From nobody Fri Oct 3 13:32:07 2025 Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C76518EB0 for ; Sun, 31 Aug 2025 09:01:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756630883; cv=none; b=mtrGm+o6NZJ+jkGuyauYKoVTyXVqSUQCqq074f9zlVWMZiAGk+dCtuyK1165KaWbmprtiuSCT0qHG3yV8desaOP7HofiKwHVyBTtdBtmJ7kTkfEiWNVGIyy+vEXK6l5SACUMIeCGX5swDV1JPwiws1UOZ6I4v5ubbrZWBPgWMck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756630883; c=relaxed/simple; bh=Uf1uVShRR8vNo6b19Nwa8ZjRQHJBlZANUYBbgbNySJc=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=tWQLGuSpaHf0url3Hoi+C9XKORbl2rMksJtKqZ4Iy3j0Bdmgzd0ycp5rZms4BBURfrz8Ura27l63F1Hi6T2EjJlqgvYg3GcoQKg9WUnYe/YAhIAu7+MGOtiLBfe1n5bR4O7QRN+xVN+pe/jeIpKl7ST7A10DQO0PTci1nBPs1+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MweJnoqR; arc=none smtp.client-ip=209.85.219.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MweJnoqR" Received: by mail-yb1-f173.google.com with SMTP id 3f1490d57ef6-e970acf352fso2726689276.2 for ; Sun, 31 Aug 2025 02:01:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756630881; x=1757235681; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Xzarj1jXPwwihpFc52AKHrUnyJlvNlaBP1cTsUIn51w=; b=MweJnoqRJCCnU4EZSExqdmuCUvE5cCEpoGJHVkXdz2mZj5Y8P7QIsxuXTMy08lbWpd 5tZ2JjQ07dDs3TvnT3ovKhox7+C/bohaDU4hy8NMKq+gmqiuCstkgliSd6IPdQrZtvii Z7e7sjHzEhGUMnycDDOssHOmfIkNZK+UyL2URKnMQXIGu24nNBjolRyAeNWJ4DC+VgEw clModHgajeAZYFcGR+fV+A2YUkcL/yURsX+sgjuublYS/pH87xszyhHLcwLEa1RBrr2r 10yQQfxnZrJax1CJtwbPoE2pHJhupiH9rwlG+5AOFWvrwROZjX5geGejJ04FVWdGxGKr wGSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756630881; x=1757235681; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xzarj1jXPwwihpFc52AKHrUnyJlvNlaBP1cTsUIn51w=; b=o9lHdwHU1rjh0O22Pz7plixP2DfsAbh3+xYc1jB9QlUJWvKGNopK7fgZlDhJjZ0Y4I F7ugMkCb6D/aGrpA/GPi70nkUqEdTAul/EvIarlRWw9g6Xq3DOEv8qnUNqlNjw4iMSfM 2hfwmV1VbktU3HTFsv7XfQPUGukdrOe7OtYjroimPjZic4/alYHcDIQAXrv7FPsT+O4m Jf2YIOU6A9X1U/KSE3DUMJ2J10HZgZ+cGxkYN9TC3MSVhkLf0bA20Q72gSMKT3E8+eTq mYkQSNsJ2fz71+QsVG0v+erdRLKs2NROkQvaCDHBaKkyN1f75/SJkxZ3DhahoiwFzjHe eiVA== X-Forwarded-Encrypted: i=1; AJvYcCWiha1NM1ulM+0ETRmxmBi+EM3y/Hjugxwsior5EsGIEm8vonzkafvBWDa6eB06ceTG7ahnZe5L4nw2zYo=@vger.kernel.org X-Gm-Message-State: AOJu0YwsQUVqHDZDE2GjJyZ8HH+8BFZdeS4jMOz/wjr7EqN1TzgkiwwS rKhqgBSjFQzXlOg8pkI2BSUQS5QLm9Sca8/cz9D3PstwV7sc8n6Qrw8ts0j8TqdGkw== X-Gm-Gg: ASbGnctwb/mRgnGTKB4U8hpQxutiv8A87JP2QwfjGcjGzzfVH8f0uKdUKvIK3EwPlet K6dMkASOLadN46ps3GdrMfV+42QYzbOrKqOtnZ7sZ4D/9YSXKePA1GVME8v1zdqbeNyhpehQCk1 2ggmHlWoclUNK/YzIOAr6y01Yo65ZB6yKe4yFYgi64pHtEqgalQH8LnK7HhhxG+RgemOw2p2TL4 apuf7MHgDhTsmyVD6Id/AXVZ4hQlRFnxA/0FhZSXkgl3FPq5NrZK11Lt0yxUgKdpBcyIutALNXh vT2mEQmdyRkyyWB4HXXH3FRNkwnq/8sfY4rMk9QzLFj7FT/v6TkCXjJUnD5x+m0WMzVvbbov6XZ B/HngE7XKOcgy5ebzuEU8g1ojtcVYhzMbLkmZYwffCURShATpIrDubUBzhiutzQ5i5a7ubz9al2 6JW0pl4T+tl9y9Tw== X-Google-Smtp-Source: AGHT+IGpRZTOsrv6OWnEdL6eZ2XWJIrkP0gSOahOzxZcUPGdu6xAkpdJXXS2GNMzOEVXcPVlQWvbAQ== X-Received: by 2002:a05:6902:158f:b0:e97:276:400d with SMTP id 3f1490d57ef6-e98a583baf1mr5704848276.36.1756630880977; Sun, 31 Aug 2025 02:01:20 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e98ac4458c7sm985933276.8.2025.08.31.02.01.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:01:19 -0700 (PDT) Date: Sun, 31 Aug 2025 02:01:16 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/7] mm: fix folio_expected_ref_count() when PG_private_2 In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" 6.16's folio_expected_ref_count() is forgetting the PG_private_2 flag, which (like PG_private, but not in addition to PG_private) counts for 1 more reference: it needs to be using folio_has_private() in place of folio_test_private(). But this went wrong earlier: folio_expected_ref_count() was based on (and replaced) mm/migrate.c's folio_expected_refs(), which has been using folio_test_private() since 6.0 converted to folios(): before that, expected_page_refs() was correctly using page_has_private(). Just a few filesystems are still using PG_private_2 a.k.a. PG_fscache. Potentially, this fix re-enables page migration on their folios; but it would not be surprising to learn that in practice those folios are not migratable for other reasons. Fixes: 86ebd50224c0 ("mm: add folio_expected_ref_count() for reference coun= t calculation") Fixes: 108ca8358139 ("mm/migrate: Convert expected_page_refs() to folio_exp= ected_refs()") Signed-off-by: Hugh Dickins Cc: --- include/linux/mm.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1ae97a0b8ec7..ee8e535eadac 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2180,8 +2180,8 @@ static inline int folio_expected_ref_count(const stru= ct folio *folio) } else { /* One reference per page from the pagecache. */ ref_count +=3D !!folio->mapping << order; - /* One reference from PG_private. */ - ref_count +=3D folio_test_private(folio); + /* One reference from PG_private or PG_private_2. */ + ref_count +=3D folio_has_private(folio); } =20 /* One reference per page table mapping. */ --=20 2.51.0 From nobody Fri Oct 3 13:32:07 2025 Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 929C52153E7 for ; Sun, 31 Aug 2025 09:06:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631163; cv=none; b=jYN3EG7EKiqnut9rhvNxjB8XqLGZFLwTjs0Z6QOBb/FakzUheLyym6LT5uZkYwRPHGgJeCmAmKi3r09lerPns2CdlyN1GEjo/k3HVjiqqBeP+GSWDDKb2nvrwDcAqZ7e0s+gXavT88ZsGcWiAFAdyaIoYCERtIDW/smb44IzR10= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631163; c=relaxed/simple; bh=5k0V3kHQiFP9hnipv1O54p2piqAq7RtCdMmFkFOKElk=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=dnIaVvyFwyKEsM7VdSPbThv020w724EENKz8pWTaTsGwNIAxAU1x2+/spy5kXwVyAby5sQNkeSUpNRIhp9pwV3L+u9MNt/CAoYFY9GRdWy8Dw45GVrvz65coLShHf9olvHP1THxE5nvWzd62M8NK7boDXSZ+rsLZvTEjmbeYb6w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AexfybYf; arc=none smtp.client-ip=209.85.219.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AexfybYf" Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-e951dfcbc5bso3008938276.3 for ; Sun, 31 Aug 2025 02:06:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631160; x=1757235960; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=cewv29th9nlXePhk/EOfBMtGSb2YntDYMH/m1gOAqlw=; b=AexfybYf3eEG2qHErqDOItmdKhSwBE9Bs2Ap50VUmUiK8XW17CDt1uI0ubviefD5B6 GDEUaHuZ1P0OmfCmk4NLwjonxvX+E3ebkQ8IJfk2z7nUMm5tPk0SL6U9X+L62Bcb4tGT Ds3pOH6InyRkfzEN0HpBqDpSN8KIxyGBMcLyzv2bwW4VTEH12LryTi3KAJ4SHqNRBnGz m48z9f7PMzN2sgLevtFc2tmLGEVpFdAaMop1XqFnehBc+4D6YTapxKfTMQcKwYuajkD8 gy1loaNVLaKpdi+winm3F4xaA4fjTgQgLj/ikLx9Rr0wwjqjfD5TE9GPIOYMk5IR5QIF R1Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631160; x=1757235960; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cewv29th9nlXePhk/EOfBMtGSb2YntDYMH/m1gOAqlw=; b=pBYKfqdKi+U9bXRLzuxqeQLJXulxF6cn3iqYExNruqNkhH2Hb0rwd3bbYyA77H2v88 2DfA2HUJA5zh308N0fzHEY7Hd+pmXawDiRzDeSGHSONfUQ/DdYFsteNLZZ2FKZIxSx/Z yTarfydUhWUcEWqZwb4oufi21/8VzeUtXsMXR386dOY4KjOTYZZVgtrN97diNzlaeazg RWgYDNd+ynu+rGnM6eljchj2e23W6y376WA1qvMgBYF3LPtATBWXWbvyU/imhu1Ihjdi nU86HS4qXb7BLeX23G3ZhqbljhOiQAG/Un3Q7P5JIvzVtcnBaIOb7pIarxZq9DmYs/g6 yaQw== X-Forwarded-Encrypted: i=1; AJvYcCUja9owVsW9EYx4IZqVNqPmbr6wl4hvtIobaU11NeqC6w5RiHjUtL/1WBZALVYbnHviDqZP9DI4xgtrUbs=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/vPBJivNTXF/zeORUFU5+hFWLcaAUczY7uwLLgOAIEd35PE3d Wx0Q5m+M9xdvX+YAOxk70g8GnW2cupDWUCIfutkbnraCfR0y4R7HdG5zBy5E+HXCnA== X-Gm-Gg: ASbGncta4R0m4IQAs1IqN+Z8XIKBx91PUuJh4uobJKCtPYmmirFDW0cCIDCCjZ2mi2g J7n3+j1L6MePa6MkTikzs1Rx1Z/+YGQcN8Gfe53BQUzcYTDNsDWCYcrlXhp3aO3o5m5+9Z1TNY4 MevzFpXrs/uIUnCN76rSG+Iq55nn2vn+NQm3zIASXXJR7VSGBBwXmLF9ZuIfWzp6kIlV8exOS2+ CIefIfQzmKkKLaZOF3AZckvKfcscgU3lc9NmnzE2UvxHtz/6Xe3xGvk6PqILo/PaX+R6i5u+qir 3aOWT461T+FlNCUf4BYfWfEMYLjexSIGm/MlB5i/Q3+DBKJcKesX8BeCCGVkGELqxWzLf0b9dXz 48Svk0Srrl5kdDTGhz/2THNxmudgft/zERnM7ywbiwlbcMdMgCWp0LNoiIyJvt0vuIVWTNMXGtF xUwU9czkYUaj50BP44OV11ALF7 X-Google-Smtp-Source: AGHT+IHZuc+H+mCEQgTcTD59v7WFBtA0Muo5YsHOfJFQ+r6CmvOZ6yRPYLQ9LbxJpAeGrP/U6wGvXQ== X-Received: by 2002:a05:6902:2490:b0:e96:efc6:8392 with SMTP id 3f1490d57ef6-e98a5851c2amr4633775276.43.1756631160112; Sun, 31 Aug 2025 02:06:00 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e98ac41ffa1sm982501276.3.2025.08.31.02.05.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:05:59 -0700 (PDT) Date: Sun, 31 Aug 2025 02:05:56 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/7] mm/gup: check ref_count instead of lru before migration In-Reply-To: Message-ID: <47c51c9a-140f-1ea1-b692-c4bae5d1fa58@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Will Deacon reports:- When taking a longterm GUP pin via pin_user_pages(), __gup_longterm_locked() tries to migrate target folios that should not be longterm pinned, for example because they reside in a CMA region or movable zone. This is done by first pinning all of the target folios anyway, collecting all of the longterm-unpinnable target folios into a list, dropping the pins that were just taken and finally handing the list off to migrate_pages() for the actual migration. It is critically important that no unexpected references are held on the folios being migrated, otherwise the migration will fail and pin_user_pages() will return -ENOMEM to its caller. Unfortunately, it is relatively easy to observe migration failures when running pKVM (which uses pin_user_pages() on crosvm's virtual address space to resolve stage-2 page faults from the guest) on a 6.15-based Pixel 6 device and this results in the VM terminating prematurely. In the failure case, 'crosvm' has called mlock(MLOCK_ONFAULT) on its mapping of guest memory prior to the pinning. Subsequently, when pin_user_pages() walks the page-table, the relevant 'pte' is not present and so the faulting logic allocates a new folio, mlocks it with mlock_folio() and maps it in the page-table. Since commit 2fbb0c10d1e8 ("mm/munlock: mlock_page() munlock_page() batch by pagevec"), mlock/munlock operations on a folio (formerly page), are deferred. For example, mlock_folio() takes an additional reference on the target folio before placing it into a per-cpu 'folio_batch' for later processing by mlock_folio_batch(), which drops the refcount once the operation is complete. Processing of the batches is coupled with the LRU batch logic and can be forcefully drained with lru_add_drain_all() but as long as a folio remains unprocessed on the batch, its refcount will be elevated. This deferred batching therefore interacts poorly with the pKVM pinning scenario as we can find ourselves in a situation where the migration code fails to migrate a folio due to the elevated refcount from the pending mlock operation. Hugh Dickins adds:- !folio_test_lru() has never been a very reliable way to tell if an lru_add_drain_all() is worth calling, to remove LRU cache references to make the folio migratable: the LRU flag may be set even while the folio is held with an extra reference in a per-CPU LRU cache. 5.18 commit 2fbb0c10d1e8 may have made it more unreliable. Then 6.11 commit 33dfe9204f29 ("mm/gup: clear the LRU flag of a page before adding to LRU batch") tried to make it reliable, by moving LRU flag clearing; but missed the mlock/munlock batches, so still unreliable as reported. And it turns out to be difficult to extend 33dfe9204f29's LRU flag clearing to the mlock/munlock batches: if they do benefit from batching, mlock/munlock cannot be so effective when easily suppressed while !LRU. Instead, switch to an expected ref_count check, which was more reliable all along: some more false positives (unhelpful drains) than before, and never a guarantee that the folio will prove migratable, but better. Note for stable backports: requires 6.16 commit 86ebd50224c0 ("mm: add folio_expected_ref_count() for reference count calculation") and 6.17 commit ("mm: fix folio_expected_ref_count() when PG_private_2"). Reported-by: Will Deacon Link: https://lore.kernel.org/linux-mm/20250815101858.24352-1-will@kernel.o= rg/ Fixes: 9a4e9f3b2d73 ("mm: update get_user_pages_longterm to migrate pages a= llocated from CMA region") Signed-off-by: Hugh Dickins Cc: --- mm/gup.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/gup.c b/mm/gup.c index adffe663594d..82aec6443c0a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2307,7 +2307,8 @@ static unsigned long collect_longterm_unpinnable_foli= os( continue; } =20 - if (!folio_test_lru(folio) && drain_allow) { + if (drain_allow && folio_ref_count(folio) !=3D + folio_expected_ref_count(folio) + 1) { lru_add_drain_all(); drain_allow =3D false; } --=20 2.51.0 From nobody Fri Oct 3 13:32:08 2025 Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE2C329A1 for ; Sun, 31 Aug 2025 09:08:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631313; cv=none; b=UBpl/lC0SVWzEdy+5VJTTeyu8QtF+Hj8S9zXxVPrMM61u2OhAlXbGCBnKQ2JkqUto7EaNQgTin/WsjyaObN9AnX7WZ/bJEnaByPhVrsCPd7dkW57aCO3VQbH6ORDRqq6vM26akZnOFVSZOBpp9cKmPkXOTiR0sPXhQSjR1GPEdE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631313; c=relaxed/simple; bh=F9zzDfAQ89ufKt+J1vm3QtfSIESD8Dr5qFKQljoOTSU=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=M1JWEgYxHFK5h94fF4HPaxAYVLxfNYFqTKK6LNvhbA8NuMqGwH0j/Z9crjozzEDgpJD6nu1tZp4CvxTvWVnODZd0VMkOraGWWUm+8ZhSZjQeMOBlHLZlm+YAR65nWTPoOVINCq1K4MC3iZyjJhKvox7cU3aPCwol6codOsq2dyI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=d01foLdL; arc=none smtp.client-ip=209.85.219.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="d01foLdL" Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-e970b511046so2599112276.0 for ; Sun, 31 Aug 2025 02:08:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631311; x=1757236111; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=u88nEX4VhKEhfqfy6sgABj7KaLHVYd51Cg9xPrUQsPg=; b=d01foLdLQuFOZkk0hNyaXds5qnMmBqhWqqnrrQzfa7RQT/O8cctngogxVFVSWTBRF5 mkWm5zOMA49uDCRCP6tObxTKdOh14wyGhrqJFxXilE8sFamz4NsGepltno7EyLSIFVUS L6xvSEmL9h6gzfZ0a9dd7bVCUZ8TMrNw9Ky7ZKc/0bhFBBYiaubF7bs5oYZp6uh5bHli fHDTvVoGmFRrleWMnyIhoMcps5caLPD7RnDIxN8S5ZMg81FHpH+ZXhKMjgLKeaQ7g0qd HXNJD4csfq7YJlxSjazyvqu8d+hg8l/DVlk8K+k6swlUdQhl51RGBIYRJ2ivQ0m0D9zG w0yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631311; x=1757236111; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=u88nEX4VhKEhfqfy6sgABj7KaLHVYd51Cg9xPrUQsPg=; b=cmloegVIqRVnq36XWskGEFWK0tmx8zsiCk0u+YDnM0lo6mNqSHgHeYUVGGRVzbh4w7 DT8tb0ZTbepOJZEipYrZU8YDmwO3V3mTr2g0x7tdKHh6cH4+HgoUYuBJ/3jeBnTB7FC+ fuemugMPdxoNqkENLqJUtJinzjPSwn5M8B+Koh6zRNW8GWraZ34+BlyP+mvn5gP4gYiF ymrHVvFnzmg5hPk0rkpOL7lrXalL7LDvZ/yEwW3txWygSjADsCvsmBi3B+4pWPpQbpBb KBVaVwXtr5+mEC/mqNzPN7m8uqrWYWj9N15gql491kwfQJXPTkj6fUnglR8oPuUd4mC8 PIAg== X-Forwarded-Encrypted: i=1; AJvYcCVSjfpAVVgBLWXeCK6rLzCEJpNq+KdZotwPuaHQDQK9BVg/nyXXf2HcryZjExph9lhOT/aFZXcInlL5hiA=@vger.kernel.org X-Gm-Message-State: AOJu0Yz5MsxUeBlLPhStK6s4kDhOuSX1vNS6rAQBnMcMjwwikLDW0suD /04EWpx4GQ4H151E6Pmvvl7Gwhj5QJMeISs694BRTXaVfA6W7C0Qu0NQf8d2QJd8Ow== X-Gm-Gg: ASbGnct+ZZbR3/JfEHap5zIIkMlOvpZPUJ/fMtub9gDRmH5mSTnQIhn3s3UyEPMsYLB bbNBDfEOgEV8y/c9pyWyuknXoabXQoeP+JalRd0Z6T5F2vJJyljwhFdcr3woKL4LoZIaYGiIhIe Fx0Ajn022WJSccH+vjCOh5CvQN8IEHfmb7Rit4bJHMzQC7P3U4b6VqBqSiLdrzxfx1LOxO/ksj/ atjVYmwTm6Gx5Mh8sl8jhBW9SGXtwFfNjwqIy7W5ZWyHEOqA4ObpMA2gPcUHQUG/JfLDQiT2oQf /CqFv7pQ9fJ3P0MKCPrQztTHHMALX8GEarlVlM3FtZzLHV4AhEf05gS+4JrURtmSJUeRGwGe0xa quBOXpmk5evq4D6BL49Ke3Ki6/PIAvvTHx1XuOnNKdVstBa0nsHAzYVjEtJcfeFLPfarXhNAbsB Z6J8Arpc0rymSYHQ== X-Google-Smtp-Source: AGHT+IF4uqF7l2E0FZ9hMKOQYbXmnG/oF/j3dZTuq2FVbemX/WEQ1z/BcNuLmVYRYcxAG32KEwnQnA== X-Received: by 2002:a05:690c:490b:b0:721:6adc:4364 with SMTP id 00721157ae682-722764cb124mr44113587b3.33.1756631310488; Sun, 31 Aug 2025 02:08:30 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7227d8ccb38sm7148137b3.35.2025.08.31.02.08.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:08:29 -0700 (PDT) Date: Sun, 31 Aug 2025 02:08:26 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/7] mm/gup: local lru_add_drain() to avoid lru_add_drain_all() In-Reply-To: Message-ID: <165ccfdb-16b5-ac11-0d66-2984293590c8@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In many cases, if collect_longterm_unpinnable_folios() does need to drain the LRU cache to release a reference, the cache in question is on this same CPU, and much more efficiently drained by a preliminary local lru_add_drain(), than the later cross-CPU lru_add_drain_all(). Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Note for clean backports: can take 6.16 commit a03db236aebf ("gup: optimize longterm pin_user_pages() for large folio") first. Signed-off-by: Hugh Dickins Cc: --- mm/gup.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/gup.c b/mm/gup.c index 82aec6443c0a..9f7c87f504a9 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2291,6 +2291,8 @@ static unsigned long collect_longterm_unpinnable_foli= os( struct folio *folio; long i =3D 0; =20 + lru_add_drain(); + for (folio =3D pofs_get_folio(pofs, i); folio; folio =3D pofs_next_folio(folio, pofs, &i)) { =20 --=20 2.51.0 From nobody Fri Oct 3 13:32:08 2025 Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 845EC14EC73 for ; Sun, 31 Aug 2025 09:11:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631502; cv=none; b=AgI9qSGvrE6O8gpWj3yt/38vzfIKMIht3USAHvFEzCUqA+cRlbvEzACP30sKVuCbHGLmC8wQ40BC7MMTagrc1IZQYG1s6RVUg5uXxc8iaSNvjtrJV0N2PkmDyKD2XIjbJWNc/5vWsKrtu5D/2gbK7FxMcJ9VyKvnm1qtn0+M+6Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631502; c=relaxed/simple; bh=81H2Wd9Bqcrfj5XROHevTWrdilBjpd3Q3RLsRn9FPPQ=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=P1cQSAv+tBD+qR73MTLGEyjJmct76UzDfl7S42X48mt6YfdhooHCXUAdAK8WX/Kmyl5tpHcqgEZyCNT2/Ko2B9IVLd+i02Nl8Izdzn+bFYS4rLfqfZXmEXmakqykbqGGf67NxZJfxdp/U+MSpUgdP2/Z4g3n8YIBkHWMly9nasY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EC6MDCy8; arc=none smtp.client-ip=209.85.219.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EC6MDCy8" Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-e96ff16fea1so3168317276.0 for ; Sun, 31 Aug 2025 02:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631499; x=1757236299; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=/75wRAcyAz/pvibic7S8snC2YyoDmLe6V6AOatHHWEE=; b=EC6MDCy8KZ6ALojsysQ7X8yrYeD4quyA1iDEML2XyBtDagRdgBOwKmrFmamANiqBRv BVEIfYVsJUN+sf4MA8W2UUvQf5C/xcacI1KeZvqz0f15zfqF3gqBL5fScv29wyzrcMAY YwGGKBtSJLiNpwP8FjyrBEhfLjSA3r8TmGmd4YkTaIlOS4DGFdb0yniB7GRou0rJP0Ka xsqFhSklpwTvwI5mv+2p3FRpBi7+P3wHmNb0NrNI9u3ccFbapHwduNLAaj+2MDcGfTAr TPWmSXNW/DnGGmPX0SL/TNwzho4t7QF1BWx7wT/bxBV8uPIsaa97TjDmxP+KpycRdyUE stag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631499; x=1757236299; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/75wRAcyAz/pvibic7S8snC2YyoDmLe6V6AOatHHWEE=; b=iOTsTa0OfytCVI/DGmZw4G9Oech/l/OqXbJzyZA5EKN/zEfUCfNnsqaS7zW1L/3cHZ lj8678I5rbSYWFh/arZ2YHWGILDchYX/jjW9fO/xZqQyfGEu3RXGFtIi1qkTqEJ+et51 1TBzRt1CaNv1iWbwkmAAQPaYn87zRmRNxHqM7fl5fO7FEst6G7kO/z/X0Tt8D7w9gv6w JEhylsqluCrF0Siym+oG01FSk+lXx1GZwE6Mxk//srL2FwlGE/z/NRjny76+zphZ+NUV Q/0nqqF+EKR3eFwfZkPiXeTpheclsm3nuLMF4j0rezJU/m22bdJpblGGzOkxWigC90EH jLlw== X-Forwarded-Encrypted: i=1; AJvYcCWyQyETYJRK+EB9xbhbU+PHwvMqJVpTTNzcBvv1JwemAyfbgkKM3mUD8QBH1XeBtmVDIq7jDRiDN3yBFFI=@vger.kernel.org X-Gm-Message-State: AOJu0YwOj3mtIQ6nWF97FmkS5JonrXCR7Q2B/xdKxTD1qzLuaBb3tdJ5 Zyd+mBTC27Ej7AU4ViRzRf/ZeyHfE8AkWOLFKliTZVwL66ZX/LdNKoDQ3414kyUyeg== X-Gm-Gg: ASbGnct+y3VTia/y2vkNaze3+ycSTuQY4QIJfFadMGzFAWyAuwZ/SHcZJS15GmXjvxl 8qaGk1Wt+ZMO5MzHxGNdjjyroXEoZ+vHtAvE7r8TpCk9Exf/fVuMkJetV4T+DHXALIgm6ccEFK+ 4bXGW2/HS3XpfxEgq3upl9R0rbzUd2Gyplv7gvE7IVrI37lHNmHBKGvV4PIwgcAGW/qSk3jZn+I K3rXqFFChB2ZTt1rofK986X/ffFLRyg8VirOpd/zZAaIvRAUYx6w2rA0CBaJr1N3h5ZiaL6KCe0 7EhffQ0wfqvDxIWyaHEnfBujOKN57PdfuYq/C6rvdYD/LfK1ok/RWQd4ccerWVLlZHFm+CkhsvU w//PivvZgzXjxKLriY5bO9ryzrrt2xoxyxXhXpy0vTpLS18ccrQ+n6nxSQ/dVKEPXv/x5gicRMz U9H9AcX8y8lX4CC/O+LdhqtT6+ X-Google-Smtp-Source: AGHT+IGLKGhwhu8szWt1srW9NOv/rN3Wnl8fJgvJJ/o/OhNDl3cOqhK0q/ihGoG3VMxSEEyhguVe3A== X-Received: by 2002:a05:6902:4387:b0:e97:fef:d53e with SMTP id 3f1490d57ef6-e98a5787930mr3875920276.17.1756631499134; Sun, 31 Aug 2025 02:11:39 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e98ac580f35sm988530276.27.2025.08.31.02.11.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:11:36 -0700 (PDT) Date: Sun, 31 Aug 2025 02:11:33 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/7] mm: Revert "mm/gup: clear the LRU flag of a page before adding to LRU batch" In-Reply-To: Message-ID: <0215a42b-99cd-612a-95f7-56f8251d99ef@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9: now that collect_longterm_unpinnable_folios() is checking ref_count instead of lru, and mlock/munlock do not participate in the revised LRU flag clearing, those changes are misleading, and enlarge the window during which mlock/munlock may miss an mlock_count update. It is possible (I'd hesitate to claim probable) that the greater likelihood of missed mlock_count updates would explain the "Realtime threads delayed due to kcompactd0" observed on 6.12 in the Link below. If that is the case, this reversion will help; but a complete solution needs also a further patch, beyond the scope of this series. Included some 80-column cleanup around folio_batch_add_and_move(). The role of folio_test_clear_lru() (before taking per-memcg lru_lock) is questionable since 6.13 removed mem_cgroup_move_account() etc; but perhaps there are still some races which need it - not examined here. Link: https://lore.kernel.org/linux-mm/DU0PR01MB10385345F7153F3341009818882= 59A@DU0PR01MB10385.eurprd01.prod.exchangelabs.com/ Signed-off-by: Hugh Dickins Cc: Acked-by: David Hildenbrand --- mm/swap.c | 50 ++++++++++++++++++++++++++------------------------ 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 3632dd061beb..6ae2d5680574 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -164,6 +164,10 @@ static void folio_batch_move_lru(struct folio_batch *f= batch, move_fn_t move_fn) for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 + /* block memcg migration while the folio moves between lru */ + if (move_fn !=3D lru_add && !folio_test_clear_lru(folio)) + continue; + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); =20 @@ -176,14 +180,10 @@ static void folio_batch_move_lru(struct folio_batch *= fbatch, move_fn_t move_fn) } =20 static void __folio_batch_add_and_move(struct folio_batch __percpu *fbatch, - struct folio *folio, move_fn_t move_fn, - bool on_lru, bool disable_irq) + struct folio *folio, move_fn_t move_fn, bool disable_irq) { unsigned long flags; =20 - if (on_lru && !folio_test_clear_lru(folio)) - return; - folio_get(folio); =20 if (disable_irq) @@ -191,8 +191,8 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, else local_lock(&cpu_fbatches.lock); =20 - if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || folio_test_large(fol= io) || - lru_cache_disabled()) + if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || + folio_test_large(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 if (disable_irq) @@ -201,13 +201,13 @@ static void __folio_batch_add_and_move(struct folio_b= atch __percpu *fbatch, local_unlock(&cpu_fbatches.lock); } =20 -#define folio_batch_add_and_move(folio, op, on_lru) \ - __folio_batch_add_and_move( \ - &cpu_fbatches.op, \ - folio, \ - op, \ - on_lru, \ - offsetof(struct cpu_fbatches, op) >=3D offsetof(struct cpu_fbatches, loc= k_irq) \ +#define folio_batch_add_and_move(folio, op) \ + __folio_batch_add_and_move( \ + &cpu_fbatches.op, \ + folio, \ + op, \ + offsetof(struct cpu_fbatches, op) >=3D \ + offsetof(struct cpu_fbatches, lock_irq) \ ) =20 static void lru_move_tail(struct lruvec *lruvec, struct folio *folio) @@ -231,10 +231,10 @@ static void lru_move_tail(struct lruvec *lruvec, stru= ct folio *folio) void folio_rotate_reclaimable(struct folio *folio) { if (folio_test_locked(folio) || folio_test_dirty(folio) || - folio_test_unevictable(folio)) + folio_test_unevictable(folio) || !folio_test_lru(folio)) return; =20 - folio_batch_add_and_move(folio, lru_move_tail, true); + folio_batch_add_and_move(folio, lru_move_tail); } =20 void lru_note_cost_unlock_irq(struct lruvec *lruvec, bool file, @@ -328,10 +328,11 @@ static void folio_activate_drain(int cpu) =20 void folio_activate(struct folio *folio) { - if (folio_test_active(folio) || folio_test_unevictable(folio)) + if (folio_test_active(folio) || folio_test_unevictable(folio) || + !folio_test_lru(folio)) return; =20 - folio_batch_add_and_move(folio, lru_activate, true); + folio_batch_add_and_move(folio, lru_activate); } =20 #else @@ -507,7 +508,7 @@ void folio_add_lru(struct folio *folio) lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) folio_set_active(folio); =20 - folio_batch_add_and_move(folio, lru_add, false); + folio_batch_add_and_move(folio, lru_add); } EXPORT_SYMBOL(folio_add_lru); =20 @@ -685,13 +686,13 @@ void lru_add_drain_cpu(int cpu) void deactivate_file_folio(struct folio *folio) { /* Deactivating an unevictable folio will not accelerate reclaim */ - if (folio_test_unevictable(folio)) + if (folio_test_unevictable(folio) || !folio_test_lru(folio)) return; =20 if (lru_gen_enabled() && lru_gen_clear_refs(folio)) return; =20 - folio_batch_add_and_move(folio, lru_deactivate_file, true); + folio_batch_add_and_move(folio, lru_deactivate_file); } =20 /* @@ -704,13 +705,13 @@ void deactivate_file_folio(struct folio *folio) */ void folio_deactivate(struct folio *folio) { - if (folio_test_unevictable(folio)) + if (folio_test_unevictable(folio) || !folio_test_lru(folio)) return; =20 if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(fo= lio)) return; =20 - folio_batch_add_and_move(folio, lru_deactivate, true); + folio_batch_add_and_move(folio, lru_deactivate); } =20 /** @@ -723,10 +724,11 @@ void folio_deactivate(struct folio *folio) void folio_mark_lazyfree(struct folio *folio) { if (!folio_test_anon(folio) || !folio_test_swapbacked(folio) || + !folio_test_lru(folio) || folio_test_swapcache(folio) || folio_test_unevictable(folio)) return; =20 - folio_batch_add_and_move(folio, lru_lazyfree, true); + folio_batch_add_and_move(folio, lru_lazyfree); } =20 void lru_add_drain(void) --=20 2.51.0 From nobody Fri Oct 3 13:32:08 2025 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4211A7D07D for ; Sun, 31 Aug 2025 09:14:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631642; cv=none; b=qeefcPx0GEU8hRu0TPvAJUzMl7CAYkrgHBgMPztuoDvvYPPq76DCv0X8arlU85tJECqQ3R/IlHYN0drQh2Yh+iUBBcEi2iXJ1tKZhNtxSSMa/BFWUVe0daigLqy6rUpgCpym0V2JG/DcUu2CcTyzJc1rJD5+wtLnm6mqKB6Frqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631642; c=relaxed/simple; bh=jlQej32j0n7qXEnY5V9OT4gzWzDjJMbxjwZlVZev3a4=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=HavnmVDLxSRGwlSsQFzBK8c722dkpUO+Nmo2qD4VDRPCFpr6+cJwJqBxtqDPIlltUVr98Cxc5qOxMUUZfApqIztkiwMtuxA573PuWFFENGah+/VQuYdv72f8j69vcFV11jNO7JJB8qiVWHl1U+VlDY9UmAo/Cj3flJ5H7pNkYKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=X0BZ/KLu; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="X0BZ/KLu" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-72290cfcf5eso769777b3.3 for ; Sun, 31 Aug 2025 02:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631640; x=1757236440; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wBznXUHFnxO7JFFRA+cSHz8XvSsYLiyW87D5PciK1Do=; b=X0BZ/KLuLmmiKWiPrD/+fqJFrsCeGdwUqFj3PlSxxJR+8GKczAgkcW2FLkbVhJGfHf 35Sz5yya5i7x7y34b/+dU3HCEq8+w5n6pxpZlTcGac6kbkgLF81oVYx7eUIZIwOEUrZZ vYNrlbF4M/zOutuF1PCFOiMgMYBqqaRzDMLdjFS23QutDNOJh2RDqAvWDltdmb3Ey0Sx g9V+iQgH2E3/VnnRFdTo6DSV5MNRfZrqrFd9njPWUQFcqgbXTRjtISksmGQ0Feilu5rC xa0xpInBNPvV7JSpmIsL1tLt3Xs5twyLTgIW9NYkikoPRHUFSYatlx+/M0yxxqkrbHaR ASIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631640; x=1757236440; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wBznXUHFnxO7JFFRA+cSHz8XvSsYLiyW87D5PciK1Do=; b=JZEtYlpxhHMj+MDHn3L34IfZkSUtfGeshroL4Um063Rj1GY4Sau+Q5jz2iCtkqk4GO 8dLeJcllt+FDzSQaRhmdnWv8UgqlFOgmRMSbuqlRsrfLdk8lix42ZFF7QpMsXADcPWvW tyqYFbkveYmdJr5A0UphDRPOV/IycjXFAUhBYjuaJPrEAhr3zPMOI/ZmhGVAmXJjeQqI vAzjb8dZLi53+PK79XtJsz7Q+SY964fhSDQEBiRiiD0iR4wUd4S9T5WG8YIKXlziUV9U Cjz9H86B2SS7L3Q+vrsFTwejaNZL7Pvs+Vv0oY2952nJj9llrv4+IIEyWznimI+PNyZD meqQ== X-Forwarded-Encrypted: i=1; AJvYcCVHrxKXHwBmk5W3nzUVwaq9YIv/xM+DS+m4NCf6UVsTna95bvsy3Pka/6oe0gstMT8dOHh7mZMBcaXiasE=@vger.kernel.org X-Gm-Message-State: AOJu0Yxqfk83jIGTEyZ/3lLFfVsgiFbRME83HBMT97aS4Asn6TsncE3Q KE4AzQUGbcTnMCJuDwPZfs3jkrRosbFIgHsOZiqrksKxJ4ZGOI7kYulsO2zkDfUbsA== X-Gm-Gg: ASbGncsU52qBa+YJSF5yIghSb9JTC3OvuJNuVCSlO+0P9zzCswJBehhvHh+NGLZSaMA 2vViQlElT3CNpnQ7K6IPmHStgB+XEoBaqHAkScM9J8x/3zbpzmNuVzYewD13mHgv5HbwA26DPMt oAoccDHfQ6hCt/UIfDUIC1Uc4yIMKY2HtvGqbVaNT6Vyl4xka/MWZiZTCfr2Ozu174A3L9mK97Q niS0JMl5UuloRYIT+IIahHm3MB3j5cyrxezAWk3mh5dXD3WMKwuKN8ktKui2vwS40SzJZconn9p oURiFQUaDf+xHfcwgCrbZL0IkBLNGYk0TkZlNMPj0Owq3pU5NMuycvkddeXvb12YMrfKV01PTGU 0DLy1JlZgTmBGdI0zfgQ2bMmJ4lg4TA4Zb2SwdRx1ONoNurK8O61w6v67q26Ln3uaUju1uyUOyY 2VYgt7n4CHH7MONi36NN3fIkeemdvbB40e4KQ= X-Google-Smtp-Source: AGHT+IF4yPQEPLuQmGbE0pb+TIdmkIJQyKnxChThhXPpgreHU7hHADg7CNY3Cer/6aWfeZDp+UC4KA== X-Received: by 2002:a05:690c:6908:b0:721:369e:44d8 with SMTP id 00721157ae682-7227657dc4emr49185577b3.51.1756631639950; Sun, 31 Aug 2025 02:13:59 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7227d67377esm7100867b3.33.2025.08.31.02.13.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:13:58 -0700 (PDT) Date: Sun, 31 Aug 2025 02:13:55 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 5/7] mm: Revert "mm: vmscan.c: fix OOM on swap stress test" In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This reverts commit 0885ef4705607936fc36a38fd74356e1c465b023: that was a fix to the reverted 33dfe9204f29b415bbc0abb1a50642d1ba94f5e9. Signed-off-by: Hugh Dickins Cc: Acked-by: David Hildenbrand --- mm/vmscan.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a48aec8bfd92..674999999cd0 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4507,7 +4507,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c } =20 /* ineligible */ - if (!folio_test_lru(folio) || zone > sc->reclaim_idx) { + if (zone > sc->reclaim_idx) { gen =3D folio_inc_gen(lruvec, folio, false); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; --=20 2.51.0 From nobody Fri Oct 3 13:32:08 2025 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27A62239E6C for ; Sun, 31 Aug 2025 09:16:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631792; cv=none; b=eqgZRtuiPOiyLPo/Z5o9dDc2+eZMdkrNE+izKydcyOkHPLtZwpfnkROdwszLSJZa+exYtCZftu2/7AhdfbmYgpC6h+UgLQ54qrwVZ86OiU+NBSIrVyx3/HIfC7Z5fgyCSerOxE9mn+ZfWhWqEaUTJc+rIzVanotg95FmQBqLDVw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631792; c=relaxed/simple; bh=autrDhEoQElahzTFBsIJy35lsPr42mb7YMvpvkEhLLY=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=fLXNvRllZIpA6UtlWczEDJKkOrcoBZeiPP0BoOFIhsvbYPN/Y+p+0sQuQ8somU6QvHKG0lash18813Pl4p6ERiNlTbDplnK+KiKYcV/QIqKse6KmAhIASWwUKWAwh96fVO7IF8SNpxClhM7Z9ZcdHi+bFu329SCsBdWSVsdtjf4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OW4Ir8g1; arc=none smtp.client-ip=209.85.128.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OW4Ir8g1" Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-71d60504788so27623657b3.2 for ; Sun, 31 Aug 2025 02:16:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631790; x=1757236590; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=fVj1TJZLZFaRr3ag7GRilGcLLKtSMUW0HsbTS6/7Wj4=; b=OW4Ir8g11eW4eLMIHBNHnuI49qhipqOLSgIGumxGFQu7qOG6HV++LaUxc5SXuKgMBF Pp3R9yEF9RII8/GhGjIMhHHhLSsyJCXR6s4V5WfQfIBe4nXIyP5YDWzXrXnpiZqfGYuY hil9I+H36sVDCHNoYTqAiRVOlN4eljPCRgpq+L5Wd8kJWQLs1+qAvHvAOCyJdkwZUxTe KKYcvirUoxE18VZoRmzWRbWhMrXRKmRFUPsyj32DGC/MPGaNNFkTA76R1GWC7DJ+LQCT VCI3ZPn3pqfS3idRhGJsitW1FRnRo7g4g6Z7HH243lXREn4mIT/d7+ODHfAK7r/XhmSt Ht5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631790; x=1757236590; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=fVj1TJZLZFaRr3ag7GRilGcLLKtSMUW0HsbTS6/7Wj4=; b=InjcDyq1fZ/2iLnHVpFArJx9K1MT1ImW0uiPYOMlTafgFQUnB6BV010DvTiR7LSMY7 YTGUKcIgnCshE1g9y6hibvz37fkwUONMUz7WmZhF/6PqqfOmf2g0gqIA1jgUmQMTgD17 btSpBfosJbT8+1BGP3NXBGzLnfi6V5rgqoxLr5cAcizLpY43ABJyhtMo1YPLyzSH7eR5 6CzcIFa4495vxHs+u+SEs1Xj2XNNPjkLSeN+vFIeH0esJOKm1AhpLIAelY9If8O7Owb4 p8EJBT0qdKYbINo7dXLXL6oQoVmmbdjy8uoKFcCJjox3EdfWJdzlj6laiuz2ewZw95zG 0fuQ== X-Forwarded-Encrypted: i=1; AJvYcCVddKAbgiOrv1m8/av/LR2RePXlkfn0ErGFPhLHKUwnAD5du+SUT9i36I+L/mZYASxHUje0NrkMo1i3siU=@vger.kernel.org X-Gm-Message-State: AOJu0YzTZ6Q7Mpou/NwvW1S9QyLWlAYJHRzG6qAZVAif+oFxiYPdOREB IGEO1A3i1nCSsTeHpIUx8w5Qg/BkxaJYcWZCxVyWMQJo/gQVkrbI0zWYj7mP4OqSIQ== X-Gm-Gg: ASbGncsqVHKimAIdQAGl9r9RWYzanD+B5qJjdhRojGshzUMPFiG6fHMd7SL5zkGXpT4 ios1Zh6rFHD8jw3CDwuACVA9+edtWMCi006wMn43FgGsHa0EgrDMa+UF5b+3pvH06hEL/tIkQVB qUcRkgbLbySpsHKVNohPVM+BBfqIrQr+cnwfXXTdbTe0trJYa50+mCEzImXNzMcuHuM6K307ssz YmM5ewAiJ6qnxj6SXWLcFW5K1ACuLabXPAq9i7qiAdXeUuAkhRac5JuyAPtxYxC7z+Gk28VmSIb eOv4M5cmVxoF6K72//VtgCV2Fjp3sx5reLkWWWrII6GmnQNOVyv1LmYrOwHd5/duH7wgacAtfr4 D7ZBXENutK82k98K/e9lVb783PqXgKDv/MZfH3gd/p9mcF4SFeu6modh1wcnVFePLTgPe4ZEBXl 4azYnugoVz3gCCPbT/eZw4kPfM X-Google-Smtp-Source: AGHT+IEvYKErI6ke0DEYK4ijjvJ+cTpg/4z2kMEoxFGpg8eCQJg+xtsOc8H11hahONN9PzGte6TGbg== X-Received: by 2002:a05:690c:e08:b0:71f:b944:1019 with SMTP id 00721157ae682-7227658e0a3mr47176947b3.52.1756631789908; Sun, 31 Aug 2025 02:16:29 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7227d634d22sm7255777b3.26.2025.08.31.02.16.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:16:28 -0700 (PDT) Date: Sun, 31 Aug 2025 02:16:25 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 6/7] mm: folio_may_be_cached() unless folio_test_large() In-Reply-To: Message-ID: <861c061c-51cd-b940-49df-9f55e1fee2c8@google.com> References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" mm/swap.c and mm/mlock.c agree to drain any per-CPU batch as soon as a large folio is added: so collect_longterm_unpinnable_folios() just wastes effort when calling lru_add_drain_all() on a large folio. But although there is good reason not to batch up PMD-sized folios, we might well benefit from batching a small number of low-order mTHPs (though unclear how that "small number" limitation will be implemented). So ask if folio_may_be_cached() rather than !folio_test_large(), to insulate those particular checks from future change. Name preferred to "folio_is_batchable" because large folios can well be put on a batch: it's just the per-CPU LRU caches, drained much later, which need care. Marked for stable, to counter the increase in lru_add_drain_all()s from "mm/gup: check ref_count instead of lru before migration". Suggested-by: David Hildenbrand Signed-off-by: Hugh Dickins Cc: Suggested-by that's making you uncomfortable, we can delete it. --- include/linux/swap.h | 10 ++++++++++ mm/gup.c | 5 +++-- mm/mlock.c | 6 +++--- mm/swap.c | 2 +- 4 files changed, 17 insertions(+), 6 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 2fe6ed2cc3fd..b49a61c32238 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -385,6 +385,16 @@ void folio_add_lru_vma(struct folio *, struct vm_area_= struct *); void mark_page_accessed(struct page *); void folio_mark_accessed(struct folio *); =20 +static inline bool folio_may_be_cached(struct folio *folio) +{ + /* + * Holding PMD-sized folios in per-CPU LRU cache unbalances accounting. + * Holding small numbers of low-order mTHP folios in per-CPU LRU cache + * will be sensible, but nobody has implemented and tested that yet. + */ + return !folio_test_large(folio); +} + extern atomic_t lru_disable_count; =20 static inline bool lru_cache_disabled(void) diff --git a/mm/gup.c b/mm/gup.c index 9f7c87f504a9..e70544c0f958 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2309,8 +2309,9 @@ static unsigned long collect_longterm_unpinnable_foli= os( continue; } =20 - if (drain_allow && folio_ref_count(folio) !=3D - folio_expected_ref_count(folio) + 1) { + if (drain_allow && folio_may_be_cached(folio) && + folio_ref_count(folio) !=3D + folio_expected_ref_count(folio) + 1) { lru_add_drain_all(); drain_allow =3D false; } diff --git a/mm/mlock.c b/mm/mlock.c index a1d93ad33c6d..427339dea380 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -255,7 +255,7 @@ void mlock_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_lru(folio)) || - folio_test_large(folio) || lru_cache_disabled()) + !folio_may_be_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } @@ -278,7 +278,7 @@ void mlock_new_folio(struct folio *folio) =20 folio_get(folio); if (!folio_batch_add(fbatch, mlock_new(folio)) || - folio_test_large(folio) || lru_cache_disabled()) + !folio_may_be_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } @@ -299,7 +299,7 @@ void munlock_folio(struct folio *folio) */ folio_get(folio); if (!folio_batch_add(fbatch, folio) || - folio_test_large(folio) || lru_cache_disabled()) + !folio_may_be_cached(folio) || lru_cache_disabled()) mlock_folio_batch(fbatch); local_unlock(&mlock_fbatch.lock); } diff --git a/mm/swap.c b/mm/swap.c index 6ae2d5680574..17438fd1f51a 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -192,7 +192,7 @@ static void __folio_batch_add_and_move(struct folio_bat= ch __percpu *fbatch, local_lock(&cpu_fbatches.lock); =20 if (!folio_batch_add(this_cpu_ptr(fbatch), folio) || - folio_test_large(folio) || lru_cache_disabled()) + !folio_may_be_cached(folio) || lru_cache_disabled()) folio_batch_move_lru(this_cpu_ptr(fbatch), move_fn); =20 if (disable_irq) --=20 2.51.0 From nobody Fri Oct 3 13:32:08 2025 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C11A34C79 for ; Sun, 31 Aug 2025 09:18:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631926; cv=none; b=eW06/JgISliEfKO0r/FWQrjOcjQStNvKTHHV/koy4bjcIaPKWlQV2PEONEdTuzKeZE3q1SrggQG1tjlELOz270O1ipxbIwlexH5DHn2T/6K0Hhz5V/PH44Y4OMDZe+zJTMkGbLS0RYTYU27g0p0sL/+AwwNfaidwFnfPJFHIdFA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756631926; c=relaxed/simple; bh=+NuMP3LEn2b85FBHF78bjPjh3jGPBvBI0QJhiJVBQo4=; h=Date:From:To:cc:Subject:In-Reply-To:Message-ID:References: MIME-Version:Content-Type; b=aAWN0R2dNfax0ldT7RgI/w9mH+0igG8RweFH+w8mTbeqs6mzEdwETvuCWqDUCXkoQWI0gdbXtQUA32/3jXiVZLELUXkRGkutBiz1J6Cb7dYFNGGu2sYwb1LTbxHq6/ZEc9RGNSoz0CGyDRBzefx1wStK8hgStOdRHt9Tkj2YwpM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZNiKo4Sj; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZNiKo4Sj" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-71d71bcab69so26906717b3.0 for ; Sun, 31 Aug 2025 02:18:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1756631924; x=1757236724; darn=vger.kernel.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=2Wrk7aQbqVPPZjpKsq8t4LHCu/svkZR7eKtEEBTzldw=; b=ZNiKo4SjBddh6DbW7utMihAlHRqco9mBT4nP/t/eRtsxlZQSwgFxuGbz4vfrxxnXjx BNFaucOdyeqjvlH/IyYCjzGMCTUzHP9Wld4Jmk0hsLJ/cuT/PZxGcIjGv8vjK29X8n4S uyz0LlrjeA2acK9UdUb6N4+uhB0PLsAlzyd/CY8gi4YuRnTo60SkLU4McGxECouTjOl+ GpvVhBaKhLFk7+dMaDfUuf83s4irCcpqJruBY88M9Z/klGrnx8mx3VXbPWMJgVqpeuOq 3Rz/KyHfRiHRL43993nLUodayeiTDWxslHpTL/RpLax2zTnpxCF7Gfpn52/h0l0kZgXX TX5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756631924; x=1757236724; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2Wrk7aQbqVPPZjpKsq8t4LHCu/svkZR7eKtEEBTzldw=; b=vHUskB93bzh6el6Bl5ATZdYgmJIWuwyGWpF6Mc3yu/sO4qPMAHG7YVkNdcVegVJFFb BhKKSxKZxvN2hA4b43gzNR+tScnFQVtEGYs+CRBTL/vffDs6l+WyPf8cYTiqoyzqmyl1 /A5GLeUAw3g03BLkuUesQjBLcb11WbmDJHPZ/KCc/0Sponu2Zs1Sd0adFLmHax4gn5Fs 7WxKznJShDo8her7GTYA2RIxEyrkCyCZeoXsylvo+gHmq2kXqv4EoItiEhQBFEZrOW9Q l1sPyFkdpKHxzn+oichLk3EvlaXLjABTC5TuCzOIfQGXvpPWLO3WdgZKbDMETvyT+InJ gZhg== X-Forwarded-Encrypted: i=1; AJvYcCUTOet9xbYigOvhgsIBydHJNu15J4e9OdTKeU9aOK1yyJ882UkELPpr9IPIKK2t49Dqj5LQX5Ne5hKgjpc=@vger.kernel.org X-Gm-Message-State: AOJu0YyfdHO1/tk2NhuN5eGpiNt4mlZq/HqnMqbEVRyCFS9UMwH5/JWm iWCEHCuxeaq5wXMITnWmnDeyl2BfiupmjcjH71N0Be6tRiVsPN+58jMFy36eNsGJeA== X-Gm-Gg: ASbGnctPPGhrx2oUBYjg4KR5fOXB+qFuJ8oDdAdLysUmM+v3bKx99VQBb09Fr4iZJAd XNUclAbquBgnXeczrLQUOaIlIhp/R6fTci/Fe6udhhBK9a+pQ+B2PQIm8sp4lb2gmaRW0jgv1VY z0qc8QYDAPjG3Dv6ze0s9Srl19CxNBIWnaEnArYYT0X0MAd/82HBleMOI02+Y2T/6VIEKGCYF6W ng/gUtHiufIOBqIOpqW4epNrOzUvIqaAhOQlBce4q+nhyGXkJim8TKj27NCFI0q96ZgI3N79vUH nxT9VgDS23CbmL9lF2Wvf1O0ZPe18GfKOggqZtsgxOnxSVdBp387zdaUzisndkSZbA2DP4SLHNj +W0CAtBWJM9lzAwI8uyIImerxSiR4D3SGi4S5mH+AP2wBq5E1fU4aJagTkaDD1GHqBu2dOozqgE oB+W9kSnNL3flBKw== X-Google-Smtp-Source: AGHT+IHx6UhN3JFnLmA4ryZLPJj6CdNI7qFb0bgmOG1cBhBLd0nyN429j6qnz3FhDBHZpgqIj4sp6Q== X-Received: by 2002:a05:690c:9a0f:b0:70e:142d:9c6e with SMTP id 00721157ae682-722764f3c47mr40781987b3.32.1756631923476; Sun, 31 Aug 2025 02:18:43 -0700 (PDT) Received: from darker.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 00721157ae682-7227d8cc1c0sm7065297b3.34.2025.08.31.02.18.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 31 Aug 2025 02:18:42 -0700 (PDT) Date: Sun, 31 Aug 2025 02:18:39 -0700 (PDT) From: Hugh Dickins To: Andrew Morton cc: Will Deacon , David Hildenbrand , Shivank Garg , Matthew Wilcox , Christoph Hellwig , Keir Fraser , Jason Gunthorpe , John Hubbard , Frederick Mayle , Peter Xu , "Aneesh Kumar K.V" , Johannes Weiner , Vlastimil Babka , Alexander Krabler , Ge Yang , Li Zhe , Chris Li , Yu Zhao , Axel Rasmussen , Yuanchu Xie , Wei Xu , Konstantin Khlebnikov , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 7/7] mm: lru_add_drain_all() do local lru_add_drain() first In-Reply-To: Message-ID: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No numbers to back this up, but it seemed obvious to me, that if there are competing lru_add_drain_all()ers, the work will be minimized if each flushes its own local queues before locking and doing cross-CPU drains. Signed-off-by: Hugh Dickins Acked-by: David Hildenbrand --- mm/swap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/swap.c b/mm/swap.c index 17438fd1f51a..4951f4d35e40 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -834,6 +834,9 @@ static inline void __lru_add_drain_all(bool force_all_c= pus) */ this_gen =3D smp_load_acquire(&lru_drain_gen); =20 + /* It helps everyone if we do our own local drain immediately. */ + lru_add_drain(); + mutex_lock(&lock); =20 /* --=20 2.51.0