From nobody Thu Oct 2 03:32:29 2025 Received: from fout-a7-smtp.messagingengine.com (fout-a7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23902302147 for ; Tue, 23 Sep 2025 11:07:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625641; cv=none; b=sSmNLLW5WugKlQy+7F/yErDSiql/b9zAnxoOn7Yd9TTQ5SH4DzRG2atRhM4+WnG460xpcxn931u6O5ZZc8oROGvOIcsoMYEUhP9CZWM9Agp2MZo5vUkAODVmqNItZzLzV+W7p5MpP1fYBkYIsH7A7npRORfrLsXuHzuWvjIPRfY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625641; c=relaxed/simple; bh=dBXptdRKA9KW6Ngd9rPNHSX/FgTaV8W/XBdmVDj1b9c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MtdXaD/hZ3+ySWI06QURIHueyhyEoIwxlC0M4RrFuv7PE86yh+JcSqurQ8lEwKpvTlINCaJbudFGgl4doUB8nQ9BU+jIP/hMKdMnuvzXjQP94M+LRK4sUiUChVEpU5Gb/xnwDbUSol7yUF7eTj7v+MwxZ+xh7C0+Y19Z2ii3NUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=SchjqWK+; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=f9y5kwR3; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="SchjqWK+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="f9y5kwR3" Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfout.phl.internal (Postfix) with ESMTP id 5B2D9EC00A5; Tue, 23 Sep 2025 07:07:19 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Tue, 23 Sep 2025 07:07:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625639; x= 1758712039; bh=Wor6PRx1QN3N2Jn0DTyxYTrVkZQE63Z4seZ89Q90skw=; b=S chjqWK+1KSkK9UwLFDRfV4UbDhPO9X3BD2D5KPkGafBjqAFI3/xKJz/aTk4Pydj0 NA8tel/EghrYm5dSgvT4O8VYL98C2aVZP4nSPIKfEX+udHzUml4EDIbCgLTnWRfG kgd3z3zU5SkOvC4xsCtaFh5J7CxQU0j6DWPuT6pU9S2xyysxzJrqVDtS6T1hqJqW Q/Z5fDWiT9O8kt0GiCbpqkcCihNVedORTQgSkXi0W9RaYhBCH+IPpmC24thrUnL0 H7BwERL31sHxzsSeiDYI+JCtvW4o07Tvm/V6kNoseh9adNXhBDtt3ys5ymlacEG0 GaSlwg9frLJ0SI3u2yS+Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625639; x=1758712039; bh=W or6PRx1QN3N2Jn0DTyxYTrVkZQE63Z4seZ89Q90skw=; b=f9y5kwR3kWPrmyRjd d4xfoymuXHvtf2tqUXImyPDrXKjSEkuYb5HAaBwpT4NKDTD/ludYfoHrkkFYQ/wt xeRacKpLSeUonOfZNe5F9Eb47xu3DfrldItqemNchQr5CEGRlM6GZ6vfwhDkykOb SJgLfWPz1u3Ilz+Q5jCcrecOHYZvQNKuBLvgSl6/IlYkO7jmMh0CyHDk4a01U6rj 5KzLRh78MAEcRFBsycWQBdyTZDY1qGKOioTBSexhg0Yw8M/E/GrCDWEljjRnCoEY 9Uw1BxPcYFrq+fPHb/ojeKyFxHsxGKMSoSrL5AfUzTj/ObUF5XO1XRed/XZAX19J MtKmA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:18 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 1/6] mm/page_vma_mapped: Track if the page is mapped across page table boundary Date: Tue, 23 Sep 2025 12:07:06 +0100 Message-ID: <20250923110711.690639-2-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau Add a PVMW_PGTABLE_CROSSSED flag that page_vma_mapped_walk() will set if the page is mapped across page table boundary. Unlike other PVMW_* flags, this one is result of page_vma_mapped_walk() and not set by the caller. folio_referenced_one() will use it to detect if it safe to mlock the folio. Signed-off-by: Kiryl Shutsemau Reviewed-by: Shakeel Butt --- include/linux/rmap.h | 5 +++++ mm/page_vma_mapped.c | 1 + 2 files changed, 6 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 6cd020eea37a..4f4659c0fc93 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -928,6 +928,11 @@ struct page *make_device_exclusive(struct mm_struct *m= m, unsigned long addr, /* Look for migration entries rather than present PTEs */ #define PVMW_MIGRATION (1 << 1) =20 +/* Result flags */ + +/* The page is mapped across page table boundary */ +#define PVMW_PGTABLE_CROSSSED (1 << 16) + struct page_vma_mapped_walk { unsigned long pfn; unsigned long nr_pages; diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e981a1a292d2..a184b88743c3 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -309,6 +309,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) } pte_unmap(pvmw->pte); pvmw->pte =3D NULL; + pvmw->flags |=3D PVMW_PGTABLE_CROSSSED; goto restart; } pvmw->pte++; --=20 2.50.1 From nobody Thu Oct 2 03:32:29 2025 Received: from fout-a7-smtp.messagingengine.com (fout-a7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A495321425 for ; Tue, 23 Sep 2025 11:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625643; cv=none; b=iYAam3g2r99S9iKsyQjtzWU1Z4OkC6tipY502DpqDYg5xJeXkwJM1HYJp+N9NtpA4XjRguAWnp7Ep84NVrul5F/q0kwE6PVE+4GwxQo/DKqfTEBQ4sVoH85czfjVk6fROJLzpb4179A15pjcXM1TgKx5ls794/T7R8uO8CQnt6Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625643; c=relaxed/simple; bh=7QCXYWOHwiU+p9/s4J6qDPLNsZQ85NJ6/lMLx1KnvP4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Am6BJOYeK575nbL3C0LtGM2LjVXeMl57NR9bdx1feeLXtyhPplJl7k97OGdfhlZ55Z1MMPsnHT3W9KHmhZm8ZQ+KoR7bNjE3TxmDulA4uuJVWBf+P2TXuXFLFUxcR66O8IWavWopKXa9UD9IiwrijREJxYdslT8v8wpnAhmE3wU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=k57Yn56F; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=RNhe5QDm; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="k57Yn56F"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="RNhe5QDm" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfout.phl.internal (Postfix) with ESMTP id 47919EC00DB; Tue, 23 Sep 2025 07:07:21 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Tue, 23 Sep 2025 07:07:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625641; x= 1758712041; bh=9h5a9glrsc1Q9FewltBTBylrIxsnIQry21VQ9ojF/aI=; b=k 57Yn56Ft7V1wdRUjQFDU16OLtJjKXC9ojXp+Xly/KRh8s+26eNrrJqXwH1D3+blc EsbPeUED693c2ckC9PcNRxqU9hWN0ht+wP6MqXnm3RkYxSTIbSFTtqMvTU1wUQAe gzZ5yc10CWXxOmkYvSqggeuJH2j/6iF9F81EskCxcTCAK0iKAkBcHAtg01wWra5d Zms63rnfsYBXBFJoI+/zZVornKHTfgfv7qLOp552azU0f8rhj96ZF7u6l4f6CSx4 x2TtEDASSbxM8cqe8LFicm6p7dXVKKCFVblLl7lqW5oQtwP5pYgFd7tjFwlnJWgW E9bX9d3rNwQxo1r2ChMKA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625641; x=1758712041; bh=9 h5a9glrsc1Q9FewltBTBylrIxsnIQry21VQ9ojF/aI=; b=RNhe5QDmEf9xtn3d0 ihdNJUhcwKphjfzODBwqVK0DRT3qzVn4HxFbUZrS84IaPQuwRIB8PhUO08D0TNpQ voIXkcBBQ95BJVGzRl6UCX2t+SYx63XQ5v2fA+HvKKbs4as+iMRkv5CUlhpK4+ag foP6nGPPEEWvPQiQdlz3f+6GSJBahvC0hvTOmWyMUkNZwCbncdljB8kD+gxdV7Sn OW0PZND/Ntl0M/4zmUT+jCkA5lD2m5dq5vwOxcgvL0LenTxkF9pIgMBeNFFPMc0P nGwJActrRumDAK/hKDxQ59Ut/wMg4MRY2Py8q+28SvudQbtXSvHQJQ3ARZWUUaJ7 KMllQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:20 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 2/6] mm/rmap: Fix a mlock race condition in folio_referenced_one() Date: Tue, 23 Sep 2025 12:07:07 +0100 Message-ID: <20250923110711.690639-3-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau The mlock_vma_folio() function requires the page table lock to be held in order to safely mlock the folio. However, folio_referenced_one() mlocks a large folios outside of the page_vma_mapped_walk() loop where the page table lock has already been dropped. Rework the mlock logic to use the same code path inside the loop for both large and small folios. Use PVMW_PGTABLE_CROSSED to detect when the folio is mapped across a page table boundary. Signed-off-by: Kiryl Shutsemau Reviewed-by: Shakeel Butt --- mm/rmap.c | 59 ++++++++++++++++++++----------------------------------- 1 file changed, 21 insertions(+), 38 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 568198e9efc2..3d0235f332de 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -851,34 +851,34 @@ static bool folio_referenced_one(struct folio *folio, { struct folio_referenced_arg *pra =3D arg; DEFINE_FOLIO_VMA_WALK(pvmw, folio, vma, address, 0); - int referenced =3D 0; - unsigned long start =3D address, ptes =3D 0; + int ptes =3D 0, referenced =3D 0; =20 while (page_vma_mapped_walk(&pvmw)) { address =3D pvmw.address; =20 if (vma->vm_flags & VM_LOCKED) { - if (!folio_test_large(folio) || !pvmw.pte) { - /* Restore the mlock which got missed */ - mlock_vma_folio(folio, vma); - page_vma_mapped_walk_done(&pvmw); - pra->vm_flags |=3D VM_LOCKED; - return false; /* To break the loop */ - } - /* - * For large folio fully mapped to VMA, will - * be handled after the pvmw loop. - * - * For large folio cross VMA boundaries, it's - * expected to be picked by page reclaim. But - * should skip reference of pages which are in - * the range of VM_LOCKED vma. As page reclaim - * should just count the reference of pages out - * the range of VM_LOCKED vma. - */ ptes++; pra->mapcount--; - continue; + + /* Only mlock fully mapped pages */ + if (pvmw.pte && ptes !=3D pvmw.nr_pages) + continue; + + /* + * All PTEs must be protected by page table lock in + * order to mlock the page. + * + * If page table boundary has been cross, current ptl + * only protect part of ptes. + */ + if (pvmw.flags & PVMW_PGTABLE_CROSSSED) + continue; + + /* Restore the mlock which got missed */ + mlock_vma_folio(folio, vma); + page_vma_mapped_walk_done(&pvmw); + pra->vm_flags |=3D VM_LOCKED; + return false; /* To break the loop */ } =20 /* @@ -914,23 +914,6 @@ static bool folio_referenced_one(struct folio *folio, pra->mapcount--; } =20 - if ((vma->vm_flags & VM_LOCKED) && - folio_test_large(folio) && - folio_within_vma(folio, vma)) { - unsigned long s_align, e_align; - - s_align =3D ALIGN_DOWN(start, PMD_SIZE); - e_align =3D ALIGN_DOWN(start + folio_size(folio) - 1, PMD_SIZE); - - /* folio doesn't cross page table boundary and fully mapped */ - if ((s_align =3D=3D e_align) && (ptes =3D=3D folio_nr_pages(folio))) { - /* Restore the mlock which got missed */ - mlock_vma_folio(folio, vma); - pra->vm_flags |=3D VM_LOCKED; - return false; /* To break the loop */ - } - } - if (referenced) folio_clear_idle(folio); if (folio_test_clear_young(folio)) --=20 2.50.1 From nobody Thu Oct 2 03:32:29 2025 Received: from fout-a7-smtp.messagingengine.com (fout-a7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06DD83218B1 for ; Tue, 23 Sep 2025 11:07:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625645; cv=none; b=X7JqIPyhL536e0owLJzmlttSJoCkcI0rxQfoa+NMb7GFwoSN5JTULsrMN7f92nKrhaJ5OhddvkYRnWbQc6ObQP1XorG9aTGmOLGW8t/8sPIWFYVqXlHW8+51WyekO3+TBTj9j071IRc0rq44or2UifNlWNF7EJbso6cCoqwwyzQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625645; c=relaxed/simple; bh=FO+dE9pku24uLN8N4NL+QT6mBqrNJmrCztRwBtxu04o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OZHYB+58nUZtrbSj2vxmilufC2SpGMeasQ6uW1H1xaLKMPKz5f41KWfmdi+jTTne5VagUe9t5o26AcJqTj1HV4uuYCncIjrbPSg1CyxBvIcJqKNftzvVM+JuLFSGp5ZlxNSDY5bnVJ1GwgDtxzK//1BIuE7Tlo742U6Y6evv5J8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=bYDfrPdI; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=GoWm7xyg; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="bYDfrPdI"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="GoWm7xyg" Received: from phl-compute-10.internal (phl-compute-10.internal [10.202.2.50]) by mailfout.phl.internal (Postfix) with ESMTP id 5A1BDEC00D6; Tue, 23 Sep 2025 07:07:23 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-10.internal (MEProxy); Tue, 23 Sep 2025 07:07:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625643; x= 1758712043; bh=BDBl7oJGV3fWpTYYjjz2yGgNlw4zY/i9bG+IwFihDok=; b=b YDfrPdIwLq3lpWq8Cp7n9DYO74ysd4WpJtuMPtFJV1iBLCNy8gx7HapavVe3MXHm RTOk95bFEv+ePlAaqUcHUnaA3pvsnDyqkQQVbncp/9BNX1gNRvPqmPV03Y/N1kEl Fd6Z3dANWDnVM9Hj9NrzKqi3swXNIMH1nuMVyr2COZJnQYOkSIG1JEgAQyCoj6tY 2Ld8OolThZ3JZG4p3rHs+GxGJuP1noXcv1VlKXrf0ytWv5gbfCCHhK6DP8vESEMq Zl1lWt+8Wr+Ix+XF79QxD7g71o+BcsORjrS85M1jtuiUAZGHSAe8s2/5ZPm33Kw0 z48EJBb6zyoMxZjh9SdVg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625643; x=1758712043; bh=B DBl7oJGV3fWpTYYjjz2yGgNlw4zY/i9bG+IwFihDok=; b=GoWm7xygmn0tkjDqZ PlF1+IkarWlbQjFHpbOFCu38GtM/aAvtp4IAWfWnyFvYZCv36BaXhCr2iGRs3IOj LRvMdIi+Goe2J7bgVLivjD0EOEHJltnuLKJ1n4ckYsgk5mlt9LG7REDvvSTVZIJS 7PxFqL0OxjshNDChp6wbgJ2UTKYZ/pWorkwlB9Z/GVDGKiJIPPhiRIUuJpUh50iv hocfaD9UHnYijdPFFS3KanNXvET14Y8+WsQxdTyNdNwr3KKpLZpDZDQ4lbytAODG Zg47EWCVG/KKuMvLcrKN2VMWmELwmQ0qq9iMUd5a11JrmTLVgrK56CLrWHYwop/3 ZukCQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:22 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 3/6] mm/rmap: mlock large folios in try_to_unmap_one() Date: Tue, 23 Sep 2025 12:07:08 +0100 Message-ID: <20250923110711.690639-4-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau Currently, try_to_unmap_once() only tries to mlock small folios. Use logic similar to folio_referenced_one() to mlock large folios: only do this for fully mapped folios and under page table lock that protects all page table entries. Signed-off-by: Kiryl Shutsemau Reviewed-by: Shakeel Butt --- mm/rmap.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 3d0235f332de..a55c3bf41287 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1870,6 +1870,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, unsigned long nr_pages =3D 1, end_addr; unsigned long pfn; unsigned long hsz =3D 0; + int ptes =3D 0; =20 /* * When racing against e.g. zap_pte_range() on another cpu, @@ -1910,10 +1911,34 @@ static bool try_to_unmap_one(struct folio *folio, s= truct vm_area_struct *vma, */ if (!(flags & TTU_IGNORE_MLOCK) && (vma->vm_flags & VM_LOCKED)) { + ptes++; + + /* + * Set 'ret' to indicate the page cannot be unmapped. + * + * Do not jump to walk_abort immediately as additional + * iteration might be required to detect fully mapped + * folio an mlock it. + */ + ret =3D false; + + /* Only mlock fully mapped pages */ + if (pvmw.pte && ptes !=3D pvmw.nr_pages) + continue; + + /* + * All PTEs must be protected by page table lock in + * order to mlock the page. + * + * If page table boundary has been cross, current ptl + * only protect part of ptes. + */ + if (pvmw.flags & PVMW_PGTABLE_CROSSSED) + goto walk_done; + /* Restore the mlock which got missed */ - if (!folio_test_large(folio)) - mlock_vma_folio(folio, vma); - goto walk_abort; + mlock_vma_folio(folio, vma); + goto walk_done; } =20 if (!pvmw.pte) { --=20 2.50.1 From nobody Thu Oct 2 03:32:29 2025 Received: from fhigh-a5-smtp.messagingengine.com (fhigh-a5-smtp.messagingengine.com [103.168.172.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37B39321F32 for ; Tue, 23 Sep 2025 11:07:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625647; cv=none; b=dwjpiXIneeSYsXXzSyzQEEIJY/JuANItMIzTzi0q+IpFAUxvhrK9Kq7UwJsqX0ncKGEYdvCgpZZzBBX1eDEkwTBVG1RALJqYxcCAhlzYAD0u0eH+U9JEceXROEx/DNh+JquDgrDZPaeNctLcG3j1vuI0tziboxkuNS7LlN1wZSc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625647; c=relaxed/simple; bh=YdfniaI8JtjbmRmFL2P1+yzqvWlKFWQCM5NeBuYYjgk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=f/YKl7wYEKeX+EXqgvQdwRYhmUW4bJsE9tHTzKHGe5d34zXALEqHmaIKWPsscGP2IEv+8+eYLraxhG2iqxkVyEwMG+gw8CCHOm1T5DIiyxrmtLwNjsxXYXSZRG8I2z8Xok+7auQAG9IRBpr4TKox/Mb6axkHYSBb/CztbYq3CwY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=n08H3Hf5; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=m0x/vk62; arc=none smtp.client-ip=103.168.172.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="n08H3Hf5"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="m0x/vk62" Received: from phl-compute-11.internal (phl-compute-11.internal [10.202.2.51]) by mailfhigh.phl.internal (Postfix) with ESMTP id 41FBE14000C0; Tue, 23 Sep 2025 07:07:25 -0400 (EDT) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-11.internal (MEProxy); Tue, 23 Sep 2025 07:07:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625645; x= 1758712045; bh=l+30FsNxlli8PpC3TpepLFbX1IePGlSZR2KUwpff5IU=; b=n 08H3Hf5JxGLXF135U68VKyoIWFff/JQy9D0ZE/ZEhu7CPWR7n0aLuH0zQHbraFYC nbboBiz7yoPXe4ZJED9Qrjc//StA3KjySXkw1OWrz2QuL8h9T0PuEZc/m+yE8IN6 r/t0Plzb56QewOKNfsXFPmTnl4uLXUjWEgXGb+BpP/Vqd2R9S5o/SiMV+wxf9h3F 5lPGzkfd2CaSdaNdcYSdZeQIwt4iriXAcUv1Kaizd1iRb7TdUII5ZgI+/nBVTtm2 VnO3iFTDylkOsbDcqkKaUcRV9qxqLkKssxM5ZtCfbOJ6TGT0RJ5NpvevUkWl9q3e qrduLtDqjogm48X6CMOig== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625645; x=1758712045; bh=l +30FsNxlli8PpC3TpepLFbX1IePGlSZR2KUwpff5IU=; b=m0x/vk62Pnle6U75M ta4YLB6sTiAwci7bsRD/ozX8OCX/XwOrDvUBr1ezdC/8F5HpUkaiw2JapAdbII6k 8s83X+kI9Xa9nbeqUv4kxskTh+Am2cN6XiYe/O0h6UxIYfAcUHXlzaQcmGj3ci0n 7ASQ76H4xEm3imy2E/o7KSCqbqOhSOEEoCmQyt81xak5G0sJ9GlIAI4EEmDkJ30R Xq12nB/Xw6SILiKbnrxlxabfOXtGx2ktJ5UCOJz99F77/SL7JDNtz/f3iT7YLfsC uBaj2tj4ohy3cPgdNAO+ULuSKGXE1yVyS2Iyt8QYNRLmdDiKsj/94pmv/SvYKevo J/sRA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:24 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 4/6] mm/fault: Try to map the entire file folio in finish_fault() Date: Tue, 23 Sep 2025 12:07:09 +0100 Message-ID: <20250923110711.690639-5-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau The finish_fault() function uses per-page fault for file folios. This only occurs for file folios smaller than PMD_SIZE. The comment suggests that this approach prevents RSS inflation. However, it only prevents RSS accounting. The folio is still mapped to the process, and the fact that it is mapped by a single PTE does not affect memory pressure. Additionally, the kernel's ability to map large folios as PMD if they are large enough does not support this argument. When possible, map large folios in one shot. This reduces the number of minor page faults and allows for TLB coalescing. Mapping large folios at once will allow the rmap code to mlock it on add, as it will recognize that it is fully mapped and mlocking is safe. Signed-off-by: Kiryl Shutsemau Reviewed-by: Shakeel Butt Reviewed-by: Baolin Wang --- mm/memory.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0ba4f6b71847..812a7d9f6531 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5386,13 +5386,8 @@ vm_fault_t finish_fault(struct vm_fault *vmf) =20 nr_pages =3D folio_nr_pages(folio); =20 - /* - * Using per-page fault to maintain the uffd semantics, and same - * approach also applies to non shmem/tmpfs faults to avoid - * inflating the RSS of the process. - */ - if (!vma_is_shmem(vma) || unlikely(userfaultfd_armed(vma)) || - unlikely(needs_fallback)) { + /* Using per-page fault to maintain the uffd semantics */ + if (unlikely(userfaultfd_armed(vma)) || unlikely(needs_fallback)) { nr_pages =3D 1; } else if (nr_pages > 1) { pgoff_t idx =3D folio_page_idx(folio, page); --=20 2.50.1 From nobody Thu Oct 2 03:32:29 2025 Received: from fout-a7-smtp.messagingengine.com (fout-a7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2A9A321F4C for ; Tue, 23 Sep 2025 11:07:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625649; cv=none; b=AdFu0YiLcrx5jNFWhM2UuQ0b71vWVWRkix6QLy1UI5Sv1juiwbAvlb/A0ShVJqKLmxdWN0LkbmVhDieBoSdl6IaOMI6HOnetFWqkW4LP18ZweMqAK7x+iuRdonX6iJlJ/TKP/XwB7PNFPYYGOocnAs6ytr/vK7WsHzodGyavvwY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625649; c=relaxed/simple; bh=lafj3u5MfFxC2O+ZOStOOH0oQz92l1iYxy2O9Q0JtL4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WE6ajZ5XGwuWT/zEK4mC9oGp5KKEU3ojCDDPdP7z42LdhZbC+KdL4LaRYmANfXaYsrNZpsxNOaLXFgb7UnyzZTbqcfZ70qZIr5FvO8Tv6nYu4U491olAT/a/j3HRH6fEXK7RHLHjE/S9xbAm0+TZ8Y7Qg7Vorh+1qIYHKG1DBDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=Eze+Ws0a; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ZTi2hcEK; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="Eze+Ws0a"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ZTi2hcEK" Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfout.phl.internal (Postfix) with ESMTP id 0127AEC00DA; Tue, 23 Sep 2025 07:07:27 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Tue, 23 Sep 2025 07:07:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625646; x= 1758712046; bh=jFlzJ6561wUPMtQD7f6sb4ugc+4wKUHkJSRo02E7hiw=; b=E ze+Ws0aoQB5hiu7xli1pwslSt17+MJcyNGgvrBHKLt47m4/Md6BjcySKTgOB++nC xA3ZbtSTPwFhnMLkMrT06I69isCgR8NWfWZz30qZOnB8xO0bFI8myTXExeXKyfGH m8sjTajuOZ2x/5irzu/Nwy8evqicCQRMNbvvq0qAha2Ls9OaawCHlCK8MUSPc6sn S3XblAcUsfnXoD3FtVJc3r1rckEfeu4PejzaKaeL+GK9YvAqWa4NHvmi1J/EXdRD dVVPFj/omcTcqlGjzMVhy/STMkwzNZzIapZfawd/KrWzV6o2zkUdaXW0UXanTyoj 6k150cvLysz/gzATxgkCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625646; x=1758712046; bh=j FlzJ6561wUPMtQD7f6sb4ugc+4wKUHkJSRo02E7hiw=; b=ZTi2hcEK11wqhlduw qtK9TkufynYkoGXDdG37857a57gfcuUX3y/iEibPSryT5Aq1Hjsa+HOzcBLj31qg i9NohoVPqio9QBMrjxupABty1nZl1Wk8BB1rNTHkaIKg6pHhEAy1cxq27bP9XdX5 mqdDpc+Kjr7xV5/FBi1tqs8z/uTMk9HJZgBpd5uiKss8JIw/Iuf3MKAt7jaI3jh7 2Wtd4x2f41kpxwiJPvX6rRJoRHLA0hp2oDBt+pyys0v1mhGpkrVbPQConU0GNIes x6Kf04NyeBIuAu+KQBq8r2Cn3YZTVBGsNYgG0AZYYmISEwE8HXbnO9ObcxEuW1po vjCRA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehjecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:26 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 5/6] mm/filemap: Map entire large folio faultaround Date: Tue, 23 Sep 2025 12:07:10 +0100 Message-ID: <20250923110711.690639-6-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau Currently, kernel only maps part of large folio that fits into start_pgoff/end_pgoff range. Map entire folio where possible. It will match finish_fault() behaviour that user hits on cold page cache. Mapping large folios at once will allow the rmap code to mlock it on add, as it will recognize that it is fully mapped and mlocking is safe. Signed-off-by: Kiryl Shutsemau --- mm/filemap.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 751838ef05e5..26cae577ba23 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3643,6 +3643,21 @@ static vm_fault_t filemap_map_folio_range(struct vm_= fault *vmf, struct page *page =3D folio_page(folio, start); unsigned int count =3D 0; pte_t *old_ptep =3D vmf->pte; + unsigned long addr0; + + /* + * Map the large folio fully where possible. + * + * The folio must not cross VMA or page table boundary. + */ + addr0 =3D addr - start * PAGE_SIZE; + if (folio_within_vma(folio, vmf->vma) && + (addr0 & PMD_MASK) =3D=3D ((addr0 + folio_size(folio) - 1) & PMD_MASK= )) { + vmf->pte -=3D start; + page -=3D start; + addr =3D addr0; + nr_pages =3D folio_nr_pages(folio); + } =20 do { if (PageHWPoison(page + count)) --=20 2.50.1 From nobody Thu Oct 2 03:32:29 2025 Received: from fout-a7-smtp.messagingengine.com (fout-a7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0050B322545 for ; Tue, 23 Sep 2025 11:07:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625651; cv=none; b=gOQbuZYnr6EEwTXF2IKfItDQYiosMbw5Bnhpw7GIiEBr7yk/pNX9IaU1RKHLp0wR5So9N+WFuegRc2iIKwmzxGJIW0Ph+laNl9qfvVm/ma+JoFZ5UrONy+k9mzkMIinGws5JOjOUECzPmdzK9Pm0BwEu+sElhfoMIZRWFYLKKU0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758625651; c=relaxed/simple; bh=zgThalph9y47lmEr32v6T0NLRVPpwukomwHn+9xtsDg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dk639sh0hCcH29132ZrFZswt+RfuQUwd6eqjIvkRTXEWisGZJgAkrp2fvp/FXJY1brhkCvSZYy82EaiJVQEXoc10jFlBS/BbVBIeRdT/2GN1cItGzUZTqRYB/0w68uOyoa5ElioWHHLBhRsLy4Wt0yzqWO9nJNKIt/EJpYMS0cI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=eBrpSZeD; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=PrhcBeG1; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="eBrpSZeD"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="PrhcBeG1" Received: from phl-compute-07.internal (phl-compute-07.internal [10.202.2.47]) by mailfout.phl.internal (Postfix) with ESMTP id 22A6FEC00DF; Tue, 23 Sep 2025 07:07:29 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-07.internal (MEProxy); Tue, 23 Sep 2025 07:07:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm3; t=1758625649; x= 1758712049; bh=gBJQwyFG/mBuy6u3kCYzOX5EZriO4heRBy+k+i4RgPY=; b=e BrpSZeDYT2luvIzf8C71a4BIoX44rg3deXZlRby5//Bqvi979R9rF8Rwykh7k3HY NDMBxDgxrom/QJk5RM0L+JywaBEWbQRVwMQ9szXMAdsMA5vbxzYuCuT2BcwPzG/9 A5KbR+5eEtKnJus7x2o0el/ud1Fqszxt0/evfJroj10y+EnptwPBWaqv8tCjmoEP TyBqC2cjJgTuG6AdrwcCwA6YNu2HsKQGxxmXM2yK8ZLQw+osTrx76ibs37rVdHN0 ilVg+/r9k/aQddk/X1vRmYsufIsUaUm8XDKj3rGEd2SCyPMsj7Mmjvx8yBBnQTAh zh89YKO0BCL34/5OoBZ/g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1758625649; x=1758712049; bh=g BJQwyFG/mBuy6u3kCYzOX5EZriO4heRBy+k+i4RgPY=; b=PrhcBeG11yAJoPQux np5WlJHQZ2qybfofYscuCW9VHC3wik4Eqp24AFrZTaPXGC7Hac6YrbWjn72rwBr0 aemGpmDjqZzD8SbIzquUN+S8hI3sXv4GYRjmJYKJdLwnAAuJ8thiaPltP907Uj6T um7EXgcYD7DSemAfDTP/1hHcLAyhY5vhMysM23XTphozrhayUVQ9Ty2Cz4P482a9 DRoSrjix1yL28pGQJlz4DMjhydUitelwNZNb4yhAs1UAj3zmAGyyNQiRoqjIfeFz Mk9aM3pGgn7hgbVDzgCjE3MFbYkAYUVcCaCRP+elZxfvG7ncdy2Tp4r1kBltgsVo i1GyQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeffedrtdeggdeitdehiecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpuffrtefokffrpgfnqfghnecuuegr ihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjug hrpefhvfevufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefmihhrhihlucfu hhhuthhsvghmrghuuceokhhirhhilhhlsehshhhuthgvmhhovhdrnhgrmhgvqeenucggtf frrghtthgvrhhnpeegveehtdfgvdfhudegffeuuddvgeevjefhveevgefhvdevieevteei vdehjefhjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhroh hmpehkihhrihhllhesshhhuhhtvghmohhvrdhnrghmvgdpnhgspghrtghpthhtohepudek pdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrkhhpmheslhhinhhugidqfhhouh hnuggrthhiohhnrdhorhhgpdhrtghpthhtohepuggrvhhiugesrhgvughhrghtrdgtohhm pdhrtghpthhtohephhhughhhugesghhoohhglhgvrdgtohhmpdhrtghpthhtohepfihilh hlhiesihhnfhhrrgguvggrugdrohhrghdprhgtphhtthhopehlohhrvghniihordhsthho rghkvghssehorhgrtghlvgdrtghomhdprhgtphhtthhopehlihgrmhdrhhhofihlvghtth esohhrrggtlhgvrdgtohhmpdhrtghpthhtohepvhgsrggskhgrsehsuhhsvgdrtgiipdhr tghpthhtoheprhhpphhtsehkvghrnhgvlhdrohhrghdprhgtphhtthhopehsuhhrvghnsg esghhoohhglhgvrdgtohhm X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 23 Sep 2025 07:07:28 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Hugh Dickins , Matthew Wilcox Cc: Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Rik van Riel , Harry Yoo , Johannes Weiner , Shakeel Butt , Baolin Wang , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Kiryl Shutsemau Subject: [PATCHv3 6/6] mm/rmap: Improve mlock tracking for large folios Date: Tue, 23 Sep 2025 12:07:11 +0100 Message-ID: <20250923110711.690639-7-kirill@shutemov.name> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20250923110711.690639-1-kirill@shutemov.name> References: <20250923110711.690639-1-kirill@shutemov.name> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kiryl Shutsemau The kernel currently does not mlock large folios when adding them to rmap, stating that it is difficult to confirm that the folio is fully mapped and safe to mlock it. This leads to a significant undercount of Mlocked in /proc/meminfo, causing problems in production where the stat was used to estimate system utilization and determine if load shedding is required. However, nowadays the caller passes a number of pages of the folio that are getting mapped, making it easy to check if the entire folio is mapped to the VMA. mlock the folio on rmap if it is fully mapped to the VMA. Mlocked in /proc/meminfo can still undercount, but the value is closer the truth and is useful for userspace. Signed-off-by: Kiryl Shutsemau Acked-by: David Hildenbrand Acked-by: Johannes Weiner Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes Reviewed-by: Baolin Wang --- mm/rmap.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index a55c3bf41287..d5b40800198c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1462,12 +1462,12 @@ static __always_inline void __folio_add_anon_rmap(s= truct folio *folio, } =20 /* - * For large folio, only mlock it if it's fully mapped to VMA. It's - * not easy to check whether the large folio is fully mapped to VMA - * here. Only mlock normal 4K folio and leave page reclaim to handle - * large folio. + * Only mlock it if the folio is fully mapped to the VMA. + * + * Partially mapped folios can be split on reclaim and part outside + * of mlocked VMA can be evicted or freed. */ - if (!folio_test_large(folio)) + if (folio_nr_pages(folio) =3D=3D nr_pages) mlock_vma_folio(folio, vma); } =20 @@ -1603,8 +1603,13 @@ static __always_inline void __folio_add_file_rmap(st= ruct folio *folio, nr =3D __folio_add_rmap(folio, page, nr_pages, vma, level, &nr_pmdmapped); __folio_mod_stat(folio, nr, nr_pmdmapped); =20 - /* See comments in folio_add_anon_rmap_*() */ - if (!folio_test_large(folio)) + /* + * Only mlock it if the folio is fully mapped to the VMA. + * + * Partially mapped folios can be split on reclaim and part outside + * of mlocked VMA can be evicted or freed. + */ + if (folio_nr_pages(folio) =3D=3D nr_pages) mlock_vma_folio(folio, vma); } =20 --=20 2.50.1