From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516892193216.21539990804263; Wed, 15 Oct 2025 01:28:12 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143201.1476959 (Exim 4.92) (envelope-from ) id 1v8wrO-00068C-LW; Wed, 15 Oct 2025 08:27:50 +0000 Received: by outflank-mailman (output) from mailman id 1143201.1476959; Wed, 15 Oct 2025 08:27:50 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrO-000685-I9; Wed, 15 Oct 2025 08:27:50 +0000 Received: by outflank-mailman (input) for mailman id 1143201; Wed, 15 Oct 2025 08:27:49 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrN-0005tz-C3 for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:27:49 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id d51649b1-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:27:48 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EE70F1A9A; Wed, 15 Oct 2025 01:27:39 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EB26A3F66E; Wed, 15 Oct 2025 01:27:42 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d51649b1-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 01/13] powerpc/64s: Do not re-activate batched TLB flush Date: Wed, 15 Oct 2025 09:27:15 +0100 Message-ID: <20251015082727.2395128-2-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516894994154100 Content-Type: text/plain; charset="utf-8" From: Alexander Gordeev Since commit b9ef323ea168 ("powerpc/64s: Disable preemption in hash lazy mmu mode") a task can not be preempted while in lazy MMU mode. Therefore, the batch re-activation code is never called, so remove it. Signed-off-by: Alexander Gordeev Signed-off-by: Kevin Brodsky --- This patch was originally posted as part of [1]; the series was not taken but this patch remains relevant. [1] https://lore.kernel.org/all/cover.1749747752.git.agordeev@linux.ibm.com/ --- arch/powerpc/include/asm/thread_info.h | 2 -- arch/powerpc/kernel/process.c | 25 ------------------------- 2 files changed, 27 deletions(-) diff --git a/arch/powerpc/include/asm/thread_info.h b/arch/powerpc/include/= asm/thread_info.h index b0f200aba2b3..97f35f9b1a96 100644 --- a/arch/powerpc/include/asm/thread_info.h +++ b/arch/powerpc/include/asm/thread_info.h @@ -154,12 +154,10 @@ void arch_setup_new_exec(void); /* Don't move TLF_NAPPING without adjusting the code in entry_32.S */ #define TLF_NAPPING 0 /* idle thread enabled NAP mode */ #define TLF_SLEEPING 1 /* suspend code enabled SLEEP mode */ -#define TLF_LAZY_MMU 3 /* tlb_batch is active */ #define TLF_RUNLATCH 4 /* Is the runlatch enabled? */ =20 #define _TLF_NAPPING (1 << TLF_NAPPING) #define _TLF_SLEEPING (1 << TLF_SLEEPING) -#define _TLF_LAZY_MMU (1 << TLF_LAZY_MMU) #define _TLF_RUNLATCH (1 << TLF_RUNLATCH) =20 #ifndef __ASSEMBLER__ diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c index eb23966ac0a9..9237dcbeee4a 100644 --- a/arch/powerpc/kernel/process.c +++ b/arch/powerpc/kernel/process.c @@ -1281,9 +1281,6 @@ struct task_struct *__switch_to(struct task_struct *p= rev, { struct thread_struct *new_thread, *old_thread; struct task_struct *last; -#ifdef CONFIG_PPC_64S_HASH_MMU - struct ppc64_tlb_batch *batch; -#endif =20 new_thread =3D &new->thread; old_thread =3D ¤t->thread; @@ -1291,14 +1288,6 @@ struct task_struct *__switch_to(struct task_struct *= prev, WARN_ON(!irqs_disabled()); =20 #ifdef CONFIG_PPC_64S_HASH_MMU - batch =3D this_cpu_ptr(&ppc64_tlb_batch); - if (batch->active) { - current_thread_info()->local_flags |=3D _TLF_LAZY_MMU; - if (batch->index) - __flush_tlb_pending(batch); - batch->active =3D 0; - } - /* * On POWER9 the copy-paste buffer can only paste into * foreign real addresses, so unprivileged processes can not @@ -1369,20 +1358,6 @@ struct task_struct *__switch_to(struct task_struct *= prev, */ =20 #ifdef CONFIG_PPC_BOOK3S_64 -#ifdef CONFIG_PPC_64S_HASH_MMU - /* - * This applies to a process that was context switched while inside - * arch_enter_lazy_mmu_mode(), to re-activate the batch that was - * deactivated above, before _switch(). This will never be the case - * for new tasks. - */ - if (current_thread_info()->local_flags & _TLF_LAZY_MMU) { - current_thread_info()->local_flags &=3D ~_TLF_LAZY_MMU; - batch =3D this_cpu_ptr(&ppc64_tlb_batch); - batch->active =3D 1; - } -#endif - /* * Math facilities are masked out of the child MSR in copy_thread. * A new task does not need to restore_math because it will --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516899435918.6497222351416; Wed, 15 Oct 2025 01:28:19 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143204.1476971 (Exim 4.92) (envelope-from ) id 1v8wra-0006T7-Tj; Wed, 15 Oct 2025 08:28:02 +0000 Received: by outflank-mailman (output) from mailman id 1143204.1476971; Wed, 15 Oct 2025 08:28:02 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wra-0006T0-OQ; Wed, 15 Oct 2025 08:28:02 +0000 Received: by outflank-mailman (input) for mailman id 1143204; Wed, 15 Oct 2025 08:28:01 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrZ-0006Qy-KE for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:01 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id d8789c25-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:27:54 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 968DD22EA; Wed, 15 Oct 2025 01:27:45 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 400C13F66E; Wed, 15 Oct 2025 01:27:48 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d8789c25-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 02/13] x86/xen: simplify flush_lazy_mmu() Date: Wed, 15 Oct 2025 09:27:16 +0100 Message-ID: <20251015082727.2395128-3-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516901568158500 Content-Type: text/plain; charset="utf-8" arch_flush_lazy_mmu_mode() is called when outstanding batched pgtable operations must be completed immediately. There should however be no need to leave and re-enter lazy MMU completely. The only part of that sequence that we really need is xen_mc_flush(); call it directly. While at it, we can also avoid preempt_disable() if we are not in lazy MMU mode - xen_get_lazy_mode() should tolerate preemption. Signed-off-by: Kevin Brodsky --- arch/x86/xen/mmu_pv.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index 2a4a8deaf612..dcb7b0989c32 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2137,14 +2137,11 @@ static void xen_enter_lazy_mmu(void) =20 static void xen_flush_lazy_mmu(void) { - preempt_disable(); - if (xen_get_lazy_mode() =3D=3D XEN_LAZY_MMU) { - arch_leave_lazy_mmu_mode(); - arch_enter_lazy_mmu_mode(); + preempt_disable(); + xen_mc_flush(); + preempt_enable(); } - - preempt_enable(); } =20 static void __init xen_post_allocator_init(void) --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516893991154.15354403843378; Wed, 15 Oct 2025 01:28:13 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143205.1476974 (Exim 4.92) (envelope-from ) id 1v8wrb-0006Vk-3o; Wed, 15 Oct 2025 08:28:03 +0000 Received: by outflank-mailman (output) from mailman id 1143205.1476974; Wed, 15 Oct 2025 08:28:03 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wra-0006V4-VR; Wed, 15 Oct 2025 08:28:02 +0000 Received: by outflank-mailman (input) for mailman id 1143205; Wed, 15 Oct 2025 08:28:01 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrZ-0006Qy-R5 for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:01 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id dba139bb-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:27:59 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E422222EE; Wed, 15 Oct 2025 01:27:50 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DF8BD3F66E; Wed, 15 Oct 2025 01:27:53 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: dba139bb-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 03/13] powerpc/mm: implement arch_flush_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:17 +0100 Message-ID: <20251015082727.2395128-4-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516896948154100 Content-Type: text/plain; charset="utf-8" Upcoming changes to the lazy_mmu API will cause arch_flush_lazy_mmu_mode() to be called when leaving a nested lazy_mmu section. Move the relevant logic from arch_leave_lazy_mmu_mode() to arch_flush_lazy_mmu_mode() and have the former call the latter. Signed-off-by: Kevin Brodsky --- .../powerpc/include/asm/book3s/64/tlbflush-hash.h | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powe= rpc/include/asm/book3s/64/tlbflush-hash.h index 146287d9580f..7704dbe8e88d 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h @@ -41,6 +41,16 @@ static inline void arch_enter_lazy_mmu_mode(void) batch->active =3D 1; } =20 +static inline void arch_flush_lazy_mmu_mode(void) +{ + struct ppc64_tlb_batch *batch; + + batch =3D this_cpu_ptr(&ppc64_tlb_batch); + + if (batch->index) + __flush_tlb_pending(batch); +} + static inline void arch_leave_lazy_mmu_mode(void) { struct ppc64_tlb_batch *batch; @@ -49,14 +59,11 @@ static inline void arch_leave_lazy_mmu_mode(void) return; batch =3D this_cpu_ptr(&ppc64_tlb_batch); =20 - if (batch->index) - __flush_tlb_pending(batch); + arch_flush_lazy_mmu_mode(); batch->active =3D 0; preempt_enable(); } =20 -#define arch_flush_lazy_mmu_mode() do {} while (0) - extern void hash__tlbiel_all(unsigned int action); =20 extern void flush_hash_page(unsigned long vpn, real_pte_t pte, int psize, --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516903922834.4985685793998; Wed, 15 Oct 2025 01:28:23 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143209.1476991 (Exim 4.92) (envelope-from ) id 1v8wre-00070X-DD; Wed, 15 Oct 2025 08:28:06 +0000 Received: by outflank-mailman (output) from mailman id 1143209.1476991; Wed, 15 Oct 2025 08:28:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wre-00070Q-8T; Wed, 15 Oct 2025 08:28:06 +0000 Received: by outflank-mailman (input) for mailman id 1143209; Wed, 15 Oct 2025 08:28:05 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrd-0005tz-Bz for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:05 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id debc29a0-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:04 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 36B1D22F8; Wed, 15 Oct 2025 01:27:56 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 361C63F66E; Wed, 15 Oct 2025 01:27:59 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: debc29a0-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 04/13] sparc/mm: implement arch_flush_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:18 +0100 Message-ID: <20251015082727.2395128-5-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516905420158500 Content-Type: text/plain; charset="utf-8" Upcoming changes to the lazy_mmu API will cause arch_flush_lazy_mmu_mode() to be called when leaving a nested lazy_mmu section. Move the relevant logic from arch_leave_lazy_mmu_mode() to arch_flush_lazy_mmu_mode() and have the former call the latter. Signed-off-by: Kevin Brodsky --- arch/sparc/include/asm/tlbflush_64.h | 2 +- arch/sparc/mm/tlb.c | 9 ++++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/arch/sparc/include/asm/tlbflush_64.h b/arch/sparc/include/asm/= tlbflush_64.h index 8b8cdaa69272..925bb5d7a4e1 100644 --- a/arch/sparc/include/asm/tlbflush_64.h +++ b/arch/sparc/include/asm/tlbflush_64.h @@ -43,8 +43,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned= long end); =20 void flush_tlb_pending(void); void arch_enter_lazy_mmu_mode(void); +void arch_flush_lazy_mmu_mode(void); void arch_leave_lazy_mmu_mode(void); -#define arch_flush_lazy_mmu_mode() do {} while (0) =20 /* Local cpu only. */ void __flush_tlb_all(void); diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c index a35ddcca5e76..7b5dfcdb1243 100644 --- a/arch/sparc/mm/tlb.c +++ b/arch/sparc/mm/tlb.c @@ -59,12 +59,19 @@ void arch_enter_lazy_mmu_mode(void) tb->active =3D 1; } =20 -void arch_leave_lazy_mmu_mode(void) +void arch_flush_lazy_mmu_mode(void) { struct tlb_batch *tb =3D this_cpu_ptr(&tlb_batch); =20 if (tb->tlb_nr) flush_tlb_pending(); +} + +void arch_leave_lazy_mmu_mode(void) +{ + struct tlb_batch *tb =3D this_cpu_ptr(&tlb_batch); + + arch_flush_lazy_mmu_mode(); tb->active =3D 0; preempt_enable(); } --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516904818100.31224564516174; Wed, 15 Oct 2025 01:28:24 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143212.1477001 (Exim 4.92) (envelope-from ) id 1v8wrk-0007VU-PK; Wed, 15 Oct 2025 08:28:12 +0000 Received: by outflank-mailman (output) from mailman id 1143212.1477001; Wed, 15 Oct 2025 08:28:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrk-0007V7-JB; Wed, 15 Oct 2025 08:28:12 +0000 Received: by outflank-mailman (input) for mailman id 1143212; Wed, 15 Oct 2025 08:28:11 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrj-0005tz-1E for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:11 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id e1ef030e-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:10 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 82ADC22FA; Wed, 15 Oct 2025 01:28:01 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 801C23F66E; Wed, 15 Oct 2025 01:28:04 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e1ef030e-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 05/13] mm: introduce CONFIG_ARCH_LAZY_MMU Date: Wed, 15 Oct 2025 09:27:19 +0100 Message-ID: <20251015082727.2395128-6-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516907508158500 Content-Type: text/plain; charset="utf-8" Architectures currently opt in for implementing lazy_mmu helpers by defining __HAVE_ARCH_ENTER_LAZY_MMU_MODE. In preparation for introducing a generic lazy_mmu layer that will require storage in task_struct, let's switch to a cleaner approach: instead of defining a macro, select a CONFIG option. This patch introduces CONFIG_ARCH_LAZY_MMU and has each arch select it when it implements lazy_mmu helpers. __HAVE_ARCH_ENTER_LAZY_MMU_MODE is removed and relies on the new CONFIG instead. On x86, lazy_mmu helpers are only implemented if PARAVIRT_XXL is selected. This creates some complications in arch/x86/boot/, because a few files manually undefine PARAVIRT* options. As a result does not define the lazy_mmu helpers, but this breaks the build as only defines them if !CONFIG_ARCH_LAZY_MMU. There does not seem to be a clean way out of this - let's just undefine that new CONFIG too. Signed-off-by: Kevin Brodsky --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 1 - arch/powerpc/include/asm/book3s/64/tlbflush-hash.h | 2 -- arch/powerpc/platforms/Kconfig.cputype | 1 + arch/sparc/Kconfig | 1 + arch/sparc/include/asm/tlbflush_64.h | 2 -- arch/x86/Kconfig | 1 + arch/x86/boot/compressed/misc.h | 1 + arch/x86/boot/startup/sme.c | 1 + arch/x86/include/asm/paravirt.h | 1 - include/linux/pgtable.h | 2 +- mm/Kconfig | 3 +++ 12 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6663ffd23f25..12d47a5f5e56 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -122,6 +122,7 @@ config ARM64 select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES select ARCH_HAS_UBSAN + select ARCH_LAZY_MMU select ARM_AMBA select ARM_ARCH_TIMER select ARM_GIC diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index aa89c2e67ebc..e3cbb10288c4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,7 +80,6 @@ static inline void queue_pte_barriers(void) } } =20 -#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE static inline void arch_enter_lazy_mmu_mode(void) { /* diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powe= rpc/include/asm/book3s/64/tlbflush-hash.h index 7704dbe8e88d..623a8a8b2d0e 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h @@ -24,8 +24,6 @@ DECLARE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch); =20 extern void __flush_tlb_pending(struct ppc64_tlb_batch *batch); =20 -#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE - static inline void arch_enter_lazy_mmu_mode(void) { struct ppc64_tlb_batch *batch; diff --git a/arch/powerpc/platforms/Kconfig.cputype b/arch/powerpc/platform= s/Kconfig.cputype index 7b527d18aa5e..a5e06aaf19cd 100644 --- a/arch/powerpc/platforms/Kconfig.cputype +++ b/arch/powerpc/platforms/Kconfig.cputype @@ -93,6 +93,7 @@ config PPC_BOOK3S_64 select IRQ_WORK select PPC_64S_HASH_MMU if !PPC_RADIX_MMU select KASAN_VMALLOC if KASAN + select ARCH_LAZY_MMU =20 config PPC_BOOK3E_64 bool "Embedded processors" diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index a630d373e645..59f17996a353 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -112,6 +112,7 @@ config SPARC64 select NEED_PER_CPU_PAGE_FIRST_CHUNK select ARCH_SUPPORTS_SCHED_SMT if SMP select ARCH_SUPPORTS_SCHED_MC if SMP + select ARCH_LAZY_MMU =20 config ARCH_PROC_KCORE_TEXT def_bool y diff --git a/arch/sparc/include/asm/tlbflush_64.h b/arch/sparc/include/asm/= tlbflush_64.h index 925bb5d7a4e1..4e1036728e2f 100644 --- a/arch/sparc/include/asm/tlbflush_64.h +++ b/arch/sparc/include/asm/tlbflush_64.h @@ -39,8 +39,6 @@ static inline void flush_tlb_range(struct vm_area_struct = *vma, =20 void flush_tlb_kernel_range(unsigned long start, unsigned long end); =20 -#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE - void flush_tlb_pending(void); void arch_enter_lazy_mmu_mode(void); void arch_flush_lazy_mmu_mode(void); diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index fa3b616af03a..85de037cad8c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -804,6 +804,7 @@ config PARAVIRT config PARAVIRT_XXL bool depends on X86_64 + select ARCH_LAZY_MMU =20 config PARAVIRT_DEBUG bool "paravirt-ops debugging" diff --git a/arch/x86/boot/compressed/misc.h b/arch/x86/boot/compressed/mis= c.h index db1048621ea2..80b3b79a1001 100644 --- a/arch/x86/boot/compressed/misc.h +++ b/arch/x86/boot/compressed/misc.h @@ -11,6 +11,7 @@ #undef CONFIG_PARAVIRT #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS +#undef CONFIG_ARCH_LAZY_MMU #undef CONFIG_KASAN #undef CONFIG_KASAN_GENERIC =20 diff --git a/arch/x86/boot/startup/sme.c b/arch/x86/boot/startup/sme.c index e7ea65f3f1d6..af74d09b68bc 100644 --- a/arch/x86/boot/startup/sme.c +++ b/arch/x86/boot/startup/sme.c @@ -24,6 +24,7 @@ #undef CONFIG_PARAVIRT #undef CONFIG_PARAVIRT_XXL #undef CONFIG_PARAVIRT_SPINLOCKS +#undef CONFIG_ARCH_LAZY_MMU =20 /* * This code runs before CPU feature bits are set. By default, the diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravir= t.h index b5e59a7ba0d0..13f9cd31c8f8 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -526,7 +526,6 @@ static inline void arch_end_context_switch(struct task_= struct *next) PVOP_VCALL1(cpu.end_context_switch, next); } =20 -#define __HAVE_ARCH_ENTER_LAZY_MMU_MODE static inline void arch_enter_lazy_mmu_mode(void) { PVOP_VCALL0(mmu.lazy_mode.enter); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 32e8457ad535..124d5fa2975f 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -231,7 +231,7 @@ static inline int pmd_dirty(pmd_t pmd) * held, but for kernel PTE updates, no lock is held). Nesting is not perm= itted * and the mode cannot be used in interrupt context. */ -#ifndef __HAVE_ARCH_ENTER_LAZY_MMU_MODE +#ifndef CONFIG_ARCH_LAZY_MMU static inline void arch_enter_lazy_mmu_mode(void) {} static inline void arch_leave_lazy_mmu_mode(void) {} static inline void arch_flush_lazy_mmu_mode(void) {} diff --git a/mm/Kconfig b/mm/Kconfig index 0e26f4fc8717..2fdcb42ca1a1 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1372,6 +1372,9 @@ config PT_RECLAIM config FIND_NORMAL_PAGE def_bool n =20 +config ARCH_LAZY_MMU + bool + source "mm/damon/Kconfig" =20 endmenu --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760516912654148.33611159632233; Wed, 15 Oct 2025 01:28:32 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143224.1477010 (Exim 4.92) (envelope-from ) id 1v8wrs-00083q-0C; Wed, 15 Oct 2025 08:28:20 +0000 Received: by outflank-mailman (output) from mailman id 1143224.1477010; Wed, 15 Oct 2025 08:28:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrr-00083d-Rz; Wed, 15 Oct 2025 08:28:19 +0000 Received: by outflank-mailman (input) for mailman id 1143224; Wed, 15 Oct 2025 08:28:18 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrp-0006Qy-QF for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:17 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id e52bc62b-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:28:15 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EAC8522FC; Wed, 15 Oct 2025 01:28:06 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C98FB3F66E; Wed, 15 Oct 2025 01:28:09 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e52bc62b-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 06/13] mm: introduce generic lazy_mmu helpers Date: Wed, 15 Oct 2025 09:27:20 +0100 Message-ID: <20251015082727.2395128-7-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760516914122154100 Content-Type: text/plain; charset="utf-8" The implementation of the lazy MMU mode is currently entirely arch-specific; core code directly calls arch helpers: arch_{enter,leave}_lazy_mmu_mode(). We are about to introduce support for nested lazy MMU sections. As things stand we'd have to duplicate that logic in every arch implementing lazy_mmu - adding to a fair amount of logic already duplicated across lazy_mmu implementations. This patch therefore introduces a new generic layer that calls the existing arch_* helpers. Two pair of calls are introduced: * lazy_mmu_mode_enable() ... lazy_mmu_mode_disable() This is the standard case where the mode is enabled for a given block of code by surrounding it with enable() and disable() calls. * lazy_mmu_mode_pause() ... lazy_mmu_mode_resume() This is for situations where the mode is temporarily disabled by first calling pause() and then resume() (e.g. to prevent any batching from occurring in a critical section). The documentation in will be updated in a subsequent patch. No functional change should be introduced at this stage. The implementation of enable()/resume() and disable()/pause() is currently identical, but nesting support will change that. Most of the call sites have been updated using the following Coccinelle script: @@ @@ { ... - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); ... - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); ... } @@ @@ { ... - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_pause(); ... - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_resume(); ... } A couple of cases are noteworthy: * madvise_*_pte_range() call arch_leave() in multiple paths, some followed by an immediate exit/rescheduling and some followed by a conditional exit. These functions assume that they are called with lazy MMU disabled and we cannot simply use pause()/resume() to address that. This patch leaves the situation unchanged by calling enable()/disable() in all cases. * x86/Xen is currently the only case where explicit handling is required for lazy MMU when context-switching. This is purely an implementation detail and using the generic lazy_mmu_mode_* functions would cause trouble when nesting support is introduced, because the generic functions must be called from the current task. For that reason we still use arch_leave() and arch_enter() there. Note: x86 calls arch_flush_lazy_mmu_mode() unconditionally in a few places, but only defines it if PARAVIRT_XXL is selected, and we are removing the fallback in . Add a new fallback definition to to keep things building. Signed-off-by: Kevin Brodsky --- arch/arm64/mm/mmu.c | 4 ++-- arch/arm64/mm/pageattr.c | 4 ++-- arch/powerpc/mm/book3s64/hash_tlb.c | 8 +++---- arch/powerpc/mm/book3s64/subpage_prot.c | 4 ++-- arch/x86/include/asm/pgtable.h | 3 ++- fs/proc/task_mmu.c | 4 ++-- include/linux/pgtable.h | 29 +++++++++++++++++++++---- mm/kasan/shadow.c | 8 +++---- mm/madvise.c | 18 +++++++-------- mm/memory.c | 16 +++++++------- mm/migrate_device.c | 4 ++-- mm/mprotect.c | 4 ++-- mm/mremap.c | 4 ++-- mm/userfaultfd.c | 4 ++-- mm/vmalloc.c | 12 +++++----- mm/vmscan.c | 12 +++++----- 16 files changed, 80 insertions(+), 58 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b8d37eb037fc..d9c8e94f140f 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -731,7 +731,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsi= gned long end) return -EINVAL; =20 mutex_lock(&pgtable_split_lock); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 /* * The split_kernel_leaf_mapping_locked() may sleep, it is not a @@ -753,7 +753,7 @@ int split_kernel_leaf_mapping(unsigned long start, unsi= gned long end) ret =3D split_kernel_leaf_mapping_locked(end); } =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); mutex_unlock(&pgtable_split_lock); return ret; } diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 5135f2d66958..e4059f13c4ed 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -110,7 +110,7 @@ static int update_range_prot(unsigned long start, unsig= ned long size, if (WARN_ON_ONCE(ret)) return ret; =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 /* * The caller must ensure that the range we are operating on does not @@ -119,7 +119,7 @@ static int update_range_prot(unsigned long start, unsig= ned long size, */ ret =3D walk_kernel_page_table_range_lockless(start, start + size, &pageattr_ops, NULL, &data); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); =20 return ret; } diff --git a/arch/powerpc/mm/book3s64/hash_tlb.c b/arch/powerpc/mm/book3s64= /hash_tlb.c index 21fcad97ae80..787f7a0e27f0 100644 --- a/arch/powerpc/mm/book3s64/hash_tlb.c +++ b/arch/powerpc/mm/book3s64/hash_tlb.c @@ -205,7 +205,7 @@ void __flush_hash_table_range(unsigned long start, unsi= gned long end) * way to do things but is fine for our needs here. */ local_irq_save(flags); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); for (; start < end; start +=3D PAGE_SIZE) { pte_t *ptep =3D find_init_mm_pte(start, &hugepage_shift); unsigned long pte; @@ -217,7 +217,7 @@ void __flush_hash_table_range(unsigned long start, unsi= gned long end) continue; hpte_need_flush(&init_mm, start, ptep, pte, hugepage_shift); } - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); local_irq_restore(flags); } =20 @@ -237,7 +237,7 @@ void flush_hash_table_pmd_range(struct mm_struct *mm, p= md_t *pmd, unsigned long * way to do things but is fine for our needs here. */ local_irq_save(flags); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); start_pte =3D pte_offset_map(pmd, addr); if (!start_pte) goto out; @@ -249,6 +249,6 @@ void flush_hash_table_pmd_range(struct mm_struct *mm, p= md_t *pmd, unsigned long } pte_unmap(start_pte); out: - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); local_irq_restore(flags); } diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book= 3s64/subpage_prot.c index ec98e526167e..07c47673bba2 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -73,13 +73,13 @@ static void hpte_flush_range(struct mm_struct *mm, unsi= gned long addr, pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); if (!pte) return; - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); for (; npages > 0; --npages) { pte_update(mm, addr, pte, 0, 0, 0); addr +=3D PAGE_SIZE; ++pte; } - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(pte - 1, ptl); } =20 diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index e33df3da6980..14fd672bc9b2 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -117,7 +117,8 @@ extern pmdval_t early_pmd_flags; #define pte_val(x) native_pte_val(x) #define __pte(x) native_make_pte(x) =20 -#define arch_end_context_switch(prev) do {} while(0) +#define arch_end_context_switch(prev) do {} while (0) +#define arch_flush_lazy_mmu_mode() do {} while (0) #endif /* CONFIG_PARAVIRT_XXL */ =20 static inline pmd_t pmd_set_flags(pmd_t pmd, pmdval_t set) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index fc35a0543f01..d16ba1d32169 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2703,7 +2703,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigne= d long start, return 0; } =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) { /* Fast path for performing exclusive WP */ @@ -2773,7 +2773,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigne= d long start, if (flush_end) flush_tlb_range(vma, start, addr); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); =20 cond_resched(); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 124d5fa2975f..194b2c3e7576 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -231,10 +231,31 @@ static inline int pmd_dirty(pmd_t pmd) * held, but for kernel PTE updates, no lock is held). Nesting is not perm= itted * and the mode cannot be used in interrupt context. */ -#ifndef CONFIG_ARCH_LAZY_MMU -static inline void arch_enter_lazy_mmu_mode(void) {} -static inline void arch_leave_lazy_mmu_mode(void) {} -static inline void arch_flush_lazy_mmu_mode(void) {} +#ifdef CONFIG_ARCH_LAZY_MMU +static inline void lazy_mmu_mode_enable(void) +{ + arch_enter_lazy_mmu_mode(); +} + +static inline void lazy_mmu_mode_disable(void) +{ + arch_leave_lazy_mmu_mode(); +} + +static inline void lazy_mmu_mode_pause(void) +{ + arch_leave_lazy_mmu_mode(); +} + +static inline void lazy_mmu_mode_resume(void) +{ + arch_enter_lazy_mmu_mode(); +} +#else +static inline void lazy_mmu_mode_enable(void) {} +static inline void lazy_mmu_mode_disable(void) {} +static inline void lazy_mmu_mode_pause(void) {} +static inline void lazy_mmu_mode_resume(void) {} #endif =20 #ifndef pte_batch_hint diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index 5d2a876035d6..c49b029d3593 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -305,7 +305,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsi= gned long addr, pte_t pte; int index; =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_pause(); =20 index =3D PFN_DOWN(addr - data->start); page =3D data->pages[index]; @@ -319,7 +319,7 @@ static int kasan_populate_vmalloc_pte(pte_t *ptep, unsi= gned long addr, } spin_unlock(&init_mm.page_table_lock); =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_resume(); =20 return 0; } @@ -482,7 +482,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, un= signed long addr, pte_t pte; int none; =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_pause(); =20 spin_lock(&init_mm.page_table_lock); pte =3D ptep_get(ptep); @@ -494,7 +494,7 @@ static int kasan_depopulate_vmalloc_pte(pte_t *ptep, un= signed long addr, if (likely(!none)) __free_page(pfn_to_page(pte_pfn(pte))); =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_resume(); =20 return 0; } diff --git a/mm/madvise.c b/mm/madvise.c index fb1c86e630b6..536026772160 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -455,7 +455,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (!start_pte) return 0; flush_tlb_batched_pending(mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); for (; addr < end; pte +=3D nr, addr +=3D nr * PAGE_SIZE) { nr =3D 1; ptent =3D ptep_get(pte); @@ -463,7 +463,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (++batch_count =3D=3D SWAP_CLUSTER_MAX) { batch_count =3D 0; if (need_resched()) { - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); cond_resched(); goto restart; @@ -499,7 +499,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (!folio_trylock(folio)) continue; folio_get(folio); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); start_pte =3D NULL; err =3D split_folio(folio); @@ -510,7 +510,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (!start_pte) break; flush_tlb_batched_pending(mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); if (!err) nr =3D 0; continue; @@ -558,7 +558,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } =20 if (start_pte) { - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); } if (pageout) @@ -677,7 +677,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, if (!start_pte) return 0; flush_tlb_batched_pending(mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); for (; addr !=3D end; pte +=3D nr, addr +=3D PAGE_SIZE * nr) { nr =3D 1; ptent =3D ptep_get(pte); @@ -727,7 +727,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, if (!folio_trylock(folio)) continue; folio_get(folio); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); start_pte =3D NULL; err =3D split_folio(folio); @@ -738,7 +738,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, if (!start_pte) break; flush_tlb_batched_pending(mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); if (!err) nr =3D 0; continue; @@ -778,7 +778,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, if (nr_swap) add_mm_counter(mm, MM_SWAPENTS, nr_swap); if (start_pte) { - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); } cond_resched(); diff --git a/mm/memory.c b/mm/memory.c index 74b45e258323..2d662dee5ae7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1254,7 +1254,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); orig_src_pte =3D src_pte; orig_dst_pte =3D dst_pte; - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 do { nr =3D 1; @@ -1323,7 +1323,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, } while (dst_pte +=3D nr, src_pte +=3D nr, addr +=3D PAGE_SIZE * nr, addr !=3D end); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(orig_src_pte, src_ptl); add_mm_rss_vec(dst_mm, rss); pte_unmap_unlock(orig_dst_pte, dst_ptl); @@ -1842,7 +1842,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, return addr; =20 flush_tlb_batched_pending(mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); do { bool any_skipped =3D false; =20 @@ -1874,7 +1874,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, direct_reclaim =3D try_get_and_clear_pmd(mm, pmd, &pmdval); =20 add_mm_rss_vec(mm, rss); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); =20 /* Do the actual TLB flush before dropping ptl */ if (force_flush) { @@ -2817,7 +2817,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_= t *pmd, mapped_pte =3D pte =3D pte_alloc_map_lock(mm, pmd, addr, &ptl); if (!pte) return -ENOMEM; - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); do { BUG_ON(!pte_none(ptep_get(pte))); if (!pfn_modify_allowed(pfn, prot)) { @@ -2827,7 +2827,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_= t *pmd, set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot))); pfn++; } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(mapped_pte, ptl); return err; } @@ -3134,7 +3134,7 @@ static int apply_to_pte_range(struct mm_struct *mm, p= md_t *pmd, return -EINVAL; } =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 if (fn) { do { @@ -3147,7 +3147,7 @@ static int apply_to_pte_range(struct mm_struct *mm, p= md_t *pmd, } *mask |=3D PGTBL_PTE_MODIFIED; =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); =20 if (mm !=3D &init_mm) pte_unmap_unlock(mapped_pte, ptl); diff --git a/mm/migrate_device.c b/mm/migrate_device.c index abd9f6850db6..dcdc46b96cc7 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -110,7 +110,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, ptep =3D pte_offset_map_lock(mm, pmdp, addr, &ptl); if (!ptep) goto again; - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 for (; addr < end; addr +=3D PAGE_SIZE, ptep++) { struct dev_pagemap *pgmap; @@ -287,7 +287,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (unmapped) flush_tlb_range(walk->vma, start, end); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(ptep - 1, ptl); =20 return 0; diff --git a/mm/mprotect.c b/mm/mprotect.c index 113b48985834..bcb183a6fd2f 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -293,7 +293,7 @@ static long change_pte_range(struct mmu_gather *tlb, target_node =3D numa_node_id(); =20 flush_tlb_batched_pending(vma->vm_mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); do { nr_ptes =3D 1; oldpte =3D ptep_get(pte); @@ -439,7 +439,7 @@ static long change_pte_range(struct mmu_gather *tlb, } } } while (pte +=3D nr_ptes, addr +=3D nr_ptes * PAGE_SIZE, addr !=3D end); - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(pte - 1, ptl); =20 return pages; diff --git a/mm/mremap.c b/mm/mremap.c index 35de0a7b910e..1e216007160d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -256,7 +256,7 @@ static int move_ptes(struct pagetable_move_control *pmc, if (new_ptl !=3D old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); flush_tlb_batched_pending(vma->vm_mm); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 for (; old_addr < old_end; old_ptep +=3D nr_ptes, old_addr +=3D nr_ptes *= PAGE_SIZE, new_ptep +=3D nr_ptes, new_addr +=3D nr_ptes * PAGE_SIZE) { @@ -301,7 +301,7 @@ static int move_ptes(struct pagetable_move_control *pmc, } } =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); if (force_flush) flush_tlb_range(vma, old_end - len, old_end); if (new_ptl !=3D old_ptl) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index af61b95c89e4..e01f7813e15c 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1100,7 +1100,7 @@ static long move_present_ptes(struct mm_struct *mm, /* It's safe to drop the reference now as the page-table is holding one. = */ folio_put(*first_src_folio); *first_src_folio =3D NULL; - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 while (true) { orig_src_pte =3D ptep_get_and_clear(mm, src_addr, src_pte); @@ -1138,7 +1138,7 @@ static long move_present_ptes(struct mm_struct *mm, break; } =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); if (src_addr > src_start) flush_tlb_range(src_vma, src_start, src_addr); =20 diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 798b2ed21e46..b9940590a40d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -105,7 +105,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long add= r, unsigned long end, if (!pte) return -ENOMEM; =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 do { if (unlikely(!pte_none(ptep_get(pte)))) { @@ -131,7 +131,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long add= r, unsigned long end, pfn++; } while (pte +=3D PFN_DOWN(size), addr +=3D size, addr !=3D end); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; return 0; } @@ -359,7 +359,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long = addr, unsigned long end, unsigned long size =3D PAGE_SIZE; =20 pte =3D pte_offset_kernel(pmd, addr); - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 do { #ifdef CONFIG_HUGETLB_PAGE @@ -378,7 +378,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long = addr, unsigned long end, WARN_ON(!pte_none(ptent) && !pte_present(ptent)); } while (pte +=3D (size >> PAGE_SHIFT), addr +=3D size, addr !=3D end); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; } =20 @@ -526,7 +526,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned lo= ng addr, if (!pte) return -ENOMEM; =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 do { struct page *page =3D pages[*nr]; @@ -548,7 +548,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned lo= ng addr, (*nr)++; } while (pte++, addr +=3D PAGE_SIZE, addr !=3D end); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); *mask |=3D PGTBL_PTE_MODIFIED; =20 return err; diff --git a/mm/vmscan.c b/mm/vmscan.c index b2fc8b626d3d..7d2d87069530 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3551,7 +3551,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, return false; } =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { unsigned long pfn; @@ -3592,7 +3592,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long = start, unsigned long end, if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &= end)) goto restart; =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); pte_unmap_unlock(pte, ptl); =20 return suitable_to_scan(total, young); @@ -3633,7 +3633,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area if (!spin_trylock(ptl)) goto done; =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 do { unsigned long pfn; @@ -3680,7 +3680,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long addr, struct vm_area =20 walk_update_folio(walk, last, gen, dirty); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); spin_unlock(ptl); done: *first =3D -1; @@ -4279,7 +4279,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) } } =20 - arch_enter_lazy_mmu_mode(); + lazy_mmu_mode_enable(); =20 pte -=3D (addr - start) / PAGE_SIZE; =20 @@ -4313,7 +4313,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) =20 walk_update_folio(walk, last, gen, dirty); =20 - arch_leave_lazy_mmu_mode(); + lazy_mmu_mode_disable(); =20 /* feedback from rmap walkers to page table walkers */ if (mm_state && suitable_to_scan(i, young)) --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760517400890431.6818730792568; Wed, 15 Oct 2025 01:36:40 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143286.1477070 (Exim 4.92) (envelope-from ) id 1v8wzc-0003hQ-BD; Wed, 15 Oct 2025 08:36:20 +0000 Received: by outflank-mailman (output) from mailman id 1143286.1477070; Wed, 15 Oct 2025 08:36:20 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzc-0003h5-6H; Wed, 15 Oct 2025 08:36:20 +0000 Received: by outflank-mailman (input) for mailman id 1143286; Wed, 15 Oct 2025 08:36:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrt-0005tz-Qa for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:21 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id e8522646-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:20 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44DD11A32; Wed, 15 Oct 2025 01:28:12 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3FFB93F66E; Wed, 15 Oct 2025 01:28:15 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e8522646-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 07/13] mm: enable lazy_mmu sections to nest Date: Wed, 15 Oct 2025 09:27:21 +0100 Message-ID: <20251015082727.2395128-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517403606154100 Despite recent efforts to prevent lazy_mmu sections from nesting, it remains difficult to ensure that it never occurs - and in fact it does occur on arm64 in certain situations (CONFIG_DEBUG_PAGEALLOC). Commit 1ef3095b1405 ("arm64/mm: Permit lazy_mmu_mode to be nested") made nesting tolerable on arm64, but without truly supporting it: the inner call to leave() disables the batching optimisation before the outer section ends. This patch actually enables lazy_mmu sections to nest by tracking the nesting level in task_struct, in a similar fashion to e.g. pagefault_{enable,disable}(). This is fully handled by the generic lazy_mmu helpers that were recently introduced. lazy_mmu sections were not initially intended to nest, so we need to clarify the semantics w.r.t. the arch_*_lazy_mmu_mode() callbacks. This patch takes the following approach: * The outermost calls to lazy_mmu_mode_{enable,disable}() trigger calls to arch_{enter,leave}_lazy_mmu_mode() - this is unchanged. * Nested calls to lazy_mmu_mode_{enable,disable}() are not forwarded to the arch via arch_{enter,leave} - lazy MMU remains enabled so the assumption is that these callbacks are not relevant. However, existing code may rely on a call to disable() to flush any batched state, regardless of nesting. arch_flush_lazy_mmu_mode() is therefore called in that situation. A separate interface was recently introduced to temporarily pause the lazy MMU mode: lazy_mmu_mode_{pause,resume}(). pause() fully exits the mode *regardless of the nesting level*, and resume() restores the mode at the same nesting level. Whether the mode is actually enabled or not at any point is tracked by a separate "enabled" field in task_struct; this makes it possible to check invariants in the generic API, and to expose a new in_lazy_mmu_mode() helper to replace the various ways arch's currently track whether the mode is enabled (this will be done in later patches). In summary (count/enabled represent the values *after* the call): lazy_mmu_mode_enable() -> arch_enter() count=3D1 enabled=3D1 lazy_mmu_mode_enable() -> =C3=B8 count=3D2 enabled=3D1 lazy_mmu_mode_pause() -> arch_leave() count=3D2 enabled=3D0 lazy_mmu_mode_resume() -> arch_enter() count=3D2 enabled=3D1 lazy_mmu_mode_disable() -> arch_flush() count=3D1 enabled=3D1 lazy_mmu_mode_disable() -> arch_leave() count=3D0 enabled=3D0 Note: in_lazy_mmu_mode() is added to to allow arch headers included by to use it. Signed-off-by: Kevin Brodsky --- Alexander Gordeev suggested that a future optimisation may need lazy_mmu_mode_{pause,resume}() to call distinct arch callbacks [1]. For now arch_{leave,enter}() are called directly, but introducing new arch callbacks should be straightforward. [1] https://lore.kernel.org/all/5a0818bb-75d4-47df-925c-0102f7d598f4-agorde= ev@linux.ibm.com/ --- arch/arm64/include/asm/pgtable.h | 12 ------ include/linux/mm_types_task.h | 5 +++ include/linux/pgtable.h | 69 ++++++++++++++++++++++++++++++-- include/linux/sched.h | 16 ++++++++ 4 files changed, 86 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index e3cbb10288c4..f15ca4d62f09 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -82,18 +82,6 @@ static inline void queue_pte_barriers(void) =20 static inline void arch_enter_lazy_mmu_mode(void) { - /* - * lazy_mmu_mode is not supposed to permit nesting. But in practice this - * does happen with CONFIG_DEBUG_PAGEALLOC, where a page allocation - * inside a lazy_mmu_mode section (such as zap_pte_range()) will change - * permissions on the linear map with apply_to_page_range(), which - * re-enters lazy_mmu_mode. So we tolerate nesting in our - * implementation. The first call to arch_leave_lazy_mmu_mode() will - * flush and clear the flag such that the remainder of the work in the - * outer nest behaves as if outside of lazy mmu mode. This is safe and - * keeps tracking simple. - */ - if (in_interrupt()) return; =20 diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a82aa80c0ba4..2ff83b85fef0 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -88,4 +88,9 @@ struct tlbflush_unmap_batch { #endif }; =20 +struct lazy_mmu_state { + u8 count; + bool enabled; +}; + #endif /* _LINUX_MM_TYPES_TASK_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 194b2c3e7576..269225a733de 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -228,28 +228,89 @@ static inline int pmd_dirty(pmd_t pmd) * of the lazy mode. So the implementation must assume preemption may be e= nabled * and cpu migration is possible; it must take steps to be robust against = this. * (In practice, for user PTE updates, the appropriate page table lock(s) = are - * held, but for kernel PTE updates, no lock is held). Nesting is not perm= itted - * and the mode cannot be used in interrupt context. + * held, but for kernel PTE updates, no lock is held). The mode cannot be = used + * in interrupt context. + * + * The lazy MMU mode is enabled for a given block of code using: + * + * lazy_mmu_mode_enable(); + * + * lazy_mmu_mode_disable(); + * + * Nesting is permitted: may itself use an enable()/disable() pair. + * A nested call to enable() has no functional effect; however disable() c= auses + * any batched architectural state to be flushed regardless of nesting. Af= ter a + * call to disable(), the caller can therefore rely on all previous page t= able + * modifications to have taken effect, but the lazy MMU mode may still be + * enabled. + * + * In certain cases, it may be desirable to temporarily pause the lazy MMU= mode. + * This can be done using: + * + * lazy_mmu_mode_pause(); + * + * lazy_mmu_mode_resume(); + * + * This sequence must only be used if the lazy MMU mode is already enabled. + * pause() ensures that the mode is exited regardless of the nesting level; + * resume() re-enters the mode at the same nesting level. must not = modify + * the lazy MMU state (i.e. it must not call any of the lazy_mmu_mode_* + * helpers). + * + * in_lazy_mmu_mode() can be used to check whether the lazy MMU mode is + * currently enabled. */ #ifdef CONFIG_ARCH_LAZY_MMU static inline void lazy_mmu_mode_enable(void) { - arch_enter_lazy_mmu_mode(); + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_BUG_ON(state->count =3D=3D U8_MAX); + /* enable() must not be called while paused */ + VM_WARN_ON(state->count > 0 && !state->enabled); + + if (state->count =3D=3D 0) { + arch_enter_lazy_mmu_mode(); + state->enabled =3D true; + } + ++state->count; } =20 static inline void lazy_mmu_mode_disable(void) { - arch_leave_lazy_mmu_mode(); + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_BUG_ON(state->count =3D=3D 0); + VM_WARN_ON(!state->enabled); + + --state->count; + if (state->count =3D=3D 0) { + state->enabled =3D false; + arch_leave_lazy_mmu_mode(); + } else { + /* Exiting a nested section */ + arch_flush_lazy_mmu_mode(); + } } =20 static inline void lazy_mmu_mode_pause(void) { + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON(state->count =3D=3D 0 || !state->enabled); + + state->enabled =3D false; arch_leave_lazy_mmu_mode(); } =20 static inline void lazy_mmu_mode_resume(void) { + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON(state->count =3D=3D 0 || state->enabled); + arch_enter_lazy_mmu_mode(); + state->enabled =3D true; } #else static inline void lazy_mmu_mode_enable(void) {} diff --git a/include/linux/sched.h b/include/linux/sched.h index cbb7340c5866..2862d8bf2160 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1441,6 +1441,10 @@ struct task_struct { =20 struct page_frag task_frag; =20 +#ifdef CONFIG_ARCH_LAZY_MMU + struct lazy_mmu_state lazy_mmu_state; +#endif + #ifdef CONFIG_TASK_DELAY_ACCT struct task_delay_info *delays; #endif @@ -1724,6 +1728,18 @@ static inline char task_state_to_char(struct task_st= ruct *tsk) return task_index_to_char(task_state_index(tsk)); } =20 +#ifdef CONFIG_ARCH_LAZY_MMU +static inline bool in_lazy_mmu_mode(void) +{ + return current->lazy_mmu_state.enabled; +} +#else +static inline bool in_lazy_mmu_mode(void) +{ + return false; +} +#endif + extern struct pid *cad_pid; =20 /* --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 176051738578435.741553844092095; Wed, 15 Oct 2025 01:36:25 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143266.1477031 (Exim 4.92) (envelope-from ) id 1v8wzO-0002PS-4a; Wed, 15 Oct 2025 08:36:06 +0000 Received: by outflank-mailman (output) from mailman id 1143266.1477031; Wed, 15 Oct 2025 08:36:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzN-0002PL-WB; Wed, 15 Oct 2025 08:36:05 +0000 Received: by outflank-mailman (input) for mailman id 1143266; Wed, 15 Oct 2025 08:36:04 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wrz-0005tz-PM for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:27 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id ebcddd7f-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:26 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E5B262308; Wed, 15 Oct 2025 01:28:17 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8ACDB3F66E; Wed, 15 Oct 2025 01:28:20 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ebcddd7f-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 08/13] arm64: mm: replace TIF_LAZY_MMU with in_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:22 +0100 Message-ID: <20251015082727.2395128-9-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517387374154100 Content-Type: text/plain; charset="utf-8" The generic lazy_mmu layer now tracks whether a task is in lazy MMU mode. As a result we no longer need a TIF flag for that purpose - let's use the new in_lazy_mmu_mode() helper instead. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 16 +++------------- arch/arm64/include/asm/thread_info.h | 3 +-- 2 files changed, 4 insertions(+), 15 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index f15ca4d62f09..944e512767db 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -62,30 +62,21 @@ static inline void emit_pte_barriers(void) =20 static inline void queue_pte_barriers(void) { - unsigned long flags; - if (in_interrupt()) { emit_pte_barriers(); return; } =20 - flags =3D read_thread_flags(); - - if (flags & BIT(TIF_LAZY_MMU)) { - /* Avoid the atomic op if already set. */ - if (!(flags & BIT(TIF_LAZY_MMU_PENDING))) - set_thread_flag(TIF_LAZY_MMU_PENDING); - } else { + if (in_lazy_mmu_mode()) + test_and_set_thread_flag(TIF_LAZY_MMU_PENDING); + else emit_pte_barriers(); - } } =20 static inline void arch_enter_lazy_mmu_mode(void) { if (in_interrupt()) return; - - set_thread_flag(TIF_LAZY_MMU); } =20 static inline void arch_flush_lazy_mmu_mode(void) @@ -103,7 +94,6 @@ static inline void arch_leave_lazy_mmu_mode(void) return; =20 arch_flush_lazy_mmu_mode(); - clear_thread_flag(TIF_LAZY_MMU); } =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/= thread_info.h index f241b8601ebd..4ff8da0767d9 100644 --- a/arch/arm64/include/asm/thread_info.h +++ b/arch/arm64/include/asm/thread_info.h @@ -84,8 +84,7 @@ void arch_setup_new_exec(void); #define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */ #define TIF_KERNEL_FPSTATE 29 /* Task is in a kernel mode FPSIMD section */ #define TIF_TSC_SIGSEGV 30 /* SIGSEGV on counter-timer access */ -#define TIF_LAZY_MMU 31 /* Task in lazy mmu mode */ -#define TIF_LAZY_MMU_PENDING 32 /* Ops pending for lazy mmu mode exit */ +#define TIF_LAZY_MMU_PENDING 31 /* Ops pending for lazy mmu mode exit */ =20 #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) #define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED) --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 176051740497320.858383572571825; Wed, 15 Oct 2025 01:36:44 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143308.1477080 (Exim 4.92) (envelope-from ) id 1v8wzk-0004Q6-KF; Wed, 15 Oct 2025 08:36:28 +0000 Received: by outflank-mailman (output) from mailman id 1143308.1477080; Wed, 15 Oct 2025 08:36:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzk-0004P0-Fs; Wed, 15 Oct 2025 08:36:28 +0000 Received: by outflank-mailman (input) for mailman id 1143308; Wed, 15 Oct 2025 08:36:27 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8ws4-0005tz-R3 for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:32 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id eee4abdc-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:31 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3C10E2309; Wed, 15 Oct 2025 01:28:23 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 37F513F66E; Wed, 15 Oct 2025 01:28:26 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: eee4abdc-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 09/13] powerpc/mm: replace batch->active with in_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:23 +0100 Message-ID: <20251015082727.2395128-10-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517407246158500 Content-Type: text/plain; charset="utf-8" The generic lazy_mmu layer now tracks whether a task is in lazy MMU mode. As a result we no longer need to track whether the per-CPU TLB batch struct is active - we know it is if in_lazy_mmu_mode() returns true. Signed-off-by: Kevin Brodsky --- arch/powerpc/include/asm/book3s/64/tlbflush-hash.h | 9 --------- arch/powerpc/mm/book3s64/hash_tlb.c | 2 +- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powe= rpc/include/asm/book3s/64/tlbflush-hash.h index 623a8a8b2d0e..bbc54690d374 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h @@ -12,7 +12,6 @@ #define PPC64_TLB_BATCH_NR 192 =20 struct ppc64_tlb_batch { - int active; unsigned long index; struct mm_struct *mm; real_pte_t pte[PPC64_TLB_BATCH_NR]; @@ -26,8 +25,6 @@ extern void __flush_tlb_pending(struct ppc64_tlb_batch *b= atch); =20 static inline void arch_enter_lazy_mmu_mode(void) { - struct ppc64_tlb_batch *batch; - if (radix_enabled()) return; /* @@ -35,8 +32,6 @@ static inline void arch_enter_lazy_mmu_mode(void) * operating on kernel page tables. */ preempt_disable(); - batch =3D this_cpu_ptr(&ppc64_tlb_batch); - batch->active =3D 1; } =20 static inline void arch_flush_lazy_mmu_mode(void) @@ -51,14 +46,10 @@ static inline void arch_flush_lazy_mmu_mode(void) =20 static inline void arch_leave_lazy_mmu_mode(void) { - struct ppc64_tlb_batch *batch; - if (radix_enabled()) return; - batch =3D this_cpu_ptr(&ppc64_tlb_batch); =20 arch_flush_lazy_mmu_mode(); - batch->active =3D 0; preempt_enable(); } =20 diff --git a/arch/powerpc/mm/book3s64/hash_tlb.c b/arch/powerpc/mm/book3s64= /hash_tlb.c index 787f7a0e27f0..72b83f582b6d 100644 --- a/arch/powerpc/mm/book3s64/hash_tlb.c +++ b/arch/powerpc/mm/book3s64/hash_tlb.c @@ -100,7 +100,7 @@ void hpte_need_flush(struct mm_struct *mm, unsigned lon= g addr, * Check if we have an active batch on this CPU. If not, just * flush now and return. */ - if (!batch->active) { + if (!in_lazy_mmu_mode()) { flush_hash_page(vpn, rpte, psize, ssize, mm_is_thread_local(mm)); put_cpu_var(ppc64_tlb_batch); return; --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 176051738918993.34483704352374; Wed, 15 Oct 2025 01:36:29 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143280.1477055 (Exim 4.92) (envelope-from ) id 1v8wzX-00032y-RI; Wed, 15 Oct 2025 08:36:15 +0000 Received: by outflank-mailman (output) from mailman id 1143280.1477055; Wed, 15 Oct 2025 08:36:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzX-00031e-MA; Wed, 15 Oct 2025 08:36:15 +0000 Received: by outflank-mailman (input) for mailman id 1143280; Wed, 15 Oct 2025 08:36:14 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wsA-0005tz-3i for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:38 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id f23dbf0d-a9a0-11f0-9d15-b5c5bf9af7f9; Wed, 15 Oct 2025 10:28:37 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D57AA2328; Wed, 15 Oct 2025 01:28:28 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8251F3F66E; Wed, 15 Oct 2025 01:28:31 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f23dbf0d-a9a0-11f0-9d15-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 10/13] sparc/mm: replace batch->active with in_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:24 +0100 Message-ID: <20251015082727.2395128-11-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517391468158500 Content-Type: text/plain; charset="utf-8" The generic lazy_mmu layer now tracks whether a task is in lazy MMU mode. As a result we no longer need to track whether the per-CPU TLB batch struct is active - we know it is if in_lazy_mmu_mode() returns true. Signed-off-by: Kevin Brodsky --- arch/sparc/include/asm/tlbflush_64.h | 1 - arch/sparc/mm/tlb.c | 9 +-------- 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/arch/sparc/include/asm/tlbflush_64.h b/arch/sparc/include/asm/= tlbflush_64.h index 4e1036728e2f..6133306ba59a 100644 --- a/arch/sparc/include/asm/tlbflush_64.h +++ b/arch/sparc/include/asm/tlbflush_64.h @@ -12,7 +12,6 @@ struct tlb_batch { unsigned int hugepage_shift; struct mm_struct *mm; unsigned long tlb_nr; - unsigned long active; unsigned long vaddrs[TLB_BATCH_NR]; }; =20 diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c index 7b5dfcdb1243..879e22c86e5c 100644 --- a/arch/sparc/mm/tlb.c +++ b/arch/sparc/mm/tlb.c @@ -52,11 +52,7 @@ void flush_tlb_pending(void) =20 void arch_enter_lazy_mmu_mode(void) { - struct tlb_batch *tb; - preempt_disable(); - tb =3D this_cpu_ptr(&tlb_batch); - tb->active =3D 1; } =20 void arch_flush_lazy_mmu_mode(void) @@ -69,10 +65,7 @@ void arch_flush_lazy_mmu_mode(void) =20 void arch_leave_lazy_mmu_mode(void) { - struct tlb_batch *tb =3D this_cpu_ptr(&tlb_batch); - arch_flush_lazy_mmu_mode(); - tb->active =3D 0; preempt_enable(); } =20 @@ -93,7 +86,7 @@ static void tlb_batch_add_one(struct mm_struct *mm, unsig= ned long vaddr, nr =3D 0; } =20 - if (!tb->active) { + if (!in_lazy_mmu_mode()) { flush_tsb_user_page(mm, vaddr, hugepage_shift); global_flush_tlb_page(mm, vaddr); goto out; --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760517384009692.0686762477309; Wed, 15 Oct 2025 01:36:24 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143275.1477041 (Exim 4.92) (envelope-from ) id 1v8wzW-0002k5-Cs; Wed, 15 Oct 2025 08:36:14 +0000 Received: by outflank-mailman (output) from mailman id 1143275.1477041; Wed, 15 Oct 2025 08:36:14 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzW-0002jy-6g; Wed, 15 Oct 2025 08:36:14 +0000 Received: by outflank-mailman (input) for mailman id 1143275; Wed, 15 Oct 2025 08:36:12 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wsG-0006Qy-PT for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:44 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id f5688f5b-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:28:42 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2EB672364; Wed, 15 Oct 2025 01:28:34 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2ADB43F66E; Wed, 15 Oct 2025 01:28:37 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f5688f5b-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 11/13] x86/xen: use lazy_mmu_state when context-switching Date: Wed, 15 Oct 2025 09:27:25 +0100 Message-ID: <20251015082727.2395128-12-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517385502154100 Content-Type: text/plain; charset="utf-8" We currently set a TIF flag when scheduling out a task that is in lazy MMU mode, in order to restore it when the task is scheduled again. The generic lazy_mmu layer now tracks whether a task is in lazy MMU mode in task_struct::lazy_mmu_state. We can therefore check that state when switching to the new task, instead of using a separate TIF flag. Signed-off-by: Kevin Brodsky --- arch/x86/include/asm/thread_info.h | 4 +--- arch/x86/xen/enlighten_pv.c | 3 +-- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thre= ad_info.h index e71e0e8362ed..0067684afb5b 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -100,8 +100,7 @@ struct thread_info { #define TIF_FORCED_TF 24 /* true if TF in eflags artificially */ #define TIF_SINGLESTEP 25 /* reenable singlestep on user return*/ #define TIF_BLOCKSTEP 26 /* set when we want DEBUGCTLMSR_BTF */ -#define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ -#define TIF_ADDR32 28 /* 32-bit address space on 64 bits */ +#define TIF_ADDR32 27 /* 32-bit address space on 64 bits */ =20 #define _TIF_SSBD BIT(TIF_SSBD) #define _TIF_SPEC_IB BIT(TIF_SPEC_IB) @@ -114,7 +113,6 @@ struct thread_info { #define _TIF_FORCED_TF BIT(TIF_FORCED_TF) #define _TIF_BLOCKSTEP BIT(TIF_BLOCKSTEP) #define _TIF_SINGLESTEP BIT(TIF_SINGLESTEP) -#define _TIF_LAZY_MMU_UPDATES BIT(TIF_LAZY_MMU_UPDATES) #define _TIF_ADDR32 BIT(TIF_ADDR32) =20 /* flags to check in __switch_to() */ diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 4806cc28d7ca..9fabe83e7546 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -426,7 +426,6 @@ static void xen_start_context_switch(struct task_struct= *prev) =20 if (this_cpu_read(xen_lazy_mode) =3D=3D XEN_LAZY_MMU) { arch_leave_lazy_mmu_mode(); - set_ti_thread_flag(task_thread_info(prev), TIF_LAZY_MMU_UPDATES); } enter_lazy(XEN_LAZY_CPU); } @@ -437,7 +436,7 @@ static void xen_end_context_switch(struct task_struct *= next) =20 xen_mc_flush(); leave_lazy(XEN_LAZY_CPU); - if (test_and_clear_ti_thread_flag(task_thread_info(next), TIF_LAZY_MMU_UP= DATES)) + if (next->lazy_mmu_state.enabled) arch_enter_lazy_mmu_mode(); } =20 --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760517392550856.4143314788386; Wed, 15 Oct 2025 01:36:32 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143277.1477049 (Exim 4.92) (envelope-from ) id 1v8wzX-0002zF-H0; Wed, 15 Oct 2025 08:36:15 +0000 Received: by outflank-mailman (output) from mailman id 1143277.1477049; Wed, 15 Oct 2025 08:36:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzX-0002z2-DF; Wed, 15 Oct 2025 08:36:15 +0000 Received: by outflank-mailman (input) for mailman id 1143277; Wed, 15 Oct 2025 08:36:14 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wsL-0006Qy-Pl for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:49 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id f8864424-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:28:48 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 78DF9236D; Wed, 15 Oct 2025 01:28:39 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 754613F66E; Wed, 15 Oct 2025 01:28:42 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: f8864424-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 12/13] mm: bail out of lazy_mmu_mode_* in interrupt context Date: Wed, 15 Oct 2025 09:27:26 +0100 Message-ID: <20251015082727.2395128-13-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517395517154100 Content-Type: text/plain; charset="utf-8" The lazy MMU mode cannot be used in interrupt context. This is documented in , but isn't consistently handled across architectures. arm64 ensures that calls to lazy_mmu_mode_* have no effect in interrupt context, because such calls do occur in certain configurations - see commit b81c688426a9 ("arm64/mm: Disable barrier batching in interrupt contexts"). Other architectures do not check this situation, most likely because it hasn't occurred so far. Both arm64 and x86/Xen also ensure that any lazy MMU optimisation is disabled while in interrupt mode (see queue_pte_barriers() and xen_get_lazy_mode() respectively). Let's handle this in the new generic lazy_mmu layer, in the same fashion as arm64: bail out of lazy_mmu_mode_* if in_interrupt(), and have in_lazy_mmu_mode() return false to disable any optimisation. Also remove the arm64 handling that is now redundant; x86/Xen has its own internal tracking so it is left unchanged. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 17 +---------------- include/linux/pgtable.h | 16 ++++++++++++++-- include/linux/sched.h | 3 +++ 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 944e512767db..a37f417c30be 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -62,37 +62,22 @@ static inline void emit_pte_barriers(void) =20 static inline void queue_pte_barriers(void) { - if (in_interrupt()) { - emit_pte_barriers(); - return; - } - if (in_lazy_mmu_mode()) test_and_set_thread_flag(TIF_LAZY_MMU_PENDING); else emit_pte_barriers(); } =20 -static inline void arch_enter_lazy_mmu_mode(void) -{ - if (in_interrupt()) - return; -} +static inline void arch_enter_lazy_mmu_mode(void) {} =20 static inline void arch_flush_lazy_mmu_mode(void) { - if (in_interrupt()) - return; - if (test_and_clear_thread_flag(TIF_LAZY_MMU_PENDING)) emit_pte_barriers(); } =20 static inline void arch_leave_lazy_mmu_mode(void) { - if (in_interrupt()) - return; - arch_flush_lazy_mmu_mode(); } =20 diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 269225a733de..718c9c788114 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -228,8 +228,8 @@ static inline int pmd_dirty(pmd_t pmd) * of the lazy mode. So the implementation must assume preemption may be e= nabled * and cpu migration is possible; it must take steps to be robust against = this. * (In practice, for user PTE updates, the appropriate page table lock(s) = are - * held, but for kernel PTE updates, no lock is held). The mode cannot be = used - * in interrupt context. + * held, but for kernel PTE updates, no lock is held). The mode is disabled + * in interrupt context and calls to the lazy_mmu API have no effect. * * The lazy MMU mode is enabled for a given block of code using: * @@ -265,6 +265,9 @@ static inline void lazy_mmu_mode_enable(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 + if (in_interrupt()) + return; + VM_BUG_ON(state->count =3D=3D U8_MAX); /* enable() must not be called while paused */ VM_WARN_ON(state->count > 0 && !state->enabled); @@ -280,6 +283,9 @@ static inline void lazy_mmu_mode_disable(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 + if (in_interrupt()) + return; + VM_BUG_ON(state->count =3D=3D 0); VM_WARN_ON(!state->enabled); =20 @@ -297,6 +303,9 @@ static inline void lazy_mmu_mode_pause(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 + if (in_interrupt()) + return; + VM_WARN_ON(state->count =3D=3D 0 || !state->enabled); =20 state->enabled =3D false; @@ -307,6 +316,9 @@ static inline void lazy_mmu_mode_resume(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 + if (in_interrupt()) + return; + VM_WARN_ON(state->count =3D=3D 0 || state->enabled); =20 arch_enter_lazy_mmu_mode(); diff --git a/include/linux/sched.h b/include/linux/sched.h index 2862d8bf2160..beb3e6cfddd9 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1731,6 +1731,9 @@ static inline char task_state_to_char(struct task_str= uct *tsk) #ifdef CONFIG_ARCH_LAZY_MMU static inline bool in_lazy_mmu_mode(void) { + if (in_interrupt()) + return false; + return current->lazy_mmu_state.enabled; } #else --=20 2.47.0 From nobody Thu Oct 30 18:37:26 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1760517379420438.57920770252065; Wed, 15 Oct 2025 01:36:19 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1143263.1477020 (Exim 4.92) (envelope-from ) id 1v8wzM-0002Bk-VI; Wed, 15 Oct 2025 08:36:04 +0000 Received: by outflank-mailman (output) from mailman id 1143263.1477020; Wed, 15 Oct 2025 08:36:04 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wzM-0002BX-PR; Wed, 15 Oct 2025 08:36:04 +0000 Received: by outflank-mailman (input) for mailman id 1143263; Wed, 15 Oct 2025 08:36:04 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1v8wsR-0006Qy-7H for xen-devel@lists.xenproject.org; Wed, 15 Oct 2025 08:28:55 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id fbbbcee6-a9a0-11f0-980a-7dc792cee155; Wed, 15 Oct 2025 10:28:53 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C3A1E2379; Wed, 15 Oct 2025 01:28:44 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id BF2653F66E; Wed, 15 Oct 2025 01:28:47 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: fbbbcee6-a9a0-11f0-980a-7dc792cee155 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v3 13/13] mm: introduce arch_wants_lazy_mmu_mode() Date: Wed, 15 Oct 2025 09:27:27 +0100 Message-ID: <20251015082727.2395128-14-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251015082727.2395128-1-kevin.brodsky@arm.com> References: <20251015082727.2395128-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1760517383674154100 Content-Type: text/plain; charset="utf-8" powerpc decides at runtime whether the lazy MMU mode should be used. To avoid the overhead associated with managing task_struct::lazy_mmu_state if the mode isn't used, introduce arch_wants_lazy_mmu_mode() and bail out of lazy_mmu_mode_* if it returns false. Add a default definition returning true, and an appropriate implementation for powerpc. Signed-off-by: Kevin Brodsky --- This patch seemed like a good idea to start with, but now I'm not so sure that the churn added to the generic layer is worth it. It provides a minor optimisation for just powerpc. x86 with XEN_PV also chooses at runtime whether to implement lazy_mmu helpers or not, but it doesn't fit this API so neatly and isn't handled here. --- .../include/asm/book3s/64/tlbflush-hash.h | 11 ++++++----- include/linux/pgtable.h | 16 ++++++++++++---- 2 files changed, 18 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h b/arch/powe= rpc/include/asm/book3s/64/tlbflush-hash.h index bbc54690d374..a91b354cf87c 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush-hash.h @@ -23,10 +23,14 @@ DECLARE_PER_CPU(struct ppc64_tlb_batch, ppc64_tlb_batch= ); =20 extern void __flush_tlb_pending(struct ppc64_tlb_batch *batch); =20 +#define arch_wants_lazy_mmu_mode arch_wants_lazy_mmu_mode +static inline bool arch_wants_lazy_mmu_mode(void) +{ + return !radix_enabled(); +} + static inline void arch_enter_lazy_mmu_mode(void) { - if (radix_enabled()) - return; /* * apply_to_page_range can call us this preempt enabled when * operating on kernel page tables. @@ -46,9 +50,6 @@ static inline void arch_flush_lazy_mmu_mode(void) =20 static inline void arch_leave_lazy_mmu_mode(void) { - if (radix_enabled()) - return; - arch_flush_lazy_mmu_mode(); preempt_enable(); } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 718c9c788114..db4f388d2a16 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -261,11 +261,19 @@ static inline int pmd_dirty(pmd_t pmd) * currently enabled. */ #ifdef CONFIG_ARCH_LAZY_MMU + +#ifndef arch_wants_lazy_mmu_mode +static inline bool arch_wants_lazy_mmu_mode(void) +{ + return true; +} +#endif + static inline void lazy_mmu_mode_enable(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 - if (in_interrupt()) + if (!arch_wants_lazy_mmu_mode() || in_interrupt()) return; =20 VM_BUG_ON(state->count =3D=3D U8_MAX); @@ -283,7 +291,7 @@ static inline void lazy_mmu_mode_disable(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 - if (in_interrupt()) + if (!arch_wants_lazy_mmu_mode() || in_interrupt()) return; =20 VM_BUG_ON(state->count =3D=3D 0); @@ -303,7 +311,7 @@ static inline void lazy_mmu_mode_pause(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 - if (in_interrupt()) + if (!arch_wants_lazy_mmu_mode() || in_interrupt()) return; =20 VM_WARN_ON(state->count =3D=3D 0 || !state->enabled); @@ -316,7 +324,7 @@ static inline void lazy_mmu_mode_resume(void) { struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; =20 - if (in_interrupt()) + if (!arch_wants_lazy_mmu_mode() || in_interrupt()) return; =20 VM_WARN_ON(state->count =3D=3D 0 || state->enabled); --=20 2.47.0