From nobody Thu Oct 30 22:56:27 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=arm.com Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1761732647715159.19853626262739; Wed, 29 Oct 2025 03:10:47 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1152622.1483193 (Exim 4.92) (envelope-from ) id 1vE38R-0003po-J8; Wed, 29 Oct 2025 10:10:31 +0000 Received: by outflank-mailman (output) from mailman id 1152622.1483193; Wed, 29 Oct 2025 10:10:31 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vE38R-0003pb-EN; Wed, 29 Oct 2025 10:10:31 +0000 Received: by outflank-mailman (input) for mailman id 1152622; Wed, 29 Oct 2025 10:10:30 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1vE38Q-0001HF-0r for xen-devel@lists.xenproject.org; Wed, 29 Oct 2025 10:10:30 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id 7ed31410-b4af-11f0-9d16-b5c5bf9af7f9; Wed, 29 Oct 2025 11:10:29 +0100 (CET) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 856C31AC1; Wed, 29 Oct 2025 03:10:20 -0700 (PDT) Received: from e123572-lin.arm.com (e123572-lin.cambridge.arm.com [10.1.194.54]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 70AF73F66E; Wed, 29 Oct 2025 03:10:23 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7ed31410-b4af-11f0-9d16-b5c5bf9af7f9 From: Kevin Brodsky To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Kevin Brodsky , Alexander Gordeev , Andreas Larsson , Andrew Morton , Boris Ostrovsky , Borislav Petkov , Catalin Marinas , Christophe Leroy , Dave Hansen , David Hildenbrand , "David S. Miller" , David Woodhouse , "H. Peter Anvin" , Ingo Molnar , Jann Horn , Juergen Gross , "Liam R. Howlett" , Lorenzo Stoakes , Madhavan Srinivasan , Michael Ellerman , Michal Hocko , Mike Rapoport , Nicholas Piggin , Peter Zijlstra , Ryan Roberts , Suren Baghdasaryan , Thomas Gleixner , Vlastimil Babka , Will Deacon , Yeoreum Yun , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, x86@kernel.org Subject: [PATCH v4 07/12] mm: enable lazy_mmu sections to nest Date: Wed, 29 Oct 2025 10:09:04 +0000 Message-ID: <20251029100909.3381140-8-kevin.brodsky@arm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20251029100909.3381140-1-kevin.brodsky@arm.com> References: <20251029100909.3381140-1-kevin.brodsky@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZM-MESSAGEID: 1761732648908154100 Despite recent efforts to prevent lazy_mmu sections from nesting, it remains difficult to ensure that it never occurs - and in fact it does occur on arm64 in certain situations (CONFIG_DEBUG_PAGEALLOC). Commit 1ef3095b1405 ("arm64/mm: Permit lazy_mmu_mode to be nested") made nesting tolerable on arm64, but without truly supporting it: the inner call to leave() disables the batching optimisation before the outer section ends. This patch actually enables lazy_mmu sections to nest by tracking the nesting level in task_struct, in a similar fashion to e.g. pagefault_{enable,disable}(). This is fully handled by the generic lazy_mmu helpers that were recently introduced. lazy_mmu sections were not initially intended to nest, so we need to clarify the semantics w.r.t. the arch_*_lazy_mmu_mode() callbacks. This patch takes the following approach: * The outermost calls to lazy_mmu_mode_{enable,disable}() trigger calls to arch_{enter,leave}_lazy_mmu_mode() - this is unchanged. * Nested calls to lazy_mmu_mode_{enable,disable}() are not forwarded to the arch via arch_{enter,leave} - lazy MMU remains enabled so the assumption is that these callbacks are not relevant. However, existing code may rely on a call to disable() to flush any batched state, regardless of nesting. arch_flush_lazy_mmu_mode() is therefore called in that situation. A separate interface was recently introduced to temporarily pause the lazy MMU mode: lazy_mmu_mode_{pause,resume}(). pause() fully exits the mode *regardless of the nesting level*, and resume() restores the mode at the same nesting level. Whether the mode is actually enabled or not at any point is tracked by a separate "active" field in task_struct; this makes it possible to check invariants in the generic API, and to expose a new in_lazy_mmu_mode() helper to replace the various ways arch's currently track whether the mode is enabled (this will be done in later patches). In summary (nesting/active represent the values *after* the call): lazy_mmu_mode_enable() -> arch_enter() nesting=3D1 active=3D1 lazy_mmu_mode_enable() -> =C3=B8 nesting=3D2 active=3D1 lazy_mmu_mode_pause() -> arch_leave() nesting=3D2 active=3D0 lazy_mmu_mode_resume() -> arch_enter() nesting=3D2 active=3D1 lazy_mmu_mode_disable() -> arch_flush() nesting=3D1 active=3D1 lazy_mmu_mode_disable() -> arch_leave() nesting=3D0 active=3D0 Note: in_lazy_mmu_mode() is added to to allow arch headers included by to use it. Signed-off-by: Kevin Brodsky --- arch/arm64/include/asm/pgtable.h | 12 ------ include/linux/mm_types_task.h | 5 +++ include/linux/pgtable.h | 67 ++++++++++++++++++++++++++++++-- include/linux/sched.h | 16 ++++++++ 4 files changed, 84 insertions(+), 16 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 54f8d6bb6f22..535435248923 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -82,18 +82,6 @@ static inline void queue_pte_barriers(void) =20 static inline void arch_enter_lazy_mmu_mode(void) { - /* - * lazy_mmu_mode is not supposed to permit nesting. But in practice this - * does happen with CONFIG_DEBUG_PAGEALLOC, where a page allocation - * inside a lazy_mmu_mode section (such as zap_pte_range()) will change - * permissions on the linear map with apply_to_page_range(), which - * re-enters lazy_mmu_mode. So we tolerate nesting in our - * implementation. The first call to arch_leave_lazy_mmu_mode() will - * flush and clear the flag such that the remainder of the work in the - * outer nest behaves as if outside of lazy mmu mode. This is safe and - * keeps tracking simple. - */ - if (in_interrupt()) return; =20 diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index a82aa80c0ba4..632d404f8191 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -88,4 +88,9 @@ struct tlbflush_unmap_batch { #endif }; =20 +struct lazy_mmu_state { + u8 nesting_level; + bool active; +}; + #endif /* _LINUX_MM_TYPES_TASK_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b5fdf32c437f..e6064e00b22d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -228,27 +228,86 @@ static inline int pmd_dirty(pmd_t pmd) * of the lazy mode. So the implementation must assume preemption may be e= nabled * and cpu migration is possible; it must take steps to be robust against = this. * (In practice, for user PTE updates, the appropriate page table lock(s) = are - * held, but for kernel PTE updates, no lock is held). Nesting is not perm= itted - * and the mode cannot be used in interrupt context. + * held, but for kernel PTE updates, no lock is held). The mode cannot be = used + * in interrupt context. + * + * The lazy MMU mode is enabled for a given block of code using: + * + * lazy_mmu_mode_enable(); + * + * lazy_mmu_mode_disable(); + * + * Nesting is permitted: may itself use an enable()/disable() pair. + * A nested call to enable() has no functional effect; however disable() c= auses + * any batched architectural state to be flushed regardless of nesting. Af= ter a + * call to disable(), the caller can therefore rely on all previous page t= able + * modifications to have taken effect, but the lazy MMU mode may still be + * enabled. + * + * In certain cases, it may be desirable to temporarily pause the lazy MMU= mode. + * This can be done using: + * + * lazy_mmu_mode_pause(); + * + * lazy_mmu_mode_resume(); + * + * This sequence must only be used if the lazy MMU mode is already enabled. + * pause() ensures that the mode is exited regardless of the nesting level; + * resume() re-enters the mode at the same nesting level. must not = modify + * the lazy MMU state (i.e. it must not call any of the lazy_mmu_mode_* + * helpers). + * + * in_lazy_mmu_mode() can be used to check whether the lazy MMU mode is + * currently enabled. */ #ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE static inline void lazy_mmu_mode_enable(void) { - arch_enter_lazy_mmu_mode(); + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON_ONCE(state->nesting_level =3D=3D U8_MAX); + /* enable() must not be called while paused */ + VM_WARN_ON(state->nesting_level > 0 && !state->active); + + if (state->nesting_level++ =3D=3D 0) { + state->active =3D true; + arch_enter_lazy_mmu_mode(); + } } =20 static inline void lazy_mmu_mode_disable(void) { - arch_leave_lazy_mmu_mode(); + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON_ONCE(state->nesting_level =3D=3D 0); + VM_WARN_ON(!state->active); + + if (--state->nesting_level =3D=3D 0) { + state->active =3D false; + arch_leave_lazy_mmu_mode(); + } else { + /* Exiting a nested section */ + arch_flush_lazy_mmu_mode(); + } } =20 static inline void lazy_mmu_mode_pause(void) { + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON(state->nesting_level =3D=3D 0 || !state->active); + + state->active =3D false; arch_leave_lazy_mmu_mode(); } =20 static inline void lazy_mmu_mode_resume(void) { + struct lazy_mmu_state *state =3D ¤t->lazy_mmu_state; + + VM_WARN_ON(state->nesting_level =3D=3D 0 || state->active); + + state->active =3D true; arch_enter_lazy_mmu_mode(); } #else diff --git a/include/linux/sched.h b/include/linux/sched.h index cbb7340c5866..11566d973f42 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1441,6 +1441,10 @@ struct task_struct { =20 struct page_frag task_frag; =20 +#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE + struct lazy_mmu_state lazy_mmu_state; +#endif + #ifdef CONFIG_TASK_DELAY_ACCT struct task_delay_info *delays; #endif @@ -1724,6 +1728,18 @@ static inline char task_state_to_char(struct task_st= ruct *tsk) return task_index_to_char(task_state_index(tsk)); } =20 +#ifdef CONFIG_ARCH_HAS_LAZY_MMU_MODE +static inline bool in_lazy_mmu_mode(void) +{ + return current->lazy_mmu_state.active; +} +#else +static inline bool in_lazy_mmu_mode(void) +{ + return false; +} +#endif + extern struct pid *cad_pid; =20 /* --=20 2.47.0