From nobody Sat Feb 7 19:38:58 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8A0031D389 for ; Mon, 29 Dec 2025 14:38:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019082; cv=none; b=CXxRkgitgcfWANiC/uOKA908BIZDlUH1ziCa+QsqTuarnO+HCW4rqECecOXHABgGOGlw14T8btRlwaTDjOXIFLJ229OCNxMW7b0u7gkkSXpo0HN/yReEO99U1S0oyUt2/cu3LRgK2AqQYRcon42NnOISOVna2hSwL2jgwhs3OMo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019082; c=relaxed/simple; bh=lC6GEfxIGK/35lvrPF2qw5D86rHWEQ6C/lNcw05H+jI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=euIMEKPMRNRhMyNxyCDx3tzMEEtpb9mtqzbTznlYkPTgtftdPTjAxNmI9MD7phNeYEynfOvOTQa0hoHdUI6eFrVkmxAUC3RlqFjq/aRtGMURDM6q9Kody+HKpqHz2dHCXq8z/w6lDO5OBHOdRs9L24nB6+/5OvzpqBhqOQHc4VE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=gmail.com; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2a12ed4d205so79822975ad.0 for ; Mon, 29 Dec 2025 06:38:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767019080; x=1767623880; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=foHDr6tTVVg2N+oq2ubI3TwM6NJy7Osy+6nFP/DCYQ8=; b=pNbHB+HwXhTTEoM6MEwywFddrF/RN5OZpeHcxNWp60WfZQbOHsR7oIQHOkknnRk33n 21Ssy0Dl3k+FGIHYBwEnr+G8xTi5K+tS9NmXYo3wkpFPHgrWizK9l3BTKFHUbfQVcP1U arZrOHF/y82BEeLCeRK6VoUdbBkq00p144BEjAtunRaLOxOQ3del590J5EUf3kw6lFXa mp3TBXMFHoZ4BFmudeMsZd4MzFwjTaN7giZwoSD+iz+AtLzcEMNgNWXweelqISgJeX0F 9Zuh0tT/7ongi0y6B/Gh8qp5bTsg3wVXtLEmlU6UjAUj0KUmpXk1jmuQYEWUI9ZNRieb TwBA== X-Forwarded-Encrypted: i=1; AJvYcCUfwh3wpeFALx5A7GNckodmo6XB6DRs87fN1gRz5jVQdfVIhl1PrkRbJ99OlPl/GC+tuR3XOjKEjAXVsx0=@vger.kernel.org X-Gm-Message-State: AOJu0YxYQoP4twILBEd/vsSQjHtg8ZFBYgoa+6EGR9AaWGxP+LcDjo80 F6zbnymjDDI5xI+Xvy5w3vGRUbEtbetwFQDggBPc9b4hub2fvnjLDc2x X-Gm-Gg: AY/fxX500fzt4QZhfKzDM26sIG9/tm51qjA6Iy0jMFnKI2ZjiC+f/QGQY75ub4lZKJ6 lqSJI15Tiipaj89WGfqepXzkLT+ePi7W0HjZzUNv/NdA8wygjq7XHX+WWl9Gh8QbeEsh2YYqJNQ 0k7H7Ob7dpxernViOyI1+/ZL7QHPAUdYZ15p+HfpjkHRlT7UvFP3X00+Kx2jErCzIYeTw8JehCq CKSABBtXXFLsMc1BXf84pJiTbpfWVuemalsgl/6/PifyUoaAmwVaQlc2PxtP/kYgsFdIN5COELf 3l+uGRhllawaQgvqfFOe4HdXkPafeYOh5gG6Q2Zet9UDjukq+B38xStw5lWzY1QAnWWUfNUQ6u/ QmGYI3T16gL34kKfnDCD49Ctm+K5s+M1aRb2DYoapQVdEW4oo/R3SK90FwztAfCKyJKghcWNR3/ S9D67CrWCD/uY5EwPWp2q2JJr8s54atbo= X-Google-Smtp-Source: AGHT+IEV772w153pVl2WpyWEJno3SC8dcLG3m0A8YzC4kscVWxvA6UP//UDe0ihAoD40pu8neBRzLg== X-Received: by 2002:a17:902:f607:b0:295:9e4e:4090 with SMTP id d9443c01a7336-2a2f2a3cea1mr322095945ad.52.1767019079997; Mon, 29 Dec 2025 06:37:59 -0800 (PST) Received: from EBJ9932692.tcent.cn ([45.8.220.167]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d7736fsm279669625ad.92.2025.12.29.06.37.50 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Dec 2025 06:37:58 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, arnd@arndb.de, david@kernel.org, lorenzo.stoakes@oracle.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, ioworker0@gmail.com, shy828301@gmail.com, riel@surriel.com, jannh@google.com, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH RESEND v1 1/3] mm/tlb: allow architectures to skip redundant TLB sync IPIs Date: Mon, 29 Dec 2025 22:36:55 +0800 Message-ID: <20251229143657.76968-2-lance.yang@linux.dev> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20251229143657.76968-1-lance.yang@linux.dev> References: <20251229143657.76968-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lance Yang When unsharing hugetlb PMD page tables, we currently send two IPIs: one for TLB invalidation, and another to synchronize with concurrent GUP-fast walkers. However, if the TLB flush already reaches all CPUs, the second IPI is redundant. GUP-fast runs with IRQs disabled, so when the TLB flush IPI completes, any concurrent GUP-fast must have finished. Add tlb_table_flush_implies_ipi_broadcast() to let architectures indicate their TLB flush provides full synchronization, enabling the redundant IPI to be skipped. Suggested-by: David Hildenbrand (Red Hat) Signed-off-by: Lance Yang --- include/asm-generic/tlb.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 4d679d2a206b..e8d99b5e831f 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -261,6 +261,20 @@ static inline void tlb_remove_table_sync_one(void) { } =20 #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ =20 +/* + * Architectures can override if their TLB flush already broadcasts IPIs t= o all + * CPUs when freeing or unsharing page tables. + * + * Return true only when the flush guarantees: + * - IPIs reach all CPUs with potentially stale paging-structure cache ent= ries + * - Synchronization with IRQ-disabled code like GUP-fast + */ +#ifndef tlb_table_flush_implies_ipi_broadcast +static inline bool tlb_table_flush_implies_ipi_broadcast(void) +{ + return false; +} +#endif =20 #ifndef CONFIG_MMU_GATHER_NO_GATHER /* --=20 2.49.0 From nobody Sat Feb 7 19:38:58 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2960E315D49 for ; Mon, 29 Dec 2025 14:38:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019093; cv=none; b=M9MCPSkyStGj8y0A/vduvDQVDzOQYJpvHjb19HTydbI1rjLeX8SGotGoYvTeEInjxLSczWihueDmSLhuRvldUu0/SWaqSrMnqmvSli2EMgjaNd6r548bcSlwlDyCeqPzYmkUpF8lBpIMWkWgO/izVmAUmHLYEGNE7f6Aas1ltfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019093; c=relaxed/simple; bh=m4iHj+AwFx2PCF5x5k1miVOJkb/O+RupWNG1Z8Fowrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JuF+MyYy95o9MAvCw3ElsYfUdTnBkCfPzCs5yad4/B9CYehvg3slGQ5IKkJB+HrwC9flEFJ6UUCbXfdMd6hdsW/ZUGjw9byjgT7rCRQPugfwko57YFsiPL0Ee6YX/Vid/aQiLPe6HXf4anPENMH6hX3yvhKrIAHe6ztXAHa/6KU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=gmail.com; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2a0c09bb78cso66632325ad.0 for ; Mon, 29 Dec 2025 06:38:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767019089; x=1767623889; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ZldRDJqtH1ig79a3NbAjYulmXt3x4GlRPqkLpWEwBdc=; b=KJyO1JQGNthNR1WUwDbEhOjD0vXf03uv76DZkpXNCGN0R/NnRzX+K+BSt9Fl4U7sJn +6tk1mURmCNcDBM9t6+8OibB0/tYuqNbI1IHFLwnQs876BBq7+KyaEVKXoJZqCQToIu8 Sp20Q1o+nR6GN+OEMo9zPcohsScw6GQPWpegZoRW+Gk2m82H2wuhyfG0R+wykMqlJNY3 r9FojluZxRcrFXL1yIVLINUt5neT0OOe1ZzfZpaqHct4AkDLxynUlgW99yoY9sOwKBbU CgO7IiYtbleu5+XVoALTarFuqPr8lyVxpQWwr8Ty0MNhgEcTDocx0vDW1t94zTBtsa6T M3vQ== X-Forwarded-Encrypted: i=1; AJvYcCXGninA5sUex+HZ7g+2aPZeIXLfzzxmZYBVyTMUPliqY0j6ENscTAFOyA6vulTJBldapt6q0mkJtHPfb+U=@vger.kernel.org X-Gm-Message-State: AOJu0YwOnNi9aarXXVffl206zlTnzqmbfRs+KztnITkYjBTgTnfAuDkI 7RpbdU/cTkAJ5gxcJcFkM6EbIzbaqZcgea3ZteDyhXiDsa8GbonEKJQj X-Gm-Gg: AY/fxX4lXUs+6I6KEHo99caY8d3JztzxxpBISkUzptDc+3c3HMlzDGBqgeOtCgdmmYT jrgtzCQUAgkswlB8MeUI1yrl5MYFgOdEBOG4vHVPVIYG2+ZLX/IogUGcuiVH2fczk/LOAgKB3FQ ZNhGno/SBEcmkMHVRDWEGh8Gem0X3VzcBfUCYqK1teikw8Xd1PPJeyYJsqHRIwP1cv3u10g6R5N M7tb0ae2/i9d0I5B9UZms8vm9uRhXXCcJXmXMbi1PpeYq5dNpI460Aq6io66qNAmgB4+SkMnjSA qrG3WzqZx4qvhJe79/EXZDITxduJW9rk+sQS9lYRQeqqXYf0lnRdR2JhtclzSIwfWVy1h9yHP89 rtJYbGxSBTay5fusqsTXzDiz27fSII7owyzWjwkXs1PY4qzyGj8lQpOytBVW70aDs9WFKKPC6Rj jHCEYwtPtqo/MpBr9utmTS X-Google-Smtp-Source: AGHT+IGbzQsFnxYP9S+gBanduwTJYp5i/avcPenp0gxUy3BJDakAIqFIQWIEeOZwEbX1DJzlvouniA== X-Received: by 2002:a17:902:cf0d:b0:298:45e5:54a4 with SMTP id d9443c01a7336-2a2f0caa6fdmr287168855ad.1.1767019089301; Mon, 29 Dec 2025 06:38:09 -0800 (PST) Received: from EBJ9932692.tcent.cn ([45.8.220.167]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d7736fsm279669625ad.92.2025.12.29.06.38.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Dec 2025 06:38:08 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, arnd@arndb.de, david@kernel.org, lorenzo.stoakes@oracle.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, ioworker0@gmail.com, shy828301@gmail.com, riel@surriel.com, jannh@google.com, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH RESEND v1 2/3] x86/mm: implement redundant IPI elimination for page table operations Date: Mon, 29 Dec 2025 22:36:56 +0800 Message-ID: <20251229143657.76968-3-lance.yang@linux.dev> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20251229143657.76968-1-lance.yang@linux.dev> References: <20251229143657.76968-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lance Yang Add a callback function flush_tlb_multi_implies_ipi_broadcast to pv_mmu_ops to explicitly track whether flush_tlb_multi IPIs provide sufficient synchronization for GUP-fast when freeing or unsharing page tables. Pass both freed_tables and unshared_tables to flush_tlb_mm_range() to ensure lazy-TLB CPUs receive IPIs and flush their paging-structure caches: flush_tlb_mm_range(..., freed_tables || unshared_tables); Suggested-by: David Hildenbrand (Red Hat) Signed-off-by: Lance Yang --- arch/x86/include/asm/paravirt_types.h | 6 ++++++ arch/x86/include/asm/tlb.h | 19 ++++++++++++++++++- arch/x86/kernel/paravirt.c | 10 ++++++++++ 3 files changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/p= aravirt_types.h index 3502939415ad..a5bd0983da1f 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -133,6 +133,12 @@ struct pv_mmu_ops { void (*flush_tlb_multi)(const struct cpumask *cpus, const struct flush_tlb_info *info); =20 + /* + * Indicates whether flush_tlb_multi IPIs provide sufficient + * synchronization for GUP-fast when freeing or unsharing page tables. + */ + bool (*flush_tlb_multi_implies_ipi_broadcast)(void); + /* Hook for intercepting the destruction of an mm_struct. */ void (*exit_mmap)(struct mm_struct *mm); void (*notify_page_enc_status_changed)(unsigned long pfn, int npages, boo= l enc); diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 866ea78ba156..3a7cdfdcea8e 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -5,10 +5,26 @@ #define tlb_flush tlb_flush static inline void tlb_flush(struct mmu_gather *tlb); =20 +#define tlb_table_flush_implies_ipi_broadcast tlb_table_flush_implies_ipi_= broadcast +static inline bool tlb_table_flush_implies_ipi_broadcast(void); + #include #include #include #include +#include + +static inline bool tlb_table_flush_implies_ipi_broadcast(void) +{ +#ifdef CONFIG_PARAVIRT + if (pv_ops.mmu.flush_tlb_multi_implies_ipi_broadcast) + return pv_ops.mmu.flush_tlb_multi_implies_ipi_broadcast(); + + return false; +#else + return !cpu_feature_enabled(X86_FEATURE_INVLPGB); +#endif +} =20 static inline void tlb_flush(struct mmu_gather *tlb) { @@ -20,7 +36,8 @@ static inline void tlb_flush(struct mmu_gather *tlb) end =3D tlb->end; } =20 - flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); + flush_tlb_mm_range(tlb->mm, start, end, stride_shift, + tlb->freed_tables || tlb->unshared_tables); } =20 static inline void invlpg(unsigned long addr) diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index ab3e172dcc69..4eaa44800b39 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -60,6 +60,15 @@ void __init native_pv_lock_init(void) static_branch_enable(&virt_spin_lock_key); } =20 +static bool native_flush_tlb_multi_implies_ipi_broadcast(void) +{ + /* Paravirt may use hypercalls that don't send real IPIs. */ + if (pv_ops.mmu.flush_tlb_multi !=3D native_flush_tlb_multi) + return false; + + return !cpu_feature_enabled(X86_FEATURE_INVLPGB); +} + struct static_key paravirt_steal_enabled; struct static_key paravirt_steal_rq_enabled; =20 @@ -173,6 +182,7 @@ struct paravirt_patch_template pv_ops =3D { .mmu.flush_tlb_kernel =3D native_flush_tlb_global, .mmu.flush_tlb_one_user =3D native_flush_tlb_one_user, .mmu.flush_tlb_multi =3D native_flush_tlb_multi, + .mmu.flush_tlb_multi_implies_ipi_broadcast =3D native_flush_tlb_multi_imp= lies_ipi_broadcast, =20 .mmu.exit_mmap =3D paravirt_nop, .mmu.notify_page_enc_status_changed =3D paravirt_nop, --=20 2.49.0 From nobody Sat Feb 7 19:38:58 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85EE4320A06 for ; Mon, 29 Dec 2025 14:38:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019105; cv=none; b=BVGz4xDzWxwedAQRIvQ2gbV6kQLSLlMI12HUJUGMs2TjUlePtSjdfg4S5ruIqZxo+tGEH8etwz6lD/LYP4qF37NDMpI3UWbo0RFnOr5MkuGU4laXNlzqM0fQmDaBWaK+ZgSmCNuZT0Xp8LYq+SrN+dAYVPJHGLJg3nZvdb1xCPw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1767019105; c=relaxed/simple; bh=QELGmcYo0okd7XeZ+8OIMQ2Ox7GoXMWa9Ttw3qK5W4U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rgnhRxbW5MVqHL172yhUFS7Ds6rFbyVJU4MxB0DXMVddC+dsQ+xb9KuldIIMVF41sRuyJIEikcoZWIHS0nGB7+cb47e0BajXX4kHq67/qQwdDWfEDoiYgHZ37aOUA1DWjRrjzXkzju5Efvtl8a0MdKyvaRIrrMuTtjKofAZDnXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=gmail.com; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2a0834769f0so90062975ad.2 for ; Mon, 29 Dec 2025 06:38:20 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1767019098; x=1767623898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=K2pBV4vyT2c4GBUcrUp/G3rP2JcNWZOiRH8LtGLdyis=; b=lheDW/dQpPOFeiussPFhtys0OBEZzZj98RbH+BApom4ikyawt1YaKDQzs+sGCuqcJz SQEGttT1Pdh6u68L+UsBvymsj7wZslbhuJX6mTwCfaKrOBWq84/JU1uFjTFHObxD/pr3 Xzj0bFORNEdoB9WZer6M71KNLU0NZTftNa0Zr9PzRcdyFkcJEvlk6c4dqKIxMZrmWyXE xxeEZYdLqlmepEht2TSNJVaXMCYX94/9HboN4/kq8LFMX1+doKgnrjN0My9bCenEdWf/ cAplYn8S9QNvg3w59IgTBlFcMqDsnWHWJEKuHjO6ZnXbJIuBXnhwdAiD6/3RPyvN1xne BlxA== X-Forwarded-Encrypted: i=1; AJvYcCXKG+cBJuQpORlJByMGqVLaaYoZOushLWJRxPH+2p8XbzCs3FYtO23CWtm9moxAc4awOAtF8axQvQORUGA=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/19TIC7yS5eQk53lFbyqemfQ5E2eODIDVE/HtQV7mkr2xs82F ff+Sp9LIIG+dBg26DGHpoYrOF/Xgg611DXMqceZ48L9XE12Ex7c1HYfG X-Gm-Gg: AY/fxX6zcTHMgV+2Mkj0mReDtIOGXAuSVX6n1rhTEcKmdoeh6V09UJL3dMI9Hp9OOOd GTEHNXzeEHcKklPua0Jv/z4wvOdZBvQbJtypXav1VAkMgIDwf7U++Ef0Z3Sk4OQ/JXrGNnQ97+C LTkR643ERtqZg/FtfqFGL3NX+UtGkGZztsu98fjR0CYw6RF9HIO8pWW5VMWPtmyfFn7riaj+tW3 uoIckWGg3Lr5uPegmWBXGjp7JJB+pDT+XkdSk1pa5J+sTvsuzlsNNkkv5pI2QDJI3t6CQLhoYHl UxZA1av1QXnfQvnMoKyr9dmEsQrTwoGHfpfClDoBDhdlChQL2RgXGfOMZTU+URu3bPmcbOFAl/T 0iN4xWGxKis8N2Qe1lHfGcQC50kTmGGnPdV0PIQbI79B4V/C6VqVBcXtKUSYswQukyFqjqTsUbF sqVPC+yGKDiFO814TIZD9f X-Google-Smtp-Source: AGHT+IGdJt3lkU5juysv6cnzuaDtBicBqS5Uo+6zRrWiZ1emfZCDgGtYUFWFKyKinXKOayvqccQJSQ== X-Received: by 2002:a17:902:ea11:b0:290:91d2:9304 with SMTP id d9443c01a7336-2a2f22052f9mr332042505ad.4.1767019098192; Mon, 29 Dec 2025 06:38:18 -0800 (PST) Received: from EBJ9932692.tcent.cn ([45.8.220.167]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d7736fsm279669625ad.92.2025.12.29.06.38.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 29 Dec 2025 06:38:17 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: will@kernel.org, aneesh.kumar@kernel.org, npiggin@gmail.com, peterz@infradead.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, arnd@arndb.de, david@kernel.org, lorenzo.stoakes@oracle.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, Liam.Howlett@oracle.com, npache@redhat.com, ryan.roberts@arm.com, dev.jain@arm.com, baohua@kernel.org, ioworker0@gmail.com, shy828301@gmail.com, riel@surriel.com, jannh@google.com, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH RESEND v1 3/3] mm: embed TLB flush IPI check in tlb_remove_table_sync_one() Date: Mon, 29 Dec 2025 22:36:57 +0800 Message-ID: <20251229143657.76968-4-lance.yang@linux.dev> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20251229143657.76968-1-lance.yang@linux.dev> References: <20251229143657.76968-1-lance.yang@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lance Yang Embed the tlb_table_flush_implies_ipi_broadcast() check directly inside tlb_remove_table_sync_one() instead of requiring every caller to check it explicitly. This relies on callers to do the right thing: flush with freed_tables=3Dtrue or unshared_tables=3Dtrue beforehand. All existing callers satisfy this requirement: 1. mm/khugepaged.c:1188 (collapse_huge_page): pmdp_collapse_flush(vma, address, pmd) -> flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE) -> flush_tlb_mm_range(mm, ..., freed_tables =3D true) -> flush_tlb_multi(mm_cpumask(mm), info) So freed_tables=3Dtrue before calling tlb_remove_table_sync_one(). 2. include/asm-generic/tlb.h:861 (tlb_flush_unshared_tables): tlb_flush_mmu_tlbonly(tlb) -> tlb_flush(tlb) -> flush_tlb_mm_range(mm, ..., unshared_tables =3D true) -> flush_tlb_multi(mm_cpumask(mm), info) unshared_tables=3Dtrue (equivalent to freed_tables for sending IPIs). 3. mm/mmu_gather.c:341 (__tlb_remove_table_one): When we can't allocate a batch page in tlb_remove_table(), we do: tlb_table_invalidate(tlb) -> tlb_flush_mmu_tlbonly(tlb) -> flush_tlb_mm_range(mm, ..., freed_tables =3D true) -> flush_tlb_multi(mm_cpumask(mm), info) Then: tlb_remove_table_one(table) -> __tlb_remove_table_one(table) // if !CONFIG_PT_RECLAIM -> tlb_remove_table_sync_one() freed_tables=3Dtrue, and this should work too. Why is tlb->freed_tables guaranteed? Because callers like pte_free_tlb() (via free_pte_range) set freed_tables=3Dtrue before calling __pte_free_tlb(), which then calls tlb_remove_table(). We cannot free page tables without freed_tables=3Dtrue. Note that tlb_remove_table_sync_one() was a NOP on bare metal x86 (CONFIG_MMU_GATHER_RCU_TABLE_FREE=3Dn) before commit a37259732a7d ("x86/mm: Make MMU_GATHER_RCU_TABLE_FREE unconditional"). 4-5. mm/khugepaged.c:1683,1819 (pmdp_get_lockless_sync macro): Same as #1. These also use pmdp_collapse_flush() beforehand. Suggested-by: David Hildenbrand (Red Hat) Signed-off-by: Lance Yang --- mm/mmu_gather.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 7468ec388455..7b588643cbae 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -276,6 +276,10 @@ static void tlb_remove_table_smp_sync(void *arg) =20 void tlb_remove_table_sync_one(void) { + /* Skip the IPI if the TLB flush already synchronized with other CPUs. */ + if (tlb_table_flush_implies_ipi_broadcast()) + return; + /* * This isn't an RCU grace period and hence the page-tables cannot be * assumed to be actually RCU-freed. --=20 2.49.0