From nobody Sat Jun 13 23:23:41 2026 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 322F3382373; Tue, 5 May 2026 10:55:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=193.142.43.55 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777978543; cv=none; b=QxtqNiExi7bboCliDRDnUsE/c5/zglRAFTSrmgNgrspkGDzqXtBwCVQ5z3LZTovLw6K1srwRed2EArt2QpZwwLTZNx1N+b32kSo2JKcH/twdsyB+Q6MNTU7dEt6b/0zrYSztu6aMJvZJqH+NQfUluxH7LkXi1waOg0OLwnqigQI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777978543; c=relaxed/simple; bh=mE2i5OWjwRo0bMKc7t6Tl7dbJWzRnJahjWeg/FVs1D4=; h=Date:From:To:Subject:Cc:In-Reply-To:References:MIME-Version: Message-ID:Content-Type; b=K25NmRg4rIM4U2q3iIynigBFd73wkhWmxEd6WGIQoeiIdrmqmgs6t77L62rIH4BqLKv+duCKRNwVqTuHRmX6un+S9qR3hSmFGCAHiVi0DppvxDfHhZLdr7ZY2fVtxc/5p0K+nHaK+LfzCU+yF25UiOTj/sc/YVCO+N5gfwFADLk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de; spf=pass smtp.mailfrom=linutronix.de; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=VtH/GXGM; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b=VPj93TZc; arc=none smtp.client-ip=193.142.43.55 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="VtH/GXGM"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="VPj93TZc" Date: Tue, 05 May 2026 10:55:38 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1777978539; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GOAvtsohx4vrV3ka5QfNjGhEgSj12IYbS10AFGjwik8=; b=VtH/GXGMMQSYN+5pQWxgsJ6iYDeHh6PqMWD8PjvMiLtfCRYY4gUVgM7saVt6LcpbxTGn99 pbKRKNJc/ZKr+1NLOs4vqdtNp/GY7Rcsih/UjspnkdfSU2FrqCB/PBToK0GynBFocdy5ep o3wMOzfgyg1PFdQKjs19sN4h3sMR5TXgpRP+WRaHbelDAh4rNHx4twhYm0iIPxjIadMKb7 2K+DXNxFxECvY5LcIIFWCQXtZiDC7O7PH8MKq13XN1qEI6cGnzlzvHjOtOexlXrXWkTFrA j5e2pKEAvtqQk/f74+FW9QLj8pfN2sGFJwuLcJSdCTC85AOUN+aeGutBjrzuKQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1777978539; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GOAvtsohx4vrV3ka5QfNjGhEgSj12IYbS10AFGjwik8=; b=VPj93TZc/hLbLseMswvq9ddjWTPd7Ty0rh0pXM0qoNwogEG1HFuWNSnQ7Mes9jGlHlAmPr YPKIz4W4FRc//qAw== From: "tip-bot2 for Aniket Gattani" Sender: tip-bot2@linutronix.de Reply-to: linux-kernel@vger.kernel.org To: linux-tip-commits@vger.kernel.org Subject: [tip: locking/core] sched/membarrier: Use per-CPU mutexes for targeted commands Cc: Aniket Gattani , "Peter Zijlstra (Intel)" , x86@kernel.org, linux-kernel@vger.kernel.org In-Reply-To: <20260503212205.3714217-2-aniketgattani@google.com> References: <20260503212205.3714217-2-aniketgattani@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-ID: <177797853818.424702.1041919714128242532.tip-bot2@tip-bot2> Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails Precedence: bulk Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The following commit has been merged into the locking/core branch of tip: Commit-ID: 89976cd73739dcb73745705a63ccc67a8be26cdf Gitweb: https://git.kernel.org/tip/89976cd73739dcb73745705a63ccc67a8= be26cdf Author: Aniket Gattani AuthorDate: Sun, 03 May 2026 21:22:03=20 Committer: Peter Zijlstra CommitterDate: Tue, 05 May 2026 12:50:48 +02:00 sched/membarrier: Use per-CPU mutexes for targeted commands Currently, the membarrier system call uses a single global mutex (`membarrier_ipi_mutex`) to serialize expedited commands. This causes significant contention on large systems when multiple threads invoke membarrier concurrently, even if they target different CPUs. This contention becomes critical when combined with CFS bandwidth throttling/unthrottling, during which interrupts can be disabled for relatively long periods on target CPUs. If membarrier is waiting for a response from such a CPU, it holds the global mutex, blocking all other membarrier calls on the system. This cascade effect can lead to hard lockups when thousands of threads stall waiting for the mutex. Optimize `MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ` when a specific CPU is targeted by introducing per-CPU mutexes. Broadcast commands and commands without a specific CPU target continue to use the global mutex. This prevents the cascade lockup scenario. As measured by the stress test introduced in the subsequent patch, on an AMD Turin machine with 384 CPUs (2 NUMA nodes with SMT=3D2), this optimization yields 200x more throughput. Signed-off-by: Aniket Gattani Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260503212205.3714217-2-aniketgattani@googl= e.com --- kernel/sched/membarrier.c | 77 +++++++++++++++++++++----------------- 1 file changed, 44 insertions(+), 33 deletions(-) diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index 6234456..3d88e90 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -164,8 +164,26 @@ | MEMBARRIER_PRIVATE_EXPEDITED_RSEQ_BITMASK \ | MEMBARRIER_CMD_GET_REGISTRATIONS) =20 +/* + * Scoped guard for memory barriers on entry and exit. + * Matches memory barriers before & after rq->curr modification in schedul= er. + */ +DEFINE_LOCK_GUARD_0(mb, smp_mb(), smp_mb()) static DEFINE_MUTEX(membarrier_ipi_mutex); +static DEFINE_PER_CPU(struct mutex, membarrier_cpu_mutexes); + #define SERIALIZE_IPI() guard(mutex)(&membarrier_ipi_mutex) +#define SERIALIZE_IPI_CPU(cpu_id) guard(mutex)(&per_cpu(membarrier_cpu_mut= exes, cpu_id)) + +static int __init membarrier_init(void) +{ + int i; + + for_each_possible_cpu(i) + mutex_init(&per_cpu(membarrier_cpu_mutexes, i)); + return 0; +} +core_initcall(membarrier_init); =20 static void ipi_mb(void *info) { @@ -315,7 +333,6 @@ static int membarrier_global_expedited(void) =20 static int membarrier_private_expedited(int flags, int cpu_id) { - cpumask_var_t tmpmask; struct mm_struct *mm =3D current->mm; smp_call_func_t ipi_func =3D ipi_mb; =20 @@ -352,30 +369,45 @@ static int membarrier_private_expedited(int flags, in= t cpu_id) * On RISC-V, this barrier pairing is also needed for the * SYNC_CORE command when switching between processes, cf. * the inline comments in membarrier_arch_switch_mm(). + * + * Memory barrier on the caller thread _after_ we finished + * waiting for the last IPI. Matches memory barriers before + * rq->curr modification in scheduler. */ - smp_mb(); /* system call entry is not a mb. */ - - if (cpu_id < 0 && !zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) - return -ENOMEM; - - SERIALIZE_IPI(); - cpus_read_lock(); - + guard(mb)(); if (cpu_id >=3D 0) { + if (cpu_id >=3D nr_cpu_ids || !cpu_possible(cpu_id)) + return 0; + + SERIALIZE_IPI_CPU(cpu_id); + guard(cpus_read_lock)(); struct task_struct *p; =20 - if (cpu_id >=3D nr_cpu_ids || !cpu_online(cpu_id)) - goto out; + if (!cpu_online(cpu_id)) + return 0; + rcu_read_lock(); p =3D rcu_dereference(cpu_rq(cpu_id)->curr); if (!p || p->mm !=3D mm) { rcu_read_unlock(); - goto out; + return 0; } rcu_read_unlock(); + /* + * smp_call_function_single() will call ipi_func() if cpu_id + * is the calling CPU. + */ + smp_call_function_single(cpu_id, ipi_func, NULL, 1); } else { + cpumask_var_t __free(free_cpumask_var) tmpmask =3D CPUMASK_VAR_NULL; int cpu; =20 + if (!zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) + return -ENOMEM; + + SERIALIZE_IPI(); + guard(cpus_read_lock)(); + rcu_read_lock(); for_each_online_cpu(cpu) { struct task_struct *p; @@ -385,15 +417,6 @@ static int membarrier_private_expedited(int flags, int= cpu_id) __cpumask_set_cpu(cpu, tmpmask); } rcu_read_unlock(); - } - - if (cpu_id >=3D 0) { - /* - * smp_call_function_single() will call ipi_func() if cpu_id - * is the calling CPU. - */ - smp_call_function_single(cpu_id, ipi_func, NULL, 1); - } else { /* * For regular membarrier, we can save a few cycles by * skipping the current cpu -- we're about to do smp_mb() @@ -420,18 +443,6 @@ static int membarrier_private_expedited(int flags, int= cpu_id) } } =20 -out: - if (cpu_id < 0) - free_cpumask_var(tmpmask); - cpus_read_unlock(); - - /* - * Memory barrier on the caller thread _after_ we finished - * waiting for the last IPI. Matches memory barriers before - * rq->curr modification in scheduler. - */ - smp_mb(); /* exit from system call is not a mb */ - return 0; } =20