From nobody Tue Jun 16 03:47:20 2026 Received: from mail-dy1-f202.google.com (mail-dy1-f202.google.com [74.125.82.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31824386549 for ; Wed, 15 Apr 2026 23:21:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295304; cv=none; b=fJcvk2z9byFg7MjB2D2Hy8O2mKoxVU9GT6zGj2G3ptWBaL4qORYT9I7XW7J5DrZFB8zfePot90rUYARnpaIrxGn8nZcjL2Fe8R8NtULfgVeDI8cCmzHqavndZEH0GFCCKWOMKI5h/GOmQ0juzf84XpB6C8PdtAzU0fHLLuLYhUI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295304; c=relaxed/simple; bh=ae1xtS698SoRwAl9b81gcSSYyvLVQWskSnVOPWRdDMs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gh7st0XbtUshQlsdx0lRFBusi7qpbGqEqL4GOw1cKaBgVRpnkLb97cfp7MgzsoY6ufjWIbNVR7fO1ldbN/YINfV84GV0L/gTV43lRQNXtoj5XjtUCaNm8zUw0ODawURUfbApvDhklKP8wfok41wzUHUouKHoXthz2yd8kHM9lmw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WlP/NUIX; arc=none smtp.client-ip=74.125.82.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WlP/NUIX" Received: by mail-dy1-f202.google.com with SMTP id 5a478bee46e88-2dd1c74508cso5747988eec.0 for ; Wed, 15 Apr 2026 16:21:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1776295301; x=1776900101; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=K0dDu/Q1pEDYYyPsxnES/b7A20+EKuhYKKH9BmyTXkk=; b=WlP/NUIXW4Y+xcwMN2H2rOqFUEVuHMAhyAV70OPV5UybgoKy0Y9/t5a26qDaaVdnS2 xueO5G3CqHCp1N6CTiDAHAr7xiilIYS7s19a1tKBsN28ZIh02EIpTSdyv1Wrdr30+yDK wRqVBGTq8jiq0dDnvAI7h8qvKr15leCFvz+dyRO84w0GJKvjRYvKErl0HHl3WYXeu0+Y /6/6/EjyvcmPDfcIllJEtAZ4GE1A9FUSGFQJ5ZN39Dd5bvcCzhun1Zibaukqh5uNGX8M hu5YapXbnepK/EvKpFRWd9te46K+nSYjiI69Lz+2CvMtRDOG+s1390cxX9g8ZYIcr3uG 1BSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776295301; x=1776900101; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=K0dDu/Q1pEDYYyPsxnES/b7A20+EKuhYKKH9BmyTXkk=; b=aSisgtCmZ66oV1Zpx2QDt44yJQoyJGPc+uS0Sr/fxxVJ3vXC3S+OcMGyCgZ4rB9IZ4 bDqEFaf1pE1ZJ1B8Hd2IVb49K3muh6Zz3LkZgNg+X+CiMi5WOe0/SjbRHoMoha2X9Hs2 9VrPVAS7p5yM1Smweg0jm6VrDO9SMZwMcuT/yNRP6htdoJZW2yc+YAFGvlgow+GtnlnF QyPQ1UkITzZ946dcqAHoZmr/K3gaYyrbb3mhx8xcYpwIOrCbn6Qr4kl3S73JtoBl4RmJ qM3Q45HQ3lmvKGKRMV3hrRlMWD7by6lWJjAgbKV0hW1zlS8K+Kr59Zu/KtrFJJL4IYS2 wKog== X-Forwarded-Encrypted: i=1; AFNElJ8i+k295fdBthv3WSLP87PV5gSHwSydF8izZ8UMPC4XhwuKuQivTDX8f9ZMP4A3Qeun1HgvklNDmftNzw8=@vger.kernel.org X-Gm-Message-State: AOJu0YzXfr355KQ1vgDpZ72wQkej3VIrACksSB2YVx+sz7deg0RpdvOu XHs063rw1nBC/Z46Ey7nWOWb7yMeAkMep5ui1b311CNehrEwY3lrpWOwGlz/vT7pA2Ecwvm83jE EAYHKBQYxgWY1gJPrD9Ktv1Hyzc1Ef5lfcg== X-Received: from dyjf14.prod.google.com ([2002:a05:7300:680e:b0:2d9:9fa3:5aa6]) (user=aniketgattani job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7300:7c0c:b0:2d4:94cc:eebb with SMTP id 5a478bee46e88-2d586ea78eamr14457155eec.13.1776295301091; Wed, 15 Apr 2026 16:21:41 -0700 (PDT) Date: Wed, 15 Apr 2026 23:21:05 +0000 In-Reply-To: <20260415232106.2803644-1-aniketgattani@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260415232106.2803644-1-aniketgattani@google.com> X-Mailer: git-send-email 2.54.0.rc1.513.gad8abe7a5a-goog Message-ID: <20260415232106.2803644-2-aniketgattani@google.com> Subject: [PATCH v2 1/2] sched/membarrier: Use per-CPU mutexes for targeted commands From: Aniket Gattani To: Mathieu Desnoyers , "Paul E . McKenney" Cc: Peter Zijlstra , Ingo Molnar , Ben Segall , Josh Don , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Aniket Gattani Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the membarrier system call uses a single global mutex (`membarrier_ipi_mutex`) to serialize expedited commands. This causes significant contention on large systems when multiple threads invoke membarrier concurrently, even if they target different CPUs. This contention becomes critical when combined with CFS bandwidth throttling/unthrottling, during which interrupts can be disabled for relatively long periods on target CPUs. If membarrier is waiting for a response from such a CPU, it holds the global mutex, blocking all other membarrier calls on the system. This cascade effect can lead to hard lockups when thousands of threads stall waiting for the mutex. Optimize `MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ` when a specific CPU is targeted by introducing per-CPU mutexes. Broadcast commands and commands without a specific CPU target continue to use the global mutex. This prevents the cascade lockup scenario. As measured by the stress test introduced in the subsequent patch, on an AMD Turin machine with 384 CPUs (2 NUMA nodes with SMT=3D2), this optimization yields 200x more throughput. Signed-off-by: Aniket Gattani --- Changes in v2: - Use different mutex macros for global vs targeted cpu membarrier (Mathi= eu). - Use (unsigned int) cpu_id >=3D nr_cpu_id (Peter). --- kernel/sched/membarrier.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/kernel/sched/membarrier.c b/kernel/sched/membarrier.c index 623445603725..7f995bd48280 100644 --- a/kernel/sched/membarrier.c +++ b/kernel/sched/membarrier.c @@ -165,7 +165,20 @@ | MEMBARRIER_CMD_GET_REGISTRATIONS) =20 static DEFINE_MUTEX(membarrier_ipi_mutex); +static DEFINE_PER_CPU(struct mutex, membarrier_cpu_mutexes); + #define SERIALIZE_IPI() guard(mutex)(&membarrier_ipi_mutex) +#define SERIALIZE_IPI_CPU(cpu_id) guard(mutex)(&per_cpu(membarrier_cpu_mut= exes, cpu_id)) + +static int __init membarrier_init(void) +{ + int i; + + for_each_possible_cpu(i) + mutex_init(&per_cpu(membarrier_cpu_mutexes, i)); + return 0; +} +core_initcall(membarrier_init); =20 static void ipi_mb(void *info) { @@ -358,14 +371,19 @@ static int membarrier_private_expedited(int flags, in= t cpu_id) if (cpu_id < 0 && !zalloc_cpumask_var(&tmpmask, GFP_KERNEL)) return -ENOMEM; =20 - SERIALIZE_IPI(); + if ((unsigned int)cpu_id >=3D nr_cpu_ids || !cpu_possible(cpu_id)) + return 0; + + SERIALIZE_IPI_CPU(cpu_id); + cpus_read_lock(); =20 if (cpu_id >=3D 0) { struct task_struct *p; =20 - if (cpu_id >=3D nr_cpu_ids || !cpu_online(cpu_id)) + if (!cpu_online(cpu_id)) goto out; + rcu_read_lock(); p =3D rcu_dereference(cpu_rq(cpu_id)->curr); if (!p || p->mm !=3D mm) { @@ -373,6 +391,11 @@ static int membarrier_private_expedited(int flags, int= cpu_id) goto out; } rcu_read_unlock(); + /* + * smp_call_function_single() will call ipi_func() if cpu_id + * is the calling CPU. + */ + smp_call_function_single(cpu_id, ipi_func, NULL, 1); } else { int cpu; =20 @@ -385,15 +408,6 @@ static int membarrier_private_expedited(int flags, int= cpu_id) __cpumask_set_cpu(cpu, tmpmask); } rcu_read_unlock(); - } - - if (cpu_id >=3D 0) { - /* - * smp_call_function_single() will call ipi_func() if cpu_id - * is the calling CPU. - */ - smp_call_function_single(cpu_id, ipi_func, NULL, 1); - } else { /* * For regular membarrier, we can save a few cycles by * skipping the current cpu -- we're about to do smp_mb() --=20 2.54.0.rc1.513.gad8abe7a5a-goog From nobody Tue Jun 16 03:47:20 2026 Received: from mail-dl1-f74.google.com (mail-dl1-f74.google.com [74.125.82.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89E303932FB for ; Wed, 15 Apr 2026 23:21:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295309; cv=none; b=qvCTW5QsaD7qWJuBODaa/8A1uqLs5A2xiy3MzdePi5Q+ayF0oT2IzsEYJQipS83tHaW3hFsX2r50VbMcJt37JFg323DXeSceJ2wO9e3FnT5wnZH3N/0qFrGrK6GmH5KV8l8G/I7z3ehIPRa1RoDwnIO+6niSNEePN7OC7hF2nWQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776295309; c=relaxed/simple; bh=Py339oSPl0sxOD7f5duoCt2jH2aS53Ig8xe4oyQoUvA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pmOaKarGtyhzT9KH2sD9YJlsgEIUwc7tl1vOEtGIcd7TZs4a72r3Bx7Jo69/lM0plvtws9ylYi01oPYIYBnwkwbE8vGPXzkCtPh2Cab2JzBHUwCADzVq66ZTIrNnsVFM/pRgxh0xl9a7Jsz/VrDg12RQ6OIgjp/tYGRAElEX5C0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z1XDEOZd; arc=none smtp.client-ip=74.125.82.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--aniketgattani.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z1XDEOZd" Received: by mail-dl1-f74.google.com with SMTP id a92af1059eb24-12c20a91932so16204453c88.1 for ; Wed, 15 Apr 2026 16:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1776295307; x=1776900107; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0T/U3yUHKoctLjbtCzv0L3UsinwxahX54+OmV3CL6JA=; b=Z1XDEOZdxG8INo5B+NKpQMgNmsAaeMHFuwY5RvsUDuzIR7ZN+eyxBny2JUYu5m+/1N tW100rgYXNND3NrXClSyUy1rLvTGL02Hj0jafFEzhEi/2tL7bfUC3zXv3X2mEqH1nmQi Iqa9obsGh4HWcJx7aoTB2bylRnJ3KVwSd2Y9YGok0IX4YNy8CXAc+NnYCkI4XakwAVth 4xvI95kaGL9oq6OEQ5D9bBd0NCRFTzMqDemadxUshmVKCgj/o/D7h4GqLQvV2P4pTJhk Ij2EDYQ71c55QwN6opf3hnwPAewac66+j4xWVySFry0jp3KzcSqD6DGc3RqhbZUV2VUJ Xjdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1776295307; x=1776900107; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0T/U3yUHKoctLjbtCzv0L3UsinwxahX54+OmV3CL6JA=; b=JWojIsrfaQE25eCE+WgsoLlwpKX3bblVrWZnibzSlUYCeW2hotaJNjJ0rroUj4zuLu GbP/c0mcBOU3fKLcOcAX145GAmu0OnjXylJoEI+YZ/9sgKw9SV+gxCZz8NbLUA4mXNz/ Mh9ey0/UL3DizMGKaYVgTZCqEYeOZG5bANUKY5YVwZaf/DINI1/ZFohzaL+uUb4HeObV vJ8e5Tdg813FCmSHK3TiRvHAChkbzuvsLCtq8HeM0jel9tkvg7Td8+QnxrYlFNYt3OX6 fLmg7sRqAQdf5AttH2O7GE/RguT/xb6788gDOVMHVUQK58vVU7lKSzq+1SBNR17XdNBB VmaQ== X-Forwarded-Encrypted: i=1; AFNElJ8P125T/nxsxBMfo/Lj1TXTN+0au8lwIC+L7Grin6gn/M3ptL7DP1Cy+zcr05bmyCtULUZuxUnBHT8HWRU=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7Oryz3rpsYfDdwPj7h3ojVPtCLMWF5VlDcyQsDQIt3x3yl5UO RSp3VKrNZKIUokKjUjp7wxcT+V1TV/2rKmB8mXNfUQvYFhRdAS27sSjbzU0E91N7UM6Vj4si5WW znkiPxCtgXcKl21yyTr1ZWyCaEHc8G0aOsQ== X-Received: from dlzz6.prod.google.com ([2002:a05:7022:486:b0:12a:ad8d:be18]) (user=aniketgattani job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:6b8d:b0:128:dedf:f56d with SMTP id a92af1059eb24-12c34eddc14mr15112308c88.24.1776295306489; Wed, 15 Apr 2026 16:21:46 -0700 (PDT) Date: Wed, 15 Apr 2026 23:21:06 +0000 In-Reply-To: <20260415232106.2803644-1-aniketgattani@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260415232106.2803644-1-aniketgattani@google.com> X-Mailer: git-send-email 2.54.0.rc1.513.gad8abe7a5a-goog Message-ID: <20260415232106.2803644-3-aniketgattani@google.com> Subject: [PATCH v2 2/2] selftests/membarrier: Add rseq stress test for CFS throttle interactions From: Aniket Gattani To: Mathieu Desnoyers , "Paul E . McKenney" Cc: Peter Zijlstra , Ingo Molnar , Ben Segall , Josh Don , linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Aniket Gattani , kernel test robot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a new stress test to exercise the interaction between targeted expedited membarrier commands and CFS bandwidth throttling. The test creates a deep cgroup hierarchy and aggressively hammers the membarrier syscall to expose lock contention and latency issues. This serves as a reliable reproducer for the `membarrier_ipi_mutex` cascade lockup, ensuring future changes to membarrier locking do not regress targeted command latency. Reported-by: kernel test robot Closes: https://lore.kernel.org/r/202604151516.Vc7Ro4LP-lkp@intel.com/ Signed-off-by: Aniket Gattani --- Changes in v2: - Removed #warning on unsupported architectures that causes build failures with W=3D1 reported by the kernel test robot. --- tools/testing/selftests/membarrier/Makefile | 5 +- .../membarrier/membarrier_rseq_stress.c | 951 ++++++++++++++++++ 2 files changed, 954 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/membarrier/membarrier_rseq_stre= ss.c diff --git a/tools/testing/selftests/membarrier/Makefile b/tools/testing/se= lftests/membarrier/Makefile index fc840e06ff56..829f95c83515 100644 --- a/tools/testing/selftests/membarrier/Makefile +++ b/tools/testing/selftests/membarrier/Makefile @@ -1,8 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only -CFLAGS +=3D -g $(KHDR_INCLUDES) +CFLAGS +=3D -g $(KHDR_INCLUDES) -pthread -I../../../../tools/include LDLIBS +=3D -lpthread =20 TEST_GEN_PROGS :=3D membarrier_test_single_thread \ - membarrier_test_multi_thread + membarrier_test_multi_thread \ + membarrier_rseq_stress =20 include ../lib.mk diff --git a/tools/testing/selftests/membarrier/membarrier_rseq_stress.c b/= tools/testing/selftests/membarrier/membarrier_rseq_stress.c new file mode 100644 index 000000000000..c188d7498610 --- /dev/null +++ b/tools/testing/selftests/membarrier/membarrier_rseq_stress.c @@ -0,0 +1,951 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Membarrier stress test for CFS throttle interactions. + * + * Reproducer for the interaction between CFS throttle and expedited memba= rrier. + */ + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +/* -- Architecture-specific rseq signature -- */ +#if defined(__x86_64__) || defined(__i386__) +# define RSEQ_SIG 0x53053053U +#elif defined(__aarch64__) +# define RSEQ_SIG 0xd428bc00U +#elif defined(__powerpc__) || defined(__powerpc64__) +# define RSEQ_SIG 0x0f000000U +#elif defined(__s390__) || defined(__s390x__) +# define RSEQ_SIG 0x0c000000U +#else +# define RSEQ_SIG 0 +# define UNSUPPORTED_ARCH 1 +#endif + +/* -- rseq ABI (kernel uapi; define locally for portability) -- */ +#define RSEQ_CPU_ID_UNINITIALIZED ((__u32)-1) + +#include + +struct rseq_abi { + __u32 cpu_id_start; + __u32 cpu_id; + __u64 rseq_cs; + __u32 flags; + __u32 node_id; + __u32 mm_cid; + char end[0]; +} __aligned(32); + +/* -- membarrier constants (not in all distro headers) -- */ +#ifndef MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ +# define MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ (1 << 7) +#endif +#ifndef MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ +# define MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ (1 << 8) +#endif +#ifndef MEMBARRIER_CMD_FLAG_CPU +# define MEMBARRIER_CMD_FLAG_CPU (1 << 0) +#endif + +/* -- Test parameters -- */ +#define N_SIBLINGS 2000 +#define NEST_DEPTH 5 +static char g_cgroup_path[4096]; +static int use_cgroup_v2; + +#define CFS_QUOTA_US 1000 +#define CFS_PERIOD_US 5000 +#define N_HAMMER_PER_CPU 25 +#define N_BURNER_PER_CPU 50 +#define MAX_STRESS_CPUS 1024 +#define TEST_DURATION_SEC 20 + +/* Latency thresholds for the sentinel */ +#define LATENCY_WARN_MS 50 +#define LATENCY_CRITICAL_MS 200 + +/* Sentinel sampling interval */ +#define SENTINEL_INTERVAL_US 500 + +/* -- Shared globals -- */ +static atomic_int g_stop; +static atomic_int g_stop_sentinel; +static atomic_long g_max_latency_us; +static atomic_long g_interval_max_latency_us; +static atomic_long g_mb_ok; +static atomic_long g_mb_err; +static int g_ncpus_stress; +static int *g_stress_cpus; + +static atomic_int g_test_ready; + +/* Per-thread rseq ABI block registered with the kernel */ +static __thread struct rseq_abi tls_rseq + __attribute__((tls_model("initial-exec"))) __aligned(32) =3D { + .cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED, +}; + +/* -- Utility -- */ +static int write_file(const char *path, const char *val) +{ + int fd =3D open(path, O_WRONLY | O_CLOEXEC); + + if (fd < 0) + return -errno; + + size_t len =3D strlen(val); + ssize_t r =3D write(fd, val, len); + + close(fd); + if (r < 0) + return -errno; + if ((size_t)r !=3D len) + return -EIO; + return 0; +} + +static uint64_t monotonic_us(void) +{ + struct timespec ts; + + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000000ULL + ts.tv_nsec / 1000ULL; +} + +static void update_max_latency(long lat) +{ + long old =3D atomic_load_explicit(&g_max_latency_us, memory_order_relaxed= ); + + while (lat > old) { + if (atomic_compare_exchange_weak_explicit(&g_max_latency_us, &old, lat, + memory_order_relaxed, memory_order_relaxed)) + break; + } + + old =3D atomic_load_explicit(&g_interval_max_latency_us, memory_order_rel= axed); + while (lat > old) { + if (atomic_compare_exchange_weak_explicit(&g_interval_max_latency_us, &o= ld, lat, + memory_order_relaxed, memory_order_relaxed)) + break; + } +} + +static void init_stress_cpus(void) +{ + cpu_set_t set; + int capacity =3D MAX_STRESS_CPUS; + + g_stress_cpus =3D malloc(capacity * sizeof(int)); + if (!g_stress_cpus) + ksft_exit_fail_msg("malloc failed for g_stress_cpus\n"); + + if (sched_getaffinity(0, sizeof(set), &set) < 0) + ksft_exit_fail_msg("sched_getaffinity failed\n"); + + for (int i =3D 0; i < CPU_SETSIZE && g_ncpus_stress < capacity; i++) { + if (CPU_ISSET(i, &set)) + g_stress_cpus[g_ncpus_stress++] =3D i; + } + + if (g_ncpus_stress =3D=3D 0) + ksft_exit_skip("No CPUs available for stress test\n"); + + ksft_print_msg("Stressing %d CPUs discovered via affinity\n", g_ncpus_str= ess); +} + +/* -- rseq / membarrier helpers -- */ +static int rseq_register_thread(void) +{ + int r =3D syscall(SYS_rseq, &tls_rseq, sizeof(tls_rseq), 0, RSEQ_SIG); + + return (r =3D=3D 0 || errno =3D=3D EBUSY || errno =3D=3D EINVAL) ? 0 : -1; +} + +static int rseq_register_thread_at(struct rseq_abi *rseq) +{ + int r =3D syscall(SYS_rseq, rseq, sizeof(*rseq), 0, RSEQ_SIG); + + return (r =3D=3D 0 || errno =3D=3D EBUSY || errno =3D=3D EINVAL) ? 0 : -1; +} + +static int membarrier_register_rseq_mm(void) +{ + return syscall(SYS_membarrier, + MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ, 0, 0); +} + +/* -- cgroup helpers -- */ +static void rm_cgroup_recursive(const char *path) +{ + DIR *dir =3D opendir(path); + + if (!dir) + return; + struct dirent *entry; + + while ((entry =3D readdir(dir)) !=3D NULL) { + if (strcmp(entry->d_name, ".") =3D=3D 0 || strcmp(entry->d_name, "..") = =3D=3D 0) + continue; + if (entry->d_type =3D=3D DT_DIR) { + char sub_path[4096]; + + snprintf(sub_path, sizeof(sub_path), "%s/%s", path, entry->d_name); + rm_cgroup_recursive(sub_path); + } + } + closedir(dir); + rmdir(path); +} + +static void cgroup_teardown(void); + +static int cgroup_setup(void) +{ + struct stat st; + + if (stat("/sys/fs/cgroup/cpu", &st) =3D=3D 0) { + use_cgroup_v2 =3D 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/dev/cgroup/cpu", &st) =3D=3D 0) { + use_cgroup_v2 =3D 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/dev/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/cgroup/cpu", &st) =3D=3D 0) { + use_cgroup_v2 =3D 0; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/cgroup/cpu/membarrier_stress_test"); + } else if (stat("/sys/fs/cgroup/cgroup.controllers", &st) =3D=3D 0) { + use_cgroup_v2 =3D 1; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/membarrier_stress_test"); + } else { + ksft_print_msg("WARN: cgroup mount not found. Using v2 at /sys/fs/cgroup= \n"); + use_cgroup_v2 =3D 1; + snprintf(g_cgroup_path, sizeof(g_cgroup_path), + "/sys/fs/cgroup/membarrier_stress_test"); + } + + /* Robust cleanup before setup */ + cgroup_teardown(); + + if (use_cgroup_v2) { + /* Enable cpu controller in root cgroup */ + if (write_file("/sys/fs/cgroup/cgroup.subtree_control", "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in /sys/fs/cgroup= \n"); + } + + if (mkdir(g_cgroup_path, 0755) < 0 && errno !=3D EEXIST) { + ksft_print_msg("mkdir base %s failed: %s\n", g_cgroup_path, strerror(err= no)); + return -1; + } + + if (use_cgroup_v2) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), "%s/cgroup.subtree_control", g_cg= roup_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in %s\n", + g_cgroup_path); + } + + for (int i =3D 0; i < N_SIBLINGS; i++) { + char sibling_path[4096]; + + snprintf(sibling_path, sizeof(sibling_path), "%s/n%d", g_cgroup_path, i); + if (mkdir(sibling_path, 0755) < 0 && errno !=3D EEXIST) { + ksft_print_msg("mkdir wide %s failed: %s\n", sibling_path, strerror(err= no)); + return -1; + } + + if (use_cgroup_v2) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), + "%s/cgroup.subtree_control", sibling_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: failed to enable cpu controller in %s\n", + sibling_path); + } + + char current_path[4096]; + + snprintf(current_path, sizeof(current_path), "%s", sibling_path); + for (int j =3D 0; j < NEST_DEPTH; j++) { + snprintf(current_path + strlen(current_path), + sizeof(current_path) - strlen(current_path), "/d%d", j); + if (mkdir(current_path, 0755) < 0 && errno !=3D EEXIST) { + ksft_print_msg("mkdir deep %s failed: %s\n", + current_path, strerror(errno)); + return -1; + } + + /* Enable for all but the leaf */ + if (use_cgroup_v2 && j < NEST_DEPTH - 1) { + char ctrl_path[4096]; + + snprintf(ctrl_path, sizeof(ctrl_path), "%s/cgroup.subtree_control", + current_path); + if (write_file(ctrl_path, "+cpu") < 0) + ksft_print_msg("WARN: cannot enable cpu controller in %s\n", + current_path); + } + } + } + + char quota[64], period[64], max_str[128]; + + snprintf(quota, sizeof(quota), "%d", CFS_QUOTA_US); + snprintf(period, sizeof(period), "%d", CFS_PERIOD_US); + snprintf(max_str, sizeof(max_str), "%d %d", CFS_QUOTA_US, CFS_PERIOD_US); + + if (use_cgroup_v2) { + char max_path[4096]; + + snprintf(max_path, sizeof(max_path), "%s/cpu.max", g_cgroup_path); + if (write_file(max_path, max_str) < 0) { + ksft_print_msg("ERROR: cannot write cpu.max at %s\n", max_path); + return -1; + } + ksft_print_msg("cgroup (v2) %s: cpu.max=3D%s\n", g_cgroup_path, max_str); + } else { + char quota_path[4096], period_path[4096]; + + snprintf(quota_path, sizeof(quota_path), "%s/cpu.cfs_quota_us", g_cgroup= _path); + snprintf(period_path, sizeof(period_path), "%s/cpu.cfs_period_us", g_cgr= oup_path); + + if (write_file(period_path, period) < 0) { + ksft_print_msg("ERROR: cannot write cpu.cfs_period_us at %s\n", + period_path); + return -1; + } + if (write_file(quota_path, quota) < 0) { + ksft_print_msg("ERROR: cannot write cpu.cfs_quota_us at %s\n", quota_pa= th); + return -1; + } + ksft_print_msg("cgroup (v1) %s: cpu.cfs_quota_us=3D%d cpu.cfs_period_us= =3D%d\n", + g_cgroup_path, CFS_QUOTA_US, CFS_PERIOD_US); + } + + return 0; +} + +static int cgroup_add_pid_to_path(pid_t pid, const char *path) +{ + char buf[32], file_path[4096]; + + snprintf(buf, sizeof(buf), "%d", (int)pid); + if (use_cgroup_v2) { + snprintf(file_path, sizeof(file_path), "%s/cgroup.procs", path); + return write_file(file_path, buf); + } + /* In v1, try tasks first, fallback to cgroup.procs */ + snprintf(file_path, sizeof(file_path), "%s/tasks", path); + int r =3D write_file(file_path, buf); + + if (r < 0) { + snprintf(file_path, sizeof(file_path), "%s/cgroup.procs", path); + r =3D write_file(file_path, buf); + } + return r; +} + +static void cgroup_teardown(void) +{ + rm_cgroup_recursive(g_cgroup_path); +} + +static void cgroup_unthrottle(void) +{ + if (use_cgroup_v2) { + char max_path[4096]; + + snprintf(max_path, sizeof(max_path), "%s/cpu.max", g_cgroup_path); + write_file(max_path, "max"); + } else { + char quota_path[4096]; + + snprintf(quota_path, sizeof(quota_path), "%s/cpu.cfs_quota_us", g_cgroup= _path); + write_file(quota_path, "-1"); + } +} + +/* -- CPU burner (inside throttled child process) -- */ +static void *burner_thread_fn(void *arg) +{ + struct rseq_abi my_rseq; + int cpu =3D (int)(uintptr_t)arg; + + memset(&my_rseq, 0, sizeof(my_rseq)); + my_rseq.cpu_id =3D RSEQ_CPU_ID_UNINITIALIZED; + + if (rseq_register_thread_at(&my_rseq) < 0) { + perror("rseq_register (burner)"); + return NULL; + } + + cpu_set_t set; + + CPU_ZERO(&set); + CPU_SET(cpu, &set); + if (sched_setaffinity(0, sizeof(set), &set) < 0) + perror("sched_setaffinity (burner)"); + + unsigned long sink =3D 0; + + while (!atomic_load_explicit(&g_stop, memory_order_relaxed)) { + sink++; + /* Prevent compiler from optimizing the loop away */ + asm volatile("" : "+g"(sink)); + } + + return NULL; +} + +static int burner_thread_fn_wrapper(void *arg) +{ + burner_thread_fn(arg); + return 0; +} + +static int leaf_child_fn(void *arg) +{ + int i =3D (int)(uintptr_t)arg; + int total_burners =3D g_ncpus_stress * N_BURNER_PER_CPU; + int n_threads_per_leaf =3D total_burners / N_SIBLINGS; + + if (i < (total_burners % N_SIBLINGS)) + n_threads_per_leaf++; + + prctl(PR_SET_PDEATHSIG, SIGTERM); + if (getppid() =3D=3D 1) + _exit(1); + + char leaf_path[4096]; + + snprintf(leaf_path, sizeof(leaf_path), "%s/n%d", g_cgroup_path, i); + for (int j =3D 0; j < NEST_DEPTH; j++) + snprintf(leaf_path + strlen(leaf_path), + sizeof(leaf_path) - strlen(leaf_path), "/d%d", j); + + int r =3D cgroup_add_pid_to_path(getpid(), leaf_path); + + if (r < 0) { + char buf[512]; + int len =3D snprintf(buf, sizeof(buf), + "[leaf child %d] failed to join cgroup %s: err %d\n", + i, leaf_path, -r); + (void)!write(2, buf, len); + _exit(1); + } + + for (int j =3D 0; j < n_threads_per_leaf; j++) { + int cpu =3D g_stress_cpus[(i * n_threads_per_leaf + j) % g_ncpus_stress]; + + /* Allocate stack via mmap (bypasses heap) */ + size_t stack_size =3D 64 * 1024; + void *stack =3D mmap(NULL, stack_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stack =3D=3D MAP_FAILED) { + const char *msg =3D "mmap stack failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + /* Use raw clone to create a thread sharing the VM and thread group */ + pid_t pid =3D clone(burner_thread_fn_wrapper, stack + stack_size, + CLONE_VM | CLONE_THREAD | CLONE_SIGHAND, + (void *)(uintptr_t)cpu); + if (pid < 0) { + const char *msg =3D "clone burner failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + } + + // Wait for SIGTERM + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, SIGTERM); + int sig; + + sigwait(&mask, &sig); + + _exit(0); +} + +struct leaf_info { + pid_t pid; + void *stack; +}; + +static int run_throttle_child(void *arg) +{ + (void)arg; + prctl(PR_SET_PDEATHSIG, SIGTERM); + if (getppid() =3D=3D 1) + _exit(1); + + int n_leafs =3D N_SIBLINGS; + + /* Block signals before spawning to avoid missing early failures */ + sigset_t mask; + + sigemptyset(&mask); + sigaddset(&mask, SIGTERM); + sigaddset(&mask, SIGCHLD); + sigprocmask(SIG_BLOCK, &mask, NULL); + + /* Use mmap for tracking structures to avoid glibc heap usage */ + struct leaf_info *leaves =3D mmap(NULL, n_leafs * sizeof(struct leaf_info= ), + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (leaves =3D=3D MAP_FAILED) { + const char *msg =3D "mmap leaves array failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + for (int i =3D 0; i < n_leafs; i++) { + size_t stack_size =3D 64 * 1024; + void *stack =3D mmap(NULL, stack_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (stack =3D=3D MAP_FAILED) { + const char *msg =3D "mmap leaf stack failed\n"; + (void)!write(2, msg, strlen(msg)); + _exit(1); + } + + leaves[i].stack =3D stack; + + pid_t pid =3D clone(leaf_child_fn, stack + stack_size, + CLONE_VM | SIGCHLD, (void *)(uintptr_t)i); + + if (pid < 0) { + const char *msg =3D "clone (leaf child) failed\n"; + (void)!write(2, msg, strlen(msg)); + + /* Clean up successfully spawned children */ + for (int j =3D 0; j < i; j++) { + kill(leaves[j].pid, SIGTERM); + waitpid(leaves[j].pid, NULL, 0); + munmap(leaves[j].stack, stack_size); + } + munmap(leaves, n_leafs * sizeof(struct leaf_info)); + + if (errno =3D=3D EAGAIN) + _exit(4); + else + _exit(1); + } + leaves[i].pid =3D pid; + } + + int failed =3D 0; + + while (1) { + int sig; + + sigwait(&mask, &sig); + + if (sig =3D=3D SIGTERM) { + break; + } else if (sig =3D=3D SIGCHLD) { + int status; + pid_t pid; + + // Reap all dead children + while ((pid =3D waitpid(-1, &status, WNOHANG)) > 0) { + for (int i =3D 0; i < n_leafs; i++) { + if (leaves[i].pid =3D=3D pid) { + leaves[i].pid =3D 0; + break; + } + } + if ((WIFEXITED(status) && WEXITSTATUS(status) !=3D 0) || + WIFSIGNALED(status)) { + char buf[128]; + int len =3D snprintf(buf, sizeof(buf), + "[manager] child %d died unexpectedly (status %d)\n", + pid, WEXITSTATUS(status)); + (void)!write(2, buf, len); + failed =3D 1; + } + } + if (failed) + break; + } + } + + // Terminate all leaf kids + for (int i =3D 0; i < n_leafs; i++) { + if (leaves[i].pid > 0) + kill(leaves[i].pid, SIGTERM); + } + + for (int i =3D 0; i < n_leafs; i++) { + if (leaves[i].pid > 0) + waitpid(leaves[i].pid, NULL, 0); + munmap(leaves[i].stack, 64 * 1024); + } + + munmap(leaves, n_leafs * sizeof(struct leaf_info)); + + _exit(failed ? 1 : 0); +} + +/* -- Membarrier hammer thread -- */ +static void *hammer_thread_fn(void *arg) +{ + int target_cpu =3D *(int *)arg; + long local_ok =3D 0; + long local_err =3D 0; + int count =3D 0; + const int batch_size =3D 1024; + + if (rseq_register_thread() < 0) { + ksft_print_msg("[hammer] rseq_register failed: %s\n", strerror(errno)); + return NULL; + } + + membarrier_register_rseq_mm(); + + while (!atomic_load_explicit(&g_stop, memory_order_relaxed)) { + int r =3D syscall(SYS_membarrier, + MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ, + MEMBARRIER_CMD_FLAG_CPU, + target_cpu); + if (__builtin_expect(r =3D=3D 0, 1)) + local_ok++; + else + local_err++; + + count++; + if (__builtin_expect(count >=3D batch_size, 0)) { + atomic_fetch_add_explicit(&g_mb_ok, local_ok, memory_order_relaxed); + atomic_fetch_add_explicit(&g_mb_err, local_err, memory_order_relaxed); + local_ok =3D 0; + local_err =3D 0; + count =3D 0; + } + } + + /* Flush any remaining counts on exit */ + if (local_ok > 0) + atomic_fetch_add_explicit(&g_mb_ok, local_ok, memory_order_relaxed); + if (local_err > 0) + atomic_fetch_add_explicit(&g_mb_err, local_err, memory_order_relaxed); + + return NULL; +} + +/* -- Latency sentinel -- */ +static void *sentinel_thread_fn(void *arg) +{ + (void)arg; + struct sched_param sp =3D { .sched_priority =3D 20 }; + + if (sched_setscheduler(0, SCHED_FIFO, &sp) < 0) + ksft_print_msg("WARN: no SCHED_FIFO for sentinel (less precise)\n"); + + while (!atomic_load_explicit(&g_test_ready, memory_order_relaxed) && + !atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + struct timespec ts =3D {0, 1000 * 1000}; /* 1ms */ + + clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); + } + + uint64_t prev =3D monotonic_us(); + + while (!atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + struct timespec ts =3D { + .tv_sec =3D 0, + .tv_nsec =3D SENTINEL_INTERVAL_US * 1000L, + }; + clock_nanosleep(CLOCK_MONOTONIC, 0, &ts, NULL); + + uint64_t now =3D monotonic_us(); + long latency_us =3D (long)(now - prev) - SENTINEL_INTERVAL_US; + + prev =3D now; + + if (latency_us <=3D 0) + continue; + + update_max_latency(latency_us); + + if (latency_us > LATENCY_CRITICAL_MS * 1000L) { + ksft_print_msg("\n[SENTINEL] CRITICAL: %ld ms delay (lockup precursor!)= \n", + latency_us / 1000); + } else if (latency_us > LATENCY_WARN_MS * 1000L) { + ksft_print_msg("\n[SENTINEL] WARN: %ld ms latency spike\n", + latency_us / 1000); + } + } + return NULL; +} + +/* -- Progress reporter -- */ +static void *reporter_thread_fn(void *arg) +{ + (void)arg; + int elapsed =3D 0; + + while (!atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) { + for (int i =3D 0; i < 5; i++) { + sleep(1); + if (atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) + break; + } + if (atomic_load_explicit(&g_stop_sentinel, memory_order_relaxed)) + break; + elapsed +=3D 5; + long interval_max =3D atomic_exchange_explicit(&g_interval_max_latency_u= s, + 0, memory_order_relaxed); + + ksft_print_msg("[%3ds] mb: ok=3D%-10ld err=3D%-8ld | max_lat=3D%ld us\n", + elapsed, + atomic_load(&g_mb_ok), + atomic_load(&g_mb_err), + interval_max); + } + return NULL; +} + +/* -- Main -- */ +int main(void) +{ + ksft_print_header(); +#ifdef UNSUPPORTED_ARCH + ksft_exit_skip("Unsupported architecture\n"); +#endif + ksft_set_plan(1); + + if (geteuid() !=3D 0) + ksft_exit_skip("Must run as root (cgroup + SCHED_FIFO)\n"); + + init_stress_cpus(); + + ksft_print_msg("=3D=3D=3D membarrier rseq + CFS unthrottle stress =3D=3D= =3D\n"); + ksft_print_msg("Stressing CPUs: %d\n", g_ncpus_stress); + ksft_print_msg("Quota: %d/%d us (~%d unthrottles/sec/CPU)\n", + CFS_QUOTA_US, CFS_PERIOD_US, + 1000000 / CFS_PERIOD_US); + ksft_print_msg("Hammer threads: %d per CPU (%d total)\n", + N_HAMMER_PER_CPU, g_ncpus_stress * N_HAMMER_PER_CPU); + ksft_print_msg("Duration: %d seconds\n\n", TEST_DURATION_SEC); + + if (cgroup_setup() < 0) { + cgroup_teardown(); + ksft_exit_skip("cgroup_setup failed (missing permissions or v2 ctrls?)\n= "); + } + + if (rseq_register_thread() < 0) { + ksft_print_msg("rseq_register (%s) failed: %s\n", __func__, strerror(err= no)); + cgroup_teardown(); + ksft_exit_skip("rseq syscall failed or not available\n"); + } + if (membarrier_register_rseq_mm() < 0) { + ksft_print_msg("MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ: %s\n" + "Kernel >=3D 5.10 with CONFIG_RSEQ required.\n", + strerror(errno)); + cgroup_teardown(); + ksft_exit_skip("membarrier register failed\n"); + } + ksft_print_msg("rseq membarrier registered OK\n"); + + sigset_t sigmask; + + sigemptyset(&sigmask); + sigaddset(&sigmask, SIGTERM); + sigprocmask(SIG_BLOCK, &sigmask, NULL); + + void *stack =3D malloc(1024 * 1024); + + if (!stack) { + perror("malloc stack"); + cgroup_teardown(); + ksft_exit_fail_msg("Malloc stack failed\n"); + } + pid_t child =3D clone(run_throttle_child, stack + 1024 * 1024, CLONE_VM |= SIGCHLD, NULL); + + if (child < 0) { + perror("clone"); + cgroup_teardown(); + ksft_exit_fail_msg("Clone failed\n"); + } + + sigprocmask(SIG_UNBLOCK, &sigmask, NULL); + ksft_print_msg("Throttle child PID %d started\n", child); + + int n_threads =3D g_ncpus_stress * N_HAMMER_PER_CPU + 2; + pthread_t *threads =3D (pthread_t *)calloc(n_threads, sizeof(pthread_t)); + int *cpuargs =3D (int *)calloc(g_ncpus_stress * N_HAMMER_PER_CPU, s= izeof(int)); + + if (!threads || !cpuargs) { + perror("calloc"); + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + ksft_exit_fail_msg("Thread allocation failed\n"); + } + + int ti =3D 0, ai =3D 0; + int r; + + ksft_print_msg("Creating sentinel thread...\n"); + r =3D pthread_create(&threads[ti], NULL, sentinel_thread_fn, NULL); + if (r !=3D 0) { + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + ksft_exit_fail_msg("pthread_create (sentinel) failed: %s\n", strerror(r)= ); + } + ti++; + + ksft_print_msg("Creating reporter thread...\n"); + r =3D pthread_create(&threads[ti], NULL, reporter_thread_fn, NULL); + if (r !=3D 0) { + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + ksft_exit_fail_msg("pthread_create (reporter) failed: %s\n", strerror(r)= ); + } + ti++; + + ksft_print_msg("Creating %d hammer threads...\n", g_ncpus_stress * N_HAMM= ER_PER_CPU); + for (int i =3D 0; i < g_ncpus_stress; i++) { + int cpu =3D g_stress_cpus[i]; + + for (int j =3D 0; j < N_HAMMER_PER_CPU; j++) { + cpuargs[ai] =3D cpu; + r =3D pthread_create(&threads[ti], NULL, hammer_thread_fn, &cpuargs[ai]= ); + if (r !=3D 0) { + ksft_print_msg("pthread_create failed at thread %d: %s\n", + ti, strerror(r)); + + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + pthread_join(threads[1], NULL); + + atomic_store(&g_stop, 1); + for (int k =3D 2; k < ti; k++) + pthread_join(threads[k], NULL); + + kill(child, SIGTERM); + waitpid(child, NULL, 0); + cgroup_teardown(); + + free(threads); + free(cpuargs); + free(g_stress_cpus); + + if (r =3D=3D EAGAIN) + ksft_exit_skip("Resource limits prevent threads\n"); + else + ksft_exit_fail_msg("Failed to create hammer thread\n"); + } + ti++; + ai++; + } + } + + ksft_print_msg("All threads running. Tip: monitor dmesg for lockups\n\n"); + + atomic_store_explicit(&g_test_ready, 1, memory_order_relaxed); + int child_failed =3D 0; + int child_status =3D 0; + + for (int i =3D 0; i < TEST_DURATION_SEC; i++) { + sleep(1); + int r =3D waitpid(child, &child_status, WNOHANG); + + if (r =3D=3D child) { + child_failed =3D 1; + break; + } + } + + atomic_store(&g_stop_sentinel, 1); + pthread_join(threads[0], NULL); + pthread_join(threads[1], NULL); + + atomic_store(&g_stop, 1); + + /* Unthrottle to allow children to exit quickly */ + cgroup_unthrottle(); + + if (!child_failed) { + kill(child, SIGTERM); + waitpid(child, NULL, 0); + } + for (int i =3D 2; i < ti; i++) + pthread_join(threads[i], NULL); + + long max_lat =3D atomic_load(&g_max_latency_us); + long total_ok =3D atomic_load(&g_mb_ok); + long total_err =3D atomic_load(&g_mb_err); + + ksft_print_msg("\n=3D=3D=3D RESULTS =3D=3D=3D\n"); + ksft_print_msg("membarrier syscalls : %ld ok %ld errors\n", total_ok, to= tal_err); + ksft_print_msg("Max scheduler latency: %ld us (%ld ms)\n", max_lat, max_= lat / 1000); + cgroup_teardown(); + free(threads); + free(cpuargs); + free(g_stress_cpus); + + if (child_failed) { + if (WIFEXITED(child_status) && WEXITSTATUS(child_status) =3D=3D 4) + ksft_exit_skip("Manager child skipped (resource limits?)\n"); + ksft_test_result_fail("membarrier_rseq_stress: Manager child died early\= n"); + ksft_exit_fail(); + } else if (total_ok =3D=3D 0) { + ksft_test_result_fail("membarrier_rseq_stress: No successful membarrier = calls\n"); + ksft_exit_fail(); + } else if (total_err > 0) { + ksft_test_result_fail("membarrier_rseq_stress: syscall errors\n"); + ksft_exit_fail(); + } else if (max_lat > LATENCY_CRITICAL_MS * 1000L) { + ksft_test_result_fail("membarrier_rseq_stress: LOCKUP PRECURSOR\n"); + ksft_exit_fail(); + } else if (max_lat > LATENCY_WARN_MS * 1000L) { + ksft_test_result_fail("membarrier_rseq_stress: significant latency spike= \n"); + ksft_exit_fail(); + } else { + ksft_test_result_pass("membarrier_rseq_stress\n"); + ksft_exit_pass(); + } + + return 0; +} --=20 2.54.0.rc1.513.gad8abe7a5a-goog