From nobody Mon Apr 6 10:32:53 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 280643D4122; Fri, 20 Mar 2026 17:57:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029433; cv=none; b=jpXwTxCljIwjc2c+VHZ1wg+POtPoAmyUuBR5GdM+1Trpk4Ik6roaXadK2+HiCSugMJJ2keqBYGxhF1n/DEl9vIPfj3llJVd8EkIdQ1tpE3Fq7++HTz8X8Zntli1TYPi040iPa7/jw2Y5KKSxhwRgX63B1zrkkXzZs1zxE7RjPsI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029433; c=relaxed/simple; bh=XXjsLSCYrZPoSfx1FEQORAbrzU8ErrlZmhCUMONwZuM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=L+M3VHF3Zlpi/whv5DOeK827+h4HSv2MWN4ZdOepKMW6FiYwngQN4TNYwWA9jsejz3G4UUngAajJdm546YINdju86xKLo3qVpvyao6RTb5dia5g8JWvuDh+EQyW7bnctNgw2tIyuSqa1/eP2IjKThpz9pyqohRB0LMyTkB4zm2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=W3M2t/RK; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="W3M2t/RK" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=Do267e3m31nTWEqviEIHaZrA779qGfFHwdpybFhU3lQ=; b=W3M2t/RKTspkygxR/jUYXC6347 3H8klbHTJcmSOooRBpb3msP3ug/h9/qNXyJ/+Dd166veelnSZM2MVC+UwUabULNA1h1ubM0XKZ8+j t5lxJnmjcMIOeidaxuY0yEZRlrYr2PqTNk4Ct/4IVpjtbu2g63mVPYtJr9uiAdZgqUKJCXj+vSL2i fDDUP7+0PbpmWyY7tq6EL0knJCr9gCObi8AEQLK0PccyUA0d4nQkne3Mn8TWGniv1K3yyv45fQfyx yjfx8fSPUn9UXfJdv+YjzCXFNOAcuz4OMz5+geyTwynH6UMCjuINaoG/P1YXsSN3HqjGmTvBLS45H bSACarhA==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w3e5u-005Nf3-5c; Fri, 20 Mar 2026 17:57:09 +0000 From: Breno Leitao Date: Fri, 20 Mar 2026 10:56:27 -0700 Subject: [PATCH v2 1/5] workqueue: fix typo in WQ_AFFN_SMT comment Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260320-workqueue_sharded-v2-1-8372930931af@debian.org> References: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> In-Reply-To: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, puranjay@kernel.org, linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Michael van der Westhuizen , kernel-team@meta.com, Chuck Lever , Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=761; i=leitao@debian.org; h=from:subject:message-id; bh=XXjsLSCYrZPoSfx1FEQORAbrzU8ErrlZmhCUMONwZuM=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpvYpr25bOynY+Sr+/qvB3irCHLXQG5E8n1SlHw e/7zJZAAmuJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCab2KawAKCRA1o5Of/Hh3 bdmhD/9NNH/ic6E0JuRUopxRmmavQF1wA/Ne5iqTU9PwVx37zFLnkORYeLafo9hTq7Xi12CnZfi bMhK0bAETDdd+n4+PqX5/zTru87IWlfilteloyuB8CEtYS/wlyTkem4Qtn33L44gECpH1fVsW+K e6lwW+kPu67z0SdKeWWwc6ItXcOsPqfBWP5aaxv3xjIG4IcREsS+5dcyhlC7FWIu4lqCBqhEO4o r6s5i4cEDd09e/nmNomcuvs0Lukkzi68B2bXDlT6mwlDFGzUV0KuZHY4Lwi4UGc+5X/2qqUQyzz oaWB5qPuwMXvWtVUrTv6B5/IhXu4gLVXu8X2FWHIvVAVsq7yv3hMJF+HB2+l+uRa0wGsiml9Zfx QiD4ZZdj4JZQqWaPdCa6kETJuxNw1fib4/VYPvSzXFfBhthhNM5TjBCCtiJuxK0bL1FMpB8TU6m OiFDqUtAyvbxW1WhWO4nsS8DwbigR+QT4edZI2dUCbTn6a/JRwMepqfSDxb/BgVOgF+FJa/10hy a0A90tk05WPnNcIdc4JC1mNRDW+f0GyAn4F2m3P0Pe/gL9pR4hzDGjpQQD2tf7LGN9LKsRlE9tp Za9hXRKgJjC/GRk31vn8uCfkmIL+TQHJjKyt9zD5+0UKpvkmzkQ00jh+aG7PuULWBo0o221+4xU w5H5loLnCYUeO+A== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Fix "poer" -> "per" in the WQ_AFFN_SMT enum comment. Signed-off-by: Breno Leitao --- include/linux/workqueue.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index a4749f56398fd..17543aec2a6e1 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -131,7 +131,7 @@ struct rcu_work { enum wq_affn_scope { WQ_AFFN_DFL, /* use system default */ WQ_AFFN_CPU, /* one pod per CPU */ - WQ_AFFN_SMT, /* one pod poer SMT */ + WQ_AFFN_SMT, /* one pod per SMT */ WQ_AFFN_CACHE, /* one pod per LLC */ WQ_AFFN_NUMA, /* one pod per NUMA node */ WQ_AFFN_SYSTEM, /* one pod across the whole system */ --=20 2.52.0 From nobody Mon Apr 6 10:32:53 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B6EF3D9030; Fri, 20 Mar 2026 17:57:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029437; cv=none; b=h0f07FLO1w6DCNJPEt4aLI9dlRAyrrJBqUOWdEdMxdN9jRQUrL9y+1gLwlDJ82ld9XvhEW1ObvMRDKyvgG/+UuEZ/0I7LO2o8qeNT7t/8YKYuP+n6oIMr8dL7c43Pf+r6w8MabCRWhbhO/4Djl22bzf+BUHcGjR/U/zXdgyF7Z0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029437; c=relaxed/simple; bh=2haM9T7RhrH3s287f0aj4MUwH8E29ZjjYaiTksLYd2c=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=P8eX2rXTIvRCnLXUuuUBOjJw182TnQgburlXklvPWkXzHHVfcei/xeJ+u12I6Mcfq/0egDRgzaAN+G38whH2x+q4ZFCRclWGY2N7jJiu36dHqwFapsd8Xw0I+mnJtk50J/Fsj3oQiKtp+A0aeneMJ/Idb5G7y5/RDw6eecyyRuk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=ojWMQJU8; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="ojWMQJU8" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=eGt+oO6gUd77Vki78P1HXz8VnZe0HmkUW7YZyAfG5l4=; b=ojWMQJU8FpR7Do1xpuDPdmCBMV tIheHbGSE0wRFUziZ3MtiObhn2MDaNn/D4tMdR80jBu4q0A7hP72uCyyHIoG/XHmjfeYGUm0ih2GN tN3bXkqNa+ifVqWzX5SRMMF2kWoD7nCLAtyHfldMdakqW3UKjpuz0J8MZoLTFnsBLyc34w6Pv0WdK t+TH/sokPbFMsxkQpRQ9Bp7cDXKIXjHMGw2dqRb0P/t6RxL9z3xCu++AAZhaw+liM9vfcVBGaNSrf A1IxaDb2hdZTciTQOHyALCKrpmDUg3lOVWTJuWWmHucxrlk21smWmB/4khPrXOmfN8iugLGI9QVcK 0on0xJDw==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w3e5y-005NfA-CT; Fri, 20 Mar 2026 17:57:13 +0000 From: Breno Leitao Date: Fri, 20 Mar 2026 10:56:28 -0700 Subject: [PATCH v2 2/5] workqueue: add WQ_AFFN_CACHE_SHARD affinity scope Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260320-workqueue_sharded-v2-2-8372930931af@debian.org> References: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> In-Reply-To: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, puranjay@kernel.org, linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Michael van der Westhuizen , kernel-team@meta.com, Chuck Lever , Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=7235; i=leitao@debian.org; h=from:subject:message-id; bh=2haM9T7RhrH3s287f0aj4MUwH8E29ZjjYaiTksLYd2c=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpvYprxdMfJdd0aSLdtoRT7SpRh+jvshNnVxoq3 H39pBnkOyGJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCab2KawAKCRA1o5Of/Hh3 bQ++D/0R3MWF4hnvgh1G6NOvdyDqoTLglQS7/eRRaza7aLU+y1IipbwSiiaFSzjgypfsJ7yb71H r6djMl/X7sJDSiPuvxguyqo4Tn2b9A+G9FhltDXqhEXW4PbMARAlqXkcWyaq6uQy3ttcxo3IB5Y hHXtrfyJg+IVhJc6EO5aBNQqfuMuzHA1USYBIJVjMdhPc1sQDsTAEoRZMKafFxgj5SY2DaYSQ69 V1GlXtCkk+kBxf8BbyTyLm4sEP5e21uTZo0+MjBQutm52YLu+/QRV1W2FGX1fPh7T9208FEcMoE OSpQXbs+NqqstBOkeo7r95b4XJu9CYWdTN8fmrorA08Tx+u6hcGZJC1NmAmGddmGXhKQqSjvguR ceVU6K5rKMjqcinOq70q2+MciYbktjZSZIWtI6D6AHd2tdsBe1/rrFZOJpr6D70rGLiANg+lFkq uKU64nBkbzfymivxPeVkvMC+Vs547MlMHUD2lvORoZqpIukOTTmgYMQRNp5L3nKLZM7yCNd6Kmj lYN4KialjsMytPHeRgsODRyeYD1iPqZgT2o3D3qGdSCLV1XV13Ffv1SYzG8Sf73QPCOXSLtAV80 YgvS7HOwdwMoxCLMZDEU5LF/1k+f5Xqh8mUBGYJEOLOxYLGONHswC7Ry7lw8NxdygN/FJ/IF2Ck qF4fnH6QuSrKMQw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao On systems where many CPUs share one LLC, unbound workqueues using WQ_AFFN_CACHE collapse to a single worker pool, causing heavy spinlock contention on pool->lock. For example, Chuck Lever measured 39% of cycles lost to native_queued_spin_lock_slowpath on a 12-core shared-L3 NFS-over-RDMA system. The existing affinity hierarchy (cpu, smt, cache, numa, system) offers no intermediate option between per-LLC and per-SMT-core granularity. Add WQ_AFFN_CACHE_SHARD, which subdivides each LLC into groups of at most wq_cache_shard_size cores (default 8, tunable via boot parameter). Shards are always split on core (SMT group) boundaries so that Hyper-Threading siblings are never placed in different pods. Cores are distributed across shards as evenly as possible -- for example, 36 cores in a single LLC with max shard size 8 produces 5 shards of 8+7+7+7+7 cores. The implementation follows the same comparator pattern as other affinity scopes: cpu_cache_shard_id() computes a per-CPU shard index on the fly from the already-initialized WQ_AFFN_CACHE and WQ_AFFN_SMT topology, and cpus_share_cache_shard() is passed to init_pod_type(). Benchmark on NVIDIA Grace (72 CPUs, single LLC, 50k items/thread), show cache_shard delivers ~5x the throughput and ~6.5x lower p50 latency compared to cache scope on this 72-core single-LLC system. Suggested-by: Tejun Heo Signed-off-by: Breno Leitao --- include/linux/workqueue.h | 1 + kernel/workqueue.c | 108 ++++++++++++++++++++++++++++++++++++++++++= ++++ 2 files changed, 109 insertions(+) diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h index 17543aec2a6e1..50bdb7e30d35f 100644 --- a/include/linux/workqueue.h +++ b/include/linux/workqueue.h @@ -133,6 +133,7 @@ enum wq_affn_scope { WQ_AFFN_CPU, /* one pod per CPU */ WQ_AFFN_SMT, /* one pod per SMT */ WQ_AFFN_CACHE, /* one pod per LLC */ + WQ_AFFN_CACHE_SHARD, /* synthetic sub-LLC shards */ WQ_AFFN_NUMA, /* one pod per NUMA node */ WQ_AFFN_SYSTEM, /* one pod across the whole system */ =20 diff --git a/kernel/workqueue.c b/kernel/workqueue.c index a050971393f1f..ebbc7971b4fa6 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -409,6 +409,7 @@ static const char * const wq_affn_names[WQ_AFFN_NR_TYPE= S] =3D { [WQ_AFFN_CPU] =3D "cpu", [WQ_AFFN_SMT] =3D "smt", [WQ_AFFN_CACHE] =3D "cache", + [WQ_AFFN_CACHE_SHARD] =3D "cache_shard", [WQ_AFFN_NUMA] =3D "numa", [WQ_AFFN_SYSTEM] =3D "system", }; @@ -431,6 +432,9 @@ module_param_named(cpu_intensive_warning_thresh, wq_cpu= _intensive_warning_thresh static bool wq_power_efficient =3D IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DE= FAULT); module_param_named(power_efficient, wq_power_efficient, bool, 0444); =20 +static unsigned int wq_cache_shard_size =3D 8; +module_param_named(cache_shard_size, wq_cache_shard_size, uint, 0444); + static bool wq_online; /* can kworkers be created yet? */ static bool wq_topo_initialized __read_mostly =3D false; =20 @@ -8107,6 +8111,104 @@ static bool __init cpus_share_numa(int cpu0, int cp= u1) return cpu_to_node(cpu0) =3D=3D cpu_to_node(cpu1); } =20 +/** + * llc_count_cores - count distinct cores (SMT groups) within a cpumask + * @pod_cpus: the cpumask to scan (typically an LLC pod) + * @smt_pt: the SMT pod type, used to identify sibling groups + * + * A core is represented by the lowest-numbered CPU in its SMT group. Retu= rns + * the number of distinct cores found in @pod_cpus. + */ +static int __init llc_count_cores(const struct cpumask *pod_cpus, + struct wq_pod_type *smt_pt) +{ + const struct cpumask *smt_cpus; + int nr_cores =3D 0, c; + + for_each_cpu(c, pod_cpus) { + smt_cpus =3D smt_pt->pod_cpus[smt_pt->cpu_pod[c]]; + if (cpumask_first(smt_cpus) =3D=3D c) + nr_cores++; + } + + return nr_cores; +} + +/** + * llc_cpu_core_pos - find a CPU's core position within a cpumask + * @cpu: the CPU to locate + * @pod_cpus: the cpumask to scan (typically an LLC pod) + * @smt_pt: the SMT pod type, used to identify sibling groups + * + * Returns the zero-based index of @cpu's core among the distinct cores in + * @pod_cpus, ordered by lowest CPU number in each SMT group. + */ +static int __init llc_cpu_core_pos(int cpu, const struct cpumask *pod_cpus, + struct wq_pod_type *smt_pt) +{ + const struct cpumask *smt_cpus; + int core_pos =3D 0, c; + + for_each_cpu(c, pod_cpus) { + smt_cpus =3D smt_pt->pod_cpus[smt_pt->cpu_pod[c]]; + if (cpumask_test_cpu(cpu, smt_cpus)) + break; + if (cpumask_first(smt_cpus) =3D=3D c) + core_pos++; + } + + return core_pos; +} + +/** + * cpu_cache_shard_id - compute the shard index for a CPU within its LLC p= od + * @cpu: the CPU to look up + * + * Returns a shard index that is unique within the CPU's LLC pod. The LLC = is + * divided into shards of at most wq_cache_shard_size cores, always split = on + * core (SMT group) boundaries so that SMT siblings are never placed in + * different shards. Cores are distributed across shards as evenly as poss= ible. + * + * Example: 36 cores with wq_cache_shard_size=3D8 gives 5 shards of + * 8+7+7+7+7 cores. + */ +static int __init cpu_cache_shard_id(int cpu) +{ + struct wq_pod_type *cache_pt =3D &wq_pod_types[WQ_AFFN_CACHE]; + struct wq_pod_type *smt_pt =3D &wq_pod_types[WQ_AFFN_SMT]; + const struct cpumask *pod_cpus; + int nr_cores, nr_shards, cores_per_shard, remainder, core_pos; + + /* CPUs in the same LLC as @cpu */ + pod_cpus =3D cache_pt->pod_cpus[cache_pt->cpu_pod[cpu]]; + nr_cores =3D llc_count_cores(pod_cpus, smt_pt); + + /* Compute number of shards from the max cores per shard */ + nr_shards =3D DIV_ROUND_UP(nr_cores, wq_cache_shard_size); + /* Distribute cores as evenly as possible across shards */ + cores_per_shard =3D nr_cores / nr_shards; + remainder =3D nr_cores % nr_shards; + + core_pos =3D llc_cpu_core_pos(cpu, pod_cpus, smt_pt); + + /* + * Map core position to shard index. The first @remainder shards have + * (cores_per_shard + 1) cores, the rest have @cores_per_shard cores. + */ + if (core_pos < remainder * (cores_per_shard + 1)) + return core_pos / (cores_per_shard + 1); + + return remainder + (core_pos - remainder * (cores_per_shard + 1)) / cores= _per_shard; +} + +static bool __init cpus_share_cache_shard(int cpu0, int cpu1) +{ + if (!cpus_share_cache(cpu0, cpu1)) + return false; + + return cpu_cache_shard_id(cpu0) =3D=3D cpu_cache_shard_id(cpu1); +} + /** * workqueue_init_topology - initialize CPU pods for unbound workqueues * @@ -8119,9 +8221,15 @@ void __init workqueue_init_topology(void) struct workqueue_struct *wq; int cpu; =20 + if (!wq_cache_shard_size) { + pr_warn("workqueue: cache_shard_size must be > 0, setting to 1\n"); + wq_cache_shard_size =3D 1; + } + init_pod_type(&wq_pod_types[WQ_AFFN_CPU], cpus_dont_share); init_pod_type(&wq_pod_types[WQ_AFFN_SMT], cpus_share_smt); init_pod_type(&wq_pod_types[WQ_AFFN_CACHE], cpus_share_cache); + init_pod_type(&wq_pod_types[WQ_AFFN_CACHE_SHARD], cpus_share_cache_shard); init_pod_type(&wq_pod_types[WQ_AFFN_NUMA], cpus_share_numa); =20 wq_topo_initialized =3D true; --=20 2.52.0 From nobody Mon Apr 6 10:32:53 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3412B3D412B; Fri, 20 Mar 2026 17:57:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029441; cv=none; b=gEzEqV2PO58JNDSzDfdVRzYE7iOis681CKmVQ//XFvyYiGUGp+KseRoALiKSt4RiOUsnEu61n9x4o0312GeN004ZKXKStqwbPeGCMMAV9ZOwhGne5aKb9talNMXU1xi3XUj/8SOve3CBtLIFZw7Wxj5KUNF6gOi5yNIwVx8PRSs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029441; c=relaxed/simple; bh=tWaivwO7eYrAhEpG850irKoR/4A9GI66PGQUBK2FMKc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eF6gNRR4AaRAJIxn9N6YSexMBkjBkT0MuE46OaqyBBVaQ+z/WmuFefIArxPSUdgW2NBrKSbAE4FUIkFrX5/43HNBQ1GTfudEmHt/ZqGbT7IMtRupBhYSMhsZ36nH8O+gJOvtmnsMkBRhaTLsIMZr78h1/BjSl+gwxP5vpKEH4Mg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=tsUKn/wy; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="tsUKn/wy" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=6xcF3MQiuG3CLGaEv7iVARfMqvoKDRBWve22AacWJmc=; b=tsUKn/wyH0UGLcorRH+T4PzRMw e1M6DBlJ/T7lmyG9h13ym7VkZaafkZ5/aZPSnRwUeM7w95+DBXLk5zBCY16xMwvJM5WmDhtZcgaAC XjsP4P/UDiE7W/r8Fjh/NdeDw7u4D+ClZdYmaink+rMJdh+jOEaVoAxTVDDU/yskLeijv7eb9+U1A a2jx4R3wg1FeLPWRrqXJJ5Cwm9NwWqLv8dujh3khTOgElkD7c2xDWPO0TN5w47QhtE6lV4Bsp/cSH Xvq7oD6loSV5BKXh05d3SKbMjoT5REYZ1jiTmdO720dTJ6K57SEc0Owz1QzSJiX/FeRMw4mjUysSt yCU2dUJQ==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w3e62-005NfM-Jn; Fri, 20 Mar 2026 17:57:17 +0000 From: Breno Leitao Date: Fri, 20 Mar 2026 10:56:29 -0700 Subject: [PATCH v2 3/5] workqueue: set WQ_AFFN_CACHE_SHARD as the default affinity scope Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260320-workqueue_sharded-v2-3-8372930931af@debian.org> References: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> In-Reply-To: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, puranjay@kernel.org, linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Michael van der Westhuizen , kernel-team@meta.com, Chuck Lever , Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=1450; i=leitao@debian.org; h=from:subject:message-id; bh=tWaivwO7eYrAhEpG850irKoR/4A9GI66PGQUBK2FMKc=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpvYpsn1Y3Ww10kwqKV9aI6PgYujQbenq4ZHTlV BwBuegQxY2JAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCab2KbAAKCRA1o5Of/Hh3 bRRvD/9aW4YO9WVdtr8xDWUMohBa+lxGoJwhiJ4Uyx/YtbrYyUzZeJbJ9nSH7Foy4+ODlYCOM09 rw2tZRPdL8gqOo6y/WagwF/3djozVdntnEfZhZJhjIWdOGpFQWw5hhNIA+u0ptUhO2IRDP6Txv7 RXws+sZ03ZVv9pSFvdqWsBnuJCONpsnw71Q8LGaQa6yqd03FdNZwfjyZIPfkHDP31i8b0i8YlTu Yk/S3wbjJj3Efy1fQnZhJGJhnHiK/Zt3n4AKlCydV+pSmlCCmZwYT17SWwKF3o9hFj0AhWnZmg9 ZgQH7M0ZKJyBzG6ZObpc9EasP8BQlS1cgdy6zptBVad2JiSmFBzoS6/kLeIXocePvJqfuXQaOD6 sWUH9a3QT/fmA3KS/QGL6F8/tLrMUfVeC175Zi8K74hWKx00CUxla2XwRckMhBwKu5rrDwRRKy4 8LRN7zHCT1Q6XUVboerdx8LrqKIFGd1POQSbCMgWi+L+ZItOzQS05eR2xBAAFsg22zt7RWjtD0w y5VNiAeohcB03ehjLVy9ZspA9NvPnkilA+aDT9VTozvkFa7G+dxgWPRamgPL43oYsdly8367p1W PCXQpj8b7Abr+6meZIsewsb5v8qSC6AcAZKCrayhbk7LQ7eQmi+1EAXmbh4puwLev5PQfQoHR9k IpbY1ZuK4ks6ewQ== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Set WQ_AFFN_CACHE_SHARD as the default affinity scope for unbound workqueues. On systems where many CPUs share one LLC, the previous default (WQ_AFFN_CACHE) collapses all CPUs to a single worker pool, causing heavy spinlock contention on pool->lock. WQ_AFFN_CACHE_SHARD subdivides each LLC into smaller groups, providing a better balance between locality and contention. Users can revert to the previous behavior with workqueue.default_affinity_scope=3Dcache. On systems with 8 or fewer cores per LLC, CACHE_SHARD produces a single shard covering the entire LLC, making it functionally identical to the previous CACHE default. The sharding only activates when an LLC has more than 8 cores. Signed-off-by: Breno Leitao --- kernel/workqueue.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/workqueue.c b/kernel/workqueue.c index ebbc7971b4fa6..6bc2e69dd5cc2 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -441,7 +441,7 @@ static bool wq_topo_initialized __read_mostly =3D false; static struct kmem_cache *pwq_cache; =20 static struct wq_pod_type wq_pod_types[WQ_AFFN_NR_TYPES]; -static enum wq_affn_scope wq_affn_dfl =3D WQ_AFFN_CACHE; +static enum wq_affn_scope wq_affn_dfl =3D WQ_AFFN_CACHE_SHARD; =20 /* buf for wq_update_unbound_pod_attrs(), protected by CPU hotplug exclusi= on */ static struct workqueue_attrs *unbound_wq_update_pwq_attrs_buf; --=20 2.52.0 From nobody Mon Apr 6 10:32:53 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 049573D1717; Fri, 20 Mar 2026 17:57:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029445; cv=none; b=dXK8SGuObGZO5GRCfcTD0WcSF4wAe1A9GB3gPDaFRhdHiqNsTnrYvtfUU/MzCh0aL/uBkFbwkOkO9pPC7bPunSHqWTGnWBWsA/DI0yw7ezoOSastp5hVz9WW0JpJaxtxwQmUADOwZl1HqrL/KoHcdBhpFflrkkQ/RycWv6gLTWE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029445; c=relaxed/simple; bh=r+H4nN8FEm5hqBTVEcbg+154s0B6+qEaV7hbQZF5BCU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Hh8A1dsyXM3SR7HDauhNNHQrSv54lQ0uZN+i7ve+cBjxRwsuhzWKbNeMTx4yGk21OUo4PDEnRyrtomd10sF4Ru79zF9qromhD1muwbZcf/RuIlM2mrWhe8G0PryeVlFYt3VuUfkRK2+ioUPoIaR4QMIJqMr6teFfyLp8F+QCoOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=krQNtRSi; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="krQNtRSi" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=VG0dZbZo9KDGej1wf7CUWCHE5nKIFztp1R/yW2wVRKY=; b=krQNtRSiD4KH6dBUVLTbh5wZ2U C59E0ggNcspYLIPiLJFSxlPy4y5mebBSN3oh4TgMMuy9ef3f3bx9+/jWIHAIMjVXwp4fQopXbj6e5 PC2OqQVB2MDoerbqc3K9UhwUIi3fFm1w8hrV+kfixzf33kz7LvvL1CnTnwgMOn4PcVzNdDSjsWrsr AsrtHLZu2YM9DAWTOr5+mB9X208QVH6yaJQPAvLac1lTj05U2Wq0FiS5RRvNrasl5akjawXV7RiOA mRCMjD0Nmt7hAVPUhJ1ZFQo4aGq6UhZa81wJPTxnoVwVGgmDHr8yle0Yk8Y6KQHy7rvYOlKhu+55b jfqRal7g==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w3e66-005Nfa-SR; Fri, 20 Mar 2026 17:57:21 +0000 From: Breno Leitao Date: Fri, 20 Mar 2026 10:56:30 -0700 Subject: [PATCH v2 4/5] tools/workqueue: add CACHE_SHARD support to wq_dump.py Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260320-workqueue_sharded-v2-4-8372930931af@debian.org> References: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> In-Reply-To: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, puranjay@kernel.org, linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Michael van der Westhuizen , kernel-team@meta.com, Chuck Lever , Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=1483; i=leitao@debian.org; h=from:subject:message-id; bh=r+H4nN8FEm5hqBTVEcbg+154s0B6+qEaV7hbQZF5BCU=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpvYpsaFOiO79PHE11A52R29Ux79F3eOeR2ERmG JvgdlJ7hMyJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCab2KbAAKCRA1o5Of/Hh3 bc2WD/9POiOiIVo0MVz5JMmDUP5PyXYm/HXvQPxroRnM/xrXp1ZRdNGKW++NTLFQA81nNLBFCig YyaFf3CIu1EGi2Hw29yri+b/f4DjeI9ATMlUpBG/orY4z14VGQGqfczUFyzx7dnRlCre4KGh7qd B3tGQ36z8UgTwTA8lIVIBonBK8YMILu8bE8pv95GVQHbrD4VGy4CpWUT+aHie255padMuHgW72z t6f4kTbZONy49xQAv9cfuxkZT2Nh7QovAZHBL+f3+pSgwI4kgAGaG5ITvMNgATax+OGEYI03O+0 VyxN2RbdxRX/rgBLQOiHriPrTNAu8H6dQBq7eRchVYu12J5d0/PqYXlMalC2sP+iktpJifcP2sh Yg8c1TKlc9dTXdevo44EbStsnxKaMZrVDOltlfX4Tycd9ZjXf8xP370WBQ2OWtQ2XZpVpXSy/mP iE5IOTL0XciwRw7NLgX6K3N0sq3U0uE1NCsIYDfUPxV0W0S8DjzLXaAsJCcn6nUUaXhPqQ99Xcr FH3zAuUe+x8fz+jPZXm+JKyHVCu7DalST6n0/gq0FrhhADER/jpbXFmcnubGWRtrfatjmDO96OU fNYGLWKq9hnw21pnZ2KrKEgO1qbFKDl4wYPZlco/kcFL2bFhDYZXsTHszeXpa92BAtpxX0rTAjt 8j4e7cYpVg6GrRg== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao The WQ_AFFN_CACHE_SHARD affinity scope was added to the kernel but wq_dump.py was not updated to enumerate it. Add the missing constant lookup and include it in the affinity scopes iteration so that drgn output shows the CACHE_SHARD pod topology alongside the other scopes. Signed-off-by: Breno Leitao --- tools/workqueue/wq_dump.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/workqueue/wq_dump.py b/tools/workqueue/wq_dump.py index d29b918306b48..06948ffcfc4b6 100644 --- a/tools/workqueue/wq_dump.py +++ b/tools/workqueue/wq_dump.py @@ -107,6 +107,7 @@ WQ_MEM_RECLAIM =3D prog['WQ_MEM_RECLAIM'] WQ_AFFN_CPU =3D prog['WQ_AFFN_CPU'] WQ_AFFN_SMT =3D prog['WQ_AFFN_SMT'] WQ_AFFN_CACHE =3D prog['WQ_AFFN_CACHE'] +WQ_AFFN_CACHE_SHARD =3D prog['WQ_AFFN_CACHE_SHARD'] WQ_AFFN_NUMA =3D prog['WQ_AFFN_NUMA'] WQ_AFFN_SYSTEM =3D prog['WQ_AFFN_SYSTEM'] =20 @@ -138,7 +139,7 @@ def print_pod_type(pt): print(f' [{cpu}]=3D{pt.cpu_pod[cpu].value_()}', end=3D'') print('') =20 -for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_NUMA, WQ_AFF= N_SYSTEM]: +for affn in [WQ_AFFN_CPU, WQ_AFFN_SMT, WQ_AFFN_CACHE, WQ_AFFN_CACHE_SHARD,= WQ_AFFN_NUMA, WQ_AFFN_SYSTEM]: print('') print(f'{wq_affn_names[affn].string_().decode().upper()}{" (default)" = if affn =3D=3D wq_affn_dfl else ""}') print_pod_type(wq_pod_types[affn]) --=20 2.52.0 From nobody Mon Apr 6 10:32:53 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AA2E3DA5A5; Fri, 20 Mar 2026 17:57:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029450; cv=none; b=HICLdnJHM0NrGH+6/ZObDfxCVAkWEVqnQRyQsbla3nM5wLVWWNr49aj7kYetf9qQvjQVAYylNKw6uFpOcll3bteWWGGtYEwMZKRHr8/0K1OHTXRMZmjOrXTHoyL6oydzbdt1vEHx+PrvJbXncZCrtV1MK/H5178u0JrowiPedUg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774029450; c=relaxed/simple; bh=fVfV+dg4pefPLaq5wbrR0HxC3vTbP1hv0mlpSnuGkSU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=G9UltzPHsN1A7EhnQP2zgbhzPik3i2+npkwKwNWe0M/eI4Zbb7VMCoEtQhuuO+tc1lleVfVcXysfRhLE+R/+4CzwYyUvrUmaWWQpKN7CVygV/7ib0tVDo4MfimzfSZBSHd8suLKX4D2mHGTWEoZ17fjlHAhLrF3Tza0lvTULTEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org; spf=none smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=NUdvJ3j6; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="NUdvJ3j6" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:In-Reply-To:References: Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description; bh=uPLEgN0Y4iP2f3Jy+ESOwggpjSmfDh5GaHCB2jXthXA=; b=NUdvJ3j6ryT8NYpywCFAenz9RW mhYJ/eybi/AR+QXBSlCv72eoblbMllR6MAP1XLMi+Camo8vfkTuPowMVo/12ZXSdsceW1dpv8K/ha w2iqVJZ+X8WfFNW8ccuU0NxlY4iahrMXbQNOrDfAPa/Mr0002BwDS2mxLBT/cYZ3u3okHap6dRiHf BGiOCFtIlZEB1mSetKWuhEBlTaQ8jjq6aUM1K0pZMTYKLp4/Xdpe3ot3XL3aBNLObgV6FwD24pGf8 40fXaOLxAvagKwVLofWiUdcKyDBHUmguEZw/uqPlHgV5u0NIg8GuTZLdENLrPyWzY8Cx9Bkalv7D3 nbOL2iXg==; Received: from authenticated user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.94.2) (envelope-from ) id 1w3e6A-005Nfl-TV; Fri, 20 Mar 2026 17:57:25 +0000 From: Breno Leitao Date: Fri, 20 Mar 2026 10:56:31 -0700 Subject: [PATCH v2 5/5] workqueue: add test_workqueue benchmark module Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260320-workqueue_sharded-v2-5-8372930931af@debian.org> References: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> In-Reply-To: <20260320-workqueue_sharded-v2-0-8372930931af@debian.org> To: Tejun Heo , Lai Jiangshan , Andrew Morton Cc: linux-kernel@vger.kernel.org, puranjay@kernel.org, linux-crypto@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Michael van der Westhuizen , kernel-team@meta.com, Chuck Lever , Breno Leitao X-Mailer: b4 0.15-dev-363b9 X-Developer-Signature: v=1; a=openpgp-sha256; l=9898; i=leitao@debian.org; h=from:subject:message-id; bh=fVfV+dg4pefPLaq5wbrR0HxC3vTbP1hv0mlpSnuGkSU=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBpvYpsQpCFFCS6Stdm13hh3rU2USIxnonVGR38q mEMNeSo1z2JAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCab2KbAAKCRA1o5Of/Hh3 bQOjD/49gPh2gnFcTxUjeB8k+HtVhOz4JJy5FQkHUNh6WYEYpnG/JUGvYoXk618cli/QxM21Y75 Unw3wSYLW2hfBUvvRG++8DxpcEE07vYGlHD6Xye1o0YlyjLHy1pTD0Pymp/y9oJtH8EG/c0yxgO 6AT7Iyv+PB6zzUVs42sfup7OWUtnNdezQqUUsENRn6MWz13UbMgvxjyyaK74AH6jRKTo2YPaHaT KA4r4WbY7qS3voEtvBJ/A49A0uPZGUZsAM28mgadY1Dt51WVZtPDAY/hFHLBnNhP0DRXmNy/HkA 4J5k2WKIWPY9H4E/xY2a16NUNWqVendA7vznOf6GISRp+pcvbmGH6LsDxtWy1qCOn6Q+8Y/T2Ve BWXbN9UbpjCT0mwuoWQdO0L92k9PmxxxqaJjkMLHzeu1pF56Me2N4OZkxxBsfDQ6x2IFkIxcyg9 zA1qUaK5eMbE5USZTMrOqUI9WRy3MJ6BbDCCv9j2e4SF0HvJY4UgBvVe0ryzVOi01QoQw6H9lqP i6ClgY/O9ypCIaOYBLH8LBLL3xRQ9fRBJYMhkYhbXhAenEt5P+OBW6ss5RaZuxj+/N9rvNhyi/C vLfovLiJfX7q4motwj3azeH2HJLZH1ic3wiCKV4dplz8mqfcyaiYV3+uIJTu0wPbI1jG6DKCOY9 NcvmzzV/x2wapfA== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Add a kernel module that benchmarks queue_work() throughput on an unbound workqueue to measure pool->lock contention under different affinity scope configurations (cache vs cache_shard). The module spawns N kthreads (default: num_online_cpus()), each bound to a different CPU. All threads start simultaneously and queue work items, measuring the latency of each queue_work() call. Results are reported as p50/p90/p95 latencies for each affinity scope. The affinity scope is switched between runs via the workqueue's sysfs affinity_scope attribute (WQ_SYSFS), avoiding the need for any new exported symbols. The module runs as __init-only, returning -EAGAIN to auto-unload, and can be re-run via insmod. Example of the output: running 50 threads, 50000 items/thread cpu 6806017 items/sec p50=3D2574 p90=3D5068 p95=3D581= 8 ns smt 6821040 items/sec p50=3D2624 p90=3D5168 p95=3D594= 9 ns cache_shard 1633653 items/sec p50=3D5337 p90=3D9694 p95=3D112= 07 ns cache 286069 items/sec p50=3D72509 p90=3D82304 p95=3D850= 09 ns numa 319403 items/sec p50=3D63745 p90=3D73480 p95=3D765= 05 ns system 308461 items/sec p50=3D66561 p90=3D75714 p95=3D780= 48 ns Signed-off-by: Breno Leitao --- lib/Kconfig.debug | 10 ++ lib/Makefile | 1 + lib/test_workqueue.c | 277 +++++++++++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 288 insertions(+) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 93f356d2b3d95..38bee649697f3 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2628,6 +2628,16 @@ config TEST_VMALLOC =20 If unsure, say N. =20 +config TEST_WORKQUEUE + tristate "Test module for stress/performance analysis of workqueue" + default n + help + This builds the "test_workqueue" module for benchmarking + workqueue throughput under contention. Useful for evaluating + affinity scope changes (e.g., cache_shard vs cache). + + If unsure, say N. + config TEST_BPF tristate "Test BPF filter functionality" depends on m && NET diff --git a/lib/Makefile b/lib/Makefile index 1b9ee167517f3..ea660cca04f40 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -79,6 +79,7 @@ UBSAN_SANITIZE_test_ubsan.o :=3D y obj-$(CONFIG_TEST_KSTRTOX) +=3D test-kstrtox.o obj-$(CONFIG_TEST_LKM) +=3D test_module.o obj-$(CONFIG_TEST_VMALLOC) +=3D test_vmalloc.o +obj-$(CONFIG_TEST_WORKQUEUE) +=3D test_workqueue.o obj-$(CONFIG_TEST_RHASHTABLE) +=3D test_rhashtable.o obj-$(CONFIG_TEST_STATIC_KEYS) +=3D test_static_keys.o obj-$(CONFIG_TEST_STATIC_KEYS) +=3D test_static_key_base.o diff --git a/lib/test_workqueue.c b/lib/test_workqueue.c new file mode 100644 index 0000000000000..a949af1d7e978 --- /dev/null +++ b/lib/test_workqueue.c @@ -0,0 +1,277 @@ +// SPDX-License-Identifier: GPL-2.0 + +/* + * Test module for stress and performance analysis of workqueue. + * + * Benchmarks queue_work() throughput on an unbound workqueue to measure + * pool->lock contention under different affinity scope configurations + * (e.g., cache vs cache_shard). + * + * The affinity scope is changed between runs via the workqueue's sysfs + * affinity_scope attribute (WQ_SYSFS). + * + * Copyright (c) 2026 Meta Platforms, Inc. and affiliates + * Copyright (c) 2026 Breno Leitao + * + */ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define WQ_NAME "bench_wq" +#define SCOPE_PATH "/sys/bus/workqueue/devices/" WQ_NAME "/affinity_scope" + +static int nr_threads; +module_param(nr_threads, int, 0444); +MODULE_PARM_DESC(nr_threads, + "Number of threads to spawn (default: 0 =3D num_online_cpus())"); + +static int wq_items =3D 50000; +module_param(wq_items, int, 0444); +MODULE_PARM_DESC(wq_items, + "Number of work items each thread queues (default: 50000)"); + +static struct workqueue_struct *bench_wq; +static atomic_t threads_done; +static DECLARE_COMPLETION(start_comp); +static DECLARE_COMPLETION(all_done_comp); + +struct thread_ctx { + struct completion work_done; + struct work_struct work; + u64 *latencies; + int cpu; + int items; +}; + +static void bench_work_fn(struct work_struct *work) +{ + struct thread_ctx *ctx =3D container_of(work, struct thread_ctx, work); + + complete(&ctx->work_done); +} + +static int bench_kthread_fn(void *data) +{ + struct thread_ctx *ctx =3D data; + ktime_t t_start, t_end; + int i; + + /* Wait for all threads to be ready */ + wait_for_completion(&start_comp); + + for (i =3D 0; i < ctx->items; i++) { + reinit_completion(&ctx->work_done); + INIT_WORK(&ctx->work, bench_work_fn); + + t_start =3D ktime_get(); + queue_work(bench_wq, &ctx->work); + t_end =3D ktime_get(); + + ctx->latencies[i] =3D ktime_to_ns(ktime_sub(t_end, t_start)); + wait_for_completion(&ctx->work_done); + } + + if (atomic_dec_and_test(&threads_done)) + complete(&all_done_comp); + + return 0; +} + +static int cmp_u64(const void *a, const void *b) +{ + u64 va =3D *(const u64 *)a; + u64 vb =3D *(const u64 *)b; + + if (va < vb) + return -1; + if (va > vb) + return 1; + return 0; +} + +static int __init set_affn_scope(const char *scope) +{ + struct file *f; + loff_t pos =3D 0; + ssize_t ret; + + f =3D filp_open(SCOPE_PATH, O_WRONLY, 0); + if (IS_ERR(f)) { + pr_err("test_workqueue: open %s failed: %ld\n", + SCOPE_PATH, PTR_ERR(f)); + return PTR_ERR(f); + } + + ret =3D kernel_write(f, scope, strlen(scope), &pos); + filp_close(f, NULL); + + if (ret < 0) { + pr_err("test_workqueue: write '%s' failed: %zd\n", scope, ret); + return ret; + } + + return 0; +} + +static int __init run_bench(int n_threads, const char *scope, const char *= label) +{ + struct task_struct **tasks; + unsigned long total_items; + struct thread_ctx *ctxs; + u64 *all_latencies; + ktime_t start, end; + int cpu, i, j, ret; + s64 elapsed_us; + + ret =3D set_affn_scope(scope); + if (ret) + return ret; + + ctxs =3D kcalloc(n_threads, sizeof(*ctxs), GFP_KERNEL); + if (!ctxs) + return -ENOMEM; + + tasks =3D kcalloc(n_threads, sizeof(*tasks), GFP_KERNEL); + if (!tasks) { + kfree(ctxs); + return -ENOMEM; + } + + total_items =3D (unsigned long)n_threads * wq_items; + all_latencies =3D kvmalloc_array(total_items, sizeof(u64), GFP_KERNEL); + if (!all_latencies) { + kfree(tasks); + kfree(ctxs); + return -ENOMEM; + } + + /* Allocate per-thread latency arrays */ + for (i =3D 0; i < n_threads; i++) { + ctxs[i].latencies =3D kvmalloc_array(wq_items, sizeof(u64), + GFP_KERNEL); + if (!ctxs[i].latencies) { + while (--i >=3D 0) + kvfree(ctxs[i].latencies); + kvfree(all_latencies); + kfree(tasks); + kfree(ctxs); + return -ENOMEM; + } + } + + atomic_set(&threads_done, n_threads); + reinit_completion(&all_done_comp); + reinit_completion(&start_comp); + + /* Create kthreads, each bound to a different online CPU */ + i =3D 0; + for_each_online_cpu(cpu) { + if (i >=3D n_threads) + break; + + ctxs[i].cpu =3D cpu; + ctxs[i].items =3D wq_items; + init_completion(&ctxs[i].work_done); + + tasks[i] =3D kthread_create(bench_kthread_fn, &ctxs[i], + "wq_bench/%d", cpu); + if (IS_ERR(tasks[i])) { + ret =3D PTR_ERR(tasks[i]); + pr_err("test_workqueue: failed to create kthread %d: %d\n", + i, ret); + while (--i >=3D 0) + kthread_stop(tasks[i]); + goto out_free; + } + + kthread_bind(tasks[i], cpu); + wake_up_process(tasks[i]); + i++; + } + + /* Start timing and release all threads */ + start =3D ktime_get(); + complete_all(&start_comp); + + /* Wait for all threads to finish */ + wait_for_completion(&all_done_comp); + + /* Drain any remaining work */ + flush_workqueue(bench_wq); + + end =3D ktime_get(); + elapsed_us =3D ktime_us_delta(end, start); + + /* Merge all per-thread latencies and sort for percentile calculation */ + j =3D 0; + for (i =3D 0; i < n_threads; i++) { + memcpy(&all_latencies[j], ctxs[i].latencies, + wq_items * sizeof(u64)); + j +=3D wq_items; + } + + sort(all_latencies, total_items, sizeof(u64), cmp_u64, NULL); + + pr_info("test_workqueue: %-16s %llu items/sec\tp50=3D%llu\tp90=3D%llu\t= p95=3D%llu ns\n", + label, + elapsed_us ? total_items * 1000000ULL / elapsed_us : 0, + all_latencies[total_items * 50 / 100], + all_latencies[total_items * 90 / 100], + all_latencies[total_items * 95 / 100]); + + ret =3D 0; +out_free: + for (i =3D 0; i < n_threads; i++) + kvfree(ctxs[i].latencies); + kvfree(all_latencies); + kfree(tasks); + kfree(ctxs); + + return ret; +} + +static const char * const bench_scopes[] =3D { + "cpu", "smt", "cache", "cache_shard", "numa", "system", +}; + +static int __init test_workqueue_init(void) +{ + int n_threads =3D nr_threads ?: num_online_cpus(); + int i; + + if (wq_items <=3D 0) { + pr_err("test_workqueue: wq_items must be > 0\n"); + return -EINVAL; + } + + bench_wq =3D alloc_workqueue(WQ_NAME, WQ_UNBOUND | WQ_SYSFS, 0); + if (!bench_wq) + return -ENOMEM; + + pr_info("test_workqueue: running %d threads, %d items/thread\n", + n_threads, wq_items); + + for (i =3D 0; i < ARRAY_SIZE(bench_scopes); i++) + run_bench(n_threads, bench_scopes[i], bench_scopes[i]); + + destroy_workqueue(bench_wq); + + return -EAGAIN; +} + +module_init(test_workqueue_init); +MODULE_AUTHOR("Breno Leitao "); +MODULE_DESCRIPTION("Stress/performance benchmark for workqueue subsystem"); +MODULE_LICENSE("GPL"); --=20 2.52.0