From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 322C41FDE31 for ; Wed, 5 Nov 2025 23:02:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383749; cv=none; b=pm9l6Ct2LJFvWsvTdloOixAxzsYrLiFzi1Uy0uG1SDvPjqZGP+fEDPOos+v3XsHEP8B0X5bZxAl2Wq2KPoIx+XGe0BTPCDrCtRsjIcjhDct8e1ZH9m0uIuNpUSls9xpxDM+7yUBXKomBjKfUF/iIhMhBhCQzeKQXrdJeFigBC2E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383749; c=relaxed/simple; bh=O+TJ1giWcgX8q8AVGe652HR0hPyTAH9TmZB/EtY9dHk=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=J/4hdUtcC/KX52/v8R7fqDDF8qTAfUxDCNBqlyT3QZYHnrhw3liLgFlCL386JJvJciT4E+GfecEWtDVSJ0p3Yp4fQV0FMALu4Rz8X7JOthj/EQinqggjxp+cLMfovXBbwh7PCZtegGk32v30v1c+szkHFxTGSXRaqTlx/hqsCk4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CS/LjCIE; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CS/LjCIE" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-295ceaf8dacso3673865ad.0 for ; Wed, 05 Nov 2025 15:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383747; x=1762988547; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=WwAIlG6TTHwluLnC19ePK0hNp9oFcPVSx2rVuCQMRhQ=; b=CS/LjCIEyPP388iMN8FFCVk34LhhJxphqF3MPao06xoHOEpgOn8h1fvlcGhow8QCkt 1/k46yMUxVes7BTiLVTXutBhUu601LmL1RPM7dqRxGt0exuUdDzZIqSw/tZRxB7OYgsp NCW/b6/9Uknyhaq5RS8HsgwdVoKGj4VDMQwimHw44rG+UpvKF/zsCwPwRQuZp0uGbncg +mRtE4xqcZ28wBgdEy/rPvCkVtf2sJkUtBRyRt/obR1Ohi/lf2zAOpvn+g2Te9fzBXtX lkdlTIL+lDkd9UuF1rpGXzymvFuW2XHErONL9tNlcNvqB17acJfK38vpoimasYY+ABsc 3S5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383747; x=1762988547; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WwAIlG6TTHwluLnC19ePK0hNp9oFcPVSx2rVuCQMRhQ=; b=qfliNxwnPGfu63fjUjuziTGqACvH5wnlWwf4wG68hnCW8/Cj4R8Gv7b0EieShvAJlo 8DZR/q6XNQUwdamuVSuugQ0RH9mZ6DPTdI71L1s45XTZQvzx24P+J/p3tB8jjetYsZV6 OW759pVuVldyr0N30n2q7411X7IEep9PHF5WnIanxeh+GdKiG2RpTvcONwqDShUEch79 steENQEO1ZjTpgD09ezbcDQOapYdC1DaJArHT1hUpID4v7sCOxtIENJEDldozd6ZQeaN iLaN106zieA7ysZzqBQbkOjSM5c6wfFfTMv6dIIy1y3d/Y6MFlEmbCcivnHUWf/1mrD/ BZ7A== X-Forwarded-Encrypted: i=1; AJvYcCUGuWooptt6NaZhiCA/daUjuHiL1kymX65Nw49S7XCwpAbaeBrvDYiZkZp+2znZvRyp5/DsoOgTs9/f+0I=@vger.kernel.org X-Gm-Message-State: AOJu0YzyArrcn3oQT5e7yR3OKCG4KdLBxDIW8ArZLt3zS5XY/gdkp5A2 MEGuKHjNvmYoBTqHmpN3JKpfvbvtZi6ck+5IX6Lz7n4Wb8hYzW/ZmPFcEs8Qx/Ae X-Gm-Gg: ASbGncvxgcPoKYS38ng7pR5N0QRSrDSgXQu5XLLzJw5/yf5BafwocjVCJ7kedJgY/C8 bPaFKpU1k9+m9G1SYTcU1G2ndYQRrmYoPi4QbIm94ugjjNONOpTK/a4Kmfq0K69ZhL+0855fjs+ 7HE8vixpcKXD5mqQzRn3dIm0K4UVTsntOkFt3WPNCmkQxVEFrsfeLP4EoHxnjH2/JydXYjZQ9S3 JwgkztE7DMGDcuZLhHHJDWUnf0vJMa+MV13dXEYh2F1b/teadTgo6A97sCxY8+utArCGBXgBZLU i67EChdXFTkdXJIvhA4wFzJN8YwrvX2GkVtSC97Jwec7XeLIawFrIAwq5HBgLSo40DMJND0ZB9a MMVOPiVQW7I5JtvlfDtidCnEzH6KEjm6qDaJ4gOM5YFZS1MqPrvzTQuNElQ0QHuID8AePCRvXdq X1WzBcjxvWd3U9JCa21G3g4wk= X-Google-Smtp-Source: AGHT+IE331S/e6OJPAmtF3ZjSt+Kn7kDN7/mWSvZsNxV7o3ouRhjWuYNtynxsbk6GSGpD8SmdbjGKQ== X-Received: by 2002:a17:902:c401:b0:295:5da6:6014 with SMTP id d9443c01a7336-2962ad2ae93mr63812035ad.22.1762383747306; Wed, 05 Nov 2025 15:02:27 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:26 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:13 -0800 Subject: [PATCH v4 1/9] rust: cpumask: Add a `Cpumask` iterator Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-1-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=2630; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=O+TJ1giWcgX8q8AVGe652HR0hPyTAH9TmZB/EtY9dHk=; b=8IczBPVeTYnGGmDoLSMEmJlu5vzuoGi7eS1E9YY+hxeS+rAglAWrTxtFqNGLBBFPmAYcnTMTJ x4pMKIkcfTyAX8RTADOq/K1QFRETacSNE1KN9TEQp5gLyy/1NwS76JQ X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add an iterator for `Cpumask` making use of C's `cpumask_next`. Acked-by: Viresh Kumar Signed-off-by: Mitchell Levy --- rust/helpers/cpumask.c | 5 +++++ rust/kernel/cpumask.rs | 48 ++++++++++++++++++++++++++++++++++++++++++++++= +- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/rust/helpers/cpumask.c b/rust/helpers/cpumask.c index eb10598a0242..d95bfa111191 100644 --- a/rust/helpers/cpumask.c +++ b/rust/helpers/cpumask.c @@ -42,6 +42,11 @@ bool rust_helper_cpumask_full(struct cpumask *srcp) return cpumask_full(srcp); } =20 +unsigned int rust_helper_cpumask_next(int n, struct cpumask *srcp) +{ + return cpumask_next(n, srcp); +} + unsigned int rust_helper_cpumask_weight(struct cpumask *srcp) { return cpumask_weight(srcp); diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index 3fcbff438670..b7401848f59e 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -6,7 +6,7 @@ =20 use crate::{ alloc::{AllocError, Flags}, - cpu::CpuId, + cpu::{self, CpuId}, prelude::*, types::Opaque, }; @@ -161,6 +161,52 @@ pub fn copy(&self, dstp: &mut Self) { } } =20 +/// Iterator for a `Cpumask`. +pub struct CpumaskIter<'a> { + mask: &'a Cpumask, + last: Option, +} + +impl<'a> CpumaskIter<'a> { + /// Creates a new `CpumaskIter` for the given `Cpumask`. + fn new(mask: &'a Cpumask) -> CpumaskIter<'a> { + Self { mask, last: None } + } +} + +impl<'a> Iterator for CpumaskIter<'a> { + type Item =3D CpuId; + + fn next(&mut self) -> Option { + // SAFETY: By the type invariant, `self.mask.as_raw` is a `struct = cpumask *`. + let next =3D unsafe { + bindings::cpumask_next( + if let Some(last) =3D self.last { + last.try_into().unwrap() + } else { + -1 + }, + self.mask.as_raw(), + ) + }; + + if next =3D=3D cpu::nr_cpu_ids() { + None + } else { + self.last =3D Some(next); + // SAFETY: `cpumask_next` returns either `nr_cpu_ids` or a val= id CPU ID. + unsafe { Some(CpuId::from_u32_unchecked(next)) } + } + } +} + +impl Cpumask { + /// Returns an iterator over the set bits in the cpumask. + pub fn iter(&self) -> CpumaskIter<'_> { + CpumaskIter::new(self) + } +} + /// A CPU Mask pointer. /// /// Rust abstraction for the C `struct cpumask_var_t`. --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AB6A62D12EE for ; Wed, 5 Nov 2025 23:02:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383751; cv=none; b=jgf0aVGWYfuMVMs0yy3m07SUVGggNn7x7xhBXUxga7z74U45QVZXh10FEs7kLDjt6XdShn5tpCVEmt6lsL1j9PI4cuGfEZWxTtlNZpZNDAeZoHNEN1ELqIzSRL55e8za00QFZvOjKlbZgv4GNE5q16C6fRxnLvu8tLTMqyGJOCQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383751; c=relaxed/simple; bh=YISdcADBQqzuvoNcCs3nt3kan2jGJvuSVytELmc4hYw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eayEoYEyWJ3qRukMSqXt3uDoJOv6RKGQEU/rZcekQxjuP7DIenlmUG5j3YxVDPRf2UaqYU1e06yciTVtthwhvaO5kAKWqG5hO/52iNNjMMH7ADyKgpHyw1A9UItf6QPVjL8Du8wvWseHoiY0l7w2n+FpYv7/Rqf9d/m4Ok372so= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CdlFPIIK; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CdlFPIIK" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-294fd2ca6acso11323465ad.0 for ; Wed, 05 Nov 2025 15:02:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383749; x=1762988549; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=N6ObyT3Tt6gveKOeNa7NjmSIElr//qB7g+FJoPS22Qs=; b=CdlFPIIKw8nPl77hy9x+De6I2lF5QOHNGi/f6NxtdwGA7td/FlTk9Zfqz+4C3x16j7 sivq/Ywzk8a1R/VMA2J+0lk4EbdHb/c/aVcnCGaFVshtxuhWdAVXnmZr+YPAgQh6aVxi d41yWYmZMJffnFvyA3UN2IsbsfXbZ/cjJa2wR+v4+VJgrrXC9MjnNJ+O/skV06NGw5fK OpnRAhaTc0KmzJFsBogDMDMTURzPfZN2FFRbQ0BbaSXG0F6SjL1SYzouh0Czci9LMp4t A53y+wzfg8x5lJHGH0v49uuavIyK0lBG0FucmgyJOlGFhDZFWQp3JYrOcjM2rHPCW6K5 nFYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383749; x=1762988549; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=N6ObyT3Tt6gveKOeNa7NjmSIElr//qB7g+FJoPS22Qs=; b=Ixtc8bKK8/rGjgya3+ijCriquUDzF1tNsLK1IZdhiAq+ulrSgJh4DGzC9DzGUMui0b yZS7Zb64774Kib18MQ+1L+ZxKpKQ326lsfuh2fvPflydagH38jlGNqdV6Uo50px8FTCH B2vLIVUCZaGyi7ZguzBgWev3YP2Dk5C0G+5bxkYxBUckKQfzXPgXmSsmp2uK9BRc9qs8 pqudiKkeBpWxKv9XHnHOhjx34kb18ljTUaZMqj3kmjZoAHgXOv55otwI6mx+at9YLIVZ Ajfkgt0OFB0zEp2tz154SKx4Z0JV3QOhRKxHGX8uv7N/ft6PJuUlLc9KJTyhriXbfFMo +WhQ== X-Forwarded-Encrypted: i=1; AJvYcCVISzOlEJa3i4DtBL8A6gK/qqwifdveInp4B/nYJhHtJz8d66mJnzq22f5f0SRkSloSjixohIhxiu+3AfM=@vger.kernel.org X-Gm-Message-State: AOJu0Yye56dzzV28EW9n1cBt/Kabje9wyjSGOY9LJ/NxGOZ0/YsrFqBm 5tq7++KCt/2iUzAzRflPquVgj9VN1ER+DnNRZO/OAJhfCwY9ghkUUOBL X-Gm-Gg: ASbGnctfYX1FAxmrqD/UTc3pumP7uBOJvhabDfmBhhRfShTk7T/9NY0L6ZMTDs7RqXR FpqKLnRoT5uy7Rl7nvCNBeTwZFTrA5Ews93yzFvRCixEl/hQhtBdd3455Lcj12Y9cbMSDXXMfyG thT4M5wPHefELIpcApAeMbW3eCOdAUGx4YPqxRnGBvzM1P3ZJTO4t4NYcqnGFENPRseq4tnrUfx +sG59kRgYfTQrwizym/aiPsfo2YMvZoCize1hak1k1/jU7G/qQ0fVGL5JOqccDqE9OXMuGnIbFB wRPhFlDrahRt3Ro3/UxDeO4Av8ym+iXov+z1ypcpCB9qfiHA7/8euSQBhJdwCY/OeqpmGciBwPJ OYYDu2n4vrDgs4DfYVYQFelsD03gnfVgnM0iQy6XmyaaIl+SXgZVFUp6j2VkGd+/L4sUgKqetA7 2fbyDUEhPqp0bl6nDi0ndOz+4= X-Google-Smtp-Source: AGHT+IGQIBe/+YtHTkbz7bKrh6tH5nkd3eR2+Az8/eXaBw/mQbeZbYIkMzrirQWpuRr+NTG9Wz3U+w== X-Received: by 2002:a17:903:3ba4:b0:271:9b0e:54c7 with SMTP id d9443c01a7336-296509646ecmr14131025ad.11.1762383748599; Wed, 05 Nov 2025 15:02:28 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:27 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:14 -0800 Subject: [PATCH v4 2/9] rust: cpumask: Add getters for globally defined cpumasks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-2-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=3255; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=YISdcADBQqzuvoNcCs3nt3kan2jGJvuSVytELmc4hYw=; b=2O8l3qrtYhENMJ4ZqlEU7sc/NrZB0x6OGnOnA5sdLydLEIj+Vo8YliGfFxguuE1tvcxcUt3d7 91hJBW1anFIDRnuwpi6vwLUehbr2kbnrL4UdQbknVNHrFetNwdFUNHM X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add getters for the global cpumasks documented in `include/linux/cpumask.h`, specifically: - cpu_possible_mask - cpu_online_mask - cpu_enabled_mask - cpu_present_mask - cpu_active_mask Acked-by: Viresh Kumar Suggested-by: Yury Norov Signed-off-by: Mitchell Levy Acked-by: Yury Norov (NVIDIA) --- rust/kernel/cpumask.rs | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index b7401848f59e..a6a130092fcb 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -77,6 +77,52 @@ pub unsafe fn as_ref<'a>(ptr: *const bindings::cpumask) = -> &'a Self { unsafe { &*ptr.cast() } } =20 + /// Get a CPU mask representing possible CPUs; has bit `cpu` set iff c= pu is populatable + #[inline] + pub fn possible_cpus() -> &'static Self { + // SAFETY: `__cpu_possible_mask` is a valid global provided by the= kernel that lives + // forever. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_possible_mask)= } + } + + /// Get a CPU mask representing online CPUs; has bit `cpu` set iff cpu= available to the + /// scheduler + #[inline] + pub fn online_cpus() -> &'static Self { + // SAFETY: `__cpu_online_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_online_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_online_mask) } + } + + /// Get a CPU mask representing enabled CPUs; has bit `cpu` set iff cp= u can be brought online + #[inline] + pub fn enabled_cpus() -> &'static Self { + // SAFETY: `__cpu_enabled_mask` is a valid global provided by the = kernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_enabled_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_enabled_mask) } + } + + /// Get a CPU mask representing present CPUs; has bit `cpu` set iff cp= u is populated + #[inline] + pub fn present_cpus() -> &'static Self { + // SAFETY: `__cpu_present_mask` is a valid global provided by the = kernel that lives + // forever. Since we wrap the returned pointer in an `Opaque`, it'= s ok that + // `__cpu_present_mask` may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_present_mask) } + } + + /// Get a CPU mask representing active CPUs; has bit `cpu` set iff cpu= is available to + /// migration. + #[inline] + pub fn active_cpus() -> &'static Self { + // SAFETY: `__cpu_active_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_active_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_active_mask) } + } + /// Obtain the raw `struct cpumask` pointer. pub fn as_raw(&self) -> *mut bindings::cpumask { let this: *const Self =3D self; --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E2C039FCE for ; Wed, 5 Nov 2025 23:02:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383751; cv=none; b=Srqll95P98EpStnmFsouSGZ7ILovDfZBQt5dWgR9KBqKBmra0uFfjT3qgpR73J046dJ/QR1shblnFCUY2ryDDiXe4GAN1rz26vlpE03dMrq6gH1KkZshKhVCv7ExC1mj2/heCLrrZBL5qx5IdoE1ox6Gc1yPA0Idwhr/GP0uCFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383751; c=relaxed/simple; bh=RzgX0CxF7lpE2xBLdNMOZqdleGvbOwroWvnQQx0XUzQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=KRkoIvf82BhzZ0ALtZMjIUaXFS0/uYHExEPcx9ptEOABQ3kamQowz50N5C77Yk0afL64HcwJ8mKb/4/pIObd542lADPY5GBWLlFSOSuyEZzOHBrOjHFVBshKt8OSeMBQk2qGtoiiMHXkgQ8FjcRmEpFwbCNLD1I8Mf9tS5t/m3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Xurdhus7; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xurdhus7" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-b8c0c0cdd61so262087a12.2 for ; Wed, 05 Nov 2025 15:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383750; x=1762988550; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=8tDmiGouP/lZqmqIQ4xLpqbP6meWUNvUiS6WQT0r7JQ=; b=Xurdhus7wqBhpTZcVHR93kfwVS1H+7KsZWpsGgxkOAkdu9lACI3zr74u0+YTjuT+ys +2W8p5ou8jJRooQeg3djL18LayhG7izB7+47B6z+f83dEkvT7ZvnE9Zfj81Rj38zRwmk J7rVfXLuUKym2sN72RUZyH+MxP06gs/VSE+wu6cBSbzVSVJjFnCC/E/qaC91lq9yKr/7 Zy5pGjDt3689V4Om3uLBTO3y8XnGOSL0KOg/4iDuoONYPpYjU5dsdWKDNnnw0SXDopR9 S/rPJ3hokmoCB2evS9JrerCaJuUi+jLXolYA016U2BC0y9lN10ER4onGttsU6yrgxdBy 7Jbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383750; x=1762988550; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8tDmiGouP/lZqmqIQ4xLpqbP6meWUNvUiS6WQT0r7JQ=; b=TPpe6UUHKkD9HAj3rEuyH8n7RvMJnAcwMf40jDbwTsCGTT4PtDQNoTN+JDeanhSk4R N4SV+ED0o/Qsmj8/4O4N4h+eSDQbBHrCRIUnphtNkQe+syALLo2qEUpfmmUpOk2CTgHZ cu68j/iZnpg1A1oM2cTPMuB777HHxGoa6mpyKEJx4PS+aGNIpe5sy1VMX1NYeBzuAfA5 UH3wJqEq0VCde/kysXssfsOzBCdKYkxM32d4JI6hVqxScxlLYu9cpyLwm3SfMed60Iq8 HceMwJkERMkT3uwKyEioJeOZKTbCgqipgM4kX5gOFwggOYXAVx8rmnziZsW50uaGoZye EdCw== X-Forwarded-Encrypted: i=1; AJvYcCWhKU826/utthJh57cqeLgNiX5gvTVJMswdQePMlkf9iHyuZ8dLHa1j0D13gTTM85UUr+Jec1xv7snIpoY=@vger.kernel.org X-Gm-Message-State: AOJu0YxU5hXexw8ZCyJEK+8KRIbhsWA0iVeLuDvWpwK0lsuyVsn8qEoB wxOmQrGyLfF2f70zZbdSlQ8VWdSKI2ajaFM3fnpPdWvCJWmRE/irAyu5 X-Gm-Gg: ASbGncuwQ2ELix5/Yw09cUVkCzfVGDz5daDCH0KiD/beaal4MpCWGxtF//O0hoK222d AQhOJIQL+qII8oMcWnONzjRkhX++n3K8V4LmNl5yFWEnMFJqcoFuM6lA9bspVdf0U14eQkMMGcA PHnaTnL6AyEQxWFTPKskrwWeQbCY6JmnD8tCSAKi8tEmetjN0QzclKCRsgHHKGj7IrIsoZ6ZpYJ GqAVVBX4su+obZBW/4Dv9kQnpRgb744GsOZHAYOrzAxxBBsfphwMtoydDvOWFicJ6HTpst/zoGw 3vbZQKUL3ptXJ6WvEaZ2ais4GAbpLJ/+8FfXWSgN0AGysrgleY2OxL7Mf5/PobGIt2C0ISzI78C iTxeFUbZ6lOqEVogFA+Ar0Q/y5PzsWhVdP+G6+NwFIBsGbbDHeSmDBRT7PR2cseBYXfUw50fgkp 5Mlnr+nyZ6t+SMbtokNe1q79U= X-Google-Smtp-Source: AGHT+IE8H0NdUg/c/thzQbZRKZeZtzM2gt17e0QQN2RbXKr1KgVctPF1CYPo5o3JzV+1jhxViXCLvA== X-Received: by 2002:a17:902:d501:b0:295:34ba:7b0b with SMTP id d9443c01a7336-2962ae0fba9mr72522505ad.35.1762383749590; Wed, 05 Nov 2025 15:02:29 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:29 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:15 -0800 Subject: [PATCH v4 3/9] rust: percpu: Add C bindings for per-CPU variable API Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-3-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=1564; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=RzgX0CxF7lpE2xBLdNMOZqdleGvbOwroWvnQQx0XUzQ=; b=BeCtd9F7suE7Dhq3+o6ebU7TlEj99UaBn9HqDmYOGvEIlw2BwbjQ9A9aTMEUtVsnUSdALbpNh qiE95luFKqoDYzljVg+qh0guvKUgr5gg+ZhgEeKyQjxdMCVh5yTgo9e X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add bindings necessary to implement a Rust per-CPU variable API, specifically per-CPU variable allocation and management of CPU preemption. Signed-off-by: Mitchell Levy Acked-by: Yury Norov (NVIDIA) --- rust/helpers/helpers.c | 2 ++ rust/helpers/percpu.c | 9 +++++++++ rust/helpers/preempt.c | 14 ++++++++++++++ 3 files changed, 25 insertions(+) diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c index 7cf7fe95e41d..2fc8d26cfe66 100644 --- a/rust/helpers/helpers.c +++ b/rust/helpers/helpers.c @@ -31,9 +31,11 @@ #include "of.c" #include "page.c" #include "pci.c" +#include "percpu.c" #include "pid_namespace.c" #include "platform.c" #include "poll.c" +#include "preempt.c" #include "property.c" #include "rbtree.c" #include "rcu.c" diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c new file mode 100644 index 000000000000..a091389f730f --- /dev/null +++ b/rust/helpers/percpu.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) +{ + return __alloc_percpu(sz, align); +} + diff --git a/rust/helpers/preempt.c b/rust/helpers/preempt.c new file mode 100644 index 000000000000..2c7529528ddd --- /dev/null +++ b/rust/helpers/preempt.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void rust_helper_preempt_disable(void) +{ + preempt_disable(); +} + +void rust_helper_preempt_enable(void) +{ + preempt_enable(); +} + --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E76172F39A9 for ; Wed, 5 Nov 2025 23:02:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383754; cv=none; b=f5Ux4rzsBMDfiHMg1ntY2+uHNfR9fEcI6QiREoyDcyb24ft6G4+xGCrwxeQWpQUkZ2jEpwOmazBEqrwpvPEmR+wNoZKwTon26ueWZIqK6owRlGLSP88eQWweNJGt0R6QTwSLCRcwjfQV2oJRsxojiETp/NC6z9LqC3GOSxJzsvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383754; c=relaxed/simple; bh=EZGO8S2GHg8ntWACzHd3jQtK+CxASeBX96N1M29X+1M=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hTlk8S3k51PopUnPOK3WkuY8PVXsGiWUeAba6DfWrX+PVz2S2Ytbnd4jHt5sRvsyo0T1059v9+I6GPXnJ/4/MRf0ivH/NaVUJ4sx357Omi8dF6KnvRAWvPBXg0Qc7xhOTf5b94yOPIKWHMlWGe2ddBF+p9pMufI0lvy8rvOoS3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RxUeG2Ec; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RxUeG2Ec" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-29633fdb2bcso3376675ad.3 for ; Wed, 05 Nov 2025 15:02:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383751; x=1762988551; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=TlxfvP2XGyBZ9ZWjp2nJWnVnTjycomxw0c/RHXKyKQo=; b=RxUeG2Ec73GJgoxDv1d9wyviqBFIv3j9rWkRJTlfugOJ2BLujwCYvEfWKvPZ1FeP2R +lD6bhJMXMvZes67JoKOp0htxZi/Xf+B2+GRPxLfsKhxGjMa0coG+1M56fL+gTHlA5Ex qyTTGg6eF+JIR5CTAne2mBx31wKIS3o17xwVDHXYv+lmtcy2Oz/VS6TrFeEOxdTM8unL R3RIHChbxcdWzGdDs0kZSAw3eak1OiXprCMH8lNSuaIEf+Es0TzMA7Rsf1RKAX+KRj8C Yvxb9kC5lsFL5UAQyhhETeuW3A9UhGPDPPfF96slqQ29LHUnl6jWC+XyJka72l3CZC/b IbRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383751; x=1762988551; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TlxfvP2XGyBZ9ZWjp2nJWnVnTjycomxw0c/RHXKyKQo=; b=hl3Fqr5uMo2GE4axkqLHPZgIRHz0PQPfhOERdKobkpzGsyPu2z81yxEIDfb7zh4GTU goE1i78fVpW0xZjDxojRTBD5uR1ygLmhMzdp6M+cfmIKOSYclAHpYekT7Quz95YWuBIQ zXBaw+rmyW7h7IUcUWKT38xdNHe1KO/mNRvn85xuzV9XLPQiXSO4FzMvPH83eKpKEx20 U/Xd6Djh59RmG0VU4y3mJaWNIYySA/VAFTpVIb0MvDYO9+CKu3V6Op9MuNL3wNTXYtms kZ0R3cuP5Xz4Ysy5GdMTvGEmdRYJGUUO8yOG9xYiZZcBjdUFBtO06xo7G32mcf8FNkRF tr2w== X-Forwarded-Encrypted: i=1; AJvYcCWuRHaW8evPXM4VKMjo34ZsI6fDiNE2cR+xecN4RJ3dK2kn8G061/Mi4aBplT6N7UGmWCuu1RSv9yAg5hE=@vger.kernel.org X-Gm-Message-State: AOJu0YzyptJCBjOb517agv0YflzbsFr4BZN2A/yDHaN8IVseKOqeHCWy lb9/bQHN+py/CD13gamAHGaXaVOEBtnPhiQIdG9byt5YZo+6ZFwQyZ3v X-Gm-Gg: ASbGncsNkFM18wMGF7216iVBFyqDAnmZyuf902kjOs5v53Fvu350lCMDyo4RoBCWDHN OmJ11NT2F7EX3docqHK3yu/qfDvaJHT5Ea7rrUZZ/2mCE9Dy0tON42rS1dfxM6sul8moh2XX/yZ q0+1eAM1FLSdSeaSUwfvZsvdrwneYn620k1odFhzl25g4K9/Qq48y6lp7uhfAykNcmqPPS3sGB6 yZStYhRBGNa79zWQ44mmVuON6+Wa/RNp260uMHFWsjS00q0ABLfPsRqDEmDG6KrjPW5368Ev0Vq gJB89xvvjbgXPlvCGrR2PosLunpX441Vwy6Lgav7LCjiOMGvMmWl2zqnglcN4oEqhlc4IZJ1aad IFbqj7ZcUBjvJqBRkiA75C494FHrvz9JZJ1ohg+8YqU3PiEXz7QKqAfITi0ZBAi1i/XF3/VWh7e uRldUuTHcWzDuXNbpI1j4oZTJT2twN1rYywA== X-Google-Smtp-Source: AGHT+IEtcWPd7dB7RyrIUrheYv6CSgAxUf5dQTAqNss54XpxzEwcb9AjvOpQDtFIWbEWDoKZJmhUyw== X-Received: by 2002:a17:903:1aa7:b0:295:98a1:7ddb with SMTP id d9443c01a7336-2962ade3309mr64506435ad.61.1762383750699; Wed, 05 Nov 2025 15:02:30 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:30 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:16 -0800 Subject: [PATCH v4 4/9] rust: percpu: introduce a rust API for static per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-4-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=25644; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=EZGO8S2GHg8ntWACzHd3jQtK+CxASeBX96N1M29X+1M=; b=VY87PCbOUQoIFarob7CleTCefLCX58T7Xi/O86w4FVkbK2dRHCYn55PyyLKrH1P8O+5VJk7sb KrXKjVUVQrzDb3qkJZBuLWCNge7F1wC5APDoK+DQXjg7loYU4SbbEUS X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Per-CPU variables are an important tool for reducing lock contention, especially in systems with many processors. They also provide a convenient way to handle data that are logically associated with a particular CPU (e.g., the currently running task). Therefore, add a Rust API to make use of statically-allocated per-CPU variables. Add a RAII `CpuGuard` type for disabling CPU preemption. Introduce unifying abstractions that can be reused for a Rust API for dynamically-allocated per-CPU variables. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- Using `rustc` 1.83.0, I can confirm that the symbols `__INIT_$id` are optimized out by the compiler and do not appear in the final `.ko` file when compiling `samples/rust/rust_percpu.rs` (introduced in a later patch in this series). --- rust/kernel/lib.rs | 3 + rust/kernel/percpu.rs | 250 ++++++++++++++++++++++++++++++++++++= ++++ rust/kernel/percpu/cpu_guard.rs | 36 ++++++ rust/kernel/percpu/static_.rs | 218 +++++++++++++++++++++++++++++++++++ 4 files changed, 507 insertions(+) diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs index f910a5ab80ba..2fdb194aa068 100644 --- a/rust/kernel/lib.rs +++ b/rust/kernel/lib.rs @@ -108,6 +108,9 @@ pub mod page; #[cfg(CONFIG_PCI)] pub mod pci; +// Only x86_64 is supported by percpu for now +#[cfg(CONFIG_X86_64)] +pub mod percpu; pub mod pid_namespace; pub mod platform; pub mod prelude; diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs new file mode 100644 index 000000000000..2fba9a165636 --- /dev/null +++ b/rust/kernel/percpu.rs @@ -0,0 +1,250 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Per-CPU variables. +//! +//! See the [`crate::define_per_cpu!`] macro and the [`PerCpu`] trait. + +pub mod cpu_guard; +mod static_; + +#[doc(inline)] +pub use static_::*; + +use crate::declare_extern_per_cpu; +use crate::percpu::cpu_guard::CpuGuard; +use crate::types::Opaque; + +use core::arch::asm; +use core::cell::{Cell, RefCell, UnsafeCell}; +use core::mem::MaybeUninit; + +use ffi::c_void; + +/// A per-CPU pointer; that is, an offset into the per-CPU area. +/// +/// Note that this type is NOT a smart pointer, it does not manage the all= ocation. +pub struct PerCpuPtr(*mut MaybeUninit); + +/// Represents exclusive access to the memory location pointed at by a par= ticular [`PerCpu`]. +pub struct PerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +/// Represents access to the memory location pointed at by a particular [`= PerCpu`] where the +/// type `T` manages access to the underlying memory to avoid aliaising tr= oubles. +/// +/// For example, `T` might be a [`Cell`] or [`RefCell`]. +pub struct CheckedPerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +impl PerCpuPtr { + /// Makes a new [`PerCpuPtr`] from a raw per-CPU pointer. + /// + /// Note that the returned [`PerCpuPtr`] is valid only as long as the = given pointer is. This + /// also requires that the allocation pointed to by `ptr` must be corr= ectly sized and aligned + /// to hold a `T`. + pub fn new(ptr: *mut MaybeUninit) -> Self { + Self(ptr) + } + + /// Get a [`&mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the current CPU + /// represented by `&self` + /// + /// # Safety + /// + /// The returned `&mut T` must follow Rust's aliasing rules. That is, = no other `&(mut) T` may + /// exist that points to the same location in memory. In practice, thi= s means that + /// [`Self::get_ref`] and [`Self::get_mut_ref`] must not be called on = another [`PerCpuPtr`] + /// that is a copy/clone of `&self` for as long as the returned refere= nce lives. If the per-CPU + /// variable represented by `&self` is available to C, the caller of t= his function must ensure + /// that the C code does not modify the variable for as long as this r= eference lives. + /// + /// `self` must point to a live allocation correctly sized and aligned= to hold a `T`. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + #[allow(clippy::mut_from_ref)] // Safety requirements prevent aliasing= issues + pub unsafe fn get_mut_ref(&self) -> &mut MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract, + // and the safety requirements of this function ensure that the re= turned reference is + // exclusive. + unsafe { &mut *(self.get_ptr()) } + } + + /// Get a [`&mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the current CPU + /// represented by `&self` + /// + /// # Safety + /// + /// The returned `&T` must follow Rust's aliasing rules. That is, no `= &mut T` may exist that + /// points to the same location in memory. In practice, this means tha= t [`Self::get_mut_ref`] + /// must not be called on another [`PerCpuPtr`] that is a copy/clon= e of `&self` for as long + /// as the returned reference lives. If the per-CPU variable represent= ed by `&self` is + /// available to C, the caller of this function must ensure that the C= code does not modify the + /// variable for as long as this reference lives. + /// + /// `self` must point to a live allocation correctly sized and aligned= to hold a `T`. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + pub unsafe fn get_ref(&self) -> &MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract. + // The safety requirements of this function ensure that the return= ed reference isn't + // aliased by a `&mut MaybeUninit`. + unsafe { &*self.get_ptr() } + } + + /// Get a [`*mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the current CPU + /// represented by `&self`. Note that if CPU preemption is not disable= d before calling this + /// function, use of the returned pointer may cause a data race withou= t some other + /// synchronization mechanism. Buyer beware! + pub fn get_ptr(&self) -> *mut MaybeUninit { + let this_cpu_off_pcpu =3D ExternStaticPerCpuSymbol::ptr(&raw const= this_cpu_off); + let mut this_cpu_area: *mut c_void; + // SAFETY: gs + this_cpu_off_pcpu is guaranteed to be a valid poin= ter because `gs` points + // to the per-CPU area and this_cpu_off_pcpu is a valid per-CPU al= location. + unsafe { + asm!( + "mov {out}, gs:[{off_val}]", + off_val =3D in(reg) this_cpu_off_pcpu.0, + out =3D out(reg) this_cpu_area, + ) + }; + + // This_cpu_area + self.0 is guaranteed to be a valid pointer by t= he per-CPU subsystem and + // the invariant that self.0 is a valid offset into the per-CPU ar= ea. + (this_cpu_area).wrapping_add(self.0 as usize).cast() + } +} + +// SAFETY: Sending a [`PerCpuPtr`] to another thread is safe because as= soon as it's sent, the +// pointer is logically referring to a different place in memory in the ot= her CPU's per-CPU area. +// In particular, this means that there are no restrictions on the type `T= `. +unsafe impl Send for PerCpuPtr {} + +// SAFETY: Two threads concurrently making use of a [`PerCpuPtr`] will = each see the `T` in their +// own per-CPU area, so there's no potential for a data race (regardless o= f whether `T` is itself +// `Sync`). +unsafe impl Sync for PerCpuPtr {} + +impl Clone for PerCpuPtr { + fn clone(&self) -> Self { + *self + } +} + +/// [`PerCpuPtr`] is just a wrapper around a pointer. +impl Copy for PerCpuPtr {} + +/// A trait representing a per-CPU variable. +/// +/// This is implemented for [`StaticPerCpu`]. The main usage of this tr= ait is to call +/// [`Self::get_mut`] to get a [`PerCpuToken`] that can be used to access = the underlying per-CPU +/// variable. +/// +/// See [`PerCpuToken::with`]. +pub trait PerCpu { + /// Produces a token, asserting that the holder has exclusive access t= o the underlying memory + /// pointed to by `self` + /// + /// # Safety + /// + /// No other [`PerCpuToken`] or [`CheckedPerCpuToken`] may exist on th= e current CPU (which is a + /// sensible notion, since we keep a [`CpuGuard`] around) that is deri= ved from the same + /// [`PerCpu`] or a clone thereof. + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T>; +} + +/// A marker trait for types that are interior mutable. Types that impleme= nt this trait can be used +/// to create "checked" per-CPU variables. See [`CheckedPerCpu`]. +pub trait InteriorMutable {} + +impl InteriorMutable for Cell {} +impl InteriorMutable for RefCell {} +impl InteriorMutable for UnsafeCell {} +impl InteriorMutable for Opaque {} + +/// A trait representing a per-CPU variable that is usable via a `&T`. +/// +/// The unsafety of [`PerCpu`] stems from the fact that the holder of a= [`PerCpuToken`] can use +/// it to get a `&mut T` to the underlying per-CPU variable. This is probl= ematic because the +/// existence of aliaising `&mut T` is undefined behavior in Rust. This ty= pe avoids that issue by +/// only allowing access via a `&T`, with the tradeoff that then `T` must = be interior mutable or +/// the underlying per-CPU variable must be a constant for the lifetime of= any corresponding +/// [`CheckedPerCpuToken`]. +/// +/// Currently, only the case where `T` is interior mutable has first-class= support, though a custom +/// implementation of [`PerCpu`]/[`CheckedPerCpu`] could be created = for the const case. +pub trait CheckedPerCpu: PerCpu { + /// Produces a token via which the holder can access the underlying pe= r-CPU variable. + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T>; +} + +impl<'a, T> PerCpuToken<'a, T> { + /// Asserts that the holder has exclusive access to the initialized un= derlying memory pointed + /// to by `ptr` on the current CPU, producing a token. + /// + /// # Safety + /// + /// No other [`PerCpuToken`] or [`CheckedPerCpuToken`] may exist on th= e current CPU (which is a + /// sensible notion, since we keep a `CpuGuard` around) that uses the = same [`PerCpuPtr`]. + /// + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed; `ptr` must be valid + /// (that is, pointing at a live per-CPU allocation correctly sized an= d aligned to hold a `T`) + /// for `'a`. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> PerCpuTok= en<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&mut T` that points at the unde= rlying per-CPU variable + /// that `&mut self` represents. + pub fn with(&mut self, func: U) + where + U: FnOnce(&mut T), + { + // SAFETY: The existence of a PerCpuToken means that the requireme= nts for get_mut_ref are + // satisfied. Likewise, the requirements for assume_init_mut are s= atisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_mut_ref().assume_init_mut() }); + } +} + +impl<'a, T> CheckedPerCpuToken<'a, T> { + /// Asserts that the memory pointed to by `ptr` is initialized on the = current CPU, producing a + /// token. + /// + /// # Safety + /// + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed; `ptr` must be valid + /// (that is, pointing at a live per-CPU allocation correctly sized an= d aligned to hold a `T`) + /// for `'a`. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> CheckedPe= rCpuToken<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&T` that points at the underlyi= ng per-CPU variable that + /// `&mut self` represents. + pub fn with(&mut self, func: U) + where + U: FnOnce(&T), + { + // SAFETY: The existence of a CheckedPerCpuToken means that the re= quirements for get_ref + // are satisfied. Likewise, the requirements for assume_init_ref a= re satisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_ref().assume_init_ref() }); + } +} + +declare_extern_per_cpu!(this_cpu_off: *mut c_void); diff --git a/rust/kernel/percpu/cpu_guard.rs b/rust/kernel/percpu/cpu_guard= .rs new file mode 100644 index 000000000000..2fb4c9218922 --- /dev/null +++ b/rust/kernel/percpu/cpu_guard.rs @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Contains abstractions for disabling CPU preemption. See [`CpuGuard`]. + +/// A RAII guard for `bindings::preempt_disable` and `bindings::preempt_en= able`. +/// +/// Guarantees preemption is disabled for as long as this object exists. +pub struct CpuGuard { + // Don't make one without using new() + _phantom: (), +} + +impl CpuGuard { + /// Create a new [`CpuGuard`]. Disables preemption for its lifetime. + pub fn new() -> Self { + // SAFETY: There are no preconditions required to call preempt_dis= able + unsafe { + bindings::preempt_disable(); + } + CpuGuard { _phantom: () } + } +} + +impl Default for CpuGuard { + fn default() -> Self { + Self::new() + } +} + +impl Drop for CpuGuard { + fn drop(&mut self) { + // SAFETY: There are no preconditions required to call preempt_ena= ble + unsafe { + bindings::preempt_enable(); + } + } +} diff --git a/rust/kernel/percpu/static_.rs b/rust/kernel/percpu/static_.rs new file mode 100644 index 000000000000..418fc2fa06f2 --- /dev/null +++ b/rust/kernel/percpu/static_.rs @@ -0,0 +1,218 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Statically allocated per-CPU variables. + +use super::*; + +/// A wrapper used for declaring static per-CPU variables. +/// +/// These symbols are "virtual" in that the linker uses them to generate o= ffsets into each CPU's +/// per-CPU area, but shouldn't be read from/written to directly. The fact= that the statics are +/// immutable prevents them being written to (generally), this struct havi= ng _val be non-public +/// prevents reading from them. +/// +/// The end-user of the per-CPU API should make use of the [`crate::define= _per_cpu!`] macro instead +/// of declaring variables of this type directly. All instances of this ty= pe must be `static` and +/// `#[link_section =3D ".data..percpu"]` (which the macro handles). +#[repr(transparent)] +pub struct StaticPerCpuSymbol { + _val: T, // generate a correctly sized type +} + +/// A wrapper for per-CPU variables declared in C. +/// +/// As with [`StaticPerCpuSymbol`], this type should not be used directly.= Instead, use the +/// [`crate::declare_extern_per_cpu!`] macro. +#[repr(transparent)] +pub struct ExternStaticPerCpuSymbol(StaticPerCpuSymbol); + +/// Holds a statically-allocated per-CPU variable. +#[derive(Clone)] +pub struct StaticPerCpu(pub(super) PerCpuPtr); + +impl StaticPerCpuSymbol { + /// Removes the per-CPU marker type from a static per-CPU symbol. + /// + /// To declare a static per-CPU variable, Rust requires that the varia= ble be `Sync`. However, + /// in the case of per-CPU variables, this is silly. See [`PerCpuSyncM= arkerType`]. Thus, our + /// statics are actually of type `StaticPerCpuSymbol>`, and this + /// produces a `StaticPerCpuSymbol`. + pub fn forward(ptr: *const Self) -> *const StaticPerCpuSymbol { + ptr.cast() + } +} + +impl ExternStaticPerCpuSymbol { + /// Gets a [`PerCpuPtr`] to the per-CPU variable. + /// + /// Usage of this [`PerCpuPtr`] must be very careful to keep in mind w= hat's happening in C. In + /// particular, the C code cannot modify the variable while any refere= nce derived from the + /// returned pointer is live. + pub fn ptr(ptr: *const Self) -> PerCpuPtr { + // These casts are OK because, ExternStaticPerCpuSymbol, StaticPer= CpuSymbol, and + // MaybeUninit are transparent, everything is just a `T` from a me= mory layout perspective. + // Casting to `mut` is OK because any usage of the returned pointe= r must satisfy the + // soundness requirements for using it as such. + PerCpuPtr::new(ptr.cast::>().cast_mut().cast= ()) + } +} + +impl StaticPerCpu { + /// Creates a [`StaticPerCpu`] from a [`StaticPerCpuSymbol`]. + /// + /// Users should probably declare static per-CPU variables with [`crat= e::define_per_cpu!`] and + /// then get instances of [`StaticPerCpu`] using [`crate::get_static_p= er_cpu!`]. + /// + /// # Safety + /// You should probably be using [`crate::get_static_per_cpu!`] instea= d. + /// + /// `ptr` must be a valid offset into the per-CPU area, sized and alig= ned for access to a `T`; + /// typically this is the address of a static in the `.data..percpu` s= ection, which is managed + /// by the per-CPU subsystem. + pub unsafe fn new(ptr: *const StaticPerCpuSymbol) -> StaticPerCpu { + let pcpu_ptr =3D PerCpuPtr::new(ptr.cast_mut().cast()); + Self(pcpu_ptr) + } +} + +impl PerCpu for StaticPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: + // 1. By the requirements of `PerCpu::get_mut`, no other `[Checked= ]PerCpuToken` exists on + // the current CPU. + // 2. The per-CPU subsystem guarantees that each CPU's instance of= a statically allocated + // variable begins with a copy of the contents of the correspon= ding symbol in + // `.data..percpu` and is therefore initialized. + // 3. The per-CPU subsystem guarantees that `self.0` is correctly = aligned for a `T`. + // 4. The per-CPU subsystem guarantees that `self.0` is lives fore= ver as a per-CPU + // allocation, and that this allocation is the proper size for = a `T`. + unsafe { PerCpuToken::new(guard, &self.0) } + } +} + +impl CheckedPerCpu for StaticPerCpu { + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: + // 1. The per-CPU subsystem guarantees that each CPU's instance of= a statically allocated + // variable begins with a copy of the contents of the correspon= ding symbol in + // `.data..percpu` and is therefore initialized. + // 2. The per-CPU subsystem guarantees that `self.0` is correctly = aligned for a `T`. + // 3. The per-CPU subsystem guarantees that `self.0` is lives fore= ver as a per-CPU + // allocation, and that this allocation is the proper size for = a `T`. + unsafe { CheckedPerCpuToken::new(guard, &self.0) } + } +} + +/// Gets a [`StaticPerCpu`] from a symbol declared with [`crate::define= _per_cpu!`]. +/// +/// # Arguments +/// +/// * `ident` - The identifier declared +#[macro_export] +macro_rules! get_static_per_cpu { + ($id:ident) =3D> { + unsafe { + // SAFETY: The signature of `StaticPerCpuSymbol::forward` guar= antees that `&raw const + // $id` is a `*const StaticPerCpuSymbol>` if the macro + // invocation compiles. + // + // Values of type `StaticPerCpuSymbol` must be created via = `define_per_cpu`, and so + // the per-CPU subsystem guarantees that the requirements for = `StaticPerCpu::new` are + // satisfied. + $crate::percpu::StaticPerCpu::new($crate::percpu::StaticPerCpu= Symbol::forward( + &raw const $id, + )) + } + }; +} + +/// Declares an [`ExternStaticPerCpuSymbol`] corresponding to a per-CPU va= riable defined in C. +#[macro_export] +macro_rules! declare_extern_per_cpu { + ($id:ident: $ty:ty) =3D> { + extern "C" { + static $id: ExternStaticPerCpuSymbol<$ty>; + } + }; +} + +/// A trait implemented by static per-CPU wrapper types. +/// +/// Internally, static per-CPU variables are declared as `static` variable= s. However, Rust doesn't +/// allow you to declare statics of a `!Sync` type. This trait is implemen= ted by the marker type +/// that is declared as `Sync` and used to declare the static. See [`crate= ::define_per_cpu!`] for +/// the gory details. +/// +/// # Safety +/// +/// Implementations must be `#[repr(transparent)]` wrappers around an `Inn= er`. This trait should +/// only be used from the [`crate::define_per_cpu!`] macro. +pub unsafe trait PerCpuSyncMarkerType { + /// The "true" type of the per-CPU variable. It must always be valid t= o cast a `*const Self` to + /// a `*const Self::Inner`, which is guarnateed by this trait's safety= requirement. + type Inner; +} + +/// Declares and initializes a static per-CPU variable. +/// +/// Analogous to the C `DEFINE_PER_CPU` macro. +/// +/// See also [`crate::get_static_per_cpu!`] for how to get a [`StaticPerCp= u`] from this +/// declaration. +/// +/// # Example +/// ``` +/// use kernel::define_per_cpu; +/// use kernel::percpu::StaticPerCpuSymbol; +/// +/// define_per_cpu!(pub MY_PERCPU: u64 =3D 0); +/// ``` +#[macro_export] +macro_rules! define_per_cpu { + ($vis:vis $id:ident: $ty:ty =3D $expr:expr) =3D> { + $crate::macros::paste! { + // We might want to have a per-CPU variable that doesn't imple= ment `Sync` (not paying + // sync overhead costs is part of the point), but Rust won't l= et us declare a static of + // a `!Sync` type. Of course, we don't actually have any synch= ronization issues, since + // each CPU will see its own copy of the variable, so we cheat= a little bit and tell + // Rust it's fine. + #[doc(hidden)] + #[allow(non_camel_case_types)] + #[repr(transparent)] // It needs to be the same size as $ty + struct [<__PRIVATE_TYPE_ $id>]($ty); + + // SAFETY: [<__PRIVATE_TYPE_ $id>] is a `#[repr(transparent)]`= wrapper around a `$ty`. + unsafe impl PerCpuSyncMarkerType for [<__PRIVATE_TYPE_ $id>] { + type Inner =3D $ty; + } + + impl [<__PRIVATE_TYPE_ $id>] { + #[doc(hidden)] + const fn new(val: $ty) -> Self { + Self(val) + } + } + + // Expand $expr outside of the unsafe block to avoid silently = allowing unsafe code to be + // used without a user-facing unsafe block + #[doc(hidden)] + static [<__INIT_ $id>]: [<__PRIVATE_TYPE_ $id>] =3D [<__PRIVAT= E_TYPE_ $id>]::new($expr); + + // SAFETY: This type will ONLY ever be used to declare a `Stat= icPerCpuSymbol` + // (which we then only ever use as input to `&raw`). Reading f= rom the symbol is + // already UB, so we won't ever actually have any variables of= this type where + // synchronization is a concern. + #[doc(hidden)] + unsafe impl Sync for [<__PRIVATE_TYPE_ $id>] {} + + // SAFETY: StaticPerCpuSymbol is #[repr(transparent)], so w= e can freely convert from + // [<__PRIVATE_TYPE_ $id>], which is also `#[repr(transparent)= ]` (i.e., everything is + // just a `$ty` from a memory layout perspective). + #[link_section =3D ".data..percpu"] + $vis static $id: StaticPerCpuSymbol<[<__PRIVATE_TYPE_ $id>]> = =3D unsafe { + core::mem::transmute_copy::< + [<__PRIVATE_TYPE_ $id>], StaticPerCpuSymbol<[<__PRIVAT= E_TYPE_ $id>]> + >(&[<__INIT_ $id>]) + }; + } + }; +} --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20F4B2F5316 for ; Wed, 5 Nov 2025 23:02:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383754; cv=none; b=MYtuONDszKnVRTPJEP9b2Nu5qtsOrebO8KZIwd80aQFA8CRRhIfkewcK+j8GzhkFdrCd8du62zI4+THpZi0Dl/cYnjUP5oyL2zqq+YBulHhhKq27wMS34FpKZhsYjhIFVgInEiapbk0sIy53dufhY7gihQmT7xn3GktkhMFW36U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383754; c=relaxed/simple; bh=mFg9QLMAwyqIxa/N6WNggn20WoF/wqWYILNGfUHwvYw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=oYjqpYYASPTjQtsOzC1NNSmAd9OYaNt2A3XAJTOZC+DKtMgoDKcY0TmC2bzFpU4s74lrQ3Yhw0eJBRTz5A2cMrDw9HcgAzdd+G/BFosuI0aHeDwWno6IJlXub7/Vo3cb/1K1rhbfg6MnGYOhFvVd6TF2wyhYd9Z70SaGCGVnNV4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mTk6P1vu; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mTk6P1vu" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-33292adb180so365995a91.3 for ; Wed, 05 Nov 2025 15:02:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383752; x=1762988552; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=WFJ1wrPBcxrrD+LBS55P13yiT8NfiLst30CpnCOiPfc=; b=mTk6P1vuXWUriJvC0nFh8CdhlAZVDPPqTdYbH2LlHcacney1GY1wo8Xd8jRNRXs/Ax HXgnpc1hlIhzQaKLc0ZA9xv58WrtVAuQ3dc6QKO/BgZyDjGffNyj3pJasyOjImxNhWvn zNCqoGOxAh7nXH9AWsiJR1TCXN5qUJ3pbyc22BeK1b9HqkIij/jBwdkehDpQPTfks3bM REjsHLyddO0H98deZXbCAS6ownS4w1ZpE+5B+G9HucbfGVEjAzaanmLN5otPR+Y/DX5L BpGVmfPo44NGSmx7MBip1U70gR0A+8nUkmSNcjpHZeMYG/kAWMiXjm5c4WtT7viLdHnd MsJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383752; x=1762988552; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WFJ1wrPBcxrrD+LBS55P13yiT8NfiLst30CpnCOiPfc=; b=p/upGh/75P4fNZzSVJ3QB4EhFsX5qKugtBzUuhG1Eb5jNPZa+HsSBipgNE3nJmyX3v fbw+vmEiV/LQLOfdBOaBI4sVH+PaHiEWo1duUE61KOxbYxuSJF0e6yCz9x/vMa/mkUyS MwXBok8fHABYhkdoBQO3vifQEZ6xJ6aDRz5Qb3ZihZHvXuX6n49wUk8f0t0O7dQsDvhf o2FqPqdt+5p7vlRND3AWS4ABl00IP0PzR7owAmDumO75XL5V8Klc2zlRdgnn0nPWigXN 0wUxLnYkslCjS/azaHX/ZQGkxkGMmtMNBk3jRjbeRwfbJJsnoNfmZ91+NYBcXnw7Geeg 8wfg== X-Forwarded-Encrypted: i=1; AJvYcCVfPk8fpniuXIRfpBqF328pQIsjCeocZ2sd4uesJ4oojYYxnQPpTMFUMVC0x5O4YJEOdU9QY28rGA2gEZA=@vger.kernel.org X-Gm-Message-State: AOJu0Ywi3wciCJsgJbVyucgxtQjlozx8QISD+TT+Iy7SFhZgePsSKGtQ C1ealVBCREJViEwdR3wP+Gq6F0WzUAbr91w24ruJLjz94m5sq9sXiEON X-Gm-Gg: ASbGncurqcyB8iBmd4LzEf28yihHhiVyy9OCF3jTc7b/9DT0mV0qVhlhO+e0q6f/Hl+ 1V7SXTzA8M+dUhcEqJvqANww4Lwkk7FIHSyo2BZmTBNpT0TO3qpRMqWJQ2cDqBvj6S7zKPkbGvC klQVbDmwyRp4j4T2P5RqkwLmSU9cxTUT33CwBni4sbkozj/E0e/RKLreaxLvOs8bZF36F+yc5h0 eeeu2MO9zPlWd5GTkPYB6kWertl8LY6MJF1dYEovKbA8fZI6UYsP3mNC2/bIHJL/3tdAme/VSd/ ubb6EJlrgmdDaPk8PBxS2hBVoge/kLZKzzX12/Vj96Q3eIeB9PJl9tR29kVyUnSzl2lfLhITZQ+ J2WOTiE33iw9gZJ2g8Q/WPNb4JJOBXoKrYR+u5NRs6nfIky3NVZJ0JgWfcXFdDleQSCcXXbJLoU SQHc+W59qtw+dPyEF6KP7OjCl111JvVdRD4g== X-Google-Smtp-Source: AGHT+IE1c9+ZrzqLcSGm3uQ2ejRBT2CJMVQlT9ke7gOJ6htC5411VVsiXKPhB80c9d8mxSq7gHQUCw== X-Received: by 2002:a17:903:1b65:b0:294:e8a0:382b with SMTP id d9443c01a7336-2962ae57374mr72286265ad.54.1762383752237; Wed, 05 Nov 2025 15:02:32 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:31 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:17 -0800 Subject: [PATCH v4 5/9] rust: percpu: introduce a rust API for dynamic per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-5-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=9789; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=mFg9QLMAwyqIxa/N6WNggn20WoF/wqWYILNGfUHwvYw=; b=rC6LvrXQz84xbSb5BvGzEQo3PK2NO2XHoHUt+ISh2JaHvuW2RktvXy7YCgzUy8bvkJ1wqFQWO bxVWrfmsXOSBxGpV0RbAHoOa2jK6XivWxBpY5qWUYmJnPND48kmTwbK X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Dynamically allocated per-CPU variables are core to many of the use-cases of per-CPU variables (e.g., ref counting). Add support for them using the core `PerCpuPtr` primitive, implementing the `PerCpu` trait. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 10 ++++ rust/kernel/percpu.rs | 30 ++++++++-- rust/kernel/percpu/dynamic.rs | 130 ++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 166 insertions(+), 4 deletions(-) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index a091389f730f..35656333dfae 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -7,3 +7,13 @@ void __percpu *rust_helper_alloc_percpu(size_t sz, size_t = align) return __alloc_percpu(sz, align); } =20 +void *rust_helper_per_cpu_ptr(void __percpu *ptr, unsigned int cpu) +{ + return per_cpu_ptr(ptr, cpu); +} + +void rust_helper_on_each_cpu(smp_call_func_t func, void *info, int wait) +{ + on_each_cpu(func, info, wait); +} + diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 2fba9a165636..294b8ffc4f62 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -1,14 +1,19 @@ // SPDX-License-Identifier: GPL-2.0 //! Per-CPU variables. //! -//! See the [`crate::define_per_cpu!`] macro and the [`PerCpu`] trait. +//! See the [`crate::define_per_cpu!`] macro, the [`DynamicPerCpu`] type, = and the [`PerCpu`] +//! trait. =20 pub mod cpu_guard; +mod dynamic; mod static_; =20 +#[doc(inline)] +pub use dynamic::*; #[doc(inline)] pub use static_::*; =20 +use crate::cpu::CpuId; use crate::declare_extern_per_cpu; use crate::percpu::cpu_guard::CpuGuard; use crate::types::Opaque; @@ -123,6 +128,23 @@ pub fn get_ptr(&self) -> *mut MaybeUninit { // the invariant that self.0 is a valid offset into the per-CPU ar= ea. (this_cpu_area).wrapping_add(self.0 as usize).cast() } + + /// Get a [`*mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the CPU represented + /// by `cpu`. Note that without some kind of synchronization, use of t= he returned pointer may + /// cause a data race. It is the caller's responsibility to use the re= turned pointer in a + /// reasonable way. + /// + /// # Returns + /// - The returned pointer is valid only if `self` is (that is, it poi= nts to a live allocation + /// correctly sized and aligned to hold a `T`) + /// - The returned pointer is valid only if the bit corresponding to `= cpu` is set in + /// [`kernel::cpumask::Cpumask::possible_cpus()`]. + pub fn get_remote_ptr(&self, cpu: CpuId) -> *mut MaybeUninit { + // SAFETY: `bindings::per_cpu_ptr` is just doing pointer arithmeti= c. The returned pointer + // may not be valid (under the conditions specified in this functi= on's documentation), but + // the act of producing the pointer is safe. + unsafe { bindings::per_cpu_ptr(self.0.cast(), cpu.as_u32()) }.cast= () + } } =20 // SAFETY: Sending a [`PerCpuPtr`] to another thread is safe because as= soon as it's sent, the @@ -146,9 +168,9 @@ impl Copy for PerCpuPtr {} =20 /// A trait representing a per-CPU variable. /// -/// This is implemented for [`StaticPerCpu`]. The main usage of this tr= ait is to call -/// [`Self::get_mut`] to get a [`PerCpuToken`] that can be used to access = the underlying per-CPU -/// variable. +/// This is implemented for both [`StaticPerCpu`] and [`DynamicPerCpu`]. The main usage of +/// this trait is to call [`Self::get_mut`] to get a [`PerCpuToken`] that = can be used to access the +/// underlying per-CPU variable. /// /// See [`PerCpuToken::with`]. pub trait PerCpu { diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs new file mode 100644 index 000000000000..1863f31a2817 --- /dev/null +++ b/rust/kernel/percpu/dynamic.rs @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Dynamically allocated per-CPU variables. + +use super::*; + +use crate::alloc::Flags; +use crate::bindings::{alloc_percpu, free_percpu}; +use crate::cpumask::Cpumask; +use crate::prelude::*; +use crate::sync::Arc; +use core::mem::{align_of, size_of, MaybeUninit}; + +/// Represents a dynamic allocation of a per-CPU variable via `alloc_percp= u`. Calls `free_percpu` +/// when dropped. +/// +/// # Contents +/// Note that the allocated memory need not be initialized, and this type = does not track when/if +/// the memory location on any particular CPU has been initialized. This m= eans that it cannot tell +/// whether it should drop the *contents* of the allocation when it is dro= pped. It is up to the +/// user to do this via something like [`core::ptr::drop_in_place`]. +pub struct PerCpuAllocation(PerCpuPtr); + +impl PerCpuAllocation { + /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, + /// initially filled with the zero value for `T`. + /// + /// Returns [`None`] under the same circumstances the C function `allo= c_percpu` returns `NULL`. + pub fn new_zero() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call `alloc_percpu`; `MaybeUnin= it` is + // `#[repr(transparent)]`, so we can cast a `*mut T` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + // alloc_percpu returns zero'ed memory + Some(Self(PerCpuPtr::new(ptr))) + } +} + +impl PerCpuAllocation { + /// Makes a per-CPU allocation sized and aligned to hold a `T`. + /// + /// Returns [`None`] under the same circumstances the C function `allo= c_percpu` returns `NULL`. + pub fn new_uninit() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call `alloc_percpu`; `MaybeUnin= it` is + // `#[repr(transparent)]`, so we can cast a `*mut T` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + Some(Self(PerCpuPtr::new(ptr))) + } +} + +impl Drop for PerCpuAllocation { + fn drop(&mut self) { + // SAFETY: self.0.0 was returned by alloc_percpu, and so was a val= id pointer into + // the percpu area, and has remained valid by the invariants of Pe= rCpuAllocation. + unsafe { free_percpu(self.0 .0.cast()) } + } +} + +/// Holds a dynamically-allocated per-CPU variable. +#[derive(Clone)] +pub struct DynamicPerCpu { + // INVARIANT: `alloc` is `Some` unless this object is in the process o= f being dropped. + // INVARIANT: The allocation held by `alloc` is sized and aligned for = a `T`. + // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by the alloc is + // initialized. + alloc: Option>>, +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `flags` - [`Flags`] used to allocate an [`Arc`] that keeps track= of the underlying + /// [`PerCpuAllocation`]. + pub fn new_zero(flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: Some(arc) }) + } +} + +impl PerCpu for DynamicPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: + // 1. Invariants of this type assure that `alloc` is `Some`. + // 2. The requirements of `PerCpu::get_mut` ensure that no other `= [Checked]PerCpuToken` + // exists on the current CPU. + // 3. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are + // initialized on each CPU. + // 4. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is + // live. + // 5. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for + // a `T`. + unsafe { PerCpuToken::new(guard, &self.alloc.as_ref().unwrap_unche= cked().0) } + } +} + +impl Drop for DynamicPerCpu { + fn drop(&mut self) { + // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. + let alloc =3D unsafe { self.alloc.take().unwrap_unchecked() }; + if let Some(unique_alloc) =3D alloc.into_unique_or_drop() { + let ptr =3D unique_alloc.0; + for cpu in Cpumask::possible_cpus().iter() { + let remote_ptr =3D ptr.get_remote_ptr(cpu); + // SAFETY: `remote_ptr` is valid because the allocation it= points to is still live, + // `cpu` appears in `Cpumask::possible_cpus()`, and the or= iginal allocation was + // sized and aligned for a `T`. + // + // This type's invariant ensures that the memory location = in each CPU's per-CPU + // area pointed at by `alloc.0` has been initialized. We h= ave a `UniqueArc`, so we + // know we're the only ones with a reference to the memory= . These two facts + // together satisfy the requirements for `assume_init_drop= `. + unsafe { + (*remote_ptr).assume_init_drop(); + } + } + } + } +} --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C35EC2F5A13 for ; Wed, 5 Nov 2025 23:02:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383756; cv=none; b=UTOvhj5rX4Xb6a3PGE4IIxoiRWnnXYgJUMqc/rem3yCgkLSA314oLUaSJ9OBrSn3AOz4g3IIcUdAzwmhWm6YB1mBzx4/DXY5VDoPkT9GM5ny/5/CVcmbwBx2JOGZ9zc9E0PQHUMPArZChL2SxClVF2moMMm/G1fMJkagxgjqLgE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383756; c=relaxed/simple; bh=ZGvpIhEgvWXGktZnhxfrMhN6hR7bkbAJOIhXp5Aq0Rs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ojLsDGYJ/oNeqju5P6vj87cCbgL4NobPvO70IiO11m8J3qmFjqwHvCvPQS/zjNqT2AdNyKIJmJRQX5DvHXl1wVJjBFcwz7dCSejcNR5mrMOq58jskJtKedVbCBTBz8tUFNnxVbykorMFZEZhpmY6bxlQD7MAGHK3ACgJEe86G1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Uuytsrba; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Uuytsrba" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2958db8ae4fso3308715ad.2 for ; Wed, 05 Nov 2025 15:02:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383754; x=1762988554; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=haGIr4TbAT28iZId1iIOWFqS56QerIllmXw1VE5QblA=; b=UuytsrbaALw8Q5my2uOdnsTlCl7N3PwpZgL1QKU9gzr2mDh+ngFN9PS0mYe+ixJ1Li GcXRXU43zsoSSx3SjBg0hhOj0vJ54NeTDGUdlcNfVYZK/ZTmNoIHais+uc3Qas479OuO orRtwpyKxE1xesyYc2jUb3GSeBKyyTZ+9lqF5qkFEImtkDGH4njnN27xDIybicFIaqIK qLQm5soW3ECVa3WvG8Yv0bLitxwgwOOB/VLKtiGevttZoLo77B8FEXjVJ7MNmbKNj0tZ ka+WoHqCe4T1tC368mSo2dLzo210tAU8JoJMgZQ8jz1ETY6iE54td43AQawKlwo/PWH+ zfHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383754; x=1762988554; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=haGIr4TbAT28iZId1iIOWFqS56QerIllmXw1VE5QblA=; b=tA4ORuXAClPaWD98pXNgh9rOcCmtSx2dJ7qytEVwBtLZf9D5dTibH4qMnUP9yfT7cS OaWY1gypfBaS2BmhPYyecPYWEdHharV9i/m83nwOJD8eYkAoUTn06TuRbJCJ98JogFrK vUM81vd4A3BEOVUZFYUxqbPnLZujHwGtlBcddaBYt2Xmg/Rc+rie48t2RRupUmfJ9ATs W5v5lK4S13DKZePOgqM+ZZ3iUAXhkyrtiBgvdPTKliD3kFu7IiUK26dZJY5r6Qr2OsU/ k9/Yc00QjQ1buxk8705Ysx99cwTUS5vkoMidcjyEGyjGsSxiBLkCL9vePVZ06QyJGb0R 9Z5g== X-Forwarded-Encrypted: i=1; AJvYcCVjz8GUOEwIOqrQvSxWk51jlKCmrFj3RpgZR8BUAu7aQJpDSI4WwjqT5LLfMLMr2mB5dTpaHIYzycVtvq4=@vger.kernel.org X-Gm-Message-State: AOJu0Ywcb9fPQkAJimGoJfAkjLkgRQo03+z8PCuCl6PrlKEF8pGJKoEk hudv0h0dXKevj6NSzflQX4oGLAx18fuxib0ibdvwS1wMbOE/ys9/PoDj X-Gm-Gg: ASbGnctgnXE/BeHmCuV4ghLMfOZS0cZGEdZVCsx3mUu3Ou07WH5t0incwnqc0qEeNhc QZeQIxMjsyjs4fUkBgD2NgvesE1iSTcu560PbpN8uLLyGhY924DK9/4/1vMaDzbFoy0GFbt+He+ 7q23wskUP0TCJaG57agmnEHy7OTrkLt+6E5zeZGv7ijFcFKDtDIc2j5lUmCDFikEws2UuanW0CZ dukMHNEq4sJRc2HCJJpRjw1FiiU6SDwgc/YnDQR48ONRuCvLe7CqqNmJhSjTFHwiH563iYMeF+f kW6zq0+JppzsknCZMWmyo3D0Gd5vqjROaGFkUu1IQtwBXctXjAFSl7UH5snTBQtv3Ciw/czKNoJ 9XJCFEDIcwonwGsGmECgivx9okLOpsybzgxQHFf29vm2vErqNystmO4qFqRt3wwcMCnYrSj6+AM UZoGJfocAQ3uEMBS8AcfKCmnQ= X-Google-Smtp-Source: AGHT+IFzjVuv6+fLc44rVR+754fMYDUlRbQZwzak3bXpRrpiq+jgHmjdZ9t766A4/ZV9UsiO7XyL/w== X-Received: by 2002:a17:902:f647:b0:295:ceaf:8d76 with SMTP id d9443c01a7336-2962add07f1mr64418025ad.47.1762383753899; Wed, 05 Nov 2025 15:02:33 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:32 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:18 -0800 Subject: [PATCH v4 6/9] rust: percpu: add a rust per-CPU variable sample Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-6-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=8909; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=ZGvpIhEgvWXGktZnhxfrMhN6hR7bkbAJOIhXp5Aq0Rs=; b=ozCkxX8AZS18Od5RqpDdJAKT5PuC+PMadw19K3gHEt4Arzb8pfY48uoQC0iTTZKu5yH9GvgEW af2cVxSKoMTDgAGQOQCroH9OOVN/z/g6/S42VKg++yjX4kPceoA4rk9 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add a short exercise for Rust's per-CPU variable API, modelled after lib/percpu_test.c Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 1 + rust/kernel/percpu.rs | 2 +- samples/rust/Kconfig | 9 +++ samples/rust/Makefile | 1 + samples/rust/rust_percpu.rs | 163 ++++++++++++++++++++++++++++++++++++++++= ++++ 5 files changed, 175 insertions(+), 1 deletion(-) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index 35656333dfae..8d83b6b86106 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include =20 void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) { diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 294b8ffc4f62..2db670c87fae 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -2,7 +2,7 @@ //! Per-CPU variables. //! //! See the [`crate::define_per_cpu!`] macro, the [`DynamicPerCpu`] type, = and the [`PerCpu`] -//! trait. +//! trait. Example usage can be found in `samples/rust/rust_percpu.rs`. =20 pub mod cpu_guard; mod dynamic; diff --git a/samples/rust/Kconfig b/samples/rust/Kconfig index 7f7371a004ee..23e35d64ac78 100644 --- a/samples/rust/Kconfig +++ b/samples/rust/Kconfig @@ -105,6 +105,15 @@ config SAMPLE_RUST_DRIVER_AUXILIARY =20 If unsure, say N. =20 +config SAMPLE_RUST_PERCPU + tristate "Per-CPU support" + depends on m + help + Enable this option to build a module which demonstrates Rust per-CPU + operations. + + If unsure, say N. + config SAMPLE_RUST_HOSTPROGS bool "Host programs" help diff --git a/samples/rust/Makefile b/samples/rust/Makefile index bd2faad63b4f..8a34d9d74754 100644 --- a/samples/rust/Makefile +++ b/samples/rust/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_SAMPLE_RUST_DRIVER_PLATFORM) +=3D rust_drive= r_platform.o obj-$(CONFIG_SAMPLE_RUST_DRIVER_FAUX) +=3D rust_driver_faux.o obj-$(CONFIG_SAMPLE_RUST_DRIVER_AUXILIARY) +=3D rust_driver_auxiliary.o obj-$(CONFIG_SAMPLE_RUST_CONFIGFS) +=3D rust_configfs.o +obj-$(CONFIG_SAMPLE_RUST_PERCPU) +=3D rust_percpu.o =20 rust_print-y :=3D rust_print_main.o rust_print_events.o =20 diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs new file mode 100644 index 000000000000..98ca1c781b6b --- /dev/null +++ b/samples/rust/rust_percpu.rs @@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +//! A simple demonstration of the rust per-CPU API. + +use core::cell::RefCell; +use core::ffi::c_void; + +use kernel::{ + bindings::on_each_cpu, + cpu::CpuId, + define_per_cpu, get_static_per_cpu, + percpu::{cpu_guard::*, *}, + pr_info, + prelude::*, +}; + +module! { + type: PerCpuMod, + name: "rust_percpu", + authors: ["Mitchell Levy"], + description: "Sample to demonstrate the Rust per-CPU API", + license: "GPL v2", +} + +struct PerCpuMod; + +define_per_cpu!(PERCPU: i64 =3D 0); +define_per_cpu!(UPERCPU: u64 =3D 0); +define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); + +impl kernel::Module for PerCpuMod { + fn init(_module: &'static ThisModule) -> Result { + pr_info!("rust percpu test start\n"); + + let mut native: i64 =3D 0; + let mut pcpu: StaticPerCpu =3D get_static_per_cpu!(PERCPU); + + // SAFETY: We only have one PerCpu that points at PERCPU + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut i64| { + pr_info!("The contents of pcpu are {}\n", val); + + native +=3D -1; + *val +=3D -1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D -1); + + native +=3D 1; + *val +=3D 1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D 0); + }); + + let mut unative: u64 =3D 0; + let mut upcpu: StaticPerCpu =3D get_static_per_cpu!(UPERCPU); + + // SAFETY: We only have one PerCpu pointing at UPERCPU + unsafe { upcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| { + unative +=3D 1; + *val +=3D 1; + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 1); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 0); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + + unative =3D 0; + *val =3D 0; + + unative =3D unative.wrapping_sub(1); + *val =3D val.wrapping_sub(1); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + assert!(unative =3D=3D *val && unative =3D=3D u64::MAX); + }); + + let mut checked_native: u64 =3D 0; + let mut checked: StaticPerCpu> =3D get_static_per_cpu= !(CHECKED); + checked.get(CpuGuard::new()).with(|val: &RefCell| { + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + + pr_info!("rust static percpu test done\n"); + + pr_info!("rust dynamic percpu test start\n"); + let mut test: DynamicPerCpu =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 1); + on_each_cpu(Some(check_percpu), (&raw mut test).cast(), 1); + } + + pr_info!("rust dynamic percpu test done\n"); + + // Return Err to unload the module + Result::Err(EINVAL) + } +} + +extern "C" fn inc_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| *val += =3D 1); +} + +extern "C" fn check_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| assert!(= *val =3D=3D 4)); +} --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 066752F5319 for ; Wed, 5 Nov 2025 23:02:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383757; cv=none; b=eHjCphvSsT3eF6+r5hTPBUdXSBcn3Mwzf8qDnpNJBWJU9jsPvkNX7rp8ZOsRt1HDWkyi0at5ThYM6sgCIIpTiT8PY92eGoZ1vmRrg0CNxVcoBnTr2NrqzN6dXm/77LGuyavQN4batHTVWBKTiTB+ZM0+kKhXxZfhR8eC5xKAKAo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383757; c=relaxed/simple; bh=BqnHuY+j+zWWmGsgTf4cjhQ+rYxxpT17fToJwK3G2cU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=K0qySab9XcMX7mTE2BBbG6ts37bunAsxQBqLQy6gVaaj6qr3q9uz0y3RKHa19zq+aKXbOdsdY0FRHLBDW3cOsv29m0v7AyYiZpaXhkRmIuP2jgSEddxkMT2PNIkR+55L7eXNtvoUrohhWnKFaT2Sq+oMuZBTEqlyl2Asx9tLmEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=f21KwWxz; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="f21KwWxz" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-29555b384acso3355915ad.1 for ; Wed, 05 Nov 2025 15:02:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383755; x=1762988555; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=JlEVe7ZQ+3aZbaXo/QWkECKjdoOcpAx/m3mobBviog0=; b=f21KwWxzPINkqh5S72/Tte/NS38rv3H9Y9JRxtefnCfwUyVWZ4hoI7aFuomForMQrU yO6FFEeDMG4AvJw3gR2oDtqd6DKi7eXuw178rBscx1x85s9iNRiy9j28oFAAHbDHKO7T +JARdr45ZX4bPGDidEj5EVozyZFBNlIfOYcVPBN1x5IgLodwjd1lx6ACNjcgB6cb62O/ cTIDsx/cFF21uYGfA9hA/paKic2tCqYGH/egxRZdhpNKc8KGYd1F3Q4zuQizf9D81dQW T/AMEFjAE+JgSbkykFKzzi5IdxiH7unbB9jWAAZFLaSbAEhxwADNDSgmWnTfNPn68zsp PVvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383755; x=1762988555; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JlEVe7ZQ+3aZbaXo/QWkECKjdoOcpAx/m3mobBviog0=; b=ZZzo+qMF3GgNxE234OYlSAGL/51zRnIei9kdUmReNJQWK0G7snCR/6sFlWJw+dhPPT W8NKOB/HinKj0x5Wy/wiiRf6EdH7Eg7u/dfLsczH3aWziVy0rMe7cvZrF26k1bEHNgdJ bO15V9m9PjOzDadzlEmwmBDNh7hphv8QI3ZWl4Ie/eJF8TgINIKla5SBAZYLduNotOUz WN4ojvhqyjvjnAm7osj4jU19asHMpHXm8y7H07kcMCJ6md1vIRL6cMcAak1EiVMuLQaf co3a8S2t6hnnTnsriSjA3tAqkPW77J+JQA1VynzaJOHwweI7Ns2PMTfTNto7OL+5uuwx efDA== X-Forwarded-Encrypted: i=1; AJvYcCXyKfPXaMEGitCcSS0bVGk6sdcHoANkGqm0RBAVHahP+RMYZgQWvvGZ5qsBI57grvOPKeyjhXUiS6ubwR0=@vger.kernel.org X-Gm-Message-State: AOJu0Yy1qvmagjeZVAcgPHPf8ctY5eLxZ52SFXesM4HhtgWDoEgxwcKL G/55UmIZT6bmY4Yj7K/4Lpi3Cue7fCn5dti3Wz/FOdwjbT2H5jTEBra+ X-Gm-Gg: ASbGnctsqoqOcVNde+SW78D2O5OmqXmo3250Pg+K2ywS6Pn3VRpZExEZhZSB+aeayOO ld1zrSouWC928upXMfdrZWkYQ3m04sRlMyDHwsmWTxH7wnOUw+1JQZXeKCTfHpbeB2pxBzvrGhO YcoQngvrUSsOdROE9DNV3Zc6G9U6/FyaZz52ePySKg9OYOZfp/ixNn0kMIb3hcCI2ajlPqUD09z 7Cgoe7YF3PvodJusUeX6FqTlKdB9R+IqjsGD8K7pM1gtui002QKecwQSMEToRRFugmTRwD6uAWc 1UBr3P6vp4sc3JJfNOnSoq2CPheCeF0wzXluQM6qD7uxx0DERHLeBoOhOkXWMtdwtQ0WFDOUSgl gK5ZiMyrJE3i56gcomKi7yXamW6xT7uHqTWd9GvRaMZYVIwBjaazW3gOq3oYVYSsQFEFVtqNJku cm4lBlc7P5tmj+nI2l3tx1IljkOdGx4+cWAg== X-Google-Smtp-Source: AGHT+IGL90UCdSoq8ldn75N4+7GWkkl8BPXpNVrFKQinz3VwFnZfYaDmILgoDXuV49BzxLMOOrniRg== X-Received: by 2002:a17:902:e751:b0:295:1277:7926 with SMTP id d9443c01a7336-2962ad2b031mr67951235ad.23.1762383754797; Wed, 05 Nov 2025 15:02:34 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:34 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:19 -0800 Subject: [PATCH v4 7/9] rust: percpu: Support non-zeroable types for DynamicPerCpu Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-7-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=10112; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=BqnHuY+j+zWWmGsgTf4cjhQ+rYxxpT17fToJwK3G2cU=; b=kTPf2rF1bKOOsD6lCvyiNpGrvnXPjibUxkDPtlxWNqeA6HklvELVCnpq28MLWOYvQwBQQswYj r6Ije0nlQwZCQZtslxG/M/AMyLX3SwifpWpGi+UEShOEycvwWb6obHp X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add functionality to `PerCpuPtr` to compute pointers to per-CPU variable slots on other CPUs. Use this facility to initialize per-CPU variables on all possible CPUs when a dynamic per-CPU variable is created with a non-zeroable type. Since `RefCell` and other `Cell`-like types fall into this category, `impl CheckedPerCpu` on `DynamicPerCpu` for these `InteriorMutable` types since they can now be used. Add examples of these usages to `samples/rust/rust_percpu.rs`. Add a test to ensure dynamic per-CPU variables properly drop their contents, done here since non-trivially dropped types often aren't `Zeroable`. Signed-off-by: Mitchell Levy --- rust/kernel/percpu/dynamic.rs | 44 +++++++++++++++++ samples/rust/rust_percpu.rs | 109 ++++++++++++++++++++++++++++++++++++++= +--- 2 files changed, 146 insertions(+), 7 deletions(-) diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index 1863f31a2817..a74c8841aeb2 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -89,6 +89,36 @@ pub fn new_zero(flags: Flags) -> Option { } } =20 +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `val` - The initial value of the per-CPU variable on all CPUs. + /// * `flags` - Flags used to allocate an [`Arc`] that keeps track of = the underlying + /// [`PerCpuAllocation`]. + pub fn new_with(val: &T, flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_uninit()?; + let ptr =3D alloc.0; + + for cpu in Cpumask::possible_cpus().iter() { + let remote_ptr =3D ptr.get_remote_ptr(cpu); + // SAFETY: `remote_ptr` is valid because `ptr` points to a liv= e allocation and `cpu` + // appears in `Cpumask::possible_cpus()`. + // + // Each CPU's slot corresponding to `ptr` is currently uniniti= alized, and no one else + // has a reference to it. Therefore, we can freely write to it= without worrying about + // the need to drop what was there or whether we're racing wit= h someone else. + unsafe { + (*remote_ptr).write(val.clone()); + } + } + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: Some(arc) }) + } +} + impl PerCpu for DynamicPerCpu { unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { // SAFETY: @@ -105,6 +135,20 @@ unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCp= uToken<'_, T> { } } =20 +impl CheckedPerCpu for DynamicPerCpu { + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: + // 1. Invariants of this type assure that `alloc` is `Some`. + // 2. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are + // initialized on each CPU. + // 3. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is + // live. + // 4. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for + // a `T`. + unsafe { CheckedPerCpuToken::new(guard, &self.alloc.as_ref().unwra= p_unchecked().0) } + } +} + impl Drop for DynamicPerCpu { fn drop(&mut self) { // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs index 98ca1c781b6b..be70ee2e513f 100644 --- a/samples/rust/rust_percpu.rs +++ b/samples/rust/rust_percpu.rs @@ -11,6 +11,7 @@ percpu::{cpu_guard::*, *}, pr_info, prelude::*, + sync::Arc, }; =20 module! { @@ -130,13 +131,81 @@ fn init(_module: &'static ThisModule) -> Result { =20 // SAFETY: No prerequisites for on_each_cpu. unsafe { - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 1); - on_each_cpu(Some(check_percpu), (&raw mut test).cast(), 1); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 1); + on_each_cpu(Some(check_percpu_u64), (&raw mut test).cast(), 1); } =20 + let mut checked: DynamicPerCpu> =3D + DynamicPerCpu::new_with(&RefCell::new(100), GFP_KERNEL).unwrap= (); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 1); + on_each_cpu(Some(check_percpu_refcell_u64), (&raw mut checked)= .cast(), 1); + } + + checked.get(CpuGuard::new()).with(|val: &RefCell| { + assert!(*val.borrow() =3D=3D 104); + + let mut checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + + let arc =3D Arc::new(0, GFP_KERNEL).unwrap(); + { + let _arc_pcpu: DynamicPerCpu> =3D + DynamicPerCpu::new_with(&arc, GFP_KERNEL).unwrap(); + } + // `arc` should be unique, since all the clones on each CPU should= be dropped when + // `_arc_pcpu` is dropped + assert!(arc.into_unique_or_drop().is_some()); + pr_info!("rust dynamic percpu test done\n"); =20 // Return Err to unload the module @@ -144,7 +213,7 @@ fn init(_module: &'static ThisModule) -> Result { } } =20 -extern "C" fn inc_percpu(info: *mut c_void) { +extern "C" fn inc_percpu_u64(info: *mut c_void) { // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); @@ -153,7 +222,7 @@ extern "C" fn inc_percpu(info: *mut c_void) { unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| *val += =3D 1); } =20 -extern "C" fn check_percpu(info: *mut c_void) { +extern "C" fn check_percpu_u64(info: *mut c_void) { // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; pr_info!("Asserting on {}\n", CpuId::current().as_u32()); @@ -161,3 +230,29 @@ extern "C" fn check_percpu(info: *mut c_void) { // SAFETY: We don't have multiple clones of pcpu in scope unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| assert!(= *val =3D=3D 4)); } + +extern "C" fn inc_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu= >)).clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let mut val =3D val.borrow_mut(); + *val +=3D 1; + }); +} + +extern "C" fn check_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu= >)).clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let val =3D val.borrow(); + assert!(*val =3D=3D 104); + }); +} --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57CAD2F60DF for ; Wed, 5 Nov 2025 23:02:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383759; cv=none; b=Z71x1PaQr7u97oemGeTe/ywbWWYXijTKhNujCOXHahi0qSncR6pN48UP6VALghqBDNUCMhrD8BVYszR3abUsXR2NWGWDHEeJDidX6HCpHTdeG4FgDiTlodqkA1k2UQrfb6wbxYfkAzMRY1NlYkaSMlAb3wJ533dlLLjbMUM7BXI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383759; c=relaxed/simple; bh=N2+WVoODdmEMs33Jmf91+GyS5dqY4IkppGnGifhlfUw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=pqEaFuX+FpnC32nb/X+ogR65BIIim/bD9wIK/TjfKdDYAi4UQbdcoVXCXh4NnIhv/xrkBpy+V963VV7e4wVuGAr4pBPnwEjGJ25WvACPvCrRbHG8gQOtzEN1TZGwRD8C8AY7RUzHc71WiURdj8d1IdM4SGkreiBilLtgeP1GtqI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N6pHZc2N; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N6pHZc2N" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2947d345949so3124115ad.3 for ; Wed, 05 Nov 2025 15:02:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383757; x=1762988557; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=kN9/UL7jFZa5RKDycC8BJ8oS9HZfDBeA7oqBiSHbTHc=; b=N6pHZc2Npl5eaVO2n/b7b04fm+P9pqcJhHCenZIGBiFSwNHKQ7Z+PfNceGuTlzVvZb EJv4CLqoG92u+KD3EufQgzAXGA3fCWCIM9TgcVyRwbnKY7P+bszKhlUIq+D9hdlGxbwR VjNet2wSSjVwiFo7DbG0UmROco+eGl3rsqZL9cUSTLsQYkAYuU1yy+six4gwq9NLOQIx SMpFphRCTc+UKhwo0cX2dnvRhBA0Ho6DpKdWx3Bka76dBaeaQIQbK2+5aKehNk+vdgXC 37dv7d6IxSMaWww5ZS3gCTjxNnTQyLBpmlIn3VTxRYkFTsy7iNbZJBYQH9RX5m2WcLN9 6diA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383757; x=1762988557; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kN9/UL7jFZa5RKDycC8BJ8oS9HZfDBeA7oqBiSHbTHc=; b=Gxd/LFsZDTEeaxo/g2BOSsGYr57yxtkc/PzkGlGCOeroGsegG+d/wB2VRtArFKTaqb riuiciE4RP7wEnysUPFJXYsuVCVvbYcfAz3lw4Oitat/zMAIKxTzIX+Bd25mmV8WRoLa CS9E0JGhwu1MvB6ceJJJaRIsjZgdApXZ1Q9G/t/X9D7DagmpV0DZRVirWn7Dn6Xa2bmT Nv4anBepjbmSFkKrcCAjlw5ziVkGnS9YSfeZQgew5kyuMIFvnfT/X1B22ZzRS7Dd1mzK fq9ShjbhXaX3SKynvtCmo6cka0Q8Y/LnuHmIt71BsGwfUOqMbVA3XbpRdl2Cq0p/IlDm Htxg== X-Forwarded-Encrypted: i=1; AJvYcCWnHGXswqQWRQTAVOVJm6NWplqxU+BbzeRVhhsbM7FgmThWICdp8ojV/c+u3ig+p30g5kyjIl6WMRNCe/Q=@vger.kernel.org X-Gm-Message-State: AOJu0YxUFi0qijJNVlXdksO+5bSs0+b/99kXdoUhMrfwZfP/L5SI+ePa Zoboww34dE502k8MlarGGkbpQ6rCYOgwuDP3uyRzDeV6TvW+Yl4VpmZY X-Gm-Gg: ASbGnct8Woi2Xr0syX2mZZVChSqH0gwHGoKX3smYUMrDaShQxOeaSmdXy5HzR2zrhBS GktEfYSyboWOrcw5Cn7avSlxbo4B/0foSooCzSQHceK5jkMdpO/hAzbfYvtXJ270u9Yb0Qs7HIi CNsTFboKV9igtLxzyWnLrH4HgsP4yq7+4TUYPHZIcGLhyxTZKUp+zaeej5uQLThnK09dd0WLizp 7ul96dkZOeNCTikFycIxrOZfgg4/JDBLTL9ws/ELvIF1jqOaohCE8CgDwoh7R8N0QDOieUdE39+ N/qY6hz5HKH8g3ZjG/ZQpT/xXwIXvH+StQ8I/Q8mnZGrzs++/xBX3DmWr72GXsrCefE8N75aWYT NisCpgqPIeRS+2eYaT67rxPtJM+Gr262iRIV5TBMMITDV+jlbgQi+nggJB9rNbMAcOU83Vlx7Ru jWPOAzeOXYCW6QT8nDNSm5hbxY45Vet1SGWw== X-Google-Smtp-Source: AGHT+IFQkxzoaevtRmWsYeKKLp4dhk9UzDcwkiSO54MpJYOtGliGQSf/Idp109RNMzxqYIG9F5TfdA== X-Received: by 2002:a17:902:ea09:b0:293:623:3260 with SMTP id d9443c01a7336-2962ae9b937mr61526055ad.57.1762383756270; Wed, 05 Nov 2025 15:02:36 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:35 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:20 -0800 Subject: [PATCH v4 8/9] rust: percpu: Add pin-hole optimizations for numerics Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-8-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=10554; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=N2+WVoODdmEMs33Jmf91+GyS5dqY4IkppGnGifhlfUw=; b=Z2rvF5hwLPBTS4udKgnmGgDRG4sF6Te7a0VTPOZNeZ9aQJCrtJODteQuQetfsAi6BEb6zjaF0 IBremlr6jdEBRfajV62taxvmYwsiKbLAK9s6VdxCrwqQp8iWjg7VVVT X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= The C implementations of `this_cpu_add`, `this_cpu_sub`, etc., are optimized to save an instruction by avoiding having to compute `this_cpu_ptr(&x)` for some per-CPU variable `x`. For example, rather than u64 *x_ptr =3D this_cpu_ptr(&x); *x_ptr +=3D 5; the implementation of `this_cpu_add` is clever enough to make use of the fact that per-CPU variables are implemented on x86 via segment registers, and so we can use only a single instruction (where we assume `&x` is already in `rax`) add gs:[rax], 5 Add this optimization via a `PerCpuNumeric` type to enable code-reuse between `DynamicPerCpu` and `StaticPerCpu`. Signed-off-by: Mitchell Levy --- rust/kernel/percpu.rs | 1 + rust/kernel/percpu/dynamic.rs | 10 ++- rust/kernel/percpu/numeric.rs | 138 ++++++++++++++++++++++++++++++++++++++= ++++ samples/rust/rust_percpu.rs | 36 +++++++++++ 4 files changed, 184 insertions(+), 1 deletion(-) diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 2db670c87fae..c1148cb36eff 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -6,6 +6,7 @@ =20 pub mod cpu_guard; mod dynamic; +pub mod numeric; mod static_; =20 #[doc(inline)] diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index a74c8841aeb2..99acbf6363f5 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -18,7 +18,7 @@ /// the memory location on any particular CPU has been initialized. This m= eans that it cannot tell /// whether it should drop the *contents* of the allocation when it is dro= pped. It is up to the /// user to do this via something like [`core::ptr::drop_in_place`]. -pub struct PerCpuAllocation(PerCpuPtr); +pub struct PerCpuAllocation(pub(super) PerCpuPtr); =20 impl PerCpuAllocation { /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, @@ -119,6 +119,14 @@ pub fn new_with(val: &T, flags: Flags) -> Option= { } } =20 +impl DynamicPerCpu { + /// Gets the allocation backing this per-CPU variable. + pub(crate) fn alloc(&self) -> &Arc> { + // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. + unsafe { self.alloc.as_ref().unwrap_unchecked() } + } +} + impl PerCpu for DynamicPerCpu { unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { // SAFETY: diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs new file mode 100644 index 000000000000..e76461f05c66 --- /dev/null +++ b/rust/kernel/percpu/numeric.rs @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Pin-hole optimizations for [`PerCpu`] where T is a numeric type. + +use super::*; +use core::arch::asm; + +/// Represents a per-CPU variable that can be manipulated with machine-int= rinsic numeric +/// operations. +pub struct PerCpuNumeric<'a, T> { + // INVARIANT: `ptr.0` is a valid offset into the per-CPU area and is i= nitialized on all CPUs + // (since we don't have a CPU guard, we have to be pessimistic and ass= ume we could be on any + // CPU). + ptr: &'a PerCpuPtr, +} + +macro_rules! impl_ops { + ($ty:ty, $reg:tt) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc().0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `StaticPerCpu`'s inv= ariant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + #[inline] + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("add gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + #[inline] + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("sub gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg) rhs, + ); + } + } + } + }; +} + +macro_rules! impl_ops_byte { + ($ty:ty) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc().0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + #[inline] + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("add gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + #[inline] + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("sub gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + }; +} + +impl_ops_byte!(i8); +impl_ops!(i16, "x"); +impl_ops!(i32, "e"); +impl_ops!(i64, "r"); +impl_ops!(isize, "r"); + +impl_ops_byte!(u8); +impl_ops!(u16, "x"); +impl_ops!(u32, "e"); +impl_ops!(u64, "r"); +impl_ops!(usize, "r"); diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs index be70ee2e513f..31ab3fcf5c6c 100644 --- a/samples/rust/rust_percpu.rs +++ b/samples/rust/rust_percpu.rs @@ -28,6 +28,26 @@ define_per_cpu!(UPERCPU: u64 =3D 0); define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); =20 +macro_rules! make_optimization_test { + ($ty:ty) =3D> { + let mut test: DynamicPerCpu<$ty> =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + { + let _ =3D CpuGuard::new(); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = *val =3D 10); + test.num().add(1); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 11)); + test.num().add(10); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 21)); + test.num().sub(5); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 16)); + } + }; +} + impl kernel::Module for PerCpuMod { fn init(_module: &'static ThisModule) -> Result { pr_info!("rust percpu test start\n"); @@ -208,6 +228,22 @@ fn init(_module: &'static ThisModule) -> Result { =20 pr_info!("rust dynamic percpu test done\n"); =20 + pr_info!("rust numeric optimizations test start\n"); + + make_optimization_test!(u8); + make_optimization_test!(u16); + make_optimization_test!(u32); + make_optimization_test!(u64); + make_optimization_test!(usize); + + make_optimization_test!(i8); + make_optimization_test!(i16); + make_optimization_test!(i32); + make_optimization_test!(i64); + make_optimization_test!(isize); + + pr_info!("rust numeric optimizations test done\n"); + // Return Err to unload the module Result::Err(EINVAL) } --=20 2.34.1 From nobody Tue Dec 16 05:52:51 2025 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEF792F60A2 for ; Wed, 5 Nov 2025 23:02:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383760; cv=none; b=ux2DWi6FlGQzUyEIitPxeXcNLEYQrxjAvweS9LOCqmEcd5Wht131+AQh93OXVn7onxY0hUCQiUCMY3i8AuqQz1VSOjhitkmr5N1bfIEFH9WrLk8Ejky0uAfl4+mRucpEJkzOI2oS4q1pFSfWz2m1TmPIEIIbg5Tutw498E3QxS8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762383760; c=relaxed/simple; bh=D3RYV7HLnBtitWmFErq2p8agS2TaxcSWS/fZMuq/eYY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=PfKy2FArSN07AYNcfEkE4tMUOWpHF7Idsd289/7BUJbaRytCvGJ6e+vfauvAzvCSDhjrh5o1hRH4VbO3GBC0lV0lPe6M/ZXbhi27zdCHHqiKzztYS7cFJbrJHVUWEQONNUByz6mcdAIM9Y8ZoLSllqUZm/hPRW5bkEB/UBrO8Y0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nDfHChHo; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nDfHChHo" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-29470bc80ceso4089075ad.1 for ; Wed, 05 Nov 2025 15:02:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1762383758; x=1762988558; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=4CKxB9LH52Z/bMQCpny1rQR1F6MuknwX7Uens3YkL/M=; b=nDfHChHoUBguPjIV+y5+ojr9zcj4KRUwDC4qQ998BvSYl1r77zW+Mz0RvKbwnKq2of +lHcn6pJNGr6swlbMwE3iFXmaHWqF5TTsTXis7DVyqcSh2pJbppnNnvK503KkQhIGJcC HxmJB+T2PttBwOM3PDD/BJBDNlErUuBoKOQMnSyiivJ8RGXt+70ChjU86XZip2BgdPL2 84FUO4lkBrZive47VG6tJP5Ukn6tjEOgQYuixiIOzrMzXEzi/6lYtvOjCkksuZuLz+WA gPaAz2+8xcHF46RlMr25NQcAB3W9swp/YgXE99c8o0YXDU0wcEtC85GIXpzcAostl++r 4yeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762383758; x=1762988558; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4CKxB9LH52Z/bMQCpny1rQR1F6MuknwX7Uens3YkL/M=; b=PckZleXrOXFibQtdVXWW9GgK7myUpIYYIYaQrSm/3sHPmAMw8r3A/NNH7S4LXZKtjq zkSOi2ksIh9x7Ynzoq3M18GosRI6Yg0VdntI79UCAknB85MZA77PDYJJABd4QZxmn4NW bZnmrIQzQCCPXEGDMsQkIBGcD/mBOfkFWDTgg4PCzGckZxX2wprGcOoSHy53hYKTkS4S +Ha2QxO4PQPRpFufePwkdf38iGc71V74AKazFcShI1HKvXDFX3tNt0y3+KVTQld92Q4R t9wXU0F4i59z5EI/jHOUaEWwCyHvNypWvcG6mCTrDE8IHw/K5onhnfoZihEa2ppxh53W vtvA== X-Forwarded-Encrypted: i=1; AJvYcCVihHgoBgqtyfODVC5sZUTa/R3Wpwfai6TyKcjUaVpqPbFlzkVnhdJFalJeVr4i+ZNy0dNaXHKaA9N6kT8=@vger.kernel.org X-Gm-Message-State: AOJu0YzA9XREZRhaiKhoTjX9fJ2MBTRNiCcPBQDusRiB2dCB+m6t7MLY ST7+ISoGO/8ABsILgGpXnqagr4XDassX7sMdUxmkVAhsI5cs1r7j/0va X-Gm-Gg: ASbGncvbQJoREV8KtOUXHB1qXPUpuLs9amdUalVUpR3nAt4IKtRYxMlJ9dSCyCSGYy1 zQGTj0XxXGU4rWK4kywslXxwYZ6IoaJsCXJs2qHSxVMPcpkHHDTpdBkr5Oo5gxF8slo74vLepmN 6F6sdXw4BWjEbmkkMww5S7HvKqH7gK26RERTK3WUKuB7wbzZ6zm5WlMn+GdmdRpofoRtB+vmO07 BBGoPjRM+OfwO55Ec0BWzHmusU3J4sS0PtD3wU4jzJyLP7xHI8+BaBJ/PJeXDN5Ltl4DGBfcZwc d4/5lYdzHZF8j7XJ+Peb53RVKX5l8Shj2KEpvWsKAioEDmuOM15YkzZyEDZ9U506GChMjQE2XCA g0uESGAcC2kyuLnTPIfErdlaBVq7hbwAfpHAhCr/6kWa6f1xN5rK1trPOeo05iaWqe2AwgsRM4j X0kLYjKN6Kn7E4j+DIq8tD+dmUKh25ifGIEA== X-Google-Smtp-Source: AGHT+IHhgn+8YeMFZ6PkDMcRo+YCf1hZhzierVrKIefbBTSFSntXm/b76xNV6gdhRNX1LQibgdMpiQ== X-Received: by 2002:a17:903:19c5:b0:295:50f5:c0e3 with SMTP id d9443c01a7336-29650c5af22mr13633165ad.14.1762383757680; Wed, 05 Nov 2025 15:02:37 -0800 (PST) Received: from mitchelllevy.localdomain ([131.107.147.147]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2965096839asm6325645ad.13.2025.11.05.15.02.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Nov 2025 15:02:37 -0800 (PST) From: Mitchell Levy Date: Wed, 05 Nov 2025 15:01:21 -0800 Subject: [PATCH v4 9/9] rust: percpu: cache per-CPU pointers in the dynamic case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20251105-rust-percpu-v4-9-984b1470adcb@gmail.com> References: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> In-Reply-To: <20251105-rust-percpu-v4-0-984b1470adcb@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1762383744; l=3408; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=D3RYV7HLnBtitWmFErq2p8agS2TaxcSWS/fZMuq/eYY=; b=F3SsT9YaklVAwHq+tWc26blxo+XKhI/hNXdvlsqIbyeoSRfdd0pXKRoUeMLZ44FBDlWmhxL/e gWSNZYhRA9NAl0i5KEtL+ADOegGqjFQajhB5DktuXDNdnfmDxdg1IZC X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Currently, the creation of a `PerCpuNumeric` requires a memory read via the `Arc` managing the dynamic allocation. While the compiler might be clever enough to consolidate these reads in some cases, the read must happen *somewhere*, which, when we're concerning ourselves with individual instructions, is a very high burden. Instead, cache the `PerCpuPointer` inside the `DynamicPerCpu` structure; then, the `Arc` is used solely to manage the allocation. Signed-off-by: Mitchell Levy --- rust/kernel/percpu/dynamic.rs | 22 ++++++++++++---------- rust/kernel/percpu/numeric.rs | 4 ++-- 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index 99acbf6363f5..dcf3e1c4f7a9 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -72,6 +72,9 @@ pub struct DynamicPerCpu { // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by the alloc is // initialized. alloc: Option>>, + // INVARIANT: `ptr` is the per-CPU pointer managed by `alloc`, which d= oes not change for the + // lifetime of `self`. + pub(super) ptr: PerCpuPtr, } =20 impl DynamicPerCpu { @@ -83,9 +86,13 @@ impl DynamicPerCpu { pub fn new_zero(flags: Flags) -> Option { let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; =20 + let ptr =3D alloc.0; let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: Some(arc) }) + Some(Self { + alloc: Some(arc), + ptr, + }) } } =20 @@ -115,15 +122,10 @@ pub fn new_with(val: &T, flags: Flags) -> Option { =20 let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: Some(arc) }) - } -} - -impl DynamicPerCpu { - /// Gets the allocation backing this per-CPU variable. - pub(crate) fn alloc(&self) -> &Arc> { - // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. - unsafe { self.alloc.as_ref().unwrap_unchecked() } + Some(Self { + alloc: Some(arc), + ptr, + }) } } =20 diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs index e76461f05c66..23a7a09216d0 100644 --- a/rust/kernel/percpu/numeric.rs +++ b/rust/kernel/percpu/numeric.rs @@ -22,7 +22,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc().0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { @@ -78,7 +78,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc().0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { --=20 2.34.1