From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D12D3AE71E for ; Fri, 10 Apr 2026 21:36:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856983; cv=none; b=QIBzFz7+r3WlykskQ4DqtFSXH0w4e6DocX6R+qHjJ78Xbz1DQGCXNRkgxkkMysnUREJj9QNGlGK5Uq4YvNGmM2qeN/NVoLEkphMXEtGaaEZTg0rjfHCe+7K7b66gRQeUWy2lXdpE+DRyJAtACajR2uZeGiEPG2hayFWoZIcyAE0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856983; c=relaxed/simple; bh=BETYXv/iPQABug1OuKQYEFZYT26teVkAoSalVy+DWvg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eGQGifSmyFFu6w0BBk3236V1QoPs9qC+7+HSVNSslXDFTr/njeJ06+To1cl4jlBIFRqMiEG2ygxPxINHfn9Q3b85JzsMfQrdUVb2SGdkNowwoeXO6tp7qGsKyBZREtEvmnylq7YqfPBdvsb9yQboyH2FCE9fPKxtuZf7/7RperM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U7ImcGtJ; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U7ImcGtJ" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-82a7ebc729dso1272169b3a.3 for ; Fri, 10 Apr 2026 14:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856981; x=1776461781; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=LSa2drr2d72jbSnbpnPMy5VtafgWDLdLn9PYujy+71c=; b=U7ImcGtJiF823NKtD4IWUpdWdAJelNVWB1noqW/OogAgKb4ptbCiYjbyeuzT2FNSoH e/lnYpP8UzzybujVDStRdz+xHx72RBVGctA5EWt88gFY9D+G2qVUeSxNm4AkaM9HbJQV Bkoh16vGwDrTsfzzClHvxNijb2GCn57rpU+2W1diCL5VZhNNGAkeglCfNSUeuY7e+kW9 rF9rsh5BKRXXjUgMU0QizzfxYqMkzzeFcAVg5YtFFkx8L1pyUHS8/S0xZC7gyylyi4lS qEyBEOIfnhwzeY7Qao4Cwjwu0gmSVPXom1GfDO7hCp5oFzDKlcfYAIP67fuLvF8+M5P5 lKHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856981; x=1776461781; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=LSa2drr2d72jbSnbpnPMy5VtafgWDLdLn9PYujy+71c=; b=Z3z/JI4GsJIBTVPd7yrRi+RHeNHVXxs67agRnbEV1Zo3vgFzuPzfiBaL3uQqFBn2TJ mu3s+CgyYxi82z3IaRFD7gNcLfrsltYaETMJ9SlXyH88IvKHWfCKpoR09PTd3Vrsl7ar NFIQ95rhV/qyDbI1iGNB0j0MlpVXAtNzrp+gTTJxPuOdMBi4XD8ZAisHNUZMoRTSel5Y HvwAGg8FXSKdUcDBQJWp3tXXVmGYz0RKt6i8S3MYxYNTymYjvII6roJXpt4HBg3U1Evf 2uNeKyX/WnwV/P46V8n6Lxl/b0WDKlLsBwzpCvnsJ+Cc4kKAwX9mAm/bPFaPGCXbAPL3 +jwQ== X-Forwarded-Encrypted: i=1; AJvYcCUzOhxTjU28orc0IUIRCHylB3iq34gebCXB8nlZYtQ6qMWlbDSbqhNQLr9DUtoeJOvKrRE3FhaRJtl8ZtM=@vger.kernel.org X-Gm-Message-State: AOJu0YxRQ59cMo50pLP6hrA8EUZu9v8esRcqcfZEMyyD5xZeWUkfcszh fbSnU6GFG7+go7S3Iqzx7PgImcu6ziDtvZPcnY4W4/hc+TTc9vGNxyPX X-Gm-Gg: AeBDievhua4Ut97CH5oy+R77JvNxTftbehL4vMJf8EiznWlA1XV5fgsbltSPhiNFIME 1RsVW7dMHrnLxt2fRJgAxJ8DXa4PdL1KsA5u5fqMqx5lYabyklUlujUMNn/k5l/9fjAa2p2EFYE FJwZa3xGieZ61xVM2G+EaPml/l+ROaPQPps0SSHCt9pFzFVerN0OOd5r3h+8pG9bxqOQAb6a36r U2WK+0etNtfPCG51o3DNWu+pLIJO9LI+vcB7a051ytfLP09o8rxxzhUcsYiqZmMLUOJPFAMOF88 2RRkzVPo/SUjyuDVCmi98K5HraNCBEPe6GAaWzC0XT41qV/4Y92hhcmIzSgzN3mnZy4vImcaiPD jHmPSc/9JT5AQMaO1w90AstCEg+sGn1IHfFRbxmp5nuwms2oPBlqdkoRS6mAV2rwFOUYyBYTKAE aW6KOmq9g37G89YvSbIVnOYhbnPu726saXNLcdzIO1azv3 X-Received: by 2002:aa7:9a84:0:b0:82c:8c6a:682b with SMTP id d2e1a72fcca58-82f0c169cb8mr5207029b3a.19.1775856981518; Fri, 10 Apr 2026 14:36:21 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:21 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:31 -0700 Subject: [PATCH v5 1/8] rust: cpumask: Add a `Cpumask` iterator Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-1-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=2878; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=BETYXv/iPQABug1OuKQYEFZYT26teVkAoSalVy+DWvg=; b=XwjVmH2pYFv3eafAe1fjdPZKz8H5eTGdm8cmz3xV4HYLP+EHPiCfuzPJVwN/Yjd+JMCLv6Wnp AS9Kw9gl3CcAMjDQbDNlLeQK3O9aaHxYfUQ5eZRr2Un5733dXgLh4dI X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add an iterator for `Cpumask` making use of C's `cpumask_next`. Acked-by: Viresh Kumar Signed-off-by: Mitchell Levy --- rust/helpers/cpumask.c | 6 ++++++ rust/kernel/cpumask.rs | 51 ++++++++++++++++++++++++++++++++++++++++++++++= +++- 2 files changed, 56 insertions(+), 1 deletion(-) diff --git a/rust/helpers/cpumask.c b/rust/helpers/cpumask.c index 5deced5b975e..76990e14dfdd 100644 --- a/rust/helpers/cpumask.c +++ b/rust/helpers/cpumask.c @@ -50,6 +50,12 @@ bool rust_helper_cpumask_full(struct cpumask *srcp) return cpumask_full(srcp); } =20 +__rust_helper +unsigned int rust_helper_cpumask_next(int n, struct cpumask *srcp) +{ + return cpumask_next(n, srcp); +} + __rust_helper unsigned int rust_helper_cpumask_weight(struct cpumask *srcp) { diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index 44bb36636ee3..b74a3fccf4b4 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -14,7 +14,10 @@ #[cfg(CONFIG_CPUMASK_OFFSTACK)] use core::ptr::{self, NonNull}; =20 -use core::ops::{Deref, DerefMut}; +use core::{ + iter::FusedIterator, + ops::{Deref, DerefMut}, +}; =20 /// A CPU Mask. /// @@ -161,6 +164,52 @@ pub fn copy(&self, dstp: &mut Self) { } } =20 +/// Iterator for a `Cpumask`. +pub struct CpumaskIter<'a> { + mask: &'a Cpumask, + /// [`None`] if no bits have been returned yet, or the index of the la= st bit returned by the + /// iterator. Equal to [`kernel::cpu::nr_cpu_ids()`] when the iterator= has been exhausted. + last: Option, +} + +impl<'a> CpumaskIter<'a> { + /// Creates a new `CpumaskIter` for the given `Cpumask`. + fn new(mask: &'a Cpumask) -> CpumaskIter<'a> { + Self { mask, last: None } + } +} + +impl<'a> Iterator for CpumaskIter<'a> { + type Item =3D CpuId; + + fn next(&mut self) -> Option { + // cpumask_next WARNs if the first argument is >=3D nr_cpu_ids whe= n CONFIG_DEBUG_PER_CPU_MAPS + // is set. So, early out in that case + if let Some(last) =3D self.last { + if last >=3D kernel::cpu::nr_cpu_ids() { + return None; + } + } + + // SAFETY: By the type invariant, `self.mask.as_raw` is a `struct = cpumask *`. + let next =3D unsafe { + bindings::cpumask_next(self.last.map_or(-1, |l| l as i32), sel= f.mask.as_raw()) + }; + + self.last =3D Some(next); + CpuId::from_u32(next) + } +} + +impl<'a> FusedIterator for CpumaskIter<'a> {} + +impl Cpumask { + /// Returns an iterator over the set bits in the cpumask. + pub fn iter(&self) -> CpumaskIter<'_> { + CpumaskIter::new(self) + } +} + /// A CPU Mask pointer. /// /// Rust abstraction for the C `struct cpumask_var_t`. --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55C523DA7F4 for ; Fri, 10 Apr 2026 21:36:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856986; cv=none; b=WvCt+4e6sUDHkh851YQ4CNi0wgVGVDPKiIcck+6T52E4AQGlcWNJ6SuNhZSF9BpwpRp3DQ9SZJ67ZpsK8oGaTHdK+01e/1i4/ALNK+ehYnafhhnfNUjUwrAg+XTKzZwJ3Wqv8Qu3wRhrAWUO9RlS2xSwdszL/l7Eq+zKvUdTR0s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856986; c=relaxed/simple; bh=nFa6XzGa0lobebiKoERWPKmf/z75UhcpF+sR2BSl5r8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=p8VNvnFF1+yseoKYXL37stRbFwcXYC65Pjt/mzjhVGP2E3lCVkHw7ZPz5cMAsyK8y156/OrTkFVd5hBk5HNOTDqUL1SdHpaureBb4bUKRuPCeENA6AXaKHdQlpPWsv075UM1B8ynpBE9Qod522CJ0p7dIUwznNtCG1DTutWnnNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Vt36GyVA; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Vt36GyVA" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-82cdb4ab547so1339764b3a.2 for ; Fri, 10 Apr 2026 14:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856983; x=1776461783; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=d/ZkXAjTN+MjWCqsB5McracnlHQTzul2ROdtvOpaZMM=; b=Vt36GyVAsPe2qsFibdE3FvvnN245tXrYImgt5Pn9F6X9vzohN1y1ZDGuw5okBh6ebg kRURf45/4xg1uKKWkHE93ZV64FNKcYjw2rXD/62bFO02VYSAGUJkAjxDExCwdbcw0aNc 0PMVPDoAwDfWfx3UAURVcXyRLqtg5zCeow07P8HxiDMGiwv6I7nHFUQ2BoO+uXdTHlg/ VCj6xI+lDw0FFeaRlRUDN1eKyL8YxppwcUWFo6H7zbK1kOslMXYpjEHf5XDS3Q2pTkOi +X3pWvhakX7Si6/tMB0jzA0w3URegYZTgp2fAaCotw2RO9oTnd1Hea3VsmfbORmSyO+d r1/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856983; x=1776461783; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=d/ZkXAjTN+MjWCqsB5McracnlHQTzul2ROdtvOpaZMM=; b=Uu5TiQ5PwgPtMrjySLG2uWt/+jccUp6m7Ihj8M4MHK++4BSAgH+wIl6fLDeReVeOVI Xaq+hhDIS4OPfmRbmMWwFlF6CwHAqf/9M8WgHP80IXoT9zr8Kl2MHXhpWBAaZHuwP9tj De0OwW/E7Xmg+MP3ryOZWirm3yvgvAGDypFiHyoC5D9XOaX5mnx7x1wQC0azyvWi5Ztm gOxD/RKmJe5LArCueizNxzTiBnTzzeux209uSOyNqG9vj6Cz4bxA0D0HlMuXa/LcNiZx D7qlYPJZk8YOw+PWgmx/WQGeZAxvT8viHcf3du+6jJ80LEv6jf6lHt6CyZ7HCXX+2IcY yCdw== X-Forwarded-Encrypted: i=1; AJvYcCWyokLA3v1j1tcmy6kEDjwE6N1Vs0BQsH4U/CxvMb8K6HYPcJKUBkIJATP0uvj164zbyW3VQepnSt/Gt88=@vger.kernel.org X-Gm-Message-State: AOJu0YyiYAEUpCPPXRx2F5nv96/gmF9wtbJ4i8UUNHL5CUM5l2Ee3qjx Ij36qQVP/g5OkqZ95FTtKfaRd1pkEtYCqHo2ZstEk0P7eT41LRU5VH2k X-Gm-Gg: AeBDievI0l/x5ys9dEMDBe2b/vL0F6SCm8WD220AZMosQnpSRuhmCMyDuGvmejjlv14 +KLaYzcIy4bATVO0zwzOJ0kTF0LLj6Tv41B96LWk/t+nWAXlDOqlzaHkSFItQFqnXJR+Sk0pKNO bBQW7Lqk4Xv/Tj0mrRkdiaBJGOMzPs563P51NBntgIEfJBYiIEsB8K6J02DEflMVMFefKiN6vTl ULNUMzeJL/XpgBkGVDWYqYDCuN8jgVhE3uDfOt0FAjI3NVpIMTEvkX10RS2VZM8PAm7ZCgqStyz zC2i3X+wSuKVbS/l2KUTU7B28Wy9zxnmNk7OpZBevjB0ENdnoEGmnLXb8nOVLUVsjEB/3P1tExf DUfJ59tY09NK45Eol/k8ojhpUYXyjOicx/3rHP8TaNeoTTcLSCGjVbEbOrgvByu7hsdlXFGWB7k jftZYevWPcmFlRT5wAQDbHQvD3/wTFBwwRW0XOrGf53qzx X-Received: by 2002:a05:6a00:3a29:b0:82f:48e:241c with SMTP id d2e1a72fcca58-82f0c2c8063mr4788741b3a.23.1775856982403; Fri, 10 Apr 2026 14:36:22 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:22 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:32 -0700 Subject: [PATCH v5 2/8] rust: cpumask: Add getters for globally defined cpumasks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-2-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=3267; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=nFa6XzGa0lobebiKoERWPKmf/z75UhcpF+sR2BSl5r8=; b=Ny8jZTTn2nKtMFz16cbqPqojORJoc5t7WJOTlcvhg5M1RO1POOoBpvS/N5eE8SvhWzbrrt1xm JYMPyeawyXqBUwkQqwAmSvTiNScQlLsQgAT/Ju6d08ZnAo5kY7HdGdW X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add getters for the global cpumasks documented in `include/linux/cpumask.h`, specifically: - cpu_possible_mask - cpu_online_mask - cpu_enabled_mask - cpu_present_mask - cpu_active_mask Acked-by: Viresh Kumar Suggested-by: Yury Norov Signed-off-by: Mitchell Levy --- rust/kernel/cpumask.rs | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index b74a3fccf4b4..2d61f1323837 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -80,6 +80,52 @@ pub unsafe fn from_raw<'a>(ptr: *const bindings::cpumask= ) -> &'a Self { unsafe { &*ptr.cast() } } =20 + /// Get a CPU mask representing possible CPUs; has bit `cpu` set iff c= pu is populatable + #[inline] + pub fn possible_cpus() -> &'static Self { + // SAFETY: `__cpu_possible_mask` is a valid global provided by the= kernel that lives + // forever. + unsafe { Cpumask::from_raw(&raw const bindings::__cpu_possible_mas= k) } + } + + /// Get a CPU mask representing online CPUs; has bit `cpu` set iff cpu= available to the + /// scheduler + #[inline] + pub fn online_cpus() -> &'static Self { + // SAFETY: `__cpu_online_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_online_mask` + // may change its value. + unsafe { Cpumask::from_raw(&raw const bindings::__cpu_online_mask)= } + } + + /// Get a CPU mask representing enabled CPUs; has bit `cpu` set iff cp= u can be brought online + #[inline] + pub fn enabled_cpus() -> &'static Self { + // SAFETY: `__cpu_enabled_mask` is a valid global provided by the = kernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_enabled_mask` + // may change its value. + unsafe { Cpumask::from_raw(&raw const bindings::__cpu_enabled_mask= ) } + } + + /// Get a CPU mask representing present CPUs; has bit `cpu` set iff cp= u is populated + #[inline] + pub fn present_cpus() -> &'static Self { + // SAFETY: `__cpu_present_mask` is a valid global provided by the = kernel that lives + // forever. Since we wrap the returned pointer in an `Opaque`, it'= s ok that + // `__cpu_present_mask` may change its value. + unsafe { Cpumask::from_raw(&raw const bindings::__cpu_present_mask= ) } + } + + /// Get a CPU mask representing active CPUs; has bit `cpu` set iff cpu= is available to + /// migration. + #[inline] + pub fn active_cpus() -> &'static Self { + // SAFETY: `__cpu_active_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_active_mask` + // may change its value. + unsafe { Cpumask::from_raw(&raw const bindings::__cpu_active_mask)= } + } + /// Obtain the raw `struct cpumask` pointer. pub fn as_raw(&self) -> *mut bindings::cpumask { let this: *const Self =3D self; --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 274FC387373 for ; Fri, 10 Apr 2026 21:36:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856985; cv=none; b=Yoa4m0mKzUdCk4zBrsqzO/yASBYA+slJaM2jGaLC6GI0qIFP0nrr0bPMhBIiHK4CKcmrcy5d3iBGqWFNE20hcwIJ2VrvDeXYz26V2YxZ8CmHYBbguRkbRNNRij8Qnf2kI7ZHUVhTBazJnemBguZJ14E1OYz/MKebZuJWEo6KlOE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856985; c=relaxed/simple; bh=LiqeN21hmlSzqeftoSl/GLCyPQV5RGEp+g7qmLBlTQE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mK0lEx4w44rYJiU4VvifVpufyCkQkoiSk7I7IKe9dvuBamZon4qmWpGuxHS5FXezS1lkUVIlVCBygYVJ1Sf+DprkaJFrlQFST852DVNaF/UoS1Fp9imkg6rTEAvwMh7rvlLkD/GK6M3yhjRPJSI2hYS6CNVwXZCDLYKT0lYsLLI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=pkbSpYJX; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pkbSpYJX" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-c7358a7a8d1so1701352a12.3 for ; Fri, 10 Apr 2026 14:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856983; x=1776461783; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=F+v45jw94JRi1dGGjjK8/w5xNCSncG1F0hGpE1jyeuw=; b=pkbSpYJXMWNg9DxqVm1BknGvL/jGfYBVdr9n6aAf+bAS67/PbwQK/5DbhiRMsMSSGl sTeskc/I+vbidZBKQ0CnRGCdvZOvbjy5TCDnVLeWMtZU70cGv6cL2gPNSIA/vok0BokC tHiddl6L9/ZZhmAY2UVxqgjuPm6aJhb0X/SLNeS4udtWFjlEp4u0vk02oSjid5Yt4HT4 OeThRKXEcZ2MCwlmpo2uZSwc5Rap0k8ZGs9trieYf25Ez/DJdT4HlOpY59HtYFU4urMN buuzb8aYkLjzYW1dpgH/uzQLbGoFsng9vJfq5C0AmFE6EpL059pLDVpt4amNPHIJVQxL IeyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856983; x=1776461783; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=F+v45jw94JRi1dGGjjK8/w5xNCSncG1F0hGpE1jyeuw=; b=Z6phA4G/RJpHx18cx33UruC+23GOXHpYzz1bKnT8MYsMkG613Tvcj9V35P9fbop1Ln 2cXQkjgJpPoq8iUzuvTcYWDLrQe9GCLqZhfHbKC0gpcy10ZyWW9jDeRDSHGbJ2d3STHW VNR3Zh/VcSpGP4HqovGofG5ob538DThj2Q/1Lt4piIM35l6Y7WrCgFOVipDNLCkUjLRg Pbw3fHDNIxa6CB6riBWWUXFwYQr2Rky6XyE/vC9BWnPXgLLedYF96LG5APZeLke2kMu0 ckGbp5sYDHpIYkeEZxjNR8r2kf1JXlqgMOC1UY7hlLSpPdMn63bJrHvshUSYOwV7hLbS kSdQ== X-Forwarded-Encrypted: i=1; AJvYcCUbo7Ix8fdI3+rVK+aanmDzXsiKwzE2kD+lpAVdiPkpdmYeng0kL859h4Kh0NhInBKjQT0I/xXrtBX+ayw=@vger.kernel.org X-Gm-Message-State: AOJu0YwCdqnqd3PSssyKgPF01vF62an9ewowCX4XlE2wJT4nrEb2r7qH WRTZvtcIhD0BXryiGAnN+PACGM4+t1lKWAB+zcsICLbNHlIDDVWihguK X-Gm-Gg: AeBDiesREK5Vwp/t4Cdl8lHDUvH37RGqdCwrfnI4/zYY+jpyEBPSz9M6T6XYuQj1nzy IXlZYEceocdbAH1YbeH9QDq1jerwsx1LKzal/Cpbr1+F6mxnHqBP5FKOVYIqYzFRU1wHGJlrSLj MwOz9mXszeu54DvYCA1lmQ9PPuIAFIopQ5sCEbRXit+aEKgGwQ+fGMKm6Fzw7vLlBm+BtL0FJoU YFpGcBBHqoyLYOFUFYs4OosUli3tK+pA6vi0knQh1Y71tS1yr/23DD0uZTeqQmEbqi5MH4ctKX8 d+roDewBddawix9it0+GR/N2/uISB2tVd2GzGiH2JqcZUrZNXx7xpPg42h54wuQ1AFD289OQYY1 eosbQoUzFro9ifp3fauPDPCjtCA847GYnz12qfdh+zHTkSS6HIGMTkTHHjnCXMDuBHc3EZr4n56 l3zCEqLMfEW3WgKaS3++if1+Nbb/QgmIfe2KzSjTnLS5p5 X-Received: by 2002:a05:6a00:2d1c:b0:82c:9897:70e0 with SMTP id d2e1a72fcca58-82f0c37f54bmr5055454b3a.47.1775856983472; Fri, 10 Apr 2026 14:36:23 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:22 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:33 -0700 Subject: [PATCH v5 3/8] rust: percpu: Add C bindings for per-CPU variable API Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-3-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=1610; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=LiqeN21hmlSzqeftoSl/GLCyPQV5RGEp+g7qmLBlTQE=; b=iUFnR3cUvYb+VI/DXA9Z3DypclDwajbL3MjM2j/sG8C8YCvKAUERXgFjlgI8fe1xS0FPR8yPj FaOyoJZfdzvBlVouoUriBmn4xf9cq/lvTRJ74Z9sbo/4C+GjPyslY/f X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add bindings necessary to implement a Rust per-CPU variable API, specifically per-CPU variable allocation and management of CPU preemption. Signed-off-by: Mitchell Levy --- rust/helpers/helpers.c | 2 ++ rust/helpers/percpu.c | 9 +++++++++ rust/helpers/preempt.c | 15 +++++++++++++++ 3 files changed, 26 insertions(+) diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c index e05c6e7e4abb..af661bb9be34 100644 --- a/rust/helpers/helpers.c +++ b/rust/helpers/helpers.c @@ -70,9 +70,11 @@ #include "of.c" #include "page.c" #include "pci.c" +#include "percpu.c" #include "pid_namespace.c" #include "platform.c" #include "poll.c" +#include "preempt.c" #include "processor.c" #include "property.c" #include "pwm.c" diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c new file mode 100644 index 000000000000..463186b8af9d --- /dev/null +++ b/rust/helpers/percpu.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +__rust_helper +void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) +{ + return __alloc_percpu(sz, align); +} diff --git a/rust/helpers/preempt.c b/rust/helpers/preempt.c new file mode 100644 index 000000000000..77dd5f10d8dc --- /dev/null +++ b/rust/helpers/preempt.c @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +__rust_helper +void rust_helper_preempt_disable(void) +{ + preempt_disable(); +} + +__rust_helper +void rust_helper_preempt_enable(void) +{ + preempt_enable(); +} --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A7993DE451 for ; Fri, 10 Apr 2026 21:36:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856987; cv=none; b=dTZg8M4BarlFs5AjpfsD/P0jt0OQ33pqpyuDTjtrgTTwfYQxyjNQdWvl36bTs6SX4BX7ByqTqCrS5NkOJuH0QDD2pjehhTGjvlHMYBb9zs/brcBzhI9NFtdDOSUfzv1+bXfZex3/+3ItM7QvwZqSwWD89EUFfSqo+mGQsmM9Sx8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856987; c=relaxed/simple; bh=6dJC9Y0VDh9MmHvE+5MiMl30etMsFxZyWjlJ75pGAVA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WJsVga4SLOzyqcE2P3OKT2c0K6dWBGNe6FLcEkMjMKRbtvQpvR/tYZxYhCP/MmdSxCTUwxeXQqTYr66jcML+UhuMpselYa3mzWOFDCBCXH7IQze6xMt2bo1PGJ5m9SR0dsVqNQc3z+lt0SPiNtoYhO0Pjjang+J07dvpbxFEICo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RgcjWh3o; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RgcjWh3o" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-82cef263bedso1246777b3a.0 for ; Fri, 10 Apr 2026 14:36:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856985; x=1776461785; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=7NpSBUaz10XvtM1OH6Hfiv98lv1+GV4Vrhyc5KaDfGM=; b=RgcjWh3o0icUmpR+KlLBP4A8yZlLS3uU3B2vXsNP3pd1k+4v6sEFmq/KTbZz1vS8xh 64CGjw3i1q5HEw9anatG+pmuVYo9PRkSzYCFy9AFbaYVI6guz1h9mYQiOuEe1eQ923++ mLaCiFhAeMP5KjKfP9PElHnLCTcdShBvfkam/tg2qkOe4QuQwNhCz2ESjj8FotZfEvbX EYtjke0ATyQh4PANvZO6Ahk3RwXYGyJXO2WDct65B6MiK4YuwTzN/0bLd7hBrnE6Y7In vVbSjVb5P+nVwKT3/IDnuN8ObDWRhsRhAz8PzPOrWxILfx2bT/y7k8liYiBSyRuWSl4X F7mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856985; x=1776461785; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=7NpSBUaz10XvtM1OH6Hfiv98lv1+GV4Vrhyc5KaDfGM=; b=rnIw+Lqt2LhXHCqUIH9YaUEaNoy2IfLkztUVgy8iDc93LRnejq07QJtRxYoNSZKA3/ KRK61FvV6MHiEclwABnFVdUhRqjM3KY3J4r0Tx4mN/BqszaUXBSJdbzJkkipsqplLJy/ GQZFOFRBWaw+KPLe3Z/Oe1h3A0+Zl+XOQk4ZbOrXbdxvFjA+ukFobXZH5yuBzc4iQjr2 POYo6j+AqoJ5W/AQwKFFI6knoHx4GfRmAeK17jOXBb9t4BB2qCRshn6ueZeyA8pcI/wH iPXMkq8K014oIoHhf04CRTWSrTP284nkoN6RFFXZ094KUYO78KAZaiTYswy+NFZzX+Ge XRxg== X-Forwarded-Encrypted: i=1; AJvYcCVLdqm29t6en8EzX9iVLy1SsTX0mpfjDeqMp7Jka42F14FBuNiC+AVgYpsD96YTNwDLbIm6HFhUEXAsSRY=@vger.kernel.org X-Gm-Message-State: AOJu0YyhCqQv7Gj7SKVEDDmoHPTtPvOh46PPdP9ANMStSU4obRDehdmb 37MrHK1zNSp3dSOfwgPXMlqJuAPYExHvDmlIt230bOYHvc0BAgDCx0cn X-Gm-Gg: AeBDiesxwyIt7ZDTzXOsOGqwOD84Y6dJSaHJm+seVM5WMMpQaIoeX7P26zbSLNPyszg su7fi4hnZJDgteOyChhElluJQLB3cILlKn/vZSXVacV7/am6O25L6T9XgBj3vArkFyMPIpVX6oT lMic16KJA905FgR2vYvE5ENPmB1cG1zb7yhq+UGxKx8Hr+cgx8k8hOSbwiNWXC64QannIhk+aLF oped+vh/ntugO7sJrsay0HXJMqUKEaejuKpg0F5IGy/Kor1WoTDV+CPfbl/OpZ3uFElcnim2HW6 KKWvXWBlKSDBf57ceTkm8fgqxZVNvOstzUb0RSEkovu45ehS4zPBaIT6/acdlkpoA5olG+B9aE/ wEiYujVr+kHUnG/UM6SGUOzSOIBhtnwc5ZMuCy37UH9ruvrlyrXqjbAjh4p/nUNSj/10ojEAo4w SnuUoJzqnu62JFCPia88c10kXyMI1fPGk5zXQIEaoZZFA2 X-Received: by 2002:aa7:838a:0:b0:82f:1aeb:fd60 with SMTP id d2e1a72fcca58-82f1aec00c3mr1557684b3a.16.1775856984445; Fri, 10 Apr 2026 14:36:24 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:24 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:34 -0700 Subject: [PATCH v5 4/8] rust: percpu: introduce a rust API for static per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-4-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=26404; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=6dJC9Y0VDh9MmHvE+5MiMl30etMsFxZyWjlJ75pGAVA=; b=o7d8Pz3fgF38hcNyFwdrEeLK1xxp+udnVZgwbjpV2TqsbjdeaV4de9aFiks1d6qZhstZS0s8x +L2I4cWbwwlBfHOni04owyA4SvAnsrsp5Ry91g2rQMEVykcmjQsTU/7 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Per-CPU variables are an important tool for reducing lock contention, especially in systems with many processors. They also provide a convenient way to handle data that are logically associated with a particular CPU (e.g., the currently running task). Therefore, add a Rust API to make use of statically-allocated per-CPU variables. Add a RAII `CpuGuard` type for disabling CPU preemption. Introduce unifying abstractions that can be reused for a Rust API for dynamically-allocated per-CPU variables. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- Using `rustc` 1.83.0, I can confirm that the symbols `__INIT_$id` are optimized out by the compiler and do not appear in the final `.ko` file when compiling `samples/rust/rust_percpu.rs` (introduced in a later patch in this series). --- rust/kernel/lib.rs | 3 + rust/kernel/percpu.rs | 259 ++++++++++++++++++++++++++++++++++++= ++++ rust/kernel/percpu/cpu_guard.rs | 48 ++++++++ rust/kernel/percpu/static_.rs | 218 +++++++++++++++++++++++++++++++++ 4 files changed, 528 insertions(+) diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs index 0fa9d820fe7c..b54aca69289c 100644 --- a/rust/kernel/lib.rs +++ b/rust/kernel/lib.rs @@ -97,6 +97,9 @@ pub mod page; #[cfg(CONFIG_PCI)] pub mod pci; +// Only x86_64 is supported by percpu for now +#[cfg(CONFIG_X86_64)] +pub mod percpu; pub mod pid_namespace; pub mod platform; pub mod prelude; diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs new file mode 100644 index 000000000000..615859d4daad --- /dev/null +++ b/rust/kernel/percpu.rs @@ -0,0 +1,259 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Per-CPU variables. +//! +//! See the [`crate::define_per_cpu!`] macro and the [`PerCpu`] trait. + +pub mod cpu_guard; +mod static_; + +#[doc(inline)] +pub use static_::*; + +use crate::{ + declare_extern_per_cpu, + percpu::cpu_guard::CpuGuard, + types::Opaque, // +}; + +use core::{ + arch::asm, + cell::{ + Cell, + RefCell, + UnsafeCell, // + }, + mem::MaybeUninit, // +}; + +use ffi::c_void; + +/// A per-CPU pointer; that is, an offset into the per-CPU area. +/// +/// Note that this type is NOT a smart pointer, it does not manage the all= ocation. +pub struct PerCpuPtr(*mut MaybeUninit); + +/// Represents exclusive access to the memory location pointed at by a par= ticular [`PerCpu`]. +pub struct PerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +/// Represents access to the memory location pointed at by a particular [`= PerCpu`] where the +/// type `T` manages access to the underlying memory to avoid aliasing tro= ubles. +/// +/// For example, `T` might be a [`Cell`] or [`RefCell`]. +pub struct CheckedPerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +impl PerCpuPtr { + /// Makes a new [`PerCpuPtr`] from a raw per-CPU pointer. + /// + /// Note that the returned [`PerCpuPtr`] is valid only as long as the = given pointer is. This + /// also requires that the allocation pointed to by `ptr` must be corr= ectly sized and aligned + /// to hold a `T`. + pub fn new(ptr: *mut MaybeUninit) -> Self { + Self(ptr) + } + + /// Get a [`&mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the current CPU + /// represented by `&self` + /// + /// # Safety + /// + /// The returned `&mut MaybeUninit` must follow Rust's aliasing rul= es. That is, no other + /// `&(mut) MaybeUninit` may exist that points to the same location= in memory. In practice, + /// this means that [`Self::get_ref`] and [`Self::get_mut_ref`] must n= ot be called on another + /// [`PerCpuPtr`] that is a copy/clone of `&self` for as long as th= e returned reference + /// lives. If the per-CPU variable represented by `&self` is available= to C, the caller of this + /// function must ensure that the C code does not modify the variable = for as long as this + /// reference lives. + /// + /// `self` must point to a live allocation correctly sized and aligned= to hold a `T`. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + #[allow(clippy::mut_from_ref)] // Safety requirements prevent aliasing= issues + pub unsafe fn get_mut_ref(&self) -> &mut MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract, + // and the safety requirements of this function ensure that the re= turned reference is + // exclusive. + unsafe { &mut *(self.get_ptr()) } + } + + /// Get a [`&MaybeUninit`](MaybeUninit) to the per-CPU variable on = the current CPU + /// represented by `&self` + /// + /// # Safety + /// + /// The returned [`&MaybeUninit`](MaybeUninit) must follow Rust's a= liasing rules. That is, + /// no `&mut MaybeUninit` may exist that points to the same locatio= n in memory. In practice, + /// this means that [`Self::get_mut_ref`] must not be called on anothe= r [`PerCpuPtr`] that + /// is a copy/clone of `&self` for as long as the returned reference l= ives. If the per-CPU + /// variable represented by `&self` is available to C, the caller of t= his function must ensure + /// that the C code does not modify the variable for as long as this r= eference lives. + /// + /// `self` must point to a live allocation correctly sized and aligned= to hold a `T`. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + pub unsafe fn get_ref(&self) -> &MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract. + // The safety requirements of this function ensure that the return= ed reference isn't + // aliased by a `&mut MaybeUninit`. + unsafe { &*self.get_ptr() } + } + + /// Get a [`*mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the current CPU + /// represented by `&self`. Note that if CPU preemption is not disable= d before calling this + /// function, use of the returned pointer may cause a data race withou= t some other + /// synchronization mechanism. Buyer beware! + pub fn get_ptr(&self) -> *mut MaybeUninit { + let this_cpu_off_pcpu =3D ExternStaticPerCpuSymbol::ptr(&raw const= this_cpu_off); + let mut this_cpu_area: *mut c_void; + // SAFETY: gs + this_cpu_off_pcpu is guaranteed to be a valid poin= ter because `gs` points + // to the per-CPU area and this_cpu_off_pcpu is a valid per-CPU al= location. + unsafe { + asm!( + "mov {out}, gs:[{off_val}]", + off_val =3D in(reg) this_cpu_off_pcpu.0, + out =3D out(reg) this_cpu_area, + ) + }; + + // This_cpu_area + self.0 is guaranteed to be a valid pointer by t= he per-CPU subsystem and + // the invariant that self.0 is a valid offset into the per-CPU ar= ea. + (this_cpu_area).wrapping_add(self.0 as usize).cast() + } +} + +// SAFETY: Sending a [`PerCpuPtr`] to another thread is safe because as= soon as it's sent, the +// pointer is logically referring to a different place in memory in the ot= her CPU's per-CPU area. +// In particular, this means that there are no restrictions on the type `T= `. +unsafe impl Send for PerCpuPtr {} + +// SAFETY: Two threads concurrently making use of a [`PerCpuPtr`] will = each see the `T` in their +// own per-CPU area, so there's no potential for a data race (regardless o= f whether `T` is itself +// `Sync`). +unsafe impl Sync for PerCpuPtr {} + +impl Clone for PerCpuPtr { + fn clone(&self) -> Self { + *self + } +} + +/// [`PerCpuPtr`] is just a wrapper around a pointer. +impl Copy for PerCpuPtr {} + +/// A trait representing a per-CPU variable. +/// +/// This is implemented for [`StaticPerCpu`]. The main usage of this tr= ait is to call +/// [`Self::get_mut`] to get a [`PerCpuToken`] that can be used to access = the underlying per-CPU +/// variable. +/// +/// See [`PerCpuToken::with`]. +pub trait PerCpu { + /// Produces a token, asserting that the holder has exclusive access t= o the underlying memory + /// pointed to by `self` + /// + /// # Safety + /// + /// No other [`PerCpuToken`] or [`CheckedPerCpuToken`] may exist on th= e current CPU (which is a + /// sensible notion, since we keep a [`CpuGuard`] around) that is deri= ved from the same + /// [`PerCpu`] or a clone thereof. + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T>; +} + +/// A marker trait for types that are interior mutable. Types that impleme= nt this trait can be used +/// to create "checked" per-CPU variables. See [`CheckedPerCpu`]. +pub trait InteriorMutable {} + +impl InteriorMutable for Cell {} +impl InteriorMutable for RefCell {} +impl InteriorMutable for UnsafeCell {} +impl InteriorMutable for Opaque {} + +/// A trait representing a per-CPU variable that is usable via a `&T`. +/// +/// The unsafety of [`PerCpu`] stems from the fact that the holder of a= [`PerCpuToken`] can use +/// it to get a `&mut T` to the underlying per-CPU variable. This is probl= ematic because the +/// existence of aliasing `&mut T` is undefined behavior in Rust. This typ= e avoids that issue by +/// only allowing access via a `&T`, with the tradeoff that then `T` must = be interior mutable or +/// the underlying per-CPU variable must be a constant for the lifetime of= any corresponding +/// [`CheckedPerCpuToken`]. +/// +/// Currently, only the case where `T` is interior mutable has first-class= support, though a custom +/// implementation of [`PerCpu`]/[`CheckedPerCpu`] could be created = for the const case. +pub trait CheckedPerCpu: PerCpu { + /// Produces a token via which the holder can access the underlying pe= r-CPU variable. + fn get(&self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T>; +} + +impl<'a, T> PerCpuToken<'a, T> { + /// Asserts that the holder has exclusive access to the initialized un= derlying memory pointed + /// to by `ptr` on the current CPU, producing a token. + /// + /// # Safety + /// + /// No other [`PerCpuToken`] or [`CheckedPerCpuToken`] may exist on th= e current CPU (which is a + /// sensible notion, since we keep a `CpuGuard` around) that uses the = same [`PerCpuPtr`]. + /// + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed; `ptr` must be valid + /// (that is, pointing at a live per-CPU allocation correctly sized an= d aligned to hold a `T`) + /// for `'a`. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> PerCpuTok= en<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&mut T` that points at the unde= rlying per-CPU variable + /// that `&mut self` represents. Forwards the return of `func`. + pub fn with(&mut self, func: U) -> V + where + U: FnOnce(&mut T) -> V, + { + // SAFETY: The existence of a PerCpuToken means that the requireme= nts for get_mut_ref are + // satisfied. Likewise, the requirements for assume_init_mut are s= atisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_mut_ref().assume_init_mut() }) + } +} + +impl<'a, T> CheckedPerCpuToken<'a, T> { + /// Asserts that the memory pointed to by `ptr` is initialized on the = current CPU, producing a + /// token. + /// + /// # Safety + /// + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed; `ptr` must be valid + /// (that is, pointing at a live per-CPU allocation correctly sized an= d aligned to hold a `T`) + /// for `'a`. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> CheckedPe= rCpuToken<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&T` that points at the underlyi= ng per-CPU variable that + /// `&mut self` represents. Forwards the return of `func`. + pub fn with(&self, func: U) -> V + where + U: FnOnce(&T) -> V, + { + // SAFETY: The existence of a CheckedPerCpuToken means that the re= quirements for get_ref + // are satisfied. Likewise, the requirements for assume_init_ref a= re satisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_ref().assume_init_ref() }) + } +} + +declare_extern_per_cpu!(this_cpu_off: *mut c_void); diff --git a/rust/kernel/percpu/cpu_guard.rs b/rust/kernel/percpu/cpu_guard= .rs new file mode 100644 index 000000000000..aba776b64bbc --- /dev/null +++ b/rust/kernel/percpu/cpu_guard.rs @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Contains abstractions for disabling CPU preemption. See [`CpuGuard`]. + +/// A RAII guard for `bindings::preempt_disable` and `bindings::preempt_en= able`. +/// +/// Guarantees preemption is disabled for as long as this object exists. +pub struct CpuGuard { + // Don't make one without using new() + _phantom: (), + /// [`CpuGuard`] is not [`Send`]: moving it to another CPU would break= its guarantees and also + /// leave the CPU it was created on with preemption disabled indefinit= ely. + /// + /// It is also not [`Sync`]: preemption being disabled on one CPU is n= ot relevant to other + /// CPUs, so we shouldn't be able to share a `&CpuGuard` between threa= ds. + _not_send_sync: core::marker::PhantomData<*const ()>, +} + +impl CpuGuard { + /// Create a new [`CpuGuard`]. Disables preemption for its lifetime. + #[inline] + pub fn new() -> Self { + // SAFETY: There are no preconditions required to call preempt_dis= able + unsafe { + bindings::preempt_disable(); + } + CpuGuard { + _phantom: (), + _not_send_sync: core::marker::PhantomData, + } + } +} + +impl Default for CpuGuard { + #[inline] + fn default() -> Self { + Self::new() + } +} + +impl Drop for CpuGuard { + #[inline] + fn drop(&mut self) { + // SAFETY: There are no preconditions required to call preempt_ena= ble + unsafe { + bindings::preempt_enable(); + } + } +} diff --git a/rust/kernel/percpu/static_.rs b/rust/kernel/percpu/static_.rs new file mode 100644 index 000000000000..70aca55295ca --- /dev/null +++ b/rust/kernel/percpu/static_.rs @@ -0,0 +1,218 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Statically allocated per-CPU variables. + +use super::*; + +/// A wrapper used for declaring static per-CPU variables. +/// +/// These symbols are "virtual" in that the linker uses them to generate o= ffsets into each CPU's +/// per-CPU area, but shouldn't be read from/written to directly. The fact= that the statics are +/// immutable prevents them being written to (generally), this struct havi= ng _val be non-public +/// prevents reading from them. +/// +/// The end-user of the per-CPU API should make use of the [`crate::define= _per_cpu!`] macro instead +/// of declaring variables of this type directly. All instances of this ty= pe must be `static` and +/// `#[link_section =3D ".data..percpu"]` (which the macro handles). +#[repr(transparent)] +pub struct StaticPerCpuSymbol { + _val: T, // generate a correctly sized type +} + +/// A wrapper for per-CPU variables declared in C. +/// +/// As with [`StaticPerCpuSymbol`], this type should not be used directly.= Instead, use the +/// [`crate::declare_extern_per_cpu!`] macro. +#[repr(transparent)] +pub struct ExternStaticPerCpuSymbol(StaticPerCpuSymbol); + +/// Holds a statically-allocated per-CPU variable. +#[derive(Clone)] +pub struct StaticPerCpu(pub(super) PerCpuPtr); + +impl StaticPerCpuSymbol { + /// Removes the per-CPU marker type from a static per-CPU symbol. + /// + /// To declare a static per-CPU variable, Rust requires that the varia= ble be `Sync`. However, + /// in the case of per-CPU variables, this is silly. See [`PerCpuSyncM= arkerType`]. Thus, our + /// statics are actually of type `StaticPerCpuSymbol>`, and this + /// produces a `StaticPerCpuSymbol`. + pub fn forward(ptr: *const Self) -> *const StaticPerCpuSymbol { + ptr.cast() + } +} + +impl ExternStaticPerCpuSymbol { + /// Gets a [`PerCpuPtr`] to the per-CPU variable. + /// + /// Usage of this [`PerCpuPtr`] must be very careful to keep in mind w= hat's happening in C. In + /// particular, the C code cannot modify the variable while any refere= nce derived from the + /// returned pointer is live. + pub fn ptr(ptr: *const Self) -> PerCpuPtr { + // These casts are OK because, ExternStaticPerCpuSymbol, StaticPer= CpuSymbol, and + // MaybeUninit are transparent, everything is just a `T` from a me= mory layout perspective. + // Casting to `mut` is OK because any usage of the returned pointe= r must satisfy the + // soundness requirements for using it as such. + PerCpuPtr::new(ptr.cast::>().cast_mut().cast= ()) + } +} + +impl StaticPerCpu { + /// Creates a [`StaticPerCpu`] from a [`StaticPerCpuSymbol`]. + /// + /// Users should probably declare static per-CPU variables with [`crat= e::define_per_cpu!`] and + /// then get instances of [`StaticPerCpu`] using [`crate::get_static_p= er_cpu!`]. + /// + /// # Safety + /// You should probably be using [`crate::get_static_per_cpu!`] instea= d. + /// + /// `ptr` must be a valid offset into the per-CPU area, sized and alig= ned for access to a `T`; + /// typically this is the address of a static in the `.data..percpu` s= ection, which is managed + /// by the per-CPU subsystem. + pub unsafe fn new(ptr: *const StaticPerCpuSymbol) -> StaticPerCpu { + let pcpu_ptr =3D PerCpuPtr::new(ptr.cast_mut().cast()); + Self(pcpu_ptr) + } +} + +impl PerCpu for StaticPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: + // 1. By the requirements of `PerCpu::get_mut`, no other `[Checked= ]PerCpuToken` exists on + // the current CPU. + // 2. The per-CPU subsystem guarantees that each CPU's instance of= a statically allocated + // variable begins with a copy of the contents of the correspon= ding symbol in + // `.data..percpu` and is therefore initialized. + // 3. The per-CPU subsystem guarantees that `self.0` is correctly = aligned for a `T`. + // 4. The per-CPU subsystem guarantees that `self.0` lives forever= as a per-CPU allocation, + // and that this allocation is the proper size for a `T`. + unsafe { PerCpuToken::new(guard, &self.0) } + } +} + +impl CheckedPerCpu for StaticPerCpu { + fn get(&self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: + // 1. The per-CPU subsystem guarantees that each CPU's instance of= a statically allocated + // variable begins with a copy of the contents of the correspon= ding symbol in + // `.data..percpu` and is therefore initialized. + // 2. The per-CPU subsystem guarantees that `self.0` is correctly = aligned for a `T`. + // 3. The per-CPU subsystem guarantees that `self.0` lives forever= as a per-CPU allocation, + // and that this allocation is the proper size for a `T`. + unsafe { CheckedPerCpuToken::new(guard, &self.0) } + } +} + +/// Gets a [`StaticPerCpu`] from a symbol declared with [`crate::define= _per_cpu!`]. +/// +/// # Arguments +/// +/// * `ident` - The identifier declared +#[macro_export] +macro_rules! get_static_per_cpu { + ($id:ident) =3D> { + unsafe { + // SAFETY: The signature of `StaticPerCpuSymbol::forward` guar= antees that `&raw const + // $id` is a `*const StaticPerCpuSymbol>` if the macro + // invocation compiles. + // + // Values of type `StaticPerCpuSymbol` must be created via = `define_per_cpu`, and so + // the per-CPU subsystem guarantees that the requirements for = `StaticPerCpu::new` are + // satisfied. + $crate::percpu::StaticPerCpu::new($crate::percpu::StaticPerCpu= Symbol::forward( + &raw const $id, + )) + } + }; +} + +/// Declares an [`ExternStaticPerCpuSymbol`] corresponding to a per-CPU va= riable defined in C. +#[macro_export] +macro_rules! declare_extern_per_cpu { + ($id:ident: $ty:ty) =3D> { + extern "C" { + static $id: ExternStaticPerCpuSymbol<$ty>; + } + }; +} + +/// A trait implemented by static per-CPU wrapper types. +/// +/// Internally, static per-CPU variables are declared as `static` variable= s. However, Rust doesn't +/// allow you to declare statics of a `!Sync` type. This trait is implemen= ted by the marker type +/// that is declared as `Sync` and used to declare the static. See [`crate= ::define_per_cpu!`] for +/// the gory details. +/// +/// # Safety +/// +/// Implementations must be `#[repr(transparent)]` wrappers around an `Inn= er`. This trait should +/// only be used from the [`crate::define_per_cpu!`] macro. +pub unsafe trait PerCpuSyncMarkerType { + /// The "true" type of the per-CPU variable. It must always be valid t= o cast a `*const Self` to + /// a `*const Self::Inner`, which is guaranteed by this trait's safety= requirement. + type Inner; +} + +/// Declares and initializes a static per-CPU variable. +/// +/// Analogous to the C `DEFINE_PER_CPU` macro. +/// +/// See also [`crate::get_static_per_cpu!`] for how to get a [`StaticPerCp= u`] from this +/// declaration. +/// +/// # Example +/// ``` +/// use kernel::define_per_cpu; +/// +/// define_per_cpu!(pub MY_PERCPU: u64 =3D 0); +/// ``` +#[macro_export] +macro_rules! define_per_cpu { + ($vis:vis $id:ident: $ty:ty =3D $expr:expr) =3D> { + $crate::macros::paste! { + // We might want to have a per-CPU variable that doesn't imple= ment `Sync` (not paying + // sync overhead costs is part of the point), but Rust won't l= et us declare a static of + // a `!Sync` type. Of course, we don't actually have any synch= ronization issues, since + // each CPU will see its own copy of the variable, so we cheat= a little bit and tell + // Rust it's fine. + #[doc(hidden)] + #[allow(non_camel_case_types)] + #[repr(transparent)] // It needs to be the same size as $ty + struct [<__PRIVATE_TYPE_ $id>]($ty); + + // SAFETY: [<__PRIVATE_TYPE_ $id>] is a `#[repr(transparent)]`= wrapper around a `$ty`. + unsafe impl $crate::percpu::PerCpuSyncMarkerType for [<__PRIVA= TE_TYPE_ $id>] { + type Inner =3D $ty; + } + + impl [<__PRIVATE_TYPE_ $id>] { + #[doc(hidden)] + const fn new(val: $ty) -> Self { + Self(val) + } + } + + // Expand $expr outside of the unsafe block to avoid silently = allowing unsafe code to be + // used without a user-facing unsafe block + #[doc(hidden)] + static [<__INIT_ $id>]: [<__PRIVATE_TYPE_ $id>] =3D [<__PRIVAT= E_TYPE_ $id>]::new($expr); + + // SAFETY: This type will ONLY ever be used to declare a `Stat= icPerCpuSymbol` + // (which we then only ever use as input to `&raw`). Reading f= rom the symbol is + // already UB, so we won't ever actually have any variables of= this type where + // synchronization is a concern. + #[doc(hidden)] + unsafe impl Sync for [<__PRIVATE_TYPE_ $id>] {} + + // SAFETY: StaticPerCpuSymbol is #[repr(transparent)], so w= e can freely convert from + // [<__PRIVATE_TYPE_ $id>], which is also `#[repr(transparent)= ]` (i.e., everything is + // just a `$ty` from a memory layout perspective). + #[link_section =3D ".data..percpu"] + $vis static $id: $crate::percpu::StaticPerCpuSymbol<[<__PRIVAT= E_TYPE_ $id>]> =3D unsafe { + core::mem::transmute_copy::< + [<__PRIVATE_TYPE_ $id>], + $crate::percpu::StaticPerCpuSymbol<[<__PRIVATE_TYPE_ $= id>]> + >(&[<__INIT_ $id>]) + }; + } + }; +} --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 701F53DEFE5 for ; Fri, 10 Apr 2026 21:36:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856988; cv=none; b=jfi/JHoRGa0TMULRU5GjbRAxC+PlPlfMCmD4T79Wf1ELLLzvMkmZwtzo7OyIrdMHmdc2XLcRGat4J3znXwkqFtMivzjhnao5ZgPuU9h0TnGOpTDFg2uJX4gzH4Da4olp4xiMQ6yq2BXXrJniSfasoT3TrmeGGUMq+k/ONkwibLM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856988; c=relaxed/simple; bh=BjduRcn3ssQj78fY+L89MJz6yy9e4tAyqjivAYqoheo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Y0Qm3ZIibusLZYV5Gb+gNiWvlbUbYlqNMwr/+AolsLidgv094SG4lIx4gc2QmTzlzRZAgzN6G2l03ACqmHRjO0d5Yb1ehSgXIACir2p64ZJ0l3RfXGr3RBCL5SGpqj19nx1W4XfmRb9jS0WJc7WGXmUJ5OAit6wYMpJMww9ionk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LIvnoNlv; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LIvnoNlv" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-82cec955160so1835314b3a.0 for ; Fri, 10 Apr 2026 14:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856986; x=1776461786; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=sjXgZakwtxdl0VN9G7aNOYuETb1QANXt4gyFlj+iho8=; b=LIvnoNlvWancibu+55M3E6sX2holU0jHxI+Vz0Nov0TfsIFK7NOR4DhF8f2Oc2joe/ yt0brppY/EqXG5NvLXeOGzPIYKodIgqz3YACf/umCpDVUSPK6nP4GQxiM7gC34DtnO/N yQSTSP/uOE2Vada16duXmxA9J/2OJ5+OO70Q8bMTyePn4DGjtyYP+Ias1PnTtuerASEP 9WBivvoykaT4rv+EqaXq/n3h+VLpQ+ou3784ijTeu/w83CPbahYuo/BLcEL/9JjAjK+P 4cf9SQFzUpn/saCvNWhaia8mg+PcMiI2jpSZrQg3Trz7mZUQlTWSgNrZMfSO4dl6ibdU hbPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856986; x=1776461786; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=sjXgZakwtxdl0VN9G7aNOYuETb1QANXt4gyFlj+iho8=; b=ZNCLIwJlsoO55LVT4/MSw9T3pc8pGbXnDG7TwTXOpLHdGihw2Ang5M7ZiAtUqkn7c7 bIqrBCt9oqwZMiX3iBwdnuQS4Yp84AsK7qgJH13e/LslUmxII0cNrpfRl2TWSRXpcLdl VaQVZW0R8qETtJVg9HBudc3W5ur9CVWi7Gzh2p2uKaVrqd3Yvre7iMAW+EDFkvaB7CGD XdHVttHZoYZPHOBuzbO3LFAcAEbAQ0tnjmXTlc+Q6fIuBqqJ0sAS4po0XzYhlSoKeXIF Dunogh+ycBfdLzCH2Yltzoxx6n65trZjXPTFdh0VPPSi/oBNyXH/lKlvBWZVRY5Nfaoa QGBA== X-Forwarded-Encrypted: i=1; AJvYcCXDgbROGZoLeDtieuU5QXtUm8L3XFgGose5hKq/UNZogyhiMIzxM+1PvTNDeJ7EBfmDoFQ5wnPqb9En/qk=@vger.kernel.org X-Gm-Message-State: AOJu0Yw2dOojqULP9sZKVdPQwYesGohr8Q+9sQUVLxBxjURa3hYpDr6P xQZuEqidmhtyak8sLxJwkVkp0u35WYQ2+spCv5fudQptdNnKHp5iqR/S X-Gm-Gg: AeBDiesF8YdkfnGzNDgVQTG52G5/tMQiTt/AqEr/OWr43niYQayXsWN5ljs1B1VhkAr 1wE/13UhURZYiRqDrPHirBstZ1H2fjPSU4u1eHDNppQHWRZiXkWh8Xhj/1D25wkZbFOTzI+aWyw ZqWGoGtTIsnAEq40hxHsxuNdHarQEJ3pnNYX+d63FQDAxmDbNYQP7jRob3nirbCKo0dqAIv8x0i KfpAGYaW4/uxLKE3NgGb2rkMCWWGj5ahScAm+KFeEaW3hehNcGCAThogtOFsNAsaykL2YCAeb0k DHYvWAH5xbk9AV7rc+DZA0swOvWvvYLg7teLNLzO7xWEiKTerhnXxRtaKS/6hqPB/Lzw/Hbllv5 sGBwvp764lvNOuPumMksA2OEP0zUs0jNss4wSEe1hrvae4nZ3s0ANdwpSKzWEH4zGeFRXTspq1+ ir6xfs9OvQ3VOifXbCKw82i/g03jEmwd8nU7RXi+Wq9HFk X-Received: by 2002:a05:6a00:94ed:b0:82c:d7e7:2f8a with SMTP id d2e1a72fcca58-82f0c2cc05emr5545403b3a.39.1775856985480; Fri, 10 Apr 2026 14:36:25 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:25 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:35 -0700 Subject: [PATCH v5 5/8] rust: percpu: introduce a rust API for dynamic per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-5-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=13283; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=BjduRcn3ssQj78fY+L89MJz6yy9e4tAyqjivAYqoheo=; b=vPb/14JfjlCNy2Aq44p4Ja3wajyFXY8LTQ+t/jJAZ8E1v/jGU8q4uLMF2F0plaErQ17GRtHuV 5wCbzE4dktkBmGXInmVsi80MCVhObpuRvyc7xVz4vPr/KoprywWlv08 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Dynamically allocated per-CPU variables are core to many of the use-cases of per-CPU variables (e.g., ref counting). Add support for them using the core `PerCpuPtr` primitive, implementing the `PerCpu` and `CheckedPerCpu` traits. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 12 +++ rust/kernel/percpu.rs | 30 +++++- rust/kernel/percpu/dynamic.rs | 217 ++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 255 insertions(+), 4 deletions(-) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index 463186b8af9d..3b2f69a96c66 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -7,3 +7,15 @@ void __percpu *rust_helper_alloc_percpu(size_t sz, size_t = align) { return __alloc_percpu(sz, align); } + +__rust_helper +void *rust_helper_per_cpu_ptr(void __percpu *ptr, unsigned int cpu) +{ + return per_cpu_ptr(ptr, cpu); +} + +__rust_helper +void rust_helper_on_each_cpu(smp_call_func_t func, void *info, int wait) +{ + on_each_cpu(func, info, wait); +} diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 615859d4daad..0ec1245038bb 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -1,15 +1,20 @@ // SPDX-License-Identifier: GPL-2.0 //! Per-CPU variables. //! -//! See the [`crate::define_per_cpu!`] macro and the [`PerCpu`] trait. +//! See the [`crate::define_per_cpu!`] macro, the [`DynamicPerCpu`] type, = and the [`PerCpu`] +//! trait. =20 pub mod cpu_guard; +mod dynamic; mod static_; =20 +#[doc(inline)] +pub use dynamic::*; #[doc(inline)] pub use static_::*; =20 use crate::{ + cpu::CpuId, declare_extern_per_cpu, percpu::cpu_guard::CpuGuard, types::Opaque, // @@ -132,6 +137,23 @@ pub fn get_ptr(&self) -> *mut MaybeUninit { // the invariant that self.0 is a valid offset into the per-CPU ar= ea. (this_cpu_area).wrapping_add(self.0 as usize).cast() } + + /// Get a [`*mut MaybeUninit`](MaybeUninit) to the per-CPU variable= on the CPU represented + /// by `cpu`. Note that without some kind of synchronization, use of t= he returned pointer may + /// cause a data race. It is the caller's responsibility to use the re= turned pointer in a + /// reasonable way. + /// + /// # Returns + /// - The returned pointer is valid only if `self` is (that is, it poi= nts to a live allocation + /// correctly sized and aligned to hold a `T`) + /// - The returned pointer is valid only if the bit corresponding to `= cpu` is set in + /// [`kernel::cpumask::Cpumask::possible_cpus()`]. + pub fn get_remote_ptr(&self, cpu: CpuId) -> *mut MaybeUninit { + // SAFETY: `bindings::per_cpu_ptr` is just doing pointer arithmeti= c. The returned pointer + // may not be valid (under the conditions specified in this functi= on's documentation), but + // the act of producing the pointer is safe. + unsafe { bindings::per_cpu_ptr(self.0.cast(), cpu.as_u32()) }.cast= () + } } =20 // SAFETY: Sending a [`PerCpuPtr`] to another thread is safe because as= soon as it's sent, the @@ -155,9 +177,9 @@ impl Copy for PerCpuPtr {} =20 /// A trait representing a per-CPU variable. /// -/// This is implemented for [`StaticPerCpu`]. The main usage of this tr= ait is to call -/// [`Self::get_mut`] to get a [`PerCpuToken`] that can be used to access = the underlying per-CPU -/// variable. +/// This is implemented for both [`StaticPerCpu`] and [`DynamicPerCpu`]. The main usage of +/// this trait is to call [`Self::get_mut`] to get a [`PerCpuToken`] that = can be used to access the +/// underlying per-CPU variable. /// /// See [`PerCpuToken::with`]. pub trait PerCpu { diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs new file mode 100644 index 000000000000..40514704b3d0 --- /dev/null +++ b/rust/kernel/percpu/dynamic.rs @@ -0,0 +1,217 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Dynamically allocated per-CPU variables. + +use super::*; + +use crate::{ + alloc::Flags, + bindings::{ + alloc_percpu, + free_percpu, // + }, + cpumask::Cpumask, + prelude::*, + sync::Arc, // +}; + +use core::mem::{ + align_of, + size_of, + MaybeUninit, // +}; + +/// Represents a dynamic allocation of a per-CPU variable via `alloc_percp= u`. Calls `free_percpu` +/// when dropped. +/// +/// # Contents +/// Note that the allocated memory need not be initialized, and this type = does not track when/if +/// the memory location on any particular CPU has been initialized. This m= eans that it cannot tell +/// whether it should drop the *contents* of the allocation when it is dro= pped. It is up to the +/// user to do this via something like [`core::ptr::drop_in_place`]. +pub struct PerCpuAllocation(PerCpuPtr); + +impl PerCpuAllocation { + /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, + /// initially filled with the zero value for `T`. + /// + /// Returns [`None`] under the same circumstances the C function `allo= c_percpu` returns `NULL`. + pub fn new_zero() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call `alloc_percpu`. The pointe= r is sized/aligned for a + // `T`, and `MaybeUninit` is `#[repr(transparent)]` (and th= us same size/align), so + // we can cast this `*mut c_void` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + // alloc_percpu returns zero'ed memory + Some(Self(PerCpuPtr::new(ptr))) + } +} + +impl PerCpuAllocation { + /// Makes a per-CPU allocation sized and aligned to hold a `T`. + /// + /// Returns [`None`] under the same circumstances the C function `allo= c_percpu` returns `NULL`. + pub fn new_uninit() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call `alloc_percpu`. The pointe= r is sized/aligned for a + // `T`, and `MaybeUninit` is `#[repr(transparent)]` (and th= us same size/align), so + // we can cast this `*mut c_void` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + Some(Self(PerCpuPtr::new(ptr))) + } +} + +impl Drop for PerCpuAllocation { + fn drop(&mut self) { + // SAFETY: self.0.0 was returned by alloc_percpu, and so was a val= id pointer into + // the percpu area, and has remained valid by the invariants of Pe= rCpuAllocation. + unsafe { free_percpu(self.0 .0.cast()) } + } +} + +/// Holds a dynamically-allocated per-CPU variable. +/// +/// # Examples +/// ``` +/// # use kernel::prelude::*; +/// # use kernel::percpu::*; +/// # use kernel::percpu::cpu_guard::CpuGuard; +/// +/// let mut pcpu: DynamicPerCpu =3D +/// DynamicPerCpu::new_zero(GFP_KERNEL).expect("failed to allocate per-C= PU variable"); +/// { +/// let _guard =3D CpuGuard::new(); +/// // SAFETY: No other `pcpu` reference +/// unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val| *val =3D 42); +/// // SAFETY: No other `pcpu` reference +/// unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val| assert!(*val =3D= =3D 42)); +/// } +/// ``` +#[derive(Clone)] +pub struct DynamicPerCpu { + // INVARIANT: `alloc` is `Some` unless this object is in the process o= f being dropped. + // INVARIANT: The allocation held by `alloc` is sized and aligned for = a `T`. + // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by the alloc is + // initialized. + alloc: Option>>, +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `flags` - [`Flags`] used to allocate an [`Arc`] that keeps track= of the underlying + /// [`PerCpuAllocation`]. + pub fn new_zero(flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: Some(arc) }) + } +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `val` - The initial value of the per-CPU variable on all CPUs. + /// * `flags` - Flags used to allocate an [`Arc`] that keeps track of = the underlying + /// [`PerCpuAllocation`]. + pub fn new_with(val: &T, flags: Flags) -> Option { + Self::new_from(|_| val.clone(), flags) + } +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `initer` - A function that takes a `CpuId` and returns the initi= al value of the per-CPU + /// variable on the corresponding CPU. Called once for each possible= CPU (i.e., + /// `Cpumask::possible_cpus().weight()` times). + /// * `flags` - Flags used to allocate an [`Arc`] that keeps track of = the underlying + /// [`PerCpuAllocation`]. + pub fn new_from(mut initer: impl FnMut(CpuId) -> T, flags: Flags) -> O= ption { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_uninit()?; + let ptr =3D alloc.0; + + let arc =3D Arc::new(alloc, flags).ok()?; + + for cpu in Cpumask::possible_cpus().iter() { + let remote_ptr =3D ptr.get_remote_ptr(cpu); + // SAFETY: `remote_ptr` is valid because `ptr` points to a liv= e allocation and `cpu` + // appears in `Cpumask::possible_cpus()`. + // + // Each CPU's slot corresponding to `ptr` is currently uniniti= alized, and no one else + // has a reference to it. Therefore, we can freely write to it= without worrying about + // the need to drop what was there or whether we're racing wit= h someone else. + unsafe { + (*remote_ptr).write(initer(cpu)); + } + } + + Some(Self { alloc: Some(arc) }) + } +} + +impl PerCpu for DynamicPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: + // 1. Invariants of this type assure that `alloc` is `Some`. + // 2. The requirements of `PerCpu::get_mut` ensure that no other `= [Checked]PerCpuToken` + // exists on the current CPU. + // 3. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are + // initialized on each CPU. + // 4. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is + // live. + // 5. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for + // a `T`. + unsafe { PerCpuToken::new(guard, &self.alloc.as_ref().unwrap_unche= cked().0) } + } +} + +impl CheckedPerCpu for DynamicPerCpu { + fn get(&self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: + // 1. Invariants of this type assure that `alloc` is `Some`. + // 2. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are + // initialized on each CPU. + // 3. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is + // live. + // 4. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for + // a `T`. + unsafe { CheckedPerCpuToken::new(guard, &self.alloc.as_ref().unwra= p_unchecked().0) } + } +} + +impl Drop for DynamicPerCpu { + fn drop(&mut self) { + // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. + let alloc =3D unsafe { self.alloc.take().unwrap_unchecked() }; + if let Some(unique_alloc) =3D Arc::into_unique_or_drop(alloc) { + let ptr =3D unique_alloc.0; + for cpu in Cpumask::possible_cpus().iter() { + let remote_ptr =3D ptr.get_remote_ptr(cpu); + // SAFETY: `remote_ptr` is valid because the allocation it= points to is still live, + // `cpu` appears in `Cpumask::possible_cpus()`, and the or= iginal allocation was + // sized and aligned for a `T`. + // + // This type's invariant ensures that the memory location = in each CPU's per-CPU + // area pointed at by `alloc.0` has been initialized. We h= ave a `UniqueArc`, so we + // know we're the only ones with a reference to the memory= . These two facts + // together satisfy the requirements for `assume_init_drop= `. + unsafe { + (*remote_ptr).assume_init_drop(); + } + } + } + } +} --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EB043DFC79 for ; Fri, 10 Apr 2026 21:36:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856989; cv=none; b=mfOkmE66SwVcvCfMzMNu93+pW6ITBMFi4512FRbvTfiqBF+Ceh+WEQTDU4UQVqHQbCVjDfZWv6VsQEko7Y6qD6b8X/hVxFND9lDB6vF+NQdO4okru4BtkaphLKEUGcS34QpqzY8FzoCz+g6CkvSrg74TjWLd9hKTLBTwQVGgsB8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856989; c=relaxed/simple; bh=MZsr3UdiERwV1f/QQfUp88/S78aGt9zYSriryeD+vLw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=sujz4lxFfmzuueHn99/+m1LzLsKy7jXBqO+Djv7T/KQQThkSR97hmLzTYZLEKc5+y0ks3t1kOEYSGHQyxgNJL9/jBqIPV24ktGi2Zyl2HKuwaQWi4qzFzhSCDd4/zp+SpGQctuPxGGFgbtAnY2AqeMJKk6sf+H4gD9AMD/DFsPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=smlNTIgN; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="smlNTIgN" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-82cebbdab08so2001340b3a.2 for ; Fri, 10 Apr 2026 14:36:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856987; x=1776461787; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=Qp18656DxFkgwOS2U3bgLEL3hq8+puEOLyZcxkp/3uY=; b=smlNTIgNbEDERR9GDG8/chy5EfUWP+Chh9YhoL6GAaaF81IP20dlFrhInzS+EHTIaY JbxKHte6sasbB37HaPOyfOGj604MkovjjASbFniFaY85iHVqULGVsWS6zAAyCCZ5p7xI vzH1S9CyDVwxJ5M5lgI7sRZ3LdIkzZJJUx//tU4EqCEoVt4LMvJ0jucm1BwrrTyi+/Im nPEPDORslG9E8Xpc9q2B4OYTzYE2AG+/wWvEBAu+OQdyOz74M6LPlsTRjQYA/Gy3nKUY rksquuzqnC1dPp+kE8LlgU9Ykp0MMpTrGZkXpy0w7rLuOIlLhgNA22HcNlzV75T8SSPW yNCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856987; x=1776461787; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Qp18656DxFkgwOS2U3bgLEL3hq8+puEOLyZcxkp/3uY=; b=gMthq01Ye2kxuJ1ueMLZsvT1oBNHwMANcX5+qAZuEYqyPftLMPQBjPINt65fLG7+NQ UGFJ08v6S+tmM9pwtZCnA9CGs3iObojudDMGykj+FH5vv19fZ469g/dqKVBjUYQF4mCs tgCtlys9hRT/YsvsBhabBZQ9hMdphVL6mepbrKazoKQ3Ydo+0yys6As0fyt1cwVtzXx4 J/acafutg/lEmmO9nybl+hKPgFU0iXCjElPjVWKI12glEBdLLMOxKNCgSHtQIrwSh6p7 4O86RoIlJWVqLfDMzzbSUDGmI5Sit/M6zPlMXHFsXgMEuOvPqdBig5F3sUxQD/qTsIL1 qJoQ== X-Forwarded-Encrypted: i=1; AJvYcCXTKV6DMlktci1oak4C5f87S7JqIMVM0oQ0aVpTcvlsu3BgqM1QnHdtRhzj5EK5ULFwEQmZSvofwPwPCC4=@vger.kernel.org X-Gm-Message-State: AOJu0YxRHwlKSoIJ3mgwLLa3lyhYUoGQ7QiH4XUOnZzlXhJoo6ZuhAFA FQOYrJJi/Pt17T0koFAQBC1VWd/doYnspSGVyYjhhB/q+5AjL4zPZq3s X-Gm-Gg: AeBDievYe6AZXUCThlFb67TM/qYkQEwAqrn47Mnc8j1AcxJ/JAx/MRPnwa1Z/PYc9fe pR8ZvQrKLG4BfvzColmeazcfux3r6/yepaFjiEhTS452Z9xgc3+Mb5dBKh3sQccCD5egiNn/c5d epWhdr02rrkQ4+ED0XA+rd3qWXAvUpPzhbneDoJUHRe7msATelybLB9vR6PPq9HNZjZbc6w02M6 okWwb/T1/QvNBrZldL4XcMS/F38uIRVEl+8Sfw6Zcu7Eq/x0dN9fv9rMBWFyaNRim4vnQRusaIl e3s1WFma3TIXpoabDCsWASNWCjwNJCcXie+5HYhC5mUJcDC9P0aZxlaPivRUTXVFzwNxTTE6cLi PsAKl5yIdfvlaCV6H0SGtUH+8MJptQd0k5eQHYUc3Q5pWZkA/5+SGJYWi64LtfJ8yEgs4OX3ZJW deC3rVVZrPMlUcN8eNfYlDtlb0jmOuRLt9pYnY+6+ZhvS1 X-Received: by 2002:a05:6a00:4388:b0:82c:daa4:ce2b with SMTP id d2e1a72fcca58-82f0c2f1536mr5606609b3a.49.1775856986535; Fri, 10 Apr 2026 14:36:26 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:26 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:36 -0700 Subject: [PATCH v5 6/8] rust: percpu: add a rust per-CPU variable sample Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-6-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=13379; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=MZsr3UdiERwV1f/QQfUp88/S78aGt9zYSriryeD+vLw=; b=rKc93uyRO4qfeWUI6y1uol7p76Ye5NgzhZp3pAGG5J/gi0WD/mvuCE/Sq1fkPMzpQCuRmoB/N Ur2QmXulKQ9BolYILXGzMRLyvFRjw8NpVKiAPCPSKhnRjb0Yh6UWMrz X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add a short exercise for Rust's per-CPU variable API, modelled after lib/percpu_test.c Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 1 + rust/kernel/percpu.rs | 2 +- samples/rust/Kconfig | 9 ++ samples/rust/Makefile | 1 + samples/rust/rust_percpu.rs | 278 ++++++++++++++++++++++++++++++++++++++++= ++++ 5 files changed, 290 insertions(+), 1 deletion(-) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index 3b2f69a96c66..173b516cd813 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include =20 __rust_helper void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 0ec1245038bb..72c83fef68ee 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -2,7 +2,7 @@ //! Per-CPU variables. //! //! See the [`crate::define_per_cpu!`] macro, the [`DynamicPerCpu`] type, = and the [`PerCpu`] -//! trait. +//! trait. Example usage can be found in `samples/rust/rust_percpu.rs`. =20 pub mod cpu_guard; mod dynamic; diff --git a/samples/rust/Kconfig b/samples/rust/Kconfig index c49ab9106345..2616dbd7b37a 100644 --- a/samples/rust/Kconfig +++ b/samples/rust/Kconfig @@ -172,6 +172,15 @@ config SAMPLE_RUST_SOC =20 If unsure, say N. =20 +config SAMPLE_RUST_PERCPU + tristate "Per-CPU support" + depends on m + help + Enable this option to build a module which demonstrates Rust per-CPU + operations. + + If unsure, say N. + config SAMPLE_RUST_HOSTPROGS bool "Host programs" help diff --git a/samples/rust/Makefile b/samples/rust/Makefile index 6c0aaa58cccc..1ce120ac0402 100644 --- a/samples/rust/Makefile +++ b/samples/rust/Makefile @@ -16,6 +16,7 @@ obj-$(CONFIG_SAMPLE_RUST_DRIVER_FAUX) +=3D rust_driver_f= aux.o obj-$(CONFIG_SAMPLE_RUST_DRIVER_AUXILIARY) +=3D rust_driver_auxiliary.o obj-$(CONFIG_SAMPLE_RUST_CONFIGFS) +=3D rust_configfs.o obj-$(CONFIG_SAMPLE_RUST_SOC) +=3D rust_soc.o +obj-$(CONFIG_SAMPLE_RUST_PERCPU) +=3D rust_percpu.o =20 rust_print-y :=3D rust_print_main.o rust_print_events.o =20 diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs new file mode 100644 index 000000000000..5adb30509bd4 --- /dev/null +++ b/samples/rust/rust_percpu.rs @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: GPL-2.0 +//! A simple demonstration of the rust per-CPU API. + +use core::cell::RefCell; +use core::ffi::c_void; + +use kernel::{ + bindings::on_each_cpu, + cpu::CpuId, + define_per_cpu, get_static_per_cpu, + percpu::{cpu_guard::*, *}, + pr_info, + prelude::*, + sync::Arc, +}; + +module! { + type: PerCpuMod, + name: "rust_percpu", + authors: ["Mitchell Levy"], + description: "Sample to demonstrate the Rust per-CPU API", + license: "GPL v2", +} + +struct PerCpuMod; + +define_per_cpu!(PERCPU: i64 =3D 0); +define_per_cpu!(UPERCPU: u64 =3D 0); +define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); + +impl kernel::Module for PerCpuMod { + fn init(_module: &'static ThisModule) -> Result { + pr_info!("rust percpu test start\n"); + + let mut native: i64 =3D 0; + let mut pcpu: StaticPerCpu =3D get_static_per_cpu!(PERCPU); + + // SAFETY: We only have one PerCpu that points at PERCPU + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut i64| { + pr_info!("The contents of pcpu are {}\n", *val); + + native +=3D -1; + *val +=3D -1; + pr_info!("Native: {}, *pcpu: {}\n", native, *val); + assert!(native =3D=3D *val && native =3D=3D -1); + + native +=3D 1; + *val +=3D 1; + pr_info!("Native: {}, *pcpu: {}\n", native, *val); + assert!(native =3D=3D *val && native =3D=3D 0); + }); + + let mut unative: u64 =3D 0; + let mut upcpu: StaticPerCpu =3D get_static_per_cpu!(UPERCPU); + + // SAFETY: We only have one PerCpu pointing at UPERCPU + unsafe { upcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| { + unative +=3D 1; + *val +=3D 1; + pr_info!("Unative: {}, *upcpu: {}\n", unative, *val); + assert!(unative =3D=3D *val && unative =3D=3D 1); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, *val); + assert!(unative =3D=3D *val && unative =3D=3D 0); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, *val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + + unative =3D 0; + *val =3D 0; + + unative =3D unative.wrapping_sub(1); + *val =3D val.wrapping_sub(1); + pr_info!("Unative: {}, *upcpu: {}\n", unative, *val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + assert!(unative =3D=3D *val && unative =3D=3D u64::MAX); + }); + + let mut checked_native: u64 =3D 0; + let checked: StaticPerCpu> =3D get_static_per_cpu!(CH= ECKED); + checked.get(CpuGuard::new()).with(|val: &RefCell| { + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + + pr_info!("rust static percpu test done\n"); + + pr_info!("rust dynamic percpu test start\n"); + let mut test: DynamicPerCpu =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 1); + on_each_cpu(Some(check_percpu_u64), (&raw mut test).cast(), 1); + } + + let checked: DynamicPerCpu> =3D + DynamicPerCpu::new_with(&RefCell::new(100), GFP_KERNEL).unwrap= (); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu( + Some(inc_percpu_refcell_u64), + (&raw const checked) as *mut c_void, + 0, + ); + on_each_cpu( + Some(inc_percpu_refcell_u64), + (&raw const checked) as *mut c_void, + 0, + ); + on_each_cpu( + Some(inc_percpu_refcell_u64), + (&raw const checked) as *mut c_void, + 0, + ); + on_each_cpu( + Some(inc_percpu_refcell_u64), + (&raw const checked) as *mut c_void, + 1, + ); + on_each_cpu( + Some(check_percpu_refcell_u64), + (&raw const checked) as *mut c_void, + 1, + ); + } + + checked.get(CpuGuard::new()).with(|val: &RefCell| { + assert!(*val.borrow() =3D=3D 104); + + let mut checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + *val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + + let arc =3D Arc::new(0, GFP_KERNEL).unwrap(); + { + let _arc_pcpu: DynamicPerCpu> =3D + DynamicPerCpu::new_with(&arc, GFP_KERNEL).unwrap(); + } + // `arc` should be unique, since all the clones on each CPU should= be dropped when + // `_arc_pcpu` is dropped + assert!(Arc::into_unique_or_drop(arc).is_some()); + + pr_info!("rust dynamic percpu test done\n"); + + // Return Err to unload the module + Result::Err(EINVAL) + } +} + +extern "C" fn inc_percpu_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| *val += =3D 1); +} + +extern "C" fn check_percpu_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| assert!(= *val =3D=3D 4)); +} + +extern "C" fn inc_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let pcpu =3D unsafe { (*(info as *const DynamicPerCpu>)).= clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let mut val =3D val.borrow_mut(); + *val +=3D 1; + }); +} + +extern "C" fn check_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let pcpu =3D unsafe { (*(info as *const DynamicPerCpu>)).= clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let val =3D val.borrow(); + assert!(*val =3D=3D 104); + }); +} --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCC2C3DB65F for ; Fri, 10 Apr 2026 21:36:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856990; cv=none; b=jrIwgvAPLeleIf8g1i3XnFo0UzYVB5rfQTKE2GeSGkb7OGW1smR80P1U/8fETSszzYuMDOcrNjYkcqQVeTEUrsrfGbZW8YVp/urqCnBRYoL11BaM1aFpdZIpp+9hSdyexyZA5i85PBEQhqIC6AO3pAaWLkXh+eiWbxOFa1tdEZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856990; c=relaxed/simple; bh=IyeQBCcNFTW60JZl2i8GXrsguuamIFDOED+ecW44t6k=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dRJ/gTz6DtWQwH+qrrjHRHgmQ5QqP+UQFEgEB2uvwYtapxnIEGmKFEJjj02Tm6LHLxcYCRfwQ6neD6JTB3IXjtqlV1E+O2rKmuCpDQxm139xVc3bOX4CUllwLDrFghGRhNI5OMiTY0OVVzGvBuHd8DzdbxW6aJMebA6VBj9JYvc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=tDz5wGIv; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tDz5wGIv" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-c76e702e01aso893100a12.3 for ; Fri, 10 Apr 2026 14:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856988; x=1776461788; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=+JzMtJCfhE/cOyRCL4deN82RoA6oSUseqvroceFE1d8=; b=tDz5wGIvoS7Vswqgc5VP3oVtqwBcqJ1zz0Z72nuIczcbg8l17HxtOFKyGnjhVgpY1U J8NKVLP93VFTGVebqUDdzTVgkGk6kcCBkvILVT2fU8MRihS7Et66xmKhHAaqNtroSqua Nu89omNpi7D2bwIWmtTWBLKyuhI38I1t+rxy9ttb/RgkL/YvVHRDF75P2UT2zV/49v3M tRwslg4qgRFbKi5O874S5GD2O6blceu+QU844b9MemiPskXlH5MpWnZQ1jIOXLbBECe1 HD0RXUBVHhMgz+NCj2E0GNlIn/rN2kKggnxIHV8FAqmvEwzkOdrcPO93bXr/M6gEOBu0 fP4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856988; x=1776461788; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=+JzMtJCfhE/cOyRCL4deN82RoA6oSUseqvroceFE1d8=; b=cBlRjbp9uV/X3GEeng6/iB4Gk0fBBFJVWlBVTcL0CHYvh3LwkkYY8EoZaOsTL/kFxC 8JiHbUXZ0XIGuXIabC2YkL5Q7U8FMqaZocuKw5F2jsMEC7PFcWMeNaAtEDdIjxVTrzUK jH22C7lxd5OILFu7dUAqrfrK9FaRVmTwFPI1fQxizn6RITlpLK/22FOfUX42pSpDlOjo Li3ApEtOXKTROOQNWpjVvCnuxIB8hlo53f4JTI25nU9f6UXfn4Nlq4xBTalKnKyQDBdL stLOkLomicLO5npLeEIV2PW4PGZCdfb5tQMlyOPHAR01b8bRpajgtfxc0hKQ80B6U19t C3VQ== X-Forwarded-Encrypted: i=1; AJvYcCUh8oNuDspnzGqkQ5qsOOyhx4h9cb691zSFjjCU366GO9NnzBDAOdF0u1yglS9z2Uwzox5/cxCrp3qf98M=@vger.kernel.org X-Gm-Message-State: AOJu0YySGZO18IdFJtYvYdFoXZwxAWMVSThjR+xLChuAcVs6VSsudR9b 8+Zgkd2r6TQ7YrKURiQ4aeF0x7cS6MtAFfajWZp9k+/2gcIyn3Mj/eHB X-Gm-Gg: AeBDietRkeBIVr9Ri9t4JzQUGrjymAQu87zK8oEItJGLVpnRR1l3Z2xD8yWLzsKk5hi a37ftuyZyB7W/A046DVmly04TfY0fUKLoGHu4irGXlGBR9Opxr0FaCtqa1Y9LUoPTjmBYVOgrYB RCS7yG5Qq6gkCZS32s2688a/DXzwQfWVihRvEZ3+7bxwaKwz1S4S2053GtQJELWj5Q+WdOJqADn QLJn67PsQPeLNxMDxEtRW3TjpKKlT7mDbl70Xw60wDcrdgYnLoZiVI45UW6nJtD54raFTiDtSKL Y7loKY0yLwPwtuT2K9IXcry617/8NvxEUb6TqC3MhymUwqz1p65RbqMsHBwrKQsADmBnCSv9Ry7 b6yDhAdM5FbL0vdefiM+mHB2ETDst5guLLu4duJPZRPLWsviUDkZ+Ke4FLK52CRJI5OG5ZttXxT W4HuBu4vbQ4oFpWWJmuLerlK3CqvmGmkOTIXrtZZGC87Ok/aodhrN4Aok= X-Received: by 2002:a05:6a20:2588:b0:398:a5c7:3dc9 with SMTP id adf61e73a8af0-39fe405dd2amr5667454637.49.1775856988108; Fri, 10 Apr 2026 14:36:28 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:27 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:37 -0700 Subject: [PATCH v5 7/8] rust: percpu: Add pin-hole optimizations for numerics Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-7-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=10544; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=IyeQBCcNFTW60JZl2i8GXrsguuamIFDOED+ecW44t6k=; b=1sEnwXSt8TG44DrXsJ2xdTtZ9Hvqtj8T/1xbiPRQKEcLDvTwN0VJ1fuUJa/x5cbQVfQfLhG1a 2UllwHEsDBTB8lfz/5JZy2p3OjK9kiS+VnYge4/oR35vnsTFhL0mRoI X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= The C implementations of `this_cpu_add`, `this_cpu_sub`, etc., are optimized to save an instruction by avoiding having to compute `this_cpu_ptr(&x)` for some per-CPU variable `x`. For example, rather than u64 *x_ptr =3D this_cpu_ptr(&x); *x_ptr +=3D 5; the implementation of `this_cpu_add` is clever enough to make use of the fact that per-CPU variables are implemented on x86 via segment registers, and so we can use only a single instruction (where we assume `&x` is already in `rax`) add gs:[rax], 5 Add this optimization via a `PerCpuNumeric` type to enable code-reuse between `DynamicPerCpu` and `StaticPerCpu`. Signed-off-by: Mitchell Levy --- rust/kernel/percpu.rs | 1 + rust/kernel/percpu/dynamic.rs | 10 ++- rust/kernel/percpu/numeric.rs | 138 ++++++++++++++++++++++++++++++++++++++= ++++ samples/rust/rust_percpu.rs | 36 +++++++++++ 4 files changed, 184 insertions(+), 1 deletion(-) diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 72c83fef68ee..ff04607ee047 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -6,6 +6,7 @@ =20 pub mod cpu_guard; mod dynamic; +pub mod numeric; mod static_; =20 #[doc(inline)] diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index 40514704b3d0..a717138b93dc 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -28,7 +28,7 @@ /// the memory location on any particular CPU has been initialized. This m= eans that it cannot tell /// whether it should drop the *contents* of the allocation when it is dro= pped. It is up to the /// user to do this via something like [`core::ptr::drop_in_place`]. -pub struct PerCpuAllocation(PerCpuPtr); +pub struct PerCpuAllocation(pub(super) PerCpuPtr); =20 impl PerCpuAllocation { /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, @@ -162,6 +162,14 @@ pub fn new_from(mut initer: impl FnMut(CpuId) -> T, fl= ags: Flags) -> Option DynamicPerCpu { + /// Gets the allocation backing this per-CPU variable. + pub(crate) fn alloc(&self) -> &Arc> { + // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. + unsafe { self.alloc.as_ref().unwrap_unchecked() } + } +} + impl PerCpu for DynamicPerCpu { unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { // SAFETY: diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs new file mode 100644 index 000000000000..13b4ab4a794d --- /dev/null +++ b/rust/kernel/percpu/numeric.rs @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Pin-hole optimizations for [`PerCpu`] where T is a numeric type. + +use super::*; +use core::arch::asm; + +/// Represents a per-CPU variable that can be manipulated with machine-int= rinsic numeric +/// operations. +pub struct PerCpuNumeric<'a, T> { + // INVARIANT: `ptr.0` is a valid offset into the per-CPU area and is i= nitialized on all CPUs + // (since we don't have a CPU guard, we have to be pessimistic and ass= ume we could be on any + // CPU). + ptr: &'a PerCpuPtr, +} + +macro_rules! impl_ops { + ($ty:ty, $reg:tt) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc().0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `StaticPerCpu`'s inv= ariant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + #[inline] + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("add gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<$ty>(), + val =3D in(reg) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + #[inline] + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("sub gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<$ty>(), + val =3D in(reg) rhs, + ); + } + } + } + }; +} + +macro_rules! impl_ops_byte { + ($ty:ty) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc().0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a [`PerCpuNumeric`] that can be used to manipulate= the underlying per-CPU + /// variable. + #[inline] + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `StaticPerCpu`'s inv= ariant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + #[inline] + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + "add gs:[{off}], {val}", + off =3D in(reg) self.ptr.0.cast::<$ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + #[inline] + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + "sub gs:[{off}], {val}", + off =3D in(reg) self.ptr.0.cast::<$ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + }; +} + +impl_ops_byte!(i8); +impl_ops!(i16, "x"); +impl_ops!(i32, "e"); +impl_ops!(i64, "r"); +impl_ops!(isize, "r"); + +impl_ops_byte!(u8); +impl_ops!(u16, "x"); +impl_ops!(u32, "e"); +impl_ops!(u64, "r"); +impl_ops!(usize, "r"); diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs index 5adb30509bd4..90f5debd3c7a 100644 --- a/samples/rust/rust_percpu.rs +++ b/samples/rust/rust_percpu.rs @@ -28,6 +28,26 @@ define_per_cpu!(UPERCPU: u64 =3D 0); define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); =20 +macro_rules! make_optimization_test { + ($ty:ty) =3D> { + let mut test: DynamicPerCpu<$ty> =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + { + let _guard =3D CpuGuard::new(); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = *val =3D 10); + test.num().add(1); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 11)); + test.num().add(10); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 21)); + test.num().sub(5); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 16)); + } + }; +} + impl kernel::Module for PerCpuMod { fn init(_module: &'static ThisModule) -> Result { pr_info!("rust percpu test start\n"); @@ -228,6 +248,22 @@ fn init(_module: &'static ThisModule) -> Result { =20 pr_info!("rust dynamic percpu test done\n"); =20 + pr_info!("rust numeric optimizations test start\n"); + + make_optimization_test!(u8); + make_optimization_test!(u16); + make_optimization_test!(u32); + make_optimization_test!(u64); + make_optimization_test!(usize); + + make_optimization_test!(i8); + make_optimization_test!(i16); + make_optimization_test!(i32); + make_optimization_test!(i64); + make_optimization_test!(isize); + + pr_info!("rust numeric optimizations test done\n"); + // Return Err to unload the module Result::Err(EINVAL) } --=20 2.34.1 From nobody Sat Apr 11 03:29:38 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1B273DC4A7 for ; Fri, 10 Apr 2026 21:36:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856991; cv=none; b=U5462ie23ASVkkHtv5fc9ODJdPBIlfP2+6vXHJpfRiMnBvL69GGocF1hqbG0doDV/XN1eaBqXjUcvTOY6nO8FNW2seVGWCCtRFHyGBT5irFr1z1LrdC39gItf5Xwx0pSZdjtxj3Nyt2eYkno/am3frxcjX99tSh4u/a+wwQRFgE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775856991; c=relaxed/simple; bh=FHBR4Wnqy/ydHjiZKFZpGeMrvO4ZgH/nNhnHVQU7j4U=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=H5anK4TCE1LFfrvCD653U7SXWLG/k0rM63y69iIeqHERMfToVGRM71wdjP2lb45Gyr1jV6UM03I43GylY6fFzuMsrOxkxduXditRk0j+fQX6s4g/WRsOn2prPmphJYRVonUHuThQXv9tsFeqAfgqnoaHyyFpkpvN9XKNk0cBMYc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A/9eqKRy; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A/9eqKRy" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-82cdb4ab547so1339779b3a.2 for ; Fri, 10 Apr 2026 14:36:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775856989; x=1776461789; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=1kY8tNpZZWpISJZ66yxQbfZrkvSCDqL724L2ETykFfw=; b=A/9eqKRyDCSXzrefsdDxmEQ6XVGgvwrdyCdJqpEsR3AzQDUmce7MMeSJLtLVZG7GCb lpA2tAh3MxquIMuz/+E4j7Fp8wXe4TgPVLRVCi3LI5WDmTny7ZIWMlzqBtRu7beWArZi +yjCht1GSerzTygfXGWYNp+fZw/ZecfMbzy6xiYOH/XDDOQwdwcwbBAOu4YAzukkOBeM 2iPYPlYuew8eQ/WuAZn29D0TL7qB5aevChSbWnA9WhERdWFmWuVtSFUtS7ZRv4Rs+Ezs netD+X+yPyMhyAh1lRKsHGl3Kb0E/A4SYgYf6jciYtrhWClr3g5K9PrP1b41AgFKcWf7 g3Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775856989; x=1776461789; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-gg:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=1kY8tNpZZWpISJZ66yxQbfZrkvSCDqL724L2ETykFfw=; b=rTY9YYT0CAOvEnOMSAqpOvSzn1koEzJiS4W/U7jXl21lmRX8cfWws3Br3odIE1tCty UtpWlsUFjWDK+/+cFALX+RFokP3amFxwQWNDXv8wqiWR9Dxj3lzh7RJs4aL+QOVR/8a4 q9bCPoARGPnnExmVrGLltzVQw5my8tBjAl2kOjt+FvKPrqG+gkrfVNplAKm6Amw5CX0J pUdtReSKwRC0QFeIO/c3V4jQqtRsJRrfyCMlLJmSOHbWMxoufKOpkUHux8ZYx/Ppu+6u nnqjJXE2dRvYpcVSVjgMB8t72kJNq0RnFvpJr7IhUU5wwtACK+L8BD2ZPjbwYrxUOkb8 7jQw== X-Forwarded-Encrypted: i=1; AJvYcCUsz9MjR8ql7n8Vq/8TBQ+Z7X9IQj+ubsD9qmynP3G13VSJiC8ETXunennPmR6NwCQQU/UErdCU6dIDsoE=@vger.kernel.org X-Gm-Message-State: AOJu0YyPYPBDHZleQ5E2hfxTsCDWk8rj0a6a5S00dAV/d2Y+lS3c/M8T g1QvjNXblH/IbDG8HQEeS/vyQAORzjgXgcoheAufrK/3+umeonEVWIo6 X-Gm-Gg: AeBDieukSWTT2OXCHYtHVkepJzkHrJQciyL9LIAEvMmFyz12uIuqhr2O9cwgS3wiCPa yqQd5q3y70W9PjjG3sisAnXeor9IP3HBf80RQ3IkPEjgLe7tI0ca+dXHCYlvSdMOnBVhTzjpXff sMKiCI6dPET+6pAIjRBJR/4Yfd1iBHupIcDSrKwuOBhHTYCW9RMDKXnTmtqGvMj4u1orTbwW4eP I9RWoB8nHnAKBypwjMNSrNhQUOBG/IAVPCXBlqvpEjTPHyj5wy/4Tv3O5MSxkY8nf5fCRx+BWwE FnqYCBnlbbuZqruCu/tMB05Hqx9VTROx0qimEIh0+/0IZvu2H833hK13hNM7U2pxZ6iEqc48oqD ZWZ4jaGlKcJH1KseroIvAmlIkFyaAcZ2VTOAThr4Hcs9rYZJvIoUJjcBKjTPeTdQeY6ZPfi5reB 0r35Z71SRTM9L6vry8s5KIXNfqOX2LgqG0Q1rPs5jBRmaD X-Received: by 2002:a05:6a00:1827:b0:82c:f041:765d with SMTP id d2e1a72fcca58-82f0c2a1cccmr5690404b3a.16.1775856988978; Fri, 10 Apr 2026 14:36:28 -0700 (PDT) Received: from mitchelllevy.localdomain ([131.107.1.135]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-82f0c4e3d41sm5111551b3a.48.2026.04.10.14.36.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 10 Apr 2026 14:36:28 -0700 (PDT) From: Mitchell Levy Date: Fri, 10 Apr 2026 14:35:38 -0700 Subject: [PATCH v5 8/8] rust: percpu: cache per-CPU pointers in the dynamic case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260410-rust-percpu-v5-8-4292380d7a41@gmail.com> References: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> In-Reply-To: <20260410-rust-percpu-v5-0-4292380d7a41@gmail.com> To: Miguel Ojeda , Alex Gaynor , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar , Boqun Feng Cc: Tyler Hicks , Allen Pais , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.15.1 X-Developer-Signature: v=1; a=ed25519-sha256; t=1775856978; l=5201; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=FHBR4Wnqy/ydHjiZKFZpGeMrvO4ZgH/nNhnHVQU7j4U=; b=ogoO04gAaNoTx75+C8BHdZL3inRGIgFDQvjKAC93cMw2HAcpU+LGo/rKWNrCe0NdbiWcyOwiV B35F+4kaHNtDVfLlWS+JakD7AUAmkILOvgE7I4iFqEvMzB6fhBKWQAz X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Currently, the creation of a `PerCpuNumeric` requires a memory read via the `Arc` managing the dynamic allocation. While the compiler might be clever enough to consolidate these reads in some cases, the read must happen *somewhere*, which, when we're concerning ourselves with individual instructions, is a very high burden. Instead, cache the `PerCpuPointer` inside the `DynamicPerCpu` structure; then, the `Arc` is used solely to manage the allocation. We might as well also use this speed-up in the standard `get` and `get_mut` accessors that give a `{Checked,}PerCpuToken` Signed-off-by: Mitchell Levy --- rust/kernel/percpu/dynamic.rs | 34 ++++++++++++++++++---------------- rust/kernel/percpu/numeric.rs | 4 ++-- 2 files changed, 20 insertions(+), 18 deletions(-) diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index a717138b93dc..675210098ed8 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -101,6 +101,9 @@ pub struct DynamicPerCpu { // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by the alloc is // initialized. alloc: Option>>, + // INVARIANT: `ptr` is the per-CPU pointer managed by `alloc`, which d= oes not change for the + // lifetime of `self`. + pub(super) ptr: PerCpuPtr, } =20 impl DynamicPerCpu { @@ -112,9 +115,13 @@ impl DynamicPerCpu { pub fn new_zero(flags: Flags) -> Option { let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; =20 + let ptr =3D alloc.0; let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: Some(arc) }) + Some(Self { + alloc: Some(arc), + ptr, + }) } } =20 @@ -158,15 +165,10 @@ pub fn new_from(mut initer: impl FnMut(CpuId) -> T, f= lags: Flags) -> Option DynamicPerCpu { - /// Gets the allocation backing this per-CPU variable. - pub(crate) fn alloc(&self) -> &Arc> { - // SAFETY: This type's invariant ensures that `self.alloc` is `Som= e`. - unsafe { self.alloc.as_ref().unwrap_unchecked() } + Some(Self { + alloc: Some(arc), + ptr, + }) } } =20 @@ -178,11 +180,11 @@ unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerC= puToken<'_, T> { // exists on the current CPU. // 3. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are // initialized on each CPU. - // 4. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is - // live. + // 4. `&mut self` holds a reference to the `PerCpuAllocation`, so = the allocation it manages + // is live. // 5. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for // a `T`. - unsafe { PerCpuToken::new(guard, &self.alloc.as_ref().unwrap_unche= cked().0) } + unsafe { PerCpuToken::new(guard, &self.ptr) } } } =20 @@ -192,11 +194,11 @@ fn get(&self, guard: CpuGuard) -> CheckedPerCpuToken<= '_, T> { // 1. Invariants of this type assure that `alloc` is `Some`. // 2. The invariants of `DynamicPerCpu` ensure that the contents o= f the allocation are // initialized on each CPU. - // 3. The existence of a reference to the `PerCpuAllocation` ensur= es that the allocation is - // live. + // 3. `&mut self` holds a reference to the `PerCpuAllocation`, so = the allocation it manages + // is live. // 4. The invariants of `DynamicPerCpu` ensure that the allocation= is sized and aligned for // a `T`. - unsafe { CheckedPerCpuToken::new(guard, &self.alloc.as_ref().unwra= p_unchecked().0) } + unsafe { CheckedPerCpuToken::new(guard, &self.ptr) } } } =20 diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs index 13b4ab4a794d..4841843b05f7 100644 --- a/rust/kernel/percpu/numeric.rs +++ b/rust/kernel/percpu/numeric.rs @@ -22,7 +22,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc().0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { @@ -78,7 +78,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc().0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { --=20 2.34.1