From nobody Wed Oct 8 19:55:46 2025 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2C8E25A330; Tue, 24 Jun 2025 22:11:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803099; cv=none; b=f94kMeORKxkW6oxo4ev4NMvzjddBbd7OQ5YXBbPOMWbhOKEqARZSxw/3u5SRKUtnRcVG21KvEshf38qUj+Ba1D1+KJAr96VDYdSn4Vn0nGXCK4iQWfPwvN5hzDcoHF11fmBvf8blu2RAACDhdij3Cgoi3v+mayZedItR3FYWOI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803099; c=relaxed/simple; bh=2fYVqdvVZOyhVdWES/18qc3/KomIzWwxDfp2+bGaXfw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=s+Jh97z71WeKQPEWSkGc8yQqZ/zb77r4tiS/m3N6apTzK6S2hXY9qdPtahJsfDYa6FvvLUwHMjiIk71IEZAWD308yDl4khukrxKp42SsLVYt9tMfELH1BtXoJAVqmxkF6HTIvRFPFhtlOzv+V/9CM3Yok0Bnw4tVKyLXe7AAxYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Wtcd1QMb; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wtcd1QMb" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2366e5e4dbaso3389825ad.1; Tue, 24 Jun 2025 15:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750803097; x=1751407897; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=k1qfz7npuSV263RU/XyMTPXjzl3/t+g8dNO+Vafq5Vc=; b=Wtcd1QMbLp6m4aiE2SYODPO5Zk4lRMX4Tav6OWdVa0AI37o0Noe5uvqiYm6AIpiI/Q rRDUJTNwYN8aBmhxUg4+uGc2ZNGNPSCM8PrFx0S/3uNcsJK7mv/0hrvfR1AL3e/EnYCU SU6sn2agj2YOv6/ZF4OiUGqsS63Vef9g1Zokl5BJbOIol+EVFdEwZThZAq7DfIbY7Rbr D7C57WEl1zjWxFizsmkNe906qOFzOvrzHanxmqod3WBcOARhojqG0tsxcEr5fup0zMXy TajBKaZ/o9m8NT8ThwyZQ09UawghtkgIOEnjcuq8DrdoqRB+AJh45OUy5dUgZIgKi/zs dBRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750803097; x=1751407897; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k1qfz7npuSV263RU/XyMTPXjzl3/t+g8dNO+Vafq5Vc=; b=JF8OUy8HStBlV5BZWnB3rIZjijIXHnoh2BjEjgAeG+Zntk2GF8W9flh7rdE/BTRDmH NSy3Z2/e4ABHFKKQnFMMH8DS5cWRQGqEiJOHxwBk6Rk6iq/VJWFRcMoRW1wXdVKkiJGn aOzr9cROJzXq0DAruklLzmnGdH9ovtGvpWVm+/zmu1LqR9GCV/ZU4VYlyJcDEUVDGhYF z1tIDgZiqqRHYe8VDJJX0QG+qb4rAwBNXZVw226pcyLMEH9NdMPSvCR6hRvmYAWFAIOn jouxLOG7Cu/36eLu0tLl255AQvOrx/CYWxJCuN5N8xBRcPymbRkmt+IEigA4kDY4OBmk 1dkQ== X-Forwarded-Encrypted: i=1; AJvYcCXPkdtnOGLxvR67/YTQEUQdG2t79ONNw1AUzmP28n3sOhDYrotHwRdJTsblgCNScrrXEzwm7wBQzq1WsbKlIQ==@vger.kernel.org X-Gm-Message-State: AOJu0YwvRdZA+7q+aKcWtAsT8lU0bfosN84z1Yr1f00IjtuAL744rVD2 6zPyPKuD+2zCyRcFo7q8W2FqjDSBASe23eIdv8oT4jfcl+Rx5rjqiZnM X-Gm-Gg: ASbGncukmSxDg9LvxNwjTnWEivZISvYteUbrIlkccKH05XSwgGJ3d8UThoshjG14aZt P38AjCLnocVJYb2ebos7Ja/1baWIowfIzmgxYOprkXn6TKleocoB8Uh3vlD8gmpQOrr2grla8ja E6mLF9LdA3ugIbZqLvaNf/m6RNS/qIejcVWaEvNbtgUAe4jtJLMGoHrdVVNY54il5kwnxUlYiAR RJXZ7dIQppCb1ZeWfmeAcEsohKue+12FEHXNN1MFFZ9oJ8J6rquc1F8FheHNAR3J1Zns60Jygi/ vn4cLB8x6vpwrUHhMsWVkOc9kxarTQkWf8hXapjZUwaV31fy0L+sTVaELqHjzawxtEHQmV/v0ES ZVU/Df2oiEKjsiJNZvIqy4FlagDz3g+bye5Ng X-Google-Smtp-Source: AGHT+IHoR9fiqp6ip71cYAnZJhavz9a3WeQJL/OnTgAV6bvnKoELPcHgQjb4d+FplsKuexEMI1F26A== X-Received: by 2002:a17:902:f68b:b0:232:59b:5923 with SMTP id d9443c01a7336-238253ec8b3mr9686685ad.23.1750803096739; Tue, 24 Jun 2025 15:11:36 -0700 (PDT) Received: from mitchelllevy.localdomain (82.sub-174-224-205.myvzw.com. [174.224.205.82]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237d8393741sm114580555ad.15.2025.06.24.15.11.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 15:11:36 -0700 (PDT) From: Mitchell Levy Date: Tue, 24 Jun 2025 15:10:39 -0700 Subject: [PATCH 1/5] rust: percpu: introduce a rust API for per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-rust-percpu-v1-1-9c59b07d2a9c@gmail.com> References: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> In-Reply-To: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin Cc: linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1750803093; l=15606; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=2fYVqdvVZOyhVdWES/18qc3/KomIzWwxDfp2+bGaXfw=; b=owgvvji1ROx81yigs8sJ/SU6nZxYmDVydO9bTpnfzG0fF3LIPscm98TeBFAnOSJ4u8J4DRJ98 R/lO168ofM9CpMzoMIIy0UV12Gsckj1oHyEm9phovR2C2EiTNuyUzc5 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add a `CpuGuard` type that disables preemption for its lifetime. Add a `PerCpuAllocation` type used to track dynamic allocations. Add a `define_per_cpu!` macro to create static per-CPU allocations. Add `DynamicPerCpu` and `StaticPerCpu` to provide a high-level API. Add a `PerCpu` trait to unify the dynamic and static cases. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- rust/helpers/helpers.c | 2 + rust/helpers/percpu.c | 9 ++ rust/helpers/preempt.c | 14 ++ rust/kernel/lib.rs | 3 + rust/kernel/percpu.rs | 299 ++++++++++++++++++++++++++++++++++++= ++++ rust/kernel/percpu/cpu_guard.rs | 35 +++++ 6 files changed, 362 insertions(+) diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c index 0f1b5d115985..d56bbe6334d3 100644 --- a/rust/helpers/helpers.c +++ b/rust/helpers/helpers.c @@ -29,7 +29,9 @@ #include "page.c" #include "platform.c" #include "pci.c" +#include "percpu.c" #include "pid_namespace.c" +#include "preempt.c" #include "rbtree.c" #include "rcu.c" #include "refcount.c" diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c new file mode 100644 index 000000000000..a091389f730f --- /dev/null +++ b/rust/helpers/percpu.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) +{ + return __alloc_percpu(sz, align); +} + diff --git a/rust/helpers/preempt.c b/rust/helpers/preempt.c new file mode 100644 index 000000000000..2c7529528ddd --- /dev/null +++ b/rust/helpers/preempt.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void rust_helper_preempt_disable(void) +{ + preempt_disable(); +} + +void rust_helper_preempt_enable(void) +{ + preempt_enable(); +} + diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs index 6b4774b2b1c3..733f9ff8b888 100644 --- a/rust/kernel/lib.rs +++ b/rust/kernel/lib.rs @@ -95,6 +95,9 @@ pub mod page; #[cfg(CONFIG_PCI)] pub mod pci; +// Only x86_64 is supported by percpu for now +#[cfg(CONFIG_X86_64)] +pub mod percpu; pub mod pid_namespace; pub mod platform; pub mod prelude; diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs new file mode 100644 index 000000000000..a912f74349e0 --- /dev/null +++ b/rust/kernel/percpu.rs @@ -0,0 +1,299 @@ +// SPDX-License-Identifier: GPL-2.0 +//! This module contains abstractions for creating and using per-CPU varia= bles from Rust. +//! See the define_per_cpu! macro and the PerCpu type. +pub mod cpu_guard; + +use bindings::{alloc_percpu, free_percpu}; + +use crate::alloc::Flags; +use crate::percpu::cpu_guard::CpuGuard; +use crate::sync::Arc; + +use core::arch::asm; + +use ffi::c_void; + +/// A per-CPU pointer; that is, an offset into the per-CPU area. +pub struct PerCpuPtr(*mut T); + +/// Represents a dynamic allocation of a per-CPU variable via alloc_percpu= . Calls free_percpu when +/// dropped. +pub struct PerCpuAllocation(PerCpuPtr); + +/// Holds a dynamically-allocated per-CPU variable. +pub struct DynamicPerCpu { + alloc: Arc>, +} + +/// Holds a statically-allocated per-CPU variable. +pub struct StaticPerCpu(PerCpuPtr); + +/// Represents exclusive access to the memory location pointed at by a par= ticular PerCpu. +pub struct PerCpuToken<'a, T> { + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +/// A wrapper used for declaring static per-CPU variables. These symbols a= re "virtual" in that the +/// linker uses them to generate offsets into each CPU's per-CPU area, but= shouldn't be read +/// from/written to directly. The fact that the statics are immutable prev= ents them being written +/// to (generally), this struct having _val be non-public prevents reading= from them. +/// +/// The end-user of the per-CPU API should make use of the define_per_cpu!= macro instead of +/// declaring variables of this type directly. +#[repr(transparent)] +pub struct StaticPerCpuSymbol { + _val: T, // generate a correctly sized type +} + +impl PerCpuPtr { + /// Makes a new PerCpuPtr from a raw per-CPU pointer. + /// + /// # Safety + /// `ptr` must be a valid per-CPU pointer. + pub unsafe fn new(ptr: *mut T) -> Self { + Self(ptr) + } + + /// Get a `&mut T` to the per-CPU variable represented by `&self` + /// + /// # Safety + /// The returned `&mut T` must follow Rust's aliasing rules. That is, = no other `&(mut) T` may + /// exist that points to the same location in memory. In practice, thi= s means that any PerCpu + /// on the same CPU holding a reference to the same PerCpuAllocation a= s `self` mut not call + /// `get_ref` for as long as the returned reference lives. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned &mut T might end up be= ing a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + #[allow(clippy::mut_from_ref)] // Safety requirements prevent aliasing= issues + pub unsafe fn get_ref(&self) -> &mut T { + let this_cpu_off_pcpu =3D core::ptr::addr_of!(this_cpu_off); + let mut this_cpu_area: *mut c_void; + // SAFETY: gs + this_cpu_off_pcpu is guaranteed to be a valid poin= ter because `gs` points + // to the per-CPU area and this_cpu_off_pcpu is a valid per-CPU al= location. + unsafe { + asm!( + "mov {out}, gs:[{off_val}]", + off_val =3D in(reg) this_cpu_off_pcpu, + out =3D out(reg) this_cpu_area, + ) + }; + // SAFETY: this_cpu_area + self.0 is guaranteed to be a valid poin= ter by the per-CPU + // subsystem and the invariant that self.0 is a valid offset into = the per-CPU area. + // + // We know no-one else has a reference to the underlying pcpu vari= able because of the + // safety requirements of this function. + unsafe { &mut *((this_cpu_area).wrapping_add(self.0 as usize) as *= mut T) } + } +} + +impl Clone for PerCpuPtr { + fn clone(&self) -> Self { + *self + } +} + +/// PerCpuPtr is just a pointer, so it's safe to copy. +impl Copy for PerCpuPtr {} + +impl PerCpuAllocation { + /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`. + /// + /// Returns `None` under the same circumstances the C function `alloc_= percpu` returns `NULL`. + pub fn new() -> Option> { + // SAFETY: No preconditions to call alloc_percpu + let ptr: *mut T =3D unsafe { alloc_percpu(size_of::(), align_of= ::()) } as *mut T; + if ptr.is_null() { + return None; + } + + Some(Self(PerCpuPtr(ptr))) + } +} + +impl Drop for PerCpuAllocation { + fn drop(&mut self) { + // SAFETY: self.0.0 was returned by alloc_percpu, and so was a val= id pointer into + // the percpu area, and has remained valid by the invariants of Pe= rCpuAllocation. + unsafe { free_percpu(self.0 .0 as *mut c_void) } + } +} + +/// A trait representing a per-CPU variable. This is implemented for both = `StaticPerCpu` and +/// `DynamicPerCpu`. The main usage of this trait is to call `get` to g= et a `PerCpuToken` that +/// can be used to access the underlying per-CPU variable. See `PerCpuToke= n::with`. +/// +/// # Safety +/// The returned value from `ptr` must be valid for the lifetime of `&mut = self`. +pub unsafe trait PerCpu { + /// Gets a `PerCpuPtr` to the per-CPU variable represented by `&mut= self` + /// + /// # Safety + /// `self` may be doing all sorts of things to track when the underlyi= ng per-CPU variable can + /// be deallocated. You almost certainly shouldn't be calling this fun= ction directly (it's + /// essentially an implementation detail of the trait), and you certai= nly shouldn't be making + /// copies of the returned `PerCpuPtr` that may outlive `&mut self`. + /// + /// Implementers of this trait should ensure that the returned `PerCpu= Ptr` is valid for + /// the lifetime of `&mut self`. + unsafe fn ptr(&mut self) -> &PerCpuPtr; + + /// Produces a token, asserting that the holder has exclusive access t= o the underlying memory + /// pointed to by `self` + /// + /// # Safety + /// `func` (or its callees that execute on the same CPU) may not call = `get_ref` on another + /// `PerCpu` that represents the same per-CPU variable as `&mut sel= f` (that is, they must + /// not be `clone()`s of each other or, in the case of statically allo= cated variables, + /// additionally can't both have come from the same `define_per_cpu!`)= for the lifetime of the + /// returned token. + /// + /// In particular, this requires that the underlying per-CPU variable = cannot ever be mutated + /// from an interrupt context, unless irqs are disabled for the lifeti= me of the returned + /// `PerCpuToken`. + unsafe fn get(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + PerCpuToken { + _guard: guard, + // SAFETY: The lifetime of the returned `PerCpuToken<'_, T>` i= s bounded by the lifetime + // of `&mut self`. + ptr: unsafe { self.ptr() }, + } + } +} + +impl StaticPerCpu { + /// Creates a new PerCpu pointing to the statically allocated varia= ble at `ptr`. End-users + /// should probably be using the `unsafe_get_per_cpu!` macro instead o= f calling this function. + /// + /// # Safety + /// `ptr` must be a valid pointer to a per-CPU variable. This means th= at it must be a valid + /// offset into the per-CPU area, and that the per-CPU area must be su= itably sized and aligned + /// to hold a `T`. + pub unsafe fn new(ptr: *mut T) -> Self { + Self(PerCpuPtr(ptr)) + } +} + +// SAFETY: The `PerCpuPtr` returned by `ptr` is valid for the lifetime = of `self` (and in fact, +// forever). +unsafe impl PerCpu for StaticPerCpu { + unsafe fn ptr(&mut self) -> &PerCpuPtr { + &self.0 + } +} + +impl Clone for StaticPerCpu { + fn clone(&self) -> Self { + Self(self.0) + } +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `flags` - Flags used to allocate an `Arc` that keeps track of th= e underlying + /// `PerCpuAllocation`. + pub fn new(flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new()?; + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: arc }) + } + + /// Wraps a `PerCpuAllocation` in a `PerCpu` + /// + /// # Arguments + /// * `alloc` - The allocation to use + /// * `flags` - The flags used to allocate an `Arc` that keeps track o= f the `PerCpuAllocation`. + pub fn new_from_allocation(alloc: PerCpuAllocation, flags: Flags) -= > Option { + let arc =3D Arc::new(alloc, flags).ok()?; + Some(Self { alloc: arc }) + } +} + +// SAFETY: The `PerCpuPtr` returned by `ptr` is valid for the lifetime = of `self` because we +// don't deallocate the underlying `PerCpuAllocation` until `self` is drop= ped. +unsafe impl PerCpu for DynamicPerCpu { + unsafe fn ptr(&mut self) -> &PerCpuPtr { + &self.alloc.0 + } +} + +impl Clone for DynamicPerCpu { + fn clone(&self) -> Self { + Self { + alloc: self.alloc.clone(), + } + } +} + +impl PerCpuToken<'_, T> { + /// Immediately invokes `func` with a `&mut T` that points at the unde= rlying per-CPU variable + /// that `&mut self` represents. + pub fn with(&mut self, func: U) + where + U: FnOnce(&mut T), + { + // SAFETY: The existence of a PerCpuToken means that the requireme= nts for get_ref are + // satisfied. + func(unsafe { self.ptr.get_ref() }); + } +} + +/// define_per_cpu! is analogous to the C DEFINE_PER_CPU macro in that it = lets you create a +/// statically allocated per-CPU variable. +/// +/// # Example +/// ``` +/// use kernel::define_per_cpu; +/// use kernel::percpu::StaticPerCpuSymbol; +/// +/// define_per_cpu!(pub MY_PERCPU: u64 =3D 0); +/// ``` +#[macro_export] +macro_rules! define_per_cpu { + ($vis:vis $id:ident: $ty:ty =3D $expr:expr) =3D> { + $crate::macros::paste! { + // Expand $expr outside of the unsafe block to avoid silently = allowing unsafe code to be + // used without a user-facing unsafe block + static [<__INIT_ $id>]: $ty =3D $expr; + + // SAFETY: StaticPerCpuSymbol is #[repr(transparent)], so w= e can freely convert from T + #[link_section =3D ".data..percpu"] + $vis static $id: StaticPerCpuSymbol<$ty> =3D unsafe { + core::mem::transmute::<$ty, StaticPerCpuSymbol<$ty>>([<__I= NIT_ $id>]) + }; + } + }; +} + +/// Gets a `PerCpu` from a symbol declared with `define_per_cpu!` or `d= eclare_extern_per_cpu!`. +/// +/// # Arguments +/// * `ident` - The identifier declared +/// +/// # Safety +/// `$id` must be declared with either `define_per_cpu!` or `declare_exter= n_per_cpu!`, and the +/// returned value must be stored in a `PerCpu` where `T` matches the d= eclared type of `$id`. +#[macro_export] +macro_rules! unsafe_get_per_cpu { + ($id:ident) =3D> {{ + $crate::percpu::StaticPerCpu::new((&$id) as *const _ as *mut _) + }}; +} + +/// Declares a StaticPerCpuSymbol corresponding to a per-CPU variable defi= ned in C. Be sure to read +/// the safety requirements of `PerCpu::get`. +#[macro_export] +macro_rules! declare_extern_per_cpu { + ($id:ident: $ty:ty) =3D> { + extern "C" { + static $id: StaticPerCpuSymbol<$ty>; + } + }; +} + +declare_extern_per_cpu!(this_cpu_off: u64); diff --git a/rust/kernel/percpu/cpu_guard.rs b/rust/kernel/percpu/cpu_guard= .rs new file mode 100644 index 000000000000..14c04b12e7f0 --- /dev/null +++ b/rust/kernel/percpu/cpu_guard.rs @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Contains abstractions for disabling CPU preemption. See `CpuGuard`. + +/// A RAII guard for bindings::preempt_disable and bindings::preempt_enabl= e. Guarantees preemption +/// is disabled for as long as this object exists. +pub struct CpuGuard { + // Don't make one without using new() + _phantom: (), +} + +impl CpuGuard { + /// Create a new CpuGuard. Disables preemption for its lifetime. + pub fn new() -> Self { + // SAFETY: There are no preconditions required to call preempt_dis= able + unsafe { + bindings::preempt_disable(); + } + CpuGuard { _phantom: () } + } +} + +impl Default for CpuGuard { + fn default() -> Self { + Self::new() + } +} + +impl Drop for CpuGuard { + fn drop(&mut self) { + // SAFETY: There are no preconditions required to call preempt_ena= ble + unsafe { + bindings::preempt_enable(); + } + } +} --=20 2.34.1 From nobody Wed Oct 8 19:55:46 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFEB525B1F0; Tue, 24 Jun 2025 22:11:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803100; cv=none; b=Z10FD3PUw0qSENk6TsAJyziusxFP7qYOZcjJLAG4Wpx7KD2kQUkbKjrZZrI7BDR5nNuKcvuJ8QOJ6YI130XAOJZaji0drXU/K/fUbrENZxIhihrCJUpIEwUKIMh/G80kdcb7pfphUjmD2Fkb3yPEs/xnWfrbNeF9xPcmhssS81I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803100; c=relaxed/simple; bh=AP8shLBDvKbYuth+8++0JymfuCshZQX8ezE8AjEF8do=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RnZjRDlrJVVfIXFQJKmgMIZ6si96GfVXnPeKVJeWlfm+3lZNhutWmQ9XfpUqERlNac3MB9iTLkRayDaHL/YHRorggfKSx7KvQ5vMPkIJsAQnEC+lh9Movy0j5fjTpKR+vZ5I3KhHXZpSJQhPaBn3Tx87NbgiYYapCqs5U0x4RuI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fDIcflTt; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fDIcflTt" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-235f9e87f78so65114745ad.2; Tue, 24 Jun 2025 15:11:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750803098; x=1751407898; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=LO0TxQnKvrLipyxCSYXSthueOvIoODhR5Ygf1iirZRk=; b=fDIcflTtHEnt/sU4fHvmfUByLBYwd1g6Hkdmc+qSao/rGUUiI38NjCMfRStSvugrNq +bMyghlvomxLos7CNN3hROvJknWer/NsxwR+ZlaQTmXjJJ3ewSFzT3qNALEPDNZykmlw IV2T1Py5tLkKi/msc7IwWlQIkey2o/VpVoyTpDe47ppDfMY+Xk05KO+MclbCbc9zWukZ 8+N5bzeu3p+3/OqPQUIUnSXX5LqiA1C5Hkuq9myt3zhVUe/6HPe70+s9aZAS9yPo1vbQ cpd2EHsvrZugD2rn8TxTCdXS8KhakFTIOONpwwuu9sxu85FTKgLJYNSi31ZWQ3l6GQao bSGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750803098; x=1751407898; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LO0TxQnKvrLipyxCSYXSthueOvIoODhR5Ygf1iirZRk=; b=JzJAIBsmZ83FhVf1uWBQXYHHN9e2CyfHtZiy0sawUlmtW9Cidnzr46K114rWeh6iFH rCBMp4ScKwfX/Wrz+WLjhPfI5Wkj11tdSsKCavqMcDStFuJe2c4mf6nlfri30YhROcQe 66L4t40XyB43oQfaoL6SrYafG91Z9AxGOCNANgflj6MmHfNj73VTyzdNFUbRx6KVtlAq vjGlMcdbAoiKmtuPVJkvUMFYB9kp4L0ydxqppcn9EOyVnfFUxCoOhdy0vZRGO+pYJA6G 7g793BWWOjYC8XXmgXCjPAA9el7YXnT6gqX5hAgdLlMMs2R7JDayY6qgiZrEf9N1odVt y25Q== X-Forwarded-Encrypted: i=1; AJvYcCVkVgOU3npGvp1tIkx5+Zwx+R19LR4bTCAxcgbw1UC7iKuCZkQzTR52/0HGoAmwV72OGiLIStd52FQHFlhsVA==@vger.kernel.org X-Gm-Message-State: AOJu0Ywpx05r1LeUGZgNMSZ+X/MPS4tN/mbRAzgILWNPbupdsdwdrc2M 6h8xmUnOwRertuJTMyBjRVARZMlyxi2MQF9UTsgYGRihlAu2fuHGP0p0 X-Gm-Gg: ASbGncuoKwbOHzU+Oxx6LnTE5RADZ/Y0/tiDgoeIGzlZ2HKyIqKyZCISTCZhDG4qTVN PS6EDp29pwQkuD6zcl36hxD1oYam6OoEhVzd81IZvJkmDjAvm5SlWQ9J5ecaT/w3/Hw4MVuCCNp stMrAZr5FblkCEgj22YQJ/cfGDjPuyGgDOtxsx30O2SkaNcCoFYJGTf88KFu+hilPV41l2E+D6x emQHY45YzMExlMOBjEaqQg1Z8my7LWRzS3XbqtZtAN4+gqgrQwXetUJ0TgyU2fNimk8/czM+aiQ RfEA/aCOtWv+Rr3Z0gwJeT7haJP6HlGMQGpQsgqYYp6+m9tycd+QFZmFQayNI+hji+qmyTrUY2b hfCRgXFu9yrE1zgl8TNPcWgDtHPc/hfpIuJqy X-Google-Smtp-Source: AGHT+IGzZk9ONGaeRCWEvkO3+3SKQAxtuncFrLeQo0zxqC9J+87qEA1081aiDzNzrowwqKyJ9OPZ4g== X-Received: by 2002:a17:903:4b2b:b0:234:de0a:b36e with SMTP id d9443c01a7336-238240827e5mr15140935ad.49.1750803098144; Tue, 24 Jun 2025 15:11:38 -0700 (PDT) Received: from mitchelllevy.localdomain (82.sub-174-224-205.myvzw.com. [174.224.205.82]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237d8393741sm114580555ad.15.2025.06.24.15.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 15:11:37 -0700 (PDT) From: Mitchell Levy Date: Tue, 24 Jun 2025 15:10:40 -0700 Subject: [PATCH 2/5] rust: rust-analyzer: add lib to dirs searched for crates Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-rust-percpu-v1-2-9c59b07d2a9c@gmail.com> References: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> In-Reply-To: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin Cc: linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1750803093; l=925; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=AP8shLBDvKbYuth+8++0JymfuCshZQX8ezE8AjEF8do=; b=pBKHWuAH/AUFsSJOOF+za31dYJq/Odb5UdYfMzE5NloODzCBbzwlWo+IzJHKrBnsWJ9iyiTqR AgI5GoK1RQoB74TbmGU12txIA42xh1HK3QuuTVPKYNnXBlWl9AQ13os X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= When generating rust-project.json, also include crates in lib/ Signed-off-by: Mitchell Levy --- scripts/generate_rust_analyzer.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_anal= yzer.py index 7c3ea2b55041..08e14ae9c1a0 100755 --- a/scripts/generate_rust_analyzer.py +++ b/scripts/generate_rust_analyzer.py @@ -152,7 +152,7 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs, core_edit # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs =3D map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs =3D map(lambda dir: srctree / dir, ("samples", "drivers", "= lib")) if external_src is not None: extra_dirs =3D [external_src] for folder in extra_dirs: --=20 2.34.1 From nobody Wed Oct 8 19:55:46 2025 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B99B25B667; Tue, 24 Jun 2025 22:11:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803102; cv=none; b=UBtd3nYlWTMfOn1xOniLu3AbD6ECacdtB6BtUYgZSkIKF5h1YvLv3/Xkt+Exc1HteNBueUpmW8qqoIao7Ddi7WM3dxQGoAAYO7Rx0zezcEKlBftGzqDP25mByrGbPvJtARTaKtBoDr5ZXnzeq9+gcxpkdbuS9JHU8YZhRKyaOus= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803102; c=relaxed/simple; bh=J2wN034E1ANJf8z7zTgfPajB+DTsr36981US4k79sBA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CqtujbTtoAIPWj/TKVgD3eGaKwSW1BqdJCijomUyTzA1RboB1hNDhJgDDPGhhQb1PWEd2Xi3vXNPmy8aMOKZVDgGbUEv/yIpIMCoYd68ijnzxpD9fd+XK0zXYqnneX/eTFtG5oiqVMs+nbV2U9z50mBVVggw6WzuTfIdQd6EVi4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=U24/A8Sc; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="U24/A8Sc" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-747abb3cd0bso231775b3a.1; Tue, 24 Jun 2025 15:11:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750803100; x=1751407900; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=hExQIcvoKTg+0rOcrZXWVdh84oK5KeLOdM+0A3CpbHg=; b=U24/A8Sciib9eZGPOhA/1+xwTM4DMRU+nDLEfGXwG/KbcPsA5V5Nz3aUcMcHrZ0WAZ Pz/hIJzzVUt8Fwbx++68Hthw2ec8HKqBjp6EjFINR7yntgfp7MYhwjnNY3FacBoondHY shiBmWfon6GoTUz6O1N9ttrcxXjcGfZnNbUtXPONAp1/hYh8W+doErDn2NiGfTDClmxr sHfVVnRidlqcrnenenKc1xeMN9q8A11pm/jEv0RuHMq7NXPcBKfEZYaISfVuP032noD4 +qj/lIh5DyLPcSwoLZM09uAC0pNCwevgzHlyNZ/qsxHD07RBM9Wjc1a0htOqhAY43HK7 rw9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750803100; x=1751407900; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hExQIcvoKTg+0rOcrZXWVdh84oK5KeLOdM+0A3CpbHg=; b=P8T+B6ua+VcmreLTwlm2Uj0CKmLNdjTqQt6Piw78ZWoqZKqjSAr/kUEiidogo88tjL AG1ViGX/2epsZWcNaYhwaRcRg9qdK5u+7KW9/o5OSpsK1wyovpPq7xYFnTIIqyACQiQJ qP2Ai4MKcSGjyMijhXR/uU9GUEKwNjkEIu8i2ym4P0YrufRp7p4nRSknT+ysBJqptHpg Jy8+tcRPbKYzuxTAXDG822zk6EebGe2mtbgviSPc0rTykgflTXQ53+9kUd0wotOai85z tj0r78hCO4HHetSinTYlamBN0GlDr+lSjjpIO9v6vsn5j5YM5RAEmgQb5rdBVgG7nVBZ aixw== X-Forwarded-Encrypted: i=1; AJvYcCWiF4oSXi93eXQtepFASFZYvYbD4B2zGrTdo7lflgvPxRGrGJvnCB6U8hYmimx62lgCmWd4S5o+5+dZG/pXIg==@vger.kernel.org X-Gm-Message-State: AOJu0YxjjeZLXHmrWqghAHrLMapjYVM9+glNRvuvYoj9KJem2y8KUG03 FQFk8G9jh/HWg7VoyoHDmJvqyO91NLaO44X0KZqhXn7ZV9IAgbeWUCmc X-Gm-Gg: ASbGncsFmC559dauJ/quE4eJDhUEvRfBQ8m8nf2K7GRKpsqN3hgM6wZ/Wy6lcbrhRB/ w9rIy1p4NuQXRzE62AzC4nudVEk18TVOWwQ5WxBd0LURHKAe4TbdROqTTwI4p+utwdlHHEBRJHQ CUDcUjLct+8xkzRib1HFqWA5AuQNs4Dj3sFk4ZA2asbQLa2EmmloX2FrHn99nTE5oI/6YznZ5qp eadidf01fuBUjJr+PGzHbQ2Y7Y0RclhNScu+PwAN1nHS8XEkwHhP0UYndLdDhSaW353AiGb19ZN IW8463NxS+URFovTJTa7xMHWXCrsY5wgaa8nXJBB6mgl11Yu2vm/oOEM+renwWzJk0soLqPMPZc BwO+Yx43Xs5VobBLlQIcvdXL6RJ7DWyT8dmbs X-Google-Smtp-Source: AGHT+IHWmfvN/QWU2UTaYOJYLgSEIUkW9IzlfVoIh8mrWtOInCaVWO4MlPJi/aEgerL1ApzcgujsRQ== X-Received: by 2002:a17:902:e74b:b0:231:9817:6ec1 with SMTP id d9443c01a7336-2382538f63bmr10265625ad.17.1750803099689; Tue, 24 Jun 2025 15:11:39 -0700 (PDT) Received: from mitchelllevy.localdomain (82.sub-174-224-205.myvzw.com. [174.224.205.82]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237d8393741sm114580555ad.15.2025.06.24.15.11.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 15:11:39 -0700 (PDT) From: Mitchell Levy Date: Tue, 24 Jun 2025 15:10:41 -0700 Subject: [PATCH 3/5] rust: percpu: add a rust per-CPU variable test Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-rust-percpu-v1-3-9c59b07d2a9c@gmail.com> References: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> In-Reply-To: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin Cc: linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1750803093; l=6875; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=J2wN034E1ANJf8z7zTgfPajB+DTsr36981US4k79sBA=; b=T9fAvCAJVHSEYQDHB7G792IQxMvoy47XXOoUkieCxnkJnVW13uXJLW6GXVuMO6r3HY6Oe+S1I iOgQJ7l3wZ8BhpuLe0yGd6tUtiR8mnLqFRxz/ygj687J7mnS6V6qdps X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add a short exercise for Rust's per-CPU variable API, modelled after lib/percpu_test.c Signed-off-by: Mitchell Levy --- lib/Kconfig.debug | 9 ++++ lib/Makefile | 1 + lib/percpu_test_rust.rs | 120 ++++++++++++++++++++++++++++++++++++++++++++= ++++ rust/helpers/percpu.c | 11 +++++ 4 files changed, 141 insertions(+) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index ebe33181b6e6..959ce156c601 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2418,6 +2418,15 @@ config PERCPU_TEST =20 If unsure, say N. =20 +config PERCPU_TEST_RUST + tristate "Rust per cpu operations test" + depends on m && DEBUG_KERNEL && RUST + help + Enable this option to build a test module which validates Rust per-cpu + operations. + + If unsure, say N. + config ATOMIC64_SELFTEST tristate "Perform an atomic64_t self-test" help diff --git a/lib/Makefile b/lib/Makefile index c38582f187dd..ab19106cc22c 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -288,6 +288,7 @@ obj-$(CONFIG_RBTREE_TEST) +=3D rbtree_test.o obj-$(CONFIG_INTERVAL_TREE_TEST) +=3D interval_tree_test.o =20 obj-$(CONFIG_PERCPU_TEST) +=3D percpu_test.o +obj-$(CONFIG_PERCPU_TEST_RUST) +=3D percpu_test_rust.o =20 obj-$(CONFIG_ASN1) +=3D asn1_decoder.o obj-$(CONFIG_ASN1_ENCODER) +=3D asn1_encoder.o diff --git a/lib/percpu_test_rust.rs b/lib/percpu_test_rust.rs new file mode 100644 index 000000000000..a9652e6ece08 --- /dev/null +++ b/lib/percpu_test_rust.rs @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: GPL-2.0 +//! A simple self test for the rust per-CPU API. + +use core::ffi::c_void; + +use kernel::{ + bindings::{on_each_cpu, smp_processor_id}, + define_per_cpu, + percpu::{cpu_guard::*, *}, + pr_info, + prelude::*, + unsafe_get_per_cpu, +}; + +module! { + type: PerCpuTestModule, + name: "percpu_test_rust", + author: "Mitchell Levy", + description: "Test code to exercise the Rust Per CPU variable API", + license: "GPL v2", +} + +struct PerCpuTestModule; + +define_per_cpu!(PERCPU: i64 =3D 0); +define_per_cpu!(UPERCPU: u64 =3D 0); + +impl kernel::Module for PerCpuTestModule { + fn init(_module: &'static ThisModule) -> Result { + pr_info!("rust percpu test start\n"); + + let mut native: i64 =3D 0; + // SAFETY: PERCPU is properly defined + let mut pcpu: StaticPerCpu =3D unsafe { unsafe_get_per_cpu!(P= ERCPU) }; + // SAFETY: We only have one PerCpu that points at PERCPU + unsafe { pcpu.get(CpuGuard::new()) }.with(|val: &mut i64| { + pr_info!("The contents of pcpu are {}\n", val); + + native +=3D -1; + *val +=3D -1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D -1); + + native +=3D 1; + *val +=3D 1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D 0); + }); + + let mut unative: u64 =3D 0; + // SAFETY: UPERCPU is properly defined + let mut upcpu: StaticPerCpu =3D unsafe { unsafe_get_per_cpu!(= UPERCPU) }; + + // SAFETY: We only have one PerCpu pointing at UPERCPU + unsafe { upcpu.get(CpuGuard::new()) }.with(|val: &mut u64| { + unative +=3D 1; + *val +=3D 1; + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 1); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 0); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + + unative =3D 0; + *val =3D 0; + + unative =3D unative.wrapping_sub(1); + *val =3D val.wrapping_sub(1); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + assert!(unative =3D=3D *val && unative =3D=3D u64::MAX); + }); + + pr_info!("rust static percpu test done\n"); + + pr_info!("rust dynamic percpu test start\n"); + let mut test: DynamicPerCpu =3D DynamicPerCpu::new(GFP_KERNEL= ).unwrap(); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu), (&raw mut test) as *mut c_void, = 0); + on_each_cpu(Some(inc_percpu), (&raw mut test) as *mut c_void, = 0); + on_each_cpu(Some(inc_percpu), (&raw mut test) as *mut c_void, = 0); + on_each_cpu(Some(inc_percpu), (&raw mut test) as *mut c_void, = 1); + on_each_cpu(Some(check_percpu), (&raw mut test) as *mut c_void= , 1); + } + + pr_info!("rust dynamic percpu test done\n"); + + // Return Err to unload the module + Result::Err(EINVAL) + } +} + +extern "C" fn inc_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Incrementing on {}\n", unsafe { smp_processor_id() }); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get(CpuGuard::new()) }.with(|val: &mut u64| *val +=3D 1); +} + +extern "C" fn check_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Asserting on {}\n", unsafe { smp_processor_id() }); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get(CpuGuard::new()) }.with(|val: &mut u64| assert!(*val= =3D=3D 4)); +} diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index a091389f730f..0e9b2fed3ebd 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -1,9 +1,20 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include =20 void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) { return __alloc_percpu(sz, align); } =20 +void rust_helper_on_each_cpu(smp_call_func_t func, void *info, int wait) +{ + on_each_cpu(func, info, wait); +} + +int rust_helper_smp_processor_id(void) +{ + return smp_processor_id(); +} + --=20 2.34.1 From nobody Wed Oct 8 19:55:46 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BE2A25BF05; Tue, 24 Jun 2025 22:11:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803103; cv=none; b=OXYbczzy+uJkhPAl3v49zXQuAtDklaOS4YbkY2u+uriMSU10MnMPXu1Nd6Q5wOlfpMs5YAXqEoDq2YNDHrdFhI8VY7mRQ+LL6ZehkN0oQYulfxuDL7ERNBRljd+UvHtFZxq4EJ+5MRfBB86bURDFYwqGHIR1O3L0opjdZnySIyg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803103; c=relaxed/simple; bh=YPpQN+Ft5ii+z9LN9DxUYct6G7gy1L7BlJzPkvs+pa4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=hTT5UM5i41KyCsoQftz+9LvOLpRD/sJbFoutpYcoAtmyr4hDuCHArCwJ9xXs/eMNY+g9f53kBjYiBiGMFOhL9lGQtiS8cBLDCP9+T/jdH/l2k8kGcAmuhytH6HVxWenckeljYB3Pa7XHEvq349Y7Mu78ppxY6MNHbif2EZIcy50= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FtyLBy+L; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FtyLBy+L" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-234b440afa7so12699975ad.0; Tue, 24 Jun 2025 15:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750803101; x=1751407901; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=MZlCvx1CGiBR7gpIxHEH/qkJGdg3V+G7iQKh210683E=; b=FtyLBy+LGSsZkUOaUl1eef4MfPGwPg2FvGCMXyUQ2HYRq0AxBlT82Pp7bGl4kuU183 rJkFfp8mKFMrhg4DtkDKZItdFGNqPkm6uOeCAlcYYa9G+wd8Q3ni7Jw1Py4g/FwegDHC pX9agmggrbVyJXjFnnEX03Vpli8YuepCCMLWf/q1ce2mTJHqCb7khdKBLlYkFw9YqWEl zwoWu8wfyDMvOiQ9B3feWQebiXkZ5s2djeBqwTMdQOGStpAn8HaaKkrAvJ4ikN7InWiT oOG2tk6dSF7Cj64DDUKtL7L9Nk6drq+BVaHKOl7mUtPuKqsFir57/pwpVQnU7ZH+XPAY Lcyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750803101; x=1751407901; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MZlCvx1CGiBR7gpIxHEH/qkJGdg3V+G7iQKh210683E=; b=BnUsgLGE2SPA41/uaoGpOGtgqX9SbUri7o2bBOpYiB4eHForvwFLfZGo8kufNczeOY Sgw0oQlF7iEFM18nQkJnZO+TvoNfDDEX9IYOLBLOhuZREECPddXOcnkvu0bgVhJXGBNR ZgMbZhQqC83RlGxt2jCSA3CHtwtAainPq3UJzNPxrOmkc1XF8LqXJz2RiWPrGpKJm2U3 TZODzg2Eu+TrI3AgOIDHE+lYDia1P5BaJBZxjLLfxkrZxsFGxEYKynzu3G9/sdu3avMa rOVAeEcMhvRNnOEzZfbmvkXG43TESsavUrPpZXZ2MAqv5Rr5ebYOpKQzf6MgBjBBhj9n nJzQ== X-Forwarded-Encrypted: i=1; AJvYcCWwRbIfDfECT+EZQWx8dZ/WWnKmsLpw4SwN1DLZx3EdNAZmd+QkJJNtuLxeQ0DOrv5aJV9UNoOw52TyD8IHLQ==@vger.kernel.org X-Gm-Message-State: AOJu0YxUFq6LwMAxWxkVFdsRtA4MWQQlho3yCkh+0JrRcaJNs8qBq8lt DIuyGrasTiIlq5Nu1Z1iYGM6cB7/svmpLTCOTHpSLTfQTOwGXIDzqXtozigkdgb1 X-Gm-Gg: ASbGncu1vB2EZdsmBAjpp4F3VPWCE29f6yx6GpFF/CxvUoeXpk4MKsAtWJZRHQy+4PA Q1srlfWLKyEe6vy5gt41tp6BO96kH2GonPWNyoRNuZVo9g0zau5MzfiZFMkkX37q/mjVa7dIjCI LoWqwp17I0eB0mzNM/9BfABVlKkkY8Tyii0v5mkr04cX3QRXFkLIpZp86+LvTocA1awDfyor638 a1RHRuuB7vIwPvCLj7j7+PfqLFXReT6wMQvgrSkO/zUy4daie1cfby18rsr7Swjw5VUOqdVli2L MO4NnPBCmRkKXLEy3vrw2H3JBjj1SXjqCywrtITzERKrDhrUj7d7oJxe9bx6H4YRu8Aj5pXaHIa DuBoRDW4pjoAerxBnJjbPkyCfsVubF5ij7RDE X-Google-Smtp-Source: AGHT+IEbmYtniKHMhkflwBj9MjSeO29aJCWX2mEvIk6KM9I9b4UHYOZVqVk9S+c5+SdXbRQY80QERQ== X-Received: by 2002:a17:903:198b:b0:233:ab04:27a with SMTP id d9443c01a7336-23824094ce2mr16400005ad.53.1750803101519; Tue, 24 Jun 2025 15:11:41 -0700 (PDT) Received: from mitchelllevy.localdomain (82.sub-174-224-205.myvzw.com. [174.224.205.82]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237d8393741sm114580555ad.15.2025.06.24.15.11.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 15:11:41 -0700 (PDT) From: Mitchell Levy Date: Tue, 24 Jun 2025 15:10:42 -0700 Subject: [PATCH 4/5] rust: percpu: Add pin-hole optimizations for numerics Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-rust-percpu-v1-4-9c59b07d2a9c@gmail.com> References: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> In-Reply-To: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin Cc: linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1750803093; l=8125; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=YPpQN+Ft5ii+z9LN9DxUYct6G7gy1L7BlJzPkvs+pa4=; b=72RLOn49lWvHD5dQo1lgVE6C04np3Nahgwl+XN/0/nxjN+3p+Qplvw1K9y82XK52dWg0C2OKr PpCyjPq8zViBVyeVjLEG/ASI20aH/vtoMvdllAXAaTrbYHwk+l5t3qu X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= The C implementations of `this_cpu_add`, `this_cpu_sub`, etc., are optimized to save an instruction by avoiding having to compute `this_cpu_ptr(&x)` for some per-CPU variable `x`. For example, rather than u64 *x_ptr =3D this_cpu_ptr(&x); *x_ptr +=3D 5; the implementation of `this_cpu_add` is clever enough to make use of the fact that per-CPU variables are implemented on x86 via segment registers, and so we can use only a single instruction (where we assume `&x` is already in `rax`) add gs:[rax], 5 Add this optimization via a `PerCpuNumeric` type to enable code-reuse between `DynamicPerCpu` and `StaticPerCpu`. Signed-off-by: Mitchell Levy --- lib/percpu_test_rust.rs | 36 +++++++++++++ rust/kernel/percpu.rs | 1 + rust/kernel/percpu/numeric.rs | 117 ++++++++++++++++++++++++++++++++++++++= ++++ 3 files changed, 154 insertions(+) diff --git a/lib/percpu_test_rust.rs b/lib/percpu_test_rust.rs index a9652e6ece08..114015435a85 100644 --- a/lib/percpu_test_rust.rs +++ b/lib/percpu_test_rust.rs @@ -25,6 +25,26 @@ define_per_cpu!(PERCPU: i64 =3D 0); define_per_cpu!(UPERCPU: u64 =3D 0); =20 +macro_rules! make_optimization_test { + ($ty:ty) =3D> { + let mut test: DynamicPerCpu<$ty> =3D DynamicPerCpu::new(GFP_KERNEL= ).unwrap(); + { + let _ =3D CpuGuard::new(); + // SAFETY: No other usage of `test` + unsafe { test.get(CpuGuard::new()) }.with(|val: &mut $ty| *val= =3D 10); + test.num().add(1); + // SAFETY: No other usage of `test` + unsafe { test.get(CpuGuard::new()) }.with(|val: &mut $ty| asse= rt_eq!(*val, 11)); + test.num().add(10); + // SAFETY: No other usage of `test` + unsafe { test.get(CpuGuard::new()) }.with(|val: &mut $ty| asse= rt_eq!(*val, 21)); + test.num().sub(5); + // SAFETY: No other usage of `test` + unsafe { test.get(CpuGuard::new()) }.with(|val: &mut $ty| asse= rt_eq!(*val, 16)); + } + }; +} + impl kernel::Module for PerCpuTestModule { fn init(_module: &'static ThisModule) -> Result { pr_info!("rust percpu test start\n"); @@ -94,6 +114,22 @@ fn init(_module: &'static ThisModule) -> Result { =20 pr_info!("rust dynamic percpu test done\n"); =20 + pr_info!("rust numeric optimizations test start\n"); + + make_optimization_test!(u8); + make_optimization_test!(u16); + make_optimization_test!(u32); + make_optimization_test!(u64); + make_optimization_test!(usize); + + make_optimization_test!(i8); + make_optimization_test!(i16); + make_optimization_test!(i32); + make_optimization_test!(i64); + make_optimization_test!(isize); + + pr_info!("rust numeric optimizations test done\n"); + // Return Err to unload the module Result::Err(EINVAL) } diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index a912f74349e0..9659bff2d889 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -2,6 +2,7 @@ //! This module contains abstractions for creating and using per-CPU varia= bles from Rust. //! See the define_per_cpu! macro and the PerCpu type. pub mod cpu_guard; +pub mod numeric; =20 use bindings::{alloc_percpu, free_percpu}; =20 diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs new file mode 100644 index 000000000000..e4008f872af1 --- /dev/null +++ b/rust/kernel/percpu/numeric.rs @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Pin-hole optimizations for PerCpu where T is a numeric type. + +use crate::percpu::*; +use core::arch::asm; + +/// Represents a per-CPU variable that can be manipulated with machine-int= rinsic numeric +/// operations. +pub struct PerCpuNumeric<'a, T> { + ptr: &'a PerCpuPtr, +} + +macro_rules! impl_ops { + ($ty:ty, $reg:tt) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU variable. + pub fn num(&self) -> PerCpuNumeric<'_, $ty> { + PerCpuNumeric { ptr: &self.alloc.0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU variable. + pub fn num(&self) -> PerCpuNumeric<'_, $ty> { + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of PerCpu. + unsafe { + asm!( + concat!("add gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0 as *mut $ty, + val =3D in(reg) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of PerCpu. + unsafe { + asm!( + concat!("sub gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0 as *mut $ty, + val =3D in(reg) rhs, + ); + } + } + } + }; +} + +macro_rules! impl_ops_byte { + ($ty:ty) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU + /// variable. + pub fn num(&self) -> PerCpuNumeric<'_, $ty> { + PerCpuNumeric { ptr: &self.alloc.0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU + /// variable. + pub fn num(&self) -> PerCpuNumeric<'_, $ty> { + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of PerCpu. + unsafe { + asm!( + concat!("add gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0 as *mut $ty, + val =3D in(reg_byte) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of PerCpu. + unsafe { + asm!( + concat!("sub gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0 as *mut $ty, + val =3D in(reg_byte) rhs, + ); + } + } + } + }; +} + +impl_ops_byte!(i8); +impl_ops!(i16, "x"); +impl_ops!(i32, "e"); +impl_ops!(i64, "r"); +impl_ops!(isize, "r"); + +impl_ops_byte!(u8); +impl_ops!(u16, "x"); +impl_ops!(u32, "e"); +impl_ops!(u64, "r"); +impl_ops!(usize, "r"); --=20 2.34.1 From nobody Wed Oct 8 19:55:46 2025 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBED525C6FB; Tue, 24 Jun 2025 22:11:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803105; cv=none; b=lGUjxbCsIU7S644NpMwZsteaaf3qf0LOdFKfkBe3HRoslKt2VtRJQVYIbkxD5Y6O/swPDWHNp074/jkiDhTpJymFeAJGSHre8gTzKL3v+z35DqOVKqxF9OW3c5m6L/GRG3QyTuebwJlwwFfbcjI3e5SPYOHgFJ/9Y1s88T1nRjQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1750803105; c=relaxed/simple; bh=QRzQkPj6599pmKka9vlWmbAK9sk/eMeNCGV0bnKtkcg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=qabGUjJxj3gyEEnBgpi1TxK3ae7z3Z/lV11pzxqWqixlLvL+nNxdsaF1670ybkB9EcydCSjZYSRBR74lsUFdUwRiSkhDyFsh9mqnXkcy7MNx1qLaMKCj2xBmTFkOdsUuKhhElkYLALQIaQpF26018H3mbNgLTyvojX7Tsfzeqb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RwRTdXCg; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RwRTdXCg" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-7424ccbef4eso4529093b3a.2; Tue, 24 Jun 2025 15:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750803103; x=1751407903; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=2zMkC9B95hVmo3huBzRRWOTMUpiB5qjyqhD5+o6Ou38=; b=RwRTdXCgxHVPHt9WNa9N4t7x8VQbNPsimYH24dXd43Jz7l2hta1nomBgek97tj4fiD vd6ow7zFUMoYlX2fqnxPKlmyV3565iCHlR8AzhcUnkD3px275SRy+e9HZNjLBQ5KTcpU G8pH92c+C1SC9hw7WsNxXsNuksveHQ3ZoZpQ30G4Z/ntLxQk3GIO9axjTM09z7JN8SDN 1svq8LgZUCVxXqmX5czelnj1jfJghvGbn0N2iC9YmB5eeJC09T9bDk3tiCrHxNKvH3HW QdX9VJN8FKpUabL30Qz3lJcpUh/L8ajUmt6DbO7Qr3zzk9KzHONH2vaRWXrWRGjWKtsJ qGDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750803103; x=1751407903; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2zMkC9B95hVmo3huBzRRWOTMUpiB5qjyqhD5+o6Ou38=; b=NuZeumH47eOqe+yY0kT45Ii8cvrUkpY2JM/MueNtoOIV4XO65b9OPdjSox7KOLAhdh TmPGIjoaJ3inopj4nQVqA+l7RXR/D5qn7ivV9E+pvyHSUtqTKHO4nMr0pp1WSaAPE+Qv gLBumyt2JnDd/EJjC7Z9Pdi4WRx8r6DBxPK2gXcu8db6uyiKCWLdRiUsdccHDDOEGm1l oRcOXan7LIgFALafYGZe23XKNYQKaAt74euJk9f1VjTL2KCVK20wiVQGINUuZpmTqJow uQrs9PBAxl3X+/qmpcFxMrcwxlN56tAEst6ELakAtZQG6J1m1aN66Q+lRykaT6m0RIxc z7SQ== X-Forwarded-Encrypted: i=1; AJvYcCVDwsQ4CG+mMtZOHJnLMeMDplP6fiYyGmQcdKgLfdF9Zv20F9lgbO8A+nJl4Biwphme5QjxMU8nkz0rIbjoaw==@vger.kernel.org X-Gm-Message-State: AOJu0YybS2mJQOQAfhf20qlgucVEGlXM3tjhi+Z5cHMLV0ERbCs2Lgpq 7X1Rsp3ghpwBO8BVfB5BVdTWp6kk+YeWwW5j2Hd+QssZyyoTr9GMUr10 X-Gm-Gg: ASbGncvUegxvkfIkNoNWE0VZp4TuRFNTIYmesF47EH1kB+PPHdW718y2Tir+AIpl1aa ttoL8gqr9AXbi1M58xNyzLY6rs0j5SkfLAF7W4J1vuUQD6Lu80EOx63/g3LGhI9roJ1JVI7ligf cT4TTQKPNomfHvzzTBkJAfs00+OXn+yNbSgIy3Zili3hZIimt8/cYQOjHpVBFtMlsymMWI0bI1h sCb3smiE8WCO4Ptmig8B7WxAQPkdnvvUAr7ldm1ZCZE8bGrwbVFgFF8LTTUkSzpXl4tXiA1CNPz qzVc0Zy+/i3NNB8q3+5utgFgbE/WSZ7U/2PDvGmJSKxupNyN24+CD2FU0Bih8E/4wf8t1WTFwk5 xhYpzHaTDP6VndRNVhR8dyTQcFHi3ZTJwj+24UppeoRra59I= X-Google-Smtp-Source: AGHT+IEzRXvMPFk+Tkm5J6SgFMC5AkOyHL1p5cr4QMS8S07R92CmifefB13pyYomizepplhvoEK0nw== X-Received: by 2002:a17:902:d58c:b0:234:d7b2:2ab9 with SMTP id d9443c01a7336-23823fcf358mr15808855ad.12.1750803103122; Tue, 24 Jun 2025 15:11:43 -0700 (PDT) Received: from mitchelllevy.localdomain (82.sub-174-224-205.myvzw.com. [174.224.205.82]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-237d8393741sm114580555ad.15.2025.06.24.15.11.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 15:11:42 -0700 (PDT) From: Mitchell Levy Date: Tue, 24 Jun 2025 15:10:43 -0700 Subject: [PATCH 5/5] rust: percpu: cache per-CPU pointers in the dynamic case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250624-rust-percpu-v1-5-9c59b07d2a9c@gmail.com> References: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> In-Reply-To: <20250624-rust-percpu-v1-0-9c59b07d2a9c@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin Cc: linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1750803093; l=3207; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=QRzQkPj6599pmKka9vlWmbAK9sk/eMeNCGV0bnKtkcg=; b=LMg9dvl4+PiuHooJCVwGOo2w2BGxVvYS+F4oaJUyJ+uCVe74yQmmnI+pUCIZjtG8VWs3ZQejN T06DBkTIpSoDOc8eOfegl6Jtj0/vPEamD9figNqDWPSdrywtF1L9+og X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Currently, the creation of a `PerCpuNumeric` requires a memory read via the `Arc` managing the dynamic allocation. While the compiler might be clever enough to consolidate these reads in some cases, the read must happen *somewhere*, which, when we're concerning ourselves with individual instructions, is a very high burden. Instead, cache the `PerCpuPointer` inside the `DynamicPerCpu` structure; then, the `Arc` is used solely to manage the allocation. Signed-off-by: Mitchell Levy --- rust/kernel/percpu.rs | 12 +++++++++--- rust/kernel/percpu/numeric.rs | 2 +- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 9659bff2d889..ffcfe02124ab 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -23,7 +23,10 @@ =20 /// Holds a dynamically-allocated per-CPU variable. pub struct DynamicPerCpu { + // INVARIANT: `ptr` is managed by `alloc` and the value of `ptr` does = not change for the + // lifetime of `self`. alloc: Arc>, + ptr: PerCpuPtr, } =20 /// Holds a statically-allocated per-CPU variable. @@ -199,9 +202,10 @@ impl DynamicPerCpu { pub fn new(flags: Flags) -> Option { let alloc: PerCpuAllocation =3D PerCpuAllocation::new()?; =20 + let ptr =3D alloc.0; let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: arc }) + Some(Self { alloc: arc, ptr }) } =20 /// Wraps a `PerCpuAllocation` in a `PerCpu` @@ -210,8 +214,9 @@ pub fn new(flags: Flags) -> Option { /// * `alloc` - The allocation to use /// * `flags` - The flags used to allocate an `Arc` that keeps track o= f the `PerCpuAllocation`. pub fn new_from_allocation(alloc: PerCpuAllocation, flags: Flags) -= > Option { + let ptr =3D alloc.0; let arc =3D Arc::new(alloc, flags).ok()?; - Some(Self { alloc: arc }) + Some(Self { alloc: arc, ptr }) } } =20 @@ -219,7 +224,7 @@ pub fn new_from_allocation(alloc: PerCpuAllocation, = flags: Flags) -> Option PerCpu for DynamicPerCpu { unsafe fn ptr(&mut self) -> &PerCpuPtr { - &self.alloc.0 + &self.ptr } } =20 @@ -227,6 +232,7 @@ impl Clone for DynamicPerCpu { fn clone(&self) -> Self { Self { alloc: self.alloc.clone(), + ptr: self.ptr, } } } diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs index e4008f872af1..1b37cc7e5c19 100644 --- a/rust/kernel/percpu/numeric.rs +++ b/rust/kernel/percpu/numeric.rs @@ -62,7 +62,7 @@ impl DynamicPerCpu<$ty> { /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU /// variable. pub fn num(&self) -> PerCpuNumeric<'_, $ty> { - PerCpuNumeric { ptr: &self.alloc.0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { --=20 2.34.1