From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA084326D51; Thu, 28 Aug 2025 19:00:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407659; cv=none; b=thLRg43BS4bVxO97h3nRr2u49c+Poasdfbb/TzYg2WxmbEc0kqNh/ktfO6lFQbLGPWbY8Nwm4sPkpVMwljODIp0vYJFCamXhzoYv1/MhkfclvJIaei1l5lQJokcwbtdnIRWJMvWmco4AwnC7CVA6KVSk2vaabI8zJnw9BTFUeSU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407659; c=relaxed/simple; bh=xrfYI1ZkHSTg2T9fsf4h4CSmtcAqiyCxcpC/fNiP8Zg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=UiL7lRozoqy7QqXwnh1o+y2xc+7zlG7dKc0mno0zSKxO21pbKi+OAP3HpUmUDkGhBoSi4sg4SU0lZ++JSUQTUY2lkMncnfPun5N8iwNReB0wg83FBCFWmOgGahbwjFnMUs+DGCbqLfXPXTWF0/rC88X43Vp5NmNvUXR0fbR9XlI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MyeEH0QV; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MyeEH0QV" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-327771edfbbso1402951a91.0; Thu, 28 Aug 2025 12:00:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407656; x=1757012456; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=xagyQTYKQ/PMa+/aB5HcbF4I+9IFEBLvR/a2RkSCass=; b=MyeEH0QVosJPEl9JBo7UJzT+Jtp3n2jaBn/S+hTXIBNEyROVGUbEyDhHrumbnSxzNH I2s3wyNXuxO+gTqhb+2XwUfxndK2KvSkLW8AIfsnZNhUgX+UnhRcSvxeB1yd0fua+G/n 7QgMZaoLbFXP0QRgrD9ZG3IhR3sAKIhJsjUROJ4IccbZQjvIWItzVIut8KK1odHtVarl 0k/uKExBEz6j+BXfLUXhmSFXv1KTIa3JxCuWlHgYMDK/ox69lQXVZnW6zUh88awngDms Tn70nxXXslzazQ9DN6xm7eHpsFgEN9jw9z10CtbsKUaG41lVn7lhhBhkHT3z7yCN8r/O gV+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407656; x=1757012456; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xagyQTYKQ/PMa+/aB5HcbF4I+9IFEBLvR/a2RkSCass=; b=flx/zQwVzmmV2K3AZrGqM5nd7tpuyRXlsmmAg7tr6Gt+/6Y7rorP+pf26JveUblNAG ndJEePRdot5R5x3EYaNRnFnbi/4XgohRAdfQd/h5tMu1Q89Db60lLXRkROc0RXhFPgC9 5gOeNkXzRex+GAb+dIMfqsZOwqsQxlykvE5GzaSHDwCS4gtR63ecle+WLPbo7e3KsWf3 GjMGHWzbv8WNc5oNXtKd6XdbM/3mYAoFReYTMRWUm/K2lTGMebv8tU/thqZmQCSNq2tI d+8vjY/q0ZgUixGLqzRJikz/6SDeTmJTWUbC5uZTp6vPhvrQ3HpkP6ibzue6PdCPB3ds t+6Q== X-Forwarded-Encrypted: i=1; AJvYcCVOCO4onF0VHXmYpaWLttxP4/yYRzkkqnvsjZHPGEwHEhGjjQXwzYasG/uABjP0d9gTPMCOb1AHt3qZDME=@vger.kernel.org, AJvYcCXz9kYXM+AcWGQVBpPHiQnIB9Y03L+U435FBQ+dhHGucMX9bZ7iNZYHbLsrBKeGuv+EGyHADhhfo1Zx/oWHpYM=@vger.kernel.org X-Gm-Message-State: AOJu0YzYwpRvrFRebo81jbM7Loo4iQr6FKJvamxRC2H50tCLRtAdtLvB 6PXgEOhf+ykVDSgQR9bR+PJAzbCgJzt22Na6aLwktqxug8J++m/Kxw7JRqh6n1Kh X-Gm-Gg: ASbGnctDYcKOaFItnMjcj2R8acEbG4Z8Mm6DM7SV3K3ZzyZ+QfemOiBcxuwhw3DuAQo 1ke0C1G81Rw8+fHF5VUD3i7klti9SdhuTNdsqo9cfznd/MJXZLgBGlX1KFjUsENUUDRYcCK3u9e 43OmLo9/5JR+JDD/iILHsqu+/bHfJ9qaG6DTaGNMyjMPV0nfiIwQ/1KgB6+HYnszLMYFkN3RDC4 ql1sQoo5EyxrX+BDJGX46kgctBm+52/eD6H6qBUQbw/d3n6XKyxjP2C3oYG3li/V1soEnfIeE/U 5CG0CFRaKBoURRULRe+uv1EvfG7QJWP4Cuy31mh1Yz03VGVquGfjMEvQrnemqjDbxP+wr/OvB74 NVPuVzQ4i1pEIiLb+bjySzkEzkubYctZMfbFAf8D94ICKzA== X-Google-Smtp-Source: AGHT+IEAvtCvavdXj+AY+/M8QNdGSGXYrE0qJtdzFxGAeh6kBsD/31fcDZSIqWtx5zI9DVEjeA1aXw== X-Received: by 2002:a17:90b:1649:b0:321:4182:2b9e with SMTP id 98e67ed59e1d1-32515e37410mr31329785a91.12.1756407655219; Thu, 28 Aug 2025 12:00:55 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.00.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:00:54 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:08 -0700 Subject: [PATCH v3 1/7] rust: percpu: introduce a rust API for per-CPU variables Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-1-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=24714; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=xrfYI1ZkHSTg2T9fsf4h4CSmtcAqiyCxcpC/fNiP8Zg=; b=M9YA0f2C9JiHjuPxHtQ2AOzf9pd50gvHz7cYbhTPcmhO+lzzK0WfSTyNYqGlrBSOW4LUZnh+5 bMt0b8R5Y6RCM5VBIRTLb9qaDJ2vwwgVP0nQ+dgyEPw7dFPP/keskKd X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Per-CPU variables are an important tool for reducing lock contention, especially in systems with many processors. They also provide a convenient way to handle data that are logically associated with a particular CPU (e.g., the currently running task). Therefore, add a Rust API to make use of per-CPU variables. Add a `CpuGuard` type that disables preemption for its lifetime. Add a `PerCpuAllocation` type used to track dynamic allocations. Add a `define_per_cpu!` macro to create static per-CPU allocations. Add `DynamicPerCpu` and `StaticPerCpu` to provide a high-level API. Add a `PerCpu` trait to unify the dynamic and static cases. Co-developed-by: Boqun Feng Signed-off-by: Boqun Feng Signed-off-by: Mitchell Levy --- rust/helpers/helpers.c | 2 + rust/helpers/percpu.c | 9 ++ rust/helpers/preempt.c | 14 +++ rust/kernel/lib.rs | 3 + rust/kernel/percpu.rs | 223 ++++++++++++++++++++++++++++++++++++= ++++ rust/kernel/percpu/cpu_guard.rs | 35 +++++++ rust/kernel/percpu/dynamic.rs | 83 +++++++++++++++ rust/kernel/percpu/static_.rs | 132 ++++++++++++++++++++++++ 8 files changed, 501 insertions(+) diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c index 7cf7fe95e41d..2fc8d26cfe66 100644 --- a/rust/helpers/helpers.c +++ b/rust/helpers/helpers.c @@ -31,9 +31,11 @@ #include "of.c" #include "page.c" #include "pci.c" +#include "percpu.c" #include "pid_namespace.c" #include "platform.c" #include "poll.c" +#include "preempt.c" #include "property.c" #include "rbtree.c" #include "rcu.c" diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c new file mode 100644 index 000000000000..a091389f730f --- /dev/null +++ b/rust/helpers/percpu.c @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) +{ + return __alloc_percpu(sz, align); +} + diff --git a/rust/helpers/preempt.c b/rust/helpers/preempt.c new file mode 100644 index 000000000000..2c7529528ddd --- /dev/null +++ b/rust/helpers/preempt.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include + +void rust_helper_preempt_disable(void) +{ + preempt_disable(); +} + +void rust_helper_preempt_enable(void) +{ + preempt_enable(); +} + diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs index ed53169e795c..ed0d5756dc55 100644 --- a/rust/kernel/lib.rs +++ b/rust/kernel/lib.rs @@ -106,6 +106,9 @@ pub mod page; #[cfg(CONFIG_PCI)] pub mod pci; +// Only x86_64 is supported by percpu for now +#[cfg(CONFIG_X86_64)] +pub mod percpu; pub mod pid_namespace; pub mod platform; pub mod prelude; diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs new file mode 100644 index 000000000000..35afcdba3ccd --- /dev/null +++ b/rust/kernel/percpu.rs @@ -0,0 +1,223 @@ +// SPDX-License-Identifier: GPL-2.0 +//! This module contains abstractions for creating and using per-CPU varia= bles from Rust. +//! See the define_per_cpu! macro and the DynamicPerCpu type, as well a= s the PerCpu trait. + +pub mod cpu_guard; +mod dynamic; +mod static_; + +#[doc(inline)] +pub use dynamic::*; +#[doc(inline)] +pub use static_::*; + +use bindings::{alloc_percpu, free_percpu}; + +use crate::alloc::Flags; +use crate::percpu::cpu_guard::CpuGuard; +use crate::prelude::*; +use crate::sync::Arc; +use crate::types::Opaque; +use crate::{declare_extern_per_cpu, get_static_per_cpu}; + +use core::arch::asm; +use core::cell::{Cell, RefCell, UnsafeCell}; +use core::mem::{align_of, size_of, MaybeUninit}; + +use ffi::c_void; + +/// A per-CPU pointer; that is, an offset into the per-CPU area. Note that= this type is NOT a smart +/// pointer, it does not manage the allocation. +pub struct PerCpuPtr(*mut MaybeUninit); + +/// Represents exclusive access to the memory location pointed at by a par= ticular PerCpu. +pub struct PerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +/// Represents access to the memory location pointed at by a particular Pe= rCpu where the type +/// `T` manages access to the underlying memory to avoid aliaising trouble= s. (For example, `T` +/// might be a `Cell` or `RefCell`.) +pub struct CheckedPerCpuToken<'a, T> { + // INVARIANT: the current CPU's memory location associated with the pe= r-CPU variable pointed at + // by `ptr` (i.e., the entry in the per-CPU area on the current CPU) h= as been initialized. + _guard: CpuGuard, + ptr: &'a PerCpuPtr, +} + +impl PerCpuPtr { + /// Makes a new PerCpuPtr from a raw per-CPU pointer. + /// + /// # Safety + /// `ptr` must be a valid per-CPU pointer. + pub unsafe fn new(ptr: *mut MaybeUninit) -> Self { + Self(ptr) + } + + /// Get a `&mut MaybeUninit` to the per-CPU variable on the current= CPU represented by `&self` + /// + /// # Safety + /// The returned `&mut T` must follow Rust's aliasing rules. That is, = no other `&(mut) T` may + /// exist that points to the same location in memory. In practice, thi= s means that `get_(mut_)ref` + /// must not be called on another `PerCpuPtr` that is a copy/clone = of `&self` for as long as + /// the returned reference lives. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + #[allow(clippy::mut_from_ref)] // Safety requirements prevent aliasing= issues + pub unsafe fn get_mut_ref(&self) -> &mut MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract, + // and the safety requirements of this function ensure that the re= turned reference is + // exclusive. + unsafe { &mut *(self.get_ptr()) } + } + + /// Get a `&MaybeUninit` to the per-CPU variable on the current CPU= represented by `&self` + /// + /// # Safety + /// The returned `&T` must follow Rust's aliasing rules. That is, no `= &mut T` may exist that + /// points to the same location in memory. In practice, this means tha= t `get_mut_ref` must not + /// be called on another `PerCpuPtr` that is a copy/clone of `&self= ` for as long as the + /// returned reference lives. + /// + /// CPU preemption must be disabled before calling this function and f= or the lifetime of the + /// returned reference. Otherwise, the returned reference might end up= being a reference to a + /// different CPU's per-CPU area, causing the potential for a data rac= e. + pub unsafe fn get_ref(&self) -> &MaybeUninit { + // SAFETY: `self.get_ptr()` returns a valid pointer to a `MaybeUni= nit` by its contract. + // The safety requirements of this function ensure that the return= ed reference isn't + // aliased by a `&mut MaybeUninit`. + unsafe { &*self.get_ptr() } + } + + /// Get a `*mut MaybeUninit` to the per-CPU variable on the current= CPU represented by + /// `&self`. Note that if CPU preemption is not disabled before callin= g this function, use of + /// the returned pointer may cause a data race without some other sync= hronization mechanism. + /// Buyer beware! + pub fn get_ptr(&self) -> *mut MaybeUninit { + let this_cpu_off_pcpu: PerCpuPtr<*mut c_void> =3D get_static_per_c= pu!(this_cpu_off).0; + let mut this_cpu_area: *mut c_void; + // SAFETY: gs + this_cpu_off_pcpu is guaranteed to be a valid poin= ter because `gs` points + // to the per-CPU area and this_cpu_off_pcpu is a valid per-CPU al= location. + unsafe { + asm!( + "mov {out}, gs:[{off_val}]", + off_val =3D in(reg) this_cpu_off_pcpu.0, + out =3D out(reg) this_cpu_area, + ) + }; + + // This_cpu_area + self.0 is guaranteed to be a valid pointer by t= he per-CPU subsystem and + // the invariant that self.0 is a valid offset into the per-CPU ar= ea. + (this_cpu_area).wrapping_add(self.0 as usize).cast() + } +} + +// SAFETY: Sending a `PerCpuPtr` to another thread is safe because as s= oon as it's sent, the +// pointer is logically referring to a different place in memory in the ot= her CPU's per-CPU area. +// In particular, this means that there are no restrictions on the type `T= `. +unsafe impl Send for PerCpuPtr {} + +// SAFETY: Two threads concurrently making use of a `PerCpuPtr` will ea= ch see the `T` in their +// own per-CPU area, so there's no potential for a data race (regardless o= f whether `T` is itself +// `Sync`). +unsafe impl Sync for PerCpuPtr {} + +impl Clone for PerCpuPtr { + fn clone(&self) -> Self { + *self + } +} + +/// `PerCpuPtr` is just a wrapper around a pointer. +impl Copy for PerCpuPtr {} + +/// A trait representing a per-CPU variable. This is implemented for both = `StaticPerCpu` and +/// `DynamicPerCpu`. The main usage of this trait is to call `get_mut` = to get a `PerCpuToken` +/// that can be used to access the underlying per-CPU variable. See `PerCp= uToken::with`. +pub trait PerCpu { + /// Produces a token, asserting that the holder has exclusive access t= o the underlying memory + /// pointed to by `self` + /// + /// # Safety + /// No other `PerCpuToken` or `CheckedPerCpuToken` may exist on the cu= rrent CPU (which is a + /// sensible notion, since we keep a `CpuGuard` around) that is derive= d from the same + /// `PerCpu` or a clone thereof. + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T>; +} + +/// A marker trait for types that are interior mutable. Types that impleme= nt this trait can be used +/// to create "checked" per-CPU variables. See `CheckedPerCpu`. +pub trait InteriorMutable {} + +impl InteriorMutable for Cell {} +impl InteriorMutable for RefCell {} +impl InteriorMutable for UnsafeCell {} +impl InteriorMutable for Opaque {} + +/// A trait representing a per-CPU variable that is usable via a `&T`. The= unsafety of `PerCpu` +/// stems from the fact that the holder of a `PerCpuToken` can use it to g= et a `&mut T` to the +/// underlying per-CPU variable. This is problematic because the existence= of aliaising `&mut T` is +/// undefined behavior in Rust. This type avoids that issue by only allowi= ng access via a `&T`, +/// with the tradeoff that then `T` must be interior mutable or the underl= ying per-CPU variable +/// must be a constant for the lifetime of the corresponding `CheckedPerCp= u`. +/// +/// Currently, only the case where `T` is interior mutable has first-class= support, though a custom +/// implementation of `PerCpu`/`CheckedPerCpu` could be created for = the const case. +pub trait CheckedPerCpu: PerCpu { + /// Produces a token via which the holder can access the underlying pe= r-CPU variable. + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T>; +} + +impl<'a, T> PerCpuToken<'a, T> { + /// # Safety + /// No other `PerCpuToken` or `CheckedPerCpuToken` may exist on the cu= rrent CPU (which is a + /// sensible notion, since we keep a `CpuGuard` around) that uses the = same `PerCpuPtr`. + /// + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> PerCpuTok= en<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&mut T` that points at the unde= rlying per-CPU variable + /// that `&mut self` represents. + pub fn with(&mut self, func: U) + where + U: FnOnce(&mut T), + { + // SAFETY: The existence of a PerCpuToken means that the requireme= nts for get_mut_ref are + // satisfied. Likewise, the requirements for assume_init_mut are s= atisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_mut_ref().assume_init_mut() }); + } +} + +impl<'a, T> CheckedPerCpuToken<'a, T> { + /// # Safety + /// The current CPU's memory location associated with the per-CPU vari= able pointed at by `ptr` + /// (i.e., the entry in the per-CPU area on this CPU) must be initiali= zed. + pub unsafe fn new(guard: CpuGuard, ptr: &'a PerCpuPtr) -> CheckedPe= rCpuToken<'a, T> { + Self { _guard: guard, ptr } + } + + /// Immediately invokes `func` with a `&T` that points at the underlyi= ng per-CPU variable that + /// `&mut self` represents. + pub fn with(&mut self, func: U) + where + U: FnOnce(&T), + { + // SAFETY: The existence of a CheckedPerCpuToken means that the re= quirements for get_ref + // are satisfied. Likewise, the requirements for assume_init_ref a= re satisfied because the + // invariants of this type ensure that on the current CPU (which i= s a sensible notion + // because we have a CpuGuard), the memory location pointed to by = `ptr` is initialized. + func(unsafe { self.ptr.get_ref().assume_init_ref() }); + } +} + +declare_extern_per_cpu!(this_cpu_off: *mut c_void); diff --git a/rust/kernel/percpu/cpu_guard.rs b/rust/kernel/percpu/cpu_guard= .rs new file mode 100644 index 000000000000..14c04b12e7f0 --- /dev/null +++ b/rust/kernel/percpu/cpu_guard.rs @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Contains abstractions for disabling CPU preemption. See `CpuGuard`. + +/// A RAII guard for bindings::preempt_disable and bindings::preempt_enabl= e. Guarantees preemption +/// is disabled for as long as this object exists. +pub struct CpuGuard { + // Don't make one without using new() + _phantom: (), +} + +impl CpuGuard { + /// Create a new CpuGuard. Disables preemption for its lifetime. + pub fn new() -> Self { + // SAFETY: There are no preconditions required to call preempt_dis= able + unsafe { + bindings::preempt_disable(); + } + CpuGuard { _phantom: () } + } +} + +impl Default for CpuGuard { + fn default() -> Self { + Self::new() + } +} + +impl Drop for CpuGuard { + fn drop(&mut self) { + // SAFETY: There are no preconditions required to call preempt_ena= ble + unsafe { + bindings::preempt_enable(); + } + } +} diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs new file mode 100644 index 000000000000..ce95e420f943 --- /dev/null +++ b/rust/kernel/percpu/dynamic.rs @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Dynamically allocated per-CPU variables. + +use super::*; + +/// Represents a dynamic allocation of a per-CPU variable via alloc_percpu= . Calls free_percpu when +/// dropped. +pub struct PerCpuAllocation(PerCpuPtr); + +impl PerCpuAllocation { + /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, + /// initially filled with the zero value for `T`. + /// + /// Returns `None` under the same circumstances the C function `alloc_= percpu` returns `NULL`. + pub fn new_zero() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call alloc_percpu; MaybeUninit<= T> is + // `#[repr(transparent)]`, so we can cast a `*mut T` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + // alloc_percpu returns zero'ed memory + Some(Self(PerCpuPtr(ptr))) + } +} + +impl PerCpuAllocation { + /// Makes a per-CPU allocation sized and aligned to hold a `T`. + /// + /// Returns `None` under the same circumstances the C function `alloc_= percpu` returns `NULL`. + pub fn new_uninit() -> Option> { + let ptr: *mut MaybeUninit =3D + // SAFETY: No preconditions to call alloc_percpu; MaybeUninit<= T> is + // `#[repr(transparent)]`, so we can cast a `*mut T` to it. + unsafe { alloc_percpu(size_of::(), align_of::()) }.cast(= ); + if ptr.is_null() { + return None; + } + + Some(Self(PerCpuPtr(ptr))) + } +} + +impl Drop for PerCpuAllocation { + fn drop(&mut self) { + // SAFETY: self.0.0 was returned by alloc_percpu, and so was a val= id pointer into + // the percpu area, and has remained valid by the invariants of Pe= rCpuAllocation. + unsafe { free_percpu(self.0 .0.cast()) } + } +} + +/// Holds a dynamically-allocated per-CPU variable. +#[derive(Clone)] +pub struct DynamicPerCpu { + // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by `alloc.0` has been + // initialized. + pub(super) alloc: Arc>, +} + +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `flags` - Flags used to allocate an `Arc` that keeps track of th= e underlying + /// `PerCpuAllocation`. + pub fn new_zero(flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: arc }) + } +} + +impl PerCpu for DynamicPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: The requirements of `PerCpu::get_mut` and this type's i= nvariant ensure that the + // requirements of `PerCpuToken::new` are met. + unsafe { PerCpuToken::new(guard, &self.alloc.0) } + } +} diff --git a/rust/kernel/percpu/static_.rs b/rust/kernel/percpu/static_.rs new file mode 100644 index 000000000000..be226dd2c3aa --- /dev/null +++ b/rust/kernel/percpu/static_.rs @@ -0,0 +1,132 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Statically allocated per-CPU variables. + +use super::*; + +/// A wrapper used for declaring static per-CPU variables. These symbols a= re "virtual" in that the +/// linker uses them to generate offsets into each CPU's per-CPU area, but= shouldn't be read +/// from/written to directly. The fact that the statics are immutable prev= ents them being written +/// to (generally), this struct having _val be non-public prevents reading= from them. +/// +/// The end-user of the per-CPU API should make use of the define_per_cpu!= macro instead of +/// declaring variables of this type directly. All instances of this type = must be `static` and +/// `#[link_section =3D ".data..percpu"]` (which the macro handles). +#[repr(transparent)] +pub struct StaticPerCpuSymbol { + _val: T, // generate a correctly sized type +} + +/// Holds a statically-allocated per-CPU variable. +#[derive(Clone)] +pub struct StaticPerCpu(pub(super) PerCpuPtr); + +impl StaticPerCpu { + /// Creates a `StaticPerCpu` from a `StaticPerCpuSymbol`. You sh= ould probably be using + /// `get_static_per_cpu!` instead. + pub fn new(ptr: *const StaticPerCpuSymbol) -> StaticPerCpu { + // SAFETY: `StaticPerCpuSymbol` is `#[repr(transparent)]`, so w= e can safely cast a + // pointer to it into a pointer to `MaybeUninit`. The validity = of it as a per-CPU + // pointer is guaranteed by the per-CPU subsystem and invariants o= f the StaticPerCpuSymbol + // type. + let pcpu_ptr =3D unsafe { PerCpuPtr::new(ptr.cast_mut().cast()) }; + Self(pcpu_ptr) + } +} + +impl PerCpu for StaticPerCpu { + unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { + // SAFETY: The requirements of `PerCpu::get_mut` and the fact that= statically-allocated + // per-CPU variables are initialized by the per-CPU subsystem ensu= re that the requirements + // of `PerCpuToken::new` are met. + unsafe { PerCpuToken::new(guard, &self.0) } + } +} + +impl CheckedPerCpu for StaticPerCpu { + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: The per-CPU subsystem guarantees that each CPU's instan= ce of a + // statically allocated variable begins with a copy of the content= s of the + // corresponding symbol in `.data..percpu`. Thus, the requirements= of + // `CheckedPerCpuToken::new` are met. + unsafe { CheckedPerCpuToken::new(guard, &self.0) } + } +} + +/// Gets a `StaticPerCpu` from a symbol declared with `define_per_cpu!`= or +/// `declare_extern_per_cpu!`. +/// +/// # Arguments +/// * `ident` - The identifier declared +#[macro_export] +macro_rules! get_static_per_cpu { + ($id:ident) =3D> { + $crate::percpu::StaticPerCpu::new((&raw const $id).cast()) + }; +} + +/// Declares a StaticPerCpuSymbol corresponding to a per-CPU variable defi= ned in C. Be sure to read +/// the safety requirements of `PerCpu::get`. +#[macro_export] +macro_rules! declare_extern_per_cpu { + ($id:ident: $ty:ty) =3D> { + extern "C" { + static $id: StaticPerCpuSymbol<$ty>; + } + }; +} + +/// define_per_cpu! is analogous to the C DEFINE_PER_CPU macro in that it = lets you create a +/// statically allocated per-CPU variable. +/// +/// # Example +/// ``` +/// use kernel::define_per_cpu; +/// use kernel::percpu::StaticPerCpuSymbol; +/// +/// define_per_cpu!(pub MY_PERCPU: u64 =3D 0); +/// ``` +#[macro_export] +macro_rules! define_per_cpu { + ($vis:vis $id:ident: $ty:ty =3D $expr:expr) =3D> { + $crate::macros::paste! { + // We might want to have a per-CPU variable that doesn't imple= ment `Sync` (not paying + // sync overhead costs is part of the point), but Rust won't l= et us declare a static of + // a `!Sync` type. Of course, we don't actually have any synch= ronization issues, since + // each CPU will see its own copy of the variable, so we cheat= a little bit and tell + // Rust it's fine. + #[doc(hidden)] + #[allow(non_camel_case_types)] + #[repr(transparent)] // It needs to be the same size as $ty + struct [<__PRIVATE_TYPE_ $id>]($ty); + + impl [<__PRIVATE_TYPE_ $id>] { + #[doc(hidden)] + const fn new(val: $ty) -> Self { + Self(val) + } + } + + // Expand $expr outside of the unsafe block to avoid silently = allowing unsafe code to be + // used without a user-facing unsafe block + #[doc(hidden)] + static [<__INIT_ $id>]: [<__PRIVATE_TYPE_ $id>] =3D [<__PRIVAT= E_TYPE_ $id>]::new($expr); + + // SAFETY: This type will ONLY ever be used to declare a `Stat= icPerCpuSymbol` + // (which we then only ever use as input to `&raw`). Reading f= rom the symbol is + // already UB, so we won't ever actually have any variables of= this type where + // synchronization is a concern. + #[doc(hidden)] + unsafe impl Sync for [<__PRIVATE_TYPE_ $id>] {} + + // SAFETY: StaticPerCpuSymbol is #[repr(transparent)], so w= e can freely convert from + // [<__PRIVATE_TYPE_ $id>], which is also `#[repr(transparent)= ]` (i.e., everything is + // just a `$ty` from a memory layout perspective). + #[link_section =3D ".data..percpu"] + $vis static $id: StaticPerCpuSymbol<[<__PRIVATE_TYPE_ $id>]> = =3D unsafe { + core::mem::transmute_copy::< + [<__PRIVATE_TYPE_ $id>], StaticPerCpuSymbol<[<__PRIVAT= E_TYPE_ $id>]> + >(&[<__INIT_ $id>]) + }; + } + }; +} --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50E4132BF2E; Thu, 28 Aug 2025 19:00:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407658; cv=none; b=TLR0q3bVItUZLBkuBZbrkleOt+d0Pie/0zuDjkOJvreYpESQFDDcfV3i9SJnAKI18DVajsDz8vuGottF/LFKMagQuBsWWKa1yjqYYLBHZ9BfnH3Aoxd5S+aDw04IUobDJEAovEQSJgxYKuJ3l4plICQm1JXqUPPCo9/QwGxNxyk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407658; c=relaxed/simple; bh=pd7OPcbrPdAWI/7kgDpMFHa6ITv6ppIv2SVJeXLx/4I=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=N+fc1s6bbedgBCx8qpBGDjqzzEjY1HDIwexI72q0EBniC+nzD6ZcvSqvXaB/8ARqYvONGDJWL6oJn/TgWeVbG7nVRjppT3xmMuBqPA41uJ9+Qu5HDXS7pKSzwT3VXQTbfna2n6oBCBtovbc7BNrhjqUJw2epn4xYoRYfIXiGOUw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YFc0p7nA; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YFc0p7nA" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-327b657924dso794134a91.0; Thu, 28 Aug 2025 12:00:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407656; x=1757012456; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=1ZhDUhaFQNfQlBtAcofKP9r1jz8p+0mi3JI78z3ft6A=; b=YFc0p7nAhg/xQuglDzFbT5NiUHgqon5JW6o/Nl7t0AeRyboh0se/ikxuKjBJ58BEHf 12fTfljQtljhy0ZOgyHCk1Dnd8PBao/T6DCwGXqE1WcnVA4rcmxqqfRsHTKccaz2dmAR 8QU3jcTPv7ELMhJPYrsB8uQKhSU+NQgaymdqXSncfGRKYMmh8m20z0JTiNmdTEAfQUoy qtLf6Cx8dkGwXQMDEThnH3ungB4dE8b7cocopZRD3PKgLsc/OeB39pNrUmm0IZOr2q6j dI6XqWxlAU9jRpOqaw6+HOQtMJQsa040uJKOzXUrTg0NCOP5E5n0I3KzH6K5tkvNFHIy 2+2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407656; x=1757012456; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1ZhDUhaFQNfQlBtAcofKP9r1jz8p+0mi3JI78z3ft6A=; b=hpILNBjwOAJnfGjovXXvWeSwMhfp3OX53NnuOeGJQs1RsPaMNEyLCDv2I0l0FBqD7p svpLDtgTeMQFPVUT8ETODWs6IuKi/HLMnY5QzsoGV9J8ZeWY/RZodwHroUZvj8/vcn5h WS1675gmflS6epjyWHSqPzi540hK62FlkyV4iClq3+Ilpx24cDIsUJEP8XWaaXu9NiYZ HDdL9YrgCKtEFPzRxwFvtAffpMDvM1MYlS9yNnPQvnERAchZ/BdQv7GTrvckLv98QGgM ZZy047Zt752M6+8seBaKrSyEYZgIQU2q3wcMpk5ril6EC2+/5JcU7DYTUCnAwvw8gfQw byHA== X-Forwarded-Encrypted: i=1; AJvYcCVvX+2NIQn8XqbaAJ5CXx453BP9qOKkmNxPlxYFPuax/1bxICc8E6ttfgERdIKIfrMVF3qwmxGT/4oFi7E=@vger.kernel.org, AJvYcCX0+kaTnnFpGk0mndSq63CMjat+EP/9dkDYSSehUt7QOQWdvH5S1LDrUtX8Wu+LOQQsuZ76mmrBbFVUOxHkk88=@vger.kernel.org X-Gm-Message-State: AOJu0YxuJEVbI54DRlQirfkIVA/9ZDj21XlYSx5y2LSIf8hgq+LBtp4r cRbdoS5rSSMrvfUxJkxHCF1SIm9bu38Z2TYdk82Wu+1OapYlLvlh9xpY1Xz+7m4V X-Gm-Gg: ASbGncv26hg0qoWBgMI2FGh4SFROKnw/jiT7YlwwFajGx03qGKhK0ZCfZ1LBkCtnIDU rIQrMqtIO13rMwZSffn2yxeiM4kOWQqKQ/EFt5yzJZM/Nww0WfmxHZk6LTACWqlonDiNhKEVsPj moHAyWj9LL/FZMjI+FQbn3Eq6awLx2dFu6c9mOO6wp8U1g03DGeMH9k2UWOimdk2BQjo4GBpWsv dLSZtPGFj/RV1SAnZIfpZUKEikCFj1KmRuaoTZ4cgrZ1AuETK9azee6Sufkjj1DTXuMG29qTNOs fm9tZj0vNB+Geec8iH2V8oGc7wjmDPJIY3nidHVYJHqWZ1GlUaW49Sg3Xch4nGY/rDuxuC++Dxc cCnBFGetQDBQZo5xcIOvDS4kDJvkNYncEOOhaUe4cavXp+6/F/b2JWmYXQ0WwVDs93kE= X-Google-Smtp-Source: AGHT+IHo7oikH+vFaY9+hLdTEQynZ5bmtJTwKQPd/bkYuk10/5MoVZOnXrdbi27adYGqQTrDB9+UAg== X-Received: by 2002:a17:90b:5101:b0:313:2206:adf1 with SMTP id 98e67ed59e1d1-32515ec3611mr32905500a91.4.1756407656141; Thu, 28 Aug 2025 12:00:56 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.00.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:00:55 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:09 -0700 Subject: [PATCH v3 2/7] rust: percpu: add a rust per-CPU variable sample Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-2-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=8590; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=pd7OPcbrPdAWI/7kgDpMFHa6ITv6ppIv2SVJeXLx/4I=; b=4mAs3HCOL9pJ49x1leBl0+kcOiuQILdFXWsH7vNLoFF1Wq1q/MBWhoCiT2WAwz0Md1dx4ui2m v3yZReoHe2iAs8mOLddeXU1f7l6uG4C5ye4DJW8gUYzwd2VbteBC/J8 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add a short exercise for Rust's per-CPU variable API, modelled after lib/percpu_test.c Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 6 ++ samples/rust/Kconfig | 9 +++ samples/rust/Makefile | 1 + samples/rust/rust_percpu.rs | 163 ++++++++++++++++++++++++++++++++++++++++= ++++ 4 files changed, 179 insertions(+) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index a091389f730f..8cc01d094752 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -1,9 +1,15 @@ // SPDX-License-Identifier: GPL-2.0 =20 #include +#include =20 void __percpu *rust_helper_alloc_percpu(size_t sz, size_t align) { return __alloc_percpu(sz, align); } =20 +void rust_helper_on_each_cpu(smp_call_func_t func, void *info, int wait) +{ + on_each_cpu(func, info, wait); +} + diff --git a/samples/rust/Kconfig b/samples/rust/Kconfig index 7f7371a004ee..23e35d64ac78 100644 --- a/samples/rust/Kconfig +++ b/samples/rust/Kconfig @@ -105,6 +105,15 @@ config SAMPLE_RUST_DRIVER_AUXILIARY =20 If unsure, say N. =20 +config SAMPLE_RUST_PERCPU + tristate "Per-CPU support" + depends on m + help + Enable this option to build a module which demonstrates Rust per-CPU + operations. + + If unsure, say N. + config SAMPLE_RUST_HOSTPROGS bool "Host programs" help diff --git a/samples/rust/Makefile b/samples/rust/Makefile index bd2faad63b4f..8a34d9d74754 100644 --- a/samples/rust/Makefile +++ b/samples/rust/Makefile @@ -10,6 +10,7 @@ obj-$(CONFIG_SAMPLE_RUST_DRIVER_PLATFORM) +=3D rust_drive= r_platform.o obj-$(CONFIG_SAMPLE_RUST_DRIVER_FAUX) +=3D rust_driver_faux.o obj-$(CONFIG_SAMPLE_RUST_DRIVER_AUXILIARY) +=3D rust_driver_auxiliary.o obj-$(CONFIG_SAMPLE_RUST_CONFIGFS) +=3D rust_configfs.o +obj-$(CONFIG_SAMPLE_RUST_PERCPU) +=3D rust_percpu.o =20 rust_print-y :=3D rust_print_main.o rust_print_events.o =20 diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs new file mode 100644 index 000000000000..98ca1c781b6b --- /dev/null +++ b/samples/rust/rust_percpu.rs @@ -0,0 +1,163 @@ +// SPDX-License-Identifier: GPL-2.0 +//! A simple demonstration of the rust per-CPU API. + +use core::cell::RefCell; +use core::ffi::c_void; + +use kernel::{ + bindings::on_each_cpu, + cpu::CpuId, + define_per_cpu, get_static_per_cpu, + percpu::{cpu_guard::*, *}, + pr_info, + prelude::*, +}; + +module! { + type: PerCpuMod, + name: "rust_percpu", + authors: ["Mitchell Levy"], + description: "Sample to demonstrate the Rust per-CPU API", + license: "GPL v2", +} + +struct PerCpuMod; + +define_per_cpu!(PERCPU: i64 =3D 0); +define_per_cpu!(UPERCPU: u64 =3D 0); +define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); + +impl kernel::Module for PerCpuMod { + fn init(_module: &'static ThisModule) -> Result { + pr_info!("rust percpu test start\n"); + + let mut native: i64 =3D 0; + let mut pcpu: StaticPerCpu =3D get_static_per_cpu!(PERCPU); + + // SAFETY: We only have one PerCpu that points at PERCPU + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut i64| { + pr_info!("The contents of pcpu are {}\n", val); + + native +=3D -1; + *val +=3D -1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D -1); + + native +=3D 1; + *val +=3D 1; + pr_info!("Native: {}, *pcpu: {}\n", native, val); + assert!(native =3D=3D *val && native =3D=3D 0); + }); + + let mut unative: u64 =3D 0; + let mut upcpu: StaticPerCpu =3D get_static_per_cpu!(UPERCPU); + + // SAFETY: We only have one PerCpu pointing at UPERCPU + unsafe { upcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| { + unative +=3D 1; + *val +=3D 1; + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 1); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D 0); + + unative =3D unative.wrapping_add((-1i64) as u64); + *val =3D val.wrapping_add((-1i64) as u64); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + + unative =3D 0; + *val =3D 0; + + unative =3D unative.wrapping_sub(1); + *val =3D val.wrapping_sub(1); + pr_info!("Unative: {}, *upcpu: {}\n", unative, val); + assert!(unative =3D=3D *val && unative =3D=3D (-1i64) as u64); + assert!(unative =3D=3D *val && unative =3D=3D u64::MAX); + }); + + let mut checked_native: u64 =3D 0; + let mut checked: StaticPerCpu> =3D get_static_per_cpu= !(CHECKED); + checked.get(CpuGuard::new()).with(|val: &RefCell| { + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + + pr_info!("rust static percpu test done\n"); + + pr_info!("rust dynamic percpu test start\n"); + let mut test: DynamicPerCpu =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 1); + on_each_cpu(Some(check_percpu), (&raw mut test).cast(), 1); + } + + pr_info!("rust dynamic percpu test done\n"); + + // Return Err to unload the module + Result::Err(EINVAL) + } +} + +extern "C" fn inc_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| *val += =3D 1); +} + +extern "C" fn check_percpu(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + // SAFETY: We don't have multiple clones of pcpu in scope + unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| assert!(= *val =3D=3D 4)); +} --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE9693375C3; Thu, 28 Aug 2025 19:00:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407660; cv=none; b=iwla7CpPMWzVhu/h21nVESPHyN38QFgtAS63gyR9pr23v5csFFxf3bsAg30kovvvRMsXe9NsFi+pe9hVHO/ar/0IDZy74ZAkBQZipB+av/ETxM3KGSAX4lzsExy3QYk5wiZDptYsuyV0zjpf6qveoiaPPCqFDKozk9zvHORjwnQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407660; c=relaxed/simple; bh=OtpCqlB9e1zWex8jQODuqGl1cz5sc9yZWh45+Mzno2g=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eZMFaRVOw9ehe+JadLorN2BOtS56EGLcsmMsvSsaKV2BjGqrwnLIt/SGC8HPTLxbL5Mk5WGpHQ6YKhaDvgwlbmMeH24WXx6X46l/KtFcGXNOVUxt9yHIibwbPO1Ne3xuvYcXtiyCrzJNgL3A4sL04ouPjyxv3HAcX9tzlqZskIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SfhAHQY9; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SfhAHQY9" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-327b657924dso794161a91.0; Thu, 28 Aug 2025 12:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407658; x=1757012458; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=kuV95K9JyY8TZOXbZAHoweF1aWF6QJFXwWOQdj7kvnY=; b=SfhAHQY9SG+LaSwqsqvowXgfSOomhbfD4wnV5HmFr3V367xlztHHGURk/cgviFruYt X8bSoBcDeTZMTqLyD3nFKiv3puOAHvTC1d7tQG+Im2b6gtUiPNzsmpDpnos7kR8h/m5O MfufvYtXEKS9lyDw/WtdNB/Voprx6Rh5nhxG2lBmScB/AjpgrjDjsEGBszGcM+LAxmUE xWrtAGi54liFU1aO8aIQtx/gi8/fQo3q5pUOYcma78h8OtUcrFgA5eGWIStAdJq0cLUY 9q1FotFPnKt74BurbWhJiRYLfQIueuk1+M2DqPYc4LaWT1JON/2F3DGx6kzHbFJFQPkH A/Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407658; x=1757012458; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kuV95K9JyY8TZOXbZAHoweF1aWF6QJFXwWOQdj7kvnY=; b=ktBCJG6NlfF8x6nInT/b+F066Hd1E/gKu7Wqs/Y/NU2JLj/QUovRSe7UOSqe48P1m8 TnVGE91w/Cc7Yk8Sr1bqzaG4Fq9wuWSD4KrjJZPzPKNrRKDVZm69boGj4sIyRl1S1P5p io6zqIBGwC4SF7fJM83mY3i8o50QbK8ukyOdNrVPS0oanh1vAVEPgmFtlN2e1AAB19/L vtpYKUX8h46u3fNFNziJS3qh9rZI/C8c0Zv3VsU/kCCIukYchRs1BJ7IEn6XosNGZOc7 99C1s1GgTUAqgDxk5DU6gjB5KLQrbR+Yj0/AcgApVcAIMYbY7Lz8f/oYMenTSqgNiRj8 4Svw== X-Forwarded-Encrypted: i=1; AJvYcCVJeeYGR+rr6cWPZdD3Bu7Y3BexkXjtG06RMJod7roB1eMJPN4UTE29NO/P9Lp7FExd2J0C7ljB3YjP8L0=@vger.kernel.org, AJvYcCVoqsyjSAnj0HclSv7IctRiCzvzm5V4O52XtMbZWj9FJbwZ5kIPuo4i7L5PYMoVCGt0kHYL2Rpipm6G0/Ccshg=@vger.kernel.org X-Gm-Message-State: AOJu0YzyAzILdw8glkSozPAGpGX8l44G+r7hIGyiKDAQl9fnAwDDXyTj 2OK3VdxOCZ/tRxlMzt5y8G09gY7VpiY7uO0sxHr8l/+m1PrsQuvNBt4YoqbPMlhz X-Gm-Gg: ASbGnct7Xy5oa0+xaEG1ked70rh3+Og6Yxx0YffJtBA5IRTj4jJXHxGB1/1vh47/uRe mtNB0KXT5E7JXhGNMHMdOl+GazcBA8epCSW4eFrd7+B8sU4HXdQvoO5fwYaBM3S//bvrJWLTcV6 D7ffPWjB7ZSv7Mzuade15a9GSe0IaDa6AS8D4LFvZHsdw0/DzQaNv0l3icMaGo3mFWIeTQxXXJ3 uloWGzLEkfvyLz2qr3Mhw9fEAOeuRDIRvXB7jXhuiq7453ywTj6/qkM+2Yh5AuvqgUycjUuRcQn WGWSgEv+3VB40Dx/vnrYX/rC+psFv58GiEN/Jhc6cZcSvvpDoEwFTsMf5CiwKdopmcBfXn061J5 7G4wwTjIn/7TKSzCcY9QZaq/7V1k+agzH0xIpxiA8I5UkFg== X-Google-Smtp-Source: AGHT+IHmhZQt+LgvfoXPy01Ao1mL5xCYpm5wePCqKk17/kEIhhYnOE66ySAPA1ychd2bemd1EStE9g== X-Received: by 2002:a17:90b:2648:b0:327:83e1:5bf with SMTP id 98e67ed59e1d1-32783e10b79mr7615856a91.28.1756407657841; Thu, 28 Aug 2025 12:00:57 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.00.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:00:56 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:10 -0700 Subject: [PATCH v3 3/7] rust: cpumask: Add a `Cpumask` iterator Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-3-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=2580; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=OtpCqlB9e1zWex8jQODuqGl1cz5sc9yZWh45+Mzno2g=; b=Q2FpQBwE86VcGQRhHVGrqBBJHXeoNsMFqG0rXzfh0HY7/r2ixF2OqX8DlbwQU04IWLevtgaFs K718VADFXt1CjEvHDLa8XOH/wgma/rXuoAzd1eGF+iseEpYpZNWDfXJ X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add an iterator for `Cpumask` making use of C's `cpumask_next`. Signed-off-by: Mitchell Levy Acked-by: Viresh Kumar --- rust/helpers/cpumask.c | 5 +++++ rust/kernel/cpumask.rs | 48 ++++++++++++++++++++++++++++++++++++++++++++++= +- 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/rust/helpers/cpumask.c b/rust/helpers/cpumask.c index eb10598a0242..d95bfa111191 100644 --- a/rust/helpers/cpumask.c +++ b/rust/helpers/cpumask.c @@ -42,6 +42,11 @@ bool rust_helper_cpumask_full(struct cpumask *srcp) return cpumask_full(srcp); } =20 +unsigned int rust_helper_cpumask_next(int n, struct cpumask *srcp) +{ + return cpumask_next(n, srcp); +} + unsigned int rust_helper_cpumask_weight(struct cpumask *srcp) { return cpumask_weight(srcp); diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index 3fcbff438670..b7401848f59e 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -6,7 +6,7 @@ =20 use crate::{ alloc::{AllocError, Flags}, - cpu::CpuId, + cpu::{self, CpuId}, prelude::*, types::Opaque, }; @@ -161,6 +161,52 @@ pub fn copy(&self, dstp: &mut Self) { } } =20 +/// Iterator for a `Cpumask`. +pub struct CpumaskIter<'a> { + mask: &'a Cpumask, + last: Option, +} + +impl<'a> CpumaskIter<'a> { + /// Creates a new `CpumaskIter` for the given `Cpumask`. + fn new(mask: &'a Cpumask) -> CpumaskIter<'a> { + Self { mask, last: None } + } +} + +impl<'a> Iterator for CpumaskIter<'a> { + type Item =3D CpuId; + + fn next(&mut self) -> Option { + // SAFETY: By the type invariant, `self.mask.as_raw` is a `struct = cpumask *`. + let next =3D unsafe { + bindings::cpumask_next( + if let Some(last) =3D self.last { + last.try_into().unwrap() + } else { + -1 + }, + self.mask.as_raw(), + ) + }; + + if next =3D=3D cpu::nr_cpu_ids() { + None + } else { + self.last =3D Some(next); + // SAFETY: `cpumask_next` returns either `nr_cpu_ids` or a val= id CPU ID. + unsafe { Some(CpuId::from_u32_unchecked(next)) } + } + } +} + +impl Cpumask { + /// Returns an iterator over the set bits in the cpumask. + pub fn iter(&self) -> CpumaskIter<'_> { + CpumaskIter::new(self) + } +} + /// A CPU Mask pointer. /// /// Rust abstraction for the C `struct cpumask_var_t`. --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9724B33EB17; Thu, 28 Aug 2025 19:01:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407662; cv=none; b=BoA+lzNiMR6M4XcLmatMcl5Tmyklb+4ikrdS+WpQHNLeamfZnN4yGR5t5Lhu3jwSkb6adDM2r2MSI2JOVRITLhqMApU2TL1AgRfSVP3QkFGGaKrFdi3PyU4eglVKWLQeIOkNdOQhr+axirWYc4eQzt3KC6luWfFgRkIw6XCLs6c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407662; c=relaxed/simple; bh=7jLeexJ2MkarSDLXB95v3y9TJT2OgU8sLC2vrrP6MLQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JrzzCrWtNutILDEgGJb1vb8wdixKplTp82A7/ZUSuITEPFTwCyVj6pTIJofBNekPMK49Lc8KbfbUDoVKsE7CN4Vf86UW9qUC25g20EZjhMKMn2XiI/PwTMcHdEGy9YVmrWQnEkO3ca2GH1pQFN8/WsNd2VqnQnvVEykGttzCIag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DrKBn7nv; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DrKBn7nv" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-32326793a85so1126422a91.1; Thu, 28 Aug 2025 12:01:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407660; x=1757012460; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=Bvt8V2L0ZNh+boeq/LSZQ3FvXwvuba2eMR3cn35JyVE=; b=DrKBn7nvFPPxJbVuUiHZQup0F5x3Nd20AhnzhcZII0ZZU83Qh1i0kNj8frumNkrR1a QtUzARz9bnikWjVwM20nQO52lLb9zWeZenz4B3TDGjK3QNQg8dbmJDnQUM1sI+jfufei RYgPu7QJAxLAc/eb1SJbhmnz73coSLjQJgWl76eGMjfKR/z6HWesUmXOMc85xOf5V0hr 2RKi5AFxDsECu2sk/XP15q+Nn1dD2nTDqNdaaYZ2qnpXpDINA/uGzD/LJiiHRuJKETq4 KbfFMdfXDlJdpIOdNYdnA1Y26cvyk6B1ocDDQLy2ld/hxmrb5hmOOqJ/+O9liLOZXcvs 18mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407660; x=1757012460; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Bvt8V2L0ZNh+boeq/LSZQ3FvXwvuba2eMR3cn35JyVE=; b=bGSB2FIVu6+ffUHSIJ3ADDrqTTbZd2oEa4Fx/iAn3pmETlwe2wpBG6ruOe2dUgK5Yn 3yqxCeUQgLue32qU6EP+uG3elV2V7ud969soDCgg0+bP+uQQAmAf8XL1YuGeNCRvZorM 1Nzt4s9NpO4YGMKm2OE1a3rVrb8cic3BWmk4DoxHagNZJce5P+kRYKhqnrQrVcWUHj2Q Pc7tP0rQkjWIzhlsDDXPOORrElOqTDNhpFV1Kpha0v1tiz2r2YWEgiGW5gnygxOEj4Yo iozHeq5xnsrduKzrwan0H23p5f1zen9PfyxhprQzs3AaWK8a1b87qa6xM2tUdixRovld mXlw== X-Forwarded-Encrypted: i=1; AJvYcCURbEIz0b/9dzEfK3mB5AnoCHEgxnrEJBmeRoTjShwAaqGYAJq007WYJHbDOMiHPaAHs14+SZLQ1tmo1BSzGr4=@vger.kernel.org, AJvYcCVv/7P+kZ62jLYSXmKsfzapxJF2Lk1+yJKou7e48d71m3Ve1mGD+oFCFY+Pk5iL3KhuqKuDh8AeS09UmGQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yw/2QKWj/L9uvm50LjIXIBJWbJEcI6fqn7TFkmOuAWPwjh0RI1j V6yaqf26EZ/P9ip8OLextbt/6+MP2XMjsovA4hl/4heJ95DReQGnZ+iWBt3aixM9 X-Gm-Gg: ASbGnctIC6n/fz/shfLMkj7xYAjGB6njIewkuTkQW8cSSySQbZoq9WOc1/DYzC//Mzc L38q0wqlYCBCEm7HJsJOr7b1aZXXVNe/1+kvOScLsu5r2d4tWuRWvpv0905YVLz6yPRK3QCMyno gcFNYL3dJQPzyEYVF5BNN2WCgu3irZlyqDbvAuokrSdY+8BFT9xScA2Wq2C4sMaY6IL4yOnNfnI aDLckYCZ9GheeIskmIrlEJdyrMilAsBDXJ8J4/X5zx2rdMVN3P8sR2aMa0VJjXtslCxDC2wi4kR bgT+Jyzsv3hu86URITuz5X5aUyVLeUzuq+MQE/XGpKuO+Zv6y8+N4+rGpTWpNJHmLymOJzlD2vC 2Y/pXJjY3RTL5llbpbqqOWFqqufsQISENF16xy0lMQwwmUQ== X-Google-Smtp-Source: AGHT+IGQ0nXDwQcd/znOgAHZEvb/bzRjgwuKUmEsfOnNkUUtXy/qInCtHGgW4wGogHPA6Xq9dBOiiw== X-Received: by 2002:a17:90b:2810:b0:325:4c48:af54 with SMTP id 98e67ed59e1d1-3254c48af7fmr24907043a91.11.1756407659077; Thu, 28 Aug 2025 12:00:59 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.00.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:00:58 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:11 -0700 Subject: [PATCH v3 4/7] rust: cpumask: Add getters for globally defined cpumasks Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-4-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=3131; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=7jLeexJ2MkarSDLXB95v3y9TJT2OgU8sLC2vrrP6MLQ=; b=zdA4F+LZY4slMG5Ve/I8Wi53or1iBeA2t76HP1S/w0KxfOdIOk6rZ4Cn8dc+CTq5sE+G9IGfS wDLwFlpvR9NCANhbfUBEg7Q4L7XNwOUKLYnJTuUUrEqPZkM8zn1ec0T X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add getters for the global cpumasks documented in `include/linux/cpumask.h`, specifically: - cpu_possible_mask - cpu_online_mask - cpu_enabled_mask - cpu_present_mask - cpu_active_mask Signed-off-by: Mitchell Levy Acked-by: Viresh Kumar --- rust/kernel/cpumask.rs | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/rust/kernel/cpumask.rs b/rust/kernel/cpumask.rs index b7401848f59e..ca9aa6875b4a 100644 --- a/rust/kernel/cpumask.rs +++ b/rust/kernel/cpumask.rs @@ -77,6 +77,52 @@ pub unsafe fn as_ref<'a>(ptr: *const bindings::cpumask) = -> &'a Self { unsafe { &*ptr.cast() } } =20 + /// Get a CPU mask representing possible CPUs; has bit `cpu` set iff c= pu is populatable + #[inline] + pub fn possible() -> &'static Self { + // SAFETY: `__cpu_possible_mask` is a valid global provided by the= kernel that lives + // forever. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_possible_mask)= } + } + + /// Get a CPU mask representing online CPUs; has bit `cpu` set iff cpu= available to the + /// scheduler + #[inline] + pub fn online() -> &'static Self { + // SAFETY: `__cpu_online_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_online_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_online_mask) } + } + + /// Get a CPU mask representing enabled CPUs; has bit `cpu` set iff cp= u can be brought online + #[inline] + pub fn enabled() -> &'static Self { + // SAFETY: `__cpu_enabled_mask` is a valid global provided by the = kernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_enabled_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_enabled_mask) } + } + + /// Get a CPU mask representing present CPUs; has bit `cpu` set iff cp= u is populated + #[inline] + pub fn present() -> &'static Self { + // SAFETY: `__cpu_present_mask` is a valid global provided by the = kernel that lives + // forever. Since we wrap the returned pointer in an `Opaque`, it'= s ok that + // `__cpu_present_mask` may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_present_mask) } + } + + /// Get a CPU mask representing active CPUs; has bit `cpu` set iff cpu= is available to + /// migration. + #[inline] + pub fn active() -> &'static Self { + // SAFETY: `__cpu_active_mask` is a valid global provided by the k= ernel that lives forever. + // Since we wrap the returned pointer in an `Opaque`, it's ok that= `__cpu_active_mask` + // may change its value. + unsafe { Cpumask::as_ref(&raw const bindings::__cpu_active_mask) } + } + /// Obtain the raw `struct cpumask` pointer. pub fn as_raw(&self) -> *mut bindings::cpumask { let this: *const Self =3D self; --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A236340DA2; Thu, 28 Aug 2025 19:01:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407663; cv=none; b=Ict7vesxPxVq86Z6an+bwi3UctXPg0AwVmx5IYzP22fM6DJOgiZGiSHaIVxWaU/wNUcJvCPhpVwpf4GA/LEvDQStMAM277hl5MI2bccHEsZb38uZRXbek//x/Nyg5fa81+Rn5rLpIRNv1UVC+tgT2Fddv+EKmv7NYAG6DQm+N90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407663; c=relaxed/simple; bh=+lEJbTpkz2WbxbYDrCnnJg9tQe8ptqmkgyWr5/FAI6E=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Uqubia+GsLk0ETk+2RRZ0YiUvZ5IGRRHSA4vFAOve9sImnJeul/H4F4Tj60yLZnVtxfGJLAqGF5wQxMJ+7pfvD+7PUX3CWYQkJCtvrADmoUODkjqLA14NqruTYEvKuXIUJY0ZF9MS5fb/Q0PY2nM7SsYd8A4Bb7M5dqnoSwaYpQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ezHTKu/E; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ezHTKu/E" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-325393d0ddaso1032201a91.3; Thu, 28 Aug 2025 12:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407661; x=1757012461; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=MV4GmSdkUNmPQjyOi2VED+Xs96RqoKvGI46q2QRv97g=; b=ezHTKu/EtcbZTtQuCmyM+rW8pXq1hrxD+493tRWaaK/v4SalfhaSK4Z+SszKRkuaC1 U0jPxa9EYUuMqyrK1dezjV+J5PdLltPkA2J3pWphrNi5yofNPz+OBXBEhpCgJk8jh9xF lzQpoHru8puTMxm4XBqbUGvp+BQw3ZGvgKyBxbX6XNJV+XwsuZqeyl+Q/Rgx/KGPS23M L0IfbQ/i85XRGMazRGw+YGMLbAziYzKK0dh4YnZ/pXDnljpb+GYCYkZXiEmdHNsTWxAr c+QXF8Y2O3euT+85EOBfDP7bbiRPzmABebeATo2+kOXhXpA2n3POFEwWT/AyR79J2Prb LzGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407661; x=1757012461; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MV4GmSdkUNmPQjyOi2VED+Xs96RqoKvGI46q2QRv97g=; b=DLQ08DqlwVVBrhgqg3odjtUIlnZiaYtM8g507uHhYpQz2q19s8nay5YuYzRGyqdg8Z lVQtEwXrdV9R2lnyn8Xvb08HebtVNDMbnnKUVfKybHjQ57r6Uu6MCl44TJARHJflZaTU IRpcrVT4doqPnzIDh7B9+9uiquQeq9pAIp6Z7ZtjadDalaOKoXvo9TzN1Iskl42/gbp9 ENrMXeuXzvEH0F+ozbpqReqMPNb5b0LriYUYAFBApZUV/+0fAF8rg8UwqNuMkzhdAMe0 xdDDxpDVEL7v2lkgBmdl531xjN4nH0coyZFTxHcWIYNKRl4jb41dgAfEQOjtc2gNhpCT G1Zw== X-Forwarded-Encrypted: i=1; AJvYcCVhyBX4w9kmYGZ8j7SjaBOsHi/Ml3hGcsKw8h7xqZ+n4GaskrcKI7od3+soITwlEPLgpBryopJ9ikyvDDuf4qo=@vger.kernel.org, AJvYcCXnpX6sDcecbRB5pHvPhRFIHh5JfPIjTo6Eluvoo2Pxbnsjl83XxD5j3fAH6vkuqAzcjCz5a+Jaetjfszk=@vger.kernel.org X-Gm-Message-State: AOJu0YwI2y2eRKotuseSHH+1rvwVO9veh0Ekhi7SYjPjqNQjbQ94ues8 yvY/+4o7gOe+lCkLyatzaEvaM8AKV8CtzCOt6FhEjvAFnj0d9ngm/gkpvedaI1nJ X-Gm-Gg: ASbGncsMvCZ4tp5/xciTJ8JkCeqcorrTpfoyFTkiv0vMt5jeYqPiwMr1/htwOJkIXfk eV/o3+EIoVKh5tkw3UzFY3mJambat9tnlYk/R8hgDuMHgIA1TwWQDft47jAsQLC1ivjHBDIIgQF tvCeRAmm+toEUG5WX8Vk7t12IvQ+swa+RJwfU7eXjMLPse0NA5SrZQskqu4o8K5WQpI9bne4uPI 0svuT+ePce7FiwocYz2KgR+4nGUZn1yr7bzSifm6Yj+N7vgdKJa3aN5hjT433gQkdfhJlvobLUe bxJ/X3ykWsOG1i85898bDTufO/Vk6+M3jbaAaJFLj/DghPqb+1sxdOyuq4fHT0v+Ez3cPHvQOc6 nymYV5llQIDn2R5yWwIyV/6Da8DgBDVKYJvMYD/Sr4J/HBg== X-Google-Smtp-Source: AGHT+IHvRbS5E1bKUmmAsVXFtXvCxFUCGJDcz57wzHvVzZP529pa37fN3+oWV7CfzyuGeLcSuQ1SWw== X-Received: by 2002:a17:90b:2ccd:b0:321:4760:c65a with SMTP id 98e67ed59e1d1-32517d1e054mr30179480a91.27.1756407660486; Thu, 28 Aug 2025 12:01:00 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.00.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:00:59 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:12 -0700 Subject: [PATCH v3 5/7] rust: percpu: Support non-zeroable types for DynamicPerCpu Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-5-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=11574; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=+lEJbTpkz2WbxbYDrCnnJg9tQe8ptqmkgyWr5/FAI6E=; b=nsWk0v7Bp3XRYlydEETW2ckzGyCaiUq6V4hFLtHmax5ZpM1J/foqW9qb/F9rKhy4xVojaLz9Z dAT4xoykz0EDR+zf6iRz4X92+VJiYDuSyBok2JQb7X66hAaNuKTChT9 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Add functionality to `PerCpuPtr` to compute pointers to per-CPU variable slots on other CPUs. Use this facility to initialize per-CPU variables on all possible CPUs when a dynamic per-CPU variable is created with a non-zeroable type. Since `RefCell` and other `Cell`-like types fall into this category, `impl CheckedPerCpu` on `DynamicPerCpu` for these `InteriorMutable` types since they can now be used. Add examples of these usages to `samples/rust/rust_percpu.rs`. Signed-off-by: Mitchell Levy --- rust/helpers/percpu.c | 5 +++ rust/kernel/percpu.rs | 15 +++++++ rust/kernel/percpu/dynamic.rs | 40 +++++++++++++++++ samples/rust/rust_percpu.rs | 99 +++++++++++++++++++++++++++++++++++++++= +--- 4 files changed, 152 insertions(+), 7 deletions(-) diff --git a/rust/helpers/percpu.c b/rust/helpers/percpu.c index 8cc01d094752..8d83b6b86106 100644 --- a/rust/helpers/percpu.c +++ b/rust/helpers/percpu.c @@ -8,6 +8,11 @@ void __percpu *rust_helper_alloc_percpu(size_t sz, size_t = align) return __alloc_percpu(sz, align); } =20 +void *rust_helper_per_cpu_ptr(void __percpu *ptr, unsigned int cpu) +{ + return per_cpu_ptr(ptr, cpu); +} + void rust_helper_on_each_cpu(smp_call_func_t func, void *info, int wait) { on_each_cpu(func, info, wait); diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index 35afcdba3ccd..c68c7520b67f 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -14,6 +14,7 @@ use bindings::{alloc_percpu, free_percpu}; =20 use crate::alloc::Flags; +use crate::cpu::CpuId; use crate::percpu::cpu_guard::CpuGuard; use crate::prelude::*; use crate::sync::Arc; @@ -115,6 +116,20 @@ pub fn get_ptr(&self) -> *mut MaybeUninit { // the invariant that self.0 is a valid offset into the per-CPU ar= ea. (this_cpu_area).wrapping_add(self.0 as usize).cast() } + + /// Get a `*mut MaybeUninit` to the per-CPU variable on the CPU rep= resented by `cpu`. Note + /// that without some kind of synchronization, use of the returned poi= nter may cause a data + /// race. It is the caller's responsibility to use the returned pointe= r in a reasonable way. + /// + /// # Safety + /// - The returned pointer is valid only if `self` is (that is, it poi= nts to a live allocation + /// correctly sized and aligned to hold a `T`) + /// - The returned pointer is valid only if the bit corresponding to `= cpu` is set in + /// `Cpumask::possible()`. + pub unsafe fn get_remote_ptr(&self, cpu: CpuId) -> *mut MaybeUninit= { + // SAFETY: The requirements of this function ensure this call is s= afe. + unsafe { bindings::per_cpu_ptr(self.0.cast(), cpu.as_u32()) }.cast= () + } } =20 // SAFETY: Sending a `PerCpuPtr` to another thread is safe because as s= oon as it's sent, the diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index ce95e420f943..64f04cef3705 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -3,6 +3,8 @@ =20 use super::*; =20 +use crate::cpumask::Cpumask; + /// Represents a dynamic allocation of a per-CPU variable via alloc_percpu= . Calls free_percpu when /// dropped. pub struct PerCpuAllocation(PerCpuPtr); @@ -74,6 +76,36 @@ pub fn new_zero(flags: Flags) -> Option { } } =20 +impl DynamicPerCpu { + /// Allocates a new per-CPU variable + /// + /// # Arguments + /// * `val` - The initial value of the per-CPU variable on all CPUs. + /// * `flags` - Flags used to allocate an `Arc` that keeps track of th= e underlying + /// `PerCpuAllocation`. + pub fn new_with(val: T, flags: Flags) -> Option { + let alloc: PerCpuAllocation =3D PerCpuAllocation::new_uninit()?; + let ptr =3D alloc.0; + + for cpu in Cpumask::possible().iter() { + // SAFETY: `ptr` is a valid allocation, and `cpu` appears in `= Cpumask::possible()` + let remote_ptr =3D unsafe { ptr.get_remote_ptr(cpu) }; + // SAFETY: Each CPU's slot corresponding to `ptr` is currently= uninitialized, and no + // one else has a reference to it. Therefore, we can freely wr= ite to it without + // worrying about the need to drop what was there or whether w= e're racing with someone + // else. `PerCpuPtr::get_remote_ptr` guarantees that the point= er is valid since we + // derived it from a valid allocation and `cpu`. + unsafe { + (*remote_ptr).write(val.clone()); + } + } + + let arc =3D Arc::new(alloc, flags).ok()?; + + Some(Self { alloc: arc }) + } +} + impl PerCpu for DynamicPerCpu { unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpuToken<'_, T> { // SAFETY: The requirements of `PerCpu::get_mut` and this type's i= nvariant ensure that the @@ -81,3 +113,11 @@ unsafe fn get_mut(&mut self, guard: CpuGuard) -> PerCpu= Token<'_, T> { unsafe { PerCpuToken::new(guard, &self.alloc.0) } } } + +impl CheckedPerCpu for DynamicPerCpu { + fn get(&mut self, guard: CpuGuard) -> CheckedPerCpuToken<'_, T> { + // SAFETY: By the invariant of `DynamicPerCpu`, the memory locatio= n in each CPU's + // per-CPU area corresponding to this variable has been initialize= d. + unsafe { CheckedPerCpuToken::new(guard, &self.alloc.0) } + } +} diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs index 98ca1c781b6b..06b322019134 100644 --- a/samples/rust/rust_percpu.rs +++ b/samples/rust/rust_percpu.rs @@ -130,13 +130,72 @@ fn init(_module: &'static ThisModule) -> Result { =20 // SAFETY: No prerequisites for on_each_cpu. unsafe { - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 0); - on_each_cpu(Some(inc_percpu), (&raw mut test).cast(), 1); - on_each_cpu(Some(check_percpu), (&raw mut test).cast(), 1); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 0); + on_each_cpu(Some(inc_percpu_u64), (&raw mut test).cast(), 1); + on_each_cpu(Some(check_percpu_u64), (&raw mut test).cast(), 1); } =20 + let mut checked: DynamicPerCpu> =3D + DynamicPerCpu::new_with(RefCell::new(100), GFP_KERNEL).unwrap(= ); + + // SAFETY: No prerequisites for on_each_cpu. + unsafe { + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 0); + on_each_cpu(Some(inc_percpu_refcell_u64), (&raw mut checked).c= ast(), 1); + on_each_cpu(Some(check_percpu_refcell_u64), (&raw mut checked)= .cast(), 1); + } + + checked.get(CpuGuard::new()).with(|val: &RefCell| { + assert!(*val.borrow() =3D=3D 104); + + let mut checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native +=3D 1; + *val.borrow_mut() +=3D 1; + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 1); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D 0); + + checked_native =3D checked_native.wrapping_add((-1i64) as u64); + val.replace_with(|old: &mut u64| old.wrapping_add((-1i64) as u= 64)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + + checked_native =3D 0; + *val.borrow_mut() =3D 0; + + checked_native =3D checked_native.wrapping_sub(1); + val.replace_with(|old: &mut u64| old.wrapping_sub(1)); + pr_info!( + "Checked native: {}, *checked: {}\n", + checked_native, + val.borrow() + ); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D (-1i64) as u64); + assert!(checked_native =3D=3D *val.borrow() && checked_native = =3D=3D u64::MAX); + }); + pr_info!("rust dynamic percpu test done\n"); =20 // Return Err to unload the module @@ -144,7 +203,7 @@ fn init(_module: &'static ThisModule) -> Result { } } =20 -extern "C" fn inc_percpu(info: *mut c_void) { +extern "C" fn inc_percpu_u64(info: *mut c_void) { // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); @@ -153,7 +212,7 @@ extern "C" fn inc_percpu(info: *mut c_void) { unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| *val += =3D 1); } =20 -extern "C" fn check_percpu(info: *mut c_void) { +extern "C" fn check_percpu_u64(info: *mut c_void) { // SAFETY: We know that info is a void *const DynamicPerCpu and D= ynamicPerCpu is Send. let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu)).clone= () }; pr_info!("Asserting on {}\n", CpuId::current().as_u32()); @@ -161,3 +220,29 @@ extern "C" fn check_percpu(info: *mut c_void) { // SAFETY: We don't have multiple clones of pcpu in scope unsafe { pcpu.get_mut(CpuGuard::new()) }.with(|val: &mut u64| assert!(= *val =3D=3D 4)); } + +extern "C" fn inc_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu= >)).clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Incrementing on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let mut val =3D val.borrow_mut(); + *val +=3D 1; + }); +} + +extern "C" fn check_percpu_refcell_u64(info: *mut c_void) { + // SAFETY: We know that info is a void *const DynamicPerCpu> and + // DynamicPerCpu> is Send. + let mut pcpu =3D unsafe { (*(info as *const DynamicPerCpu= >)).clone() }; + // SAFETY: smp_processor_id has no preconditions + pr_info!("Asserting on {}\n", CpuId::current().as_u32()); + + pcpu.get(CpuGuard::new()).with(|val: &RefCell| { + let val =3D val.borrow(); + assert!(*val =3D=3D 104); + }); +} --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F3B93342C92; Thu, 28 Aug 2025 19:01:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407666; cv=none; b=Ahaf0JKB1kvXdOpmopJW/jmn/ciETAbBr0uFx9gAZvbhinFD+ABuAlQLl7muKH1Ocd9s8DmHE48RyO0lodOMOpaJAnMtFGnVL/foS0RkbJjwh4BwwRApnSKzoZpmZpanPGnsKJXydp1EnGO8+RJOrIOBykUGXX8Y17PlgdATkB0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407666; c=relaxed/simple; bh=FkPibw/wTeWHpca1nOecK532ZikijOEsQLvkO8BhfQs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OPVYaTo381umDuEoBKYIO5fkMjH83EY6Y3on9lsBw4H4jfhXZfxhw1AUxTs/kZvrMwevmrEbsOodLB1ajWbeLXipq/a0lGybF4u+K5Rrx1Jzos182VxxJPa+X5sPuL2mtxizpYwEvtzdZ8nlu4rao5EiU4skUs0fKvkm0CFaiV4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UJo+oXkV; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UJo+oXkV" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-b475dfb4f42so950944a12.0; Thu, 28 Aug 2025 12:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407664; x=1757012464; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=aAngSwZJY8wvhRdb+Us7Bf51kKhX3hLhPg8BAwh+qjA=; b=UJo+oXkVARHFGPiWFrrYhj7vYH2cg00ppTzLUIaEwu8OWNh4rTVCazq7uPXSRSLySC PY2CqRC10ZdB0IST3+jEoxkx5a5O/6tHSlhiBqUxgrEaaKPZC+XQrixVbckOj8X0x3Bh McLbWXMwrVN4Acnsh7HvUareYylRNAr/itodPsxhmeJYoWM94nxvaJx84pAO1Az0x4Ov DNIQE8kFhdVLaRHfAdyWY3djvKD095AU+ulrex/JB1YxYOBoM4BDaqYOPPUdoXsqLdkd POvlQ3A2wrm67cX88GjBCP7yBPFgMhq2NLipUzAprDrPDorRw8lhqZCSVsKoGD1zWtbn ioLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407664; x=1757012464; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aAngSwZJY8wvhRdb+Us7Bf51kKhX3hLhPg8BAwh+qjA=; b=gRbbdyeI56K0ncSuZBh6EakIsaneV5c6oK7L8wuP/MtEP+4YGduQIiuKvnGeh4Qxqd jJFH7Qhei0UP61bCgYc8zHjOc5NvteTxGJBkJHYE+kwm/BTAhs8q/yjGHxVl09WKB4MI ZWGcecAibBgKmyezYrcbG9L4tcVQrbfF+HGVGh4S8blm0D3J97DXMqYiQbkHTn7uJhaZ Y3Nwsjv+25F26tdz8WDr1KRfcTW8N2J0Z7bZU5QZJxFVy0TfDF1IUD6i1OfchPV+0RJe YsMnKAmK/3wI38hT3PAzmHJVJo5aUNtfXRnHMT03YyQl0+wH41IWp/FGZELhi7hRDdcH lLtA== X-Forwarded-Encrypted: i=1; AJvYcCU77X+6Jc7usM4AwbwjtCW7dosHc+nUAC8cA9uOlQUlb0Y9Ru7LuAGgEh6ydHPJP0yyR0bZQ8mgotGR0oaRtnY=@vger.kernel.org, AJvYcCUHoHox/M6WeXexhXIlBKusdGZwNrqgR8iVstGt/hNJR2ZsOhSqWZAv3woXsMTYKIWnT0+8Y9ka5yMhcFY=@vger.kernel.org X-Gm-Message-State: AOJu0YwajRBAlC4xACLSUYQcNqLkJ0c8iQaTWFZbJZrSoxvbtiYp/cIM 4C9I7Vw3w88AET+I1zARuLqvx5Y08swgbnpkaz04HgcllxGQg7DpCJhO6nk2AKop X-Gm-Gg: ASbGncsMZ2t1srvyBqrHpyezSg2KTct2FfZrOahY3nvWztAbq7aXscUtLy3E3UR9W48 DBCuckEcKp4vwh12S6maAaRjoJcjSl7puUu0okx40i2UDknzElIAQ1HgCNgJTl7FcHmf7ZNuzJa pPlL05zR8KAgE1zPUy7dehKXg/DV77/soUNDNfn+ti8nGA6n+kU1nFaCJ9rknFjSUZRL/xAIned JX0jY8n+X+FVjOZGm5AF1yaO81Z6KBPeAI5icdoo/BzUpmTdtElj2C1p8v3aO5thc4+BcZe+r35 8eB3MH7oC2toAdH+a8matO4R+m9Ii3ydvzIgDtrsB6P27RkwvaNzuQpLvpBVcvSEKBBcTLtshh/ 3lBiVIhKYxUSj1luAX9FKHmzOJJECdD9Cx8vlQ03Nx3KtfQ== X-Google-Smtp-Source: AGHT+IE26/lrl4f3dGjJW6QGStTgLdmIa26piskyaGIsHE5TO9Dbkag/vLov52+DzHKlj+iq5a9ckw== X-Received: by 2002:a17:90b:3ec9:b0:327:5360:5b11 with SMTP id 98e67ed59e1d1-327536061fdmr12677471a91.27.1756407662991; Thu, 28 Aug 2025 12:01:02 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.01.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:01:01 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:13 -0700 Subject: [PATCH v3 6/7] rust: percpu: Add pin-hole optimizations for numerics Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-6-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=9614; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=FkPibw/wTeWHpca1nOecK532ZikijOEsQLvkO8BhfQs=; b=G3nMpuN/5qYzhXKfS67YeHPg6iQlvPE/KO79sTBsc4eHDPXaEL+ZfiGMssCN5FNO96iohtJUT XZe3XfI2Na1Dz4CdbL+lGwMMy+olOmlki7WomhFHkbFKPpofXIRqhnD X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= The C implementations of `this_cpu_add`, `this_cpu_sub`, etc., are optimized to save an instruction by avoiding having to compute `this_cpu_ptr(&x)` for some per-CPU variable `x`. For example, rather than u64 *x_ptr =3D this_cpu_ptr(&x); *x_ptr +=3D 5; the implementation of `this_cpu_add` is clever enough to make use of the fact that per-CPU variables are implemented on x86 via segment registers, and so we can use only a single instruction (where we assume `&x` is already in `rax`) add gs:[rax], 5 Add this optimization via a `PerCpuNumeric` type to enable code-reuse between `DynamicPerCpu` and `StaticPerCpu`. Signed-off-by: Mitchell Levy --- rust/kernel/percpu.rs | 1 + rust/kernel/percpu/dynamic.rs | 2 +- rust/kernel/percpu/numeric.rs | 128 ++++++++++++++++++++++++++++++++++++++= ++++ samples/rust/rust_percpu.rs | 36 ++++++++++++ 4 files changed, 166 insertions(+), 1 deletion(-) diff --git a/rust/kernel/percpu.rs b/rust/kernel/percpu.rs index c68c7520b67f..c693d16518d2 100644 --- a/rust/kernel/percpu.rs +++ b/rust/kernel/percpu.rs @@ -4,6 +4,7 @@ =20 pub mod cpu_guard; mod dynamic; +pub mod numeric; mod static_; =20 #[doc(inline)] diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index 64f04cef3705..aad08e4b4251 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -7,7 +7,7 @@ =20 /// Represents a dynamic allocation of a per-CPU variable via alloc_percpu= . Calls free_percpu when /// dropped. -pub struct PerCpuAllocation(PerCpuPtr); +pub struct PerCpuAllocation(pub(super) PerCpuPtr); =20 impl PerCpuAllocation { /// Dynamically allocates a space in the per-CPU area suitably sized a= nd aligned to hold a `T`, diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs new file mode 100644 index 000000000000..4de93f653f0e --- /dev/null +++ b/rust/kernel/percpu/numeric.rs @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: GPL-2.0 +//! Pin-hole optimizations for PerCpu where T is a numeric type. + +use super::*; +use core::arch::asm; + +/// Represents a per-CPU variable that can be manipulated with machine-int= rinsic numeric +/// operations. +pub struct PerCpuNumeric<'a, T> { + // INVARIANT: `ptr.0` is a valid offset into the per-CPU area and is i= nitialized on all CPUs + // (since we don't have a CPU guard, we have to be pessimistic and ass= ume we could be on any + // CPU). + ptr: &'a PerCpuPtr, +} + +macro_rules! impl_ops { + ($ty:ty, $reg:tt) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU variable. + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc.0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU variable. + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `StaticPerCpu`'s inv= ariant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("add gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("sub gs:[{off}], {val:", $reg, "}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg) rhs, + ); + } + } + } + }; +} + +macro_rules! impl_ops_byte { + ($ty:ty) =3D> { + impl DynamicPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU + /// variable. + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.alloc.0 } + } + } + impl StaticPerCpu<$ty> { + /// Returns a `PerCpuNumeric` that can be used to manipulate t= he underlying per-CPU + /// variable. + pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { + // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that + // this pointer is valid and initialized on all CPUs. + PerCpuNumeric { ptr: &self.0 } + } + } + + impl PerCpuNumeric<'_, $ty> { + /// Adds `rhs` to the per-CPU variable. + pub fn add(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("add gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + impl PerCpuNumeric<'_, $ty> { + /// Subtracts `rhs` from the per-CPU variable. + pub fn sub(&mut self, rhs: $ty) { + // SAFETY: `self.ptr.0` is a valid offset into the per-CPU= area (i.e., valid as a + // pointer relative to the `gs` segment register) by the i= nvariants of this type. + unsafe { + asm!( + concat!("sub gs:[{off}], {val}"), + off =3D in(reg) self.ptr.0.cast::<*mut $ty>(), + val =3D in(reg_byte) rhs, + ); + } + } + } + }; +} + +impl_ops_byte!(i8); +impl_ops!(i16, "x"); +impl_ops!(i32, "e"); +impl_ops!(i64, "r"); +impl_ops!(isize, "r"); + +impl_ops_byte!(u8); +impl_ops!(u16, "x"); +impl_ops!(u32, "e"); +impl_ops!(u64, "r"); +impl_ops!(usize, "r"); diff --git a/samples/rust/rust_percpu.rs b/samples/rust/rust_percpu.rs index 06b322019134..e3a46a053b8d 100644 --- a/samples/rust/rust_percpu.rs +++ b/samples/rust/rust_percpu.rs @@ -27,6 +27,26 @@ define_per_cpu!(UPERCPU: u64 =3D 0); define_per_cpu!(CHECKED: RefCell =3D RefCell::new(0)); =20 +macro_rules! make_optimization_test { + ($ty:ty) =3D> { + let mut test: DynamicPerCpu<$ty> =3D DynamicPerCpu::new_zero(GFP_K= ERNEL).unwrap(); + { + let _ =3D CpuGuard::new(); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = *val =3D 10); + test.num().add(1); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 11)); + test.num().add(10); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 21)); + test.num().sub(5); + // SAFETY: No other usage of `test` + unsafe { test.get_mut(CpuGuard::new()) }.with(|val: &mut $ty| = assert_eq!(*val, 16)); + } + }; +} + impl kernel::Module for PerCpuMod { fn init(_module: &'static ThisModule) -> Result { pr_info!("rust percpu test start\n"); @@ -198,6 +218,22 @@ fn init(_module: &'static ThisModule) -> Result { =20 pr_info!("rust dynamic percpu test done\n"); =20 + pr_info!("rust numeric optimizations test start\n"); + + make_optimization_test!(u8); + make_optimization_test!(u16); + make_optimization_test!(u32); + make_optimization_test!(u64); + make_optimization_test!(usize); + + make_optimization_test!(i8); + make_optimization_test!(i16); + make_optimization_test!(i32); + make_optimization_test!(i64); + make_optimization_test!(isize); + + pr_info!("rust numeric optimizations test done\n"); + // Return Err to unload the module Result::Err(EINVAL) } --=20 2.34.1 From nobody Fri Oct 3 15:36:02 2025 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E4A543431E3; Thu, 28 Aug 2025 19:01:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407668; cv=none; b=s5q/jRQp7LLKmJjGMq51vdMBVd7wgHsDXOX3y3RyJcnmBSK8GFf7EzufGfSylF4fDlHT3eOiuE428fPaEg6WygM1Zqj13KLgL3DP2r97M5SXZXk/1aH68UwhvX5LEhZdXin892IJTF3W0ktAw34hanKqK+X4HJz46B13pUdIaVI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756407668; c=relaxed/simple; bh=E3oVGdPxyaJFgaTqf2y8btrQgXMEMCM/zUm8ETp1y5c=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=JIp/rBzbGngawNhVqToFHYPBI74CbmFUxRWh4bp2RrwL40coaN5mA7cIpc+QCU5XZxuUgUrrANMbxvdA5Bs6Nn3c+K/3Jx9hXZKpcQyMPdlE3Hk4w8c2Q/ANG6V8AxLREluXmo70UoVVnTPsv2cOExS+COVqSwhoDN1QtC48JCs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aa082iQd; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aa082iQd" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-32783de3e87so1355578a91.0; Thu, 28 Aug 2025 12:01:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1756407665; x=1757012465; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=PofwN+sl6dw2dDWCuPOORWJpa23jS48M28BtJ3SIOeU=; b=aa082iQdXbXlU8y2SKgpyUO7arztkbj6ukbBvWkbqus/rKXYcMm1mz/NqSATwizwUy G7dGJ6lJvQdjOOdDd5qRZLFng5fQKJTBIncHNrzIPnxcnSg1So5fJkMhHlsAxXQZLqc7 gbefAUc/8jH1d+dhcHPHOe4ScCUtQ/+XSivakZ5YO6UlRQKyX3pS59b3hZ4F6prZVgSh T/p773XFEi62RO3zn6aC356JVIJfkGPLY7yZFuvVQ6aqhG5eGKJX7KbYkp1TDczYlZ7C vIHLQtW5k9aXYqvh/MmNGYrEpYrXCMrwqyF8bmBHxuG/8U2GHP0FTqv+6d0G2XqUQxNO iG0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756407665; x=1757012465; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PofwN+sl6dw2dDWCuPOORWJpa23jS48M28BtJ3SIOeU=; b=kaDvyF7BE8GuN6T2CbgESh+PcSn9lVLhrSK6gT+ar0WPIljLeEhEW6nHYUxZh95e99 3EUUK6oPzDF1Cco+OJqIL91aIRySZs9ZBRRvahnk2vbb+HVvbuk6uswDZ3z0aB6wW7VO ff5zScn4G5HIt2/w/Xx8ZIL1kntBpxXVQVKlCf0GrdPfq9AOdQH+WqH9swXIsL3IhMch IXW6xLgNQ9i3/v9/fNfPo1iKeDY44lSuTLXI6in8SOHbsO2IUT1Pak6XD5++WOrZ01oZ 39HbJGIWFKTtcdgxCfMfqsjVZ/HKY4GYS0YZZ0iYJGiL7SziRVnku6eX93FdlA7MQ0Ug cOOA== X-Forwarded-Encrypted: i=1; AJvYcCUT9su3dRvy3sgQu6mrODPdul5ex0pjntb8+1y0fr0Fqx5frgJzr485WpleEjBGFQSO491VdL+o5UbsGdM=@vger.kernel.org, AJvYcCVIVfflyDUchTVJtcCJQfITHIjkMY1ug1DMWalHWtBaTKuL/uQN+hvqDEJJOyV4DVWvK+42rTR1x4fyD1fH31Y=@vger.kernel.org X-Gm-Message-State: AOJu0Yyk5NHap/GWT2P1DUbXJbV5AvP2H7Ei64gqEWTnWvzvPBQ93Ex7 ifCg7qPO8zI0yx7YrOsxQFxlJgJSTzVWJQUiT7taI2/D4Vchvhn8FzOdRZ1pXSux X-Gm-Gg: ASbGncv5BcT21VyQMVdvJh2lLX+wBnT6VtYM29A0tMaSfGNNItQ4db4F8fZUc1GaHBX 10wz3FHQHWpxRQXTMAa4+lvkQdFEZL4O9QXZIGNWUDEV5qPT/4eAbw9XByFPjCfoVxg8mOPc3rH V+C15EQrayi5sa5kIwDidx61fk+JPP6IIXiTX2xakazzSKxzykK61bURJQ2OF0GntrOsE/vWttX rRVuAZGaEgMKBVP5pU9TDvmLsHG6dzlhdoqJIvIV2QiRRINJ94evnpw6RIorDJKsREPvuw3uBHD 4LrZM58ODBkcCtcFAOMJKh6kTq1Fknu8Uv3RqmnB/eXpVDxyGVGf2XPNm1bj/geBhMupna5LeKA CIJlIZLZmddiekyaM4fBOUdrDP5ywRBKY07GvXsCS+KGfxuYwsxukGdSa X-Google-Smtp-Source: AGHT+IFS3JVNNhR3zNJyk8gBLIrCVFra0me8HIrQNR73Yc/VuaZrqjY1lLTK7uNwZToUsptvYvMAug== X-Received: by 2002:a17:90b:52cc:b0:327:d4e9:c4ea with SMTP id 98e67ed59e1d1-327d4e9c960mr1680821a91.4.1756407664152; Thu, 28 Aug 2025 12:01:04 -0700 (PDT) Received: from mitchelllevy.localdomain ([174.127.224.194]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-327da8e5ad5sm405729a91.18.2025.08.28.12.01.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 12:01:03 -0700 (PDT) From: Mitchell Levy Date: Thu, 28 Aug 2025 12:00:14 -0700 Subject: [PATCH v3 7/7] rust: percpu: cache per-CPU pointers in the dynamic case Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250828-rust-percpu-v3-7-4dd92e1e7904@gmail.com> References: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> In-Reply-To: <20250828-rust-percpu-v3-0-4dd92e1e7904@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Andreas Hindborg , Alice Ryhl , Trevor Gross , Andrew Morton , Dennis Zhou , Tejun Heo , Christoph Lameter , Danilo Krummrich , Benno Lossin , Yury Norov , Viresh Kumar Cc: Tyler Hicks , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org, linux-mm@kvack.org, Mitchell Levy X-Mailer: b4 0.14.0 X-Developer-Signature: v=1; a=ed25519-sha256; t=1756407651; l=3005; i=levymitchell0@gmail.com; s=20240719; h=from:subject:message-id; bh=E3oVGdPxyaJFgaTqf2y8btrQgXMEMCM/zUm8ETp1y5c=; b=EzwFDp+SiJdNfpKZyQsIfCPW4wWXpRD3O7suWpzgYPTt0ZCT0wH9WnekbdPgj5jQlRyhAXkKS RUSwrYD8Pa1BCS54cUq+TOw4bSu3hPrC3vCTdt3FbdvBg5XQJzJ5KH3 X-Developer-Key: i=levymitchell0@gmail.com; a=ed25519; pk=n6kBmUnb+UNmjVkTnDwrLwTJAEKUfs2e8E+MFPZI93E= Currently, the creation of a `PerCpuNumeric` requires a memory read via the `Arc` managing the dynamic allocation. While the compiler might be clever enough to consolidate these reads in some cases, the read must happen *somewhere*, which, when we're concerning ourselves with individual instructions, is a very high burden. Instead, cache the `PerCpuPointer` inside the `DynamicPerCpu` structure; then, the `Arc` is used solely to manage the allocation. Signed-off-by: Mitchell Levy --- rust/kernel/percpu/dynamic.rs | 8 ++++++-- rust/kernel/percpu/numeric.rs | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/rust/kernel/percpu/dynamic.rs b/rust/kernel/percpu/dynamic.rs index aad08e4b4251..8683e94bcd9f 100644 --- a/rust/kernel/percpu/dynamic.rs +++ b/rust/kernel/percpu/dynamic.rs @@ -58,7 +58,10 @@ fn drop(&mut self) { pub struct DynamicPerCpu { // INVARIANT: The memory location in each CPU's per-CPU area pointed a= t by `alloc.0` has been // initialized. + // INVARIANT: `ptr` is the per-CPU pointer managed by `alloc`, which d= oes not change for the + // lifetime of `self`. pub(super) alloc: Arc>, + pub(super) ptr: PerCpuPtr, } =20 impl DynamicPerCpu { @@ -70,9 +73,10 @@ impl DynamicPerCpu { pub fn new_zero(flags: Flags) -> Option { let alloc: PerCpuAllocation =3D PerCpuAllocation::new_zero()?; =20 + let ptr =3D alloc.0; let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: arc }) + Some(Self { alloc: arc, ptr }) } } =20 @@ -102,7 +106,7 @@ pub fn new_with(val: T, flags: Flags) -> Option { =20 let arc =3D Arc::new(alloc, flags).ok()?; =20 - Some(Self { alloc: arc }) + Some(Self { alloc: arc, ptr }) } } =20 diff --git a/rust/kernel/percpu/numeric.rs b/rust/kernel/percpu/numeric.rs index 4de93f653f0e..3412c2de85fd 100644 --- a/rust/kernel/percpu/numeric.rs +++ b/rust/kernel/percpu/numeric.rs @@ -20,7 +20,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc.0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { @@ -71,7 +71,7 @@ impl DynamicPerCpu<$ty> { pub fn num(&mut self) -> PerCpuNumeric<'_, $ty> { // The invariant is satisfied because `DynamicPerCpu`'s in= variant guarantees that // this pointer is valid and initialized on all CPUs. - PerCpuNumeric { ptr: &self.alloc.0 } + PerCpuNumeric { ptr: &self.ptr } } } impl StaticPerCpu<$ty> { --=20 2.34.1