From nobody Sat Sep 26 22:53:30 2026 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4478C364058 for ; Fri, 28 Aug 2026 15:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931677; cv=none; b=oflC3kOeCxZWaqr2qXyxXJNGEwAe9elMchqgIXlTBrswlFCuyfUqamHDRnc5OWV1YBMTOpb6VGAN98ov+B8eibHsHw+YLW5LAXvBEYUHfFewda6XWdOaTPFYCtJhnHyCSZkg5lDOSj/IVW1KH69Pa2ZsOVfMOcrMlxmXoCmtHP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931677; c=relaxed/simple; bh=VkZbjXeNxSt5oZpSD6SfLMeOquHToVqWskiDwdeRjxQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s+vTZOa9OR4XxayH7aTCXCnu/+ldhKB6rEVeZZ2qhYzJNgJ/ZPSgDzbCHbVDdd4KS5H+YkLmcD2T/e2johLcLS6LWfYrc3d3p/j+0pozmGrglHhs0AMS8YIHrnvEwQCPn4W+FMhaRT45uJs+rFirYWEuu1sNrH6s0o3bfWJ1K1U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=GuVnUXy8; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="GuVnUXy8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1787931237; bh=Wi5zWpvsXSQP3flQPE51y11BL6wE16BItDUgzsf8CHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GuVnUXy8ppPMwyFVWKz4IYWO0cqWdQxtlBQIWe8ec5a4+Wq/lSrb68rSge132WDiC UJPue3uMXqesjHCqWmtZ/lYgfzFZ2+wW0cqHnmyAnVudJIx79cCFVjNG0IhMBZh6M8 XjdtD7cT1xbf2ZtQOvT9nzGyjMTioxUeMn9uTewJo4IkTv2WKWS05ALeWo32VV/BS4 yCA6I/1F4kx2BJI07btPi0mu2iOUaACvOMLMsxHvvT1jJWoSkfocFUAkxb5KpBeu9J gu3rDiUpkxC/PtMHatS45tMyCIxIQakSxaQQe7ofmPChfjcU2Q2W6VMaPMekGm9bCu VRI8HSrJ4s3VA== Received: from laura.localdomain (199-193-172-8.cpe.axion.ca [199.193.172.8]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4hWj910PR3zTCY; Fri, 28 Aug 2026 11:33:57 -0400 (EDT) From: odion@efficios.com To: Mathieu Desnoyers Cc: Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , LKML , Thomas Gleixner , Dmitry Vyukov , David Matlack , Marco Elver , Sean Christopherson , Wei Liu , Florian Weimer , Mathias Stearn , Chris Kennelly , Blake Oler , Rich Felker , Matthew Wilcox , Greg Kroah-Hartman , Carlos O'Donell , Olivier Dion Subject: [RFC PATCH 1/5] rseq: uapi: add rseq operation definitions Date: Fri, 28 Aug 2026 11:33:39 -0400 Message-ID: <20260828153349.8061-2-odion@efficios.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260828153349.8061-1-odion@efficios.com> References: <20260828153349.8061-1-odion@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Olivier Dion Introduce userspace ABI for rseq operations: a per-thread list of operations the kernel applies on return to user space. The main motivation for this work is to encourage TCMalloc to migrate to RSEQ v2 [0] and use the glibc RSEQ region. Indeed, TCMalloc relies on the behavior of RSEQ v1, that reset the cpu_id bits in the RSEQ shared region, to invalidate a per-cpu pointer cached in a TLS. This hack requires TCMalloc users to use a glibc tunable to disable RSEQ registration for threads so that TCMalloc can register its own region, overlapping the TLS cache. Overall, this puts the users in a situation of choosing between a fast sched_getcpu() and TCMalloc, plus some downsides such as requiring a initial-exec model for the cache TLS in a shared-library and not being compatible with the glibc RSEQ operations aim at solving this by offering operations that are executed by the kernel, on behalf of a task after it is scheduled, before returning to userspace. Operations are per-task and registered throught prctl. They are opt-in and only introduce overhead on tasks that register them. Add enum rseq_op_type and the rseq_op_node / rseq_op_reset / rseq_op_reset_with_stride structures, the RSEQ_CS_FLAG_RSEQ_OP_* availability/enabled flags, the RSEQ_OP_LIST_LIMIT walk bound, and the PR_RSEQ_OP prctl with its REGISTER/UNREGISTER sub-commands. The operation list is a circular doubly-linked list anchored by a kernel-owned sentinel embedded in struct rseq. [0] Documentation/userspace-api/rseq.rst (Optimized RSEQ v2) Link: https://lore.kernel.org/lkml/20260428221058.149538293@kernel.org Signed-off-by: Olivier Dion --- include/uapi/linux/prctl.h | 12 +++++ include/uapi/linux/rseq.h | 104 ++++++++++++++++++++++++++++++++++--- 2 files changed, 108 insertions(+), 8 deletions(-) diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h index b6ec6f693719..4cb6356a271a 100644 --- a/include/uapi/linux/prctl.h +++ b/include/uapi/linux/prctl.h @@ -396,6 +396,18 @@ struct prctl_mm_map { */ # define PR_RSEQ_SLICE_EXT_ENABLE 0x01 =20 +/* + * RSEQ operation registration. + * + * arg3 is the user address of a struct rseq_op_node embedded in one of the + * rseq operation structures (see uapi/linux/rseq.h). Registering the first + * operation enables rseq operation processing for the thread; unregisteri= ng + * the last one disables it. + */ +#define PR_RSEQ_OP 82 +# define PR_RSEQ_OP_REGISTER 1 +# define PR_RSEQ_OP_UNREGISTER 2 + /* * Get or set the control flow integrity (CFI) configuration for the * current thread. diff --git a/include/uapi/linux/rseq.h b/include/uapi/linux/rseq.h index ca6fe1f9d05e..b664d1991c48 100644 --- a/include/uapi/linux/rseq.h +++ b/include/uapi/linux/rseq.h @@ -13,6 +13,11 @@ #include #include =20 +/* + * Maximum number of nodes walked in the rseq operation list. + */ +#define RSEQ_OP_LIST_LIMIT 2048 + enum rseq_cpu_id_state { RSEQ_CPU_ID_UNINITIALIZED =3D -1, RSEQ_CPU_ID_REGISTRATION_FAILED =3D -2, @@ -33,6 +38,8 @@ enum rseq_cs_flags_bit { /* User read only feature flags */ RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT =3D 4, RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT =3D 5, + RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT =3D 6, + RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT =3D 7, }; =20 enum rseq_cs_flags { @@ -47,6 +54,10 @@ enum rseq_cs_flags { (1U << RSEQ_CS_FLAG_SLICE_EXT_AVAILABLE_BIT), RSEQ_CS_FLAG_SLICE_EXT_ENABLED =3D (1U << RSEQ_CS_FLAG_SLICE_EXT_ENABLED_BIT), + RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE =3D + (1U << RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE_BIT), + RSEQ_CS_FLAG_RSEQ_OP_ENABLED =3D + (1U << RSEQ_CS_FLAG_RSEQ_OP_ENABLED_BIT), }; =20 /* @@ -86,6 +97,77 @@ struct rseq_slice_ctrl { }; }; =20 +/* + * enum rseq_op_type - Type of an rseq operation + * @RSEQ_OP_RESET: Plain reset. Uses struct rseq_op_reset. + * @RSEQ_OP_RESET_WITH_STRIDE_CPUID: Reset indexed by the current CPU ID. + * Uses struct rseq_op_reset_with_stride. + * @RSEQ_OP_RESET_WITH_STRIDE_MMCID: Reset indexed by the current MM CID. + * Uses struct rseq_op_reset_with_stride. + */ +enum rseq_op_type { + RSEQ_OP_RESET, + RSEQ_OP_RESET_WITH_STRIDE_CPUID, + RSEQ_OP_RESET_WITH_STRIDE_MMCID, + RSEQ_OP_NR, +}; + +/* + * struct rseq_op_node - Common header linking an rseq operation into the = list + * @next: Address of the next node. Owned by the kernel. + * @prev: Address of the previous node. Owned by the kernel. + * @type: Operation type. See enum rseq_op_type. + * @reserved: Must be zero on registration. + * + * User space allocates the node, sets @type and zeroes @next, @prev and + * @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, n= ode). + * The kernel owns @next and @prev for the lifetime of the registration and + * links the node into a circular doubly-linked list anchored by an intern= al + * sentinel in struct rseq. User space must not touch @next or @prev while= the + * node is registered. + */ +struct rseq_op_node { + __u64 next; + __u64 prev; + struct { + __u8 type; /* enum rseq_op_type */ + __u8 reserved[7]; + }; +}; + +/* + * struct rseq_op_reset - Reset one word to a value on return to user space + * @node: Operation list node. + * @src: Address of the source word, or 0 to reset @dst to zero. + * @dst: Address of the destination word. + * @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). + */ +struct rseq_op_reset { + struct rseq_op_node node; + __u64 src; + __u64 dst; + __u32 len; +}; + +/* + * struct rseq_op_reset_with_stride - Reset one word in a strided array + * @node: Operation list node. + * @src: Address of the source word, or 0 to reset the slot to zero. + * @dst: Base address of the strided destination array. + * @dst_stride: Stride in bytes between consecutive array slots. + * @len: Word length in bytes. Must be 4 or 8 (8 is 64-bit only). + * + * The destination slot is @dst + @dst_stride * index, where index is the + * current CPU ID or MM CID depending on the operation type. + */ +struct rseq_op_reset_with_stride { + struct rseq_op_node node; + __u64 src; + __u64 dst; + __u64 dst_stride; + __u32 len; +}; + /* * The original size and alignment of the allocation for struct rseq is * 32 bytes. @@ -191,15 +273,21 @@ struct rseq { struct rseq_slice_ctrl slice_ctrl; =20 /* - * Before rseq became extensible, its original size was 32 bytes even - * though the active rseq area was only 20 bytes. - * Exposing a 32 bytes feature size would make life needlessly painful - * for userspace. Therefore, add a reserved byte after byte 32 - * to bump the rseq feature size from 32 to 33. - * The next field to be added to the rseq area will be larger - * than one byte, and will replace this reserved byte. + * Sentinel of the circular doubly-linked list of rseq operations + * registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by + * the kernel: it is initialized to point to itself on registration and + * user space must never read or write it directly. + * + * The kernel only use next and prev from rseq_op_list. The rest of the + * bytes are reserved for later usage and should be zeroed. */ - __u8 __reserved; + union { + struct rseq_op_node rseq_op_list; + struct { + __u64 op_used[2]; + __u64 reserved; + }; + }; =20 /* * Flexible array member at end of structure, after last feature field. --=20 2.54.0 From nobody Sat Sep 26 22:53:30 2026 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 453FA369D6E for ; Fri, 28 Aug 2026 15:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931675; cv=none; b=aOsIcdnmVs8qpt24jrtZauYm3vim4lPgDLJzX06mJ5cKTCHnIDN6cA/wrDcGzb6X2YBszCiG9SapjoxaNa0xQhLDD5BhhzEG4K5FewmJrXdaXw0/wHQHchqAoTY6xqcQkn3awiWjD+5QhH64XRXx2ArDADUttGtAfmKWMBcoyxo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931675; c=relaxed/simple; bh=E9d8N9EeeOpwm9M+PlRsmfUC4jVIvkf0iXl59EcBCFY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KP7tRaZQ9wn3tgneQLkxQBzu2MsmS3FWZNIzq3QouOpECfpa1M3PUEelY7lUxWcDTo5vtS9iQZ+ihbAb1z9Z5f9XuiPOksIAqEmSMyZLn0cMBTosQuzR8W1OBEhpaaynyA3XvzI97Zhz9AkFZxXuavnXVlB7ykJsF9IepqV1Op8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=lAtZHNdY; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="lAtZHNdY" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1787931237; bh=TNUZtdikO0Z4HeeE4t3Hz9wtWi9KTEiLCEoF5STwecc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lAtZHNdYzqmEZRtKXcBDPdKbqSpCOuQ23B0JolKDdr1e9c+iW9u+hrOPo8EOXF0A8 nwQU7RZkE5wbEXiPx1hkliIZAuDkW14WdCAcB3nJvca55FfKfjji6mLBRWiZs1yCe8 baxN/Nu0U8QtEz9yYgIO8k1S0ZMgaFdvXmbLnIiDUnb1D+gSpesX/XX2UP6HFGew/U JArQ0pjD+vnQfJuRVZFEnDLmN3gO0Vh3WddI+dlIQYdck/doB3pFm8CLzQlUk/ASPR KPEDgo0ESAiNNuor5IweJkxdHWuvbRjn7cj7BB4KORPDL5PxMDgTdCL+fZtXA5uIgl 6CYj3e5GkvUYQ== Received: from laura.localdomain (199-193-172-8.cpe.axion.ca [199.193.172.8]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4hWj912JJ1zTCZ; Fri, 28 Aug 2026 11:33:57 -0400 (EDT) From: odion@efficios.com To: Mathieu Desnoyers Cc: Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , LKML , Thomas Gleixner , Dmitry Vyukov , David Matlack , Marco Elver , Sean Christopherson , Wei Liu , Florian Weimer , Mathias Stearn , Chris Kennelly , Blake Oler , Rich Felker , Matthew Wilcox , Greg Kroah-Hartman , Carlos O'Donell , Olivier Dion Subject: [RFC PATCH 2/5] rseq: add per-task rseq operation state Date: Fri, 28 Aug 2026 11:33:40 -0400 Message-ID: <20260828153349.8061-3-odion@efficios.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260828153349.8061-1-odion@efficios.com> References: <20260828153349.8061-1-odion@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Olivier Dion Add the task-side state for rseq operations: the rseq_event::rseq_op state indicating operation processing is enabled for the task, and the nr_ops counter in struct rseq_data. nr_ops is edge triggered: its 0<->1 transition enables/disables rseq_event::rseq_op and the user visible RSEQ_CS_FLAG_RSEQ_OP_ENABLED flag. This makes so that only thread that enable RSEQ operations through prctl takes a performance hit. Include rseq_op in the sched-switch raise condition so pending operations force a return through the rseq exit path, and declare the rseq_op_prctl() entry point (with a CONFIG_RSEQ=3Dn stub). Signed-off-by: Olivier Dion --- include/linux/rseq.h | 10 +++++++++- include/linux/rseq_types.h | 9 +++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/linux/rseq.h b/include/linux/rseq.h index 7ef79b25e714..d1e33906d863 100644 --- a/include/linux/rseq.h +++ b/include/linux/rseq.h @@ -69,7 +69,9 @@ static __always_inline void rseq_sched_switch_event(struc= t task_struct *t) * was via interrupt from user space. ev->has_rseq does not have * to be evaluated here because rseq_v2() implies has_rseq. */ - bool raise =3D ev->user_irq | ev->ids_changed; + bool raise =3D (ev->user_irq | + ev->ids_changed | + ev->rseq_op); =20 if (raise) { ev->sched_switch =3D true; @@ -172,6 +174,8 @@ static inline unsigned int rseq_alloc_align(void) return 1U << get_count_order(offsetof(struct rseq, end)); } =20 +int rseq_op_prctl(unsigned long arg2, unsigned long arg3); + #else /* CONFIG_RSEQ */ static inline bool rseq_v2(struct task_struct *t) { return false; } static inline void rseq_handle_slowpath(struct pt_regs *regs) { } @@ -182,6 +186,10 @@ static inline void rseq_force_update(void) { } static inline void rseq_virt_userspace_exit(void) { } static inline void rseq_fork(struct task_struct *t, u64 clone_flags) { } static inline void rseq_execve(struct task_struct *t) { } +static inline int rseq_op_prctl(unsigned long arg2, unsigned long arg3) +{ + return -ENOTSUPP; +} #endif /* !CONFIG_RSEQ */ =20 #ifdef CONFIG_DEBUG_RSEQ diff --git a/include/linux/rseq_types.h b/include/linux/rseq_types.h index 85739a63e85e..059292695c34 100644 --- a/include/linux/rseq_types.h +++ b/include/linux/rseq_types.h @@ -23,6 +23,7 @@ struct rseq; * exit to user * @ids_changed: Indicator that IDs need to be updated * @user_irq: True on interrupt entry from user mode + * @rseq_op: Rseq operation processing is enabled for the task * @has_rseq: Greater than 0 if the task has a rseq pointer installed. * Contains the RSEQ version number * @error: Compound error code for the slow path to analyze @@ -44,6 +45,7 @@ struct rseq_event { u8 sched_switch; u8 ids_changed; u8 user_irq; + u8 rseq_op; }; }; =20 @@ -115,6 +117,7 @@ struct rseq_slice { * @event: Storage for event management * @ids: Storage for cached CPU ID and MM CID * @slice: Storage for time slice extension data + * @nr_ops: Number of registered rseq operations */ struct rseq_data { struct rseq __user *usrptr; @@ -125,6 +128,12 @@ struct rseq_data { #ifdef CONFIG_RSEQ_SLICE_EXTENSION struct rseq_slice slice; #endif + /* + * Number of rseq operations registered for the task. Edge triggered: + * the 0<->1 transition enables/disables rseq_event::rseq_op and the + * RSEQ_CS_FLAG_RSEQ_OP_ENABLED user flag. + */ + u32 nr_ops; }; =20 #else /* CONFIG_RSEQ */ --=20 2.54.0 From nobody Sat Sep 26 22:53:30 2026 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 45275369D60 for ; Fri, 28 Aug 2026 15:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931676; cv=none; b=rk0uMjdq2qPHSaPGSGrxVdjjM0jyiA7y42nV8UXLtAC+tzikbOXWc143XMPqLqiJcIyEhWADKV0GltFwuU1B2hW7ukife0nE8kc9JZbCBy4SqsJ9ZGP51cB6IaO0ZtbM/VeGzXh6Zgy8UL54jWlfjSG7XYaRRP1UhTk430FmnzY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931676; c=relaxed/simple; bh=WNns8viny/2WJbIUPFjH3LBFLJG7s3VzHGVJtwZZrak=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u9UzE2VVbxQuPTUq4c5FrH4ptSC1qqEK1CkXmJo1VrQ7c6B+TII8l1uAc343dwzQxFOsnJu2YWUl2yV1sjrIuFbEunphdKJdQF8Af+5MFyeDggNBc5omIoYLFRqvZN9fQCGXXOY/A9eR1chJIHWysgwX4ysDzTjqS3icVHxymq4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=XMOorbxQ; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="XMOorbxQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1787931237; bh=dkAdbX6C4yV4urrZ88atFWLuos5sbXNdTh5kny1Q5wo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XMOorbxQxhymAlJAXFWkMFvHaEFMYNpKdUPEuiSC5hd2o2JwqffHxIsqWE7fzTn5a 9ZdaetYF0XpR+fBpRQE2hHYenw/oce1HcYxq16zIzgde5CyR/OBnmlVhLjUn6OG75I dGN/0F7SgGnDO7h0PUxKfAxMdiaMTjEjs+l2GHcsR5oKHBKxO/hKJJ/Tgl9In6xS7n 3jDXWn+qSi/htPl/HzuB/E9Nl5e5P/u3pVOklpgr8aqVYig59AOp+mIIpPvJBTxcg0 4GFu4ZYyeHvk/Y3hC8xhlyWi2ZRPhcphTvExZlN9pkPmcQon6gfRYExe55kWI3Nmln zM7lWmQQLVhOg== Received: from laura.localdomain (199-193-172-8.cpe.axion.ca [199.193.172.8]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4hWj914mh1zSNW; Fri, 28 Aug 2026 11:33:57 -0400 (EDT) From: odion@efficios.com To: Mathieu Desnoyers Cc: Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , LKML , Thomas Gleixner , Dmitry Vyukov , David Matlack , Marco Elver , Sean Christopherson , Wei Liu , Florian Weimer , Mathias Stearn , Chris Kennelly , Blake Oler , Rich Felker , Matthew Wilcox , Greg Kroah-Hartman , Carlos O'Donell , Olivier Dion Subject: [RFC PATCH 3/5] rseq: apply operations on exit to user space Date: Fri, 28 Aug 2026 11:33:41 -0400 Message-ID: <20260828153349.8061-4-odion@efficios.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260828153349.8061-1-odion@efficios.com> References: <20260828153349.8061-1-odion@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Olivier Dion Add rseq_apply_ops(), which walks the task's circular operation list on return to user space and applies each operation. Wire the walk into rseq_exit_user_update() for both the ids-unchanged and ids-changed paths, gated on rseq_event::rseq_op. Since operation processing is a persistent per-task state rather than a one-shot event, introduce rseq_clear_one_shot_events() which preserves the rseq_op bit when clearing events on the way out. len, src and dst are re-read from user memory and re-validated on every application, since user space may change them after registration. A broken ABI contract will fault the process. Signed-off-by: Olivier Dion --- include/linux/rseq_entry.h | 157 +++++++++++++++++++++++++++++++++++-- 1 file changed, 152 insertions(+), 5 deletions(-) diff --git a/include/linux/rseq_entry.h b/include/linux/rseq_entry.h index ed9da6e41a2a..df083b0ee14a 100644 --- a/include/linux/rseq_entry.h +++ b/include/linux/rseq_entry.h @@ -247,6 +247,137 @@ static __always_inline bool rseq_grant_slice_extensio= n(unsigned long ti_work, un #define rseq_slice_clear_user(rseq, efault) do { } while (0) #endif /* !CONFIG_RSEQ_SLICE_EXTENSION */ =20 +/* + * Validate and perform a single reset word operation: reset @dst to *@src, + * or to zero when @src is 0. @len, @src and @dst come from user memory and + * are re-validated on every application, since user space may have changed + * them since registration. A broken ABI contract returns false, which fau= lts + * the process. + */ +static __always_inline bool rseq_op_reset_word(u64 src, u64 dst, u32 len, + void __user *node) +{ + if (unlikely(!IS_ALIGNED(dst, len) || (src && !IS_ALIGNED(src, len)))) { + pr_info_ratelimited("rseq: bad operation alignment len=3D%u src=3D%llx d= st=3D%llx from %p\n", + len, src, dst, node); + return false; + } + + switch (len) { + case 4: { + u32 __user *udst =3D (u32 __user *)dst; + u32 v =3D 0; + + if (src) { + u32 __user *usrc =3D (u32 __user *)src; + + scoped_user_read_access(usrc, efault) + unsafe_get_user(v, usrc, efault); + } + scoped_user_write_access(udst, efault) + unsafe_put_user(v, udst, efault); + return true; + } + case 8: { + u64 __user *udst =3D (u64 __user *)dst; + u64 v =3D 0; + + if (src) { + u64 __user *usrc =3D (u64 __user *)src; + + scoped_user_read_access(usrc, efault) + unsafe_get_user(v, usrc, efault); + } + scoped_user_write_access(udst, efault) + unsafe_put_user(v, udst, efault); + return true; + } + default: + pr_info("rseq: bad operation length=3D%u from %p\n", len, node); + return false; + } +efault: + pr_info("rseq: fault while applying operation from %p\n", node); + return false; +} + +static __always_inline bool rseq_apply_ops(struct task_struct *t) +{ + struct rseq __user *rseq =3D t->rseq.usrptr; + struct rseq_op_node __user *sentinel =3D &rseq->rseq_op_list; + struct rseq_op_node __user *node; + unsigned int limit =3D RSEQ_OP_LIST_LIMIT; + u64 next; + + WARN_ONCE(!t->rseq.event.rseq_op, "rseq operations not enabled"); + + scoped_user_read_access(sentinel, efault) + unsafe_get_user(next, &sentinel->next, efault); + + node =3D (struct rseq_op_node __user *)next; + + while (node !=3D sentinel && limit--) { + u8 type; + + scoped_user_read_access(node, efault) { + unsafe_get_user(type, &node->type, efault); + unsafe_get_user(next, &node->next, efault); + } + + switch (type) { + case RSEQ_OP_RESET: { + struct rseq_op_reset __user *op =3D + (struct rseq_op_reset __user *)node; + u64 src, dst; + u32 len; + + scoped_user_read_access(op, efault) { + unsafe_get_user(src, &op->src, efault); + unsafe_get_user(dst, &op->dst, efault); + unsafe_get_user(len, &op->len, efault); + } + + if (!rseq_op_reset_word(src, dst, len, node)) + return false; + break; + } + case RSEQ_OP_RESET_WITH_STRIDE_CPUID: /* fall through */ + case RSEQ_OP_RESET_WITH_STRIDE_MMCID: { + struct rseq_op_reset_with_stride __user *op =3D + (struct rseq_op_reset_with_stride __user *)node; + u64 src, dst, dst_stride; + u32 len, index; + + scoped_user_read_access(op, efault) { + unsafe_get_user(src, &op->src, efault); + unsafe_get_user(dst, &op->dst, efault); + unsafe_get_user(dst_stride, &op->dst_stride, efault); + unsafe_get_user(len, &op->len, efault); + } + index =3D (type =3D=3D RSEQ_OP_RESET_WITH_STRIDE_CPUID) ? + t->rseq.ids.cpu_id : t->rseq.ids.mm_cid; + dst +=3D dst_stride * index; + + if (!rseq_op_reset_word(src, dst, len, node)) + return false; + + break; + } + default: + pr_info("rseq: bad operation type=3D%u from %p\n", + type, node); + return false; + } + + node =3D (struct rseq_op_node __user *)next; + } + + return node =3D=3D sentinel; +efault: + pr_info("rseq: fault while walking operation list\n"); + return false; +} + bool rseq_debug_update_user_cs(struct task_struct *t, struct pt_regs *regs= , unsigned long csaddr); =20 static __always_inline void rseq_note_user_irq_entry(void) @@ -632,6 +763,10 @@ static __always_inline bool rseq_exit_user_update(stru= ct pt_regs *regs, struct t if (unlikely(!rseq_update_user_cs(t, regs, csaddr))) return false; } + + if (t->rseq.event.rseq_op && !rseq_apply_ops(t)) + return false; + return true; } =20 @@ -642,11 +777,22 @@ static __always_inline bool rseq_exit_user_update(str= uct pt_regs *regs, struct t .node_id =3D cpu_to_node(cpu), }; =20 - return rseq_update_usr(t, regs, &ids); + if (!rseq_update_usr(t, regs, &ids)) + return false; + + if (t->rseq.event.rseq_op && !rseq_apply_ops(t)) + return false; + + return true; efault: return false; } =20 +static __always_inline void rseq_clear_one_shot_events(struct rseq_event *= ev) +{ + ev->events &=3D (struct rseq_event){ .rseq_op =3D true }.events; +} + static __always_inline bool __rseq_exit_to_user_mode_restart(struct pt_reg= s *regs) { struct task_struct *t =3D current; @@ -674,8 +820,9 @@ static __always_inline bool __rseq_exit_to_user_mode_re= start(struct pt_regs *reg if (unlikely(!rseq_exit_user_update(regs, t))) return true; } - /* Clear state so next entry starts from a clean slate */ - t->rseq.event.events =3D 0; + /* Clear one-shot events so next entry starts from a clean slate */ + rseq_clear_one_shot_events(&t->rseq.event); + return false; } =20 @@ -730,7 +877,7 @@ static __always_inline void rseq_syscall_exit_to_user_m= ode(void) /* Needed to remove the store for the !lockdep case */ if (IS_ENABLED(CONFIG_LOCKDEP)) { WARN_ON_ONCE(ev->sched_switch); - ev->events =3D 0; + rseq_clear_one_shot_events(ev); } } =20 @@ -747,7 +894,7 @@ static __always_inline void rseq_irqentry_exit_to_user_= mode(void) * interrupt did not result in a schedule and therefore the * rseq processing could not clear it. */ - ev->events =3D 0; + rseq_clear_one_shot_events(ev); } =20 void __rseq_debug_syscall_return(struct pt_regs *regs); --=20 2.54.0 From nobody Sat Sep 26 22:53:30 2026 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4513F369225 for ; Fri, 28 Aug 2026 15:41:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931676; cv=none; b=UBudoo6tqSES0b9SgKKEfRPUsQ9UP6cHOTUwIEG7Aw5hHDUceXO8lhOthbtAxmUiQNuPETEwq68+ywxHuYlXkwkZKjASg5KQQAs2WJy/VWC5w+yB56btKeraUbaXgLVlFaV5LtUdXA7YN/BUGVoIfAN0OFH6p77ucCInVWcWUCg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931676; c=relaxed/simple; bh=LxsOcjm6T6TGxVtZFKZeBX3WPoCIdhvpHW4CNWEtAgI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=axYQ2d1xNdQ8TulwQMOGzH4Ga/oyJQXscLn7WFHBk3TRCh6fGnt+DWfvjJb4R/mApQNEluXTuNDPG8dLdyGrs7VFitMGaz2+CGLjD51fprOJLOeMD+s2+5uzvFLvfmFaoGXdZAi37YNOYCGyFsbe+HKeWau10ACNlOWdLaLAoQw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=ePSdnMsJ; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="ePSdnMsJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1787931238; bh=5V6XawWF2fwKJpYjSWTgOA6QLkuanYWwMg4glTCKIGo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ePSdnMsJjfJxlsSrF/SP7O0aBdutdmtpTUR55F64yIC8Z6jjeZvXMU1uDL0S6U2UB QlmQ0Tz8SCp7C4xCS9ZVqZbRs0vxFM33VObmWDH0D1cJ8aB76i+yAi6DFK7LRMJKea n9k5uoJa1mxo8K1YwRDM5x7rn5EjOvM7VQa6AtN8w9AdOoigJvsnZmMVBZNaA9BtIK AUw4Sm5c0lEKrP9Okn2KRKV76SLbP0JLQaDLWl40/2bTa2UFYPGb8OfhYk2ecQuaKX xAzFW1ToU3TZZBq+wTro1k/zSHVWx0Zs3GceANP96XM1gRpB42/PS34mpN6E9gc63u 6+ir0LFjKDnLg== Received: from laura.localdomain (199-193-172-8.cpe.axion.ca [199.193.172.8]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4hWj916ySpzTCb; Fri, 28 Aug 2026 11:33:57 -0400 (EDT) From: odion@efficios.com To: Mathieu Desnoyers Cc: Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , LKML , Thomas Gleixner , Dmitry Vyukov , David Matlack , Marco Elver , Sean Christopherson , Wei Liu , Florian Weimer , Mathias Stearn , Chris Kennelly , Blake Oler , Rich Felker , Matthew Wilcox , Greg Kroah-Hartman , Carlos O'Donell , Olivier Dion Subject: [RFC PATCH 4/5] rseq: register and unregister operations via prctl Date: Fri, 28 Aug 2026 11:33:42 -0400 Message-ID: <20260828153349.8061-5-odion@efficios.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260828153349.8061-1-odion@efficios.com> References: <20260828153349.8061-1-odion@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Olivier Dion Implement the PR_RSEQ_OP prctl. Registration splices a user-provided pristine node into the head of the circular operation list, and unregistration unsplices it after validating it is properly linked between its neighbours; the kernel owns the node's next/prev links for the lifetime of the registration. The 0<->1 nr_ops transition reflects the enabled state into rseq_event::rseq_op and the user visible flag via rseq_op_update_enabled(). Initialize the operation list sentinel as an empty self-pointing circular list on rseq registration, and reset nr_ops / rseq_op state there. Apply pending operations on the slow path and on signal delivery, and factor the common "clear error and force SIGSEGV" fixup into force_fault(). Signed-off-by: Olivier Dion --- kernel/rseq.c | 220 +++++++++++++++++++++++++++++++++++++++++++++++--- kernel/sys.c | 5 ++ 2 files changed, 212 insertions(+), 13 deletions(-) diff --git a/kernel/rseq.c b/kernel/rseq.c index e75e3a5e312c..a277c11dae99 100644 --- a/kernel/rseq.c +++ b/kernel/rseq.c @@ -253,17 +253,18 @@ static bool rseq_handle_cs(struct task_struct *t, str= uct pt_regs *regs) static void rseq_slowpath_update_usr(struct pt_regs *regs) { /* - * Preserve has_rseq and user_irq state. The generic entry code clears - * user_irq on the way out, the non-generic entry architectures are not - * setting user_irq. + * Preserve has_rseq, rseq_op and user_irq state. The generic entry + * code clears user_irq on the way out, the non-generic entry + * architectures are not setting user_irq. */ const struct rseq_event evt_mask =3D { .has_rseq =3D RSEQ_HAS_RSEQ_VERSION_MASK, + .rseq_op =3D true, .user_irq =3D true, }; struct task_struct *t =3D current; struct rseq_ids ids; - bool event; + bool event, should_fault =3D false; =20 if (unlikely(t->flags & PF_EXITING)) return; @@ -300,7 +301,12 @@ static void rseq_slowpath_update_usr(struct pt_regs *r= egs) =20 ids.node_id =3D cpu_to_node(ids.cpu_id); =20 - if (unlikely(!rseq_update_usr(t, regs, &ids))) { + if (unlikely(!rseq_update_usr(t, regs, &ids))) + should_fault =3D true; + else if (t->rseq.event.rseq_op && unlikely(!rseq_apply_ops(t))) + should_fault =3D true; + + if (should_fault) { /* * Clear the errors just in case this might survive magically, but * leave the rest intact. @@ -329,8 +335,20 @@ void __rseq_handle_slowpath(struct pt_regs *regs) rseq_slowpath_update_usr(regs); } =20 +static inline void force_fault(int sig) +{ + /* + * Clear the errors just in case this might survive magically, but leave + * the rest intact. + */ + current->rseq.event.error =3D 0; + force_sigsegv(sig); +} + void __rseq_signal_deliver(int sig, struct pt_regs *regs) { + bool should_fault =3D false; + rseq_stat_inc(rseq_stats.signal); =20 /* @@ -339,14 +357,13 @@ void __rseq_signal_deliver(int sig, struct pt_regs *r= egs) * the interrupted context as after this point the instruction * pointer in @regs points to the signal handler. */ - if (unlikely(!rseq_handle_cs(current, regs))) { - /* - * Clear the errors just in case this might survive - * magically, but leave the rest intact. - */ - current->rseq.event.error =3D 0; - force_sigsegv(sig); - } + if (unlikely(!rseq_handle_cs(current, regs))) + should_fault =3D true; + else if (current->rseq.event.rseq_op && unlikely(!rseq_apply_ops(current)= )) + should_fault =3D true; + + if (should_fault) + force_fault(sig); =20 /* * In legacy mode, force the update of IDs before returning to user @@ -436,6 +453,8 @@ static long rseq_register(struct rseq __user * rseq, u3= 2 rseq_len, int flags, u3 rseqfl |=3D RSEQ_CS_FLAG_SLICE_EXT_ENABLED; } } + if (version > 1) + rseqfl |=3D RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE; =20 scoped_user_write_access(rseq, efault) { /* @@ -458,8 +477,16 @@ static long rseq_register(struct rseq __user * rseq, u= 32 rseq_len, int flags, u3 * registrations. */ if (version > 1) { + u64 sentinel =3D (u64)&rseq->rseq_op_list; + if (IS_ENABLED(CONFIG_RSEQ_SLICE_EXTENSION)) unsafe_put_user(0U, &rseq->slice_ctrl.all, efault); + /* + * Initialize the rseq operation list sentinel as an + * empty circular doubly-linked list pointing to itself. + */ + unsafe_put_user(sentinel, &rseq->rseq_op_list.next, efault); + unsafe_put_user(sentinel, &rseq->rseq_op_list.prev, efault); } } =20 @@ -474,6 +501,12 @@ static long rseq_register(struct rseq __user * rseq, u= 32 rseq_len, int flags, u3 #ifdef CONFIG_RSEQ_SLICE_EXTENSION current->rseq.slice.state.enabled =3D !!(rseqfl & RSEQ_CS_FLAG_SLICE_EXT_= ENABLED); #endif + /* + * A fresh registration starts with no operations, so operation + * processing is disabled until the first one is registered. + */ + current->rseq.event.rseq_op =3D false; + current->rseq.nr_ops =3D 0; =20 /* * Ensure the cpu_id_start and cpu_id fields are updated before @@ -887,3 +920,164 @@ device_initcall(rseq_slice_init); #else static void rseq_slice_ext_init(struct dentry *root_dir) { } #endif /* CONFIG_RSEQ_SLICE_EXTENSION */ + +/* + * Reflect the operation enabled state into the user visible flags field. + * Called on the 0<->1 transition of nr_ops. The kill path is taken on fau= lt + * because losing this update leaves user space and kernel state inconsist= ent. + */ +static int rseq_op_update_enabled(struct task_struct *t, bool enable) +{ + struct rseq __user *rseq =3D t->rseq.usrptr; + u32 rflags; + + if (get_user(rflags, &rseq->flags)) + return -EFAULT; + + rflags &=3D ~RSEQ_CS_FLAG_RSEQ_OP_ENABLED; + rflags |=3D RSEQ_CS_FLAG_RSEQ_OP_AVAILABLE; + if (enable) + rflags |=3D RSEQ_CS_FLAG_RSEQ_OP_ENABLED; + + if (put_user(rflags, &rseq->flags)) + return -EFAULT; + + t->rseq.event.rseq_op =3D enable; + return 0; +} + +/* + * Register @node at the head of the circular doubly-linked operation list + * anchored by the kernel owned sentinel in struct rseq. + */ +static int rseq_op_register(struct task_struct *t, struct rseq_op_node __u= ser *node) +{ + struct rseq_op_node __user *sentinel =3D &t->rseq.usrptr->rseq_op_list; + struct rseq_op_node __user *first; + u64 next, prev, first_addr; + u8 type, i; + + if (t->rseq.nr_ops >=3D RSEQ_OP_LIST_LIMIT) + return -ENOSPC; + + if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node))) + return -EINVAL; + if (!access_ok(node, sizeof(*node))) + return -EFAULT; + + /* + * The node links are owned by the kernel. User space must present a + * pristine node: next, prev and the reserved bytes all zeroed, and a + * known operation type. + */ + if (get_user(next, &node->next) || get_user(prev, &node->prev) || + get_user(type, &node->type)) + return -EFAULT; + if (next || prev) + return -EINVAL; + if (type >=3D RSEQ_OP_NR) + return -EINVAL; + for (i =3D 1; i < sizeof(node->reserved); i++) { + u8 r; + + if (get_user(r, &node->reserved[i])) + return -EFAULT; + if (r) + return -EINVAL; + } + + /* Splice the node in right after the sentinel. */ + if (get_user(first_addr, &sentinel->next)) + goto die; + first =3D (struct rseq_op_node __user *)first_addr; + + if (put_user((u64)(unsigned long)first, &node->next) || + put_user((u64)(unsigned long)sentinel, &node->prev) || + put_user((u64)(unsigned long)node, &first->prev) || + put_user((u64)(unsigned long)node, &sentinel->next)) + goto die; + + t->rseq.nr_ops +=3D 1; + + if (t->rseq.nr_ops =3D=3D 1) + return rseq_op_update_enabled(t, true) ? -EFAULT : 0; + return 0; +die: + force_sig(SIGSEGV); + return -EFAULT; +} + +/* + * Unregister @node from the operation list. The node links are validated + * against its neighbours to reject bogus or double unregistration. + */ +static int rseq_op_unregister(struct task_struct *t, struct rseq_op_node _= _user *node) +{ + struct rseq_op_node __user *prev, *next; + u64 prev_addr, next_addr, tmp; + + if (!t->rseq.nr_ops) + return -ENOENT; + + if (!IS_ALIGNED((unsigned long)node, __alignof__(struct rseq_op_node))) + return -EINVAL; + if (!access_ok(node, sizeof(*node))) + return -EFAULT; + + if (get_user(next_addr, &node->next) || get_user(prev_addr, &node->prev)) + return -EFAULT; + prev =3D (struct rseq_op_node __user *)prev_addr; + next =3D (struct rseq_op_node __user *)next_addr; + + /* A registered node always has both links set. */ + if (!prev || !next) + return -EINVAL; + + /* Verify the node is properly linked between its neighbours. */ + if (get_user(tmp, &prev->next)) + goto die; + if (tmp !=3D (u64)(unsigned long)node) + return -EINVAL; + if (get_user(tmp, &next->prev)) + goto die; + if (tmp !=3D (u64)(unsigned long)node) + return -EINVAL; + + /* Unsplice and clear the node links so it can be reused. */ + if (put_user(next_addr, &prev->next) || + put_user(prev_addr, &next->prev) || + put_user(0ULL, &node->next) || + put_user(0ULL, &node->prev)) + goto die; + + t->rseq.nr_ops -=3D 1; + + if (t->rseq.nr_ops =3D=3D 0) + return rseq_op_update_enabled(t, false) ? -EFAULT : 0; + return 0; +die: + force_sig(SIGSEGV); + return -EFAULT; +} + +int rseq_op_prctl(unsigned long arg2, unsigned long arg3) +{ + struct rseq_op_node __user *node =3D (struct rseq_op_node __user *)arg3; + struct task_struct *t =3D current; + + if (!t->rseq.usrptr) + return -ENXIO; + if (!rseq_v2(t)) + return -ENOTSUPP; + if (!node) + return -EINVAL; + + switch (arg2) { + case PR_RSEQ_OP_REGISTER: + return rseq_op_register(t, node); + case PR_RSEQ_OP_UNREGISTER: + return rseq_op_unregister(t, node); + default: + return -EINVAL; + } +} diff --git a/kernel/sys.c b/kernel/sys.c index df69bd71de03..494c91b03c3f 100644 --- a/kernel/sys.c +++ b/kernel/sys.c @@ -2889,6 +2889,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, a= rg2, unsigned long, arg3, return -EINVAL; error =3D rseq_slice_extension_prctl(arg2, arg3); break; + case PR_RSEQ_OP: + if (arg4 || arg5) + return -EINVAL; + error =3D rseq_op_prctl(arg2, arg3); + break; case PR_GET_CFI: if (arg2 !=3D PR_CFI_BRANCH_LANDING_PADS) return -EINVAL; --=20 2.54.0 From nobody Sat Sep 26 22:53:30 2026 Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 95E4D326939 for ; Fri, 28 Aug 2026 15:34:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=158.69.130.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931247; cv=none; b=p0ZJeBE5xeLOKZ5RyrQTGY3GreQ+bNoMEriutRAU4NjuQ/gjdf9pZi7EHMf+8HXDSHw1mJ5zaNrosZG0NpQDE//q+Cl0+/WNzqryHwgaCXRBtw96m1AGScTt/lXZw1Ol86dR1SNrMUXJPW5DS0aaDhFfbnI/80R1MPE3y0N6E/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787931247; c=relaxed/simple; bh=S2cYy7vhOT57cVqfL/vpOpekNqT+Eo0KcUG4NFUxEVw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IXQqUG7wuwub9s1c9IcPjeRHJhH0pWp6rizd1u/m6L/Ctlzth3Y1T8oYdxhnLjlc/NdFxyz8GmEGrG0LcWqHUs+Lj1aHSSs2ICkVn2BCWxd2V3GBIDRovgHw24bKCK74LHk9J7iKBz1v8+6CN9oKDvGWHkz9p+iWuTrAXOLMOUI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com; spf=pass smtp.mailfrom=efficios.com; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b=LuVWlOwZ; arc=none smtp.client-ip=158.69.130.18 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=efficios.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=efficios.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=efficios.com header.i=@efficios.com header.b="LuVWlOwZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=efficios.com; s=smtpout1; t=1787931238; bh=s3vazJV1gATKw5/G/0f4A7CGJjt06s+h9fsOEt1EOeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LuVWlOwZYOBzGpIEYIu2HxTBVud85sC7dNkGPjtwGyCyvK4W1guXGGBmbZfbnloT+ BjRZESyNEQENyHyNSqOStD+udkW62O0Nkg7mXdUDl9pXSzEe5gUYGXghkNoUqLMKQ0 DvbB5P1zjgWhMIr1NwEcAUNocT91URrTSjrkA6/+70lyppZfw5mTPqIRaezIYKMuVl 1fTegKLCnq+vCkwbie6Mdmz/81H/By0/2Ug3CTy346txXJgmphxDYa56Oqjn4UVAMX us3EltrPuCOKETGreYX8vZMREh7uVbM5R03MzJnwiD6/WUfK0qy54hK1ehg1WGT7Ys n78rTlsryaQOA== Received: from laura.localdomain (199-193-172-8.cpe.axion.ca [199.193.172.8]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4hWj92220FzTCc; Fri, 28 Aug 2026 11:33:58 -0400 (EDT) From: odion@efficios.com To: Mathieu Desnoyers Cc: Peter Zijlstra , "Paul E. McKenney" , Boqun Feng , LKML , Thomas Gleixner , Dmitry Vyukov , David Matlack , Marco Elver , Sean Christopherson , Wei Liu , Florian Weimer , Mathias Stearn , Chris Kennelly , Blake Oler , Rich Felker , Matthew Wilcox , Greg Kroah-Hartman , Carlos O'Donell , Olivier Dion Subject: [RFC PATCH 5/5] selftests/rseq: add coverage for rseq operations Date: Fri, 28 Aug 2026 11:33:43 -0400 Message-ID: <20260828153349.8061-6-odion@efficios.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260828153349.8061-1-odion@efficios.com> References: <20260828153349.8061-1-odion@efficios.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Olivier Dion Add selftest-side support for the rseq operation ABI and wire a new rseq_op_test into the rseq selftest suite. The new test covers: - reset before entering a signal handler, - reset on return to userspace after poll(), - reset after scheduler-driven preemption, - no-operation cases where nothing should be reset, - register-time rejection of non-pristine nodes and unknown types, - fatal cases for corrupted list links (self and transitive cycles), - invalid destination and next pointers, - unsupported lengths, - unaligned destinations, and - NULL-source clearing. Also extend the selftest ABI/helpers with the rseq operation node definitions and prctl-based register/unregister helpers, and add a dedicated runner that disables glibc's implicit rseq registration. Signed-off-by: Olivier Dion --- tools/testing/selftests/rseq/.gitignore | 4 +- tools/testing/selftests/rseq/Makefile | 4 +- tools/testing/selftests/rseq/rseq-abi.h | 146 ++++++++++++++++++++---- tools/testing/selftests/rseq/rseq.c | 23 +++- tools/testing/selftests/rseq/rseq.h | 75 ++++++++++++ 5 files changed, 223 insertions(+), 29 deletions(-) diff --git a/tools/testing/selftests/rseq/.gitignore b/tools/testing/selfte= sts/rseq/.gitignore index ec01d164c1f0..8d5f02d79b98 100644 --- a/tools/testing/selftests/rseq/.gitignore +++ b/tools/testing/selftests/rseq/.gitignore @@ -2,7 +2,6 @@ basic_percpu_ops_test basic_percpu_ops_mm_cid_test basic_test -basic_rseq_op_test param_test param_test_benchmark param_test_compare_twice @@ -11,3 +10,6 @@ param_test_mm_cid_benchmark param_test_mm_cid_compare_twice syscall_errors_test slice_test +check_optimized +legacy_check +rseq_op_test \ No newline at end of file diff --git a/tools/testing/selftests/rseq/Makefile b/tools/testing/selftest= s/rseq/Makefile index 50d69e22ee7a..5e761df0e5aa 100644 --- a/tools/testing/selftests/rseq/Makefile +++ b/tools/testing/selftests/rseq/Makefile @@ -22,12 +22,14 @@ TEST_GEN_PROGS_EXTENDED =3D librseq.so \ param_test_compare_twice \ param_test_mm_cid \ param_test_mm_cid_compare_twice \ + rseq_op_test \ syscall_errors_test \ legacy_check \ slice_test \ check_optimized =20 -TEST_PROGS =3D run_param_test.sh run_syscall_errors_test.sh run_legacy_che= ck.sh run_timeslice_test.sh +TEST_PROGS =3D run_param_test.sh run_syscall_errors_test.sh run_legacy_che= ck.sh run_timeslice_test.sh \ + run_rseq_op_test.sh =20 TEST_FILES :=3D settings =20 diff --git a/tools/testing/selftests/rseq/rseq-abi.h b/tools/testing/selfte= sts/rseq/rseq-abi.h index 5f4ea2152c2f..051636f84194 100644 --- a/tools/testing/selftests/rseq/rseq-abi.h +++ b/tools/testing/selftests/rseq/rseq-abi.h @@ -26,6 +26,10 @@ enum rseq_abi_cs_flags_bit { RSEQ_ABI_CS_FLAG_NO_RESTART_ON_PREEMPT_BIT =3D 0, RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT =3D 1, RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT =3D 2, + RSEQ_ABI_CS_FLAG_SLICE_EXT_AVAILABLE_BIT =3D 4, + RSEQ_ABI_CS_FLAG_SLICE_EXT_ENABLED_BIT =3D 5, + RSEQ_ABI_CS_FLAG_RSEQ_OP_AVAILABLE_BIT =3D 6, + RSEQ_ABI_CS_FLAG_RSEQ_OP_ENABLED_BIT =3D 7, }; =20 enum rseq_abi_cs_flags { @@ -35,6 +39,14 @@ enum rseq_abi_cs_flags { (1U << RSEQ_ABI_CS_FLAG_NO_RESTART_ON_SIGNAL_BIT), RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE =3D (1U << RSEQ_ABI_CS_FLAG_NO_RESTART_ON_MIGRATE_BIT), + RSEQ_ABI_CS_FLAG_SLICE_EXT_AVAILABLE =3D + (1U << RSEQ_ABI_CS_FLAG_SLICE_EXT_AVAILABLE_BIT), + RSEQ_ABI_CS_FLAG_SLICE_EXT_ENABLED =3D + (1U << RSEQ_ABI_CS_FLAG_SLICE_EXT_ENABLED_BIT), + RSEQ_ABI_CS_FLAG_RSEQ_OP_AVAILABLE =3D + (1U << RSEQ_ABI_CS_FLAG_RSEQ_OP_AVAILABLE_BIT), + RSEQ_ABI_CS_FLAG_RSEQ_OP_ENABLED =3D + (1U << RSEQ_ABI_CS_FLAG_RSEQ_OP_ENABLED_BIT), }; =20 /* @@ -74,6 +86,103 @@ struct rseq_abi_slice_ctrl { }; }; =20 +union rseq_ptr { + __u64 ptr64; + + /* + * The "arch" field provides architecture accessor for + * the ptr field based on architecture pointer size and + * endianness. + */ + struct { +#ifdef __LP64__ + __u64 ptr; +#elif defined(__BYTE_ORDER) ? (__BYTE_ORDER =3D=3D __BIG_ENDIAN) : defined= (__BIG_ENDIAN) + __u32 padding; /* Initialized to zero. */ + __u32 ptr; +#else + __u32 ptr; + __u32 padding; /* Initialized to zero. */ +#endif + } arch; +}; + +/* + * Maximum number of nodes walked in the rseq operation list. + */ +#define RSEQ_ABI_OP_LIST_LIMIT 2048 + +/* + * enum rseq_abi_op_type - Type of an rseq operation + * @RSEQ_ABI_OP_RESET: Plain reset. Uses struct rseq_abi_op_reset. + * @RSEQ_ABI_OP_RESET_WITH_STRIDE_CPUID: Reset indexed by the current CPU = ID. + * Uses struct rseq_abi_op_reset_with_stride. + * @RSEQ_ABI_OP_RESET_WITH_STRIDE_MMCID: Reset indexed by the current MM C= ID. + * Uses struct rseq_abi_op_reset_with_stride. + */ +enum rseq_abi_op_type { + RSEQ_ABI_OP_RESET, + RSEQ_ABI_OP_RESET_WITH_STRIDE_CPUID, + RSEQ_ABI_OP_RESET_WITH_STRIDE_MMCID, + RSEQ_ABI_OP_NR, +}; + +/* + * struct rseq_abi_op_node - Common header linking an rseq operation into = the list + * @next: Address of the next node. Owned by the kernel. + * @prev: Address of the previous node. Owned by the kernel. + * @type: Operation type. See enum rseq_abi_op_type. + * @reserved: Must be zero on registration. + * + * User space allocates the node, sets @type and zeroes @next, @prev and + * @reserved before passing it to prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, n= ode). + * The kernel owns @next and @prev for the lifetime of the registration and + * links the node into a circular doubly-linked list anchored by an intern= al + * sentinel in struct rseq_abi. User space must not touch @next or @prev w= hile + * the node is registered. + */ +struct rseq_abi_op_node { + __u64 next; + __u64 prev; + struct { + __u8 type; /* enum rseq_abi_op_type */ + __u8 reserved[7]; + }; +}; + +/* + * struct rseq_abi_op_reset - Reset one word to a value on return to user = space + * @node: Operation list node. + * @src: Address of the source word, or 0 to reset @dst to zero. + * @dst: Address of the destination word. + * @len: Word length in bytes. Must be 4 or 8. + */ +struct rseq_abi_op_reset { + struct rseq_abi_op_node node; + __u64 src; + __u64 dst; + __u32 len; +}; + +/* + * struct rseq_abi_op_reset_with_stride - Reset one word in a strided array + * @node: Operation list node. + * @src: Address of the source word, or 0 to reset the slot to zero. + * @dst: Base address of the strided destination array. + * @dst_stride: Stride in bytes between consecutive array slots. + * @len: Word length in bytes. Must be 4 or 8. + * + * The destination slot is @dst + @dst_stride * index, where index is the + * current CPU ID or MM CID depending on the operation type. + */ +struct rseq_abi_op_reset_with_stride { + struct rseq_abi_op_node node; + __u64 src; + __u64 dst; + __u64 dst_stride; + __u32 len; +}; + /* * struct rseq_abi is aligned on 4 * 8 bytes to ensure it is always * contained within a single cache-line. @@ -127,26 +236,7 @@ struct rseq_abi { * atomicity semantics. This field should only be updated by the * thread which registered this data structure. Aligned on 64-bit. */ - union { - __u64 ptr64; - - /* - * The "arch" field provides architecture accessor for - * the ptr field based on architecture pointer size and - * endianness. - */ - struct { -#ifdef __LP64__ - __u64 ptr; -#elif defined(__BYTE_ORDER) ? (__BYTE_ORDER =3D=3D __BIG_ENDIAN) : defined= (__BIG_ENDIAN) - __u32 padding; /* Initialized to zero. */ - __u32 ptr; -#else - __u32 ptr; - __u32 padding; /* Initialized to zero. */ -#endif - } arch; - } rseq_cs; + union rseq_ptr rseq_cs; =20 /* * Restartable sequences flags field. @@ -192,9 +282,21 @@ struct rseq_abi { struct rseq_abi_slice_ctrl slice_ctrl; =20 /* - * Place holder to push the size above 32 bytes. + * Sentinel of the circular doubly-linked list of rseq operations + * registered via prctl(PR_RSEQ_OP, ...). Fully owned and maintained by + * the kernel: it is initialized to point to itself on registration and + * user space must never read or write it directly. + * + * The kernel only use next and prev from rseq_op_list. The rest of the + * bytes are reserved for later usage and should be zeroed. */ - __u8 __reserved; + union { + struct rseq_abi_op_node rseq_op_list; + struct { + __u64 op_used[2]; + __u64 reserved; + }; + }; =20 /* * Flexible array member at end of structure, after last feature field. diff --git a/tools/testing/selftests/rseq/rseq.c b/tools/testing/selftests/= rseq/rseq.c index be0d0a97031e..6a5406f3353b 100644 --- a/tools/testing/selftests/rseq/rseq.c +++ b/tools/testing/selftests/rseq/rseq.c @@ -116,6 +116,17 @@ bool rseq_available(void) } } =20 +/* The rseq areas need to be at least 32 bytes. */ +static +unsigned int get_rseq_min_alloc_size(void) +{ + unsigned int alloc_size =3D rseq_size; + + if ((int) alloc_size < ORIG_RSEQ_ALLOC_SIZE) + alloc_size =3D ORIG_RSEQ_ALLOC_SIZE; + return alloc_size; +} + /* * Return the feature size supported by the kernel. * @@ -261,12 +272,14 @@ void rseq_init(void) =20 /* rseq flags are deprecated, always set to 0. */ rseq_flags =3D 0; + { + unsigned int rseq_kernel_feature_size =3D get_rseq_kernel_feature_size(); =20 - /* - * Set the size to 0 until at least one thread registers to mimic the - * libc behavior. - */ - rseq_size =3D 0; + if (rseq_kernel_feature_size <=3D RSEQ_THREAD_AREA_ALLOC_SIZE) + rseq_size =3D rseq_kernel_feature_size; + else + rseq_size =3D ORIG_RSEQ_ALLOC_SIZE; + } } =20 static __attribute__((destructor)) diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/= rseq/rseq.h index c62ebb9290c0..8f65d272e7cf 100644 --- a/tools/testing/selftests/rseq/rseq.h +++ b/tools/testing/selftests/rseq/rseq.h @@ -18,6 +18,9 @@ #include #include #include +#include +#include +#include #include "rseq-abi.h" #include "compiler.h" =20 @@ -395,4 +398,76 @@ int rseq_cmpeqv_trymemcpy_storev(enum rseq_mo rseq_mo,= enum rseq_percpu_mode per } } =20 +/* + * prctl commands for the rseq operation list. The kernel owns the list: t= he + * feature is enabled implicitly by registering the first operation and + * disabled by unregistering the last one. There is no explicit enable kno= b. + */ +#ifndef PR_RSEQ_OP +#define PR_RSEQ_OP 82 +#define PR_RSEQ_OP_REGISTER 1 +#define PR_RSEQ_OP_UNREGISTER 2 +#endif + +/* + * Initialize a plain reset operation. Sets the operation type, the destin= ation + * and source words, the word length, and leaves the node in a pristine st= ate + * (next, prev and reserved bytes zeroed) as required by the kernel at + * registration time. + */ +static inline +void rseq_op_reset_init(struct rseq_abi_op_reset *op, + void *dst, void *src, size_t len) +{ + op->node.next =3D 0; + op->node.prev =3D 0; + op->node.type =3D RSEQ_ABI_OP_RESET; + memset(op->node.reserved, 0, sizeof(op->node.reserved)); + op->src =3D (__u64)(unsigned long)src; + op->dst =3D (__u64)(unsigned long)dst; + op->len =3D len; +} + +/* + * Initialize a strided reset operation indexed by the current CPU ID or M= M CID + * depending on @type. + */ +static inline +void rseq_op_reset_with_stride_init(struct rseq_abi_op_reset_with_stride *= op, + enum rseq_abi_op_type type, + void *dst, void *src, + size_t dst_stride, size_t len) +{ + op->node.next =3D 0; + op->node.prev =3D 0; + op->node.type =3D type; + memset(op->node.reserved, 0, sizeof(op->node.reserved)); + op->src =3D (__u64)(unsigned long)src; + op->dst =3D (__u64)(unsigned long)dst; + op->dst_stride =3D (__u64)dst_stride; + op->len =3D len; +} + +/* + * Register an rseq operation node. The kernel links the pristine node int= o its + * internal list and enables operation processing when the first node is + * registered. Returns the prctl() return value (0 on success). + */ +static inline +int rseq_op_register(struct rseq_abi_op_node *node) +{ + return prctl(PR_RSEQ_OP, PR_RSEQ_OP_REGISTER, (unsigned long)node, 0, 0); +} + +/* + * Unregister an rseq operation node. The kernel unlinks the node, clears = its + * next/prev links, and disables operation processing when the last node is + * unregistered. Returns the prctl() return value (0 on success). + */ +static inline +int rseq_op_unregister(struct rseq_abi_op_node *node) +{ + return prctl(PR_RSEQ_OP, PR_RSEQ_OP_UNREGISTER, (unsigned long)node, 0, 0= ); +} + #endif /* RSEQ_H_ */ --=20 2.54.0