From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135819; cv=none; d=zohomail.com; s=zohoarc; b=B7BQOO2CfQbmfirqs1tho/Mj5TRk0UJNT7WmwP7m9L/AIhpy5Uk14TevbcwtchWnAhNCeKnzuHsTegqpG86Ekzls3ZHYepn2t+1kcdUXZuLNXNR8+lGMY4pKsXdYMxJcy+waHZSZUPCdusn2JhyQGKkCMiEgzMTbZCY0Y6JUC3I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135819; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+LMR5vylWFXJjVBWLTvtKp/AcLzLdtKKT/apfY5nlpQ=; b=dQnTDgsxi8zfdiQcoQVsLAwVFbz6QMPk3i5ACGTKNS8SL8/NYYmPzEbqcMng1l59z3LFyasoLC5lf1XWhoLziVxvk3Pci1kQYERf6tO1Qw4a5skNwnq7O885fQ+IVbKXjy7gMOc/AhDVdMmiDmc4IejFCZY47YYc+oR0dXQpYhk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135819610305.178421849894; Fri, 11 Sep 2026 07:10:19 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416854.1645774 (Exim 4.92) (envelope-from ) id 1x51wt-0005Jh-4l; Fri, 11 Sep 2026 14:09:51 +0000 Received: by outflank-mailman (output) from mailman id 1416854.1645774; Fri, 11 Sep 2026 14:09:51 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wt-0005Ja-1z; Fri, 11 Sep 2026 14:09:51 +0000 Received: by outflank-mailman (input) for mailman id 1416854; Fri, 11 Sep 2026 14:09:49 +0000 Received: from mx.expurgate.net ([194.145.224.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wr-0005JU-Ed for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:09:49 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51wq-008feO-Rj for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:09:48 +0200 Received: from [10.42.69.5] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40ba3-8faa-0a2a0a5109dd-0a2a4505898e-24 for ; Fri, 11 Sep 2026 16:09:48 +0200 Received: from [209.85.128.170] (helo=mail-yw1-f170.google.com) by tlsNG-c201ff.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bab-4cb1-0a2a45050019-d15580aacc87-3 for ; Fri, 11 Sep 2026 16:09:48 +0200 Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-836c8bde2dcso8056307b3.0 for ; Fri, 11 Sep 2026 07:09:48 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120ef650a6sm22355976d6.0.2026.09.11.07.09.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:44 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135787; x=1789740587; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+LMR5vylWFXJjVBWLTvtKp/AcLzLdtKKT/apfY5nlpQ=; b=dk7uol5T3YTxr4y+210IUhr6PsWhsjyaIP9E3IDSVCSx0Wx79wdPvXs3wTBVY5bums 70jHoJ7dg4C4jc4H0PQwF5WX1CNxUZky89z5HeLrwGd0utYhxNsoqQj3pyiwOrAVrNSC +9xN8QmVwIrmDK3Zh9DNQlhoAXklNG+Tg1KzFKQB3bSpoAyqSb0Dcp6czqzc5usdVhvN sG3rN1MVTRZyZ0Mrw4EEWwLi9xLRYE4yrMer5Rz7Lzszh8PRlrOzVvEsOwptHhvTyGNB XG8ARsvQv4m+Zyy73E0EctN5k9xfJXh0+DC3tnzOQys+7sO4iUCNBwrqaPgVHV92j2yN 6+jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135787; x=1789740587; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+LMR5vylWFXJjVBWLTvtKp/AcLzLdtKKT/apfY5nlpQ=; b=JmlmaVvvZ/STlR34YLilwO7YNILGPl9yFWSxTa7powzFOMkIT95FfJve9nQ6hLiLYu +fD2SuyjnMYE9cwm7Gi5D0EFEDPU8uddMICL2dUIQoAD7xCS374zRS2MUA1dBVHLOH4Z 3DqtxPdM0Mf9ZaIpFe/2Mr+BX/uSi9djoxZ2bj7ySqV8Ihk5bGzCo7ObmJ6Ud2FafBvc lGKdzxCKIBedItDZjWU+k4CZGL/iiOa4TWfGtIPm44tb3Qo7OZqpo9tv+L9EVss1xrjH Yb5XvGzEeOywk2Pf7quhORfvcoYCaF0C9LsjSiLRtBQ7F2lGXoHyw70KZYdEi7sbAf0G 3uxA== X-Forwarded-Encrypted: i=1; AKwUvBwmlyKW4CxydxmhyghtiMgfzvXaqWBOHDgLZUR25LU39eef65vVej/Ck0/BjTQU/kEylnVMKoGlHgU=@lists.xenproject.org X-Gm-Message-State: AFuF++lCKfHv0VOYufeJWk0/jS5KyohRYuB51B4GMfoEJMox8Y0sYAkg ejyvmAVHE8IozccwP4CKNKdw4Pxz1xwc04xW4EyAEwooiA42xl2BfgqtrTzQWaAPxZg= X-Gm-Gg: AYBFou1EYvsR+hZG/M/W2d+cu1+dFuPg3ZDFiXspHE4hXrExkgndRZ+Txmqcl7w/8OW actKErk6/JFuXMomk74kXQp4mMxbFtycuQ6njiwaU4UApxvY+JqwRPSq6zvSphE9pwjXiLImuIt r6G53nAjXdRC4YeNAlFcqUNDuBGcpEWBrhg6rSSJSlrNdIxnVNuUoTK+6RS//ed+o5JhyTZlwAt YSEVUU0jW7DuEoWd0AH7eqwbincl9kvkzf849QGyICAWuSiy4GyzwlxE7T4AmMemMn1de1t0abz nYCnGnqUEVqQZZrun9sdqsDpSAOjYn/abVqI3dQnPLZyS60iQzGitYmRFhuEf74Rd9zNPGhmUGH pdUQE2TQT295a5sPEY+R9U0cncBGxpB3psMkZJlA2+bVGPe1B9OrKVmZBk1mGZ693zLF+ez3+Gm iDb3ydqL1jPxt8hX3us9G11zwho2nheC11jEmUN5x0Ra3qfAZZmBE4540dQ93LoIBsS/wYw6QpW 9Q4+New7Fn0uT/jhkIMleOuOoZks14r1uSzFdomXP1Cnxm8gW0Tq/xYt11a4OVyQwA= X-Received: by 2002:a53:ac84:0:b0:671:2bb8:ceb0 with SMTP id 956f58d0204a3-6712bb8d198mr592173d50.65.1789135786904; Fri, 11 Sep 2026 07:09:46 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:39 +0000 Subject: [PATCH RFC v2 01/15] rcu-tasks: Add per-task trampoline nesting count MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-1-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=7684; i=josef@toxicpanda.com; h=from:subject:message-id; bh=5hfL8mlrp+YP87pgAlSpHeSx0zMtfymEMHCNh6a40UA=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QPWvMx3TEehoRUZzyZvbL0GIDlx2yxh9C8YdBR0qgc4jATwF0zYzIPzSM+uo5u0rpazhBa063Yn SUZFFg50YiQg= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-c201ff/1789135788-738BE2A1-27375FBD/0/0 X-purgate-type: clean X-purgate-size: 7686 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135821816158500 Tasks RCU exists so that ftrace, BPF and kprobes can free trampoline text once no task can still be executing in it. Today the only way a task tells Tasks RCU "I am not in a trampoline" is a voluntary context switch, so a preempted task is always assumed to be inside one. Add task_struct::rcu_tramp_nesting so that trampolines can say so directly: a trampoline increments it before calling out and decrements it before returning, and while it is non-zero the task must not be treated as Tasks-RCU quiescent. Provide rcu_tasks_trampoline_enter() and rcu_tasks_trampoline_exit() for C users, report the count in the Tasks RCU stall output, and, under CONFIG_PROVE_RCU, assert that it is zero on every return to userspace since no task can legitimately reach userspace with a trampoline on its stack. Only current ever writes the count and every nested user (interrupts running their own trampolines) is balanced, so plain accesses suffice. The callbacks reached from static trampolines (return_to_handler, the rethook and kretprobe trampolines) are covered by the preempt_disable() in the ftrace recursion protection rather than by the count; note that dependency in trace_recursion.h so it is not lost if the preempt_disable() is ever removed from there. Nothing increments the count and nothing consults it for quiescent-state decisions yet; both come in later patches. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/irq-entry-common.h | 2 ++ include/linux/rcupdate.h | 37 +++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + include/linux/trace_recursion.h | 11 +++++++++++ kernel/fork.c | 1 + kernel/rcu/tasks.h | 3 ++- 6 files changed, 54 insertions(+), 1 deletion(-) diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 0bb6c03481fa..8da571622000 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validat= e(void) { /* Ensure that kernel state is sane for a return to userspace */ kmap_assert_nomap(); + rcu_tasks_trampoline_assert_none(); lockdep_assert_irqs_disabled(); lockdep_sys_exit(); } diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 44c07a66edff..b5c666c82479 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -180,6 +180,37 @@ static inline void rcu_nocb_flush_deferred_wakeup(void= ) { } #ifdef CONFIG_TASKS_RCU_GENERIC =20 # ifdef CONFIG_TASKS_RCU + +/* + * Trampoline nesting: dynamically allocated text (ftrace trampolines, BPF + * trampoline images, kprobe optinsn slots) that relies on Tasks RCU for i= ts + * lifetime brackets itself with an increment/decrement of + * current->rcu_tramp_nesting. While the count is non-zero the task is in= side, + * or was called from, such text and an involuntary context switch must no= t be + * treated as a Tasks RCU quiescent state. + * + * Only current writes the count and only current (or an interrupt on the = same + * CPU) reads it, so plain accesses suffice. + */ +static __always_inline void rcu_tasks_trampoline_enter(void) +{ + current->rcu_tramp_nesting++; + barrier(); +} + +static __always_inline void rcu_tasks_trampoline_exit(void) +{ + barrier(); + current->rcu_tramp_nesting--; +} + +/* A task must never reach userspace with a trampoline on its stack. */ +static __always_inline void rcu_tasks_trampoline_assert_none(void) +{ + if (IS_ENABLED(CONFIG_PROVE_RCU)) + WARN_ON_ONCE(current->rcu_tramp_nesting); +} + # define rcu_tasks_classic_qs(t, preempt) \ do { \ if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ @@ -192,6 +223,9 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); # define rcu_tasks_classic_qs(t, preempt) do { } while (0) # define call_rcu_tasks call_rcu # define synchronize_rcu_tasks synchronize_rcu +static inline void rcu_tasks_trampoline_enter(void) { } +static inline void rcu_tasks_trampoline_exit(void) { } +static inline void rcu_tasks_trampoline_assert_none(void) { } # endif =20 #define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) @@ -208,6 +242,9 @@ void exit_tasks_rcu_finish(void); #define rcu_tasks_classic_qs(t, preempt) do { } while (0) #define rcu_tasks_qs(t, preempt) do { } while (0) #define rcu_note_voluntary_context_switch(t) do { } while (0) +static inline void rcu_tasks_trampoline_enter(void) { } +static inline void rcu_tasks_trampoline_exit(void) { } +static inline void rcu_tasks_trampoline_assert_none(void) { } #define call_rcu_tasks call_rcu #define synchronize_rcu_tasks synchronize_rcu static inline void exit_tasks_rcu_start(void) { } diff --git a/include/linux/sched.h b/include/linux/sched.h index 8b3d47a325cc..d2e7b1b3c9d2 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -956,6 +956,7 @@ struct task_struct { unsigned long rcu_tasks_nvcsw; u8 rcu_tasks_holdout; u8 rcu_tasks_idx; + int rcu_tramp_nesting; int rcu_tasks_idle_cpu; struct list_head rcu_tasks_holdout_list; int rcu_tasks_exit_cpu; diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursio= n.h index e6ca052b2a85..2da23a52ca4a 100644 --- a/include/linux/trace_recursion.h +++ b/include/linux/trace_recursion.h @@ -153,6 +153,17 @@ static __always_inline int trace_test_and_set_recursio= n(unsigned long ip, unsign current->trace_recursion =3D val; barrier(); =20 + /* + * Callbacks reached from static trampoline text (return_to_handler, + * the rethook and kretprobe trampolines) do not maintain + * current->rcu_tramp_nesting themselves; they rely on this + * preempt_disable() to keep the task from being preempted, and thus + * from reporting a Tasks RCU quiescent state, while an ftrace_ops or + * its data is in use. If the preempt_disable() is ever removed from + * the recursion protection, this must rcu_tasks_trampoline_enter() + * here and rcu_tasks_trampoline_exit() in trace_clear_recursion() + * instead. See CONFIG_RCU_TASKS_PREEMPT_QS. + */ preempt_disable_notrace(); =20 return bit; diff --git a/kernel/fork.c b/kernel/fork.c index 416758c8a3d4..cfe3a8e53fbd 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1869,6 +1869,7 @@ static inline void rcu_copy_process(struct task_struc= t *p) #endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_TASKS_RCU p->rcu_tasks_holdout =3D false; + p->rcu_tramp_nesting =3D 0; INIT_LIST_HEAD(&p->rcu_tasks_holdout_list); p->rcu_tasks_idle_cpu =3D -1; INIT_LIST_HEAD(&p->rcu_tasks_exit_list); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 627295396cd9..1662ba18bf34 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1113,10 +1113,11 @@ static void check_holdout_task(struct task_struct *= t, *firstreport =3D false; } cpu =3D task_cpu(t); - pr_alert("%p: %c%c nvcsw: %lu/%lu holdout: %d idle_cpu: %d/%d\n", + pr_alert("%p: %c%c nvcsw: %lu/%lu holdout: %d tramp_nesting: %d idle_cpu:= %d/%d\n", t, ".I"[is_idle_task(t)], "N."[cpu < 0 || !tick_nohz_full_cpu(cpu)], t->rcu_tasks_nvcsw, t->nvcsw, t->rcu_tasks_holdout, + data_race(t->rcu_tramp_nesting), data_race(t->rcu_tasks_idle_cpu), cpu); sched_show_task(t); } --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135811; cv=none; d=zohomail.com; s=zohoarc; b=TZGKBxp5pcNxamHNRQ010rlsbAmrEH0k7CT8Vo1VUecw8sOVovxZGie/uAqFeVi3029+wJ3Dq/EjHRhVSB8NdMY18lID+VRm+Pjyxr6w/FnZO3tRUOirua1fm4+6haLuyHNKqmMTcNH5fGtzsDkRq/gE9QexZ3zbf6nqbnkOabQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135811; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=aiCVs9APKjX6sZKgVhMsXAxqlrvFEVJaRs9F0HXfb1vWT6clSguX7A3YpRzwWBHwDfYPhFTn657nu8bxrY03PJ2Viazn8ldj0TahLSs/s8GrrHh3sQGXbpFxvxnsW9A0D1zPMHthWjAQccNIcLRUL650gA91sVUUohwUqmfax/o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135811340278.03797096231085; Fri, 11 Sep 2026 07:10:11 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416855.1645783 (Exim 4.92) (envelope-from ) id 1x51ww-0005Wa-Ez; Fri, 11 Sep 2026 14:09:54 +0000 Received: by outflank-mailman (output) from mailman id 1416855.1645783; Fri, 11 Sep 2026 14:09:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51ww-0005WT-B2; Fri, 11 Sep 2026 14:09:54 +0000 Received: by outflank-mailman (input) for mailman id 1416855; Fri, 11 Sep 2026 14:09:53 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wv-0005WB-9m for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:09:53 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51wu-009TbZ-HP for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:09:52 +0200 Received: from [10.42.69.2] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40bad-bab6-0a2a0a5309dd-0a2a4502afbc-20 for ; Fri, 11 Sep 2026 16:09:52 +0200 Received: from [74.125.230.140] (helo=mail-qv2-f12.google.com) by tlsNG-720697.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40baf-6ca4-0a2a45020019-4a7de68c8e02-3 for ; Fri, 11 Sep 2026 16:09:52 +0200 Received: by mail-qv2-f12.google.com with SMTP id 6a1803df08f44-90cdfbd148cso9161746d6.1 for ; Fri, 11 Sep 2026 07:09:52 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f478bf1sm21734556d6.25.2026.09.11.07.09.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:48 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135791; x=1789740591; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=I9KwG09aoUS+hu71dRDVO81iZgY5dD1qArXVwta18hRKE+F5BdR3WqOc0bNMtJp4lo 4KTV5NQ4G4+jV2iIcw9/pwqr8YBtblTpwtOtzN+ZWhF+uVOp9k/wkR4TM3NnSNJuDk73 wtOCJPUrYDjBwI9dYrniNoR1X48Zv+SzHHzVP/9MhS9UDROrGKvzojQi9SuiSCoB5wAt rXqIjpIt93z9dLRiMVJhzL6mOxfjEQrJj2jv4ffNAPH2dC0bdx0LYro3KXGFRfLDAdpJ QKwItdDXf2HoTieSQ05SyzZPnRKkFH7lyO0yvJhzk7G9v1QgRZnRDmcyJJxGpfjXWVBx /2iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135791; x=1789740591; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=XSpVtNQJaM5x3ebQcgdHw85pVIcpQX+ggrfnQxj52RdRwimGIQhuwXE25fdHuhd+Y/ rkEnUg1ZOSIysnbymHf+lu7at628Fv406nk9qt4RnXqye1CWo99NQAoLRjPgcWLDBJDV MU4ewTFvkTJd7tRj7luv7t7A1mJRjzQUH8cN7soy8HkkfGHJOIZR33JZYekF57FP1wwH INLwqLZUwOlgoeGtxH/JDreDNOJcSbFZf1kHCDwMi+UvXx4bdeBy5NWXF22MZv7ONLoe GYT4VYvdRZskUNLX8EdNN/6QywToTr9MLsHlx5XRhJxFu+FTTEgsj1qfF6lRLbuaScd2 IU/Q== X-Forwarded-Encrypted: i=1; AKwUvBzykGfl8SbzWkNVda9oCSAeaYyo0S+e1eBM5fqfCYq6wdil97e6HThqF9eaTXcd3DCZ2k8/U6S0UJs=@lists.xenproject.org X-Gm-Message-State: AFuF++mFfFfnfXM6XWPrB9MQYgLSWXcRMR6/wi82HpErOKC7g5e9lULC RCMxbQzFmmWLOz/dK0UjHQ+0AW4Vj6dL9IUJdlXOjPhKkPHoOj591jP6+0Pz8JXtXms= X-Gm-Gg: AYBFou16v9FqXSe2Tqc3WvLwrvh92P0G+MFO6Hzz0MdBepD0bPwlNU+6mRmkG4mqF14 kU2vkB6j0EQLHlqUhrbO4KgQDqx2Dqn0a6bJrNqDiwz4PQeqmb/XNLIOXwGzTxX+f7H3mMr6ChJ ycenTqvDSdN4sGABs5Zx9vFm6FDyQ7GAOO8EAwPOIdlE0Ffi7bDvtDf+owa4ox4c8ADH3moLKUd n/b6DiEhwQKnPDdXJObVVMixzSkADOJAWvQfwMXfhX+58Lp5zxPoYAJnqF9tsouDPvekgdhZl3t FHbvmr6UqiqKcXqqob3PEmHbBw5FPW9xS04dVd6ZMn18xRxL4F0T0ztJzzCdm4af3CXPMH+20vT oX6JrAAAXFLkBFJ5dBxpNd4r0Q+NtqagQPs0fXxDW/0s7jnZT7ah9lomEk14XUw3awGJsBxkAdy WVMd9HnUA93fEtwQRno4CDytADlQ/n1H/IdRp1ji02JNqujCLympEtn74vkK8oaN5emRPx4JWwz 5yMoibvCDcBlePsT24ncc40jG0P3dbbamLqzRPVmmpsH88J7dduWINi X-Received: by 2002:a05:6214:5297:b0:912:bed:c40 with SMTP id 6a1803df08f44-91212116f69mr58502326d6.34.1789135790416; Fri, 11 Sep 2026 07:09:50 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:40 +0000 Subject: [PATCH RFC v2 02/15] entry: Pass pt_regs to irqentry_exit_cond_resched() MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-2-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=4149; i=josef@toxicpanda.com; h=from:subject:message-id; bh=lYdFz9QIpBbv18s7vNFL415owdJ/HvkbTUnKX5a6LVk=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QNLFUziLYUgokPYUCSvxyZJ4wGM33t0Exk3VASr1V813+Mcb6CwBQQCYTgKz4pDcXnm0a03w7j0 ZSuWxmEEtwQ4= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-720697/1789135792-31DC92AC-F39C4332/0/0 X-purgate-type: clean X-purgate-size: 4151 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135813953158500 The irq-exit preemption path is about to need the interrupted context's registers to decide whether the preemption may be reported to Tasks RCU as a quiescent state. irqentry_exit_to_kernel_mode_preempt() already has them; hand them down through irqentry_exit_cond_resched(), its PREEMPT_DYNAMIC static-call and static-key variants, and raw_irqentry_exit_cond_resched(). The only caller outside the generic entry code is Xen PV's upcall handler, which has regs as well. No functional change. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/xen/enlighten_pv.c | 2 +- include/linux/irq-entry-common.h | 12 ++++++------ kernel/entry/common.c | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 2c64b388f616..3d85035f5624 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct p= t_regs *regs) =20 inhcall =3D get_and_clear_inhcall(); if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) { - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); instrumentation_end(); restore_inhcall(inhcall); } else { diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 8da571622000..fc04725ae46b 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -348,21 +348,21 @@ typedef struct irqentry_state { * * Conditional reschedule with additional sanity checks. */ -void raw_irqentry_exit_cond_resched(void); +void raw_irqentry_exit_cond_resched(struct pt_regs *regs); =20 #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) #define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched #define irqentry_exit_cond_resched_dynamic_disabled NULL DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); -#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() +#define irqentry_exit_cond_resched(regs) static_call(irqentry_exit_cond_re= sched)(regs) #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void); -#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched() +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs); +#define irqentry_exit_cond_resched(regs) dynamic_irqentry_exit_cond_resche= d(regs) #endif #else /* CONFIG_PREEMPT_DYNAMIC */ -#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() +#define irqentry_exit_cond_resched(regs) raw_irqentry_exit_cond_resched(re= gs) #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 /** @@ -467,7 +467,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt= (struct pt_regs *regs, return; =20 if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); } =20 /** diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e3d381fd3d25..e4acd50bd81a 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -134,7 +134,7 @@ static inline bool arch_irqentry_exit_need_resched(void= ); static inline bool arch_irqentry_exit_need_resched(void) { return true; } #endif =20 -void raw_irqentry_exit_cond_resched(void) +void raw_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!preempt_count()) { /* Sanity check RCU and thread stack */ @@ -150,11 +150,11 @@ void raw_irqentry_exit_cond_resched(void) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resc= hed); #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void) +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) return; - raw_irqentry_exit_cond_resched(); + raw_irqentry_exit_cond_resched(regs); } #endif #endif --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135826; cv=none; d=zohomail.com; s=zohoarc; b=O3DmksDpRyU2ALO+OUbWKktMg9Z7ob4MLg0Cc8HGFgVUThRoRMuCJEJ2vrnqUz30Y+fidt1nQOjNaB9YIvsj6yMCZarDQ9nxDHTH5mfixBpeCezqtLjbtuWUyoZZkpW0KCygumXg7cqzDRQfduJOYL2K3cGdwT+mTmocn/KbcM8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135826; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=c88rGMLTws06IbDiZ6+QO44UtrNvdJ7MCjkuNiFKda4GoCk/E9zHnbu91LNJtUVGrwmgCzkntDGVCui8oGGhZ1nADfYyKIbjn0/16oMHzi7P8fLbU+2c3zydKFPEF2gLVSKKwRbY47VU7ktVQwihFkYXwfAr0v8h7yVXqtLJX4w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135826016806.7929294814707; Fri, 11 Sep 2026 07:10:26 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416856.1645792 (Exim 4.92) (envelope-from ) id 1x51wy-0005jf-Md; Fri, 11 Sep 2026 14:09:56 +0000 Received: by outflank-mailman (output) from mailman id 1416856.1645792; Fri, 11 Sep 2026 14:09:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wy-0005jY-J0; Fri, 11 Sep 2026 14:09:56 +0000 Received: by outflank-mailman (input) for mailman id 1416856; Fri, 11 Sep 2026 14:09:55 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wx-0005he-Bp for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:09:55 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51ww-00E9V9-Oz for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:09:54 +0200 Received: from [10.42.69.11] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40b97-2eae-0a2a0a5409dd-0a2a450bac0c-30 for ; Fri, 11 Sep 2026 16:09:54 +0200 Received: from [209.85.160.178] (helo=mail-qt1-f178.google.com) by tlsNG-42698a.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bb1-b7e8-0a2a450b0019-d155a0b2c932-3 for ; Fri, 11 Sep 2026 16:09:54 +0200 Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-5218927884fso15772411cf.3 for ; Fri, 11 Sep 2026 07:09:54 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca50cd05sm20570811cf.29.2026.09.11.07.09.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:51 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135793; x=1789740593; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=kVyTMRfXf7wDivyTuDFHix8y410Go95Qeg51c+AX3MnfvkkY67XHNgz6JLU11gfjgN wQyLDpEjKI1HAugLWm4hy61wyzWo6+Gix4YyB3I/rAb+6IdV3vArQMrUoZSWwdQdMkj0 bHsXPCj7+gtu7h3jku1mYZAy7K9y312ayhpwGTf3fiIZU+33NNPeACN1RIoQQ3T4fXk8 HeaoAJfSytexb0q29w1OOvAIx6XlkpQYw/TMkhi8JUbloH/mvf51lKXIDDHMyr8Wwn/z gtF8U2NTFpKThApkE+ou8dNxMER4TcifpnALQN8Xkwb/KGA0aKrmnF4x/716vePMfzpF lA8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135793; x=1789740593; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=aPMDRb8l99PXZ2lejYyyjz+WfA0LqkdCX4BtTXmKf/UvulSH2NwTHJIx68qtOf0TVP KAQmY6S7V8VHkUt5Nv2aLqG8By99mpPMsEdTphB8eI2aOguSbj+sFWVY2YHxUittea2O ebPPM4s+4h0Cpwv/+uD7LOLaCa+bH5j6ApY62JXV3IxI9BFmXgeBEMk4WHYbjJU3Fjdp OP3WJChKXlrDKKRYqoigFZhIjb7eRTEkbVpiTqmmDHbP7m+x9hvnd03PS9ws2Jt/6iHY WRkLyHQ9JJ/Ew7V/OyOUzwTOryH+frV272GINI+WLqZKndbWnHKeBuLYGcsmCmO8Ro64 jXhg== X-Forwarded-Encrypted: i=1; AKwUvByS7FOFs8iYgb83Hu1dSf0udJ2piFkLPf4cULguS7+GhoC2/VaNUvmVp7bCcfBCLBdLsSLKFAH9u/o=@lists.xenproject.org X-Gm-Message-State: AFuF++kWmTkN4yeHuWBMZuHPy7qV0ujP1lS2Z61N34rb9aVtn6Xo+/rJ fejW6jpD2eia/A2oBU2gRDgfvIl9k2BgtGW43e4Ea+btAm2ZfUYG2ZcFU1W/8kUrPrI= X-Gm-Gg: AYBFou0L4xVRysNwCwNIV1VB6TUqp4WRry6x+CdZknZmHb48my5v4yDD7lJPMilojWa IJKimIIiIWpf29NjxI6TA74FPwhg7CLnbwdWxzc1a70v71a50Z6pTre+kNHY18CVr80fXV5XxVh FnHnPdr+HscODCSgeN60vqjevwop/cMgqiviuznGDzkxRM6zLFsCi1ChOzk660K0UoirX9h2crt qL8M0kgO3fBsWEZgJS6Jus49F4Go9yF3iUAsWsq2KOqKNXTXrB9WZE0tO6HpTuBdKLY0r5DEXiH o83o5UlKlmUE+/cfVcYpk7n7fZsq+EqLrlFe3qW1U8j++CtKgxo31eOC5e9i9NEBZ7CcXDgUI/g tpKmViiS6PCK0QFX+Xowo7RdsP9pAUPKz+rG8WGaqhQ0oqR0XYPe8KX/3CaHhVmd2caicDN1pg6 i+MZGAvABAUn1JHuK80n2nY4kp/lkVA1RH8O921r+gghFjuskUk+2lhhODrvT/wOuSVa8sF7k8F Qn9d2HkIj+/ZxgyWCfzjrCAWS1FQOjxny5Al43P7WeimH8ji2nYF6Qc X-Received: by 2002:a05:622a:393:b0:530:a441:c52d with SMTP id d75a77b69052e-530c876307fmr64046411cf.46.1789135792869; Fri, 11 Sep 2026 07:09:52 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:41 +0000 Subject: [PATCH RFC v2 03/15] rcu-tasks: Hold trampoline nesting across irq-exit preemption in trampoline text MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-3-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=9197; i=josef@toxicpanda.com; h=from:subject:message-id; bh=CNudSyp9EGSFLIPrtB5sqvKMksuv/pGd9p7EfdyuASs=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QDP3NG/b7q989dUqjalqec1jdmkHZIx2wGCVq3J1Gir+zOd7a4FhOwI8Us9/xLubjdPD5eVSQFN Ltn5AyJDBWwY= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-42698a/1789135794-182F49EA-ECC8F3A6/0/0 X-purgate-type: clean X-purgate-size: 9199 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135827691158500 A trampoline's own rcu_tramp_nesting increment and decrement live inside the trampoline, so there is a window of a few instructions on entry and exit where the count is zero while the CPU is executing trampoline text (or text on the way into one, such as a static ftrace stub holding a direct-call target). In that window the task has not called out, so it can only be preempted from an interrupt, and the interrupted instruction pointer identifies where it is. Add rcu_tasks_ip_in_trampoline(), which treats any IP outside core kernel and module text as potentially Tasks-RCU-protected (ftrace trampolines, BPF images and programs, kprobe slots are all dynamically allocated text; is_ftrace_trampoline() and friends are deliberately not used because text being torn down may already be unregistered from them while a task still stands on it), plus a __weak arch_rcu_tasks_ip_in_trampoline() for core text an architecture needs to flag. On irq-exit preemption, if the IP matches, hold the count elevated across preempt_schedule_irq(). Introduce ARCH_HAS_RCU_TASKS_PREEMPT_QS / RCU_TASKS_PREEMPT_QS to gate this; no architecture selects it yet, so the check compiles away and there is no functional change. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/rcupdate.h | 17 +++++++++++++++++ kernel/entry/common.c | 23 ++++++++++++++++++++++- kernel/rcu/Kconfig | 10 ++++++++++ kernel/rcu/tasks.h | 38 ++++++++++++++++++++++++++++++++++++++ kernel/rcu/update.c | 2 ++ 5 files changed, 89 insertions(+), 1 deletion(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index b5c666c82479..0a408e36ea15 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -173,6 +173,9 @@ static inline void rcu_nocb_flush_deferred_wakeup(void)= { } =20 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ =20 +/* Arch hook for rcu_tasks_ip_in_trampoline(); see kernel/rcu/tasks.h. */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); + /* * Note a quasi-voluntary context switch for RCU-tasks's benefit. * This is a macro rather than an inline function to avoid #include hell. @@ -189,6 +192,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(void= ) { } * or was called from, such text and an involuntary context switch must no= t be * treated as a Tasks RCU quiescent state. * + * The increment and decrement themselves live inside the trampoline, so t= here + * is a window of a few instructions at entry (before the increment) and e= xit + * (after the decrement) where the count is zero but the CPU is executing + * trampoline text, or text on the way into one (a static ftrace stub or a + * return thunk holding the trampoline's address). In that window the task + * cannot be preempted synchronously, only from an interrupt, so the irq-e= xit + * preemption path covers it by checking regs->ip with + * rcu_tasks_ip_in_trampoline() and holding the count elevated across + * preempt_schedule_irq() when it matches. + * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. */ @@ -211,6 +224,8 @@ static __always_inline void rcu_tasks_trampoline_assert= _none(void) WARN_ON_ONCE(current->rcu_tramp_nesting); } =20 +bool rcu_tasks_ip_in_trampoline(unsigned long ip); + # define rcu_tasks_classic_qs(t, preempt) \ do { \ if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ @@ -226,6 +241,7 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } +static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } # endif =20 #define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) @@ -245,6 +261,7 @@ void exit_tasks_rcu_finish(void); static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } +static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } #define call_rcu_tasks call_rcu #define synchronize_rcu_tasks synchronize_rcu static inline void exit_tasks_rcu_start(void) { } diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e4acd50bd81a..cd3feaca6420 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -134,6 +134,27 @@ static inline bool arch_irqentry_exit_need_resched(voi= d); static inline bool arch_irqentry_exit_need_resched(void) { return true; } #endif =20 +/* + * Preempt the interrupted kernel context. If the interrupt landed in text + * that may be a Tasks-RCU-protected trampoline (see + * rcu_tasks_trampoline_enter()), hold current->rcu_tramp_nesting elevated + * across the context switch so that it is not mistaken for a Tasks RCU + * quiescent state. This closes the few-instruction windows at trampoline + * entry/exit where the trampoline's own increment has not yet run or its + * decrement already has. + */ +static void irqentry_preempt(struct pt_regs *regs) +{ + bool in_tramp =3D IS_ENABLED(CONFIG_RCU_TASKS_PREEMPT_QS) && + rcu_tasks_ip_in_trampoline(instruction_pointer(regs)); + + if (in_tramp) + rcu_tasks_trampoline_enter(); + preempt_schedule_irq(); + if (in_tramp) + rcu_tasks_trampoline_exit(); +} + void raw_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!preempt_count()) { @@ -142,7 +163,7 @@ void raw_irqentry_exit_cond_resched(struct pt_regs *reg= s) if (IS_ENABLED(CONFIG_DEBUG_ENTRY)) WARN_ON_ONCE(!on_thread_stack()); if (need_resched() && arch_irqentry_exit_need_resched()) - preempt_schedule_irq(); + irqentry_preempt(regs); } } #ifdef CONFIG_PREEMPT_DYNAMIC diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 332df7a7a634..999f8228a13d 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -107,6 +107,16 @@ config TASKS_RCU default NEED_TASKS_RCU && PREEMPTION select IRQ_WORK =20 +# Selected by architectures whose ftrace, BPF and kprobe trampolines maint= ain +# current->rcu_tramp_nesting and which use the generic irqentry code, so t= hat +# a preemption outside any trampoline can be treated as a Tasks RCU +# quiescent state. See rcu_tasks_trampoline_enter(). +config ARCH_HAS_RCU_TASKS_PREEMPT_QS + bool + +config RCU_TASKS_PREEMPT_QS + def_bool TASKS_RCU && ARCH_HAS_RCU_TASKS_PREEMPT_QS && GENERIC_IRQ_ENTRY + config FORCE_TASKS_RUDE_RCU bool "Force selection of Tasks Rude RCU" depends on RCU_EXPERT diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1662ba18bf34..a801ec4a951b 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1089,6 +1089,44 @@ static void rcu_tasks_postscan(struct list_head *hop) timer_delete_sync(&tasks_rcu_exit_stall_timer); } =20 +/* + * Architectures selecting ARCH_HAS_RCU_TASKS_PREEMPT_QS override this to = flag + * core kernel text that must be treated like a trampoline, e.g. static ft= race + * entry stubs and return thunks that run with a trampoline address in han= d. + */ +bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + return false; +} + +/** + * rcu_tasks_ip_in_trampoline - Could a task interrupted at @ip be a Tasks= RCU reader? + * @ip: interrupted instruction pointer + * + * Called from the irq-exit preemption path with interrupts disabled, to d= ecide + * whether the imminent preemption may be reported as a Tasks RCU quiescent + * state when current->rcu_tramp_nesting is zero. Returns true, meaning "= do + * not report", when @ip is: + * + * - outside static kernel and module text, i.e. possibly in an ftrace + * trampoline, BPF trampoline image or program, kprobe insn/optinsn slo= t or + * other dynamically allocated text whose lifetime Tasks RCU guards. T= his + * deliberately does not consult is_ftrace_trampoline() and friends: te= xt + * being torn down may already be unregistered there while a task still + * stands on it; + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(). + * + * A false positive only defers the quiescent state to the task's next + * context switch. + */ +bool rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + if (core_kernel_text(ip)) + return arch_rcu_tasks_ip_in_trampoline(ip); + return !is_module_text_address(ip); +} +NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); + /* See if tasks are still holding out, complain if so. */ static void check_holdout_task(struct task_struct *t, bool needreport, bool *firstreport) diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index b62735a67884..23be7e97c3b5 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -41,6 +41,8 @@ #include #include #include +#include +#include #include #include #include --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135821; cv=none; d=zohomail.com; s=zohoarc; b=Ji+XVjJOJL83dt3zGpz8XB5T9vzXu2jqDztqjVLhdmlXS7tKDLwxp4exfhbN50grIOxH7XCJ27fg57yV19Wt29nNYIqdXw9u7xkjGg4+4HtbpZvveO/grOJQ2NWt/CiX5TtHimZ3Rp4MB3p3DlvLCnwIZCPH+ZPASqkZkH1JxvQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135821; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=5run1FbtWhLPrIbIpcsdoDpDcAIfw1LixKxJKXXjMZI=; b=fl2D40GT1ExJOmmU2xiyYLDqLTzNBPn1vuExuSaLzguK3BnkqAlfafZ4SB7suTqNgrx35jt7YJfo4+TRq9OSLqVUoFXnX/3I+DqyrBOT+SzteJkkCK2xQBiI7oYfiGLXHOewA9ha+2pkjAvdvBEHa+laNwxEAUOQKPx30TtVcUY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135821265773.6274618713726; Fri, 11 Sep 2026 07:10:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416857.1645801 (Exim 4.92) (envelope-from ) id 1x51x0-0005ws-TH; Fri, 11 Sep 2026 14:09:58 +0000 Received: by outflank-mailman (output) from mailman id 1416857.1645801; Fri, 11 Sep 2026 14:09:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51x0-0005wl-Pl; Fri, 11 Sep 2026 14:09:58 +0000 Received: by outflank-mailman (input) for mailman id 1416857; Fri, 11 Sep 2026 14:09:57 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51wz-0005wJ-Q3 for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:09:57 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51wz-009TfM-74 for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:09:57 +0200 Received: from [10.42.69.9] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40b98-bab6-0a2a0a5309dd-0a2a4509a768-46 for ; Fri, 11 Sep 2026 16:09:57 +0200 Received: from [209.85.128.179] (helo=mail-yw1-f179.google.com) by tlsNG-bad1c0.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bb4-be1a-0a2a45090019-d15580b3d16e-3 for ; Fri, 11 Sep 2026 16:09:57 +0200 Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-8664769db58so7635247b3.3 for ; Fri, 11 Sep 2026 07:09:56 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f4d0700sm22061896d6.38.2026.09.11.07.09.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:53 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135795; x=1789740595; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5run1FbtWhLPrIbIpcsdoDpDcAIfw1LixKxJKXXjMZI=; b=jc1RD6ctevXEPgn+sQIvLqY7WwEgJkddRw6MasTL4Xn9a5+TFC16wWwWMTnuyTrxOo gh4aCNpDC10bh0OWzBsA3aaO6iuSllQncokB6ZlhBYSg/TV/n3sk62NPPI0K+0RRwJ3t a+7XBZFHNz2hebuHzIS6jY7MuzS20+H7uE1TPmaW1TTj8h6MyTL6NKd4t63FLksP/4DS PrAVafVIWZxf41WWC37cs9UaZUbJTNYcYRrnKdI7GI4kafjAdzmaxe2EzANM7M+fUzlM fz4Hs258ubG5K8GtRLrB94G5bF7j98DcyohhTmhQeFGc5ZTOSPFIB5zr3fq3EymfT5Cg qs/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135795; x=1789740595; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=5run1FbtWhLPrIbIpcsdoDpDcAIfw1LixKxJKXXjMZI=; b=Qb8PPgpBCePXjJ39MoCoHPNUm8DM+5dTe21DwO/KoEMH6LzV1j1xeUkgCY/DbMIqnx kWAk2KXf9gXg2eysWeenhw8DA/VOHxJQqrJ1qbMI+82k6jyZx5lRa4i1kcBb0QbqL8cZ ap1fcCV7oOYvMHjuRwu1UjBDW5ZFMGK4Tf4Hlu3ANkgco4m9WD23f1PtnHkiWAGpWFae WmIFZAIXXktjiNJOAc4sNT3p+K+w6C80NAIV78JU5Uf4FpWaHl4/ctgVSuSbY4wVW4zl A+LxhpPwXjnw4MHdjuznq4rYZWzac4mnCKm7iMCQOH34KcE7T7Xj4qTZR4Zvk8vzfJyK jLNQ== X-Forwarded-Encrypted: i=1; AKwUvBzAHPG39eyYt2GYBdHHVC8NgoSk2iZnhyOcjIeW9YtZIyefNseZ7aFCtDHqaavJmLckLwRTS0GIPrg=@lists.xenproject.org X-Gm-Message-State: AFuF++kZS4YupNoVo5gY8ICfgO55+2Oc6mqhDDpO4vHu35Aj3FyioLh5 oCEa16gZj7hZ42hlhICAJuQftwyWNn9wZ1PBC1wkywRC/ZgNyrizvWg/wFYbcayul0k= X-Gm-Gg: AYBFou13e50IkhAYgclDyCIpfBoiYyf5ZJhndj7mirPnAq01sXkw4USzuD8R8XiJSWd bMsFB8Xr/53UmfTDM3by1cDW7+4JnjV0kJWkpB5hy5BcTCAk5WCPphgDqezqNxCjmAifS8L9ta5 Zv8dtUdq3tH79hIwjrz51VeE6xJX+cXwrpNw4NL7l+yhntNd5WMFRHIxMwyG3gKNDWHMkHrGrzo Hi7WuV7LflmwNiMD21PnGO58PXx51x/72zliZnSsqPSzlP7PMNvrar0O3MN9VQp865/WbmNwWj+ /n8K9SG2Wzx53csnnzmec/BDsCkU7ZBDeMOuWvGN14vO3ggOp+B/qaMCCu1xTJ8haiNFB8bkGj3 bMRdvD3kFN23eix/r6IueoF2S9ahY/RV3v5H/cc3HuHzNfU+/B/K/ZmtjGyc+5/1QJbQ7viV4lM CBRUipdafQyB1t4idoSlqUtWzNXPpjhx2qwG3V7f+pIXmjjWXML6WoKnEcZGLZNwLxHzbeEutJU auwv6ViM/sEwWkcGuIGcRhZsPZsqYCaXlyduzWUJfH8O/f65OvyrMBA X-Received: by 2002:a05:690c:10c:b0:873:5c0f:27f with SMTP id 00721157ae682-884b183fa8fmr15728927b3.41.1789135795285; Fri, 11 Sep 2026 07:09:55 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:42 +0000 Subject: [PATCH RFC v2 04/15] kprobes: Let Tasks RCU recognise tasks preempted in an optprobe jump window MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-4-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=12385; i=josef@toxicpanda.com; h=from:subject:message-id; bh=zaxzL0DRdr6YBFEcTSGtZpzKzuVZf+tDLqNK8lLQ3vQ=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QDkHo+7Zxxr001o9pThi3R9UNNh0hJGGQszxQdwGKWyIfzZokuP81SU5bPZjsOb84RB2Rvd/rcr ztRWEWANpTgE= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-bad1c0/1789135797-BF4D3034-B7F0E660/0/0 X-purgate-type: clean X-purgate-size: 12387 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135823913158500 kprobe_optimizer() is the one synchronize_rcu_tasks() user that is not about trampoline text: it waits for tasks that were preempted on an instruction boundary inside the bytes it is about to overwrite with the optimized jump, so that none of them resumes into the middle of the new instruction. Such a task sits in ordinary kernel or module text with rcu_tramp_nesting =3D=3D 0, and can only have got there via an irq-exit preemption. Add kprobe_in_optimized_region(), a lockless and conservative form of get_optimized_kprobe() that reports whether any registered kprobe lies within MAX_OPTIMIZED_LENGTH before the given address regardless of its optimization state. The hash walk is only done while kprobe_optimizer() is actually inside its synchronize_rcu_tasks(), tracked by a flag it sets around the call; otherwise the check is a single load. The kprobe hash is RCU-protected and every free path waits for a grace period after unhashing, so the lockless walk is safe from any context with preemption disabled. Unlike trampoline text, which a task can only be interrupted in while the trampoline exists, these bytes are ordinary text a task may have been parked in since before the kprobe was registered, and the optimizer may start waiting while that task is already switched out. So the check cannot be made once at preemption time the way the trampoline cases are: have irqentry_preempt() record the interrupted IP in current->rcu_tasks_irq_ip for the duration of the preemption, and add rcu_tasks_irq_ip_holds() to test it, to be evaluated at every quiescent-state decision once preemption becomes a quiescent state -- each pass through __schedule() in preempt_schedule_irq()'s loop as well as any remote check. A task switched out synchronously cannot have a resume point inside such a window (a call there returns beyond it), so only the irq-exit IP needs checking, and preempt_schedule_irq() cannot nest, so one slot per task suffices. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/kprobes.h | 8 +++++++- include/linux/rcupdate.h | 17 +++++++++++++++++ include/linux/sched.h | 1 + kernel/entry/common.c | 13 +++++++++++-- kernel/fork.c | 1 + kernel/kprobes.c | 46 ++++++++++++++++++++++++++++++++++++++++++++= ++ kernel/rcu/tasks.h | 23 +++++++++++++++++++++++ 7 files changed, 106 insertions(+), 3 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index e6de7ae55bda..74cc48c04417 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long = addr) } #endif /* !CONFIG_KPROBES */ =20 -#ifndef CONFIG_OPTPROBES +#ifdef CONFIG_OPTPROBES +bool kprobe_in_optimized_region(unsigned long addr); +#else /* !CONFIG_OPTPROBES */ static inline bool is_kprobe_optinsn_slot(unsigned long addr) { return false; } +static inline bool kprobe_in_optimized_region(unsigned long addr) +{ + return false; +} #endif /* !CONFIG_OPTPROBES */ =20 #ifdef CONFIG_KRETPROBES diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 0a408e36ea15..4cfe096d624f 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -202,6 +202,14 @@ bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); * rcu_tasks_ip_in_trampoline() and holding the count elevated across * preempt_schedule_irq() when it matches. * + * The one non-trampoline user, kprobe jump optimization, waits for tasks + * preempted inside ordinary instruction bytes it is about to overwrite. A + * task can be parked there from before the kprobe even existed, so that + * cannot be decided once at preemption time: irqentry_preempt() records t= he + * interrupted IP in current->rcu_tasks_irq_ip for the duration of the + * preemption and rcu_tasks_irq_ip_holds() checks it at every quiescent-st= ate + * decision, locally and from the grace-period kthread. + * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. */ @@ -225,6 +233,13 @@ static __always_inline void rcu_tasks_trampoline_asser= t_none(void) } =20 bool rcu_tasks_ip_in_trampoline(unsigned long ip); +bool rcu_tasks_irq_ip_holds(struct task_struct *t); + +/* Record where current is being irq-preempted; 0 once it has resumed. */ +static __always_inline void rcu_tasks_note_irq_ip(unsigned long ip) +{ + WRITE_ONCE(current->rcu_tasks_irq_ip, ip); +} =20 # define rcu_tasks_classic_qs(t, preempt) \ do { \ @@ -242,6 +257,7 @@ static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } +static inline void rcu_tasks_note_irq_ip(unsigned long ip) { } # endif =20 #define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) @@ -262,6 +278,7 @@ static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } +static inline void rcu_tasks_note_irq_ip(unsigned long ip) { } #define call_rcu_tasks call_rcu #define synchronize_rcu_tasks synchronize_rcu static inline void exit_tasks_rcu_start(void) { } diff --git a/include/linux/sched.h b/include/linux/sched.h index d2e7b1b3c9d2..7f0bdc81fba3 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -957,6 +957,7 @@ struct task_struct { u8 rcu_tasks_holdout; u8 rcu_tasks_idx; int rcu_tramp_nesting; + unsigned long rcu_tasks_irq_ip; int rcu_tasks_idle_cpu; struct list_head rcu_tasks_holdout_list; int rcu_tasks_exit_cpu; diff --git a/kernel/entry/common.c b/kernel/entry/common.c index cd3feaca6420..b372f2670d4f 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -141,16 +141,25 @@ static inline bool arch_irqentry_exit_need_resched(vo= id) { return true; } * across the context switch so that it is not mistaken for a Tasks RCU * quiescent state. This closes the few-instruction windows at trampoline * entry/exit where the trampoline's own increment has not yet run or its - * decrement already has. + * decrement already has. The interrupted IP is also recorded for the + * duration, for conditions that must be re-evaluated at each quiescent-st= ate + * decision rather than once here (see rcu_tasks_irq_ip_holds()); nested + * irq-exit preemption cannot happen inside preempt_schedule_irq(), so one + * slot per task is enough. */ static void irqentry_preempt(struct pt_regs *regs) { + unsigned long ip =3D instruction_pointer(regs); bool in_tramp =3D IS_ENABLED(CONFIG_RCU_TASKS_PREEMPT_QS) && - rcu_tasks_ip_in_trampoline(instruction_pointer(regs)); + rcu_tasks_ip_in_trampoline(ip); =20 if (in_tramp) rcu_tasks_trampoline_enter(); + if (IS_ENABLED(CONFIG_RCU_TASKS_PREEMPT_QS)) + rcu_tasks_note_irq_ip(ip); preempt_schedule_irq(); + if (IS_ENABLED(CONFIG_RCU_TASKS_PREEMPT_QS)) + rcu_tasks_note_irq_ip(0); if (in_tramp) rcu_tasks_trampoline_exit(); } diff --git a/kernel/fork.c b/kernel/fork.c index cfe3a8e53fbd..1277603bc472 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1870,6 +1870,7 @@ static inline void rcu_copy_process(struct task_struc= t *p) #ifdef CONFIG_TASKS_RCU p->rcu_tasks_holdout =3D false; p->rcu_tramp_nesting =3D 0; + p->rcu_tasks_irq_ip =3D 0; INIT_LIST_HEAD(&p->rcu_tasks_holdout_list); p->rcu_tasks_idle_cpu =3D -1; INIT_LIST_HEAD(&p->rcu_tasks_exit_list); diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 6337da5cab9e..cf2ea278fdf5 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opco= de_t *addr) return NULL; } =20 +/* + * True while kprobe_optimizer() is waiting for its Tasks RCU grace period. + * Only in that window can a preemption inside an optprobe's jump region + * matter to it, so kprobe_in_optimized_region() does no work otherwise. + */ +static bool kprobe_optimizer_waiting; + +/** + * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimiz= ed + * kprobe replaces? + * @addr: kernel text address, typically an interrupted instruction pointer + * + * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks = that + * were preempted on an instruction boundary inside the region about to be + * overwritten by the optimized jump; such a task must not report a Tasks = RCU + * quiescent state when it is preempted (see rcu_tasks_ip_in_trampoline()). + * This is the lockless, conservative form of get_optimized_kprobe(): it d= oes + * not care whether the kprobe found is, or ever will be, optimized. May = be + * called from any context with preemption disabled; the kprobe hash is + * RCU-protected and every free path waits for a grace period after unhash= ing. + * + * The hash walk only runs while the optimizer is actually waiting. A + * preemption that does not observe kprobe_optimizer_waiting predates the + * grace period (its leading synchronize_rcu() publishes the store to every + * interrupts-disabled reader before any task is sampled as a holdout); su= ch a + * task is then an ordinary preempted holdout, and the jump is not written + * until it has run again and left the region. + */ +bool kprobe_in_optimized_region(unsigned long addr) +{ + int i; + + if (!READ_ONCE(kprobe_optimizer_waiting)) + return false; + + for (i =3D 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + if (get_kprobe((kprobe_opcode_t *)addr - i)) + return true; + return false; +} +NOKPROBE_SYMBOL(kprobe_in_optimized_region); + /* Optimization staging list, protected by 'kprobe_mutex' */ static LIST_HEAD(optimizing_list); static LIST_HEAD(unoptimizing_list); @@ -644,8 +686,12 @@ static void kprobe_optimizer(void) * to 2nd-Nth byte of jump instruction. This wait is for avoiding it. * Note that on non-preemptive kernel, this is transparently converted * to synchronoze_sched() to wait for all interrupts to have completed. + * kprobe_optimizer_waiting lets Tasks RCU recognise tasks preempted + * in such a region while we wait, see kprobe_in_optimized_region(). */ + WRITE_ONCE(kprobe_optimizer_waiting, true); synchronize_rcu_tasks(); + WRITE_ONCE(kprobe_optimizer_waiting, false); =20 /* Step 3: Optimize kprobes after quiesence period */ do_optimize_kprobes(); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index a801ec4a951b..0e46d8fe4d8e 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1127,6 +1127,29 @@ bool rcu_tasks_ip_in_trampoline(unsigned long ip) } NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); =20 +/** + * rcu_tasks_irq_ip_holds - Is @t irq-preempted somewhere that must hold o= ff Tasks RCU? + * @t: a task inside preempt_schedule_irq() (t->rcu_tasks_irq_ip !=3D 0), = or not + * + * Unlike trampoline text, which a task can only be interrupted in while t= he + * trampoline exists, the bytes kprobe_optimizer() is about to overwrite w= ith a + * jump are ordinary text a task may have been parked in since before the + * kprobe was registered, and the optimizer may start waiting while the ta= sk is + * already switched out. So this is evaluated against the IP recorded by + * irqentry_preempt() at every quiescent-state decision -- each pass throu= gh + * __schedule() in preempt_schedule_irq()'s loop, and the grace-period + * kthread's scans -- rather than once at preemption time. A task switche= d out + * synchronously cannot have a resume point inside such a window (a call t= here + * returns beyond it), so only the irq-exit IP needs checking. + */ +bool rcu_tasks_irq_ip_holds(struct task_struct *t) +{ + unsigned long ip =3D READ_ONCE(t->rcu_tasks_irq_ip); + + return ip && kprobe_in_optimized_region(ip); +} +NOKPROBE_SYMBOL(rcu_tasks_irq_ip_holds); + /* See if tasks are still holding out, complain if so. */ static void check_holdout_task(struct task_struct *t, bool needreport, bool *firstreport) --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135825; cv=none; d=zohomail.com; s=zohoarc; b=m2erX0pu7xrfJKoln033SUwz/Qz+aU+/IvPrAWQtgdvbZ2QVvmFshR+BTM2W/YEZ4nBkbKWn57A53Unozh0+5VANvvDq+XNR9aXArGqKxudEk47uvf+hsDHEFeBFsmVCpNAwF6VdvAycBAm3jQymrgulJK+5q+/fmFv9gATKL2Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135825; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=HhDqsBRCDYPA+bw813nyWKwnWX3/qBDcJfc7aNSFHb97M7rC1bajcMW0Hj5HhqEPEM9Kr/aloDvlrFaBa5pFq2GURGpvWsMIdKFL8aY4lle/At7Iz7ZSmwXagXglKo9FS4ALJEWtvuFEei7AtAzWHyR8QktB+wisVENjgUFE/+A= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135825926436.1410061867962; Fri, 11 Sep 2026 07:10:25 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416858.1645810 (Exim 4.92) (envelope-from ) id 1x51x3-0006Br-9p; Fri, 11 Sep 2026 14:10:01 +0000 Received: by outflank-mailman (output) from mailman id 1416858.1645810; Fri, 11 Sep 2026 14:10:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51x3-0006Bi-61; Fri, 11 Sep 2026 14:10:01 +0000 Received: by outflank-mailman (input) for mailman id 1416858; Fri, 11 Sep 2026 14:10:00 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51x2-00069z-28 for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:10:00 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51x1-00E9V9-FE for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:09:59 +0200 Received: from [10.42.69.4] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40bb2-2eae-0a2a0a5409dd-0a2a450497e8-20 for ; Fri, 11 Sep 2026 16:09:59 +0200 Received: from [209.85.128.178] (helo=mail-yw1-f178.google.com) by tlsNG-ebf023.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bb6-b57f-0a2a45040019-d15580b2c9a0-3 for ; Fri, 11 Sep 2026 16:09:59 +0200 Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-86c0e4dd49cso9235347b3.3 for ; Fri, 11 Sep 2026 07:09:58 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e8047a4dsm255725685a.23.2026.09.11.07.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:56 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135798; x=1789740598; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=NiE7GNK5Xk8hFJegGxEnIGYK9q7Gp54jBAMQnpFvn5tlf0pr4bh/q5kTIwv2dX54gN iYUGX2k3NxDC09CYG478fRCWm0zxLdSHnlUzCHXqnQuUJDFkajKBIDyH+WZqxBTPuIvm e6XJsFYOCETuU+DUKHhva7e+OFb2whbU9brY/H2FlGFQ1/9cPkf08EQV3IhtuVdcgNIf c3F/OX5QXcdlDegna68S3DUoD3DtujQrAPUU13ZfAOaA79ZcuscVuFOJ5f8aaIgc/R+G dJdWukyO/pYMhlMxSMkRJiMTZjPs7r4p3edv9W2/eB70UrckNicUwNcJ6heVFqJtoTGt e5UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135798; x=1789740598; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=HBaaLJS5G4VAf23aqx/A+Rb8K2L5rXzsLluHzAbGgIe6EPszG5N1U5l7SSK+cIVV9a pxXpCWNOz5W3Qze27z0UThcLlfvcsbHPBl8fSE2xI7clm1H33p9NL/MXKFHTSecNBr+4 R5me2dMPcfJ+QqScIsFNU7odJcyQ7XogXqiCgQT55hBK1b6cRvVbhhykN1P1v+WO8VrP xDb5TUTjZ6s+f0qb8Xlk9R6mOophuGacQqesFzWwZmRuYUcN+dOy0pKlut+mkcCeLApN Efxq9kY6PYDjAhJG8aBGcZ22fQeBQ2Am3L3mAr1hnwDklXe+J+Qen4tQpUy6kXKK/3uZ sVFA== X-Forwarded-Encrypted: i=1; AKwUvBywBBhhciAi3Bif872zNSZwKIGvB97r1I7/TWSWgFvPyifKMKSw76JBcp7Jo+PVDfaJLKvGitMNs50=@lists.xenproject.org X-Gm-Message-State: AFuF++kiNOV7fNu7BdPAYrrz4OqNX3v5xOO/BE+zRIf0NrW1ebgDJ7ko eh3oDtdel9tcz0i55DIiE0jJYRfkmHOWPo5aWTq4jZjxH1LL2PW/TqFpNcU78cbw9Ko= X-Gm-Gg: AYBFou2/X3RlO0N4k4HhMGC7t6iSnPxL0pFsVO4ZtHYof8qeUEv7DCsWHe4hig34ifg rHqwNJBSh3wQfOHGT3pHEoMrNzMT4eSVc6m0Q7j3x8XTy9oh36S91yasP6Ub0/097c/cp8titbo CkxFzf4CF7GnvCw5OkMUinAJ0ZUYkcvGpoYbvNcLtrxVAEnY/7yjDei+C5Xp6fKFm2jkExMQ5nA JDdDmjxNbu5yJJHqJJgoU4yCs8xMDgdTBf3mhynaQZTwJCi+gTySfaVyGmNZEsVAH7hfXmvQWSQ A2vTApnmjzImxE25DsLb06u40z+mZ9z5Z+7s/LcegxBECP3H6gJv+4VE6zkrwbszO7fcCwHHcEw Ueuzd0bhJ6V19qHbTRTvVuE8XtFwWpc/MJdZEJIcsJ/Q2NzeQJRwC9U+HvPMcvbKUWiHoYM3aSh Ln/7IdKxoeOCP4Pr3RlC70Xwtii6JY+3eLoiUNsZjrnqALjh2Rsh8zggPHicEkYqcVihMnweARh NsdgKSB1aeezig4KTRsQzUqPdQkEHtI7tNtTtPVa94D7o5LRcCjgiBm X-Received: by 2002:a05:690c:e64c:20b0:873:5c7b:c107 with SMTP id 00721157ae682-884b338f76bmr10673077b3.63.1789135797554; Fri, 11 Sep 2026 07:09:57 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:43 +0000 Subject: [PATCH RFC v2 05/15] ftrace: Mark modules hosting direct-call trampolines for Tasks RCU MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-5-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=7014; i=josef@toxicpanda.com; h=from:subject:message-id; bh=0t7KiskdL/z6tLq58DM0lyjoNIyrb4NbcoG2+xIJz7Y=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QIYpNGNhK8IIkgKHr1QxZn5VS4Cl+AINmPPfcsvZi4ZGk+7mTHv8oPmVPnKO/4ckp4yjA+b+rY+ zCDlRK/TVmQM= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-ebf023/1789135799-C36C2B50-0FAA8C3D/0/0 X-purgate-type: clean X-purgate-size: 7016 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135827677158500 An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Once preemption becomes a Tasks RCU quiescent state, such a trampoline must hold current->rcu_tramp_nesting across its call-out like the ftrace and BPF trampolines do, so document that in register_ftrace_direct(). That still leaves the few instructions before the increment and after the decrement. For BPF images those are in dynamically allocated text that rcu_tasks_ip_in_trampoline() already treats as protected, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag, set by every register/modify path when the direct address is module text, and have rcu_tasks_ip_in_trampoline() treat a task interrupted anywhere in such a module as a potential reader. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 23 +++++++++++++++++++++-- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..ea4727f53fab 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_ip_in_trampoline(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 0e46d8fe4d8e..1b9fe1bfa591 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1114,16 +1114,35 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigne= d long ip) * deliberately does not consult is_ftrace_trampoline() and friends: te= xt * being torn down may already be unregistered there while a task still * stands on it; - * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(). + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(); + * - in the text of a module that hosts an ftrace direct-call trampoline, + * which covers the instructions before that trampoline's increment and + * after its decrement (see ftrace_direct_mark_module()). * * A false positive only defers the quiescent state to the task's next * context switch. */ bool rcu_tasks_ip_in_trampoline(unsigned long ip) { + bool ret =3D true; + if (core_kernel_text(ip)) return arch_rcu_tasks_ip_in_trampoline(ip); - return !is_module_text_address(ip); + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod =3D __module_text_address(ip); + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + if (mod) + ret =3D READ_ONCE(mod->ftrace_direct_tramp); +#else + if (mod) + ret =3D false; +#endif + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..14f27b887231 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) ops->trampoline =3D 0; } =20 +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_ip_in_trampoline() recognises on its own = (see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must hold + * current->rcu_tramp_nesting across its call-out (see register_ftrace_dir= ect()); + * marking the owning module here covers the instructions before that incr= ement + * and after the decrement, where a task interrupted in the module's text = must + * not be treated as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed un= der + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#ifdef CONFIG_MODULES + struct module *mod; + + guard(rcu)(); + mod =3D __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task = is + * executing it or is preempted in something it called. On architectures = that + * select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU quiesc= ent + * state unless current->rcu_tramp_nesting is non-zero, so the trampoline = must + * increment it before calling out and decrement it before returning, as t= he + * ftrace and BPF trampolines do (see rcu_tasks_trampoline_enter() and + * samples/ftrace/ftrace-direct.h). The few instructions before the incre= ment + * and after the decrement are covered by the irq-exit IP check: automatic= ally + * for trampolines outside kernel and module text (e.g. BPF images), and v= ia + * ftrace_direct_mark_module() for trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) ops->flags |=3D MULTI_FLAGS; ops->trampoline =3D FTRACE_REGS_ADDR; ops->direct_call =3D addr; + ftrace_direct_mark_module(addr); =20 err =3D register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, = struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } =20 @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, = struct ftrace_hash *hash, b tmp =3D __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct =3D entry->direct; } } --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135821; cv=none; d=zohomail.com; s=zohoarc; b=av+o2XxasOrWhupxBJT1F1lBVfnJSSSXzylE6vgKAOLe2j5h4f60+21h28mYZtRriRkFlDGTrqx+MijhTGoxpo52jEByPe3uBKMbdmksImD/YEF/gWU2CO37r7qp/+tvzXaTK4TgKYcjis73MEKSFB4cy/wisB6z9+OC2OUhAAA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135821; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=ernFqVtV599i4uOPCDQSceYqFipgcM6HSwYHmr5J6ZiIqKLVkcGQYD2pDjxdusVG8Cyf60p+xGDmRe19gDIgYnnjnlhkKKO44edXTVf1CUNCb0kI+QCapDjFj/V6pQhkx/1m4XG9sXj9RqH5EWfo54pAAGIiJTCtaqxaanZSrxU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135821467363.5023834858206; Fri, 11 Sep 2026 07:10:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416859.1645819 (Exim 4.92) (envelope-from ) id 1x51x7-00072L-Gk; Fri, 11 Sep 2026 14:10:05 +0000 Received: by outflank-mailman (output) from mailman id 1416859.1645819; Fri, 11 Sep 2026 14:10:05 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51x7-00071p-DO; Fri, 11 Sep 2026 14:10:05 +0000 Received: by outflank-mailman (input) for mailman id 1416859; Fri, 11 Sep 2026 14:10:03 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51x5-0006Zz-5H for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:10:03 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51x4-00E9V9-IH for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:10:02 +0200 Received: from [10.42.69.4] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40bb2-2eae-0a2a0a5409dd-0a2a450497e8-36 for ; Fri, 11 Sep 2026 16:10:02 +0200 Received: from [209.85.222.179] (helo=mail-qk1-f179.google.com) by tlsNG-ebf023.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bb9-b57f-0a2a45040019-d155deb3d53d-3 for ; Fri, 11 Sep 2026 16:10:02 +0200 Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-9399798ca61so108279285a.1 for ; Fri, 11 Sep 2026 07:10:02 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939fd0ebaf6sm17627485a.35.2026.09.11.07.09.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:58 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135801; x=1789740601; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=mfj7AjBLWfzEl9nUZ0orOhFTHNS2xjO/JOc0e3l3vLXE+Kjmig9qmhP6KdYuKi8R3L 61MECn4zxk346aeXVdb6lpRm6pbIc8aN/DtLTrAlFNXF+zIcojkMjm9UPdUuTxUdGA/k ANkZTprAtM1q0GHOaMyFaWqhxntmJHhIUU2a5GNvzOSZizmRrOLa7BCyARunr2XE8IJJ jOeghTyczxvegjEHFZ0xovxk7qIEMffWZEYgt+ilADOCbRR7RFD+VZ7csEJJYP860ioL RFjg/oJvb4gwk5ymD2sFj9ZZaIJMeti8rgrvwPH2YCCE8tKtVRmS6S7zNZlsYO6hVFQS EUkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135801; x=1789740601; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=Ein14Q0IB13Mgxp92z7M16nAuI4ACam/pv+zZpLZrecLrnkX9Vir7kJAys3d82rX5q j/iOQ8wM+IyeVsGP0T/S9EVUIrSQKjPTOySbL3pXxApaQahxg2xNV4AHtkkQAtpJF/Ks w1MgsAQJIyAq5SpuzqK3vmeV2OZShle0Nx9yH0sZvLZNGDEoM0hhPDFOAAvVfiXhvMzR dvmP4/ooR1aRKDStIQiX6JzQtdYZZiwvGte18tzvY98n0TfOtt7GAzlXjb0kcKisT378 DyPaeOZFGNI4Qqr4hmZCo+vpoFZ2qtnPDlpbhXz6C6rwfyA6kYm8sxqBlEq7dNVoMuTn 06fw== X-Forwarded-Encrypted: i=1; AKwUvBxGQvJo8TbdG45i2yJsxG/25DY1AfGu7ZRDQ/EA3rCAtGMntZyhGuRAXllcLnvrQnKAn8Iuf9Jn0d4=@lists.xenproject.org X-Gm-Message-State: AFuF++l3dj7aI/vpwNRkC7wWvGofo069mOh5OKla8svnXPdvvM/PBrrF ppEBEMLRlK7lsEpvJ32VmJ+ARm4PSoXm14aaSxw/XLc5DK62hYd/l4pXehE5ZRjNpAg= X-Gm-Gg: AYBFou23Zh4yzcxYkMDMAQ9ojBHAl+b0Pum4ZuhLx+R/dnGjeOjvbfT+L83+l2GwaOO VmnPkvf7dSTPp7Wk3/h1c1xH/6npfNw2DiTVQvrxQEubvXQbNMP6zqT0T+i2mtxAF7jsdR23Rwd vhDHOaHVcQaJcI5oMpGEONfxjnlbCuQcmqNXft65tFA+2qY4wpq9C95mXJ7TqqEhwa3P7zPvRnh oeAPddQYQrWThueTqFqxVxcjrcq++JR/W8lF1CJ6+LZm6vZWFS82RcFD63EdHoEDOwxBlveeB3J loUUglMtMnErSzxnJkobna3JrGBjDUpHyPPHyKGU7aBKt75Ht6MCMZZ7CeI+MgEol+zAHvTIwzn +kD4rJuEXph1dQZhPks+K+hTSmL9q4m1HI/nS+dpTytv8qZniTFLcJHXZKWUWzPSrJDfnhRpTMC c0tlykXSKkjkc2HGYCCDug+kdMa6rmSka0GrclnjFtxaYa6oeydGg4ecptL+X2oe38daD8HmVon DstaQ5VOZ7puaPsOFLVi1wPbK8qJiY4W7PAQVBqPL2YhdIfMw2mXZPD X-Received: by 2002:a05:620a:2589:b0:939:112:d526 with SMTP id af79cd13be357-939ea07c29cmr585929585a.18.1789135800554; Fri, 11 Sep 2026 07:10:00 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:44 +0000 Subject: [PATCH RFC v2 06/15] x86/ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-6-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=7822; i=josef@toxicpanda.com; h=from:subject:message-id; bh=bmu7zPpxbnsCSHBhyK2WDk3M9VMJMCkgBrW5uszUBuU=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QHW3FDsl5CxgF20PG9LmVTMTg46RLyWS2XjYhSinsXHd+9ujqg/53BxxPRKz/SrjYV+XGJTF9Bc FIbELyLBq3wc= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-ebf023/1789135802-53CC7B50-55E21C92/0/0 X-purgate-type: clean X-purgate-size: 7824 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135823876158500 Bracket the call out to the ftrace_ops callback in ftrace_caller and ftrace_regs_caller with an increment/decrement of current->rcu_tramp_nesting. The instructions sit inside the region that create_trampoline() copies for per-ops dynamic trampolines, so those inherit them; the %rip-relative per-CPU reference to current_task is fixed up by text_poke_apply_relocation() like CALL_DEPTH_ACCOUNT's. %rdx is dead at both points (about to be loaded with the ops pointer on entry, restored by restore_mcount_regs on exit). Two pieces of core text still run with the count at zero while holding the address of a Tasks-RCU-protected trampoline they are about to enter: the static stubs themselves, whose direct-call tails keep a BPF trampoline address on the stack until the final RET, and, under CONFIG_MITIGATION_RETHUNK, the return thunk that RET expands to. Add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and linker symbols around .text..__x86.return_thunk and .text..__x86.rethunk_safe, and provide arch_rcu_tasks_ip_in_trampoline() covering [ftrace_caller, ftrace_static_tramp_end) and both thunk ranges so the irq-exit check treats a task interrupted there as still inside a trampoline. The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which x86 does not select until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/asm-offsets.c | 3 +++ arch/x86/kernel/ftrace.c | 37 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/ftrace_64.S | 43 +++++++++++++++++++++++++++++++++++++++= ++++ arch/x86/kernel/vmlinux.lds.S | 4 ++++ 4 files changed, 87 insertions(+) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 081816888f7a..4f3b1caa5a30 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -46,6 +46,9 @@ static void __used common(void) #ifdef CONFIG_STACKPROTECTOR OFFSET(TASK_stack_canary, task_struct, stack_canary); #endif +#ifdef CONFIG_TASKS_RCU + OFFSET(TASK_rcu_tramp_nesting, task_struct, rcu_tramp_nesting); +#endif =20 BLANK(); OFFSET(pbe_address, pbe, address); diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 17d6edfcb7e0..8f63cd4b543c 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -275,6 +275,43 @@ static inline void tramp_free(void *tramp) execmem_free(tramp); } =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +extern void ftrace_static_tramp_end(void); +extern char __return_thunk_start[], __return_thunk_end[]; +extern char __rethunk_safe_start[], __rethunk_safe_end[]; + +/* + * See rcu_tasks_ip_in_trampoline(). Some core kernel text behaves like a + * trampoline for Tasks RCU purposes because a task executing there with + * rcu_tramp_nesting =3D=3D 0 may still be about to enter a Tasks-RCU-prot= ected + * trampoline whose address it already holds: + * + * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_t= ramp + * stubs, which carry a direct-call target on the stack until their fin= al + * RET, and + * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUN= K, + * which run after leaving the stubs above and before landing in that + * target. + */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + if (ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end) + return true; +#ifdef CONFIG_MITIGATION_RETPOLINE + if (ip >=3D (unsigned long)__return_thunk_start && + ip < (unsigned long)__return_thunk_end) + return true; +#endif +#ifdef CONFIG_MITIGATION_SRSO + if (ip >=3D (unsigned long)__rethunk_safe_start && + ip < (unsigned long)__rethunk_safe_end) + return true; +#endif + return false; +} +#endif /* CONFIG_RCU_TASKS_PREEMPT_QS */ + /* Defined as markers to the end of the ftrace default trampolines */ extern void ftrace_regs_caller_end(void); extern void ftrace_caller_end(void); diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 62c1c93aa1c6..902472c41798 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -145,6 +146,27 @@ SYM_FUNC_END(ftrace_stub_graph) =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). These = live + * inside the region copied into dynamic trampolines; the %rip-relative pe= r-CPU + * reference is fixed up by text_poke_apply_relocation() in create_trampol= ine(). + * The increment must precede the function_trace_op load: between that loa= d and + * the call, the ops pointer in %rdx is protected only by Tasks RCU. + */ +.macro RCU_TASKS_TRAMP_ENTER reg:req +#ifdef CONFIG_TASKS_RCU + movq PER_CPU_VAR(current_task), \reg + incl TASK_rcu_tramp_nesting(\reg) +#endif +.endm + +.macro RCU_TASKS_TRAMP_EXIT reg:req +#ifdef CONFIG_TASKS_RCU + movq PER_CPU_VAR(current_task), \reg + decl TASK_rcu_tramp_nesting(\reg) +#endif +.endm + SYM_FUNC_START(__fentry__) ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT @@ -163,6 +185,8 @@ SYM_FUNC_START(ftrace_caller) leaq MCOUNT_REG_SIZE+8(%rsp), %rcx movq %rcx, RSP(%rsp) =20 + RCU_TASKS_TRAMP_ENTER %rdx + SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -181,6 +205,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + RCU_TASKS_TRAMP_EXIT %rdx + /* Handlers can change the RIP */ movq RIP(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -209,6 +235,8 @@ SYM_FUNC_START(ftrace_regs_caller) =20 CALL_DEPTH_ACCOUNT =20 + RCU_TASKS_TRAMP_ENTER %rdx + SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -246,6 +274,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + RCU_TASKS_TRAMP_EXIT %rdx + /* Copy flags back to SS, to restore them */ movq EFLAGS(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -328,6 +358,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp) RET SYM_FUNC_END(ftrace_stub_direct_tramp) =20 +/* + * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text = by + * rcu_tasks_ip_in_trampoline(): after RCU_TASKS_TRAMP_EXIT the stubs may + * still hold a direct-call target (a BPF trampoline) on the stack until t= he + * final RET, and that target's lifetime is guarded by Tasks RCU. With + * return thunks the RET itself runs elsewhere; arch_rcu_tasks_ip_in_tramp= oline() + * covers the thunk text too. + */ +SYM_CODE_START_NOALIGN(ftrace_static_tramp_end) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR +SYM_CODE_END(ftrace_static_tramp_end) + #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 2438b89a4620..e546283dc267 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -151,7 +151,9 @@ SECTIONS * definition. */ . =3D srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 2= 0); + __rethunk_safe_start =3D .; *(.text..__x86.rethunk_safe) + __rethunk_safe_end =3D .; #endif ALIGN_ENTRY_TEXT_END =20 @@ -162,7 +164,9 @@ SECTIONS SOFTIRQENTRY_TEXT #ifdef CONFIG_MITIGATION_RETPOLINE *(.text..__x86.indirect_thunk) + __return_thunk_start =3D .; *(.text..__x86.return_thunk) + __return_thunk_end =3D .; #endif STATIC_CALL_TEXT *(.gnu.warning) --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE8E03176EE for ; Fri, 11 Sep 2026 14:10:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135809; cv=none; b=FrpM1N8h1XBX6/farWd6X4e2RX/TYi+qnXovm4pcvE4dASDYiV9z6keW5NNUx8jwPxG9huJGRI5Gv9Jmz9NAXpxHUSKZi9Q3/RaFrjhqOQycKpWnFLRutLIgjxyMf1AHcQkdFDDbYuKKYDLfQKG+AGrVGsw41+alO2HG4XhbxaI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135809; c=relaxed/simple; bh=8cFDFeD2IXPN8Vlj1EymtW9Bi32PSZwsbXS468MRSTs=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=DBUzyeWtGIMF0q8z6+A3reymDi/pkuiiKiv8yO7nmaXyxvn4BSgCdE4SvbW4m+NrzkLRI/fbZC0wWha/0CU2k0cwGHd+T95umZSTpp+6E4rAGuwdtARbjumzhkh48txSG/0zXIVNSz+OmP/a9/pNaqKKcYrBh/GGqSb7+AmmWLQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=AO0u/AX8; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="AO0u/AX8" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-939e028b3aeso125593185a.0 for ; Fri, 11 Sep 2026 07:10:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135804; x=1789740604; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hbVlQmHCpb9zv+5BDFsE+SK081jwlXFhk4tZhnaWY5k=; b=AO0u/AX8T2KO9cUT21S8NZRAtESkc/8lg+v96HISVkTT7B10Mjmvm8BmfwPFrST/Ea eHAw7XB0udEm4icZnkQxCqHzVeES3CSJFA/jmnILW85HT4C+GozarSvEh+N3OPILfXNn Hb2DXQGzHE07x0W/wCsgPkopcf60quSzzFv7BjvxE9CQ29+58ttPSF+RleZKLtOAvJL5 xDjsow38dudkVjd+2EjMtOniOpMgAJ6gvOdCW6m+1L8UD8/XeCmjM9IQNUVeoLFPBUjI +2qLeWf9wL+LL1De3OC7dOk9Rek4M4/hZTsrTcsUl520tXWy37O4+V/SmPQZ9SGRmGsU WdAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135804; x=1789740604; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=hbVlQmHCpb9zv+5BDFsE+SK081jwlXFhk4tZhnaWY5k=; b=dPR5h8WPSjRhgACDuT8vDUPH4+XNAPPfOYLE8PmXNl3Zvfuz64CqRiMEFpyphF8uo2 u87LszvRttzgjWweCWj9V2O9zXWKA8Ys/q8ajrvhVgf/ycDHaN3wIY0J1jNpQKqJuuYV zkBZkgHGh4Sv2dq2vdPtPOnhlAtGD16HFduBreVvKwV2KI3flKaXcGHG/3FHf6W9aMk8 VftqlfVLcOOcYeTqXYTmhaBUFcO8kk/mjxFcEBMOktV7ECg1wUzLSoYB4aCw+ZnQ1ItZ NMUXjZIILkfdOM5rdXtaxKcaeoOqOSZM5gN9NlL5cNr1SgxqAnUXQvI/Aj3SByKCw3PO m18Q== X-Forwarded-Encrypted: i=1; AKwUvBwMkO2qE+Qw1rVG6KkU9kU9EXNIRxjLhMeLn0ptAuUK8I23/m5WppxTDcwrVovW4BvWblfjUf357Is4K00=@vger.kernel.org X-Gm-Message-State: AFuF++k4tDosrOXSmrL+DbAn9wr9+nIU1fWxqqCwKxuDE0hLfM/xOTKv WzFerZyfbiyb5Vvaa2yFYtNPfvm/PAjgQuaQ3fygMXSV/MXW8afINVA2kEaRhBIl2uA= X-Gm-Gg: AYBFou1avx1jprKL7SyHAxYvXFQqH/yg07JSFfZDfZmSxcHBoO1uuKs1ZBA6Qbmv/yw J5+yCOGn34mzzjEJMkzV17VGJ57/hXxAv/NAH8l57f0GroFXp/Of5fDPrh9amzIlcYFZaVvtPXP hZAbID/fIm54J3nv2A4hrevZRNKvRxDiSrvl0shLV4go+y0mJcDS1IRa6uUPAd9ARdYPlaTvnwd /V4EG2XGTo+27gVjLAQiL7LpU70v3PrvJnjbQMBXKpQf3ELok+EEl2KKSidXHDKaz54C0HdhHaO +sArAXX+Wlr/p879/Td5q+la6lkzdOMlKnoIN77UBppeKl2n0r6r1niPJC5+BPuFJr5J5kx4R9l 4L28BLIx25BZzMDTUy3fBcfIgCuGX//EoKmorcbC3jY6UHGDhB0Nosfqm279xckTelf88S8GS2U L8V4NxrN0prH9TxUlRXLI+5jRhe9qeia1pfiXosNX8l2hjKnrGICp7h6R+Ku3YZvGlEX+x/vJ5E vwz6VMBCXbrgS8ApKdYF7mUIkMYDB7SmMFnhblKSOUyb9YwATL3KCm5 X-Received: by 2002:a05:620a:198d:b0:939:3a81:326 with SMTP id af79cd13be357-939ea18c166mr526461185a.37.1789135803738; Fri, 11 Sep 2026 07:10:03 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939f1dd527asm125166385a.16.2026.09.11.07.10.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:01 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:45 +0000 Subject: [PATCH RFC v2 07/15] x86/kprobes: Maintain Tasks RCU trampoline nesting in the optprobe template Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-7-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=2636; i=josef@toxicpanda.com; h=from:subject:message-id; bh=8cFDFeD2IXPN8Vlj1EymtW9Bi32PSZwsbXS468MRSTs=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QE88Fo4D8KnMZviii3UcxtNsSOWitctfeeqTUeKrfDl5ze3tUkJnL+Us2LrW6ea29znHZAO8PVa SQS9QIpNv5Ao= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA The jump-optimized kprobe template calls optimized_callback() with preemption still enabled for its first few instructions, so bracket the call with an increment/decrement of current->rcu_tramp_nesting. The template lives in .rodata and is memcpy()d into each optinsn slot without relocation processing, so the per-CPU reference to current_task must be an absolute %gs: address (R_X86_64_32S, relocated for KASLR like any other) rather than %rip-relative. %rax has already been saved by SAVE_REGS_STRING and is dead after the call. The slot itself is dynamically allocated text, so the instructions before the increment and after the decrement are covered by the irq-exit IP check. 64-bit only; 32-bit x86 does not take part. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/kprobes/opt.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 3f8fea52619f..f722520bb989 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -31,6 +31,7 @@ #include #include #include +#include =20 #include "common.h" =20 @@ -101,6 +102,23 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr,= unsigned long val) *(unsigned long *)addr =3D val; } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). The + * template is memcpy()d into the slot without relocation processing, so t= he + * per-CPU reference must be absolute, not %rip-relative. + */ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_X86_64) +#define OPTPROBE_RCU_TASKS_ENTER \ + " movq %gs:current_task, %rax\n" \ + " incl " __stringify(TASK_rcu_tramp_nesting) "(%rax)\n" +#define OPTPROBE_RCU_TASKS_EXIT \ + " movq %gs:current_task, %rax\n" \ + " decl " __stringify(TASK_rcu_tramp_nesting) "(%rax)\n" +#else +#define OPTPROBE_RCU_TASKS_ENTER +#define OPTPROBE_RCU_TASKS_EXIT +#endif + asm ( ".pushsection .rodata\n" ".global optprobe_template_entry\n" @@ -114,6 +132,7 @@ asm ( "optprobe_template_clac:\n" ASM_NOP3 SAVE_REGS_STRING + OPTPROBE_RCU_TASKS_ENTER " movq %rsp, %rsi\n" ".global optprobe_template_val\n" "optprobe_template_val:\n" @@ -122,6 +141,7 @@ asm ( ".global optprobe_template_call\n" "optprobe_template_call:\n" ASM_NOP5 + OPTPROBE_RCU_TASKS_EXIT /* Copy 'regs->flags' into 'regs->ss'. */ " movq 18*8(%rsp), %rdx\n" " movq %rdx, 20*8(%rsp)\n" --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789135833; cv=none; d=zohomail.com; s=zohoarc; b=Vu0PUSMSv1SLy/34mhDkHSXegVOyrA/9BOMXoEoX7yUv/+yiCB1tOlsvO18JSuiN9v0G3adcMwDyvH5SS3NcY+uiTOt/eLF5NUOnb6o8oRmkHZcklqY0jU33X2oYfqup//A/OYCvuBpV/WsojgMczRvKa3w/b/KLIkjzWaIat+E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789135833; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=V+bQT2l6DSRsFjacMHs7M5XNLVkMvfpDksc21k7fi5yCI5HaOtQaFq4/GByDlN+5ucWLMJsVaq5uYVxU9waTUM1tFLsC4RvXThSneguXHv9O2KbYl5bQ0CdWld2P8W6Wkcp9K0YKnxh83bsKH9Q84+ywCV4VLje6g0YBSTnqRg8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789135833007437.7765499480414; Fri, 11 Sep 2026 07:10:33 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1416861.1645834 (Exim 4.92) (envelope-from ) id 1x51xE-0007z0-BC; Fri, 11 Sep 2026 14:10:12 +0000 Received: by outflank-mailman (output) from mailman id 1416861.1645834; Fri, 11 Sep 2026 14:10:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51xE-0007yW-1P; Fri, 11 Sep 2026 14:10:12 +0000 Received: by outflank-mailman (input) for mailman id 1416861; Fri, 11 Sep 2026 14:10:10 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x51xC-0007sY-Lf for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 14:10:10 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x51xC-00E9cU-2O for xen-devel@lists.xenproject.org; Fri, 11 Sep 2026 16:10:10 +0200 Received: from [10.42.69.8] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa40bc0-bab6-0a2a0a5309dd-0a2a4508af86-6 for ; Fri, 11 Sep 2026 16:10:10 +0200 Received: from [209.85.219.52] (helo=mail-qv1-f52.google.com) by tlsNG-c1860d.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa40bc1-f659-0a2a45080019-d155db34f044-3 for ; Fri, 11 Sep 2026 16:10:09 +0200 Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-910316a2fc9so11546576d6.2 for ; Fri, 11 Sep 2026 07:10:09 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e7f184d9sm254193785a.12.2026.09.11.07.10.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:05 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135808; x=1789740608; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=ZbHGPsWwzRok43ZcjJKavwsPNS6W0mLdK9348ZPWknyWWlld1xfZCspzjln3natrNC fLPC4iTskVhP4u303UmjS+g0Ip/p/VhH7SO2uyC/vNl/uM3oZuYvO5smkIN8jQ0BQNUU +L04CtvCiDJjyTDTRNrR18hJVI6xXOF7OSvu3RLpX0rpp83yDGrkle2BErDx8pXnVvZj W/nuPUeBb5BbtwiF+baCgXI4BqNs3bzacBuBGqZqMkrbqISQYKKXu+fVdrofH1dy/jVe In6ZuqGyNu9ihBM+XHQZ7vpyquj3dNRiF1xnAElzj6wrMigAVXg055zkKu6thr4ArAl8 9Z/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135808; x=1789740608; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=DYencswdLXs79jqpcedU1zonAw8jAn64Xd6zQlbtl6Xz0/SHVVI+tQl3hLqhuyQkDb E+aRazfDRKRrOeSMD2vzwUbwBoobg3DLZQyA9KeQ0acYNA0EVnoovyaVbh9A6gd4VplC 4kKX85VzV2l/6XD7TF4VDFZn+VdRHNywHRpIt94eQbCCugCPJy4ZDqy9Xqzano8Oopnr BCwSCZ1hs3C2Sy5/ofUlGncWH9AJNXkOGsRq3RShB3BposrFJC+CWDFcNguXnnSUE3gC AOeU2sI+8WPlAcHHmzVaBzwqRK0Vx/IGPKcluW7iqjY+uoFBWZE85YEhFvmih2aZ2ZMY 5Abw== X-Forwarded-Encrypted: i=1; AKwUvBxDFulrSOLP+mobzBfJjiUa35QNVvt/1/g2LcPqiboC4BPgK8oot9XqfTeGdLvyoRG0oZ3mZHZLefM=@lists.xenproject.org X-Gm-Message-State: AFuF++kwY+LsWxZk/DcFIZCSGQ1cE+dbnraXA/jRBSTlFkNRlADKYbYF uYfugZDObyQrzSeyCFu7mi6oefjAHR2s2DSEdeaaANPNp0YXszo+5KnEWf7ELabFFNk= X-Gm-Gg: AYBFou2zm2M8ndv9oDjqiJ87taADzhsT0zlBzBRt4Hw+XnWX+snZJKvv21pZYQIirTK kGSi9GnyT5i3kNlofAHxpADALiUfXrxR5baWX1VRIt3FoUHcopBwTWbOnc/8EilBgAZk6nj/9Ri KJobqeuGchU87v5BA2BXQggAn8Gi/8tImrbDvDWHF0Mp7iSx4ZwV+72xS96m7uySIpL/o1AI/Ux DhtDiX1RzKu7mx3jiGBMgV2T5z/xbXg3zyDuOZnxcNtRUCxBPI2zqxoziGY62nI89LqnQTHXtof HiWzt1TFcQsmTJ41QzOpJgZymyHdk78XNqJhwuOT49JB/HmTB+bxgp8JCxzU3tTy7xUc0s2E/PU qOzkK7Oomoy3G8vjfzS3ciIfETlUY3AAxeT86FjVzlQRk6llgD0meba8Ze9nd+gpAHF6hE1Kcb9 CoMlXxgATEzyp4ng52V2LTV/+1SMzQaZTTA5tR6N2LTD6J/0YEzme73A+D1nPDu57vr+jAKqi4/ J5k0uMSlPktP94jTCN2eijN967nt7w8Ug3DRR13Yl/PUfJ8DcODbXvB X-Received: by 2002:a05:620a:25c7:b0:939:bd4e:b2fe with SMTP id af79cd13be357-939ea2877e9mr555790285a.40.1789135808266; Fri, 11 Sep 2026 07:10:08 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:46 +0000 Subject: [PATCH RFC v2 08/15] bpf, x86: Maintain Tasks RCU trampoline nesting in the BPF trampoline MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-8-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=4464; i=josef@toxicpanda.com; h=from:subject:message-id; bh=ZoHASPcgXPkyj8nhWq6Qxe/k1AFcC7iFexyU3IrpvR8=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QNGMQXlnJ9mLTB5EylkME94aCeeA53rjkqD+SHndTEJHVSfeCPYIUtSGAIP6GARv7tJy5dM1hBK jP1T+H+zt3A0= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-c1860d/1789135810-D534D87B-3E826EF8/0/0 X-purgate-type: clean X-purgate-size: 4466 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789135833689158500 Emit an increment of current->rcu_tramp_nesting once the trampoline's frame is set up and a decrement before the final register restore, so that a task preempted while running fentry/fexit/fmod_ret/LSM programs or the __bpf_tramp_enter()/__bpf_tramp_exit() glue is not treated as Tasks-RCU quiescent. Drop the count around the call to the original function: that may run arbitrarily long without sleeping and must not pin a Tasks RCU grace period, and the trampoline frame above it is held by im->pcref rather than by Tasks RCU (see bpf_tramp_image_put()). The fmod_ret early-exit branch and the ip_after_call -> ip_epilogue poke both skip the decrement/increment pair around the original call, so the count stays balanced on every path. The sequence is "mov r11, gs:[current_task]; inc/dec dword [r11 + off]"; r11 is scratch at every emission point and (u32)¤t_task is a valid sign-extended %gs-absolute with the current per-CPU layout, the same form the JIT already uses for this_cpu_off. The image is dynamically allocated text, so the instructions outside the bracketed region are covered by the irq-exit IP check. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++++++++++++++++++++++++++++= ++ 1 file changed, 43 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2853e87797a7..a375c1b7bd50 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -722,6 +722,31 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg= , u8 *ip) *pprog =3D prog; } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). + * + * mov r11, QWORD PTR gs:[current_task] + * inc/dec DWORD PTR [r11 + offsetof(struct task_struct, rcu_tramp_nesti= ng)] + * + * r11 (AUX_REG) is scratch in the trampoline at every point this is emitt= ed. + */ +static void emit_rcu_tasks_tramp_nesting(u8 **pprog, bool enter) +{ +#ifdef CONFIG_TASKS_RCU + u8 *prog =3D *pprog; + + /* mov r11, gs:[abs32] */ + EMIT2(0x65, 0x4C); + EMIT3(0x8B, 0x1C, 0x25); + EMIT((u32)(unsigned long)¤t_task, 4); + /* inc/dec dword ptr [r11 + disp32] */ + EMIT3(0x41, 0xFF, enter ? 0x83 : 0x8B); + EMIT(offsetof(struct task_struct, rcu_tramp_nesting), 4); + + *pprog =3D prog; +#endif +} + static void emit_return(u8 **pprog, u8 *ip) { u8 *prog =3D *pprog; @@ -3610,6 +3635,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im /* mov QWORD PTR [rbp - rbx_off], rbx */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); =20 + /* + * From here until the matching decrement before the final return, a + * preemption of this task is not a Tasks RCU quiescent state. The + * instructions above this point are covered by the irq-exit IP check. + */ + emit_rcu_tasks_tramp_nesting(&prog, true); + func_meta =3D nr_regs; /* Store number of argument registers of the traced function */ emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); @@ -3670,6 +3702,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size); } =20 + /* + * The original function may run for a long time without + * sleeping; do not let it pin a Tasks RCU grace period. The + * trampoline frame above it is held by im->pcref + * (__bpf_tramp_enter()), not by Tasks RCU, across the call. + */ + emit_rcu_tasks_tramp_nesting(&prog, false); if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); EMIT2(0xff, 0xd3); /* call *rbx */ @@ -3680,6 +3719,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im goto cleanup; } } + emit_rcu_tasks_tramp_nesting(&prog, true); /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); im->ip_after_call =3D image + (prog - (u8 *)rw_image); @@ -3741,6 +3781,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_rcu_tasks_tramp_nesting(&prog, false); + emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); =20 EMIT1(0xC9); /* leave */ --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12E1633A9E1 for ; Fri, 11 Sep 2026 14:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135816; cv=none; b=J4cE8DRPxGj1QvhGIWkZos8sQJcc4FT6DyrG0gYkVAihMr6t5unMWFnW2ElNL38cfEGUdebBq/eJkyo6xOIKYZ43v+94ECgdoY4o2Kk7gSkHsBYJnXv0khXua2TPI9g0Yxrb8zIjyoc7dvms76Emr3Dh7paNni+rW0Pzqs2LobY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135816; c=relaxed/simple; bh=xqTPn5S4ER1XAZhtIYfJFNG5V6jdP8hXJAwXIfHAJeM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=o31vGFY9rs7ZKEHTvHopqBncU9rirO7hsDX9Gt30wKDKv0K7valVXwBLW2/Xf5OdSLsaKVLUvVDiobCn7tjDwOk855bBsdGjR803/bnBRPTYrwC2KeGyBrG31hR4DzEjrY9bw7Fk5PEuM67pwxfvH6IPvE9HezbJmaAMa3/HwhM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=RL7TN91v; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="RL7TN91v" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-9399daa3c8eso76786985a.1 for ; Fri, 11 Sep 2026 07:10:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135813; x=1789740613; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=RL7TN91v33ejtXycFyxg3fUnBjQf8abI+hin2UNGZI2wBeWwf49htfb9Aj8vUw2Muq k9u8JLOpfzYnL+lg0qTqWffW0ltVG7wjneIpkZseOnt26NGyUT2jtGsu3wJxAwhF9TfT Q7tUWoFJCzyfPptfXsbE6n8fgKUbf9pfjLsavhbujWDYGXt48pF1kZryFdJgRiRIX4ge pIeqsVTdFT92F/zIFJZMB18PnPCOKsOH1fvdO3mIZVGLVrTrMhmrC6bHOTAOhAsTrZJJ W95Cqb6b3057kaKanyW0Z8xbp2flwvBJ4Aa1e2hOY5E02BLr7W9PPTVAWk0kBW/CyAoq rIDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135813; x=1789740613; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=L7/PdEgQoCXtFo4pyw0xdFVjKNH7rlwejO7Z445bnJji0QsAO+PGzPtoVnBgjbItG3 RS/94ubGsfhX7JpNJSWdW3ZFSLA8mprxhXIhErAK3ynlCPXJvS6F0eJHBvbsKSSLf7zm XF767X/ZRLPh2pi47cNP2BPX+ULLWyvpUteqDGKWbMIjmPUGFm7CIzsU7bn1iUtmfU34 k6HU5ebqTCexK0qV0tE9qN7+hzdWx6BF2Qevvky2+0xLWLiC0igYoGoK7eijfBd9Tx6T JwSwsNtVbYmVObCcyfyEIjOw0oXnk06A6QMdy9tgoYmA5IHaR917bC/csnAilCJwfuwN iPxA== X-Forwarded-Encrypted: i=1; AKwUvByMaPVmYRIGLA5pFVVspFtu+5dT21imPpYzTjcW16ImqxX5XQRBWFKgW2G33tfqw3/jHjeqabxRHTaM3ck=@vger.kernel.org X-Gm-Message-State: AFuF++lEQxWtqrvyBlUlAFh8/fArKnGSzT9murzRAnlK/Ks6sjsSt+Q1 M5gfnoVEqKzOaknFLLRuRTRCzadGn7DaNALTgfJp4I9hLk0YHUB5ftfpgewIYBVvpEY= X-Gm-Gg: AYBFou35AxBenBsVdUZR8Fx2IQjm8MAYaLTETeAxTa5dRAHEeH6FKuKruMiQUT7wzYf TehZG3Vb/VsZB5OroJ3bpLTidawCfX6JLwoM+w6AupWCv9e/DMOiEQRiJ8rmMFy5tLZmpy6hpAG 9VuiUZpg4EA5f25O3M/JN8V3tFrND9tM9i27W+PRKdJbuVJbwafZpPjgJb4Nvojzf0GomTAxWt4 xD1pUud7krJ/sbN47oum2rV2Ad39kATr0TPtoeY+gkjC7bCBaBp9LZSrSqE/9mIIeyBIB6apmqt l0o/6gmp+njNGBmBcJuBYunMNa5gPVyHPGMlfCcHL+zMxiYDkP447Zr8SVQKoqvGBkHoOAHft8i 9kEnxCvWevKsmy8vou+0Xr/Zu/hFZPI1Ua+4KHSsSwNK6U9TYs1OyGeZUMzHjibclr/sOqR0zFI oQankuHO5+0b1uWvK03o+geQNUArwHiHXkBvIMbJdX/qWY37ycURoOeiGpjt/ZP+51X6xWAVPQC C7y6M62unw807sVxeOm89Q0KjDipANfuSPXc7+DoqZsR7vr839aIb4s X-Received: by 2002:a05:620a:6287:b0:939:6dea:374b with SMTP id af79cd13be357-939ea1af385mr553869585a.47.1789135812492; Fri, 11 Sep 2026 07:10:12 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80bbab7sm247620785a.34.2026.09.11.07.10.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:10 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:47 +0000 Subject: [PATCH RFC v2 09/15] arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-9-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=5217; i=josef@toxicpanda.com; h=from:subject:message-id; bh=xqTPn5S4ER1XAZhtIYfJFNG5V6jdP8hXJAwXIfHAJeM=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QDdDEdIwM05h7b7JvpgB0Ncpvko1HStiAd9VzLg7h8HjWsQnjthe7VWx1bLRiDUbfVMTbRgUiKe YsgeGrk28wQI= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Bracket the call out to the ftrace_ops callback in ftrace_caller with an increment/decrement of current->rcu_tramp_nesting, using x12/w13 which are scratch there. The read-modify-write is not atomic, but only current modifies the count and every interrupting user is balanced, so nothing is lost. ftrace_caller itself, including the early CALL_OPS direct path and the late direct tail that carry a BPF trampoline address in x17 with the count at zero, is static kernel text: add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and provide arch_rcu_tasks_ip_in_trampoline() covering [ftrace_caller, ftrace_static_tramp_end) so the irq-exit check treats a task interrupted anywhere in it as inside a trampoline. The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which arm64 does not select until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry-ftrace.S | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/kernel/ftrace.c | 16 ++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offset= s.c index 9c853ed3ceab..f6655a284f18 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -39,6 +39,9 @@ int main(void) DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); +#endif +#ifdef CONFIG_TASKS_RCU + DEFINE(TSK_RCU_TRAMP_NESTING, offsetof(struct task_struct, rcu_tramp_nes= ting)); #endif BLANK(); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_conte= xt)); diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftr= ace.S index 025140caafe7..46a102e7199a 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -14,6 +14,33 @@ #include =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). The wh= ole + * of ftrace_caller is treated as trampoline text by the irq-exit IP check= (see + * arch_rcu_tasks_ip_in_trampoline()), so these only need to bracket the c= all + * out to ops->func; everything before the increment and after the decreme= nt, + * including the direct-call tails that carry a BPF trampoline address in = x17, + * is covered by that. The count is only modified by current and every ne= sted + * user (interrupts) is balanced, so a plain ldr/add/str is sufficient. + */ + .macro rcu_tasks_tramp_enter, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + add \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + + .macro rcu_tasks_tramp_exit, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + sub \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + /* * Due to -fpatchable-function-entry=3D2, the compiler has placed two NOPs= before * the regular function prologue. For an enabled callsite, ftrace_init_nop= () and @@ -94,6 +121,8 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE =20 + rcu_tasks_tramp_enter x12, w13 + /* Prepare arguments for the tracer func */ sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) @@ -111,6 +140,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) bl ftrace_stub // func(ip, parent_ip, op, regs) #endif =20 + rcu_tasks_tramp_exit x12, w13 + /* * At the callsite x0-x8 and x19-x30 were live. Any C code will have prese= rved * x19-x29 per the AAPCS, and we created frame records upon entry, so we n= eed @@ -178,6 +209,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp) SYM_CODE_END(ftrace_stub_direct_tramp) #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +/* End of [ftrace_caller, ...) for arch_rcu_tasks_ip_in_trampoline(). */ +SYM_CODE_START(ftrace_static_tramp_end) +SYM_CODE_END(ftrace_static_tramp_end) + #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ =20 /* diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index e1a3c0b3a051..1b7ac2afed0d 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -17,6 +17,22 @@ #include #include =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +extern void ftrace_static_tramp_end(void); + +/* + * See rcu_tasks_ip_in_trampoline(). ftrace_caller and ftrace_stub_direct= _tramp + * are core kernel text but must be treated as trampolines: a task preempt= ed in + * them may be carrying an ops pointer (x11) or a direct-call BPF trampoli= ne + * address (x17) whose lifetime is guarded only by Tasks RCU. + */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + return ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end; +} +#endif + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct fregs_offset { const char *name; --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46213369D57 for ; Fri, 11 Sep 2026 14:10:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135819; cv=none; b=QZxd3cbh3tC4lKbAIRLngNymyeGCZgCOtP+DuCQ5QDbehlZn86MWRkKjFB/aUGJwRFAHKC9bnSFo7JJpDd/J61F6UKvwu2Rkp9u37G0tGx6nda9ifypurDR0M63oPFIkCKWQkya6or839gZg7yCO67eRKJOOMgwpPTki3rx+g4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135819; c=relaxed/simple; bh=MVjrbMDjnAqQ8aHx7Jc4sWMH68fI2crdoTABU++vCCI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=RNEekXsZtplamDKMUA8qLwk6+EkP4i5zJ0n+uDCEVpDm31I7Ns0JHD18Q09W7J5unnn3CPOxbtK4MW1pYeG9lkLoSgMbeBSVIxR0oVNSBGSGPOoWt1td+TmyN9UJxdzgFrTBaTuLhGtA+ZaRUsiIhAKzIuaxZlRdfDsVjzoVySc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=V/jf2qeX; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="V/jf2qeX" Received: by mail-qk2-f12.google.com with SMTP id af79cd13be357-93910cadeb7so101377685a.0 for ; Fri, 11 Sep 2026 07:10:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135816; x=1789740616; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9b8kkjvSg0tb+ScPvKP3GAz1Pld5wxO+feqfkbC4jq4=; b=V/jf2qeXIvFHzb3i8uU2lXQ1gE+0HHweIvmhB2+6xvaItDBfYbudP6XTknXoUcoRpM JI2aM1Rb8d1vZph0Gjd1qTZRXB/t2SnAHqC0cL/3vcKY3yV7JgoElEOsFeJjz5tCP0zw ZR5fM5xZS9xpYPuykSOFqN4fB6PE145qsn5PXNJF+MwFexRSUvIMmBBAopiKW5l8MYOw 1M2tzbSxQRpTD7z12U2XZxb0+EYoKwQML47Chmwx3wGnQkf6DaQOtIC9QDK0Xe9+q4Kb toncWkdvtv8n0BRoOYFauRuD473UYE2ErkXYDwRZ0oAs+/TL5Xt3kuOXG5/ZWl623IJY 4N9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135816; x=1789740616; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9b8kkjvSg0tb+ScPvKP3GAz1Pld5wxO+feqfkbC4jq4=; b=eoELJb7a4HKQvq5T4glocVIl6VLdrW4FCHGQf6CesJOdwzwLOyrYqmnIDEK2gIrOgQ TL9kjkTJWm1NxrqA+SI2QLndUko/IqVUwpz7nH0aPexNHHW8g8XTGuY7yyfDO17icg1i XQfBYSHWEswMxEPW+zhPSGIxk3GdNTfXAnrWmTfqJcwj5Bl9Wf+ThWzi7E5dFQNroWte hwPs6rdIfokkfE0tNNMnx4NtZJMBI/G/JpBL8NKAtlDuVoGV2Hf+9sjWcAuyd35brkvV hioYIIhOMg7WYxkkxJGDbZzx6j0Wc6wtO3xpOz/kwTA+0/de8IAjfIywA719L0AT5C5s NmgA== X-Forwarded-Encrypted: i=1; AKwUvBzH1lu1RygZoxdOcQDm44yAyOXWMvmuwtvUFWvvgEuUMHCIuAzZOQP7/c4wiA3iArTD6ylEQZDLi5OkbGk=@vger.kernel.org X-Gm-Message-State: AFuF++nwkNvjGxfws150a1gqL8SBqRXf3F8KqAB12p1/MT6XgrYHXkrb 9T7EY2knsaRvAzR7K6wpXYnqHpIyz6u3NtbDuVlx6IOl0LDBK1FxSvFKUZHyRL2Hr/c= X-Gm-Gg: AYBFou0UUKrjmQENUQZHe4PcYDUwW3NiZINxUFyE7W9Lne7jBwprs5uPv2o46LgWAzK uKzJzzK5STqPAR2Y+i4t6uGSZ7ChdnLtJwsvQQ8SlKavMkoye+3LX9RNZlpJttwvPDowyPUl6Bw xdSV2RQOE/9rnfDECCYxUPaUfBgtW8dQxw+P3pcm/QadUxbLg3cia0sRN2b12cxsdJwPf45GKqt 75Tqs9V8xeFNqsd3QOSfPi/PREaMoGcYnqW68zWe4yIuNQqrpq/uHR9xn9qts/FC14sUpBMJn9c rJ3m7hTgooW7LXH6KwcLeSQnmmVXV7qHC/r4xd93eISKKMODK+dZzMcJPnMpCqT4PAVqzVmC5bO 687wAEkp8LMeerdm1EfV13fegv3lAZk6EkeuhADioYe6Ig1QxCOTdWequbPcbrHdFn+CemVieU1 RkBMYuGnPPKnTWQPM+Y0DKZz5cA/gwKt/0QJhX8SPaP79oClVi2r+MDWV5YE4nIaaB2bB75SI13 KjjPYrLdgKPmw9PmxwWgn83AC2AV7VIz10EI9446Y0BV/3p1m7Zr/wT X-Received: by 2002:a05:620a:4103:b0:939:7f14:ebc6 with SMTP id af79cd13be357-939ea036d6cmr589566285a.3.1789135815998; Fri, 11 Sep 2026 07:10:15 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80bc225sm248184485a.36.2026.09.11.07.10.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:14 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:48 +0000 Subject: [PATCH RFC v2 10/15] bpf, arm64: Maintain Tasks RCU trampoline nesting in the BPF trampoline Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-10-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=4364; i=josef@toxicpanda.com; h=from:subject:message-id; bh=MVjrbMDjnAqQ8aHx7Jc4sWMH68fI2crdoTABU++vCCI=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QLeRHDw//rEtUsP8CEBuDuvfZIDhzXHIZMrHDgrgPTpnzRVCinhLEvplAgnBDiwNcYGB3QeJu+7 QOU1v97vhHA4= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Same scheme as x86: emit "mrs x10, sp_el0; ldr/add|sub/str w11" to bump current->rcu_tramp_nesting after the callee-saved registers are stored and to drop it before they are restored, and release it around the call to the original function, which im->pcref protects and which must not pin a Tasks RCU grace period. x10/x11 are scratch at every emission point; the fmod_ret cbnz target lies after the decrement/increment pair around the original call, and the ip_after_call nop follows the re-increment, so the count is balanced on every path. BUILD_BUG_ON guards the LDR/STR immediate range for the task_struct offset. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/net/bpf_jit_comp.c | 46 +++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 46 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41db..5c9a7bde5cc9 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2591,6 +2591,34 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx,= u8 dst, u8 src, bool nullab emit(A64_SUB(0, dst, src, base_lo), ctx); } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). + * + * mrs x10, sp_el0 + * ldr w11, [x10, #offsetof(struct task_struct, rcu_tramp_nesting)] + * add/sub w11, w11, #1 + * str w11, [x10, #...] + * + * x10/x11 are scratch in the trampoline at every point this is emitted. + */ +static void emit_rcu_tasks_tramp_nesting(struct jit_ctx *ctx, bool enter) +{ +#ifdef CONFIG_TASKS_RCU + const int off =3D offsetof(struct task_struct, rcu_tramp_nesting); + const u8 tsk =3D A64_R(10), cnt =3D A64_R(11); + + BUILD_BUG_ON(off & 3 || off >=3D SZ_16K); /* LDR/STR (imm12, scaled) */ + + emit(A64_MRS_SP_EL0(tsk), ctx); + emit(A64_LDR32I(cnt, tsk, off), ctx); + if (enter) + emit(A64_ADD_I(0, cnt, cnt, 1), ctx); + else + emit(A64_SUB_I(0, cnt, cnt, 1), ctx); + emit(A64_STR32I(cnt, tsk, off), ctx); +#endif +} + static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, const struct btf_func_model *m, const struct arg_aux *a, bool for_call_origin, bool is_struct_ops, u64 arena_base) @@ -2854,6 +2882,13 @@ static int prepare_trampoline(struct jit_ctx *ctx, s= truct bpf_tramp_image *im, emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx); =20 + /* + * From here until the matching decrement in the epilogue, a preemption + * of this task is not a Tasks RCU quiescent state. The instructions + * above this point are covered by the irq-exit IP check. + */ + emit_rcu_tasks_tramp_nesting(ctx, true); + if (flags & BPF_TRAMP_F_CALL_ORIG) { /* for the first pass, assume the worst case */ if (!ctx->image) @@ -2898,12 +2933,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, = struct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_CALL_ORIG) { /* the original func takes kernel addresses, never converted ones */ save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0); + /* + * The original function may run for a long time without + * sleeping; do not let it pin a Tasks RCU grace period. The + * trampoline frame above it is held by im->pcref + * (__bpf_tramp_enter()), not by Tasks RCU, across the call. + */ + emit_rcu_tasks_tramp_nesting(ctx, false); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); emit(A64_RET(A64_R(10)), ctx); /* store return value */ emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); + emit_rcu_tasks_tramp_nesting(ctx, true); /* reserve a nop for bpf_tramp_image_put */ im->ip_after_call =3D ctx->ro_image + ctx->idx; emit(A64_NOP, ctx); @@ -2945,6 +2988,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, st= ruct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_RESTORE_REGS) restore_args(ctx, bargs_off, a->regs_for_args); =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_rcu_tasks_tramp_nesting(ctx, false); + /* restore callee saved register x19 and x20 */ emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx); --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A494369D6A for ; Fri, 11 Sep 2026 14:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135823; cv=none; b=HAwSYoh713iQxawulKhLWjsPf4YZAF43OiPanGgYLs0MkHY6eYT131WPQYjPYGyl3HzLQM76t22h3e/SHS1UX2Ho0DLR3/zfJyOkJOPfHuXUoljdR8VDlE+MrP9q4hqF7KYibJ4iJ/Z0eCTQkPzzkAvpJbHTUSk6nO5GMKhXr7U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135823; c=relaxed/simple; bh=hPcqGdFxo+b+ijS7jNB+8GMowmEDvopzKnpNmpphVrY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YCe78PjlvBdBCLWYj8VM7NZCoZIwzy5vZjwxn4PDz/IwNfEpp8G+Ayy3Heue1SplovyzcuFapLI3kYJYEaQk1ThpFz+BVcRKxjxd8KQcVLjrhPvBpRafNqkwFQYTpF3cLfIe46BZbiKli+svfGfM6IJIV1jS9Vr5yE8aIV4z6EQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=J6fC02vx; arc=none smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="J6fC02vx" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-5306baf6b53so15000001cf.1 for ; Fri, 11 Sep 2026 07:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135820; x=1789740620; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=J6fC02vxUj7R8uLNkm6j2UFmhumP9masIbwPFYObyCpfnCpl3MwEPR2KHNb3TyycME sNpKatHEdR199uSZcOxHQdrLrPW7i5wb1ApmozfvvD0hBYo2mTiACAsndyXwmdKVEdka 08gLK3Be7se5sydXOPVnlve+QHJoL+v/vGwXUs7VmnoCYAYO78aZFdPbcl27Cxkq97PH mMXdGQjeed5kX+kAkvMU5nCtrn9ory3l/CIoX7AuYKpRmhW2FdiprLBawGjB7tYJOAnG if093zQPO+yjtGUUhHQemjKilV6M3h1sN/CklO3LZU1tzpqLhfQQbX+isj8hXyteX8NZ Oa/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135820; x=1789740620; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=pI74BXb+STexKz09aqF7h1OgFx4I1w6mB+dlJx+uhzEF3GoN/8HQCTsQP+IeVTMt3F p/WiW2pyA4uC7ARgsiVpm0/4JhQZ1vI2fdOIZxqOBN3V30iLBy4At6tn/QFaHccKkoTW xHzchGqxN9h1G2PRXhqi8LzSOegY09Crt2Vo2nViu36STeYHvD4Z21KpZbwP7OZ4rLRy Mbj6w5lJOF+IFyxkhs5T0oZyc/HGBYE3Os7NMSiUqXLnYxcxUo7TIxHBw9okklSDdvhD l9/9jUEc6zAhIroCBmzHxDkGymYXdwWDBrV9u3CPH9ZZXHkLYdI4O2e4Vbo1CuTC1zbu In+g== X-Forwarded-Encrypted: i=1; AKwUvBw0JKB5DSctN9MVyx03pqaV3fmF77AehMEXsuUT2BtQLfc3Xfk/sIdwfbzaz6n3FyAhKEa076fwtAbqnfQ=@vger.kernel.org X-Gm-Message-State: AFuF++nbvQcZb2D4V1Qo+E+qWN04QPs+mJPsWvbLeNs/Dte+6AnUAVZj vsV5M2eI9E41AGP7XyazxTVmbgkk0OUBQY7S/By4YjOYnQ5OQR9t0bVnsz5z69c/ELc= X-Gm-Gg: AYBFou2pdvCfybgKTc167cchKVu3wmkk2N3G5/g3huYD95uksfbpKpYiHTW5OspGtm+ 2BgYQi4/KMpwDMd0xM1DSf8srzeHujNrlionm87pLlBVrq52IkviQ4wWUfrrV5SyEDDW++w+WVp Yk3uFBmbzPVTJwHa9eO30GX/3stvQS0Cc4Fee6w9GoBLYb7UHMcgTk5687DBqevkMrrAOwfYs07 2dO+XCuawMv+yHBNi08MNfgr7x0+F3tSYUWjR/XBndDEMUj3EwF6i1k6EJg5XatGhYAHlsoi2zP x2A2bcHxi1OarkzkIvdCykX0mJN5gkoNkUFtDIOSB24UpHHIBzqPjUgK5Gn1GmcR6Ukbsi7ht0w mB0tHKghtggPOct4yaKHTUlOs3pMlXGt8xNVNCm70AEgfUQH76++y1T6J0SSSdkcsCKbltw7OrD 16aTUdhuzS2FGvss2JMnHruKbANzKRu0QYR0+7lcnutVm1q/spgRtsvYaVuanCegJovOOG1AJ/U rH8a5GzogPZsNvX5RFaF3ZvHpXlW0CDnwOQRneS20ZHGyl/LEjN2ZYGcACF2SPsqt4= X-Received: by 2002:a05:622a:4114:b0:530:da2a:757 with SMTP id d75a77b69052e-530da2a1198mr24113211cf.33.1789135818762; Fri, 11 Sep 2026 07:10:18 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530e0aaddbdsm2707391cf.2.2026.09.11.07.10.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:17 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:49 +0000 Subject: [PATCH RFC v2 11/15] samples: ftrace: Maintain Tasks RCU trampoline nesting in direct-call trampolines Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-11-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=10811; i=josef@toxicpanda.com; h=from:subject:message-id; bh=hPcqGdFxo+b+ijS7jNB+8GMowmEDvopzKnpNmpphVrY=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QKhN0qkThHUh5I4+Grit8p3XEg0LAgV/aMtyHMka/HOmaDw0JgyPUhuISBQoEJ4d9x0FD6DquBp sygcZlr+QrA8= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Follow the register_ftrace_direct() contract in the sample modules: on x86-64 and arm64, have each hand-written trampoline increment current->rcu_tramp_nesting before calling its C handler and decrement it before returning, via a small shared samples/ftrace/ftrace-direct.h. %r11 and x12/w13 are used as scratch; both are caller-saved, non-argument registers and therefore dead on entry to and exit from an fentry trampoline. The header pulls in the generated asm-offsets.h only on those two architectures, since it is not generally safe to include from C (PPC32's TASK_SIZE and arm64's TRAMP_VALIAS clash with the C definitions; the latter is worked around locally with push_macro/pop_macro). Other architectures get empty macros and are unchanged. Assisted-by: LLM Signed-off-by: Josef Bacik --- samples/ftrace/ftrace-direct-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi.c | 5 +++ samples/ftrace/ftrace-direct-too.c | 5 +++ samples/ftrace/ftrace-direct.c | 5 +++ samples/ftrace/ftrace-direct.h | 64 +++++++++++++++++++++++++= ++++ 6 files changed, 97 insertions(+) diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-= direct-modify.c index 164d9dd6fd92..eb8230fa4242 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -73,7 +74,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func1\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -85,7 +88,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func2\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -141,11 +146,13 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func1\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -153,11 +160,13 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func2\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/f= trace-direct-multi-modify.c index b03766c6217b..c8f1062e5d1a 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -77,10 +78,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -92,10 +95,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -154,6 +159,7 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -162,6 +168,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -169,6 +176,7 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -177,6 +185,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-d= irect-multi.c index 3fe6ddaf0b69..bc6a88dd4ffc 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #include #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include @@ -56,10 +57,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -101,6 +104,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -109,6 +113,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-dir= ect-too.c index bf2411aa6fd7..247e418644a2 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -61,6 +62,7 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n" @@ -70,6 +72,7 @@ asm ( " popq %rdx\n" " popq %rsi\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -110,6 +113,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #48\n" " stp x9, x30, [sp]\n" " stp x0, x1, [sp, #16]\n" @@ -119,6 +123,7 @@ asm ( " ldp x0, x1, [sp, #16]\n" " ldp x2, x3, [sp, #32]\n" " add sp, sp, #48\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 5368c8c39cbb..9e1964baf28b 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -3,6 +3,7 @@ =20 #include /* for wake_up_process() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -54,9 +55,11 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -97,6 +100,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -104,6 +108,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h new file mode 100644 index 000000000000..d0313f33f47f --- /dev/null +++ b/samples/ftrace/ftrace-direct.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _SAMPLES_FTRACE_DIRECT_H +#define _SAMPLES_FTRACE_DIRECT_H + +#include + +/* + * A direct-call trampoline is entered with no lock, refcount or RCU marker + * held; only Tasks RCU keeps it (and, for a module, its text) alive while= a + * task is inside it or preempted in something it called. On architectures + * that select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU + * quiescent state unless current->rcu_tramp_nesting is non-zero, so the + * trampoline must raise it before calling out and drop it afterwards, exa= ctly + * like the ftrace and BPF trampolines do. See rcu_tasks_trampoline_enter= () + * and register_ftrace_direct(). The instructions before the increment and + * after the decrement are covered by ftrace_direct_mark_module(). + * + * These expand to instruction strings for use inside the samples' asm() + * trampolines. The scratch register is caller-saved and not an argument + * register, so it is dead on entry to and exit from an fentry trampoline. + * + * The generated asm-offsets.h is only pulled in on the architectures that= need + * it here: it is not generally safe to include from C (e.g. PPC32's TASK_= SIZE + * and arm64's TRAMP_VALIAS clash with the C definitions), which is why the + * samples themselves guard their own include of it. + */ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_X86_64) + +#include + +#define RCU_TASKS_TRAMP_ENTER \ + " movq %gs:current_task(%rip), %r11\n" \ + " incl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" +#define RCU_TASKS_TRAMP_EXIT \ + " movq %gs:current_task(%rip), %r11\n" \ + " decl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" + +#elif defined(CONFIG_TASKS_RCU) && defined(CONFIG_ARM64) + +/* arm64's asm-offsets.h redefines TRAMP_VALIAS from . */ +#pragma push_macro("TRAMP_VALIAS") +#undef TRAMP_VALIAS +#include +#pragma pop_macro("TRAMP_VALIAS") + +#define RCU_TASKS_TRAMP_ENTER \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " add w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" +#define RCU_TASKS_TRAMP_EXIT \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " sub w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" + +#else + +#define RCU_TASKS_TRAMP_ENTER +#define RCU_TASKS_TRAMP_EXIT + +#endif + +#endif /* _SAMPLES_FTRACE_DIRECT_H */ --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qv2-f12.google.com (mail-qv2-f12.google.com [74.125.230.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABD1B36EA8B for ; Fri, 11 Sep 2026 14:10:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135824; cv=none; b=TNXcF5PHXK3SOOBQAYg2CCxq8+gqpi8Fg68tP4PAm7e7G8sIo61LfjZYF90j0TNtiZO29H453JtEiV6Rr0ADsrAdwiWAQV3vgKQyOLQRQUTe/OoSru+sQiBVsVwobt+aJ3iypUVt6rp77BJZfme7I73laMXsSk74BpDfu0i599Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135824; c=relaxed/simple; bh=c9hUQ9kAAl2b2IbncqDZq1k2ejJxuc1g83EyvfAIkis=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=lrGmpBIxGnEzjZsDK5ZaKvCmoJSlRtmsC7xI2rUHkibTa0ByOG6R6cIOrfrdpMkr++wPjrCTeYVW+DDcl2I6nFgcfvlcISo/9YyB/iyy2G6w2gYKFIWhxGvsmR0hhMVKfH9jU8VQ2lVy7kGZ/jeLPDyrpzXpc+JR2w4jMpSa+6o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=VNKysXO3; arc=none smtp.client-ip=74.125.230.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="VNKysXO3" Received: by mail-qv2-f12.google.com with SMTP id 6a1803df08f44-90cdfc9b6e4so8995836d6.3 for ; Fri, 11 Sep 2026 07:10:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135821; x=1789740621; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=twz7P/UgrzFqFrDTfXn+06wBF94/L3aT/pnjRwe1Smk=; b=VNKysXO35hDRh0I5aaik3B+ZyQfCj/A/akUeSWwO4t2rLjIxuWhPn+XPKU7gnGywL3 lG8fEKUQC7QElqObZ/b7r5Ip7EIJO13BG03nXDxPK2pGwu8aeUNKR93XiG4KF0VzFvws nlCDPYyeCxTBoVXwy/8O2LmEYq0pnIuyJ+tKpthhet9V30qNJXTmbhJgiRASHmq6cfbS Q/coTcxK4ImFJ2+MaByBx8YRIUAo0f0lnheaKFvC1i2EQ2TFNZlBVQ9LvXPZKPBALc4h X1/IYd9gJBBR8JgQ7InEmdp4f+BNcd7l6jbrB1/TTUmuDzCUYJq+0KZMA9NtqZnErt1h gqCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135821; x=1789740621; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=twz7P/UgrzFqFrDTfXn+06wBF94/L3aT/pnjRwe1Smk=; b=b7Httgqiex+KzI0kwwkYiVaPXvYzzhw+d2UJz9Ny7t2kEhzxTlGnfo2YhsLGaoYp5j 9wLVzGLDWD1isXSlVsHJPCYJ4CCiB1yaUabkifLaw2IUFbD4L5IlVAud86Apnss/9J0U UM2NpNm8V5s0h/1PG637cpl5B2y2lbGmftHQd0ggB79cUuEYyY+4mfuATJZxpFhdRvQl 9q0N0mEkPw3Qs9tAc/MF35AxF0vU+H13kQj9ftlXsHWZR3jTyFxP0AMVE14sxoaO6H7W h+RWwhwWuSDGAh2gXGX4ARnl7ypCXHZBimUjKNex1UQv22yMcCzo8niJ/j/6d6MymF1X IUqw== X-Forwarded-Encrypted: i=1; AKwUvByhyqshf6M1etmdGXHFISXUmE9lMwp8Ne/lltGAXKxVlDW53iYjMpfIQyeNQsH72cOCkNFXEXJNTjOo+GY=@vger.kernel.org X-Gm-Message-State: AFuF++nj56oUgwSW2AV5MoxShTxLyYekyBvAjNU2GzE6jhMApMLs+NmV rqbKNPY5obzlDjug42Dhp2nT4BAKOEB05oi8QXOew6fic7kzLzRIqkGcgtYRSGkCrHk= X-Gm-Gg: AYBFou2H1PMMmmAep4nWp5VLQO2C6JMKW7gcmuzkhDB47Y1Nl7aIELim+t2nccr/BlY K82RBqNn9fMgkrEYCbVG9L6AKHZd59OvcKHWKPOLHERo5cPKiUqwmpb+Aa69cnwVEEA0pay5KC+ wgd1LzSy7WLA6iLFfCTiCm3usO/NkBXkquQmhvXmhYteu2VPouzoxLtLDDxAT/pRvY2KEKP0OR0 uttym+Q+h/b4DiUogEqW48//hNYTS4pSghdivKyUWqanePFzATEKuARgz0BlUssE0t1vp10Hx2M ZfeIVI8rHfCDvMc5iv4s/RxzdbE+M0uDiMuj6IGRwYPQIGCVTG0QaYdq+oEbWDTwGm44+7LJ6Nx uODSITUHEWqx9ad42PSbZzQ5TrlmDwXLM9HA4fY8L8512Bjd0XZSK9YgUH3IRsg8rJDQx3oqJtK 8+VSNBjRBJK30ewyzUhCplGZu2KlJFT430+I4cJwV3ghRB1wHmNP/MBHxa41jbWp77hLSRMuEiE 0pmbYIuKtslQIXvYgZwoDkjnp+0kFBtsDRahOtYsVKWifj2oNmp2LFD X-Received: by 2002:a05:620a:690e:b0:939:639b:2e1f with SMTP id af79cd13be357-939ea04922amr525733885a.7.1789135821311; Fri, 11 Sep 2026 07:10:21 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80dc002sm248697485a.39.2026.09.11.07.10.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:20 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:50 +0000 Subject: [PATCH RFC v2 12/15] rcutorture: Bracket Tasks RCU readers with trampoline nesting Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-12-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=1492; i=josef@toxicpanda.com; h=from:subject:message-id; bh=c9hUQ9kAAl2b2IbncqDZq1k2ejJxuc1g83EyvfAIkis=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QDpeCdAsuDTbsWwi8oqAy5KN1IjS7Ye/ysM/qLEJAJ0SGsvHuCkttXUFQ2AlX09A0IwbwXmGUQ4 qHaGMzc7/Vgg= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA rcutorture's tasks flavor models a Tasks RCU reader as "any stretch of kernel code", and rcu_read_delay() deliberately preempts inside it to check that a preemption does not end the read-side critical section. Once preemption outside a trampoline becomes a quiescent state that model no longer matches what Tasks RCU protects, and the readers would report false too-short grace periods. Have tasks_torture_read_lock()/unlock() raise and drop current->rcu_tramp_nesting so the reader models a trampoline, which is the thing Tasks RCU actually guards; the deliberate preemption inside it then continues to be, correctly, not a quiescent state. Assisted-by: LLM Signed-off-by: Josef Bacik --- kernel/rcu/rcutorture.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 794937e13e7c..df6dd708cea7 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1144,11 +1144,17 @@ static struct rcu_torture_ops trivial_preempt_ops = =3D { =20 static int tasks_torture_read_lock(void) { + /* + * Model a trampoline: with CONFIG_RCU_TASKS_PREEMPT_QS a preemption is + * otherwise a quiescent state and rcu_read_delay() preempts on purpose. + */ + rcu_tasks_trampoline_enter(); return 0; } =20 static void tasks_torture_read_unlock(int idx) { + rcu_tasks_trampoline_exit(); } =20 static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C34DC374197 for ; Fri, 11 Sep 2026 14:10:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135828; cv=none; b=Y96bCu4lYWWnopY2AqZUIJfz5kHCZ+6jeBDiQUjnUK2Ms4hH/KbxA5DEtC+FfUTurTL9uVUplhH3+mXOWIcflyN/8/218lHR1Pa1Vdx9rLxYuCT0yO4sKPbJPYVAnOln1P2M5xWtC5A7/iZUfiFUodKpRUPsEa10C4O1KbBVGPk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135828; c=relaxed/simple; bh=782IiL7t8C+T7tHuP+fr7fuNKyJIY5LyrSPvir4aYJ4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=U0ScqaclHNIqyidmBMaHWsYziVHCsUETRG2uyMnUU40MsF2Q/hwFzFyfmibpQjd3lQDabL4w0Mz4rPT3zTdE3jIONBAFCTqerbNhXfM/bBE6nq7OhL5UOlAZ28I+uevd5DdbUWLbhjVM5XGLSbSrW9AKw9tFHN713cS8HGFt4E8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=hLLaGoJZ; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="hLLaGoJZ" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-939b5ded99dso92999085a.2 for ; Fri, 11 Sep 2026 07:10:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135824; x=1789740624; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WT/8H/cMArsJRoufbqdgzsmAKwU5NTozjyFMdjWjsc8=; b=hLLaGoJZppM4zK0atM90Nj0fQpwFsj08kX3VvrzD4jvzgzs21Z/Q39hmrmbyIItc2C aShIS1x6G8SXe4GGjuEtCAI9geSAnURZzQxRnzLyouKxXli0tHlnrIvAn6sUpJpkyTEV XBdd8IRda0wYXPgk2zKw4L+ITQviagPa944zKvrYnOZaqTvlDwLeJY5khHbqYHi0hjgI 1gjpYIe5YegKxBIE2C6OA/J9lR1d5wdZZhrOdVaRSoKWoGNOq/ART0GjYkGjGSSmr9/M nvpGRr84qKU/m2R9LkOW3u6tSuRx9c4MagdFtvZA52PZIpAHLSCX2dMjZDN/AlrMqneS +s0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135824; x=1789740624; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=WT/8H/cMArsJRoufbqdgzsmAKwU5NTozjyFMdjWjsc8=; b=htmTkgwhwtcG73H80Y2s1UvlKc4Kde6yP+afntJckKzOR1oxXNzhLaD/VZLagaWEo0 /fvvyzCHFTqr+WPmb+4rhh1fV/bydBmqWWbdkmudtmlR66vOOh3sxyqDYzjjvQA82d2J nCpxNJBIbCKFYKeS0+oRRtcXha7rvRVKxUyL1p1Pg5miuaD4v61rEF0A2y3rv2TTdu6y FSFh94JR8H6Ami2MaMXlab6tn0OQdV3hjsCIPlK0qTXzu4Rzo38WR2C+fr87kfc54izT JemyiWG+5/jTqRC/42Wcy1stX/Y2TevUwVam4jUuW6BhVN0r2rphjoVGuUabFXiw0VSn XClQ== X-Forwarded-Encrypted: i=1; AKwUvBwgPIcbQGfB8vvuwyiapDekJ0lEDhxgzzHeXje0Pf+u2BACsVWkTzFoyT34ksgu050mfS8g2YZwNwuSaWk=@vger.kernel.org X-Gm-Message-State: AFuF++mp6QREhLqip6gexr5KgP3+Zuf2oxkcYU9PuGlyyGF+0UCzl6Mm 4osTH3Q138YUkSulyYcOMiT6gGwHiMtEQFCmRcL5Njwa63ddnB/iNqe9CIBfUq/SWV4= X-Gm-Gg: AYBFou2c2DJDCghdG1cQqJKxx35ChARjz6JfG7kvB6w1Q77mRDM4i9gk6SRmyvHX4Dw PKYlVKutVtPDNdctGmKkc3fUkQIcj2KsM7JZN0WUbY15WFFOA6dRRHy4e3/pvEBOHQrxOlSITRr yl/rpVRfsocCLcppOnnEl6s1KCemCXkWcRyHDxvnzTqUT4zdSgI/3k7jzif8YCkQPEd/RSBtvw4 HbgVd+MeojN5GA3ZN/j6UUS4Z8l3iZvdeuaw1PNt6IAKx8h+JoKXTQKrNP8/FXtv6nOWK9iA5WF EyRVDwdY1xlAbm8ybSrSLauWGoZPTNtKheNjncbiMCu1BsKI5z3lz1cK/eTvvuMz/nNu0iM0gW2 vM9omQjXm1m9+j0hAa3ZN0ujcm7jevnEt4rb3kRVQQLlculPAniP6myst4wkX1lgW7y0Nw9tu/O eoKVE67cXuGlrG7CoZUnemQSo6urjk0V751dahXfUoypnivDJx//hFtekA491yBJjOnozvcvT8Y jcHrTBFXfWYpsXYktuIS1deVZMSiPu7O5ytIm6rVUvFIMRDPBK71YAg X-Received: by 2002:a05:620a:444d:b0:939:cab7:8651 with SMTP id af79cd13be357-939ea0cff3cmr508758285a.21.1789135824016; Fri, 11 Sep 2026 07:10:24 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80801c0sm248912485a.30.2026.09.11.07.10.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:22 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:51 +0000 Subject: [PATCH RFC v2 13/15] rcu-tasks: Treat preemption outside trampolines as a quiescent state Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-13-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=13377; i=josef@toxicpanda.com; h=from:subject:message-id; bh=782IiL7t8C+T7tHuP+fr7fuNKyJIY5LyrSPvir4aYJ4=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QLUZnTIeq2lPY+NhGtgUM19AxK92AiGBaR4ReDiwvRRgaRGOQrROuHTyqXvXkEa5pxrRzrRbB/J mvbDi+9g+vAw= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Tasks RCU only accepts a voluntary context switch, usermode or idle as a quiescent state, because a task that was preempted may be sitting in a trampoline whose text is about to be freed. On PREEMPT_LAZY kernels, where cond_resched() is a no-op and CPU-bound kernel threads only ever lose the CPU through preemption, that means any long-running kthread or kworker stalls every synchronize_rcu_tasks() caller -- BPF and LSM program detach and DYNAMIC ftrace_ops teardown via ftrace_shutdown(), and kprobe (un)registration via the jump optimizer, which waits under kprobe_mutex, text_mutex and cpus_read_lock() -- for its entire run, unless someone sprinkles cond_resched_tasks_rcu_qs() into it. A cgroup writeback worker draining a large cgwb for eleven minutes was enough to back 40+ tasks up behind trampoline_mutex and trip the hung-task panic. With the previous patches, every Tasks-RCU-protected trampoline on x86-64 and arm64 (ftrace_caller and its dynamic copies, BPF trampoline images, the optprobe template, out-of-line direct trampolines) holds current->rcu_tramp_nesting across its call-out, and the irq-exit preemption path holds it across preempt_schedule_irq() whenever the interrupted IP is somewhere the counter cannot cover: trampoline entry/exit instructions and other dynamically allocated text, the static ftrace stubs and x86 return thunks on the way into a direct-call target, modules hosting their own direct trampolines. The kprobe jump-optimization window, which is ordinary text a task may have been parked in before the kprobe existed, is instead re-checked against the recorded irq-preemption IP at each decision (rcu_tasks_irq_ip_holds()). A task that is context-switched with the count at zero and no such IP therefore cannot be inside, called from, or about to resume into anything Tasks RCU protects. So let rcu_tasks_classic_qs() clear the holdout flag on a preemption too when rcu_tramp_nesting is zero, on architectures that select ARCH_HAS_RCU_TASKS_PREEMPT_QS, and select it for x86-64 and for arm64 with DYNAMIC_FTRACE_WITH_ARGS. A running holdout is already poked via rcu_request_urgent_qs_task(), which makes the next tick set NEED_RESCHED; the resulting preemption -- from irq exit, or synchronously at the next preempt_enable() -- now retires it, so a Tasks RCU grace period is bounded by roughly a tick plus the longest preempt-disabled section instead of by the longest stretch without a voluntary schedule(). Other architectures keep the voluntary-only rule. Update the Tasks RCU comments, Documentation/RCU (Requirements.rst, checklist.rst) and the FORCE_TASKS_RCU help text to match. Cost: one load of current plus an inc/dec per trampoline entry and exit, and on irq-exit preemption one core_kernel_text() check plus, with OPTPROBES, MAX_OPTIMIZED_LENGTH-1 lockless kprobe hash lookups. Not covered: x86-32 and the other GENERIC_IRQ_ENTRY architectures, and return_to_handler / the rethook trampoline, whose C callees take the ftrace recursion lock before touching any ops. Tested under QEMU (x86-64, PREEMPT_LAZY, PREEMPT_RCU=3Dn, PROVE_RCU, with and without PREEMPT_DYNAMIC) against a kthread spinning in-kernel for 30s with the function tracer, an ftrace kprobe, an optimized kprobe and fentry/fexit programs live: synchronize_rcu_tasks() 29.7s -> 0.1-0.3s, ftrace_shutdown() of a DYNAMIC ops 27s -> 0.2-0.8s, the ftrace-direct sample modules load/fire/unload in ~2.5s each during the spin, no warnings. arm64 is build-tested only. Assisted-by: LLM Signed-off-by: Josef Bacik --- .../RCU/Design/Requirements/Requirements.rst | 28 ++++++++++++++++--= ---- Documentation/RCU/checklist.rst | 8 ++++++- arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/rcupdate.h | 15 +++++++++++- kernel/rcu/Kconfig | 7 +++--- kernel/rcu/tasks.h | 15 ++++++++---- 7 files changed, 59 insertions(+), 16 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index 8101fe6229d5..428b5e8f4b4e 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2739,13 +2739,27 @@ userspace execution also delimit tasks-RCU read-sid= e critical sections. Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to interact with them. =20 -Note well that involuntary context switches are *not* Tasks-RCU quiescent -states. After all, in preemptible kernels, a task executing code in a -trampoline might be preempted. In this case, the Tasks-RCU grace period -clearly cannot end until that task resumes and its execution leaves that -trampoline. This means, among other things, that cond_resched() does -not provide a Tasks RCU quiescent state. (Instead, use rcu_softirq_qs() -from softirq or rcu_tasks_classic_qs() otherwise.) +Note well that, by default, involuntary context switches are *not* +Tasks-RCU quiescent states. After all, in preemptible kernels, a task +executing code in a trampoline might be preempted. In this case, the +Tasks-RCU grace period clearly cannot end until that task resumes and its +execution leaves that trampoline. This means, among other things, that +cond_resched() does not provide a Tasks RCU quiescent state. (Instead, +use rcu_softirq_qs() from softirq or rcu_tasks_classic_qs() otherwise.) + +Architectures that select ``CONFIG_ARCH_HAS_RCU_TASKS_PREEMPT_QS`` relax +this: there, every trampoline whose lifetime Tasks RCU guards (the ftrace +and BPF trampolines, optprobe slots, out-of-line ftrace direct-call +trampolines) increments ``current->rcu_tramp_nesting`` before calling out +and decrements it before returning, and the irq-exit preemption path +covers the few instructions the counter cannot (see +rcu_tasks_ip_in_trampoline() and rcu_tasks_irq_ip_holds()). A task that +is preempted with that count at zero is therefore known not to be in, or +called from, any trampoline, and such a preemption *is* a Tasks-RCU +quiescent state. The obligation moves to the trampolines: anything that +relies on synchronize_rcu_tasks() to protect code a task may be preempted +in must maintain the count (see register_ftrace_direct()), or Tasks RCU +will not wait for it on those architectures. =20 The tasks-RCU API is quite compact, consisting only of call_rcu_tasks(), synchronize_rcu_tasks(), and diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.= rst index 4b30f701225f..28df48fecac7 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -252,7 +252,13 @@ over a rather long period of time, but improvements ar= e always welcome! a. If the updater uses synchronize_rcu_tasks() or call_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from - blocking. + blocking. On architectures that select + CONFIG_ARCH_HAS_RCU_TASKS_PREEMPT_QS an involuntary + context switch is also a quiescent state unless + current->rcu_tramp_nesting is non-zero, so a reader + there is a trampoline that maintains that count (see + rcu_tasks_trampoline_enter()), not an arbitrary + stretch of kernel code. =20 b. If the updater uses call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then the diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b5a51b0ef944..0e6c1e0b236f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -44,6 +44,7 @@ config ARM64 select ARCH_HAS_PREEMPT_LAZY select ARCH_HAS_PTDUMP select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_RCU_TASKS_PREEMPT_QS if DYNAMIC_FTRACE_WITH_ARGS select ARCH_HAS_HW_PTE_YOUNG select ARCH_HAS_SETUP_DMA_OPS select ARCH_HAS_SET_DIRECT_MAP diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..0a6427019345 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -99,6 +99,7 @@ config X86 select ARCH_HAS_PREEMPT_LAZY select ARCH_HAS_PTDUMP select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_RCU_TASKS_PREEMPT_QS if X86_64 select ARCH_HAS_HW_PTE_YOUNG select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 4cfe096d624f..9509f99ec965 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -210,6 +210,11 @@ bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); * preemption and rcu_tasks_irq_ip_holds() checks it at every quiescent-st= ate * decision, locally and from the grace-period kthread. * + * With both in place, on architectures that select + * ARCH_HAS_RCU_TASKS_PREEMPT_QS, a preemption with rcu_tramp_nesting =3D= =3D 0 is + * a Tasks RCU quiescent state, and a CPU-bound kernel thread no longer ne= eds + * to volunteer one via cond_resched_tasks_rcu_qs(). + * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. */ @@ -241,9 +246,17 @@ static __always_inline void rcu_tasks_note_irq_ip(unsi= gned long ip) WRITE_ONCE(current->rcu_tasks_irq_ip, ip); } =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +#define rcu_tasks_preempt_is_qs(t) \ + (!READ_ONCE((t)->rcu_tramp_nesting) && !rcu_tasks_irq_ip_holds(t)) +#else +#define rcu_tasks_preempt_is_qs(t) false +#endif + # define rcu_tasks_classic_qs(t, preempt) \ do { \ - if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ + if (READ_ONCE((t)->rcu_tasks_holdout) && \ + (!(preempt) || rcu_tasks_preempt_is_qs(t))) \ WRITE_ONCE((t)->rcu_tasks_holdout, false); \ } while (0) void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func); diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 999f8228a13d..8e7c94329105 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -94,9 +94,10 @@ config FORCE_TASKS_RCU default n help This option force-enables a task-based RCU implementation - that uses only voluntary context switch (not preemption!), - idle, and user-mode execution as quiescent states. Not for - manual selection in most cases. + that uses only voluntary context switch (not preemption, unless + the architecture selects ARCH_HAS_RCU_TASKS_PREEMPT_QS and the + task is outside any trampoline), idle, and user-mode execution + as quiescent states. Not for manual selection in most cases. =20 config NEED_TASKS_RCU bool diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1b9fe1bfa591..bab08a666dc0 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -905,7 +905,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) // // Simple variant of RCU whose quiescent states are voluntary context // switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle. -// As such, grace periods can take one good long time. There are no +// With CONFIG_RCU_TASKS_PREEMPT_QS, a preemption taken while the task is +// not inside a trampoline (current->rcu_tramp_nesting =3D=3D 0, see +// rcu_tasks_trampoline_enter()) is a quiescent state as well; without it, +// grace periods can take one good long time. There are no // read-side primitives similar to rcu_read_lock() and rcu_read_unlock() // because this implementation is intended to get the system into a safe // state for some of the manipulations involved in tracing and the like. @@ -1263,8 +1266,11 @@ static void tasks_rcu_exit_stall(struct timer_list *= unused) * period elapses, in other words after all currently executing rcu-tasks * read-side critical sections have completed. call_rcu_tasks() assumes * that the read-side critical sections end at a voluntary context - * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idl= e, - * or transition to usermode execution. As such, there are no read-side + * switch, cond_resched_tasks_rcu_qs(), entry into idle, transition to + * usermode execution, or, with CONFIG_RCU_TASKS_PREEMPT_QS, a preemption + * taken outside any trampoline (current->rcu_tramp_nesting =3D=3D 0, see + * rcu_tasks_trampoline_enter()); otherwise a preemption is not a + * quiescent state. As such, there are no read-side * primitives analogous to rcu_read_lock() and rcu_read_unlock() because * this primitive is intended to determine that all tasks have passed * through a safe state, not so much for data-structure synchronization. @@ -1286,7 +1292,8 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks); * executing rcu-tasks read-side critical sections have elapsed. These * read-side critical sections are delimited by calls to schedule(), * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls - * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched(). + * to synchronize_rcu_tasks(), (in theory, anyway) cond_resched(), and, + * with CONFIG_RCU_TASKS_PREEMPT_QS, preemption outside any trampoline. * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81D95374A16 for ; Fri, 11 Sep 2026 14:10:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135829; cv=none; b=gm9pF1q7/Jwyo8dhROkdUNHp6m36Uoa+dJGZfLQkP7Dfa7+d1fxfxGUYjFA3dwmVUjl6Nruicf7G4VzIgjUob/eEoJVXnFN+4fntydvd7aWn2YPrbrbOQNc7RLgWK0jNPyRCstl+SIB6Jw+4NyHQ7psfFXcAoWTOaIA7t7PlDCY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135829; c=relaxed/simple; bh=CHfEJf4XyY3zn8RyomjOM2Uc3mVFT7MUycrwjg2RabY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=SN1EwEhlpeeMBBv7C0Vx3c2lc0mQp5seUMkwlClEFX9kNAXhzGJhSkMhVHWrhr/oFXPjxgouJjGPeU5B+dkIAS0tWh38jN0XdY4hAvUlxjIduZTensWeuzhaLj0I7Bwo0M7ZjhvmJ/+J7Y8oXAXiFlpAd8t0smk8qFJ+cNFNAd4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=mt4DmxKM; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="mt4DmxKM" Received: by mail-qk2-f12.google.com with SMTP id af79cd13be357-939109fafd7so84824385a.2 for ; Fri, 11 Sep 2026 07:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135826; x=1789740626; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=tDcwMwF0CqD/SLUwt7lLP/Zeny+NN6n073A4kvHv5Ng=; b=mt4DmxKMFOa54vKaT7P7rC0nknUec5SG40ykFBll3pCY7tJxqt/x+SOCkXo3KNe18M yedzxk53shlT7hl4bnCF93vW8Y8pISo2hDuG8GRWZ/63qMENZom+pJ0aLDwuU3F8URLY sw2cIbuAA0M8+dqVCdaDdIwG5iib83Y3t4wTDLLWgp+XYVAQr1orQF86xBjSQEUfDkuI m1WVGkAxwPOHg/lTHRo1qpU7itzlq3ZKGLp28y4/aiV/Z3+87tG6AJ4lwxpq8U8eYIam 2Zoa7ytfUf8k+ZLC8eMWbTFPq7Fa+oTvHEHxNMUgVKDtXmGQ6Pl0liwZ2dOx/DYlFOQP Q1mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135826; x=1789740626; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=tDcwMwF0CqD/SLUwt7lLP/Zeny+NN6n073A4kvHv5Ng=; b=VPuOuZGAQz8fjo++Obw7pwG6FG5z0qoL8zlxiw4KH0DQ3VZwDinHiNpG3PyTJvmvHB YbzhNk9MbMgipThIrlYDnJkenga4ae9+Ub3VzQtuU1GBmVfvAz9CqR3isJmhFgFgVQfi uAt1Bx1Ct7N6qx9WM7jmnuWEToTRughG9f5ujZILHO+X8JBgKc5q5GPoDknUr74+G++4 klhEbqmAj7LNZ0/NqVDRjZLlG3ZehljGR2R0yccBRU3OFfaNuxwYNQfmRQFUyPQpZHuA D9iEW9dzw1ZaffHhkJrzRA6k4OYYZlLiHfY/qIh9EjPqQs5vfv1nWXW9SEFLyvTvRWak I67g== X-Forwarded-Encrypted: i=1; AKwUvBybVD4yQd43IQF3ElOmkZo41g2xXKLolBmhCSi/5o/5M+m9zSmJgJNl+pvcP4TVfTHrARPBRni4j5BttkU=@vger.kernel.org X-Gm-Message-State: AFuF++l9+FjJjXuLTh/fizF68+XD12x5mSDqDMmJOaX6n6rHMmWNeZFK rnP1vkuOHfHDaoEpELkn9xmnLWyScFqnNhJWoOWHG3wO0hsOyiqrshvXhv3iz/N5uXs= X-Gm-Gg: AYBFou3e4tHQBIgP3Vpi8KRIrpKVBO8agheFKhgaurD85lSaelCBPdWWuRKwBL0R0QL nUkt3Pp9AsWU+B9uAWLnJHnyVFJXDDo3ONvC0I2cZEegBQ8CekpSjZb196jTgiyqzG0Bv3Jl5q5 vFZ+XHzw0lOYB6LdL5HIt/O1DNn9pJU5SBiuv5CsQeloDm/Lnqcg2Wh53N0i9G9pW+zWbVPVWAH D1ijTWQ1oOfXLz/gltpn5dG0tDYD1uu4/PWyDw96Bf3WhWOnWhWg4GZKpAY//VoLgbpZXyttVae YUveslOru2oYNeLIa2XqPqStp4eizDN3LffHFwdvQzjSEubYsZfFbPzhaynTxs6436MN42xx21M p5dZkudSH1soREFsVebn7hNZeOa1GN4MMUG8TJxSBb3Vczf3PrcvXoTJbLAxLvKAWMUtByZVutP ZghSK4TELfIBb3nAZuK7ThQ4T1yTumERSa9zNXvNpwWwsmBPzpnhHYAvh4M/irJKWZkKGkqxHwe ZqEWzHpANxLc67s6N/NvZ4TyyBkoQT58M4bSjanacnzNUDMqNjoqC9U X-Received: by 2002:a05:620a:2a01:b0:92e:6c15:24cd with SMTP id af79cd13be357-939ea096ac1mr525489385a.15.1789135826040; Fri, 11 Sep 2026 07:10:26 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939f19f94a1sm128350685a.42.2026.09.11.07.10.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:25 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:52 +0000 Subject: [PATCH RFC v2 14/15] rcu-tasks: Retire switched-out tasks with no trampoline nesting at scan time Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-14-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=3915; i=josef@toxicpanda.com; h=from:subject:message-id; bh=CHfEJf4XyY3zn8RyomjOM2Uc3mVFT7MUycrwjg2RabY=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QCsPahyd7Oix02uJZsqv3hNKRWBYJbR9ZWsmEzIdYJha+M3Xp7J5xDeZFd/3etSxBmrSG0gewnR pnh9QNzgnXwY= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA The previous patch lets a task report its own quiescent state when it is preempted with rcu_tramp_nesting =3D=3D 0, but a task that was preempted before the grace period started and simply has not run since is still listed as a holdout until it next passes through __schedule(). As Paul pointed out, the grace-period kthread can settle that case itself. For a task that is switched out, task_call_func() pins it and !task_curr() tells us it left the CPU through __schedule(), whose locking orders its last rcu_tramp_nesting and rcu_tasks_irq_ip updates before ours; a task preempted from irq exit inside trampoline text has the count held non-zero across the switch by irqentry_preempt(). So a pinned, not-running task with a zero count is neither in nor called from a trampoline, and with rcu_tasks_irq_ip_holds() also clear (the kprobe jump window, which can open after the task was switched out) it is quiescent now, whether or not it ever runs again. Check that in rcu_tasks_pertask() so such tasks never become holdouts, and in check_holdout_task() so they are retired on the next scan. This is the only remote reader of the count, and it only reads it for a pinned, switched-out task, so the trampoline-side increment and decrement stay plain on every preemption model, PREEMPT_RT included. Only under CONFIG_RCU_TASKS_PREEMPT_QS; other architectures are unchanged. Suggested-by: Paul E. McKenney Assisted-by: LLM Signed-off-by: Josef Bacik --- kernel/rcu/tasks.h | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index bab08a666dc0..ba432bd922e2 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1014,10 +1014,39 @@ static bool rcu_tasks_is_holdout(struct task_struct= *t) return true; } =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +/* task_call_func() callback: is @t switched out with no trampoline in pla= y? */ +static int rcu_tasks_switched_out_clean(struct task_struct *t, void *arg) +{ + /* + * With @t pinned, !task_curr() means it last left the CPU through + * __schedule(), so its rcu_tramp_nesting and rcu_tasks_irq_ip are + * stable and ordered before the rq lock we hold. A task preempted + * from irq exit inside trampoline text has the count held non-zero + * across the switch by irqentry_preempt(), so zero here means neither + * in nor called from a trampoline; rcu_tasks_irq_ip_holds() covers the + * one case that can become true after the task was switched out (a + * kprobe jump-optimization window). Both clear: already quiescent, + * whether or not it ever runs again. + */ + return !task_curr(t) && !READ_ONCE(t->rcu_tramp_nesting) && + !rcu_tasks_irq_ip_holds(t); +} + +/* Is @t, right now, switched out somewhere that is a quiescent state? */ +static bool rcu_tasks_preempted_qs(struct task_struct *t) +{ + return task_call_func(t, rcu_tasks_switched_out_clean, NULL); +} +#else +static bool rcu_tasks_preempted_qs(struct task_struct *t) { return false; } +#endif + /* Per-task initial processing. */ static void rcu_tasks_pertask(struct task_struct *t, struct list_head *hop) { - if (t !=3D current && rcu_tasks_is_holdout(t)) { + if (t !=3D current && rcu_tasks_is_holdout(t) && + !rcu_tasks_preempted_qs(t)) { get_task_struct(t); t->rcu_tasks_nvcsw =3D READ_ONCE(t->nvcsw); WRITE_ONCE(t->rcu_tasks_holdout, true); @@ -1181,6 +1210,7 @@ static void check_holdout_task(struct task_struct *t, if (!READ_ONCE(t->rcu_tasks_holdout) || t->rcu_tasks_nvcsw !=3D READ_ONCE(t->nvcsw) || !rcu_tasks_is_holdout(t) || + rcu_tasks_preempted_qs(t) || (IS_ENABLED(CONFIG_NO_HZ_FULL) && !is_idle_task(t) && READ_ONCE(t->rcu_tasks_idle_cpu) >=3D 0)) { WRITE_ONCE(t->rcu_tasks_holdout, false); --=20 2.55.0 From nobody Thu Sep 24 20:23:20 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B850328B7F for ; Fri, 11 Sep 2026 14:10:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135845; cv=none; b=aIthOWeu+71RaRCeahj2UCTpJLszbevtpOgWLnxKZqKAOIJYbBzVxSBHd/dWMhIzaCnU5QaaK4F9XQu/mcKVorPyj8v8UJQOHAmSc4+q/E9C/cISy/ysqLc+DKmyCpKkatzPZkqN4Dak+BXOoxYhpUHGyxW2W+Jl58nYXGMDRlA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135845; c=relaxed/simple; bh=K9+ZKIgsGRgsc959M4tS0rMKOB/eZlOXTQ8Ho0gK2m4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=V4eL5g276WlbKBcTPRHZ91z71N8+WjbxppyrjhjT//4GEOZPBGnOT1QaATZ2KtNjbUa5BPHJ1FOEbOhhl6kzjt1N2of6pRArah0PzABKDoZDWIqIXvinziar5uYrSQPr/KwzQSFhK/MxK1bcPKeJxPevxTLuZ7E22NX5mQSAE5E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=DlojXtNQ; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="DlojXtNQ" Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-52fb76543e1so4032561cf.0 for ; Fri, 11 Sep 2026 07:10:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135842; x=1789740642; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qrCxqh0lkRJGK/msNx4sQvIno2OGGDXwsMjyZ+inPRg=; b=DlojXtNQUMI+4RY2899Vd4IPmVGYjdfubWPeJfZpIUvjJBU3wym3CQSinEvyvDr9pK 082SvIG7BLzZLFY79cZL8B70om+Olx3fXxpAcQQR1qCc591L7hzuznE4vhfcDug9oYzN Nr2s2ZDFpiRGrI4U6t3n8BBycDUuL98yWwZaNenafr0YfxKl8p5iZtwwD+H5tGylC0ct r0vS+8Sfrp3pA0mXnjnnJCktcs+aqWnZCuT0uONmPA/cwW3Lx0yD3xB0L3dhCTO7sASw wE1kkzSkDG1EJ1nLU/KszNnGOj1bg3TQDvvf/zzzO5i6M4HjgFs7HaRiQ3jZqZRXMvvx Cg3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135842; x=1789740642; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qrCxqh0lkRJGK/msNx4sQvIno2OGGDXwsMjyZ+inPRg=; b=od2fDcNdS5FqJ2/jvJCneFGPNnNiivnQg1bB6oigyMjxsfS+csWzkfyRbfijRr6Pg8 5hRVUGLY8Um4W0QnrCTepiHiKCD4vkQgQ3hvyZRWYx04bjDtY/sATL7i77fUBI+TyVVo HQ8fd/Koje6qTcm+grliwj9Kpzx/o7nucw+c+aQuHRuBk1E9t2X/zJaP7T08vrsygl3d rMMADBRA9n3MPQiHEL0i7rJidN21cxSwXeSfQWN8YUyx9v9rTtds4ezQ2uqam8+BhJA8 lh+qVFPittR49pxvctg+ymVGjeLRjrcvT0zxa3qCr/Ad5DSGH/2rDqb8RwTTI4eOtnAL /Z/A== X-Forwarded-Encrypted: i=1; AKwUvBxwRGD9k4RXQs78eDFowfzUalYS+ZF8bud27UB+qH9AJm1QFdAZoW4t7mUlC5ig8gJWTzOf1Ay6edz2wyY=@vger.kernel.org X-Gm-Message-State: AFuF++mFoHxF88Re5p8P3fzJwjC9ZXI5CLz22LwhUR8uWWx1VFKqXEEK CevVFkCNHiXrsIuAU7xpObVICSkLCCJWjioACBd1WmAt7pVFXP2hLxQhd1VVtmnM/ro= X-Gm-Gg: AYBFou0hsXFdisi4AnCbN1f6pq02HLxiEgoJ2yENRnpEHgGbkROCE/eS9LXOcW7jG7X fOQ+nBUQa2sFA+HpyWaRef1itU8aKNHsrNeq1AhC+RO50WyhVpws/GUqEM0H45rHs3i8zOJs20n oZgWzyrKCU44wii6YMl4BaNHOnm5b67q6lgeHvHmnzVgO6UTw3BqK0PwtDLSnXzGVAEzORt1hnD yCXgNbsTo+LubtUNYWWaoMz5lT15hJ8MntkvyuaGbMfwfCsCvbXoTFsu61lt9W8x9VUDA64qG5B z0FQUna/+1S1MfUTWl9mdwvGenF1jJhi1FVfgBHrenKQgM2E1uN/xbSE7weSjn67bu2TDjmA5bc T3uYx8gMwReIrASGQH8ifofHXpMvdh8CxuXtJJUOL0TUyLekEuEHKLD4ydI1c72ZAlBPADWnpGn lhYywlnHyN6hMu9eQAIb795T1TTbf1EqAN5C14dsoGKDoI69cmtFgCKkNNKtghGoHUiTTV7NjVq 6c/tq8vtSWTCy2GLwDpQzVCx+46cStdYO+7fjH8HI9OP/z4Re6UtWjI X-Received: by 2002:a05:622a:53c5:b0:51c:b98c:f772 with SMTP id d75a77b69052e-530c86f6195mr51379741cf.15.1789135827987; Fri, 11 Sep 2026 07:10:27 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca460a32sm20960711cf.9.2026.09.11.07.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:27 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:53 +0000 Subject: [PATCH RFC v2 15/15] rcu-tasks: Kick running holdouts through the scheduler Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-15-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=1790; i=josef@toxicpanda.com; h=from:subject:message-id; bh=K9+ZKIgsGRgsc959M4tS0rMKOB/eZlOXTQ8Ho0gK2m4=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QFxu9w6oKo935Kq6bYpzSj4sd/pud6qAfpTImYrmpDnci7BPIoYh09XWotebu7q/uE6z/VGg6k/ nk0bYtqCcqAg= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA A holdout that is running on a CPU with nothing else runnable is only preempted if the tick acts on rcu_request_urgent_qs_task()'s flag, and there may be no tick. Now that a preemption outside a trampoline is a quiescent state, have check_holdout_task() call resched_cpu() on a running holdout as well, so the scheduler IPIs it and it goes through __schedule() and reports (or, if it is inside a trampoline, does not report) its own state with purely local ordering. Nothing reads a running task's rcu_tramp_nesting remotely. Only under CONFIG_RCU_TASKS_PREEMPT_QS; other architectures are unchanged. Suggested-by: Paul E. McKenney Assisted-by: LLM Signed-off-by: Josef Bacik --- kernel/rcu/tasks.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index ba432bd922e2..02d2592ab7a3 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1038,8 +1038,18 @@ static bool rcu_tasks_preempted_qs(struct task_struc= t *t) { return task_call_func(t, rcu_tasks_switched_out_clean, NULL); } + +/* Make a running holdout pass through __schedule() soon, tick or no tick.= */ +static void rcu_tasks_kick_running(struct task_struct *t) +{ + int cpu =3D task_cpu(t); + + if (task_curr(t) && cpu_online(cpu)) + resched_cpu(cpu); +} #else static bool rcu_tasks_preempted_qs(struct task_struct *t) { return false; } +static void rcu_tasks_kick_running(struct task_struct *t) { } #endif =20 /* Per-task initial processing. */ @@ -1219,6 +1229,7 @@ static void check_holdout_task(struct task_struct *t, return; } rcu_request_urgent_qs_task(t); + rcu_tasks_kick_running(t); if (!needreport) return; if (*firstreport) { --=20 2.55.0