From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066280; cv=none; d=zohomail.com; s=zohoarc; b=apiQnlsDd0QbFELyEUaDfif/1qeSSxGBsfSVjMTrQqI/jn+ucrMfQs/ajNTrQOeb87V6DrnWlcUfN3oW0I4SfrfAP5ZBNmKO7oXQ0uZ5/aFxzM01kAePvqsZN/rgWIzujaaYV0halyX+4azYgCwpS4OY5pHYfoUyRcsfVXdRFpg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066280; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=AwJvzjOst2er6K+UDT8eDw5lj3PdpLVthroXGjj7dNM=; b=cLE+eFALrrvbFhizBsYxHGHbeMpYX+mztDCjyjBFgl0iAbsLWcBQ2mjz8dK/rqzKIcR2N4e9Xr7I76sIhnOf+Hs7N14LAgBT9kbHdyEacvw4uitoOYHYwCt90YJ+Uob3KkhUZbMoVAUU52uqlfOoWVivWosfyXNiT3DCDQ/4ap4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066280983314.732477378437; Thu, 10 Sep 2026 11:51:20 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415149.1644642 (Exim 4.92) (envelope-from ) id 1x4jrM-0004Wm-Fn; Thu, 10 Sep 2026 18:50:56 +0000 Received: by outflank-mailman (output) from mailman id 1415149.1644642; Thu, 10 Sep 2026 18:50:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrM-0004Wf-CZ; Thu, 10 Sep 2026 18:50:56 +0000 Received: by outflank-mailman (input) for mailman id 1415149; Thu, 10 Sep 2026 18:50:54 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrK-0004WP-PT for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:50:54 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrK-00GcVR-6S for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:50:54 +0200 Received: from [10.42.69.6] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fbec-bab6-0a2a0a5309dd-0a2a4506e7e0-40 for ; Thu, 10 Sep 2026 20:50:54 +0200 Received: from [74.125.230.140] (helo=mail-qv2-f12.google.com) by tlsNG-16d1c6.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc0d-195a-0a2a45060019-4a7de68ca226-3 for ; Thu, 10 Sep 2026 20:50:54 +0200 Received: by mail-qv2-f12.google.com with SMTP id 6a1803df08f44-90cdfc9b6ebso282126d6.2 for ; Thu, 10 Sep 2026 11:50:53 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca45b61dsm239421cf.6.2026.09.10.11.50.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:50:51 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066252; x=1789671052; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=AwJvzjOst2er6K+UDT8eDw5lj3PdpLVthroXGjj7dNM=; b=PBu+rAgA+SRJuxl4EK9t6NvRu4M4sZV5P5sauLJDEzhZKuXE5i1/xq+yC4LiZhkK2N U1sx8IkgrfsGMrCxYlhRxIIjCoUctbOtcxkLY+Hthys74dYcel0SUWGqFWjyReiyXfeE K7BLpafsbM9RDlpBTeixuMveUj0PLES6OtVTunqqyG7gBup66j2GfBsh/ao46DjT728w jwWI0BnL0ysw4ct6Ih7iybU+5u0/GM/ZZdhCy5wKNc9v2Pj+pw4/fZHp5KxZlfB7yLjQ KAqvxNXb9gKu271LM0GzKPjcN+/0YPGujrnB5OLrYU1PPx7Z0y0P3yFHtQnB0QapEjRk NgDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066252; x=1789671052; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=AwJvzjOst2er6K+UDT8eDw5lj3PdpLVthroXGjj7dNM=; b=jo3YMgzuzYyFlfLJmDMtlDyHofTfJbdQXMH8DGq83Wo6H5lsnGSYfWdEZtfUg3zARD pH6mTRM0QVMNIo6UHJbk1mr1kZ8cEMVsa3OGHTrlGnlwdfBaDYVyVbzXWNdm6Hcyi7UU /2lPTFJtOwLw91PE0R+NJuPObNiITDTE4rVXIecbNpgz0Xu+sDoQJ/l4KgTYaZmoNDBg 6MR+ESqbPZDKSExOxfgLOSlZATnY9z+X/hejqWm4xqosG3eUyEY0ZW7Qq8I8Yrc2Lnmz UbJJSSrAdrH4L+cJq+be1SA1omGt5MR7MEJBLk3bCV8HIHaeRkOAERLRKVqcWry7iZA/ ou9A== X-Forwarded-Encrypted: i=1; AKwUvBx517ID/pJYa1ijc55h2DD0d/bLX3yC/ZHYhrSzzXHYWsOotL7P/mW0F/ZxrHxv5HLEszg4fS0vAok=@lists.xenproject.org X-Gm-Message-State: AFuF++mPXVlD8l6amng25G9F1X0yVmlW+oUzDxsMcV2tPZOn4LU+ag5Z aSPhMhqJLT+gIGIPUvwhMhpGGeEBZ1Mb6FNHqS7As1lBO20w+LygQWMpFFPIXLXY/TE= X-Gm-Gg: AYBFou2nZe8U5RL3j+p9D2BFEsHjppdHu9AMRNm3r62VnqSmKUOGn1OrqsrW5MZJlK8 QcAtvP/gC2xERLBsPwQC1XGaxV2B3LORrRnMywCCsviXNp3+30bMd9M2xdfO6iRa09rWwebdRZ5 qihfYtJPgAiMyFlDpQE25DCiYGYVi0QYcRiXlbkCKK3gMEamNwSuangtnNJpAa+5nH37o0Ufb2Z vmIfam/+2qFxdY3g8WseOFRY7bmAY6XDCVaSIIggnd7kEn82emE3s5LQHGzBXRtl1JqlYQWz1il FNjO/PXZ4pNQHd/tjKaqgG3H3CsKOCleNSCRYAjnV0VHh5AKRDPp1p92mNBhps6xG3EojST7B6C qETLBFPyrjQkHuF1Y2JBFHtQqQ79WF5wUOoxW5HI4Q89iVcrzbRTNOtQpezqSXPdLSh8ifFh0Bp DWCgctRzRZ621Y0bhOHvaepuarbMtn/ScsZyT5gHM+v5q8JiGH2WeJlVo1pTfuQLops0Rgry0dj GIUoeRgIaldknH1+FWRBsYOWMTh+MdRxUOqmUKKqULux8CaOqOrKgYCSg== X-Received: by 2002:a05:622a:1b8b:b0:530:5ed1:561a with SMTP id d75a77b69052e-530c876e50amr12423551cf.43.1789066252311; Thu, 10 Sep 2026 11:50:52 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:24 +0000 Subject: [PATCH RFC 01/13] rcu-tasks: Add per-task trampoline nesting count MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-1-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-16d1c6/1789066254-F580D77B-06DA44E2/0/0 X-purgate-type: clean X-purgate-size: 6292 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066283269158500 Tasks RCU exists so that ftrace, BPF and kprobes can free trampoline text once no task can still be executing in it. Today the only way a task tells Tasks RCU "I am not in a trampoline" is a voluntary context switch, so a preempted task is always assumed to be inside one. Add task_struct::rcu_tramp_nesting so that trampolines can say so directly: a trampoline increments it before calling out and decrements it before returning, and while it is non-zero the task must not be treated as Tasks-RCU quiescent. Provide rcu_tasks_trampoline_enter() and rcu_tasks_trampoline_exit() for C users, report the count in the Tasks RCU stall output, and, under CONFIG_PROVE_RCU, assert that it is zero on every return to userspace since no task can legitimately reach userspace with a trampoline on its stack. Only current ever writes the count and every nested user (interrupts running their own trampolines) is balanced, so plain accesses suffice. Nothing increments the count and nothing consults it for quiescent-state decisions yet; both come in later patches. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/irq-entry-common.h | 2 ++ include/linux/rcupdate.h | 37 +++++++++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + kernel/fork.c | 1 + kernel/rcu/tasks.h | 3 ++- 5 files changed, 43 insertions(+), 1 deletion(-) diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 0bb6c03481fa..8da571622000 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validat= e(void) { /* Ensure that kernel state is sane for a return to userspace */ kmap_assert_nomap(); + rcu_tasks_trampoline_assert_none(); lockdep_assert_irqs_disabled(); lockdep_sys_exit(); } diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 44c07a66edff..b5c666c82479 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -180,6 +180,37 @@ static inline void rcu_nocb_flush_deferred_wakeup(void= ) { } #ifdef CONFIG_TASKS_RCU_GENERIC =20 # ifdef CONFIG_TASKS_RCU + +/* + * Trampoline nesting: dynamically allocated text (ftrace trampolines, BPF + * trampoline images, kprobe optinsn slots) that relies on Tasks RCU for i= ts + * lifetime brackets itself with an increment/decrement of + * current->rcu_tramp_nesting. While the count is non-zero the task is in= side, + * or was called from, such text and an involuntary context switch must no= t be + * treated as a Tasks RCU quiescent state. + * + * Only current writes the count and only current (or an interrupt on the = same + * CPU) reads it, so plain accesses suffice. + */ +static __always_inline void rcu_tasks_trampoline_enter(void) +{ + current->rcu_tramp_nesting++; + barrier(); +} + +static __always_inline void rcu_tasks_trampoline_exit(void) +{ + barrier(); + current->rcu_tramp_nesting--; +} + +/* A task must never reach userspace with a trampoline on its stack. */ +static __always_inline void rcu_tasks_trampoline_assert_none(void) +{ + if (IS_ENABLED(CONFIG_PROVE_RCU)) + WARN_ON_ONCE(current->rcu_tramp_nesting); +} + # define rcu_tasks_classic_qs(t, preempt) \ do { \ if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ @@ -192,6 +223,9 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); # define rcu_tasks_classic_qs(t, preempt) do { } while (0) # define call_rcu_tasks call_rcu # define synchronize_rcu_tasks synchronize_rcu +static inline void rcu_tasks_trampoline_enter(void) { } +static inline void rcu_tasks_trampoline_exit(void) { } +static inline void rcu_tasks_trampoline_assert_none(void) { } # endif =20 #define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) @@ -208,6 +242,9 @@ void exit_tasks_rcu_finish(void); #define rcu_tasks_classic_qs(t, preempt) do { } while (0) #define rcu_tasks_qs(t, preempt) do { } while (0) #define rcu_note_voluntary_context_switch(t) do { } while (0) +static inline void rcu_tasks_trampoline_enter(void) { } +static inline void rcu_tasks_trampoline_exit(void) { } +static inline void rcu_tasks_trampoline_assert_none(void) { } #define call_rcu_tasks call_rcu #define synchronize_rcu_tasks synchronize_rcu static inline void exit_tasks_rcu_start(void) { } diff --git a/include/linux/sched.h b/include/linux/sched.h index 8b3d47a325cc..d2e7b1b3c9d2 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -956,6 +956,7 @@ struct task_struct { unsigned long rcu_tasks_nvcsw; u8 rcu_tasks_holdout; u8 rcu_tasks_idx; + int rcu_tramp_nesting; int rcu_tasks_idle_cpu; struct list_head rcu_tasks_holdout_list; int rcu_tasks_exit_cpu; diff --git a/kernel/fork.c b/kernel/fork.c index 416758c8a3d4..cfe3a8e53fbd 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1869,6 +1869,7 @@ static inline void rcu_copy_process(struct task_struc= t *p) #endif /* #ifdef CONFIG_PREEMPT_RCU */ #ifdef CONFIG_TASKS_RCU p->rcu_tasks_holdout =3D false; + p->rcu_tramp_nesting =3D 0; INIT_LIST_HEAD(&p->rcu_tasks_holdout_list); p->rcu_tasks_idle_cpu =3D -1; INIT_LIST_HEAD(&p->rcu_tasks_exit_list); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 627295396cd9..1662ba18bf34 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1113,10 +1113,11 @@ static void check_holdout_task(struct task_struct *= t, *firstreport =3D false; } cpu =3D task_cpu(t); - pr_alert("%p: %c%c nvcsw: %lu/%lu holdout: %d idle_cpu: %d/%d\n", + pr_alert("%p: %c%c nvcsw: %lu/%lu holdout: %d tramp_nesting: %d idle_cpu:= %d/%d\n", t, ".I"[is_idle_task(t)], "N."[cpu < 0 || !tick_nohz_full_cpu(cpu)], t->rcu_tasks_nvcsw, t->nvcsw, t->rcu_tasks_holdout, + data_race(t->rcu_tramp_nesting), data_race(t->rcu_tasks_idle_cpu), cpu); sched_show_task(t); } --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066272; cv=none; d=zohomail.com; s=zohoarc; b=RCh0vI+U532u+O+k7Xnb8zDrXAk93bfoPWv+TGAniFUi49ucSEoObI9ybLifjpERc8fvp+ED0ZFBDh3Na3482Ecv6ofhZJq3MKtpGKJK1Q8bp1f8+hVCmoV8OKsqc7At0W7yI0yy7R6FCkqng71xnXtkAxVJCkyChRbXbqzvZa4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066272; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=H7wSWIL0VPD1fdq24NT7SObCwyyogmx/+Qqy+uT9slksIdO5XZvwJRE7DWyPDpN11zdh454O5eEJnGagd4ZywMExLkepLrvSDDwUc6abErQpob5EK5eccJju7JHpCLgfFqFDs2CERGk0/cQ7QWWx2nRdyphfFgbeRBJBQ5a84S8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066272740637.8227095322311; Thu, 10 Sep 2026 11:51:12 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415150.1644650 (Exim 4.92) (envelope-from ) id 1x4jrO-0004kf-PT; Thu, 10 Sep 2026 18:50:58 +0000 Received: by outflank-mailman (output) from mailman id 1415150.1644650; Thu, 10 Sep 2026 18:50:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrO-0004kY-Mp; Thu, 10 Sep 2026 18:50:58 +0000 Received: by outflank-mailman (input) for mailman id 1415150; Thu, 10 Sep 2026 18:50:57 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrN-0004eI-2o for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:50:57 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrM-00GcVR-Fk for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:50:56 +0200 Received: from [10.42.69.1] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fbc3-bab6-0a2a0a5309dd-0a2a4501c7de-46 for ; Thu, 10 Sep 2026 20:50:56 +0200 Received: from [209.85.219.52] (helo=mail-qv1-f52.google.com) by tlsNG-d62444.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc0f-5984-0a2a45010019-d155db34cc92-3 for ; Thu, 10 Sep 2026 20:50:56 +0200 Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-90cc39e06bdso569076d6.0 for ; Thu, 10 Sep 2026 11:50:55 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e8bf9d51sm34965885a.14.2026.09.10.11.50.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:50:53 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066255; x=1789671055; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=qZ4kkEzaVD5EgqYkuvzJR0rh3kr43L38Y9KsooAwlzNbXwSgGYpoVimRX4tD3AHCR3 r7DOLAIbdTC22OQi9+UBQnDMkp9upzu82IMCvOA4eZH2YtP7okowTR5d7Qvv8bBYCOhR iQzoe4cPytufFkpkvBxaaD8cA36sVB0JkiBS0oZdNLK9oTLK0VRCJuj7pwgr8XcwHXFg n5wMgJkZ777AOLa3LNlLBUK33hDSVVE8n6KyH/HSi1IRNgtHlBP93zKGTuqcp6dCqM3E l4tHJXuMzTPhKcF+DaqlzupEnwVNndvNjpRu1o8gN4tMrtMkqb/oP8YZXdnuctFN9/5m +ILQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066255; x=1789671055; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dxBJRgyuHY3W/iJApny4mLX1c4wnn/TYh39J8mu564M=; b=sTNbfU93Mxp54VoEGtB/BKrsUkSoaNMEFMQMmOOV1gtPZb1dJoK12XCK4vKZpWY/Xe Hb9SZKMuJC9PgkOo3M1gOhCkoq6LF+J2/K1LsaTot3/H5FJosBa4WIHAkMzcNTkIRHXY TBudaRcPY2f7O7OwAnSMV7HZYiYAxsPkecL22iqZZfR90AaUL0jWL36juO0PFG+MLCVM zfi9faO+zNJs16aA+AcpNjYRGOKPb7exy3P8ukorQm4Zp7TZHRMcVWRc1OsKTG5/rU7M So8ynfWzy8AXv3iHC0/UZyux4XGubATnlTP8Dcb4yamaHEw0kUUeowuiQC1rLBTSKJy5 ZUrw== X-Forwarded-Encrypted: i=1; AKwUvBzxbHUAJsmEHZfjZTyekggmfIaAuGxwBGdjiZJkoFLZ+Z8zVFnjGpsZ1930pha/2o4FseIAprQ7Tf8=@lists.xenproject.org X-Gm-Message-State: AFuF++l9hA2tKsgG/bIv7x2xiemmRZJ/ZMpm9Reg5ZWg102sTYBWQgqK iZBvqERejyzoU+sxZYXeJp1W1nwiCVNjzbH4cpfJKRH4b+Bwy35s1Px9AreUkWViVVE= X-Gm-Gg: AYBFou1B8ABJWvYDLakBiwvvATzGxYlInAvOxnL41zTb03hGMx4bm/RR+XMJDJ5TxCV 8A/aET1HGsnjp8B/wDQ6M7//cIIfXwwmIrD+4XEBL2qa3wJSbXR4lH2F0+cq8HymALgTDMS1DJw yH9xbnp5LB3gJ1C3IlyI13t7OpFipmiaaeJsozLf0hALeo8cqGfF6F04Zk7CCm+2/G+XDw93xUU w/2QVu0f5wZ+K9kevSHxKaUGTFtvlEnHNGp6CMo/gLvBQkNEwWeF8zK+GUgM7yY/JZ8SuEP2GkI KhXO0twbeoAsx88nklqPqEhS9vwI4fU3emtHPcQhNKkyokZe72sk4IFvciY3cVFQzhMHLsihJUS g7HzfdKA2hbgcNrpMLiWEzoTOHrUe2z4j8k01gPa/qkQgr6UOQOEIdz3oUW6+lknPXcBb1jfj4S 7G8vjj4qzENDycQJBmDfW96mHlqLQlB6AdKOMekt8SHVqexwLuE1afeoS3zoCL5512xLe1Hb0ed ad5bF+SWTojZFy4jQqLKL/LUDeluyWn2GZlDciJMR7gdL45AXdEA5C6 X-Received: by 2002:a05:620a:280d:b0:939:78ab:3c5f with SMTP id af79cd13be357-939ea0137a8mr26249085a.9.1789066254480; Thu, 10 Sep 2026 11:50:54 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:25 +0000 Subject: [PATCH RFC 02/13] entry: Pass pt_regs to irqentry_exit_cond_resched() MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-2-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-d62444/1789066256-C5146757-93FF6A3D/0/0 X-purgate-type: clean X-purgate-size: 4151 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066275790158500 The irq-exit preemption path is about to need the interrupted context's registers to decide whether the preemption may be reported to Tasks RCU as a quiescent state. irqentry_exit_to_kernel_mode_preempt() already has them; hand them down through irqentry_exit_cond_resched(), its PREEMPT_DYNAMIC static-call and static-key variants, and raw_irqentry_exit_cond_resched(). The only caller outside the generic entry code is Xen PV's upcall handler, which has regs as well. No functional change. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/xen/enlighten_pv.c | 2 +- include/linux/irq-entry-common.h | 12 ++++++------ kernel/entry/common.c | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 2c64b388f616..3d85035f5624 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct p= t_regs *regs) =20 inhcall =3D get_and_clear_inhcall(); if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) { - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); instrumentation_end(); restore_inhcall(inhcall); } else { diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 8da571622000..fc04725ae46b 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -348,21 +348,21 @@ typedef struct irqentry_state { * * Conditional reschedule with additional sanity checks. */ -void raw_irqentry_exit_cond_resched(void); +void raw_irqentry_exit_cond_resched(struct pt_regs *regs); =20 #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) #define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched #define irqentry_exit_cond_resched_dynamic_disabled NULL DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); -#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() +#define irqentry_exit_cond_resched(regs) static_call(irqentry_exit_cond_re= sched)(regs) #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void); -#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched() +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs); +#define irqentry_exit_cond_resched(regs) dynamic_irqentry_exit_cond_resche= d(regs) #endif #else /* CONFIG_PREEMPT_DYNAMIC */ -#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() +#define irqentry_exit_cond_resched(regs) raw_irqentry_exit_cond_resched(re= gs) #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 /** @@ -467,7 +467,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt= (struct pt_regs *regs, return; =20 if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); } =20 /** diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e3d381fd3d25..e4acd50bd81a 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -134,7 +134,7 @@ static inline bool arch_irqentry_exit_need_resched(void= ); static inline bool arch_irqentry_exit_need_resched(void) { return true; } #endif =20 -void raw_irqentry_exit_cond_resched(void) +void raw_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!preempt_count()) { /* Sanity check RCU and thread stack */ @@ -150,11 +150,11 @@ void raw_irqentry_exit_cond_resched(void) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resc= hed); #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void) +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) return; - raw_irqentry_exit_cond_resched(); + raw_irqentry_exit_cond_resched(regs); } #endif #endif --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066282; cv=none; d=zohomail.com; s=zohoarc; b=F3nFrVIeNC2Q5eDzgcJYY11T2FjbGLjAwD1Wu5CuW6SYIe1DjuJipgK43W7Xz4VXYKZL3rhHDovIKcQeWT5pguC+FaTmiUeWg5/5RyavPfAZJSBQTNh44iLm8WNSyBAfpmyt+uiCXGUZHS8Ev5xDutLDwO1zABshb6qtiTcKhyE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066282; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=m1fNranstR4PQYCajbNE5u6WXmU8puPx7xBxfPTjbZheJIz91222bY6Q4yLB/dc/4ms0BV2ml/mORC4fLzw0sssS9+tYrAUGcjjcR5wE2pYFsUBwrNaz38CfHS0SrMXbt8rTuX/vr1VpxpLinLI/sddiomw4HiB1XLqpG/eRzO4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066282143291.89635747214504; Thu, 10 Sep 2026 11:51:22 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415151.1644660 (Exim 4.92) (envelope-from ) id 1x4jrR-0004zP-0N; Thu, 10 Sep 2026 18:51:01 +0000 Received: by outflank-mailman (output) from mailman id 1415151.1644660; Thu, 10 Sep 2026 18:51:00 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrQ-0004zF-Tu; Thu, 10 Sep 2026 18:51:00 +0000 Received: by outflank-mailman (input) for mailman id 1415151; Thu, 10 Sep 2026 18:50:59 +0000 Received: from mx.expurgate.net ([194.145.224.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrP-0004kl-FF for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:50:59 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrO-00BoS1-Cm for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:50:58 +0200 Received: from [10.42.69.10] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fbf1-e002-0a2a0a5209dd-0a2a450a87e0-40 for ; Thu, 10 Sep 2026 20:50:58 +0200 Received: from [74.125.230.204] (helo=mail-qk2-f12.google.com) by tlsNG-4011c0.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc11-f2d2-0a2a450a0019-4a7de6ccccad-3 for ; Thu, 10 Sep 2026 20:50:58 +0200 Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-52fb769ca02so2300301cf.0 for ; Thu, 10 Sep 2026 11:50:57 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f5e880csm2372196d6.49.2026.09.10.11.50.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:50:55 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066257; x=1789671057; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=NxGCuF9TxJR+E2COmaVZM7zVngkAJEnkCFSxGZZnEPX9z3qcj4pYNuE5no3CkArp63 o3DeEcYrarqrAo8TyOuxll4Qf0ENKCY+A7GkEIpB5zBL90C5644Zgg39jpcu8or2mWod stMWWdenxCdBW2D+bfikZ4nmpEct97GiKH8LCaLDQhxCutnl4SN4rIabhFJpB/wI5jyB bjX2Ua4acmVf5+pW1qV5MeOPYKUr4F5wR/i2W3c52SWC/JjVih4xA187u1GrOpmp/6jc 7GzLXF5OmdoFwTSL0dzbZnEKQt5t897h2f9iTH4sFr4BDCiTccKWf6Yzp2/qq8KGvxXC UVng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066257; x=1789671057; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Vr/1/6kD89CuecwWXrcFDfjNkcZoJvv75czWOOFkQAs=; b=Xy0MMEn5blLMOj6McyQPDnCMQsNjjcX3LyjEoyqoXbS4V9JuvW3JT5MlayHVYAGbKt XplM95r9AfiCg705NPxIvNXr9fupbo98lX/GpRMnMgeNjs92iNdurVn9HQ84IOEbkIsI fXIRlp37ADK+KQeIqk4C42A4kHKViJHAj16AmJa/9If+1/4HVtjgWddxDE+tfFCgPrUG cW9VuO8kVlkUE87oAPVACobVHXkqrXllEjxISdkqiGgYzvxJ+TYnZA55p4HbQY3ANnKg H73DeqmNh00WC4PkGDVoDkhFV4t2WgLtHaSOUBrvZTolBpBe3HUJn11obNq4AqAJCymh oMSA== X-Forwarded-Encrypted: i=1; AKwUvByH6SJ7xdhRdj/1I2HKpcLJospR28iRKTuzNRXiLeBbqZ55mwG0S7Zzxakp1IFB0C7/h/mNjpQVy6s=@lists.xenproject.org X-Gm-Message-State: AFuF++nqjJJzs6WoJRIYJyRIDkYYl9EHz6N3SnK8gg58p6G2wvDaKOxm avjfnCsXjVbnOM5XMXSh7sat+8Tmy/n34qs5eV7jbMm+10VloK5QTryIbhEUw0Z4lR8= X-Gm-Gg: AYBFou2AeaPNobDsyg6MtAgpej6IcpGnOxtOVVSok2czbQtdrYrIMgti4tXVZNeSt05 104AvSiS7gjhp/c5dJ/w6z5uPObTuhfnXjVbFJry15yeR0tYUEwHMeOC+n4iPRUcvGwQ38p5Xi/ 3Bc0YUCw0EEcaoyIkEmPqLWxTdayHaVhhYNwAYXJcbumOQCP4EEzFqqCMkwfipkbDgl0rL5NzFE D7OeLmpNUyhzQZoZG8tjddNFyTBaU9XX7T1ttZp46kmWs0/pXaqI38jOM66FGS08oRfgQYJyxs5 w1+BBV2TqTjLDR3sArxOZzN+5n482A0c1lQvartHweT0SDdWf4y+NXBSd+x7MJ29eQI8OTu8UjI DIAjNuaRfRAW5CihhuxJdZBJIL9YA84Oh0SJMIVI46cZTYdCSxBPzgfxjkVW/6zuJSCbdk0n20O xOm8FQ3iIlQJhsnmxE2c2LcGYkONg2sxTl4Q2zXtlXEt4FV73n+plJ1osJCBTXMIgiaFSGZq+Ig jYrdL95Hfl2+l5+4f1diKnh3PNYSVn++Jz5ZG11kPkKGK1hmUvdb7g+go+bFQzNALU= X-Received: by 2002:ac8:5a87:0:b0:530:8b38:b9f4 with SMTP id d75a77b69052e-530c86f329fmr16033471cf.37.1789066256481; Thu, 10 Sep 2026 11:50:56 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:26 +0000 Subject: [PATCH RFC 03/13] rcu-tasks: Hold trampoline nesting across irq-exit preemption in trampoline text MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-3-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-4011c0/1789066258-51EC2CFC-E3177B4E/0/0 X-purgate-type: clean X-purgate-size: 9199 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066283295158500 A trampoline's own rcu_tramp_nesting increment and decrement live inside the trampoline, so there is a window of a few instructions on entry and exit where the count is zero while the CPU is executing trampoline text (or text on the way into one, such as a static ftrace stub holding a direct-call target). In that window the task has not called out, so it can only be preempted from an interrupt, and the interrupted instruction pointer identifies where it is. Add rcu_tasks_ip_in_trampoline(), which treats any IP outside core kernel and module text as potentially Tasks-RCU-protected (ftrace trampolines, BPF images and programs, kprobe slots are all dynamically allocated text; is_ftrace_trampoline() and friends are deliberately not used because text being torn down may already be unregistered from them while a task still stands on it), plus a __weak arch_rcu_tasks_ip_in_trampoline() for core text an architecture needs to flag. On irq-exit preemption, if the IP matches, hold the count elevated across preempt_schedule_irq(). Introduce ARCH_HAS_RCU_TASKS_PREEMPT_QS / RCU_TASKS_PREEMPT_QS to gate this; no architecture selects it yet, so the check compiles away and there is no functional change. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/rcupdate.h | 17 +++++++++++++++++ kernel/entry/common.c | 23 ++++++++++++++++++++++- kernel/rcu/Kconfig | 10 ++++++++++ kernel/rcu/tasks.h | 38 ++++++++++++++++++++++++++++++++++++++ kernel/rcu/update.c | 2 ++ 5 files changed, 89 insertions(+), 1 deletion(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index b5c666c82479..0a408e36ea15 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -173,6 +173,9 @@ static inline void rcu_nocb_flush_deferred_wakeup(void)= { } =20 #endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */ =20 +/* Arch hook for rcu_tasks_ip_in_trampoline(); see kernel/rcu/tasks.h. */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); + /* * Note a quasi-voluntary context switch for RCU-tasks's benefit. * This is a macro rather than an inline function to avoid #include hell. @@ -189,6 +192,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(void= ) { } * or was called from, such text and an involuntary context switch must no= t be * treated as a Tasks RCU quiescent state. * + * The increment and decrement themselves live inside the trampoline, so t= here + * is a window of a few instructions at entry (before the increment) and e= xit + * (after the decrement) where the count is zero but the CPU is executing + * trampoline text, or text on the way into one (a static ftrace stub or a + * return thunk holding the trampoline's address). In that window the task + * cannot be preempted synchronously, only from an interrupt, so the irq-e= xit + * preemption path covers it by checking regs->ip with + * rcu_tasks_ip_in_trampoline() and holding the count elevated across + * preempt_schedule_irq() when it matches. + * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. */ @@ -211,6 +224,8 @@ static __always_inline void rcu_tasks_trampoline_assert= _none(void) WARN_ON_ONCE(current->rcu_tramp_nesting); } =20 +bool rcu_tasks_ip_in_trampoline(unsigned long ip); + # define rcu_tasks_classic_qs(t, preempt) \ do { \ if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ @@ -226,6 +241,7 @@ void rcu_tasks_torture_stats_print(char *tt, char *tf); static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } +static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } # endif =20 #define rcu_tasks_qs(t, preempt) rcu_tasks_classic_qs((t), (preempt)) @@ -245,6 +261,7 @@ void exit_tasks_rcu_finish(void); static inline void rcu_tasks_trampoline_enter(void) { } static inline void rcu_tasks_trampoline_exit(void) { } static inline void rcu_tasks_trampoline_assert_none(void) { } +static inline bool rcu_tasks_ip_in_trampoline(unsigned long ip) { return f= alse; } #define call_rcu_tasks call_rcu #define synchronize_rcu_tasks synchronize_rcu static inline void exit_tasks_rcu_start(void) { } diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e4acd50bd81a..cd3feaca6420 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -134,6 +134,27 @@ static inline bool arch_irqentry_exit_need_resched(voi= d); static inline bool arch_irqentry_exit_need_resched(void) { return true; } #endif =20 +/* + * Preempt the interrupted kernel context. If the interrupt landed in text + * that may be a Tasks-RCU-protected trampoline (see + * rcu_tasks_trampoline_enter()), hold current->rcu_tramp_nesting elevated + * across the context switch so that it is not mistaken for a Tasks RCU + * quiescent state. This closes the few-instruction windows at trampoline + * entry/exit where the trampoline's own increment has not yet run or its + * decrement already has. + */ +static void irqentry_preempt(struct pt_regs *regs) +{ + bool in_tramp =3D IS_ENABLED(CONFIG_RCU_TASKS_PREEMPT_QS) && + rcu_tasks_ip_in_trampoline(instruction_pointer(regs)); + + if (in_tramp) + rcu_tasks_trampoline_enter(); + preempt_schedule_irq(); + if (in_tramp) + rcu_tasks_trampoline_exit(); +} + void raw_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!preempt_count()) { @@ -142,7 +163,7 @@ void raw_irqentry_exit_cond_resched(struct pt_regs *reg= s) if (IS_ENABLED(CONFIG_DEBUG_ENTRY)) WARN_ON_ONCE(!on_thread_stack()); if (need_resched() && arch_irqentry_exit_need_resched()) - preempt_schedule_irq(); + irqentry_preempt(regs); } } #ifdef CONFIG_PREEMPT_DYNAMIC diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 332df7a7a634..999f8228a13d 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -107,6 +107,16 @@ config TASKS_RCU default NEED_TASKS_RCU && PREEMPTION select IRQ_WORK =20 +# Selected by architectures whose ftrace, BPF and kprobe trampolines maint= ain +# current->rcu_tramp_nesting and which use the generic irqentry code, so t= hat +# a preemption outside any trampoline can be treated as a Tasks RCU +# quiescent state. See rcu_tasks_trampoline_enter(). +config ARCH_HAS_RCU_TASKS_PREEMPT_QS + bool + +config RCU_TASKS_PREEMPT_QS + def_bool TASKS_RCU && ARCH_HAS_RCU_TASKS_PREEMPT_QS && GENERIC_IRQ_ENTRY + config FORCE_TASKS_RUDE_RCU bool "Force selection of Tasks Rude RCU" depends on RCU_EXPERT diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 1662ba18bf34..a801ec4a951b 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1089,6 +1089,44 @@ static void rcu_tasks_postscan(struct list_head *hop) timer_delete_sync(&tasks_rcu_exit_stall_timer); } =20 +/* + * Architectures selecting ARCH_HAS_RCU_TASKS_PREEMPT_QS override this to = flag + * core kernel text that must be treated like a trampoline, e.g. static ft= race + * entry stubs and return thunks that run with a trampoline address in han= d. + */ +bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + return false; +} + +/** + * rcu_tasks_ip_in_trampoline - Could a task interrupted at @ip be a Tasks= RCU reader? + * @ip: interrupted instruction pointer + * + * Called from the irq-exit preemption path with interrupts disabled, to d= ecide + * whether the imminent preemption may be reported as a Tasks RCU quiescent + * state when current->rcu_tramp_nesting is zero. Returns true, meaning "= do + * not report", when @ip is: + * + * - outside static kernel and module text, i.e. possibly in an ftrace + * trampoline, BPF trampoline image or program, kprobe insn/optinsn slo= t or + * other dynamically allocated text whose lifetime Tasks RCU guards. T= his + * deliberately does not consult is_ftrace_trampoline() and friends: te= xt + * being torn down may already be unregistered there while a task still + * stands on it; + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(). + * + * A false positive only defers the quiescent state to the task's next + * context switch. + */ +bool rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + if (core_kernel_text(ip)) + return arch_rcu_tasks_ip_in_trampoline(ip); + return !is_module_text_address(ip); +} +NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); + /* See if tasks are still holding out, complain if so. */ static void check_holdout_task(struct task_struct *t, bool needreport, bool *firstreport) diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index b62735a67884..23be7e97c3b5 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -41,6 +41,8 @@ #include #include #include +#include +#include #include #include #include --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066282; cv=none; d=zohomail.com; s=zohoarc; b=mahjJBwfTiY3xg59/MEZMCPPX3FnAp4pjKShlFmKf9RwrT2oT32Qmcfg3No3+ra8b7sNlcZdY0PpszH47FvS/A/yKCsgYZq/zUUw+GsE2czuZZXWXUHjKwmHeCchmfSJyaadz/NedJ5xE/zGICgPE84qpyI47Mv563pJlhB0+zQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066282; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=aLCJ034DJzI4gEh3LNSHB0+4siN8J5WsqqEdRoq/eTA=; b=Ro4MINqCDzIvaJ5BE0HPfio1ZIEj3mWHW4wZvqC2jYEEFdb71wtSScuUmX9IQ/AJYHqgNJNu7PbCwfcCJgMhXjRWC4Ea9jNj2Itk/eUMOvZpNd88UYMKfslkVgTGdI4kieCBRyVkAdd5xCBIb6RsjZFrUvM0kKxrSL9HgB+ms5Y= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066282955935.5101205895289; Thu, 10 Sep 2026 11:51:22 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415152.1644669 (Exim 4.92) (envelope-from ) id 1x4jrS-0005DS-7Q; Thu, 10 Sep 2026 18:51:02 +0000 Received: by outflank-mailman (output) from mailman id 1415152.1644669; Thu, 10 Sep 2026 18:51:02 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrS-0005DJ-4I; Thu, 10 Sep 2026 18:51:02 +0000 Received: by outflank-mailman (input) for mailman id 1415152; Thu, 10 Sep 2026 18:51:01 +0000 Received: from mx.expurgate.net ([194.145.224.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrQ-0004zA-U2 for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:01 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrQ-00BoS1-Am for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:00 +0200 Received: from [10.42.69.4] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fbd5-e002-0a2a0a5209dd-0a2a4504b836-42 for ; Thu, 10 Sep 2026 20:51:00 +0200 Received: from [209.85.222.174] (helo=mail-qk1-f174.google.com) by tlsNG-ebf023.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc13-b57f-0a2a45040019-d155deaeec8c-3 for ; Thu, 10 Sep 2026 20:51:00 +0200 Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-92e85499ffbso1147785a.0 for ; Thu, 10 Sep 2026 11:50:59 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e807eba7sm47389685a.29.2026.09.10.11.50.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:50:57 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066259; x=1789671059; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=aLCJ034DJzI4gEh3LNSHB0+4siN8J5WsqqEdRoq/eTA=; b=WxQ3hfhb6GZ0qAc8VKLjFNT/T/sLaymAqWoXJTyrD6ltoe3GbJmEnK2pM0FoIvS+gU ZRW/viscNX7K0Rbz2z7+P6YxJ7H1I4DhJzQcplGU78OG4K6eMSU+xMXqMvgYSSlh+0ss Oh6oYK7FLke6Uz7Md5gBfUxusrFrIa6i0foszxiwUTOHZClKVSRUbUvC1+d11u+i+y1i Hvjx+C2ZSPqan/NeTlspOSlTcpnkyoZumyjfM5PGJmaw5qi0gkj1vQOR00lOA6OuTvBQ dhK6BN9Qc3Wlchdc++1LlY1T5X49G3xbQiBdYdLphDwpbz1ASFamQd8Rw/G4TXFCD7bi zBqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066259; x=1789671059; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=aLCJ034DJzI4gEh3LNSHB0+4siN8J5WsqqEdRoq/eTA=; b=jfodkSJdE8UvQUn8BRqsotdbRVR8C6Rn/zJFmovxcNe9CvgttcYdQVJFU6/S5C9yHc 5CFBFKRY4MbeaqomTBWpxr2AnHZvlXC8yJ+ik9vi09nx1sQcbXRDZCuvqqgpUmCEr9/R k5ahjXcj2MUN1KEuGxLJleulg6hV2ziFDhQL1jmnZOGD0CoQcAO771bLhRPknbEUG7LL AtnF6JrxLKLYbhk6rvxQiYyk3NOf+cVnkOgeq6En6TfizhdpRdmor7FDyonQND6aFqoe IOR4zb0t0c82u8MoUJWkYOz6hLxTTKO93GL6JM9uYSQw96chGhIaH36CJLYf9Z3QfR8C bwyQ== X-Forwarded-Encrypted: i=1; AKwUvBwwAS451I2hV6DnFtesynT6nXWqhd4vcb9EB6NqbUuwt8REfcQVWiK2QliPqBwkS/QJ73EJZ8Usf1M=@lists.xenproject.org X-Gm-Message-State: AFuF++k4w6w0ZNablti72nTbe6gctKtGanumT5Usx3x45/QW6OTw6Ezx gMCnAUMTWM9sGN8IZkJN3mwIdVOOfcrSmEIP0JevkK6RUaPTv1Lqtfzt33UEL8SG5SI= X-Gm-Gg: AYBFou1nL2Jcdtykfa1RP/q3/NnkDrpC0ZwP9wUQT85bweHZFOTcEU31DBlTzwXWuHW hYQ76aR4kUavujGfFt7NuYZElWwMJIT3NY+bVNsmYTon3F6NBuDfn/rHX8vMWdvUf5JjxxLUElB tTQ874Yx9CdEvrtNVk7du9pKw9ckhefV7mj4ByZuyEBykvatwdh2Etqw96uDfchfrDEAyAEwxSp bXsRnu2vKoPOnmcAg2j91LyWE/yOH+/PfAdxFHL0cdyYycyDxRFAwzMu+vb6Z9KDZWds6lTZMJy 0BjcVOQF6Dssf5jsv6MaZaIv/hq2NH6n+nsGXTTFgM4j0olnOivAQbOTDvyICPsOrYCI0k6eIJo OulpkmsI756lrSfftqCHlTL4V/JOaD6B6AN6dQIL8ta1IR3brR4yElcz7NtGHeA47uoICanz+22 BgprT5rhvIVxXIUCLCTT8GEIOC/GE9gp1PaKhwrZUDTNOb1S6hIzDerd3K8YxpI1skTc+8XuvH4 uDJuMf3xxjn26rHz7Gz1UXioF5A+9ctbBx66YJVk+PTVvh8wYz1o5Q/iB0gUbm0U3Q= X-Received: by 2002:a05:620a:6191:b0:939:a6e4:2267 with SMTP id af79cd13be357-939ea2bff00mr14491685a.50.1789066258414; Thu, 10 Sep 2026 11:50:58 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:27 +0000 Subject: [PATCH RFC 04/13] kprobes: Let Tasks RCU recognise tasks preempted in an optprobe jump window MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-4-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-ebf023/1789066260-C20D5B50-279DC295/0/0 X-purgate-type: clean X-purgate-size: 5160 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066283270158500 kprobe_optimizer() is the one synchronize_rcu_tasks() user that is not about trampoline text: it waits for tasks that were preempted on an instruction boundary inside the bytes it is about to overwrite with the optimized jump, so that none of them resumes into the middle of the new instruction. Such a task sits in ordinary kernel or module text with rcu_tramp_nesting =3D=3D 0, and can only have got there via an irq-exit preemption. Add kprobe_in_optimized_region(), a lockless and conservative form of get_optimized_kprobe() that reports whether any registered kprobe lies within MAX_OPTIMIZED_LENGTH before the given address regardless of its optimization state, and have rcu_tasks_ip_in_trampoline() consult it so that a task interrupted there keeps holding off the Tasks RCU grace period once preemption becomes a quiescent state. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/kprobes.h | 8 +++++++- include/linux/rcupdate.h | 4 +++- kernel/kprobes.c | 24 ++++++++++++++++++++++++ kernel/rcu/tasks.h | 6 ++++++ 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index e6de7ae55bda..74cc48c04417 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long = addr) } #endif /* !CONFIG_KPROBES */ =20 -#ifndef CONFIG_OPTPROBES +#ifdef CONFIG_OPTPROBES +bool kprobe_in_optimized_region(unsigned long addr); +#else /* !CONFIG_OPTPROBES */ static inline bool is_kprobe_optinsn_slot(unsigned long addr) { return false; } +static inline bool kprobe_in_optimized_region(unsigned long addr) +{ + return false; +} #endif /* !CONFIG_OPTPROBES */ =20 #ifdef CONFIG_KRETPROBES diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 0a408e36ea15..e9afbbb1b061 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -200,7 +200,9 @@ bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); * cannot be preempted synchronously, only from an interrupt, so the irq-e= xit * preemption path covers it by checking regs->ip with * rcu_tasks_ip_in_trampoline() and holding the count elevated across - * preempt_schedule_irq() when it matches. + * preempt_schedule_irq() when it matches. The same check covers the one + * non-trampoline user, kprobe jump optimization, which waits for tasks + * preempted inside the instruction bytes it is about to overwrite. * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 6337da5cab9e..76f146edb0e5 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -511,6 +511,30 @@ static struct kprobe *get_optimized_kprobe(kprobe_opco= de_t *addr) return NULL; } =20 +/** + * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimiz= ed + * kprobe replaces? + * @addr: kernel text address, typically an interrupted instruction pointer + * + * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks = that + * were preempted on an instruction boundary inside the region about to be + * overwritten by the optimized jump; such a task must not report a Tasks = RCU + * quiescent state when it is preempted (see rcu_tasks_ip_in_trampoline()). + * This is the lockless, conservative form of get_optimized_kprobe(): it d= oes + * not care whether the kprobe found is, or ever will be, optimized. May = be + * called from any context with preemption disabled. + */ +bool kprobe_in_optimized_region(unsigned long addr) +{ + int i; + + for (i =3D 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + if (get_kprobe((kprobe_opcode_t *)addr - i)) + return true; + return false; +} +NOKPROBE_SYMBOL(kprobe_in_optimized_region); + /* Optimization staging list, protected by 'kprobe_mutex' */ static LIST_HEAD(optimizing_list); static LIST_HEAD(unoptimizing_list); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index a801ec4a951b..a55dc2a20fb7 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1114,6 +1114,9 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned = long ip) * deliberately does not consult is_ftrace_trampoline() and friends: te= xt * being torn down may already be unregistered there while a task still * stands on it; + * - inside the bytes following a registered kprobe that jump optimization + * may overwrite, which kprobe_optimizer() protects with + * synchronize_rcu_tasks(); * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(). * * A false positive only defers the quiescent state to the task's next @@ -1121,6 +1124,9 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned = long ip) */ bool rcu_tasks_ip_in_trampoline(unsigned long ip) { + if (kprobe_in_optimized_region(ip)) + return true; + if (core_kernel_text(ip)) return arch_rcu_tasks_ip_in_trampoline(ip); return !is_module_text_address(ip); --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066286; cv=none; d=zohomail.com; s=zohoarc; b=JWdkSxywtHFTrmDkuTG3hTX8ma8c8VIv8bV7Fz5zbFFh5OAHZgz/KqPM9Ys4Vksh0u1LBt5Mxg3GmZrOpzovvXD6hlZgLRYYTgBkyM5iqAhIYjA+y0q7I+St445Cd4BAvOma6UNvXbQ2WPuwsiiu2rEVoBpAo9yAatwVxtc9UME= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066286; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=+J5TsPokEtcBZgoC4/eVwMDHhzV3+rip9BUfjH6oWiE=; b=PgF+dJBbvEosMmg7DskvzzudU5F1lNaF48hH/k3UGsFWKGMY8M8NUMrWwV0pp/LM1bUfNx6999B3vWSmJJy0WxSusYANufqRtK+cNsnMWKTFFUSWcDib3A67Pof58RIbF9ub+hkwIv8ntJMkE76hRKrdSL3NYbq65sD5c6Ew2aY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066286535119.37128627128936; Thu, 10 Sep 2026 11:51:26 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415153.1644678 (Exim 4.92) (envelope-from ) id 1x4jrU-0005UO-KM; Thu, 10 Sep 2026 18:51:04 +0000 Received: by outflank-mailman (output) from mailman id 1415153.1644678; Thu, 10 Sep 2026 18:51:04 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrU-0005UB-HA; Thu, 10 Sep 2026 18:51:04 +0000 Received: by outflank-mailman (input) for mailman id 1415153; Thu, 10 Sep 2026 18:51:02 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrS-0005Jj-Qg for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:02 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrS-001Y5I-6h for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:02 +0200 Received: from [10.42.69.11] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fbe6-2eae-0a2a0a5409dd-0a2a450b9c5c-44 for ; Thu, 10 Sep 2026 20:51:02 +0200 Received: from [209.85.160.172] (helo=mail-qt1-f172.google.com) by tlsNG-42698a.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc15-b7e8-0a2a450b0019-d155a0aca9f1-3 for ; Thu, 10 Sep 2026 20:51:02 +0200 Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-52fa9c055b5so15088301cf.1 for ; Thu, 10 Sep 2026 11:51:01 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca45bbd7sm230971cf.8.2026.09.10.11.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:50:59 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066260; x=1789671060; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+J5TsPokEtcBZgoC4/eVwMDHhzV3+rip9BUfjH6oWiE=; b=k2R7qgxLhaHpZMFV/FyIOtpDdDYG7ItfL9mSXNlvwpkXAdW03/OTnYYWXvByjTyssM 3KNGBgdzufCpH+kpKr2qrqcYaxOSqT5LYw0nKbpDUv033fnfAlir1Qhf6/YLs0GDQ/+j 395RLubVGLbincc4DSqwSnkWvYYT7mvbPD/sWDoKXIWNnh6QDILp3GK1TQSfVVbG7yu3 6EIyo8xBhleT7oEXJW2I91L48zMHrY/LuRhk2107nspNYaNCxib7rfppkpReHw+CI1zr YbgmgVh8GLTmW9xi/x293MLfuMRVD2C3ucShGQ2Ezf5RALWXpFlTyZkv4CUYQtA4SdDg jGZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066260; x=1789671060; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=+J5TsPokEtcBZgoC4/eVwMDHhzV3+rip9BUfjH6oWiE=; b=eSk5sFgApxPkP5RDZE4teOHL+1jfD0MKJNyJJDvK36JbdPr74HDHlUF02AhdxdvJu3 H4WMXzoi66Ws8yIxnBCV//FPzScwY2EFG68cBI1Z5amKDF+ZxTcGSEzCv6wQmqVdKZDt nN31xlPXQK/Rxj00Ypr6iRvJ1p0XG+tuhXUOoFK0A8Mx5LQyz0ojaSqTL3jv7UBW/NZD JXEzafclB3v28v9dLry2np+rL7nb6RBKHP7dUyK6Y0695t/bQwWZtneCvoDg6u8aNRwH rBRmEQYRBDDrtrqU0n13TVFxB5pVfTKwI+5CrnK05LZRV2aai78IPJEc1mliKdk7zneX pWzA== X-Forwarded-Encrypted: i=1; AKwUvBzR6wGBwNRucABzd8SheaRAeYyVmx6LjFfTdV2Jm1BPZYtROM4ZqVRPkUttRFOITAQKysBZA9KR5O0=@lists.xenproject.org X-Gm-Message-State: AFuF++meniDitsiMzoGgliQSbhZMDvZV9azrOfhe1iiEz4hiiMwhC2R5 R4PKFYID/MH3tUUd/8eds+PIgpxY3oAfOhHZiLLO5YRKOuJ0YyR4MmXpd/RbSfKx4tI= X-Gm-Gg: AYBFou2JcePGI8RETKalPi96RUeBHawaENDHZ0dl1fguPumgQnPy9XAbXO/GTmNxHE7 IXE61XlnDtj02rIaSySn+LheCLlJqKKmyfQj5XYS45jYHep/+N7DJGFpL8qmWDgaNjvTsjhGm3+ t3GSLGuTSlAz3V4XAF2DhKEs0+Nq4KgFVhItvY8tfK0cAGBt3ojAZOtEzkrEJNJhvfJBv5xOU7c mVL9udXJyFfUHilXNk5/q6ZY4+wCY8U1kH/9NYju7bdwBUldy3bNrcPxogbO1BmVrnoDOMHeq/k UXfsXyZGa+Sz9PCHISFrCpK3XQVVR8brXtVfUfpk4aW+mIId+6yEWLL+Th36WEmKPjPEHdNv3BZ tCxDmc4DinylRnYt0fn5Hpnf9WbtnchNrlDQzAFkUbuG50fc4pgo/ErgWKNCiHR0Lt4lZuqGKEf XyWxFRal2DRZWrsNSQ/iTG+I8khf4olbVM7ER632U5xBcg6HcgMbvjSWVxvDIvSJY7wjfFyYugF gZ8Qs8Z/Yhr+L2XR4aMvF8DxuZ1hDCwoE6DIKCEGoazfLROQ8HHSc2w X-Received: by 2002:ac8:6f19:0:b0:530:4773:113d with SMTP id d75a77b69052e-530b35ec4f6mr85380621cf.27.1789066260401; Thu, 10 Sep 2026 11:51:00 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:28 +0000 Subject: [PATCH RFC 05/13] ftrace: Mark modules hosting direct-call trampolines for Tasks RCU MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-5-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-42698a/1789066262-AB0DD9EA-7F2523CD/0/0 X-purgate-type: clean X-purgate-size: 7070 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066287211158500 An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Once preemption becomes a Tasks RCU quiescent state, such a trampoline must hold current->rcu_tramp_nesting across its call-out like the ftrace and BPF trampolines do, so document that in register_ftrace_direct(). That still leaves the few instructions before the increment and after the decrement. For BPF images those are in dynamically allocated text that rcu_tasks_ip_in_trampoline() already treats as protected, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag, set by every register/modify path when the direct address is module text, and have rcu_tasks_ip_in_trampoline() treat a task interrupted anywhere in such a module as a potential reader. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 23 +++++++++++++++++++++-- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..ea4727f53fab 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_ip_in_trampoline(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index a55dc2a20fb7..df68a330769a 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1117,19 +1117,38 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigne= d long ip) * - inside the bytes following a registered kprobe that jump optimization * may overwrite, which kprobe_optimizer() protects with * synchronize_rcu_tasks(); - * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(). + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampol= ine(); + * - in the text of a module that hosts an ftrace direct-call trampoline, + * which covers the instructions before that trampoline's increment and + * after its decrement (see ftrace_direct_mark_module()). * * A false positive only defers the quiescent state to the task's next * context switch. */ bool rcu_tasks_ip_in_trampoline(unsigned long ip) { + bool ret =3D true; + if (kprobe_in_optimized_region(ip)) return true; =20 if (core_kernel_text(ip)) return arch_rcu_tasks_ip_in_trampoline(ip); - return !is_module_text_address(ip); + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod =3D __module_text_address(ip); + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + if (mod) + ret =3D READ_ONCE(mod->ftrace_direct_tramp); +#else + if (mod) + ret =3D false; +#endif + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..14f27b887231 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) ops->trampoline =3D 0; } =20 +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_ip_in_trampoline() recognises on its own = (see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must hold + * current->rcu_tramp_nesting across its call-out (see register_ftrace_dir= ect()); + * marking the owning module here covers the instructions before that incr= ement + * and after the decrement, where a task interrupted in the module's text = must + * not be treated as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed un= der + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#ifdef CONFIG_MODULES + struct module *mod; + + guard(rcu)(); + mod =3D __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task = is + * executing it or is preempted in something it called. On architectures = that + * select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU quiesc= ent + * state unless current->rcu_tramp_nesting is non-zero, so the trampoline = must + * increment it before calling out and decrement it before returning, as t= he + * ftrace and BPF trampolines do (see rcu_tasks_trampoline_enter() and + * samples/ftrace/ftrace-direct.h). The few instructions before the incre= ment + * and after the decrement are covered by the irq-exit IP check: automatic= ally + * for trampolines outside kernel and module text (e.g. BPF images), and v= ia + * ftrace_direct_mark_module() for trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) ops->flags |=3D MULTI_FLAGS; ops->trampoline =3D FTRACE_REGS_ADDR; ops->direct_call =3D addr; + ftrace_direct_mark_module(addr); =20 err =3D register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, = struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } =20 @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, = struct ftrace_hash *hash, b tmp =3D __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct =3D entry->direct; } } --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066282; cv=none; d=zohomail.com; s=zohoarc; b=QD9SYbqmYAHxCSeKBD6A9LB2H4n5M5D78nal4XaWBhPXjioIZZGS/k9aIldZwnt3NfYFBV9lV3ZtZxJ6pizQ8+HkJd79WUVEutjln7WIXkLIa7pPV2TGXeYdVkLm1eey4queFvIZgjC4k3zaMMyOvOfskfjhSoxsfqIqudhNab8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066282; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=CXri9TuH33xtHtBe4zODZzC5ZTQ4gwh2PPULlIm604+ruVUw4ilMDRYLrU46mBEbBibOOPfIOpvvzfE+aYTloTtIAvWnKyWYbKsSt9vZ9lnVNvxtwnefd+D3f52xXTcjlxiuK4Hf9DSFTcYsTrqD5pyT1BdaEkOCnixPkWtyPgc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066282669969.5620310839055; Thu, 10 Sep 2026 11:51:22 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415154.1644688 (Exim 4.92) (envelope-from ) id 1x4jrW-0005lY-W2; Thu, 10 Sep 2026 18:51:06 +0000 Received: by outflank-mailman (output) from mailman id 1415154.1644688; Thu, 10 Sep 2026 18:51:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrW-0005lL-Qf; Thu, 10 Sep 2026 18:51:06 +0000 Received: by outflank-mailman (input) for mailman id 1415154; Thu, 10 Sep 2026 18:51:05 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrV-0005XH-0p for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:05 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrU-00GcVR-DT for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:04 +0200 Received: from [10.42.69.5] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc02-bab6-0a2a0a5309dd-0a2a4505895e-24 for ; Thu, 10 Sep 2026 20:51:04 +0200 Received: from [209.85.160.175] (helo=mail-qt1-f175.google.com) by tlsNG-c201ff.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc17-4cb1-0a2a45050019-d155a0afe0fa-3 for ; Thu, 10 Sep 2026 20:51:04 +0200 Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-5306d609317so1134411cf.2 for ; Thu, 10 Sep 2026 11:51:03 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f45a780sm2547836d6.10.2026.09.10.11.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:01 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066263; x=1789671063; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=iMdfafnxHmbTYoQLU5GtDkFqgWcS9zyZTR6ftc+GmmdOJzaZZG+SXqYLtWTaurTkrO eJEGeV6ENiZWYXHBlfoyEor5VIJcGMRtWRHNjfo1uPOntmfmGeUqxWUD7BrXgn/tWbun hYZF8cAysRcbJJCC5iVZH/oN7WFVLY9cZFQAmFgPcinc1h7aqLKbVlDjvgTylwLDEUHL /b4PoGO1Mbx8y5M0cf8H2dQkGk/d7ViGDoAnmPcpyWOQLejgcmvgLPFRFkdqggaUnXCW kHTtsqrmzyk+THRkWTUNDMHoZsIgP92KrA6o14bkZvBGs8PPJWHWg3cLJ7hOzXAMXSZO zITQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066263; x=1789671063; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=sAaaOhrJ7Jcw63o+Xz8kKBY4jQKDr2HCsih3+9envV4=; b=ZXqyKIkYdPfwEKLl2m8HDiUkv9sv/VAp3qkDC1b/5g1EUbsWE02PQXeHE5boM6/31i DLFBFaRnW6ZsygnQLsET4p5AA3cjoQn2sX8iO3JyCYM4GIcdIHcFSAy6pPVpR2XZsi6s p6sScywp+JNSfgvxufgPTLWmoxw/6kqJ5r4RiRwPdtJ51M1xZl29CrygMLp0BE621j5J HBzyQBfvSUbg4rUxY/2OGLguX6waYPT6wylZyHIC4SFqjPvyaVTffqX0MkfeGujKcrC/ WSe5OIPOlKY3zPb9yKFVMB2oGHyGp/wVygP6ZQdHs2Ra9/6H/q24tKgtgMguUlKazZo4 hI5Q== X-Forwarded-Encrypted: i=1; AKwUvBwaV3L/XH7uGrlBizYSPSeQHF0vZ0R2yUHlIZjuEQvm/eFBcqtboc8XSR8426e5u3kj8MEBq7Zql6M=@lists.xenproject.org X-Gm-Message-State: AFuF++kjGdFcpOrK4jMIqZMCwvWLe+ltLtvVzYjVfhdbUQWj32AMUT/U h8f+sDaYDMamgYd93CWvlcKEZpKuZTJjye1DuGjOqIOox8UdvolHPRVPurxhiGp2WMk= X-Gm-Gg: AYBFou11C1PEK0++Iz+yGR5aJiOkpt+em+ikRflk7ENIW9GblQHtcmdtvjrOoNzWlwd VfON1BiY0RVWE/n/19JyvMTPDfXVi9hadinKI/RqWEQmkztujBhi1pXxo7SKXOnEPVi+EUUSbFk 9R8hpItcfaXYfbIiuj1dAAXrLRotGr68QfmXSzBEYFlcGt06APvodelp88C2GyEnpRfigmIqSeX +1Bw/58qw6iCv8Y6hzYVN+Dv4P+Hy8VLKTqXXGp4ZwqeItwi+TGDOdRMBrWNlJXt0kBYq8OSiPj NdDbBILdVlUFq+NnXZCF8HAI8dEvzdtSAa1W4rf/PBTxga0oUSXjoAilJy3vnm4WYqLw0NRpKHz jqYIk0WqrByS6Ef5UTJws36QDnOdBO60YzE1AQqalG1fHO7NoVCCkgW0bKjDlLYMTAGSrd7+qnm pZTMqwAi41t3q5UEdOGoAhNmFbNNC5/2IYB34Z/FVOmwuZ7+e1POtimxyrYd1NMesJdNpPGB6tg cMO9aBD5Xl9yUQG79aiC5cOHLV1OnDSgQ/mBz9T21QHWXM89tkHzCxH X-Received: by 2002:a05:622a:1309:b0:530:42d4:4c98 with SMTP id d75a77b69052e-530c8756e8amr10238141cf.42.1789066262237; Thu, 10 Sep 2026 11:51:02 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:29 +0000 Subject: [PATCH RFC 06/13] x86/ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-6-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-c201ff/1789066264-72EB12A1-9D871752/0/0 X-purgate-type: clean X-purgate-size: 7824 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066283259158500 Bracket the call out to the ftrace_ops callback in ftrace_caller and ftrace_regs_caller with an increment/decrement of current->rcu_tramp_nesting. The instructions sit inside the region that create_trampoline() copies for per-ops dynamic trampolines, so those inherit them; the %rip-relative per-CPU reference to current_task is fixed up by text_poke_apply_relocation() like CALL_DEPTH_ACCOUNT's. %rdx is dead at both points (about to be loaded with the ops pointer on entry, restored by restore_mcount_regs on exit). Two pieces of core text still run with the count at zero while holding the address of a Tasks-RCU-protected trampoline they are about to enter: the static stubs themselves, whose direct-call tails keep a BPF trampoline address on the stack until the final RET, and, under CONFIG_MITIGATION_RETHUNK, the return thunk that RET expands to. Add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and linker symbols around .text..__x86.return_thunk and .text..__x86.rethunk_safe, and provide arch_rcu_tasks_ip_in_trampoline() covering [ftrace_caller, ftrace_static_tramp_end) and both thunk ranges so the irq-exit check treats a task interrupted there as still inside a trampoline. The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which x86 does not select until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/asm-offsets.c | 3 +++ arch/x86/kernel/ftrace.c | 37 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/ftrace_64.S | 43 +++++++++++++++++++++++++++++++++++++++= ++++ arch/x86/kernel/vmlinux.lds.S | 4 ++++ 4 files changed, 87 insertions(+) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 081816888f7a..4f3b1caa5a30 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -46,6 +46,9 @@ static void __used common(void) #ifdef CONFIG_STACKPROTECTOR OFFSET(TASK_stack_canary, task_struct, stack_canary); #endif +#ifdef CONFIG_TASKS_RCU + OFFSET(TASK_rcu_tramp_nesting, task_struct, rcu_tramp_nesting); +#endif =20 BLANK(); OFFSET(pbe_address, pbe, address); diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 17d6edfcb7e0..8f63cd4b543c 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -275,6 +275,43 @@ static inline void tramp_free(void *tramp) execmem_free(tramp); } =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +extern void ftrace_static_tramp_end(void); +extern char __return_thunk_start[], __return_thunk_end[]; +extern char __rethunk_safe_start[], __rethunk_safe_end[]; + +/* + * See rcu_tasks_ip_in_trampoline(). Some core kernel text behaves like a + * trampoline for Tasks RCU purposes because a task executing there with + * rcu_tramp_nesting =3D=3D 0 may still be about to enter a Tasks-RCU-prot= ected + * trampoline whose address it already holds: + * + * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_t= ramp + * stubs, which carry a direct-call target on the stack until their fin= al + * RET, and + * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUN= K, + * which run after leaving the stubs above and before landing in that + * target. + */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + if (ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end) + return true; +#ifdef CONFIG_MITIGATION_RETPOLINE + if (ip >=3D (unsigned long)__return_thunk_start && + ip < (unsigned long)__return_thunk_end) + return true; +#endif +#ifdef CONFIG_MITIGATION_SRSO + if (ip >=3D (unsigned long)__rethunk_safe_start && + ip < (unsigned long)__rethunk_safe_end) + return true; +#endif + return false; +} +#endif /* CONFIG_RCU_TASKS_PREEMPT_QS */ + /* Defined as markers to the end of the ftrace default trampolines */ extern void ftrace_regs_caller_end(void); extern void ftrace_caller_end(void); diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 62c1c93aa1c6..902472c41798 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -145,6 +146,27 @@ SYM_FUNC_END(ftrace_stub_graph) =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). These = live + * inside the region copied into dynamic trampolines; the %rip-relative pe= r-CPU + * reference is fixed up by text_poke_apply_relocation() in create_trampol= ine(). + * The increment must precede the function_trace_op load: between that loa= d and + * the call, the ops pointer in %rdx is protected only by Tasks RCU. + */ +.macro RCU_TASKS_TRAMP_ENTER reg:req +#ifdef CONFIG_TASKS_RCU + movq PER_CPU_VAR(current_task), \reg + incl TASK_rcu_tramp_nesting(\reg) +#endif +.endm + +.macro RCU_TASKS_TRAMP_EXIT reg:req +#ifdef CONFIG_TASKS_RCU + movq PER_CPU_VAR(current_task), \reg + decl TASK_rcu_tramp_nesting(\reg) +#endif +.endm + SYM_FUNC_START(__fentry__) ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT @@ -163,6 +185,8 @@ SYM_FUNC_START(ftrace_caller) leaq MCOUNT_REG_SIZE+8(%rsp), %rcx movq %rcx, RSP(%rsp) =20 + RCU_TASKS_TRAMP_ENTER %rdx + SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -181,6 +205,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + RCU_TASKS_TRAMP_EXIT %rdx + /* Handlers can change the RIP */ movq RIP(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -209,6 +235,8 @@ SYM_FUNC_START(ftrace_regs_caller) =20 CALL_DEPTH_ACCOUNT =20 + RCU_TASKS_TRAMP_ENTER %rdx + SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -246,6 +274,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + RCU_TASKS_TRAMP_EXIT %rdx + /* Copy flags back to SS, to restore them */ movq EFLAGS(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -328,6 +358,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp) RET SYM_FUNC_END(ftrace_stub_direct_tramp) =20 +/* + * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text = by + * rcu_tasks_ip_in_trampoline(): after RCU_TASKS_TRAMP_EXIT the stubs may + * still hold a direct-call target (a BPF trampoline) on the stack until t= he + * final RET, and that target's lifetime is guarded by Tasks RCU. With + * return thunks the RET itself runs elsewhere; arch_rcu_tasks_ip_in_tramp= oline() + * covers the thunk text too. + */ +SYM_CODE_START_NOALIGN(ftrace_static_tramp_end) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR +SYM_CODE_END(ftrace_static_tramp_end) + #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 2438b89a4620..e546283dc267 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -151,7 +151,9 @@ SECTIONS * definition. */ . =3D srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 2= 0); + __rethunk_safe_start =3D .; *(.text..__x86.rethunk_safe) + __rethunk_safe_end =3D .; #endif ALIGN_ENTRY_TEXT_END =20 @@ -162,7 +164,9 @@ SECTIONS SOFTIRQENTRY_TEXT #ifdef CONFIG_MITIGATION_RETPOLINE *(.text..__x86.indirect_thunk) + __return_thunk_start =3D .; *(.text..__x86.return_thunk) + __return_thunk_end =3D .; #endif STATIC_CALL_TEXT *(.gnu.warning) --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066293; cv=none; d=zohomail.com; s=zohoarc; b=TECqrPVl9S8xhMq6ow5FKBigCLTUFzRuv3UjJ/iLi4rECB50ZI9yBR5XiJaBFDW2KrVl/F4TEOMe/cHtdGc6lIVxrcD/B9WtSFUPqaP2apjP9m8VZcVNXBs+UeqkHeoVcJwaDLiGTcGGG/SvjKiHGB8Yl40ihil91an3v+9ad7c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066293; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=hbVlQmHCpb9zv+5BDFsE+SK081jwlXFhk4tZhnaWY5k=; b=WcgPjCxKWAv/qgKHCXLvmXUqSPUKN3PtrlZSmOIvnmIApLUCb9+ZisNbS9kIqnpI5gtdGaVncCxFD96B8K1nbEWjNhycuS7UrhTOTuRV6jypy+fSMBDBcLfFsF050Uer6rIluJ2P0tdxVQ/UmH0BsgNnJco6Y1MbLlIdKtyQDs4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066293276354.83098578379224; Thu, 10 Sep 2026 11:51:33 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415156.1644696 (Exim 4.92) (envelope-from ) id 1x4jrZ-00063q-5M; Thu, 10 Sep 2026 18:51:09 +0000 Received: by outflank-mailman (output) from mailman id 1415156.1644696; Thu, 10 Sep 2026 18:51:09 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrZ-00063h-19; Thu, 10 Sep 2026 18:51:09 +0000 Received: by outflank-mailman (input) for mailman id 1415156; Thu, 10 Sep 2026 18:51:07 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrX-0005uX-PR for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:07 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrX-00BfN4-5j for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:07 +0200 Received: from [10.42.69.12] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc15-e002-0a2a0a5209dd-0a2a450cd430-16 for ; Thu, 10 Sep 2026 20:51:07 +0200 Received: from [209.85.160.176] (helo=mail-qt1-f176.google.com) by tlsNG-d25034.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc19-f479-0a2a450c0019-d155a0b0d829-3 for ; Thu, 10 Sep 2026 20:51:06 +0200 Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-52de50e77ffso1049931cf.2 for ; Thu, 10 Sep 2026 11:51:06 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530ca47efa2sm216291cf.11.2026.09.10.11.51.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:04 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066265; x=1789671065; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hbVlQmHCpb9zv+5BDFsE+SK081jwlXFhk4tZhnaWY5k=; b=eAC6lVWZHl5D9Win+zVmKYJI7dkTccT2i5OXUlFLhS38FccSerR8CLEoFoRa70FFgK mYMBcZ+SDERtkY//x3PazflZ/mfTOb88zK/r0gcJHUz3wlsn/evx//FGyWswxw2oqmB0 BhCbKohO91paAvYYlM8TudUGEnIAvZpVNDEWZlJjSbNtAzCDTc1RVY+ZA35AwHBW41ui AKmFe99X38O2ripXIasAkr3P1oCOc9I3c7bStL414zxoAkeoaBA+y59hp3vTonbimYWW QsLmwVeBFO1v0VWAj6hyfyM+bKqHq1CuN0FHlvOgfLKfPooxlCEmZPTATU6UUapvpaXU cimQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066265; x=1789671065; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=hbVlQmHCpb9zv+5BDFsE+SK081jwlXFhk4tZhnaWY5k=; b=tNhbhUPItUitCjq5bu5hWOimXEK/C8cRYGQ01P+1qe1e0NH4nJhu1/LAmrDZPnOKUW VS+RDH0NeVvaTJxasQTYFThMpxa6uqS0GkLSFQYFxvSXZbPgCT+/ETu+0cDy/3QM8Mwo xeG10wAepCZ6h1Bk0XTIPMvJ30OrqSch2stmZUK4j1Qi8Koeuf1a1mK9Jp3V8bsO/CX5 Ty4TTgPaWsaakcPy5I4K4/ZZum2fgTbSichjyBXzeax5t9EJEBg1Jc1VQQ8RgA2w0X22 lozcKOk386oah1XPEF7btFsfauKF1oEQ4Zpa04EoPCs/71vYiVgrdJDtmR9wz0Nf1veF Hoqw== X-Forwarded-Encrypted: i=1; AKwUvBxXLSuW3MTNrVttqTWa2YU082zPuuZ3PzmnRjM1afqR5/LilWKnbclclWXbEtE2MkgH85pBUxpc7ZY=@lists.xenproject.org X-Gm-Message-State: AFuF++nTAVxGDkwBbkyqfCwVlSq3P9RDa4t/rC4ZtsmLEyRbKAxy3x+s qguCZv2IrGbslDHPqvEdbLDt9tjA918qvhGHat1b/nLJPJ/CtZTy2uawSESzr1DBJVc= X-Gm-Gg: AYBFou0KN+5Sw1pi2mM1YvdDEXGN24hbhuJCPpP1zH7opUWiMt7EEzj0MOHlB5NwEZ3 PJIwpvUhFaxM3QPtWTbTWjhqXJBBjt78bk5M8GBXZyF56FLJnehVbIyIYwwmPIPHZcP3tdkMGbc qiiXXjY2n45ao5NlFUAjcJEp+LZdtrqc7ePij9dnAtYskeTWI24YgTqb51l1UmOyumBmKZ96aHV 1l7hVyNLw4T2pRx1VK4rxJrUYnwOEgQUrTtJWLvS/zx34yzf7DqoUYKdtjK4kq4v7HtV0kbnhOC vj5l7O/AIGO4iurZ4iyZsKvgnsmOZK+ykJc/IiPcfq1S35X9CuZkk3mKWoMPX+9QDQ1OE/GFn1x cImen90+wBoqdW9O69PUUH7Qs4b/e3KtTsD8fo5OTQXdzS88VnCYb1xa8mZv0NJbiGBn/TG+lfp UtV7Y+CVfPj1D8PcbjFaxONwPY8IBoGpv1Fbc/eS5moJZcffp/Ix/dQc9ohorgeyFVFApbGdgjz 0O0UfZ3Qn6rRHN3kKRqA+xLZ8oQFjiK4nn1pMqa/aaSVV/JQCQEtQbq X-Received: by 2002:a05:622a:1309:b0:530:42e4:effc with SMTP id d75a77b69052e-530c87566f0mr10949261cf.41.1789066264886; Thu, 10 Sep 2026 11:51:04 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:30 +0000 Subject: [PATCH RFC 07/13] x86/kprobes: Maintain Tasks RCU trampoline nesting in the optprobe template MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-7-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-d25034/1789066267-022C0A5B-FD6BC4CB/0/0 X-purgate-type: clean X-purgate-size: 2638 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066295241158500 The jump-optimized kprobe template calls optimized_callback() with preemption still enabled for its first few instructions, so bracket the call with an increment/decrement of current->rcu_tramp_nesting. The template lives in .rodata and is memcpy()d into each optinsn slot without relocation processing, so the per-CPU reference to current_task must be an absolute %gs: address (R_X86_64_32S, relocated for KASLR like any other) rather than %rip-relative. %rax has already been saved by SAVE_REGS_STRING and is dead after the call. The slot itself is dynamically allocated text, so the instructions before the increment and after the decrement are covered by the irq-exit IP check. 64-bit only; 32-bit x86 does not take part. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/kprobes/opt.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 3f8fea52619f..f722520bb989 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -31,6 +31,7 @@ #include #include #include +#include =20 #include "common.h" =20 @@ -101,6 +102,23 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr,= unsigned long val) *(unsigned long *)addr =3D val; } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). The + * template is memcpy()d into the slot without relocation processing, so t= he + * per-CPU reference must be absolute, not %rip-relative. + */ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_X86_64) +#define OPTPROBE_RCU_TASKS_ENTER \ + " movq %gs:current_task, %rax\n" \ + " incl " __stringify(TASK_rcu_tramp_nesting) "(%rax)\n" +#define OPTPROBE_RCU_TASKS_EXIT \ + " movq %gs:current_task, %rax\n" \ + " decl " __stringify(TASK_rcu_tramp_nesting) "(%rax)\n" +#else +#define OPTPROBE_RCU_TASKS_ENTER +#define OPTPROBE_RCU_TASKS_EXIT +#endif + asm ( ".pushsection .rodata\n" ".global optprobe_template_entry\n" @@ -114,6 +132,7 @@ asm ( "optprobe_template_clac:\n" ASM_NOP3 SAVE_REGS_STRING + OPTPROBE_RCU_TASKS_ENTER " movq %rsp, %rsi\n" ".global optprobe_template_val\n" "optprobe_template_val:\n" @@ -122,6 +141,7 @@ asm ( ".global optprobe_template_call\n" "optprobe_template_call:\n" ASM_NOP5 + OPTPROBE_RCU_TASKS_EXIT /* Copy 'regs->flags' into 'regs->ss'. */ " movq 18*8(%rsp), %rdx\n" " movq %rdx, 20*8(%rsp)\n" --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066298; cv=none; d=zohomail.com; s=zohoarc; b=Fb1aHk0j5rc73GzQZYvtNjMQ295WjlCdUEoBr4u+bzEnHrmMiWn6NnWn/ubvXvelni+xtbUhsRTXv6F/6FQYWLttMu1Uf4ezdyHSJsaauvxPzlkLtD0RMRLgADEGIMSDjcaksxhi8jxMmHfex3DWYB7JoZMJQE0lcf+CrGQ85Vk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066298; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=LhsubAci6ASgSs3sGHDL0Q6BLGhta1yJ0VrVRUhb4RaKE3+X6lrUunjR9dr0RSnd4miXdDyJ9/sR8/URWfshi3jeI6qcMHfbqASlVUWQQDRhNGHeNKTARdihzwOwGpvgvlCgnfOghM0hkcNLbSzXaY096rGbtmIn12oytpgBfZo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066298847132.94463323770026; Thu, 10 Sep 2026 11:51:38 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415165.1644713 (Exim 4.92) (envelope-from ) id 1x4jre-0006fa-04; Thu, 10 Sep 2026 18:51:14 +0000 Received: by outflank-mailman (output) from mailman id 1415165.1644713; Thu, 10 Sep 2026 18:51:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrd-0006eZ-Os; Thu, 10 Sep 2026 18:51:13 +0000 Received: by outflank-mailman (input) for mailman id 1415165; Thu, 10 Sep 2026 18:51:13 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrc-0006Qa-RT for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:12 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrc-001YAS-7x for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:12 +0200 Received: from [10.42.69.4] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc16-2eae-0a2a0a5409dd-0a2a4504d6e0-18 for ; Thu, 10 Sep 2026 20:51:12 +0200 Received: from [209.85.160.170] (helo=mail-qt1-f170.google.com) by tlsNG-ebf023.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc1f-b57f-0a2a45040019-d155a0aad1c4-3 for ; Thu, 10 Sep 2026 20:51:12 +0200 Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-53091987029so1493231cf.3 for ; Thu, 10 Sep 2026 11:51:11 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f47a239sm2556366d6.26.2026.09.10.11.51.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:06 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066271; x=1789671071; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=LFkPgazcrw8BpnABGKjUhCWQdoyba0TX6luIrM7Qm2WQnLD+cmgGm4oyPaBHLo0Rz8 +gSfqx4/X/hO1JevE9rUYXf6wqBZyUSEg57kdqhiPB1Mc5lJl9glmGEUAVtvbWLo9OJi HX6rDOsv60e5yamxLDrTK13hYLVIh74DAW3rpmO6x7Fdt7Zte/a3+PO/YC7pg1Gg32mZ 5XPzdsGFwAyiYR2EAp0q++ogLg2Wb2YR8G3AhM9iNZ6+mazhE01UTXdHXjXy8DFeepFO X3XldodNBS2xmxTCoPaDLEiI/OE2yKL0Ab0nlvMxNDYYy8AepnHEq+F/0snFuDOZ7oJ6 tahw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066271; x=1789671071; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=GctmtDuxfshuBde7qmWn7P5qaQ14r66IXlZUeq2cXfxMWWpEszgA0d574IzWLeqohi LwgLLfbcOpFWcYhODBdI7sgs4JqlYy3crIst6R4QYs60O9Bcezen8jUT9cT4CFq156nH KWR4fSw0Sv7a7vMnBOSebGnNusFVa7rFTq/CW2qxiboBaDS5Vv6M0V3Mwr4cPdZ5/lH3 fv3okBJ7nnFbUdJN2hNvIDcajNo4q9MLupV60l5daojqw5mUoRgr0vFwneQ7PVe95PNU v2oaEAm8W945wBg+XuLJxwen9wevH4dlbIDXOCYifUZY67K4fS9r36g27SpG9bJPaScw 2CKg== X-Forwarded-Encrypted: i=1; AKwUvBywYzarpWE7zVfpyzEiPFHJAF0DFPSRo0Z2All2QI6aOVUoy96H4oX2OMbsYHBZlJ+O3XctKfpGOXM=@lists.xenproject.org X-Gm-Message-State: AFuF++mKi0tYTULD+/6sqkbO8GtXAdr5gAi1YqcrRNEX0lfS1lYhPK/m WeDAJvNr+2pKpyU6TIr1UXLJdAuH4hCKWn/9FrsQ697CjP1VJvraUBKQMMMpL1rVM5A= X-Gm-Gg: AYBFou2NmfOqz6SucL/tYH2WjTbl4l0Imcizx63p3q+rm5LNFvINnZxCsI98FNQt62h OAwJdF0idXcI13aE8cCQCV1mQDynU63R8DIovemuqtwTE167Vv3OuHzLQ8ErvW+DeoZLAS3Heeb htDK9WJff7c61y4goajGwujgNKcl3PU0asIf3fycLXyUMaxbeEsNx72VIfycJAFlpcOKCXlkn7E +Zw3RkHQeUO1ekD335Cp6HIrvXhZ74X43OwFkcw1mczFsALyH83DSrmsM3/VdwmC0l20TNa15KA EQVBkWrERHVGlJXM8ZIoGeNuzPixBYC/CMCsAMx8sRm2FVjM2EE1bBoh4jIG2rVpIIbFlXuiwqs SKB0HMMxti5caijhtfy0tki1VCNc11HreXlRsJ5NQhSFlN3PjatHRiqNApC1enu1hcb+O26/be8 xV/l6QyMn10178tUr7xj16Km2mZSKlIEKyftQQGyYN/cBVIomT3SpG/CsMflz1RVAoC2d6a8Xxy Z9bhmLzk6V69crU6TYYpfRx19eqzIyTCQIIK/nHjIqO0hQy+xXth6CK X-Received: by 2002:a05:622a:2294:b0:530:b2e4:d597 with SMTP id d75a77b69052e-530c87487damr11129971cf.50.1789066266848; Thu, 10 Sep 2026 11:51:06 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:31 +0000 Subject: [PATCH RFC 08/13] bpf, x86: Maintain Tasks RCU trampoline nesting in the BPF trampoline MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-8-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-ebf023/1789066272-C24CBB50-345CEFD1/0/0 X-purgate-type: clean X-purgate-size: 4466 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066299294158500 Emit an increment of current->rcu_tramp_nesting once the trampoline's frame is set up and a decrement before the final register restore, so that a task preempted while running fentry/fexit/fmod_ret/LSM programs or the __bpf_tramp_enter()/__bpf_tramp_exit() glue is not treated as Tasks-RCU quiescent. Drop the count around the call to the original function: that may run arbitrarily long without sleeping and must not pin a Tasks RCU grace period, and the trampoline frame above it is held by im->pcref rather than by Tasks RCU (see bpf_tramp_image_put()). The fmod_ret early-exit branch and the ip_after_call -> ip_epilogue poke both skip the decrement/increment pair around the original call, so the count stays balanced on every path. The sequence is "mov r11, gs:[current_task]; inc/dec dword [r11 + off]"; r11 is scratch at every emission point and (u32)¤t_task is a valid sign-extended %gs-absolute with the current per-CPU layout, the same form the JIT already uses for this_cpu_off. The image is dynamically allocated text, so the instructions outside the bracketed region are covered by the irq-exit IP check. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++++++++++++++++++++++++++++= ++ 1 file changed, 43 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2853e87797a7..a375c1b7bd50 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -722,6 +722,31 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg= , u8 *ip) *pprog =3D prog; } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). + * + * mov r11, QWORD PTR gs:[current_task] + * inc/dec DWORD PTR [r11 + offsetof(struct task_struct, rcu_tramp_nesti= ng)] + * + * r11 (AUX_REG) is scratch in the trampoline at every point this is emitt= ed. + */ +static void emit_rcu_tasks_tramp_nesting(u8 **pprog, bool enter) +{ +#ifdef CONFIG_TASKS_RCU + u8 *prog =3D *pprog; + + /* mov r11, gs:[abs32] */ + EMIT2(0x65, 0x4C); + EMIT3(0x8B, 0x1C, 0x25); + EMIT((u32)(unsigned long)¤t_task, 4); + /* inc/dec dword ptr [r11 + disp32] */ + EMIT3(0x41, 0xFF, enter ? 0x83 : 0x8B); + EMIT(offsetof(struct task_struct, rcu_tramp_nesting), 4); + + *pprog =3D prog; +#endif +} + static void emit_return(u8 **pprog, u8 *ip) { u8 *prog =3D *pprog; @@ -3610,6 +3635,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im /* mov QWORD PTR [rbp - rbx_off], rbx */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); =20 + /* + * From here until the matching decrement before the final return, a + * preemption of this task is not a Tasks RCU quiescent state. The + * instructions above this point are covered by the irq-exit IP check. + */ + emit_rcu_tasks_tramp_nesting(&prog, true); + func_meta =3D nr_regs; /* Store number of argument registers of the traced function */ emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); @@ -3670,6 +3702,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size); } =20 + /* + * The original function may run for a long time without + * sleeping; do not let it pin a Tasks RCU grace period. The + * trampoline frame above it is held by im->pcref + * (__bpf_tramp_enter()), not by Tasks RCU, across the call. + */ + emit_rcu_tasks_tramp_nesting(&prog, false); if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); EMIT2(0xff, 0xd3); /* call *rbx */ @@ -3680,6 +3719,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im goto cleanup; } } + emit_rcu_tasks_tramp_nesting(&prog, true); /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); im->ip_after_call =3D image + (prog - (u8 *)rw_image); @@ -3741,6 +3781,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_rcu_tasks_tramp_nesting(&prog, false); + emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); =20 EMIT1(0xC9); /* leave */ --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066291; cv=none; d=zohomail.com; s=zohoarc; b=GFj8Juncc/5mCTaZtAXInCL2V/7mW/q2muXmqq8d9EcFZT6iIIF/dFEHUuzZBuwubt4w7DrdlqGZPH7e1BBl57kS5+YSWZfcQRqADADMtfYYSJTmH5nYLllqmrUXLedelUjw3Ng1J/UiR088hN21OgTU4N/VaBlfRRF4uhYrhAs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066291; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=EOpBFIcgZsfmkGGREIf98PkQQvjbbyd031ReUUntO+lnvBuEc/kQclDYgutRVTk6A6X92AcitYgkrezpb7kyJT4KP6YepIHO2sEvqSThZgzOInNi+ztwB+2n/R4DEjdTbnKvOrGDENCtoC4TcnB7UXVan7FQDXA8bMjtfevSNYY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 178906629140354.49830369423273; Thu, 10 Sep 2026 11:51:31 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415163.1644704 (Exim 4.92) (envelope-from ) id 1x4jrc-0006Pj-II; Thu, 10 Sep 2026 18:51:12 +0000 Received: by outflank-mailman (output) from mailman id 1415163.1644704; Thu, 10 Sep 2026 18:51:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrc-0006PV-F9; Thu, 10 Sep 2026 18:51:12 +0000 Received: by outflank-mailman (input) for mailman id 1415163; Thu, 10 Sep 2026 18:51:11 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrb-0006N7-NO for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:11 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrb-00Gcad-3w for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:11 +0200 Received: from [10.42.69.2] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc17-bab6-0a2a0a5309dd-0a2a4502e940-10 for ; Thu, 10 Sep 2026 20:51:11 +0200 Received: from [209.85.222.172] (helo=mail-qk1-f172.google.com) by tlsNG-720697.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc1e-6ca4-0a2a45020019-d155deaca438-3 for ; Thu, 10 Sep 2026 20:51:10 +0200 Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-92ed19f4d60so13505785a.0 for ; Thu, 10 Sep 2026 11:51:10 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e7f1da12sm50063385a.9.2026.09.10.11.51.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:08 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066269; x=1789671069; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=Kw2a+A60926yogG1g0r4S8egw4amFvo40bjc7giALBVUBsmb7WmCNZjPflHSUYpHHR ARvgH3T/actVPbBkWQhNsRf7ENpYgVuuJ/dZztDZ/fTNeb9VELLQc8NluXiFVQgoFvOt Py6yUYYdm5cQEX0LjvfWsIi5A/FjGSSoa3/ft3jkuZX0bmzoLbfkhcOXAqgntgHX3zGP bDre1BTWI0is9DXY0g7SI0qQk9hQZdqCO0WXocCEs1gViQkVFX6oVOY66/HdmDxQrtA4 MnMlvOzv4PKkrDjtLdA6jGFinWRiWad3h4u0o++w6akBg/boanqEgMXa/q8eZdHkycwy YKiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066269; x=1789671069; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=lJwBdA+2jcR/Yq41IekeHbJ9z3aXH3BT8kJzl0ov/xPTyArY9uKl+q8bTJulvFWm+w j8ky7Ge59Xznpsu+XDRfiM6CyIhIQ4rm/aivCun8ghAcPsmVdGzzfSFPClx9pgCzMaiY 5yzo9cUalr75mC74a8/pohP31qfXXc1FG6FT3bPZsiKbI6GlzeqKDPXTitwW0iMFFsPj /QxicE9BlJKnbMDoqSHClwcsfxIygFeRwCstjVgjX1X4nIP6BiKLnpL9oqpemZfy5Q9/ Ha/I7pQndPYVR6ydm04Rb6jwxSIyB5nmQexkyci1iQxDvdhDlSsGiztxaLk7WDCplrSw wSrw== X-Forwarded-Encrypted: i=1; AKwUvBzqa4TaVMW2V7ybBGevtT16IJWY/0fcy3THDSetJYwF1JhDC5lafBRKG0aUsg8JZ17GvE938lgWoho=@lists.xenproject.org X-Gm-Message-State: AFuF++mat/9V6y54+McACBm43VdQsckpAPtsYqeR0a/qVawL+gPY+0Xn dPYzn5eJX+Vjyb4EgJlnoCuiCdLsCUO8wo22ISa6eq/QrUspAVGFYYCoypWoCJLwbdM= X-Gm-Gg: AYBFou24Sj95eA8I/JHAEBhJXmoLaxCsmqrHZMy1aQwq5rRnFarBLTZBjpeYvrbOKUN 2wUXcBgrJ+eOT+5p4sJCRV4B3450+HBzaeG3lfUUqU2TuvyVTmT8dsUukASajT/2iTEMpuHaX/2 dSogTHEjmPBdrhq/nwWY5hEAy7jqWRZxA8FzmVXiOPlg8kNm+sSzwbcenkKdc8MtUQoIpp/hlmv kyi5EVIZcFNfpuzmJtM6WpKgE1LOaTUne+5eHlsJx4WwsJzQgkNtSSQNutqOEJfjabMnGgGTmYO JB6gFEO9ofEt2K4/+jETAqepFTUOa2rA9X2TSlrqmxmpMGdoxyzOOB0foApvkbamptaAXHkYBO0 rxbjxzQYYx3XNBghcdB/FVyOF6XQZLI9BG9vF2biZkLOL0dTfPIhPzeFGossAmQfuEL8eOkkb3y yy4bKHwKEYVW98Lxv4TCx8hAVTIU5KEsOxuD904MHHK71mmlixRe+0ZkME+G34cLTABsmye1r/u rRB+lIzi7mQ8WHj27hz9lx2B/m3lAI26f20qQq1gEm9v9oqHgGtJGSm1lJFrkaPEAU= X-Received: by 2002:a05:620a:1a0a:b0:939:ca72:bef8 with SMTP id af79cd13be357-939d7fe4f58mr706492885a.41.1789066269332; Thu, 10 Sep 2026 11:51:09 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:32 +0000 Subject: [PATCH RFC 09/13] arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-9-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-720697/1789066271-323D42AC-D8908D05/0/0 X-purgate-type: clean X-purgate-size: 5219 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066293176158500 Bracket the call out to the ftrace_ops callback in ftrace_caller with an increment/decrement of current->rcu_tramp_nesting, using x12/w13 which are scratch there. The read-modify-write is not atomic, but only current modifies the count and every interrupting user is balanced, so nothing is lost. ftrace_caller itself, including the early CALL_OPS direct path and the late direct tail that carry a BPF trampoline address in x17 with the count at zero, is static kernel text: add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and provide arch_rcu_tasks_ip_in_trampoline() covering [ftrace_caller, ftrace_static_tramp_end) so the irq-exit check treats a task interrupted anywhere in it as inside a trampoline. The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which arm64 does not select until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry-ftrace.S | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/kernel/ftrace.c | 16 ++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offset= s.c index 9c853ed3ceab..f6655a284f18 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -39,6 +39,9 @@ int main(void) DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); +#endif +#ifdef CONFIG_TASKS_RCU + DEFINE(TSK_RCU_TRAMP_NESTING, offsetof(struct task_struct, rcu_tramp_nes= ting)); #endif BLANK(); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_conte= xt)); diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftr= ace.S index 025140caafe7..46a102e7199a 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -14,6 +14,33 @@ #include =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). The wh= ole + * of ftrace_caller is treated as trampoline text by the irq-exit IP check= (see + * arch_rcu_tasks_ip_in_trampoline()), so these only need to bracket the c= all + * out to ops->func; everything before the increment and after the decreme= nt, + * including the direct-call tails that carry a BPF trampoline address in = x17, + * is covered by that. The count is only modified by current and every ne= sted + * user (interrupts) is balanced, so a plain ldr/add/str is sufficient. + */ + .macro rcu_tasks_tramp_enter, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + add \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + + .macro rcu_tasks_tramp_exit, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + sub \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + /* * Due to -fpatchable-function-entry=3D2, the compiler has placed two NOPs= before * the regular function prologue. For an enabled callsite, ftrace_init_nop= () and @@ -94,6 +121,8 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE =20 + rcu_tasks_tramp_enter x12, w13 + /* Prepare arguments for the tracer func */ sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) @@ -111,6 +140,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) bl ftrace_stub // func(ip, parent_ip, op, regs) #endif =20 + rcu_tasks_tramp_exit x12, w13 + /* * At the callsite x0-x8 and x19-x30 were live. Any C code will have prese= rved * x19-x29 per the AAPCS, and we created frame records upon entry, so we n= eed @@ -178,6 +209,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp) SYM_CODE_END(ftrace_stub_direct_tramp) #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +/* End of [ftrace_caller, ...) for arch_rcu_tasks_ip_in_trampoline(). */ +SYM_CODE_START(ftrace_static_tramp_end) +SYM_CODE_END(ftrace_static_tramp_end) + #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ =20 /* diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index e1a3c0b3a051..1b7ac2afed0d 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -17,6 +17,22 @@ #include #include =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +extern void ftrace_static_tramp_end(void); + +/* + * See rcu_tasks_ip_in_trampoline(). ftrace_caller and ftrace_stub_direct= _tramp + * are core kernel text but must be treated as trampolines: a task preempt= ed in + * them may be carrying an ops pointer (x11) or a direct-call BPF trampoli= ne + * address (x17) whose lifetime is guarded only by Tasks RCU. + */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + return ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end; +} +#endif + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct fregs_offset { const char *name; --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066295; cv=none; d=zohomail.com; s=zohoarc; b=jQUq1RV2ltjcunqToGEwLTBKEsshlsacgBlQNyxAIxd1FFOFkJWXtHS3U2wnkcnQojGHE64gOVbIwDlVZwM6nfcqfFAwj4IjnQqCFFwi6tFbYhFebjUyrIa5GVvT6kdl0aVP4FrhZvg3UV2fMTgbIGh7rPmUnniJMGB3Ji40gZU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066295; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=9b8kkjvSg0tb+ScPvKP3GAz1Pld5wxO+feqfkbC4jq4=; b=HhH+3k1e5+Wg5WUOU98xeRcD1nJ2ppaZ4MY8o/t0MRjFOfEwGHFhsookba8g2ovlQybrS9+ZanYCwgkvb9gldngYUurxiHor2aSk0OPazNZ+oJFXe6RelUg0kjnmNBCPQhW9j2N7yC3uPchQJy2YwXY4mELplIqBGHlvQ9qdSZA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066295596689.3952426619741; Thu, 10 Sep 2026 11:51:35 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415166.1644724 (Exim 4.92) (envelope-from ) id 1x4jrf-0006xs-Cd; Thu, 10 Sep 2026 18:51:15 +0000 Received: by outflank-mailman (output) from mailman id 1415166.1644724; Thu, 10 Sep 2026 18:51:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrf-0006wR-4G; Thu, 10 Sep 2026 18:51:15 +0000 Received: by outflank-mailman (input) for mailman id 1415166; Thu, 10 Sep 2026 18:51:13 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrd-0006ar-K6 for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:13 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrd-00Gcad-0t for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:13 +0200 Received: from [10.42.69.6] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc15-bab6-0a2a0a5309dd-0a2a450695de-20 for ; Thu, 10 Sep 2026 20:51:13 +0200 Received: from [209.85.219.50] (helo=mail-qv1-f50.google.com) by tlsNG-16d1c6.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc1f-195a-0a2a45060019-d155db32e079-3 for ; Thu, 10 Sep 2026 20:51:12 +0200 Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-91053c27a91so244926d6.2 for ; Thu, 10 Sep 2026 11:51:12 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f49444bsm2501436d6.29.2026.09.10.11.51.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:10 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066271; x=1789671071; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9b8kkjvSg0tb+ScPvKP3GAz1Pld5wxO+feqfkbC4jq4=; b=Q21Q9PgSNWn30sbhwRwkPQ0MP1CC7bhnzuzlr17Fv62cS+tWK1Kk6ga5KtBb6F4ZZA 67IYtDgL8xIY9NsRt+uU+WV1rWRDuyk1re0fzX29gqfIrD4FqxA9R+tXXL7dNKKMy/6y l4kZikIvJdM27uX5FuW3SqZBQdRDKdrbLdvpKl00MB9nEHlCmk7QWHROSTCA+w0O++n2 pHR3+1DdZkoodd7Ct68VlLlAFjKICIxYz7ZZS4ah8f7XhKw5P1VKsaUOGjabgsx4uD2g S7eobRTZT6etd4pXfMRbczPQ7FzkoQbHVOZpqRjqsm2qWpwYkM9qwJ86gExNIcLYhxKt 2ELQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066271; x=1789671071; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9b8kkjvSg0tb+ScPvKP3GAz1Pld5wxO+feqfkbC4jq4=; b=C+yxX3GpHw0SXqI/HjJFSz0pN1K/MqDv4WsYVvVbrYLb7NFdFPLYcUOw4M/xpTYjtR 9PbFv24sYuLHQ56S/0XZzXnrVCK854NTLu+H4P+Gg2E9HfOuGtZxEEviRRlWk4S6r3Zq rIXjVUwNUg8efS1+pfc90RuK6AoCpHBKc8DNQGySBz36Xoy9U7402rWktd8taekaDUm+ pCNzyk5smovfYDXg72b3RvEZiGj9xhURg7IlN65W5qauvj9aOFCKU1vbfYhAcbPvXRcM pGaO9xwZJTWpIsBxWvrzpTRQbRNM6u2owRla8u8+g5gIJdtN6zf9r/bT4YZLsMlPf0bv t8KA== X-Forwarded-Encrypted: i=1; AKwUvBwP2tAHRbqRN89w4khKAMNlxNs5lCgsV+d7R0U3zl/h+hUJTTZHESmd2GlQWi9hjCHWcRwKC3QHIh0=@lists.xenproject.org X-Gm-Message-State: AFuF++kln520KST5YVE1DOX/FHvebKqrQBb63CeJmo/Yz2iStj1R5nUM cSAwnnwHzrfoaSLz0E9SyjwRkUV1EJ4lbto3g3s3NiOsizsA0xfvsKIj22Mu3+MHzZ0= X-Gm-Gg: AYBFou1koypL7wQgeFZMOngIcFtOQTqmQCoB7IC83Hfxt6ut91rSphBERbKubpDjb7X Bj47dF+0X+ZsFUj5KbGcDrRKH1YU/RbEIcU3cQ5XNMtowsIioi7Ea85Ns53WQPFyVQ+yIo83KwC r1skq6+b0IvvABpjXOzl7aad2UIKpxSTA0kktAxsL3UtrWWUgRel9seEeTssZyUoVK2gpbVSUNN 8QTB3xmJOql8S6q0G5wUlhx6qqPIWP6/hv2v/wSVqikE0zAlDzkvmthOBdtqOamevNZoVvGHqWc i7mA0C8RhjnEmd76EBurBlKGQ+7o/CIToPHGzdFuD9arm0SLA26Go1vRCeqKJ+3f5ak9oG3eObj TzRf0zlexErvJhbK24nA9LzfX07bapRIf+ZbSQhh5GQ9QwCvl7d6eHERVTOBQcWu7KraBdQKk+2 jIkwxwlDEV/K0ZxLPO1UVDFujlFHGNWA9/KB+sPNN5BNfMiJnAy+Vvb++pC1vXLR1cue+WYxkWm UYIpGx7OE+VbwnhBk/YHjRf02MYuk5EIR8jQQb1XNfERXjkZeH/8I0N X-Received: by 2002:a05:6214:5544:b0:910:52ab:59dc with SMTP id 6a1803df08f44-912120e2526mr945166d6.23.1789066271231; Thu, 10 Sep 2026 11:51:11 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:33 +0000 Subject: [PATCH RFC 10/13] bpf, arm64: Maintain Tasks RCU trampoline nesting in the BPF trampoline MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-10-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-16d1c6/1789066272-F5C0F77B-F2CA3D70/0/0 X-purgate-type: clean X-purgate-size: 4366 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066297219158500 Same scheme as x86: emit "mrs x10, sp_el0; ldr/add|sub/str w11" to bump current->rcu_tramp_nesting after the callee-saved registers are stored and to drop it before they are restored, and release it around the call to the original function, which im->pcref protects and which must not pin a Tasks RCU grace period. x10/x11 are scratch at every emission point; the fmod_ret cbnz target lies after the decrement/increment pair around the original call, and the ip_after_call nop follows the re-increment, so the count is balanced on every path. BUILD_BUG_ON guards the LDR/STR immediate range for the task_struct offset. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/net/bpf_jit_comp.c | 46 +++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 46 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41db..5c9a7bde5cc9 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -2591,6 +2591,34 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx,= u8 dst, u8 src, bool nullab emit(A64_SUB(0, dst, src, base_lo), ctx); } =20 +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). + * + * mrs x10, sp_el0 + * ldr w11, [x10, #offsetof(struct task_struct, rcu_tramp_nesting)] + * add/sub w11, w11, #1 + * str w11, [x10, #...] + * + * x10/x11 are scratch in the trampoline at every point this is emitted. + */ +static void emit_rcu_tasks_tramp_nesting(struct jit_ctx *ctx, bool enter) +{ +#ifdef CONFIG_TASKS_RCU + const int off =3D offsetof(struct task_struct, rcu_tramp_nesting); + const u8 tsk =3D A64_R(10), cnt =3D A64_R(11); + + BUILD_BUG_ON(off & 3 || off >=3D SZ_16K); /* LDR/STR (imm12, scaled) */ + + emit(A64_MRS_SP_EL0(tsk), ctx); + emit(A64_LDR32I(cnt, tsk, off), ctx); + if (enter) + emit(A64_ADD_I(0, cnt, cnt, 1), ctx); + else + emit(A64_SUB_I(0, cnt, cnt, 1), ctx); + emit(A64_STR32I(cnt, tsk, off), ctx); +#endif +} + static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, const struct btf_func_model *m, const struct arg_aux *a, bool for_call_origin, bool is_struct_ops, u64 arena_base) @@ -2854,6 +2882,13 @@ static int prepare_trampoline(struct jit_ctx *ctx, s= truct bpf_tramp_image *im, emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx); =20 + /* + * From here until the matching decrement in the epilogue, a preemption + * of this task is not a Tasks RCU quiescent state. The instructions + * above this point are covered by the irq-exit IP check. + */ + emit_rcu_tasks_tramp_nesting(ctx, true); + if (flags & BPF_TRAMP_F_CALL_ORIG) { /* for the first pass, assume the worst case */ if (!ctx->image) @@ -2898,12 +2933,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, = struct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_CALL_ORIG) { /* the original func takes kernel addresses, never converted ones */ save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0); + /* + * The original function may run for a long time without + * sleeping; do not let it pin a Tasks RCU grace period. The + * trampoline frame above it is held by im->pcref + * (__bpf_tramp_enter()), not by Tasks RCU, across the call. + */ + emit_rcu_tasks_tramp_nesting(ctx, false); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); emit(A64_RET(A64_R(10)), ctx); /* store return value */ emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); + emit_rcu_tasks_tramp_nesting(ctx, true); /* reserve a nop for bpf_tramp_image_put */ im->ip_after_call =3D ctx->ro_image + ctx->idx; emit(A64_NOP, ctx); @@ -2945,6 +2988,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, st= ruct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_RESTORE_REGS) restore_args(ctx, bargs_off, a->regs_for_args); =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_rcu_tasks_tramp_nesting(ctx, false); + /* restore callee saved register x19 and x20 */ emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx); --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066307; cv=none; d=zohomail.com; s=zohoarc; b=Lit5xxU9yL8zPeL21Kc1bMk8K4IwjrFS6q8W+BuVe/xkxmcmbfrvFO8mNFzwrHLmvIBGjbBdPlXsgdAr9hCFlgdNBSkyhM1xQ2LARQ6vmPuC56TPeRP4ApBAI9MrpkdVr6pBzQFrW9Sel5IT+PVJ3RGV9IXeKHnF5E3bxFAx+nw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066307; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=HOD3O7Zgq6szuGZyEhyAnv0PHDrKYei5bZUKVSYk0EKgMoiei+j8esKxpIfiDBzGXxccTdY9XskG5h9loJP/hnJux4dyMEbguMdO/E48BRgwceoD55bQANLTv4nE5EyYzPnAhrlu26TyGWrsRExSBdBunukwO7hs3VnBYSoHrCw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066307516109.5941443400186; Thu, 10 Sep 2026 11:51:47 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415168.1644732 (Exim 4.92) (envelope-from ) id 1x4jrg-0007Fn-UG; Thu, 10 Sep 2026 18:51:16 +0000 Received: by outflank-mailman (output) from mailman id 1415168.1644732; Thu, 10 Sep 2026 18:51:16 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrg-0007Fc-Ow; Thu, 10 Sep 2026 18:51:16 +0000 Received: by outflank-mailman (input) for mailman id 1415168; Thu, 10 Sep 2026 18:51:15 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrf-0006xY-DN for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:15 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jre-00BfN4-Pl for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:14 +0200 Received: from [10.42.69.11] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc1e-e002-0a2a0a5209dd-0a2a450bc178-6 for ; Thu, 10 Sep 2026 20:51:14 +0200 Received: from [209.85.222.54] (helo=mail-ua1-f54.google.com) by tlsNG-42698a.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc21-b7e8-0a2a450b0019-d155de36cd42-3 for ; Thu, 10 Sep 2026 20:51:14 +0200 Received: by mail-ua1-f54.google.com with SMTP id a1e0cc1a2514c-97ca74bd6e0so133957241.1 for ; Thu, 10 Sep 2026 11:51:14 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f47aeb7sm2503906d6.24.2026.09.10.11.51.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:12 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066273; x=1789671073; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=Kadw4Z2/sWrsZQ4RBjIq1A92bzOgmRIDQFsHMY6ls3XPG0iNxF5uxa7iMsUnuY8SgZ o+OHhjrVZj4ZM8zm1qkwxROheA0c8H7zbOIOwOH7qTBHAerKPw1ZuI6E11giqZE2FmDO hszt4sTydrMqO1LakF+p+iCMHZhOPWMBkA9LQ2yDGY43m8uPcUVO8CBszR3oR8mamZh3 TJI8v+05wegGK5Tv/Q0OFTKF5XZP4TwOESIVh/sY6Eatwc+bpgBMSS3zvjSmtwlXpTuM UyJqt5zn1HKGnSgVlllwhUVMKxMpVe1lOoRVigJnXHMXPc0QpcAfLKnxAsdMRJS3RJCs /xHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066273; x=1789671073; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=hGi7HXzkLZuNsReNiltvSGQbu5eCiP8RmUZh6mQPFbGwU63PWmWmpUJurwrWOdWGj+ S55eKzxAo3Lp79uwQVr8VPkMaISysGV8DuwghLdq71jgq0g1Wgjp1e7ApFpEEG3Rm5YT yWUuzxNlUcLL3u6JbE6k3sGPMwnzt58X6Zq8HOfXfEdaL6fQyl8eVGvVF9Ekas+Kgws3 0pZVxtxsWv11Oa5BHQN/LgDeC/FqN5Kw474nkQytDj4uSSTf5tJztWcYPFEOTK601ZLN SLL655hgz3aVLuV3V2TUasQ7sXYa8hkVeXY0ywbkAccsPOhpkhGIogThsvg1vyk0suw4 +y1g== X-Forwarded-Encrypted: i=1; AKwUvBzYkdbJ7tkyDkC0O19yN5SKLSNjBAG+f5j2fzue5tXr9COkWmqD9W8jY/KrWJY9qU66J6ULr/VJIUA=@lists.xenproject.org X-Gm-Message-State: AFuF++k+xU9P+v4vkW4oHRhyZFVDAaMcnxbAc9B6hEBcZff6yjH8lI1A xJSEJ8VpnITkAR9vun65CFac2drfdJb7LpuFc5+u9ADGW60fjy/NMfdJQtWiS26Mbu8= X-Gm-Gg: AYBFou0tkze9Gmwd9Bt7+qMEdHbnGZKE50+alExBvQg8gwvsWtEFvj9+siV8r6ugCEw 63OJEHdgqy9IaZXjsYYPUqzNwim/GaNA+wVm9Ayy0oRkOclCCj45yfP+Noyb0Bla3ry6QxnYGv6 inuZ/r1lpKNC84UF6fqcUVDMs2ZE+76c5spAjY7zUcv7HwvdmQ/Ie+5gnp/4zaDVwT5PCOsn4UY GRzzAPkVHQlQnK4gNuvU+uR77bkunp3fBhalXlcDVxsLfCajkUePle7b2aVh98pXX1irkz3C6wj 5bojXZC4iNZsRMamHedpXQ9ONe7uEQ5+0cklfhOXoze/wl3dK7sWrt77HDnULmaU4I49o4Ff77p RYsAX5GVMzJY8LRhKa3Aj44+dDOS+de8gliQ0nU4imFjog+LxmEP5uiDHVJxKznm7As421xMT7M 2T0UcVNiBAahB6ktYsKnY772W7vN9BnNXVm6ghKNlXZa6kqZnkN24Bvekg5RxMGfK5b4iSd6ltk 2rfR/HfE+ImC1SScl0znN9faSqjLtRmWAetp4WjvLqXtBeuQ7AYLNdj X-Received: by 2002:a67:e117:0:b0:790:f691:2d5b with SMTP id ada2fe7eead31-792a5b8b8a7mr1152469137.4.1789066272932; Thu, 10 Sep 2026 11:51:12 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:34 +0000 Subject: [PATCH RFC 11/13] samples: ftrace: Maintain Tasks RCU trampoline nesting in direct-call trampolines MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-11-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-42698a/1789066274-AA4DB9EA-9A047A62/0/0 X-purgate-type: clean X-purgate-size: 10813 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066309388158500 Follow the register_ftrace_direct() contract in the sample modules: on x86-64 and arm64, have each hand-written trampoline increment current->rcu_tramp_nesting before calling its C handler and decrement it before returning, via a small shared samples/ftrace/ftrace-direct.h. %r11 and x12/w13 are used as scratch; both are caller-saved, non-argument registers and therefore dead on entry to and exit from an fentry trampoline. The header pulls in the generated asm-offsets.h only on those two architectures, since it is not generally safe to include from C (PPC32's TASK_SIZE and arm64's TRAMP_VALIAS clash with the C definitions; the latter is worked around locally with push_macro/pop_macro). Other architectures get empty macros and are unchanged. Assisted-by: LLM Signed-off-by: Josef Bacik --- samples/ftrace/ftrace-direct-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi.c | 5 +++ samples/ftrace/ftrace-direct-too.c | 5 +++ samples/ftrace/ftrace-direct.c | 5 +++ samples/ftrace/ftrace-direct.h | 64 +++++++++++++++++++++++++= ++++ 6 files changed, 97 insertions(+) diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-= direct-modify.c index 164d9dd6fd92..eb8230fa4242 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -73,7 +74,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func1\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -85,7 +88,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func2\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -141,11 +146,13 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func1\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -153,11 +160,13 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func2\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/f= trace-direct-multi-modify.c index b03766c6217b..c8f1062e5d1a 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -77,10 +78,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -92,10 +95,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -154,6 +159,7 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -162,6 +168,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -169,6 +176,7 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -177,6 +185,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-d= irect-multi.c index 3fe6ddaf0b69..bc6a88dd4ffc 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #include #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include @@ -56,10 +57,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -101,6 +104,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -109,6 +113,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-dir= ect-too.c index bf2411aa6fd7..247e418644a2 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -61,6 +62,7 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n" @@ -70,6 +72,7 @@ asm ( " popq %rdx\n" " popq %rsi\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -110,6 +113,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #48\n" " stp x9, x30, [sp]\n" " stp x0, x1, [sp, #16]\n" @@ -119,6 +123,7 @@ asm ( " ldp x0, x1, [sp, #16]\n" " ldp x2, x3, [sp, #32]\n" " add sp, sp, #48\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 5368c8c39cbb..9e1964baf28b 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -3,6 +3,7 @@ =20 #include /* for wake_up_process() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -54,9 +55,11 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -97,6 +100,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -104,6 +108,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h new file mode 100644 index 000000000000..d0313f33f47f --- /dev/null +++ b/samples/ftrace/ftrace-direct.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _SAMPLES_FTRACE_DIRECT_H +#define _SAMPLES_FTRACE_DIRECT_H + +#include + +/* + * A direct-call trampoline is entered with no lock, refcount or RCU marker + * held; only Tasks RCU keeps it (and, for a module, its text) alive while= a + * task is inside it or preempted in something it called. On architectures + * that select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU + * quiescent state unless current->rcu_tramp_nesting is non-zero, so the + * trampoline must raise it before calling out and drop it afterwards, exa= ctly + * like the ftrace and BPF trampolines do. See rcu_tasks_trampoline_enter= () + * and register_ftrace_direct(). The instructions before the increment and + * after the decrement are covered by ftrace_direct_mark_module(). + * + * These expand to instruction strings for use inside the samples' asm() + * trampolines. The scratch register is caller-saved and not an argument + * register, so it is dead on entry to and exit from an fentry trampoline. + * + * The generated asm-offsets.h is only pulled in on the architectures that= need + * it here: it is not generally safe to include from C (e.g. PPC32's TASK_= SIZE + * and arm64's TRAMP_VALIAS clash with the C definitions), which is why the + * samples themselves guard their own include of it. + */ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_X86_64) + +#include + +#define RCU_TASKS_TRAMP_ENTER \ + " movq %gs:current_task(%rip), %r11\n" \ + " incl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" +#define RCU_TASKS_TRAMP_EXIT \ + " movq %gs:current_task(%rip), %r11\n" \ + " decl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" + +#elif defined(CONFIG_TASKS_RCU) && defined(CONFIG_ARM64) + +/* arm64's asm-offsets.h redefines TRAMP_VALIAS from . */ +#pragma push_macro("TRAMP_VALIAS") +#undef TRAMP_VALIAS +#include +#pragma pop_macro("TRAMP_VALIAS") + +#define RCU_TASKS_TRAMP_ENTER \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " add w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" +#define RCU_TASKS_TRAMP_EXIT \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " sub w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" + +#else + +#define RCU_TASKS_TRAMP_ENTER +#define RCU_TASKS_TRAMP_EXIT + +#endif + +#endif /* _SAMPLES_FTRACE_DIRECT_H */ --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066298; cv=none; d=zohomail.com; s=zohoarc; b=KgoMl/G8ud5crsp/IX3zxNtq/pBucTZKVekj7m/T1zpcV05y2FipASUPKK6y3RvPJFUgG5uCADxcGntUAW/e/xS5VRXtz7SmROQo8BYzhWcPm+vT1Y2RwMTGlZtRhJxpBWiYhZLDztqFOWx6fA+cQBEZrStwieu9UGzoG0auz8w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066298; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=twz7P/UgrzFqFrDTfXn+06wBF94/L3aT/pnjRwe1Smk=; b=DqiQLlMkrw//H8thMmqtc4U/J5cgrGn/tZYxA4Zi3ayiW2B/Nhkr5ikRqphS/iyy84KR7HfPs1E5Xz8ECtnAPfkd7rPiF9JDtQtwj029HAd8BMH7Iaf2FlnP1oaldo1GjQy0g8wQHKb5UIP9DglZ2WxnWn2KTKwaTq3L6VHvjoQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066298120205.23036417876767; Thu, 10 Sep 2026 11:51:38 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415172.1644741 (Exim 4.92) (envelope-from ) id 1x4jrj-0007i6-BA; Thu, 10 Sep 2026 18:51:19 +0000 Received: by outflank-mailman (output) from mailman id 1415172.1644741; Thu, 10 Sep 2026 18:51:19 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrj-0007gV-4m; Thu, 10 Sep 2026 18:51:19 +0000 Received: by outflank-mailman (input) for mailman id 1415172; Thu, 10 Sep 2026 18:51:17 +0000 Received: from mx.expurgate.net ([194.145.224.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrh-0007L9-Dn for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:17 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrg-009F3X-QO for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:16 +0200 Received: from [10.42.69.9] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc22-8faa-0a2a0a5109dd-0a2a4509ce40-8 for ; Thu, 10 Sep 2026 20:51:16 +0200 Received: from [209.85.222.178] (helo=mail-qk1-f178.google.com) by tlsNG-bad1c0.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc23-be1a-0a2a45090019-d155deb2e929-3 for ; Thu, 10 Sep 2026 20:51:16 +0200 Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-92edb12cdf2so541995785a.3 for ; Thu, 10 Sep 2026 11:51:16 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e7f184f5sm49053285a.15.2026.09.10.11.51.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:14 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066275; x=1789671075; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=twz7P/UgrzFqFrDTfXn+06wBF94/L3aT/pnjRwe1Smk=; b=a/W+E3oqSR9eqkKr341DAJy1tNKaFaQBR82chod6S1YlxphovVFLNAMXtqIvf8WOvK FSQk35n2Wzlxm/D9zLeEY5lh9DgKjyMKron8scWdp6Y9A0j5cmSKLM/kCLFxlDSxPM9D QsFSU4p3CqRtKttdxiELkDbyLUy1EOr/GUkP+1yVNbEsi/NjijRlusDHWgE8RkpC0OiH WaaSQqnngGs0jCAZfdPqHyobeQCRewa0U9ExLrcvbDDRFQa66LC2r610vdTP+AsYuCtf xQbTcKtHoGFlMFYQ/EYB4L4v5MDkmDj24cDaxLtQupeMabo+yPoCTlwZEOyOMVfSf4Pb 5PnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066275; x=1789671075; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=twz7P/UgrzFqFrDTfXn+06wBF94/L3aT/pnjRwe1Smk=; b=WXGKxb3Q6YNYI8yaewDqDgA4cfBiGDsxVNunbLywB8OL2h59szXIn02aO4GCwDPfq5 wqEZ4uNCg2jzBoJx6q3ZmliL4d8iy34KeBNMg/wRoTrKnFyxvqzVpEGVkvrwH5438hdq 8HJQN35hy/o+omZiqVGBL3qdZ4U8uIDG6eaL4j2b4gdcc2LiH8jwaavpJkHXNy2h1eDu vJcJoVKb5qT2igyvmcMHwdr9g2LHGNcz3tDlGCk85okgP2qQ+BlZ4XjA8CvYN7RC087e PkMKNr7M0BhjBlge0p1k9Od6NXv4tKWiSJyE78CzrGursprcNlv5OylekrfQzzr3icyy 2xlg== X-Forwarded-Encrypted: i=1; AKwUvBwt8s9YCby8blA7OeByO02KfwWKmF74UYLsZKaTfWTuKj79J8HYl5kVh/j5Ei3z1QxmOh/4svuOiZk=@lists.xenproject.org X-Gm-Message-State: AFuF++nIhZdcqnTU2jDCVgDUx6JqbvY/NfyPlEeMc69ehZxjWx0x1NAd AsCsq/KSc1DwHVubYl9YEvlAHYYU9i0JRv9HLVLsJj3alw5VthsD5pqnJdWrzIvZQG8= X-Gm-Gg: AYBFou0RWmVthrE8BKK2CB8KD6TfguQimw5kS0qUPLtegP4MWhS9oNTd8LumDYDkC5z mOojv4ZWReRZAisyYBUcsTnU2DtBiEGiXctRbwz9cRzKfFIrDiyrvD6MJgL0duMGFkEUpDujlRB hxey7RYcXSpAQ6Uv+uNlc1EpzGAVXd/grT0qM0zpdV2jWj3Ocgj4IHCNFV3wWllshbzDkYHm6pq OayUsGZSfogqz8I2ZYCnDpj44IFqMjhEw3BQUcMCYFTo/s3Dc3IyZ9NER8YYUhvotLVNLjbYtTQ gMNrNQ/qqLZKvu+eDBO38wc0mzyk04z0AdlvlIGv6h9Wo2RE0ZIFxFhBS2ATwyKi7ZVmr7q1XDG BvAt/z2qy2jACf0CPgrBKziYSYs8G8qfN7ehSQp8EVZrt2f/d4sQjBmUKxuvwsA0fRvoNKrsTxK HWP3fwNu3EZQ+/TVN0HlZdEc/Ct+cq4bqdRHMjzeVw2zdcoeYYYhq+2101pApcMcjtHGco1zrKE Hm27BqPM/lKR5zCUglIbdnPzGbmqBxMxzHmI23K0x/RMtf/FZY70sg/ X-Received: by 2002:a05:620a:269d:b0:939:6de9:208b with SMTP id af79cd13be357-939ea2b3827mr16779985a.48.1789066274997; Thu, 10 Sep 2026 11:51:14 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:35 +0000 Subject: [PATCH RFC 12/13] rcutorture: Bracket Tasks RCU readers with trampoline nesting MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-12-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-bad1c0/1789066276-FC817034-32A0BD17/0/0 X-purgate-type: clean X-purgate-size: 1494 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066299381158500 rcutorture's tasks flavor models a Tasks RCU reader as "any stretch of kernel code", and rcu_read_delay() deliberately preempts inside it to check that a preemption does not end the read-side critical section. Once preemption outside a trampoline becomes a quiescent state that model no longer matches what Tasks RCU protects, and the readers would report false too-short grace periods. Have tasks_torture_read_lock()/unlock() raise and drop current->rcu_tramp_nesting so the reader models a trampoline, which is the thing Tasks RCU actually guards; the deliberate preemption inside it then continues to be, correctly, not a quiescent state. Assisted-by: LLM Signed-off-by: Josef Bacik --- kernel/rcu/rcutorture.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 794937e13e7c..df6dd708cea7 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1144,11 +1144,17 @@ static struct rcu_torture_ops trivial_preempt_ops = =3D { =20 static int tasks_torture_read_lock(void) { + /* + * Model a trampoline: with CONFIG_RCU_TASKS_PREEMPT_QS a preemption is + * otherwise a quiescent state and rcu_read_delay() preempts on purpose. + */ + rcu_tasks_trampoline_enter(); return 0; } =20 static void tasks_torture_read_unlock(int idx) { + rcu_tasks_trampoline_exit(); } =20 static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) --=20 2.55.0 From nobody Thu Sep 24 19:51:08 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789066307; cv=none; d=zohomail.com; s=zohoarc; b=byJihfyQ6RLC51w3y4tgu1PXLQJBVbSXfpdiIzJfLedZo9OnBlPTByKYCq6VWhT5OIa/bUcitEIwpNs2ZFBHvq9juGodA74oyfgbU9WQ1u8YEp3EEe0OIAnNXQXMXibDXmrLP3wYWaVPMScFyKVa++lT2g40pcSdtycckHAKXRs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789066307; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=lBIdGtm0mo8q5590yjvGhNrcgQrhEpuiN5lFQ+hlgEA=; b=BxYTIcz51lSKnYYx1HX1iTevnmPCMnz2SVsifE5o1UaA5Renb75M0IWeaCN9k6wIHVI7/K1VZtaKqSjsVPBKG6iBPhtiBj8NkQhil+OY3HaJLPL/+Dh4vVyhTGprxOSuiPa9amwVc8ww3Hi5gd795iXlJqV1NofZHvOkq+2GBHs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789066307078344.06740332210416; Thu, 10 Sep 2026 11:51:47 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1415175.1644750 (Exim 4.92) (envelope-from ) id 1x4jrm-0008Fj-I8; Thu, 10 Sep 2026 18:51:22 +0000 Received: by outflank-mailman (output) from mailman id 1415175.1644750; Thu, 10 Sep 2026 18:51:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrm-0008FG-DG; Thu, 10 Sep 2026 18:51:22 +0000 Received: by outflank-mailman (input) for mailman id 1415175; Thu, 10 Sep 2026 18:51:20 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x4jrk-0007tI-6l for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 18:51:20 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x4jrj-00BfN4-K3 for xen-devel@lists.xenproject.org; Thu, 10 Sep 2026 20:51:19 +0200 Received: from [10.42.69.7] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aa2fc04-e002-0a2a0a5209dd-0a2a4507a3ac-48 for ; Thu, 10 Sep 2026 20:51:19 +0200 Received: from [209.85.222.171] (helo=mail-qk1-f171.google.com) by tlsNG-ef75cf.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aa2fc26-b4ea-0a2a45070019-d155deabed0b-3 for ; Thu, 10 Sep 2026 20:51:19 +0200 Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-9377f717879so14385a.1 for ; Thu, 10 Sep 2026 11:51:19 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80f8142sm49848385a.43.2026.09.10.11.51.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Sep 2026 11:51:17 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789066278; x=1789671078; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lBIdGtm0mo8q5590yjvGhNrcgQrhEpuiN5lFQ+hlgEA=; b=n70nH+yfFKCjVXMvhQx9IC0guXrh6b42XHu/VnPWPC/DwaNL+S4GVG945Var7cJq1T A1rISvERthLCOwTYQoccWBVof9gfUWEWT+T47NE151PRpuz7eIfZzfKZhWuBScjW+oIR 7JIPIbBdr0TpzYggSKMaNUmaTxLJy3uKmmQKWBOKpyOVmcQs7gb+EE/5o/cxoEsCz0Ob 3KvMVDG0U5fFBgcKPTyZFxTo1K5CeqEHtc6L5v1Qnpl1h8PrB7O7U4tGK5jas7WEqgX6 0PzS0lPDRsDzGM5Zr1QvSKLEzO+RBo+wxOtB/0eTKrRCFClp1TMZFxX+bN8uRceK/ZCx FtWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789066278; x=1789671078; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=lBIdGtm0mo8q5590yjvGhNrcgQrhEpuiN5lFQ+hlgEA=; b=S+jAaf9mhEZlACR1ak9hX7UWkXUFW0QjEz0AMWUMdk9C3C+IGbjUZ9Q5ZygldGQ8iK sJ86pH63GvkySavKRiFH17R+UrAxo/6OlTzmk5BYd0oqUmf+hRmsxbzmh/T5KZ1ttFTI B3I74inUNjNKeOKJ0DBU7ZgM/zgCGHemJOX2Djd0pA3UOBhdr4F02nLLljO5bxmURNLX D9muqzut70epzOo8NbMIVZKaxg8QUCskGOXFyoenLf6Q8Gmi6qRgTClEmQg27ePtUGpL DH09qfM9ZAdWxdfmqfgxYPhmWq4zmbeipy8JYJaryzzhCJovA8yN9CwJJaNn+hBK/vK0 Ankw== X-Forwarded-Encrypted: i=1; AKwUvByE8aUBdUXLw5fZZ2Q16TyO2EWc6+hbUdKylP4mpVdaPrhKcsslq/HV0lRy1QkCGSjmByb9+CIRk34=@lists.xenproject.org X-Gm-Message-State: AFuF++lEGdbRRPUAUtKQoa4FnBLfTWEZAkyfSKHNqU7rgHU9aJ4x9x1z s8FKPngDVGz7zQxS+f0DAc6UgNsB+GHDJzMsFd+xHNAVoGYkZSvtoqUP2Gw7+YaH9ss= X-Gm-Gg: AYBFou275ImTeqwBZ1FH67IRvTrz3uYnAFv2wxF0+FgvNW6kv9rZ2QYLqTnMhlXsvHE crWmeSviRzfUMectFFgKkCmvYC8q+PEJXIgWWGtew9bqHep7Tu3BvMiuk0HKRh1cDmDoRFWdge5 NOn3lU+9lMr3BzsdS8CFfkPt7wKYEq0otNpl98kxVNqa8WMS1W6/g2vgqm/9zTVF8x7NpRSJiAo 5daAQ6ngB1ufvkANoxPKfz/t5+VznfbjpXrdAE9E57z/A2m4YzXhKu4Ut72JZgqPLevBvvrtRh+ OmRz1DuOM9uA6dw5BA9EdWNfjr3MhNIheM2YhKTt/vC8l6KrkyuDT3+1zVTx5SwNA3JguXfvveM o8WLag2NElue0gGUi/q1EzlFYuOVcHu6qXUlQnTUXr59ZVsflog0ev7urhCitzvS7bQ0BywOFGn +QYrNYE5aeojMNNI9o9L0K4mI7KL2xk8DtyJO31MaZ7pGNYdUSribIM0AvFuXhuQqxoAaWWCNkW zvSDL9ty9SctQu75RGM98uioR/f9TWXrGSCMM9tjbGE9uD4Qb4gcO0k X-Received: by 2002:a05:620a:6a04:b0:937:4e12:5db6 with SMTP id af79cd13be357-939ea2a1e5dmr15410485a.43.1789066277808; Thu, 10 Sep 2026 11:51:17 -0700 (PDT) From: Josef Bacik Date: Thu, 10 Sep 2026 18:50:36 +0000 Subject: [PATCH RFC 13/13] rcu-tasks: Treat preemption outside trampolines as a quiescent state MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260910-b4-rcu-tasks-preempt-qs-v1-13-d4469f4cc101@toxicpanda.com> References: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> In-Reply-To: <20260910-b4-rcu-tasks-preempt-qs-v1-0-d4469f4cc101@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-purgate-ID: tlsNG-ef75cf/1789066279-A46CAAE4-03B96413/0/0 X-purgate-type: clean X-purgate-size: 9398 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789066307323158500 Tasks RCU only accepts a voluntary context switch, usermode or idle as a quiescent state, because a task that was preempted may be sitting in a trampoline whose text is about to be freed. On PREEMPT_LAZY kernels, where cond_resched() is a no-op and CPU-bound kernel threads only ever lose the CPU through preemption, that means any long-running kthread or kworker stalls every synchronize_rcu_tasks() caller -- BPF and LSM program detach and DYNAMIC ftrace_ops teardown via ftrace_shutdown(), and kprobe (un)registration via the jump optimizer, which waits under kprobe_mutex, text_mutex and cpus_read_lock() -- for its entire run, unless someone sprinkles cond_resched_tasks_rcu_qs() into it. A cgroup writeback worker draining a large cgwb for eleven minutes was enough to back 40+ tasks up behind trampoline_mutex and trip the hung-task panic. With the previous patches, every Tasks-RCU-protected trampoline on x86-64 and arm64 (ftrace_caller and its dynamic copies, BPF trampoline images, the optprobe template, out-of-line direct trampolines) holds current->rcu_tramp_nesting across its call-out, and the irq-exit preemption path holds it across preempt_schedule_irq() whenever the interrupted IP is somewhere the counter cannot cover: trampoline entry/exit instructions and other dynamically allocated text, the static ftrace stubs and x86 return thunks on the way into a direct-call target, modules hosting their own direct trampolines, and the kprobe jump-optimization window. A task that is context-switched with the count at zero therefore cannot be inside, called from, or about to resume into anything Tasks RCU protects. So let rcu_tasks_classic_qs() clear the holdout flag on a preemption too when rcu_tramp_nesting is zero, on architectures that select ARCH_HAS_RCU_TASKS_PREEMPT_QS, and select it for x86-64 and for arm64 with DYNAMIC_FTRACE_WITH_ARGS. A running holdout is already poked via rcu_request_urgent_qs_task(), which makes the next tick set NEED_RESCHED; the resulting preemption -- from irq exit, or synchronously at the next preempt_enable() -- now retires it, so a Tasks RCU grace period is bounded by roughly a tick plus the longest preempt-disabled section instead of by the longest stretch without a voluntary schedule(). Other architectures keep the voluntary-only rule. Update the Tasks RCU documentation comments and the FORCE_TASKS_RCU help text to match. Cost: one load of current plus an inc/dec per trampoline entry and exit, and on irq-exit preemption one core_kernel_text() check plus, with OPTPROBES, MAX_OPTIMIZED_LENGTH-1 lockless kprobe hash lookups. Not covered: x86-32 and the other GENERIC_IRQ_ENTRY architectures, and return_to_handler / the rethook trampoline, whose C callees take the ftrace recursion lock before touching any ops. Tested under QEMU (x86-64, PREEMPT_LAZY, PREEMPT_RCU=3Dn, PROVE_RCU, with and without PREEMPT_DYNAMIC) against a kthread spinning in-kernel for 30s with the function tracer, an ftrace kprobe, an optimized kprobe and fentry/fexit programs live: synchronize_rcu_tasks() 29.7s -> 0.1-0.3s, ftrace_shutdown() of a DYNAMIC ops 27s -> 0.2-0.8s, the ftrace-direct sample modules load/fire/unload in ~2.5s each during the spin, no warnings. arm64 is build-tested only. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + include/linux/rcupdate.h | 14 +++++++++++++- kernel/rcu/Kconfig | 7 ++++--- kernel/rcu/tasks.h | 15 +++++++++++---- 5 files changed, 30 insertions(+), 8 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b5a51b0ef944..0e6c1e0b236f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -44,6 +44,7 @@ config ARM64 select ARCH_HAS_PREEMPT_LAZY select ARCH_HAS_PTDUMP select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_RCU_TASKS_PREEMPT_QS if DYNAMIC_FTRACE_WITH_ARGS select ARCH_HAS_HW_PTE_YOUNG select ARCH_HAS_SETUP_DMA_OPS select ARCH_HAS_SET_DIRECT_MAP diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..0a6427019345 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -99,6 +99,7 @@ config X86 select ARCH_HAS_PREEMPT_LAZY select ARCH_HAS_PTDUMP select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_RCU_TASKS_PREEMPT_QS if X86_64 select ARCH_HAS_HW_PTE_YOUNG select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index e9afbbb1b061..356b1d6ef226 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -204,6 +204,11 @@ bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip); * non-trampoline user, kprobe jump optimization, which waits for tasks * preempted inside the instruction bytes it is about to overwrite. * + * With both in place, on architectures that select + * ARCH_HAS_RCU_TASKS_PREEMPT_QS, a preemption with rcu_tramp_nesting =3D= =3D 0 is + * a Tasks RCU quiescent state, and a CPU-bound kernel thread no longer ne= eds + * to volunteer one via cond_resched_tasks_rcu_qs(). + * * Only current writes the count and only current (or an interrupt on the = same * CPU) reads it, so plain accesses suffice. */ @@ -228,9 +233,16 @@ static __always_inline void rcu_tasks_trampoline_asser= t_none(void) =20 bool rcu_tasks_ip_in_trampoline(unsigned long ip); =20 +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +#define rcu_tasks_preempt_is_qs(t) (!READ_ONCE((t)->rcu_tramp_nesting)) +#else +#define rcu_tasks_preempt_is_qs(t) false +#endif + # define rcu_tasks_classic_qs(t, preempt) \ do { \ - if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ + if (READ_ONCE((t)->rcu_tasks_holdout) && \ + (!(preempt) || rcu_tasks_preempt_is_qs(t))) \ WRITE_ONCE((t)->rcu_tasks_holdout, false); \ } while (0) void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func); diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 999f8228a13d..8e7c94329105 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -94,9 +94,10 @@ config FORCE_TASKS_RCU default n help This option force-enables a task-based RCU implementation - that uses only voluntary context switch (not preemption!), - idle, and user-mode execution as quiescent states. Not for - manual selection in most cases. + that uses only voluntary context switch (not preemption, unless + the architecture selects ARCH_HAS_RCU_TASKS_PREEMPT_QS and the + task is outside any trampoline), idle, and user-mode execution + as quiescent states. Not for manual selection in most cases. =20 config NEED_TASKS_RCU bool diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index df68a330769a..78d2b78d3043 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -905,7 +905,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) // // Simple variant of RCU whose quiescent states are voluntary context // switch, cond_resched_tasks_rcu_qs(), user-space execution, and idle. -// As such, grace periods can take one good long time. There are no +// With CONFIG_RCU_TASKS_PREEMPT_QS, a preemption taken while the task is +// not inside a trampoline (current->rcu_tramp_nesting =3D=3D 0, see +// rcu_tasks_trampoline_enter()) is a quiescent state as well; without it, +// grace periods can take one good long time. There are no // read-side primitives similar to rcu_read_lock() and rcu_read_unlock() // because this implementation is intended to get the system into a safe // state for some of the manipulations involved in tracing and the like. @@ -1246,8 +1249,11 @@ static void tasks_rcu_exit_stall(struct timer_list *= unused) * period elapses, in other words after all currently executing rcu-tasks * read-side critical sections have completed. call_rcu_tasks() assumes * that the read-side critical sections end at a voluntary context - * switch (not a preemption!), cond_resched_tasks_rcu_qs(), entry into idl= e, - * or transition to usermode execution. As such, there are no read-side + * switch, cond_resched_tasks_rcu_qs(), entry into idle, transition to + * usermode execution, or, with CONFIG_RCU_TASKS_PREEMPT_QS, a preemption + * taken outside any trampoline (current->rcu_tramp_nesting =3D=3D 0, see + * rcu_tasks_trampoline_enter()); otherwise a preemption is not a + * quiescent state. As such, there are no read-side * primitives analogous to rcu_read_lock() and rcu_read_unlock() because * this primitive is intended to determine that all tasks have passed * through a safe state, not so much for data-structure synchronization. @@ -1269,7 +1275,8 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks); * executing rcu-tasks read-side critical sections have elapsed. These * read-side critical sections are delimited by calls to schedule(), * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls - * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched(). + * to synchronize_rcu_tasks(), (in theory, anyway) cond_resched(), and, + * with CONFIG_RCU_TASKS_PREEMPT_QS, preemption outside any trampoline. * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function --=20 2.55.0