From nobody Thu Sep 24 23:33:31 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B9FA3D9042 for ; Fri, 18 Sep 2026 14:49:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789742989; cv=none; b=ehJPF+l13UsdTukgSD7A+9xbpJYiiKD9nPESUB21GVePCdA6jD2fILyfAFX31LUV5BBZeYZ5endq/8jaSyffMwNtCaRqw3OezT+ABi3F0cQvrWFP11y9IOImGmaBGF2iEBTuk7GgCJl+ZxO3/Yvr/9tUDrb9bM7kB7QO5YYMQIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789742989; c=relaxed/simple; bh=vAWpKfoMcBcYFQiIgPpQZt2gAuw45VdrXu4vr32WnR0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=n3DXxK7kywnip3S8E8whvzVcBRR2CvNpe578WTKxCuVlFFVHbb4EbI16A+weaQEQ3mQa4vRyJdc8yDKAyw0+dgI7NRzacg7K1elbCrlW0DKLkQhOl5Z2UHZ1Z4VViXxKMxxnbs7L3ceaW50r1aTugJWqrkTPgoFkMiM/vpXb6sI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=EGgWMV6z; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="EGgWMV6z" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-52fb76bcb1eso9540131cf.3 for ; Fri, 18 Sep 2026 07:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742986; x=1790347786; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fr6kFLYx3hqPZHfOilkUN+oxGtJHBsVQ4xuLKdXd4To=; b=EGgWMV6z55IWJOyB4m5gstrvtiOTmMAqGfGnjHURxdaGzeYa03npTcSQXQNWsp1OpG 1yWwSagHCAftWamOKnzrLXSxGiJylsxr1dLNHVkGrloWdpP39BYC3mezixCFJZcO5nwC WOOeu7aVYfU+EWYnCT3JXk5/BvlWKWuC4ejairXDOeZAb+z2Ay6+XhvEEYdYgMQBHtnY q6Pq/0g8lBxdkrvDj8N3hYyKwWUBzRvMrMDdE7Kf/m22XebcYb6bjfU0nC4Lses4n2sm J6geooV1c7gyLOcNoH9XvntGJiknXX/ERvaJHLTq1AObEkkRg2wtIx2owDlnuJjjmjJ0 4TmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742986; x=1790347786; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=fr6kFLYx3hqPZHfOilkUN+oxGtJHBsVQ4xuLKdXd4To=; b=ve0XKU3ViX0iSnQGGmJycO8qkpFKHqAIcZAjkd9c9ibXZSrmI6UXbbmyEio3g+gmDp /teK5zYNB1nZDotsa49HNcnUVPPmUqhEd/PEU8kzRs3HsNlXT4ubNDq0ZTzeOPpGTnt0 YVdBqXzUu4p5uI064+PfczCr0ZZYv7/faCp9p20ujPx9ODsS56sbGZ0Ukqz0fvRR8LGA Ax8dWr6CEXtDc3GFYeI942deztshww6H+gHT8aJlmGYJX50ih9zFpgb4K4mN4vKKdXwn Hdht38N+ewS8ubWmBGUbiX8PoxocE1Fzp9Oje0YB5PZUKnZVyAuT0vFouu7PGr0qRk4w 8r/g== X-Forwarded-Encrypted: i=1; AKwUvBxPAylKFUAA2K442JAImagxt2yG+W9wc+TpjzOk84vS8AfeJH9XyqmfocKLuCN48TtM63UrYi6+I3MCCi8=@vger.kernel.org X-Gm-Message-State: AFuF++mjAknpnSExWSiPwpGMXtGBe9nRptR/X35oxwttsxlicMxu1ePu dBS+Y+lRBwakZX5fw+kYAUtd07oIuEk8SfKVsXWJ/Iihif3gdAHXuwcmAAlYLmYavwc= X-Gm-Gg: AYBFou0JP9Pue7hCFaLTt66TE04oXb8IAgPMuVt/PjS99G9BHlLV5FYU+p4XUpHGaEx hZOtJXTa2e/7GW9c1z8dURycXpXCWFc3ileLTkzzfg/T4vieWViUqu49iwYZJY9uCwTMSIeVc0x 1+ylckRuHIEZtdl/xLsL03lijjxX4+MtPA7aodQyyr0gzHhKDC09/2XD0fVLBpCkyuA34dtNJpe 5BvZmcCCKDuw9SI/pEH/HRcuNB3ol3XE4JQZRzRTekuPa34lxLVfZ5U1tLp1jIM22Yxdq08MZl2 kgEq3l+HIzi2PjpQuf4lq/hUFmeTZ2ce5yidVcLeiEovofu7WwMfsKUZT1LGuOC1mqbFVboqFUs h9j15U23ZUnMLA9JToYnNQA10c8Tzv0y163Xw0eBuXPcz9XOZlVUTql2hICSEpt4Z3viB5aH/uv c4qcP6ftWYgaqTxBamasJTtebbzvPun2dIMVp17xt59X9M3p9LbiLeC31OS7cbwyK0rE0vEdp49 8VytELKsyRhv5Krp5eOILLwMhzPlSz9GxiBLCu6FTrDBw92RVistqX6jkRfYuVr/N8= X-Received: by 2002:a05:622a:652:b0:530:efbc:9a8e with SMTP id d75a77b69052e-5329e3767camr46085191cf.5.1789742985599; Fri, 18 Sep 2026 07:49:45 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532abe9a4f4sm3338251cf.0.2026.09.18.07.49.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:44 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:20 +0000 Subject: [PATCH RFC v4 01/13] entry: Pass pt_regs to irqentry_exit_cond_resched() Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-1-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742979; l=4297; i=josef@toxicpanda.com; h=from:subject:message-id; bh=vAWpKfoMcBcYFQiIgPpQZt2gAuw45VdrXu4vr32WnR0=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QLUE3k3CkS2ybrdoKxDg2rM2yr7vcCp6RSWtB3hgoOxoISfCxswmRrxWH4MEQd0eZavwNPRgmIn mcD0WDNOMyg4= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA The irq-exit preemption path is about to need the interrupted context's registers to decide whether the preemption may be reported to Tasks RCU as a quiescent state. irqentry_exit_to_kernel_mode_preempt() already has them; hand them down through irqentry_exit_cond_resched(), its PREEMPT_DYNAMIC static-call and static-key variants, and raw_irqentry_exit_cond_resched(). The only caller outside the generic entry code is Xen PV's upcall handler, which has regs as well. No functional change. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/xen/enlighten_pv.c | 2 +- include/linux/irq-entry-common.h | 13 +++++++------ kernel/entry/common.c | 6 +++--- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/arch/x86/xen/enlighten_pv.c b/arch/x86/xen/enlighten_pv.c index 2c64b388f616..3d85035f5624 100644 --- a/arch/x86/xen/enlighten_pv.c +++ b/arch/x86/xen/enlighten_pv.c @@ -739,7 +739,7 @@ __visible noinstr void xen_pv_evtchn_do_upcall(struct p= t_regs *regs) =20 inhcall =3D get_and_clear_inhcall(); if (inhcall && !WARN_ON_ONCE(state.exit_rcu)) { - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); instrumentation_end(); restore_inhcall(inhcall); } else { diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index 0bb6c03481fa..e19b41ee6b18 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -343,24 +343,25 @@ typedef struct irqentry_state { =20 /** * irqentry_exit_cond_resched - Conditionally reschedule on return from in= terrupt + * @regs: Pointer to pt_regs of interrupted context * * Conditional reschedule with additional sanity checks. */ -void raw_irqentry_exit_cond_resched(void); +void raw_irqentry_exit_cond_resched(struct pt_regs *regs); =20 #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) #define irqentry_exit_cond_resched_dynamic_enabled raw_irqentry_exit_cond_= resched #define irqentry_exit_cond_resched_dynamic_disabled NULL DECLARE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_res= ched); -#define irqentry_exit_cond_resched() static_call(irqentry_exit_cond_resche= d)() +#define irqentry_exit_cond_resched(regs) static_call(irqentry_exit_cond_re= sched)(regs) #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DECLARE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void); -#define irqentry_exit_cond_resched() dynamic_irqentry_exit_cond_resched() +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs); +#define irqentry_exit_cond_resched(regs) dynamic_irqentry_exit_cond_resche= d(regs) #endif #else /* CONFIG_PREEMPT_DYNAMIC */ -#define irqentry_exit_cond_resched() raw_irqentry_exit_cond_resched() +#define irqentry_exit_cond_resched(regs) raw_irqentry_exit_cond_resched(re= gs) #endif /* CONFIG_PREEMPT_DYNAMIC */ =20 /** @@ -465,7 +466,7 @@ static inline void irqentry_exit_to_kernel_mode_preempt= (struct pt_regs *regs, return; =20 if (IS_ENABLED(CONFIG_PREEMPTION)) - irqentry_exit_cond_resched(); + irqentry_exit_cond_resched(regs); } =20 /** diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e3d381fd3d25..e4acd50bd81a 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -134,7 +134,7 @@ static inline bool arch_irqentry_exit_need_resched(void= ); static inline bool arch_irqentry_exit_need_resched(void) { return true; } #endif =20 -void raw_irqentry_exit_cond_resched(void) +void raw_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!preempt_count()) { /* Sanity check RCU and thread stack */ @@ -150,11 +150,11 @@ void raw_irqentry_exit_cond_resched(void) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resc= hed); #elif defined(CONFIG_HAVE_PREEMPT_DYNAMIC_KEY) DEFINE_STATIC_KEY_TRUE(sk_dynamic_irqentry_exit_cond_resched); -void dynamic_irqentry_exit_cond_resched(void) +void dynamic_irqentry_exit_cond_resched(struct pt_regs *regs) { if (!static_branch_unlikely(&sk_dynamic_irqentry_exit_cond_resched)) return; - raw_irqentry_exit_cond_resched(); + raw_irqentry_exit_cond_resched(regs); } #endif #endif --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C2AA4B048D for ; Fri, 18 Sep 2026 14:49:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789742996; cv=none; b=cP5Rlc32qdFAvg3qa/w4V2dN+I1N3NVVOvA7PidVWWwkIGTjmfvq1/h+eHrISBq1yuPfhZ4dOF8OwOf0REHhVQhFL112005GXBCRg91GzD6xy+JRMaV0MmivpLMKRpkd9sfP/Ajxqdz5a7+qD2AiCmPptiGqhXUZ8JL+njNAxPQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789742996; c=relaxed/simple; bh=qhXNBlJJp01Ei07kqFZDAUfEsCu0gSp6eIBNqqgCukM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=g2+Gzmvh+xJIgoTY5s9msG3JjRDpa4dHeGzO4VyKxIzQNMUSMrIDzYywAFxCGA5Gan2na6U7yssxCTpodiATWr7DLK5qmYmxIUEutKcktc7bKFrmgmrlLGjvXmul3EEQE2pu/8yjmZ+cB+dlBNcm3VcrKzchTgFByzRYIo4L+wo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=O7qjVTZE; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="O7qjVTZE" Received: by mail-qk2-f13.google.com with SMTP id af79cd13be357-93a0299c787so63380685a.1 for ; Fri, 18 Sep 2026 07:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742988; x=1790347788; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=myll629Qfz4YrwGmTVALnxB11mMoUz/LpucXoYBTMJw=; b=O7qjVTZEZZVg9nrIexxNs66QSwV91aC51kyoY501FP02Wsqt/4sbYBpC4hDuYEGW+w e9N8WAGthO1F8XRvxSRHKV3iHoKD6p8XEdrWjU2PE6ZwdDrs2AyvIBl5PgUmHguxa7Zz qpsmLuFNfY8V75JJHz7alQLZj8GZtkzqhutOk6G4WUsyQ/8litkVg9qVmbpVGRG9LabZ KT1CbYF89oX9y3Y4WYMdjV/pkszAjGvSvo3b7X4qhDGgcHOpab69A1k9JXPKn1pmsAZx H5dSJ9ANYDBT4UUrTIhAFvezISfWTaYPzB662tdJf6cNg9+DhKiuwA9v9229QLcI5a6m yX/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742988; x=1790347788; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=myll629Qfz4YrwGmTVALnxB11mMoUz/LpucXoYBTMJw=; b=kRRuE0rWHUmVDwyobYMWlHkMh5no9cBzzLz+ZGUk1Zg4Ghxayb9WZL8V35yoyEkX/8 R3EgWeINX4+aoqA8RKvJ+trvYW275sGnjxJDXruaGspg0zUB+/BOWvLkR3J3UPQ1UuEQ diG7SZXt5mLhGM3M4XMiC6BBEC3TKLGAGCbAjEcuisq2yrt6t8hMZyG1GlagRsyTwp9+ XZ7e8IFlzleHqgGt4Z/zj0qsRTf/l6lG5c/nXlyCAifKudLk3vLARgvAp0whWJVj8zsd 2PDTpdqIxigMN2VQCVbdyJSAFb1ceSV/0Lb7uTS1bOuO2xfp55jNytFcnRqXSghvQh7P 7uNA== X-Forwarded-Encrypted: i=1; AKwUvBx/fjZm3Ke/dV+gL2hECaLt3MAXsGOINXhfXWIUJElZ2YPaqnPU7V7laKS0xXBOLdvdKvLMq0ADbJ+R39k=@vger.kernel.org X-Gm-Message-State: AFuF++kntKNEOpZJI0CZRlikYDJ2VyqkIMbRAXYFT/PibrXrSsVXrCFG UqU6c0jRY67ZkYoeYOIrvo7ZUasGyRFbmCF4JMLmRy5MjwUoswpDWgKZQfI9080NNQw= X-Gm-Gg: AYBFou1JifbGC3gEc3H1taXk0vTw0ha2BlfAKqNguUTzYigzuDaf0A8lsOz3hGmn5Q0 J/8T2NHoURND33nxfJpHoPfKfTALx6or8OKU8AmnyQjg7QY/rrAJFZxl1RKfNlyiTCFYCHfsfc0 O1GWSj0m5Ptbi70DjFhsEFTTFcg8/v6bXfJ6R4Cntx30QSlHt6ZLl6U6kAN2a2KAGeLiyf9Xpmn sDp5bB267heHvCcoXhBybu13SgDCGNi9JapPyDfrwWh0Q8ki2i1DWXyv6APU+atDX1FddJqyQxI 5Z1UwFtdRONy/AER+WL0wRSzLeSQjQbGJetr5nO5hxY2WKDSTP0bff2rOkwdon+I81PyugWnoTI ABIJfpKqDRy3NoE883aUyKr0MXVernr/rhMHk8xHrlPiNEtVnHZH2UMgJpfJQZKIPAu9soI+zQt b24SZHcbkAIx5YFBLM8TGMKkvVDrXZFY8mAvg6Xd4DkkmZIY/ypE5wSgju9scXSDLqmxRG8glwa zi+4qvhWTAdRS+c1cWSWlq6fpOPAhW4DBXbUsGgWz5S3PQZI5rNLQ/v+XTmMzdpy1o= X-Received: by 2002:a05:620a:2983:b0:93b:d7a3:45d3 with SMTP id af79cd13be357-93bdc8b6a27mr357058685a.60.1789742988018; Fri, 18 Sep 2026 07:49:48 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9125800e368sm14410356d6.8.2026.09.18.07.49.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:47 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:21 +0000 Subject: [PATCH RFC v4 02/13] rcu-tasks: Add a Tasks RCU implementation for reader-marked trampolines Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-2-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742979; l=31523; i=josef@toxicpanda.com; h=from:subject:message-id; bh=qhXNBlJJp01Ei07kqFZDAUfEsCu0gSp6eIBNqqgCukM=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QKxRe2CaBiEe6pLgkvnyEKMb+x/jOl++z2W4Kx5A0gKz1iBWZzIljfaAJXHRJiKTIhXpTun61uP /1AaTpYl8kwI= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Tasks RCU waits for every task to pass through a voluntary context switch, usermode or idle, because a preempted task might be sitting in a trampoline that is about to be freed and nothing marks it as such. With PREEMPT_LAZY that is a poor fit for servers: cond_resched() is a no-op, so a CPU-bound kthread only ever leaves the CPU by preemption, and one such kthread holds every synchronize_rcu_tasks() caller -- ftrace and BPF trampoline teardown under their mutexes, the kprobe jump optimizer under text_mutex and cpus_read_lock() -- hostage for as long as it runs. Following the discussion on v2, take the other road: let the architecture make its trampolines Tasks Trace RCU readers. When an architecture selects HAVE_RCU_TRAMPOLINE_READERS it promises that every trampoline whose lifetime Tasks RCU guards enters rcu_read_lock_trace() (or its assembly equivalent) before calling out and leaves it before returning, so a task anywhere inside such a call-out, preempted or not, is an ordinary Tasks Trace reader. That leaves the few instructions of trampoline text before the reader is entered and after it is left (plus, in a later patch, the bytes a kprobe jump optimization is about to overwrite). A task can only linger there by being interrupted there, and such text never calls anything that schedules, so instead of tracking tasks we track CPUs: every pass through __schedule() is a per-CPU quiescent event, except that the one context switch that can catch a task at an arbitrary instruction -- a preemption from irq exit -- first records the interrupted IP in the task and parks it on a per-CPU list for the duration (reusing the fields and lists the classic flavor keeps for its exit-path bookkeeping), and, if the IP is inside such "unmarked" text, puts the task on a short holdout list; the task takes itself off at its next context switch outside such a preemption or irq-exit check that finds it elsewhere. Usermode (the existing tick hook, or a nohz_full CPU in an RCU extended quiescent state) and idle count as well. rcu_tasks_trampoline_text() does the classification: anything outside core and module text, plus an arch hook for things like static ftrace stubs and return thunks. The grace period, run by the existing rcu_tasks kthread so that call_rcu_tasks(), synchronize_rcu_tasks() and rcu_barrier_tasks() keep their names and callers, is: wait for every online CPU to context switch or be seen in an RCU extended quiescent state (nudging stragglers with resched_cpu() after a jiffy), drain the holdout list as it stood, synchronize_rcu_tasks_trace() for everything inside the readers, then one more CPU pass and drain for tasks that have since left the reader into the trailing instructions. That is bounded by a few jiffies, preempt-off latency and an SRCU grace period rather than by the longest stretch any task runs without sleeping, needs no per-task scan, and makes cond_resched_tasks_rcu_qs() unnecessary on such architectures. Unlike the classic flavor it also waits for an idle task caught in a trampoline, since an idle CPU only counts while RCU is not watching it. rcu_tasks_wait_irq_preempted() walks the parked lists for the one caller (the kprobe jump optimizer, later in the series) that makes ordinary text unsafe to be parked in and so has to wait out tasks that were preempted there before it said so. The classic implementation is untouched and remains the default; the new one is built only as CONFIG_TASKS_RCU_TRAMPOLINE_READERS when the architecture opts in and uses the generic irq entry code, whose reschedule check gains the rcu_tasks_irq_resched() call. Nothing selects it yet. Suggested-by: Paul E. McKenney Suggested-by: Alexei Starovoitov Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/rcupdate.h | 24 ++- include/linux/sched.h | 1 + kernel/entry/common.c | 8 +- kernel/fork.c | 1 + kernel/rcu/Kconfig | 22 +++ kernel/rcu/tasks.h | 459 +++++++++++++++++++++++++++++++++++++++++++= +++- kernel/rcu/update.c | 2 + 7 files changed, 508 insertions(+), 9 deletions(-) diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 44c07a66edff..79e4b14f83b9 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -50,6 +50,23 @@ token_context_lock_instance(RCU, RCU_BH); /* Exported common interfaces */ void call_rcu(struct rcu_head *head, rcu_callback_t func); void rcu_barrier_tasks(void); + +/* + * Trampoline-reader Tasks RCU (CONFIG_TASKS_RCU_TRAMPOLINE_READERS), see + * kernel/rcu/tasks.h. rcu_tasks_irq_resched_enter()/_exit() bracket the + * irq-exit preemption; rcu_tasks_trampoline_text() and the arch_ override + * classify an interrupted IP; rcu_tasks_wait_irq_preempted() lets a caller + * wait out tasks already preempted somewhere it is about to make unsafe. + */ +void rcu_tasks_irq_resched_enter(unsigned long ip); +void rcu_tasks_irq_resched_exit(void); +bool rcu_tasks_trampoline_text(unsigned long ip); +bool arch_rcu_tasks_trampoline_text(unsigned long ip); +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS +void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip)); +#else +static inline void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned lo= ng ip)) { } +#endif void synchronize_rcu(void); =20 /* @@ -180,11 +197,16 @@ static inline void rcu_nocb_flush_deferred_wakeup(voi= d) { } #ifdef CONFIG_TASKS_RCU_GENERIC =20 # ifdef CONFIG_TASKS_RCU -# define rcu_tasks_classic_qs(t, preempt) \ +# ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS +void rcu_tasks_note_qs(struct task_struct *t, bool preempt); +# define rcu_tasks_classic_qs(t, preempt) rcu_tasks_note_qs((t), (preempt= )) +# else +# define rcu_tasks_classic_qs(t, preempt) \ do { \ if (!(preempt) && READ_ONCE((t)->rcu_tasks_holdout)) \ WRITE_ONCE((t)->rcu_tasks_holdout, false); \ } while (0) +# endif void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func); void synchronize_rcu_tasks(void); void rcu_tasks_torture_stats_print(char *tt, char *tf); diff --git a/include/linux/sched.h b/include/linux/sched.h index 8b3d47a325cc..15beb44caa2c 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -957,6 +957,7 @@ struct task_struct { u8 rcu_tasks_holdout; u8 rcu_tasks_idx; int rcu_tasks_idle_cpu; + unsigned long rcu_tasks_irq_ip; struct list_head rcu_tasks_holdout_list; int rcu_tasks_exit_cpu; struct list_head rcu_tasks_exit_list; diff --git a/kernel/entry/common.c b/kernel/entry/common.c index e4acd50bd81a..94318519998c 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include =20 @@ -141,8 +142,13 @@ void raw_irqentry_exit_cond_resched(struct pt_regs *re= gs) rcu_irq_exit_check_preempt(); if (IS_ENABLED(CONFIG_DEBUG_ENTRY)) WARN_ON_ONCE(!on_thread_stack()); - if (need_resched() && arch_irqentry_exit_need_resched()) + if (need_resched() && arch_irqentry_exit_need_resched()) { + if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)) + rcu_tasks_irq_resched_enter(instruction_pointer(regs)); preempt_schedule_irq(); + if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)) + rcu_tasks_irq_resched_exit(); + } } } #ifdef CONFIG_PREEMPT_DYNAMIC diff --git a/kernel/fork.c b/kernel/fork.c index 416758c8a3d4..8077336bb136 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1871,6 +1871,7 @@ static inline void rcu_copy_process(struct task_struc= t *p) p->rcu_tasks_holdout =3D false; INIT_LIST_HEAD(&p->rcu_tasks_holdout_list); p->rcu_tasks_idle_cpu =3D -1; + p->rcu_tasks_irq_ip =3D 0; INIT_LIST_HEAD(&p->rcu_tasks_exit_list); #endif /* #ifdef CONFIG_TASKS_RCU */ #ifdef CONFIG_TASKS_TRACE_RCU diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index 332df7a7a634..bbab14bc14c3 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -107,6 +107,28 @@ config TASKS_RCU default NEED_TASKS_RCU && PREEMPTION select IRQ_WORK =20 +config HAVE_RCU_TRAMPOLINE_READERS + bool + help + Select this if the architecture uses the generic irq entry code and + every trampoline whose lifetime Tasks RCU guards on it (ftrace + trampolines, kprobe out-of-line and optimized-probe slots, BPF + trampolines, out-of-line ftrace direct-call trampolines) enters a + Tasks Trace RCU read-side critical section before calling out of + the trampoline and leaves it before returning, and any core text + that runs on behalf of such a trampoline outside that reader is + reported by arch_rcu_tasks_trampoline_text(). The assembly readers + use the this_cpu_inc() form of SRCU-fast, hence !NEED_SRCU_NMI_SAFE. + +config TASKS_RCU_TRAMPOLINE_READERS + def_bool TASKS_RCU && HAVE_RCU_TRAMPOLINE_READERS && GENERIC_IRQ_ENTRY &&= !NEED_SRCU_NMI_SAFE + select TASKS_TRACE_RCU + help + Implement the Tasks RCU grace period as a per-CPU pass over + context switches and irq-exit reschedules outside trampoline text + plus a Tasks Trace RCU grace period, instead of waiting for every + task to voluntarily context switch. See kernel/rcu/tasks.h. + config FORCE_TASKS_RUDE_RCU bool "Force selection of Tasks Rude RCU" depends on RCU_EXPERT diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 627295396cd9..f03be742be48 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -152,7 +152,7 @@ static struct rcu_tasks rt_name =3D \ .kname =3D #rt_name, \ } =20 -#ifdef CONFIG_TASKS_RCU +#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READ= ERS) =20 /* Report delay of scan exiting tasklist in rcu_tasks_postscan(). */ static void tasks_rcu_exit_stall(struct timer_list *unused); @@ -802,7 +802,7 @@ static void rcu_tasks_torture_stats_print_generic(struc= t rcu_tasks *rtp, char *t =20 #endif // #ifndef CONFIG_TINY_RCU =20 -#if defined(CONFIG_TASKS_RCU) +#if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMPOLINE_READ= ERS) =20 //////////////////////////////////////////////////////////////////////// // @@ -897,10 +897,444 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) rtp->postgp_func(rtp); } =20 -#endif /* #if defined(CONFIG_TASKS_RCU) */ +#endif /* #if defined(CONFIG_TASKS_RCU) && !defined(CONFIG_TASKS_RCU_TRAMP= OLINE_READERS) */ =20 #ifdef CONFIG_TASKS_RCU =20 +static int rcu_tasks_lazy_ms =3D -1; +module_param(rcu_tasks_lazy_ms, int, 0444); + +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + +//////////////////////////////////////////////////////////////////////// +// +// Tasks RCU for architectures whose trampolines are Tasks Trace RCU +// readers (CONFIG_HAVE_RCU_TRAMPOLINE_READERS). +// +// On these architectures every piece of text whose lifetime Tasks RCU +// guards -- ftrace trampolines, kprobe optinsn slots, BPF trampoline +// images, out-of-line ftrace direct-call trampolines -- enters a Tasks +// Trace RCU read-side critical section before calling out of itself and +// leaves it before returning, so a task anywhere inside such a call-out, +// preempted or not, is an ordinary rcu_read_lock_trace() reader and +// synchronize_rcu_tasks_trace() waits for it. +// +// What that cannot cover is the handful of instructions in the trampoline +// before the reader is entered and after it is left, and the one user that +// has no trampoline at all: the bytes after a kprobe that the jump +// optimizer is about to overwrite. A task can only linger in such +// "unmarked" text by being interrupted there; unmarked text never calls +// anything that could schedule. So a context switch on a CPU tells us th= at +// whatever that CPU was running is out of unmarked text, with one +// exception: a preemption from the irq-exit path, which can happen at any +// instruction boundary. That path has the interrupted pt_regs in hand, so +// just before it preempts it records the IP in the task and checks it +// (rcu_tasks_trampoline_text()); if it is inside unmarked text the task +// goes on a short holdout list first, and takes itself off again at its +// next context switch outside such a preemption or its next irq-exit +// check that finds it elsewhere. With that, every pass through +// __schedule() is a per-CPU quiescent event, as are usermode and idle. +// +// A grace period is then: +// +// 1. Wait for every online CPU to context switch or be found in an RCU +// extended quiescent state (deep idle, nohz_full userspace), nudging +// stragglers with resched_cpu(). Afterwards no task is in the leading +// unmarked instructions of a dying trampoline unless it is on the +// holdout list. +// 2. Wait for the holdout list (as it stood) to drain. +// 3. synchronize_rcu_tasks_trace(), for everything inside the readers. +// 4. Repeat 1 and 2 for tasks that have since left the reader and are in +// the trailing unmarked instructions. +// +// which is bounded by a few jiffies plus preempt-off latency plus an SRCU +// grace period, independent of how long any task runs without sleeping. +// Unlike the classic implementation this does wait for an idle task caught +// in a trampoline, since an idle CPU only counts while RCU is not watching +// it. + +static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp); +void call_rcu_tasks(struct rcu_head *rhp, rcu_callback_t func); +DEFINE_RCU_TASKS(rcu_tasks, rcu_tasks_tramp_wait_gp, call_rcu_tasks, "RCU = Tasks"); + +/* Per-CPU count of Tasks RCU quiescent events, and the GP kthread's snaps= hot. */ +static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_seq); +static DEFINE_PER_CPU(unsigned long, rcu_tasks_qs_snap); + +/* + * Tasks currently switched out by an irq-exit preemption are kept, with t= he + * interrupted IP, on the per-CPU rtp_exit_list of the CPU that preempted = them + * (reusing the list, lock and task_struct fields the classic flavor uses = for + * its exit-path bookkeeping, which this flavor does not need), so that + * rcu_tasks_wait_irq_preempted() can find them without a tasklist scan and + * regardless of where they are in exit. + */ + +/* Tasks last seen preempted inside unmarked trampoline text. */ +static LIST_HEAD(rcu_tasks_tramp_holdouts); +static DEFINE_RAW_SPINLOCK(rcu_tasks_tramp_lock); + +/* CPUs / holdouts the current grace period is still waiting for. */ +static struct cpumask rcu_tasks_pending_cpus; +static LIST_HEAD(rcu_tasks_gp_holdouts); + +/** + * arch_rcu_tasks_trampoline_text - Does the architecture treat @ip as unm= arked trampoline text? + * @ip: kernel text address inside core kernel text + * + * See rcu_tasks_trampoline_text(). Architectures override this to flag + * core text that runs on behalf of a trampoline outside its Tasks Trace + * reader, e.g. static ftrace entry stubs or return thunks that hold a + * trampoline address they are about to jump to. + */ +bool __weak arch_rcu_tasks_trampoline_text(unsigned long ip) +{ + return false; +} + +/** + * rcu_tasks_trampoline_text - Is @ip in text Tasks RCU protects but no re= ader marks? + * @ip: an interrupted instruction pointer + * + * True when a task interrupted at @ip may be executing, or about to enter + * or return into, text whose lifetime depends on synchronize_rcu_tasks() + * without being inside the Tasks Trace reader that text takes around its + * call-outs: + * + * - anything outside core kernel and module text (ftrace and BPF + * trampolines, kprobe slots and other dynamically allocated text; this + * deliberately does not ask is_ftrace_trampoline() and friends, since + * text being torn down may already be unregistered there); + * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(). + * + * A false positive only makes the task a holdout until its next quiescent + * event. Called with interrupts disabled from the irq-exit path. + */ +bool rcu_tasks_trampoline_text(unsigned long ip) +{ + if (core_kernel_text(ip)) + return arch_rcu_tasks_trampoline_text(ip); + return !is_module_text_address(ip); +} +NOKPROBE_SYMBOL(rcu_tasks_trampoline_text); + +/* Note a Tasks RCU quiescent event on this CPU. */ +static void rcu_tasks_qs_event(void) +{ + unsigned long *seq; + + guard(preempt_notrace)(); + seq =3D this_cpu_ptr(&rcu_tasks_qs_seq); + /* Order a preceding rcu_tasks_tramp_hold() before the count. */ + smp_store_release(seq, *seq + 1); +} + +static void rcu_tasks_tramp_hold(struct task_struct *t) +{ + unsigned long flags; + + if (t->rcu_tasks_holdout) + return; + raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags); + list_add_tail(&t->rcu_tasks_holdout_list, &rcu_tasks_tramp_holdouts); + WRITE_ONCE(t->rcu_tasks_holdout, true); + raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags); +} + +static void rcu_tasks_tramp_release(struct task_struct *t) +{ + unsigned long flags; + + if (likely(!t->rcu_tasks_holdout)) + return; + raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags); + list_del_init(&t->rcu_tasks_holdout_list); + WRITE_ONCE(t->rcu_tasks_holdout, false); + raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags); +} + +/** + * rcu_tasks_irq_resched_enter - Tasks RCU hook for the irq-exit reschedul= e check + * @ip: instruction pointer of the interrupted (task-level) context + * + * Called with interrupts disabled when an interrupt returning to kernel + * mode is about to preempt_schedule_irq(), the one context switch that can + * catch a task inside unmarked trampoline text. Record where the task is + * parked for as long as it is (rcu_tasks_wait_irq_preempted() looks at + * that), and if it is inside such text make it a holdout before + * __schedule() reports the quiescent event; if it is not, this is as good + * as a voluntary switch for ending an earlier hold. + */ +void rcu_tasks_irq_resched_enter(unsigned long ip) +{ + struct task_struct *t =3D current; + struct rcu_tasks_percpu *rtpcp =3D this_cpu_ptr(rcu_tasks.rtpcpu); + + lockdep_assert_irqs_disabled(); + WRITE_ONCE(t->rcu_tasks_irq_ip, ip); + t->rcu_tasks_exit_cpu =3D smp_processor_id(); + raw_spin_lock_rcu_node(rtpcp); + list_add(&t->rcu_tasks_exit_list, &rtpcp->rtp_exit_list); + raw_spin_unlock_rcu_node(rtpcp); + + if (unlikely(rcu_tasks_trampoline_text(ip))) + rcu_tasks_tramp_hold(t); + else + rcu_tasks_tramp_release(t); +} +NOKPROBE_SYMBOL(rcu_tasks_irq_resched_enter); + +/** + * rcu_tasks_irq_resched_exit - preempt_schedule_irq() has returned + * + * The task is running again (possibly elsewhere) and about to return to t= he + * interrupted context; it is no longer parked anywhere. + */ +void rcu_tasks_irq_resched_exit(void) +{ + struct task_struct *t =3D current; + struct rcu_tasks_percpu *rtpcp =3D per_cpu_ptr(rcu_tasks.rtpcpu, t->rcu_t= asks_exit_cpu); + + lockdep_assert_irqs_disabled(); + raw_spin_lock_rcu_node(rtpcp); + list_del_init(&t->rcu_tasks_exit_list); + raw_spin_unlock_rcu_node(rtpcp); + WRITE_ONCE(t->rcu_tasks_irq_ip, 0); +} +NOKPROBE_SYMBOL(rcu_tasks_irq_resched_exit); + +/** + * rcu_tasks_note_qs - Tasks RCU hook for a context switch or explicit QS + * @t: current + * @preempt: this is a preemption rather than a voluntary switch + * + * Every pass through __schedule() (and cond_resched_tasks_rcu_qs(), and a + * tick from userspace or idle) is a quiescent event for this CPU: unmarked + * trampoline text never calls anything that schedules, and the irq-exit + * path has already made @t a holdout if it is preempting inside such text. + * Any of these outside an irq-exit preemption also shows @t itself to be + * outside, ending an earlier hold -- including cond_resched() under + * PREEMPT_DYNAMIC's none/voluntary modes, where the irq-exit path is off. + */ +void rcu_tasks_note_qs(struct task_struct *t, bool preempt) +{ + WARN_ON_ONCE(t !=3D current); + if (!READ_ONCE(t->rcu_tasks_irq_ip)) + rcu_tasks_tramp_release(t); + rcu_tasks_qs_event(); +} +EXPORT_SYMBOL_GPL(rcu_tasks_note_qs); /* cond_resched_tasks_rcu_qs() */ + +/** + * rcu_tasks_wait_irq_preempted - wait for tasks irq-preempted inside @ins= ide + * @inside: predicate on a task's recorded irq-exit preemption IP + * + * For a caller about to make some ordinary text unsafe to be parked in + * (the kprobe jump optimizer): once the caller has arranged for + * rcu_tasks_trampoline_text() to cover that text, new irq-exit preemptions + * there become holdouts, but a task preempted there earlier is invisible + * to the grace period. Wait until no parked task's recorded preemption IP + * is inside; a following synchronize_rcu_tasks() then covers the rest. + * The leading synchronize_rcu() orders the caller's arrangement against + * preemptions in flight, which run with interrupts disabled. + */ +void rcu_tasks_wait_irq_preempted(bool (*inside)(unsigned long ip)) +{ + struct task_struct *t; + unsigned long flags; + int cpu, kick; + bool found; + + synchronize_rcu(); + for (;;) { + found =3D false; + for_each_possible_cpu(cpu) { + struct rcu_tasks_percpu *rtpcp =3D per_cpu_ptr(rcu_tasks.rtpcpu, cpu); + + kick =3D -1; + raw_spin_lock_irqsave_rcu_node(rtpcp, flags); + list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) { + if (inside(READ_ONCE(t->rcu_tasks_irq_ip))) { + found =3D true; + if (task_curr(t)) + kick =3D task_cpu(t); + } + } + raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); + if (kick >=3D 0) + resched_cpu(kick); + } + if (!found) + return; + schedule_timeout_uninterruptible(1); + } +} + +/* Has @cpu passed a quiescent event since the snapshot, or need it not? */ +static bool rcu_tasks_cpu_quiescent(int cpu) +{ + if (!cpu_online(cpu)) + return true; + /* Pairs with the release in rcu_tasks_qs_event(). */ + if (smp_load_acquire(per_cpu_ptr(&rcu_tasks_qs_seq, cpu)) !=3D + per_cpu(rcu_tasks_qs_snap, cpu)) + return true; + /* + * Idle or nohz_full userspace in an RCU extended quiescent state: no + * task-level kernel frames can be live in a trampoline there, and + * whatever ran before has switched out. An idle CPU that RCU is + * watching (an interrupt from idle, or the traceable part of the idle + * loop) is deliberately not let through: the idle task may be in a + * trampoline with that interrupt on top, and since it is never + * preempted from irq exit nothing else would catch it. It gets the + * resched_cpu() like anyone else and counts once the idle loop itself + * schedules, which it cannot do from inside a trampoline. + */ + return !(ct_rcu_watching_cpu(cpu) & CT_RCU_WATCHING); +} + +/* Rate-limited stall report; returns true if the caller should add detail= . */ +static bool rcu_tasks_tramp_stall(struct rcu_tasks *rtp, unsigned long *la= streport, + const char *what) +{ + int rtst =3D READ_ONCE(rcu_task_stall_timeout); + + if (rtst <=3D 0 || !time_after(jiffies, *lastreport + rtst)) + return false; + *lastreport =3D jiffies; + pr_err("INFO: %s: %s, grace period %lu is %lu jiffies old\n", rtp->kname, + what, rcu_seq_current(&rtp->tasks_gp_seq), jiffies - rtp->gp_start= ); + return true; +} + +/* + * Steps 1/4: wait until every online CPU has context switched or is in an + * RCU extended quiescent state. A CPU that has neither after a jiffy is + * asked to switch with resched_cpu(), which takes it through + * rcu_tasks_irq_resched_enter() and __schedule() (or, from userspace, a + * guest or the idle loop, straight to __schedule()). + */ +static void rcu_tasks_tramp_wait_cpus(struct rcu_tasks *rtp, unsigned long= *lastreport) +{ + struct cpumask *pending =3D &rcu_tasks_pending_cpus; + unsigned long start; + int cpu; + + /* + * The quiescent events run with preemption (in practice interrupts) + * disabled, so after this any event we go on to count began after the + * caller's updates -- the unpublished trampoline, and whatever + * rcu_tasks_trampoline_text() consults -- were visible to it. + */ + synchronize_rcu(); + + start =3D jiffies; + for_each_online_cpu(cpu) { + per_cpu(rcu_tasks_qs_snap, cpu) =3D READ_ONCE(per_cpu(rcu_tasks_qs_seq, = cpu)); + __cpumask_set_cpu(cpu, pending); + } + /* Snapshots before the checks below; pairs with rcu_tasks_qs_event(). */ + smp_mb(); + + for (;;) { + for_each_cpu(cpu, pending) + if (rcu_tasks_cpu_quiescent(cpu)) + __cpumask_clear_cpu(cpu, pending); + if (cpumask_empty(pending)) + break; + if (time_after(jiffies, start)) { + for_each_cpu(cpu, pending) + resched_cpu(cpu); + rtp->n_ipis +=3D cpumask_weight(pending); + } + schedule_timeout_idle(1); + if (rcu_tasks_tramp_stall(rtp, lastreport, "CPUs without a quiescent eve= nt")) + pr_err("\tCPUs: %*pbl\n", cpumask_pr_args(pending)); + } +} + +/* + * Steps 2/4: wait for the tasks that were holdouts when we looked to stop + * being holdouts. They are moved to a private list so that tasks becoming + * holdouts later (in live trampolines) cannot keep us here; each removes + * itself via rcu_tasks_tramp_release() wherever it is queued. + */ +static void rcu_tasks_tramp_wait_holdouts(struct rcu_tasks *rtp, unsigned = long *lastreport) +{ + struct task_struct *t; + unsigned long flags; + int cpu; + + raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags); + list_splice_tail_init(&rcu_tasks_tramp_holdouts, &rcu_tasks_gp_holdouts); + raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags); + + for (;;) { + struct cpumask *kick =3D &rcu_tasks_pending_cpus; + struct task_struct *show[8]; + int nshow =3D 0, i; + bool empty, report; + + report =3D rcu_tasks_tramp_stall(rtp, lastreport, + "tasks preempted in trampoline text"); + cpumask_clear(kick); + raw_spin_lock_irqsave(&rcu_tasks_tramp_lock, flags); + empty =3D list_empty(&rcu_tasks_gp_holdouts); + list_for_each_entry(t, &rcu_tasks_gp_holdouts, rcu_tasks_holdout_list) { + if (task_curr(t)) + __cpumask_set_cpu(task_cpu(t), kick); + if (report && nshow < ARRAY_SIZE(show)) + show[nshow++] =3D get_task_struct(t); + } + raw_spin_unlock_irqrestore(&rcu_tasks_tramp_lock, flags); + /* Never printk under the lock the irq-exit path takes. */ + for (i =3D 0; i < nshow; i++) { + sched_show_task(show[i]); + put_task_struct(show[i]); + } + if (empty) + break; + for_each_cpu(cpu, kick) + resched_cpu(cpu); + rtp->n_ipis +=3D cpumask_weight(kick); + schedule_timeout_idle(1); + } +} + +/* Wait for one trampoline-reader Tasks RCU grace period. */ +static void rcu_tasks_tramp_wait_gp(struct rcu_tasks *rtp) +{ + unsigned long lastreport =3D jiffies; + + set_tasks_gp_state(rtp, RTGS_WAIT_SCAN_HOLDOUTS); + rcu_tasks_tramp_wait_cpus(rtp, &lastreport); + rcu_tasks_tramp_wait_holdouts(rtp, &lastreport); + + set_tasks_gp_state(rtp, RTGS_WAIT_READERS); + synchronize_rcu_tasks_trace(); + + set_tasks_gp_state(rtp, RTGS_SCAN_HOLDOUTS); + rcu_tasks_tramp_wait_cpus(rtp, &lastreport); + rcu_tasks_tramp_wait_holdouts(rtp, &lastreport); + + set_tasks_gp_state(rtp, RTGS_POST_GP); +} + +static int __init rcu_spawn_tasks_kthread(void) +{ + rcu_tasks.gp_sleep =3D HZ / 10; + if (rcu_tasks_lazy_ms >=3D 0) + rcu_tasks.lazy_jiffies =3D msecs_to_jiffies(rcu_tasks_lazy_ms); + rcu_tasks.wait_state =3D TASK_IDLE; + rcu_spawn_tasks_kthread_generic(&rcu_tasks); + return 0; +} + +void exit_tasks_rcu_start(void) { } +void exit_tasks_rcu_finish(void) { } + +#else /* #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */ + //////////////////////////////////////////////////////////////////////// // // Simple variant of RCU whose quiescent states are voluntary context @@ -1173,6 +1607,8 @@ static void tasks_rcu_exit_stall(struct timer_list *u= nused) #endif // #ifndef CONFIG_TINY_RCU } =20 +#endif /* #else #ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */ + /** * call_rcu_tasks() - Queue an RCU for invocation task-based grace period * @rhp: structure to be used for queueing the RCU updates. @@ -1187,6 +1623,12 @@ static void tasks_rcu_exit_stall(struct timer_list *= unused) * primitives analogous to rcu_read_lock() and rcu_read_unlock() because * this primitive is intended to determine that all tasks have passed * through a safe state, not so much for data-structure synchronization. + * On CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels a preemption outside + * trampoline text also ends one, and a reader whose protected window + * spans preemptible code must additionally be a Tasks Trace RCU reader + * (rcu_read_lock_trace(), as the trampolines there take around their + * call-outs); an arbitrary stretch of preemptible kernel code is not + * protected. * * See the description of call_rcu() for more detailed information on * memory ordering guarantees. @@ -1205,7 +1647,9 @@ EXPORT_SYMBOL_GPL(call_rcu_tasks); * executing rcu-tasks read-side critical sections have elapsed. These * read-side critical sections are delimited by calls to schedule(), * cond_resched_tasks_rcu_qs(), idle execution, userspace execution, calls - * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched(). + * to synchronize_rcu_tasks(), and (in theory, anyway) cond_resched(); + * on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels also by preemption + * outside trampoline text, see call_rcu_tasks(). * * This is a very specialized primitive, intended only for a few uses in * tracing and other situations requiring manipulation of function @@ -1233,9 +1677,7 @@ void rcu_barrier_tasks(void) } EXPORT_SYMBOL_GPL(rcu_barrier_tasks); =20 -static int rcu_tasks_lazy_ms =3D -1; -module_param(rcu_tasks_lazy_ms, int, 0444); - +#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS static int __init rcu_spawn_tasks_kthread(void) { rcu_tasks.gp_sleep =3D HZ / 10; @@ -1251,6 +1693,7 @@ static int __init rcu_spawn_tasks_kthread(void) rcu_spawn_tasks_kthread_generic(&rcu_tasks); return 0; } +#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */ =20 #if !defined(CONFIG_TINY_RCU) void show_rcu_tasks_classic_gp_kthread(void) @@ -1279,6 +1722,7 @@ void rcu_tasks_get_gp_data(int *flags, unsigned long = *gp_seq) } EXPORT_SYMBOL_GPL(rcu_tasks_get_gp_data); =20 +#ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS /* * Protect against tasklist scan blind spot while the task is exiting and * may be removed from the tasklist. Do this by adding the task to yet @@ -1322,6 +1766,7 @@ void exit_tasks_rcu_finish(void) list_del_init(&t->rcu_tasks_exit_list); raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags); } +#endif /* #ifndef CONFIG_TASKS_RCU_TRAMPOLINE_READERS */ =20 #else /* #ifdef CONFIG_TASKS_RCU */ void exit_tasks_rcu_start(void) { } diff --git a/kernel/rcu/update.c b/kernel/rcu/update.c index b62735a67884..a122b8d1effb 100644 --- a/kernel/rcu/update.c +++ b/kernel/rcu/update.c @@ -40,7 +40,9 @@ #include #include #include +#include #include +#include #include #include #include --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743025; cv=none; d=zohomail.com; s=zohoarc; b=EZBQjkMzU19dAghM3AAhhYZkqs1sXK1i6aD8oUwze4v3IQjbUu+5YHns6OYPWtWKbPQCIIZ2yPWwG300yjJ5cJipZlqjY+3LJ6IZSx4W/Z5wyZL9DQXyIX4PNK1PiD29enzYLA2iCHkxjJth6v8MchGUW01fgOxa6zq01+Y4L98= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743025; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=n57jpnbqR3xEpFvVL4pj+x/9TynCnpwMD9+qM+zc540=; b=Yp2D6hDdfAkS1OCYGCXnK7gZ28SZccmbkxeRbTFbAhi9JP8OBM0pwlMyDDu1/IXr/R4kDBT8Ynm+N8wAO1kQObQA2K+ipWVNyXTvQAgVHcCwAFIYAg8ncO6pjpMSaS286Ypao0owzGn/vahWj9eyS9LS/LUncDAdQSCVpO43PoQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743025834964.4624603549057; Fri, 18 Sep 2026 07:50:25 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425495.1649419 (Exim 4.92) (envelope-from ) id 1x7ZuW-0004i2-2R; Fri, 18 Sep 2026 14:49:56 +0000 Received: by outflank-mailman (output) from mailman id 1425495.1649419; Fri, 18 Sep 2026 14:49:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuV-0004hr-VI; Fri, 18 Sep 2026 14:49:55 +0000 Received: by outflank-mailman (input) for mailman id 1425495; Fri, 18 Sep 2026 14:49:54 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuT-0004gK-Tt for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:49:54 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7ZuT-009hUR-Ab for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:49:53 +0200 Received: from [10.42.69.12] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f7a-e002-0a2a0a5209dd-0a2a450c824a-40 for ; Fri, 18 Sep 2026 16:49:53 +0200 Received: from [74.125.230.235] (helo=mail-qk2-f43.google.com) by tlsNG-d25034.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f8e-f479-0a2a450c0019-4a7de6eb9422-3 for ; Fri, 18 Sep 2026 16:49:51 +0200 Received: by mail-qk2-f43.google.com with SMTP id af79cd13be357-93910cc46c7so64988585a.3 for ; Fri, 18 Sep 2026 07:49:51 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-93be0e90276sm152288185a.21.2026.09.18.07.49.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:49 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742990; x=1790347790; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=n57jpnbqR3xEpFvVL4pj+x/9TynCnpwMD9+qM+zc540=; b=obcMTyDqmBcYTxnvLA82tgwSacvI6i7PRIPfpNn/dXRTTd5CjE4YXV84Dc6p9+yZ4Z 6HFNIl1Xxy4yH/nuxldhwzrWT1ive5gBF28iXyAjvW71eQZm9Msiiv5Kmzc8yjcyW9dB rcsfQUIjd6pw4OA5b0OpuFTWe2Iq9Ns8JxEaOa1YCS01ovKuziiDFORbB+bXTzCUc+Nj MCh5TZ3f/Xxd1I6a+TDF8weKZhbhS4prF149xctCVVmclKjoX/727lClZzHDP/oxyxFg LjYwwttB7+5OYjexn46WIIsqnVFAZkcBXpzzXykknCdHuZhtVZPgvy4jBjyDxwBZNvYd Vplw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742990; x=1790347790; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=n57jpnbqR3xEpFvVL4pj+x/9TynCnpwMD9+qM+zc540=; b=T3EhY/yQ208Ft1BXxWfQP6K2u3zzZ2aRUBRm3kbBlGXwv1JAgsdnr+mq0Ege6cxTg2 feQgPxaxD439795D/WsUCieHAF/LZeYOnjFffg3EmB1E1NGoLK7/HM0k0m3LaN4lNJl2 IhUXeCb3xmUIJPjJ/DgsHyzOMes6SBS3e41YXWBslW7oPgFBFVS/d21KJIQ6/hPZ2i5Z KOd2Kf0XotlRdOyvuZrYww2kxTtaGBmD+CU+mNcsuxUI13NLrw3QGwEbfW0QWPXWbOwy VNbHAW8IjK2Y3+zyHRaWxnT3Keb3F/Qh+68F2A0PAWnN2vuf5qZEnrhWPJZF8/yqQKYX FIAQ== X-Forwarded-Encrypted: i=1; AKwUvBwjNnoTow5U4GJmgeN6lE9zNssHQiyqrhzVlBP7lO8eHsXj//hzh4xf76TSF5gQUmQ9TIW2fwK1FR4=@lists.xenproject.org X-Gm-Message-State: AFuF++mdTSVhkCwx5kSyScjK66T45BvsiXGnjvzXFfevTzDO7b0GCBss hvLZDuI2mbJPz37Jl3Q1NgmQVMswVKNWWwsDTMC2Awk1KV3Nm9gzmQkFOyACPaq2AAU= X-Gm-Gg: AYBFou2NHTz172ie584vk7d+Ll66ez9l8imDIhIM0rwJv9/us5IOkW7RlHD4XDC1v2B A/ifmqG+bTcN2M003weKd6iejqE4BMJVAck3ljZFo0HNKM7wOupHLCevzYh5Yvx77qhI2ZeQYVf Oq0ib7tNLOaU1i3rPfC/N47lWwFdv0EteknVE6fVslY6FF57S/g0iygvO64mYYURdc2ktKrBwsd HL8oIUOFdeGM2rZyXmVkiF7G4AFcd+ezgZSIdgFzjr73CdvoQlq0UC0bQ/gaWtN6GVVImq78OYm BNH2qHG+l762P65t8QsKMro5Xi582BI42yx4i4/83FyK1OnbfS7jReSlD2qkTD4eSh/9JtUZuxd oTyNhdDKSyzzM7aI+K+eTQ6q3J1yKM1kKnLnXC8qpvpczByPhOCy9tlx4HsO8jYc5W0G/eJ4BZz lKN8C6jynvhAUN+v9LePv39qrOAp0DHMwP6aY9ieWn45Dzb/hyCxl8wb/3Si5AYpKjBGkG23qPN vqpfHBtypJtT+3sts1VJk5JxAiFNueQR5v02ehaoMs99YpPHtJ7VJDO X-Received: by 2002:a05:620a:1724:b0:939:365a:49a5 with SMTP id af79cd13be357-93bdc6e5e09mr400777085a.27.1789742989989; Fri, 18 Sep 2026 07:49:49 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:22 +0000 Subject: [PATCH RFC v4 03/13] kprobes: Expose the optprobe jump window to Tasks RCU MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-3-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=7473; i=josef@toxicpanda.com; h=from:subject:message-id; bh=97rXNEsO0S02JFiFJq38moNkWCht7E9TlyxYtLTVeYI=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QLDPJjh7XcPwoZnncjeIz0ekrFrEehwu9vxSbP/wbLngYBuNUBgEdoEdYmb8mwvrWGuV6QiYW9q iu8TI3m4Kfgo= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-d25034/1789742991-50D3CA5B-AAE1A061/0/0 X-purgate-type: clean X-purgate-size: 7475 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743026539158500 kprobe_optimizer() is the one synchronize_rcu_tasks() user that is not about trampoline text: it waits for tasks that were interrupted on an instruction boundary inside the bytes it is about to overwrite with the optimized jump, so that none of them resumes into the middle of the new instruction. Those bytes are ordinary kernel or module text with no Tasks Trace reader around them, so on CONFIG_TASKS_RCU_TRAMPOLINE_READERS kernels the irq-exit quiescent-state check has to be told about them. Add kprobe_in_optimized_region(), a lockless and conservative form of get_optimized_kprobe() that reports whether any registered kprobe lies within MAX_OPTIMIZED_LENGTH before the given address regardless of its optimization state, and have rcu_tasks_trampoline_text() consult it for core and module text so that a task interrupted there becomes a holdout rather than a quiescent event. The hash walk only runs while kprobe_optimizer() is actually inside its synchronize_rcu_tasks(), tracked by a flag it sets around the call; otherwise the check is a single load. That check cannot see a task that was already preempted in the region before the flag went up (possibly before the kprobe even existed), and the new grace period does not otherwise wait for a preempted task to run again, so before synchronize_rcu_tasks() the optimizer calls rcu_tasks_wait_irq_preempted() to wait until no parked task's recorded irq-exit preemption IP is inside such a region; its leading synchronize_rcu() also publishes the flag to every (interrupts- disabled) check in flight. The kprobe hash is RCU-protected and every free path waits for a grace period after unhashing, so the lockless walk from the irq-exit path is safe. On other configurations the flag is set and cleared but nothing reads it and rcu_tasks_wait_irq_preempted() is a stub; the classic implementation already waits for such tasks. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/kprobes.h | 8 +++++++- kernel/kprobes.c | 50 +++++++++++++++++++++++++++++++++++++++++++++= ++++ kernel/rcu/tasks.h | 11 ++++++++--- 3 files changed, 65 insertions(+), 4 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index e6de7ae55bda..74cc48c04417 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -530,11 +530,17 @@ static inline bool is_kprobe_insn_slot(unsigned long = addr) } #endif /* !CONFIG_KPROBES */ =20 -#ifndef CONFIG_OPTPROBES +#ifdef CONFIG_OPTPROBES +bool kprobe_in_optimized_region(unsigned long addr); +#else /* !CONFIG_OPTPROBES */ static inline bool is_kprobe_optinsn_slot(unsigned long addr) { return false; } +static inline bool kprobe_in_optimized_region(unsigned long addr) +{ + return false; +} #endif /* !CONFIG_OPTPROBES */ =20 #ifdef CONFIG_KRETPROBES diff --git a/kernel/kprobes.c b/kernel/kprobes.c index 6337da5cab9e..e460fba83e4a 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -511,6 +511,48 @@ static struct kprobe *get_optimized_kprobe(kprobe_opco= de_t *addr) return NULL; } =20 +/* + * True while kprobe_optimizer() is waiting for its Tasks RCU grace period. + * Only in that window can an interruption inside an optprobe's jump region + * matter to it, so kprobe_in_optimized_region() does no work otherwise. + */ +static bool kprobe_optimizer_waiting; + +/** + * kprobe_in_optimized_region - Could @addr be inside bytes a jump-optimiz= ed + * kprobe replaces? + * @addr: kernel text address, typically an interrupted instruction pointer + * + * kprobe_optimizer() relies on synchronize_rcu_tasks() to wait for tasks = that + * were interrupted on an instruction boundary inside the region about to = be + * overwritten by the optimized jump. Where Tasks RCU is built on + * reader-marked trampolines that region has no reader, so the irq-exit + * quiescent-state check asks this instead (see rcu_tasks_trampoline_text(= )). + * This is the lockless, conservative form of get_optimized_kprobe(): it d= oes + * not care whether the kprobe found is, or ever will be, optimized. May = be + * called from any context with preemption disabled; the kprobe hash is + * RCU-protected and every free path waits for a grace period after unhash= ing. + * + * The hash walk only runs while the optimizer is actually waiting. A task + * that was preempted in such a region before the flag went up is invisible + * to that check, so the optimizer first waits those out by their recorded + * preemption IP (rcu_tasks_wait_irq_preempted(), whose leading + * synchronize_rcu() also publishes the flag to every check in flight). + */ +bool kprobe_in_optimized_region(unsigned long addr) +{ + int i; + + if (!READ_ONCE(kprobe_optimizer_waiting)) + return false; + + for (i =3D 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + if (get_kprobe((kprobe_opcode_t *)addr - i)) + return true; + return false; +} +NOKPROBE_SYMBOL(kprobe_in_optimized_region); + /* Optimization staging list, protected by 'kprobe_mutex' */ static LIST_HEAD(optimizing_list); static LIST_HEAD(unoptimizing_list); @@ -644,8 +686,16 @@ static void kprobe_optimizer(void) * to 2nd-Nth byte of jump instruction. This wait is for avoiding it. * Note that on non-preemptive kernel, this is transparently converted * to synchronoze_sched() to wait for all interrupts to have completed. + * kprobe_optimizer_waiting lets a reader-marked-trampoline Tasks RCU + * recognise tasks interrupted in such a region while we wait, and + * rcu_tasks_wait_irq_preempted() (a no-op elsewhere) first waits + * out any that were preempted there before we said so; see + * kprobe_in_optimized_region(). */ + WRITE_ONCE(kprobe_optimizer_waiting, true); + rcu_tasks_wait_irq_preempted(kprobe_in_optimized_region); synchronize_rcu_tasks(); + WRITE_ONCE(kprobe_optimizer_waiting, false); =20 /* Step 3: Optimize kprobes after quiesence period */ do_optimize_kprobes(); diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index f03be742be48..eb1388dd8a61 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1005,7 +1005,9 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned l= ong ip) * trampolines, kprobe slots and other dynamically allocated text; this * deliberately does not ask is_ftrace_trampoline() and friends, since * text being torn down may already be unregistered there); - * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(). + * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(); + * - the bytes after a kprobe that a pending jump optimization is about to + * overwrite, the one synchronize_rcu_tasks() user with no trampoline. * * A false positive only makes the task a holdout until its next quiescent * event. Called with interrupts disabled from the irq-exit path. @@ -1013,8 +1015,11 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned = long ip) bool rcu_tasks_trampoline_text(unsigned long ip) { if (core_kernel_text(ip)) - return arch_rcu_tasks_trampoline_text(ip); - return !is_module_text_address(ip); + return arch_rcu_tasks_trampoline_text(ip) || + kprobe_in_optimized_region(ip); + if (is_module_text_address(ip)) + return kprobe_in_optimized_region(ip); + return true; } NOKPROBE_SYMBOL(rcu_tasks_trampoline_text); =20 --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743021; cv=none; d=zohomail.com; s=zohoarc; b=nptL3FHLVKt50zCkqhUgyQQTFDDS2vOhzdncuLNMkgegpLldB1vQi/dY2oAMCYpb7lSkIlJdVRScaz5RRnoGGAfL/rIkVOeEsu1uMUFc0xxC0I7uu5TCruoK3WYDHX5C+XWjLd2N1GE31HFTRpT1Fi/8tMUmZxItGMZn2EJWYXQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743021; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=cjEABplA81VfGcXS6Z1Q3Y7Xt5Unb50ofxErtTl6HPY=; b=PUnhH/F/DNf9JfZo/qJXdp4WMx9fmfRzF4nnzBV2IBsfms0XElqkb7d9splYMk1h8k17zS60kzZnMjbPRqQsZX7v23UTa2pQvMExh+JvKQr2CQ1RX6uG+mXBIYp8jsx59udvbGuytCXVJpSXBZeGc98c+3wlo8/ldQNfnTw3yRY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 178974302099431.914488399020797; Fri, 18 Sep 2026 07:50:20 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425496.1649426 (Exim 4.92) (envelope-from ) id 1x7ZuW-0004kq-Eq; Fri, 18 Sep 2026 14:49:56 +0000 Received: by outflank-mailman (output) from mailman id 1425496.1649426; Fri, 18 Sep 2026 14:49:56 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuW-0004kO-6q; Fri, 18 Sep 2026 14:49:56 +0000 Received: by outflank-mailman (input) for mailman id 1425496; Fri, 18 Sep 2026 14:49:54 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuU-0004gV-6t for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:49:54 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7ZuT-0054h4-K5 for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:49:53 +0200 Received: from [10.42.69.6] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f85-8faa-0a2a0a5109dd-0a2a4506b4ba-32 for ; Fri, 18 Sep 2026 16:49:53 +0200 Received: from [74.125.230.204] (helo=mail-qk2-f12.google.com) by tlsNG-16d1c6.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f90-195a-0a2a45060019-4a7de6ccdddf-3 for ; Fri, 18 Sep 2026 16:49:53 +0200 Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-530e28a62abso11097351cf.1 for ; Fri, 18 Sep 2026 07:49:53 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532a19a5ce3sm14843771cf.12.2026.09.18.07.49.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:51 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742992; x=1790347792; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=cjEABplA81VfGcXS6Z1Q3Y7Xt5Unb50ofxErtTl6HPY=; b=oVOsWb1uu7G1/KWzNdrb+K6tlGrWQQfJXrgP/yh5l7HBkpyVd5K/O0W7LG4A2zVmD5 jPCDrwFeVmHVnoGrxyBzPwOZepOgalpNkxmfcOTskWTtjHmNLd5aORZbu0DcRbFa2g1h MN2MaQr17qXRQEegdLacnAhPf5RYBVKKbiLcGbstuAkcSFs9Iv2KXeeELZ2kJzukU9Ll 25t5puE6paTM5fOVaqWkPs7mGQflFbsE4e2tc9OG8Myv14Kf64fzDqWGW6FbkjinefJd VJkZ46AkxKJ0A5HhaRrIpHpdhyFD2kEJ9CoteR0NBw63zCFGJrhiPguujwmsMxnDTgvA IcYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742992; x=1790347792; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=cjEABplA81VfGcXS6Z1Q3Y7Xt5Unb50ofxErtTl6HPY=; b=Hn9Xpqp+fOO6iIN0CbDhMUUiyVWdXgB0c7JX8/cv+kkV5WFF6qHxRoYKRUxY/xg6Hn oqq32/bineZCBkREKEgkR/RUxIP7FVo8u3imixrXuXyKvlFLpt+1M6aVcq+EnQDVbSoJ OK/PERJSrsppJIwVg/JemyqxfvcjiL8h35BZjOopJ9ot4NJWPL41+ZeoMypPpLgixbX8 W9pYajxuyYGNDgdIMj0y0llMUqOm/EAvdm/PoFPAtLjaU58/GRzzpulb5Sdb4FKZ2aE+ /ktOLPQaeomhrFOQEW2xuBxx2YGpbWfgqslwKJaQWdEldIuDFigLpoeOzCwfJM/+poCW bC/g== X-Forwarded-Encrypted: i=1; AKwUvBwtNL74APxGkUbWKsVAIt+30mwlgY0FrxNgatz6BWWinCrGU8863RN4dsi7bKLLKf3AWBkqGvsgvVg=@lists.xenproject.org X-Gm-Message-State: AFuF++lp+mMd47QFtRMqB/KnR1v53hHUZknEGzGPTu3vMawQpvibI1hP aWcqlj/O4TreKhzaFlLgX1al6oh4gJgaM5GtiNjjkx1rcM47esgiGbNZUPx73hjL6iE= X-Gm-Gg: AYBFou1WpKgE7GqieRlmEibgKojboCbtgZypCKjHdEySKaS1KWnr0ZNGLblQtCdCb8B igVmgT3LXe/mOBAHxdJGI4trtczy58sJcJKt2YQUyiUxIhfON2mbELlZXg3oWbated68jWK8GJN t4Xiu9e3hXhwgsTI8/KCkInHv8sTSRUoGj4JM8FF1OCE5JPivkh0JqVJOp3005Xc2gT405BlRzN OJ9ECsxOjwtOyC8NRGLZMphkh7NoRGTGyAAbUNWWIPwMf77la+G9pW2jrsvznFul2EPKUSBchXE WRNu14gipFbVfoyRInXUaAT1KcdIjo23HMnizW8mWCjdTA4E32YXR6SwIhxZvHrgupLHSXAZehY aOM2Jtq6YPFJjaZwSYcqc2eWlrYym96VxuuSjmDmA9/zcqKjMKEo7irtpuo8qDrcBCfodYyZGcU Y3w2b6HmSAIwzBHDmekdcSsYhO12fCRm8scPnNBSmceJUjeU5ZgbURZTGsW9tyHsNs+0JaE2Z9W uN9JKjO9BxF1jogOfxNcUNdwAdBaNVoeqSzZgXw/YdY0a/qQFC/AcRvAg== X-Received: by 2002:ac8:7f0d:0:b0:52f:9e5c:33ff with SMTP id d75a77b69052e-5329e3767d5mr46806381cf.13.1789742991795; Fri, 18 Sep 2026 07:49:51 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:23 +0000 Subject: [PATCH RFC v4 04/13] ftrace: Mark modules hosting direct-call trampolines for Tasks RCU MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-4-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=7065; i=josef@toxicpanda.com; h=from:subject:message-id; bh=WxuURWfyPfVrbwoJYzsme+dJrO0g2fOexfelnDzg3io=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QOzJNf4EPWurTiDx3kIhXFyZwPBWmFPff1giOF5VzhAiMbXyfZ1cbtvqDh0lVG3P1j1LapzQR9r O/9+GtEnVpA0= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-16d1c6/1789742993-F560A77B-3845039B/0/0 X-purgate-type: clean X-purgate-size: 7067 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743022553158500 An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Where Tasks RCU is built on reader-marked trampolines, such a trampoline must be a Tasks Trace reader across its call-out like the ftrace and BPF trampolines are, so document that in register_ftrace_direct(). That still leaves the few instructions before the reader is entered and after it is left. For BPF images those are in dynamically allocated text that rcu_tasks_trampoline_text() already treats as unmarked trampoline text, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag (under CONFIG_TASKS_RCU_TRAMPOLINE_READERS, its only consumer), set by every register/modify path when the direct address is module text, and have rcu_tasks_trampoline_text() treat a task interrupted anywhere in such a module as a potential holdout. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 18 +++++++++++++++--- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 61 insertions(+), 3 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..82ca4f774725 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_trampoline_text(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index eb1388dd8a61..42ea6e0e61cb 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1006,6 +1006,8 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned l= ong ip) * deliberately does not ask is_ftrace_trampoline() and friends, since * text being torn down may already be unregistered there); * - whatever the architecture adds via arch_rcu_tasks_trampoline_text(); + * - the text of a module that hosts an out-of-line ftrace direct-call + * trampoline (see ftrace_direct_mark_module()); * - the bytes after a kprobe that a pending jump optimization is about to * overwrite, the one synchronize_rcu_tasks() user with no trampoline. * @@ -1014,12 +1016,22 @@ bool __weak arch_rcu_tasks_trampoline_text(unsigned= long ip) */ bool rcu_tasks_trampoline_text(unsigned long ip) { + bool ret =3D true; + if (core_kernel_text(ip)) return arch_rcu_tasks_trampoline_text(ip) || kprobe_in_optimized_region(ip); - if (is_module_text_address(ip)) - return kprobe_in_optimized_region(ip); - return true; + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod =3D __module_text_address(ip); + + if (mod) + ret =3D READ_ONCE(mod->ftrace_direct_tramp) || + kprobe_in_optimized_region(ip); + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_trampoline_text); =20 diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..f69f71591358 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) ops->trampoline =3D 0; } =20 +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_trampoline_text() recognises on its own (= see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must be a Tasks + * Trace reader across its call-out (see register_ftrace_direct()); markin= g the + * owning module here covers the instructions before it enters that reader= and + * after it leaves it, where a task interrupted in the module's text must = not be + * counted as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed un= der + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#if defined(CONFIG_MODULES) && defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) + struct module *mod; + + guard(rcu)(); + mod =3D __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, uns= igned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task = is + * executing it or is preempted in something it called. On architectures = that + * select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a tas= k if + * it is a Tasks Trace RCU reader, so the trampoline must enter one + * (rcu_read_lock_trace() or an assembly equivalent) before calling out and + * leave it before returning, just as that option requires of the in-kernel + * ftrace and BPF trampolines. The few instructions before and after are + * covered by the irq-exit check: automatically for trampolines outside ke= rnel + * and module text (e.g. BPF images), and via ftrace_direct_mark_module() = for + * trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, un= signed long addr) ops->flags |=3D MULTI_FLAGS; ops->trampoline =3D FTRACE_REGS_ADDR; ops->direct_call =3D addr; + ftrace_direct_mark_module(addr); =20 err =3D register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsign= ed long addr) =20 lockdep_assert_held_once(&direct_mutex); =20 + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash =3D ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, = struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } =20 @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, = struct ftrace_hash *hash, b tmp =3D __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct =3D entry->direct; } } --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743023; cv=none; d=zohomail.com; s=zohoarc; b=WN1olMO/FkUYphF1HdngnyKRUDN6eCLgOd1fqIxAkI10sHDyl4bR+f1r7SiEToYwBS4A8QHGSnmHFfp7WBKhoPVvyDAxvZEUOR6q++xlo/CRLKjYA/pl/6DLpY1Yio0FMmUfOWn7pT1uaTOTbhi1plYUyX8E0tCREJuwiOEOAhw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743023; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=pPH1+eiN4EhHtK6rDK3I01/qZvQ6oNbinGntBhruq3Q=; b=SfUFTHLnUBaNBDkCjJZpPAX1s9c7W8z+P66lFY1UyO8cvrMjHUaa4phOX15PEZs9rULa5S6XbbVDjP1ep7cpPFMc13N/p9MAsR+Y9MIj2Un8quGEmkkh3b32MvFqt37dzmu/3EBrVlWiUJFN8/Kac9eHfsc7FLjM7OGvD6CjaBo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743023054332.94398621231153; Fri, 18 Sep 2026 07:50:23 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425497.1649439 (Exim 4.92) (envelope-from ) id 1x7ZuX-00059O-P6; Fri, 18 Sep 2026 14:49:57 +0000 Received: by outflank-mailman (output) from mailman id 1425497.1649439; Fri, 18 Sep 2026 14:49:57 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuX-000597-K3; Fri, 18 Sep 2026 14:49:57 +0000 Received: by outflank-mailman (input) for mailman id 1425497; Fri, 18 Sep 2026 14:49:56 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuW-0004ic-AE for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:49:56 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7ZuV-0054gK-Jm for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:49:55 +0200 Received: from [10.42.69.7] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f71-bab6-0a2a0a5309dd-0a2a4507c416-44 for ; Fri, 18 Sep 2026 16:49:55 +0200 Received: from [74.125.230.140] (helo=mail-qv2-f12.google.com) by tlsNG-ef75cf.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f92-b4ea-0a2a45070019-4a7de68c8b0c-3 for ; Fri, 18 Sep 2026 16:49:55 +0200 Received: by mail-qv2-f12.google.com with SMTP id 6a1803df08f44-9106feddbdfso7371256d6.0 for ; Fri, 18 Sep 2026 07:49:54 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9125800e368sm14412066d6.8.2026.09.18.07.49.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:52 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742994; x=1790347794; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=pPH1+eiN4EhHtK6rDK3I01/qZvQ6oNbinGntBhruq3Q=; b=i+Idf0G0nZ3uTlabI3uFjNO/t9kVswVLJb7snaSrP3fRvh1bqvyICrIqqIu9utHrKT ojZShzk3duHWyjYd7hLeQQUKv9OsNOL9JJjJU9KshuGYTiENQ8/6CNG6um+vbSJbcC9a hG3FEUu4WbApAdBRgAYXsxyU5lddTJF8z/0/I75ny1mQGWq75ByuqJqUS4ifTzWrtggw 01TD6708hgWx7LBC1SFUiccGcQ+o799QM4x9yrjHmSFa0myO69wmB8vzUEDeAT8zz3fG hIvTUE7jlNVmFkoaPgbIngyKYVmJaj4ovDlSawVVKeETe00iTkW5OhDMliX4EAy1vvus cPGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742994; x=1790347794; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=pPH1+eiN4EhHtK6rDK3I01/qZvQ6oNbinGntBhruq3Q=; b=y0wFHl+Z5G6ny7wf2Bo6N869ivLb//rrgEVBB20JjqApUyLscLN15qpYnnvmgoR+s4 2QQ+cg0yHY9KUAaYXM7iTVpOgzV832FxFyXHBU9LASzgzy7NN0y1ytyyrWYZWkccUOvc vWtAqsZQa/ygiIanQNHZabeWMgk2LmCme5faIVy1KlXqFvP31cbnDSqULHgCNWcmX4pS OQ6R7IUnH3MwK8jrArwFWXTxzHD2yfDZZXYd4FpkBb/9LZvO87wI2adVhJEZJ14SKN6r grckJ8bhiyZAMHA+2kwykNOVSwo1zY0b1MBVQ/wF31fCU7bGyak5QDbEaPMSH6pMdNIM LjCQ== X-Forwarded-Encrypted: i=1; AKwUvBzFjcFDl4c2MtUhr82FGhQeq39eszoqinoKDaYBzQ461c31z3fnbdzXSxlxtHuZ4IZv3pfGwP2lf1k=@lists.xenproject.org X-Gm-Message-State: AFuF++nFXPKb/JjwfMlo2Em3f9Jw+rqJZGS4gawS+2fFABY4Qiox+l2o XmArg5YT2D/kIWdkvI+KpWvcoml00VEiJbJIlj4lK7j/uYhEL9TVqwNPntmO21QLaE4= X-Gm-Gg: AYBFou1kkNrt7WTVgWzSK4s27qqltEjAgVW5qc7I6uj5CdoeN+HokllO4ccy+i0GkiU ApucfqAGzbe47C2Oa3oghPTOPbe6mo8G2HE/spKanh7on/eF98IeeHE0hGB4GWn8Zf15fKPswpq RGoL8lKpWh+ZiFYFl7r8ckWbjPy8BtKR6HU8MWLuyxFnY7ZilAbJ2aDojLbMfxHYy9kBwL/WA9S b8EBRrG1tzTHwc2kpHvbGnLwELZVBM/Ng8TFI/HLOdj+kpI7jSUsaA8gAJtO+k9uoHnqdHSIcQB ONOt5zlDp3fMn3SwiBdXW4Ct5RpX4NTipU4cGWFOauQfuFgAIrgJikojpfH4tlT8qBJ4BExgPEq 1hRGc8TTr845EIq7YW+pXr7SOsbGxvhCR8wAzLVESyz+B8o3XK+xIfOOCFZhvxxVOi+xKcaAVs/ 9zcDipkZz/tnFeuBVLHWnBEInP0lCLXqoLVhsmkvw3Y0x7ZnPqtOMKfljsFT/UGk5ZoRQJpFnuZ zug9mQPaMmMc5wCOhkrHUAEV1w7vK8p1uE3bc2qoELRXbWpRgZvjVE7d6rPWLhMqzQ= X-Received: by 2002:a05:6214:5d02:b0:910:705a:2c5e with SMTP id 6a1803df08f44-91254bcf2c3mr42299806d6.18.1789742993553; Fri, 18 Sep 2026 07:49:53 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:24 +0000 Subject: [PATCH RFC v4 05/13] x86/ftrace: Take a Tasks Trace reader around ftrace_caller's call-out MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-5-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=9964; i=josef@toxicpanda.com; h=from:subject:message-id; bh=EBgIXBiL5Z/I6XbFK2Nq9sAUVoT2dZ9k9OxjIjZChGA=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QE5RsSfIxHeGJCFPi3zOG/JnoKeCeLOBzXS0wrZafOedFHnVb+Jk6Kqa2Y7pWw6LloZYUQvCiGv 7bLPKG2JGuwc= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-ef75cf/1789742995-A6AD8AE4-89A271E0/0/0 X-purgate-type: clean X-purgate-size: 9966 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743024645158500 For HAVE_RCU_TRAMPOLINE_READERS the ftrace trampolines must be Tasks Trace RCU readers while they call out, since that -- and not the absence of a voluntary context switch -- is what synchronize_rcu_tasks() will wait for before ftrace_shutdown() frees a dynamic trampoline or its ops. Open-code rcu_read_lock_trace() and rcu_read_unlock_trace() in ftrace_caller and ftrace_regs_caller: bump current->trc_reader_nesting and, for the outermost reader, do the SRCU-fast per-CPU increment on rcu_tasks_trace_srcu_struct and stash the counter pointer in current->trc_reader_scp, exactly as the C inlines do (including the smp_mb() when CONFIG_TASKS_TRACE_RCU_NO_MB is not set). The lock sits before the function_trace_op load, because between that load and the call the ops pointer is protected only by Tasks RCU, and the unlock after the call returns. The sequences are inside the region that create_trampoline() copies for per-ops trampolines; their %rip-relative references are fixed up by text_poke_apply_relocation() like CALL_DEPTH_ACCOUNT's. %rax and %rcx are dead at both points. Two pieces of core text still run outside that reader while holding the address of a Tasks-RCU-protected trampoline they are about to enter: the static stubs themselves, whose direct-call tails keep a BPF trampoline address on the stack until the final RET, and, under CONFIG_MITIGATION_RETHUNK, the return thunk that RET expands to. Add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and linker symbols around .text..__x86.return_thunk and .text..__x86.rethunk_safe, and provide arch_rcu_tasks_trampoline_text() covering [ftrace_caller, ftrace_static_tramp_end) and both thunk ranges so the irq-exit check treats a task interrupted there as a holdout. All of this is built only under CONFIG_TASKS_RCU_TRAMPOLINE_READERS, which x86 does not enable until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/asm-offsets.c | 8 +++++ arch/x86/kernel/ftrace.c | 43 +++++++++++++++++++++++++++ arch/x86/kernel/ftrace_64.S | 69 +++++++++++++++++++++++++++++++++++++++= ++++ arch/x86/kernel/vmlinux.lds.S | 4 +++ 4 files changed, 124 insertions(+) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 081816888f7a..876c3986419a 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -46,6 +47,13 @@ static void __used common(void) #ifdef CONFIG_STACKPROTECTOR OFFSET(TASK_stack_canary, task_struct, stack_canary); #endif +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + OFFSET(TASK_trc_reader_nesting, task_struct, trc_reader_nesting); + OFFSET(TASK_trc_reader_scp, task_struct, trc_reader_scp); + OFFSET(SRCU_srcu_ctrp, srcu_struct, srcu_ctrp); + OFFSET(SRCU_CTR_srcu_locks, srcu_ctr, srcu_locks); + OFFSET(SRCU_CTR_srcu_unlocks, srcu_ctr, srcu_unlocks); +#endif =20 BLANK(); OFFSET(pbe_address, pbe, address); diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c index 17d6edfcb7e0..9babaed483eb 100644 --- a/arch/x86/kernel/ftrace.c +++ b/arch/x86/kernel/ftrace.c @@ -275,6 +275,49 @@ static inline void tramp_free(void *tramp) execmem_free(tramp); } =20 +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS +extern void ftrace_static_tramp_end(void); +extern char __return_thunk_start[], __return_thunk_end[]; +extern char __rethunk_safe_start[], __rethunk_safe_end[]; + +/* + * The SRCU-fast increments in TRACE_RCU_READ_LOCK/UNLOCK (ftrace_64.S) ar= e the + * this_cpu_inc() form. + */ +static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)); + +/* + * See rcu_tasks_trampoline_text(). Some core kernel text behaves like a + * trampoline for Tasks RCU purposes because a task executing there outside + * any Tasks Trace reader may still be about to enter a Tasks-RCU-protected + * trampoline whose address it already holds: + * + * - the static ftrace_caller / ftrace_regs_caller / ftrace_stub_direct_t= ramp + * stubs, which carry a direct-call target on the stack until their fin= al + * RET, and + * - the return thunks that RET expands to under CONFIG_MITIGATION_RETHUN= K, + * which run after leaving the stubs above and before landing in that + * target. + */ +bool arch_rcu_tasks_trampoline_text(unsigned long ip) +{ + if (ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end) + return true; +#ifdef CONFIG_MITIGATION_RETPOLINE + if (ip >=3D (unsigned long)__return_thunk_start && + ip < (unsigned long)__return_thunk_end) + return true; +#endif +#ifdef CONFIG_MITIGATION_SRSO + if (ip >=3D (unsigned long)__rethunk_safe_start && + ip < (unsigned long)__rethunk_safe_end) + return true; +#endif + return false; +} +#endif /* CONFIG_TASKS_RCU_TRAMPOLINE_READERS */ + /* Defined as markers to the end of the ftrace default trampolines */ extern void ftrace_regs_caller_end(void); extern void ftrace_caller_end(void); diff --git a/arch/x86/kernel/ftrace_64.S b/arch/x86/kernel/ftrace_64.S index 62c1c93aa1c6..5d8cb3861978 100644 --- a/arch/x86/kernel/ftrace_64.S +++ b/arch/x86/kernel/ftrace_64.S @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -145,6 +146,53 @@ SYM_FUNC_END(ftrace_stub_graph) =20 #ifdef CONFIG_DYNAMIC_FTRACE =20 +/* + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see + * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS: = the + * trampoline and the ftrace_ops it is about to load are kept alive by Tas= ks + * RCU only while we are inside this reader, so the lock must precede the + * function_trace_op load and the unlock must follow the call. These live + * inside the region copied into dynamic trampolines; the %rip-relative + * references are fixed up by text_poke_apply_relocation() in + * create_trampoline(). Clobbers %rax, %rcx and flags. + */ +.macro TRACE_RCU_READ_LOCK +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + movq PER_CPU_VAR(current_task), %rcx + movl TASK_trc_reader_nesting(%rcx), %eax + incl TASK_trc_reader_nesting(%rcx) + testl %eax, %eax + jnz .Ltrl_nested_\@ + movq rcu_tasks_trace_srcu_struct+SRCU_srcu_ctrp(%rip), %rax + incq %gs:SRCU_CTR_srcu_locks(%rax) + movq %rax, TASK_trc_reader_scp(%rcx) +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB + lock addl $0, -4(%rsp) /* smp_mb() */ +#endif +.Ltrl_nested_\@: +#endif +.endm + +.macro TRACE_RCU_READ_UNLOCK +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + movq PER_CPU_VAR(current_task), %rcx + movl TASK_trc_reader_nesting(%rcx), %eax + subl $1, %eax + jnz .Ltru_nested_\@ + /* Outermost: pick up scp before an interrupt can see nesting =3D=3D 0. */ + movq TASK_trc_reader_scp(%rcx), %rax + movl $0, TASK_trc_reader_nesting(%rcx) +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB + lock addl $0, -4(%rsp) /* smp_mb() */ +#endif + incq %gs:SRCU_CTR_srcu_unlocks(%rax) + jmp .Ltru_done_\@ +.Ltru_nested_\@: + movl %eax, TASK_trc_reader_nesting(%rcx) +.Ltru_done_\@: +#endif +.endm + SYM_FUNC_START(__fentry__) ANNOTATE_NOENDBR CALL_DEPTH_ACCOUNT @@ -163,6 +211,8 @@ SYM_FUNC_START(ftrace_caller) leaq MCOUNT_REG_SIZE+8(%rsp), %rcx movq %rcx, RSP(%rsp) =20 + TRACE_RCU_READ_LOCK + SYM_INNER_LABEL(ftrace_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -181,6 +231,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + TRACE_RCU_READ_UNLOCK + /* Handlers can change the RIP */ movq RIP(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -209,6 +261,8 @@ SYM_FUNC_START(ftrace_regs_caller) =20 CALL_DEPTH_ACCOUNT =20 + TRACE_RCU_READ_LOCK + SYM_INNER_LABEL(ftrace_regs_caller_op_ptr, SYM_L_GLOBAL) ANNOTATE_NOENDBR /* Load the ftrace_ops into the 3rd parameter */ @@ -246,6 +300,8 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) ANNOTATE_NOENDBR call ftrace_stub =20 + TRACE_RCU_READ_UNLOCK + /* Copy flags back to SS, to restore them */ movq EFLAGS(%rsp), %rax movq %rax, MCOUNT_REG_SIZE(%rsp) @@ -328,6 +384,19 @@ SYM_FUNC_START(ftrace_stub_direct_tramp) RET SYM_FUNC_END(ftrace_stub_direct_tramp) =20 +/* + * [ftrace_caller, ftrace_static_tramp_end) is treated as trampoline text = by + * rcu_tasks_trampoline_text(): outside TRACE_RCU_READ_LOCK/UNLOCK the stu= bs + * may still hold a direct-call trampoline address (ORIG_RAX / the return + * address they RET to) that only Tasks RCU keeps alive. With return thun= ks + * the RET itself runs elsewhere; arch_rcu_tasks_trampoline_text() covers + * those too. + */ +SYM_CODE_START_NOALIGN(ftrace_static_tramp_end) + UNWIND_HINT_UNDEFINED + ANNOTATE_NOENDBR +SYM_CODE_END(ftrace_static_tramp_end) + #else /* ! CONFIG_DYNAMIC_FTRACE */ =20 SYM_FUNC_START(__fentry__) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 2438b89a4620..e546283dc267 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -151,7 +151,9 @@ SECTIONS * definition. */ . =3D srso_alias_untrain_ret | (1 << 2) | (1 << 8) | (1 << 14) | (1 << 2= 0); + __rethunk_safe_start =3D .; *(.text..__x86.rethunk_safe) + __rethunk_safe_end =3D .; #endif ALIGN_ENTRY_TEXT_END =20 @@ -162,7 +164,9 @@ SECTIONS SOFTIRQENTRY_TEXT #ifdef CONFIG_MITIGATION_RETPOLINE *(.text..__x86.indirect_thunk) + __return_thunk_start =3D .; *(.text..__x86.return_thunk) + __return_thunk_end =3D .; #endif STATIC_CALL_TEXT *(.gnu.warning) --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743024; cv=none; d=zohomail.com; s=zohoarc; b=LcCnRRFd8uzkk0rI+Iib01ve5yIb4MWIZqVbpRfzbjup7Q+VI6OOjByrHbSLvTRmaI9VaTXbjdbAGzwF2XaRDUTBcTuio+wjuDfjsAb5lfbqAxtyiVm/ZH28YftFrp0E0iEkljJz2jTieOLWQgsVTA6fkNBjKPaEw7ERaYa6KIU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743024; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=9lDajmUPmaz8BTxhPEk6XaACKGtk0v9RwI05AN+Obys=; b=noeb8lsnU0gLoOOY1lhcFlbRp0vUOBgahRY9hAjSsqbdeZS36IZw1WabE+uwCU/i7S4PPH4SU5RerI5zQCpCayTI/Px62E1qzGqYmryI0mb2YWUu7i1HBfi2ZdpA+hpixpMCt19l4xgQF9YrWh9mdqDNkjil6OdAY+fk53Z/zvE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743024169779.4012740897053; Fri, 18 Sep 2026 07:50:24 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425498.1649446 (Exim 4.92) (envelope-from ) id 1x7ZuZ-0005Nx-1Y; Fri, 18 Sep 2026 14:49:59 +0000 Received: by outflank-mailman (output) from mailman id 1425498.1649446; Fri, 18 Sep 2026 14:49:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuY-0005Nq-U9; Fri, 18 Sep 2026 14:49:58 +0000 Received: by outflank-mailman (input) for mailman id 1425498; Fri, 18 Sep 2026 14:49:57 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuX-00057Q-Lt for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:49:57 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7ZuX-0054h4-2d for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:49:57 +0200 Received: from [10.42.69.11] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f8b-8faa-0a2a0a5109dd-0a2a450bb8d4-18 for ; Fri, 18 Sep 2026 16:49:57 +0200 Received: from [74.125.230.209] (helo=mail-qk2-f17.google.com) by tlsNG-42698a.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f94-b7e8-0a2a450b0019-4a7de6d1806a-3 for ; Fri, 18 Sep 2026 16:49:56 +0200 Received: by mail-qk2-f17.google.com with SMTP id af79cd13be357-93910ca5aa7so73891185a.1 for ; Fri, 18 Sep 2026 07:49:56 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9125808d8fcsm14099356d6.45.2026.09.18.07.49.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:54 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742995; x=1790347795; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9lDajmUPmaz8BTxhPEk6XaACKGtk0v9RwI05AN+Obys=; b=iu8PZYYhVy56tQ09y3Ij134WzEnOsZy5ohnNPTqGAxeyMYvujJZ8bzzrvtDXnXhrl1 UrsS1mUjG8Wm+h1pfbNMYNDMtXwIoOHvotlrONC/YBSZnrsR7mWom4iOhPiJ8OVSqsjY v3wqM1yhnrnTj9cZ+y2YHwIUWMB5kbFjoSjpTV4n1+Hr/m+hgIgSe4eb017UPDu+x3CG o4ITiCkckYZR0hTmS9wjToy+s53S18N3yk5h/TbBS527Ng39Vv6Q7Yv0IWKn5/PEaQCE fUz9aiuiUQqPMlJgIpi5xA3lTdoHKfjOmJiEyEkotBZYrpmwbxDwXdOoCQROv/+qN1Ws CGnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742995; x=1790347795; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9lDajmUPmaz8BTxhPEk6XaACKGtk0v9RwI05AN+Obys=; b=Ly+HXYxEFUtlRZ43Mxj0qQvsZn2O71B538bUGKUUp22si1Zyve6WJJhj7ci59Lm+Gl Plbr4xNwB0pWqSWN/X7kgvIVlqjvjYUQ6px3Vb6uQRUnZu+s4LYUr9Law0+jUTYjyMg8 hPxegL3npF6r2CdWoW3AJKnO5JkrbfqVTo6FqynWn5W1+Bapl6JCR8uWQZ8JdJu+zVuJ wpasdoSStXVnx+3FtwrxC5WNlTRWtQK2tHxQzgW5+bdGeNV3uGHrJlbP9ApDAxwjnWec ORIR25Zvxxbm6cVIjkZRHYjewFkAxUfD7zBF69VmNwzcY6aHi2oWAYMW8/ZEUCsN9FsX 68JA== X-Forwarded-Encrypted: i=1; AKwUvBwm/wx7AGPZa3nnpTRg0abt4CUZ3P1lViWNGw8x67Ul8II39v5eHvm+a+EcJN55IGItsQ3nTDPJvsQ=@lists.xenproject.org X-Gm-Message-State: AFuF++ksecH852WEL8ntEiQk6mOrw7VlwvgIvxcgXrL5Jx+q+295NrAd pWvFA9vPUcIEjcHEXUV9lWqIUOfXZNefesSDM7szPgBdtAtIPQBcJK0EIUGo1WdRxCM= X-Gm-Gg: AYBFou03a+eZ5ofL4Y1CAsVd7IeKs/MKhSujSQyoGTtZe8+w4R5XrlcbNLLkskYdkCB ePwuU/EYRT9mkg7N8Jltmj6WHvWSc4m/pmFHxTmew3tkDtActpzTSRTygoDfNqTxSgY/OOSoD/P 3O8f1ZRhpKBPHYVXFWIABAzhToxd6fELtj1uOoAjsT9wE4HVzYtUhO62ybPefuKxEV7s+s/VmqM qH9aI4X6eZr6J0HrX8ISaRn1q2SHYFR91lMW4QP1F+VWa6Lj5KbO3ONaN2jLyFLwBR6895uDzc/ 7q5a9zSQFCruB9TWRiSW/22YEwjHbJoI4Jc5D7PiEV5CRflVPD9f+jCbSCLd/PdqsZINi9G+f4X UytoYZWAM9mLlhXZRtfv8JYQfNBWNfOGyB7tcp2Z8DXwSffFMGB6q5xxQvsSeXsL4zhA0sgIdbM AlisOofnMiVU33+VuvH0V/3cvy5+R0nw+0XrlBeRXFEMNgLnSfUMB1D/LxusWtSVJiVpp5WeyGK iRFf6yi0j7yrz9nwot1wkNspWq5MyZv6RsVbfkIXhOqml+xrcUHLtjb X-Received: by 2002:a05:620a:2591:b0:939:3626:558b with SMTP id af79cd13be357-93bdc6e9664mr386099885a.5.1789742995331; Fri, 18 Sep 2026 07:49:55 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:25 +0000 Subject: [PATCH RFC v4 06/13] x86/kprobes: Take a Tasks Trace reader in the optprobe template MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-6-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=3977; i=josef@toxicpanda.com; h=from:subject:message-id; bh=cyOEpv7E+99KbsYBQF12i4qmiz/SB2W5CJ1v6BiG3CI=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QEjEDkoMCTK8/SkEBNPAmAQa/DHNGXCcOq6l1exqCfOV/HOva5SRV+bPucef9uXdDcfTxk/sgpw 4F0zavhp0AA0= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-42698a/1789742997-A92CC9EA-AB086A2A/0/0 X-purgate-type: clean X-purgate-size: 3979 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743024537158500 The jump-optimized kprobe template calls optimized_callback() from a dynamically allocated slot with preemption enabled, and only Tasks RCU keeps that slot alive under a task preempted in the callback. For HAVE_RCU_TRAMPOLINE_READERS that means the template must be a Tasks Trace reader across the call, so open-code rcu_read_lock_trace() and rcu_read_unlock_trace() around it as ftrace_64.S does. The template lives in .rodata and is memcpy()d into each slot without relocation processing, so the references to current_task and rcu_tasks_trace_srcu_struct are absolute (R_X86_64_32S, relocated for KASLR like any other) rather than %rip-relative. %rax and %rcx have already been saved by SAVE_REGS_STRING and are dead after the call. The slot itself is dynamically allocated text, so the instructions before the lock and after the unlock are covered by the irq-exit check. 64-bit only; 32-bit x86 does not take part. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/kernel/kprobes/opt.c | 44 +++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 44 insertions(+) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 3f8fea52619f..68a5de6cdabe 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -31,6 +31,7 @@ #include #include #include +#include =20 #include "common.h" =20 @@ -101,6 +102,47 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr,= unsigned long val) *(unsigned long *)addr =3D val; } =20 +/* + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() around the c= all + * to optimized_callback(), see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the + * equivalent macros in ftrace_64.S. The template is memcpy()d into the s= lot + * without relocation processing, so memory references must be absolute + * rather than %rip-relative. %rax and %rcx are free at both points. + */ +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB +#define OPTPROBE_TRACE_RCU_MB " lock addl $0, -4(%rsp)\n" +#else +#define OPTPROBE_TRACE_RCU_MB +#endif +#define OPTPROBE_TRACE_RCU_READ_LOCK \ + " movq %gs:current_task, %rcx\n" \ + " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \ + " incl " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \ + " testl %eax, %eax\n" \ + " jnz 1f\n" \ + " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) ", %rax= \n" \ + " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%rax)\n" \ + " movq %rax, " __stringify(TASK_trc_reader_scp) "(%rcx)\n" \ + OPTPROBE_TRACE_RCU_MB \ + "1:\n" +#define OPTPROBE_TRACE_RCU_READ_UNLOCK \ + " movq %gs:current_task, %rcx\n" \ + " movl " __stringify(TASK_trc_reader_nesting) "(%rcx), %eax\n" \ + " subl $1, %eax\n" \ + " jnz 2f\n" \ + " movq " __stringify(TASK_trc_reader_scp) "(%rcx), %rax\n" \ + " movl $0, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \ + OPTPROBE_TRACE_RCU_MB \ + " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%rax)\n" \ + " jmp 3f\n" \ + "2: movl %eax, " __stringify(TASK_trc_reader_nesting) "(%rcx)\n" \ + "3:\n" +#else +#define OPTPROBE_TRACE_RCU_READ_LOCK +#define OPTPROBE_TRACE_RCU_READ_UNLOCK +#endif + asm ( ".pushsection .rodata\n" ".global optprobe_template_entry\n" @@ -114,6 +156,7 @@ asm ( "optprobe_template_clac:\n" ASM_NOP3 SAVE_REGS_STRING + OPTPROBE_TRACE_RCU_READ_LOCK " movq %rsp, %rsi\n" ".global optprobe_template_val\n" "optprobe_template_val:\n" @@ -122,6 +165,7 @@ asm ( ".global optprobe_template_call\n" "optprobe_template_call:\n" ASM_NOP5 + OPTPROBE_TRACE_RCU_READ_UNLOCK /* Copy 'regs->flags' into 'regs->ss'. */ " movq 18*8(%rsp), %rdx\n" " movq %rdx, 20*8(%rsp)\n" --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743027; cv=none; d=zohomail.com; s=zohoarc; b=ey+hDR3lxCO9hZ9s5fyTOHWiUIYlyQAHvwevLxpvqIEIerwfycOXJ0wn1n4dThQivnYkr4nYjaSBPweKyQFcXn3zkmIM/jZ55sO/LRI88mKQP2dLy5nm+skZu8/MZLdVAXnFrV4cDfVFqE09mzaDxzAEwhNDV+CrZt1tNRUHaV8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743027; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Tx2UYwtxbz9ZqVNvSp5mlXDS3jyz+Ej3sfBe8tuEQZo=; b=G9Vr5tttmIlyuTK0iYRGC4wU3O95x+yh5uhRruHWQ/MJtXscs+aIMYbvE2LxV9FyAdKdWsoXLd4M1rnnI217HiVk7mojoVsAW4+OOAcQLSU3HLQVJ+QKEFnomuWMJCigXFTGDzWWCE3Yq+67gFPuKZjuuR32CSdB3NFsASBO4sA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743027952937.042995596212; Fri, 18 Sep 2026 07:50:27 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425499.1649456 (Exim 4.92) (envelope-from ) id 1x7Zub-0005gJ-Gd; Fri, 18 Sep 2026 14:50:01 +0000 Received: by outflank-mailman (output) from mailman id 1425499.1649456; Fri, 18 Sep 2026 14:50:01 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zub-0005g8-CS; Fri, 18 Sep 2026 14:50:01 +0000 Received: by outflank-mailman (input) for mailman id 1425499; Fri, 18 Sep 2026 14:49:59 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7ZuZ-0005U8-Kw for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:49:59 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7ZuZ-0054h4-1p for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:49:59 +0200 Received: from [10.42.69.7] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f97-8faa-0a2a0a5109dd-0a2a4507eda8-0 for ; Fri, 18 Sep 2026 16:49:59 +0200 Received: from [209.85.160.171] (helo=mail-qt1-f171.google.com) by tlsNG-ef75cf.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f95-b4ea-0a2a45070019-d155a0aba40a-3 for ; Fri, 18 Sep 2026 16:49:58 +0200 Received: by mail-qt1-f171.google.com with SMTP id d75a77b69052e-52ff0eea420so6539841cf.0 for ; Fri, 18 Sep 2026 07:49:58 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532ac48df0fsm2560241cf.6.2026.09.18.07.49.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:56 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742997; x=1790347797; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Tx2UYwtxbz9ZqVNvSp5mlXDS3jyz+Ej3sfBe8tuEQZo=; b=E2IjlHcSBercr7jNqDVvN2nxUYcKbDPPAxG954ib+qwqRPH4B2zHgxgl3Mf5wLPlJF NpB9PJTUt/CLqL+PafCC6572u3RtwiGzrZGJYwP15sR35Y/fcGPgZANWrf9XWN93Sf51 a5mt5LWiXYXQSP5HFcwvF4yIGw5n2l6Nt9xOGyFCk1aqHDvi/TBHXJwRKBWvQHOxetuM QIr1W2qL4j4ndipkysE0HkFfxxVkFzzcSG9cxOl0i6wXf2p1bFr9ls71fX076STrb/A1 7KrRgghjFfBAt0KuZrqqtK4a9RxnGtWVcu6cdpEX/Hxj+qjxHcSDLMHs7q9QR6+Ymulm mG5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742997; x=1790347797; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Tx2UYwtxbz9ZqVNvSp5mlXDS3jyz+Ej3sfBe8tuEQZo=; b=laKZIP2poUZGfHVY5ctY7ZF53zLCo7wV8Z+LiuJGj9FhdVyYgBUaHC0w8Pn13h4026 YxfD6WnPcV9GJODJ2L56NJpEADVE2hJrBgF51RFYxuW0ds5h6EB8FrKNZH4h1LeNVnlZ RO90gynm3w29kVrE63a5gJ2kgCMKd8SpqGiDsgFu6nJezEkVzvTXK5Wk8u4CyvwPm4RA Md5vWtopRPIvSQdq7GHW/WRw5OpYdGIyXgUypzcgKrkNuyreSvSZGgt4CGRGfwe6GWHx Oh8TR6m/tRFJr1pMkAVYyo3ut0uoBoJtb92kJWkhjPat6ksHuw9//o5+rQGh+Dgvx8H1 ANKA== X-Forwarded-Encrypted: i=1; AKwUvBxT+1JA0ELvlBm1nF8GrTbPGP79cyNoK+w5vgukIdK0xFAT/Q1WBGZTyl0YwAli2uePxN4UvGmpycE=@lists.xenproject.org X-Gm-Message-State: AFuF++lPxhu5MNLBz3IBI4rOZUZ2o71PJJaCUyiRNXYo+Xjaj6Jl/mno e+raof6OnTut7SIom8jVuKkx4b7AH4RjcRMWic9fuXBwafXHeQckKCyKLUrlR3BWYRc= X-Gm-Gg: AYBFou3jx5kaxcbz563ef8oXnms7Nx4tYqGhhVy4DscApVyUVX06Hw+R+sg1L8idlKJ FUMluPzzv1IxkQUmhSA0eUYpE5F7RPmREurvE5xS85JxosVDW+tvp0a2kbm+7Y9BX+QC+4K87YX qX5Em2zOcjHzlcrPnL4LcEOr3kQHyWbYJ5GAtRwkrnLIbA0LUuOx/IGnMApEonXHaY7eNjdwnfO GYtl6jTZw4MFxzEinx3ELSKTZMubW15rEr7waVOybr4lFrI1t5dL48Sf8evZ6T3dDPLDjs6XNou AYRYxzYs+CmjjcSXPIxB/SZboIEyNaBCGRkZOK49iN5De0CcE3v+E5o7aRfblreL7hpJnzfHboR Q1HaYzGAVjxidiHIVyrJmB47GhTHPgeMlW4Q4iOVq7cWMG60r2qOucvyaqYnPt9WdRf7hjNHEZ0 aEpXGRDDFxISZQVkBfpqA+ye8NU8kyzZrHaOq/f8/1y+NZgrs0iO4V5CNatfYFAZn+UvbSLLWFL ocywhJ15DhobhQvSDxUsrSD8/Yku3lXFUactiVjdLVq2qOCJOZ06rY6 X-Received: by 2002:ac8:5f4f:0:b0:531:e51:12f9 with SMTP id d75a77b69052e-5328cfc0a9fmr106591791cf.24.1789742997160; Fri, 18 Sep 2026 07:49:57 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:26 +0000 Subject: [PATCH RFC v4 07/13] bpf, x86: Take a Tasks Trace reader in the trampoline around its call-outs MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-7-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=8086; i=josef@toxicpanda.com; h=from:subject:message-id; bh=JGl/QtTBddM7nyy/qfbMR7PzifAXs0WCS5aHiUuitqA=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QIEBfe7NVaGlqFqLQwZKbFw8BpHmlfuvIjJBnOTYn9qtDCjTJL+2uxQrDvhCu7CSaCn6ZdAis0V Bpsr/bJffvAk= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-ef75cf/1789742999-35EC6AE4-2408CD1E/0/0 X-purgate-type: clean X-purgate-size: 8088 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743028790158500 On HAVE_RCU_TRAMPOLINE_READERS kernels Tasks RCU keeps a BPF trampoline image allocated only while a task using it is a Tasks Trace RCU reader or is executing text that rcu_tasks_trampoline_text() recognises. The image itself is such text, but the C glue and the programs it calls are not, and only sleepable programs take rcu_read_lock_trace() today. Have the x86-64 JIT open-code rcu_read_lock_trace() and rcu_read_unlock_trace() in the trampoline, as ftrace_64.S does for ftrace_caller: one reader from just after the frame is set up to just before the original function is called, covering __bpf_tramp_enter() and the fentry and fmod_ret programs, and a second one from just after the original function returns to just before the final register restore, covering the fexit programs and __bpf_tramp_exit(). The original function itself runs outside both, since it may run for a long time and the image is pinned by im->pcref across it. Trampolines that do not call the original function get a single reader around all their programs. The second reader is entered before ip_after_call, so the ip_after_call -> ip_epilogue jump that bpf_tramp_image_put() patches in is inside it, and the fmod_ret early-exit branch lands after that point still holding the first reader, so exactly one is held on every path. The sequence uses r10 and r11, which are scratch at each emission point, and references current_task and rcu_tasks_trace_srcu_struct by absolute sign-extended address, the form the JIT already relies on for this_cpu_off. Sleepable programs' own rcu_read_lock_trace() simply nests. Nothing is emitted on other configurations. Suggested-by: Alexei Starovoitov Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/net/bpf_jit_comp.c | 113 ++++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 113 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2853e87797a7..c991f7ceacdf 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -722,6 +723,97 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg= , u8 *ip) *pprog =3D prog; } =20 +/* + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the + * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent + * macros in arch/x86/kernel/ftrace_64.S. The image is not relocated, so + * current_task and rcu_tasks_trace_srcu_struct are referenced by absolute + * (sign-extended 32-bit) address, the form the JIT already relies on for + * this_cpu_off. Uses r10 and r11, which are scratch at every emission + * point, and clobbers flags. + * + * lock: unlock: + * mov r11, gs:[current_task] mov r11, gs:[current_task] + * mov r10d, [r11+nesting] mov r10d, [r11+nesting] + * inc dword ptr [r11+nesting] sub r10d, 1 + * test r10d, r10d jnz 2f + * jnz 1f mov r10, [r11+scp] + * mov r10, [&srcu.srcu_ctrp] mov dword ptr [r11+nesting], 0 + * inc qword ptr gs:[r10+locks] (smp_mb) + * mov [r11+scp], r10 inc qword ptr gs:[r10+unlocks] + * (smp_mb) jmp 3f + * 1: 2: mov [r11+nesting], r10d + * 3: + */ +static void emit_trace_rcu_reader(u8 **pprog, bool lock) +{ +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + const u32 nesting =3D offsetof(struct task_struct, trc_reader_nesting); + const u32 scp =3D offsetof(struct task_struct, trc_reader_scp); + const bool mb =3D !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB); + u8 *prog =3D *pprog; + + BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)); + BUILD_BUG_ON(offsetof(struct srcu_ctr, srcu_locks) !=3D 0); + BUILD_BUG_ON(offsetof(struct srcu_ctr, srcu_unlocks) !=3D 8); + + /* mov r11, gs:[abs32 current_task] */ + EMIT2(0x65, 0x4C); + EMIT3(0x8B, 0x1C, 0x25); + EMIT((u32)(unsigned long)¤t_task, 4); + /* mov r10d, dword ptr [r11 + nesting] */ + EMIT3(0x45, 0x8B, 0x93); + EMIT(nesting, 4); + + if (lock) { + /* inc dword ptr [r11 + nesting] */ + EMIT3(0x41, 0xFF, 0x83); + EMIT(nesting, 4); + /* test r10d, r10d */ + EMIT3(0x45, 0x85, 0xD2); + /* jnz 1f */ + EMIT2(X86_JNE, 8 + 4 + 7 + (mb ? 6 : 0)); + /* mov r10, qword ptr [abs32 &rcu_tasks_trace_srcu_struct.srcu_ctrp] */ + EMIT4(0x4C, 0x8B, 0x14, 0x25); + EMIT((u32)(unsigned long)&rcu_tasks_trace_srcu_struct.srcu_ctrp, 4); + /* inc qword ptr gs:[r10] */ + EMIT4(0x65, 0x49, 0xFF, 0x02); + /* mov qword ptr [r11 + scp], r10 */ + EMIT3(0x4D, 0x89, 0x93); + EMIT(scp, 4); + /* smp_mb(): lock add dword ptr [rsp - 4], 0 */ + if (mb) + EMIT2_off32(0xF0, 0x83, 0x00FC2444); + /* 1: */ + } else { + /* sub r10d, 1 */ + EMIT4(0x41, 0x83, 0xEA, 0x01); + /* jnz 2f */ + EMIT2(X86_JNE, 7 + 11 + (mb ? 6 : 0) + 5 + 2); + /* mov r10, qword ptr [r11 + scp] */ + EMIT3(0x4D, 0x8B, 0x93); + EMIT(scp, 4); + /* mov dword ptr [r11 + nesting], 0 */ + EMIT3(0x41, 0xC7, 0x83); + EMIT(nesting, 4); + EMIT(0, 4); + if (mb) + EMIT2_off32(0xF0, 0x83, 0x00FC2444); + /* inc qword ptr gs:[r10 + 8] */ + EMIT4(0x65, 0x49, 0xFF, 0x42); + EMIT1(0x08); + /* jmp 3f */ + EMIT2(0xEB, 7); + /* 2: mov dword ptr [r11 + nesting], r10d */ + EMIT3(0x45, 0x89, 0x93); + EMIT(nesting, 4); + /* 3: */ + } + + *pprog =3D prog; +#endif +} + static void emit_return(u8 **pprog, u8 *ip) { u8 *prog =3D *pprog; @@ -3610,6 +3702,16 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im /* mov QWORD PTR [rbp - rbx_off], rbx */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); =20 + /* + * Tasks RCU keeps this image alive only while we are a Tasks Trace + * reader; the instructions before this point (and after the final + * unlock) are covered by the irq-exit IP check. One reader spans + * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one + * the fexit progs and __bpf_tramp_exit(); the original function runs + * outside both, with the image pinned by im->pcref instead. + */ + emit_trace_rcu_reader(&prog, true); + func_meta =3D nr_regs; /* Store number of argument registers of the traced function */ emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); @@ -3660,6 +3762,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im } =20 if (flags & BPF_TRAMP_F_CALL_ORIG) { + emit_trace_rcu_reader(&prog, false); restore_regs(m, &prog, regs_off); save_args(m, &prog, arg_stack_off, true, flags, 0); =20 @@ -3682,6 +3785,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_= tramp_image *im, void *rw_im } /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); + /* + * Second reader. Taken before ip_after_call so that the + * ip_after_call -> ip_epilogue jump patched in at teardown is + * inside it too; the fmod_ret early exit jumps past this still + * holding the first reader, so either way exactly one is held. + */ + emit_trace_rcu_reader(&prog, true); im->ip_after_call =3D image + (prog - (u8 *)rw_image); emit_nops(&prog, X86_PATCH_SIZE); } @@ -3737,6 +3847,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_t= ramp_image *im, void *rw_im LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size); } =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_trace_rcu_reader(&prog, false); + /* restore return value of orig_call or fentry prog back into RAX */ if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Received: from mail-qk2-f43.google.com (mail-qk2-f43.google.com [74.125.230.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC3804AD7C9 for ; Fri, 18 Sep 2026 14:50:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743010; cv=none; b=qQdEA8TLhNp/gLJFD3vMNaJCNJrHGWyYyLh6uJdOOeA2X5powNEybB9uht5mVP3axHN+50QkMoyfZpLQH8ibUqhAcmAm2IhaSmC/CJDtdKLfXvlZvybrFZeho8YZrfvu0HclrSiUhLizF3jEvNfsFhhpzX0+Hi8ogcEHHIrYN3M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743010; c=relaxed/simple; bh=Aqm0C5BBMBkqFusbG93kcm6leRi28F4VKStTPzg6dAE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=t3F5HiNGBDq89WxbTW1t7eITl3ZtAivu5itQzrifyJDm5T8ildMdoKl8z3nzpRITyv9GXoriMi3baIh0clK6lZh/zr7eLc2oDPz49h8Oy/LDKzwpugsVOs6PS98bbWxqmZfrystkddsCOPFNNNPeHPwnW+WE/MxHACeMKRMjmcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=AyexF3lD; arc=none smtp.client-ip=74.125.230.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="AyexF3lD" Received: by mail-qk2-f43.google.com with SMTP id af79cd13be357-93910c9ff1cso68811185a.3 for ; Fri, 18 Sep 2026 07:50:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789742999; x=1790347799; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ua8lq0Nx07F8Y+BgJ1JiUwgoDOin8IZmspGmQtj7ybE=; b=AyexF3lDDG3gwT5NW6ieFJ+0B5vIoCHqmzIILR6w4zRjhBrffJbg70uuPgKrLzPmY1 MIkCUQ3jEj8gY9GSax/ZvCpoCVZwdxqa34lU7IfG+uFy95QziGkm+0YANEyvghHUFBjJ MkMeZo7o41xeavy3Ynymth+7GWQrtzBCGgZ8nEAehTa9XwWfNj1QfoGuLpuf2pNeQuTT xetaP/dS6OHt8/ql2oUHmpBUaATU183RIo3crkZraShZPT1DcUkIdu7uQ0r5fCOwnvkb DjwUPD1nj+Oi0ai920R3sie2ryPNiWczsku8N03v1dTq9QfKT2LGKukO9rjGgoVYZgdB aE3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789742999; x=1790347799; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ua8lq0Nx07F8Y+BgJ1JiUwgoDOin8IZmspGmQtj7ybE=; b=h+VQ2v4D3e8EIIIp2vLA3vS7rdHy8p1KloyhpjPBxRCT9xVpmgUjLBfj7ZKAilIZIA 4JhueYm0QCI4gnsLKCpb01CIzqcDVtxojA8JqeVgY9aHbIfJI2hf25KScVjz6lzrknWC U9PJiABxhjQiUNqN2NE9kM+an2xLeSvOUfw8yvvGfIg5+j1DiL5fYE57nFX4LfSa6L4e lgL7sThn/jKQz5VXh8d7o8g3X14NDJzHZoGkNiNJsf3MC5XGZRqFSp7eauZya7X+lCz9 Dv7CtoczSo+8DEtDUsVamNjSm5ZjHPG8qxLSZOyFUyrpaQ+9v6qIEN3A7vPRj/KMeATb CHmQ== X-Forwarded-Encrypted: i=1; AKwUvBwMEQPWEN7E5Grr+MhDBHX+T4Hs+tiD9m0AIv/Ip6J/yHcchLTInhdVfBN3KMACzLn6u0M7VanQggK6zQc=@vger.kernel.org X-Gm-Message-State: AFuF++lJf4P1KoAjkA6FUnN+LworqvpwV/mSdUH70eYupgvfNHhHCq3+ +lCX2IOVngpJ6rL2fWKdfidWIcRmMbLfSFxGMooGgiEqPbnGjaFkEdmo5hveK8Pn/OQ= X-Gm-Gg: AYBFou3kGKI4sUTfGtbo3HwvyvCvGlEVyyvQSez00eyrjLw/bUH6ylE7y2BQ62SR0B/ 4Dh6Xt3Bconh0IvOsyaHlB6ytIVyqSCoxZZz6E7dm37cwh3sm7q3I3JbFjfm9GvjEB6oK9+27pu 0q+Vl8kzVz+b9SZj820b3WQ6hO/pKsao7gbG7upCn+C4zHwi8GfPj0FktOoBeUiVO9w6hK1H/9p xe95Q3th7TYb/EPhYLyDhrpZBBKuz6jZahRZoE7swQJlC2lL4VU5b7UdAlzdh/8DLV9xzBx8yea +ELEwCqiB0tpyvmv35YRfN44eeD5dt8MCjlCkQDDimXWmIqZsr1MjsW4t2S9Sd5mEkpoH1YYxpe lTKdBBODuKLXaCeSc1gdn33DM5U9x72pXUXThAsU3hieXYAMUUPEIxqswl2F4NAIikNdT96ctN2 0zqATV4Yj5LGLmc1waUuDTbj8MuA5URirhtd8/Yr2uoHU4CGRKhXVjUED0DPwVrg9Ers6NG8R5u nBkWHPhaTT9ZZotwopv0Ucv8qnbOGlJ1N2GMXLaHItyRpuq84uO59qJ X-Received: by 2002:a05:620a:1a05:b0:939:733d:cbe0 with SMTP id af79cd13be357-93bdc65cdefmr384366585a.3.1789742998818; Fri, 18 Sep 2026 07:49:58 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-93be0edf295sm154059185a.33.2026.09.18.07.49.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:58 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:27 +0000 Subject: [PATCH RFC v4 08/13] arm64: ftrace: Take a Tasks Trace reader around ftrace_caller's call-out Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-8-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=7839; i=josef@toxicpanda.com; h=from:subject:message-id; bh=Aqm0C5BBMBkqFusbG93kcm6leRi28F4VKStTPzg6dAE=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QBDwhofZWNUZ4nqzwdQchm82AWmFCaXImzqI0Y5t/lZTWR+xCeGh1w/K8fFkLJ01W3nr6p09Evj VkL1iXbfRFwc= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA For HAVE_RCU_TRAMPOLINE_READERS the ftrace trampoline must be a Tasks Trace RCU reader while it calls out, since that is what synchronize_rcu_tasks() will wait for before ftrace_shutdown() frees an ftrace_ops (or, with CALL_OPS, lets its owner free it) under a task preempted in the callback. Open-code rcu_read_lock_trace() and rcu_read_unlock_trace() around the call to ops->func in ftrace_caller: bump current->trc_reader_nesting via sp_el0 and, for the outermost reader, do the SRCU-fast per-CPU increment on rcu_tasks_trace_srcu_struct and stash the counter pointer in current->trc_reader_scp, as the C inlines do (including the dmb when CONFIG_TASKS_TRACE_RCU_NO_MB is not set). The per-CPU increment is an LL/SC add on this CPU's counter; being migrated between reading the per-CPU offset and the store-exclusive only means another CPU's counter is incremented atomically instead, which SRCU sums over anyway. x12-x16 are free at both points. arm64 has no return thunks and, with CALL_OPS, no dynamic ftrace trampolines, but ftrace_caller itself carries the ops pointer in x11 from before the reader is entered and a direct-call BPF trampoline address in x17 until the final br/ret after it is left, so mark the end of the static trampoline text and provide arch_rcu_tasks_trampoline_text() covering [ftrace_caller, ftrace_static_tramp_end). Built only under CONFIG_TASKS_RCU_TRAMPOLINE_READERS, which arm64 does not enable until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/kernel/asm-offsets.c | 8 +++++ arch/arm64/kernel/entry-ftrace.S | 74 ++++++++++++++++++++++++++++++++++++= ++++ arch/arm64/kernel/ftrace.c | 20 +++++++++++ 3 files changed, 102 insertions(+) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offset= s.c index 9c853ed3ceab..f6a8fb1f9b43 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -10,6 +10,7 @@ =20 #include #include +#include #include #include #include @@ -39,6 +40,13 @@ int main(void) DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); +#endif +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + DEFINE(TSK_TRC_READER_NESTING, offsetof(struct task_struct, trc_reader_n= esting)); + DEFINE(TSK_TRC_READER_SCP, offsetof(struct task_struct, trc_reader_scp)); + DEFINE(SRCU_SRCU_CTRP, offsetof(struct srcu_struct, srcu_ctrp)); + DEFINE(SRCU_CTR_SRCU_LOCKS, offsetof(struct srcu_ctr, srcu_locks)); + DEFINE(SRCU_CTR_SRCU_UNLOCKS, offsetof(struct srcu_ctr, srcu_unlocks)); #endif BLANK(); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_conte= xt)); diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftr= ace.S index 025140caafe7..fc2805eb9e15 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -14,6 +14,72 @@ #include =20 #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +/* + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace(), see + * include/linux/rcupdate_trace.h and CONFIG_HAVE_RCU_TRAMPOLINE_READERS. = The + * whole of ftrace_caller is treated as trampoline text by the irq-exit ch= eck + * (see arch_rcu_tasks_trampoline_text()), so these only need to bracket t= he + * call out to ops->func; everything before the lock and after the unlock, + * including the direct-call tails that carry a BPF trampoline address in = x17, + * is covered by that. + * + * The SRCU-fast per-CPU increment is done LL/SC on this CPU's counter; be= ing + * migrated between reading the per-CPU offset and the store-exclusive only + * means another CPU's counter is (atomically) incremented, which SRCU sums + * over anyway. Ordering between the nesting count and the scp stash only + * matters against interrupts on this CPU, which observe program order. + * Clobbers x12-x16 and the flags. + */ + .macro trace_rcu_srcu_inc, addr:req, tmp:req, wtmp2:req +8888: ldxr \tmp, [\addr] + add \tmp, \tmp, #1 + stxr \wtmp2, \tmp, [\addr] + cbnz \wtmp2, 8888b + .endm + + .macro trace_rcu_read_lock +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + mrs x12, sp_el0 // current + ldr w13, [x12, #TSK_TRC_READER_NESTING] + add w14, w13, #1 + str w14, [x12, #TSK_TRC_READER_NESTING] + cbnz w13, .Ltrl_nested\@ // interrupted a reader: done + ldr_l x13, rcu_tasks_trace_srcu_struct + SRCU_SRCU_CTRP + str x13, [x12, #TSK_TRC_READER_SCP] + get_this_cpu_offset x14 + add x14, x14, x13 + add x14, x14, #SRCU_CTR_SRCU_LOCKS + trace_rcu_srcu_inc x14, x15, w16 +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB + dmb ish +#endif +.Ltrl_nested\@: +#endif + .endm + + .macro trace_rcu_read_unlock +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + mrs x12, sp_el0 // current + ldr w13, [x12, #TSK_TRC_READER_NESTING] + subs w13, w13, #1 + b.ne .Ltru_nested\@ + /* Outermost: pick up scp before an interrupt can see nesting =3D=3D 0. */ + ldr x14, [x12, #TSK_TRC_READER_SCP] + str wzr, [x12, #TSK_TRC_READER_NESTING] +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB + dmb ish +#endif + get_this_cpu_offset x15 + add x14, x14, x15 + add x14, x14, #SRCU_CTR_SRCU_UNLOCKS + trace_rcu_srcu_inc x14, x15, w16 + b .Ltru_done\@ +.Ltru_nested\@: + str w13, [x12, #TSK_TRC_READER_NESTING] +.Ltru_done\@: +#endif + .endm + /* * Due to -fpatchable-function-entry=3D2, the compiler has placed two NOPs= before * the regular function prologue. For an enabled callsite, ftrace_init_nop= () and @@ -94,6 +160,8 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE =20 + trace_rcu_read_lock + /* Prepare arguments for the tracer func */ sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) @@ -111,6 +179,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) bl ftrace_stub // func(ip, parent_ip, op, regs) #endif =20 + trace_rcu_read_unlock + /* * At the callsite x0-x8 and x19-x30 were live. Any C code will have prese= rved * x19-x29 per the AAPCS, and we created frame records upon entry, so we n= eed @@ -178,6 +248,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp) SYM_CODE_END(ftrace_stub_direct_tramp) #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ =20 +/* End of [ftrace_caller, ...) for arch_rcu_tasks_trampoline_text(). */ +SYM_CODE_START(ftrace_static_tramp_end) +SYM_CODE_END(ftrace_static_tramp_end) + #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ =20 /* diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index e1a3c0b3a051..5f4193f15cd9 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -17,6 +17,26 @@ #include #include =20 +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS +extern void ftrace_static_tramp_end(void); + +/* The SRCU-fast increments in entry-ftrace.S are the this_cpu_inc() form.= */ +static_assert(!IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)); + +/* + * See rcu_tasks_trampoline_text(). ftrace_caller and ftrace_stub_direct_= tramp + * are core kernel text but must be treated as trampolines: a task interru= pted + * in them outside the Tasks Trace reader may be carrying an ops pointer (= x11) + * or a direct-call BPF trampoline address (x17) whose lifetime is guarded= only + * by Tasks RCU. + */ +bool arch_rcu_tasks_trampoline_text(unsigned long ip) +{ + return ip >=3D (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end; +} +#endif + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct fregs_offset { const char *name; --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743028; cv=none; d=zohomail.com; s=zohoarc; b=EqYIWVaRpj2Fa/sfHsY4EHF95uFKDrCpKfk0kcwV6wpuGj+HCmvI/7inGl7v9ISKG2LwHtbpMdczn+E0EkTeMiYFLgSj250EIFCE6wEKGG5LKnPwbjLDq3QH65wwGFCGW6sEKcAX9wfl9JUwjKalwSvSKsfkx8GONTRLw/q+jgA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743028; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Q8bPS+4IlAeB++9fkK/U7jkDL1GJ1oJuOJz5hfsv6cg=; b=Tz2ZQT7vRQbkr6MjkFdaXlUqEFBlFob0HP9uwv61Ze+CMkgx9QRl3jmTrGhizCLl69PaWbLB7NOhABAaW2bB4yRwqSxoa4yh4kMxoO1UbIxdN/Ei/DRLy1403WRQAy1CVkMSeJvIVYUEsMoW6S2XkgLJe0mTo9t3h55M1DnluPU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743028305404.796867167576; Fri, 18 Sep 2026 07:50:28 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425509.1649488 (Exim 4.92) (envelope-from ) id 1x7Zuk-0007a9-E3; Fri, 18 Sep 2026 14:50:10 +0000 Received: by outflank-mailman (output) from mailman id 1425509.1649488; Fri, 18 Sep 2026 14:50:10 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zuk-0007Yo-4D; Fri, 18 Sep 2026 14:50:10 +0000 Received: by outflank-mailman (input) for mailman id 1425509; Fri, 18 Sep 2026 14:50:08 +0000 Received: from mx.expurgate.net ([195.190.135.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zui-0007GO-Jh for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:50:08 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7Zui-009ha3-0e for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:50:08 +0200 Received: from [10.42.69.2] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f96-e002-0a2a0a5209dd-0a2a450294da-18 for ; Fri, 18 Sep 2026 16:50:08 +0200 Received: from [74.125.230.141] (helo=mail-qv2-f13.google.com) by tlsNG-720697.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f99-6ca4-0a2a45020019-4a7de68d8058-3 for ; Fri, 18 Sep 2026 16:50:02 +0200 Received: by mail-qv2-f13.google.com with SMTP id 6a1803df08f44-90cdfc9b6e3so9327956d6.1 for ; Fri, 18 Sep 2026 07:50:02 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-93be0e95a4asm151763985a.18.2026.09.18.07.49.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:49:59 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789743001; x=1790347801; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Q8bPS+4IlAeB++9fkK/U7jkDL1GJ1oJuOJz5hfsv6cg=; b=Wwprqy+zlj0lZVzG+Bme27kUhv6Qvn/3LCKyK4nmM7Ejxm+MFWSp25P3ipS5JT9eV2 Igs9p1phJetTI2qFzl/AhrSHwz9dA1+HG7WyXwCe0IcEVJCUKCoqmGvLal26JfmKJrwo +qp5sSVGVMk4Ric6dwSyJktF7QMXcdO6RZPB1sbPvYNe9ztZgBBNRqgtutA1EW/tRomL kSYA65y0hPgaQGyoXnOPoWpJhFfc9d6l6al5duIbJDO+dFQvtX2jxX1vUIOmL7PsW1fz jFre/LyX+ZJ3ZKHyZ7K7Q+vTagyvfVwji7rL+UU/Hd97lizKTakoKbGkNUrH2Ma4HlqG rpdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789743001; x=1790347801; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Q8bPS+4IlAeB++9fkK/U7jkDL1GJ1oJuOJz5hfsv6cg=; b=bfBuoQrKuy3tTYsTgj73mAVtjEc/JfFq3D4SdFk+8CWlIPujqWYyh4SVD/hpAIxsoi 4ffnJqHTZ2S+r9I8udaa9iBEZxVJCqSUBVvKyFbgiA5a/MOs9a9Jjb3XHNX7pDsOWPQI jgDHL2ICzB5PT17GaoFGjJ1tSxWCdYVoJkPCbzlwGOCtreUIeIN1ICeIfBPyrgyHIym8 keVkD2tiD+OevxI7kGEtkJQOMWxK5H20b5i+nBhH1eZnfHaYD1HrkhwEAmp8/YmRmdML CiDuMR2vKf1QIBfup/4xqydDFBxcRwzJBKUTBDGBntiWY2EZdg7ytGQIRtNETJM6syOM daqg== X-Forwarded-Encrypted: i=1; AKwUvBzR/duahPry7kFyipESCKxNUXlHlC2l6fFGRmg5+pkOiVnbdkEwfi+9HoEJsQT1Eizcz/zpbvYQENg=@lists.xenproject.org X-Gm-Message-State: AFuF++kxlk5tftl4WyE/FYwElwGV41z59V7SXHWMYI/VxpaTCwQcXird zeHfnTn483Z9jSm9pZ6qnm2/UgAtp7U8piQO5s7zjjBU3Jjt9D/Tmp8jHLqQLlOgWSA= X-Gm-Gg: AYBFou3e+LWt/prxi1OQbP70P35F2nXVDsy07oV4b1OURlay3SN20wmjg5LMiAtCgAn Nc9vz5XwlIiEAEJhOn2qwFAgj56ljxDg794sTZiLC/d9Jp69sWl5shWKCXbT+k3+h3rmT651itE x0uZvIIZbvM3KSBlqIxlcOnJKPyrz7Jf5RpZ+ZYxU0J8xU7Oag8zefQ3Qi0Web+FPqUkaM2X46y LbFyib72iuBtmNGU1T28ji5bu4KgPfCO1fTJ1e1wqsbXa8nXrFCYWVBb/SxvgX7p5xster0lvQb BEHA0/yLBL1eq7ZJvLCwELed3zbA4fkbggKDyV4BzL0mTV88n+jRA/FVKv2t//j+DsiCUciHb8A RP67wMU2JTYRIHDGNfwDVLFptJXjcJSN7slyk+z1HlpgkVSoT58caMCN4ruhohYCLpz2mKGwcX8 ZGQ4+iYURNOd6QIKRHR/MUeNACI3Vc2JUlnDiAF+OVGBiePOVrzAjWqzu+bj+k7X6KDvfCsroUS DhcPyOPeY8VtQ0PSDmbEtCqbcQqyA5JvninXGvrBasF1L9L3MuRGPiT X-Received: by 2002:a05:620a:a2c1:20b0:93b:e9d2:5ece with SMTP id af79cd13be357-93be9d25ee8mr69803785a.39.1789743000697; Fri, 18 Sep 2026 07:50:00 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:28 +0000 Subject: [PATCH RFC v4 09/13] bpf, arm64: Take a Tasks Trace reader in the trampoline around its call-outs MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-9-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=7138; i=josef@toxicpanda.com; h=from:subject:message-id; bh=yHC4KMZfT4OhoIiDiBb7tvVf2glio0Mt2yiw+oyROR0=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QMextWmuyzGMJVeNI8yxyv6Z0gZzWAgn/7KthFAEN2WlTA4mSkVfiREApm3yp97Xx2XO4eWHESm TkxgsjmQ6cwg= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-720697/1789743002-F14AE2AC-8470F112/0/0 X-purgate-type: clean X-purgate-size: 7140 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743030539158500 Same scheme as x86: have the arm64 BPF JIT open-code rcu_read_lock_trace() and rcu_read_unlock_trace() in the trampoline, one reader from after the callee-saved registers are stored to just before the original function is called (covering __bpf_tramp_enter() and the fentry/fmod_ret programs) and a second from just after it returns to just before those registers are restored (covering the fexit programs and __bpf_tramp_exit()), with the original function itself outside both and pinned by im->pcref. The second reader is entered before ip_after_call and the fmod_ret cbnz lands past it still holding the first, so exactly one is held on every path; trampolines without an original call get a single reader. The sequence mirrors entry-ftrace.S: current via sp_el0, trc_reader_nesting bumped, and for the outermost reader the SRCU-fast per-CPU counter incremented LL/SC and the counter pointer stashed in trc_reader_scp (plus the dmb when CONFIG_TASKS_TRACE_RCU_NO_MB is not set). x10-x15 are scratch at every emission point. Nothing is emitted on other configurations. Suggested-by: Alexei Starovoitov Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/net/bpf_jit_comp.c | 90 +++++++++++++++++++++++++++++++++++++++= ++++ 1 file changed, 90 insertions(+) diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index c18e005a41db..87f9d53caf2c 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -14,6 +14,7 @@ #include #include #include +#include #include =20 #include @@ -2591,6 +2592,74 @@ static void emit_arena_arg_conv(struct jit_ctx *ctx,= u8 dst, u8 src, bool nullab emit(A64_SUB(0, dst, src, base_lo), ctx); } =20 +/* + * Open-coded rcu_read_lock_trace() / rcu_read_unlock_trace() for the + * trampoline, see CONFIG_HAVE_RCU_TRAMPOLINE_READERS and the equivalent m= acros + * in arch/arm64/kernel/entry-ftrace.S. The SRCU-fast per-CPU increment i= s an + * LL/SC add on this CPU's counter; being migrated between reading the per= -CPU + * offset and the store-exclusive only means another CPU's counter is + * incremented atomically instead, which SRCU sums over anyway. Uses x10-= x15, + * which are scratch at every emission point, and the flags. + */ +static void emit_trace_rcu_reader(struct jit_ctx *ctx, bool lock) +{ +#ifdef CONFIG_TASKS_RCU_TRAMPOLINE_READERS + const int nesting =3D offsetof(struct task_struct, trc_reader_nesting); + const int scp =3D offsetof(struct task_struct, trc_reader_scp); + const bool mb =3D !IS_ENABLED(CONFIG_TASKS_TRACE_RCU_NO_MB); + const u8 tsk =3D A64_R(10), n =3D A64_R(11), tmp =3D A64_R(12); + const u8 ctr =3D A64_R(13), addr =3D A64_R(14), val =3D A64_R(15); + + BUILD_BUG_ON(IS_ENABLED(CONFIG_NEED_SRCU_NMI_SAFE)); + /* LDR/STR (immediate, unsigned offset) ranges */ + BUILD_BUG_ON((nesting & 3) || nesting >=3D SZ_16K || (scp & 7) || scp >= =3D SZ_32K); + + emit(A64_MRS_SP_EL0(tsk), ctx); /* current */ + emit(A64_LDR32I(n, tsk, nesting), ctx); + if (lock) { + emit(A64_ADD_I(0, tmp, n, 1), ctx); + emit(A64_STR32I(tmp, tsk, nesting), ctx); + /* interrupted a reader: done */ + emit(A64_CBNZ(0, n, 12 + mb), ctx); + /* scp =3D rcu_tasks_trace_srcu_struct.srcu_ctrp; current->trc_reader_sc= p =3D scp */ + emit_addr_mov_i64(ctr, (u64)&rcu_tasks_trace_srcu_struct.srcu_ctrp, ctx); + emit(A64_LDR64I(ctr, ctr, 0), ctx); + emit(A64_STR64I(ctr, tsk, scp), ctx); + } else { + emit(A64_SUB_I(0, n, n, 1), ctx); + /* still nested: just store the count */ + emit(A64_CBNZ(0, n, 11 + mb), ctx); + /* outermost: pick up scp before an interrupt can see nesting =3D=3D 0 */ + emit(A64_LDR64I(ctr, tsk, scp), ctx); + emit(A64_STR32I(A64_ZR, tsk, nesting), ctx); + if (mb) + emit(A64_DMB_ISH, ctx); + } + /* this_cpu_inc(scp->srcu_locks / srcu_unlocks) */ + if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + emit(A64_MRS_TPIDR_EL2(addr), ctx); + else + emit(A64_MRS_TPIDR_EL1(addr), ctx); + emit(A64_ADD(1, addr, addr, ctr), ctx); + if (!lock) + emit(A64_ADD_I(1, addr, addr, offsetof(struct srcu_ctr, srcu_unlocks)), = ctx); + emit(A64_LDXR(1, val, addr), ctx); + emit(A64_ADD_I(1, val, val, 1), ctx); + emit(A64_STXR(1, val, addr, tmp), ctx); + emit(A64_CBNZ(0, tmp, -3), ctx); + if (lock) { + if (mb) + emit(A64_DMB_ISH, ctx); + /* 1: */ + } else { + emit(A64_B(2), ctx); + /* 2: */ + emit(A64_STR32I(n, tsk, nesting), ctx); + /* 3: */ + } +#endif +} + static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off, const struct btf_func_model *m, const struct arg_aux *a, bool for_call_origin, bool is_struct_ops, u64 arena_base) @@ -2854,6 +2923,16 @@ static int prepare_trampoline(struct jit_ctx *ctx, s= truct bpf_tramp_image *im, emit(A64_STR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_STR64I(A64_R(20), A64_SP, regs_off + 8), ctx); =20 + /* + * Tasks RCU keeps this image alive only while we are a Tasks Trace + * reader; the instructions before this point (and after the final + * unlock) are covered by the irq-exit IP check. One reader spans + * __bpf_tramp_enter() and the fentry/fmod_ret progs, a second one the + * fexit progs and __bpf_tramp_exit(); the original function runs + * outside both, with the image pinned by im->pcref instead. + */ + emit_trace_rcu_reader(ctx, true); + if (flags & BPF_TRAMP_F_CALL_ORIG) { /* for the first pass, assume the worst case */ if (!ctx->image) @@ -2898,12 +2977,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, = struct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_CALL_ORIG) { /* the original func takes kernel addresses, never converted ones */ save_args(ctx, bargs_off, oargs_off, m, a, true, is_struct_ops, 0); + emit_trace_rcu_reader(ctx, false); /* call original func */ emit(A64_LDR64I(A64_R(10), A64_SP, retaddr_off), ctx); emit(A64_ADR(A64_LR, AARCH64_INSN_SIZE * 2), ctx); emit(A64_RET(A64_R(10)), ctx); /* store return value */ emit(A64_STR64I(A64_R(0), A64_SP, retval_off), ctx); + /* + * Second reader. Taken before ip_after_call so that the branch + * to the epilogue patched in at teardown is inside it too; the + * fmod_ret early exit lands past this still holding the first + * reader, so either way exactly one is held. + */ + emit_trace_rcu_reader(ctx, true); /* reserve a nop for bpf_tramp_image_put */ im->ip_after_call =3D ctx->ro_image + ctx->idx; emit(A64_NOP, ctx); @@ -2945,6 +3032,9 @@ static int prepare_trampoline(struct jit_ctx *ctx, st= ruct bpf_tramp_image *im, if (flags & BPF_TRAMP_F_RESTORE_REGS) restore_args(ctx, bargs_off, a->regs_for_args); =20 + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_trace_rcu_reader(ctx, false); + /* restore callee saved register x19 and x20 */ emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx); emit(A64_LDR64I(A64_R(20), A64_SP, regs_off + 8), ctx); --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743021; cv=none; d=zohomail.com; s=zohoarc; b=ZyMeOovwLzVk51QfPLObfuoUeyi7ETkn2KauuyBeEhzhNUxNx/ySkZwPZFiNXy+GXIcvdQv1KJmsnW9Jgvbj+G2PI89t8hHKT6+Pqc9ci0+h0NNxgDUxWyDPqw0cMAL4+5Sd2otMDW044f6NqDUjFOQtrnrLO09dIpe02zp6G4I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743021; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Nk/uI/WquaXkRgHHd/dzhScVzrCkrWpp6IlwsBXJcUg=; b=RVhxDYrNAHrxxl/hwF2T3pCKBfsPExKjhVwh4aSiyD6FZxk+t+xX8oQrNCkmGETwnUCcXkSTcUvN8Jz2tx/vzvsTRRGXWvlSWd1NJLO3waKboGv6Mcs8UFnunS51zxQlzPThDG5V5LABJrKHabKcRE/s4AKW/9Mj4Z1wJs3iNTg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1789743021500731.3267211529753; Fri, 18 Sep 2026 07:50:21 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425505.1649474 (Exim 4.92) (envelope-from ) id 1x7Zuh-0006wu-BU; Fri, 18 Sep 2026 14:50:07 +0000 Received: by outflank-mailman (output) from mailman id 1425505.1649474; Fri, 18 Sep 2026 14:50:07 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zuh-0006wB-5p; Fri, 18 Sep 2026 14:50:07 +0000 Received: by outflank-mailman (input) for mailman id 1425505; Fri, 18 Sep 2026 14:50:05 +0000 Received: from mx.expurgate.net ([195.190.135.10]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zuf-0006du-I5 for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:50:05 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7Zue-0054nV-V1 for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:50:04 +0200 Received: from [10.42.69.11] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f8b-8faa-0a2a0a5109dd-0a2a450bb8d4-34 for ; Fri, 18 Sep 2026 16:50:04 +0200 Received: from [74.125.230.204] (helo=mail-qk2-f12.google.com) by tlsNG-42698a.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4f9b-b7e8-0a2a450b0019-4a7de6ccadd4-3 for ; Fri, 18 Sep 2026 16:50:04 +0200 Received: by mail-qk2-f12.google.com with SMTP id af79cd13be357-939695b5741so77872185a.1 for ; Fri, 18 Sep 2026 07:50:04 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-93be0cd1807sm155785785a.4.2026.09.18.07.50.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:50:01 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789743003; x=1790347803; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Nk/uI/WquaXkRgHHd/dzhScVzrCkrWpp6IlwsBXJcUg=; b=caweufXTUdydEpY+1sjgq5PBziP0tWwC6FwroISpqzZOIFhDozqGYFyIDsE50GH64h U7HxwV6hVrk74OoMneABgCt8MDbeTerDH4fFZeZB+DlLWgLKruXiAhhq89XSVGhDubC6 6gM8hC3IxseJyiuyxQ+qd599l66zsLn8ABw25LNhWyB013BCuiQ/xFI7vdP6uOt/fuWM /olbXsu5LytxOjA68Ppo5rbHvLStBlvQQTsx2WF/SkCd21Zmj6OUo4feI6ZSk+ueUNzu 4ZEIbdiOop9Y0tnZDHEqREGMqTRzEfyb45c+wfwtFsQpjQynWkSg46kIb47HVOnazpds Qm0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789743003; x=1790347803; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Nk/uI/WquaXkRgHHd/dzhScVzrCkrWpp6IlwsBXJcUg=; b=flxY+hioavvEZOMlj8e62ljSNYfpwLtiQCy4Ls0IhHTs0xtHXPOg4g2Ka0sVLC/Wdo pX5cTpeSg8vyMW0fbbe86F5vL9q9baly8JXw+TaO+s95WktyliFQutnVM1R26p5Z6MVc NoT3wnMXA8/Pn3DfPi1wgffCRRKgtj5uvxlBAX8rztXCztm9KYjoVYD7VDOxoWOq12vo PToSET656M7X5x8BYnb+4twOP+bNwimX6d8eplXdRQB1ZfpWzzUqy6kQD1IHKGNAUrqF gttJYiBKU9vgDccX0i6EUp6gUDZ9dxZrUCvVQhf0LkZwQurF3Rn5vMG99XDOsPv/eKTI 5LKg== X-Forwarded-Encrypted: i=1; AKwUvBwPP/TIqmWK/s1t9bBl5ONdaJdnO+idElC+TijZ0mdA4vvPvseHobLdT0UDSIGU4xs16WpO+NaEZDg=@lists.xenproject.org X-Gm-Message-State: AFuF++nXu3Ghb10gzL5i0ibNvxjTPUWJhbON6QMtWHpr09lNbwA9XfYF gLDk085tdYx12VsfsvDuFrrB5c1ajwgZr0PdW62LHG1diDZT+xqMBkHT2DVNmOGE6To= X-Gm-Gg: AYBFou142nJ/WrmU2QYLhnODQZOpldZNAUHZXgjwUiXELqlCyJdASgSHU/kmuaQT5cX aHCtW77HNscJJ5CwVQuLgffQ0F+Mj4kuxpPu2bcHhTpbLcRo+ey2FIM0+2Tc2jirpBTJEZm8Opq W9HJ+QSoUCjiWSEAaGHhUxSMlYUIK8VUbIncqDZLuPb7OFyv4L+F0aeSueKiijyL9ur9OFlMEEj UU0Tce9yELoLTkgoWnHgN/7O4+zcW4N9Utv8Am7eTpFZJEsb/zJici6rAJxbSv2wHtMb1CW4gAv rSY4QPoUmSg8d7IUCPmyP01axW9rm2frYFHwTyvtKUchyPsdGN82V6umKqjiagjbyxY9maBF9PX tTFQdMq8c2dJiaeZ1loAtYmKrsAVcjuHf1irUVbJ+r0gSCD0Dwf/bMOK123Et4yVO9JHjl+cczi NEcNAAWhJaVxKHA6Pk55/6H6njuvCRsbzektOcVOb3+BQeJ5WNSyhlhx0UdsyaTo1ZrkKbww4o1 rZaQ6z/GhYpLQUcroXwHX+irhiRT7CJSFj77GFrAKrBhrAexXf3dxeh X-Received: by 2002:a05:620a:40d1:b0:939:6dfc:8abd with SMTP id af79cd13be357-93bdc7ab0bemr371381385a.52.1789743002924; Fri, 18 Sep 2026 07:50:02 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:29 +0000 Subject: [PATCH RFC v4 10/13] samples: ftrace: Make the direct-call trampolines Tasks Trace readers MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-10-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=13193; i=josef@toxicpanda.com; h=from:subject:message-id; bh=I7rYeQZ4spsxQVfaE2Sayb+dFXGxTZqIS2lZCUAJvtU=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QKXeVxQcH6ty+hUqDmOn27Ev2Cl/LtWOK4T4/1GJkZcNB9TzuskGzxBUIC7NaI2lvgHsCIRsimJ ERl1bJQ3J9QQ= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-42698a/1789743004-AA2C49EA-D3384BF3/0/0 X-purgate-type: clean X-purgate-size: 13195 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743022587158500 The sample direct trampolines are exactly the kind of out-of-line register_ftrace_direct() user whose lifetime depends on Tasks RCU waiting for a task inside them: nothing else stops rmmod while a task is preempted in my_direct_func(). On HAVE_RCU_TRAMPOLINE_READERS architectures that wait only covers Tasks Trace RCU readers, so give the samples a small shared header with rcu_read_lock_trace() and rcu_read_unlock_trace() open-coded as instruction strings for x86-64 and arm64 -- the same sequences as ftrace_64.S and entry-ftrace.S, using caller-saved non-argument scratch registers -- and bracket every call-out with them. The instructions outside the bracket are module text, covered by ftrace_direct_mark_module(). Other architectures get empty definitions. Assisted-by: LLM Signed-off-by: Josef Bacik --- samples/ftrace/ftrace-direct-modify.c | 9 ++ samples/ftrace/ftrace-direct-multi-modify.c | 9 ++ samples/ftrace/ftrace-direct-multi.c | 5 ++ samples/ftrace/ftrace-direct-too.c | 5 ++ samples/ftrace/ftrace-direct.c | 5 ++ samples/ftrace/ftrace-direct.h | 126 ++++++++++++++++++++++++= ++++ 6 files changed, 159 insertions(+) diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-= direct-modify.c index 164d9dd6fd92..937c8d8c2a1b 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -73,7 +74,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " call my_direct_func1\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -85,7 +88,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " call my_direct_func2\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -141,11 +146,13 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func1\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -153,11 +160,13 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func2\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/f= trace-direct-multi-modify.c index b03766c6217b..e12e5c8b83f0 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -77,10 +78,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n" " popq %rdi\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -92,10 +95,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n" " popq %rdi\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -154,6 +159,7 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -162,6 +168,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp1, .-my_tramp1\n" =20 @@ -169,6 +176,7 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -177,6 +185,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-d= irect-multi.c index 3fe6ddaf0b69..a970464ed378 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #include #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include @@ -56,10 +57,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n" " popq %rdi\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -101,6 +104,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -109,6 +113,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-dir= ect-too.c index bf2411aa6fd7..abc098c2ab7a 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -3,6 +3,7 @@ =20 #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -61,6 +62,7 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n" @@ -70,6 +72,7 @@ asm ( " popq %rdx\n" " popq %rsi\n" " popq %rdi\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -110,6 +113,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #48\n" " stp x9, x30, [sp]\n" " stp x0, x1, [sp, #16]\n" @@ -119,6 +123,7 @@ asm ( " ldp x0, x1, [sp, #16]\n" " ldp x2, x3, [sp, #32]\n" " add sp, sp, #48\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 5368c8c39cbb..99b65ad2fccc 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -3,6 +3,7 @@ =20 #include /* for wake_up_process() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -54,9 +55,11 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + TRACE_RCU_READ_LOCK " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n" + TRACE_RCU_READ_UNLOCK " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -97,6 +100,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + TRACE_RCU_READ_LOCK " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -104,6 +108,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + TRACE_RCU_READ_UNLOCK " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h new file mode 100644 index 000000000000..726f67048ff5 --- /dev/null +++ b/samples/ftrace/ftrace-direct.h @@ -0,0 +1,126 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _SAMPLES_FTRACE_DIRECT_H +#define _SAMPLES_FTRACE_DIRECT_H + +#include + +/* + * A direct-call trampoline is entered with no lock, refcount or RCU marker + * held; only Tasks RCU keeps it (and, for a module, its text) alive while= a + * task is inside it or preempted in something it called. On architectures + * that select HAVE_RCU_TRAMPOLINE_READERS, Tasks RCU only waits for such a + * task while it is a Tasks Trace RCU reader, so the trampoline must enter= one + * before calling out and leave it afterwards, exactly like the ftrace and= BPF + * trampolines do. See register_ftrace_direct(). The instructions before= the + * lock and after the unlock are covered by ftrace_direct_mark_module(). + * + * These are rcu_read_lock_trace() / rcu_read_unlock_trace() open-coded as + * instruction strings for use inside the samples' asm() trampolines, after + * the versions in arch/x86/kernel/ftrace_64.S and + * arch/arm64/kernel/entry-ftrace.S. The scratch registers are caller-sav= ed + * and not argument registers, so they are dead on entry to and exit from = an + * fentry trampoline; the flags are clobbered. + * + * The generated asm-offsets.h is only pulled in on the architectures that= need + * it here: it is not generally safe to include from C (e.g. PPC32's TASK_= SIZE + * and arm64's TRAMP_VALIAS clash with the C definitions). + */ +#if defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_X86_64) + +#include + +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB +#define TRACE_RCU_MB " lock addl $0, -4(%rsp)\n" +#else +#define TRACE_RCU_MB +#endif + +#define TRACE_RCU_READ_LOCK \ + " movq %gs:current_task(%rip), %r11\n" \ + " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \ + " incl " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \ + " testl %r10d, %r10d\n" \ + " jnz 771f\n" \ + " movq rcu_tasks_trace_srcu_struct+" __stringify(SRCU_srcu_ctrp) "(%rip),= %r10\n" \ + " incq %gs:" __stringify(SRCU_CTR_srcu_locks) "(%r10)\n" \ + " movq %r10, " __stringify(TASK_trc_reader_scp) "(%r11)\n" \ + TRACE_RCU_MB \ + "771:\n" + +#define TRACE_RCU_READ_UNLOCK \ + " movq %gs:current_task(%rip), %r11\n" \ + " movl " __stringify(TASK_trc_reader_nesting) "(%r11), %r10d\n" \ + " subl $1, %r10d\n" \ + " jnz 772f\n" \ + " movq " __stringify(TASK_trc_reader_scp) "(%r11), %r10\n" \ + " movl $0, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \ + TRACE_RCU_MB \ + " incq %gs:" __stringify(SRCU_CTR_srcu_unlocks) "(%r10)\n" \ + " jmp 773f\n" \ + "772: movl %r10d, " __stringify(TASK_trc_reader_nesting) "(%r11)\n" \ + "773:\n" + +#elif defined(CONFIG_TASKS_RCU_TRAMPOLINE_READERS) && defined(CONFIG_ARM64) + +#include +#include +/* arm64's asm-offsets.h redefines TRAMP_VALIAS from . */ +#pragma push_macro("TRAMP_VALIAS") +#undef TRAMP_VALIAS +#include +#pragma pop_macro("TRAMP_VALIAS") + +#ifndef CONFIG_TASKS_TRACE_RCU_NO_MB +#define TRACE_RCU_MB " dmb ish\n" +#else +#define TRACE_RCU_MB +#endif + +#define TRACE_RCU_SRCU_CTRP "rcu_tasks_trace_srcu_struct+" __stringify(SRC= U_SRCU_CTRP) + +/* x14 =3D this CPU's offset; then atomically increment the long at x14 + = \areg */ +#define TRACE_RCU_PERCPU_INC(areg) \ + ALTERNATIVE(" mrs x14, tpidr_el1\n", " mrs x14, tpidr_el2\n", \ + ARM64_HAS_VIRT_HOST_EXTN) \ + " add x14, x14, " areg "\n" \ + "778: ldxr x15, [x14]\n" \ + " add x15, x15, #1\n" \ + " stxr w16, x15, [x14]\n" \ + " cbnz w16, 778b\n" + +#define TRACE_RCU_READ_LOCK \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \ + " add w14, w13, #1\n" \ + " str w14, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \ + " cbnz w13, 771f\n" \ + " adrp x13, " TRACE_RCU_SRCU_CTRP "\n" \ + " ldr x13, [x13, #:lo12:" TRACE_RCU_SRCU_CTRP "]\n" \ + " str x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \ + " add x13, x13, #" __stringify(SRCU_CTR_SRCU_LOCKS) "\n" \ + TRACE_RCU_PERCPU_INC("x13") \ + TRACE_RCU_MB \ + "771:\n" + +#define TRACE_RCU_READ_UNLOCK \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \ + " subs w13, w13, #1\n" \ + " b.ne 772f\n" \ + " ldr x13, [x12, #" __stringify(TSK_TRC_READER_SCP) "]\n" \ + " str wzr, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \ + TRACE_RCU_MB \ + " add x13, x13, #" __stringify(SRCU_CTR_SRCU_UNLOCKS) "\n" \ + TRACE_RCU_PERCPU_INC("x13") \ + " b 773f\n" \ + "772: str w13, [x12, #" __stringify(TSK_TRC_READER_NESTING) "]\n" \ + "773:\n" + +#else + +#define TRACE_RCU_READ_LOCK +#define TRACE_RCU_READ_UNLOCK + +#endif + +#endif /* _SAMPLES_FTRACE_DIRECT_H */ --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Received: from mail-qv2-f21.google.com (mail-qv2-f21.google.com [74.125.230.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A02784F96BD for ; Fri, 18 Sep 2026 14:50:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.149 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743012; cv=none; b=QIwRggA5Qu90K1ej0fJGSk6HlSqRAohddNwQlgL8aefmTZq0t6Oxty18+TNLvaGAFM+KS8FlQZrbtqwUvKDrX8ZsCkMz7BkdOsM39kP3t98uKws+rjkUeqfLUZP8ioyH2VJ1M4b+yQrEokUJ265YBaTFnhBGVhq8Ff3TCVUoQmI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743012; c=relaxed/simple; bh=O8YdW4YSHUYyF3GwEQN16WzJM7UkZZFfh8RzVwv7xJM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=QLmtk28SvSz3Pc+zZfQwHMu+F8b2stkPkQQVlQmcpdSDkOhYF0oI33HuR8CD4WTdHVP/wgSpv7XvF6QY6m9mB4BL5UeivImTrziJeTVzz+v3KJ3e4/7ULC1usRL9P6H4wLB9siKbbxWnrT22Cujl0Of7Yz0jLsCT2Ys3MCGFrX8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=FMenZX4f; arc=none smtp.client-ip=74.125.230.149 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="FMenZX4f" Received: by mail-qv2-f21.google.com with SMTP id 6a1803df08f44-90cdfbd148aso7068906d6.2 for ; Fri, 18 Sep 2026 07:50:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789743005; x=1790347805; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=MkP56mJgKixcgLXoBxrfzz0NyJo14L+hjKFrNfKE/2k=; b=FMenZX4fA7IYYaGVRJ74JORgL+Hbw07S5YoZW625s6FlgiimPPEplqFaCu1De7ETu4 I56W5/QhtPJLAVld/nTmIBJ790mBWOk4n7LzyWhs6fxs4yoGOGz8+bA413+8tWlDifNi eGaqh5ZQLGJv6ATyMyx69py90tY3ihIdArxWnxgWnx20AoCpfJOmLXHZ6jPiqjewxtkl 4hHaFlHfDqpjuhO/JTWOc9DzQIA5MDDBdiuvZ4pA2o70a/HyAJCwStZ7YEx1/81uitv+ nayrgUTXY2IEJbJmPVoTWY8y9BjR4RcTR4F4zbD1VBRDOG+k+5ZFEiAh9VvJzwwtR+xc 41bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789743005; x=1790347805; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=MkP56mJgKixcgLXoBxrfzz0NyJo14L+hjKFrNfKE/2k=; b=GlWi1pL/u7rae0tA2aZ5U68o9EYuVAka/NOyB+Bava/nKLDugWdBpcVIZmLs/OVDgP RlpzJ9rrBSWXtfEE2Xj8LyleFwXN4la6XQ5QHrwt+RhqAwYrnMTHcyolTjzgwN1gXg16 vJ/3GHLHPLD/Sd6s2i5DI8+Eo3SSQrG2/+tvz9DVZqc8ma+633FTqe8Q0xDLXuhlyAJs ZeIwA1MoRFyb9v67TSnOi8j1WI5o9x0lVdQ+07H8rIMr0SdxTBF/0yc/U4PQMKSxx51I 14PYSbPMJNEsCPmO7/r/+4jgcLsPFfnJGoLaJT3Qt4E8WzEoD7YtQpO90JSEIP/iMNbF UKBQ== X-Forwarded-Encrypted: i=1; AKwUvByuuYcXgRPEHoS5bt9grJdA+qpJ1LPAHuDdQoAuN7uVXA68GUIcQcGig/FNxAUuFB1z80kmx8eYh/NgEqg=@vger.kernel.org X-Gm-Message-State: AFuF++lBU2f4IsdanzkF0GDLV2/FAduUGljxHIAfSrI/RoK1Hycnrlfg WhKs7WGgq8ZWBMjnlWFL3xuDNdTLeFGasmF2miDPYcC5vrADLcfVOk/ls3KdcaZ4gx0= X-Gm-Gg: AYBFou3f4/FHK/S8G+1cxlDlRDCwv76dDSY/2JaaNon5Cyqn8kDjeSv20v2T2mCKGK+ FYEus/sb2qHkhUe0vfk6IkVu8K8SbvOlx3eXPNlkLIU5QtkUYANMUoHPgcOw6MR2TxaWSPeH3lM VffuiHhdFZUiAB4ptrc7s+sYGJ5KT5VhTO71xLhpx4WptKcvle/f6fmvbyQJrEBdyo93i53fbTs PveuZov3aB6atzhjXeBC0Sd5qOjGRzdO/g7u4znr756CM2Eh0a/bmNb81ZqjdnK7fWSkL/96SPO aNeuCTuX9wYzh7p2dlFqiYrUi7z4QshUHrMP6Ddfo6k28Q7rS8RTV1zqK4wewB94Rza9QmzuAWY w/5SSG7ftB3eunbUtoFF4NQQVyDJcdoPmnnrZmUtxaB50heCfQPn4sECo8dkp2UO+umFXpqYuey S/1dEky3qVHkIo1Shgwwjv/xsYJFpMN2+HXLvpduFvf3jBViLFqtP4LRCxHy9qLQM9JISgPL4rU K6hEqJ8hQ86AmV5rY7W+ycbSRtrSsiZtcSuvO7yWT65NUODW5rQNryx X-Received: by 2002:a05:6214:1307:b0:910:4a86:5969 with SMTP id 6a1803df08f44-91254bd5293mr46077526d6.23.1789743004909; Fri, 18 Sep 2026 07:50:04 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-91258056195sm14090716d6.29.2026.09.18.07.50.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:50:04 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:30 +0000 Subject: [PATCH RFC v4 11/13] rcutorture: Make Tasks RCU readers Tasks Trace readers where required Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-11-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=1716; i=josef@toxicpanda.com; h=from:subject:message-id; bh=O8YdW4YSHUYyF3GwEQN16WzJM7UkZZFfh8RzVwv7xJM=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QCK2SKWNSvb8uUIHABRfNQrEJPCWCZeIZnAqX/3f5X/BpUmomCy4y1MVWrdYzX/ihyvEJhShdIA kincQ831+1ws= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA rcutorture's "tasks" flavor has empty readlock/readunlock hooks because a classic Tasks RCU reader is simply code that does not block. Under CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preemption outside trampoline text is also a quiescent state, and the thing real readers (trampolines) do to stay protected across their call-outs is take rcu_read_lock_trace(), so have the torture readers do the same there. Otherwise a preempted torture reader would rightly be treated as quiescent and the test would report false-positive too-short grace periods. Assisted-by: LLM Signed-off-by: Josef Bacik --- kernel/rcu/rcutorture.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/rcu/rcutorture.c b/kernel/rcu/rcutorture.c index 794937e13e7c..ab870ef09af0 100644 --- a/kernel/rcu/rcutorture.c +++ b/kernel/rcu/rcutorture.c @@ -1142,13 +1142,23 @@ static struct rcu_torture_ops trivial_preempt_ops = =3D { * Definitions for RCU-tasks torture testing. */ =20 +/* + * A classic Tasks RCU reader is any stretch of kernel code that does not + * voluntarily block. With CONFIG_TASKS_RCU_TRAMPOLINE_READERS a preempti= on + * outside trampoline text also ends it, and what a trampoline does to stay + * protected across its call-out is take a Tasks Trace reader, so model th= at. + */ static int tasks_torture_read_lock(void) { + if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)) + rcu_read_lock_trace(); return 0; } =20 static void tasks_torture_read_unlock(int idx) { + if (IS_ENABLED(CONFIG_TASKS_RCU_TRAMPOLINE_READERS)) + rcu_read_unlock_trace(); } =20 static void rcu_tasks_torture_deferred_free(struct rcu_torture *p) --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Received: from mail-qv2-f12.google.com (mail-qv2-f12.google.com [74.125.230.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2741C4F4036 for ; Fri, 18 Sep 2026 14:50:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743014; cv=none; b=dyXwwOQgnZKh1FrCL85DWN8onWjsI0e5bYSsteAVtWlxsXYPiGD4Przu6Tejw/FBU7OKNZrOg/g06PnZTIoQLkHPpvwMPcial8wF45F22zIjjccSdTJbRefCD+SWQu0PDcraalGMztnSMbp+jJbdTKXNDP4WOAG28QpJVHLNM6g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789743014; c=relaxed/simple; bh=FRa7j81kn+FZifN+tU/RXt2NlHMQaOXWqjJvWoyULIE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WyZrp8+LqXBqyPNHf3sO3yFtGhqWnIoJNNfgYgmErF1GbKC93LKVtszp8WWWr6UWuT7K9USOtLhNXSt4hDwgSE4i1dHBReftR0k41zAuMZVAZ7mpQGoV1f6M2gFK3EFVIFXmT8gk2cFrAaoO6iaGLVznLGQgDQdVqaID4E6oLJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=eaAruMPi; arc=none smtp.client-ip=74.125.230.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="eaAruMPi" Received: by mail-qv2-f12.google.com with SMTP id 6a1803df08f44-90ce6d5271dso4554336d6.1 for ; Fri, 18 Sep 2026 07:50:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789743009; x=1790347809; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=z+PbvTl7j6crAVqwwSi1N6wOs6bBuNm88XPaAXlYwG4=; b=eaAruMPiqNCETlndUIMEsTI8wwe4IyMYRp7IY+Jq1Pz5C6glptp6M48r3gAtXl3R3o o0mk+4/fYf/RyVbjMLlqirV3r0k7aLq8D3SJBS9TnmTyroJxI1lWFXcCmVmMQh49977z 6BQCEXpuZs7auPVq7cO2mxTj47CQGQVRNaycwaX/MxEuIHuS5LluhN/YiJkMHk4Mwv2V o7U9rfACfUocBZcnkxbCCUpznnuADY5zCOTAOYF19t0uxRlYXEoRTgNXQZgU5edp5tMF Ls0N2NnNmCjfe2UterERkyBV/iPZwAwtdi7ArbkvvGykip2Bk11CMa7dpDOgmU7PrcuI bjfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789743009; x=1790347809; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=z+PbvTl7j6crAVqwwSi1N6wOs6bBuNm88XPaAXlYwG4=; b=Uxdf5R9fhG/3Mkyl8bdWzpoh3x4ckjQO/nvfqT1QPmBM95LFj2Euv9YrsOjuiAf5yn DahWLE8n/d9+7hgpfOb0/hmQv6zz+kM2Ss/Ke2jPP58xRGVEqTinYjzw8wHMyxq9+eYc QaX949A4/ooQU4ooJe7KsGvVIoO1ioLrVx++rOzYW5wQ5nfxnXlwNY3U3vvdYwEPsuKh JtxMXrPc4IecM3G7QovR2MsludJciOfoU2MVyi60zP2G/jMMBT8LBu3Njbri4HkWvgT8 5xd2Rcxfde8ONsHMKOrPxokrpIOD2R+s5n4+mfyrKdxXx12BH+5c6gz1ITq4Y01KcxlN d/YQ== X-Forwarded-Encrypted: i=1; AKwUvBy14DQfGR58MmHJ0j5abSfbQdCE4QC6neNWe/GxzVcU8m/r21/bHmKQbOwDLAQxM7J00PUXTJ/1EKY7Dmc=@vger.kernel.org X-Gm-Message-State: AFuF++kDLnkZV0Bn/YifH0Kud3Pcf+LagG7Lkz+S87stmvHcyvCmMYeh IGm21zY1Gyux8CGp0y6gbnWLBgHlpuNDlxHMVo0lyL/HtMa3duYmPFZxiNMN3fIIJ/I= X-Gm-Gg: AYBFou3F6fhpuUPSDuEVXbz6NqbwaL0UFd8H+LlwqkmYKkZ4s/8gOTOMq5tPjMwZ1CY mRCPjxYK0n+RPn8BOEPGFHaTAJ+t5iet9Tm+9RT2CW9IURl64inFeLRYtj6rVeAM9iGNMeCQRNJ gdTXFZPR0sdphFpHmY2Z0dXHlenRsjQgJvAYFKN8oTkaJgk09AcQESxjNThW96vt9VvwajSObsk mgjVL5E2otzrQ1czhz6Hr1080ggXgBKN9HRphDlKgCNWgW6JXvhUtek3z7On9e/pmDbpPMRYwWF dN+JN+Lib5eQFVlxk73tVWUiBY6Yu2i77h+3P0TxdardhmEGyEubjXdaihiqdpLWlBSsBEzZvzt 27NHiCa6vtyWPC2I2KjhrQ7j8x0woB6NLEu/fXbqMP6NShUd8v8WbS8AZaWMsKgIRTmMdgW/Ftg FOOJIebckdOo0TL2BD6M3Usa3/2DcLXMOL0F/GmaDXkichiw+9kAYhkgVnMeWo8+kdYC0zM13Uk 2YQG9KGfm18rxrv5XKSW1q2C2RfX2RbM+TLDGRI+6vrG7Qy0gj7twcWDQ== X-Received: by 2002:a05:6214:ca6:b0:912:3d05:eb11 with SMTP id 6a1803df08f44-91254e4feadmr40041586d6.5.1789743008667; Fri, 18 Sep 2026 07:50:08 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9125802d79bsm13900886d6.24.2026.09.18.07.50.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:50:06 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:31 +0000 Subject: [PATCH RFC v4 12/13] rcu-tasks-trace: Assert no reader is held on return to userspace Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-12-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=2581; i=josef@toxicpanda.com; h=from:subject:message-id; bh=FRa7j81kn+FZifN+tU/RXt2NlHMQaOXWqjJvWoyULIE=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QAsi6H715Q7GMOS9YXNtKesBwx2lesjeCIIWVvPLg8W7bMTSPfSKqF2zG8KnHJqpv18BR10X5Rc 7g2eq40cmPAk= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA With trampolines now taking rcu_read_lock_trace() open-coded from assembly and JIT-emitted code, an unbalanced reader would silently turn every later Tasks Trace grace period on that task into a stall. No task can legitimately reach userspace with current->trc_reader_nesting non-zero, so under CONFIG_PROVE_RCU check it in the generic entry code's return-to-user validation, next to the existing kmap and lockdep assertions. Compiles away otherwise. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/irq-entry-common.h | 2 ++ include/linux/rcupdate_trace.h | 7 +++++++ 2 files changed, 9 insertions(+) diff --git a/include/linux/irq-entry-common.h b/include/linux/irq-entry-com= mon.h index e19b41ee6b18..d42568a488a7 100644 --- a/include/linux/irq-entry-common.h +++ b/include/linux/irq-entry-common.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include #include @@ -214,6 +215,7 @@ static __always_inline void __exit_to_user_mode_validat= e(void) { /* Ensure that kernel state is sane for a return to userspace */ kmap_assert_nomap(); + rcu_tasks_trace_assert_idle(); lockdep_assert_irqs_disabled(); lockdep_sys_exit(); } diff --git a/include/linux/rcupdate_trace.h b/include/linux/rcupdate_trace.h index 273c59a03251..6034c6b19479 100644 --- a/include/linux/rcupdate_trace.h +++ b/include/linux/rcupdate_trace.h @@ -211,6 +211,12 @@ unsigned long rcu_tasks_trace_batches_completed(void); // Placeholders to enable stepwise transition. void __init rcu_tasks_trace_suppress_unused(void); =20 +/* A task must never reach userspace inside an rcu_read_lock_trace() reade= r. */ +static inline void rcu_tasks_trace_assert_idle(void) +{ + WARN_ON_ONCE(IS_ENABLED(CONFIG_PROVE_RCU) && READ_ONCE(current->trc_reade= r_nesting)); +} + #else static inline unsigned long rcu_tasks_trace_batches_completed(void) { retu= rn 0; } /* @@ -220,6 +226,7 @@ static inline unsigned long rcu_tasks_trace_batches_com= pleted(void) { return 0; static inline void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback= _t func) { BUG(); } static inline void rcu_read_lock_trace(void) { BUG(); } static inline void rcu_read_unlock_trace(void) { BUG(); } +static inline void rcu_tasks_trace_assert_idle(void) { } #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ =20 DEFINE_LOCK_GUARD_0(rcu_tasks_trace, --=20 2.55.0 From nobody Thu Sep 24 23:33:31 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1789743029; cv=none; d=zohomail.com; s=zohoarc; b=Dz/koZnrBlHm4p3mrI4Nq2ys4ZZpEOxLpTO+tzuB9d1f4x7A+v4tbZmJ5Q9L3ITc/bvALettzik08eqpobLRr7M1BfNs72O09UbNDSCGzUKVUWEPANvxitQ81XXXY5FvtfYWuUU+NoCepvJfax4IH+Wfwc6AxGlvHVJZ5C10mRw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1789743029; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=A8c5rUXaTXu9B0UqDL2lNbjmJqabE8+fQOLsAHyr+Co=; b=fhALIhH3B269skl2YzJ5eM9oSdNbf2u8reax9GOXIRyjGetjpoqYZjHlGpY/+XZEGAQBm73SNBm/Py84adYyQlNNNPS3p9EP1gZNzZm1MEsbWbjKSH3YtRPoNJR/zMCuqndgD6CUsV5fCj7jh1xlATd6xJk/SB0QeBzudxkfveQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 178974302975181.38278587897446; Fri, 18 Sep 2026 07:50:29 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1425520.1649510 (Exim 4.92) (envelope-from ) id 1x7Zup-0000Fu-TQ; Fri, 18 Sep 2026 14:50:15 +0000 Received: by outflank-mailman (output) from mailman id 1425520.1649510; Fri, 18 Sep 2026 14:50:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zup-0000FE-Ol; Fri, 18 Sep 2026 14:50:15 +0000 Received: by outflank-mailman (input) for mailman id 1425520; Fri, 18 Sep 2026 14:50:14 +0000 Received: from mx.expurgate.net ([194.145.224.20]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1x7Zuo-0008PH-2F for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 14:50:14 +0000 Received: from mx.expurgate.net (helo=localhost) by mx.expurgate.net with esmtp id 1x7Zun-00H86X-Ev for xen-devel@lists.xenproject.org; Fri, 18 Sep 2026 16:50:13 +0200 Received: from [10.42.69.4] (helo=localhost) by localhost with ESMTP (eXpurgate MTA 0.9.1) (envelope-from ) id 6aad4f8a-bab6-0a2a0a5309dd-0a2a4504a334-32 for ; Fri, 18 Sep 2026 16:50:13 +0200 Received: from [74.125.230.204] (helo=mail-qk2-f12.google.com) by tlsNG-ebf023.mxtls.expurgate.net with ESMTPS (eXpurgate 4.57.1) (envelope-from ) id 6aad4fa3-b57f-0a2a45040019-4a7de6ccf00b-3 for ; Fri, 18 Sep 2026 16:50:12 +0200 Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-52fb76ef19aso7956491cf.2 for ; Fri, 18 Sep 2026 07:50:12 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532aac41311sm5612191cf.10.2026.09.18.07.50.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2026 07:50:09 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" Authentication-Results: eu.smtp.expurgate.cloud; dkim=pass header.s=google header.d=toxicpanda.com header.i="@toxicpanda.com" header.h="Cc:To:In-Reply-To:References:Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789743011; x=1790347811; darn=lists.xenproject.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=A8c5rUXaTXu9B0UqDL2lNbjmJqabE8+fQOLsAHyr+Co=; b=RrtbBLuO81ggjT/jg5rcGpFCBb1wap4lt4eXzYssx9gOx7qVYHx84f8IvvVH1C2n3v 0J1H4hsfijtC7dNtXattJ5A6krhl8fKegVIFA23L+QL5B8Hfj62ICcKGUgWgLFARS06d C7HqyvzRottBa7PMJew8i6x2j9ihU43a4Py1ku8p9X3+4U3HNyGtpQZ9VC91ACBIYRCR oRbSVXFJk0dLNXKVuqsYDwsQBJrUQ+X7o93FL554ayaoQsjZg1Tw6AYnIvs4AjmhPOYJ fQGySwmYTXTcfW/h485ZLE8ehGAG25PCpzQX79lDWKeBoA+VJ3tpKUre/YUSV0udvfGd 06nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789743011; x=1790347811; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=A8c5rUXaTXu9B0UqDL2lNbjmJqabE8+fQOLsAHyr+Co=; b=nhMGYRJyGWTGsimErcPGAK1ElO2mPEv2z58HBGb7Bar8rNpoVK1ug8+NV1wf/idwAi 5vUEe7f91oACy82rigJL4MopcTuaOZaLDX2LyApa05i9mgUqopbcV115/rnmRP1wd27c iT4yYSYa06k1wvSW2i9BoPKQbi5AIYq9RIRDKjwzCO7zfy3+J477Mazcg/rMw3dDe4HS kU8oA7Fx9ULoJQvaC25e5PlMW/St4SBFaX/0PhOQdIMQKpah0rBuELNK9nspFnMzZM/G k5NFxFyc9W8weylywa8mGJBxkoBqCUJwl7TOec/eA4jSPNj6x+wFP9sJjk9QKOtenooY gd/Q== X-Forwarded-Encrypted: i=1; AKwUvBzLJ0ClcwmEhbaH/EEGEulrLJynphdvS59tKbVavZj0Dyo/v+VjAw9rWiKxgYGUG6IEvzrfWejp1SE=@lists.xenproject.org X-Gm-Message-State: AFuF++nh32iyUUkfy2T0EcZzb1/plmokomv3Llw7w0ALPeYt8zpGmx7u jRmm2ImG/K+hkkYbfpAj5HPDY6eooB++cQX3RTTDxPOSnS9IUbugDFpntlxmpVYtUg0= X-Gm-Gg: AYBFou3opspiqGHuinxLW9/VWtOwi6iQEiJx9RleWrXhQKilwFDTOzcZznzEVneTckt hBHQ5XoceIjp3r4wOi2InsORlVI+zSpb27iwSgeT5f5xZRU0PD57QDj3K2ssD07AwK6d3obIRcZ CGBRoe0YLb8DkbY2BD3ze/RsGO+fELyi9I5pYl9z/BRW/bq0/E+fqdR8dLF9bt7EV8OfhhEiGGL VELt8KKGbJnoB6Bzo3LcmpincNICe/SMnADhOrHj/WaY/1YPHnG0NYCWlwc3NKAXjRO4BpkV9nX l7U/4odmM2/v0ixAlGjkdFwvA8/s2ZhNQZt2OeKJyhCox2H3PbrCw3Moew2P3QQwN+pEnP8KJ2a 24LdsAwVTG/wCl/+DDyG3X06uFqyFAZavQUFIinW4/T+mqTy5pADH/dq7oY5Ns0VFeZ35/Kti2T /FHbYk68TMBT7MAf/IhBGSIgFAwx6mw0rn9XRRJQqJgUZcZ0EWEa1dP6A0EcoU421+WznYCx1KK X3jFKV8zieDgd9jXi2mK/PcS1nX8CovegCmPFUVCnuMvpGr/w5CabcM X-Received: by 2002:a05:622a:2606:b0:532:9add:50bb with SMTP id d75a77b69052e-5329e402a69mr42121701cf.75.1789743010763; Fri, 18 Sep 2026 07:50:10 -0700 (PDT) From: Josef Bacik Date: Fri, 18 Sep 2026 14:49:32 +0000 Subject: [PATCH RFC v4 13/13] x86, arm64: Build Tasks RCU on Tasks Trace readers in trampolines MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260918-b4-rcu-tasks-preempt-qs-v4-13-63f0e9d69661@toxicpanda.com> References: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> In-Reply-To: <20260918-b4-rcu-tasks-preempt-qs-v4-0-63f0e9d69661@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai , "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789742980; l=6284; i=josef@toxicpanda.com; h=from:subject:message-id; bh=VsJyzZYJ6ksShlnC7Yiz29R4LafGY+0/mhkU2yBL4O8=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QEwAhBVideBDLteTwvOfpHAYkoAWUm3JvOEwqQNA35GYhU6H/frrHaAxlVzl5POPAnXUpm7Q7V1 WCFlv6PR8OAI= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-purgate-ID: tlsNG-ebf023/1789743012-502E4B50-B2ABC6E4/0/0 X-purgate-type: clean X-purgate-size: 6286 X-ZohoMail-DKIM: pass (identity @toxicpanda.com) X-ZM-MESSAGEID: 1789743030512158500 With the preceding patches every trampoline whose lifetime Tasks RCU guards on x86-64 and arm64 -- ftrace_caller and its copies, the optprobe template, BPF trampolines, and the sample direct-call trampolines -- is a Tasks Trace RCU reader around its call-out, and the text outside that reader is known to rcu_tasks_trampoline_text(). Select HAVE_RCU_TRAMPOLINE_READERS on both (x86-64 with SMP for Tree SRCU and DYNAMIC_FTRACE, which is where its ftrace_caller changes and arch_rcu_tasks_trampoline_text() live; arm64 with DYNAMIC_FTRACE_WITH_ARGS likewise), which switches CONFIG_TASKS_RCU to the implementation added earlier in the series: a Tasks RCU grace period becomes a per-CPU pass over context switches and irq-exit reschedules outside trampoline text plus a Tasks Trace grace period, bounded by a few jiffies and preempt-off latency instead of by the longest stretch any task runs without sleeping. Other architectures keep the classic implementation. Update Documentation/RCU and the FORCE_TASKS_RCU help text to describe the variant and the obligation it places on trampolines. Assisted-by: LLM Signed-off-by: Josef Bacik --- .../RCU/Design/Requirements/Requirements.rst | 24 ++++++++++++++++++= ++++ Documentation/RCU/checklist.rst | 7 ++++++- arch/arm64/Kconfig | 1 + arch/x86/Kconfig | 1 + kernel/rcu/Kconfig | 6 ++++-- 5 files changed, 36 insertions(+), 3 deletions(-) diff --git a/Documentation/RCU/Design/Requirements/Requirements.rst b/Docum= entation/RCU/Design/Requirements/Requirements.rst index 8101fe6229d5..c059c463e083 100644 --- a/Documentation/RCU/Design/Requirements/Requirements.rst +++ b/Documentation/RCU/Design/Requirements/Requirements.rst @@ -2756,6 +2756,30 @@ synchronize_rcu(), and rcu_barrier(), respectively. = In three APIs are therefore implemented by separate functions that check for voluntary context switches. =20 +Architectures that select ``CONFIG_HAVE_RCU_TRAMPOLINE_READERS`` keep the +same three APIs but implement the grace period differently +(``CONFIG_TASKS_RCU_TRAMPOLINE_READERS``). There, every trampoline whose +lifetime Tasks RCU guards enters a Tasks Trace RCU read-side critical +section (rcu_read_lock_trace() or its assembly equivalent) before calling +out and leaves it before returning, so a task anywhere inside such a +call-out is an ordinary Tasks Trace reader whether or not it is +preempted. The few trampoline instructions outside that reader can only +be occupied by a task that was interrupted there, so the grace period +additionally waits for each CPU to pass through a context switch, and the +irq-exit preemption path, the only switch that can catch a task inside +such text (rcu_tasks_trampoline_text()), briefly makes such a task a +holdout until it is next seen elsewhere. On such kernels an involuntary +context switch outside trampoline text *is* a Tasks-RCU quiescent state, +a Tasks RCU grace period no longer depends on how long any task runs +without sleeping, cond_resched_tasks_rcu_qs() is unnecessary, and the +obligation moves to the trampolines: anything that relies on +synchronize_rcu_tasks() to protect code a task may be preempted in must +take the Tasks Trace reader (see register_ftrace_direct()), or, where +that is impossible because the code is ordinary text with no trampoline +of its own, make it known to rcu_tasks_trampoline_text() and wait out +tasks already parked there with rcu_tasks_wait_irq_preempted(), as the +kprobe jump optimizer does. + Tasks Rude RCU ~~~~~~~~~~~~~~ =20 diff --git a/Documentation/RCU/checklist.rst b/Documentation/RCU/checklist.= rst index 4b30f701225f..87dd506bffef 100644 --- a/Documentation/RCU/checklist.rst +++ b/Documentation/RCU/checklist.rst @@ -252,7 +252,12 @@ over a rather long period of time, but improvements ar= e always welcome! a. If the updater uses synchronize_rcu_tasks() or call_rcu_tasks(), then the readers must refrain from executing voluntary context switches, that is, from - blocking. + blocking. On architectures that select + CONFIG_HAVE_RCU_TRAMPOLINE_READERS a reader must in + addition be a Tasks Trace RCU reader (that is what the + trampolines there do around their call-outs) or be text + that rcu_tasks_trampoline_text() recognises; an arbitrary + stretch of preemptible kernel code is not protected. =20 b. If the updater uses call_rcu_tasks_trace() or synchronize_rcu_tasks_trace(), then the diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b5a51b0ef944..bf0e56006863 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -218,6 +218,7 @@ config ARM64 select HAVE_PERF_REGS select HAVE_PERF_USER_STACK_DUMP select HAVE_PREEMPT_DYNAMIC_KEY + select HAVE_RCU_TRAMPOLINE_READERS if DYNAMIC_FTRACE_WITH_ARGS select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE select HAVE_POSIX_CPU_TIMERS_TASK_WORK diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 15fd9ec5ecac..64c3814eb745 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -288,6 +288,7 @@ config X86 select MMU_GATHER_RCU_TABLE_FREE select MMU_GATHER_MERGE_VMAS select HAVE_POSIX_CPU_TIMERS_TASK_WORK + select HAVE_RCU_TRAMPOLINE_READERS if X86_64 && SMP && DYNAMIC_FTRACE select HAVE_REGS_AND_STACK_ACCESS_API select HAVE_RELIABLE_STACKTRACE if UNWINDER_ORC || STACK_VALIDATION select HAVE_FUNCTION_ARG_ACCESS_API diff --git a/kernel/rcu/Kconfig b/kernel/rcu/Kconfig index bbab14bc14c3..341b68b972ef 100644 --- a/kernel/rcu/Kconfig +++ b/kernel/rcu/Kconfig @@ -95,8 +95,10 @@ config FORCE_TASKS_RCU help This option force-enables a task-based RCU implementation that uses only voluntary context switch (not preemption!), - idle, and user-mode execution as quiescent states. Not for - manual selection in most cases. + idle, and user-mode execution as quiescent states, or, on + HAVE_RCU_TRAMPOLINE_READERS architectures, the variant built on + Tasks Trace RCU readers in trampolines. Not for manual + selection in most cases. =20 config NEED_TASKS_RCU bool --=20 2.55.0