From nobody Mon Jun 8 18:55:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E08BF3F789C; Wed, 27 May 2026 12:12:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883962; cv=none; b=P9s6bzFGJPfR6kYBmSNUApf/apRUo97X313UQk33lIixETJIHQVrQ6qKCv7fkvLu9Q4spJoRuzF5GH9NBXns50/UllSEDBDWODVqMTQnJwllsCqPkS8TM/CUJ9lyHNb3DVJNIx6N2F1SchRhQe77CVW2J1H78UV9JayMooQ5bhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883962; c=relaxed/simple; bh=h3ExjGlQ/IfRfWlCDyhOHXurhQr8Q4T+X+082Em90IQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S9MI6QXLSPWlyZQuOBV5HjLexkU6KAFUnedJVlV+3c2Fu6nl/UL3Kg8aLB7PdEUSuNDQX5tgs7MEOmL6z+/MN00jEEI8SNQuX5veIsO3crUPajELN3eUeIl27xSOzoojI84n+LglR5IytOgleOezxZtMjdaxncEoIJ9lvaZv97s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YfwtHf0f; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YfwtHf0f" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D2601F00A3A; Wed, 27 May 2026 12:12:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779883958; bh=Gvz8PiwvGk6Y72u5ZGZh0QDA7xEjtRQQxZQEXXA8yJQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YfwtHf0fCRwSCsi0j+YSaW/YJZ2YIrGStbHkIksFR7bUuKI0CBtGimWacS+sceB0v 3RA84khyfuBEAr35jrluhAudumFm5x4Js412+QPGwuRrWKbO/k1CaPCtlHOM+ZG6Lj ouasmclyuaKE3TJkM1O1Ldlra6/sk2mdoiSXHuIEIQz2kv37voWgJdi/wV8g26lb83 GWgi3KODu4gnvHhtn91x7XCiDW46NmRbTWhPQJYaMCFEpzoCXSDsbHG4pmdhdDbYX0 HyIb2apud3R6SNaVlE1sbJbUu7JRCb+OEg+u8tdeQgjz8pHHKsxOXNzW2UEYxxFhaE J+252EyBhmNBg== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 1/4] perf/core: Fix sched_task callbacks for CPU-wide branch stack events Date: Wed, 27 May 2026 05:11:57 -0700 Message-ID: <20260527121207.2312181-2-puranjay@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260527121207.2312181-1-puranjay@kernel.org> References: <20260527121207.2312181-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" perf_pmu_sched_task() returns early when cpuctx->task_ctx is non-NULL, deferring to perf_ctx_sched_task_cb() in the context sched_in/out paths. But perf_ctx_sched_task_cb() only walks the task context's pmu_ctx_list -- PMUs that have only CPU-wide events are not on that list and their sched_task callback is silently skipped. On ARM64 with CPU-wide branch recording: perf record -b -e cycles -a -- ls armv8pmu_sched_task() is skipped whenever the scheduled task has an unrelated perf event (e.g. a software event), and branch records leak across task boundaries. A second problem exists in __perf_pmu_sched_task(): it passes cpc->task_epc directly to pmu->sched_task(), but task_epc is NULL for PMUs with only CPU-wide events. When perf_pmu_sched_task() does reach the loop (because cpuctx->task_ctx is NULL), this causes a NULL pointer dereference: Unable to handle kernel NULL pointer dereference at virtual address 00[.] PC is at armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0 Fix both: - Remove the blanket early return in perf_pmu_sched_task() when cpuctx->task_ctx is set. Instead, skip individual CPCs that have a task_epc (those are handled by perf_ctx_sched_task_cb()). CPCs without a task_epc are CPU-only and must be handled here. - Fall back to &cpc->epc in __perf_pmu_sched_task() when task_epc is NULL, so the callback always gets a valid pmu_ctx. Fixes: bd2756811766 ("perf: Rewrite core context handling") Signed-off-by: Puranjay Mohan --- kernel/events/core.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 6d1f8bad7e1c..6604f6e8f352 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -3906,7 +3906,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu= _context *cpc, perf_ctx_lock(cpuctx, cpuctx->task_ctx); perf_pmu_disable(pmu); =20 - pmu->sched_task(cpc->task_epc, task, sched_in); + pmu->sched_task(cpc->task_epc ? cpc->task_epc : &cpc->epc, + task, sched_in); =20 perf_pmu_enable(pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); @@ -3919,12 +3920,20 @@ static void perf_pmu_sched_task(struct task_struct = *prev, struct perf_cpu_context *cpuctx =3D this_cpu_ptr(&perf_cpu_context); struct perf_cpu_pmu_context *cpc; =20 - /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ - if (prev =3D=3D next || cpuctx->task_ctx) + if (prev =3D=3D next) return; =20 - list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) + list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) { + /* + * PMUs with per-task events are handled by + * perf_ctx_sched_task_cb() via perf_event_context_sched_in/out + * when a task context is active. + */ + if (cpuctx->task_ctx && cpc->task_epc) + continue; + __perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in); + } } =20 static void perf_event_switch(struct task_struct *task, --=20 2.53.0-Meta From nobody Mon Jun 8 18:55:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E2823F44EB; Wed, 27 May 2026 12:12:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883966; cv=none; b=HpDb0O4a7+Hrp/EHErZVOAOmuSzIiDQSLooetV1L9mxMj3FuSN6iaZE9VmwQI9TJYJKbpNebCWBEjJVHS9lSeXYmqGZkfdjkLoTP1mFAObEIAkYOYvP4jVGXyFn1yaHBhY06RBxEVi8X7khrzru2XZrJYZ0n9T5BVRSJGgxWGSk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883966; c=relaxed/simple; bh=ECf/L8cm8+F05E4oCNG+j2SuvufsDhOisGsADV9b7gI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=V7MwEiiFYOGDtMQXyu4xaTZiYz/gCFwqeDifBYbI5Qmpf4sht7lz9a+WnwCpyuB7LAzCDQ+Uaz1TgMEoBAplCMh8XgJuG3WHjVWDvopzFvnl9za4H+CYMa/crTM9odaap/oq615QDF+j5qz7rYm/O7y0fb2olDMzAcFLER/NyOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=A4T6SXkb; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="A4T6SXkb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA6FE1F000E9; Wed, 27 May 2026 12:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779883963; bh=6no4BGDS0XyYdBrexKoo6TT3tPuXPbKEDRVyJ4oD9Bg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=A4T6SXkbAlc54eDKw65mlkSVgcFAfv+QCqI6M7Eqoj4vFARUv2EOY3fIBfVW0xv2H MynfGT2ZiSPWsJRuX+AzZkU4oh/nFxWZOG9i3D/mzZenfWXHaJh9ncMgHsjLFjXcCx 7A7atXEQJDJYgWXAZiK86m6DTRnAQv61S+aOkAcGaedRko2FEsgCYeADBJK28goY2E xyTx6OYYQ77qUlPZCUZYxVycOP1AvbgNLtKM9axnbp70Ngin8pZWTipAm6TNWTtHvf tXiqXY7lM3yvojNpVnCTikZTtctK6Rfp6wTsdF2g712JE+Nu+3/XaCjSIRxhVxyRqf x1M/Kkxf0MB/w== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 2/4] perf: Use a union to clear branch entry bitfields Date: Wed, 27 May 2026 05:11:58 -0700 Message-ID: <20260527121207.2312181-3-puranjay@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260527121207.2312181-1-puranjay@kernel.org> References: <20260527121207.2312181-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" perf_clear_branch_entry_bitfields() zeroes individual bitfields of struct perf_branch_entry but has repeatedly fallen out of sync when new fields were added (new_type and priv were missed). Wrap the bitfields in an anonymous struct inside a union with a u64 bitfields member, and clear them all with a single assignment. This avoids having to update the clearing function every time a new bitfield is added. Fixes: bfe4daf850f4 ("perf/core: Add perf_clear_branch_entry_bitfields() he= lper") Signed-off-by: Puranjay Mohan --- include/linux/perf_event.h | 9 +-------- include/uapi/linux/perf_event.h | 25 +++++++++++++++---------- tools/include/uapi/linux/perf_event.h | 25 +++++++++++++++---------- 3 files changed, 31 insertions(+), 28 deletions(-) diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 48d851fbd8ea..f7360c43f902 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -1474,14 +1474,7 @@ static inline u32 perf_sample_data_size(struct perf_= sample_data *data, */ static inline void perf_clear_branch_entry_bitfields(struct perf_branch_en= try *br) { - br->mispred =3D 0; - br->predicted =3D 0; - br->in_tx =3D 0; - br->abort =3D 0; - br->cycles =3D 0; - br->type =3D 0; - br->spec =3D PERF_BR_SPEC_NA; - br->reserved =3D 0; + br->bitfields =3D 0; } =20 extern void perf_output_sample(struct perf_output_handle *handle, diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_even= t.h index fd10aa8d697f..c2e7b1b1c4fa 100644 --- a/include/uapi/linux/perf_event.h +++ b/include/uapi/linux/perf_event.h @@ -1491,16 +1491,21 @@ union perf_mem_data_src { struct perf_branch_entry { __u64 from; __u64 to; - __u64 mispred : 1, /* target mispredicted */ - predicted : 1, /* target predicted */ - in_tx : 1, /* in transaction */ - abort : 1, /* transaction abort */ - cycles : 16, /* cycle count to last branch */ - type : 4, /* branch type */ - spec : 2, /* branch speculation info */ - new_type : 4, /* additional branch type */ - priv : 3, /* privilege level */ - reserved : 31; + union { + struct { + __u64 mispred : 1, /* target mispredicted */ + predicted : 1, /* target predicted */ + in_tx : 1, /* in transaction */ + abort : 1, /* transaction abort */ + cycles : 16, /* cycle count to last branch */ + type : 4, /* branch type */ + spec : 2, /* branch speculation info */ + new_type : 4, /* additional branch type */ + priv : 3, /* privilege level */ + reserved : 31; + }; + __u64 bitfields; + }; }; =20 /* Size of used info bits in struct perf_branch_entry */ diff --git a/tools/include/uapi/linux/perf_event.h b/tools/include/uapi/lin= ux/perf_event.h index fd10aa8d697f..c2e7b1b1c4fa 100644 --- a/tools/include/uapi/linux/perf_event.h +++ b/tools/include/uapi/linux/perf_event.h @@ -1491,16 +1491,21 @@ union perf_mem_data_src { struct perf_branch_entry { __u64 from; __u64 to; - __u64 mispred : 1, /* target mispredicted */ - predicted : 1, /* target predicted */ - in_tx : 1, /* in transaction */ - abort : 1, /* transaction abort */ - cycles : 16, /* cycle count to last branch */ - type : 4, /* branch type */ - spec : 2, /* branch speculation info */ - new_type : 4, /* additional branch type */ - priv : 3, /* privilege level */ - reserved : 31; + union { + struct { + __u64 mispred : 1, /* target mispredicted */ + predicted : 1, /* target predicted */ + in_tx : 1, /* in transaction */ + abort : 1, /* transaction abort */ + cycles : 16, /* cycle count to last branch */ + type : 4, /* branch type */ + spec : 2, /* branch speculation info */ + new_type : 4, /* additional branch type */ + priv : 3, /* privilege level */ + reserved : 31; + }; + __u64 bitfields; + }; }; =20 /* Size of used info bits in struct perf_branch_entry */ --=20 2.53.0-Meta From nobody Mon Jun 8 18:55:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30059402420; Wed, 27 May 2026 12:12:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883969; cv=none; b=AybNPr0jQtM4lHkUaewBs5RVMZ2O5g4VX1lolXdYB8KDnXGU/wzAjQIcMz1kr+wawYN0maK/MmhHHNg6MAQxXw+A6u5NIgxt/pyqObpy0quMUoR+uppm6NNIG0zc6HGSu1x04W6tqQQkn50KoYE/hsF3rvuUOkeyo7WYjgohy2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883969; c=relaxed/simple; bh=Ic3lDxiGW2cdIu0QVlSWoepJxl7Uv7qCT4dztXqaQzA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E5UYZrAaPI3iryln6Y1lVZUM342b49N2AD+INUNRzAlj5e+7Z6krWcHWpXGcyxPPhQbG4yJqEQ4A4yQGAmiP60NvFYo498B7Rv9Tn5gSpzipIbaqQCEhjEbk940tsmd6vSg4IGtC5yNyTkmp0ZLuNhvDHfaJ23eOili0CQtYzEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VDKFk7to; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VDKFk7to" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 753181F000E9; Wed, 27 May 2026 12:12:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779883966; bh=RppFL9mNgicTaOiU0sqQ/gT8rSlf9eCMwVU6Ih6AvC8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=VDKFk7tosXeD6tSCj7ICrBik/3NjvBiJ+FNnEZk2ueTvVV6V+r7ltEnV/dMdeNqOl c5SvvQshivzd+1xKyO1HIv95wiCxQnW9LN92Cq6J+EbQMfrWmz/zqAmtbBeuk39Qmy oHaasbHFxqIeUoL01BTrnpqM4RBVyxxZBgKx//UOMyaCfC9+W7jqKow2sB78H7JtF7 IjaqgqecR8eM8oykq8CmuvGPsM34CAvKxRVnliFDdH5DZ23kyp+7Tl1OI/rVs85leH xv8fC4MRaMWSu3j+K47JSLcqfd8z+7eoY6xE2q8fGk3S5gSwhdOxs2eHJph9lhPliU G5v2xHmqOH/3Q== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 3/4] perf/arm64: Add BRBE support for bpf_get_branch_snapshot() Date: Wed, 27 May 2026 05:11:59 -0700 Message-ID: <20260527121207.2312181-4-puranjay@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260527121207.2312181-1-puranjay@kernel.org> References: <20260527121207.2312181-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable bpf_get_branch_snapshot() on ARM64 by implementing the perf_snapshot_branch_stack static call for BRBE. BRBE is paused before masking exceptions to avoid branch buffer pollution from trace_hardirqs_off(). Exceptions are then masked with local_daif_save() to prevent PMU overflow pseudo-NMIs from interfering. If an overflow between pause and DAIF save re-enables BRBE, the snapshot detects this via BRBFCR_EL1.PAUSED and bails out. Branch records are read using perf_entry_from_brbe_regset() with a NULL event pointer to bypass event-specific filtering. The buffer is invalidated after reading. Introduce a for_each_brbe_entry() iterator to deduplicate bank iteration between brbe_read_filtered_entries() and the snapshot. Signed-off-by: Puranjay Mohan Reviewed-by: Rob Herring (Arm) --- drivers/perf/arm_brbe.c | 127 ++++++++++++++++++++++++++++++++------- drivers/perf/arm_brbe.h | 9 +++ drivers/perf/arm_pmuv3.c | 5 +- 3 files changed, 119 insertions(+), 22 deletions(-) diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index ba554e0c846c..aede95e27784 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "arm_brbe.h" =20 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT | \ @@ -256,6 +257,14 @@ static bool valid_brbe_version(int brbe_version) brbe_version =3D=3D ID_AA64DFR0_EL1_BRBE_BRBE_V1P1; } =20 +static __always_inline bool cpu_has_brbe(void) +{ + u64 aa64dfr0 =3D read_sysreg_s(SYS_ID_AA64DFR0_EL1); + int brbe =3D cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_E= L1_BRBE_SHIFT); + + return valid_brbe_version(brbe); +} + static void select_brbe_bank(int bank) { u64 brbfcr; @@ -271,6 +280,20 @@ static void select_brbe_bank(int bank) isb(); } =20 +static inline void __brbe_advance(int *bank, int *idx, int nr_hw) +{ + if (++(*idx) >=3D BRBE_BANK_MAX_ENTRIES && + *bank * BRBE_BANK_MAX_ENTRIES + *idx < nr_hw) { + *idx =3D 0; + select_brbe_bank(++(*bank)); + } +} + +#define for_each_brbe_entry(idx, nr_hw) \ + for (int __bank =3D (select_brbe_bank(0), 0), idx =3D 0; \ + __bank * BRBE_BANK_MAX_ENTRIES + idx < (nr_hw); \ + __brbe_advance(&__bank, &idx, (nr_hw))) + static bool __read_brbe_regset(struct brbe_regset *entry, int idx) { entry->brbinf =3D get_brbinf_reg(idx); @@ -474,11 +497,9 @@ unsigned int brbe_num_branch_records(const struct arm_= pmu *armpmu) =20 void brbe_probe(struct arm_pmu *armpmu) { - u64 brbidr, aa64dfr0 =3D read_sysreg_s(SYS_ID_AA64DFR0_EL1); - u32 brbe; + u64 brbidr; =20 - brbe =3D cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_EL1_B= RBE_SHIFT); - if (!valid_brbe_version(brbe)) + if (!cpu_has_brbe()) return; =20 brbidr =3D read_sysreg_s(SYS_BRBIDR0_EL1); @@ -618,10 +639,10 @@ static bool perf_entry_from_brbe_regset(int index, st= ruct perf_branch_entry *ent =20 brbe_set_perf_entry_type(entry, brbinf); =20 - if (!branch_sample_no_cycles(event)) + if (!event || !branch_sample_no_cycles(event)) entry->cycles =3D brbinf_get_cycles(brbinf); =20 - if (!branch_sample_no_flags(event)) { + if (!event || !branch_sample_no_flags(event)) { /* Mispredict info is available for source only and complete branch reco= rds. */ if (!brbe_record_is_target_only(brbinf)) { entry->mispred =3D brbinf_get_mispredict(brbinf); @@ -774,32 +795,96 @@ void brbe_read_filtered_entries(struct perf_branch_st= ack *branch_stack, { struct arm_pmu *cpu_pmu =3D to_arm_pmu(event->pmu); int nr_hw =3D brbe_num_branch_records(cpu_pmu); - int nr_banks =3D DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES); int nr_filtered =3D 0; u64 branch_sample_type =3D event->attr.branch_sample_type; DECLARE_BITMAP(event_type_mask, PERF_BR_ARM64_MAX); =20 prepare_event_branch_type_mask(branch_sample_type, event_type_mask); =20 - for (int bank =3D 0; bank < nr_banks; bank++) { - int nr_remaining =3D nr_hw - (bank * BRBE_BANK_MAX_ENTRIES); - int nr_this_bank =3D min(nr_remaining, BRBE_BANK_MAX_ENTRIES); + for_each_brbe_entry(i, nr_hw) { + struct perf_branch_entry *pbe =3D &branch_stack->entries[nr_filtered]; =20 - select_brbe_bank(bank); + if (!perf_entry_from_brbe_regset(i, pbe, event)) + break; =20 - for (int i =3D 0; i < nr_this_bank; i++) { - struct perf_branch_entry *pbe =3D &branch_stack->entries[nr_filtered]; + if (!filter_branch_record(pbe, branch_sample_type, event_type_mask)) + continue; =20 - if (!perf_entry_from_brbe_regset(i, pbe, event)) - goto done; + nr_filtered++; + } =20 - if (!filter_branch_record(pbe, branch_sample_type, event_type_mask)) - continue; + branch_stack->nr =3D nr_filtered; +} =20 - nr_filtered++; - } +/* + * Best-effort BRBE snapshot for BPF tracing. Pause BRBE to avoid + * self-recording and return 0 if the snapshot state appears disturbed. + */ +int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsi= gned int cnt) +{ + unsigned long flags; + int nr_hw, nr_copied =3D 0; + u64 brbfcr, brbcr; + + if (!cnt) + return 0; + + /* Guard against running on a CPU without BRBE (e.g. big.LITTLE). */ + if (!cpu_has_brbe()) + return 0; + + /* + * Pause BRBE first to avoid recording our own branches. The + * sysreg read/write and ISB are branchless, so pausing before + * checking BRBCR avoids polluting the buffer with our own + * conditional branches. + */ + brbfcr =3D read_sysreg_s(SYS_BRBFCR_EL1); + brbcr =3D read_sysreg_s(SYS_BRBCR_EL1); + write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + + /* Bail out if BRBE is not enabled (BRBCR_EL1 =3D=3D 0). */ + if (!brbcr) { + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + return 0; } =20 -done: - branch_stack->nr =3D nr_filtered; + /* Block local exception delivery while reading the buffer. */ + flags =3D local_daif_save(); + + /* + * A PMU overflow before local_daif_save() could have re-enabled + * BRBE, clearing the PAUSED bit. The overflow handler already + * restored BRBE to its correct state, so just bail out. + */ + if (!(read_sysreg_s(SYS_BRBFCR_EL1) & BRBFCR_EL1_PAUSED)) { + local_daif_restore(flags); + return 0; + } + + nr_hw =3D FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, + read_sysreg_s(SYS_BRBIDR0_EL1)); + + for_each_brbe_entry(i, nr_hw) { + if (nr_copied >=3D cnt) + break; + + if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL)) + break; + + nr_copied++; + } + + brbe_invalidate(); + + /* Restore BRBCR before unpausing via BRBFCR, matching brbe_enable(). */ + write_sysreg_s(brbcr, SYS_BRBCR_EL1); + isb(); + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + /* Ensure BRBE is unpaused before returning to the caller. */ + isb(); + local_daif_restore(flags); + + return nr_copied; } diff --git a/drivers/perf/arm_brbe.h b/drivers/perf/arm_brbe.h index b7c7d8796c86..c2a1824437fb 100644 --- a/drivers/perf/arm_brbe.h +++ b/drivers/perf/arm_brbe.h @@ -10,6 +10,7 @@ struct arm_pmu; struct perf_branch_stack; struct perf_event; +struct perf_branch_entry; =20 #ifdef CONFIG_ARM64_BRBE void brbe_probe(struct arm_pmu *arm_pmu); @@ -22,6 +23,8 @@ void brbe_disable(void); bool brbe_branch_attr_valid(struct perf_event *event); void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, const struct perf_event *event); +int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, + unsigned int cnt); #else static inline void brbe_probe(struct arm_pmu *arm_pmu) { } static inline unsigned int brbe_num_branch_records(const struct arm_pmu *a= rmpmu) @@ -44,4 +47,10 @@ static void brbe_read_filtered_entries(struct perf_branc= h_stack *branch_stack, const struct perf_event *event) { } + +static inline int arm_brbe_snapshot_branch_stack(struct perf_branch_entry = *entries, + unsigned int cnt) +{ + return 0; +} #endif diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 8014ff766cff..1a9f129a0f94 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -1449,8 +1449,11 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, c= har *name, cpu_pmu->set_event_filter =3D armv8pmu_set_event_filter; =20 cpu_pmu->pmu.event_idx =3D armv8pmu_user_event_idx; - if (brbe_num_branch_records(cpu_pmu)) + if (brbe_num_branch_records(cpu_pmu)) { cpu_pmu->pmu.sched_task =3D armv8pmu_sched_task; + static_call_update(perf_snapshot_branch_stack, + arm_brbe_snapshot_branch_stack); + } =20 cpu_pmu->name =3D name; cpu_pmu->map_event =3D map_event; --=20 2.53.0-Meta From nobody Mon Jun 8 18:55:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A63D3FE660; Wed, 27 May 2026 12:12:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883971; cv=none; b=eGOO+77m+LGXW+bMx2+EhGjyM1RIUUhOCoO//+olG84tXPWN3cy4pFO66mTsFs0eO5KqQ4ccUpYzdv7xe+RbB1caKtEAFFwEMAyDCltuFvH8fxs60UtLhgAhwrPgSwP3S90ts0lBhg5BKF+x5eQSU4qrIwa5RbXupbtk4J5PeNQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779883971; c=relaxed/simple; bh=xGU+zthPArmUciFE3I4/MTbw9lB8zAY6DEuEIIvSCT0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PXQgZ7f/Fyp4hk11osOfaPqeKPaQ9C1xTOHlFu9QOMvCdeAQXd7OYL8fLeDv5i582udG3Q64qV3oSU4jkMYUcpsU1RkbMWhyw9oPxwUE8gZP0qB18OZ5xo7NuT753hIP0LJ1mWeDF9IqqfSnyXLStQa1KLB36Ja6cJ40ciqLdC4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YDnomG9T; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YDnomG9T" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7C511F000E9; Wed, 27 May 2026 12:12:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1779883970; bh=2I29sVWPGLqUjIDyjNAdYt3UxHapRuSjOGldiuvjnFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YDnomG9T/2ke5Rv5/HWio2Pv40kda1zsQ4ipmwuz1tZU2N6mznX/nEB1PIRxVejpm 8MjwhVuv3sZafyTNpSVtn5o283JlpvGkKVt+T+w/bwjaC1kpKaUr8ZtLFNGONEQnhP y83MWM+EwN4ovMGJjCuc4rvIGDZCAfEwHOyDOZz16Qv44uQ3f+gudgHHQqV7cz7xib 6Y2sgglGn272YMw5arS+W1mOnnAM/gWtFXT6O1CEBRogZtTdzdlGWvMB67WlaOwwKk y4azC2jo+CqXj3t3MN2whp0jecPR8xQxiQh6xxRI7/XLrypTkynAirbPivYfTBnGwA SwD0CwaSZUtMQ== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v4 4/4] selftests/bpf: Adjust wasted entries threshold for ARM64 BRBE Date: Wed, 27 May 2026 05:12:00 -0700 Message-ID: <20260527121207.2312181-5-puranjay@kernel.org> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260527121207.2312181-1-puranjay@kernel.org> References: <20260527121207.2312181-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The get_branch_snapshot test checks that bpf_get_branch_snapshot() doesn't waste too many branch entries on infrastructure overhead. The threshold of < 10 was calibrated for x86 where about 7 entries are wasted. On ARM64, the BPF trampoline generates more branches than x86, resulting in about 13 wasted entries. The overhead comes from the BPF trampoline calling __bpf_prog_enter_recur which on ARM64 makes out-of-line calls to __rcu_read_lock and generates more conditional branches than x86: [#12] bpf_testmod_loop_test+0x40 -> bpf_trampoline_...+0x48 [#11] bpf_trampoline_...+0x68 -> __bpf_prog_enter_recur+0x0 [#10] __bpf_prog_enter_recur+0x20 -> __bpf_prog_enter_recur+0x118 [#09] __bpf_prog_enter_recur+0x154 -> __bpf_prog_enter_recur+0x160 [#08] __bpf_prog_enter_recur+0x164 -> __bpf_prog_enter_recur+0x2c [#07] __bpf_prog_enter_recur+0x2c -> __rcu_read_lock+0x0 [#06] __rcu_read_lock+0x18 -> __bpf_prog_enter_recur+0x30 [#05] __bpf_prog_enter_recur+0x9c -> __bpf_prog_enter_recur+0xf0 [#04] __bpf_prog_enter_recur+0xf4 -> __bpf_prog_enter_recur+0xa8 [#03] __bpf_prog_enter_recur+0xb8 -> __bpf_prog_enter_recur+0x100 [#02] __bpf_prog_enter_recur+0x114 -> bpf_trampoline_...+0x6c [#01] bpf_trampoline_...+0x78 -> bpf_prog_...test1+0x0 [#00] bpf_prog_...test1+0x58 -> arm_brbe_snapshot_branch_stack+0x0 Use an architecture-specific threshold of < 14 for ARM64 to accommodate this overhead while still detecting regressions. Signed-off-by: Puranjay Mohan --- .../selftests/bpf/prog_tests/get_branch_snapshot.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c b= /tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c index 0394a1156d99..8d1a3480767f 100644 --- a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c +++ b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c @@ -116,13 +116,18 @@ void serial_test_get_branch_snapshot(void) =20 ASSERT_GT(skel->bss->test1_hits, 6, "find_looptest_in_lbr"); =20 - /* Given we stop LBR in software, we will waste a few entries. + /* Given we stop LBR/BRBE in software, we will waste a few entries. * But we should try to waste as few as possible entries. We are at - * about 7 on x86_64 systems. - * Add a check for < 10 so that we get heads-up when something - * changes and wastes too many entries. + * about 7 on x86_64 and about 13 on arm64 systems (the arm64 BPF + * trampoline generates more branches than x86_64). + * Add a check so that we get heads-up when something changes and + * wastes too many entries. */ +#if defined(__aarch64__) + ASSERT_LT(skel->bss->wasted_entries, 14, "check_wasted_entries"); +#else ASSERT_LT(skel->bss->wasted_entries, 10, "check_wasted_entries"); +#endif =20 cleanup: get_branch_snapshot__destroy(skel); --=20 2.53.0-Meta