From nobody Sat Jun 20 15:19:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10B06450917; Tue, 16 Jun 2026 15:57:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625455; cv=none; b=kCKXnaRx++2yDVr1j4dxzkrP92KUmfnvb7Tosv5Ni/VFBwjl6Mj9lYgw9AiTzknEr0gMt5mFQXSa/HoIDOKkFEEruEHGLI6/YXuAvHLHJOg6UBp2EPGmarLGFveQAbbUY2IwjaICo1gbJo+pSn+uIxaIwHxGWcjfovjUUgCgUcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625455; c=relaxed/simple; bh=h3ExjGlQ/IfRfWlCDyhOHXurhQr8Q4T+X+082Em90IQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hfJIEoES3VGEDtJeVw5gPn5xDjWBGHuki+NyENCW5aFGdv037e3gcDpggSNPK2Cju8Uv513nkqJtYfcM/SJqPNmaMppqD29u9IOdrqR9tx5586AqeklAn7buw7ZLDYALPXEe3JkTqMH/F+SdoLh0CD0wNWxev/UtRb+CMvjBt04= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PNqB2RPy; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PNqB2RPy" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B408F1F000E9; Tue, 16 Jun 2026 15:57:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781625453; bh=Gvz8PiwvGk6Y72u5ZGZh0QDA7xEjtRQQxZQEXXA8yJQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=PNqB2RPyAc+8k7fzb3OAUcI7VhHpEsjB0I3i56mjT3hIfD6R4dWgKDnA64XGFSKfy j10q2wlupMNlWbBR3AnaunTkzmlOePMENVTIBm46MDzjTegV4hN7tRKlEGsTN4Katd FSoNjRs4K84iZQXIMqp5OUtE5gVDBF1MWDj8x65XSG3oeOXoELdWuq5utUOScgzDHq RN9xO4nMVYN6Wa69nTl6uNwKq0ZLnlAkL+WYLfMSu4SB8hdaeibPEmk3py9dmeWpv/ 87YMcRJwVwO0IWtSiFDkDzzQTBN32RkwXkmJOv2z486uxnhnPrRDYtmXTBqaHZYIoE gh7smqnJv52iw== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v5 1/4] perf/core: Fix sched_task callbacks for CPU-wide branch stack events Date: Tue, 16 Jun 2026 08:57:06 -0700 Message-ID: <20260616155716.2631508-2-puranjay@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260616155716.2631508-1-puranjay@kernel.org> References: <20260616155716.2631508-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" perf_pmu_sched_task() returns early when cpuctx->task_ctx is non-NULL, deferring to perf_ctx_sched_task_cb() in the context sched_in/out paths. But perf_ctx_sched_task_cb() only walks the task context's pmu_ctx_list -- PMUs that have only CPU-wide events are not on that list and their sched_task callback is silently skipped. On ARM64 with CPU-wide branch recording: perf record -b -e cycles -a -- ls armv8pmu_sched_task() is skipped whenever the scheduled task has an unrelated perf event (e.g. a software event), and branch records leak across task boundaries. A second problem exists in __perf_pmu_sched_task(): it passes cpc->task_epc directly to pmu->sched_task(), but task_epc is NULL for PMUs with only CPU-wide events. When perf_pmu_sched_task() does reach the loop (because cpuctx->task_ctx is NULL), this causes a NULL pointer dereference: Unable to handle kernel NULL pointer dereference at virtual address 00[.] PC is at armv8pmu_sched_task+0x14/0x50 Call trace: armv8pmu_sched_task+0x14/0x50 (P) perf_pmu_sched_task+0xac/0x108 __perf_event_task_sched_out+0x6c/0xe0 Fix both: - Remove the blanket early return in perf_pmu_sched_task() when cpuctx->task_ctx is set. Instead, skip individual CPCs that have a task_epc (those are handled by perf_ctx_sched_task_cb()). CPCs without a task_epc are CPU-only and must be handled here. - Fall back to &cpc->epc in __perf_pmu_sched_task() when task_epc is NULL, so the callback always gets a valid pmu_ctx. Fixes: bd2756811766 ("perf: Rewrite core context handling") Signed-off-by: Puranjay Mohan --- kernel/events/core.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/kernel/events/core.c b/kernel/events/core.c index 6d1f8bad7e1c..6604f6e8f352 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -3906,7 +3906,8 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu= _context *cpc, perf_ctx_lock(cpuctx, cpuctx->task_ctx); perf_pmu_disable(pmu); =20 - pmu->sched_task(cpc->task_epc, task, sched_in); + pmu->sched_task(cpc->task_epc ? cpc->task_epc : &cpc->epc, + task, sched_in); =20 perf_pmu_enable(pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); @@ -3919,12 +3920,20 @@ static void perf_pmu_sched_task(struct task_struct = *prev, struct perf_cpu_context *cpuctx =3D this_cpu_ptr(&perf_cpu_context); struct perf_cpu_pmu_context *cpc; =20 - /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ - if (prev =3D=3D next || cpuctx->task_ctx) + if (prev =3D=3D next) return; =20 - list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) + list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) { + /* + * PMUs with per-task events are handled by + * perf_ctx_sched_task_cb() via perf_event_context_sched_in/out + * when a task context is active. + */ + if (cpuctx->task_ctx && cpc->task_epc) + continue; + __perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in); + } } =20 static void perf_event_switch(struct task_struct *task, --=20 2.53.0-Meta From nobody Sat Jun 20 15:19:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE1B14508E9; Tue, 16 Jun 2026 15:57:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625460; cv=none; b=mtaiAjCsjPwRgEg2FmNZ4us1QK1vVYwqVWMLfx7CU3yJ229thjwRP1rWsLiaVUFtNZe6MYcWzpuxmOHXUXYdyrRfMIOLGGqTG/6o11egXBcGdIc9Wooy/25H21V0awupvw1qKTS945MpdXPclIof3/NlhTaqdx4795QPBjKk1zQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625460; c=relaxed/simple; bh=/t22Ufo1TWQl+8eUGdEHLy9GAgphqs2agBvYPBixQ68=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DNFWYc68dSnmXmHorAdmQSoLCxiZViolP4gGrdFQaVWyl+V0+UcNDymQzKKhUk5NW2f8yDE4w2lvO93smjDpfTPAG/+WIuT9BCwFpqejmq8PsN4rTgrM4hG9R56JtVfy9B8wa/0LlTVLjpzZinqwSrW/UrWgylgJVPIsdIvxNPw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=g3MFfWnb; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="g3MFfWnb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E2B81F00A3A; Tue, 16 Jun 2026 15:57:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781625458; bh=+VVoEj6gKLm6QpDzoKPZEs8KuFBJW/1u5UDllJ4ILPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=g3MFfWnbbjmGUc3L793M2E9oL6Z2upxdxulI4iWXaWHZu6RksYYoKjmGSp+2GqfSy cgFTdz9frDYcnMdcAtgpWTvoBl0VQGnLnXhIsNC3lJo4hgByeJe7UMBCqtI2/aA5O7 x51vWyjKqZthh0Oz6+ar5ipR5H0vXOr53neW+RVSU0MvfWJKSxhXzODdJHIj5yLI7o ZxyVRLzixfwVnB8PZZevUGQ4L7xo/1tyBESPqKSrznvTVnH/rgfIXUs3t/6XngyqW8 IjhV5pqJcZ964PznEce84RdsNKWtl+4VQMfb8vQPsURaApZMx4ietWNOIk/1XgMQ9o 0wdLO3Ihu/qog== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v5 2/4] perf/core: Clear the whole branch entry in perf_clear_branch_entry() Date: Tue, 16 Jun 2026 08:57:07 -0700 Message-ID: <20260616155716.2631508-3-puranjay@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260616155716.2631508-1-puranjay@kernel.org> References: <20260616155716.2631508-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" perf_clear_branch_entry_bitfields() resets the bitfields of struct perf_branch_entry one at a time and deliberately leaves from/to alone, since callers overwrite those immediately. The list of assignments has to be kept in sync with the struct by hand, and it has already drifted: new_type and priv were added to perf_branch_entry but never cleared here, so stale values can leak into the records handed to userspace. Clear the entry with a single struct assignment instead: *br =3D (struct perf_branch_entry){ }; Every caller writes from/to right after the clear, so zeroing them as well is harmless and the dead stores are elided on the x86 LBR read paths. There is no longer anything to keep in sync when a field is added. The helper no longer touches only the bitfields, so rename it to perf_clear_branch_entry() and update the callers, fixing up the br+nr/br+out spacing on the touched lines while at it. Fixes: b190bc4ac9e6 ("perf: Extend branch type classification") Fixes: 5402d25aa571 ("perf: Capture branch privilege information") Suggested-by: James Clark Signed-off-by: Puranjay Mohan --- arch/x86/events/amd/brs.c | 2 +- arch/x86/events/amd/lbr.c | 2 +- arch/x86/events/intel/lbr.c | 6 +++--- drivers/perf/arm_brbe.c | 2 +- include/linux/perf_event.h | 16 ++-------------- 5 files changed, 8 insertions(+), 20 deletions(-) diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c index 06f35a6b58a5..68c5f42965e9 100644 --- a/arch/x86/events/amd/brs.c +++ b/arch/x86/events/amd/brs.c @@ -343,7 +343,7 @@ void amd_brs_drain(void) =20 rdmsrq(brs_from(brs_idx), from); =20 - perf_clear_branch_entry_bitfields(br+nr); + perf_clear_branch_entry(br + nr); =20 br[nr].from =3D from; br[nr].to =3D to; diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c index d24da377df77..08401fd60585 100644 --- a/arch/x86/events/amd/lbr.c +++ b/arch/x86/events/amd/lbr.c @@ -181,7 +181,7 @@ void amd_pmu_lbr_read(void) entry.to.split.reserved) continue; =20 - perf_clear_branch_entry_bitfields(br + out); + perf_clear_branch_entry(br + out); =20 br[out].from =3D sign_ext_branch_ip(entry.from.split.ip); br[out].to =3D sign_ext_branch_ip(entry.to.split.ip); diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c index 72f2adcda7c6..295da179fa74 100644 --- a/arch/x86/events/intel/lbr.c +++ b/arch/x86/events/intel/lbr.c @@ -755,7 +755,7 @@ void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc) =20 rdmsrq(x86_pmu.lbr_from + lbr_idx, msr_lastbranch.lbr); =20 - perf_clear_branch_entry_bitfields(br); + perf_clear_branch_entry(br); =20 br->from =3D msr_lastbranch.from; br->to =3D msr_lastbranch.to; @@ -846,7 +846,7 @@ void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc) if (abort && x86_pmu.lbr_double_abort && out > 0) out--; =20 - perf_clear_branch_entry_bitfields(br+out); + perf_clear_branch_entry(br + out); br[out].from =3D from; br[out].to =3D to; br[out].mispred =3D mis; @@ -920,7 +920,7 @@ static void intel_pmu_store_lbr(struct cpu_hw_events *c= puc, to =3D rdlbr_to(i, lbr); info =3D rdlbr_info(i, lbr); =20 - perf_clear_branch_entry_bitfields(e); + perf_clear_branch_entry(e); =20 e->from =3D from; e->to =3D to; diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index ba554e0c846c..effbdeacfcbb 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -604,7 +604,7 @@ static bool perf_entry_from_brbe_regset(int index, stru= ct perf_branch_entry *ent return false; =20 brbinf =3D bregs.brbinf; - perf_clear_branch_entry_bitfields(entry); + perf_clear_branch_entry(entry); if (brbe_record_is_complete(brbinf)) { entry->from =3D bregs.brbsrc; entry->to =3D bregs.brbtgt; diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 48d851fbd8ea..e034be4a473a 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -1467,21 +1467,9 @@ static inline u32 perf_sample_data_size(struct perf_= sample_data *data, return size; } =20 -/* - * Clear all bitfields in the perf_branch_entry. - * The to and from fields are not cleared because they are - * systematically modified by caller. - */ -static inline void perf_clear_branch_entry_bitfields(struct perf_branch_en= try *br) +static inline void perf_clear_branch_entry(struct perf_branch_entry *br) { - br->mispred =3D 0; - br->predicted =3D 0; - br->in_tx =3D 0; - br->abort =3D 0; - br->cycles =3D 0; - br->type =3D 0; - br->spec =3D PERF_BR_SPEC_NA; - br->reserved =3D 0; + *br =3D (struct perf_branch_entry){ }; } =20 extern void perf_output_sample(struct perf_output_handle *handle, --=20 2.53.0-Meta From nobody Sat Jun 20 15:19:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2C52466B69; Tue, 16 Jun 2026 15:57:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625463; cv=none; b=Gucv9MTDDQFnR0lMHmf3Nk5JAKgazMz4Npx26v9vnFdS4odENBKdeDRdFMwpMs5HaHo1byR93bbuxlvXNyuVP8dVgPvihqMGbAOPJ+yaSzGZhqGOfgB9C6YSf/W4NhpBf9uJ4JGTnYMC07uDTT5D9Oj5VFSfo4l95S0W5+3+CuM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625463; c=relaxed/simple; bh=aes1XEmYBDl2RmYRB/wQuLLgjq8XkF/pQwwZsWdtxr0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B692jS387+PtI3mmCa6LBaJyYqbhwVBu0ZrhZY81UyrQtvml/kG8Jn1vKeq9cOk1Uv5b9bS8bvnnmmlnWqWORAgqv53FnvTJx6vJ3XMPiwoZqTN+0KWrUB670C5KDWIBpINYNcGT9lCbnaarsZR/ZQLKe/IbG9tDHK1/bY/dDlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XSczpPvt; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XSczpPvt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6B9B21F00A3A; Tue, 16 Jun 2026 15:57:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781625461; bh=8sgGW/qEFwPxZcovDLzHGJGJ+O9xnnJBPViL03Al5ao=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=XSczpPvtXcnnisWlEMeu/A6NHYYOUbsDG0MP/BqBk4r/YpnvF8KmEAFtYN3XuvEd+ HN5nSo3HjZo5i2YdUnlpMY4WRIa59jsg4O/WZVyleuOEoXRdNfGFOwJK1dQf83XZy6 wxje0P88CYhbylHt5sSGJ+wKHVszHfmHNTAzz7B8RPKAS9Yo4h5w5ZVLLIFYe7y/jE sOcsRWiO0xewKhH8W5Luf+sUzNj4sZyfH0m1728/3PdAlkk1fjv5XaTS8Qih87c+zD 2a90ne1lczfTgobg7WYqkznhZxpFCsplxIOsZ3pbWQCAN8OzaZXZkp6YVqT1f5Outz yCqVe2fkpoyQQ== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v5 3/4] perf/arm64: Add BRBE support for bpf_get_branch_snapshot() Date: Tue, 16 Jun 2026 08:57:08 -0700 Message-ID: <20260616155716.2631508-4-puranjay@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260616155716.2631508-1-puranjay@kernel.org> References: <20260616155716.2631508-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable bpf_get_branch_snapshot() on ARM64 by implementing the perf_snapshot_branch_stack static call for BRBE. BRBE is paused before masking exceptions to avoid branch buffer pollution from trace_hardirqs_off(). Exceptions are then masked with local_daif_save() to prevent PMU overflow pseudo-NMIs from interfering. If an overflow between pause and DAIF save re-enables BRBE, the snapshot detects this via BRBFCR_EL1.PAUSED and bails out. Branch records are read using perf_entry_from_brbe_regset() with a NULL event pointer to bypass event-specific filtering. The buffer is invalidated after reading. Introduce a for_each_brbe_entry() iterator to deduplicate bank iteration between brbe_read_filtered_entries() and the snapshot. Signed-off-by: Puranjay Mohan Reviewed-by: Rob Herring (Arm) --- drivers/perf/arm_brbe.c | 128 ++++++++++++++++++++++++++++++++------- drivers/perf/arm_brbe.h | 9 +++ drivers/perf/arm_pmuv3.c | 5 +- 3 files changed, 120 insertions(+), 22 deletions(-) diff --git a/drivers/perf/arm_brbe.c b/drivers/perf/arm_brbe.c index effbdeacfcbb..a141ad7abcf2 100644 --- a/drivers/perf/arm_brbe.c +++ b/drivers/perf/arm_brbe.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "arm_brbe.h" =20 #define BRBFCR_EL1_BRANCH_FILTERS (BRBFCR_EL1_DIRECT | \ @@ -256,6 +257,14 @@ static bool valid_brbe_version(int brbe_version) brbe_version =3D=3D ID_AA64DFR0_EL1_BRBE_BRBE_V1P1; } =20 +static __always_inline bool cpu_has_brbe(void) +{ + u64 aa64dfr0 =3D read_sysreg_s(SYS_ID_AA64DFR0_EL1); + int brbe =3D cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_E= L1_BRBE_SHIFT); + + return valid_brbe_version(brbe); +} + static void select_brbe_bank(int bank) { u64 brbfcr; @@ -271,6 +280,20 @@ static void select_brbe_bank(int bank) isb(); } =20 +static inline void __brbe_advance(int *bank, int *idx, int nr_hw) +{ + if (++(*idx) >=3D BRBE_BANK_MAX_ENTRIES && + *bank * BRBE_BANK_MAX_ENTRIES + *idx < nr_hw) { + *idx =3D 0; + select_brbe_bank(++(*bank)); + } +} + +#define for_each_brbe_entry(idx, nr_hw) \ + for (int __bank =3D (select_brbe_bank(0), 0), idx =3D 0; \ + __bank * BRBE_BANK_MAX_ENTRIES + idx < (nr_hw); \ + __brbe_advance(&__bank, &idx, (nr_hw))) + static bool __read_brbe_regset(struct brbe_regset *entry, int idx) { entry->brbinf =3D get_brbinf_reg(idx); @@ -474,11 +497,9 @@ unsigned int brbe_num_branch_records(const struct arm_= pmu *armpmu) =20 void brbe_probe(struct arm_pmu *armpmu) { - u64 brbidr, aa64dfr0 =3D read_sysreg_s(SYS_ID_AA64DFR0_EL1); - u32 brbe; + u64 brbidr; =20 - brbe =3D cpuid_feature_extract_unsigned_field(aa64dfr0, ID_AA64DFR0_EL1_B= RBE_SHIFT); - if (!valid_brbe_version(brbe)) + if (!cpu_has_brbe()) return; =20 brbidr =3D read_sysreg_s(SYS_BRBIDR0_EL1); @@ -618,10 +639,10 @@ static bool perf_entry_from_brbe_regset(int index, st= ruct perf_branch_entry *ent =20 brbe_set_perf_entry_type(entry, brbinf); =20 - if (!branch_sample_no_cycles(event)) + if (!event || !branch_sample_no_cycles(event)) entry->cycles =3D brbinf_get_cycles(brbinf); =20 - if (!branch_sample_no_flags(event)) { + if (!event || !branch_sample_no_flags(event)) { /* Mispredict info is available for source only and complete branch reco= rds. */ if (!brbe_record_is_target_only(brbinf)) { entry->mispred =3D brbinf_get_mispredict(brbinf); @@ -774,32 +795,97 @@ void brbe_read_filtered_entries(struct perf_branch_st= ack *branch_stack, { struct arm_pmu *cpu_pmu =3D to_arm_pmu(event->pmu); int nr_hw =3D brbe_num_branch_records(cpu_pmu); - int nr_banks =3D DIV_ROUND_UP(nr_hw, BRBE_BANK_MAX_ENTRIES); int nr_filtered =3D 0; u64 branch_sample_type =3D event->attr.branch_sample_type; DECLARE_BITMAP(event_type_mask, PERF_BR_ARM64_MAX); =20 prepare_event_branch_type_mask(branch_sample_type, event_type_mask); =20 - for (int bank =3D 0; bank < nr_banks; bank++) { - int nr_remaining =3D nr_hw - (bank * BRBE_BANK_MAX_ENTRIES); - int nr_this_bank =3D min(nr_remaining, BRBE_BANK_MAX_ENTRIES); + for_each_brbe_entry(i, nr_hw) { + struct perf_branch_entry *pbe =3D &branch_stack->entries[nr_filtered]; =20 - select_brbe_bank(bank); + if (!perf_entry_from_brbe_regset(i, pbe, event)) + break; =20 - for (int i =3D 0; i < nr_this_bank; i++) { - struct perf_branch_entry *pbe =3D &branch_stack->entries[nr_filtered]; + if (!filter_branch_record(pbe, branch_sample_type, event_type_mask)) + continue; =20 - if (!perf_entry_from_brbe_regset(i, pbe, event)) - goto done; + nr_filtered++; + } =20 - if (!filter_branch_record(pbe, branch_sample_type, event_type_mask)) - continue; + branch_stack->nr =3D nr_filtered; +} =20 - nr_filtered++; - } +/* + * Best-effort BRBE snapshot for BPF tracing. Pause BRBE to avoid + * self-recording and return 0 if the snapshot state appears disturbed. + */ +int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, unsi= gned int cnt) +{ + unsigned long flags; + int nr_hw, nr_copied =3D 0; + u64 brbfcr, brbcr; + + if (!cnt) + return 0; + + /* Guard against running on a CPU without BRBE (e.g. big.LITTLE). */ + if (!cpu_has_brbe()) + return 0; + + /* + * Pause BRBE first to avoid recording our own branches. The + * sysreg read/write and ISB are branchless, so pausing before + * checking BRBCR avoids polluting the buffer with our own + * conditional branches. + */ + brbfcr =3D read_sysreg_s(SYS_BRBFCR_EL1); + brbcr =3D read_sysreg_s(SYS_BRBCR_EL1); + write_sysreg_s(brbfcr | BRBFCR_EL1_PAUSED, SYS_BRBFCR_EL1); + isb(); + + /* Bail out if BRBE is not enabled (BRBCR_EL1 =3D=3D 0). */ + if (!brbcr) { + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + isb(); + return 0; } =20 -done: - branch_stack->nr =3D nr_filtered; + /* Block local exception delivery while reading the buffer. */ + flags =3D local_daif_save(); + + /* + * A PMU overflow before local_daif_save() could have re-enabled + * BRBE, clearing the PAUSED bit. The overflow handler already + * restored BRBE to its correct state, so just bail out. + */ + if (!(read_sysreg_s(SYS_BRBFCR_EL1) & BRBFCR_EL1_PAUSED)) { + local_daif_restore(flags); + return 0; + } + + nr_hw =3D FIELD_GET(BRBIDR0_EL1_NUMREC_MASK, + read_sysreg_s(SYS_BRBIDR0_EL1)); + + for_each_brbe_entry(i, nr_hw) { + if (nr_copied >=3D cnt) + break; + + if (!perf_entry_from_brbe_regset(i, &entries[nr_copied], NULL)) + break; + + nr_copied++; + } + + brbe_invalidate(); + + /* Restore BRBCR before unpausing via BRBFCR, matching brbe_enable(). */ + write_sysreg_s(brbcr, SYS_BRBCR_EL1); + isb(); + write_sysreg_s(brbfcr, SYS_BRBFCR_EL1); + /* Ensure BRBE is unpaused before returning to the caller. */ + isb(); + local_daif_restore(flags); + + return nr_copied; } diff --git a/drivers/perf/arm_brbe.h b/drivers/perf/arm_brbe.h index b7c7d8796c86..c2a1824437fb 100644 --- a/drivers/perf/arm_brbe.h +++ b/drivers/perf/arm_brbe.h @@ -10,6 +10,7 @@ struct arm_pmu; struct perf_branch_stack; struct perf_event; +struct perf_branch_entry; =20 #ifdef CONFIG_ARM64_BRBE void brbe_probe(struct arm_pmu *arm_pmu); @@ -22,6 +23,8 @@ void brbe_disable(void); bool brbe_branch_attr_valid(struct perf_event *event); void brbe_read_filtered_entries(struct perf_branch_stack *branch_stack, const struct perf_event *event); +int arm_brbe_snapshot_branch_stack(struct perf_branch_entry *entries, + unsigned int cnt); #else static inline void brbe_probe(struct arm_pmu *arm_pmu) { } static inline unsigned int brbe_num_branch_records(const struct arm_pmu *a= rmpmu) @@ -44,4 +47,10 @@ static void brbe_read_filtered_entries(struct perf_branc= h_stack *branch_stack, const struct perf_event *event) { } + +static inline int arm_brbe_snapshot_branch_stack(struct perf_branch_entry = *entries, + unsigned int cnt) +{ + return 0; +} #endif diff --git a/drivers/perf/arm_pmuv3.c b/drivers/perf/arm_pmuv3.c index 8014ff766cff..1a9f129a0f94 100644 --- a/drivers/perf/arm_pmuv3.c +++ b/drivers/perf/arm_pmuv3.c @@ -1449,8 +1449,11 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, c= har *name, cpu_pmu->set_event_filter =3D armv8pmu_set_event_filter; =20 cpu_pmu->pmu.event_idx =3D armv8pmu_user_event_idx; - if (brbe_num_branch_records(cpu_pmu)) + if (brbe_num_branch_records(cpu_pmu)) { cpu_pmu->pmu.sched_task =3D armv8pmu_sched_task; + static_call_update(perf_snapshot_branch_stack, + arm_brbe_snapshot_branch_stack); + } =20 cpu_pmu->name =3D name; cpu_pmu->map_event =3D map_event; --=20 2.53.0-Meta From nobody Sat Jun 20 15:19:24 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E318746AF10; Tue, 16 Jun 2026 15:57:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625466; cv=none; b=BXfZNo1ZblqkhRU8zaZ6CvarXJV/vkTN87ZUV+ez5LOFymWVhknC1JsWs7qnquR297TS2mpfFUwhHnDKXzOejxE38yVK8IwYEdgzPhx5sGuLV/ODVNBaZRl8w4gkgzrtHWyDyRvWZAgrAM1MBjoqS8WdyfofGhLzUN1AFlmjBsc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781625466; c=relaxed/simple; bh=xGU+zthPArmUciFE3I4/MTbw9lB8zAY6DEuEIIvSCT0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qAPhFuLVZQA4gPBGb5ntRR4HG6eTXXvsz2dzcwiV/vOICFQxWUXdy9UEyYu0/tEJ7SyY06JkmXXWn7Vd+tMY+LegiZ3KvWNZFjJrLaXrhIiwwRcq+vuDo27/mz/TSGijm7hYp1bHmeiV878LGdQSQlHku1XuE4ouOSit9uA0JkQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hvMrhoDs; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hvMrhoDs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A96C1F000E9; Tue, 16 Jun 2026 15:57:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1781625464; bh=2I29sVWPGLqUjIDyjNAdYt3UxHapRuSjOGldiuvjnFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hvMrhoDsIv55pDkqtMQJkIvBvYtP03iTeGsHOKYQksDENhjkdvt9oTdqQ6yGdQlKd fOW7NTIQXoVV2mbxwF15GJj2IJpSsejIFL9HNkDRbSZFD9B1OlmtQycYJ0cmPFb0kw 1Drcf4CbSV259gNGECV3IW90U1lovSnbmhAnZjzVOeJdmOUuzeQZHBaE7xDofPcJez bpoflV05cCVyOiJfYSKL7WioHrVHh5WF6Lr+uKIfhwPi4LyznsS/UqeZCOcyyDPMYd PE0o/Au1LOj7t+lvbuOtwYQaM7iCDKtlhvp2DYc9ed78wZECHVpWozbf3kXeQz5nn5 lGQiaqEE3z8QA== From: Puranjay Mohan To: bpf@vger.kernel.org Cc: Puranjay Mohan , Puranjay Mohan , Alexei Starovoitov , Daniel Borkmann , John Fastabend , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , Will Deacon , Mark Rutland , Catalin Marinas , Leo Yan , Rob Herring , Peter Zijlstra , Ingo Molnar , Arnaldo Carvalho de Melo , Namhyung Kim , James Clark , Ian Rogers , Adrian Hunter , Shuah Khan , Breno Leitao , Ravi Bangoria , Stephane Eranian , Kumar Kartikeya Dwivedi , Usama Arif , linux-arm-kernel@lists.infradead.org, linux-perf-users@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com Subject: [PATCH v5 4/4] selftests/bpf: Adjust wasted entries threshold for ARM64 BRBE Date: Tue, 16 Jun 2026 08:57:09 -0700 Message-ID: <20260616155716.2631508-5-puranjay@kernel.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260616155716.2631508-1-puranjay@kernel.org> References: <20260616155716.2631508-1-puranjay@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The get_branch_snapshot test checks that bpf_get_branch_snapshot() doesn't waste too many branch entries on infrastructure overhead. The threshold of < 10 was calibrated for x86 where about 7 entries are wasted. On ARM64, the BPF trampoline generates more branches than x86, resulting in about 13 wasted entries. The overhead comes from the BPF trampoline calling __bpf_prog_enter_recur which on ARM64 makes out-of-line calls to __rcu_read_lock and generates more conditional branches than x86: [#12] bpf_testmod_loop_test+0x40 -> bpf_trampoline_...+0x48 [#11] bpf_trampoline_...+0x68 -> __bpf_prog_enter_recur+0x0 [#10] __bpf_prog_enter_recur+0x20 -> __bpf_prog_enter_recur+0x118 [#09] __bpf_prog_enter_recur+0x154 -> __bpf_prog_enter_recur+0x160 [#08] __bpf_prog_enter_recur+0x164 -> __bpf_prog_enter_recur+0x2c [#07] __bpf_prog_enter_recur+0x2c -> __rcu_read_lock+0x0 [#06] __rcu_read_lock+0x18 -> __bpf_prog_enter_recur+0x30 [#05] __bpf_prog_enter_recur+0x9c -> __bpf_prog_enter_recur+0xf0 [#04] __bpf_prog_enter_recur+0xf4 -> __bpf_prog_enter_recur+0xa8 [#03] __bpf_prog_enter_recur+0xb8 -> __bpf_prog_enter_recur+0x100 [#02] __bpf_prog_enter_recur+0x114 -> bpf_trampoline_...+0x6c [#01] bpf_trampoline_...+0x78 -> bpf_prog_...test1+0x0 [#00] bpf_prog_...test1+0x58 -> arm_brbe_snapshot_branch_stack+0x0 Use an architecture-specific threshold of < 14 for ARM64 to accommodate this overhead while still detecting regressions. Signed-off-by: Puranjay Mohan --- .../selftests/bpf/prog_tests/get_branch_snapshot.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c b= /tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c index 0394a1156d99..8d1a3480767f 100644 --- a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c +++ b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c @@ -116,13 +116,18 @@ void serial_test_get_branch_snapshot(void) =20 ASSERT_GT(skel->bss->test1_hits, 6, "find_looptest_in_lbr"); =20 - /* Given we stop LBR in software, we will waste a few entries. + /* Given we stop LBR/BRBE in software, we will waste a few entries. * But we should try to waste as few as possible entries. We are at - * about 7 on x86_64 systems. - * Add a check for < 10 so that we get heads-up when something - * changes and wastes too many entries. + * about 7 on x86_64 and about 13 on arm64 systems (the arm64 BPF + * trampoline generates more branches than x86_64). + * Add a check so that we get heads-up when something changes and + * wastes too many entries. */ +#if defined(__aarch64__) + ASSERT_LT(skel->bss->wasted_entries, 14, "check_wasted_entries"); +#else ASSERT_LT(skel->bss->wasted_entries, 10, "check_wasted_entries"); +#endif =20 cleanup: get_branch_snapshot__destroy(skel); --=20 2.53.0-Meta