arch/arm64/Kbuild | 4 ++++ 1 file changed, 4 insertions(+)
The arm64 kernel doesn't boot with annotated branches
(PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
Bisecting it, I found that disabling branch profiling in arch/arm64/mm
solved the problem. Narrowing down a bit further, I found that
physaddr.c is the file that needs to have branch profiling disabled to
get the machine to boot.
I suspect that it might invoke some ftrace helper very early in the boot
process and ftrace is still not enabled(!?).
Rather than playing whack-a-mole with individual files, disable branch
profiling for the entire arch/arm64 tree, similar to what x86 already
does in arch/x86/Kbuild.
Cc: stable@vger.kernel.org
Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
Signed-off-by: Breno Leitao <leitao@debian.org>
---
Changes in v2:
- Expand the scope to arch/arm64 instead of just physaddr.c
- Link to v1: https://lore.kernel.org/all/20251231-annotated-v1-1-9db1c0d03062@debian.org/
---
arch/arm64/Kbuild | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild
index 5bfbf7d79c99..d876bc0e5421 100644
--- a/arch/arm64/Kbuild
+++ b/arch/arm64/Kbuild
@@ -1,4 +1,8 @@
# SPDX-License-Identifier: GPL-2.0-only
+
+# Branch profiling isn't noinstr-safe
+subdir-ccflags-$(CONFIG_TRACE_BRANCH_PROFILING) += -DDISABLE_BRANCH_PROFILING
+
obj-y += kernel/ mm/ net/
obj-$(CONFIG_KVM) += kvm/
obj-$(CONFIG_XEN) += xen/
---
base-commit: c8ebd433459bcbf068682b09544e830acd7ed222
change-id: 20251231-annotated-75de3f33cd7b
Best regards,
--
Breno Leitao <leitao@debian.org>On Tue, 06 Jan 2026 02:16:35 -0800, Breno Leitao wrote:
> The arm64 kernel doesn't boot with annotated branches
> (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
>
> Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> solved the problem. Narrowing down a bit further, I found that
> physaddr.c is the file that needs to have branch profiling disabled to
> get the machine to boot.
>
> [...]
Applied to arm64 (for-next/misc), thanks!
[1/1] arm64: Disable branch profiling for all arm64 code
https://git.kernel.org/arm64/c/f22c81bebf8b
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
> The arm64 kernel doesn't boot with annotated branches
> (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
>
> Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> solved the problem. Narrowing down a bit further, I found that
> physaddr.c is the file that needs to have branch profiling disabled to
> get the machine to boot.
>
> I suspect that it might invoke some ftrace helper very early in the boot
> process and ftrace is still not enabled(!?).
>
> Rather than playing whack-a-mole with individual files, disable branch
> profiling for the entire arch/arm64 tree, similar to what x86 already
> does in arch/x86/Kbuild.
>
> Cc: stable@vger.kernel.org
> Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
> Signed-off-by: Breno Leitao <leitao@debian.org>
I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL
is unsound in a number of places, so I'd prefer to remove that Fixes tag
and backport this for all stable trees.
Regardless, I'm in favour of disabling CONFIG_DEBUG_VIRTUAL widely, so:
Acked-by: Mark Rutland <mark.rutland@arm.com>
Mark.
> ---
> Changes in v2:
> - Expand the scope to arch/arm64 instead of just physaddr.c
> - Link to v1: https://lore.kernel.org/all/20251231-annotated-v1-1-9db1c0d03062@debian.org/
> ---
> arch/arm64/Kbuild | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/arch/arm64/Kbuild b/arch/arm64/Kbuild
> index 5bfbf7d79c99..d876bc0e5421 100644
> --- a/arch/arm64/Kbuild
> +++ b/arch/arm64/Kbuild
> @@ -1,4 +1,8 @@
> # SPDX-License-Identifier: GPL-2.0-only
> +
> +# Branch profiling isn't noinstr-safe
> +subdir-ccflags-$(CONFIG_TRACE_BRANCH_PROFILING) += -DDISABLE_BRANCH_PROFILING
> +
> obj-y += kernel/ mm/ net/
> obj-$(CONFIG_KVM) += kvm/
> obj-$(CONFIG_XEN) += xen/
>
> ---
> base-commit: c8ebd433459bcbf068682b09544e830acd7ed222
> change-id: 20251231-annotated-75de3f33cd7b
>
> Best regards,
> --
> Breno Leitao <leitao@debian.org>
>
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
> On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
> > The arm64 kernel doesn't boot with annotated branches
> > (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
> >
> > Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> > solved the problem. Narrowing down a bit further, I found that
> > physaddr.c is the file that needs to have branch profiling disabled to
> > get the machine to boot.
> >
> > I suspect that it might invoke some ftrace helper very early in the boot
> > process and ftrace is still not enabled(!?).
> >
> > Rather than playing whack-a-mole with individual files, disable branch
> > profiling for the entire arch/arm64 tree, similar to what x86 already
> > does in arch/x86/Kbuild.
> >
> > Cc: stable@vger.kernel.org
> > Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
> > Signed-off-by: Breno Leitao <leitao@debian.org>
>
> I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL
> is unsound in a number of places, so I'd prefer to remove that Fixes tag
> and backport this for all stable trees.
That is fair, thanks for the review.
Should I submit a new version without the fixes tag, or, do you guys do
it while merging the patch?
thanks
--breno
On Tue, Jan 06, 2026 at 06:05:37AM -0800, Breno Leitao wrote:
> On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
> > On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
> > > The arm64 kernel doesn't boot with annotated branches
> > > (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
> > >
> > > Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> > > solved the problem. Narrowing down a bit further, I found that
> > > physaddr.c is the file that needs to have branch profiling disabled to
> > > get the machine to boot.
> > >
> > > I suspect that it might invoke some ftrace helper very early in the boot
> > > process and ftrace is still not enabled(!?).
> > >
> > > Rather than playing whack-a-mole with individual files, disable branch
> > > profiling for the entire arch/arm64 tree, similar to what x86 already
> > > does in arch/x86/Kbuild.
> > >
> > > Cc: stable@vger.kernel.org
> > > Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
> > > Signed-off-by: Breno Leitao <leitao@debian.org>
> >
> > I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL
> > is unsound in a number of places, so I'd prefer to remove that Fixes tag
> > and backport this for all stable trees.
>
> That is fair, thanks for the review.
>
> Should I submit a new version without the fixes tag, or, do you guys do
> it while merging the patch?
I assume that Catalin or Will can handle that when applying (if they
agree with me); no need to respin.
Mark.
On Tue, Jan 06, 2026 at 03:31:16PM +0000, Mark Rutland wrote:
> On Tue, Jan 06, 2026 at 06:05:37AM -0800, Breno Leitao wrote:
> > On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
> > > On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
> > > > The arm64 kernel doesn't boot with annotated branches
> > > > (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
> > > >
> > > > Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> > > > solved the problem. Narrowing down a bit further, I found that
> > > > physaddr.c is the file that needs to have branch profiling disabled to
> > > > get the machine to boot.
> > > >
> > > > I suspect that it might invoke some ftrace helper very early in the boot
> > > > process and ftrace is still not enabled(!?).
> > > >
> > > > Rather than playing whack-a-mole with individual files, disable branch
> > > > profiling for the entire arch/arm64 tree, similar to what x86 already
> > > > does in arch/x86/Kbuild.
> > > >
> > > > Cc: stable@vger.kernel.org
> > > > Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
> > > > Signed-off-by: Breno Leitao <leitao@debian.org>
> > >
> > > I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL
> > > is unsound in a number of places, so I'd prefer to remove that Fixes tag
> > > and backport this for all stable trees.
> >
> > That is fair, thanks for the review.
> >
> > Should I submit a new version without the fixes tag, or, do you guys do
> > it while merging the patch?
>
> I assume that Catalin or Will can handle that when applying (if they
> agree with me); no need to respin.
Thanks Mark.
On Tue, Jan 06, 2026 at 12:21:47PM +0000, Mark Rutland wrote:
> On Tue, Jan 06, 2026 at 02:16:35AM -0800, Breno Leitao wrote:
> > The arm64 kernel doesn't boot with annotated branches
> > (PROFILE_ANNOTATED_BRANCHES) enabled and CONFIG_DEBUG_VIRTUAL together.
> >
> > Bisecting it, I found that disabling branch profiling in arch/arm64/mm
> > solved the problem. Narrowing down a bit further, I found that
> > physaddr.c is the file that needs to have branch profiling disabled to
> > get the machine to boot.
> >
> > I suspect that it might invoke some ftrace helper very early in the boot
> > process and ftrace is still not enabled(!?).
> >
> > Rather than playing whack-a-mole with individual files, disable branch
> > profiling for the entire arch/arm64 tree, similar to what x86 already
> > does in arch/x86/Kbuild.
> >
> > Cc: stable@vger.kernel.org
> > Fixes: ec6d06efb0bac ("arm64: Add support for CONFIG_DEBUG_VIRTUAL")
> > Signed-off-by: Breno Leitao <leitao@debian.org>
>
> I don't think ec6d06efb0bac is to blame here, and CONFIG_DEBUG_VIRTUAL
> is unsound in a number of places, so I'd prefer to remove that Fixes tag
> and backport this for all stable trees.
>
> Regardless, I'm in favour of disabling CONFIG_DEBUG_VIRTUAL widely, so:
>
> Acked-by: Mark Rutland <mark.rutland@arm.com>
Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both
places in my reply.
Mark.
On Tue, 6 Jan 2026 12:24:09 +0000 Mark Rutland <mark.rutland@arm.com> wrote: > Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both > places in my reply. Note, it could still be useful ;-) I've been running my yearly branch profiling build on both my main workstation and my server. I post the results from my server publicly (this is updated every night): https://rostedt.org/branches/current If you check out the branch_annotated file, you can see there's still quite a bit that gets it wrong. Some of these is because of bad assumptions by the developer, others is because the code moved around causing new branches to make later annotated branches go the opposite way. Note, its still slow to download, not only because my internet is 20mbs up, but also because that web server is running on a machine with this profiling still enabled. I'll rebuild the kernel and disable it this weekend. -- Steve
Hello Steven, On Tue, Jan 06, 2026 at 11:11:42AM -0500, Steven Rostedt wrote: > On Tue, 6 Jan 2026 12:24:09 +0000 > Mark Rutland <mark.rutland@arm.com> wrote: > > > > Whoops; s/CONFIG_DEBUG_VIRTUAL/PROFILE_ANNOTATED_BRANCHES/ in both > > places in my reply. > > Note, it could still be useful ;-) > > I've been running my yearly branch profiling build on both my main > workstation and my server. I post the results from my server publicly (this > is updated every night): > > https://rostedt.org/branches/current > > If you check out the branch_annotated file, you can see there's still quite > a bit that gets it wrong. Some of these is because of bad assumptions by > the developer, others is because the code moved around causing new branches > to make later annotated branches go the opposite way. I am starting to look and remove some of these likely/unlikely hint that are 100% wrong on some very sane configuration (arm64 baremetal hosts running a webserver). So far, these are the fixes I have in flight now. * https://lore.kernel.org/all/20260105-dcache-v1-1-f0d904b4a7c2@debian.org/ * https://lore.kernel.org/all/20260106-blk_unlikely-v1-1-90fb556a6776@debian.org/ * https://lore.kernel.org/all/20260105-annotated_idle-v1-1-10ddf0771b58@debian.org/
On Wed, 7 Jan 2026 01:37:36 -0800 Breno Leitao <leitao@debian.org> wrote: > I am starting to look and remove some of these likely/unlikely hint that > are 100% wrong on some very sane configuration (arm64 baremetal hosts > running a webserver). > > So far, these are the fixes I have in flight now. > > * https://lore.kernel.org/all/20260105-dcache-v1-1-f0d904b4a7c2@debian.org/ > * https://lore.kernel.org/all/20260106-blk_unlikely-v1-1-90fb556a6776@debian.org/ > * https://lore.kernel.org/all/20260105-annotated_idle-v1-1-10ddf0771b58@debian.org/ Nice. Thanks for letting me know. I'm happy to see that someone besides me is using the annotated branch profiler to fix real issues in the kernel. I would love it if more people did the same. Anyway, please advertise this as much as possible. If it shows that it is helping to fix real bugs, maybe it would attract the interest of more developers, and more importantly, it keeps the tooling in place. I'm constantly fighting to keep it from being removed from the kernel :-p -- Steve
© 2016 - 2026 Red Hat, Inc.