[PATCH v3 0/3] KHO: kfence + KHO memory corruption fix

Pasha Tatashin posted 3 patches 3 months, 2 weeks ago
include/linux/gfp.h              |  3 ++
kernel/Kconfig.kexec             |  9 ++++
kernel/Makefile                  |  1 +
kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
kernel/kexec_handover_debug.c    | 25 +++++++++++
kernel/kexec_handover_internal.h | 16 +++++++
6 files changed, 100 insertions(+), 26 deletions(-)
create mode 100644 kernel/kexec_handover_debug.c
create mode 100644 kernel/kexec_handover_internal.h
[PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Pasha Tatashin 3 months, 2 weeks ago
This series fixes a memory corruption bug in KHO that occurs when KFENCE
is enabled.

The root cause is that KHO metadata, allocated via kzalloc(), can be
randomly serviced by kfence_alloc(). When a kernel boots via KHO, the
early memblock allocator is restricted to a "scratch area". This forces
the KFENCE pool to be allocated within this scratch area, creating a
conflict. If KHO metadata is subsequently placed in this pool, it gets
corrupted during the next kexec operation.

Patch 1/3 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG)
that adds checks to detect and fail any operation that attempts to place
KHO metadata or preserved memory within the scratch area. This serves as
a validation and diagnostic tool to confirm the problem without
affecting production builds.

Patch 2/3 Increases bitmap to PAGE_SIZE, so buddy allocator can be used.

Patch 3/3 Provides the fix by modifying KHO to allocate its metadata
directly from the buddy allocator instead of slab. This bypasses the
KFENCE interception entirely.

Pasha Tatashin (3):
  liveupdate: kho: warn and fail on metadata or preserved memory in
    scratch area
  liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE
  liveupdate: kho: allocate metadata directly from the buddy allocator

 include/linux/gfp.h              |  3 ++
 kernel/Kconfig.kexec             |  9 ++++
 kernel/Makefile                  |  1 +
 kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
 kernel/kexec_handover_debug.c    | 25 +++++++++++
 kernel/kexec_handover_internal.h | 16 +++++++
 6 files changed, 100 insertions(+), 26 deletions(-)
 create mode 100644 kernel/kexec_handover_debug.c
 create mode 100644 kernel/kexec_handover_internal.h


base-commit: 6548d364a3e850326831799d7e3ea2d7bb97ba08
-- 
2.51.0.869.ge66316f041-goog
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Mike Rapoport 3 months, 2 weeks ago
On Mon, Oct 20, 2025 at 08:08:49PM -0400, Pasha Tatashin wrote:
> This series fixes a memory corruption bug in KHO that occurs when KFENCE
> is enabled.
> 
> The root cause is that KHO metadata, allocated via kzalloc(), can be
> randomly serviced by kfence_alloc(). When a kernel boots via KHO, the
> early memblock allocator is restricted to a "scratch area". This forces
> the KFENCE pool to be allocated within this scratch area, creating a
> conflict. If KHO metadata is subsequently placed in this pool, it gets
> corrupted during the next kexec operation.
> 
> Patch 1/3 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG)
> that adds checks to detect and fail any operation that attempts to place
> KHO metadata or preserved memory within the scratch area. This serves as
> a validation and diagnostic tool to confirm the problem without
> affecting production builds.
> 
> Patch 2/3 Increases bitmap to PAGE_SIZE, so buddy allocator can be used.
> 
> Patch 3/3 Provides the fix by modifying KHO to allocate its metadata
> directly from the buddy allocator instead of slab. This bypasses the
> KFENCE interception entirely.
> 
> Pasha Tatashin (3):
>   liveupdate: kho: warn and fail on metadata or preserved memory in
>     scratch area
>   liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE
>   liveupdate: kho: allocate metadata directly from the buddy allocator

With liveupdate: dropped from the subjects

Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
 
>  include/linux/gfp.h              |  3 ++
>  kernel/Kconfig.kexec             |  9 ++++
>  kernel/Makefile                  |  1 +
>  kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
>  kernel/kexec_handover_debug.c    | 25 +++++++++++
>  kernel/kexec_handover_internal.h | 16 +++++++
>  6 files changed, 100 insertions(+), 26 deletions(-)
>  create mode 100644 kernel/kexec_handover_debug.c
>  create mode 100644 kernel/kexec_handover_internal.h
> 
> 
> base-commit: 6548d364a3e850326831799d7e3ea2d7bb97ba08
> -- 
> 2.51.0.869.ge66316f041-goog
> 

-- 
Sincerely yours,
Mike.
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Pasha Tatashin 3 months, 2 weeks ago
On Tue, Oct 21, 2025 at 2:01 AM Mike Rapoport <rppt@kernel.org> wrote:
>
> On Mon, Oct 20, 2025 at 08:08:49PM -0400, Pasha Tatashin wrote:
> > This series fixes a memory corruption bug in KHO that occurs when KFENCE
> > is enabled.
> >
> > The root cause is that KHO metadata, allocated via kzalloc(), can be
> > randomly serviced by kfence_alloc(). When a kernel boots via KHO, the
> > early memblock allocator is restricted to a "scratch area". This forces
> > the KFENCE pool to be allocated within this scratch area, creating a
> > conflict. If KHO metadata is subsequently placed in this pool, it gets
> > corrupted during the next kexec operation.
> >
> > Patch 1/3 introduces a debug-only feature (CONFIG_KEXEC_HANDOVER_DEBUG)
> > that adds checks to detect and fail any operation that attempts to place
> > KHO metadata or preserved memory within the scratch area. This serves as
> > a validation and diagnostic tool to confirm the problem without
> > affecting production builds.
> >
> > Patch 2/3 Increases bitmap to PAGE_SIZE, so buddy allocator can be used.
> >
> > Patch 3/3 Provides the fix by modifying KHO to allocate its metadata
> > directly from the buddy allocator instead of slab. This bypasses the
> > KFENCE interception entirely.
> >
> > Pasha Tatashin (3):
> >   liveupdate: kho: warn and fail on metadata or preserved memory in
> >     scratch area
> >   liveupdate: kho: Increase metadata bitmap size to PAGE_SIZE
> >   liveupdate: kho: allocate metadata directly from the buddy allocator
>
> With liveupdate: dropped from the subjects

I noticed "liveupdate: " subject prefix left over only after sending
these patches. Andrew, would you like me to resend them, or could you
remove the prefix from these patches?

> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
>
> >  include/linux/gfp.h              |  3 ++
> >  kernel/Kconfig.kexec             |  9 ++++
> >  kernel/Makefile                  |  1 +
> >  kernel/kexec_handover.c          | 72 ++++++++++++++++++++------------
> >  kernel/kexec_handover_debug.c    | 25 +++++++++++
> >  kernel/kexec_handover_internal.h | 16 +++++++
> >  6 files changed, 100 insertions(+), 26 deletions(-)
> >  create mode 100644 kernel/kexec_handover_debug.c
> >  create mode 100644 kernel/kexec_handover_internal.h
> >
> >
> > base-commit: 6548d364a3e850326831799d7e3ea2d7bb97ba08
> > --
> > 2.51.0.869.ge66316f041-goog
> >
>
> --
> Sincerely yours,
> Mike.
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Andrew Morton 3 months, 2 weeks ago
On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <pasha.tatashin@soleen.com> wrote:

> > With liveupdate: dropped from the subjects
> 
> I noticed "liveupdate: " subject prefix left over only after sending
> these patches. Andrew, would you like me to resend them, or could you
> remove the prefix from these patches?

No problem.

What should we do about -stable kernels?

It doesn't seem worthwhile to backport a 3-patch series for a pretty
obscure bug.  Perhaps we could merge a patch which disables this
combination in Kconfig, as a 6.18-rcX hotfix with a cc:stable.

Then for 6.19-rc1 we add this series and a fourth patch which undoes
that Kconfig change?
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Pasha Tatashin 3 months, 2 weeks ago
On Tue, Oct 21, 2025 at 4:53 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <pasha.tatashin@soleen.com> wrote:
>
> > > With liveupdate: dropped from the subjects
> >
> > I noticed "liveupdate: " subject prefix left over only after sending
> > these patches. Andrew, would you like me to resend them, or could you
> > remove the prefix from these patches?
>
> No problem.
>
> What should we do about -stable kernels?
>
> It doesn't seem worthwhile to backport a 3-patch series for a pretty
> obscure bug.  Perhaps we could merge a patch which disables this

We are using KHO and have had obscure crashes due to this memory
corruption, with stacks all over the place. I would prefer this fix to
be properly backported to stable so we can also automatically consume
it once we switch to the upstream KHO. I do not think disabling kfence
in the Google fleet to resolve this problem would work for us, so if
it is not going to be part of stable, we would have to backport it
manually anyway.

Thanks,
Pasha

> combination in Kconfig, as a 6.18-rcX hotfix with a cc:stable.
>
> Then for 6.19-rc1 we add this series and a fourth patch which undoes
> that Kconfig change?
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Andrew Morton 3 months, 2 weeks ago
On Tue, 21 Oct 2025 20:15:04 -0400 Pasha Tatashin <pasha.tatashin@soleen.com> wrote:

> On Tue, Oct 21, 2025 at 4:53 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <pasha.tatashin@soleen.com> wrote:
> >
> > > > With liveupdate: dropped from the subjects
> > >
> > > I noticed "liveupdate: " subject prefix left over only after sending
> > > these patches. Andrew, would you like me to resend them, or could you
> > > remove the prefix from these patches?
> >
> > No problem.
> >
> > What should we do about -stable kernels?
> >
> > It doesn't seem worthwhile to backport a 3-patch series for a pretty
> > obscure bug.  Perhaps we could merge a patch which disables this
> 
> We are using KHO and have had obscure crashes due to this memory
> corruption, with stacks all over the place. I would prefer this fix to
> be properly backported to stable so we can also automatically consume
> it once we switch to the upstream KHO.

Oh.

I added this important info to the [0/N] changelog, added

Fixes: fc33e4b44b27 ("kexec: enable KHO support for memory preservation")
Cc: <stable@vger.kernel.org>

to all three and moved this into mm.git's mm-hotfixes branch.

Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Mike Rapoport 3 months, 2 weeks ago
On Tue, Oct 21, 2025 at 08:15:04PM -0400, Pasha Tatashin wrote:
> On Tue, Oct 21, 2025 at 4:53 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Tue, 21 Oct 2025 12:04:47 -0400 Pasha Tatashin <pasha.tatashin@soleen.com> wrote:
> >
> > > > With liveupdate: dropped from the subjects
> > >
> > > I noticed "liveupdate: " subject prefix left over only after sending
> > > these patches. Andrew, would you like me to resend them, or could you
> > > remove the prefix from these patches?
> >
> > No problem.
> >
> > What should we do about -stable kernels?
> >
> > It doesn't seem worthwhile to backport a 3-patch series for a pretty
> > obscure bug.  Perhaps we could merge a patch which disables this
> 
> We are using KHO and have had obscure crashes due to this memory
> corruption, with stacks all over the place. I would prefer this fix to
> be properly backported to stable so we can also automatically consume
> it once we switch to the upstream KHO. I do not think disabling kfence
> in the Google fleet to resolve this problem would work for us, so if
> it is not going to be part of stable, we would have to backport it
> manually anyway.

The backport to stable is only relevant to 6.17 that's going to be EOL soon
anyway. Do you really think it's worth the effort?
 
> Thanks,
> Pasha
> 
> > combination in Kconfig, as a 6.18-rcX hotfix with a cc:stable.
> >
> > Then for 6.19-rc1 we add this series and a fourth patch which undoes
> > that Kconfig change?

-- 
Sincerely yours,
Mike.
Re: [PATCH v3 0/3] KHO: kfence + KHO memory corruption fix
Posted by Andrew Morton 3 months, 2 weeks ago
On Wed, 22 Oct 2025 08:48:34 +0300 Mike Rapoport <rppt@kernel.org> wrote:

> > We are using KHO and have had obscure crashes due to this memory
> > corruption, with stacks all over the place. I would prefer this fix to
> > be properly backported to stable so we can also automatically consume
> > it once we switch to the upstream KHO. I do not think disabling kfence
> > in the Google fleet to resolve this problem would work for us, so if
> > it is not going to be part of stable, we would have to backport it
> > manually anyway.
> 
> The backport to stable is only relevant to 6.17 that's going to be EOL soon
> anyway. Do you really think it's worth the effort?

If some organization is basing their next kernel on 6.17 then they'd
like it.

Do we assume that all organizations follow the LTS schedule?  I haven't
been doing that.