We're including maintainers from all subsystems that currently is or
will be expected to participate in live update to ensure alignment on
the path forward for compatibility.

Currently, Live Update Orchestrator (LUO) and its file handlers rely on
monolithic compatibility strings (such as "luo-v5" and "memfd-v1") to
validate ABI compatibility across kexec live updates. Any modification
to serialized structures requires bumping the version string, which
strictly breaks compatibility between adjacent kernels even when changes
are additive, backwards-compatible, or optional.

This RFC series transitions LUO and subsystem file handlers to use
granular feature bitmasks instead of compatibility strings. We replace
the compatibility string with a header structure that includes some
reserved space to define the features that are included after the
feature.

We include 3 bits of data for each feature: supported, active, and
required. When the supported bit is set in the next kernel, the previous
kernel can serialize the feature and pass it to the next kernel. When
the active bit is set, it means the previous occurred and the data for
that feature is valid. When the required bit is set, it means that the
next kernel must have the associated supported bit in order to be
compatible.

If a feature is required in that it cannot live update to a kernel that
does not support the feature, it is expected that the feature not be
active or required initially. This allows for an intermediate upgrade
path.

Overview of Changes:
1. Define feature header and migrate luo_ser (Patch 1):
  - Introudce the feature header with the supported, active, and
    required bits.
  - Replace the luo compatibility string with this new header.
  - Validate the preserved size is at least the size of the header. This
    breaks compatibility with the old way.

2. Export feature header information to vmlinux (Patch 2):
  - Adds a .liveupdate_features section in vmlinux containing
  - Adds helper macros to export feature information for a given
    subsystem
  - Export luo's feature information

3. Memfd Handler Migration (Patch 3):
  - Migrate memfd to use the feature header instead of compatibility
    strings

Logan Odell (3):
  luo: Move to feature flags instead of compatibility strings
  luo: Export feature support to vmlinux section
  luo: memfd: Move to feature flags instead of compatibility strings

 include/asm-generic/vmlinux.lds.h |  12 ++++
 include/linux/kho/abi/luo.h       | 114 +++++++++++++++++++++++++-----
 include/linux/kho/abi/memfd.h     |  41 +++++++----
 include/linux/liveupdate.h        |  22 ++++--
 kernel/liveupdate/luo_core.c      |  60 ++++++++++++----
 kernel/liveupdate/luo_file.c      |  28 ++++----
 kernel/liveupdate/luo_flb.c       |   2 +-
 lib/tests/liveupdate.c            |   2 +-
 mm/memfd_luo.c                    |  40 ++++++++---
 9 files changed, 246 insertions(+), 75 deletions(-)

-- 
2.55.0.979.g7e5102b832-goog

On Wed, Sep 02, 2026 at 07:34:49PM -0700, Logan Odell wrote:
> We're including maintainers from all subsystems that currently is or
> will be expected to participate in live update to ensure alignment on
> the path forward for compatibility.
> 
> Currently, Live Update Orchestrator (LUO) and its file handlers rely on
> monolithic compatibility strings (such as "luo-v5" and "memfd-v1") to
> validate ABI compatibility across kexec live updates. Any modification
> to serialized structures requires bumping the version string, which
> strictly breaks compatibility between adjacent kernels even when changes
> are additive, backwards-compatible, or optional.

That was the intention, I aruged strongly that upstream does not want
to maintain a CSP matrix of endless kernel version combinations. That
is far too much work to push on maintainers. Upstream would do much
less, maybe only same-version, depending.

> This RFC series transitions LUO and subsystem file handlers to use
> granular feature bitmasks instead of compatibility strings. We replace
> the compatibility string with a header structure that includes some
> reserved space to define the features that are included after the
> feature.

The compatability string is only a small part of it, you also need a
serializing ABI that can handle some random mixmash of these
features. Ie the various TLV schemes that were all proposed. What is
the plan here?

Jason

On 2026-09-04 01:00 PM, Jason Gunthorpe wrote:
> On Wed, Sep 02, 2026 at 07:34:49PM -0700, Logan Odell wrote:
> > We're including maintainers from all subsystems that currently is or
> > will be expected to participate in live update to ensure alignment on
> > the path forward for compatibility.
> > 
> > Currently, Live Update Orchestrator (LUO) and its file handlers rely on
> > monolithic compatibility strings (such as "luo-v5" and "memfd-v1") to
> > validate ABI compatibility across kexec live updates. Any modification
> > to serialized structures requires bumping the version string, which
> > strictly breaks compatibility between adjacent kernels even when changes
> > are additive, backwards-compatible, or optional.
> 
> That was the intention, I aruged strongly that upstream does not want
> to maintain a CSP matrix of endless kernel version combinations. That
> is far too much work to push on maintainers. Upstream would do much
> less, maybe only same-version, depending.

We received basically the opposite stance from Sean regarding the KVM
LUO ABI:

  https://lore.kernel.org/kvm/aoSCCTTBn9D5hqzk@google.com/

So we're trying to use this series to get some alignment across LUO
ABIs.

> > This RFC series transitions LUO and subsystem file handlers to use
> > granular feature bitmasks instead of compatibility strings. We replace
> > the compatibility string with a header structure that includes some
> > reserved space to define the features that are included after the
> > feature.
> 
> The compatability string is only a small part of it, you also need a
> serializing ABI that can handle some random mixmash of these
> features. Ie the various TLV schemes that were all proposed. What is
> the plan here?

The proposal here (which is inspired by the KVM UAPI) is to ensure every
LUO ABI struct has 2 properties:

 1. A field to encode options/features (e.g. u64 flags).
 2. A way way to grow without breaking backward compatibility (e.g. so
    we can add new fields).

Each flag can mean whatever it needs to. e.g. It can indicate the
precence of one or more fields (i.e. new fields in the struct), or it
can mean a field now has a different meaning (i.e. union in the struct).

This would enable adding support for new features without breaking
backward compatibility. Downstream users would have to ensure their
kernel does not start using a new feature while it can still rollback to
a version that does not support the new feature.

On Fri, Sep 04, 2026 at 10:24:21PM +0000, David Matlack wrote:

> The proposal here (which is inspired by the KVM UAPI) is to ensure every
> LUO ABI struct has 2 properties:
> 
>  1. A field to encode options/features (e.g. u64 flags).
>  2. A way way to grow without breaking backward compatibility (e.g. so
>     we can add new fields).
> 
> Each flag can mean whatever it needs to. e.g. It can indicate the
> precence of one or more fields (i.e. new fields in the struct), or it
> can mean a field now has a different meaning (i.e. union in the struct).
> 
> This would enable adding support for new features without breaking
> backward compatibility. Downstream users would have to ensure their
> kernel does not start using a new feature while it can still rollback to
> a version that does not support the new feature.

This was never the biggest problem. The main issue was the functional
behaviors of the kernel that cannot be represented simply as data in a
struct with some flag bits.

Like for instance kernel A supports memfd folio sizes far larger than
kernel B because we fixed MAX_ORDER. You can't fix that just with
simplistic flags.

Jason

On Fri, Sep 04, 2026, Jason Gunthorpe wrote:
> On Fri, Sep 04, 2026 at 10:24:21PM +0000, David Matlack wrote:
> 
> > The proposal here (which is inspired by the KVM UAPI) is to ensure every
> > LUO ABI struct has 2 properties:
> > 
> >  1. A field to encode options/features (e.g. u64 flags).
> >  2. A way way to grow without breaking backward compatibility (e.g. so
> >     we can add new fields).
> > 
> > Each flag can mean whatever it needs to. e.g. It can indicate the
> > precence of one or more fields (i.e. new fields in the struct), or it
> > can mean a field now has a different meaning (i.e. union in the struct).
> > 
> > This would enable adding support for new features without breaking
> > backward compatibility. Downstream users would have to ensure their
> > kernel does not start using a new feature while it can still rollback to
> > a version that does not support the new feature.
> 
> This was never the biggest problem. The main issue was the functional
> behaviors of the kernel that cannot be represented simply as data in a
> struct with some flag bits.
> 
> Like for instance kernel A supports memfd folio sizes far larger than
> kernel B because we fixed MAX_ORDER. You can't fix that just with
> simplistic flags.

Can you elaborate on why the folio sizes matter?  Honest question, because I don't
understand why the serialization format wouldn't express things as "N contiguous
pages starting at PFN X".  Then the implementation would rebuild its folios as
appropriate.

I could see things like HugeTLB not working if someone booted the kernel with
support for only 1GiB pages and then tried to feed it payload with sub-1GiB ranges.
But to me, those sorts of things fall into the "well yeah, don't do that" category.

On Wed, Sep 09, 2026 at 05:58:28PM -0700, Sean Christopherson wrote:
> On Fri, Sep 04, 2026, Jason Gunthorpe wrote:
> > On Fri, Sep 04, 2026 at 10:24:21PM +0000, David Matlack wrote:
> > 
> > > The proposal here (which is inspired by the KVM UAPI) is to ensure every
> > > LUO ABI struct has 2 properties:
> > > 
> > >  1. A field to encode options/features (e.g. u64 flags).
> > >  2. A way way to grow without breaking backward compatibility (e.g. so
> > >     we can add new fields).
> > > 
> > > Each flag can mean whatever it needs to. e.g. It can indicate the
> > > precence of one or more fields (i.e. new fields in the struct), or it
> > > can mean a field now has a different meaning (i.e. union in the struct).
> > > 
> > > This would enable adding support for new features without breaking
> > > backward compatibility. Downstream users would have to ensure their
> > > kernel does not start using a new feature while it can still rollback to
> > > a version that does not support the new feature.
> > 
> > This was never the biggest problem. The main issue was the functional
> > behaviors of the kernel that cannot be represented simply as data in a
> > struct with some flag bits.
> > 
> > Like for instance kernel A supports memfd folio sizes far larger than
> > kernel B because we fixed MAX_ORDER. You can't fix that just with
> > simplistic flags.
> 
> Can you elaborate on why the folio sizes matter?  Honest question, because I don't
> understand why the serialization format wouldn't express things as "N contiguous
> pages starting at PFN X".  Then the implementation would rebuild its folios as
> appropriate.

That's an idyllic view, yes, but my point is (IIRC) we didn't do
exactly that for memfd.

Sometimes you can do more and more work to try and be more and more
general but this is *alot* of work and even then eventually hits
problematic limits. Like what do you do with the sealing flags? That's
ABI breaking if the successor does not support them, and downgrades
make exactly that possible.

A CSPish user can do things like patch the new sealing flag into their
current kernel (while preventing userspace from using it), ensure
everything is updated to that, then jump ahead to a newer kernel and
enjoy the new flag with full downgrade support. There is so much more
control on their part that makes the problem far more managably simple
that upstream does not get to have.

This is why I think the very idea we can support any version pair is
too much to ask for. We should focus on supporting a small set of
version pairs and not making it too invasive or hard in the kernel or
on the maintainers.

Thus live update within a stable branch only is my proposal for
upstream support.

If it really succeeds at that and it becomes very popular, then let's
discuss upstreaming doing additional version combinations.

> I could see things like HugeTLB not working if someone booted the kernel with
> support for only 1GiB pages and then tried to feed it payload with sub-1GiB ranges.
> But to me, those sorts of things fall into the "well yeah, don't do that" category.

Okay, how about worse, todays kernel has hugetlbfs and there are
patches around to luo serialize that. Lots and lots of talks about a
post-hugetlbfs world out there.

Do we want to constrain what is possible to ensure we accomodate this
hugetlbfs serialization? I vote no. 

Do we want to reject the hugetlbfs serialization until we have a year
of debate outlining every possible ABI scenario? I also vote no.

Should we make a downgrade round trip a downstream problem? I think
so!

Jason

On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> On Wed, Sep 09, 2026 at 05:58:28PM -0700, Sean Christopherson wrote:
> > On Fri, Sep 04, 2026, Jason Gunthorpe wrote:
> > > On Fri, Sep 04, 2026 at 10:24:21PM +0000, David Matlack wrote:
> > > 
> > > > The proposal here (which is inspired by the KVM UAPI) is to ensure every
> > > > LUO ABI struct has 2 properties:
> > > > 
> > > >  1. A field to encode options/features (e.g. u64 flags).
> > > >  2. A way way to grow without breaking backward compatibility (e.g. so
> > > >     we can add new fields).
> > > > 
> > > > Each flag can mean whatever it needs to. e.g. It can indicate the
> > > > precence of one or more fields (i.e. new fields in the struct), or it
> > > > can mean a field now has a different meaning (i.e. union in the struct).
> > > > 
> > > > This would enable adding support for new features without breaking
> > > > backward compatibility. Downstream users would have to ensure their
> > > > kernel does not start using a new feature while it can still rollback to
> > > > a version that does not support the new feature.
> > > 
> > > This was never the biggest problem. The main issue was the functional
> > > behaviors of the kernel that cannot be represented simply as data in a
> > > struct with some flag bits.
> > > 
> > > Like for instance kernel A supports memfd folio sizes far larger than
> > > kernel B because we fixed MAX_ORDER. You can't fix that just with
> > > simplistic flags.
> > 
> > Can you elaborate on why the folio sizes matter?  Honest question, because I don't
> > understand why the serialization format wouldn't express things as "N contiguous
> > pages starting at PFN X".  Then the implementation would rebuild its folios as
> > appropriate.
> 
> That's an idyllic view, yes, but my point is (IIRC) we didn't do
> exactly that for memfd.

Well, y'all screwed up then.  I don't see why past mistakes should force other
subsystems to support a flawed implementation.  Learn from the mistakes, add v2
of serialization for memfd, and move on.

> Sometimes you can do more and more work to try and be more and more
> general but this is *alot* of work and even then eventually hits
> problematic limits. Like what do you do with the sealing flags? That's
> ABI breaking if the successor does not support them, and downgrades
> make exactly that possible.
>
> A CSPish user can do things like patch the new sealing flag into their
> current kernel (while preventing userspace from using it), ensure
> everything is updated to that, then jump ahead to a newer kernel and
> enjoy the new flag with full downgrade support. There is so much more
> control on their part that makes the problem far more managably simple
> that upstream does not get to have.

I guess maybe we have a different definition of ABI? 

I'm not saying that upstream has to be 100% forwards and backwards compatible.
I'm saying the serialization payload itself should communicate what features are
effectively required.  I.e. *if* there are incompatibilities, they should be
naturally expressed in the serialization format, not communicated out-of-band
through magic numbers.

The scenario you describe fits exactly with what I am proposing.  Until something
actually starts using the new sealing flag, the CSP can downgrade to older kernels
at will.  And if the user cares about downgrading, then they need to prevent the
flag from being used until the new kernel is rollback-safe and deployed to enough
hosts to prevent stockout.

> This is why I think the very idea we can support any version pair is
> too much to ask for. We should focus on supporting a small set of
> version pairs and not making it too invasive or hard in the kernel or
> on the maintainers.
>
> Thus live update within a stable branch only is my proposal for
> upstream support.
> 
> If it really succeeds at that and it becomes very popular, then let's
> discuss upstreaming doing additional version combinations.

Why on earth would we have version numbers in the first place?  IMO, monotically
increasing version numbers are flat out the worst way to communicate features.

I am completely against supporting any scheme that relies on magic version numbers.
It creates problems where none need exist, and checking for compatibility can't be
sanely done in a programmatic way, because by definition it relies on magic numbers.

E.g. if the ABI for a given component hasn't changed, why should anyone care if
the overall "version" of the kernel is ahead or behind by N kernels?

> > I could see things like HugeTLB not working if someone booted the kernel with
> > support for only 1GiB pages and then tried to feed it payload with sub-1GiB ranges.
> > But to me, those sorts of things fall into the "well yeah, don't do that" category.
> 
> Okay, how about worse, todays kernel has hugetlbfs and there are
> patches around to luo serialize that. Lots and lots of talks about a
> post-hugetlbfs world out there.

And?  Adding a compatibility layer to a future kernel so that it understands an
incoming HugeTLBFS payload should be trivial.  I can totally see not wanting to
support serializing a post-HugeTBLFS kernel's memory representation into the "old"
format, though even that probably wouldn't be all that difficult.

> Do we want to constrain what is possible to ensure we accomodate this
> hugetlbfs serialization? I vote no.

In what way is providing strong ABI guarantees for individual components
constraining HugeTBLFS serialization?

> Do we want to reject the hugetlbfs serialization until we have a year
> of debate outlining every possible ABI scenario? I also vote no.

That's a bit of a strawman argument.  Is designing a forward-looking ABI easy?
No, but IMO "a year" is a massive exaggeration of the effort required to come up
with a scheme that can survive a variety of plausible upgrade/downgrade scenarios.

And again, I'm not saying we have to support infinite compatibility.  If some
future kernel drops HugeTLBFS, and we decide not to provide a shim to support
downgrading (which IMO is totally reasonable), then it's on the user to understand
that moving to that new kernel is a one-way street.  Given that dropping something
like HugeTLBFS would require significant changes in the software stack, I think
it's perfectly fine to put the burden of understanding the implications on the end
user (though realistically, there would be a copious amount of documentation and
deprecation warnings).

> Should we make a downgrade round trip a downstream problem? I think
> so!

Hard NAK.  There will inevitably be boundaries that cannot be crossed, but I am
not at all ok punting on downgrades.  To me, that's basically saying "we want to
add just enough support upstream so that it's not too painful to carry full support
out-of-tree".  That completely goes against the spirit of open source and upstream
Linux, and I want no part of it.

On 9/10/26 17:35, Sean Christopherson wrote:
> On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
>> Sometimes you can do more and more work to try and be more and more
>> general but this is *alot* of work and even then eventually hits
>> problematic limits. Like what do you do with the sealing flags? That's
>> ABI breaking if the successor does not support them, and downgrades
>> make exactly that possible.
>>
>> A CSPish user can do things like patch the new sealing flag into their
>> current kernel (while preventing userspace from using it), ensure
>> everything is updated to that, then jump ahead to a newer kernel and
>> enjoy the new flag with full downgrade support. There is so much more
>> control on their part that makes the problem far more managably simple
>> that upstream does not get to have.
>
> I guess maybe we have a different definition of ABI? I'm not saying
> that upstream has to be 100% forwards and backwards compatible. I'm
> saying the serialization payload itself should communicate what
> features are effectively required.
I strongly agree with Sean on this, like really really agree.

All you need is serializing *actions*.  Make the destination a small 
interpreter not something that read structs.  memfd/guest_memfd is 
already created by a bunch of actions, which are syscalls, so it 
shouldn't be hard to either come up with the actions or parse them in 
the destination.

An example matching (going by memory) what is now in place for memfd:

- memfd_create(name[], flags)
- memfd_map(folios[], index)
- memfd_finish(seals, pos, size, mode)

So:

#define MEMFD_LUO_CREATE 0
#define MEMFD_LUO_MAP 1
#define MEMFD_LUO_FINISH 2
struct memfd_luo_op {
	/* 0 = end */
	u32 size;
	u32 op;
	union {
		struct {
			u32 flags;
			char name[];
		} memfd_luo_create;
		struct {
			u32 flags;
		} memfd_luo_secret;
		struct {
			u64 i_size;
			u64 f_pos;
			u32 f_seals;
			u32 i_mode;
		} memfd_luo_finish;
		struct {
			u64 index;
			struct memfd_luo_folio src_folios[]; // whatever
		} memfd_luo_map;
	};
} __aligned(8);

You write almost everything at prepare, just ensure there is room for 
finish and write that on freeze.

Want to move secret memfds?  Sure they're different in underlying 
implementation but they can share LUO serialization format almost 
entirely.  Make it a new op instead of create.. you have 4 billion 
possible ops, adding them isn't quite free but not too expensive either.

In fact memfd is the easy case, almost always you'll have a more 
complicated initialization sequence and a huge explosion of 
possibilities, but the good thing is that the kernel *already* has to 
initialize its data structures from actions.  We're not quite 
serializing syscalls but pretty close, in fact for KVM a lot of code 
could be shared between ioctls and LUO receiving side.

It doesn't have to match exactly userspace, for example you wouldn't 
really need to transmit MFD_ALLOW_SEALING because it's implicit in the 
seals you transmit.  That said, taking inspiration doesn't hurt; just 
remember to *always* validate unknown flags.

>> If it really succeeds at that and it becomes very popular, then let's
>> discuss upstreaming doing additional version combinations.
> 
> Why on earth would we have version numbers in the first place?  IMO, monotically
> increasing version numbers are flat out the worst way to communicate features.

This, too.  KVM has been at API version 12 since 2007.  It is not 
userspace compatible with 2007 vintage QEMU, because a couple 
misfeatures were removed after 10 years or so of waiting, so I guess 
technically it would be 15 or 16, but it doesn't matter because no one 
checks KVM_API_VERSION.  If a ioctl works it works, if it doesn't you 
get a much better message than "KVM API version mismatch".

>> Okay, how about worse, todays kernel has hugetlbfs and there are
>> patches around to luo serialize that. Lots and lots of talks about a
>> post-hugetlbfs world out there.
> 
> And?  Adding a compatibility layer to a future kernel so that it understands an
> incoming HugeTLBFS payload should be trivial.  I can totally see not wanting to
> support serializing a post-HugeTBLFS kernel's memory representation into the "old"
> format, though even that probably wouldn't be all that difficult.

Yeah, hugetlbfs is an implementation detail *of the destination* not the 
source.  The destination somehow needs to take the 1GB area and donate 
it to hugetlbfs.  That's not the source's problem.  All the 
source->destination ABI contains is MFD_HUGETLB and MFD_HUGE_*, which 
promise to the destination a certain alignment of all map requests.
>> Do we want to reject the hugetlbfs serialization until we have a year
>> of debate outlining every possible ABI scenario? I also vote no.
> 
> That's a bit of a strawman argument.  Is designing a forward-looking ABI easy?

No, but it's also not *that* hard if you have a half-decent userspace ABI.

> Hard NAK.  There will inevitably be boundaries that cannot be crossed, but I am
> not at all ok punting on downgrades.  To me, that's basically saying "we want to
> add just enough support upstream so that it's not too painful to carry full support
> out-of-tree".  That completely goes against the spirit of open source and upstream
> Linux, and I want no part of it.
100%.  And I'll add, what happened to "we don't break userspace"?  This 
does the intentional opposite in the hope that no one cares about using 
this feature upstream.  Which in the long term hurts downstream forks as 
much as upstream.

Paolo

On Thu, Sep 10, 2026 at 08:35:54AM -0700, Sean Christopherson wrote:

> I guess maybe we have a different definition of ABI? 
> 
> I'm not saying that upstream has to be 100% forwards and backwards compatible.
> I'm saying the serialization payload itself should communicate what features are
> effectively required.  I.e. *if* there are incompatibilities, they should be
> naturally expressed in the serialization format, not communicated out-of-band
> through magic numbers.

The ABI strings were introduced specifically because extension makes
the actual compatibility indeterminate by userspace. 

Keep in mind the actual goal here. Someone has kernel A and they need
to blind kexec into kernel B and NOT have the machine explode, or all
the VMs sitting on it lost.

Meaning you must have a way to determine before the kexec if kernel A
is producing something B will *accept*. Accept is not "parse and fail
with EOPNOTSUPP" like most uapi schems. Aceept means bring in and
actually fully support and use.

So how do you solve this problem? You MUST declare in some kind of
manifest exactly what ABIs are supported, in some way.

> The scenario you describe fits exactly with what I am proposing.  

It does not. What is really wanted here is to tell kernel A to only
support ABI 1 for memfd and so kernel A will fail to serialize if it
cannot do it because a newer seal flag was used.

We do not want to succeed to serialize then fail to accept after
kexec and have a dead machine.

This is not anything like a normal uapi compatability problem.

> actually starts using the new sealing flag, the CSP can downgrade to
> older kernels at will.  And if the user cares about downgrading,
> then they need to prevent the flag from being used until the new
> kernel is rollback-safe and deployed to enough hosts to prevent
> stockout.

Yeah, CSP broadly has to do exactly this across a wide range of
topics. It is a further reason why this feature is not exactly usable
by a "mainstream" user :\

> > This is why I think the very idea we can support any version pair is
> > too much to ask for. We should focus on supporting a small set of
> > version pairs and not making it too invasive or hard in the kernel or
> > on the maintainers.
> >
> > Thus live update within a stable branch only is my proposal for
> > upstream support.
> > 
> > If it really succeeds at that and it becomes very popular, then let's
> > discuss upstreaming doing additional version combinations.
>
> Why on earth would we have version numbers in the first place?  IMO, monotically
> increasing version numbers are flat out the worst way to communicate
> features.

As above, discoverablility is a key requirement.

Each version number is a very specific upstream defined ABI, in the
sense if kernel A emits version X and kernel B accepts version X then
kexec *must* work.

You can make some manifest in other more complicated ways, but I'm
deeply skeptical that is really going to bring any value. It feels
like it is just increasing the testing matrix :\
 
> > > I could see things like HugeTLB not working if someone booted the kernel with
> > > support for only 1GiB pages and then tried to feed it payload with sub-1GiB ranges.
> > > But to me, those sorts of things fall into the "well yeah, don't do that" category.
> > 
> > Okay, how about worse, todays kernel has hugetlbfs and there are
> > patches around to luo serialize that. Lots and lots of talks about a
> > post-hugetlbfs world out there.
> 
> And?  Adding a compatibility layer to a future kernel so that it
> understands an incoming HugeTLBFS payload should be trivial.

From my experience that's optimistic :(

> > Do we want to constrain what is possible to ensure we accomodate this
> > hugetlbfs serialization? I vote no.
> 
> In what way is providing strong ABI guarantees for individual components
> constraining HugeTBLFS serialization?

I bet it will. Other things we've looked at seemed to be like that.
Even the above about "yall screwed up" with memfd has the problem
already. I don't believe we can ever do this so right that it won't be
constraining to the kernel internals.

> > Do we want to reject the hugetlbfs serialization until we have a year
> > of debate outlining every possible ABI scenario? I also vote no.
> 
> That's a bit of a strawman argument.  Is designing a forward-looking ABI easy?
> No, but IMO "a year" is a massive exaggeration of the effort required to come up
> with a scheme that can survive a variety of plausible upgrade/downgrade scenarios.

Have you tried to get anything merged into the kernel lately? I've got
lots of uncontroversial stuff pushed out past 4 months already. Some
luo patches are close to a year already and don't even have any
controversy.

> And again, I'm not saying we have to support infinite compatibility.  

Okay, I said same stable branch only, do you have some wider
limitation in mind?

> > Should we make a downgrade round trip a downstream problem? I think
> > so!
> 
> Hard NAK.  There will inevitably be boundaries that cannot be crossed, but I am
> not at all ok punting on downgrades.  To me, that's basically saying "we want to
> add just enough support upstream so that it's not too painful to carry full support
> out-of-tree".  That completely goes against the spirit of open source and upstream
> Linux, and I want no part of it.

I generally agree with you sentiment, but I think this is a unique
case. I've asked around a fair bit, this is sufficiently complicated,
requires alot of userspace that the CSPs are not open sourcing so has
a very minimal usage foot print out side their world. I found one
other possible user that might be more open source oriented..

So, if I was feeling unreasonable I'd say stay out of the upstream
kernel entirely.

Though, I think this could grow and maybe some open source ecosystem
will develop around it. I don't know. I'm willing to give it a
chance.

HOWEVER upstream is not some kind of free outsourcing for the CSP's
proprietary forks! Do not ask maintainers to do significant and
burdensome work that only a CSP is ever going to consume and can only
really work in a closed proprietary environment. There is no "spirit
of open source" in that kind of demand. I will be NAKing anything like
that in my subsystems, I am not signing up to do live update stable
ABI so the CSPs alone can have a better proprietary product.

This is how I come to my conclusion that upstream should support same
stable branch only at this point. It minimizes the burden, it is a
decent trail of the technology, and if things go well with a quality
open ecosystem then sure, upstream can change its mind.

Jason

On 9/10/26 19:12, Jason Gunthorpe wrote:
> Keep in mind the actual goal here. Someone has kernel A and they need
> to blind kexec into kernel B and NOT have the machine explode, or all
> the VMs sitting on it lost.
> 
> Meaning you must have a way to determine before the kexec if kernel A
> is producing something B will *accept*. Accept is not "parse and fail
> with EOPNOTSUPP" like most uapi schems. Aceept means bring in and
> actually fully support and use.
>
> So how do you solve this problem? You MUST declare in some kind of
> manifest exactly what ABIs are supported, in some way.

You still have to pass out of band what B will accept.  Passing a string 
or a bitvector doesn't change much.

The problem, again is that memfd is the easy case.  KVM would bump the 
version number on every other release, as even a new serialized MSR will 
be an incompatibility.

Rather, forwards kexec *must* work (again, that's nothing but a variant 
of "we don't break userspace") and for backwards kexec, well, you must 
know what you're doing.  QEMU has been doing backwards live migration 
forever, and QEMU is a gnarly C program that has grown by accretion as 
we were learning all this stuff, so it's not impossible at all.

> Yeah, CSP broadly has to do exactly this across a wide range of
> topics. It is a further reason why this feature is not exactly usable
> by a "mainstream" user :\
It's not easy, but you aren't even trying to do it right in the kernel. 
You are starting from *a* solution and saying that it makes the feature 
hard to use.

And I'm not saying to dismiss your work or ability, quite the contrary 
in fact!  It's just that you're solving for the wrong complexity, and 
only partially so because (if I'm not wrong) there's still the question 
of how to bring the manifest of kernel B into kernel A.

Paolo

On Fri, Sep 11, 2026 at 07:16:23PM +0200, Paolo Bonzini wrote:

> Rather, forwards kexec *must* work (again, that's nothing but a variant of
> "we don't break userspace") and for backwards kexec, well, you must know
> what you're doing.

Again, I strongly disagree with this MUST. This is not a variant of
"we don't break userspace". It is internal kabi and the kernel gets to
decide which versions it supports. There are no promises here. It is
not a breaking change to say we won't support A->B.

Community needs to agree on what these parameters are. I've been clear
my stake in the ground is same stable version only as a starting point.

This is typical in the industry, nobody does live update where you can
jump ahead multiple years of versions. Sequential releases only is
very normal.

> QEMU has been doing backwards live migration forever, and QEMU is a
> gnarly C program that has grown by accretion as we were learning all
> this stuff, so it's not impossible at all.

I agree it is not impossible, you can always make things harder and
more complicated to keep it working. I don't want the kernel to turn
into a "gnarly C program" because we over engineered this.

> It's not easy, but you aren't even trying to do it right in the kernel. You

Right, I've been pretty clear about this, I want to start with
same-stable-version only and if that works out then consider if we
want to do more.

I am not doing this for any sort of technical reason. Not because it
is impossible, but because it is HARD. I do not want to force anyone
to take on hard and difficult projects like this for a niche feature
only a few CSPs are likely to use. If we take this approach LUO will
fail because maintainers will be scared to take on this extra work and
simply won't merge LUO patches.

There needs to be some balance that we can try this out and see if a
broader use case develops before we commit to doing the hardest
possible version of it.

A niche feature only for CSPs like live update must not become a boat
anchor on kernel progress!

> fact!  It's just that you're solving for the wrong complexity, and only
> partially so because (if I'm not wrong) there's still the question of how to
> bring the manifest of kernel B into kernel A.

There is a proposal to encode information into ELF sections. Feature
flags should flow through there just fine, I think.

I have not seen a proposal to re-inject this information back into the
A kernel as a 'restriction' so it fails luo instead of producing
something unsupported.

Jason

On 2026-09-10 02:12 PM, Jason Gunthorpe wrote:
> On Thu, Sep 10, 2026 at 08:35:54AM -0700, Sean Christopherson wrote:
> 
> > I guess maybe we have a different definition of ABI? 
> > 
> > I'm not saying that upstream has to be 100% forwards and backwards compatible.
> > I'm saying the serialization payload itself should communicate what features are
> > effectively required.  I.e. *if* there are incompatibilities, they should be
> > naturally expressed in the serialization format, not communicated out-of-band
> > through magic numbers.
> 
> The ABI strings were introduced specifically because extension makes
> the actual compatibility indeterminate by userspace. 
> 
> Keep in mind the actual goal here. Someone has kernel A and they need
> to blind kexec into kernel B and NOT have the machine explode, or all
> the VMs sitting on it lost.
> 
> Meaning you must have a way to determine before the kexec if kernel A
> is producing something B will *accept*. Accept is not "parse and fail
> with EOPNOTSUPP" like most uapi schems. Aceept means bring in and
> actually fully support and use.
> 
> So how do you solve this problem? You MUST declare in some kind of
> manifest exactly what ABIs are supported, in some way.

I think this series solves this problem in a fairly clean way without
relying on version numbers.

Each ABI is now extensible with a set of structured featured flags that
are exposed to userspace. Userspace can inspect the flags that the
kernel supports and confirm the next kernel also supports them.

I think there is still room for improvement, like determining what
features are used at runtime rather than statically at compile time, or
allowing userspace to disable use of certain features to control
compatability, but I think these things can be built into this type of
model.

> > The scenario you describe fits exactly with what I am proposing.  
> 
> It does not. What is really wanted here is to tell kernel A to only
> support ABI 1 for memfd and so kernel A will fail to serialize if it
> cannot do it because a newer seal flag was used.
> 
> We do not want to succeed to serialize then fail to accept after
> kexec and have a dead machine.
> 
> This is not anything like a normal uapi compatability problem.
> 
> > actually starts using the new sealing flag, the CSP can downgrade to
> > older kernels at will.  And if the user cares about downgrading,
> > then they need to prevent the flag from being used until the new
> > kernel is rollback-safe and deployed to enough hosts to prevent
> > stockout.
> 
> Yeah, CSP broadly has to do exactly this across a wide range of
> topics. It is a further reason why this feature is not exactly usable
> by a "mainstream" user :\
> 
> > > This is why I think the very idea we can support any version pair is
> > > too much to ask for. We should focus on supporting a small set of
> > > version pairs and not making it too invasive or hard in the kernel or
> > > on the maintainers.
> > >
> > > Thus live update within a stable branch only is my proposal for
> > > upstream support.
> > > 
> > > If it really succeeds at that and it becomes very popular, then let's
> > > discuss upstreaming doing additional version combinations.
> >
> > Why on earth would we have version numbers in the first place?  IMO, monotically
> > increasing version numbers are flat out the worst way to communicate
> > features.
> 
> As above, discoverablility is a key requirement.
> 
> Each version number is a very specific upstream defined ABI, in the
> sense if kernel A emits version X and kernel B accepts version X then
> kexec *must* work.
> 
> You can make some manifest in other more complicated ways, but I'm
> deeply skeptical that is really going to bring any value. It feels
> like it is just increasing the testing matrix :\

The value I see of the flag-based approach over the version-based
approach is:

 - Each component can have one ABI struct that extends over time and one
   serialization/deserialization routines, rather than N for the N
   supported current versions. Supporting multiple versions within a
   single kernel would be required for upgrade/downgrade. Maybe there is
   a way to make the multi-versioning support maintainable but it seems
   like it will be messy to me.

 - Features can be managed individually. Let's say a downstream user
   wants to use a new upstream feature. If we had a versioning model
   they would have to backport the entire version delta from their
   current kernel to that feature upstream. With flags they can backport
   and use an individual feature.

I agree testing matrix becomes more complex but maybe that can be
mitigated with your suggestion that upstream only "officially" supports
(i.e. tests) some constrained version sets like within a stable branch?

>  
> > > > I could see things like HugeTLB not working if someone booted the kernel with
> > > > support for only 1GiB pages and then tried to feed it payload with sub-1GiB ranges.
> > > > But to me, those sorts of things fall into the "well yeah, don't do that" category.
> > > 
> > > Okay, how about worse, todays kernel has hugetlbfs and there are
> > > patches around to luo serialize that. Lots and lots of talks about a
> > > post-hugetlbfs world out there.
> > 
> > And?  Adding a compatibility layer to a future kernel so that it
> > understands an incoming HugeTLBFS payload should be trivial.
> 
> From my experience that's optimistic :(
> 
> > > Do we want to constrain what is possible to ensure we accomodate this
> > > hugetlbfs serialization? I vote no.
> > 
> > In what way is providing strong ABI guarantees for individual components
> > constraining HugeTBLFS serialization?
> 
> I bet it will. Other things we've looked at seemed to be like that.
> Even the above about "yall screwed up" with memfd has the problem
> already. I don't believe we can ever do this so right that it won't be
> constraining to the kernel internals.
> 
> > > Do we want to reject the hugetlbfs serialization until we have a year
> > > of debate outlining every possible ABI scenario? I also vote no.
> > 
> > That's a bit of a strawman argument.  Is designing a forward-looking ABI easy?
> > No, but IMO "a year" is a massive exaggeration of the effort required to come up
> > with a scheme that can survive a variety of plausible upgrade/downgrade scenarios.
> 
> Have you tried to get anything merged into the kernel lately? I've got
> lots of uncontroversial stuff pushed out past 4 months already. Some
> luo patches are close to a year already and don't even have any
> controversy.
> 
> > And again, I'm not saying we have to support infinite compatibility.  
> 
> Okay, I said same stable branch only, do you have some wider
> limitation in mind?
> 
> > > Should we make a downgrade round trip a downstream problem? I think
> > > so!
> > 
> > Hard NAK.  There will inevitably be boundaries that cannot be crossed, but I am
> > not at all ok punting on downgrades.  To me, that's basically saying "we want to
> > add just enough support upstream so that it's not too painful to carry full support
> > out-of-tree".  That completely goes against the spirit of open source and upstream
> > Linux, and I want no part of it.
> 
> I generally agree with you sentiment, but I think this is a unique
> case. I've asked around a fair bit, this is sufficiently complicated,
> requires alot of userspace that the CSPs are not open sourcing so has
> a very minimal usage foot print out side their world. I found one
> other possible user that might be more open source oriented..
> 
> So, if I was feeling unreasonable I'd say stay out of the upstream
> kernel entirely.
> 
> Though, I think this could grow and maybe some open source ecosystem
> will develop around it. I don't know. I'm willing to give it a
> chance.
> 
> HOWEVER upstream is not some kind of free outsourcing for the CSP's
> proprietary forks! Do not ask maintainers to do significant and
> burdensome work that only a CSP is ever going to consume and can only
> really work in a closed proprietary environment. There is no "spirit
> of open source" in that kind of demand. I will be NAKing anything like
> that in my subsystems, I am not signing up to do live update stable
> ABI so the CSPs alone can have a better proprietary product.
> 
> This is how I come to my conclusion that upstream should support same
> stable branch only at this point. It minimizes the burden, it is a
> decent trail of the technology, and if things go well with a quality
> open ecosystem then sure, upstream can change its mind.
> 
> Jason

On Thu, Sep 10, 2026 at 09:27:58PM +0000, David Matlack wrote:
> Each ABI is now extensible with a set of structured featured flags that
> are exposed to userspace. Userspace can inspect the flags that the
> kernel supports and confirm the next kernel also supports them.

I'm fine with that bit, it is really just a different way to encode an
ABI ID #. If ABI 5 is spelled b11111 instead it doesn't materially
change how things work.

It was always the case you can use the same struct extension techinque
with a version number. A version can't handle 'holes' in the feature
mask, but I haven't thought that was a meaningful use case.

"if (id > 10)" instead "if (id & FEAT)" isn't a big coding difference.

My concern is the implied position that the kernel must make full use
of this to maximize compatability or bust. That's what has always
concerned me about all these proposals since the start.

I do not want to have arguments upstream about not changing things
because we didn't do a perfect job preserving luo compatability with
upgrade and downgrade for arbitary kernel versions. That has always
been my position and worry.

So, change the simple version to a u64 features[] tuple (be sure to
use an array, we will need alot of them!!) and I'm fine. But I'm not
going to promise that every upstream kernel will strictly only have an
increasing string of 1's. You are going to get some that look like
b1110000 and still wont work with old kernels.

Sean's initial email was asking for strict never-break ABI
compatability rules on top, and that is what I've been reacting to.

Jason

On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> Sean's initial email was asking for strict never-break ABI compatability
> rules on top, and that is what I've been reacting to.

No, I was never asking for that.  I think we just have a different interpretation
of ABI, or rather are talking about different pieces of ABI.

I am still asking for never-break serialization format compatibility, i.e. the
more literal save/restore ABI.  I'm not asking for full backwards/forwards
compatibility across all kernels version, i.e. the higher level "kernel" ABI.

To phrase things differently: I am a-ok if kernels are inherently incompatible
because they fundamentally operate differently and/or support different features.
I am not ok if we end up with incompatible kernels because the save/restore
interfaces and payloads are poorly designed, lack abstraction, etc.

On Thu, Sep 10, 2026 at 03:42:36PM -0700, Sean Christopherson wrote:
> On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> > Sean's initial email was asking for strict never-break ABI compatability
> > rules on top, and that is what I've been reacting to.
> 
> No, I was never asking for that.  I think we just have a different interpretation
> of ABI, or rather are talking about different pieces of ABI.
> 
> I am still asking for never-break serialization format compatibility, i.e. the
> more literal save/restore ABI.  I'm not asking for full backwards/forwards
> compatibility across all kernels version, i.e. the higher level "kernel" ABI.
> 
> To phrase things differently: I am a-ok if kernels are inherently incompatible
> because they fundamentally operate differently and/or support different features.
> I am not ok if we end up with incompatible kernels because the save/restore
> interfaces and payloads are poorly designed, lack abstraction, etc.  

I can more agree with this, it is mostly what I thought we'd end up
doing anyhow with version numbers and growing the structs not
replacing them.

So you still end up with a version label (string I guess?) because we
can break it from time to time, just the version should have the
feature mechanism layered under it? I'm fine with that

Jason

On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> On Thu, Sep 10, 2026 at 03:42:36PM -0700, Sean Christopherson wrote:
> > On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> > > Sean's initial email was asking for strict never-break ABI compatability
> > > rules on top, and that is what I've been reacting to.
> > 
> > No, I was never asking for that.  I think we just have a different interpretation
> > of ABI, or rather are talking about different pieces of ABI.
> > 
> > I am still asking for never-break serialization format compatibility, i.e. the
> > more literal save/restore ABI.  I'm not asking for full backwards/forwards
> > compatibility across all kernels version, i.e. the higher level "kernel" ABI.
> > 
> > To phrase things differently: I am a-ok if kernels are inherently incompatible
> > because they fundamentally operate differently and/or support different features.
> > I am not ok if we end up with incompatible kernels because the save/restore
> > interfaces and payloads are poorly designed, lack abstraction, etc.  
> 
> I can more agree with this, it is mostly what I thought we'd end up
> doing anyhow with version numbers and growing the structs not
> replacing them.
> 
> So you still end up with a version label (string I guess?) because we
> can break it from time to time, 
   
Why bother with a label/string?  I don't understand what it buys us; I see only
pain.  Make the features discoverable and punt on all of the "is this pile of
insanity my company has created compatible with this other pile of insanity?" to
layers above the kernel.

IMO, trying to capture compatibility in a version string is doomed to fail.  To
be useful, the version would have to be extremely sensitive to changes, otherwise
userspace will have to do its own compatibility analysis/checks anyways.  And at
that point, a verson string will be too restrictive, e.g. will mismatch due to
changes in features the deployment isn't using.

> just the version should have the feature mechanism layered under it? I'm fine
> with that
> 
> Jason

On Fri, Sep 11, 2026 at 06:44:50AM -0700, Sean Christopherson wrote:
> On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> > On Thu, Sep 10, 2026 at 03:42:36PM -0700, Sean Christopherson wrote:
> > > On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> > > > Sean's initial email was asking for strict never-break ABI compatability
> > > > rules on top, and that is what I've been reacting to.
> > > 
> > > No, I was never asking for that.  I think we just have a different interpretation
> > > of ABI, or rather are talking about different pieces of ABI.
> > > 
> > > I am still asking for never-break serialization format compatibility, i.e. the
> > > more literal save/restore ABI.  I'm not asking for full backwards/forwards
> > > compatibility across all kernels version, i.e. the higher level "kernel" ABI.
> > > 
> > > To phrase things differently: I am a-ok if kernels are inherently incompatible
> > > because they fundamentally operate differently and/or support different features.
> > > I am not ok if we end up with incompatible kernels because the save/restore
> > > interfaces and payloads are poorly designed, lack abstraction, etc.  
> > 
> > I can more agree with this, it is mostly what I thought we'd end up
> > doing anyhow with version numbers and growing the structs not
> > replacing them.
> > 
> > So you still end up with a version label (string I guess?) because we
> > can break it from time to time, 
>    
> Why bother with a label/string?  I don't understand what it buys us; I see only
> pain.  Make the features discoverable and punt on all of the "is this pile of
> insanity my company has created compatible with this other pile of insanity?" to
> layers above the kernel.

It is one layer above, the serialization ends up with 10 memfds a
couple iommus, kvm, pci device, etc, etch. Each one starts with a
string label saying what it even is - to dispatch it to the proper
de-serialization code, then some detail about where it is (eg a handle
to recover a FD later) and then the actual object serialization
data itself.

So the features are fine to handle the ABI within the object and a
memfd-v1, memfd-v2 is fine to handle labeling the object within the
stream and gives us an option to reboot the ABI if some reason comes
up someday.

If features are a perfect success then we will never need to go beyond
memfd-v1. I'm just saying we should have the option and plan for it.

Jason

On Thu, 2026-09-10 at 19:57 -0300, Jason Gunthorpe wrote:
> On Thu, Sep 10, 2026 at 03:42:36PM -0700, Sean Christopherson wrote:
> > On Thu, Sep 10, 2026, Jason Gunthorpe wrote:
> > > Sean's initial email was asking for strict never-break ABI compatability
> > > rules on top, and that is what I've been reacting to.
> > 
> > No, I was never asking for that.  I think we just have a different interpretation
> > of ABI, or rather are talking about different pieces of ABI.
> > 
> > I am still asking for never-break serialization format compatibility, i.e. the
> > more literal save/restore ABI.  I'm not asking for full backwards/forwards
> > compatibility across all kernels version, i.e. the higher level "kernel" ABI.
> > 
> > To phrase things differently: I am a-ok if kernels are inherently incompatible
> > because they fundamentally operate differently and/or support different features.
> > I am not ok if we end up with incompatible kernels because the save/restore
> > interfaces and payloads are poorly designed, lack abstraction, etc.  
> 
> I can more agree with this, it is mostly what I thought we'd end up
> doing anyhow with version numbers and growing the structs not
> replacing them.
> 
> So you still end up with a version label (string I guess?) because we
> can break it from time to time, just the version should have the
> feature mechanism layered under it? I'm fine with that

Seems reasonable. I'd note that downgrade support *is* important though.

So where possible, those new features which are inherently incompatible
should be opt-out. That way the admin can disable them when first
upgrading to a new kernel version, and enable them only in a
*subsequent* update, once that new kernel version has actually rolled
out successfully.

Again, it's OK if we sometimes can't manage that, but not if it happens
just because of poor design, as Sean said above.

The thing that *really* concerns me here though, is latency. When we
pause guests and kexec underneath them, every *millisecond* is
perceived as steal time. I worry that we're designing handover
protocols to look *pretty*, and not necessarily go *fast* enough.

I'd really love to see these patch series come in with benchmarks.