Hey,

The idea in this series is old-ish and really never let go of me and so
I wanted to at least dump it onto the list once even if it's just for
illustrative purposes.

I still had parts of an implementation laying around that I started
montsh ago alongside FD_PREPARE()/FD_ADD(). I sat down and finished it.

We've spoken about this idea a few times over the years that we could
reserve fds and files on a task and then install or a clean them up on
syscall success or error and get rid of most of the complicated cleanup
dance that we have in a lot of code. In particular drm. It last came up
during the Rust file descriptor reservation discussion quite some time
ago.

TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
allocates a descriptor like get_unused_fd_flags() does and records it in
a slot on the task. fd_stage() attaches the file to that slot and
returns the number.

When the syscall returns success the exit path installs every staged
file. When it returns an error it drops the descriptors and the files.

So a caller reserves, hands the number to userspace whenever it wants,
creates the file, stages it and returns errors without unwinding
anything.

fd_prepare() returns the slot itself, as a const pointer. The
preexisting fd_prepare_fd() and fd_prepare_file() give access to the fd
and file.

get_unused_fd_flags() and fd_install() don't change. A descriptor is
only reserved where a caller asks for it. And open(), dup() and all
other syscalls that maximize speed simply use FD_ADD().

The task keeps two slots inline. For SCM_RIGHTS and multi-descriptor
ioctls a spill array is added. It stick with the task.

Reservations belong to the thread and the syscall that made them. A
child of fork() starts without any. A thread can't unshare its fdtagble
with outstanding reservations. Kernel threads never return to userspace
so nothing would commit. Anything left at exit is a bug and gets warned
about and dropped.

Christian

Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
---
Christian Brauner (50):
      file: install files on syscall exit
      entry: commit fds on syscall exit
      alpha: commit fds on syscall exit
      ARC: commit fds on syscall exit
      ARM: commit fds on syscall exit
      arm64: commit fds on syscall exit
      csky: commit fds on syscall exit
      hexagon: commit fds on syscall exit
      m68k: commit fds on syscall exit
      microblaze: commit fds on syscall exit
      MIPS: commit fds on syscall exit
      nios2: commit fds on syscall exit
      openrisc: commit fds on syscall exit
      parisc: commit fds on syscall exit
      sh: commit fds on syscall exit
      sparc: commit fds on syscall exit
      um: commit fds on syscall exit
      xtensa: commit fds on syscall exit
      file: require the syscall exit hook from every architecture
      file: warn when the descriptor table is unshared with slots
      io_uring: commit fds per request
      net: install SCM_RIGHTS descriptors when recvmsg() returns
      file: open-code receive_fd()'s immediate install
      file: make FD_ADD() a standalone immediate install
      file: reimplement FD_PREPARE() on the deferred fd_prepare() path
      dma-buf: stop unwinding sync file descriptors by hand
      drm/amdkfd: stop collecting CRIU dma-buf descriptors for a final install
      drm/msm: install the out-fence descriptor when the ioctl returns
      drm/virtio: install the out-fence descriptor when the ioctl returns
      drm/vmwgfx: install the out-fence descriptor when the ioctl returns
      vfio: install the migration data descriptor when the ioctl returns
      liveupdate: install the session descriptors when the ioctl returns
      io_uring/zcrx: install the exported descriptor when the request returns
      sctp: install the peeloff descriptor when the syscall returns
      ALSA: compress: install the task descriptors when the ioctl returns
      nitro_enclaves: install the enclave descriptor when the ioctl returns
      tpm: vtpm_proxy: install the server descriptor when the ioctl returns
      perf: stop putting the event descriptor back on failure
      seccomp: stop putting the listener descriptor back on failure
      KVM: stop putting descriptors back on failure
      KVM: guest_memfd: stop putting the descriptor back on failure
      drm: stop unwinding descriptors by hand
      drm/amdgpu: stop unwinding the fence descriptor by hand
      drm/etnaviv: install the out-fence descriptor when the ioctl returns
      accel/habanalabs: stop putting the dma-buf descriptor back on failure
      xen/gntdev-dmabuf: stop putting the descriptor back on failure
      iio: buffer: install the buffer descriptor when the ioctl returns
      misc: fastrpc: install the dma-buf descriptor when the ioctl returns
      iommufd: stop putting descriptors back on failure
      Drivers: hv: mshv: stop putting descriptors back on failure

 arch/alpha/include/asm/thread_info.h         |   2 +
 arch/alpha/kernel/entry.S                    |  24 ++-
 arch/alpha/kernel/ptrace.c                   |   3 +
 arch/arc/include/asm/thread_info.h           |   2 +
 arch/arc/kernel/entry.S                      |   5 +
 arch/arc/kernel/ptrace.c                     |   4 +
 arch/arm/include/asm/thread_info.h           |   2 +
 arch/arm/kernel/entry-common.S               |   2 +
 arch/arm/kernel/ptrace.c                     |   4 +
 arch/arm64/include/asm/thread_info.h         |   4 +-
 arch/arm64/kernel/ptrace.c                   |   4 +
 arch/csky/include/asm/thread_info.h          |   2 +
 arch/csky/kernel/entry.S                     |   9 +
 arch/csky/kernel/ptrace.c                    |   4 +
 arch/hexagon/include/asm/thread_info.h       |   2 +
 arch/hexagon/kernel/traps.c                  |   4 +
 arch/m68k/68000/entry.S                      |  10 ++
 arch/m68k/coldfire/entry.S                   |  12 ++
 arch/m68k/include/asm/thread_info.h          |   2 +
 arch/m68k/kernel/entry.S                     |   2 +
 arch/m68k/kernel/ptrace.c                    |   3 +
 arch/microblaze/include/asm/thread_info.h    |   5 +-
 arch/microblaze/kernel/ptrace.c              |   4 +
 arch/mips/include/asm/thread_info.h          |   5 +-
 arch/mips/kernel/ptrace.c                    |   4 +
 arch/nios2/include/asm/thread_info.h         |   2 +
 arch/nios2/kernel/entry.S                    |   5 +
 arch/nios2/kernel/ptrace.c                   |   3 +
 arch/openrisc/include/asm/thread_info.h      |   5 +-
 arch/openrisc/kernel/entry.S                 |   5 +-
 arch/openrisc/kernel/ptrace.c                |   4 +
 arch/parisc/include/asm/thread_info.h        |   2 +
 arch/parisc/kernel/entry.S                   |  12 ++
 arch/parisc/kernel/ptrace.c                  |   4 +
 arch/powerpc/platforms/cell/spufs/inode.c    |  12 +-
 arch/sh/include/asm/thread_info.h            |  10 +-
 arch/sh/kernel/entry-common.S                |   8 +-
 arch/sh/kernel/ptrace_32.c                   |   4 +
 arch/sparc/include/asm/thread_info_32.h      |   2 +
 arch/sparc/include/asm/thread_info_64.h      |   9 +-
 arch/sparc/kernel/entry.S                    |   2 +-
 arch/sparc/kernel/ptrace_32.c                |   4 +
 arch/sparc/kernel/ptrace_64.c                |   4 +
 arch/sparc/kernel/syscalls.S                 |   4 +-
 arch/um/include/asm/thread_info.h            |   2 +
 arch/um/kernel/ptrace.c                      |   4 +
 arch/xtensa/include/asm/thread_info.h        |   2 +
 arch/xtensa/kernel/entry.S                   |   5 +
 arch/xtensa/kernel/ptrace.c                  |   4 +
 drivers/accel/habanalabs/common/memory.c     |  13 +-
 drivers/char/tpm/tpm_vtpm_proxy.c            |  30 +---
 drivers/dma-buf/dma-buf.c                    |  33 ++--
 drivers/dma-buf/sw_sync.c                    |  40 ++---
 drivers/dma-buf/sync_file.c                  |  52 ++----
 drivers/gpio/gpiolib-cdev.c                  |  18 +-
 drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c       |  16 +-
 drivers/gpu/drm/amd/amdkfd/kfd_chardev.c     |  67 ++------
 drivers/gpu/drm/drm_lease.c                  |  20 +--
 drivers/gpu/drm/drm_prime.c                  |  13 +-
 drivers/gpu/drm/drm_syncobj.c                |  44 ++---
 drivers/gpu/drm/etnaviv/etnaviv_gem_submit.c |  14 +-
 drivers/gpu/drm/msm/msm_gem_submit.c         |  22 +--
 drivers/gpu/drm/msm/msm_gem_vma.c            |  22 +--
 drivers/gpu/drm/msm/msm_perfcntr.c           |   6 +-
 drivers/gpu/drm/virtio/virtgpu_submit.c      |  32 ++--
 drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c      |  29 ++--
 drivers/hv/mshv_root_main.c                  |  27 +--
 drivers/hv/mshv_vtl_main.c                   |  20 +--
 drivers/iio/industrialio-buffer.c            |  26 +--
 drivers/iommu/iommufd/eventq.c               |  39 ++---
 drivers/media/mc/mc-request.c                |   6 +-
 drivers/misc/fastrpc.c                       |  19 +--
 drivers/misc/ntsync.c                        |   6 +-
 drivers/vfio/vfio_main.c                     |  25 +--
 drivers/virt/nitro_enclaves/ne_misc_dev.c    |  33 ++--
 drivers/xen/gntdev-dmabuf.c                  |  14 +-
 fs/eventfd.c                                 |   6 +-
 fs/eventpoll.c                               |   6 +-
 fs/exec.c                                    |   3 +
 fs/file.c                                    | 245 +++++++++++++++++++++++++--
 fs/namespace.c                               |  12 +-
 fs/nsfs.c                                    |   6 +-
 fs/xfs/xfs_handle.c                          |   6 +-
 include/linux/entry-common.h                 |   8 +-
 include/linux/file.h                         | 160 +++++------------
 include/linux/sched.h                        |  21 +++
 include/linux/thread_info.h                  |   2 +
 io_uring/io_uring.c                          |  27 +++
 io_uring/mock_file.c                         |   5 +-
 io_uring/zcrx.c                              |  25 +--
 kernel/bpf/bpf_iter.c                        |   6 +-
 kernel/bpf/token.c                           |   6 +-
 kernel/events/core.c                         |  31 ++--
 kernel/exit.c                                |   1 +
 kernel/fork.c                                |   3 +
 kernel/liveupdate/luo_core.c                 |  48 ++----
 kernel/liveupdate/luo_session.c              |  24 +--
 kernel/seccomp.c                             |  13 +-
 mm/userfaultfd.c                             |   6 +-
 net/core/scm.c                               |  13 +-
 net/handshake/netlink.c                      |  20 ++-
 net/kcm/kcmsock.c                            |   5 +-
 net/sctp/socket.c                            |  78 ++++-----
 sound/core/compress_offload.c                |  24 ++-
 virt/kvm/guest_memfd.c                       |  20 +--
 virt/kvm/kvm_main.c                          |  65 +++----
 106 files changed, 962 insertions(+), 861 deletions(-)
---
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
change-id: 20260915-work-fd-reserve-unify-folded-b89d3154b3d0

On Tue, Sep 15, 2026 at 1:31 PM Christian Brauner <brauner@kernel.org> wrote:
> TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
> allocates a descriptor like get_unused_fd_flags() does and records it in
> a slot on the task. fd_stage() attaches the file to that slot and
> returns the number.

I'm not really a fan but I feel like if you want to do this, you
should be using task_work so you don't have to touch the entry/exit
paths directly? (And if task_work is too inefficient somehow, you
could tinker with the task_work implementation.)

> When the syscall returns success the exit path installs every staged
> file. When it returns an error it drops the descriptors and the files.

Is this mainly about stuff like "we installed a file descriptor and
then the following put_user() failed"? Because if so, I think a nicer
fix would be to have a policy of "if userspace provides unwritable
memory to a syscall, just keep going and pretend the access worked",
and maybe have a sysctl that kills the process when this happens to
emphasize that userspace should not be doing this.

On Tue, 15 Sept 2026 at 10:52, Jann Horn <jannh@google.com> wrote:
>
> Is this mainly about stuff like "we installed a file descriptor and
> then the following put_user() failed"? Because if so, I think a nicer
> fix would be to have a policy of "if userspace provides unwritable
> memory to a syscall, just keep going and pretend the access worked",
> and maybe have a sysctl that kills the process when this happens to
> emphasize that userspace should not be doing this.

We've done that before, where we just ignore put_user() errors and the
user gets whatever the user gets.

It is maybe not optimal, but it's fine. You can find quite a lot of
unchecked put_user() calls with a pattern like

        git grep '^[[:space:]]*put_user(.*);'

and some of them are in core code - see the two in kernel/fork.c, for example.

One of them says "if userspace has not set up a proper pointer then
tough luck". The other one doesn't even bother with a comment.

The scheduler has two cases too, although one of them is admittedly
for another error case.

So yes, saying "if you pass bogus arguments, you get what you get" is
a valid model. It's perhaps not the *preferred* model, but it's not
wrong.

It *would* be wrong to take code that already has error handling and
remove the error handling, though.

                  Linus

On Tue, 15 Sep 2026 12:08:57 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Tue, 15 Sept 2026 at 10:52, Jann Horn <jannh@google.com> wrote:
> >
> > Is this mainly about stuff like "we installed a file descriptor and
> > then the following put_user() failed"? Because if so, I think a nicer
> > fix would be to have a policy of "if userspace provides unwritable
> > memory to a syscall, just keep going and pretend the access worked",
> > and maybe have a sysctl that kills the process when this happens to
> > emphasize that userspace should not be doing this.  
> 
> We've done that before, where we just ignore put_user() errors and the
> user gets whatever the user gets.
> 
> It is maybe not optimal, but it's fine. You can find quite a lot of
> unchecked put_user() calls with a pattern like
> 
>         git grep '^[[:space:]]*put_user(.*);'
> 
> and some of them are in core code - see the two in kernel/fork.c, for example.
> 
> One of them says "if userspace has not set up a proper pointer then
> tough luck". The other one doesn't even bother with a comment.

I think the code should try to return EFAULT (IIRC that is too hard
in one of the exec cases).
Otherwise very unexpected things might happen if the memory is just
readonly.
If you ignore the error and the pointer is invalid the application will
get a SIGSEGV and (usually) die.
But winding back kernel data because a user copy failed is likely to
be problematic/difficult and at best have error path code that isn't
really tested.

As well as writing fd numbers to userspace, some sockopt code tries to
wind back if the write to optlen fails (which has been read earlier).

I've forgotten which Unix converted EFAULT to SIGSEGV in the system
call exit code - I'm sure one of the ones I've used did.

David

> 
> The scheduler has two cases too, although one of them is admittedly
> for another error case.
> 
> So yes, saying "if you pass bogus arguments, you get what you get" is
> a valid model. It's perhaps not the *preferred* model, but it's not
> wrong.
> 
> It *would* be wrong to take code that already has error handling and
> remove the error handling, though.
> 
>                   Linus
>

On Tue, 15 Sept 2026 at 04:31, Christian Brauner <brauner@kernel.org> wrote:
>
> TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
> allocates a descriptor like get_unused_fd_flags() does and records it in
> a slot on the task. fd_stage() attaches the file to that slot and
> returns the number.

Honestly, I am *not* a fan. At all.

I think this adds complexity rather than removing it.

The diffstat tells a story:

>  106 files changed, 962 insertions(+), 861 deletions(-)

in how this adds a hundred lines more than it deletes.

But to make things worse, it adds them in bad places: low-level
architecture assembly code.

Those extra lines are *not* making complicated code simpler. Quite the
reverse. They are taking fairly straightforward "just deal with errors
with normal cleanup in the place where it makes sense" and makes the
*straightforward* case simpler, while making the big picture more
complicated and subtle, and adding new code to places that are not
simple and just blindly do somethign that makes no sense in that
context.

IOW: it's more code, and it's more abstraction, and it DOES NOT HELP.

And yes, most of the well-maintained modern platforms (read: x86 and
arm64) use the generic infrastructure and they don't show that
low-level asm effect. So for those cases, it's mainly just  that added
conceptual complexity and a new odd rule - and makes a *successful*
system call go through that "extra work" and indirection phase that it
didn't use to go through.

So it just spreads out the work, and moves it to a less obvious place.

All for the very questionable advantage that you can now randomly add
a file descriptor without the few lines of fairly straightforward
rollback.

IOW, I do not see a single real upside to this, and I see immediate
downsides to it, and a more abstract complicated machinery.

I do *not* see why we woudl want to make this be a "every system call
exit" kind of thing. If there are places that think it's complicated
to do, they could have *their* local little stack of fd's pending. Why
force it onto the low-level system call entry for every system call
when there are only a small handful of actual real cases (the vfs
itself, and random ioctl's by odd driver subsystems).

              Linus

On Tue, Sep 15, 2026 at 09:02:31AM -0700, Linus Torvalds wrote:
> On Tue, 15 Sept 2026 at 04:31, Christian Brauner <brauner@kernel.org> wrote:
> >
> > TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
> > allocates a descriptor like get_unused_fd_flags() does and records it in
> > a slot on the task. fd_stage() attaches the file to that slot and
> > returns the number.
> 
> Honestly, I am *not* a fan. At all.

I'm really glad that you don't like it! I hate it too.

To quote back from the cover letter:

> The idea in this series is old-ish and really never let go of me and so
> I wanted to at least dump it onto the list once even if it's just for
> illustrative purposes.

I think the cleanup-based solution we've added a couple of releases ago
is the better appraoch. Which is why I chose it over this.

But this install-on-the-way-out idea has been brought up quite a few
times as the "better approach". I never really believed it would work
nicely. Now we have an implementation on the list that we can point to
and we can move on from this.

On Tue, Sep 15, 2026 at 9:02 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Tue, 15 Sept 2026 at 04:31, Christian Brauner <brauner@kernel.org> wrote:
> >
> > TL;DR, this lets arch code handle fd install and cleanup. fd_prepare()
> > allocates a descriptor like get_unused_fd_flags() does and records it in
> > a slot on the task. fd_stage() attaches the file to that slot and
> > returns the number.
>
> Honestly, I am *not* a fan. At all.
>
> I think this adds complexity rather than removing it.
>
> The diffstat tells a story:
>
> >  106 files changed, 962 insertions(+), 861 deletions(-)
>
> in how this adds a hundred lines more than it deletes.
>
> But to make things worse, it adds them in bad places: low-level
> architecture assembly code.
>
> Those extra lines are *not* making complicated code simpler. Quite the
> reverse. They are taking fairly straightforward "just deal with errors
> with normal cleanup in the place where it makes sense" and makes the
> *straightforward* case simpler, while making the big picture more
> complicated and subtle, and adding new code to places that are not
> simple and just blindly do somethign that makes no sense in that
> context.

With fd's it is more than just about error rollback.. exposing an fd
to userspace mid-way thru an ioctl is an easy security bug to write..

OTOH, "just use FD_PREPARE() / fd_publish()"..

BR,
-R

On Tue, 15 Sept 2026 at 15:21, Rob Clark <rob.clark@oss.qualcomm.com> wrote:
>
> With fd's it is more than just about error rollback.. exposing an fd
> to userspace mid-way thru an ioctl is an easy security bug to write..

That isn't very common code. It's something that is done by a few
strange cases, and yes, those cases should just do it right.

But "those few cases should do it right" does not mean "make a whole
new subsystem for this thing that makes a simple "open" actually be
slower and stranger".

             Linus