arch/Kconfig | 7 + arch/arm64/Kconfig | 1 + arch/arm64/kernel/process.c | 109 +++++++ arch/x86/Kconfig | 1 + arch/x86/kernel/process.c | 10 +- arch/x86/kernel/process_64.c | 139 +++++++++ include/linux/io_uring.h | 9 + include/linux/io_uring_types.h | 48 +++- include/linux/sched.h | 2 +- include/linux/thread_handoff.h | 76 +++++ include/trace/events/io_uring.h | 114 ++++++++ init/Kconfig | 11 + io_uring/Makefile | 1 + io_uring/handoff.c | 395 +++++++++++++++++++++++++ io_uring/handoff.h | 103 +++++++ io_uring/io-wq.c | 270 +++++++++++++++++- io_uring/io-wq.h | 15 + io_uring/io_uring.c | 310 ++++++++++++++------ io_uring/io_uring.h | 46 ++- io_uring/kbuf.c | 5 +- io_uring/msg_ring.c | 2 +- io_uring/opdef.c | 29 ++ io_uring/opdef.h | 2 + io_uring/rw.c | 2 +- io_uring/splice.c | 6 + io_uring/tctx.c | 16 +- io_uring/tctx.h | 2 + io_uring/tw.c | 14 +- io_uring/uring_cmd.c | 5 +- kernel/Makefile | 1 + kernel/fork.c | 6 +- kernel/sched/core.c | 36 +++ kernel/thread_handoff.c | 490 ++++++++++++++++++++++++++++++++ 33 files changed, 2167 insertions(+), 116 deletions(-)
Hi,
io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to
io-wq when that isn't possible. For a range of opcodes it isn't possible
at all, as there's no nonblocking path in the kernel for them: fsync,
statx, openat, the *at family, xattr, fadvise, splice, etc. Those are
punted unconditionally, and the punt costs a thread wakeup, a context
switch and a task_work completion round trip per request. io_uring HAS
to be cautious to prevent accidental blocking in the kernel, even if the
operations predominantly never block. Sad story. Examples of that are
things like an fdatasync that doesn't block, statx that hits dcache,
openat for O_TMPFILE, etc. All of those would've completed inline just
fine, but io_uring just cannot rely on that.
This series issues those requests inline in blocking mode instead, and
only pays for the offload if the request actually blocks. But by the
time it blocks, the submitter is deep in the kernel with the request on
its stack, so the work can't be moved to another thread. What we can
move is the identity. If the submitting task blocks, an idle io-wq
worker takes over its user visible identity (tid, signal state,
credentials, scheduling attributes, cgroup, user register state),
finishes the io_uring_enter() call and returns to userspace as the
submitter. The original task finishes the request as an
io-wq worker and joins the pool. Userspace is none the wiser, hopefully,
the same tid came back from the syscall, it's just on a different
task_struct. Folks that have been around a while may remember earlier
attempts at this about 20 years ago.
We catch the blocking through a scheduler hook. A task in a blocking
inline issue carries PF_IO_HANDOFF, and sched_submit_work() calls into
io_uring for it, next to the existing io-wq and workqueue hooks. This is
where the identity is handed off, with the uring_lock still held by the
blocking task and released for the promoted worker on its behalf.
Structure of the series:
1 kernel: the thread identity handoff itself, in
kernel/thread_handoff.c. Independent of io_uring.
2 sched: the PF_IO_HANDOFF hook.
3-4 arm64 and x86 support. The arch hooks sync live register state
before the source blocks and load it on the destination.
5-9 io_uring prep: helper cleanups, a tctx node list, tracking of
uring_lock sections in the issue path so the hook knows when
the lock may be dropped, splitting io_uring_enter() so it can
be resumed by another task, and keeping the block plug on the
submitter's stack.
10 io-wq: claiming an idle worker for a handoff, and keeping a
couple of idle spares around as targets.
11-12 io_uring: the handoff itself, and deferring the identity
migration to the end of the submission so a batch of blocking
SQEs costs one migration rather than one per SQE.
13 io_uring: issue blockable requests inline in blocking mode.
This is where behavior changes.
14 tracepoints.
15 treat IOSQE_ASYNC the same way. Separate as it's a userspace
visible policy change, and I'm not sure yet it should be done.
Some things are refused for handoff up front: traced tasks, per-task
perf contexts, PI futexes, audit contexts, armed per-thread CPU timers,
core scheduling cookies, vfork parents. Per-thread accounting moves with
the handoff, so the counters userspace sees for a tid stay monotonic.
Known gaps are LSM state kept in the task rather than the cred, and
PR_SET_IO_FLUSHER. Neither moves, and not moving them can only ever
restrict.
Reads and writes on files with FMODE_NOWAIT are excluded.
They have a working nonblocking path and poll retry, and a handoff per
op would be worse than that. The handoff covers requests that previously
would always have been handed to io-wq upfront. uring_cmd is excluded
too, as drivers like ublk bind state to the submitting task.
Some test results:
Measured in a virtme-ng guest, 8 vcpu, non-debug x86 config, same
kernel with a sysctl toggle for turning the feature on and off. CPU
is the usage of the whole process including io-wq workers, as a
percentage of one CPU. Mean of two runs.
ops/s cpu
=====================================================
fsync, tmpfs, qd 1
baseline 28.2k 96%
handoff 221k 100%
change +681% +5%
fsync, tmpfs, qd 8
baseline 195k 141%
handoff 526k 100%
change +170% -29%
fsync, tmpfs, qd 32
baseline 357k 219%
handoff 611k 100%
change +71% -54%
fsync, ext4 (flushes, always blocks), qd 1
baseline 9.6k 66%
handoff 7.1k 91%
change -26% +37%
fsync, ext4 (flushes, always blocks), qd 8
baseline 29.8k 212%
handoff 14.7k 146%
change -51% -31%
fsync, ext4 (flushes, always blocks), qd 32
baseline 42.2k 270%
handoff 14.6k 144%
change -65% -47%
statx, ext4, qd 1
baseline 23.6k 96%
handoff 121k 100%
change +414% +5%
statx, ext4, qd 8
baseline 129k 234%
handoff 179k 100%
change +38% -57%
statx, ext4, qd 32
baseline 197k 276%
handoff 140k 100%
change -29% -64%
statx, tmpfs, qd 1
baseline 24.4k 96%
handoff 117k 100%
change +378% +4%
statx, tmpfs, qd 8
baseline 131k 232%
handoff 180k 100%
change +37% -57%
statx, tmpfs, qd 32
baseline 219k 295%
handoff 190k 100%
change -13% -66%
fadvise DONTNEED, ext4, qd 1
baseline 26.1k 95%
handoff 150k 100%
change +478% +5%
fadvise DONTNEED, ext4, qd 8
baseline 127k 170%
handoff 252k 100%
change +99% -41%
fadvise DONTNEED, ext4, qd 32
baseline 270k 234%
handoff 273k 100%
change +1% -57%
renameat, ext4, qd 1
baseline 7.4k 98%
handoff 13.4k 100%
change +81% +2%
renameat, ext4, qd 8
baseline 12.0k 544%
handoff 14.6k 100%
change +21% -82%
renameat, ext4, qd 32
baseline 11.2k 500%
handoff 14.3k 100%
change +28% -80%
renameat, tmpfs, qd 1
baseline 15.0k 97%
handoff 48.5k 99%
change +223% +2%
renameat, tmpfs, qd 8
baseline 65.6k 514%
handoff 55.4k 98%
change -15% -81%
renameat, tmpfs, qd 32
baseline 39.4k 462%
handoff 38.9k 98%
change -1% -79%
openat O_TMPFILE, ext4, qd 1
baseline 6.3k 100%
handoff 11.3k 100%
change +81% +1%
openat O_TMPFILE, ext4, qd 8
baseline 23.0k 258%
handoff 12.9k 99%
change -44% -62%
openat O_TMPFILE, ext4, qd 32
baseline 26.2k 248%
handoff 13.4k 99%
change -49% -60%
openat O_TMPFILE, tmpfs, qd 1
baseline 13.4k 95%
handoff 43.0k 98%
change +222% +3%
openat O_TMPFILE, tmpfs, qd 8
baseline 57.8k 220%
handoff 51.7k 96%
change -11% -56%
openat O_TMPFILE, tmpfs, qd 32
baseline 69.3k 218%
handoff 56.8k 96%
change -18% -56%
splice to pipe, ext4, qd 1
baseline 19.4k 96%
handoff 21.1k 98%
change +8% +2%
splice to pipe, ext4, qd 8
baseline 48.2k 176%
handoff 48.4k 176%
change +0% +0%
splice to pipe, ext4, qd 32
baseline 53.3k 188%
handoff 48.1k 182%
change -10% -3%
As you can tell, normal QD=1 type issues see big wins. Conversely, for
higher queue depth, there are losses. The losses are generally from one
of two reasons:
1) The syscall part is fairly expensive, and previously we farmed all of
this work across a bunch of io-wq workers, and the parallelization
there helps performance. For QD=1 that obviously isn't the case. The
more expensive the lower level kernel parts are, the lower the QD
required to see a perf loss. openat is the obvious worse case for this.
2) The syscall part ALWAYS blocks. For this case, io-wq is going to be
quicker, just punt the opcode upfront. fsync on ext4, as shown in the table
above, is indeed that case. Each of those block.
I've got some ideas for how to mitigate the losses for higher queue
depths, but a) I didn't think they were THAT interesting for an RFC, as
low QD is generally what people do with these kinds of ops, and b) the
main concept behind this handoff is really the interesting part right
now.
Passes the full liburing test suite, on both x86-64 and arm64. Other
archs don't support this yet.
This is obviously an RFC, in terms of what I'd love people to take a
closer look at:
- The scheduler hook and the identity move itself, kernel/thread_handoff.c.
Is the set of refused states complete enough, and is moving
thread group leadership this way (the leader must stay first on
->thread_head, like de_thread() keeps it) acceptable / kosher.
- The x86 and arm64 register state handling. x86 refuses AMX users,
and arm64 refuses SME.
- Whether anyone relies on a task's user identity staying on one
task_struct in ways not covered above in the series.
Patches are against 7.3-rc2. Also available at:
git://git.kernel.dk/linux.git io_uring-thread-handoff.3
arch/Kconfig | 7 +
arch/arm64/Kconfig | 1 +
arch/arm64/kernel/process.c | 109 +++++++
arch/x86/Kconfig | 1 +
arch/x86/kernel/process.c | 10 +-
arch/x86/kernel/process_64.c | 139 +++++++++
include/linux/io_uring.h | 9 +
include/linux/io_uring_types.h | 48 +++-
include/linux/sched.h | 2 +-
include/linux/thread_handoff.h | 76 +++++
include/trace/events/io_uring.h | 114 ++++++++
init/Kconfig | 11 +
io_uring/Makefile | 1 +
io_uring/handoff.c | 395 +++++++++++++++++++++++++
io_uring/handoff.h | 103 +++++++
io_uring/io-wq.c | 270 +++++++++++++++++-
io_uring/io-wq.h | 15 +
io_uring/io_uring.c | 310 ++++++++++++++------
io_uring/io_uring.h | 46 ++-
io_uring/kbuf.c | 5 +-
io_uring/msg_ring.c | 2 +-
io_uring/opdef.c | 29 ++
io_uring/opdef.h | 2 +
io_uring/rw.c | 2 +-
io_uring/splice.c | 6 +
io_uring/tctx.c | 16 +-
io_uring/tctx.h | 2 +
io_uring/tw.c | 14 +-
io_uring/uring_cmd.c | 5 +-
kernel/Makefile | 1 +
kernel/fork.c | 6 +-
kernel/sched/core.c | 36 +++
kernel/thread_handoff.c | 490 ++++++++++++++++++++++++++++++++
33 files changed, 2167 insertions(+), 116 deletions(-)
--
Jens Axboe
Jens Axboe <axboe@kernel.dk> writes: > Hi, > > io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to > io-wq when that isn't possible. For a range of opcodes it isn't possible > at all, as there's no nonblocking path in the kernel for them: fsync, > statx, openat, the *at family, xattr, fadvise, splice, etc. Those are > punted unconditionally, and the punt costs a thread wakeup, a context > switch and a task_work completion round trip per request. io_uring HAS > to be cautious to prevent accidental blocking in the kernel, even if the > operations predominantly never block. Sad story. Examples of that are > things like an fdatasync that doesn't block, statx that hits dcache, > openat for O_TMPFILE, etc. All of those would've completed inline just > fine, but io_uring just cannot rely on that. > > This series issues those requests inline in blocking mode instead, and > only pays for the offload if the request actually blocks. But by the > time it blocks, the submitter is deep in the kernel with the request on > its stack, so the work can't be moved to another thread. What we can > move is the identity. If the submitting task blocks, an idle io-wq > worker takes over its user visible identity (tid, signal state, > credentials, scheduling attributes, cgroup, user register state), > finishes the io_uring_enter() call and returns to userspace as the > submitter. The original task finishes the request as an > io-wq worker and joins the pool. Userspace is none the wiser, hopefully, > the same tid came back from the syscall, it's just on a different > task_struct. Folks that have been around a while may remember earlier > attempts at this about 20 years ago. I don't see anything immediately wrong, but I suspect I am just not looking hard enough. In my time working with the kernel I have never seen anyone actually get this kind of thing correct. The handoff that we do during exec has a bug with posix timers that I think is 23 years old that we just caught, and still hasn't been merged to Linus. There was the old daemonize call that got it wrong so often I added kthreadd. Maybe you want something like the old solaris doors, or vfork. Perform a synchronous task switch to this other thread, and call this function in the other thread. Then block waiting on the other thread until the other thread blocks, or the function you called finishes. Is there a reason you didn't try and do it that way? Just a synchronous switch to and from a thread in your thread pool? You aren't changing the mm so I really doubt changing the stack pointer and a registers will be that expensive. Eric
On 9/18/26 10:33 PM, Eric W. Biederman wrote: > Jens Axboe <axboe@kernel.dk> writes: > >> Hi, >> >> io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to >> io-wq when that isn't possible. For a range of opcodes it isn't possible >> at all, as there's no nonblocking path in the kernel for them: fsync, >> statx, openat, the *at family, xattr, fadvise, splice, etc. Those are >> punted unconditionally, and the punt costs a thread wakeup, a context >> switch and a task_work completion round trip per request. io_uring HAS >> to be cautious to prevent accidental blocking in the kernel, even if the >> operations predominantly never block. Sad story. Examples of that are >> things like an fdatasync that doesn't block, statx that hits dcache, >> openat for O_TMPFILE, etc. All of those would've completed inline just >> fine, but io_uring just cannot rely on that. >> >> This series issues those requests inline in blocking mode instead, and >> only pays for the offload if the request actually blocks. But by the >> time it blocks, the submitter is deep in the kernel with the request on >> its stack, so the work can't be moved to another thread. What we can >> move is the identity. If the submitting task blocks, an idle io-wq >> worker takes over its user visible identity (tid, signal state, >> credentials, scheduling attributes, cgroup, user register state), >> finishes the io_uring_enter() call and returns to userspace as the >> submitter. The original task finishes the request as an >> io-wq worker and joins the pool. Userspace is none the wiser, hopefully, >> the same tid came back from the syscall, it's just on a different >> task_struct. Folks that have been around a while may remember earlier >> attempts at this about 20 years ago. > > I don't see anything immediately wrong, but I suspect I am just > not looking hard enough. > > In my time working with the kernel I have never seen anyone actually get > this kind of thing correct. > > The handoff that we do during exec has a bug with posix timers that > I think is 23 years old that we just caught, and still hasn't been > merged to Linus. > > There was the old daemonize call that got it wrong so often I added > kthreadd. I agree entirely with you, which is why this is (deeply) and RFC and I mostly pulled it to (some notion of) completion so I could run some testing and see how it performs. > Maybe you want something like the old solaris doors, or vfork. > Perform a synchronous task switch to this other thread, and call this > function in the other thread. Then block waiting on the other thread > until the other thread blocks, or the function you called finishes. > > Is there a reason you didn't try and do it that way? > Just a synchronous switch to and from a thread in your thread pool? > > You aren't changing the mm so I really doubt changing the stack pointer > and a registers will be that expensive. And replies like this are also why I wanted to get it out, because I think it's a problem worth solving, and it's the best way to solicit ideas. I think there's some potential in your suggestion, let me try and dig at it a little bit and experiment... I'll be back with more details. -- Jens Axboe
On Fri, Sep 11, 2026 at 09:40:50AM -0600, Jens Axboe wrote: > This series issues those requests inline in blocking mode instead, and > only pays for the offload if the request actually blocks. But by the > time it blocks, the submitter is deep in the kernel with the request on > its stack, so the work can't be moved to another thread. What we can > move is the identity. If the submitting task blocks, an idle io-wq > worker takes over its user visible identity (tid, signal state, > credentials, scheduling attributes, cgroup, user register state), > finishes the io_uring_enter() call and returns to userspace as the > submitter. The original task finishes the request as an > io-wq worker and joins the pool. Userspace is none the wiser, hopefully, > the same tid came back from the syscall, it's just on a different > task_struct. I'm still struggling my way through this thing. So thread T1 is doing this blocking syscall. When it actually blocks, you hand-over the userspace identifying part to another thread T2, which will return to userspace as if it were T1. Is this really a hand-over, or a swap? I would imagine we not have two threads with the same tid and all that. Also, I'm a wee bit confused, why not swap out the kernel stack with a io-wq worker and have the original thread return to userspace. This seems like a better defined situation. In so far as anything here is well defined. Swapping the kernel state seems like a simpler endeavour than swapping all that is or might be user visible. > Folks that have been around a while may remember earlier > attempts at this about 20 years ago. That was the whole threadlet thing, right? Very hazy memories of that. I'll have to go read that back, I'm sure I still have that in my inbox *somewhere*. Surely we abandoned that approach for a reason. ISTR there being some significant ick there, much like the thing you're proposing now. Not sure yet on how things compare. Will have to dig through that stuff again.
On 9/14/26 1:22 PM, Peter Zijlstra wrote: > On Fri, Sep 11, 2026 at 09:40:50AM -0600, Jens Axboe wrote: > >> This series issues those requests inline in blocking mode instead, and >> only pays for the offload if the request actually blocks. But by the >> time it blocks, the submitter is deep in the kernel with the request on >> its stack, so the work can't be moved to another thread. What we can >> move is the identity. If the submitting task blocks, an idle io-wq >> worker takes over its user visible identity (tid, signal state, >> credentials, scheduling attributes, cgroup, user register state), >> finishes the io_uring_enter() call and returns to userspace as the >> submitter. The original task finishes the request as an >> io-wq worker and joins the pool. Userspace is none the wiser, hopefully, >> the same tid came back from the syscall, it's just on a different >> task_struct. > > I'm still struggling my way through this thing. I don't blame you. > So thread T1 is doing this blocking syscall. When it actually blocks, > you hand-over the userspace identifying part to another thread T2, which > will return to userspace as if it were T1. Correct. > Is this really a hand-over, or a swap? I would imagine we not have two > threads with the same tid and all that. It's a swap, yes we won't have two threads with the same tid. > Also, I'm a wee bit confused, why not swap out the kernel stack with a > io-wq worker and have the original thread return to userspace. This > seems like a better defined situation. In so far as anything here is > well defined. > > Swapping the kernel state seems like a simpler endeavour than swapping > all that is or might be user visible. I think we'd just be trading one set of problems for another. In terms of the kernel side, we have the following set of issues around swapping the kernel stack instead: - 'current' itself, this is used throughout the waiting helpers. Things like wait_queue_entries that use private = current. - Any kind of sleeping lock that also stashes away the value of 'current;. - signal interrupible sleeps via TASK_KILLABLE - Just like the userspace side, we have a bunch of kernel side state as well in the task_struct. Not a complete list, but: - plug, for any kind of in-progress IO submission state - journal_info - reclaim_state - io_uring - PF_MEMALLOC and friends - task_work - lockdep - preempt/irq stuff in thread_info - Kernel stack itself in ways it's tied to the task_struct - stack canary - KASAN - vma tracking - thread.sp - pt_regs Probably not a complete list, but it's a start... I do agree that the kernel stack swap seems like the more immediate idea, but I think it's a harder problem. >> Folks that have been around a while may remember earlier >> attempts at this about 20 years ago. > > That was the whole threadlet thing, right? Very hazy memories of that. > I'll have to go read that back, I'm sure I still have that in my inbox > *somewhere*. Right, and it's actually pretty related to what we're discussing above. The first one was Zach's fibrils which attempted the kernel stack swap, and then we had threadlets from Ingo which went the route that this patch set is also taking and dealt with userspace state swaps instead. > Surely we abandoned that approach for a reason. ISTR there being some > significant ick there, much like the thing you're proposing now. Well like you that state has long since been swapped out on my end, my recollection was disagreement on the API, not the swapping itself. But also extremely hazy there... > Not sure yet on how things compare. Will have to dig through that stuff > again. Let me know if you find something interesting there. -- Jens Axboe
Jens Axboe <axboe@kernel.dk> writes: > Hi, > > io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to > io-wq when that isn't possible. For a range of opcodes it isn't possible > at all, as there's no nonblocking path in the kernel for them: fsync, > statx, openat, the *at family, xattr, fadvise, splice, etc. Those are > punted unconditionally, and the punt costs a thread wakeup, a context > switch and a task_work completion round trip per request. io_uring HAS > to be cautious to prevent accidental blocking in the kernel, even if the > operations predominantly never block. Sad story. Examples of that are > things like an fdatasync that doesn't block, statx that hits dcache, > openat for O_TMPFILE, etc. All of those would've completed inline just > fine, but io_uring just cannot rely on that. > > This series issues those requests inline in blocking mode instead, and > only pays for the offload if the request actually blocks. But by the > time it blocks, the submitter is deep in the kernel with the request on > its stack, so the work can't be moved to another thread. What we can > move is the identity. If the submitting task blocks, an idle io-wq > worker takes over its user visible identity (tid, signal state, > credentials, scheduling attributes, cgroup, user register state), > finishes the io_uring_enter() call and returns to userspace as the > submitter. The original task finishes the request as an > io-wq worker and joins the pool. Userspace is none the wiser, hopefully, > the same tid came back from the syscall, it's just on a different > task_struct. Folks that have been around a while may remember earlier > attempts at this about 20 years ago. This is both really cool and seems like very dangerous thing :) Count me amazed. I worry this impersonating method will become as tricky as the kthread impersonating model that you replaced with the user workers, though. I haven't looked at your patches yet, but I wonder how you handle other tasks that have a reference to your task_struct. I was actually working something much simpler to improve this problem, which still require subsystems to cooperate, but largely reduces issue: My idea was to reuse the non_block_count which already exists in task_struct preserved for every kernel config that has io_uring. We we scope the inline path with it. We then provide new mutex, semaphore callers that will check the flag and fail refusing to sleep, similar to a try_lock. The new callers are required because we want subsystems to opt-in the behavior, properly clean after themselves, and return EWOULDBLOCK. This is why we need to clean blocking paths in io_uring. sched throws a WARN_ON if we schedule out with the counter> 0, making it easy to find issues. It has the downside of still requiring fixes to every path and we need to handle every new case that comes by, but it is much cleaner than plumbing a nonblock flag several layers down the stack across each subsystem or having subsystem-specific details in io_uring, which is what we have today. On the upper side, it is much less complex than your approach. It also allow us to just back off during memory allocations that would block, solving the memory allocations anywhere in the submission path, not only inside ->issue(), which we discussed recently on discord. I'll give a try to this series and report back. > > We catch the blocking through a scheduler hook. A task in a blocking > inline issue carries PF_IO_HANDOFF, and sched_submit_work() calls into > io_uring for it, next to the existing io-wq and workqueue hooks. This is > where the identity is handed off, with the uring_lock still held by the > blocking task and released for the promoted worker on its behalf. > > Structure of the series: > > 1 kernel: the thread identity handoff itself, in > kernel/thread_handoff.c. Independent of io_uring. > 2 sched: the PF_IO_HANDOFF hook. > 3-4 arm64 and x86 support. The arch hooks sync live register state > before the source blocks and load it on the destination. > 5-9 io_uring prep: helper cleanups, a tctx node list, tracking of > uring_lock sections in the issue path so the hook knows when > the lock may be dropped, splitting io_uring_enter() so it can > be resumed by another task, and keeping the block plug on the > submitter's stack. > 10 io-wq: claiming an idle worker for a handoff, and keeping a > couple of idle spares around as targets. > 11-12 io_uring: the handoff itself, and deferring the identity > migration to the end of the submission so a batch of blocking > SQEs costs one migration rather than one per SQE. > 13 io_uring: issue blockable requests inline in blocking mode. > This is where behavior changes. > 14 tracepoints. > 15 treat IOSQE_ASYNC the same way. Separate as it's a userspace > visible policy change, and I'm not sure yet it should be done. > > Some things are refused for handoff up front: traced tasks, per-task > perf contexts, PI futexes, audit contexts, armed per-thread CPU timers, > core scheduling cookies, vfork parents. Per-thread accounting moves with > the handoff, so the counters userspace sees for a tid stay monotonic. > Known gaps are LSM state kept in the task rather than the cred, and > PR_SET_IO_FLUSHER. Neither moves, and not moving them can only ever > restrict. > > Reads and writes on files with FMODE_NOWAIT are excluded. > They have a working nonblocking path and poll retry, and a handoff per > op would be worse than that. The handoff covers requests that previously > would always have been handed to io-wq upfront. uring_cmd is excluded > too, as drivers like ublk bind state to the submitting task. > > Some test results: > > Measured in a virtme-ng guest, 8 vcpu, non-debug x86 config, same > kernel with a sysctl toggle for turning the feature on and off. CPU > is the usage of the whole process including io-wq workers, as a > percentage of one CPU. Mean of two runs. > > ops/s cpu > ===================================================== > fsync, tmpfs, qd 1 > baseline 28.2k 96% > handoff 221k 100% > change +681% +5% > fsync, tmpfs, qd 8 > baseline 195k 141% > handoff 526k 100% > change +170% -29% > fsync, tmpfs, qd 32 > baseline 357k 219% > handoff 611k 100% > change +71% -54% > > fsync, ext4 (flushes, always blocks), qd 1 > baseline 9.6k 66% > handoff 7.1k 91% > change -26% +37% > fsync, ext4 (flushes, always blocks), qd 8 > baseline 29.8k 212% > handoff 14.7k 146% > change -51% -31% > fsync, ext4 (flushes, always blocks), qd 32 > baseline 42.2k 270% > handoff 14.6k 144% > change -65% -47% > > statx, ext4, qd 1 > baseline 23.6k 96% > handoff 121k 100% > change +414% +5% > statx, ext4, qd 8 > baseline 129k 234% > handoff 179k 100% > change +38% -57% > statx, ext4, qd 32 > baseline 197k 276% > handoff 140k 100% > change -29% -64% > > statx, tmpfs, qd 1 > baseline 24.4k 96% > handoff 117k 100% > change +378% +4% > statx, tmpfs, qd 8 > baseline 131k 232% > handoff 180k 100% > change +37% -57% > statx, tmpfs, qd 32 > baseline 219k 295% > handoff 190k 100% > change -13% -66% > > fadvise DONTNEED, ext4, qd 1 > baseline 26.1k 95% > handoff 150k 100% > change +478% +5% > fadvise DONTNEED, ext4, qd 8 > baseline 127k 170% > handoff 252k 100% > change +99% -41% > fadvise DONTNEED, ext4, qd 32 > baseline 270k 234% > handoff 273k 100% > change +1% -57% > > renameat, ext4, qd 1 > baseline 7.4k 98% > handoff 13.4k 100% > change +81% +2% > renameat, ext4, qd 8 > baseline 12.0k 544% > handoff 14.6k 100% > change +21% -82% > renameat, ext4, qd 32 > baseline 11.2k 500% > handoff 14.3k 100% > change +28% -80% > > renameat, tmpfs, qd 1 > baseline 15.0k 97% > handoff 48.5k 99% > change +223% +2% > renameat, tmpfs, qd 8 > baseline 65.6k 514% > handoff 55.4k 98% > change -15% -81% > renameat, tmpfs, qd 32 > baseline 39.4k 462% > handoff 38.9k 98% > change -1% -79% > > openat O_TMPFILE, ext4, qd 1 > baseline 6.3k 100% > handoff 11.3k 100% > change +81% +1% > openat O_TMPFILE, ext4, qd 8 > baseline 23.0k 258% > handoff 12.9k 99% > change -44% -62% > openat O_TMPFILE, ext4, qd 32 > baseline 26.2k 248% > handoff 13.4k 99% > change -49% -60% > > openat O_TMPFILE, tmpfs, qd 1 > baseline 13.4k 95% > handoff 43.0k 98% > change +222% +3% > openat O_TMPFILE, tmpfs, qd 8 > baseline 57.8k 220% > handoff 51.7k 96% > change -11% -56% > openat O_TMPFILE, tmpfs, qd 32 > baseline 69.3k 218% > handoff 56.8k 96% > change -18% -56% > > splice to pipe, ext4, qd 1 > baseline 19.4k 96% > handoff 21.1k 98% > change +8% +2% > splice to pipe, ext4, qd 8 > baseline 48.2k 176% > handoff 48.4k 176% > change +0% +0% > splice to pipe, ext4, qd 32 > baseline 53.3k 188% > handoff 48.1k 182% > change -10% -3% > > As you can tell, normal QD=1 type issues see big wins. Conversely, for > higher queue depth, there are losses. The losses are generally from one > of two reasons: > > 1) The syscall part is fairly expensive, and previously we farmed all of > this work across a bunch of io-wq workers, and the parallelization > there helps performance. For QD=1 that obviously isn't the case. The > more expensive the lower level kernel parts are, the lower the QD > required to see a perf loss. openat is the obvious worse case for this. > > 2) The syscall part ALWAYS blocks. For this case, io-wq is going to be > quicker, just punt the opcode upfront. fsync on ext4, as shown in the table > above, is indeed that case. Each of those block. > > I've got some ideas for how to mitigate the losses for higher queue > depths, but a) I didn't think they were THAT interesting for an RFC, as > low QD is generally what people do with these kinds of ops, and b) the > main concept behind this handoff is really the interesting part right > now. > > Passes the full liburing test suite, on both x86-64 and arm64. Other > archs don't support this yet. > > This is obviously an RFC, in terms of what I'd love people to take a > closer look at: > > - The scheduler hook and the identity move itself, kernel/thread_handoff.c. > Is the set of refused states complete enough, and is moving > thread group leadership this way (the leader must stay first on > ->thread_head, like de_thread() keeps it) acceptable / kosher. > - The x86 and arm64 register state handling. x86 refuses AMX users, > and arm64 refuses SME. > - Whether anyone relies on a task's user identity staying on one > task_struct in ways not covered above in the series. > > Patches are against 7.3-rc2. Also available at: > > git://git.kernel.dk/linux.git io_uring-thread-handoff.3 > > arch/Kconfig | 7 + > arch/arm64/Kconfig | 1 + > arch/arm64/kernel/process.c | 109 +++++++ > arch/x86/Kconfig | 1 + > arch/x86/kernel/process.c | 10 +- > arch/x86/kernel/process_64.c | 139 +++++++++ > include/linux/io_uring.h | 9 + > include/linux/io_uring_types.h | 48 +++- > include/linux/sched.h | 2 +- > include/linux/thread_handoff.h | 76 +++++ > include/trace/events/io_uring.h | 114 ++++++++ > init/Kconfig | 11 + > io_uring/Makefile | 1 + > io_uring/handoff.c | 395 +++++++++++++++++++++++++ > io_uring/handoff.h | 103 +++++++ > io_uring/io-wq.c | 270 +++++++++++++++++- > io_uring/io-wq.h | 15 + > io_uring/io_uring.c | 310 ++++++++++++++------ > io_uring/io_uring.h | 46 ++- > io_uring/kbuf.c | 5 +- > io_uring/msg_ring.c | 2 +- > io_uring/opdef.c | 29 ++ > io_uring/opdef.h | 2 + > io_uring/rw.c | 2 +- > io_uring/splice.c | 6 + > io_uring/tctx.c | 16 +- > io_uring/tctx.h | 2 + > io_uring/tw.c | 14 +- > io_uring/uring_cmd.c | 5 +- > kernel/Makefile | 1 + > kernel/fork.c | 6 +- > kernel/sched/core.c | 36 +++ > kernel/thread_handoff.c | 490 ++++++++++++++++++++++++++++++++ > 33 files changed, 2167 insertions(+), 116 deletions(-) > > -- > Jens Axboe > -- Gabriel Krisman Bertazi
On 9/11/26 11:33 AM, Gabriel Krisman Bertazi wrote: > Jens Axboe <axboe@kernel.dk> writes: > >> Hi, >> >> io_uring issues requests inline with IO_URING_F_NONBLOCK and punts to >> io-wq when that isn't possible. For a range of opcodes it isn't possible >> at all, as there's no nonblocking path in the kernel for them: fsync, >> statx, openat, the *at family, xattr, fadvise, splice, etc. Those are >> punted unconditionally, and the punt costs a thread wakeup, a context >> switch and a task_work completion round trip per request. io_uring HAS >> to be cautious to prevent accidental blocking in the kernel, even if the >> operations predominantly never block. Sad story. Examples of that are >> things like an fdatasync that doesn't block, statx that hits dcache, >> openat for O_TMPFILE, etc. All of those would've completed inline just >> fine, but io_uring just cannot rely on that. >> >> This series issues those requests inline in blocking mode instead, and >> only pays for the offload if the request actually blocks. But by the >> time it blocks, the submitter is deep in the kernel with the request on >> its stack, so the work can't be moved to another thread. What we can >> move is the identity. If the submitting task blocks, an idle io-wq >> worker takes over its user visible identity (tid, signal state, >> credentials, scheduling attributes, cgroup, user register state), >> finishes the io_uring_enter() call and returns to userspace as the >> submitter. The original task finishes the request as an >> io-wq worker and joins the pool. Userspace is none the wiser, hopefully, >> the same tid came back from the syscall, it's just on a different >> task_struct. Folks that have been around a while may remember earlier >> attempts at this about 20 years ago. > > This is both really cool and seems like very dangerous thing :) Count me Oh yeah, it's definitely crazy and deeply an RFC. > amazed. I worry this impersonating method will become as tricky as the > kthread impersonating model that you replaced with the user workers, > though. I haven't looked at your patches yet, but I wonder how you > handle other tasks that have a reference to your task_struct. That one was different, because these are normal threads, not kthreads. They are created similarly to if you did pthread_create() in userspace, this is what io-wq workers are already. So it's mostly as safe as io-wq already is, by design, which is why the PF_IO_WORKER work happened and why kthreads haven't been used since back in the early 5.x days. So I don't think there's too much to worry about on the security front, it's mostly a "this will confuse the application" kind of thing because something has been missed. And yes that is no good either, but it's not a security concern. That's VERY different from the kthread case, where if you missed some kind of personality, then congrats you're now running with fully elevated privileges. > I was actually working something much simpler to improve this problem, > which still require subsystems to cooperate, but largely reduces issue: > > My idea was to reuse the non_block_count which already exists in > task_struct preserved for every kernel config that has io_uring. We we > scope the inline path with it. We then provide new mutex, semaphore > callers that will check the flag and fail refusing to sleep, similar to > a try_lock. The new callers are required because we want subsystems to > opt-in the behavior, properly clean after themselves, and return > EWOULDBLOCK. This is why we need to clean blocking paths in io_uring. > sched throws a WARN_ON if we schedule out with the counter> 0, making it > easy to find issues. I think that would be a tough sell, mostly because of how many locking primitives we have and how widely they are used, and how difficult (or impossible) it is to introduce error paths for code that previously had none. That alone would make it a non-starter for me. Let alone is that it'd be a continual whack-a-mole kind of work, it'll never be fully done. > It has the downside of still requiring fixes to every path and we need > to handle every new case that comes by, but it is much cleaner than > plumbing a nonblock flag several layers down the stack across each > subsystem or having subsystem-specific details in io_uring, which is > what we have today. On the upper side, it is much less complex than > your approach. It also allow us to just back off during memory > allocations that would block, solving the memory allocations anywhere in > the submission path, not only inside ->issue(), which we discussed > recently on discord. I think you'll find it'll be a lot MORE complicated than my approach! Backing out error handling is going to be impossible in some cases, think file systems for example. How would those cases be handled? > I'll give a try to this series and report back. Thanks! -- Jens Axboe
Jens Axboe <axboe@kernel.dk> writes: > On 9/11/26 11:33 AM, Gabriel Krisman Bertazi wrote: >> It has the downside of still requiring fixes to every path and we need >> to handle every new case that comes by, but it is much cleaner than >> plumbing a nonblock flag several layers down the stack across each >> subsystem or having subsystem-specific details in io_uring, which is >> what we have today. On the upper side, it is much less complex than >> your approach. It also allow us to just back off during memory >> allocations that would block, solving the memory allocations anywhere in >> the submission path, not only inside ->issue(), which we discussed >> recently on discord. > > I think you'll find it'll be a lot MORE complicated than my approach! > Backing out error handling is going to be impossible in some cases, > think file systems for example. How would those cases be handled? I understand there are many cases where it would be impossible, most, if not all of them, involving FS. But I naively imagine we could back-off those early, before we get to the critical session, without even trying the nonblock approach, similar to what we do now, and punt to the io-wq, which is not going away anyway. What I'd like to solve is drop the logic of other parts of the kernel that we need to keep in the io_uring layer, such as which network protocols will block and which won't. It is a hard sell, yes. And it does gives off vibes of re-editing the PF_MEMALLOC_NORECLAIM controversy. Long term, though, it would be much better for the kernel that can reliably handle failure and back-off than keep going, including for predictable performance. -- Gabriel Krisman Bertazi
On 9/22/26 4:05 PM, Gabriel Krisman Bertazi wrote: > Jens Axboe <axboe@kernel.dk> writes: > >> On 9/11/26 11:33 AM, Gabriel Krisman Bertazi wrote: >>> It has the downside of still requiring fixes to every path and we need >>> to handle every new case that comes by, but it is much cleaner than >>> plumbing a nonblock flag several layers down the stack across each >>> subsystem or having subsystem-specific details in io_uring, which is >>> what we have today. On the upper side, it is much less complex than >>> your approach. It also allow us to just back off during memory >>> allocations that would block, solving the memory allocations anywhere in >>> the submission path, not only inside ->issue(), which we discussed >>> recently on discord. >> >> I think you'll find it'll be a lot MORE complicated than my approach! >> Backing out error handling is going to be impossible in some cases, >> think file systems for example. How would those cases be handled? > > I understand there are many cases where it would be impossible, most, if > not all of them, involving FS. But I naively imagine we could back-off > those early, before we get to the critical session, without even trying > the nonblock approach, similar to what we do now, and punt to the io-wq, > which is not going away anyway. What I'd like to solve is drop the > logic of other parts of the kernel that we need to keep in the io_uring > layer, such as which network protocols will block and which won't. The problem is that you don't always know until you're at the point of no return. If it was that simple, we would not be talking about these patches :-) -- Jens Axboe
© 2016 - 2026 Red Hat, Inc.