drivers/base/core.c | 1 + drivers/iommu/dma-iommu.c | 9 +- include/linux/netdevice.h | 21 +++ include/linux/skbuff.h | 7 +- include/linux/swiotlb.h | 63 ++++++++ include/net/sock.h | 46 ++++++ kernel/dma/direct.h | 11 ++ kernel/dma/swiotlb.c | 296 ++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 61 +++++++- net/core/page_pool.c | 25 +++- net/core/sock.c | 101 +++++++++++-- 11 files changed, 617 insertions(+), 24 deletions(-)
The use of swiotlb, common in Confidential Computing, causes an extra data copy on each I/O. Focusing on network sockets: - on tx, the copy has a high chance of happening in the tx softirq handler (especially with greedy senders where the device queue is often full) - on rx, it is guaranteed to happen in the rx softirq handler. Thus, on top of the copy cost, swiotlb concentrates the overhead on an already constrained resource (CPUs processing network interrupts). Reduce or remove the extra copy by conditionally allocating socket buffers directly from the swiotlb buffer pool. The feature is controlled by runtime parameters to set the percentage of swiotlb buffers that can be used for this purpose. This avoids stranding the entire swiotlb pool in socket buffers. The implementation is made of four main parts: - introduce a swiotlb page allocator that can be used instead of regular pages, and teach __free_frozen_pages(), free_unref_folio() how to handle them - dynamically track the leaf device for each tx network socket, so we can tell at copy_from_user() time whether we need to use swiotlb for this socket - modify skb_page_frag_refill() to allocate from swiotlb if needed. This implements the copy elision for the transmit path - modify __page_pool_alloc_page_order() to allocate from swiotlb if needed. This implements the copy elision for the receive path. The savings are especially visible with fewer queues. In synthetic benchmarks, senders with 1-2 queues would cap around 50Gbps with conventional swiotlb, and reach over 170Gbps with the feature enabled. OPEN ISSUES Currently the swiotlb allocator looks for free slots using an approximately linear scan of each pool (with some hints to likely candidates) and then does a linear scan of subsequent pools. This works extremely well when the number of pools matches the number of CPUs, and there is plenty of memory available. In fact, it is almost unbeatable by any more complex strategy. Under high load or buffer fragmentation, a CPU might repeatedly do a full scan of its starting pool before finding a suitable candidate. Even worse, with multiple tx/rx queues, what happens is that multiple CPUs will trail each other on the same sequence of pools. The effect is that some allocations will end up costing O(100us) and more. I have tried to implement two improvements: - a buddy allocator on top of each pool, so to make it quicker to find a candidate of the requested size - make each CPU use a different sequence to explore other pools in case one is full, so they will not end up queueing one after the other While they are very effective on the tails, for low load scenarios the current linear allocators is better. Thus this will take more investigation. --- v1 -> v2: - split components into separate commits - simplified allocator, no need for a new page type - many code cleanups - also implement the rx side Luigi Rizzo (5): swiotlb: enforce pool nareas and nslabs invariants swiotlb/mm: Implement SWIOTLB nocopy page allocator net/swiotlb: Track bounce device per socket net: Divert socket allocations to SWIOTLB for nocopy TX swiotlb: Implement RX nocopy with fast recycling eviction drivers/base/core.c | 1 + drivers/iommu/dma-iommu.c | 9 +- include/linux/netdevice.h | 21 +++ include/linux/skbuff.h | 7 +- include/linux/swiotlb.h | 63 ++++++++ include/net/sock.h | 46 ++++++ kernel/dma/direct.h | 11 ++ kernel/dma/swiotlb.c | 296 ++++++++++++++++++++++++++++++++++++-- mm/page_alloc.c | 61 +++++++- net/core/page_pool.c | 25 +++- net/core/sock.c | 101 +++++++++++-- 11 files changed, 617 insertions(+), 24 deletions(-) -- 2.55.0.766.g2966f0265a-goog
On 24.08.26 17:29, Luigi Rizzo wrote: > The use of swiotlb, common in Confidential Computing, causes an extra > data copy on each I/O. Focusing on network sockets: > - on tx, the copy has a high chance of happening in the tx softirq handler > (especially with greedy senders where the device queue is often full) > - on rx, it is guaranteed to happen in the rx softirq handler. > Thus, on top of the copy cost, swiotlb concentrates the overhead on an > already constrained resource (CPUs processing network interrupts). > > Reduce or remove the extra copy by conditionally allocating socket buffers > directly from the swiotlb buffer pool. > Isn't it dangerous for RX to expose kernel structures to the HV? If SKB the linear area is exposed to the HW, the headroom and tailroom are up for grabs for the HV: the HV could modify them in a TOCTOU fashioon. > The feature is controlled by runtime parameters to set the percentage of > swiotlb buffers that can be used for this purpose. This avoids stranding > the entire swiotlb pool in socket buffers. > > The implementation is made of four main parts: > - introduce a swiotlb page allocator that can be used instead of > regular pages, and teach __free_frozen_pages(), free_unref_folio() > how to handle them > - dynamically track the leaf device for each tx network socket, > so we can tell at copy_from_user() time whether we need to use > swiotlb for this socket > - modify skb_page_frag_refill() to allocate from swiotlb if needed. > This implements the copy elision for the transmit path > - modify __page_pool_alloc_page_order() to allocate from swiotlb if needed. > This implements the copy elision for the receive path. > > The savings are especially visible with fewer queues. In synthetic > benchmarks, senders with 1-2 queues would cap around 50Gbps with > conventional swiotlb, and reach over 170Gbps with the feature enabled. > > OPEN ISSUES > > Currently the swiotlb allocator looks for free slots using an > approximately linear scan of each pool (with some hints to likely > candidates) and then does a linear scan of subsequent pools. > This works extremely well when the number of pools matches the number of > CPUs, and there is plenty of memory available. In fact, it is almost > unbeatable by any more complex strategy. > > Under high load or buffer fragmentation, a CPU might repeatedly do a > full scan of its starting pool before finding a suitable candidate. > Even worse, with multiple tx/rx queues, what happens is that multiple CPUs > will trail each other on the same sequence of pools. The effect is that > some allocations will end up costing O(100us) and more. I encountered this as well: even with maxed out swiotlb memory the page_pool will suck a lot of pages from there. And TX allocations are left scrambling for scraps. Why can't we create per device pools instead on relying on the swiotb? > > I have tried to implement two improvements: > - a buddy allocator on top of each pool, so to make it quicker to find a > candidate of the requested size > - make each CPU use a different sequence to explore other pools in case > one is full, so they will not end up queueing one after the other > While they are very effective on the tails, for low load scenarios the > current linear allocators is better. Thus this will take more > investigation. > > --- > v1 -> v2: > > - split components into separate commits > - simplified allocator, no need for a new page type > - many code cleanups > - also implement the rx side > > Luigi Rizzo (5): > swiotlb: enforce pool nareas and nslabs invariants > swiotlb/mm: Implement SWIOTLB nocopy page allocator > net/swiotlb: Track bounce device per socket > net: Divert socket allocations to SWIOTLB for nocopy TX > swiotlb: Implement RX nocopy with fast recycling eviction > > drivers/base/core.c | 1 + > drivers/iommu/dma-iommu.c | 9 +- > include/linux/netdevice.h | 21 +++ > include/linux/skbuff.h | 7 +- > include/linux/swiotlb.h | 63 ++++++++ > include/net/sock.h | 46 ++++++ > kernel/dma/direct.h | 11 ++ > kernel/dma/swiotlb.c | 296 ++++++++++++++++++++++++++++++++++++-- > mm/page_alloc.c | 61 +++++++- > net/core/page_pool.c | 25 +++- > net/core/sock.c | 101 +++++++++++-- > 11 files changed, 617 insertions(+), 24 deletions(-) >
syzbot ci has tested the following series [v2] swiotlb: avoid swiotlb copy on network sockets https://lore.kernel.org/all/20260824152932.1583506-1-lrizzo@google.com * [PATCH v2 1/5] swiotlb: enforce pool nareas and nslabs invariants * [PATCH v2 2/5] swiotlb/mm: Implement SWIOTLB nocopy page allocator * [PATCH v2 3/5] net/swiotlb: Track bounce device per socket * [PATCH v2 4/5] net: Divert socket allocations to SWIOTLB for nocopy TX * [PATCH v2 5/5] swiotlb: Implement RX nocopy with fast recycling eviction and found the following issue: general protection fault in page_pool_put_unrefed_netmem Full report is available here: https://ci.syzbot.org/series/b17756c4-983f-4d7a-8415-3da06c0dda99 *** general protection fault in page_pool_put_unrefed_netmem tree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: 0a0d1d55dad570724bf8c7ea83409639cfb4be9b arch: amd64 compiler: Debian clang version 22.1.8 (++20260613092233+e80beda6e255-1~exp1~20260613092250.77), Debian LLD 22.1.8 config: https://ci.syzbot.org/builds/20e51ca2-88da-4674-9797-a62c64743095/config syz repro: https://ci.syzbot.org/findings/4385dd7f-d1fc-4627-ad19-6862c780a90c/syz_repro Oops: general protection fault, probably for non-canonical address 0xdffffc0000000082: 0000 [#1] SMP KASAN PTI KASAN: null-ptr-deref in range [0x0000000000000410-0x0000000000000417] CPU: 0 UID: 0 PID: 5836 Comm: syz.2.19 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:swiotlb_addr_in_default_pool include/linux/swiotlb.h:177 [inline] RIP: 0010:swiotlb_is_nocopy_addr include/linux/swiotlb.h:184 [inline] RIP: 0010:__page_pool_put_page net/core/page_pool.c:851 [inline] RIP: 0010:page_pool_put_unrefed_netmem+0x22a/0xd20 net/core/page_pool.c:930 Code: 5c 63 00 00 89 c5 31 ff 89 c6 e8 61 17 04 f8 85 ed 0f 84 07 08 00 00 e8 54 12 04 f8 49 81 c4 10 04 00 00 4c 89 e0 48 c1 e8 03 <42> 80 3c 28 00 74 08 4c 89 e7 e8 37 c7 73 f8 4d 8b 24 24 4d 85 e4 RSP: 0018:ffffc900039df230 EFLAGS: 00010206 RAX: 0000000000000082 RBX: 0000000000000000 RCX: ffff888160efda00 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: 0000000000000001 R08: ffffffff89c3aca5 R09: 0000000000000000 R10: 0000000000000000 R11: ffffffff8ed5a4e0 R12: 0000000000000410 R13: dffffc0000000000 R14: ffffea0004450300 R15: 0000000004450300 FS: 00007f08f83e86c0(0000) GS:ffff88818d6fc000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffd455a8fe8 CR3: 00000001134ac000 CR4: 00000000000006f0 Call Trace: <TASK> page_pool_put_netmem include/net/page_pool/helpers.h:343 [inline] page_pool_put_full_netmem include/net/page_pool/helpers.h:373 [inline] napi_pp_put_page+0x133/0x1e0 net/core/skbuff.c:1036 skb_pp_recycle net/core/skbuff.c:1047 [inline] skb_free_head net/core/skbuff.c:1091 [inline] skb_release_data+0x8c3/0xab0 net/core/skbuff.c:1122 skb_release_all net/core/skbuff.c:1197 [inline] __kfree_skb+0x5d/0x210 net/core/skbuff.c:1211 llc_rcv+0x77a/0xcb0 net/llc/llc_input.c:-1 __netif_receive_skb_list_ptype net/core/dev.c:6312 [inline] __netif_receive_skb_list_core+0x784/0x830 net/core/dev.c:6354 __netif_receive_skb_list net/core/dev.c:6406 [inline] netif_receive_skb_list_internal+0x995/0xcf0 net/core/dev.c:6497 netif_receive_skb_list+0x67/0x480 net/core/dev.c:6549 xdp_recv_frames net/bpf/test_run.c:268 [inline] xdp_test_run_batch net/bpf/test_run.c:349 [inline] bpf_test_run_xdp_live+0x1875/0x1c20 net/bpf/test_run.c:378 bpf_prog_test_run_xdp+0x7d8/0x11d0 net/bpf/test_run.c:1463 bpf_prog_test_run+0x2c5/0x340 kernel/bpf/syscall.c:4804 __sys_bpf+0xa22/0xd90 kernel/bpf/syscall.c:6385 __do_sys_bpf kernel/bpf/syscall.c:6486 [inline] __se_sys_bpf kernel/bpf/syscall.c:6483 [inline] __x64_sys_bpf+0xba/0xd0 kernel/bpf/syscall.c:6483 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f08f759e0d9 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f08f83e8028 EFLAGS: 00000246 ORIG_RAX: 0000000000000141 RAX: ffffffffffffffda RBX: 00007f08f7825fa0 RCX: 00007f08f759e0d9 RDX: 0000000000000048 RSI: 0000200000000600 RDI: 000000000000000a RBP: 00007f08f7635024 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f08f7826038 R14: 00007f08f7825fa0 R15: 00007fffc4c69778 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- RIP: 0010:swiotlb_addr_in_default_pool include/linux/swiotlb.h:177 [inline] RIP: 0010:swiotlb_is_nocopy_addr include/linux/swiotlb.h:184 [inline] RIP: 0010:__page_pool_put_page net/core/page_pool.c:851 [inline] RIP: 0010:page_pool_put_unrefed_netmem+0x22a/0xd20 net/core/page_pool.c:930 Code: 5c 63 00 00 89 c5 31 ff 89 c6 e8 61 17 04 f8 85 ed 0f 84 07 08 00 00 e8 54 12 04 f8 49 81 c4 10 04 00 00 4c 89 e0 48 c1 e8 03 <42> 80 3c 28 00 74 08 4c 89 e7 e8 37 c7 73 f8 4d 8b 24 24 4d 85 e4 RSP: 0018:ffffc900039df230 EFLAGS: 00010206 RAX: 0000000000000082 RBX: 0000000000000000 RCX: ffff888160efda00 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: 0000000000000001 R08: ffffffff89c3aca5 R09: 0000000000000000 R10: 0000000000000000 R11: ffffffff8ed5a4e0 R12: 0000000000000410 R13: dffffc0000000000 R14: ffffea0004450300 R15: 0000000004450300 FS: 00007f08f83e86c0(0000) GS:ffff88818d6fc000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffd455a8fe8 CR3: 00000001134ac000 CR4: 00000000000006f0 ---------------- Code disassembly (best guess), 2 bytes skipped: 0: 00 00 add %al,(%rax) 2: 89 c5 mov %eax,%ebp 4: 31 ff xor %edi,%edi 6: 89 c6 mov %eax,%esi 8: e8 61 17 04 f8 call 0xf804176e d: 85 ed test %ebp,%ebp f: 0f 84 07 08 00 00 je 0x81c 15: e8 54 12 04 f8 call 0xf804126e 1a: 49 81 c4 10 04 00 00 add $0x410,%r12 21: 4c 89 e0 mov %r12,%rax 24: 48 c1 e8 03 shr $0x3,%rax * 28: 42 80 3c 28 00 cmpb $0x0,(%rax,%r13,1) <-- trapping instruction 2d: 74 08 je 0x37 2f: 4c 89 e7 mov %r12,%rdi 32: e8 37 c7 73 f8 call 0xf873c76e 37: 4d 8b 24 24 mov (%r12),%r12 3b: 4d 85 e4 test %r12,%r12 *** If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message: Tested-by: syzbot@syzkaller.appspotmail.com --- This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at syzkaller@googlegroups.com. To test a fix for this bug, please reply with `#syz test` (on a separate line) and attach the patch to the email. Notes: - The patch will be applied on top of the tested series (as an incremental fix). - To test a new version of the whole series, please send it directly to syzbot@lists.linux.dev. - Arguments like custom git repos and branches are not supported.
© 2016 - 2026 Red Hat, Inc.