From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F1294F68AB for ; Wed, 14 Oct 2026 11:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; cv=none; b=ibfzpUx4iAl7jpcyE0Ax18j98GYgFhsf0M8yYY2oZpxCZASv2b+GW4M4GAhfvn8rGiB272XuDhWWF9iuYms6ZKEzJjFgDBpGAMeWrHtAmlagx41HH/1OzjzhsaHtqpzz2NxoMPFLg9kXYibNlPCms3zO+x8rsWyowOa7i3sVzGs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; c=relaxed/simple; bh=J5l2XSVFm0v/kjrCLEIjTJDoAt/lho1+MPKngAjlObQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=kxyjLNu87VqV4JYytGnW2wrEyySA4FM5lCAJM2lgeUkWZiY+i5AqSCJ+cIclfeg/gzrG7IyyNF2/uayl9wnP4coilu01H4Hxb/G+8yfXpbHQgbOFtiDl8MATQonrcg0ps25gKORzI3uaKP8Dlj/RPNmXK6B+v5DOt8Qzf1M4+lk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XKKvN0xz; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XKKvN0xz" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-49e73611928so37265425e9.1 for ; Wed, 14 Oct 2026 04:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975741; x=1792580541; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=2mkSqDyENes6nJoSVFzqdx//VmH5Dkg6GQfLEw4/f/g=; b=XKKvN0xzMfeIkcAoWL/9Us/0xD9CVcvnaMDo9y+EHJjh3ENZKvvO62qznFaIyh5hBU 1+vPL2gfmHqoCKlEfYsGUkIdZ430MAP1OQcm+CBMcEawwIz4ChXmt4p/gfsCzPnCGdpM XJz1Fz+36h3iB0hifnChChWRo5qN8lnwNJlTw4yc4VXdg8gASsKVQHlQVfFpa2/qYJRi nCn5BG86QXAt1vWq67F8UtrCMQtFMcERdrKC3C5Jwl96kR1/kfnvPAxFNEwPFsq076X/ UYH4x/jIgdGm1aRsIF+o8kIVT86XNO+4Y4KrgYpUop0psKB8BeyG+GndfFMci92Y6w55 m3+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975741; x=1792580541; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2mkSqDyENes6nJoSVFzqdx//VmH5Dkg6GQfLEw4/f/g=; b=j+foQeIDqNJ6yIJgliCBqgMUaXcoPBJgA+0c5/Hvl51xOu/wnUAW1cXXxu8VBVFYoX 0Sn6DZe0aQdGqvBxWOYgXKheqIdMBtSZ+3R/6z6VHB9f6NyAx07LD/3QgShQxAY4rBbe gudpuJIEEYtUd5767/iqnx4t5Zw8ny1cWrbUV4/RG1dzZCyyCBdgqh95B1HllIIIN4Ep fQBeb6ceucs3G+x2zPUZGQZda1qGk4yFRne3h5PVc5YgUXIHTWy4TFeVxJ72vU88IFZj V/fAeMKtc6Hn60apjWommAJTIvVBaeFd0cHOoUT2OZ1sDO+llqrx83pDXYZZ4GJ5N9cF yPAw== X-Forwarded-Encrypted: i=1; AKwUvBz9ckc41tw5MB/jtpkI4u8zwEVK40kqzwZNVzO/9UcqHqp3/vIRMZ+/XVe7KNJ6B+YC2BN2GAhrY/xuXxU=@vger.kernel.org X-Gm-Message-State: AFq9FYI9WQpvsSDrDnsRfU3ZwKUqCZ8mHV/U+2wYjrOM3tyPvLIyQYGz kev9rr8uhTo8KU7TDbf3C9oLsNio1TK5J/CuT87deR/l4r5gpCeYz3Du X-Gm-Gg: AYBFou0iJliTjMjGUecVN0xvS2Sr8HjyqLudTEWEcD3MXFbMX5xeA6Hu865k+Q/K+D3 9kg6YGNKpN2eJnQAd9aJ7yku5MExYmpDkLM4msPMKC7yKBxx74lF4EjflZod2r2S1Fp0LreK4UH INQcwyR12QTFtjIF3Orx426YK1POdFwGI7nNCk0Ofr1v0239/v9cwGZ+qaY3lSQdhn5dbVewQHZ 1cVeGS/lyoNqb8Bi/OIGqMJllYIRhukYcl83iEMVjS/bzlmqqYOx7WxlBnRGlmMzuwmix7mw/Yu ikNOYUN07LzeceFi/qwAWTl+hCFqm1OaNKUKgchW8tQsUVnX4IExEfK7aF73EMuPCYcGkOk2ujF BqUiTNx5VenKvSfxDUEmuegS9atm8HOrvxJl+w8+784TcBw08IODXBq4HTPjYgTVEEZW/ZRTBaV WvRYOCAyn31E4IuTQRU9ZTMlmjwIqD2WP84b+vxkxCB+wVxt1lBmwhCuvPw6PX3ptPJCCCvHLDn //NbS3dzH/mSp7r10oLlD131wZRJXNlartv/29wuqt0L5sjiidjfHfPF5VVec/IUhSN8shL5iTa X-Received: by 2002:a05:600c:46ca:b0:4a1:7a1e:6910 with SMTP id 5b1f17b1804b1-4a18e48363amr311310525e9.4.1791975741146; Wed, 14 Oct 2026 04:02:21 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:19 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 01/13] dma-buf: introduce initial file I/O infrastructure Date: Wed, 14 Oct 2026 12:01:59 +0100 Message-ID: <49e24334953262e85d5616c9afde88c680e4f3a3.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The goal is to be able to natively use dma-buf in the read-write / IO path. This patch adds basic building blocks serving as a glue and API between drivers and upper layer subsystems providing the uAPI. Later patches implement it for NVMe raw block devices and expose it to the user space via io_uring. There are two main objects. struct dma_buf_io_ctx and struct dma_buf_io_map. The ctx is used during initial registration and serves as an interface between the upper layer user like io_uring and to the importer subsystem / driver. The map represents the actual dma map established for the target device[s] with dma_buf_map_attachment() and stored in a device specific format. The context is created via a new file operation ->init_dma_buf_io_ctx. The ctx-map separation exists to facilitate map invalidation (see dma_buf_io_invalidate_mappings()). A ctx can create multiple maps during its lifetime, but there can only be no more than one (active) map attached to it. Invalidation drops the active map if present, and any new I/O request will try to create a new one. The primary task of dma_buf_io_map is to count requests using it and wait for their completion when we want to destroy the DMA map. [un]mapping and any work with dma addresses is delegated to the importer driver via an ops table stored in the ctx, see struct dma_buf_io_ops. Only the target driver / subsystem knows about devices it wants to use the dma-buf with, especially in case of multi-device filesystems or stacking in the future. Acked-by: Matthew Brost Tested-by: Matthew Brost Acked-by: Christian K=C3=B6nig Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-buf-io.c | 229 +++++++++++++++++++++++++++++++++++ include/linux/dma-buf-io.h | 125 +++++++++++++++++++ include/linux/fs.h | 2 + 4 files changed, 357 insertions(+), 1 deletion(-) create mode 100644 drivers/dma-buf/dma-buf-io.c create mode 100644 include/linux/dma-buf-io.h diff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile index b25d7550bacf..523731b0f83e 100644 --- a/drivers/dma-buf/Makefile +++ b/drivers/dma-buf/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y :=3D dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o + dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o obj-$(CONFIG_DMABUF_HEAPS) +=3D dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) +=3D heaps/ obj-$(CONFIG_SYNC_FILE) +=3D sync_file.o diff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c new file mode 100644 index 000000000000..21296bd3dde1 --- /dev/null +++ b/drivers/dma-buf/dma-buf-io.c @@ -0,0 +1,229 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Common infrastructure for supporing dma-buf in the I/O path. + * + * Copyright (C) 2026 Pavel Begunkov + */ +#include +#include + +static void dma_buf_io_map_release(struct dma_buf_io_map *map) +{ + struct dma_buf_io_ctx *ctx =3D map->ctx; + + dma_resv_assert_held(ctx->dmabuf->resv); + + percpu_ref_exit(&map->refs); + ctx->dev_ops->unmap(ctx, map); +} + +static void dma_buf_io_map_refs_cb(struct percpu_ref *ref) +{ + struct dma_buf_io_map *map =3D container_of(ref, struct dma_buf_io_map, r= efs); + + complete(&map->drained); +} + +static void dma_buf_io_kill_map(struct dma_buf_io_ctx *ctx) +{ + struct dma_buf_io_map *map; + + dma_resv_assert_held(ctx->dmabuf->resv); + + map =3D rcu_dereference_protected(ctx->map, + dma_resv_held(ctx->dmabuf->resv)); + if (!map) + return; + + rcu_assign_pointer(ctx->map, NULL); + percpu_ref_kill(&map->refs); + /* make sure the map is not visible via ctx->map */ + synchronize_rcu_expedited(); + wait_for_completion(&map->drained); + dma_buf_io_map_release(map); +} + +int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map = *map, + struct sg_table *sgt) +{ + unsigned seg_shift =3D ~0U; + struct scatterlist *sg; + unsigned long tmp; + int ret; + + for_each_sgtable_dma_sg(sgt, sg, tmp) + seg_shift =3D min(seg_shift, __ffs(sg_dma_len(sg))); + + ret =3D percpu_ref_init(&map->refs, dma_buf_io_map_refs_cb, 0, GFP_KERNEL= ); + if (ret) + return ret; + init_completion(&map->drained); + map->min_seg_shift =3D seg_shift; + map->ctx =3D ctx; + return 0; +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, "DMA_BUF"); + +static struct dma_buf_io_map *__dma_buf_io_create_map(struct dma_buf_io_ct= x *ctx) +{ + struct dma_buf *dmabuf =3D ctx->dmabuf; + struct dma_buf_io_map *map; + long ret; + + dma_resv_assert_held(dmabuf->resv); + + if (ctx->is_dead) + return ERR_PTR(-ENOENT); + + map =3D __dma_buf_io_get_map(ctx); + if (map) + return map; + + map =3D ctx->dev_ops->map(ctx); + if (IS_ERR(map)) + return map; + + if (WARN_ON_ONCE(!map->min_seg_shift)) + return ERR_PTR(-EFAULT); + + ret =3D dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL, + true, MAX_SCHEDULE_TIMEOUT); + if (ret <=3D 0) { + if (!ret) + ret =3D -EAGAIN; + dma_buf_io_map_release(map); + return ERR_PTR(ret); + } + + /* get a reference for the caller */ + percpu_ref_get(&map->refs); + rcu_assign_pointer(ctx->map, map); + return map; +} + +/** + * dma_buf_io_create_map - Create a new or reference an existing map. + * @ctx: [in] dma-buf I/O context + * + * A slow path part of dma_buf_io_get_map() and shouldn't be used directly. + * Behaves in the same way but allowed to block. + * + * Returns a map on success, or ERR_PTR encoded errno otherwise. + */ +struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx) +{ + struct dma_buf_io_map *map; + int ret; + + ret =3D dma_resv_lock_interruptible(ctx->dmabuf->resv, NULL); + if (ret) + return ERR_PTR(ret); + + map =3D __dma_buf_io_create_map(ctx); + dma_resv_unlock(ctx->dmabuf->resv); + return map; +} + +static void dma_buf_io_ctx_shutdown(struct dma_buf_io_ctx *ctx) +{ + dma_resv_lock(ctx->dmabuf->resv, NULL); + ctx->is_dead =3D true; + dma_buf_io_kill_map(ctx); + dma_resv_unlock(ctx->dmabuf->resv); +} + +/* + * dma_buf_io_ctx_create - create a new dma-buf I/O context for the given = file. + * @file: [in] file for which the context is created + * @dmabuf: [in] buffer to create a context from + * @dir: [in] direction of DMA transfers + * @out_ctx [out] resulting ctx + * + * Create a new I/O context from the given DMA buffer for the given file. + * It only works with files implementing ->init_dma_buf_io_ctx and fails + * with -EOPNOTSUPP otherwise. Once created, the user can create a map + * and use it to issue I/O like read / write requests to @file. + */ +int dma_buf_io_ctx_create(struct file *file, + struct dma_buf *dmabuf, + enum dma_data_direction dir, + struct dma_buf_io_ctx **out_ctx) +{ + struct dma_buf_io_ctx *ctx; + int ret; + + if (!file->f_op->init_dma_buf_io_ctx) + return -EOPNOTSUPP; + + ctx =3D kzalloc_obj(*ctx); + if (!ctx) + return -ENOMEM; + ctx->dir =3D dir; + ctx->dmabuf =3D dmabuf; + get_dma_buf(dmabuf); + + ret =3D file->f_op->init_dma_buf_io_ctx(file, ctx); + if (ret) { + kfree(ctx); + dma_buf_put(dmabuf); + return ret; + } + + if (WARN_ON_ONCE(!ctx->dev_ops || + !ctx->dev_ops->map || + !ctx->dev_ops->unmap || + !ctx->dev_ops->release)) + return -EINVAL; + + *out_ctx =3D ctx; + return 0; +} + +/** + * dma_buf_io_ctx_release - destroy the I/O context + * @ctx: [in] dma-buf I/O context + * + * Destroy the context. It makes sure all maps are waited on and destroyed, + * and can block for I/O to complete. It detaches from the importer and + * destroys all its resources via the ->release callback, and frees the + * context. + */ +void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx) +{ + dma_buf_io_ctx_shutdown(ctx); + + if (WARN_ON_ONCE(rcu_dereference_protected(ctx->map, true))) + return; + + ctx->dev_ops->release(ctx); + dma_buf_put(ctx->dmabuf); + kfree(ctx); +} + +/** + * dma_buf_io_invalidate_mappings - invalidate the current map + * @ctx: [in] dma-buf I/O context + * + * Importer's helper for invalidating the current map. Waits for all I/O + * using the map to complete and destroyes the map. Called with the + * dma-buf reservation lock held. + */ +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx) +{ + dma_buf_io_kill_map(ctx); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, "DMA_BUF"); + +/** + * dma_buf_io_detach - detach the @ctx from the importer. + * @ctx: [in] dma-buf I/O context + * + * Importer's helper for removing the I/O ctx. It kills all maps and + * prevents creation of new ones. Upon return it's safe for the importer + * to release all its resources such as dma-buf attachments. + */ +void dma_buf_io_detach(struct dma_buf_io_ctx *ctx) +{ + dma_buf_io_ctx_shutdown(ctx); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_detach, "DMA_BUF"); diff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h new file mode 100644 index 000000000000..b9bbeca1744f --- /dev/null +++ b/include/linux/dma-buf-io.h @@ -0,0 +1,125 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __DMA_BUF_IO_H__ +#define __DMA_BUF_IO_H__ + +#include + +struct dma_buf_io_ctx; +struct dma_buf_io_map; + +struct dma_buf_io_ops { + /* + * Create a new map for the given ctx. Called with the reservation + * lock held. + */ + struct dma_buf_io_map *(*map)(struct dma_buf_io_ctx *ctx); + + /* + * Clean up device specific parts of the @map. Called with the + * reservation lock held. + */ + void (*unmap)(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map); + + /* + * The user tries to destroy the ctx. Release all device specific + * parts of the token. + */ + void (*release)(struct dma_buf_io_ctx *); +}; + +struct dma_buf_io_map { + /* + * Counts attached requests and other users. Device specific unmapping + * is deferred until all refs are dropped. + */ + struct percpu_ref refs; + /* + * Shift for the minimum segment size of the mapping. + */ + unsigned min_seg_shift; + + struct dma_buf_io_ctx *ctx; + struct completion drained; +}; + +struct dma_buf_io_ctx { + struct dma_buf_io_map __rcu *map; + struct dma_buf *dmabuf; + enum dma_data_direction dir; + bool is_dead; + + void *dev_priv; + const struct dma_buf_io_ops *dev_ops; +}; + +int dma_buf_io_ctx_create(struct file *file, + struct dma_buf *dmabuf, + enum dma_data_direction dir, + struct dma_buf_io_ctx **ctx); +void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx); + +struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx); + +static inline struct dma_buf_io_map * +__dma_buf_io_get_map(struct dma_buf_io_ctx *ctx) +{ + struct dma_buf_io_map *map; + + guard(rcu)(); + + map =3D rcu_dereference(ctx->map); + if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs))) + return NULL; + + return map; +} + +/** + * dma_buf_io_get_map - Create a new or reference an existing map. + * @ctx: [in] dma-buf I/O context + * @nowait: [in] specifies whether it's allowed to block + * + * Create a new or acquire an existing map. When succeeds, it returns + * a referenced map, which should eventually be put down with + * dma_buf_io_map_drop(). If @nowait is set and the function would block, + * it return -EAGAIN, and the caller is expected to retry from a context + * where blocking is allowed. + * + * Returns a map on success, or ERR_PTR encoded errno otherwise. + */ +static inline struct dma_buf_io_map * +dma_buf_io_get_map(struct dma_buf_io_ctx *ctx, bool nowait) +{ + struct dma_buf_io_map *map; + + map =3D __dma_buf_io_get_map(ctx); + if (likely(map)) + return map; + + if (nowait) + return ERR_PTR(-EAGAIN); + return dma_buf_io_create_map(ctx); +} + +/** + * dma_buf_io_map_drop - Drop the map. + * @map: [in] dma-buf I/O map + * + * Put down a reference for the @map acquired via dma_buf_io_get_map(). + * Should be called as soon as I/O is completed. + */ +static inline void dma_buf_io_map_drop(struct dma_buf_io_map *map) +{ + percpu_ref_put(&map->refs); +} + +/* + * Device API + */ + +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx); +int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map = *map, + struct sg_table *sgt); +void dma_buf_io_detach(struct dma_buf_io_ctx *ctx); + +#endif diff --git a/include/linux/fs.h b/include/linux/fs.h index f9d1e05e8ae6..05c1ff495732 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1914,6 +1914,7 @@ struct dir_context { #define COPY_FILE_SPLICE (1 << 0) =20 struct io_uring_cmd; +struct dma_buf_io_ctx; struct offset_ctx; =20 struct file_operations { @@ -1960,6 +1961,7 @@ struct file_operations { int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *, unsigned int poll_flags); int (*mmap_prepare)(struct vm_area_desc *); + int (*init_dma_buf_io_ctx)(struct file *, struct dma_buf_io_ctx *); } __randomize_layout; =20 /* Supports async buffered reads */ --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 116004F68C8 for ; Wed, 14 Oct 2026 11:02:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; cv=none; b=niPjjurOYt7Z7dmNCKEkKzlLYMSC8qMORQaOc1GvXZV26g2PQ6wA0J34FnNIC9WsHkWMzAjU6nDBUrMWCmXZB7WQ64wvHO/Pa5EwVAA9jW0wmxQFUflB8piQpm4fGkmqH54iEPxzvny8Q/8HF9qPc5g6pK2DFwOSofOIM6+76/w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; c=relaxed/simple; bh=r0p2adqwMj/5lT7C+TYVmZHcVYZ+gdU2HSJ4o7+sSAQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PaNxuZ9OeBNI26Gm2WRhtl0jJIeJYc4KkCfWCRgPs/ruvtVQzaUeRaZG9Psvqo7EkB04jO7jOSlTTqu5eWgRezTEPJ10Kz+0BbGYUgGCB9QhDhz1n2NIycVCS7xShz+qzPZy55Cz9473WlWg047qWPVjqresbUUYA5BOMzDkwJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AlZajlsF; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AlZajlsF" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-4a1af482519so13100325e9.1 for ; Wed, 14 Oct 2026 04:02:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975743; x=1792580543; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=u2SC7dcI1Y+PPMkABrALMaBjWk253olXayI/6F0cq90=; b=AlZajlsFuXNaw+WSC4WWNqvtnnlJsFoEwUA49HXTnNp0KUIgaki8UMeIQFfXiew5RR nRsVHSGB0v2wsK81ajZ+w0926oaAQXpl5IG7QFo+hAvpcGDs8MUTHaAYlh9O/0geKjs7 LVgdmzayccV8J4e/jsDDXlN1d1p6obd8D2rlsCgLcfC7lXB+RFn+pHGzFV8hKRwP7ZTN wIwJ9ybi7RrKdY+g61g8xHUOwRN/bVPedQoA/V8vyIQtO/pTGzVZIZYk0BSseto93l6/ IDg6fC43p0Uux/LxB/Ar46+trrtxI6JOyWvCSi7jTJtqrDbItzaH8VlXUO0KA8iFM6xg EX7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975743; x=1792580543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=u2SC7dcI1Y+PPMkABrALMaBjWk253olXayI/6F0cq90=; b=pEWzBV1aarakTA90pNa6oeeFM/oL41vFknx+3E/7Sza1sSiwmP4zkmPV55rTz3T4nt jZQtr5sA73mJkVLj3wGV6iqvoUWjfhUDcZN64LeizUJnTLnU43uwNrxC/azB1ypCZl55 3x0ynQzrDdhdSKqyUjLKfqScq75Yaqw7p9JrgkKGSxTyxFmrZFCfLcUJAURXKtu0uXLL zv/39+Q+om1E3kixm6yG+nzun/qZdPuV69NvhXOBPI1VLKGjIbzLzUXKLDM1bNkKQplz kHJjmc0Oz//+UV/I2jSl43u2vRLK127X/jKfKCJ1bHWTT/D1DQS35bIIV2SW/SMxp8Fw IaPw== X-Forwarded-Encrypted: i=1; AKwUvByF0S6ZJIa4T0LNBRFs/IxbPU+GcDmrQMF46DA87n/MPSuDlehAAqfnEBbg8LeVYvALRzhzRiu5Y/WYlUw=@vger.kernel.org X-Gm-Message-State: AFq9FYKfp9z2JHqex4UchyKA90O++Qu1a1cZIoFh5pvZLqoRn8F/7/nH 5au+9CDLs0dG++0CH5c3SPdFk7e4xj5hhEtoukKAXTayV0o6WhMt9g+9 X-Gm-Gg: AYBFou06weqbNzK2Y/12vTgKVSYKFGG3psYqcUdylWjGwzU6YLEP2t/c4C/XOn8PQnZ tw561DgmpTcYNhcPg5EhYn2mBzOu3/fA9CXfVoCO6WUVBd4z6CVKZDDQvcTEYgs8TpWmBPK5USk kB9J35sYHTM8WKnHS/rqLsaKrSESm2elJu9dzvZ/V/eA9jTKpwqepo8rGbFB/8GbHEN4aMn4rU5 4XHVUwx66Gz15T3GSmaXhvyrapIIUkX3pjlldoa99pXuwYX/Mb4mL13Tr32ZeAtn9T0CHGUQ66g DahZVbfTvxJduUjQSg6OtkZQTHRuDISo3nwzjS9lDyeboyBwA1rWrCkIDY3vlh20FraAxLrp1DM zdP5hl8Eao+YPBAZbe2hMm/HbBR/65QZ7EWhDtdMm/nRp1IWwOQ6XxX//ZuESBCA8N2ZSxyzKWd sV2djzyZ3/xUCoGZksIBDKtOlpGT1fP3aey67HnSMl21iKNzjCYzrgTYn2o5iv7Si7nPiU9AGOm Cb7y5RSqSnMglDPSFJbn/9k57t+pz79/1aw5KcaaIxmbpVkvDfX5rduiN2kSZbf7tCn+guiUHlu X-Received: by 2002:a05:600c:3146:b0:4a0:34a:588c with SMTP id 5b1f17b1804b1-4a18e4a7cb9mr304322515e9.14.1791975742685; Wed, 14 Oct 2026 04:02:22 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:21 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 02/13] iov_iter: add iterator type for dmabuf maps Date: Wed, 14 Oct 2026 12:02:00 +0100 Message-ID: <6aedb1c606f20ae5bd7fa358829966384c6e738f.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a new iterator type for dmabuf maps. The map in an opaque object with internals and format specific to the subsystem / driver, and only it can use that subsystem / driver for issuing IO. The task of the middle layers is to pass the map / iterator further down, maybe doing basic splitting and length checking. The iterator can only be used by operations of the file the associated map was created for. Suggested-by: Keith Busch Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- include/linux/iov_iter.h | 2 ++ include/linux/uio.h | 11 +++++++++++ lib/iov_iter.c | 29 +++++++++++++++++++++++------ 3 files changed, 36 insertions(+), 6 deletions(-) diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h index f9a17fbbd398..2f91884d733b 100644 --- a/include/linux/iov_iter.h +++ b/include/linux/iov_iter.h @@ -310,6 +310,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size= _t len, void *priv, return iterate_folioq(iter, len, priv, priv2, step); if (iov_iter_is_xarray(iter)) return iterate_xarray(iter, len, priv, priv2, step); + if (WARN_ON_ONCE(iov_iter_is_dmabuf_map(iter))) + return 0; return iterate_discard(iter, len, priv, priv2, step); } =20 diff --git a/include/linux/uio.h b/include/linux/uio.h index fe2e985d74d2..638c1116e912 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -12,6 +12,7 @@ =20 struct page; struct folio_queue; +struct dma_buf_io_map; =20 typedef unsigned int __bitwise iov_iter_extraction_t; =20 @@ -29,6 +30,7 @@ enum iter_type { ITER_FOLIOQ, ITER_XARRAY, ITER_DISCARD, + ITER_DMABUF_MAP, }; =20 #define ITER_SOURCE 1 // =3D=3D WRITE @@ -71,6 +73,7 @@ struct iov_iter { const struct folio_queue *folioq; struct xarray *xarray; void __user *ubuf; + struct dma_buf_io_map *dmabuf_map; }; size_t count; }; @@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov= _iter *i) return iov_iter_type(i) =3D=3D ITER_XARRAY; } =20 +static inline bool iov_iter_is_dmabuf_map(const struct iov_iter *i) +{ + return iov_iter_type(i) =3D=3D ITER_DMABUF_MAP; +} + static inline unsigned char iov_iter_rw(const struct iov_iter *i) { return i->data_source ? WRITE : READ; @@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter *i, unsigned = int direction, unsigned int first_slot, unsigned int offset, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xa= rray *xarray, loff_t start, size_t count); +void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction, + struct dma_buf_io_map *map, + loff_t off, size_t count); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 2072c04e99d0..6831a5d9396e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(i->count < size)) size =3D i->count; - if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i))) { + if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i)) || + unlikely(iov_iter_is_dmabuf_map(i))) { i->iov_offset +=3D size; i->count -=3D size; } else if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) { @@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) return; } unroll -=3D i->iov_offset; - if (iov_iter_is_xarray(i) || iter_is_ubuf(i)) { + if (iov_iter_is_xarray(i) || iter_is_ubuf(i) || + iov_iter_is_dmabuf_map(i)) { BUG(); /* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int = direction, } EXPORT_SYMBOL(iov_iter_xarray); =20 +void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction, + struct dma_buf_io_map *map, + loff_t off, size_t count) +{ + WARN_ON(direction & ~(READ | WRITE)); + *i =3D (struct iov_iter){ + .iter_type =3D ITER_DMABUF_MAP, + .data_source =3D direction, + .dmabuf_map =3D map, + .count =3D count, + .iov_offset =3D off, + }; +} + /** * iov_iter_discard - Initialise an I/O iterator that discards data * @i: The iterator to initialise. @@ -856,7 +872,7 @@ unsigned long iov_iter_alignment(const struct iov_iter = *i) return iov_iter_alignment_bvec(i); =20 /* With both xarray and folioq types, we're dealing with whole folios. */ - if (iov_iter_is_folioq(i)) + if (iov_iter_is_folioq(i) || iov_iter_is_dmabuf_map(i)) return i->iov_offset | i->count; if (iov_iter_is_xarray(i)) return (i->xarray_start + i->iov_offset) | i->count; @@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_i= ter *i) size_t size =3D i->count; unsigned k; =20 - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i)) return 0; =20 if (WARN_ON(!iter_is_iovec(i))) @@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf); void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) { if (WARN_ON_ONCE(!iov_iter_is_bvec(i) && !iter_is_iovec(i) && - !iter_is_ubuf(i)) && !iov_iter_is_kvec(i)) + !iter_is_ubuf(i) && !iov_iter_is_kvec(i) && + !iov_iter_is_dmabuf_map(i))) return; i->iov_offset =3D state->iov_offset; i->count =3D state->count; - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i)) return; /* * For the *vec iters, nr_segs + iov is constant - if we increment --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FBF84F0535 for ; Wed, 14 Oct 2026 11:02:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; cv=none; b=VF4s8PGf05IKNbWD9O4Y9yg3AEQ6WEKsz0BDvZWGdDtnsvQkPN+sFjmZzMIOrxi8kl79V41xkI2dvDFESj/PtNd65sgHqnUyR1SOoNX+8W+6Wn7uLfoHG59c2Y77dOmjmdA3ixGyNCF55kNkKrlGIGRnNta33iBUoIHGfc0inho= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975752; c=relaxed/simple; bh=lqtmMOq9NyJHWkCTixF/EfRvdzo9ti4x7BRbdW0dvpo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=buXCWNNLVJpC1B6Thc8QzJYpeR5FmMtVgDxrRowELpYBsSwmbb9hQ18uG+tt4OgmSoZPMNznSPiXk6uvlCPPfnBN1tftluoGHj+KOl63n5SfFDEHUQi/qI4f7KNJaMqS1tAeutDCX+tOVkanS3uyWalf7nHwtQcMDH0Gmb+XYNs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BbcPzze0; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BbcPzze0" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-4a061a13884so34064155e9.3 for ; Wed, 14 Oct 2026 04:02:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975744; x=1792580544; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BooWm7URUcyrH2E2qkRWxlx7KQoz7GWFpr53v56Fl9k=; b=BbcPzze0G7sm+U5rosix6MrInL+KqWVYUz/mHVM3BMQP/SryDr3Py7OMsHqeLXd+3P jEqNiRD2Z0O0iKhJhpjmu+LkcLNRcavskjQ27c1dKeaoXy0cdhkn7cqoJ22a6fcfg1aY InWRYCOFBvzFPkF3C6E9yoDQi3oTwwOyTSYcAw5ZCpcbIYhJWr6xKtKo+xI4CbKc8Fu2 RoCfZ7KOs3Ym91wo+zsy6kem38FW2wYmwShp9T60G2ARGP6+X5Qkj2RaVx+F8FQ0lHqU im0hTpkNBRcsw7wP8BxUAja02M9pv0ztPpDmuuWutnfrwHZJdLoJiMQOXyBfiLA5F3HZ vsJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975744; x=1792580544; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BooWm7URUcyrH2E2qkRWxlx7KQoz7GWFpr53v56Fl9k=; b=f2WESFSDc+mSMTirQw6nvazFU5/DDDBLyT6cfQeRP52cPwe14sE4kn0aamQtcOeJ1y EAPR+kblicGSSoOEIozaVou/d+BonZV/bvHxrqruaDmGHWuItOEXRRtAzfpDWSZJQlBW djvEu+WdrN7Q1neVBZNVYkqyPd3HLHFx0CEW3GR3W7fzVMb6CpvfiQd662kEqeKsa0Qz jSbHr68GIlyFlFdZ1r5m8O0B95VDi8Jx7XJMRB3ZW1doB/++Chtg7QjI0KhNnrWMYzGD IFdesD4GRWLaQhnI1BnxEz64bLXGFY+CVAfoqGHMTwNOdU9VRcPUNaHdcqe4Umlwa875 bFAQ== X-Forwarded-Encrypted: i=1; AKwUvBxujZfNuvzH0+5PdPwMcgxbAFfA3athGYA+dyJRDDJRtb+hrluMI/HSVpFxpf7bEfh9A2xGJagsIiCwc9g=@vger.kernel.org X-Gm-Message-State: AFq9FYLeGFd3WO2/HkSRH65oiDmgrk64lUDiNvxWrbjv/LL4CAjfVwKy Ezu0149nqMQhDP1wZnZBcP7tLffcgydOK4CmibCkQ0Qp1JGvSxa8LdQL X-Gm-Gg: AYBFou2ThkYUXQ11gGo7+NQByJ4dpKQbEfPmfcVtNOEJjWdPxU6tIKt8KMk0zM5wP3J YHU+kpW18u9sraY1O38o8xhBUOZyPjhkZW0ZuLyAW+/R6lKPsZvNeb+RtEH8BGDFnsxJ598z5fB BhVEOmDCza2W/EJc6uhjK9tpL3Pby7PN/XOyC62k8xRuWeH4kPSMlwt5WE64n9FmS+D/1TReDth Z+dyNBQ1L4CYgIpRW+g7waQrmonsIJZl0ujlL60jm7DTKVYwH4gkrPZneOX/814/EY8JrkGoubS 1tafVN9NL+aL6F4oHs4/1Byy5tPw9OMSLW6H0vRABavNtQxXvX4jDPofwRAJtnNXPB3+Ks7QkKK jQdrMdlhMAZhmy6iJQ7B+ggzxy28UNZoS+txJhkxqs6wxG6MNmm85zFLWEhI/YlROUjcHDSwdma KJ+JMF6hXxUjMZ9+chzOlojd2oiVU00szwUHE6xT4ho+YjX/YNXhRyo+6wyYyghtQcdCNxXdrlr LFADa2uiITeTxej6DjUbiyRV1oDI+Lp7VgC6NPYpPpQaA7AQv+ofYbSbbJBNdwAwn/RUmUrSmR3 BA== X-Received: by 2002:a05:600c:a088:b0:4a0:333:253a with SMTP id 5b1f17b1804b1-4a18e49c331mr307406695e9.14.1791975744203; Wed, 14 Oct 2026 04:02:24 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:23 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 03/13] block: always adjust bi_offset on bio_advance_iter Date: Wed, 14 Oct 2026 12:02:01 +0100 Message-ID: <8e5a67c2f84b6df99ff57a7b3bf7b289999e0468.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend bio_no_advance_iter() iteration to also increment the offset. It's not currently needed because the currently listed request types don't have a buffer, but will be useful once we add bios backed by dma-buf, which don't have a bvec but work with a single range. Suggested-by: Christoph Hellwig Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- drivers/md/dm-io-rewind.c | 6 ++++-- include/linux/bio.h | 12 ++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c index 04f3fc8aeb6f..b22719c6411c 100644 --- a/drivers/md/dm-io-rewind.c +++ b/drivers/md/dm-io-rewind.c @@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bi= o *bio, iter->bi_sector -=3D bytes >> 9; =20 /* No advance means no rewind */ - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size +=3D bytes; - else + iter->bi_offset -=3D bytes; + } else { dm_bvec_iter_rewind(bio->bi_io_vec, iter, bytes); + } } =20 /** diff --git a/include/linux/bio.h b/include/linux/bio.h index 17944e44b584..892ca469c570 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -113,11 +113,13 @@ static inline void bio_advance_iter(const struct bio = *bio, { iter->bi_sector +=3D bytes >> 9; =20 - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -=3D bytes; - else + iter->bi_offset +=3D bytes; + } else { bvec_iter_advance(bio->bi_io_vec, iter, bytes); /* TODO: It is reasonable to complete bio with error here. */ + } } =20 /* @bytes should be less or equal to bvec[i->bi_idx].bv_len */ @@ -127,10 +129,12 @@ static inline void bio_advance_iter_single(const stru= ct bio *bio, { iter->bi_sector +=3D bytes >> 9; =20 - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -=3D bytes; - else + iter->bi_offset +=3D bytes; + } else { bvec_iter_advance_single(bio->bi_io_vec, iter, bytes); + } } =20 void __bio_advance(struct bio *, unsigned bytes); --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 889504F7CCB for ; Wed, 14 Oct 2026 11:02:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975758; cv=none; b=iDkMOHwGfiCMMMFEXarGAZs2ssCWNgj9ncsAAh0E2JtXON4uKMbtKnAi1OVsCPdIKYWHku0GL2sdDHt+xNx2j3z2gORCg7d1/Casn3ElgZR49GWAzh9LqE4o1e6jtZ0dIUvcBp59J1DAwVYko26xSJLpLuir8xNk3Ur1ZW6H4TM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975758; c=relaxed/simple; bh=WSYuJ2CSe5MQPAXNnecfuqbBxEyZXfqX7o9X0AvpCs0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Dus3vAk5ENh+eYM9rniYCnfb3f+OmWzji8l3j8y9yajZNGGsrDY+mqhBIOGCUit+1C3ilMjbp3itpHznqFLlFdXFa/WU7a/VYH3SY+8oYQfbZ2n3hCRiFAH2DyMdegigCK92tGqv+5HEMpNmWMVfF/jqqX0mwUWTq4z7AkU95sA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Fh5gUEV9; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Fh5gUEV9" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4956869750eso27895395e9.2 for ; Wed, 14 Oct 2026 04:02:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975746; x=1792580546; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=VU42MfuIm5UeQYpyXXcu/MIJJ21cHTHg0N7CAKjDZ/U=; b=Fh5gUEV9/pTrif2y3igFxrzWK2GROFsN8gAuGMscScSb3sGlOBfDqYtGgmT18ii9Lp J+ZELvQEwtA1VEfm1eQasee4QZ4ZdMv6T2AW4JOsnTR9yY7TxD0UCik1gtZaVm0b6DEM IupI3fbmTbGSlhNZlDtBzciC30KBrnJ80lSNPjCIAbkEvudFox+bpqAbeE/eal01zWTS ervzo99r5/HL6pQWHCQx4/Z40AnL8hN963ZyBsY7e2Kd4A7RHx2XztmlTWhaUR9QO3f3 e8CuhhCqPEwlm5YwODvdJDF3lhc6ntt+/r6B2Zb3DpCQP6rlSQaPera9tXkWwMoo+b7I +Mzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975746; x=1792580546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=VU42MfuIm5UeQYpyXXcu/MIJJ21cHTHg0N7CAKjDZ/U=; b=GmakZSy3qtNvAdr3e2FjYtNDuz5EySgeaT+fWXDgE7lBW6HWi3Jb6tSgLo+4ccYwPJ kfPZJ/FmnJTRaWGl0+7GHyIln9LB/KABUqHW+ub6iqzslUwXwOpztVll4Xu4/CUxDHdx tvichlzRhlAnT1yWDVgzUqxy8OoFXvk6TpnaWUwX8vmQmmfVZQJQABztlM1he9Gc68v9 +e3DQFxg+ZLTr9TewrQQapJ+SL5GyVfNgrxp8NKQOAUiNLXM3KTq17YJY9Hvkr2yy282 KbbaFGg9tGbHj8Hkc3izAKI9fk50Uk95O54DKGZEQ8DpDUioXBGNMlSy9TLA27mNDhw+ PY9w== X-Forwarded-Encrypted: i=1; AKwUvBxMcORQSvqINKN6t3n8s1U3eY98Un44zrTo4h8CbQE58EMRs+WXbx898GK+yiIUuKylRlpfgSNBNsTCbtI=@vger.kernel.org X-Gm-Message-State: AFq9FYIg5hyQZCRrLlAL6UvhzvfhhTlX3HWM3A0adEoXSaa0t3FtURLc 3qr0ZEeRV0sa6iHJ/ag1Vpu3YNJkoI3uSJIxBh5rUqHXVKRKObwx3PIK X-Gm-Gg: AYBFou0Vv3dpJh2jEtWD/DoU9nger7L/dhmRDK8FGUJd0p5Bo2CEhqpLZrp73j0uZ83 6MR0XeUX6svsjzkOnDIePjJhE2z9eK/KOooeGfYT9PytfMWYqOE4ZprrsSRHbX8wbFJAxe8W4HN nrGqtmOH2AFnFSsEVx+RIdVOxyphl+myC3VgLaSUsmHGnWCfrxrUl0TXUUanrAie6OwjJyADbpK hESH4HhGq4goNKEkH+5BKYBIzHHCakENRXg8APLboYTue3EpJrnzVZuBscLZcWMXPii3bxFuEFJ TF2UfhbmLhFdhlzsh6cK4NgpkQK0HgI0yqkwwqbLj0gztj3aD00wvgvOerFl0e9txW0zCRxBh4+ 9DQROhTuuobpI3CR9HxvbnXRy8+IKxHepN4IrFza/CJm2ei32LkUfFP06ZfkzMnOSMPUTfyBzX5 gPM02hXCuEYFkCJJou5C52gMPkkOYuR7sRblc0DQZe4roczOhdFbSbSh5ap/9CVexz6GF/Ms1EB rA6fQ/1MhIERtOJnNe9Stqf9UfEjfVJUEz1ndIVvJ7+U3KVA1fBp7P/ok6bVre3m1JhbSwBT9kF X-Received: by 2002:a05:600c:8184:b0:4a0:1c17:4fb2 with SMTP id 5b1f17b1804b1-4a18e4d854cmr271523965e9.31.1791975745861; Wed, 14 Oct 2026 04:02:25 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:25 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 04/13] block: introduce dma map backed bio type Date: Wed, 14 Oct 2026 12:02:02 +0100 Message-ID: <39b964cf7510f42755188563da63d3d5acaf2aa9.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Premapped buffers don't require a generic bio_vec since these have already been dma mapped. Repurpose the bi_io_vec space to store dmabuf maps as they are mutually exclusive. The bio splitting differs from the normal path because it's already pre-mapped and for the block layer it's just an offset into the dma-buf. The actual segmentation is only available to the importer driver and not the block layer, however, it doesn't contain alignment gaps and we don't have to check it. For the same reason we can't precisely split by the number of segments, but we use ->min_seg_shift stored in the map to calculate the minimum number of bytes a request consisting of lim->max_segments full segments can cover and split by that. It's stricter and can add extra splitting. E.g. for a {4K, 4G} segmentation, the min segment size is 4K, and we'll split it into bios of (4K * lim->max_segments) bytes each, but it should be good enough for now to cover the most popular use cases. Suggested-by: Keith Busch Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Reviewed-by: Christoph Hellwig Tested-by: Anuj Gupta --- block/bio.c | 15 +++++++++-- block/blk-merge.c | 52 +++++++++++++++++++++++++++++++++++++++ block/blk-mq.c | 6 +++++ block/fops.c | 2 +- include/linux/bio.h | 9 ++++--- include/linux/blk-mq.h | 7 ++++++ include/linux/blk_types.h | 14 ++++++++++- include/linux/bvec.h | 3 ++- 8 files changed, 99 insertions(+), 9 deletions(-) diff --git a/block/bio.c b/block/bio.c index b48091c7663f..1e0d9714c541 100644 --- a/block/bio.c +++ b/block/bio.c @@ -881,7 +881,11 @@ static int __bio_clone(struct bio *bio, struct bio *bi= o_src, gfp_t gfp) bio->bi_write_stream =3D bio_src->bi_write_stream; bio->bi_bvec_gap_bit =3D bio_src->bi_bvec_gap_bit; bio->bi_iter =3D bio_src->bi_iter; - bio->bi_io_vec =3D bio_src->bi_io_vec; + + if (op_is_dmabuf(bio->bi_opf)) + bio->bi_dmabuf_map =3D bio_src->bi_dmabuf_map; + else + bio->bi_io_vec =3D bio_src->bi_io_vec; =20 if (bio->bi_bdev) { if (bio->bi_bdev =3D=3D bio_src->bi_bdev && @@ -1204,16 +1208,23 @@ EXPORT_SYMBOL_GPL(__bio_release_pages); =20 bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter) { - if (!iov_iter_is_bvec(iter)) + if (!iov_iter_is_bvec(iter) && !iov_iter_is_dmabuf_map(iter)) return false; =20 WARN_ON_ONCE(bio->bi_max_vecs); =20 + static_assert(offsetof(struct bio, bi_io_vec) =3D=3D + offsetof(struct bio, bi_dmabuf_map)); + static_assert(offsetof(struct iov_iter, bvec) =3D=3D + offsetof(struct iov_iter, dmabuf_map)); + bio->bi_io_vec =3D (struct bio_vec *)iter->bvec; bio->bi_iter.bi_idx =3D 0; bio->bi_iter.bi_offset =3D iter->iov_offset; bio->bi_iter.bi_size =3D iov_iter_count(iter); bio_set_flag(bio, BIO_CLONED); + if (iov_iter_is_dmabuf_map(iter)) + bio->bi_opf |=3D REQ_NOMERGE | REQ_DMABUF; return true; } =20 diff --git a/block/blk-merge.c b/block/blk-merge.c index 258a726071d1..501ddc01c449 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -9,6 +9,7 @@ #include #include #include +#include =20 #include =20 @@ -319,6 +320,41 @@ static inline unsigned int bvec_seg_gap(struct bio_vec= *bvprv, return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len); } =20 +static inline int bio_split_io_at_dmabuf(struct bio *bio, + const struct queue_limits *lim, unsigned *segs, + unsigned max_bytes, unsigned len_align_mask, + unsigned start_align_mask) +{ + unsigned bytes =3D min(bio->bi_iter.bi_size, max_bytes); + unsigned seg_shift =3D bio->bi_dmabuf_map->min_seg_shift; + unsigned offset =3D bio->bi_iter.bi_offset & ((1U << seg_shift) - 1); + u64 max_segs_bytes; + + /* + * dma-buf maps don't expose the underlying segmentation, but they're + * guaranteed to not have alignment gaps, we only need to check the + * start and length alignment. + */ + if ((bio->bi_iter.bi_offset & start_align_mask) || + (bio->bi_iter.bi_size & len_align_mask)) + return -EINVAL; + + /* Presented as a single contiguous range into the dma-buf */ + *segs =3D 1; + + /* + * Limit by the number of segments by using the minimal segment size. + * Any I/O consisting of N full segments should be able to cover at + * least N multiplied by the segment size. It's stricter than walking + * the segments and might cause extra splitting. + */ + max_segs_bytes =3D (u64)lim->max_segments << seg_shift; + bytes =3D min_t(u64, bytes, max_segs_bytes - offset); + if (bytes !=3D bio->bi_iter.bi_size) + return bytes; + return 0; +} + /** * bio_split_io_at - check if and where to split a bio * @bio: [in] bio to be split @@ -346,6 +382,19 @@ int bio_split_io_at(struct bio *bio, const struct queu= e_limits *lim, len_align_mask |=3D (bc->bc_key->crypto_cfg.data_unit_size - 1); } =20 + if (op_is_dmabuf(bio->bi_opf)) { + int ret; + + ret =3D bio_split_io_at_dmabuf(bio, lim, &nsegs, max_bytes, + len_align_mask, start_align_mask); + if (ret < 0) + return ret; + if (!ret) + goto out; + bytes =3D ret; + goto split; + } + bio_for_each_bvec(bv, bio, iter) { if (bv.bv_offset & start_align_mask || bv.bv_len & len_align_mask) @@ -376,6 +425,7 @@ int bio_split_io_at(struct bio *bio, const struct queue= _limits *lim, bvprvp =3D &bvprv; } =20 +out: *segs =3D nsegs; bio->bi_bvec_gap_bit =3D ffs(gaps); return 0; @@ -493,6 +543,8 @@ unsigned int blk_recalc_rq_segments(struct request *rq) =20 if (!rq->bio) return 0; + if (op_is_dmabuf(rq->bio->bi_opf)) + return 1; =20 switch (bio_op(rq->bio)) { case REQ_OP_DISCARD: diff --git a/block/blk-mq.c b/block/blk-mq.c index e929f94b25c7..1b7f8cbbda8b 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -3150,6 +3150,12 @@ void blk_mq_submit_bio(struct bio *bio) goto queue_exit; } =20 + if (unlikely(op_is_dmabuf(bio->bi_opf) && + bdev_has_integrity_csum(bio->bi_bdev))) { + bio_endio_status(bio, BLK_STS_NOTSUPP); + goto queue_exit; + } + if ((bio->bi_opf & REQ_POLLED) && !blk_mq_can_poll(q)) { bio_endio_status(bio, BLK_STS_NOTSUPP); goto queue_exit; diff --git a/block/fops.c b/block/fops.c index 9905ed24a157..827dc9eecba7 100644 --- a/block/fops.c +++ b/block/fops.c @@ -362,7 +362,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *i= ocb, * Users don't rely on the iterator being in any particular * state for async I/O returning -EIOCBQUEUED, hence we can * avoid expensive iov_iter_advance(). Bypass - * bio_iov_iter_get_pages() and set the bvec directly. + * bio_iov_iter_get_pages() and set the bvec/dmabuf directly. */ if (!bio_iov_iter_set(bio, iter)) { ret =3D blkdev_iov_iter_get_pages(bio, iter, bdev); diff --git a/include/linux/bio.h b/include/linux/bio.h index 892ca469c570..7a794ce723b8 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio *= bio) { return bio_op(bio) =3D=3D REQ_OP_DISCARD || bio_op(bio) =3D=3D REQ_OP_SECURE_ERASE || - bio_op(bio) =3D=3D REQ_OP_WRITE_ZEROES; + bio_op(bio) =3D=3D REQ_OP_WRITE_ZEROES || + op_is_dmabuf(bio->bi_opf); } =20 static inline void *bio_data(struct bio *bio) @@ -438,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio *b= io) =20 /* * Calculate number of bvec segments that should be allocated to fit data - * pointed by @iter. If @iter is backed by bvec it's going to be reused - * instead of allocating a new one. + * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec ar= ray / + * the dma map are going to be reused, and so no extra allocation is requi= red. */ static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_seg= s) { - if (iov_iter_is_bvec(iter)) + if (iov_iter_is_bvec(iter) || iov_iter_is_dmabuf_map(iter)) return 0; return iov_iter_npages(iter, max_segs); } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index af878597afb8..7c7504c84e09 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -1017,6 +1017,13 @@ static inline void *blk_mq_rq_to_pdu(struct request = *rq) return rq + 1; } =20 +static inline bool blk_mq_rq_is_dmabuf(struct request *rq) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return false; + return rq->bio && op_is_dmabuf(rq->bio->bi_opf); +} + static inline struct blk_mq_hw_ctx *queue_hctx(struct request_queue *q, in= t id) { struct blk_mq_hw_ctx *hctx; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 98e21b4cbf32..0cc09b975d8f 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -233,7 +233,12 @@ struct bio { atomic_t __bi_remaining; =20 /* The actual vec list, preserved by bio_reset() */ - struct bio_vec *bi_io_vec; + union { + struct bio_vec *bi_io_vec; + /* Driver specific dma map, valid IFF REQ_DMABUF is set */ + struct dma_buf_io_map *bi_dmabuf_map; + }; + struct bvec_iter bi_iter; =20 union { @@ -402,6 +407,7 @@ enum req_flag_bits { __REQ_DRV, /* for driver use */ __REQ_FS_PRIVATE, /* for file system (submitter) use */ __REQ_ATOMIC, /* for atomic write operations */ + __REQ_DMABUF, /* Using premmaped dma buffers */ /* * Command specific flags, keep last: */ @@ -434,6 +440,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) #define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_DMABUF (__force blk_opf_t)(1ULL << __REQ_DMABUF) =20 #define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP) =20 @@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op) return (op & REQ_OP_MASK) =3D=3D REQ_OP_DISCARD; } =20 +static inline bool op_is_dmabuf(blk_opf_t op) +{ + return op & REQ_DMABUF; +} + /* * Check if a bio or request operation is a zone management operation. */ diff --git a/include/linux/bvec.h b/include/linux/bvec.h index fc566ee1c1ff..b63914ff56e3 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -108,7 +108,8 @@ struct bvec_iter { unsigned int bi_idx; =20 /* - * Current offset in the bvec entry pointed to by `bi_idx`. + * Current offset in the bvec entry pointed to by `bi_idx` or into + * a dma-buf map. */ unsigned int bi_offset; } __packed __aligned(4); --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A98AC4F68CD for ; Wed, 14 Oct 2026 11:02:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975757; cv=none; b=q1ibI0uO/rD/nlAdsMAKQHR7iqq59N7hUOdbJ4n8DH+aTUjBiCHY16O2Z6xnU6aAuGuJcvMLouulo3eshT9G6EZJflhCfFOeGZOlMyO50vVa4O3xyqnbtfctg7lJ0zPhGm2Dv4HEBICjw8QvqOtF0udJsFC2c60Ym5ndOupSvD0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975757; c=relaxed/simple; bh=poMU3IMk+s1i+OL6PspfjWv+5wjZjdnLVtCEDY3Hn+k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BAfFQ3xdf7nNfHYQGzjby2783a9Fb/Hx9m+szgIPQjRJHHAu2njlItyXi7TvFKaveXWDau/WvGGp9zVYT7G39UsizxJw584AqLcl6zC94GZxOapyBVjf8JGQPzlXvt0ZVuKoeDCyC7Jzxl6HF0B9nLu76ODSs7vao7O4nZIjQMg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lgNO+bZQ; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lgNO+bZQ" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-4a16aaf2067so10596135e9.0 for ; Wed, 14 Oct 2026 04:02:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975748; x=1792580548; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=sjQY12JZJkEEoHUMGZ5pLsy3Pmh4yJs101KM8nm2S/U=; b=lgNO+bZQ/hnx2yCAL1IbW7Iij3HMfcE7QFPPFXXIrHszCMDgkl/yukvU2NsT0LuA8k XTYrAF3J7+KZ0mLr0dyTtvudv3W8K26zyQSPyCr+Gl0xk+zlTDr5MBDbiaFF48RmlAZz G6liAnDuYEL/AowlsBFYqbYiVjwVhPygUXqs/cVM22qW03HxCWxgizK6LJsvuuhl6iAh wfuzAaoNxC1STbH744wcuWm88gHQl91TjBIK3JCJQipocLiR/mh+CuAg8aw3+QxLDujS 3Pkne/qKlxAPAsIfCOlBTxlVI9+VKvE9PhcJ4PziH9BhgLsbVjblTnzNLLkEySovYlse KAOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975748; x=1792580548; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=sjQY12JZJkEEoHUMGZ5pLsy3Pmh4yJs101KM8nm2S/U=; b=CDSgY6PWzYG5Tpca32EWFLR2kOdZI64zQigdM2n5wJG70wtLP3N8vvZmk006HMU8K7 9hX7v8x743DevSmT2lXFic0IZCrAzypygp2eJecWjJNsPTMhiKKUgBo0adLo8XT0fzUd Cm+bFcc8bJPHnf2Dcj4xOR41hwj+0a4OcUvOTEnsDJGrTekO06djHgA62M80yYCvgzFP dxi5OQCmxKL4zlIw1Sm6u+tAHYpdCQmvxRk9d91cQBMqoieo9ooZ+pOuFtUd0Wp9CPKy Mmou2G9VT1OW4gYYK5+4Egyc9/oWbiu/v/kt4iRoVqIVtDOvz4b2p2MVpXYjrNJMAtdO kCFQ== X-Forwarded-Encrypted: i=1; AKwUvBx88VwZq4B6MC6EMMJEvNuBvr8gjVBky8VTm/0Dd7XnEk0qwENeE0rve+s4I18or6AYPVrtkXPFAGcnDjA=@vger.kernel.org X-Gm-Message-State: AFq9FYLjuyn+ZgPSClUFlWRAfgjpH2bASzCYifeW3VtwJ10IiH1tZCfg 5XZEJB94WGGK1TWPJT7sHGnloK7/ExO9s8MvTTbj7MyIXT8GwNQLsdUC X-Gm-Gg: AYBFou3Q42zgrlMrBlAUDG/Pzb+0sTLt3at3KvWRXWtg29HZ314PHcAOLdPUIwNBmNu crVsLw0yI/G2ibLhj+0t7XF/3c5x6DViLOqelk/i6VpI/SJyOIRdMvB2sB8nwRVy4dTmmy15ezJ jFTu1fGx2X5fMBrOGq5Lh+mQo4L8HQthhQ4UxFaJQHCCLTyRvz8u4cRL2jqwKKcY7hFwwpIJNtB Jjr/yQm3OuEEuo5bP5IoP/S6cCPJUOpfmqeJ1GCdDdj0mVWcBjZfQeFvpyM72oX6RCpP3clM5jp 34PXnQd6PqOxcI+c1/kkhBMza3RBPUm9Htj5/cZn6UPYJJ3PRb8OMMwVmnF9p7ktLSh3W2ugduS 7Afe8l8/ckyxbXU+TPtX/5+XaVjiWQzrn7eUhgWd2R2+y1P8L2tl5Z3vqvtRUW9Mw8n7le4mesO n9ted1F0cBj0EyXzrD+ibcZwjmrcaalifgB1VWtkHqdIXN7DmvmRAXJaiZCNzUpAEC0aQSnhPXL Y9SC8+k8H0k8oGGbW70JSdM432NjxGGU/fqSPo5QhbYYlXVe0QWAZn9Y3e5VjVB4Zh0hcRa09Er X-Received: by 2002:a05:600c:8716:b0:4a1:62b8:9aa1 with SMTP id 5b1f17b1804b1-4a18e44885dmr316691205e9.2.1791975747413; Wed, 14 Oct 2026 04:02:27 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:26 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 05/13] block: add dma-buf support for raw bdev Date: Wed, 14 Oct 2026 12:02:03 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a simple proxy implementation of init_dma_buf_io_ctx() forwarding the call to a new struct block_device_operations operation. Also reject dma-buf backed iterators for buffered IO. Reviewed-by: Christoph Hellwig [pavel: reject dma-buf without O_DIRECT] Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- block/fops.c | 24 ++++++++++++++++++++++++ include/linux/blkdev.h | 2 ++ 2 files changed, 26 insertions(+) diff --git a/block/fops.c b/block/fops.c index 827dc9eecba7..6c6700df943c 100644 --- a/block/fops.c +++ b/block/fops.c @@ -730,6 +730,9 @@ blkdev_direct_write(struct kiocb *iocb, struct iov_iter= *from) =20 static ssize_t blkdev_buffered_write(struct kiocb *iocb, struct iov_iter *= from) { + if (unlikely(iov_iter_is_dmabuf_map(from))) + return -EOPNOTSUPP; + return iomap_file_buffered_write(iocb, from, &blkdev_iomap_ops, NULL, NULL); } @@ -855,6 +858,11 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, st= ruct iov_iter *to) if (ret < 0 || !count) goto reexpand; } + if (unlikely(iov_iter_is_dmabuf_map(to))) { + if (!ret) + ret =3D -EOPNOTSUPP; + goto reexpand; + } =20 /* * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize @@ -958,6 +966,21 @@ static int blkdev_mmap_prepare(struct vm_area_desc *de= sc) return generic_file_mmap_prepare(desc); } =20 +static int blkdev_init_dma_buf_io_ctx(struct file *file, + struct dma_buf_io_ctx *ctx) +{ + struct block_device *bdev =3D file_bdev(file); + struct gendisk *disk =3D bdev->bd_disk; + + if (!(file->f_flags & O_DIRECT)) + return -EINVAL; + if (!disk->fops->init_dma_buf_io_ctx) + return -EOPNOTSUPP; + if (bdev_has_integrity_csum(bdev)) + return -EOPNOTSUPP; + return disk->fops->init_dma_buf_io_ctx(bdev, ctx); +} + const struct file_operations def_blk_fops =3D { .open =3D blkdev_open, .release =3D blkdev_release, @@ -976,6 +999,7 @@ const struct file_operations def_blk_fops =3D { .fallocate =3D blkdev_fallocate, .uring_cmd =3D blkdev_uring_cmd, .fop_flags =3D FOP_BUFFER_RASYNC | FOP_DONTCACHE, + .init_dma_buf_io_ctx =3D blkdev_init_dma_buf_io_ctx, }; =20 static __init int blkdev_init(void) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index d003a9d2d1f6..2d5c30823aab 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1599,6 +1599,8 @@ struct block_device_operations { /* returns the length of the identifier or a negative errno: */ int (*get_unique_id)(struct gendisk *disk, u8 id[16], enum blk_unique_id id_type); + int (*init_dma_buf_io_ctx)(struct block_device *, + struct dma_buf_io_ctx *); struct module *owner; const struct pr_ops *pr_ops; =20 --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15B914F68C3 for ; Wed, 14 Oct 2026 11:02:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975759; cv=none; b=DYJKE2335ULTvlN4AYpqQmSeZ1p8MTsfz2YdMGQAXLpo7Y7ul54C/qvmIZNq0uOHgoiUN5WILLwSA5m+1PP80d8R8DhvnJD8r8JCUJ96u51dvKglSql04aGOjPMPbb+aqKeCRtFBnuBL7WlvKthSOVcqluOtURHox0Lvv/KSCDM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975759; c=relaxed/simple; bh=0a99AkCdYYlGDSLNyvcoHvxyA0h7cLM0HZDY04APdmg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jcIMBGI082KHp1UCfwGhzEUszV96Qd8gzW+xtDrCguVN1z8/T2/uf9uu79P7eE65pw11ZWQDu/rMs8DeRPatZXSp1q2RayFaIiAowP76zPa9Ts5tfxmseh/SXft+nWs5zdP0l34RUcCICs/fYBcCpHfaqnXwVr3pLl2igklxPcc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q1kO+xqz; arc=none smtp.client-ip=209.85.128.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q1kO+xqz" Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-4a1635f7c89so8281235e9.2 for ; Wed, 14 Oct 2026 04:02:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975750; x=1792580550; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SVgIYOF9sl5M4BcUqDAxFWbO5yiW+j2eX94ter+J4Q4=; b=Q1kO+xqzcnhyOMC/CmPun+hCV1NJP2CvZTgmiHVIE3lQoK3G8GakCw172Frx2IGI08 ed/GXliK+ytILgZIPBI6mPV3kdAExQTxHL9gorMW6XZNqNtMKC+0vSb5wIGNtkZ9H+J8 xrBqKwGb9Z3zpRrCtymlGgvQxofTGShk/4z2Dwt9/Yjto2bpYbAJwymXvZCXOa31/bqy LrwObguWN9VakhbrX+q6QBCXtcKbH5Z5v7JrPpCSRhLfBvhz+yoHw2gVd153XlzqEAQ6 +3ESc3ET8xIh5CnYo31zLwr6xXUV5wkQZpMfIo/qscwkDYOMfbrVfwDJDs1oulfcUUZc ZWBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975750; x=1792580550; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SVgIYOF9sl5M4BcUqDAxFWbO5yiW+j2eX94ter+J4Q4=; b=hbVxHeVZ7jQGdevhXzMYhy0arE8God6KaqC4ruKwuN7ijw2/RWeOX13Nj+VjORiBQt 3GP+7zC7iAjt0Cj0Dw6lmoF7C7PqOO/dX9vgI7oP8u3BOwjR+dySfmiVo1E6wkkToZwj c+BSf1E70hHuWl2P+7tJW7BHjYiA7WyXbDATDjIVUaKwERYo1Mcjh/d13R0jDkfzh7HH /2yC6XKwG18JAqFNNQxPPsYBhgeo8/NI5PB1PW/G4KEWmQVg6xw0XhWyisVaDLImngOz QP0rZPDLr5z3Q78gc3oRjxEZTXMRzCNxX+D+ZhK/em6A6M84gHi4Wg91ethTZ86WjKjo uHgg== X-Forwarded-Encrypted: i=1; AKwUvBwbBsTON0BvOsedETsnMl/WrN8fG2BzbbOqQMvaLLLWU5SINqzATZVsw2hws/BnrLdipI2ipKuScKyoNW4=@vger.kernel.org X-Gm-Message-State: AFq9FYIkR8tIci0CBu3XhUhAZIof5QaL8/4tlgEwytZ0W4+7peSqFvwi Ni2FzWptKMIRk9AismRvY9emxdH2aosW29//2lUYTbVKan404R3yhjQ1 X-Gm-Gg: AYBFou1dD4WLHPkoMUgghpZ8kFhsbC1GEP6MbjrsBIes6rrjM4GYVLn5HKxI1CuRAw8 ot+vJJybSNRWroTWYYaVuEtXrvo/BrlITliq1zTsxTmAS3ZjjgXzVXX4ckK7mfmXl+YZpHhrPBk BzSepPeuUFPEfpVW7M3D85x9k6InKQudPME+hcYmjrLg1PzT+ILoZVTSHIzpSstFcPC8albb5Xb 554vW9Y8c8XQxTcpKKL2c8PxQhuqZLcxbs4Ypmj7qp7a9xrwHNUTJXbZr+4BnnXGFmyFKOKbu7v ILmLJwaxxp111mSjxMnF0rmmIhcPwO8U66gIw3UMfiqjpHpWclEPtXEtGFwnOgV4iLECshfnh0z t54zfzGlm1d6KhEC6xb6U4Q0cZJnxL3NmQbwL4T5Qi+gcACODFR3acKuUYD++Ub1IyRJm4gMtWG b91/D56LKovz4d2oSBK1sPcjfPLMit1Z3HnsPgwNRAYmS+aGwG4HxJtya2KK8uPPctMbOA/60zf e9ttTwOhgKPNygmig51HJZC5D3TV8GvvNsErMmeBlJXCOVbsUG93umHnpNSufaxlPacvP1nJhXa X-Received: by 2002:a05:600c:3e87:b0:4a0:a34:6e7d with SMTP id 5b1f17b1804b1-4a18e4bacbfmr282470095e9.18.1791975749999; Wed, 14 Oct 2026 04:02:29 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:29 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 06/13] nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl Date: Wed, 14 Oct 2026 12:02:04 +0100 Message-ID: <42788b01948b58b891476006cb7ee9f9f5e917db.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Anuj Gupta Rename the blk_dma_iter based nvme_pci_sgl_set_data() to nvme_pci_dma_iter_set_sgl(). Suggested-by: Christoph Hellwig Reviewed-by: Christoph Hellwig Signed-off-by: Anuj Gupta Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- drivers/nvme/host/pci.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 77317e5d00f9..a65b77726e4b 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1147,7 +1147,7 @@ static blk_status_t nvme_pci_setup_data_prp(struct re= quest *req, return BLK_STS_IOERR; } =20 -static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge, +static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge, struct blk_dma_iter *iter) { sge->addr =3D cpu_to_le64(iter->addr); @@ -1177,7 +1177,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct re= quest *req, iod->cmd.common.flags =3D NVME_CMD_SGL_METABUF; =20 if (entries =3D=3D 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) { - nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter); + nvme_pci_dma_iter_set_sgl(&iod->cmd.common.dptr.sgl, iter); iod->total_len +=3D iter->len; return BLK_STS_OK; } @@ -1199,7 +1199,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct re= quest *req, iter->status =3D BLK_STS_IOERR; break; } - nvme_pci_sgl_set_data(&sg_list[mapped++], iter); + nvme_pci_dma_iter_set_sgl(&sg_list[mapped++], iter); iod->total_len +=3D iter->len; } while (blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter)); =20 @@ -1359,13 +1359,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct= request *req) iod->cmd.common.metadata =3D cpu_to_le64(sgl_dma); if (entries =3D=3D 1) { iod->meta_total_len =3D iter.len; - nvme_pci_sgl_set_data(sg_list, &iter); + nvme_pci_dma_iter_set_sgl(sg_list, &iter); return BLK_STS_OK; } =20 sgl_dma +=3D sizeof(*sg_list); do { - nvme_pci_sgl_set_data(&sg_list[++i], &iter); + nvme_pci_dma_iter_set_sgl(&sg_list[++i], &iter); iod->meta_total_len +=3D iter.len; } while (blk_rq_integrity_dma_map_iter_next(req, dev->dev, &iter)); =20 --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EF8E4F5E06 for ; Wed, 14 Oct 2026 11:02:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975765; cv=none; b=n94gBQu804KiZR9lFbIzXdiRHr7+A1NdVWTqqEJjO6jJORsRg9Dj8UL6ClM83yexK8iapGI3MHG1f95JtLjCPY6Uxvjhpwjzqwbg0hmA7oHeaD6En+Bc12KFAGtMIHJI52VWItSjmhGcFR1655LIbVMh0laPa4101AK6XRZ6kdE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975765; c=relaxed/simple; bh=ixhh2GRwYMc7RdtfH5U8ta21UkU3OCGiQaTzkgMFwUQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nwbDgjJoC4CF419YbWHM3/FE19UJg7wu5DWhJ3cNCrxiTzCM7/1WN8PWxbxEBAI39dFSrB5GGWDNvgvDZ6/0gA6Gmo7WpZ9b0LHqYoftib8fyCPqtF8HLbxFZTjP9idMBWPS4Yknk4SVPSHhmIT8n/eFNDYSxl1WMEVnuHlJM1w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IxObNTWs; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IxObNTWs" Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-49fbb2acc90so34172095e9.2 for ; Wed, 14 Oct 2026 04:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975752; x=1792580552; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dgD4ZRKr/dSiiTwqauwPBK2KpAxV+NZiR9fa1tWOdeA=; b=IxObNTWs+PDB9SxaVo/xi/TR8A4oRf6WOMUUYDuW+6Oc9XKsSLDQwd16g/yhkcSOap 5fh8vZbP+Nsh+w+Bx574uiPaO8B0pve1FzLTWEHUZcZmddoSIKsToqmmwxpkSc21Usq5 LY3/jrepf4MRcithEC9Muzv3L8CGmqfTO/RfoujNuWwA+jKyMDK9spsofsuh3ZDxwVpf Otcgbs704uqpzs1FnGtkxZDcQ7X+DB+KxAJemVSynnkIkT4H+bl12/J6oYzVwiBvbWSw DSQ+wzNR+SQW2E5JbaxtIVMiV4SaN1EtPGgLtRxClcVq0jP4zolIz4TNTf/hCKOc23gQ AnMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975752; x=1792580552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=dgD4ZRKr/dSiiTwqauwPBK2KpAxV+NZiR9fa1tWOdeA=; b=XcCqGZsnyjPVlNoD6g88hlU+B5ewQIbHbU4Sf8qxD6bfB5MzxY5MMb4PIDP8SGM/pY T2J+LVKOns49+U/WYRdANbcqe05jQegNKqwDEp2G6WnN61/jg6qMVdLil18dX/co4oHw Fb59dQ0fpDED3kI9V9WENScKIYWXt4RM82sW7s27eyi9UZkC7kUH6jB0M2fIWx8M+GDi SVMPHQLpRQ1mUcF2ql+IXIQhoNNFJMlo6tdJt6y/gsD781KcXClQwEn1/kWs4igr9pwF BjWQQBe5BJ0WpvcNqhUOlxrHTkhqEhVoigYKY72qdQDCd8MjtqGFp41JdJywbMLQDKtf +AWA== X-Forwarded-Encrypted: i=1; AKwUvBzqtS2Zh6IsywfoOWB8A+D5yL+TDUTBgEwxrb61/Qt5+HPaLh2Yj5WU4lmX+85nNYhThgQiOPcWc5Fx+YI=@vger.kernel.org X-Gm-Message-State: AFq9FYK0ctRhUDIzVNte0h7uQsAqY/Mol3lAVCZqqU8IWfm/0nYQ/iWZ 8jJ6b2H0sOgX1Sp4aDeC0HEkHF4QXmuWNbrhYzxh2ah53P6BhrIcun+D X-Gm-Gg: AYBFou2Xaqxq7LhiecZEL4YWDVtTbJMgakMfZuEzvQZvV4AKAr1FkUu1jntA7bkAbow DVx/PAQyCpTTBk4EiYrHmV0/ng+2v1ml/9OHTea4yORwDHZ5VwyD+ZyInBSv9h/SxMK3DZflmRv zbUgJJ/+gQg9W76itNm09E4KAtCj8iTQmsXUggAaAh44s4LYUF9dCV14FRyRdZbzokUp6XKlrVT Eb8ixwrjRbKQ1RsJv3qaQg33mItyx6CtNI2KWpBxSqYn7XZC3zo7nDfYCxHQA9/Mm44TJGtTSw3 AV7Fp5z+hL7FUQnaWJlLUa4FUCa5UjxJYgsS1FAc3hpSQTRlszLDMVV+HcidDckhrTHXUsPzand l5THPxhJ8/VYi2f99vszFEqY1dU+WFMJblNrqN4gU2ZiWN17XQ4Vyp1GJ/6wDBUeQ9REJg9CaFw d5Mpd1tovtzE0753T6Aq9bQwtog3sKBgsGPLH6WPCH/Yb8H8OM18dKLhH78ttn/8UmRTQdDe5A9 ba7Z3fboB2AiNn69bNLSJKSnJxL3g4t396ZAP8l9o9iQtS4hNlL0N2sFeEBBnIpXmfCny4juJfX X-Received: by 2002:a05:600c:4e11:b0:4a1:7f41:b9dc with SMTP id 5b1f17b1804b1-4a18e4d67famr306600635e9.23.1791975751569; Wed, 14 Oct 2026 04:02:31 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:30 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 07/13] nvme-pci: implement dma-buf backed requests Date: Wed, 14 Oct 2026 12:02:05 +0100 Message-ID: <9540e74b5d2dd1a5dc0d436bc0c2a6f5e18fc389.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable BIO_DMABUF_MAP backed requests. On registration we map the dma-buf and store it as a prp list, which is then used to initialise requests. All attached contexts are stored in a new list dmabuf_ctxs, and additions/removals are synchronised with dmabuf_lock. Suggested-by: Keith Busch Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Reviewed-by: Christoph Hellwig Tested-by: Anuj Gupta --- drivers/nvme/host/Makefile | 1 + drivers/nvme/host/core.c | 12 +++ drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/pci-dmabuf.c | 169 +++++++++++++++++++++++++++++++++ drivers/nvme/host/pci-dmabuf.h | 75 +++++++++++++++ drivers/nvme/host/pci.c | 93 ++++++++++++++++++ 6 files changed, 352 insertions(+) create mode 100644 drivers/nvme/host/pci-dmabuf.c create mode 100644 drivers/nvme/host/pci-dmabuf.h diff --git a/drivers/nvme/host/Makefile b/drivers/nvme/host/Makefile index 67563a69f7dc..8288b986d713 100644 --- a/drivers/nvme/host/Makefile +++ b/drivers/nvme/host/Makefile @@ -21,6 +21,7 @@ nvme-core-$(CONFIG_NVME_HWMON) +=3D hwmon.o nvme-core-$(CONFIG_NVME_HOST_AUTH) +=3D auth.o =20 nvme-y +=3D pci.o +nvme-$(CONFIG_DMA_SHARED_BUFFER) +=3D pci-dmabuf.o =20 nvme-fabrics-y +=3D fabrics.o =20 diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index 7eb0f9fe6b18..2f9b2262f77c 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2708,6 +2708,17 @@ static int nvme_report_zones(struct gendisk *disk, s= ector_t sector, #define nvme_report_zones NULL #endif /* CONFIG_BLK_DEV_ZONED */ =20 +static int nvme_init_dma_buf_io_ctx(struct block_device *bdev, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_ns *ns =3D bdev->bd_disk->private_data; + struct nvme_ctrl *ctrl =3D ns->ctrl; + + if (!ctrl->ops->init_dma_buf_io_ctx) + return -EINVAL; + return ctrl->ops->init_dma_buf_io_ctx(ctrl, ctx); +} + const struct block_device_operations nvme_bdev_ops =3D { .owner =3D THIS_MODULE, .ioctl =3D nvme_ioctl, @@ -2718,6 +2729,7 @@ const struct block_device_operations nvme_bdev_ops = =3D { .get_unique_id =3D nvme_get_unique_id, .report_zones =3D nvme_report_zones, .pr_ops =3D &nvme_pr_ops, + .init_dma_buf_io_ctx =3D nvme_init_dma_buf_io_ctx, }; =20 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u32 mask, u32 val, diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index a643c4723a69..be32e9a5ca5a 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -668,6 +668,8 @@ struct nvme_ctrl_ops { int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); void (*print_device_info)(struct nvme_ctrl *ctrl); bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); + int (*init_dma_buf_io_ctx)(struct nvme_ctrl *ctrl, + struct dma_buf_io_ctx *ctx); unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin); }; =20 diff --git a/drivers/nvme/host/pci-dmabuf.c b/drivers/nvme/host/pci-dmabuf.c new file mode 100644 index 000000000000..161e4ffa6919 --- /dev/null +++ b/drivers/nvme/host/pci-dmabuf.c @@ -0,0 +1,169 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Code for supporting dma-buf I/O maps in the NVMe driver. + * Copyright (C) 2026 Pavel Begunkov + */ + +#include + +#include "pci-dmabuf.h" + +static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *att= ach) +{ + struct dma_buf_io_ctx *ctx =3D attach->importer_priv; + + dma_buf_io_invalidate_mappings(ctx); +} + +static const struct dma_buf_attach_ops nvme_dmabuf_importer_ops =3D { + .invalidate_mappings =3D nvme_dmabuf_invalidate_mappings, + .allow_peer2peer =3D true, +}; + +static struct dma_buf_io_map *nvme_dma_buf_io_map(struct dma_buf_io_ctx *c= tx) +{ + unsigned nr_entries =3D ctx->dmabuf->size / NVME_CTRL_PAGE_SIZE; + struct nvme_dma_buf_io_ctx *nvme_ctx =3D ctx->dev_priv; + struct dma_buf_attachment *attach =3D nvme_ctx->attach; + unsigned long tmp, i =3D 0; + struct nvme_dmabuf_map *map; + struct scatterlist *sg; + struct sg_table *sgt; + int ret; + + dma_resv_assert_held(ctx->dmabuf->resv); + + if (ctx->dmabuf->size % NVME_CTRL_PAGE_SIZE) + return ERR_PTR(-EINVAL); + + map =3D kvmalloc_flex(*map, dma_list, nr_entries); + if (!map) + return ERR_PTR(-ENOMEM); + map->nr_entries =3D nr_entries; + + sgt =3D dma_buf_map_attachment(attach, ctx->dir); + if (IS_ERR(sgt)) { + ret =3D PTR_ERR(sgt); + sgt =3D NULL; + goto err; + } + + for_each_sgtable_dma_sg(sgt, sg, tmp) { + dma_addr_t dma_addr =3D sg_dma_address(sg); + unsigned long sg_len =3D sg_dma_len(sg); + + if ((sg_len % NVME_CTRL_PAGE_SIZE) || + (dma_addr % NVME_CTRL_PAGE_SIZE)) { + ret =3D -EINVAL; + goto err; + } + while (sg_len) { + map->dma_list[i++] =3D dma_addr; + dma_addr +=3D NVME_CTRL_PAGE_SIZE; + sg_len -=3D NVME_CTRL_PAGE_SIZE; + } + } + + ret =3D dma_buf_io_init_map(ctx, &map->base, sgt); + if (ret) + goto err; + map->sgt =3D sgt; + return &map->base; +err: + if (sgt) + dma_buf_unmap_attachment(attach, sgt, ctx->dir); + kvfree(map); + return ERR_PTR(ret); +} + +static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx *ctx, + struct dma_buf_io_map *map_base) +{ + struct nvme_dma_buf_io_ctx *nvme_ctx =3D ctx->dev_priv; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(map_base); + + dma_resv_assert_held(ctx->dmabuf->resv); + + dma_buf_unmap_attachment(nvme_ctx->attach, map->sgt, ctx->dir); + kvfree(map); +} + +static void nvme_dma_buf_io_release(struct dma_buf_io_ctx *base_ctx) +{ + struct nvme_dma_buf_io_ctx *ctx =3D base_ctx->dev_priv; + struct nvme_dma_buf_list *buf_list =3D ctx->buf_list; + + mutex_lock(&buf_list->lock); + if (!list_empty(&ctx->list)) { + dma_buf_detach(ctx->base->dmabuf, ctx->attach); + list_del_init(&ctx->list); + } + mutex_unlock(&buf_list->lock); + + nvme_put_ctrl(ctx->ctrl); + kfree(ctx); +} + +void __nvme_pci_remove_dmabuf(struct nvme_dma_buf_list *list) +{ + struct nvme_dma_buf_io_ctx *ctx, *tmp; + + mutex_lock(&list->lock); + list->dead =3D true; + list_for_each_entry_safe(ctx, tmp, &list->ctxs, list) { + dma_buf_io_detach(ctx->base); + dma_buf_detach(ctx->base->dmabuf, ctx->attach); + list_del_init(&ctx->list); + } + mutex_unlock(&list->lock); +} + +const struct dma_buf_io_ops nvme_dma_buf_io_ops =3D { + .map =3D nvme_dma_buf_io_map, + .unmap =3D nvme_dma_buf_io_unmap, + .release =3D nvme_dma_buf_io_release, +}; + +static int nvme_init_io_ctx_locked(struct nvme_ctrl *ctrl, + struct nvme_dma_buf_list *buf_list, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_dma_buf_io_ctx *nvme_ctx; + struct dma_buf_attachment *attach; + + if (buf_list->dead) + return -ENODEV; + + nvme_ctx =3D kzalloc_obj(*nvme_ctx); + if (!nvme_ctx) + return -ENOMEM; + + attach =3D dma_buf_dynamic_attach(ctx->dmabuf, ctrl->dev, + &nvme_dmabuf_importer_ops, ctx); + if (IS_ERR(attach)) { + kfree(nvme_ctx); + return PTR_ERR(attach); + } + + nvme_get_ctrl(ctrl); + list_add(&nvme_ctx->list, &buf_list->ctxs); + nvme_ctx->attach =3D attach; + nvme_ctx->base =3D ctx; + nvme_ctx->ctrl =3D ctrl; + nvme_ctx->buf_list =3D buf_list; + ctx->dev_priv =3D nvme_ctx; + ctx->dev_ops =3D &nvme_dma_buf_io_ops; + return 0; +} + +int __nvme_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct nvme_dma_buf_list *buf_list, + struct dma_buf_io_ctx *ctx) +{ + int ret; + + mutex_lock(&buf_list->lock); + ret =3D nvme_init_io_ctx_locked(ctrl, buf_list, ctx); + mutex_unlock(&buf_list->lock); + return ret; +} diff --git a/drivers/nvme/host/pci-dmabuf.h b/drivers/nvme/host/pci-dmabuf.h new file mode 100644 index 000000000000..b3c9fd485878 --- /dev/null +++ b/drivers/nvme/host/pci-dmabuf.h @@ -0,0 +1,75 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2026 Pavel Begunkov + */ + +#ifndef _NVME_PCI_DMABUF_H +#define _NVME_PCI_DMABUF_H + +#include + +#include "nvme.h" + +struct nvme_dma_buf_list { + struct list_head ctxs; + struct mutex lock; + bool dead; +}; + +struct nvme_dma_buf_io_ctx { + struct dma_buf_attachment *attach; + struct nvme_dma_buf_list *buf_list; + struct dma_buf_io_ctx *base; + struct nvme_ctrl *ctrl; + struct list_head list; +}; + +struct nvme_dmabuf_map { + struct dma_buf_io_map base; + struct sg_table *sgt; + unsigned nr_entries; + dma_addr_t dma_list[] __counted_by(nr_entries); +}; + +static inline struct nvme_dmabuf_map *to_nvme_dmabuf_map( + struct dma_buf_io_map *map) +{ + return container_of(map, struct nvme_dmabuf_map, base); +} + +static inline struct nvme_dmabuf_map *nvme_req_to_map(struct request *req) +{ + struct dma_buf_io_map *map =3D req->bio->bi_dmabuf_map; + + return to_nvme_dmabuf_map(map); +} + +void __nvme_pci_remove_dmabuf(struct nvme_dma_buf_list *list); +int __nvme_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct nvme_dma_buf_list *buf_list, + struct dma_buf_io_ctx *ctx); + +static inline void nvme_pci_remove_dmabuf(struct nvme_dma_buf_list *list) +{ + if (IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + __nvme_pci_remove_dmabuf(list); +} + +static inline int nvme_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct nvme_dma_buf_list *buf_list, + struct dma_buf_io_ctx *ctx) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + return __nvme_init_dma_buf_io_ctx(ctrl, buf_list, ctx); +} + +static inline void nvme_init_buf_list(struct nvme_dma_buf_list *buf_list) +{ + INIT_LIST_HEAD(&buf_list->ctxs); + mutex_init(&buf_list->lock); + buf_list->dead =3D false; +} + + +#endif /* _NVME_PCI_DMABUF_H */ \ No newline at end of file diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index a65b77726e4b..a0506c93fa16 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -30,6 +30,7 @@ =20 #include "trace.h" #include "nvme.h" +#include "pci-dmabuf.h" =20 #define SQ_SIZE(q) ((q)->q_depth << (q)->sqes) #define CQ_SIZE(q) ((q)->q_depth * sizeof(struct nvme_completion)) @@ -318,6 +319,7 @@ struct nvme_dev { bool hmb; struct sg_table *hmb_sgt; mempool_t *dmavec_mempool; + struct nvme_dma_buf_list dmabufs; =20 /* shadow doorbell buffer support: */ __le32 *dbbuf_dbs; @@ -865,6 +867,76 @@ static void nvme_free_descriptors(struct request *req) } } =20 +static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req, + struct nvme_queue *nvmeq) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_dmabuf_map *map =3D nvme_req_to_map(req); + unsigned bvec_done =3D req->bio->bi_iter.bi_offset; + unsigned map_idx =3D bvec_done / NVME_CTRL_PAGE_SIZE; + unsigned offset =3D bvec_done & (NVME_CTRL_PAGE_SIZE - 1); + int length =3D blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset); + dma_addr_t *dma_list =3D map->dma_list; + u64 prp1_dma =3D dma_list[map_idx++] + offset; + u64 dma_addr, prp2_dma; + dma_addr_t prp_dma; + __le64 *prp_list; + unsigned i; + + if (length <=3D 0) { + prp2_dma =3D 0; + goto done; + } + + if (length <=3D NVME_CTRL_PAGE_SIZE) { + prp2_dma =3D dma_list[map_idx]; + goto done; + } + + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <=3D + NVME_SMALL_POOL_SIZE / sizeof(__le64)) + iod->flags |=3D IOD_SMALL_DESCRIPTOR; + + prp_list =3D dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) + return BLK_STS_RESOURCE; + + iod->descriptors[iod->nr_descriptors++] =3D prp_list; + prp2_dma =3D prp_dma; + i =3D 0; + for (;;) { + if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { + __le64 *old_prp_list =3D prp_list; + + prp_list =3D dma_pool_alloc(nvmeq->descriptor_pools.large, + GFP_ATOMIC, &prp_dma); + if (!prp_list) + goto free_prps; + iod->descriptors[iod->nr_descriptors++] =3D prp_list; + prp_list[0] =3D old_prp_list[i - 1]; + old_prp_list[i - 1] =3D cpu_to_le64(prp_dma); + i =3D 1; + } + + dma_addr =3D dma_list[map_idx++]; + prp_list[i++] =3D cpu_to_le64(dma_addr); + + length -=3D NVME_CTRL_PAGE_SIZE; + if (length <=3D 0) + break; + } +done: + iod->cmd.common.dptr.prp1 =3D cpu_to_le64(prp1_dma); + iod->cmd.common.dptr.prp2 =3D cpu_to_le64(prp2_dma); + return BLK_STS_OK; +free_prps: + iod->cmd.common.dptr.prp1 =3D cpu_to_le64(prp1_dma); + iod->cmd.common.dptr.prp2 =3D cpu_to_le64(prp2_dma); + nvme_free_descriptors(req); + return BLK_STS_RESOURCE; +} + static void nvme_free_prps(struct request *req, unsigned int attrs) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); @@ -943,6 +1015,9 @@ static void nvme_unmap_data(struct request *req) struct device *dma_dev =3D nvmeq->dev->dev; unsigned int attrs =3D 0; =20 + if (blk_mq_rq_is_dmabuf(req)) + goto out_free_desc; + if (iod->flags & IOD_SINGLE_SEGMENT) { static_assert(offsetof(union nvme_data_ptr, prp1) =3D=3D offsetof(union nvme_data_ptr, sgl.addr)); @@ -967,6 +1042,7 @@ static void nvme_unmap_data(struct request *req) nvme_free_prps(req, attrs); } =20 +out_free_desc: if (iod->nr_descriptors) nvme_free_descriptors(req); } @@ -1257,6 +1333,9 @@ static blk_status_t nvme_map_data(struct request *req) struct blk_dma_iter iter; blk_status_t ret; =20 + if (blk_mq_rq_is_dmabuf(req)) + return nvme_rq_setup_dmabuf_map(req, nvmeq); + /* * Try to skip the DMA iterator for single segment requests, as that * significantly improves performances for small I/O sizes. @@ -3357,6 +3436,9 @@ static void nvme_dev_disable(struct nvme_dev *dev, bo= ol shutdown) nvme_unquiesce_admin_queue(&dev->ctrl); } mutex_unlock(&dev->shutdown_lock); + + if (shutdown) + nvme_pci_remove_dmabuf(&dev->dmabufs); } =20 static int nvme_disable_prepare_reset(struct nvme_dev *dev, bool shutdown) @@ -3564,6 +3646,14 @@ static unsigned long nvme_pci_get_virt_boundary(stru= ct nvme_ctrl *ctrl, return 0; } =20 +static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_dev *dev =3D to_nvme_dev(ctrl); + + return nvme_init_dma_buf_io_ctx(ctrl, &dev->dmabufs, ctx); +} + static const struct nvme_ctrl_ops nvme_pci_ctrl_ops =3D { .name =3D "pcie", .module =3D THIS_MODULE, @@ -3579,6 +3669,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = =3D { .print_device_info =3D nvme_pci_print_device_info, .supports_pci_p2pdma =3D nvme_pci_supports_pci_p2pdma, .get_virt_boundary =3D nvme_pci_get_virt_boundary, + .init_dma_buf_io_ctx =3D nvme_pci_init_dma_buf_io_ctx, }; =20 static int nvme_dev_map(struct nvme_dev *dev) @@ -3701,6 +3792,7 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci= _dev *pdev, return ERR_PTR(-ENOMEM); INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work); mutex_init(&dev->shutdown_lock); + nvme_init_buf_list(&dev->dmabufs); =20 dev->nr_write_queues =3D write_queues; dev->nr_poll_queues =3D poll_queues; @@ -4346,5 +4438,6 @@ MODULE_AUTHOR("Matthew Wilcox = "); MODULE_LICENSE("GPL"); MODULE_VERSION("1.0"); MODULE_DESCRIPTION("NVMe host PCIe transport driver"); +MODULE_IMPORT_NS("DMA_BUF"); module_init(nvme_init); module_exit(nvme_exit); --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EC064FD79D for ; Wed, 14 Oct 2026 11:02:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975764; cv=none; b=LUOCbJr01h4QhJbq5/w9gMYJCwXHk9qpXXPdcv47lw6HVf74/Efmg5kvwuIQJbaev9EDrzTT0QqLS9P4SukpE3HW1i56Y1beuVS6ckQSqGx9M7Wkd8eNg9tlnBlSW9qH0FihE6glBGcCWtEefRzGlmfsZn0jb0l+v2zGgO5jQd4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975764; c=relaxed/simple; bh=iy9U6gJyxks5ymTnPsPTZg2hiScYxD3ewbuzhpHoVzk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WC+AFYVr1g+ryTHe507fzFsgaU2ZXog/JFhZMWocVi0vFiejwoS9LF0DWFPVAmeaiZx/REpWRYD3brK3cIh8N+p5aJjRCXe3l379u0v/yDHgVVDCPJAti7YMqnwO8wxa9WGcCquz1w6HTkBhSAJwh+nfDCF9N/foIqPdn3vQe1A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Mbk6OF0g; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Mbk6OF0g" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4a16aaf2067so10597505e9.0 for ; Wed, 14 Oct 2026 04:02:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975754; x=1792580554; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SSeH33Tp73HrlxrRD40ZRw/kp0xshqtHbX/nV8WDcGA=; b=Mbk6OF0gK7+DKGMPYpyHbb0WOqDBuwPId7kiJSuyuugw2TVNLlQgDdp9ISEGBYBg21 4n+JvfBFwz5XQDe13kYeyamYFLGuYIjL8AT/c1aCRCG/cZaAgYH6K0QNnEzYKha8kEA1 IfpzpRDFGFrBImE2ubyAMf2UspBXbfR/HUO3IkpSx+LXn0GMuOttMxH+30oPVBtFZ0CT WrgnLG5ThL28AR6WmZWJlgtVuZdffNwdKAYH5cVPdP/EkrVhLHUbAy/D30YPNHGdV+N3 9NwFIkVK1B7hnyXP915ToZjLxrFNdZSust3p8lQ/SyeDACvZXefTnsmAsRa0SO/LOYjC BImw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975754; x=1792580554; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=SSeH33Tp73HrlxrRD40ZRw/kp0xshqtHbX/nV8WDcGA=; b=NNChIzlfZy4Fg122vAGmEyA/vVcoztbzgr0ZCqg7JmGNA1Q4SPvy/Txtq0prM7/hRH imsm1eZ87g7z3oF/C2/9SJfyMPxvBwNn5m1WodDuwcK9ri9huiaMMfBjvL153fh9dgz7 cnEILk5aNV2XVR3QhcFwKVjpFLNTwcsmA5aEzHPmYZ1LcaOM8C7lc7Pp/OkIAV6DSVne JuXAUfrkDMSGDIfVNEuOeMN1Oj+uI6ad71oqLX3Qi8gwYUsm5173qyJf/oKfhlVTmUba zfP3ucxFXofBd9cq4jLiMIc6pMQICkbpqb+CMTARL+oyxxfcS76BDzfo0xmS4t5vn/qo dfQw== X-Forwarded-Encrypted: i=1; AKwUvBypde5ANJ+0wgr7oVNLLdlJWGs9ek5gnpz5ihRbhlCK/ZN0kbeZXV0Xsv3OdB54+dC/oD5FUhk4+dUBeno=@vger.kernel.org X-Gm-Message-State: AFq9FYKBIowwzXkPRLnd9wjqg7NNeBH9vsVbZVS1e6M5smyx+Rk+oAit PCasNIZTaK8J7TOh8Ro5DQQwJu7LH9LHydUv12QhhWJg0lNPM1bat3xf X-Gm-Gg: AYBFou3pLHLfOtFRz8KUM9u80DN3bjH7JEh1g2t2ceF6PK0gkAb9EV0bCGlLIMUA0JW ffNfdpBpE6ubQg2VRFtG/FpAKbaTbpP6brSZkohwQzg9b02xJkgZatzGLEMUUrU5QfmxF9HqI0g gknL6yxlcGJpukF+R9bE03zWDDT86OzTRhyPs5sQNeJpi5sGU41BgRCHmpEB/xjyeZYKbtw3Q24 5EN5Zr6j89PotNmYDTu2OI15rlpuOMB3vI5jI09G4J7TIAjDH3evdW/B4jwNgj/HlL715TSManz 7Mhb3EKt5s3L74j4Bufd2tAD4pIiOCm48yYLiZAxSbELDXlLAwmaXC4CNWyLb/Gmvs2rj99sK89 gh3MuY8AzQKeX++c3+4xvzWWMxaz3nyFvb4i2HIxriwIBLfUCfeODcubvY7IKgGmlVoJ3Ltnj6l eHpiUB2Ivcz7dYoTkxt/dEaK9RMgsxTiaAGpDkDLUjk0Zo/kD4gYd2R/IfBMKlxOcjRw8gn7Wk8 zt0IiWN14YeAFCMcx7+dWvl7IZudaHrxIzljzKNGaHpYm+qJbYJkKaDonlKi060INa5xoOHU7XB X-Received: by 2002:a05:600c:34cf:b0:4a0:159b:d2a9 with SMTP id 5b1f17b1804b1-4a18e4d68aamr259014055e9.24.1791975754208; Wed, 14 Oct 2026 04:02:34 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:32 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 08/13] nvme-pci: add SGL support for the dmabuf path Date: Wed, 14 Oct 2026 12:02:06 +0100 Message-ID: <86fb4caeddaa58fafad106548f82950782ea04fd.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Anuj Gupta Add SGL support in addition to PRP for dmabuf-backed requests, building the descriptor list from the mapping's sg_table. Reviewed-by: Christoph Hellwig Signed-off-by: Anuj Gupta Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- drivers/nvme/host/pci.c | 150 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 146 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index a0506c93fa16..79b3c1092e4d 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1223,12 +1223,18 @@ static blk_status_t nvme_pci_setup_data_prp(struct = request *req, return BLK_STS_IOERR; } =20 +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge, + dma_addr_t addr, u32 len) +{ + sge->addr =3D cpu_to_le64(addr); + sge->length =3D cpu_to_le32(len); + sge->type =3D NVME_SGL_FMT_DATA_DESC << 4; +} + static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge, struct blk_dma_iter *iter) { - sge->addr =3D cpu_to_le64(iter->addr); - sge->length =3D cpu_to_le32(iter->len); - sge->type =3D NVME_SGL_FMT_DATA_DESC << 4; + nvme_pci_sgl_set_data(sge, iter->addr, iter->len); } =20 static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge, @@ -1239,6 +1245,142 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_de= sc *sge, sge->type =3D NVME_SGL_FMT_LAST_SEG_DESC << 4; } =20 +static unsigned int nvme_pci_dmabuf_sgl_nents(struct request *req) +{ + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + struct scatterlist *sg; + unsigned long tmp; + size_t offset =3D bio->bi_iter.bi_offset; + size_t remaining =3D blk_rq_payload_bytes(req); + unsigned int nents =3D 0; + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len =3D sg_dma_len(sg); + + if (!remaining) + break; + if (offset >=3D sg_len) { + offset -=3D sg_len; + continue; + } + + sg_len -=3D offset; + offset =3D 0; + + do { + size_t chunk =3D min(remaining, sg_len); + + nents++; + sg_len -=3D chunk; + remaining -=3D chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + return 0; + + return nents; +} + +static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request *req, + struct nvme_queue *nvmeq, unsigned int entries) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct nvme_dmabuf_map *map =3D nvme_req_to_map(req); + size_t length =3D blk_rq_payload_bytes(req); + struct nvme_sgl_desc *sg_list =3D &iod->cmd.common.dptr.sgl; + size_t offset =3D req->bio->bi_iter.bi_offset; + bool pooled =3D entries > 1; + dma_addr_t sgl_dma =3D 0; + unsigned int mapped =3D 0; + unsigned long tmp; + struct scatterlist *sg; + size_t remaining =3D length; + + if (!entries) + return BLK_STS_IOERR; + + iod->cmd.common.flags =3D NVME_CMD_SGL_METABUF; + iod->total_len =3D length; + + /* entries =3D=3D 1 fits in the inline descriptor; more needs a pool. */ + if (pooled) { + if (entries <=3D NVME_SMALL_POOL_SIZE / sizeof(*sg_list)) + iod->flags |=3D IOD_SMALL_DESCRIPTOR; + + sg_list =3D dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &sgl_dma); + if (!sg_list) + return BLK_STS_RESOURCE; + iod->descriptors[iod->nr_descriptors++] =3D sg_list; + } + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len =3D sg_dma_len(sg); + dma_addr_t addr =3D sg_dma_address(sg); + + if (!remaining) + break; + if (offset >=3D sg_len) { + offset -=3D sg_len; + continue; + } + + addr +=3D offset; + sg_len -=3D offset; + offset =3D 0; + + do { + u32 chunk =3D min_t(size_t, remaining, sg_len); + + if (WARN_ON_ONCE(mapped =3D=3D entries)) + goto err_free; + nvme_pci_sgl_set_data(&sg_list[mapped++], addr, chunk); + + addr +=3D chunk; + sg_len -=3D chunk; + remaining -=3D chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + goto err_free; + + if (pooled) + nvme_pci_sgl_set_seg(&iod->cmd.common.dptr.sgl, sgl_dma, + mapped); + return BLK_STS_OK; +err_free: + if (pooled) { + iod->nr_descriptors--; + dma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma); + } + return BLK_STS_IOERR; +} + +static blk_status_t nvme_rq_setup_dmabuf(struct request *req, + struct nvme_queue *nvmeq, enum nvme_use_sgl use_sgl) +{ + unsigned int entries; + size_t avg_seg; + + if (use_sgl =3D=3D SGL_UNSUPPORTED) + return nvme_rq_setup_dmabuf_map(req, nvmeq); + + entries =3D nvme_pci_dmabuf_sgl_nents(req); + + if (use_sgl =3D=3D SGL_FORCED) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + avg_seg =3D entries ? + DIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0; + if (sgl_threshold && avg_seg >=3D sgl_threshold) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + return nvme_rq_setup_dmabuf_map(req, nvmeq); +} + static blk_status_t nvme_pci_setup_data_sgl(struct request *req, struct blk_dma_iter *iter) { @@ -1334,7 +1476,7 @@ static blk_status_t nvme_map_data(struct request *req) blk_status_t ret; =20 if (blk_mq_rq_is_dmabuf(req)) - return nvme_rq_setup_dmabuf_map(req, nvmeq); + return nvme_rq_setup_dmabuf(req, nvmeq, use_sgl); =20 /* * Try to skip the DMA iterator for single segment requests, as that --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 050934FDE60 for ; Wed, 14 Oct 2026 11:02:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975766; cv=none; b=OGNiF7b+16/eUZBO+FctRNaspa/zEcxfUDq9AoNQAp5QMWaxb0TPyZJmMKU7g6eBnV0bjy19xJErtrOoo+fxeDbqzUClXudQRMVkXl4YDYtP99WB0xhG4CMkfoPbicKOpvHOXwi9wKzCQxHGYkSn/KEoLDBkvN1uSp4BxjMOVpk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975766; c=relaxed/simple; bh=VfD6HJzx1Y38GpK6rDkYF3jfCC0ZHJ/tqnfgL8fmZrk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KQ/Otk8g8BduYm9SkWhaVIEXpGXGGonTyvxDyr+sa/qgaNPYkTe0LOb1mXSHg6DrNYMkhK2bWCAX8SB/3/VXSFrGBNfS1ZdEPbtx0J7HKzSzNHPIEf9Pw5/hoqe6jrePfj2oER+ujHSQ0yinI6hasWYcVG5dvYmoF/q9D/XFwDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=E3TXbZ0F; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="E3TXbZ0F" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-4a17baef589so7409935e9.0 for ; Wed, 14 Oct 2026 04:02:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975756; x=1792580556; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XniK4D9I5IxaguHxPaSeNTaZUyaBimf3PNveaq2tpgg=; b=E3TXbZ0FCh/WllUwLQrvmEis4IBA34kDY1oulmfCb5z5QhMLc6rbgbAXbTYOl6o6Am xMiFRrCvYbQe8r/txbAws1RhwPEULtHySTadC7kW54UTrdC08T55wA+Wx1CMS3yXinEZ RmVhIjK4TzvPHRG13lMTkHIuv8y5rDuujvjHeE0yceRaIqJ+Cg7vuDcv7KyVi2OzR06Y pjsmdppc7XLFSUXuJtpo8epflMfy/QFxiFjgtfOAc+yrvbhk376kqbgFY0pesePWlKeK WKJRYBTl/8c4tsSAQarZt25zgS7k7EnERHCq5JQO86bA4oZ6GXNtMOjkzrQ5Gxx8bZlO FVGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975756; x=1792580556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XniK4D9I5IxaguHxPaSeNTaZUyaBimf3PNveaq2tpgg=; b=VzbIViQ3m0rrrkcQ1YOvYpVxVsMtNa1h2QLmNnxkKdswVz++SHMYJPer/PaqzoxPQF q/z7yEK3sKbS6pExGqTh5mgKgmFuMVypHRnvhRIz7L6LUG3pRZfL7qyv6M5D9znUkI8k CscrPlFPck7GfyzFtU0OYQORqSRZpCshVQ2/97fGP5Qi6RETQAJa9YUnG61/2CLOtNDn 3EP7zijp/Ls0VUyto8o5y7ky6Tj43Ou/cGFyN4xnhhlHhhfokIX2dT0c8CGzLOA5z6QU 2xMuReqzVy/sEzC+lujOIKqwmueaYvtx5yFxKy12wj6nCFnDqJCd4xvPZGElF4nVw676 21TA== X-Forwarded-Encrypted: i=1; AKwUvBy6rZs7GnVILDU4n51pf9z8yrigCDMEa+UZX4O0ASiLuHGYfok6R8WhnaRbOYeKvS6PoAGd8Ki5OyzSDwI=@vger.kernel.org X-Gm-Message-State: AFq9FYIwO6mCzenu2Q2PyJM027GzMJzi3VJkebmSSEknyQh+1kOuu0GA 39heQX29EF3vDNG/QON/1EJLzqVO2GKjTS41gOJ0b9cv8+L34k+Vufso X-Gm-Gg: AYBFou2lN9BHWgdtsHYZkzWwbqKXDEwl3lvRoHxL33rJAx2oKPa0FM3GddmazyL4LHV 2qwGYSScDf1UBShFPwtb55hNM0yLSODZCzOdeeIYHdv/0noznfazgpmfD7th/5F3+ZCyt+7wsgd bkvrS6Ja6SRBauf1UnIFH4xTWglxDrbKWv5zjSBqHeBYr+gGc2Nfct2Lbl2lZJct2629XxORsIO LI/8Y3XMcMsRkSapm3+7ujdXjGoJrTa9vO44B4CKI9VIUihxtFcvJTQaC+iEUD7E7LwH/H6o5Pb 6lpKdCoXtOff8C8T5SDNXFjtLCQlnZo7lF9i4tpzwSf6oOW/kMsBLaApX8uCR2eZqeohK7q+ce7 sRkzUMzDSipkhXfT0CKK2j3BG9AgMICt269xRxWb4yGPzx8TYiQcdfe0HyPqAwMMZNmDyHBU+XG 7j9TWTJ1f/bjLoBXSG3dlCCb0xQKuBNtbTiHUINnP2U66Ci5gzye69HpqgsoEVOBtRSqC+fITLb Ae8uEkmgGRMaupyeEmuy6CL2F2VRd8WCjvpYNjEcHnqRuQwXM3LCkwW4ySMuH2YVraoqUDFyCHl X-Received: by 2002:a05:600c:1549:b0:4a0:25c:8af2 with SMTP id 5b1f17b1804b1-4a18e491051mr343632765e9.17.1791975755701; Wed, 14 Oct 2026 04:02:35 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:35 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 09/13] io_uring/rsrc: introduce buf registration structure Date: Wed, 14 Oct 2026 12:02:07 +0100 Message-ID: <178bbcbc7a0a9cf017284e8b15d5434fb546fcaf.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation to following changes, instead of passing an iovec for buffer registration introduce a new structure. It'll be moved to uapi later, but for now it's initialised early from a user provided iovec. Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- io_uring/rsrc.c | 50 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 15 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 1efaf29514e8..7234dcdb8372 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,8 +27,13 @@ struct io_rsrc_update { u32 offset; }; =20 +struct io_uring_regbuf_desc { + __u64 uaddr; + __u64 size; +}; + static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, - struct iovec *iov); + struct io_uring_regbuf_desc *desc); =20 static int hpage_acct_ref(struct io_ring_ctx *ctx, struct page *hpage, bool *acct_new) @@ -81,6 +86,15 @@ static bool hpage_acct_unref(struct io_ring_ctx *ctx, st= ruct page *hpage) =20 #define IO_CACHED_BVECS_SEGS 32 =20 +static void io_iov_to_regbuf_desc(const struct iovec *iov, + struct io_uring_regbuf_desc *desc) +{ + *desc =3D (struct io_uring_regbuf_desc) { + .uaddr =3D (u64)(uintptr_t)iov->iov_base, + .size =3D iov->iov_len, + }; +} + int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -398,6 +412,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 @@ -411,7 +426,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, err =3D -EFAULT; break; } - node =3D io_sqe_buffer_register(ctx, iov); + + io_iov_to_regbuf_desc(iov, &desc); + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { err =3D PTR_ERR(node); break; @@ -870,26 +887,29 @@ bool io_check_coalesce_buffer(struct page **page_arra= y, int nr_pages, } =20 static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, - struct iovec *iov) + struct io_uring_regbuf_desc *desc) { + unsigned long uaddr =3D (unsigned long)desc->uaddr; + size_t size =3D desc->size; struct io_mapped_ubuf *imu =3D NULL; struct page **pages =3D NULL; struct io_rsrc_node *node; unsigned long off; - size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; bool coalesced =3D false; =20 - if (!iov->iov_base) { - if (iov->iov_len) + if (uaddr !=3D desc->uaddr || size !=3D desc->size) + return ERR_PTR(-EOVERFLOW); + + if (!uaddr) { + if (size) return ERR_PTR(-EFAULT); /* remove the buffer without installing a new one */ return NULL; } =20 - ret =3D io_validate_user_buf_range((unsigned long)iov->iov_base, - iov->iov_len); + ret =3D io_validate_user_buf_range(uaddr, size); if (ret) return ERR_PTR(ret); =20 @@ -898,8 +918,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(stru= ct io_ring_ctx *ctx, return ERR_PTR(-ENOMEM); =20 ret =3D -ENOMEM; - pages =3D io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, - &nr_pages); + pages =3D io_pin_pages(uaddr, size, &nr_pages); if (IS_ERR(pages)) { ret =3D PTR_ERR(pages); pages =3D NULL; @@ -921,10 +940,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, if (ret) goto done; =20 - size =3D iov->iov_len; /* store original address for later verification */ - imu->ubuf =3D (unsigned long) iov->iov_base; - imu->len =3D iov->iov_len; + imu->ubuf =3D uaddr; + imu->len =3D size; imu->folio_shift =3D PAGE_SHIFT; imu->release =3D io_release_ubuf; imu->priv =3D imu; @@ -934,7 +952,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(stru= ct io_ring_ctx *ctx, imu->folio_shift =3D data.folio_shift; refcount_set(&imu->refs, 1); =20 - off =3D (unsigned long)iov->iov_base & ~PAGE_MASK; + off =3D uaddr & ~PAGE_MASK; if (coalesced) off +=3D data.first_folio_page_idx << PAGE_SHIFT; =20 @@ -986,6 +1004,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, v= oid __user *arg, memset(iov, 0, sizeof(*iov)); =20 for (i =3D 0; i < nr_args; i++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 @@ -1009,7 +1028,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, = void __user *arg, } } =20 - node =3D io_sqe_buffer_register(ctx, iov); + io_iov_to_regbuf_desc(iov, &desc); + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { ret =3D PTR_ERR(node); break; --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC06C4FDE7C for ; Wed, 14 Oct 2026 11:02:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975768; cv=none; b=oJzfUzeVK8MJUi3rOicpjfbvviQPFgi/Lz5znejLgPri3HA9+vvBpcrVtt9b32+jZYehDpALjh65KlZIOsC1Ov3WF9ZFcHp2aukPpTZIu8y63t+sPJlxSjKIT2xD2FosrHBSj9O1Nmk+Cz+xS0KdZCG1+geQLV7Z74ram9ZtooU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975768; c=relaxed/simple; bh=bXbAIgu8XiflD4WoySdMtpdgz0ClWZkdJULVv1/YSpQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=glSSN+wlaARFma1snhO3HjpR8rCnMY2pSK+aSH++dhkDs5/n+EBivUcrEVztopurnmzlolVoMeQRZCZYb7ewpehihKBJNPefvjFTipd66ZexCnj+7ptyN5x84PKdRILGn8qI7O1ZlzyXEi/4piuGmSIr99O0nw3Dse/Y0U9KalI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=S3s9X1X9; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="S3s9X1X9" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4a0213948d3so24332495e9.2 for ; Wed, 14 Oct 2026 04:02:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975757; x=1792580557; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jun/zHGshv7Ie+bKLV3PJNKMmcdIRnJjgz+qOkaWbDg=; b=S3s9X1X9ZyM1k3WQK4h8OuQCT0lsPxlnppeiT1DeKYkK/djtH+0WnsPFa+SkeIf4C7 uFvQ+07P2PDt9PecHHfAoeTLFSR8aLrbBkEbDOrhfVn9PnrfMznQWTLSHjApVj7eUqJx cvV61xPG+qHamx4Zo5y/I75U7zDwjn0RnyB9XpRylgXX9gjw/cCxK0kt6EPhJzYpRbXT 2HxRcvw1vY5ITgvge3KPMAWPdXELxdarEG10HabCCCmuowo80rWx1Yw5cT5b9cUasVNX m+j+Q14RwwUc9dYG6xeQr845C4qSI3n1memHUyQ53/XVYjddHaogSn6roXt1U/ByaCF4 WHHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975757; x=1792580557; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=jun/zHGshv7Ie+bKLV3PJNKMmcdIRnJjgz+qOkaWbDg=; b=Q4aCOvQLQmX2ZQrkOiDyXK3VWwCQGVfjnYfqASomlmQGyWfg0ZTVxFo/Gb1yT0Roht Xieyauj/oV2m+CMBZWNbkQtMeNUS6kW9iDPJEZO+GCJVahqpqlk1dVpwsspm0bDLj4+/ ise3T9UGARU+jpEWxYZ0zalrQk2+4ICKWc09W4w3eKgt5kYUZ28oyVkOqUyYSJH6LLvv shipa18UwiCCgFA1rTzinL3SYg5lQvL+YRDtgyJMyUkktsJM66bXKjb7rXs5u9OQ53uZ nL0AVEXvmow95FzGwQHEr5ZXxznJ3T7NEgqDc6vcAg620F50xh1wjaNSINC/mWYdeTz5 g46w== X-Forwarded-Encrypted: i=1; AKwUvBy2mzxdxzRfSG6KDY05Ehz7+HMnuj+/fim0FEfUjvelDHZ51SHOsQeVV0EQAX0Th8dNmevp2VQX5JUJhjM=@vger.kernel.org X-Gm-Message-State: AFq9FYLROt98IK2mgsnywRa6dk4QggWO2pd/fwrRMNr1kchbZGMZlkYL VDFhZerHyWpqxiiDtOIfhw37w4BHxOtX1u42rjZTFIAD3sh9r13Ej9ON X-Gm-Gg: AYBFou31YXu4KKTllap2TbRRcyRR4eGOWPzj75pLR6X+ybQi9ThZ+EymV8agHhbAsSK ZhHg0EX3SRZyzmEzpBy5So+wHlTbJX9vfhe6YIg+Fi4EtIk2PosezaYeiMY5M1F/OeeqWzTgAlM lnfNmIZ/fJ8DBP8sUkiVNMd3OpjE1/tRgLUMPSJ4kpkL5idDmIcdArsfdUK+fxAuZaAvv+V+xp2 sZym1FI/19JSXZSf/ngOTxjcFuPmrwB2SSTE6vP05AwMHWOXfl/n/20oYRXXxzWXNDbXC4UKiJ/ 9eokBOzEpXy02+9p7A1s+OERJuE5U2elwQZ9WNrSzx01kRPqgppLPeCidU7R+E+hhv8GizAk75I QR/iVp+4YF0yhJEK7hW4FB90jemiiiTNSaMp0DI4clU9ox78Zq/mvMLaJpYmT9OAjNNjEy2bHhI igAUWpRAzETwk/++1WJyrAVK+d/oX09BARfrMt7tQpVtDk3a7+qDt+04kzIf824Sq7I/ktrxniA SIzaiPDNHYLV6yXhAzzNlWhlir12F6/EiyFfMoBO/JVPwV7Ik6U83GXQB6qhCFwFalC2PZGXNwd X-Received: by 2002:a05:600d:8488:10b0:4a1:9078:911 with SMTP id 5b1f17b1804b1-4a190780997mr206827605e9.31.1791975757484; Wed, 14 Oct 2026 04:02:37 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:36 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 10/13] io_uring/rsrc: extend buffer update Date: Wed, 14 Oct 2026 12:02:08 +0100 Message-ID: <0d60f1ebab80ef594a78ea27c2edd5ad33faa640.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We need to pass more information to buffer registration than we can fit into a single struct iovec. This patch allows users to optionally pass struct io_uring_regbuf_desc. Apart from having more space for future use cases, it also introduces registration types. Currently, the type can be either of IO_REGBUF_TYPE_UADDR, which mirrors the iovec path, or IO_REGBUF_TYPE_EMPTY for leaving a buffer table slot empty. The next patch introduces a dmabuf backed type, and can be useful for other extensions like splicing a list of user addresses (i.e. iovec[]), interoperability with zcrx, kernel allocated memory like was brough up by Cristoph. Note, the type only represents a registration option, which is distinct from how io_uring internally stores it. The flags field is not used yet but always useful to have, e.g. we can encode read-only / write-only restrictions using it. Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- include/uapi/linux/io_uring.h | 27 +++++++++++++- io_uring/rsrc.c | 68 +++++++++++++++++++++-------------- 2 files changed, 68 insertions(+), 27 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index b42b66b424a7..abb590833a0f 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -791,13 +791,38 @@ struct io_uring_rsrc_update { =20 struct io_uring_rsrc_update2 { __u32 offset; - __u32 resv; + __u32 flags; __aligned_u64 data; __aligned_u64 tags; __u32 nr; __u32 resv2; }; =20 +/* struct io_uring_rsrc_update2::flags */ +enum io_uring_rsrc_reg_flags { + /* + * Use the extended descriptor format for buffer updates, + * see struct io_uring_regbuf_desc + */ + IORING_RSRC_UPDATE_EXTENDED =3D 1U << 1, +}; + +/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */ +enum io_uring_regbuf_type { + IO_REGBUF_TYPE_EMPTY, + IO_REGBUF_TYPE_UADDR, + + __IO_REGBUF_TYPE_MAX, +}; + +struct io_uring_regbuf_desc { + __u32 type; /* enum io_uring_regbuf_type */ + __u32 flags; + __u64 size; + __u64 uaddr; + __u64 __resv[7]; +}; + /* Skip updating fd indexes set to this value in the fd table */ #define IORING_REGISTER_FILES_SKIP (-2) =20 diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7234dcdb8372..54774729ee67 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,11 +27,6 @@ struct io_rsrc_update { u32 offset; }; =20 -struct io_uring_regbuf_desc { - __u64 uaddr; - __u64 size; -}; - static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_uring_regbuf_desc *desc); =20 @@ -90,9 +85,12 @@ static void io_iov_to_regbuf_desc(const struct iovec *io= v, struct io_uring_regbuf_desc *desc) { *desc =3D (struct io_uring_regbuf_desc) { + .type =3D IO_REGBUF_TYPE_UADDR, .uaddr =3D (u64)(uintptr_t)iov->iov_base, .size =3D iov->iov_len, }; + if (!desc->uaddr) + desc->type =3D IO_REGBUF_TYPE_EMPTY; } =20 int __io_account_mem(struct user_struct *user, unsigned long nr_pages) @@ -340,6 +338,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ct= x, return -ENXIO; if (up->offset + nr_args > ctx->file_table.data.nr) return -EINVAL; + if (up->flags) + return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { u64 tag =3D 0; @@ -399,9 +399,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, struct io_uring_rsrc_update2 *up, unsigned int nr_args) { + bool extended =3D up->flags & IORING_RSRC_UPDATE_EXTENDED; u64 __user *tags =3D u64_to_user_ptr(up->tags); - struct iovec fast_iov, *iov; - struct iovec __user *uvec; u64 user_data =3D up->data; __u32 done; int i, err; @@ -410,29 +409,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx= *ctx, return -ENXIO; if (up->offset + nr_args > ctx->buf_table.nr) return -EINVAL; + if (up->flags & ~IORING_RSRC_UPDATE_EXTENDED) + return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 - uvec =3D u64_to_user_ptr(user_data); - iov =3D iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); - if (IS_ERR(iov)) { - err =3D PTR_ERR(iov); - break; - } if (tags && copy_from_user(&tag, &tags[done], sizeof(tag))) { err =3D -EFAULT; break; } =20 - io_iov_to_regbuf_desc(iov, &desc); + if (extended) { + if (copy_from_user(&desc, u64_to_user_ptr(user_data), + sizeof(desc))) { + err =3D -EFAULT; + break; + } + user_data +=3D sizeof(desc); + } else { + struct iovec __user *uvec =3D u64_to_user_ptr(user_data); + struct iovec fast_iov, *iov; + + if (io_is_compat(ctx)) + user_data +=3D sizeof(struct compat_iovec); + else + user_data +=3D sizeof(struct iovec); + + iov =3D iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); + if (IS_ERR(iov)) { + err =3D PTR_ERR(iov); + break; + } + io_iov_to_regbuf_desc(iov, &desc); + } + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { err =3D PTR_ERR(node); break; } + if (tag) { if (!node) { err =3D -EINVAL; @@ -443,10 +462,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx = *ctx, i =3D array_index_nospec(up->offset + done, ctx->buf_table.nr); io_reset_rsrc_node(ctx, &ctx->buf_table, i); ctx->buf_table.nodes[i] =3D node; - if (io_is_compat(ctx)) - user_data +=3D sizeof(struct compat_iovec); - else - user_data +=3D sizeof(struct iovec); } return done ? done : err; } @@ -481,7 +496,7 @@ int io_register_files_update(struct io_ring_ctx *ctx, v= oid __user *arg, memset(&up, 0, sizeof(up)); if (copy_from_user(&up, arg, sizeof(struct io_uring_rsrc_update))) return -EFAULT; - if (up.resv || up.resv2) + if (up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, IORING_RSRC_FILE, &up, nr_args); } @@ -495,7 +510,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, vo= id __user *arg, return -EINVAL; if (copy_from_user(&up, arg, sizeof(up))) return -EFAULT; - if (!up.nr || up.resv || up.resv2) + if (!up.nr || up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, type, &up, up.nr); } @@ -595,12 +610,9 @@ int io_files_update(struct io_kiocb *req, unsigned int= issue_flags) struct io_uring_rsrc_update2 up2; int ret; =20 + memset(&up2, 0, sizeof(up2)); up2.offset =3D up->offset; up2.data =3D up->arg; - up2.nr =3D 0; - up2.tags =3D 0; - up2.resv =3D 0; - up2.resv2 =3D 0; =20 if (up->offset =3D=3D IORING_FILE_INDEX_ALLOC) { ret =3D io_files_update_with_index_alloc(req, issue_flags); @@ -901,9 +913,13 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, =20 if (uaddr !=3D desc->uaddr || size !=3D desc->size) return ERR_PTR(-EOVERFLOW); + if (desc->type >=3D __IO_REGBUF_TYPE_MAX) + return ERR_PTR(-EINVAL); + if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) || desc->flags) + return ERR_PTR(-EINVAL); =20 - if (!uaddr) { - if (size) + if (desc->type =3D=3D IO_REGBUF_TYPE_EMPTY) { + if (uaddr || size) return ERR_PTR(-EFAULT); /* remove the buffer without installing a new one */ return NULL; --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F0BA4FECCF for ; Wed, 14 Oct 2026 11:02:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975770; cv=none; b=ZZIXlkdzsui4JbSI9+9uHk4txDdv/EXnZ8A37e9rA5JRbleFbNSUaEVHd6sCgsafUeM1TDU5dFrQOPAEwP4jUq+Oj35r3xSTl4oUj7nuDGMdPtIEK+TNUHi8cdo/IYmnXB3m5ZJyS8c3A9DueO6X9znrNKIkBUdaMf3TaAV8UCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975770; c=relaxed/simple; bh=Yz9mg10bqVdj7cVQ9CUCuhVSPCWk27GUE+7ZZMs5kDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aZ9elFeRWu+5topUNG+Bx2tLbqwkxNpsa6zXOXceEncoa/ZZeXZj/XAGDxzWJkwZPhl5BurV7V+cEC7GeCA78O9hsSFymMqkWplYE7lYkXqLJxCBbfFiWq/NYHfMm27eyoAHU5LHciud7OELUd847pg9gehEKWveNwnU2XbzeB4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=P69uopP0; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="P69uopP0" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-49d05d51553so5096255e9.2 for ; Wed, 14 Oct 2026 04:02:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975759; x=1792580559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9CL2BRiyUp3CaHGocQHkhsGpx+V6MY3r8MUNMnInA+U=; b=P69uopP0dGw4EOaBNSimaiFj01eeZtTBN1IzIjPctYGXGMK186M0vk1uak1vfbhoQC Z4f+zsvI9w8AQB+S/B67WktuHbeO4aifFH3/USoHAC1vXOCw3R8LPpeyZMpRONe+jiPB O4HTkJ+LZNMzsFHzYsEh+Yl0LWLLuseBNmsmShyR6tR4yuUQ/jFIpRGSIizgydVD58ZA 4Rj1t0FIjWI7vqAXOufrWsQuWJOF0+50IM8pry+1R8j2WxScMO82jJuux0xQmRW5+Guf JgxxxEYU9mcdVtgbr3+oQrqjr8IBZB1JkgUMoI7oPP25Edx2aEfEfGvRoN4cSV2XO+t5 PHjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975759; x=1792580559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=9CL2BRiyUp3CaHGocQHkhsGpx+V6MY3r8MUNMnInA+U=; b=Nf+3dUDv5B78Abl6Oq0XzuECjPrFvgWmelJDHdps/hvgbivSW1rjSY9ToBA8WwelSn yyVXYQ04/6sRVIRGGAkGfaQepZTb6weB6e82REEbnQxYPWtoM64ekzW/3s34ReUeNzeH rq+zi3FI6Zz+bcVHUZsi//TFR6pd36y3ypCrRx1505K4iSPToGgyKYML6jgOr9Iq+wb0 lIuKd3Ny+qB74LnzpxSBor3X214CjNcAZpmD5zm8urDPtB+CkIW+0VZ4LIhO5Iri1T4O id94hHzxjyvI4EiVfS1RKwuM8IVAFP3ej9HSWYHfJMevX2nqp92M5TG+bfsXB1s0qBfu MQew== X-Forwarded-Encrypted: i=1; AKwUvBxMNXXlzC7ZeBocs53QRVjrB+vSb5/PBmLWOEU7nNopd5ABUpSgub4Bdcto56icgfzflsiijEGVvELBJT8=@vger.kernel.org X-Gm-Message-State: AFq9FYIufTp4K73dVqSVxvlj6ydfhIP8k47IkG8Ccph8sh5zxy43+MCe CVseLOlgYD8IWyC3KjKWfcV5L88Ya1hbIlLGLyobyQdSr0NoAyzIZdA5 X-Gm-Gg: AYBFou0Wfc9g+dHq06xnGnxLXx13/M2+m1Ib52bs0OhGuOwOe5JY+XYIIoqATnYgY0u jbPjlrfgmXSjcgdiRvmZuNXkhcQdPOLXiFC6Z67+4XTYN+LB10QCHN7SejyIggMXYOWkG+fUKUr USBVL2nMc/CbZgukb5KSU99/RGOOz9cIlt7HWakFPHPEBSAZbI2JL4RfjjeNaQLytwWc19OZist nOBhn3xZmx+Md2ZdjKJUqpu/82N2pTyyCl7h0RNIg9P4KCAFsayigtMRnqub67LUsFmDzHdUpCc eweTvLHEEgZBi1dJaNVvk46SBoo/K05lSnBq6QmyEijtNqyRXdYWGDvMeAyR9Vo5XkVNS5OFwlL m3tlslnk/UH4CyFCwSP27VNvyHq7b5oAq3mZZo/QSHnQeFFnxfglPqlKM7lkAOz/pZieczMES9d 2pPCS/drQ0/tk1jp8eL4ciWHltY5W9auEx0zhGGNttsNLsSky+6ha8ixgqUznyReQT2JnVqy4U7 TxPsDXLdzG4+ep4DH/Q4Ty6pOwaqw/0YIppIgOivfeDO7RFlDfz35tmRZLNOvMYdZAb5Lj9YILf X-Received: by 2002:a05:600c:4747:b0:4a1:77db:45d9 with SMTP id 5b1f17b1804b1-4a18e47aa9amr321452385e9.6.1791975758979; Wed, 14 Oct 2026 04:02:38 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:38 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 11/13] io_uring/rsrc: add uncloneable regbuf flag Date: Wed, 14 Oct 2026 12:02:09 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It's hard to implement cloning if the internal structure needs to be mutable and/or relies on other ring resources. In preparation to such buffer types, add a flag indicating that the buffer can't be cloned. It might be possible to add cloning in the future for them, but that would likely need reallocating the structure and reacquiring resources in case by case manner. Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- io_uring/rsrc.c | 5 +++++ io_uring/rsrc.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 54774729ee67..532866a1e9b9 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1458,6 +1458,11 @@ static int io_clone_buffers(struct io_ring_ctx *ctx,= struct io_ring_ctx *src_ctx if (!src_node) { dst_node =3D NULL; } else { + if (src_node->buf->flags & IO_REGBUF_F_UNCLONEABLE) { + io_rsrc_data_free(ctx, &data); + return -EOPNOTSUPP; + } + dst_node =3D io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); if (!dst_node) { io_rsrc_data_free(ctx, &data); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 9ef88383b363..73d7b1ebf0b6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -26,7 +26,8 @@ struct io_rsrc_node { }; =20 enum { - IO_REGBUF_F_KBUF =3D 1, + IO_REGBUF_F_KBUF =3D 1 << 0, + IO_REGBUF_F_UNCLONEABLE =3D 1 << 1, }; =20 struct io_mapped_ubuf { --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50690502563 for ; Wed, 14 Oct 2026 11:02:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975773; cv=none; b=kUcyetyqTyofWrxR+7DkrJJUK+kmI+elksK61YYJYUZFmT6X2dpjMKcSXxByN6S5vatu5CykNLyQT0EHCXAEfjl74GK7O4dfEyje9/OQP1iXQwwPBsoT/v9/08YbHIgnjoUhJv3/E5xq7eSt8Ma1prkub2KtTHiqToGW5woQ5Ig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975773; c=relaxed/simple; bh=r60ha7f86tSXOy7awDRSeceZcMhT0AOlgJWaxT3jtGs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=S+IUUnoUXxX1ee7oCpM5T1Efx5WeB09n7ueoZRe7+JF9f10Seig0toUn0hXPBruCVDMUtCpEeRkWU/NMM5bjk1+F7aPofb8FWyYrjUrgbhoy7cnpsX5k4YF6R2vFR3mBguwPdJsvp61q7678+Ndr0XMeCkTH7UDkVARN9r/CvVM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ec2J/9XW; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ec2J/9XW" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-4a16af2a232so31350705e9.1 for ; Wed, 14 Oct 2026 04:02:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975762; x=1792580562; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ays4SVba18pfRPB/Hpqu6QDaxGmpKJaoA/2lD097Khk=; b=ec2J/9XWQns9tj6FMC0xey6hE+ChJX7QMWPnnNC7ck9YAiDfM4dCYfGC6PGN7ASton nGvx8FSUjPxR0wkhYfPj0/Dsbdedi0x0PmmFkimO9rEEA3FoIph8o+wg3EGhp1ImbFz4 Bhlqj6cUD9Ra0r+nlzXuM0ofwOuKMVJMycpHCtyNDKF6HPylhF8jl9c812YN1970grG8 REu40iu1vOL7XzqQto+JHM1Xc6laDsEY0vfM/QX3uELyezwGFtN0e3Vg8aoagLyEsVVj TOpRkqKEB0cisgogZoA77w2SUi8berzjTITZmgHXbkVsEZRWfLXIhkjjJ1rRkbR8qMWv oipg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975762; x=1792580562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ays4SVba18pfRPB/Hpqu6QDaxGmpKJaoA/2lD097Khk=; b=Q5mvZzS6uCeENmp8S2Tyn74c3BFNg55X36fJSDBE9nukFURlRwCUu1iBvPWF6pE258 OKeKUzPyp8LZa9oL3UTpz6HlboMf/6rQNu3AM00P0ZZ2H/JWpdrJaPR3IiUvkeJnVyH0 U8zHEY5Cjb3K5UhWKo2YvEA7rvUR6cjoFG5i8uWNhcPC/CPlWz6q3wiOtXiRYucjE8hK /Sb07sapjCK4nwGPFEiGuDtPqM/0ZqoVjBcgbzOr1weQpsSzbf1b50qCF2gM6P97X0jF hUjKHmgdVg7wj7SrJQ5mxlWhMkXT4tykddcV6QdiDcIGKAzjkb4mpZmXSt8YWHa1dLjB MjeA== X-Forwarded-Encrypted: i=1; AKwUvBzyHMSKcE/hHfn6bP1A63RyyQn4EmuCHrVd2TeTWZGIKU8aIgl9LKgXlx8UNaoJdYXou5Hct1guUqXFzXg=@vger.kernel.org X-Gm-Message-State: AFq9FYIMMaE9yBO36SodqJaeqEZoEkxvz7sE77ZQn5YPfvqE3t6fFgIT HzT2IBhUIYhsjZgYeJqY64g+NLhveLyRJc3Fk46DMPIqtVii05Hv4wtH X-Gm-Gg: AYBFou26NZcr1ftj2erLKCnWG0d4DtOBTBRvhyWx4ClIbw/3RGv6Hn9gTF0Dei6RF/p ymn1IgPH2uOHSo7iP694wZuDATdJ4PUchNUlTcAzd98umsX7Zp6VOEgN9qwa0bxYKbHH2ntNKLS /3LaikYNStw5YmMuHw/z+uaRPku2uGCblNhbKvSvPhuyRLf6Ul2kJa9r9/iOhgZDtruWyead8XW m1JY0r7QjAySkKpapwdIQSnV8+8YT9lHP/Iq3bTAYxD5U/zxoqc7LjbQ/O81mHg7FJG4eRhHtm9 2F6q1rEpgeo/EH9hvvZedlLZXwvZXMeVtNhqeLumAGk3WEVHdruTjuw4FGSAVNdHtBT/LNNvUVE nWvshm2sF1jONahfqHoV/pXEGBBRVWUb7X8Z5lOMbjqHD7i1Qj2/I87n7tbeLYg8uQ+xehERJsm dCE79daevLqyYJTazYnbuEX1mi/5INjQFybpqlpRbCIy1pGsh21OPX6y8VSOBZIMwW3GYQ0gNAo hV68AHlYsyyc3jCK/SSI1LjCUzXxnaCM3cLGmUHdtisaHkOgDr6ib9VEQGWGTP6CMGsrimUNHJw X-Received: by 2002:a05:600c:154c:b0:49f:ce78:3562 with SMTP id 5b1f17b1804b1-4a18e4a9159mr329602185e9.19.1791975762395; Wed, 14 Oct 2026 04:02:42 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:39 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 12/13] io_uring/rsrc: add regbuf import flags Date: Wed, 14 Oct 2026 12:02:10 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We'll have special registered buffer types that can't be used with all opcodes and need special handling. Add separate flags to control registered buffer import, which will be used to specify what kind of buffers the caller can handle. Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- io_uring/rsrc.c | 8 ++++---- io_uring/rsrc.h | 24 ++++++++++++++++++++---- 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 532866a1e9b9..f42d201738cd 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1317,9 +1317,9 @@ inline struct io_rsrc_node *io_find_buf_node(struct i= o_kiocb *req, return NULL; } =20 -int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, +int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags) + unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node *node; =20 @@ -1733,9 +1733,9 @@ static int io_kern_bvec_size(struct iovec *iov, unsig= ned nr_iovs, return 0; } =20 -int io_import_reg_vec(int ddir, struct iov_iter *iter, +int __io_import_reg_vec(int ddir, struct iov_iter *iter, struct io_kiocb *req, struct iou_vec *vec, - unsigned nr_iovs, unsigned issue_flags) + unsigned nr_iovs, unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node *node; struct io_mapped_ubuf *imu; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 73d7b1ebf0b6..351995b61e68 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -62,12 +62,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsig= ned nr); =20 struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, unsigned issue_flags); -int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, +int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags); -int io_import_reg_vec(int ddir, struct iov_iter *iter, + unsigned issue_flags, unsigned import_flags); +int __io_import_reg_vec(int ddir, struct iov_iter *iter, struct io_kiocb *req, struct iou_vec *vec, - unsigned nr_iovs, unsigned issue_flags); + unsigned nr_iovs, unsigned issue_flags, + unsigned import_flags); + +static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter = *iter, + u64 buf_addr, size_t len, int ddir, + unsigned issue_flags) +{ + return __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0= ); +} + +static inline int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned issue_flags) +{ + return __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0); +} + int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv, const struct iovec __user *uvec, size_t uvec_segs); =20 --=20 2.54.0 From nobody Sat Oct 24 13:19:47 2026 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43942509F10 for ; Wed, 14 Oct 2026 11:02:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975778; cv=none; b=SnmA3eX50/HN+E2w1IJRIsVfVipvvRc3RCKBY7WoxJ1c1JTMPd3LrHPmSuFbwgpeX78zCJl+UlNyg2uJvjwcl471xTDy1hgz/a4uz53/bLHvxyUr/30IJ7OgsVC10Wz6p223yBFP3tZfCl53UnMF+lXdPJM4Si91hzgsQ+9rCSs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791975778; c=relaxed/simple; bh=BJdHB/ejV+CxAqyvc96eqP9MAroM09p6ojnCtOYYwsA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Sw2J64yRmw19ROTmA8ALLMlYpYKo3md/wROUurUQIvAR7760A7yJVm2N7e/ykmho9hUnfjT8bwF0PNdIO3pki4pUMRiiHi88wnTstTt8f8u7tN2EQmcK734Z1YXCKUV9wyoOk6GuhREzcfeEy8hX7ttp2l2GfSNuf1ObcDkx5qM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LXztwm+E; arc=none smtp.client-ip=209.85.221.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LXztwm+E" Received: by mail-wr1-f42.google.com with SMTP id ffacd0b85a97d-486e70f2457so165108f8f.2 for ; Wed, 14 Oct 2026 04:02:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1791975765; x=1792580565; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Az43PS/qS00pmQNyOpOlhDk/xvxDJzb8VYH9IXe9HrU=; b=LXztwm+EV5bzwbHwCnOFt/9wdvnsQzYoL9wea7A1yAoa9jyT/7JKdYHBYKI3Yi9JK8 hPIlem5d/KFlq/TG2e+tuD2kA5qI1r3zB2oErHZ2UNuFLdQ1ci1tN6mghSnRs1cVMVLv wJUpm3zZjegIDOIulYSU1A7Sy/oZ3kVGo01l4AQytnuSbiJJDJ+qAOBcXhAJdzi4EKCb cGMDZJSSA7UiRp7LWuIYbIfrf8WUnDucK4iOiF24wCWy8C/j9j6MG10SVUMLVCit0lTe Urwvt1R82WFSBGmEtIoDg7Tlpgk9svV6fQ4USrMwjT1rv6r6lFWN90233ucJXA1hLkM5 b+tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1791975765; x=1792580565; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Az43PS/qS00pmQNyOpOlhDk/xvxDJzb8VYH9IXe9HrU=; b=Ez+uesNlix49K5wZ6EL/YqNp8KAqfM7Qcv4sucxGYNg2Xu02GRlJ+j6YhiMx9ifv+i 5A+7suXNLenmLD4Cga1t0tWHV2psvyEhgRMKFnr/o2OSvFIdyHdd1cTN+tdVJmXaEL/e VAcfSl/ZsqYiJX0J4eRyuB19epc3FPkmVCYpOS7WiOFmtORFer0X6k4skuXGkDz9B4WK +1erTpqoTOCkFh1TZ5L1uFi6PI4W9IJsOov3KgsTvWmPK2dp+OgTu/cK9uYWN3iLILjz 0JYHPGESXj+maPO+s9LzDPuvIdRBTocytILQ01S2zOGkwtRQMPt2V5s2KOzwiYEviiSp z0WQ== X-Forwarded-Encrypted: i=1; AKwUvBw6QQllLotpjmM3RJ5Z2xqP5QvggKGzHklhE4nuZ5xkaromOMMS7rSqkKDUp9qZOI50Nv/ficOAa6BkD2Q=@vger.kernel.org X-Gm-Message-State: AFq9FYI+39PzK6ceUxxjMEuZFEbs6jbIhRWOsEWwiD/OMoZz12cBGEOe 9EtExUXenqT928LeV+KpImUr4oqKIbSpLlFQmLXfXavq3e8mrUl70HUx X-Gm-Gg: AYBFou1pNY/AeJGuUe5jsPkmI9lU7sZHsZl7foVugD1hgcIVV4ttMGSex6g091Za6o4 rOe9pDfwr57D7Q8xXW2tzyOEC7nO515uJj28vDMeU28+Meh8HZnfgQk/ynFlKUTppUm4jHZ9FZj JKIS+Cc5/SYPuq0ImrBse1uAEVDUkQPUwxThFuMH42QVV8E10EyAtnTVlyMI6UYstoDu1V8sH2V rFFTSa63E+WmTSQdqHRWpk9L0FpK9D+zxfQNU2hdebm5IvwtiIK2TU34PeN8tyw/FUfq9ziao8a a5jaVBI2JrD0EUf0gBtjLdiQc2sSRf3MxjH/hQ8QMf3Srfp4ghck85KWkBhQFHOHNUwN60ky3GV ASpcQeU0o2UEEDCfNEBnjqC7QX4zDs7mM7pimY4Agwv+5tjAs8YjHEllImqqdp1N/zvE1JskodJ 8hwWaXOcXdRsqLT03rizdiET+Mos5S1fnge4hMWyGoGmLlQqbM16smCqs/9i/UzU1ja41OzkZuM DxPIzLIoC8NwHcz90PEZLvM/kZMQ0l51Dq6nLr/SpgKVRaGve22izq8jzmYqfTGeQQFc2Uq7SED M92JZSwgUUQ= X-Received: by 2002:a05:600c:e547:20b0:4a1:946c:953d with SMTP id 5b1f17b1804b1-4a1946c968cmr174343455e9.27.1791975765052; Wed, 14 Oct 2026 04:02:45 -0700 (PDT) Received: from 127.mynet ([2a01:4b00:bd21:4f00:7cc6:d3ca:494:116c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4a1b45eb78csm38078845e9.2.2026.10.14.04.02.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Oct 2026 04:02:44 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v11 13/13] io_uring/rsrc: add dmabuf backed registered buffers Date: Wed, 14 Oct 2026 12:02:11 +0100 Message-ID: <4aa230df871ee275536f367087b44ebfebb88cc1.1791975048.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement dmabuf backed registered buffers. To register them, the user should specify IO_REGBUF_TYPE_DMABUF for the regitration and pass the desired dmabuf fd and a file for which it should be registered. From there, it can be used with io_uring read/write requests IORING_OP_{READ,WRITE}_FIXED) as normal. The requests should be issued against the file specified during registration, and otherwise they'll be failed. The user should also be prepared to handle spurious -EAGAIN by reissuing the request. Internally, dmabuf registered buffers is an optin feature for io_uring request opcodes and they should pass a special flag on import to use it. Suggested-by: David Wei Suggested-by: Vishal Verma Suggested-by: Tushar Gohad Signed-off-by: Pavel Begunkov Reviewed-by tag, but I tested it on NVMe with both an AMD GPU dma-buf Tested-by: Anuj Gupta --- include/linux/io_uring_types.h | 5 +++ include/uapi/linux/io_uring.h | 6 ++- io_uring/Makefile | 1 + io_uring/dma-buf.c | 66 +++++++++++++++++++++++++++++ io_uring/dma-buf.h | 76 ++++++++++++++++++++++++++++++++++ io_uring/rsrc.c | 59 ++++++++++++++++++++++++-- io_uring/rsrc.h | 5 +++ io_uring/rw.c | 33 +++++++++++---- 8 files changed, 237 insertions(+), 14 deletions(-) create mode 100644 io_uring/dma-buf.c create mode 100644 io_uring/dma-buf.h diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 90fea94ad202..73cd11d173df 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -11,6 +11,7 @@ =20 struct iou_loop_params; struct io_uring_bpf_ops; +struct dma_buf_io_map; =20 enum { /* @@ -610,6 +611,7 @@ enum { REQ_F_IMPORT_BUFFER_BIT, REQ_F_SQE_COPIED_BIT, REQ_F_IOPOLL_BIT, + REQ_F_DMABUF_BIT, =20 /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -705,6 +707,8 @@ enum { REQ_F_SQE_COPIED =3D IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT), /* request must be iopolled to completion (set in ->issue()) */ REQ_F_IOPOLL =3D IO_REQ_FLAG(REQ_F_IOPOLL_BIT), + /* there is a dma map attached to request that needs to be dropped */ + REQ_F_DMABUF =3D IO_REQ_FLAG(REQ_F_DMABUF_BIT), }; =20 struct io_tw_req { @@ -827,6 +831,7 @@ struct io_kiocb { /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; struct io_wq_work work; + struct dma_buf_io_map *dmabuf_map; =20 struct io_big_cqe { u64 extra1; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index abb590833a0f..b456f45fa8de 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -811,6 +811,7 @@ enum io_uring_rsrc_reg_flags { enum io_uring_regbuf_type { IO_REGBUF_TYPE_EMPTY, IO_REGBUF_TYPE_UADDR, + IO_REGBUF_TYPE_DMABUF, =20 __IO_REGBUF_TYPE_MAX, }; @@ -820,7 +821,10 @@ struct io_uring_regbuf_desc { __u32 flags; __u64 size; __u64 uaddr; - __u64 __resv[7]; + + __s32 dmabuf_fd; + __s32 target_fd; + __u64 __resv[6]; }; =20 /* Skip updating fd indexes set to this value in the fd table */ diff --git a/io_uring/Makefile b/io_uring/Makefile index c54e328d1410..965bb60e9f7b 100644 --- a/io_uring/Makefile +++ b/io_uring/Makefile @@ -26,3 +26,4 @@ obj-$(CONFIG_PROC_FS) +=3D fdinfo.o obj-$(CONFIG_IO_URING_MOCK_FILE) +=3D mock_file.o obj-$(CONFIG_IO_URING_BPF) +=3D bpf_filter.o obj-$(CONFIG_IO_URING_BPF_OPS) +=3D bpf-ops.o +obj-$(CONFIG_DMA_SHARED_BUFFER) +=3D dma-buf.o diff --git a/io_uring/dma-buf.c b/io_uring/dma-buf.c new file mode 100644 index 000000000000..1a1a88640823 --- /dev/null +++ b/io_uring/dma-buf.c @@ -0,0 +1,66 @@ +#include "dma-buf.h" + +static inline void io_release_dmabuf(void *priv) +{ + struct io_buf_dma *bd =3D priv; + + fput(bd->target_file); + dma_buf_io_ctx_release(bd->ctx); + kfree(bd); +} + +int __io_register_dmabuf(struct io_ring_ctx *ctx, + struct io_uring_regbuf_desc *desc, + struct io_mapped_ubuf *imu) +{ + struct io_buf_dma *bd =3D NULL; + struct file *target_file =3D NULL; + struct dma_buf *dmabuf =3D NULL; + int ret; + + if (ctx->flags & IORING_SETUP_IOPOLL) + return -EOPNOTSUPP; + if (desc->uaddr || desc->size) + return -EINVAL; + + bd =3D kzalloc(sizeof(*bd), GFP_KERNEL); + if (!bd) + return -ENOMEM; + + target_file =3D fget(desc->target_fd); + if (!target_file) { + ret =3D -EBADF; + goto err; + } + dmabuf =3D dma_buf_get(desc->dmabuf_fd); + if (IS_ERR(dmabuf)) { + ret =3D PTR_ERR(dmabuf); + dmabuf =3D NULL; + goto err; + } + ret =3D io_validate_user_buf_range(0, dmabuf->size); + if (ret) + goto err; + + ret =3D dma_buf_io_ctx_create(target_file, dmabuf, DMA_BIDIRECTIONAL, + &bd->ctx); + if (ret) + goto err; + + bd->target_file =3D target_file; + imu->len =3D dmabuf->size; + imu->release =3D io_release_dmabuf; + imu->priv =3D bd; + imu->flags =3D IO_REGBUF_F_DMABUF | IO_REGBUF_F_UNCLONEABLE; + imu->dir =3D IO_BUF_DEST | IO_BUF_SOURCE; + dma_buf_put(dmabuf); + return 0; +err: + kfree(bd); + if (target_file) + fput(target_file); + if (dmabuf) + dma_buf_put(dmabuf); + return ret; +} + diff --git a/io_uring/dma-buf.h b/io_uring/dma-buf.h new file mode 100644 index 000000000000..a876a9e5e272 --- /dev/null +++ b/io_uring/dma-buf.h @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef IOU_DMA_BUF_H +#define IOU_DMA_BUF_H + +#include +#include + +#include "rsrc.h" + +struct io_buf_dma { + struct dma_buf_io_ctx *ctx; + struct file *target_file; +}; + +int __io_register_dmabuf(struct io_ring_ctx *ctx, + struct io_uring_regbuf_desc *desc, + struct io_mapped_ubuf *imu); + +static inline void io_detach_dmabuf_map(struct io_kiocb *req) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return; + if (!(req->flags & REQ_F_DMABUF) || !req->dmabuf_map) + return; + dma_buf_io_map_drop(req->dmabuf_map); + req->dmabuf_map =3D NULL; +} + +static inline int io_attach_dmabuf_map(struct io_kiocb *req, + struct iov_iter *iter, + unsigned issue_flags) +{ + bool nowait =3D issue_flags & IO_URING_F_NONBLOCK; + struct dma_buf_io_map *map; + struct io_buf_dma *bd; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + if (!(req->flags & REQ_F_DMABUF)) + return 0; + + bd =3D req->buf_node->buf->priv; + map =3D dma_buf_io_get_map(bd->ctx, nowait); + if (unlikely(IS_ERR(map))) + return PTR_ERR(map); + req->dmabuf_map =3D map; + iter->dmabuf_map =3D map; + return 0; +} + +static inline int io_import_dmabuf_map(struct io_kiocb *req, + int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + size_t len, size_t offset) +{ + struct io_buf_dma *bd =3D imu->priv; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + if (req->file !=3D bd->target_file) + return -EBADF; + req->flags |=3D REQ_F_DMABUF; + iov_iter_dmabuf_map(iter, ddir, NULL, offset, len); + return 0; +} + +static inline int io_register_dmabuf(struct io_ring_ctx *ctx, + struct io_uring_regbuf_desc *desc, + struct io_mapped_ubuf *imu) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + return __io_register_dmabuf(ctx, desc, imu); +} + +#endif /* IOU_DMA_BUF_H */ diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index f42d201738cd..2e89e091afbb 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -10,6 +10,7 @@ #include #include #include +#include =20 #include =20 @@ -19,6 +20,7 @@ #include "rsrc.h" #include "memmap.h" #include "register.h" +#include "dma-buf.h" =20 struct io_rsrc_update { struct file *file; @@ -898,6 +900,38 @@ bool io_check_coalesce_buffer(struct page **page_array= , int nr_pages, return true; } =20 +static struct io_rsrc_node *io_register_dmabuf_desc(struct io_ring_ctx *ct= x, + struct io_uring_regbuf_desc *desc) +{ + struct io_rsrc_node *node =3D NULL; + struct io_mapped_ubuf *imu =3D NULL; + int ret; + + node =3D io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); + if (!node) + return ERR_PTR(-ENOMEM); + + imu =3D io_alloc_imu(ctx, 0); + if (!imu) { + ret =3D -ENOMEM; + goto err; + } + memset(imu, 0, sizeof(*imu)); + refcount_set(&imu->refs, 1); + + ret =3D io_register_dmabuf(ctx, desc, imu); + if (ret) + goto err; + node->buf =3D imu; + return node; +err: + if (imu) + io_free_imu(ctx, imu); + if (node) + io_cache_free(&ctx->node_cache, node); + return ERR_PTR(ret); +} + static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_uring_regbuf_desc *desc) { @@ -918,6 +952,12 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) || desc->flags) return ERR_PTR(-EINVAL); =20 + if (desc->type =3D=3D IO_REGBUF_TYPE_DMABUF) + return io_register_dmabuf_desc(ctx, desc); + + if (desc->dmabuf_fd || desc->target_fd) + return ERR_PTR(-EINVAL); + if (desc->type =3D=3D IO_REGBUF_TYPE_EMPTY) { if (uaddr || size) return ERR_PTR(-EFAULT); @@ -1242,9 +1282,12 @@ static int io_import_kbuf(int ddir, struct iov_iter = *iter, return 0; } =20 -static int io_import_fixed(int ddir, struct iov_iter *iter, +static int io_import_fixed(struct io_kiocb *req, + int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, - u64 buf_addr, size_t len) + u64 buf_addr, size_t len, + unsigned issue_flags, + unsigned import_flags) { const struct bio_vec *bvec; size_t folio_mask; @@ -1264,6 +1307,11 @@ static int io_import_fixed(int ddir, struct iov_iter= *iter, =20 offset =3D buf_addr - imu->ubuf; =20 + if (imu->flags & IO_REGBUF_F_DMABUF) { + if (!(import_flags & IO_REGBUF_IMPORT_ALLOW_DMABUF)) + return -EFAULT; + return io_import_dmabuf_map(req, ddir, iter, imu, len, offset); + } if (imu->flags & IO_REGBUF_F_KBUF) return io_import_kbuf(ddir, iter, imu, len, offset); =20 @@ -1326,7 +1374,8 @@ int __io_import_reg_buf(struct io_kiocb *req, struct = iov_iter *iter, node =3D io_find_buf_node(req, issue_flags); if (!node) return -EFAULT; - return io_import_fixed(ddir, iter, node->buf, buf_addr, len); + return io_import_fixed(req, ddir, iter, node->buf, buf_addr, len, + issue_flags, import_flags); } =20 static int io_buffer_acct_cloned_hpages(struct io_ring_ctx *ctx, @@ -1753,7 +1802,9 @@ int __io_import_reg_vec(int ddir, struct iov_iter *it= er, iovec_off =3D vec->nr - nr_iovs; iov =3D vec->iovec + iovec_off; =20 - if (imu->flags & IO_REGBUF_F_KBUF) { + if (imu->flags & IO_REGBUF_F_DMABUF) { + return -EOPNOTSUPP; + } else if (imu->flags & IO_REGBUF_F_KBUF) { int ret =3D io_kern_bvec_size(iov, nr_iovs, imu, &nr_segs); =20 if (unlikely(ret)) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 351995b61e68..d893cc5e580a 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -28,6 +28,11 @@ struct io_rsrc_node { enum { IO_REGBUF_F_KBUF =3D 1 << 0, IO_REGBUF_F_UNCLONEABLE =3D 1 << 1, + IO_REGBUF_F_DMABUF =3D 1 << 3, +}; + +enum { + IO_REGBUF_IMPORT_ALLOW_DMABUF =3D 1 << 1, }; =20 struct io_mapped_ubuf { diff --git a/io_uring/rw.c b/io_uring/rw.c index 999d9a1cc5a5..d33b925a848e 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -23,6 +23,7 @@ #include "rsrc.h" #include "poll.h" #include "rw.h" +#include "dma-buf.h" =20 static void io_complete_rw(struct kiocb *kiocb, long res); static void io_complete_rw_iopoll(struct kiocb *kiocb, long res); @@ -358,13 +359,18 @@ static int io_init_rw_fixed(struct io_kiocb *req, uns= igned int issue_flags, struct io_async_rw *io =3D req->async_data; int ret; =20 - if (io->bytes_done) - return 0; + if (!io->bytes_done) { + ret =3D __io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, + issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF); + if (ret) + return ret; + iov_iter_save_state(&io->iter, &io->iter_state); + } =20 - ret =3D io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, - issue_flags); - iov_iter_save_state(&io->iter, &io->iter_state); - return ret; + ret =3D io_attach_dmabuf_map(req, &io->iter, issue_flags); + if (ret) + return ret; + return 0; } =20 int io_prep_read_fixed(struct io_kiocb *req, const struct io_uring_sqe *sq= e) @@ -577,6 +583,8 @@ static void io_complete_rw(struct kiocb *kiocb, long re= s) struct io_rw *rw =3D container_of(kiocb, struct io_rw, kiocb); struct io_kiocb *req =3D cmd_to_io_kiocb(rw); =20 + io_detach_dmabuf_map(req); + /* ring owner may block in freeze_super() before task_work runs */ if (kiocb->ki_flags & IOCB_WRITE) io_req_end_write(req); @@ -698,7 +706,7 @@ static ssize_t loop_rw_iter(int ddir, struct io_rw *rw,= struct iov_iter *iter) !(kiocb->ki_filp->f_flags & O_NONBLOCK)) return -EAGAIN; if ((req->flags & REQ_F_BUF_NODE) && - (req->buf_node->buf->flags & IO_REGBUF_F_KBUF)) + (req->buf_node->buf->flags & (IO_REGBUF_F_KBUF|IO_REGBUF_F_DMABUF))) return -EFAULT; =20 ppos =3D io_kiocb_ppos(kiocb); @@ -1226,7 +1234,10 @@ int io_read_fixed(struct io_kiocb *req, unsigned int= issue_flags) if (unlikely(ret)) return ret; =20 - return io_read(req, issue_flags); + ret =3D io_read(req, issue_flags); + if (ret !=3D -EIOCBQUEUED) + io_detach_dmabuf_map(req); + return ret; } =20 int io_write_fixed(struct io_kiocb *req, unsigned int issue_flags) @@ -1237,7 +1248,11 @@ int io_write_fixed(struct io_kiocb *req, unsigned in= t issue_flags) if (unlikely(ret)) return ret; =20 - return io_write(req, issue_flags); + ret =3D io_write(req, issue_flags); + if (ret !=3D -EIOCBQUEUED) + io_detach_dmabuf_map(req); + return ret; + } =20 void io_rw_fail(struct io_kiocb *req) --=20 2.54.0