From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4483D4A012B for ; Mon, 21 Sep 2026 13:39:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997968; cv=none; b=g4FahNhLqM0rv1lpUtWuJ2eflkk0HCzBjQiEN+drIDF0TCe4y/K2pzB8Q3005bzP51DNV2nwF22kyoUJo4xJrz7EYjJvaTKPxBQxo2BYGLGE5waw8m2gvEBPfefJbV7SafkoDuWPYWw6WvBke9eolvUw3QIah6YUhIrVGS0lUtg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997968; c=relaxed/simple; bh=KGtOg1J+fiW5XKDlyAKJq3Uud5eNfzJnff4wxetsBsI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=udL0dXPHDTE8xuRRdnQHmnjWW7zuxIyrqyHgyKzOZd3DNVlNPT8fDNoSaHmISaNoprxOboSDq1eKyvXrQTe6oNZEUDIH7mwXO8PZmxoT+izCiMHXTEnAPozp9V9Kazr/iu5GzTlFeCtETRT6Y5j76errNvP2nfBr7IYEYH8dlAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LVkwyZNZ; arc=none smtp.client-ip=74.125.225.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LVkwyZNZ" Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49ccf3ca626so14791005e9.0 for ; Mon, 21 Sep 2026 06:39:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997963; x=1790602763; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wbsLds5gQB7DU8emxytzuWlkdnk2zS91BnQHPN4/e+k=; b=LVkwyZNZ42L6lQaMY2Qo/a0agomCRfitzL7Y4O6GrtZWBdJm0vZ9HQgEBfIz3EE1nI kGh3LZn0NR06KymVJW6KkDskX65nsEPK6QCUvmsHWphHUODwnagK4k/zxtgQdGpKMuXt j1GiyRl1VA8Kx/MXidy+90XBcYc/zkAoZAvOnX7UywLoct6QzVNtCiNILQyTBvanHEAL qqHZrcK5rgA99cj0r9Vt8OM+dYPDZy+HM1It7az3wbQ+fk1HuXchVWCeR1eR/+mk4a/Y q0V8gwwn45bqaNH/pkc8BRkuR7wJw+j5GxRXwlDVVZZcFORs9Qt0xRdnR77bkdA4mpH/ F8ag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997963; x=1790602763; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wbsLds5gQB7DU8emxytzuWlkdnk2zS91BnQHPN4/e+k=; b=qChOrdMahJSAJ9uN7Z93tu0B3UcMFL4A19FlFPImXiLmgBvDRBi6olsgxRgvOKugdF 89eWOh+MTEQlkvIYilloEX+HbUzyRBdKRWsifgVriFtC5GJowjPgXcK550mgO9xVNymZ 4ArZHL8GWGb54Zlf95eO2ktV7E42DTNNndnD2vgfX8o6pPEkXLIoRasntKMaw9cjdexA SRF9Yx2Lay/SEY9SyCjh9h2vUzj4SZQAPFZdX4zJErZMlFbXPhXcYCSJu+Zie4E1ENyr v8wmrT85yTT+7FCgDEroK6JbDziuMGNjNQgm0t5HSRMGetQ5PjNmkSbKjjMbKXc2bbCV oLjg== X-Forwarded-Encrypted: i=1; AKwUvByFYKBrRnQ0hPT/KJP6yQRi9ieJE/faIwHC5Nq7HTp/c/ccmPvioHQO/0o8yLCZT2AEw496bK597C0GCdM=@vger.kernel.org X-Gm-Message-State: AFuF++k18NwEzHDunqWCHjHoSqNHxpB2GdPFyUTc6s8SB1CrOpqh56Y+ Dmk0xPBQks+6X/IrkaVkTomJquoXy+tDA/qaS4LKWcr9M62qhrCNKMIt X-Gm-Gg: AYBFou1e4nXz/o7WdkAzM1kqcGqBBglO8mY0FT95QnYFoktTyAIgNPdDLJ2SbWHzKUy na1MrkQqC9PR7vAtNLcVuDLiHLjuAKcC2aCudnFZ0avVaSclk1bpnhiazYARoECJcM/Q4Qfo1QG Y1vGHO1Zhz9U1a2fFy0rm+1gExvXIXVGtL5CKPdKPmxBKdVDV/eN/SbrE82DXl5gtiWi9YQkEmi rg7ty9vs4XVjgx+kjDJb/b8dBL7UJf7bBZ5l8RoZuRM8qHhs/7Hy8OfzkfLOfqWD6bJ3RnuHDI6 lshaS/HlAQ+2LKWQKtg3kWS8x1aAAejkLSPianAeo+3gZGGcugtUu7WU9wfXaRJW1yYCXUDXgaX ZhRgjBAalJRikCO3T+dQD71n75LJP4/iNtu00OlSe9F+fQzvwpoAxnCRKD0ju6uwkXuKSbuvsSV /eMOfQ/wNe2pQTuY1SEtEcqLr+/Y4qCD9pcSSC4J7tzie8sDqaOLM0nIduLrY9n7wGjC5wbQMvG htCCcuabHATJb/6SUlhJcXRBrffAi2O2Xdmjo1iLwvu1FtShtS8pt4i7rgoDSCo+PmT1HjXxbzK ElOgVhFUGqyyYr1Jk4HI2iUuS8AoCw== X-Received: by 2002:a05:600c:4e50:b0:49c:ffde:45ff with SMTP id 5b1f17b1804b1-49fc5728f5amr140455545e9.17.1789997963087; Mon, 21 Sep 2026 06:39:23 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:22 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 01/13] dma-buf: introduce initial file I/O infrastructure Date: Mon, 21 Sep 2026 14:38:45 +0100 Message-ID: <5490ee42c4452fd4198b245ead20fcf7438c066e.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The goal is to be able to natively use dma-buf in the read-write / IO path. This patch adds basic building blocks serving as a glue and API between drivers and upper layer subsystems providing the uAPI. Later patches implement it for NVMe raw block devices and expose it to the user space via io_uring. There are two main objects. struct dma_buf_io_ctx and struct dma_buf_io_map. The ctx is used during initial registration and serves as an interface between the upper layer user like io_uring and to the importer subsystem / driver. The map represents the actual dma map established for the target device[s] with dma_buf_map_attachment() and stored in a device specific format. The context is created via a new file operation ->init_dma_buf_io_ctx. The ctx-map separation exists to support map invalidation (see dma_buf_io_invalidate_mappings()). A ctx can create multiple maps during its lifetime, but there can only be no more than one (active) map attached to it. Invalidation drops the active map if present, and the next map will only be attempted to be created once there is a new request that wants to use the dma-buf IO ctx. The primary task of the dma_buf_io_map object is to count requests using it and to wait for their completion when we want to destroy the DMA map. [un]mapping and any work with dma addresses is delegated to the importer driver via an ops table stored in the ctx, see struct dma_buf_io_ops. Only the target driver / subsystem knows about devices it wants to use the dma-buf with, especially in case of multi-device filesystems or stacking in the future. Signed-off-by: Pavel Begunkov --- drivers/dma-buf/Makefile | 2 +- drivers/dma-buf/dma-buf-io.c | 219 +++++++++++++++++++++++++++++++++++ include/linux/dma-buf-io.h | 113 ++++++++++++++++++ include/linux/fs.h | 2 + 4 files changed, 335 insertions(+), 1 deletion(-) create mode 100644 drivers/dma-buf/dma-buf-io.c create mode 100644 include/linux/dma-buf-io.h diff --git a/drivers/dma-buf/Makefile b/drivers/dma-buf/Makefile index b25d7550bacf..523731b0f83e 100644 --- a/drivers/dma-buf/Makefile +++ b/drivers/dma-buf/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only obj-y :=3D dma-buf.o dma-fence.o dma-fence-array.o dma-fence-chain.o \ - dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o + dma-fence-unwrap.o dma-resv.o dma-buf-mapping.o dma-buf-io.o obj-$(CONFIG_DMABUF_HEAPS) +=3D dma-heap.o obj-$(CONFIG_DMABUF_HEAPS) +=3D heaps/ obj-$(CONFIG_SYNC_FILE) +=3D sync_file.o diff --git a/drivers/dma-buf/dma-buf-io.c b/drivers/dma-buf/dma-buf-io.c new file mode 100644 index 000000000000..8312637a299f --- /dev/null +++ b/drivers/dma-buf/dma-buf-io.c @@ -0,0 +1,219 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Common infrastructure for supporing dma-buf in the I/O path. + * + * Copyright (C) 2026 Pavel Begunkov + */ +#include +#include + +static void dma_buf_io_put_ctx(struct dma_buf_io_ctx *ctx) +{ + might_sleep(); + + if (WARN_ON_ONCE(rcu_dereference_protected(ctx->map, true))) + return; + + ctx->dev_ops->release(ctx); + + dma_buf_put(ctx->dmabuf); + mutex_destroy(&ctx->map_mutex); + mutex_destroy(&ctx->map_create_mutex); + kfree(ctx); +} + +static void dma_buf_io_map_release_work(struct work_struct *work) +{ + struct dma_buf_io_map *map =3D container_of(work, struct dma_buf_io_map, + release_work); + struct dma_buf_io_ctx *ctx =3D map->ctx; + struct dma_buf *dmabuf =3D ctx->dmabuf; + + dma_resv_lock(dmabuf->resv, NULL); + ctx->dev_ops->unmap(ctx, map); + dma_resv_unlock(dmabuf->resv); + + percpu_ref_exit(&map->refs); + kfree(map); + + atomic_dec(&ctx->all_maps); + wake_up(&ctx->maps_wq); +} + +static void dma_buf_io_map_refs_release(struct percpu_ref *ref) +{ + struct dma_buf_io_map *map =3D container_of(ref, struct dma_buf_io_map, r= efs); + struct dma_buf_io_ctx *ctx =3D map->ctx; + + /* There are no more requests using the map. */ + atomic_dec(&ctx->active_maps); + wake_up(&ctx->maps_wq); + + /* might sleep, use a worker */ + INIT_WORK(&map->release_work, dma_buf_io_map_release_work); + queue_work(system_percpu_wq, &map->release_work); +} + +static void dma_buf_io_wait_active_maps(struct dma_buf_io_ctx *ctx) +{ + wait_event(ctx->maps_wq, atomic_read(&ctx->active_maps) =3D=3D 0); +} + +static void dma_buf_io_wait_maps(struct dma_buf_io_ctx *ctx) +{ + wait_event(ctx->maps_wq, atomic_read(&ctx->all_maps) =3D=3D 0); +} + +int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map = *map, + struct sg_table *sgt) +{ + unsigned seg_shift =3D ~0U; + struct scatterlist *sg; + unsigned long tmp; + int ret; + + for_each_sgtable_dma_sg(sgt, sg, tmp) + seg_shift =3D min(seg_shift, __ffs(sg_dma_len(sg))); + + ret =3D percpu_ref_init(&map->refs, dma_buf_io_map_refs_release, 0, + GFP_KERNEL); + if (ret) + return ret; + map->min_seg_shift =3D seg_shift; + map->ctx =3D ctx; + return 0; +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_init_map, "DMA_BUF"); + +struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx) +{ + struct dma_buf *dmabuf =3D ctx->dmabuf; + struct dma_buf_io_map *map; + long ret; + + guard(mutex)(&ctx->map_create_mutex); + + scoped_guard(mutex, &ctx->map_mutex) { + if (ctx->maps_killed) + return ERR_PTR(-ENOENT); + /* recheck under the lock in case it has already been re-created */ + map =3D __dma_buf_io_get_map(ctx); + if (map) + return map; + } + + dma_buf_io_wait_active_maps(ctx); + + ret =3D dma_resv_lock_interruptible(dmabuf->resv, NULL); + if (ret) + return ERR_PTR(ret); + + ret =3D dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL, + true, MAX_SCHEDULE_TIMEOUT); + if (ret <=3D 0) { + if (!ret) + ret =3D -EAGAIN; + dma_resv_unlock(dmabuf->resv); + return ERR_PTR(ret); + } + + map =3D ctx->dev_ops->map(ctx); + dma_resv_unlock(dmabuf->resv); + + if (IS_ERR(map)) + return map; + if (WARN_ON_ONCE(!map->min_seg_shift)) + return ERR_PTR(-EFAULT); + + atomic_inc(&ctx->active_maps); + atomic_inc(&ctx->all_maps); + /* get a reference for the caller */ + percpu_ref_get(&map->refs); + + scoped_guard(mutex, &ctx->map_mutex) + rcu_assign_pointer(ctx->map, map); + return map; +} + +static void dma_buf_io_kill_maps(struct dma_buf_io_ctx *ctx, bool final) +{ + struct dma_buf_io_map *map; + + scoped_guard(mutex, &ctx->map_mutex) { + if (final) + ctx->maps_killed =3D true; + + map =3D rcu_dereference_protected(ctx->map, + lockdep_is_held(&ctx->map_mutex)); + if (!map) + return; + rcu_assign_pointer(ctx->map, NULL); + percpu_ref_kill(&map->refs); + } +} + +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx) +{ + dma_buf_io_kill_maps(ctx, false); + dma_buf_io_wait_active_maps(ctx); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_invalidate_mappings, "DMA_BUF"); + +void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx) +{ + /* Remove and wait for the last map, there should be no new ones. */ + dma_buf_io_kill_maps(ctx, true); + dma_buf_io_wait_maps(ctx); + dma_buf_io_put_ctx(ctx); +} + +int dma_buf_io_ctx_create(struct file *file, + struct dma_buf *dmabuf, + enum dma_data_direction dir, + struct dma_buf_io_ctx **out_ctx) +{ + struct dma_buf_io_ctx *ctx; + int ret; + + if (!file->f_op->init_dma_buf_io_ctx) + return -EOPNOTSUPP; + + ctx =3D kmalloc_obj(*ctx); + if (!ctx) + return -ENOMEM; + + memset(ctx, 0, sizeof(*ctx)); + ctx->dir =3D dir; + ctx->dmabuf =3D dmabuf; + get_dma_buf(dmabuf); + mutex_init(&ctx->map_mutex); + mutex_init(&ctx->map_create_mutex); + atomic_set(&ctx->active_maps, 0); + atomic_set(&ctx->all_maps, 0); + init_waitqueue_head(&ctx->maps_wq); + + ret =3D file->f_op->init_dma_buf_io_ctx(file, ctx); + if (ret) { + kfree(ctx); + dma_buf_put(dmabuf); + return ret; + } + + if (WARN_ON_ONCE(!ctx->dev_ops || + !ctx->dev_ops->map || + !ctx->dev_ops->unmap || + !ctx->dev_ops->release)) + return -EINVAL; + + *out_ctx =3D ctx; + return 0; +} + +void dma_buf_io_detach(struct dma_buf_io_ctx *ctx) +{ + guard(mutex)(&ctx->map_create_mutex); + + dma_buf_io_kill_maps(ctx, true); + dma_buf_io_wait_maps(ctx); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_io_detach, "DMA_BUF"); diff --git a/include/linux/dma-buf-io.h b/include/linux/dma-buf-io.h new file mode 100644 index 000000000000..5ea92fc78582 --- /dev/null +++ b/include/linux/dma-buf-io.h @@ -0,0 +1,113 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __DMA_BUF_IO_H__ +#define __DMA_BUF_IO_H__ + +#include + +struct dma_buf_io_ctx; +struct dma_buf_io_map; + +struct dma_buf_io_ops { + /* + * Create a new map for the given ctx. Called with the reservation + * lock held. + */ + struct dma_buf_io_map *(*map)(struct dma_buf_io_ctx *ctx); + + /* + * Clean up device specific parts of the @map. Called with the + * reservation lock held. + */ + void (*unmap)(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map *map); + + /* + * The user tries to destroy the ctx. Release all device specific + * parts of the token. + */ + void (*release)(struct dma_buf_io_ctx *); +}; + +struct dma_buf_io_map { + /* + * Counts attached requests and other users. Device specific unmapping + * is deferred until all refs are dropped. + */ + struct percpu_ref refs; + /* + * Shift for the minimum segment size of the mapping. + */ + unsigned min_seg_shift; + + struct work_struct release_work; + struct dma_buf_io_ctx *ctx; +}; + +struct dma_buf_io_ctx { + struct dma_buf_io_map __rcu *map; + struct dma_buf *dmabuf; + enum dma_data_direction dir; + + /* synchronises map reassignment */ + struct mutex map_mutex; + struct mutex map_create_mutex; + /* maps that may still be in use. */ + atomic_t active_maps; + atomic_t all_maps; + struct wait_queue_head maps_wq; + bool maps_killed; + + void *dev_priv; + const struct dma_buf_io_ops *dev_ops; +}; + +int dma_buf_io_ctx_create(struct file *file, + struct dma_buf *dmabuf, + enum dma_data_direction dir, + struct dma_buf_io_ctx **ctx); +void dma_buf_io_ctx_release(struct dma_buf_io_ctx *ctx); + +struct dma_buf_io_map *dma_buf_io_create_map(struct dma_buf_io_ctx *ctx); + +static inline struct dma_buf_io_map * +__dma_buf_io_get_map(struct dma_buf_io_ctx *ctx) +{ + struct dma_buf_io_map *map; + + guard(rcu)(); + + map =3D rcu_dereference(ctx->map); + if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs))) + return NULL; + + return map; +} + +static inline struct dma_buf_io_map * +dma_buf_io_get_map(struct dma_buf_io_ctx *ctx, bool nowait) +{ + struct dma_buf_io_map *map; + + map =3D __dma_buf_io_get_map(ctx); + if (likely(map)) + return map; + + if (nowait) + return ERR_PTR(-EAGAIN); + return dma_buf_io_create_map(ctx); +} + +static inline void dma_buf_io_map_drop(struct dma_buf_io_map *map) +{ + percpu_ref_put(&map->refs); +} + +/* + * Device API + */ + +void dma_buf_io_invalidate_mappings(struct dma_buf_io_ctx *ctx); +int dma_buf_io_init_map(struct dma_buf_io_ctx *ctx, struct dma_buf_io_map = *map, + struct sg_table *sgt); +void dma_buf_io_detach(struct dma_buf_io_ctx *ctx); + +#endif diff --git a/include/linux/fs.h b/include/linux/fs.h index f9d1e05e8ae6..05c1ff495732 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1914,6 +1914,7 @@ struct dir_context { #define COPY_FILE_SPLICE (1 << 0) =20 struct io_uring_cmd; +struct dma_buf_io_ctx; struct offset_ctx; =20 struct file_operations { @@ -1960,6 +1961,7 @@ struct file_operations { int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *, unsigned int poll_flags); int (*mmap_prepare)(struct vm_area_desc *); + int (*init_dma_buf_io_ctx)(struct file *, struct dma_buf_io_ctx *); } __randomize_layout; =20 /* Supports async buffered reads */ --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD2514A014D for ; Mon, 21 Sep 2026 13:39:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997971; cv=none; b=a0HNU89d/AW5UrhldKU3/qANLm3BV6NF1L5fDE6hliNvSbcfAYJgk1p924Gm1SQDBuc5lfe0vxA66k+Z8R8dhsX7D3nfQdWC3dDLTkp8q//JA93vz/fXr54JfVEhoxG+Sl/PPOERii4GaDE2H87Tc6z/tAKAar1ImtlcnvR7dec= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997971; c=relaxed/simple; bh=9vQH2FkifwfjUkmpno2ejJJq+JJRqKfmbGSl7/camvA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mhLzL3wZLzt75zxufXAEshSPHueQ/zeMk/Kt41NqrF9rHFek1RZZFd2kk2BtC+5oXr6BGLHmXl5TwVM3bcNnQtT0Sw/H18XhfAova8WS2mOBXJ0VhQ4s5eHZfkv/NZ+UEPzeh8CJBelqipSjfvNoBcPDRclH9sWFE4KbYI4bR6Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OBX0Kho9; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OBX0Kho9" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49e79a408deso16962715e9.2 for ; Mon, 21 Sep 2026 06:39:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997967; x=1790602767; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=lhfYmNfKHeufPvJ/pLV04jPXJys5SvK0KBUyCDM0noo=; b=OBX0Kho9R6MTUYOALWN0nKWJp+S/nhUEONjYX7gfgpUph7N899PxoK1kfuxxth8oGW Lx7JG0hz5qtBxnGYBGsLP+g29A0o2Zb8vkwyzYXyedoUivYksnuI0+Gvc+sGOXmH6Js8 rKguejXDcfFbgT8m9ZkU3v0KtoFTFi4Lz+YeS+k2mHz8YvWxCHbh1oEwQASW/h6QJDFX oG8hZN25N0tXeiMWvuBN4QlZVp25ZdEVJbYjZ/ZVeMu6CaHhapvjfs6KkFIuZiTEj5gX kBRHccMC2MTXllk821eiqbU9KSkLU9biO+Z8IYmep5AZd6asZ+Ny50m8BszeTh5kxAAm 5yGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997967; x=1790602767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=lhfYmNfKHeufPvJ/pLV04jPXJys5SvK0KBUyCDM0noo=; b=2dT1awsiqgVgL6y2W+VVWS6WeOG/X5/rTgrJGNjOyJDk0smyP68U84KyQcBrhd2Mka dNOu2odpKg4hd81hHJOZkjtSw9337ISIw4S+51f/ikT+H4+KebHbywCGU5oJSCrA1CPt YVbFN0Z3uzFxX+aASwLnWGJxc9CREZmoyfBYZEEnNc5whXUHilGGKD38UvwnHZWhSucU BVV7ViIVWZ+G2eD5XqkWogwknVp/l3jwDAZXvVQQtAV7dFYgZruyBsjADGSaCi3exNr4 RAzpgQsh2/elISwE1+DfmGo5LrZzMNXfzjAmNLVsjRNDrnqCD3bTyVEvzfBsYRzd1E3w km0Q== X-Forwarded-Encrypted: i=1; AKwUvByJqAOPm/ukqeE1CqpdUMJFmhmGsGMZ6BAS6P2Muq/8q0sLN0B06JNGTPxnoZMC1h558h3G+pjKxxoPHkQ=@vger.kernel.org X-Gm-Message-State: AFuF++mJpmJSTSI3cqGgZGn0YG0udJQMne6a/knbpU6EgGC/g76++wxR G2xUNHAoOkfpWazH/TBMunKigg6bru0/daB+L5UXyAZh0i4rY+NVIRkN X-Gm-Gg: AYBFou0wb4CQiMrSziVIFUCPmIazDVHBKvUXyw4y9GR/rQGetQdrXmdHDoRBtA+XjbT pU6z0acrnCOVXX5OjmCLGHEgZ7MRqEcyG92BO1rvG3oYZe964EHRq3Vc3jBQUXGYF9bQLzSjsua SxpfS81TrXVB5elVTWowFb14My9+HEBBQyN6aM+urLveNrWXON/Ib3NkVdU45u5O4576atjdqj2 kBY9rSGuFyRyu79/Yrh4XULmO6+0fiOYJ/UVasJOGiLFnIDw40iyoyqTmG8c+cI/tFJbdbZrT/a VqfBEmesriBIk+Z36wYBWyreBlE+PDcQmV3z9k2wXukHCXnsxzxtPgk4hKwy10ys4nqzkMMBWOd 8n5C+Bz/gXVC3ZmdHevHWzvIiLljEq1UWlkWrZT1vDRCFRu/PHbd6yll5nigV+Ywer2kYgbt6P4 eP9kqD66KFIsSFG2MGyqz4hP6ILwatXA3czR53kLunlwLi//j/+lZFUdL0uCtneKXSplzAGHT1y IUp2ou3LZ3S8heSoZe8G1G4uVPJLPmbxYOQN5Sdld7HvAjruMfVODIp1kwJ+d1FAdHOb6HtWqbX /HDf3q3YE5Qd3XA4aBKFwni9vNOcZQ== X-Received: by 2002:a05:600c:19cb:b0:49e:8184:f619 with SMTP id 5b1f17b1804b1-49fc57410f2mr151435725e9.22.1789997966776; Mon, 21 Sep 2026 06:39:26 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:26 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 02/13] iov_iter: add iterator type for dmabuf maps Date: Mon, 21 Sep 2026 14:38:46 +0100 Message-ID: <6b78003332b57ec3a4552d7393464636793aff76.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce a new iterator type for dmabuf maps. The map in an opaque object with internals and format specific to the subsystem / driver, and only it can use that subsystem / driver for issuing IO. The task of the middle layers is to pass the map / iterator further down, maybe doing basic splitting and length checking. The iterator can only be used by operations of the file the associated map was created for. Suggested-by: Keith Busch Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- include/linux/uio.h | 11 +++++++++++ lib/iov_iter.c | 29 +++++++++++++++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index fe2e985d74d2..638c1116e912 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -12,6 +12,7 @@ =20 struct page; struct folio_queue; +struct dma_buf_io_map; =20 typedef unsigned int __bitwise iov_iter_extraction_t; =20 @@ -29,6 +30,7 @@ enum iter_type { ITER_FOLIOQ, ITER_XARRAY, ITER_DISCARD, + ITER_DMABUF_MAP, }; =20 #define ITER_SOURCE 1 // =3D=3D WRITE @@ -71,6 +73,7 @@ struct iov_iter { const struct folio_queue *folioq; struct xarray *xarray; void __user *ubuf; + struct dma_buf_io_map *dmabuf_map; }; size_t count; }; @@ -155,6 +158,11 @@ static inline bool iov_iter_is_xarray(const struct iov= _iter *i) return iov_iter_type(i) =3D=3D ITER_XARRAY; } =20 +static inline bool iov_iter_is_dmabuf_map(const struct iov_iter *i) +{ + return iov_iter_type(i) =3D=3D ITER_DMABUF_MAP; +} + static inline unsigned char iov_iter_rw(const struct iov_iter *i) { return i->data_source ? WRITE : READ; @@ -300,6 +308,9 @@ void iov_iter_folio_queue(struct iov_iter *i, unsigned = int direction, unsigned int first_slot, unsigned int offset, size_t count); void iov_iter_xarray(struct iov_iter *i, unsigned int direction, struct xa= rray *xarray, loff_t start, size_t count); +void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction, + struct dma_buf_io_map *map, + loff_t off, size_t count); ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages, size_t maxsize, unsigned maxpages, size_t *start); ssize_t iov_iter_get_pages_alloc2(struct iov_iter *i, struct page ***pages, diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 2072c04e99d0..6831a5d9396e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -575,7 +575,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size) { if (unlikely(i->count < size)) size =3D i->count; - if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i))) { + if (likely(iter_is_ubuf(i)) || unlikely(iov_iter_is_xarray(i)) || + unlikely(iov_iter_is_dmabuf_map(i))) { i->iov_offset +=3D size; i->count -=3D size; } else if (likely(iter_is_iovec(i) || iov_iter_is_kvec(i))) { @@ -631,7 +632,8 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll) return; } unroll -=3D i->iov_offset; - if (iov_iter_is_xarray(i) || iter_is_ubuf(i)) { + if (iov_iter_is_xarray(i) || iter_is_ubuf(i) || + iov_iter_is_dmabuf_map(i)) { BUG(); /* We should never go beyond the start of the specified * range since we might then be straying into pages that * aren't pinned. @@ -775,6 +777,20 @@ void iov_iter_xarray(struct iov_iter *i, unsigned int = direction, } EXPORT_SYMBOL(iov_iter_xarray); =20 +void iov_iter_dmabuf_map(struct iov_iter *i, unsigned int direction, + struct dma_buf_io_map *map, + loff_t off, size_t count) +{ + WARN_ON(direction & ~(READ | WRITE)); + *i =3D (struct iov_iter){ + .iter_type =3D ITER_DMABUF_MAP, + .data_source =3D direction, + .dmabuf_map =3D map, + .count =3D count, + .iov_offset =3D off, + }; +} + /** * iov_iter_discard - Initialise an I/O iterator that discards data * @i: The iterator to initialise. @@ -856,7 +872,7 @@ unsigned long iov_iter_alignment(const struct iov_iter = *i) return iov_iter_alignment_bvec(i); =20 /* With both xarray and folioq types, we're dealing with whole folios. */ - if (iov_iter_is_folioq(i)) + if (iov_iter_is_folioq(i) || iov_iter_is_dmabuf_map(i)) return i->iov_offset | i->count; if (iov_iter_is_xarray(i)) return (i->xarray_start + i->iov_offset) | i->count; @@ -872,7 +888,7 @@ unsigned long iov_iter_gap_alignment(const struct iov_i= ter *i) size_t size =3D i->count; unsigned k; =20 - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i)) return 0; =20 if (WARN_ON(!iter_is_iovec(i))) @@ -1469,11 +1485,12 @@ EXPORT_SYMBOL_GPL(import_ubuf); void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state) { if (WARN_ON_ONCE(!iov_iter_is_bvec(i) && !iter_is_iovec(i) && - !iter_is_ubuf(i)) && !iov_iter_is_kvec(i)) + !iter_is_ubuf(i) && !iov_iter_is_kvec(i) && + !iov_iter_is_dmabuf_map(i))) return; i->iov_offset =3D state->iov_offset; i->count =3D state->count; - if (iter_is_ubuf(i)) + if (iter_is_ubuf(i) || iov_iter_is_dmabuf_map(i)) return; /* * For the *vec iters, nr_segs + iov is constant - if we increment --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CFEB4A1E17 for ; Mon, 21 Sep 2026 13:39:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997975; cv=none; b=ptaD4f7DgvGmsFpjnXboyRCE4J7SXKVCbTvaHMlPqKFEN8eGcahWI2Vt0jYNVbJENp9EO6rhnuxaE9048Dr7ltpwbvtioopfPmCmLMvCAn6nmRJ7dWUL+VkIBRuFk1YvvB/kzhQYgOxX8tShEkHJ5GKysYqewUuHPEShJ07DRlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997975; c=relaxed/simple; bh=lqtmMOq9NyJHWkCTixF/EfRvdzo9ti4x7BRbdW0dvpo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l1L0nYpksj9HfFYQ9m+51ezg+HjrI/bIyyuTqO4qGGnTB/gSyq6dydD4CeWwj4pQn5w0fotegiWy/rOK0sTo9FMY9mj2371EkIZ9tfkAHm/q77xhH+2/L/hsCcs46ir++KM4ky13OfgENEpkvspBaPB8/fBBGHvSrqVOfSyAHrs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=knfo6iGm; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="knfo6iGm" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49cd5462b69so15884835e9.1 for ; Mon, 21 Sep 2026 06:39:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997970; x=1790602770; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=BooWm7URUcyrH2E2qkRWxlx7KQoz7GWFpr53v56Fl9k=; b=knfo6iGma+p5YGcX3sMSvqn1cE2jcWgAb6FR1sf0KBcsHhjevqWGwL62g0WiVWH9QJ sWgyh70nu9IKWRBj8XEB9jUkkeFvfLXHWLalsArfPtCBJ3D3Tg+r+2Cuo3BcyADPcEo3 4JmrQsMvNG/kwKpbfkzdsP56gViW31u9cssaB6RQ2dWvsWXo+GU54HD29p6uDFzMTOf1 uMt4nhViArd593RfEJdF/K2iI+XuF5U4Apza/2hmuN9BM+zw89gFCw7jvd04fK7MRE5m wDy6w27RO7GItjGoRvKawp8A6E2+iE2UESJhHD9by/Qr/iDMYlaa5NumYkYIO/S+hPZj q6oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997970; x=1790602770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=BooWm7URUcyrH2E2qkRWxlx7KQoz7GWFpr53v56Fl9k=; b=dedh1qHh7V11x3KTDFuMSIXj0irnA7pXUaaZTEbgkaTzM/GSNrTFhE1/TYPrc4k7UE 68c3zB3Shvs4BUyUfoWZBp5zelQaT0cRy4fQUX7l5SpdXdhMr5IB2RIzWTgVJYSNWrlG VBuCs13C8NEREkfPHqYXoKZKV7n8Rc3mUSKNc2VoNzI8MB0Hxo1A1vmVYtakKaJwXMVr Pk6G2mrP53Br7yLd6cwRgZ5qCQ9DzjT7qUOl5MZBQKSsN9Di6S9FiFxp9asIZ7UZhfii 19PgM+CMmN/gqx9qsMAlZFvtwCdPkR4Q3KCxetzp4ak0WVUlHx0C7m1ftjktrl9+fA9O lk0A== X-Forwarded-Encrypted: i=1; AKwUvBxLuBz9Rbuc0AleXDSrNBDGSrl3lrA37LuzYR/GEXEFvzop9noZEr5ueLoyPStemHV6dCE5sC97rhB1OHI=@vger.kernel.org X-Gm-Message-State: AFuF++kGcfrwtY1zWUKWuQQjCAd4py3IWeEteeNYZrn10FHoZAjMaMVM D036UDbamTWGNKOCHU+5TkJ1HquB5ELx5jeq15JTZRTvkqA10lpIwEmw X-Gm-Gg: AYBFou3bSIPSMesMOCahwQnsLRDWyrY9AOekamz36kW/CQuvYQLeovoKDUdbXQjpC05 HYPdmytnNkTS6SHtyp2Ln3/ZaPDCxREFrS+TqiXxVgsdX2CXUJNvQSKhAHQ573chss8n0RZwmDm +gkRKrgvRnQyv9LqOW/MTiXDw2tevhnbYxSdjB0IVhch7BKUPlaxwctarrS6DPMmEL70IuxQsfh N22dhi8hhRxGYdO47CCYNRBORYaT6pcCYMTvRo7Pa7o0F8MF9ineokYldGfEWLc6EhUrDUS/DOL TVRoCHDfsdaWleeo4pzOCDK/nRcUE8PtmEDNKKoLgdiNdpyp8aq458r77ZOx7k4qEPK6wvbTPtW J0wRmYiI67yHOQdetyTzqFIPfo0Y8Zan0sml6CKdcq5u69CT1bxiMqhQjfarvt7kRvHE9qB6u4z yCDuZS6/4wH2XLTwWFJ3ICWX2Iy49lcOV14H8+D+b5sfBZ/e892F+7FIjZgJamhUFWVJBqpZslQ i80ppnmkiRzVjn334b2n03fTSDnY6h/LfYUSMJv55aj7tDgBb2iN9FSAGtelEKub2M4WSTtsKsD 70DyqWkFyxk40v+3+Wa+XdiGhlTYEg== X-Received: by 2002:a05:600c:c0c3:10b0:49e:823a:9dec with SMTP id 5b1f17b1804b1-49fc5a10b5fmr145292005e9.3.1789997970323; Mon, 21 Sep 2026 06:39:30 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:29 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 03/13] block: always adjust bi_offset on bio_advance_iter Date: Mon, 21 Sep 2026 14:38:47 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extend bio_no_advance_iter() iteration to also increment the offset. It's not currently needed because the currently listed request types don't have a buffer, but will be useful once we add bios backed by dma-buf, which don't have a bvec but work with a single range. Suggested-by: Christoph Hellwig Reviewed-by: Christoph Hellwig Signed-off-by: Pavel Begunkov --- drivers/md/dm-io-rewind.c | 6 ++++-- include/linux/bio.h | 12 ++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/drivers/md/dm-io-rewind.c b/drivers/md/dm-io-rewind.c index 04f3fc8aeb6f..b22719c6411c 100644 --- a/drivers/md/dm-io-rewind.c +++ b/drivers/md/dm-io-rewind.c @@ -113,10 +113,12 @@ static inline void dm_bio_rewind_iter(const struct bi= o *bio, iter->bi_sector -=3D bytes >> 9; =20 /* No advance means no rewind */ - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size +=3D bytes; - else + iter->bi_offset -=3D bytes; + } else { dm_bvec_iter_rewind(bio->bi_io_vec, iter, bytes); + } } =20 /** diff --git a/include/linux/bio.h b/include/linux/bio.h index 17944e44b584..892ca469c570 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -113,11 +113,13 @@ static inline void bio_advance_iter(const struct bio = *bio, { iter->bi_sector +=3D bytes >> 9; =20 - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -=3D bytes; - else + iter->bi_offset +=3D bytes; + } else { bvec_iter_advance(bio->bi_io_vec, iter, bytes); /* TODO: It is reasonable to complete bio with error here. */ + } } =20 /* @bytes should be less or equal to bvec[i->bi_idx].bv_len */ @@ -127,10 +129,12 @@ static inline void bio_advance_iter_single(const stru= ct bio *bio, { iter->bi_sector +=3D bytes >> 9; =20 - if (bio_no_advance_iter(bio)) + if (bio_no_advance_iter(bio)) { iter->bi_size -=3D bytes; - else + iter->bi_offset +=3D bytes; + } else { bvec_iter_advance_single(bio->bi_io_vec, iter, bytes); + } } =20 void __bio_advance(struct bio *, unsigned bytes); --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 935814A260D for ; Mon, 21 Sep 2026 13:39:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997978; cv=none; b=FjCoXnXu9kpg2cjSEVZDH3g5ySE0VX0WxDwC83xjZcp3N2+CKELZ2zmacMdsIEpchVjxxaNdGHyyMwIIMHBc1lfu5hWUWF9V5BHgH8LSLZ5YTfDamWMNqM9/H6TyrglxfXZV/Iy0KRW0RSwUEis1wXKr4XN6x0M3mwdNTEz12A0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997978; c=relaxed/simple; bh=SPDNlCQKU7wLOqZjUSkqlJqBFbH1fKRBG8UUakIzbGM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i6UHPsf6txvhQpAmw/s3B3uneUQVmItvlsNjTpT8+wkPkTISVZzIV76nltLMM2ycQ3w7JCZD6FzaSkV8q7tVEorChJFvxnqevEcJqBMEqm1SBdch2CZh7mE9aUirE11NKbpfP/GcC36YaYS/lxhwAx/B8TWfQitsnuI/X1o7NKs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QIWdvbLK; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QIWdvbLK" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49d097b4939so15408865e9.0 for ; Mon, 21 Sep 2026 06:39:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997974; x=1790602774; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dxtQQDz5ZZ2oYc2awZZ8esOC2pl4iP5YFfGRHxTLRW8=; b=QIWdvbLKkogRcYJH+iaXmHhctEYaQ4/GsuR16GpElAIEMO5QmaRyQShcZSMDYOlTia GX/NZJT5j80zh5ZOpYCsGEbGBkmZ19ENMcSSUauPXrMhcbaYhoA7QosQ4qkUKJaBZUBM nThyYZcAoe427veumNsRQo71PBoiMUzNKW///44EFMafdhVqrXHzJbL28P0TxE+LQuRO gSUF+FQupwtZgrWJgti7xIRZObsAe+gDGg8uRNX0ZLlcJptZCqhcwIUKX69fiDEK2T/0 tl9vBRdLyfPYJbCR/DmJO+PdRzqnmK8xuiNmn0xqPnIZD2r4tpQeBe9kR6bIEQEKMmxg M2fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997974; x=1790602774; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=dxtQQDz5ZZ2oYc2awZZ8esOC2pl4iP5YFfGRHxTLRW8=; b=waQ30sB5pYV69YsdurGqeSepO5QzRiIlO/0oy9tL9wcgNTfhD/3M+hXvfU+LBY2Itv waopoEpQE24cv3HbFBUfMiKhAhjAnWi8IY/szzNkEDsgSVPGlcv+GIDTqi9Jj8LD6Igy Sin9EhTqN2i9nbKL4voaNNhupPizmPH4WRDPYvPxWy9U9TKkUKwwHqJTWB9AJSe4ojBr jU7FFOxEJNsx4DkSKRdc2foURio+Z+FEaK9xlf3sIC5xdnq9dMcoaZQ22kqNKViC7APB hDvRKVM1GzD5aBhYukfCkA7131AvsD41Pf7DYBmZzFyNuIXX3GLnCyqnRtZ7vOh8LKmu CWNQ== X-Forwarded-Encrypted: i=1; AKwUvBzq8izZwqDzx1uCt/tDX3lGTkBi0jslr+P22q8P+1tK4VH19tiw0syRcn69n8euafQ8Z2HhqLqptVQB1Mw=@vger.kernel.org X-Gm-Message-State: AFuF++nyEv/5Hoigzd6zJrDK+4ChXTWTu74mqlUszIWKbTEekep/eY4I ZnG7FmhyI1iyFbcldh1/RK1dKp812FSvhtbZ14lQneGI700DGxrhQ1EC X-Gm-Gg: AYBFou0y4kfeSAqSKVuNtxCN6CCFFq9UbybqYJjuKZCvBTJsQo+qMxyLCdUaabCPVrC gphiag584ztTJ+IzUWUSV/ov12KiZtT3ytzLmVf4pKa1artSfM+hvM1a2n9JeImfnanaaD4V6md JwK+fuwcUKR56TAOgrtJ9IItnBg2XGS7DJjaHi8xtXgbYqcJP7WWXWZcsKSZl36Q6+Dk/z0869x /lx8DSvDRwD4f2RbFY41BUh3eqOso/uROvnPKyB3UbWW/S+FcuBWFzL38RJMNBbvgO6WF8lmOT7 lqUMF6mt/1qDvZImkiQxCwbPhprngAwKXMF/EuJopciIDnsJ875Obxzj7PDEIs+tnxSMjXG5S50 pHM26i/AseTQsQllsPTsRD/AS3C7E8PZ2gtokQqy8wDlHHeFnjwrbbWEmUQGwORkmH1Sm0molaJ MJ7pduOksMaFHdoZK9ilLDG4cgqEov3bT8jUAD1LyltAkDuWyEgIRaIAtGzjouNvFpqNgDg9pb+ 7YP9Q02tLNsC6hyxMlyhAs+xTVKCDIpyLtqS7nDVR4N9rON8igy4lNqkVk0vZhw3NtzvRBpxS2A eJTMnNXZsfNm3Irf3pV3GTs3RnvK8w== X-Received: by 2002:a05:600d:1a:b0:49e:6778:c2bd with SMTP id 5b1f17b1804b1-49fc5a10c99mr131242505e9.6.1789997973654; Mon, 21 Sep 2026 06:39:33 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:32 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 04/13] block: introduce dma map backed bio type Date: Mon, 21 Sep 2026 14:38:48 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Premapped buffers don't require a generic bio_vec since these have already been dma mapped. Repurpose the bi_io_vec space to strore dmabuf maps as they are mutually exclusive. Suggested-by: Keith Busch Signed-off-by: Pavel Begunkov --- block/bio.c | 15 +++++++++++-- block/blk-merge.c | 45 +++++++++++++++++++++++++++++++++++++++ block/fops.c | 2 +- include/linux/bio.h | 9 ++++---- include/linux/blk-mq.h | 7 ++++++ include/linux/blk_types.h | 14 +++++++++++- include/linux/bvec.h | 3 ++- 7 files changed, 86 insertions(+), 9 deletions(-) diff --git a/block/bio.c b/block/bio.c index b48091c7663f..1e0d9714c541 100644 --- a/block/bio.c +++ b/block/bio.c @@ -881,7 +881,11 @@ static int __bio_clone(struct bio *bio, struct bio *bi= o_src, gfp_t gfp) bio->bi_write_stream =3D bio_src->bi_write_stream; bio->bi_bvec_gap_bit =3D bio_src->bi_bvec_gap_bit; bio->bi_iter =3D bio_src->bi_iter; - bio->bi_io_vec =3D bio_src->bi_io_vec; + + if (op_is_dmabuf(bio->bi_opf)) + bio->bi_dmabuf_map =3D bio_src->bi_dmabuf_map; + else + bio->bi_io_vec =3D bio_src->bi_io_vec; =20 if (bio->bi_bdev) { if (bio->bi_bdev =3D=3D bio_src->bi_bdev && @@ -1204,16 +1208,23 @@ EXPORT_SYMBOL_GPL(__bio_release_pages); =20 bool bio_iov_iter_set(struct bio *bio, const struct iov_iter *iter) { - if (!iov_iter_is_bvec(iter)) + if (!iov_iter_is_bvec(iter) && !iov_iter_is_dmabuf_map(iter)) return false; =20 WARN_ON_ONCE(bio->bi_max_vecs); =20 + static_assert(offsetof(struct bio, bi_io_vec) =3D=3D + offsetof(struct bio, bi_dmabuf_map)); + static_assert(offsetof(struct iov_iter, bvec) =3D=3D + offsetof(struct iov_iter, dmabuf_map)); + bio->bi_io_vec =3D (struct bio_vec *)iter->bvec; bio->bi_iter.bi_idx =3D 0; bio->bi_iter.bi_offset =3D iter->iov_offset; bio->bi_iter.bi_size =3D iov_iter_count(iter); bio_set_flag(bio, BIO_CLONED); + if (iov_iter_is_dmabuf_map(iter)) + bio->bi_opf |=3D REQ_NOMERGE | REQ_DMABUF; return true; } =20 diff --git a/block/blk-merge.c b/block/blk-merge.c index 258a726071d1..f0ece378c2d1 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -9,6 +9,7 @@ #include #include #include +#include =20 #include =20 @@ -319,6 +320,36 @@ static inline unsigned int bvec_seg_gap(struct bio_vec= *bvprv, return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len); } =20 +static inline int bio_split_io_at_dmabuf(struct bio *bio, + const struct queue_limits *lim, unsigned *segs, + unsigned max_bytes, unsigned len_align_mask, + unsigned start_align_mask) +{ + unsigned bytes =3D min(bio->bi_iter.bi_size, max_bytes); + unsigned seg_shift =3D bio->bi_dmabuf_map->min_seg_shift; + unsigned offset =3D bio->bi_iter.bi_offset & ((1U << seg_shift) - 1); + u64 max_segs_bytes; + + if ((bio->bi_iter.bi_offset & start_align_mask) || + (bio->bi_iter.bi_size & len_align_mask)) + return -EINVAL; + + /* single contiguous range into the dma-buf */ + *segs =3D 1; + + /* + * Limit by the number of segments. We don't expose the underlying + * mapping layout, but with a known minimum segment size, any I/O + * consisting of N full segments should be able to cover at least + * this much. + */ + max_segs_bytes =3D (u64)lim->max_segments << seg_shift; + bytes =3D min_t(u64, bytes, max_segs_bytes - offset); + if (bytes !=3D bio->bi_iter.bi_size) + return bytes; + return 0; +} + /** * bio_split_io_at - check if and where to split a bio * @bio: [in] bio to be split @@ -346,6 +377,19 @@ int bio_split_io_at(struct bio *bio, const struct queu= e_limits *lim, len_align_mask |=3D (bc->bc_key->crypto_cfg.data_unit_size - 1); } =20 + if (op_is_dmabuf(bio->bi_opf)) { + int ret; + + ret =3D bio_split_io_at_dmabuf(bio, lim, &nsegs, max_bytes, + len_align_mask, start_align_mask); + if (ret < 0) + return ret; + if (!ret) + goto out; + bytes =3D ret; + goto split; + } + bio_for_each_bvec(bv, bio, iter) { if (bv.bv_offset & start_align_mask || bv.bv_len & len_align_mask) @@ -376,6 +420,7 @@ int bio_split_io_at(struct bio *bio, const struct queue= _limits *lim, bvprvp =3D &bvprv; } =20 +out: *segs =3D nsegs; bio->bi_bvec_gap_bit =3D ffs(gaps); return 0; diff --git a/block/fops.c b/block/fops.c index b182d0b30748..09fa42f8d8fa 100644 --- a/block/fops.c +++ b/block/fops.c @@ -362,7 +362,7 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *i= ocb, * Users don't rely on the iterator being in any particular * state for async I/O returning -EIOCBQUEUED, hence we can * avoid expensive iov_iter_advance(). Bypass - * bio_iov_iter_get_pages() and set the bvec directly. + * bio_iov_iter_get_pages() and set the bvec/dmabuf directly. */ if (!bio_iov_iter_set(bio, iter)) { ret =3D blkdev_iov_iter_get_pages(bio, iter, bdev); diff --git a/include/linux/bio.h b/include/linux/bio.h index 892ca469c570..7a794ce723b8 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -80,7 +80,8 @@ static inline bool bio_no_advance_iter(const struct bio *= bio) { return bio_op(bio) =3D=3D REQ_OP_DISCARD || bio_op(bio) =3D=3D REQ_OP_SECURE_ERASE || - bio_op(bio) =3D=3D REQ_OP_WRITE_ZEROES; + bio_op(bio) =3D=3D REQ_OP_WRITE_ZEROES || + op_is_dmabuf(bio->bi_opf); } =20 static inline void *bio_data(struct bio *bio) @@ -438,12 +439,12 @@ static inline void bio_wouldblock_error(struct bio *b= io) =20 /* * Calculate number of bvec segments that should be allocated to fit data - * pointed by @iter. If @iter is backed by bvec it's going to be reused - * instead of allocating a new one. + * pointed by @iter. If @iter is backed by a bvec or a dmabuf, the bvec ar= ray / + * the dma map are going to be reused, and so no extra allocation is requi= red. */ static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_seg= s) { - if (iov_iter_is_bvec(iter)) + if (iov_iter_is_bvec(iter) || iov_iter_is_dmabuf_map(iter)) return 0; return iov_iter_npages(iter, max_segs); } diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index af878597afb8..7c7504c84e09 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -1017,6 +1017,13 @@ static inline void *blk_mq_rq_to_pdu(struct request = *rq) return rq + 1; } =20 +static inline bool blk_mq_rq_is_dmabuf(struct request *rq) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return false; + return rq->bio && op_is_dmabuf(rq->bio->bi_opf); +} + static inline struct blk_mq_hw_ctx *queue_hctx(struct request_queue *q, in= t id) { struct blk_mq_hw_ctx *hctx; diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 98e21b4cbf32..0cc09b975d8f 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -233,7 +233,12 @@ struct bio { atomic_t __bi_remaining; =20 /* The actual vec list, preserved by bio_reset() */ - struct bio_vec *bi_io_vec; + union { + struct bio_vec *bi_io_vec; + /* Driver specific dma map, valid IFF REQ_DMABUF is set */ + struct dma_buf_io_map *bi_dmabuf_map; + }; + struct bvec_iter bi_iter; =20 union { @@ -402,6 +407,7 @@ enum req_flag_bits { __REQ_DRV, /* for driver use */ __REQ_FS_PRIVATE, /* for file system (submitter) use */ __REQ_ATOMIC, /* for atomic write operations */ + __REQ_DMABUF, /* Using premmaped dma buffers */ /* * Command specific flags, keep last: */ @@ -434,6 +440,7 @@ enum req_flag_bits { #define REQ_DRV (__force blk_opf_t)(1ULL << __REQ_DRV) #define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE) #define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC) +#define REQ_DMABUF (__force blk_opf_t)(1ULL << __REQ_DMABUF) =20 #define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP) =20 @@ -487,6 +494,11 @@ static inline bool op_is_discard(blk_opf_t op) return (op & REQ_OP_MASK) =3D=3D REQ_OP_DISCARD; } =20 +static inline bool op_is_dmabuf(blk_opf_t op) +{ + return op & REQ_DMABUF; +} + /* * Check if a bio or request operation is a zone management operation. */ diff --git a/include/linux/bvec.h b/include/linux/bvec.h index fc566ee1c1ff..b63914ff56e3 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -108,7 +108,8 @@ struct bvec_iter { unsigned int bi_idx; =20 /* - * Current offset in the bvec entry pointed to by `bi_idx`. + * Current offset in the bvec entry pointed to by `bi_idx` or into + * a dma-buf map. */ unsigned int bi_offset; } __packed __aligned(4); --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0B384A2A74 for ; Mon, 21 Sep 2026 13:39:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997981; cv=none; b=aFdoC9tOF1E3BwPH0i3MIBnRJ7dE2FGdUaSFjeBs9H9ZtgSXe1NOmynVKzah1A+yOrhG7AZHpkGtpz7LNiGRBwl2wxEBp19zLGgQUrXwIC13+TJNBbQcBcW8Dck0QdSjEFXNIYD3SkzqQTQmQvflM+wKTfA1oR4twAQx+d2YOGo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997981; c=relaxed/simple; bh=4botMd0JRxgaa+oDvZMUZvvmXYyk4mBy6faGFHabX+8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=embdUTtFRC6FHW1euFfGshcw13DiVwST/Jzv0qAESo9SrtLNn3VYKlteeBvJDVINRuCWS49EzWX9ip1GuSotRRSYNTPenrzYiv/rYkHw4JqJ8cnTftcyIt/pchQCD9a1WWPEqDdwD2S2RiBqckkd4GTDX8mZIKqqFDXeTGtfoPE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AvtWke4Q; arc=none smtp.client-ip=74.125.225.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AvtWke4Q" Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49b912d3931so22495465e9.3 for ; Mon, 21 Sep 2026 06:39:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997977; x=1790602777; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LfiQtDTMXplRc+HRsAYVyvIssf+oWzUZ0qqIFwpPEw8=; b=AvtWke4QL/CX8yeQjGnGXUT6cnsoMNLqIBwXjteSRY5/9bQPHev7G54OANcwmJx1AB 4nGDMQSYF0qcecwqD6CnHNlluEYCkX0XRx6/pnrXiOcrmNBO+M0k2I5cNYMDZcCBHJ29 hTW4MhkrM9St39NWbtNbza7M+MEZnHfoPgqfYT6l8jZzY+4mcFfeWBkkkds+ZC+aqbR6 zGIAWHsrL/7dgqZoMOZtCRKDerTwe4+VAbavp/NrQvl32oeo5K2Xxq2S+ULOS6S/92Mb ajEM+77gXhDuQOD3ol75VXsDZd+oaIslcCtSVxKcguq3mA1FMVlWvVCHuY3BeAZyaDBo z5mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997977; x=1790602777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LfiQtDTMXplRc+HRsAYVyvIssf+oWzUZ0qqIFwpPEw8=; b=yylf6tVBjcvAikI8Z7i4pUwwAvBoPVxmiU38j4+lW8x/JPpY9DhRzyFXHmo0MiE/Lz uw0bJKHBZ2/feV6PHMwFBh+jqf4no7ymp3XdfPTctxh1KoA/yuUIFYyMzMXcomoGNMYW xk6k7S9Ml6SD51S0HqiQiL8BB82wXmj+48PWwmZb8gy9fTOvRNE7DlcIL6s0efSQz/Ph Ft01K0CVFhizwoJsnzZ+hP76YDvMtyn+2Yq39clPOypTrLq6UtRQ78B43/I014rdn/ki VrnVl8MWHf2XygoN/aEsLDG01bXXEc/EOmX4/3oLhpun3vv8JlRYN+4glKpNbKCD3sIC 54Yg== X-Forwarded-Encrypted: i=1; AKwUvBx1A3pSxI2DBUeB/CgeZzYj3NLTjYs+R0wnUPAXYcwsoi6osYXcBYt3aiFGnkgSqk5rY7VmoXjttMB4SUo=@vger.kernel.org X-Gm-Message-State: AFuF++k6BOoYgpurp1VoufaTgYm7P4LpeHN+Dns2iLmZxBVDH5tfCIQc LEDmPq95wGYI3GZ1Js5ScsoQiEonT8+uGq82BKXbG+awRVVQ8ccy11Gx X-Gm-Gg: AYBFou10159gbpxWsI/kYDOfxAeXr9RUXmQ4/cqcQpMCvaXglwisOKj1f/+1MEgBG8F AFAlTjR6lhB4lX5DWlIj+gASwpJl6oBWb2w6SkR7Iw/2d/CJINwH68uSr5BpIrh86IEkr5x0Mw/ 8ixM1Ef9ONtYn2shEnVeXaSjnQk8ErXUdkTpTZmgxbM6rParsC61h8igPWxxMAZHk2t1M7qTOFY 2PnnrLfiU1tp9N8JO4VI3qqEO3xugR5tQCfcKj/48gpT+Q0hX6A4RXSplbWBsibs0Thz/JJB5i7 FmaXL+Vo/qRSTMJGSF7znm9Vlb9XsQKpLI8BZWUMafKzYkwsuD+FaIGFq1bB+hYxm55JMVGIK+9 4tJaIh5KAlqljh3DwCJdcAwJINDQgXoLieCf+/f7lyGzZcfncQYrm3YEs/u8r4dbZMRSPqN4sMH AjcCr0t5mL8iY9tJZ1YUQXB5Ao+3Q3QD2kd3OuVh4GVt+UP+A8FbNaQI3gNB64IAYUwl2WYkiGY D5TJM/IQMGJQeRVPoozUMf3eFtE187Z9PoEgsNEA+LT4NxH8r7C1sleLGlIhjyeOrw4phVfoDS7 vRQzZKgU9A2hmz5XQhwrH7rMaP2zDOVsB5NthS4Y X-Received: by 2002:a05:600c:c493:b0:49e:74b6:740f with SMTP id 5b1f17b1804b1-49fc573603amr145209165e9.17.1789997977065; Mon, 21 Sep 2026 06:39:37 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:35 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 05/13] block: add dma-buf support for raw bdev Date: Mon, 21 Sep 2026 14:38:49 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a simple proxy implementation of init_dma_buf_io_ctx() forwarding the call to a new struct block_device_operations operation. Also reject dma-buf backed iterators for buffered IO. Reviewed-by: Christoph Hellwig [pavel: reject dma-buf without O_DIRECT] Signed-off-by: Pavel Begunkov --- block/fops.c | 22 +++++++++++++++++++++- include/linux/blkdev.h | 2 ++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/block/fops.c b/block/fops.c index 09fa42f8d8fa..0252d26969a4 100644 --- a/block/fops.c +++ b/block/fops.c @@ -782,7 +782,8 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, st= ruct iov_iter *from) =20 if (iocb->ki_flags & IOCB_DIRECT) { ret =3D blkdev_direct_write(iocb, from); - if (ret >=3D 0 && iov_iter_count(from)) { + if (ret >=3D 0 && iov_iter_count(from) && + !iov_iter_is_dmabuf_map(from)) { ret =3D direct_write_fallback(iocb, from, ret, blkdev_buffered_write(iocb, from)); need_sync =3D true; @@ -795,6 +796,9 @@ static ssize_t blkdev_write_iter(struct kiocb *iocb, st= ruct iov_iter *from) need_sync =3D true; } } else { + if (unlikely(iov_iter_is_dmabuf_map(from))) + return -EOPNOTSUPP; + /* * Take i_rwsem and invalidate_lock to avoid racing with * set_blocksize changing i_blkbits/folio order and punching @@ -850,6 +854,8 @@ static ssize_t blkdev_read_iter(struct kiocb *iocb, str= uct iov_iter *to) if (ret < 0 || !count) goto reexpand; } + if (unlikely(iov_iter_is_dmabuf_map(to))) + return -EOPNOTSUPP; =20 /* * Take i_rwsem and invalidate_lock to avoid racing with set_blocksize @@ -953,6 +959,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc *de= sc) return generic_file_mmap_prepare(desc); } =20 +static int blkdev_init_dma_buf_io_ctx(struct file *file, + struct dma_buf_io_ctx *ctx) +{ + struct block_device *bdev =3D file_bdev(file); + struct gendisk *disk =3D bdev->bd_disk; + + if (!(file->f_flags & O_DIRECT)) + return -EINVAL; + if (!disk->fops->init_dma_buf_io_ctx) + return -EOPNOTSUPP; + return disk->fops->init_dma_buf_io_ctx(bdev, ctx); +} + const struct file_operations def_blk_fops =3D { .open =3D blkdev_open, .release =3D blkdev_release, @@ -971,6 +990,7 @@ const struct file_operations def_blk_fops =3D { .fallocate =3D blkdev_fallocate, .uring_cmd =3D blkdev_uring_cmd, .fop_flags =3D FOP_BUFFER_RASYNC | FOP_DONTCACHE, + .init_dma_buf_io_ctx =3D blkdev_init_dma_buf_io_ctx, }; =20 static __init int blkdev_init(void) diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index d003a9d2d1f6..2d5c30823aab 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -1599,6 +1599,8 @@ struct block_device_operations { /* returns the length of the identifier or a negative errno: */ int (*get_unique_id)(struct gendisk *disk, u8 id[16], enum blk_unique_id id_type); + int (*init_dma_buf_io_ctx)(struct block_device *, + struct dma_buf_io_ctx *); struct module *owner; const struct pr_ops *pr_ops; =20 --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 756F04A3F1C for ; Mon, 21 Sep 2026 13:39:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997985; cv=none; b=QyLZk37l66fLMe4T9HeuvkjQIlJHLv2YmFFCR/Af4BHk3MTl9KOIeHxqZwwibkX9Fg+/8LPv8yK8B811EPfnqc+i/hAKVlap2GXO6Xs9fRWedsGWqON/9TfRYicRZNskUCg8qeXGaql4mRJVnMriSGGpA+3ilqMS+zvKbpUF+ng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997985; c=relaxed/simple; bh=9SPI/n3EIcq8zekl6ra6d1sEPJO99PtcJFr3WLfKF58=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mZ/yhq4Zvuv0Ou3qsLd/jaKVTGm63SYdcluSPVCxghlAUOlqv+TZ8wSBtjkwuqCam+KdwF1eXofSsT5aVBPBKjGRCevdouhz67tC6Pla34GYiSlI1RCYH+BjCGs/QODESnlPzOEylD/32hXp4D71MDHdlXQ7w7NM58KuvUj+1/M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZePgUMjo; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZePgUMjo" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49e71cdb22bso22404395e9.2 for ; Mon, 21 Sep 2026 06:39:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997981; x=1790602781; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=47gGjdY3fuMrT1DeMzW6GgwCiiKM0sHeOAMnZA9asVU=; b=ZePgUMjo3bdLvAMWIRPEFJI5BCePo13Gm8knq3ApU3jddjX9byCA7hV89fPDVyzvfo q2aZ253eucAEsLaMXQwrHHC5fpGckPXcRuNjDpa8nnVlt6HUBkuv62e6muH1Z7U5+/Tx Zwkv3Cm2J1luxKAGLzuQmYFoSjo18mgmiusLCgouNoJPmVHlmSgrTqAJo8tqEvYL8GXl j0OQtGlh0roAl4hwbZZTzwUipTnLTXP1vCK74fHw4Iqq6V6rS56Cq/U6Sn69Pt3z6lkF gT2jECOiRhDR2yuhSmbS7DWYfaDhPGy/ZgNupQdo5uLSmG5f+qpyvD5FA4/drEUOgIdS T4dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997981; x=1790602781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=47gGjdY3fuMrT1DeMzW6GgwCiiKM0sHeOAMnZA9asVU=; b=IWQjpmom313hb1qws6IE5L3PP6QkCJ84YgBtRxfsQYg/7q1PboNjhbgQZfDUM+mrNF VLmkiA0/qf3VQl7r59D87bgh/MaKyC8jSFF+8yQEdn5XNIeFTo16SeRVAeRg216ypt/O v9uioXqWVGW+hIS2+TpaOZKeMgF7/0I8dfsHVttMmD5rQd+KvmhTF++oEATrM9g2j4pv iiUch4te25IUMyQzP8nmtZBofJ3ein2B5LnYUZTM8lV+X6kOkljwdpztxXGBzvJJKlj+ KxVdAAWjoOTcCq/7HfcLbgZqOfWHnBLj3/TzTKGk4rPVT+wuoyDL3BjWYU2h1RwdjcK8 A88Q== X-Forwarded-Encrypted: i=1; AKwUvByzRee4m8anuk7p2V033Gunf4UORjX0sn0KeqKUa79eQ+iuAfvqFWjeMUJVGXws0ZN01zr9jh2iKCIRs7k=@vger.kernel.org X-Gm-Message-State: AFuF++lEBm+aMjsKatIP77ZulbkfHIR7ryV4QMM+odN8z3gMhbb27ELB /FtJ6DtEUsfVT9GB0OwHJ93O20rO2eekvR0ldPANE+k6wA3Kj4mYhenxJGn66w== X-Gm-Gg: AYBFou1Bgrljl6GYD3PDmN42zGxUSrG7mvdHkMxnn6a5i2ZpkhZMEweMt6Rr71ExKYD E38G5TGpnpNF3uilen2Rlqdt1ypB6HyH/hVLbgjg5nIZNe8mdKpot44gQFQRk9pViRY7eMTvuFj mXbToZW0YMDW02RR6FzVJAjMp0Ltj+b3bOn53fBDywjqyv2dgOT6D3o8HboKdXYNi9tfw2HS5cj XYm0wYfL0uD5ZJq+Im0KxnngDzz257wBAcgpbe6h8ZB/tv2LTT5iWFKcxE1PQGpzY1hvKQJNUcr /kcZ2IerEMbPJQd8agpSBxaooHhwozGDOIISqGhHjdL5J4FTcVr/jHGAMxlx/wI5QJpQOz/RArV zTdf0YarKuB0oEdhkSX5ByOTTxwpSEN39J3ut+fFUPz3ZIe8ukLYN1SxhKPxkUxPIShconvFmO0 tdhbjfHnpAraB9lQhyFG0XvdJRLcEk+totGlD66VqMmEKJna9viny007m2C+R2mHYZbgQtuFstZ vXkOvEgRulrq5CFl1ZY2qh5FxXNFrb1YMZOBT75njDet95ar9GJOqJjWoSlE8QVlRoTQhmEZosb vpvIGKUWDEo8tPjaOU9cX2EkStarfA== X-Received: by 2002:a05:600c:6211:b0:49c:fa20:cbfb with SMTP id 5b1f17b1804b1-49fc5728f84mr137193975e9.18.1789997980506; Mon, 21 Sep 2026 06:39:40 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:39 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 06/13] nvme-pci: implement dma-buf backed requests Date: Mon, 21 Sep 2026 14:38:50 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable BIO_DMABUF_MAP backed requests. On registration we map the dma-buf and store it as a prp list, which is then used to initialise requests. All attached contexts are stored in a new list dmabuf_ctxs, and additions/removals are synchronised with dmabuf_lock. Suggested-by: Keith Busch Signed-off-by: Pavel Begunkov --- drivers/nvme/host/core.c | 12 ++ drivers/nvme/host/nvme.h | 2 + drivers/nvme/host/pci.c | 325 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 339 insertions(+) diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index beea23d04a70..d81462e864a4 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -2699,6 +2699,17 @@ static int nvme_report_zones(struct gendisk *disk, s= ector_t sector, #define nvme_report_zones NULL #endif /* CONFIG_BLK_DEV_ZONED */ =20 +static int nvme_init_dma_buf_io_ctx(struct block_device *bdev, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_ns *ns =3D bdev->bd_disk->private_data; + struct nvme_ctrl *ctrl =3D ns->ctrl; + + if (!ctrl->ops->init_dma_buf_io_ctx) + return -EINVAL; + return ctrl->ops->init_dma_buf_io_ctx(ctrl, ctx); +} + const struct block_device_operations nvme_bdev_ops =3D { .owner =3D THIS_MODULE, .ioctl =3D nvme_ioctl, @@ -2709,6 +2720,7 @@ const struct block_device_operations nvme_bdev_ops = =3D { .get_unique_id =3D nvme_get_unique_id, .report_zones =3D nvme_report_zones, .pr_ops =3D &nvme_pr_ops, + .init_dma_buf_io_ctx =3D nvme_init_dma_buf_io_ctx, }; =20 static int nvme_wait_ready(struct nvme_ctrl *ctrl, u32 mask, u32 val, diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index 2cff9fcbf740..befb7faba9ca 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -667,6 +667,8 @@ struct nvme_ctrl_ops { int (*get_address)(struct nvme_ctrl *ctrl, char *buf, int size); void (*print_device_info)(struct nvme_ctrl *ctrl); bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); + int (*init_dma_buf_io_ctx)(struct nvme_ctrl *ctrl, + struct dma_buf_io_ctx *ctx); unsigned long (*get_virt_boundary)(struct nvme_ctrl *ctrl, bool is_admin); }; =20 diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 5440cf18b55b..3d645324f04e 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -27,6 +27,8 @@ #include #include #include +#include +#include =20 #include "trace.h" #include "nvme.h" @@ -318,6 +320,8 @@ struct nvme_dev { bool hmb; struct sg_table *hmb_sgt; mempool_t *dmavec_mempool; + struct list_head dmabuf_ctxs; + struct mutex dmabuf_lock; =20 /* shadow doorbell buffer support: */ __le32 *dbbuf_dbs; @@ -397,6 +401,13 @@ struct nvme_queue { struct completion delete_done; }; =20 +struct nvme_dmabuf_map { + struct dma_buf_io_map base; + struct sg_table *sgt; + unsigned nr_entries; + dma_addr_t dma_list[]; +}; + /* bits for iod->flags */ enum nvme_iod_flags { /* this command has been aborted by the timeout handler */ @@ -865,6 +876,140 @@ static void nvme_free_descriptors(struct request *req) } } =20 +static inline struct nvme_dmabuf_map * +to_nvme_dmabuf_map(struct dma_buf_io_map *map) +{ + return container_of(map, struct nvme_dmabuf_map, base); +} + +static void nvme_dmabuf_map_sync_for_cpu(struct nvme_dev *nvme_dev, + struct request *req) +{ + struct device *dev =3D nvme_dev->dev; + enum dma_data_direction dma_dir; + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + dma_addr_t *dma_list =3D map->dma_list; + unsigned offset =3D bio->bi_iter.bi_offset; + unsigned map_idx =3D offset / NVME_CTRL_PAGE_SIZE; + int length =3D blk_rq_payload_bytes(req) + + (offset & (NVME_CTRL_PAGE_SIZE - 1)); + + dma_dir =3D rq_data_dir(req) =3D=3D READ ? DMA_FROM_DEVICE : DMA_TO_DEVIC= E; + + while (length > 0) { + dma_sync_single_for_cpu(dev, dma_list[map_idx++], + NVME_CTRL_PAGE_SIZE, dma_dir); + length -=3D NVME_CTRL_PAGE_SIZE; + } +} + +static void nvme_dmabuf_map_sync_for_device(struct nvme_dev *nvme_dev, + struct request *req) +{ + struct device *dev =3D nvme_dev->dev; + enum dma_data_direction dma_dir; + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + dma_addr_t *dma_list =3D map->dma_list; + unsigned offset =3D bio->bi_iter.bi_offset; + unsigned map_idx =3D offset / NVME_CTRL_PAGE_SIZE; + int length =3D blk_rq_payload_bytes(req) + + (offset & (NVME_CTRL_PAGE_SIZE - 1)); + + dma_dir =3D rq_data_dir(req) =3D=3D READ ? DMA_FROM_DEVICE : DMA_TO_DEVIC= E; + + while (length > 0) { + dma_sync_single_for_device(dev, dma_list[map_idx++], + NVME_CTRL_PAGE_SIZE, dma_dir); + length -=3D NVME_CTRL_PAGE_SIZE; + } +} + +static void nvme_rq_clean_dmabuf_map(struct nvme_dev *dev, + struct request *req) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + + nvme_dmabuf_map_sync_for_cpu(dev, req); + + if (iod->nr_descriptors) + nvme_free_descriptors(req); +} + +static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req, + struct nvme_queue *nvmeq) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + unsigned bvec_done =3D bio->bi_iter.bi_offset; + unsigned map_idx =3D bvec_done / NVME_CTRL_PAGE_SIZE; + unsigned offset =3D bvec_done & (NVME_CTRL_PAGE_SIZE - 1); + int length =3D blk_rq_payload_bytes(req) - (NVME_CTRL_PAGE_SIZE - offset); + dma_addr_t *dma_list =3D map->dma_list; + u64 prp1_dma =3D dma_list[map_idx++] + offset; + u64 dma_addr, prp2_dma; + dma_addr_t prp_dma; + __le64 *prp_list; + unsigned i; + + nvme_dmabuf_map_sync_for_device(nvmeq->dev, req); + + if (length <=3D 0) { + prp2_dma =3D 0; + goto done; + } + + if (length <=3D NVME_CTRL_PAGE_SIZE) { + prp2_dma =3D dma_list[map_idx]; + goto done; + } + + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <=3D + NVME_SMALL_POOL_SIZE / sizeof(__le64)) + iod->flags |=3D IOD_SMALL_DESCRIPTOR; + + prp_list =3D dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &prp_dma); + if (!prp_list) + return BLK_STS_RESOURCE; + + iod->descriptors[iod->nr_descriptors++] =3D prp_list; + prp2_dma =3D prp_dma; + i =3D 0; + for (;;) { + if (i =3D=3D NVME_CTRL_PAGE_SIZE >> 3) { + __le64 *old_prp_list =3D prp_list; + + prp_list =3D dma_pool_alloc(nvmeq->descriptor_pools.large, + GFP_ATOMIC, &prp_dma); + if (!prp_list) + goto free_prps; + iod->descriptors[iod->nr_descriptors++] =3D prp_list; + prp_list[0] =3D old_prp_list[i - 1]; + old_prp_list[i - 1] =3D cpu_to_le64(prp_dma); + i =3D 1; + } + + dma_addr =3D dma_list[map_idx++]; + prp_list[i++] =3D cpu_to_le64(dma_addr); + + length -=3D NVME_CTRL_PAGE_SIZE; + if (length <=3D 0) + break; + } +done: + iod->cmd.common.dptr.prp1 =3D cpu_to_le64(prp1_dma); + iod->cmd.common.dptr.prp2 =3D cpu_to_le64(prp2_dma); + return BLK_STS_OK; +free_prps: + iod->cmd.common.dptr.prp1 =3D cpu_to_le64(prp1_dma); + iod->cmd.common.dptr.prp2 =3D cpu_to_le64(prp2_dma); + nvme_free_descriptors(req); + return BLK_STS_RESOURCE; +} + static void nvme_free_prps(struct request *req, unsigned int attrs) { struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); @@ -943,6 +1088,11 @@ static void nvme_unmap_data(struct request *req) struct device *dma_dev =3D nvmeq->dev->dev; unsigned int attrs =3D 0; =20 + if (blk_mq_rq_is_dmabuf(req)) { + nvme_rq_clean_dmabuf_map(nvmeq->dev, req); + return; + } + if (iod->flags & IOD_SINGLE_SEGMENT) { static_assert(offsetof(union nvme_data_ptr, prp1) =3D=3D offsetof(union nvme_data_ptr, sgl.addr)); @@ -1257,6 +1407,9 @@ static blk_status_t nvme_map_data(struct request *req) struct blk_dma_iter iter; blk_status_t ret; =20 + if (blk_mq_rq_is_dmabuf(req)) + return nvme_rq_setup_dmabuf_map(req, nvmeq); + /* * Try to skip the DMA iterator for single segment requests, as that * significantly improves performances for small I/O sizes. @@ -2284,6 +2437,170 @@ static int nvme_create_queue(struct nvme_queue *nvm= eq, int qid, bool polled) return result; } =20 +#ifdef CONFIG_DMA_SHARED_BUFFER + +struct nvme_dma_buf_io_ctx { + struct dma_buf_attachment *attach; + struct dma_buf_io_ctx *ctx; + struct nvme_dev *dev; + struct list_head list; +}; + +static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *att= ach) +{ + struct dma_buf_io_ctx *ctx =3D attach->importer_priv; + + dma_buf_io_invalidate_mappings(ctx); +} + +const struct dma_buf_attach_ops nvme_dmabuf_importer_ops =3D { + .invalidate_mappings =3D nvme_dmabuf_invalidate_mappings, + .allow_peer2peer =3D true, +}; + +static struct dma_buf_io_map *nvme_dma_buf_io_map(struct dma_buf_io_ctx *c= tx) +{ + unsigned nr_entries =3D ctx->dmabuf->size / NVME_CTRL_PAGE_SIZE; + struct nvme_dma_buf_io_ctx *nvme_ctx =3D ctx->dev_priv; + struct dma_buf_attachment *attach =3D nvme_ctx->attach; + unsigned long tmp, i =3D 0; + struct nvme_dmabuf_map *map; + struct scatterlist *sg; + struct sg_table *sgt; + int ret; + + dma_resv_assert_held(ctx->dmabuf->resv); + + map =3D kmalloc_flex(*map, dma_list, nr_entries); + if (!map) + return ERR_PTR(-ENOMEM); + + sgt =3D dma_buf_map_attachment(attach, ctx->dir); + if (IS_ERR(sgt)) { + ret =3D PTR_ERR(sgt); + sgt =3D NULL; + goto err; + } + + for_each_sgtable_dma_sg(sgt, sg, tmp) { + dma_addr_t dma_addr =3D sg_dma_address(sg); + unsigned long sg_len =3D sg_dma_len(sg); + + if (sg_len % NVME_CTRL_PAGE_SIZE) { + ret =3D -EINVAL; + goto err; + } + while (sg_len) { + map->dma_list[i++] =3D dma_addr; + dma_addr +=3D NVME_CTRL_PAGE_SIZE; + sg_len -=3D NVME_CTRL_PAGE_SIZE; + } + } + + ret =3D dma_buf_io_init_map(ctx, &map->base, sgt); + if (ret) + goto err; + map->nr_entries =3D nr_entries; + map->sgt =3D sgt; + return &map->base; +err: + if (sgt) + dma_buf_unmap_attachment(attach, sgt, ctx->dir); + kfree(map); + return ERR_PTR(ret); +} + +static void nvme_dma_buf_io_unmap(struct dma_buf_io_ctx *ctx, + struct dma_buf_io_map *map_base) +{ + struct nvme_dma_buf_io_ctx *nvme_ctx =3D ctx->dev_priv; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(map_base); + + dma_resv_assert_held(ctx->dmabuf->resv); + + dma_buf_unmap_attachment(nvme_ctx->attach, map->sgt, ctx->dir); +} + +static void nvme_dma_buf_io_release(struct dma_buf_io_ctx *ctx) +{ + struct nvme_dma_buf_io_ctx *nvme_ctx =3D ctx->dev_priv; + struct nvme_dev *dev =3D nvme_ctx->dev; + + mutex_lock(&dev->dmabuf_lock); + if (!list_empty(&nvme_ctx->list)) { + dma_buf_detach(ctx->dmabuf, nvme_ctx->attach); + list_del_init(&nvme_ctx->list); + } + mutex_unlock(&dev->dmabuf_lock); + + nvme_put_ctrl(&dev->ctrl); + kfree(nvme_ctx); +} + +const struct dma_buf_io_ops nvme_dma_buf_io_ops =3D { + .map =3D nvme_dma_buf_io_map, + .unmap =3D nvme_dma_buf_io_unmap, + .release =3D nvme_dma_buf_io_release, +}; + +static int __nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_dev *dev =3D to_nvme_dev(ctrl); + struct nvme_dma_buf_io_ctx *nvme_ctx; + struct dma_buf_attachment *attach; + + nvme_ctx =3D kzalloc_obj(*nvme_ctx); + if (!nvme_ctx) + return -ENOMEM; + + attach =3D dma_buf_dynamic_attach(ctx->dmabuf, dev->dev, + &nvme_dmabuf_importer_ops, ctx); + if (IS_ERR(attach)) { + kfree(nvme_ctx); + return PTR_ERR(attach); + } + + nvme_get_ctrl(ctrl); + list_add(&nvme_ctx->list, &dev->dmabuf_ctxs); + nvme_ctx->attach =3D attach; + nvme_ctx->ctx =3D ctx; + nvme_ctx->dev =3D dev; + ctx->dev_priv =3D nvme_ctx; + ctx->dev_ops =3D &nvme_dma_buf_io_ops; + return 0; +} + +static int nvme_pci_init_dma_buf_io_ctx(struct nvme_ctrl *ctrl, + struct dma_buf_io_ctx *ctx) +{ + struct nvme_dev *dev =3D to_nvme_dev(ctrl); + int ret; + + mutex_lock(&dev->dmabuf_lock); + ret =3D __nvme_pci_init_dma_buf_io_ctx(ctrl, ctx); + mutex_unlock(&dev->dmabuf_lock); + return ret; +} + +static void nvme_pci_remove_dmabuf(struct nvme_dev *dev) +{ + struct nvme_dma_buf_io_ctx *ctx, *tmp; + + mutex_lock(&dev->dmabuf_lock); + list_for_each_entry_safe(ctx, tmp, &dev->dmabuf_ctxs, list) { + dma_buf_io_detach(ctx->ctx); + dma_buf_detach(ctx->ctx->dmabuf, ctx->attach); + list_del_init(&ctx->list); + } + mutex_unlock(&dev->dmabuf_lock); +} +#else +static void nvme_pci_remove_dmabuf(struct nvme_dev *dev) +{ +} +#endif + static const struct blk_mq_ops nvme_mq_admin_ops =3D { .queue_rq =3D nvme_queue_rq, .complete =3D nvme_pci_complete_rq, @@ -3316,6 +3633,8 @@ static void nvme_dev_disable(struct nvme_dev *dev, bo= ol shutdown) struct pci_dev *pdev =3D to_pci_dev(dev->dev); bool dead; =20 + nvme_pci_remove_dmabuf(dev); + mutex_lock(&dev->shutdown_lock); dead =3D nvme_pci_ctrl_is_dead(dev); if (state =3D=3D NVME_CTRL_LIVE || state =3D=3D NVME_CTRL_RESETTING) { @@ -3579,6 +3898,9 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = =3D { .print_device_info =3D nvme_pci_print_device_info, .supports_pci_p2pdma =3D nvme_pci_supports_pci_p2pdma, .get_virt_boundary =3D nvme_pci_get_virt_boundary, +#ifdef CONFIG_DMA_SHARED_BUFFER + .init_dma_buf_io_ctx =3D nvme_pci_init_dma_buf_io_ctx, +#endif }; =20 static int nvme_dev_map(struct nvme_dev *dev) @@ -3701,6 +4023,8 @@ static struct nvme_dev *nvme_pci_alloc_dev(struct pci= _dev *pdev, return ERR_PTR(-ENOMEM); INIT_WORK(&dev->ctrl.reset_work, nvme_reset_work); mutex_init(&dev->shutdown_lock); + INIT_LIST_HEAD(&dev->dmabuf_ctxs); + mutex_init(&dev->dmabuf_lock); =20 dev->nr_write_queues =3D write_queues; dev->nr_poll_queues =3D poll_queues; @@ -4344,5 +4668,6 @@ MODULE_AUTHOR("Matthew Wilcox = "); MODULE_LICENSE("GPL"); MODULE_VERSION("1.0"); MODULE_DESCRIPTION("NVMe host PCIe transport driver"); +MODULE_IMPORT_NS("DMA_BUF"); module_init(nvme_init); module_exit(nvme_exit); --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92BF74A43E2 for ; Mon, 21 Sep 2026 13:39:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997987; cv=none; b=jeUPtoWMZuJ0+v7peoERqHtyIt3d6S0NpSBgVECQba29N02HBGNIY8iMnu2hIXQpwl8W0PmyDgkfqJ9EsKuy6yhOV0vheGof1yIdkpz2vqVg5RZWVQtdhiQqYonhOa8vMaJDfJtGqMkYgiEObfO+IPkHj5ZuIAnhEtgOzYH6Apk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997987; c=relaxed/simple; bh=smDKQevZZNHOYCALAT7ddU35ycdlQd+jI7KY7MHT/MI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qmmt1eum4eQ+T/szv5s9WWX7xDcoTrfAsVPh4Yd9Er3aEwKmknxSw4LBzfneJvXq9IszllyS1QdZVYJLle9i/cTFrvZJKwxwZcKxwEEA8DjMmlMDMB9xYE8k6u8B72ki/WzL9PQo21bCFjypGNnWlfQ4lASg5jgBKM8J4eDft0E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WjBxfpz+; arc=none smtp.client-ip=74.125.225.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WjBxfpz+" Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49e620fa473so17788205e9.1 for ; Mon, 21 Sep 2026 06:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997984; x=1790602784; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=EHTjFNHsfPkUevAeFTZNd93AhuEIP81I0y/DmV6dihY=; b=WjBxfpz+Z59BT444gfGRH3Mtq36vASFYOARX3FT6YjCghKwWC/pXcTF9wK+Ub3vqBC Sc4dhztOy6cixWLJTdmnW6aAAdKy6TPhMsPXXeutJhvrLzPjLqgtQiZ/zqfnBxCoUEeo EgLHekMf8Whef56EqGaYHtNlXeHKNnxM3wA1LbUHwyWakaZ++biQR//LT7RcYO3npvW9 SrP/iKtyrib9w9UzVAEsPN7zVMbNF6NcA+QrrmokozaB0DMuC94DsYPRf/Krt0l6+z9r nvOpE25tQCq/X/7/Tg+hp4Ccl3nkS9CWkMQgEOeLKAo4Eh3RG9wpVUSStGcSORqd1XWa YPPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997984; x=1790602784; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=EHTjFNHsfPkUevAeFTZNd93AhuEIP81I0y/DmV6dihY=; b=EjrsWElRAx363oNaMwCM5BxTwvwYlrX9ubim9Rsy52xms1vDjVcCOTw8Yz6sG9HGfy v3Cfffr8qwmduu0OBlFdTvXjAmCrpXb5phTiquMI/cqqFedgVOaDvwLMIi2lk0eBlLtb xRlR/jAjElko8KNyu+UmBWjaj5g7fl9XxxBc25S2OC82DEV0lB6irWXFUu6VOYpf5Hfl zvXVExxdkD82GtseWg4HRgjMPTzZr6dMWmrwxfcoH1mTpL4vsb+7XH0Rk1AXoSdQ8J/g aaluj4YLRTcXrCOrcwgRMcad5+cdKxxNO4CY5QXjm2Qqje7RyF4LrDv2Y05b60y58sci WcAw== X-Forwarded-Encrypted: i=1; AKwUvBwe8fiagmfm0sZHhjBJN+NXDjf2bcQ+lvkyxyvikCQ7pHNHRGdUzZGHs7axCR6QLMON0HTwprWrzWXKJa4=@vger.kernel.org X-Gm-Message-State: AFuF++majypC2w24mKXNI+0Kt4wR/qp9dzZjLErpLaunlZP6qKhSoyHK o1hph0ogD5EXtG/oSV5VrxP6abcefDU7QnpYQ3pQl+r0wvIeW49HJSuF X-Gm-Gg: AYBFou0VNAQV0KCdHXahdDGQcZU4tgl0RFdjYS2wNvWSEIvCOB1+Pa7uOxjpLO3NCCa z0D3HXLjKfKFuT2Ll+PnK5EDr8lDNduXb+INNVlJ6l4zl0fgu1pDd2/4gKfBseKdJFBlb2cJMVi IQIdkhjfU7djctCuN40fggBheP5fzgRy9pzp07kc7YlDElGibWMS7AuVBTR+VAET7VZRgvaWft2 kNmxbBLEbZgjbGYUnqRhjJPHxwZ70adhcwsPp24a5VUNuKxUeOPd++7AEEdwdzu97Y/zxDWbl9r Ch062Y521GnmFb5ECcntDy4QblDlmUVFZYHUur8kuAcec4ODgkE3L6uh6FgUHxYZPA8jgho2ssf 0DAZl2dqADf99FMYj93cNf1ASB+LayyNSTOKRUe8IoUtrNaxlVPl0LP28topu+qgp/CG+byhTRy r7IKVAcbfJt6cLycRJmJ6Q+1VASfDVkpVxIJdFaGEshr3rXSvYDQBLrVmPviY1vG51QmtwCxDyY BnJTJNIv898D03szB3vpSo0hgNDcwdDRF1/om7XWGcvTW1Z8wLhG3VPWr8Kt4K5A0AF8+Ph+QjR hJMe++kTT9t5kB8/1/BhIbZP/mrF0g== X-Received: by 2002:a05:600c:4505:b0:49a:a101:4157 with SMTP id 5b1f17b1804b1-49fc56db92fmr154368075e9.7.1789997983642; Mon, 21 Sep 2026 06:39:43 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:43 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 07/13] nvme-pci: rename nvme_pci_sgl_set_data to nvme_pci_dma_iter_set_sgl Date: Mon, 21 Sep 2026 14:38:51 +0100 Message-ID: <114bc2087c7811c849fafbf1392abd7edfb24471.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Anuj Gupta Rename the blk_dma_iter based nvme_pci_sgl_set_data() to nvme_pci_dma_iter_set_sgl(). Suggested-by: Christoph Hellwig Reviewed-by: Christoph Hellwig Signed-off-by: Anuj Gupta Signed-off-by: Pavel Begunkov --- drivers/nvme/host/pci.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 3d645324f04e..8dc4002fa696 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1297,7 +1297,7 @@ static blk_status_t nvme_pci_setup_data_prp(struct re= quest *req, return BLK_STS_IOERR; } =20 -static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge, +static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge, struct blk_dma_iter *iter) { sge->addr =3D cpu_to_le64(iter->addr); @@ -1327,7 +1327,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct re= quest *req, iod->cmd.common.flags =3D NVME_CMD_SGL_METABUF; =20 if (entries =3D=3D 1 || blk_rq_dma_map_coalesce(&iod->dma_state)) { - nvme_pci_sgl_set_data(&iod->cmd.common.dptr.sgl, iter); + nvme_pci_dma_iter_set_sgl(&iod->cmd.common.dptr.sgl, iter); iod->total_len +=3D iter->len; return BLK_STS_OK; } @@ -1349,7 +1349,7 @@ static blk_status_t nvme_pci_setup_data_sgl(struct re= quest *req, iter->status =3D BLK_STS_IOERR; break; } - nvme_pci_sgl_set_data(&sg_list[mapped++], iter); + nvme_pci_dma_iter_set_sgl(&sg_list[mapped++], iter); iod->total_len +=3D iter->len; } while (blk_rq_dma_map_iter_next(req, nvmeq->dev->dev, iter)); =20 @@ -1512,13 +1512,13 @@ static blk_status_t nvme_pci_setup_meta_iter(struct= request *req) iod->cmd.common.metadata =3D cpu_to_le64(sgl_dma); if (entries =3D=3D 1) { iod->meta_total_len =3D iter.len; - nvme_pci_sgl_set_data(sg_list, &iter); + nvme_pci_dma_iter_set_sgl(sg_list, &iter); return BLK_STS_OK; } =20 sgl_dma +=3D sizeof(*sg_list); do { - nvme_pci_sgl_set_data(&sg_list[++i], &iter); + nvme_pci_dma_iter_set_sgl(&sg_list[++i], &iter); iod->meta_total_len +=3D iter.len; } while (blk_rq_integrity_dma_map_iter_next(req, dev->dev, &iter)); =20 --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wr2-f20.google.com (mail-wr2-f20.google.com [74.125.225.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F9994A49B6 for ; Mon, 21 Sep 2026 13:39:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.84 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997991; cv=none; b=QBiES52pVAb7CqBPbCKYix8Qdw//wFfrRTsGtly5ih7djiAk2+J/mcpwbJTFQd6FLdkoGNMBSk/QWIF1tA/R/I0djWBzu17TDcPM0wL9qD3v5huYhnK4T4qkNRNLbziPilc+UpRhdWVjnMyhuCfaeasH90CTVlaQRv9sVPc/sik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997991; c=relaxed/simple; bh=eWGOuzxLl4ww8+RUexIOLldrycXKg0mPt+5jK5IMoA4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Rac6K9mU2IYDf+oY0oMEDXfkP3iG+sSwERLzxk40LIMAymsi9DjrpRimr6XbaUsrzLK2zkKBbQEiCjTRpYLdPuGlcNisCLSSjqZ39k+0rcvjIGIroYKRQ9rWRFCqao27mcMgXXMsy3T7hSFbsiQQgsh8EuH9gmuTQYCOQcSFT0w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gK+XMALd; arc=none smtp.client-ip=74.125.225.84 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gK+XMALd" Received: by mail-wr2-f20.google.com with SMTP id ffacd0b85a97d-482f6356256so994785f8f.1 for ; Mon, 21 Sep 2026 06:39:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997987; x=1790602787; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=mH7BvLrcnLu8WRQAlFxz4vQAXzatlrTdCQIygZg9AG4=; b=gK+XMALddrevx7LCr/o4gyVSrrjoDnmkQraNj2cJtBSYgY8Vq0Ds4bOmrZ4AZLvhGF G59CcB5xGKPtHFNVnJZK1Yt9CSspq7EiPPaUdvQ3rRHIcTNiwzJdhr8xOQo4zr/KRnib 5hJGe068wZgBxLOrfi2k7r59cZ94oRwAhJ50wfWV6TYK6QhW+i3M+Cxb0T2F9jm6K6JE RtDiP+S4IQ1sJaloDkxsUwzaBVLsqUyqbLNh0ZxgGTsSlv925Ylv26dcG3kwfaTHbOS8 x2uFl6mU6+8nMZqI7vB2ORFZAFGaaScOgI1Dr6sbKoutAQV4mE2xz4OZC4NLRrSL7Xus Mzyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997987; x=1790602787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=mH7BvLrcnLu8WRQAlFxz4vQAXzatlrTdCQIygZg9AG4=; b=RiYbdXhIzKAUKbs38bvM5+nx6kkNSgT1z4aiO6+jv6BtECg26GpLDQ4UGkLlY1WUpR GFTEacDLYnPTK8yWuet6Hv5OV4BxKTNVQOYvyBII0CS5ztaH2+llr5BMXAUFCRP9pHZ0 kF6DMxt31Ly2zaLefr8YAw6QhSP2LGJRSm5IBssrW1nNOlhq+WLkkNxvyrxxQYIW2Piv yqzyLjMKi2+DzhBIh/BsRPKpa0gA27Cy9HH/BH1FRnyV/WhCCLypXfvcWLv8ItNmU4u9 BDQVmdAM4ghnrnUKAzz5kJFFVV2Eo1MgNDntnWECRDwC4xY4rBPvs9JeluPZ+pej54lz 8CbQ== X-Forwarded-Encrypted: i=1; AKwUvByrBGtjf3rstuPYvr24FTxZPRqT1OxbJZP05HQiY5VUe2Cd32CyzUWk+AgD2pjxDRVlpkgyaImx601sths=@vger.kernel.org X-Gm-Message-State: AFuF++nQ76VkBS1sUv6loBG+zSwgzLXEEyB1N1MhA0NnvUb1rmH5T2Z8 MPazMp11AUHH79mdEwm2cfIJsSWriAUdNXbOGv2o9JHMPPd3EPjXCYRy X-Gm-Gg: AYBFou2vZDhBNwBR1T5Aq55EIUYOY5w8c80kJ320jAWj4Ria4Lb+mqF5V9xrTxFcV+y P6dSjqX3N70FeBBntOkz1eqFyZAaZS650L4TGlEHZ6TNNZnXDM9W5n1rSqRXKZFKF/JlR2gUhRL zuAsA12QUDw3VuZ/GP783BQNcf7DZ4RkxCV8jHVZTxk+ueRIhs2H0RFgr4dbY+qdV+ykEXGW/Qc Lnh0J9uy6KMsGIH4fc3zDlW4MsZnkMm/8vja7kczjnB5ssJi9CT+VCFq1y9kYBYF0qmphEo2qjr X+rifHJQ4Qy27QesM0agqte57OvqRqGfiTn9VhT7ldUoXpOVQMN4l3don4jrkH9em6fQ0b56k7p utv8kJT8vrZTbDbxY6t1HguMQFNhai5psst8ogKNUAMdrWQvURfNl7z+B4pH+NOAqPIrpNIClO2 IAcAOzoDzDi47/fXF8e/k8WSkP6ktJFCG050/mh7xyUS/ZsnTmok6kifev2Qk/v75DIQtsdnl9o mZX6ByTDX6hdUqfjpb4omcwJZk/coTvMeBAsS5jCGdOmZ5BRaDr5BTaXifejhtWYipW1YdYvpKs AStER0U2Lx97bumtdoMAUHtvH2WUsg== X-Received: by 2002:a05:600c:1d06:b0:49e:8222:3451 with SMTP id 5b1f17b1804b1-49fc5001358mr167903285e9.14.1789997987144; Mon, 21 Sep 2026 06:39:47 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:46 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 08/13] nvme-pci: add SGL support for the dmabuf path Date: Mon, 21 Sep 2026 14:38:52 +0100 Message-ID: <1a50e7e88263f97403f8df8583dbb1953600f2a9.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Anuj Gupta Add SGL support in addition to PRP for dmabuf-backed requests, building the descriptor list from the mapping's sg_table. Reviewed-by: Christoph Hellwig Signed-off-by: Anuj Gupta Signed-off-by: Pavel Begunkov --- drivers/nvme/host/pci.c | 153 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 149 insertions(+), 4 deletions(-) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 8dc4002fa696..4756f114154c 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -1297,12 +1297,18 @@ static blk_status_t nvme_pci_setup_data_prp(struct = request *req, return BLK_STS_IOERR; } =20 +static void nvme_pci_sgl_set_data(struct nvme_sgl_desc *sge, + dma_addr_t addr, u32 len) +{ + sge->addr =3D cpu_to_le64(addr); + sge->length =3D cpu_to_le32(len); + sge->type =3D NVME_SGL_FMT_DATA_DESC << 4; +} + static void nvme_pci_dma_iter_set_sgl(struct nvme_sgl_desc *sge, struct blk_dma_iter *iter) { - sge->addr =3D cpu_to_le64(iter->addr); - sge->length =3D cpu_to_le32(iter->len); - sge->type =3D NVME_SGL_FMT_DATA_DESC << 4; + nvme_pci_sgl_set_data(sge, iter->addr, iter->len); } =20 static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge, @@ -1313,6 +1319,145 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_de= sc *sge, sge->type =3D NVME_SGL_FMT_LAST_SEG_DESC << 4; } =20 +static unsigned int nvme_pci_dmabuf_sgl_nents(struct request *req) +{ + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + struct scatterlist *sg; + unsigned long tmp; + size_t offset =3D bio->bi_iter.bi_offset; + size_t remaining =3D blk_rq_payload_bytes(req); + unsigned int nents =3D 0; + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len =3D sg_dma_len(sg); + + if (!remaining) + break; + if (offset >=3D sg_len) { + offset -=3D sg_len; + continue; + } + + sg_len -=3D offset; + offset =3D 0; + + do { + size_t chunk =3D min(remaining, sg_len); + + nents++; + sg_len -=3D chunk; + remaining -=3D chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + return 0; + + return nents; +} + +static blk_status_t nvme_rq_setup_dmabuf_sgl(struct request *req, + struct nvme_queue *nvmeq, unsigned int entries) +{ + struct nvme_iod *iod =3D blk_mq_rq_to_pdu(req); + struct bio *bio =3D req->bio; + struct nvme_dmabuf_map *map =3D to_nvme_dmabuf_map(bio->bi_dmabuf_map); + size_t length =3D blk_rq_payload_bytes(req); + struct nvme_sgl_desc *sg_list =3D &iod->cmd.common.dptr.sgl; + bool pooled =3D entries > 1; + dma_addr_t sgl_dma =3D 0; + unsigned int mapped =3D 0; + unsigned long tmp; + struct scatterlist *sg; + size_t offset =3D bio->bi_iter.bi_offset; + size_t remaining =3D length; + + if (!entries) + return BLK_STS_IOERR; + + iod->cmd.common.flags =3D NVME_CMD_SGL_METABUF; + iod->total_len =3D length; + + nvme_dmabuf_map_sync_for_device(nvmeq->dev, req); + + /* entries =3D=3D 1 fits in the inline descriptor; more needs a pool. */ + if (pooled) { + if (entries <=3D NVME_SMALL_POOL_SIZE / sizeof(*sg_list)) + iod->flags |=3D IOD_SMALL_DESCRIPTOR; + + sg_list =3D dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC, + &sgl_dma); + if (!sg_list) + return BLK_STS_RESOURCE; + iod->descriptors[iod->nr_descriptors++] =3D sg_list; + } + + for_each_sgtable_dma_sg(map->sgt, sg, tmp) { + size_t sg_len =3D sg_dma_len(sg); + dma_addr_t addr =3D sg_dma_address(sg); + + if (!remaining) + break; + if (offset >=3D sg_len) { + offset -=3D sg_len; + continue; + } + + addr +=3D offset; + sg_len -=3D offset; + offset =3D 0; + + do { + u32 chunk =3D min_t(size_t, remaining, sg_len); + + if (WARN_ON_ONCE(mapped =3D=3D entries)) + goto err_free; + nvme_pci_sgl_set_data(&sg_list[mapped++], addr, chunk); + + addr +=3D chunk; + sg_len -=3D chunk; + remaining -=3D chunk; + } while (sg_len && remaining); + } + + if (unlikely(remaining)) + goto err_free; + + if (pooled) + nvme_pci_sgl_set_seg(&iod->cmd.common.dptr.sgl, sgl_dma, + mapped); + return BLK_STS_OK; +err_free: + if (pooled) { + iod->nr_descriptors--; + dma_pool_free(nvme_dma_pool(nvmeq, iod), sg_list, sgl_dma); + } + return BLK_STS_IOERR; +} + +static blk_status_t nvme_rq_setup_dmabuf(struct request *req, + struct nvme_queue *nvmeq, enum nvme_use_sgl use_sgl) +{ + unsigned int entries; + size_t avg_seg; + + if (use_sgl =3D=3D SGL_UNSUPPORTED) + return nvme_rq_setup_dmabuf_map(req, nvmeq); + + entries =3D nvme_pci_dmabuf_sgl_nents(req); + + if (use_sgl =3D=3D SGL_FORCED) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + avg_seg =3D entries ? + DIV_ROUND_UP(blk_rq_payload_bytes(req), entries) : 0; + if (sgl_threshold && avg_seg >=3D sgl_threshold) + return nvme_rq_setup_dmabuf_sgl(req, nvmeq, entries); + + return nvme_rq_setup_dmabuf_map(req, nvmeq); +} + static blk_status_t nvme_pci_setup_data_sgl(struct request *req, struct blk_dma_iter *iter) { @@ -1408,7 +1553,7 @@ static blk_status_t nvme_map_data(struct request *req) blk_status_t ret; =20 if (blk_mq_rq_is_dmabuf(req)) - return nvme_rq_setup_dmabuf_map(req, nvmeq); + return nvme_rq_setup_dmabuf(req, nvmeq, use_sgl); =20 /* * Try to skip the DMA iterator for single segment requests, as that --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 348FA4A4EF7 for ; Mon, 21 Sep 2026 13:39:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997996; cv=none; b=EvcYuDywntsHE+9mWKcSqkYhkFV6k5Sxqm5t65fJ/qaTtpICqXkTKTKcXsfW6EIxhshtvc+pIR/5WBYb54eSB4BuFO/OMsVv3Rw/iPJvJlDkUqaiDMz7XQgQFFyaO/FOtlIPyrAMGEVkglZ0wAFwqfg5XfHLIk5q5ZVEOw1jXrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997996; c=relaxed/simple; bh=Z4vOxks9qTIFK+TCf+DcsECaWrDahRJWHvnK/Oy0YsE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ou+1wQ6DNvpNzSBoMkDJOWxmYj+BwiBS9K193x7qcFAbKOwzXfUbAgO2OQzrO1Lj1m0BL+Tx5ewVot/bLxNignfRNHIpLgfmqPhStODPDRCxioY7XLQGP6xNEkNkJJTkz8GiVJuFK6Q/mTf4GSpk6lJ6MYmUcwemUzRio9CFLiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fcYiafm8; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fcYiafm8" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49ccfd61ecaso26283455e9.3 for ; Mon, 21 Sep 2026 06:39:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997991; x=1790602791; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/RT09IM6iAPFk2VI7P8d4NpW1v04uxG67FoGdSGxGhI=; b=fcYiafm8WU+PsHRjH0kBI4wrr69zkXEz7/dHrIMdS1Z/IFxyLYqxbvjL/7VyDJi8Ld Jpa6Vc6CfYPjuicYz74Ga+Dh7Vw+gpR24aYwywaCtE3A+VObPrOoZ569w6NQghrXi81l hCzCS9HVe9HBEa1PYdxPSP2GcnXEYJYpB9W1KeLKNW2IqMxw25mx5mD1Negm+gU82j6P Bwr1NPRIt5MNZn/ynUqrtU0IVtWykr8mO1MM++j+r/IQGxxuUGg+vYgADzl6XI4GjeZ2 bU0fPWvVUZ+5faGWAp2h219j1GTQ518m4kyuyFj3qUdctv2uFuJsniJ676LO29k4Vvr6 8wRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997991; x=1790602791; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/RT09IM6iAPFk2VI7P8d4NpW1v04uxG67FoGdSGxGhI=; b=jAxicmVNsk3aS4fNQo67RbwGRCN8VHkWZ2NExbiFl0HSJMkQve4miC9BGIz9FpOhk1 YkhE07R5iIAszfmmBSlJ8cDhNsKobOp6x8qH7ECNgaAjEXHJW5Q5M8/t5RhypQcuCUFR JvlGcPJnrnYPlqiwSI/1E1lOpJgKUhJP8Lov/KXEX4D6mtBzHRW0JLZN8ILSDgAaI/S/ EQuGik0bn3nWAZDaiM/EHUrSvqxnu0vjxzNAE3yvFkIrtcvwo7UEeqwfu4GfNuNku1Mu YYAEn0Rqz4WL2InLNgx9XgGcAQnLwQ2jCDuoFLU1XMvkSSA3LEvurXfaRhHudVy9Fn5S vnwg== X-Forwarded-Encrypted: i=1; AKwUvBxAG0t1l0lo+3Ap3uKqfwi8DSkaCPhbMyI7BWTfFArbKPZslXYogSSeMpPrYCa4YW2uGW8z5lxErSzdeT4=@vger.kernel.org X-Gm-Message-State: AFuF++l8PGNLYGV53PUFeTE7o8C3jhuTNyQ47aKguBKugVNfu51W9Tu/ gVmcpsEduw87RsVywVbdN680lCSnf25B66Re0n39LIkbGP/5uC9MgweJ X-Gm-Gg: AYBFou2lPQ9CWFDalH6aHxOZFwgbQ6stAi3ZG4jt8+HcsaFbosdclKKsZe8kMfHBsHr ZQnv4vruz+ln8EXHNC2Uz+sSLYhsRnFzoyFGbHGkNkmbX1YNqhNKb98gtnE+LDQwSv0LRJco7zp Ds1Zn+RzUdkr/Crpv0N+wAslz5WmtVLv/gupVIDgU12R3WqJuJTZXow9D2ZMzGeHCP8tOvAC+KU /OHLSYm7GeGzUpjzHjpQ+OTHsXF5UNMdBpYDq+R+A6cwvhYzwigOT/WH5EVo+CAL6blo7KQVwrE e8eXGd69Oy1wUCS4TlD9i5kmPQCRvqN56f/JdsEJ5eDyqfryLPKS5x1L1C6nP2Fhynrb4r6iXYr L06rHFdkZRlCNP8gMdAM3R82dqs8DktrtP82IHV2PUyVaQdvSme1+QmYNRX8qfb0i260EBUEXTU nJ6ALnN2duYi0md3vyCsQz7HCvlaFgsF5kuO+HgnulNNbO4NZimcGOtUGn54FnoS9PHYhdyW3I3 6l7fThiVGIKBbL2ZEAOxHTQhNigkCR4vUXv4J099J18lQuYDfkqZl7G2DrL0gQC5IrvYld7CAc/ LuW7SV6ps4uAm97SitwqJdzxTPVIDQ== X-Received: by 2002:a05:600c:e558:20b0:49f:ce73:5e97 with SMTP id 5b1f17b1804b1-49fce735f5bmr54726715e9.35.1789997991415; Mon, 21 Sep 2026 06:39:51 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:49 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 09/13] io_uring/rsrc: introduce buf registration structure Date: Mon, 21 Sep 2026 14:38:53 +0100 Message-ID: <13677c50b853ec0518528ad03ba06012342f1316.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation to following changes, instead of passing an iovec for buffer registration introduce a new structure. It'll be moved to uapi later, but for now it's initialised early from a user provided iovec. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 47 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 51b46e624ddd..da767b34eb55 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,8 +27,13 @@ struct io_rsrc_update { u32 offset; }; =20 +struct io_uring_regbuf_desc { + __u64 uaddr; + __u64 size; +}; + static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, - struct iovec *iov); + struct io_uring_regbuf_desc *desc); =20 static int hpage_acct_ref(struct io_ring_ctx *ctx, struct page *hpage, bool *acct_new) @@ -81,6 +86,15 @@ static bool hpage_acct_unref(struct io_ring_ctx *ctx, st= ruct page *hpage) =20 #define IO_CACHED_BVECS_SEGS 32 =20 +static void io_iov_to_regbuf_desc(const struct iovec *iov, + struct io_uring_regbuf_desc *desc) +{ + *desc =3D (struct io_uring_regbuf_desc) { + .uaddr =3D (u64)(uintptr_t)iov->iov_base, + .size =3D iov->iov_len, + }; +} + int __io_account_mem(struct user_struct *user, unsigned long nr_pages) { unsigned long page_limit, cur_pages, new_pages; @@ -381,6 +395,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 @@ -394,7 +409,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, err =3D -EFAULT; break; } - node =3D io_sqe_buffer_register(ctx, iov); + + io_iov_to_regbuf_desc(iov, &desc); + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { err =3D PTR_ERR(node); break; @@ -853,26 +870,26 @@ bool io_check_coalesce_buffer(struct page **page_arra= y, int nr_pages, } =20 static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, - struct iovec *iov) + struct io_uring_regbuf_desc *desc) { + unsigned long uaddr =3D (unsigned long)desc->uaddr; + size_t size =3D desc->size; struct io_mapped_ubuf *imu =3D NULL; struct page **pages =3D NULL; struct io_rsrc_node *node; unsigned long off; - size_t size; int ret, nr_pages, i; struct io_imu_folio_data data; bool coalesced =3D false; =20 - if (!iov->iov_base) { - if (iov->iov_len) + if (!uaddr) { + if (size) return ERR_PTR(-EFAULT); /* remove the buffer without installing a new one */ return NULL; } =20 - ret =3D io_validate_user_buf_range((unsigned long)iov->iov_base, - iov->iov_len); + ret =3D io_validate_user_buf_range(uaddr, size); if (ret) return ERR_PTR(ret); =20 @@ -881,8 +898,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(stru= ct io_ring_ctx *ctx, return ERR_PTR(-ENOMEM); =20 ret =3D -ENOMEM; - pages =3D io_pin_pages((unsigned long) iov->iov_base, iov->iov_len, - &nr_pages); + pages =3D io_pin_pages(uaddr, size, &nr_pages); if (IS_ERR(pages)) { ret =3D PTR_ERR(pages); pages =3D NULL; @@ -904,10 +920,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, if (ret) goto done; =20 - size =3D iov->iov_len; /* store original address for later verification */ - imu->ubuf =3D (unsigned long) iov->iov_base; - imu->len =3D iov->iov_len; + imu->ubuf =3D uaddr; + imu->len =3D size; imu->folio_shift =3D PAGE_SHIFT; imu->release =3D io_release_ubuf; imu->priv =3D imu; @@ -917,7 +932,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(stru= ct io_ring_ctx *ctx, imu->folio_shift =3D data.folio_shift; refcount_set(&imu->refs, 1); =20 - off =3D (unsigned long)iov->iov_base & ~PAGE_MASK; + off =3D uaddr & ~PAGE_MASK; if (coalesced) off +=3D data.first_folio_page_idx << PAGE_SHIFT; =20 @@ -969,6 +984,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, vo= id __user *arg, memset(iov, 0, sizeof(*iov)); =20 for (i =3D 0; i < nr_args; i++) { + struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 @@ -992,7 +1008,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, v= oid __user *arg, } } =20 - node =3D io_sqe_buffer_register(ctx, iov); + io_iov_to_regbuf_desc(iov, &desc); + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { ret =3D PTR_ERR(node); break; --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C1C8A4A64F4 for ; Mon, 21 Sep 2026 13:39:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997999; cv=none; b=rx+YHG24zxQHx6xtbf96rjPj4d0LgklT6a5g6Pfkj+t9M/9t5VgVHwI5cpuE6wBsHnBDjppdtwBKE1FxTKWMRCB3ds97tPv7yddDpOQS3zjOJXniP4PShthj8+V3TLOw+1eaxXNnuoT/pzOltuMA/Oh6rHCe9QRMLiWXR2zn3Bk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789997999; c=relaxed/simple; bh=FAuNY7f+2IN9h+0fEp3doBi1UJ2umEK7JqOTw5NHGgE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XgaxsjE/D8crWOYFTgwaXJyhHmK/CMUOgbtdTtTNnVsvB1PhV3mwU7zHkFsszCo9/cZ+1obJnA1h/k26K9mb7i7MR9SW3LUkWzEDIy8IKw4pn2GDDNybcbBbRhMxpydWkByg7OSOcOhyZtZ7Zv8WwJ075xajcAIW6+GzAEJeYqo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Keh0fKHa; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Keh0fKHa" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49e79a408deso16965715e9.2 for ; Mon, 21 Sep 2026 06:39:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997995; x=1790602795; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=FutSyws4EmX9/QkptnDh8TTYgLLGT7M1LNenj3p2Roc=; b=Keh0fKHaMyar4vCS+5xjzHGUiEWWO9NXVY+9gcea4IWUKxAcY/n9H6E2J9kzCi6stL Icd9ZHvK5Hu/7Ine5phjVDuKIyqJ9xudPb0cUbYNxdR2d4aNYFrTgsL3DjWZWxnninKq 5y7eV2+KupNbXrsaoPbS8vFfySoylJcS/0KLvgjo8YHwf1RPSCTk3YtVpp+laeVfg1cU gMGYztxpKbeMr1u2KsG2RlkyaqS1+5cFQhjrxwA36EbJKE/ATQSrBFek+xc3wLVST3I/ igpYEBgGtEG4JYq81lqIOzjnFcObs0yReSs5++RXP5SAmflGGISI3ENtCqbDIZo3ZtoR tliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997995; x=1790602795; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=FutSyws4EmX9/QkptnDh8TTYgLLGT7M1LNenj3p2Roc=; b=e2oGirq+pJu/7E9v+SKcXQvXS1lrIS+G8vAxuv/8lWqRl5qVXZ2gQlvLZLIcaV86lI Q+iAQy4M2ncRziMLTJHKMQJHi3cSjmTeMy6SyLqY3J+4j5+PY/XJiwvaxQtsISPtojOg Z8ZrMxR8nAtzxCd+pKUUH+AQI1oLezGjvk1T0CClkklDDipiNACPZ140ABDiHWEdNYF8 cMMLI6Y60JIEB4SMsKdjyLxsrOdhVTQFWhNyZR/r5vUEIJYYpUupKj9SMflXQ1zz9DXX cuAYA/sgj/GSnX3TiPfG8RDsoY7XPYe6jlrvXAHIKESSFPtYj/Z0hOiC/p/2vanaQd4J NCKQ== X-Forwarded-Encrypted: i=1; AKwUvBwCnGxGtkkrZfJvdRgE7wLPXwig5/9SlqS+rE/jd1Il2UpZLGGBIWR3XA0jTmDwNQQszJiWyU6JmcdnL6E=@vger.kernel.org X-Gm-Message-State: AFuF++mXa6K73r6UHeRoisAojyAfco40M4fYVdsBF+K8/sABWe0V1nCP BLrSpTSYuU+1v3EPPrrNhIFJgOyxNlYvUL2/nthLVSff+UinItisWD0w X-Gm-Gg: AYBFou0P8rzeRfB+0rEZ9kLbXbK2MGAGlsHm4udPjdDoZJ4dtwopR83os8IrdREwI10 79q2XSPYYK5xwGMmJ/kxCTfwqyv62LKrf+3itV02Td2bhUZKgFzNlOB2ashpLgMrQJYMkm3iOTK MJM7+fsDmqQZdcMSHS8x97zV1iA6zab7X+aZ0eYf2lLOYuZgBj0PLXcbP9tbb++UO8dFGsVnMNE GP6xF+6I6yKuikBFKSm7OsEcl8Je77JFogoR/58h4XmZekh/nNbwM5UeIXIYtkZbz2WPDjfqf4n 7Wiw88lpR8TQqYTEE3CbgF7/dcb+yZoB5zBEMBdRnAf0cQUYG2xbC3c7Ng1/R0mz4RFsPP/KpZf zT2CB8S8henc0Xm5lOM4NjmayPwTxOSE9pOBXD+S3n3HQjJMLikWqqkeF5xoMHm+T4NtGbvXauK Wd+i5EZB4ihcXCn9Jrq08xsuM+a4hZIFOt/dY1YrTRUngFzM7OVt6ggL9mZO1KclTk+Sg/rAxqw IKOJERGVtPcpoCA28AIF8M11X6qRCCDXKYKHw1cnF5BrP7I34EfYiDOa1U7U6ZDuL4uU79CXuAb RLMTNr8y5E5BXIXWt9qIrMk+Z3K7/fw= X-Received: by 2002:a05:600c:a013:b0:49e:6891:27a8 with SMTP id 5b1f17b1804b1-49fc574a9bcmr133446705e9.25.1789997994878; Mon, 21 Sep 2026 06:39:54 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:54 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 10/13] io_uring/rsrc: extend buffer update Date: Mon, 21 Sep 2026 14:38:54 +0100 Message-ID: <947ff623f7e6844af4435f258ad939fa2192c0e4.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We need to pass more information to buffer registration than we can fit into a single struct iovec. This patch allows users to optionally pass struct io_uring_regbuf_desc. Apart from having more space for future use cases, it also introduces registration types. Currently, the type can be either of IO_REGBUF_TYPE_UADDR, which mirrors the iovec path, or IO_REGBUF_TYPE_EMPTY for leaving a buffer table slot empty. The next patch introduces a dmabuf backed type, and can be useful for other extensions like splicing a list of user addresses (i.e. iovec[]), interoperability with zcrx, kernel allocated memory like was brough up by Cristoph. Note, the type only represents a registration option, which is distinct from how io_uring internally stores it. The flags field is not used yet but always useful to have, e.g. we can encode read-only / write-only restrictions using it. Signed-off-by: Pavel Begunkov --- include/uapi/linux/io_uring.h | 27 +++++++++++++- io_uring/rsrc.c | 69 ++++++++++++++++++++++------------- 2 files changed, 69 insertions(+), 27 deletions(-) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 909fb7aea638..98b259901185 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -790,13 +790,38 @@ struct io_uring_rsrc_update { =20 struct io_uring_rsrc_update2 { __u32 offset; - __u32 resv; + __u32 flags; __aligned_u64 data; __aligned_u64 tags; __u32 nr; __u32 resv2; }; =20 +/* struct io_uring_rsrc_update2::flags */ +enum io_uring_rsrc_reg_flags { + /* + * Use the extended descriptor format for buffer updates, + * see struct io_uring_regbuf_desc + */ + IORING_RSRC_UPDATE_EXTENDED =3D 1U << 1, +}; + +/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */ +enum io_uring_regbuf_type { + IO_REGBUF_TYPE_EMPTY, + IO_REGBUF_TYPE_UADDR, + + __IO_REGBUF_TYPE_MAX, +}; + +struct io_uring_regbuf_desc { + __u32 type; /* enum io_uring_regbuf_type */ + __u32 flags; + __u64 size; + __u64 uaddr; + __u64 __resv[7]; +}; + /* Skip updating fd indexes set to this value in the fd table */ #define IORING_REGISTER_FILES_SKIP (-2) =20 diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index da767b34eb55..83a7ad9f5bc7 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -27,11 +27,6 @@ struct io_rsrc_update { u32 offset; }; =20 -struct io_uring_regbuf_desc { - __u64 uaddr; - __u64 size; -}; - static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_uring_regbuf_desc *desc); =20 @@ -90,9 +85,12 @@ static void io_iov_to_regbuf_desc(const struct iovec *io= v, struct io_uring_regbuf_desc *desc) { *desc =3D (struct io_uring_regbuf_desc) { + .type =3D IO_REGBUF_TYPE_UADDR, .uaddr =3D (u64)(uintptr_t)iov->iov_base, .size =3D iov->iov_len, }; + if (!desc->uaddr) + desc->type =3D IO_REGBUF_TYPE_EMPTY; } =20 int __io_account_mem(struct user_struct *user, unsigned long nr_pages) @@ -323,6 +321,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ct= x, return -ENXIO; if (up->offset + nr_args > ctx->file_table.data.nr) return -EINVAL; + if (up->flags) + return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { u64 tag =3D 0; @@ -382,9 +382,8 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *= ctx, struct io_uring_rsrc_update2 *up, unsigned int nr_args) { + bool extended =3D up->flags & IORING_RSRC_UPDATE_EXTENDED; u64 __user *tags =3D u64_to_user_ptr(up->tags); - struct iovec fast_iov, *iov; - struct iovec __user *uvec; u64 user_data =3D up->data; __u32 done; int i, err; @@ -393,29 +392,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx= *ctx, return -ENXIO; if (up->offset + nr_args > ctx->buf_table.nr) return -EINVAL; + if (up->flags & ~IORING_RSRC_UPDATE_EXTENDED) + return -EINVAL; =20 for (done =3D 0; done < nr_args; done++) { struct io_uring_regbuf_desc desc; struct io_rsrc_node *node; u64 tag =3D 0; =20 - uvec =3D u64_to_user_ptr(user_data); - iov =3D iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); - if (IS_ERR(iov)) { - err =3D PTR_ERR(iov); - break; - } if (tags && copy_from_user(&tag, &tags[done], sizeof(tag))) { err =3D -EFAULT; break; } =20 - io_iov_to_regbuf_desc(iov, &desc); + if (extended) { + if (copy_from_user(&desc, u64_to_user_ptr(user_data), + sizeof(desc))) { + err =3D -EFAULT; + break; + } + user_data +=3D sizeof(desc); + } else { + struct iovec __user *uvec =3D u64_to_user_ptr(user_data); + struct iovec fast_iov, *iov; + + if (io_is_compat(ctx)) + user_data +=3D sizeof(struct compat_iovec); + else + user_data +=3D sizeof(struct iovec); + + iov =3D iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx)); + if (IS_ERR(iov)) { + err =3D PTR_ERR(iov); + break; + } + io_iov_to_regbuf_desc(iov, &desc); + } + node =3D io_sqe_buffer_register(ctx, &desc); if (IS_ERR(node)) { err =3D PTR_ERR(node); break; } + if (tag) { if (!node) { err =3D -EINVAL; @@ -426,10 +445,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx = *ctx, i =3D array_index_nospec(up->offset + done, ctx->buf_table.nr); io_reset_rsrc_node(ctx, &ctx->buf_table, i); ctx->buf_table.nodes[i] =3D node; - if (io_is_compat(ctx)) - user_data +=3D sizeof(struct compat_iovec); - else - user_data +=3D sizeof(struct iovec); } return done ? done : err; } @@ -464,7 +479,7 @@ int io_register_files_update(struct io_ring_ctx *ctx, v= oid __user *arg, memset(&up, 0, sizeof(up)); if (copy_from_user(&up, arg, sizeof(struct io_uring_rsrc_update))) return -EFAULT; - if (up.resv || up.resv2) + if (up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, IORING_RSRC_FILE, &up, nr_args); } @@ -478,7 +493,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, vo= id __user *arg, return -EINVAL; if (copy_from_user(&up, arg, sizeof(up))) return -EFAULT; - if (!up.nr || up.resv || up.resv2) + if (!up.nr || up.resv2) return -EINVAL; return __io_register_rsrc_update(ctx, type, &up, up.nr); } @@ -578,12 +593,9 @@ int io_files_update(struct io_kiocb *req, unsigned int= issue_flags) struct io_uring_rsrc_update2 up2; int ret; =20 + memset(&up2, 0, sizeof(up2)); up2.offset =3D up->offset; up2.data =3D up->arg; - up2.nr =3D 0; - up2.tags =3D 0; - up2.resv =3D 0; - up2.resv2 =3D 0; =20 if (up->offset =3D=3D IORING_FILE_INDEX_ALLOC) { ret =3D io_files_update_with_index_alloc(req, issue_flags); @@ -882,8 +894,13 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, struct io_imu_folio_data data; bool coalesced =3D false; =20 - if (!uaddr) { - if (size) + if (desc->type >=3D __IO_REGBUF_TYPE_MAX) + return ERR_PTR(-EINVAL); + if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) || desc->flags) + return ERR_PTR(-EINVAL); + + if (desc->type =3D=3D IO_REGBUF_TYPE_EMPTY) { + if (uaddr || size) return ERR_PTR(-EFAULT); /* remove the buffer without installing a new one */ return NULL; --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f13.google.com (mail-wm2-f13.google.com [74.125.225.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08F754A689B for ; Mon, 21 Sep 2026 13:39:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.141 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998002; cv=none; b=rl6tdIk4e495Eb3b/FIQ8YEetjtLhO18PkLHSu5U53dbv2I9QeDow1ZfvFt1FU7yDDnQd1v1vaFN/LcJigVp7BxAjuTjsflBFPSP8Da3jGHfyLz3/nlcqJc4jpwZWLHEs2obiEGu4CejEL66lTrdPRvji95ZK8iVmjFVa+aK+R0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998002; c=relaxed/simple; bh=iU1u1X7cruHQchsBcaFSFlP5wT6XT+YL719bzSe6aAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E6bkKzl6iIrZsapoUag0BJzGxgKBtfEOpnm/xFp0zybrQe442NbiDlFHRaaIEFHYi2CwRcZepPeM12nDG2w70oEISBHrbUFP+UJyQcWuDGUyCoKE74IZMmsDeHmjoarAJjbCem7QZ4E1Xha0D9WWchLfVL0bRzZJStcYPqradrk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=elBFyK9R; arc=none smtp.client-ip=74.125.225.141 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="elBFyK9R" Received: by mail-wm2-f13.google.com with SMTP id 5b1f17b1804b1-49e7bcb94d3so20695365e9.2 for ; Mon, 21 Sep 2026 06:39:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789997998; x=1790602798; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0usqXOyq3o4rQ+AIVUUfsp1gP3m25NdfBnNXnNuf6eU=; b=elBFyK9RLFilzpG/TmAgpW9bxj8muxXlg99N53H3eV4np8s8VqfEDigO00OvlkGv7n Gmwp4C5/ob2Pro9ciGOeYH6ThlUqwkdH+stElgaGkZsza5457AQprteLFhFBtMI0NNRk AjwFF26AsXoiu/xAOtkESEfPzteWMQET5/oE/Mjkkrm+YL5fhMU5QBjId+7Sb2FcMAlj nUSk/v7Om4jnAvdy6dLPkgGs+LKIIX5PIFXd/irK4vABWAb68tvsCguy0Y0XBP38kdoR yj4nft0NVHK7jUM/UdVXstPVaL1fw88520VcxrM6MKV/PzME4POYRepXQ7MQFJacuUMj koMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789997998; x=1790602798; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=0usqXOyq3o4rQ+AIVUUfsp1gP3m25NdfBnNXnNuf6eU=; b=O+Tdk42IzmWtw+UF8V1zDle87DCLAeFo2GNnKqGoWyz7PV8Aftm1Bwu/RQv0SXXXhx Y9vegYAc4siL45QGoD/lYf7/nzUQ6JuhG1CmuiDh4FzdH7IO1Y2ldE410ReAJuoUSC+U 317Mo+zt1EICSaxvkySgrQWvGQ58vzWlSiBqVBvVcJq8UyYcDMu2OCaWdOrbci7HgPns G17L+0w5A0P8/coAfvoBVKO2T9C27Lc7T7sVHMClw9XFOGTWz4hf/gSFFy+cQ+SBQqqm amd0qU1Se8UtBoqM2Ia2y7JOCxJ5rRh9na73XY5VNZShzzmiVr1cp62SWKyY3zvYY/qf MGZw== X-Forwarded-Encrypted: i=1; AKwUvBwS1RTuKuUCXMfvjSEoZK7s9ZCoYRyRH81JMtEbUXQcQcYemvIauzDyG7FBT/Q8d3M5aZwOWRQWyxa5l9I=@vger.kernel.org X-Gm-Message-State: AFuF++lw1Dqpv2BBW8TJqIDNhGAK439etHwdURdaSYgW7dtNvR9g7fUD J0Ubp9zaE1EEWMZv8Mwl9Q2HePkYtYlqkF0rAq394mpAcLpvjhyxHmsO X-Gm-Gg: AYBFou2HvSPYmSw/0fbv1iUvzcT2YY48ymCiE9y/nTye/PsypFpFlJZeJWNjJiXAaHe VL3mLiS4W1YzHsMoKxbADINy5LO3WJUmcXzFCWxCsdL6tYgRxbHaRCopdekq0LWFx9lUFuIxM1/ hsiAlTT1tF8TPmfq90PL/pnOR1fQcTZWkGnseIV4fktV8wxTFnnT/9fdS7bsaWYJnX15aGAZxOy Ye0v6xglziVQgQ5arI2xwv1o0Gz1mADII1twnk92JML89kMdCBrZBWJzKDZ4trMxPzsTbnKO6bY HMbvV+zsm379DU2EQDzhn8EyTUAFEH0gUU+xI9ZYDYEVL3cpKxt1JuFbFtMwpVncrnwEndTghK0 ihYqMVq9gjRvWmnTPw2yZMk0c2FQ/cLiMXtByUCXL3L1GquvLjJxZpofbfgAdL5/2gjM6NHMR48 ZUDaohGwpheMW0yRCbQGNxrFqe+TLnhykdGKuqVadw+sZir8uS0FTJb2kodvZK0fTwPeuTeCqLD heqPxY55xjkmM4zsw5ImbeiHI2r6/eU/R/pd6cJ/T//tF8p3G/HmnBmnmtZ1Ngzi+cTNOPF9aKj tY4NodirMBtv4IWpIL61zcLh6OihyqWQ+hFNS0wc X-Received: by 2002:a05:600c:34c8:b0:49e:732e:6b79 with SMTP id 5b1f17b1804b1-49fc574eff6mr149861745e9.32.1789997998153; Mon, 21 Sep 2026 06:39:58 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:39:57 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 11/13] io_uring/rsrc: add uncloneable regbuf flag Date: Mon, 21 Sep 2026 14:38:55 +0100 Message-ID: <9b71636a03f2345bcc09127d77eb6b294a1d0938.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It's hard to implement cloning if the internal structure needs to be mutable and/or relies on other ring resources. In preparation to such buffer types, add a flag indicating that the buffer can't be cloned. It might be possible to add cloning in the future for them, but that would likely need reallocating the structure and reacquiring resources in case by case manner. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 5 +++++ io_uring/rsrc.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 83a7ad9f5bc7..0b526b094dc9 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1434,6 +1434,11 @@ static int io_clone_buffers(struct io_ring_ctx *ctx,= struct io_ring_ctx *src_ctx if (!src_node) { dst_node =3D NULL; } else { + if (src_node->buf->flags & IO_REGBUF_F_UNCLONEABLE) { + io_rsrc_data_free(ctx, &data); + return -ENOMEM; + } + dst_node =3D io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); if (!dst_node) { io_rsrc_data_free(ctx, &data); diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 9ef88383b363..73d7b1ebf0b6 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -26,7 +26,8 @@ struct io_rsrc_node { }; =20 enum { - IO_REGBUF_F_KBUF =3D 1, + IO_REGBUF_F_KBUF =3D 1 << 0, + IO_REGBUF_F_UNCLONEABLE =3D 1 << 1, }; =20 struct io_mapped_ubuf { --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F20AA4A8429 for ; Mon, 21 Sep 2026 13:40:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998006; cv=none; b=WwdFHPfe41UWeyn5ZvKC8vLB8WNRyXwz8D/C4bzc7iC7GEiBrICOU/Li1iyCdwZ2hXH2VdwFZu6ubg3NQPvUUEHbg3yrQY7UP8ir/4+BKlTANkeU7eeWTDjLcrtaZCGN/xfXNuz3u2mS2Pqeo1CWr9ygILtj6Cn3xnrG9KwbOqg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998006; c=relaxed/simple; bh=kBi8BbQVDcPE/2Q3Wr6NlZISspLakO5UHtrHPWtVj5g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BFz5sh5syYlp9Jdynl9xo91+7HML4XW2jWg4f+SpdPT3OwFt16TIn7OAU6IODLY3YvZMT29GKmf7TTd8mrSvVR8N43ie1oqkiOTiR9fV7ovXoCVnRcFJU8osCWVCv1JfVHDTAUM+jnbKmfr5M+jtyM7A/k7yU7cOgozjxa6zT3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JQ9CBXZR; arc=none smtp.client-ip=74.125.225.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JQ9CBXZR" Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49b965f447cso25338675e9.3 for ; Mon, 21 Sep 2026 06:40:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789998002; x=1790602802; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/FMRvm1BaZ6kTlzIVu1IAGTERBBcZPL0byDe3SvrYcY=; b=JQ9CBXZRm3oNHhMB9wylLRIlXa4CC36rQ6napKSfXH/L2NTDUxJRw9n2arEgVyOGCJ dBo1amanLtLrdVlrm5sFkX/mnbKGxe+wGrGWNCULR7CtnrT/YATYjy1uzLXdKKWS9JQ1 k1oJCKHjvr3Llo+PIPy6XnmltxVqCw7pOAyVPvc8rz4/fcC3se2c1FTOsc1E4muCFOiR hEMVhDMP61600m2UZRrw6izfO2SBVGHwl+zoe5ShdEG1SacCAzwlZlUfuCoK9qenb7rk bkpGv1/jeYb17tOn4g/3KhbiuH7O0swwQAyBle2RnTOm154P7pESPQH/VSDUDjYnph9Y 0O6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789998002; x=1790602802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/FMRvm1BaZ6kTlzIVu1IAGTERBBcZPL0byDe3SvrYcY=; b=HHYP1pmf1nALD0HX1OsowDhdbK50oFEAm0Efz+o9wOJLP3IEgzCN6XV4LeJEKDO2l9 eKCx0Z5oZ5iNMTKGoZkZtLSaS7Im/Jc6kw98SHSsEBbJHh+Dz5JKre006gswhBgtWCZ7 s1pBPu4DZaZSIT58U+FhcX5DbwDPV18udF5tKf9VGTvniCZnQmA9UsIJrt/OW4Xef2mH WieRQU2u1hQctRS3B/e/R66CP28E2TgB1cwpP+AEVS6f9xSr1w6yJtSvk5fJ1GgmSmR7 KvHKYJXABi0x7obMEU8/1VybHFKyYwwCvjfvQkbqVudaMLfJaQW8CwQUeL6QGBFnwvyW WeeA== X-Forwarded-Encrypted: i=1; AKwUvBwHAzVWPwitENZ0CO7dEbCCV0mGSKFx+aY4ilJ/DWoSzE0JrG69lKxSQRSbNHcFgphcGIbhHKE5ZzHizbU=@vger.kernel.org X-Gm-Message-State: AFuF++m0oMu5s6RQ61Mb6GDDXUi8orNF0ZFW4INgWxxBiIf7uhcLZ3So +tW9wIVavLggOKQCd4sA/hYzc1oGZJS0MIcuSCQuBWa112KlC9Y9hYmJ X-Gm-Gg: AYBFou3VtcopsTje92BxCey7cYlY6BMCno09NxnhWAVHeQg1Y3S2bT7F5cuJHRFIyC2 J6aMCMTNZVjpcAjWa9MRoVLl88TPN/daBYJpLjLqdlakAje3FI6UmQxCYjtoEzNhdTGJOD4iXPp LU43wKAaGhIchyMehW/+/x+WSangO1b9OFOFiG0eV8lcSymlz/8f6v4EyfHpAKjlz3rcP1jVHOV wFIO8pQc+GGR6ogwpoMzZjh/iwJXvO/Bj3MG2OuzAUk2Z10tTc3tCezEAsdoPiDLQfCKtDHKqsi TOabXwHR8w8bt0c58oxg5D3oUqjz6SMjGyTq9jZ+exlTC7uRBZGhSP9Uou/S7SUXJ2AlZAhgIp4 Bq7zNUFdWdQv1S/PV/Fv3MYb8rxd1vULDaawIUrQmLKEUIJdV+V7s8HkC5kAjr5R5yvD7PCxSIN k9b+yxOwPfIeVN3fv1gknURLUxKsMQ3vxXZvWumShr9i0IUn9xHnLpbe4WQkVkK2N67pQrti1do XBWBWlM1W8rLdCgC6iZeSfuSWkczMQqn+JZ/BTYYGyIj4mM6uYTliBRzm2I7XlxbHvCeKFFchEO +6vWA26hvK0DrfIsISA1zO3iaCApsg== X-Received: by 2002:a05:600c:3107:b0:49c:ff9f:f6b6 with SMTP id 5b1f17b1804b1-49fc5669346mr153997265e9.6.1789998002057; Mon, 21 Sep 2026 06:40:02 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.39.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:40:01 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 12/13] io_uring/rsrc: add regbuf import flags Date: Mon, 21 Sep 2026 14:38:56 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" We'll have special registered buffer types that can't be used with all opcodes and need special handling. Add separate flags to control registered buffer import, which will be used to specify what kind of buffers the caller can handle. Signed-off-by: Pavel Begunkov --- io_uring/rsrc.c | 8 ++++---- io_uring/rsrc.h | 24 ++++++++++++++++++++---- 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 0b526b094dc9..79e3c686ecc1 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -1298,9 +1298,9 @@ inline struct io_rsrc_node *io_find_buf_node(struct i= o_kiocb *req, return NULL; } =20 -int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, +int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags) + unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node *node; =20 @@ -1709,9 +1709,9 @@ static int io_kern_bvec_size(struct iovec *iov, unsig= ned nr_iovs, return 0; } =20 -int io_import_reg_vec(int ddir, struct iov_iter *iter, +int __io_import_reg_vec(int ddir, struct iov_iter *iter, struct io_kiocb *req, struct iou_vec *vec, - unsigned nr_iovs, unsigned issue_flags) + unsigned nr_iovs, unsigned issue_flags, unsigned import_flags) { struct io_rsrc_node *node; struct io_mapped_ubuf *imu; diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 73d7b1ebf0b6..351995b61e68 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -62,12 +62,28 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsig= ned nr); =20 struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req, unsigned issue_flags); -int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, +int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter, u64 buf_addr, size_t len, int ddir, - unsigned issue_flags); -int io_import_reg_vec(int ddir, struct iov_iter *iter, + unsigned issue_flags, unsigned import_flags); +int __io_import_reg_vec(int ddir, struct iov_iter *iter, struct io_kiocb *req, struct iou_vec *vec, - unsigned nr_iovs, unsigned issue_flags); + unsigned nr_iovs, unsigned issue_flags, + unsigned import_flags); + +static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter = *iter, + u64 buf_addr, size_t len, int ddir, + unsigned issue_flags) +{ + return __io_import_reg_buf(req, iter, buf_addr, len, ddir, issue_flags, 0= ); +} + +static inline int io_import_reg_vec(int ddir, struct iov_iter *iter, + struct io_kiocb *req, struct iou_vec *vec, + unsigned nr_iovs, unsigned issue_flags) +{ + return __io_import_reg_vec(ddir, iter, req, vec, nr_iovs, issue_flags, 0); +} + int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv, const struct iovec __user *uvec, size_t uvec_segs); =20 --=20 2.54.0 From nobody Thu Sep 24 18:37:40 2026 Received: from mail-wm2-f12.google.com (mail-wm2-f12.google.com [74.125.225.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA36A4908D4 for ; Mon, 21 Sep 2026 13:40:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.225.140 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998016; cv=none; b=iqZRvZhN0b0vQx22nKRMFj3VABKFUfEobERomrgZWfRx7t3svrju0Nx1fPyIe+mjvbEB7QAkW2DRb/DBNXPUUJba/SeBEf3aUnc08Us5PDhoHYZq3ymdyJ2Q7nLLWS6Ayj53PBiBiXDKgCdblMsIgZmw1/ZhhIT491Yafmc/1HM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789998016; c=relaxed/simple; bh=rQeme4mOWh7F18Z9nPG3uk3qctIZf5OLv1gSfE+PgWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lzcDrlPgzPXVWlljl8pB+7J7HI+c8oETXQLs070P2/MgbqWQsZKNoax3OIIEQpw1oU2iL8Whx0pLMMCVMCeBLFqWDHyqPzbNkbQ3rtHlJp30p6EVnvJ21jN2tyUdcICmTolS4qDdWgNKyUHIcefCBQ0N+KvxjOdzMeJNnAhEF8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jxIZy9jf; arc=none smtp.client-ip=74.125.225.140 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jxIZy9jf" Received: by mail-wm2-f12.google.com with SMTP id 5b1f17b1804b1-49cd38e0f79so16750535e9.3 for ; Mon, 21 Sep 2026 06:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789998007; x=1790602807; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=NH6ys1CTIdRnJg9xiwn6C8F70peAM9vhMfijPWQg7aQ=; b=jxIZy9jfmpn4opY9FjkyRZdXTyPXCrh0HQMs+fGII8epOkWnlFgPrv/UUagLaxzrWk xm762F2oaq3WFcGFyYyVcHxv+34epuI4mFuF5GfYp4pzcyLcEhSykNjopO09ZLHJGBIq gkUbqZEQVchoIL0xI7uFqcUIqlxKsLHmr77rfvEIeAI0LYs2cVaEiscvinV2a8oIhqQi hCvCtkODR5hFQq0RJgSwe+8LWRNqg79rYiOWfQ+nNbNH0txiHg9eR4R0xN2J6dCDgRfn UGYkURYXsEf5Am7WobG1QJbC7uggr4f6gleG9EYWDwPGUL7aeuAlR4uEs7ANT20FSRGy NN/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789998007; x=1790602807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=NH6ys1CTIdRnJg9xiwn6C8F70peAM9vhMfijPWQg7aQ=; b=DnSR8UhXKjvbFLt8JvQuV95//EWvnqjuTyTT8Ravz2w+YmzVefaLSztpSLNg7m21HK XtYDJvKBpIo+HqwZSxAuT81Zi5saBr3PV91RDzlE3fBbPu0l/aRtWE9/tNyoq2MUFEeN 37+jue3Gl3Wbflx4yhSDEES2h98143Fnb1Z3Tix1uGBh0c25wB/QnPwKCJnIKqisDD8l kmC++ibPsTQIzgMYrg0UkyfTu1D10IGHwusz5vyfPlfJ9DRMo1UOQdNObTosV5U/DJf8 8dX4uJcp6YLtTAcFzam3LAqxiDKqTK3ObLjELwcLjWGK1mAUdMdafBSgDnUj+BrvL9mn vJog== X-Forwarded-Encrypted: i=1; AKwUvBwZfpF9/rg6UVQ3gQA9nXKTHcCoW+sPl7UMtS1diohKcmNMVBdIW2ag/+KbuI6iC9idICXIU/v1uJJ9jCc=@vger.kernel.org X-Gm-Message-State: AFuF++m37jj0CuIbIHP3CHJ68hh5m8GUrzkSc+pA8Ve5VnNHCbrZsx6X Kvy90J371qW1N2+Zj24IT7cD7Daa7MTouZPypiVHhl3jY7cOhnyoTTap X-Gm-Gg: AYBFou368zY9ZNv43KP3YXHeELTh01pT3Fjfxdgq5mhi/aDTz7Tl1D3QHgOGuEfGJUU htws3SmjiCYLBcWI94WxJkc4iU4ERMkRFJ3LNRulYgrPm9iEYL54IRwx9uT6UG2V39dPXazU+Pl aZdSn5ZWmlnoycM51VSB6aoWVmHN4KbSjAbmFdQFMdMAPI9vnCdAvoXxWSE2AJhA0wn1e8q730A o+3iEzMBHjWM8sJXxefWrc96uqsghQEmcf5KEheNyft1nDATqqAgfEdFNi8ATZN8/u1lr+XT55W fcfpaT2ErS4hxSO+c6qoj9UYAuGk2VutkhEXN2RqlEWxhQkBRJ5zxRCx9NsuNNm0WF3iZTAFJN8 oznJp9C/m69h6M9K5iFoazyV4L3max9fpsLvYA2XO2UOqQe8i1WYxEieSsB3ABKTTMyiZvZ+ajB UHGyNCKQ5SXLTeMwvYYRUD8kToWB5T97bSvsK99LeDP33G2R/qvZs2p2vZD+sm9Jj7MbIOyO8M1 WJcJF11cQPLeSe4ILC4FKR/tekBfdcmTZ/ZFVFPBlbiwKLRbqHu1eUTSZ6G5MRqY+0Ilgw95g7O /XgcwFWZJvH+YIQoxaWeR38VR7SyYQ== X-Received: by 2002:a05:600c:8709:b0:49d:870:7a69 with SMTP id 5b1f17b1804b1-49fc573514bmr197593185e9.12.1789998007283; Mon, 21 Sep 2026 06:40:07 -0700 (PDT) Received: from 127.0.0.1localhost (82-132-213-26.dab.02.net. [82.132.213.26]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fc585920fsm617171605e9.4.2026.09.21.06.40.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 06:40:06 -0700 (PDT) From: Pavel Begunkov To: linux-block@vger.kernel.org Cc: asml.silence@gmail.com, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-nvme@lists.infradead.org, linux-fsdevel@vger.kernel.org, io-uring@vger.kernel.org, Christoph Hellwig , Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Keith Busch , Sagi Grimberg , Alexander Viro , Christian Brauner , Jan Kara , Andrew Morton , Jens Axboe , Nitesh Shetty , Kanchan Joshi , Anuj Gupta , Tushar Gohad , William Power , Phil Cayton , Matthew Brost , Alasdair Kergon , Mike Snitzer , Mikulas Patocka , Benjamin Marzinski , dm-devel@lists.linux.dev Subject: [PATCH v6 13/13] io_uring/rsrc: add dmabuf backed registered buffers Date: Mon, 21 Sep 2026 14:38:57 +0100 Message-ID: <0d82253d5e930981e6a21503e145a1ebd280f995.1789997898.git.asml.silence@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement dmabuf backed registered buffers. To register them, the user should specify IO_REGBUF_TYPE_DMABUF for the regitration and pass the desired dmabuf fd and a file for which it should be registered. From there, it can be used with io_uring read/write requests IORING_OP_{READ,WRITE}_FIXED) as normal. The requests should be issued against the file specified during registration, and otherwise they'll be failed. The user should also be prepared to handle spurious -EAGAIN by reissuing the request. Internally, dmabuf registered buffers is an optin feature for io_uring request opcodes and they should pass a special flag on import to use it. Suggested-by: David Wei Suggested-by: Vishal Verma Suggested-by: Tushar Gohad Signed-off-by: Pavel Begunkov --- include/linux/io_uring_types.h | 5 + include/uapi/linux/io_uring.h | 6 +- io_uring/io_uring.c | 3 +- io_uring/rsrc.c | 161 ++++++++++++++++++++++++++++++++- io_uring/rsrc.h | 18 ++++ io_uring/rw.c | 6 +- 6 files changed, 191 insertions(+), 8 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 90fea94ad202..3d41ca622114 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -11,6 +11,7 @@ =20 struct iou_loop_params; struct io_uring_bpf_ops; +struct dma_buf_io_map; =20 enum { /* @@ -610,6 +611,7 @@ enum { REQ_F_IMPORT_BUFFER_BIT, REQ_F_SQE_COPIED_BIT, REQ_F_IOPOLL_BIT, + REQ_F_DROP_DMABUF_BIT, =20 /* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT, @@ -705,6 +707,8 @@ enum { REQ_F_SQE_COPIED =3D IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT), /* request must be iopolled to completion (set in ->issue()) */ REQ_F_IOPOLL =3D IO_REQ_FLAG(REQ_F_IOPOLL_BIT), + /* there is a dma map attached to request that needs to be dropped */ + REQ_F_DROP_DMABUF =3D IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT), }; =20 struct io_tw_req { @@ -827,6 +831,7 @@ struct io_kiocb { /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; struct io_wq_work work; + struct dma_buf_io_map *dmabuf_map; =20 struct io_big_cqe { u64 extra1; diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index 98b259901185..c39297e8beb6 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -810,6 +810,7 @@ enum io_uring_rsrc_reg_flags { enum io_uring_regbuf_type { IO_REGBUF_TYPE_EMPTY, IO_REGBUF_TYPE_UADDR, + IO_REGBUF_TYPE_DMABUF, =20 __IO_REGBUF_TYPE_MAX, }; @@ -819,7 +820,10 @@ struct io_uring_regbuf_desc { __u32 flags; __u64 size; __u64 uaddr; - __u64 __resv[7]; + + __s32 dmabuf_fd; + __s32 target_fd; + __u64 __resv[6]; }; =20 /* Skip updating fd indexes set to this value in the fd table */ diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index a67b2adeda36..267b2a43131a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -109,7 +109,7 @@ =20 #define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | IO_REQ_LINK_FLAGS | \ REQ_F_REISSUE | REQ_F_POLLED | \ - IO_REQ_CLEAN_FLAGS) + IO_REQ_CLEAN_FLAGS | REQ_F_DROP_DMABUF) =20 #define IO_TCTX_REFS_CACHE_NR (1U << 10) =20 @@ -1134,6 +1134,7 @@ static void io_free_batch_list(struct io_ring_ctx *ct= x, io_queue_next(req); if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS)) io_clean_op(req); + io_req_drop_dmabuf(req); } io_put_file(req); io_req_put_rsrc_nodes(req); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 79e3c686ecc1..cb7495df13b7 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -10,6 +10,7 @@ #include #include #include +#include =20 #include =20 @@ -881,6 +882,93 @@ bool io_check_coalesce_buffer(struct page **page_array= , int nr_pages, return true; } =20 +struct io_regbuf_dma { + struct dma_buf_io_ctx *ctx; + struct file *target_file; +}; + +static void io_release_reg_dmabuf(void *priv) +{ + struct io_regbuf_dma *db =3D priv; + + fput(db->target_file); + dma_buf_io_ctx_release(db->ctx); +} + +static struct io_rsrc_node *io_register_dmabuf(struct io_ring_ctx *ctx, + struct io_uring_regbuf_desc *desc) +{ + struct io_rsrc_node *node =3D NULL; + struct io_mapped_ubuf *imu =3D NULL; + struct io_regbuf_dma *regbuf =3D NULL; + struct file *target_file =3D NULL; + struct dma_buf *dmabuf =3D NULL; + int ret; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return ERR_PTR(-EOPNOTSUPP); + if (ctx->flags & IORING_SETUP_IOPOLL) + return ERR_PTR(-EOPNOTSUPP); + if (desc->uaddr || desc->size) + return ERR_PTR(-EINVAL); + + ret =3D -ENOMEM; + node =3D io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER); + if (!node) + return ERR_PTR(-ENOMEM); + imu =3D io_alloc_imu(ctx, 0); + if (!imu) + goto err; + regbuf =3D kzalloc(sizeof(*regbuf), GFP_KERNEL); + if (!regbuf) + goto err; + + ret =3D -EBADF; + target_file =3D fget(desc->target_fd); + if (!target_file) + goto err; + + dmabuf =3D dma_buf_get(desc->dmabuf_fd); + if (IS_ERR(dmabuf)) { + ret =3D PTR_ERR(dmabuf); + dmabuf =3D NULL; + goto err; + } + ret =3D io_validate_user_buf_range(0, dmabuf->size); + if (ret) + goto err; + + ret =3D dma_buf_io_ctx_create(target_file, dmabuf, DMA_BIDIRECTIONAL, + ®buf->ctx); + if (ret) + goto err; + + regbuf->target_file =3D target_file; + imu->nr_bvecs =3D 0; + imu->ubuf =3D 0; + imu->len =3D dmabuf->size; + imu->folio_shift =3D 0; + imu->release =3D io_release_reg_dmabuf; + imu->priv =3D regbuf; + imu->flags =3D IO_REGBUF_F_DMABUF; + imu->dir =3D IO_BUF_DEST | IO_BUF_SOURCE; + refcount_set(&imu->refs, 1); + node->buf =3D imu; + dma_buf_put(dmabuf); + return node; +err: + kfree(regbuf); + if (imu) + io_free_imu(ctx, imu); + if (node) + io_cache_free(&ctx->node_cache, node); + if (target_file) + fput(target_file); + if (dmabuf) + dma_buf_put(dmabuf); + return ERR_PTR(ret); +} + static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx, struct io_uring_regbuf_desc *desc) { @@ -899,6 +987,12 @@ static struct io_rsrc_node *io_sqe_buffer_register(str= uct io_ring_ctx *ctx, if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)) || desc->flags) return ERR_PTR(-EINVAL); =20 + if (desc->type =3D=3D IO_REGBUF_TYPE_DMABUF) + return io_register_dmabuf(ctx, desc); + + if (desc->dmabuf_fd || desc->target_fd) + return ERR_PTR(-EINVAL); + if (desc->type =3D=3D IO_REGBUF_TYPE_EMPTY) { if (uaddr || size) return ERR_PTR(-EFAULT); @@ -1223,9 +1317,59 @@ static int io_import_kbuf(int ddir, struct iov_iter = *iter, return 0; } =20 -static int io_import_fixed(int ddir, struct iov_iter *iter, +void io_drop_dmabuf_node(struct io_kiocb *req) +{ + struct io_mapped_ubuf *imu; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return; + if (WARN_ON_ONCE(req->buf_node->type !=3D IORING_RSRC_BUFFER)) + return; + imu =3D req->buf_node->buf; + if (WARN_ON_ONCE(!(imu->flags & IO_REGBUF_F_DMABUF))) + return; + dma_buf_io_map_drop(req->dmabuf_map); + req->flags &=3D ~REQ_F_DROP_DMABUF; +} + +static int io_import_dmabuf(struct io_kiocb *req, + int ddir, struct iov_iter *iter, + struct io_mapped_ubuf *imu, + size_t len, size_t offset, + unsigned issue_flags) +{ + bool nowait =3D issue_flags & IO_URING_F_NONBLOCK; + struct io_regbuf_dma *db =3D imu->priv; + struct dma_buf_io_map *map; + + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return -EOPNOTSUPP; + if (!len) + return -EFAULT; + if (req->file !=3D db->target_file) + return -EBADF; + + if (req->flags & REQ_F_DROP_DMABUF) { + map =3D req->dmabuf_map; + goto init_iter; + } + + map =3D dma_buf_io_get_map(db->ctx, nowait); + if (unlikely(IS_ERR(map))) + return PTR_ERR(map); + req->dmabuf_map =3D map; + req->flags |=3D REQ_F_DROP_DMABUF; +init_iter: + iov_iter_dmabuf_map(iter, ddir, map, offset, len); + return 0; +} + +static int io_import_fixed(struct io_kiocb *req, + int ddir, struct iov_iter *iter, struct io_mapped_ubuf *imu, - u64 buf_addr, size_t len) + u64 buf_addr, size_t len, + unsigned issue_flags, + unsigned import_flags) { const struct bio_vec *bvec; size_t folio_mask; @@ -1245,6 +1389,12 @@ static int io_import_fixed(int ddir, struct iov_iter= *iter, =20 offset =3D buf_addr - imu->ubuf; =20 + if (imu->flags & IO_REGBUF_F_DMABUF) { + if (!(import_flags & IO_REGBUF_IMPORT_ALLOW_DMABUF)) + return -EFAULT; + return io_import_dmabuf(req, ddir, iter, imu, len, offset, + issue_flags); + } if (imu->flags & IO_REGBUF_F_KBUF) return io_import_kbuf(ddir, iter, imu, len, offset); =20 @@ -1307,7 +1457,8 @@ int __io_import_reg_buf(struct io_kiocb *req, struct = iov_iter *iter, node =3D io_find_buf_node(req, issue_flags); if (!node) return -EFAULT; - return io_import_fixed(ddir, iter, node->buf, buf_addr, len); + return io_import_fixed(req, ddir, iter, node->buf, buf_addr, len, + issue_flags, import_flags); } =20 static int io_buffer_acct_cloned_hpages(struct io_ring_ctx *ctx, @@ -1729,7 +1880,9 @@ int __io_import_reg_vec(int ddir, struct iov_iter *it= er, iovec_off =3D vec->nr - nr_iovs; iov =3D vec->iovec + iovec_off; =20 - if (imu->flags & IO_REGBUF_F_KBUF) { + if (imu->flags & IO_REGBUF_F_DMABUF) { + return -EOPNOTSUPP; + } else if (imu->flags & IO_REGBUF_F_KBUF) { int ret =3D io_kern_bvec_size(iov, nr_iovs, imu, &nr_segs); =20 if (unlikely(ret)) diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 351995b61e68..8e36a954169e 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -28,6 +28,11 @@ struct io_rsrc_node { enum { IO_REGBUF_F_KBUF =3D 1 << 0, IO_REGBUF_F_UNCLONEABLE =3D 1 << 1, + IO_REGBUF_F_DMABUF =3D 1 << 3, +}; + +enum { + IO_REGBUF_IMPORT_ALLOW_DMABUF =3D 1 << 1, }; =20 struct io_mapped_ubuf { @@ -170,4 +175,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou= _vec *iv) io_vec_free(iv); } =20 +void io_drop_dmabuf_node(struct io_kiocb *req); + +static inline void io_req_drop_dmabuf(struct io_kiocb *req) +{ + if (!IS_ENABLED(CONFIG_DMA_SHARED_BUFFER)) + return; + if (!(req->flags & REQ_F_DROP_DMABUF)) + return; + if (WARN_ON_ONCE(!(req->flags & REQ_F_BUF_NODE))) + return; + io_drop_dmabuf_node(req); +} + #endif diff --git a/io_uring/rw.c b/io_uring/rw.c index 0c9494fd21be..a13dd2138c06 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -367,8 +367,8 @@ static int io_init_rw_fixed(struct io_kiocb *req, unsig= ned int issue_flags, if (io->bytes_done) return 0; =20 - ret =3D io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, - issue_flags); + ret =3D __io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir, + issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF); iov_iter_save_state(&io->iter, &io->iter_state); return ret; } @@ -583,6 +583,8 @@ static void io_complete_rw(struct kiocb *kiocb, long re= s) struct io_rw *rw =3D container_of(kiocb, struct io_rw, kiocb); struct io_kiocb *req =3D cmd_to_io_kiocb(rw); =20 + io_req_drop_dmabuf(req); + /* ring owner may block in freeze_super() before task_work runs */ if (kiocb->ki_flags & IOCB_WRITE) io_req_end_write(req); --=20 2.54.0