drivers/dma-buf/dma-buf.c | 36 ++++++++++++++++++++++++++++++++---- drivers/iommu/iommufd/pages.c | 2 +- drivers/iommu/iommufd/selftest.c | 12 ++++-------- drivers/vfio/pci/vfio_pci_dmabuf.c | 27 ++++++--------------------- include/linux/dma-buf.h | 31 +++++++++++++++++++++++++++++++ 5 files changed, 74 insertions(+), 34 deletions(-)
This series implements a dma-buf “revoke” mechanism: to allow a dma-buf
exporter to explicitly invalidate (“kill”) a shared buffer after it has
been distributed to importers, so that further CPU and device access is
prevented and importers reliably observe failure.
Today, dma-buf effectively provides “if you have the fd, you can keep using
the memory indefinitely.” That assumption breaks down when an exporter must
reclaim, reset, evict, or otherwise retire backing memory after it has been
shared. Concrete cases include GPU reset and recovery where old allocations
become unsafe to access, memory eviction/overcommit where backing storage
must be withdrawn, and security or isolation situations where continued access
must be prevented. While drivers can sometimes approximate this with
exporter-specific fencing and policy, there is no core dma-buf state transition
that communicates “this buffer is no longer valid; fail access” across all
access paths.
The change in this series is to introduce a core “revoked” state on the dma-buf
object and a corresponding exporter-triggered revoke operation. Once a dma-buf
is revoked, new access paths are blocked so that attempts to DMA-map, vmap, or
mmap the buffer fail in a consistent way.
In addition, the series aims to invalidate existing access as much as the kernel
allows: device mappings are torn down where possible so devices and IOMMUs cannot
continue DMA.
The semantics are intentionally simple: revoke is a one-way, permanent transition
for the lifetime of that dma-buf instance.
From a compatibility perspective, users that never invoke revoke are unaffected,
and exporters that adopt it gain a core-supported enforcement mechanism rather
than relying on ad hoc driver behavior. The intent is to keep the interface
minimal and avoid imposing policy; the series provides the mechanism to terminate
access, with policy remaining in the exporter and higher-level components.
BTW, see this megathread [1] for additional context.
Ironically, it was posted exactly one year ago.
[1] https://lore.kernel.org/all/20250107142719.179636-2-yilun.xu@linux.intel.com/
Thanks
Cc: linux-rdma@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-media@vger.kernel.org
Cc: dri-devel@lists.freedesktop.org
Cc: linaro-mm-sig@lists.linaro.org
Cc: kvm@vger.kernel.org
Cc: iommu@lists.linux.dev
To: Jason Gunthorpe <jgg@ziepe.ca>
To: Leon Romanovsky <leon@kernel.org>
To: Sumit Semwal <sumit.semwal@linaro.org>
To: Christian König <christian.koenig@amd.com>
To: Alex Williamson <alex@shazbot.org>
To: Kevin Tian <kevin.tian@intel.com>
To: Joerg Roedel <joro@8bytes.org>
To: Will Deacon <will@kernel.org>
To: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
Leon Romanovsky (4):
dma-buf: Introduce revoke semantics
vfio: Use dma-buf revoke semantics
iommufd: Require DMABUF revoke semantics
iommufd/selftest: Reuse dma-buf revoke semantics
drivers/dma-buf/dma-buf.c | 36 ++++++++++++++++++++++++++++++++----
drivers/iommu/iommufd/pages.c | 2 +-
drivers/iommu/iommufd/selftest.c | 12 ++++--------
drivers/vfio/pci/vfio_pci_dmabuf.c | 27 ++++++---------------------
include/linux/dma-buf.h | 31 +++++++++++++++++++++++++++++++
5 files changed, 74 insertions(+), 34 deletions(-)
---
base-commit: 9ace4753a5202b02191d54e9fdf7f9e3d02b85eb
change-id: 20251221-dmabuf-revoke-b90ef16e4236
Best regards,
--
Leon Romanovsky <leonro@nvidia.com>
On 1/11/26 11:37, Leon Romanovsky wrote: > This series implements a dma-buf “revoke” mechanism: to allow a dma-buf > exporter to explicitly invalidate (“kill”) a shared buffer after it has > been distributed to importers, so that further CPU and device access is > prevented and importers reliably observe failure. We already have that. This is what the move_notify is all about. > Today, dma-buf effectively provides “if you have the fd, you can keep using > the memory indefinitely.” That assumption breaks down when an exporter must > reclaim, reset, evict, or otherwise retire backing memory after it has been > shared. Concrete cases include GPU reset and recovery where old allocations > become unsafe to access, memory eviction/overcommit where backing storage > must be withdrawn, and security or isolation situations where continued access > must be prevented. While drivers can sometimes approximate this with > exporter-specific fencing and policy, there is no core dma-buf state transition > that communicates “this buffer is no longer valid; fail access” across all > access paths. It's not correct that there is no DMA-buf handling for this use case. > The change in this series is to introduce a core “revoked” state on the dma-buf > object and a corresponding exporter-triggered revoke operation. Once a dma-buf > is revoked, new access paths are blocked so that attempts to DMA-map, vmap, or > mmap the buffer fail in a consistent way. > > In addition, the series aims to invalidate existing access as much as the kernel > allows: device mappings are torn down where possible so devices and IOMMUs cannot > continue DMA. > > The semantics are intentionally simple: revoke is a one-way, permanent transition > for the lifetime of that dma-buf instance. > > From a compatibility perspective, users that never invoke revoke are unaffected, > and exporters that adopt it gain a core-supported enforcement mechanism rather > than relying on ad hoc driver behavior. The intent is to keep the interface > minimal and avoid imposing policy; the series provides the mechanism to terminate > access, with policy remaining in the exporter and higher-level components. As far as I can see that patch set is completely superfluous. The move_notify mechanism has been implemented exactly to cover this use case and is in use for a couple of years now. What exactly is missing? Regards, Christian. > > BTW, see this megathread [1] for additional context. > Ironically, it was posted exactly one year ago. > > [1] https://lore.kernel.org/all/20250107142719.179636-2-yilun.xu@linux.intel.com/ > > Thanks > > Cc: linux-rdma@vger.kernel.org > Cc: linux-kernel@vger.kernel.org > Cc: linux-media@vger.kernel.org > Cc: dri-devel@lists.freedesktop.org > Cc: linaro-mm-sig@lists.linaro.org > Cc: kvm@vger.kernel.org > Cc: iommu@lists.linux.dev > To: Jason Gunthorpe <jgg@ziepe.ca> > To: Leon Romanovsky <leon@kernel.org> > To: Sumit Semwal <sumit.semwal@linaro.org> > To: Christian König <christian.koenig@amd.com> > To: Alex Williamson <alex@shazbot.org> > To: Kevin Tian <kevin.tian@intel.com> > To: Joerg Roedel <joro@8bytes.org> > To: Will Deacon <will@kernel.org> > To: Robin Murphy <robin.murphy@arm.com> > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > --- > Leon Romanovsky (4): > dma-buf: Introduce revoke semantics > vfio: Use dma-buf revoke semantics > iommufd: Require DMABUF revoke semantics > iommufd/selftest: Reuse dma-buf revoke semantics > > drivers/dma-buf/dma-buf.c | 36 ++++++++++++++++++++++++++++++++---- > drivers/iommu/iommufd/pages.c | 2 +- > drivers/iommu/iommufd/selftest.c | 12 ++++-------- > drivers/vfio/pci/vfio_pci_dmabuf.c | 27 ++++++--------------------- > include/linux/dma-buf.h | 31 +++++++++++++++++++++++++++++++ > 5 files changed, 74 insertions(+), 34 deletions(-) > --- > base-commit: 9ace4753a5202b02191d54e9fdf7f9e3d02b85eb > change-id: 20251221-dmabuf-revoke-b90ef16e4236 > > Best regards, > -- > Leon Romanovsky <leonro@nvidia.com> >
On Mon, Jan 12, 2026 at 11:04:38AM +0100, Christian König wrote: > On 1/11/26 11:37, Leon Romanovsky wrote: > > This series implements a dma-buf “revoke” mechanism: to allow a dma-buf > > exporter to explicitly invalidate (“kill”) a shared buffer after it has > > been distributed to importers, so that further CPU and device access is > > prevented and importers reliably observe failure. > > We already have that. This is what the move_notify is all about. > > > Today, dma-buf effectively provides “if you have the fd, you can keep using > > the memory indefinitely.” That assumption breaks down when an exporter must > > reclaim, reset, evict, or otherwise retire backing memory after it has been > > shared. Concrete cases include GPU reset and recovery where old allocations > > become unsafe to access, memory eviction/overcommit where backing storage > > must be withdrawn, and security or isolation situations where continued access > > must be prevented. While drivers can sometimes approximate this with > > exporter-specific fencing and policy, there is no core dma-buf state transition > > that communicates “this buffer is no longer valid; fail access” across all > > access paths. > > It's not correct that there is no DMA-buf handling for this use case. > > > The change in this series is to introduce a core “revoked” state on the dma-buf > > object and a corresponding exporter-triggered revoke operation. Once a dma-buf > > is revoked, new access paths are blocked so that attempts to DMA-map, vmap, or > > mmap the buffer fail in a consistent way. > > > > In addition, the series aims to invalidate existing access as much as the kernel > > allows: device mappings are torn down where possible so devices and IOMMUs cannot > > continue DMA. > > > > The semantics are intentionally simple: revoke is a one-way, permanent transition > > for the lifetime of that dma-buf instance. > > > > From a compatibility perspective, users that never invoke revoke are unaffected, > > and exporters that adopt it gain a core-supported enforcement mechanism rather > > than relying on ad hoc driver behavior. The intent is to keep the interface > > minimal and avoid imposing policy; the series provides the mechanism to terminate > > access, with policy remaining in the exporter and higher-level components. > > As far as I can see that patch set is completely superfluous. > > The move_notify mechanism has been implemented exactly to cover this use case and is in use for a couple of years now. > > What exactly is missing? From what I can tell, the missing piece is what happens after .move_notify() is called. According to the documentation, the exporter remains valid, and the importer is expected to recreate all mappings. include/linux/dma-buf.h: 471 * Mappings stay valid and are not directly affected by this callback. 472 * But the DMA-buf can now be in a different physical location, so all 473 * mappings should be destroyed and re-created as soon as possible. 474 * 475 * New mappings can be created after this callback returns, and will 476 * point to the new location of the DMA-buf. Call to dma_buf_move_notify() does not prevent new attachments to that exporter, while "revoke" does. In the current code, the importer is not aware that the exporter no longer exists and will continue calling dma_buf_map_attachment(). In summary, the current implementation allows a single .attach() check but permits multiple .map_dma_buf() calls. With "revoke", we gain the ability to block any subsequent .map_dma_buf() operations. Main use case is VFIO as exporter and IOMMUFD as importer. Thanks > > Regards, > Christian. > > > > > BTW, see this megathread [1] for additional context. > > Ironically, it was posted exactly one year ago. > > > > [1] https://lore.kernel.org/all/20250107142719.179636-2-yilun.xu@linux.intel.com/ > > > > Thanks > > > > Cc: linux-rdma@vger.kernel.org > > Cc: linux-kernel@vger.kernel.org > > Cc: linux-media@vger.kernel.org > > Cc: dri-devel@lists.freedesktop.org > > Cc: linaro-mm-sig@lists.linaro.org > > Cc: kvm@vger.kernel.org > > Cc: iommu@lists.linux.dev > > To: Jason Gunthorpe <jgg@ziepe.ca> > > To: Leon Romanovsky <leon@kernel.org> > > To: Sumit Semwal <sumit.semwal@linaro.org> > > To: Christian König <christian.koenig@amd.com> > > To: Alex Williamson <alex@shazbot.org> > > To: Kevin Tian <kevin.tian@intel.com> > > To: Joerg Roedel <joro@8bytes.org> > > To: Will Deacon <will@kernel.org> > > To: Robin Murphy <robin.murphy@arm.com> > > > > Signed-off-by: Leon Romanovsky <leonro@nvidia.com> > > --- > > Leon Romanovsky (4): > > dma-buf: Introduce revoke semantics > > vfio: Use dma-buf revoke semantics > > iommufd: Require DMABUF revoke semantics > > iommufd/selftest: Reuse dma-buf revoke semantics > > > > drivers/dma-buf/dma-buf.c | 36 ++++++++++++++++++++++++++++++++---- > > drivers/iommu/iommufd/pages.c | 2 +- > > drivers/iommu/iommufd/selftest.c | 12 ++++-------- > > drivers/vfio/pci/vfio_pci_dmabuf.c | 27 ++++++--------------------- > > include/linux/dma-buf.h | 31 +++++++++++++++++++++++++++++++ > > 5 files changed, 74 insertions(+), 34 deletions(-) > > --- > > base-commit: 9ace4753a5202b02191d54e9fdf7f9e3d02b85eb > > change-id: 20251221-dmabuf-revoke-b90ef16e4236 > > > > Best regards, > > -- > > Leon Romanovsky <leonro@nvidia.com> > > > >
On 1/12/26 13:19, Leon Romanovsky wrote: > On Mon, Jan 12, 2026 at 11:04:38AM +0100, Christian König wrote: >> On 1/11/26 11:37, Leon Romanovsky wrote: >>> This series implements a dma-buf “revoke” mechanism: to allow a dma-buf >>> exporter to explicitly invalidate (“kill”) a shared buffer after it has >>> been distributed to importers, so that further CPU and device access is >>> prevented and importers reliably observe failure. >> >> We already have that. This is what the move_notify is all about. >> >>> Today, dma-buf effectively provides “if you have the fd, you can keep using >>> the memory indefinitely.” That assumption breaks down when an exporter must >>> reclaim, reset, evict, or otherwise retire backing memory after it has been >>> shared. Concrete cases include GPU reset and recovery where old allocations >>> become unsafe to access, memory eviction/overcommit where backing storage >>> must be withdrawn, and security or isolation situations where continued access >>> must be prevented. While drivers can sometimes approximate this with >>> exporter-specific fencing and policy, there is no core dma-buf state transition >>> that communicates “this buffer is no longer valid; fail access” across all >>> access paths. >> >> It's not correct that there is no DMA-buf handling for this use case. >> >>> The change in this series is to introduce a core “revoked” state on the dma-buf >>> object and a corresponding exporter-triggered revoke operation. Once a dma-buf >>> is revoked, new access paths are blocked so that attempts to DMA-map, vmap, or >>> mmap the buffer fail in a consistent way. >>> >>> In addition, the series aims to invalidate existing access as much as the kernel >>> allows: device mappings are torn down where possible so devices and IOMMUs cannot >>> continue DMA. >>> >>> The semantics are intentionally simple: revoke is a one-way, permanent transition >>> for the lifetime of that dma-buf instance. >>> >>> From a compatibility perspective, users that never invoke revoke are unaffected, >>> and exporters that adopt it gain a core-supported enforcement mechanism rather >>> than relying on ad hoc driver behavior. The intent is to keep the interface >>> minimal and avoid imposing policy; the series provides the mechanism to terminate >>> access, with policy remaining in the exporter and higher-level components. >> >> As far as I can see that patch set is completely superfluous. >> >> The move_notify mechanism has been implemented exactly to cover this use case and is in use for a couple of years now. >> >> What exactly is missing? > > From what I can tell, the missing piece is what happens after .move_notify() > is called. According to the documentation, the exporter remains valid, and > the importer is expected to recreate all mappings. > > include/linux/dma-buf.h: > 471 * Mappings stay valid and are not directly affected by this callback. > 472 * But the DMA-buf can now be in a different physical location, so all > 473 * mappings should be destroyed and re-created as soon as possible. > 474 * > 475 * New mappings can be created after this callback returns, and will > 476 * point to the new location of the DMA-buf. > > Call to dma_buf_move_notify() does not prevent new attachments to that > exporter, while "revoke" does. In the current code, the importer is not aware > that the exporter no longer exists and will continue calling > dma_buf_map_attachment(). Yeah and that is perfectly intentional. > In summary, the current implementation allows a single .attach() check but > permits multiple .map_dma_buf() calls. With "revoke", we gain the ability to > block any subsequent .map_dma_buf() operations. Clear NAK to that plan. This is not something DMA-buf should need to deal with and as far as I can see is incompatible with the UAPI. If a DMA-buf can no longer be attached or mapped then the relevant callbacks just need to return an error code. Existing mappings can be invalidated with the move_notify callback and that functionality should be sufficient to prevent importers from accessing the backing store. Existing attachments should stay around until the importer drops their usage. In other words the exporter can't force an importer to drop their attachments, that would be a violation of the UAPI. Regards, Christian. > > Main use case is VFIO as exporter and IOMMUFD as importer. > > Thanks > >> >> Regards, >> Christian. >> >>> >>> BTW, see this megathread [1] for additional context. >>> Ironically, it was posted exactly one year ago. >>> >>> [1] https://lore.kernel.org/all/20250107142719.179636-2-yilun.xu@linux.intel.com/ >>> >>> Thanks >>> >>> Cc: linux-rdma@vger.kernel.org >>> Cc: linux-kernel@vger.kernel.org >>> Cc: linux-media@vger.kernel.org >>> Cc: dri-devel@lists.freedesktop.org >>> Cc: linaro-mm-sig@lists.linaro.org >>> Cc: kvm@vger.kernel.org >>> Cc: iommu@lists.linux.dev >>> To: Jason Gunthorpe <jgg@ziepe.ca> >>> To: Leon Romanovsky <leon@kernel.org> >>> To: Sumit Semwal <sumit.semwal@linaro.org> >>> To: Christian König <christian.koenig@amd.com> >>> To: Alex Williamson <alex@shazbot.org> >>> To: Kevin Tian <kevin.tian@intel.com> >>> To: Joerg Roedel <joro@8bytes.org> >>> To: Will Deacon <will@kernel.org> >>> To: Robin Murphy <robin.murphy@arm.com> >>> >>> Signed-off-by: Leon Romanovsky <leonro@nvidia.com> >>> --- >>> Leon Romanovsky (4): >>> dma-buf: Introduce revoke semantics >>> vfio: Use dma-buf revoke semantics >>> iommufd: Require DMABUF revoke semantics >>> iommufd/selftest: Reuse dma-buf revoke semantics >>> >>> drivers/dma-buf/dma-buf.c | 36 ++++++++++++++++++++++++++++++++---- >>> drivers/iommu/iommufd/pages.c | 2 +- >>> drivers/iommu/iommufd/selftest.c | 12 ++++-------- >>> drivers/vfio/pci/vfio_pci_dmabuf.c | 27 ++++++--------------------- >>> include/linux/dma-buf.h | 31 +++++++++++++++++++++++++++++++ >>> 5 files changed, 74 insertions(+), 34 deletions(-) >>> --- >>> base-commit: 9ace4753a5202b02191d54e9fdf7f9e3d02b85eb >>> change-id: 20251221-dmabuf-revoke-b90ef16e4236 >>> >>> Best regards, >>> -- >>> Leon Romanovsky <leonro@nvidia.com> >>> >> >>
On Mon, Jan 12, 2026 at 01:57:25PM +0100, Christian König wrote: > Clear NAK to that plan. This is not something DMA-buf should need to > deal with and as far as I can see is incompatible with the UAPI. We had this discussion with Simona and you a while back and there was a pretty clear direction we needed to add a revoke to sit inbetween pin and move. I think Leon has no quite got the "dmabuf lingo" down right to explain this. https://lore.kernel.org/dri-devel/Z4Z4NKqVG2Vbv98Q@phenom.ffwll.local/ Since you mention pin here, I think that's another aspect of the revocable vs dynamic question. Dynamic buffers are expected to sometimes just move around for no reason, and importers must be able to cope. For recovable exporters/importers I'd expect that movement is not happening, meaning it's pinned until the single terminal revocation. And maybe I read the kvm stuff wrong, but it reads more like the latter to me when crawling through the pfn code. The issue is that DMABUF only offers two attachment options today, pin and move. iommufd/kvm can implement pin, but not move because they don't support faulting. vfio and others don't need move with faulting but they do need pin with a terminal, emergency, revocation. The purpose of revoke is to add a new negotiated attachment mode between exporter and importer that behaves the same as pin up until the user does something catastrophic (like ubind a driver) then a revoke invalidation is used to clean everything up safely. You are right that the existing move_notify already meets this semantic, and today VFIO exporter, RDMA ODP importer even implement this. Upon VFIO revoke move_notify() will invalidate and map() will fail. RDMA ODP then HW fails all faults. The problem revoke is designed to solve is that many importers have hardware that can either be DMA'ing or failing. There is no fault mechanims that can be used to implement the full "move around for no reason" semantics that are implied by move_notify. Thus they can't implement move_notify! Revoke allows this less capable HW to still be usable with exporters, so long as exporters promise only to issue an invalidation for a "single terminal revocation". Which does nicely match the needs of exporters which are primarily pin based. IOW this is an enhancement to pin modes to add a terminal error case invalidation to pinned attachments. It is not intended to be UAPI changing, and Leon is not trying to say that importers have to drop their attachment. The attachment just becomes permanently non-present. Jason
On 1/12/26 15:14, Jason Gunthorpe wrote: > On Mon, Jan 12, 2026 at 01:57:25PM +0100, Christian König wrote: >> Clear NAK to that plan. This is not something DMA-buf should need to >> deal with and as far as I can see is incompatible with the UAPI. > > We had this discussion with Simona and you a while back and there was > a pretty clear direction we needed to add a revoke to sit inbetween > pin and move. I think Leon has no quite got the "dmabuf lingo" down > right to explain this. I was already wondering why this was clearly not what we have discussed before. > https://lore.kernel.org/dri-devel/Z4Z4NKqVG2Vbv98Q@phenom.ffwll.local/ > > Since you mention pin here, I think that's another aspect of the revocable > vs dynamic question. Dynamic buffers are expected to sometimes just move > around for no reason, and importers must be able to cope. > > For recovable exporters/importers I'd expect that movement is not > happening, meaning it's pinned until the single terminal revocation. And > maybe I read the kvm stuff wrong, but it reads more like the latter to me > when crawling through the pfn code. > > The issue is that DMABUF only offers two attachment options today, pin > and move. iommufd/kvm can implement pin, but not move because they > don't support faulting. > > vfio and others don't need move with faulting but they do need pin > with a terminal, emergency, revocation. Yeah, I know that this is confusing. But that use case is already supported and we just need to properly document things. The move_notify callback can be called even after pin() in the case of PCIe hotplug for example. We could potentially rename the callback to something like invalidate_mappings. And yes, I know that we had a few issues with that because we didn't properly documented things... > The purpose of revoke is to add a new negotiated attachment mode > between exporter and importer that behaves the same as pin up until > the user does something catastrophic (like ubind a driver) then a > revoke invalidation is used to clean everything up safely. With or with pin() you need to guarantee to the importer that the DMA address you gave out stay valid until the importer had a chance to free up it's mappings. It is intentionally done this way to properly support PCIe hot plug because even when a device is gone the address space can't be re-used until all importers stated that they stopped their DMA. > You are right that the existing move_notify already meets this > semantic, and today VFIO exporter, RDMA ODP importer even implement > this. Upon VFIO revoke move_notify() will invalidate and map() will > fail. RDMA ODP then HW fails all faults. > > The problem revoke is designed to solve is that many importers have > hardware that can either be DMA'ing or failing. There is no fault > mechanims that can be used to implement the full "move around for no > reason" semantics that are implied by move_notify. In this case just call dma_buf_pin(). We already support that approach for RDMA devices which can't do ODP. Regards, Christian. > Thus they can't implement move_notify! > > Revoke allows this less capable HW to still be usable with exporters, > so long as exporters promise only to issue an invalidation for a > "single terminal revocation". Which does nicely match the needs of > exporters which are primarily pin based. > > IOW this is an enhancement to pin modes to add a terminal error case > invalidation to pinned attachments. > > It is not intended to be UAPI changing, and Leon is not trying to say > that importers have to drop their attachment. The attachment just > becomes permanently non-present. > > Jason
On Mon, Jan 12, 2026 at 03:56:32PM +0100, Christian König wrote:
> > The problem revoke is designed to solve is that many importers have
> > hardware that can either be DMA'ing or failing. There is no fault
> > mechanims that can be used to implement the full "move around for no
> > reason" semantics that are implied by move_notify.
>
> In this case just call dma_buf_pin(). We already support that
> approach for RDMA devices which can't do ODP.
That alone isn't good enough - the patch adding the non-ODP support
also contained this:
static void
ib_umem_dmabuf_unsupported_move_notify(struct dma_buf_attachment *attach)
{
struct ib_umem_dmabuf *umem_dmabuf = attach->importer_priv;
ibdev_warn_ratelimited(umem_dmabuf->umem.ibdev,
"Invalidate callback should not be called when memory is pinned\n");
}
static struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_ops = {
.allow_peer2peer = true,
.move_notify = ib_umem_dmabuf_unsupported_move_notify,
};
So we can't just allow it to attach to exporters that are going to
start calling move_notify while pinned.
Looking around I don't see anyone else doing something like this, and
reading your remarks I think EFA guys got it wrong. So I'm wondering
if this should not have been allowed. Unfortunately 5 years later I'm
pretty sure it is being used in places where we don't have HW support
to invalidate at all, and it is now well established uAPI that we
can't just break.
Which is why we are coming to negotiation because at least the above
isn't going to work if move_notify is called for revoke reasons, and
we'd like to block attaching exporters that need revoke for the above.
So, would you be happier with this if we documented that move_notify
can be called for pinned importers for revoke purposes and figure out
something to mark the above as special so exporters can fail pin if
they are going to call move_notify?
Then this series would transform into documentation, making VFIO
accept pin and continue to call move_notify as it does right now, and
some logic to reject the RDMA non-ODP importer.
Jason
On 1/12/26 16:35, Jason Gunthorpe wrote:
> On Mon, Jan 12, 2026 at 03:56:32PM +0100, Christian König wrote:
>>> The problem revoke is designed to solve is that many importers have
>>> hardware that can either be DMA'ing or failing. There is no fault
>>> mechanims that can be used to implement the full "move around for no
>>> reason" semantics that are implied by move_notify.
>>
>> In this case just call dma_buf_pin(). We already support that
>> approach for RDMA devices which can't do ODP.
>
> That alone isn't good enough - the patch adding the non-ODP support
> also contained this:
>
> static void
> ib_umem_dmabuf_unsupported_move_notify(struct dma_buf_attachment *attach)
> {
> struct ib_umem_dmabuf *umem_dmabuf = attach->importer_priv;
>
> ibdev_warn_ratelimited(umem_dmabuf->umem.ibdev,
> "Invalidate callback should not be called when memory is pinned\n");
> }
Yeah, I know. That's what I meant we have to better document this.
>
> static struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_ops = {
> .allow_peer2peer = true,
> .move_notify = ib_umem_dmabuf_unsupported_move_notify,
> };
>
> So we can't just allow it to attach to exporters that are going to
> start calling move_notify while pinned.
The point is exporters are already doing this.
> Looking around I don't see anyone else doing something like this, and
> reading your remarks I think EFA guys got it wrong. So I'm wondering
> if this should not have been allowed. Unfortunately 5 years later I'm
> pretty sure it is being used in places where we don't have HW support
> to invalidate at all, and it is now well established uAPI that we
> can't just break.
>
> Which is why we are coming to negotiation because at least the above
> isn't going to work if move_notify is called for revoke reasons, and
> we'd like to block attaching exporters that need revoke for the above.
Ah, yes that makes sense. This is clearly a new requirement.
So basically for PCIe hotplug was a rare event were we said we have some problems with non-ODP but we can live with that, but for this use case here it's more like a perfectly normal condition that userspace can trigger.
So the exporter wants to reject importers which can't handle a mapping invalidation while the BO is pinned, correct?
>
> So, would you be happier with this if we documented that move_notify
> can be called for pinned importers for revoke purposes and figure out
> something to mark the above as special so exporters can fail pin if
> they are going to call move_notify?
That would work for me. I mean it is already current practice, we just never fully documented it.
>
> Then this series would transform into documentation, making VFIO
> accept pin and continue to call move_notify as it does right now, and
> some logic to reject the RDMA non-ODP importer.
I think we just need to expose this with flags or similar from the importer side. As far as I know RDMA without ODP is currently the only one really needing this (except for cross device scanout, but that is special anyway).
Christian.
>
> Jason
On Mon, Jan 12, 2026 at 05:12:36PM +0100, Christian König wrote:
> > static struct dma_buf_attach_ops ib_umem_dmabuf_attach_pinned_ops = {
> > .allow_peer2peer = true,
> > .move_notify = ib_umem_dmabuf_unsupported_move_notify,
> > };
> >
> > So we can't just allow it to attach to exporters that are going to
> > start calling move_notify while pinned.
>
> The point is exporters are already doing this.
:( So obviously this doesn't work fully correctly..
> > Which is why we are coming to negotiation because at least the above
> > isn't going to work if move_notify is called for revoke reasons, and
> > we'd like to block attaching exporters that need revoke for the above.
>
> Ah, yes that makes sense. This is clearly a new requirement.
>
> So basically for PCIe hotplug was a rare event were we said we have
> some problems with non-ODP but we can live with that, but for this
> use case here it's more like a perfectly normal condition that
> userspace can trigger.
Yes that seems to be exactly the case. I didn't know about the PCI RAS
case until now :(
> So the exporter wants to reject importers which can't handle a
> mapping invalidation while the BO is pinned, correct?
Yes. I think at a minimum exporters where it is a normal use case
should block it so unpriv user space cannot trigger incorrect behavior
/ ignored invalidation. ie VFIO will trigger this based on unpriv user
system calls.
I supposed we have to retain the PCI RAS misbehavior for now at least.
It would probably be uAPI regression to start blocking some of the
existing ones.
It also seems we should invest in the RDMA side to minimize where this
is used.
> > So, would you be happier with this if we documented that move_notify
> > can be called for pinned importers for revoke purposes and figure out
> > something to mark the above as special so exporters can fail pin if
> > they are going to call move_notify?
>
> That would work for me. I mean it is already current practice, we
> just never fully documented it.
OK
> > Then this series would transform into documentation, making VFIO
> > accept pin and continue to call move_notify as it does right now, and
> > some logic to reject the RDMA non-ODP importer.
>
> I think we just need to expose this with flags or similar from the
> importer side. As far as I know RDMA without ODP is currently the
> only one really needing this (except for cross device scanout, but
> that is special anyway).
I did not see any other importers with an obvious broken move_notify,
so I hope this is right. Even iommufd has a working move_notify
(disruptive, but working).
How do you feel about an enum in the ops:
+enum dma_buf_move_notify_level {
+ /*
+ * The importer can pause HW access while move_notify is running
+ * and cleanly handle dynamic changes to the DMA mapping without
+ * any disruption.
+ */
+ DMA_BUF_MOVE_NOTIFY_FAULTING = 0,
+ /*
+ * The importer can stop HW access and disruptively fail any
+ * of its DMA activity. move_notify should only be called if the
+ * exporter is experiencing an unusual error and can accept
+ * that the importer will be disrupted.
+ */
+ DMA_BUF_MOVE_NOTIFY_REVOKING,
+ /*
+ * move_notify is not supported at all and must not be called. Do not
+ * introduce new drivers using this, it has significant draw backs
+ * around PCI error handling and other cases. It has the most limited
+ * set of compatible importers.
+ */
+ DMA_BUF_MOVE_NOTIFY_UNSUPPORTED,
+};
+
/**
* struct dma_buf_attach_ops - importer operations for an attachment
*
@@ -457,6 +480,8 @@ struct dma_buf_attach_ops {
*/
bool allow_peer2peer;
+ enum dma_buf_move_notify_level move_notify_level;
+
/**
* @move_notify: [optional] notification that the DMA-buf is moving
*
Jason
On Mon, Jan 12, 2026 at 10:14:40AM -0400, Jason Gunthorpe wrote: > On Mon, Jan 12, 2026 at 01:57:25PM +0100, Christian König wrote: > > Clear NAK to that plan. This is not something DMA-buf should need to > > deal with and as far as I can see is incompatible with the UAPI. > > We had this discussion with Simona and you a while back and there was > a pretty clear direction we needed to add a revoke to sit inbetween > pin and move. I think Leon has no quite got the "dmabuf lingo" down > right to explain this. > > https://lore.kernel.org/dri-devel/Z4Z4NKqVG2Vbv98Q@phenom.ffwll.local/ <...> > It is not intended to be UAPI changing, and Leon is not trying to say > that importers have to drop their attachment. The attachment just > becomes permanently non-present. Leon also ensures that no UAPI semantic changes are introduced here; the existing interface is simply extended. Thanks > > Jason
© 2016 - 2026 Red Hat, Inc.