> -----Original Message----- > From: Yi Liu [mailto:yi.l.liu@intel.com] > Sent: 14 April 2022 11:47 > To: alex.williamson@redhat.com; cohuck@redhat.com; > qemu-devel@nongnu.org > Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; > mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; > jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; > jgg@nvidia.com; nicolinc@nvidia.com; eric.auger@redhat.com; > eric.auger.pro@gmail.com; kevin.tian@intel.com; yi.l.liu@intel.com; > chao.p.peng@intel.com; yi.y.sun@intel.com; peterx@redhat.com > Subject: [RFC 00/18] vfio: Adopt iommufd > > With the introduction of iommufd[1], the linux kernel provides a generic > interface for userspace drivers to propagate their DMA mappings to kernel > for assigned devices. This series does the porting of the VFIO devices > onto the /dev/iommu uapi and let it coexist with the legacy implementation. > Other devices like vpda, vfio mdev and etc. are not considered yet. > > For vfio devices, the new interface is tied with device fd and iommufd > as the iommufd solution is device-centric. This is different from legacy > vfio which is group-centric. To support both interfaces in QEMU, this > series introduces the iommu backend concept in the form of different > container classes. The existing vfio container is named legacy container > (equivalent with legacy iommu backend in this series), while the new > iommufd based container is named as iommufd container (may also be > mentioned > as iommufd backend in this series). The two backend types have their own > way to setup secure context and dma management interface. Below diagram > shows how it looks like with both BEs. > > VFIO > AddressSpace/Memory > +-------+ +----------+ +-----+ +-----+ > | pci | | platform | | ap | | ccw | > +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ > | | | | | AddressSpace > | > | | | | +------------+---------+ > +---V-----------V-----------V--------V----+ / > | VFIOAddressSpace | <------------+ > | | | MemoryListener > | VFIOContainer list | > +-------+----------------------------+----+ > | | > | | > +-------V------+ +--------V----------+ > | iommufd | | vfio legacy | > | container | | container | > +-------+------+ +--------+----------+ > | | > | /dev/iommu | /dev/vfio/vfio > | /dev/vfio/devices/vfioX | /dev/vfio/$group_id > Userspace | | > > ===========+============================+========================== > ====== > Kernel | device fd | > +---------------+ | group/container fd > | (BIND_IOMMUFD | | > (SET_CONTAINER/SET_IOMMU) > | ATTACH_IOAS) | | device fd > | | | > | +-------V------------V-----------------+ > iommufd | | vfio | > (map/unmap | +---------+--------------------+-------+ > ioas_copy) | | | map/unmap > | | | > +------V------+ +-----V------+ +------V--------+ > | iommfd core | | device | | vfio iommu | > +-------------+ +------------+ +---------------+ > > [Secure Context setup] > - iommufd BE: uses device fd and iommufd to setup secure context > (bind_iommufd, attach_ioas) > - vfio legacy BE: uses group fd and container fd to setup secure context > (set_container, set_iommu) > [Device access] > - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX > - vfio legacy BE: device fd is retrieved from group fd ioctl > [DMA Mapping flow] > - VFIOAddressSpace receives MemoryRegion add/del via MemoryListener > - VFIO populates DMA map/unmap via the container BEs > *) iommufd BE: uses iommufd > *) vfio legacy BE: uses container fd > > This series qomifies the VFIOContainer object which acts as a base class > for a container. This base class is derived into the legacy VFIO container > and the new iommufd based container. The base class implements generic > code > such as code related to memory_listener and address space management > whereas > the derived class implements callbacks that depend on the kernel user space > being used. > > The selection of the backend is made on a device basis using the new > iommufd option (on/off/auto). By default the iommufd backend is selected > if supported by the host and by QEMU (iommufd KConfig). This option is > currently available only for the vfio-pci device. For other types of > devices, it does not yet exist and the legacy BE is chosen by default. > > Test done: > - PCI and Platform device were tested > - ccw and ap were only compile-tested > - limited device hotplug test > - vIOMMU test run for both legacy and iommufd backends (limited tests) > > This series was co-developed by Eric Auger and me based on the exploration > iommufd kernel[2], complete code of this series is available in[3]. As > iommufd kernel is in the early step (only iommufd generic interface is in > mailing list), so this series hasn't made the iommufd backend fully on par > with legacy backend w.r.t. features like p2p mappings, coherency tracking, > live migration, etc. This series hasn't supported PCI devices without FLR > neither as the kernel doesn't support VFIO_DEVICE_PCI_HOT_RESET when > userspace > is using iommufd. The kernel needs to be updated to accept device fd list for > reset when userspace is using iommufd. Related work is in progress by > Jason[4]. > > TODOs: > - Add DMA alias check for iommufd BE (group level) > - Make pci.c to be BE agnostic. Needs kernel change as well to fix the > VFIO_DEVICE_PCI_HOT_RESET gap > - Cleanup the VFIODevice fields as it's used in both BEs > - Add locks > - Replace list with g_tree > - More tests > > Patch Overview: > > - Preparation: > 0001-scripts-update-linux-headers-Add-iommufd.h.patch > 0002-linux-headers-Import-latest-vfio.h-and-iommufd.h.patch > 0003-hw-vfio-pci-fix-vfio_pci_hot_reset_result-trace-poin.patch > 0004-vfio-pci-Use-vbasedev-local-variable-in-vfio_realize.patch > > 0005-vfio-common-Rename-VFIOGuestIOMMU-iommu-into-iommu_m.patch > 0006-vfio-common-Split-common.c-into-common.c-container.c.patch > > - Introduce container object and covert existing vfio to use it: > 0007-vfio-Add-base-object-for-VFIOContainer.patch > 0008-vfio-container-Introduce-vfio_attach-detach_device.patch > 0009-vfio-platform-Use-vfio_-attach-detach-_device.patch > 0010-vfio-ap-Use-vfio_-attach-detach-_device.patch > 0011-vfio-ccw-Use-vfio_-attach-detach-_device.patch > 0012-vfio-container-obj-Introduce-attach-detach-_device-c.patch > 0013-vfio-container-obj-Introduce-VFIOContainer-reset-cal.patch > > - Introduce iommufd based container: > 0014-hw-iommufd-Creation.patch > 0015-vfio-iommufd-Implement-iommufd-backend.patch > 0016-vfio-iommufd-Add-IOAS_COPY_DMA-support.patch > > - Add backend selection for vfio-pci: > 0017-vfio-as-Allow-the-selection-of-a-given-iommu-backend.patch > 0018-vfio-pci-Add-an-iommufd-option.patch > > [1] > https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com > / > [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 > [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 Hi, I had a go with the above branches on our ARM64 platform trying to pass-through a VF dev, but Qemu reports an error as below, [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) I think this happens for the dev BAR addr range. I haven't debugged the kernel yet to see where it actually reports that. Maybe I am missing something. Please let me know. Thanks, Shameer
Hi Shameer, On 4/26/22 11:47 AM, Shameerali Kolothum Thodi wrote: > >> -----Original Message----- >> From: Yi Liu [mailto:yi.l.liu@intel.com] >> Sent: 14 April 2022 11:47 >> To: alex.williamson@redhat.com; cohuck@redhat.com; >> qemu-devel@nongnu.org >> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; >> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger@redhat.com; >> eric.auger.pro@gmail.com; kevin.tian@intel.com; yi.l.liu@intel.com; >> chao.p.peng@intel.com; yi.y.sun@intel.com; peterx@redhat.com >> Subject: [RFC 00/18] vfio: Adopt iommufd >> >> With the introduction of iommufd[1], the linux kernel provides a generic >> interface for userspace drivers to propagate their DMA mappings to kernel >> for assigned devices. This series does the porting of the VFIO devices >> onto the /dev/iommu uapi and let it coexist with the legacy implementation. >> Other devices like vpda, vfio mdev and etc. are not considered yet. >> >> For vfio devices, the new interface is tied with device fd and iommufd >> as the iommufd solution is device-centric. This is different from legacy >> vfio which is group-centric. To support both interfaces in QEMU, this >> series introduces the iommu backend concept in the form of different >> container classes. The existing vfio container is named legacy container >> (equivalent with legacy iommu backend in this series), while the new >> iommufd based container is named as iommufd container (may also be >> mentioned >> as iommufd backend in this series). The two backend types have their own >> way to setup secure context and dma management interface. Below diagram >> shows how it looks like with both BEs. >> >> VFIO >> AddressSpace/Memory >> +-------+ +----------+ +-----+ +-----+ >> | pci | | platform | | ap | | ccw | >> +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ >> | | | | | AddressSpace >> | >> | | | | +------------+---------+ >> +---V-----------V-----------V--------V----+ / >> | VFIOAddressSpace | <------------+ >> | | | MemoryListener >> | VFIOContainer list | >> +-------+----------------------------+----+ >> | | >> | | >> +-------V------+ +--------V----------+ >> | iommufd | | vfio legacy | >> | container | | container | >> +-------+------+ +--------+----------+ >> | | >> | /dev/iommu | /dev/vfio/vfio >> | /dev/vfio/devices/vfioX | /dev/vfio/$group_id >> Userspace | | >> >> ===========+============================+========================== >> ====== >> Kernel | device fd | >> +---------------+ | group/container fd >> | (BIND_IOMMUFD | | >> (SET_CONTAINER/SET_IOMMU) >> | ATTACH_IOAS) | | device fd >> | | | >> | +-------V------------V-----------------+ >> iommufd | | vfio | >> (map/unmap | +---------+--------------------+-------+ >> ioas_copy) | | | map/unmap >> | | | >> +------V------+ +-----V------+ +------V--------+ >> | iommfd core | | device | | vfio iommu | >> +-------------+ +------------+ +---------------+ >> >> [Secure Context setup] >> - iommufd BE: uses device fd and iommufd to setup secure context >> (bind_iommufd, attach_ioas) >> - vfio legacy BE: uses group fd and container fd to setup secure context >> (set_container, set_iommu) >> [Device access] >> - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX >> - vfio legacy BE: device fd is retrieved from group fd ioctl >> [DMA Mapping flow] >> - VFIOAddressSpace receives MemoryRegion add/del via MemoryListener >> - VFIO populates DMA map/unmap via the container BEs >> *) iommufd BE: uses iommufd >> *) vfio legacy BE: uses container fd >> >> This series qomifies the VFIOContainer object which acts as a base class >> for a container. This base class is derived into the legacy VFIO container >> and the new iommufd based container. The base class implements generic >> code >> such as code related to memory_listener and address space management >> whereas >> the derived class implements callbacks that depend on the kernel user space >> being used. >> >> The selection of the backend is made on a device basis using the new >> iommufd option (on/off/auto). By default the iommufd backend is selected >> if supported by the host and by QEMU (iommufd KConfig). This option is >> currently available only for the vfio-pci device. For other types of >> devices, it does not yet exist and the legacy BE is chosen by default. >> >> Test done: >> - PCI and Platform device were tested >> - ccw and ap were only compile-tested >> - limited device hotplug test >> - vIOMMU test run for both legacy and iommufd backends (limited tests) >> >> This series was co-developed by Eric Auger and me based on the exploration >> iommufd kernel[2], complete code of this series is available in[3]. As >> iommufd kernel is in the early step (only iommufd generic interface is in >> mailing list), so this series hasn't made the iommufd backend fully on par >> with legacy backend w.r.t. features like p2p mappings, coherency tracking, >> live migration, etc. This series hasn't supported PCI devices without FLR >> neither as the kernel doesn't support VFIO_DEVICE_PCI_HOT_RESET when >> userspace >> is using iommufd. The kernel needs to be updated to accept device fd list for >> reset when userspace is using iommufd. Related work is in progress by >> Jason[4]. >> >> TODOs: >> - Add DMA alias check for iommufd BE (group level) >> - Make pci.c to be BE agnostic. Needs kernel change as well to fix the >> VFIO_DEVICE_PCI_HOT_RESET gap >> - Cleanup the VFIODevice fields as it's used in both BEs >> - Add locks >> - Replace list with g_tree >> - More tests >> >> Patch Overview: >> >> - Preparation: >> 0001-scripts-update-linux-headers-Add-iommufd.h.patch >> 0002-linux-headers-Import-latest-vfio.h-and-iommufd.h.patch >> 0003-hw-vfio-pci-fix-vfio_pci_hot_reset_result-trace-poin.patch >> 0004-vfio-pci-Use-vbasedev-local-variable-in-vfio_realize.patch >> >> 0005-vfio-common-Rename-VFIOGuestIOMMU-iommu-into-iommu_m.patch >> 0006-vfio-common-Split-common.c-into-common.c-container.c.patch >> >> - Introduce container object and covert existing vfio to use it: >> 0007-vfio-Add-base-object-for-VFIOContainer.patch >> 0008-vfio-container-Introduce-vfio_attach-detach_device.patch >> 0009-vfio-platform-Use-vfio_-attach-detach-_device.patch >> 0010-vfio-ap-Use-vfio_-attach-detach-_device.patch >> 0011-vfio-ccw-Use-vfio_-attach-detach-_device.patch >> 0012-vfio-container-obj-Introduce-attach-detach-_device-c.patch >> 0013-vfio-container-obj-Introduce-VFIOContainer-reset-cal.patch >> >> - Introduce iommufd based container: >> 0014-hw-iommufd-Creation.patch >> 0015-vfio-iommufd-Implement-iommufd-backend.patch >> 0016-vfio-iommufd-Add-IOAS_COPY_DMA-support.patch >> >> - Add backend selection for vfio-pci: >> 0017-vfio-as-Allow-the-selection-of-a-given-iommu-backend.patch >> 0018-vfio-pci-Add-an-iommufd-option.patch >> >> [1] >> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com >> / >> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 >> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 > Hi, > > I had a go with the above branches on our ARM64 platform trying to pass-through > a VF dev, but Qemu reports an error as below, > > [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) > qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address > qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) > > I think this happens for the dev BAR addr range. I haven't debugged the kernel > yet to see where it actually reports that. Does it prevent your assigned device from working? I have such errors too but this is a known issue. This is due to the fact P2P DMA is not supported yet. Thanks Eric > > Maybe I am missing something. Please let me know. > > Thanks, > Shameer >
> -----Original Message-----
> From: Eric Auger [mailto:eric.auger@redhat.com]
> Sent: 26 April 2022 12:45
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi
> Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; cohuck@redhat.com;
> qemu-devel@nongnu.org
> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com;
> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com;
> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org;
> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com;
> kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com;
> peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org>
> Subject: Re: [RFC 00/18] vfio: Adopt iommufd
[...]
> >>
> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com
> >> /
> >> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6
> >> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1
> > Hi,
> >
> > I had a go with the above branches on our ARM64 platform trying to
> pass-through
> > a VF dev, but Qemu reports an error as below,
> >
> > [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002)
> > qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address
> > qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0,
> 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address)
> >
> > I think this happens for the dev BAR addr range. I haven't debugged the
> kernel
> > yet to see where it actually reports that.
> Does it prevent your assigned device from working? I have such errors
> too but this is a known issue. This is due to the fact P2P DMA is not
> supported yet.
>
Yes, the basic tests all good so far. I am still not very clear how it works if
the map() fails though. It looks like it fails in,
iommufd_ioas_map()
iopt_map_user_pages()
iopt_map_pages()
..
pfn_reader_pin_pages()
So does it mean it just works because the page is resident()?
Thanks,
Shameer
On Tue, 26 Apr 2022 12:43:35 +0000 Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: > > -----Original Message----- > > From: Eric Auger [mailto:eric.auger@redhat.com] > > Sent: 26 April 2022 12:45 > > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi > > Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; cohuck@redhat.com; > > qemu-devel@nongnu.org > > Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; > > mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; > > jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; > > jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; > > kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; > > peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org> > > Subject: Re: [RFC 00/18] vfio: Adopt iommufd > > [...] > > > >> > > https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com > > >> / > > >> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 > > >> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 > > > Hi, > > > > > > I had a go with the above branches on our ARM64 platform trying to > > pass-through > > > a VF dev, but Qemu reports an error as below, > > > > > > [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) > > > qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address > > > qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, > > 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) > > > > > > I think this happens for the dev BAR addr range. I haven't debugged the > > kernel > > > yet to see where it actually reports that. > > Does it prevent your assigned device from working? I have such errors > > too but this is a known issue. This is due to the fact P2P DMA is not > > supported yet. > > > > Yes, the basic tests all good so far. I am still not very clear how it works if > the map() fails though. It looks like it fails in, > > iommufd_ioas_map() > iopt_map_user_pages() > iopt_map_pages() > .. > pfn_reader_pin_pages() > > So does it mean it just works because the page is resident()? No, it just means that you're not triggering any accesses that require peer-to-peer DMA support. Any sort of test where the device is only performing DMA to guest RAM, which is by far the standard use case, will work fine. This also doesn't affect vCPU access to BAR space. It's only a failure of the mappings of the BAR space into the IOAS, which is only used when a device tries to directly target another device's BAR space via DMA. Thanks, Alex
Hi, Alex On 2022/4/27 上午12:35, Alex Williamson wrote: > On Tue, 26 Apr 2022 12:43:35 +0000 > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: > >>> -----Original Message----- >>> From: Eric Auger [mailto:eric.auger@redhat.com] >>> Sent: 26 April 2022 12:45 >>> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi >>> Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; cohuck@redhat.com; >>> qemu-devel@nongnu.org >>> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; >>> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >>> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >>> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; >>> kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; >>> peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org> >>> Subject: Re: [RFC 00/18] vfio: Adopt iommufd >> [...] >> >>>>> >>> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com >>>>> / >>>>> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 >>>>> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 >>>> Hi, >>>> >>>> I had a go with the above branches on our ARM64 platform trying to >>> pass-through >>>> a VF dev, but Qemu reports an error as below, >>>> >>>> [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) >>>> qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address >>>> qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, >>> 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) >>>> I think this happens for the dev BAR addr range. I haven't debugged the >>> kernel >>>> yet to see where it actually reports that. >>> Does it prevent your assigned device from working? I have such errors >>> too but this is a known issue. This is due to the fact P2P DMA is not >>> supported yet. >>> >> Yes, the basic tests all good so far. I am still not very clear how it works if >> the map() fails though. It looks like it fails in, >> >> iommufd_ioas_map() >> iopt_map_user_pages() >> iopt_map_pages() >> .. >> pfn_reader_pin_pages() >> >> So does it mean it just works because the page is resident()? > No, it just means that you're not triggering any accesses that require > peer-to-peer DMA support. Any sort of test where the device is only > performing DMA to guest RAM, which is by far the standard use case, > will work fine. This also doesn't affect vCPU access to BAR space. > It's only a failure of the mappings of the BAR space into the IOAS, > which is only used when a device tries to directly target another > device's BAR space via DMA. Thanks, I also get this issue when trying adding prereg listenner + container->prereg_listener = vfio_memory_prereg_listener; + memory_listener_register(&container->prereg_listener, + &address_space_memory); host kernel log: iommufd_ioas_map 1 iova=8000000000, iova1=8000000000, cmd->iova=8000000000, cmd->user_va=9c495000, cmd->length=10000 iopt_alloc_area input area=859a2d00 iova=8000000000 iopt_alloc_area area=859a2d00 iova=8000000000 pin_user_pages_remote rc=-14 qemu log: vfio_prereg_listener_region_add iommufd_map iova=0x8000000000 qemu-system-aarch64: IOMMU_IOAS_MAP failed: Bad address qemu-system-aarch64: vfio_dma_map(0xaaaafb96a930, 0x8000000000, 0x10000, 0xffff9c495000) = -14 (Bad address) qemu-system-aarch64: (null) double free or corruption (fasttop) Aborted (core dumped) With hack of ignoring address 0x8000000000 in map and unmap, kernel can boot. Thanks
Hi Zhangfei, On 2022/5/9 22:24, Zhangfei Gao wrote: > Hi, Alex > > On 2022/4/27 上午12:35, Alex Williamson wrote: >> On Tue, 26 Apr 2022 12:43:35 +0000 >> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: >> >>>> -----Original Message----- >>>> From: Eric Auger [mailto:eric.auger@redhat.com] >>>> Sent: 26 April 2022 12:45 >>>> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; Yi >>>> Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; cohuck@redhat.com; >>>> qemu-devel@nongnu.org >>>> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; >>>> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >>>> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >>>> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; >>>> kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; >>>> peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org> >>>> Subject: Re: [RFC 00/18] vfio: Adopt iommufd >>> [...] >>>> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com >>>>>> / >>>>>> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 >>>>>> [3] https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 >>>>> Hi, >>>>> >>>>> I had a go with the above branches on our ARM64 platform trying to >>>> pass-through >>>>> a VF dev, but Qemu reports an error as below, >>>>> >>>>> [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> 0002) >>>>> qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address >>>>> qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, >>>> 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) >>>>> I think this happens for the dev BAR addr range. I haven't debugged the >>>> kernel >>>>> yet to see where it actually reports that. >>>> Does it prevent your assigned device from working? I have such errors >>>> too but this is a known issue. This is due to the fact P2P DMA is not >>>> supported yet. >>> Yes, the basic tests all good so far. I am still not very clear how it >>> works if >>> the map() fails though. It looks like it fails in, >>> >>> iommufd_ioas_map() >>> iopt_map_user_pages() >>> iopt_map_pages() >>> .. >>> pfn_reader_pin_pages() >>> >>> So does it mean it just works because the page is resident()? >> No, it just means that you're not triggering any accesses that require >> peer-to-peer DMA support. Any sort of test where the device is only >> performing DMA to guest RAM, which is by far the standard use case, >> will work fine. This also doesn't affect vCPU access to BAR space. >> It's only a failure of the mappings of the BAR space into the IOAS, >> which is only used when a device tries to directly target another >> device's BAR space via DMA. Thanks, > > I also get this issue when trying adding prereg listenner > > + container->prereg_listener = vfio_memory_prereg_listener; > + memory_listener_register(&container->prereg_listener, > + &address_space_memory); > > host kernel log: > iommufd_ioas_map 1 iova=8000000000, iova1=8000000000, cmd->iova=8000000000, > cmd->user_va=9c495000, cmd->length=10000 > iopt_alloc_area input area=859a2d00 iova=8000000000 > iopt_alloc_area area=859a2d00 iova=8000000000 > pin_user_pages_remote rc=-14 > > qemu log: > vfio_prereg_listener_region_add > iommufd_map iova=0x8000000000 > qemu-system-aarch64: IOMMU_IOAS_MAP failed: Bad address > qemu-system-aarch64: vfio_dma_map(0xaaaafb96a930, 0x8000000000, 0x10000, > 0xffff9c495000) = -14 (Bad address) > qemu-system-aarch64: (null) > double free or corruption (fasttop) > Aborted (core dumped) > > With hack of ignoring address 0x8000000000 in map and unmap, kernel can boot. do you know if the iova 0x8000000000 guest RAM or MMIO? Currently, iommufd kernel part doesn't support mapping device BAR MMIO. This is a known gap. -- Regards, Yi Liu
Hi Hi, Zhangfei, On 5/10/22 05:17, Yi Liu wrote: > Hi Zhangfei, > > On 2022/5/9 22:24, Zhangfei Gao wrote: >> Hi, Alex >> >> On 2022/4/27 上午12:35, Alex Williamson wrote: >>> On Tue, 26 Apr 2022 12:43:35 +0000 >>> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: >>> >>>>> -----Original Message----- >>>>> From: Eric Auger [mailto:eric.auger@redhat.com] >>>>> Sent: 26 April 2022 12:45 >>>>> To: Shameerali Kolothum Thodi >>>>> <shameerali.kolothum.thodi@huawei.com>; Yi >>>>> Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; >>>>> cohuck@redhat.com; >>>>> qemu-devel@nongnu.org >>>>> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; >>>>> farman@linux.ibm.com; >>>>> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >>>>> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >>>>> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; >>>>> kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; >>>>> peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org> >>>>> Subject: Re: [RFC 00/18] vfio: Adopt iommufd >>>> [...] >>>>> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com >>>>> >>>>>>> / >>>>>>> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 >>>>>>> [3] >>>>>>> https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 >>>>>> Hi, >>>>>> >>>>>> I had a go with the above branches on our ARM64 platform trying to >>>>> pass-through >>>>>> a VF dev, but Qemu reports an error as below, >>>>>> >>>>>> [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> >>>>>> 0002) >>>>>> qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address >>>>>> qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, >>>>> 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) >>>>>> I think this happens for the dev BAR addr range. I haven't >>>>>> debugged the >>>>> kernel >>>>>> yet to see where it actually reports that. >>>>> Does it prevent your assigned device from working? I have such errors >>>>> too but this is a known issue. This is due to the fact P2P DMA is not >>>>> supported yet. >>>> Yes, the basic tests all good so far. I am still not very clear how >>>> it works if >>>> the map() fails though. It looks like it fails in, >>>> >>>> iommufd_ioas_map() >>>> iopt_map_user_pages() >>>> iopt_map_pages() >>>> .. >>>> pfn_reader_pin_pages() >>>> >>>> So does it mean it just works because the page is resident()? >>> No, it just means that you're not triggering any accesses that require >>> peer-to-peer DMA support. Any sort of test where the device is only >>> performing DMA to guest RAM, which is by far the standard use case, >>> will work fine. This also doesn't affect vCPU access to BAR space. >>> It's only a failure of the mappings of the BAR space into the IOAS, >>> which is only used when a device tries to directly target another >>> device's BAR space via DMA. Thanks, >> >> I also get this issue when trying adding prereg listenner >> >> + container->prereg_listener = vfio_memory_prereg_listener; >> + memory_listener_register(&container->prereg_listener, >> + &address_space_memory); >> >> host kernel log: >> iommufd_ioas_map 1 iova=8000000000, iova1=8000000000, >> cmd->iova=8000000000, cmd->user_va=9c495000, cmd->length=10000 >> iopt_alloc_area input area=859a2d00 iova=8000000000 >> iopt_alloc_area area=859a2d00 iova=8000000000 >> pin_user_pages_remote rc=-14 >> >> qemu log: >> vfio_prereg_listener_region_add >> iommufd_map iova=0x8000000000 >> qemu-system-aarch64: IOMMU_IOAS_MAP failed: Bad address >> qemu-system-aarch64: vfio_dma_map(0xaaaafb96a930, 0x8000000000, >> 0x10000, 0xffff9c495000) = -14 (Bad address) >> qemu-system-aarch64: (null) >> double free or corruption (fasttop) >> Aborted (core dumped) >> >> With hack of ignoring address 0x8000000000 in map and unmap, kernel >> can boot. > > do you know if the iova 0x8000000000 guest RAM or MMIO? Currently, > iommufd kernel part doesn't support mapping device BAR MMIO. This is a > known gap. In qemu arm virt machine this indeed matches the PCI MMIO region. Thanks Eric
On 2022/5/10 下午2:51, Eric Auger wrote: > Hi Hi, Zhangfei, > > On 5/10/22 05:17, Yi Liu wrote: >> Hi Zhangfei, >> >> On 2022/5/9 22:24, Zhangfei Gao wrote: >>> Hi, Alex >>> >>> On 2022/4/27 上午12:35, Alex Williamson wrote: >>>> On Tue, 26 Apr 2022 12:43:35 +0000 >>>> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: >>>> >>>>>> -----Original Message----- >>>>>> From: Eric Auger [mailto:eric.auger@redhat.com] >>>>>> Sent: 26 April 2022 12:45 >>>>>> To: Shameerali Kolothum Thodi >>>>>> <shameerali.kolothum.thodi@huawei.com>; Yi >>>>>> Liu <yi.l.liu@intel.com>; alex.williamson@redhat.com; >>>>>> cohuck@redhat.com; >>>>>> qemu-devel@nongnu.org >>>>>> Cc: david@gibson.dropbear.id.au; thuth@redhat.com; >>>>>> farman@linux.ibm.com; >>>>>> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >>>>>> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >>>>>> jgg@nvidia.com; nicolinc@nvidia.com; eric.auger.pro@gmail.com; >>>>>> kevin.tian@intel.com; chao.p.peng@intel.com; yi.y.sun@intel.com; >>>>>> peterx@redhat.com; Zhangfei Gao <zhangfei.gao@linaro.org> >>>>>> Subject: Re: [RFC 00/18] vfio: Adopt iommufd >>>>> [...] >>>>>> https://lore.kernel.org/kvm/0-v1-e79cd8d168e8+6-iommufd_jgg@nvidia.com >>>>>> >>>>>>>> / >>>>>>>> [2] https://github.com/luxis1999/iommufd/tree/iommufd-v5.17-rc6 >>>>>>>> [3] >>>>>>>> https://github.com/luxis1999/qemu/tree/qemu-for-5.17-rc6-vm-rfcv1 >>>>>>> Hi, >>>>>>> >>>>>>> I had a go with the above branches on our ARM64 platform trying to >>>>>> pass-through >>>>>>> a VF dev, but Qemu reports an error as below, >>>>>>> >>>>>>> [ 0.444728] hisi_sec2 0000:00:01.0: enabling device (0000 -> >>>>>>> 0002) >>>>>>> qemu-system-aarch64-iommufd: IOMMU_IOAS_MAP failed: Bad address >>>>>>> qemu-system-aarch64-iommufd: vfio_container_dma_map(0xaaaafeb40ce0, >>>>>> 0x8000000000, 0x10000, 0xffffb40ef000) = -14 (Bad address) >>>>>>> I think this happens for the dev BAR addr range. I haven't >>>>>>> debugged the >>>>>> kernel >>>>>>> yet to see where it actually reports that. >>>>>> Does it prevent your assigned device from working? I have such errors >>>>>> too but this is a known issue. This is due to the fact P2P DMA is not >>>>>> supported yet. >>>>> Yes, the basic tests all good so far. I am still not very clear how >>>>> it works if >>>>> the map() fails though. It looks like it fails in, >>>>> >>>>> iommufd_ioas_map() >>>>> iopt_map_user_pages() >>>>> iopt_map_pages() >>>>> .. >>>>> pfn_reader_pin_pages() >>>>> >>>>> So does it mean it just works because the page is resident()? >>>> No, it just means that you're not triggering any accesses that require >>>> peer-to-peer DMA support. Any sort of test where the device is only >>>> performing DMA to guest RAM, which is by far the standard use case, >>>> will work fine. This also doesn't affect vCPU access to BAR space. >>>> It's only a failure of the mappings of the BAR space into the IOAS, >>>> which is only used when a device tries to directly target another >>>> device's BAR space via DMA. Thanks, >>> I also get this issue when trying adding prereg listenner >>> >>> + container->prereg_listener = vfio_memory_prereg_listener; >>> + memory_listener_register(&container->prereg_listener, >>> + &address_space_memory); >>> >>> host kernel log: >>> iommufd_ioas_map 1 iova=8000000000, iova1=8000000000, >>> cmd->iova=8000000000, cmd->user_va=9c495000, cmd->length=10000 >>> iopt_alloc_area input area=859a2d00 iova=8000000000 >>> iopt_alloc_area area=859a2d00 iova=8000000000 >>> pin_user_pages_remote rc=-14 >>> >>> qemu log: >>> vfio_prereg_listener_region_add >>> iommufd_map iova=0x8000000000 >>> qemu-system-aarch64: IOMMU_IOAS_MAP failed: Bad address >>> qemu-system-aarch64: vfio_dma_map(0xaaaafb96a930, 0x8000000000, >>> 0x10000, 0xffff9c495000) = -14 (Bad address) >>> qemu-system-aarch64: (null) >>> double free or corruption (fasttop) >>> Aborted (core dumped) >>> >>> With hack of ignoring address 0x8000000000 in map and unmap, kernel >>> can boot. >> do you know if the iova 0x8000000000 guest RAM or MMIO? Currently, >> iommufd kernel part doesn't support mapping device BAR MMIO. This is a >> known gap. > In qemu arm virt machine this indeed matches the PCI MMIO region. Thanks Yi and Eric, Then will wait for the updated iommufd kernel for the PCI MMIO region. Another question, How to get the iommu_domain in the ioctl. qemu can get container->ioas_id. kernel can get ioas via the ioas_id. But how to get the domain? Currently I am hacking with ioas->iopt.next_domain_id, which is increasing. domain = xa_load(&ioas->iopt.domains, ioas->iopt.next_domain_id-1); Any idea? Thanks
On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: > Thanks Yi and Eric, > Then will wait for the updated iommufd kernel for the PCI MMIO region. > > Another question, > How to get the iommu_domain in the ioctl. The ID of the iommu_domain (called the hwpt) it should be returned by the vfio attach ioctl. Jason
On 2022/5/10 20:45, Jason Gunthorpe wrote: > On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >> Thanks Yi and Eric, >> Then will wait for the updated iommufd kernel for the PCI MMIO region. >> >> Another question, >> How to get the iommu_domain in the ioctl. > > The ID of the iommu_domain (called the hwpt) it should be returned by > the vfio attach ioctl. yes, hwpt_id is returned by the vfio attach ioctl and recorded in qemu. You can query page table related capabilities with this id. https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ -- Regards, Yi Liu
On 2022/5/10 下午10:08, Yi Liu wrote: > On 2022/5/10 20:45, Jason Gunthorpe wrote: >> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>> Thanks Yi and Eric, >>> Then will wait for the updated iommufd kernel for the PCI MMIO region. >>> >>> Another question, >>> How to get the iommu_domain in the ioctl. >> >> The ID of the iommu_domain (called the hwpt) it should be returned by >> the vfio attach ioctl. > > yes, hwpt_id is returned by the vfio attach ioctl and recorded in > qemu. You can query page table related capabilities with this id. > > https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ > Thanks Yi, Do we use iommufd_hw_pagetable_from_id in kernel? The qemu send hwpt_id via ioctl. Currently VFIOIOMMUFDContainer has hwpt_list, Which member is good to save hwpt_id, IOMMUTLBEntry? In kernel ioctl: iommufd_vfio_ioctl @dev: Device to get an iommu_domain for iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, struct device *dev) But iommufd_vfio_ioctl seems no para dev? Thanks
Hi Zhangfei, On 2022/5/11 22:17, zhangfei.gao@foxmail.com wrote: > > > On 2022/5/10 下午10:08, Yi Liu wrote: >> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>> Thanks Yi and Eric, >>>> Then will wait for the updated iommufd kernel for the PCI MMIO region. >>>> >>>> Another question, >>>> How to get the iommu_domain in the ioctl. >>> >>> The ID of the iommu_domain (called the hwpt) it should be returned by >>> the vfio attach ioctl. >> >> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >> qemu. You can query page table related capabilities with this id. >> >> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >> > Thanks Yi, > > Do we use iommufd_hw_pagetable_from_id in kernel? > > The qemu send hwpt_id via ioctl. > Currently VFIOIOMMUFDContainer has hwpt_list, > Which member is good to save hwpt_id, IOMMUTLBEntry? currently, we don't make use of hwpt yet in the version we have in the qemu branch. I have a change to make use of it. Also, it would be used in future for nested translation setup and also dirty page bit support query for a given domain. > > In kernel ioctl: iommufd_vfio_ioctl > @dev: Device to get an iommu_domain for > iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, struct > device *dev) > But iommufd_vfio_ioctl seems no para dev? there is. you can look at the vfio_group_set_iommufd(), it loops the device_list provided by vfio. And the device info is passed to iommufd. > Thanks > > > > -- Regards, Yi Liu
Hi, Yi On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: > > > On 2022/5/10 下午10:08, Yi Liu wrote: >> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>> Thanks Yi and Eric, >>>> Then will wait for the updated iommufd kernel for the PCI MMIO region. >>>> >>>> Another question, >>>> How to get the iommu_domain in the ioctl. >>> >>> The ID of the iommu_domain (called the hwpt) it should be returned by >>> the vfio attach ioctl. >> >> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >> qemu. You can query page table related capabilities with this id. >> >> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >> > Thanks Yi, > > Do we use iommufd_hw_pagetable_from_id in kernel? > > The qemu send hwpt_id via ioctl. > Currently VFIOIOMMUFDContainer has hwpt_list, > Which member is good to save hwpt_id, IOMMUTLBEntry? Can VFIOIOMMUFDContainer have multi hwpt? Since VFIOIOMMUFDContainer has hwpt_list now. If so, how to get specific hwpt from map/unmap_notify in hw/vfio/as.c, where no vbasedev can be used for compare. I am testing with a workaround, adding VFIOIOASHwpt *hwpt in VFIOIOMMUFDContainer. And save hwpt when vfio_device_attach_container. > > In kernel ioctl: iommufd_vfio_ioctl > @dev: Device to get an iommu_domain for > iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, > struct device *dev) > But iommufd_vfio_ioctl seems no para dev? We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not need dev. iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) Thanks >
Hi Zhangfei, On 2022/5/12 17:01, zhangfei.gao@foxmail.com wrote: > > Hi, Yi > > On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: >> >> >> On 2022/5/10 下午10:08, Yi Liu wrote: >>> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>>> Thanks Yi and Eric, >>>>> Then will wait for the updated iommufd kernel for the PCI MMIO region. >>>>> >>>>> Another question, >>>>> How to get the iommu_domain in the ioctl. >>>> >>>> The ID of the iommu_domain (called the hwpt) it should be returned by >>>> the vfio attach ioctl. >>> >>> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >>> qemu. You can query page table related capabilities with this id. >>> >>> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >>> >> Thanks Yi, >> >> Do we use iommufd_hw_pagetable_from_id in kernel? >> >> The qemu send hwpt_id via ioctl. >> Currently VFIOIOMMUFDContainer has hwpt_list, >> Which member is good to save hwpt_id, IOMMUTLBEntry? > > Can VFIOIOMMUFDContainer have multi hwpt? yes, it is possible > Since VFIOIOMMUFDContainer has hwpt_list now. > If so, how to get specific hwpt from map/unmap_notify in hw/vfio/as.c, > where no vbasedev can be used for compare. > > I am testing with a workaround, adding VFIOIOASHwpt *hwpt in > VFIOIOMMUFDContainer. > And save hwpt when vfio_device_attach_container. > >> >> In kernel ioctl: iommufd_vfio_ioctl >> @dev: Device to get an iommu_domain for >> iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, struct >> device *dev) >> But iommufd_vfio_ioctl seems no para dev? > > We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not need dev. > iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) this is not good. dev is passed in to this function to allocate domain and also check sw_msi things. If you pass in a NULL, it may even unable to get a domain for the hwpt. It won't work I guess. > Thanks >> > -- Regards, Yi Liu
On 2022/5/17 下午4:55, Yi Liu wrote: > Hi Zhangfei, > > On 2022/5/12 17:01, zhangfei.gao@foxmail.com wrote: >> >> Hi, Yi >> >> On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: >>> >>> >>> On 2022/5/10 下午10:08, Yi Liu wrote: >>>> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>>>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>>>> Thanks Yi and Eric, >>>>>> Then will wait for the updated iommufd kernel for the PCI MMIO >>>>>> region. >>>>>> >>>>>> Another question, >>>>>> How to get the iommu_domain in the ioctl. >>>>> >>>>> The ID of the iommu_domain (called the hwpt) it should be returned by >>>>> the vfio attach ioctl. >>>> >>>> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >>>> qemu. You can query page table related capabilities with this id. >>>> >>>> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >>>> >>>> >>> Thanks Yi, >>> >>> Do we use iommufd_hw_pagetable_from_id in kernel? >>> >>> The qemu send hwpt_id via ioctl. >>> Currently VFIOIOMMUFDContainer has hwpt_list, >>> Which member is good to save hwpt_id, IOMMUTLBEntry? >> >> Can VFIOIOMMUFDContainer have multi hwpt? > > yes, it is possible Then how to get hwpt_id in map/unmap_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) > >> Since VFIOIOMMUFDContainer has hwpt_list now. >> If so, how to get specific hwpt from map/unmap_notify in >> hw/vfio/as.c, where no vbasedev can be used for compare. >> >> I am testing with a workaround, adding VFIOIOASHwpt *hwpt in >> VFIOIOMMUFDContainer. >> And save hwpt when vfio_device_attach_container. >> >>> >>> In kernel ioctl: iommufd_vfio_ioctl >>> @dev: Device to get an iommu_domain for >>> iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, >>> struct device *dev) >>> But iommufd_vfio_ioctl seems no para dev? >> >> We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not need dev. >> iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) > > this is not good. dev is passed in to this function to allocate domain > and also check sw_msi things. If you pass in a NULL, it may even unable > to get a domain for the hwpt. It won't work I guess. The iommufd_hw_pagetable_from_id can be used for 1, allocate domain, which need para dev case IOMMUFD_OBJ_IOAS hwpt = iommufd_hw_pagetable_auto_get(ictx, ioas, dev); 2. Just return allocated domain via hwpt_id, which does not need dev. case IOMMUFD_OBJ_HW_PAGETABLE: return container_of(obj, struct iommufd_hw_pagetable, obj); By the way, any plan of the nested mode? Thanks
On 2022/5/18 15:22, zhangfei.gao@foxmail.com wrote: > > > On 2022/5/17 下午4:55, Yi Liu wrote: >> Hi Zhangfei, >> >> On 2022/5/12 17:01, zhangfei.gao@foxmail.com wrote: >>> >>> Hi, Yi >>> >>> On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: >>>> >>>> >>>> On 2022/5/10 下午10:08, Yi Liu wrote: >>>>> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>>>>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>>>>> Thanks Yi and Eric, >>>>>>> Then will wait for the updated iommufd kernel for the PCI MMIO region. >>>>>>> >>>>>>> Another question, >>>>>>> How to get the iommu_domain in the ioctl. >>>>>> >>>>>> The ID of the iommu_domain (called the hwpt) it should be returned by >>>>>> the vfio attach ioctl. >>>>> >>>>> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >>>>> qemu. You can query page table related capabilities with this id. >>>>> >>>>> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >>>>> >>>> Thanks Yi, >>>> >>>> Do we use iommufd_hw_pagetable_from_id in kernel? >>>> >>>> The qemu send hwpt_id via ioctl. >>>> Currently VFIOIOMMUFDContainer has hwpt_list, >>>> Which member is good to save hwpt_id, IOMMUTLBEntry? >>> >>> Can VFIOIOMMUFDContainer have multi hwpt? >> >> yes, it is possible > Then how to get hwpt_id in map/unmap_notify(IOMMUNotifier *n, IOMMUTLBEntry > *iotlb) in map/unmap, should use ioas_id instead of hwpt_id > >> >>> Since VFIOIOMMUFDContainer has hwpt_list now. >>> If so, how to get specific hwpt from map/unmap_notify in hw/vfio/as.c, >>> where no vbasedev can be used for compare. >>> >>> I am testing with a workaround, adding VFIOIOASHwpt *hwpt in >>> VFIOIOMMUFDContainer. >>> And save hwpt when vfio_device_attach_container. >>> >>>> >>>> In kernel ioctl: iommufd_vfio_ioctl >>>> @dev: Device to get an iommu_domain for >>>> iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, >>>> struct device *dev) >>>> But iommufd_vfio_ioctl seems no para dev? >>> >>> We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not need dev. >>> iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) >> >> this is not good. dev is passed in to this function to allocate domain >> and also check sw_msi things. If you pass in a NULL, it may even unable >> to get a domain for the hwpt. It won't work I guess. > > The iommufd_hw_pagetable_from_id can be used for > 1, allocate domain, which need para dev > case IOMMUFD_OBJ_IOAS > hwpt = iommufd_hw_pagetable_auto_get(ictx, ioas, dev); this is used when attaching ioas. > 2. Just return allocated domain via hwpt_id, which does not need dev. > case IOMMUFD_OBJ_HW_PAGETABLE: > return container_of(obj, struct iommufd_hw_pagetable, obj); yes, this would be the usage in nesting. you may check my below branch. It's for nesting integration. https://github.com/luxis1999/iommufd/tree/iommufd-v5.18-rc4-nesting > By the way, any plan of the nested mode? I'm working with Eric, Nic on it. Currently, I've got the above kernel branch, QEMU side is also WIP. > Thanks -- Regards, Yi Liu
> -----Original Message----- > From: Yi Liu [mailto:yi.l.liu@intel.com] > Sent: 18 May 2022 15:01 > To: zhangfei.gao@foxmail.com; Jason Gunthorpe <jgg@nvidia.com>; > Zhangfei Gao <zhangfei.gao@linaro.org> > Cc: eric.auger@redhat.com; Alex Williamson <alex.williamson@redhat.com>; > Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; > cohuck@redhat.com; qemu-devel@nongnu.org; > david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; > mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; > jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; > nicolinc@nvidia.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; > chao.p.peng@intel.com; yi.y.sun@intel.com; peterx@redhat.com > Subject: Re: [RFC 00/18] vfio: Adopt iommufd > > On 2022/5/18 15:22, zhangfei.gao@foxmail.com wrote: > > > > > > On 2022/5/17 下午4:55, Yi Liu wrote: > >> Hi Zhangfei, > >> > >> On 2022/5/12 17:01, zhangfei.gao@foxmail.com wrote: > >>> > >>> Hi, Yi > >>> > >>> On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: > >>>> > >>>> > >>>> On 2022/5/10 下午10:08, Yi Liu wrote: > >>>>> On 2022/5/10 20:45, Jason Gunthorpe wrote: > >>>>>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: > >>>>>>> Thanks Yi and Eric, > >>>>>>> Then will wait for the updated iommufd kernel for the PCI MMIO > region. > >>>>>>> > >>>>>>> Another question, > >>>>>>> How to get the iommu_domain in the ioctl. > >>>>>> > >>>>>> The ID of the iommu_domain (called the hwpt) it should be returned > by > >>>>>> the vfio attach ioctl. > >>>>> > >>>>> yes, hwpt_id is returned by the vfio attach ioctl and recorded in > >>>>> qemu. You can query page table related capabilities with this id. > >>>>> > >>>>> > https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ > >>>>> > >>>> Thanks Yi, > >>>> > >>>> Do we use iommufd_hw_pagetable_from_id in kernel? > >>>> > >>>> The qemu send hwpt_id via ioctl. > >>>> Currently VFIOIOMMUFDContainer has hwpt_list, > >>>> Which member is good to save hwpt_id, IOMMUTLBEntry? > >>> > >>> Can VFIOIOMMUFDContainer have multi hwpt? > >> > >> yes, it is possible > > Then how to get hwpt_id in map/unmap_notify(IOMMUNotifier *n, > IOMMUTLBEntry > > *iotlb) > > in map/unmap, should use ioas_id instead of hwpt_id > > > > >> > >>> Since VFIOIOMMUFDContainer has hwpt_list now. > >>> If so, how to get specific hwpt from map/unmap_notify in hw/vfio/as.c, > >>> where no vbasedev can be used for compare. > >>> > >>> I am testing with a workaround, adding VFIOIOASHwpt *hwpt in > >>> VFIOIOMMUFDContainer. > >>> And save hwpt when vfio_device_attach_container. > >>> > >>>> > >>>> In kernel ioctl: iommufd_vfio_ioctl > >>>> @dev: Device to get an iommu_domain for > >>>> iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, > >>>> struct device *dev) > >>>> But iommufd_vfio_ioctl seems no para dev? > >>> > >>> We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not > need dev. > >>> iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) > >> > >> this is not good. dev is passed in to this function to allocate domain > >> and also check sw_msi things. If you pass in a NULL, it may even unable > >> to get a domain for the hwpt. It won't work I guess. > > > > The iommufd_hw_pagetable_from_id can be used for > > 1, allocate domain, which need para dev > > case IOMMUFD_OBJ_IOAS > > hwpt = iommufd_hw_pagetable_auto_get(ictx, ioas, dev); > > this is used when attaching ioas. > > > 2. Just return allocated domain via hwpt_id, which does not need dev. > > case IOMMUFD_OBJ_HW_PAGETABLE: > > return container_of(obj, struct iommufd_hw_pagetable, obj); > > yes, this would be the usage in nesting. you may check my below > branch. It's for nesting integration. > > https://github.com/luxis1999/iommufd/tree/iommufd-v5.18-rc4-nesting > > > By the way, any plan of the nested mode? > I'm working with Eric, Nic on it. Currently, I've got the above kernel > branch, QEMU side is also WIP. Hi Yi/Eric, I had a look at the above nesting kernel and Qemu branches and as mentioned in the cover letter it is not working on ARM yet. IIUC, to get it working via the iommufd the main thing is we need a way to configure the phys SMMU in nested mode and setup the mappings for the stage 2. The Cache/PASID related changes looks more straight forward. I had quite a few hacks to get it working on ARM, but still a WIP. So just wondering do you guys have something that can be shared yet? Please let me know. Thanks, Shameer
Hi Shameer, On 6/28/22 10:14, Shameerali Kolothum Thodi wrote: > >> -----Original Message----- >> From: Yi Liu [mailto:yi.l.liu@intel.com] >> Sent: 18 May 2022 15:01 >> To: zhangfei.gao@foxmail.com; Jason Gunthorpe <jgg@nvidia.com>; >> Zhangfei Gao <zhangfei.gao@linaro.org> >> Cc: eric.auger@redhat.com; Alex Williamson <alex.williamson@redhat.com>; >> Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; >> cohuck@redhat.com; qemu-devel@nongnu.org; >> david@gibson.dropbear.id.au; thuth@redhat.com; farman@linux.ibm.com; >> mjrosato@linux.ibm.com; akrowiak@linux.ibm.com; pasic@linux.ibm.com; >> jjherne@linux.ibm.com; jasowang@redhat.com; kvm@vger.kernel.org; >> nicolinc@nvidia.com; eric.auger.pro@gmail.com; kevin.tian@intel.com; >> chao.p.peng@intel.com; yi.y.sun@intel.com; peterx@redhat.com >> Subject: Re: [RFC 00/18] vfio: Adopt iommufd >> >> On 2022/5/18 15:22, zhangfei.gao@foxmail.com wrote: >>> >>> On 2022/5/17 下午4:55, Yi Liu wrote: >>>> Hi Zhangfei, >>>> >>>> On 2022/5/12 17:01, zhangfei.gao@foxmail.com wrote: >>>>> Hi, Yi >>>>> >>>>> On 2022/5/11 下午10:17, zhangfei.gao@foxmail.com wrote: >>>>>> >>>>>> On 2022/5/10 下午10:08, Yi Liu wrote: >>>>>>> On 2022/5/10 20:45, Jason Gunthorpe wrote: >>>>>>>> On Tue, May 10, 2022 at 08:35:00PM +0800, Zhangfei Gao wrote: >>>>>>>>> Thanks Yi and Eric, >>>>>>>>> Then will wait for the updated iommufd kernel for the PCI MMIO >> region. >>>>>>>>> Another question, >>>>>>>>> How to get the iommu_domain in the ioctl. >>>>>>>> The ID of the iommu_domain (called the hwpt) it should be returned >> by >>>>>>>> the vfio attach ioctl. >>>>>>> yes, hwpt_id is returned by the vfio attach ioctl and recorded in >>>>>>> qemu. You can query page table related capabilities with this id. >>>>>>> >>>>>>> >> https://lore.kernel.org/kvm/20220414104710.28534-16-yi.l.liu@intel.com/ >>>>>> Thanks Yi, >>>>>> >>>>>> Do we use iommufd_hw_pagetable_from_id in kernel? >>>>>> >>>>>> The qemu send hwpt_id via ioctl. >>>>>> Currently VFIOIOMMUFDContainer has hwpt_list, >>>>>> Which member is good to save hwpt_id, IOMMUTLBEntry? >>>>> Can VFIOIOMMUFDContainer have multi hwpt? >>>> yes, it is possible >>> Then how to get hwpt_id in map/unmap_notify(IOMMUNotifier *n, >> IOMMUTLBEntry >>> *iotlb) >> in map/unmap, should use ioas_id instead of hwpt_id >> >>>>> Since VFIOIOMMUFDContainer has hwpt_list now. >>>>> If so, how to get specific hwpt from map/unmap_notify in hw/vfio/as.c, >>>>> where no vbasedev can be used for compare. >>>>> >>>>> I am testing with a workaround, adding VFIOIOASHwpt *hwpt in >>>>> VFIOIOMMUFDContainer. >>>>> And save hwpt when vfio_device_attach_container. >>>>> >>>>>> In kernel ioctl: iommufd_vfio_ioctl >>>>>> @dev: Device to get an iommu_domain for >>>>>> iommufd_hw_pagetable_from_id(struct iommufd_ctx *ictx, u32 pt_id, >>>>>> struct device *dev) >>>>>> But iommufd_vfio_ioctl seems no para dev? >>>>> We can set dev=Null since IOMMUFD_OBJ_HW_PAGETABLE does not >> need dev. >>>>> iommufd_hw_pagetable_from_id(ictx, hwpt_id, NULL) >>>> this is not good. dev is passed in to this function to allocate domain >>>> and also check sw_msi things. If you pass in a NULL, it may even unable >>>> to get a domain for the hwpt. It won't work I guess. >>> The iommufd_hw_pagetable_from_id can be used for >>> 1, allocate domain, which need para dev >>> case IOMMUFD_OBJ_IOAS >>> hwpt = iommufd_hw_pagetable_auto_get(ictx, ioas, dev); >> this is used when attaching ioas. >> >>> 2. Just return allocated domain via hwpt_id, which does not need dev. >>> case IOMMUFD_OBJ_HW_PAGETABLE: >>> return container_of(obj, struct iommufd_hw_pagetable, obj); >> yes, this would be the usage in nesting. you may check my below >> branch. It's for nesting integration. >> >> https://github.com/luxis1999/iommufd/tree/iommufd-v5.18-rc4-nesting >> >>> By the way, any plan of the nested mode? >> I'm working with Eric, Nic on it. Currently, I've got the above kernel >> branch, QEMU side is also WIP. > Hi Yi/Eric, > > I had a look at the above nesting kernel and Qemu branches and as mentioned > in the cover letter it is not working on ARM yet. > > IIUC, to get it working via the iommufd the main thing is we need a way to configure > the phys SMMU in nested mode and setup the mappings for the stage 2. The > Cache/PASID related changes looks more straight forward. > > I had quite a few hacks to get it working on ARM, but still a WIP. So just wondering > do you guys have something that can be shared yet? I am working on the respin based on latest iommufd kernel branches and qemu RFC v2 but it is still WIP. I will share as soon as possible. Eric > > Please let me know. > > Thanks, > Shameer
© 2016 - 2026 Red Hat, Inc.