From: Alistair Francis <alistair.francis@wdc.com>

This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
endpoint. This is a similar approach to the NVMe PCIe Endpoint
(drivers/nvme/target/pci-epf.c) but for SCSI.

This does end up being somewhat similar to the pci-epf.c code, but
re-written for SCSI.

This approach allows a PCIe Endpoint device (tested on a
radxa-rock5b) to setup what appears to be a SCSI device, using an
existing SCSI backend (tested using scsi_debug).

At this point a host can connect over PCIe, ensure virtio_pci and
virtio_scsi is loaded and on PCIe rescan will see a scsi device.

There are a few pain points with this approach though:
 1. We have to use the Legacy SCSI VirtIO driver. This is because the
    Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
    capabilities. So we can't advertise the VirtIO Common configuration
    capability, which means we can't be a modern VirtIO SCSI device.

    This is unfortunate, but there doesn't seem to be any way around
    this, at least with the current hardware.

 1.2. Legacy virtio devices only have 32 feature bits and therefore can't
    set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
    vring_use_map_api() function will return false.

    Currently Linux endpoint devices use the legacy virtio interface as
    they aren't able to advertise the Common configuration capability.
    As most PCI endpoint capable PCIe controllers do not allow modifying the
    capability list, and thus are unable to advertise the Common configuration
    capability. This means the device's inbound TLPs fault on the host
    SMMU because the vring descriptors carry raw physical addresses.

    This series adds a quirk that forces a subset of legacy virtio devices
    to use the DMA Map API (vring_use_map_api() will return true),
    which fixes this issue.

    It's unideal that we have to hard code a quirk to basically just
    advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
    are stuck with legacy virtio devices there isn't much else we can do.

 2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
    respond fast enough to the host. This means we effectivly burn a CPU
    to read and write some values. But as there are no intterupts
    generated on these events and we need to be very quick there isn't
    another option.

With two Raxda Rock5bs connected together and the IOMMU turned off I see
performance numbers like this

root@radxa-rock5b:~# fio-test.sh /dev/sda
Running on /dev/sda...
  Rnd read,    4KB,  QD=1, 1 job :  IOPS=223, BW=895KiB/s (917kB/s)
  Rnd read,    4KB, QD=32, 1 job :  IOPS=7415, BW=29.0MiB/s (30.4MB/s)
  Rnd read,    4KB, QD=32, 4 jobs:  IOPS=20.2k, BW=79.0MiB/s (82.8MB/s)
  Rnd read,  128KB,  QD=1, 1 job :  IOPS=213, BW=26.7MiB/s (28.0MB/s)
  Rnd read,  128KB, QD=32, 1 job :  IOPS=1482, BW=185MiB/s (194MB/s)
  Rnd read,  128KB, QD=32, 4 jobs:  IOPS=2599, BW=325MiB/s (341MB/s)
  Rnd read,  512KB,  QD=1, 1 job :  IOPS=184, BW=92.1MiB/s (96.6MB/s)
  Rnd read,  512KB, QD=32, 1 job :  IOPS=1079, BW=540MiB/s (566MB/s)
  Rnd read,  512KB, QD=32, 4 jobs:  IOPS=1284, BW=642MiB/s (674MB/s)
  Rnd write,   4KB,  QD=1, 1 job :  IOPS=222, BW=889KiB/s (911kB/s)
  Rnd write,   4KB, QD=32, 1 job :  IOPS=7433, BW=29.0MiB/s (30.4MB/s)
  Rnd write,   4KB, QD=32, 4 jobs:  IOPS=20.3k, BW=79.1MiB/s (83.0MB/s)
  Rnd write, 128KB,  QD=1, 1 job :  IOPS=203, BW=25.5MiB/s (26.7MB/s)
  Rnd write, 128KB, QD=32, 1 job :  IOPS=1521, BW=190MiB/s (199MB/s)
  Rnd write, 128KB, QD=32, 4 jobs:  IOPS=2927, BW=366MiB/s (384MB/s)
  Seq read,  128KB,  QD=1, 1 job :  IOPS=207, BW=25.9MiB/s (27.2MB/s)
  Seq read,  128KB, QD=32, 1 job :  IOPS=1538, BW=192MiB/s (202MB/s)
  Seq read,  512KB,  QD=1, 1 job :  IOPS=183, BW=91.7MiB/s (96.2MB/s)
  Seq read,  512KB, QD=32, 1 job :  IOPS=1356, BW=678MiB/s (711MB/s)
  Seq read,    1MB, QD=32, 1 job :  IOPS=646, BW=647MiB/s (678MB/s)
  Seq write, 128KB,  QD=1, 1 job :  IOPS=209, BW=26.1MiB/s (27.4MB/s)
  Seq write, 128KB, QD=32, 1 job :  IOPS=1576, BW=197MiB/s (207MB/s)
  Seq write, 512KB,  QD=1, 1 job :  IOPS=166, BW=83.3MiB/s (87.4MB/s)
  Seq write, 512KB, QD=32, 1 job :  IOPS=891, BW=446MiB/s (468MB/s)
  Seq write,   1MB, QD=32, 1 job :  IOPS=539, BW=540MiB/s (566MB/s)
  Rnd rdwr, 4K..1MB, QD=8, 4 jobs:  IOPS=453, BW=228MiB/s (239MB/s)
 IOPS=478, BW=241MiB/s (253MB/s)

claude-opus-4-8 was used to parse the crash dumps and IOMMU faults
during testing to narrow down where issues where are how to fix them

Alistair Francis (2):
  virtio_pci: Add a quirk to force DMA Map API for certain legacy
    devices
  scsi: Initial commit of VirtIO PCIe Endpoint Driver

 drivers/scsi/Kconfig               |   12 +
 drivers/scsi/Makefile              |    1 +
 drivers/scsi/virtio-scsi-pci-epf.c | 3099 ++++++++++++++++++++++++++++
 drivers/virtio/virtio_pci_legacy.c |   27 +
 drivers/virtio/virtio_ring.c       |    7 +
 include/linux/virtio.h             |    5 +
 6 files changed, 3151 insertions(+)
 create mode 100644 drivers/scsi/virtio-scsi-pci-epf.c

-- 
2.55.0

On 9/1/26 10:46, alistair23@gmail.com wrote:
> From: Alistair Francis <alistair.francis@wdc.com>
> 
> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> endpoint. This is a similar approach to the NVMe PCIe Endpoint
> (drivers/nvme/target/pci-epf.c) but for SCSI.
> 
> This does end up being somewhat similar to the pci-epf.c code, but
> re-written for SCSI.
> 
> This approach allows a PCIe Endpoint device (tested on a
> radxa-rock5b) to setup what appears to be a SCSI device, using an
> existing SCSI backend (tested using scsi_debug).
> 
> At this point a host can connect over PCIe, ensure virtio_pci and
> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> 
> There are a few pain points with this approach though:
>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
>     capabilities. So we can't advertise the VirtIO Common configuration
>     capability, which means we can't be a modern VirtIO SCSI device.
> 
>     This is unfortunate, but there doesn't seem to be any way around
>     this, at least with the current hardware.
> 
>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
>     vring_use_map_api() function will return false.
> 
>     Currently Linux endpoint devices use the legacy virtio interface as
>     they aren't able to advertise the Common configuration capability.
>     As most PCI endpoint capable PCIe controllers do not allow modifying the
>     capability list, and thus are unable to advertise the Common configuration
>     capability. This means the device's inbound TLPs fault on the host
>     SMMU because the vring descriptors carry raw physical addresses.
> 
>     This series adds a quirk that forces a subset of legacy virtio devices
>     to use the DMA Map API (vring_use_map_api() will return true),
>     which fixes this issue.
> 
>     It's unideal that we have to hard code a quirk to basically just
>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
>     are stuck with legacy virtio devices there isn't much else we can do.
> 
>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
>     respond fast enough to the host. This means we effectivly burn a CPU
>     to read and write some values. But as there are no intterupts
>     generated on these events and we need to be very quick there isn't
>     another option.
> 
> With two Raxda Rock5bs connected together and the IOMMU turned off I see
> performance numbers like this
> 
> root@radxa-rock5b:~# fio-test.sh /dev/sda
> Running on /dev/sda...
>   Rnd read,    4KB,  QD=1, 1 job :  IOPS=223, BW=895KiB/s (917kB/s)
>   Rnd read,    4KB, QD=32, 1 job :  IOPS=7415, BW=29.0MiB/s (30.4MB/s)
>   Rnd read,    4KB, QD=32, 4 jobs:  IOPS=20.2k, BW=79.0MiB/s (82.8MB/s)
>   Rnd read,  128KB,  QD=1, 1 job :  IOPS=213, BW=26.7MiB/s (28.0MB/s)
>   Rnd read,  128KB, QD=32, 1 job :  IOPS=1482, BW=185MiB/s (194MB/s)
>   Rnd read,  128KB, QD=32, 4 jobs:  IOPS=2599, BW=325MiB/s (341MB/s)
>   Rnd read,  512KB,  QD=1, 1 job :  IOPS=184, BW=92.1MiB/s (96.6MB/s)
>   Rnd read,  512KB, QD=32, 1 job :  IOPS=1079, BW=540MiB/s (566MB/s)
>   Rnd read,  512KB, QD=32, 4 jobs:  IOPS=1284, BW=642MiB/s (674MB/s)
>   Rnd write,   4KB,  QD=1, 1 job :  IOPS=222, BW=889KiB/s (911kB/s)
>   Rnd write,   4KB, QD=32, 1 job :  IOPS=7433, BW=29.0MiB/s (30.4MB/s)
>   Rnd write,   4KB, QD=32, 4 jobs:  IOPS=20.3k, BW=79.1MiB/s (83.0MB/s)
>   Rnd write, 128KB,  QD=1, 1 job :  IOPS=203, BW=25.5MiB/s (26.7MB/s)
>   Rnd write, 128KB, QD=32, 1 job :  IOPS=1521, BW=190MiB/s (199MB/s)
>   Rnd write, 128KB, QD=32, 4 jobs:  IOPS=2927, BW=366MiB/s (384MB/s)
>   Seq read,  128KB,  QD=1, 1 job :  IOPS=207, BW=25.9MiB/s (27.2MB/s)
>   Seq read,  128KB, QD=32, 1 job :  IOPS=1538, BW=192MiB/s (202MB/s)
>   Seq read,  512KB,  QD=1, 1 job :  IOPS=183, BW=91.7MiB/s (96.2MB/s)
>   Seq read,  512KB, QD=32, 1 job :  IOPS=1356, BW=678MiB/s (711MB/s)
>   Seq read,    1MB, QD=32, 1 job :  IOPS=646, BW=647MiB/s (678MB/s)
>   Seq write, 128KB,  QD=1, 1 job :  IOPS=209, BW=26.1MiB/s (27.4MB/s)
>   Seq write, 128KB, QD=32, 1 job :  IOPS=1576, BW=197MiB/s (207MB/s)
>   Seq write, 512KB,  QD=1, 1 job :  IOPS=166, BW=83.3MiB/s (87.4MB/s)
>   Seq write, 512KB, QD=32, 1 job :  IOPS=891, BW=446MiB/s (468MB/s)
>   Seq write,   1MB, QD=32, 1 job :  IOPS=539, BW=540MiB/s (566MB/s)
>   Rnd rdwr, 4K..1MB, QD=8, 4 jobs:  IOPS=453, BW=228MiB/s (239MB/s)
>  IOPS=478, BW=241MiB/s (253MB/s)
> 
> claude-opus-4-8 was used to parse the crash dumps and IOMMU faults
> during testing to narrow down where issues where are how to fix them
> 
> Alistair Francis (2):
>   virtio_pci: Add a quirk to force DMA Map API for certain legacy
>     devices
>   scsi: Initial commit of VirtIO PCIe Endpoint Driver
> 
>  drivers/scsi/Kconfig               |   12 +
>  drivers/scsi/Makefile              |    1 +
>  drivers/scsi/virtio-scsi-pci-epf.c | 3099 ++++++++++++++++++++++++++++
>  drivers/virtio/virtio_pci_legacy.c |   27 +
>  drivers/virtio/virtio_ring.c       |    7 +
>  include/linux/virtio.h             |    5 +
>  6 files changed, 3151 insertions(+)
>  create mode 100644 drivers/scsi/virtio-scsi-pci-epf.c

We need this documented too, with a reference to it in the PCI/endpoint
documentation.
See Documentation/PCI/endpoint/index.rst for what I did for the nvme endpoint
driver, with the main documentation for that driver in
Documentation/nvme/nvme-pci-endpoint-target.rst.

For this one, maybe add a similar file under Documentation/scsi ?


-- 
Damien Le Moal
Western Digital Research

On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
> From: Alistair Francis <alistair.francis@wdc.com>
> 
> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> endpoint. This is a similar approach to the NVMe PCIe Endpoint
> (drivers/nvme/target/pci-epf.c) but for SCSI.
> 
> This does end up being somewhat similar to the pci-epf.c code, but
> re-written for SCSI.
> 
> This approach allows a PCIe Endpoint device (tested on a
> radxa-rock5b) to setup what appears to be a SCSI device, using an
> existing SCSI backend (tested using scsi_debug).
> 
> At this point a host can connect over PCIe, ensure virtio_pci and
> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> 
> There are a few pain points with this approach though:
>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
>     capabilities. So we can't advertise the VirtIO Common configuration
>     capability, which means we can't be a modern VirtIO SCSI device.
> 
>     This is unfortunate, but there doesn't seem to be any way around
>     this, at least with the current hardware.
> 
>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
>     vring_use_map_api() function will return false.
> 
>     Currently Linux endpoint devices use the legacy virtio interface as
>     they aren't able to advertise the Common configuration capability.
>     As most PCI endpoint capable PCIe controllers do not allow modifying the
>     capability list, and thus are unable to advertise the Common configuration
>     capability. This means the device's inbound TLPs fault on the host
>     SMMU because the vring descriptors carry raw physical addresses.
> 
>     This series adds a quirk that forces a subset of legacy virtio devices
>     to use the DMA Map API (vring_use_map_api() will return true),
>     which fixes this issue.
> 
>     It's unideal that we have to hard code a quirk to basically just
>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
>     are stuck with legacy virtio devices there isn't much else we can do.
> 
>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
>     respond fast enough to the host. This means we effectivly burn a CPU
>     to read and write some values. But as there are no intterupts
>     generated on these events and we need to be very quick there isn't
>     another option.
> 
> With two Raxda Rock5bs connected together and the IOMMU turned off I see
> performance numbers like this
> 
> root@radxa-rock5b:~# fio-test.sh /dev/sda
> Running on /dev/sda...
>   Rnd read,    4KB,  QD=1, 1 job :  IOPS=223, BW=895KiB/s (917kB/s)
>   Rnd read,    4KB, QD=32, 1 job :  IOPS=7415, BW=29.0MiB/s (30.4MB/s)
>   Rnd read,    4KB, QD=32, 4 jobs:  IOPS=20.2k, BW=79.0MiB/s (82.8MB/s)
>   Rnd read,  128KB,  QD=1, 1 job :  IOPS=213, BW=26.7MiB/s (28.0MB/s)
>   Rnd read,  128KB, QD=32, 1 job :  IOPS=1482, BW=185MiB/s (194MB/s)
>   Rnd read,  128KB, QD=32, 4 jobs:  IOPS=2599, BW=325MiB/s (341MB/s)
>   Rnd read,  512KB,  QD=1, 1 job :  IOPS=184, BW=92.1MiB/s (96.6MB/s)
>   Rnd read,  512KB, QD=32, 1 job :  IOPS=1079, BW=540MiB/s (566MB/s)
>   Rnd read,  512KB, QD=32, 4 jobs:  IOPS=1284, BW=642MiB/s (674MB/s)
>   Rnd write,   4KB,  QD=1, 1 job :  IOPS=222, BW=889KiB/s (911kB/s)
>   Rnd write,   4KB, QD=32, 1 job :  IOPS=7433, BW=29.0MiB/s (30.4MB/s)
>   Rnd write,   4KB, QD=32, 4 jobs:  IOPS=20.3k, BW=79.1MiB/s (83.0MB/s)
>   Rnd write, 128KB,  QD=1, 1 job :  IOPS=203, BW=25.5MiB/s (26.7MB/s)
>   Rnd write, 128KB, QD=32, 1 job :  IOPS=1521, BW=190MiB/s (199MB/s)
>   Rnd write, 128KB, QD=32, 4 jobs:  IOPS=2927, BW=366MiB/s (384MB/s)
>   Seq read,  128KB,  QD=1, 1 job :  IOPS=207, BW=25.9MiB/s (27.2MB/s)
>   Seq read,  128KB, QD=32, 1 job :  IOPS=1538, BW=192MiB/s (202MB/s)
>   Seq read,  512KB,  QD=1, 1 job :  IOPS=183, BW=91.7MiB/s (96.2MB/s)
>   Seq read,  512KB, QD=32, 1 job :  IOPS=1356, BW=678MiB/s (711MB/s)
>   Seq read,    1MB, QD=32, 1 job :  IOPS=646, BW=647MiB/s (678MB/s)
>   Seq write, 128KB,  QD=1, 1 job :  IOPS=209, BW=26.1MiB/s (27.4MB/s)
>   Seq write, 128KB, QD=32, 1 job :  IOPS=1576, BW=197MiB/s (207MB/s)
>   Seq write, 512KB,  QD=1, 1 job :  IOPS=166, BW=83.3MiB/s (87.4MB/s)
>   Seq write, 512KB, QD=32, 1 job :  IOPS=891, BW=446MiB/s (468MB/s)
>   Seq write,   1MB, QD=32, 1 job :  IOPS=539, BW=540MiB/s (566MB/s)
>   Rnd rdwr, 4K..1MB, QD=8, 4 jobs:  IOPS=453, BW=228MiB/s (239MB/s)
>  IOPS=478, BW=241MiB/s (253MB/s)
> 
> claude-opus-4-8 was used to parse the crash dumps and IOMMU faults
> during testing to narrow down where issues where are how to fix them
> 
> Alistair Francis (2):
>   virtio_pci: Add a quirk to force DMA Map API for certain legacy
>     devices
>   scsi: Initial commit of VirtIO PCIe Endpoint Driver
> 
>  drivers/scsi/Kconfig               |   12 +
>  drivers/scsi/Makefile              |    1 +
>  drivers/scsi/virtio-scsi-pci-epf.c | 3099 ++++++++++++++++++++++++++++
>  drivers/virtio/virtio_pci_legacy.c |   27 +
>  drivers/virtio/virtio_ring.c       |    7 +
>  include/linux/virtio.h             |    5 +
>  6 files changed, 3151 insertions(+)
>  create mode 100644 drivers/scsi/virtio-scsi-pci-epf.c
> 
> -- 
> 2.55.0
> 

+ linux-pci
+ PCI endpoint folks, Mani and Shunsuke have done a few presentations at
  various conferences about virtio/virtio-net/virtio-console + the PCI
  endpoint subsystem.

On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
> On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
> > From: Alistair Francis <alistair.francis@wdc.com>
> > 
> > This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> > endpoint. This is a similar approach to the NVMe PCIe Endpoint
> > (drivers/nvme/target/pci-epf.c) but for SCSI.
> > 
> > This does end up being somewhat similar to the pci-epf.c code, but
> > re-written for SCSI.
> > 
> > This approach allows a PCIe Endpoint device (tested on a
> > radxa-rock5b) to setup what appears to be a SCSI device, using an
> > existing SCSI backend (tested using scsi_debug).
> > 
> > At this point a host can connect over PCIe, ensure virtio_pci and
> > virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> > 
> > There are a few pain points with this approach though:
> >  1. We have to use the Legacy SCSI VirtIO driver. This is because the
> >     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
> >     capabilities. So we can't advertise the VirtIO Common configuration
> >     capability, which means we can't be a modern VirtIO SCSI device.
> > 
> >     This is unfortunate, but there doesn't seem to be any way around
> >     this, at least with the current hardware.
> > 
> >  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
> >     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
> >     vring_use_map_api() function will return false.
> > 
> >     Currently Linux endpoint devices use the legacy virtio interface as
> >     they aren't able to advertise the Common configuration capability.
> >     As most PCI endpoint capable PCIe controllers do not allow modifying the
> >     capability list, and thus are unable to advertise the Common configuration
> >     capability. This means the device's inbound TLPs fault on the host
> >     SMMU because the vring descriptors carry raw physical addresses.
> > 
> >     This series adds a quirk that forces a subset of legacy virtio devices
> >     to use the DMA Map API (vring_use_map_api() will return true),
> >     which fixes this issue.
> > 
> >     It's unideal that we have to hard code a quirk to basically just
> >     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
> >     are stuck with legacy virtio devices there isn't much else we can do.
> > 
> >  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
> >     respond fast enough to the host. This means we effectivly burn a CPU
> >     to read and write some values. But as there are no intterupts
> >     generated on these events and we need to be very quick there isn't
> >     another option.
> > 

Most of these pain points will go away if you use virtio-msg [1] transport
instead of the virtio-pci transport. Using the virtio-pci transport on a real
PCIe device without a way to trap and emulate the config space requests will
always be racy.

- Mani

[1] https://lore.kernel.org/virtio-comment/cover.1781514628.git.bertrand.marquis@arm.com

-- 
மணிவண்ணன் சதாசிவம்

On 9/3/26 20:51, Manivannan Sadhasivam wrote:
> On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
>> On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
>>> From: Alistair Francis <alistair.francis@wdc.com>
>>>
>>> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
>>> endpoint. This is a similar approach to the NVMe PCIe Endpoint
>>> (drivers/nvme/target/pci-epf.c) but for SCSI.
>>>
>>> This does end up being somewhat similar to the pci-epf.c code, but
>>> re-written for SCSI.
>>>
>>> This approach allows a PCIe Endpoint device (tested on a
>>> radxa-rock5b) to setup what appears to be a SCSI device, using an
>>> existing SCSI backend (tested using scsi_debug).
>>>
>>> At this point a host can connect over PCIe, ensure virtio_pci and
>>> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
>>>
>>> There are a few pain points with this approach though:
>>>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
>>>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
>>>     capabilities. So we can't advertise the VirtIO Common configuration
>>>     capability, which means we can't be a modern VirtIO SCSI device.
>>>
>>>     This is unfortunate, but there doesn't seem to be any way around
>>>     this, at least with the current hardware.
>>>
>>>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
>>>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
>>>     vring_use_map_api() function will return false.
>>>
>>>     Currently Linux endpoint devices use the legacy virtio interface as
>>>     they aren't able to advertise the Common configuration capability.
>>>     As most PCI endpoint capable PCIe controllers do not allow modifying the
>>>     capability list, and thus are unable to advertise the Common configuration
>>>     capability. This means the device's inbound TLPs fault on the host
>>>     SMMU because the vring descriptors carry raw physical addresses.
>>>
>>>     This series adds a quirk that forces a subset of legacy virtio devices
>>>     to use the DMA Map API (vring_use_map_api() will return true),
>>>     which fixes this issue.
>>>
>>>     It's unideal that we have to hard code a quirk to basically just
>>>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
>>>     are stuck with legacy virtio devices there isn't much else we can do.
>>>
>>>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
>>>     respond fast enough to the host. This means we effectivly burn a CPU
>>>     to read and write some values. But as there are no intterupts
>>>     generated on these events and we need to be very quick there isn't
>>>     another option.
>>>
> 
> Most of these pain points will go away if you use virtio-msg [1] transport
> instead of the virtio-pci transport. Using the virtio-pci transport on a real
> PCIe device without a way to trap and emulate the config space requests will
> always be racy.

There is nothing inherently racy about the config space. It is about the fact
that most PCI endpoint controllers:
1) Do not raise an interrupt when PCI BARs or config space is written by the
host RC, and
2) All PCI endpoint controllers that Linux supports do not allow drivers to
create extended capabilities in the config space that can then be emulated in
the endpoint driver (enabling that would require 1 to be supported, obviously).

(2) can be delt with quirks. Not great, but simple enough. And in this case, we
need it more because of the virtio-pci specs, which are not great to start with.

And for (1), the only real problem that causes is that an endpoint driver needs
to poll PCI BARs/submission queues to see if the host issued commands. Again not
great, but that works just fine. Alistair's point about burning a CPU doing that
is simply so that we can reduce command latency and get good enough performance.

We went through all of that already with the NVMe PCI endpoint. Works well
enough and does what is intended, which is the same here for the virtio-scsi
endpoint driver: create a platform where one can emulate a SCSI device to
experiment with new features etc. This is all intended as a development/test
tool, not for production use.

I do not know virtio-msg. First time I hear about it. And I am not sure if there
is a standard way of exposing a SCSI host through that.


-- 
Damien Le Moal
Western Digital Research

On Fri, Sep 04, 2026 at 09:12:11AM +0900, Damien Le Moal wrote:
> On 9/3/26 20:51, Manivannan Sadhasivam wrote:
> > On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
> >> On Tue, Sep 01, 2026 at 11:46:48AM +1000, alistair23@gmail.com wrote:
> >>> From: Alistair Francis <alistair.francis@wdc.com>
> >>>
> >>> This series adds a VirtIO SCSI endpoint built on top of the VirtIO PCIe
> >>> endpoint. This is a similar approach to the NVMe PCIe Endpoint
> >>> (drivers/nvme/target/pci-epf.c) but for SCSI.
> >>>
> >>> This does end up being somewhat similar to the pci-epf.c code, but
> >>> re-written for SCSI.
> >>>
> >>> This approach allows a PCIe Endpoint device (tested on a
> >>> radxa-rock5b) to setup what appears to be a SCSI device, using an
> >>> existing SCSI backend (tested using scsi_debug).
> >>>
> >>> At this point a host can connect over PCIe, ensure virtio_pci and
> >>> virtio_scsi is loaded and on PCIe rescan will see a scsi device.
> >>>
> >>> There are a few pain points with this approach though:
> >>>  1. We have to use the Legacy SCSI VirtIO driver. This is because the
> >>>     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't add
> >>>     capabilities. So we can't advertise the VirtIO Common configuration
> >>>     capability, which means we can't be a modern VirtIO SCSI device.
> >>>
> >>>     This is unfortunate, but there doesn't seem to be any way around
> >>>     this, at least with the current hardware.
> >>>
> >>>  1.2. Legacy virtio devices only have 32 feature bits and therefore can't
> >>>     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This means the
> >>>     vring_use_map_api() function will return false.
> >>>
> >>>     Currently Linux endpoint devices use the legacy virtio interface as
> >>>     they aren't able to advertise the Common configuration capability.
> >>>     As most PCI endpoint capable PCIe controllers do not allow modifying the
> >>>     capability list, and thus are unable to advertise the Common configuration
> >>>     capability. This means the device's inbound TLPs fault on the host
> >>>     SMMU because the vring descriptors carry raw physical addresses.
> >>>
> >>>     This series adds a quirk that forces a subset of legacy virtio devices
> >>>     to use the DMA Map API (vring_use_map_api() will return true),
> >>>     which fixes this issue.
> >>>
> >>>     It's unideal that we have to hard code a quirk to basically just
> >>>     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1) as we
> >>>     are stuck with legacy virtio devices there isn't much else we can do.
> >>>
> >>>  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in order to
> >>>     respond fast enough to the host. This means we effectivly burn a CPU
> >>>     to read and write some values. But as there are no intterupts
> >>>     generated on these events and we need to be very quick there isn't
> >>>     another option.
> >>>
> > 
> > Most of these pain points will go away if you use virtio-msg [1] transport
> > instead of the virtio-pci transport. Using the virtio-pci transport on a real
> > PCIe device without a way to trap and emulate the config space requests will
> > always be racy.
> 
> There is nothing inherently racy about the config space. It is about the fact
> that most PCI endpoint controllers:
> 1) Do not raise an interrupt when PCI BARs or config space is written by the
> host RC, and
> 2) All PCI endpoint controllers that Linux supports do not allow drivers to
> create extended capabilities in the config space that can then be emulated in
> the endpoint driver (enabling that would require 1 to be supported, obviously).
> 
> (2) can be delt with quirks. Not great, but simple enough. And in this case, we
> need it more because of the virtio-pci specs, which are not great to start with.
> 
> And for (1), the only real problem that causes is that an endpoint driver needs
> to poll PCI BARs/submission queues to see if the host issued commands. Again not
> great, but that works just fine. Alistair's point about burning a CPU doing that
> is simply so that we can reduce command latency and get good enough performance.
> 
> We went through all of that already with the NVMe PCI endpoint. Works well
> enough and does what is intended, which is the same here for the virtio-scsi
> endpoint driver: create a platform where one can emulate a SCSI device to
> experiment with new features etc. This is all intended as a development/test
> tool, not for production use.
> 

The race is inherently present in the way the virtqueue setup is done. Spec
defines many registers with side-effects. Like device_feature_select,
device_status, queue_select, queue_reset etc... Fabricating responses for these
registers properly would require trap-and-emulate as the endpoint cannot
reliably generate the response, once written. It may work sometimes, but not
always.

> I do not know virtio-msg. First time I hear about it. And I am not sure if there
> is a standard way of exposing a SCSI host through that.
> 

virtio-msg is a message based transport, just like virtio-pci. It has no
knowledge of the top level Virtio protocol. Using it will eliminate all the
shortcomings of the virtio-pci transport as the config steps between the
driver and device is message-response based.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

On Fri, 2026-09-04 at 09:12 +0900, Damien Le Moal wrote:
> On 9/3/26 20:51, Manivannan Sadhasivam wrote:
> > On Tue, Sep 01, 2026 at 10:35:49AM +0200, Niklas Cassel wrote:
> > > On Tue, Sep 01, 2026 at 11:46:48AM +1000,
> > > alistair23@gmail.com wrote:
> > > > From: Alistair Francis <alistair.francis@wdc.com>
> > > > 
> > > > This series adds a VirtIO SCSI endpoint built on top of the
> > > > VirtIO PCIe
> > > > endpoint. This is a similar approach to the NVMe PCIe Endpoint
> > > > (drivers/nvme/target/pci-epf.c) but for SCSI.
> > > > 
> > > > This does end up being somewhat similar to the pci-epf.c code,
> > > > but
> > > > re-written for SCSI.
> > > > 
> > > > This approach allows a PCIe Endpoint device (tested on a
> > > > radxa-rock5b) to setup what appears to be a SCSI device, using
> > > > an
> > > > existing SCSI backend (tested using scsi_debug).
> > > > 
> > > > At this point a host can connect over PCIe, ensure virtio_pci
> > > > and
> > > > virtio_scsi is loaded and on PCIe rescan will see a scsi
> > > > device.
> > > > 
> > > > There are a few pain points with this approach though:
> > > >  1. We have to use the Legacy SCSI VirtIO driver. This is
> > > > because the
> > > >     Raxda Rock5b (and AFAIK all PCIe Endpoint hardware) can't
> > > > add
> > > >     capabilities. So we can't advertise the VirtIO Common
> > > > configuration
> > > >     capability, which means we can't be a modern VirtIO SCSI
> > > > device.
> > > > 
> > > >     This is unfortunate, but there doesn't seem to be any way
> > > > around
> > > >     this, at least with the current hardware.
> > > > 
> > > >  1.2. Legacy virtio devices only have 32 feature bits and
> > > > therefore can't
> > > >     set the VIRTIO_F_ACCESS_PLATFORM (bit 33) feature. This
> > > > means the
> > > >     vring_use_map_api() function will return false.
> > > > 
> > > >     Currently Linux endpoint devices use the legacy virtio
> > > > interface as
> > > >     they aren't able to advertise the Common configuration
> > > > capability.
> > > >     As most PCI endpoint capable PCIe controllers do not allow
> > > > modifying the
> > > >     capability list, and thus are unable to advertise the
> > > > Common configuration
> > > >     capability. This means the device's inbound TLPs fault on
> > > > the host
> > > >     SMMU because the vring descriptors carry raw physical
> > > > addresses.
> > > > 
> > > >     This series adds a quirk that forces a subset of legacy
> > > > virtio devices
> > > >     to use the DMA Map API (vring_use_map_api() will return
> > > > true),
> > > >     which fixes this issue.
> > > > 
> > > >     It's unideal that we have to hard code a quirk to basically
> > > > just
> > > >     advertise the VIRTIO_F_ACCESS_PLATFORM feature, but (see 1)
> > > > as we
> > > >     are stuck with legacy virtio devices there isn't much else
> > > > we can do.
> > > > 
> > > >  2. We have to pin scsit_pci_epf_poll_cfg_thread() on a CPU in
> > > > order to
> > > >     respond fast enough to the host. This means we effectivly
> > > > burn a CPU
> > > >     to read and write some values. But as there are no
> > > > intterupts
> > > >     generated on these events and we need to be very quick
> > > > there isn't
> > > >     another option.
> > > > 
> > 
> > Most of these pain points will go away if you use virtio-msg [1]
> > transport
> > instead of the virtio-pci transport. Using the virtio-pci transport
> > on a real
> > PCIe device without a way to trap and emulate the config space
> > requests will
> > always be racy.
> 
> There is nothing inherently racy about the config space. It is about
> the fact
> that most PCI endpoint controllers:
> 1) Do not raise an interrupt when PCI BARs or config space is written
> by the
> host RC, and
> 2) All PCI endpoint controllers that Linux supports do not allow
> drivers to
> create extended capabilities in the config space that can then be
> emulated in
> the endpoint driver (enabling that would require 1 to be supported,
> obviously).
> 
> (2) can be delt with quirks. Not great, but simple enough. And in
> this case, we
> need it more because of the virtio-pci specs, which are not great to
> start with.
> 
> And for (1), the only real problem that causes is that an endpoint
> driver needs
> to poll PCI BARs/submission queues to see if the host issued
> commands. Again not
> great, but that works just fine. Alistair's point about burning a CPU
> doing that
> is simply so that we can reduce command latency and get good enough
> performance.

It is actually racy. If we don't burn a CPU to check we end up racing,
with the host as we are too slow to update the config space.

> 
> We went through all of that already with the NVMe PCI endpoint. Works
> well
> enough and does what is intended, which is the same here for the
> virtio-scsi
> endpoint driver: create a platform where one can emulate a SCSI
> device to
> experiment with new features etc. This is all intended as a
> development/test
> tool, not for production use.
> 
> I do not know virtio-msg. First time I hear about it. And I am not
> sure if there
> is a standard way of exposing a SCSI host through that.

virtio-msg-amp does seem promising. I'll dig into it a bit more and
keep an eye on it.

As virtio-msg-amp is very new though, I'm not sure it solves the
problem right now.

Alistair

On 9/4/26 12:19, Alistair Francis wrote:
>>> Most of these pain points will go away if you use virtio-msg [1]
>>> transport
>>> instead of the virtio-pci transport. Using the virtio-pci transport
>>> on a real
>>> PCIe device without a way to trap and emulate the config space
>>> requests will
>>> always be racy.
>>
>> There is nothing inherently racy about the config space. It is about
>> the fact
>> that most PCI endpoint controllers:
>> 1) Do not raise an interrupt when PCI BARs or config space is written
>> by the
>> host RC, and
>> 2) All PCI endpoint controllers that Linux supports do not allow
>> drivers to
>> create extended capabilities in the config space that can then be
>> emulated in
>> the endpoint driver (enabling that would require 1 to be supported,
>> obviously).
>>
>> (2) can be delt with quirks. Not great, but simple enough. And in
>> this case, we
>> need it more because of the virtio-pci specs, which are not great to
>> start with.
>>
>> And for (1), the only real problem that causes is that an endpoint
>> driver needs
>> to poll PCI BARs/submission queues to see if the host issued
>> commands. Again not
>> great, but that works just fine. Alistair's point about burning a CPU
>> doing that
>> is simply so that we can reduce command latency and get good enough
>> performance.
> 
> It is actually racy. If we don't burn a CPU to check we end up racing,
> with the host as we are too slow to update the config space.

How come ? At least for nvme, it does not matter how slow the endpoint is to
pull commands from the SQ: if the SQ becomes full, the host just stops
submitting. There is no race. I would expect virtio-pci/scsi to have a similar
race-safe protocol mechanism. If there is indeed a race, then we are talking
about a deficiency of the protocols rather than the transport.

-- 
Damien Le Moal
Western Digital Research

On Fri, 2026-09-04 at 13:41 +0900, Damien Le Moal wrote:
> On 9/4/26 12:19, Alistair Francis wrote:
> > > > Most of these pain points will go away if you use virtio-msg
> > > > [1]
> > > > transport
> > > > instead of the virtio-pci transport. Using the virtio-pci
> > > > transport
> > > > on a real
> > > > PCIe device without a way to trap and emulate the config space
> > > > requests will
> > > > always be racy.
> > > 
> > > There is nothing inherently racy about the config space. It is
> > > about
> > > the fact
> > > that most PCI endpoint controllers:
> > > 1) Do not raise an interrupt when PCI BARs or config space is
> > > written
> > > by the
> > > host RC, and
> > > 2) All PCI endpoint controllers that Linux supports do not allow
> > > drivers to
> > > create extended capabilities in the config space that can then be
> > > emulated in
> > > the endpoint driver (enabling that would require 1 to be
> > > supported,
> > > obviously).
> > > 
> > > (2) can be delt with quirks. Not great, but simple enough. And in
> > > this case, we
> > > need it more because of the virtio-pci specs, which are not great
> > > to
> > > start with.
> > > 
> > > And for (1), the only real problem that causes is that an
> > > endpoint
> > > driver needs
> > > to poll PCI BARs/submission queues to see if the host issued
> > > commands. Again not
> > > great, but that works just fine. Alistair's point about burning a
> > > CPU
> > > doing that
> > > is simply so that we can reduce command latency and get good
> > > enough
> > > performance.
> > 
> > It is actually racy. If we don't burn a CPU to check we end up
> > racing,
> > with the host as we are too slow to update the config space.
> 
> How come ? At least for nvme, it does not matter how slow the
> endpoint is to
> pull commands from the SQ: if the SQ becomes full, the host just
> stops
> submitting. There is no race. I would expect virtio-pci/scsi to have
> a similar
> race-safe protocol mechanism. If there is indeed a race, then we are
> talking
> about a deficiency of the protocols rather than the transport.

The issue is in the config space, setting up VirtIO-PCI. From memory
the QueuePFN (which indicates the currently used page number of the
queue) causes all sorts of issues if you don't zero it fast enough.

Alistair