Hi all,

This series adds a vfio_pci variant driver for the s390-specific
Internal Shared Memory (ISM) devices used for inter-VM communication
including SMC-D.

This is a prerequisite for an in-development open-source user space
driver stack that will allow to use ISM devices to provide remote
console and block device functionality. This stack will be part of
s390-tools.

This driver would also allow QEMU to mediate access to an ISM device,
enabling a form of PCI pass-through even for guests whose hardware
cannot directly execute PCI accesses, such as nested guests.

On s390, kernel primitives such as ioread() and iowrite() are switched
over from function handle based PCI load/stores instructions to PCI
memory-I/O (MIO) loads/stores when these are available and not
explicitly disabled. Since these instructions cannot be used with ISM
devices, ensure that classic function handle-based PCI instructions are
used instead.

The driver is still required even when MIO instructions are disabled, as
the ISM device relies on the PCI store‑block (PCISTB) instruction to
perform write operations.

Thank you,
Julian

Signed-off-by: Julian Ruess <julianr@linux.ibm.com>
---
Changes in v4:
- Fix bug with < 8 byte reads. For code simplicity, only support 8 byte reads.
- Fix leak of ivpcd.
- Fix cache replacement by implementing a per-device kmem_cache.
- Link to v3: https://lore.kernel.org/r/20260305-vfio_pci_ism-v3-0-1217076c81d9@linux.ibm.com

Changes in v3:
- Add comments to ism_vfio_pci_do_io_r() and ism_vfio_pci_do_io_w().
- Format Kconfig.
- Add 4k boundary check to ism_vfio_pci_do_io_w().
- Use kmem_cache instead of kzalloc in ism_vfio_pci_do_io_w().
- Add error handler to struct ism_vfio_pci_driver.
- Link to v2: https://lore.kernel.org/r/20260224-vfio_pci_ism-v2-0-f010945373fa@linux.ibm.com

Changes in v2:
- Remove common code patch that sets VFIO_PCI_OFFSET_SHIFT to 48.
- Implement ism_vfio_pci_ioctl_get_region_info() to have own region
  offsets.
- For config space accesses, rename vfio_config_do_rw() to
  vfio_pci_config_rw_single() and export it.
- Use zdev->maxstbl instead of ZPCI_BOUNDARY_SIZE.
- Add comment that zPCI must not use MIO instructions for config space
  access.
- Rework patch descriptions.
- Update license info.
- Link to v1: https://lore.kernel.org/r/20260212-vfio_pci_ism-v1-0-333262ade074@linux.ibm.com

---
Julian Ruess (3):
      vfio/pci: Rename vfio_config_do_rw() to vfio_pci_config_rw_single() and export it
      vfio/ism: Implement vfio_pci driver for ISM devices
      MAINTAINERS: add VFIO ISM PCI DRIVER section

 MAINTAINERS                        |   6 +
 drivers/vfio/pci/Kconfig           |   2 +
 drivers/vfio/pci/Makefile          |   2 +
 drivers/vfio/pci/ism/Kconfig       |  10 ++
 drivers/vfio/pci/ism/Makefile      |   3 +
 drivers/vfio/pci/ism/main.c        | 345 +++++++++++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c |   8 +-
 drivers/vfio/pci/vfio_pci_priv.h   |   4 +
 8 files changed, 377 insertions(+), 3 deletions(-)
---
base-commit: 0257f64bdac7fdca30fa3cae0df8b9ecbec7733a
change-id: 20250227-vfio_pci_ism-0ccc2e472247

Best regards,
-- 
Julian Ruess <julianr@linux.ibm.com>

On Fri, 13 Mar 2026 15:40:27 +0100
Julian Ruess <julianr@linux.ibm.com> wrote:

> Hi all,
> 
> This series adds a vfio_pci variant driver for the s390-specific
> Internal Shared Memory (ISM) devices used for inter-VM communication
> including SMC-D.
> 
> This is a prerequisite for an in-development open-source user space
> driver stack that will allow to use ISM devices to provide remote
> console and block device functionality. This stack will be part of
> s390-tools.
> 
> This driver would also allow QEMU to mediate access to an ISM device,
> enabling a form of PCI pass-through even for guests whose hardware
> cannot directly execute PCI accesses, such as nested guests.
> 
> On s390, kernel primitives such as ioread() and iowrite() are switched
> over from function handle based PCI load/stores instructions to PCI
> memory-I/O (MIO) loads/stores when these are available and not
> explicitly disabled. Since these instructions cannot be used with ISM
> devices, ensure that classic function handle-based PCI instructions are
> used instead.
> 
> The driver is still required even when MIO instructions are disabled, as
> the ISM device relies on the PCI store‑block (PCISTB) instruction to
> perform write operations.
> 
> Thank you,
> Julian
> 
> Signed-off-by: Julian Ruess <julianr@linux.ibm.com>
> ---
> Changes in v4:
> - Fix bug with < 8 byte reads. For code simplicity, only support 8 byte reads.

Does the ISM device define sub-8-byte accesses as valid?  It looks like
if pread() doesn't return the desired size QEMU will fill the return
with -1.  Unless such accesses are classified as undefined by ISM,
doesn't that suggest a potential data corruption issue to the guest
driver?  Thanks,

Alex

On Fri Mar 13, 2026 at 4:41 PM CET, Alex Williamson wrote:
> On Fri, 13 Mar 2026 15:40:27 +0100
> Julian Ruess <julianr@linux.ibm.com> wrote:
>
>> Hi all,
>> 
>> This series adds a vfio_pci variant driver for the s390-specific
>> Internal Shared Memory (ISM) devices used for inter-VM communication
>> including SMC-D.
>> 
>> This is a prerequisite for an in-development open-source user space
>> driver stack that will allow to use ISM devices to provide remote
>> console and block device functionality. This stack will be part of
>> s390-tools.
>> 
>> This driver would also allow QEMU to mediate access to an ISM device,
>> enabling a form of PCI pass-through even for guests whose hardware
>> cannot directly execute PCI accesses, such as nested guests.
>> 
>> On s390, kernel primitives such as ioread() and iowrite() are switched
>> over from function handle based PCI load/stores instructions to PCI
>> memory-I/O (MIO) loads/stores when these are available and not
>> explicitly disabled. Since these instructions cannot be used with ISM
>> devices, ensure that classic function handle-based PCI instructions are
>> used instead.
>> 
>> The driver is still required even when MIO instructions are disabled, as
>> the ISM device relies on the PCI store‑block (PCISTB) instruction to
>> perform write operations.
>> 
>> Thank you,
>> Julian
>> 
>> Signed-off-by: Julian Ruess <julianr@linux.ibm.com>
>> ---
>> Changes in v4:
>> - Fix bug with < 8 byte reads. For code simplicity, only support 8 byte reads.
>
> Does the ISM device define sub-8-byte accesses as valid?  It looks like
> if pread() doesn't return the desired size QEMU will fill the return
> with -1.  Unless such accesses are classified as undefined by ISM,
> doesn't that suggest a potential data corruption issue to the guest
> driver?  Thanks,
>
> Alex

Hi Alex,

thanks for the quick feedback!

We are currently developing this extension for a non‑QEMU vfio user space
driver. Reads smaller than 8 bytes are theoretically valid, but they are not
used by this driver nor the existing in-kernel driver at the moment. We could
extend this in the future if needed.

vfio‑pci based PCI pass-through of the ISM device is already possible without
this extension. In that case, the ISM driver in the guest kernel accesses the
BARs directly through hardware virtualization, without using the new access
routines provided by this variant driver.

Julian

On Mon, 16 Mar 2026 13:33:04 +0100
"Julian Ruess" <julianr@linux.ibm.com> wrote:

> On Fri Mar 13, 2026 at 4:41 PM CET, Alex Williamson wrote:
> > On Fri, 13 Mar 2026 15:40:27 +0100
> > Julian Ruess <julianr@linux.ibm.com> wrote:
> >  
> >> Hi all,
> >> 
> >> This series adds a vfio_pci variant driver for the s390-specific
> >> Internal Shared Memory (ISM) devices used for inter-VM communication
> >> including SMC-D.
> >> 
> >> This is a prerequisite for an in-development open-source user space
> >> driver stack that will allow to use ISM devices to provide remote
> >> console and block device functionality. This stack will be part of
> >> s390-tools.
> >> 
> >> This driver would also allow QEMU to mediate access to an ISM device,
> >> enabling a form of PCI pass-through even for guests whose hardware
> >> cannot directly execute PCI accesses, such as nested guests.
> >> 
> >> On s390, kernel primitives such as ioread() and iowrite() are switched
> >> over from function handle based PCI load/stores instructions to PCI
> >> memory-I/O (MIO) loads/stores when these are available and not
> >> explicitly disabled. Since these instructions cannot be used with ISM
> >> devices, ensure that classic function handle-based PCI instructions are
> >> used instead.
> >> 
> >> The driver is still required even when MIO instructions are disabled, as
> >> the ISM device relies on the PCI store‑block (PCISTB) instruction to
> >> perform write operations.
> >> 
> >> Thank you,
> >> Julian
> >> 
> >> Signed-off-by: Julian Ruess <julianr@linux.ibm.com>
> >> ---
> >> Changes in v4:
> >> - Fix bug with < 8 byte reads. For code simplicity, only support 8 byte reads.  
> >
> > Does the ISM device define sub-8-byte accesses as valid?  It looks like
> > if pread() doesn't return the desired size QEMU will fill the return
> > with -1.  Unless such accesses are classified as undefined by ISM,
> > doesn't that suggest a potential data corruption issue to the guest
> > driver?  Thanks,
> >
> > Alex  
> 
> Hi Alex,
> 
> thanks for the quick feedback!
> 
> We are currently developing this extension for a non‑QEMU vfio user space
> driver. Reads smaller than 8 bytes are theoretically valid, but they are not
> used by this driver nor the existing in-kernel driver at the moment. We could
> extend this in the future if needed.
> 
> vfio‑pci based PCI pass-through of the ISM device is already possible without
> this extension. In that case, the ISM driver in the guest kernel accesses the
> BARs directly through hardware virtualization, without using the new access
> routines provided by this variant driver.

Hi Julian,

The cover letter argues a secondary use case with QEMU, especially in a
nested environment.  The ISM range appears to be an interface to a
variety of device types, console and block are noted.  It's also noted
in the implementation that the z/Architecture allows sub-8-byte access.

I think we need to be cautious that the existence of this driver makes
it available for use with QEMU and other VMMs.  In the case of QEMU
vfio_region_ops will allow single-byte access by default.

The restricted access width is positioned as a simplification here, but
it needs to be evaluated against all the use cases.  Unless we're 100%
sure none of those use cases rely on sub-8-byte accesses, we might be
setting ourselves up for hacks later to fix or detect partial access
support.

I'll leave it to IBM folks to determine if this is indeed a
simplification for long term support of all use cases and not a short
term fix for the short term use case.  Thanks,

Alex

On Mon, 2026-03-16 at 13:03 -0600, Alex Williamson wrote:
> On Mon, 16 Mar 2026 13:33:04 +0100
> "Julian Ruess" <julianr@linux.ibm.com> wrote:
> 
> > On Fri Mar 13, 2026 at 4:41 PM CET, Alex Williamson wrote:
> > > On Fri, 13 Mar 2026 15:40:27 +0100
> > > Julian Ruess <julianr@linux.ibm.com> wrote:
> > >  
--- snip ---
> > 
> > Hi Alex,
> > 
> > thanks for the quick feedback!
> > 
> > We are currently developing this extension for a non‑QEMU vfio user space
> > driver. Reads smaller than 8 bytes are theoretically valid, but they are not
> > used by this driver nor the existing in-kernel driver at the moment. We could
> > extend this in the future if needed.
> > 
> > vfio‑pci based PCI pass-through of the ISM device is already possible without
> > this extension. In that case, the ISM driver in the guest kernel accesses the
> > BARs directly through hardware virtualization, without using the new access
> > routines provided by this variant driver.
> 
> Hi Julian,
> 
> The cover letter argues a secondary use case with QEMU, especially in a
> nested environment.  The ISM range appears to be an interface to a
> variety of device types, console and block are noted.  It's also noted
> in the implementation that the z/Architecture allows sub-8-byte access.
> 
> I think we need to be cautious that the existence of this driver makes
> it available for use with QEMU and other VMMs.  In the case of QEMU
> vfio_region_ops will allow single-byte access by default.
> 
> The restricted access width is positioned as a simplification here, but
> it needs to be evaluated against all the use cases.  Unless we're 100%
> sure none of those use cases rely on sub-8-byte accesses, we might be
> setting ourselves up for hacks later to fix or detect partial access
> support.
> 
> I'll leave it to IBM folks to determine if this is indeed a
> simplification for long term support of all use cases and not a short
> term fix for the short term use case.  Thanks,
> 
> Alex

Hi Alex,

Thank you for the insights and expertise it is highly appreciated. Your
reasoning makes sense to me and I agree this simplification looks like
it may be ok for now but could cause more trouble than it is worth
later and there's really no reason not to just support < 8 byte
accesses too.

One thing I'd like to explain though since you mention ISM potentially
being an interface to different device types. This part of the cover
letter is easy to misunderstand since we haven't yet send out patches
opening up that direction. There is only one ISM device and even future
versions would likely just iterate but serve the same purpose. The
multiple device types including consoles and block devices will not
replace this ISM device but rather sit on top of a virtual bus called
the ISM Peer Bus where the ISM device will serve as a communication
channel between two Linux instances connected via an ISM device.
Specifically in the current design there will be a vfio-pci based user-
space daemon/driver on one Linux instance that provides virtual
consoles and block devices for other Linux instances on the other side
of ISM based communication channels. Those Linux instances will use
kernel integrated drivers for the console and block devices. Currently
there are no plans for passing these devices through to guests since we
can also just pass-through an ISM device and ISM devices being virtual
we're not limited in how many we can have.

Thanks,
Niklas