Hi,

This patch series introduces support for error recovery for passthrough PCI
devices on System Z (s390x). This is the user space component for the Linux
kernel patches [1]. The kernel patches were merged for 7.3 and it had some
significant changes in VFIO API since the last QEMU series was posted. But the
design for QEMU component is still the same.

For QEMU on eventfd notification for PCI error from vfio-pci driver we call the
vfio error handler. We can use a per device error handler callback to override
the default vfio error handler.  For s390x specific error handler, we retrieve
the architecture specific PCI error information and inject the information into
the guest. Once the guest receives the error information, the guest drivers will
drive the error recovery.  Typically recovery involves a device reset which
translate to CLP disable/enable cycle for the device.

I would appreciate some feedback on this patch series.

Thanks 
Farhan

[1] https://lore.kernel.org/all/20260818164326.387bb27b@shazbot.org/

ChangeLog
---------
v3 https://lore.kernel.org/qemu-devel/20250925174852.1302-1-alifm@linux.ibm.com/
v3 -> v4
    - Include linux headers for 7.3-rc1.
    - Rework VFIO API changes based on the kernel API (patch 3).
    - Address Markus's comments from v3 (patch 2).

v2 https://lore.kernel.org/qemu-devel/20250825212434.2255-1-alifm@linux.ibm.com/
v2 -> v3
    - Update arch_err_handler to err_handler and include Error ** in
    function definition. (patch 2)

    - Introduce helper function to hide the internal indirection of device_feature()
    (patch 3)

    - Update function definitions to include Error ** (patch 4)
    


v1 https://lore.kernel.org/qemu-devel/20250813174152.1238-1-alifm@linux.ibm.com/
v1 -> v2
   - Use VFIO_DEVICE_FEATURE ioctl to get device error information.
   (Based on Alex's feedback on kernel series)

Farhan Ali (4):
  linux-headers: Update Linux header to 7.3-rc1
  vfio/pci: Add an error handler callback
  s390x/pci: Add PCI error handling for vfio pci devices
  s390x/pci: Reset a device in error state

 hw/s390x/s390-pci-bus.c                     |  16 ++
 hw/s390x/s390-pci-vfio-stubs.c              |  10 +
 hw/s390x/s390-pci-vfio.c                    | 110 +++++++++++
 hw/vfio/pci.c                               |  27 ++-
 hw/vfio/pci.h                               |   1 +
 include/hw/s390x/s390-pci-bus.h             |   2 +
 include/hw/s390x/s390-pci-vfio.h            |   2 +
 include/standard-headers/drm/drm_fourcc.h   | 209 ++++++++++++++++++--
 include/standard-headers/linux/ethtool.h    |   6 +
 include/standard-headers/linux/fuse.h       |  63 +++++-
 include/standard-headers/linux/virtio_ids.h |   1 +
 linux-headers/asm-arm64/kvm.h               |   1 +
 linux-headers/asm-riscv/kvm.h               |  13 ++
 linux-headers/linux/iommufd.h               |  45 ++++-
 linux-headers/linux/kvm.h                   |   1 +
 linux-headers/linux/userfaultfd.h           |  54 ++++-
 linux-headers/linux/vduse.h                 |  28 +++
 linux-headers/linux/vfio.h                  |  21 ++
 linux-headers/linux/vfio_zdev.h             |   2 +
 19 files changed, 579 insertions(+), 33 deletions(-)

-- 
2.43.0

On 8/31/26 20:31, Farhan Ali wrote:
> Hi,
> 
> This patch series introduces support for error recovery for passthrough PCI
> devices on System Z (s390x). This is the user space component for the Linux
> kernel patches [1]. The kernel patches were merged for 7.3 and it had some
> significant changes in VFIO API since the last QEMU series was posted. But the
> design for QEMU component is still the same.
> 
> For QEMU on eventfd notification for PCI error from vfio-pci driver we call the
> vfio error handler. We can use a per device error handler callback to override
> the default vfio error handler.  For s390x specific error handler, we retrieve
> the architecture specific PCI error information and inject the information into
> the guest. Once the guest receives the error information, the guest drivers will
> drive the error recovery.  Typically recovery involves a device reset which
> translate to CLP disable/enable cycle for the device.
> 
> I would appreciate some feedback on this patch series.

Hi Farhan,

Have you seen Shameer's kernel series for PCI error recovery [1] ?
It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET,
with a self-contained handler independent of the legacy err_notifier.

The QEMU side [2] just probes the feature, installs the eventfd, and
suppresses the legacy vm_stop path with a bool flag : no callback in
generic VFIO PCI code, which is cleaner.

This is also the pattern used by the other s390 VFIO devices: vfio-ccw
has three dedicated eventfds (io_notifier, crw_notifier, req_notifier)
and vfio-ap has two (req_notifier, cfg_notifier).

For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to
support SET with an eventfd field. The ABI is still new so the struct
can be extended, keeping the s390 error recovery code self-contained,
with no changes to generic VFIO  PCI code. I think this approach is
preferable.

Thanks,

C.

[1] https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/
[2] https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be

On 9/3/2026 2:19 AM, Cédric Le Goater wrote:
> On 8/31/26 20:31, Farhan Ali wrote:
>> Hi,
>>
>> This patch series introduces support for error recovery for 
>> passthrough PCI
>> devices on System Z (s390x). This is the user space component for the 
>> Linux
>> kernel patches [1]. The kernel patches were merged for 7.3 and it had 
>> some
>> significant changes in VFIO API since the last QEMU series was 
>> posted. But the
>> design for QEMU component is still the same.
>>
>> For QEMU on eventfd notification for PCI error from vfio-pci driver 
>> we call the
>> vfio error handler. We can use a per device error handler callback to 
>> override
>> the default vfio error handler.  For s390x specific error handler, we 
>> retrieve
>> the architecture specific PCI error information and inject the 
>> information into
>> the guest. Once the guest receives the error information, the guest 
>> drivers will
>> drive the error recovery.  Typically recovery involves a device reset 
>> which
>> translate to CLP disable/enable cycle for the device.
>>
>> I would appreciate some feedback on this patch series.
>
Hi Cedric,

> Hi Farhan,
>
> Have you seen Shameer's kernel series for PCI error recovery [1] ?
> It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET,
> with a self-contained handler independent of the legacy err_notifier.
>
> The QEMU side [2] just probes the feature, installs the eventfd, and
> suppresses the legacy vm_stop path with a bool flag : no callback in
> generic VFIO PCI code, which is cleaner.
>
> This is also the pattern used by the other s390 VFIO devices: vfio-ccw
> has three dedicated eventfds (io_notifier, crw_notifier, req_notifier)
> and vfio-ap has two (req_notifier, cfg_notifier).

My thought process was to use the existing legacy err_notifier and let 
each vfio-pci device handle the error. I thought the callback would also 
provide some flexibility on each device handling the error.

I can take a look at Shameer's approach and see if its something we can 
do. Just briefly looking at the QEMU code, it looks like it it ties the 
eventfd to the generic vfio-pci code (of course this is in flux). This 
would become problematic for s390x as unfortunately AER recovery will 
not work for zPCI devices due to platform firmware being involved in the 
recovery process.

> For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to
> support SET with an eventfd field. The ABI is still new so the struct
> can be extended, keeping the s390 error recovery code self-contained,
> with no changes to generic VFIO  PCI code. I think this approach is
> preferable.
>
I would like to get some feedback from Alex to see if we could extend 
the ABI.

Thanks

Farhan


> Thanks,
>
> C.
>
> [1] 
> https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/
> [2] 
> https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be
>
>

On 9/3/2026 10:55 AM, Farhan Ali wrote:
>
> On 9/3/2026 2:19 AM, Cédric Le Goater wrote:
>> On 8/31/26 20:31, Farhan Ali wrote:
>>> Hi,
>>>
>>> This patch series introduces support for error recovery for 
>>> passthrough PCI
>>> devices on System Z (s390x). This is the user space component for 
>>> the Linux
>>> kernel patches [1]. The kernel patches were merged for 7.3 and it 
>>> had some
>>> significant changes in VFIO API since the last QEMU series was 
>>> posted. But the
>>> design for QEMU component is still the same.
>>>
>>> For QEMU on eventfd notification for PCI error from vfio-pci driver 
>>> we call the
>>> vfio error handler. We can use a per device error handler callback 
>>> to override
>>> the default vfio error handler.  For s390x specific error handler, 
>>> we retrieve
>>> the architecture specific PCI error information and inject the 
>>> information into
>>> the guest. Once the guest receives the error information, the guest 
>>> drivers will
>>> drive the error recovery.  Typically recovery involves a device 
>>> reset which
>>> translate to CLP disable/enable cycle for the device.
>>>
>>> I would appreciate some feedback on this patch series.
>>
> Hi Cedric,
>
>> Hi Farhan,
>>
>> Have you seen Shameer's kernel series for PCI error recovery [1] ?
>> It uses a dedicated eventfd registered via VFIO_DEVICE_FEATURE_SET,
>> with a self-contained handler independent of the legacy err_notifier.
>>
>> The QEMU side [2] just probes the feature, installs the eventfd, and
>> suppresses the legacy vm_stop path with a bool flag : no callback in
>> generic VFIO PCI code, which is cleaner.
>>
>> This is also the pattern used by the other s390 VFIO devices: vfio-ccw
>> has three dedicated eventfds (io_notifier, crw_notifier, req_notifier)
>> and vfio-ap has two (req_notifier, cfg_notifier).
>
> My thought process was to use the existing legacy err_notifier and let 
> each vfio-pci device handle the error. I thought the callback would 
> also provide some flexibility on each device handling the error.
>
> I can take a look at Shameer's approach and see if its something we 
> can do. Just briefly looking at the QEMU code, it looks like it it 
> ties the eventfd to the generic vfio-pci code (of course this is in 
> flux). This would become problematic for s390x as unfortunately AER 
> recovery will not work for zPCI devices due to platform firmware being 
> involved in the recovery process.
>
>> For zPCI, it would mean extending VFIO_DEVICE_FEATURE_ZPCI_ERROR to
>> support SET with an eventfd field. The ABI is still new so the struct
>> can be extended, keeping the s390 error recovery code self-contained,
>> with no changes to generic VFIO  PCI code. I think this approach is
>> preferable.
>>
> I would like to get some feedback from Alex to see if we could extend 
> the ABI.
>
>
I had been thinking about this a bit more, and I think having a another 
eventfd for s390 to just notify on error would just be unnecessary 
duplication. Because now we will have the legacy err_notifier + s390 
error notifer eventfds for the same purpose but with different handlers. 
Can we replace fd handler for the legacy err_notifier from 
vfio_err_notifier_handler() to an s390 specific handler for s390 
devices? This would also avoid any callback in the vfio core common code.

Thanks

Farhan

>
>
>> Thanks,
>>
>> C.
>>
>> [1] 
>> https://lore.kernel.org/all/20260901093217.8539-1-skolothumtho@nvidia.com/
>> [2] 
>> https://github.com/shamiali2008/qemu-master/commit/e45458710cd8509fa5bd81f4376126b77b6f40be
>>
>>
>