This RFC proposes automatically enabling PCIe Atomic Ops completer
support on PCIe root ports within the VM given the restrictions that
a) vfio-pci provides a capability on the vfio device indicating enabled
support for various operations, and b) requiring a configuration of
installing the vfio-pci as a cold-plug device immediately downstream of
the root port, thus avoiding a secondary issue for enabling Atomic Ops
routing.  This idea was outlined in reply[1] to another proposal
suggesting manual device options to pcie-root-port to do the same.

This support depends on this[2] kernel patch enabling the host Atomic Op
reporting.

There are however some concerns with an automatic approach.  First is
the issue of hotplug.  If we've marked Atomic Ops completion as
supported by the root port and subsequently unplug the endpoint that was
the basis for that enablement, do we let that capability remain?  Should
the device exit function clear that capability?  Is it even legal to
clear capabilities on the root port at runtime?  I'm not sure it's safe
to assume that other devices on the same host would have similar
capabilities, but regardless, it's not safe to assume the VM is even
running on the same host for a subsequent hot-add.

We could potentially require the vfio-pci-nohotplug variant, but this
then becomes a usability barrier that not only do we require management
tools to pick the correct device flavor in QEMU, but we lose support for
things like unplugging a device for migration and re-plugging a
replacement on the target system.

Potentially the best solution might be to clear these capability bits in
the QEMU device exit function.  I don't know if the spec specifically
disallows this, but I suspect we could get away with it from a guest
perspective.

In a similar vein, the second concern is that PCIe slots are composable
in QEMU, ie. a user could create a multi-function device where the
Atomic Ops support between devices is in conflict.  The simple solution
would be to virtualize DEVCAP2 on the device to only report Atomic Ops
capabilities when supported by the host routing (which would also
simplify the hot-plug situation), but it does not appear to be common
practice for Linux in-kernel drivers to look at this register on the
endpoint.

Maybe the solution is that we only enable Atomic Ops on the root port
for a device iff the device is at slot address zero and the
multifunction bit is clear.  Additionally the device exit function
should remove the capabilities it has enabled.

I'll work on a v2 with these changes but discussion is welcome whether
such a solution would be acceptable.  Thanks,

Alex

[1]https://lore.kernel.org/all/20230518161309.369a5d6c.alex.williamson@redhat.com/
[2]https://lore.kernel.org/all/20230519214748.402003-1-alex.williamson@redhat.com/

Alex Williamson (4):
  linux-headers: Update for vfio capability reporting AtomicOps
  vfio: Implement a common device info helper
  pcie: Add a PCIe capability version helper
  vfio/pci: Enable AtomicOps completers on root ports

 hw/pci/pcie.c                 |  7 ++++
 hw/s390x/s390-pci-vfio.c      | 37 +++-----------------
 hw/vfio/common.c              | 46 ++++++++++++++++++------
 hw/vfio/pci.c                 | 66 +++++++++++++++++++++++++++++++++++
 include/hw/pci/pcie.h         |  1 +
 include/hw/vfio/vfio-common.h |  1 +
 linux-headers/linux/vfio.h    | 14 ++++++++
 7 files changed, 129 insertions(+), 43 deletions(-)

-- 
2.39.2