Since v5.19, vfio-pci makes use of runtime power management on devices.
This has the effect of potentially putting entire sub-hierarchies into
lower power states, which has exposed some gaps in the PCI subsystem
around power management support.

The first issue is that lspci accesses the VPD sysfs interface, which
does not provide the same power management wrappers as general config
space.

The next covers PME, where we attempt to skip devices based on their PCI
power state, but don't protect changes to that state or look at the
overall runtime power management state of the device.

This latter patch addresses the issue noted by Eric in the follow-ups to
v1 linked below.

These patches are logically independent, but only together resolve an
issue on Eric's system where a pair of endpoints bound to vfio-pci and
unused by userspace drivers trigger faults through lspci and PME
polling.  Thanks,

Alex 

v1: https://lore.kernel.org/all/20230707151044.1311544-1-alex.williamson@redhat.com/

Alex Williamson (2):
  PCI/VPD: Add runtime power management to sysfs interface
  PCI: Fix runtime PM race with PME polling

 drivers/pci/pci.c | 23 ++++++++++++++++-------
 drivers/pci/vpd.c | 34 ++++++++++++++++++++++++++++++++--
 2 files changed, 48 insertions(+), 9 deletions(-)

-- 
2.40.1

On Thu, Aug 03, 2023 at 11:12:31AM -0600, Alex Williamson wrote:
> Since v5.19, vfio-pci makes use of runtime power management on devices.
> This has the effect of potentially putting entire sub-hierarchies into
> lower power states, which has exposed some gaps in the PCI subsystem
> around power management support.
> 
> The first issue is that lspci accesses the VPD sysfs interface, which
> does not provide the same power management wrappers as general config
> space.
> 
> The next covers PME, where we attempt to skip devices based on their PCI
> power state, but don't protect changes to that state or look at the
> overall runtime power management state of the device.
> 
> This latter patch addresses the issue noted by Eric in the follow-ups to
> v1 linked below.
> 
> These patches are logically independent, but only together resolve an
> issue on Eric's system where a pair of endpoints bound to vfio-pci and
> unused by userspace drivers trigger faults through lspci and PME
> polling.  Thanks,
> 
> Alex 
> 
> v1: https://lore.kernel.org/all/20230707151044.1311544-1-alex.williamson@redhat.com/
> 
> Alex Williamson (2):
>   PCI/VPD: Add runtime power management to sysfs interface
>   PCI: Fix runtime PM race with PME polling
> 
>  drivers/pci/pci.c | 23 ++++++++++++++++-------
>  drivers/pci/vpd.c | 34 ++++++++++++++++++++++++++++++++--
>  2 files changed, 48 insertions(+), 9 deletions(-)

Applied with the tweak below to pci/vpd for v6.6, thanks!  The idea is
to match the pci_get_func0_dev() so the get/put balance is clear
without having to analyze PCI_DEV_FLAGS_VPD_REF_F0 usage:

-       if (dev != vpd_dev)
+       if (dev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0)

On Thu, 10 Aug 2023 13:29:44 -0500
Bjorn Helgaas <helgaas@kernel.org> wrote:

> On Thu, Aug 03, 2023 at 11:12:31AM -0600, Alex Williamson wrote:
> > Since v5.19, vfio-pci makes use of runtime power management on devices.
> > This has the effect of potentially putting entire sub-hierarchies into
> > lower power states, which has exposed some gaps in the PCI subsystem
> > around power management support.
> > 
> > The first issue is that lspci accesses the VPD sysfs interface, which
> > does not provide the same power management wrappers as general config
> > space.
> > 
> > The next covers PME, where we attempt to skip devices based on their PCI
> > power state, but don't protect changes to that state or look at the
> > overall runtime power management state of the device.
> > 
> > This latter patch addresses the issue noted by Eric in the follow-ups to
> > v1 linked below.
> > 
> > These patches are logically independent, but only together resolve an
> > issue on Eric's system where a pair of endpoints bound to vfio-pci and
> > unused by userspace drivers trigger faults through lspci and PME
> > polling.  Thanks,
> > 
> > Alex 
> > 
> > v1: https://lore.kernel.org/all/20230707151044.1311544-1-alex.williamson@redhat.com/
> > 
> > Alex Williamson (2):
> >   PCI/VPD: Add runtime power management to sysfs interface
> >   PCI: Fix runtime PM race with PME polling
> > 
> >  drivers/pci/pci.c | 23 ++++++++++++++++-------
> >  drivers/pci/vpd.c | 34 ++++++++++++++++++++++++++++++++--
> >  2 files changed, 48 insertions(+), 9 deletions(-)  
> 
> Applied with the tweak below to pci/vpd for v6.6, thanks!  The idea is
> to match the pci_get_func0_dev() so the get/put balance is clear
> without having to analyze PCI_DEV_FLAGS_VPD_REF_F0 usage:
> 
> -       if (dev != vpd_dev)
> +       if (dev->dev_flags & PCI_DEV_FLAGS_VPD_REF_F0)
> 

Looks good, thanks!

Alex