[v3] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

[PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

Posted by Niklas Schnelle 1 month, 3 weeks ago

Hi Bjorn,

Doing additional testing for a distribution backport of commit
05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
enabling/disabling SR-IOV") Benjamin found a hang with s390's
recover attribute. Further investigation showed this to be a deadlock by
recursively trying to take pci_rescan_remove lock when removing a PF
with enabled SR-IOV.

The issue can be reproduced on both s390 and x86_64 with:

    $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
    $ echo 1 > /sys/bus/pci/devices/<pf>/remove

As this seems worse than the original, hard to hit, race fixed by the
cited commit I think we first want to revert the broken fix.

Following that patch 2 attempts to fix the original issue by taking the
PCI rescan/remove lock directly before calling into the driver's
sriov_configure() callback enforcing the rule that this should only
be called with the pci_rescan_remove_lock held.

Thanks,
Niklas

Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
Changes in v3:
- Rebased on v6.19-rc1, also verified issue is still there and the fix
  still works
- Added more of the lockdep splat for better context
- Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com

Changes in v2:
- Collected R-b from Benjamin
- Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com

---
Niklas Schnelle (2):
      Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
      PCI/IOV: Fix race between SR-IOV enable/disable and hotplug

 drivers/pci/iov.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
---
base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
change-id: 20251029-revert_sriov_lock-aef4557f360f

Best regards,
-- 
Niklas Schnelle

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

Posted by Bjorn Helgaas 4 days, 3 hours ago

On Tue, Dec 16, 2025 at 11:14:01PM +0100, Niklas Schnelle wrote:
> Hi Bjorn,
> 
> Doing additional testing for a distribution backport of commit
> 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
> enabling/disabling SR-IOV") Benjamin found a hang with s390's
> recover attribute. Further investigation showed this to be a deadlock by
> recursively trying to take pci_rescan_remove lock when removing a PF
> with enabled SR-IOV.
> 
> The issue can be reproduced on both s390 and x86_64 with:
> 
>     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
>     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> 
> As this seems worse than the original, hard to hit, race fixed by the
> cited commit I think we first want to revert the broken fix.
> 
> Following that patch 2 attempts to fix the original issue by taking the
> PCI rescan/remove lock directly before calling into the driver's
> sriov_configure() callback enforcing the rule that this should only
> be called with the pci_rescan_remove_lock held.
> 
> Thanks,
> Niklas
> 
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
> Changes in v3:
> - Rebased on v6.19-rc1, also verified issue is still there and the fix
>   still works
> - Added more of the lockdep splat for better context
> - Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com
> 
> Changes in v2:
> - Collected R-b from Benjamin
> - Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com
> 
> ---
> Niklas Schnelle (2):
>       Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
>       PCI/IOV: Fix race between SR-IOV enable/disable and hotplug
> 
>  drivers/pci/iov.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> ---
> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
> change-id: 20251029-revert_sriov_lock-aef4557f360f

Applied to pci/iov for v6.20, thanks!

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

Posted by Thorsten Leemhuis 5 days, 12 hours ago

Lo! Top-posting to facilitate processing.

The issue fixed by these patches is on my list on tracked regressions.
Makes me wonder: Did it fall through the cracks due to the festive
season, was this fixes in a different way, or is this obsolete for some
reason?

Ciao, Thorsten

On 12/16/25 23:14, Niklas Schnelle wrote:
> Hi Bjorn,
> 
> Doing additional testing for a distribution backport of commit
> 05703271c3cd ("PCI/IOV: Add PCI rescan-remove locking when
> enabling/disabling SR-IOV") Benjamin found a hang with s390's
> recover attribute. Further investigation showed this to be a deadlock by
> recursively trying to take pci_rescan_remove lock when removing a PF
> with enabled SR-IOV.
> 
> The issue can be reproduced on both s390 and x86_64 with:
> 
>     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
>     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> 
> As this seems worse than the original, hard to hit, race fixed by the
> cited commit I think we first want to revert the broken fix.
> 
> Following that patch 2 attempts to fix the original issue by taking the
> PCI rescan/remove lock directly before calling into the driver's
> sriov_configure() callback enforcing the rule that this should only
> be called with the pci_rescan_remove_lock held.
> 
> Thanks,
> Niklas
> 
> Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
> ---
> Changes in v3:
> - Rebased on v6.19-rc1, also verified issue is still there and the fix
>   still works
> - Added more of the lockdep splat for better context
> - Link to v2: https://lore.kernel.org/r/20251119-revert_sriov_lock-v2-0-ea50eb1e8f96@linux.ibm.com
> 
> Changes in v2:
> - Collected R-b from Benjamin
> - Link to v1: https://lore.kernel.org/r/20251030-revert_sriov_lock-v1-0-70f82ade426f@linux.ibm.com
> 
> ---
> Niklas Schnelle (2):
>       Revert "PCI/IOV: Add PCI rescan-remove locking when enabling/disabling SR-IOV"
>       PCI/IOV: Fix race between SR-IOV enable/disable and hotplug
> 
>  drivers/pci/iov.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> ---
> base-commit: 8f0b4cce4481fb22653697cced8d0d04027cb1e8
> change-id: 20251029-revert_sriov_lock-aef4557f360f
> 
> Best regards,

Re: [PATCH v3 0/2] PCI/IOV: Fix deadlock when removing PF with enabled SR-IOV

Posted by Niklas Schnelle 4 days, 12 hours ago

On Sun, 2026-02-01 at 16:56 +0100, Thorsten Leemhuis wrote:
> Lo! Top-posting to facilitate processing.
> 
> The issue fixed by these patches is on my list on tracked regressions.
> Makes me wonder: Did it fall through the cracks due to the festive
> season, was this fixes in a different way, or is this obsolete for some
> reason?
> 
> Ciao, Thorsten
> 
> On 12/16/25 23:14, Niklas Schnelle wrote:
> > 
--- snip ---
> > The issue can be reproduced on both s390 and x86_64 with:
> > 
> >     $ echo <NUM> > /sys/bus/pci/devices/<pf>/sriov_numvfs
> >     $ echo 1 > /sys/bus/pci/devices/<pf>/remove
> > 
> > As this seems worse than the original, hard to hit, race fixed by the
> > cited commit I think we first want to revert the broken fix.

Hi Thorsten,

The issue is definitely still current, I just reproduced with the above
instructions on v6.19-rc8.

I fear in general there may still be some further issues with SR-IOV PF
disable path but this is still the best fix I know for my broken commit
(see Fixes tag) and the issue it tried to fix.

Thanks,
Niklas