drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++--- drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++-- 2 files changed, 296 insertions(+), 47 deletions(-)
This series fixes doorbell bit/vector handling for the EPF-based NTB
pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
without changing the on-the-wire doorbell mapping.
Background / problem
====================
ntb_hw_epf historically applies an extra offset when ringing peer
doorbells: the link event uses the first interrupt slot, and doorbells
start from the third slot (i.e. a second slot is effectively unused).
pci-epf-vntb carries the matching offset on the EP side as well.
As long as db_vector_count()/db_vector_mask() are not implemented, this
mismatch is mostly masked. Doorbell events are effectively treated as
"can hit any QP" and the off-by-one vector numbering does not surface
clearly.
However, once per-vector handling is enabled, the current state becomes
problematic:
- db_valid_mask exposes bits that do not correspond to real doorbells
(link/unused slots leak into the mask).
- ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
expects a 0-based db_vector for doorbells.
- On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
it directly calls pci_epc_raise_irq(), which can sleep.
Why NOT fix the root offset?
============================
The natural "root" fix would be to remove the historical extra offset in
the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
Unfortunately this would lead to interoperability issues when mixing old
and new kernel versions (old/new peers). A new side would ring a
different interrupt slot than what an old peer expects, leading to
missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
are implemented.
Therefore this series intentionally keeps the legacy offset, and instead
fixes the surrounding pieces so the mapping is documented and handled
consistently in masks, vector numbering, and per-vector reporting.
Dependencies
============
This v4 is prepared on top of the current pci/endpoint:
commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
doorbell trigger")
plus these patches as contextual prerequisites:
- [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
- [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
(These touch the same code part as this series and would otherwise conflict
with it. They should land upstream first in my opinion.)
Practical benefit
=================
This also enables ntb_netdev's multi-queue performance scale:
[PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
Each netdev queue maps to an ntb_transport queue pair. With correct
db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
only the queue pairs covered by the doorbell vector it received. Without
those callbacks, the NTB core falls back to the full valid doorbell mask,
so each doorbell interrupt can still wake all queue pairs. See "Testing /
Results" section in the above link.
What this series does
=====================
- pci-epf-vntb:
- Document the legacy offset.
- Defer MSI doorbell raises to process context to avoid sleeping in
atomic context. This becomes relevant once multiple doorbells are
raised concurrently at a high rate.
- Report doorbell vectors as 0-based to ntb_db_event().
- Reject unusable doorbell counts and guard configfs writes after EPC
attach with -EOPNOTSUPP.
- Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
- ntb_hw_epf:
- Document the legacy offset in ntb_epf_peer_db_set().
- Fix db_valid_mask to cover only real doorbell bits.
- Report 0-based db_vector to ntb_db_event() (accounting for the
unused slot).
- Pass per-vector context as request_irq() dev_id instead of deriving the
device vector from Linux IRQ numbers.
- Keep db_val as a bitmask and fix db_read/db_clear semantics
accordingly.
- Implement db_vector_count()/db_vector_mask().
Compatibility
=============
By keeping the legacy offset intact, this series aims to remain
compatible across mixed kernel versions. The observable changes are
limited to correct mask/vector reporting and safer execution context
handling.
Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
applied separately through the respective trees. They are sent together so
the two sides of the same legacy layout do not drift apart, and the
resulting code stays easier to follow.
The plan is still to take the whole series through the PCI EP tree once the
remaining NTB acks are collected. See:
https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
Note
====
Most v3->v4 changes are responses to Sashiko feedback:
https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
Sashiko also flagged a broader driver-unbind lifetime concern around the
vNTB PCI driver. That is real, but it predates this series and needs a
proper .remove() path, including NTB unregistration and work cancellation.
I am keeping it as follow-up lifecycle work:
https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
This v4 only clears the deferred doorbell state added by this series around
EPC init/cleanup.
---
Changelog
=========
Changes since v3:
- Rebased on the dependencies listed above.
- For Sashiko feedback on state added by this series: clear
peer_db_pending around work enable/disable, mask peer_db_set() with
db_valid_mask, and use __ffs64().
- For pre-existing issues made more relevant by per-vector semantics:
stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
per-vector request_irq() context, ignore the reserved slot, and
validate vectors before BIT_ULL()/ntb_db_event().
- Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
cleanups.
- Dropped Reviewed-by tags from patches that changed to non-trivial extent.
Changes since v2:
- No functional changes.
- Rebased onto current pci/endpoint
e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
* Patch 2 needed a trivial context-only adjustment while rebasing, due
to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
cmd_handler work in epf_ntb_epc_cleanup").
- Picked up additional Reviewed-by tags from Frank.
- Fixed the incorrect v2 series title.
Changes since v1:
- Addressed feedback from Dave (add a source code comment, introduce
enum to eliminate magic numbers)
- Updated source code comment in Patch 2.
- No functional changes, so retained Reviewed-by tags by Frank and Dave.
Thank you both for the review.
v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
Best regards,
Koichiro
Koichiro Den (12):
PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
context
PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
ntb_db_event()
PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
doorbells
NTB: epf: Document legacy doorbell slot offset in
ntb_epf_peer_db_set()
NTB: epf: Make db_valid_mask cover only real doorbell bits
NTB: epf: Report 0-based doorbell vector via ntb_db_event()
NTB: epf: Fix doorbell bitmask and IRQ vector handling
NTB: epf: Implement db_vector_count/mask for doorbells
drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
2 files changed, 296 insertions(+), 47 deletions(-)
--
2.51.0
On Wed, 13 May 2026 11:49:11 +0900, Koichiro Den wrote:
> This series fixes doorbell bit/vector handling for the EPF-based NTB
> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
> without changing the on-the-wire doorbell mapping.
>
>
> Background / problem
> ====================
>
> [...]
Applied, thanks!
[01/12] PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
commit: 683f8a747ab35886cc9bef96025712011309eae5
[02/12] PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic context
commit: 84dd5950df2f540e52bd6cbcb5682cb4dff5f2a5
[03/12] PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via ntb_db_event()
commit: 7c54a7152588e8892a7f7f2c69117f00c1bf24f4
[04/12] PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
commit: 86b48f07a5a398ce1cd88e2f06ddab52c2181df0
[05/12] PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
commit: 9da8fecbe9173cc1d48667b2b821e4d1f383edbb
[06/12] PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
commit: 12f666206720cb462ac4eaeb61a03e492737e1ef
[07/12] PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for doorbells
commit: d7eacad42e2d812872fc7da83466657b7759dde1
[08/12] NTB: epf: Document legacy doorbell slot offset in ntb_epf_peer_db_set()
commit: d3de5eb64ecd836160afad84aa3ff639acb489cb
[09/12] NTB: epf: Make db_valid_mask cover only real doorbell bits
commit: 615dc10082a96423743d3a7b8b90eed80580916d
[10/12] NTB: epf: Report 0-based doorbell vector via ntb_db_event()
commit: c8220ccb30e8c092db9b6fa7dcb06dfd5340457d
[11/12] NTB: epf: Fix doorbell bitmask and IRQ vector handling
commit: 1c981bf355e8ab78c45717fbd322bd67fafa8082
[12/12] NTB: epf: Implement db_vector_count/mask for doorbells
commit: e6ff30150738af5c7aa20fad08b74cbf9b86d3a2
Best regards,
--
Manivannan Sadhasivam <mani@kernel.org>
On 5/12/26 7:49 PM, Koichiro Den wrote:
> This series fixes doorbell bit/vector handling for the EPF-based NTB
> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
> without changing the on-the-wire doorbell mapping.
>
>
> Background / problem
> ====================
>
> ntb_hw_epf historically applies an extra offset when ringing peer
> doorbells: the link event uses the first interrupt slot, and doorbells
> start from the third slot (i.e. a second slot is effectively unused).
> pci-epf-vntb carries the matching offset on the EP side as well.
>
> As long as db_vector_count()/db_vector_mask() are not implemented, this
> mismatch is mostly masked. Doorbell events are effectively treated as
> "can hit any QP" and the off-by-one vector numbering does not surface
> clearly.
>
> However, once per-vector handling is enabled, the current state becomes
> problematic:
>
> - db_valid_mask exposes bits that do not correspond to real doorbells
> (link/unused slots leak into the mask).
> - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
> expects a 0-based db_vector for doorbells.
> - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
> it directly calls pci_epc_raise_irq(), which can sleep.
>
>
> Why NOT fix the root offset?
> ============================
>
> The natural "root" fix would be to remove the historical extra offset in
> the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
> Unfortunately this would lead to interoperability issues when mixing old
> and new kernel versions (old/new peers). A new side would ring a
> different interrupt slot than what an old peer expects, leading to
> missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
> are implemented.
>
> Therefore this series intentionally keeps the legacy offset, and instead
> fixes the surrounding pieces so the mapping is documented and handled
> consistently in masks, vector numbering, and per-vector reporting.
>
>
> Dependencies
> ============
>
> This v4 is prepared on top of the current pci/endpoint:
> commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
> doorbell trigger")
I can't seem to apply the patch set via 'b4 shazam' on top of f5aa97125491. Trying to set it up for review.
DJ
>
> plus these patches as contextual prerequisites:
>
> - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
> https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
>
> - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
> https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
>
> (These touch the same code part as this series and would otherwise conflict
> with it. They should land upstream first in my opinion.)
>
>
> Practical benefit
> =================
>
> This also enables ntb_netdev's multi-queue performance scale:
>
> [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
> https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
>
> Each netdev queue maps to an ntb_transport queue pair. With correct
> db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
> only the queue pairs covered by the doorbell vector it received. Without
> those callbacks, the NTB core falls back to the full valid doorbell mask,
> so each doorbell interrupt can still wake all queue pairs. See "Testing /
> Results" section in the above link.
>
>
> What this series does
> =====================
>
> - pci-epf-vntb:
>
> - Document the legacy offset.
> - Defer MSI doorbell raises to process context to avoid sleeping in
> atomic context. This becomes relevant once multiple doorbells are
> raised concurrently at a high rate.
> - Report doorbell vectors as 0-based to ntb_db_event().
> - Reject unusable doorbell counts and guard configfs writes after EPC
> attach with -EOPNOTSUPP.
> - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
>
> - ntb_hw_epf:
>
> - Document the legacy offset in ntb_epf_peer_db_set().
> - Fix db_valid_mask to cover only real doorbell bits.
> - Report 0-based db_vector to ntb_db_event() (accounting for the
> unused slot).
> - Pass per-vector context as request_irq() dev_id instead of deriving the
> device vector from Linux IRQ numbers.
> - Keep db_val as a bitmask and fix db_read/db_clear semantics
> accordingly.
> - Implement db_vector_count()/db_vector_mask().
>
>
> Compatibility
> =============
>
> By keeping the legacy offset intact, this series aims to remain
> compatible across mixed kernel versions. The observable changes are
> limited to correct mask/vector reporting and safer execution context
> handling.
>
> Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
> applied separately through the respective trees. They are sent together so
> the two sides of the same legacy layout do not drift apart, and the
> resulting code stays easier to follow.
>
> The plan is still to take the whole series through the PCI EP tree once the
> remaining NTB acks are collected. See:
> https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
>
>
> Note
> ====
>
> Most v3->v4 changes are responses to Sashiko feedback:
> https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
>
> Sashiko also flagged a broader driver-unbind lifetime concern around the
> vNTB PCI driver. That is real, but it predates this series and needs a
> proper .remove() path, including NTB unregistration and work cancellation.
> I am keeping it as follow-up lifecycle work:
>
> https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
>
> This v4 only clears the deferred doorbell state added by this series around
> EPC init/cleanup.
>
>
> ---
> Changelog
> =========
>
> Changes since v3:
> - Rebased on the dependencies listed above.
> - For Sashiko feedback on state added by this series: clear
> peer_db_pending around work enable/disable, mask peer_db_set() with
> db_valid_mask, and use __ffs64().
> - For pre-existing issues made more relevant by per-vector semantics:
> stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
> per-vector request_irq() context, ignore the reserved slot, and
> validate vectors before BIT_ULL()/ntb_db_event().
> - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
> bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
> cleanups.
> - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
>
> Changes since v2:
> - No functional changes.
> - Rebased onto current pci/endpoint
> e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
> * Patch 2 needed a trivial context-only adjustment while rebasing, due
> to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
> cmd_handler work in epf_ntb_epc_cleanup").
> - Picked up additional Reviewed-by tags from Frank.
> - Fixed the incorrect v2 series title.
>
> Changes since v1:
> - Addressed feedback from Dave (add a source code comment, introduce
> enum to eliminate magic numbers)
> - Updated source code comment in Patch 2.
> - No functional changes, so retained Reviewed-by tags by Frank and Dave.
> Thank you both for the review.
>
> v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
> v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
> v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
>
>
> Best regards,
> Koichiro
>
>
> Koichiro Den (12):
> PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
> PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
> context
> PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
> ntb_db_event()
> PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
> PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
> PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
> PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
> doorbells
> NTB: epf: Document legacy doorbell slot offset in
> ntb_epf_peer_db_set()
> NTB: epf: Make db_valid_mask cover only real doorbell bits
> NTB: epf: Report 0-based doorbell vector via ntb_db_event()
> NTB: epf: Fix doorbell bitmask and IRQ vector handling
> NTB: epf: Implement db_vector_count/mask for doorbells
>
> drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
> drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
> 2 files changed, 296 insertions(+), 47 deletions(-)
>
On Tue, May 19, 2026 at 02:29:51PM -0700, Dave Jiang wrote:
>
>
> On 5/12/26 7:49 PM, Koichiro Den wrote:
> > This series fixes doorbell bit/vector handling for the EPF-based NTB
> > pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
> > per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
> > without changing the on-the-wire doorbell mapping.
> >
> >
> > Background / problem
> > ====================
> >
> > ntb_hw_epf historically applies an extra offset when ringing peer
> > doorbells: the link event uses the first interrupt slot, and doorbells
> > start from the third slot (i.e. a second slot is effectively unused).
> > pci-epf-vntb carries the matching offset on the EP side as well.
> >
> > As long as db_vector_count()/db_vector_mask() are not implemented, this
> > mismatch is mostly masked. Doorbell events are effectively treated as
> > "can hit any QP" and the off-by-one vector numbering does not surface
> > clearly.
> >
> > However, once per-vector handling is enabled, the current state becomes
> > problematic:
> >
> > - db_valid_mask exposes bits that do not correspond to real doorbells
> > (link/unused slots leak into the mask).
> > - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
> > expects a 0-based db_vector for doorbells.
> > - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
> > it directly calls pci_epc_raise_irq(), which can sleep.
> >
> >
> > Why NOT fix the root offset?
> > ============================
> >
> > The natural "root" fix would be to remove the historical extra offset in
> > the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
> > Unfortunately this would lead to interoperability issues when mixing old
> > and new kernel versions (old/new peers). A new side would ring a
> > different interrupt slot than what an old peer expects, leading to
> > missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
> > are implemented.
> >
> > Therefore this series intentionally keeps the legacy offset, and instead
> > fixes the surrounding pieces so the mapping is documented and handled
> > consistently in masks, vector numbering, and per-vector reporting.
> >
> >
> > Dependencies
> > ============
> >
> > This v4 is prepared on top of the current pci/endpoint:
> > commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
> > doorbell trigger")
>
> I can't seem to apply the patch set via 'b4 shazam' on top of f5aa97125491. Trying to set it up for review.
Hi Dave,
>
> DJ
>
> >
> > plus these patches as contextual prerequisites:
> >
> > - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
> > https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
> >
> > - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
> > https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
> >
> > (These touch the same code part as this series and would otherwise conflict
> > with it. They should land upstream first in my opinion.)
This series depends on the two prerequisite series listed above.
Best regards,
Koichiro
> >
> >
> > Practical benefit
> > =================
> >
> > This also enables ntb_netdev's multi-queue performance scale:
> >
> > [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
> > https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
> >
> > Each netdev queue maps to an ntb_transport queue pair. With correct
> > db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
> > only the queue pairs covered by the doorbell vector it received. Without
> > those callbacks, the NTB core falls back to the full valid doorbell mask,
> > so each doorbell interrupt can still wake all queue pairs. See "Testing /
> > Results" section in the above link.
> >
> >
> > What this series does
> > =====================
> >
> > - pci-epf-vntb:
> >
> > - Document the legacy offset.
> > - Defer MSI doorbell raises to process context to avoid sleeping in
> > atomic context. This becomes relevant once multiple doorbells are
> > raised concurrently at a high rate.
> > - Report doorbell vectors as 0-based to ntb_db_event().
> > - Reject unusable doorbell counts and guard configfs writes after EPC
> > attach with -EOPNOTSUPP.
> > - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
> >
> > - ntb_hw_epf:
> >
> > - Document the legacy offset in ntb_epf_peer_db_set().
> > - Fix db_valid_mask to cover only real doorbell bits.
> > - Report 0-based db_vector to ntb_db_event() (accounting for the
> > unused slot).
> > - Pass per-vector context as request_irq() dev_id instead of deriving the
> > device vector from Linux IRQ numbers.
> > - Keep db_val as a bitmask and fix db_read/db_clear semantics
> > accordingly.
> > - Implement db_vector_count()/db_vector_mask().
> >
> >
> > Compatibility
> > =============
> >
> > By keeping the legacy offset intact, this series aims to remain
> > compatible across mixed kernel versions. The observable changes are
> > limited to correct mask/vector reporting and safer execution context
> > handling.
> >
> > Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
> > applied separately through the respective trees. They are sent together so
> > the two sides of the same legacy layout do not drift apart, and the
> > resulting code stays easier to follow.
> >
> > The plan is still to take the whole series through the PCI EP tree once the
> > remaining NTB acks are collected. See:
> > https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
> >
> >
> > Note
> > ====
> >
> > Most v3->v4 changes are responses to Sashiko feedback:
> > https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
> >
> > Sashiko also flagged a broader driver-unbind lifetime concern around the
> > vNTB PCI driver. That is real, but it predates this series and needs a
> > proper .remove() path, including NTB unregistration and work cancellation.
> > I am keeping it as follow-up lifecycle work:
> >
> > https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
> >
> > This v4 only clears the deferred doorbell state added by this series around
> > EPC init/cleanup.
> >
> >
> > ---
> > Changelog
> > =========
> >
> > Changes since v3:
> > - Rebased on the dependencies listed above.
> > - For Sashiko feedback on state added by this series: clear
> > peer_db_pending around work enable/disable, mask peer_db_set() with
> > db_valid_mask, and use __ffs64().
> > - For pre-existing issues made more relevant by per-vector semantics:
> > stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
> > per-vector request_irq() context, ignore the reserved slot, and
> > validate vectors before BIT_ULL()/ntb_db_event().
> > - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
> > bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
> > cleanups.
> > - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
> >
> > Changes since v2:
> > - No functional changes.
> > - Rebased onto current pci/endpoint
> > e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
> > * Patch 2 needed a trivial context-only adjustment while rebasing, due
> > to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
> > cmd_handler work in epf_ntb_epc_cleanup").
> > - Picked up additional Reviewed-by tags from Frank.
> > - Fixed the incorrect v2 series title.
> >
> > Changes since v1:
> > - Addressed feedback from Dave (add a source code comment, introduce
> > enum to eliminate magic numbers)
> > - Updated source code comment in Patch 2.
> > - No functional changes, so retained Reviewed-by tags by Frank and Dave.
> > Thank you both for the review.
> >
> > v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
> > v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
> > v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
> >
> >
> > Best regards,
> > Koichiro
> >
> >
> > Koichiro Den (12):
> > PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
> > PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
> > context
> > PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
> > ntb_db_event()
> > PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
> > PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
> > PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
> > PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
> > doorbells
> > NTB: epf: Document legacy doorbell slot offset in
> > ntb_epf_peer_db_set()
> > NTB: epf: Make db_valid_mask cover only real doorbell bits
> > NTB: epf: Report 0-based doorbell vector via ntb_db_event()
> > NTB: epf: Fix doorbell bitmask and IRQ vector handling
> > NTB: epf: Implement db_vector_count/mask for doorbells
> >
> > drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
> > drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
> > 2 files changed, 296 insertions(+), 47 deletions(-)
> >
>
On 5/20/26 11:29 PM, Koichiro Den wrote:
> On Tue, May 19, 2026 at 02:29:51PM -0700, Dave Jiang wrote:
>>
>>
>> On 5/12/26 7:49 PM, Koichiro Den wrote:
>>> This series fixes doorbell bit/vector handling for the EPF-based NTB
>>> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
>>> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
>>> without changing the on-the-wire doorbell mapping.
>>>
>>>
>>> Background / problem
>>> ====================
>>>
>>> ntb_hw_epf historically applies an extra offset when ringing peer
>>> doorbells: the link event uses the first interrupt slot, and doorbells
>>> start from the third slot (i.e. a second slot is effectively unused).
>>> pci-epf-vntb carries the matching offset on the EP side as well.
>>>
>>> As long as db_vector_count()/db_vector_mask() are not implemented, this
>>> mismatch is mostly masked. Doorbell events are effectively treated as
>>> "can hit any QP" and the off-by-one vector numbering does not surface
>>> clearly.
>>>
>>> However, once per-vector handling is enabled, the current state becomes
>>> problematic:
>>>
>>> - db_valid_mask exposes bits that do not correspond to real doorbells
>>> (link/unused slots leak into the mask).
>>> - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
>>> expects a 0-based db_vector for doorbells.
>>> - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
>>> it directly calls pci_epc_raise_irq(), which can sleep.
>>>
>>>
>>> Why NOT fix the root offset?
>>> ============================
>>>
>>> The natural "root" fix would be to remove the historical extra offset in
>>> the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
>>> Unfortunately this would lead to interoperability issues when mixing old
>>> and new kernel versions (old/new peers). A new side would ring a
>>> different interrupt slot than what an old peer expects, leading to
>>> missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
>>> are implemented.
>>>
>>> Therefore this series intentionally keeps the legacy offset, and instead
>>> fixes the surrounding pieces so the mapping is documented and handled
>>> consistently in masks, vector numbering, and per-vector reporting.
>>>
>>>
>>> Dependencies
>>> ============
>>>
>>> This v4 is prepared on top of the current pci/endpoint:
>>> commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
>>> doorbell trigger")
>>
>> I can't seem to apply the patch set via 'b4 shazam' on top of f5aa97125491. Trying to set it up for review.
>
> Hi Dave,
>
>>
>> DJ
>>
>>>
>>> plus these patches as contextual prerequisites:
>>>
>>> - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
>>> https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
>>>
>>> - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
>>> https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
>>>
>>> (These touch the same code part as this series and would otherwise conflict
>>> with it. They should land upstream first in my opinion.)
>
> This series depends on the two prerequisite series listed above.
They don't apply cleanly either. Do you have a public repo somewhere that has everything applied? That may be the easiest way to do things if a series has complicated dependencies.
DJ
>
> Best regards,
> Koichiro
>
>>>
>>>
>>> Practical benefit
>>> =================
>>>
>>> This also enables ntb_netdev's multi-queue performance scale:
>>>
>>> [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
>>> https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
>>>
>>> Each netdev queue maps to an ntb_transport queue pair. With correct
>>> db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
>>> only the queue pairs covered by the doorbell vector it received. Without
>>> those callbacks, the NTB core falls back to the full valid doorbell mask,
>>> so each doorbell interrupt can still wake all queue pairs. See "Testing /
>>> Results" section in the above link.
>>>
>>>
>>> What this series does
>>> =====================
>>>
>>> - pci-epf-vntb:
>>>
>>> - Document the legacy offset.
>>> - Defer MSI doorbell raises to process context to avoid sleeping in
>>> atomic context. This becomes relevant once multiple doorbells are
>>> raised concurrently at a high rate.
>>> - Report doorbell vectors as 0-based to ntb_db_event().
>>> - Reject unusable doorbell counts and guard configfs writes after EPC
>>> attach with -EOPNOTSUPP.
>>> - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
>>>
>>> - ntb_hw_epf:
>>>
>>> - Document the legacy offset in ntb_epf_peer_db_set().
>>> - Fix db_valid_mask to cover only real doorbell bits.
>>> - Report 0-based db_vector to ntb_db_event() (accounting for the
>>> unused slot).
>>> - Pass per-vector context as request_irq() dev_id instead of deriving the
>>> device vector from Linux IRQ numbers.
>>> - Keep db_val as a bitmask and fix db_read/db_clear semantics
>>> accordingly.
>>> - Implement db_vector_count()/db_vector_mask().
>>>
>>>
>>> Compatibility
>>> =============
>>>
>>> By keeping the legacy offset intact, this series aims to remain
>>> compatible across mixed kernel versions. The observable changes are
>>> limited to correct mask/vector reporting and safer execution context
>>> handling.
>>>
>>> Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
>>> applied separately through the respective trees. They are sent together so
>>> the two sides of the same legacy layout do not drift apart, and the
>>> resulting code stays easier to follow.
>>>
>>> The plan is still to take the whole series through the PCI EP tree once the
>>> remaining NTB acks are collected. See:
>>> https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
>>>
>>>
>>> Note
>>> ====
>>>
>>> Most v3->v4 changes are responses to Sashiko feedback:
>>> https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
>>>
>>> Sashiko also flagged a broader driver-unbind lifetime concern around the
>>> vNTB PCI driver. That is real, but it predates this series and needs a
>>> proper .remove() path, including NTB unregistration and work cancellation.
>>> I am keeping it as follow-up lifecycle work:
>>>
>>> https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
>>>
>>> This v4 only clears the deferred doorbell state added by this series around
>>> EPC init/cleanup.
>>>
>>>
>>> ---
>>> Changelog
>>> =========
>>>
>>> Changes since v3:
>>> - Rebased on the dependencies listed above.
>>> - For Sashiko feedback on state added by this series: clear
>>> peer_db_pending around work enable/disable, mask peer_db_set() with
>>> db_valid_mask, and use __ffs64().
>>> - For pre-existing issues made more relevant by per-vector semantics:
>>> stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
>>> per-vector request_irq() context, ignore the reserved slot, and
>>> validate vectors before BIT_ULL()/ntb_db_event().
>>> - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
>>> bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
>>> cleanups.
>>> - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
>>>
>>> Changes since v2:
>>> - No functional changes.
>>> - Rebased onto current pci/endpoint
>>> e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
>>> * Patch 2 needed a trivial context-only adjustment while rebasing, due
>>> to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
>>> cmd_handler work in epf_ntb_epc_cleanup").
>>> - Picked up additional Reviewed-by tags from Frank.
>>> - Fixed the incorrect v2 series title.
>>>
>>> Changes since v1:
>>> - Addressed feedback from Dave (add a source code comment, introduce
>>> enum to eliminate magic numbers)
>>> - Updated source code comment in Patch 2.
>>> - No functional changes, so retained Reviewed-by tags by Frank and Dave.
>>> Thank you both for the review.
>>>
>>> v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
>>> v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
>>> v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
>>>
>>>
>>> Best regards,
>>> Koichiro
>>>
>>>
>>> Koichiro Den (12):
>>> PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
>>> PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
>>> context
>>> PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
>>> ntb_db_event()
>>> PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
>>> PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
>>> PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
>>> PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
>>> doorbells
>>> NTB: epf: Document legacy doorbell slot offset in
>>> ntb_epf_peer_db_set()
>>> NTB: epf: Make db_valid_mask cover only real doorbell bits
>>> NTB: epf: Report 0-based doorbell vector via ntb_db_event()
>>> NTB: epf: Fix doorbell bitmask and IRQ vector handling
>>> NTB: epf: Implement db_vector_count/mask for doorbells
>>>
>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
>>> 2 files changed, 296 insertions(+), 47 deletions(-)
>>>
>>
>
On Tue, May 26, 2026 at 07:45:11AM -0700, Dave Jiang wrote:
>
>
> On 5/20/26 11:29 PM, Koichiro Den wrote:
> > On Tue, May 19, 2026 at 02:29:51PM -0700, Dave Jiang wrote:
> >>
> >>
> >> On 5/12/26 7:49 PM, Koichiro Den wrote:
> >>> This series fixes doorbell bit/vector handling for the EPF-based NTB
> >>> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
> >>> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
> >>> without changing the on-the-wire doorbell mapping.
> >>>
> >>>
> >>> Background / problem
> >>> ====================
> >>>
> >>> ntb_hw_epf historically applies an extra offset when ringing peer
> >>> doorbells: the link event uses the first interrupt slot, and doorbells
> >>> start from the third slot (i.e. a second slot is effectively unused).
> >>> pci-epf-vntb carries the matching offset on the EP side as well.
> >>>
> >>> As long as db_vector_count()/db_vector_mask() are not implemented, this
> >>> mismatch is mostly masked. Doorbell events are effectively treated as
> >>> "can hit any QP" and the off-by-one vector numbering does not surface
> >>> clearly.
> >>>
> >>> However, once per-vector handling is enabled, the current state becomes
> >>> problematic:
> >>>
> >>> - db_valid_mask exposes bits that do not correspond to real doorbells
> >>> (link/unused slots leak into the mask).
> >>> - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
> >>> expects a 0-based db_vector for doorbells.
> >>> - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
> >>> it directly calls pci_epc_raise_irq(), which can sleep.
> >>>
> >>>
> >>> Why NOT fix the root offset?
> >>> ============================
> >>>
> >>> The natural "root" fix would be to remove the historical extra offset in
> >>> the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
> >>> Unfortunately this would lead to interoperability issues when mixing old
> >>> and new kernel versions (old/new peers). A new side would ring a
> >>> different interrupt slot than what an old peer expects, leading to
> >>> missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
> >>> are implemented.
> >>>
> >>> Therefore this series intentionally keeps the legacy offset, and instead
> >>> fixes the surrounding pieces so the mapping is documented and handled
> >>> consistently in masks, vector numbering, and per-vector reporting.
> >>>
> >>>
> >>> Dependencies
> >>> ============
> >>>
> >>> This v4 is prepared on top of the current pci/endpoint:
> >>> commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
> >>> doorbell trigger")
> >>
> >> I can't seem to apply the patch set via 'b4 shazam' on top of f5aa97125491. Trying to set it up for review.
> >
> > Hi Dave,
> >
> >>
> >> DJ
> >>
> >>>
> >>> plus these patches as contextual prerequisites:
> >>>
> >>> - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
> >>> https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
> >>>
> >>> - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
> >>> https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
> >>>
> >>> (These touch the same code part as this series and would otherwise conflict
> >>> with it. They should land upstream first in my opinion.)
> >
> > This series depends on the two prerequisite series listed above.
>
> They don't apply cleanly either. Do you have a public repo somewhere that has everything applied? That may be the easiest way to do things if a series has complicated dependencies.
I rechecked this with a fresh worktree based off of f5aa97125491:
git worktree add /tmp/f5aa97125491 f5aa97125491
cd /tmp/f5aa97125491
# 1st dependency
b4 am 20260304083028.1391068-1-den@valinux.co.jp
git am ./20260304_den_ntb_epf_fix_ntb_hw_epf_isr_issues.mbx
# 2nd dependency
b4 am 20260407124421.282766-1-mani@kernel.org
git am ./20260407_manivannan_sadhasivam_pci_endpoint_pci_epf_v_ntb_a_couple_of_fixes.mbx
# this v4 series
b4 am 20260513024923.451765-1-den@valinux.co.jp
git am ./v4_20260513_den_pci_endpoint_pci_epf_vntb_ntb_epf_enable_per_doorbell_bit_handling.mbx
And this gives me the following commit log:
git log --reverse --oneline f5aa97125491..
# 1d1465542cc0 NTB: epf: Fix request_irq() unwind in ntb_epf_init_isr()
# 94bd43a9e435 NTB: epf: Avoid pci_irq_vector() from hardirq context
# 3cab4db0f647 PCI: endpoint: pci-epf-vntb: Add check to detect 'db_count' value of 0
# 5449f10512ae PCI: endpoint: pci-epf-ntb: Add check to detect 'db_count' value of 0
# e3dbdcceb2fd PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
# e1c167a58888 PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic context
# 6c54fd0c9c9c PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via ntb_db_event()
# 28a30b79ff03 PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
# cb5fbd37bdc6 PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
# 9ed0f4db85b4 PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
# 575bd3aa530b PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for doorbells
# 41f26aa279af NTB: epf: Document legacy doorbell slot offset in ntb_epf_peer_db_set()
# 6b2e2f80a61a NTB: epf: Make db_valid_mask cover only real doorbell bits
# 44d1bb070521 NTB: epf: Report 0-based doorbell vector via ntb_db_event()
# 4e4d37b4c2a8 NTB: epf: Fix doorbell bitmask and IRQ vector handling
# 1e5829682361 (HEAD) NTB: epf: Implement db_vector_count/mask for doorbells
I don't have a public repo at the moment, sorry. Could you try the above? If it
still fails, please let me know.
Best regards,
Koichiro
>
> DJ
>
> >
> > Best regards,
> > Koichiro
> >
> >>>
> >>>
> >>> Practical benefit
> >>> =================
> >>>
> >>> This also enables ntb_netdev's multi-queue performance scale:
> >>>
> >>> [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
> >>> https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
> >>>
> >>> Each netdev queue maps to an ntb_transport queue pair. With correct
> >>> db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
> >>> only the queue pairs covered by the doorbell vector it received. Without
> >>> those callbacks, the NTB core falls back to the full valid doorbell mask,
> >>> so each doorbell interrupt can still wake all queue pairs. See "Testing /
> >>> Results" section in the above link.
> >>>
> >>>
> >>> What this series does
> >>> =====================
> >>>
> >>> - pci-epf-vntb:
> >>>
> >>> - Document the legacy offset.
> >>> - Defer MSI doorbell raises to process context to avoid sleeping in
> >>> atomic context. This becomes relevant once multiple doorbells are
> >>> raised concurrently at a high rate.
> >>> - Report doorbell vectors as 0-based to ntb_db_event().
> >>> - Reject unusable doorbell counts and guard configfs writes after EPC
> >>> attach with -EOPNOTSUPP.
> >>> - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
> >>>
> >>> - ntb_hw_epf:
> >>>
> >>> - Document the legacy offset in ntb_epf_peer_db_set().
> >>> - Fix db_valid_mask to cover only real doorbell bits.
> >>> - Report 0-based db_vector to ntb_db_event() (accounting for the
> >>> unused slot).
> >>> - Pass per-vector context as request_irq() dev_id instead of deriving the
> >>> device vector from Linux IRQ numbers.
> >>> - Keep db_val as a bitmask and fix db_read/db_clear semantics
> >>> accordingly.
> >>> - Implement db_vector_count()/db_vector_mask().
> >>>
> >>>
> >>> Compatibility
> >>> =============
> >>>
> >>> By keeping the legacy offset intact, this series aims to remain
> >>> compatible across mixed kernel versions. The observable changes are
> >>> limited to correct mask/vector reporting and safer execution context
> >>> handling.
> >>>
> >>> Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
> >>> applied separately through the respective trees. They are sent together so
> >>> the two sides of the same legacy layout do not drift apart, and the
> >>> resulting code stays easier to follow.
> >>>
> >>> The plan is still to take the whole series through the PCI EP tree once the
> >>> remaining NTB acks are collected. See:
> >>> https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
> >>>
> >>>
> >>> Note
> >>> ====
> >>>
> >>> Most v3->v4 changes are responses to Sashiko feedback:
> >>> https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
> >>>
> >>> Sashiko also flagged a broader driver-unbind lifetime concern around the
> >>> vNTB PCI driver. That is real, but it predates this series and needs a
> >>> proper .remove() path, including NTB unregistration and work cancellation.
> >>> I am keeping it as follow-up lifecycle work:
> >>>
> >>> https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
> >>>
> >>> This v4 only clears the deferred doorbell state added by this series around
> >>> EPC init/cleanup.
> >>>
> >>>
> >>> ---
> >>> Changelog
> >>> =========
> >>>
> >>> Changes since v3:
> >>> - Rebased on the dependencies listed above.
> >>> - For Sashiko feedback on state added by this series: clear
> >>> peer_db_pending around work enable/disable, mask peer_db_set() with
> >>> db_valid_mask, and use __ffs64().
> >>> - For pre-existing issues made more relevant by per-vector semantics:
> >>> stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
> >>> per-vector request_irq() context, ignore the reserved slot, and
> >>> validate vectors before BIT_ULL()/ntb_db_event().
> >>> - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
> >>> bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
> >>> cleanups.
> >>> - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
> >>>
> >>> Changes since v2:
> >>> - No functional changes.
> >>> - Rebased onto current pci/endpoint
> >>> e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
> >>> * Patch 2 needed a trivial context-only adjustment while rebasing, due
> >>> to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
> >>> cmd_handler work in epf_ntb_epc_cleanup").
> >>> - Picked up additional Reviewed-by tags from Frank.
> >>> - Fixed the incorrect v2 series title.
> >>>
> >>> Changes since v1:
> >>> - Addressed feedback from Dave (add a source code comment, introduce
> >>> enum to eliminate magic numbers)
> >>> - Updated source code comment in Patch 2.
> >>> - No functional changes, so retained Reviewed-by tags by Frank and Dave.
> >>> Thank you both for the review.
> >>>
> >>> v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
> >>> v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
> >>> v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
> >>>
> >>>
> >>> Best regards,
> >>> Koichiro
> >>>
> >>>
> >>> Koichiro Den (12):
> >>> PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
> >>> PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
> >>> context
> >>> PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
> >>> ntb_db_event()
> >>> PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
> >>> PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
> >>> PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
> >>> PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
> >>> doorbells
> >>> NTB: epf: Document legacy doorbell slot offset in
> >>> ntb_epf_peer_db_set()
> >>> NTB: epf: Make db_valid_mask cover only real doorbell bits
> >>> NTB: epf: Report 0-based doorbell vector via ntb_db_event()
> >>> NTB: epf: Fix doorbell bitmask and IRQ vector handling
> >>> NTB: epf: Implement db_vector_count/mask for doorbells
> >>>
> >>> drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
> >>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
> >>> 2 files changed, 296 insertions(+), 47 deletions(-)
> >>>
> >>
> >
>
On 5/26/26 8:56 AM, Koichiro Den wrote:
> On Tue, May 26, 2026 at 07:45:11AM -0700, Dave Jiang wrote:
>>
>>
>> On 5/20/26 11:29 PM, Koichiro Den wrote:
>>> On Tue, May 19, 2026 at 02:29:51PM -0700, Dave Jiang wrote:
>>>>
>>>>
>>>> On 5/12/26 7:49 PM, Koichiro Den wrote:
>>>>> This series fixes doorbell bit/vector handling for the EPF-based NTB
>>>>> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
>>>>> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
>>>>> without changing the on-the-wire doorbell mapping.
>>>>>
>>>>>
>>>>> Background / problem
>>>>> ====================
>>>>>
>>>>> ntb_hw_epf historically applies an extra offset when ringing peer
>>>>> doorbells: the link event uses the first interrupt slot, and doorbells
>>>>> start from the third slot (i.e. a second slot is effectively unused).
>>>>> pci-epf-vntb carries the matching offset on the EP side as well.
>>>>>
>>>>> As long as db_vector_count()/db_vector_mask() are not implemented, this
>>>>> mismatch is mostly masked. Doorbell events are effectively treated as
>>>>> "can hit any QP" and the off-by-one vector numbering does not surface
>>>>> clearly.
>>>>>
>>>>> However, once per-vector handling is enabled, the current state becomes
>>>>> problematic:
>>>>>
>>>>> - db_valid_mask exposes bits that do not correspond to real doorbells
>>>>> (link/unused slots leak into the mask).
>>>>> - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
>>>>> expects a 0-based db_vector for doorbells.
>>>>> - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
>>>>> it directly calls pci_epc_raise_irq(), which can sleep.
>>>>>
>>>>>
>>>>> Why NOT fix the root offset?
>>>>> ============================
>>>>>
>>>>> The natural "root" fix would be to remove the historical extra offset in
>>>>> the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
>>>>> Unfortunately this would lead to interoperability issues when mixing old
>>>>> and new kernel versions (old/new peers). A new side would ring a
>>>>> different interrupt slot than what an old peer expects, leading to
>>>>> missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
>>>>> are implemented.
>>>>>
>>>>> Therefore this series intentionally keeps the legacy offset, and instead
>>>>> fixes the surrounding pieces so the mapping is documented and handled
>>>>> consistently in masks, vector numbering, and per-vector reporting.
>>>>>
>>>>>
>>>>> Dependencies
>>>>> ============
>>>>>
>>>>> This v4 is prepared on top of the current pci/endpoint:
>>>>> commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
>>>>> doorbell trigger")
>>>>
>>>> I can't seem to apply the patch set via 'b4 shazam' on top of f5aa97125491. Trying to set it up for review.
>>>
>>> Hi Dave,
>>>
>>>>
>>>> DJ
>>>>
>>>>>
>>>>> plus these patches as contextual prerequisites:
>>>>>
>>>>> - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
>>>>> https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
>>>>>
>>>>> - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
>>>>> https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
>>>>>
>>>>> (These touch the same code part as this series and would otherwise conflict
>>>>> with it. They should land upstream first in my opinion.)
>>>
>>> This series depends on the two prerequisite series listed above.
>>
>> They don't apply cleanly either. Do you have a public repo somewhere that has everything applied? That may be the easiest way to do things if a series has complicated dependencies.
>
> I rechecked this with a fresh worktree based off of f5aa97125491:
>
> git worktree add /tmp/f5aa97125491 f5aa97125491
> cd /tmp/f5aa97125491
>
> # 1st dependency
> b4 am 20260304083028.1391068-1-den@valinux.co.jp
> git am ./20260304_den_ntb_epf_fix_ntb_hw_epf_isr_issues.mbx
>
> # 2nd dependency
> b4 am 20260407124421.282766-1-mani@kernel.org
> git am ./20260407_manivannan_sadhasivam_pci_endpoint_pci_epf_v_ntb_a_couple_of_fixes.mbx
>
> # this v4 series
> b4 am 20260513024923.451765-1-den@valinux.co.jp
> git am ./v4_20260513_den_pci_endpoint_pci_epf_vntb_ntb_epf_enable_per_doorbell_bit_handling.mbx
>
> And this gives me the following commit log:
>
> git log --reverse --oneline f5aa97125491..
> # 1d1465542cc0 NTB: epf: Fix request_irq() unwind in ntb_epf_init_isr()
> # 94bd43a9e435 NTB: epf: Avoid pci_irq_vector() from hardirq context
> # 3cab4db0f647 PCI: endpoint: pci-epf-vntb: Add check to detect 'db_count' value of 0
> # 5449f10512ae PCI: endpoint: pci-epf-ntb: Add check to detect 'db_count' value of 0
> # e3dbdcceb2fd PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
> # e1c167a58888 PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic context
> # 6c54fd0c9c9c PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via ntb_db_event()
> # 28a30b79ff03 PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
> # cb5fbd37bdc6 PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
> # 9ed0f4db85b4 PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
> # 575bd3aa530b PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for doorbells
> # 41f26aa279af NTB: epf: Document legacy doorbell slot offset in ntb_epf_peer_db_set()
> # 6b2e2f80a61a NTB: epf: Make db_valid_mask cover only real doorbell bits
> # 44d1bb070521 NTB: epf: Report 0-based doorbell vector via ntb_db_event()
> # 4e4d37b4c2a8 NTB: epf: Fix doorbell bitmask and IRQ vector handling
> # 1e5829682361 (HEAD) NTB: epf: Implement db_vector_count/mask for doorbells
>
> I don't have a public repo at the moment, sorry. Could you try the above? If it
> still fails, please let me know.
I'm able to apply with the above steps. Thanks!
>
> Best regards,
> Koichiro
>
>>
>> DJ
>>
>>>
>>> Best regards,
>>> Koichiro
>>>
>>>>>
>>>>>
>>>>> Practical benefit
>>>>> =================
>>>>>
>>>>> This also enables ntb_netdev's multi-queue performance scale:
>>>>>
>>>>> [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
>>>>> https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
>>>>>
>>>>> Each netdev queue maps to an ntb_transport queue pair. With correct
>>>>> db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
>>>>> only the queue pairs covered by the doorbell vector it received. Without
>>>>> those callbacks, the NTB core falls back to the full valid doorbell mask,
>>>>> so each doorbell interrupt can still wake all queue pairs. See "Testing /
>>>>> Results" section in the above link.
>>>>>
>>>>>
>>>>> What this series does
>>>>> =====================
>>>>>
>>>>> - pci-epf-vntb:
>>>>>
>>>>> - Document the legacy offset.
>>>>> - Defer MSI doorbell raises to process context to avoid sleeping in
>>>>> atomic context. This becomes relevant once multiple doorbells are
>>>>> raised concurrently at a high rate.
>>>>> - Report doorbell vectors as 0-based to ntb_db_event().
>>>>> - Reject unusable doorbell counts and guard configfs writes after EPC
>>>>> attach with -EOPNOTSUPP.
>>>>> - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
>>>>>
>>>>> - ntb_hw_epf:
>>>>>
>>>>> - Document the legacy offset in ntb_epf_peer_db_set().
>>>>> - Fix db_valid_mask to cover only real doorbell bits.
>>>>> - Report 0-based db_vector to ntb_db_event() (accounting for the
>>>>> unused slot).
>>>>> - Pass per-vector context as request_irq() dev_id instead of deriving the
>>>>> device vector from Linux IRQ numbers.
>>>>> - Keep db_val as a bitmask and fix db_read/db_clear semantics
>>>>> accordingly.
>>>>> - Implement db_vector_count()/db_vector_mask().
>>>>>
>>>>>
>>>>> Compatibility
>>>>> =============
>>>>>
>>>>> By keeping the legacy offset intact, this series aims to remain
>>>>> compatible across mixed kernel versions. The observable changes are
>>>>> limited to correct mask/vector reporting and safer execution context
>>>>> handling.
>>>>>
>>>>> Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
>>>>> applied separately through the respective trees. They are sent together so
>>>>> the two sides of the same legacy layout do not drift apart, and the
>>>>> resulting code stays easier to follow.
>>>>>
>>>>> The plan is still to take the whole series through the PCI EP tree once the
>>>>> remaining NTB acks are collected. See:
>>>>> https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
>>>>>
>>>>>
>>>>> Note
>>>>> ====
>>>>>
>>>>> Most v3->v4 changes are responses to Sashiko feedback:
>>>>> https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
>>>>>
>>>>> Sashiko also flagged a broader driver-unbind lifetime concern around the
>>>>> vNTB PCI driver. That is real, but it predates this series and needs a
>>>>> proper .remove() path, including NTB unregistration and work cancellation.
>>>>> I am keeping it as follow-up lifecycle work:
>>>>>
>>>>> https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
>>>>>
>>>>> This v4 only clears the deferred doorbell state added by this series around
>>>>> EPC init/cleanup.
>>>>>
>>>>>
>>>>> ---
>>>>> Changelog
>>>>> =========
>>>>>
>>>>> Changes since v3:
>>>>> - Rebased on the dependencies listed above.
>>>>> - For Sashiko feedback on state added by this series: clear
>>>>> peer_db_pending around work enable/disable, mask peer_db_set() with
>>>>> db_valid_mask, and use __ffs64().
>>>>> - For pre-existing issues made more relevant by per-vector semantics:
>>>>> stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
>>>>> per-vector request_irq() context, ignore the reserved slot, and
>>>>> validate vectors before BIT_ULL()/ntb_db_event().
>>>>> - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
>>>>> bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
>>>>> cleanups.
>>>>> - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
>>>>>
>>>>> Changes since v2:
>>>>> - No functional changes.
>>>>> - Rebased onto current pci/endpoint
>>>>> e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
>>>>> * Patch 2 needed a trivial context-only adjustment while rebasing, due
>>>>> to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
>>>>> cmd_handler work in epf_ntb_epc_cleanup").
>>>>> - Picked up additional Reviewed-by tags from Frank.
>>>>> - Fixed the incorrect v2 series title.
>>>>>
>>>>> Changes since v1:
>>>>> - Addressed feedback from Dave (add a source code comment, introduce
>>>>> enum to eliminate magic numbers)
>>>>> - Updated source code comment in Patch 2.
>>>>> - No functional changes, so retained Reviewed-by tags by Frank and Dave.
>>>>> Thank you both for the review.
>>>>>
>>>>> v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
>>>>> v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
>>>>> v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Koichiro
>>>>>
>>>>>
>>>>> Koichiro Den (12):
>>>>> PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
>>>>> PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
>>>>> context
>>>>> PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
>>>>> ntb_db_event()
>>>>> PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
>>>>> PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
>>>>> PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
>>>>> PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
>>>>> doorbells
>>>>> NTB: epf: Document legacy doorbell slot offset in
>>>>> ntb_epf_peer_db_set()
>>>>> NTB: epf: Make db_valid_mask cover only real doorbell bits
>>>>> NTB: epf: Report 0-based doorbell vector via ntb_db_event()
>>>>> NTB: epf: Fix doorbell bitmask and IRQ vector handling
>>>>> NTB: epf: Implement db_vector_count/mask for doorbells
>>>>>
>>>>> drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
>>>>> drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
>>>>> 2 files changed, 296 insertions(+), 47 deletions(-)
>>>>>
>>>>
>>>
>>
On Wed, May 13, 2026 at 11:49:11AM +0900, Koichiro Den wrote:
> This series fixes doorbell bit/vector handling for the EPF-based NTB
> pair (ntb_hw_epf <-> pci-epf-*ntb). Its primary goal is to enable safe
> per-db-vector handling in the NTB core and clients (e.g. ntb_transport),
> without changing the on-the-wire doorbell mapping.
>
FYI, I'm waiting for Acks from NTB maintainers for all the NTB patches.
- Mani
>
> Background / problem
> ====================
>
> ntb_hw_epf historically applies an extra offset when ringing peer
> doorbells: the link event uses the first interrupt slot, and doorbells
> start from the third slot (i.e. a second slot is effectively unused).
> pci-epf-vntb carries the matching offset on the EP side as well.
>
> As long as db_vector_count()/db_vector_mask() are not implemented, this
> mismatch is mostly masked. Doorbell events are effectively treated as
> "can hit any QP" and the off-by-one vector numbering does not surface
> clearly.
>
> However, once per-vector handling is enabled, the current state becomes
> problematic:
>
> - db_valid_mask exposes bits that do not correspond to real doorbells
> (link/unused slots leak into the mask).
> - ntb_db_event() is fed with 1-based/shifted vectors, while NTB core
> expects a 0-based db_vector for doorbells.
> - On pci-epf-vntb, .peer_db_set() may be called in atomic context, but
> it directly calls pci_epc_raise_irq(), which can sleep.
>
>
> Why NOT fix the root offset?
> ============================
>
> The natural "root" fix would be to remove the historical extra offset in
> the peer_db_set() doorbell paths for ntb_hw_epf and pci-epf-vntb.
> Unfortunately this would lead to interoperability issues when mixing old
> and new kernel versions (old/new peers). A new side would ring a
> different interrupt slot than what an old peer expects, leading to
> missed or misrouted doorbells, once db_vector_count()/db_vector_mask()
> are implemented.
>
> Therefore this series intentionally keeps the legacy offset, and instead
> fixes the surrounding pieces so the mapping is documented and handled
> consistently in masks, vector numbering, and per-vector reporting.
>
>
> Dependencies
> ============
>
> This v4 is prepared on top of the current pci/endpoint:
> commit f5aa97125491 ("misc: pci_endpoint_test: remove dead BAR read before
> doorbell trigger")
>
> plus these patches as contextual prerequisites:
>
> - [PATCH 0/2] NTB: epf: Fix ntb_hw_epf ISR issues
> https://lore.kernel.org/ntb/20260304083028.1391068-1-den@valinux.co.jp/
>
> - [PATCH 0/2] PCI: endpoint: pci-epf-{v}ntb: A couple of fixes
> https://lore.kernel.org/linux-pci/20260407124421.282766-1-mani@kernel.org/
>
> (These touch the same code part as this series and would otherwise conflict
> with it. They should land upstream first in my opinion.)
>
>
> Practical benefit
> =================
>
> This also enables ntb_netdev's multi-queue performance scale:
>
> [PATCH v3 0/4] net: ntb_netdev: Add Multi-queue support
> https://lore.kernel.org/netdev/20260305155639.1885517-1-den@valinux.co.jp/
>
> Each netdev queue maps to an ntb_transport queue pair. With correct
> db_vector_count() and db_vector_mask() callbacks, ntb_transport can wake
> only the queue pairs covered by the doorbell vector it received. Without
> those callbacks, the NTB core falls back to the full valid doorbell mask,
> so each doorbell interrupt can still wake all queue pairs. See "Testing /
> Results" section in the above link.
>
>
> What this series does
> =====================
>
> - pci-epf-vntb:
>
> - Document the legacy offset.
> - Defer MSI doorbell raises to process context to avoid sleeping in
> atomic context. This becomes relevant once multiple doorbells are
> raised concurrently at a high rate.
> - Report doorbell vectors as 0-based to ntb_db_event().
> - Reject unusable doorbell counts and guard configfs writes after EPC
> attach with -EOPNOTSUPP.
> - Fix db_valid_mask and implement db_vector_count()/db_vector_mask().
>
> - ntb_hw_epf:
>
> - Document the legacy offset in ntb_epf_peer_db_set().
> - Fix db_valid_mask to cover only real doorbell bits.
> - Report 0-based db_vector to ntb_db_event() (accounting for the
> unused slot).
> - Pass per-vector context as request_irq() dev_id instead of deriving the
> device vector from Linux IRQ numbers.
> - Keep db_val as a bitmask and fix db_read/db_clear semantics
> accordingly.
> - Implement db_vector_count()/db_vector_mask().
>
>
> Compatibility
> =============
>
> By keeping the legacy offset intact, this series aims to remain
> compatible across mixed kernel versions. The observable changes are
> limited to correct mask/vector reporting and safer execution context
> handling.
>
> Patches 1-7 (PCI Endpoint) and 8-12 (NTB) are independent and can be
> applied separately through the respective trees. They are sent together so
> the two sides of the same legacy layout do not drift apart, and the
> resulting code stays easier to follow.
>
> The plan is still to take the whole series through the PCI EP tree once the
> remaining NTB acks are collected. See:
> https://lore.kernel.org/linux-pci/rnzsnp5de4qf5w7smebkmqekpuaqckltx73rj6ha3q2nrby5yp@7hsgvdzvjkp6/
>
>
> Note
> ====
>
> Most v3->v4 changes are responses to Sashiko feedback:
> https://sashiko.dev/#/patchset/20260323031544.2598111-1-den%40valinux.co.jp
>
> Sashiko also flagged a broader driver-unbind lifetime concern around the
> vNTB PCI driver. That is real, but it predates this series and needs a
> proper .remove() path, including NTB unregistration and work cancellation.
> I am keeping it as follow-up lifecycle work:
>
> https://lore.kernel.org/linux-pci/20260226084142.2226875-1-den@valinux.co.jp/
>
> This v4 only clears the deferred doorbell state added by this series around
> EPC init/cleanup.
>
>
> ---
> Changelog
> =========
>
> Changes since v3:
> - Rebased on the dependencies listed above.
> - For Sashiko feedback on state added by this series: clear
> peer_db_pending around work enable/disable, mask peer_db_set() with
> db_valid_mask, and use __ffs64().
> - For pre-existing issues made more relevant by per-vector semantics:
> stop deriving ntb_hw_epf vectors from Linux IRQ numbers, pass
> per-vector request_irq() context, ignore the reserved slot, and
> validate vectors before BIT_ULL()/ntb_db_event().
> - Added adjacent hardening: pci-epf-vntb db_count range/configfs guards,
> bounded db_vector masks, ntb_hw_epf mw_count precheck, and helper guard
> cleanups.
> - Dropped Reviewed-by tags from patches that changed to non-trivial extent.
>
> Changes since v2:
> - No functional changes.
> - Rebased onto current pci/endpoint
> e022f0c72c7f ("selftests: pci_endpoint: Skip reserved BARs").
> * Patch 2 needed a trivial context-only adjustment while rebasing, due
> to commit d799984233a5 ("PCI: endpoint: pci-epf-vntb: Stop
> cmd_handler work in epf_ntb_epc_cleanup").
> - Picked up additional Reviewed-by tags from Frank.
> - Fixed the incorrect v2 series title.
>
> Changes since v1:
> - Addressed feedback from Dave (add a source code comment, introduce
> enum to eliminate magic numbers)
> - Updated source code comment in Patch 2.
> - No functional changes, so retained Reviewed-by tags by Frank and Dave.
> Thank you both for the review.
>
> v3: https://lore.kernel.org/linux-pci/20260323031544.2598111-1-den@valinux.co.jp/
> v2: https://lore.kernel.org/linux-pci/20260227084955.3184017-1-den@valinux.co.jp/
> v1: https://lore.kernel.org/linux-pci/20260224133459.1741537-1-den@valinux.co.jp/
>
>
> Best regards,
> Koichiro
>
>
> Koichiro Den (12):
> PCI: endpoint: pci-epf-vntb: Document legacy MSI doorbell offset
> PCI: endpoint: pci-epf-vntb: Defer pci_epc_raise_irq() out of atomic
> context
> PCI: endpoint: pci-epf-vntb: Report 0-based doorbell vector via
> ntb_db_event()
> PCI: endpoint: pci-epf-vntb: Reject unusable doorbell counts
> PCI: endpoint: pci-epf-vntb: Guard configfs writes after EPC attach
> PCI: endpoint: pci-epf-vntb: Exclude reserved slots from db_valid_mask
> PCI: endpoint: pci-epf-vntb: Implement db_vector_count/mask for
> doorbells
> NTB: epf: Document legacy doorbell slot offset in
> ntb_epf_peer_db_set()
> NTB: epf: Make db_valid_mask cover only real doorbell bits
> NTB: epf: Report 0-based doorbell vector via ntb_db_event()
> NTB: epf: Fix doorbell bitmask and IRQ vector handling
> NTB: epf: Implement db_vector_count/mask for doorbells
>
> drivers/ntb/hw/epf/ntb_hw_epf.c | 133 ++++++++---
> drivers/pci/endpoint/functions/pci-epf-vntb.c | 210 ++++++++++++++++--
> 2 files changed, 296 insertions(+), 47 deletions(-)
>
> --
> 2.51.0
>
--
மணிவண்ணன் சதாசிவம்
© 2016 - 2026 Red Hat, Inc.