Documentation/PCI/endpoint/pci-vntb-howto.rst | 16 +- drivers/ntb/hw/epf/ntb_hw_epf.c | 143 +++++- drivers/ntb/msi.c | 19 +- drivers/ntb/ntb_transport.c | 127 +++++- drivers/ntb/test/ntb_perf.c | 16 +- drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++-- include/linux/ntb.h | 133 +++++- 7 files changed, 801 insertions(+), 76 deletions(-)
Hi,
vNTB currently spends one BAR per MW. BAR count therefore limits the
number of MWs, even when one BAR has room for several. A large BAR,
especially a Resizable BAR, can instead be split into logical MWs and
leave more BARs for config and doorbells.
This series adds packed_mws, which exposes up to 16 equal-sized logical
MWs in one BAR. ntb_transport uses the new group operations to configure
them atomically.
The immediate use case is increasing the number of ntb_transport queues.
The resulting performance gain is shown in the Testing section.
Based on the latest pci/endpoint.
Why not BAR subrange mappings?
==============================
An earlier proposal used mwN_offset and BAR subrange mappings to place
independently backed MWs in one BAR:
https://lore.kernel.org/r/20260312165005.1148676-11-den@valinux.co.jp/
BAR subrange mapping is currently implemented only by DWC, and that
design still needs one hardware mapping per MW in each direction. This
series instead requires one contiguous backing range and maps the whole
group once in each direction. The initial implementation intentionally
trades arbitrary MW placement for a constant mapping count as the MW
count grows. More flexible layouts can be added when a concrete use
case requires them.
Testing
=======
The packed data path was tested with ntb_transport and ntb_netdev. An
`iperf3 -ub0 -l 65482 -P 2` UDP saturation run produced the following
results when only one Resizable BAR was available for MWs:
Configuration Receiver throughput
---------------------------------------------- -------------------
num_mws=1, packed_mws unset (default 0) 607 Mbits/sec
num_mws=2, packed_mws=2, combined channels=2 1.19 Gbits/sec
Meanwhile, I realized that the in-tree ntb_test.sh does not fully pass
with the pci-epf-vntb/ntb_hw_epf setup on the base tree. I had to apply
a separate collection of local PCI endpoint and NTB fixes before this
setup could be used for regression testing. These fixes address
pre-existing issues and are not prerequisites of the packed MW
implementation.
With packed_mws unset and ntb_msi_tests disabled, ntb_test.sh reported
the same pass/unsupported results before and after this series. One run
follows:
$ sudo ./ntb_test.sh -r ${EP} -m 1048576 -p 28 0000:01:00.0 0001:10:00.0
Starting ntb_tool tests...
Running port tests on: 0000:01:00.0 / 0001:10:00.0
Local port 0 with index 0 on remote host
Peer port 0 with index 0 on local host
Passed
Running link tests on: 0000:01:00.0 / 0001:10:00.0
Passed
Running link tests on: 0001:10:00.0 / 0000:01:00.0
Passed
Running db tests on: 0000:01:00.0 / 0001:10:00.0
Passed
Running db tests on: 0001:10:00.0 / 0000:01:00.0
Passed
Running spad tests on: 0000:01:00.0 / 0001:10:00.0
Passed
Running spad tests on: 0001:10:00.0 / 0000:01:00.0
Passed
Running msg tests on: 0000:01:00.0 / 0001:10:00.0
Unsupported
Running msg tests on: 0001:10:00.0 / 0000:01:00.0
Unsupported
Running mw0 tests on: 0000:01:00.0 / 0001:10:00.0
Passed
Running mw0 tests on: 0001:10:00.0 / 0000:01:00.0
Passed
Starting ntb_pingpong tests...
Running ping pong tests on: 0000:01:00.0 / 0001:10:00.0
Passed
Starting ntb_perf tests...
Running local perf test without DMA
Peer 0 test statistics:
0: copied 268435456 bytes in 137063 usecs, 1958 MBytes/s
Passed
Running remote perf test without DMA
Peer 0 test statistics:
0: copied 268435456 bytes in 139188 usecs, 1928 MBytes/s
Passed
For comparison, the before-series run reported 1997 MBytes/s locally
and 1950 MBytes/s remotely. No obvious regression was seen in this
single-run comparison.
Control layout compatibility
============================
The endpoint selects the control layout. Non-packed pci-epf-vntb keeps
the version 0 layout. packed_mws selects version 1. "Old" and "new"
below mean ntb_hw_epf without and with this series.
EP configuration Old ntb_hw_epf New ntb_hw_epf
---------------- ------------------------ ------------------------
non-packed (v0) Supported Supported
packed (v1) Not supported [1] Supported
[1] The old driver accepts two or four MWs but looks for a separate BAR
for each MW. It rejects larger MW counts at probe.
Packed MW compatibility with in-tree NTB clients
================================================
The in-tree clients handle a packed MW group as follows:
Client Behavior
-------------- ------------------------------------------------------
ntb_transport Uses group operations. Without a singleton MW, MSI
mode falls back to doorbells.
ntb_perf Rejects a non-singleton MW group at probe.
ntb_tool Loads normally. Per-MW inbound translation setup
returns -EOPNOTSUPP for a non-singleton group.
Doorbell and scratchpad access is unaffected.
ntb_msi_test Fails probe unless the MW reserved for MSI is a
singleton.
ntb_pingpong Does not use MWs and is unaffected.
Further client support can be added separately when needed. A final
singleton MW on another BAR can also be added later if MSI mode is needed
with packed MWs.
Notes
=====
* About pci-epf-ntb:
pci-epf-ntb remains on the legacy version 0 layout; this series does
not add version 1 layout support to it. A pci-epf-ntb bridge should
therefore keep its existing behavior even when its nodes run a mix of
old and new kernels. I do not have hardware for the pci-epf-ntb bridge
topology.
* About existing inbound MW clear behavior:
pci-epf-vntb already leaves the inbound BAR mapping in place when an
MW translation is cleared. This series keeps that behavior for packed
MWs. This limitation is independent of packing.
pci_epc_clear_bar() is not suitable here because it can clear the BAR
address assigned by the host. An EPF can instead restore a persistent
fallback mapping with pci_epc_set_bar(), as pci-epf-test does:
https://lore.kernel.org/r/20250908161942.534799-2-cassel@kernel.org/
pci-epf-vntb has no such fallback mapping today. Adding one, or an EPC
operation to unmap only the inbound translation, should be separate
work.
Best regards,
Koichiro
Koichiro Den (11):
NTB: Add atomic MW translation group operations
NTB: epf: Parse a versioned packed MW layout
PCI: endpoint: pci-epf-vntb: Add packed MW layout handling
PCI: endpoint: pci-epf-vntb: Implement MW group translation callbacks
PCI: endpoint: pci-epf-vntb: Allocate packed outbound MW space
PCI: endpoint: pci-epf-vntb: Add outbound MW group commands
NTB: epf: Implement MW group translation callbacks
NTB: perf: Reject grouped memory windows
NTB/msi: Require a singleton memory window
NTB: ntb_transport: Use atomic MW translation groups
PCI: endpoint: pci-epf-vntb: Expose packed MWs through configfs
Documentation/PCI/endpoint/pci-vntb-howto.rst | 16 +-
drivers/ntb/hw/epf/ntb_hw_epf.c | 143 +++++-
drivers/ntb/msi.c | 19 +-
drivers/ntb/ntb_transport.c | 127 +++++-
drivers/ntb/test/ntb_perf.c | 16 +-
drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++--
include/linux/ntb.h | 133 +++++-
7 files changed, 801 insertions(+), 76 deletions(-)
--
2.51.0
On 8/3/26 11:04 AM, Koichiro Den wrote:
> Hi,
>
> vNTB currently spends one BAR per MW. BAR count therefore limits the
> number of MWs, even when one BAR has room for several. A large BAR,
> especially a Resizable BAR, can instead be split into logical MWs and
> leave more BARs for config and doorbells.
>
> This series adds packed_mws, which exposes up to 16 equal-sized logical
> MWs in one BAR. ntb_transport uses the new group operations to configure
> them atomically.
>
> The immediate use case is increasing the number of ntb_transport queues.
> The resulting performance gain is shown in the Testing section.
>
> Based on the latest pci/endpoint.
>
>
> Why not BAR subrange mappings?
> ==============================
>
> An earlier proposal used mwN_offset and BAR subrange mappings to place
> independently backed MWs in one BAR:
>
> https://lore.kernel.org/r/20260312165005.1148676-11-den@valinux.co.jp/
>
> BAR subrange mapping is currently implemented only by DWC, and that
> design still needs one hardware mapping per MW in each direction. This
> series instead requires one contiguous backing range and maps the whole
> group once in each direction. The initial implementation intentionally
> trades arbitrary MW placement for a constant mapping count as the MW
> count grows. More flexible layouts can be added when a concrete use
> case requires them.
>
Hi,
To be clear, this is totally outside of the scope of any PCI or NTB
standards or specs? or did I overlook that part of the discussion?
Thanks.
>
> Testing
> =======
>
> The packed data path was tested with ntb_transport and ntb_netdev. An
> `iperf3 -ub0 -l 65482 -P 2` UDP saturation run produced the following
> results when only one Resizable BAR was available for MWs:
>
> Configuration Receiver throughput
> ---------------------------------------------- -------------------
> num_mws=1, packed_mws unset (default 0) 607 Mbits/sec
> num_mws=2, packed_mws=2, combined channels=2 1.19 Gbits/sec
>
> Meanwhile, I realized that the in-tree ntb_test.sh does not fully pass
> with the pci-epf-vntb/ntb_hw_epf setup on the base tree. I had to apply
> a separate collection of local PCI endpoint and NTB fixes before this
> setup could be used for regression testing. These fixes address
> pre-existing issues and are not prerequisites of the packed MW
> implementation.
>
> With packed_mws unset and ntb_msi_tests disabled, ntb_test.sh reported
> the same pass/unsupported results before and after this series. One run
> follows:
>
> $ sudo ./ntb_test.sh -r ${EP} -m 1048576 -p 28 0000:01:00.0 0001:10:00.0
> Starting ntb_tool tests...
> Running port tests on: 0000:01:00.0 / 0001:10:00.0
> Local port 0 with index 0 on remote host
> Peer port 0 with index 0 on local host
> Passed
> Running link tests on: 0000:01:00.0 / 0001:10:00.0
> Passed
> Running link tests on: 0001:10:00.0 / 0000:01:00.0
> Passed
> Running db tests on: 0000:01:00.0 / 0001:10:00.0
> Passed
> Running db tests on: 0001:10:00.0 / 0000:01:00.0
> Passed
> Running spad tests on: 0000:01:00.0 / 0001:10:00.0
> Passed
> Running spad tests on: 0001:10:00.0 / 0000:01:00.0
> Passed
> Running msg tests on: 0000:01:00.0 / 0001:10:00.0
> Unsupported
> Running msg tests on: 0001:10:00.0 / 0000:01:00.0
> Unsupported
> Running mw0 tests on: 0000:01:00.0 / 0001:10:00.0
> Passed
> Running mw0 tests on: 0001:10:00.0 / 0000:01:00.0
> Passed
>
> Starting ntb_pingpong tests...
> Running ping pong tests on: 0000:01:00.0 / 0001:10:00.0
> Passed
>
> Starting ntb_perf tests...
> Running local perf test without DMA
> Peer 0 test statistics:
> 0: copied 268435456 bytes in 137063 usecs, 1958 MBytes/s
> Passed
> Running remote perf test without DMA
> Peer 0 test statistics:
> 0: copied 268435456 bytes in 139188 usecs, 1928 MBytes/s
> Passed
>
> For comparison, the before-series run reported 1997 MBytes/s locally
> and 1950 MBytes/s remotely. No obvious regression was seen in this
> single-run comparison.
>
>
> Control layout compatibility
> ============================
>
> The endpoint selects the control layout. Non-packed pci-epf-vntb keeps
> the version 0 layout. packed_mws selects version 1. "Old" and "new"
> below mean ntb_hw_epf without and with this series.
>
> EP configuration Old ntb_hw_epf New ntb_hw_epf
> ---------------- ------------------------ ------------------------
> non-packed (v0) Supported Supported
> packed (v1) Not supported [1] Supported
>
> [1] The old driver accepts two or four MWs but looks for a separate BAR
> for each MW. It rejects larger MW counts at probe.
>
>
> Packed MW compatibility with in-tree NTB clients
> ================================================
>
> The in-tree clients handle a packed MW group as follows:
>
> Client Behavior
> -------------- ------------------------------------------------------
> ntb_transport Uses group operations. Without a singleton MW, MSI
> mode falls back to doorbells.
> ntb_perf Rejects a non-singleton MW group at probe.
> ntb_tool Loads normally. Per-MW inbound translation setup
> returns -EOPNOTSUPP for a non-singleton group.
> Doorbell and scratchpad access is unaffected.
> ntb_msi_test Fails probe unless the MW reserved for MSI is a
> singleton.
> ntb_pingpong Does not use MWs and is unaffected.
>
> Further client support can be added separately when needed. A final
> singleton MW on another BAR can also be added later if MSI mode is needed
> with packed MWs.
>
>
> Notes
> =====
>
> * About pci-epf-ntb:
>
> pci-epf-ntb remains on the legacy version 0 layout; this series does
> not add version 1 layout support to it. A pci-epf-ntb bridge should
> therefore keep its existing behavior even when its nodes run a mix of
> old and new kernels. I do not have hardware for the pci-epf-ntb bridge
> topology.
>
> * About existing inbound MW clear behavior:
>
> pci-epf-vntb already leaves the inbound BAR mapping in place when an
> MW translation is cleared. This series keeps that behavior for packed
> MWs. This limitation is independent of packing.
>
> pci_epc_clear_bar() is not suitable here because it can clear the BAR
> address assigned by the host. An EPF can instead restore a persistent
> fallback mapping with pci_epc_set_bar(), as pci-epf-test does:
>
> https://lore.kernel.org/r/20250908161942.534799-2-cassel@kernel.org/
>
> pci-epf-vntb has no such fallback mapping today. Adding one, or an EPC
> operation to unmap only the inbound translation, should be separate
> work.
>
> Best regards,
> Koichiro
>
>
> Koichiro Den (11):
> NTB: Add atomic MW translation group operations
> NTB: epf: Parse a versioned packed MW layout
> PCI: endpoint: pci-epf-vntb: Add packed MW layout handling
> PCI: endpoint: pci-epf-vntb: Implement MW group translation callbacks
> PCI: endpoint: pci-epf-vntb: Allocate packed outbound MW space
> PCI: endpoint: pci-epf-vntb: Add outbound MW group commands
> NTB: epf: Implement MW group translation callbacks
> NTB: perf: Reject grouped memory windows
> NTB/msi: Require a singleton memory window
> NTB: ntb_transport: Use atomic MW translation groups
> PCI: endpoint: pci-epf-vntb: Expose packed MWs through configfs
>
> Documentation/PCI/endpoint/pci-vntb-howto.rst | 16 +-
> drivers/ntb/hw/epf/ntb_hw_epf.c | 143 +++++-
> drivers/ntb/msi.c | 19 +-
> drivers/ntb/ntb_transport.c | 127 +++++-
> drivers/ntb/test/ntb_perf.c | 16 +-
> drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++--
> include/linux/ntb.h | 133 +++++-
> 7 files changed, 801 insertions(+), 76 deletions(-)
>
--
~Randy
On Mon, Aug 03, 2026 at 04:00:45PM -0700, Randy Dunlap wrote:
>
>
> On 8/3/26 11:04 AM, Koichiro Den wrote:
> > Hi,
> >
> > vNTB currently spends one BAR per MW. BAR count therefore limits the
> > number of MWs, even when one BAR has room for several. A large BAR,
> > especially a Resizable BAR, can instead be split into logical MWs and
> > leave more BARs for config and doorbells.
> >
> > This series adds packed_mws, which exposes up to 16 equal-sized logical
> > MWs in one BAR. ntb_transport uses the new group operations to configure
> > them atomically.
> >
> > The immediate use case is increasing the number of ntb_transport queues.
> > The resulting performance gain is shown in the Testing section.
> >
> > Based on the latest pci/endpoint.
> >
> >
> > Why not BAR subrange mappings?
> > ==============================
> >
> > An earlier proposal used mwN_offset and BAR subrange mappings to place
> > independently backed MWs in one BAR:
> >
> > https://lore.kernel.org/r/20260312165005.1148676-11-den@valinux.co.jp/
> >
> > BAR subrange mapping is currently implemented only by DWC, and that
> > design still needs one hardware mapping per MW in each direction. This
> > series instead requires one contiguous backing range and maps the whole
> > group once in each direction. The initial implementation intentionally
> > trades arbitrary MW placement for a constant mapping count as the MW
> > count grows. More flexible layouts can be added when a concrete use
> > case requires them.
> >
>
> Hi,
> To be clear, this is totally outside of the scope of any PCI or NTB
> standards or specs? or did I overlook that part of the discussion?
> Thanks.
Hi Randy,
No, you didn't overlook anything. This isn't part of any PCI or NTB spec.
This series just proposes a new 'packed_mws' configfs knob for the
vNTB/ntb_hw_epf pair. PCI still sees a normal BAR.
Thanks,
Koichiro
>
> >
> > Testing
> > =======
> >
> > The packed data path was tested with ntb_transport and ntb_netdev. An
> > `iperf3 -ub0 -l 65482 -P 2` UDP saturation run produced the following
> > results when only one Resizable BAR was available for MWs:
> >
> > Configuration Receiver throughput
> > ---------------------------------------------- -------------------
> > num_mws=1, packed_mws unset (default 0) 607 Mbits/sec
> > num_mws=2, packed_mws=2, combined channels=2 1.19 Gbits/sec
> >
> > Meanwhile, I realized that the in-tree ntb_test.sh does not fully pass
> > with the pci-epf-vntb/ntb_hw_epf setup on the base tree. I had to apply
> > a separate collection of local PCI endpoint and NTB fixes before this
> > setup could be used for regression testing. These fixes address
> > pre-existing issues and are not prerequisites of the packed MW
> > implementation.
> >
> > With packed_mws unset and ntb_msi_tests disabled, ntb_test.sh reported
> > the same pass/unsupported results before and after this series. One run
> > follows:
> >
> > $ sudo ./ntb_test.sh -r ${EP} -m 1048576 -p 28 0000:01:00.0 0001:10:00.0
> > Starting ntb_tool tests...
> > Running port tests on: 0000:01:00.0 / 0001:10:00.0
> > Local port 0 with index 0 on remote host
> > Peer port 0 with index 0 on local host
> > Passed
> > Running link tests on: 0000:01:00.0 / 0001:10:00.0
> > Passed
> > Running link tests on: 0001:10:00.0 / 0000:01:00.0
> > Passed
> > Running db tests on: 0000:01:00.0 / 0001:10:00.0
> > Passed
> > Running db tests on: 0001:10:00.0 / 0000:01:00.0
> > Passed
> > Running spad tests on: 0000:01:00.0 / 0001:10:00.0
> > Passed
> > Running spad tests on: 0001:10:00.0 / 0000:01:00.0
> > Passed
> > Running msg tests on: 0000:01:00.0 / 0001:10:00.0
> > Unsupported
> > Running msg tests on: 0001:10:00.0 / 0000:01:00.0
> > Unsupported
> > Running mw0 tests on: 0000:01:00.0 / 0001:10:00.0
> > Passed
> > Running mw0 tests on: 0001:10:00.0 / 0000:01:00.0
> > Passed
> >
> > Starting ntb_pingpong tests...
> > Running ping pong tests on: 0000:01:00.0 / 0001:10:00.0
> > Passed
> >
> > Starting ntb_perf tests...
> > Running local perf test without DMA
> > Peer 0 test statistics:
> > 0: copied 268435456 bytes in 137063 usecs, 1958 MBytes/s
> > Passed
> > Running remote perf test without DMA
> > Peer 0 test statistics:
> > 0: copied 268435456 bytes in 139188 usecs, 1928 MBytes/s
> > Passed
> >
> > For comparison, the before-series run reported 1997 MBytes/s locally
> > and 1950 MBytes/s remotely. No obvious regression was seen in this
> > single-run comparison.
> >
> >
> > Control layout compatibility
> > ============================
> >
> > The endpoint selects the control layout. Non-packed pci-epf-vntb keeps
> > the version 0 layout. packed_mws selects version 1. "Old" and "new"
> > below mean ntb_hw_epf without and with this series.
> >
> > EP configuration Old ntb_hw_epf New ntb_hw_epf
> > ---------------- ------------------------ ------------------------
> > non-packed (v0) Supported Supported
> > packed (v1) Not supported [1] Supported
> >
> > [1] The old driver accepts two or four MWs but looks for a separate BAR
> > for each MW. It rejects larger MW counts at probe.
> >
> >
> > Packed MW compatibility with in-tree NTB clients
> > ================================================
> >
> > The in-tree clients handle a packed MW group as follows:
> >
> > Client Behavior
> > -------------- ------------------------------------------------------
> > ntb_transport Uses group operations. Without a singleton MW, MSI
> > mode falls back to doorbells.
> > ntb_perf Rejects a non-singleton MW group at probe.
> > ntb_tool Loads normally. Per-MW inbound translation setup
> > returns -EOPNOTSUPP for a non-singleton group.
> > Doorbell and scratchpad access is unaffected.
> > ntb_msi_test Fails probe unless the MW reserved for MSI is a
> > singleton.
> > ntb_pingpong Does not use MWs and is unaffected.
> >
> > Further client support can be added separately when needed. A final
> > singleton MW on another BAR can also be added later if MSI mode is needed
> > with packed MWs.
> >
> >
> > Notes
> > =====
> >
> > * About pci-epf-ntb:
> >
> > pci-epf-ntb remains on the legacy version 0 layout; this series does
> > not add version 1 layout support to it. A pci-epf-ntb bridge should
> > therefore keep its existing behavior even when its nodes run a mix of
> > old and new kernels. I do not have hardware for the pci-epf-ntb bridge
> > topology.
> >
> > * About existing inbound MW clear behavior:
> >
> > pci-epf-vntb already leaves the inbound BAR mapping in place when an
> > MW translation is cleared. This series keeps that behavior for packed
> > MWs. This limitation is independent of packing.
> >
> > pci_epc_clear_bar() is not suitable here because it can clear the BAR
> > address assigned by the host. An EPF can instead restore a persistent
> > fallback mapping with pci_epc_set_bar(), as pci-epf-test does:
> >
> > https://lore.kernel.org/r/20250908161942.534799-2-cassel@kernel.org/
> >
> > pci-epf-vntb has no such fallback mapping today. Adding one, or an EPC
> > operation to unmap only the inbound translation, should be separate
> > work.
> >
> > Best regards,
> > Koichiro
> >
> >
> > Koichiro Den (11):
> > NTB: Add atomic MW translation group operations
> > NTB: epf: Parse a versioned packed MW layout
> > PCI: endpoint: pci-epf-vntb: Add packed MW layout handling
> > PCI: endpoint: pci-epf-vntb: Implement MW group translation callbacks
> > PCI: endpoint: pci-epf-vntb: Allocate packed outbound MW space
> > PCI: endpoint: pci-epf-vntb: Add outbound MW group commands
> > NTB: epf: Implement MW group translation callbacks
> > NTB: perf: Reject grouped memory windows
> > NTB/msi: Require a singleton memory window
> > NTB: ntb_transport: Use atomic MW translation groups
> > PCI: endpoint: pci-epf-vntb: Expose packed MWs through configfs
> >
> > Documentation/PCI/endpoint/pci-vntb-howto.rst | 16 +-
> > drivers/ntb/hw/epf/ntb_hw_epf.c | 143 +++++-
> > drivers/ntb/msi.c | 19 +-
> > drivers/ntb/ntb_transport.c | 127 +++++-
> > drivers/ntb/test/ntb_perf.c | 16 +-
> > drivers/pci/endpoint/functions/pci-epf-vntb.c | 423 ++++++++++++++++--
> > include/linux/ntb.h | 133 +++++-
> > 7 files changed, 801 insertions(+), 76 deletions(-)
> >
>
> --
> ~Randy
>
© 2016 - 2026 Red Hat, Inc.