Hi,

This is v2, part 3 of three series for PCI endpoint DMA.

The three series are:

  * part 1: dmaengine: dw-edma: Prepare for PCI EP DMA
  * part 2: PCI: endpoint: Expose endpoint DMA resources
  * part 3: PCI: endpoint: Add PCI DMA endpoint function

This series adds the host-side metadata parser, the pci-epf-dma endpoint
function driver, and documentation.

The endpoint function exposes selected endpoint-integrated DMA channels as
a separate PCI DMA controller function. The host-side dw-edma-pcie driver
discovers the BAR metadata, requests the final layout, and registers the
exposed channels with DMAengine. Host clients then submit transfers through
the regular DMAengine API. The endpoint function keeps the metadata BAR
stable and uses a separate DMA window BAR for resources that need dynamic
subrange mappings.

No fixed PCI ID is assigned by this series. Users provide the PCI
vendor/device ID through configfs and bind dw-edma-pcie explicitly, for
example with driver_override.


Dependencies
============

This series depends on parts 1 and 2, applied on top of pci/endpoint:

  [PATCH v2 00/12] dmaengine: dw-edma: Prepare for PCI EP DMA (part 1/3)
  https://lore.kernel.org/dmaengine/20260525062420.3315904-1-den@valinux.co.jp/

  [PATCH v2 0/3] PCI: endpoint: Expose endpoint DMA resources (part 2/3)
  https://lore.kernel.org/linux-pci/20260525063129.3316894-1-den@valinux.co.jp/


Note
====

This series touches both dmaengine and PCI endpoint code. I kept the
dw-edma-pcie metadata parser together with the endpoint function so the
metadata producer and consumer can be reviewed in one place.

If the general direction looks acceptable, the dw-edma-pcie patch may need
a dmaengine Ack if this series is routed through the PCI endpoint tree.


Tested on
=========

The RC-to-EP data path was tested with a small out-of-tree DMAengine
client. The host submits a DMA_MEM_TO_DEV transfer through dw-edma-pcie,
which uses a DesignWare eDMA read channel to copy host memory into
endpoint memory.

Tested with:

  * R-Car S4 as endpoint and R-Car S4 as root complex
  * RK3588 as endpoint and CD8180 as root complex


---
Changelog
=========

Changes in v2:
  - Follow the part 1/3 and part 2/3 v2 channel-claim model: pci-epf-dma
    now claims delegated channels through DMAengine filter information from
    EPC auxiliary resources.
  - Select raw-address dw-edma-pcie platform ops from the endpoint DMA
    match entry instead of using a match flag.

v1: https://lore.kernel.org/linux-pci/20260521063638.2843021-1-den@valinux.co.jp/


Best regards,
Koichiro


Koichiro Den (3):
  dmaengine: dw-edma-pcie: Discover endpoint DMA metadata
  PCI: endpoint: Add DMA endpoint function
  Documentation: PCI: Add PCI DMA endpoint function documentation

 Documentation/PCI/endpoint/index.rst          |    2 +
 .../PCI/endpoint/pci-dma-function.rst         |  182 +++
 Documentation/PCI/endpoint/pci-dma-howto.rst  |  200 +++
 drivers/dma/dw-edma/dw-edma-pcie.c            |  374 ++++-
 drivers/pci/endpoint/functions/Kconfig        |   14 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-dma.c  | 1366 +++++++++++++++++
 7 files changed, 2138 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/PCI/endpoint/pci-dma-function.rst
 create mode 100644 Documentation/PCI/endpoint/pci-dma-howto.rst
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-dma.c

-- 
2.51.0

On Mon, May 25, 2026 at 03:34:53PM +0900, Koichiro Den wrote:
> Hi,
> 
> This is v2, part 3 of three series for PCI endpoint DMA.
> 
> The three series are:
> 
>   * part 1: dmaengine: dw-edma: Prepare for PCI EP DMA
>   * part 2: PCI: endpoint: Expose endpoint DMA resources
>   * part 3: PCI: endpoint: Add PCI DMA endpoint function
> 
> This series adds the host-side metadata parser, the pci-epf-dma endpoint
> function driver, and documentation.
> 
> The endpoint function exposes selected endpoint-integrated DMA channels as
> a separate PCI DMA controller function. The host-side dw-edma-pcie driver
> discovers the BAR metadata, requests the final layout, and registers the
> exposed channels with DMAengine. Host clients then submit transfers through
> the regular DMAengine API. The endpoint function keeps the metadata BAR
> stable and uses a separate DMA window BAR for resources that need dynamic
> subrange mappings.
> 
> No fixed PCI ID is assigned by this series. Users provide the PCI
> vendor/device ID through configfs and bind dw-edma-pcie explicitly, for
> example with driver_override.
> 
> 
> Dependencies
> ============
> 
> This series depends on parts 1 and 2, applied on top of pci/endpoint:
> 
>   [PATCH v2 00/12] dmaengine: dw-edma: Prepare for PCI EP DMA (part 1/3)
>   https://lore.kernel.org/dmaengine/20260525062420.3315904-1-den@valinux.co.jp/
> 
>   [PATCH v2 0/3] PCI: endpoint: Expose endpoint DMA resources (part 2/3)
>   https://lore.kernel.org/linux-pci/20260525063129.3316894-1-den@valinux.co.jp/
> 
> 
> Note
> ====
> 
> This series touches both dmaengine and PCI endpoint code. I kept the
> dw-edma-pcie metadata parser together with the endpoint function so the
> metadata producer and consumer can be reviewed in one place.
> 
> If the general direction looks acceptable, the dw-edma-pcie patch may need
> a dmaengine Ack if this series is routed through the PCI endpoint tree.
> 
> 
> Tested on
> =========
> 
> The RC-to-EP data path was tested with a small out-of-tree DMAengine
> client. The host submits a DMA_MEM_TO_DEV transfer through dw-edma-pcie,
> which uses a DesignWare eDMA read channel to copy host memory into
> endpoint memory.
> 
> Tested with:
> 
>   * R-Car S4 as endpoint and R-Car S4 as root complex
>   * RK3588 as endpoint and CD8180 as root complex
> 
> 
> ---
> Changelog
> =========
> 
> Changes in v2:
>   - Follow the part 1/3 and part 2/3 v2 channel-claim model: pci-epf-dma
>     now claims delegated channels through DMAengine filter information from
>     EPC auxiliary resources.
>   - Select raw-address dw-edma-pcie platform ops from the endpoint DMA
>     match entry instead of using a match flag.
> 
> v1: https://lore.kernel.org/linux-pci/20260521063638.2843021-1-den@valinux.co.jp/
> 
> 
> Best regards,
> Koichiro

Hi Mani,

I would like to ask you for your high-level opinion on the direction of this
series.

Previously, I have tried two different approaches for the same objective:
avoiding the extra CPU memcpy (or local DMA memcpy) in NTB transport on both EP
and RC sides.

1. Put dw-edma-specific handling under drivers/ntb/hw and let the (new) NTB
   driver carry the metadata needed for channel delegation.

   [RFC PATCH v4 00/38] NTB transport backed by PCI EP embedded DMA
   https://lore.kernel.org/all/20260118135440.1958279-1-den@valinux.co.jp/

2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
   would create an auxiliary device, and a new dw-edma-aux driver would create
   the delegated DMA channels on the RC side.

   [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
   https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/

   I added an ASCII diagram for the overview as a follow-up comment here:
   https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/)

Now, this v2 series takes a third direction. It moves the DMA controller out of
vNTB/NTB-specific ABI and exposes it as a separate PCI endpoint DMA function.
The host then discovers it as a DMA controller function. The initial host-side
driver is the existing dw-edma-pcie driver, and dw-edma-aux is no longer needed.

My current thinking is that this is the cleanest among the previous attempts.
But this is mostly an architecture question, so I would like to know whether
this direction looks acceptable to you.

In short, do you agree with the direction of this series, that endpoint DMA
channel delegation should be modeled as a separate PCI endpoint DMA function?

If you think the vNTB-integrated direction is preferable, or if this should be
modeled differently in the endpoint framework, I would rather adjust the
direction as early as possible, before building the NTB transport on top of it.

Best regards,
Koichiro

> 
> 
> Koichiro Den (3):
>   dmaengine: dw-edma-pcie: Discover endpoint DMA metadata
>   PCI: endpoint: Add DMA endpoint function
>   Documentation: PCI: Add PCI DMA endpoint function documentation
> 
>  Documentation/PCI/endpoint/index.rst          |    2 +
>  .../PCI/endpoint/pci-dma-function.rst         |  182 +++
>  Documentation/PCI/endpoint/pci-dma-howto.rst  |  200 +++
>  drivers/dma/dw-edma/dw-edma-pcie.c            |  374 ++++-
>  drivers/pci/endpoint/functions/Kconfig        |   14 +
>  drivers/pci/endpoint/functions/Makefile       |    1 +
>  drivers/pci/endpoint/functions/pci-epf-dma.c  | 1366 +++++++++++++++++
>  7 files changed, 2138 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/PCI/endpoint/pci-dma-function.rst
>  create mode 100644 Documentation/PCI/endpoint/pci-dma-howto.rst
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-dma.c
> 
> -- 
> 2.51.0
>

On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
> 
> I would like to ask you for your high-level opinion on the direction of this
> series.
> 
> Previously, I have tried two different approaches for the same objective:
> avoiding the extra CPU memcpy (or local DMA memcpy) in NTB transport on both EP
> and RC sides.
> 
> 1. Put dw-edma-specific handling under drivers/ntb/hw and let the (new) NTB
>    driver carry the metadata needed for channel delegation.
> 
>    [RFC PATCH v4 00/38] NTB transport backed by PCI EP embedded DMA
>    https://lore.kernel.org/all/20260118135440.1958279-1-den@valinux.co.jp/
> 
> 2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
>    would create an auxiliary device, and a new dw-edma-aux driver would create
>    the delegated DMA channels on the RC side.
> 
>    [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
>    https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/
> 
>    I added an ASCII diagram for the overview as a follow-up comment here:
>    https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/)
> 
> Now, this v2 series takes a third direction. It moves the DMA controller out of
> vNTB/NTB-specific ABI and exposes it as a separate PCI endpoint DMA function.
> The host then discovers it as a DMA controller function. The initial host-side
> driver is the existing dw-edma-pcie driver, and dw-edma-aux is no longer needed.
> 
> My current thinking is that this is the cleanest among the previous attempts.
> But this is mostly an architecture question, so I would like to know whether
> this direction looks acceptable to you.
> 
> In short, do you agree with the direction of this series, that endpoint DMA
> channel delegation should be modeled as a separate PCI endpoint DMA function?
> 
> If you think the vNTB-integrated direction is preferable, or if this should be
> modeled differently in the endpoint framework, I would rather adjust the
> direction as early as possible, before building the NTB transport on top of it.

Hello Koichiro,

I think it would have been nice if your overall goal was more clearly
described in the cover letter.

AFAICT, you goal is for "upper layer NTB consumers" to be able to use these
DMA channels.


Since these DMAengine channels will exposed on the host side, I assume that
these "upper layer NTB consumers" are also on the host side.
Could you perhaps give some specific examples of drivers on the host side
that will use these DMA channels?

How will these drivers on the host side know to use the correct DMA channel,
i.e. the DMA channel that is backed by this new PCI DMA EPF?
(And not some other random DMA channel, in case the SoC has multiple DMA
channels.)

If you need to configure your endpoint SoC to bind to the PCI DMA EPF,
don't you need the endpoint SoC to bind to the pci-epf-ntb or pci-epf-vntb
driver? I know that some endpoint controllers can bind to multiple EPFs.
Is the intention for the endpoint SoC to bind both to this new and PCI
DMA EPF and pci-epf-vntb ?

If so, but do really all endpoint controllers / endpoint controller drivers
support binding to multiple EPFs?


Kind regards,
Niklas

On Mon, May 25, 2026 at 10:35:46AM +0200, Niklas Cassel wrote:
> On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
> > 
> > I would like to ask you for your high-level opinion on the direction of this
> > series.
> > 
> > Previously, I have tried two different approaches for the same objective:
> > avoiding the extra CPU memcpy (or local DMA memcpy) in NTB transport on both EP
> > and RC sides.
> > 
> > 1. Put dw-edma-specific handling under drivers/ntb/hw and let the (new) NTB
> >    driver carry the metadata needed for channel delegation.
> > 
> >    [RFC PATCH v4 00/38] NTB transport backed by PCI EP embedded DMA
> >    https://lore.kernel.org/all/20260118135440.1958279-1-den@valinux.co.jp/
> > 
> > 2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
> >    would create an auxiliary device, and a new dw-edma-aux driver would create
> >    the delegated DMA channels on the RC side.
> > 
> >    [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
> >    https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/
> > 
> >    I added an ASCII diagram for the overview as a follow-up comment here:
> >    https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/)
> > 
> > Now, this v2 series takes a third direction. It moves the DMA controller out of
> > vNTB/NTB-specific ABI and exposes it as a separate PCI endpoint DMA function.
> > The host then discovers it as a DMA controller function. The initial host-side
> > driver is the existing dw-edma-pcie driver, and dw-edma-aux is no longer needed.
> > 
> > My current thinking is that this is the cleanest among the previous attempts.
> > But this is mostly an architecture question, so I would like to know whether
> > this direction looks acceptable to you.
> > 
> > In short, do you agree with the direction of this series, that endpoint DMA
> > channel delegation should be modeled as a separate PCI endpoint DMA function?
> > 
> > If you think the vNTB-integrated direction is preferable, or if this should be
> > modeled differently in the endpoint framework, I would rather adjust the
> > direction as early as possible, before building the NTB transport on top of it.
> 
> Hello Koichiro,

Hello Niklas,

> 
> I think it would have been nice if your overall goal was more clearly
> described in the cover letter.

Fair enough. Part 1 describes the use case:
https://lore.kernel.org/dmaengine/20260525062420.3315904-1-den@valinux.co.jp/
but part 3 should probably have stated the overall goal as well.

> 
> AFAICT, you goal is for "upper layer NTB consumers" to be able to use these
> DMA channels.
> 
> 
> Since these DMAengine channels will exposed on the host side, I assume that
> these "upper layer NTB consumers" are also on the host side.
> Could you perhaps give some specific examples of drivers on the host side
> that will use these DMA channels?

I have not submitted the first real consumer code (= NTB transport backed by PCI
EP DMA) yet. I plan to do that after checking whether the direction taken by
this series is acceptable.

That said, the consumer would be something like:
https://lore.kernel.org/ntb/20260118135440.1958279-27-den@valinux.co.jp/
although the naming "ntb_transport_edma" is no longer suitable (it would be
"ntb_transport_ep_dma" or something similar). Also, the old RFC holds
dw-edma-specific handling under drivers/ntb/hw, which is what I am trying to
avoid with this series, so the whole drivers/ntb/hw/edma/ would no longer be
needed.

With this direction, the NTB transport would use a DMA engine provider exposed
by the separate PCI DMA EPF, while the data path would still be very close to
the old RFC.

> 
> How will these drivers on the host side know to use the correct DMA channel,
> i.e. the DMA channel that is backed by this new PCI DMA EPF?
> (And not some other random DMA channel, in case the SoC has multiple DMA
> channels.)

I believe the NTB transport client driver should not request a channel by
capability mask alone, even if a specific dma_transaction_type for this sort of
transfer is added. It will need to know the specific PCI DMA EPF device to use,
for example through configuration. One option would be to let the admin specify
the PCI BDF of the DMA function when loading/configuring the NTB transport
client.

> 
> If you need to configure your endpoint SoC to bind to the PCI DMA EPF,
> don't you need the endpoint SoC to bind to the pci-epf-ntb or pci-epf-vntb
> driver? I know that some endpoint controllers can bind to multiple EPFs.
> Is the intention for the endpoint SoC to bind both to this new and PCI
> DMA EPF and pci-epf-vntb ?

Yes, for the NTB transport backed by PCI EP DMA, the endpoint side would expose
both functions, vNTB and PCI DMA EPF.

> 
> If so, but do really all endpoint controllers / endpoint controller drivers
> support binding to multiple EPFs?

No. For example, R-Car S4's PCIe controller supports multi-functions, while
RK3588's PCIe controller seems not. So with this scheme, RK3588 would not
support the NTB transport backed by PCI EP DMA.

That restriction should be documented with the new NTB transport, which I will
submit if the direction taken by this series is acceptable.

Best regards,
Koichiro

> 
> 
> Kind regards,
> Niklas

On 25 May 2026 16:03:35 CEST, Koichiro Den <den@valinux.co.jp> wrote:
>On Mon, May 25, 2026 at 10:35:46AM +0200, Niklas Cassel wrote:
>> On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
>> 
>> If so, but do really all endpoint controllers / endpoint controller drivers
>> support binding to multiple EPFs?
>
>No. For example, R-Car S4's PCIe controller supports multi-functions, while
>RK3588's PCIe controller seems not. So with this scheme, RK3588 would not
>support the NTB transport backed by PCI EP DMA.

I guess it depends on 'max-functions' being set in device tree or not:
https://github.com/torvalds/linux/blob/master/drivers/pci/controller/dwc/pcie-designware-ep.c#L1365-L1367

AFAICT, this value depends on how the DWC PCI controller hardware configuration.


>
>That restriction should be documented with the new NTB transport, which I will
>submit if the direction taken by this series is acceptable.

This is easy for me to say, since I am not the NTB maintainer, but it would be nice if we could somehow come up with a design where we don't only support EPCs that have 'max-functions' != 1, because IIRC, most PCI EPCs have 'max-functions' == 1.


Kind regards,
Nikla

On Mon, May 25, 2026 at 10:32:06PM +0200, Niklas Cassel wrote:
> On 25 May 2026 16:03:35 CEST, Koichiro Den <den@valinux.co.jp> wrote:
> >On Mon, May 25, 2026 at 10:35:46AM +0200, Niklas Cassel wrote:
> >> On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
> >> 
> >That restriction should be documented with the new NTB transport, which I will
> >submit if the direction taken by this series is acceptable.
> 
> This is easy for me to say, since I am not the NTB maintainer, but it would be nice if we could somehow come up with a design where we don't only support EPCs that have 'max-functions' != 1, because IIRC, most PCI EPCs have 'max-functions' == 1.

Yes, that's fair point. As a quick check on v7.1-rc5, among DWC-based EP nodes,
only 6 out of 45 set max-functions > 1 (about 13%). Assuming there are no cases
where the hardware supports more functions than the DT advertises, that means only
about 13% of DWC-based EP instances described in DT could support the "NTB
transport backed by PCI EP DMA" use case. If I also count non-DWC EP nodes, I
get 15 out of 64 (about 23%).

If supporting single-function EPCs is a requirement, then the separate PCI DMA
EPF model is not a good choice for that NTB transport use case. We would need to
keep the DMA delegation metadata inside the vNTB function, or use some other
single-function design.

That is basically option 2 from my earlier mail:
https://lore.kernel.org/linux-pci/xnfnxv64hpil6if4ikyohxnarvsekbmjcc37k5zej264ix46z3@qtu6xj2uy3xi/

    [snip]
    2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
       would create an auxiliary device, and a new dw-edma-aux driver would create
       the delegated DMA channels on the RC side.
    
       [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
       https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/
    
       I added an ASCII diagram for the overview as a follow-up comment here:
       https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/
    [snip]

Do you prefer the vNTB-integrated model over this series?

Best regards,
Koichiro

> 
> 
> Kind regards,
> Nikla

Hello Koichiro,

On 26 May 2026 04:04:07 CEST, Koichiro Den <den@valinux.co.jp> wrote:
>On Mon, May 25, 2026 at 10:32:06PM +0200, Niklas Cassel wrote:
>> On 25 May 2026 16:03:35 CEST, Koichiro Den <den@valinux.co.jp> wrote:
>> >On Mon, May 25, 2026 at 10:35:46AM +0200, Niklas Cassel wrote:
>> >> On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
>> >> 
>> >That restriction should be documented with the new NTB transport, which I will
>> >submit if the direction taken by this series is acceptable.
>> 
>> This is easy for me to say, since I am not the NTB maintainer, but it would be nice if we could somehow come up with a design where we don't only support EPCs that have 'max-functions' != 1, because IIRC, most PCI EPCs have 'max-functions' == 1.
>
>Yes, that's fair point. As a quick check on v7.1-rc5, among DWC-based EP nodes,
>only 6 out of 45 set max-functions > 1 (about 13%). Assuming there are no cases
>where the hardware supports more functions than the DT advertises, that means only
>about 13% of DWC-based EP instances described in DT could support the "NTB
>transport backed by PCI EP DMA" use case. If I also count non-DWC EP nodes, I
>get 15 out of 64 (about 23%).

The only DMA "backend" added in your 3-part series is the eDMA in DWC-based controllers.

So if all three of your series lands, then 13% of the DWC-based endpoint controllers can theoretically use this new feature.


>
>If supporting single-function EPCs is a requirement, then the separate PCI DMA
>EPF model is not a good choice for that NTB transport use case. We would need to
>keep the DMA delegation metadata inside the vNTB function, or use some other
>single-function design.
>
>That is basically option 2 from my earlier mail:
>https://lore.kernel.org/linux-pci/xnfnxv64hpil6if4ikyohxnarvsekbmjcc37k5zej264ix46z3@qtu6xj2uy3xi/
>
>    [snip]
>    2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
>       would create an auxiliary device, and a new dw-edma-aux driver would create
>       the delegated DMA channels on the RC side.
>    
>       [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
>       https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/
>    
>       I added an ASCII diagram for the overview as a follow-up comment here:
>       https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/
>    [snip]
>
>Do you prefer the vNTB-integrated model over this series?

My take:

I do think that the design in  this series is more elegant that the vNTB-integrated model.

However, if the design in this series only supports 13% of DWC-based endpoint controllers, when the vNTB-integrated model can support 100% of DWC-based endpoint controllers...

What good it is to have an elegant design if in reality, it supports drastically fewer SoCs?

But please don't listen only to my opinion, Mani is the maintainer, so it would be interesting to hear his thoughts as well.


Kind regards,
Niklas

On Tue, May 26, 2026 at 07:35:06AM +0200, Niklas Cassel wrote:
> Hello Koichiro,
> 
> On 26 May 2026 04:04:07 CEST, Koichiro Den <den@valinux.co.jp> wrote:
> >On Mon, May 25, 2026 at 10:32:06PM +0200, Niklas Cassel wrote:
> >> On 25 May 2026 16:03:35 CEST, Koichiro Den <den@valinux.co.jp> wrote:
> >> >On Mon, May 25, 2026 at 10:35:46AM +0200, Niklas Cassel wrote:
> >> >> On Mon, May 25, 2026 at 04:05:02PM +0900, Koichiro Den wrote:
> >> >> 
> >> >That restriction should be documented with the new NTB transport, which I will
> >> >submit if the direction taken by this series is acceptable.
> >> 
> >> This is easy for me to say, since I am not the NTB maintainer, but it would be nice if we could somehow come up with a design where we don't only support EPCs that have 'max-functions' != 1, because IIRC, most PCI EPCs have 'max-functions' == 1.
> >
> >Yes, that's fair point. As a quick check on v7.1-rc5, among DWC-based EP nodes,
> >only 6 out of 45 set max-functions > 1 (about 13%). Assuming there are no cases
> >where the hardware supports more functions than the DT advertises, that means only
> >about 13% of DWC-based EP instances described in DT could support the "NTB
> >transport backed by PCI EP DMA" use case. If I also count non-DWC EP nodes, I
> >get 15 out of 64 (about 23%).
> 
> The only DMA "backend" added in your 3-part series is the eDMA in DWC-based controllers.
> 
> So if all three of your series lands, then 13% of the DWC-based endpoint controllers can theoretically use this new feature.
> 
> 
> >
> >If supporting single-function EPCs is a requirement, then the separate PCI DMA
> >EPF model is not a good choice for that NTB transport use case. We would need to
> >keep the DMA delegation metadata inside the vNTB function, or use some other
> >single-function design.
> >
> >That is basically option 2 from my earlier mail:
> >https://lore.kernel.org/linux-pci/xnfnxv64hpil6if4ikyohxnarvsekbmjcc37k5zej264ix46z3@qtu6xj2uy3xi/
> >
> >    [snip]
> >    2. Treat endpoint DMA as a first-class part of vNTB. The RC-side ntb_hw_epf
> >       would create an auxiliary device, and a new dw-edma-aux driver would create
> >       the delegated DMA channels on the RC side.
> >    
> >       [PATCH 00/15] PCI: endpoint: Remote DMA support via vNTB
> >       https://lore.kernel.org/linux-pci/20260312165005.1148676-1-den@valinux.co.jp/
> >    
> >       I added an ASCII diagram for the overview as a follow-up comment here:
> >       https://lore.kernel.org/all/sn67hi7kljh7cgmgodatb3naz2astlaklqfobdbxyyzgoohxqb@4nnetbhqwba4/
> >    [snip]
> >
> >Do you prefer the vNTB-integrated model over this series?
> 
> My take:
> 
> I do think that the design in  this series is more elegant that the vNTB-integrated model.
> 
> However, if the design in this series only supports 13% of DWC-based endpoint controllers, when the vNTB-integrated model can support 100% of DWC-based endpoint controllers...
> 
> What good it is to have an elegant design if in reality, it supports drastically fewer SoCs?
> 
> But please don't listen only to my opinion, Mani is the maintainer, so it would be interesting to hear his thoughts as well.

Yes, I also think the architecture of this series is much cleaner. The option 2
series may look like it overloads and complicates vNTB a bit too much, and the
auxiliary device created from ntb_hw_epf only for the channel delegation purpose
may look awkward to some.

The coverage concern is a real downside of this direction though. This is a
trade-off between a cleaner PCI/DMA model and broader EPC coverage. On my side,
R-Car Gen4+ is the main target, so the multi-function requirement is acceptable.
In that sense, I am also curious whether future DWC-based SoCs will typically
support more than one function or not.

Thanks for sharing your thoughts, Niklas. I would also like to hear Mani's view
on this series vs. previous attempts. Any comments from others are also very
welcome.

Best regards,
Koichiro

> 
> 
> Kind regards,
> Niklas

On Tue, May 26, 2026 at 03:23:57PM +0900, Koichiro Den wrote:
> On Tue, May 26, 2026 at 07:35:06AM +0200, Niklas Cassel wrote:
> 
> Yes, I also think the architecture of this series is much cleaner. The option 2
> series may look like it overloads and complicates vNTB a bit too much, and the
> auxiliary device created from ntb_hw_epf only for the channel delegation purpose
> may look awkward to some.
> 
> The coverage concern is a real downside of this direction though. This is a
> trade-off between a cleaner PCI/DMA model and broader EPC coverage. On my side,
> R-Car Gen4+ is the main target, so the multi-function requirement is acceptable.
> In that sense, I am also curious whether future DWC-based SoCs will typically
> support more than one function or not.

It seems that the number of supported physical functions is controlled by
the configurable PCIe IP-core synthesize parameter CX_NFUNC.


Looking at the code, if the device tree property 'max-functions' is missing,
it will set epc->max_functions to 1:

$ git show f8aed6ec624f

It has been like this since the initial EP support in the DWC driver, added
in 2017.

However, a missing 'max-functions' device tree property does not necessarily
mean that the IP-core was configured with CX_NFUNC == 1.


Right now, I don't see any register to get CX_NFUNC.

Perhaps it is possible to write some code that figures out CX_NFUNC, by
writing different values to the iATU registers, somewhat similar to how we
detect the number of inbound and outbound iATUs:
https://github.com/torvalds/linux/blob/v7.1-rc5/drivers/pci/controller/dwc/pcie-designware.c#L998-L1001



I added EP mode support for the pcie-dw-rockchip driver, but I don't know
the CX_NFUNC parameter value, so I could not add a 'max-functions' property.

While it is possible that some DWC-based PCIe endpoint controllers are
configured with CX_NFUNC == 1, it is also possible that some people simply
did now add a 'max-functions' property, because they did not know the CX_NFUNC
value, just like me.


Shawn Lin, do you perhaps know the CX_NFUNC value for rk3588 and rk3568 ?


Kind regards,
Niklas