drivers/net/ethernet/microsoft/mana/mana_en.c | 22 ++- .../ethernet/microsoft/mana/mana_ethtool.c | 164 ++++++++++++++---- include/net/mana/mana.h | 8 + 3 files changed, 163 insertions(+), 31 deletions(-)
On some ARM64 platforms with 4K PAGE_SIZE, utilizing page_pool
fragments for allocation in the RX refill path (~2kB buffer per fragment)
causes 15-20% throughput regression under high connection counts
(>16 TCP streams at 180+ Gbps). Using full-page buffers on these
platforms shows no regression and restores line-rate performance.
This behavior is observed on a single platform; other platforms
perform better with page_pool fragments, indicating this is not a
page_pool issue but platform-specific.
This series adds an ethtool private flag "full-page-rx" to let the
user opt in to one RX buffer per page:
ethtool --set-priv-flags eth0 full-page-rx on
There is no behavioral change by default. The flag can be persisted
via udev rule for affected platforms.
Changes in v6:
- Added missed maintainers.
Changes in v5:
- Split prep refactor into separate patch (patch 1/2)
Changes in v4:
- Dropping the smbios string parsing and add ethtool priv flag
to reconfigure the queues with full page rx buffers.
Changes in v3:
- changed u8* to char*
Changes in v2:
- separate reading string index and the string, remove inline.
Dipayaan Roy (2):
net: mana: refactor mana_get_strings() and mana_get_sset_count() to
use switch
net: mana: force full-page RX buffers via ethtool private flag
drivers/net/ethernet/microsoft/mana/mana_en.c | 22 ++-
.../ethernet/microsoft/mana/mana_ethtool.c | 164 ++++++++++++++----
include/net/mana/mana.h | 8 +
3 files changed, 163 insertions(+), 31 deletions(-)
--
2.43.0
On Tue, 7 Apr 2026 12:59:17 -0700 Dipayaan Roy wrote: > This behavior is observed on a single platform; other platforms > perform better with page_pool fragments, indicating this is not a > page_pool issue but platform-specific. Well, someone has to run some experiments and confirm other ARM platforms are not impacted, with data. I was hoping to do it myself but doesn't look like that will happen in time for the merge window :( > Changes in v6: > - Added missed maintainers. STOP REPOSTING PATCHES FOR NO REASON.
On Thu, 9 Apr 2026 18:35:09 -0700 Jakub Kicinski wrote: > On Tue, 7 Apr 2026 12:59:17 -0700 Dipayaan Roy wrote: > > This behavior is observed on a single platform; other platforms > > perform better with page_pool fragments, indicating this is not a > > page_pool issue but platform-specific. > > Well, someone has to run some experiments and confirm other ARM > platforms are not impacted, with data. I was hoping to do it myself > but doesn't look like that will happen in time for the merge window :( Please repost with the perf analysis on other commercially available ARM platform. Something like: This is a workaround applicable to only some platforms. Modifying driver X to use a similar workaround on [Ampere Max|nVidia Grace|Amazon Graviton 3|..] the performance for split pages is y% higher than when using single pages. -- pw-bot: cr
On Sun, Apr 12, 2026 at 12:59:17PM -0700, Jakub Kicinski wrote: > On Thu, 9 Apr 2026 18:35:09 -0700 Jakub Kicinski wrote: > > On Tue, 7 Apr 2026 12:59:17 -0700 Dipayaan Roy wrote: > > > This behavior is observed on a single platform; other platforms > > > perform better with page_pool fragments, indicating this is not a > > > page_pool issue but platform-specific. > > > > Well, someone has to run some experiments and confirm other ARM > > platforms are not impacted, with data. I was hoping to do it myself > > but doesn't look like that will happen in time for the merge window :( > > Please repost with the perf analysis on other commercially available > ARM platform. Something like: > > This is a workaround applicable to only some platforms. Modifying > driver X to use a similar workaround on [Ampere Max|nVidia > Grace|Amazon Graviton 3|..] the performance for split pages is > y% higher than when using single pages. > -- > pw-bot: cr Hi Jakub, I ran the same experiment on an alternate ARM64 platform from a different vendor, which I was able to access only recently. I still see roughly a 5% overhead from the atomic refcount operation itself, but on that platform there is no throughput drop when using page fragments versus full-page mode. In both cases, the setup reaches line rate. That suggests the atomic overhead alone does not explain the throughput loss on the specific hardware we are discussing. I also received an update from the hardware team. They collected PCIe traces and observed stalls on this particular ARM64 prcossor when running with page fragments, while those stalls are not seen in full-page mode. The exact root cause is still under investigation, but their current assessment is that this is likely a microarchitectural issue in the PCIe root port. Based on that, they are asking for a software workaround that uses full pages until the issue is fully understood. For that reason, I am asking whether this could be accepted as an ethtool private flag rather than as a generic driver change, since the problem is still specific to one CPU/platform. Please let me know whether you think this patch with private flag would be acceptable here. Regards Dipayaan Roy
On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: > I still see roughly a 5% overhead from the atomic refcount operation > itself, but on that platform there is no throughput drop when using > page fragments versus full-page mode. That seems to contradict your claim that it's a problem with a specific platform.. Since we're in the merge window I asked David Wei to try to experiment with disabling page fragmentation on the ARM64 platforms we have at Meta. If it repros we should use the generic rx-buf-len ringparam because more NICs may want to implement this strategy.
On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: > On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: > > I still see roughly a 5% overhead from the atomic refcount operation > > itself, but on that platform there is no throughput drop when using > > page fragments versus full-page mode. > > That seems to contradict your claim that it's a problem with a specific > platform.. Since we're in the merge window I asked David Wei to try to > experiment with disabling page fragmentation on the ARM64 platforms we > have at Meta. If it repros we should use the generic rx-buf-len > ringparam because more NICs may want to implement this strategy. Hi Jakub, Thanks. I think I was not precise enough in my previous reply. What I meant is that the atomic refcount cost itself does not appear to be unique to the affected platform. I see a similar ~5% overhead on another ARM64 platformi (different vendor) as well. However, on that platform there is no throughput delta between fragment mode and full-page mode; both reach line rate. On the affected platform, fragment mode shows an additional ~15% throughput drop versus full-page mode. So the current data suggests that the atomic overhead is common, but the throughput regression is not explained by that overhead alone and likely depends on an additional platform-specific factor. Separately, the hardware team collected PCIe traces on the affected platform and reported stalls in the fragment-mode case that are not seen in full-page mode. They are still investigating the root cause, but their current hypothesis is that this is related to that platform’s PCIe/root-port microarchitecture rather than to page_pool refcounting alone. That said, I agree the right direction depends on whether this reproduces on other ARM64 platforms. If David is able to reproduce the same behavior, then using the generic rx-buf-len ringparam sounds like the better direction. Please let me know what David finds, and I can rework the patch accordingly. Regards Dipayaan Roy
On 2026-04-23 05:48, Dipayaan Roy wrote: > On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: >> On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: >>> I still see roughly a 5% overhead from the atomic refcount operation >>> itself, but on that platform there is no throughput drop when using >>> page fragments versus full-page mode. >> >> That seems to contradict your claim that it's a problem with a specific >> platform.. Since we're in the merge window I asked David Wei to try to >> experiment with disabling page fragmentation on the ARM64 platforms we >> have at Meta. If it repros we should use the generic rx-buf-len >> ringparam because more NICs may want to implement this strategy. > > Hi Jakub, > > Thanks. I think I was not precise enough in my previous reply. > > What I meant is that the atomic refcount cost itself does not appear to > be unique to the affected platform. I see a similar ~5% overhead on > another ARM64 platformi (different vendor) as well. However, on that platform > there is no throughput delta between fragment mode and full-page mode; both reach > line rate. > > On the affected platform, fragment mode shows an additional ~15% > throughput drop versus full-page mode. So the current data suggests that > the atomic overhead is common, but the throughput regression is not > explained by that overhead alone and likely depends on an additional > platform-specific factor. > > Separately, the hardware team collected PCIe traces on the affected > platform and reported stalls in the fragment-mode case that are not seen > in full-page mode. They are still investigating the root cause, but > their current hypothesis is that this is related to that platform’s > PCIe/root-port microarchitecture rather than to page_pool refcounting > alone. > > That said, I agree the right direction depends on whether this > reproduces on other ARM64 platforms. If David is able to reproduce the > same behavior, then using the generic rx-buf-len ringparam sounds like > the better direction. > > Please let me know what David finds, and I can rework the patch > accordingly. I ran a test on Grace, 4 KB pages, 72 cores, 1 NUMA node. Broadcom NIC, bnxt driver, 50 Gbps bandwidth. Hacked it up to either give me 1 or 2 frags per page. No agg ring, no HDS, no HW GRO. Use 1 combined queue only for the server. Affinitized its net rx softirq to run on core 4. Ran iperf3 server, taskset onto cpu cores 32-47. The iperf3 client is running on a host w/ same hw in the same region. Using 32 queues, no softirq affinities. The idea is to hammer page->pp_ref_count from different cores. * 1 frag/page -> 32.3 Gbps * 2 frags/page -> 36.0 Gbps Comparing perf, for 2 frags/page the cost of skb_release_data() hitting pp_ref_count goes up, as expected. Is this what you see? When you say there's a +5% overhead, what function? Overall tput is higher with multiple frags. That's to be expected w/ page pool. There are some 200 Gbps NICs but they're mlx5 so I'd have to redo the driver hack. Are you going to re-implement this change with rx-buf-len instead of a private flag? If so, I won't spend more time running this test. > > > Regards > Dipayaan Roy
On Fri, Apr 24, 2026 at 01:05:24PM -0700, David Wei wrote: > On 2026-04-23 05:48, Dipayaan Roy wrote: > > On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: > > > On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: > > > > I still see roughly a 5% overhead from the atomic refcount operation > > > > itself, but on that platform there is no throughput drop when using > > > > page fragments versus full-page mode. > > > > > > That seems to contradict your claim that it's a problem with a specific > > > platform.. Since we're in the merge window I asked David Wei to try to > > > experiment with disabling page fragmentation on the ARM64 platforms we > > > have at Meta. If it repros we should use the generic rx-buf-len > > > ringparam because more NICs may want to implement this strategy. > > > > Hi Jakub, > > > > Thanks. I think I was not precise enough in my previous reply. > > > > What I meant is that the atomic refcount cost itself does not appear to > > be unique to the affected platform. I see a similar ~5% overhead on > > another ARM64 platformi (different vendor) as well. However, on that platform > > there is no throughput delta between fragment mode and full-page mode; both reach > > line rate. > > > > On the affected platform, fragment mode shows an additional ~15% > > throughput drop versus full-page mode. So the current data suggests that > > the atomic overhead is common, but the throughput regression is not > > explained by that overhead alone and likely depends on an additional > > platform-specific factor. > > > > Separately, the hardware team collected PCIe traces on the affected > > platform and reported stalls in the fragment-mode case that are not seen > > in full-page mode. They are still investigating the root cause, but > > their current hypothesis is that this is related to that platform’s > > PCIe/root-port microarchitecture rather than to page_pool refcounting > > alone. > > > > That said, I agree the right direction depends on whether this > > reproduces on other ARM64 platforms. If David is able to reproduce the > > same behavior, then using the generic rx-buf-len ringparam sounds like > > the better direction. > > > > Please let me know what David finds, and I can rework the patch > > accordingly. > > I ran a test on Grace, 4 KB pages, 72 cores, 1 NUMA node. > > Broadcom NIC, bnxt driver, 50 Gbps bandwidth. Hacked it up to either > give me 1 or 2 frags per page. No agg ring, no HDS, no HW GRO. > > Use 1 combined queue only for the server. Affinitized its net rx softirq > to run on core 4. > > Ran iperf3 server, taskset onto cpu cores 32-47. The iperf3 client is > running on a host w/ same hw in the same region. Using 32 queues, no > softirq affinities. The idea is to hammer page->pp_ref_count from > different cores. > > * 1 frag/page -> 32.3 Gbps > * 2 frags/page -> 36.0 Gbps > > Comparing perf, for 2 frags/page the cost of skb_release_data() hitting > pp_ref_count goes up, as expected. Is this what you see? When you say > there's a +5% overhead, what function? > > Overall tput is higher with multiple frags. That's to be expected w/ > page pool. Hi David, Thanks for running this. Your results are consistent with mine. I have tested this on 2 ARM64 platforms from different vendors, running ntttcp and iperf3 using 4k as base page size. In my observation I see both platforms show a 5% overhead in napi_pp_put_page (~3.9%) and page_pool_alloc_frag_netmem (~1.9%) when running in fragment mode, both stalling on the LSE ldaddal atomic that maintains pp_ref_count. This seems to be same as your observation as well. However in my observation one of the platform shows 15% drop in throughput when in fragment mode vs page mode. The other platform I ran the test on infact performs slighty better in fragment mode than in full page mode (simillar observation as yours). So the atomic refcount overhead appears to be common across ARM64 platforms, but it does not cause a throughput regression. The throughput regression seems specific to one platform only for which we want to have the full page work around, also the HW team has identified PCIe stalls in fragment mode that are absent in full-page mode. Their investigation points to a suspected microarchitectural issue in the PCIe root port. IMO, there seems to be no issue with page_pool itself. Given that: - Grace shows fragments are faster (your data) - A second ARM64 platform shows no regression (my data) - Only the affected platform shows a throughput drop - The HW team suspects this to a platform-specific PCIe issue, also form our experiment data the drop in throughput seems to be platform specific only. I believe this remains a platform-specific workaround rather than a generic issue. Would a private flag still be acceptable for this case? > > There are some 200 Gbps NICs but they're mlx5 so I'd have to redo the > driver hack. Are you going to re-implement this change with rx-buf-len > instead of a private flag? If so, I won't spend more time running this > test. > I can go either way depending on what Jakub prefers. Hi Jakub, with this new data from David, is it convincing enough for a mana driver specific private flag, which can be set from user space by a udev rule by detecting the underlying platform? If not then I will send the next version with the other rxbuflen approach. > > > > > > Regards > > Dipayaan Roy Thanks and Regards Dipayaan Roy
On Sat, 25 Apr 2026 01:05:43 -0700 Dipayaan Roy wrote: > Hi Jakub, > with this new data from David, is it convincing enough for a mana driver > specific private flag, which can be set from user space by a udev rule > by detecting the underlying platform? If not then I will send the next > version with the other rxbuflen approach. I think so, thank you both for the testing. Please look out for the net-next opening up and repost the patches. (The reopening is delayed, it was supposed to happen already but I can't get a clean run out of our CI, sigh)
On Mon, Apr 27, 2026 at 01:17:45PM -0700, Jakub Kicinski wrote: > On Sat, 25 Apr 2026 01:05:43 -0700 Dipayaan Roy wrote: > > Hi Jakub, > > with this new data from David, is it convincing enough for a mana driver > > specific private flag, which can be set from user space by a udev rule > > by detecting the underlying platform? If not then I will send the next > > version with the other rxbuflen approach. > > I think so, thank you both for the testing. > Please look out for the net-next opening up and repost the patches. > (The reopening is delayed, it was supposed to happen already but I > can't get a clean run out of our CI, sigh) Hi Jakub, I have reposted the patches now. Regards Dipayaan Roy
On 2026-04-25 01:05, Dipayaan Roy wrote: > On Fri, Apr 24, 2026 at 01:05:24PM -0700, David Wei wrote: >> On 2026-04-23 05:48, Dipayaan Roy wrote: >>> On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: >>>> On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: >>>>> I still see roughly a 5% overhead from the atomic refcount operation >>>>> itself, but on that platform there is no throughput drop when using >>>>> page fragments versus full-page mode. >>>> >>>> That seems to contradict your claim that it's a problem with a specific >>>> platform.. Since we're in the merge window I asked David Wei to try to >>>> experiment with disabling page fragmentation on the ARM64 platforms we >>>> have at Meta. If it repros we should use the generic rx-buf-len >>>> ringparam because more NICs may want to implement this strategy. >>> >>> Hi Jakub, >>> >>> Thanks. I think I was not precise enough in my previous reply. >>> >>> What I meant is that the atomic refcount cost itself does not appear to >>> be unique to the affected platform. I see a similar ~5% overhead on >>> another ARM64 platformi (different vendor) as well. However, on that platform >>> there is no throughput delta between fragment mode and full-page mode; both reach >>> line rate. >>> >>> On the affected platform, fragment mode shows an additional ~15% >>> throughput drop versus full-page mode. So the current data suggests that >>> the atomic overhead is common, but the throughput regression is not >>> explained by that overhead alone and likely depends on an additional >>> platform-specific factor. >>> >>> Separately, the hardware team collected PCIe traces on the affected >>> platform and reported stalls in the fragment-mode case that are not seen >>> in full-page mode. They are still investigating the root cause, but >>> their current hypothesis is that this is related to that platform’s >>> PCIe/root-port microarchitecture rather than to page_pool refcounting >>> alone. >>> >>> That said, I agree the right direction depends on whether this >>> reproduces on other ARM64 platforms. If David is able to reproduce the >>> same behavior, then using the generic rx-buf-len ringparam sounds like >>> the better direction. >>> >>> Please let me know what David finds, and I can rework the patch >>> accordingly. >> >> I ran a test on Grace, 4 KB pages, 72 cores, 1 NUMA node. >> >> Broadcom NIC, bnxt driver, 50 Gbps bandwidth. Hacked it up to either >> give me 1 or 2 frags per page. No agg ring, no HDS, no HW GRO. >> >> Use 1 combined queue only for the server. Affinitized its net rx softirq >> to run on core 4. >> >> Ran iperf3 server, taskset onto cpu cores 32-47. The iperf3 client is >> running on a host w/ same hw in the same region. Using 32 queues, no >> softirq affinities. The idea is to hammer page->pp_ref_count from >> different cores. >> >> * 1 frag/page -> 32.3 Gbps >> * 2 frags/page -> 36.0 Gbps >> >> Comparing perf, for 2 frags/page the cost of skb_release_data() hitting >> pp_ref_count goes up, as expected. Is this what you see? When you say >> there's a +5% overhead, what function? >> >> Overall tput is higher with multiple frags. That's to be expected w/ >> page pool. > > Hi David, > > Thanks for running this. Your results are consistent with mine. > > I have tested this on 2 ARM64 platforms from different vendors, > running ntttcp and iperf3 using 4k as base page size. > In my observation I see both platforms show a 5% overhead in > napi_pp_put_page (~3.9%) and page_pool_alloc_frag_netmem (~1.9%) > when running in fragment mode, both stalling on the LSE ldaddal > atomic that maintains pp_ref_count. > This seems to be same as your observation as well. However in my > observation one of the platform shows 15% drop in throughput when > in fragment mode vs page mode. The other platform I ran the test on > infact performs slighty better in fragment mode than in full page > mode (simillar observation as yours). That's not what I observe. I don't see napi_pp_put_page at all, and page_pool_alloc_frag_netmem is actually lower with 2 frags/page (4.06%) than 1 frag/page (5.73%). The main difference is in skb_release_data which goes from 0.85% (1 frag/page) to 3.32% (2 frags/page). > > So the atomic refcount overhead appears to be common across ARM64 > platforms, but it does not cause a throughput regression. > The throughput regression seems specific to one platform only for which > we want to have the full page work around, also the HW team has > identified PCIe stalls in fragment mode that are absent in full-page mode. > Their investigation points to a suspected microarchitectural > issue in the PCIe root port. IMO, there seems to be no issue with > page_pool itself. > > Given that: > - Grace shows fragments are faster (your data) > - A second ARM64 platform shows no regression (my data) > - Only the affected platform shows a throughput drop > - The HW team suspects this to a platform-specific PCIe issue, > also form our experiment data the drop in throughput seems to > be platform specific only. > > I believe this remains a platform-specific workaround rather than > a generic issue. Would a private flag still be acceptable for this > case? > > >> >> There are some 200 Gbps NICs but they're mlx5 so I'd have to redo the >> driver hack. Are you going to re-implement this change with rx-buf-len >> instead of a private flag? If so, I won't spend more time running this >> test. >> > I can go either way depending on what Jakub prefers. > Hi Jakub, > with this new data from David, is it convincing enough for a mana driver > specific private flag, which can be set from user space by a udev rule > by detecting the underlying platform? If not then I will send the next > version with the other rxbuflen approach. >>> >>> >>> Regards >>> Dipayaan Roy > > > Thanks and Regards > Dipayaan Roy
On 2026-04-23 05:48, Dipayaan Roy wrote: > On Thu, Apr 16, 2026 at 08:31:46AM -0700, Jakub Kicinski wrote: >> On Tue, 14 Apr 2026 09:00:56 -0700 Dipayaan Roy wrote: >>> I still see roughly a 5% overhead from the atomic refcount operation >>> itself, but on that platform there is no throughput drop when using >>> page fragments versus full-page mode. >> >> That seems to contradict your claim that it's a problem with a specific >> platform.. Since we're in the merge window I asked David Wei to try to >> experiment with disabling page fragmentation on the ARM64 platforms we >> have at Meta. If it repros we should use the generic rx-buf-len >> ringparam because more NICs may want to implement this strategy. > > Hi Jakub, > > Thanks. I think I was not precise enough in my previous reply. > > What I meant is that the atomic refcount cost itself does not appear to > be unique to the affected platform. I see a similar ~5% overhead on > another ARM64 platformi (different vendor) as well. However, on that platform > there is no throughput delta between fragment mode and full-page mode; both reach > line rate. > > On the affected platform, fragment mode shows an additional ~15% > throughput drop versus full-page mode. So the current data suggests that > the atomic overhead is common, but the throughput regression is not > explained by that overhead alone and likely depends on an additional > platform-specific factor. > > Separately, the hardware team collected PCIe traces on the affected > platform and reported stalls in the fragment-mode case that are not seen > in full-page mode. They are still investigating the root cause, but > their current hypothesis is that this is related to that platform’s > PCIe/root-port microarchitecture rather than to page_pool refcounting > alone. > > That said, I agree the right direction depends on whether this > reproduces on other ARM64 platforms. If David is able to reproduce the > same behavior, then using the generic rx-buf-len ringparam sounds like > the better direction. > > Please let me know what David finds, and I can rework the patch > accordingly. Hi Dipayaan. Can you please share more details on your testing setup? * What are you using as the test client/server? iperf3 or something else? * What do you mean specifically by "5% overhead from the atomic refcount operation"? Some specific function? * What are you using to measure? perf? * How many queues, what is the napi softirq affinity? * How many NUMA nodes? Does the problem only appear when crossing? Thanks, David > > > Regards > Dipayaan Roy
On Thu, 23 Apr 2026 05:48:11 -0700 Dipayaan Roy wrote: > What I meant is that the atomic refcount cost itself does not appear to > be unique to the affected platform. I see a similar ~5% overhead on > another ARM64 platformi (different vendor) as well. However, on that platform > there is no throughput delta between fragment mode and full-page mode; both reach > line rate. I wonder if it wouldn't be more expedient at this stage to just switch to rx-buf-len rather than investigating in more detail. But we can wait for more data if you prefer. > Please let me know what David finds, and I can rework the patch > accordingly. Haven't heard back. I pinged him now.
© 2016 - 2026 Red Hat, Inc.