On Thu, Aug 14, 2025 at 11:46:35AM +0100, Pavel Begunkov wrote:
> On 8/13/25 16:39, Dragos Tatulea wrote:
> > Hi Pavel,
> >
> > On Fri, Aug 08, 2025 at 03:54:23PM +0100, Pavel Begunkov wrote:
> > > [...]
> > > For example, for 200Gbit broadcom NIC, 4K vs 32K buffers, and napi and
> > > userspace pinned to the same CPU:
> > >
> > > packets=23987040 (MB=2745098), rps=199559 (MB/s=22837)
> > > CPU %usr %nice %sys %iowait %irq %soft %idle
> > > 0 1.53 0.00 27.78 2.72 1.31 66.45 0.22
> > > packets=24078368 (MB=2755550), rps=200319 (MB/s=22924)
> > > CPU %usr %nice %sys %iowait %irq %soft %idle
> > > 0 0.69 0.00 8.26 31.65 1.83 57.00 0.57
> > >
> > > And for napi and userspace on different CPUs:
> > >
> > > packets=10725082 (MB=1227388), rps=198285 (MB/s=22692)
> > > CPU %usr %nice %sys %iowait %irq %soft %idle
> > > 0 0.10 0.00 0.50 0.00 0.50 74.50 24.40
> > > 1 4.51 0.00 44.33 47.22 2.08 1.85 0.00
> > > packets=14026235 (MB=1605175), rps=198388 (MB/s=22703)
> > > CPU %usr %nice %sys %iowait %irq %soft %idle
> > > 0 0.10 0.00 0.70 0.00 1.00 43.78 54.42
> > > 1 1.09 0.00 31.95 62.91 1.42 2.63 0.00
> > >
I forgot to ask: what is the MTU here?
> > What did you use for this benchmark, send-zerocopy? Could you share a
> > branch and how you ran it please?
> >
> > I have added some initial support to mlx5 for rx-buf-len and would like
> > to benchmark it and compare it to what you posted.
>
> You can use this branch:
> https://github.com/isilence/liburing.git zcrx/rx-buf-len
>
> # server
> examples/zcrx -p <port> -q <queue_idx> -i <interface_name> -A1 \
> -B <rx_buf_len> -S <area size / memory provided>
>
> "-A1" here is for using huge pages, so don't forget to configure
> /proc/sys/vm/nr_hugepages.
>
> # client
> examples/send-zerocopy -6 tcp -D <ip addr> -p <port>
> -t <runtime secs>
> -l -b1 -n1 -z1 -d -s<send size>
>
Thanks a lot for the branch and the instructions Pavel! I am playing
with them now and seeing some preliminary good results. Will post
them once we share the patches.
> I had to play with the client a bit for it to keep up with
> the server. "-l" enables huge pages, and had to bump up the
> send size. You can also add -v to both for a basic payload
> verification.
>
I see what you mean. I also had to make the rx memory larger once
rx-buf-len >= 32K. Otherwise the traffic was hanging after a second or
so. This is probably related to the currently known issue where if a
page_pool is too small due to incorrect sizing of the buffer, mlx5 hangs
on first refill. That still needs fixing.
Thanks,
Dragos