When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
thread hang if I "configure" its slave network device by taking its link
down and bringing it back up. Attempting to just `ls` the mount point in
this state gives EHOSTDOWN.

I believe the following is the root-cause of the hang: When the link is
taken down, the corresponding GID Table Entry is marked as pending deletion
and has its slave ndev set to NULL. All sends on RDMA connections still
using that GID Table Entry fail at MAD creation. SMB Direct eventually
detects this and tries to disconnect / reconnect to recover. Unfortunately,
disconnecting requires successfully sending either a DREQ or a DREP. Since
neither of them even post, the connection remains in the RDMA_CM_CONNECT
state, and no callback moves it out. The SMB Direct layer never gets the
RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.

Fix this in the CMA. If we call `rdma_disconnect` on a connected connection
and we fail to send both a DREP and a DREQ; disconnect, and thereby send
the `RDMA_CM_EVENT_DISCONNECTED` event to SMB Direct.

I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a
mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2'
of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
I reproduce the hang. With this patch applied, SMB Direct immediately
disconnects and reconnects when the slave device is brought down then up.
Listing and reading files from the mount point also work afterwards.

Artifacts for reproducing the hang and testing my fix are available at

    https://github.com/ammrat13/linux-cifs

There, you can find: logs from the "bad" and "good" kernels, the kernel
configuration, the VMs' configuration, and a playbook to trigger the hang.

Ammar Ratnani (2):
  infiniband: force disconnect if DREP and DREQ fail
  infiniband: trace force disconnections

 drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
 drivers/infiniband/core/cma_trace.h |  1 +
 2 files changed, 38 insertions(+), 4 deletions(-)

-- 
2.55.0

On Tue, Sep 08, 2026 at 08:44:25AM -0700, Ammar Ratnani wrote:
> When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
> thread hang if I "configure" its slave network device by taking its link
> down and bringing it back up. Attempting to just `ls` the mount point in
> this state gives EHOSTDOWN.
> 
> I believe the following is the root-cause of the hang: When the link is
> taken down, the corresponding GID Table Entry is marked as pending deletion
> and has its slave ndev set to NULL. All sends on RDMA connections still
> using that GID Table Entry fail at MAD creation. SMB Direct eventually
> detects this and tries to disconnect / reconnect to recover. Unfortunately,
> disconnecting requires successfully sending either a DREQ or a DREP. Since
> neither of them even post, the connection remains in the RDMA_CM_CONNECT
> state, and no callback moves it out. The SMB Direct layer never gets the
> RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.
> 
> Fix this in the CMA. If we call `rdma_disconnect` on a connected connection
> and we fail to send both a DREP and a DREQ; disconnect, and thereby send
> the `RDMA_CM_EVENT_DISCONNECTED` event to SMB Direct.
> 
> I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a
> mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2'
> of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
> I reproduce the hang. With this patch applied, SMB Direct immediately
> disconnects and reconnects when the slave device is brought down then up.
> Listing and reading files from the mount point also work afterwards.
> 
> Artifacts for reproducing the hang and testing my fix are available at
> 
>     https://github.com/ammrat13/linux-cifs
> 
> There, you can find: logs from the "bad" and "good" kernels, the kernel
> configuration, the VMs' configuration, and a playbook to trigger the hang.
> 
> Ammar Ratnani (2):
>   infiniband: force disconnect if DREP and DREQ fail
>   infiniband: trace force disconnections

I squashed the second patch into the first one, fixed the subject, removed
the useless AI-generated comments, placed trace call into right place, and
pushed the result.

Thanks

> 
>  drivers/infiniband/core/cma.c       | 41 ++++++++++++++++++++++++++---
>  drivers/infiniband/core/cma_trace.h |  1 +
>  2 files changed, 38 insertions(+), 4 deletions(-)
> 
> -- 
> 2.55.0

On Tue, 08 Sep 2026 08:44:25 -0700, Ammar Ratnani wrote:
> When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
> thread hang if I "configure" its slave network device by taking its link
> down and bringing it back up. Attempting to just `ls` the mount point in
> this state gives EHOSTDOWN.
> 
> I believe the following is the root-cause of the hang: When the link is
> taken down, the corresponding GID Table Entry is marked as pending deletion
> and has its slave ndev set to NULL. All sends on RDMA connections still
> using that GID Table Entry fail at MAD creation. SMB Direct eventually
> detects this and tries to disconnect / reconnect to recover. Unfortunately,
> disconnecting requires successfully sending either a DREQ or a DREP. Since
> neither of them even post, the connection remains in the RDMA_CM_CONNECT
> state, and no callback moves it out. The SMB Direct layer never gets the
> RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.
> 
> [...]

Applied, thanks!

[1/2] infiniband: force disconnect if DREP and DREQ fail
      https://git.kernel.org/rdma/rdma/c/ba88a28b47a703
[2/2] infiniband: trace force disconnections
      (no commit info)

Best regards,
-- 
Leon Romanovsky <leon@kernel.org>

Hi Ammar,

thanks for the fixes!
I let the rdma maintainers comment on the patches in detail,
but it's good to fix these problems where they happen.

> When I mount a CIFS share using SMB Direct over RoCEv2, I observe a kernel
> thread hang if I "configure" its slave network device by taking its link
> down and bringing it back up. Attempting to just `ls` the mount point in
> this state gives EHOSTDOWN.
> 
> I believe the following is the root-cause of the hang: When the link is
> taken down, the corresponding GID Table Entry is marked as pending deletion
> and has its slave ndev set to NULL. All sends on RDMA connections still
> using that GID Table Entry fail at MAD creation. SMB Direct eventually
> detects this and tries to disconnect / reconnect to recover. Unfortunately,
> disconnecting requires successfully sending either a DREQ or a DREP. Since
> neither of them even post, the connection remains in the RDMA_CM_CONNECT
> state, and no callback moves it out. The SMB Direct layer never gets the
> RDMA_CM_EVENT_DISCONNECTED it's waiting for, and hangs.
> 
> Fix this in the CMA. If we call `rdma_disconnect` on a connected connection
> and we fail to send both a DREP and a DREQ; disconnect, and thereby send
> the `RDMA_CM_EVENT_DISCONNECTED` event to SMB Direct.
> 
> I tested this change in QEMU using RXE. I ran Ubuntu 26.04.1 with a
> mainline kernel. On commit 9f0346dcbea3 ("Merge tag 'driver-core-7.3-rc2'
> of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core"),
> I reproduce the hang. With this patch applied, SMB Direct immediately
> disconnects and reconnects when the slave device is brought down then up.
> Listing and reading files from the mount point also work afterwards.
> 
> Artifacts for reproducing the hang and testing my fix are available at
> 
>      https://github.com/ammrat13/linux-cifs
Given you have some automation to reproduce it I'm
wondering if you could also test the iwarp case, see
https://lore.kernel.org/linux-rdma/20260805000159.321645-2-yunseong.kim@est.tech/

That was reported as fix for ksmbd, but I guess it
will also happen for the case your're seeing with rxe.
And your fixes are likely also good for ksmbd.

Thanks!
metze

On Tue, Sep 08, 2026 at 06:16:21PM +0200, Stefan Metzmacher wrote:
> Hi Ammar,
> 
> thanks for the fixes!
> I let the rdma maintainers comment on the patches in detail,
> but it's good to fix these problems where they happen.

I think that patches are ok.

Thanks

Thank you for the very quick response, Stefan!

> Given you have some automation to reproduce it I'm
> wondering if you could also test the iwarp case, see
> https://lore.kernel.org/linux-rdma/20260805000159.321645-2-yunseong.kim@est.tech/

I tried running my setup for iWARP. I used the old "bad" kernel, and the
only thing I changed was that I used an siw device instead of an rxe. I
do not get a hang in that case. To be more precise, I see the following
at the end of dmesg:

    [   40.006460] siw0: Port: 1 Link DOWN
    [   40.012912] siw0: Port: 1 Link ACTIVE

But I don't see any message of the form "INFO: task ... blocked for more
than ... seconds." Additionally, I'm able to list and read a file off
the CIFS mount, even after bringing the interface down and up.

For this test, I used commit 2beb1b31a12b ("Merge tag 'bpf-fixes' of
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf"). This is a later
tree than what I used in my last email, but I confirmed that I get the
same behavior I described in that email.

Thank you,
- Ammar Ratnani