RDMA/erdma: complete object teardown when the destroy command fails

[PATCH rdma-next] RDMA/erdma: complete object teardown when the destroy command fails

Posted by Leon Romanovsky 2 days, 9 hours ago

From: Leon Romanovsky <leonro@nvidia.com>

erdma_destroy_qp(), erdma_destroy_cq(), erdma_dereg_mr(), and
erdma_destroy_ah() returned early when erdma_post_cmd_wait() failed,
leaking the queue buffers, MTTs, doorbells and the STAG, QPN, CQN and AHN
identifiers. A command timeout clears ERDMA_CMDQ_STATE_OK_BIT and
permanently disables the command queue, so no retry can succeed; the RDMA
core keeps the object after a failed destructor and forced uverbs cleanup
then nulls the pointers, making the resources unreachable.

Warn on failure but release every software-owned resource and return
success, since during terminal destruction the hardware command result is
only diagnostic.

Fixes: 155055771704 ("RDMA/erdma: Add verbs implementation")
Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
---
 drivers/infiniband/hw/erdma/erdma_verbs.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index 71e3e8618a61..ab6abbab029e 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -1302,8 +1302,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
 
 	ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
 				  true);
+	/*
+	 * A timeout disables the command queue, so retry cannot succeed.  Treat
+	 * terminal command failures as diagnostic; propagating them can make
+	 * forced uverbs cleanup discard the last software resource pointers.
+	 */
 	if (ret)
-		return ret;
+		ibdev_warn_ratelimited(&dev->ibdev,
+				       "failed to deregister MR 0x%x: %d\n",
+				       ibmr->lkey, ret);
 
 	erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8);
 
@@ -1329,7 +1336,9 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
 	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
 				  true);
 	if (err)
-		return err;
+		ibdev_warn_ratelimited(&dev->ibdev,
+				       "failed to destroy CQ %u: %d\n",
+				       cq->cqn, err);
 
 	if (rdma_is_kernel_res(&cq->ibcq.res)) {
 		dma_free_coherent(&dev->pdev->dev, cq->depth << CQE_SHIFT,
@@ -1377,7 +1386,9 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
 	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
 				  true);
 	if (err)
-		return err;
+		ibdev_warn_ratelimited(&dev->ibdev,
+				       "failed to destroy QP %u: %d\n",
+				       QP_ID(qp), err);
 
 	erdma_qp_put(qp);
 	wait_for_completion(&qp->safe_free);
@@ -2279,7 +2290,9 @@ int erdma_destroy_ah(struct ib_ah *ibah, u32 flags)
 	ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
 				  flags & RDMA_DESTROY_AH_SLEEPABLE);
 	if (ret)
-		return ret;
+		ibdev_warn_ratelimited(&dev->ibdev,
+				       "failed to destroy AH %u: %d\n",
+				       ah->ahn, ret);
 
 	erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_AH], ah->ahn);
 

---
base-commit: 0e8e94c15091041ea8910cbfcade5a9c7cfe3f90
change-id: 20260722-b4-qp-and-cq-memory-are-leaked-if-the-d-760a0f866319

Best regards,
--  
Leon Romanovsky <leonro@nvidia.com>

Re: [PATCH rdma-next] RDMA/erdma: complete object teardown when the destroy command fails

Posted by Leon Romanovsky 1 day, 10 hours ago

On Wed, 22 Jul 2026 16:13:54 +0300, Leon Romanovsky wrote:
> erdma_destroy_qp(), erdma_destroy_cq(), erdma_dereg_mr(), and
> erdma_destroy_ah() returned early when erdma_post_cmd_wait() failed,
> leaking the queue buffers, MTTs, doorbells and the STAG, QPN, CQN and AHN
> identifiers. A command timeout clears ERDMA_CMDQ_STATE_OK_BIT and
> permanently disables the command queue, so no retry can succeed; the RDMA
> core keeps the object after a failed destructor and forced uverbs cleanup
> then nulls the pointers, making the resources unreachable.
> 
> [...]

Applied, thanks!

[1/1] RDMA/erdma: complete object teardown when the destroy command fails
      https://git.kernel.org/rdma/rdma/c/652befcba956ef

Best regards,
-- 
Leon Romanovsky <leonro@nvidia.com>

Re: [PATCH rdma-next] RDMA/erdma: complete object teardown when the destroy command fails

Posted by Cheng Xu 1 day, 14 hours ago


On 7/22/26 9:13 PM, Leon Romanovsky wrote:
> From: Leon Romanovsky <leonro@nvidia.com>
> 
> erdma_destroy_qp(), erdma_destroy_cq(), erdma_dereg_mr(), and
> erdma_destroy_ah() returned early when erdma_post_cmd_wait() failed,
> leaking the queue buffers, MTTs, doorbells and the STAG, QPN, CQN and AHN
> identifiers. A command timeout clears ERDMA_CMDQ_STATE_OK_BIT and
> permanently disables the command queue, so no retry can succeed; the RDMA
> core keeps the object after a failed destructor and forced uverbs cleanup
> then nulls the pointers, making the resources unreachable.
> 
> Warn on failure but release every software-owned resource and return
> success, since during terminal destruction the hardware command result is
> only diagnostic.
> 
> Fixes: 155055771704 ("RDMA/erdma: Add verbs implementation")
> Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> ---
>  drivers/infiniband/hw/erdma/erdma_verbs.c | 21 +++++++++++++++++----
>  1 file changed, 17 insertions(+), 4 deletions(-)
> 

Thanks Leon, this makes sense to me.

Acked-by: Cheng Xu <chengyou@linux.alibaba.com>

Thanks,
Cheng Xu

> diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
> index 71e3e8618a61..ab6abbab029e 100644
> --- a/drivers/infiniband/hw/erdma/erdma_verbs.c
> +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
> @@ -1302,8 +1302,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
>  
>  	ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
>  				  true);
> +	/*
> +	 * A timeout disables the command queue, so retry cannot succeed.  Treat
> +	 * terminal command failures as diagnostic; propagating them can make
> +	 * forced uverbs cleanup discard the last software resource pointers.
> +	 */
>  	if (ret)
> -		return ret;
> +		ibdev_warn_ratelimited(&dev->ibdev,
> +				       "failed to deregister MR 0x%x: %d\n",
> +				       ibmr->lkey, ret);
>  
>  	erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8);
>  
> @@ -1329,7 +1336,9 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
>  	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
>  				  true);
>  	if (err)
> -		return err;
> +		ibdev_warn_ratelimited(&dev->ibdev,
> +				       "failed to destroy CQ %u: %d\n",
> +				       cq->cqn, err);
>  
>  	if (rdma_is_kernel_res(&cq->ibcq.res)) {
>  		dma_free_coherent(&dev->pdev->dev, cq->depth << CQE_SHIFT,
> @@ -1377,7 +1386,9 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
>  	err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
>  				  true);
>  	if (err)
> -		return err;
> +		ibdev_warn_ratelimited(&dev->ibdev,
> +				       "failed to destroy QP %u: %d\n",
> +				       QP_ID(qp), err);
>  
>  	erdma_qp_put(qp);
>  	wait_for_completion(&qp->safe_free);
> @@ -2279,7 +2290,9 @@ int erdma_destroy_ah(struct ib_ah *ibah, u32 flags)
>  	ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL,
>  				  flags & RDMA_DESTROY_AH_SLEEPABLE);
>  	if (ret)
> -		return ret;
> +		ibdev_warn_ratelimited(&dev->ibdev,
> +				       "failed to destroy AH %u: %d\n",
> +				       ah->ahn, ret);
>  
>  	erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_AH], ah->ahn);
>  
> 
> ---
> base-commit: 0e8e94c15091041ea8910cbfcade5a9c7cfe3f90
> change-id: 20260722-b4-qp-and-cq-memory-are-leaked-if-the-d-760a0f866319
> 
> Best regards,
> --  
> Leon Romanovsky <leonro@nvidia.com>