From nobody Mon Sep 15 11:12:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC60AC67871 for ; Thu, 12 Jan 2023 14:32:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237447AbjALOcN (ORCPT ); Thu, 12 Jan 2023 09:32:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236847AbjALObW (ORCPT ); Thu, 12 Jan 2023 09:31:22 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C31FC5E09D for ; Thu, 12 Jan 2023 06:22:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673533348; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F/AVPewFoadGFAzU9enda46Kg7wI9ci0LM+wGYIMzIQ=; b=LhZcJRqXZCBwWT/FbJfF84KdkmKvfp8RsBLRYeWFTZSxIb1tjyp45ZzpDewjqj6HpUDflp CkRTLCOHWSZ7D7OK40k/UoPVt8lX7D1Vo8I5kopsDHaW41oDw+y/89TBY/XDb1PVrIDyJ1 37IzMCk1j/0yASei1a0n/rf9AgJrBlk= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-120-ji8upy0INVe9E7i5Mn-w2A-1; Thu, 12 Jan 2023 09:22:24 -0500 X-MC-Unique: ji8upy0INVe9E7i5Mn-w2A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 12C1E38041DB; Thu, 12 Jan 2023 14:22:24 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.192.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 719B52166B29; Thu, 12 Jan 2023 14:22:22 +0000 (UTC) From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: mst@redhat.com, elic@nvidia.com Cc: linux-kernel@vger.kernel.org, parav@nvidia.com, lulu@redhat.com, jasowang@redhat.com, virtualization@lists.linux-foundation.org, sgarzare@redhat.com, si-wei.liu@oracle.com Subject: [RFC 1/3] vdpa/mlx5: reset iotlb at dup_iotlb Date: Thu, 12 Jan 2023 15:22:16 +0100 Message-Id: <20230112142218.725622-2-eperezma@redhat.com> In-Reply-To: <20230112142218.725622-1-eperezma@redhat.com> References: <20230112142218.725622-1-eperezma@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Regular memory region changes already reset cvq iotlb at set_map. However this is not true if CVQ and data VQs are in different ASID. Clean the CVQ iotlb every time we hit dup_iotlb. Fixes: 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for contr= ol and data") Signed-off-by: Eugenio P=C3=A9rez --- drivers/vdpa/mlx5/core/mr.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 0a1e0b0dc37e..ae34dcac9a3f 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -456,6 +456,8 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, struc= t vhost_iotlb *src) u64 start =3D 0, last =3D ULLONG_MAX; int err; =20 + vhost_iotlb_reset(mvdev->cvq.iotlb); + if (!src) { err =3D vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOS= T_ACCESS_RW); return err; --=20 2.31.1 From nobody Mon Sep 15 11:12:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CD0FC54EBC for ; Thu, 12 Jan 2023 14:32:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237406AbjALOcH (ORCPT ); Thu, 12 Jan 2023 09:32:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238056AbjALObV (ORCPT ); Thu, 12 Jan 2023 09:31:21 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79D545933A for ; Thu, 12 Jan 2023 06:22:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673533350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LuOhWcvMHZUxEhlH6k+bJeMOhO9553Adrn/Panfxguk=; b=VdhL75XuOaaFToRZ+WCRPqCNHhSAXUoUltsD0RC2IFRQbM1K+4QbBIc38iOnt8aQJv5/yr vk2fAwieC8o2GW+rOhI2OaaXRWaWKsQe+0yC9JaUB0N0kgd/Xc3VgzR7YFIcZfNymtCPw2 ScKmWZiYT6/Nn0SE4ypD6RLhLMjHpHc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-55-z_941EakMCm6hFbNtIPxgg-1; Thu, 12 Jan 2023 09:22:26 -0500 X-MC-Unique: z_941EakMCm6hFbNtIPxgg-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 00080100F902; Thu, 12 Jan 2023 14:22:25 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.192.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C7162166B26; Thu, 12 Jan 2023 14:22:24 +0000 (UTC) From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: mst@redhat.com, elic@nvidia.com Cc: linux-kernel@vger.kernel.org, parav@nvidia.com, lulu@redhat.com, jasowang@redhat.com, virtualization@lists.linux-foundation.org, sgarzare@redhat.com, si-wei.liu@oracle.com Subject: [RFC 2/3] vdpa/mlx5: conditionally delete cvq iotlb in destroy_mr Date: Thu, 12 Jan 2023 15:22:17 +0100 Message-Id: <20230112142218.725622-3-eperezma@redhat.com> In-Reply-To: <20230112142218.725622-1-eperezma@redhat.com> References: <20230112142218.725622-1-eperezma@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org mlx5_vdpa_destroy_mr can be called by setting a map to data ASID after populating control virtqueue ASID iotlb. Control vq iotlb must not be cleared, since it will not be populated again. Adding a conditional in the function so the caller specifies if it is resetting, cleaning, or just changing data memory. Fixes: 8fcd20c30704 ("vdpa/mlx5: Support different address spaces for contr= ol and data") Signed-off-by: Eugenio P=C3=A9rez --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 +- drivers/vdpa/mlx5/core/mr.c | 5 +++-- drivers/vdpa/mlx5/net/mlx5_vnet.c | 12 ++++++------ 3 files changed, 10 insertions(+), 9 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/ml= x5_vdpa.h index 058fbe28107e..000b144019ec 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -119,7 +119,7 @@ int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvde= v, struct vhost_iotlb *io bool *change_map, unsigned int asid); int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *i= otlb, unsigned int asid); -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool delete_cvq_iot= lb); =20 #define mlx5_vdpa_warn(__dev, format, ...) = \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func= __, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index ae34dcac9a3f..878ee94efa78 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -491,7 +491,7 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvdev= , struct mlx5_vdpa_mr *mr } } =20 -void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) +void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev, bool delete_cvq_iot= lb) { struct mlx5_vdpa_mr *mr =3D &mvdev->mr; =20 @@ -499,7 +499,8 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) if (!mr->initialized) goto out; =20 - prune_iotlb(mvdev); + if (delete_cvq_iotlb) + prune_iotlb(mvdev); if (mr->user_mr) destroy_user_mr(mvdev, mr); else diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5= _vnet.c index 6632651b1e54..1f1f341f602b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2433,7 +2433,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev = *mvdev, goto err_mr; =20 teardown_driver(ndev); - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] =3D=3D= asid); err =3D mlx5_vdpa_create_mr(mvdev, iotlb, asid); if (err) goto err_mr; @@ -2449,7 +2449,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev = *mvdev, return 0; =20 err_setup: - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr(mvdev, mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] =3D=3D= asid); err_mr: return err; } @@ -2578,7 +2578,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *= vdev, u8 status) return; =20 err_setup: - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr(&ndev->mvdev, true); ndev->mvdev.status |=3D VIRTIO_CONFIG_S_FAILED; err_clear: up_write(&ndev->reslock); @@ -2604,7 +2604,7 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) down_write(&ndev->reslock); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr(&ndev->mvdev); + mlx5_vdpa_destroy_mr(&ndev->mvdev, true); ndev->mvdev.status =3D 0; ndev->cur_num_vqs =3D 0; ndev->mvdev.cvq.received_desc =3D 0; @@ -2691,7 +2691,7 @@ static void mlx5_vdpa_free(struct vdpa_device *vdev) ndev =3D to_mlx5_vdpa_ndev(mvdev); =20 free_resources(ndev); - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr(mvdev, true); if (!is_zero_ether_addr(ndev->config.mac)) { pfmdev =3D pci_get_drvdata(pci_physfn(mvdev->mdev->pdev)); mlx5_mpfs_del_mac(pfmdev, ndev->config.mac); @@ -3214,7 +3214,7 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_= mdev, const char *name, err_res2: free_resources(ndev); err_mr: - mlx5_vdpa_destroy_mr(mvdev); + mlx5_vdpa_destroy_mr(mvdev, true); err_res: mlx5_vdpa_free_resources(&ndev->mvdev); err_mpfs: --=20 2.31.1 From nobody Mon Sep 15 11:12:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1FDDC54EBD for ; Thu, 12 Jan 2023 14:32:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238250AbjALOcT (ORCPT ); Thu, 12 Jan 2023 09:32:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237137AbjALOb0 (ORCPT ); Thu, 12 Jan 2023 09:31:26 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84D0259300 for ; Thu, 12 Jan 2023 06:22:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673533352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xS+WhDCpVAXfoHuOOZG8UbA19f02GKhB5mYWzIxAKis=; b=YurhqiEPS1qkTpy6H3O7DLRyIhjLkKXd7k5H6m/CcStTNwF8pFglFlhCK39yyoRLh4JDpm Snr9jwKkTFJwtq9tU1aZyy4DobraJpDiS0JarnHqDh1pQVSZXu1zlaFhUoklU/x1cVqGd+ Y2FDhvwdKpDXc3SLe2gPbkN5ZKU4Qtc= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-450-MofelG4rOpKRCjq502kgtQ-1; Thu, 12 Jan 2023 09:22:28 -0500 X-MC-Unique: MofelG4rOpKRCjq502kgtQ-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.rdu2.redhat.com [10.11.54.6]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E1B4F2805599; Thu, 12 Jan 2023 14:22:27 +0000 (UTC) Received: from eperezma.remote.csb (unknown [10.39.192.72]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4975B2166B29; Thu, 12 Jan 2023 14:22:26 +0000 (UTC) From: =?UTF-8?q?Eugenio=20P=C3=A9rez?= To: mst@redhat.com, elic@nvidia.com Cc: linux-kernel@vger.kernel.org, parav@nvidia.com, lulu@redhat.com, jasowang@redhat.com, virtualization@lists.linux-foundation.org, sgarzare@redhat.com, si-wei.liu@oracle.com Subject: [RFC 3/3] vdpa/mlx5: take iommu_lock at dup_iotlb Date: Thu, 12 Jan 2023 15:22:18 +0100 Message-Id: <20230112142218.725622-4-eperezma@redhat.com> In-Reply-To: <20230112142218.725622-1-eperezma@redhat.com> References: <20230112142218.725622-1-eperezma@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.6 Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Both iommu changes and lookup are protected by mlx5_vdpa_net->reslock at this moment, but: * These iotlb changes / queries are not in the fast data path. * reslock belongs to netdev, while dup_iotlb seems generic. * It's located in a different file than the lock it needs to hold Justifies the lock acquisition. Fixes: 5262912ef3cf ("vdpa/mlx5: Add support for control VQ and MAC setting= ") Signed-off-by: Eugenio P=C3=A9rez --- drivers/vdpa/mlx5/core/mr.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 878ee94efa78..e9c8a7f8ee1d 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -454,13 +454,15 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, str= uct vhost_iotlb *src) { struct vhost_iotlb_map *map; u64 start =3D 0, last =3D ULLONG_MAX; - int err; + int err =3D 0; + + spin_lock(&mvdev->cvq.iommu_lock); =20 vhost_iotlb_reset(mvdev->cvq.iotlb); =20 if (!src) { err =3D vhost_iotlb_add_range(mvdev->cvq.iotlb, start, last, start, VHOS= T_ACCESS_RW); - return err; + goto out; } =20 for (map =3D vhost_iotlb_itree_first(src, start, last); map; @@ -468,9 +470,12 @@ static int dup_iotlb(struct mlx5_vdpa_dev *mvdev, stru= ct vhost_iotlb *src) err =3D vhost_iotlb_add_range(mvdev->cvq.iotlb, map->start, map->last, map->addr, map->perm); if (err) - return err; + goto out; } - return 0; + +out: + spin_unlock(&mvdev->cvq.iommu_lock); + return err; } =20 static void prune_iotlb(struct mlx5_vdpa_dev *mvdev) --=20 2.31.1