From nobody Wed Dec 17 06:10:03 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22D99C25B6B for ; Thu, 26 Oct 2023 07:17:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344495AbjJZHRy (ORCPT ); Thu, 26 Oct 2023 03:17:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344159AbjJZHRk (ORCPT ); Thu, 26 Oct 2023 03:17:40 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F90318D for ; Thu, 26 Oct 2023 00:17:38 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 39Q3efbu019277; Thu, 26 Oct 2023 07:17:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=XlIgyRieA91Md9S9/xgEZlQmQWyurhBSVRrrLd2gz0E=; b=YXE5XDADTRwpZkdZbXV9JzPtdaWchQv8EU4nP27SeK7NarnPwC/s4eUKjFMgfUEC/uIL fHBfnl54ZmkyIpBblmDzbuuIB0tTuEdCqBV0GRubZxUX9jRNE7ciI3pme7cagcR3U+cf LMQnD+L4hJKiZi/QdErThjrcllCy0nKaHBrYU5qfj5fEB5AHVigHGLQxj8VZ2m2gLlDl GqBzFFDvWRYqXBgJL6xh1eVoB+68fgJAyN8bObeDL7EE/e9NQsxhBcOkyK/+su60E0ch yDlL0M2KppY2Ycx/H7AhUBu9wHjiCsPBOIXyO7ixgntNbISC5jVP9ugWH7t+F8WbW/0e CA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tv68tj8t1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 26 Oct 2023 07:17:33 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39Q6tbAL015200; Thu, 26 Oct 2023 07:17:32 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTP id 3tv537t53r-7; Thu, 26 Oct 2023 07:17:32 +0000 From: Si-Wei Liu To: jasowang@redhat.com, mst@redhat.com, eperezma@redhat.com, sgarzare@redhat.com, dtatulea@nvidia.com Cc: virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 6/7] vdpa/mlx5: implement .reset_map driver op Date: Thu, 26 Oct 2023 00:14:39 -0700 Message-Id: <1698304480-18463-7-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1698304480-18463-1-git-send-email-si-wei.liu@oracle.com> References: <1698304480-18463-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-26_04,2023-10-25_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2310170001 definitions=main-2310260060 X-Proofpoint-ORIG-GUID: BHJUXJLEQ3LETPBUsG9ZNfDyrohSFrXd X-Proofpoint-GUID: BHJUXJLEQ3LETPBUsG9ZNfDyrohSFrXd Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Since commit 6f5312f80183 ("vdpa/mlx5: Add support for running with virtio_vdpa"), mlx5_vdpa starts with preallocate 1:1 DMA MR at device creation time. This 1:1 DMA MR will be implicitly destroyed while the first .set_map call is invoked, in which case callers like vhost-vdpa will start to set up custom mappings. When the .reset callback is invoked, the custom mappings will be cleared and the 1:1 DMA MR will be re-created. In order to reduce excessive memory mapping cost in live migration, it is desirable to decouple the vhost-vdpa IOTLB abstraction from the virtio device life cycle, i.e. mappings can be kept around intact across virtio device reset. Leverage the .reset_map callback, which is meant to destroy the regular MR (including cvq mapping) on the given ASID and recreate the initial DMA mapping. That way, the device .reset op runs free from having to maintain and clean up memory mappings by itself. Additionally, implement .compat_reset to cater for older userspace, which may wish to see mapping to be cleared during reset. Co-developed-by: Dragos Tatulea Signed-off-by: Dragos Tatulea Signed-off-by: Si-Wei Liu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 17 +++++++++++++++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 27 ++++++++++++++++++++++++--- 3 files changed, 42 insertions(+), 3 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/ml= x5_vdpa.h index db988ced5a5d..84547d998bcf 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -127,6 +127,7 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mv= dev, struct vhost_iotlb *iotlb, unsigned int asid); int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvdev); +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); =20 #define mlx5_vdpa_warn(__dev, format, ...) = \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func= __, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 66530e28f327..2197c46e563a 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -645,3 +645,20 @@ int mlx5_vdpa_create_dma_mr(struct mlx5_vdpa_dev *mvde= v) =20 return mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, 0); } + +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +{ + if (asid >=3D MLX5_VDPA_NUM_AS) + return -EINVAL; + + mlx5_vdpa_destroy_mr(mvdev, mvdev->mr[asid]); + + if (asid =3D=3D 0 && MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { + if (mlx5_vdpa_create_dma_mr(mvdev)) + mlx5_vdpa_warn(mvdev, "create DMA MR failed\n"); + } else { + mlx5_vdpa_update_cvq_iotlb(mvdev, NULL, asid); + } + + return 0; +} diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5= _vnet.c index f4516a2d5bb0..12ac3397f39b 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2876,7 +2876,7 @@ static void init_group_to_asid_map(struct mlx5_vdpa_d= ev *mvdev) mvdev->group2asid[i] =3D 0; } =20 -static int mlx5_vdpa_reset(struct vdpa_device *vdev) +static int mlx5_vdpa_compat_reset(struct vdpa_device *vdev, u32 flags) { struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev); struct mlx5_vdpa_net *ndev =3D to_mlx5_vdpa_ndev(mvdev); @@ -2888,7 +2888,8 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) unregister_link_notifier(ndev); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); + if (flags & VDPA_RESET_F_CLEAN_MAP) + mlx5_vdpa_destroy_mr_resources(&ndev->mvdev); ndev->mvdev.status =3D 0; ndev->mvdev.suspended =3D false; ndev->cur_num_vqs =3D 0; @@ -2899,7 +2900,8 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) init_group_to_asid_map(mvdev); ++mvdev->generation; =20 - if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { + if ((flags & VDPA_RESET_F_CLEAN_MAP) && + MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { if (mlx5_vdpa_create_dma_mr(mvdev)) mlx5_vdpa_warn(mvdev, "create MR failed\n"); } @@ -2908,6 +2910,11 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) return 0; } =20 +static int mlx5_vdpa_reset(struct vdpa_device *vdev) +{ + return mlx5_vdpa_compat_reset(vdev, 0); +} + static size_t mlx5_vdpa_get_config_size(struct vdpa_device *vdev) { return sizeof(struct virtio_net_config); @@ -2987,6 +2994,18 @@ static int mlx5_vdpa_set_map(struct vdpa_device *vde= v, unsigned int asid, return err; } =20 +static int mlx5_vdpa_reset_map(struct vdpa_device *vdev, unsigned int asid) +{ + struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev); + struct mlx5_vdpa_net *ndev =3D to_mlx5_vdpa_ndev(mvdev); + int err; + + down_write(&ndev->reslock); + err =3D mlx5_vdpa_reset_mr(mvdev, asid); + up_write(&ndev->reslock); + return err; +} + static struct device *mlx5_get_vq_dma_dev(struct vdpa_device *vdev, u16 id= x) { struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev); @@ -3250,11 +3269,13 @@ static const struct vdpa_config_ops mlx5_vdpa_ops = =3D { .get_status =3D mlx5_vdpa_get_status, .set_status =3D mlx5_vdpa_set_status, .reset =3D mlx5_vdpa_reset, + .compat_reset =3D mlx5_vdpa_compat_reset, .get_config_size =3D mlx5_vdpa_get_config_size, .get_config =3D mlx5_vdpa_get_config, .set_config =3D mlx5_vdpa_set_config, .get_generation =3D mlx5_vdpa_get_generation, .set_map =3D mlx5_vdpa_set_map, + .reset_map =3D mlx5_vdpa_reset_map, .set_group_asid =3D mlx5_set_group_asid, .get_vq_dma_dev =3D mlx5_get_vq_dma_dev, .free =3D mlx5_vdpa_free, --=20 2.39.3