From nobody Thu Dec 18 07:50:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D9A5C04E69 for ; Tue, 15 Aug 2023 01:46:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234056AbjHOBqJ (ORCPT ); Mon, 14 Aug 2023 21:46:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234032AbjHOBpm (ORCPT ); Mon, 14 Aug 2023 21:45:42 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7E2EC1729 for ; Mon, 14 Aug 2023 18:45:41 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37EJOKZa021657; Tue, 15 Aug 2023 01:45:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=nLT8ivYqbPa/ni/5McWMzueAucXhDm8989/yEvkFPHY=; b=aXrdTwJy+yeO8HNT0VyeZdFNctBH7QegVEMO33XJ6IT3tOBOiEBgnGJVPeizym1cfTPA se5/G4OsZj/mPcXltkSiN8TsBC9FMy2CK3drdbROcuEA5xeTn5+Iyl7K0JTyqBmcALlQ PBtY0zmDA0c2egrqYL6WTC+V8oW7pXJEyxv/3Ha2lV1FxaQmpTvkLLfm+bYecHs3YcVS BRtQ7l7Cq0A7fsxsKgmlZN+WeTFrTPpWb3bdZGgOux9HvFv2avbmgzvx66BSyiAoRCUJ Cl5SZlAIXT6c+b8ZgvfLymo9WxqoOt6Rt4uqR3Pxo3EQIndpKGdfYYnsFYZtefR32oof NA== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3se2yfkwfk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:31 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37ENrmfi006642; Tue, 15 Aug 2023 01:45:30 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sey2ckehu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:30 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37F1jTK9011831; Tue, 15 Aug 2023 01:45:30 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3sey2ckehb-2; Tue, 15 Aug 2023 01:45:30 +0000 From: Si-Wei Liu To: jasowang@redhat.com Cc: eperezma@redhat.com, gal@nvidia.com, linux-kernel@vger.kernel.org, mst@redhat.com, si-wei.liu@oracle.com, virtualization@lists.linux-foundation.org, xuanzhuo@linux.alibaba.com Subject: [PATCH RFC 1/4] vdpa: introduce .reset_map operation callback Date: Mon, 14 Aug 2023 18:43:24 -0700 Message-Id: <1692063807-5018-2-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> References: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-08-15_01,2023-08-10_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 adultscore=0 spamscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308150013 X-Proofpoint-ORIG-GUID: PdXkjGB6HasQq-Gd8GSCExarDykM-Q2G X-Proofpoint-GUID: PdXkjGB6HasQq-Gd8GSCExarDykM-Q2G Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Si-Wei Liu --- include/linux/vdpa.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index db1b0ea..3a3878d 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -314,6 +314,12 @@ struct vdpa_map_file { * @iova: iova to be unmapped * @size: size of the area * Returns integer: success (0) or error (< 0) + * @reset_map: Reset device memory mapping (optional) + * Needed for device that using device + * specific DMA translation (on-chip IOMMU) + * @vdev: vdpa device + * @asid: address space identifier + * Returns integer: success (0) or error (< 0) * @get_vq_dma_dev: Get the dma device for a specific * virtqueue (optional) * @vdev: vdpa device @@ -390,6 +396,7 @@ struct vdpa_config_ops { u64 iova, u64 size, u64 pa, u32 perm, void *opaque); int (*dma_unmap)(struct vdpa_device *vdev, unsigned int asid, u64 iova, u64 size); + int (*reset_map)(struct vdpa_device *vdev, unsigned int asid); int (*set_group_asid)(struct vdpa_device *vdev, unsigned int group, unsigned int asid); struct device *(*get_vq_dma_dev)(struct vdpa_device *vdev, u16 idx); --=20 1.8.3.1 From nobody Thu Dec 18 07:50:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9392C001E0 for ; Tue, 15 Aug 2023 01:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234007AbjHOBqF (ORCPT ); Mon, 14 Aug 2023 21:46:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234030AbjHOBpm (ORCPT ); Mon, 14 Aug 2023 21:45:42 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E78C51B2 for ; Mon, 14 Aug 2023 18:45:39 -0700 (PDT) Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37EJOwWk015360; Tue, 15 Aug 2023 01:45:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=ibbe3IEzpkg/bmF2R++TxgWBCqasnu95SyHMSrDgL+E=; b=r/lBNgg4JdrEPjKXss9dB0zS9XP+jNmXTT7BzDmH3v03vvNnu9W1TDM8dKk7yYXdNmLR CRQub5Z2yHRhSB119K5HMn7t+ZHbXH5A+d99tJ9jz7SfSXsyG+NRr8vaPFvVke1cRa4/ pOQJgzPqRtQZ07QzeBtUCRIbM9ohitMorAdgrknqhmpG0YrJWPfftQyFLnBKDoomupMS AuNNYjepmkv1PRqgc9EAILirXe62DMKwsetM44UUQL9g4rpSBXzBXPOjE4HfpCY9kGs7 Jw/8vhU1OeuPeyTGzIXB6WrbTozP+Cnb3zDe2wX9kl+XrP7nH1/Xz583TI/HHDZ9qlry Aw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3se2xwkw3v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:32 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37ENwIAI006639; Tue, 15 Aug 2023 01:45:31 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sey2ckej3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:31 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37F1jTKB011831; Tue, 15 Aug 2023 01:45:30 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3sey2ckehb-3; Tue, 15 Aug 2023 01:45:30 +0000 From: Si-Wei Liu To: jasowang@redhat.com Cc: eperezma@redhat.com, gal@nvidia.com, linux-kernel@vger.kernel.org, mst@redhat.com, si-wei.liu@oracle.com, virtualization@lists.linux-foundation.org, xuanzhuo@linux.alibaba.com Subject: [PATCH RFC 2/4] vdpa/mlx5: implement .reset_map driver op Date: Mon, 14 Aug 2023 18:43:25 -0700 Message-Id: <1692063807-5018-3-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> References: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-08-15_01,2023-08-10_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 adultscore=0 spamscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308150013 X-Proofpoint-ORIG-GUID: Rsj_cxSGT8Sqbjc4kcPZGnM6JlCrM26h X-Proofpoint-GUID: Rsj_cxSGT8Sqbjc4kcPZGnM6JlCrM26h Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This patch is based on top of the "vdpa/mlx5: Fixes for ASID handling" series [1]. [1] vdpa/mlx5: Fixes for ASID handling https://lore.kernel.org/virtualization/20230802171231.11001-1-dtatulea@nvid= ia.com/ Signed-off-by: Si-Wei Liu --- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 1 + drivers/vdpa/mlx5/core/mr.c | 72 +++++++++++++++++++++-------------= ---- drivers/vdpa/mlx5/net/mlx5_vnet.c | 18 +++++++--- 3 files changed, 54 insertions(+), 37 deletions(-) diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/ml= x5_vdpa.h index b53420e..5c9a25a 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -123,6 +123,7 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, st= ruct vhost_iotlb *iotlb, unsigned int asid); void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev); void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *mvdev, unsigned int a= sid); +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid); =20 #define mlx5_vdpa_warn(__dev, format, ...) = \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func= __, __LINE__, \ diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c index 5a1971fc..c8d64fc 100644 --- a/drivers/vdpa/mlx5/core/mr.c +++ b/drivers/vdpa/mlx5/core/mr.c @@ -489,21 +489,15 @@ static void destroy_user_mr(struct mlx5_vdpa_dev *mvd= ev, struct mlx5_vdpa_mr *mr } } =20 -static void _mlx5_vdpa_destroy_cvq_mr(struct mlx5_vdpa_dev *mvdev, unsigne= d int asid) +static void _mlx5_vdpa_destroy_cvq_mr(struct mlx5_vdpa_dev *mvdev) { - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] !=3D asid) - return; - prune_iotlb(mvdev); } =20 -static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev, unsigne= d int asid) +static void _mlx5_vdpa_destroy_dvq_mr(struct mlx5_vdpa_dev *mvdev) { struct mlx5_vdpa_mr *mr =3D &mvdev->mr; =20 - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] !=3D asid) - return; - if (!mr->initialized) return; =20 @@ -521,8 +515,10 @@ void mlx5_vdpa_destroy_mr_asid(struct mlx5_vdpa_dev *m= vdev, unsigned int asid) =20 mutex_lock(&mr->mkey_mtx); =20 - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - _mlx5_vdpa_destroy_cvq_mr(mvdev, asid); + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] =3D=3D asid) + _mlx5_vdpa_destroy_dvq_mr(mvdev); + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] =3D=3D asid) + _mlx5_vdpa_destroy_cvq_mr(mvdev); =20 mutex_unlock(&mr->mkey_mtx); } @@ -534,25 +530,17 @@ void mlx5_vdpa_destroy_mr(struct mlx5_vdpa_dev *mvdev) } =20 static int _mlx5_vdpa_create_cvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) + struct vhost_iotlb *iotlb) { - if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] !=3D asid) - return 0; - return dup_iotlb(mvdev, iotlb); } =20 static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev, - struct vhost_iotlb *iotlb, - unsigned int asid) + struct vhost_iotlb *iotlb) { struct mlx5_vdpa_mr *mr =3D &mvdev->mr; int err; =20 - if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] !=3D asid) - return 0; - if (mr->initialized) return 0; =20 @@ -574,20 +562,18 @@ static int _mlx5_vdpa_create_mr(struct mlx5_vdpa_dev = *mvdev, { int err; =20 - err =3D _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); - if (err) - return err; - - err =3D _mlx5_vdpa_create_cvq_mr(mvdev, iotlb, asid); - if (err) - goto out_err; + if (mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP] =3D=3D asid) { + err =3D _mlx5_vdpa_create_dvq_mr(mvdev, iotlb, asid); + if (err) + return err; + } + if (mvdev->group2asid[MLX5_VDPA_CVQ_GROUP] =3D=3D asid) { + err =3D _mlx5_vdpa_create_cvq_mr(mvdev, iotlb); + if (err) + return err; + } =20 return 0; - -out_err: - _mlx5_vdpa_destroy_dvq_mr(mvdev, asid); - - return err; } =20 int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *i= otlb, @@ -601,6 +587,28 @@ int mlx5_vdpa_create_mr(struct mlx5_vdpa_dev *mvdev, s= truct vhost_iotlb *iotlb, return err; } =20 +int mlx5_vdpa_reset_mr(struct mlx5_vdpa_dev *mvdev, unsigned int asid) +{ + struct mlx5_vdpa_mr *mr =3D &mvdev->mr; + int err =3D 0; + + if (asid !=3D 0) + return 0; + + mutex_lock(&mr->mkey_mtx); + if (!mr->user_mr) + goto out; + _mlx5_vdpa_destroy_dvq_mr(mvdev); + if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { + err =3D _mlx5_vdpa_create_dvq_mr(mvdev, NULL, 0); + if (err) + mlx5_vdpa_warn(mvdev, "create DMA MR failed\n"); + } +out: + mutex_unlock(&mr->mkey_mtx); + return err; +} + int mlx5_vdpa_handle_set_map(struct mlx5_vdpa_dev *mvdev, struct vhost_iot= lb *iotlb, bool *change_map, unsigned int asid) { diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5= _vnet.c index 61c10ba..399a690 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -2816,7 +2816,6 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) unregister_link_notifier(ndev); teardown_driver(ndev); clear_vqs_ready(ndev); - mlx5_vdpa_destroy_mr(&ndev->mvdev); ndev->mvdev.status =3D 0; ndev->mvdev.suspended =3D false; ndev->cur_num_vqs =3D 0; @@ -2827,10 +2826,6 @@ static int mlx5_vdpa_reset(struct vdpa_device *vdev) init_group_to_asid_map(mvdev); ++mvdev->generation; =20 - if (MLX5_CAP_GEN(mvdev->mdev, umem_uid_0)) { - if (mlx5_vdpa_create_mr(mvdev, NULL, 0)) - mlx5_vdpa_warn(mvdev, "create MR failed\n"); - } up_write(&ndev->reslock); =20 return 0; @@ -2895,6 +2890,18 @@ static int mlx5_vdpa_set_map(struct vdpa_device *vde= v, unsigned int asid, return err; } =20 +static int mlx5_vdpa_reset_map(struct vdpa_device *vdev, unsigned int asid) +{ + struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev); + struct mlx5_vdpa_net *ndev =3D to_mlx5_vdpa_ndev(mvdev); + int err; + + down_write(&ndev->reslock); + err =3D mlx5_vdpa_reset_mr(mvdev, asid); + up_write(&ndev->reslock); + return err; +} + static struct device *mlx5_get_vq_dma_dev(struct vdpa_device *vdev, u16 id= x) { struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev); @@ -3154,6 +3161,7 @@ static int mlx5_set_group_asid(struct vdpa_device *vd= ev, u32 group, .set_config =3D mlx5_vdpa_set_config, .get_generation =3D mlx5_vdpa_get_generation, .set_map =3D mlx5_vdpa_set_map, + .reset_map =3D mlx5_vdpa_reset_map, .set_group_asid =3D mlx5_set_group_asid, .get_vq_dma_dev =3D mlx5_get_vq_dma_dev, .free =3D mlx5_vdpa_free, --=20 1.8.3.1 From nobody Thu Dec 18 07:50:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9BF1C41513 for ; Tue, 15 Aug 2023 01:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233992AbjHOBqD (ORCPT ); Mon, 14 Aug 2023 21:46:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234031AbjHOBpm (ORCPT ); Mon, 14 Aug 2023 21:45:42 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3F9321736 for ; Mon, 14 Aug 2023 18:45:41 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37EJOkBi004848; Tue, 15 Aug 2023 01:45:32 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=OORoP4T2YEFFgRyKnOBujnSlwX+77mpB31iRz09hv1E=; b=N9ywFWQO7PCLuehItKulZ1kwGFyOCVP7peZWO2tK05xBqYB6UqjtekU0gCD3Mf6/35Dw Dtsr8m6QrD6iasLarytiSwWmW/WJuArlgGulRg5Qhhvxd+9mZu8VSfJQZGOKGRUkKJwA JSZtOJyS7iQO9ZER5kVzOHsNV/fNj7taExR+0A3aRxusFgPUg4UL6RSlSgKWltwpb6L6 E23LQIysLLcM3WIrgHcfsgN7h22KUoPZWgCoKAtcC7oekWNnfQCDI/qfdxh6zbn0/ca3 rPcgotVzLOhv9gvZWKdUvcnbigB22kxhRYGiz6uzyxOTTne0Z5IQNC29J3xh+ZHnx3Dw yw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3se349bx00-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:32 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37ENVNaq005498; Tue, 15 Aug 2023 01:45:31 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sey2ckejc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:31 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37F1jTKD011831; Tue, 15 Aug 2023 01:45:31 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3sey2ckehb-4; Tue, 15 Aug 2023 01:45:31 +0000 From: Si-Wei Liu To: jasowang@redhat.com Cc: eperezma@redhat.com, gal@nvidia.com, linux-kernel@vger.kernel.org, mst@redhat.com, si-wei.liu@oracle.com, virtualization@lists.linux-foundation.org, xuanzhuo@linux.alibaba.com Subject: [PATCH RFC 3/4] vhost-vdpa: should restore 1:1 dma mapping before detaching driver Date: Mon, 14 Aug 2023 18:43:26 -0700 Message-Id: <1692063807-5018-4-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> References: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-08-15_01,2023-08-10_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 adultscore=0 spamscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308150013 X-Proofpoint-GUID: DrfwY-3wagKT8OcSBAb3meBPXhlpWGWn X-Proofpoint-ORIG-GUID: DrfwY-3wagKT8OcSBAb3meBPXhlpWGWn Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Si-Wei Liu --- drivers/vhost/vdpa.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index b43e868..62b0a01 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -131,6 +131,15 @@ static struct vhost_vdpa_as *vhost_vdpa_find_alloc_as(= struct vhost_vdpa *v, return vhost_vdpa_alloc_as(v, asid); } =20 +static void vhost_vdpa_reset_map(struct vhost_vdpa *v, u32 asid) +{ + struct vdpa_device *vdpa =3D v->vdpa; + const struct vdpa_config_ops *ops =3D vdpa->config; + + if (ops->reset_map) + ops->reset_map(vdpa, asid); +} + static int vhost_vdpa_remove_as(struct vhost_vdpa *v, u32 asid) { struct vhost_vdpa_as *as =3D asid_to_as(v, asid); @@ -140,6 +149,14 @@ static int vhost_vdpa_remove_as(struct vhost_vdpa *v, = u32 asid) =20 hlist_del(&as->hash_link); vhost_vdpa_iotlb_unmap(v, &as->iotlb, 0ULL, 0ULL - 1, asid); + /* + * Devices with on-chip IOMMU need to restore iotlb + * to 1:1 identity mapping before vhost-vdpa is going + * to be removed and detached from the device. Give + * them a chance to do so, as this cannot be done + * efficiently via the whole-range unmap call above. + */ + vhost_vdpa_reset_map(v, asid); kfree(as); =20 return 0; --=20 1.8.3.1 From nobody Thu Dec 18 07:50:39 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3E13C04A6A for ; Tue, 15 Aug 2023 01:46:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233967AbjHOBqB (ORCPT ); Mon, 14 Aug 2023 21:46:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234023AbjHOBpl (ORCPT ); Mon, 14 Aug 2023 21:45:41 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0D10120 for ; Mon, 14 Aug 2023 18:45:39 -0700 (PDT) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 37EJOsLs023643; Tue, 15 Aug 2023 01:45:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2023-03-30; bh=GmzgeWelKQolP+WJ3LzphyiaUevasb/FBdXmsCIpQxQ=; b=svb6UsQlDcyzKHQWQBOUwDekQZBvWah2JVajwcrnuSsg3NJZI95lC5oGhmaWoLp9KLbk AWdP7eUsDNyp1b38QDncnP8/g5cCCoLTZLGoVjPFxiYeG1QKow9HQUKgpWOdrJgd3lyS sVW6/WNkMQ1iPriL9ZGK1xJYlwsghXTaKT5tTFHy7RLvagdJyzVNRQHKiydSCL6pJycx 8H+qZkksAXQN/RQlIb9ydc4vD/LQJ+huLqwbKPPlGvXUsGwcKBy/IFFTjcO1a512Cy+x xLEjfvQMkcvjnRn1uSCa0UxSAKNxMVF/lxm/S582TKxBX++huAZPyVJzWXKWB99sw1bC Nw== Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.appoci.oracle.com [130.35.100.223]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3se2w5uusv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:33 +0000 Received: from pps.filterd (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 37ENE9HV005491; Tue, 15 Aug 2023 01:45:32 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3sey2ckejq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 15 Aug 2023 01:45:32 +0000 Received: from iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 37F1jTKF011831; Tue, 15 Aug 2023 01:45:31 GMT Received: from ban25x6uut24.us.oracle.com (ban25x6uut24.us.oracle.com [10.153.73.24]) by iadpaimrmta01.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3sey2ckehb-5; Tue, 15 Aug 2023 01:45:31 +0000 From: Si-Wei Liu To: jasowang@redhat.com Cc: eperezma@redhat.com, gal@nvidia.com, linux-kernel@vger.kernel.org, mst@redhat.com, si-wei.liu@oracle.com, virtualization@lists.linux-foundation.org, xuanzhuo@linux.alibaba.com Subject: [PATCH RFC 4/4] vhost-vdpa: introduce IOTLB_PERSIST backend feature bit Date: Mon, 14 Aug 2023 18:43:27 -0700 Message-Id: <1692063807-5018-5-git-send-email-si-wei.liu@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> References: <1692063807-5018-1-git-send-email-si-wei.liu@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26 definitions=2023-08-15_01,2023-08-10_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 adultscore=0 spamscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2306200000 definitions=main-2308150013 X-Proofpoint-ORIG-GUID: AfewiI6wC-SYHxFKlpzUEzQILf1BjsDv X-Proofpoint-GUID: AfewiI6wC-SYHxFKlpzUEzQILf1BjsDv Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Signed-off-by: Si-Wei Liu --- drivers/vhost/vdpa.c | 16 +++++++++++++++- include/uapi/linux/vhost_types.h | 2 ++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index 62b0a01..75092a7 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -406,6 +406,14 @@ static bool vhost_vdpa_can_resume(const struct vhost_v= dpa *v) return ops->resume; } =20 +static bool vhost_vdpa_has_persistent_map(const struct vhost_vdpa *v) +{ + struct vdpa_device *vdpa =3D v->vdpa; + const struct vdpa_config_ops *ops =3D vdpa->config; + + return (!ops->set_map && !ops->dma_map) || ops->reset_map; +} + static long vhost_vdpa_get_features(struct vhost_vdpa *v, u64 __user *feat= urep) { struct vdpa_device *vdpa =3D v->vdpa; @@ -697,7 +705,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *file= p, return -EFAULT; if (features & ~(VHOST_VDPA_BACKEND_FEATURES | BIT_ULL(VHOST_BACKEND_F_SUSPEND) | - BIT_ULL(VHOST_BACKEND_F_RESUME))) + BIT_ULL(VHOST_BACKEND_F_RESUME) | + BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST))) return -EOPNOTSUPP; if ((features & BIT_ULL(VHOST_BACKEND_F_SUSPEND)) && !vhost_vdpa_can_suspend(v)) @@ -705,6 +714,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *file= p, if ((features & BIT_ULL(VHOST_BACKEND_F_RESUME)) && !vhost_vdpa_can_resume(v)) return -EOPNOTSUPP; + if ((features & BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST)) && + !vhost_vdpa_has_persistent_map(v)) + return -EOPNOTSUPP; vhost_set_backend_features(&v->vdev, features); return 0; } @@ -758,6 +770,8 @@ static long vhost_vdpa_unlocked_ioctl(struct file *file= p, features |=3D BIT_ULL(VHOST_BACKEND_F_SUSPEND); if (vhost_vdpa_can_resume(v)) features |=3D BIT_ULL(VHOST_BACKEND_F_RESUME); + if (vhost_vdpa_has_persistent_map(v)) + features |=3D BIT_ULL(VHOST_BACKEND_F_IOTLB_PERSIST); if (copy_to_user(featurep, &features, sizeof(features))) r =3D -EFAULT; break; diff --git a/include/uapi/linux/vhost_types.h b/include/uapi/linux/vhost_ty= pes.h index d3aad12a..a7f8fc6 100644 --- a/include/uapi/linux/vhost_types.h +++ b/include/uapi/linux/vhost_types.h @@ -181,5 +181,7 @@ struct vhost_vdpa_iova_range { #define VHOST_BACKEND_F_SUSPEND 0x4 /* Device can be resumed */ #define VHOST_BACKEND_F_RESUME 0x5 +/* IOTLB don't flush memory mapping across device reset */ +#define VHOST_BACKEND_F_IOTLB_PERSIST 0x6 =20 #endif --=20 1.8.3.1