From nobody Sun Sep 28 17:03:47 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1757471987; cv=none; d=zohomail.com; s=zohoarc; b=m+rtF73m3E7+KKHug1cVLCo1G+XAjCJhF6Az7ywsVXGoVmAjq0JhHQWdVVwRqXyX63Lm6oC3NK+slx3ZO6xlrW9ADDoEQ2+W452c0kyssQC2SgCr1Iab6uTGXLTRuAVLXiGdBaSa37vtC0L9/J6jBAUez4kp5BlPdcVS4ute9BE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757471987; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=aT1zRy9Tfyh9DmjKax35LCoRE4EN9fwkOxx8hsTZnBM=; b=fr1X5H2NmYjS/zSlfQFB7qNk//oDHD1oah2POyrpI6/8bWMekPslySvF1jb5IItaH5L0sP7EooRQFrRT+VFawdFW/C+4yavL7JHQEvB/SiHlmj3qi15S7V2gbtuFMJPqtgEGzelo+Y9rz4HeRbNEQ7Tzsn6tRhzZM6pGoX4B2tM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757471987176818.575367227245; Tue, 9 Sep 2025 19:39:47 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwAib-00070g-RK; Tue, 09 Sep 2025 22:37:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAiZ-0006zr-N4 for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:37:55 -0400 Received: from mgamail.intel.com ([192.198.163.13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAiO-0005fl-Pc for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:37:55 -0400 Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:34 -0700 Received: from unknown (HELO gnr-sp-2s-612.sh.intel.com) ([10.112.230.229]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:31 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1757471864; x=1789007864; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IDytYjxDTRyFHmWdI60FpD+xs5DrtvuV9RPonrF1C40=; b=P1qni6c1J0HA+zTmVqgw3W/ZMADQIaZx5tiMjL+epPcpQI4epCdSTb7s a3aGIEielW4tjVQkvEgxxVKMHvGRhIDVFO59Vnd/sMV8PyCxk/WY1ifvm IWbIQ4o0ZXEpJ3GGRyO7InrHGDJxm0z3/9hU/N4/0naXEbwX9skRhg0+e Ho5RNXyjZgIiEkBadgu761/ijsFbAhrUyBADib9fSfPbGxXasf2qnoJgK 65SCYYu9fjrk8tTkbqui5o7C5wVmvrJnBNUj1IREjfqcM/NVpWjrgHACF HHul58p+m0ix3abbk7nMBGMFt51Zsj2LfFkaKhgjQHqikA6JPFOCPRL2B w==; X-CSE-ConnectionGUID: is8t5F9WTQ2GOt19ufdP8Q== X-CSE-MsgGUID: OJSPUJ4RQCmvzozuYMsYXg== X-IronPort-AV: E=McAfee;i="6800,10657,11548"; a="62402788" X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="62402788" X-CSE-ConnectionGUID: emAKSSdbRiuTZK0Mr0FSkQ== X-CSE-MsgGUID: Pgv23ujORpGp4heygaemaw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="196930557" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, mst@redhat.com, jasowang@redhat.com, yi.l.liu@intel.com, clement.mathieu--drif@eviden.com, eric.auger@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, xudong.hao@intel.com, giovanni.cabiddu@intel.com, mark.gross@intel.com, arjan.van.de.ven@intel.com, Zhenzhong Duan Subject: [PATCH 1/5] vfio/iommufd: Add framework code to support getting dirty bitmap before unmap Date: Tue, 9 Sep 2025 22:36:57 -0400 Message-ID: <20250910023701.244356-2-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250910023701.244356-1-zhenzhong.duan@intel.com> References: <20250910023701.244356-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.198.163.13; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1757471990384124100 Content-Type: text/plain; charset="utf-8" Currently we support device and iommu dirty tracking, device dirty tracking is preferred. Add the framework code in iommufd_cdev_unmap_one() to choose either device or iommu dirty tracking, just like vfio_legacy_dma_unmap_one(). Signed-off-by: Zhenzhong Duan Tested-by: Xudong Hao Tested-by: Giovannio Cabiddu --- hw/vfio/iommufd.c | 56 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 45 insertions(+), 11 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 48c590b6a9..b5d6e54c45 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -58,33 +58,67 @@ static int iommufd_cdev_map_file(const VFIOContainerBas= e *bcontainer, iova, size, fd, start, readonly); } =20 -static int iommufd_cdev_unmap(const VFIOContainerBase *bcontainer, - hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb, bool unmap_all) +static int iommufd_cdev_unmap_one(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb) { const VFIOIOMMUFDContainer *container =3D container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); + bool need_dirty_sync =3D false; + Error *local_err =3D NULL; + IOMMUFDBackend *be =3D container->be; + uint32_t ioas_id =3D container->ioas_id; + int ret; + + if (iotlb && vfio_container_dirty_tracking_is_started(bcontainer)) { + if (!vfio_container_devices_dirty_tracking_is_supported(bcontainer= ) && + bcontainer->dirty_pages_supported) { + /* TODO: query dirty bitmap before DMA unmap */ + return iommufd_backend_unmap_dma(be, ioas_id, iova, size); + } + + need_dirty_sync =3D true; + } + + ret =3D iommufd_backend_unmap_dma(be, ioas_id, iova, size); + if (ret) { + return ret; + } =20 + if (need_dirty_sync) { + ret =3D vfio_container_query_dirty_bitmap(bcontainer, iova, size, + iotlb->translated_addr, + &local_err); + if (ret) { + error_report_err(local_err); + return ret; + } + } + + return 0; +} + +static int iommufd_cdev_unmap(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb, bool unmap_all) +{ /* unmap in halves */ if (unmap_all) { Int128 llsize =3D int128_rshift(int128_2_64(), 1); int ret; =20 - ret =3D iommufd_backend_unmap_dma(container->be, container->ioas_i= d, - 0, int128_get64(llsize)); + ret =3D iommufd_cdev_unmap_one(bcontainer, 0, int128_get64(llsize), + iotlb); =20 if (ret =3D=3D 0) { - ret =3D iommufd_backend_unmap_dma(container->be, container->io= as_id, - int128_get64(llsize), - int128_get64(llsize)); + ret =3D iommufd_cdev_unmap_one(bcontainer, int128_get64(llsize= ), + int128_get64(llsize), iotlb); } =20 return ret; } =20 - /* TODO: Handle dma_unmap_bitmap with iotlb args (migration) */ - return iommufd_backend_unmap_dma(container->be, - container->ioas_id, iova, size); + return iommufd_cdev_unmap_one(bcontainer, iova, size, iotlb); } =20 static bool iommufd_cdev_kvm_device_add(VFIODevice *vbasedev, Error **errp) --=20 2.47.1 From nobody Sun Sep 28 17:03:47 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1757471988; cv=none; d=zohomail.com; s=zohoarc; b=XpblvvcJPo49ss8jPMCZebrgaxLLmBtjPKgIMcELVUQzH8op4l8CjIz0yq7wx7tOhvahTIbzi+obfBFE/B8hqKLY2nFt4jBoLg9our2Uo7VrGtB2LoWr3X/Bgvd2VoYc8ywjiBSHwiHKlOvmn9vlWc192qPNEksADi38Z2fLipw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757471988; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=0ktra5W9Ln75xvJRQfliltuS8C1E/vosyRtFuDwRWZI=; b=JCYJ9vyBaS34jBgezCJBHMLmwh5npiZWs31Qs3u7FyCWzXFTC7lV5mMy7n7WIKd/24Xq+0D0VvVsGNRXxLTPnFgi9T5wNW1JtIT4VR6BL6HNSgcmReQnpC5XN1OUvF7drmm5Z1UEp+Ojps6e0D3Ybqc2ojA6fJQ68+UXNKCpcGk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757471988085307.53090916721555; Tue, 9 Sep 2025 19:39:48 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwAid-00070C-4I; Tue, 09 Sep 2025 22:37:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAiZ-0006zs-Ol for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:37:55 -0400 Received: from mgamail.intel.com ([192.198.163.13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAiP-0005gB-JW for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:37:55 -0400 Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:37 -0700 Received: from unknown (HELO gnr-sp-2s-612.sh.intel.com) ([10.112.230.229]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:34 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1757471865; x=1789007865; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mQiyyT422oTCwhFWFR51BUCTzu/XqXJsSl9tJ8GNNIU=; b=MLuI/CkepmyJsVwvu9SgzFcuiiKeXdBWhDdWMfEviCk9329ZUsp7TzbB BYw2t82WacZ3yLYmmrQkB2y75yW8Y5m2I4jFBXLn+k00V7MjHFJb2DKMa qXkBhLcufbIT2OWoN62/d4qxY1FSdGQiE6ZrTm66lM2eidhzimsqdLJJt qBmLjph0FrDbWJfvNm2byWoRt70RTcJD1QEDH36FJOPOGLZuPwiqEKuHl AVq2b9ybf8sxL8zZdJoRUrAKJhsIUC2vz2GemtBnFr8l42lUczygx+nGz E54Xj0NDSnreQEXQiFqFWKC0Go0gVggyKsExcotqgiN78H03Gqzc9rc4I Q==; X-CSE-ConnectionGUID: Gx91fnUqSvyeiXvKNk/y5w== X-CSE-MsgGUID: YG7Qi/RUSpKmK2Yfobdq7w== X-IronPort-AV: E=McAfee;i="6800,10657,11548"; a="62402795" X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="62402795" X-CSE-ConnectionGUID: 36X9HAdyTACBq/aJyKqQ2A== X-CSE-MsgGUID: XsxdeO6gQb+hBUZjF96CBQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="196930562" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, mst@redhat.com, jasowang@redhat.com, yi.l.liu@intel.com, clement.mathieu--drif@eviden.com, eric.auger@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, xudong.hao@intel.com, giovanni.cabiddu@intel.com, mark.gross@intel.com, arjan.van.de.ven@intel.com, Zhenzhong Duan Subject: [PATCH 2/5] vfio/iommufd: Query dirty bitmap before DMA unmap Date: Tue, 9 Sep 2025 22:36:58 -0400 Message-ID: <20250910023701.244356-3-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250910023701.244356-1-zhenzhong.duan@intel.com> References: <20250910023701.244356-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.198.163.13; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1757471991180116600 Content-Type: text/plain; charset="utf-8" When a existing mapping is unmapped, there could already be dirty bits which need to be recorded before unmap. If query dirty bitmap fails, we still need to do unmapping or else there is stale mapping and it's risky to guest. Co-developed-by: Joao Martins Signed-off-by: Joao Martins Signed-off-by: Zhenzhong Duan Tested-by: Xudong Hao Tested-by: Giovannio Cabiddu --- hw/vfio/iommufd.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index b5d6e54c45..0057488ce9 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -73,7 +73,13 @@ static int iommufd_cdev_unmap_one(const VFIOContainerBas= e *bcontainer, if (iotlb && vfio_container_dirty_tracking_is_started(bcontainer)) { if (!vfio_container_devices_dirty_tracking_is_supported(bcontainer= ) && bcontainer->dirty_pages_supported) { - /* TODO: query dirty bitmap before DMA unmap */ + ret =3D vfio_container_query_dirty_bitmap(bcontainer, iova, si= ze, + iotlb->translated_addr, + &local_err); + if (ret) { + error_report_err(local_err); + } + /* Unmap stale mapping even if query dirty bitmap fails */ return iommufd_backend_unmap_dma(be, ioas_id, iova, size); } =20 --=20 2.47.1 From nobody Sun Sep 28 17:03:47 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1757471989; cv=none; d=zohomail.com; s=zohoarc; b=nOMBJ+cYXSOxWdGHiqG91/rmW4sb9cKFlx40PLFGmQ9sd7D5W2K7P/sxbVeHDVBQSjKdZ5pQVyDKzqdOLykkILvVdplftQ5qFhNmY+E2BOn19Ikncs6Fioavhm8uKMunxs1+R0QrGcUmeMf2B8+yUL6cwiZEpp2xB6VL308tycU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757471989; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=r8dqxY/KlQFCEa6NCxmjAITFLTCeH4sWdzQwKOwyNqQ=; b=BvGD5BTRW4C3GCsYLPMtZ7TlWKJVa3KN5WFf5JurZrQeXYVYvpzQ8uc3tk0RQLWazzDXcKVXGlGpjb7vFuP8jRKx4PFdJaAtrnjnK4KRHjuqUOzx0kUrQ4HsXMwMwraz90Cq8zKTYd5ZJ3SNuSUYPIgVcYlhAdi6HkcJCHH3+zM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757471988679812.2398963084933; Tue, 9 Sep 2025 19:39:48 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwAih-00071I-44; Tue, 09 Sep 2025 22:38:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAie-000713-7R for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:38:00 -0400 Received: from mgamail.intel.com ([192.198.163.13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAib-0005fl-Gb for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:37:59 -0400 Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:41 -0700 Received: from unknown (HELO gnr-sp-2s-612.sh.intel.com) ([10.112.230.229]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:38 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1757471877; x=1789007877; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tdZgGAXGGAq4Pg6STXrJDzN4iFO8GejtPxlwh1FrhXU=; b=mtWVY+3wB27M4Xdzlw6H6NxrOvTAzM34YLqzCBFe6v1CJFdjKpIl0VdP LP3J3n6Z5G5lCOKW1K8RyNE/vgLt3qaznWLdLJlqUVZx0kOxZMTjqhiB7 hWfWnA/pzTcNIQBZ9XSVp92sTlcL/SbT/w2jYRfkBbamejG3j7PC9wHdg ngbDqqQ1AIrG248hZqp3vqfzTguV7ElZ1KVYLnsO1eGzuyRLzvA3ibP81 Y4MSifGoBaxvzcKd4PvZhjJuqk239cjiS5HIozgNLb7joyWwxDU2Y843o kK2lgG3f0mhVR5guu9DVYjb7zt9votlHbm0jxkgXab7W56uCkrt5uRFkl w==; X-CSE-ConnectionGUID: UidPoc1eRAWHfG13hRGZaw== X-CSE-MsgGUID: 4Nd5qdG0R8uBjfyKl2vEJg== X-IronPort-AV: E=McAfee;i="6800,10657,11548"; a="62402804" X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="62402804" X-CSE-ConnectionGUID: qIhbZV7lRfu8R4qQbjT/xw== X-CSE-MsgGUID: B5bGZjbETziHd9N5mWHLUw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="196930571" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, mst@redhat.com, jasowang@redhat.com, yi.l.liu@intel.com, clement.mathieu--drif@eviden.com, eric.auger@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, xudong.hao@intel.com, giovanni.cabiddu@intel.com, mark.gross@intel.com, arjan.van.de.ven@intel.com, Zhenzhong Duan Subject: [PATCH 3/5] vfio/iommufd: Add IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR flag support Date: Tue, 9 Sep 2025 22:36:59 -0400 Message-ID: <20250910023701.244356-4-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250910023701.244356-1-zhenzhong.duan@intel.com> References: <20250910023701.244356-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.198.163.13; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1757471991361116600 Content-Type: text/plain; charset="utf-8" Pass IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR when doing the last dirty bitmap query right before unmap, no PTEs flushes. This accelerates the query without issue because unmap will tear down the mapping anyway. Add a new element dirty_tracking_flags in VFIOIOMMUFDContainer to be used for the flags of iommufd dirty tracking. Currently it is set to either IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR or 0 based on the scenario. Co-developed-by: Joao Martins Signed-off-by: Joao Martins Signed-off-by: Zhenzhong Duan Tested-by: Xudong Hao Tested-by: Giovannio Cabiddu --- hw/vfio/vfio-iommufd.h | 1 + include/system/iommufd.h | 2 +- backends/iommufd.c | 5 +++-- hw/vfio/iommufd.c | 6 +++++- backends/trace-events | 2 +- 5 files changed, 11 insertions(+), 5 deletions(-) diff --git a/hw/vfio/vfio-iommufd.h b/hw/vfio/vfio-iommufd.h index 07ea0f4304..e0af241c75 100644 --- a/hw/vfio/vfio-iommufd.h +++ b/hw/vfio/vfio-iommufd.h @@ -26,6 +26,7 @@ typedef struct VFIOIOMMUFDContainer { VFIOContainerBase bcontainer; IOMMUFDBackend *be; uint32_t ioas_id; + uint64_t dirty_tracking_flags; QLIST_HEAD(, VFIOIOASHwpt) hwpt_list; } VFIOIOMMUFDContainer; =20 diff --git a/include/system/iommufd.h b/include/system/iommufd.h index c9c72ffc45..63898e7b0d 100644 --- a/include/system/iommufd.h +++ b/include/system/iommufd.h @@ -64,7 +64,7 @@ bool iommufd_backend_set_dirty_tracking(IOMMUFDBackend *b= e, uint32_t hwpt_id, bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *be, uint32_t hwpt_id, uint64_t iova, ram_addr_t size, uint64_t page_size, uint64_t *data, - Error **errp); + uint64_t flags, Error **errp); bool iommufd_backend_invalidate_cache(IOMMUFDBackend *be, uint32_t id, uint32_t data_type, uint32_t entry_l= en, uint32_t *entry_num, void *data, diff --git a/backends/iommufd.c b/backends/iommufd.c index 2a33c7ab0b..3c4f6157e2 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -361,7 +361,7 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend *b= e, uint32_t hwpt_id, uint64_t iova, ram_addr_t size, uint64_t page_size, uint64_t *data, - Error **errp) + uint64_t flags, Error **errp) { int ret; struct iommu_hwpt_get_dirty_bitmap get_dirty_bitmap =3D { @@ -371,11 +371,12 @@ bool iommufd_backend_get_dirty_bitmap(IOMMUFDBackend = *be, .length =3D size, .page_size =3D page_size, .data =3D (uintptr_t)data, + .flags =3D flags, }; =20 ret =3D ioctl(be->fd, IOMMU_HWPT_GET_DIRTY_BITMAP, &get_dirty_bitmap); trace_iommufd_backend_get_dirty_bitmap(be->fd, hwpt_id, iova, size, - page_size, ret ? errno : 0); + flags, page_size, ret ? errno := 0); if (ret) { error_setg_errno(errp, errno, "IOMMU_HWPT_GET_DIRTY_BITMAP (iova: 0x%"HWADDR_PR= Ix diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 0057488ce9..c897aa6b17 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -62,7 +62,7 @@ static int iommufd_cdev_unmap_one(const VFIOContainerBase= *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb) { - const VFIOIOMMUFDContainer *container =3D + VFIOIOMMUFDContainer *container =3D container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); bool need_dirty_sync =3D false; Error *local_err =3D NULL; @@ -73,9 +73,12 @@ static int iommufd_cdev_unmap_one(const VFIOContainerBas= e *bcontainer, if (iotlb && vfio_container_dirty_tracking_is_started(bcontainer)) { if (!vfio_container_devices_dirty_tracking_is_supported(bcontainer= ) && bcontainer->dirty_pages_supported) { + container->dirty_tracking_flags =3D + IOMMU_HWPT_GET_DIRTY_BITMAP_NO_CLEAR; ret =3D vfio_container_query_dirty_bitmap(bcontainer, iova, si= ze, iotlb->translated_addr, &local_err); + container->dirty_tracking_flags =3D 0; if (ret) { error_report_err(local_err); } @@ -248,6 +251,7 @@ static int iommufd_query_dirty_bitmap(const VFIOContain= erBase *bcontainer, if (!iommufd_backend_get_dirty_bitmap(container->be, hwpt->hwpt_id, iova, size, page_size, (uint64_t *)vbmap->bitmap, + container->dirty_tracking_fl= ags, errp)) { return -EINVAL; } diff --git a/backends/trace-events b/backends/trace-events index 56132d3fd2..e1992ba12f 100644 --- a/backends/trace-events +++ b/backends/trace-events @@ -19,5 +19,5 @@ iommufd_backend_alloc_ioas(int iommufd, uint32_t ioas) " = iommufd=3D%d ioas=3D%d" iommufd_backend_alloc_hwpt(int iommufd, uint32_t dev_id, uint32_t pt_id, u= int32_t flags, uint32_t hwpt_type, uint32_t len, uint64_t data_ptr, uint32_= t out_hwpt_id, int ret) " iommufd=3D%d dev_id=3D%u pt_id=3D%u flags=3D0x%x = hwpt_type=3D%u len=3D%u data_ptr=3D0x%"PRIx64" out_hwpt=3D%u (%d)" iommufd_backend_free_id(int iommufd, uint32_t id, int ret) " iommufd=3D%d = id=3D%d (%d)" iommufd_backend_set_dirty(int iommufd, uint32_t hwpt_id, bool start, int r= et) " iommufd=3D%d hwpt=3D%u enable=3D%d (%d)" -iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t i= ova, uint64_t size, uint64_t page_size, int ret) " iommufd=3D%d hwpt=3D%u i= ova=3D0x%"PRIx64" size=3D0x%"PRIx64" page_size=3D0x%"PRIx64" (%d)" +iommufd_backend_get_dirty_bitmap(int iommufd, uint32_t hwpt_id, uint64_t i= ova, uint64_t size, uint64_t flags, uint64_t page_size, int ret) " iommufd= =3D%d hwpt=3D%u iova=3D0x%"PRIx64" size=3D0x%"PRIx64" flags=3D0x%"PRIx64" p= age_size=3D0x%"PRIx64" (%d)" iommufd_backend_invalidate_cache(int iommufd, uint32_t id, uint32_t data_t= ype, uint32_t entry_len, uint32_t entry_num, uint32_t done_num, uint64_t da= ta_ptr, int ret) " iommufd=3D%d id=3D%u data_type=3D%u entry_len=3D%u entry= _num=3D%u done_num=3D%u data_ptr=3D0x%"PRIx64" (%d)" --=20 2.47.1 From nobody Sun Sep 28 17:03:47 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1757472008; cv=none; d=zohomail.com; s=zohoarc; b=NKH1M5+qtB/oWoRm9oopFe045f2/EtqDifFm8UNqKupanwAC0b0GvtwiFt6qzBnolvW0b190ASLYFpGwhWNHb2yLx9Fx0FUZAGJ1koDW0mhEhoRHE1u9bt2NPgQ1j2nL8dDgFuY+DV8G2qdWzZZReMd0C6rh4clVp5YIiIw6How= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757472008; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=N04iPNaFLP1vax0eKL9KloBAvyM2/IeXQIOFkzMnJvM=; b=dIgTLhhQe3hAnJTgJxH5iHxutsuLqT8JeOun2Y0A8VKKOpiV0iumF8+JKFXWFyA+nurQkkwaJy9YkSHtw4aDRn2bxf3Zvif6/hBXn589x60+q+z2Ftn8Va5lqEsYENPerIsQM+dhCLox1mjLUClzGNZDkPatfPvD+vYHMCeCg90= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757472008370913.6004144482491; Tue, 9 Sep 2025 19:40:08 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwAij-00072M-CL; Tue, 09 Sep 2025 22:38:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAih-00071g-CC for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:38:03 -0400 Received: from mgamail.intel.com ([192.198.163.13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAid-0005gB-Hb for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:38:02 -0400 Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:44 -0700 Received: from unknown (HELO gnr-sp-2s-612.sh.intel.com) ([10.112.230.229]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:41 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1757471880; x=1789007880; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Zu28Wx2gkxOgWLPOk59CCYz0qv1zGQ8gGNxyBTmRFXs=; b=IehuGJcF0PW3lCW/WpZlIb+JDbsp3BnXCVI0fMwCxumqy1wgQLghOWrc 9KjzyHSAYZaty63osWuP1I7WFbLvum/OS/idZ4IEutAEdQlv1W+BN3+zI /lpVmzNOAUh1c7S1EKOfVDIXjfalO+pprHcxLPvWI2TQcveVdFFv9LygP w56MYWxtQNKb+6CmumPs4WiEvikwCYM2oKcEVMIUpnplmRXuQGkIeJfUv 4AmMDwDfmfHnWi0elO02rkcplwe78yleRBD1OXu7It/Q9ahxQMJTqSUWN doq+1KNE4Ze5FnMV/TZXt5Ls6tl4FokRjZvT3NUIWp/jrysXXZeogszON g==; X-CSE-ConnectionGUID: SncOXWMWR/eKIwpodOfpDA== X-CSE-MsgGUID: WzewmLO4SeCdxXFJamIRhQ== X-IronPort-AV: E=McAfee;i="6800,10657,11548"; a="62402809" X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="62402809" X-CSE-ConnectionGUID: XAi2f26PQTeJ4IDTnLvINA== X-CSE-MsgGUID: nTqWyxooQhSqGBJUve8h1Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="196930581" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, mst@redhat.com, jasowang@redhat.com, yi.l.liu@intel.com, clement.mathieu--drif@eviden.com, eric.auger@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, xudong.hao@intel.com, giovanni.cabiddu@intel.com, mark.gross@intel.com, arjan.van.de.ven@intel.com, Zhenzhong Duan Subject: [PATCH 4/5] intel_iommu: Optimize unmap_bitmap during migration Date: Tue, 9 Sep 2025 22:37:00 -0400 Message-ID: <20250910023701.244356-5-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250910023701.244356-1-zhenzhong.duan@intel.com> References: <20250910023701.244356-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.198.163.13; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1757472009076116600 Content-Type: text/plain; charset="utf-8" If a VFIO device in guest switches from IOMMU domain to block domain, vtd_address_space_unmap() is called to unmap whole address space. If that happens during migration, migration fails with legacy VFIO backend as below: Status: failed (vfio_container_dma_unmap(0x561bbbd92d90, 0x100000000000, 0x= 100000000000) =3D -7 (Argument list too long)) Because legacy VFIO limits maximum bitmap size to 256MB which maps to 8TB on 4K page system, when 16TB sized UNMAP notification is sent, unmap_bitmap ioctl fails. There is no such limitation with iommufd backend, but it's still not optimal to allocate large bitmap. Optimize it by iterating over DMAMap list to unmap each range with active mapping when migration is active. If migration is not active, unmapping the whole address space in one go is optimal. Signed-off-by: Zhenzhong Duan Tested-by: Giovannio Cabiddu --- hw/i386/intel_iommu.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 83c5e44413..6876dae727 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -37,6 +37,7 @@ #include "system/system.h" #include "hw/i386/apic_internal.h" #include "kvm/kvm_i386.h" +#include "migration/misc.h" #include "migration/vmstate.h" #include "trace.h" =20 @@ -4423,6 +4424,42 @@ static void vtd_dev_unset_iommu_device(PCIBus *bus, = void *opaque, int devfn) vtd_iommu_unlock(s); } =20 +/* + * Unmapping a large range in one go is not optimal during migration becau= se + * a large dirty bitmap needs to be allocated while there may be only small + * mappings, iterate over DMAMap list to unmap each range with active mapp= ing. + */ +static void vtd_address_space_unmap_in_migration(VTDAddressSpace *as, + IOMMUNotifier *n) +{ + const DMAMap *map; + const DMAMap target =3D { + .iova =3D n->start, + .size =3D n->end, + }; + IOVATree *tree =3D as->iova_tree; + + /* + * DMAMap is created during IOMMU page table sync, it's either 4KB or = huge + * page size and always a power of 2 in size. So the range of DMAMap c= ould + * be used for UNMAP notification directly. + */ + while ((map =3D iova_tree_find(tree, &target))) { + IOMMUTLBEvent event; + + event.type =3D IOMMU_NOTIFIER_UNMAP; + event.entry.iova =3D map->iova; + event.entry.addr_mask =3D map->size; + event.entry.target_as =3D &address_space_memory; + event.entry.perm =3D IOMMU_NONE; + /* This field is meaningless for unmap */ + event.entry.translated_addr =3D 0; + memory_region_notify_iommu_one(n, &event); + + iova_tree_remove(tree, *map); + } +} + /* Unmap the whole range in the notifier's scope. */ static void vtd_address_space_unmap(VTDAddressSpace *as, IOMMUNotifier *n) { @@ -4432,6 +4469,11 @@ static void vtd_address_space_unmap(VTDAddressSpace = *as, IOMMUNotifier *n) IntelIOMMUState *s =3D as->iommu_state; DMAMap map; =20 + if (migration_is_running()) { + vtd_address_space_unmap_in_migration(as, n); + return; + } + /* * Note: all the codes in this function has a assumption that IOVA * bits are no more than VTD_MGAW bits (which is restricted by --=20 2.47.1 From nobody Sun Sep 28 17:03:47 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1757472010; cv=none; d=zohomail.com; s=zohoarc; b=K5If0LemSgfwh5morlKUsiUsE2PRHvbQolQCjDYVEkrebAf4RxxtyYJmUaPesMaqW2ZddBsQu/oGoWgpN5grJipNd4WWavPzeVx/S7u0OGRrpO/ZROzWiKS+ceyrD4ltPeIlqWAnBAiV7F4HLQFYKBeFuCSJ8Gwd3diBIIX0lpQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1757472010; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Tp6gFssqUASWPfVUTCHLhLaNW+NFWZ0cfncqkf8tLiA=; b=NhBkeZYSfYI+k+wjaXT18CFGKE2pDy8ekVgcL4xva6uf9v8US/sfg8NvbKgfQ+hMfAI2osHV7zNuvEbuYi3EBN9G5jNdTeXAsPKjNdIpTLOIQaIJSmsMFe9HX25W7hBjBrKldatvUg+vDFFpSBQLwGSwIbPowZg6HUOZhqWe4sw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1757472010331659.8466215904677; Tue, 9 Sep 2025 19:40:10 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1uwAin-00073H-46; Tue, 09 Sep 2025 22:38:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAil-00072p-Fo for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:38:07 -0400 Received: from mgamail.intel.com ([192.198.163.13]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1uwAif-0005fl-Rf for qemu-devel@nongnu.org; Tue, 09 Sep 2025 22:38:07 -0400 Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:48 -0700 Received: from unknown (HELO gnr-sp-2s-612.sh.intel.com) ([10.112.230.229]) by fmviesa002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2025 19:37:45 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1757471881; x=1789007881; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7WAwv9vUQFAur+V3PKq2isaFy9n/1SVHStpDpcXSul8=; b=Ekgn2Y9trF9qxDxud49mrHnLZE0gPhXMlqn66vdBweGEyL+mlgesO1Cp KTg6EvBi8ZSJ2XzK/LzczB24ygu6+wUAjlVgjfl+ZvM0T7/TcP5zWoMdy +Ipwtt0ZRRw0+V2sEP22XntC5ja76s/kBgre2+4QUOoH7RGV2RvOlpU/q aEp7mQDpqb8J39bXdelDR97hO4xV3BA8YWQYGbvyainCDK9mUyVKO5hn0 IlXVD4e/gkvLsa0pR5bKrJxPEzGd0cO7S9O7PiLweg88qt/d5WDMpGOYX 3C6X1lqrxwPrLETBYRuP+hycgsx4CFxSN//rkZ/F4+O3dKKROaigM31ul g==; X-CSE-ConnectionGUID: sX8wJ3wpRVSqum6pO8CnVQ== X-CSE-MsgGUID: 9mqAzyRBQfWATOmB9NYLIw== X-IronPort-AV: E=McAfee;i="6800,10657,11548"; a="62402818" X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="62402818" X-CSE-ConnectionGUID: 93xHmiL7S9auX+SLpuoK0A== X-CSE-MsgGUID: /cs5QtL4SNO1hZrywD0y6A== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.18,253,1751266800"; d="scan'208";a="196930595" From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, mst@redhat.com, jasowang@redhat.com, yi.l.liu@intel.com, clement.mathieu--drif@eviden.com, eric.auger@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, xudong.hao@intel.com, giovanni.cabiddu@intel.com, mark.gross@intel.com, arjan.van.de.ven@intel.com, Zhenzhong Duan , Jason Zeng Subject: [PATCH 5/5] vfio/migration: Allow live migration with vIOMMU without VFs using device dirty tracking Date: Tue, 9 Sep 2025 22:37:01 -0400 Message-ID: <20250910023701.244356-6-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250910023701.244356-1-zhenzhong.duan@intel.com> References: <20250910023701.244356-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=192.198.163.13; envelope-from=zhenzhong.duan@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_PASS=-0.001, T_SPF_HELO_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @intel.com) X-ZM-MESSAGEID: 1757472013137124100 Content-Type: text/plain; charset="utf-8" Commit e46883204c38 ("vfio/migration: Block migration with vIOMMU") introduces a migration blocker when vIOMMU is enabled, because we need to calculate the IOVA ranges for device dirty tracking. But this is unnecessary for iommu dirty tracking. Limit the vfio_viommu_preset() check to those devices which use device dirty tracking. This allows live migration with VFIO devices which use iommu dirty tracking. Introduce a helper vfio_device_dirty_pages_disabled() to facilicate it. Suggested-by: Jason Zeng Co-developed-by: Joao Martins Signed-off-by: Joao Martins Signed-off-by: Zhenzhong Duan Tested-by: Xudong Hao Tested-by: Giovannio Cabiddu --- include/hw/vfio/vfio-device.h | 10 ++++++++++ hw/vfio/container-base.c | 5 +---- hw/vfio/device.c | 6 ++++++ hw/vfio/migration.c | 6 +++--- 4 files changed, 20 insertions(+), 7 deletions(-) diff --git a/include/hw/vfio/vfio-device.h b/include/hw/vfio/vfio-device.h index 6e4d5ccdac..0c663a49d5 100644 --- a/include/hw/vfio/vfio-device.h +++ b/include/hw/vfio/vfio-device.h @@ -148,6 +148,16 @@ bool vfio_device_irq_set_signaling(VFIODevice *vbasede= v, int index, int subindex =20 void vfio_device_reset_handler(void *opaque); bool vfio_device_is_mdev(VFIODevice *vbasedev); +/** + * vfio_device_dirty_pages_disabled: Check if device dirty tracking will be + * used for a VFIO device + * + * @vbasedev: The VFIODevice to transform + * + * Return: true if either @vbasedev doesn't support device dirty tracking = or + * is forcedly disabled from command line, otherwise false. + */ +bool vfio_device_dirty_pages_disabled(VFIODevice *vbasedev); bool vfio_device_hiod_create_and_realize(VFIODevice *vbasedev, const char *typename, Error **err= p); bool vfio_device_attach(char *name, VFIODevice *vbasedev, diff --git a/hw/vfio/container-base.c b/hw/vfio/container-base.c index 56304978e1..39c88812f8 100644 --- a/hw/vfio/container-base.c +++ b/hw/vfio/container-base.c @@ -177,10 +177,7 @@ bool vfio_container_devices_dirty_tracking_is_supporte= d( VFIODevice *vbasedev; =20 QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { - if (vbasedev->device_dirty_page_tracking =3D=3D ON_OFF_AUTO_OFF) { - return false; - } - if (!vbasedev->dirty_pages_supported) { + if (vfio_device_dirty_pages_disabled(vbasedev)) { return false; } } diff --git a/hw/vfio/device.c b/hw/vfio/device.c index 52a1996dc4..d0975e0e40 100644 --- a/hw/vfio/device.c +++ b/hw/vfio/device.c @@ -400,6 +400,12 @@ bool vfio_device_is_mdev(VFIODevice *vbasedev) return subsys && (strcmp(subsys, "/sys/bus/mdev") =3D=3D 0); } =20 +bool vfio_device_dirty_pages_disabled(VFIODevice *vbasedev) +{ + return (!vbasedev->dirty_pages_supported || + vbasedev->device_dirty_page_tracking =3D=3D ON_OFF_AUTO_OFF); +} + bool vfio_device_hiod_create_and_realize(VFIODevice *vbasedev, const char *typename, Error **err= p) { diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 4c06e3db93..810a5cb157 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -1183,8 +1183,7 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Err= or **errp) return !vfio_block_migration(vbasedev, err, errp); } =20 - if ((!vbasedev->dirty_pages_supported || - vbasedev->device_dirty_page_tracking =3D=3D ON_OFF_AUTO_OFF) && + if (vfio_device_dirty_pages_disabled(vbasedev) && !vbasedev->iommu_dirty_tracking) { if (vbasedev->enable_migration =3D=3D ON_OFF_AUTO_AUTO) { error_setg(&err, @@ -1202,7 +1201,8 @@ bool vfio_migration_realize(VFIODevice *vbasedev, Err= or **errp) goto out_deinit; } =20 - if (vfio_viommu_preset(vbasedev)) { + if (!vfio_device_dirty_pages_disabled(vbasedev) && + vfio_viommu_preset(vbasedev)) { error_setg(&err, "%s: Migration is currently not supported " "with vIOMMU enabled", vbasedev->name); goto add_blocker; --=20 2.47.1