From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759025; cv=pass; d=zohomail.com; s=zohoarc; b=RB/uesMKpIwY41lQhOAEgwGlQRO25f8sm7PDwbH38aynb0mBh8K9+Y7Ziwj2yu2fHz97ilo7Ersn4bUAzzJchgKZfvFDIm62HbEjJa2Eq+3+aWFuRSF/eVx7mWb6qtHg1qxNT5jO+/ScX/8eNE1c0rnprUkXfylyHpP4cJNoLvw= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759025; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=E/pI6QIyQ/Go4ZOjqy/UhuGjYejwDocw1PpsfkeoGVA=; b=mZ+J6wSeN0Y1GKKWGcTX9ATWJ3BNWIDvuuAMMb9JucvOTgz4LrUGv2olkHROt7Lb+ZXSR0aOOf4tnVNXPn2zWkjAnt5AmPUkoXF/cDQKATpmcKk9j0Am//mYZJ6LQg83NRfZ2wo4n0M/RtCX1hfoYSJSMqKpak45W+5xNeWloPk= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 16747590259521020.1768707083116; Thu, 26 Jan 2023 10:50:25 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KE-0004Oo-4b; Thu, 26 Jan 2023 13:50:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7K5-0004NK-5m for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:09 -0500 Received: from mail-dm6nam10on2062c.outbound.protection.outlook.com ([2a01:111:f400:7e88::62c] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7K1-0002OL-AN for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:08 -0500 Received: from MW4PR04CA0101.namprd04.prod.outlook.com (2603:10b6:303:83::16) by SA1PR12MB6726.namprd12.prod.outlook.com (2603:10b6:806:255::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Thu, 26 Jan 2023 18:49:59 +0000 Received: from CO1NAM11FT100.eop-nam11.prod.protection.outlook.com (2603:10b6:303:83:cafe::3d) by MW4PR04CA0101.outlook.office365.com (2603:10b6:303:83::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33 via Frontend Transport; Thu, 26 Jan 2023 18:49:59 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT100.mail.protection.outlook.com (10.13.175.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:49:59 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:49:58 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:49:57 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:49:53 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xj4WUdXyWSJ6nmjFFmSk0irlU0JGwip5VG5PIldqAAUo7XcxsZM1JX1lwvIkFDVgx7CR8Wkpg5dbzCDNa4sc2My+f/PPk3MNwLsmcWd4ttNwAQJvAyk5sTsrHOPgH6fRvIfFO+r/ExzBRGzNaqWAPUSxdx83aoRYd5uqLHIPtWtEmJ2uMJYQwhMpUp/lbPsTtlH+8CZBxBQgYKm9O+7acY+uLYSGBAsdLlm+jrQFprLT0Fv5YHlYtP6X0qAMU+/wgFfQhgj6jKcOQ+Og7r5AN7c4zzEJtlePZ7gv1vQKItUAUNcvbUukF16hGsod0NrV4PaiFw7jYx2uvRKGkMC/kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=E/pI6QIyQ/Go4ZOjqy/UhuGjYejwDocw1PpsfkeoGVA=; b=UviSQts90HpMwgfZzsEo4bHDNyf3YcDtfhngs4jutHGuzAI8XlTf0xcJqqjNXRYGFJ+ZDtXmuEua8cD1hS6kPQyT5yufMnHW1axuf7P2mNM18Nbw7sNthnaQNG2fbVyDd32xA8zqEmgbkm/6L/aD8S1C+lsaRWUmzhmxQluBsirbk1O/1S043uFDLduANScxxebyYSYFxQKH/0+EAkyCXpoBcIm8hQvtFEzGZqIc4FxteOYJv9z0CoofLuxdiNeoRpEd0KeT8uKn6HB9wgAUrPaUwIEn8qaP75qGeC75+mLoy54wQkD6Xph9yPIGz9F/kJppqh3WRBNB+3HcHB7bLQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E/pI6QIyQ/Go4ZOjqy/UhuGjYejwDocw1PpsfkeoGVA=; b=AUBnZ9S84PSvMLhjd4fQbLv9jIP35J8sdhqVNpj5AnCW6/1VVLKdaGdl43sWUTA5sCEEFNoLePiA/dsY0t/e8xvTKlVzcwBN3p57/U2Va1lztotESmmqXQG52KcRLKUa2y/4EIz17zKqYXaJx2sAq4IVZThuRiOOj59Dn1t1nNnyYtVx4HVNR2srA+nKfaKKSLecBmloK0EbMTZlnrf61OV6bSzmHSFfXD+8WkYWjAKFoVSqA9XguVMlJNMCSv/oiKBVUNF9+XweAMfS6DbL0BckVC6lRq9662alZAu/PNU243DAFgrTODE6q4jcx9PeCVNMVeKmnTCPB8DextERPQ== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 01/18] vfio/migration: Add VFIO migration pre-copy support Date: Thu, 26 Jan 2023 20:49:31 +0200 Message-ID: <20230126184948.10478-2-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT100:EE_|SA1PR12MB6726:EE_ X-MS-Office365-Filtering-Correlation-Id: 6cbcbfa9-d3e1-4572-24ea-08daffce2048 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BYIAyJQeYYzsneuvKzUSTI6hGEyEOSOwF2CE+1r9uOjNXH/+ZolE+5j73r6wA25CmHpzEjJWXguKN6+fUudcZX2IXaF6t4ZlfuEXete6AdHi7OkFFinIqnwAs1K1LSWlKmeUkt2mqUCgCbXOBQJVPDKdAL2aw64EODRfkNlB8Wxr5rqt/3KU4jVElAGIP6ab69NxsVmXI5FOVx9cZzX0AbkwHl9LaOeuUt9XXa7324RRUjNY6FYLL3s82ZkhtihEALN6c1o2AER7indrgCCpGTZNnEuv9/p1J9ctMBOMwpNdAfsA2b64LWOMmUEbCUIKJx+hMNgVPyqHXQU5ay96I9TIBFV62uimV1Fxfs2V2fvvpnZ9HpS86VJVcgm3RJNn4DHZDydF+dagpK/nlx/xlhBCi2YkBqz8KIDQsdFd4KSgXJzU8bJQu+GwLA3QdtqeHKOIhfnvRi8lcZILVkOGebea8/wF95UqshCEuM1HTJEK2+Dx+vCqdqS68cfu2IAwSYLTH4ftu3yKi9UvqsyHOghmjy7SvIP02afb4ArUQFZNcRdO4RDEXQ8YLd9LUGAvEO7ejorAWeGC3df5qwcOVd/WPEyW4XDy/8FYWsEd8i06cgT+CnDEN8k7PQN4nLvpg+A/zUq98/K/YBQXb0aUJY7d9PvJTEIkcWhVaNbAkmETJxt4T9IZ22wG/HgbMozr97v6qOyELklCJSqlSl+hevPjbbDbTrKKslKZb+r64+6ZW804OnrOMGGN1mDtECJVZ20if9hodc4Fjxf48aTNU4F0RacovFu29W3BdBo5Bdk= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(396003)(136003)(346002)(376002)(451199018)(36840700001)(46966006)(40470700004)(36860700001)(26005)(356005)(83380400001)(7636003)(5660300002)(82740400003)(7416002)(7696005)(30864003)(41300700001)(2906002)(86362001)(82310400005)(8936002)(4326008)(40480700001)(40460700003)(6666004)(316002)(186003)(336012)(8676002)(47076005)(2616005)(478600001)(54906003)(6916009)(70586007)(36756003)(966005)(426003)(1076003)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:49:59.5197 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6cbcbfa9-d3e1-4572-24ea-08daffce2048 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT100.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB6726 Received-SPF: softfail client-ip=2a01:111:f400:7e88::62c; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759028002100003 Content-Type: text/plain; charset="utf-8" Pre-copy support allows the VFIO device data to be transferred while the VM is running. This helps to accommodate VFIO devices that have a large amount of data that needs to be transferred, and it can reduce migration downtime. Pre-copy support is optional in VFIO migration protocol v2. Implement pre-copy of VFIO migration protocol v2 and use it for devices that support it. Full description of it can be found here [1]. [1] https://lore.kernel.org/kvm/20221206083438.37807-3-yishaih@nvidia.com/ Signed-off-by: Avihai Horon --- docs/devel/vfio-migration.rst | 29 ++++++--- include/hw/vfio/vfio-common.h | 3 + hw/vfio/common.c | 8 ++- hw/vfio/migration.c | 112 ++++++++++++++++++++++++++++++++-- hw/vfio/trace-events | 5 +- 5 files changed, 140 insertions(+), 17 deletions(-) diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst index 1d50c2fe5f..51f5e1a537 100644 --- a/docs/devel/vfio-migration.rst +++ b/docs/devel/vfio-migration.rst @@ -7,12 +7,14 @@ the guest is running on source host and restoring this sa= ved state on the destination host. This document details how saving and restoring of VFIO devices is done in QEMU. =20 -Migration of VFIO devices currently consists of a single stop-and-copy pha= se. -During the stop-and-copy phase the guest is stopped and the entire VFIO de= vice -data is transferred to the destination. - -The pre-copy phase of migration is currently not supported for VFIO device= s. -Support for VFIO pre-copy will be added later on. +Migration of VFIO devices consists of two phases: the optional pre-copy ph= ase, +and the stop-and-copy phase. The pre-copy phase is iterative and allows to +accommodate VFIO devices that have a large amount of data that needs to be +transferred. The iterative pre-copy phase of migration allows for the gues= t to +continue whilst the VFIO device state is transferred to the destination, t= his +helps to reduce the total downtime of the VM. VFIO devices can choose to s= kip +the pre-copy phase of migration by not reporting the VFIO_MIGRATION_PRE_CO= PY +flag in VFIO_DEVICE_FEATURE_MIGRATION ioctl. =20 A detailed description of the UAPI for VFIO device migration can be found = in the comment for the ``vfio_device_mig_state`` structure in the header file @@ -29,6 +31,12 @@ VFIO implements the device hooks for the iterative appro= ach as follows: driver, which indicates the amount of data that the vendor driver has ye= t to save for the VFIO device. =20 +* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is + active only if the VFIO device is in pre-copy states. + +* A ``save_live_iterate`` function that reads the VFIO device's data from = the + vendor driver during iterative phase. + * A ``save_state`` function to save the device config space if it is prese= nt. =20 * A ``save_live_complete_precopy`` function that sets the VFIO device in @@ -91,8 +99,10 @@ Flow of state changes during Live migration =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 Below is the flow of state change during live migration. -The values in the brackets represent the VM state, the migration state, and +The values in the parentheses represent the VM state, the migration state,= and the VFIO device state, respectively. +The text in the square brackets represents the flow if the VFIO device sup= ports +pre-copy. =20 Live migration save path ------------------------ @@ -104,11 +114,12 @@ Live migration save path | migrate_init spawns migration_thread Migration thread then calls each device's .save_setup() - (RUNNING, _SETUP, _RUNNING) + (RUNNING, _SETUP, _RUNNING [_PRE_COPY]) | - (RUNNING, _ACTIVE, _RUNNING) + (RUNNING, _ACTIVE, _RUNNING [_PRE_COPY]) If device is active, get pending_bytes by .save_live_pending() If total pending_bytes >=3D threshold_size, call .save_live_iter= ate() + [Data of VFIO device for pre-copy phase is copied] Iterate till total pending bytes converge and are less than thresh= old | On migration completion, vCPU stops and calls .save_live_complete_precop= y for diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 5f8e7a02fe..88c2194fb9 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -67,7 +67,10 @@ typedef struct VFIOMigration { int data_fd; void *data_buffer; size_t data_buffer_size; + uint64_t mig_flags; uint64_t stop_copy_size; + uint64_t precopy_init_size; + uint64_t precopy_dirty_size; } VFIOMigration; =20 typedef struct VFIOAddressSpace { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 9a0dbee6b4..93b18c5e3d 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -357,7 +357,9 @@ static bool vfio_devices_all_dirty_tracking(VFIOContain= er *container) =20 if ((vbasedev->pre_copy_dirty_page_tracking =3D=3D ON_OFF_AUTO= _OFF) && (migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING = || - migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_= P2P)) { + migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_= P2P || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY= || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY= _P2P)) { return false; } } @@ -387,7 +389,9 @@ static bool vfio_devices_all_running_and_mig_active(VFI= OContainer *container) } =20 if (migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING || - migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_P= 2P) { + migration->device_state =3D=3D VFIO_DEVICE_STATE_RUNNING_P= 2P || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY = || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY_= P2P) { continue; } else { return false; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 760f667e04..2a0a663023 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -69,6 +69,10 @@ static const char *mig_state_to_str(enum vfio_device_mig= _state state) return "RESUMING"; case VFIO_DEVICE_STATE_RUNNING_P2P: return "RUNNING_P2P"; + case VFIO_DEVICE_STATE_PRE_COPY: + return "PRE_COPY"; + case VFIO_DEVICE_STATE_PRE_COPY_P2P: + return "PRE_COPY_P2P"; default: return "UNKNOWN STATE"; } @@ -237,6 +241,11 @@ static int vfio_save_block(QEMUFile *f, VFIOMigration = *migration) data_size =3D read(migration->data_fd, migration->data_buffer, migration->data_buffer_size); if (data_size < 0) { + /* Pre-copy emptied all the device state for now */ + if (errno =3D=3D ENOMSG) { + return 1; + } + return -errno; } if (data_size =3D=3D 0) { @@ -260,6 +269,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) VFIODevice *vbasedev =3D opaque; VFIOMigration *migration =3D vbasedev->migration; uint64_t stop_copy_size =3D VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE; + int ret; =20 qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); =20 @@ -273,6 +283,23 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) return -ENOMEM; } =20 + if (migration->mig_flags & VFIO_MIGRATION_PRE_COPY) { + switch (migration->device_state) { + case VFIO_DEVICE_STATE_RUNNING: + ret =3D vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_P= RE_COPY, + VFIO_DEVICE_STATE_RUNNING); + if (ret) { + return ret; + } + break; + case VFIO_DEVICE_STATE_STOP: + /* vfio_save_complete_precopy() will go to STOP_COPY */ + break; + default: + return -EINVAL; + } + } + trace_vfio_save_setup(vbasedev->name, migration->data_buffer_size); =20 qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); @@ -287,6 +314,12 @@ static void vfio_save_cleanup(void *opaque) =20 g_free(migration->data_buffer); migration->data_buffer =3D NULL; + + if (migration->mig_flags & VFIO_MIGRATION_PRE_COPY) { + migration->precopy_init_size =3D 0; + migration->precopy_dirty_size =3D 0; + } + vfio_migration_cleanup(vbasedev); trace_vfio_save_cleanup(vbasedev->name); } @@ -301,9 +334,55 @@ static void vfio_save_pending(void *opaque, uint64_t t= hreshold_size, =20 *res_precopy_only +=3D migration->stop_copy_size; =20 + if (migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY_P2P) { + if (migration->precopy_init_size) { + /* + * Initial size should be transferred during pre-copy phase so + * stop-copy phase will not be slowed down. Report threshold_s= ize + * to force another pre-copy iteration. + */ + *res_precopy_only +=3D threshold_size; + } else { + *res_precopy_only +=3D migration->precopy_dirty_size; + } + } + trace_vfio_save_pending(vbasedev->name, *res_precopy_only, *res_postcopy_only, *res_compatible, - migration->stop_copy_size); + migration->stop_copy_size, + migration->precopy_init_size, + migration->precopy_dirty_size); +} + +static bool vfio_is_active_iterate(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + + return migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY_P2P; +} + +static int vfio_save_iterate(QEMUFile *f, void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret; + + ret =3D vfio_save_block(f, migration); + if (ret < 0) { + return ret; + } + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); + + trace_vfio_save_iterate(vbasedev->name); + + /* + * A VFIO device's pre-copy dirty_bytes is not guaranteed to reach zer= o. + * Return 1 so following handlers will not be potentially blocked. + */ + return 1; } =20 static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) @@ -312,7 +391,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void= *opaque) enum vfio_device_mig_state recover_state; int ret; =20 - /* We reach here with device state STOP only */ + /* We reach here with device state STOP or STOP_COPY only */ recover_state =3D VFIO_DEVICE_STATE_STOP; ret =3D vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, recover_state); @@ -430,6 +509,8 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { .save_setup =3D vfio_save_setup, .save_cleanup =3D vfio_save_cleanup, .save_live_pending =3D vfio_save_pending, + .is_active_iterate =3D vfio_is_active_iterate, + .save_live_iterate =3D vfio_save_iterate, .save_live_complete_precopy =3D vfio_save_complete_precopy, .save_state =3D vfio_save_state, .load_setup =3D vfio_load_setup, @@ -442,13 +523,19 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { static void vfio_vmstate_change(void *opaque, bool running, RunState state) { VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; enum vfio_device_mig_state new_state; int ret; =20 if (running) { new_state =3D VFIO_DEVICE_STATE_RUNNING; } else { - new_state =3D VFIO_DEVICE_STATE_STOP; + new_state =3D + ((migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY_P2= P) && + (state =3D=3D RUN_STATE_FINISH_MIGRATE || state =3D=3D RUN_ST= ATE_PAUSED)) ? + VFIO_DEVICE_STATE_STOP_COPY : + VFIO_DEVICE_STATE_STOP; } =20 ret =3D vfio_migration_set_state(vbasedev, new_state, @@ -496,6 +583,9 @@ static int vfio_migration_data_notifier(NotifierWithRet= urn *n, void *data) { VFIOMigration *migration =3D container_of(n, VFIOMigration, migration_= data); VFIODevice *vbasedev =3D migration->vbasedev; + struct vfio_precopy_info precopy =3D { + .argsz =3D sizeof(precopy), + }; PrecopyNotifyData *pnd =3D data; =20 if (pnd->reason !=3D PRECOPY_NOTIFY_AFTER_BITMAP_SYNC) { @@ -515,8 +605,21 @@ static int vfio_migration_data_notifier(NotifierWithRe= turn *n, void *data) migration->stop_copy_size =3D VFIO_MIG_STOP_COPY_SIZE; } =20 + if ((migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY || + migration->device_state =3D=3D VFIO_DEVICE_STATE_PRE_COPY_P2P)) { + if (ioctl(migration->data_fd, VFIO_MIG_GET_PRECOPY_INFO, &precopy)= ) { + migration->precopy_init_size =3D 0; + migration->precopy_dirty_size =3D 0; + } else { + migration->precopy_init_size =3D precopy.initial_bytes; + migration->precopy_dirty_size =3D precopy.dirty_bytes; + } + } + trace_vfio_migration_data_notifier(vbasedev->name, - migration->stop_copy_size); + migration->stop_copy_size, + migration->precopy_init_size, + migration->precopy_dirty_size); =20 return 0; } @@ -588,6 +691,7 @@ static int vfio_migration_init(VFIODevice *vbasedev) migration->vbasedev =3D vbasedev; migration->device_state =3D VFIO_DEVICE_STATE_RUNNING; migration->data_fd =3D -1; + migration->mig_flags =3D mig_flags; =20 oid =3D vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); if (oid) { diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index db9cb94952..37724579e3 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -154,7 +154,7 @@ vfio_load_cleanup(const char *name) " (%s)" vfio_load_device_config_state(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret)= " (%s) size 0x%"PRIx64" ret %d" -vfio_migration_data_notifier(const char *name, uint64_t stopcopy_size) " (= %s) stopcopy size 0x%"PRIx64 +vfio_migration_data_notifier(const char *name, uint64_t stopcopy_size, uin= t64_t precopy_init_size, uint64_t precopy_dirty_size) " (%s) stopcopy size = 0x%"PRIx64" precopy initial size 0x%"PRIx64" precopy dirty size 0x%"PRIx64 vfio_migration_probe(const char *name) " (%s)" vfio_migration_set_state(const char *name, const char *state) " (%s) state= %s" vfio_migration_state_notifier(const char *name, const char *state) " (%s) = state %s" @@ -162,6 +162,7 @@ vfio_save_block(const char *name, int data_size) " (%s)= data_size %d" vfio_save_cleanup(const char *name) " (%s)" vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d" vfio_save_device_config_state(const char *name) " (%s)" -vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, u= int64_t compatible, uint64_t stopcopy_size) " (%s) precopy 0x%"PRIx64" post= copy 0x%"PRIx64" compatible 0x%"PRIx64" stopcopy size 0x%"PRIx64 +vfio_save_iterate(const char *name) " (%s)" +vfio_save_pending(const char *name, uint64_t precopy, uint64_t postcopy, u= int64_t compatible, uint64_t stopcopy_size, uint64_t precopy_init_size, uin= t64_t precopy_dirty_size) " (%s) precopy 0x%"PRIx64" postcopy 0x%"PRIx64" c= ompatible 0x%"PRIx64" stopcopy size 0x%"PRIx64" precopy initial size 0x%"PR= Ix64" precopy dirty size 0x%"PRIx64 vfio_save_setup(const char *name, uint64_t data_buffer_size) " (%s) data b= uffer size 0x%"PRIx64 vfio_vmstate_change(const char *name, int running, const char *reason, con= st char *dev_state) " (%s) running %d reason %s device state %s" --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759079; cv=pass; d=zohomail.com; s=zohoarc; b=TPsGHkT3XtFr+rMvbxZtETfc02JBf2nENSYudzxYQuECzcBpMuKXvwDQRDt4+lVThfUM1NfBWNOJ5/wbHSH/ythMJj3KlvFWYlZLLxZQYEjlo9FEDSRuXkpQ/6SN3pQYFLouhgHdZFHbJ7YLqTrkGPuMQN/7xYZ061U1o5fnBH4= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759079; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=IR2dsXmWhu6AYqO59bBSGoeNj6rBH2k8HoU3pGDzRGQ=; b=Jir9GBMEh1F6IBBveDVwnVLZzuOSKUMxu84GrpzMRwc2dV0uOHgQGl4PqT+DbemPQ/ctNszK+WrRxvynlQ4koqXtSuWETwnDABevQGD4BZQqDly0ltHl/zOnNia5XEVqIh7cEVgT5ZmEyTx8eguUdgd65WRr/BnQHXe7RaSsT+o= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759078575105.1517740896063; Thu, 26 Jan 2023 10:51:18 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KC-0004OJ-TF; Thu, 26 Jan 2023 13:50:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7K5-0004NM-Mf for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:09 -0500 Received: from mail-mw2nam10on2058.outbound.protection.outlook.com ([40.107.94.58] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7K3-0002SA-Ke for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:09 -0500 Received: from MW4PR04CA0113.namprd04.prod.outlook.com (2603:10b6:303:83::28) by SA0PR12MB4349.namprd12.prod.outlook.com (2603:10b6:806:98::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:03 +0000 Received: from CO1NAM11FT100.eop-nam11.prod.protection.outlook.com (2603:10b6:303:83:cafe::11) by MW4PR04CA0113.outlook.office365.com (2603:10b6:303:83::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:03 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT100.mail.protection.outlook.com (10.13.175.133) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:03 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:02 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:02 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:49:58 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z7vS8eUJrsChBJD1RW4wFjbSokfEkaCatrLnlyF9OiUROePqLrcXyQt53XJQg5io5yGD0SAT8xEbHRp+HzJTm/9jKXfA/+ueK5R7hR4ncoEoPr9Q8F+nr7HB1GuWqVMV/Pao+N+j8OmKu2lddd067OlMj7i3fn732e35GN9pI/NZbVUDwZf2cwJmVdrrq/4LHIDxlg7caJe+Y/08tDM0XJti06wyS/Y5QqFNgHKoQHht/FfLSzAdeI4qQ1YS/8W30E/ePxebZ5RcDJYU4ntI3rXgwqtH72t26oRp7oujH8yMs0TOX5lXAEJyby0m4qnQ+FL3WnwlDIAXAESrGWIUNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IR2dsXmWhu6AYqO59bBSGoeNj6rBH2k8HoU3pGDzRGQ=; b=XGWPfrnqMjf3XM1lBipuhb0t3/4OB5veU7fFkFlJhIAJnkpG5cLNLdi2xqdBTSFEX+xp1G7ANlGTxg3BWGrdsXXaTKJbw5eYfMLlO3JEkvzu1TVbU5bdyyo8sODqHPImgflyF1hbFaOMQwwkWoh05EPVImIKkZKv+94/tW1U9DM9rn9K9vlv0iYwAuJW5ww6kDHJfVExI1iEJ/47O2GxlJoMhy6RbWqRfu3lC25qaxRIicJZPY3fWV8j7iTh1prPYmQxERD4hLH2kvz15hxSWtAUg3gIjq/rijN3NgwiZ4ebGdWo+0zdynAGeICmYeswIhKyR9N9RVcTTgSlfYAo6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IR2dsXmWhu6AYqO59bBSGoeNj6rBH2k8HoU3pGDzRGQ=; b=HNNlnv3eWzb5pzOyOTFcyLvftgFEhdaancYVTogwHr0jYsm5qhR01hH8K7wtOxBAKTBc84Jsy3DLg0p8ipYwhn5XhOp6LkSYi9JRot1XJ21QyhlfvfuU6SDJuyAp7MHRgrjZkZCmC+qi9ft/4BH3pe5QDOZVKR4geyoC+GCN8ESoMBUBARAuzsv/TYwCxPwKUSX8FAxls3OOjE5/hakgcFJBe3FUsOnB9QMJlEymDUfv/UcSOEHbDLPt+A4QkUbyXlhl+/dy4emr0Gem+AUFK8LNYjzkiOTJ7cBJtzERaDT3C+aQPBT/qrL2uTt7jjbjdgb6S5/ZJ/PLbYp5OcY8EQ== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 02/18] vfio/common: Fix error reporting in vfio_get_dirty_bitmap() Date: Thu, 26 Jan 2023 20:49:32 +0200 Message-ID: <20230126184948.10478-3-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT100:EE_|SA0PR12MB4349:EE_ X-MS-Office365-Filtering-Correlation-Id: 46af893d-782c-42bf-ade6-08daffce22c4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kfGR6xUjgleBCqgNAWtFIf2xnqlGp43s1at41RBCgdZJUDv/iLha+jx1A/W+MAvv39tPswg1mD+Gn5syS5rpfzKuCI4Nj1qvEZEQ2yYdojerYYNIbEy7qxKcZGi14oqKLfIQfSnJStEygOW2Tbu3+49IRhl9Os0ObsIjW2zQCbXTbuKjVut/rzjL/SARjddRZat01OR+jUO60UY8YhorP9MAAVWsZizX43N+2/WCeIMRGUGRHtlPruuPUa/rPYia4wa3KIx1ij30hwanHuj3N3ZSBMhXwHIqOr7CTnocaXcdX3CscR1stgXgxJ3cAi7jZpBWEF8j8evwpn3KAPvqw+MdWjhqSqQY6C8qb3sgWAsIYm+iTM979qJOvOO+brr5QU2DhxCNGchc2p5HDOyM3RwydoAoycyj3cvv2gnFt09tsicpH8O3dn7OhvdCBP4a/thd/G6VJbiwubCbFpjZgiXt3ar1/kFx8IPsbkAl7GotNN6D21cAsP/aVwNGjK+LGcYA55RguAtuNfg0gibpzxWLYlg2bk4FZ7+Y7qWBWMXRozfq0gyuwRGb48lEa5gtcgx+NkltbRPvn36mEoe6bQgf5u1eG4srSm33T5Tr/wx0jpQ7vBVXA6JIAVJr50vvfF6Pqv+TyjGR8gtYt+fVoN3L/uns96itwuv1UmWCXjBrQjivbBmvAGMarVk3+5IaVYxi2JaUEEHMADE64kb/GBOEI4FFojvMVO53IfzJpQo= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(136003)(396003)(39860400002)(346002)(451199018)(40470700004)(36840700001)(46966006)(41300700001)(8936002)(26005)(6666004)(47076005)(36860700001)(426003)(86362001)(40480700001)(40460700003)(356005)(82740400003)(36756003)(70206006)(82310400005)(54906003)(7696005)(70586007)(316002)(4326008)(6916009)(8676002)(7636003)(2616005)(336012)(478600001)(186003)(1076003)(7416002)(5660300002)(2906002)(4744005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:03.7069 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 46af893d-782c-42bf-ade6-08daffce22c4 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT100.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4349 Received-SPF: softfail client-ip=40.107.94.58; envelope-from=avihaih@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759080107100001 Content-Type: text/plain; charset="utf-8" Return -errno instead of -1 if VFIO_IOMMU_DIRTY_PAGES ioctl fails in vfio_get_dirty_bitmap(). Signed-off-by: Avihai Horon Reviewed-by: C=C3=A9dric Le Goater --- hw/vfio/common.c | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 93b18c5e3d..d892609cf1 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1288,6 +1288,7 @@ static int vfio_get_dirty_bitmap(VFIOContainer *conta= iner, uint64_t iova, =20 ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); if (ret) { + ret =3D -errno; error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, (uint64_t)range->size, errno); --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759052; cv=pass; d=zohomail.com; s=zohoarc; b=Ufm9NvDJTxaCxU2AlJIYuVwUilzMjFstAJTFxjr4q9or3uon5553CK0iuRqRh/wEBuMQeHhnasPT4/WSKcE4DT0QeS7vsgfYPWWShnCakpOjKqobqYxaYzcztahCrEJwgNl36/IiveG2Aw4GhZ4UHDJzMljw7a0sybuang97Buk= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759052; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WjBf/rSgs42hm3eLxOhwIYXl87rKvUmqpjFmi0IwVko=; b=EUhzB64GVWtOcZOEhzIC3TtLnZGsYxGzHNtCZ8Z+wO5bQuVUQwB4hXTSUhdtr8MuY6l+Pb96kHgm3JRK8qQXPTomMV2TuxUpIXRj1ct7HpES8KLhSefv3K/FEbEYbdJiZQJR/Kl2HK++0zHE+b8mAh8nfCBSNj5z5pBiYbo2FDw= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759052899178.97722768158724; Thu, 26 Jan 2023 10:50:52 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KJ-0004PJ-K3; Thu, 26 Jan 2023 13:50:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KI-0004P4-4u for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:22 -0500 Received: from mail-mw2nam12on20611.outbound.protection.outlook.com ([2a01:111:f400:fe5a::611] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KG-0002TF-7o for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:21 -0500 Received: from DS7PR03CA0269.namprd03.prod.outlook.com (2603:10b6:5:3b3::34) by PH7PR12MB5709.namprd12.prod.outlook.com (2603:10b6:510:1e0::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Thu, 26 Jan 2023 18:50:16 +0000 Received: from DM6NAM11FT089.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b3:cafe::58) by DS7PR03CA0269.outlook.office365.com (2603:10b6:5:3b3::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:16 +0000 Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT089.mail.protection.outlook.com (10.13.173.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6023.16 via Frontend Transport; Thu, 26 Jan 2023 18:50:15 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:07 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:06 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:02 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JGISUYGmYu3h0HzFVju+wLmgwNJpeizxNK51bTRM5sTz4X+siXsXLDlUBvWJ8UjtUB2LR2Ar6yL/wuT6eg4MTaPauY2EcYHuSWG3ipJXRC6CrGJlBt1/JulDjvQSaUQnzdYD5GNl0b9ZKVAgilFFPRX+Cjb1y3pCxrM5UAsJbRZt7kaPBWoR+uvUD4vuh0Mq0GxmFDWHfcs5eTOLU2NujDagKCJGx1NIYALlTi9DU7Kww0aOA9MALFlEtawsWNan8zhXQm2Ix2q+NJcA1MDvAGa8Sq/N/k6atS4hI7kh1vh3m9bd27CFrDwH7PfRD3xgBIjon0caTkqt6sDzX71dCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WjBf/rSgs42hm3eLxOhwIYXl87rKvUmqpjFmi0IwVko=; b=bcZHFX/ZpGGIJQ1DcVX+xhJXLNPl8gn28FC0Rn6DUHhLoZ7uYbt6KdoJIPtVOp+/0+ksj3vQJ4/CRv0urV42U340oUygkg4KJs6dwpoRfKI6Ou8vKEiB3DHZDOVKK1wrKmncF1XMFMiBEXcFcB4BpKLIullyoxCmiqhW7edF1jXP1NAWo0h7B+RVKadLTHGuR+SsW2td7qqXrl2YncDTYGnodLyqlpLwN01MHGDykhef9J+OfaCLnBo9dffxaJPDJJ42BIj2hVy3b0aDY3qTqGbmKA+m/Q1v95DNCEL/t5TrtjwvLFcpNZlgFA372itDSSQbkWl6Ps2zGdke81j1Gg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WjBf/rSgs42hm3eLxOhwIYXl87rKvUmqpjFmi0IwVko=; b=Fr8KKl83gA8Ndwpi+hR2RKOHZZEEgJrtCQOvAzMj8XarU9RZSlMVYGo3rYjW/Cg5CtBGB/1nySmLHWc7CS8H/sKjNIgJpwRsGKSfU2qjHsuQkTz+leXpclNH7qDA4Xz395bDozin6cGltBDWDX0BQTqa+Ykb1A3MoHgNx0idwjmCcixsfO3HTzI600h7FAd+szhwpQHcYYftCK5Eo370/QhvCznz8TeWg2JnpZIZQcqhC+Be0FcWKIALqoItDaZEcHH3ywuaZ3/NQzeUvxirQqw82hrfk7/TB8cREzFlSBGgoXRmQ5OlBCOldBQ59wowmG0cFZg+gAUiiz8Pwmqjiw== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 03/18] vfio/common: Fix wrong %m usages Date: Thu, 26 Jan 2023 20:49:33 +0200 Message-ID: <20230126184948.10478-4-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT089:EE_|PH7PR12MB5709:EE_ X-MS-Office365-Filtering-Correlation-Id: 44af71e7-69dd-4655-9157-08daffce2a08 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jAfbmWp4Sx0diaUppNLPk4JW/fb9azmDTl5vQOsEQBpaRgVZYxAtnAR/me5JO6OkeAWvY0ngIFaAtgAK6tWJ/WPavZViLJPc1cDoVaSgYOGm4tEalKZFbDF62v4u5Kbeh+6XI2P8tiw4P+u8CApNl5NNaw5/C9/G0wtz2nfCF1+mRoK9jZK6OAB9eRztF3OGZprMQynhiecAQUs3nsQzXz4HblkU9UIaxuo2XqWVCY4mmDqiSlsU75+lIdsTxa7UIqJAMNJ2eR+J3P2KwbcpGan+NiRXVJEzf7fiyQk+dta8wH11SCrYHfFftxMykRfIC/McraEeOh/r7GMgxu9ucbKRA8IYvXRAtCPtWCSNnamlNL7FixDdFvTiSh8E9CNLpc4B04OnJ3IUhWFur5FV+o1z6X8bxD8eIg/h60YY6Tn5riR+Cg5JuDp+PBSV+GiDE8pQ5C/LW8CDtlxZK3e8gYkELiFemW/aKbcJqAMgZm+X8TpIO5Qr1EQyQrod1tQZilrAMk0/1nRcVCYJfTCXNu2y+W4lUg62zVCiGWqql282djsCtoStg9CanMDnfhMS/ekYrtUr04HLjgw5c65ZD9H9VcL+SP4YFzLm7BcoG9J9ncVgomwbK0kIWFDYga33wEgvjl21ksMUkSrTbCmX3cJGO+Iy9ATdEL0UfZix6sO6ZGTn71+0O6PySheVn7eqymdVx8HDKBA42CDM4ZPPGQ== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(346002)(376002)(136003)(396003)(451199018)(46966006)(36840700001)(40470700004)(54906003)(478600001)(70586007)(7636003)(40480700001)(36756003)(82740400003)(40460700003)(336012)(316002)(70206006)(2906002)(7696005)(47076005)(2616005)(426003)(4326008)(356005)(8676002)(36860700001)(41300700001)(1076003)(83380400001)(82310400005)(6916009)(6666004)(86362001)(5660300002)(26005)(7416002)(8936002)(186003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:15.8490 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 44af71e7-69dd-4655-9157-08daffce2a08 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT089.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5709 Received-SPF: softfail client-ip=2a01:111:f400:fe5a::611; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759053983100002 Content-Type: text/plain; charset="utf-8" There are several places where the %m conversion is used if one of vfio_dma_map(), vfio_dma_unmap() or vfio_get_dirty_bitmap() fail. The %m usage in these places is wrong since %m relies on errno value while the above functions don't report errors via errno. Fix it by using strerror() with the returned value instead. Signed-off-by: Avihai Horon Reviewed-by: C=C3=A9dric Le Goater --- hw/vfio/common.c | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index d892609cf1..643418f6f1 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -656,17 +656,17 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, I= OMMUTLBEntry *iotlb) read_only); if (ret) { error_report("vfio_dma_map(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx", %p) =3D %d (%m)", + "0x%"HWADDR_PRIx", %p) =3D %d (%s)", container, iova, - iotlb->addr_mask + 1, vaddr, ret); + iotlb->addr_mask + 1, vaddr, ret, strerror(-ret)); } } else { ret =3D vfio_dma_unmap(container, iova, iotlb->addr_mask + 1, iotl= b); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx") =3D %d (%m)", + "0x%"HWADDR_PRIx") =3D %d (%s)", container, iova, - iotlb->addr_mask + 1, ret); + iotlb->addr_mask + 1, ret, strerror(-ret)); } } out: @@ -1048,8 +1048,9 @@ static void vfio_listener_region_add(MemoryListener *= listener, vaddr, section->readonly); if (ret) { error_setg(&err, "vfio_dma_map(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx", %p) =3D %d (%m)", - container, iova, int128_get64(llsize), vaddr, ret); + "0x%"HWADDR_PRIx", %p) =3D %d (%s)", + container, iova, int128_get64(llsize), vaddr, ret, + strerror(-ret)); if (memory_region_is_ram_device(section->mr)) { /* Allow unexpected mappings not to be fatal for RAM devices */ error_report_err(err); @@ -1181,16 +1182,18 @@ static void vfio_listener_region_del(MemoryListener= *listener, ret =3D vfio_dma_unmap(container, iova, int128_get64(llsize), = NULL); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx") =3D %d (%m)", - container, iova, int128_get64(llsize), ret); + "0x%"HWADDR_PRIx") =3D %d (%s)", + container, iova, int128_get64(llsize), ret, + strerror(-ret)); } iova +=3D int128_get64(llsize); } ret =3D vfio_dma_unmap(container, iova, int128_get64(llsize), NULL= ); if (ret) { error_report("vfio_dma_unmap(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx") =3D %d (%m)", - container, iova, int128_get64(llsize), ret); + "0x%"HWADDR_PRIx") =3D %d (%s)", + container, iova, int128_get64(llsize), ret, + strerror(-ret)); } } =20 @@ -1337,9 +1340,9 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier= *n, IOMMUTLBEntry *iotlb) translated_addr); if (ret) { error_report("vfio_iommu_map_dirty_notify(%p, 0x%"HWADDR_PRIx"= , " - "0x%"HWADDR_PRIx") =3D %d (%m)", - container, iova, - iotlb->addr_mask + 1, ret); + "0x%"HWADDR_PRIx") =3D %d (%s)", + container, iova, iotlb->addr_mask + 1, ret, + strerror(-ret)); } } rcu_read_unlock(); --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759042; cv=pass; d=zohomail.com; s=zohoarc; b=TmhwXjBBzoNWjxjfzb1a7EtlFoSjKXk1pr1KpX+QHXwB1XSH68/AmkxS2qx5wvEwHw/eHla6kcvYzuV5NeDk4594z18MusYeV7sYzN8PxTGoYkGp7fwqmVF1I54P26GSHm9ffVyp75ntSva3sm9WD14uB6H+I53AQmX1cCE8S7Y= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759042; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SI3B9FEdnfEMvYhdUGPQx1QkhgBoanyT6YNvzlHwQoo=; b=itCDxPKD1e5ERlQbXNY/6oYnsLlVkhTIg5XGlz96bO8OT3fW35lHy7ylO1tKpPvzoIx0cLY5k0SjFfSE3d+1KCsZLtyVyAldtQY2jzH2aDt+EIpzXdytmJuHQEZfzWpRH8Ix/GH7rW+rFihxULyoHQ2G4enVwRLeVxgeTYy/BRE= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759042712936.4919460398575; Thu, 26 Jan 2023 10:50:42 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KT-0004XI-NR; Thu, 26 Jan 2023 13:50:33 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KS-0004Wp-12 for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:32 -0500 Received: from mail-bn8nam12on2055.outbound.protection.outlook.com ([40.107.237.55] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KP-0002Ys-SR for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:31 -0500 Received: from BN9PR03CA0941.namprd03.prod.outlook.com (2603:10b6:408:108::16) by CY5PR12MB6597.namprd12.prod.outlook.com (2603:10b6:930:43::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21; Thu, 26 Jan 2023 18:50:25 +0000 Received: from BN8NAM11FT010.eop-nam11.prod.protection.outlook.com (2603:10b6:408:108:cafe::2b) by BN9PR03CA0941.outlook.office365.com (2603:10b6:408:108::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:25 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT010.mail.protection.outlook.com (10.13.177.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.17 via Frontend Transport; Thu, 26 Jan 2023 18:50:24 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:11 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:11 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:07 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H0U/TZ4Hwko0bfq6DjvhdeD2LvZWyTXuGqPMi7+fISto4iUpRReD9BPCPFe6VsVU+27VTBgpBs6nyMVjUQ7cmNYR7L39yCLNet+Sfyre2U/aN7V34/C0O/4nzO841E29jw5u3lZCEV3dedumA4xC+LaLsxmv4FhEHYsA+hW8I+m0R4BAqwLMOmpliLYq2VOj6aRlzAe/PNkF1i9abEMZe5BEElm6Z68Ub8vyMa5DObW2Ek/rLkDaLJa+IhKcix9FGzUZ/eOzL60eqVBxyFl/NA2iTFRVgWXpf0g/ZPw68XnFCQgHj2PzQ0sMNDl8/ZDSJ8UyoJnCBnKNhU5ClxfKtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SI3B9FEdnfEMvYhdUGPQx1QkhgBoanyT6YNvzlHwQoo=; b=SieLpfmW0q4GNUOIXMNh21wg+/4rtky8JXdxJVUvCx+K1SsamZfVwOlq8XQIxW4M9vJinNa7J1jqgmjgHIB4qse9NiDQLaNWFh76VevKdHhTtp4KAIQ9CZlPFz3M/NR7pu5bB8xpfZdOSIN9T4ScsFsYovyqMJpMRnqhwkr58ExTeHi8lybOFg8E03uTRVN50onDJ3R+MFUrsBW7tneNHLIgMd8d4HwD+XoG+x7GjiJDwHlqpTxxGahZrffJjj1BgZmJxSk7zDV/T/fcyFqk6pkhwdvwCj+10Sn668d+h9qYFWj6WZralxCR9SgWmpMUfjek2j/2RFCsqjRA15Y6iA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SI3B9FEdnfEMvYhdUGPQx1QkhgBoanyT6YNvzlHwQoo=; b=ruFVtKLJMxE8H1a3Dm/p8WPSqn561JVUdqQs1ThtUfM32FfY7sFa08kVhPrHUvOE8TloEbY81ewC9PqAHADjdgqD47ifrSnLPQaP0KRZdzZbDt409vk8WKOFtXnd3yAn/D3br0swWEVXqtBjU4wEchkAyKLRu5oyjqLIYOtbgFwoIZpGf4hQwmpDj6voyzmHxm6bSrkZ80MLCQWdQtYOQjm86hC1EG+ieaM8KQsTViCCwmybh4B4iXq+puYMAYDnRaFQ0Ru91LyXVyFYJcm+QdfcWMEaiDMchYjBb4d/R/AKderhrX/CIr59skzCa5FxH3lZo2pgFUUSYGa0A9isAA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 04/18] vfio/common: Abort migration if dirty log start/stop/sync fails Date: Thu, 26 Jan 2023 20:49:34 +0200 Message-ID: <20230126184948.10478-5-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT010:EE_|CY5PR12MB6597:EE_ X-MS-Office365-Filtering-Correlation-Id: e904a555-67c4-416c-232c-08daffce2f55 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 2YaLwxDWRT3x8tziZfC9xPDvLpOBAgQMoM5folZz6jwkBuvo6agtZKu3aS2pE36h5+O2S52RkMczSzu2MTiwHY+Iz+r3OQkPu5AajSWpVDYRsTR+HzVfq1wJQQmLZcpv09GDNXRJ5/slSQzCV6A3jDKBW0DwGYlmejyZY+rp2ajTJ8XCmibpDfYt5Eq4SYBnU5DpD7exlKUPkzE19FY9GHmsI/DIs1p6agXSuAshMpjg1Gj7+lt9NsnWAltxH7vYb5GWUrKY9x4L9G/OON6e3oZhCsefKakAdrV+l13dqYDWQUZTukvgPE06zbXwEQSkfoDP2EhUkfBN7FLEi5IffxFR+fPqOKMBih5j47AhkZWZKt/fTaKkiTDvJyyUzLuwaBqPt5P9gpI3Nex3m/zD3qRqD2YQBGy4XZ/wKbqpFpZowlqaf/ppAVe7LRAsuFSawt3kHpFYZ0RPXGybfV9ep0b4RjFXafuea0o7KV2C9uQg5ITDV+JZLIvou3RCyKVlJION6GMOMj2vkaYIknpaarHUQKnYBndxFnSNYBMAYIQGGLx9Lj9nVVGqLwCh4DnIB5t9mLjbbaw2RLXM7ghvDl8nNa7RqX18MktGWhxxPGeZqVMnCYVJ1aw4qFpjXipjhODIFEmHNLuQccV0aznoSzIcTJFYKqop24N+T+rkPiXtG3uU/64uPWxIOcuCuO0X1zglM8xULMrtJdt+/EPeU4/UJNEbi/hXpRB3KH/+6fM= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(39860400002)(346002)(376002)(136003)(451199018)(46966006)(40470700004)(36840700001)(478600001)(86362001)(2906002)(36756003)(40460700003)(5660300002)(7416002)(4326008)(7636003)(7696005)(82310400005)(6916009)(82740400003)(40480700001)(41300700001)(8676002)(70586007)(70206006)(356005)(8936002)(6666004)(1076003)(36860700001)(54906003)(83380400001)(316002)(336012)(186003)(47076005)(26005)(426003)(2616005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:24.6926 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e904a555-67c4-416c-232c-08daffce2f55 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT010.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6597 Received-SPF: softfail client-ip=40.107.237.55; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759043972100003 Content-Type: text/plain; charset="utf-8" If VFIO dirty pages log start/stop/sync fails during migration, migration should be aborted as pages dirtied by VFIO devices might not be reported properly. This is not the case today, where in such scenario only an error is printed. Fix it by aborting migration in the above scenario. Fixes: 758b96b61d5c ("vfio/migrate: Move switch of dirty tracking into vfio= _memory_listener") Fixes: b6dd6504e303 ("vfio: Add vfio_listener_log_sync to mark dirty pages") Fixes: 9e7b0442f23a ("vfio: Add ioctl to get dirty pages bitmap during dma = unmap") Signed-off-by: Avihai Horon Reviewed-by: C=C3=A9dric Le Goater --- hw/vfio/common.c | 53 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 45 insertions(+), 8 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 643418f6f1..8e8ffbc046 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -41,6 +41,7 @@ #include "qapi/error.h" #include "migration/migration.h" #include "migration/misc.h" +#include "migration/qemu-file.h" #include "sysemu/tpm.h" =20 VFIOGroupList vfio_group_list =3D @@ -337,6 +338,19 @@ bool vfio_mig_active(void) return true; } =20 +static void vfio_set_migration_error(int err) +{ + MigrationState *ms =3D migrate_get_current(); + + if (migration_is_setup_or_active(ms->state)) { + WITH_QEMU_LOCK_GUARD(&ms->qemu_file_lock) { + if (ms->to_dst_file) { + qemu_file_set_error(ms->to_dst_file, err); + } + } + } +} + static bool vfio_devices_all_dirty_tracking(VFIOContainer *container) { VFIOGroup *group; @@ -633,6 +647,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOM= MUTLBEntry *iotlb) if (iotlb->target_as !=3D &address_space_memory) { error_report("Wrong target AS \"%s\", only system memory is allowe= d", iotlb->target_as->name ? iotlb->target_as->name : "no= ne"); + vfio_set_migration_error(-EINVAL); return; } =20 @@ -667,6 +682,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOM= MUTLBEntry *iotlb) "0x%"HWADDR_PRIx") =3D %d (%s)", container, iova, iotlb->addr_mask + 1, ret, strerror(-ret)); + vfio_set_migration_error(ret); } } out: @@ -1212,7 +1228,7 @@ static void vfio_listener_region_del(MemoryListener *= listener, } } =20 -static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool st= art) +static int vfio_set_dirty_page_tracking(VFIOContainer *container, bool sta= rt) { int ret; struct vfio_iommu_type1_dirty_bitmap dirty =3D { @@ -1220,7 +1236,7 @@ static void vfio_set_dirty_page_tracking(VFIOContaine= r *container, bool start) }; =20 if (!container->dirty_pages_supported) { - return; + return 0; } =20 if (start) { @@ -1231,23 +1247,34 @@ static void vfio_set_dirty_page_tracking(VFIOContai= ner *container, bool start) =20 ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); if (ret) { + ret =3D -errno; error_report("Failed to set dirty tracking flag 0x%x errno: %d", dirty.flags, errno); } + + return ret; } =20 static void vfio_listener_log_global_start(MemoryListener *listener) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + int ret; =20 - vfio_set_dirty_page_tracking(container, true); + ret =3D vfio_set_dirty_page_tracking(container, true); + if (ret) { + vfio_set_migration_error(ret); + } } =20 static void vfio_listener_log_global_stop(MemoryListener *listener) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + int ret; =20 - vfio_set_dirty_page_tracking(container, false); + ret =3D vfio_set_dirty_page_tracking(container, false); + if (ret) { + vfio_set_migration_error(ret); + } } =20 static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, @@ -1323,19 +1350,18 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifi= er *n, IOMMUTLBEntry *iotlb) VFIOContainer *container =3D giommu->container; hwaddr iova =3D iotlb->iova + giommu->iommu_offset; ram_addr_t translated_addr; + int ret =3D -EINVAL; =20 trace_vfio_iommu_map_dirty_notify(iova, iova + iotlb->addr_mask); =20 if (iotlb->target_as !=3D &address_space_memory) { error_report("Wrong target AS \"%s\", only system memory is allowe= d", iotlb->target_as->name ? iotlb->target_as->name : "no= ne"); - return; + goto out; } =20 rcu_read_lock(); if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL)) { - int ret; - ret =3D vfio_get_dirty_bitmap(container, iova, iotlb->addr_mask + = 1, translated_addr); if (ret) { @@ -1346,6 +1372,11 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifie= r *n, IOMMUTLBEntry *iotlb) } } rcu_read_unlock(); + +out: + if (ret) { + vfio_set_migration_error(ret); + } } =20 static int vfio_ram_discard_get_dirty_bitmap(MemoryRegionSection *section, @@ -1438,13 +1469,19 @@ static void vfio_listener_log_sync(MemoryListener *= listener, MemoryRegionSection *section) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); + int ret; =20 if (vfio_listener_skipped_section(section)) { return; } =20 if (vfio_devices_all_dirty_tracking(container)) { - vfio_sync_dirty_bitmap(container, section); + ret =3D vfio_sync_dirty_bitmap(container, section); + if (ret) { + error_report("vfio: Failed to sync dirty bitmap, err: %d (%s)"= , ret, + strerror(-ret)); + vfio_set_migration_error(ret); + } } } =20 --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759316; cv=pass; d=zohomail.com; s=zohoarc; b=Rbmiz2iXIwaafUkF5takadO2NYL9CancsG3kxVZZtnWtW+iEVdiw2yCfyOg1mUzaNEEhMf53B7P1COve37Wva11o/Q2sLEMU1k2WMT6BPQX6HGyhbjJgcIKGmpsRbzCwAUJGAIimBxx7BZ9ojrOVK1boixwFp960vaRjCveotIU= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759316; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Y/uiyTM0m3IoX61iU1bXPh+iEIQYz5WiCCIPcMEA0xY=; b=Sw/fXr1YpQv3biOquTukoHy6bvgXTk88nWOV/7w6OvvbkSZNM8ZHYetlHZOviWqSGem8yo65CEBNHB5StxiEuI8P0+a5JNVZyigiQtzqWVPwPl8QaWMTZ2qOM2NgJH1KfRa7238PZoTyD9Rj4xpCdC/E/6dhP2D5znx4Z+sp4Q8= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759316578204.12635477120352; Thu, 26 Jan 2023 10:55:16 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KZ-0004Yp-BM; Thu, 26 Jan 2023 13:50:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KX-0004YD-SS for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:37 -0500 Received: from mail-dm6nam11on2086.outbound.protection.outlook.com ([40.107.223.86] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KV-0002di-KW for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:37 -0500 Received: from BN9PR03CA0792.namprd03.prod.outlook.com (2603:10b6:408:13f::17) by PH7PR12MB6834.namprd12.prod.outlook.com (2603:10b6:510:1b4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:32 +0000 Received: from BN8NAM11FT025.eop-nam11.prod.protection.outlook.com (2603:10b6:408:13f:cafe::75) by BN9PR03CA0792.outlook.office365.com (2603:10b6:408:13f::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:31 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT025.mail.protection.outlook.com (10.13.177.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:31 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:16 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:15 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:11 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bvTgBxfO6dDDfG6iNO+Z4zw/lZin0UdpXFLEMfbNwv1n8rgxHJG25PsOVRa7pRET9gDQphBjwNca/5pOyUVMac5FPNM4SOnhz07OK3nIcpMBTf33Fd8iCDZkgVCDdk3TNtJNbS9mKKRfN2bc0qMkMhyywuo347Jgl+ZYwpKe64lzN2di+jOLfyZXIiFsOyPmrHHkV1y0t2LS1v/aKjJBgtAofcH7CdqA2bOzAZbidBLRTAq3PD189lA31vazBNlsJXeuIMZghUll1oqPW84Y+U8ATRnT/ubwLHkEVExGMlX4wjX3+3209NdKKKB3VleD9kiszlKyeZfYm83T5FO4/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Y/uiyTM0m3IoX61iU1bXPh+iEIQYz5WiCCIPcMEA0xY=; b=PtHHBpX4Va1ZyhioCRpq0R7YXZ/R2QNaO8m4em6yDlgGleTH7S+ir4qDxUkU1Lgt66C393fopFC/CwmTLaAuP6eUYD1MpaUKbvlulgk64tRu357vcqplCNMVtXYOXtKzFNzrAYmncW5YC53TRGZDqHuABf18SH8lBvq/oQ0DVcxzNW9NIcLIpvZHSzxSYu8V1KvSSogcgbpIvUTG6RVq7Hx/clrqAf+BJVU7Tf27azn32cyak60DftvKcneFZvDFt08A0yIXBwytM2zVFSvIMwyP1Cs2R1NMTIBJahSu7r7y7lhC32OrSQ7oMoyK9G5z/UVeoT+kA2ulbSF1NHqmnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y/uiyTM0m3IoX61iU1bXPh+iEIQYz5WiCCIPcMEA0xY=; b=gzZcOoYLokbRfGEWIP5PM8rZ7pqtkkg7YemRU4XY7buIFyqAizmmXVT6mJTQyK5QyftFD2UpXZzVfL8uE3sdE58E/jzaGQBS8gdhDmElBUy/IsRGUWbKWO1vhqT86RxcTxj24lgDQDFgdF0e62TZTSe9evFFWKw1/rEOwO0YGUAlixg4kWHuk9m3A+AB1wb14NlecIwLQMgRw4nkblwuqUMukoXjpu5Lostzy4zvcxxj34xubSmj2KqtdPEzZmzVfWkRHT+boe837glgfyAA1N3XyOa7guOBE+QmcSp+hG40HK4ZSRnnBPJPaW6r7uIsFeRoWF7xwLQZluxqC4OiHg== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 05/18] vfio/common: Add VFIOBitmap and (de)alloc functions Date: Thu, 26 Jan 2023 20:49:35 +0200 Message-ID: <20230126184948.10478-6-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT025:EE_|PH7PR12MB6834:EE_ X-MS-Office365-Filtering-Correlation-Id: 76da9b2f-fa43-49d7-ca1d-08daffce3380 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hBuZulqTmWd95tXcH/MEnFizgI3MH6iv0qvlqM/ruwGqDxziscj0k5GixmgQ9ygK8Bh4doGyy2dJQlBmlJ4GLh2gWAUG1MZtjpmLV2Zc+Wf8gQbsWOTU0rkaYq7iq4E9Nz0Rw/CDnen3L1wNP0ajHr+7CjeX90RZs7MsM/gv02i4or3swFlD0/SP9LzB0HDhZ7xrrotCEXyjp1QjQ6K5djY4gUcCbfpSJFRu0jIDmDj/QEnSjO84o3ckOyR3eJFgtKYYV7b6g9EuNwwopu1BuWtXOCxOVKElA+YvkB20tGulZVTCbIUzSGe7Owk8nrTQ7O3wcKz1m2SVbHur9R/USGS42yH5/zkYMBOcsck3fw3sXX8ct8wOVD6u4Lm6duEAYFpgJwSKSa8iR041No+kd84GMSDOWJpN+obTgEUQ8Bcn8tut7xOhCpNKecUMy98Buef5jeE3N4Bo8R/lbZY3OZ5Pi8AEdGWZNY3PILCYKbCMC0ocB7Vd/U8pwYimFMGHH2dRX66/A9HBkg5d3efTjyfY2fLE95EFWMoW+icNWcSK42nV6tWuMid1RW0GT1fX8KTI6E8hpP2jAa6mKVOSZvgS1U5mtPv53dlRVLDPxxmM75g+x5oR/Fj4BjlXu2I84Ft5l82N3ajJ+va95XQ3A/rQR3W4udjfU0Syo/KRuK8KkO6RVkmm4etkq9uvwzodDtVCrjy9paYM5sg1CRy//A== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(136003)(346002)(39860400002)(396003)(451199018)(40470700004)(46966006)(36840700001)(82310400005)(356005)(1076003)(2616005)(36860700001)(7636003)(40460700003)(2906002)(316002)(478600001)(82740400003)(54906003)(36756003)(8936002)(7696005)(41300700001)(66899018)(86362001)(40480700001)(426003)(83380400001)(186003)(26005)(6666004)(336012)(4326008)(70586007)(5660300002)(70206006)(7416002)(8676002)(6916009)(47076005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:31.6739 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 76da9b2f-fa43-49d7-ca1d-08daffce3380 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT025.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6834 Received-SPF: softfail client-ip=40.107.223.86; envelope-from=avihaih@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759317851100001 Content-Type: text/plain; charset="utf-8" There are already two places where dirty page bitmap allocation and calculations are done in open code. With device dirty page tracking being added in next patches, there are going to be even more places. To avoid code duplication, introduce VFIOBitmap struct and corresponding alloc and dealloc functions and use them where applicable. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 89 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 60 insertions(+), 29 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 8e8ffbc046..e554573eb5 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -319,6 +319,41 @@ const MemoryRegionOps vfio_region_ops =3D { * Device state interfaces */ =20 +typedef struct { + unsigned long *bitmap; + hwaddr size; + hwaddr pages; +} VFIOBitmap; + +static VFIOBitmap *vfio_bitmap_alloc(hwaddr size) +{ + VFIOBitmap *vbmap =3D g_try_new0(VFIOBitmap, 1); + if (!vbmap) { + errno =3D ENOMEM; + + return NULL; + } + + vbmap->pages =3D REAL_HOST_PAGE_ALIGN(size) / qemu_real_host_page_size= (); + vbmap->size =3D ROUND_UP(vbmap->pages, sizeof(__u64) * BITS_PER_BYTE)= / + BITS_PER_BYTE; + vbmap->bitmap =3D g_try_malloc0(vbmap->size); + if (!vbmap->bitmap) { + g_free(vbmap); + errno =3D ENOMEM; + + return NULL; + } + + return vbmap; +} + +static void vfio_bitmap_dealloc(VFIOBitmap *vbmap) +{ + g_free(vbmap->bitmap); + g_free(vbmap); +} + bool vfio_mig_active(void) { VFIOGroup *group; @@ -421,9 +456,14 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *contai= ner, { struct vfio_iommu_type1_dma_unmap *unmap; struct vfio_bitmap *bitmap; - uint64_t pages =3D REAL_HOST_PAGE_ALIGN(size) / qemu_real_host_page_si= ze(); + VFIOBitmap *vbmap; int ret; =20 + vbmap =3D vfio_bitmap_alloc(size); + if (!vbmap) { + return -errno; + } + unmap =3D g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); =20 unmap->argsz =3D sizeof(*unmap) + sizeof(*bitmap); @@ -437,35 +477,28 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *conta= iner, * qemu_real_host_page_size to mark those dirty. Hence set bitmap_pgsi= ze * to qemu_real_host_page_size. */ - bitmap->pgsize =3D qemu_real_host_page_size(); - bitmap->size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / - BITS_PER_BYTE; + bitmap->size =3D vbmap->size; + bitmap->data =3D (__u64 *)vbmap->bitmap; =20 - if (bitmap->size > container->max_dirty_bitmap_size) { - error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, - (uint64_t)bitmap->size); + if (vbmap->size > container->max_dirty_bitmap_size) { + error_report("UNMAP: Size of bitmap too big 0x%"PRIx64, vbmap->siz= e); ret =3D -E2BIG; goto unmap_exit; } =20 - bitmap->data =3D g_try_malloc0(bitmap->size); - if (!bitmap->data) { - ret =3D -ENOMEM; - goto unmap_exit; - } - ret =3D ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); if (!ret) { - cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->da= ta, - iotlb->translated_addr, pages); + cpu_physical_memory_set_dirty_lebitmap(vbmap->bitmap, + iotlb->translated_addr, vbmap->pages); } else { error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); } =20 - g_free(bitmap->data); unmap_exit: g_free(unmap); + vfio_bitmap_dealloc(vbmap); + return ret; } =20 @@ -1282,7 +1315,7 @@ static int vfio_get_dirty_bitmap(VFIOContainer *conta= iner, uint64_t iova, { struct vfio_iommu_type1_dirty_bitmap *dbitmap; struct vfio_iommu_type1_dirty_bitmap_get *range; - uint64_t pages; + VFIOBitmap *vbmap; int ret; =20 if (!container->dirty_pages_supported) { @@ -1292,6 +1325,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *cont= ainer, uint64_t iova, return 0; } =20 + vbmap =3D vfio_bitmap_alloc(size); + if (!vbmap) { + return -errno; + } + dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); =20 dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); @@ -1306,15 +1344,8 @@ static int vfio_get_dirty_bitmap(VFIOContainer *cont= ainer, uint64_t iova, * to qemu_real_host_page_size. */ range->bitmap.pgsize =3D qemu_real_host_page_size(); - - pages =3D REAL_HOST_PAGE_ALIGN(range->size) / qemu_real_host_page_size= (); - range->bitmap.size =3D ROUND_UP(pages, sizeof(__u64) * BITS_PER_BYTE) / - BITS_PER_BYTE; - range->bitmap.data =3D g_try_malloc0(range->bitmap.size); - if (!range->bitmap.data) { - ret =3D -ENOMEM; - goto err_out; - } + range->bitmap.size =3D vbmap->size; + range->bitmap.data =3D (__u64 *)vbmap->bitmap; =20 ret =3D ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); if (ret) { @@ -1325,14 +1356,14 @@ static int vfio_get_dirty_bitmap(VFIOContainer *con= tainer, uint64_t iova, goto err_out; } =20 - cpu_physical_memory_set_dirty_lebitmap((unsigned long *)range->bitmap.= data, - ram_addr, pages); + cpu_physical_memory_set_dirty_lebitmap(vbmap->bitmap, ram_addr, + vbmap->pages); =20 trace_vfio_get_dirty_bitmap(container->fd, range->iova, range->size, range->bitmap.size, ram_addr); err_out: - g_free(range->bitmap.data); g_free(dbitmap); + vfio_bitmap_dealloc(vbmap); =20 return ret; } --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759092; cv=pass; d=zohomail.com; s=zohoarc; b=WyMsUeB/kGYWQVI/XBxPmwtDpWuAaf4nZwc+bCPBXQ5Vs7VsJQWREbmVlanYqUv2a3+8r8bh/aM8V8tNYCdEYctwKa1dUPmSYHeU+XgO9QlUlYAUe/ssqJzl+Q++QgJNAxl2TLZnscaU/zR05/44yoaYMdc9dqs2R1lJKs5PaWk= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759092; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=PkntuEtOlJY33+Zvdent9wgvbmBGEGFIQIBt4XgDqjM=; b=np65qP8xmMhqzoo9QQ/EsLLIcF2Rh2W9Fi2TPoX8CcB46EPvbhT3WZdsSrieReHLgLKr5oDm8UvAVTlbgZikgMxKUAoZD2ch9HQNNzRwTc69ofaueIX/I402lqEvBsmsCiX9orHaeIPYovEYO/NOyrwtuMZXvyMwB1MijZIOalI= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759092457977.0204747491713; Thu, 26 Jan 2023 10:51:32 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LF-0005Fz-AJ; Thu, 26 Jan 2023 13:51:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kd-0004cJ-EU for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:54 -0500 Received: from mail-bn8nam11on20601.outbound.protection.outlook.com ([2a01:111:f400:7eae::601] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kb-0002jb-At for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:42 -0500 Received: from BN1PR10CA0012.namprd10.prod.outlook.com (2603:10b6:408:e0::17) by BN9PR12MB5196.namprd12.prod.outlook.com (2603:10b6:408:11d::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:37 +0000 Received: from BN8NAM11FT004.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e0:cafe::dc) by BN1PR10CA0012.outlook.office365.com (2603:10b6:408:e0::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:37 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT004.mail.protection.outlook.com (10.13.176.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:37 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:20 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:20 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:16 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SPSM8dbPw8ox1Udk2HLGhXQdTvN3UgZLHwVGBdKj9ONsvYCYQlQOt17e10o1t0cYiOV3M8CQLRS8sFLSjZY9IGwXFvjV5vv0Q7N4H73L+HbXdPMUA7W0bRbdiq0CML2r7NuYyeHpIHMNq4z21uTakhT67xFOZVVESbcSBjuTzGxnpzYEai6qmQ2x2Il0q/BCHU2vXQqlMXHLm5RioCzdnTVZaulq8fxkRKVhykCtHbkGFDZn/wFMdTxIqDCYelSBEzP3ddMhhFEMiZHuzLDVNrmu0g2y0c+noqnDFfu9nZE8TLSh1sKpNBoVix9RikXkbNSPsWHkXwMBQaW3M2ou7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PkntuEtOlJY33+Zvdent9wgvbmBGEGFIQIBt4XgDqjM=; b=XOW1Tmd3xUjnLYP4KpjD+z3kKa9zgMXCXIlAB8T3bvOZBbQoebMyi3H2cSxcxkti5RzTgBzXGPcFKVTyHbxmcM6FyooA6TsF2uGSAlMdJkBjrecjfku2mbuotJPRP1CzaaTL3OrISC3rqE0SJSbKVBmyxKhuDJ1vwOdkITEl3sPcxFPcWY+X7qlp7g4B9GysxlNJnK9fuy6Qye1X57UKkBtXUR9grL3+/d7OQnTFPJizCOKpDFbH4Q4D/gcMuEQuRNe/EjvHExuPRNgs9KUIHbUQxwMa+oFACf2YVnCSheAGLkdnQsL039YLPUnXivbZF+JzD8jTyCajtqATc/awfw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PkntuEtOlJY33+Zvdent9wgvbmBGEGFIQIBt4XgDqjM=; b=Qnx/IOhkvO+mJZb80zqFs2z6feqOlo8styCuqN9Xyu7+WAlwtkF1zPcmC8PKBFFOLPlB+kzn4TbmgyRwoI5bot/RiAALEtPFpUFN740ZTnDgPNQTIMIgW00qXdZSNvk+HdyINpMkVY/K9x3loWVbSBBy8TLY2pBpoRPs/qcMq9TZFumiThEuyCYKXjHfERaZhQ4mtIKh5e/BwUNLiC6aKoRyt7jgWUiHSIPnYs7ggd3T6wDWu+5hQWwgDWfVI+Gi6/o/V6jiMHZffyk0uBq6Va1VX/SdNGcjfKsvK9RnqITdnwmXFjdoeJNRv78f1TNHaWt/Y36GgA9HV2E57AGaWA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 06/18] util: Add iova_tree_nnodes() Date: Thu, 26 Jan 2023 20:49:36 +0200 Message-ID: <20230126184948.10478-7-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT004:EE_|BN9PR12MB5196:EE_ X-MS-Office365-Filtering-Correlation-Id: 2d9e5348-0566-4b37-0bcc-08daffce36bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rpX5Y/W7JHsq6rXXZDnNBuAJEllupXoHysH4DI3caWFQJkaoYDuK6jyLgf52JTqzLaq6gh9gw2YLfiAJ3DZ7xW4UqhFGIbog/43NJHfJhuwNgJx/7XlSeJWMTk5/IzAOEfdnF0fUt0oyq6cK2PsHuQprSKA/NnwVqYYzRpgeO2u+2QGx1xYr1Qz/NxXD7Aj80nYYFXG2M3G0PY5NvTYCRA7x+iadLV8UZpchaqnJsIzEazORzyaGqO+IouKg/c1xGS0bjV6S8Qn5Z25jhnUqUliwJ+BxCdUySJdDTB333XNu0TtK8JQqxXkmTNY7eQ0ChJSm/JhaiytlRcqBpUFwm2OlQC2bQhZVo+mcaEs0QQq8URrnFE75xcmGSIehAwhNJQogI3NtOjoVYt070t2lei/XOlPD0qb222Nwx431KBoMzQrUeXH+azHzHaxYtB+F0LVsd1IEy9g907EECLugeWdFBjmEfqoe9wEReV7gDYtc746qdG9oPmWFJ95eBCo3DmrgmD9gCI7IALmwff+xdiz5p3T6kh7FAsu3sEMWksryME4ph60qs1eb18c8z7GGk8tctO3lWQlupagLG6MFrAUNCBWaTSaD6miGR5wk3hyHZDV3KVBYlEGtJzBtxBrPsUyoCBToHBtjX+GV2eYG57bNdSwnXkXSWflw7Bjr8OMiuHscWs9OErqoCtEQchCq9oX/uchDyblpu2raOnFvvQ== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(346002)(396003)(39860400002)(136003)(451199018)(46966006)(40470700004)(36840700001)(70206006)(316002)(54906003)(41300700001)(8676002)(7636003)(356005)(6916009)(70586007)(5660300002)(7416002)(40460700003)(86362001)(36756003)(8936002)(4326008)(40480700001)(2906002)(36860700001)(82740400003)(1076003)(26005)(186003)(478600001)(7696005)(426003)(6666004)(47076005)(82310400005)(336012)(2616005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:37.1045 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2d9e5348-0566-4b37-0bcc-08daffce36bb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT004.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5196 Received-SPF: softfail client-ip=2a01:111:f400:7eae::601; envelope-from=avihaih@nvidia.com; helo=NAM11-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759094184100001 Content-Type: text/plain; charset="utf-8" From: Joao Martins Add iova_tree_nnodes() which returns the number of nodes in the IOVA tree. Signed-off-by: Joao Martins Acked-by: Peter Xu --- include/qemu/iova-tree.h | 11 +++++++++++ util/iova-tree.c | 5 +++++ 2 files changed, 16 insertions(+) diff --git a/include/qemu/iova-tree.h b/include/qemu/iova-tree.h index 8528e5c98f..7bb80783ce 100644 --- a/include/qemu/iova-tree.h +++ b/include/qemu/iova-tree.h @@ -164,4 +164,15 @@ int iova_tree_alloc_map(IOVATree *tree, DMAMap *map, h= waddr iova_begin, */ void iova_tree_destroy(IOVATree *tree); =20 +/** + * iova_tree_nnodes: + * + * @tree: the iova tree to consult + * + * Returns the number of nodes in the iova tree + * + * Return: >=3D0 for the number of nodes. + */ +gint iova_tree_nnodes(IOVATree *tree); + #endif diff --git a/util/iova-tree.c b/util/iova-tree.c index 536789797e..6141a6229b 100644 --- a/util/iova-tree.c +++ b/util/iova-tree.c @@ -280,3 +280,8 @@ void iova_tree_destroy(IOVATree *tree) g_tree_destroy(tree->tree); g_free(tree); } + +gint iova_tree_nnodes(IOVATree *tree) +{ + return g_tree_nnodes(tree->tree); +} --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759061; cv=pass; d=zohomail.com; s=zohoarc; b=gg6kVRj5vrrOb6PjR2VWx3+zmIak5hXGVS7vXN2lKtTWX6OVPdVsDASlzzMWMp8SvCsDhu+CWuks3vaGq0OMhWTuXVoVzW9vk7a6MeB/jYODgeEUUL8b19X/REb/9tc5X24tfVBrAMrqurtJQ89X89VGEAK6+t4tZXnwvw1/xIw= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759061; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=1+b4GN+Lj86ioZsZiH/9lvinX0evvu9nbrnzBNdUx9o=; b=F0Qfb6b++AzziyuBpx4CMW+hZOYdFJPKD7PBNDFplRhfusgKBYDp7WOSGN6tP4rRPu2NQ8lRgVFN5c3NwuB4qNZVtYZa5Kr5h25VxaWuMBcY3c9hWgNxoKLEqYEZQjv7jXjn2bBmGGJ0Vjaw2Z0DiO5KGk/3fxTIu5GLTxZAXmU= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759061403270.14142629095306; Thu, 26 Jan 2023 10:51:01 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7KU-0004Xh-Fq; Thu, 26 Jan 2023 13:50:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KT-0004XH-IN for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:33 -0500 Received: from mail-bn8nam11on20600.outbound.protection.outlook.com ([2a01:111:f400:7eae::600] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7KQ-0002Yv-31 for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:33 -0500 Received: from MW4PR03CA0335.namprd03.prod.outlook.com (2603:10b6:303:dc::10) by DS7PR12MB6165.namprd12.prod.outlook.com (2603:10b6:8:9a::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Thu, 26 Jan 2023 18:50:27 +0000 Received: from CO1NAM11FT086.eop-nam11.prod.protection.outlook.com (2603:10b6:303:dc:cafe::ea) by MW4PR03CA0335.outlook.office365.com (2603:10b6:303:dc::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:26 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT086.mail.protection.outlook.com (10.13.175.73) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:50:26 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:25 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:24 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:20 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ByXg3EeduBnLh/ZeGBryBb/Ysp3D0EsVOi0oxe/fwUQEtT+H+v/OWgBIyYAfR4na2FfRgpticw8dO0BNh1ayOKUwu1D7bhLV9pIWCg6TFbx+jd9UhzRpz3QCCD/Hy8C4akRUc2KdS8C5jjhe/W6HDljFP1WNJa50y2NcD7X+IB8rgRDON+0fRwCNfVZSwWgvmKxizQv/k+n2oJdMIglnIyvfCVbEtl2lu2UbYVf3+kluAVVU+2a14fLdgJRQddKuOHiWdibJd+CTE2G6tUhrkvd+2ujROl0NViRA3y+FbNz4eFfD6+gWgCo6CK3i95YV/L9fl8pWudKtRdppE+vO0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1+b4GN+Lj86ioZsZiH/9lvinX0evvu9nbrnzBNdUx9o=; b=Oeycili6HGoB96ImYOnQj5A2oHuskf9eXxVb0aKbwKW/k7Xqb2fn8vuoo2834g0aU7xhIDVVXw12yukHbMskjf9FbSPnK/yP50yIbI//If79dctrJ98luk0+ogCwsfobcRASH0gm4a/dhalWdam2yNLj2KthTIaE1wksxAcMnlaSVBNq4+Q1TSCM6cAlEhPmBc3V3/kPa82Ncz8Av3N1uDLM7UAK7xnFf43XfTGUh/qAjvK12psSIrlo8yIOHv+UmuSZ1dMvjeDKyBxNt34kEEDqXr7G516N1g76bYQZUJ00d6DSa2lYELUv6XRxtvx+2okgivbjqVlFVPj1wEGHGA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1+b4GN+Lj86ioZsZiH/9lvinX0evvu9nbrnzBNdUx9o=; b=W84oNAHJsyliLYx/Cw2L62JiAzeUhN6M/CWttEhYd1R2QvjfU4JBASU1HOkNEMMtuipUuL6a8+8NsohtMuVZSxyLBWEi7/hZOg2UCHmzOHB0gvYa6hyAclrOLBmFopDiJPpFp8OPrFAecHfOmGmdUa8UD6V36844FcE6GLYrg/xnCmFz7aXXc4I/cFPLx0XNnZlncWfwjVdstpOgQyws74aeH1EtBWnkOvgWvjSKxb5JupSxede7n6e+F1+taqDDFtq0nzxLI/oV6bGpiwyv8LMVWjzWNSfNTAD4wORwFxiJLiwZlSHc8GEnevWBl+hQIyJl5xI/S5ZlIzPe/OHAjA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 07/18] util: Extend iova_tree_foreach() to take data argument Date: Thu, 26 Jan 2023 20:49:37 +0200 Message-ID: <20230126184948.10478-8-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT086:EE_|DS7PR12MB6165:EE_ X-MS-Office365-Filtering-Correlation-Id: 0829c7bd-d53b-46d5-9a89-08daffce306c X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ceZD+x6UspjGNo4cvqf/xqH399jR6hQ6sGsNLnicx97sQZlGoueF/GWUxuv8vZHVPHQRDDeN9GW+awPr+YU2FghPZhixa00mbXuUb5uvXwgsoBv6YVbPTZpgdw5r2aDrBJjt0b1O7smziiijJJwVSNthiamMcaW0CJQIX28tfrNCymN27x8dqmIzi9Tq9OhpuhxHt+BhK3cfqZqd6v2EXpbUZPNspGu5ZdzIUsM0PNF5mbIYGlpmbbB1gj/SvivyBMsF8aGbiqulN4tmY7Y/jANZdd/+ZzsEBJTnO1Ws9siGEkp63vhbEGZrMUe02ldWwtXqa6ZVCvV+/DtJwkjZ/meDP6dGxlnq5qK6TCCq6EeoMYmnK0nsqWDMz7LOVlzHUrfdcU6yUglV3b1jtAaybZjROib6aaJRfv+p58sbHirDzZD+kR46Lk/APad7p48QgukSeOp73W4mCDzKqm+ZYiwexWpslFr+pcWjkcw2ty6QG2pqKq66M+mPRDFRJvck/TlNlFPHLS1x2/nKVGxs0dFq5yocZhhZ1LvaeU+42ObddXZlhI4YlBc8dmmZgkd4MLPz/24oQ2lQuKx7+M/7VwHUJEfIcSYMR07urvG7BJVBPS4QKEgYGQTXRttPWZFpxUqBJhg9NbsLlE18A/vOkajDsQ16aHD2I1DPh6+QN5pWEovs1mN5Kro8+kMHPxyL7nnrtildtacFBTS6fdKyqg== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(396003)(136003)(346002)(376002)(451199018)(36840700001)(46966006)(40470700004)(36860700001)(26005)(356005)(83380400001)(7636003)(5660300002)(82740400003)(7416002)(7696005)(41300700001)(2906002)(86362001)(82310400005)(8936002)(4326008)(40480700001)(40460700003)(6666004)(316002)(186003)(336012)(8676002)(47076005)(2616005)(478600001)(54906003)(6916009)(70586007)(36756003)(426003)(1076003)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:26.6178 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 0829c7bd-d53b-46d5-9a89-08daffce306c X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT086.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6165 Received-SPF: softfail client-ip=2a01:111:f400:7eae::600; envelope-from=avihaih@nvidia.com; helo=NAM11-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759062015100002 Content-Type: text/plain; charset="utf-8" From: Joao Martins Extend iova_tree_foreach() to take data argument to be passed and used by the iterator. While at it, fix a documentation error: The documentation says iova_tree_foreach() returns a value even though it is a void function. Signed-off-by: Joao Martins Acked-by: Peter Xu --- include/qemu/iova-tree.h | 8 +++++--- util/iova-tree.c | 18 ++++++++++++++---- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/include/qemu/iova-tree.h b/include/qemu/iova-tree.h index 7bb80783ce..1332dce014 100644 --- a/include/qemu/iova-tree.h +++ b/include/qemu/iova-tree.h @@ -38,7 +38,7 @@ typedef struct DMAMap { hwaddr size; /* Inclusive */ IOMMUAccessFlags perm; } QEMU_PACKED DMAMap; -typedef gboolean (*iova_tree_iterator)(DMAMap *map); +typedef gboolean (*iova_tree_iterator)(DMAMap *map, gpointer data); =20 /** * iova_tree_new: @@ -129,12 +129,14 @@ const DMAMap *iova_tree_find_address(const IOVATree *= tree, hwaddr iova); * * @tree: the iova tree to iterate on * @iterator: the interator for the mappings, return true to stop + * @data: data to be passed to the iterator * * Iterate over the iova tree. * - * Return: 1 if found any overlap, 0 if not, <0 if error. + * Return: None. */ -void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator); +void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator, + gpointer data); =20 /** * iova_tree_alloc_map: diff --git a/util/iova-tree.c b/util/iova-tree.c index 6141a6229b..9845427b86 100644 --- a/util/iova-tree.c +++ b/util/iova-tree.c @@ -42,6 +42,11 @@ typedef struct IOVATreeFindIOVAArgs { const DMAMap *result; } IOVATreeFindIOVAArgs; =20 +typedef struct IOVATreeIterator { + iova_tree_iterator fn; + gpointer data; +} IOVATreeIterator; + /** * Iterate args to the next hole * @@ -151,17 +156,22 @@ int iova_tree_insert(IOVATree *tree, const DMAMap *ma= p) static gboolean iova_tree_traverse(gpointer key, gpointer value, gpointer data) { - iova_tree_iterator iterator =3D data; + IOVATreeIterator *iterator =3D data; DMAMap *map =3D key; =20 g_assert(key =3D=3D value); =20 - return iterator(map); + return iterator->fn(map, iterator->data); } =20 -void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator) +void iova_tree_foreach(IOVATree *tree, iova_tree_iterator iterator, + gpointer data) { - g_tree_foreach(tree->tree, iova_tree_traverse, iterator); + IOVATreeIterator arg =3D { + .fn =3D iterator, + .data =3D data, + }; + g_tree_foreach(tree->tree, iova_tree_traverse, &arg); } =20 void iova_tree_remove(IOVATree *tree, DMAMap map) --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759195; cv=pass; d=zohomail.com; s=zohoarc; b=Aj/Hlbgx3FkRXCNJ+P/uZ/4HZfSVDpzTJtWO3Zxsdk07B8ExV21Or4V/Kvg0kpr8pECsuXvlFKnzjLDc+KEodWa9u2JRZeUupxWCaie8spDOZFZYxfmuONdE/s3CfT4AZFuUWsrSI1WUHnO65Ql/ERxUCx7IGyizd3BOuA3yoXc= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759195; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=nHUZ5+DutMpYKJqRocWZ30J9p5qBIDWTIHGKUSpMLGM=; b=IMSvuEqHhJj/itJa1P2mN41sIsqXfdGgrNTRBT07XMrB/utPRZZtEhqXa1wZtVRmb6/geQX0d5im1wZhxV7EcMrg3eIKwKjynquxmfiGqWLlZhXUSWat6osRtqL1zZH5KC+x7Kho63AOkp3A6F4YdOC6Yysrl5xnRyiVC6yS5f0= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759195093627.121594345328; Thu, 26 Jan 2023 10:53:15 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7L0-00050z-Va; Thu, 26 Jan 2023 13:51:09 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kj-0004gh-JZ for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:57 -0500 Received: from mail-bn7nam10on2079.outbound.protection.outlook.com ([40.107.92.79] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kh-0002oY-CD for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:49 -0500 Received: from BN9PR03CA0784.namprd03.prod.outlook.com (2603:10b6:408:13f::9) by SA1PR12MB7411.namprd12.prod.outlook.com (2603:10b6:806:2b1::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:42 +0000 Received: from BN8NAM11FT025.eop-nam11.prod.protection.outlook.com (2603:10b6:408:13f:cafe::f0) by BN9PR03CA0784.outlook.office365.com (2603:10b6:408:13f::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:42 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT025.mail.protection.outlook.com (10.13.177.136) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:41 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:30 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:29 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:25 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y7RTsDEHCnK4pTAomP5WhlYzDNLAu8Pm/TSgiAXmVIpsn6ONp3Zv3PF+T7Vzbpw4PK8PQ5a0yvp0mvp1JRlpJI3DpbBGDVzcq73VEccb+ZE3Ux/wXH649QUbD0uOFw3yg4ccNguBYcXVA9qO5Lv3XZhzsodWOFNZxo9TdGbRDdLLTHxI++nI85OPanWKqYmIWSWB1eXWxcN/b7a4VkdTdDs48zCe2XZLEVwN5u/mbf5ZDflSBKt9U+MQGweao3ysg/hLwtTHvBwpRZr+AdT9ZxCSZ52VeoKKy0dnwPl8JkOfvf6/Nas6f0oUnQVGrQ8qd/wNiesou8aMosl8mdHkUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nHUZ5+DutMpYKJqRocWZ30J9p5qBIDWTIHGKUSpMLGM=; b=l/TvXyiT9FCtKwFanzthG3kJrOYyUWDQ+pKUajUTzOzmq39CFlJeCyK93ANDb5YOI+g7xTLZA3BeEYxlkQfXMj8yRf3kZpiQCHuhgcqmKf5sTCO2jcewXtzkKllfTf/hhUMJf4G7HQg1kdyB5whWyiZXKv5B49f/3Jbg97LHjDV2tMrvrp0EcRc61v8YR3PehpxVjY3mu2pgGSe7FqpyoubjeuBWJTHhG2sigcxrim/RaEyO6m0AwtWwIgCm14yMQ0lK/jOM//GPcWNL5hGzYwwp9zSOeSigOileROEljHZgyzBr2MIta7YsabXJFV8zYEILr7LR6t/DQ3CQE1rOZQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nHUZ5+DutMpYKJqRocWZ30J9p5qBIDWTIHGKUSpMLGM=; b=oemzGMmv0Qm9EnjjSvoagw641gBwRTCAvc7AgPvrH/EVsTwpCY/yeNr4rb6biv3NKa5/jEmYtZ4oy/C2hFcq7NlbbIYpbO5EyxOD6pBP2J9UeJM+s08nmrQlT1dSoDq/g57Uq586x+t8M+fwmsF694T2nnW8sPWEqHHAcBENxpkZddtb8eR+1EnJ2a/FfvX4fWfblPChfSBIldCYI47fvw2HQZ+5PgTHo3wS4AVNU9y2ODzt/NDxoM2SGLWoj9hd6T6pAW94AZ3pxU0WcEwgXf/Pt/z7Mxo8UgSs5dFxf9YMNuJUJaGAYt6d1pUmNSak43vZ49LO8D+mooaOQazCkg== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 08/18] vfio/common: Record DMA mapped IOVA ranges Date: Thu, 26 Jan 2023 20:49:38 +0200 Message-ID: <20230126184948.10478-9-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT025:EE_|SA1PR12MB7411:EE_ X-MS-Office365-Filtering-Correlation-Id: 26dec0fb-91a0-41eb-8031-08daffce39d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8bf51rHYnkc+FIxRYBK6X74DrEuwqrSNGZhPQe+FIh05AbvWNJfYhVTG4CpZ7W7SEvhTXnqpOeUvSftfKKnsCLBy40LA4MdaxwcnfKlfbZSL8jAXqmTwIy/X4e/YnSHQx4Uj7tfNUCnyqgytvSAIT8SujQsKMsoq2/UnPo8SlJo2UKeWrtlaWK6T2fBn1y0on+YM0auTLWstmvD5DT5da2yOoYBxRFUNmKQ3jpYz6WtMuuo6aW7e4n1v0af88ZvpErQxgtubvSEcEapjdIhUbI58nW6NcCndLhCB2K8mr67HIZp1kc73e67w3R7MikFEo5hC8XCWp+gOGsbcP/WLGDnXfdBimAxQgkNtDK3dEeaOH2OPHsCO+pK620jlVcJjaUdgaI+agjZlbWZa+leqHZF0X+K54SKH2qd6y6Ml8oZCPsie/9ppJypBWhsL+3i+p/ujD0Nix9f28GuWiN6if9NliFi28mFKv1LYKmz5ynsznlNp6oGBdhAsbOd4toabLe8awj9d0DHUv/Q0FhO7cGTUyCslkoA2d4iy3FU+8Yb9/T0mUkztDW+1cZhT3yA5TUKrGuL175xTDMYNYYmUgfVVYsqOilx0frNfX89jR1B7eqdqoqjVkk3IxmiVCrRcSdCCTffT0wFigJCz7BNJEizp8zP5oauKSfYDQBota3FRAQMQJmON4/f/dG6fbgl4KBtzs33I4bPu62chzpem5Q== X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(136003)(396003)(39860400002)(346002)(451199018)(46966006)(36840700001)(40470700004)(2906002)(5660300002)(356005)(83380400001)(336012)(426003)(7416002)(86362001)(40480700001)(40460700003)(6666004)(1076003)(82310400005)(70206006)(70586007)(478600001)(6916009)(7696005)(41300700001)(36860700001)(186003)(2616005)(26005)(7636003)(47076005)(316002)(36756003)(54906003)(8936002)(4326008)(8676002)(82740400003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:41.4855 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 26dec0fb-91a0-41eb-8031-08daffce39d3 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT025.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7411 Received-SPF: softfail client-ip=40.107.92.79; envelope-from=avihaih@nvidia.com; helo=NAM10-BN7-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759196738100001 Content-Type: text/plain; charset="utf-8" From: Joao Martins According to the device DMA logging uAPI, IOVA ranges to be logged by the device must be provided all at once upon DMA logging start. As preparation for the following patches which will add device dirty page tracking, keep a record of all DMA mapped IOVA ranges so later they can be used for DMA logging start. Note that when vIOMMU is enabled DMA mapped IOVA ranges are not tracked. This is due to the dynamic nature of vIOMMU DMA mapping/unmapping. Following patches will address the vIOMMU case specifically. Signed-off-by: Joao Martins Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 86 +++++++++++++++++++++++++++++++++-- 2 files changed, 86 insertions(+), 3 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 88c2194fb9..d54000d7ae 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -23,6 +23,7 @@ =20 #include "exec/memory.h" #include "qemu/queue.h" +#include "qemu/iova-tree.h" #include "qemu/notify.h" #include "ui/console.h" #include "hw/display/ramfb.h" @@ -94,6 +95,8 @@ typedef struct VFIOContainer { uint64_t max_dirty_bitmap_size; unsigned long pgsizes; unsigned int dma_max_mappings; + IOVATree *mappings; + QemuMutex mappings_mutex; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index e554573eb5..fafc361cea 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -43,6 +43,7 @@ #include "migration/misc.h" #include "migration/qemu-file.h" #include "sysemu/tpm.h" +#include "qemu/iova-tree.h" =20 VFIOGroupList vfio_group_list =3D QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -373,6 +374,11 @@ bool vfio_mig_active(void) return true; } =20 +static bool vfio_have_giommu(VFIOContainer *container) +{ + return !QLIST_EMPTY(&container->giommu_list); +} + static void vfio_set_migration_error(int err) { MigrationState *ms =3D migrate_get_current(); @@ -450,6 +456,51 @@ static bool vfio_devices_all_running_and_mig_active(VF= IOContainer *container) return true; } =20 +static int vfio_record_mapping(VFIOContainer *container, hwaddr iova, + hwaddr size, bool readonly) +{ + DMAMap map =3D { + .iova =3D iova, + .size =3D size - 1, /* IOVATree is inclusive, so subtract 1 from s= ize */ + .perm =3D readonly ? IOMMU_RO : IOMMU_RW, + }; + int ret; + + if (vfio_have_giommu(container)) { + return 0; + } + + WITH_QEMU_LOCK_GUARD(&container->mappings_mutex) { + ret =3D iova_tree_insert(container->mappings, &map); + if (ret) { + if (ret =3D=3D IOVA_ERR_INVALID) { + ret =3D -EINVAL; + } else if (ret =3D=3D IOVA_ERR_OVERLAP) { + ret =3D -EEXIST; + } + } + } + + return ret; +} + +static void vfio_erase_mapping(VFIOContainer *container, hwaddr iova, + hwaddr size) +{ + DMAMap map =3D { + .iova =3D iova, + .size =3D size - 1, /* IOVATree is inclusive, so subtract 1 from s= ize */ + }; + + if (vfio_have_giommu(container)) { + return; + } + + WITH_QEMU_LOCK_GUARD(&container->mappings_mutex) { + iova_tree_remove(container->mappings, map); + } +} + static int vfio_dma_unmap_bitmap(VFIOContainer *container, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb) @@ -550,6 +601,8 @@ static int vfio_dma_unmap(VFIOContainer *container, DIRTY_CLIENTS_NOCODE); } =20 + vfio_erase_mapping(container, iova, size); + return 0; } =20 @@ -563,6 +616,16 @@ static int vfio_dma_map(VFIOContainer *container, hwad= dr iova, .iova =3D iova, .size =3D size, }; + int ret; + + ret =3D vfio_record_mapping(container, iova, size, readonly); + if (ret) { + error_report("vfio: Failed to record mapping, iova: 0x%" HWADDR_PR= Ix + ", size: 0x" RAM_ADDR_FMT ", ret: %d (%s)", + iova, size, ret, strerror(-ret)); + + return ret; + } =20 if (!readonly) { map.flags |=3D VFIO_DMA_MAP_FLAG_WRITE; @@ -579,8 +642,12 @@ static int vfio_dma_map(VFIOContainer *container, hwad= dr iova, return 0; } =20 + ret =3D -errno; error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); - return -errno; + + vfio_erase_mapping(container, iova, size); + + return ret; } =20 static void vfio_host_win_add(VFIOContainer *container, @@ -2134,16 +2201,23 @@ static int vfio_connect_container(VFIOGroup *group,= AddressSpace *as, QLIST_INIT(&container->giommu_list); QLIST_INIT(&container->hostwin_list); QLIST_INIT(&container->vrdl_list); + container->mappings =3D iova_tree_new(); + if (!container->mappings) { + error_setg(errp, "Cannot allocate DMA mappings tree"); + ret =3D -ENOMEM; + goto free_container_exit; + } + qemu_mutex_init(&container->mappings_mutex); =20 ret =3D vfio_init_container(container, group->fd, errp); if (ret) { - goto free_container_exit; + goto destroy_mappings_exit; } =20 ret =3D vfio_ram_block_discard_disable(container, true); if (ret) { error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); - goto free_container_exit; + goto destroy_mappings_exit; } =20 switch (container->iommu_type) { @@ -2279,6 +2353,10 @@ listener_release_exit: enable_discards_exit: vfio_ram_block_discard_disable(container, false); =20 +destroy_mappings_exit: + qemu_mutex_destroy(&container->mappings_mutex); + iova_tree_destroy(container->mappings); + free_container_exit: g_free(container); =20 @@ -2333,6 +2411,8 @@ static void vfio_disconnect_container(VFIOGroup *grou= p) } =20 trace_vfio_disconnect_container(container->fd); + qemu_mutex_destroy(&container->mappings_mutex); + iova_tree_destroy(container->mappings); close(container->fd); g_free(container); =20 --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759316; cv=pass; d=zohomail.com; s=zohoarc; b=HIAghzi/aL2o95ea543jsmJCgZLX1kDL5jzpPCL9gSzoYZCmmNJntbTzXm9x55pZVtZ1T+13Bsmr/EdnIMHJE2UIKrwzqpmY9A2OnLgATubOeXMJ++FtQZmcxGa6uU3htRiaV3xKfv1sWHluyOsP1bQobDpDV+qILh+AwgRa2OI= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759316; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=VDhUpA/lzIC42tK94W5IVr5b7fPs8P3l4LlOB5P7rsA=; b=l9gI6ZtW28iGZJYAh4/9zh2R6Il8GLLIFvSVv0j48/Y+8OgBJ+0CroWQpq7kArKG5Vdr9WrxMkBCfFcGsrjg9mk1W3f7faRoc68xQBpFfWRBteh8m1hxnKHOLkBg/KAYpKjDoC32nDc2motYVPEHRI7Ef38clBwsdAYHIBqnVTk= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759316882111.99942915217628; Thu, 26 Jan 2023 10:55:16 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LI-0005gx-3O; Thu, 26 Jan 2023 13:51:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kj-0004gr-Lt for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:57 -0500 Received: from mail-dm6nam04on2047.outbound.protection.outlook.com ([40.107.102.47] helo=NAM04-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kh-0002ob-BD for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:49 -0500 Received: from BN1PR10CA0030.namprd10.prod.outlook.com (2603:10b6:408:e0::35) by CO6PR12MB5428.namprd12.prod.outlook.com (2603:10b6:5:35c::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:43 +0000 Received: from BN8NAM11FT004.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e0:cafe::c2) by BN1PR10CA0030.outlook.office365.com (2603:10b6:408:e0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:43 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT004.mail.protection.outlook.com (10.13.176.164) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:43 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:34 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:34 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:30 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RaVFjZ/KVmTVWp1o+AUk7gOPMMqkOMOQ++0CfKN6Ezie1gx1WJtZfCdR7M6NfvomhYd7NY0/PRzNcqMSbhnndDs0XN/p2BKs6DUkTeaBKyROrz9R+qvDW5JVfjgOQP9gZblrhe6kLxd1BBaeSjw1qmP6ecYDKlcZg1HiO4g0VKbngDJ/AN6JPs6793c2C1RpTpjepxTXZy1+SniZhszV1EUsoqbT5Jsay44x2HWuRWkRgEhS8PbEw5uvmn5ZO81qm8LPp3/0vY/NqP/Ldgi46z3gKbqiq5TbZ3ibt6n0cKPs1GmrbiIK4covUSYRGRxULFilolRn/B/dS68x6pzAFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VDhUpA/lzIC42tK94W5IVr5b7fPs8P3l4LlOB5P7rsA=; b=Lgc55ZRFWmr3XmCbX8rE7pzCE34I/K1Wc3S0rRCNM2MR4BxjvuiglpcLy3xL88b7WlKoa/Zeyzyy7PNLx0srsj0bQEQcmKRhbPVOV48gHv0sLc2eR+yfgZr6x6Lx3+02tN9fQ7U82G48IgoCaJOi2tX4/oqoGQw9ZnRT1xNSw+YtaDjmKAfBStaLW62/8xGFbE7EHY9XpA4wJaM6efiIJT3PegP8lUBnUsu47D7m7XVtd1LxXUmY5BGnXPiP9RitmPqbr52Wa3BfTd1LL2eZgxNfdaJefbUc/1i/0CnCAMAY1OPOvu2WkPOuJWX66h3RcOnHuSIxdFtjK70LihZ1QQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VDhUpA/lzIC42tK94W5IVr5b7fPs8P3l4LlOB5P7rsA=; b=Ii3DgqcKaTqrHNtnGQBTigaSvaXE2tF/0UJqkzlE01IeaAOsT3Fbag6purtph85CpLz9kfma19khf0P7MO72HJGgCCMFjsV20NqNUuGQhNv8FX1SqxBJ71R2gidTRPfX6fhbBzBUfXQB4LixThN1dn5OOk3rfd6Mbn1MMHnioxcSyE02b10pPlvqw0/v7JZqQBFC7qJQGG8ZLVA1Yhj96Twz0QOtTm6O3pbiG08KEJIabTU0+/8ZcBVrwtFHfTTtjBmTgsT1dE1GpvAVQZ0vNQLhsrM06hNqmcMsHIjFOH1fx4/bz3d2SSAtkTLZXL1j46QjExsTLyZjoukOy1DUlQ== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 09/18] vfio/common: Add device dirty page tracking start/stop Date: Thu, 26 Jan 2023 20:49:39 +0200 Message-ID: <20230126184948.10478-10-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT004:EE_|CO6PR12MB5428:EE_ X-MS-Office365-Filtering-Correlation-Id: da20e7b5-8dc5-45b1-12a7-08daffce3a58 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s3hjxv58ydtTkeRgFRt4GFcoPq60uX1bwkBfRIP2DyRgmv28mtObJgMsV8nXI1iZt0Os9OSE3sRU7k+lyI+mPKhzvVMr11ZbF99Yh+MakDHeP7EY8HOdBzQAXxYMwYBKjAyHwhM1wG6jzuvD5d7D40775XgoCYcZ2ev1y1KTba2tdnDrHwe1XSXb6MXK87gmlPawWXbez9sf/nMpOzaFe68q+tyG5fgUt0GSv3nzYdkaDXuvy4mA3B9n/qUdNoan/BUcIEFVBr+d1ygbcmCCsG5PtqBLIiZR1WrCU2nC6TfTI9CEE816RihHSaL++JPVWFlLgS26HOofxgICXDrZLSHv2d+qdADf2ZqJGiRlCRfY2RPFFJnw7+D2LtGxZA2o2sLurVL2IjDlivXMDfQWCDrP97cqN9G27p8+hpJ7jedaYG0TZJ9n3/uq1xDt84R2I9+v/E48ZGQoXFx2+2qgKDUKf11jiyz2CyZFLiBYyjpyHFzC6oTsm3En+eTNwTEdklUOP0LoH9Jvq4mmyKbGtRznE63VQErIz6XpumMeLc/QtI6UmKIwWKXzf+kBo/iZ93ELBvTJCIhCsgtpp+S22QzV8RT4Ybd0VVrVGK5BXZeE3jNhu1KHKSHiS8OdCkutL5LZ+wbowLtqvFhDALa+ZPPtwmEYdlgquNjqGOcTqr+uZ5WIzV2myJUEYAwv4oxXDIknqUadFslXiKRYKb/ixpWZ48+l/Tp465c2qbMS/s8= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(136003)(396003)(376002)(346002)(39860400002)(451199018)(46966006)(40470700004)(36840700001)(40460700003)(36756003)(7696005)(8676002)(54906003)(478600001)(316002)(1076003)(5660300002)(7416002)(2906002)(6916009)(4326008)(70586007)(41300700001)(70206006)(8936002)(82740400003)(7636003)(36860700001)(356005)(40480700001)(336012)(26005)(86362001)(2616005)(82310400005)(83380400001)(426003)(186003)(47076005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:43.1667 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: da20e7b5-8dc5-45b1-12a7-08daffce3a58 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT004.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR12MB5428 Received-SPF: softfail client-ip=40.107.102.47; envelope-from=avihaih@nvidia.com; helo=NAM04-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759317966100003 Content-Type: text/plain; charset="utf-8" From: Joao Martins Add device dirty page tracking start/stop functionality. This uses the device DMA logging uAPI to start and stop dirty page tracking by device. Device dirty page tracking is used only if all devices within a container support device dirty page tracking. Signed-off-by: Joao Martins Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 2 + hw/vfio/common.c | 211 +++++++++++++++++++++++++++++++++- 2 files changed, 211 insertions(+), 2 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index d54000d7ae..cde6ffb9d6 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -151,6 +151,8 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + bool dirty_pages_supported; + bool dirty_tracking; } VFIODevice; =20 struct VFIODeviceOps { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index fafc361cea..005c060c67 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -422,6 +422,22 @@ static bool vfio_devices_all_dirty_tracking(VFIOContai= ner *container) return true; } =20 +static bool vfio_devices_all_device_dirty_tracking(VFIOContainer *containe= r) +{ + VFIOGroup *group; + VFIODevice *vbasedev; + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (!vbasedev->dirty_pages_supported) { + return false; + } + } + } + + return true; +} + /* * Check if all VFIO devices are running and migration is active, which is * essentially equivalent to the migration being in pre-copy phase. @@ -1355,13 +1371,192 @@ static int vfio_set_dirty_page_tracking(VFIOContai= ner *container, bool start) return ret; } =20 +static int vfio_devices_dma_logging_set(VFIOContainer *container, + struct vfio_device_feature *featur= e) +{ + bool status =3D (feature->flags & VFIO_DEVICE_FEATURE_MASK) =3D=3D + VFIO_DEVICE_FEATURE_DMA_LOGGING_START; + VFIODevice *vbasedev; + VFIOGroup *group; + int ret =3D 0; + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + if (vbasedev->dirty_tracking =3D=3D status) { + continue; + } + + ret =3D ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature); + if (ret) { + ret =3D -errno; + error_report("%s: Failed to set DMA logging %s, err %d (%s= )", + vbasedev->name, status ? "start" : "stop", re= t, + strerror(errno)); + goto out; + } + vbasedev->dirty_tracking =3D status; + } + } + +out: + return ret; +} + +static int vfio_devices_dma_logging_stop(VFIOContainer *container) +{ + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature), + sizeof(uint64_t))] =3D {}; + struct vfio_device_feature *feature =3D (struct vfio_device_feature *)= buf; + + feature->argsz =3D sizeof(buf); + feature->flags =3D VFIO_DEVICE_FEATURE_SET; + feature->flags |=3D VFIO_DEVICE_FEATURE_DMA_LOGGING_STOP; + + return vfio_devices_dma_logging_set(container, feature); +} + +static gboolean vfio_device_dma_logging_range_add(DMAMap *map, gpointer da= ta) +{ + struct vfio_device_feature_dma_logging_range **out =3D data; + struct vfio_device_feature_dma_logging_range *range =3D *out; + + range->iova =3D map->iova; + /* IOVATree is inclusive, DMA logging uAPI isn't, so add 1 to length */ + range->length =3D map->size + 1; + + *out =3D ++range; + + return false; +} + +static gboolean vfio_iova_tree_get_first(DMAMap *map, gpointer data) +{ + DMAMap *first =3D data; + + first->iova =3D map->iova; + first->size =3D map->size; + + return true; +} + +static gboolean vfio_iova_tree_get_last(DMAMap *map, gpointer data) +{ + DMAMap *last =3D data; + + last->iova =3D map->iova; + last->size =3D map->size; + + return false; +} + +static struct vfio_device_feature * +vfio_device_feature_dma_logging_start_create(VFIOContainer *container) +{ + struct vfio_device_feature *feature; + size_t feature_size; + struct vfio_device_feature_dma_logging_control *control; + struct vfio_device_feature_dma_logging_range *ranges; + unsigned int max_ranges; + unsigned int cur_ranges; + + feature_size =3D sizeof(struct vfio_device_feature) + + sizeof(struct vfio_device_feature_dma_logging_control); + feature =3D g_malloc0(feature_size); + feature->argsz =3D feature_size; + feature->flags =3D VFIO_DEVICE_FEATURE_SET; + feature->flags |=3D VFIO_DEVICE_FEATURE_DMA_LOGGING_START; + + control =3D (struct vfio_device_feature_dma_logging_control *)feature-= >data; + control->page_size =3D qemu_real_host_page_size(); + + QEMU_LOCK_GUARD(&container->mappings_mutex); + + /* + * DMA logging uAPI guarantees to support at least num_ranges that fit= s into + * a single host kernel page. To be on the safe side, use this as a li= mit + * from which to merge to a single range. + */ + max_ranges =3D qemu_real_host_page_size() / sizeof(*ranges); + cur_ranges =3D iova_tree_nnodes(container->mappings); + control->num_ranges =3D (cur_ranges <=3D max_ranges) ? cur_ranges : 1; + ranges =3D g_try_new0(struct vfio_device_feature_dma_logging_range, + control->num_ranges); + if (!ranges) { + g_free(feature); + errno =3D ENOMEM; + + return NULL; + } + + control->ranges =3D (uint64_t)ranges; + if (cur_ranges <=3D max_ranges) { + iova_tree_foreach(container->mappings, + vfio_device_dma_logging_range_add, &ranges); + } else { + DMAMap first, last; + + iova_tree_foreach(container->mappings, vfio_iova_tree_get_first, + &first); + iova_tree_foreach(container->mappings, vfio_iova_tree_get_last, &l= ast); + ranges->iova =3D first.iova; + /* IOVATree is inclusive, DMA logging uAPI isn't, so add 1 to leng= th */ + ranges->length =3D (last.iova - first.iova) + last.size + 1; + } + + return feature; +} + +static void vfio_device_feature_dma_logging_start_destroy( + struct vfio_device_feature *feature) +{ + struct vfio_device_feature_dma_logging_control *control =3D + (struct vfio_device_feature_dma_logging_control *)feature->data; + struct vfio_device_feature_dma_logging_range *ranges =3D + (struct vfio_device_feature_dma_logging_range *)control->ranges; + + g_free(ranges); + g_free(feature); +} + +static int vfio_devices_dma_logging_start(VFIOContainer *container) +{ + struct vfio_device_feature *feature; + int ret; + + feature =3D vfio_device_feature_dma_logging_start_create(container); + if (!feature) { + return -errno; + } + + ret =3D vfio_devices_dma_logging_set(container, feature); + if (ret) { + vfio_devices_dma_logging_stop(container); + } + + vfio_device_feature_dma_logging_start_destroy(feature); + + return ret; +} + static void vfio_listener_log_global_start(MemoryListener *listener) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); int ret; =20 - ret =3D vfio_set_dirty_page_tracking(container, true); + if (vfio_devices_all_device_dirty_tracking(container)) { + if (vfio_have_giommu(container)) { + /* Device dirty page tracking currently doesn't support vIOMMU= */ + return; + } + + ret =3D vfio_devices_dma_logging_start(container); + } else { + ret =3D vfio_set_dirty_page_tracking(container, true); + } + if (ret) { + error_report("vfio: Could not start dirty page tracking, err: %d (= %s)", + ret, strerror(-ret)); vfio_set_migration_error(ret); } } @@ -1371,8 +1566,20 @@ static void vfio_listener_log_global_stop(MemoryList= ener *listener) VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); int ret; =20 - ret =3D vfio_set_dirty_page_tracking(container, false); + if (vfio_devices_all_device_dirty_tracking(container)) { + if (vfio_have_giommu(container)) { + /* Device dirty page tracking currently doesn't support vIOMMU= */ + return; + } + + ret =3D vfio_devices_dma_logging_stop(container); + } else { + ret =3D vfio_set_dirty_page_tracking(container, false); + } + if (ret) { + error_report("vfio: Could not stop dirty page tracking, err: %d (%= s)", + ret, strerror(-ret)); vfio_set_migration_error(ret); } } --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759209; cv=pass; d=zohomail.com; s=zohoarc; b=jalJ/qqHR6dGwCo96KcVNrtilXOCvrLhElb80QuwBXu6DP2vW2GJeVPjSP45UlLa9X9i5tlDNiyC4mHmhKm2M6cXzJ3BNNlEWuBLA1i4+Zpn/7f27BWqBG5avD5ZKBw+Vi4jrR+MCWT4DQidcSRKt93g7+UTWKct6XAdn/QDxbM= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759209; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=vQ+NskwLHTnpthg5KMKf7zCi+c2zgVf/AnBQH4DHTSA=; b=lkeVSkN8k1Pj+3yMC2RIZQvdNLIvpXOEpkO4JRNp0vgZ2Uip/UaRUYOBkHHtKnqrS3r5np9PlrTM7QQbZpaWFa7cGfOCKbZiFco+SESXEiIaumlxOF5++ot2qNOD/d2G1IhpZpxDMlasn9v39PccUDJpXKPzO0EoC0OymvxsUNY= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759209338935.6759077558868; Thu, 26 Jan 2023 10:53:29 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LU-00063X-Rh; Thu, 26 Jan 2023 13:51:37 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kf-0004fI-Kn for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:56 -0500 Received: from mail-dm6nam10on20616.outbound.protection.outlook.com ([2a01:111:f400:7e88::616] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Kd-0002lW-HQ for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:50:45 -0500 Received: from MW4PR04CA0297.namprd04.prod.outlook.com (2603:10b6:303:89::32) by DM6PR12MB4973.namprd12.prod.outlook.com (2603:10b6:5:1b7::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:50:40 +0000 Received: from CO1NAM11FT113.eop-nam11.prod.protection.outlook.com (2603:10b6:303:89:cafe::93) by MW4PR04CA0297.outlook.office365.com (2603:10b6:303:89::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33 via Frontend Transport; Thu, 26 Jan 2023 18:50:40 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT113.mail.protection.outlook.com (10.13.174.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:39 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:39 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:38 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:34 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RK0sQIk0dj4Tl9SLq9XMyG/hTEDQqxWUPJd3siCeugcme0ERGfqeqkO/OJiGdZDrJvIPTrBkW5JmJZoAlYIa1C8NPJo6nIV3MQgTVRQ48Dkavs5p1W9scRtA+bex5E/AewT3rRMlbukMlUrVVMrM/udDrMfTYd2CJ5wgi8DZvIYsYyX9Y01DkDdMOaEaoiyTzahEeZxEuMMePg0Ss2XNTaiyd9IYB9IVuqxhOuZrLW6ZjMPAiPStTfh3JRM5bLj3Wu5TsHideoHkHYBlo33m9jHeIqvajpadXFJjj7l78iwOkc8q/ZJQixBXvzE8/3kSlUFYtEFWC8jVKn8B9GVLww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vQ+NskwLHTnpthg5KMKf7zCi+c2zgVf/AnBQH4DHTSA=; b=Rhvg+nDGg+BfNX1cRw3G8q+B21ZCubM54ZS89mfYlQhuVvK2OTQboY+XfPY5gb2qk17tUe4Q42F12voItxWPxGBkiIiRXVoAUYafLGgByp3cWd2YHKW86bqpg43gcH6nDLtq1C5Ty6nbN+mt1kFiZ6zmGcEt6KyrVwZn+2QHonz/HbUe+/cU+UIGNM6+mLrFv731M3bUmqwsl2rAeMwBASfvFywDXsDSmCRwLhPofJS6ypzgdBh94ScqfW9eRMpWhW7XmuqHM6ayOBAScyf+V/bVRPGzs95O4irkp3JVl2cMVw4QlV4lCulWFCAnDuK/vIDV90ak++gXL+faFoDBxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vQ+NskwLHTnpthg5KMKf7zCi+c2zgVf/AnBQH4DHTSA=; b=QZUqkV+zyv1SDaOjvAhNKHtxKszj8PYufxKqrfZRz13AS6mGyUO3zit16W8Nc79zegn8PdLpo0OBB+IujU8inlU5QHDrQo7NAeFOHI01j+DghWiQQn4sBBYv6nJMEFwfUqy5010oxRxr5rAY2/CjhN2sr2/rwAhOZiPriJMznBZ1sg1sgDmVJppc/gqPr48qe8Y7JQPRBByfwBltan5+jyeqgea7zYmP6YICEUvV93NOKDVe2L9g6M7uSgXl7AWhgaYDWnc5gs4lv7zCXA63mWjm23hGdZ/bV6fuDucye66DPm3LgHsZu26mvSN8+qw4kxAkvtfX6W8G1bbfSzewmA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 10/18] vfio/common: Extract code from vfio_get_dirty_bitmap() to new function Date: Thu, 26 Jan 2023 20:49:40 +0200 Message-ID: <20230126184948.10478-11-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT113:EE_|DM6PR12MB4973:EE_ X-MS-Office365-Filtering-Correlation-Id: 43abef5e-7c8c-42dc-afb1-08daffce3839 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IR7B/r3D5VqR1Kfl+Njp6YyYZkBNCrI+PE2xfMIYdJGghZiRTcZ1KyO8bUALH5DJftWPHE2w8dPaoirN27gsaCSEo5ddPA7+Zrfviyl8+/SI52NnhOpBPRqLEoo4l78ejSQ/TYfqcz8BT2gRkD+ZtdXxWl5GFLfICJP1/aAKZrrsOCcgu7cZaqyq5PYjg0E/fUctW+vN+1spK3lSR9xKmHyoQ6LtPMojnl5tj1WCOJAITlt4QN/XoLE7GWoyFi689O9n+kr9aBXp+V/ed1yVZHvcz95ODdzh+TosMLSA/yZ/L4BZYTsue+iP8ouU6wOHXulhM61DwS3zD//NFxhNGYFiIq06q3Sz7Azv+SZJs5I32CBaOhk+hOq1TIbZDy/KXEjqt3EPVXCKV23i6hcsKBkJnKrGwSLApYh1FuJ1Bc/wE9zjG1/oLlA1SjdbYYww5zvufdzizpGkMLktfMD4Et/PEYlz6plYuAqktRRwssZcDY4DpmvMhl4rqJ2hK9/MbTKoadZ8yohAAf0OG5wp1fM84KxM29ms6UwfgQ69DLAzq9jbzCEeDWpHq3BbW6yKBfk8Y4k533/E5jVfpYDSv3ipVi80/24BYfSkpNLhRRrY8KGf/+G7yEZDegbKC8e/7UEjBaYNcgXCBlRnz4pxTu4GfqYa9WIBgv0hL96mUpgLLKpRY66GdxVG983TXrpgByHmJAihFkcTDHXecpoQUSSuh6GE+ft14nTWcHDuvz8= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(136003)(376002)(346002)(396003)(451199018)(40470700004)(46966006)(36840700001)(66899018)(47076005)(8676002)(40460700003)(36756003)(7696005)(356005)(86362001)(36860700001)(7636003)(2616005)(82310400005)(426003)(26005)(83380400001)(336012)(186003)(54906003)(40480700001)(478600001)(7416002)(5660300002)(6666004)(316002)(1076003)(6916009)(41300700001)(82740400003)(8936002)(70586007)(70206006)(4326008)(2906002)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:39.7059 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 43abef5e-7c8c-42dc-afb1-08daffce3839 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT113.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4973 Received-SPF: softfail client-ip=2a01:111:f400:7e88::616; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759210668100001 Content-Type: text/plain; charset="utf-8" Extract the VFIO_IOMMU_DIRTY_PAGES ioctl code in vfio_get_dirty_bitmap() to its own function. This will help the code to be more readable after next patch will add device dirty page bitmap sync functionality. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 53 ++++++++++++++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 005c060c67..3caa73d6f7 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1584,26 +1584,13 @@ static void vfio_listener_log_global_stop(MemoryLis= tener *listener) } } =20 -static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, - uint64_t size, ram_addr_t ram_addr) +static int vfio_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *v= bmap, + hwaddr iova, hwaddr size) { struct vfio_iommu_type1_dirty_bitmap *dbitmap; struct vfio_iommu_type1_dirty_bitmap_get *range; - VFIOBitmap *vbmap; int ret; =20 - if (!container->dirty_pages_supported) { - cpu_physical_memory_set_dirty_range(ram_addr, size, - tcg_enabled() ? DIRTY_CLIENTS_= ALL : - DIRTY_CLIENTS_NOCODE); - return 0; - } - - vbmap =3D vfio_bitmap_alloc(size); - if (!vbmap) { - return -errno; - } - dbitmap =3D g_malloc0(sizeof(*dbitmap) + sizeof(*range)); =20 dbitmap->argsz =3D sizeof(*dbitmap) + sizeof(*range); @@ -1627,16 +1614,42 @@ static int vfio_get_dirty_bitmap(VFIOContainer *con= tainer, uint64_t iova, error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, (uint64_t)range->size, errno); - goto err_out; + } + + g_free(dbitmap); + + return ret; +} + +static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr) +{ + VFIOBitmap *vbmap; + int ret; + + if (!container->dirty_pages_supported) { + cpu_physical_memory_set_dirty_range(ram_addr, size, + tcg_enabled() ? DIRTY_CLIENTS_= ALL : + DIRTY_CLIENTS_NOCODE); + return 0; + } + + vbmap =3D vfio_bitmap_alloc(size); + if (!vbmap) { + return -errno; + } + + ret =3D vfio_query_dirty_bitmap(container, vbmap, iova, size); + if (ret) { + goto out; } =20 cpu_physical_memory_set_dirty_lebitmap(vbmap->bitmap, ram_addr, vbmap->pages); =20 - trace_vfio_get_dirty_bitmap(container->fd, range->iova, range->size, - range->bitmap.size, ram_addr); -err_out: - g_free(dbitmap); + trace_vfio_get_dirty_bitmap(container->fd, iova, size, vbmap->size, + ram_addr); +out: vfio_bitmap_dealloc(vbmap); =20 return ret; --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759270; cv=pass; d=zohomail.com; s=zohoarc; b=IsbNi+dcFn4RkDkn9rts3S5qJVVLSWJgXL1W9wsTvGf3wA/OHAVbUJJpwKOfwND2jWWo4D/a7SXMHSCabvhfZMBewp9+lrCwVaFVJI2Hn6+w1ExQw1XU//QB2xo/RUyRF4sCZ0lisrJnEyPh/X0ExsmX0FNLd56pZLrYmlrCX50= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759270; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=IW2YpEbwYjtEzjPon1/scu8kulvXmZs+gcjsw+mhPxg=; b=LVXu86sqKT+UoPKxSr62VhtM7dyZq7LQiA25FvEXD0xTdcZNlh5L6dJeruyTFQSs7GxbPX9EgPzCLwLUnemTuoPDuG6CK29Lbzw7AiY2VhHxO6euTWffXKdGrkMoToO2NQr04hCCm34mT6WG3Lbysm9V6gKL5e2ApKfW+ROKvy4= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759270264587.7346056171809; Thu, 26 Jan 2023 10:54:30 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LW-0006Fa-5K; Thu, 26 Jan 2023 13:51:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7L0-0005F3-Tb for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:09 -0500 Received: from mail-mw2nam12on20630.outbound.protection.outlook.com ([2a01:111:f400:fe5a::630] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Ky-0002uK-EE for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:06 -0500 Received: from MW4PR03CA0201.namprd03.prod.outlook.com (2603:10b6:303:b8::26) by CH3PR12MB8546.namprd12.prod.outlook.com (2603:10b6:610:15f::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21; Thu, 26 Jan 2023 18:50:49 +0000 Received: from CO1NAM11FT057.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b8:cafe::a2) by MW4PR03CA0201.outlook.office365.com (2603:10b6:303:b8::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:50:49 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT057.mail.protection.outlook.com (10.13.174.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:50:48 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:43 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:43 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:39 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U+xzKnoMFqIou52X4NVp0n383oPFF7lQYwxQe5FhpoHMyy3FdLJEc7W5viXhF4jBpdr6ItaNS6hGMibDlHtzUq1Z5pSMacKvMFXBfVOmP76NxVWEIKJ6CRCRvUqTDBEESFfLnqgMv2BFKShYvj3bUX65i+tLz9LXyrX8QeOgG5+OjbpvjE/0dPAw0Na/fxiB3xNOLFPt39FIgyChqvhKYLsFx00Qui6ANKFJL0pb2FkX4q6DOo1n0+PIaeAKkoVxT4Xi/TyXHwJ6qZUZKc1FN1Bx8OH8GBJbsKiLgaQI8W2UOypPbxj7t9eE12xDYTaOU0Z0+2kY+1KZhYg+vk7+VQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IW2YpEbwYjtEzjPon1/scu8kulvXmZs+gcjsw+mhPxg=; b=AhP2Jig8udIRJ/eOjFuyyh/OrzCbLjY1h0D4P2ShhMaXExF8e0ghBLAl58JcYJ+ecrQFJlH9t95KlNu0lkkESsfnFf1b7QhWXHFgiALfLrtYojEtzgfDAk8NuXrXIJrin8M6XEMlhViFQd8Uh1KXn/JL6IzCCgO6tmQQFemORJkixJiBiT2ZZYI6dB2DGcjDCNqvqU9KUQtHNEPy6BOrssgpGTU6JzTPw+aZGaaYIjKC8FqwcV5jUZ1ccsVQYJ6nCVshFtkbN+wCmWiOmhzmOc+OZSgRQuUvKk896NlQszfJHm7ech5LMwKz/Nbs1ggIyAfJS9cYiP3qWT85oAwrOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IW2YpEbwYjtEzjPon1/scu8kulvXmZs+gcjsw+mhPxg=; b=LH4+vW3OeS+9u4VDJUdUer/AyC4cqE3huI6Qk5UP16rHDmNk/JcBjqYbxJXPpcZHzo7bQ3t9CIRQ96w9QUtClQasCE3d2s7/IOJe2I55s9Yx6ink9vwwndYw6qU6qE4fILhxtBQRO1q/a127XeKf2FXxrhdnNxJNF/NI/4x8ehq3Zm5MDhJSX2Y5GFlDnyNFd3hKMO8TrHjnPKvIV4v901qy1IVooNftTJdhxiBlXziWlfXxJ5eNyx2iXobs3TW782922LjIHWM5YwPLFqG0E+s4+sJMdy5ev0JR6r48EAcDpvCbUtJCj+ei1D60wkn7fzTbo64P33nEgBW7IL+t5g== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 11/18] vfio/common: Add device dirty page bitmap sync Date: Thu, 26 Jan 2023 20:49:41 +0200 Message-ID: <20230126184948.10478-12-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT057:EE_|CH3PR12MB8546:EE_ X-MS-Office365-Filtering-Correlation-Id: e679f3b1-0582-4da5-de9e-08daffce3dbb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: u+4DLvm/Cd3b2Q2wKlmgzSgv8q24BRmibZgwRgQCtiGP3KqjUNd17cf7jmWiMfMLHAAmmJ7ybMwQRiSh9LjTJnuCtQbc2yk7PUF2z+C+mWdJReb2/xWpeKo0TI6dS1ES37jQYk2X1bkqOckQOhAWTGMameDDCew6QMLxay9mLOue9jo/qKn4qCUS05ksZxPM12vHZ5acSVde48805WEO17O3PWwJwLvkDIi9g/MesQ1J7RDM6boccSRmXpJq80VteGM5VkuV0eAaW44QNylc4U8kLc6Bs//DYzVO9K0EGUJGArBttEDFxYxMDfALpbZ22e118lTEXHVQzLO4Llo4U3Vu/P+HTwNhVg7be2SMgYlrXAksc59SsHQJNO4H/v8LojaEP/7HiUtc6h239jPlqaDUlSq32PWRgU8Tyf0H8IfZgS4eSGmeMRQpNGae+jnYQf8ptrO9zU2Rna4YnUhuPFybLn74qtepmFejgezjH/77O3yCwvaqxbTN5ApdTVL7OjMGM37GHq5JAod5LhbA6kd6N4wOiqGGLU+iFnYG6BZcDgWMAeI8gXJdzRS6NyutQVzEgLleTsb352w9P8qpxtnuF5u7GFsKNWgLAIQi+0t7m3pAStc1nZjAoLfkB589KjBo2MNGKJk0vWVm8c+MzvDfv5Sa1MGXuU7d2vQ2mtEwFDzHCeu+S8lPdE6GazOAPCA/dh+jwfOl5ey/CmyFz9AxHYtmcQqaKGALK4HbfYE= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(396003)(136003)(346002)(39860400002)(451199018)(40470700004)(46966006)(36840700001)(316002)(47076005)(82310400005)(86362001)(40480700001)(36756003)(40460700003)(356005)(36860700001)(7636003)(54906003)(82740400003)(7416002)(2906002)(8676002)(66899018)(6916009)(41300700001)(4326008)(70206006)(70586007)(8936002)(5660300002)(186003)(26005)(1076003)(2616005)(7696005)(83380400001)(426003)(6666004)(478600001)(336012)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:48.9477 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e679f3b1-0582-4da5-de9e-08daffce3dbb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT057.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8546 Received-SPF: softfail client-ip=2a01:111:f400:fe5a::630; envelope-from=avihaih@nvidia.com; helo=NAM12-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759271376100001 Content-Type: text/plain; charset="utf-8" From: Joao Martins Add device dirty page bitmap sync functionality. This uses the device DMA logging uAPI to sync dirty page bitmap from the device. Device dirty page bitmap sync is used only if all devices within a container support device dirty page tracking. Signed-off-by: Joao Martins Signed-off-by: Avihai Horon --- hw/vfio/common.c | 93 ++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 82 insertions(+), 11 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 3caa73d6f7..0003f2421d 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -355,6 +355,9 @@ static void vfio_bitmap_dealloc(VFIOBitmap *vbmap) g_free(vbmap); } =20 +static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, + uint64_t size, ram_addr_t ram_addr); + bool vfio_mig_active(void) { VFIOGroup *group; @@ -582,10 +585,19 @@ static int vfio_dma_unmap(VFIOContainer *container, .iova =3D iova, .size =3D size, }; + int ret; =20 - if (iotlb && container->dirty_pages_supported && - vfio_devices_all_running_and_mig_active(container)) { - return vfio_dma_unmap_bitmap(container, iova, size, iotlb); + if (iotlb && vfio_devices_all_running_and_mig_active(container)) { + if (!vfio_devices_all_device_dirty_tracking(container) && + container->dirty_pages_supported) { + return vfio_dma_unmap_bitmap(container, iova, size, iotlb); + } + + ret =3D vfio_get_dirty_bitmap(container, iova, size, + iotlb->translated_addr); + if (ret) { + return ret; + } } =20 while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { @@ -611,12 +623,6 @@ static int vfio_dma_unmap(VFIOContainer *container, return -errno; } =20 - if (iotlb && vfio_devices_all_running_and_mig_active(container)) { - cpu_physical_memory_set_dirty_range(iotlb->translated_addr, size, - tcg_enabled() ? DIRTY_CLIENTS_= ALL : - DIRTY_CLIENTS_NOCODE); - } - vfio_erase_mapping(container, iova, size); =20 return 0; @@ -1584,6 +1590,65 @@ static void vfio_listener_log_global_stop(MemoryList= ener *listener) } } =20 +static int vfio_device_dma_logging_report(VFIODevice *vbasedev, hwaddr iov= a, + hwaddr size, void *bitmap) +{ + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature) + + sizeof(struct vfio_device_feature_dma_logging_repo= rt), + sizeof(uint64_t))] =3D {}; + struct vfio_device_feature *feature =3D (struct vfio_device_feature *)= buf; + struct vfio_device_feature_dma_logging_report *report =3D + (struct vfio_device_feature_dma_logging_report *)feature->data; + + report->iova =3D iova; + report->length =3D size; + report->page_size =3D qemu_real_host_page_size(); + report->bitmap =3D (uint64_t)bitmap; + + feature->argsz =3D sizeof(buf); + feature->flags =3D + VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT; + + if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { + return -errno; + } + + return 0; +} + +static int vfio_devices_query_dirty_bitmap(VFIOContainer *container, + VFIOBitmap *vbmap, hwaddr iova, + hwaddr size) +{ + VFIODevice *vbasedev; + VFIOGroup *group; + int ret; + + if (vfio_have_giommu(container)) { + /* Device dirty page tracking currently doesn't support vIOMMU */ + bitmap_set(vbmap->bitmap, 0, vbmap->pages); + + return 0; + } + + QLIST_FOREACH(group, &container->group_list, container_next) { + QLIST_FOREACH(vbasedev, &group->device_list, next) { + ret =3D vfio_device_dma_logging_report(vbasedev, iova, size, + vbmap->bitmap); + if (ret) { + error_report("%s: Failed to get DMA logging report, iova: " + "0x%" HWADDR_PRIx ", size: 0x%" HWADDR_PRIx + ", err: %d (%s)", + vbasedev->name, iova, size, ret, strerror(-re= t)); + + return ret; + } + } + } + + return 0; +} + static int vfio_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *v= bmap, hwaddr iova, hwaddr size) { @@ -1627,7 +1692,8 @@ static int vfio_get_dirty_bitmap(VFIOContainer *conta= iner, uint64_t iova, VFIOBitmap *vbmap; int ret; =20 - if (!container->dirty_pages_supported) { + if (!container->dirty_pages_supported && + !vfio_devices_all_device_dirty_tracking(container)) { cpu_physical_memory_set_dirty_range(ram_addr, size, tcg_enabled() ? DIRTY_CLIENTS_= ALL : DIRTY_CLIENTS_NOCODE); @@ -1639,7 +1705,12 @@ static int vfio_get_dirty_bitmap(VFIOContainer *cont= ainer, uint64_t iova, return -errno; } =20 - ret =3D vfio_query_dirty_bitmap(container, vbmap, iova, size); + if (vfio_devices_all_device_dirty_tracking(container)) { + ret =3D vfio_devices_query_dirty_bitmap(container, vbmap, iova, si= ze); + } else { + ret =3D vfio_query_dirty_bitmap(container, vbmap, iova, size); + } + if (ret) { goto out; } --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759317; cv=pass; d=zohomail.com; s=zohoarc; b=DpdFYwyNXMsRmULQ3JYu7VNhGywDDdSCN0XQWp9wt99IS4dRseSYncB2+B98FrKlML6ufwFdhtuDLRwA5FKkO4IR9RAQHPkkY4D8cK7je6FIsmMiiL/ywDSgvNz6xuGNqgc0bJtIEB9QWtN1doo/Gdelu8wNIc6Co1HVYTu33Sg= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759317; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=kSqK6swuTag6Nz82TO12D7shl3s6wvNTutEgUGVwe7o=; b=Hh6XACBQcfJfEunxt1bo5gDg8wM4Ecv8MWXZsnXplEPXSm0fOfhj8m5ITiUpMhEjg4gqocEWE3ewI3ikiIkStZGgLmc0po4xmjICLQVb4sa0SYG2jvOBYX8PVg9/FAXuYAPBM7Wa3yyvWsalJ13QsUy0ITslxs28fiGK6fs/gE4= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759317097214.50360890667253; Thu, 26 Jan 2023 10:55:17 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7MC-0006ex-Qp; Thu, 26 Jan 2023 13:52:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Ku-00052J-Mh for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:03 -0500 Received: from mail-co1nam11on20616.outbound.protection.outlook.com ([2a01:111:f400:7eab::616] helo=NAM11-CO1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7Ks-0002tK-Pi for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:00 -0500 Received: from MW4PR03CA0196.namprd03.prod.outlook.com (2603:10b6:303:b8::21) by LV2PR12MB5944.namprd12.prod.outlook.com (2603:10b6:408:14f::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Thu, 26 Jan 2023 18:50:55 +0000 Received: from CO1NAM11FT057.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b8:cafe::32) by MW4PR03CA0196.outlook.office365.com (2603:10b6:303:b8::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:50:55 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT057.mail.protection.outlook.com (10.13.174.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:50:54 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:48 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:47 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:43 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OMWP3dMpcv61eJpR+v2RpXWblhdvS+xgjCuXYH/w3Sdgzag5m2VU8iJMbpIRd8Zfd9QH8aFXzDHlb6OO+3SAxqLOnI6kf6deUmPn5XfhZMJA3aGhqO/NhpdHykkaYXwbETr0Wy8N6zPE87Fek/K44wVhBkxgRWjdzc070dgGQfoXHkcH4z5nz52xWlNsMKmtem/jfMl3u9x4ajfLBObSuFgDFuWBVWBzY/Ri503ll9bgt6IeSkYTUUcsd+nhGMCFVm3BRUG39tnb+UeLRrylp6+va2IVJ5iOR107P3e1JM3LcVZrpnUsanPLcLLJ6A5+JfJY1ZyBO/coFMkpm53wCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kSqK6swuTag6Nz82TO12D7shl3s6wvNTutEgUGVwe7o=; b=F2bSTRXa09rkuR2dVI0vg9F5y+vzMC3WNKmsBEkUD/5c50swh/MBUuGi85PIJji+5mF5RRd9d4bUr+Si71VbpJ6jy82t4iwo3+raO96jLV4mBGpTWUlikckB7dq/64fj2MzRnnSXRWgcJCB30lAn5FoGm7f1TAoJ0+InXbnPJvrpQpJcedfk8+YbmmkDSo2ILZaJK+s00eyX/kUZb0UXG0uqXXtQgn9Ze9b4bugJ7EA0IpJU2/ZtSMSSJFFti+fYh4RC/Ysz40lgHy2m8IT/ceGe3mofxHEuH8VVukhvrFDpRVgkbE2VwYIXNQhZFzzP1buTfp7tChlwS31/ezj+AQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kSqK6swuTag6Nz82TO12D7shl3s6wvNTutEgUGVwe7o=; b=izRUqn9v2uZOu5AhXPoV0gnCg9Wz28YDSSVX1s1Db93+mEtCRhz7jBvQfoHKpJpVDBXnWOIHFMDNeyRHiSs4/mkGWs2KMFciHUMnGDQvIOR3qonRCLw3Z6GTFT+Yh+YJBCiDVTc5BJ4bZfXYAxKiuYtyJP4/FwGnU/vvynvJHngQqT9k90Nd8ERGGxMRI6axbci6IvxZPQU/fhk0Eg9r+n1Ukts4Fxxgye9ccIFoZ4x8nX1E9LcZpWU2QAHCVSvcz+L9x+3HWxuYIeYmaZJiOI2AST1Z91L8GVcVxmhcNPmYb5zQhxM4edyXKREc2ON+IDNwsfb3iowtf8HzVLEOuw== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 12/18] vfio/common: Extract vIOMMU code from vfio_sync_dirty_bitmap() Date: Thu, 26 Jan 2023 20:49:42 +0200 Message-ID: <20230126184948.10478-13-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT057:EE_|LV2PR12MB5944:EE_ X-MS-Office365-Filtering-Correlation-Id: 9abe5fc3-467e-4134-8f32-08daffce4156 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0M1BruHPP1lnaIbPEdgnAaOFkweDM8ccTVzTPBaW1yK5Es2AKF/eZZdc4O8ElIVHFxuLRq4kYU/HDFsiCi2d+jJe9KnkfrtMhkW1/D+TQxUUEkLR+T1dylsag2HIA37JDXj2NTHLim3j8Ie+8eLQkwA7z4abq9ryHc1N9yRC5HWsV5ZxsIAhyqcZoqPEsHlNIgMxrCbVGfAfA50k0M3fp1+AFLA/SSJityXdzExwYBhopTQTjQKeKEHmHo/NjqcJlWc9CTU8lMOIC7PsOZZqLqLAK6LZ0ZyF+6A8x/Y2SU5PSlsQIjkgihDoVhV0/vzR1NIoDMmnQqwvaI5QA7T0DUOok7LaDFxl+vbHGu3UQElZVjbFol60TpUJsAq8vKsql9V/cKUzYczt6FJztgfdLpXBtIRee5Z57TI+m1SyNaZx0UiVA//C+WD7rxa47tIWC89QlzMg+L7hyOjupDEo0pLvx8gbjNqiF3yCNGFc4IWE49iDCDM9APiFHy4cKd+IoPlETeGmbFOUJT+Lyorpbf9mcbIvqJa3nJVi3wOZ/pl8hQ+JY70VAeFP6CxMJHO+8j5sni+YA2C80ABRSzqgIHjxgu3CKSrW/CsIkCPBZ8RDYyf0zMZN2UkRxmVaKMF1Vju+g/GGwjACWMlpdD/N/BbW1Upxf2pnR0MB8/mcIBHZn52QjiaodA9uIpMOvOO5sChqwEXYaObcLo0XRWYnZd0q7XDzjHzqIl5UdVGmlfA= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(396003)(136003)(346002)(376002)(451199018)(36840700001)(46966006)(40470700004)(36860700001)(26005)(356005)(83380400001)(7636003)(5660300002)(82740400003)(7416002)(7696005)(41300700001)(2906002)(86362001)(82310400005)(8936002)(4326008)(40480700001)(40460700003)(316002)(186003)(336012)(8676002)(47076005)(2616005)(478600001)(54906003)(6916009)(70586007)(36756003)(426003)(1076003)(70206006)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:50:54.9790 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9abe5fc3-467e-4134-8f32-08daffce4156 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT057.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5944 Received-SPF: softfail client-ip=2a01:111:f400:7eab::616; envelope-from=avihaih@nvidia.com; helo=NAM11-CO1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759317861100002 Content-Type: text/plain; charset="utf-8" Extract vIOMMU code from vfio_sync_dirty_bitmap() to a new function and restructure the code. This is done as preparation for the following patches which will add vIOMMU support to device dirty page tracking. No functional changes intended. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 63 +++++++++++++++++++++++++++++------------------- 1 file changed, 38 insertions(+), 25 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 0003f2421d..9792c2c935 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1811,37 +1811,50 @@ static int vfio_sync_ram_discard_listener_dirty_bit= map(VFIOContainer *container, &vrdl); } =20 +static int vfio_sync_iommu_dirty_bitmap(VFIOContainer *container, + MemoryRegionSection *section) +{ + VFIOGuestIOMMU *giommu; + bool found =3D false; + Int128 llend; + vfio_giommu_dirty_notifier gdn; + int idx; + + QLIST_FOREACH(giommu, &container->giommu_list, giommu_next) { + if (MEMORY_REGION(giommu->iommu_mr) =3D=3D section->mr && + giommu->n.start =3D=3D section->offset_within_region) { + found =3D true; + break; + } + } + + if (!found) { + return 0; + } + + gdn.giommu =3D giommu; + idx =3D memory_region_iommu_attrs_to_index(giommu->iommu_mr, + MEMTXATTRS_UNSPECIFIED); + + llend =3D int128_add(int128_make64(section->offset_within_region), + section->size); + llend =3D int128_sub(llend, int128_one()); + + iommu_notifier_init(&gdn.n, vfio_iommu_map_dirty_notify, IOMMU_NOTIFIE= R_MAP, + section->offset_within_region, int128_get64(llend), + idx); + memory_region_iommu_replay(giommu->iommu_mr, &gdn.n); + + return 0; +} + static int vfio_sync_dirty_bitmap(VFIOContainer *container, MemoryRegionSection *section) { ram_addr_t ram_addr; =20 if (memory_region_is_iommu(section->mr)) { - VFIOGuestIOMMU *giommu; - - QLIST_FOREACH(giommu, &container->giommu_list, giommu_next) { - if (MEMORY_REGION(giommu->iommu_mr) =3D=3D section->mr && - giommu->n.start =3D=3D section->offset_within_region) { - Int128 llend; - vfio_giommu_dirty_notifier gdn =3D { .giommu =3D giommu }; - int idx =3D memory_region_iommu_attrs_to_index(giommu->iom= mu_mr, - MEMTXATTRS_UNSPECIF= IED); - - llend =3D int128_add(int128_make64(section->offset_within_= region), - section->size); - llend =3D int128_sub(llend, int128_one()); - - iommu_notifier_init(&gdn.n, - vfio_iommu_map_dirty_notify, - IOMMU_NOTIFIER_MAP, - section->offset_within_region, - int128_get64(llend), - idx); - memory_region_iommu_replay(giommu->iommu_mr, &gdn.n); - break; - } - } - return 0; + return vfio_sync_iommu_dirty_bitmap(container, section); } else if (memory_region_has_ram_discard_manager(section->mr)) { return vfio_sync_ram_discard_listener_dirty_bitmap(container, sect= ion); } --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759254; cv=pass; d=zohomail.com; s=zohoarc; b=V/tLXaLxm6Yqd2TnAc75LO7UQpMgiCY1RAFWIx45sCJDeuXKd1g80nsYUjl1Tf8RVJiuUSDpjr6fGWxHApWCGtF+EtGU7kw/mqu9HGAaaGdpAbeuXQCMbp+lxXhzc3U5vfZIg0JOxFR7SElws6MDCYMm9rPLyYjUsn6PhzZn7Go= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759254; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=uuWcVYbadQ793r5UWyDS1lhsUw5avGS07TznOJIoXAA=; b=WLbsUBURRauIeVjy6VRpoeKuav7b4+diVkackXM8fBiLzjJa1u7YotLjOWjfKc8/BDmzKdARQzGZg6nuJMfF3+/ze5GOGbW0skuCSX1PUy6Uup0aSUhjAJKqlOMDyK1ybBSD/WX0Thmqj9IJmr6Hhj8h4juoU4rlOAaCr2yeFtQ= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759254556912.8473090605773; Thu, 26 Jan 2023 10:54:14 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LL-00061e-Jy; Thu, 26 Jan 2023 13:51:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7L3-0005Fq-OE for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:15 -0500 Received: from mail-bn8nam12on20613.outbound.protection.outlook.com ([2a01:111:f400:fe5b::613] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7L0-0002ut-Qa for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:09 -0500 Received: from MW3PR05CA0008.namprd05.prod.outlook.com (2603:10b6:303:2b::13) by CH2PR12MB4972.namprd12.prod.outlook.com (2603:10b6:610:69::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:51:03 +0000 Received: from CO1NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2b:cafe::1a) by MW3PR05CA0008.outlook.office365.com (2603:10b6:303:2b::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:03 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT014.mail.protection.outlook.com (10.13.175.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:51:03 +0000 Received: from drhqmail203.nvidia.com (10.126.190.182) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:52 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail203.nvidia.com (10.126.190.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:52 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:48 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JYXgNfK6NBo/R6/GLeysry6wbOQlC7ru8PTJD6YJ5utcYroXHe3NigZ4Kusqn8GxenwsvKlYIc7uVg4QQTBBfw+Yy3fpngpuDqrT4pqOE5huAGuHGNSzUSKRD1cytq8OxsIbq2HbDZaMUT2yx4MwPqUMAHr9MLCCyEEJWAIkdeG9W+VUYryIHa2QZxFBOrLK+ClrWFdMaiO8oAhP+08VciHsjERIQlUGTnFUu1kO+P2vxLXm0B0/lQwU7Um2Pkuhen0bgIEy9XHZRmyaLCZckFJBCNdqjBVqv10FDFpYxiiYQBpd1jvjF3xnElcyWNYIPmJePPfJUboRLx/7rbupWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=uuWcVYbadQ793r5UWyDS1lhsUw5avGS07TznOJIoXAA=; b=Jc3Uknm7Cq++DgAmaXceys5mQxbOz6+Mr2iMlB/RHQnTLUTe9J1DAVBDd4o2u56Kv7sq+qrVKfQQ02fdEgP2YPtAABeIJmz5QuBFKkGrQGEir9AiQmNndNMdqtraeUarny3vs66/VXwrpyM8iLG/0v50rOhXnvDVrs4LZcsSfmKdKoLDCuc+bcvDcGmrrQQFGedzUQz256oV00hF90R8c6ohHK6y+TMQz6Kwu9nir7EndXReC6wz9saCqfchhHFVndgfKrHXXhAY8C4mM1sOTwQ6gRXpWbM8aq9vFlIQ/Ml8q5zQOrxlhjSzZ9+KIlV07HinVUeyE5cvOF5pde5ywQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=uuWcVYbadQ793r5UWyDS1lhsUw5avGS07TznOJIoXAA=; b=AsVOE+IoaS9F+IIxqdlfBredMYfKMo9u20WzdfoWc/YrxhY887yX2PKzI5huGhT498wh21tObyYk0DQaKGyMuOBOPSdVPS/WSH2WcqcLA3T2ADm5vR7dFoUNa3kAIzbBLsD4aJQLbA/A6jG21nl8Ol5YVs7/JrxVQGiDusBNR6pb7jLvGFn/Rna2GUIuMPpric97q/uu8+WdI8fwoSqT13nbNjZlc9jwa4Qqk4rt2BgD/4k7b89/i7wGObKOVutu5oZ95srxFGyMRPhHteIsP6La4yGia4hg5N29Xfs7Zj1y/IhB2rXHD7mpq15y8pegSt4UT5Ns0MMXnen4QnqOjA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 13/18] memory/iommu: Add IOMMU_ATTR_MAX_IOVA attribute Date: Thu, 26 Jan 2023 20:49:43 +0200 Message-ID: <20230126184948.10478-14-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT014:EE_|CH2PR12MB4972:EE_ X-MS-Office365-Filtering-Correlation-Id: 27ba2aca-ce2b-4fb0-abc0-08daffce462f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tG6YwENbKJfBSK2gwTl0SsxKzWF2+M9TIcvnzJDz56DiTvGB4ERL+gh/szGkOlc7XXT48BX68C8b6+L4x9Sr1SCRGxOREne0DgdtcdjOLhU9K0Dsh9C4LRmOCDphqXFeEHlBhmBFHYwOjsbbdyJDiLnC/r93Dwd1uXjKOarg4YgRq4qHkNkPixdqTT0J9FVtJjVNOJJmuiRZn57mQb944/gcvtuU/nt9IwWPZc05cFWtifU5ktDqKPT/y5OUfNasEMXzEs7rql8gyrfMStWAEDE4X2i1UbxFmVIyjSfwyT8qErnvxZZtLj+z2FS7x2PlNOQqmitoHqVN1avjbaZDsNAgeaeb3btDAoE0fsR0/qDGwSVIDUOoV7o9I19+31VqDnFLv6YvqyfSOyNH47wwU86BMRwaEDmQHjkJca8zEycxS0m3X3W61D6x7OGT9UnIGAUZFaX0w5w18OVoKbxgQ+5EMrWW+k+SEOVkMrsqJFxvvScT/pHHpi12HwTwcHSB6hH6GHF7jsozgz5rkLvc/cOOfiZD+wox9YvlNZ+YYFOyDhDPgU3JUhxJBphx1VJDrfN7+pap8BRbTHUyWnafhR0oMxo1uUki/C0zGNItwJy3+vZOzuUqaMUa2HPmhHeo8qBiTgxoOUIIq4X7DT2ED+iwcd/nojT+xSH4Jd94oeIjc02sYMmCrhd/fq9VWMUpuWlbmZH2iSkCeEdvrb6GqA== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(346002)(39860400002)(396003)(136003)(376002)(451199018)(36840700001)(40470700004)(46966006)(86362001)(40460700003)(40480700001)(4744005)(5660300002)(356005)(7416002)(336012)(83380400001)(426003)(26005)(186003)(2616005)(47076005)(7636003)(36860700001)(8676002)(82740400003)(316002)(36756003)(54906003)(4326008)(8936002)(70586007)(70206006)(478600001)(6666004)(82310400005)(1076003)(6916009)(7696005)(41300700001)(2906002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:03.1284 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 27ba2aca-ce2b-4fb0-abc0-08daffce462f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4972 Received-SPF: softfail client-ip=2a01:111:f400:fe5b::613; envelope-from=avihaih@nvidia.com; helo=NAM12-BN8-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759255313100001 Content-Type: text/plain; charset="utf-8" Add a new IOMMU attribute IOMMU_ATTR_MAX_IOVA which indicates the maximal IOVA that an IOMMU can use. This attribute will be used by VFIO device dirty page tracking so it can track the entire IOVA space when needed (i.e. when vIOMMU is enabled). Signed-off-by: Avihai Horon Acked-by: Peter Xu --- include/exec/memory.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/include/exec/memory.h b/include/exec/memory.h index c37ffdbcd1..910067a3a5 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -290,7 +290,8 @@ typedef struct MemoryRegionClass { =20 =20 enum IOMMUMemoryRegionAttr { - IOMMU_ATTR_SPAPR_TCE_FD + IOMMU_ATTR_SPAPR_TCE_FD, + IOMMU_ATTR_MAX_IOVA, }; =20 /* --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759330; cv=pass; d=zohomail.com; s=zohoarc; b=SG4exc6U25ehyE3AY08a3hG86ZZNJXz2lsemq0JpmCfT54UhMUhwfSHkB0LwFP3yOMNDkF7Dt2UrcUZFql7DZ69Cb4U/KBn1LeceuxV0aqOlChIVfLhtutgZQywY78zcXSDpkTtHn6oPi09SFkJvs36qkbGwm2dKSeeDy9SwFvU= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759330; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=QertlL506fJI6k9SbZ+Jkp8StwMukfuvViPrVm8LCws=; b=h7SJaNUw111ykdWTJBk8bHQMyT0HGOKgRnfJkgmBYiULrefkrvtI2uxaTeaZirNBs8EaGxXzJqcFi//yqEJZSjrzVj2mwk7pPnAaif1j2iWd1YQwmkWyhIH2ydINl044FIMEIMfHYNMv3Lzv4MKL6aOglVxqgoYkeyxKUn8JQeU= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759330917931.309465298297; Thu, 26 Jan 2023 10:55:30 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LZ-0006RT-CX; Thu, 26 Jan 2023 13:51:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LA-0005Me-OI for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:21 -0500 Received: from mail-dm6nam11on20601.outbound.protection.outlook.com ([2a01:111:f400:7eaa::601] helo=NAM11-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7L7-0002vE-HY for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:15 -0500 Received: from MW3PR05CA0019.namprd05.prod.outlook.com (2603:10b6:303:2b::24) by CY5PR12MB6429.namprd12.prod.outlook.com (2603:10b6:930:3b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6002.33; Thu, 26 Jan 2023 18:51:09 +0000 Received: from CO1NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2b:cafe::55) by MW3PR05CA0019.outlook.office365.com (2603:10b6:303:2b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:51:09 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT014.mail.protection.outlook.com (10.13.175.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:51:09 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:57 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:50:56 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:52 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TqF+KL7fKv4oDeLtE+p9+SbbpDUBO7c2gTx17KCrGY2STkUJj5tAKxHbv+H8s5T+w+E5UyKirQ4dBQglZBWqG2aNa3ceFPOLpkLx0UPhpnVViOq9boxAy5rEugOGhnnSg6IgtHTqLOq9sRqNSUIaMcuvsPBz1SCMFUWAUgax7NmmwG1Z/KbLDL2ukwzuoehaDvdntIVy+1hTjsRpHrRzKEpCOzIoxpYB0ECHNM62pZZbnFFmsUcGusTej2mVv+HgEgzK1/Use3gWYLYU7XMr0ll/d9COMHGfclDqb1dnzSDpa8aFlt1Uz8HTe4oMhYVhX1L2aXLyQhT6UQNl3Exjmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QertlL506fJI6k9SbZ+Jkp8StwMukfuvViPrVm8LCws=; b=AaBRdrq3cml+zJrle4KcJ+KpZTWO2kI92FtLJBid8wFlC/mKO1lBP9PkRnuBwedLOAeYCFTlEMnhx2zyLpmmOKrkcS0Br579bkwXycw77N2l0x4JD1Laja28YCfAfTDm2ytgCMavib5zd2kunYvujlymVdwR7sD845Uof879nQKDyavXwdvGfKB7nV/QzgKVtUiLi/rts/2Su2pTn5np8L84hMnSnE4dGwmvZ9Sy+SVlw7ntS1idXDuf3xKykzP9HcQAgAvz5WHvMFtbHm6cNQANbRa/LMzqJ4ad/xKFF+7GI4MGpXLiep76uc9ueSzdrm9cQbjsMEmZsrIugHUljw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QertlL506fJI6k9SbZ+Jkp8StwMukfuvViPrVm8LCws=; b=WBRdgC1mTjYOEYTHqj+yYMESyME2JeoBm/falUkk2bXVCXyF/QvHqD0XEa5rRHuWnu2W/2dXuoYKOr8SAW7KsEKqs/PwLiIO9FSYSG5iX+aj2Te6kimWiSt7Ths76F29HUqjN1rwo4tuNKlg29007XRwAsC9tioiS2x7/XbGSIZ5/NoUGql/txtt9DgGL3SSzh+Isx5/a0YhKW/CGh6P5YInOSvT7+QBwMIxFCLTImPB2w/PXWtn2fCPzznGP8S1qea5Rw+AeCPIKT65QzYhd2Exmwlls135+smJjWaizdVCV546v+cT5JTTziQJohehzf1fYhvMXrDtnr3i7xQctA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 14/18] intel-iommu: Implement get_attr() method Date: Thu, 26 Jan 2023 20:49:44 +0200 Message-ID: <20230126184948.10478-15-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT014:EE_|CY5PR12MB6429:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d541ad8-6ea7-4957-c678-08daffce49c0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Y4H6jhYRVLHJtUG7E/asE23o0z3VdSKduwrcvMtwZxhWHG2CtcwtKW97RiKvnbM6waB+amXIv/pxoZrB0G3ljAV+ahm9IQ3I6uqZG8omDs23oiq34DK9WHFTWV6o73lYyc7b11tvZWxElR6xWeQ/2vZ+A0A+b2EmOSzGH/+dUV6NL/al/wzH5rIUdp3+5mjukpEuUzt+iPDotrsPWJp6atIZOVsKU6g4USeOyO0M7DERRmi/ZHmtcHdn6RE00anU/jueOplN81lOkSm1w687CCMbnUsKxDPisbUh4BZTVQtxxFRCUv4QiFQtKnfRauG0QzutdIURDGYtDeYx8xdpyb5vvdLFaQL4C40LROhEKXlM8nKHopbjdaEfyZJbiehb8vDOH+IiYZkAf+Pqr+fCLj36VEZ/3ew+cRjWu8DjuV2pcVF1ibK1UCj6WDRUT9dvJ93ffC9+Gc+SfLQLUVz+22qUoXvQZQwOdQy1GJ8Qdn89xLFzuyE8bvdd++fjRK9TL1JK1ddx2sSkRm2hDbkKDLrnNLk/C5otSDukVE8/py3IuCsiDB4WQzCjV0mWYw01MyCayaCwF1ZH3jGQsPty/IwrIzD3OGg8UC9a8bYYgKNkR+IgwMpZQ32EehlhcaZ0B915KKzjusRjNoBcM3ng34cz9D2Oe4qBDtqR3SHUKOZn6mo0rgODA3qBDt303zQyF6843Y4lvCLPTCsY7TSkdA== X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(136003)(376002)(346002)(39860400002)(451199018)(46966006)(40470700004)(36840700001)(5660300002)(36860700001)(86362001)(186003)(2616005)(2906002)(82740400003)(4326008)(7636003)(82310400005)(7416002)(41300700001)(356005)(8936002)(54906003)(40460700003)(40480700001)(426003)(70206006)(8676002)(6666004)(6916009)(316002)(336012)(478600001)(7696005)(47076005)(26005)(70586007)(36756003)(1076003); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:09.1125 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9d541ad8-6ea7-4957-c678-08daffce49c0 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6429 Received-SPF: softfail client-ip=2a01:111:f400:7eaa::601; envelope-from=avihaih@nvidia.com; helo=NAM11-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759331976100001 Content-Type: text/plain; charset="utf-8" Implement get_attr() method and use the address width property to report the IOMMU_ATTR_MAX_IOVA attribute. Signed-off-by: Avihai Horon Acked-by: Peter Xu --- hw/i386/intel_iommu.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/hw/i386/intel_iommu.c b/hw/i386/intel_iommu.c index 98a5c304a7..b0068b0df4 100644 --- a/hw/i386/intel_iommu.c +++ b/hw/i386/intel_iommu.c @@ -3841,6 +3841,23 @@ static void vtd_iommu_replay(IOMMUMemoryRegion *iomm= u_mr, IOMMUNotifier *n) return; } =20 +static int vtd_iommu_get_attr(IOMMUMemoryRegion *iommu_mr, + enum IOMMUMemoryRegionAttr attr, void *data) +{ + VTDAddressSpace *vtd_as =3D container_of(iommu_mr, VTDAddressSpace, io= mmu); + IntelIOMMUState *s =3D vtd_as->iommu_state; + + if (attr =3D=3D IOMMU_ATTR_MAX_IOVA) { + hwaddr *max_iova =3D data; + + *max_iova =3D (1ULL << s->aw_bits) - 1; + + return 0; + } + + return -EINVAL; +} + /* Do the initialization. It will also be called when reset, so pay * attention when adding new initialization stuff. */ @@ -4173,6 +4190,7 @@ static void vtd_iommu_memory_region_class_init(Object= Class *klass, imrc->translate =3D vtd_iommu_translate; imrc->notify_flag_changed =3D vtd_iommu_notify_flag_changed; imrc->replay =3D vtd_iommu_replay; + imrc->get_attr =3D vtd_iommu_get_attr; } =20 static const TypeInfo vtd_iommu_memory_region_info =3D { --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759286; cv=pass; d=zohomail.com; s=zohoarc; b=dW5F69aINrVPe/zHpvtE8SoylvMnL/Ksy/TZlUmNJttRpvcxwxPVyBw74DTXr5aM9na3GDSZ1OVH2E1MbTZ0+qnhraglrgXBxI18plQii5dzsGEpxrezQJNsGCBxoVUbxov//76e1LW5GtGghnfv3vhPkR0cZJz79WZc0J4Rrb4= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759286; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=j5yF22RW/SrZOIWrGxgv6eqD3vJgAmcuK9WMhCuu+6Q=; b=DPoLiYSU6EZda8nbW2/BM7JYDHgUz3kW9akwMGerJ5GOKqkwuipf+qLRmzBSZTb9La5XjENKfeAm7fs9I5xfejpWdgSxalMMkJEVC0VQkSW3Zi4KgW6GVu8Tttl6ajCWxi/9krEG3MLxkcJqnXalLQ6TVjG1L0R0XMLVPIbYhtI= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759286225116.73835812561765; Thu, 26 Jan 2023 10:54:46 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7MC-0006ec-Pa; Thu, 26 Jan 2023 13:52:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LD-0005Q3-8Z for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:21 -0500 Received: from mail-dm6nam10on2058.outbound.protection.outlook.com ([40.107.93.58] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7L9-0002vX-Bu for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:17 -0500 Received: from MW3PR05CA0008.namprd05.prod.outlook.com (2603:10b6:303:2b::13) by PH8PR12MB7303.namprd12.prod.outlook.com (2603:10b6:510:220::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:51:12 +0000 Received: from CO1NAM11FT014.eop-nam11.prod.protection.outlook.com (2603:10b6:303:2b:cafe::1a) by MW3PR05CA0008.outlook.office365.com (2603:10b6:303:2b::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:12 +0000 Received: from mail.nvidia.com (216.228.118.232) by CO1NAM11FT014.mail.protection.outlook.com (10.13.175.99) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.21 via Frontend Transport; Thu, 26 Jan 2023 18:51:12 +0000 Received: from drhqmail201.nvidia.com (10.126.190.180) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:01 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail201.nvidia.com (10.126.190.180) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:01 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:50:57 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mdb1IgGlKBA3jNWxym3UX+TEVOP1ahYI6RtF1uHH5xUNcA5zkXm2zJ0s/sJDsG8d0ypinqbwe2oq6LztnkCBWnB3lkBqzqHMVYfxQbwif5zuDl/vdhRMY1v+VM4GxDOIQqpzd5TGMbV6x+T2cIjnX9p5VLyWh/ins55Cugiu/PBbNJ5YJy29dnPcrQBeFad3F8K92zvJCqfNdYHnwajORkcoiJc9nsDiRjuDYYCXtPxqdc/570tbCKdhQwF7awHJ73kARn/7ZkIHRw96LA1uX76VQ1SNWnNF3NbbFCPF73QXdDKvObiKpD6a+eTfOHGG0OPvuIKNUnOCpoR0Zqo1LA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j5yF22RW/SrZOIWrGxgv6eqD3vJgAmcuK9WMhCuu+6Q=; b=PU0WYT7S1tcN8g4t4rjYkoCvltvAKaMOdqKIoeAojNIIjFpajaGwBrHIjOtefsrFpTWsgGBY3S73Y1YMj15ufl7aQrZ2JwpMiT3caaoimccvbhs55C7X9ejVyokDpwegt12CQVwgUeq4hg9YA05vFDJG+YqXmtSHJSfwH7O1jdM8XpQQ/z5MBIp/1JPp6dbOZ5edr5pg+BIR+wJOGwNq/G9ccOTObtrwlNa8b5bl5wyNfAmmK5hdI1kiesN5giYl+o79afTbPBNjJTIJf27XK2mPh9jWMmY3M1uDW4M2bqQ4SFyhQeH1mmTIY6L18cj5RB5VrEnl38Ri+c5mzi9ASw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j5yF22RW/SrZOIWrGxgv6eqD3vJgAmcuK9WMhCuu+6Q=; b=lyGyYz8QnEEvL+WcrBJdhWtKY1GqGGJf7fqsl9zHgZ03N/ipFha0C0ojEihgnTwoxOT0AwEU7yaQywLuOG4TVEZWCLaOrvUmqDZxG3vEso/shzyFSTx6+zuEOCMDgjRcNFtFNCKiusDnPWTKefNBV96NZ526M+h2DcpTaDOhqHD1D7dIVvlPxF4jG9pUf7+2PCXThK92lvxeGiPUaJSSYojm/lTrA42Uw4YYQpKxJaiKrF6+2MUibZ5qqd5/s8w0aL3OoGaWcFF9R5DePXDDOUIuIMO6gqiTX8xLG7CCqJvQ2fgQkPdHoLLz0wh0pjRZesUooe3l1WqlEmXF/ym/Qg== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 15/18] vfio/common: Support device dirty page tracking with vIOMMU Date: Thu, 26 Jan 2023 20:49:45 +0200 Message-ID: <20230126184948.10478-16-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO1NAM11FT014:EE_|PH8PR12MB7303:EE_ X-MS-Office365-Filtering-Correlation-Id: 641d6370-0ea1-4936-d4be-08daffce4b79 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N0BVoAVUQ0xMowC6I9zUg7SIPanvjQtzdXOnGFBy/+GAAKP/Nc0WATpy9N8sKK1vZsD0nuQsy+BcIU2eVlWZqKC5yP6vP9XHGg6AMKCmUEdsO89VIlu1L24jNQdDmN/bNW5+u3A3EPolBLJOo4kd36PdGjxxv6mENKKzdc2YWKVkA0enzOxChQkYY2bjBVMIQYhIeCfCNCidPiQTTME9ePyR5Mq4VaXteiDrY3wxZhRFrPpvtD7iC8iW+O4bmkyGRBanshkW2wzxIaN8gj9jPsjhX2JHzTBufZdcDjTC61Ys9XhQ/Qi+3IrhEgBsyXZX/o9ox7/mc2ysr+R9HwqrnoCnYeTF54xGv8eembN0y3Ro6IbfWHNsceJ04g2U9/PlY/pqAXseJVOz6N/YHj7zfWmBIQsx7HCAhQoeloqEwo54TE2w6qsNSZj9v2MjwxYSwprMC90/nuBM1LJA8636axy73Bw9eHlcvWUkJ3EYYpyGvVbSfwLMQK3glOrjZ6eJI/VKsLyscIoHs2QYvKf8kziBe5tFN3R0BlPz8AtWtA2IfhGSsWpWoJg0znHx4pGQF305oZEki/ZUivjL+6wcw4ocCc/8BtOq7f7mc2YXbDOF6sg53JcXqe3dLwZ4TUi2pkc+df5tVmV1lkqrg+Lsp7KSILHVI964LZuwfpomaKUyRS4T16UARVyvUMTDewaOFIGMIjXSldYUhZbaOd42e5GeI1VOIda6/AHTTRfsVso= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(376002)(39860400002)(136003)(346002)(451199018)(46966006)(36840700001)(40470700004)(2906002)(426003)(36860700001)(336012)(83380400001)(5660300002)(41300700001)(8936002)(7416002)(8676002)(478600001)(2616005)(1076003)(4326008)(26005)(186003)(6666004)(86362001)(40460700003)(356005)(82740400003)(47076005)(40480700001)(7636003)(70586007)(6916009)(70206006)(36756003)(54906003)(7696005)(82310400005)(316002)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:12.0030 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 641d6370-0ea1-4936-d4be-08daffce4b79 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT014.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB7303 Received-SPF: softfail client-ip=40.107.93.58; envelope-from=avihaih@nvidia.com; helo=NAM10-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759287701100001 Content-Type: text/plain; charset="utf-8" Currently, device dirty page tracking with vIOMMU is not supported - RAM pages are perpetually marked dirty in this case. When vIOMMU is used, IOVA ranges are DMA mapped/unmapped on the fly as the vIOMMU maps/unmaps them. These IOVA ranges can potentially be mapped anywhere in the vIOMMU IOVA space. Due to this dynamic nature of vIOMMU mapping/unmapping, tracking only the currently mapped IOVA ranges, as done in the non-vIOMMU case, doesn't work very well. Instead, to support device dirty tracking when vIOMMU is enabled, track the entire vIOMMU IOVA space. If that fails (IOVA space can be rather big and we might hit HW limitation), try tracking smaller range while marking untracked ranges dirty. Signed-off-by: Avihai Horon --- include/hw/vfio/vfio-common.h | 2 + hw/vfio/common.c | 153 ++++++++++++++++++++++++++++++---- 2 files changed, 138 insertions(+), 17 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index cde6ffb9d6..15109c311d 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -97,6 +97,8 @@ typedef struct VFIOContainer { unsigned int dma_max_mappings; IOVATree *mappings; QemuMutex mappings_mutex; + /* Represents the range [0, giommu_tracked_range) not inclusive */ + hwaddr giommu_tracked_range; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 9792c2c935..c3a27cbbd5 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -44,6 +44,8 @@ #include "migration/qemu-file.h" #include "sysemu/tpm.h" #include "qemu/iova-tree.h" +#include "hw/boards.h" +#include "hw/mem/memory-device.h" =20 VFIOGroupList vfio_group_list =3D QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -377,6 +379,38 @@ bool vfio_mig_active(void) return true; } =20 +static uint64_t vfio_get_ram_size(void) +{ + MachineState *ms =3D MACHINE(qdev_get_machine()); + uint64_t plugged_size; + + plugged_size =3D get_plugged_memory_size(); + if (plugged_size =3D=3D (uint64_t)-1) { + plugged_size =3D 0; + } + + return ms->ram_size + plugged_size; +} + +static int vfio_giommu_get_max_iova(VFIOContainer *container, hwaddr *max_= iova) +{ + VFIOGuestIOMMU *giommu; + int ret; + + giommu =3D QLIST_FIRST(&container->giommu_list); + if (!giommu) { + return -ENOENT; + } + + ret =3D memory_region_iommu_get_attr(giommu->iommu_mr, IOMMU_ATTR_MAX_= IOVA, + max_iova); + if (ret) { + return ret; + } + + return 0; +} + static bool vfio_have_giommu(VFIOContainer *container) { return !QLIST_EMPTY(&container->giommu_list); @@ -1456,7 +1490,8 @@ static gboolean vfio_iova_tree_get_last(DMAMap *map, = gpointer data) } =20 static struct vfio_device_feature * -vfio_device_feature_dma_logging_start_create(VFIOContainer *container) +vfio_device_feature_dma_logging_start_create(VFIOContainer *container, + bool giommu) { struct vfio_device_feature *feature; size_t feature_size; @@ -1475,6 +1510,16 @@ vfio_device_feature_dma_logging_start_create(VFIOCon= tainer *container) control =3D (struct vfio_device_feature_dma_logging_control *)feature-= >data; control->page_size =3D qemu_real_host_page_size(); =20 + if (giommu) { + ranges =3D g_malloc0(sizeof(*ranges)); + ranges->iova =3D 0; + ranges->length =3D container->giommu_tracked_range; + control->num_ranges =3D 1; + control->ranges =3D (uint64_t)ranges; + + return feature; + } + QEMU_LOCK_GUARD(&container->mappings_mutex); =20 /* @@ -1524,12 +1569,12 @@ static void vfio_device_feature_dma_logging_start_d= estroy( g_free(feature); } =20 -static int vfio_devices_dma_logging_start(VFIOContainer *container) +static int vfio_devices_dma_logging_start(VFIOContainer *container, bool g= iommu) { struct vfio_device_feature *feature; int ret; =20 - feature =3D vfio_device_feature_dma_logging_start_create(container); + feature =3D vfio_device_feature_dma_logging_start_create(container, gi= ommu); if (!feature) { return -errno; } @@ -1544,18 +1589,85 @@ static int vfio_devices_dma_logging_start(VFIOConta= iner *container) return ret; } =20 +/* + * This value is used in the second attempt to start device dirty tracking= with + * vIOMMU, if the first attempt fails. It should be in the middle, not too= big + * and not too small, allowing devices with HW limitations to do device di= rty + * tracking while covering a fair amount of the IOVA space. + * + * This arbitrary value was chosen becasue it is the minimum value of Intel + * IOMMU max IOVA and mlx5 device supports tracking a range of this size. + */ +#define VFIO_GIOMMU_RETRY_IOVA ((1ULL << 39) - 1) + +#define VFIO_GIOMMU_RETRY_COUNT 3 +static int vfio_devices_start_dirty_page_tracking(VFIOContainer *container) +{ + hwaddr giommu_max_iova, iova_size, iova_retry_size, ram_size; + hwaddr iova_to_track[VFIO_GIOMMU_RETRY_COUNT] =3D {}; + int ret; + int i; + + if (!vfio_have_giommu(container)) { + return vfio_devices_dma_logging_start(container, false); + } + + /* + * With vIOMMU we try to track the entire IOVA space. As the IOVA spac= e can + * be rather big, devices might not be able to track it due to HW + * limitations. Therefore, retry tracking smaller ranges as follows: + * (1) Retry tracking a smaller part of the IOVA space. + * (2) Retry tracking a range in the size of the physical memory. + * (3) If all fail, give up. + */ + ret =3D vfio_giommu_get_max_iova(container, &giommu_max_iova); + if (!ret && !REAL_HOST_PAGE_ALIGN(giommu_max_iova)) { + giommu_max_iova -=3D qemu_real_host_page_size(); + } + + iova_size =3D ret ? 0 : giommu_max_iova; + iova_retry_size =3D iova_size ? MIN(VFIO_GIOMMU_RETRY_IOVA, iova_size = / 2) : + VFIO_GIOMMU_RETRY_IOVA; + ram_size =3D vfio_get_ram_size(); + + iova_to_track[0] =3D REAL_HOST_PAGE_ALIGN(iova_size); + iova_to_track[1] =3D REAL_HOST_PAGE_ALIGN(iova_retry_size); + iova_to_track[2] =3D REAL_HOST_PAGE_ALIGN(MIN(ram_size, iova_retry_siz= e / 2)); + + for (i =3D 0; i < VFIO_GIOMMU_RETRY_COUNT; i++) { + if (!iova_to_track[i]) { + continue; + } + + container->giommu_tracked_range =3D iova_to_track[i]; + ret =3D vfio_devices_dma_logging_start(container, true); + if (!ret) { + break; + } + + if (i < VFIO_GIOMMU_RETRY_COUNT - 1) { + warn_report("Failed to start device dirty tracking with vIOMMU= " + "with range of size 0x%" HWADDR_PRIx + ", err: %d. Retrying with range " + "of size 0x%" HWADDR_PRIx, + iova_to_track[i], ret, iova_to_track[i + 1]); + } else { + error_report("Failed to start device dirty tracking with vIOMM= U " + "with range of size 0x%" HWADDR_PRIx ", err: %d", + iova_to_track[i], ret); + } + } + + return ret; +} + static void vfio_listener_log_global_start(MemoryListener *listener) { VFIOContainer *container =3D container_of(listener, VFIOContainer, lis= tener); int ret; =20 if (vfio_devices_all_device_dirty_tracking(container)) { - if (vfio_have_giommu(container)) { - /* Device dirty page tracking currently doesn't support vIOMMU= */ - return; - } - - ret =3D vfio_devices_dma_logging_start(container); + ret =3D vfio_devices_start_dirty_page_tracking(container); } else { ret =3D vfio_set_dirty_page_tracking(container, true); } @@ -1573,11 +1685,6 @@ static void vfio_listener_log_global_stop(MemoryList= ener *listener) int ret; =20 if (vfio_devices_all_device_dirty_tracking(container)) { - if (vfio_have_giommu(container)) { - /* Device dirty page tracking currently doesn't support vIOMMU= */ - return; - } - ret =3D vfio_devices_dma_logging_stop(container); } else { ret =3D vfio_set_dirty_page_tracking(container, false); @@ -1616,6 +1723,17 @@ static int vfio_device_dma_logging_report(VFIODevice= *vbasedev, hwaddr iova, return 0; } =20 +static bool vfio_iommu_range_is_device_tracked(VFIOContainer *container, + hwaddr iova, hwaddr size) +{ + /* Check overflow */ + if (iova + size < iova) { + return false; + } + + return iova + size <=3D container->giommu_tracked_range; +} + static int vfio_devices_query_dirty_bitmap(VFIOContainer *container, VFIOBitmap *vbmap, hwaddr iova, hwaddr size) @@ -1625,10 +1743,11 @@ static int vfio_devices_query_dirty_bitmap(VFIOCont= ainer *container, int ret; =20 if (vfio_have_giommu(container)) { - /* Device dirty page tracking currently doesn't support vIOMMU */ - bitmap_set(vbmap->bitmap, 0, vbmap->pages); + if (!vfio_iommu_range_is_device_tracked(container, iova, size)) { + bitmap_set(vbmap->bitmap, 0, vbmap->pages); =20 - return 0; + return 0; + } } =20 QLIST_FOREACH(group, &container->group_list, container_next) { --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759102; cv=pass; d=zohomail.com; s=zohoarc; b=EueZmY4cFrzOl0cQWNClb1Vr9woaSAdETz6jE+qZq16qBvDX0tfVrfj77691eAosCoGH/CQ+dnZMGsY+qmnyP6Vq9vm90GEApiHYK6DPBAGAespMEGOTxFsUruKCPuR+MKjNUz74U8g86Xd1tvZxK2l++hL95AGjYkloGrjKWQE= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759102; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=X/A8ipv6Lf7VuO3oB5cyO3T7DidUjD5L14Pjy0wSW5M=; b=nj5euKPNrUQ42FW9wkcXuBX+UcLIFGLUX8qltyaAy1Y1bbpmR+3hlry5XIxdBK4IvofGT6gckRpvAqxlGgYI7Y0kwwmUekz9M2p/ETdFwSH6SVScQMB7LEFJdXWFgQxBmzpX0T/i5/ogKlVO2lFOaIXUHR2n7cFoJEy1joMGRxE= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759102157279.4341434188441; Thu, 26 Jan 2023 10:51:42 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LK-0005uv-5p; Thu, 26 Jan 2023 13:51:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LG-0005Zm-Vq for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:23 -0500 Received: from mail-sn1nam02on2060a.outbound.protection.outlook.com ([2a01:111:f400:7ea9::60a] helo=NAM02-SN1-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LE-0002wP-VC for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:22 -0500 Received: from DM6PR11CA0044.namprd11.prod.outlook.com (2603:10b6:5:14c::21) by DM6PR12MB4057.namprd12.prod.outlook.com (2603:10b6:5:213::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:51:16 +0000 Received: from DM6NAM11FT073.eop-nam11.prod.protection.outlook.com (2603:10b6:5:14c:cafe::86) by DM6PR11CA0044.outlook.office365.com (2603:10b6:5:14c::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.17 via Frontend Transport; Thu, 26 Jan 2023 18:51:16 +0000 Received: from mail.nvidia.com (216.228.118.232) by DM6NAM11FT073.mail.protection.outlook.com (10.13.173.152) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.17 via Frontend Transport; Thu, 26 Jan 2023 18:51:16 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:06 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:05 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:51:01 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Hr1wTGQOH+yye5pxMKiJ59z5y3LyT8KsoYM6I67VNVJJ4rqs86ZmJaaKFHwT8LsIbIwnD7rWBeGDagqUA9AbPgQqB5AgG1x66F3+bsc6V8jMaUnjEbJBQqaPSo0LlbMd6YgAmvH9VKaSoSXx0AsiOET9huu3IZu7Dbgai7c/ZQ1McC83aA9aCc9sP2KtWgYr2AX4TEnadB86wHGCUR8HRSSbYQc/BaJjO3fBMaqSzaaL9/n1j75XV6SYArY02QdaN9lOhK/QxFfPs3InWPrifsvfJMtNnObpOhp9cr8TLcDb9YrPN5ux7Zn5tFHbLHvs6qIKRZKR/ojHJi33kqTWsg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X/A8ipv6Lf7VuO3oB5cyO3T7DidUjD5L14Pjy0wSW5M=; b=mfccVmbIQTIwc9ngRihgJMfj+xWzvmDbnnukv5uJAgFtwZyj6GNAuM2tLH8zRybPgf8WJYcPNvJqCNHHT+e17fg06zSYY2h4fwI1K+evltOsEw89MtB1Ekwf3oeatwUmiG5NdvB+6Af9c95hvHdZqOAiHaLvF3n01hMUyzCyc3xXmvvPktar7Y8X58TySThMIf81Lwvpa+FRW4/IXv4sX7IAN0a8btxNU/BoMdsP1lXXHoZSF9Q88UA8ZiNxG1VEA3uBYuPaNFZ2k8bFTF/5mRgVThulyplHgeclyIPl/E/WadVckAmXF18Yfu53NAneWTCvDGnekRJ2xL/LffjTdA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.232) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X/A8ipv6Lf7VuO3oB5cyO3T7DidUjD5L14Pjy0wSW5M=; b=Nn91ga9NSplZUxRgxv+GiNIJhBjA/RrvDqTaaMesdDk6IsMDlHiLxKmkg4giSm0LCsmye5mrQE4ptL/yNCk21RpeMUUeRsr4Md+ICjWiopRjD/v/lkMKqKdbsd229HnT/HobKB9ya/ZfSqRdSsy4iXtfYbN0fzxrhXrCXgZ1pfAf6CXaUjSipcmoUYUS02I8xOkbWXUKBlRaIZPeIwAvwHedtP6m9DX+XIWtaf4aAYuNB970ArLODjdeQu3UObaRPRr35oi4whMKyFK1oyaswXPoXrBjiufvluuEHM41J1xP9OmW8kyXtWs9awQ0e/1Jh+vXLF6tu27iVcoR72v7wQ== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.232) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.232 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 16/18] vfio/common: Optimize device dirty page tracking with vIOMMU Date: Thu, 26 Jan 2023 20:49:46 +0200 Message-ID: <20230126184948.10478-17-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT073:EE_|DM6PR12MB4057:EE_ X-MS-Office365-Filtering-Correlation-Id: 02a72282-4d94-4b9d-398c-08daffce4e3f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ov/HlA/Ia8JrdtuFCiCC2F1TQYMBmz8KzWlbKLoq2Q7H9dtqrfTC9heeWmSYurstHn2fX7EnPPcCY+o0U83vETA8lH9zaBSUI9X6lrBDu4qMPQHDambzKGsFZRq88n+gC3Lac+6fvbqCSaVOk2ZMQ2qDyKbMhltiKC8hB2nUsljzlxL4coL2PkTc2mLyyaIa6mFN20HhhMjPvpuwqb+UUYmqTKOJa9KEf858PvzbzCCihol3DJR87vwbo1bcu7nMjFS6VKk8pYx9/yJI9XWXRXzlAbsvZ3BoC0O7EsCTvXZnQMMki0JJHxEWnMEAsNMov1vb9hthc/EzrhRiyJrAIpvuUHvplwslIX9iWKzjbD7Kab9rDnGuJYuH2M/vaWmELgX+t3oNXTRVMDfOV7vUaCMPxjWsdIkgyKGAWwFOgi2UnG9um1wDxT4RXftWv3rHbyfQAPlLcoZQOfo8lyZm80VNjUQ73nDSGGxeccy32TDPjjdRL6JSDiYKKGNO70jl3HgSw2cTvlJe/lGAzxUehGKwMu1ytvQJ8u7xE4jyihreCmeheM8StaiyzEPbAc5AGdaEVLf+BmJwYJFCkJhlG3aG50QGIGtYywPAK8f+9kocgZh2xIac3fnd9ocefx0BDx0G2n9SamtvL4ukwjpXqkywBBhZr5+mDa0L5gIxeqK3Z/BzjMXDw8bCVHz6oPWWD5sZJjGn6GnxR7XkocvySzIkaV99YsfbRUE0MruXckc= X-Forefront-Antispam-Report: CIP:216.228.118.232; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge1.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(346002)(39860400002)(136003)(376002)(451199018)(36840700001)(40470700004)(46966006)(66899018)(47076005)(8676002)(86362001)(36756003)(7696005)(40460700003)(40480700001)(356005)(7636003)(36860700001)(426003)(2616005)(26005)(336012)(83380400001)(82310400005)(186003)(5660300002)(478600001)(7416002)(6666004)(316002)(54906003)(1076003)(82740400003)(70206006)(70586007)(41300700001)(8936002)(6916009)(2906002)(4326008)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:16.6208 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 02a72282-4d94-4b9d-398c-08daffce4e3f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.232]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT073.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4057 Received-SPF: softfail client-ip=2a01:111:f400:7ea9::60a; envelope-from=avihaih@nvidia.com; helo=NAM02-SN1-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759104234100002 Content-Type: text/plain; charset="utf-8" When vIOMMU is enabled, syncing dirty page bitmaps is done by replaying the vIOMMU mappings and querying the dirty bitmap for each mapping. With device dirty tracking this causes a lot of overhead, since the HW is queried many times (even with small idle guest this can end up with thousands of calls to HW). Optimize this by de-coupling dirty bitmap query from vIOMMU replay. Now a single dirty bitmap is queried per vIOMMU MR section, which is then used for all corresponding vIOMMU mappings within that MR section. Signed-off-by: Avihai Horon --- hw/vfio/common.c | 85 ++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 2 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c3a27cbbd5..4f27cd669f 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1848,8 +1848,42 @@ out: typedef struct { IOMMUNotifier n; VFIOGuestIOMMU *giommu; + VFIOBitmap *vbmap; } vfio_giommu_dirty_notifier; =20 +static int vfio_iommu_set_dirty_bitmap(VFIOContainer *container, + vfio_giommu_dirty_notifier *gdn, + hwaddr iova, hwaddr size, + ram_addr_t ram_addr) +{ + VFIOBitmap *vbmap =3D gdn->vbmap; + VFIOBitmap *dst_vbmap; + hwaddr start_iova =3D REAL_HOST_PAGE_ALIGN(gdn->n.start); + hwaddr copy_offset; + + dst_vbmap =3D vfio_bitmap_alloc(size); + if (!dst_vbmap) { + return -errno; + } + + if (!vfio_iommu_range_is_device_tracked(container, iova, size)) { + bitmap_set(dst_vbmap->bitmap, 0, dst_vbmap->pages); + + goto out; + } + + copy_offset =3D (iova - start_iova) / qemu_real_host_page_size(); + bitmap_copy_with_src_offset(dst_vbmap->bitmap, vbmap->bitmap, copy_off= set, + dst_vbmap->pages); + +out: + cpu_physical_memory_set_dirty_lebitmap(dst_vbmap->bitmap, ram_addr, + dst_vbmap->pages); + vfio_bitmap_dealloc(dst_vbmap); + + return 0; +} + static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *i= otlb) { vfio_giommu_dirty_notifier *gdn =3D container_of(n, @@ -1870,8 +1904,15 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifie= r *n, IOMMUTLBEntry *iotlb) =20 rcu_read_lock(); if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL)) { - ret =3D vfio_get_dirty_bitmap(container, iova, iotlb->addr_mask + = 1, - translated_addr); + if (gdn->vbmap) { + ret =3D vfio_iommu_set_dirty_bitmap(container, gdn, iova, + iotlb->addr_mask + 1, + translated_addr); + } else { + ret =3D vfio_get_dirty_bitmap(container, iova, iotlb->addr_mas= k + 1, + translated_addr); + } + if (ret) { error_report("vfio_iommu_map_dirty_notify(%p, 0x%"HWADDR_PRIx"= , " "0x%"HWADDR_PRIx") =3D %d (%s)", @@ -1935,6 +1976,7 @@ static int vfio_sync_iommu_dirty_bitmap(VFIOContainer= *container, { VFIOGuestIOMMU *giommu; bool found =3D false; + VFIOBitmap *vbmap =3D NULL; Int128 llend; vfio_giommu_dirty_notifier gdn; int idx; @@ -1952,6 +1994,7 @@ static int vfio_sync_iommu_dirty_bitmap(VFIOContainer= *container, } =20 gdn.giommu =3D giommu; + gdn.vbmap =3D NULL; idx =3D memory_region_iommu_attrs_to_index(giommu->iommu_mr, MEMTXATTRS_UNSPECIFIED); =20 @@ -1959,11 +2002,49 @@ static int vfio_sync_iommu_dirty_bitmap(VFIOContain= er *container, section->size); llend =3D int128_sub(llend, int128_one()); =20 + /* + * Optimize device dirty tracking if the MR section is at least partia= lly + * tracked. Optimization is done by querying a single dirty bitmap for= the + * entire range instead of querying dirty bitmap for each vIOMMU mappi= ng. + */ + if (vfio_devices_all_device_dirty_tracking(container)) { + hwaddr start =3D REAL_HOST_PAGE_ALIGN(section->offset_within_regio= n); + hwaddr end =3D int128_get64(llend); + hwaddr size; + int ret; + + if (start >=3D container->giommu_tracked_range) { + goto notifier_init; + } + + size =3D REAL_HOST_PAGE_ALIGN( + MIN(container->giommu_tracked_range - 1, end) - start); + + vbmap =3D vfio_bitmap_alloc(size); + if (!vbmap) { + return -errno; + } + + ret =3D vfio_devices_query_dirty_bitmap(container, vbmap, start, s= ize); + if (ret) { + vfio_bitmap_dealloc(vbmap); + + return ret; + } + + gdn.vbmap =3D vbmap; + } + +notifier_init: iommu_notifier_init(&gdn.n, vfio_iommu_map_dirty_notify, IOMMU_NOTIFIE= R_MAP, section->offset_within_region, int128_get64(llend), idx); memory_region_iommu_replay(giommu->iommu_mr, &gdn.n); =20 + if (vbmap) { + vfio_bitmap_dealloc(vbmap); + } + return 0; } =20 --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759204; cv=pass; d=zohomail.com; s=zohoarc; b=InivGPSqc9yamaPM20swPl+sdRqSo4Kw7U039YuJJ4TptUMrP/s8mWrKnVf5PG6kiOo63IrmvlFiEp5mPGIU+1Wks0Uqy7azahK9bz2xWixc1+k1xUzMEXEBrf8qGedXUasWCIoB5RqR5CQpIor4pujsAEbwKmyNh9CGs5/cLHs= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759204; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hPPDKm8V3Lo9Aeu+bYelnDegV5GuooD1YAYk2pR/vow=; b=PHuV0TQcUFDksQyWudlAgxAS7Lt1SLPjQBZmP63SKhqZByT1ivQTl8yyrTgNVrq3NRFimI6FjcRFgUs3kCak58/j73kKD4FL6NotYXUVNjM46RmTq7Rk4ZeNnE5i3bP5SkPt0UMp/0UXvkUfJzeLe2lhgkb3XefYDVnrMiwH3+c= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759204666521.537323891976; Thu, 26 Jan 2023 10:53:24 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7LK-0005vZ-9p; Thu, 26 Jan 2023 13:51:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LI-0005nP-LF for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:24 -0500 Received: from mail-mw2nam10on2062c.outbound.protection.outlook.com ([2a01:111:f400:7e89::62c] helo=NAM10-MW2-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LG-0002wy-WF for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:24 -0500 Received: from BN9PR03CA0598.namprd03.prod.outlook.com (2603:10b6:408:10d::33) by CY5PR12MB6406.namprd12.prod.outlook.com (2603:10b6:930:3d::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:51:19 +0000 Received: from BN8NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10d:cafe::28) by BN9PR03CA0598.outlook.office365.com (2603:10b6:408:10d::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:19 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT022.mail.protection.outlook.com (10.13.176.112) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:19 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:10 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:10 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:51:06 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kIK31kDQG3XC0+bqJsut7IcijgjVjoIRazCPfuSw410H7GrBqngn58kG63+wAcbVTmCV7Jp8F6hgyzBKd9H6Wxf3/Z60iIck1CIHHJAuExhOK5cgEn1g5GHI5mHBSJEo5tyxemldCinD/+fbBH4t0QCD9byL7dDbqJJ48OApVQbfUSJvEOfyPOC3nBvYbvxcPyaDCKYLSLTw3bfK8cjj3CK6yntbSXP4Qq0ovbFbWJc3UEZrsAXKKrOczYHL1YpItsXydJl5Ueu7MZ5BEs45N0co6Utg6xqBb05YSkwVVQjuBRIKiQ6iNYsmFwz8gkkENqlhF52JnjWZf/zx3CFG2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=hPPDKm8V3Lo9Aeu+bYelnDegV5GuooD1YAYk2pR/vow=; b=FC99RKJIeyPdL6kNg66r01kUtK37xU842+36oKluFUf/8+LGekZ+wx8AIh6XJphtaJhRbRknjI7fCFg/IwLqNoeolYAS1WSW3aYJcPjGWF5lGnDQBRsQNxQ8b6e2Rx2sbnkUrVJ4F/HW21J+Z13s++5MmUCtC3xOi4xGiKgF+ZbwiVVLk44gd9bjo5ZAHGypJZ/7Jvl0wkZUJwXSBoBlpuD2BSG2as6shtujD+MoqIL0SGNid/tu7xdPuJQrHc2jjH2vluR5+I+IeqoYwLzzEF7emzlboFynfxNEW8osD3S2wZlRqdAJfMxYoDuVyE5Zcf+b36BOpLCxzPtXfU7/MA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hPPDKm8V3Lo9Aeu+bYelnDegV5GuooD1YAYk2pR/vow=; b=uttY4B1igTZLsOdKh/R7KdwR+uWa+jknRrnotz5aQD66TKvDpX7p8A6Qh+XFRkvuE2aQK16Sihany/54AgF1uWs10vI0l1fbogPG/zWPJ4h3bxl1S8PNROa3/uAf9TXCPMooywTsGBpPveVv2BcojRTxM1fTvsrqhjGym9gA4iNYGn1MYYen/TGCBJcYxNxyW42LtMx5hAbB7TaBV3pqvh5wYWt3D03z7Lg13awqHuLmf1fXiEkeycWR1PYwYTneaoaq2SH2+Fy4nkC+L9e8yjU/oXTD+tgPWUTuQEPeP+nLlp+DudCPKwBzYbtDK6ORf28EAYfn9Lpheo4vVMZhog== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 17/18] vfio/migration: Query device dirty page tracking support Date: Thu, 26 Jan 2023 20:49:47 +0200 Message-ID: <20230126184948.10478-18-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT022:EE_|CY5PR12MB6406:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ba4003f-40dc-45b8-b406-08daffce5011 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: s6g2VE4qOslm737xGqAVcwumuPYbZQDZT/ynyx5qCZtmfHmNgbjCov2w30o8tNSwDFd6ixIO2siIZkxmSY33Vi1ZGJA7nUDsW7nGtj3M806qVioKCb5IqFqRP6IPNpkSdjlAFYVRmoFDDRxVGLEGA3qN39TSXbxCrhTezuyIMlFGM1SN25RBWgGEpi7Bjr91kKkJJUp3Ewd0jlX58HZU+1St/4YpjqjRiL8or4n1YB9g3/2L61xnhRQ99H35S/4oI2ZuewuSeaWb6UiM90OsaiGuItuXYCg8esLq8/Raaj3V5su/wcDkNNil9C+ZOpreTVBFkw+Eo5kjDFx6aJ9Rhl6QpP50pwAZ7hYr9/7WRMa6DlqylcqfFSpLb4s15W+s5ON0CNCURPbXspXhhAF/NB6uI+osRjn38n2kEpIYjQP5u5auq8hKDYQ7V7WEGZ58ZJKlwTeZtaJS1IcIZ2RNLEbBbaY7cwikVTJELZmISMyUxT1UgH676mdJqueWIU3Ao4nS0AgFvl5kA51gUcRhUCcUXw/C3R2q6yPJ55xmCjRaflqu+2/g4NwbXO4bER1BNLTRL/1YOztWrnzs5Uer1U3rgeVdC+PJFc1eyEM+MkP8lm0scwNgmTzRNmoWXI/VgKm3wSpDichKxLgLD/w7kXr7C5MGJ50DsUfqm30pZRTkiUAztu5REkyiGLggk57mtEEnr7y5N0vxJcHrSdTQ0sL5Qx+vTbTaoMk64wMJYb8= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(376002)(136003)(39860400002)(346002)(396003)(451199018)(40470700004)(36840700001)(46966006)(36756003)(40460700003)(7696005)(356005)(54906003)(8676002)(316002)(1076003)(478600001)(6666004)(5660300002)(7416002)(2906002)(4326008)(70206006)(70586007)(41300700001)(82740400003)(6916009)(8936002)(7636003)(36860700001)(86362001)(40480700001)(82310400005)(336012)(26005)(186003)(426003)(2616005)(47076005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:19.6154 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ba4003f-40dc-45b8-b406-08daffce5011 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6406 Received-SPF: softfail client-ip=2a01:111:f400:7e89::62c; envelope-from=avihaih@nvidia.com; helo=NAM10-MW2-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759206637100003 Content-Type: text/plain; charset="utf-8" From: Joao Martins Now that everything has been set up for device dirty page tracking, query the device for device dirty page tracking support. Signed-off-by: Joao Martins Signed-off-by: Avihai Horon --- hw/vfio/migration.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 2a0a663023..5aeda47345 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -658,6 +658,19 @@ static int vfio_migration_query_flags(VFIODevice *vbas= edev, uint64_t *mig_flags) return 0; } =20 +static bool vfio_dma_logging_supported(VFIODevice *vbasedev) +{ + uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature), + sizeof(uint64_t))] =3D {}; + struct vfio_device_feature *feature =3D (struct vfio_device_feature *)= buf; + + feature->argsz =3D sizeof(buf); + feature->flags =3D + VFIO_DEVICE_FEATURE_PROBE | VFIO_DEVICE_FEATURE_DMA_LOGGING_START; + + return !ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature); +} + static int vfio_migration_init(VFIODevice *vbasedev) { int ret; @@ -693,6 +706,8 @@ static int vfio_migration_init(VFIODevice *vbasedev) migration->data_fd =3D -1; migration->mig_flags =3D mig_flags; =20 + vbasedev->dirty_pages_supported =3D vfio_dma_logging_supported(vbasede= v); + oid =3D vmstate_if_get_id(VMSTATE_IF(DEVICE(obj))); if (oid) { path =3D g_strdup_printf("%s/vfio", oid); --=20 2.26.3 From nobody Thu Oct 31 23:55:26 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass(p=reject dis=none) header.from=nvidia.com ARC-Seal: i=2; a=rsa-sha256; t=1674759254; cv=pass; d=zohomail.com; s=zohoarc; b=nNvEkLO++FG7q88uPRoLpEVhaIS+rCxt/eNt2WHDoHU3cVmWou8CJ8HmGH5wsxjsogfOfEz70MLCtoKG/L1cYGmq1HLxMLjNuN+KVuXWY2ciRQU4xadm7odIhnvhWey7fIr4faBVlEvWprVdjNGD+zju3LXnWSGECDRoP7kYp6E= ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674759254; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=wbDwoweDd3MH+MNMpLa0Y6rjEtRiBgczu1QLVX78AFg=; b=PvaRK0G8LYQFrFQJh5+tsdBlXMJ9xhdPsIF49Fiyu3ROjOoH/uKjsn+WrleS6LH1MclOlJTlhHlW3LzxaByLnBBPHepshctEMOq5yKmskhY/vYOa1fhOXy4wAGWQP/LA93BOQVt8w/eD7WNN8PUfHzwHIx1dLQQ2r7r8wJF7ErI= ARC-Authentication-Results: i=2; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; arc=pass (i=1 dmarc=pass fromdomain=nvidia.com); dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1674759254619752.5780230700037; Thu, 26 Jan 2023 10:54:14 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pL7MG-0006rB-8B; Thu, 26 Jan 2023 13:52:24 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LP-00066h-29 for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:35 -0500 Received: from mail-dm6nam12on2040.outbound.protection.outlook.com ([40.107.243.40] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pL7LL-0002xc-Cu for qemu-devel@nongnu.org; Thu, 26 Jan 2023 13:51:30 -0500 Received: from BN9PR03CA0576.namprd03.prod.outlook.com (2603:10b6:408:10d::11) by IA1PR12MB6161.namprd12.prod.outlook.com (2603:10b6:208:3eb::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22; Thu, 26 Jan 2023 18:51:24 +0000 Received: from BN8NAM11FT022.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10d:cafe::55) by BN9PR03CA0576.outlook.office365.com (2603:10b6:408:10d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:24 +0000 Received: from mail.nvidia.com (216.228.118.233) by BN8NAM11FT022.mail.protection.outlook.com (10.13.176.112) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.22 via Frontend Transport; Thu, 26 Jan 2023 18:51:24 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:15 -0800 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.36; Thu, 26 Jan 2023 10:51:14 -0800 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.986.36 via Frontend Transport; Thu, 26 Jan 2023 10:51:10 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nF2S0VlwHK1CHMHLWoIrcJ4crDHyrSe99c2iYQHFiCfgoZOl920mZjjCmMa51hjldEgqIVkNbO1jJu+BHOd0aQJsU8sJqp57FyQjrwIVsueWmKJe+H9kxiGu8TI4Bd2cfH3+fgplXEAwLKW98OPfq7aeY4YGhQnOvRTj+MubKD5636ZJKopkK2yLXVAO1t8IWAe6GO6tybw6WYz9kYb/heR/HUJx+bddZZ1YjgD1aO7nrymbzoyyyMS5tpWcppcgfguBzAGFwQAcQlr0Fz51xjkyTzF9vHZmKB0x1Wphq1JH6Pxox9V1Dv6lDmMqY+pi53Uz9cLefsuZMo7VqgYtWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wbDwoweDd3MH+MNMpLa0Y6rjEtRiBgczu1QLVX78AFg=; b=HBNK+CSAe5iY1H/nbI4L+UXw1Bqu1NhqQqVq7ENFNYwVi59wCASQc9THIHi6utQTLS62MgmwqCrdAKeV5znFNd0IdCR09Tb5OoykaGNMnY7kEy9mJ8zyenm8QbiBWl99BwUwJ9qOnpYz/3N0UtAjuZSp5PrvROjdm25v3QVgmO45yqdJ9VV74GSXu0A5pUbe+LgKBoV7OFrVpja2nXwSRgjjIr6HmN5/dp0Md0LFhxzJFdklShh9LtIgBqcnSJVmRZHfJStQ4ijPBqG5FnJk+rA/RGmj2gqvqQIbM0ADYaB6egTDAv6JhyP8zA4NAwQdbiLeBP03JK+r+yAElT9HTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=nongnu.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wbDwoweDd3MH+MNMpLa0Y6rjEtRiBgczu1QLVX78AFg=; b=XRHkFG5gQmIEuWvlSGQWErgK4yLPO2YRaf8L2hGNrO9hDDwnqMIjK8CFuLnbjAVEhP4kyhDXE2Q+cX+J4GsL91pAHpcK3fq/qF7MOblZIY4S9ZIz/3BbqNLa4w9iw7GeBVdP1sCk0E+JvkGmuUemw1SCn3r2mCv5cHluhy/mFieP/yyXgO2SCoVaSsHY7f5PIMK99ROe0qrVlS1S9BGs6TYC82uMZZqoEp9Rsz3JOSpCVAbr7Wxjz98C8ejzQtz86JNP50gfvBinIRMFnLIxGfCDhM7l6Nr81kfh2aUKWbEhAuDEqWhfUphuh1vv2Ge9bH7tzDbJOiw4gXxdkGDZHA== X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C From: Avihai Horon To: CC: Alex Williamson , "Michael S. Tsirkin" , Peter Xu , Jason Wang , Marcel Apfelbaum , "Paolo Bonzini" , Richard Henderson , Eduardo Habkost , "David Hildenbrand" , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Yishai Hadas , Jason Gunthorpe , Maor Gottlieb , Avihai Horon , Kirti Wankhede , Tarun Gupta , Joao Martins Subject: [PATCH 18/18] docs/devel: Document VFIO device dirty page tracking Date: Thu, 26 Jan 2023 20:49:48 +0200 Message-ID: <20230126184948.10478-19-avihaih@nvidia.com> X-Mailer: git-send-email 2.21.3 In-Reply-To: <20230126184948.10478-1-avihaih@nvidia.com> References: <20230126184948.10478-1-avihaih@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT022:EE_|IA1PR12MB6161:EE_ X-MS-Office365-Filtering-Correlation-Id: bdab153a-24f1-495d-3743-08daffce52eb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1MdNuVbcFhiEDOjmAhMC+Mn6GbxmwzZgeahVg7U2qQRcHQz0iADtsJ1mT/b58m6Fbq8k6O+hruaFVAzu4dUAwXbkQUR0x1aEtY6pz3TasQPDqkVYhP3rnKL5ogXNZAnepXlgQ8FBBmeJHYx9QYDxmdN1sDdId0S5hvZPERmbWzlTlXwG/ICaeJxGMu7Y7EEhB6+y5lasaQoA51lu702sVC/yUJgTxOrxi0VAo2qWfbUW4hr/vLAFZp7wDmh89wzNnnxnmurjqCDvbK5nEGZ7isFUTlhH7QEjNMkWiz6JE2rHzp1/6eEqwjV9UrThOfmhCxn+6nvNuAFhZV56qNB54MCyivAPZoOE8nF7Mbg6NT8d5m16E6gPaNdCaQRASEe4F7p1EGgFOxTVZcaczqsDZg7c5/ooNaPuYG5i0gvnmfZ/uYLWkCKPaacCjNZLD8rAaCDxOayw9JwFLYIiCflFep8crlbTX0L+SnX1aDSAvRpRWgqi79bpQvJpJQ956pHL0gvb5sAImblDRFCbLuAzwKaIRuAEGCVB+KI4jPr2YPDF97dMeiGtD0IoHnn1BDxn8FBXeoikib73AlUe2FkHiKXa7Wfh/qG5HkZ+4e//LxXAR5BIBMGrUMjDGRr9cFrXw2hpqgTkYqHqO3XfN5ZoJ6k/CJL8KovXKqKRqUjaL9s5AshhK0sRbUUoiMapC8KGiTytDToKZcAsfHitwPEsqQu4RSd7s+CRFr9G+KT/+eE= X-Forefront-Antispam-Report: CIP:216.228.118.233; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:dc7edge2.nvidia.com; CAT:NONE; SFS:(13230025)(4636009)(396003)(39860400002)(376002)(346002)(136003)(451199018)(46966006)(36840700001)(40470700004)(40460700003)(7636003)(356005)(2906002)(36860700001)(1076003)(2616005)(186003)(26005)(6666004)(40480700001)(86362001)(426003)(83380400001)(336012)(4326008)(6916009)(7416002)(70586007)(70206006)(5660300002)(47076005)(8676002)(36756003)(41300700001)(7696005)(316002)(82740400003)(54906003)(8936002)(478600001)(82310400005)(14143004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jan 2023 18:51:24.3808 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: bdab153a-24f1-495d-3743-08daffce52eb X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.118.233]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT022.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6161 Received-SPF: softfail client-ip=40.107.243.40; envelope-from=avihaih@nvidia.com; helo=NAM12-DM6-obe.outbound.protection.outlook.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @Nvidia.com) X-ZM-MESSAGEID: 1674759255337100002 Content-Type: text/plain; charset="utf-8" Adjust the VFIO dirty page tracking documentation and add a section to describe device dirty page tracking. Signed-off-by: Avihai Horon --- docs/devel/vfio-migration.rst | 50 ++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/docs/devel/vfio-migration.rst b/docs/devel/vfio-migration.rst index 51f5e1a537..c02f62e534 100644 --- a/docs/devel/vfio-migration.rst +++ b/docs/devel/vfio-migration.rst @@ -63,22 +63,37 @@ System memory dirty pages tracking ---------------------------------- =20 A ``log_global_start`` and ``log_global_stop`` memory listener callback in= forms -the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync`` -memory listener callback marks those system memory pages as dirty which are -used for DMA by the VFIO device. The dirty pages bitmap is queried per -container. All pages pinned by the vendor driver through external APIs hav= e to -be marked as dirty during migration. When there are CPU writes, CPU dirty = page -tracking can identify dirtied pages, but any page pinned by the vendor dri= ver -can also be written by the device. There is currently no device or IOMMU -support for dirty page tracking in hardware. +the VFIO dirty tracking module to start and stop dirty page tracking. A +``log_sync`` memory listener callback queries the dirty page bitmap from t= he +dirty tracking module and marks system memory pages which were DMA-ed by t= he +VFIO device as dirty. The dirty page bitmap is queried per container. + +Currently there are two ways dirty page tracking can be done: +(1) Device dirty tracking: +In this method the device is responsible to log and report its DMAs. This +method can be used only if the device is capable of tracking its DMAs. +Discovering device capability, starting and stopping dirty tracking, and +syncing the dirty bitmaps from the device are done using the DMA logging u= API. +More info about the uAPI can be found in the comments of the +``vfio_device_feature_dma_logging_control`` and +``vfio_device_feature_dma_logging_report`` structures in the header file +linux-headers/linux/vfio.h. + +(2) VFIO IOMMU module: +In this method dirty tracking is done by IOMMU. However, there is currentl= y no +IOMMU support for dirty page tracking. For this reason, all pages are +perpetually marked dirty, unless the device driver pins pages through exte= rnal +APIs in which case only those pinned pages are perpetually marked dirty. + +If the above two methods are not supported, all pages are perpetually mark= ed +dirty by QEMU. =20 By default, dirty pages are tracked during pre-copy as well as stop-and-co= py -phase. So, a page pinned by the vendor driver will be copied to the destin= ation -in both phases. Copying dirty pages in pre-copy phase helps QEMU to predic= t if -it can achieve its downtime tolerances. If QEMU during pre-copy phase keeps -finding dirty pages continuously, then it understands that even in stop-an= d-copy -phase, it is likely to find dirty pages and can predict the downtime -accordingly. +phase. So, a page marked as dirty will be copied to the destination in both +phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it = can +achieve its downtime tolerances. If QEMU during pre-copy phase keeps findi= ng +dirty pages continuously, then it understands that even in stop-and-copy p= hase, +it is likely to find dirty pages and can predict the downtime accordingly. =20 QEMU also provides a per device opt-out option ``pre-copy-dirty-page-track= ing`` which disables querying the dirty bitmap during pre-copy phase. If it is s= et to @@ -89,10 +104,9 @@ System memory dirty pages tracking when vIOMMU is enabl= ed --------------------------------------------------------- =20 With vIOMMU, an IO virtual address range can get unmapped while in pre-copy -phase of migration. In that case, the unmap ioctl returns any dirty pages = in -that range and QEMU reports corresponding guest physical pages dirty. Duri= ng -stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped -pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for t= hose +phase of migration. In that case, dirty page bitmap for this range is quer= ied +and synced with QEMU. During stop-and-copy phase, an IOMMU notifier is use= d to +get a callback for mapped pages and then dirty page bitmap is fetched for = those mapped ranges. =20 Flow of state changes during Live migration --=20 2.26.3