From nobody Thu Apr 3 10:15:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 174112682197557.19709190883941; Tue, 4 Mar 2025 14:20:21 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tpaZk-0005RF-EX; Tue, 04 Mar 2025 17:17:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tpaYJ-000478-Oa for qemu-devel@nongnu.org; Tue, 04 Mar 2025 17:15:53 -0500 Received: from vps-ovh.mhejs.net ([145.239.82.108]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tpaXY-0001Us-PB for qemu-devel@nongnu.org; Tue, 04 Mar 2025 17:15:50 -0500 Received: from MUA by vps-ovh.mhejs.net with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.98) (envelope-from ) id 1tpaQZ-00000000LcL-3KZI; Tue, 04 Mar 2025 23:07:51 +0100 From: "Maciej S. Szmigiero" To: Peter Xu , Fabiano Rosas Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Eric Blake , Markus Armbruster , =?UTF-8?q?Daniel=20P=20=2E=20Berrang=C3=A9?= , Avihai Horon , Joao Martins , qemu-devel@nongnu.org Subject: [PATCH v6 36/36] vfio/migration: Update VFIO migration documentation Date: Tue, 4 Mar 2025 23:04:03 +0100 Message-ID: <6bbaaab45a9b8a2707c63570635255183eb1e997.1741124640.git.maciej.szmigiero@oracle.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=145.239.82.108; envelope-from=mhej@vps-ovh.mhejs.net; helo=vps-ovh.mhejs.net X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1741126823856019100 Content-Type: text/plain; charset="utf-8" From: "Maciej S. Szmigiero" Update the VFIO documentation at docs/devel/migration describing the changes brought by the multifd device state transfer. Signed-off-by: Maciej S. Szmigiero --- docs/devel/migration/vfio.rst | 79 +++++++++++++++++++++++++++++++---- 1 file changed, 72 insertions(+), 7 deletions(-) diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index c49482eab66d..fb76a38c333f 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -67,14 +67,39 @@ VFIO implements the device hooks for the iterative appr= oach as follows: * A ``switchover_ack_needed`` function that checks if the VFIO device uses "switchover-ack" migration capability when this capability is enabled. =20 -* A ``save_state`` function to save the device config space if it is prese= nt. +* A ``switchover_start`` function that in the multifd mode starts a thread= that + reassembles the multifd received data and loads it in-order into the dev= ice. + In the non-multifd mode this function is a NOP. =20 -* A ``save_live_complete_precopy`` function that sets the VFIO device in - _STOP_COPY state and iteratively copies the data for the VFIO device unt= il - the vendor driver indicates that no data remains. +* A ``save_state`` function to save the device config space if it is prese= nt + in the non-multifd mode. + In the multifd mode it just emits either a dummy EOS marker or + "all iterables were loaded" flag for configurations that need to defer + loading device config space after them. =20 -* A ``load_state`` function that loads the config section and the data - sections that are generated by the save functions above. +* A ``save_live_complete_precopy`` function that in the non-multifd mode s= ets + the VFIO device in _STOP_COPY state and iteratively copies the data for = the + VFIO device until the vendor driver indicates that no data remains. + In the multifd mode it just emits a dummy EOS marker. + +* A ``save_live_complete_precopy_thread`` function that in the multifd mode + provides thread handler performing multifd device state transfer. + It sets the VFIO device to _STOP_COPY state, iteratively reads the data + from the VFIO device and queues it for multifd transmission until the ve= ndor + driver indicates that no data remains. + After that, it saves the device config space and queues it for multifd + transfer too. + In the non-multifd mode this thread is a NOP. + +* A ``load_state`` function that loads the data sections that are generated + by the main migration channel save functions above. + In the non-multifd mode it also loads the config section, while in the + multifd mode it handles the optional "all iterables were loaded" flag if + it is in use. + +* A ``load_state_buffer`` function that loads the device state and the dev= ice + config that arrived via multifd channels. + It's used only in the multifd mode. =20 * ``cleanup`` functions for both save and load that perform any migration related cleanup. @@ -176,8 +201,11 @@ Live migration save path Then the VFIO device is put in _STOP_COPY state (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) .save_live_complete_precopy() is called for each active device - For the VFIO device, iterate in .save_live_complete_precopy() until + For the VFIO device: in the non-multifd mode iterate in + .save_live_complete_precopy() until pending data is 0 + In the multifd mode this iteration is done in + .save_live_complete_precopy_thread() instead. | (POSTMIGRATE, _COMPLETED, _STOP_COPY) Migraton thread schedules cleanup bottom half and exits @@ -194,6 +222,9 @@ Live migration resume path (RESTORE_VM, _ACTIVE, _STOP) | For each device, .load_state() is called for that device section data + transmitted via the main migration channel. + For data transmitted via multifd channels .load_state_buffer() is cal= led + instead. (RESTORE_VM, _ACTIVE, _RESUMING) | At the end, .load_cleanup() is called for each device and vCPUs are star= ted @@ -206,3 +237,37 @@ Postcopy =3D=3D=3D=3D=3D=3D=3D=3D =20 Postcopy migration is currently not supported for VFIO devices. + +Multifd +=3D=3D=3D=3D=3D=3D=3D + +Starting from QEMU version 10.0 there's a possibility to transfer VFIO dev= ice +_STOP_COPY state via multifd channels. This helps reduce downtime - especi= ally +with multiple VFIO devices or with devices having a large migration state. +As an additional benefit, setting the VFIO device to _STOP_COPY state and +saving its config space is also parallelized (run in a separate thread) in +such migration mode. + +The multifd VFIO device state transfer is controlled by +"x-migration-multifd-transfer" VFIO device property. This property default= s to +AUTO, which means that VFIO device state transfer via multifd channels is +attempted in configurations that otherwise support it. + +Since the target QEMU needs to load device state buffers in-order it needs= to +queue incoming buffers until they can be loaded into the device. +This means that a malicious QEMU source could theoretically cause the targ= et +QEMU to allocate unlimited amounts of memory for such buffers-in-flight. + +The "x-migration-max-queued-buffers" property allows capping the maximum c= ount +of these VFIO device state buffers queued at the destination. + +Because a malicious QEMU source causing OOM on the target is not expected = to be +a realistic threat in most of VFIO live migration use cases and the right = value +depends on the particular setup by default this queued buffers limit is +disabled by setting it to UINT64_MAX. + +Some host platforms (like ARM64) require that VFIO device config is loaded= only +after all iterables were loaded. +Such interlocking is controlled by "x-migration-load-config-after-iter" VF= IO +device property, which in its default setting (AUTO) does so only on platf= orms +that actually require it.