From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270557; cv=none; d=zohomail.com; s=zohoarc; b=GfdaMAV9fTTgorEX3d2RqKyMFWgUFvV8+F0KKDGtum/rhm3RGx1lMstg/qH62eo3+4LeiWwuDfd/l4WpkB5dCFpA+Gy9DyCpvw5KIW+rzagQON/qusA1S0Pmp4PGlWds/cU910Z9sLAzZvs0s341LBkqULJBgg8h7iA5s0nYKKA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270557; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=jsDCMU6OOVfdTbqxRxkrrzYD3sdMuJak9AFaBj57dak=; b=LzjeO6CqlwNd/O4B5XaBevqFPUGvxnpN6Fz09FZ3ZeT8gRCbdrSrwGIfNNDa71urwjyMHXpDXNG/ZyjD6vrHIj6VBKo17fVQPJGgsFTjqAPRF8kuukERHalcETNMgyE03yezKutGLvrkg6Zip5cqmSX0Q7l4y76FUgjeVQaRSDE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270557460895.9829518506782; Thu, 6 Mar 2025 06:15:57 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqBzz-0006mE-B8; Thu, 06 Mar 2025 09:14:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzx-0006ka-Aq for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzv-0001Z4-3z for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:53 -0500 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-301-Nf3AbrjyNMWt5vJ7pNoglA-1; Thu, 06 Mar 2025 09:14:32 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A618A1954B1E; Thu, 6 Mar 2025 14:14:29 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6398E18001D3; Thu, 6 Mar 2025 14:14:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jsDCMU6OOVfdTbqxRxkrrzYD3sdMuJak9AFaBj57dak=; b=eJKOVuESYQ1TNoQcBxWC5mYivwg4qfg8FEclUU1s39DT8D82w9qBo7wPhTB4TvnxMubHFq QupQy6UoBJD0E4V9y6FCeHwY3IuAtEUNmL9oZhwfPTFIHAMaK0nVpE+BZAhHRX0tWYh2R8 89pkFW7VZoELlM0TBP/6IQ4GcEhr9Zs= X-MC-Unique: Nf3AbrjyNMWt5vJ7pNoglA-1 X-Mimecast-MFC-AGG-ID: Nf3AbrjyNMWt5vJ7pNoglA_1741270470 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Tony Krowiak , Eric Farman , Eric Auger , Kirti Wankhede Subject: [PULL 01/42] vfio: Add property documentation Date: Thu, 6 Mar 2025 15:13:37 +0100 Message-ID: <20250306141419.2015340-2-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270558738019000 Investigate the git history to uncover when and why the VFIO properties were introduced and update the models. This is mostly targeting vfio-pci device, since vfio-platform, vfio-ap and vfio-ccw devices are simpler. Sort the properties based on the QEMU version in which they were introduced. Cc: Tony Krowiak Cc: Eric Farman Cc: Eric Auger Reviewed-by: Kirti Wankhede Reviewed-by: Anthony Krowiak Reviewed-by: Eric Farman # vfio-ccw Reviewed-by: Alex Williamson Reviewed-by: Eric Auger Link: https://lore.kernel.org/qemu-devel/20250217173455.449983-1-clg@redhat= .com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/ap.c | 9 ++++ hw/vfio/ccw.c | 15 ++++++ hw/vfio/pci.c | 125 +++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/platform.c | 24 +++++++++ 4 files changed, 173 insertions(+) diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index 30b08ad375d5ecae886c5000fbaa364799fe76d0..c7ab4ff57ada0ed0e5a76f52b5a= 05c86ca4fe0b4 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -257,6 +257,15 @@ static void vfio_ap_class_init(ObjectClass *klass, voi= d *data) dc->hotpluggable =3D true; device_class_set_legacy_reset(dc, vfio_ap_reset); dc->bus_type =3D TYPE_AP_BUS; + + object_class_property_set_description(klass, /* 3.1 */ + "sysfsdev", + "Host sysfs path of assigned dev= ice"); +#ifdef CONFIG_IOMMUFD + object_class_property_set_description(klass, /* 9.0 */ + "iommufd", + "Set host IOMMUFD backend device= "); +#endif } =20 static const TypeInfo vfio_ap_info =3D { diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 67bc137f9be6d43c5970c6271f3cdbfffd8a32de..ea766ae26c744c05515e1cc80a4= 86a3462d82834 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -717,6 +717,21 @@ static void vfio_ccw_class_init(ObjectClass *klass, vo= id *data) cdc->handle_halt =3D vfio_ccw_handle_halt; cdc->handle_clear =3D vfio_ccw_handle_clear; cdc->handle_store =3D vfio_ccw_handle_store; + + object_class_property_set_description(klass, /* 2.10 */ + "sysfsdev", + "Host sysfs path of assigned dev= ice"); + object_class_property_set_description(klass, /* 3.0 */ + "force-orb-pfch", + "Force unlimited prefetch"); +#ifdef CONFIG_IOMMUFD + object_class_property_set_description(klass, /* 9.0 */ + "iommufd", + "Set host IOMMUFD backend device= "); +#endif + object_class_property_set_description(klass, /* 9.2 */ + "loadparm", + "Define which devices that can b= e used for booting"); } =20 static const TypeInfo vfio_ccw_info =3D { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 89d900e9cf0ce364f7c813d81b0317bb3b3e80ca..4f92b50b133060c9199079a0ab6= 20793ecdac0ee 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3433,6 +3433,122 @@ static void vfio_pci_dev_class_init(ObjectClass *kl= ass, void *data) pdc->exit =3D vfio_exitfn; pdc->config_read =3D vfio_pci_read_config; pdc->config_write =3D vfio_pci_write_config; + + object_class_property_set_description(klass, /* 1.3 */ + "host", + "Host PCI address [domain:] of assigned device"); + object_class_property_set_description(klass, /* 1.3 */ + "x-intx-mmap-timeout-ms", + "When EOI is not provided by KVM= /QEMU, wait time " + "(milliseconds) to re-enable dev= ice direct access " + "after INTx (DEBUG)"); + object_class_property_set_description(klass, /* 1.5 */ + "x-vga", + "Expose VGA address spaces for d= evice"); + object_class_property_set_description(klass, /* 2.3 */ + "x-req", + "Disable device request notifica= tion support (DEBUG)"); + object_class_property_set_description(klass, /* 2.4 and 2.5 */ + "x-no-mmap", + "Disable MMAP for device. Allows= to trace MMIO " + "accesses (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-no-kvm-intx", + "Disable direct VFIO->KVM INTx i= njection. Allows to " + "trace INTx interrupts (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-no-kvm-msi", + "Disable direct VFIO->KVM MSI in= jection. Allows to " + "trace MSI interrupts (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-no-kvm-msix", + "Disable direct VFIO->KVM MSIx i= njection. Allows to " + "trace MSIx interrupts (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-pci-vendor-id", + "Override PCI Vendor ID with pro= vided value (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-pci-device-id", + "Override PCI device ID with pro= vided value (DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-pci-sub-vendor-id", + "Override PCI Subsystem Vendor I= D with provided value " + "(DEBUG)"); + object_class_property_set_description(klass, /* 2.5 */ + "x-pci-sub-device-id", + "Override PCI Subsystem Device I= D with provided value " + "(DEBUG)"); + object_class_property_set_description(klass, /* 2.6 */ + "sysfsdev", + "Host sysfs path of assigned dev= ice"); + object_class_property_set_description(klass, /* 2.7 */ + "x-igd-opregion", + "Expose host IGD OpRegion to gue= st"); + object_class_property_set_description(klass, /* 2.7 (See c4c45e943e51)= */ + "x-igd-gms", + "Override IGD data stolen memory= size (32MiB units)"); + object_class_property_set_description(klass, /* 2.11 */ + "x-nv-gpudirect-clique", + "Add NVIDIA GPUDirect capability= indicating P2P DMA " + "clique for device [0-15]"); + object_class_property_set_description(klass, /* 2.12 */ + "x-no-geforce-quirks", + "Disable GeForce quirks (for NVI= DIA Quadro/GRID/Tesla). " + "Improves performance"); + object_class_property_set_description(klass, /* 2.12 */ + "display", + "Enable display support for devi= ce, ex. vGPU"); + object_class_property_set_description(klass, /* 2.12 */ + "x-msix-relocation", + "Specify MSI-X MMIO relocation t= o the end of specified " + "existing BAR or new BAR to avoi= d virtualization overhead " + "due to adjacent device register= s"); + object_class_property_set_description(klass, /* 3.0 */ + "x-no-kvm-ioeventfd", + "Disable registration of ioevent= fds with KVM (DEBUG)"); + object_class_property_set_description(klass, /* 3.0 */ + "x-no-vfio-ioeventfd", + "Disable linking of KVM ioeventf= ds to VFIO ioeventfds " + "(DEBUG)"); + object_class_property_set_description(klass, /* 3.1 */ + "x-balloon-allowed", + "Override allowing ballooning wi= th device (DEBUG, DANGER)"); + object_class_property_set_description(klass, /* 3.2 */ + "xres", + "Set X display resolution the vG= PU should use"); + object_class_property_set_description(klass, /* 3.2 */ + "yres", + "Set Y display resolution the vG= PU should use"); + object_class_property_set_description(klass, /* 5.2 */ + "x-pre-copy-dirty-page-tracking", + "Disable dirty pages tracking du= ring iterative phase " + "(DEBUG)"); + object_class_property_set_description(klass, /* 5.2, 8.0 non-experimet= al */ + "enable-migration", + "Enale device migration. Also re= quires a host VFIO PCI " + "variant or mdev driver with mig= ration support enabled"); + object_class_property_set_description(klass, /* 8.1 */ + "vf-token", + "Specify UUID VF token. Required= for VF when PF is owned " + "by another VFIO driver"); +#ifdef CONFIG_IOMMUFD + object_class_property_set_description(klass, /* 9.0 */ + "iommufd", + "Set host IOMMUFD backend device= "); +#endif + object_class_property_set_description(klass, /* 9.1 */ + "x-device-dirty-page-tracking", + "Disable device dirty page track= ing and use " + "container-based dirty page trac= king (DEBUG)"); + object_class_property_set_description(klass, /* 9.1 */ + "migration-events", + "Emit VFIO migration QAPI event = when a VFIO device " + "changes its migration state. Fo= r management applications"); + object_class_property_set_description(klass, /* 9.1 */ + "skip-vsc-check", + "Skip config space check for Ven= dor Specific Capability. " + "Setting to false will enforce s= trict checking of VSC content " + "(DEBUG)"); } =20 static const TypeInfo vfio_pci_dev_info =3D { @@ -3461,6 +3577,15 @@ static void vfio_pci_nohotplug_dev_class_init(Object= Class *klass, void *data) =20 device_class_set_props(dc, vfio_pci_dev_nohotplug_properties); dc->hotpluggable =3D false; + + object_class_property_set_description(klass, /* 3.1 */ + "ramfb", + "Enable ramfb to provide pre-boo= t graphics for devices " + "enabling display option"); + object_class_property_set_description(klass, /* 8.2 */ + "x-ramfb-migrate", + "Override default migration supp= ort for ramfb support " + "(DEBUG)"); } =20 static const TypeInfo vfio_pci_nohotplug_dev_info =3D { diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index f491f4dc9543c2ea3a7af4e51ee82fbc7ee6e4bb..d9faaa73959ad36aa3a835b8796= 4ab940928bd9f 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -672,6 +672,30 @@ static void vfio_platform_class_init(ObjectClass *klas= s, void *data) dc->desc =3D "VFIO-based platform device assignment"; sbc->connect_irq_notifier =3D vfio_start_irqfd_injection; set_bit(DEVICE_CATEGORY_MISC, dc->categories); + + object_class_property_set_description(klass, /* 2.4 */ + "host", + "Host device name of assigned de= vice"); + object_class_property_set_description(klass, /* 2.4 and 2.5 */ + "x-no-mmap", + "Disable MMAP for device. Allows= to trace MMIO " + "accesses (DEBUG)"); + object_class_property_set_description(klass, /* 2.4 */ + "mmap-timeout-ms", + "When EOI is not provided by KVM= /QEMU, wait time " + "(milliseconds) to re-enable dev= ice direct access " + "after level interrupt (DEBUG)"); + object_class_property_set_description(klass, /* 2.4 */ + "x-irqfd", + "Allow disabling irqfd support (= DEBUG)"); + object_class_property_set_description(klass, /* 2.6 */ + "sysfsdev", + "Host sysfs path of assigned dev= ice"); +#ifdef CONFIG_IOMMUFD + object_class_property_set_description(klass, /* 9.0 */ + "iommufd", + "Set host IOMMUFD backend device= "); +#endif } =20 static const TypeInfo vfio_platform_dev_info =3D { --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270508; cv=none; d=zohomail.com; s=zohoarc; b=W6xy8Y9cQtXpu2C1Z7C0afPGN1/IMV2C0mCbJSSSK5MpZLAi2kuMjGt5WHGWkHWjL9kdvwfYH/ZFDp8Z8isEPk7Benp0JNZDwUig914+bSXuNj2nEUI6XrIhBIAWco5ozRzGpY1pybemnVAUspeZJ/xRb28bdLLuxl3XNZZtTA8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270508; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=96NNoeYmbYX997FSeXO0A+RNZHQyES2WQ6gVPAsS7hQ=; b=HY7RQQFzPEdhoxeZzidTrnz61yYopReaj2jtZhkDmHSNXfzbj0g2WomiA94bD0pLo+hPOfiZhkjgM2LlpwHMDKUTm2IytyEwZ6ZaIhryO2W0jpiIpgAQ4AhIj4E9eoyylt/dxlLDWyxTb/JwIHx+MsC9F4NefABa1DQeC7qgtiw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270508700222.92488003846393; Thu, 6 Mar 2025 06:15:08 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqBzs-0006j8-ME; Thu, 06 Mar 2025 09:14:48 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzq-0006i7-Dl for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzo-0001YK-Ok for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:46 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-18-k_qVS7vDPveblp5hoSpzUA-1; Thu, 06 Mar 2025 09:14:35 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AB613180035E; Thu, 6 Mar 2025 14:14:34 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2BA1F18009BC; Thu, 6 Mar 2025 14:14:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270483; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=96NNoeYmbYX997FSeXO0A+RNZHQyES2WQ6gVPAsS7hQ=; b=A2TxifiFaVIM1h6ODNXFrAeR9nB4WOkqkXPDwAirq2OP+GCvBhwR4IJGyN062xt7h1tJ7f rjoLolGhZXXSmTZnOIiSpZgHcI0xA8TlcTu5a4jAnbUIK0nLl00/rU4T0HkVlenCa6DzfH uYQwkpi04er1hjLDfO+fqHjro0CgyFY= X-MC-Unique: k_qVS7vDPveblp5hoSpzUA-1 X-Mimecast-MFC-AGG-ID: k_qVS7vDPveblp5hoSpzUA_1741270475 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Eric Farman Subject: [PULL 02/42] vfio/ccw: Replace warn_once_pfch() with warn_report_once() Date: Thu, 6 Mar 2025 15:13:38 +0100 Message-ID: <20250306141419.2015340-3-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270510087019000 Use the common helper warn_report_once() instead of implementing its own. Cc: Eric Farman Reviewed-by: Eric Farman Link: https://lore.kernel.org/qemu-devel/20250214161936.1720039-1-clg@redha= t.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/ccw.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index ea766ae26c744c05515e1cc80a486a3462d82834..e5e0d9e3e7ed124f242b3eda345= bc973e418a64c 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -51,17 +51,8 @@ struct VFIOCCWDevice { EventNotifier crw_notifier; EventNotifier req_notifier; bool force_orb_pfch; - bool warned_orb_pfch; }; =20 -static inline void warn_once_pfch(VFIOCCWDevice *vcdev, SubchDev *sch, - const char *msg) -{ - warn_report_once_cond(&vcdev->warned_orb_pfch, - "vfio-ccw (devno %x.%x.%04x): %s", - sch->cssid, sch->ssid, sch->devno, msg); -} - static void vfio_ccw_compute_needs_reset(VFIODevice *vdev) { vdev->needs_reset =3D false; @@ -83,7 +74,8 @@ static IOInstEnding vfio_ccw_handle_request(SubchDev *sch) =20 if (!(sch->orb.ctrl0 & ORB_CTRL0_MASK_PFCH) && vcdev->force_orb_pfch) { sch->orb.ctrl0 |=3D ORB_CTRL0_MASK_PFCH; - warn_once_pfch(vcdev, sch, "PFCH flag forced"); + warn_report_once("vfio-ccw (devno %x.%x.%04x): PFCH flag forced", + sch->cssid, sch->ssid, sch->devno); } =20 QEMU_BUILD_BUG_ON(sizeof(region->orb_area) !=3D sizeof(ORB)); --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270509; cv=none; d=zohomail.com; s=zohoarc; b=JSwXbbhkTkqytQlNCTSDnA752AB60eleWiV2Wm/XgXzVb14Has7HfMuH4SZ3Zomq4MzbogfcfHZ4/RcpMsRTpi6POQOa4vu9Ih+HfYkStdsdI4G3yQ0Eg9x0fOv1D3PewP0gZigI7wseUdLs4qOTaocGywqH4K8QZlzIJ3p+m6U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270509; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=s48Mv6Z7UKnwAVAi7fOx0Bp9CMRWP4ErHPxAcs9KJ60=; b=mb+Xmq3wz9hlK1uI/dRKWjEHOJqomYLtg5uaNQ3StCPddX4XxKDOpb8f7LQDmkA0Cy2oxtXDqRx5yCQrxpL9K6Yush4R3KIKLagIrngk+sMnYl1UlNvbFEfMQIe7dJK/iz3J8OJ8DOQGjSukhk27NnJ2ZZ6r+p9xQm3kkQuYcm8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270509012991.4349557292492; Thu, 6 Mar 2025 06:15:09 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqBzr-0006ih-Si; Thu, 06 Mar 2025 09:14:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzq-0006i6-Dl for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzn-0001Y1-VO for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:46 -0500 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-184-4KLrfvfPOJmK7fXh0tFzfA-1; Thu, 06 Mar 2025 09:14:38 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DB99A19560B4; Thu, 6 Mar 2025 14:14:37 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 135851801748; Thu, 6 Mar 2025 14:14:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s48Mv6Z7UKnwAVAi7fOx0Bp9CMRWP4ErHPxAcs9KJ60=; b=VxnCIt9yGvM+mxTjNUkBi2FRl3yBmh66gi4y59DoDMrcU++MZvKWYq4uXy8rZhZR1xXXDk VpDirC2zxgU7zyZJ2Sh3zRlQ4jmUAj83R+XwClu+vGclP5lCHHO+zIwmvgjpIixLxQaY5U X1TV64n0D/1tm1Pil4qopO3Z+BNAK0E= X-MC-Unique: 4KLrfvfPOJmK7fXh0tFzfA-1 X-Mimecast-MFC-AGG-ID: 4KLrfvfPOJmK7fXh0tFzfA_1741270478 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Michael S. Tsirkin" , Marcel Apfelbaum , Eric Auger , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 03/42] hw/pci: Basic support for PCI power management Date: Thu, 6 Mar 2025 15:13:39 +0100 Message-ID: <20250306141419.2015340-4-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270510247019000 From: Alex Williamson The memory and IO BARs for devices are only accessible in the D0 power state. In other power states the PCI spec defines that the device responds to TLPs and messages with an Unsupported Request response. To approximate this behavior, consider the BARs as unmapped when the device is not in the D0 power state. This makes the BARs inaccessible and has the additional bonus for vfio-pci that we don't attempt to DMA map BARs for devices in a non-D0 power state. To support this, an interface is added for devices to register the PM capability, which allows central tracking to enforce valid transitions and unmap BARs in non-D0 states. NB. We currently have device models (eepro100 and pcie_pci_bridge) that register a PM capability but do not set wmask to enable writes to the power state field. In order to maintain migration compatibility, this new helper does not manage the wmask to enable guest writes to initiate a power state change. The contents and write access of the PM capability are still managed by the caller. Cc: Michael S. Tsirkin Cc: Marcel Apfelbaum Signed-off-by: Alex Williamson Reviewed-by: Eric Auger Reviewed-by: Michael S. Tsirkin Link: https://lore.kernel.org/qemu-devel/20250225215237.3314011-2-alex.will= iamson@redhat.com Signed-off-by: C=C3=A9dric Le Goater --- include/hw/pci/pci.h | 3 ++ include/hw/pci/pci_device.h | 3 ++ hw/pci/pci.c | 93 ++++++++++++++++++++++++++++++++++++- hw/pci/trace-events | 2 + 4 files changed, 99 insertions(+), 2 deletions(-) diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index 4002bbeebde0968aec07146f045d0dcfabcda36e..c220cc844962416b6b3eaac4283= 548e66a00ca61 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -216,6 +216,8 @@ enum { QEMU_PCIE_ARI_NEXTFN_1 =3D (1 << QEMU_PCIE_ARI_NEXTFN_1_BITNR), #define QEMU_PCIE_EXT_TAG_BITNR 13 QEMU_PCIE_EXT_TAG =3D (1 << QEMU_PCIE_EXT_TAG_BITNR), +#define QEMU_PCI_CAP_PM_BITNR 14 + QEMU_PCI_CAP_PM =3D (1 << QEMU_PCI_CAP_PM_BITNR), }; =20 typedef struct PCIINTxRoute { @@ -676,5 +678,6 @@ static inline void pci_irq_deassert(PCIDevice *pci_dev) MSIMessage pci_get_msi_message(PCIDevice *dev, int vector); void pci_set_enabled(PCIDevice *pci_dev, bool state); void pci_set_power(PCIDevice *pci_dev, bool state); +int pci_pm_init(PCIDevice *pci_dev, uint8_t offset, Error **errp); =20 #endif diff --git a/include/hw/pci/pci_device.h b/include/hw/pci/pci_device.h index add208edfabdae8902598042bbcd4efbec0a80b8..345b12eaac1a192bbeace8ff522= 7bf04c5edbcc7 100644 --- a/include/hw/pci/pci_device.h +++ b/include/hw/pci/pci_device.h @@ -105,6 +105,9 @@ struct PCIDevice { /* Capability bits */ uint32_t cap_present; =20 + /* Offset of PM capability in config space */ + uint8_t pm_cap; + /* Offset of MSI-X capability in config space */ uint8_t msix_cap; =20 diff --git a/hw/pci/pci.c b/hw/pci/pci.c index 1d42847ef04457e49af535b646c0beb0c543dedf..2ef7f617491498d6e6853e91e9f= b7321037475f5 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -435,6 +435,84 @@ static void pci_msi_trigger(PCIDevice *dev, MSIMessage= msg) attrs, NULL); } =20 +/* + * Register and track a PM capability. If wmask is also enabled for the p= ower + * state field of the pmcsr register, guest writes may change the device PM + * state. BAR access is only enabled while the device is in the D0 state. + * Return the capability offset or negative error code. + */ +int pci_pm_init(PCIDevice *d, uint8_t offset, Error **errp) +{ + int cap =3D pci_add_capability(d, PCI_CAP_ID_PM, offset, PCI_PM_SIZEOF= , errp); + + if (cap < 0) { + return cap; + } + + d->pm_cap =3D cap; + d->cap_present |=3D QEMU_PCI_CAP_PM; + + return cap; +} + +static uint8_t pci_pm_state(PCIDevice *d) +{ + uint16_t pmcsr; + + if (!(d->cap_present & QEMU_PCI_CAP_PM)) { + return 0; + } + + pmcsr =3D pci_get_word(d->config + d->pm_cap + PCI_PM_CTRL); + + return pmcsr & PCI_PM_CTRL_STATE_MASK; +} + +/* + * Update the PM capability state based on the new value stored in config + * space respective to the old, pre-write state provided. If the new value + * is rejected (unsupported or invalid transition) restore the old value. + * Return the resulting PM state. + */ +static uint8_t pci_pm_update(PCIDevice *d, uint32_t addr, int l, uint8_t o= ld) +{ + uint16_t pmc; + uint8_t new; + + if (!(d->cap_present & QEMU_PCI_CAP_PM) || + !range_covers_byte(addr, l, d->pm_cap + PCI_PM_CTRL)) { + return old; + } + + new =3D pci_pm_state(d); + if (new =3D=3D old) { + return old; + } + + pmc =3D pci_get_word(d->config + d->pm_cap + PCI_PM_PMC); + + /* + * Transitions to D1 & D2 are only allowed if supported. Devices may + * only transition to higher D-states or to D0. + */ + if ((!(pmc & PCI_PM_CAP_D1) && new =3D=3D 1) || + (!(pmc & PCI_PM_CAP_D2) && new =3D=3D 2) || + (old && new && new < old)) { + pci_word_test_and_clear_mask(d->config + d->pm_cap + PCI_PM_CTRL, + PCI_PM_CTRL_STATE_MASK); + pci_word_test_and_set_mask(d->config + d->pm_cap + PCI_PM_CTRL, + old); + trace_pci_pm_bad_transition(d->name, pci_dev_bus_num(d), + PCI_SLOT(d->devfn), PCI_FUNC(d->devfn), + old, new); + return old; + } + + trace_pci_pm_transition(d->name, pci_dev_bus_num(d), PCI_SLOT(d->devfn= ), + PCI_FUNC(d->devfn), old, new); + return new; +} + static void pci_reset_regions(PCIDevice *dev) { int r; @@ -474,6 +552,11 @@ static void pci_do_device_reset(PCIDevice *dev) pci_get_word(dev->wmask + PCI_INTERRUPT_LINE= ) | pci_get_word(dev->w1cmask + PCI_INTERRUPT_LI= NE)); dev->config[PCI_CACHE_LINE_SIZE] =3D 0x0; + /* Default PM state is D0 */ + if (dev->cap_present & QEMU_PCI_CAP_PM) { + pci_word_test_and_clear_mask(dev->config + dev->pm_cap + PCI_PM_CT= RL, + PCI_PM_CTRL_STATE_MASK); + } pci_reset_regions(dev); pci_update_mappings(dev); =20 @@ -1606,7 +1689,7 @@ static void pci_update_mappings(PCIDevice *d) continue; =20 new_addr =3D pci_bar_address(d, i, r->type, r->size); - if (!d->enabled) { + if (!d->enabled || pci_pm_state(d)) { new_addr =3D PCI_BAR_UNMAPPED; } =20 @@ -1672,6 +1755,7 @@ uint32_t pci_default_read_config(PCIDevice *d, =20 void pci_default_write_config(PCIDevice *d, uint32_t addr, uint32_t val_in= , int l) { + uint8_t new_pm_state, old_pm_state =3D pci_pm_state(d); int i, was_irq_disabled =3D pci_irq_disabled(d); uint32_t val =3D val_in; =20 @@ -1684,11 +1768,16 @@ void pci_default_write_config(PCIDevice *d, uint32_= t addr, uint32_t val_in, int d->config[addr + i] =3D (d->config[addr + i] & ~wmask) | (val & wm= ask); d->config[addr + i] &=3D ~(val & w1cmask); /* W1C: Write 1 to Clea= r */ } + + new_pm_state =3D pci_pm_update(d, addr, l, old_pm_state); + if (ranges_overlap(addr, l, PCI_BASE_ADDRESS_0, 24) || ranges_overlap(addr, l, PCI_ROM_ADDRESS, 4) || ranges_overlap(addr, l, PCI_ROM_ADDRESS1, 4) || - range_covers_byte(addr, l, PCI_COMMAND)) + range_covers_byte(addr, l, PCI_COMMAND) || + !!new_pm_state !=3D !!old_pm_state) { pci_update_mappings(d); + } =20 if (ranges_overlap(addr, l, PCI_COMMAND, 2)) { pci_update_irq_disabled(d, was_irq_disabled); diff --git a/hw/pci/trace-events b/hw/pci/trace-events index e98f575a9d19ba9ce8d391fef1326aa317c322f9..6a9968962f4afcc2c0a33ced257= 94bea7282efbb 100644 --- a/hw/pci/trace-events +++ b/hw/pci/trace-events @@ -1,6 +1,8 @@ # See docs/devel/tracing.rst for syntax documentation. =20 # pci.c +pci_pm_bad_transition(const char *dev, uint32_t bus, uint32_t slot, uint32= _t func, uint8_t old, uint8_t new) "%s %02x:%02x.%x REJECTED PM transition = D%d->D%d" +pci_pm_transition(const char *dev, uint32_t bus, uint32_t slot, uint32_t f= unc, uint8_t old, uint8_t new) "%s %02x:%02x.%x PM transition D%d->D%d" pci_update_mappings_del(const char *dev, uint32_t bus, uint32_t slot, uint= 32_t func, int bar, uint64_t addr, uint64_t size) "%s %02x:%02x.%x %d,0x%"P= RIx64"+0x%"PRIx64 pci_update_mappings_add(const char *dev, uint32_t bus, uint32_t slot, uint= 32_t func, int bar, uint64_t addr, uint64_t size) "%s %02x:%02x.%x %d,0x%"P= RIx64"+0x%"PRIx64 pci_route_irq(int dev_irq, const char *dev_path, int parent_irq, const cha= r *parent_path) "IRQ %d @%s -> IRQ %d @%s" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270576; cv=none; d=zohomail.com; s=zohoarc; b=aEKSQS77lPCFn0zOFhm5izYEz1bq6L0iKon63ySHPdGzabimNjwLY8WfC7bcLcamHWTSitop2dDzG00Nu9KT7B7D3m+mHSyJ/u9CMp4R5c55i8eAWRiKTTPgJk7fdIGdScTWxiZhziLs5TJuIH5CstEqQK8MnC83teYUnTU527E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270576; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=S7xl4PVyNoAGYZsF4VN6ErV6AJOrELxS7uo5iP6Ku9o=; b=YOuILsve+DDVF4iGQ6L/3o0r4VEfCgh3s2bNh7f3eomAdES/M7SJA7sfO5g/m6HdOpSZJ4J7b8cQKJtNAV1f3f3mmnldTr2fXxexXglMwB/D1Gpq2pk1iNgLVlejFW9HbBVcEBh6DFsi+cZHGrbliaRTAqVfXGSFsIKEXwvpACY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270576007242.6539109844175; Thu, 6 Mar 2025 06:16:16 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC00-0006nP-GL; Thu, 06 Mar 2025 09:14:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzy-0006mC-VN for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:54 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzw-0001ZE-5s for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:54 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-296-L3geaJjbN9Wcz8T4d3qzYQ-1; Thu, 06 Mar 2025 09:14:45 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 11FC51828A87; Thu, 6 Mar 2025 14:14:44 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 62F6018001D3; Thu, 6 Mar 2025 14:14:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270491; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S7xl4PVyNoAGYZsF4VN6ErV6AJOrELxS7uo5iP6Ku9o=; b=fupyYCH0T/sd8qJBauY/+zqST9or8TtxkmKRL8EotFMR1SisJoKT+SdA9z014gYE9qJoJX ulUn83SHkJO1y4+JJl4NGW8GjVVCmt2Q7Cnu70OW/0I01/s+NISNJCUx/vsLs8+sfE22kb OE7kZa+j6uWRZbCHUi+PNn8ARg+XIjo= X-MC-Unique: L3geaJjbN9Wcz8T4d3qzYQ-1 X-Mimecast-MFC-AGG-ID: L3geaJjbN9Wcz8T4d3qzYQ_1741270484 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Dmitry Fleytman , Akihiko Odaki , Jason Wang , Stefan Weil , Sriram Yagnaraman , Keith Busch , Klaus Jensen , Jesper Devantier , "Michael S. Tsirkin" , Marcel Apfelbaum , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Eric Auger Subject: [PULL 04/42] pci: Use PCI PM capability initializer Date: Thu, 6 Mar 2025 15:13:40 +0100 Message-ID: <20250306141419.2015340-5-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270581034019000 From: Alex Williamson Switch callers directly initializing the PCI PM capability with pci_add_capability() to use pci_pm_init(). Cc: Dmitry Fleytman Cc: Akihiko Odaki Cc: Jason Wang Cc: Stefan Weil Cc: Sriram Yagnaraman Cc: Keith Busch Cc: Klaus Jensen Cc: Jesper Devantier Cc: Michael S. Tsirkin Cc: Marcel Apfelbaum Cc: C=C3=A9dric Le Goater Signed-off-by: Alex Williamson Reviewed-by: Eric Auger Reviewed-by: Akihiko Odaki Reviewed-by: Michael S. Tsirkin Link: https://lore.kernel.org/qemu-devel/20250225215237.3314011-3-alex.will= iamson@redhat.com Signed-off-by: C=C3=A9dric Le Goater --- hw/net/e1000e.c | 3 +-- hw/net/eepro100.c | 4 +--- hw/net/igb.c | 3 +-- hw/nvme/ctrl.c | 3 +-- hw/pci-bridge/pcie_pci_bridge.c | 2 +- hw/vfio/pci.c | 7 ++++++- hw/virtio/virtio-pci.c | 3 +-- 7 files changed, 12 insertions(+), 13 deletions(-) diff --git a/hw/net/e1000e.c b/hw/net/e1000e.c index f637853073e2610dd93b4de006f539093883fe07..b72cbab7e88904e3e6e7eb8d190= 4d68ca2e64d9d 100644 --- a/hw/net/e1000e.c +++ b/hw/net/e1000e.c @@ -372,8 +372,7 @@ static int e1000e_add_pm_capability(PCIDevice *pdev, uint8_t offset, uint16_t pmc) { Error *local_err =3D NULL; - int ret =3D pci_add_capability(pdev, PCI_CAP_ID_PM, offset, - PCI_PM_SIZEOF, &local_err); + int ret =3D pci_pm_init(pdev, offset, &local_err); =20 if (local_err) { error_report_err(local_err); diff --git a/hw/net/eepro100.c b/hw/net/eepro100.c index 6d853229aec2ed285d212a48bdfc69812e58c30f..29a39865a6086b8f175be6be003= 11487bc59f25c 100644 --- a/hw/net/eepro100.c +++ b/hw/net/eepro100.c @@ -551,9 +551,7 @@ static void e100_pci_reset(EEPRO100State *s, Error **er= rp) if (info->power_management) { /* Power Management Capabilities */ int cfg_offset =3D 0xdc; - int r =3D pci_add_capability(&s->dev, PCI_CAP_ID_PM, - cfg_offset, PCI_PM_SIZEOF, - errp); + int r =3D pci_pm_init(&s->dev, cfg_offset, errp); if (r < 0) { return; } diff --git a/hw/net/igb.c b/hw/net/igb.c index c965fc2fb68a37699c1cc4be0be2a6467522ccc0..e318df40e0138d5101f99680d17= b97917e6fb559 100644 --- a/hw/net/igb.c +++ b/hw/net/igb.c @@ -356,8 +356,7 @@ static int igb_add_pm_capability(PCIDevice *pdev, uint8_t offset, uint16_t pmc) { Error *local_err =3D NULL; - int ret =3D pci_add_capability(pdev, PCI_CAP_ID_PM, offset, - PCI_PM_SIZEOF, &local_err); + int ret =3D pci_pm_init(pdev, offset, &local_err); =20 if (local_err) { error_report_err(local_err); diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index e62c6a358828b82d6f0cef5de9b49291f17bde96..518d02dc66706e2d2e86f1705db= 52188a97a67fc 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -8600,8 +8600,7 @@ static int nvme_add_pm_capability(PCIDevice *pci_dev,= uint8_t offset) Error *err =3D NULL; int ret; =20 - ret =3D pci_add_capability(pci_dev, PCI_CAP_ID_PM, offset, - PCI_PM_SIZEOF, &err); + ret =3D pci_pm_init(pci_dev, offset, &err); if (err) { error_report_err(err); return ret; diff --git a/hw/pci-bridge/pcie_pci_bridge.c b/hw/pci-bridge/pcie_pci_bridg= e.c index fd4514a595ce08d82773376bd7294a43a551762a..9fa656b43b42624fd597ec6df5d= 986e71e5f06f0 100644 --- a/hw/pci-bridge/pcie_pci_bridge.c +++ b/hw/pci-bridge/pcie_pci_bridge.c @@ -52,7 +52,7 @@ static void pcie_pci_bridge_realize(PCIDevice *d, Error *= *errp) goto cap_error; } =20 - pos =3D pci_add_capability(d, PCI_CAP_ID_PM, 0, PCI_PM_SIZEOF, errp); + pos =3D pci_pm_init(d, 0, errp); if (pos < 0) { goto pm_error; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 4f92b50b133060c9199079a0ab620793ecdac0ee..d33b795af079ae6999a5b4e4074= c202c5a9de362 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2216,7 +2216,12 @@ static bool vfio_add_std_cap(VFIOPCIDevice *vdev, ui= nt8_t pos, Error **errp) case PCI_CAP_ID_PM: vfio_check_pm_reset(vdev, pos); vdev->pm_cap =3D pos; - ret =3D pci_add_capability(pdev, cap_id, pos, size, errp) >=3D 0; + ret =3D pci_pm_init(pdev, pos, errp) >=3D 0; + /* + * PCI-core config space emulation needs write access to the power + * state enabled for tracking BAR mapping relative to PM state. + */ + pci_set_word(pdev->wmask + pos + PCI_PM_CTRL, PCI_PM_CTRL_STATE_MA= SK); break; case PCI_CAP_ID_AF: vfio_check_af_flr(vdev, pos); diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index c773a9130c7e8890b93d20a17b099940284060be..afe8b5551c5cc5d33a184f25ac0= 5efa2c126e796 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -2204,8 +2204,7 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Er= ror **errp) pos =3D pcie_endpoint_cap_init(pci_dev, 0); assert(pos > 0); =20 - pos =3D pci_add_capability(pci_dev, PCI_CAP_ID_PM, 0, - PCI_PM_SIZEOF, errp); + pos =3D pci_pm_init(pci_dev, 0, errp); if (pos < 0) { return; } --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270515; cv=none; d=zohomail.com; s=zohoarc; b=BFejRwYhWFFtI2nyACM8jjEhRN59qw4xvE64xA9AMb8QzOp6fXkAVeXy20xClCUkFEO1tWqq3ImxUcltnxGlhNrU30xxFEHoL3OkySHDDQdf/sIHIXa3mcSGuuYlc7fvauWkwkai3M9XY+2+AGlCcyiGxwVnUDpaz9BwrfVYM3U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270515; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=k/mUuly+YXerGoJ1N3+hUtW3oITKd6pZGZ38TrFk4Go=; b=Z6YyAYbqgE6+z00OVelYAe0IE6ZNBMWoqwpAC/ezF/g+JJaTNLCeuYM+qvEF4azLCQYCHqK8RmpLh2Yrj7v4HHh7khpNK32cvCMguZ5/KkOQF5kIaUngTSyonRq2LSvc6BgtEJgT8bcD+X4jwMj7PJfcmLMIKBG+8NZBekjuzxA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270515664760.1296760407726; Thu, 6 Mar 2025 06:15:15 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqBzz-0006ma-Cx; Thu, 06 Mar 2025 09:14:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzx-0006kX-9O for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqBzv-0001ZA-J2 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:14:53 -0500 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-120-SaMLVixZNRatAXW8gDf33Q-1; Thu, 06 Mar 2025 09:14:48 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 606CE19560B8; Thu, 6 Mar 2025 14:14:47 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8F324180174C; Thu, 6 Mar 2025 14:14:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270491; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k/mUuly+YXerGoJ1N3+hUtW3oITKd6pZGZ38TrFk4Go=; b=TY6vu/OwQwrJ5hpR3mPw0wuWV+qVeWL4fc+ZzPxVBgjip/wGArLhXt8H4PakZvmpYcRvN/ Lo/KTQrxVzo/VOIjdvB6SUTfT7P12e6xH7+0JIf11VW1V9nt2PVAuTl2HC898kTNCx18vo xaQLmFHyUBXHbfoNh6fJOQMxBSpkCgI= X-MC-Unique: SaMLVixZNRatAXW8gDf33Q-1 X-Mimecast-MFC-AGG-ID: SaMLVixZNRatAXW8gDf33Q_1741270487 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Zhenzhong Duan , Eric Auger , "Michael S. Tsirkin" Subject: [PULL 05/42] vfio/pci: Delete local pm_cap Date: Thu, 6 Mar 2025 15:13:41 +0100 Message-ID: <20250306141419.2015340-6-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270516098019000 From: Alex Williamson This is now redundant to PCIDevice.pm_cap. Cc: C=C3=A9dric Le Goater Reviewed-by: Zhenzhong Duan Reviewed-by: Eric Auger Signed-off-by: Alex Williamson Reviewed-by: Michael S. Tsirkin Link: https://lore.kernel.org/qemu-devel/20250225215237.3314011-4-alex.will= iamson@redhat.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/pci.h | 1 - hw/vfio/pci.c | 9 ++++----- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 43c166680abbc2460c75fae33565c3fdeb2c40d8..d638c781f6f168a71d3a44b48ba= 0c4edfd334f44 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -160,7 +160,6 @@ struct VFIOPCIDevice { int32_t bootindex; uint32_t igd_gms; OffAutoPCIBAR msix_relo; - uint8_t pm_cap; uint8_t nv_gpudirect_clique; bool pci_aer; bool req_enabled; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index d33b795af079ae6999a5b4e4074c202c5a9de362..a8db19d8d2f9707693500493eb2= 37c7362d35c06 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2215,7 +2215,6 @@ static bool vfio_add_std_cap(VFIOPCIDevice *vdev, uin= t8_t pos, Error **errp) break; case PCI_CAP_ID_PM: vfio_check_pm_reset(vdev, pos); - vdev->pm_cap =3D pos; ret =3D pci_pm_init(pdev, pos, errp) >=3D 0; /* * PCI-core config space emulation needs write access to the power @@ -2412,17 +2411,17 @@ void vfio_pci_pre_reset(VFIOPCIDevice *vdev) vfio_disable_interrupts(vdev); =20 /* Make sure the device is in D0 */ - if (vdev->pm_cap) { + if (pdev->pm_cap) { uint16_t pmcsr; uint8_t state; =20 - pmcsr =3D vfio_pci_read_config(pdev, vdev->pm_cap + PCI_PM_CTRL, 2= ); + pmcsr =3D vfio_pci_read_config(pdev, pdev->pm_cap + PCI_PM_CTRL, 2= ); state =3D pmcsr & PCI_PM_CTRL_STATE_MASK; if (state) { pmcsr &=3D ~PCI_PM_CTRL_STATE_MASK; - vfio_pci_write_config(pdev, vdev->pm_cap + PCI_PM_CTRL, pmcsr,= 2); + vfio_pci_write_config(pdev, pdev->pm_cap + PCI_PM_CTRL, pmcsr,= 2); /* vfio handles the necessary delay here */ - pmcsr =3D vfio_pci_read_config(pdev, vdev->pm_cap + PCI_PM_CTR= L, 2); + pmcsr =3D vfio_pci_read_config(pdev, pdev->pm_cap + PCI_PM_CTR= L, 2); state =3D pmcsr & PCI_PM_CTRL_STATE_MASK; if (state) { error_report("vfio: Unable to power on device, stuck in D%= d", --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270596; cv=none; d=zohomail.com; s=zohoarc; b=c9/koDle+p47JzMNwWHo8rYWZ1WaERucNShA/9hailFD3nqFASJKRMR9hGUd0kUJ1hdQxrfKeTpxUMRoDUyP0VCcjfBgJjk/GK25TIwzQWRvnzJE/DK4cTMegT5GrMj5VPn9HApQeMzMa0HwWwd1izZF/48NHcwIvtExXTCyY5Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270596; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=dWRgq8lQFjl5YTPNjjtxvseUG3VlTRuWZlzIlG7saZ4=; b=jg33MjulH0zHdrXAzLwosn5evAiFSZ4LRn9lQIA6zgMpLquYqc60DxLaJJac2gWy+qkYr564vbJaqDn/qPnBThCNrhuO0K6bgUi+k5LtQ14LGremQ7ofmLtWBfcbmqcpJdA6T3BvyiQejVTYjdGJTEHtTwtwmGjI8NPqo68/oYU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270596608912.2643213628401; Thu, 6 Mar 2025 06:16:36 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0i-0007Rb-MO; Thu, 06 Mar 2025 09:15:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0E-000744-V6 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0C-0001mB-OV for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:10 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-564-ZY6_RaIMN6uhBnTQ9n42sA-1; Thu, 06 Mar 2025 09:14:51 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 60A0519560B3; Thu, 6 Mar 2025 14:14:50 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 024FA180174D; Thu, 6 Mar 2025 14:14:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dWRgq8lQFjl5YTPNjjtxvseUG3VlTRuWZlzIlG7saZ4=; b=Q6C/DfdqLwbMl/q+WMZUgTKfkx2Iqiwr4TadXPZ9AisCPcc+sLmTAvxufkLJwN/D+lr7i0 2beu92Cu64X0Y5I8lj+v5q9euqS6nfpJruqx3jiFHxeaHBafixhYhsogYK9NLrdS71O1bJ lweHVVvQGXvjR8Czt9Ue5V6JsamZL9M= X-MC-Unique: ZY6_RaIMN6uhBnTQ9n42sA-1 X-Mimecast-MFC-AGG-ID: ZY6_RaIMN6uhBnTQ9n42sA_1741270490 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Michael S. Tsirkin" , Marcel Apfelbaum , Zhenzhong Duan , Eric Auger , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 06/42] pcie, virtio: Remove redundant pm_cap Date: Thu, 6 Mar 2025 15:13:42 +0100 Message-ID: <20250306141419.2015340-7-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270598681019000 From: Alex Williamson The pm_cap on the PCIExpressDevice object can be distilled down to the new instance on the PCIDevice object. Cc: Michael S. Tsirkin Cc: Marcel Apfelbaum Reviewed-by: Michael S. Tsirkin Reviewed-by: Zhenzhong Duan Reviewed-by: Eric Auger Signed-off-by: Alex Williamson Link: https://lore.kernel.org/qemu-devel/20250225215237.3314011-5-alex.will= iamson@redhat.com Signed-off-by: C=C3=A9dric Le Goater --- include/hw/pci/pcie.h | 2 -- hw/pci-bridge/pcie_pci_bridge.c | 1 - hw/virtio/virtio-pci.c | 8 +++----- 3 files changed, 3 insertions(+), 8 deletions(-) diff --git a/include/hw/pci/pcie.h b/include/hw/pci/pcie.h index b8d59732bc636971b0640e63f036c994a65db932..70a5de09de3952feb4488f8eebe= 78f4dbf8d930d 100644 --- a/include/hw/pci/pcie.h +++ b/include/hw/pci/pcie.h @@ -58,8 +58,6 @@ typedef enum { struct PCIExpressDevice { /* Offset of express capability in config space */ uint8_t exp_cap; - /* Offset of Power Management capability in config space */ - uint8_t pm_cap; =20 /* SLOT */ bool hpev_notified; /* Logical AND of conditions for hot plug event. diff --git a/hw/pci-bridge/pcie_pci_bridge.c b/hw/pci-bridge/pcie_pci_bridg= e.c index 9fa656b43b42624fd597ec6df5d986e71e5f06f0..2429503cfbbf5aaa8977c053226= dc436d3f3dded 100644 --- a/hw/pci-bridge/pcie_pci_bridge.c +++ b/hw/pci-bridge/pcie_pci_bridge.c @@ -56,7 +56,6 @@ static void pcie_pci_bridge_realize(PCIDevice *d, Error *= *errp) if (pos < 0) { goto pm_error; } - d->exp.pm_cap =3D pos; pci_set_word(d->config + pos + PCI_PM_PMC, 0x3); =20 pcie_cap_arifwd_init(d); diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c index afe8b5551c5cc5d33a184f25ac05efa2c126e796..3ca3f849d39157b39a09f4174db= 22487f627c709 100644 --- a/hw/virtio/virtio-pci.c +++ b/hw/virtio/virtio-pci.c @@ -2209,8 +2209,6 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Er= ror **errp) return; } =20 - pci_dev->exp.pm_cap =3D pos; - /* * Indicates that this function complies with revision 1.2 of the * PCI Power Management Interface Specification. @@ -2309,11 +2307,11 @@ static bool virtio_pci_no_soft_reset(PCIDevice *dev) { uint16_t pmcsr; =20 - if (!pci_is_express(dev) || !dev->exp.pm_cap) { + if (!pci_is_express(dev) || !(dev->cap_present & QEMU_PCI_CAP_PM)) { return false; } =20 - pmcsr =3D pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL); + pmcsr =3D pci_get_word(dev->config + dev->pm_cap + PCI_PM_CTRL); =20 /* * When No_Soft_Reset bit is set and the device @@ -2342,7 +2340,7 @@ static void virtio_pci_bus_reset_hold(Object *obj, Re= setType type) =20 if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) { pci_word_test_and_clear_mask( - dev->config + dev->exp.pm_cap + PCI_PM_CTRL, + dev->config + dev->pm_cap + PCI_PM_CTRL, PCI_PM_CTRL_STATE_MASK); } } --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270727; cv=none; d=zohomail.com; s=zohoarc; b=GBssOiY5GWCKUF67kIgbM/y0jchT5aMK/krvMHWsfrdErhwR67VxORO8m28cRVuy7H5CXkCjg9CYm12iRFuuIT3Ebc/cqM5koWy+rZvpFyy35yGCVpti9s3CjvjcAnKOXo6OTFRzpsN9HgIEAcp6+oMk2eN9H1Y1z7fzw+uxL4c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270727; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=sOP2UCzhPbCeiRkfJuaSQp64INrHzLLVntxr+JH2q3k=; b=BDPRQz+mNrGVPhhj9s0hP0/Je6/CxowcDpfJJyrPDZsGgyIK3qHacLIyZ3xY6WBdmSuvNkpfVZsyrXYNoxIa8Pi36GtANjCPWpsth7+mu/Tesykso0C6vVnHxrDjdcKnS+MgmqWyO2Jia7bUg2mg2HCuwOmoyfLNKcRMczL0IF0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270727615580.4107988795382; Thu, 6 Mar 2025 06:18:47 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0B-0006tZ-3v; Thu, 06 Mar 2025 09:15:07 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC08-0006qw-Tn for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC07-0001aZ-4q for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:04 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-27-farwqlZaPzGiEPI_mBFUkg-1; Thu, 06 Mar 2025 09:14:54 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3EA831828A9B; Thu, 6 Mar 2025 14:14:53 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DD32718001D3; Thu, 6 Mar 2025 14:14:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sOP2UCzhPbCeiRkfJuaSQp64INrHzLLVntxr+JH2q3k=; b=WqdBvjNv1GyEnN8LXffLBCGTXqa9jvicCaYpjroA5V/0+qO8kav7cGSbkPsf31+zUr6PlB dUBpGEpvqLr+tALYyMDw1K4aTd0/Ha1QvBdxTHWyhUsqS3qiZoNo6XRcy3FWyU2CEIP03x kZjhiQ13l39DWrRXd5DbbEBt18cgTGU= X-MC-Unique: farwqlZaPzGiEPI_mBFUkg-1 X-Mimecast-MFC-AGG-ID: farwqlZaPzGiEPI_mBFUkg_1741270493 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Zhenzhong Duan , Eric Auger , "Michael S. Tsirkin" Subject: [PULL 07/42] hw/vfio/pci: Re-order pre-reset Date: Thu, 6 Mar 2025 15:13:43 +0100 Message-ID: <20250306141419.2015340-8-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270728957019100 From: Alex Williamson We want the device in the D0 power state going into reset, but the config write can enable the BARs in the address space, which are then removed from the address space once we clear the memory enable bit in the command register. Re-order to clear the command bit first, so the power state change doesn't enable the BARs. Cc: C=C3=A9dric Le Goater Reviewed-by: Zhenzhong Duan Reviewed-by: Eric Auger Signed-off-by: Alex Williamson Reviewed-by: Michael S. Tsirkin Link: https://lore.kernel.org/qemu-devel/20250225215237.3314011-6-alex.will= iamson@redhat.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/pci.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a8db19d8d2f9707693500493eb237c7362d35c06..c1cee280ae4bbfc5a479aa5d1f2= 290065667a677 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2410,6 +2410,15 @@ void vfio_pci_pre_reset(VFIOPCIDevice *vdev) =20 vfio_disable_interrupts(vdev); =20 + /* + * Stop any ongoing DMA by disconnecting I/O, MMIO, and bus master. + * Also put INTx Disable in known state. + */ + cmd =3D vfio_pci_read_config(pdev, PCI_COMMAND, 2); + cmd &=3D ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER | + PCI_COMMAND_INTX_DISABLE); + vfio_pci_write_config(pdev, PCI_COMMAND, cmd, 2); + /* Make sure the device is in D0 */ if (pdev->pm_cap) { uint16_t pmcsr; @@ -2429,15 +2438,6 @@ void vfio_pci_pre_reset(VFIOPCIDevice *vdev) } } } - - /* - * Stop any ongoing DMA by disconnecting I/O, MMIO, and bus master. - * Also put INTx Disable in known state. - */ - cmd =3D vfio_pci_read_config(pdev, PCI_COMMAND, 2); - cmd &=3D ~(PCI_COMMAND_IO | PCI_COMMAND_MEMORY | PCI_COMMAND_MASTER | - PCI_COMMAND_INTX_DISABLE); - vfio_pci_write_config(pdev, PCI_COMMAND, cmd, 2); } =20 void vfio_pci_post_reset(VFIOPCIDevice *vdev) --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270526; cv=none; d=zohomail.com; s=zohoarc; b=kcy2T7Ye4IUqxGM8BBzl3Xe1Ja+qmwOWAdkGDQneFH3WQ/vdiFy3vFMaK9mdEguFh+K/LjIL4TKAoan+h7GF1c3neHrwp6KNCr8NP1c7sThLRbDwcAk91mSP7SEEDUmEeOLuQUQuCFk5DqYIqnOmxdrg875NpVO7z6/zsOJxrxM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270526; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=GbTP8IzGP3RAVbnH0sdHeVXyTt1UNb7Ycl82RIrpHdQ=; b=LPzCLbVOHQxtoIlPi1Lj0DkaGIUVX16mW9SE2jszYSk4kpHDM3otAvsZ7lQdCGyt5gl7sHV83HK60M8bFmqlzYITGHYbO8TU2TfDrdGRg6FZlHBLg7zwQkJgm0qkLs/iYHMmQD6tViBHOrLe7ctNhIpe9Pkvapuf+IzUQYQLkLg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270526937178.8099426937332; Thu, 6 Mar 2025 06:15:26 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC07-0006px-ND; Thu, 06 Mar 2025 09:15:03 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC05-0006pG-OS for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:01 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC04-0001Zt-3p for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:01 -0500 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-582-rflt-SbcPVaR574y1f6ylA-1; Thu, 06 Mar 2025 09:14:56 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6424B1955DDF; Thu, 6 Mar 2025 14:14:55 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id B903B18009BC; Thu, 6 Mar 2025 14:14:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GbTP8IzGP3RAVbnH0sdHeVXyTt1UNb7Ycl82RIrpHdQ=; b=PiR0bOEvaEiccBSSUAqGOQk0sM+f+Tr+bp7M09xX+lysIZGI/vrk7egEj7Tp4LJW1DL4rJ /N3YKB0Ruz6ulF+tD1I6SspKbQyymzgD3cCK/Y+tFOlsJWciT/x2kVeOD+FuKqUqzCF0rU MKpi5QD/UEerp23msivh/oS0pCmtMA0= X-MC-Unique: rflt-SbcPVaR574y1f6ylA-1 X-Mimecast-MFC-AGG-ID: rflt-SbcPVaR574y1f6ylA_1741270495 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Tomita Moeko , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 08/42] MAINTAINERS: Add myself as vfio-igd maintainer Date: Thu, 6 Mar 2025 15:13:44 +0100 Message-ID: <20250306141419.2015340-9-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270528280019000 From: Tomita Moeko As suggested by C=C3=A9dric, I'm glad to be a maintainer of vfio-igd. Signed-off-by: Tomita Moeko Reviewed-by: Alex Williamson Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/20250227162741.9860-1-tomitamoeko@= gmail.com Signed-off-by: C=C3=A9dric Le Goater --- MAINTAINERS | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/MAINTAINERS b/MAINTAINERS index 692628cd7801f2f5b7d18ca451bed3439b01e217..1a920e7dc4cfc637b5f976ae91d= cf40eb7f172e3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2186,10 +2186,17 @@ M: C=C3=A9dric Le Goater S: Supported F: hw/vfio/* F: include/hw/vfio/ -F: docs/igd-assign.txt F: docs/devel/migration/vfio.rst F: qapi/vfio.json =20 +vfio-igd +M: Alex Williamson +M: C=C3=A9dric Le Goater +M: Tomita Moeko +S: Supported +F: hw/vfio/igd.c +F: docs/igd-assign.txt + vfio-ccw M: Eric Farman M: Matthew Rosato --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270866; cv=none; d=zohomail.com; s=zohoarc; b=f5dTyNHkKSwwENAZd96mLrvUKwbwFmXG79EvkxSOYFkCS306Ydm64pb9WzQwBVUHVzl5Lr1sc7tMa3o/Y1n8s2tLdZLY2PcZUa4SXtvF2BWf48OYvnpTPqgLwHmzHW3AQhVFXIrfk5a6KYu2NZ25XAE1WowB+zRLTN2KBbQi5xU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270866; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=wG2/ud91oUoTPzCY10U6/QHq4DEf7p6K3TUNPURNbp4=; b=CKhr0cXv0yP4UORsK8HhGM4fFWVnrqEFQaxuJR4/00zFJvaBXT6V+bimZKVCjCTriCBnkHnOL1vOCIAZWsOWQ3w8lTNcJ1bijY4vBvbA82QXDVsFQlmIrh9hpfeX67mJPx8hFj3agz0/mL1aXs7uhJACBqLZIbxDNztla0+lQaY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270866878166.967643235492; Thu, 6 Mar 2025 06:21:06 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC08-0006qM-GP; Thu, 06 Mar 2025 09:15:04 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC06-0006pP-GC for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:02 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC04-0001Zv-45 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:02 -0500 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-690-HmRxqHLGPpCkJy1N-ttJCQ-1; Thu, 06 Mar 2025 09:14:58 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 838391955D52 for ; Thu, 6 Mar 2025 14:14:57 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DFD1118009BC; Thu, 6 Mar 2025 14:14:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270499; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wG2/ud91oUoTPzCY10U6/QHq4DEf7p6K3TUNPURNbp4=; b=BtX+JrqEGBt+Nk7hyRMpt+SdjN39iOhrm5LaMDAQ7Ush3dfldXUCE40JTULpLP2XbtVPJO GFYA2Rq+mCcc6Bkxefl+/y1dfPr8ksMlES7nw/vtdFV821Px0ydHtV4zDvfT31WpuATV17 EnJ9NG2JYNxhEDSsDlpcL2x1R0Pz7tk= X-MC-Unique: HmRxqHLGPpCkJy1N-ttJCQ-1 X-Mimecast-MFC-AGG-ID: HmRxqHLGPpCkJy1N-ttJCQ_1741270497 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Eric Auger , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 09/42] vfio-platform: Deprecate all forms of vfio-platform devices Date: Thu, 6 Mar 2025 15:13:45 +0100 Message-ID: <20250306141419.2015340-10-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270868008019000 From: Eric Auger As an outcome of KVM forum 2024 "vfio-platform: live and let die?" talk, let's deprecate vfio-platform devices. Signed-off-by: Eric Auger Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/20250305124225.952791-1-eric.auger= @redhat.com [ clg: Fixed spelling in vfio-amd-xgbe section ] Signed-off-by: C=C3=A9dric Le Goater --- docs/about/deprecated.rst | 25 +++++++++++++++++++++++++ hw/vfio/amd-xgbe.c | 2 ++ hw/vfio/calxeda-xgmac.c | 2 ++ hw/vfio/platform.c | 1 + 4 files changed, 30 insertions(+) diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst index abadf8de2708681fb281b07e5ffc671f0c94cfe1..589951b1368327d74e4478feab9= 03d22c13552b7 100644 --- a/docs/about/deprecated.rst +++ b/docs/about/deprecated.rst @@ -434,6 +434,31 @@ Stream ``reconnect`` (since 9.2) The ``reconnect`` option only allows specifiying second granularity timeou= ts, which is not enough for all types of use cases, use ``reconnect-ms`` inste= ad. =20 +VFIO device options +''''''''''''''''''' + +``-device vfio-calxeda-xgmac`` (since 10.0) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The vfio-calxeda-xgmac device allows to assign a host Calxeda Highbank +10Gb XGMAC Ethernet controller device ("calxeda,hb-xgmac" compatibility +string) to a guest. Calxeda HW has been ewasted now and there is no point +keeping that device. + +``-device vfio-amd-xgbe`` (since 10.0) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The vfio-amd-xgbe device allows to assign a host AMD 10GbE controller +to a guest ("amd,xgbe-seattle-v1a" compatibility string). AMD "Seattle" +is not supported anymore and there is no point keeping that device. + +``-device vfio-platform`` (since 10.0) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The vfio-platform device allows to assign a host platform device +to a guest in a generic manner. Integrating a new device into +the vfio-platform infrastructure requires some adaptation at +both kernel and qemu level. No such attempt has been done for years +and the conclusion is that vfio-platform has not got any traction. +PCIe passthrough shall be the mainline solution. + CPU device properties ''''''''''''''''''''' =20 diff --git a/hw/vfio/amd-xgbe.c b/hw/vfio/amd-xgbe.c index aaa96903db0a4bb881078a3f75518ac8469a99fa..5927503b5c452c4065a82786a4b= da714749f0705 100644 --- a/hw/vfio/amd-xgbe.c +++ b/hw/vfio/amd-xgbe.c @@ -15,12 +15,14 @@ #include "hw/vfio/vfio-amd-xgbe.h" #include "migration/vmstate.h" #include "qemu/module.h" +#include "qemu/error-report.h" =20 static void amd_xgbe_realize(DeviceState *dev, Error **errp) { VFIOPlatformDevice *vdev =3D VFIO_PLATFORM_DEVICE(dev); VFIOAmdXgbeDeviceClass *k =3D VFIO_AMD_XGBE_DEVICE_GET_CLASS(dev); =20 + warn_report("-device vfio-amd-xgbe is deprecated"); vdev->compat =3D g_strdup("amd,xgbe-seattle-v1a"); vdev->num_compat =3D 1; =20 diff --git a/hw/vfio/calxeda-xgmac.c b/hw/vfio/calxeda-xgmac.c index b016d42b496190ae585df1e0d90843c1f94e4497..a5ef262def9500feba5c799081e= 4196baba2068e 100644 --- a/hw/vfio/calxeda-xgmac.c +++ b/hw/vfio/calxeda-xgmac.c @@ -15,12 +15,14 @@ #include "hw/vfio/vfio-calxeda-xgmac.h" #include "migration/vmstate.h" #include "qemu/module.h" +#include "qemu/error-report.h" =20 static void calxeda_xgmac_realize(DeviceState *dev, Error **errp) { VFIOPlatformDevice *vdev =3D VFIO_PLATFORM_DEVICE(dev); VFIOCalxedaXgmacDeviceClass *k =3D VFIO_CALXEDA_XGMAC_DEVICE_GET_CLASS= (dev); =20 + warn_report("-device vfio-calxeda-xgmac is deprecated"); vdev->compat =3D g_strdup("calxeda,hb-xgmac"); vdev->num_compat =3D 1; =20 diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index d9faaa73959ad36aa3a835b87964ab940928bd9f..67bc57409c1f5a6978690c3dc07= d249ea0248aa0 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -575,6 +575,7 @@ static void vfio_platform_realize(DeviceState *dev, Err= or **errp) VFIODevice *vbasedev =3D &vdev->vbasedev; int i; =20 + warn_report("-device vfio-platform is deprecated"); qemu_mutex_init(&vdev->intp_mutex); =20 trace_vfio_platform_realize(vbasedev->sysfsdev ? --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270565; cv=none; d=zohomail.com; s=zohoarc; b=YbQpWCWZpd+jMmm1J02L3SwNPkH7mP57xLu0e8Rsc2KW7hKlB1vOdJrtxFWWVIMs6U4+NbLqF5zqComWoAQphC+Tdj4tY6pPZ35izC9L1tsDQNkklwYfduwsLNl+ecjAYluRQebokkF+/KRerm842QTPmawZkU6Vk+UHx5Tpr5s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270565; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ZRAn4E0ADclT5NKArQvMv+JwT4fuP9snV/n8bxajaCQ=; b=AENzYXA7H6XAf5nsyjAbnuPKTpJF854PUrN8SJ5oJUF2VXd5kgpmP/oWWiIdanWK7exRRsvZ02OnevIhlroGrsaqBH5s4hQdiE465RcKREXnyyI7Dr9V0/v9LVuVeKUEDzAlb/u5o/j7pLcBZZ2epq7VLBhRAMQl076jBESrD1A= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270565776459.65698908780837; Thu, 6 Mar 2025 06:16:05 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0F-00071k-44; Thu, 06 Mar 2025 09:15:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0B-0006tl-N7 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:08 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC09-0001lg-W2 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:07 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-198-TN0raHs7NtqJiiXAoXRzbw-1; Thu, 06 Mar 2025 09:15:02 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B398318001E3; Thu, 6 Mar 2025 14:15:00 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 09B3618001D3; Thu, 6 Mar 2025 14:14:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270505; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZRAn4E0ADclT5NKArQvMv+JwT4fuP9snV/n8bxajaCQ=; b=NwfRbTaB0Qkl9TnKQ5LzKbk+gYqQjuIGD2q7VSEvF23p/SupUVtqyno636UC53nWGv9wiH iDfalGejbDTALEOGteRRdPH5cE65P0Ozkt37L4ijYhGwcKhClF+9YPgZFZKTyGd6qhTx75 MvmmWRSZ1cY2CeYOwOTa4D37fM/xdSY= X-MC-Unique: TN0raHs7NtqJiiXAoXRzbw-1 X-Mimecast-MFC-AGG-ID: TN0raHs7NtqJiiXAoXRzbw_1741270501 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 10/42] migration: Clarify that {load, save}_cleanup handlers can run without setup Date: Thu, 6 Mar 2025 15:13:46 +0100 Message-ID: <20250306141419.2015340-11-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270568468019100 From: "Maciej S. Szmigiero" It's possible for {load,save}_cleanup SaveVMHandlers to get called without the corresponding {load,save}_setup handler being called first. One such example is if {load,save}_setup handler of a proceeding device returns error. In this case the migration core cleanup code will call all corresponding cleanup handlers, even for these devices which haven't had its setup handler called. Since this behavior can generate some surprises let's clearly document it in these SaveVMHandlers description. Reviewed-by: Fabiano Rosas Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/991636623fb780350f493b5f045cb17e13= ce4c0f.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/register.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/include/migration/register.h b/include/migration/register.h index f60e797894e5faacdf55d2d6de175074ac58944f..0b02927383208d1457f21da7ad4= 900ba53e93638 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -69,7 +69,9 @@ typedef struct SaveVMHandlers { /** * @save_cleanup * - * Uninitializes the data structures on the source + * Uninitializes the data structures on the source. + * Note that this handler can be called even if save_setup + * wasn't called earlier. * * @opaque: data pointer passed to register_savevm_live() */ @@ -244,6 +246,8 @@ typedef struct SaveVMHandlers { * @load_cleanup * * Uninitializes the data structures on the destination. + * Note that this handler can be called even if load_setup + * wasn't called earlier. * * @opaque: data pointer passed to register_savevm_live() * --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270556; cv=none; d=zohomail.com; s=zohoarc; b=YQl2wZXY67MdGY4B4GiZI03lRMSWvdcYY5AR5NPdN0DrvkFBMTxLxtEkfguD42CxEuZqg4m0FCXOj1cWtFbtxp9Ur+EnK00Rjv8QBk6gPk+g58n1uJwhSjirGfbwQKqBPxc08qT7S9F5jJNNsZ/euJEYM0520nk23zYKY/GRIWM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270556; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PaUCwVBWNisuA+DLJLKKBbuf+EdU9eDWEbHvI3b6clg=; b=GqfWPlptwi8U+TEoP76xxs6AZU6u93GnE+/RvYB3OjTqWUT/PPBVW2M4okQIBz8Xh4mpz76QcafEJ+YXuQ6oF21WbeRIlY4Pequz/kOIlRjQNOMkwQH6+7LbSVMdoM/vtKuvYUlH/t0X+CI2yZw5r4KloAiI+H9TbErj+anzJlE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270556916707.8437658415523; Thu, 6 Mar 2025 06:15:56 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0v-0007qy-Ic; Thu, 06 Mar 2025 09:15:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0F-0007AT-Nq for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:22 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0D-0001mG-1K for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:11 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-117-N7jfAxwKPCG_AgiubYe-6A-1; Thu, 06 Mar 2025 09:15:05 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E1948180025B; Thu, 6 Mar 2025 14:15:03 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 3CE5818009BC; Thu, 6 Mar 2025 14:15:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PaUCwVBWNisuA+DLJLKKBbuf+EdU9eDWEbHvI3b6clg=; b=PEH6eB3HFVdLaJG/fK3Bta9gK7ReZlQb68OGm10jcjZo7+o3gz7PLdPGAq5PGqB+lE2o8U TAykSit+volRP9OJz6XX6Ilk76Qgddz8LO4b5KMFojKsfdpIb6PzHwvmVJ6e2t2NmRu7TJ WbnK+WmdrC1b8bAL/peLyZBrdrSAVZA= X-MC-Unique: N7jfAxwKPCG_AgiubYe-6A-1 X-Mimecast-MFC-AGG-ID: N7jfAxwKPCG_AgiubYe-6A_1741270504 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Peter Xu Subject: [PULL 11/42] thread-pool: Remove thread_pool_submit() function Date: Thu, 6 Mar 2025 15:13:47 +0100 Message-ID: <20250306141419.2015340-12-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270560608019100 From: "Maciej S. Szmigiero" This function name conflicts with one used by a future generic thread pool function and it was only used by one test anyway. Update the trace event name in thread_pool_submit_aio() accordingly. Acked-by: Fabiano Rosas Reviewed-by: C=C3=A9dric Le Goater Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/6830f07777f939edaf0a2d301c39adcaaf= 3817f0.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/block/thread-pool.h | 3 +-- tests/unit/test-thread-pool.c | 6 +++--- util/thread-pool.c | 7 +------ util/trace-events | 2 +- 4 files changed, 6 insertions(+), 12 deletions(-) diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h index 948ff5f30c31de452361578dec9fa252817b4a69..4f6694026123e2b6b503eab3ee2= 3cf51db4bdef1 100644 --- a/include/block/thread-pool.h +++ b/include/block/thread-pool.h @@ -30,13 +30,12 @@ ThreadPool *thread_pool_new(struct AioContext *ctx); void thread_pool_free(ThreadPool *pool); =20 /* - * thread_pool_submit* API: submit I/O requests in the thread's + * thread_pool_submit_{aio,co} API: submit I/O requests in the thread's * current AioContext. */ BlockAIOCB *thread_pool_submit_aio(ThreadPoolFunc *func, void *arg, BlockCompletionFunc *cb, void *opaque); int coroutine_fn thread_pool_submit_co(ThreadPoolFunc *func, void *arg); -void thread_pool_submit(ThreadPoolFunc *func, void *arg); =20 void thread_pool_update_params(ThreadPool *pool, struct AioContext *ctx); =20 diff --git a/tests/unit/test-thread-pool.c b/tests/unit/test-thread-pool.c index 1483e53473db56d43508fc3c4363dbdfbd64cf1f..33407b595d3565614c631a58759= bc55be1bcfac3 100644 --- a/tests/unit/test-thread-pool.c +++ b/tests/unit/test-thread-pool.c @@ -43,10 +43,10 @@ static void done_cb(void *opaque, int ret) active--; } =20 -static void test_submit(void) +static void test_submit_no_complete(void) { WorkerTestData data =3D { .n =3D 0 }; - thread_pool_submit(worker_cb, &data); + thread_pool_submit_aio(worker_cb, &data, NULL, NULL); while (data.n =3D=3D 0) { aio_poll(ctx, true); } @@ -236,7 +236,7 @@ int main(int argc, char **argv) ctx =3D qemu_get_current_aio_context(); =20 g_test_init(&argc, &argv, NULL); - g_test_add_func("/thread-pool/submit", test_submit); + g_test_add_func("/thread-pool/submit-no-complete", test_submit_no_comp= lete); g_test_add_func("/thread-pool/submit-aio", test_submit_aio); g_test_add_func("/thread-pool/submit-co", test_submit_co); g_test_add_func("/thread-pool/submit-many", test_submit_many); diff --git a/util/thread-pool.c b/util/thread-pool.c index 27eb777e855b62818a64cc5ad4f4937903b64ed1..2f751d55b33ffcbde584c99030c= 5b46b583b87d3 100644 --- a/util/thread-pool.c +++ b/util/thread-pool.c @@ -256,7 +256,7 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPoolFunc *func= , void *arg, =20 QLIST_INSERT_HEAD(&pool->head, req, all); =20 - trace_thread_pool_submit(pool, req, arg); + trace_thread_pool_submit_aio(pool, req, arg); =20 qemu_mutex_lock(&pool->lock); if (pool->idle_threads =3D=3D 0 && pool->cur_threads < pool->max_threa= ds) { @@ -290,11 +290,6 @@ int coroutine_fn thread_pool_submit_co(ThreadPoolFunc = *func, void *arg) return tpc.ret; } =20 -void thread_pool_submit(ThreadPoolFunc *func, void *arg) -{ - thread_pool_submit_aio(func, arg, NULL, NULL); -} - void thread_pool_update_params(ThreadPool *pool, AioContext *ctx) { qemu_mutex_lock(&pool->lock); diff --git a/util/trace-events b/util/trace-events index 49a4962e18869d9793349283fce5c5d4a33efe3e..5be12d7fab8943458097d7d03bf= 1fe41e330aba6 100644 --- a/util/trace-events +++ b/util/trace-events @@ -14,7 +14,7 @@ aio_co_schedule_bh_cb(void *ctx, void *co) "ctx %p co %p" reentrant_aio(void *ctx, const char *name) "ctx %p name %s" =20 # thread-pool.c -thread_pool_submit(void *pool, void *req, void *opaque) "pool %p req %p op= aque %p" +thread_pool_submit_aio(void *pool, void *req, void *opaque) "pool %p req %= p opaque %p" thread_pool_complete(void *pool, void *req, void *opaque, int ret) "pool %= p req %p opaque %p ret %d" thread_pool_cancel(void *req, void *opaque) "req %p opaque %p" =20 --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270765; cv=none; d=zohomail.com; s=zohoarc; b=ikgkP7M1+viJzze3VXwmx7Iqf2dNALZwAUEB7GCtkCSD737hu/XzCyIEsHYYSF73mmwJfHtZHxLR2YfHx6+5wFVrP2SRRApFrjTgmWCol5rxf0bHPv68aUAqqJKJWecDdtaR9cZC/pK3g9dJDgva3cJq+OGqUedsp6nGPThiHig= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270765; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=WS7YUvhiU0H9JPDNMiJqsDDCj4ZF6iHMJo/TyYhiLQU=; b=eCUXZMWmq2LmQ/ja1S/9Dektabm5kNjI4iaI1Tx3+6/W6iCeWgzjLBNl5lUvEhdOfItZzNbowMvLyP8J1e9zPaEz6dM31yLZcZZdmYno7vyqN3R7dJLIoPkqwycHAYk8tR7364xUblkheKAB9RblfZgmJ86Fe755htQK3WV/66w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270765375286.73729994350845; Thu, 6 Mar 2025 06:19:25 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0m-0007ZQ-6g; Thu, 06 Mar 2025 09:15:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0Q-0007Hd-9Z for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:33 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0J-0001my-F9 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:17 -0500 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-92-sfGV4ux4NoiUzLEPqYF6Rg-1; Thu, 06 Mar 2025 09:15:08 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2E7E519560B2; Thu, 6 Mar 2025 14:15:07 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6A5D318001D3; Thu, 6 Mar 2025 14:15:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WS7YUvhiU0H9JPDNMiJqsDDCj4ZF6iHMJo/TyYhiLQU=; b=bxrVKz0F4gfpZCX1UczT8Z3Pox59qfBNYeciHGiCehlseax2MYOBOoyM570rHExTGygKiw TnHN9lOeZ7QQVxF5br34arigjfD9Duq74YeEb+IolsKTRQBlrxJZF9cP1xu6DXmmwU0JhZ t9iODLFrqNUuxv8dWiG6fT7LjP8EzZw= X-MC-Unique: sfGV4ux4NoiUzLEPqYF6Rg-1 X-Mimecast-MFC-AGG-ID: sfGV4ux4NoiUzLEPqYF6Rg_1741270507 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= , Peter Xu Subject: [PULL 12/42] thread-pool: Rename AIO pool functions to *_aio() and data types to *Aio Date: Thu, 6 Mar 2025 15:13:48 +0100 Message-ID: <20250306141419.2015340-13-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270769418019100 From: "Maciej S. Szmigiero" These names conflict with ones used by future generic thread pool equivalents. Generic names should belong to the generic pool type, not specific (AIO) type. Acked-by: Fabiano Rosas Reviewed-by: C=C3=A9dric Le Goater Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/70f9e0fb4b01042258a1a57996c64d1977= 9dc7f0.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/block/aio.h | 8 ++--- include/block/thread-pool.h | 8 ++--- util/async.c | 6 ++-- util/thread-pool.c | 58 ++++++++++++++++++------------------- util/trace-events | 4 +-- 5 files changed, 42 insertions(+), 42 deletions(-) diff --git a/include/block/aio.h b/include/block/aio.h index 43883a8a33a8efcab92da58a8071ac082365ed25..b2ab3514de23ac8b744b23e9cb9= d3a237a050a35 100644 --- a/include/block/aio.h +++ b/include/block/aio.h @@ -54,7 +54,7 @@ typedef void QEMUBHFunc(void *opaque); typedef bool AioPollFn(void *opaque); typedef void IOHandler(void *opaque); =20 -struct ThreadPool; +struct ThreadPoolAio; struct LinuxAioState; typedef struct LuringState LuringState; =20 @@ -207,7 +207,7 @@ struct AioContext { /* Thread pool for performing work and receiving completion callbacks. * Has its own locking. */ - struct ThreadPool *thread_pool; + struct ThreadPoolAio *thread_pool; =20 #ifdef CONFIG_LINUX_AIO struct LinuxAioState *linux_aio; @@ -500,8 +500,8 @@ void aio_set_event_notifier_poll(AioContext *ctx, */ GSource *aio_get_g_source(AioContext *ctx); =20 -/* Return the ThreadPool bound to this AioContext */ -struct ThreadPool *aio_get_thread_pool(AioContext *ctx); +/* Return the ThreadPoolAio bound to this AioContext */ +struct ThreadPoolAio *aio_get_thread_pool(AioContext *ctx); =20 /* Setup the LinuxAioState bound to this AioContext */ struct LinuxAioState *aio_setup_linux_aio(AioContext *ctx, Error **errp); diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h index 4f6694026123e2b6b503eab3ee23cf51db4bdef1..6f27eb085b451ecc712aeff3859= 44541a729c83d 100644 --- a/include/block/thread-pool.h +++ b/include/block/thread-pool.h @@ -24,10 +24,10 @@ =20 typedef int ThreadPoolFunc(void *opaque); =20 -typedef struct ThreadPool ThreadPool; +typedef struct ThreadPoolAio ThreadPoolAio; =20 -ThreadPool *thread_pool_new(struct AioContext *ctx); -void thread_pool_free(ThreadPool *pool); +ThreadPoolAio *thread_pool_new_aio(struct AioContext *ctx); +void thread_pool_free_aio(ThreadPoolAio *pool); =20 /* * thread_pool_submit_{aio,co} API: submit I/O requests in the thread's @@ -36,7 +36,7 @@ void thread_pool_free(ThreadPool *pool); BlockAIOCB *thread_pool_submit_aio(ThreadPoolFunc *func, void *arg, BlockCompletionFunc *cb, void *opaque); int coroutine_fn thread_pool_submit_co(ThreadPoolFunc *func, void *arg); +void thread_pool_update_params(ThreadPoolAio *pool, struct AioContext *ctx= ); =20 -void thread_pool_update_params(ThreadPool *pool, struct AioContext *ctx); =20 #endif diff --git a/util/async.c b/util/async.c index 0fe29436090dc383f1a867ce109c90c1d763d6e7..47e3d35a263fa98777da5b341f0= d7cf287b92c45 100644 --- a/util/async.c +++ b/util/async.c @@ -369,7 +369,7 @@ aio_ctx_finalize(GSource *source) QEMUBH *bh; unsigned flags; =20 - thread_pool_free(ctx->thread_pool); + thread_pool_free_aio(ctx->thread_pool); =20 #ifdef CONFIG_LINUX_AIO if (ctx->linux_aio) { @@ -435,10 +435,10 @@ GSource *aio_get_g_source(AioContext *ctx) return &ctx->source; } =20 -ThreadPool *aio_get_thread_pool(AioContext *ctx) +ThreadPoolAio *aio_get_thread_pool(AioContext *ctx) { if (!ctx->thread_pool) { - ctx->thread_pool =3D thread_pool_new(ctx); + ctx->thread_pool =3D thread_pool_new_aio(ctx); } return ctx->thread_pool; } diff --git a/util/thread-pool.c b/util/thread-pool.c index 2f751d55b33ffcbde584c99030c5b46b583b87d3..908194dc070fb90390c493f0c7f= b10062dcb5c15 100644 --- a/util/thread-pool.c +++ b/util/thread-pool.c @@ -23,9 +23,9 @@ #include "block/thread-pool.h" #include "qemu/main-loop.h" =20 -static void do_spawn_thread(ThreadPool *pool); +static void do_spawn_thread(ThreadPoolAio *pool); =20 -typedef struct ThreadPoolElement ThreadPoolElement; +typedef struct ThreadPoolElementAio ThreadPoolElementAio; =20 enum ThreadState { THREAD_QUEUED, @@ -33,9 +33,9 @@ enum ThreadState { THREAD_DONE, }; =20 -struct ThreadPoolElement { +struct ThreadPoolElementAio { BlockAIOCB common; - ThreadPool *pool; + ThreadPoolAio *pool; ThreadPoolFunc *func; void *arg; =20 @@ -47,13 +47,13 @@ struct ThreadPoolElement { int ret; =20 /* Access to this list is protected by lock. */ - QTAILQ_ENTRY(ThreadPoolElement) reqs; + QTAILQ_ENTRY(ThreadPoolElementAio) reqs; =20 /* This list is only written by the thread pool's mother thread. */ - QLIST_ENTRY(ThreadPoolElement) all; + QLIST_ENTRY(ThreadPoolElementAio) all; }; =20 -struct ThreadPool { +struct ThreadPoolAio { AioContext *ctx; QEMUBH *completion_bh; QemuMutex lock; @@ -62,10 +62,10 @@ struct ThreadPool { QEMUBH *new_thread_bh; =20 /* The following variables are only accessed from one AioContext. */ - QLIST_HEAD(, ThreadPoolElement) head; + QLIST_HEAD(, ThreadPoolElementAio) head; =20 /* The following variables are protected by lock. */ - QTAILQ_HEAD(, ThreadPoolElement) request_list; + QTAILQ_HEAD(, ThreadPoolElementAio) request_list; int cur_threads; int idle_threads; int new_threads; /* backlog of threads we need to create */ @@ -76,14 +76,14 @@ struct ThreadPool { =20 static void *worker_thread(void *opaque) { - ThreadPool *pool =3D opaque; + ThreadPoolAio *pool =3D opaque; =20 qemu_mutex_lock(&pool->lock); pool->pending_threads--; do_spawn_thread(pool); =20 while (pool->cur_threads <=3D pool->max_threads) { - ThreadPoolElement *req; + ThreadPoolElementAio *req; int ret; =20 if (QTAILQ_EMPTY(&pool->request_list)) { @@ -131,7 +131,7 @@ static void *worker_thread(void *opaque) return NULL; } =20 -static void do_spawn_thread(ThreadPool *pool) +static void do_spawn_thread(ThreadPoolAio *pool) { QemuThread t; =20 @@ -148,14 +148,14 @@ static void do_spawn_thread(ThreadPool *pool) =20 static void spawn_thread_bh_fn(void *opaque) { - ThreadPool *pool =3D opaque; + ThreadPoolAio *pool =3D opaque; =20 qemu_mutex_lock(&pool->lock); do_spawn_thread(pool); qemu_mutex_unlock(&pool->lock); } =20 -static void spawn_thread(ThreadPool *pool) +static void spawn_thread(ThreadPoolAio *pool) { pool->cur_threads++; pool->new_threads++; @@ -173,8 +173,8 @@ static void spawn_thread(ThreadPool *pool) =20 static void thread_pool_completion_bh(void *opaque) { - ThreadPool *pool =3D opaque; - ThreadPoolElement *elem, *next; + ThreadPoolAio *pool =3D opaque; + ThreadPoolElementAio *elem, *next; =20 defer_call_begin(); /* cb() may use defer_call() to coalesce work */ =20 @@ -184,8 +184,8 @@ restart: continue; } =20 - trace_thread_pool_complete(pool, elem, elem->common.opaque, - elem->ret); + trace_thread_pool_complete_aio(pool, elem, elem->common.opaque, + elem->ret); QLIST_REMOVE(elem, all); =20 if (elem->common.cb) { @@ -217,10 +217,10 @@ restart: =20 static void thread_pool_cancel(BlockAIOCB *acb) { - ThreadPoolElement *elem =3D (ThreadPoolElement *)acb; - ThreadPool *pool =3D elem->pool; + ThreadPoolElementAio *elem =3D (ThreadPoolElementAio *)acb; + ThreadPoolAio *pool =3D elem->pool; =20 - trace_thread_pool_cancel(elem, elem->common.opaque); + trace_thread_pool_cancel_aio(elem, elem->common.opaque); =20 QEMU_LOCK_GUARD(&pool->lock); if (elem->state =3D=3D THREAD_QUEUED) { @@ -234,16 +234,16 @@ static void thread_pool_cancel(BlockAIOCB *acb) } =20 static const AIOCBInfo thread_pool_aiocb_info =3D { - .aiocb_size =3D sizeof(ThreadPoolElement), + .aiocb_size =3D sizeof(ThreadPoolElementAio), .cancel_async =3D thread_pool_cancel, }; =20 BlockAIOCB *thread_pool_submit_aio(ThreadPoolFunc *func, void *arg, BlockCompletionFunc *cb, void *opaque) { - ThreadPoolElement *req; + ThreadPoolElementAio *req; AioContext *ctx =3D qemu_get_current_aio_context(); - ThreadPool *pool =3D aio_get_thread_pool(ctx); + ThreadPoolAio *pool =3D aio_get_thread_pool(ctx); =20 /* Assert that the thread submitting work is the same running the pool= */ assert(pool->ctx =3D=3D qemu_get_current_aio_context()); @@ -290,7 +290,7 @@ int coroutine_fn thread_pool_submit_co(ThreadPoolFunc *= func, void *arg) return tpc.ret; } =20 -void thread_pool_update_params(ThreadPool *pool, AioContext *ctx) +void thread_pool_update_params(ThreadPoolAio *pool, AioContext *ctx) { qemu_mutex_lock(&pool->lock); =20 @@ -317,7 +317,7 @@ void thread_pool_update_params(ThreadPool *pool, AioCon= text *ctx) qemu_mutex_unlock(&pool->lock); } =20 -static void thread_pool_init_one(ThreadPool *pool, AioContext *ctx) +static void thread_pool_init_one(ThreadPoolAio *pool, AioContext *ctx) { if (!ctx) { ctx =3D qemu_get_aio_context(); @@ -337,14 +337,14 @@ static void thread_pool_init_one(ThreadPool *pool, Ai= oContext *ctx) thread_pool_update_params(pool, ctx); } =20 -ThreadPool *thread_pool_new(AioContext *ctx) +ThreadPoolAio *thread_pool_new_aio(AioContext *ctx) { - ThreadPool *pool =3D g_new(ThreadPool, 1); + ThreadPoolAio *pool =3D g_new(ThreadPoolAio, 1); thread_pool_init_one(pool, ctx); return pool; } =20 -void thread_pool_free(ThreadPool *pool) +void thread_pool_free_aio(ThreadPoolAio *pool) { if (!pool) { return; diff --git a/util/trace-events b/util/trace-events index 5be12d7fab8943458097d7d03bf1fe41e330aba6..bd8f25fb592097b32d2017e9623= 823635d042691 100644 --- a/util/trace-events +++ b/util/trace-events @@ -15,8 +15,8 @@ reentrant_aio(void *ctx, const char *name) "ctx %p name %= s" =20 # thread-pool.c thread_pool_submit_aio(void *pool, void *req, void *opaque) "pool %p req %= p opaque %p" -thread_pool_complete(void *pool, void *req, void *opaque, int ret) "pool %= p req %p opaque %p ret %d" -thread_pool_cancel(void *req, void *opaque) "req %p opaque %p" +thread_pool_complete_aio(void *pool, void *req, void *opaque, int ret) "po= ol %p req %p opaque %p ret %d" +thread_pool_cancel_aio(void *req, void *opaque) "req %p opaque %p" =20 # buffer.c buffer_resize(const char *buf, size_t olen, size_t len) "%s: old %zd, new = %zd" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270640; cv=none; d=zohomail.com; s=zohoarc; b=BoHBdbwz42FG/vNu8bA57Giinpe+kdD0I98HL9YUD/96V8FFzKCRuWpTLlcOHwfUdiGLECNBNZOaRfhCrkZBUZy9ihzgGCByC33JKycevQ1HLh2zJHU8qMocPEgOxIEyVpwgHXcyiY6WG0WvnHF7tuqM5xdQTxc1e4PWSfp7ZI4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270640; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ORsMgR5tNUkV2umt2LiJWC8AWrhfAOEM9RMcEU3OdRk=; b=G+bUKZRFhE/hX9j6rn6Qi2GSfkurd+m2E26n8jdzHhtHfoOdhCbMS4WNjtpp4ZIUAG584jmrRNVwcXTDbTUgUZaK7I52Ds92p1/TQ3w/wws/OVFaSIJOiq0UgGHEIyU2AqsoXD/uS0f+H+crOqnZ6Pu0NK79CVk2xa6kmOjtlhM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270640218541.2560971028718; Thu, 6 Mar 2025 06:17:20 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0z-000842-I2; Thu, 06 Mar 2025 09:15:57 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0Z-0007MF-26 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0S-0001nK-1r for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:27 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-ZqouXhPaO6CfDQxDWiUCDg-1; Thu, 06 Mar 2025 09:15:10 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D978A180AF5B; Thu, 6 Mar 2025 14:15:09 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id AF79D18001D3; Thu, 6 Mar 2025 14:15:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270519; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ORsMgR5tNUkV2umt2LiJWC8AWrhfAOEM9RMcEU3OdRk=; b=YMj673rgglNVZFQW5cZL6uDp9JZgV/qPIJqdypq+CPVJOm0nVcbZ89tpPml6fjwzD2xrR8 wdIzdBlRPKECvEIFU0pw+T4w9qxAH3lzXU8iz5dTUxXwaAwtq3eaBnQxRaCXOXYuhWlMHG cukj3+9qVFamHvQ6KiqCa4X5hUNejVI= X-MC-Unique: ZqouXhPaO6CfDQxDWiUCDg-1 X-Mimecast-MFC-AGG-ID: ZqouXhPaO6CfDQxDWiUCDg_1741270510 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 13/42] thread-pool: Implement generic (non-AIO) pool support Date: Thu, 6 Mar 2025 15:13:49 +0100 Message-ID: <20250306141419.2015340-14-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270643120019100 From: "Maciej S. Szmigiero" Migration code wants to manage device data sending threads in one place. QEMU has an existing thread pool implementation, however it is limited to queuing AIO operations only and essentially has a 1:1 mapping between the current AioContext and the AIO ThreadPool in use. Implement generic (non-AIO) ThreadPool by essentially wrapping Glib's GThreadPool. This brings a few new operations on a pool: * thread_pool_wait() operation waits until all the submitted work requests have finished. * thread_pool_set_max_threads() explicitly sets the maximum thread count in the pool. * thread_pool_adjust_max_threads_to_work() adjusts the maximum thread count in the pool to equal the number of still waiting in queue or unfinished wor= k. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/b1efaebdbea7cb7068b8fb741487770123= 83e12b.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/block/thread-pool.h | 51 ++++++++++++++++ util/thread-pool.c | 119 ++++++++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h index 6f27eb085b451ecc712aeff385944541a729c83d..dd48cf07e85f886e9efc44916e4= 018141f352233 100644 --- a/include/block/thread-pool.h +++ b/include/block/thread-pool.h @@ -38,5 +38,56 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPoolFunc *func,= void *arg, int coroutine_fn thread_pool_submit_co(ThreadPoolFunc *func, void *arg); void thread_pool_update_params(ThreadPoolAio *pool, struct AioContext *ctx= ); =20 +/* ------------------------------------------- */ +/* Generic thread pool types and methods below */ +typedef struct ThreadPool ThreadPool; + +/* Create a new thread pool. Never returns NULL. */ +ThreadPool *thread_pool_new(void); + +/* + * Free the thread pool. + * Waits for all the previously submitted work to complete before performi= ng + * the actual freeing operation. + */ +void thread_pool_free(ThreadPool *pool); + +/* + * Submit a new work (task) for the pool. + * + * @opaque_destroy is an optional GDestroyNotify for the @opaque argument + * to the work function at @func. + */ +void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, + void *opaque, GDestroyNotify opaque_destroy); + +/* + * Submit a new work (task) for the pool, making sure it starts getting + * processed immediately, launching a new thread for it if necessary. + * + * @opaque_destroy is an optional GDestroyNotify for the @opaque argument + * to the work function at @func. + */ +void thread_pool_submit_immediate(ThreadPool *pool, ThreadPoolFunc *func, + void *opaque, GDestroyNotify opaque_dest= roy); + +/* + * Wait for all previously submitted work to complete before returning. + * + * Can be used as a barrier between two sets of tasks executed on a thread + * pool without destroying it or in a performance sensitive path where the + * caller just wants to wait for all tasks to complete while deferring the + * pool free operation for later, less performance sensitive time. + */ +void thread_pool_wait(ThreadPool *pool); + +/* Set the maximum number of threads in the pool. */ +bool thread_pool_set_max_threads(ThreadPool *pool, int max_threads); + +/* + * Adjust the maximum number of threads in the pool to give each task its + * own thread (exactly one thread per task). + */ +bool thread_pool_adjust_max_threads_to_work(ThreadPool *pool); =20 #endif diff --git a/util/thread-pool.c b/util/thread-pool.c index 908194dc070fb90390c493f0c7fb10062dcb5c15..d2ead6b7285765821f0fa956132= b207b5985b015 100644 --- a/util/thread-pool.c +++ b/util/thread-pool.c @@ -374,3 +374,122 @@ void thread_pool_free_aio(ThreadPoolAio *pool) qemu_mutex_destroy(&pool->lock); g_free(pool); } + +struct ThreadPool { + GThreadPool *t; + size_t cur_work; + QemuMutex cur_work_lock; + QemuCond all_finished_cond; +}; + +typedef struct { + ThreadPoolFunc *func; + void *opaque; + GDestroyNotify opaque_destroy; +} ThreadPoolElement; + +static void thread_pool_func(gpointer data, gpointer user_data) +{ + ThreadPool *pool =3D user_data; + g_autofree ThreadPoolElement *el =3D data; + + el->func(el->opaque); + + if (el->opaque_destroy) { + el->opaque_destroy(el->opaque); + } + + QEMU_LOCK_GUARD(&pool->cur_work_lock); + + assert(pool->cur_work > 0); + pool->cur_work--; + + if (pool->cur_work =3D=3D 0) { + qemu_cond_signal(&pool->all_finished_cond); + } +} + +ThreadPool *thread_pool_new(void) +{ + ThreadPool *pool =3D g_new(ThreadPool, 1); + + pool->cur_work =3D 0; + qemu_mutex_init(&pool->cur_work_lock); + qemu_cond_init(&pool->all_finished_cond); + + pool->t =3D g_thread_pool_new(thread_pool_func, pool, 0, TRUE, NULL); + /* + * g_thread_pool_new() can only return errors if initial thread(s) + * creation fails but we ask for 0 initial threads above. + */ + assert(pool->t); + + return pool; +} + +void thread_pool_free(ThreadPool *pool) +{ + /* + * With _wait =3D TRUE this effectively waits for all + * previously submitted work to complete first. + */ + g_thread_pool_free(pool->t, FALSE, TRUE); + + qemu_cond_destroy(&pool->all_finished_cond); + qemu_mutex_destroy(&pool->cur_work_lock); + + g_free(pool); +} + +void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, + void *opaque, GDestroyNotify opaque_destroy) +{ + ThreadPoolElement *el =3D g_new(ThreadPoolElement, 1); + + el->func =3D func; + el->opaque =3D opaque; + el->opaque_destroy =3D opaque_destroy; + + WITH_QEMU_LOCK_GUARD(&pool->cur_work_lock) { + pool->cur_work++; + } + + /* + * Ignore the return value since this function can only return errors + * if creation of an additional thread fails but even in this case the + * provided work is still getting queued (just for the existing thread= s). + */ + g_thread_pool_push(pool->t, el, NULL); +} + +void thread_pool_submit_immediate(ThreadPool *pool, ThreadPoolFunc *func, + void *opaque, GDestroyNotify opaque_dest= roy) +{ + thread_pool_submit(pool, func, opaque, opaque_destroy); + thread_pool_adjust_max_threads_to_work(pool); +} + +void thread_pool_wait(ThreadPool *pool) +{ + QEMU_LOCK_GUARD(&pool->cur_work_lock); + + while (pool->cur_work > 0) { + qemu_cond_wait(&pool->all_finished_cond, + &pool->cur_work_lock); + } +} + +bool thread_pool_set_max_threads(ThreadPool *pool, + int max_threads) +{ + assert(max_threads > 0); + + return g_thread_pool_set_max_threads(pool->t, max_threads, NULL); +} + +bool thread_pool_adjust_max_threads_to_work(ThreadPool *pool) +{ + QEMU_LOCK_GUARD(&pool->cur_work_lock); + + return thread_pool_set_max_threads(pool, pool->cur_work); +} --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270931; cv=none; d=zohomail.com; s=zohoarc; b=fzqcUwShJtV0b4zOQ3DrmJf+oFNGT68GToERakBK7MlM27UOS3wUB0ulbdi8OAXth4kc136U/8BMTIb8Q2FcnOeRh3epyW1GcijH9qtXy2QUYFvrD3gq2sIEDrsDec673fTVZ9QzdYWf5ASo1FrB/g7RG/IbFrgwFCegvLkVCQc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270931; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=nx4bd6ERtY3K+2G5g7ASy2iRMEjQ7poAzNL7XntscSA=; b=jfHOpVM7ON+0MNf81zmI5axQAZT6GxWUBvO0vzJjciPJ6Y9PuqB67mM2+7Pkrkmytu0hDOxogK63VYx0h7+lgAFLR7dYxsbEtzLJBT/TIRQL+nV7Brf+pEEY8ADE+IjQIjN8+3rW98GIncfy1/fUTx16bB+VuHqLoDn1yu57STI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270931692501.35785734960984; Thu, 6 Mar 2025 06:22:11 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0y-0007xf-9a; Thu, 06 Mar 2025 09:15:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0d-0007NF-Ll for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:36 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0a-0001on-JQ for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:34 -0500 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-542-3Gt8Rl0cNVirCFxRBrHizg-1; Thu, 06 Mar 2025 09:15:13 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8EC2F19560BB; Thu, 6 Mar 2025 14:15:12 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 64CB318009BC; Thu, 6 Mar 2025 14:15:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nx4bd6ERtY3K+2G5g7ASy2iRMEjQ7poAzNL7XntscSA=; b=ANt60jDmJJi4dLps7AgkYviRx1lTaDcAuLpeSwXgdHJpM325rE1gYjSDuq9BHLaugXCbjS 6FHBl4RP2Wt2zQUsX9K+ex0Srlo5JCv4T6z4CFdGrPhPgZX4un9H/gf6hl3SIh1iMpro8V ImlpXQchFnjUd3fGBVOSpJMSITaMQNc= X-MC-Unique: 3Gt8Rl0cNVirCFxRBrHizg-1 X-Mimecast-MFC-AGG-ID: 3Gt8Rl0cNVirCFxRBrHizg_1741270512 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Zhang Chen , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 14/42] migration: Add MIG_CMD_SWITCHOVER_START and its load handler Date: Thu, 6 Mar 2025 15:13:50 +0100 Message-ID: <20250306141419.2015340-15-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270933718019100 From: "Maciej S. Szmigiero" This QEMU_VM_COMMAND sub-command and its switchover_start SaveVMHandler is used to mark the switchover point in main migration stream. It can be used to inform the destination that all pre-switchover main migration stream data has been sent/received so it can start to process post-switchover data that it might have received via other migration channels like the multifd ones. Add also the relevant MigrationState bit stream compatibility property and its hw_compat entry. Reviewed-by: Fabiano Rosas Reviewed-by: Zhang Chen # for the COLO part Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/311be6da85fc7e49a7598684d80aa63177= 8dcbce.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/client-options.h | 4 +++ include/migration/register.h | 12 +++++++++ migration/migration.h | 2 ++ migration/savevm.h | 1 + hw/core/machine.c | 1 + migration/colo.c | 3 +++ migration/migration-hmp-cmds.c | 2 ++ migration/migration.c | 2 ++ migration/options.c | 9 +++++++ migration/savevm.c | 39 ++++++++++++++++++++++++++++++ migration/trace-events | 1 + scripts/analyze-migration.py | 11 +++++++++ 12 files changed, 87 insertions(+) diff --git a/include/migration/client-options.h b/include/migration/client-= options.h index 59f4b55cf4f744f222c161ab0b5b9828b53198be..289c9d776221ce3cf35d86f91ce= 3313fd79427d3 100644 --- a/include/migration/client-options.h +++ b/include/migration/client-options.h @@ -10,6 +10,10 @@ #ifndef QEMU_MIGRATION_CLIENT_OPTIONS_H #define QEMU_MIGRATION_CLIENT_OPTIONS_H =20 + +/* properties */ +bool migrate_send_switchover_start(void); + /* capabilities */ =20 bool migrate_background_snapshot(void); diff --git a/include/migration/register.h b/include/migration/register.h index 0b02927383208d1457f21da7ad4900ba53e93638..ff0faf5f68c8e9ac0fd1144ad91= b3fab731d8a2f 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -279,6 +279,18 @@ typedef struct SaveVMHandlers { * otherwise */ bool (*switchover_ack_needed)(void *opaque); + + /** + * @switchover_start + * + * Notifies that the switchover has started. Called only on + * the destination. + * + * @opaque: data pointer passed to register_savevm_live() + * + * Returns zero to indicate success and negative for error + */ + int (*switchover_start)(void *opaque); } SaveVMHandlers; =20 /** diff --git a/migration/migration.h b/migration/migration.h index 4639e2a7e42f6b25c1fe45ff1a9c13695306480a..7b4278e2a32b2044fad4cf7f7a5= defd4435333ea 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -400,6 +400,8 @@ struct MigrationState { bool send_configuration; /* Whether we send section footer during migration */ bool send_section_footer; + /* Whether we send switchover start notification during migration */ + bool send_switchover_start; =20 /* Needed by postcopy-pause state */ QemuSemaphore postcopy_pause_sem; diff --git a/migration/savevm.h b/migration/savevm.h index 7957460062cad0de7129e5f03b9f8d51ba5dc1f9..58f871a7ed9ce8498a16f76ab66= b08ef665aee73 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -53,6 +53,7 @@ void qemu_savevm_send_postcopy_listen(QEMUFile *f); void qemu_savevm_send_postcopy_run(QEMUFile *f); void qemu_savevm_send_postcopy_resume(QEMUFile *f); void qemu_savevm_send_recv_bitmap(QEMUFile *f, char *block_name); +void qemu_savevm_maybe_send_switchover_start(QEMUFile *f); =20 void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name, uint16_t len, diff --git a/hw/core/machine.c b/hw/core/machine.c index b68b8b94a3c43865ec89d9a957289e42d1b26da1..d1ddc3a3db595bd746e8f39b006= a8bf9cd9a52d6 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -44,6 +44,7 @@ GlobalProperty hw_compat_9_2[] =3D { { "virtio-balloon-pci-non-transitional", "vectors", "0" }, { "virtio-mem-pci", "vectors", "0" }, { "migration", "multifd-clean-tls-termination", "false" }, + { "migration", "send-switchover-start", "off"}, }; const size_t hw_compat_9_2_len =3D G_N_ELEMENTS(hw_compat_9_2); =20 diff --git a/migration/colo.c b/migration/colo.c index 9a8e5fbe9b94e9fa56b3c59d3ae4271c1368d7f8..c976b3ff344d7ddf8f850c5a601= b3fe42a503610 100644 --- a/migration/colo.c +++ b/migration/colo.c @@ -452,6 +452,9 @@ static int colo_do_checkpoint_transaction(MigrationStat= e *s, bql_unlock(); goto out; } + + qemu_savevm_maybe_send_switchover_start(s->to_dst_file); + /* Note: device state is saved into buffer */ ret =3D qemu_save_device_state(fb); =20 diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 3347e34c489136106e29baf077bb93fb2f95ef1d..49c26daed359fbc04474be212cf= 7b5c8748043d5 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -46,6 +46,8 @@ static void migration_global_dump(Monitor *mon) ms->send_configuration ? "on" : "off"); monitor_printf(mon, "send-section-footer: %s\n", ms->send_section_footer ? "on" : "off"); + monitor_printf(mon, "send-switchover-start: %s\n", + ms->send_switchover_start ? "on" : "off"); monitor_printf(mon, "clear-bitmap-shift: %u\n", ms->clear_bitmap_shift); } diff --git a/migration/migration.c b/migration/migration.c index c597aa707e5718bfd15ba6f175b4efbd870ee96f..9e9db26667f1adbeaae428be9b9= 7750755c1c144 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2891,6 +2891,8 @@ static bool migration_switchover_start(MigrationState= *s, Error **errp) =20 precopy_notify_complete(); =20 + qemu_savevm_maybe_send_switchover_start(s->to_dst_file); + return true; } =20 diff --git a/migration/options.c b/migration/options.c index bb259d192a932d7af1ada20254a020cf3b2178a2..b0ac2ea4083ffbdf20788e0f508= ab9f02a955f99 100644 --- a/migration/options.c +++ b/migration/options.c @@ -93,6 +93,8 @@ const Property migration_properties[] =3D { send_configuration, true), DEFINE_PROP_BOOL("send-section-footer", MigrationState, send_section_footer, true), + DEFINE_PROP_BOOL("send-switchover-start", MigrationState, + send_switchover_start, true), DEFINE_PROP_BOOL("multifd-flush-after-each-section", MigrationState, multifd_flush_after_each_section, false), DEFINE_PROP_UINT8("x-clear-bitmap-shift", MigrationState, @@ -209,6 +211,13 @@ bool migrate_auto_converge(void) return s->capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE]; } =20 +bool migrate_send_switchover_start(void) +{ + MigrationState *s =3D migrate_get_current(); + + return s->send_switchover_start; +} + bool migrate_background_snapshot(void) { MigrationState *s =3D migrate_get_current(); diff --git a/migration/savevm.c b/migration/savevm.c index 4046faf0091e8ec48c47d4265f325236596a8cc5..faebf47ef51f2c5e6c49df831e1= efe585c1a38a9 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -90,6 +90,7 @@ enum qemu_vm_cmd { MIG_CMD_ENABLE_COLO, /* Enable COLO */ MIG_CMD_POSTCOPY_RESUME, /* resume postcopy on dest */ MIG_CMD_RECV_BITMAP, /* Request for recved bitmap on dst */ + MIG_CMD_SWITCHOVER_START, /* Switchover start notification */ MIG_CMD_MAX }; =20 @@ -109,6 +110,7 @@ static struct mig_cmd_args { [MIG_CMD_POSTCOPY_RESUME] =3D { .len =3D 0, .name =3D "POSTCOPY_RESU= ME" }, [MIG_CMD_PACKAGED] =3D { .len =3D 4, .name =3D "PACKAGED" }, [MIG_CMD_RECV_BITMAP] =3D { .len =3D -1, .name =3D "RECV_BITMAP" = }, + [MIG_CMD_SWITCHOVER_START] =3D { .len =3D 0, .name =3D "SWITCHOVER_ST= ART" }, [MIG_CMD_MAX] =3D { .len =3D -1, .name =3D "MAX" }, }; =20 @@ -1201,6 +1203,19 @@ void qemu_savevm_send_recv_bitmap(QEMUFile *f, char = *block_name) qemu_savevm_command_send(f, MIG_CMD_RECV_BITMAP, len + 1, (uint8_t *)b= uf); } =20 +static void qemu_savevm_send_switchover_start(QEMUFile *f) +{ + trace_savevm_send_switchover_start(); + qemu_savevm_command_send(f, MIG_CMD_SWITCHOVER_START, 0, NULL); +} + +void qemu_savevm_maybe_send_switchover_start(QEMUFile *f) +{ + if (migrate_send_switchover_start()) { + qemu_savevm_send_switchover_start(f); + } +} + bool qemu_savevm_state_blocked(Error **errp) { SaveStateEntry *se; @@ -1687,6 +1702,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **err= p) =20 ret =3D qemu_file_get_error(f); if (ret =3D=3D 0) { + qemu_savevm_maybe_send_switchover_start(f); qemu_savevm_state_complete_precopy(f, false); ret =3D qemu_file_get_error(f); } @@ -2383,6 +2399,26 @@ static int loadvm_process_enable_colo(MigrationIncom= ingState *mis) return ret; } =20 +static int loadvm_postcopy_handle_switchover_start(void) +{ + SaveStateEntry *se; + + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + int ret; + + if (!se->ops || !se->ops->switchover_start) { + continue; + } + + ret =3D se->ops->switchover_start(se->opaque); + if (ret < 0) { + return ret; + } + } + + return 0; +} + /* * Process an incoming 'QEMU_VM_COMMAND' * 0 just a normal return @@ -2481,6 +2517,9 @@ static int loadvm_process_command(QEMUFile *f) =20 case MIG_CMD_ENABLE_COLO: return loadvm_process_enable_colo(mis); + + case MIG_CMD_SWITCHOVER_START: + return loadvm_postcopy_handle_switchover_start(); } =20 return 0; diff --git a/migration/trace-events b/migration/trace-events index 58c0f07f5b2dc3c7550239eab0c410561a21d645..c506e11a2e1d37a4aa22035248b= cce47ac5956b1 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -39,6 +39,7 @@ savevm_send_postcopy_run(void) "" savevm_send_postcopy_resume(void) "" savevm_send_colo_enable(void) "" savevm_send_recv_bitmap(char *name) "%s" +savevm_send_switchover_start(void) "" savevm_state_setup(void) "" savevm_state_resume_prepare(void) "" savevm_state_header(void) "" diff --git a/scripts/analyze-migration.py b/scripts/analyze-migration.py index 8e1fbf4c9d9f13151535f95bfbee4f32ce8d7500..67631ac43e9f4807b16d3ae62b4= eff2d0e340943 100755 --- a/scripts/analyze-migration.py +++ b/scripts/analyze-migration.py @@ -620,7 +620,9 @@ class MigrationDump(object): QEMU_VM_SUBSECTION =3D 0x05 QEMU_VM_VMDESCRIPTION =3D 0x06 QEMU_VM_CONFIGURATION =3D 0x07 + QEMU_VM_COMMAND =3D 0x08 QEMU_VM_SECTION_FOOTER=3D 0x7e + QEMU_MIG_CMD_SWITCHOVER_START =3D 0x0b =20 def __init__(self, filename): self.section_classes =3D { @@ -685,6 +687,15 @@ def read(self, desc_only =3D False, dump_memory =3D Fa= lse, elif section_type =3D=3D self.QEMU_VM_SECTION_PART or section_= type =3D=3D self.QEMU_VM_SECTION_END: section_id =3D file.read32() self.sections[section_id].read() + elif section_type =3D=3D self.QEMU_VM_COMMAND: + command_type =3D file.read16() + command_data_len =3D file.read16() + if command_type !=3D self.QEMU_MIG_CMD_SWITCHOVER_START: + raise Exception("Unknown QEMU_VM_COMMAND: %x" % + (command_type)) + if command_data_len !=3D 0: + raise Exception("Invalid SWITCHOVER_START length: %x" % + (command_data_len)) elif section_type =3D=3D self.QEMU_VM_SECTION_FOOTER: read_section_id =3D file.read32() if read_section_id !=3D section_id: --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270640; cv=none; d=zohomail.com; s=zohoarc; b=YIeirmipjm9hgBsQ8PJzDos2qo/RKQ4vTGJxNqMsDfix5nJA5vmnBp/BIfJ1fZ1kpNfHxRWSO6QyO2issQaUSlz2U5lpRMmnsQDr5+mBymf7zH94VyMpNYkOHSCsXead8h/TJU1JyAEIpuxgBljKhmQQVIcEEMOPTszS29ZDdiE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270640; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=cttGIj/EgxZzLx7fzWWIjmhxHltQh8WylsSt0tpo05g=; b=g4DHu41bvCqU8AmfoT7CX+7udkN9AL07Cw1gFswREwvdekGkS81bC2cQ/1n595uPsf9yqfFZnnSBpyLRrAZdDlt/SwhboUpbBnai3gGuyH3HUAW40fM9i1arWkv7t01GozUwCYLcD3V3Elr7brLMXE/cKR7lqlV5kX14MD72Hcg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270640138137.01009162069818; Thu, 6 Mar 2025 06:17:20 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC0v-0007qp-Hk; Thu, 06 Mar 2025 09:15:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0U-0007Hq-LO for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0Q-0001nT-46 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:25 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-549-XAI3TW-2MkGN7_aj0iTOTA-1; Thu, 06 Mar 2025 09:15:16 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 33B7D180AF69; Thu, 6 Mar 2025 14:15:15 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 15D2D18001D3; Thu, 6 Mar 2025 14:15:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270520; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cttGIj/EgxZzLx7fzWWIjmhxHltQh8WylsSt0tpo05g=; b=MvRxb28AV1FipyAqAZnukhnz312f9M+9Y1z0hs+iBDP3VqUW0rH4x6Sgl6fW4cAfJLjT6N vz7kT9CZMwLwF1r79LHSLOp98duzrg6+bcBwRdHSbZHoe0++8YdwuuqRs0S2/p6Iskb8xM 4XtejrlvHP90WCD9vzDjBDYfkqnFZUA= X-MC-Unique: XAI3TW-2MkGN7_aj0iTOTA-1 X-Mimecast-MFC-AGG-ID: XAI3TW-2MkGN7_aj0iTOTA_1741270515 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 15/42] migration: Add qemu_loadvm_load_state_buffer() and its handler Date: Thu, 6 Mar 2025 15:13:51 +0100 Message-ID: <20250306141419.2015340-16-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270640895019000 From: "Maciej S. Szmigiero" qemu_loadvm_load_state_buffer() and its load_state_buffer SaveVMHandler allow providing device state buffer to explicitly specified device via its idstr and instance id. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/71ca753286b87831ced4afd422e2e2bed0= 71af25.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/register.h | 15 +++++++++++++++ migration/savevm.h | 3 +++ migration/savevm.c | 23 +++++++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/include/migration/register.h b/include/migration/register.h index ff0faf5f68c8e9ac0fd1144ad91b3fab731d8a2f..58891aa54b76cccdea9e442be9b= f528a41593e26 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -229,6 +229,21 @@ typedef struct SaveVMHandlers { */ int (*load_state)(QEMUFile *f, void *opaque, int version_id); =20 + /** + * @load_state_buffer (invoked outside the BQL) + * + * Load device state buffer provided to qemu_loadvm_load_state_buffer(= ). + * + * @opaque: data pointer passed to register_savevm_live() + * @buf: the data buffer to load + * @len: the data length in buffer + * @errp: pointer to Error*, to store an error if it happens. + * + * Returns true to indicate success and false for errors. + */ + bool (*load_state_buffer)(void *opaque, char *buf, size_t len, + Error **errp); + /** * @load_setup * diff --git a/migration/savevm.h b/migration/savevm.h index 58f871a7ed9ce8498a16f76ab66b08ef665aee73..cb58434a9437f7f9752ae7ae029= 81e9927d4ce85 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -71,4 +71,7 @@ int qemu_loadvm_approve_switchover(void); int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, bool in_postcopy); =20 +bool qemu_loadvm_load_state_buffer(const char *idstr, uint32_t instance_id, + char *buf, size_t len, Error **errp); + #endif diff --git a/migration/savevm.c b/migration/savevm.c index faebf47ef51f2c5e6c49df831e1efe585c1a38a9..7c1aa8ad7b9d5c1460467c36e99= 23ed13768fbad 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -3060,6 +3060,29 @@ int qemu_loadvm_approve_switchover(void) return migrate_send_rp_switchover_ack(mis); } =20 +bool qemu_loadvm_load_state_buffer(const char *idstr, uint32_t instance_id, + char *buf, size_t len, Error **errp) +{ + SaveStateEntry *se; + + se =3D find_se(idstr, instance_id); + if (!se) { + error_setg(errp, + "Unknown idstr %s or instance id %u for load state buff= er", + idstr, instance_id); + return false; + } + + if (!se->ops || !se->ops->load_state_buffer) { + error_setg(errp, + "idstr %s / instance %u has no load state buffer operat= ion", + idstr, instance_id); + return false; + } + + return se->ops->load_state_buffer(se->opaque, buf, len, errp); +} + bool save_snapshot(const char *name, bool overwrite, const char *vmstate, bool has_devices, strList *devices, Error **errp) { --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270584; cv=none; d=zohomail.com; s=zohoarc; b=l4e1lzbnwL3Q825u4+27gcn7bOb3unvyddoXkyonPanXX1UEE5w8bPtJStoPtbJsV20c+F7qndcNDcdGVYmQokR0G12d/PWSmulCMJMUbe/OjbPnaEdT/u4CKJPctlhlCTdYiSKnwUQsGAst4D1m0qgDW9ANKJS86YGkmiXob7I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270584; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=lBo8krjcIEnw3jWVq23SGS/3MsVWrcDF6+cmTHeCWXU=; b=JYG/w/24FTOLa82kVUOfeEBprFPmwmVjBdu7S0Kd332uXX0olhq5bdnoGAMIawgLtfIanNZeiF+0oRNpzA2+Dh84LFLz4eUPyIcM9jyG1xxZUSjZHlL0yCRCU9m4HYU+tyDE37I9GeXutPC4hX977j9XNiKKev1dC6a/dcc/hq8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270584727187.0751419114266; Thu, 6 Mar 2025 06:16:24 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC19-0008SK-NW; Thu, 06 Mar 2025 09:16:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0a-0007MN-Qv for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0X-0001o7-ME for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:32 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-317-qUgFnNE7MG6oznCWyK7O_Q-1; Thu, 06 Mar 2025 09:15:19 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 43C56180AF67; Thu, 6 Mar 2025 14:15:18 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2F44418009BC; Thu, 6 Mar 2025 14:15:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lBo8krjcIEnw3jWVq23SGS/3MsVWrcDF6+cmTHeCWXU=; b=gtIGxbXoUZCr+jfTWQQEpwBwYKOBy7v8aHeEzVT4bwrMbRzWAxFvu6EE1DLgPjbNVjZVsc LgJljzGYCYy9+Ifpz0LWeXciCcsi/FhQBOkFZx+aRX5R+CRK/1Q3oygoQT8NcQZYua/Pao 3qKq3gv2daPp+sCaNU9Xf3UCNHB1Mho= X-MC-Unique: qUgFnNE7MG6oznCWyK7O_Q-1 X-Mimecast-MFC-AGG-ID: qUgFnNE7MG6oznCWyK7O_Q_1741270518 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 16/42] migration: Always take BQL for migration_incoming_state_destroy() Date: Thu, 6 Mar 2025 15:13:52 +0100 Message-ID: <20250306141419.2015340-17-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270586954019000 From: "Maciej S. Szmigiero" All callers to migration_incoming_state_destroy() other than postcopy_ram_listen_thread() do this call with BQL held. Since migration_incoming_state_destroy() ultimately calls "load_cleanup" SaveVMHandlers and it will soon call BQL-sensitive code it makes sense to always call that function under BQL rather than to have it deal with both cases (with BQL and without BQL). Add the necessary bql_lock() and bql_unlock() to postcopy_ram_listen_thread(). qemu_loadvm_state_main() in postcopy_ram_listen_thread() could call "load_state" SaveVMHandlers that are expecting BQL to be held. In principle, the only devices that should be arriving on migration channel serviced by postcopy_ram_listen_thread() are those that are postcopiable and whose load handlers are safe to be called without BQL being held. But nothing currently prevents the source from sending data for "unsafe" devices which would cause trouble there. Add a TODO comment there so it's clear that it would be good to improve handling of such (erroneous) case in the future. Acked-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/21bb5ca337b1d5a802e697f553f37faf29= 6b5ff4.1741193259.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/migration.c | 13 +++++++++++++ migration/savevm.c | 4 ++++ 2 files changed, 17 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 9e9db26667f1adbeaae428be9b97750755c1c144..0bf70ea9717d73b0816f6ae52b9= 9ae67924e8030 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -402,10 +402,23 @@ void migration_incoming_state_destroy(void) struct MigrationIncomingState *mis =3D migration_incoming_get_current(= ); =20 multifd_recv_cleanup(); + /* * RAM state cleanup needs to happen after multifd cleanup, because * multifd threads can use some of its states (receivedmap). + * The VFIO load_cleanup() implementation is BQL-sensitive. It requires + * BQL must NOT be taken when recycling load threads, so that it won't + * block the load threads from making progress on address space + * modification operations. + * + * To make it work, we could try to not take BQL for all load_cleanup(= ), + * or conditionally unlock BQL only if bql_locked() in VFIO. + * + * Since most existing call sites take BQL for load_cleanup(), make + * it simple by taking BQL always as the rule, so that VFIO can unlock + * BQL and retake unconditionally. */ + assert(bql_locked()); qemu_loadvm_state_cleanup(); =20 if (mis->to_src_file) { diff --git a/migration/savevm.c b/migration/savevm.c index 7c1aa8ad7b9d5c1460467c36e9923ed13768fbad..3e86b572cfa82c201b1bf935080= a2e0ca651be0e 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -1986,6 +1986,8 @@ static void *postcopy_ram_listen_thread(void *opaque) * in qemu_file, and thus we must be blocking now. */ qemu_file_set_blocking(f, true); + + /* TODO: sanity check that only postcopiable data will be loaded here = */ load_res =3D qemu_loadvm_state_main(f, mis); =20 /* @@ -2046,7 +2048,9 @@ static void *postcopy_ram_listen_thread(void *opaque) * (If something broke then qemu will have to exit anyway since it's * got a bad migration state). */ + bql_lock(); migration_incoming_state_destroy(); + bql_unlock(); =20 rcu_unregister_thread(); mis->have_listen_thread =3D false; --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271054; cv=none; d=zohomail.com; s=zohoarc; b=E2Ok+U4x/oXoJ6TqfFjye9lQtjxhLjLyqBNq6BUe5WKjSTje9aF4UqyivfVQ+JUParYFic48tJc8o+ji3M7cbCtBRteMIGrZc/I6sPtPCRodfTOwEf9O+jXZPuWIL9Bdh7ulPSG+Q6gHNGha0IH0HZ49tr30K/ZvHg+5DGEzSkY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271054; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=pRTQCy+UqYQQonEzfSl2x3lH0mLrSbC/jP7b/oTheS4=; b=gAyGMEflhCU2gg0J/JQiGB5852rUXRLMy5xHWmZyjIbsR/C5D4LAMguMpYYpqs5UJ/jWKgpUFget7zEk++RTm6BvB/X4ZQjPD87TFXV4vgCTeWO47YBI0LROTWrTV8Cadh9IXd1DUxWFvZw+X3TfSJVpXVBbQ0734904EKM9DxY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271054952529.8442824342609; Thu, 6 Mar 2025 06:24:14 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC1p-0000sF-47; Thu, 06 Mar 2025 09:16:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0Z-0007MG-27 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:35 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0U-0001np-5Q for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:27 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-605-z00htaDNNNGi-OSQBy25Mg-1; Thu, 06 Mar 2025 09:15:22 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 49C0019560B7; Thu, 6 Mar 2025 14:15:21 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C73C81801748; Thu, 6 Mar 2025 14:15:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270525; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pRTQCy+UqYQQonEzfSl2x3lH0mLrSbC/jP7b/oTheS4=; b=DXbjHURz0Hk+tDPfZwhRnn6M8xKNKqNgjzv46v8DJDH60gTR0dio1hH+6vHa9tLg2uwMqb SfL5wTGOddSvgKYE54stW3bsHsfdrit8RqMgGwvZKUYmupkdAkUoQ1NL8ymj+TuJ50rPgy DSXQftbPI8AbgiG+4H5KfXjlV6rjYXQ= X-MC-Unique: z00htaDNNNGi-OSQBy25Mg-1 X-Mimecast-MFC-AGG-ID: z00htaDNNNGi-OSQBy25Mg_1741270521 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 17/42] error: define g_autoptr() cleanup function for the Error type Date: Thu, 6 Mar 2025 15:13:53 +0100 Message-ID: <20250306141419.2015340-18-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271055850019100 From: "Maciej S. Szmigiero" Automatic memory management helps avoid memory safety issues. Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/a5843c5fa64d7e5239a4316092ec0ef0d1= 0c2320.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/qapi/error.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/qapi/error.h b/include/qapi/error.h index f5fe2162623e5770d652f7415ebc25172d97616e..41e38163804904874e5d114b699= a3e2a93d3fc6d 100644 --- a/include/qapi/error.h +++ b/include/qapi/error.h @@ -437,6 +437,8 @@ Error *error_copy(const Error *err); */ void error_free(Error *err); =20 +G_DEFINE_AUTOPTR_CLEANUP_FUNC(Error, error_free) + /* * Convenience function to assert that *@errp is set, then silently free i= t. */ --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270799; cv=none; d=zohomail.com; s=zohoarc; b=nmGGUFThCPibIhMZGh2crr93QC1As3w+/WhALj5PR4tbU2ViPvw9eVMERTchUVHnuXHXa4e3fIRAlg9eAxvzTPIzALuNCaCa82MP1/tW1eA3ga276pEjozXqdUGTbDpUFGyfyDTgvG5lwBi0sVMo1RAWjhC6znZbquKnMvJurQ8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270799; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=e7Bt2EpAIGgMArz6Dk8AbOWgsYJFhPuLViysJsREzs8=; b=cQN2yWi7y+kc9b/6bDxxFzgZUsGfSQCvxKI1M+ox5n1ORWIhqys0QRJt3wGWgxJQs3zo/wy5E2UwYVP30wYphz+FwBxJvMVUT9Y/dCljJu76OGzNkpbVrav65nvb+eR843QRLuRr3duzXnN0r9RI7hdRFgkJAsWk+CCUivBf+XY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270799542997.4909231102906; Thu, 6 Mar 2025 06:19:59 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC1W-0000fD-VY; Thu, 06 Mar 2025 09:16:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0m-0007cu-Si for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:45 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0k-0001px-Gg for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:44 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-86-Bmp2nXKAPAyBXX5uxFHJ4Q-1; Thu, 06 Mar 2025 09:15:24 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A16F719560B0; Thu, 6 Mar 2025 14:15:23 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C499B180174C; Thu, 6 Mar 2025 14:15:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270542; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e7Bt2EpAIGgMArz6Dk8AbOWgsYJFhPuLViysJsREzs8=; b=eWz4t8APhBAkf0IUEzj1FGpeoBzsTD2FuE6ykQxdkDxrlGJFi6g0ZdE8FHunIIrFHfidJs 4rYOClAgBAptyy++VMVTHloehqj6pT/rt1fg3dHZTfk9S00xbGGVri3Qw/i+CWfTX8vUES IDWlMQzbIi0Pf9cUFVh/TIdvrUKevE4= X-MC-Unique: Bmp2nXKAPAyBXX5uxFHJ4Q-1 X-Mimecast-MFC-AGG-ID: Bmp2nXKAPAyBXX5uxFHJ4Q_1741270523 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 18/42] migration: Add thread pool of optional load threads Date: Thu, 6 Mar 2025 15:13:54 +0100 Message-ID: <20250306141419.2015340-19-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270801885019000 From: "Maciej S. Szmigiero" Some drivers might want to make use of auxiliary helper threads during VM state loading, for example to make sure that their blocking (sync) I/O operations don't block the rest of the migration process. Add a migration core managed thread pool to facilitate this use case. The migration core will wait for these threads to finish before (re)starting the VM at destination. Reviewed-by: Fabiano Rosas Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/b09fd70369b6159c75847e69f235cb908b= 02570c.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/misc.h | 3 ++ include/qemu/typedefs.h | 2 + migration/migration.h | 5 +++ migration/savevm.h | 2 +- migration/migration.c | 2 +- migration/savevm.c | 95 +++++++++++++++++++++++++++++++++++++++- 6 files changed, 105 insertions(+), 4 deletions(-) diff --git a/include/migration/misc.h b/include/migration/misc.h index c660be80954abdd768e419d2ab892fc034c7349d..4c171f4e897e2bea9016a1559be= f3e89c165b176 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -45,9 +45,12 @@ bool migrate_ram_is_ignored(RAMBlock *block); /* migration/block.c */ =20 AnnounceParameters *migrate_announce_params(void); + /* migration/savevm.c */ =20 void dump_vmstate_json_to_file(FILE *out_fp); +void qemu_loadvm_start_load_thread(MigrationLoadThread function, + void *opaque); =20 /* migration/migration.c */ void migration_object_init(void); diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h index 3d84efcac47a2f1a34f177a1ed0df5aeae04fdb3..fd23ff7771b1bd6cd51f48bbc5b= e86aa57982307 100644 --- a/include/qemu/typedefs.h +++ b/include/qemu/typedefs.h @@ -131,5 +131,7 @@ typedef struct IRQState *qemu_irq; * Function types */ typedef void (*qemu_irq_handler)(void *opaque, int n, int level); +typedef bool (*MigrationLoadThread)(void *opaque, bool *should_quit, + Error **errp); =20 #endif /* QEMU_TYPEDEFS_H */ diff --git a/migration/migration.h b/migration/migration.h index 7b4278e2a32b2044fad4cf7f7a5defd4435333ea..d53f7cad84d8e4a8bd8546f94d6= 35c9733d71961 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -43,6 +43,7 @@ #define MIGRATION_THREAD_DST_PREEMPT "mig/dst/preempt" =20 struct PostcopyBlocktimeContext; +typedef struct ThreadPool ThreadPool; =20 #define MIGRATION_RESUME_ACK_VALUE (1) =20 @@ -187,6 +188,10 @@ struct MigrationIncomingState { Coroutine *colo_incoming_co; QemuSemaphore colo_incoming_sem; =20 + /* Optional load threads pool and its thread exit request flag */ + ThreadPool *load_threads; + bool load_threads_abort; + /* * PostcopyBlocktimeContext to keep information for postcopy * live migration, to calculate vCPU block time diff --git a/migration/savevm.h b/migration/savevm.h index cb58434a9437f7f9752ae7ae02981e9927d4ce85..138c39a7f9f97f69957eac63f33= 8e4807bd7e8c5 100644 --- a/migration/savevm.h +++ b/migration/savevm.h @@ -64,7 +64,7 @@ void qemu_savevm_live_state(QEMUFile *f); int qemu_save_device_state(QEMUFile *f); =20 int qemu_loadvm_state(QEMUFile *f); -void qemu_loadvm_state_cleanup(void); +void qemu_loadvm_state_cleanup(MigrationIncomingState *mis); int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis); int qemu_load_device_state(QEMUFile *f); int qemu_loadvm_approve_switchover(void); diff --git a/migration/migration.c b/migration/migration.c index 0bf70ea9717d73b0816f6ae52b99ae67924e8030..1833cfe3580cd6e587c6c7cb754= 458c34baf61e8 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -419,7 +419,7 @@ void migration_incoming_state_destroy(void) * BQL and retake unconditionally. */ assert(bql_locked()); - qemu_loadvm_state_cleanup(); + qemu_loadvm_state_cleanup(mis); =20 if (mis->to_src_file) { /* Tell source that we are done */ diff --git a/migration/savevm.c b/migration/savevm.c index 3e86b572cfa82c201b1bf935080a2e0ca651be0e..1abc365570e324dd85f8a95adeb= 1a95f57b73264 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -54,6 +54,7 @@ #include "qemu/job.h" #include "qemu/main-loop.h" #include "block/snapshot.h" +#include "block/thread-pool.h" #include "qemu/cutils.h" #include "io/channel-buffer.h" #include "io/channel-file.h" @@ -131,6 +132,35 @@ static struct mig_cmd_args { * generic extendable format with an exception for two old entities. */ =20 +/***********************************************************/ +/* Optional load threads pool support */ + +static void qemu_loadvm_thread_pool_create(MigrationIncomingState *mis) +{ + assert(!mis->load_threads); + mis->load_threads =3D thread_pool_new(); + mis->load_threads_abort =3D false; +} + +static void qemu_loadvm_thread_pool_destroy(MigrationIncomingState *mis) +{ + qatomic_set(&mis->load_threads_abort, true); + + bql_unlock(); /* Load threads might be waiting for BQL */ + g_clear_pointer(&mis->load_threads, thread_pool_free); + bql_lock(); +} + +static bool qemu_loadvm_thread_pool_wait(MigrationState *s, + MigrationIncomingState *mis) +{ + bql_unlock(); /* Let load threads do work requiring BQL */ + thread_pool_wait(mis->load_threads); + bql_lock(); + + return !migrate_has_error(s); +} + /***********************************************************/ /* savevm/loadvm support */ =20 @@ -2783,16 +2813,68 @@ static int qemu_loadvm_state_setup(QEMUFile *f, Err= or **errp) return 0; } =20 -void qemu_loadvm_state_cleanup(void) +struct LoadThreadData { + MigrationLoadThread function; + void *opaque; +}; + +static int qemu_loadvm_load_thread(void *thread_opaque) +{ + struct LoadThreadData *data =3D thread_opaque; + MigrationIncomingState *mis =3D migration_incoming_get_current(); + g_autoptr(Error) local_err =3D NULL; + + if (!data->function(data->opaque, &mis->load_threads_abort, &local_err= )) { + MigrationState *s =3D migrate_get_current(); + + /* + * Can't set load_threads_abort here since processing of main migr= ation + * channel data could still be happening, resulting in launching o= f new + * load threads. + */ + + assert(local_err); + + /* + * In case of multiple load threads failing which thread error + * return we end setting is purely arbitrary. + */ + migrate_set_error(s, local_err); + } + + return 0; +} + +void qemu_loadvm_start_load_thread(MigrationLoadThread function, + void *opaque) +{ + MigrationIncomingState *mis =3D migration_incoming_get_current(); + struct LoadThreadData *data; + + /* We only set it from this thread so it's okay to read it directly */ + assert(!mis->load_threads_abort); + + data =3D g_new(struct LoadThreadData, 1); + data->function =3D function; + data->opaque =3D opaque; + + thread_pool_submit_immediate(mis->load_threads, qemu_loadvm_load_threa= d, + data, g_free); +} + +void qemu_loadvm_state_cleanup(MigrationIncomingState *mis) { SaveStateEntry *se; =20 trace_loadvm_state_cleanup(); + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { if (se->ops && se->ops->load_cleanup) { se->ops->load_cleanup(se->opaque); } } + + qemu_loadvm_thread_pool_destroy(mis); } =20 /* Return true if we should continue the migration, or false. */ @@ -2943,6 +3025,7 @@ out: =20 int qemu_loadvm_state(QEMUFile *f) { + MigrationState *s =3D migrate_get_current(); MigrationIncomingState *mis =3D migration_incoming_get_current(); Error *local_err =3D NULL; int ret; @@ -2952,6 +3035,8 @@ int qemu_loadvm_state(QEMUFile *f) return -EINVAL; } =20 + qemu_loadvm_thread_pool_create(mis); + ret =3D qemu_loadvm_state_header(f); if (ret) { return ret; @@ -2983,12 +3068,18 @@ int qemu_loadvm_state(QEMUFile *f) =20 /* When reaching here, it must be precopy */ if (ret =3D=3D 0) { - if (migrate_has_error(migrate_get_current())) { + if (migrate_has_error(migrate_get_current()) || + !qemu_loadvm_thread_pool_wait(s, mis)) { ret =3D -EINVAL; } else { ret =3D qemu_file_get_error(f); } } + /* + * Set this flag unconditionally so we'll catch further attempts to + * start additional threads via an appropriate assert() + */ + qatomic_set(&mis->load_threads_abort, true); =20 /* * Try to read in the VMDESC section as well, so that dumping tools th= at --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270797; cv=none; d=zohomail.com; s=zohoarc; b=hshZNyGZAuSigeIa+rDr4XSblx25hXWCUCFAyqPjdChLRx/pA+pi7WcAKWRIPYhU208zPeFoa83VuW5R08YEyr45nj2t8I5o4UV0SlIpanS4nvaGJcVkIiItRRp2ZHl2T0NtfB1GvvtJsYmdGqEQZ7/cAWTF10xPiFXZVB8BGfU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270797; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=eQLanTp3GDSohaLUCQp6avjWsH1FyqzHgFyDaJKhHhk=; b=NKWs4DYhwfgSBcVYFKOZZiv/G4fwV9/ZnyH3k/hFEhe+blhJS6CYzn63nWWm3AYOKOByen0CAyPX+18TpXXZ67J+F32aCq2vquFGvLZyP2RXQxoozJNSGy99BXO6Vc3rJ3vg+mej90DOE2z1zDFEgzFZsNugJ2kEYyWgEVWbC+8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270797079486.49612607233473; Thu, 6 Mar 2025 06:19:57 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC10-00087A-HL; Thu, 06 Mar 2025 09:15:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0d-0007NG-MH for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:36 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0b-0001ow-FR for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:34 -0500 Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-608-hi75gCTENn-0uqNdd5x9AA-1; Thu, 06 Mar 2025 09:15:27 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 573941954B31; Thu, 6 Mar 2025 14:15:26 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2995E1801748; Thu, 6 Mar 2025 14:15:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQLanTp3GDSohaLUCQp6avjWsH1FyqzHgFyDaJKhHhk=; b=cP+abiR0MxmY0MOwvSjyswa4zZxS7zeMfJBZ0sPY8afgdNeGfAw7+Va2V6IMuv9wC0/0aZ LHDkuIH8xpkAjxzK63HDTlDlWXU3C4EOzW/hgN0ededQxCoVMwIntYbvRxAMcAvrr7TuJd ulJyywoYeiebDOFhDfqQt5wYz2YjIKQ= X-MC-Unique: hi75gCTENn-0uqNdd5x9AA-1 X-Mimecast-MFC-AGG-ID: hi75gCTENn-0uqNdd5x9AA_1741270526 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 19/42] migration/multifd: Split packet into header and RAM data Date: Thu, 6 Mar 2025 15:13:55 +0100 Message-ID: <20250306141419.2015340-20-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270799095019100 From: "Maciej S. Szmigiero" Read packet header first so in the future we will be able to differentiate between a RAM multifd packet and a device state multifd packet. Since these two are of different size we can't read the packet body until we know which packet type it is. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/832ad055fe447561ac1ad565d61658660c= b3f63f.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/multifd.h | 5 +++++ migration/multifd.c | 55 ++++++++++++++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 11 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index cf408ff72140a51fef7bdfc90ba11044992cfd2e..f7156f66c0f6b3872c12af7ccda= b55806509fac2 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -69,6 +69,11 @@ typedef struct { uint32_t magic; uint32_t version; uint32_t flags; +} __attribute__((packed)) MultiFDPacketHdr_t; + +typedef struct { + MultiFDPacketHdr_t hdr; + /* maximum number of allocated pages */ uint32_t pages_alloc; /* non zero pages */ diff --git a/migration/multifd.c b/migration/multifd.c index 215ad0414a799cfb389c501331de5e505600905d..3b47e63c2c4a6a65751b4eb4e95= 01a443695ff8f 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -209,10 +209,10 @@ void multifd_send_fill_packet(MultiFDSendParams *p) =20 memset(packet, 0, p->packet_len); =20 - packet->magic =3D cpu_to_be32(MULTIFD_MAGIC); - packet->version =3D cpu_to_be32(MULTIFD_VERSION); + packet->hdr.magic =3D cpu_to_be32(MULTIFD_MAGIC); + packet->hdr.version =3D cpu_to_be32(MULTIFD_VERSION); =20 - packet->flags =3D cpu_to_be32(p->flags); + packet->hdr.flags =3D cpu_to_be32(p->flags); packet->next_packet_size =3D cpu_to_be32(p->next_packet_size); =20 packet_num =3D qatomic_fetch_inc(&multifd_send_state->packet_num); @@ -228,12 +228,12 @@ void multifd_send_fill_packet(MultiFDSendParams *p) p->flags, p->next_packet_size); } =20 -static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +static int multifd_recv_unfill_packet_header(MultiFDRecvParams *p, + const MultiFDPacketHdr_t *hdr, + Error **errp) { - const MultiFDPacket_t *packet =3D p->packet; - uint32_t magic =3D be32_to_cpu(packet->magic); - uint32_t version =3D be32_to_cpu(packet->version); - int ret =3D 0; + uint32_t magic =3D be32_to_cpu(hdr->magic); + uint32_t version =3D be32_to_cpu(hdr->version); =20 if (magic !=3D MULTIFD_MAGIC) { error_setg(errp, "multifd: received packet magic %x, expected %x", @@ -247,7 +247,16 @@ static int multifd_recv_unfill_packet(MultiFDRecvParam= s *p, Error **errp) return -1; } =20 - p->flags =3D be32_to_cpu(packet->flags); + p->flags =3D be32_to_cpu(hdr->flags); + + return 0; +} + +static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +{ + const MultiFDPacket_t *packet =3D p->packet; + int ret =3D 0; + p->next_packet_size =3D be32_to_cpu(packet->next_packet_size); p->packet_num =3D be64_to_cpu(packet->packet_num); p->packets_recved++; @@ -1165,14 +1174,18 @@ static void *multifd_recv_thread(void *opaque) } =20 while (true) { + MultiFDPacketHdr_t hdr; uint32_t flags =3D 0; bool has_data =3D false; + uint8_t *pkt_buf; + size_t pkt_len; + p->normal_num =3D 0; =20 if (use_packets) { struct iovec iov =3D { - .iov_base =3D (void *)p->packet, - .iov_len =3D p->packet_len + .iov_base =3D (void *)&hdr, + .iov_len =3D sizeof(hdr) }; =20 if (multifd_recv_should_exit()) { @@ -1191,6 +1204,26 @@ static void *multifd_recv_thread(void *opaque) break; } =20 + ret =3D multifd_recv_unfill_packet_header(p, &hdr, &local_err); + if (ret) { + break; + } + + pkt_buf =3D (uint8_t *)p->packet + sizeof(hdr); + pkt_len =3D p->packet_len - sizeof(hdr); + + ret =3D qio_channel_read_all_eof(p->c, (char *)pkt_buf, pkt_le= n, + &local_err); + if (!ret) { + /* EOF */ + error_setg(&local_err, "multifd: unexpected EOF after pack= et header"); + break; + } + + if (ret =3D=3D -1) { + break; + } + qemu_mutex_lock(&p->mutex); ret =3D multifd_recv_unfill_packet(p, &local_err); if (ret) { --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271023; cv=none; d=zohomail.com; s=zohoarc; b=b5/nzWIuclr/A5MxxcTsYMiAFPqgfkdaCTU9Ja2wn+kSLdwj6rngVpqxbWXvuYaMyod1KbBxu83bSa4aGp2t4N5NvqcPLj0l7znfwQPJNaX2u5+4fyfJSJMWwA53M9qxLkNueslIVzByi5yUhYPexNA+f5ycZj6ZvxcaHeIJ29c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271023; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=TmvjSK2wG1iCeODaoW8+kT8fos4M/jV8WwkpxI4oRg8=; b=dSjge5xzkpae/IJf2yHre0Uo5qUgZsHWZBxnEEgBdbjId7jzp7VTdzB/TedYMglM7JQpC9OFJZGah8UaxoZulqOq2t1RGJylhUa7Zt6g4DndXHDsIDq2xZPVzQS2j4P+WgMayakeY5xgKphjYnO9Z3Y7tSbOIs/P4fyt9+VAvYA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271023425750.606335382579; Thu, 6 Mar 2025 06:23:43 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2F-0001Ii-S9; Thu, 06 Mar 2025 09:17:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0v-0007q3-BA for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:53 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0q-0001r1-V5 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:53 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-424-tn0YLadFO-K0cAoSgGuFew-1; Thu, 06 Mar 2025 09:15:29 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0E00D1800349; Thu, 6 Mar 2025 14:15:29 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D3361180174E; Thu, 6 Mar 2025 14:15:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270548; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TmvjSK2wG1iCeODaoW8+kT8fos4M/jV8WwkpxI4oRg8=; b=Z4gsGyuLSsoCiEss95swSvgYpnfGcMDArzDpKx9fKAIDcuEutYzlx5Vi8ngdcuqV5zD+ty h5ibUjSoakPs/J98ewx28K2SE0Ba0zOpayf8PqStoKew6jBQ4XG730jP5libL/LNSVekxX cEaR6/grGS8m4bR2woX7iVhkDXcUeTs= X-MC-Unique: tn0YLadFO-K0cAoSgGuFew-1 X-Mimecast-MFC-AGG-ID: tn0YLadFO-K0cAoSgGuFew_1741270529 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 20/42] migration/multifd: Device state transfer support - receive side Date: Thu, 6 Mar 2025 15:13:56 +0100 Message-ID: <20250306141419.2015340-21-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271024801019000 From: "Maciej S. Szmigiero" Add a basic support for receiving device state via multifd channels - channels that are shared with RAM transfers. Depending whether MULTIFD_FLAG_DEVICE_STATE flag is present or not in the packet header either device state (MultiFDPacketDeviceState_t) or RAM data (existing MultiFDPacket_t) is read. The received device state data is provided to qemu_loadvm_load_state_buffer() function for processing in the device's load_state_buffer handler. Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/9b86f806c134e7815ecce0eee84f0e0e34= aa0146.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/multifd.h | 19 ++++++++- migration/multifd.c | 101 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 108 insertions(+), 12 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index f7156f66c0f6b3872c12af7ccdab55806509fac2..d682c5a9b74388bca7ea129d766= 46dc1e5fe1446 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -62,6 +62,12 @@ MultiFDRecvData *multifd_get_recv_data(void); #define MULTIFD_FLAG_UADK (8 << 1) #define MULTIFD_FLAG_QATZIP (16 << 1) =20 +/* + * If set it means that this packet contains device state + * (MultiFDPacketDeviceState_t), not RAM data (MultiFDPacket_t). + */ +#define MULTIFD_FLAG_DEVICE_STATE (32 << 1) + /* This value needs to be a multiple of qemu_target_page_size() */ #define MULTIFD_PACKET_SIZE (512 * 1024) =20 @@ -94,6 +100,16 @@ typedef struct { uint64_t offset[]; } __attribute__((packed)) MultiFDPacket_t; =20 +typedef struct { + MultiFDPacketHdr_t hdr; + + char idstr[256]; + uint32_t instance_id; + + /* size of the next packet that contains the actual data */ + uint32_t next_packet_size; +} __attribute__((packed)) MultiFDPacketDeviceState_t; + typedef struct { /* number of used pages */ uint32_t num; @@ -227,8 +243,9 @@ typedef struct { =20 /* thread local variables. No locking required */ =20 - /* pointer to the packet */ + /* pointers to the possible packet types */ MultiFDPacket_t *packet; + MultiFDPacketDeviceState_t *packet_dev_state; /* size of the next packet that contains pages */ uint32_t next_packet_size; /* packets received through this channel */ diff --git a/migration/multifd.c b/migration/multifd.c index 3b47e63c2c4a6a65751b4eb4e9501a443695ff8f..01f427d8ed036417559a0573d36= 9172f72b508e1 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -21,6 +21,7 @@ #include "file.h" #include "migration.h" #include "migration-stats.h" +#include "savevm.h" #include "socket.h" #include "tls.h" #include "qemu-file.h" @@ -252,14 +253,24 @@ static int multifd_recv_unfill_packet_header(MultiFDR= ecvParams *p, return 0; } =20 -static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +static int multifd_recv_unfill_packet_device_state(MultiFDRecvParams *p, + Error **errp) +{ + MultiFDPacketDeviceState_t *packet =3D p->packet_dev_state; + + packet->instance_id =3D be32_to_cpu(packet->instance_id); + p->next_packet_size =3D be32_to_cpu(packet->next_packet_size); + + return 0; +} + +static int multifd_recv_unfill_packet_ram(MultiFDRecvParams *p, Error **er= rp) { const MultiFDPacket_t *packet =3D p->packet; int ret =3D 0; =20 p->next_packet_size =3D be32_to_cpu(packet->next_packet_size); p->packet_num =3D be64_to_cpu(packet->packet_num); - p->packets_recved++; =20 /* Always unfill, old QEMUs (<9.0) send data along with SYNC */ ret =3D multifd_ram_unfill_packet(p, errp); @@ -270,6 +281,17 @@ static int multifd_recv_unfill_packet(MultiFDRecvParam= s *p, Error **errp) return ret; } =20 +static int multifd_recv_unfill_packet(MultiFDRecvParams *p, Error **errp) +{ + p->packets_recved++; + + if (p->flags & MULTIFD_FLAG_DEVICE_STATE) { + return multifd_recv_unfill_packet_device_state(p, errp); + } + + return multifd_recv_unfill_packet_ram(p, errp); +} + static bool multifd_send_should_exit(void) { return qatomic_read(&multifd_send_state->exiting); @@ -1057,6 +1079,7 @@ static void multifd_recv_cleanup_channel(MultiFDRecvP= arams *p) p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; + g_clear_pointer(&p->packet_dev_state, g_free); g_free(p->normal); p->normal =3D NULL; g_free(p->zero); @@ -1158,6 +1181,34 @@ void multifd_recv_sync_main(void) trace_multifd_recv_sync_main(multifd_recv_state->packet_num); } =20 +static int multifd_device_state_recv(MultiFDRecvParams *p, Error **errp) +{ + g_autofree char *dev_state_buf =3D NULL; + int ret; + + dev_state_buf =3D g_malloc(p->next_packet_size); + + ret =3D qio_channel_read_all(p->c, dev_state_buf, p->next_packet_size,= errp); + if (ret !=3D 0) { + return ret; + } + + if (p->packet_dev_state->idstr[sizeof(p->packet_dev_state->idstr) - 1] + !=3D 0) { + error_setg(errp, "unterminated multifd device state idstr"); + return -1; + } + + if (!qemu_loadvm_load_state_buffer(p->packet_dev_state->idstr, + p->packet_dev_state->instance_id, + dev_state_buf, p->next_packet_size, + errp)) { + ret =3D -1; + } + + return ret; +} + static void *multifd_recv_thread(void *opaque) { MigrationState *s =3D migrate_get_current(); @@ -1176,6 +1227,7 @@ static void *multifd_recv_thread(void *opaque) while (true) { MultiFDPacketHdr_t hdr; uint32_t flags =3D 0; + bool is_device_state =3D false; bool has_data =3D false; uint8_t *pkt_buf; size_t pkt_len; @@ -1209,8 +1261,14 @@ static void *multifd_recv_thread(void *opaque) break; } =20 - pkt_buf =3D (uint8_t *)p->packet + sizeof(hdr); - pkt_len =3D p->packet_len - sizeof(hdr); + is_device_state =3D p->flags & MULTIFD_FLAG_DEVICE_STATE; + if (is_device_state) { + pkt_buf =3D (uint8_t *)p->packet_dev_state + sizeof(hdr); + pkt_len =3D sizeof(*p->packet_dev_state) - sizeof(hdr); + } else { + pkt_buf =3D (uint8_t *)p->packet + sizeof(hdr); + pkt_len =3D p->packet_len - sizeof(hdr); + } =20 ret =3D qio_channel_read_all_eof(p->c, (char *)pkt_buf, pkt_le= n, &local_err); @@ -1235,12 +1293,17 @@ static void *multifd_recv_thread(void *opaque) /* recv methods don't know how to handle the SYNC flag */ p->flags &=3D ~MULTIFD_FLAG_SYNC; =20 - /* - * Even if it's a SYNC packet, this needs to be set - * because older QEMUs (<9.0) still send data along with - * the SYNC packet. - */ - has_data =3D p->normal_num || p->zero_num; + if (is_device_state) { + has_data =3D p->next_packet_size > 0; + } else { + /* + * Even if it's a SYNC packet, this needs to be set + * because older QEMUs (<9.0) still send data along with + * the SYNC packet. + */ + has_data =3D p->normal_num || p->zero_num; + } + qemu_mutex_unlock(&p->mutex); } else { /* @@ -1269,14 +1332,29 @@ static void *multifd_recv_thread(void *opaque) } =20 if (has_data) { - ret =3D multifd_recv_state->ops->recv(p, &local_err); + if (is_device_state) { + assert(use_packets); + ret =3D multifd_device_state_recv(p, &local_err); + } else { + ret =3D multifd_recv_state->ops->recv(p, &local_err); + } if (ret !=3D 0) { break; } + } else if (is_device_state) { + error_setg(&local_err, + "multifd: received empty device state packet"); + break; } =20 if (use_packets) { if (flags & MULTIFD_FLAG_SYNC) { + if (is_device_state) { + error_setg(&local_err, + "multifd: received SYNC device state packet= "); + break; + } + qemu_sem_post(&multifd_recv_state->sem_sync); qemu_sem_wait(&p->sem_sync); } @@ -1345,6 +1423,7 @@ int multifd_recv_setup(Error **errp) p->packet_len =3D sizeof(MultiFDPacket_t) + sizeof(uint64_t) * page_count; p->packet =3D g_malloc0(p->packet_len); + p->packet_dev_state =3D g_malloc0(sizeof(*p->packet_dev_state)= ); } p->name =3D g_strdup_printf(MIGRATION_THREAD_DST_MULTIFD, i); p->normal =3D g_new0(ram_addr_t, page_count); --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270999; cv=none; d=zohomail.com; s=zohoarc; b=WdivySy1pDp3jLHxy0yDrITuiIbM88O9+CCKJpE+JZFsrudGXVdlv4xee1qjLZ9iusyATcmBetZkz9eDWW3TKUzc560bFD8H6dV5ycRmQSx6mAFC3VZwTkbIYrhXALg/rtoenOEhtIVjTn4JXZLg4aR1QLxG6ESgG2G1yQn8/Mk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270999; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=FNw/KaJyd7djxjrv3Hx0XbcUdwV3P/GfQA35U0gME/I=; b=hdinbutPTDiWA8gcsOVoVUzciYKPjcrKcb8fqN3FoFfddAA9fWEhRPZJKV3se9VvOAo9P5/o/OC5Zym1Hj2Aekas1py8GbhHhHh7N4ZZg1KMkCJHHZUIozvpQSFufQzZQ3sNXQVqlTbADOEaKVX2tQUQKrr0TRD9+EyqihrVr/I= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270999434733.8629703712901; Thu, 6 Mar 2025 06:23:19 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC1A-0008UJ-4P; Thu, 06 Mar 2025 09:16:08 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0f-0007RU-Co for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:37 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0d-0001pD-Gi for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:37 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-210-E2sSZGpxPy2Y5kLjsH0vQQ-1; Thu, 06 Mar 2025 09:15:32 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 6D8A41801A12; Thu, 6 Mar 2025 14:15:31 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6AD17180174E; Thu, 6 Mar 2025 14:15:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FNw/KaJyd7djxjrv3Hx0XbcUdwV3P/GfQA35U0gME/I=; b=FCbk6RGsU1FUle12Aklj9AtLDcMrQ5dM1SL1FKgApCXjYe+VVvcaUFsBqGEwbaDt/8dlZx LYsM6SF12/mjtOqW3Rxfd1KOnGN+OyF4BKHLek7Wyi6iEJSsZYPq4/nNTTRPdJf4H5jxNb UkGPf27ZWwUoCQ49G0ABVCCsfA1wVho= X-MC-Unique: E2sSZGpxPy2Y5kLjsH0vQQ-1 X-Mimecast-MFC-AGG-ID: E2sSZGpxPy2Y5kLjsH0vQQ_1741270531 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 21/42] migration/multifd: Make multifd_send() thread safe Date: Thu, 6 Mar 2025 15:13:57 +0100 Message-ID: <20250306141419.2015340-22-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271000558019000 From: "Maciej S. Szmigiero" multifd_send() function is currently not thread safe, make it thread safe by holding a lock during its execution. This way it will be possible to safely call it concurrently from multiple threads. Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/dd0f3bcc02ca96a7d523ca58ea69e495a3= 3b453b.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/multifd.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/migration/multifd.c b/migration/multifd.c index 01f427d8ed036417559a0573d369172f72b508e1..add6f86175c25c0f7925a919c26= 7240826fab54c 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -50,6 +50,10 @@ typedef struct { =20 struct { MultiFDSendParams *params; + + /* multifd_send() body is not thread safe, needs serialization */ + QemuMutex multifd_send_mutex; + /* * Global number of generated multifd packets. * @@ -339,6 +343,8 @@ bool multifd_send(MultiFDSendData **send_data) return false; } =20 + QEMU_LOCK_GUARD(&multifd_send_state->multifd_send_mutex); + /* We wait here, until at least one channel is ready */ qemu_sem_wait(&multifd_send_state->channels_ready); =20 @@ -507,6 +513,7 @@ static void multifd_send_cleanup_state(void) socket_cleanup_outgoing_migration(); qemu_sem_destroy(&multifd_send_state->channels_created); qemu_sem_destroy(&multifd_send_state->channels_ready); + qemu_mutex_destroy(&multifd_send_state->multifd_send_mutex); g_free(multifd_send_state->params); multifd_send_state->params =3D NULL; g_free(multifd_send_state); @@ -887,6 +894,7 @@ bool multifd_send_setup(void) thread_count =3D migrate_multifd_channels(); multifd_send_state =3D g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params =3D g_new0(MultiFDSendParams, thread_count); + qemu_mutex_init(&multifd_send_state->multifd_send_mutex); qemu_sem_init(&multifd_send_state->channels_created, 0); qemu_sem_init(&multifd_send_state->channels_ready, 0); qatomic_set(&multifd_send_state->exiting, 0); --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270932; cv=none; d=zohomail.com; s=zohoarc; b=my8Javo7fJWR4QRLdU9GxsQPQEL1mMNx02MG0PXPCDHKleXwCxnwgJfnsobAibWHKhUZIYIYUyqnYIq7tl8TT1UmWlY26jsqCBDlQI/ElAj0Ekbb4rUxNWez5nucyr/Ok0sKucw+AC7aWwP/s16HuDsfuzuzTsktS6G4/+7tevI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270932; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=IiyKTyQI8RufAMjHzPz249f8Jax8j1tG0nyxgwzWL7Y=; b=QoZO1BH6FZaWsEvVKxkMsk9edkUxhCDPDhb+hu1oCyL7tfirtoOTwucDhQGV3+s0Ev79MiA4rapsspHR5LQBBxUVUpMdCskK1L7x8ETRlbFyCJCnEIunWpuwvfLnHM+4007tHUgNqyV3hj5tsJTV9jRWuIAUDF9RppDqSYc3mgA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270932083539.1160208223673; Thu, 6 Mar 2025 06:22:12 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2F-0001IP-Sf; Thu, 06 Mar 2025 09:17:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0m-0007cv-St for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:45 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0k-0001q1-Sn for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:44 -0500 Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-332-RcEXq44iMzO8HELNl656VA-1; Thu, 06 Mar 2025 09:15:36 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0F4BE1955DB9; Thu, 6 Mar 2025 14:15:35 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CF16218001D3; Thu, 6 Mar 2025 14:15:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270542; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IiyKTyQI8RufAMjHzPz249f8Jax8j1tG0nyxgwzWL7Y=; b=VtCnbiSpHhyIdDCeUfSZSdYrw31OyvPS64R0JYcoGoTfH2g3AtsN2J68q0BUa/4a6v9UTj PiFxQEsGKxatnsIzfrHLi+RE/zo4pVUv8gz+lS8q9CZ1szziBmiADZ0gCj9Y3nXDOvPs1A C1mW4GmV/Q8Q2vRgAx+aO6mvP6pGhK0= X-MC-Unique: RcEXq44iMzO8HELNl656VA-1 X-Mimecast-MFC-AGG-ID: RcEXq44iMzO8HELNl656VA_1741270535 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 22/42] migration/multifd: Add an explicit MultiFDSendData destructor Date: Thu, 6 Mar 2025 15:13:58 +0100 Message-ID: <20250306141419.2015340-23-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270933430019100 From: "Maciej S. Szmigiero" This way if there are fields there that needs explicit disposal (like, for example, some attached buffers) they will be handled appropriately. Add a related assert to multifd_set_payload_type() in order to make sure that this function is only used to fill a previously empty MultiFDSendData with some payload, not the other way around. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/6755205f2b95abbed251f87061feee1c0e= 410836.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/multifd.h | 5 +++++ migration/multifd-nocomp.c | 3 +-- migration/multifd.c | 31 ++++++++++++++++++++++++++++--- 3 files changed, 34 insertions(+), 5 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index d682c5a9b74388bca7ea129d76646dc1e5fe1446..8d639eec69fecc4cf393abeb9d9= 87be854db2242 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -149,6 +149,9 @@ static inline bool multifd_payload_empty(MultiFDSendDat= a *data) static inline void multifd_set_payload_type(MultiFDSendData *data, MultiFDPayloadType type) { + assert(multifd_payload_empty(data)); + assert(type !=3D MULTIFD_PAYLOAD_NONE); + data->type =3D type; } =20 @@ -365,6 +368,8 @@ static inline void multifd_send_prepare_header(MultiFDS= endParams *p) void multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc); bool multifd_send(MultiFDSendData **send_data); MultiFDSendData *multifd_send_data_alloc(void); +void multifd_send_data_clear(MultiFDSendData *data); +void multifd_send_data_free(MultiFDSendData *data); =20 static inline uint32_t multifd_ram_page_size(void) { diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index 1325dba97cea876a94082e43107b99f428db7bcb..e46e79d8b272122c210bb895387= dfa133d218d46 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -42,8 +42,7 @@ void multifd_ram_save_setup(void) =20 void multifd_ram_save_cleanup(void) { - g_free(multifd_ram_send); - multifd_ram_send =3D NULL; + g_clear_pointer(&multifd_ram_send, multifd_send_data_free); } =20 static void multifd_set_file_bitmap(MultiFDSendParams *p) diff --git a/migration/multifd.c b/migration/multifd.c index add6f86175c25c0f7925a919c267240826fab54c..c8508cadab2d00c5c7f3e267660= 38e9f5a287dcc 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -123,6 +123,32 @@ MultiFDSendData *multifd_send_data_alloc(void) return g_malloc0(size_minus_payload + max_payload_size); } =20 +void multifd_send_data_clear(MultiFDSendData *data) +{ + if (multifd_payload_empty(data)) { + return; + } + + switch (data->type) { + default: + /* Nothing to do */ + break; + } + + data->type =3D MULTIFD_PAYLOAD_NONE; +} + +void multifd_send_data_free(MultiFDSendData *data) +{ + if (!data) { + return; + } + + multifd_send_data_clear(data); + + g_free(data); +} + static bool multifd_use_packets(void) { return !migrate_mapped_ram(); @@ -496,8 +522,7 @@ static bool multifd_send_cleanup_channel(MultiFDSendPar= ams *p, Error **errp) qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name =3D NULL; - g_free(p->data); - p->data =3D NULL; + g_clear_pointer(&p->data, multifd_send_data_free); p->packet_len =3D 0; g_free(p->packet); p->packet =3D NULL; @@ -695,7 +720,7 @@ static void *multifd_send_thread(void *opaque) (uint64_t)p->next_packet_size + p->packet_len); =20 p->next_packet_size =3D 0; - multifd_set_payload_type(p->data, MULTIFD_PAYLOAD_NONE); + multifd_send_data_clear(p->data); =20 /* * Making sure p->data is published before saying "we're --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270768; cv=none; d=zohomail.com; s=zohoarc; b=GtCEZKkHZFm5h/F6bzmLpByItK82bom5EeaDKh6zeHeAWqY9ZqkIed3iN+/3NqR1Dt+KRU1MMv3WYiwnIo7xOTq/RaAOrsNSypcJuxvRwnNOmLgz4n29/AJAWr5FgCvv6ixghFwKaEsVrO3JolYpYmMlMAS3kKEdtErKRnGUTwc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270768; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=fsq6RYHJYPgz1FwPZzTCEp8H4G4mPxLoeIS+HZrZY9M=; b=jsBeafT3aCJI6NCNeHnzU90M1qxeDk3jG8yfNCfpDSgqeQe1eEMdI1mU1kbW04fS4WJly1c/BZO9NCCQoZapVkQ4L9Kd2entQH7KtVdrp+q4ZpSb5gNH6amcNVkET9IpRrTLDGFgMrcAYCyYahsqY2dQOC3mBJrqFyepb8J0LOw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270768557728.4972508626576; Thu, 6 Mar 2025 06:19:28 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC1C-0008WO-Ki; Thu, 06 Mar 2025 09:16:11 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0t-0007nV-F8 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:52 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0q-0001qz-Ms for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:51 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-421-7yv6TTKWPJ6rSg0oXmIU4w-1; Thu, 06 Mar 2025 09:15:39 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 9BCFA19560B8; Thu, 6 Mar 2025 14:15:38 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 95E701801753; Thu, 6 Mar 2025 14:15:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270547; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fsq6RYHJYPgz1FwPZzTCEp8H4G4mPxLoeIS+HZrZY9M=; b=b7YdexGIonNxm9evzmQvKHnhUx+r2hv1fW4ajrSjek1U0mNM/Pbpkvaa/Dq0xtboe8nihV W9+3kvjPiY7upYkI3EheL12YTt0MtU7byRQ4L5j+erwGRQrT0hhLJnbJk1twUPt//mH0ff 6STfThK7Tv1Vtzyl+2dJQ7GIZMXCcik= X-MC-Unique: 7yv6TTKWPJ6rSg0oXmIU4w-1 X-Mimecast-MFC-AGG-ID: 7yv6TTKWPJ6rSg0oXmIU4w_1741270538 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 23/42] migration/multifd: Device state transfer support - send side Date: Thu, 6 Mar 2025 15:13:59 +0100 Message-ID: <20250306141419.2015340-24-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270769751019000 From: "Maciej S. Szmigiero" A new function multifd_queue_device_state() is provided for device to queue its state for transmission via a multifd channel. Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/ebd55768d3e5fecb5eb3f197bad9c0c07e= 5bc084.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/misc.h | 4 ++ migration/multifd.h | 34 ++++++--- migration/multifd-device-state.c | 118 +++++++++++++++++++++++++++++++ migration/multifd-nocomp.c | 14 +++- migration/multifd.c | 42 +++++++++-- migration/meson.build | 1 + 6 files changed, 197 insertions(+), 16 deletions(-) create mode 100644 migration/multifd-device-state.c diff --git a/include/migration/misc.h b/include/migration/misc.h index 4c171f4e897e2bea9016a1559bef3e89c165b176..bd3b725fa0b7efc4551aa19d2ac= 540f24bde4763 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -118,4 +118,8 @@ bool migrate_is_uri(const char *uri); bool migrate_uri_parse(const char *uri, MigrationChannel **channel, Error **errp); =20 +/* migration/multifd-device-state.c */ +bool multifd_queue_device_state(char *idstr, uint32_t instance_id, + char *data, size_t len); + #endif diff --git a/migration/multifd.h b/migration/multifd.h index 8d639eec69fecc4cf393abeb9d987be854db2242..aa679d8bbe8358e5cbab0add9c4= bf007fa55961b 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -127,13 +127,22 @@ struct MultiFDRecvData { off_t file_offset; }; =20 +typedef struct { + char *idstr; + uint32_t instance_id; + char *buf; + size_t buf_len; +} MultiFDDeviceState_t; + typedef enum { MULTIFD_PAYLOAD_NONE, MULTIFD_PAYLOAD_RAM, + MULTIFD_PAYLOAD_DEVICE_STATE, } MultiFDPayloadType; =20 typedef union MultiFDPayload { MultiFDPages_t ram; + MultiFDDeviceState_t device_state; } MultiFDPayload; =20 struct MultiFDSendData { @@ -146,6 +155,11 @@ static inline bool multifd_payload_empty(MultiFDSendDa= ta *data) return data->type =3D=3D MULTIFD_PAYLOAD_NONE; } =20 +static inline bool multifd_payload_device_state(MultiFDSendData *data) +{ + return data->type =3D=3D MULTIFD_PAYLOAD_DEVICE_STATE; +} + static inline void multifd_set_payload_type(MultiFDSendData *data, MultiFDPayloadType type) { @@ -198,8 +212,9 @@ typedef struct { =20 /* thread local variables. No locking required */ =20 - /* pointer to the packet */ + /* pointers to the possible packet types */ MultiFDPacket_t *packet; + MultiFDPacketDeviceState_t *packet_device_state; /* size of the next packet that contains pages */ uint32_t next_packet_size; /* packets sent through this channel */ @@ -358,13 +373,6 @@ bool multifd_send_prepare_common(MultiFDSendParams *p); void multifd_send_zero_page_detect(MultiFDSendParams *p); void multifd_recv_zero_page_process(MultiFDRecvParams *p); =20 -static inline void multifd_send_prepare_header(MultiFDSendParams *p) -{ - p->iov[0].iov_len =3D p->packet_len; - p->iov[0].iov_base =3D p->packet; - p->iovs_num++; -} - void multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc); bool multifd_send(MultiFDSendData **send_data); MultiFDSendData *multifd_send_data_alloc(void); @@ -389,4 +397,14 @@ bool multifd_ram_sync_per_section(void); size_t multifd_ram_payload_size(void); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); + +size_t multifd_device_state_payload_size(void); + +void multifd_send_data_clear_device_state(MultiFDDeviceState_t *device_sta= te); + +void multifd_device_state_send_setup(void); +void multifd_device_state_send_cleanup(void); + +void multifd_device_state_send_prepare(MultiFDSendParams *p); + #endif diff --git a/migration/multifd-device-state.c b/migration/multifd-device-st= ate.c new file mode 100644 index 0000000000000000000000000000000000000000..e383e75b1a024c1885b41bcc3ae= e034fd6f71f9d --- /dev/null +++ b/migration/multifd-device-state.c @@ -0,0 +1,118 @@ +/* + * Multifd device state migration + * + * Copyright (C) 2024,2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "qemu/lockable.h" +#include "migration/misc.h" +#include "multifd.h" + +static struct { + QemuMutex queue_job_mutex; + + MultiFDSendData *send_data; +} *multifd_send_device_state; + +size_t multifd_device_state_payload_size(void) +{ + return sizeof(MultiFDDeviceState_t); +} + +void multifd_device_state_send_setup(void) +{ + assert(!multifd_send_device_state); + multifd_send_device_state =3D g_malloc(sizeof(*multifd_send_device_sta= te)); + + qemu_mutex_init(&multifd_send_device_state->queue_job_mutex); + + multifd_send_device_state->send_data =3D multifd_send_data_alloc(); +} + +void multifd_device_state_send_cleanup(void) +{ + g_clear_pointer(&multifd_send_device_state->send_data, + multifd_send_data_free); + + qemu_mutex_destroy(&multifd_send_device_state->queue_job_mutex); + + g_clear_pointer(&multifd_send_device_state, g_free); +} + +void multifd_send_data_clear_device_state(MultiFDDeviceState_t *device_sta= te) +{ + g_clear_pointer(&device_state->idstr, g_free); + g_clear_pointer(&device_state->buf, g_free); +} + +static void multifd_device_state_fill_packet(MultiFDSendParams *p) +{ + MultiFDDeviceState_t *device_state =3D &p->data->u.device_state; + MultiFDPacketDeviceState_t *packet =3D p->packet_device_state; + + packet->hdr.flags =3D cpu_to_be32(p->flags); + strncpy(packet->idstr, device_state->idstr, sizeof(packet->idstr) - 1); + packet->idstr[sizeof(packet->idstr) - 1] =3D 0; + packet->instance_id =3D cpu_to_be32(device_state->instance_id); + packet->next_packet_size =3D cpu_to_be32(p->next_packet_size); +} + +static void multifd_prepare_header_device_state(MultiFDSendParams *p) +{ + p->iov[0].iov_len =3D sizeof(*p->packet_device_state); + p->iov[0].iov_base =3D p->packet_device_state; + p->iovs_num++; +} + +void multifd_device_state_send_prepare(MultiFDSendParams *p) +{ + MultiFDDeviceState_t *device_state =3D &p->data->u.device_state; + + assert(multifd_payload_device_state(p->data)); + + multifd_prepare_header_device_state(p); + + assert(!(p->flags & MULTIFD_FLAG_SYNC)); + + p->next_packet_size =3D device_state->buf_len; + if (p->next_packet_size > 0) { + p->iov[p->iovs_num].iov_base =3D device_state->buf; + p->iov[p->iovs_num].iov_len =3D p->next_packet_size; + p->iovs_num++; + } + + p->flags |=3D MULTIFD_FLAG_NOCOMP | MULTIFD_FLAG_DEVICE_STATE; + + multifd_device_state_fill_packet(p); +} + +bool multifd_queue_device_state(char *idstr, uint32_t instance_id, + char *data, size_t len) +{ + /* Device state submissions can come from multiple threads */ + QEMU_LOCK_GUARD(&multifd_send_device_state->queue_job_mutex); + MultiFDDeviceState_t *device_state; + + assert(multifd_payload_empty(multifd_send_device_state->send_data)); + + multifd_set_payload_type(multifd_send_device_state->send_data, + MULTIFD_PAYLOAD_DEVICE_STATE); + device_state =3D &multifd_send_device_state->send_data->u.device_state; + device_state->idstr =3D g_strdup(idstr); + device_state->instance_id =3D instance_id; + device_state->buf =3D g_memdup2(data, len); + device_state->buf_len =3D len; + + if (!multifd_send(&multifd_send_device_state->send_data)) { + multifd_send_data_clear(multifd_send_device_state->send_data); + return false; + } + + return true; +} diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index e46e79d8b272122c210bb895387dfa133d218d46..c00804652383ed0496769c134ea= 2c10e0ae91f18 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -14,6 +14,7 @@ #include "exec/ramblock.h" #include "exec/target_page.h" #include "file.h" +#include "migration-stats.h" #include "multifd.h" #include "options.h" #include "qapi/error.h" @@ -85,6 +86,13 @@ static void multifd_nocomp_send_cleanup(MultiFDSendParam= s *p, Error **errp) return; } =20 +static void multifd_ram_prepare_header(MultiFDSendParams *p) +{ + p->iov[0].iov_len =3D p->packet_len; + p->iov[0].iov_base =3D p->packet; + p->iovs_num++; +} + static void multifd_send_prepare_iovs(MultiFDSendParams *p) { MultiFDPages_t *pages =3D &p->data->u.ram; @@ -118,7 +126,7 @@ static int multifd_nocomp_send_prepare(MultiFDSendParam= s *p, Error **errp) * Only !zerocopy needs the header in IOV; zerocopy will * send it separately. */ - multifd_send_prepare_header(p); + multifd_ram_prepare_header(p); } =20 multifd_send_prepare_iovs(p); @@ -133,6 +141,8 @@ static int multifd_nocomp_send_prepare(MultiFDSendParam= s *p, Error **errp) if (ret !=3D 0) { return -1; } + + stat64_add(&mig_stats.multifd_bytes, p->packet_len); } =20 return 0; @@ -431,7 +441,7 @@ int multifd_ram_flush_and_sync(QEMUFile *f) bool multifd_send_prepare_common(MultiFDSendParams *p) { MultiFDPages_t *pages =3D &p->data->u.ram; - multifd_send_prepare_header(p); + multifd_ram_prepare_header(p); multifd_send_zero_page_detect(p); =20 if (!pages->normal_num) { diff --git a/migration/multifd.c b/migration/multifd.c index c8508cadab2d00c5c7f3e26766038e9f5a287dcc..3625c9a37c0e046b7a851dfe3d9= 4fefa6887534d 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -12,6 +12,7 @@ =20 #include "qemu/osdep.h" #include "qemu/cutils.h" +#include "qemu/iov.h" #include "qemu/rcu.h" #include "exec/target_page.h" #include "system/system.h" @@ -19,6 +20,7 @@ #include "qemu/error-report.h" #include "qapi/error.h" #include "file.h" +#include "migration/misc.h" #include "migration.h" #include "migration-stats.h" #include "savevm.h" @@ -111,7 +113,9 @@ MultiFDSendData *multifd_send_data_alloc(void) * added to the union in the future are larger than * (MultiFDPages_t + flex array). */ - max_payload_size =3D MAX(multifd_ram_payload_size(), sizeof(MultiFDPay= load)); + max_payload_size =3D MAX(multifd_ram_payload_size(), + multifd_device_state_payload_size()); + max_payload_size =3D MAX(max_payload_size, sizeof(MultiFDPayload)); =20 /* * Account for any holes the compiler might insert. We can't pack @@ -130,6 +134,9 @@ void multifd_send_data_clear(MultiFDSendData *data) } =20 switch (data->type) { + case MULTIFD_PAYLOAD_DEVICE_STATE: + multifd_send_data_clear_device_state(&data->u.device_state); + break; default: /* Nothing to do */ break; @@ -232,6 +239,7 @@ static int multifd_recv_initial_packet(QIOChannel *c, E= rror **errp) return msg.id; } =20 +/* Fills a RAM multifd packet */ void multifd_send_fill_packet(MultiFDSendParams *p) { MultiFDPacket_t *packet =3D p->packet; @@ -524,6 +532,7 @@ static bool multifd_send_cleanup_channel(MultiFDSendPar= ams *p, Error **errp) p->name =3D NULL; g_clear_pointer(&p->data, multifd_send_data_free); p->packet_len =3D 0; + g_clear_pointer(&p->packet_device_state, g_free); g_free(p->packet); p->packet =3D NULL; multifd_send_state->ops->send_cleanup(p, errp); @@ -536,6 +545,7 @@ static void multifd_send_cleanup_state(void) { file_cleanup_outgoing_migration(); socket_cleanup_outgoing_migration(); + multifd_device_state_send_cleanup(); qemu_sem_destroy(&multifd_send_state->channels_created); qemu_sem_destroy(&multifd_send_state->channels_ready); qemu_mutex_destroy(&multifd_send_state->multifd_send_mutex); @@ -694,16 +704,32 @@ static void *multifd_send_thread(void *opaque) * qatomic_store_release() in multifd_send(). */ if (qatomic_load_acquire(&p->pending_job)) { + bool is_device_state =3D multifd_payload_device_state(p->data); + size_t total_size; + p->flags =3D 0; p->iovs_num =3D 0; assert(!multifd_payload_empty(p->data)); =20 - ret =3D multifd_send_state->ops->send_prepare(p, &local_err); - if (ret !=3D 0) { - break; + if (is_device_state) { + multifd_device_state_send_prepare(p); + } else { + ret =3D multifd_send_state->ops->send_prepare(p, &local_er= r); + if (ret !=3D 0) { + break; + } } =20 + /* + * The packet header in the zerocopy RAM case is accounted for + * in multifd_nocomp_send_prepare() - where it is actually + * being sent. + */ + total_size =3D iov_size(p->iov, p->iovs_num); + if (migrate_mapped_ram()) { + assert(!is_device_state); + ret =3D file_write_ramblock_iov(p->c, p->iov, p->iovs_num, &p->data->u.ram, &local_err); } else { @@ -716,8 +742,7 @@ static void *multifd_send_thread(void *opaque) break; } =20 - stat64_add(&mig_stats.multifd_bytes, - (uint64_t)p->next_packet_size + p->packet_len); + stat64_add(&mig_stats.multifd_bytes, total_size); =20 p->next_packet_size =3D 0; multifd_send_data_clear(p->data); @@ -938,6 +963,9 @@ bool multifd_send_setup(void) p->packet_len =3D sizeof(MultiFDPacket_t) + sizeof(uint64_t) * page_count; p->packet =3D g_malloc0(p->packet_len); + p->packet_device_state =3D g_malloc0(sizeof(*p->packet_device_= state)); + p->packet_device_state->hdr.magic =3D cpu_to_be32(MULTIFD_MAGI= C); + p->packet_device_state->hdr.version =3D cpu_to_be32(MULTIFD_VE= RSION); } p->name =3D g_strdup_printf(MIGRATION_THREAD_SRC_MULTIFD, i); p->write_flags =3D 0; @@ -973,6 +1001,8 @@ bool multifd_send_setup(void) assert(p->iov); } =20 + multifd_device_state_send_setup(); + return true; =20 err: diff --git a/migration/meson.build b/migration/meson.build index d3bfe84d62045e757ee4cca3f5d0db9a87d3450c..9aa48b290e2a58533bcabdf3efd= 67313471fbd48 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -25,6 +25,7 @@ system_ss.add(files( 'migration-hmp-cmds.c', 'migration.c', 'multifd.c', + 'multifd-device-state.c', 'multifd-nocomp.c', 'multifd-zlib.c', 'multifd-zero-page.c', --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271036; cv=none; d=zohomail.com; s=zohoarc; b=Z+colk4BNIh54IhvSFxSOHoYG4T+q3TSKGOYDd3TvQdFnivC4A8RtVswJq1Xkq4QXC3UM1D/xyRR6PPbpgsx3jTXoE4upju/ynRcW1ady9XwY6kIpoAl2VVgXaZFuegAqe4/DCPfYMi/dGCwqR/JWm87lOwAWIXW6tiOYuhj8fg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271036; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=LkN4XUrq6jE3dyokE7nPL9/x74SW8PtHvl3wLbGC1Ns=; b=KaEW6XDL2hmPApNtJ7xa4XJ0w9BJEITPDkGB+V41lnmvOOP7kzdnPC0/NMTKHmD8/GUO5Z7rfHNHKIR294h5Kib5JPKtBoxXJYiUKNWvdY91QZAVsYl8BRG4SaM3dWMF7Pj7QSSL8+STTVl+e+LJnFvJWgJGdKRAKlf5ZeJXqvw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271036586720.7479555110065; Thu, 6 Mar 2025 06:23:56 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3D-0002Qm-60; Thu, 06 Mar 2025 09:18:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC10-0008Cv-VN for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:59 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0y-0001sv-4R for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:58 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-564-NZwlYBCqNGG2xeCXCnG01g-1; Thu, 06 Mar 2025 09:15:42 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 465C61800349; Thu, 6 Mar 2025 14:15:41 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 24A6C180174C; Thu, 6 Mar 2025 14:15:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LkN4XUrq6jE3dyokE7nPL9/x74SW8PtHvl3wLbGC1Ns=; b=DdxYooD9wnYazeyuxxwCHlTQQSLyDDQLhwW8cytwyPc6pKf59a4NL73CKwC1PpqmtmARq0 3zALTIKdwROKY2BImsN1Z34wOZ9fPh9acpXecaKn4SYV3RCCkY97WqsaWJYBpZL60VI8kQ +c2PSZCsu0J9nin+rB5vPqku49qDdPI= X-MC-Unique: NZwlYBCqNGG2xeCXCnG01g-1 X-Mimecast-MFC-AGG-ID: NZwlYBCqNGG2xeCXCnG01g_1741270541 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , Peter Xu , "Maciej S. Szmigiero" , Fabiano Rosas , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 24/42] migration/multifd: Make MultiFDSendData a struct Date: Thu, 6 Mar 2025 15:14:00 +0100 Message-ID: <20250306141419.2015340-25-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271038994019000 From: Peter Xu The newly introduced device state buffer can be used for either storing VFIO's read() raw data, but already also possible to store generic device states. After noticing that device states may not easily provide a max buffer size (also the fact that RAM MultiFDPages_t after all also want to have flexibility on managing offset[] array), it may not be a good idea to stick with union on MultiFDSendData.. as it won't play well with such flexibility. Switch MultiFDSendData to a struct. It won't consume a lot more space in reality, after all the real buffers were already dynamically allocated, so it's so far only about the two structs (pages, device_state) that will be duplicated, but they're small. With this, we can remove the pretty hard to understand alloc size logic. Because now we can allocate offset[] together with the SendData, and properly free it when the SendData is freed. [MSS: Make sure to clear possible device state payload before freeing MultiFDSendData, remove placeholders for other patches not included] Signed-off-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Acked-by: Fabiano Rosas Link: https://lore.kernel.org/qemu-devel/7b02baba8e6ddb23ef7c349d312b9b631d= b09d7e.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/multifd.h | 15 +++++++++------ migration/multifd-device-state.c | 5 ----- migration/multifd-nocomp.c | 13 ++++++------- migration/multifd.c | 25 +++++++------------------ 4 files changed, 22 insertions(+), 36 deletions(-) diff --git a/migration/multifd.h b/migration/multifd.h index aa679d8bbe8358e5cbab0add9c4bf007fa55961b..2d337e7b3b52fd005a6c342fb55= 2975e8f13bdf1 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -115,9 +115,13 @@ typedef struct { uint32_t num; /* number of normal pages */ uint32_t normal_num; + /* + * Pointer to the ramblock. NOTE: it's caller's responsibility to make + * sure the pointer is always valid! + */ RAMBlock *block; - /* offset of each page */ - ram_addr_t offset[]; + /* offset array of each page, managed by multifd */ + ram_addr_t *offset; } MultiFDPages_t; =20 struct MultiFDRecvData { @@ -140,7 +144,7 @@ typedef enum { MULTIFD_PAYLOAD_DEVICE_STATE, } MultiFDPayloadType; =20 -typedef union MultiFDPayload { +typedef struct MultiFDPayload { MultiFDPages_t ram; MultiFDDeviceState_t device_state; } MultiFDPayload; @@ -394,12 +398,11 @@ void multifd_ram_save_cleanup(void); int multifd_ram_flush_and_sync(QEMUFile *f); bool multifd_ram_sync_per_round(void); bool multifd_ram_sync_per_section(void); -size_t multifd_ram_payload_size(void); +void multifd_ram_payload_alloc(MultiFDPages_t *pages); +void multifd_ram_payload_free(MultiFDPages_t *pages); void multifd_ram_fill_packet(MultiFDSendParams *p); int multifd_ram_unfill_packet(MultiFDRecvParams *p, Error **errp); =20 -size_t multifd_device_state_payload_size(void); - void multifd_send_data_clear_device_state(MultiFDDeviceState_t *device_sta= te); =20 void multifd_device_state_send_setup(void); diff --git a/migration/multifd-device-state.c b/migration/multifd-device-st= ate.c index e383e75b1a024c1885b41bcc3aee034fd6f71f9d..64d8ca180167b09abf13c930232= 6404be236717e 100644 --- a/migration/multifd-device-state.c +++ b/migration/multifd-device-state.c @@ -20,11 +20,6 @@ static struct { MultiFDSendData *send_data; } *multifd_send_device_state; =20 -size_t multifd_device_state_payload_size(void) -{ - return sizeof(MultiFDDeviceState_t); -} - void multifd_device_state_send_setup(void) { assert(!multifd_send_device_state); diff --git a/migration/multifd-nocomp.c b/migration/multifd-nocomp.c index c00804652383ed0496769c134ea2c10e0ae91f18..ffe75256c9fbe459dee4eb4aa8f= 80ae013ff2d87 100644 --- a/migration/multifd-nocomp.c +++ b/migration/multifd-nocomp.c @@ -25,15 +25,14 @@ =20 static MultiFDSendData *multifd_ram_send; =20 -size_t multifd_ram_payload_size(void) +void multifd_ram_payload_alloc(MultiFDPages_t *pages) { - uint32_t n =3D multifd_ram_page_count(); + pages->offset =3D g_new0(ram_addr_t, multifd_ram_page_count()); +} =20 - /* - * We keep an array of page offsets at the end of MultiFDPages_t, - * add space for it in the allocation. - */ - return sizeof(MultiFDPages_t) + n * sizeof(ram_addr_t); +void multifd_ram_payload_free(MultiFDPages_t *pages) +{ + g_clear_pointer(&pages->offset, g_free); } =20 void multifd_ram_save_setup(void) diff --git a/migration/multifd.c b/migration/multifd.c index 3625c9a37c0e046b7a851dfe3d94fefa6887534d..dfb5189f0ea3da97370b81e2dcf= 5a7f86252ae29 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -105,26 +105,12 @@ struct { =20 MultiFDSendData *multifd_send_data_alloc(void) { - size_t max_payload_size, size_minus_payload; + MultiFDSendData *new =3D g_new0(MultiFDSendData, 1); =20 - /* - * MultiFDPages_t has a flexible array at the end, account for it - * when allocating MultiFDSendData. Use max() in case other types - * added to the union in the future are larger than - * (MultiFDPages_t + flex array). - */ - max_payload_size =3D MAX(multifd_ram_payload_size(), - multifd_device_state_payload_size()); - max_payload_size =3D MAX(max_payload_size, sizeof(MultiFDPayload)); - - /* - * Account for any holes the compiler might insert. We can't pack - * the structure because that misaligns the members and triggers - * Waddress-of-packed-member. - */ - size_minus_payload =3D sizeof(MultiFDSendData) - sizeof(MultiFDPayload= ); + multifd_ram_payload_alloc(&new->u.ram); + /* Device state allocates its payload on-demand */ =20 - return g_malloc0(size_minus_payload + max_payload_size); + return new; } =20 void multifd_send_data_clear(MultiFDSendData *data) @@ -151,8 +137,11 @@ void multifd_send_data_free(MultiFDSendData *data) return; } =20 + /* This also free's device state payload */ multifd_send_data_clear(data); =20 + multifd_ram_payload_free(&data->u.ram); + g_free(data); } =20 --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270768; cv=none; d=zohomail.com; s=zohoarc; b=QBa7u+/5xfcQrhQJvCks5ku3EBhVCESt7o91sBBambi9+IUeCCFAvrdc0anxADawbqDmTVirjK6gieaK/FTElRP+P0n3e77xDbT05hZYpBOwjzq02a4lv7oMq0l0Rov7o9+Cs3xOjbPYYI28TNusCHYIOV/6w48PIKKllfz9Nu0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270768; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4qMRgHQOMtuGn9LMz4mIxSzpfeNWUHwblusGaJqRnNg=; b=MZi56k13KoaD+dVXlSFA1NqkWhTk6I6URarG8Ox6VEAH72DiAk8Q/7o9hI0NpNblMdDFg/pdUFL4GQDCA/t/7w0igkU83HwsWk2kjwMeRmp7Fz7jdosaXlV6mcNMGgSurLi1ziGzVDNzixxaFTaYBFLMNh9tg37nm2kGfiE9WeE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270768394834.5095482040682; Thu, 6 Mar 2025 06:19:28 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2J-0001lq-Nt; Thu, 06 Mar 2025 09:17:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC10-00089r-NU for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:58 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0y-0001st-8T for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:58 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-515-P-xAzp9bPkOpbXLVoffkdw-1; Thu, 06 Mar 2025 09:15:45 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 17D7A180AF73; Thu, 6 Mar 2025 14:15:44 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C6CC518001D3; Thu, 6 Mar 2025 14:15:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270555; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4qMRgHQOMtuGn9LMz4mIxSzpfeNWUHwblusGaJqRnNg=; b=KPKA6zBXXiV1SPMX7DDLED3d0aDSv1BDp6K6CVTSEEi8G10heb2HpjEUYPOOSE5cQ9oETa aIEEhOCw82ufQk93ralFnPQQjz1Oi5JsjCJJX98ntPFUMPaXJUQuBT5sz7a0Lh+ZsXq0g9 738if1LrpcMrTnov7qPeDozwv/cD1q8= X-MC-Unique: P-xAzp9bPkOpbXLVoffkdw-1 X-Mimecast-MFC-AGG-ID: P-xAzp9bPkOpbXLVoffkdw_1741270544 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 25/42] migration/multifd: Add multifd_device_state_supported() Date: Thu, 6 Mar 2025 15:14:01 +0100 Message-ID: <20250306141419.2015340-26-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270769515019000 From: "Maciej S. Szmigiero" Since device state transfer via multifd channels requires multifd channels with packets and is currently not compatible with multifd compression add an appropriate query function so device can learn whether it can actually make use of it. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/1ff0d98b85f470e5a33687406e877583b8= fab74e.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/misc.h | 1 + migration/multifd-device-state.c | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/include/migration/misc.h b/include/migration/misc.h index bd3b725fa0b7efc4551aa19d2ac540f24bde4763..273ebfca6256265507cd6aa28a1= 04a3db1ac106b 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -121,5 +121,6 @@ bool migrate_uri_parse(const char *uri, MigrationChanne= l **channel, /* migration/multifd-device-state.c */ bool multifd_queue_device_state(char *idstr, uint32_t instance_id, char *data, size_t len); +bool multifd_device_state_supported(void); =20 #endif diff --git a/migration/multifd-device-state.c b/migration/multifd-device-st= ate.c index 64d8ca180167b09abf13c9302326404be236717e..3097ffa31025971bb9b893c53da= d352d517148d9 100644 --- a/migration/multifd-device-state.c +++ b/migration/multifd-device-state.c @@ -13,6 +13,7 @@ #include "qemu/lockable.h" #include "migration/misc.h" #include "multifd.h" +#include "options.h" =20 static struct { QemuMutex queue_job_mutex; @@ -111,3 +112,9 @@ bool multifd_queue_device_state(char *idstr, uint32_t i= nstance_id, =20 return true; } + +bool multifd_device_state_supported(void) +{ + return migrate_multifd() && !migrate_mapped_ram() && + migrate_multifd_compression() =3D=3D MULTIFD_COMPRESSION_NONE; +} --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271200; cv=none; d=zohomail.com; s=zohoarc; b=FKHkf+LxJTkvuu0t3folOk35hPVZYrmKHWmMqVUp6OIkTqkCbwqerkE0L+CI5pLRSmrSs9SCNp6PA683opb52p8as8gWo8IK7a1OS1hf/fDe5V9jnrS2UFjVw2me/Q21223TgAWOp40xVb3t2cqOavrl9dkdGOEko2sgbZNh91c= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271200; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=A7lprOMmLSeSMFwax8dV7y5sGDyWfHydF1R9GGLiQ90=; b=iq05QWlTlrQ6vee3wTaeJ0xiUc66FyNwK9e7kbErN+tYV+fNt2jJ0AOnLi+Jrc7U6OINRJSiE1Zr3ihhYxICfgPwP8+d2HYOXQlN9nDWQD/kRbpfGrqDRViqlyF2M19BYTYRUN9DNmKELy/h5fe7m+zJ2IYfNoZ52ukrf0bK1cA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271200698532.6221823887412; Thu, 6 Mar 2025 06:26:40 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2K-0001wv-TJ; Thu, 06 Mar 2025 09:17:22 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC12-0008JR-6a for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:02 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0z-0001tM-Qc for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:59 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-678-ID13grh_Oyue_d_OYy_96A-1; Thu, 06 Mar 2025 09:15:47 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 12110180025B; Thu, 6 Mar 2025 14:15:47 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 93F261801748; Thu, 6 Mar 2025 14:15:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270557; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A7lprOMmLSeSMFwax8dV7y5sGDyWfHydF1R9GGLiQ90=; b=FkstkCDvB58BSangxzIiIyeEx/wAuEoLTpChakgQrXgCO7kToOHUY3i2B0ONqZLAlc9LhD dYj59vtZbtUm4NxDmzNFNUsYkbzls8YspELSWcJjmJ/IDNLccCDzUjYQvKDpRO17RyGHY1 lCtYV5ztkNYZMf6EXvO/u5NuRk0XFjM= X-MC-Unique: ID13grh_Oyue_d_OYy_96A-1 X-Mimecast-MFC-AGG-ID: ID13grh_Oyue_d_OYy_96A_1741270547 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 26/42] migration: Add save_live_complete_precopy_thread handler Date: Thu, 6 Mar 2025 15:14:02 +0100 Message-ID: <20250306141419.2015340-27-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271202685019100 From: "Maciej S. Szmigiero" This SaveVMHandler helps device provide its own asynchronous transmission of the remaining data at the end of a precopy phase via multifd channels, in parallel with the transfer done by save_live_complete_precopy handlers. These threads are launched only when multifd device state transfer is supported. Management of these threads in done in the multifd migration code, wrapping them in the generic thread pool. Signed-off-by: Maciej S. Szmigiero Reviewed-by: Peter Xu Link: https://lore.kernel.org/qemu-devel/eac74a4ca7edd8968bbf72aa07b9041c76= 364a16.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/migration/misc.h | 17 ++++++ include/migration/register.h | 19 +++++++ include/qemu/typedefs.h | 3 ++ migration/multifd-device-state.c | 92 ++++++++++++++++++++++++++++++++ migration/savevm.c | 40 +++++++++++++- 5 files changed, 170 insertions(+), 1 deletion(-) diff --git a/include/migration/misc.h b/include/migration/misc.h index 273ebfca6256265507cd6aa28a104a3db1ac106b..8fd36eba1da72004bbc01cf1804= 3180b21e10e41 100644 --- a/include/migration/misc.h +++ b/include/migration/misc.h @@ -119,8 +119,25 @@ bool migrate_uri_parse(const char *uri, MigrationChann= el **channel, Error **errp); =20 /* migration/multifd-device-state.c */ +typedef struct SaveLiveCompletePrecopyThreadData { + SaveLiveCompletePrecopyThreadHandler hdlr; + char *idstr; + uint32_t instance_id; + void *handler_opaque; +} SaveLiveCompletePrecopyThreadData; + bool multifd_queue_device_state(char *idstr, uint32_t instance_id, char *data, size_t len); bool multifd_device_state_supported(void); =20 +void +multifd_spawn_device_state_save_thread(SaveLiveCompletePrecopyThreadHandle= r hdlr, + char *idstr, uint32_t instance_id, + void *opaque); + +bool multifd_device_state_save_thread_should_exit(void); + +void multifd_abort_device_state_save_threads(void); +bool multifd_join_device_state_save_threads(void); + #endif diff --git a/include/migration/register.h b/include/migration/register.h index 58891aa54b76cccdea9e442be9bf528a41593e26..c041ce32f2fcf0e0a102ee5a5d6= 9ec0a84c5feac 100644 --- a/include/migration/register.h +++ b/include/migration/register.h @@ -105,6 +105,25 @@ typedef struct SaveVMHandlers { */ int (*save_live_complete_precopy)(QEMUFile *f, void *opaque); =20 + /** + * @save_live_complete_precopy_thread (invoked in a separate thread) + * + * Called at the end of a precopy phase from a separate worker thread + * in configurations where multifd device state transfer is supported + * in order to perform asynchronous transmission of the remaining data= in + * parallel with @save_live_complete_precopy handlers. + * When postcopy is enabled, devices that support postcopy will skip t= his + * step. + * + * @d: a #SaveLiveCompletePrecopyThreadData containing parameters that= the + * handler may need, including this device section idstr and instance_= id, + * and opaque data pointer passed to register_savevm_live(). + * @errp: pointer to Error*, to store an error if it happens. + * + * Returns true to indicate success and false for errors. + */ + SaveLiveCompletePrecopyThreadHandler save_live_complete_precopy_thread; + /* This runs both outside and inside the BQL. */ =20 /** diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h index fd23ff7771b1bd6cd51f48bbc5be86aa57982307..42ed4e6be150acc39c842e1158c= b87f3a8f5662d 100644 --- a/include/qemu/typedefs.h +++ b/include/qemu/typedefs.h @@ -108,6 +108,7 @@ typedef struct QString QString; typedef struct RAMBlock RAMBlock; typedef struct Range Range; typedef struct ReservedRegion ReservedRegion; +typedef struct SaveLiveCompletePrecopyThreadData SaveLiveCompletePrecopyTh= readData; typedef struct SHPCDevice SHPCDevice; typedef struct SSIBus SSIBus; typedef struct TCGCPUOps TCGCPUOps; @@ -133,5 +134,7 @@ typedef struct IRQState *qemu_irq; typedef void (*qemu_irq_handler)(void *opaque, int n, int level); typedef bool (*MigrationLoadThread)(void *opaque, bool *should_quit, Error **errp); +typedef bool (*SaveLiveCompletePrecopyThreadHandler)(SaveLiveCompletePreco= pyThreadData *d, + Error **errp); =20 #endif /* QEMU_TYPEDEFS_H */ diff --git a/migration/multifd-device-state.c b/migration/multifd-device-st= ate.c index 3097ffa31025971bb9b893c53dad352d517148d9..94222d0eb0d8f4dd0de0f5e5d6d= 742e9b3b6edf5 100644 --- a/migration/multifd-device-state.c +++ b/migration/multifd-device-state.c @@ -10,7 +10,10 @@ */ =20 #include "qemu/osdep.h" +#include "qapi/error.h" #include "qemu/lockable.h" +#include "block/thread-pool.h" +#include "migration.h" #include "migration/misc.h" #include "multifd.h" #include "options.h" @@ -19,6 +22,9 @@ static struct { QemuMutex queue_job_mutex; =20 MultiFDSendData *send_data; + + ThreadPool *threads; + bool threads_abort; } *multifd_send_device_state; =20 void multifd_device_state_send_setup(void) @@ -29,10 +35,14 @@ void multifd_device_state_send_setup(void) qemu_mutex_init(&multifd_send_device_state->queue_job_mutex); =20 multifd_send_device_state->send_data =3D multifd_send_data_alloc(); + + multifd_send_device_state->threads =3D thread_pool_new(); + multifd_send_device_state->threads_abort =3D false; } =20 void multifd_device_state_send_cleanup(void) { + g_clear_pointer(&multifd_send_device_state->threads, thread_pool_free); g_clear_pointer(&multifd_send_device_state->send_data, multifd_send_data_free); =20 @@ -118,3 +128,85 @@ bool multifd_device_state_supported(void) return migrate_multifd() && !migrate_mapped_ram() && migrate_multifd_compression() =3D=3D MULTIFD_COMPRESSION_NONE; } + +static void multifd_device_state_save_thread_data_free(void *opaque) +{ + SaveLiveCompletePrecopyThreadData *data =3D opaque; + + g_clear_pointer(&data->idstr, g_free); + g_free(data); +} + +static int multifd_device_state_save_thread(void *opaque) +{ + SaveLiveCompletePrecopyThreadData *data =3D opaque; + g_autoptr(Error) local_err =3D NULL; + + if (!data->hdlr(data, &local_err)) { + MigrationState *s =3D migrate_get_current(); + + /* + * Can't call abort_device_state_save_threads() here since new + * save threads could still be in process of being launched + * (if, for example, the very first save thread launched exited + * with an error very quickly). + */ + + assert(local_err); + + /* + * In case of multiple save threads failing which thread error + * return we end setting is purely arbitrary. + */ + migrate_set_error(s, local_err); + } + + return 0; +} + +bool multifd_device_state_save_thread_should_exit(void) +{ + return qatomic_read(&multifd_send_device_state->threads_abort); +} + +void +multifd_spawn_device_state_save_thread(SaveLiveCompletePrecopyThreadHandle= r hdlr, + char *idstr, uint32_t instance_id, + void *opaque) +{ + SaveLiveCompletePrecopyThreadData *data; + + assert(multifd_device_state_supported()); + assert(multifd_send_device_state); + + assert(!qatomic_read(&multifd_send_device_state->threads_abort)); + + data =3D g_new(SaveLiveCompletePrecopyThreadData, 1); + data->hdlr =3D hdlr; + data->idstr =3D g_strdup(idstr); + data->instance_id =3D instance_id; + data->handler_opaque =3D opaque; + + thread_pool_submit_immediate(multifd_send_device_state->threads, + multifd_device_state_save_thread, + data, + multifd_device_state_save_thread_data_fre= e); +} + +void multifd_abort_device_state_save_threads(void) +{ + assert(multifd_device_state_supported()); + + qatomic_set(&multifd_send_device_state->threads_abort, true); +} + +bool multifd_join_device_state_save_threads(void) +{ + MigrationState *s =3D migrate_get_current(); + + assert(multifd_device_state_supported()); + + thread_pool_wait(multifd_send_device_state->threads); + + return !migrate_has_error(s); +} diff --git a/migration/savevm.c b/migration/savevm.c index 1abc365570e324dd85f8a95adeb1a95f57b73264..5c4fdfd95eebf8d52fa02f23430= 90e4bac6c2d73 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -37,6 +37,7 @@ #include "migration/register.h" #include "migration/global_state.h" #include "migration/channel-block.h" +#include "multifd.h" #include "ram.h" #include "qemu-file.h" #include "savevm.h" @@ -1527,6 +1528,24 @@ int qemu_savevm_state_complete_precopy_iterable(QEMU= File *f, bool in_postcopy) int64_t start_ts_each, end_ts_each; SaveStateEntry *se; int ret; + bool multifd_device_state =3D multifd_device_state_supported(); + + if (multifd_device_state) { + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { + SaveLiveCompletePrecopyThreadHandler hdlr; + + if (!se->ops || (in_postcopy && se->ops->has_postcopy && + se->ops->has_postcopy(se->opaque)) || + !se->ops->save_live_complete_precopy_thread) { + continue; + } + + hdlr =3D se->ops->save_live_complete_precopy_thread; + multifd_spawn_device_state_save_thread(hdlr, + se->idstr, se->instance= _id, + se->opaque); + } + } =20 QTAILQ_FOREACH(se, &savevm_state.handlers, entry) { if (!se->ops || @@ -1552,16 +1571,35 @@ int qemu_savevm_state_complete_precopy_iterable(QEM= UFile *f, bool in_postcopy) save_section_footer(f, se); if (ret < 0) { qemu_file_set_error(f, ret); - return -1; + goto ret_fail_abort_threads; } end_ts_each =3D qemu_clock_get_us(QEMU_CLOCK_REALTIME); trace_vmstate_downtime_save("iterable", se->idstr, se->instance_id, end_ts_each - start_ts_each); } =20 + if (multifd_device_state) { + if (migrate_has_error(migrate_get_current())) { + multifd_abort_device_state_save_threads(); + } + + if (!multifd_join_device_state_save_threads()) { + qemu_file_set_error(f, -EINVAL); + return -1; + } + } + trace_vmstate_downtime_checkpoint("src-iterable-saved"); =20 return 0; + +ret_fail_abort_threads: + if (multifd_device_state) { + multifd_abort_device_state_save_threads(); + multifd_join_device_state_save_threads(); + } + + return -1; } =20 int qemu_savevm_state_complete_precopy_non_iterable(QEMUFile *f, --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271031; cv=none; d=zohomail.com; s=zohoarc; b=RYHCBbCvh+IfnUDGyXg5EtI1obRKZPLcjk6LPPyoiNe/km0NDzQ/PqA8iyo/IGtQNBxtlaSnvCbUSc5HXu8XRCysdOSyGMq1R6mVqoTPen7vqLteNqOzHNeOvYjFyF2oqXJDB+F7ySdEKEwwfUCdiUc/ULW9MDFkV95asirnpRM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271031; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ggH3+7adeq8R4LD9PInlmJ5WYrvAy9ls9kmNXg6IBiE=; b=QUaHZH4UNrDWb4UaBpgSE6YqJ9B09g3BJwLgmYSEIbCbQGy63d8iphXyTW76n6RJu7p3AOeZjDNJN7pCYGqwGYnVs4HPhb5Y4QxO+UznRHG0DjYUtJX1OsVu9eXwYuucLpBPqH1ZDg0VNZ+i4jxZ2aCUZHYuY+gJC5C6+fJkuxo= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271031792775.0533031254836; Thu, 6 Mar 2025 06:23:51 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC1p-0000sE-3V; Thu, 06 Mar 2025 09:16:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0z-000858-4C for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:57 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0x-0001sS-A8 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:56 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-73-Cl-qnIg-MAOUoCsJa4NnDg-1; Thu, 06 Mar 2025 09:15:50 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 50E58180AF5A; Thu, 6 Mar 2025 14:15:49 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9E20718001D3; Thu, 6 Mar 2025 14:15:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270554; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ggH3+7adeq8R4LD9PInlmJ5WYrvAy9ls9kmNXg6IBiE=; b=CSAzs40qrxOJGhEk7QjULxF4xDHsZcqt+PyhDY8R7g6Q+3dpls46xCjync8+qWCCZT8m4s Ky+CihZlPlhP2YQqcIN6ny+ok/HYzsNTfV3Gs/q/J7Y32vDNbVeOtQzdT6jHSUSXgDgpu6 0UY4Fc4QFtje7sW7OK+3F8Q8Qbhib2I= X-MC-Unique: Cl-qnIg-MAOUoCsJa4NnDg-1 X-Mimecast-MFC-AGG-ID: Cl-qnIg-MAOUoCsJa4NnDg_1741270549 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 27/42] vfio/migration: Add load_device_config_state_start trace event Date: Thu, 6 Mar 2025 15:14:03 +0100 Message-ID: <20250306141419.2015340-28-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271033801019100 From: "Maciej S. Szmigiero" And rename existing load_device_config_state trace event to load_device_config_state_end for consistency since it is triggered at the end of loading of the VFIO device config state. This way both the start and end points of particular device config loading operation (a long, BQL-serialized operation) are known. Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/1b6c5a2097e64c272eb7e53f9e4cca4b79= 581b38.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration.c | 4 +++- hw/vfio/trace-events | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index adfa752db5272e37d73fc0a435a0834e74e3f2fe..03890eaa48a9b36a9b934b5bb6d= 4814fadf65a8b 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -285,6 +285,8 @@ static int vfio_load_device_config_state(QEMUFile *f, v= oid *opaque) VFIODevice *vbasedev =3D opaque; uint64_t data; =20 + trace_vfio_load_device_config_state_start(vbasedev->name); + if (vbasedev->ops && vbasedev->ops->vfio_load_config) { int ret; =20 @@ -303,7 +305,7 @@ static int vfio_load_device_config_state(QEMUFile *f, v= oid *opaque) return -EINVAL; } =20 - trace_vfio_load_device_config_state(vbasedev->name); + trace_vfio_load_device_config_state_end(vbasedev->name); return qemu_file_get_error(f); } =20 diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index c5385e1a4f98ea703c433f9923a37a776fe63e78..a02c668f28a43f73ed2db9f1582= 7f26fed0994c3 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -150,7 +150,8 @@ vfio_display_edid_write_error(void) "" =20 # migration.c vfio_load_cleanup(const char *name) " (%s)" -vfio_load_device_config_state(const char *name) " (%s)" +vfio_load_device_config_state_start(const char *name) " (%s)" +vfio_load_device_config_state_end(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret)= " (%s) size %"PRIu64" ret %d" vfio_migration_realize(const char *name) " (%s)" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270727; cv=none; d=zohomail.com; s=zohoarc; b=ivsvNMizUOGLRcLojLYIr0q4cm/CmsNwDjYNqBKVwFk/XiUv0j9e+qPgqAs+Ybp8BHtdlFLJRS1TfIwC9wgm33SUlP82ymZXWocK7b02KZjvAKWWSg7tcaad+ufDvK5PlVnXwu4yE1uwwRtc+QDBfWrZGfBuE1jAJu7QrVjxp70= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270727; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=E6L+WpHb5duQkrD2m5irW1vavd9Ca4RFcgsWm+X6qFY=; b=ZgVvoQ7afDK7Ll4tD66Y6ZxpjVOz9l9/vkfiWJAf0sM6nD/M8eDfBich5zizXDy3dpSDnyPL3C+DPbR5vasRTgEZCPUe4JMaMuMZw0m/uoz3wpE4m2XXrBQqxyKQCqZAdmad8MIt+kQkncEPbu8P629pTJF5dviDlbZGobeE8dg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 174127072779394.36526120482449; Thu, 6 Mar 2025 06:18:47 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2J-0001m0-W1; Thu, 06 Mar 2025 09:17:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0z-00085e-Bd for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:57 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC0x-0001sP-Li for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:15:57 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-533-JwIUFvReMl6GKNnml96Usg-1; Thu, 06 Mar 2025 09:15:52 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B298618001DE; Thu, 6 Mar 2025 14:15:51 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id CB39C18001D3; Thu, 6 Mar 2025 14:15:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270553; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E6L+WpHb5duQkrD2m5irW1vavd9Ca4RFcgsWm+X6qFY=; b=Y4Wabgt9MDnvuXLT1PJ08wplExkfTapqu1T/s6+hxrjofqsRhQYuTOaZxZtggxgKoxMZik 892izngJbWhl+SSbckxNBwLL/oTwdMN3XNZtEy5CK5+xjiD5PP3XuaTysnbKjzTY2T+fUf V72FoWw7E0y/k4/yE49cSBDKBme6SjQ= X-MC-Unique: JwIUFvReMl6GKNnml96Usg-1 X-Mimecast-MFC-AGG-ID: JwIUFvReMl6GKNnml96Usg_1741270551 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 28/42] vfio/migration: Convert bytes_transferred counter to atomic Date: Thu, 6 Mar 2025 15:14:04 +0100 Message-ID: <20250306141419.2015340-29-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270729347019000 From: "Maciej S. Szmigiero" So it can be safety accessed from multiple threads. This variable type needs to be changed to unsigned long since 32-bit host platforms lack the necessary addition atomics on 64-bit variables. Using 32-bit counters on 32-bit host platforms should not be a problem in practice since they can't realistically address more memory anyway. Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/dc391771d2d9ad0f311994f0cb9e666da5= 64aeaf.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 03890eaa48a9b36a9b934b5bb6d4814fadf65a8b..5532787be63b9925a70202536a7= f5796ea1fc8d1 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -55,7 +55,7 @@ */ #define VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE (1 * MiB) =20 -static int64_t bytes_transferred; +static unsigned long bytes_transferred; =20 static const char *mig_state_to_str(enum vfio_device_mig_state state) { @@ -391,7 +391,7 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigrati= on *migration) qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); qemu_put_be64(f, data_size); qemu_put_buffer(f, migration->data_buffer, data_size); - bytes_transferred +=3D data_size; + qatomic_add(&bytes_transferred, data_size); =20 trace_vfio_save_block(migration->vbasedev->name, data_size); =20 @@ -1013,12 +1013,12 @@ static int vfio_block_migration(VFIODevice *vbasede= v, Error *err, Error **errp) =20 int64_t vfio_mig_bytes_transferred(void) { - return bytes_transferred; + return MIN(qatomic_read(&bytes_transferred), INT64_MAX); } =20 void vfio_reset_bytes_transferred(void) { - bytes_transferred =3D 0; + qatomic_set(&bytes_transferred, 0); } =20 /* --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271257; cv=none; d=zohomail.com; s=zohoarc; b=bN33laNSj9Jc7JkzKD9rIPg+EoMRRcIpSpVG4vpdDkepK8C6doLc0vv8a4kaIgV1+7KrO/WZPJ8XBRAvwJyu6DLotiy4BeiV/Kns4D/66WmMSbT3APxq0B0vm5EKl9u5nPkNagY7ZvgBzw+KNy18Bgep9Mfamr3mt2hk+LKM8VI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271257; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=RbOAsJqzSFDg+zHRkVpv2sHEQw8t+X1jLkRex3WYLHU=; b=DhlkdxB+x/q3uBKbc358XTSTn7a8GdQZcEZz3BtmktBQU2ImB4cX8z64yZuGw7f6gbu6gWAJ4oixaat8QsagOgKoQ3spXJxJZVtsUpCs249jk8FtLYQp9IGWloR4rBDbVH6MVy8QlKtPGrzjY7FDLSarP3RUcqoaq71W+19Mk6g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271257854743.4615000771909; Thu, 6 Mar 2025 06:27:37 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3E-0002RF-7D; Thu, 06 Mar 2025 09:18:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC16-0008S2-4P for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:04 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC14-0001uS-F7 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:03 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-688-UKXHTFaAOteoQ4n2Dva3jA-1; Thu, 06 Mar 2025 09:15:57 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EE84119560A1; Thu, 6 Mar 2025 14:15:54 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 42D441801748; Thu, 6 Mar 2025 14:15:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RbOAsJqzSFDg+zHRkVpv2sHEQw8t+X1jLkRex3WYLHU=; b=D+e1SL/xPiBwcNi/VTcSBllwXW14dQQgcdyXRDi/QMwqalXLb5ZdQnhUMqBiPa+2PZEf3w SSIolBHrbt4cn8SQsN0pRfxyjqxbQNJO216g4SAeVxEhVTQvTxVB9RskTq0cHbUe6PDubj kfwPuF5PB7IQHXap/PFtwHoszMYE180= X-MC-Unique: UKXHTFaAOteoQ4n2Dva3jA-1 X-Mimecast-MFC-AGG-ID: UKXHTFaAOteoQ4n2Dva3jA_1741270555 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 29/42] vfio/migration: Add vfio_add_bytes_transferred() Date: Thu, 6 Mar 2025 15:14:05 +0100 Message-ID: <20250306141419.2015340-30-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271259941019000 From: "Maciej S. Szmigiero" This way bytes_transferred can also be incremented in other translation units than migration.c. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/d1fbc27ac2417b49892f354ba20f6c6b3f= 7209f8.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 1 + hw/vfio/migration.c | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ac35136a11051b079cd9d04e6becd344a0e0f7e7..5c84ebb0029853bf408ffb7d308= a603435bd234b 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -274,6 +274,7 @@ void vfio_unblock_multiple_devices_migration(void); bool vfio_viommu_preset(VFIODevice *vbasedev); int64_t vfio_mig_bytes_transferred(void); void vfio_reset_bytes_transferred(void); +void vfio_mig_add_bytes_transferred(unsigned long val); bool vfio_device_state_is_running(VFIODevice *vbasedev); bool vfio_device_state_is_precopy(VFIODevice *vbasedev); =20 diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 5532787be63b9925a70202536a7f5796ea1fc8d1..51c056e152aae694ecc1b590c1a= 94dd8e5277c4a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -391,7 +391,7 @@ static ssize_t vfio_save_block(QEMUFile *f, VFIOMigrati= on *migration) qemu_put_be64(f, VFIO_MIG_FLAG_DEV_DATA_STATE); qemu_put_be64(f, data_size); qemu_put_buffer(f, migration->data_buffer, data_size); - qatomic_add(&bytes_transferred, data_size); + vfio_mig_add_bytes_transferred(data_size); =20 trace_vfio_save_block(migration->vbasedev->name, data_size); =20 @@ -1021,6 +1021,11 @@ void vfio_reset_bytes_transferred(void) qatomic_set(&bytes_transferred, 0); } =20 +void vfio_mig_add_bytes_transferred(unsigned long val) +{ + qatomic_add(&bytes_transferred, val); +} + /* * Return true when either migration initialized or blocker registered. * Currently only return false when adding blocker fails which will --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271009; cv=none; d=zohomail.com; s=zohoarc; b=n53blzJ2TRQCpaAndkGWYj55pzgE4bO9E5uLU4JQWkObdMf9y5DBD66Fsn/jBuiy1qYYIESpQByU9mfGLNsTmte0S0diLrh2KUpQnylzYF53GJmqdBod68pQL5T0c/EQe1xciOoTFOSqsrAWce3tHKnBTRCLUd7m53G6RjmcIZA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271009; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Bb2pXYyk+rCZrdsQNj/06TLTTEn4q37gLE7EXsSn5qo=; b=KFHmxCqzu9MpWYah7Ybw1rPrwg2ETkcjtxKPTnIKqnWEvFtva8AYLkTTSIvT+edv2gm0QHvNSdNN9qUNEpW0Ca7S1AY+bZgedNOpm2svB0XN18zICj70uZsWB6WpqBh+735Jw/ysJG9ZaTCBe5qNWkniCdpEV+iqZ5cJ0588wWI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 174127100967183.3889920439592; Thu, 6 Mar 2025 06:23:29 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC4F-0004Dt-Do; Thu, 06 Mar 2025 09:19:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1g-0000sH-N5 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1a-00028o-TR for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:39 -0500 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-144-Zf7mE1gINzCa3Tn8RfUS8Q-1; Thu, 06 Mar 2025 09:15:58 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 27C00195421B; Thu, 6 Mar 2025 14:15:57 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 85F9C18001D3; Thu, 6 Mar 2025 14:15:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270593; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bb2pXYyk+rCZrdsQNj/06TLTTEn4q37gLE7EXsSn5qo=; b=i7fVQOWW7IgGvstDVX+CxMO/b4U5hv1pTj/q/yhzCjAJI9g8OPceVZDHlplUKo6HqddOqd ZXXYMoDQT3KFNI2n3PA8++JHNwR878bEY3zpJqkE00A+B8iNT+BZyj9nkIYO1wfbGBoFsa wRI9u/SFzi3G+4R1Kf/UtHM8YCtK+lM= X-MC-Unique: Zf7mE1gINzCa3Tn8RfUS8Q-1 X-Mimecast-MFC-AGG-ID: Zf7mE1gINzCa3Tn8RfUS8Q_1741270557 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 30/42] vfio/migration: Move migration channel flags to vfio-common.h header file Date: Thu, 6 Mar 2025 15:14:06 +0100 Message-ID: <20250306141419.2015340-31-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271010567019000 From: "Maciej S. Szmigiero" This way they can also be referenced in other translation units than migration.c. Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/26a940f6b22c1b685818251b7a3ddbbca6= 01b1d6.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 17 +++++++++++++++++ hw/vfio/migration.c | 17 ----------------- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 5c84ebb0029853bf408ffb7d308a603435bd234b..bf5d5208712925f140e55583a2c= 0fe216184d046 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -36,6 +36,23 @@ =20 #define VFIO_MSG_PREFIX "vfio %s: " =20 +/* + * Flags to be used as unique delimiters for VFIO devices in the migration + * stream. These flags are composed as: + * 0xffffffff =3D> MSB 32-bit all 1s + * 0xef10 =3D> Magic ID, represents emulated (virtual) function IO + * 0x0000 =3D> 16-bits reserved for flags + * + * The beginning of state information is marked by _DEV_CONFIG_STATE, + * _DEV_SETUP_STATE, or _DEV_DATA_STATE, respectively. The end of a + * certain state information is marked by _END_OF_STATE. + */ +#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) +#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) +#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) +#define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) +#define VFIO_MIG_FLAG_DEV_INIT_DATA_SENT (0xffffffffef100005ULL) + enum { VFIO_DEVICE_TYPE_PCI =3D 0, VFIO_DEVICE_TYPE_PLATFORM =3D 1, diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 51c056e152aae694ecc1b590c1a94dd8e5277c4a..a9b0970604aa3e15fe2fb241228= 6e48cac3b03cc 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -31,23 +31,6 @@ #include "trace.h" #include "hw/hw.h" =20 -/* - * Flags to be used as unique delimiters for VFIO devices in the migration - * stream. These flags are composed as: - * 0xffffffff =3D> MSB 32-bit all 1s - * 0xef10 =3D> Magic ID, represents emulated (virtual) function IO - * 0x0000 =3D> 16-bits reserved for flags - * - * The beginning of state information is marked by _DEV_CONFIG_STATE, - * _DEV_SETUP_STATE, or _DEV_DATA_STATE, respectively. The end of a - * certain state information is marked by _END_OF_STATE. - */ -#define VFIO_MIG_FLAG_END_OF_STATE (0xffffffffef100001ULL) -#define VFIO_MIG_FLAG_DEV_CONFIG_STATE (0xffffffffef100002ULL) -#define VFIO_MIG_FLAG_DEV_SETUP_STATE (0xffffffffef100003ULL) -#define VFIO_MIG_FLAG_DEV_DATA_STATE (0xffffffffef100004ULL) -#define VFIO_MIG_FLAG_DEV_INIT_DATA_SENT (0xffffffffef100005ULL) - /* * This is an arbitrary size based on migration of mlx5 devices, where typ= ically * total device migration size is on the order of 100s of MB. Testing with --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270876; cv=none; d=zohomail.com; s=zohoarc; b=Q5h7IpGQRG1lBFKJQUVthvRA0/9rmZYQ3eScWYWoPVFLJ90dM7KVJcBxd6WeM/ctK3B1paWgmPGPzlw7f7UBd0By9Y6DhtiQLYXLnQVOZXfJSFyqWiCyIt8tAdOjOlIRDvviV3er/1hBvlE/kNpTwcJCsXWj5gtcpKQ8XoQMyWg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270876; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=8Vxi0S3m0kmWM2L3BzdSS+w3nmDL9rl89fpxIkNRBKs=; b=NZUYB19zTRAztxaikMOh31eC+4WPK/2GmNJFY1pzNosh1K2QY3Ao2V0RRUF2fuT85HPpjyJN8xhqGdBhDc5pChssldTSTclYWocfnhPaOd84P86pflGElhIA7jX4VU1RidKEME8ve3nSjZet5fvwgXIj9zGtdCP4fwxgcbkVTSE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270876967720.1832821130238; Thu, 6 Mar 2025 06:21:16 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3f-0002uC-U2; Thu, 06 Mar 2025 09:18:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC17-0008U5-Ch for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:05 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC15-0001v2-Gj for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:05 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-371-P6gtLt6GMraPGeO-r1Ql2w-1; Thu, 06 Mar 2025 09:16:00 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 424701828AA4; Thu, 6 Mar 2025 14:15:59 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A169C18001D3; Thu, 6 Mar 2025 14:15:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8Vxi0S3m0kmWM2L3BzdSS+w3nmDL9rl89fpxIkNRBKs=; b=DQ2p2a8I61TQzkjjezv/We2/YzcjjVPhJs8si7VvAtNkcEIfPAyUgK/XmWVvxRHW+fAkbn rfrUQGn7REJs0tfWfBZFdjNshme3Tcx/4WSztUZVhuMUSbRaHM0yDBEmhk9YnKg9JXV56v Id6WIeFP8WHtlZe5pOBJwQ/7bmv0Ro0= X-MC-Unique: P6gtLt6GMraPGeO-r1Ql2w-1 X-Mimecast-MFC-AGG-ID: P6gtLt6GMraPGeO-r1Ql2w_1741270559 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 31/42] vfio/migration: Multifd device state transfer support - basic types Date: Thu, 6 Mar 2025 15:14:07 +0100 Message-ID: <20250306141419.2015340-32-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270883045019100 From: "Maciej S. Szmigiero" Add basic types and flags used by VFIO multifd device state transfer support. Since we'll be introducing a lot of multifd transfer specific code, add a new file migration-multifd.c to home it, wired into main VFIO migration code (migration.c) via migration-multifd.h header file. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/4eedd529e6617f80f3d6a66d7268a0db2b= c173fa.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration-multifd.h | 17 +++++++++++++++++ hw/vfio/migration-multifd.c | 33 +++++++++++++++++++++++++++++++++ hw/vfio/migration.c | 1 + hw/vfio/meson.build | 1 + 4 files changed, 52 insertions(+) create mode 100644 hw/vfio/migration-multifd.h create mode 100644 hw/vfio/migration-multifd.c diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h new file mode 100644 index 0000000000000000000000000000000000000000..5b221c6e16b0265314034a6060b= 588e984b8ac5e --- /dev/null +++ b/hw/vfio/migration-multifd.h @@ -0,0 +1,17 @@ +/* + * Multifd VFIO migration + * + * Copyright (C) 2024,2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef HW_VFIO_MIGRATION_MULTIFD_H +#define HW_VFIO_MIGRATION_MULTIFD_H + +#include "hw/vfio/vfio-common.h" + +#endif diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c new file mode 100644 index 0000000000000000000000000000000000000000..fa594b33fdd1112e1d3ecf52a35= fecb551e29999 --- /dev/null +++ b/hw/vfio/migration-multifd.c @@ -0,0 +1,33 @@ +/* + * Multifd VFIO migration + * + * Copyright (C) 2024,2025 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "hw/vfio/vfio-common.h" +#include "migration/misc.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/lockable.h" +#include "qemu/main-loop.h" +#include "qemu/thread.h" +#include "migration/qemu-file.h" +#include "migration-multifd.h" +#include "trace.h" + +#define VFIO_DEVICE_STATE_CONFIG_STATE (1) + +#define VFIO_DEVICE_STATE_PACKET_VER_CURRENT (0) + +typedef struct VFIODeviceStatePacket { + uint32_t version; + uint32_t idx; + uint32_t flags; + uint8_t data[0]; +} QEMU_PACKED VFIODeviceStatePacket; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index a9b0970604aa3e15fe2fb2412286e48cac3b03cc..dc1fe4e717a465f442634f6e65e= 9c56df3c084d7 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -23,6 +23,7 @@ #include "migration/qemu-file.h" #include "migration/register.h" #include "migration/blocker.h" +#include "migration-multifd.h" #include "qapi/error.h" #include "qapi/qapi-events-vfio.h" #include "exec/ramlist.h" diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index bba776f75cc718956b37f68cb542b378a509d77f..260d65febd6bcd1935cfa9f59cc= 08aae21fc97fd 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -5,6 +5,7 @@ vfio_ss.add(files( 'container-base.c', 'container.c', 'migration.c', + 'migration-multifd.c', 'cpr.c', )) vfio_ss.add(when: 'CONFIG_PSERIES', if_true: files('spapr.c')) --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270784; cv=none; d=zohomail.com; s=zohoarc; b=AN7j40iTDyUPIV9y8Fbssc+LHwjwROQugp/tdUlzKXef3g63JilSf2ItwVcPwo+H9Cw+WUvDf6jdGmKB80VHQ0UynzHG0EFlwx3AlaqBo2kUHbbqDejJopghrOtMj311mfhmHOAaegSfLmem9oT2nh6DctkKinQFMov6HYjeXTE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270784; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=4PEo31mSNERiGC5mMve2THeVlhL4GmE+0RiWcz7ZlWI=; b=BMM9QNltNBBRoP/nw33T4NBSAop/09MKeyDZihQKRBh/T0/Rv9zsRAg2MYSqASqk+GdaO+5F0pl2r+tRe1naA2mNcd97dnB1njZzFfBsrNaTkPqHr5ctAs+yxr1zncdpFdDA196XVoECIE9by0sIaOH9x5mbRc1BsRyqVuccCts= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270784543288.83522950639247; Thu, 6 Mar 2025 06:19:44 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3U-0002SW-MS; Thu, 06 Mar 2025 09:18:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1F-0000Oz-DK for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:15 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1D-0001zy-Mj for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:13 -0500 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-170-WGd7WDaUOg2Yof-lYkMP2Q-1; Thu, 06 Mar 2025 09:16:02 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 73F1019560B7; Thu, 6 Mar 2025 14:16:01 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BEB7B180174D; Thu, 6 Mar 2025 14:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4PEo31mSNERiGC5mMve2THeVlhL4GmE+0RiWcz7ZlWI=; b=Ylg6WggnuNMMvVCPRCSxQN1JIaD6qBF0Gg1u+6Jj8ep5R+fE3nK2qlgfUeI8Q0t8THc9j1 drgr5oIYCJ44Drqw0qNdkiUi0uFqIsooP4jvYD+oyLrbwSMvSiaTK0g6SDUnO+9liZehyl e9t28wuM7ky8Cqkl7SRWeYdfQdPlW10= X-MC-Unique: WGd7WDaUOg2Yof-lYkMP2Q-1 X-Mimecast-MFC-AGG-ID: WGd7WDaUOg2Yof-lYkMP2Q_1741270561 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 32/42] vfio/migration: Multifd device state transfer - add support checking function Date: Thu, 6 Mar 2025 15:14:08 +0100 Message-ID: <20250306141419.2015340-33-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270787003019100 From: "Maciej S. Szmigiero" Add vfio_multifd_transfer_supported() function that tells whether the multifd device state transfer is supported. Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/8ce50256f341b3d47342bb217cb5fbb2de= b14639.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration-multifd.h | 2 ++ hw/vfio/migration-multifd.c | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h index 5b221c6e16b0265314034a6060b588e984b8ac5e..1b60d5f67a1c76cdac21d71a7fb= 7335f4e194ffe 100644 --- a/hw/vfio/migration-multifd.h +++ b/hw/vfio/migration-multifd.h @@ -14,4 +14,6 @@ =20 #include "hw/vfio/vfio-common.h" =20 +bool vfio_multifd_transfer_supported(void); + #endif diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index fa594b33fdd1112e1d3ecf52a35fecb551e29999..79fae0b6296f9244317ed5215e4= 950703c45a717 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -31,3 +31,9 @@ typedef struct VFIODeviceStatePacket { uint32_t flags; uint8_t data[0]; } QEMU_PACKED VFIODeviceStatePacket; + +bool vfio_multifd_transfer_supported(void) +{ + return multifd_device_state_supported() && + migrate_send_switchover_start(); +} --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270981; cv=none; d=zohomail.com; s=zohoarc; b=DjbuYfXicOINZ6tgPupNArJ30QGj+NiM7vQrFKw0UljmI4GbaKygpTWShk1Sb8a6A0rVM0SOrzyT2Ok7bubmMkZDmS2c9XheooWf/95Ft3zG+3Q9ueoCI2xxy51hm6JeDvtLaJtpW3NwJdXn0lTY99wrj1/YvnE7SMEEWebsbwk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270981; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=9tE6LKSxXlL5BjHPli5rp4ZzG1kP0l59yDZWkLvT/qs=; b=Fd+93dRzG7pT0bccGtDkj4SML78/kuOdAJJ+dDasVNe6wFmrH44nRVVxPjrAU3EYwRskKAoF7CR4uMVb8UPIxL5VIN/4G3hMATT/G2WN2DoFUSdueZCbd4RfFr4DhauUWn2lEfGEXlOgOEpTQOHO3PZiM1EWXPVSzfmUCEwgbec= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 174127098198725.821078866605035; Thu, 6 Mar 2025 06:23:01 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3s-0003BY-6j; Thu, 06 Mar 2025 09:18:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1U-0000lf-9M for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1S-00025e-8V for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:27 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-315-zH1qSnrdPB6958rPOerx6Q-1; Thu, 06 Mar 2025 09:16:04 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 11AA41955EF4; Thu, 6 Mar 2025 14:16:04 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F17DD18009BC; Thu, 6 Mar 2025 14:16:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270585; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9tE6LKSxXlL5BjHPli5rp4ZzG1kP0l59yDZWkLvT/qs=; b=gEj+rz9rCFkj3nUPh+KSjIycEX6kmt+Zf4BQOHaWmWVAspBgPB4SMQkZVRzTZV+BZqgWB2 I/XDDPBdjsEvjGBWdv7SVyKVCGOwaRK1qS2ygIlQXgzHIE8nfsgafR3RsrmcRF+s7jq0MH GlTpsi1tWboAzxavRE/G7JTGS6zMtUo= X-MC-Unique: zH1qSnrdPB6958rPOerx6Q-1 X-Mimecast-MFC-AGG-ID: zH1qSnrdPB6958rPOerx6Q_1741270564 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 33/42] vfio/migration: Multifd setup/cleanup functions and associated VFIOMultifd Date: Thu, 6 Mar 2025 15:14:09 +0100 Message-ID: <20250306141419.2015340-34-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270982569019000 From: "Maciej S. Szmigiero" Add multifd setup/cleanup functions and an associated VFIOMultifd data structure that will contain most of the receive-side data together with its init/cleanup methods. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/c0520523053b1087787152ddf2163257d3= 030be0.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration-multifd.h | 4 ++++ include/hw/vfio/vfio-common.h | 3 +++ hw/vfio/migration-multifd.c | 44 +++++++++++++++++++++++++++++++++++ 3 files changed, 51 insertions(+) diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h index 1b60d5f67a1c76cdac21d71a7fb7335f4e194ffe..2a7a76164f291d182172775524a= 5b11c0a560c58 100644 --- a/hw/vfio/migration-multifd.h +++ b/hw/vfio/migration-multifd.h @@ -14,6 +14,10 @@ =20 #include "hw/vfio/vfio-common.h" =20 +bool vfio_multifd_setup(VFIODevice *vbasedev, bool alloc_multifd, Error **= errp); +void vfio_multifd_cleanup(VFIODevice *vbasedev); + bool vfio_multifd_transfer_supported(void); +bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev); =20 #endif diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index bf5d5208712925f140e55583a2c0fe216184d046..40382390692dcd772f2ceb6689d= 9b31dc5e8b8d5 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -78,6 +78,8 @@ typedef struct VFIORegion { uint8_t nr; /* cache the region number for debug */ } VFIORegion; =20 +typedef struct VFIOMultifd VFIOMultifd; + typedef struct VFIOMigration { struct VFIODevice *vbasedev; VMChangeStateEntry *vm_state; @@ -89,6 +91,7 @@ typedef struct VFIOMigration { uint64_t mig_flags; uint64_t precopy_init_size; uint64_t precopy_dirty_size; + VFIOMultifd *multifd; bool initial_data_sent; =20 bool event_save_iterate_started; diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index 79fae0b6296f9244317ed5215e4950703c45a717..091dc43210ad1459d5114da1833= 6e73f6cb0baf9 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -32,8 +32,52 @@ typedef struct VFIODeviceStatePacket { uint8_t data[0]; } QEMU_PACKED VFIODeviceStatePacket; =20 +typedef struct VFIOMultifd { +} VFIOMultifd; + +static VFIOMultifd *vfio_multifd_new(void) +{ + VFIOMultifd *multifd =3D g_new(VFIOMultifd, 1); + + return multifd; +} + +static void vfio_multifd_free(VFIOMultifd *multifd) +{ + g_free(multifd); +} + +void vfio_multifd_cleanup(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + + g_clear_pointer(&migration->multifd, vfio_multifd_free); +} + bool vfio_multifd_transfer_supported(void) { return multifd_device_state_supported() && migrate_send_switchover_start(); } + +bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev) +{ + return false; +} + +bool vfio_multifd_setup(VFIODevice *vbasedev, bool alloc_multifd, Error **= errp) +{ + VFIOMigration *migration =3D vbasedev->migration; + + if (!vfio_multifd_transfer_enabled(vbasedev)) { + /* Nothing further to check or do */ + return true; + } + + if (alloc_multifd) { + assert(!migration->multifd); + migration->multifd =3D vfio_multifd_new(); + } + + return true; +} --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270798; cv=none; d=zohomail.com; s=zohoarc; b=EyMb+fcep6wlofG1LmysylsZbDOZObqm0dm0GrMiyZEosiI8e9cHVpsOo405H5aKHwlza7Wr6+wIodbz7NzHz+u2YC8F9tv2NMt67koicwTe4GDlhOHdAMAkOKdbSXurYt2m6V/Ad/Sll2zS40R3kg4B75UO/o9XSfInMTHjvYE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270798; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=0IzeVlm8ShaMbeT8/mVTvAuhmG3dHX9hBq2h899cedg=; b=l5ARVxX1b1EJWnyx3DaJtrWAklAEpy0lOv/DDMFVUgI+77qK4q3E6BwjhbIvl7OTe6ZfwI4yrhSv6iNJuPGqOusq4O/1KikG6ttcPTUIFBSyWWhrWfZleaKfxM7pWuG69EgRWewpITDUmZjxYi+sG0yP/0W46ydvUFnNLW6efWs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270798625514.6947584859361; Thu, 6 Mar 2025 06:19:58 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC2F-0001JG-TQ; Thu, 06 Mar 2025 09:17:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1J-0000gQ-83 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:23 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1C-0001zA-Ta for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:16 -0500 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-508-7YgSuEipM5eP7WSIIrAYqA-1; Thu, 06 Mar 2025 09:16:07 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2EB591954B1C; Thu, 6 Mar 2025 14:16:06 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8C32818001D3; Thu, 6 Mar 2025 14:16:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0IzeVlm8ShaMbeT8/mVTvAuhmG3dHX9hBq2h899cedg=; b=cM77PU24ci6erAZh792ewjOnmtAlPDD9aGMAPvUXQRtIOAgbnMOi+Q741RQvjAC1xAg+vx V0qPrR9qpeOrOrhgy6eh6U4MwV9/WtOd9TB0C9TPdgBy70DYCGvB0a2j2YAUXqipaVF3j5 V4/E/tGnr3n75T7Hzv76PSWC0D/eS6E= X-MC-Unique: 7YgSuEipM5eP7WSIIrAYqA-1 X-Mimecast-MFC-AGG-ID: 7YgSuEipM5eP7WSIIrAYqA_1741270566 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 34/42] vfio/migration: Setup and cleanup multifd transfer in these general methods Date: Thu, 6 Mar 2025 15:14:10 +0100 Message-ID: <20250306141419.2015340-35-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270799819019000 From: "Maciej S. Szmigiero" Wire VFIO multifd transfer specific setup and cleanup functions into general VFIO load/save setup and cleanup methods. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/b1f864a65fafd4fdab1f89230df52e46ae= 41f2ac.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index dc1fe4e717a465f442634f6e65e9c56df3c084d7..3c8286ae62300122582ae5ced26= f5cbf5742818a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -453,6 +453,10 @@ static int vfio_save_setup(QEMUFile *f, void *opaque, = Error **errp) uint64_t stop_copy_size =3D VFIO_MIG_DEFAULT_DATA_BUFFER_SIZE; int ret; =20 + if (!vfio_multifd_setup(vbasedev, false, errp)) { + return -EINVAL; + } + qemu_put_be64(f, VFIO_MIG_FLAG_DEV_SETUP_STATE); =20 vfio_query_stop_copy_size(vbasedev, &stop_copy_size); @@ -509,6 +513,9 @@ static void vfio_save_cleanup(void *opaque) Error *local_err =3D NULL; int ret; =20 + /* Currently a NOP, done for symmetry with load_cleanup() */ + vfio_multifd_cleanup(vbasedev); + /* * Changing device state from STOP_COPY to STOP can take time. Do it h= ere, * after migration has completed, so it won't increase downtime. @@ -674,15 +681,28 @@ static void vfio_save_state(QEMUFile *f, void *opaque) static int vfio_load_setup(QEMUFile *f, void *opaque, Error **errp) { VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + int ret; =20 - return vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RESUMING, - vbasedev->migration->device_state, err= p); + if (!vfio_multifd_setup(vbasedev, true, errp)) { + return -EINVAL; + } + + ret =3D vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_RESUMING, + migration->device_state, errp); + if (ret) { + return ret; + } + + return 0; } =20 static int vfio_load_cleanup(void *opaque) { VFIODevice *vbasedev =3D opaque; =20 + vfio_multifd_cleanup(vbasedev); + vfio_migration_cleanup(vbasedev); trace_vfio_load_cleanup(vbasedev->name); =20 --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270979; cv=none; d=zohomail.com; s=zohoarc; b=OxJvY+uIiM7myR64og0ksrAhpMS1d+d3OUVgnNZIO89lLqWc0RNS514nN7V8XDwM7UJGlMv76ynejDUnFtqMRvC7IeuEbmMvgxnhg6B23mss4GeeHbEK555m+7eWn5Nl6wBXfXIbM6TTDlhSNaKwwVTGpaeR3pxefpvt2yjVFq0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270979; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=fL2jDoLkbNFzfcsI8GHD0gwVFThbZF+dj4p0Hq4j3rw=; b=ITUGlrhmKDrbu0vAj6Fx215arZcr9c4AVarOG3moqXOTXkz3DGYbYRCwqphGBUB6O592yK3udCl1j2Za5PWxTdrhnUaIk7LQWWojOtir7KzEAQH5kRV9/AUL1A0gJY8PGNMnAH0tQuJJjjZlZOONLX+0iEXFdMTExJ81zkcA7NU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270979133799.9836723956802; Thu, 6 Mar 2025 06:22:59 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC4B-0003zL-M1; Thu, 06 Mar 2025 09:19:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1e-0000sG-Mh for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1X-00027m-Nd for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:35 -0500 Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-650-dNUYE8_EMP6zXm3kRnVMeQ-1; Thu, 06 Mar 2025 09:16:09 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8BDCE1955E9C; Thu, 6 Mar 2025 14:16:08 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A99CD18009BC; Thu, 6 Mar 2025 14:16:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fL2jDoLkbNFzfcsI8GHD0gwVFThbZF+dj4p0Hq4j3rw=; b=X+MeZnpc5ZzGsHJPPGu6V6y7RxVIbCwra0NOh/0UhDyGu9aBFaf52zTlqedcITJ7uFQAIW u4MYS2PwpTcXrh/BKN7lk7KYvqR2QHX4VO3AKIl6Syp3ltTh3zZF8CVaoEo307ZB2z6Xhl KkjsGpRbd7IQA7yzSPpUFiQG5y+I3/w= X-MC-Unique: dNUYE8_EMP6zXm3kRnVMeQ-1 X-Mimecast-MFC-AGG-ID: dNUYE8_EMP6zXm3kRnVMeQ_1741270568 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 35/42] vfio/migration: Multifd device state transfer support - received buffers queuing Date: Thu, 6 Mar 2025 15:14:11 +0100 Message-ID: <20250306141419.2015340-36-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270981814019100 From: "Maciej S. Szmigiero" The multifd received data needs to be reassembled since device state packets sent via different multifd channels can arrive out-of-order. Therefore, each VFIO device state packet carries a header indicating its position in the stream. The raw device state data is saved into a VFIOStateBuffer for later in-order loading into the device. The last such VFIO device state packet should have VFIO_DEVICE_STATE_CONFIG_STATE flag set and carry the device config state. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/e3bff515a8d61c582b94b409eb12a45b1a= 143a69.1741124640.git.maciej.szmigiero@oracle.com [ clg: - Reordered savevm_vfio_handlers - Added load_state_buffer documentation ] Signed-off-by: C=C3=A9dric Le Goater --- docs/devel/migration/vfio.rst | 7 ++ hw/vfio/migration-multifd.h | 3 + hw/vfio/migration-multifd.c | 163 ++++++++++++++++++++++++++++++++++ hw/vfio/migration.c | 4 + hw/vfio/trace-events | 1 + 5 files changed, 178 insertions(+) diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index c49482eab66d8e831ea1c2c791fc895b51893e4d..8b1f28890a0ba708cc1e49d87b1= 6e17a5d8f7c88 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -76,6 +76,10 @@ VFIO implements the device hooks for the iterative appro= ach as follows: * A ``load_state`` function that loads the config section and the data sections that are generated by the save functions above. =20 +* A ``load_state_buffer`` function that loads the device state and the dev= ice + config that arrived via multifd channels. + It's used only in the multifd mode. + * ``cleanup`` functions for both save and load that perform any migration related cleanup. =20 @@ -194,6 +198,9 @@ Live migration resume path (RESTORE_VM, _ACTIVE, _STOP) | For each device, .load_state() is called for that device section data + transmitted via the main migration channel. + For data transmitted via multifd channels .load_state_buffer() is cal= led + instead. (RESTORE_VM, _ACTIVE, _RESUMING) | At the end, .load_cleanup() is called for each device and vCPUs are star= ted diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h index 2a7a76164f291d182172775524a5b11c0a560c58..8c6320fcb484ca9f779e14d4f9d= 814081d2f760e 100644 --- a/hw/vfio/migration-multifd.h +++ b/hw/vfio/migration-multifd.h @@ -20,4 +20,7 @@ void vfio_multifd_cleanup(VFIODevice *vbasedev); bool vfio_multifd_transfer_supported(void); bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev); =20 +bool vfio_multifd_load_state_buffer(void *opaque, char *data, size_t data_= size, + Error **errp); + #endif diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index 091dc43210ad1459d5114da18336e73f6cb0baf9..79df11b7baa991763039b395c34= 931e796e711b0 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -32,18 +32,181 @@ typedef struct VFIODeviceStatePacket { uint8_t data[0]; } QEMU_PACKED VFIODeviceStatePacket; =20 +/* type safety */ +typedef struct VFIOStateBuffers { + GArray *array; +} VFIOStateBuffers; + +typedef struct VFIOStateBuffer { + bool is_present; + char *data; + size_t len; +} VFIOStateBuffer; + typedef struct VFIOMultifd { + VFIOStateBuffers load_bufs; + QemuCond load_bufs_buffer_ready_cond; + QemuMutex load_bufs_mutex; /* Lock order: this lock -> BQL */ + uint32_t load_buf_idx; + uint32_t load_buf_idx_last; } VFIOMultifd; =20 +static void vfio_state_buffer_clear(gpointer data) +{ + VFIOStateBuffer *lb =3D data; + + if (!lb->is_present) { + return; + } + + g_clear_pointer(&lb->data, g_free); + lb->is_present =3D false; +} + +static void vfio_state_buffers_init(VFIOStateBuffers *bufs) +{ + bufs->array =3D g_array_new(FALSE, TRUE, sizeof(VFIOStateBuffer)); + g_array_set_clear_func(bufs->array, vfio_state_buffer_clear); +} + +static void vfio_state_buffers_destroy(VFIOStateBuffers *bufs) +{ + g_clear_pointer(&bufs->array, g_array_unref); +} + +static void vfio_state_buffers_assert_init(VFIOStateBuffers *bufs) +{ + assert(bufs->array); +} + +static unsigned int vfio_state_buffers_size_get(VFIOStateBuffers *bufs) +{ + return bufs->array->len; +} + +static void vfio_state_buffers_size_set(VFIOStateBuffers *bufs, + unsigned int size) +{ + g_array_set_size(bufs->array, size); +} + +static VFIOStateBuffer *vfio_state_buffers_at(VFIOStateBuffers *bufs, + unsigned int idx) +{ + return &g_array_index(bufs->array, VFIOStateBuffer, idx); +} + +/* called with load_bufs_mutex locked */ +static bool vfio_load_state_buffer_insert(VFIODevice *vbasedev, + VFIODeviceStatePacket *packet, + size_t packet_total_size, + Error **errp) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + VFIOStateBuffer *lb; + + vfio_state_buffers_assert_init(&multifd->load_bufs); + if (packet->idx >=3D vfio_state_buffers_size_get(&multifd->load_bufs))= { + vfio_state_buffers_size_set(&multifd->load_bufs, packet->idx + 1); + } + + lb =3D vfio_state_buffers_at(&multifd->load_bufs, packet->idx); + if (lb->is_present) { + error_setg(errp, "%s: state buffer %" PRIu32 " already filled", + vbasedev->name, packet->idx); + return false; + } + + assert(packet->idx >=3D multifd->load_buf_idx); + + lb->data =3D g_memdup2(&packet->data, packet_total_size - sizeof(*pack= et)); + lb->len =3D packet_total_size - sizeof(*packet); + lb->is_present =3D true; + + return true; +} + +bool vfio_multifd_load_state_buffer(void *opaque, char *data, size_t data_= size, + Error **errp) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + VFIODeviceStatePacket *packet =3D (VFIODeviceStatePacket *)data; + + if (!vfio_multifd_transfer_enabled(vbasedev)) { + error_setg(errp, + "%s: got device state packet but not doing multifd tran= sfer", + vbasedev->name); + return false; + } + + assert(multifd); + + if (data_size < sizeof(*packet)) { + error_setg(errp, "%s: packet too short at %zu (min is %zu)", + vbasedev->name, data_size, sizeof(*packet)); + return false; + } + + if (packet->version !=3D VFIO_DEVICE_STATE_PACKET_VER_CURRENT) { + error_setg(errp, "%s: packet has unknown version %" PRIu32, + vbasedev->name, packet->version); + return false; + } + + if (packet->idx =3D=3D UINT32_MAX) { + error_setg(errp, "%s: packet index is invalid", vbasedev->name); + return false; + } + + trace_vfio_load_state_device_buffer_incoming(vbasedev->name, packet->i= dx); + + /* + * Holding BQL here would violate the lock order and can cause + * a deadlock once we attempt to lock load_bufs_mutex below. + */ + assert(!bql_locked()); + + WITH_QEMU_LOCK_GUARD(&multifd->load_bufs_mutex) { + /* config state packet should be the last one in the stream */ + if (packet->flags & VFIO_DEVICE_STATE_CONFIG_STATE) { + multifd->load_buf_idx_last =3D packet->idx; + } + + if (!vfio_load_state_buffer_insert(vbasedev, packet, data_size, + errp)) { + return false; + } + + qemu_cond_signal(&multifd->load_bufs_buffer_ready_cond); + } + + return true; +} + static VFIOMultifd *vfio_multifd_new(void) { VFIOMultifd *multifd =3D g_new(VFIOMultifd, 1); =20 + vfio_state_buffers_init(&multifd->load_bufs); + + qemu_mutex_init(&multifd->load_bufs_mutex); + + multifd->load_buf_idx =3D 0; + multifd->load_buf_idx_last =3D UINT32_MAX; + qemu_cond_init(&multifd->load_bufs_buffer_ready_cond); + return multifd; } =20 static void vfio_multifd_free(VFIOMultifd *multifd) { + vfio_state_buffers_destroy(&multifd->load_bufs); + qemu_cond_destroy(&multifd->load_bufs_buffer_ready_cond); + qemu_mutex_destroy(&multifd->load_bufs_mutex); + g_free(multifd); } =20 diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 3c8286ae62300122582ae5ced26f5cbf5742818a..2cdb92356e0a2afb64109c10536= c857b19f7e7c5 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -802,6 +802,10 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { .load_cleanup =3D vfio_load_cleanup, .load_state =3D vfio_load_state, .switchover_ack_needed =3D vfio_switchover_ack_needed, + /* + * Multifd support + */ + .load_state_buffer =3D vfio_multifd_load_state_buffer, }; =20 /* ---------------------------------------------------------------------- = */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index a02c668f28a43f73ed2db9f15827f26fed0994c3..404ea079b25c49fe25f4c9b05f0= cde4f0536fdd7 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -154,6 +154,7 @@ vfio_load_device_config_state_start(const char *name) "= (%s)" vfio_load_device_config_state_end(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret)= " (%s) size %"PRIu64" ret %d" +vfio_load_state_device_buffer_incoming(const char *name, uint32_t idx) " (= %s) idx %"PRIu32 vfio_migration_realize(const char *name) " (%s)" vfio_migration_set_device_state(const char *name, const char *state) " (%s= ) state %s" vfio_migration_set_state(const char *name, const char *new_state, const ch= ar *recover_state) " (%s) new state %s, recover state %s" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270796; cv=none; d=zohomail.com; s=zohoarc; b=eEDdv9EoGJ+4r1BWJ8/+aLKdjgIU8SQ1lKejW6QkC5Zcs/C2Y6m/+dkguglwVSmLucDuop5KhYjkVztAp9HswSzKhZUyjFQDTeCtUajE8qEYNDAiFosa0fQ94NFlOKslBaiCiXMb9Ng3zZkdr73280SKkf8LR5vFXo6PyAA+cWk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270796; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ZP4TLHbC3WazgKO8vlgr/7iKmqPs2/SyqnCf0Vx7TnA=; b=TRFk8OFhe9jUVtBd1lJfsNa1dydmiKQTYfCutDRQiXy2vCDEpWCVGrIR+Z+Q+pwynwQrK9HtAVAzwGVDHMW8VxM5RDkhQkhMNYf6dtIgKE0A4T8uGZCA8hyNNaxpzd5/I/SqmT4J8sxN3f3v5RFrigmMfvwxMXezrVk63VSPOMs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17412707968571014.298510718263; Thu, 6 Mar 2025 06:19:56 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3U-0002SM-NI; Thu, 06 Mar 2025 09:18:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1J-0000gR-94 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:23 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1G-00020d-JC for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:16 -0500 Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-683-uZh2R9XTNuyInRZ8rZS4zQ-1; Thu, 06 Mar 2025 09:16:12 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 8BD8F1954223; Thu, 6 Mar 2025 14:16:11 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 251CE18009BC; Thu, 6 Mar 2025 14:16:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270573; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZP4TLHbC3WazgKO8vlgr/7iKmqPs2/SyqnCf0Vx7TnA=; b=EyraR0I/GHnkdpBpe/ew0sgIj/kAzrcDbkB/xPFtNnJ4/qWbxgwY13gi1v9Wf41ri+XVk7 l+dz6GZ4nmgBTdcDIpAJyTqXAcsAAkBLvX3BmnjNXh1VcOdFsuxFSI+eWtnWFEBTFoG/3R YD9SAX9M828JvxxW4pjBKbDwRkMrI/A= X-MC-Unique: uZh2R9XTNuyInRZ8rZS4zQ-1 X-Mimecast-MFC-AGG-ID: uZh2R9XTNuyInRZ8rZS4zQ_1741270571 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 36/42] vfio/migration: Multifd device state transfer support - load thread Date: Thu, 6 Mar 2025 15:14:12 +0100 Message-ID: <20250306141419.2015340-37-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270798064019000 From: "Maciej S. Szmigiero" Add a thread which loads the VFIO device state buffers that were received via multifd. Each VFIO device that has multifd device state transfer enabled has one such thread, which is created using migration core API qemu_loadvm_start_load_thread(). Since it's important to finish loading device state transferred via the main migration channel (via save_live_iterate SaveVMHandler) before starting loading the data asynchronously transferred via multifd the thread doing the actual loading of the multifd transferred data is only started from switchover_start SaveVMHandler. switchover_start handler is called when MIG_CMD_SWITCHOVER_START sub-command of QEMU_VM_COMMAND is received via the main migration channel. This sub-command is only sent after all save_live_iterate data have already been posted so it is safe to commence loading of the multifd-transferred device state upon receiving it - loading of save_live_iterate data happens synchronously in the main migration thread (much like the processing of MIG_CMD_SWITCHOVER_START) so by the time MIG_CMD_SWITCHOVER_START is processed all the proceeding data must have already been loaded. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/9abe612d775aaf42e31646796acd2363c7= 23a57a.1741124640.git.maciej.szmigiero@oracle.com [ clg: - Reordered savevm_vfio_handlers - Added switchover_start documentation ] Signed-off-by: C=C3=A9dric Le Goater --- docs/devel/migration/vfio.rst | 4 + hw/vfio/migration-multifd.h | 2 + hw/vfio/migration-multifd.c | 226 ++++++++++++++++++++++++++++++++++ hw/vfio/migration.c | 12 ++ hw/vfio/trace-events | 7 ++ 5 files changed, 251 insertions(+) diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index 8b1f28890a0ba708cc1e49d87b16e17a5d8f7c88..d6cf60890c43f2ebf085940a233= f8b85a900b6b3 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -67,6 +67,10 @@ VFIO implements the device hooks for the iterative appro= ach as follows: * A ``switchover_ack_needed`` function that checks if the VFIO device uses "switchover-ack" migration capability when this capability is enabled. =20 +* A ``switchover_start`` function that in the multifd mode starts a thread= that + reassembles the multifd received data and loads it in-order into the dev= ice. + In the non-multifd mode this function is a NOP. + * A ``save_state`` function to save the device config space if it is prese= nt. =20 * A ``save_live_complete_precopy`` function that sets the VFIO device in diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h index 8c6320fcb484ca9f779e14d4f9d814081d2f760e..f0d28fcef2ea737fc550a34ff26= 6a34848bca67c 100644 --- a/hw/vfio/migration-multifd.h +++ b/hw/vfio/migration-multifd.h @@ -23,4 +23,6 @@ bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev); bool vfio_multifd_load_state_buffer(void *opaque, char *data, size_t data_= size, Error **errp); =20 +int vfio_multifd_switchover_start(VFIODevice *vbasedev); + #endif diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index 79df11b7baa991763039b395c34931e796e711b0..2eef27604e4f8f1f4db74d132a4= d4bf2906a4cc1 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -44,8 +44,12 @@ typedef struct VFIOStateBuffer { } VFIOStateBuffer; =20 typedef struct VFIOMultifd { + bool load_bufs_thread_running; + bool load_bufs_thread_want_exit; + VFIOStateBuffers load_bufs; QemuCond load_bufs_buffer_ready_cond; + QemuCond load_bufs_thread_finished_cond; QemuMutex load_bufs_mutex; /* Lock order: this lock -> BQL */ uint32_t load_buf_idx; uint32_t load_buf_idx_last; @@ -186,6 +190,178 @@ bool vfio_multifd_load_state_buffer(void *opaque, cha= r *data, size_t data_size, return true; } =20 +static bool vfio_load_bufs_thread_load_config(VFIODevice *vbasedev, + Error **errp) +{ + error_setg(errp, "not yet there"); + return false; +} + +static VFIOStateBuffer *vfio_load_state_buffer_get(VFIOMultifd *multifd) +{ + VFIOStateBuffer *lb; + unsigned int bufs_len; + + bufs_len =3D vfio_state_buffers_size_get(&multifd->load_bufs); + if (multifd->load_buf_idx >=3D bufs_len) { + assert(multifd->load_buf_idx =3D=3D bufs_len); + return NULL; + } + + lb =3D vfio_state_buffers_at(&multifd->load_bufs, + multifd->load_buf_idx); + if (!lb->is_present) { + return NULL; + } + + return lb; +} + +static bool vfio_load_state_buffer_write(VFIODevice *vbasedev, + VFIOStateBuffer *lb, + Error **errp) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + g_autofree char *buf =3D NULL; + char *buf_cur; + size_t buf_len; + + if (!lb->len) { + return true; + } + + trace_vfio_load_state_device_buffer_load_start(vbasedev->name, + multifd->load_buf_idx); + + /* lb might become re-allocated when we drop the lock */ + buf =3D g_steal_pointer(&lb->data); + buf_cur =3D buf; + buf_len =3D lb->len; + while (buf_len > 0) { + ssize_t wr_ret; + int errno_save; + + /* + * Loading data to the device takes a while, + * drop the lock during this process. + */ + qemu_mutex_unlock(&multifd->load_bufs_mutex); + wr_ret =3D write(migration->data_fd, buf_cur, buf_len); + errno_save =3D errno; + qemu_mutex_lock(&multifd->load_bufs_mutex); + + if (wr_ret < 0) { + error_setg(errp, + "%s: writing state buffer %" PRIu32 " failed: %d", + vbasedev->name, multifd->load_buf_idx, errno_save); + return false; + } + + assert(wr_ret <=3D buf_len); + buf_len -=3D wr_ret; + buf_cur +=3D wr_ret; + } + + trace_vfio_load_state_device_buffer_load_end(vbasedev->name, + multifd->load_buf_idx); + + return true; +} + +static bool vfio_load_bufs_thread_want_exit(VFIOMultifd *multifd, + bool *should_quit) +{ + return multifd->load_bufs_thread_want_exit || qatomic_read(should_quit= ); +} + +/* + * This thread is spawned by vfio_multifd_switchover_start() which gets + * called upon encountering the switchover point marker in main migration + * stream. + * + * It exits after either: + * * completing loading the remaining device state and device config, OR: + * * encountering some error while doing the above, OR: + * * being forcefully aborted by the migration core by it setting should_q= uit + * or by vfio_load_cleanup_load_bufs_thread() setting + * multifd->load_bufs_thread_want_exit. + */ +static bool vfio_load_bufs_thread(void *opaque, bool *should_quit, Error *= *errp) +{ + VFIODevice *vbasedev =3D opaque; + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + bool ret =3D false; + + trace_vfio_load_bufs_thread_start(vbasedev->name); + + assert(multifd); + QEMU_LOCK_GUARD(&multifd->load_bufs_mutex); + + assert(multifd->load_bufs_thread_running); + + while (true) { + VFIOStateBuffer *lb; + + /* + * Always check cancellation first after the buffer_ready wait bel= ow in + * case that cond was signalled by vfio_load_cleanup_load_bufs_thr= ead(). + */ + if (vfio_load_bufs_thread_want_exit(multifd, should_quit)) { + error_setg(errp, "operation cancelled"); + goto thread_exit; + } + + assert(multifd->load_buf_idx <=3D multifd->load_buf_idx_last); + + lb =3D vfio_load_state_buffer_get(multifd); + if (!lb) { + trace_vfio_load_state_device_buffer_starved(vbasedev->name, + multifd->load_buf_= idx); + qemu_cond_wait(&multifd->load_bufs_buffer_ready_cond, + &multifd->load_bufs_mutex); + continue; + } + + if (multifd->load_buf_idx =3D=3D multifd->load_buf_idx_last) { + break; + } + + if (multifd->load_buf_idx =3D=3D 0) { + trace_vfio_load_state_device_buffer_start(vbasedev->name); + } + + if (!vfio_load_state_buffer_write(vbasedev, lb, errp)) { + goto thread_exit; + } + + if (multifd->load_buf_idx =3D=3D multifd->load_buf_idx_last - 1) { + trace_vfio_load_state_device_buffer_end(vbasedev->name); + } + + multifd->load_buf_idx++; + } + + if (!vfio_load_bufs_thread_load_config(vbasedev, errp)) { + goto thread_exit; + } + + ret =3D true; + +thread_exit: + /* + * Notify possibly waiting vfio_load_cleanup_load_bufs_thread() that + * this thread is exiting. + */ + multifd->load_bufs_thread_running =3D false; + qemu_cond_signal(&multifd->load_bufs_thread_finished_cond); + + trace_vfio_load_bufs_thread_end(vbasedev->name); + + return ret; +} + static VFIOMultifd *vfio_multifd_new(void) { VFIOMultifd *multifd =3D g_new(VFIOMultifd, 1); @@ -198,11 +374,41 @@ static VFIOMultifd *vfio_multifd_new(void) multifd->load_buf_idx_last =3D UINT32_MAX; qemu_cond_init(&multifd->load_bufs_buffer_ready_cond); =20 + multifd->load_bufs_thread_running =3D false; + multifd->load_bufs_thread_want_exit =3D false; + qemu_cond_init(&multifd->load_bufs_thread_finished_cond); + return multifd; } =20 +/* + * Terminates vfio_load_bufs_thread by setting + * multifd->load_bufs_thread_want_exit and signalling all the conditions + * the thread could be blocked on. + * + * Waits for the thread to signal that it had finished. + */ +static void vfio_load_cleanup_load_bufs_thread(VFIOMultifd *multifd) +{ + /* The lock order is load_bufs_mutex -> BQL so unlock BQL here first */ + bql_unlock(); + WITH_QEMU_LOCK_GUARD(&multifd->load_bufs_mutex) { + while (multifd->load_bufs_thread_running) { + multifd->load_bufs_thread_want_exit =3D true; + + qemu_cond_signal(&multifd->load_bufs_buffer_ready_cond); + qemu_cond_wait(&multifd->load_bufs_thread_finished_cond, + &multifd->load_bufs_mutex); + } + } + bql_lock(); +} + static void vfio_multifd_free(VFIOMultifd *multifd) { + vfio_load_cleanup_load_bufs_thread(multifd); + + qemu_cond_destroy(&multifd->load_bufs_thread_finished_cond); vfio_state_buffers_destroy(&multifd->load_bufs); qemu_cond_destroy(&multifd->load_bufs_buffer_ready_cond); qemu_mutex_destroy(&multifd->load_bufs_mutex); @@ -244,3 +450,23 @@ bool vfio_multifd_setup(VFIODevice *vbasedev, bool all= oc_multifd, Error **errp) =20 return true; } + +int vfio_multifd_switchover_start(VFIODevice *vbasedev) +{ + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + + assert(multifd); + + /* The lock order is load_bufs_mutex -> BQL so unlock BQL here first */ + bql_unlock(); + WITH_QEMU_LOCK_GUARD(&multifd->load_bufs_mutex) { + assert(!multifd->load_bufs_thread_running); + multifd->load_bufs_thread_running =3D true; + } + bql_lock(); + + qemu_loadvm_start_load_thread(vfio_load_bufs_thread, vbasedev); + + return 0; +} diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 2cdb92356e0a2afb64109c10536c857b19f7e7c5..815ad8fc93c7155d831838f9ccf= cfd1955e588bc 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -788,6 +788,17 @@ static bool vfio_switchover_ack_needed(void *opaque) return vfio_precopy_supported(vbasedev); } =20 +static int vfio_switchover_start(void *opaque) +{ + VFIODevice *vbasedev =3D opaque; + + if (vfio_multifd_transfer_enabled(vbasedev)) { + return vfio_multifd_switchover_start(vbasedev); + } + + return 0; +} + static const SaveVMHandlers savevm_vfio_handlers =3D { .save_prepare =3D vfio_save_prepare, .save_setup =3D vfio_save_setup, @@ -806,6 +817,7 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { * Multifd support */ .load_state_buffer =3D vfio_multifd_load_state_buffer, + .switchover_start =3D vfio_switchover_start, }; =20 /* ---------------------------------------------------------------------- = */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 404ea079b25c49fe25f4c9b05f0cde4f0536fdd7..d6b7e34faa392890aad34173b34= fd8c8c9df5e84 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -149,12 +149,19 @@ vfio_display_edid_update(uint32_t prefx, uint32_t pre= fy) "%ux%u" vfio_display_edid_write_error(void) "" =20 # migration.c +vfio_load_bufs_thread_start(const char *name) " (%s)" +vfio_load_bufs_thread_end(const char *name) " (%s)" vfio_load_cleanup(const char *name) " (%s)" vfio_load_device_config_state_start(const char *name) " (%s)" vfio_load_device_config_state_end(const char *name) " (%s)" vfio_load_state(const char *name, uint64_t data) " (%s) data 0x%"PRIx64 vfio_load_state_device_data(const char *name, uint64_t data_size, int ret)= " (%s) size %"PRIu64" ret %d" vfio_load_state_device_buffer_incoming(const char *name, uint32_t idx) " (= %s) idx %"PRIu32 +vfio_load_state_device_buffer_start(const char *name) " (%s)" +vfio_load_state_device_buffer_starved(const char *name, uint32_t idx) " (%= s) idx %"PRIu32 +vfio_load_state_device_buffer_load_start(const char *name, uint32_t idx) "= (%s) idx %"PRIu32 +vfio_load_state_device_buffer_load_end(const char *name, uint32_t idx) " (= %s) idx %"PRIu32 +vfio_load_state_device_buffer_end(const char *name) " (%s)" vfio_migration_realize(const char *name) " (%s)" vfio_migration_set_device_state(const char *name, const char *state) " (%s= ) state %s" vfio_migration_set_state(const char *name, const char *new_state, const ch= ar *recover_state) " (%s) new state %s, recover state %s" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270993; cv=none; d=zohomail.com; s=zohoarc; b=G95acrzLTmUPfr+iEzm1+HaontrwngT35NSLOSvEiUmktAWMoNAPRmJSoURf8QCMcgKqDBueeqsoBQD4HWFfOqSP4yEltDjKaeS1NJgfv+iF4nzE0Vf+g/obiY/4iwck/TZTHdEgGl0VNADGQLc+QZcyi0Ag64+9rpoZQYh5S1A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270993; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=L1KeHcnkM0S5lcdMHwWh6X32Ga/cnuREcHAcx8VowSQ=; b=SUUkCuhhE8pppgsOChJ627BnDXn7mgOzXZsr6VprMRN2hgaeMk3XWj+vfWSyUnTPyJ/RrZPiki6tl3pyo1zM2PX5oXA1cBLdC+II/Zx2jtDgLMFA0FL4pZREgWfXoT3QlUpYiwouvpz7Shjl/H5wVNylzpFL71zkMNH3tSiWrI0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270993262481.81534324758275; Thu, 6 Mar 2025 06:23:13 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3h-0002wL-3F; Thu, 06 Mar 2025 09:18:45 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1O-0000jl-M7 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:29 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1L-00023E-Dg for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:20 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-161-pXcvA0plN6SFiENyJW-_Rw-1; Thu, 06 Mar 2025 09:16:15 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 770461801A00; Thu, 6 Mar 2025 14:16:14 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 131D618009BC; Thu, 6 Mar 2025 14:16:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270578; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L1KeHcnkM0S5lcdMHwWh6X32Ga/cnuREcHAcx8VowSQ=; b=i6rT2QD2vOn4ujEcuGbp9SHXrHd91Smvzd/FTpc8vqIePU8g/Kx0bcl+tukBzqxW6FdXFu 1mv2Es88zHuBNSRkEx3ngSehcdYMIal5I5ODgP6IndDKiqf6IFBn40drKlExWVPCbUdNLT Tgwn1s31feXNmNPtM80X5LnoQa5kMb4= X-MC-Unique: pXcvA0plN6SFiENyJW-_Rw-1 X-Mimecast-MFC-AGG-ID: pXcvA0plN6SFiENyJW-_Rw_1741270574 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , Fabiano Rosas , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 37/42] migration/qemu-file: Define g_autoptr() cleanup function for QEMUFile Date: Thu, 6 Mar 2025 15:14:13 +0100 Message-ID: <20250306141419.2015340-38-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270994459019000 From: "Maciej S. Szmigiero" Automatic memory management helps avoid memory safety issues. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/2fd01d773a783d572dcf538a064a98cc09= e75c12.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- migration/qemu-file.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/migration/qemu-file.h b/migration/qemu-file.h index 3e47a20621a7e9b57822f0f8911de02dd5e63293..f5b9f430e04b1933747eba80587= f9d88726c56a8 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -33,6 +33,8 @@ QEMUFile *qemu_file_new_input(QIOChannel *ioc); QEMUFile *qemu_file_new_output(QIOChannel *ioc); int qemu_fclose(QEMUFile *f); =20 +G_DEFINE_AUTOPTR_CLEANUP_FUNC(QEMUFile, qemu_fclose) + /* * qemu_file_transferred: * --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271020; cv=none; d=zohomail.com; s=zohoarc; b=R5MM6/5foh6bNeJDBIvWxX++nlNvSYgFr+GTaLZOk15jefKXGgUFI6rn4R+ED81mK3oeayalOLdL+98IUAAprPt3TV+5pnkrfbce+0ml/W6Csi6FQWEDALiWSlL4FXlmt1EGltyjVdZMIA0J6lASXDUo0dH5AiK3OHlGgwRoL6A= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271020; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=J9csUtQRcZmArM3bAaUsw3uaSxuokMjwDwmzHM3/r20=; b=DKOXH2xCJKjetUjDPzFTjvd6mcqVYyNQKHjkX8dfTbtALkKPIsNakDuAOEvAE3a00COlsHmJo1bI6T6GwytFv3d6u6FboWdGGqV9E+8Tr7oITiX+hAm3ZkuIK1ZdoBo17EMA0uj0KjezTEsey+5pBobh6m8KkrwTXlEY7PYDkcc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271020459634.3542052284186; Thu, 6 Mar 2025 06:23:40 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC4M-0004xH-3l; Thu, 06 Mar 2025 09:19:26 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC2A-00014Z-T1 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:17:11 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC28-0002B5-V5 for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:17:10 -0500 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-270-yohFUGqtPiW0WZntNgfejw-1; Thu, 06 Mar 2025 09:16:17 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 976441955DC0; Thu, 6 Mar 2025 14:16:16 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F338418001D3; Thu, 6 Mar 2025 14:16:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J9csUtQRcZmArM3bAaUsw3uaSxuokMjwDwmzHM3/r20=; b=VBXr1WZgFw0bACXEIl2HdJLcVb7BOM1u0bQcogDxMlsoF7R8AnTFin8+NCreQlFC943Plb vJQ4ndktvJvsXwy3ATR3toVw4zzQkMP2aoajP3gi0Ybh1QbRqfPwkZzxofJyDxRdSjVD0g Gkgd5DEPD2vFeGaNp+2sw69t4jKkz10= X-MC-Unique: yohFUGqtPiW0WZntNgfejw-1 X-Mimecast-MFC-AGG-ID: yohFUGqtPiW0WZntNgfejw_1741270576 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 38/42] vfio/migration: Multifd device state transfer support - config loading support Date: Thu, 6 Mar 2025 15:14:14 +0100 Message-ID: <20250306141419.2015340-39-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271021773019100 From: "Maciej S. Szmigiero" Load device config received via multifd using the existing machinery behind vfio_load_device_config_state(). Also, make sure to process the relevant main migration channel flags. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/5dbd3f3703ec1097da2cf82a7262233452= 146fee.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 2 ++ hw/vfio/migration-multifd.c | 49 +++++++++++++++++++++++++++++++++-- hw/vfio/migration.c | 9 ++++++- 3 files changed, 57 insertions(+), 3 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 40382390692dcd772f2ceb6689d9b31dc5e8b8d5..9d72ac1eae8a50e06f32d1c4633= 2c9fd3adddce2 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -298,6 +298,8 @@ void vfio_mig_add_bytes_transferred(unsigned long val); bool vfio_device_state_is_running(VFIODevice *vbasedev); bool vfio_device_state_is_precopy(VFIODevice *vbasedev); =20 +int vfio_load_device_config_state(QEMUFile *f, void *opaque); + #ifdef CONFIG_LINUX int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info); diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index 2eef27604e4f8f1f4db74d132a4d4bf2906a4cc1..1d81233c755fec825d94c040ff6= cf87eb6551f01 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -17,6 +17,7 @@ #include "qemu/lockable.h" #include "qemu/main-loop.h" #include "qemu/thread.h" +#include "io/channel-buffer.h" #include "migration/qemu-file.h" #include "migration-multifd.h" #include "trace.h" @@ -193,8 +194,52 @@ bool vfio_multifd_load_state_buffer(void *opaque, char= *data, size_t data_size, static bool vfio_load_bufs_thread_load_config(VFIODevice *vbasedev, Error **errp) { - error_setg(errp, "not yet there"); - return false; + VFIOMigration *migration =3D vbasedev->migration; + VFIOMultifd *multifd =3D migration->multifd; + VFIOStateBuffer *lb; + g_autoptr(QIOChannelBuffer) bioc =3D NULL; + g_autoptr(QEMUFile) f_out =3D NULL, f_in =3D NULL; + uint64_t mig_header; + int ret; + + assert(multifd->load_buf_idx =3D=3D multifd->load_buf_idx_last); + lb =3D vfio_state_buffers_at(&multifd->load_bufs, multifd->load_buf_id= x); + assert(lb->is_present); + + bioc =3D qio_channel_buffer_new(lb->len); + qio_channel_set_name(QIO_CHANNEL(bioc), "vfio-device-config-load"); + + f_out =3D qemu_file_new_output(QIO_CHANNEL(bioc)); + qemu_put_buffer(f_out, (uint8_t *)lb->data, lb->len); + + ret =3D qemu_fflush(f_out); + if (ret) { + error_setg(errp, "%s: load config state flush failed: %d", + vbasedev->name, ret); + return false; + } + + qio_channel_io_seek(QIO_CHANNEL(bioc), 0, 0, NULL); + f_in =3D qemu_file_new_input(QIO_CHANNEL(bioc)); + + mig_header =3D qemu_get_be64(f_in); + if (mig_header !=3D VFIO_MIG_FLAG_DEV_CONFIG_STATE) { + error_setg(errp, "%s: expected FLAG_DEV_CONFIG_STATE but got %" PR= Ix64, + vbasedev->name, mig_header); + return false; + } + + bql_lock(); + ret =3D vfio_load_device_config_state(f_in, vbasedev); + bql_unlock(); + + if (ret < 0) { + error_setg(errp, "%s: vfio_load_device_config_state() failed: %d", + vbasedev->name, ret); + return false; + } + + return true; } =20 static VFIOStateBuffer *vfio_load_state_buffer_get(VFIOMultifd *multifd) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 815ad8fc93c7155d831838f9ccfcfd1955e588bc..2ca3fa08d48667e1c338d3e1480= 23afed7251ac4 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -264,7 +264,7 @@ static int vfio_save_device_config_state(QEMUFile *f, v= oid *opaque, return ret; } =20 -static int vfio_load_device_config_state(QEMUFile *f, void *opaque) +int vfio_load_device_config_state(QEMUFile *f, void *opaque) { VFIODevice *vbasedev =3D opaque; uint64_t data; @@ -723,6 +723,13 @@ static int vfio_load_state(QEMUFile *f, void *opaque, = int version_id) switch (data) { case VFIO_MIG_FLAG_DEV_CONFIG_STATE: { + if (vfio_multifd_transfer_enabled(vbasedev)) { + error_report("%s: got DEV_CONFIG_STATE in main migration " + "channel but doing multifd transfer", + vbasedev->name); + return -EINVAL; + } + return vfio_load_device_config_state(f, opaque); } case VFIO_MIG_FLAG_DEV_SETUP_STATE: --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271057; cv=none; d=zohomail.com; s=zohoarc; b=UamdvvGvatc0Hyr5izkidDOsX8K34ZRIlx05LqE+MOw/yiu8sfi9Iu9VAuBhgXhe5pgIxh7gcbTz8czeyR+zJhLxTW048VboieTu8a+ueL8gFNq+spS6CT7YnXRRGtZZmtfNgnsYiSTw8E/jqk4px7gtzOU0iUS6DwficI0FByE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271057; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=3HZy1XPT45Um+BbusYDbqxnkO9sHgbG8uYLs/KQVWSs=; b=clGnHEy2i0DSO/EVBb/MYCle8QjcXbQIUMqQSPuFJICOv8bXLpr6jlh6aYoHbK0UanZTj7vSIkceGibKdRyyOhbFBJocYmqIV+pQkraVyN2fvfIB0FAuDgW/zI6IEU3TOpC15zQwEsPKrEr6XhzsDhSmuqMfxjgzUBSJZDq4tlM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271057567373.1227508537688; Thu, 6 Mar 2025 06:24:17 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC4L-0004p7-5q; Thu, 06 Mar 2025 09:19:25 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1d-0000s7-Vr for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1X-00027c-NT for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:33 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-454-AiZFGSlgNB2d1z1Hav11uw-1; Thu, 06 Mar 2025 09:16:20 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 0F95F1828A9C; Thu, 6 Mar 2025 14:16:19 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1E5311801748; Thu, 6 Mar 2025 14:16:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270590; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3HZy1XPT45Um+BbusYDbqxnkO9sHgbG8uYLs/KQVWSs=; b=FZzkNDDkWchX27o/C5DAdYp03qI2N9A/3SnwoJUtWm2dhEU87O0c/1/WKX2o+g6yXW8CjH 8YmW+56iIYBiacnTZE4g4Bkw15yCis/Gr7JO4KJewN7TI2zvsyWnoU8t41xiRIlhVsZTFQ CuJzs797gq2QhcFSS5z2QV1nRsJktWc= X-MC-Unique: AiZFGSlgNB2d1z1Hav11uw-1 X-Mimecast-MFC-AGG-ID: AiZFGSlgNB2d1z1Hav11uw_1741270579 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 39/42] vfio/migration: Multifd device state transfer support - send side Date: Thu, 6 Mar 2025 15:14:15 +0100 Message-ID: <20250306141419.2015340-40-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271061127019100 From: "Maciej S. Szmigiero" Implement the multifd device state transfer via additional per-device thread inside save_live_complete_precopy_thread handler. Switch between doing the data transfer in the new handler and doing it in the old save_state handler depending if VFIO multifd transfer is enabled or not. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/4d727e2e0435e0022d50004e4740776328= 30e08d.1741124640.git.maciej.szmigiero@oracle.com [ clg: - Reordered savevm_vfio_handlers - Updated save_live_complete_precopy* documentation ] Signed-off-by: C=C3=A9dric Le Goater --- docs/devel/migration/vfio.rst | 19 ++++- hw/vfio/migration-multifd.h | 6 ++ include/hw/vfio/vfio-common.h | 6 ++ hw/vfio/migration-multifd.c | 142 ++++++++++++++++++++++++++++++++++ hw/vfio/migration.c | 22 ++++-- hw/vfio/trace-events | 2 + 6 files changed, 189 insertions(+), 8 deletions(-) diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index d6cf60890c43f2ebf085940a233f8b85a900b6b3..a803a09bc15619cfc46384908d5= 0860ca135d26e 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -71,11 +71,23 @@ VFIO implements the device hooks for the iterative appr= oach as follows: reassembles the multifd received data and loads it in-order into the dev= ice. In the non-multifd mode this function is a NOP. =20 -* A ``save_state`` function to save the device config space if it is prese= nt. +* A ``save_state`` function to save the device config space if it is prese= nt + in the non-multifd mode. + In the multifd mode it just emits either a dummy EOS marker. =20 * A ``save_live_complete_precopy`` function that sets the VFIO device in _STOP_COPY state and iteratively copies the data for the VFIO device unt= il the vendor driver indicates that no data remains. + In the multifd mode it just emits a dummy EOS marker. + +* A ``save_live_complete_precopy_thread`` function that in the multifd mode + provides thread handler performing multifd device state transfer. + It sets the VFIO device to _STOP_COPY state, iteratively reads the data + from the VFIO device and queues it for multifd transmission until the ve= ndor + driver indicates that no data remains. + After that, it saves the device config space and queues it for multifd + transfer too. + In the non-multifd mode this thread is a NOP. =20 * A ``load_state`` function that loads the config section and the data sections that are generated by the save functions above. @@ -184,8 +196,11 @@ Live migration save path Then the VFIO device is put in _STOP_COPY state (FINISH_MIGRATE, _ACTIVE, _STOP_COPY) .save_live_complete_precopy() is called for each active device - For the VFIO device, iterate in .save_live_complete_precopy() until + For the VFIO device: in the non-multifd mode iterate in + .save_live_complete_precopy() until pending data is 0 + In the multifd mode this iteration is done in + .save_live_complete_precopy_thread() instead. | (POSTMIGRATE, _COMPLETED, _STOP_COPY) Migraton thread schedules cleanup bottom half and exits diff --git a/hw/vfio/migration-multifd.h b/hw/vfio/migration-multifd.h index f0d28fcef2ea737fc550a34ff266a34848bca67c..a664051eb8ae03bc41cb7f9362a= ce840f41066ff 100644 --- a/hw/vfio/migration-multifd.h +++ b/hw/vfio/migration-multifd.h @@ -23,6 +23,12 @@ bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev); bool vfio_multifd_load_state_buffer(void *opaque, char *data, size_t data_= size, Error **errp); =20 +void vfio_multifd_emit_dummy_eos(VFIODevice *vbasedev, QEMUFile *f); + +bool +vfio_multifd_save_complete_precopy_thread(SaveLiveCompletePrecopyThreadDat= a *d, + Error **errp); + int vfio_multifd_switchover_start(VFIODevice *vbasedev); =20 #endif diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 9d72ac1eae8a50e06f32d1c46332c9fd3adddce2..961931d9f457e170f26afabc484= a1e7c39675533 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -298,6 +298,7 @@ void vfio_mig_add_bytes_transferred(unsigned long val); bool vfio_device_state_is_running(VFIODevice *vbasedev); bool vfio_device_state_is_precopy(VFIODevice *vbasedev); =20 +int vfio_save_device_config_state(QEMUFile *f, void *opaque, Error **errp); int vfio_load_device_config_state(QEMUFile *f, void *opaque); =20 #ifdef CONFIG_LINUX @@ -314,6 +315,11 @@ struct vfio_info_cap_header * vfio_get_device_info_cap(struct vfio_device_info *info, uint16_t id); struct vfio_info_cap_header * vfio_get_cap(void *ptr, uint32_t cap_offset, uint16_t id); + +int vfio_migration_set_state(VFIODevice *vbasedev, + enum vfio_device_mig_state new_state, + enum vfio_device_mig_state recover_state, + Error **errp); #endif =20 bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp); diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index 1d81233c755fec825d94c040ff6cf87eb6551f01..bfb9a72fa4503e2ea00731a2ba5= 28a54d09ed4c5 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -496,6 +496,148 @@ bool vfio_multifd_setup(VFIODevice *vbasedev, bool al= loc_multifd, Error **errp) return true; } =20 +void vfio_multifd_emit_dummy_eos(VFIODevice *vbasedev, QEMUFile *f) +{ + assert(vfio_multifd_transfer_enabled(vbasedev)); + + /* + * Emit dummy NOP data on the main migration channel since the actual + * device state transfer is done via multifd channels. + */ + qemu_put_be64(f, VFIO_MIG_FLAG_END_OF_STATE); +} + +static bool +vfio_save_complete_precopy_thread_config_state(VFIODevice *vbasedev, + char *idstr, + uint32_t instance_id, + uint32_t idx, + Error **errp) +{ + g_autoptr(QIOChannelBuffer) bioc =3D NULL; + g_autoptr(QEMUFile) f =3D NULL; + int ret; + g_autofree VFIODeviceStatePacket *packet =3D NULL; + size_t packet_len; + + bioc =3D qio_channel_buffer_new(0); + qio_channel_set_name(QIO_CHANNEL(bioc), "vfio-device-config-save"); + + f =3D qemu_file_new_output(QIO_CHANNEL(bioc)); + + if (vfio_save_device_config_state(f, vbasedev, errp)) { + return false; + } + + ret =3D qemu_fflush(f); + if (ret) { + error_setg(errp, "%s: save config state flush failed: %d", + vbasedev->name, ret); + return false; + } + + packet_len =3D sizeof(*packet) + bioc->usage; + packet =3D g_malloc0(packet_len); + packet->version =3D VFIO_DEVICE_STATE_PACKET_VER_CURRENT; + packet->idx =3D idx; + packet->flags =3D VFIO_DEVICE_STATE_CONFIG_STATE; + memcpy(&packet->data, bioc->data, bioc->usage); + + if (!multifd_queue_device_state(idstr, instance_id, + (char *)packet, packet_len)) { + error_setg(errp, "%s: multifd config data queuing failed", + vbasedev->name); + return false; + } + + vfio_mig_add_bytes_transferred(packet_len); + + return true; +} + +/* + * This thread is spawned by the migration core directly via + * .save_live_complete_precopy_thread SaveVMHandler. + * + * It exits after either: + * * completing saving the remaining device state and device config, OR: + * * encountering some error while doing the above, OR: + * * being forcefully aborted by the migration core by + * multifd_device_state_save_thread_should_exit() returning true. + */ +bool +vfio_multifd_save_complete_precopy_thread(SaveLiveCompletePrecopyThreadDat= a *d, + Error **errp) +{ + VFIODevice *vbasedev =3D d->handler_opaque; + VFIOMigration *migration =3D vbasedev->migration; + bool ret =3D false; + g_autofree VFIODeviceStatePacket *packet =3D NULL; + uint32_t idx; + + if (!vfio_multifd_transfer_enabled(vbasedev)) { + /* Nothing to do, vfio_save_complete_precopy() does the transfer. = */ + return true; + } + + trace_vfio_save_complete_precopy_thread_start(vbasedev->name, + d->idstr, d->instance_id= ); + + /* We reach here with device state STOP or STOP_COPY only */ + if (vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_STOP_COPY, + VFIO_DEVICE_STATE_STOP, errp)) { + goto thread_exit; + } + + packet =3D g_malloc0(sizeof(*packet) + migration->data_buffer_size); + packet->version =3D VFIO_DEVICE_STATE_PACKET_VER_CURRENT; + + for (idx =3D 0; ; idx++) { + ssize_t data_size; + size_t packet_size; + + if (multifd_device_state_save_thread_should_exit()) { + error_setg(errp, "operation cancelled"); + goto thread_exit; + } + + data_size =3D read(migration->data_fd, &packet->data, + migration->data_buffer_size); + if (data_size < 0) { + error_setg(errp, "%s: reading state buffer %" PRIu32 " failed:= %d", + vbasedev->name, idx, errno); + goto thread_exit; + } else if (data_size =3D=3D 0) { + break; + } + + packet->idx =3D idx; + packet_size =3D sizeof(*packet) + data_size; + + if (!multifd_queue_device_state(d->idstr, d->instance_id, + (char *)packet, packet_size)) { + error_setg(errp, "%s: multifd data queuing failed", vbasedev->= name); + goto thread_exit; + } + + vfio_mig_add_bytes_transferred(packet_size); + } + + if (!vfio_save_complete_precopy_thread_config_state(vbasedev, + d->idstr, + d->instance_id, + idx, errp)) { + goto thread_exit; + } + + ret =3D true; + +thread_exit: + trace_vfio_save_complete_precopy_thread_end(vbasedev->name, ret); + + return ret; +} + int vfio_multifd_switchover_start(VFIODevice *vbasedev) { VFIOMigration *migration =3D vbasedev->migration; diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 2ca3fa08d48667e1c338d3e148023afed7251ac4..416643ddd69ab64c252eadd0770= 2a9d5dad0db47 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -120,10 +120,10 @@ static void vfio_migration_set_device_state(VFIODevic= e *vbasedev, vfio_migration_send_event(vbasedev); } =20 -static int vfio_migration_set_state(VFIODevice *vbasedev, - enum vfio_device_mig_state new_state, - enum vfio_device_mig_state recover_sta= te, - Error **errp) +int vfio_migration_set_state(VFIODevice *vbasedev, + enum vfio_device_mig_state new_state, + enum vfio_device_mig_state recover_state, + Error **errp) { VFIOMigration *migration =3D vbasedev->migration; uint64_t buf[DIV_ROUND_UP(sizeof(struct vfio_device_feature) + @@ -238,8 +238,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vb= asedev, return ret; } =20 -static int vfio_save_device_config_state(QEMUFile *f, void *opaque, - Error **errp) +int vfio_save_device_config_state(QEMUFile *f, void *opaque, Error **errp) { VFIODevice *vbasedev =3D opaque; int ret; @@ -638,6 +637,11 @@ static int vfio_save_complete_precopy(QEMUFile *f, voi= d *opaque) int ret; Error *local_err =3D NULL; =20 + if (vfio_multifd_transfer_enabled(vbasedev)) { + vfio_multifd_emit_dummy_eos(vbasedev, f); + return 0; + } + trace_vfio_save_complete_precopy_start(vbasedev->name); =20 /* We reach here with device state STOP or STOP_COPY only */ @@ -669,6 +673,11 @@ static void vfio_save_state(QEMUFile *f, void *opaque) Error *local_err =3D NULL; int ret; =20 + if (vfio_multifd_transfer_enabled(vbasedev)) { + vfio_multifd_emit_dummy_eos(vbasedev, f); + return; + } + ret =3D vfio_save_device_config_state(f, opaque, &local_err); if (ret) { error_prepend(&local_err, @@ -825,6 +834,7 @@ static const SaveVMHandlers savevm_vfio_handlers =3D { */ .load_state_buffer =3D vfio_multifd_load_state_buffer, .switchover_start =3D vfio_switchover_start, + .save_live_complete_precopy_thread =3D vfio_multifd_save_complete_prec= opy_thread, }; =20 /* ---------------------------------------------------------------------- = */ diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index d6b7e34faa392890aad34173b34fd8c8c9df5e84..9347e3a5f6607ec6907f9b426da= 9ab90553292cf 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -171,6 +171,8 @@ vfio_save_block_precopy_empty_hit(const char *name) " (= %s)" vfio_save_cleanup(const char *name) " (%s)" vfio_save_complete_precopy(const char *name, int ret) " (%s) ret %d" vfio_save_complete_precopy_start(const char *name) " (%s)" +vfio_save_complete_precopy_thread_start(const char *name, const char *idst= r, uint32_t instance_id) " (%s) idstr %s instance %"PRIu32 +vfio_save_complete_precopy_thread_end(const char *name, int ret) " (%s) re= t %d" vfio_save_device_config_state(const char *name) " (%s)" vfio_save_iterate(const char *name, uint64_t precopy_init_size, uint64_t p= recopy_dirty_size) " (%s) precopy initial size %"PRIu64" precopy dirty size= %"PRIu64 vfio_save_iterate_start(const char *name) " (%s)" --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741270870; cv=none; d=zohomail.com; s=zohoarc; b=n06f95fAC8b7gQCpmyxn+GyrK5LWStSkMpwPxQyme0tCDPONgisLmRsWwtm4p9Ssb0mb39MaxsteDfgwMX0ICVPTyb7vkJclldFE6xDVqAmd4ZUJRlZTlTlAqOxp09ccxM00Ybchkrzj9t3QAqqfxhQVMFucSvmT7STQMpiZpLU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741270870; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=rUvV7qq9uzaYU21iu4zHJBfeyJeO0F4FuZGYn1fB/w8=; b=gPOzVbZm88FXhmOvqArGBNR30drFs17JVLUXZqudhujLcCEzzYdNlfNzvVcnKwVsaj2yJetCGPAe7ggudoqG2TRs8T2UaPFBOLwVUBjeFyVcDliP2GCDg/NHYffLd/RWxtzbM9mNFx8PTIsDlhn1z8ME+hwpQ++W7XqTuoPRRKw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741270870008707.3814316008695; Thu, 6 Mar 2025 06:21:10 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3l-00035x-Uu; Thu, 06 Mar 2025 09:18:51 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1S-0000lW-Ie for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:34 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1Q-00025L-It for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:26 -0500 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-426-jnK-QQmxP_GVoFLVmlVLbw-1; Thu, 06 Mar 2025 09:16:22 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 38B751808869; Thu, 6 Mar 2025 14:16:21 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8BE051801748; Thu, 6 Mar 2025 14:16:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rUvV7qq9uzaYU21iu4zHJBfeyJeO0F4FuZGYn1fB/w8=; b=CHjsT1AHfx7QD8oOKjiSdr/pYp7YCYQC+MlX82eW8F6s/YjWHgTIjRzhNXXheHpWsCFQ+j Qpn/Hf9ij2ji38ZgP5jvq3ATTMnhRbjWbo1Vzh+AL5B7Qyq+kJXdzNlSo/dOsSewwuM9fH SlF7lR4UQk5tv2tT9XVboE/4Bw7Ee1w= X-MC-Unique: jnK-QQmxP_GVoFLVmlVLbw-1 X-Mimecast-MFC-AGG-ID: jnK-QQmxP_GVoFLVmlVLbw_1741270581 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 40/42] vfio/migration: Add x-migration-multifd-transfer VFIO property Date: Thu, 6 Mar 2025 15:14:16 +0100 Message-ID: <20250306141419.2015340-41-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741270872056019000 From: "Maciej S. Szmigiero" This property allows configuring whether to transfer the particular device state via multifd channels when live migrating that device. It defaults to AUTO, which means that VFIO device state transfer via multifd channels is attempted in configurations that otherwise support it. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/d6dbb326e3d53c7104d62c96c9e3dd64e1= c7b940.1741124640.git.maciej.szmigiero@oracle.com [ clg: Added documentation ] Signed-off-by: C=C3=A9dric Le Goater --- docs/devel/migration/vfio.rst | 15 +++++++++++++++ include/hw/vfio/vfio-common.h | 2 ++ hw/vfio/migration-multifd.c | 18 +++++++++++++++++- hw/vfio/pci.c | 7 +++++++ 4 files changed, 41 insertions(+), 1 deletion(-) diff --git a/docs/devel/migration/vfio.rst b/docs/devel/migration/vfio.rst index a803a09bc15619cfc46384908d50860ca135d26e..673e354754c8ad2f6ed5364516e= f72c75d084610 100644 --- a/docs/devel/migration/vfio.rst +++ b/docs/devel/migration/vfio.rst @@ -232,3 +232,18 @@ Postcopy =3D=3D=3D=3D=3D=3D=3D=3D =20 Postcopy migration is currently not supported for VFIO devices. + +Multifd +=3D=3D=3D=3D=3D=3D=3D + +Starting from QEMU version 10.0 there's a possibility to transfer VFIO dev= ice +_STOP_COPY state via multifd channels. This helps reduce downtime - especi= ally +with multiple VFIO devices or with devices having a large migration state. +As an additional benefit, setting the VFIO device to _STOP_COPY state and +saving its config space is also parallelized (run in a separate thread) in +such migration mode. + +The multifd VFIO device state transfer is controlled by +"x-migration-multifd-transfer" VFIO device property. This property default= s to +AUTO, which means that VFIO device state transfer via multifd channels is +attempted in configurations that otherwise support it. diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 961931d9f457e170f26afabc484a1e7c39675533..04b123a6c929a8f47d740fc5433= b54dadd32f731 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -91,6 +91,7 @@ typedef struct VFIOMigration { uint64_t mig_flags; uint64_t precopy_init_size; uint64_t precopy_dirty_size; + bool multifd_transfer; VFIOMultifd *multifd; bool initial_data_sent; =20 @@ -153,6 +154,7 @@ typedef struct VFIODevice { bool no_mmap; bool ram_block_discard_allowed; OnOffAuto enable_migration; + OnOffAuto migration_multifd_transfer; bool migration_events; VFIODeviceOps *ops; unsigned int num_irqs; diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index bfb9a72fa4503e2ea00731a2ba528a54d09ed4c5..aacddc503bb8ba6c40e38c97ce9= e2b3ca9430e12 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -476,18 +476,34 @@ bool vfio_multifd_transfer_supported(void) =20 bool vfio_multifd_transfer_enabled(VFIODevice *vbasedev) { - return false; + VFIOMigration *migration =3D vbasedev->migration; + + return migration->multifd_transfer; } =20 bool vfio_multifd_setup(VFIODevice *vbasedev, bool alloc_multifd, Error **= errp) { VFIOMigration *migration =3D vbasedev->migration; =20 + if (vbasedev->migration_multifd_transfer =3D=3D ON_OFF_AUTO_AUTO) { + migration->multifd_transfer =3D vfio_multifd_transfer_supported(); + } else { + migration->multifd_transfer =3D + vbasedev->migration_multifd_transfer =3D=3D ON_OFF_AUTO_ON; + } + if (!vfio_multifd_transfer_enabled(vbasedev)) { /* Nothing further to check or do */ return true; } =20 + if (!vfio_multifd_transfer_supported()) { + error_setg(errp, + "%s: Multifd device transfer requested but unsupported = in the current config", + vbasedev->name); + return false; + } + if (alloc_multifd) { assert(!migration->multifd); migration->multifd =3D vfio_multifd_new(); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c1cee280ae4bbfc5a479aa5d1f2290065667a677..1bbf15cea32657ef2a9ee37b8e1= dee392d5dbcb7 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3381,6 +3381,9 @@ static const Property vfio_pci_dev_properties[] =3D { VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, vbasedev.enable_migration, ON_OFF_AUTO_AUTO), + DEFINE_PROP_ON_OFF_AUTO("x-migration-multifd-transfer", VFIOPCIDevice, + vbasedev.migration_multifd_transfer, + ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("migration-events", VFIOPCIDevice, vbasedev.migration_events, false), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), @@ -3553,6 +3556,10 @@ static void vfio_pci_dev_class_init(ObjectClass *kla= ss, void *data) "Skip config space check for Ven= dor Specific Capability. " "Setting to false will enforce s= trict checking of VSC content " "(DEBUG)"); + object_class_property_set_description(klass, /* 10.0 */ + "x-migration-multifd-transfer", + "Transfer this device state via " + "multifd channels when live migr= ating it"); } =20 static const TypeInfo vfio_pci_dev_info =3D { --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271036; cv=none; d=zohomail.com; s=zohoarc; b=f21/w3r6Zf7+7rGoDT4hnnjyzX4NSU8dZuo9xet63YXg2sFStoIH0V6Y8aDNESQKX/5XLIB1Kum2uR+MJ4NViiMDfvO3sGQPqinOX8HlT6pZsYUvKA8e+KtxxRcS+zHZbRAUCqFLbtNrXWxoK20W5xG9c8blT3rq65utjH+Tnlo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271036; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=BU+dkKqyovOGnsyjcACGTgAqOZVrzySUjaZmfUrJeks=; b=jeD24J2Ryr3EtINJ/ytm9/f0aqjn0ZTUns0x725Ht0kuDXiQcdyIR8NpheQ5LIvy552VJFdk7CA4/gnTRSSRtx9Kz0uz7yrtZIYnpd/8Y+gBreC/kbiR/lPHxlGozSKAWXPCGnEP0QHf2Io5CV37EogOLTUP+2o94QBLyAS1cUU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271036587562.8426941985192; Thu, 6 Mar 2025 06:23:56 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC3c-0002oo-Kt; Thu, 06 Mar 2025 09:18:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1U-0000lz-Ny for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:36 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1S-00026D-Km for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:28 -0500 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-685-Fn0hhD5bPHu-w_WB6J5lVw-1; Thu, 06 Mar 2025 09:16:24 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 91866180AF6C; Thu, 6 Mar 2025 14:16:23 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BC68A1801751; Thu, 6 Mar 2025 14:16:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270585; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BU+dkKqyovOGnsyjcACGTgAqOZVrzySUjaZmfUrJeks=; b=VzuaQK21lQxrOQXPkODR7qhYSaGWxgToekdL8LcRai6QcV8PRnBalU0KzURdlB1XH3TWC/ /MThY8IXzj/bpszYFvjpfJ3LkRdMzwYHdNEAAMfbfue5fpxuLG2b19jtywG+sCAV1tBtbE oEwW6LWgyMXaGZ7lDRT2NkTz1sWbuP8= X-MC-Unique: Fn0hhD5bPHu-w_WB6J5lVw-1 X-Mimecast-MFC-AGG-ID: Fn0hhD5bPHu-w_WB6J5lVw_1741270583 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 41/42] vfio/migration: Make x-migration-multifd-transfer VFIO property mutable Date: Thu, 6 Mar 2025 15:14:17 +0100 Message-ID: <20250306141419.2015340-42-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271037858019100 From: "Maciej S. Szmigiero" DEFINE_PROP_ON_OFF_AUTO() property isn't runtime-mutable so using it would mean that the source VM would need to decide upfront at startup time whether it wants to do a multifd device state transfer at some point. Source VM can run for a long time before being migrated so it is desirable to have a fallback mechanism to the old way of transferring VFIO device state if it turns to be necessary. This brings this property to the same mutability level as ordinary migration parameters, which too can be adjusted at the run time. Signed-off-by: Maciej S. Szmigiero Reviewed-by: C=C3=A9dric Le Goater Link: https://lore.kernel.org/qemu-devel/f2f2d66bda477da3e6cb8c0311006cff36= e8651d.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/vfio/migration-multifd.c | 4 ++++ hw/vfio/pci.c | 20 +++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/hw/vfio/migration-multifd.c b/hw/vfio/migration-multifd.c index aacddc503bb8ba6c40e38c97ce9e2b3ca9430e12..233724710b377afd40aabe5ef5b= 846f654f9865d 100644 --- a/hw/vfio/migration-multifd.c +++ b/hw/vfio/migration-multifd.c @@ -485,6 +485,10 @@ bool vfio_multifd_setup(VFIODevice *vbasedev, bool all= oc_multifd, Error **errp) { VFIOMigration *migration =3D vbasedev->migration; =20 + /* + * Make a copy of this setting at the start in case it is changed + * mid-migration. + */ if (vbasedev->migration_multifd_transfer =3D=3D ON_OFF_AUTO_AUTO) { migration->multifd_transfer =3D vfio_multifd_transfer_supported(); } else { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 1bbf15cea32657ef2a9ee37b8e1dee392d5dbcb7..fdbc15885d442ee20343e6b7127= d54a70b1688ce 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3357,6 +3357,8 @@ static void vfio_instance_init(Object *obj) pci_dev->cap_present |=3D QEMU_PCI_CAP_EXPRESS; } =20 +static PropertyInfo vfio_pci_migration_multifd_transfer_prop; + static const Property vfio_pci_dev_properties[] =3D { DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), DEFINE_PROP_UUID_NODEFAULT("vf-token", VFIOPCIDevice, vf_token), @@ -3381,9 +3383,10 @@ static const Property vfio_pci_dev_properties[] =3D { VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, vbasedev.enable_migration, ON_OFF_AUTO_AUTO), - DEFINE_PROP_ON_OFF_AUTO("x-migration-multifd-transfer", VFIOPCIDevice, - vbasedev.migration_multifd_transfer, - ON_OFF_AUTO_AUTO), + DEFINE_PROP("x-migration-multifd-transfer", VFIOPCIDevice, + vbasedev.migration_multifd_transfer, + vfio_pci_migration_multifd_transfer_prop, OnOffAuto, + .set_default =3D true, .defval.i =3D ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("migration-events", VFIOPCIDevice, vbasedev.migration_events, false), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), @@ -3608,6 +3611,17 @@ static const TypeInfo vfio_pci_nohotplug_dev_info = =3D { =20 static void register_vfio_pci_dev_type(void) { + /* + * Ordinary ON_OFF_AUTO property isn't runtime-mutable, but source VM = can + * run for a long time before being migrated so it is desirable to hav= e a + * fallback mechanism to the old way of transferring VFIO device state= if + * it turns to be necessary. + * The following makes this type of property have the same mutability = level + * as ordinary migration parameters. + */ + vfio_pci_migration_multifd_transfer_prop =3D qdev_prop_on_off_auto; + vfio_pci_migration_multifd_transfer_prop.realized_set_allowed =3D true; + type_register_static(&vfio_pci_dev_info); type_register_static(&vfio_pci_nohotplug_dev_info); } --=20 2.48.1 From nobody Wed Apr 2 13:31:15 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1741271329; cv=none; d=zohomail.com; s=zohoarc; b=bKSxvL8xf9ENF3pBQspyrcC1UUm5rndjFQXGvNSu/xhIqOxExwP+AozqolYbtfloyt2DFN7nCka3mzIQYFELYiFa7oaE1SdyfcY204jbe/um/9uSJmngD+mHgH6XIoJ0xSVICLhYG3xUn2QQQhd+IQYqxjeWwodBUIlv9C8XJuI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1741271329; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=XmAyrBFPHwb6k7X1yaZprES8+vgfxwM21eUICd7cFnY=; b=ZHiRknTZyDn48/oAK43lrbT2QjiA0/MhxAsmcvhfxzDT/UDSCNt2t9UYY41k7e6Z4ctKM1TXJJy/EibQpE/BBVx4uOnjTno88/JN19YsNs2YQ7UwH0+sQYWOIspFzov6qpJ2UmnUsJ/j2ujI6bLEzISyJjPXeuRtSY/GuppIn/o= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1741271329902527.0081718779351; Thu, 6 Mar 2025 06:28:49 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tqC4H-0004Yt-CI; Thu, 06 Mar 2025 09:19:21 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1g-0000sI-TV for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:46 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tqC1d-00028y-Mc for qemu-devel@nongnu.org; Thu, 06 Mar 2025 09:16:40 -0500 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-14-0ZhYLJgfNf-hBr8EyFWl9Q-1; Thu, 06 Mar 2025 09:16:27 -0500 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id CE25B1954218; Thu, 6 Mar 2025 14:16:26 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.33.141]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2E09F18009BC; Thu, 6 Mar 2025 14:16:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741270594; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XmAyrBFPHwb6k7X1yaZprES8+vgfxwM21eUICd7cFnY=; b=CPlY3kdk0r0BGiGahkJ4Gop4BGALQqJ7isjcDgDZG6WnBxZeEkNeqahr5GKb7Xx7jFL5Fo 449pVI6GeWSaW1ciXOPYDsgYM9xmOZKy5eg0eatdb0VkaEUlvtDS/BzHmky4YnlwNgYH6T R5wnyC8cXqOnnWoWkEN5Or0UCKSkgqM= X-MC-Unique: 0ZhYLJgfNf-hBr8EyFWl9Q-1 X-Mimecast-MFC-AGG-ID: 0ZhYLJgfNf-hBr8EyFWl9Q_1741270586 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Alex Williamson , "Maciej S. Szmigiero" , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 42/42] hw/core/machine: Add compat for x-migration-multifd-transfer VFIO property Date: Thu, 6 Mar 2025 15:14:18 +0100 Message-ID: <20250306141419.2015340-43-clg@redhat.com> In-Reply-To: <20250306141419.2015340-1-clg@redhat.com> References: <20250306141419.2015340-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1741271332172019000 From: "Maciej S. Szmigiero" Add a hw_compat entry for recently added x-migration-multifd-transfer VFIO property. Reviewed-by: C=C3=A9dric Le Goater Signed-off-by: Maciej S. Szmigiero Link: https://lore.kernel.org/qemu-devel/92c354f0457c152d1f267cc258c6967fff= 551cb1.1741124640.git.maciej.szmigiero@oracle.com Signed-off-by: C=C3=A9dric Le Goater --- hw/core/machine.c | 1 + 1 file changed, 1 insertion(+) diff --git a/hw/core/machine.c b/hw/core/machine.c index d1ddc3a3db595bd746e8f39b006a8bf9cd9a52d6..f52a4f2273b2fa0cfa58d2703be= 88bcf7fac2786 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -45,6 +45,7 @@ GlobalProperty hw_compat_9_2[] =3D { { "virtio-mem-pci", "vectors", "0" }, { "migration", "multifd-clean-tls-termination", "false" }, { "migration", "send-switchover-start", "off"}, + { "vfio-pci", "x-migration-multifd-transfer", "off" }, }; const size_t hw_compat_9_2_len =3D G_N_ELEMENTS(hw_compat_9_2); =20 --=20 2.48.1