From nobody Thu May 16 11:41:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710098449555583.270258028236; Sun, 10 Mar 2024 12:20:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rjOi6-0006Bs-Sh; Sun, 10 Mar 2024 15:19:50 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rjOi4-0006BM-LZ for qemu-devel@nongnu.org; Sun, 10 Mar 2024 15:19:48 -0400 Received: from mail.ozlabs.org ([2404:9400:2221:ea00::3] helo=gandalf.ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rjOhy-0001Qx-Bh for qemu-devel@nongnu.org; Sun, 10 Mar 2024 15:19:46 -0400 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4Tt8q93rztz4wyp; Mon, 11 Mar 2024 06:19:33 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4Tt8q76N41z4wcl; Mon, 11 Mar 2024 06:19:31 +1100 (AEDT) From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Steve Sistare , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 1/2] vfio: register container for cpr Date: Sun, 10 Mar 2024 20:19:23 +0100 Message-ID: <20240310191924.201680-2-clg@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240310191924.201680-1-clg@redhat.com> References: <20240310191924.201680-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2404:9400:2221:ea00::3; envelope-from=SRS0=sXSr=KQ=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -39 X-Spam_score: -4.0 X-Spam_bar: ---- X-Spam_report: (-4.0 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.248, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1710098451416100011 From: Steve Sistare Define entry points to perform per-container cpr-specific initialization and teardown. Signed-off-by: Steve Sistare Reviewed-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-common.h | 3 +++ hw/vfio/container.c | 11 ++++++++++- hw/vfio/cpr.c | 19 +++++++++++++++++++ hw/vfio/iommufd.c | 6 ++++++ hw/vfio/meson.build | 1 + 5 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 hw/vfio/cpr.c diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 4a6c262f778b9bf4bf5760ffd548b9b6ea5bad33..b9da6c08ef41174610eb92726c5= 90309a53696a3 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -205,6 +205,9 @@ void vfio_detach_device(VFIODevice *vbasedev); int vfio_kvm_device_add_fd(int fd, Error **errp); int vfio_kvm_device_del_fd(int fd, Error **errp); =20 +int vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **err= p); +void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); + extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; typedef QLIST_HEAD(VFIODeviceList, VFIODevice) VFIODeviceList; diff --git a/hw/vfio/container.c b/hw/vfio/container.c index bd25b9fbad2e717e63c2ab0e331186e5f63cef49..096d77eac3946a9c38fc2a98116= b93353f71f06e 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -621,10 +621,15 @@ static int vfio_connect_container(VFIOGroup *group, A= ddressSpace *as, goto free_container_exit; } =20 + ret =3D vfio_cpr_register_container(bcontainer, errp); + if (ret) { + goto free_container_exit; + } + ret =3D vfio_ram_block_discard_disable(container, true); if (ret) { error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"= ); - goto free_container_exit; + goto unregister_container_exit; } =20 assert(bcontainer->ops->setup); @@ -667,6 +672,9 @@ listener_release_exit: enable_discards_exit: vfio_ram_block_discard_disable(container, false); =20 +unregister_container_exit: + vfio_cpr_unregister_container(bcontainer); + free_container_exit: g_free(container); =20 @@ -710,6 +718,7 @@ static void vfio_disconnect_container(VFIOGroup *group) vfio_container_destroy(bcontainer); =20 trace_vfio_disconnect_container(container->fd); + vfio_cpr_unregister_container(bcontainer); close(container->fd); g_free(container); =20 diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c new file mode 100644 index 0000000000000000000000000000000000000000..3bede5494804e4aa291663c1e7c= 01d5dc5d3a407 --- /dev/null +++ b/hw/vfio/cpr.c @@ -0,0 +1,19 @@ +/* + * Copyright (c) 2021-2024 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or late= r. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "hw/vfio/vfio-common.h" +#include "qapi/error.h" + +int vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **err= p) +{ + return 0; +} + +void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer) +{ +} diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 9bfddc1360895413176a9f170e29e89027384a66..e1be2244947172367e2a1f85c59= 967c20bfcff77 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -411,6 +411,11 @@ found_container: goto err_listener_register; } =20 + ret =3D vfio_cpr_register_container(bcontainer, errp); + if (ret) { + goto err_listener_register; + } + /* * TODO: examine RAM_BLOCK_DISCARD stuff, should we do group level * for discarding incompatibility check as well? @@ -461,6 +466,7 @@ static void iommufd_cdev_detach(VFIODevice *vbasedev) iommufd_cdev_ram_block_discard_disable(false); } =20 + vfio_cpr_unregister_container(bcontainer); iommufd_cdev_detach_container(vbasedev, container); iommufd_cdev_container_destroy(container); vfio_put_address_space(space); diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index bb98493b53e858c53181e224f9cb46892838a8be..bba776f75cc718956b37f68cb54= 2b378a509d77f 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -5,6 +5,7 @@ vfio_ss.add(files( 'container-base.c', 'container.c', 'migration.c', + 'cpr.c', )) vfio_ss.add(when: 'CONFIG_PSERIES', if_true: files('spapr.c')) vfio_ss.add(when: 'CONFIG_IOMMUFD', if_true: files( --=20 2.44.0 From nobody Thu May 16 11:41:41 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1710098449536598.8128292164299; Sun, 10 Mar 2024 12:20:49 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rjOi2-0006BB-B9; Sun, 10 Mar 2024 15:19:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rjOi0-0006Ai-9J for qemu-devel@nongnu.org; Sun, 10 Mar 2024 15:19:44 -0400 Received: from mail.ozlabs.org ([2404:9400:2221:ea00::3] helo=gandalf.ozlabs.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rjOhw-0001Qy-R9 for qemu-devel@nongnu.org; Sun, 10 Mar 2024 15:19:43 -0400 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by gandalf.ozlabs.org (Postfix) with ESMTP id 4Tt8qC3ptxz4wyq; Mon, 11 Mar 2024 06:19:35 +1100 (AEDT) Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.ozlabs.org (Postfix) with ESMTPSA id 4Tt8qB0gGjz4wcl; Mon, 11 Mar 2024 06:19:33 +1100 (AEDT) From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Steve Sistare , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [PULL 2/2] vfio: allow cpr-reboot migration if suspended Date: Sun, 10 Mar 2024 20:19:24 +0100 Message-ID: <20240310191924.201680-3-clg@redhat.com> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240310191924.201680-1-clg@redhat.com> References: <20240310191924.201680-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2404:9400:2221:ea00::3; envelope-from=SRS0=sXSr=KQ=redhat.com=clg@ozlabs.org; helo=gandalf.ozlabs.org X-Spam_score_int: -39 X-Spam_score: -4.0 X-Spam_bar: ---- X-Spam_report: (-4.0 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.248, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1710098451365100008 From: Steve Sistare Allow cpr-reboot for vfio if the guest is in the suspended runstate. The guest drivers' suspend methods flush outstanding requests and re-initialize the devices, and thus there is no device state to save and restore. The user is responsible for suspending the guest before initiating cpr, such as by issuing guest-suspend-ram to the qemu guest agent. Relax the vfio blocker so it does not apply to cpr, and add a notifier that verifies the guest is suspended. Signed-off-by: Steve Sistare Reviewed-by: C=C3=A9dric Le Goater --- include/hw/vfio/vfio-container-base.h | 1 + hw/vfio/common.c | 2 +- hw/vfio/cpr.c | 20 ++++++++++++++++++++ hw/vfio/migration.c | 2 +- 4 files changed, 23 insertions(+), 2 deletions(-) diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-c= ontainer-base.h index b2813b0c117985425c842d91f011bb895955d738..3582d5f97a37877b2adfc0d0b06= 996c82403f8b7 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -49,6 +49,7 @@ typedef struct VFIOContainerBase { QLIST_ENTRY(VFIOContainerBase) next; QLIST_HEAD(, VFIODevice) device_list; GList *iova_ranges; + NotifierWithReturn cpr_reboot_notifier; } VFIOContainerBase; =20 typedef struct VFIOGuestIOMMU { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 059bfdc07a85e2eb908df828c1f42104d683e911..ff88c3f31ca660b3c0a79060110= 0fdc6116192a0 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -128,7 +128,7 @@ int vfio_block_multiple_devices_migration(VFIODevice *v= basedev, Error **errp) error_setg(&multiple_devices_migration_blocker, "Multiple VFIO devices migration is supported only if all o= f " "them support P2P migration"); - ret =3D migrate_add_blocker(&multiple_devices_migration_blocker, errp); + ret =3D migrate_add_blocker_normal(&multiple_devices_migration_blocker= , errp); =20 return ret; } diff --git a/hw/vfio/cpr.c b/hw/vfio/cpr.c index 3bede5494804e4aa291663c1e7c01d5dc5d3a407..392c2dd95d14664bade0ae74655= e91bc839504c0 100644 --- a/hw/vfio/cpr.c +++ b/hw/vfio/cpr.c @@ -7,13 +7,33 @@ =20 #include "qemu/osdep.h" #include "hw/vfio/vfio-common.h" +#include "migration/misc.h" #include "qapi/error.h" +#include "sysemu/runstate.h" + +static int vfio_cpr_reboot_notifier(NotifierWithReturn *notifier, + MigrationEvent *e, Error **errp) +{ + if (e->type =3D=3D MIG_EVENT_PRECOPY_SETUP && + !runstate_check(RUN_STATE_SUSPENDED) && !vm_get_suspended()) { + + error_setg(errp, + "VFIO device only supports cpr-reboot for runstate suspended"); + + return -1; + } + return 0; +} =20 int vfio_cpr_register_container(VFIOContainerBase *bcontainer, Error **err= p) { + migration_add_notifier_mode(&bcontainer->cpr_reboot_notifier, + vfio_cpr_reboot_notifier, + MIG_MODE_CPR_REBOOT); return 0; } =20 void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer) { + migration_remove_notifier(&bcontainer->cpr_reboot_notifier); } diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 50140eda872b3d5c9438459e207682de9be06511..2050ac8897231ff89cc223f0570= d5c7a65dede9e 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -889,7 +889,7 @@ static int vfio_block_migration(VFIODevice *vbasedev, E= rror *err, Error **errp) vbasedev->migration_blocker =3D error_copy(err); error_free(err); =20 - return migrate_add_blocker(&vbasedev->migration_blocker, errp); + return migrate_add_blocker_normal(&vbasedev->migration_blocker, errp); } =20 /* ---------------------------------------------------------------------- = */ --=20 2.44.0