From nobody Sat Nov 15 07:46:05 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=yandex-team.ru ARC-Seal: i=1; a=rsa-sha256; t=1755103890; cv=none; d=zohomail.com; s=zohoarc; b=kpqguS8NSY9MDPAp1+0yOzShuq5pGNRT8F+nbNS7+Be8TBTTBBFDGR2SFp6J+5qMi7hJ+8nhKNv/rfLwmw5uzaWFUXfv4MWBRESNnHtRNO/H0wOEXd4weUydfmCDrKzaBypdtqECkocVuFymo2jTckG88ADI5pI6JRzq3NAGMtU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1755103890; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ySTklhZeyl9879IHDj6pOytJKMMT2EjW0woilwTiBYw=; b=LcFs9+/CyQ0JH7iZD+/HXHteoC6624kufGjVPBfqw918cNCKYcH6jVE8YzhoU8V1iA+H4P/QHqad4hdQbUrosFZiHoivH96QPu49J7/VroXKV4RIR9iIzrJR4vsUMCUvbFsOMK+BdPu2T+JLnKHDnflPgCPhIkW2hN7b8wB5owQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17551038900701012.2832499980879; Wed, 13 Aug 2025 09:51:30 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1umEg5-00038p-0y; Wed, 13 Aug 2025 12:50:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1umEfw-00030Y-OM; Wed, 13 Aug 2025 12:50:08 -0400 Received: from forwardcorp1d.mail.yandex.net ([2a02:6b8:c41:1300:1:45:d181:df01]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1umEfk-0007tu-Hi; Wed, 13 Aug 2025 12:50:07 -0400 Received: from mail-nwsmtp-smtp-corp-main-56.klg.yp-c.yandex.net (mail-nwsmtp-smtp-corp-main-56.klg.yp-c.yandex.net [IPv6:2a02:6b8:c42:cf2d:0:640:140f:0]) by forwardcorp1d.mail.yandex.net (Yandex) with ESMTPS id BD302813EE; Wed, 13 Aug 2025 19:49:27 +0300 (MSK) Received: from vsementsov-lin.. (unknown [2a02:6bf:8080:167::1:21]) by mail-nwsmtp-smtp-corp-main-56.klg.yp-c.yandex.net (smtpcorp/Yandex) with ESMTPSA id wmOQI00FoSw0-tyTiIsXB; Wed, 13 Aug 2025 19:49:27 +0300 X-Yandex-Fwd: 1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex-team.ru; s=default; t=1755103767; bh=ySTklhZeyl9879IHDj6pOytJKMMT2EjW0woilwTiBYw=; h=Message-ID:Date:In-Reply-To:Cc:Subject:References:To:From; b=VUWTwCtS6WZ5e6lVny+K3mw/0Dr11WoxdBM0KS3vtRM80sB1mSpcT5Vqacgkbnb8H Dt/vscMqWcgJKsEjnUxvnP+l/FskCQYTcKKND2VrWfSOyM/X0EPp2R6vuI+XUpjlbt fRNwXnbNHPJWRuvtKJIapP6p5+Yd2eTOwWDfoUKs= Authentication-Results: mail-nwsmtp-smtp-corp-main-56.klg.yp-c.yandex.net; dkim=pass header.i=@yandex-team.ru From: Vladimir Sementsov-Ogievskiy To: mst@redhat.com, peterx@redhat.com, farosas@suse.de, raphael@enfabrica.net Cc: sgarzare@redhat.com, marcandre.lureau@redhat.com, pbonzini@redhat.com, kwolf@redhat.com, hreitz@redhat.com, berrange@redhat.com, eblake@redhat.com, armbru@redhat.com, qemu-devel@nongnu.org, qemu-block@nongnu.org, steven.sistare@oracle.com, den-plotnikov@yandex-team.ru, vsementsov@yandex-team.ru Subject: [PATCH 31/33] vhost-user-blk: support vhost backend migration Date: Wed, 13 Aug 2025 19:48:52 +0300 Message-ID: <20250813164856.950363-32-vsementsov@yandex-team.ru> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250813164856.950363-1-vsementsov@yandex-team.ru> References: <20250813164856.950363-1-vsementsov@yandex-team.ru> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a02:6b8:c41:1300:1:45:d181:df01; envelope-from=vsementsov@yandex-team.ru; helo=forwardcorp1d.mail.yandex.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @yandex-team.ru) X-ZM-MESSAGEID: 1755103892656116600 Content-Type: text/plain; charset="utf-8" Opt-out backend initialization code, and instead get the state from migration channel (including inflight region). Signed-off-by: Vladimir Sementsov-Ogievskiy --- hw/block/vhost-user-blk.c | 185 +++++++++++++++++++++++------ include/hw/virtio/vhost-user-blk.h | 2 + migration/options.c | 7 ++ migration/options.h | 1 + qapi/migration.json | 15 ++- 5 files changed, 169 insertions(+), 41 deletions(-) diff --git a/hw/block/vhost-user-blk.c b/hw/block/vhost-user-blk.c index c8bc2c78e6..2e6ef6477e 100644 --- a/hw/block/vhost-user-blk.c +++ b/hw/block/vhost-user-blk.c @@ -17,6 +17,7 @@ */ =20 #include "qemu/osdep.h" +#include "qapi-types-run-state.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/cutils.h" @@ -32,6 +33,11 @@ #include "system/system.h" #include "system/runstate.h" #include "trace.h" +#include "migration/qemu-file.h" +#include "migration/migration.h" +#include "migration/options.h" +#include "qemu/event_notifier.h" +#include =20 static const int user_feature_bits[] =3D { VIRTIO_BLK_F_SIZE_MAX, @@ -159,32 +165,35 @@ static int vhost_user_blk_start(VirtIODevice *vdev, E= rror **errp) =20 s->dev.acked_features =3D vdev->guest_features; =20 - ret =3D vhost_dev_prepare_inflight(&s->dev, vdev); - if (ret < 0) { - error_setg_errno(errp, -ret, "Error setting inflight format"); - goto err_guest_notifiers; - } - - if (!s->inflight->addr) { - ret =3D vhost_dev_get_inflight(&s->dev, s->queue_size, s->inflight= ); + if (!s->dev.migrating_backend) { + ret =3D vhost_dev_prepare_inflight(&s->dev, vdev); if (ret < 0) { - error_setg_errno(errp, -ret, "Error getting inflight"); + error_setg_errno(errp, -ret, "Error setting inflight format"); goto err_guest_notifiers; } - } =20 - ret =3D vhost_dev_set_inflight(&s->dev, s->inflight); - if (ret < 0) { - error_setg_errno(errp, -ret, "Error setting inflight"); - goto err_guest_notifiers; - } + if (!s->inflight->addr) { + ret =3D vhost_dev_get_inflight(&s->dev, s->queue_size, s->infl= ight); + if (ret < 0) { + error_setg_errno(errp, -ret, "Error getting inflight"); + goto err_guest_notifiers; + } + } =20 - /* guest_notifier_mask/pending not used yet, so just unmask - * everything here. virtio-pci will do the right thing by - * enabling/disabling irqfd. - */ - for (i =3D 0; i < s->dev.nvqs; i++) { - vhost_virtqueue_mask(&s->dev, vdev, i, false); + ret =3D vhost_dev_set_inflight(&s->dev, s->inflight); + if (ret < 0) { + error_setg_errno(errp, -ret, "Error setting inflight"); + goto err_guest_notifiers; + } + + /* + * guest_notifier_mask/pending not used yet, so just unmask + * everything here. virtio-pci will do the right thing by + * enabling/disabling irqfd. + */ + for (i =3D 0; i < s->dev.nvqs; i++) { + vhost_virtqueue_mask(&s->dev, vdev, i, false); + } } =20 s->dev.vq_index_end =3D s->dev.nvqs; @@ -231,6 +240,10 @@ static int vhost_user_blk_stop(VirtIODevice *vdev) force_stop =3D s->skip_get_vring_base_on_force_shutdown && qemu_force_shutdown_requested(); =20 + s->dev.migrating_backend =3D s->dev.migrating_backend || + (runstate_check(RUN_STATE_FINISH_MIGRATE) && + migrate_local_vhost_user_blk()); + ret =3D force_stop ? vhost_dev_force_stop(&s->dev, vdev, true) : vhost_dev_stop(&s->dev, vdev, true); =20 @@ -343,7 +356,9 @@ static void vhost_user_blk_reset(VirtIODevice *vdev) vhost_dev_free_inflight(s->inflight); } =20 -static int vhost_user_blk_connect(DeviceState *dev, Error **errp) +static int vhost_user_blk_connect(DeviceState *dev, + bool migrating_backend, + Error **errp) { VirtIODevice *vdev =3D VIRTIO_DEVICE(dev); VHostUserBlk *s =3D VHOST_USER_BLK(vdev); @@ -359,6 +374,7 @@ static int vhost_user_blk_connect(DeviceState *dev, Err= or **errp) s->dev.nvqs =3D s->num_queues; s->dev.vqs =3D s->vhost_vqs; s->dev.vq_index =3D 0; + s->dev.migrating_backend =3D migrating_backend; =20 vhost_dev_set_config_notifier(&s->dev, &blk_ops); =20 @@ -409,7 +425,7 @@ static void vhost_user_blk_event(void *opaque, QEMUChrE= vent event) =20 switch (event) { case CHR_EVENT_OPENED: - if (vhost_user_blk_connect(dev, &local_err) < 0) { + if (vhost_user_blk_connect(dev, false, &local_err) < 0) { error_report_err(local_err); qemu_chr_fe_disconnect(&s->chardev); return; @@ -428,31 +444,37 @@ static void vhost_user_blk_event(void *opaque, QEMUCh= rEvent event) } } =20 -static int vhost_user_blk_realize_connect(VHostUserBlk *s, Error **errp) +static int vhost_user_blk_realize_connect(VHostUserBlk *s, + bool migrating_backend, + Error **errp) { DeviceState *dev =3D DEVICE(s); int ret; =20 s->connected =3D false; =20 - ret =3D qemu_chr_fe_wait_connected(&s->chardev, errp); - if (ret < 0) { - return ret; + if (!migrating_backend) { + ret =3D qemu_chr_fe_wait_connected(&s->chardev, errp); + if (ret < 0) { + return ret; + } } =20 - ret =3D vhost_user_blk_connect(dev, errp); + ret =3D vhost_user_blk_connect(dev, migrating_backend, errp); if (ret < 0) { qemu_chr_fe_disconnect(&s->chardev); return ret; } assert(s->connected); =20 - ret =3D vhost_dev_get_config(&s->dev, (uint8_t *)&s->blkcfg, - VIRTIO_DEVICE(s)->config_len, errp); - if (ret < 0) { - qemu_chr_fe_disconnect(&s->chardev); - vhost_dev_cleanup(&s->dev); - return ret; + if (!migrating_backend) { + ret =3D vhost_dev_get_config(&s->dev, (uint8_t *)&s->blkcfg, + VIRTIO_DEVICE(s)->config_len, errp); + if (ret < 0) { + qemu_chr_fe_disconnect(&s->chardev); + vhost_dev_cleanup(&s->dev); + return ret; + } } =20 return 0; @@ -469,6 +491,11 @@ static void vhost_user_blk_device_realize(DeviceState = *dev, Error **errp) =20 trace_vhost_user_blk_device_realize(); =20 + if (s->incoming_backend && !runstate_check(RUN_STATE_INMIGRATE)) { + error_setg(errp, "__yc_local-incoming can be used " + "only for incoming migration"); + } + if (!s->chardev.chr) { error_setg(errp, "chardev is mandatory"); return; @@ -517,7 +544,7 @@ static void vhost_user_blk_device_realize(DeviceState *= dev, Error **errp) error_report_err(*errp); *errp =3D NULL; } - ret =3D vhost_user_blk_realize_connect(s, errp); + ret =3D vhost_user_blk_realize_connect(s, s->incoming_backend, err= p); } while (ret < 0 && retries--); =20 if (ret < 0) { @@ -525,9 +552,12 @@ static void vhost_user_blk_device_realize(DeviceState = *dev, Error **errp) } =20 /* we're fully initialized, now we can operate, so add the handler */ - qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, - vhost_user_blk_event, NULL, (void *)dev, - NULL, true); + if (!s->incoming_backend) { + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, + vhost_user_blk_event, NULL, (void *)dev, + NULL, true); + } + trace_vhost_user_blk_device_realize_finish(); return; =20 @@ -592,6 +622,79 @@ static const VMStateDescription vmstate_vhost_user_blk= =3D { }, }; =20 +static void vhost_user_blk_save(VirtIODevice *vdev, QEMUFile *f) +{ + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + struct vhost_dev *hdev =3D vhost_user_blk_get_vhost(vdev); + + if (!hdev->migrating_backend) { + return; + } + + qemu_file_put_fd(f, s->inflight->fd); + qemu_put_be64(f, s->inflight->size); + qemu_put_be64(f, s->inflight->offset); + qemu_put_be16(f, s->inflight->queue_size); + + vhost_save_backend(hdev, f); +} + +static int vhost_user_blk_load(VirtIODevice *vdev, QEMUFile *f, + int version_id) +{ + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + struct vhost_dev *hdev =3D vhost_user_blk_get_vhost(vdev); + + if (!hdev->migrating_backend) { + return 0; + } + + s->inflight->fd =3D qemu_file_get_fd(f); + qemu_get_be64s(f, &s->inflight->size); + qemu_get_be64s(f, &s->inflight->offset); + qemu_get_be16s(f, &s->inflight->queue_size); + + s->inflight->addr =3D mmap(0, s->inflight->size, PROT_READ | PROT_WRIT= E, + MAP_SHARED, s->inflight->fd, s->inflight->off= set); + if (s->inflight->addr =3D=3D MAP_FAILED) { + return -EINVAL; + } + + vhost_load_backend(hdev, f); + + return 0; +} + +static int vhost_user_blk_post_load(VirtIODevice *vdev) +{ + VHostUserBlk *s =3D VHOST_USER_BLK(vdev); + struct vhost_dev *hdev =3D vhost_user_blk_get_vhost(vdev); + DeviceState *dev =3D &s->parent_obj.parent_obj; + + if (!hdev->migrating_backend) { + return 0; + } + + memcpy(&s->blkcfg, vdev->config, vdev->config_len); + + /* we're fully initialized, now we can operate, so add the handler */ + qemu_chr_fe_set_handlers(&s->chardev, NULL, NULL, + vhost_user_blk_event, NULL, (void *)dev, + NULL, true); + + return 0; +} + +static bool vhost_user_blk_skip_migration_log(VirtIODevice *vdev) +{ + /* + * Note that hdev->migrating_backend is false at this moment, + * as logging is being setup during outging migration setup stage, + * which is far before vm stop. + */ + return migrate_local_vhost_user_blk(); +} + static const Property vhost_user_blk_properties[] =3D { DEFINE_PROP_CHR("chardev", VHostUserBlk, chardev), DEFINE_PROP_UINT16("num-queues", VHostUserBlk, num_queues, @@ -605,6 +708,8 @@ static const Property vhost_user_blk_properties[] =3D { VIRTIO_BLK_F_WRITE_ZEROES, true), DEFINE_PROP_BOOL("skip-get-vring-base-on-force-shutdown", VHostUserBlk, skip_get_vring_base_on_force_shutdown, false), + DEFINE_PROP_BOOL("local-incoming", VHostUserBlk, + incoming_backend, false), }; =20 static void vhost_user_blk_class_init(ObjectClass *klass, const void *data) @@ -624,6 +729,10 @@ static void vhost_user_blk_class_init(ObjectClass *kla= ss, const void *data) vdc->set_status =3D vhost_user_blk_set_status; vdc->reset =3D vhost_user_blk_reset; vdc->get_vhost =3D vhost_user_blk_get_vhost; + vdc->save =3D vhost_user_blk_save; + vdc->load =3D vhost_user_blk_load; + vdc->post_load =3D vhost_user_blk_post_load, + vdc->skip_vhost_migration_log =3D vhost_user_blk_skip_migration_log; } =20 static const TypeInfo vhost_user_blk_info =3D { diff --git a/include/hw/virtio/vhost-user-blk.h b/include/hw/virtio/vhost-u= ser-blk.h index a10f785672..b06f55fd6f 100644 --- a/include/hw/virtio/vhost-user-blk.h +++ b/include/hw/virtio/vhost-user-blk.h @@ -52,6 +52,8 @@ struct VHostUserBlk { bool started_vu; =20 bool skip_get_vring_base_on_force_shutdown; + + bool incoming_backend; }; =20 #endif diff --git a/migration/options.c b/migration/options.c index dffb6910f4..11b719c81b 100644 --- a/migration/options.c +++ b/migration/options.c @@ -269,6 +269,13 @@ bool migrate_local_char_socket(void) return s->capabilities[MIGRATION_CAPABILITY_LOCAL_CHAR_SOCKET]; } =20 +bool migrate_local_vhost_user_blk(void) +{ + MigrationState *s =3D migrate_get_current(); + + return s->capabilities[MIGRATION_CAPABILITY_LOCAL_VHOST_USER_BLK]; +} + bool migrate_ignore_shared(void) { MigrationState *s =3D migrate_get_current(); diff --git a/migration/options.h b/migration/options.h index 40971f0aa0..5a40ac073d 100644 --- a/migration/options.h +++ b/migration/options.h @@ -31,6 +31,7 @@ bool migrate_dirty_bitmaps(void); bool migrate_events(void); bool migrate_mapped_ram(void); bool migrate_local_char_socket(void); +bool migrate_local_vhost_user_blk(void); bool migrate_ignore_shared(void); bool migrate_late_block_activate(void); bool migrate_multifd(void); diff --git a/qapi/migration.json b/qapi/migration.json index 4f282d168e..ead7f4d17c 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -520,11 +520,19 @@ # @local-char-socket: Migrate socket chardevs open file descriptors. # Only may be used when migration channel is unix socket. Only # involves socket chardevs with "support-local-migration" option -# enabled. (since 10.2) +# enabled. For target device also @local-incoming option must +# be specified (since 10.2) +# +# @local-vhost-user-blk: Migrate vhost-user-blk locally, keeping +# backend alive. Open file descriptors and backend-related state are +# migrated. Only may be used when migration channel is unix socket. +# For target device also @local-incoming option must be specified +# (since 10.2) # # Features: # -# @unstable: Members @x-colo and @x-ignore-shared are experimental. +# @unstable: Members @x-colo, @x-ignore-shared, @local-char-socket, +# @local-vhost-user-blk are experimental. # @deprecated: Member @zero-blocks is deprecated as being part of # block migration which was already removed. # @@ -542,7 +550,8 @@ 'validate-uuid', 'background-snapshot', 'zero-copy-send', 'postcopy-preempt', 'switchover-ack', 'dirty-limit', 'mapped-ram', - { 'name': 'local-char-socket', 'features': [ 'unstable' ] } ] } + { 'name': 'local-char-socket', 'features': [ 'unstable' ] }, + { 'name': 'local-vhost-user-blk', 'features': [ 'unstable' ] } = ] } =20 ## # @MigrationCapabilityStatus: --=20 2.48.1