docs/interop/vhost-user.rst | 56 ++++++++++++++++++----------- hw/block/vhost-user-blk.c | 28 +++++++++++++++ hw/virtio/vhost-user.c | 5 +++ hw/virtio/vhost.c | 57 ++++++++++++++++++++++++++++++ include/hw/virtio/vhost-user-blk.h | 1 + include/hw/virtio/vhost-user.h | 2 ++ include/hw/virtio/vhost.h | 6 ++++ include/migration/vmstate.h | 10 ++++++ 8 files changed, 145 insertions(+), 20 deletions(-)
v6:
- fix documentation about new protocol feature
- add check to ensure that inflight buffer subsection has been successfully loaded
- disable support for the new feature if in-flight or inflight migration is not supported.
v5:
Make protocol feature flag instead of GET_VRING_BASE msg parameter,
so all changes in other devices is no longer needed.
Now back-end may set this feature for QEMU. This feature must be set
with in-flight migration parameter in vhost-user-blk.
v4:
While testing inflight migration, I notices a problem with the fact that
GET_VRING_BASE is needed during migration, so the back-end stops
dirtying pages and synchronizes `last_avail` counter with QEMU. So after
migration in-flight I/O requests will be looks like resubmited on destination vm.
However, in new logic, we no longer need to wait for in-flight requests
to be complete at GET_VRING_BASE message. So support new parameter
`should_drain` in the GET_VRING_BASE to allow back-end stop vrings
immediately without waiting for in-flight I/O requests to complete.
Also:
- modify vhost-user rst
- refactor on vhost-user-blk.c, now `should_drain` is based on
device parameter `inflight-migration`
v3:
- use pre_load_errp instead of pre_load in vhost.c
- change vhost-user-blk property to
"skip-get-vring-base-inflight-migration"
- refactor vhost-user-blk.c, by moving vhost_user_blk_inflight_needed() higher
v2:
- rewrite migration using VMSD instead of qemufile API
- add vhost-user-blk parameter instead of migration capability
I don't know if VMSD was used cleanly in migration implementation, so
feel free for comments.
Based on Vladimir's work:
[PATCH v2 00/25] vhost-user-blk: live-backend local migration
which was based on:
- [PATCH v4 0/7] chardev: postpone connect
(which in turn is based on [PATCH 0/2] remove deprecated 'reconnect' options)
- [PATCH v3 00/23] vhost refactoring and fixes
- [PATCH v8 14/19] migration: introduce .pre_incoming() vmsd handler
Based-on: <20250924133309.334631-1-vsementsov@yandex-team.ru>
Based-on: <20251015212051.1156334-1-vsementsov@yandex-team.ru>
Based-on: <20251015145808.1112843-1-vsementsov@yandex-team.ru>
Based-on: <20251015132136.1083972-15-vsementsov@yandex-team.ru>
Based-on: <20251016114104.1384675-1-vsementsov@yandex-team.ru>
---
Hi!
During inter-host migration, waiting for disk requests to be drained
in the vhost-user backend can incur significant downtime.
This can be avoided if QEMU migrates the inflight region in
vhost-user-blk.
Thus, during the qemu migration, with protocol feature flag the vhost-user
back-end can immediately stop vrings, so all in-flight requests will be
migrated to another host.
At first, I tried to implement migration for all vhost-user devices that support inflight at once,
but this would require a lot of changes both in vhost-user-blk (to transfer it to the base class) and
in the vhost-user-base base class (inflight implementation and remodeling + a large refactor).
Therefore, for now I decided to leave this idea for later and
implement the migration of the inflight region first for vhost-user-blk.
Alexandr Moshkov (5):
vhost-user.rst: specify vhost-user back-end action on GET_VRING_BASE
vhost-user: introduce protocol feature for skip drain on
GET_VRING_BASE
vmstate: introduce VMSTATE_VBUFFER_UINT64
vhost: add vmstate for inflight region with inner buffer
vhost-user-blk: support inter-host inflight migration
docs/interop/vhost-user.rst | 56 ++++++++++++++++++-----------
hw/block/vhost-user-blk.c | 28 +++++++++++++++
hw/virtio/vhost-user.c | 5 +++
hw/virtio/vhost.c | 57 ++++++++++++++++++++++++++++++
include/hw/virtio/vhost-user-blk.h | 1 +
include/hw/virtio/vhost-user.h | 2 ++
include/hw/virtio/vhost.h | 6 ++++
include/migration/vmstate.h | 10 ++++++
8 files changed, 145 insertions(+), 20 deletions(-)
--
2.34.1
On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: Peter: Please review the migration aspects (especially the vmstates). Thank you! > v6: > - fix documentation about new protocol feature > - add check to ensure that inflight buffer subsection has been successfully loaded > - disable support for the new feature if in-flight or inflight migration is not supported. Aside from the comment I posted: Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> This patch series should go through Michael Tsirkin's VIRTIO/vhost tree.
On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: > > Peter: Please review the migration aspects (especially the vmstates). > Thank you! Looks good from my side as long as it's based on VMSD, I appreciate that change from the old versions where it used to use qemufile APIs. The major question here is if this series depends on Vladimir's other series, while there's still one patch that is not-for-merge: https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t Does it automatically mark this series RFC as well? Personally speaking, a new migration cap would work all fine, we should have discussed it somewhere previously. Said that, "local-vhost-user-blk" capability is likely not the right one. IMHO it should be either "local" or "fd-passing" / "fd-passthrough" (or something generic) as the name. If we are not sure if we will leverage more than "passing the FDs around", we can make it as simple as "local" as a new migration capability. Then migration's misc.h should export a function migrate_is_local() then device code can probe that in its own vmstate handling paths on save/load. A note to Vladimir: please remember to add a check to enforce UNIX socket when a formal patch 23 will be proposed some day, no matter what is the name of the capability. It should fail qmp "migrate" or qmp "migrate_incoming" command if the main URI is not a unix socket. Thanks, -- Peter Xu
On 13.01.26 21:56, Peter Xu wrote: > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: >> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: >> >> Peter: Please review the migration aspects (especially the vmstates). >> Thank you! > > Looks good from my side as long as it's based on VMSD, I appreciate that > change from the old versions where it used to use qemufile APIs. > > The major question here is if this series depends on Vladimir's other > series No, it does not. And if we can proceed with merging these series first, I'll be happy to rebase on top of it. > while there's still one patch that is not-for-merge: > > https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t > > Does it automatically mark this series RFC as well? > > Personally speaking, a new migration cap would work all fine, we should > have discussed it somewhere previously. Said that, "local-vhost-user-blk" > capability is likely not the right one. IMHO it should be either "local" > or "fd-passing" / "fd-passthrough" (or something generic) as the name. If > we are not sure if we will leverage more than "passing the FDs around", we > can make it as simple as "local" as a new migration capability. > > Then migration's misc.h should export a function migrate_is_local() then > device code can probe that in its own vmstate handling paths on save/load. > > A note to Vladimir: please remember to add a check to enforce UNIX socket > when a formal patch 23 will be proposed some day, no matter what is the > name of the capability. It should fail qmp "migrate" or qmp > "migrate_incoming" command if the main URI is not a unix socket. > Thanks! I'll keep that in mind when prepare next version. -- Best regards, Vladimir
On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 13.01.26 21:56, Peter Xu wrote: > > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: > > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: > > > > > > Peter: Please review the migration aspects (especially the vmstates). > > > Thank you! > > > > Looks good from my side as long as it's based on VMSD, I appreciate that > > change from the old versions where it used to use qemufile APIs. > > > > The major question here is if this series depends on Vladimir's other > > series > > No, it does not. And if we can proceed with merging these series first, I'll > be happy to rebase on top of it. I thought it requires migrate_local_vhost_user_blk() be present? The inflight feature should not be enabled only if there's a hint that it's a local migration.. I'll comment inline on the patch later. > > > while there's still one patch that is not-for-merge: > > > > https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t > > > > Does it automatically mark this series RFC as well? > > > > Personally speaking, a new migration cap would work all fine, we should > > have discussed it somewhere previously. Said that, "local-vhost-user-blk" > > capability is likely not the right one. IMHO it should be either "local" > > or "fd-passing" / "fd-passthrough" (or something generic) as the name. If > > we are not sure if we will leverage more than "passing the FDs around", we > > can make it as simple as "local" as a new migration capability. > > > > Then migration's misc.h should export a function migrate_is_local() then > > device code can probe that in its own vmstate handling paths on save/load. > > > > A note to Vladimir: please remember to add a check to enforce UNIX socket > > when a formal patch 23 will be proposed some day, no matter what is the > > name of the capability. It should fail qmp "migrate" or qmp > > "migrate_incoming" command if the main URI is not a unix socket. > > > > Thanks! I'll keep that in mind when prepare next version. > > -- > Best regards, > Vladimir > -- Peter Xu
On 14.01.26 15:22, Peter Xu wrote: > On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote: >> On 13.01.26 21:56, Peter Xu wrote: >>> On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: >>>> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: >>>> >>>> Peter: Please review the migration aspects (especially the vmstates). >>>> Thank you! >>> >>> Looks good from my side as long as it's based on VMSD, I appreciate that >>> change from the old versions where it used to use qemufile APIs. >>> >>> The major question here is if this series depends on Vladimir's other >>> series >> >> No, it does not. And if we can proceed with merging these series first, I'll >> be happy to rebase on top of it. > > I thought it requires migrate_local_vhost_user_blk() be present? The > inflight feature should not be enabled only if there's a hint that it's a > local migration.. Oh right, I missed it. We discussed that Alexandr will rebase the series on master without dependency on my RFC. > > I'll comment inline on the patch later. > >> >>> while there's still one patch that is not-for-merge: >>> >>> https://lore.kernel.org/all/20251016114104.1384675-24-vsementsov@yandex-team.ru/#t >>> >>> Does it automatically mark this series RFC as well? >>> >>> Personally speaking, a new migration cap would work all fine, we should >>> have discussed it somewhere previously. Said that, "local-vhost-user-blk" >>> capability is likely not the right one. IMHO it should be either "local" >>> or "fd-passing" / "fd-passthrough" (or something generic) as the name. If >>> we are not sure if we will leverage more than "passing the FDs around", we >>> can make it as simple as "local" as a new migration capability. >>> >>> Then migration's misc.h should export a function migrate_is_local() then >>> device code can probe that in its own vmstate handling paths on save/load. >>> >>> A note to Vladimir: please remember to add a check to enforce UNIX socket >>> when a formal patch 23 will be proposed some day, no matter what is the >>> name of the capability. It should fail qmp "migrate" or qmp >>> "migrate_incoming" command if the main URI is not a unix socket. >>> >> >> Thanks! I'll keep that in mind when prepare next version. >> >> -- >> Best regards, >> Vladimir >> > -- Best regards, Vladimir
On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 14.01.26 15:22, Peter Xu wrote: > > On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > On 13.01.26 21:56, Peter Xu wrote: > > > > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: > > > > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: > > > > > > > > > > Peter: Please review the migration aspects (especially the vmstates). > > > > > Thank you! > > > > > > > > Looks good from my side as long as it's based on VMSD, I appreciate that > > > > change from the old versions where it used to use qemufile APIs. > > > > > > > > The major question here is if this series depends on Vladimir's other > > > > series > > > > > > No, it does not. And if we can proceed with merging these series first, I'll > > > be happy to rebase on top of it. > > > > I thought it requires migrate_local_vhost_user_blk() be present? The > > inflight feature should not be enabled only if there's a hint that it's a > > local migration.. > > Oh right, I missed it. > > We discussed that Alexandr will rebase the series on master without dependency > on my RFC. The problem is IIUC the new INFLIGHT feature bit will be declared as supported to vhost-user-block after applying this series. Then if we start a remote migration (rather than local) it'll be automatically (and wrongly) enabled? AFAIU, the dependency makes sense, at least to the patch to introduce the "local" / ... capability? -- Peter Xu
On 14.01.26 18:17, Peter Xu wrote: > On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: >> On 14.01.26 15:22, Peter Xu wrote: >>> On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote: >>>> On 13.01.26 21:56, Peter Xu wrote: >>>>> On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: >>>>>> On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: >>>>>> >>>>>> Peter: Please review the migration aspects (especially the vmstates). >>>>>> Thank you! >>>>> >>>>> Looks good from my side as long as it's based on VMSD, I appreciate that >>>>> change from the old versions where it used to use qemufile APIs. >>>>> >>>>> The major question here is if this series depends on Vladimir's other >>>>> series >>>> >>>> No, it does not. And if we can proceed with merging these series first, I'll >>>> be happy to rebase on top of it. >>> >>> I thought it requires migrate_local_vhost_user_blk() be present? The >>> inflight feature should not be enabled only if there's a hint that it's a >>> local migration.. >> >> Oh right, I missed it. >> >> We discussed that Alexandr will rebase the series on master without dependency >> on my RFC. > > The problem is IIUC the new INFLIGHT feature bit will be declared as > supported to vhost-user-block after applying this series. Then if we start > a remote migration (rather than local) it'll be automatically (and wrongly) > enabled? > No, not so. We develop inflight-region migraiton (this series) exactly for remote migration, not for local. My series about backend transfer (fd-migration) will migrate inflight-region the other way - by migrating its FD. > AFAIU, the dependency makes sense, at least to the patch to introduce the > "local" / ... capability? > -- Best regards, Vladimir
On Wed, Jan 14, 2026 at 08:54:50PM +0300, Vladimir Sementsov-Ogievskiy wrote: > On 14.01.26 18:17, Peter Xu wrote: > > On Wed, Jan 14, 2026 at 05:35:53PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > On 14.01.26 15:22, Peter Xu wrote: > > > > On Wed, Jan 14, 2026 at 09:19:10AM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > > > On 13.01.26 21:56, Peter Xu wrote: > > > > > > On Tue, Jan 13, 2026 at 01:12:42PM -0500, Stefan Hajnoczi wrote: > > > > > > > On Tue, Jan 13, 2026 at 02:58:09PM +0500, Alexandr Moshkov wrote: > > > > > > > > > > > > > > Peter: Please review the migration aspects (especially the vmstates). > > > > > > > Thank you! > > > > > > > > > > > > Looks good from my side as long as it's based on VMSD, I appreciate that > > > > > > change from the old versions where it used to use qemufile APIs. > > > > > > > > > > > > The major question here is if this series depends on Vladimir's other > > > > > > series > > > > > > > > > > No, it does not. And if we can proceed with merging these series first, I'll > > > > > be happy to rebase on top of it. > > > > > > > > I thought it requires migrate_local_vhost_user_blk() be present? The > > > > inflight feature should not be enabled only if there's a hint that it's a > > > > local migration.. > > > > > > Oh right, I missed it. > > > > > > We discussed that Alexandr will rebase the series on master without dependency > > > on my RFC. > > > > The problem is IIUC the new INFLIGHT feature bit will be declared as > > supported to vhost-user-block after applying this series. Then if we start > > a remote migration (rather than local) it'll be automatically (and wrongly) > > enabled? > > > > No, not so. > > We develop inflight-region migraiton (this series) exactly for remote migration, > not for local. > > My series about backend transfer (fd-migration) will migrate inflight-region the > other way - by migrating its FD. Ah, I see.. Yes I think you're right then there's no dependency needed. Please go ahead with the repost. Thanks, -- Peter Xu
© 2016 - 2026 Red Hat, Inc.