hw/virtio/vhost-kernel.c | 6 + hw/virtio/vhost-vsock.c | 208 +++++++++++++++++++++++++++--- hw/virtio/vhost.c | 55 ++++++-- include/hw/virtio/vhost-backend.h | 1 + include/hw/virtio/vhost-vsock.h | 4 + include/hw/virtio/vhost.h | 32 +++++ 6 files changed, 275 insertions(+), 31 deletions(-)
v2 -> v3:
Re-design device ownership hand-off (suggested by Dongli). Do not rely
on SETUP migration notifiers, as this approach forces us to reorder
generic migration code (see v2 discussion). Instead:
* .pre_save() releases ownership on the source (RESET_OWNER) once VM
is stopped;
* .realize() is adjusted to defer device ownership acquisition for an
incoming CPR;
* .post_load() actually claims the device (VHOST_SET_OWNER);
* FAILED migration event callback re-aquires the ownership on the
source after re_save released it.
v2: https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com
Andrey Drobyshev (7):
vhost: add vhost_reset_owner op
vhost-vsock: don't reset connections during CPR
vhost-vsock: fix FD leak in realize()
vhost-vsock: preserve vhost FD during CPR
vhost: factor out vhost_dev_init_backend()
vhost: add vhost_dev_set_owner() / vhost_dev_reset_owner() helpers
vhost-vsock: hand off device ownership across CPR
hw/virtio/vhost-kernel.c | 6 +
hw/virtio/vhost-vsock.c | 208 +++++++++++++++++++++++++++---
hw/virtio/vhost.c | 55 ++++++--
include/hw/virtio/vhost-backend.h | 1 +
include/hw/virtio/vhost-vsock.h | 4 +
include/hw/virtio/vhost.h | 32 +++++
6 files changed, 275 insertions(+), 31 deletions(-)
--
2.47.1
On 26.06.26 19:46, Andrey Drobyshev wrote: > v2 -> v3: > > Re-design device ownership hand-off (suggested by Dongli). Do not rely > on SETUP migration notifiers, as this approach forces us to reorder > generic migration code (see v2 discussion). Instead: > > * .pre_save() releases ownership on the source (RESET_OWNER) once VM > is stopped; > * .realize() is adjusted to defer device ownership acquisition for an > incoming CPR; > * .post_load() actually claims the device (VHOST_SET_OWNER); > * FAILED migration event callback re-aquires the ownership on the > source after re_save released it. > > v2:https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com Hi! Two notes: 1. Did you consider migrating needed FDs through main migration channel, without use of CPR, like I do in (not yet landed) "[PATCH v19 00/15] virtio-net: live-TAP local migration" [1] 2. I don't know how much vhost-vsock differs from vhost-net, but for vhost-net I remember that RESET_OWNER + SET_OWNER is effectively equal to simply recreating vhost device on target, because in kernel vhost device is hardly bound to the process itself, and can't be passed to another process. So, we can pass only "empty" FD, not the initialized vhost device. That's why in [1] I only pass tap-fds (and some additional state), but vhost fds are simply reopened on target (or passed by mgmt app). Does vhost-vsock work in a different way? Is there real sense in passing vhost fds here? [1] https://lore.kernel.org/qemu-devel/20260714154246.1242856-1-vsementsov@yandex-team.ru/ -- Best regards, Vladimir
On 7/14/26 6:03 PM, Vladimir Sementsov-Ogievskiy wrote: > On 26.06.26 19:46, Andrey Drobyshev wrote: >> v2 -> v3: >> >> Re-design device ownership hand-off (suggested by Dongli). Do not rely >> on SETUP migration notifiers, as this approach forces us to reorder >> generic migration code (see v2 discussion). Instead: >> >> * .pre_save() releases ownership on the source (RESET_OWNER) once VM >> is stopped; >> * .realize() is adjusted to defer device ownership acquisition for an >> incoming CPR; >> * .post_load() actually claims the device (VHOST_SET_OWNER); >> * FAILED migration event callback re-aquires the ownership on the >> source after re_save released it. >> >> v2:https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com > > Hi! > Hello Vladimir! > Two notes: > > 1. Did you consider migrating needed FDs through main migration channel, without > use of CPR, like I do in (not yet landed) "[PATCH v19 00/15] virtio-net: live-TAP local migration" [1] > AFAIU you're doing local same-host migration as a way to update QEMU binary. With this approach transferring FDs through the main channel is indeed cleaner. However in our own downstream we use enhanced in-place memory preservation which is based entirely on cpr-exec migration mode. So, to answer your question: yes, we considered it, and it might work for cpr-transfer where we're also doing same-host migration. But in cpr-exec case there's only one QEMU process involved => we can't use UNIX socket for migration transport => no SCM_RIGHTS FDs passing => FDs should be passed via additional transport, no way around it. > 2. I don't know how much vhost-vsock differs from vhost-net, but for vhost-net I remember > that RESET_OWNER + SET_OWNER is effectively equal to simply recreating vhost device on target, > because in kernel vhost device is hardly bound to the process itself, and can't be passed to > another process. So, we can pass only "empty" FD, not the initialized vhost device. > That's why in [1] I only pass tap-fds (and some additional state), but vhost fds are simply > reopened on target (or passed by mgmt app). > Does vhost-vsock work in a different way? Is there real sense in passing vhost fds here? > > > [1] https://lore.kernel.org/qemu-devel/20260714154246.1242856-1-vsementsov@yandex-team.ru/ > I'm also not extremely keen on vhost-net internals, but as I understand with vhost-net we have 2 FDs: TAP FD for the network endpoint and vhost FD for the memory tables, vring addresses etc. The TAP FD is process-independent state - and we pass it via migration channel. The vhost FD is entirely per owning process - and we're able to close it on source, reopen on target and bind to the passed TAP FD. However with vhost-vsock there's only one FD. It's used both for plumbing and the endpoint. At the very least, vhost device state holds guest CID registration in the global vhost_vsock_hash. If we go the "recreate/reopen the device" path - CID is removed from the hash in vhost_vsock_dev_release(), and then we're gonna get ECONNRESET for any incoming packets => connections are dead. We can't afford that. Andrey
On 15.07.26 13:25, Andrey Drobyshev wrote:
> On 7/14/26 6:03 PM, Vladimir Sementsov-Ogievskiy wrote:
>> On 26.06.26 19:46, Andrey Drobyshev wrote:
>>> v2 -> v3:
>>>
>>> Re-design device ownership hand-off (suggested by Dongli). Do not rely
>>> on SETUP migration notifiers, as this approach forces us to reorder
>>> generic migration code (see v2 discussion). Instead:
>>>
>>> * .pre_save() releases ownership on the source (RESET_OWNER) once VM
>>> is stopped;
>>> * .realize() is adjusted to defer device ownership acquisition for an
>>> incoming CPR;
>>> * .post_load() actually claims the device (VHOST_SET_OWNER);
>>> * FAILED migration event callback re-aquires the ownership on the
>>> source after re_save released it.
>>>
>>> v2:https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com
>>
>> Hi!
>>
>
> Hello Vladimir!
>
>> Two notes:
>>
>> 1. Did you consider migrating needed FDs through main migration channel, without
>> use of CPR, like I do in (not yet landed) "[PATCH v19 00/15] virtio-net: live-TAP local migration" [1]
>>
>
> AFAIU you're doing local same-host migration as a way to update QEMU
> binary. With this approach transferring FDs through the main channel is
> indeed cleaner. However in our own downstream we use enhanced in-place
> memory preservation which is based entirely on cpr-exec migration mode.
>
> So, to answer your question: yes, we considered it, and it might work
> for cpr-transfer where we're also doing same-host migration. But in
> cpr-exec case there's only one QEMU process involved => we can't use
> UNIX socket for migration transport => no SCM_RIGHTS FDs passing => FDs
> should be passed via additional transport, no way around it.
Hmm. Looking at code for fd-passing through migration channel:
static bool load_fd(QEMUFile *f, void *pv, size_t size,
const VMStateField *field, Error **errp)
{
int32_t *v = pv;
if (migrate_mode() == MIG_MODE_CPR_EXEC) {
qemu_get_sbe32s(f, v);
return true;
}
...
Seems like passing FDs through main channel (like in my TAP-series) would
work both for local migration (with two processes and UNIX socket) and for
CPR_EXEC mode. So the FD is passed as simple number in case of cpr-exec.
So, if simply pass FDs through migration state, it should work both for
CPR_EXEC and local migration.
(hmm, still, keeping in mind that TAP-series API still not negotiated,
it's simpler to start from pure CPR approach, and add local-migration
support in future)
>
>
>> 2. I don't know how much vhost-vsock differs from vhost-net, but for vhost-net I remember
>> that RESET_OWNER + SET_OWNER is effectively equal to simply recreating vhost device on target,
>> because in kernel vhost device is hardly bound to the process itself, and can't be passed to
>> another process. So, we can pass only "empty" FD, not the initialized vhost device.
>> That's why in [1] I only pass tap-fds (and some additional state), but vhost fds are simply
>> reopened on target (or passed by mgmt app).
>> Does vhost-vsock work in a different way? Is there real sense in passing vhost fds here?
>>
>>
>> [1] https://lore.kernel.org/qemu-devel/20260714154246.1242856-1-vsementsov@yandex-team.ru/
>>
>
> I'm also not extremely keen on vhost-net internals, but as I understand
> with vhost-net we have 2 FDs: TAP FD for the network endpoint and vhost
> FD for the memory tables, vring addresses etc. The TAP FD is
> process-independent state - and we pass it via migration channel. The
> vhost FD is entirely per owning process - and we're able to close it on
> source, reopen on target and bind to the passed TAP FD.
Yes, my understanding matches.
>
> However with vhost-vsock there's only one FD. It's used both for
> plumbing and the endpoint. At the very least, vhost device state holds
> guest CID registration in the global vhost_vsock_hash. If we go the
> "recreate/reopen the device" path - CID is removed from the hash in
> vhost_vsock_dev_release(), and then we're gonna get ECONNRESET for any
> incoming packets => connections are dead. We can't afford that.
>
OK, thanks for explanation!
--
Best regards,
Vladimir
On 7/15/26 12:48 PM, Vladimir Sementsov-Ogievskiy wrote:
> On 15.07.26 13:25, Andrey Drobyshev wrote:
>> On 7/14/26 6:03 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> On 26.06.26 19:46, Andrey Drobyshev wrote:
>>>> v2 -> v3:
>>>>
>>>> Re-design device ownership hand-off (suggested by Dongli). Do not rely
>>>> on SETUP migration notifiers, as this approach forces us to reorder
>>>> generic migration code (see v2 discussion). Instead:
>>>>
>>>> * .pre_save() releases ownership on the source (RESET_OWNER) once VM
>>>> is stopped;
>>>> * .realize() is adjusted to defer device ownership acquisition for an
>>>> incoming CPR;
>>>> * .post_load() actually claims the device (VHOST_SET_OWNER);
>>>> * FAILED migration event callback re-aquires the ownership on the
>>>> source after re_save released it.
>>>>
>>>> v2:https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com
>>>
>>> Hi!
>>>
>>
>> Hello Vladimir!
>>
>>> Two notes:
>>>
>>> 1. Did you consider migrating needed FDs through main migration channel, without
>>> use of CPR, like I do in (not yet landed) "[PATCH v19 00/15] virtio-net: live-TAP local migration" [1]
>>>
>>
>> AFAIU you're doing local same-host migration as a way to update QEMU
>> binary. With this approach transferring FDs through the main channel is
>> indeed cleaner. However in our own downstream we use enhanced in-place
>> memory preservation which is based entirely on cpr-exec migration mode.
>>
>> So, to answer your question: yes, we considered it, and it might work
>> for cpr-transfer where we're also doing same-host migration. But in
>> cpr-exec case there's only one QEMU process involved => we can't use
>> UNIX socket for migration transport => no SCM_RIGHTS FDs passing => FDs
>> should be passed via additional transport, no way around it.
>
>
> Hmm. Looking at code for fd-passing through migration channel:
>
> static bool load_fd(QEMUFile *f, void *pv, size_t size,
> const VMStateField *field, Error **errp)
> {
> int32_t *v = pv;
>
> if (migrate_mode() == MIG_MODE_CPR_EXEC) {
> qemu_get_sbe32s(f, v);
> return true;
> }
> ...
>
>
> Seems like passing FDs through main channel (like in my TAP-series) would
> work both for local migration (with two processes and UNIX socket) and for
> CPR_EXEC mode. So the FD is passed as simple number in case of cpr-exec.
>
> So, if simply pass FDs through migration state, it should work both for
> CPR_EXEC and local migration.
>
> (hmm, still, keeping in mind that TAP-series API still not negotiated,
> it's simpler to start from pure CPR approach, and add local-migration
> support in future)
>
Yes, my claim was too harsh - the FD passing mechanism itself probably
can be implemented via the main migration channel as well, even for
cpr-exec. E.g. if it's not a UNIX socket but a plain file on file
system - we should be good.
But then the issue is timing: VMSTATE_FD delivers the FD during the
device's vmstate load, i.e. around post_load. And cpr_state is loaded
early, before device realization, so the FD is available at realize
time. And for instance our vhost-vsock case does need FD already during
realize to read the backend feature set:
vhost_vsock_device_realize() ->
vhost_dev_init_backend() ->
vhost_dev_init_features()
So for me the exact transport used underneath for FDs passing is not
that important. I'm just exploiting the CPR API to get the passed FD at
an early .realize() stage. If we can somehow guarantee an early FDs
delivery through the main channel - that'll work.
Andrey
>>
>>
>>> 2. I don't know how much vhost-vsock differs from vhost-net, but for vhost-net I remember
>>> that RESET_OWNER + SET_OWNER is effectively equal to simply recreating vhost device on target,
>>> because in kernel vhost device is hardly bound to the process itself, and can't be passed to
>>> another process. So, we can pass only "empty" FD, not the initialized vhost device.
>>> That's why in [1] I only pass tap-fds (and some additional state), but vhost fds are simply
>>> reopened on target (or passed by mgmt app).
>>> Does vhost-vsock work in a different way? Is there real sense in passing vhost fds here?
>>>
>>>
>>> [1] https://lore.kernel.org/qemu-devel/20260714154246.1242856-1-vsementsov@yandex-team.ru/
>>>
>>
>> I'm also not extremely keen on vhost-net internals, but as I understand
>> with vhost-net we have 2 FDs: TAP FD for the network endpoint and vhost
>> FD for the memory tables, vring addresses etc. The TAP FD is
>> process-independent state - and we pass it via migration channel. The
>> vhost FD is entirely per owning process - and we're able to close it on
>> source, reopen on target and bind to the passed TAP FD.
>
> Yes, my understanding matches.
>
>>
>> However with vhost-vsock there's only one FD. It's used both for
>> plumbing and the endpoint. At the very least, vhost device state holds
>> guest CID registration in the global vhost_vsock_hash. If we go the
>> "recreate/reopen the device" path - CID is removed from the hash in
>> vhost_vsock_dev_release(), and then we're gonna get ECONNRESET for any
>> incoming packets => connections are dead. We can't afford that.
>>
>
> OK, thanks for explanation!
>
>
On 26.06.2026 18:46, Andrey Drobyshev wrote: > v2 -> v3: > > Re-design device ownership hand-off (suggested by Dongli). Do not rely > on SETUP migration notifiers, as this approach forces us to reorder > generic migration code (see v2 discussion). Instead: > > * .pre_save() releases ownership on the source (RESET_OWNER) once VM > is stopped; > * .realize() is adjusted to defer device ownership acquisition for an > incoming CPR; > * .post_load() actually claims the device (VHOST_SET_OWNER); > * FAILED migration event callback re-aquires the ownership on the > source after re_save released it. > > v2: https://lore.kernel.org/qemu-devel/57c9c9b3-d758-489b-95b8-d16c258b9b0c@virtuozzo.com > > Andrey Drobyshev (7): > vhost: add vhost_reset_owner op > vhost-vsock: don't reset connections during CPR > vhost-vsock: fix FD leak in realize() > vhost-vsock: preserve vhost FD during CPR > vhost: factor out vhost_dev_init_backend() > vhost: add vhost_dev_set_owner() / vhost_dev_reset_owner() helpers > vhost-vsock: hand off device ownership across CPR > These patches look okay CPR-wise, so: Acked-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com> # for CPR But the vsock parts still need review by their maintainers/reviewers. Thanks, Maciej
© 2016 - 2026 Red Hat, Inc.