Hi all!

Here is a migration for TAP net backend, including its properties and
open fds.

With this new feature, management software doesn't need to initialize
new TAP and do a switch to it. Nothing should be done around
virtio-net in local migration: it just migrates and continues to use
same TAP device. So we avoid extra logic in management software, extra
allocations in kernel (for new TAP), and corresponding extra delay in
migration downtime.

v19:

02: add r-b by Markus
03: pass parameter name to improve error message; info str: "no" -> ""
09: better documentation wording, keep a-b marks
10: new
13: - better wording
    - keep new property false by default
    - better error message
    - move to errp-variants of pre/post _load functions

v19 is pushed to

    https://gitlab.com/vsementsov/qemu.git
      tag: up-tap-fd-migration-with-bk-opt-v19

To run the test, use sudo, as test needs to configure TAP device:

    sudo PYTHONPATH=python:tests/functional \
    QEMU_TEST_QEMU_BINARY=$PWD/build/qemu-system-x86_64 \
    MESON_BUILD_ROOT=$PWD/build \
    ./build/pyvenv/bin/python3 tests/functional/x86_64/test_tap_migration.py

Or, to test the feature by hand, you may follow the instruction.

The walkthrough uses four terminals:

  source-cmd  -- source VM console (serial output, guest login)
  source-qmp  -- QMP connection to the source QEMU
  target-cmd  -- target VM console (serial output, guest login after migration)
  target-qmp  -- QMP connection to the target QEMU

1. Prerequisites
----------------

    QEMU=/path/to/your/build/qemu-system-x86_64

# download same image as in test

    wget -O /tmp/alpine.iso "https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/x86_64/alpine-standard-3.22.1-x86_64.iso"


# prepare tap device (be careful to not break your own networks)

    sudo ip tuntap add dev tap0 mode tap multi_queue
    sudo ip addr add 192.168.100.1/24 dev tap0
    sudo ip link set tap0 up


2. Start source VM
------------------

In source-cmd, run:

    $QEMU \
        -name source \
        -machine q35 \
        -accel kvm \
        -m 1G \
        -object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
        -machine memory-backend=ram0 \
        -drive file=/tmp/alpine.iso,media=cdrom,format=raw \
        -device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
        -serial stdio \
        -nographic \
        -S \
        -qmp unix:/tmp/qmp-source.sock,server=on,wait=off

Note: the netdev and virtio-net device are added via QMP below for symmetry
with the target.  On the source you could also pass them on the command line
(with local-migration-supported=on).

In source-qmp, connect and add the TAP netdev and virtio-net device:

    socat - UNIX-CONNECT:/tmp/qmp-source.sock

    {"execute": "qmp_capabilities"}

    {"execute": "netdev_add", "arguments": {
        "id": "netdev.1",
        "type": "tap",
        "ifname": "tap0",
        "queues": 4,
        "vnet_hdr": true,
        "script": "no",
        "downscript": "no",
        "local-migration-supported": true
    }}

    {"execute": "device_add", "arguments": {
        "driver": "virtio-net-pci",
        "id": "vnet.1",
        "netdev": "netdev.1",
        "bus": "pci.1",
        "mq": true,
        "vectors": 18,
        "romfile": "",
        "disable-legacy": "off"
    }}

    {"execute": "cont"}

Wait for Alpine to boot in source-cmd.  When you see the login prompt, log
in as root (no password):

    localhost login: root

Configure the guest network:

    ip addr add 192.168.100.2/24 dev eth0
    ip link set eth0 up

Verify connectivity from the guest:

    ping -c 3 192.168.100.1

And from the host (in any spare terminal):

    ping -c 3 192.168.100.2


3. Start target VM
------------------

The TAP netdev must be created via QMP after enabling the "local" migration
parameter — the target will not open tap0 itself; instead it will receive the
TAP file descriptors from the source over the migration channel.

In target-cmd, run:

    $QEMU \
        -name target \
        -machine q35 \
        -accel kvm \
        -m 1G \
        -object memory-backend-file,id=ram0,size=1G,mem-path=/dev/shm/qemu_migration_test,share=on \
        -machine memory-backend=ram0 \
        -drive file=/tmp/alpine.iso,media=cdrom,format=raw \
        -device pcie-pci-bridge,id=pci.1,bus=pcie.0 \
        -serial stdio \
        -nographic \
        -qmp unix:/tmp/qmp-target.sock,server=on,wait=off \
        -incoming defer

In target-qmp, connect, enable local migration, and add the TAP netdev and
virtio-net device.  The "local" parameter must be set before creating the TAP.
Do not pass ifname/fd — the fd will arrive via the migration channel:

    socat - UNIX-CONNECT:/tmp/qmp-target.sock

    {"execute": "qmp_capabilities"}

    {"execute": "migrate-set-capabilities", "arguments": {
        "capabilities": [
            {"capability": "events",          "state": true},
            {"capability": "x-ignore-shared", "state": true}
        ]
    }}

    {"execute": "migrate-set-parameters", "arguments": {"local": true}}

    {"execute": "netdev_add", "arguments": {
        "id": "netdev.1",
        "type": "tap",
        "queues": 4,
        "script": "",
        "downscript": "",
        "local-migration-supported": true
    }}

    {"execute": "device_add", "arguments": {
        "driver": "virtio-net-pci",
        "id": "vnet.1",
        "netdev": "netdev.1",
        "bus": "pci.1",
        "mq": true,
        "vectors": 18,
        "romfile": "",
        "disable-legacy": "off"
    }}

4. Start migration
------------------

In target-qmp, tell the target to listen for the incoming migration:

    {"execute": "migrate-incoming",
     "arguments": {"uri": "unix:/tmp/migration.sock"}}

In source-qmp, configure migration capabilities and parameters:

    {"execute": "migrate-set-capabilities", "arguments": {
        "capabilities": [
            {"capability": "events",          "state": true},
            {"capability": "x-ignore-shared", "state": true}
        ]
    }}

    {"execute": "migrate-set-parameters", "arguments": {"local": true}}

In source-qmp, trigger the migration:

    {"execute": "migrate",
     "arguments": {"uri": "unix:/tmp/migration.sock"}}

Poll migration status until it completes (source-qmp):

    {"execute": "query-migrate"}
    # repeat until "status" == "completed"

Or just wait for the MIGRATION event that QEMU emits automatically:

    # {"event": "MIGRATION", "data": {"status": "completed"}, ...}

Once the source reports "completed", resume the target VM (target-qmp):

    {"execute": "cont"}

The target VM is now running with the migrated state and the TAP file
descriptors that were passed from the source. Still, target console
(in target-cmd) may still be empty, until you at least press Enter in
it.

Verify that the guest is still reachable from the host:

    ping -c 3 192.168.100.2

And from inside the guest in target-cmd:

    ping -c 3 192.168.100.1


5. Cleanup
----------

Shut down the target VM (target-qmp):

    {"execute": "quit"}

Shut down the source VM (source-qmp):

    {"execute": "quit"}

Remove the TAP device:

    sudo ip tuntap del tap0 mode tap multi_queue

Remove the shared memory file:

    rm /dev/shm/qemu_migration_test

Remove leftover sockets if they still exist:

    rm -f /tmp/migration.sock /tmp/qmp-source.sock /tmp/qmp-target.sock 


Vladimir Sementsov-Ogievskiy (15):
  net/tap: rework tap_parse_script
  net/tap: improve script/downscript options documentation
  net/tap: deprecate "no" as special value for script/downscript
  net/tap: move vhost-net open() calls to tap_parse_vhost_fds()
  net/tap: move vhost initialization to tap_setup_vhost()
  net/tap: use container_of instead of DO_UPCAST
  net/tap: QOMify tap backend
  net/tap: add TYPE_VMSTATE_IF interface
  qapi: add local migration parameter
  migration/channel: check that transfer is UNIX socket when "local" set
  virtio-net: support local migration of backend
  net/tap: disable read polling for stopped VM
  net/tap: support local migration with virtio-net
  tests/functional: add skipWithoutSudo() decorator
  tests/functional: add test_tap_migration

 docs/about/deprecated.rst                     |  18 +
 docs/system/i386/microvm.rst                  |   4 +-
 docs/system/i386/xenpvh.rst                   |   2 +-
 docs/system/ppc/ppce500.rst                   |   4 +-
 docs/system/riscv/microchip-icicle-kit.rst    |   2 +-
 docs/system/riscv/sifive_u.rst                |   2 +-
 hw/net/virtio-net.c                           |  89 +++-
 include/hw/virtio/virtio-net.h                |   1 +
 include/migration/misc.h                      |   2 +
 include/migration/vmstate.h                   |   2 +
 include/net/net.h                             |   9 +
 include/net/tap.h                             |   2 +
 migration/channel.c                           |  17 +
 migration/options.c                           |  18 +-
 net/net.c                                     |  14 +-
 net/tap.c                                     | 441 +++++++++++++----
 qapi/migration.json                           |  15 +-
 qapi/net.json                                 |  38 +-
 qemu-options.hx                               |  12 +-
 tests/functional/qemu_test/decorators.py      |  16 +
 tests/functional/x86_64/meson.build           |   1 +
 tests/functional/x86_64/test_tap_migration.py | 455 ++++++++++++++++++
 22 files changed, 1057 insertions(+), 107 deletions(-)
 create mode 100755 tests/functional/x86_64/test_tap_migration.py

-- 
2.43.0

On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Here is a migration for TAP net backend, including its properties and
> open fds.
> 
> With this new feature, management software doesn't need to initialize
> new TAP and do a switch to it. Nothing should be done around
> virtio-net in local migration: it just migrates and continues to use
> same TAP device. So we avoid extra logic in management software, extra
> allocations in kernel (for new TAP), and corresponding extra delay in
> migration downtime.

This is quite a big patch set, - is this really worth the effort to do
all this just for *local* migration?  What's the possible use case for
this in real, - am I right this is just about upgrading the host qemu?
And with that in mind, isn't it sufficient to use what we already have
(namely, create new tap, start new qemu instance, and migrate the usual
way), and tolerate some very minor downtime while the networking code
learns the new network topology (isn't it happening almost instantly
anyway, and if not, the management can help by sending gratitious ARP)?

I wonder what's the use for this at yandex?

Thanks,

/mjt

On 15.07.26 18:01, Michael Tokarev wrote:
> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Here is a migration for TAP net backend, including its properties and
>> open fds.
>>
>> With this new feature, management software doesn't need to initialize
>> new TAP and do a switch to it. Nothing should be done around
>> virtio-net in local migration: it just migrates and continues to use
>> same TAP device. So we avoid extra logic in management software, extra
>> allocations in kernel (for new TAP), and corresponding extra delay in
>> migration downtime.
> 
> This is quite a big patch set,

You probably haven't been following Steve's work (about 5 years) on
CPR Live Update :) My patch set is small.

> - is this really worth the effort to do
> all this just for *local* migration?  What's the possible use case for
> this in real, - am I right this is just about upgrading the host qemu?

Exactly. You say "just for", but actually local migration is a lot more
"massive" operation for us. Remote migrations are done mostly to release the
physical server for service/redeploy (you should first migrate all the
vms to other servers).  That's a relatively seldom operation.

On the contrary, when updating to new QEMU version, you migrate _all_
vms on _all_ servers.  These (minimal) downtimes affects all the
customers, some are sensitive to freezes.

> And with that in mind, isn't it sufficient to use what we already have
> (namely, create new tap, start new qemu instance, and migrate the usual
> way), and tolerate some very minor downtime while the networking code
> learns the new network topology (isn't it happening almost instantly
> anyway, and if not, the management can help by sending gratitious ARP)?
> 
> I wonder what's the use for this at yandex?
> 

This series together with similar about vhost-user-blk local migration
gives several times win in freeze-time. It's significant for us.

-- 
Best regards,
Vladimir

On Wed, Jul 15, 2026 at 11:10:28PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 15.07.26 18:01, Michael Tokarev wrote:
> > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > Hi all!
> > > 
> > > Here is a migration for TAP net backend, including its properties and
> > > open fds.
> > > 
> > > With this new feature, management software doesn't need to initialize
> > > new TAP and do a switch to it. Nothing should be done around
> > > virtio-net in local migration: it just migrates and continues to use
> > > same TAP device. So we avoid extra logic in management software, extra
> > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > migration downtime.
> > 
> > This is quite a big patch set,
> 
> You probably haven't been following Steve's work (about 5 years) on
> CPR Live Update :) My patch set is small.
> 
> > - is this really worth the effort to do
> > all this just for *local* migration?  What's the possible use case for
> > this in real, - am I right this is just about upgrading the host qemu?
> 
> Exactly. You say "just for", but actually local migration is a lot more
> "massive" operation for us. Remote migrations are done mostly to release the
> physical server for service/redeploy (you should first migrate all the
> vms to other servers).  That's a relatively seldom operation.
> 
> On the contrary, when updating to new QEMU version, you migrate _all_
> vms on _all_ servers.  These (minimal) downtimes affects all the
> customers, some are sensitive to freezes.
> 
> > And with that in mind, isn't it sufficient to use what we already have
> > (namely, create new tap, start new qemu instance, and migrate the usual
> > way), and tolerate some very minor downtime while the networking code
> > learns the new network topology (isn't it happening almost instantly
> > anyway, and if not, the management can help by sending gratitious ARP)?
> > 
> > I wonder what's the use for this at yandex?
> > 
> 
> This series together with similar about vhost-user-blk local migration
> gives several times win in freeze-time. It's significant for us.
> 
> -- 
> Best regards,
> Vladimir


I believe it is.
But let's focus on what exactly are we saving here.
It is not that clear.

Is it the overhead of tap losing state such as
bridge forgetting the tap mac. Or what?


-- 
MST

On 15.07.26 23:20, Michael S. Tsirkin wrote:
> On Wed, Jul 15, 2026 at 11:10:28PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 15.07.26 18:01, Michael Tokarev wrote:
>>> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>>>> Hi all!
>>>>
>>>> Here is a migration for TAP net backend, including its properties and
>>>> open fds.
>>>>
>>>> With this new feature, management software doesn't need to initialize
>>>> new TAP and do a switch to it. Nothing should be done around
>>>> virtio-net in local migration: it just migrates and continues to use
>>>> same TAP device. So we avoid extra logic in management software, extra
>>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>>> migration downtime.
>>>
>>> This is quite a big patch set,
>>
>> You probably haven't been following Steve's work (about 5 years) on
>> CPR Live Update :) My patch set is small.
>>
>>> - is this really worth the effort to do
>>> all this just for *local* migration?  What's the possible use case for
>>> this in real, - am I right this is just about upgrading the host qemu?
>>
>> Exactly. You say "just for", but actually local migration is a lot more
>> "massive" operation for us. Remote migrations are done mostly to release the
>> physical server for service/redeploy (you should first migrate all the
>> vms to other servers).  That's a relatively seldom operation.
>>
>> On the contrary, when updating to new QEMU version, you migrate _all_
>> vms on _all_ servers.  These (minimal) downtimes affects all the
>> customers, some are sensitive to freezes.
>>
>>> And with that in mind, isn't it sufficient to use what we already have
>>> (namely, create new tap, start new qemu instance, and migrate the usual
>>> way), and tolerate some very minor downtime while the networking code
>>> learns the new network topology (isn't it happening almost instantly
>>> anyway, and if not, the management can help by sending gratitious ARP)?
>>>
>>> I wonder what's the use for this at yandex?
>>>
>>
>> This series together with similar about vhost-user-blk local migration
>> gives several times win in freeze-time. It's significant for us.
>>
>> -- 
>> Best regards,
>> Vladimir
> 
> 
> I believe it is.
> But let's focus on what exactly are we saving here.
> It is not that clear.
> 
> Is it the overhead of tap losing state such as
> bridge forgetting the tap mac. Or what?
> 

I don't have a full expertize here, but the overhead is in switching
to a new TAP in our cloud virtual networking component. Could this
switching be optimized, to always work very fast? That's a good
question that I don't have an answer to. Probably yes.

We decided to go another way: drop extra components from the process,
and avoid any extra switching at all. Both cloud networking component, and
storage (with vhost-usr-blk live-udpate series) don't even know, that
QEMU is migrating. That seems to be a clear design for local migration,
and shows good results.

-- 
Best regards,
Vladimir

On Wed, Jul 15, 2026 at 11:48:02PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 15.07.26 23:20, Michael S. Tsirkin wrote:
> > On Wed, Jul 15, 2026 at 11:10:28PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 15.07.26 18:01, Michael Tokarev wrote:
> > > > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > > > Hi all!
> > > > > 
> > > > > Here is a migration for TAP net backend, including its properties and
> > > > > open fds.
> > > > > 
> > > > > With this new feature, management software doesn't need to initialize
> > > > > new TAP and do a switch to it. Nothing should be done around
> > > > > virtio-net in local migration: it just migrates and continues to use
> > > > > same TAP device. So we avoid extra logic in management software, extra
> > > > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > > > migration downtime.
> > > > 
> > > > This is quite a big patch set,
> > > 
> > > You probably haven't been following Steve's work (about 5 years) on
> > > CPR Live Update :) My patch set is small.
> > > 
> > > > - is this really worth the effort to do
> > > > all this just for *local* migration?  What's the possible use case for
> > > > this in real, - am I right this is just about upgrading the host qemu?
> > > 
> > > Exactly. You say "just for", but actually local migration is a lot more
> > > "massive" operation for us. Remote migrations are done mostly to release the
> > > physical server for service/redeploy (you should first migrate all the
> > > vms to other servers).  That's a relatively seldom operation.
> > > 
> > > On the contrary, when updating to new QEMU version, you migrate _all_
> > > vms on _all_ servers.  These (minimal) downtimes affects all the
> > > customers, some are sensitive to freezes.
> > > 
> > > > And with that in mind, isn't it sufficient to use what we already have
> > > > (namely, create new tap, start new qemu instance, and migrate the usual
> > > > way), and tolerate some very minor downtime while the networking code
> > > > learns the new network topology (isn't it happening almost instantly
> > > > anyway, and if not, the management can help by sending gratitious ARP)?
> > > > 
> > > > I wonder what's the use for this at yandex?
> > > > 
> > > 
> > > This series together with similar about vhost-user-blk local migration
> > > gives several times win in freeze-time. It's significant for us.
> > > 
> > > -- 
> > > Best regards,
> > > Vladimir
> > 
> > 
> > I believe it is.
> > But let's focus on what exactly are we saving here.
> > It is not that clear.
> > 
> > Is it the overhead of tap losing state such as
> > bridge forgetting the tap mac. Or what?
> > 
> 
> I don't have a full expertize here, but the overhead is in switching
> to a new TAP in our cloud virtual networking component.


So what is wrong with
1. using a persistent TAP
or
2. starting a server and handing out these FDs?

> Could this
> switching be optimized, to always work very fast? That's a good
> question that I don't have an answer to. Probably yes.
> 
> We decided to go another way: drop extra components from the process,
> and avoid any extra switching at all. Both cloud networking component, and
> storage (with vhost-usr-blk live-udpate series) don't even know, that
> QEMU is migrating. That seems to be a clear design for local migration,
> and shows good results.
> 
> -- 
> Best regards,
> Vladimir

On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > Hi all!
> > 
> > Here is a migration for TAP net backend, including its properties and
> > open fds.
> > 
> > With this new feature, management software doesn't need to initialize
> > new TAP and do a switch to it. Nothing should be done around
> > virtio-net in local migration: it just migrates and continues to use
> > same TAP device. So we avoid extra logic in management software, extra
> > allocations in kernel (for new TAP), and corresponding extra delay in
> > migration downtime.
> 
> This is quite a big patch set, - is this really worth the effort to do
> all this just for *local* migration?  What's the possible use case for
> this in real, - am I right this is just about upgrading the host qemu?
> And with that in mind, isn't it sufficient to use what we already have
> (namely, create new tap, start new qemu instance, and migrate the usual
> way), and tolerate some very minor downtime while the networking code
> learns the new network topology (isn't it happening almost instantly
> anyway, and if not, the management can help by sending gratitious ARP)?
> 
> I wonder what's the use for this at yandex?
> 
> Thanks,
> 
> /mjt

Well just theoretically, imagine a big VM, reserving twice the amount
of memory just to migrate is not nice at all.

On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
> On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > Hi all!
> > > 
> > > Here is a migration for TAP net backend, including its properties and
> > > open fds.
> > > 
> > > With this new feature, management software doesn't need to initialize
> > > new TAP and do a switch to it. Nothing should be done around
> > > virtio-net in local migration: it just migrates and continues to use
> > > same TAP device. So we avoid extra logic in management software, extra
> > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > migration downtime.
> > 
> > This is quite a big patch set, - is this really worth the effort to do
> > all this just for *local* migration?  What's the possible use case for
> > this in real, - am I right this is just about upgrading the host qemu?
> > And with that in mind, isn't it sufficient to use what we already have
> > (namely, create new tap, start new qemu instance, and migrate the usual
> > way), and tolerate some very minor downtime while the networking code
> > learns the new network topology (isn't it happening almost instantly
> > anyway, and if not, the management can help by sending gratitious ARP)?
> > 
> > I wonder what's the use for this at yandex?
> > 
> > Thanks,
> > 
> > /mjt
> 
> Well just theoretically, imagine a big VM, reserving twice the amount
> of memory just to migrate is not nice at all.

Don't we already have the ability to skip memory transfer by setting
the "x-ignore-shared" capability, assuming the VM RAM has a shared
memory backing.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
> On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
> > On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> > > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > > Hi all!
> > > > 
> > > > Here is a migration for TAP net backend, including its properties and
> > > > open fds.
> > > > 
> > > > With this new feature, management software doesn't need to initialize
> > > > new TAP and do a switch to it. Nothing should be done around
> > > > virtio-net in local migration: it just migrates and continues to use
> > > > same TAP device. So we avoid extra logic in management software, extra
> > > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > > migration downtime.
> > > 
> > > This is quite a big patch set, - is this really worth the effort to do
> > > all this just for *local* migration?  What's the possible use case for
> > > this in real, - am I right this is just about upgrading the host qemu?
> > > And with that in mind, isn't it sufficient to use what we already have
> > > (namely, create new tap, start new qemu instance, and migrate the usual
> > > way), and tolerate some very minor downtime while the networking code
> > > learns the new network topology (isn't it happening almost instantly
> > > anyway, and if not, the management can help by sending gratitious ARP)?
> > > 
> > > I wonder what's the use for this at yandex?
> > > 
> > > Thanks,
> > > 
> > > /mjt
> > 
> > Well just theoretically, imagine a big VM, reserving twice the amount
> > of memory just to migrate is not nice at all.
> 
> Don't we already have the ability to skip memory transfer by setting
> the "x-ignore-shared" capability, assuming the VM RAM has a shared
> memory backing.

Right, IIUC all similar single-host migrations like this series or CPR (or
anything else...) should always need to enable x-ignore-shared in the first
place.  That's almost always the starting point of optimizing local
migrations.. no matter how the memory will be shared (by the same pool of
page cache, or persisted over kexec, etc.).

Thanks,

-- 
Peter Xu

On 15.07.26 19:00, Peter Xu wrote:
> On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
>> On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
>>> On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
>>>> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Hi all!
>>>>>
>>>>> Here is a migration for TAP net backend, including its properties and
>>>>> open fds.
>>>>>
>>>>> With this new feature, management software doesn't need to initialize
>>>>> new TAP and do a switch to it. Nothing should be done around
>>>>> virtio-net in local migration: it just migrates and continues to use
>>>>> same TAP device. So we avoid extra logic in management software, extra
>>>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>>>> migration downtime.
>>>>
>>>> This is quite a big patch set, - is this really worth the effort to do
>>>> all this just for *local* migration?  What's the possible use case for
>>>> this in real, - am I right this is just about upgrading the host qemu?
>>>> And with that in mind, isn't it sufficient to use what we already have
>>>> (namely, create new tap, start new qemu instance, and migrate the usual
>>>> way), and tolerate some very minor downtime while the networking code
>>>> learns the new network topology (isn't it happening almost instantly
>>>> anyway, and if not, the management can help by sending gratitious ARP)?
>>>>
>>>> I wonder what's the use for this at yandex?
>>>>
>>>> Thanks,
>>>>
>>>> /mjt
>>>
>>> Well just theoretically, imagine a big VM, reserving twice the amount
>>> of memory just to migrate is not nice at all.
>>
>> Don't we already have the ability to skip memory transfer by setting
>> the "x-ignore-shared" capability, assuming the VM RAM has a shared
>> memory backing.
> 
> Right, IIUC all similar single-host migrations like this series or CPR (or
> anything else...) should always need to enable x-ignore-shared in the first
> place.  That's almost always the starting point of optimizing local
> migrations.. no matter how the memory will be shared (by the same pool of
> page cache, or persisted over kexec, etc.).
> 

Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.

This series optimizes TAP recreating. Not only skip recreating but also allow to
exclude cloud-networking component form live-update entirely, making the process
simpler (less components involved), and as I already said, reducing corresponding
downtime.

-- 
Best regards,
Vladimir

On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 15.07.26 19:00, Peter Xu wrote:
> > On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
> > > > On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> > > > > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > > > > Hi all!
> > > > > > 
> > > > > > Here is a migration for TAP net backend, including its properties and
> > > > > > open fds.
> > > > > > 
> > > > > > With this new feature, management software doesn't need to initialize
> > > > > > new TAP and do a switch to it. Nothing should be done around
> > > > > > virtio-net in local migration: it just migrates and continues to use
> > > > > > same TAP device. So we avoid extra logic in management software, extra
> > > > > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > > > > migration downtime.
> > > > > 
> > > > > This is quite a big patch set, - is this really worth the effort to do
> > > > > all this just for *local* migration?  What's the possible use case for
> > > > > this in real, - am I right this is just about upgrading the host qemu?
> > > > > And with that in mind, isn't it sufficient to use what we already have
> > > > > (namely, create new tap, start new qemu instance, and migrate the usual
> > > > > way), and tolerate some very minor downtime while the networking code
> > > > > learns the new network topology (isn't it happening almost instantly
> > > > > anyway, and if not, the management can help by sending gratitious ARP)?
> > > > > 
> > > > > I wonder what's the use for this at yandex?
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > /mjt
> > > > 
> > > > Well just theoretically, imagine a big VM, reserving twice the amount
> > > > of memory just to migrate is not nice at all.
> > > 
> > > Don't we already have the ability to skip memory transfer by setting
> > > the "x-ignore-shared" capability, assuming the VM RAM has a shared
> > > memory backing.
> > 
> > Right, IIUC all similar single-host migrations like this series or CPR (or
> > anything else...) should always need to enable x-ignore-shared in the first
> > place.  That's almost always the starting point of optimizing local
> > migrations.. no matter how the memory will be shared (by the same pool of
> > page cache, or persisted over kexec, etc.).
> > 
> 
> Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
> 
> This series optimizes TAP recreating. Not only skip recreating but also allow to
> exclude cloud-networking component form live-update entirely, making the process
> simpler (less components involved), and as I already said, reducing corresponding
> downtime.
> 
> -- 
> Best regards,
> Vladimir

So can you explain, how is this better than
1. a persistent tap
2. a non persistent tap that some server gets a hold of

IOW why does qemu need to bother.
-- 
MST

On 15.07.26 23:32, Michael S. Tsirkin wrote:
> On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 15.07.26 19:00, Peter Xu wrote:
>>> On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
>>>> On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
>>>>> On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
>>>>>> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>> Hi all!
>>>>>>>
>>>>>>> Here is a migration for TAP net backend, including its properties and
>>>>>>> open fds.
>>>>>>>
>>>>>>> With this new feature, management software doesn't need to initialize
>>>>>>> new TAP and do a switch to it. Nothing should be done around
>>>>>>> virtio-net in local migration: it just migrates and continues to use
>>>>>>> same TAP device. So we avoid extra logic in management software, extra
>>>>>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>>>>>> migration downtime.
>>>>>>
>>>>>> This is quite a big patch set, - is this really worth the effort to do
>>>>>> all this just for *local* migration?  What's the possible use case for
>>>>>> this in real, - am I right this is just about upgrading the host qemu?
>>>>>> And with that in mind, isn't it sufficient to use what we already have
>>>>>> (namely, create new tap, start new qemu instance, and migrate the usual
>>>>>> way), and tolerate some very minor downtime while the networking code
>>>>>> learns the new network topology (isn't it happening almost instantly
>>>>>> anyway, and if not, the management can help by sending gratitious ARP)?
>>>>>>
>>>>>> I wonder what's the use for this at yandex?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /mjt
>>>>>
>>>>> Well just theoretically, imagine a big VM, reserving twice the amount
>>>>> of memory just to migrate is not nice at all.
>>>>
>>>> Don't we already have the ability to skip memory transfer by setting
>>>> the "x-ignore-shared" capability, assuming the VM RAM has a shared
>>>> memory backing.
>>>
>>> Right, IIUC all similar single-host migrations like this series or CPR (or
>>> anything else...) should always need to enable x-ignore-shared in the first
>>> place.  That's almost always the starting point of optimizing local
>>> migrations.. no matter how the memory will be shared (by the same pool of
>>> page cache, or persisted over kexec, etc.).
>>>
>>
>> Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
>>
>> This series optimizes TAP recreating. Not only skip recreating but also allow to
>> exclude cloud-networking component form live-update entirely, making the process
>> simpler (less components involved), and as I already said, reducing corresponding
>> downtime.
>>
>> -- 
>> Best regards,
>> Vladimir
> 
> So can you explain, how is this better than
> 1. a persistent tap

You mean open same tap device both on source and target?

This will require some additional steps anyway, to avoid packet loss,
like keeping queues disabled on target until post_load.

Another thing is MAX_TAP_QUEUES=256 in kernel: this is a problem, if you have more
than 128 queues already opened on source. Seems cleaner just pass already opened
queues to the target.

And finally, on hosts with many CPUs, TAP queue allocate noticeable amount
of RAM, so having x2 queues during migration would be an overhead.

> 2. a non persistent tap that some server gets a hold of

You mean, just pass FDs externally, instead of using QEMUs migration stream?
That's possible. But requires mgmt tool to store (or get from source) and pass
these FDs. Requires mgmt to even know about these FDs. But why? QEMU already
can pass FDs through migration for vfio devices (CPR), why is TAP worse?

Like with persistent tap, it will require some changes in Qemu anyway, to avoid packet loss
(like patch 12/15 here).

> 
> IOW why does qemu need to bother.

QEMU owns the TAP fd and has full knowledge of its state. Pushing this responsibility
to an external tool means the tool needs to understand QEMU internals just to pass
an fd that QEMU already has. I think, that's a worse separation of concerns.

Of course, there are other ways to do TAP local migration. But looking at wider
picture, where we want to migrate not only TAP, but also vfio devices (already
implemented as CPR migration, but may be updated to use similar approach as in
this series, to use one migration channel), vhost-user-blk (my another series
in flight) and vhost-user-fs (not yet published), vhost-vsock migration
("[PATCH v3 0/7] migration/cpr: support vhost-vsock devices" in flight series),
it seems a good generic approach: simply pass backends (including open FDs) to
the target, not involving mgmt. Qemu has full knowledge about these FDs and
owns the whole state. Migrating them in QEMU looks correct for me.

-- 
Best regards,
Vladimir

On Thu, Jul 16, 2026 at 01:11:05AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 15.07.26 23:32, Michael S. Tsirkin wrote:
> > On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 15.07.26 19:00, Peter Xu wrote:
> > > > On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
> > > > > On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
> > > > > > On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> > > > > > > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > > > > > > Hi all!
> > > > > > > > 
> > > > > > > > Here is a migration for TAP net backend, including its properties and
> > > > > > > > open fds.
> > > > > > > > 
> > > > > > > > With this new feature, management software doesn't need to initialize
> > > > > > > > new TAP and do a switch to it. Nothing should be done around
> > > > > > > > virtio-net in local migration: it just migrates and continues to use
> > > > > > > > same TAP device. So we avoid extra logic in management software, extra
> > > > > > > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > > > > > > migration downtime.
> > > > > > > 
> > > > > > > This is quite a big patch set, - is this really worth the effort to do
> > > > > > > all this just for *local* migration?  What's the possible use case for
> > > > > > > this in real, - am I right this is just about upgrading the host qemu?
> > > > > > > And with that in mind, isn't it sufficient to use what we already have
> > > > > > > (namely, create new tap, start new qemu instance, and migrate the usual
> > > > > > > way), and tolerate some very minor downtime while the networking code
> > > > > > > learns the new network topology (isn't it happening almost instantly
> > > > > > > anyway, and if not, the management can help by sending gratitious ARP)?
> > > > > > > 
> > > > > > > I wonder what's the use for this at yandex?
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > 
> > > > > > > /mjt
> > > > > > 
> > > > > > Well just theoretically, imagine a big VM, reserving twice the amount
> > > > > > of memory just to migrate is not nice at all.
> > > > > 
> > > > > Don't we already have the ability to skip memory transfer by setting
> > > > > the "x-ignore-shared" capability, assuming the VM RAM has a shared
> > > > > memory backing.
> > > > 
> > > > Right, IIUC all similar single-host migrations like this series or CPR (or
> > > > anything else...) should always need to enable x-ignore-shared in the first
> > > > place.  That's almost always the starting point of optimizing local
> > > > migrations.. no matter how the memory will be shared (by the same pool of
> > > > page cache, or persisted over kexec, etc.).
> > > > 
> > > 
> > > Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
> > > 
> > > This series optimizes TAP recreating. Not only skip recreating but also allow to
> > > exclude cloud-networking component form live-update entirely, making the process
> > > simpler (less components involved), and as I already said, reducing corresponding
> > > downtime.
> > > 
> > > -- 
> > > Best regards,
> > > Vladimir
> > 
> > So can you explain, how is this better than
> > 1. a persistent tap
> 
> You mean open same tap device both on source and target?
> 
> This will require some additional steps anyway, to avoid packet loss,
> like keeping queues disabled on target until post_load.
> 
> Another thing is MAX_TAP_QUEUES=256 in kernel: this is a problem, if you have more
> than 128 queues already opened on source. Seems cleaner just pass already opened
> queues to the target.
> 
> And finally, on hosts with many CPUs, TAP queue allocate noticeable amount
> of RAM, so having x2 queues during migration would be an overhead.

ah I forgot. yes queues do get 

> > 2. a non persistent tap that some server gets a hold of
> 
> You mean, just pass FDs externally, instead of using QEMUs migration stream?
> That's possible. But requires mgmt tool to store (or get from source) and pass
> these FDs. Requires mgmt to even know about these FDs.

e.g. libvirt already does, right? it creates them?

> But why? QEMU already
> can pass FDs through migration for vfio devices (CPR), why is TAP worse?

it's not that it's worse. it's that we are growing bespoke mechanisms so far.
so if qemu gets tap fd on command line then what? how does that
interact?

> Like with persistent tap, it will require some changes in Qemu anyway, to avoid packet loss
> (like patch 12/15 here).

that one is more like a bugfix.

> > 
> > IOW why does qemu need to bother.
> 
> QEMU owns the TAP fd and has full knowledge of its state. Pushing this responsibility
> to an external tool means the tool needs to understand QEMU internals just to pass
> an fd that QEMU already has. I think, that's a worse separation of concerns.
> 
> Of course, there are other ways to do TAP local migration. But looking at wider
> picture, where we want to migrate not only TAP, but also vfio devices (already
> implemented as CPR migration, but may be updated to use similar approach as in
> this series, to use one migration channel), vhost-user-blk (my another series
> in flight) and vhost-user-fs (not yet published), vhost-vsock migration
> ("[PATCH v3 0/7] migration/cpr: support vhost-vsock devices" in flight series),
> it seems a good generic approach: simply pass backends (including open FDs) to
> the target, not involving mgmt. Qemu has full knowledge about these FDs and
> owns the whole state. Migrating them in QEMU looks correct for me.

if you find a way to generalize things and reuse them for your
purposes without intrusive changes all over qemu, fine.

But this one is poking as far as frontend code even.


> -- 
> Best regards,
> Vladimir

On 16.07.26 01:45, Michael S. Tsirkin wrote:
> On Thu, Jul 16, 2026 at 01:11:05AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 15.07.26 23:32, Michael S. Tsirkin wrote:
>>> On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 15.07.26 19:00, Peter Xu wrote:
>>>>> On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
>>>>>> On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
>>>>>>> On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
>>>>>>>> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>> Hi all!
>>>>>>>>>
>>>>>>>>> Here is a migration for TAP net backend, including its properties and
>>>>>>>>> open fds.
>>>>>>>>>
>>>>>>>>> With this new feature, management software doesn't need to initialize
>>>>>>>>> new TAP and do a switch to it. Nothing should be done around
>>>>>>>>> virtio-net in local migration: it just migrates and continues to use
>>>>>>>>> same TAP device. So we avoid extra logic in management software, extra
>>>>>>>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>>>>>>>> migration downtime.
>>>>>>>>
>>>>>>>> This is quite a big patch set, - is this really worth the effort to do
>>>>>>>> all this just for *local* migration?  What's the possible use case for
>>>>>>>> this in real, - am I right this is just about upgrading the host qemu?
>>>>>>>> And with that in mind, isn't it sufficient to use what we already have
>>>>>>>> (namely, create new tap, start new qemu instance, and migrate the usual
>>>>>>>> way), and tolerate some very minor downtime while the networking code
>>>>>>>> learns the new network topology (isn't it happening almost instantly
>>>>>>>> anyway, and if not, the management can help by sending gratitious ARP)?
>>>>>>>>
>>>>>>>> I wonder what's the use for this at yandex?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> /mjt
>>>>>>>
>>>>>>> Well just theoretically, imagine a big VM, reserving twice the amount
>>>>>>> of memory just to migrate is not nice at all.
>>>>>>
>>>>>> Don't we already have the ability to skip memory transfer by setting
>>>>>> the "x-ignore-shared" capability, assuming the VM RAM has a shared
>>>>>> memory backing.
>>>>>
>>>>> Right, IIUC all similar single-host migrations like this series or CPR (or
>>>>> anything else...) should always need to enable x-ignore-shared in the first
>>>>> place.  That's almost always the starting point of optimizing local
>>>>> migrations.. no matter how the memory will be shared (by the same pool of
>>>>> page cache, or persisted over kexec, etc.).
>>>>>
>>>>
>>>> Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
>>>>
>>>> This series optimizes TAP recreating. Not only skip recreating but also allow to
>>>> exclude cloud-networking component form live-update entirely, making the process
>>>> simpler (less components involved), and as I already said, reducing corresponding
>>>> downtime.
>>>>
>>>> -- 
>>>> Best regards,
>>>> Vladimir
>>>
>>> So can you explain, how is this better than
>>> 1. a persistent tap
>>
>> You mean open same tap device both on source and target?
>>
>> This will require some additional steps anyway, to avoid packet loss,
>> like keeping queues disabled on target until post_load.
>>
>> Another thing is MAX_TAP_QUEUES=256 in kernel: this is a problem, if you have more
>> than 128 queues already opened on source. Seems cleaner just pass already opened
>> queues to the target.
>>
>> And finally, on hosts with many CPUs, TAP queue allocate noticeable amount
>> of RAM, so having x2 queues during migration would be an overhead.
> 
> ah I forgot. yes queues do get
> 
>>> 2. a non persistent tap that some server gets a hold of
>>
>> You mean, just pass FDs externally, instead of using QEMUs migration stream?
>> That's possible. But requires mgmt tool to store (or get from source) and pass
>> these FDs. Requires mgmt to even know about these FDs.
> 
> e.g. libvirt already does, right? it creates them?

Not sure. But even if it creates and passes them to QEMU. Does it store them?
Libvirt may restart, and it shouldn't break further migration.

So, we'll need also an API to get FDs from QEMU. To get FDs from QEMU just
to pass them to target QEMU - I don't think it's more clear solution.

> 
>> But why? QEMU already
>> can pass FDs through migration for vfio devices (CPR), why is TAP worse?
> 
> it's not that it's worse. it's that we are growing bespoke mechanisms so far.
> so if qemu gets tap fd on command line then what? how does that
> interact?

Do you mean how this series handles that case? No problem: given on source from command line
FDs will migrate to the target the same way. Or I don't understand the question.

> 
>> Like with persistent tap, it will require some changes in Qemu anyway, to avoid packet loss
>> (like patch 12/15 here).
> 
> that one is more like a bugfix.
> 
>>>
>>> IOW why does qemu need to bother.
>>
>> QEMU owns the TAP fd and has full knowledge of its state. Pushing this responsibility
>> to an external tool means the tool needs to understand QEMU internals just to pass
>> an fd that QEMU already has. I think, that's a worse separation of concerns.
>>
>> Of course, there are other ways to do TAP local migration. But looking at wider
>> picture, where we want to migrate not only TAP, but also vfio devices (already
>> implemented as CPR migration, but may be updated to use similar approach as in
>> this series, to use one migration channel), vhost-user-blk (my another series
>> in flight) and vhost-user-fs (not yet published), vhost-vsock migration
>> ("[PATCH v3 0/7] migration/cpr: support vhost-vsock devices" in flight series),
>> it seems a good generic approach: simply pass backends (including open FDs) to
>> the target, not involving mgmt. Qemu has full knowledge about these FDs and
>> owns the whole state. Migrating them in QEMU looks correct for me.
> 
> if you find a way to generalize things and reuse them for your
> purposes without intrusive changes all over qemu, fine.

That's not only my purposes (i.e., as I understand, not only Yandex use case).
Originally the work was started by Oracle (CPR), and I see interest from other
companies too.

> 
> But this one is poking as far as frontend code even.
> 

You mean "[PATCH v19 11/15] virtio-net: support local migration of backend" ?

CPR-approach avoids it, because FDs are passed _before_ any devices created
on target. Is that better? In this way I don't really like the fact that we
start to use (call some ioctls) in target QEMU the FDs, which are still
actively used in running source QEMU . I don't like it, but that works,
and I started working on live-migration on TAP from CPR tap series by
Steve. Still, such approach doesn't work with vhost-user-blk, as trying
to share (even for initialization) the fd between source and running
target may just break things. So I decided to move to similar approach for both
net and storage. Do you think it's better to return to CPR-approach for TAP
(accepting that it won't generalize to vhost-user) like in
"[PATCH v4 0/8] Live update: tap and vhost" -
https://lore.kernel.org/qemu-devel/20260128-cpr-tap-v4-0-48e334d4216b@akamai.com/ -
where both approaches were compared and mine was preferred?


-- 
Best regards,
Vladimir

On Thu, Jul 16, 2026 at 03:30:26AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> On 16.07.26 01:45, Michael S. Tsirkin wrote:
> > On Thu, Jul 16, 2026 at 01:11:05AM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > On 15.07.26 23:32, Michael S. Tsirkin wrote:
> > > > On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > > > > On 15.07.26 19:00, Peter Xu wrote:
> > > > > > On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
> > > > > > > On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
> > > > > > > > > On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
> > > > > > > > > > Hi all!
> > > > > > > > > > 
> > > > > > > > > > Here is a migration for TAP net backend, including its properties and
> > > > > > > > > > open fds.
> > > > > > > > > > 
> > > > > > > > > > With this new feature, management software doesn't need to initialize
> > > > > > > > > > new TAP and do a switch to it. Nothing should be done around
> > > > > > > > > > virtio-net in local migration: it just migrates and continues to use
> > > > > > > > > > same TAP device. So we avoid extra logic in management software, extra
> > > > > > > > > > allocations in kernel (for new TAP), and corresponding extra delay in
> > > > > > > > > > migration downtime.
> > > > > > > > > 
> > > > > > > > > This is quite a big patch set, - is this really worth the effort to do
> > > > > > > > > all this just for *local* migration?  What's the possible use case for
> > > > > > > > > this in real, - am I right this is just about upgrading the host qemu?
> > > > > > > > > And with that in mind, isn't it sufficient to use what we already have
> > > > > > > > > (namely, create new tap, start new qemu instance, and migrate the usual
> > > > > > > > > way), and tolerate some very minor downtime while the networking code
> > > > > > > > > learns the new network topology (isn't it happening almost instantly
> > > > > > > > > anyway, and if not, the management can help by sending gratitious ARP)?
> > > > > > > > > 
> > > > > > > > > I wonder what's the use for this at yandex?
> > > > > > > > > 
> > > > > > > > > Thanks,
> > > > > > > > > 
> > > > > > > > > /mjt
> > > > > > > > 
> > > > > > > > Well just theoretically, imagine a big VM, reserving twice the amount
> > > > > > > > of memory just to migrate is not nice at all.
> > > > > > > 
> > > > > > > Don't we already have the ability to skip memory transfer by setting
> > > > > > > the "x-ignore-shared" capability, assuming the VM RAM has a shared
> > > > > > > memory backing.
> > > > > > 
> > > > > > Right, IIUC all similar single-host migrations like this series or CPR (or
> > > > > > anything else...) should always need to enable x-ignore-shared in the first
> > > > > > place.  That's almost always the starting point of optimizing local
> > > > > > migrations.. no matter how the memory will be shared (by the same pool of
> > > > > > page cache, or persisted over kexec, etc.).
> > > > > > 
> > > > > 
> > > > > Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
> > > > > 
> > > > > This series optimizes TAP recreating. Not only skip recreating but also allow to
> > > > > exclude cloud-networking component form live-update entirely, making the process
> > > > > simpler (less components involved), and as I already said, reducing corresponding
> > > > > downtime.
> > > > > 
> > > > > -- 
> > > > > Best regards,
> > > > > Vladimir
> > > > 
> > > > So can you explain, how is this better than
> > > > 1. a persistent tap
> > > 
> > > You mean open same tap device both on source and target?
> > > 
> > > This will require some additional steps anyway, to avoid packet loss,
> > > like keeping queues disabled on target until post_load.
> > > 
> > > Another thing is MAX_TAP_QUEUES=256 in kernel: this is a problem, if you have more
> > > than 128 queues already opened on source. Seems cleaner just pass already opened
> > > queues to the target.
> > > 
> > > And finally, on hosts with many CPUs, TAP queue allocate noticeable amount
> > > of RAM, so having x2 queues during migration would be an overhead.
> > 
> > ah I forgot. yes queues do get
> > 
> > > > 2. a non persistent tap that some server gets a hold of
> > > 
> > > You mean, just pass FDs externally, instead of using QEMUs migration stream?
> > > That's possible. But requires mgmt tool to store (or get from source) and pass
> > > these FDs. Requires mgmt to even know about these FDs.
> > 
> > e.g. libvirt already does, right? it creates them?
> 
> Not sure. But even if it creates and passes them to QEMU. Does it store them?
> Libvirt may restart, and it shouldn't break further migration.
> 
> So, we'll need also an API to get FDs from QEMU. To get FDs from QEMU just
> to pass them to target QEMU - I don't think it's more clear solution.
> 
> > 
> > > But why? QEMU already
> > > can pass FDs through migration for vfio devices (CPR), why is TAP worse?
> > 
> > it's not that it's worse. it's that we are growing bespoke mechanisms so far.
> > so if qemu gets tap fd on command line then what? how does that
> > interact?
> 
> Do you mean how this series handles that case? No problem: given on source from command line
> FDs will migrate to the target the same way. Or I don't understand the question.

But right now target gives its own FDs.

> > 
> > > Like with persistent tap, it will require some changes in Qemu anyway, to avoid packet loss
> > > (like patch 12/15 here).
> > 
> > that one is more like a bugfix.
> > 
> > > > 
> > > > IOW why does qemu need to bother.
> > > 
> > > QEMU owns the TAP fd and has full knowledge of its state. Pushing this responsibility
> > > to an external tool means the tool needs to understand QEMU internals just to pass
> > > an fd that QEMU already has. I think, that's a worse separation of concerns.
> > > 
> > > Of course, there are other ways to do TAP local migration. But looking at wider
> > > picture, where we want to migrate not only TAP, but also vfio devices (already
> > > implemented as CPR migration, but may be updated to use similar approach as in
> > > this series, to use one migration channel), vhost-user-blk (my another series
> > > in flight) and vhost-user-fs (not yet published), vhost-vsock migration
> > > ("[PATCH v3 0/7] migration/cpr: support vhost-vsock devices" in flight series),
> > > it seems a good generic approach: simply pass backends (including open FDs) to
> > > the target, not involving mgmt. Qemu has full knowledge about these FDs and
> > > owns the whole state. Migrating them in QEMU looks correct for me.
> > 
> > if you find a way to generalize things and reuse them for your
> > purposes without intrusive changes all over qemu, fine.
> 
> That's not only my purposes (i.e., as I understand, not only Yandex use case).
> Originally the work was started by Oracle (CPR), and I see interest from other
> companies too.
> 
> > 
> > But this one is poking as far as frontend code even.
> > 
> 
> You mean "[PATCH v19 11/15] virtio-net: support local migration of backend" ?
> 
> CPR-approach avoids it, because FDs are passed _before_ any devices created
> on target. Is that better? In this way I don't really like the fact that we
> start to use (call some ioctls) in target QEMU the FDs, which are still
> actively used in running source QEMU . I don't like it, but that works,
> and I started working on live-migration on TAP from CPR tap series by
> Steve. Still, such approach doesn't work with vhost-user-blk, as trying
> to share (even for initialization) the fd between source and running
> target may just break things. So I decided to move to similar approach for both
> net and storage. Do you think it's better to return to CPR-approach for TAP
> (accepting that it won't generalize to vhost-user) like in
> "[PATCH v4 0/8] Live update: tap and vhost" -
> https://lore.kernel.org/qemu-devel/20260128-cpr-tap-v4-0-48e334d4216b@akamai.com/ -
> where both approaches were compared and mine was preferred?

Hmm. Can CPR use yours?

> 
> -- 
> Best regards,
> Vladimir

On 16.07.26 08:18, Michael S. Tsirkin wrote:
> On Thu, Jul 16, 2026 at 03:30:26AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> On 16.07.26 01:45, Michael S. Tsirkin wrote:
>>> On Thu, Jul 16, 2026 at 01:11:05AM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> On 15.07.26 23:32, Michael S. Tsirkin wrote:
>>>>> On Wed, Jul 15, 2026 at 11:21:23PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> On 15.07.26 19:00, Peter Xu wrote:
>>>>>>> On Wed, Jul 15, 2026 at 04:52:48PM +0100, Daniel P. Berrangé wrote:
>>>>>>>> On Wed, Jul 15, 2026 at 11:48:40AM -0400, Michael S. Tsirkin wrote:
>>>>>>>>> On Wed, Jul 15, 2026 at 06:01:39PM +0300, Michael Tokarev wrote:
>>>>>>>>>> On 14.07.2026 18:42, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>>>> Hi all!
>>>>>>>>>>>
>>>>>>>>>>> Here is a migration for TAP net backend, including its properties and
>>>>>>>>>>> open fds.
>>>>>>>>>>>
>>>>>>>>>>> With this new feature, management software doesn't need to initialize
>>>>>>>>>>> new TAP and do a switch to it. Nothing should be done around
>>>>>>>>>>> virtio-net in local migration: it just migrates and continues to use
>>>>>>>>>>> same TAP device. So we avoid extra logic in management software, extra
>>>>>>>>>>> allocations in kernel (for new TAP), and corresponding extra delay in
>>>>>>>>>>> migration downtime.
>>>>>>>>>>
>>>>>>>>>> This is quite a big patch set, - is this really worth the effort to do
>>>>>>>>>> all this just for *local* migration?  What's the possible use case for
>>>>>>>>>> this in real, - am I right this is just about upgrading the host qemu?
>>>>>>>>>> And with that in mind, isn't it sufficient to use what we already have
>>>>>>>>>> (namely, create new tap, start new qemu instance, and migrate the usual
>>>>>>>>>> way), and tolerate some very minor downtime while the networking code
>>>>>>>>>> learns the new network topology (isn't it happening almost instantly
>>>>>>>>>> anyway, and if not, the management can help by sending gratitious ARP)?
>>>>>>>>>>
>>>>>>>>>> I wonder what's the use for this at yandex?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> /mjt
>>>>>>>>>
>>>>>>>>> Well just theoretically, imagine a big VM, reserving twice the amount
>>>>>>>>> of memory just to migrate is not nice at all.
>>>>>>>>
>>>>>>>> Don't we already have the ability to skip memory transfer by setting
>>>>>>>> the "x-ignore-shared" capability, assuming the VM RAM has a shared
>>>>>>>> memory backing.
>>>>>>>
>>>>>>> Right, IIUC all similar single-host migrations like this series or CPR (or
>>>>>>> anything else...) should always need to enable x-ignore-shared in the first
>>>>>>> place.  That's almost always the starting point of optimizing local
>>>>>>> migrations.. no matter how the memory will be shared (by the same pool of
>>>>>>> page cache, or persisted over kexec, etc.).
>>>>>>>
>>>>>>
>>>>>> Yes, sharing RAM between source and target + enabling x-ignore-shared is a first thing to do.
>>>>>>
>>>>>> This series optimizes TAP recreating. Not only skip recreating but also allow to
>>>>>> exclude cloud-networking component form live-update entirely, making the process
>>>>>> simpler (less components involved), and as I already said, reducing corresponding
>>>>>> downtime.
>>>>>>
>>>>>> -- 
>>>>>> Best regards,
>>>>>> Vladimir
>>>>>
>>>>> So can you explain, how is this better than
>>>>> 1. a persistent tap
>>>>
>>>> You mean open same tap device both on source and target?
>>>>
>>>> This will require some additional steps anyway, to avoid packet loss,
>>>> like keeping queues disabled on target until post_load.
>>>>
>>>> Another thing is MAX_TAP_QUEUES=256 in kernel: this is a problem, if you have more
>>>> than 128 queues already opened on source. Seems cleaner just pass already opened
>>>> queues to the target.
>>>>
>>>> And finally, on hosts with many CPUs, TAP queue allocate noticeable amount
>>>> of RAM, so having x2 queues during migration would be an overhead.
>>>
>>> ah I forgot. yes queues do get
>>>
>>>>> 2. a non persistent tap that some server gets a hold of
>>>>
>>>> You mean, just pass FDs externally, instead of using QEMUs migration stream?
>>>> That's possible. But requires mgmt tool to store (or get from source) and pass
>>>> these FDs. Requires mgmt to even know about these FDs.
>>>
>>> e.g. libvirt already does, right? it creates them?
>>
>> Not sure. But even if it creates and passes them to QEMU. Does it store them?
>> Libvirt may restart, and it shouldn't break further migration.
>>
>> So, we'll need also an API to get FDs from QEMU. To get FDs from QEMU just
>> to pass them to target QEMU - I don't think it's more clear solution.
>>
>>>
>>>> But why? QEMU already
>>>> can pass FDs through migration for vfio devices (CPR), why is TAP worse?
>>>
>>> it's not that it's worse. it's that we are growing bespoke mechanisms so far.
>>> so if qemu gets tap fd on command line then what? how does that
>>> interact?
>>
>> Do you mean how this series handles that case? No problem: given on source from command line
>> FDs will migrate to the target the same way. Or I don't understand the question.
> 
> But right now target gives its own FDs.

You can't start local incoming migration of TAP and also pass FDs (by fd= or fds=),
that's checked in 13/15.

> 
>>>
>>>> Like with persistent tap, it will require some changes in Qemu anyway, to avoid packet loss
>>>> (like patch 12/15 here).
>>>
>>> that one is more like a bugfix.
>>>
>>>>>
>>>>> IOW why does qemu need to bother.
>>>>
>>>> QEMU owns the TAP fd and has full knowledge of its state. Pushing this responsibility
>>>> to an external tool means the tool needs to understand QEMU internals just to pass
>>>> an fd that QEMU already has. I think, that's a worse separation of concerns.
>>>>
>>>> Of course, there are other ways to do TAP local migration. But looking at wider
>>>> picture, where we want to migrate not only TAP, but also vfio devices (already
>>>> implemented as CPR migration, but may be updated to use similar approach as in
>>>> this series, to use one migration channel), vhost-user-blk (my another series
>>>> in flight) and vhost-user-fs (not yet published), vhost-vsock migration
>>>> ("[PATCH v3 0/7] migration/cpr: support vhost-vsock devices" in flight series),
>>>> it seems a good generic approach: simply pass backends (including open FDs) to
>>>> the target, not involving mgmt. Qemu has full knowledge about these FDs and
>>>> owns the whole state. Migrating them in QEMU looks correct for me.
>>>
>>> if you find a way to generalize things and reuse them for your
>>> purposes without intrusive changes all over qemu, fine.
>>
>> That's not only my purposes (i.e., as I understand, not only Yandex use case).
>> Originally the work was started by Oracle (CPR), and I see interest from other
>> companies too.
>>
>>>
>>> But this one is poking as far as frontend code even.
>>>
>>
>> You mean "[PATCH v19 11/15] virtio-net: support local migration of backend" ?
>>
>> CPR-approach avoids it, because FDs are passed _before_ any devices created
>> on target. Is that better? In this way I don't really like the fact that we
>> start to use (call some ioctls) in target QEMU the FDs, which are still
>> actively used in running source QEMU . I don't like it, but that works,
>> and I started working on live-migration on TAP from CPR tap series by
>> Steve. Still, such approach doesn't work with vhost-user-blk, as trying
>> to share (even for initialization) the fd between source and running
>> target may just break things. So I decided to move to similar approach for both
>> net and storage. Do you think it's better to return to CPR-approach for TAP
>> (accepting that it won't generalize to vhost-user) like in
>> "[PATCH v4 0/8] Live update: tap and vhost" -
>> https://lore.kernel.org/qemu-devel/20260128-cpr-tap-v4-0-48e334d4216b@akamai.com/ -
>> where both approaches were compared and mine was preferred?
> 
> Hmm. Can CPR use yours?
> 

Yes. For cpr-transfer, TAP will just migrate through main migration channel, like
other (non-CPR) devices. For cpr-exec, Peter noted [1] that cpr-exec removes
FD_CLOEXEC for passed fds. But for TAP FDs we don't set this flag, so it should
work as is. And this series uses the same core mechanism to pass FDs as CPR (see
load_fd() / save_fd() in migration/vmstate-types.c), which in the case of CPR_EXEC
passes FDs simply as integers.

[1] https://lore.kernel.org/qemu-devel/aYN1I0WMl6vqaZx4@x1.local/

-- 
Best regards,
Vladimir

On 7/14/26, 11:43 AM, "Vladimir Sementsov-Ogievskiy" <vsementsov@yandex-team.ru <mailto:vsementsov@yandex-team.ru>> wrote:


> Hi all!
>
>
> Here is a migration for TAP net backend, including its properties and
> open fds.

Reviewed-by: Ben Chaney <bchaney@akamai.com>

I've gone through this patch set again. I had a few minor questions that I'll send inline, but overall LGTM!

Thanks,
        Ben

On 15.07.26 17:46, Chaney, Ben wrote:
> 
> On 7/14/26, 11:43 AM, "Vladimir Sementsov-Ogievskiy" <vsementsov@yandex-team.ru <mailto:vsementsov@yandex-team.ru>> wrote:
> 
> 
>> Hi all!
>>
>>
>> Here is a migration for TAP net backend, including its properties and
>> open fds.
> 
> Reviewed-by: Ben Chaney <bchaney@akamai.com>
> 
> I've gone through this patch set again. I had a few minor questions that I'll send inline, but overall LGTM!
> 
> Thanks,
>          Ben
> 

Thanks!

-- 
Best regards,
Vladimir