1. Patchew
  2. QEMU
  3. View series

[PATCH v2 0/1] migration: multifd live migration improvement

Li Zhang posted 1 patch 2 years, 4 months ago
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20211203115533.31534-1-lizhang@suse.de
Maintainers: Juan Quintela <quintela@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>
migration/multifd.c | 3 +++
1 file changed, 3 insertions(+)
[PATCH v2 0/1] migration: multifd live migration improvement
Posted by Li Zhang 2 years, 4 months ago 
When testing live migration with multifd channels (8, 16, or a bigger number)
and using qemu -incoming (without "defer"), if a network error occurs
(for example, triggering the kernel SYN flooding detection),
the migration fails and the guest hangs forever.

The test environment and the command line is as the following:

QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
Host OS: SLE 15  with kernel: 5.14.5-1-default
Network Card: mlx5 100Gbps
Network card: Intel Corporation I350 Gigabit (1Gbps)

Source:
qemu-system-x86_64 -M q35 -smp 32 -nographic \
        -serial telnet:10.156.208.153:4321,server,nowait \
        -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
        -monitor stdio
Dest:
qemu-system-x86_64 -M q35 -smp 32 -nographic \
        -serial telnet:10.156.208.154:4321,server,nowait \
        -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
        -monitor stdio \
        -incoming tcp:1.0.8.154:4000

(qemu) migrate_set_parameter max-bandwidth 100G
(qemu) migrate_set_capability multifd on
(qemu) migrate_set_parameter multifd-channels 16

The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.

If a network problem happens, TCP ACK is not received by destination
and the destination resets the connection with RST.

No.     Time    Source  Destination     Protocol        Length  Info
119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
No.     Time    Source  Destination     Protocol        Length  Info
125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0

kernel log:
[334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
[334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
[334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
[334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
[335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
[335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.

There are two problems here:
1. On the send side, the main thread is blocked on qemu_thread_join and 
   send threads are blocked on sendmsg
2. On receive side, the receive threads are blocked on qemu_sem_wait to 
   wait for a semaphore. 

The patch is to fix the first problem, and the guest doesn't hang any more. 
But there is no better solution to fix the second problem yet. 

Li Zhang (1):
  multifd: Shut down the QIO channels to avoid blocking the send threads
    when they are terminated.

 migration/multifd.c | 3 +++
 1 file changed, 3 insertions(+)

-- 
2.31.1
Re: [PATCH v2 0/1] migration: multifd live migration improvement
Posted by Dr. David Alan Gilbert 2 years, 4 months ago 
* Li Zhang (lizhang@suse.de) wrote:
> When testing live migration with multifd channels (8, 16, or a bigger number)
> and using qemu -incoming (without "defer"), if a network error occurs
> (for example, triggering the kernel SYN flooding detection),
> the migration fails and the guest hangs forever.
> 
> The test environment and the command line is as the following:
> 
> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
> Host OS: SLE 15  with kernel: 5.14.5-1-default
> Network Card: mlx5 100Gbps
> Network card: Intel Corporation I350 Gigabit (1Gbps)
> 
> Source:
> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>         -serial telnet:10.156.208.153:4321,server,nowait \
>         -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>         -monitor stdio
> Dest:
> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>         -serial telnet:10.156.208.154:4321,server,nowait \
>         -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>         -monitor stdio \
>         -incoming tcp:1.0.8.154:4000
> 
> (qemu) migrate_set_parameter max-bandwidth 100G
> (qemu) migrate_set_capability multifd on
> (qemu) migrate_set_parameter multifd-channels 16
> 
> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
> 
> If a network problem happens, TCP ACK is not received by destination
> and the destination resets the connection with RST.
> 
> No.     Time    Source  Destination     Protocol        Length  Info
> 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
> No.     Time    Source  Destination     Protocol        Length  Info
> 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0
> 
> kernel log:
> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.

Should we document somewhere how to avoid that?  Is there something we
should be doing in the connection code to avoid it?

Dave

> There are two problems here:
> 1. On the send side, the main thread is blocked on qemu_thread_join and 
>    send threads are blocked on sendmsg
> 2. On receive side, the receive threads are blocked on qemu_sem_wait to 
>    wait for a semaphore. 
> 
> The patch is to fix the first problem, and the guest doesn't hang any more. 
> But there is no better solution to fix the second problem yet. 
> 
> Li Zhang (1):
>   multifd: Shut down the QIO channels to avoid blocking the send threads
>     when they are terminated.
> 
>  migration/multifd.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> -- 
> 2.31.1
> 
-- 
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Re: [PATCH v2 0/1] migration: multifd live migration improvement
Posted by Li Zhang 2 years, 4 months ago 
On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
> * Li Zhang (lizhang@suse.de) wrote:
>> When testing live migration with multifd channels (8, 16, or a bigger number)
>> and using qemu -incoming (without "defer"), if a network error occurs
>> (for example, triggering the kernel SYN flooding detection),
>> the migration fails and the guest hangs forever.
>>
>> The test environment and the command line is as the following:
>>
>> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
>> Host OS: SLE 15  with kernel: 5.14.5-1-default
>> Network Card: mlx5 100Gbps
>> Network card: Intel Corporation I350 Gigabit (1Gbps)
>>
>> Source:
>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>          -serial telnet:10.156.208.153:4321,server,nowait \
>>          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>          -monitor stdio
>> Dest:
>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>          -serial telnet:10.156.208.154:4321,server,nowait \
>>          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>          -monitor stdio \
>>          -incoming tcp:1.0.8.154:4000
>>
>> (qemu) migrate_set_parameter max-bandwidth 100G
>> (qemu) migrate_set_capability multifd on
>> (qemu) migrate_set_parameter multifd-channels 16
>>
>> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>>
>> If a network problem happens, TCP ACK is not received by destination
>> and the destination resets the connection with RST.
>>
>> No.     Time    Source  Destination     Protocol        Length  Info
>> 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
>> No.     Time    Source  Destination     Protocol        Length  Info
>> 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0
>>
>> kernel log:
>> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> Should we document somewhere how to avoid that?  Is there something we
> should be doing in the connection code to avoid it?

We should use the command line -incoming defer in QEMU command line 
instead of -incoming ip:port.

And the backlog of the socket will be set as the same as  multifd 
channels,  this problem doesn't happen as far as I test.

If we use --incoming ip:port in the QEMU command line, the backlog of 
the socket is always 1, it will cause the SYN flooding.


>
> Dave
>
>> There are two problems here:
>> 1. On the send side, the main thread is blocked on qemu_thread_join and
>>     send threads are blocked on sendmsg
>> 2. On receive side, the receive threads are blocked on qemu_sem_wait to
>>     wait for a semaphore.
>>
>> The patch is to fix the first problem, and the guest doesn't hang any more.
>> But there is no better solution to fix the second problem yet.
>>
>> Li Zhang (1):
>>    multifd: Shut down the QIO channels to avoid blocking the send threads
>>      when they are terminated.
>>
>>   migration/multifd.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> -- 
>> 2.31.1
>>
Re: [PATCH v2 0/1] migration: multifd live migration improvement
Posted by Daniel P. Berrangé 2 years, 4 months ago 
On Tue, Dec 07, 2021 at 02:45:10PM +0100, Li Zhang wrote:
> 
> On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
> > * Li Zhang (lizhang@suse.de) wrote:
> > > When testing live migration with multifd channels (8, 16, or a bigger number)
> > > and using qemu -incoming (without "defer"), if a network error occurs
> > > (for example, triggering the kernel SYN flooding detection),
> > > the migration fails and the guest hangs forever.
> > > 
> > > The test environment and the command line is as the following:
> > > 
> > > QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
> > > Host OS: SLE 15  with kernel: 5.14.5-1-default
> > > Network Card: mlx5 100Gbps
> > > Network card: Intel Corporation I350 Gigabit (1Gbps)
> > > 
> > > Source:
> > > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> > >          -serial telnet:10.156.208.153:4321,server,nowait \
> > >          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> > >          -monitor stdio
> > > Dest:
> > > qemu-system-x86_64 -M q35 -smp 32 -nographic \
> > >          -serial telnet:10.156.208.154:4321,server,nowait \
> > >          -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
> > >          -monitor stdio \
> > >          -incoming tcp:1.0.8.154:4000
> > > 
> > > (qemu) migrate_set_parameter max-bandwidth 100G
> > > (qemu) migrate_set_capability multifd on
> > > (qemu) migrate_set_parameter multifd-channels 16
> > > 
> > > The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
> > > 
> > > If a network problem happens, TCP ACK is not received by destination
> > > and the destination resets the connection with RST.
> > > 
> > > No.     Time    Source  Destination     Protocol        Length  Info
> > > 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
> > > No.     Time    Source  Destination     Protocol        Length  Info
> > > 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0
> > > 
> > > kernel log:
> > > [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > > [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > > [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > > [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > > [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > > [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
> > Should we document somewhere how to avoid that?  Is there something we
> > should be doing in the connection code to avoid it?
> 
> We should use the command line -incoming defer in QEMU command line instead
> of -incoming ip:port.
> 
> And the backlog of the socket will be set as the same as  multifd channels, 
> this problem doesn't happen as far as I test.
> 
> If we use --incoming ip:port in the QEMU command line, the backlog of the
> socket is always 1, it will cause the SYN flooding.

Do we send migration parameters from the src to the dst QEMU ?

There are a bunch of things that we need to set to the same
value on the src and dst. If we sent any relevant MigrationParameters
fields to the dst, when the first/main migration chanel is opened, it
could validate that it is configured in a way that is compatible with
the src. If it isn't, it can drop the main channel immediately. This
would trigger the src to fail the migration and we couldn't get stuck
setting up the secondary data channels for multifd.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [PATCH v2 0/1] migration: multifd live migration improvement
Posted by Li Zhang 2 years, 4 months ago 
On 12/7/21 3:16 PM, Daniel P. Berrangé wrote:
> On Tue, Dec 07, 2021 at 02:45:10PM +0100, Li Zhang wrote:
>> On 12/6/21 8:54 PM, Dr. David Alan Gilbert wrote:
>>> * Li Zhang (lizhang@suse.de) wrote:
>>>> When testing live migration with multifd channels (8, 16, or a bigger number)
>>>> and using qemu -incoming (without "defer"), if a network error occurs
>>>> (for example, triggering the kernel SYN flooding detection),
>>>> the migration fails and the guest hangs forever.
>>>>
>>>> The test environment and the command line is as the following:
>>>>
>>>> QEMU verions: QEMU emulator version 6.2.91 (v6.2.0-rc1-47-gc5fbdd60cf)
>>>> Host OS: SLE 15  with kernel: 5.14.5-1-default
>>>> Network Card: mlx5 100Gbps
>>>> Network card: Intel Corporation I350 Gigabit (1Gbps)
>>>>
>>>> Source:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>>           -serial telnet:10.156.208.153:4321,server,nowait \
>>>>           -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>>           -monitor stdio
>>>> Dest:
>>>> qemu-system-x86_64 -M q35 -smp 32 -nographic \
>>>>           -serial telnet:10.156.208.154:4321,server,nowait \
>>>>           -m 4096 -enable-kvm -hda /var/lib/libvirt/images/openSUSE-15.3.img \
>>>>           -monitor stdio \
>>>>           -incoming tcp:1.0.8.154:4000
>>>>
>>>> (qemu) migrate_set_parameter max-bandwidth 100G
>>>> (qemu) migrate_set_capability multifd on
>>>> (qemu) migrate_set_parameter multifd-channels 16
>>>>
>>>> The guest hangs when executing the command: migrate -d tcp:1.0.8.154:4000.
>>>>
>>>> If a network problem happens, TCP ACK is not received by destination
>>>> and the destination resets the connection with RST.
>>>>
>>>> No.     Time    Source  Destination     Protocol        Length  Info
>>>> 119     1.021169        1.0.8.153       1.0.8.154       TCP     1410    60166 → 4000 [PSH, ACK] Seq=65 Ack=1 Win=62720 Len=1344 TSval=1338662881 TSecr=1399531897
>>>> No.     Time    Source  Destination     Protocol        Length  Info
>>>> 125     1.021181        1.0.8.154       1.0.8.153       TCP     54      4000 → 60166 [RST] Seq=1 Win=0 Len=0
>>>>
>>>> kernel log:
>>>> [334520.229445] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334562.994919] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334695.519927] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [334734.689511] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [335687.740415] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>>> [335730.013598] TCP: request_sock_TCP: Possible SYN flooding on port 4000. Sending cookies.  Check SNMP counters.
>>> Should we document somewhere how to avoid that?  Is there something we
>>> should be doing in the connection code to avoid it?
>> We should use the command line -incoming defer in QEMU command line instead
>> of -incoming ip:port.
>>
>> And the backlog of the socket will be set as the same as  multifd channels,
>> this problem doesn't happen as far as I test.
>>
>> If we use --incoming ip:port in the QEMU command line, the backlog of the
>> socket is always 1, it will cause the SYN flooding.
> Do we send migration parameters from the src to the dst QEMU ?

No, I don't think we send migration parameters from the src to the dest 
QEMU.

I set migration parameters on both sides from qemu monitor seperately.

> There are a bunch of things that we need to set to the same
> value on the src and dst. If we sent any relevant MigrationParameters
> fields to the dst, when the first/main migration chanel is opened, it
> could validate that it is configured in a way that is compatible with
> the src. If it isn't, it can drop the main channel immediately. This
> would trigger the src to fail the migration and we couldn't get stuck
> setting up the secondary data channels for multifd.

OK,  currently, we have same parameters on both sides if we set them the 
same parameters.

If we use -incoming tcp:ip:port because the multifd is disabled by 
default and backlog is 1 when the socket is created.

Here is the function which set the backlog:

static void
socket_start_incoming_migration_internal(SocketAddress *saddr,
                                          Error **errp)
{
     QIONetListener *listener = qio_net_listener_new();
     MigrationIncomingState *mis = migration_incoming_get_current();
     size_t i;
     int num = 1;

     qio_net_listener_set_name(listener, "migration-socket-listener");

     if (migrate_use_multifd()) {
         num = migrate_multifd_channels();
     }
   ...

}

The process  with -incoming tcp:ip:port is as the following:

1.   Create qemu process with command line -incoming tcp:ip:port

2.   socket_start_incoming_migration_internal  is called and backlog is: 
num=1, multifd is disabled, num = migrate_multifd_channels() is not called

3.   Enable multifd and set multifd parameters, but the backlog is still 
1, because the it couldn't be changed anymore.

4.   Run migration

The process with -incoming defer is as the following:

1. Create qemu process with command line -incoming defer

2. Enable multifd and set multifd parameters

3. Execute the command (qemu) migrate_incoming tcp:ip:port

4. Call socket_start_incoming_migration_internal then the backlog is 
set: num = migrate_multifd_channels();

5. Run migration

>
> Regards,
> Daniel

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.