[RFC PATCH 00/27] vDPA software assisted live migration

Eugenio Pérez posted 27 patches 3 days, 11 hours ago
Test checkpatch failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20201120185105.279030-1-eperezma@redhat.com
hw/virtio/vhost-sw-lm-ring.h      |  39 +++
include/hw/virtio/vhost.h         |   5 +
include/hw/virtio/virtio-access.h |   8 +-
include/hw/virtio/virtio.h        |   4 +
hw/net/virtio-net.c               |  39 ++-
hw/virtio/vhost-backend.c         |  29 ++
hw/virtio/vhost-sw-lm-ring.c      | 268 +++++++++++++++++++
hw/virtio/vhost.c                 | 431 +++++++++++++++++++++++++-----
hw/virtio/virtio.c                |  18 +-
hw/virtio/meson.build             |   2 +-
10 files changed, 758 insertions(+), 85 deletions(-)
create mode 100644 hw/virtio/vhost-sw-lm-ring.h
create mode 100644 hw/virtio/vhost-sw-lm-ring.c

[RFC PATCH 00/27] vDPA software assisted live migration

Posted by Eugenio Pérez 3 days, 11 hours ago
This series enable vDPA software assisted live migration for vhost-net
devices. This is a new method of vhost devices migration: Instead of
relay on vDPA device's dirty logging capability, SW assisted LM
intercepts dataplane, forwarding the descriptors between VM and device.

In this migration mode, qemu offers a new vring to the device to
read and write into, and disable vhost notifiers, processing guest and
vhost notifications in qemu. On used buffer relay, qemu will mark the
dirty memory as with plain virtio-net devices. This way, devices does
not need to have dirty page logging capability.

This series is a POC doing SW LM for vhost-net devices, which already
have dirty page logging capabilities. None of the changes have actual
effect with current devices until last two patches (26 and 27) are
applied, but they can be rebased on top of any other. These checks the
device to meet all requirements, and disable vhost-net devices logging
so migration goes through SW LM. This last patch is not meant to be
applied in the final revision, it is in the series just for testing
purposes.

For use SW assisted LM these vhost-net devices need to be instantiated:
* With IOMMU (iommu_platform=on,ats=on)
* Without event_idx (event_idx=off)

Just the notification forwarding (with no descriptor relay) can be
achieved with patches 7 and 9, and starting migration. Partial applies
between 13 and 24 will not work while migrating on source, and patch
25 is needed for the destination to resume network activity.

It is based on the ideas of DPDK SW assisted LM, in the series of
DPDK's https://patchwork.dpdk.org/cover/48370/ .

Comments are welcome.

Thanks!

Eugenio Pérez (27):
  vhost: Add vhost_dev_can_log
  vhost: Add device callback in vhost_migration_log
  vhost: Move log resize/put to vhost_dev_set_log
  vhost: add vhost_kernel_set_vring_enable
  vhost: Add hdev->dev.sw_lm_vq_handler
  virtio: Add virtio_queue_get_used_notify_split
  vhost: Route guest->host notification through qemu
  vhost: Add a flag for software assisted Live Migration
  vhost: Route host->guest notification through qemu
  vhost: Allocate shadow vring
  virtio: const-ify all virtio_tswap* functions
  virtio: Add virtio_queue_full
  vhost: Send buffers to device
  virtio: Remove virtio_queue_get_used_notify_split
  vhost: Do not invalidate signalled used
  virtio: Expose virtqueue_alloc_element
  vhost: add vhost_vring_set_notification_rcu
  vhost: add vhost_vring_poll_rcu
  vhost: add vhost_vring_get_buf_rcu
  vhost: Return used buffers
  vhost: Add vhost_virtqueue_memory_unmap
  vhost: Add vhost_virtqueue_memory_map
  vhost: unmap qemu's shadow virtqueues on sw live migration
  vhost: iommu changes
  vhost: Do not commit vhost used idx on vhost_virtqueue_stop
  vhost: Add vhost_hdev_can_sw_lm
  vhost: forbid vhost devices logging

 hw/virtio/vhost-sw-lm-ring.h      |  39 +++
 include/hw/virtio/vhost.h         |   5 +
 include/hw/virtio/virtio-access.h |   8 +-
 include/hw/virtio/virtio.h        |   4 +
 hw/net/virtio-net.c               |  39 ++-
 hw/virtio/vhost-backend.c         |  29 ++
 hw/virtio/vhost-sw-lm-ring.c      | 268 +++++++++++++++++++
 hw/virtio/vhost.c                 | 431 +++++++++++++++++++++++++-----
 hw/virtio/virtio.c                |  18 +-
 hw/virtio/meson.build             |   2 +-
 10 files changed, 758 insertions(+), 85 deletions(-)
 create mode 100644 hw/virtio/vhost-sw-lm-ring.h
 create mode 100644 hw/virtio/vhost-sw-lm-ring.c

-- 
2.18.4


Re: [RFC PATCH 00/27] vDPA software assisted live migration

Posted by no-reply@patchew.org 3 days, 11 hours ago
Patchew URL: https://patchew.org/QEMU/20201120185105.279030-1-eperezma@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20201120185105.279030-1-eperezma@redhat.com
Subject: [RFC PATCH 00/27] vDPA software assisted live migration

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20201117173635.29101-1-alex.bennee@linaro.org -> patchew/20201117173635.29101-1-alex.bennee@linaro.org
 * [new tag]         patchew/20201120185105.279030-1-eperezma@redhat.com -> patchew/20201120185105.279030-1-eperezma@redhat.com
Switched to a new branch 'test'
af2fe22 vhost: forbid vhost devices logging
405925c vhost: Add vhost_hdev_can_sw_lm
7f2955b vhost: Do not commit vhost used idx on vhost_virtqueue_stop
b68d3d5 vhost: iommu changes
74f282a vhost: unmap qemu's shadow virtqueues on sw live migration
c999e86 vhost: Add vhost_virtqueue_memory_map
6e5f219 vhost: Add vhost_virtqueue_memory_unmap
d5054c8 vhost: Return used buffers
806db46 vhost: add vhost_vring_get_buf_rcu
b6b8168 vhost: add vhost_vring_poll_rcu
ca44882 vhost: add vhost_vring_set_notification_rcu
3183f62 virtio: Expose virtqueue_alloc_element
4ead0ac vhost: Do not invalidate signalled used
ceb76a4 virtio: Remove virtio_queue_get_used_notify_split
6aacdfe vhost: Send buffers to device
9fcc98d virtio: Add virtio_queue_full
41da0f8 virtio: const-ify all virtio_tswap* functions
a3c92f1 vhost: Allocate shadow vring
b40b3f7 vhost: Route host->guest notification through qemu
92ea117 vhost: Add a flag for software assisted Live Migration
ace8c10 vhost: Route guest->host notification through qemu
6be11a6 virtio: Add virtio_queue_get_used_notify_split
47d4485 vhost: Add hdev->dev.sw_lm_vq_handler
7dbc8e5 vhost: add vhost_kernel_set_vring_enable
4e51c5c vhost: Move log resize/put to vhost_dev_set_log
b964abc vhost: Add device callback in vhost_migration_log
264504e vhost: Add vhost_dev_can_log

=== OUTPUT BEGIN ===
1/27 Checking commit 264504ee0018 (vhost: Add vhost_dev_can_log)
2/27 Checking commit b964abc315bd (vhost: Add device callback in vhost_migration_log)
3/27 Checking commit 4e51c5cec56d (vhost: Move log resize/put to vhost_dev_set_log)
4/27 Checking commit 7dbc8e5c64c3 (vhost: add vhost_kernel_set_vring_enable)
5/27 Checking commit 47d4485458d1 (vhost: Add hdev->dev.sw_lm_vq_handler)
6/27 Checking commit 6be11a63e3b4 (virtio: Add virtio_queue_get_used_notify_split)
7/27 Checking commit ace8c1034f83 (vhost: Route guest->host notification through qemu)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#24: 
new file mode 100644

total: 0 errors, 1 warnings, 245 lines checked

Patch 7/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
8/27 Checking commit 92ea11700e32 (vhost: Add a flag for software assisted Live Migration)
WARNING: Block comments use a leading /* on a separate line
#46: FILE: hw/virtio/vhost.c:1581:
+        /* We've been called after migration is completed, so no need to

WARNING: Block comments use * on subsequent lines
#47: FILE: hw/virtio/vhost.c:1582:
+        /* We've been called after migration is completed, so no need to
+           disable it again

total: 0 errors, 2 warnings, 45 lines checked

Patch 8/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
9/27 Checking commit b40b3f79355c (vhost: Route host->guest notification through qemu)
10/27 Checking commit a3c92f15b554 (vhost: Allocate shadow vring)
11/27 Checking commit 41da0f8e02b2 (virtio: const-ify all virtio_tswap* functions)
12/27 Checking commit 9fcc98da9cd9 (virtio: Add virtio_queue_full)
13/27 Checking commit 6aacdfe0e0ba (vhost: Send buffers to device)
ERROR: memory barrier without comment
#50: FILE: hw/virtio/vhost-sw-lm-ring.c:45:
+    smp_rmb();

WARNING: Block comments use a leading /* on a separate line
#98: FILE: hw/virtio/vhost-sw-lm-ring.c:93:
+/* virtqueue_add:

WARNING: Block comments use a leading /* on a separate line
#125: FILE: hw/virtio/vhost-sw-lm-ring.c:120:
+    /* Put entry in available array (but don't update avail->idx until they

WARNING: Block comments use a trailing */ on a separate line
#126: FILE: hw/virtio/vhost-sw-lm-ring.c:121:
+     * do sync). */

ERROR: g_free(NULL) is safe this check is probably not required
#147: FILE: hw/virtio/vhost-sw-lm-ring.c:142:
+    if (vq->ring_id_maps[host_head]) {
+        g_free(vq->ring_id_maps[host_head]);

ERROR: braces {} are necessary for all arms of this statement
#185: FILE: hw/virtio/vhost-sw-lm-ring.c:181:
+    for (i = 0; i < num - 1; i++)
[...]

ERROR: code indent should never use tabs
#207: FILE: hw/virtio/vhost-sw-lm-ring.h:23:
+^I                    struct vhost_vring_addr *addr);$

ERROR: space required before the open parenthesis '('
#247: FILE: hw/virtio/vhost.c:986:
+    } while(!virtio_queue_empty(vq));

total: 5 errors, 3 warnings, 275 lines checked

Patch 13/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

14/27 Checking commit ceb76a4401b8 (virtio: Remove virtio_queue_get_used_notify_split)
15/27 Checking commit 4ead0ac8457f (vhost: Do not invalidate signalled used)
16/27 Checking commit 3183f62db3dc (virtio: Expose virtqueue_alloc_element)
17/27 Checking commit ca44882af152 (vhost: add vhost_vring_set_notification_rcu)
ERROR: memory barrier without comment
#45: FILE: hw/virtio/vhost-sw-lm-ring.c:83:
+    smp_mb();

total: 1 errors, 0 warnings, 45 lines checked

Patch 17/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

18/27 Checking commit b6b8168b9fe7 (vhost: add vhost_vring_poll_rcu)
ERROR: memory barrier without comment
#37: FILE: hw/virtio/vhost-sw-lm-ring.c:98:
+    smp_rmb();

total: 1 errors, 0 warnings, 38 lines checked

Patch 18/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

19/27 Checking commit 806db46e9194 (vhost: add vhost_vring_get_buf_rcu)
20/27 Checking commit d5054c8556bf (vhost: Return used buffers)
21/27 Checking commit 6e5f2192254a (vhost: Add vhost_virtqueue_memory_unmap)
22/27 Checking commit c999e86cf7f0 (vhost: Add vhost_virtqueue_memory_map)
23/27 Checking commit 74f282a80019 (vhost: unmap qemu's shadow virtqueues on sw live migration)
24/27 Checking commit b68d3d5fb839 (vhost: iommu changes)
25/27 Checking commit 7f2955b8e788 (vhost: Do not commit vhost used idx on vhost_virtqueue_stop)
WARNING: Block comments use a leading /* on a separate line
#40: FILE: hw/virtio/vhost.c:1442:
+        /* Connection to the backend is unusable, so let's sync internal

total: 0 errors, 1 warnings, 27 lines checked

Patch 25/27 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
26/27 Checking commit 405925c6b54a (vhost: Add vhost_hdev_can_sw_lm)
27/27 Checking commit af2fe2219d39 (vhost: forbid vhost devices logging)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20201120185105.279030-1-eperezma@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [RFC PATCH 00/27] vDPA software assisted live migration

Posted by Eugenio Perez Martin 3 days, 11 hours ago
The main intention with this POC/RFC is to serve as a base to
implement SW LM in vdpa devices.

To implement in vhost-vdpa devices, the high priority is to achieve an
interface for vdpa to stop the device without losing its state, i.e.,
the avail_idx the destination device should start fetching descriptors
from. Following this POC, an implementation on vdpa_sim will be
performed.

Apart from that, there is a TODO list about this series, they will be
solved as the code is marked as valid. They don't affect the device,
just internal qemu's code, and in case of change of direction it is
easy to modify or delete. Comments about these are welcome.

- Currently, it hijacks the log mechanism to know when migration is
starting/done. Maybe it would be cleaner to forward migrate status
from virtio_vmstate_change, since there is no need for the memory
listener. However, this could make "memory backend" abstraction (also
TODO) more complicated. This would drop patches 2,3 entirely.
- There is a reverse search in a list on "vhost_dev_from_virtio" for
each notification. Not really efficient, and it leads to a race
condition at device destruction.
- Implement new capabilities (no iommu, packed vq, event_idx, ...)
- Lot of assertions need to be converted to proper error handling.

Thanks!

On Fri, Nov 20, 2020 at 8:02 PM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> This series enable vDPA software assisted live migration for vhost-net
> devices. This is a new method of vhost devices migration: Instead of
> relay on vDPA device's dirty logging capability, SW assisted LM
> intercepts dataplane, forwarding the descriptors between VM and device.
>
> In this migration mode, qemu offers a new vring to the device to
> read and write into, and disable vhost notifiers, processing guest and
> vhost notifications in qemu. On used buffer relay, qemu will mark the
> dirty memory as with plain virtio-net devices. This way, devices does
> not need to have dirty page logging capability.
>
> This series is a POC doing SW LM for vhost-net devices, which already
> have dirty page logging capabilities. None of the changes have actual
> effect with current devices until last two patches (26 and 27) are
> applied, but they can be rebased on top of any other. These checks the
> device to meet all requirements, and disable vhost-net devices logging
> so migration goes through SW LM. This last patch is not meant to be
> applied in the final revision, it is in the series just for testing
> purposes.
>
> For use SW assisted LM these vhost-net devices need to be instantiated:
> * With IOMMU (iommu_platform=on,ats=on)
> * Without event_idx (event_idx=off)
>
> Just the notification forwarding (with no descriptor relay) can be
> achieved with patches 7 and 9, and starting migration. Partial applies
> between 13 and 24 will not work while migrating on source, and patch
> 25 is needed for the destination to resume network activity.
>
> It is based on the ideas of DPDK SW assisted LM, in the series of
> DPDK's https://patchwork.dpdk.org/cover/48370/ .
>
> Comments are welcome.
>
> Thanks!
>
> Eugenio Pérez (27):
>   vhost: Add vhost_dev_can_log
>   vhost: Add device callback in vhost_migration_log
>   vhost: Move log resize/put to vhost_dev_set_log
>   vhost: add vhost_kernel_set_vring_enable
>   vhost: Add hdev->dev.sw_lm_vq_handler
>   virtio: Add virtio_queue_get_used_notify_split
>   vhost: Route guest->host notification through qemu
>   vhost: Add a flag for software assisted Live Migration
>   vhost: Route host->guest notification through qemu
>   vhost: Allocate shadow vring
>   virtio: const-ify all virtio_tswap* functions
>   virtio: Add virtio_queue_full
>   vhost: Send buffers to device
>   virtio: Remove virtio_queue_get_used_notify_split
>   vhost: Do not invalidate signalled used
>   virtio: Expose virtqueue_alloc_element
>   vhost: add vhost_vring_set_notification_rcu
>   vhost: add vhost_vring_poll_rcu
>   vhost: add vhost_vring_get_buf_rcu
>   vhost: Return used buffers
>   vhost: Add vhost_virtqueue_memory_unmap
>   vhost: Add vhost_virtqueue_memory_map
>   vhost: unmap qemu's shadow virtqueues on sw live migration
>   vhost: iommu changes
>   vhost: Do not commit vhost used idx on vhost_virtqueue_stop
>   vhost: Add vhost_hdev_can_sw_lm
>   vhost: forbid vhost devices logging
>
>  hw/virtio/vhost-sw-lm-ring.h      |  39 +++
>  include/hw/virtio/vhost.h         |   5 +
>  include/hw/virtio/virtio-access.h |   8 +-
>  include/hw/virtio/virtio.h        |   4 +
>  hw/net/virtio-net.c               |  39 ++-
>  hw/virtio/vhost-backend.c         |  29 ++
>  hw/virtio/vhost-sw-lm-ring.c      | 268 +++++++++++++++++++
>  hw/virtio/vhost.c                 | 431 +++++++++++++++++++++++++-----
>  hw/virtio/virtio.c                |  18 +-
>  hw/virtio/meson.build             |   2 +-
>  10 files changed, 758 insertions(+), 85 deletions(-)
>  create mode 100644 hw/virtio/vhost-sw-lm-ring.h
>  create mode 100644 hw/virtio/vhost-sw-lm-ring.c
>
> --
> 2.18.4
>
>