This version of the live update patch series integrates live update into the
live migration framework.  The new interfaces are:
  * mode (migration parameter)
  * cpr-exec-args (migration parameter)
  * file (migration URI)
  * migrate-mode-enable (command-line argument)
  * only-cpr-capable (command-line argument)

Provide the cpr-exec and cpr-reboot migration modes for live update.  These
save and restore VM state, with minimal guest pause time, so that qemu may be
updated to a new version in between.  The caller sets the mode parameter
before invoking the migrate or migrate-incoming commands.

In cpr-reboot mode, the migrate command saves state to a file, allowing
one to quit qemu, reboot to an updated kernel, start an updated version of
qemu, and resume via the migrate-incoming command.  The caller must specify
a migration URI that writes to and reads from a file.  Unlike normal mode,
the use of certain local storage options does not block the migration, but
the caller must not modify guest block devices between the quit and restart.
The guest RAM memory-backend must be shared, and the @x-ignore-shared
migration capability must be set, to avoid saving it to the file.  Guest RAM
must be non-volatile across reboot, which can be achieved by backing it with
a dax device, or /dev/shm PKRAM as proposed in
https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com
but this is not enforced.  The restarted qemu arguments must match those used
to initially start qemu, plus the -incoming option.

The reboot mode supports vfio devices if the caller first suspends the guest,
such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
drivers' suspend methods flush outstanding requests and re-initialize the
devices, and thus there is no device state to save and restore.  After
issuing migrate-incoming, the caller must issue a system_wakeup command to
resume.

In cpr-exec mode, the migrate command saves state to a file and directly
exec's a new version of qemu on the same host, replacing the original process
while retaining its PID.  The caller must specify a migration URI that writes
to and reads from a file, and resumes execution via the migrate-incoming
command.  Arguments for the new qemu process are taken from the cpr-exec-args
migration parameter, and must include the -incoming option.

Guest RAM must be backed by a memory backend with share=on, but cannot be
memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
ram is efficiently preserved in place, albeit with new virtual addresses.
In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
causes secondary guest ram blocks (those not specified on the command line)
to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
their values are saved in special cpr state which is retrieved after exec,
and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
are not migrated, the caller must disable all capabilities related to page
and block copy.  The implementation ignores all related parameters.

The exec mode supports vfio devices by preserving the vfio container, group,
device, and event descriptors across the qemu re-exec, and by updating DMA
mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
VFIO_DMA_MAP_FLAG_VADDR as defined in
  https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com
and integrated in Linux kernel 5.12.

Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
The software update is performed while the guest is running to minimize
downtime.

window 1                                        | window 2
                                                |
# qemu-system-$arch ...                         |
  -migrate-mode-enable cpr-exec                 |
QEMU 7.0.50 monitor - type 'help' ...           |
(qemu) info status                              |
VM status: running                              |
                                                | # yum update qemu
(qemu) migrate_set_parameter mode cpr-exec      |
(qemu) migrate_set_parameter cpr-exec-args      |
  qemu-system-$arch ... -incoming defer         |
(qemu) migrate -d file:/tmp/qemu.sav            |
QEMU 7.0.51 monitor - type 'help' ...           |
(qemu) info status                              |
VM status: paused (inmigrate)                   |
(qemu) migrate_incoming file:/tmp/qemu.sav      |
(qemu) info status                              |
VM status: running                              |


Here is an example of updating the host kernel using reboot mode.

window 1                                        | window 2
                                                |
# qemu-system-$arch ... mem-path=/dev/dax0.0    |
  -migrate-mode-enable cpr-reboot               |
QEMU 7.0.50 monitor - type 'help' ...           |
(qemu) info status                              |
VM status: running                              |
                                                | # yum update kernel-uek
(qemu) migrate_set_parameter mode cpr-reboot    |
(qemu) migrate -d file:/tmp/qemu.sav            |
(qemu) quit                                     |
                                                |
# systemctl kexec                               |
kexec_core: Starting new kernel                 |
...                                             |
                                                |
# qemu-system-$arch mem-path=/dev/dax0.0 ...    |
  -incoming defer                               |
QEMU 7.0.51 monitor - type 'help' ...           |
(qemu) info status                              |
VM status: paused (inmigrate)                   |
(qemu) migrate_incoming file:/tmp/qemu.sav      |
(qemu) info status                              |
VM status: running                              |

Changes from V8 to V9:
  vfio:
    - free all cpr state during unwind in vfio_connect_container
    - change cpr_resave_fd to return void, and avoid new unwind cases
    - delete incorrect .unmigratable=1 in vmstate handlers
    - add route batching in vfio_claim_vectors
    - simplified vfio intx cpr code
    - fix commit message for 'recover from unmap-all-vaddr failure'
    - verify suspended runstate for cpr-reboot mode
  Other:
    - delete cpr-save, cpr-exec, cpr-load
    - delete ram block vmstate handlers that were added in V8
    - rename cpr-enable option to migrate-mode-enable
    - add file URI for migration
    - add mode and cpr-exec-args migration parameters
    - add per-mode migration blockers
    - add mode checks in migration notifiers
    - fix suspended runstate during migration
    - replace RAM_ANON flag with RAM_NAMED_FILE
    - support memory-backend-epc

Steve Sistare (44):
  migration: fix populate_vfio_info                  ---  reboot mode  ---
  memory: RAM_NAMED_FILE flag
  migration: file URI
  migration: mode parameter
  migration: migrate-enable-mode option
  migration: simplify blockers
  migration: per-mode blockers
  cpr: relax some blockers
  cpr: reboot mode

  qdev-properties: strList                           ---  exec mode ---
  qapi: strList_from_string
  qapi: QAPI_LIST_LENGTH
  qapi: strv_from_strList
  qapi: strList unit tests
  migration: cpr-exec-args parameter
  migration: simplify notifiers
  migration: check mode in notifiers
  memory: flat section iterator
  oslib: qemu_clear_cloexec
  vl: helper to request re-exec
  cpr: preserve extra state
  cpr: exec mode
  cpr: add exec-mode blockers
  cpr: ram block blockers
  cpr: only-cpr-capable
  cpr: Mismatched GPAs fix
  hostmem-memfd: cpr support
  hostmem-epc: cpr support

  pci: export msix_is_pending                       --- vfio for exec ---
  vfio-pci: refactor for cpr
  vfio-pci: cpr part 1 (fd and dma)
  vfio-pci: cpr part 2 (msi)
  vfio-pci: cpr part 3 (intx)
  vfio-pci: recover from unmap-all-vaddr failure

  chardev: cpr framework                            --- misc for exec ---
  chardev: cpr for simple devices
  chardev: cpr for pty
  python/machine: QEMUMachine full_args
  python/machine: QEMUMachine reopen_qmp_connection
  tests/avocado: add cpr regression test

  vl: start on wakeup request                       --- vfio for reboot ---
  migration: fix suspended runstate
  migration: notifier error reporting
  vfio: allow cpr-reboot migration if suspended

Mark Kanda, Steve Sistare (2):
  vhost: reset vhost devices for cpr
  chardev: cpr for sockets

 MAINTAINERS                         |  14 ++
 accel/xen/xen-all.c                 |   3 +
 backends/hostmem-epc.c              |  18 +-
 backends/hostmem-file.c             |   1 +
 backends/hostmem-memfd.c            |  22 ++-
 backends/tpm/tpm_emulator.c         |  11 +-
 block/parallels.c                   |   7 +-
 block/qcow.c                        |   7 +-
 block/vdi.c                         |   7 +-
 block/vhdx.c                        |   7 +-
 block/vmdk.c                        |   7 +-
 block/vpc.c                         |   7 +-
 block/vvfat.c                       |   7 +-
 chardev/char-mux.c                  |   1 +
 chardev/char-null.c                 |   1 +
 chardev/char-pty.c                  |  16 +-
 chardev/char-serial.c               |   1 +
 chardev/char-socket.c               |  48 +++++
 chardev/char-stdio.c                |  31 +++
 chardev/char.c                      |  49 ++++-
 dump/dump.c                         |   4 +-
 gdbstub.c                           |   1 +
 hmp-commands.hx                     |   2 +-
 hw/9pfs/9p.c                        |  11 +-
 hw/core/qdev-properties-system.c    |  12 ++
 hw/core/qdev-properties.c           |  44 +++++
 hw/display/virtio-gpu-base.c        |   8 +-
 hw/intc/arm_gic_kvm.c               |   3 +-
 hw/intc/arm_gicv3_its_kvm.c         |   3 +-
 hw/intc/arm_gicv3_kvm.c             |   3 +-
 hw/misc/ivshmem.c                   |   8 +-
 hw/net/virtio-net.c                 |  10 +-
 hw/pci/msix.c                       |   2 +-
 hw/pci/pci.c                        |  12 ++
 hw/ppc/pef.c                        |   2 +-
 hw/ppc/spapr.c                      |   2 +-
 hw/ppc/spapr_events.c               |   2 +-
 hw/ppc/spapr_rtas.c                 |   2 +-
 hw/remote/proxy.c                   |   7 +-
 hw/s390x/s390-virtio-ccw.c          |   9 +-
 hw/scsi/vhost-scsi.c                |   9 +-
 hw/vfio/common.c                    | 235 +++++++++++++++++++----
 hw/vfio/cpr.c                       | 177 ++++++++++++++++++
 hw/vfio/meson.build                 |   1 +
 hw/vfio/migration.c                 |  23 +--
 hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
 hw/vfio/trace-events                |   1 +
 hw/virtio/vhost-vdpa.c              |   6 +-
 hw/virtio/vhost.c                   |  32 +++-
 include/chardev/char-socket.h       |   1 +
 include/chardev/char.h              |   5 +
 include/exec/memory.h               |  48 +++++
 include/exec/ram_addr.h             |   1 +
 include/exec/ramblock.h             |   1 +
 include/hw/pci/msix.h               |   1 +
 include/hw/qdev-properties-system.h |   4 +
 include/hw/qdev-properties.h        |   3 +
 include/hw/vfio/vfio-common.h       |  12 ++
 include/hw/virtio/vhost.h           |   1 +
 include/migration/blocker.h         |  69 ++++++-
 include/migration/cpr-state.h       |  30 +++
 include/migration/cpr.h             |  20 ++
 include/migration/misc.h            |  13 +-
 include/migration/vmstate.h         |   2 +
 include/qapi/util.h                 |  28 +++
 include/qemu/osdep.h                |   9 +
 include/sysemu/runstate.h           |   2 +
 migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
 migration/cpr.c                     |  85 +++++++++
 migration/file.c                    |  62 ++++++
 migration/file.h                    |  14 ++
 migration/meson.build               |   3 +
 migration/migration.c               | 268 +++++++++++++++++++++++---
 migration/ram.c                     |  24 ++-
 migration/target.c                  |   1 +
 migration/trace-events              |  12 ++
 monitor/hmp-cmds.c                  |  59 +++---
 monitor/hmp.c                       |   3 +
 monitor/qmp.c                       |   4 +
 python/qemu/machine/machine.py      |  14 ++
 qapi/char.json                      |   7 +-
 qapi/migration.json                 |  68 ++++++-
 qapi/qapi-util.c                    |  37 ++++
 qemu-options.hx                     |  50 ++++-
 replay/replay.c                     |   4 +
 softmmu/memory.c                    |  31 ++-
 softmmu/physmem.c                   | 100 +++++++++-
 softmmu/runstate.c                  |  42 ++++-
 softmmu/vl.c                        |  10 +
 stubs/cpr-state.c                   |  26 +++
 stubs/meson.build                   |   2 +
 stubs/migr-blocker.c                |   9 +-
 stubs/migration.c                   |  33 ++++
 target/i386/kvm/kvm.c               |   8 +-
 target/i386/nvmm/nvmm-all.c         |   4 +-
 target/i386/sev.c                   |   2 +-
 target/i386/whpx/whpx-all.c         |   3 +-
 tests/avocado/cpr.py                | 176 ++++++++++++++++++
 tests/unit/meson.build              |   1 +
 tests/unit/test-strlist.c           |  81 ++++++++
 trace-events                        |   1 +
 ui/spice-core.c                     |   5 +-
 ui/vdagent.c                        |   5 +-
 util/oslib-posix.c                  |   9 +
 util/oslib-win32.c                  |   4 +
 105 files changed, 2781 insertions(+), 330 deletions(-)
 create mode 100644 hw/vfio/cpr.c
 create mode 100644 include/migration/cpr-state.h
 create mode 100644 include/migration/cpr.h
 create mode 100644 migration/cpr-state.c
 create mode 100644 migration/cpr.c
 create mode 100644 migration/file.c
 create mode 100644 migration/file.h
 create mode 100644 stubs/cpr-state.c
 create mode 100644 stubs/migration.c
 create mode 100644 tests/avocado/cpr.py
 create mode 100644 tests/unit/test-strlist.c

-- 
1.8.3.1

This series desperately needs review in its intersection with live migration.
The code in other areas has been reviewed and revised multiple times -- thank you!

David, Juan, can you spare some time to review this?  I have done my best to order 
the patches logically (see the labelled groups in this email), and to provide 
complete and clear cover letter and commit messages. Can I do anything to facilitate,
like doing a code walk through via zoom?

And of course, I welcome anyone's feedback.

Here is the original posting.

https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/

- Steve

On 7/26/2022 12:09 PM, Steve Sistare wrote:
> This version of the live update patch series integrates live update into the
> live migration framework.  The new interfaces are:
>   * mode (migration parameter)
>   * cpr-exec-args (migration parameter)
>   * file (migration URI)
>   * migrate-mode-enable (command-line argument)
>   * only-cpr-capable (command-line argument)
> 
> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
> save and restore VM state, with minimal guest pause time, so that qemu may be
> updated to a new version in between.  The caller sets the mode parameter
> before invoking the migrate or migrate-incoming commands.
> 
> In cpr-reboot mode, the migrate command saves state to a file, allowing
> one to quit qemu, reboot to an updated kernel, start an updated version of
> qemu, and resume via the migrate-incoming command.  The caller must specify
> a migration URI that writes to and reads from a file.  Unlike normal mode,
> the use of certain local storage options does not block the migration, but
> the caller must not modify guest block devices between the quit and restart.
> The guest RAM memory-backend must be shared, and the @x-ignore-shared
> migration capability must be set, to avoid saving it to the file.  Guest RAM
> must be non-volatile across reboot, which can be achieved by backing it with
> a dax device, or /dev/shm PKRAM as proposed in
> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com
> but this is not enforced.  The restarted qemu arguments must match those used
> to initially start qemu, plus the -incoming option.
> 
> The reboot mode supports vfio devices if the caller first suspends the guest,
> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
> drivers' suspend methods flush outstanding requests and re-initialize the
> devices, and thus there is no device state to save and restore.  After
> issuing migrate-incoming, the caller must issue a system_wakeup command to
> resume.
> 
> In cpr-exec mode, the migrate command saves state to a file and directly
> exec's a new version of qemu on the same host, replacing the original process
> while retaining its PID.  The caller must specify a migration URI that writes
> to and reads from a file, and resumes execution via the migrate-incoming
> command.  Arguments for the new qemu process are taken from the cpr-exec-args
> migration parameter, and must include the -incoming option.
> 
> Guest RAM must be backed by a memory backend with share=on, but cannot be
> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
> ram is efficiently preserved in place, albeit with new virtual addresses.
> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
> causes secondary guest ram blocks (those not specified on the command line)
> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
> their values are saved in special cpr state which is retrieved after exec,
> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
> are not migrated, the caller must disable all capabilities related to page
> and block copy.  The implementation ignores all related parameters.
> 
> The exec mode supports vfio devices by preserving the vfio container, group,
> device, and event descriptors across the qemu re-exec, and by updating DMA
> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
> VFIO_DMA_MAP_FLAG_VADDR as defined in
>   https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com
> and integrated in Linux kernel 5.12.
> 
> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
> The software update is performed while the guest is running to minimize
> downtime.
> 
> window 1                                        | window 2
>                                                 |
> # qemu-system-$arch ...                         |
>   -migrate-mode-enable cpr-exec                 |
> QEMU 7.0.50 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: running                              |
>                                                 | # yum update qemu
> (qemu) migrate_set_parameter mode cpr-exec      |
> (qemu) migrate_set_parameter cpr-exec-args      |
>   qemu-system-$arch ... -incoming defer         |
> (qemu) migrate -d file:/tmp/qemu.sav            |
> QEMU 7.0.51 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: paused (inmigrate)                   |
> (qemu) migrate_incoming file:/tmp/qemu.sav      |
> (qemu) info status                              |
> VM status: running                              |
> 
> 
> Here is an example of updating the host kernel using reboot mode.
> 
> window 1                                        | window 2
>                                                 |
> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>   -migrate-mode-enable cpr-reboot               |
> QEMU 7.0.50 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: running                              |
>                                                 | # yum update kernel-uek
> (qemu) migrate_set_parameter mode cpr-reboot    |
> (qemu) migrate -d file:/tmp/qemu.sav            |
> (qemu) quit                                     |
>                                                 |
> # systemctl kexec                               |
> kexec_core: Starting new kernel                 |
> ...                                             |
>                                                 |
> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>   -incoming defer                               |
> QEMU 7.0.51 monitor - type 'help' ...           |
> (qemu) info status                              |
> VM status: paused (inmigrate)                   |
> (qemu) migrate_incoming file:/tmp/qemu.sav      |
> (qemu) info status                              |
> VM status: running                              |
> 
> Changes from V8 to V9:
>   vfio:
>     - free all cpr state during unwind in vfio_connect_container
>     - change cpr_resave_fd to return void, and avoid new unwind cases
>     - delete incorrect .unmigratable=1 in vmstate handlers
>     - add route batching in vfio_claim_vectors
>     - simplified vfio intx cpr code
>     - fix commit message for 'recover from unmap-all-vaddr failure'
>     - verify suspended runstate for cpr-reboot mode
>   Other:
>     - delete cpr-save, cpr-exec, cpr-load
>     - delete ram block vmstate handlers that were added in V8
>     - rename cpr-enable option to migrate-mode-enable
>     - add file URI for migration
>     - add mode and cpr-exec-args migration parameters
>     - add per-mode migration blockers
>     - add mode checks in migration notifiers
>     - fix suspended runstate during migration
>     - replace RAM_ANON flag with RAM_NAMED_FILE
>     - support memory-backend-epc
> 
> Steve Sistare (44):
>   migration: fix populate_vfio_info                  ---  reboot mode  ---
>   memory: RAM_NAMED_FILE flag
>   migration: file URI
>   migration: mode parameter
>   migration: migrate-enable-mode option
>   migration: simplify blockers
>   migration: per-mode blockers
>   cpr: relax some blockers
>   cpr: reboot mode
> 
>   qdev-properties: strList                           ---  exec mode ---
>   qapi: strList_from_string
>   qapi: QAPI_LIST_LENGTH
>   qapi: strv_from_strList
>   qapi: strList unit tests
>   migration: cpr-exec-args parameter
>   migration: simplify notifiers
>   migration: check mode in notifiers
>   memory: flat section iterator
>   oslib: qemu_clear_cloexec
>   vl: helper to request re-exec
>   cpr: preserve extra state
>   cpr: exec mode
>   cpr: add exec-mode blockers
>   cpr: ram block blockers
>   cpr: only-cpr-capable
>   cpr: Mismatched GPAs fix
>   hostmem-memfd: cpr support
>   hostmem-epc: cpr support
> 
>   pci: export msix_is_pending                       --- vfio for exec ---
>   vfio-pci: refactor for cpr
>   vfio-pci: cpr part 1 (fd and dma)
>   vfio-pci: cpr part 2 (msi)
>   vfio-pci: cpr part 3 (intx)
>   vfio-pci: recover from unmap-all-vaddr failure
> 
>   chardev: cpr framework                            --- misc for exec ---
>   chardev: cpr for simple devices
>   chardev: cpr for pty
>   python/machine: QEMUMachine full_args
>   python/machine: QEMUMachine reopen_qmp_connection
>   tests/avocado: add cpr regression test
> 
>   vl: start on wakeup request                       --- vfio for reboot ---
>   migration: fix suspended runstate
>   migration: notifier error reporting
>   vfio: allow cpr-reboot migration if suspended
> 
> Mark Kanda, Steve Sistare (2):
>   vhost: reset vhost devices for cpr
>   chardev: cpr for sockets
> 
>  MAINTAINERS                         |  14 ++
>  accel/xen/xen-all.c                 |   3 +
>  backends/hostmem-epc.c              |  18 +-
>  backends/hostmem-file.c             |   1 +
>  backends/hostmem-memfd.c            |  22 ++-
>  backends/tpm/tpm_emulator.c         |  11 +-
>  block/parallels.c                   |   7 +-
>  block/qcow.c                        |   7 +-
>  block/vdi.c                         |   7 +-
>  block/vhdx.c                        |   7 +-
>  block/vmdk.c                        |   7 +-
>  block/vpc.c                         |   7 +-
>  block/vvfat.c                       |   7 +-
>  chardev/char-mux.c                  |   1 +
>  chardev/char-null.c                 |   1 +
>  chardev/char-pty.c                  |  16 +-
>  chardev/char-serial.c               |   1 +
>  chardev/char-socket.c               |  48 +++++
>  chardev/char-stdio.c                |  31 +++
>  chardev/char.c                      |  49 ++++-
>  dump/dump.c                         |   4 +-
>  gdbstub.c                           |   1 +
>  hmp-commands.hx                     |   2 +-
>  hw/9pfs/9p.c                        |  11 +-
>  hw/core/qdev-properties-system.c    |  12 ++
>  hw/core/qdev-properties.c           |  44 +++++
>  hw/display/virtio-gpu-base.c        |   8 +-
>  hw/intc/arm_gic_kvm.c               |   3 +-
>  hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>  hw/intc/arm_gicv3_kvm.c             |   3 +-
>  hw/misc/ivshmem.c                   |   8 +-
>  hw/net/virtio-net.c                 |  10 +-
>  hw/pci/msix.c                       |   2 +-
>  hw/pci/pci.c                        |  12 ++
>  hw/ppc/pef.c                        |   2 +-
>  hw/ppc/spapr.c                      |   2 +-
>  hw/ppc/spapr_events.c               |   2 +-
>  hw/ppc/spapr_rtas.c                 |   2 +-
>  hw/remote/proxy.c                   |   7 +-
>  hw/s390x/s390-virtio-ccw.c          |   9 +-
>  hw/scsi/vhost-scsi.c                |   9 +-
>  hw/vfio/common.c                    | 235 +++++++++++++++++++----
>  hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>  hw/vfio/meson.build                 |   1 +
>  hw/vfio/migration.c                 |  23 +--
>  hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>  hw/vfio/trace-events                |   1 +
>  hw/virtio/vhost-vdpa.c              |   6 +-
>  hw/virtio/vhost.c                   |  32 +++-
>  include/chardev/char-socket.h       |   1 +
>  include/chardev/char.h              |   5 +
>  include/exec/memory.h               |  48 +++++
>  include/exec/ram_addr.h             |   1 +
>  include/exec/ramblock.h             |   1 +
>  include/hw/pci/msix.h               |   1 +
>  include/hw/qdev-properties-system.h |   4 +
>  include/hw/qdev-properties.h        |   3 +
>  include/hw/vfio/vfio-common.h       |  12 ++
>  include/hw/virtio/vhost.h           |   1 +
>  include/migration/blocker.h         |  69 ++++++-
>  include/migration/cpr-state.h       |  30 +++
>  include/migration/cpr.h             |  20 ++
>  include/migration/misc.h            |  13 +-
>  include/migration/vmstate.h         |   2 +
>  include/qapi/util.h                 |  28 +++
>  include/qemu/osdep.h                |   9 +
>  include/sysemu/runstate.h           |   2 +
>  migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>  migration/cpr.c                     |  85 +++++++++
>  migration/file.c                    |  62 ++++++
>  migration/file.h                    |  14 ++
>  migration/meson.build               |   3 +
>  migration/migration.c               | 268 +++++++++++++++++++++++---
>  migration/ram.c                     |  24 ++-
>  migration/target.c                  |   1 +
>  migration/trace-events              |  12 ++
>  monitor/hmp-cmds.c                  |  59 +++---
>  monitor/hmp.c                       |   3 +
>  monitor/qmp.c                       |   4 +
>  python/qemu/machine/machine.py      |  14 ++
>  qapi/char.json                      |   7 +-
>  qapi/migration.json                 |  68 ++++++-
>  qapi/qapi-util.c                    |  37 ++++
>  qemu-options.hx                     |  50 ++++-
>  replay/replay.c                     |   4 +
>  softmmu/memory.c                    |  31 ++-
>  softmmu/physmem.c                   | 100 +++++++++-
>  softmmu/runstate.c                  |  42 ++++-
>  softmmu/vl.c                        |  10 +
>  stubs/cpr-state.c                   |  26 +++
>  stubs/meson.build                   |   2 +
>  stubs/migr-blocker.c                |   9 +-
>  stubs/migration.c                   |  33 ++++
>  target/i386/kvm/kvm.c               |   8 +-
>  target/i386/nvmm/nvmm-all.c         |   4 +-
>  target/i386/sev.c                   |   2 +-
>  target/i386/whpx/whpx-all.c         |   3 +-
>  tests/avocado/cpr.py                | 176 ++++++++++++++++++
>  tests/unit/meson.build              |   1 +
>  tests/unit/test-strlist.c           |  81 ++++++++
>  trace-events                        |   1 +
>  ui/spice-core.c                     |   5 +-
>  ui/vdagent.c                        |   5 +-
>  util/oslib-posix.c                  |   9 +
>  util/oslib-win32.c                  |   4 +
>  105 files changed, 2781 insertions(+), 330 deletions(-)
>  create mode 100644 hw/vfio/cpr.c
>  create mode 100644 include/migration/cpr-state.h
>  create mode 100644 include/migration/cpr.h
>  create mode 100644 migration/cpr-state.c
>  create mode 100644 migration/cpr.c
>  create mode 100644 migration/file.c
>  create mode 100644 migration/file.h
>  create mode 100644 stubs/cpr-state.c
>  create mode 100644 stubs/migration.c
>  create mode 100644 tests/avocado/cpr.py
>  create mode 100644 tests/unit/test-strlist.c
>

Hi Steve,

In the current design you have, we have to specify both the command line 
parameter "-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".

Is it possible to opt-in to the CPR mode just once over the monitor 
instead of having to specify it twice on the command line?
This would also match the live migration model: You do not need to 
necessarily "opt in" to live migration mode through
a command line parameter, you simply request it when you need to. Can 
CPR behave the same way?

This would also make switching over to a CPR-capable version of QEMU 
much simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would 
no longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and 
then work on a libvirt patch later when
we're ready to do so.


Comments?

- Michael


On 12/7/22 09:48, Steven Sistare wrote:
> This series desperately needs review in its intersection with live migration.
> The code in other areas has been reviewed and revised multiple times -- thank you!
>
> David, Juan, can you spare some time to review this?  I have done my best to order
> the patches logically (see the labelled groups in this email), and to provide
> complete and clear cover letter and commit messages. Can I do anything to facilitate,
> like doing a code walk through via zoom?
>
> And of course, I welcome anyone's feedback.
>
> Here is the original posting.
>
> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$  
>
> - Steve
>
> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>> This version of the live update patch series integrates live update into the
>> live migration framework.  The new interfaces are:
>>    * mode (migration parameter)
>>    * cpr-exec-args (migration parameter)
>>    * file (migration URI)
>>    * migrate-mode-enable (command-line argument)
>>    * only-cpr-capable (command-line argument)
>>
>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>> save and restore VM state, with minimal guest pause time, so that qemu may be
>> updated to a new version in between.  The caller sets the mode parameter
>> before invoking the migrate or migrate-incoming commands.
>>
>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>> one to quit qemu, reboot to an updated kernel, start an updated version of
>> qemu, and resume via the migrate-incoming command.  The caller must specify
>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>> the use of certain local storage options does not block the migration, but
>> the caller must not modify guest block devices between the quit and restart.
>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>> must be non-volatile across reboot, which can be achieved by backing it with
>> a dax device, or /dev/shm PKRAM as proposed in
>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$  
>> but this is not enforced.  The restarted qemu arguments must match those used
>> to initially start qemu, plus the -incoming option.
>>
>> The reboot mode supports vfio devices if the caller first suspends the guest,
>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>> drivers' suspend methods flush outstanding requests and re-initialize the
>> devices, and thus there is no device state to save and restore.  After
>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>> resume.
>>
>> In cpr-exec mode, the migrate command saves state to a file and directly
>> exec's a new version of qemu on the same host, replacing the original process
>> while retaining its PID.  The caller must specify a migration URI that writes
>> to and reads from a file, and resumes execution via the migrate-incoming
>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>> migration parameter, and must include the -incoming option.
>>
>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>> ram is efficiently preserved in place, albeit with new virtual addresses.
>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>> causes secondary guest ram blocks (those not specified on the command line)
>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>> their values are saved in special cpr state which is retrieved after exec,
>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>> are not migrated, the caller must disable all capabilities related to page
>> and block copy.  The implementation ignores all related parameters.
>>
>> The exec mode supports vfio devices by preserving the vfio container, group,
>> device, and event descriptors across the qemu re-exec, and by updating DMA
>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>    https://urldefense.com/v3/__https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3IK0BrTn$  
>> and integrated in Linux kernel 5.12.
>>
>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>> The software update is performed while the guest is running to minimize
>> downtime.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ...                         |
>>    -migrate-mode-enable cpr-exec                 |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update qemu
>> (qemu) migrate_set_parameter mode cpr-exec      |
>> (qemu) migrate_set_parameter cpr-exec-args      |
>>    qemu-system-$arch ... -incoming defer         |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>>
>> Here is an example of updating the host kernel using reboot mode.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>    -migrate-mode-enable cpr-reboot               |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update kernel-uek
>> (qemu) migrate_set_parameter mode cpr-reboot    |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> (qemu) quit                                     |
>>                                                  |
>> # systemctl kexec                               |
>> kexec_core: Starting new kernel                 |
>> ...                                             |
>>                                                  |
>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>    -incoming defer                               |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>> Changes from V8 to V9:
>>    vfio:
>>      - free all cpr state during unwind in vfio_connect_container
>>      - change cpr_resave_fd to return void, and avoid new unwind cases
>>      - delete incorrect .unmigratable=1 in vmstate handlers
>>      - add route batching in vfio_claim_vectors
>>      - simplified vfio intx cpr code
>>      - fix commit message for 'recover from unmap-all-vaddr failure'
>>      - verify suspended runstate for cpr-reboot mode
>>    Other:
>>      - delete cpr-save, cpr-exec, cpr-load
>>      - delete ram block vmstate handlers that were added in V8
>>      - rename cpr-enable option to migrate-mode-enable
>>      - add file URI for migration
>>      - add mode and cpr-exec-args migration parameters
>>      - add per-mode migration blockers
>>      - add mode checks in migration notifiers
>>      - fix suspended runstate during migration
>>      - replace RAM_ANON flag with RAM_NAMED_FILE
>>      - support memory-backend-epc
>>
>> Steve Sistare (44):
>>    migration: fix populate_vfio_info                  ---  reboot mode  ---
>>    memory: RAM_NAMED_FILE flag
>>    migration: file URI
>>    migration: mode parameter
>>    migration: migrate-enable-mode option
>>    migration: simplify blockers
>>    migration: per-mode blockers
>>    cpr: relax some blockers
>>    cpr: reboot mode
>>
>>    qdev-properties: strList                           ---  exec mode ---
>>    qapi: strList_from_string
>>    qapi: QAPI_LIST_LENGTH
>>    qapi: strv_from_strList
>>    qapi: strList unit tests
>>    migration: cpr-exec-args parameter
>>    migration: simplify notifiers
>>    migration: check mode in notifiers
>>    memory: flat section iterator
>>    oslib: qemu_clear_cloexec
>>    vl: helper to request re-exec
>>    cpr: preserve extra state
>>    cpr: exec mode
>>    cpr: add exec-mode blockers
>>    cpr: ram block blockers
>>    cpr: only-cpr-capable
>>    cpr: Mismatched GPAs fix
>>    hostmem-memfd: cpr support
>>    hostmem-epc: cpr support
>>
>>    pci: export msix_is_pending                       --- vfio for exec ---
>>    vfio-pci: refactor for cpr
>>    vfio-pci: cpr part 1 (fd and dma)
>>    vfio-pci: cpr part 2 (msi)
>>    vfio-pci: cpr part 3 (intx)
>>    vfio-pci: recover from unmap-all-vaddr failure
>>
>>    chardev: cpr framework                            --- misc for exec ---
>>    chardev: cpr for simple devices
>>    chardev: cpr for pty
>>    python/machine: QEMUMachine full_args
>>    python/machine: QEMUMachine reopen_qmp_connection
>>    tests/avocado: add cpr regression test
>>
>>    vl: start on wakeup request                       --- vfio for reboot ---
>>    migration: fix suspended runstate
>>    migration: notifier error reporting
>>    vfio: allow cpr-reboot migration if suspended
>>
>> Mark Kanda, Steve Sistare (2):
>>    vhost: reset vhost devices for cpr
>>    chardev: cpr for sockets
>>
>>   MAINTAINERS                         |  14 ++
>>   accel/xen/xen-all.c                 |   3 +
>>   backends/hostmem-epc.c              |  18 +-
>>   backends/hostmem-file.c             |   1 +
>>   backends/hostmem-memfd.c            |  22 ++-
>>   backends/tpm/tpm_emulator.c         |  11 +-
>>   block/parallels.c                   |   7 +-
>>   block/qcow.c                        |   7 +-
>>   block/vdi.c                         |   7 +-
>>   block/vhdx.c                        |   7 +-
>>   block/vmdk.c                        |   7 +-
>>   block/vpc.c                         |   7 +-
>>   block/vvfat.c                       |   7 +-
>>   chardev/char-mux.c                  |   1 +
>>   chardev/char-null.c                 |   1 +
>>   chardev/char-pty.c                  |  16 +-
>>   chardev/char-serial.c               |   1 +
>>   chardev/char-socket.c               |  48 +++++
>>   chardev/char-stdio.c                |  31 +++
>>   chardev/char.c                      |  49 ++++-
>>   dump/dump.c                         |   4 +-
>>   gdbstub.c                           |   1 +
>>   hmp-commands.hx                     |   2 +-
>>   hw/9pfs/9p.c                        |  11 +-
>>   hw/core/qdev-properties-system.c    |  12 ++
>>   hw/core/qdev-properties.c           |  44 +++++
>>   hw/display/virtio-gpu-base.c        |   8 +-
>>   hw/intc/arm_gic_kvm.c               |   3 +-
>>   hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>   hw/intc/arm_gicv3_kvm.c             |   3 +-
>>   hw/misc/ivshmem.c                   |   8 +-
>>   hw/net/virtio-net.c                 |  10 +-
>>   hw/pci/msix.c                       |   2 +-
>>   hw/pci/pci.c                        |  12 ++
>>   hw/ppc/pef.c                        |   2 +-
>>   hw/ppc/spapr.c                      |   2 +-
>>   hw/ppc/spapr_events.c               |   2 +-
>>   hw/ppc/spapr_rtas.c                 |   2 +-
>>   hw/remote/proxy.c                   |   7 +-
>>   hw/s390x/s390-virtio-ccw.c          |   9 +-
>>   hw/scsi/vhost-scsi.c                |   9 +-
>>   hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>   hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>   hw/vfio/meson.build                 |   1 +
>>   hw/vfio/migration.c                 |  23 +--
>>   hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>   hw/vfio/trace-events                |   1 +
>>   hw/virtio/vhost-vdpa.c              |   6 +-
>>   hw/virtio/vhost.c                   |  32 +++-
>>   include/chardev/char-socket.h       |   1 +
>>   include/chardev/char.h              |   5 +
>>   include/exec/memory.h               |  48 +++++
>>   include/exec/ram_addr.h             |   1 +
>>   include/exec/ramblock.h             |   1 +
>>   include/hw/pci/msix.h               |   1 +
>>   include/hw/qdev-properties-system.h |   4 +
>>   include/hw/qdev-properties.h        |   3 +
>>   include/hw/vfio/vfio-common.h       |  12 ++
>>   include/hw/virtio/vhost.h           |   1 +
>>   include/migration/blocker.h         |  69 ++++++-
>>   include/migration/cpr-state.h       |  30 +++
>>   include/migration/cpr.h             |  20 ++
>>   include/migration/misc.h            |  13 +-
>>   include/migration/vmstate.h         |   2 +
>>   include/qapi/util.h                 |  28 +++
>>   include/qemu/osdep.h                |   9 +
>>   include/sysemu/runstate.h           |   2 +
>>   migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>   migration/cpr.c                     |  85 +++++++++
>>   migration/file.c                    |  62 ++++++
>>   migration/file.h                    |  14 ++
>>   migration/meson.build               |   3 +
>>   migration/migration.c               | 268 +++++++++++++++++++++++---
>>   migration/ram.c                     |  24 ++-
>>   migration/target.c                  |   1 +
>>   migration/trace-events              |  12 ++
>>   monitor/hmp-cmds.c                  |  59 +++---
>>   monitor/hmp.c                       |   3 +
>>   monitor/qmp.c                       |   4 +
>>   python/qemu/machine/machine.py      |  14 ++
>>   qapi/char.json                      |   7 +-
>>   qapi/migration.json                 |  68 ++++++-
>>   qapi/qapi-util.c                    |  37 ++++
>>   qemu-options.hx                     |  50 ++++-
>>   replay/replay.c                     |   4 +
>>   softmmu/memory.c                    |  31 ++-
>>   softmmu/physmem.c                   | 100 +++++++++-
>>   softmmu/runstate.c                  |  42 ++++-
>>   softmmu/vl.c                        |  10 +
>>   stubs/cpr-state.c                   |  26 +++
>>   stubs/meson.build                   |   2 +
>>   stubs/migr-blocker.c                |   9 +-
>>   stubs/migration.c                   |  33 ++++
>>   target/i386/kvm/kvm.c               |   8 +-
>>   target/i386/nvmm/nvmm-all.c         |   4 +-
>>   target/i386/sev.c                   |   2 +-
>>   target/i386/whpx/whpx-all.c         |   3 +-
>>   tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>   tests/unit/meson.build              |   1 +
>>   tests/unit/test-strlist.c           |  81 ++++++++
>>   trace-events                        |   1 +
>>   ui/spice-core.c                     |   5 +-
>>   ui/vdagent.c                        |   5 +-
>>   util/oslib-posix.c                  |   9 +
>>   util/oslib-win32.c                  |   4 +
>>   105 files changed, 2781 insertions(+), 330 deletions(-)
>>   create mode 100644 hw/vfio/cpr.c
>>   create mode 100644 include/migration/cpr-state.h
>>   create mode 100644 include/migration/cpr.h
>>   create mode 100644 migration/cpr-state.c
>>   create mode 100644 migration/cpr.c
>>   create mode 100644 migration/file.c
>>   create mode 100644 migration/file.h
>>   create mode 100644 stubs/cpr-state.c
>>   create mode 100644 stubs/migration.c
>>   create mode 100644 tests/avocado/cpr.py
>>   create mode 100644 tests/unit/test-strlist.c
>>

Another option could be to expose "-migrate-mode-disable" (instead of 
enable) and just enable all 3 modes by default,
since we are already required to switch from "normal" mode to a 
CPR-specific mode when it is time to do a live update,
if the intention is to preserve the capability to completely prevent a 
running QEMU from using these modes
before the VM starts up.

- Michael

On 6/6/23 17:15, Michael Galaxy wrote:
> Hi Steve,
>
> In the current design you have, we have to specify both the command 
> line parameter "-migrate-mode-enable cpr-reboot"
> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>
> Is it possible to opt-in to the CPR mode just once over the monitor 
> instead of having to specify it twice on the command line?
> This would also match the live migration model: You do not need to 
> necessarily "opt in" to live migration mode through
> a command line parameter, you simply request it when you need to. Can 
> CPR behave the same way?
>
> This would also make switching over to a CPR-capable version of QEMU 
> much simpler and would even make it work for
> existing libvirt-managed guests as their command line parameters would 
> no longer need to change. This would allow us to
> simply power-off and power-on existing VMs to make them CPR-capable 
> and then work on a libvirt patch later when
> we're ready to do so.
>
>
> Comments?
>
> - Michael
>
>
> On 12/7/22 09:48, Steven Sistare wrote:
>> This series desperately needs review in its intersection with live migration.
>> The code in other areas has been reviewed and revised multiple times -- thank you!
>>
>> David, Juan, can you spare some time to review this?  I have done my best to order
>> the patches logically (see the labelled groups in this email), and to provide
>> complete and clear cover letter and commit messages. Can I do anything to facilitate,
>> like doing a code walk through via zoom?
>>
>> And of course, I welcome anyone's feedback.
>>
>> Here is the original posting.
>>
>> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$  
>>
>> - Steve
>>
>> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>>> This version of the live update patch series integrates live update into the
>>> live migration framework.  The new interfaces are:
>>>    * mode (migration parameter)
>>>    * cpr-exec-args (migration parameter)
>>>    * file (migration URI)
>>>    * migrate-mode-enable (command-line argument)
>>>    * only-cpr-capable (command-line argument)
>>>
>>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>>> save and restore VM state, with minimal guest pause time, so that qemu may be
>>> updated to a new version in between.  The caller sets the mode parameter
>>> before invoking the migrate or migrate-incoming commands.
>>>
>>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>>> one to quit qemu, reboot to an updated kernel, start an updated version of
>>> qemu, and resume via the migrate-incoming command.  The caller must specify
>>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>>> the use of certain local storage options does not block the migration, but
>>> the caller must not modify guest block devices between the quit and restart.
>>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>>> must be non-volatile across reboot, which can be achieved by backing it with
>>> a dax device, or /dev/shm PKRAM as proposed in
>>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$  
>>> but this is not enforced.  The restarted qemu arguments must match those used
>>> to initially start qemu, plus the -incoming option.
>>>
>>> The reboot mode supports vfio devices if the caller first suspends the guest,
>>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>>> drivers' suspend methods flush outstanding requests and re-initialize the
>>> devices, and thus there is no device state to save and restore.  After
>>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>>> resume.
>>>
>>> In cpr-exec mode, the migrate command saves state to a file and directly
>>> exec's a new version of qemu on the same host, replacing the original process
>>> while retaining its PID.  The caller must specify a migration URI that writes
>>> to and reads from a file, and resumes execution via the migrate-incoming
>>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>>> migration parameter, and must include the -incoming option.
>>>
>>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>>> ram is efficiently preserved in place, albeit with new virtual addresses.
>>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>>> causes secondary guest ram blocks (those not specified on the command line)
>>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>>> their values are saved in special cpr state which is retrieved after exec,
>>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>>> are not migrated, the caller must disable all capabilities related to page
>>> and block copy.  The implementation ignores all related parameters.
>>>
>>> The exec mode supports vfio devices by preserving the vfio container, group,
>>> device, and event descriptors across the qemu re-exec, and by updating DMA
>>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>>    https://urldefense.com/v3/__https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3IK0BrTn$  
>>> and integrated in Linux kernel 5.12.
>>>
>>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>>> The software update is performed while the guest is running to minimize
>>> downtime.
>>>
>>> window 1                                        | window 2
>>>                                                  |
>>> # qemu-system-$arch ...                         |
>>>    -migrate-mode-enable cpr-exec                 |
>>> QEMU 7.0.50 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>                                                  | # yum update qemu
>>> (qemu) migrate_set_parameter mode cpr-exec      |
>>> (qemu) migrate_set_parameter cpr-exec-args      |
>>>    qemu-system-$arch ... -incoming defer         |
>>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>>> QEMU 7.0.51 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: paused (inmigrate)                   |
>>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>
>>>
>>> Here is an example of updating the host kernel using reboot mode.
>>>
>>> window 1                                        | window 2
>>>                                                  |
>>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>>    -migrate-mode-enable cpr-reboot               |
>>> QEMU 7.0.50 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>                                                  | # yum update kernel-uek
>>> (qemu) migrate_set_parameter mode cpr-reboot    |
>>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>>> (qemu) quit                                     |
>>>                                                  |
>>> # systemctl kexec                               |
>>> kexec_core: Starting new kernel                 |
>>> ...                                             |
>>>                                                  |
>>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>>    -incoming defer                               |
>>> QEMU 7.0.51 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: paused (inmigrate)                   |
>>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>
>>> Changes from V8 to V9:
>>>    vfio:
>>>      - free all cpr state during unwind in vfio_connect_container
>>>      - change cpr_resave_fd to return void, and avoid new unwind cases
>>>      - delete incorrect .unmigratable=1 in vmstate handlers
>>>      - add route batching in vfio_claim_vectors
>>>      - simplified vfio intx cpr code
>>>      - fix commit message for 'recover from unmap-all-vaddr failure'
>>>      - verify suspended runstate for cpr-reboot mode
>>>    Other:
>>>      - delete cpr-save, cpr-exec, cpr-load
>>>      - delete ram block vmstate handlers that were added in V8
>>>      - rename cpr-enable option to migrate-mode-enable
>>>      - add file URI for migration
>>>      - add mode and cpr-exec-args migration parameters
>>>      - add per-mode migration blockers
>>>      - add mode checks in migration notifiers
>>>      - fix suspended runstate during migration
>>>      - replace RAM_ANON flag with RAM_NAMED_FILE
>>>      - support memory-backend-epc
>>>
>>> Steve Sistare (44):
>>>    migration: fix populate_vfio_info                  ---  reboot mode  ---
>>>    memory: RAM_NAMED_FILE flag
>>>    migration: file URI
>>>    migration: mode parameter
>>>    migration: migrate-enable-mode option
>>>    migration: simplify blockers
>>>    migration: per-mode blockers
>>>    cpr: relax some blockers
>>>    cpr: reboot mode
>>>
>>>    qdev-properties: strList                           ---  exec mode ---
>>>    qapi: strList_from_string
>>>    qapi: QAPI_LIST_LENGTH
>>>    qapi: strv_from_strList
>>>    qapi: strList unit tests
>>>    migration: cpr-exec-args parameter
>>>    migration: simplify notifiers
>>>    migration: check mode in notifiers
>>>    memory: flat section iterator
>>>    oslib: qemu_clear_cloexec
>>>    vl: helper to request re-exec
>>>    cpr: preserve extra state
>>>    cpr: exec mode
>>>    cpr: add exec-mode blockers
>>>    cpr: ram block blockers
>>>    cpr: only-cpr-capable
>>>    cpr: Mismatched GPAs fix
>>>    hostmem-memfd: cpr support
>>>    hostmem-epc: cpr support
>>>
>>>    pci: export msix_is_pending                       --- vfio for exec ---
>>>    vfio-pci: refactor for cpr
>>>    vfio-pci: cpr part 1 (fd and dma)
>>>    vfio-pci: cpr part 2 (msi)
>>>    vfio-pci: cpr part 3 (intx)
>>>    vfio-pci: recover from unmap-all-vaddr failure
>>>
>>>    chardev: cpr framework                            --- misc for exec ---
>>>    chardev: cpr for simple devices
>>>    chardev: cpr for pty
>>>    python/machine: QEMUMachine full_args
>>>    python/machine: QEMUMachine reopen_qmp_connection
>>>    tests/avocado: add cpr regression test
>>>
>>>    vl: start on wakeup request                       --- vfio for reboot ---
>>>    migration: fix suspended runstate
>>>    migration: notifier error reporting
>>>    vfio: allow cpr-reboot migration if suspended
>>>
>>> Mark Kanda, Steve Sistare (2):
>>>    vhost: reset vhost devices for cpr
>>>    chardev: cpr for sockets
>>>
>>>   MAINTAINERS                         |  14 ++
>>>   accel/xen/xen-all.c                 |   3 +
>>>   backends/hostmem-epc.c              |  18 +-
>>>   backends/hostmem-file.c             |   1 +
>>>   backends/hostmem-memfd.c            |  22 ++-
>>>   backends/tpm/tpm_emulator.c         |  11 +-
>>>   block/parallels.c                   |   7 +-
>>>   block/qcow.c                        |   7 +-
>>>   block/vdi.c                         |   7 +-
>>>   block/vhdx.c                        |   7 +-
>>>   block/vmdk.c                        |   7 +-
>>>   block/vpc.c                         |   7 +-
>>>   block/vvfat.c                       |   7 +-
>>>   chardev/char-mux.c                  |   1 +
>>>   chardev/char-null.c                 |   1 +
>>>   chardev/char-pty.c                  |  16 +-
>>>   chardev/char-serial.c               |   1 +
>>>   chardev/char-socket.c               |  48 +++++
>>>   chardev/char-stdio.c                |  31 +++
>>>   chardev/char.c                      |  49 ++++-
>>>   dump/dump.c                         |   4 +-
>>>   gdbstub.c                           |   1 +
>>>   hmp-commands.hx                     |   2 +-
>>>   hw/9pfs/9p.c                        |  11 +-
>>>   hw/core/qdev-properties-system.c    |  12 ++
>>>   hw/core/qdev-properties.c           |  44 +++++
>>>   hw/display/virtio-gpu-base.c        |   8 +-
>>>   hw/intc/arm_gic_kvm.c               |   3 +-
>>>   hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>>   hw/intc/arm_gicv3_kvm.c             |   3 +-
>>>   hw/misc/ivshmem.c                   |   8 +-
>>>   hw/net/virtio-net.c                 |  10 +-
>>>   hw/pci/msix.c                       |   2 +-
>>>   hw/pci/pci.c                        |  12 ++
>>>   hw/ppc/pef.c                        |   2 +-
>>>   hw/ppc/spapr.c                      |   2 +-
>>>   hw/ppc/spapr_events.c               |   2 +-
>>>   hw/ppc/spapr_rtas.c                 |   2 +-
>>>   hw/remote/proxy.c                   |   7 +-
>>>   hw/s390x/s390-virtio-ccw.c          |   9 +-
>>>   hw/scsi/vhost-scsi.c                |   9 +-
>>>   hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>>   hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>>   hw/vfio/meson.build                 |   1 +
>>>   hw/vfio/migration.c                 |  23 +--
>>>   hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>>   hw/vfio/trace-events                |   1 +
>>>   hw/virtio/vhost-vdpa.c              |   6 +-
>>>   hw/virtio/vhost.c                   |  32 +++-
>>>   include/chardev/char-socket.h       |   1 +
>>>   include/chardev/char.h              |   5 +
>>>   include/exec/memory.h               |  48 +++++
>>>   include/exec/ram_addr.h             |   1 +
>>>   include/exec/ramblock.h             |   1 +
>>>   include/hw/pci/msix.h               |   1 +
>>>   include/hw/qdev-properties-system.h |   4 +
>>>   include/hw/qdev-properties.h        |   3 +
>>>   include/hw/vfio/vfio-common.h       |  12 ++
>>>   include/hw/virtio/vhost.h           |   1 +
>>>   include/migration/blocker.h         |  69 ++++++-
>>>   include/migration/cpr-state.h       |  30 +++
>>>   include/migration/cpr.h             |  20 ++
>>>   include/migration/misc.h            |  13 +-
>>>   include/migration/vmstate.h         |   2 +
>>>   include/qapi/util.h                 |  28 +++
>>>   include/qemu/osdep.h                |   9 +
>>>   include/sysemu/runstate.h           |   2 +
>>>   migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>>   migration/cpr.c                     |  85 +++++++++
>>>   migration/file.c                    |  62 ++++++
>>>   migration/file.h                    |  14 ++
>>>   migration/meson.build               |   3 +
>>>   migration/migration.c               | 268 +++++++++++++++++++++++---
>>>   migration/ram.c                     |  24 ++-
>>>   migration/target.c                  |   1 +
>>>   migration/trace-events              |  12 ++
>>>   monitor/hmp-cmds.c                  |  59 +++---
>>>   monitor/hmp.c                       |   3 +
>>>   monitor/qmp.c                       |   4 +
>>>   python/qemu/machine/machine.py      |  14 ++
>>>   qapi/char.json                      |   7 +-
>>>   qapi/migration.json                 |  68 ++++++-
>>>   qapi/qapi-util.c                    |  37 ++++
>>>   qemu-options.hx                     |  50 ++++-
>>>   replay/replay.c                     |   4 +
>>>   softmmu/memory.c                    |  31 ++-
>>>   softmmu/physmem.c                   | 100 +++++++++-
>>>   softmmu/runstate.c                  |  42 ++++-
>>>   softmmu/vl.c                        |  10 +
>>>   stubs/cpr-state.c                   |  26 +++
>>>   stubs/meson.build                   |   2 +
>>>   stubs/migr-blocker.c                |   9 +-
>>>   stubs/migration.c                   |  33 ++++
>>>   target/i386/kvm/kvm.c               |   8 +-
>>>   target/i386/nvmm/nvmm-all.c         |   4 +-
>>>   target/i386/sev.c                   |   2 +-
>>>   target/i386/whpx/whpx-all.c         |   3 +-
>>>   tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>>   tests/unit/meson.build              |   1 +
>>>   tests/unit/test-strlist.c           |  81 ++++++++
>>>   trace-events                        |   1 +
>>>   ui/spice-core.c                     |   5 +-
>>>   ui/vdagent.c                        |   5 +-
>>>   util/oslib-posix.c                  |   9 +
>>>   util/oslib-win32.c                  |   4 +
>>>   105 files changed, 2781 insertions(+), 330 deletions(-)
>>>   create mode 100644 hw/vfio/cpr.c
>>>   create mode 100644 include/migration/cpr-state.h
>>>   create mode 100644 include/migration/cpr.h
>>>   create mode 100644 migration/cpr-state.c
>>>   create mode 100644 migration/cpr.c
>>>   create mode 100644 migration/file.c
>>>   create mode 100644 migration/file.h
>>>   create mode 100644 stubs/cpr-state.c
>>>   create mode 100644 stubs/migration.c
>>>   create mode 100644 tests/avocado/cpr.py
>>>   create mode 100644 tests/unit/test-strlist.c
>>>

On 6/7/2023 11:55 AM, Michael Galaxy wrote:
> Another option could be to expose "-migrate-mode-disable" (instead of enable) and just enable all 3 modes by default,
> since we are already required to switch from "normal" mode to a CPR-specific mode when it is time to do a live update,
> if the intention is to preserve the capability to completely prevent a running QEMU from using these modes
> before the VM starts up.
> 
> - Michael
> 
> On 6/6/23 17:15, Michael Galaxy wrote:
>> Hi Steve,
>>
>> In the current design you have, we have to specify both the command line parameter "-migrate-mode-enable cpr-reboot"
>> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>>
>> Is it possible to opt-in to the CPR mode just once over the monitor instead of having to specify it twice on the command line?
>> This would also match the live migration model: You do not need to necessarily "opt in" to live migration mode through
>> a command line parameter, you simply request it when you need to. Can CPR behave the same way?
>>
>> This would also make switching over to a CPR-capable version of QEMU much simpler and would even make it work for
>> existing libvirt-managed guests as their command line parameters would no longer need to change. This would allow us to
>> simply power-off and power-on existing VMs to make them CPR-capable and then work on a libvirt patch later when
>> we're ready to do so.
>>
>>
>> Comments?

Hi Michael,
  Requiring -migrate-enable-mode allows qemu to initialize objects
differently, if necessary, so that migration for a mode is not blocked.
See callers of migrate_mode_enabled.  There is only one so far, in
ram_block_add.  If the mode is cpr-exec, then it creates anonymous ram
blocks using memfd_create, else using MAP_ANON.  In the V7 series, this
was controlled by a '-machine memfd-alloc=on' option.  

migrate-enable-mode is more future proof for the user.  If something new must
initialize differently to support cpr, then it adds a call to migrate_mode_enabled,
and the command line remains the same.  However, I could be persuaded to go either way.

A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
option.  It needs to know which mode will be used, in order to check a
mode-specific blocker list.

- Steve

Hi Steve,


On 6/7/23 12:37, Steven Sistare wrote:
> On 6/7/2023 11:55 AM, Michael Galaxy wrote:
>> Another option could be to expose "-migrate-mode-disable" (instead of enable) and just enable all 3 modes by default,
>> since we are already required to switch from "normal" mode to a CPR-specific mode when it is time to do a live update,
>> if the intention is to preserve the capability to completely prevent a running QEMU from using these modes
>> before the VM starts up.
>>
>> - Michael
>>
>> On 6/6/23 17:15, Michael Galaxy wrote:
>>> Hi Steve,
>>>
>>> In the current design you have, we have to specify both the command line parameter "-migrate-mode-enable cpr-reboot"
>>> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>>>
>>> Is it possible to opt-in to the CPR mode just once over the monitor instead of having to specify it twice on the command line?
>>> This would also match the live migration model: You do not need to necessarily "opt in" to live migration mode through
>>> a command line parameter, you simply request it when you need to. Can CPR behave the same way?
>>>
>>> This would also make switching over to a CPR-capable version of QEMU much simpler and would even make it work for
>>> existing libvirt-managed guests as their command line parameters would no longer need to change. This would allow us to
>>> simply power-off and power-on existing VMs to make them CPR-capable and then work on a libvirt patch later when
>>> we're ready to do so.
>>>
>>>
>>> Comments?
> Hi Michael,
>    Requiring -migrate-enable-mode allows qemu to initialize objects
> differently, if necessary, so that migration for a mode is not blocked.
> See callers of migrate_mode_enabled.  There is only one so far, in
> ram_block_add.  If the mode is cpr-exec, then it creates anonymous ram
> blocks using memfd_create, else using MAP_ANON.  In the V7 series, this
> was controlled by a '-machine memfd-alloc=on' option.
>
> migrate-enable-mode is more future proof for the user.  If something new must
> initialize differently to support cpr, then it adds a call to migrate_mode_enabled,
> and the command line remains the same.  However, I could be persuaded to go either way.


OK, so it is cpr-exec that needs this option (because of ram block 
allocation), not really cpr-reboot.
Could the option then be made to only be required for cpr-exec and not 
cpr-reboot, then,
since cpr-reboot doesn't require that consideration?


> A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
> option.  It needs to know which mode will be used, in order to check a
> mode-specific blocker list.

Still, only-cpr-capable is also optional. If and only if one needs this 
option, the mode could be
specified as part of the option itself, rather than requiring an extra 
command line parameter, no?

Further, in many clouds (including ours), our VM-management is 
generational with the
development of the software versioning, so we *always* run tests and 
know whether or not a VM is CPR-capable.

If it is not CPR-capable, we would never run the VM in the first place, 
which means we would never
really use that option at all.

I do see the appeal of the option, but could we simplify it, since it is 
opt-in?

- Michael


>
> - Steve

On 6/12/2023 10:59 AM, Michael Galaxy wrote:
> Hi Steve,
> On 6/7/23 12:37, Steven Sistare wrote:
>> On 6/7/2023 11:55 AM, Michael Galaxy wrote:
>>> Another option could be to expose "-migrate-mode-disable" (instead of enable) and just enable all 3 modes by default,
>>> since we are already required to switch from "normal" mode to a CPR-specific mode when it is time to do a live update,
>>> if the intention is to preserve the capability to completely prevent a running QEMU from using these modes
>>> before the VM starts up.
>>>
>>> - Michael
>>>
>>> On 6/6/23 17:15, Michael Galaxy wrote:
>>>> Hi Steve,
>>>>
>>>> In the current design you have, we have to specify both the command line parameter "-migrate-mode-enable cpr-reboot"
>>>> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>>>>
>>>> Is it possible to opt-in to the CPR mode just once over the monitor instead of having to specify it twice on the command line?
>>>> This would also match the live migration model: You do not need to necessarily "opt in" to live migration mode through
>>>> a command line parameter, you simply request it when you need to. Can CPR behave the same way?
>>>>
>>>> This would also make switching over to a CPR-capable version of QEMU much simpler and would even make it work for
>>>> existing libvirt-managed guests as their command line parameters would no longer need to change. This would allow us to
>>>> simply power-off and power-on existing VMs to make them CPR-capable and then work on a libvirt patch later when
>>>> we're ready to do so.
>>>>
>>>>
>>>> Comments?
>> Hi Michael,
>>    Requiring -migrate-enable-mode allows qemu to initialize objects
>> differently, if necessary, so that migration for a mode is not blocked.
>> See callers of migrate_mode_enabled.  There is only one so far, in
>> ram_block_add.  If the mode is cpr-exec, then it creates anonymous ram
>> blocks using memfd_create, else using MAP_ANON.  In the V7 series, this
>> was controlled by a '-machine memfd-alloc=on' option.
>>
>> migrate-enable-mode is more future proof for the user.  If something new must
>> initialize differently to support cpr, then it adds a call to migrate_mode_enabled,
>> and the command line remains the same.  However, I could be persuaded to go either way.
> 
> OK, so it is cpr-exec that needs this option (because of ram block allocation), not really cpr-reboot.
> Could the option then be made to only be required for cpr-exec and not cpr-reboot, then,
> since cpr-reboot doesn't require that consideration?

In a different forum Juan said this is a memory issue, so it should be expressed
as a memory related option.  So, I will delete -migrate-enable-mode and revert back
to -machine memfd-alloc, as defined in the V7 patch series.

>> A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
>> option.  It needs to know which mode will be used, in order to check a
>> mode-specific blocker list.
> 
> Still, only-cpr-capable is also optional. If and only if one needs this option, the mode could be
> specified as part of the option itself, rather than requiring an extra command line parameter, no?

Yes, I will make that change.

- Steve

> Further, in many clouds (including ours), our VM-management is generational with the
> development of the software versioning, so we *always* run tests and know whether or not a VM is CPR-capable.
> 
> If it is not CPR-capable, we would never run the VM in the first place, which means we would never
> really use that option at all.
> 
> I do see the appeal of the option, but could we simplify it, since it is opt-in?

  Good morning,

On 7/10/23 10:10, Steven Sistare wrote:
> On 6/12/2023 10:59 AM, Michael Galaxy wrote:
>> Hi Steve,
>> On 6/7/23 12:37, Steven Sistare wrote:
>>> On 6/7/2023 11:55 AM, Michael Galaxy wrote:
>>>> Another option could be to expose "-migrate-mode-disable" (instead of enable) and just enable all 3 modes by default,
>>>> since we are already required to switch from "normal" mode to a CPR-specific mode when it is time to do a live update,
>>>> if the intention is to preserve the capability to completely prevent a running QEMU from using these modes
>>>> before the VM starts up.
>>>>
>>>> - Michael
>>>>
>>>> On 6/6/23 17:15, Michael Galaxy wrote:
>>>>> Hi Steve,
>>>>>
>>>>> In the current design you have, we have to specify both the command line parameter "-migrate-mode-enable cpr-reboot"
>>>>> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>>>>>
>>>>> Is it possible to opt-in to the CPR mode just once over the monitor instead of having to specify it twice on the command line?
>>>>> This would also match the live migration model: You do not need to necessarily "opt in" to live migration mode through
>>>>> a command line parameter, you simply request it when you need to. Can CPR behave the same way?
>>>>>
>>>>> This would also make switching over to a CPR-capable version of QEMU much simpler and would even make it work for
>>>>> existing libvirt-managed guests as their command line parameters would no longer need to change. This would allow us to
>>>>> simply power-off and power-on existing VMs to make them CPR-capable and then work on a libvirt patch later when
>>>>> we're ready to do so.
>>>>>
>>>>>
>>>>> Comments?
>>> Hi Michael,
>>>     Requiring -migrate-enable-mode allows qemu to initialize objects
>>> differently, if necessary, so that migration for a mode is not blocked.
>>> See callers of migrate_mode_enabled.  There is only one so far, in
>>> ram_block_add.  If the mode is cpr-exec, then it creates anonymous ram
>>> blocks using memfd_create, else using MAP_ANON.  In the V7 series, this
>>> was controlled by a '-machine memfd-alloc=on' option.
>>>
>>> migrate-enable-mode is more future proof for the user.  If something new must
>>> initialize differently to support cpr, then it adds a call to migrate_mode_enabled,
>>> and the command line remains the same.  However, I could be persuaded to go either way.
>> OK, so it is cpr-exec that needs this option (because of ram block allocation), not really cpr-reboot.
>> Could the option then be made to only be required for cpr-exec and not cpr-reboot, then,
>> since cpr-reboot doesn't require that consideration?
> In a different forum Juan said this is a memory issue, so it should be expressed
> as a memory related option.  So, I will delete -migrate-enable-mode and revert back
> to -machine memfd-alloc, as defined in the V7 patch series.

Acknowledged. I'm going to try to get my reviewed-by's in soon. Sorry I 
haven't done it sooner.

We've finished testing these patches on our systems and are moving forward.


>>> A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
>>> option.  It needs to know which mode will be used, in order to check a
>>> mode-specific blocker list.
>> Still, only-cpr-capable is also optional. If and only if one needs this option, the mode could be
>> specified as part of the option itself, rather than requiring an extra command line parameter, no?
> Yes, I will make that change.
>
> - Steve

Acknowledged.

Greetings,

For what its worth, our team has been aggressively testing this patch 
series and to date have not found any deficiencies or bottlenecks 
whatsoever.

Overall, with a very well-tuned system (and linux kernel), we are 
getting live update downtimes just above 15 seconds, and the vast 
majority of that downtime is kind of wasted time during kexec 
(particularly with PCI device probing, if anybody has any tips on that) 
and has nothing to do whatsoever with the QEMU-side of things. (We use 
this patchset in a PMEM-based configuration which has been working out 
extremely well so far).

We also tested "back-to-back" live updates as you would expect to do in 
production.

Overall we've done thousands of live updates over the past few months on 
many different types of hardware without failure.

Steven has been really responsive in answering some of our usability 
questions and we were able to fix those issues.

We will continue our testing throughout the year with more 
heavily-loaded workloads, but all in all we would very much be 
interested in seeing further reviews on this patch series from others.
*
*---
Tested-by: Michael Galaxy <mgalaxy@akamai.com>

On 12/7/22 09:48, Steven Sistare wrote:
> This series desperately needs review in its intersection with live migration.
> The code in other areas has been reviewed and revised multiple times -- thank you!
>
> David, Juan, can you spare some time to review this?  I have done my best to order
> the patches logically (see the labelled groups in this email), and to provide
> complete and clear cover letter and commit messages. Can I do anything to facilitate,
> like doing a code walk through via zoom?
>
> And of course, I welcome anyone's feedback.
>
> Here is the original posting.
>
> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$  
>
> - Steve
>
> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>> This version of the live update patch series integrates live update into the
>> live migration framework.  The new interfaces are:
>>    * mode (migration parameter)
>>    * cpr-exec-args (migration parameter)
>>    * file (migration URI)
>>    * migrate-mode-enable (command-line argument)
>>    * only-cpr-capable (command-line argument)
>>
>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>> save and restore VM state, with minimal guest pause time, so that qemu may be
>> updated to a new version in between.  The caller sets the mode parameter
>> before invoking the migrate or migrate-incoming commands.
>>
>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>> one to quit qemu, reboot to an updated kernel, start an updated version of
>> qemu, and resume via the migrate-incoming command.  The caller must specify
>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>> the use of certain local storage options does not block the migration, but
>> the caller must not modify guest block devices between the quit and restart.
>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>> must be non-volatile across reboot, which can be achieved by backing it with
>> a dax device, or /dev/shm PKRAM as proposed in
>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$  
>> but this is not enforced.  The restarted qemu arguments must match those used
>> to initially start qemu, plus the -incoming option.
>>
>> The reboot mode supports vfio devices if the caller first suspends the guest,
>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>> drivers' suspend methods flush outstanding requests and re-initialize the
>> devices, and thus there is no device state to save and restore.  After
>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>> resume.
>>
>> In cpr-exec mode, the migrate command saves state to a file and directly
>> exec's a new version of qemu on the same host, replacing the original process
>> while retaining its PID.  The caller must specify a migration URI that writes
>> to and reads from a file, and resumes execution via the migrate-incoming
>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>> migration parameter, and must include the -incoming option.
>>
>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>> ram is efficiently preserved in place, albeit with new virtual addresses.
>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>> causes secondary guest ram blocks (those not specified on the command line)
>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>> their values are saved in special cpr state which is retrieved after exec,
>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>> are not migrated, the caller must disable all capabilities related to page
>> and block copy.  The implementation ignores all related parameters.
>>
>> The exec mode supports vfio devices by preserving the vfio container, group,
>> device, and event descriptors across the qemu re-exec, and by updating DMA
>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>    https://urldefense.com/v3/__https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3IK0BrTn$  
>> and integrated in Linux kernel 5.12.
>>
>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>> The software update is performed while the guest is running to minimize
>> downtime.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ...                         |
>>    -migrate-mode-enable cpr-exec                 |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update qemu
>> (qemu) migrate_set_parameter mode cpr-exec      |
>> (qemu) migrate_set_parameter cpr-exec-args      |
>>    qemu-system-$arch ... -incoming defer         |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>>
>> Here is an example of updating the host kernel using reboot mode.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>    -migrate-mode-enable cpr-reboot               |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update kernel-uek
>> (qemu) migrate_set_parameter mode cpr-reboot    |
>> (qemu) migrate -dfile:/tmp/qemu.sav             |
>> (qemu) quit                                     |
>>                                                  |
>> # systemctl kexec                               |
>> kexec_core: Starting new kernel                 |
>> ...                                             |
>>                                                  |
>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>    -incoming defer                               |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incomingfile:/tmp/qemu.sav       |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>> Changes from V8 to V9:
>>    vfio:
>>      - free all cpr state during unwind in vfio_connect_container
>>      - change cpr_resave_fd to return void, and avoid new unwind cases
>>      - delete incorrect .unmigratable=1 in vmstate handlers
>>      - add route batching in vfio_claim_vectors
>>      - simplified vfio intx cpr code
>>      - fix commit message for 'recover from unmap-all-vaddr failure'
>>      - verify suspended runstate for cpr-reboot mode
>>    Other:
>>      - delete cpr-save, cpr-exec, cpr-load
>>      - delete ram block vmstate handlers that were added in V8
>>      - rename cpr-enable option to migrate-mode-enable
>>      - add file URI for migration
>>      - add mode and cpr-exec-args migration parameters
>>      - add per-mode migration blockers
>>      - add mode checks in migration notifiers
>>      - fix suspended runstate during migration
>>      - replace RAM_ANON flag with RAM_NAMED_FILE
>>      - support memory-backend-epc
>>
>> Steve Sistare (44):
>>    migration: fix populate_vfio_info                  ---  reboot mode  ---
>>    memory: RAM_NAMED_FILE flag
>>    migration: file URI
>>    migration: mode parameter
>>    migration: migrate-enable-mode option
>>    migration: simplify blockers
>>    migration: per-mode blockers
>>    cpr: relax some blockers
>>    cpr: reboot mode
>>
>>    qdev-properties: strList                           ---  exec mode ---
>>    qapi: strList_from_string
>>    qapi: QAPI_LIST_LENGTH
>>    qapi: strv_from_strList
>>    qapi: strList unit tests
>>    migration: cpr-exec-args parameter
>>    migration: simplify notifiers
>>    migration: check mode in notifiers
>>    memory: flat section iterator
>>    oslib: qemu_clear_cloexec
>>    vl: helper to request re-exec
>>    cpr: preserve extra state
>>    cpr: exec mode
>>    cpr: add exec-mode blockers
>>    cpr: ram block blockers
>>    cpr: only-cpr-capable
>>    cpr: Mismatched GPAs fix
>>    hostmem-memfd: cpr support
>>    hostmem-epc: cpr support
>>
>>    pci: export msix_is_pending                       --- vfio for exec ---
>>    vfio-pci: refactor for cpr
>>    vfio-pci: cpr part 1 (fd and dma)
>>    vfio-pci: cpr part 2 (msi)
>>    vfio-pci: cpr part 3 (intx)
>>    vfio-pci: recover from unmap-all-vaddr failure
>>
>>    chardev: cpr framework                            --- misc for exec ---
>>    chardev: cpr for simple devices
>>    chardev: cpr for pty
>>    python/machine: QEMUMachine full_args
>>    python/machine: QEMUMachine reopen_qmp_connection
>>    tests/avocado: add cpr regression test
>>
>>    vl: start on wakeup request                       --- vfio for reboot ---
>>    migration: fix suspended runstate
>>    migration: notifier error reporting
>>    vfio: allow cpr-reboot migration if suspended
>>
>> Mark Kanda, Steve Sistare (2):
>>    vhost: reset vhost devices for cpr
>>    chardev: cpr for sockets
>>
>>   MAINTAINERS                         |  14 ++
>>   accel/xen/xen-all.c                 |   3 +
>>   backends/hostmem-epc.c              |  18 +-
>>   backends/hostmem-file.c             |   1 +
>>   backends/hostmem-memfd.c            |  22 ++-
>>   backends/tpm/tpm_emulator.c         |  11 +-
>>   block/parallels.c                   |   7 +-
>>   block/qcow.c                        |   7 +-
>>   block/vdi.c                         |   7 +-
>>   block/vhdx.c                        |   7 +-
>>   block/vmdk.c                        |   7 +-
>>   block/vpc.c                         |   7 +-
>>   block/vvfat.c                       |   7 +-
>>   chardev/char-mux.c                  |   1 +
>>   chardev/char-null.c                 |   1 +
>>   chardev/char-pty.c                  |  16 +-
>>   chardev/char-serial.c               |   1 +
>>   chardev/char-socket.c               |  48 +++++
>>   chardev/char-stdio.c                |  31 +++
>>   chardev/char.c                      |  49 ++++-
>>   dump/dump.c                         |   4 +-
>>   gdbstub.c                           |   1 +
>>   hmp-commands.hx                     |   2 +-
>>   hw/9pfs/9p.c                        |  11 +-
>>   hw/core/qdev-properties-system.c    |  12 ++
>>   hw/core/qdev-properties.c           |  44 +++++
>>   hw/display/virtio-gpu-base.c        |   8 +-
>>   hw/intc/arm_gic_kvm.c               |   3 +-
>>   hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>   hw/intc/arm_gicv3_kvm.c             |   3 +-
>>   hw/misc/ivshmem.c                   |   8 +-
>>   hw/net/virtio-net.c                 |  10 +-
>>   hw/pci/msix.c                       |   2 +-
>>   hw/pci/pci.c                        |  12 ++
>>   hw/ppc/pef.c                        |   2 +-
>>   hw/ppc/spapr.c                      |   2 +-
>>   hw/ppc/spapr_events.c               |   2 +-
>>   hw/ppc/spapr_rtas.c                 |   2 +-
>>   hw/remote/proxy.c                   |   7 +-
>>   hw/s390x/s390-virtio-ccw.c          |   9 +-
>>   hw/scsi/vhost-scsi.c                |   9 +-
>>   hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>   hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>   hw/vfio/meson.build                 |   1 +
>>   hw/vfio/migration.c                 |  23 +--
>>   hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>   hw/vfio/trace-events                |   1 +
>>   hw/virtio/vhost-vdpa.c              |   6 +-
>>   hw/virtio/vhost.c                   |  32 +++-
>>   include/chardev/char-socket.h       |   1 +
>>   include/chardev/char.h              |   5 +
>>   include/exec/memory.h               |  48 +++++
>>   include/exec/ram_addr.h             |   1 +
>>   include/exec/ramblock.h             |   1 +
>>   include/hw/pci/msix.h               |   1 +
>>   include/hw/qdev-properties-system.h |   4 +
>>   include/hw/qdev-properties.h        |   3 +
>>   include/hw/vfio/vfio-common.h       |  12 ++
>>   include/hw/virtio/vhost.h           |   1 +
>>   include/migration/blocker.h         |  69 ++++++-
>>   include/migration/cpr-state.h       |  30 +++
>>   include/migration/cpr.h             |  20 ++
>>   include/migration/misc.h            |  13 +-
>>   include/migration/vmstate.h         |   2 +
>>   include/qapi/util.h                 |  28 +++
>>   include/qemu/osdep.h                |   9 +
>>   include/sysemu/runstate.h           |   2 +
>>   migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>   migration/cpr.c                     |  85 +++++++++
>>   migration/file.c                    |  62 ++++++
>>   migration/file.h                    |  14 ++
>>   migration/meson.build               |   3 +
>>   migration/migration.c               | 268 +++++++++++++++++++++++---
>>   migration/ram.c                     |  24 ++-
>>   migration/target.c                  |   1 +
>>   migration/trace-events              |  12 ++
>>   monitor/hmp-cmds.c                  |  59 +++---
>>   monitor/hmp.c                       |   3 +
>>   monitor/qmp.c                       |   4 +
>>   python/qemu/machine/machine.py      |  14 ++
>>   qapi/char.json                      |   7 +-
>>   qapi/migration.json                 |  68 ++++++-
>>   qapi/qapi-util.c                    |  37 ++++
>>   qemu-options.hx                     |  50 ++++-
>>   replay/replay.c                     |   4 +
>>   softmmu/memory.c                    |  31 ++-
>>   softmmu/physmem.c                   | 100 +++++++++-
>>   softmmu/runstate.c                  |  42 ++++-
>>   softmmu/vl.c                        |  10 +
>>   stubs/cpr-state.c                   |  26 +++
>>   stubs/meson.build                   |   2 +
>>   stubs/migr-blocker.c                |   9 +-
>>   stubs/migration.c                   |  33 ++++
>>   target/i386/kvm/kvm.c               |   8 +-
>>   target/i386/nvmm/nvmm-all.c         |   4 +-
>>   target/i386/sev.c                   |   2 +-
>>   target/i386/whpx/whpx-all.c         |   3 +-
>>   tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>   tests/unit/meson.build              |   1 +
>>   tests/unit/test-strlist.c           |  81 ++++++++
>>   trace-events                        |   1 +
>>   ui/spice-core.c                     |   5 +-
>>   ui/vdagent.c                        |   5 +-
>>   util/oslib-posix.c                  |   9 +
>>   util/oslib-win32.c                  |   4 +
>>   105 files changed, 2781 insertions(+), 330 deletions(-)
>>   create mode 100644 hw/vfio/cpr.c
>>   create mode 100644 include/migration/cpr-state.h
>>   create mode 100644 include/migration/cpr.h
>>   create mode 100644 migration/cpr-state.c
>>   create mode 100644 migration/cpr.c
>>   create mode 100644 migration/file.c
>>   create mode 100644 migration/file.h
>>   create mode 100644 stubs/cpr-state.c
>>   create mode 100644 stubs/migration.c
>>   create mode 100644 tests/avocado/cpr.py
>>   create mode 100644 tests/unit/test-strlist.c
>>

Hey Steven,

Have you done any "back-to-back" live update testing before?

I am still doing extensive testing on this myself. I am running into a 
bug that I have not yet diagnosed.

It involves the following:

1. Perform a live update (I'm using kexec + PMEM-based live updates). => 
VM comes back just fine. All is well.
2. Using the *same* QEMU instance: Try to perform another live update 
agaig,
this results in the following error:

monitor> migrate_incoming file:qemu.sav
qemu-system-x86_64: check_section_footer: Read section footer failed: -5
qemu-system-x86_64: load of migration failed: Invalid argument

I'm going to start diving into the code soon, but just thought I would 
ask first.

- Michael

On 12/7/22 09:48, Steven Sistare wrote:

> This series desperately needs review in its intersection with live migration.
> The code in other areas has been reviewed and revised multiple times -- thank you!
>
> David, Juan, can you spare some time to review this?  I have done my best to order
> the patches logically (see the labelled groups in this email), and to provide
> complete and clear cover letter and commit messages. Can I do anything to facilitate,
> like doing a code walk through via zoom?
>
> And of course, I welcome anyone's feedback.
>
> Here is the original posting.
>
> https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$
>
> - Steve
>
> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>> This version of the live update patch series integrates live update into the
>> live migration framework.  The new interfaces are:
>>    * mode (migration parameter)
>>    * cpr-exec-args (migration parameter)
>>    * file (migration URI)
>>    * migrate-mode-enable (command-line argument)
>>    * only-cpr-capable (command-line argument)
>>
>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>> save and restore VM state, with minimal guest pause time, so that qemu may be
>> updated to a new version in between.  The caller sets the mode parameter
>> before invoking the migrate or migrate-incoming commands.
>>
>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>> one to quit qemu, reboot to an updated kernel, start an updated version of
>> qemu, and resume via the migrate-incoming command.  The caller must specify
>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>> the use of certain local storage options does not block the migration, but
>> the caller must not modify guest block devices between the quit and restart.
>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>> must be non-volatile across reboot, which can be achieved by backing it with
>> a dax device, or /dev/shm PKRAM as proposed in
>> https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$
>> but this is not enforced.  The restarted qemu arguments must match those used
>> to initially start qemu, plus the -incoming option.
>>
>> The reboot mode supports vfio devices if the caller first suspends the guest,
>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>> drivers' suspend methods flush outstanding requests and re-initialize the
>> devices, and thus there is no device state to save and restore.  After
>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>> resume.
>>
>> In cpr-exec mode, the migrate command saves state to a file and directly
>> exec's a new version of qemu on the same host, replacing the original process
>> while retaining its PID.  The caller must specify a migration URI that writes
>> to and reads from a file, and resumes execution via the migrate-incoming
>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>> migration parameter, and must include the -incoming option.
>>
>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>> ram is efficiently preserved in place, albeit with new virtual addresses.
>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>> causes secondary guest ram blocks (those not specified on the command line)
>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>> their values are saved in special cpr state which is retrieved after exec,
>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>> are not migrated, the caller must disable all capabilities related to page
>> and block copy.  The implementation ignores all related parameters.
>>
>> The exec mode supports vfio devices by preserving the vfio container, group,
>> device, and event descriptors across the qemu re-exec, and by updating DMA
>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>    https://urldefense.com/v3/__https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3IK0BrTn$
>> and integrated in Linux kernel 5.12.
>>
>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>> The software update is performed while the guest is running to minimize
>> downtime.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ...                         |
>>    -migrate-mode-enable cpr-exec                 |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update qemu
>> (qemu) migrate_set_parameter mode cpr-exec      |
>> (qemu) migrate_set_parameter cpr-exec-args      |
>>    qemu-system-$arch ... -incoming defer         |
>> (qemu) migrate -d file:/tmp/qemu.sav            |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>>
>> Here is an example of updating the host kernel using reboot mode.
>>
>> window 1                                        | window 2
>>                                                  |
>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>    -migrate-mode-enable cpr-reboot               |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                  | # yum update kernel-uek
>> (qemu) migrate_set_parameter mode cpr-reboot    |
>> (qemu) migrate -d file:/tmp/qemu.sav            |
>> (qemu) quit                                     |
>>                                                  |
>> # systemctl kexec                               |
>> kexec_core: Starting new kernel                 |
>> ...                                             |
>>                                                  |
>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>    -incoming defer                               |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>> Changes from V8 to V9:
>>    vfio:
>>      - free all cpr state during unwind in vfio_connect_container
>>      - change cpr_resave_fd to return void, and avoid new unwind cases
>>      - delete incorrect .unmigratable=1 in vmstate handlers
>>      - add route batching in vfio_claim_vectors
>>      - simplified vfio intx cpr code
>>      - fix commit message for 'recover from unmap-all-vaddr failure'
>>      - verify suspended runstate for cpr-reboot mode
>>    Other:
>>      - delete cpr-save, cpr-exec, cpr-load
>>      - delete ram block vmstate handlers that were added in V8
>>      - rename cpr-enable option to migrate-mode-enable
>>      - add file URI for migration
>>      - add mode and cpr-exec-args migration parameters
>>      - add per-mode migration blockers
>>      - add mode checks in migration notifiers
>>      - fix suspended runstate during migration
>>      - replace RAM_ANON flag with RAM_NAMED_FILE
>>      - support memory-backend-epc
>>
>> Steve Sistare (44):
>>    migration: fix populate_vfio_info                  ---  reboot mode  ---
>>    memory: RAM_NAMED_FILE flag
>>    migration: file URI
>>    migration: mode parameter
>>    migration: migrate-enable-mode option
>>    migration: simplify blockers
>>    migration: per-mode blockers
>>    cpr: relax some blockers
>>    cpr: reboot mode
>>
>>    qdev-properties: strList                           ---  exec mode ---
>>    qapi: strList_from_string
>>    qapi: QAPI_LIST_LENGTH
>>    qapi: strv_from_strList
>>    qapi: strList unit tests
>>    migration: cpr-exec-args parameter
>>    migration: simplify notifiers
>>    migration: check mode in notifiers
>>    memory: flat section iterator
>>    oslib: qemu_clear_cloexec
>>    vl: helper to request re-exec
>>    cpr: preserve extra state
>>    cpr: exec mode
>>    cpr: add exec-mode blockers
>>    cpr: ram block blockers
>>    cpr: only-cpr-capable
>>    cpr: Mismatched GPAs fix
>>    hostmem-memfd: cpr support
>>    hostmem-epc: cpr support
>>
>>    pci: export msix_is_pending                       --- vfio for exec ---
>>    vfio-pci: refactor for cpr
>>    vfio-pci: cpr part 1 (fd and dma)
>>    vfio-pci: cpr part 2 (msi)
>>    vfio-pci: cpr part 3 (intx)
>>    vfio-pci: recover from unmap-all-vaddr failure
>>
>>    chardev: cpr framework                            --- misc for exec ---
>>    chardev: cpr for simple devices
>>    chardev: cpr for pty
>>    python/machine: QEMUMachine full_args
>>    python/machine: QEMUMachine reopen_qmp_connection
>>    tests/avocado: add cpr regression test
>>
>>    vl: start on wakeup request                       --- vfio for reboot ---
>>    migration: fix suspended runstate
>>    migration: notifier error reporting
>>    vfio: allow cpr-reboot migration if suspended
>>
>> Mark Kanda, Steve Sistare (2):
>>    vhost: reset vhost devices for cpr
>>    chardev: cpr for sockets
>>
>>   MAINTAINERS                         |  14 ++
>>   accel/xen/xen-all.c                 |   3 +
>>   backends/hostmem-epc.c              |  18 +-
>>   backends/hostmem-file.c             |   1 +
>>   backends/hostmem-memfd.c            |  22 ++-
>>   backends/tpm/tpm_emulator.c         |  11 +-
>>   block/parallels.c                   |   7 +-
>>   block/qcow.c                        |   7 +-
>>   block/vdi.c                         |   7 +-
>>   block/vhdx.c                        |   7 +-
>>   block/vmdk.c                        |   7 +-
>>   block/vpc.c                         |   7 +-
>>   block/vvfat.c                       |   7 +-
>>   chardev/char-mux.c                  |   1 +
>>   chardev/char-null.c                 |   1 +
>>   chardev/char-pty.c                  |  16 +-
>>   chardev/char-serial.c               |   1 +
>>   chardev/char-socket.c               |  48 +++++
>>   chardev/char-stdio.c                |  31 +++
>>   chardev/char.c                      |  49 ++++-
>>   dump/dump.c                         |   4 +-
>>   gdbstub.c                           |   1 +
>>   hmp-commands.hx                     |   2 +-
>>   hw/9pfs/9p.c                        |  11 +-
>>   hw/core/qdev-properties-system.c    |  12 ++
>>   hw/core/qdev-properties.c           |  44 +++++
>>   hw/display/virtio-gpu-base.c        |   8 +-
>>   hw/intc/arm_gic_kvm.c               |   3 +-
>>   hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>   hw/intc/arm_gicv3_kvm.c             |   3 +-
>>   hw/misc/ivshmem.c                   |   8 +-
>>   hw/net/virtio-net.c                 |  10 +-
>>   hw/pci/msix.c                       |   2 +-
>>   hw/pci/pci.c                        |  12 ++
>>   hw/ppc/pef.c                        |   2 +-
>>   hw/ppc/spapr.c                      |   2 +-
>>   hw/ppc/spapr_events.c               |   2 +-
>>   hw/ppc/spapr_rtas.c                 |   2 +-
>>   hw/remote/proxy.c                   |   7 +-
>>   hw/s390x/s390-virtio-ccw.c          |   9 +-
>>   hw/scsi/vhost-scsi.c                |   9 +-
>>   hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>   hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>   hw/vfio/meson.build                 |   1 +
>>   hw/vfio/migration.c                 |  23 +--
>>   hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>   hw/vfio/trace-events                |   1 +
>>   hw/virtio/vhost-vdpa.c              |   6 +-
>>   hw/virtio/vhost.c                   |  32 +++-
>>   include/chardev/char-socket.h       |   1 +
>>   include/chardev/char.h              |   5 +
>>   include/exec/memory.h               |  48 +++++
>>   include/exec/ram_addr.h             |   1 +
>>   include/exec/ramblock.h             |   1 +
>>   include/hw/pci/msix.h               |   1 +
>>   include/hw/qdev-properties-system.h |   4 +
>>   include/hw/qdev-properties.h        |   3 +
>>   include/hw/vfio/vfio-common.h       |  12 ++
>>   include/hw/virtio/vhost.h           |   1 +
>>   include/migration/blocker.h         |  69 ++++++-
>>   include/migration/cpr-state.h       |  30 +++
>>   include/migration/cpr.h             |  20 ++
>>   include/migration/misc.h            |  13 +-
>>   include/migration/vmstate.h         |   2 +
>>   include/qapi/util.h                 |  28 +++
>>   include/qemu/osdep.h                |   9 +
>>   include/sysemu/runstate.h           |   2 +
>>   migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>   migration/cpr.c                     |  85 +++++++++
>>   migration/file.c                    |  62 ++++++
>>   migration/file.h                    |  14 ++
>>   migration/meson.build               |   3 +
>>   migration/migration.c               | 268 +++++++++++++++++++++++---
>>   migration/ram.c                     |  24 ++-
>>   migration/target.c                  |   1 +
>>   migration/trace-events              |  12 ++
>>   monitor/hmp-cmds.c                  |  59 +++---
>>   monitor/hmp.c                       |   3 +
>>   monitor/qmp.c                       |   4 +
>>   python/qemu/machine/machine.py      |  14 ++
>>   qapi/char.json                      |   7 +-
>>   qapi/migration.json                 |  68 ++++++-
>>   qapi/qapi-util.c                    |  37 ++++
>>   qemu-options.hx                     |  50 ++++-
>>   replay/replay.c                     |   4 +
>>   softmmu/memory.c                    |  31 ++-
>>   softmmu/physmem.c                   | 100 +++++++++-
>>   softmmu/runstate.c                  |  42 ++++-
>>   softmmu/vl.c                        |  10 +
>>   stubs/cpr-state.c                   |  26 +++
>>   stubs/meson.build                   |   2 +
>>   stubs/migr-blocker.c                |   9 +-
>>   stubs/migration.c                   |  33 ++++
>>   target/i386/kvm/kvm.c               |   8 +-
>>   target/i386/nvmm/nvmm-all.c         |   4 +-
>>   target/i386/sev.c                   |   2 +-
>>   target/i386/whpx/whpx-all.c         |   3 +-
>>   tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>   tests/unit/meson.build              |   1 +
>>   tests/unit/test-strlist.c           |  81 ++++++++
>>   trace-events                        |   1 +
>>   ui/spice-core.c                     |   5 +-
>>   ui/vdagent.c                        |   5 +-
>>   util/oslib-posix.c                  |   9 +
>>   util/oslib-win32.c                  |   4 +
>>   105 files changed, 2781 insertions(+), 330 deletions(-)
>>   create mode 100644 hw/vfio/cpr.c
>>   create mode 100644 include/migration/cpr-state.h
>>   create mode 100644 include/migration/cpr.h
>>   create mode 100644 migration/cpr-state.c
>>   create mode 100644 migration/cpr.c
>>   create mode 100644 migration/file.c
>>   create mode 100644 migration/file.h
>>   create mode 100644 stubs/cpr-state.c
>>   create mode 100644 stubs/migration.c
>>   create mode 100644 tests/avocado/cpr.py
>>   create mode 100644 tests/unit/test-strlist.c
>>

To make forward progress on this series and reduce its size, I will be posting
those of its patches that can be independently integrated and have some value
on their own, to a reduced distribution of reviewers for each.  This is what
I plan to break out:

migration: fix populate_vfio_info

memory: RAM_NAMED_FILE flag

memory: flat section iterator

oslib: qemu_clear_cloexec

migration: simplify blockers

migration: simplify notifiers

python/machine: QEMUMachine full_args

python/machine: QEMUMachine reopen_qmp_connection

qapi: strList_from_string
qapi: QAPI_LIST_LENGTH
qapi: strv_from_strList
qapi: strList unit tests

- Steve

On 12/7/2022 10:48 AM, Steven Sistare wrote:
> This series desperately needs review in its intersection with live migration.
> The code in other areas has been reviewed and revised multiple times -- thank you!
> 
> David, Juan, can you spare some time to review this?  I have done my best to order 
> the patches logically (see the labelled groups in this email), and to provide 
> complete and clear cover letter and commit messages. Can I do anything to facilitate,
> like doing a code walk through via zoom?
> 
> And of course, I welcome anyone's feedback.
> 
> Here is the original posting.
> 
> https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/
> 
> - Steve
> 
> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>> This version of the live update patch series integrates live update into the
>> live migration framework.  The new interfaces are:
>>   * mode (migration parameter)
>>   * cpr-exec-args (migration parameter)
>>   * file (migration URI)
>>   * migrate-mode-enable (command-line argument)
>>   * only-cpr-capable (command-line argument)
>>
>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>> save and restore VM state, with minimal guest pause time, so that qemu may be
>> updated to a new version in between.  The caller sets the mode parameter
>> before invoking the migrate or migrate-incoming commands.
>>
>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>> one to quit qemu, reboot to an updated kernel, start an updated version of
>> qemu, and resume via the migrate-incoming command.  The caller must specify
>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>> the use of certain local storage options does not block the migration, but
>> the caller must not modify guest block devices between the quit and restart.
>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>> must be non-volatile across reboot, which can be achieved by backing it with
>> a dax device, or /dev/shm PKRAM as proposed in
>> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com
>> but this is not enforced.  The restarted qemu arguments must match those used
>> to initially start qemu, plus the -incoming option.
>>
>> The reboot mode supports vfio devices if the caller first suspends the guest,
>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>> drivers' suspend methods flush outstanding requests and re-initialize the
>> devices, and thus there is no device state to save and restore.  After
>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>> resume.
>>
>> In cpr-exec mode, the migrate command saves state to a file and directly
>> exec's a new version of qemu on the same host, replacing the original process
>> while retaining its PID.  The caller must specify a migration URI that writes
>> to and reads from a file, and resumes execution via the migrate-incoming
>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>> migration parameter, and must include the -incoming option.
>>
>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>> ram is efficiently preserved in place, albeit with new virtual addresses.
>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>> causes secondary guest ram blocks (those not specified on the command line)
>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>> their values are saved in special cpr state which is retrieved after exec,
>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>> are not migrated, the caller must disable all capabilities related to page
>> and block copy.  The implementation ignores all related parameters.
>>
>> The exec mode supports vfio devices by preserving the vfio container, group,
>> device, and event descriptors across the qemu re-exec, and by updating DMA
>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>   https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com
>> and integrated in Linux kernel 5.12.
>>
>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>> The software update is performed while the guest is running to minimize
>> downtime.
>>
>> window 1                                        | window 2
>>                                                 |
>> # qemu-system-$arch ...                         |
>>   -migrate-mode-enable cpr-exec                 |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                 | # yum update qemu
>> (qemu) migrate_set_parameter mode cpr-exec      |
>> (qemu) migrate_set_parameter cpr-exec-args      |
>>   qemu-system-$arch ... -incoming defer         |
>> (qemu) migrate -d file:/tmp/qemu.sav            |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>>
>> Here is an example of updating the host kernel using reboot mode.
>>
>> window 1                                        | window 2
>>                                                 |
>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>   -migrate-mode-enable cpr-reboot               |
>> QEMU 7.0.50 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: running                              |
>>                                                 | # yum update kernel-uek
>> (qemu) migrate_set_parameter mode cpr-reboot    |
>> (qemu) migrate -d file:/tmp/qemu.sav            |
>> (qemu) quit                                     |
>>                                                 |
>> # systemctl kexec                               |
>> kexec_core: Starting new kernel                 |
>> ...                                             |
>>                                                 |
>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>   -incoming defer                               |
>> QEMU 7.0.51 monitor - type 'help' ...           |
>> (qemu) info status                              |
>> VM status: paused (inmigrate)                   |
>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>> (qemu) info status                              |
>> VM status: running                              |
>>
>> Changes from V8 to V9:
>>   vfio:
>>     - free all cpr state during unwind in vfio_connect_container
>>     - change cpr_resave_fd to return void, and avoid new unwind cases
>>     - delete incorrect .unmigratable=1 in vmstate handlers
>>     - add route batching in vfio_claim_vectors
>>     - simplified vfio intx cpr code
>>     - fix commit message for 'recover from unmap-all-vaddr failure'
>>     - verify suspended runstate for cpr-reboot mode
>>   Other:
>>     - delete cpr-save, cpr-exec, cpr-load
>>     - delete ram block vmstate handlers that were added in V8
>>     - rename cpr-enable option to migrate-mode-enable
>>     - add file URI for migration
>>     - add mode and cpr-exec-args migration parameters
>>     - add per-mode migration blockers
>>     - add mode checks in migration notifiers
>>     - fix suspended runstate during migration
>>     - replace RAM_ANON flag with RAM_NAMED_FILE
>>     - support memory-backend-epc
>>
>> Steve Sistare (44):
>>   migration: fix populate_vfio_info                  ---  reboot mode  ---
>>   memory: RAM_NAMED_FILE flag
>>   migration: file URI
>>   migration: mode parameter
>>   migration: migrate-enable-mode option
>>   migration: simplify blockers
>>   migration: per-mode blockers
>>   cpr: relax some blockers
>>   cpr: reboot mode
>>
>>   qdev-properties: strList                           ---  exec mode ---
>>   qapi: strList_from_string
>>   qapi: QAPI_LIST_LENGTH
>>   qapi: strv_from_strList
>>   qapi: strList unit tests
>>   migration: cpr-exec-args parameter
>>   migration: simplify notifiers
>>   migration: check mode in notifiers
>>   memory: flat section iterator
>>   oslib: qemu_clear_cloexec
>>   vl: helper to request re-exec
>>   cpr: preserve extra state
>>   cpr: exec mode
>>   cpr: add exec-mode blockers
>>   cpr: ram block blockers
>>   cpr: only-cpr-capable
>>   cpr: Mismatched GPAs fix
>>   hostmem-memfd: cpr support
>>   hostmem-epc: cpr support
>>
>>   pci: export msix_is_pending                       --- vfio for exec ---
>>   vfio-pci: refactor for cpr
>>   vfio-pci: cpr part 1 (fd and dma)
>>   vfio-pci: cpr part 2 (msi)
>>   vfio-pci: cpr part 3 (intx)
>>   vfio-pci: recover from unmap-all-vaddr failure
>>
>>   chardev: cpr framework                            --- misc for exec ---
>>   chardev: cpr for simple devices
>>   chardev: cpr for pty
>>   python/machine: QEMUMachine full_args
>>   python/machine: QEMUMachine reopen_qmp_connection
>>   tests/avocado: add cpr regression test
>>
>>   vl: start on wakeup request                       --- vfio for reboot ---
>>   migration: fix suspended runstate
>>   migration: notifier error reporting
>>   vfio: allow cpr-reboot migration if suspended
>>
>> Mark Kanda, Steve Sistare (2):
>>   vhost: reset vhost devices for cpr
>>   chardev: cpr for sockets
>>
>>  MAINTAINERS                         |  14 ++
>>  accel/xen/xen-all.c                 |   3 +
>>  backends/hostmem-epc.c              |  18 +-
>>  backends/hostmem-file.c             |   1 +
>>  backends/hostmem-memfd.c            |  22 ++-
>>  backends/tpm/tpm_emulator.c         |  11 +-
>>  block/parallels.c                   |   7 +-
>>  block/qcow.c                        |   7 +-
>>  block/vdi.c                         |   7 +-
>>  block/vhdx.c                        |   7 +-
>>  block/vmdk.c                        |   7 +-
>>  block/vpc.c                         |   7 +-
>>  block/vvfat.c                       |   7 +-
>>  chardev/char-mux.c                  |   1 +
>>  chardev/char-null.c                 |   1 +
>>  chardev/char-pty.c                  |  16 +-
>>  chardev/char-serial.c               |   1 +
>>  chardev/char-socket.c               |  48 +++++
>>  chardev/char-stdio.c                |  31 +++
>>  chardev/char.c                      |  49 ++++-
>>  dump/dump.c                         |   4 +-
>>  gdbstub.c                           |   1 +
>>  hmp-commands.hx                     |   2 +-
>>  hw/9pfs/9p.c                        |  11 +-
>>  hw/core/qdev-properties-system.c    |  12 ++
>>  hw/core/qdev-properties.c           |  44 +++++
>>  hw/display/virtio-gpu-base.c        |   8 +-
>>  hw/intc/arm_gic_kvm.c               |   3 +-
>>  hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>  hw/intc/arm_gicv3_kvm.c             |   3 +-
>>  hw/misc/ivshmem.c                   |   8 +-
>>  hw/net/virtio-net.c                 |  10 +-
>>  hw/pci/msix.c                       |   2 +-
>>  hw/pci/pci.c                        |  12 ++
>>  hw/ppc/pef.c                        |   2 +-
>>  hw/ppc/spapr.c                      |   2 +-
>>  hw/ppc/spapr_events.c               |   2 +-
>>  hw/ppc/spapr_rtas.c                 |   2 +-
>>  hw/remote/proxy.c                   |   7 +-
>>  hw/s390x/s390-virtio-ccw.c          |   9 +-
>>  hw/scsi/vhost-scsi.c                |   9 +-
>>  hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>  hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>  hw/vfio/meson.build                 |   1 +
>>  hw/vfio/migration.c                 |  23 +--
>>  hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>  hw/vfio/trace-events                |   1 +
>>  hw/virtio/vhost-vdpa.c              |   6 +-
>>  hw/virtio/vhost.c                   |  32 +++-
>>  include/chardev/char-socket.h       |   1 +
>>  include/chardev/char.h              |   5 +
>>  include/exec/memory.h               |  48 +++++
>>  include/exec/ram_addr.h             |   1 +
>>  include/exec/ramblock.h             |   1 +
>>  include/hw/pci/msix.h               |   1 +
>>  include/hw/qdev-properties-system.h |   4 +
>>  include/hw/qdev-properties.h        |   3 +
>>  include/hw/vfio/vfio-common.h       |  12 ++
>>  include/hw/virtio/vhost.h           |   1 +
>>  include/migration/blocker.h         |  69 ++++++-
>>  include/migration/cpr-state.h       |  30 +++
>>  include/migration/cpr.h             |  20 ++
>>  include/migration/misc.h            |  13 +-
>>  include/migration/vmstate.h         |   2 +
>>  include/qapi/util.h                 |  28 +++
>>  include/qemu/osdep.h                |   9 +
>>  include/sysemu/runstate.h           |   2 +
>>  migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>  migration/cpr.c                     |  85 +++++++++
>>  migration/file.c                    |  62 ++++++
>>  migration/file.h                    |  14 ++
>>  migration/meson.build               |   3 +
>>  migration/migration.c               | 268 +++++++++++++++++++++++---
>>  migration/ram.c                     |  24 ++-
>>  migration/target.c                  |   1 +
>>  migration/trace-events              |  12 ++
>>  monitor/hmp-cmds.c                  |  59 +++---
>>  monitor/hmp.c                       |   3 +
>>  monitor/qmp.c                       |   4 +
>>  python/qemu/machine/machine.py      |  14 ++
>>  qapi/char.json                      |   7 +-
>>  qapi/migration.json                 |  68 ++++++-
>>  qapi/qapi-util.c                    |  37 ++++
>>  qemu-options.hx                     |  50 ++++-
>>  replay/replay.c                     |   4 +
>>  softmmu/memory.c                    |  31 ++-
>>  softmmu/physmem.c                   | 100 +++++++++-
>>  softmmu/runstate.c                  |  42 ++++-
>>  softmmu/vl.c                        |  10 +
>>  stubs/cpr-state.c                   |  26 +++
>>  stubs/meson.build                   |   2 +
>>  stubs/migr-blocker.c                |   9 +-
>>  stubs/migration.c                   |  33 ++++
>>  target/i386/kvm/kvm.c               |   8 +-
>>  target/i386/nvmm/nvmm-all.c         |   4 +-
>>  target/i386/sev.c                   |   2 +-
>>  target/i386/whpx/whpx-all.c         |   3 +-
>>  tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>  tests/unit/meson.build              |   1 +
>>  tests/unit/test-strlist.c           |  81 ++++++++
>>  trace-events                        |   1 +
>>  ui/spice-core.c                     |   5 +-
>>  ui/vdagent.c                        |   5 +-
>>  util/oslib-posix.c                  |   9 +
>>  util/oslib-win32.c                  |   4 +
>>  105 files changed, 2781 insertions(+), 330 deletions(-)
>>  create mode 100644 hw/vfio/cpr.c
>>  create mode 100644 include/migration/cpr-state.h
>>  create mode 100644 include/migration/cpr.h
>>  create mode 100644 migration/cpr-state.c
>>  create mode 100644 migration/cpr.c
>>  create mode 100644 migration/file.c
>>  create mode 100644 migration/file.h
>>  create mode 100644 stubs/cpr-state.c
>>  create mode 100644 stubs/migration.c
>>  create mode 100644 tests/avocado/cpr.py
>>  create mode 100644 tests/unit/test-strlist.c
>>

Hi Steve,

On 7/2/23 19:44, Steven Sistare wrote:
> To make forward progress on this series and reduce its size, I will be posting
> those of its patches that can be independently integrated and have some value
> on their own, to a reduced distribution of reviewers for each.  This is what
> I plan to break out:
> 
> migration: fix populate_vfio_info
> 
> memory: RAM_NAMED_FILE flag
> 
> memory: flat section iterator
> 
> oslib: qemu_clear_cloexec
> 
> migration: simplify blockers
> 
> migration: simplify notifiers
> 
> python/machine: QEMUMachine full_args
> 
> python/machine: QEMUMachine reopen_qmp_connection
> 
> qapi: strList_from_string
> qapi: QAPI_LIST_LENGTH
> qapi: strv_from_strList
> qapi: strList unit tests

The break out looks good, but I guess it is a bit obsolete (as of
today), you probably already re-posted some of these patches as
separate series. Do you mind replying with a "ping" to the posted
ones which still applies and need to be reviewed?

Thanks,

Phil.

> - Steve
> 
> On 12/7/2022 10:48 AM, Steven Sistare wrote:
>> This series desperately needs review in its intersection with live migration.
>> The code in other areas has been reviewed and revised multiple times -- thank you!

On 5/30/2023 9:38 AM, Philippe Mathieu-Daudé wrote:
> Hi Steve,
> 
> On 7/2/23 19:44, Steven Sistare wrote:
>> To make forward progress on this series and reduce its size, I will be posting
>> those of its patches that can be independently integrated and have some value
>> on their own, to a reduced distribution of reviewers for each.  This is what
>> I plan to break out:
>>
>> migration: fix populate_vfio_info
>>
>> memory: RAM_NAMED_FILE flag
>>
>> memory: flat section iterator
>>
>> oslib: qemu_clear_cloexec
>>
>> migration: simplify blockers
>>
>> migration: simplify notifiers
>>
>> python/machine: QEMUMachine full_args
>>
>> python/machine: QEMUMachine reopen_qmp_connection
>>
>> qapi: strList_from_string
>> qapi: QAPI_LIST_LENGTH
>> qapi: strv_from_strList
>> qapi: strList unit tests
> 
> The break out looks good, but I guess it is a bit obsolete (as of
> today), you probably already re-posted some of these patches as
> separate series. Do you mind replying with a "ping" to the posted
> ones which still applies and need to be reviewed?

Yes, I re-posted this list of patches, some of which have been accepted
or are close.  I will reply to the threads that are still open and cc you, 
but I will first verify that they still apply cleanly, and update them if not.
Thanks!

- Steve

and of course I immediately screwed up and forgot to delete series numbers from
the email titles.  I will resend.

- Steve

On 2/7/2023 1:44 PM, Steven Sistare wrote:
> To make forward progress on this series and reduce its size, I will be posting
> those of its patches that can be independently integrated and have some value
> on their own, to a reduced distribution of reviewers for each.  This is what
> I plan to break out:
> 
> migration: fix populate_vfio_info
> 
> memory: RAM_NAMED_FILE flag
> 
> memory: flat section iterator
> 
> oslib: qemu_clear_cloexec
> 
> migration: simplify blockers
> 
> migration: simplify notifiers
> 
> python/machine: QEMUMachine full_args
> 
> python/machine: QEMUMachine reopen_qmp_connection
> 
> qapi: strList_from_string
> qapi: QAPI_LIST_LENGTH
> qapi: strv_from_strList
> qapi: strList unit tests
> 
> - Steve
> 
> On 12/7/2022 10:48 AM, Steven Sistare wrote:
>> This series desperately needs review in its intersection with live migration.
>> The code in other areas has been reviewed and revised multiple times -- thank you!
>>
>> David, Juan, can you spare some time to review this?  I have done my best to order 
>> the patches logically (see the labelled groups in this email), and to provide 
>> complete and clear cover letter and commit messages. Can I do anything to facilitate,
>> like doing a code walk through via zoom?
>>
>> And of course, I welcome anyone's feedback.
>>
>> Here is the original posting.
>>
>> https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/
>>
>> - Steve
>>
>> On 7/26/2022 12:09 PM, Steve Sistare wrote:
>>> This version of the live update patch series integrates live update into the
>>> live migration framework.  The new interfaces are:
>>>   * mode (migration parameter)
>>>   * cpr-exec-args (migration parameter)
>>>   * file (migration URI)
>>>   * migrate-mode-enable (command-line argument)
>>>   * only-cpr-capable (command-line argument)
>>>
>>> Provide the cpr-exec and cpr-reboot migration modes for live update.  These
>>> save and restore VM state, with minimal guest pause time, so that qemu may be
>>> updated to a new version in between.  The caller sets the mode parameter
>>> before invoking the migrate or migrate-incoming commands.
>>>
>>> In cpr-reboot mode, the migrate command saves state to a file, allowing
>>> one to quit qemu, reboot to an updated kernel, start an updated version of
>>> qemu, and resume via the migrate-incoming command.  The caller must specify
>>> a migration URI that writes to and reads from a file.  Unlike normal mode,
>>> the use of certain local storage options does not block the migration, but
>>> the caller must not modify guest block devices between the quit and restart.
>>> The guest RAM memory-backend must be shared, and the @x-ignore-shared
>>> migration capability must be set, to avoid saving it to the file.  Guest RAM
>>> must be non-volatile across reboot, which can be achieved by backing it with
>>> a dax device, or /dev/shm PKRAM as proposed in
>>> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com
>>> but this is not enforced.  The restarted qemu arguments must match those used
>>> to initially start qemu, plus the -incoming option.
>>>
>>> The reboot mode supports vfio devices if the caller first suspends the guest,
>>> such as by issuing guest-suspend-ram to the qemu guest agent.  The guest
>>> drivers' suspend methods flush outstanding requests and re-initialize the
>>> devices, and thus there is no device state to save and restore.  After
>>> issuing migrate-incoming, the caller must issue a system_wakeup command to
>>> resume.
>>>
>>> In cpr-exec mode, the migrate command saves state to a file and directly
>>> exec's a new version of qemu on the same host, replacing the original process
>>> while retaining its PID.  The caller must specify a migration URI that writes
>>> to and reads from a file, and resumes execution via the migrate-incoming
>>> command.  Arguments for the new qemu process are taken from the cpr-exec-args
>>> migration parameter, and must include the -incoming option.
>>>
>>> Guest RAM must be backed by a memory backend with share=on, but cannot be
>>> memory-backend-ram.  The memory is re-mmap'd in the updated process, so guest
>>> ram is efficiently preserved in place, albeit with new virtual addresses.
>>> In addition, the '-migrate-mode-enable cpr-exec' option is required.  This
>>> causes secondary guest ram blocks (those not specified on the command line)
>>> to be allocated by mmap'ing a memfd.  The memfds are kept open across exec,
>>> their values are saved in special cpr state which is retrieved after exec,
>>> and they are re-mmap'd.  Since guest RAM is not copied, and storage blocks
>>> are not migrated, the caller must disable all capabilities related to page
>>> and block copy.  The implementation ignores all related parameters.
>>>
>>> The exec mode supports vfio devices by preserving the vfio container, group,
>>> device, and event descriptors across the qemu re-exec, and by updating DMA
>>> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and
>>> VFIO_DMA_MAP_FLAG_VADDR as defined in
>>>   https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sistare@oracle.com
>>> and integrated in Linux kernel 5.12.
>>>
>>> Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode.
>>> The software update is performed while the guest is running to minimize
>>> downtime.
>>>
>>> window 1                                        | window 2
>>>                                                 |
>>> # qemu-system-$arch ...                         |
>>>   -migrate-mode-enable cpr-exec                 |
>>> QEMU 7.0.50 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>                                                 | # yum update qemu
>>> (qemu) migrate_set_parameter mode cpr-exec      |
>>> (qemu) migrate_set_parameter cpr-exec-args      |
>>>   qemu-system-$arch ... -incoming defer         |
>>> (qemu) migrate -d file:/tmp/qemu.sav            |
>>> QEMU 7.0.51 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: paused (inmigrate)                   |
>>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>
>>>
>>> Here is an example of updating the host kernel using reboot mode.
>>>
>>> window 1                                        | window 2
>>>                                                 |
>>> # qemu-system-$arch ... mem-path=/dev/dax0.0    |
>>>   -migrate-mode-enable cpr-reboot               |
>>> QEMU 7.0.50 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>                                                 | # yum update kernel-uek
>>> (qemu) migrate_set_parameter mode cpr-reboot    |
>>> (qemu) migrate -d file:/tmp/qemu.sav            |
>>> (qemu) quit                                     |
>>>                                                 |
>>> # systemctl kexec                               |
>>> kexec_core: Starting new kernel                 |
>>> ...                                             |
>>>                                                 |
>>> # qemu-system-$arch mem-path=/dev/dax0.0 ...    |
>>>   -incoming defer                               |
>>> QEMU 7.0.51 monitor - type 'help' ...           |
>>> (qemu) info status                              |
>>> VM status: paused (inmigrate)                   |
>>> (qemu) migrate_incoming file:/tmp/qemu.sav      |
>>> (qemu) info status                              |
>>> VM status: running                              |
>>>
>>> Changes from V8 to V9:
>>>   vfio:
>>>     - free all cpr state during unwind in vfio_connect_container
>>>     - change cpr_resave_fd to return void, and avoid new unwind cases
>>>     - delete incorrect .unmigratable=1 in vmstate handlers
>>>     - add route batching in vfio_claim_vectors
>>>     - simplified vfio intx cpr code
>>>     - fix commit message for 'recover from unmap-all-vaddr failure'
>>>     - verify suspended runstate for cpr-reboot mode
>>>   Other:
>>>     - delete cpr-save, cpr-exec, cpr-load
>>>     - delete ram block vmstate handlers that were added in V8
>>>     - rename cpr-enable option to migrate-mode-enable
>>>     - add file URI for migration
>>>     - add mode and cpr-exec-args migration parameters
>>>     - add per-mode migration blockers
>>>     - add mode checks in migration notifiers
>>>     - fix suspended runstate during migration
>>>     - replace RAM_ANON flag with RAM_NAMED_FILE
>>>     - support memory-backend-epc
>>>
>>> Steve Sistare (44):
>>>   migration: fix populate_vfio_info                  ---  reboot mode  ---
>>>   memory: RAM_NAMED_FILE flag
>>>   migration: file URI
>>>   migration: mode parameter
>>>   migration: migrate-enable-mode option
>>>   migration: simplify blockers
>>>   migration: per-mode blockers
>>>   cpr: relax some blockers
>>>   cpr: reboot mode
>>>
>>>   qdev-properties: strList                           ---  exec mode ---
>>>   qapi: strList_from_string
>>>   qapi: QAPI_LIST_LENGTH
>>>   qapi: strv_from_strList
>>>   qapi: strList unit tests
>>>   migration: cpr-exec-args parameter
>>>   migration: simplify notifiers
>>>   migration: check mode in notifiers
>>>   memory: flat section iterator
>>>   oslib: qemu_clear_cloexec
>>>   vl: helper to request re-exec
>>>   cpr: preserve extra state
>>>   cpr: exec mode
>>>   cpr: add exec-mode blockers
>>>   cpr: ram block blockers
>>>   cpr: only-cpr-capable
>>>   cpr: Mismatched GPAs fix
>>>   hostmem-memfd: cpr support
>>>   hostmem-epc: cpr support
>>>
>>>   pci: export msix_is_pending                       --- vfio for exec ---
>>>   vfio-pci: refactor for cpr
>>>   vfio-pci: cpr part 1 (fd and dma)
>>>   vfio-pci: cpr part 2 (msi)
>>>   vfio-pci: cpr part 3 (intx)
>>>   vfio-pci: recover from unmap-all-vaddr failure
>>>
>>>   chardev: cpr framework                            --- misc for exec ---
>>>   chardev: cpr for simple devices
>>>   chardev: cpr for pty
>>>   python/machine: QEMUMachine full_args
>>>   python/machine: QEMUMachine reopen_qmp_connection
>>>   tests/avocado: add cpr regression test
>>>
>>>   vl: start on wakeup request                       --- vfio for reboot ---
>>>   migration: fix suspended runstate
>>>   migration: notifier error reporting
>>>   vfio: allow cpr-reboot migration if suspended
>>>
>>> Mark Kanda, Steve Sistare (2):
>>>   vhost: reset vhost devices for cpr
>>>   chardev: cpr for sockets
>>>
>>>  MAINTAINERS                         |  14 ++
>>>  accel/xen/xen-all.c                 |   3 +
>>>  backends/hostmem-epc.c              |  18 +-
>>>  backends/hostmem-file.c             |   1 +
>>>  backends/hostmem-memfd.c            |  22 ++-
>>>  backends/tpm/tpm_emulator.c         |  11 +-
>>>  block/parallels.c                   |   7 +-
>>>  block/qcow.c                        |   7 +-
>>>  block/vdi.c                         |   7 +-
>>>  block/vhdx.c                        |   7 +-
>>>  block/vmdk.c                        |   7 +-
>>>  block/vpc.c                         |   7 +-
>>>  block/vvfat.c                       |   7 +-
>>>  chardev/char-mux.c                  |   1 +
>>>  chardev/char-null.c                 |   1 +
>>>  chardev/char-pty.c                  |  16 +-
>>>  chardev/char-serial.c               |   1 +
>>>  chardev/char-socket.c               |  48 +++++
>>>  chardev/char-stdio.c                |  31 +++
>>>  chardev/char.c                      |  49 ++++-
>>>  dump/dump.c                         |   4 +-
>>>  gdbstub.c                           |   1 +
>>>  hmp-commands.hx                     |   2 +-
>>>  hw/9pfs/9p.c                        |  11 +-
>>>  hw/core/qdev-properties-system.c    |  12 ++
>>>  hw/core/qdev-properties.c           |  44 +++++
>>>  hw/display/virtio-gpu-base.c        |   8 +-
>>>  hw/intc/arm_gic_kvm.c               |   3 +-
>>>  hw/intc/arm_gicv3_its_kvm.c         |   3 +-
>>>  hw/intc/arm_gicv3_kvm.c             |   3 +-
>>>  hw/misc/ivshmem.c                   |   8 +-
>>>  hw/net/virtio-net.c                 |  10 +-
>>>  hw/pci/msix.c                       |   2 +-
>>>  hw/pci/pci.c                        |  12 ++
>>>  hw/ppc/pef.c                        |   2 +-
>>>  hw/ppc/spapr.c                      |   2 +-
>>>  hw/ppc/spapr_events.c               |   2 +-
>>>  hw/ppc/spapr_rtas.c                 |   2 +-
>>>  hw/remote/proxy.c                   |   7 +-
>>>  hw/s390x/s390-virtio-ccw.c          |   9 +-
>>>  hw/scsi/vhost-scsi.c                |   9 +-
>>>  hw/vfio/common.c                    | 235 +++++++++++++++++++----
>>>  hw/vfio/cpr.c                       | 177 ++++++++++++++++++
>>>  hw/vfio/meson.build                 |   1 +
>>>  hw/vfio/migration.c                 |  23 +--
>>>  hw/vfio/pci.c                       | 336 ++++++++++++++++++++++++++++-----
>>>  hw/vfio/trace-events                |   1 +
>>>  hw/virtio/vhost-vdpa.c              |   6 +-
>>>  hw/virtio/vhost.c                   |  32 +++-
>>>  include/chardev/char-socket.h       |   1 +
>>>  include/chardev/char.h              |   5 +
>>>  include/exec/memory.h               |  48 +++++
>>>  include/exec/ram_addr.h             |   1 +
>>>  include/exec/ramblock.h             |   1 +
>>>  include/hw/pci/msix.h               |   1 +
>>>  include/hw/qdev-properties-system.h |   4 +
>>>  include/hw/qdev-properties.h        |   3 +
>>>  include/hw/vfio/vfio-common.h       |  12 ++
>>>  include/hw/virtio/vhost.h           |   1 +
>>>  include/migration/blocker.h         |  69 ++++++-
>>>  include/migration/cpr-state.h       |  30 +++
>>>  include/migration/cpr.h             |  20 ++
>>>  include/migration/misc.h            |  13 +-
>>>  include/migration/vmstate.h         |   2 +
>>>  include/qapi/util.h                 |  28 +++
>>>  include/qemu/osdep.h                |   9 +
>>>  include/sysemu/runstate.h           |   2 +
>>>  migration/cpr-state.c               | 362 ++++++++++++++++++++++++++++++++++++
>>>  migration/cpr.c                     |  85 +++++++++
>>>  migration/file.c                    |  62 ++++++
>>>  migration/file.h                    |  14 ++
>>>  migration/meson.build               |   3 +
>>>  migration/migration.c               | 268 +++++++++++++++++++++++---
>>>  migration/ram.c                     |  24 ++-
>>>  migration/target.c                  |   1 +
>>>  migration/trace-events              |  12 ++
>>>  monitor/hmp-cmds.c                  |  59 +++---
>>>  monitor/hmp.c                       |   3 +
>>>  monitor/qmp.c                       |   4 +
>>>  python/qemu/machine/machine.py      |  14 ++
>>>  qapi/char.json                      |   7 +-
>>>  qapi/migration.json                 |  68 ++++++-
>>>  qapi/qapi-util.c                    |  37 ++++
>>>  qemu-options.hx                     |  50 ++++-
>>>  replay/replay.c                     |   4 +
>>>  softmmu/memory.c                    |  31 ++-
>>>  softmmu/physmem.c                   | 100 +++++++++-
>>>  softmmu/runstate.c                  |  42 ++++-
>>>  softmmu/vl.c                        |  10 +
>>>  stubs/cpr-state.c                   |  26 +++
>>>  stubs/meson.build                   |   2 +
>>>  stubs/migr-blocker.c                |   9 +-
>>>  stubs/migration.c                   |  33 ++++
>>>  target/i386/kvm/kvm.c               |   8 +-
>>>  target/i386/nvmm/nvmm-all.c         |   4 +-
>>>  target/i386/sev.c                   |   2 +-
>>>  target/i386/whpx/whpx-all.c         |   3 +-
>>>  tests/avocado/cpr.py                | 176 ++++++++++++++++++
>>>  tests/unit/meson.build              |   1 +
>>>  tests/unit/test-strlist.c           |  81 ++++++++
>>>  trace-events                        |   1 +
>>>  ui/spice-core.c                     |   5 +-
>>>  ui/vdagent.c                        |   5 +-
>>>  util/oslib-posix.c                  |   9 +
>>>  util/oslib-win32.c                  |   4 +
>>>  105 files changed, 2781 insertions(+), 330 deletions(-)
>>>  create mode 100644 hw/vfio/cpr.c
>>>  create mode 100644 include/migration/cpr-state.h
>>>  create mode 100644 include/migration/cpr.h
>>>  create mode 100644 migration/cpr-state.c
>>>  create mode 100644 migration/cpr.c
>>>  create mode 100644 migration/file.c
>>>  create mode 100644 migration/file.h
>>>  create mode 100644 stubs/cpr-state.c
>>>  create mode 100644 stubs/migration.c
>>>  create mode 100644 tests/avocado/cpr.py
>>>  create mode 100644 tests/unit/test-strlist.c
>>>