hw/ppc/spapr.c | 158 +++++++++++++++++++++++++++++++++++++++++--- hw/ppc/spapr_drc.c | 97 ++++++++++++++++++++++----- hw/ppc/spapr_events.c | 24 ++++--- hw/ppc/spapr_pci.c | 5 +- include/hw/pci-host/spapr.h | 3 + include/hw/ppc/spapr.h | 24 ++++++- include/hw/ppc/spapr_drc.h | 8 +-- 7 files changed, 273 insertions(+), 46 deletions(-)
v9:
- patch 1 (*new*): added a qtail in sPAPRMachineState called pending_dimm_unplugs
that stores the DIMM LMB state during the unplug process.
- patch 2 (*new*): merged v8-patch1 and v8-patch2: removing detach_cb and
detach_cb_opaque.
- patch 3:
* removed dk->vmsd entry. We're using vmstate_register instead
* added 'awaiting_allocation' flag in the DRC migration
- patch 4 (*new*): migrating spapr->pending_dimm_unplugs qtailq to allow
for an ongoing PCDIMM unplug to continue after a migration.
v8:
- new patch added: 'removing spapr_drc_detach_cb opaques'. This new patch removes
the need for the detach_cb_opaques inside the removal callback functions. See
the commit message of the patch for more info.
v7:
- removed the first patch. DRC registration is now done by vmstate_register
in patch 2.
- added a new patch that changes spapr_dr_connector_new to receive as argument
the detach_cb.
- removed the callback logic of patch 2 since there is no need to restore the
detach_cb on post-load due to the detach_cb on spapr_dr_connector_new change.
- added word separators in the VMSD names of patch 3 and 4.
v6: - Rebased with QEMU master after 6+ months.
- Simplified the logic in patch 1.
- Reworked patch 2: added CPU DRC migration, removed a function pointer from DRC
class and minor improvements.
- Added clarifications from the previous v5 discussions in the commit messages.
v5: - Rebased to David's ppc-for-2.8.
v4: - Introduce a way to set customized instance_id in SaveStateEntry. Use it
to set instance_id for DRC using its unique index to address David
Gibson's concern.
- Rename VMS_CSTM to VMS_LINKED based on Paolo Bonzini's suggestions.
- Clean up qjson stuff in put_qtailq.
- Add trace for put_qtailq and get_qtailq based on David Gilbert's
suggestion.
- Based on David's ppc-for-2.7.
v3: - Simplify overall design followng discussion with Paolo. No longer need
metadata to migrate QTAILQ.
- Extend VMStateInfo instead of adding similar fields to VMStateField.
- Clean up macros in qemu/queue.h.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg05695.html)
v2: - Introduce a general approach to migrate QTAILQ in qemu/queue.h.
- Migrate signalled field in the DRC state.
- Put the newly added migrating fields in subsections so that backward
migration is not broken.
- Set detach_cb field right after migration so that a migrated hot-unplug
event could finish its course.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg04188.html)
v1: - Inital version.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg02601.html)
To make guest devices (PCI, CPU and memory) hotplug work together
with guest migration, spapr drc state needs be transmitted in
migration. This patch defines the VMStateDescription struct for
spapr drc state to enable it.
To fix the potential racing between hotplug events on guest and
guest migration, ccs_list and pending_events of spapr state need be
transmitted in migration. This patch set also takes care of it.
Daniel Henrique Barboza (4):
hw/ppc/spapr.c: adding pending_dimm_unplugs to sPAPRMachineState
hw/ppc: removing drc->detach_cb and drc->detach_cb_opaque
hw/ppc: migrating the DRC state of hotplugged devices
hw/ppc/spapr.c: migrate pending_dimm_unplugs of spapr state
Jianjun Duan (2):
migration: spapr: migrate ccs_list in spapr state
migration: spapr: migrate pending_events of spapr state
hw/ppc/spapr.c | 158 +++++++++++++++++++++++++++++++++++++++++---
hw/ppc/spapr_drc.c | 97 ++++++++++++++++++++++-----
hw/ppc/spapr_events.c | 24 ++++---
hw/ppc/spapr_pci.c | 5 +-
include/hw/pci-host/spapr.h | 3 +
include/hw/ppc/spapr.h | 24 ++++++-
include/hw/ppc/spapr_drc.h | 8 +--
7 files changed, 273 insertions(+), 46 deletions(-)
--
2.9.3
Hi Daniel, I'd like to test your patch series: is there a way to generate easily the problem your series wants to fix? Thanks, Laurent On 05/05/2017 22:47, Daniel Henrique Barboza wrote: > v9: > - patch 1 (*new*): added a qtail in sPAPRMachineState called pending_dimm_unplugs > that stores the DIMM LMB state during the unplug process. > - patch 2 (*new*): merged v8-patch1 and v8-patch2: removing detach_cb and > detach_cb_opaque. > - patch 3: > * removed dk->vmsd entry. We're using vmstate_register instead > * added 'awaiting_allocation' flag in the DRC migration > - patch 4 (*new*): migrating spapr->pending_dimm_unplugs qtailq to allow > for an ongoing PCDIMM unplug to continue after a migration. > > v8: > - new patch added: 'removing spapr_drc_detach_cb opaques'. This new patch removes > the need for the detach_cb_opaques inside the removal callback functions. See > the commit message of the patch for more info. > > v7: > - removed the first patch. DRC registration is now done by vmstate_register > in patch 2. > - added a new patch that changes spapr_dr_connector_new to receive as argument > the detach_cb. > - removed the callback logic of patch 2 since there is no need to restore the > detach_cb on post-load due to the detach_cb on spapr_dr_connector_new change. > - added word separators in the VMSD names of patch 3 and 4. > > v6: - Rebased with QEMU master after 6+ months. > - Simplified the logic in patch 1. > - Reworked patch 2: added CPU DRC migration, removed a function pointer from DRC > class and minor improvements. > - Added clarifications from the previous v5 discussions in the commit messages. > > v5: - Rebased to David's ppc-for-2.8. > > v4: - Introduce a way to set customized instance_id in SaveStateEntry. Use it > to set instance_id for DRC using its unique index to address David > Gibson's concern. > - Rename VMS_CSTM to VMS_LINKED based on Paolo Bonzini's suggestions. > - Clean up qjson stuff in put_qtailq. > - Add trace for put_qtailq and get_qtailq based on David Gilbert's > suggestion. > > - Based on David's ppc-for-2.7. > > v3: - Simplify overall design followng discussion with Paolo. No longer need > metadata to migrate QTAILQ. > - Extend VMStateInfo instead of adding similar fields to VMStateField. > - Clean up macros in qemu/queue.h. > (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg05695.html) > > v2: - Introduce a general approach to migrate QTAILQ in qemu/queue.h. > - Migrate signalled field in the DRC state. > - Put the newly added migrating fields in subsections so that backward > migration is not broken. > - Set detach_cb field right after migration so that a migrated hot-unplug > event could finish its course. > (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg04188.html) > > v1: - Inital version. > (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg02601.html) > > > To make guest devices (PCI, CPU and memory) hotplug work together > with guest migration, spapr drc state needs be transmitted in > migration. This patch defines the VMStateDescription struct for > spapr drc state to enable it. > > To fix the potential racing between hotplug events on guest and > guest migration, ccs_list and pending_events of spapr state need be > transmitted in migration. This patch set also takes care of it. > > > > Daniel Henrique Barboza (4): > hw/ppc/spapr.c: adding pending_dimm_unplugs to sPAPRMachineState > hw/ppc: removing drc->detach_cb and drc->detach_cb_opaque > hw/ppc: migrating the DRC state of hotplugged devices > hw/ppc/spapr.c: migrate pending_dimm_unplugs of spapr state > > Jianjun Duan (2): > migration: spapr: migrate ccs_list in spapr state > migration: spapr: migrate pending_events of spapr state > > hw/ppc/spapr.c | 158 +++++++++++++++++++++++++++++++++++++++++--- > hw/ppc/spapr_drc.c | 97 ++++++++++++++++++++++----- > hw/ppc/spapr_events.c | 24 ++++--- > hw/ppc/spapr_pci.c | 5 +- > include/hw/pci-host/spapr.h | 3 + > include/hw/ppc/spapr.h | 24 ++++++- > include/hw/ppc/spapr_drc.h | 8 +-- > 7 files changed, 273 insertions(+), 46 deletions(-) >
Hi Laurent, To easily reproduce the error scenario: 1 - start QEMU in both source and target host with the same devices (target qemu with the extra --incoming to accept the migration) 2 - in both QEMU instances add one or more cpus with device add: device_add host-spapr-cpu-core,id=core1,core-id=1 3 - execute the migration 4 - after the migration is completed try to detach the cpu(s) you added in the target QEMU: device_del core1 There will be no errors thrown in the QEMU console but you'll notice in the VM that the cpus will still be there. dmesg will show something like this: [ +0.482307] cgroup: new mount options do not match the existing superblock, will be ignored [Apr19 14:47] random: crng init done [Apr19 14:51] pseries-hotplug-cpu: CPU with drc index 10000008 already exists [ +0.020319] cpu 1 (hwid 8) Ready to die... [ +0.020550] pseries-hotplug-cpu: Failed to release drc (10000008) for CPU <NULL>, rc: -1 [Apr19 14:56] cpu 1 (hwid 8) Ready to die... [ +0.014549] pseries-hotplug-cpu: Failed to release drc (10000008) for CPU <NULL>, rc: -1 The bug was originally reproduced using virsh: # virsh setvcpus avocado-vt-vm1-migration 64 --live # virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://<target_ip>/system --timeout 60 # virsh -c 'qemu+ssh://<target_ip>/system' setvcpus avocado-vt-vm1-migration 8 --live error: operation failed: vcpu unplug request timed out This happens because virsh is hot plugging the CPUs in both source and target QEMUs and then migrating the VM, instead of starting the target QEMU with the extra CPUs. When trying to hot unplug the CPUs after the migration, it fails because these hot plugged CPUs at the target didn't pass the DRC state changes that happened in the source. This patch series aims to solve this scenario by migrating the relevant DRC states to allow for proper device removal after a migration. You can see more info in this launchpad bug: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1677552 Thanks, Daniel On 05/11/2017 03:38 PM, Laurent Vivier wrote: > Hi Daniel, > > I'd like to test your patch series: is there a way to generate easily > the problem your series wants to fix? > > Thanks, > Laurent > > On 05/05/2017 22:47, Daniel Henrique Barboza wrote: >> v9: >> - patch 1 (*new*): added a qtail in sPAPRMachineState called pending_dimm_unplugs >> that stores the DIMM LMB state during the unplug process. >> - patch 2 (*new*): merged v8-patch1 and v8-patch2: removing detach_cb and >> detach_cb_opaque. >> - patch 3: >> * removed dk->vmsd entry. We're using vmstate_register instead >> * added 'awaiting_allocation' flag in the DRC migration >> - patch 4 (*new*): migrating spapr->pending_dimm_unplugs qtailq to allow >> for an ongoing PCDIMM unplug to continue after a migration. >> >> v8: >> - new patch added: 'removing spapr_drc_detach_cb opaques'. This new patch removes >> the need for the detach_cb_opaques inside the removal callback functions. See >> the commit message of the patch for more info. >> >> v7: >> - removed the first patch. DRC registration is now done by vmstate_register >> in patch 2. >> - added a new patch that changes spapr_dr_connector_new to receive as argument >> the detach_cb. >> - removed the callback logic of patch 2 since there is no need to restore the >> detach_cb on post-load due to the detach_cb on spapr_dr_connector_new change. >> - added word separators in the VMSD names of patch 3 and 4. >> >> v6: - Rebased with QEMU master after 6+ months. >> - Simplified the logic in patch 1. >> - Reworked patch 2: added CPU DRC migration, removed a function pointer from DRC >> class and minor improvements. >> - Added clarifications from the previous v5 discussions in the commit messages. >> >> v5: - Rebased to David's ppc-for-2.8. >> >> v4: - Introduce a way to set customized instance_id in SaveStateEntry. Use it >> to set instance_id for DRC using its unique index to address David >> Gibson's concern. >> - Rename VMS_CSTM to VMS_LINKED based on Paolo Bonzini's suggestions. >> - Clean up qjson stuff in put_qtailq. >> - Add trace for put_qtailq and get_qtailq based on David Gilbert's >> suggestion. >> >> - Based on David's ppc-for-2.7. >> >> v3: - Simplify overall design followng discussion with Paolo. No longer need >> metadata to migrate QTAILQ. >> - Extend VMStateInfo instead of adding similar fields to VMStateField. >> - Clean up macros in qemu/queue.h. >> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg05695.html) >> >> v2: - Introduce a general approach to migrate QTAILQ in qemu/queue.h. >> - Migrate signalled field in the DRC state. >> - Put the newly added migrating fields in subsections so that backward >> migration is not broken. >> - Set detach_cb field right after migration so that a migrated hot-unplug >> event could finish its course. >> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg04188.html) >> >> v1: - Inital version. >> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg02601.html) >> >> >> To make guest devices (PCI, CPU and memory) hotplug work together >> with guest migration, spapr drc state needs be transmitted in >> migration. This patch defines the VMStateDescription struct for >> spapr drc state to enable it. >> >> To fix the potential racing between hotplug events on guest and >> guest migration, ccs_list and pending_events of spapr state need be >> transmitted in migration. This patch set also takes care of it. >> >> >> >> Daniel Henrique Barboza (4): >> hw/ppc/spapr.c: adding pending_dimm_unplugs to sPAPRMachineState >> hw/ppc: removing drc->detach_cb and drc->detach_cb_opaque >> hw/ppc: migrating the DRC state of hotplugged devices >> hw/ppc/spapr.c: migrate pending_dimm_unplugs of spapr state >> >> Jianjun Duan (2): >> migration: spapr: migrate ccs_list in spapr state >> migration: spapr: migrate pending_events of spapr state >> >> hw/ppc/spapr.c | 158 +++++++++++++++++++++++++++++++++++++++++--- >> hw/ppc/spapr_drc.c | 97 ++++++++++++++++++++++----- >> hw/ppc/spapr_events.c | 24 ++++--- >> hw/ppc/spapr_pci.c | 5 +- >> include/hw/pci-host/spapr.h | 3 + >> include/hw/ppc/spapr.h | 24 ++++++- >> include/hw/ppc/spapr_drc.h | 8 +-- >> 7 files changed, 273 insertions(+), 46 deletions(-) >> > >
© 2016 - 2024 Red Hat, Inc.