From: Mykola Kvach <mykola_kvach@epam.com>
This is part 2 of version 5 of the ARM Xen system suspend/resume patch
series, based on earlier work by Mirela Simonovic and Mykyta Poturai.
This version is ported to Xen master (4.21-unstable) and includes
extensive improvements based on reviewer feedback. The patch series
restructures code to improve robustness, maintainability, and implements
system Suspend-to-RAM support on ARM64 hardware domains.
At a high-level, this patch series provides:
- Support for Host system suspend/resume via PSCI SYSTEM_SUSPEND (ARM64)
- Suspend/resume infrastructure for CPU context, timers, GICv2/GICv3 and IPMMU-VMSA
- Proper error propagation and recovery throughout the suspend/resume flow
Key updates in this series:
- Introduced architecture-specific suspend/resume infrastructure (new `suspend.c`, `suspend.h`, low-level context save/restore in `head.S`)
- Integrated GICv2/GICv3 suspend and resume, including memory-backed context save/restore with error handling
- Added time and IRQ suspend/resume hooks, ensuring correct timer/interrupt state across suspend cycles
- Implemented proper PSCI SYSTEM_SUSPEND invocation and version checks
- Improved state management and recovery in error cases during suspend/resume
- Added support for IPMMU-VMSA context save/restore
---
TODOs:
- Test system suspend with llc_coloring_enabled set and verify functionality
- Implement SMMUv3 suspend/resume handlers
- Enable "xl suspend" support on ARM
- Properly disable Xen timer watchdog from relevant services (only init.d left)
- Add suspend/resume CI test for ARM (QEMU if feasible)
- Investigate feasibility and need for implementing system suspend on ARM32
---
Changes introduced in V5:
- Add support for IPMMU-VMSA context save/restore
- Add support for GICv3 context save/restore
- Select HAS_SYSTEM_SUSPEND in ARM_64 instead of ARM
- Check llc_coloring_enabled instead of LLC_COLORING during the selection
of HAS_SYSTEM_SUSPEND config
- Call host_system_suspend from guest PSCI system suspend instead of
arch_domain_shutdown, reducing the complexity of the new code
Changes introduced in V4:
- Remove the prior tasklet-based workaround in favor of a more
straightforward and safer solution.
- Rework the approach by adding explicit system state checks around
request_irq and release_irq calls; skip these calls during suspend
and resume states to avoid unsafe memory operations when IRQs are
disabled.
- Prevent reinitialization of local IRQ descriptors on system resume.
- Restore the state of local IRQs during system resume for secondary CPUs.
- Drop code for saving and restoring VCPU context (see part 1 of the patch
series for details).
- Remove IOMMU suspend and resume calls until these features are implemented.
- Move system suspend logic to arch_domain_shutdown, invoked from
domain_shutdown.
- Add console_end_sync to the resume path after system suspend.
- Drop unnecessary DAIF masking; interrupts are already masked on resume.
- Remove leftover TLB flush instructions; flushing is handled in enable_mmu.
- Avoid setting x19 in hyp_resume as it is not required.
- Replace prepare_secondary_mm with set_init_ttbr, and call it from system_suspend.
- Produce a build-time error for ARM32 when CONFIG_SYSTEM_SUSPEND is enabled.
- Use register_t instead of uint64_t in the cpu_context structure.
- Apply minor fixes such as renaming functions, updating comments, and
modifying commit messages to accurately reflect the changes introduced
by this patch series.
For earlier changelogs, please refer to the previous cover letters.
Previous versions:
V1: https://marc.info/?l=xen-devel&m=154202231501850&w=2
V2: https://marc.info/?l=xen-devel&m=166514782207736&w=2
V3: https://lists.xen.org/archives/html/xen-devel/2025-03/msg00168.html
Mirela Simonovic (6):
xen/arm: Add suspend and resume timer helpers
xen/arm: gic-v2: Implement GIC suspend/resume functions
xen/arm: Add support for system suspend triggered by hardware domain
xen/arm: Implement PSCI SYSTEM_SUSPEND call (host interface)
xen/arm: Resume memory management on Xen resume
xen/arm: Save/restore context on suspend/resume
Mykola Kvach (4):
xen/arm: gic-v3: Implement GICv3 suspend/resume functions
xen/arm: Prevent crash during disable_nonboot_cpus on suspend
xen/arm: irq: avoid local IRQ descriptors reinit on system resume
xen/arm: irq: Restore state of local IRQs during system resume
Oleksandr Tyshchenko (2):
iommu/ipmmu-vmsa: Implement suspend/resume callbacks
xen/arm: Suspend/resume IOMMU on Xen suspend/resume
xen/arch/arm/Kconfig | 1 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/arm64/head.S | 113 ++++++++++
xen/arch/arm/gic-v2.c | 154 +++++++++++++
xen/arch/arm/gic-v3.c | 233 ++++++++++++++++++++
xen/arch/arm/gic.c | 35 +++
xen/arch/arm/include/asm/gic.h | 12 +
xen/arch/arm/include/asm/mm.h | 2 +
xen/arch/arm/include/asm/psci.h | 1 +
xen/arch/arm/include/asm/suspend.h | 44 ++++
xen/arch/arm/include/asm/time.h | 5 +
xen/arch/arm/irq.c | 48 ++++
xen/arch/arm/mmu/smpboot.c | 2 +-
xen/arch/arm/psci.c | 23 +-
xen/arch/arm/suspend.c | 196 +++++++++++++++++
xen/arch/arm/tee/ffa_notif.c | 2 +-
xen/arch/arm/time.c | 56 ++++-
xen/arch/arm/vpsci.c | 12 +-
xen/common/domain.c | 4 +
xen/drivers/passthrough/arm/ipmmu-vmsa.c | 269 +++++++++++++++++++++++
20 files changed, 1197 insertions(+), 16 deletions(-)
create mode 100644 xen/arch/arm/include/asm/suspend.h
create mode 100644 xen/arch/arm/suspend.c
--
2.48.1