xen/arm64: Suspend to RAM support for Xen

[PATCH 00/19] xen/arm64: Suspend to RAM support for Xen

Posted by Mykyta Poturai 1 year, 7 months ago

This is a series from Mirela Simonovic. Ported to 4.16 and with
added changes suggested here
https://lore.kernel.org/all/CAKPH-NjmaZENb8gT=+FobrAycRF01_--6GuRA2ck9Di5wiudhA@mail.gmail.com

This series contains support for suspend to RAM (in the following text just
'suspend') for Xen on arm64. The implementation is aligned with the design
specification that has been proposed on xen-devel list:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

At a high-level the series contains:
1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
2) Support for resuming a guest on an interrupt targeted to that guest
3) Support for suspending Xen after dom0 finalizes the suspend
4) Support for resuming Xen on an interrupt that is targeted to a guest

--------------------------------------------------------------------------------
In more details:

*** About suspending/resuming guests

The patches included in this series allow PSCI compliant guests that have
support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
suspend and resume on top of Xen without any EL1 code changes.

During their suspend procedure guests will hot-unplug their secondary CPUs,
triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
Guests will save/restore their own EL1 context on suspend/resume.

A guest is expected to leave enabled interrupts that are considered to be its
wake-up sources. Those interrupts will be able to wake up the guest. This holds
regardless of the state of the underlying software layers, i.e. whether Xen gets
suspended or not doesn't affect the ability of the guest to wake up.

First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from which
the guest assumes to start on resume. On resume, guests assume to be running in
an environment whose state matches the CPU state after reset, e.g. with reset
register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
VCPU context and save the resume entry point into program counter before the
guest's VCPU gets scheduled in on resume. This is done when the guest resumes.
Xen also needs to take care that the guest's view of GIC and timer gets saved.
Also, while a guest is suspended its watchdogs are paused, in order to avoid
watchdog triggered shutdown of a guest that has been asleep for a period of time
that is longer than the watchdog period.

After this point, from Xen's point of view a suspended guest has one VCPU
blocked, waiting for an interrupt. When such an interrupt comes, Xen will
unblock the VCPU of the suspended domain, which results in the guest
resuming.

*** About suspending/resuming Xen

Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
considered to be the decision maker for EL1 and EL2.
On suspend, Xen will first freeze all domains. Then, Xen disables physical
secondary CPUs, which leads to physical CPU_OFF to be called by each secondary
CPU. After that Xen finalizes the suspend from the boot CPU.

This consists of suspending the timer, i.e. suppressing its interrupts (we don't
want to be woken up by a timer, there is no VCPU ready to be scheduled). Then
the state of GIC is saved, console is suspended, and CPU context is saved. The
saved context tells where Xen needs to continue execution on resume.
Since Xen will resume with MMU disabled, the first thing to do in resume is to
resume memory management in order to be able to access the context that needs to
be restored (we know virtual address of the context data). Finally Xen calls
SYSTEM_SUSPEND PSCI to the EL3.

When resuming, all the steps done in suspend need to be reverted. This is
completed by unblocking dom0's VCPU, because we always want the dom0 to
resume,
regardless of the target domain whose interrupt woke up Xen.

*** Handling of unprivileged guests during Xen suspend/resume

Any domU that is not suspended when dom0 suspends will be frozen, domUs that are
already suspended remain suspended. On resume the suspended domUs still remain
suspended (unless their wake interrupt caused Xen to wake) while the
others will be thawed.

For more details please refer to patches or the design specification:
https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

Juergen Gross (1):
xen: don't free percpu areas during suspend

Mirela Simonovic (15):
xen/arm: Implement PSCI system suspend
xen/arm: While a domain is suspended put its watchdogs on pause
xen/arm: Trigger Xen suspend when Dom0 completes suspend
xen/x86: Move freeze/thaw_domains into common files
xen/arm: Freeze domains on suspend and thaw them on resume
xen/arm: Disable/enable non-boot physical CPUs on suspend/resume
xen/arm: Add rcu_barrier() before enabling non-boot CPUs on resume
xen/arm: Implement GIC suspend/resume functions (gicv2 only)
xen/arm: Suspend/resume GIC on system suspend/resume
xen/arm: Suspend/resume timer interrupt generation
xen/arm: Implement PSCI SYSTEM_SUSPEND call (physical interface)
xen/arm: Resume memory management on Xen resume
xen/arm: Save/restore context on suspend/resume
xen/arm: Resume Dom0 after Xen resumes
xen/arm: Suspend/resume console on Xen suspend/resume

Mykyta Poturai (2):
watchdog: Introduce a separate struct for watchdog timers
timers: Don't migrate timers during suspend

Oleksandr Andrushchenko (1):
Fix misleading indentation gcc warning

--
2.37.1

Re: [PATCH 00/19] xen/arm64: Suspend to RAM support for Xen

Posted by Julien Grall 1 year, 5 months ago

Hi,

On 07/10/2022 14:08, Mykyta Poturai wrote:
> This is a series from Mirela Simonovic. Ported to 4.16 and with
> added changes suggested here
> https://lore.kernel.org/all/CAKPH-NjmaZENb8gT=+FobrAycRF01_--6GuRA2ck9Di5wiudhA@mail.gmail.com
> 
> This series contains support for suspend to RAM (in the following text just
> 'suspend') for Xen on arm64. The implementation is aligned with the design
> specification that has been proposed on xen-devel list:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
> 
> At a high-level the series contains:
> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
> 2) Support for resuming a guest on an interrupt targeted to that guest
> 3) Support for suspending Xen after dom0 finalizes the suspend
> 4) Support for resuming Xen on an interrupt that is targeted to a guest

 From the previous discussion, there were a few dubious code in 
cpu_disable() (e.g. mdelay()). So did you go through the code to confirm 
it is fully Arm compliant?

> 
> 
> --------------------------------------------------------------------------------
> In more details:
> 
> *** About suspending/resuming guests
> 
> The patches included in this series allow PSCI compliant guests that have
> support for suspend to RAM (e.g. echo mem > /sys/power/state in Linux) to
> suspend and resume on top of Xen without any EL1 code changes.

What's the mechanism to tell the guest to suspend?

> 
> During their suspend procedure guests will hot-unplug their secondary CPUs,
> triggering Xen's virtual CPU_OFF PSCI implementation, and then finalize the
> suspend from their boot CPU, triggering Xen's virtual SYSTEM_SUSPEND PSCI.
> Guests will save/restore their own EL1 context on suspend/resume.
> 
> A guest is expected to leave enabled interrupts that are considered to be its
> wake-up sources. Those interrupts will be able to wake up the guest. This holds
> regardless of the state of the underlying software layers, i.e. whether Xen gets
> suspended or not doesn't affect the ability of the guest to wake up.
> 
> First argument of SYSTEM_SUSPEND PSCI call is a resume entry point, from which
> the guest assumes to start on resume. On resume, guests assume to be running in
> an environment whose state matches the CPU state after reset, e.g. with reset
> register values, MMU disabled, etc. To ensure this, Xen has to 'reset' the
> VCPU context and save the resume entry point into program counter before the
> guest's VCPU gets scheduled in on resume. This is done when the guest resumes.
> Xen also needs to take care that the guest's view of GIC and timer gets saved.
> Also, while a guest is suspended its watchdogs are paused, in order to avoid
> watchdog triggered shutdown of a guest that has been asleep for a period of time
> that is longer than the watchdog period.
> 
> After this point, from Xen's point of view a suspended guest has one VCPU
> blocked, waiting for an interrupt. When such an interrupt comes, Xen will
> unblock the VCPU of the suspended domain, which results in the guest
> resuming.
> 
> *** About suspending/resuming Xen
> 
> Xen starts its own suspend procedure once dom0 is suspended. Dom0 is
> considered to be the decision maker for EL1 and EL2.

Can you explain why dom0 is the decision maker here? Also, what about 
cases where there is no dom0?

> On suspend, Xen will first freeze all domains. Then, Xen disables physical
> secondary CPUs, which leads to physical CPU_OFF to be called by each secondary
> CPU. After that Xen finalizes the suspend from the boot CPU.
> 
> This consists of suspending the timer, i.e. suppressing its interrupts (we don't
> want to be woken up by a timer, there is no VCPU ready to be scheduled). Then
> the state of GIC is saved, console is suspended, and CPU context is saved. The
> saved context tells where Xen needs to continue execution on resume.
> Since Xen will resume with MMU disabled, the first thing to do in resume is to
> resume memory management in order to be able to access the context that needs to
> be restored (we know virtual address of the context data). Finally Xen calls
> SYSTEM_SUSPEND PSCI to the EL3.
> 
> When resuming, all the steps done in suspend need to be reverted. This is
> completed by unblocking dom0's VCPU, because we always want the dom0 to
> resume,
> regardless of the target domain whose interrupt woke up Xen.
> 
> *** Handling of unprivileged guests during Xen suspend/resume
> 
> Any domU that is not suspended when dom0 suspends will be frozen,
Wouldn't this mess up with the timers in the guest OS?

> domUs that are
> already suspended remain suspended. On resume the suspended domUs still remain
> suspended (unless their wake interrupt caused Xen to wake) while the
> others will be thawed.
> 
> For more details please refer to patches or the design specification:
> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html

This is a 5 years old design and there were some comments on it. So I 
think it would be best if a new version is sent.

Cheers,

-- 
Julien Grall

Re: [PATCH 00/19] xen/arm64: Suspend to RAM support for Xen

Posted by Julien Grall 1 year, 5 months ago


On 06/12/2022 20:32, Julien Grall wrote:
> Hi,
> 
> On 07/10/2022 14:08, Mykyta Poturai wrote:
>> This is a series from Mirela Simonovic. Ported to 4.16 and with
>> added changes suggested here
>> https://lore.kernel.org/all/CAKPH-NjmaZENb8gT=+FobrAycRF01_--6GuRA2ck9Di5wiudhA@mail.gmail.com
>>
>> This series contains support for suspend to RAM (in the following text 
>> just
>> 'suspend') for Xen on arm64. The implementation is aligned with the 
>> design
>> specification that has been proposed on xen-devel list:
>> https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg01574.html
>>
>> At a high-level the series contains:
>> 1) Support for suspending guests via virtual PSCI SYSTEM_SUSPEND call
>> 2) Support for resuming a guest on an interrupt targeted to that guest
>> 3) Support for suspending Xen after dom0 finalizes the suspend
>> 4) Support for resuming Xen on an interrupt that is targeted to a guest
> 
>  From the previous discussion, there were a few dubious code in 
> cpu_disable() (e.g. mdelay()). So did you go through the code to confirm 
> it is fully Arm compliant?

Another missing pieces is that we don't free anything related to the CPU 
that is been powered down. But AFAICT, we will end up to allocate the 
memory again for things like page tables, per-cpu...

Cheers,

-- 
Julien Grall