[PATCH 06/16] xen/arm: Introduce system suspend config option

Mykola Kvach posted 16 patches 11 months, 1 week ago
[PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Mykola Kvach 11 months, 1 week ago
From: Mykola Kvach <mykola_kvach@epam.com>

This option enables the system suspend support. This is the
mechanism that allows the system to be suspended to RAM and
later resumed.

Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
---
 xen/arch/arm/Kconfig | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index a26d3e1182..5834af16ab 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
 config ARM32_HARDEN_BRANCH_PREDICTOR
     def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
 
+config SYSTEM_SUSPEND
+	bool "System suspend support"
+	default y
+	depends on ARM_64
+	help
+	  This option enables the system suspend support. This is the
+	  mechanism that allows the system to be suspended to RAM and
+	  later resumed.
+
+	  If unsure, say Y.
+
 source "arch/arm/platforms/Kconfig"
 
 source "common/Kconfig"
-- 
2.43.0
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Jan Beulich 11 months ago
On 05.03.2025 10:11, Mykola Kvach wrote:
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
>  config ARM32_HARDEN_BRANCH_PREDICTOR
>      def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
>  
> +config SYSTEM_SUSPEND
> +	bool "System suspend support"
> +	default y
> +	depends on ARM_64
> +	help
> +	  This option enables the system suspend support. This is the
> +	  mechanism that allows the system to be suspended to RAM and
> +	  later resumed.
> +
> +	  If unsure, say Y.

I wonder if something like this makes sense to place in an arch-specific
Kconfig. It's also not becoming clear here why only Arm64 would permit it.

Jan
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Mykola Kvach 10 months, 3 weeks ago
Hi,

On Thu, Mar 13, 2025 at 5:37 PM Jan Beulich <jbeulich@suse.com> wrote:
>
> On 05.03.2025 10:11, Mykola Kvach wrote:
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
> >  config ARM32_HARDEN_BRANCH_PREDICTOR
> >      def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
> >
> > +config SYSTEM_SUSPEND
> > +     bool "System suspend support"
> > +     default y
> > +     depends on ARM_64
> > +     help
> > +       This option enables the system suspend support. This is the
> > +       mechanism that allows the system to be suspended to RAM and
> > +       later resumed.
> > +
> > +       If unsure, say Y.
>
> I wonder if something like this makes sense to place in an arch-specific

Maybe it makes sense, but only if we are not planning to cover
suspend/resume related code for x86 as well.

> Kconfig. It's also not becoming clear here why only Arm64 would permit it.

If I understand your comment correctly, you’re suggesting that we
could use this for x86 as well. However, in that case, we would need
to make a lot of changes in other places that are not related to this
patch series, which is specifically focused on adding suspend/resume
support for Arm64. I believe that is outside the scope of this patch
series. However, this config was requested in one of the previous
patch series. The primary reason for adding this config was to reduce
the binary size for platforms where it isn’t used. I also think it can
be useful for debugging purposes, such as for identifying regressions.

As for Arm32, it’s not supported at the moment, but I hope support
will be added in the future.

>
> Jan

Best regards,
Mykola
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Jan Beulich 10 months, 3 weeks ago
On 21.03.2025 10:49, Mykola Kvach wrote:
> Hi,
> 
> On Thu, Mar 13, 2025 at 5:37 PM Jan Beulich <jbeulich@suse.com> wrote:
>>
>> On 05.03.2025 10:11, Mykola Kvach wrote:
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
>>>  config ARM32_HARDEN_BRANCH_PREDICTOR
>>>      def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
>>>
>>> +config SYSTEM_SUSPEND
>>> +     bool "System suspend support"
>>> +     default y
>>> +     depends on ARM_64
>>> +     help
>>> +       This option enables the system suspend support. This is the
>>> +       mechanism that allows the system to be suspended to RAM and
>>> +       later resumed.
>>> +
>>> +       If unsure, say Y.
>>
>> I wonder if something like this makes sense to place in an arch-specific
> 
> Maybe it makes sense, but only if we are not planning to cover
> suspend/resume related code for x86 as well.
> 
>> Kconfig. It's also not becoming clear here why only Arm64 would permit it.
> 
> If I understand your comment correctly, you’re suggesting that we
> could use this for x86 as well.

Or PPC / RISC-V once they progress enough.

> However, in that case, we would need
> to make a lot of changes in other places that are not related to this
> patch series, which is specifically focused on adding suspend/resume
> support for Arm64. I believe that is outside the scope of this patch
> series.

Considering that - give or take bugs - S3 is working on x86, I'm not
sure what lots of changes you're thinking of. In fact ...

> However, this config was requested in one of the previous
> patch series. The primary reason for adding this config was to reduce
> the binary size for platforms where it isn’t used. I also think it can
> be useful for debugging purposes, such as for identifying regressions.

... that's what I'd see as a (future) option on x86 as well. 

> As for Arm32, it’s not supported at the moment, but I hope support
> will be added in the future.

Which is another data point towards this wanting to move to common
code, with a per-arch-selected HAVE_* as dependency. To cover that it's
always-on for x86, an ..._ALWAYS_ON setting may want introducing as
well (or some shorthand approach to limit [future] churn).

Jan

Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Grygorii Strashko 10 months, 3 weeks ago

On 13.03.25 17:37, Jan Beulich wrote:
> On 05.03.2025 10:11, Mykola Kvach wrote:
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
>>   config ARM32_HARDEN_BRANCH_PREDICTOR
>>       def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
>>   
>> +config SYSTEM_SUSPEND
>> +	bool "System suspend support"
>> +	default y
>> +	depends on ARM_64
>> +	help
>> +	  This option enables the system suspend support. This is the
>> +	  mechanism that allows the system to be suspended to RAM and
>> +	  later resumed.
>> +
>> +	  If unsure, say Y.
> 
> I wonder if something like this makes sense to place in an arch-specific
> Kconfig. It's also not becoming clear here why only Arm64 would permit it.

Taking into account that
- System suspend-2-ram support on x86 is enabled by default and It's going to be supported on ARM also;
- follow up patches are adding #ifdef CONFIG_SYSTEM_SUSPEND not only in Arm specific code;
I think, it deserve to be generic option (in some way), enabled by default on x86.

Also, Arches can declare that suspend is possible, like ARCH_SUSPEND_POSSIBLE, then
generic configs would not need to be fixed every time when System suspend-2-ram enabled
for new arch.

Personally I'd introduce separate arch/Kconfig.power (or common Kconfig.power) file
for PM options (A least there is also cpufreq/cpuidel, and could be other, PM specific things).

Best regards,
-grygorii
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Julien Grall 11 months ago
Hi,

On 05/03/2025 09:11, Mykola Kvach wrote:
> From: Mykola Kvach <mykola_kvach@epam.com>
> 
> This option enables the system suspend support. This is the
> mechanism that allows the system to be suspended to RAM and
> later resumed.
> 
> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
> ---
>   xen/arch/arm/Kconfig | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index a26d3e1182..5834af16ab 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
>   config ARM32_HARDEN_BRANCH_PREDICTOR
>       def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
>   
> +config SYSTEM_SUSPEND
> +	bool "System suspend support"
> +	default y

The default should likely be no until everything is working.

> +	depends on ARM_64

I think this also needs to depends on !LLC_COLORING (unless you 
confirmed cache coloring is working) and UNSUPPORTED.

> +	help
> +	  This option enables the system suspend support. This is the
> +	  mechanism that allows the system to be suspended to RAM and
> +	  later resumed.

You seem to also tie guest suspend/resunme to this option. Is it intended?

Cheers,

-- 
Julien Grall
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Mykola Kvach 10 months, 3 weeks ago
Hi,

On Wed, Mar 12, 2025 at 12:29 AM Julien Grall <julien@xen.org> wrote:
>
> Hi,
>
> On 05/03/2025 09:11, Mykola Kvach wrote:
> > From: Mykola Kvach <mykola_kvach@epam.com>
> >
> > This option enables the system suspend support. This is the
> > mechanism that allows the system to be suspended to RAM and
> > later resumed.
> >
> > Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> > Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
> > ---
> >   xen/arch/arm/Kconfig | 11 +++++++++++
> >   1 file changed, 11 insertions(+)
> >
> > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > index a26d3e1182..5834af16ab 100644
> > --- a/xen/arch/arm/Kconfig
> > +++ b/xen/arch/arm/Kconfig
> > @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
> >   config ARM32_HARDEN_BRANCH_PREDICTOR
> >       def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
> >
> > +config SYSTEM_SUSPEND
> > +     bool "System suspend support"
> > +     default y
>
> The default should likely be no until everything is working.

got it!

>
> > +     depends on ARM_64
>
> I think this also needs to depends on !LLC_COLORING (unless you
> confirmed cache coloring is working) and UNSUPPORTED.

Sure! I'll add the dependency.

>
> > +     help
> > +       This option enables the system suspend support. This is the
> > +       mechanism that allows the system to be suspended to RAM and
> > +       later resumed.
>
> You seem to also tie guest suspend/resunme to this option. Is it intended?

From the guest's perspective, it is a system suspend. However, it looks like the
description should be enhanced. Thank you for pointing that out.

>
> Cheers,
>
> --
> Julien Grall
>

Best regards,
Mykola
Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Grygorii Strashko 10 months, 3 weeks ago

On 21.03.25 11:48, Mykola Kvach wrote:
> Hi,
> 
> On Wed, Mar 12, 2025 at 12:29 AM Julien Grall <julien@xen.org> wrote:
>>
>> Hi,
>>
>> On 05/03/2025 09:11, Mykola Kvach wrote:
>>> From: Mykola Kvach <mykola_kvach@epam.com>
>>>
>>> This option enables the system suspend support. This is the
>>> mechanism that allows the system to be suspended to RAM and
>>> later resumed.
>>>
>>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
>>> Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
>>> ---
>>>    xen/arch/arm/Kconfig | 11 +++++++++++
>>>    1 file changed, 11 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index a26d3e1182..5834af16ab 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
>>>    config ARM32_HARDEN_BRANCH_PREDICTOR
>>>        def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
>>>
>>> +config SYSTEM_SUSPEND
>>> +     bool "System suspend support"
>>> +     default y
>>
>> The default should likely be no until everything is working.
> 
> got it!
> 
>>
>>> +     depends on ARM_64
>>
>> I think this also needs to depends on !LLC_COLORING (unless you
>> confirmed cache coloring is working) and UNSUPPORTED.
> 
> Sure! I'll add the dependency.
> 
>>
>>> +     help
>>> +       This option enables the system suspend support. This is the
>>> +       mechanism that allows the system to be suspended to RAM and
>>> +       later resumed.
>>
>> You seem to also tie guest suspend/resunme to this option. Is it intended?
> 
>  From the guest's perspective, it is a system suspend. However, it looks like the
> description should be enhanced. Thank you for pointing that out.

s2r = "suspend to ram"

You definitely need consider and clarify ARM64 Guest System s2r and
XEN system s2r. First can be supported without second, while the XEN system s2r
depends on Guests System s2r support and required guests to be properly suspended
before allowing XEN to enter system s2r.

You can't call freeze_domains() and blindly pause some domain, because if it's not
suspend and has passed through HW which is in the middle of transaction -> DEADBEEF.

-- 
Best regards,
-grygorii

Re: [PATCH 06/16] xen/arm: Introduce system suspend config option
Posted by Mykola Kvach 10 months, 3 weeks ago
HI,

On Fri, Mar 21, 2025 at 4:58 PM Grygorii Strashko
<grygorii_strashko@epam.com> wrote:
>
>
>
> On 21.03.25 11:48, Mykola Kvach wrote:
> > Hi,
> >
> > On Wed, Mar 12, 2025 at 12:29 AM Julien Grall <julien@xen.org> wrote:
> >>
> >> Hi,
> >>
> >> On 05/03/2025 09:11, Mykola Kvach wrote:
> >>> From: Mykola Kvach <mykola_kvach@epam.com>
> >>>
> >>> This option enables the system suspend support. This is the
> >>> mechanism that allows the system to be suspended to RAM and
> >>> later resumed.
> >>>
> >>> Signed-off-by: Mykyta Poturai <mykyta_poturai@epam.com>
> >>> Signed-off-by: Mykola Kvach <mykola_kvach@epam.com>
> >>> ---
> >>>    xen/arch/arm/Kconfig | 11 +++++++++++
> >>>    1 file changed, 11 insertions(+)
> >>>
> >>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> >>> index a26d3e1182..5834af16ab 100644
> >>> --- a/xen/arch/arm/Kconfig
> >>> +++ b/xen/arch/arm/Kconfig
> >>> @@ -475,6 +475,17 @@ config ARM64_HARDEN_BRANCH_PREDICTOR
> >>>    config ARM32_HARDEN_BRANCH_PREDICTOR
> >>>        def_bool y if ARM_32 && HARDEN_BRANCH_PREDICTOR
> >>>
> >>> +config SYSTEM_SUSPEND
> >>> +     bool "System suspend support"
> >>> +     default y
> >>
> >> The default should likely be no until everything is working.
> >
> > got it!
> >
> >>
> >>> +     depends on ARM_64
> >>
> >> I think this also needs to depends on !LLC_COLORING (unless you
> >> confirmed cache coloring is working) and UNSUPPORTED.
> >
> > Sure! I'll add the dependency.
> >
> >>
> >>> +     help
> >>> +       This option enables the system suspend support. This is the
> >>> +       mechanism that allows the system to be suspended to RAM and
> >>> +       later resumed.
> >>
> >> You seem to also tie guest suspend/resunme to this option. Is it intended?
> >
> >  From the guest's perspective, it is a system suspend. However, it looks like the
> > description should be enhanced. Thank you for pointing that out.
>
> s2r = "suspend to ram"
>
> You definitely need consider and clarify ARM64 Guest System s2r and
> XEN system s2r. First can be supported without second, while the XEN system s2r
> depends on Guests System s2r support and required guests to be properly suspended
> before allowing XEN to enter system s2r.
>

This is exactly what...

> You can't call freeze_domains() and blindly pause some domain, because if it's not
> suspend and has passed through HW which is in the middle of transaction -> DEADBEEF.

... should happen. x86 works in the same way—we call domain_pause when
performing system suspend. All domains are paused except the one that
is eligible to request system suspend.

>
> --
> Best regards,
> -grygorii

Best regards,
Mykola