Hi,


The first patch fixes GIC and brings the secondary CPU. Unfortunately,
this brings up to light an annoying issue with CPUIDLE driver.


Overview of the problem
=======================
On Exynos4210, by default Linux kernel uses cpuidle driver which tries
to enter low power mode, called AFTR (Arm Off, Top Running).  On real
hardware this brings some power savings.  This AFTR mode requires second
CPU to be off, so the driver (coupled cpuidle driver) when system is idle:
1. Turns off second CPU,
2. Enters AFTR on CPU0.

However the QEMU system is then totally unresponsive (e.g. on serial console)
and RCU stalls appear from time to time.  I spent some time on it and did not
find the real cause behind the lag.  Maybe it is because the second CPU
does not really power down itself and system just burns the cycles under spin
locks?

Looking at recent stable kernels:
 - 3.10, 3.16 - works fine with two CPUs (no CPUIDLE driver),
 - 4.1 and newer - stalls and are unresponsive due to CPUIDLE being enabled.

The cpuidle driver is not relying on DTS. It is just enabled in exynos_defconfig
and works.


Question
========
I am quite new to QEMU.  I do not know the internals (yet), nor the design
how hardware should be emulated in such subtle details.

The questions I have are:
1. What is the preferred way to solve it? Maybe some a QEMU
   workaround to disable the cpuidle is okay?
2. Is it worth implementing a proper secondary CPU power down and 
   at the end proper AFTR cpuidle? It might be quite difficult...
3. How such issues with deep sleep modes were solved for other
   ARM targets?


Best regards,
Krzysztof


Krzysztof Kozlowski (5):
  hw/intc/exynos4210_gic: Fix GIC memory mappings for secondary CPU
  hw/intc/exynos4210_gic: Use more meaningful name for local variable
  hw/timer/exynos4210_mct: Cleanup indentation and empty new lines
  hw/timer/exynos4210_mct: Fix checkpatch style errors
  hw/timer/exynos4210_mct: Remove unused defines

 hw/intc/exynos4210_gic.c  | 36 ++++++++++++++++++++--------------
 hw/timer/exynos4210_mct.c | 50 ++++++++++++++++++++---------------------------
 2 files changed, 42 insertions(+), 44 deletions(-)

-- 
2.9.3

Hi,

This series seems to have some coding style problems. See output below for
more information:

Message-id: 20170301182618.22395-1-krzk@kernel.org
Type: series
Subject: [Qemu-devel] [RFC 0/5] hw: arm: exynos: Bring up secondary CPU + CPUIDLE issue

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

# Useful git options
git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/20170301182618.22395-1-krzk@kernel.org -> patchew/20170301182618.22395-1-krzk@kernel.org
Switched to a new branch 'test'
acc22b2 hw/timer/exynos4210_mct: Remove unused defines
312c3fc hw/timer/exynos4210_mct: Fix checkpatch style errors
fb89224 hw/timer/exynos4210_mct: Cleanup indentation and empty new lines
040f3f8 hw/intc/exynos4210_gic: Use more meaningful name for local variable
20aa421 hw/intc/exynos4210_gic: Fix GIC memory mappings for secondary CPU

=== OUTPUT BEGIN ===
Checking PATCH 1/5: hw/intc/exynos4210_gic: Fix GIC memory mappings for secondary CPU...
Checking PATCH 2/5: hw/intc/exynos4210_gic: Use more meaningful name for local variable...
Checking PATCH 3/5: hw/timer/exynos4210_mct: Cleanup indentation and empty new lines...
ERROR: spaces required around that '&' (ctx:VxV)
#64: FILE: hw/timer/exynos4210_mct.c:1163:
+        if (offset&0x4) {
                   ^

total: 1 errors, 0 warnings, 93 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 4/5: hw/timer/exynos4210_mct: Fix checkpatch style errors...
Checking PATCH 5/5: hw/timer/exynos4210_mct: Remove unused defines...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

On 1 March 2017 at 18:26, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> Overview of the problem
> =======================
> On Exynos4210, by default Linux kernel uses cpuidle driver which tries
> to enter low power mode, called AFTR (Arm Off, Top Running).  On real
> hardware this brings some power savings.  This AFTR mode requires second
> CPU to be off, so the driver (coupled cpuidle driver) when system is idle:
> 1. Turns off second CPU,
> 2. Enters AFTR on CPU0.
>
> However the QEMU system is then totally unresponsive (e.g. on serial console)
> and RCU stalls appear from time to time.  I spent some time on it and did not
> find the real cause behind the lag.  Maybe it is because the second CPU
> does not really power down itself and system just burns the cycles under spin
> locks?

Possibly so. You might check whether it still has this behaviour with
current head of git now that the multi-threaded TCG support has landed.

> I am quite new to QEMU.  I do not know the internals (yet), nor the design
> how hardware should be emulated in such subtle details.
>
> The questions I have are:
> 1. What is the preferred way to solve it? Maybe some a QEMU
>    workaround to disable the cpuidle is okay?
> 2. Is it worth implementing a proper secondary CPU power down and
>    at the end proper AFTR cpuidle? It might be quite difficult...
> 3. How such issues with deep sleep modes were solved for other
>    ARM targets?

The general answer to 3 is "we don't bother implementing deep sleep
and mostly this hasn't caused problems, but more by luck than
judgement". Most of our boards aren't SMP; until now SMP TCG guests
have been slower than single core so even on a nominally SMP guest
board most users probably use (the default) -smp 1.

 The SMP board that gets most use is the "virt" board whose
power-down functionality works through our PSCI implementation.
The other board we have which has power-down is the imx6 one:
hw/misc/imx6_src.c is the system reset controller model which
does this. Basically the model of the whatever-it-is that
causes the CPU to be powered down does it via the APIs in
target/arm/arm-powerctl.h, which lets you turn CPUs off and on.
When a CPU is off then we don't try to let it execute instructions,
so it shouldn't chew host CPU at the expense of other guest CPUs.

Minor caution with that API: arm_set_cpu_on() is asynchronous,
so if you need to tell the guest "CPU now actually booted" then
you'll need to use an async_run_on_cpu() callback the way the
imx6_src.c code does.

If the "CPU on" behaviour you need is "CPU powers on as if
starting up in standard reset" rather than "CPU powers up
and appears to start execution at given address", then it's
probably best to improve the arm_powerctl APIs rather than
fudge it by trying to work out the arguments to pass to
arm_set_cpu_on() to make its "and now go to this address"
code not do anything. (The APIs are targeted at the current
users which both want "do things which are done in the guest
firmware or real hardware".)

thanks
-- PMM

On Thu, Mar 02, 2017 at 04:56:22PM +0000, Peter Maydell wrote:
> On 1 March 2017 at 18:26, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > Overview of the problem
> > =======================
> > On Exynos4210, by default Linux kernel uses cpuidle driver which tries
> > to enter low power mode, called AFTR (Arm Off, Top Running).  On real
> > hardware this brings some power savings.  This AFTR mode requires second
> > CPU to be off, so the driver (coupled cpuidle driver) when system is idle:
> > 1. Turns off second CPU,
> > 2. Enters AFTR on CPU0.
> >
> > However the QEMU system is then totally unresponsive (e.g. on serial console)
> > and RCU stalls appear from time to time.  I spent some time on it and did not
> > find the real cause behind the lag.  Maybe it is because the second CPU
> > does not really power down itself and system just burns the cycles under spin
> > locks?
> 
> Possibly so. You might check whether it still has this behaviour with
> current head of git now that the multi-threaded TCG support has landed.

No changes.

> > I am quite new to QEMU.  I do not know the internals (yet), nor the design
> > how hardware should be emulated in such subtle details.
> >
> > The questions I have are:
> > 1. What is the preferred way to solve it? Maybe some a QEMU
> >    workaround to disable the cpuidle is okay?
> > 2. Is it worth implementing a proper secondary CPU power down and
> >    at the end proper AFTR cpuidle? It might be quite difficult...
> > 3. How such issues with deep sleep modes were solved for other
> >    ARM targets?
> 
> The general answer to 3 is "we don't bother implementing deep sleep
> and mostly this hasn't caused problems, but more by luck than
> judgement". Most of our boards aren't SMP; until now SMP TCG guests
> have been slower than single core so even on a nominally SMP guest
> board most users probably use (the default) -smp 1.
> 
>  The SMP board that gets most use is the "virt" board whose
> power-down functionality works through our PSCI implementation.
> The other board we have which has power-down is the imx6 one:
> hw/misc/imx6_src.c is the system reset controller model which
> does this. Basically the model of the whatever-it-is that
> causes the CPU to be powered down does it via the APIs in
> target/arm/arm-powerctl.h, which lets you turn CPUs off and on.
> When a CPU is off then we don't try to let it execute instructions,
> so it shouldn't chew host CPU at the expense of other guest CPUs.
> 
> Minor caution with that API: arm_set_cpu_on() is asynchronous,
> so if you need to tell the guest "CPU now actually booted" then
> you'll need to use an async_run_on_cpu() callback the way the
> imx6_src.c code does.
> 
> If the "CPU on" behaviour you need is "CPU powers on as if
> starting up in standard reset" rather than "CPU powers up
> and appears to start execution at given address", then it's
> probably best to improve the arm_powerctl APIs rather than
> fudge it by trying to work out the arguments to pass to
> arm_set_cpu_on() to make its "and now go to this address"
> code not do anything. (The APIs are targeted at the current
> users which both want "do things which are done in the guest
> firmware or real hardware".)

Thanks for the hints! Especially the existing implementations might be
quite useful.

The cpuidle driver expects the CPU to come up to the same resume
address. I'll see what others are doing and learn from that. 

As I mentioned before, on current master thish patchset makes everything
more laggish and unresponsive (because of cpuidle) so actually I think
the first commit could be postponed for now. At least till I figure out
solution for the cpuidle problem.


Best regards,
Krzysztof