[Xen-devel] [PATCH v6 00/20] xen: add core scheduling support

Juergen Gross posted 20 patches 4 years, 5 months ago
Failed in applying to current master (apply log)
docs/misc/xen-command-line.pandoc |   28 +
xen/arch/arm/domain.c             |    2 +-
xen/arch/x86/Kconfig              |    1 +
xen/arch/x86/acpi/power.c         |    4 +
xen/arch/x86/domain.c             |   26 +-
xen/arch/x86/sysctl.c             |    5 +
xen/common/Kconfig                |    3 +
xen/common/cpupool.c              |  232 +++++--
xen/common/domain.c               |    8 +-
xen/common/domctl.c               |    2 +-
xen/common/sched_arinc653.c       |    4 +-
xen/common/sched_credit.c         |   73 ++-
xen/common/sched_credit2.c        |   32 +-
xen/common/sched_null.c           |   11 +-
xen/common/sched_rt.c             |   18 +-
xen/common/schedule.c             | 1300 ++++++++++++++++++++++++++++++-------
xen/common/softirq.c              |    6 +-
xen/include/asm-arm/current.h     |    1 +
xen/include/asm-x86/current.h     |   19 +-
xen/include/asm-x86/smp.h         |    7 +
xen/include/xen/sched-if.h        |   86 ++-
xen/include/xen/sched.h           |   26 +-
xen/include/xen/softirq.h         |    1 +
23 files changed, 1532 insertions(+), 363 deletions(-)
[Xen-devel] [PATCH v6 00/20] xen: add core scheduling support
Posted by Juergen Gross 4 years, 5 months ago
Add support for core- and socket-scheduling in the Xen hypervisor.

Via boot parameter sched-gran=core (or sched-gran=socket)
it is possible to change the scheduling granularity from cpu (the
default) to either whole cores or even sockets.

All logical cpus (threads) of the core or socket are always scheduled
together. This means that on a core always vcpus of the same domain
will be active, and those vcpus will always be scheduled at the same
time.

This is achieved by switching the scheduler to no longer see vcpus as
the primary object to schedule, but "schedule units". Each schedule
unit consists of as many vcpus as each core has threads on the current
system. The vcpu->unit relation is fixed.

I have done some very basic performance testing: on a 4 cpu system
(2 cores with 2 threads each) I did a "make -j 4" for building the Xen
hypervisor. With This test has been run on dom0, once with no other
guest active and once with another guest with 4 vcpus running the same
test. The results are (always elapsed time, system time, user time):

sched-gran=cpu,    no other guest: 116.10 177.65 207.84
sched-gran=core,   no other guest: 114.04 175.47 207.45
sched-gran=cpu,    other guest:    202.30 334.21 384.63
sched-gran=core,   other guest:    207.24 293.04 371.37

The performance tests have been performed with credit2, the other
schedulers are tested only briefly to be able to create a domain in a
cpupool.

Cpupools have been moderately tested (cpu add/remove, create, destroy,
move domain).

Cpu on-/offlining has been moderately tested, too.

Changes in V6:
- only added new patch 20 (sent as v5 20/19 before)

Changes in V5:
- dropped patches 1-27 as they already went in
- added comments in 2 patches

Changes in V4:
- comments addressed
- former patch 36 merged into patch 32

Changes in V3:
- comments addressed
- former patch 26 carved out and sent separately
- some minor bugs fixed

Changes in V2:
- comments addressed
- some patches merged into one
- idle scheduler related patches split off to own series
- some patches are already applied
- some bugs fixed (e.g. crashes when powering off)

Changes in V1:
- cpupools are working now
- cpu on-/offlining working now
- all schedulers working now
- renamed "items" to "units"
- introduction of "idle scheduler"
- several new patches (see individual patches, mostly splits of
  former patches or cpupool and cpu on-/offlining support)
- all review comments addressed
- some minor changes (see individual patches)

Changes in RFC V2:
- ARM is building now
- HVM domains are working now
- idling will always be done with idle_vcpu active
- other small changes see individual patches

Juergen Gross (20):
  xen/sched: add code to sync scheduling of all vcpus of a sched unit
  xen/sched: introduce unit_runnable_state()
  xen/sched: add support for multiple vcpus per sched unit where missing
  xen/sched: modify cpupool_domain_cpumask() to be an unit mask
  xen/sched: support allocating multiple vcpus into one sched unit
  xen/sched: add a percpu resource index
  xen/sched: add fall back to idle vcpu when scheduling unit
  xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling aware
  xen/sched: move per-cpu variable scheduler to struct sched_resource
  xen/sched: move per-cpu variable cpupool to struct sched_resource
  xen/sched: reject switching smt on/off with core scheduling active
  xen/sched: prepare per-cpupool scheduling granularity
  xen/sched: split schedule_cpu_switch()
  xen/sched: protect scheduling resource via rcu
  xen/sched: support multiple cpus per scheduling resource
  xen/sched: support differing granularity in schedule_cpu_[add/rm]()
  xen/sched: support core scheduling for moving cpus to/from cpupools
  xen/sched: disable scheduling when entering ACPI deep sleep states
  xen/sched: add scheduling granularity enum
  docs: add "sched-gran" boot parameter documentation

 docs/misc/xen-command-line.pandoc |   28 +
 xen/arch/arm/domain.c             |    2 +-
 xen/arch/x86/Kconfig              |    1 +
 xen/arch/x86/acpi/power.c         |    4 +
 xen/arch/x86/domain.c             |   26 +-
 xen/arch/x86/sysctl.c             |    5 +
 xen/common/Kconfig                |    3 +
 xen/common/cpupool.c              |  232 +++++--
 xen/common/domain.c               |    8 +-
 xen/common/domctl.c               |    2 +-
 xen/common/sched_arinc653.c       |    4 +-
 xen/common/sched_credit.c         |   73 ++-
 xen/common/sched_credit2.c        |   32 +-
 xen/common/sched_null.c           |   11 +-
 xen/common/sched_rt.c             |   18 +-
 xen/common/schedule.c             | 1300 ++++++++++++++++++++++++++++++-------
 xen/common/softirq.c              |    6 +-
 xen/include/asm-arm/current.h     |    1 +
 xen/include/asm-x86/current.h     |   19 +-
 xen/include/asm-x86/smp.h         |    7 +
 xen/include/xen/sched-if.h        |   86 ++-
 xen/include/xen/sched.h           |   26 +-
 xen/include/xen/softirq.h         |    1 +
 23 files changed, 1532 insertions(+), 363 deletions(-)

-- 
2.16.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v6 00/20] xen: add core scheduling support
Posted by Sergey Dyasli 4 years, 5 months ago
Hi Juergen,

Looks like we've hit the first Xen crash with core scheduling patches applied.
The log is below. From my analysis it seems that CSCHED_PCPU is NULL.
I suspect this is connected to commit b0000b128adb
("sched: populate cpupool0 only after all cpus are up")

Could you take a look, please?

(XEN) [  164.775206] Brought up 448 CPUs
(XEN) [  164.827979] Testing NMI watchdog on all CPUs: ok
(XEN) [  168.064678] ----[ Xen-4.13.0-8.0.12-d  x86_64  debug=y   Not tainted ]----
(XEN) [  168.128822] CPU:    79
(XEN) [  168.191394] RIP:    e008:[<ffff82d0802467dc>] set_timer+0x39/0x1f7
(XEN) [  168.255921] RFLAGS: 0000000000010002   CONTEXT: hypervisor
(XEN) [  168.319752] rax: 0000000000000048   rbx: 0000000000000020   rcx: ffff82d0805b8008
(XEN) [  168.382563] rdx: ffff8462efce7fff   rsi: 0000002721c0ad80   rdi: 0000000000000020
(XEN) [  168.445194] rbp: ffff8462efce7de0   rsp: ffff8462efce7da0   r8:  0000000000008eea
(XEN) [  168.507630] r9:  0000000000000000   r10: 3333333333333333   r11: 0000000000000008
(XEN) [  168.569849] r12: ffff82d080488f20   r13: 00000027216dad05   r14: 000000272167bef6
(XEN) [  168.631879] r15: 0000000000000008   cr0: 000000008005003b   cr4: 00000000003526e0
(XEN) [  168.693743] cr3: 00000000a5a9d000   cr2: 0000000000000048
(XEN) [  168.754868] fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) [  168.816304] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) [  168.877390] Xen code around <ffff82d0802467dc> (set_timer+0x39/0x1f7):
(XEN) [  168.938117]  48 8d 47 28 48 89 45 c0 <66> 44 8b 67 28 45 0f b7 e4 41 81 fc ff ff 00 00
(XEN) [  169.015525] Xen stack trace from rsp=ffff8462efce7da0:
(XEN) [  169.075579]    0000000000000048 0000000000000286 ffff8382cd650f08 000000000000004f
(XEN) [  169.139205]    ffff82d080488f20 00000027216dad05 000000272167bef6 0000000000000008
(XEN) [  169.202660]    ffff8462efce7e00 ffff82d08022c1f4 000000000000004f ffff8382cd650e70
(XEN) [  169.265928]    ffff8462efce7e20 ffff82d080241dfe 000000272167bef6 ffff8382cd650f08
(XEN) [  169.329022]    ffff8462efce7ea0 ffff82d0802eda52 ffff8462000000ff 00000020802ae6a5
(XEN) [  169.392015]    ffff8462efce7e60 ffff82d0802ae7d1 ffff82d0805c0200 0000000000000000
(XEN) [  169.455119]    0000000000000000 0000000000000000 0000000000002780 0000000000002780
(XEN) [  169.518238]    ffff82d0805bda80 000000000000004f ffff82d0805b7270 000000000000004f
(XEN) [  169.581477]    ffff8462efce7ef0 ffff82d08027939e 0000000000000000 0000004f000000c8
(XEN) [  169.644501]    ffff8382cd64a000 ffff8382cd64a000 ffff8382cd50ab30 000000000000004f
(XEN) [  169.707365]    0000000000000000 0000000000000000 ffff8462efce7ec0 0000000000000000
(XEN) [  169.770036]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  169.832519]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  169.894813]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  169.956916]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  170.018831]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  170.080574]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  170.142124]    0000e0100000004f ffff8382cd64a000 000000b24d08e000 00000000003526e0
(XEN) [  170.203508]    0000000000000000 0000000000000000 0000060100000000 0000000000000000
(XEN) [  170.264698] Xen call trace:
(XEN) [  170.321356]    [<ffff82d0802467dc>] set_timer+0x39/0x1f7
(XEN) [  170.378154]    [<ffff82d08022c1f4>] sched_credit.c#csched_tick_resume+0x54/0x59
(XEN) [  170.435049]    [<ffff82d080241dfe>] sched_tick_resume+0x67/0x86
(XEN) [  170.491383]    [<ffff82d0802eda52>] mwait-idle.c#mwait_idle+0x32b/0x357
(XEN) [  170.547579]    [<ffff82d08027939e>] domain.c#idle_loop+0xa6/0xc2
(XEN) [  170.603301]
(XEN) [  170.657509] Running stub recovery selftests...
(XEN) [  170.701789] Pagetable walk from 0000000000000048:
(XEN) [  170.755603] traps.c:1564: GPF (0000): ffff82d0bffff041 [ffff82d0bffff041] -> ffff82d0803893f2
(XEN) [  170.800367]  L4[0x000] = 00000082cfb9c063 ffffffffffffffff
(XEN) [  170.854150] traps.c:759: Trap 12: ffff82d0bffff040 [ffff82d0bffff040] -> ffff82d0803893f2
(XEN) [  170.900493]  L3[0x000] = 00000082cfb9b063 ffffffffffffffff
(XEN) [  170.954060] traps.c:1098: Trap 3: ffff82d0bffff041 [ffff82d0bffff041] -> ffff82d0803893f2
(XEN) [  170.998631]  L2[0x000] = 00000082cfb9a063 ffffffffffffffff
(XEN) [  171.058140]  L1[0x000] = 0000000000000000 ffffffffffffffff
(XEN) [  171.130737]
(XEN) [  171.190190] ****************************************
(XEN) [  171.254241] Panic on CPU 79:
(XEN) [  171.315034] FATAL PAGE FAULT
(XEN) [  171.375744] [error_code=0000]
(XEN) [  171.436129] Faulting linear address: 0000000000000048
(XEN) [  171.499198] ****************************************

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v6 00/20] xen: add core scheduling support
Posted by Jürgen Groß 4 years, 5 months ago
On 03.10.19 11:47, Sergey Dyasli wrote:
> Hi Juergen,
> 
> Looks like we've hit the first Xen crash with core scheduling patches applied.
> The log is below. From my analysis it seems that CSCHED_PCPU is NULL.
> I suspect this is connected to commit b0000b128adb
> ("sched: populate cpupool0 only after all cpus are up")
> 
> Could you take a look, please?

The main reason is that sched_tick_resume() should call
sched_do_tick_resume() only with the scheduling lock held.

This has been a latent bug since ages, but my patches (especially
"sched: add minimalistic idle scheduler for free cpus" in combination
with "sched: populate cpupool0 only after all cpus are up") is
triggering it much easier now.

In the past you'd need to remove a cpu from a cpupool with null, rt or
arinc653 scheduler with default scheduler being credit in order to have
a chance hitting the bug.

I'll send a patch.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel