The following sequence of events may lead a debug build of Xen to crash
when using the null scheduler:
1. domain creation (e.g. d1) failed due to bad configuration
2. complete_domain_destroy() was deferred
3. domain creation (e.g. d2) succeeds
At this point, d2 is running, while the zombie d1 is not fully cleaned
up:
(XEN) Online Cpus: 0-3
(XEN) Cpupool 0:
(XEN) Cpus: 0-3
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Scheduler: null Scheduler (null)
(XEN) cpus_free = 3
(XEN) Domain info:
(XEN) Domain: 0
(XEN) 1: [0.0] pcpu=0
(XEN) 2: [0.1] pcpu=1
(XEN) Domain: 1
(XEN) 3: [1.0] pcpu=2
(XEN) Domain: 2
(XEN) 4: [2.0] pcpu=2
4. complete_domain_destroy() gets called for d1 and triggers the
following:
(XEN) Xen call trace:
(XEN) [<00000a0000322ed4>] null.c#unit_deassign+0x2d8/0xb70 (PC)
(XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 (LR)
(XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8
(XEN) [<00000a0000336404>] sched_destroy_vcpu+0x354/0x8fc
(XEN) [<00000a0000227324>] domain.c#complete_domain_destroy+0x11c/0x49c
(XEN) [<00000a000029fbd0>] rcupdate.c#rcu_do_batch+0x94/0x3d0
(XEN) [<00000a00002a10c0>] rcupdate.c#__rcu_process_callbacks+0x160/0x5f4
(XEN) [<00000a00002a1e60>] rcupdate.c#rcu_process_callbacks+0xcc/0x1b0
(XEN) [<00000a00002a3460>] softirq.c#__do_softirq+0x1f4/0x3d8
(XEN) [<00000a00002a37c4>] do_softirq+0x14/0x1c
(XEN) [<00000a0000465260>] traps.c#check_for_pcpu_work+0x30/0xb8
(XEN) [<00000a000046bb08>] leave_hypervisor_to_guest+0x28/0x198
(XEN) [<00000a0000409c84>] entry.o#guest_sync_slowpath+0xac/0xd8
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion 'npc->unit == unit' failed at common/sched/null.c:383
(XEN) ****************************************
Fix by skipping unit_deassign() when the unit to be removed does not
match the pcpu's currently assigned unit.
Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
---
See c2eae2614c8f ("sched/null: avoid crash after failed domU creation")
Another consequence of deferring complete_domain_destroy() is that
eventually domains may fail to be scheduled by the null scheduler,
despite domains/vcpus having previously been running on available pcpus
have been destroyed.
(XEN) common/sched/null.c:639: WARNING: d4v0 not assigned to any CPU!
... but that is a problem for another day. After all, I'm not sure how
common of a use case dynamic domain creation & destruction really is for
the null scheduler.
---
xen/common/sched/null.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/xen/common/sched/null.c b/xen/common/sched/null.c
index 7e31440e5b84..c8e327e3cdd0 100644
--- a/xen/common/sched/null.c
+++ b/xen/common/sched/null.c
@@ -557,7 +557,7 @@ static void cf_check null_unit_remove(
cpu = sched_unit_master(unit);
npc = get_sched_res(cpu)->sched_priv;
- if ( npc->unit )
+ if ( npc->unit == unit )
unit_deassign(prv, unit);
out:
base-commit: befc384d21784affa3daf2abc85b02500e4dc545
--
2.49.0On 03.04.25 23:48, Stewart Hildebrand wrote: > The following sequence of events may lead a debug build of Xen to crash > when using the null scheduler: > > 1. domain creation (e.g. d1) failed due to bad configuration > 2. complete_domain_destroy() was deferred > 3. domain creation (e.g. d2) succeeds > > At this point, d2 is running, while the zombie d1 is not fully cleaned > up: > > (XEN) Online Cpus: 0-3 > (XEN) Cpupool 0: > (XEN) Cpus: 0-3 > (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource > (XEN) Scheduler: null Scheduler (null) > (XEN) cpus_free = 3 > (XEN) Domain info: > (XEN) Domain: 0 > (XEN) 1: [0.0] pcpu=0 > (XEN) 2: [0.1] pcpu=1 > (XEN) Domain: 1 > (XEN) 3: [1.0] pcpu=2 > (XEN) Domain: 2 > (XEN) 4: [2.0] pcpu=2 > > 4. complete_domain_destroy() gets called for d1 and triggers the > following: > > (XEN) Xen call trace: > (XEN) [<00000a0000322ed4>] null.c#unit_deassign+0x2d8/0xb70 (PC) > (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 (LR) > (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 > (XEN) [<00000a0000336404>] sched_destroy_vcpu+0x354/0x8fc > (XEN) [<00000a0000227324>] domain.c#complete_domain_destroy+0x11c/0x49c > (XEN) [<00000a000029fbd0>] rcupdate.c#rcu_do_batch+0x94/0x3d0 > (XEN) [<00000a00002a10c0>] rcupdate.c#__rcu_process_callbacks+0x160/0x5f4 > (XEN) [<00000a00002a1e60>] rcupdate.c#rcu_process_callbacks+0xcc/0x1b0 > (XEN) [<00000a00002a3460>] softirq.c#__do_softirq+0x1f4/0x3d8 > (XEN) [<00000a00002a37c4>] do_softirq+0x14/0x1c > (XEN) [<00000a0000465260>] traps.c#check_for_pcpu_work+0x30/0xb8 > (XEN) [<00000a000046bb08>] leave_hypervisor_to_guest+0x28/0x198 > (XEN) [<00000a0000409c84>] entry.o#guest_sync_slowpath+0xac/0xd8 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Assertion 'npc->unit == unit' failed at common/sched/null.c:383 > (XEN) **************************************** > > Fix by skipping unit_deassign() when the unit to be removed does not > match the pcpu's currently assigned unit. > > Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com> Reviewed-by: Juergen Gross <jgross@suse.com> Juergen
On 03.04.2025 23:48, Stewart Hildebrand wrote: > The following sequence of events may lead a debug build of Xen to crash > when using the null scheduler: > > 1. domain creation (e.g. d1) failed due to bad configuration > 2. complete_domain_destroy() was deferred > 3. domain creation (e.g. d2) succeeds > > At this point, d2 is running, while the zombie d1 is not fully cleaned > up: > > (XEN) Online Cpus: 0-3 > (XEN) Cpupool 0: > (XEN) Cpus: 0-3 > (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource > (XEN) Scheduler: null Scheduler (null) > (XEN) cpus_free = 3 > (XEN) Domain info: > (XEN) Domain: 0 > (XEN) 1: [0.0] pcpu=0 > (XEN) 2: [0.1] pcpu=1 > (XEN) Domain: 1 > (XEN) 3: [1.0] pcpu=2 > (XEN) Domain: 2 > (XEN) 4: [2.0] pcpu=2 > > 4. complete_domain_destroy() gets called for d1 and triggers the > following: > > (XEN) Xen call trace: > (XEN) [<00000a0000322ed4>] null.c#unit_deassign+0x2d8/0xb70 (PC) > (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 (LR) > (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 > (XEN) [<00000a0000336404>] sched_destroy_vcpu+0x354/0x8fc > (XEN) [<00000a0000227324>] domain.c#complete_domain_destroy+0x11c/0x49c > (XEN) [<00000a000029fbd0>] rcupdate.c#rcu_do_batch+0x94/0x3d0 > (XEN) [<00000a00002a10c0>] rcupdate.c#__rcu_process_callbacks+0x160/0x5f4 > (XEN) [<00000a00002a1e60>] rcupdate.c#rcu_process_callbacks+0xcc/0x1b0 > (XEN) [<00000a00002a3460>] softirq.c#__do_softirq+0x1f4/0x3d8 > (XEN) [<00000a00002a37c4>] do_softirq+0x14/0x1c > (XEN) [<00000a0000465260>] traps.c#check_for_pcpu_work+0x30/0xb8 > (XEN) [<00000a000046bb08>] leave_hypervisor_to_guest+0x28/0x198 > (XEN) [<00000a0000409c84>] entry.o#guest_sync_slowpath+0xac/0xd8 > (XEN) > (XEN) **************************************** > (XEN) Panic on CPU 0: > (XEN) Assertion 'npc->unit == unit' failed at common/sched/null.c:383 > (XEN) **************************************** > > Fix by skipping unit_deassign() when the unit to be removed does not > match the pcpu's currently assigned unit. > > Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com> Just one remark here: This almost certainly wants a Fixes: tag. Jan
On 04.04.25 09:24, Jan Beulich wrote: > On 03.04.2025 23:48, Stewart Hildebrand wrote: >> The following sequence of events may lead a debug build of Xen to crash >> when using the null scheduler: >> >> 1. domain creation (e.g. d1) failed due to bad configuration >> 2. complete_domain_destroy() was deferred >> 3. domain creation (e.g. d2) succeeds >> >> At this point, d2 is running, while the zombie d1 is not fully cleaned >> up: >> >> (XEN) Online Cpus: 0-3 >> (XEN) Cpupool 0: >> (XEN) Cpus: 0-3 >> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource >> (XEN) Scheduler: null Scheduler (null) >> (XEN) cpus_free = 3 >> (XEN) Domain info: >> (XEN) Domain: 0 >> (XEN) 1: [0.0] pcpu=0 >> (XEN) 2: [0.1] pcpu=1 >> (XEN) Domain: 1 >> (XEN) 3: [1.0] pcpu=2 >> (XEN) Domain: 2 >> (XEN) 4: [2.0] pcpu=2 >> >> 4. complete_domain_destroy() gets called for d1 and triggers the >> following: >> >> (XEN) Xen call trace: >> (XEN) [<00000a0000322ed4>] null.c#unit_deassign+0x2d8/0xb70 (PC) >> (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 (LR) >> (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 >> (XEN) [<00000a0000336404>] sched_destroy_vcpu+0x354/0x8fc >> (XEN) [<00000a0000227324>] domain.c#complete_domain_destroy+0x11c/0x49c >> (XEN) [<00000a000029fbd0>] rcupdate.c#rcu_do_batch+0x94/0x3d0 >> (XEN) [<00000a00002a10c0>] rcupdate.c#__rcu_process_callbacks+0x160/0x5f4 >> (XEN) [<00000a00002a1e60>] rcupdate.c#rcu_process_callbacks+0xcc/0x1b0 >> (XEN) [<00000a00002a3460>] softirq.c#__do_softirq+0x1f4/0x3d8 >> (XEN) [<00000a00002a37c4>] do_softirq+0x14/0x1c >> (XEN) [<00000a0000465260>] traps.c#check_for_pcpu_work+0x30/0xb8 >> (XEN) [<00000a000046bb08>] leave_hypervisor_to_guest+0x28/0x198 >> (XEN) [<00000a0000409c84>] entry.o#guest_sync_slowpath+0xac/0xd8 >> (XEN) >> (XEN) **************************************** >> (XEN) Panic on CPU 0: >> (XEN) Assertion 'npc->unit == unit' failed at common/sched/null.c:383 >> (XEN) **************************************** >> >> Fix by skipping unit_deassign() when the unit to be removed does not >> match the pcpu's currently assigned unit. >> >> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com> > > Just one remark here: This almost certainly wants a Fixes: tag. This will probably be d671da3be3ca (the initial NULL scheduler commit). Juergen
On 04.04.2025 09:38, Jürgen Groß wrote:
> On 04.04.25 09:24, Jan Beulich wrote:
>> On 03.04.2025 23:48, Stewart Hildebrand wrote:
>>> The following sequence of events may lead a debug build of Xen to crash
>>> when using the null scheduler:
>>>
>>> 1. domain creation (e.g. d1) failed due to bad configuration
>>> 2. complete_domain_destroy() was deferred
>>> 3. domain creation (e.g. d2) succeeds
>>>
>>> At this point, d2 is running, while the zombie d1 is not fully cleaned
>>> up:
>>>
>>> (XEN) Online Cpus: 0-3
>>> (XEN) Cpupool 0:
>>> (XEN) Cpus: 0-3
>>> (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
>>> (XEN) Scheduler: null Scheduler (null)
>>> (XEN) cpus_free = 3
>>> (XEN) Domain info:
>>> (XEN) Domain: 0
>>> (XEN) 1: [0.0] pcpu=0
>>> (XEN) 2: [0.1] pcpu=1
>>> (XEN) Domain: 1
>>> (XEN) 3: [1.0] pcpu=2
>>> (XEN) Domain: 2
>>> (XEN) 4: [2.0] pcpu=2
>>>
>>> 4. complete_domain_destroy() gets called for d1 and triggers the
>>> following:
>>>
>>> (XEN) Xen call trace:
>>> (XEN) [<00000a0000322ed4>] null.c#unit_deassign+0x2d8/0xb70 (PC)
>>> (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8 (LR)
>>> (XEN) [<00000a000032457c>] null.c#null_unit_remove+0x670/0xba8
>>> (XEN) [<00000a0000336404>] sched_destroy_vcpu+0x354/0x8fc
>>> (XEN) [<00000a0000227324>] domain.c#complete_domain_destroy+0x11c/0x49c
>>> (XEN) [<00000a000029fbd0>] rcupdate.c#rcu_do_batch+0x94/0x3d0
>>> (XEN) [<00000a00002a10c0>] rcupdate.c#__rcu_process_callbacks+0x160/0x5f4
>>> (XEN) [<00000a00002a1e60>] rcupdate.c#rcu_process_callbacks+0xcc/0x1b0
>>> (XEN) [<00000a00002a3460>] softirq.c#__do_softirq+0x1f4/0x3d8
>>> (XEN) [<00000a00002a37c4>] do_softirq+0x14/0x1c
>>> (XEN) [<00000a0000465260>] traps.c#check_for_pcpu_work+0x30/0xb8
>>> (XEN) [<00000a000046bb08>] leave_hypervisor_to_guest+0x28/0x198
>>> (XEN) [<00000a0000409c84>] entry.o#guest_sync_slowpath+0xac/0xd8
>>> (XEN)
>>> (XEN) ****************************************
>>> (XEN) Panic on CPU 0:
>>> (XEN) Assertion 'npc->unit == unit' failed at common/sched/null.c:383
>>> (XEN) ****************************************
>>>
>>> Fix by skipping unit_deassign() when the unit to be removed does not
>>> match the pcpu's currently assigned unit.
>>>
>>> Signed-off-by: Stewart Hildebrand <stewart.hildebrand@amd.com>
>>
>> Just one remark here: This almost certainly wants a Fixes: tag.
>
> This will probably be d671da3be3ca (the initial NULL scheduler commit).
The code in question was added by c2eae2614c8f ("sched/null: avoid crash
after failed domU creation").
Jan
© 2016 - 2025 Red Hat, Inc.