From nobody Wed May 1 02:11:36 2024 Delivered-To: importer@patchew.org Received-SPF: none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1583238701; cv=none; d=zohomail.com; s=zohoarc; b=hCJ7pWKRFUDGHqB7RE3lADV4asA0X5iSoDrQeDTORlYgz3B59SWhUDOF/Q0g+YgXUO1u1OsW7DZzoeayWGkWDsWj06plwZfxAYS7t7P+kujqf2He33M6wt59PCA3JH/xsQ1FhoGSNF7KGe8R5DfWO9mR3yna4axYQK3QA5PiprM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1583238701; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=8VrEBH2SGiBeGorMY6bTTPRkJvbnB+mw23OwvSUbL1A=; b=MY40gyESPNHNJZAnk9Wl6tEe7i9l9skFHglpfK59YGLij95ywVA/74GZcZC8eYdpRZx32ThjqXr+k3vprScbQAIvO5zj0zskeRMwdMNgGR37fF7qMqm9gd735ERn4gs62vuYpE6LhrAKz1YFCw80j1l7IOsAOU9QCOaDV7mRv9k= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1583238701266140.9790947592977; Tue, 3 Mar 2020 04:31:41 -0800 (PST) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j96hU-00058o-Ax; Tue, 03 Mar 2020 12:31:04 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1j96hS-00058j-US for xen-devel@lists.xenproject.org; Tue, 03 Mar 2020 12:31:02 +0000 Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id d77654ba-5d4a-11ea-82f6-bc764e2007e4; Tue, 03 Mar 2020 12:31:01 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id DA885B109; Tue, 3 Mar 2020 12:31:00 +0000 (UTC) X-Inumbo-ID: d77654ba-5d4a-11ea-82f6-bc764e2007e4 X-Virus-Scanned: by amavisd-new at test-mx.suse.de From: Juergen Gross To: xen-devel@lists.xenproject.org Date: Tue, 3 Mar 2020 13:30:58 +0100 Message-Id: <20200303123058.27210-1-jgross@suse.com> X-Mailer: git-send-email 2.16.4 Subject: [Xen-devel] [PATCH] xen/sched: fix cpu offlining with core scheduling X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , George Dunlap , Dario Faggioli MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Offlining a cpu with core scheduling active can result in a hanging system. Reason is the scheduling resource and unit of the to be removed cpus needs to be split in order to remove the cpu from its cpupool and move it to the idle scheduler. In case one of the involved cpus happens to have received a sched slave event due to a vcpu former having been running on that cpu being woken up again, it can happen that this cpu will enter sched_wait_rendezvous_in() while its scheduling resource is just about to be split. It might wait for ever for the other sibling to join, which will never happen due to the resources already being modified. This can easily be avoided by: - resetting the rendezvous counters of the idle unit which is kept - checking for a new scheduling resource in sched_wait_rendezvous_in() after reacquiring the scheduling lock and resetting the counters in that case without scheduling another vcpu - moving schedule resource modifications (in schedule_cpu_rm()) and retrieving (schedule(), sched_slave() is fine already, others are not critical) into locked regions Reported-by: Igor Druzhinin Signed-off-by: Juergen Gross --- xen/common/sched/core.c | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c index 7e8e7d2c39..723283ed00 100644 --- a/xen/common/sched/core.c +++ b/xen/common/sched/core.c @@ -2415,7 +2415,8 @@ static struct sched_unit *sched_wait_rendezvous_in(st= ruct sched_unit *prev, { struct sched_unit *next; struct vcpu *v; - unsigned int gran =3D get_sched_res(cpu)->granularity; + struct sched_resource *sr =3D get_sched_res(cpu); + unsigned int gran =3D sr->granularity; =20 if ( !--prev->rendezvous_in_cnt ) { @@ -2482,6 +2483,19 @@ static struct sched_unit *sched_wait_rendezvous_in(s= truct sched_unit *prev, atomic_set(&prev->next_task->rendezvous_out_cnt, 0); prev->rendezvous_in_cnt =3D 0; } + + /* + * Check for scheduling resourced switched. This happens when we a= re + * moved away from our cpupool and cpus are subject of the idle + * scheduler now. + */ + if ( unlikely(sr !=3D get_sched_res(cpu)) ) + { + ASSERT(is_idle_unit(prev)); + atomic_set(&prev->next_task->rendezvous_out_cnt, 0); + prev->rendezvous_in_cnt =3D 0; + return NULL; + } } =20 return prev->next_task; @@ -2538,7 +2552,10 @@ static void sched_slave(void) =20 next =3D sched_wait_rendezvous_in(prev, &lock, cpu, now); if ( !next ) + { + rcu_read_unlock(&sched_res_rculock); return; + } =20 pcpu_schedule_unlock_irq(lock, cpu); =20 @@ -2567,11 +2584,11 @@ static void schedule(void) =20 rcu_read_lock(&sched_res_rculock); =20 + lock =3D pcpu_schedule_lock_irq(cpu); + sr =3D get_sched_res(cpu); gran =3D sr->granularity; =20 - lock =3D pcpu_schedule_lock_irq(cpu); - if ( prev->rendezvous_in_cnt ) { /* @@ -2599,7 +2616,10 @@ static void schedule(void) cpumask_raise_softirq(mask, SCHED_SLAVE_SOFTIRQ); next =3D sched_wait_rendezvous_in(prev, &lock, cpu, now); if ( !next ) + { + rcu_read_unlock(&sched_res_rculock); return; + } } else { @@ -3151,7 +3171,10 @@ int schedule_cpu_rm(unsigned int cpu) per_cpu(sched_res_idx, cpu_iter) =3D 0; if ( cpu_iter =3D=3D cpu ) { - idle_vcpu[cpu_iter]->sched_unit->priv =3D NULL; + unit =3D idle_vcpu[cpu_iter]->sched_unit; + unit->priv =3D NULL; + atomic_set(&unit->next_task->rendezvous_out_cnt, 0); + unit->rendezvous_in_cnt =3D 0; } else { @@ -3182,6 +3205,8 @@ int schedule_cpu_rm(unsigned int cpu) } sr->scheduler =3D &sched_idle_ops; sr->sched_priv =3D NULL; + sr->granularity =3D 1; + sr->cpupool =3D NULL; =20 smp_mb(); sr->schedule_lock =3D &sched_free_cpu_lock; @@ -3194,9 +3219,6 @@ int schedule_cpu_rm(unsigned int cpu) sched_free_udata(old_ops, vpriv_old); sched_free_pdata(old_ops, ppriv_old, cpu); =20 - sr->granularity =3D 1; - sr->cpupool =3D NULL; - out: rcu_read_unlock(&sched_res_rculock); xfree(sr_new); --=20 2.16.4 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel