From nobody Sat May 4 11:42:48 2024 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1564122418; cv=none; d=zoho.com; s=zohoarc; b=gb7NArOoj9Z51stw8ZufCrZ0c1wBaXZkrSeUxlwQHAH/kmagfgCjb2nzP2/Vupj3IIsJtM+PnxoB+VIOJ5MebtZVAGiPKbJRH5DsR2fk2kJEREtcpEk4Vb5pDslZ2EHLxTsicnIqf/r8cwV2z3J7rw1dVOiBqjiqDKeXXMLo1Vk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564122418; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=L/Avbx22qE7G5slZ+2rtTpVKxlNCPuXeCZHP/RnFNK8=; b=OrqrjdHIN1mnICVvrfysSHD/1i1omCIEdb8aAbkywJ9c7oDdBrWFqcDUpXedMj6c0209fHtYZZ68vplJVG5J0LcNjUaBkUWKej8NIWs3RHDVG29yUr7LTY0iIDQUTo5UVboJfsKG0HWS3hwSrfQIRpniLsvqKnXkJyldxQOSGi8= ARC-Authentication-Results: i=1; mx.zoho.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1564122418781814.4652306395029; Thu, 25 Jul 2019 23:26:58 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtfv-00038Q-9q; Fri, 26 Jul 2019 06:25:55 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtft-00038K-NA for xen-devel@lists.xenproject.org; Fri, 26 Jul 2019 06:25:53 +0000 Received: from mx1.suse.de (unknown [195.135.220.15]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 3730ec1e-af6e-11e9-8980-bc764e045a96; Fri, 26 Jul 2019 06:25:52 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 924C1AD31; Fri, 26 Jul 2019 06:25:51 +0000 (UTC) X-Inumbo-ID: 3730ec1e-af6e-11e9-8980-bc764e045a96 X-Virus-Scanned: by amavisd-new at test-mx.suse.de From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 26 Jul 2019 08:25:51 +0200 Message-ID: <156412235104.2385.3911161728130674771.stgit@Palanthas> In-Reply-To: <156412188377.2385.12588508835559819141.stgit@Palanthas> References: <156412188377.2385.12588508835559819141.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 1/4] xen: sched: refector code around vcpu_deassign() in null scheduler X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: George Dunlap Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" vcpu_deassign() is called only once (in _vcpu_remove()). Let's consolidate the two functions into one. No functional change intended. Signed-off-by: Dario Faggioli Acked-by: George Dunlap --- xen/common/sched_null.c | 76 ++++++++++++++++++++++---------------------= ---- 1 file changed, 35 insertions(+), 41 deletions(-) diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index c02c1b9c1f..c47c1b5aae 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -358,9 +358,14 @@ static void vcpu_assign(struct null_private *prv, stru= ct vcpu *v, } } =20 -static void vcpu_deassign(struct null_private *prv, struct vcpu *v, - unsigned int cpu) +static void vcpu_deassign(struct null_private *prv, struct vcpu *v) { + unsigned int bs; + unsigned int cpu =3D v->processor; + struct null_vcpu *wvc; + + ASSERT(list_empty(&null_vcpu(v)->waitq_elem)); + per_cpu(npc, cpu).vcpu =3D NULL; cpumask_set_cpu(cpu, &prv->cpus_free); =20 @@ -377,6 +382,32 @@ static void vcpu_deassign(struct null_private *prv, st= ruct vcpu *v, d.cpu =3D cpu; __trace_var(TRC_SNULL_VCPU_DEASSIGN, 1, sizeof(d), &d); } + + spin_lock(&prv->waitq_lock); + + /* + * If v is assigned to a pCPU, let's see if there is someone waiting, + * suitable to be assigned to it (prioritizing vcpus that have + * soft-affinity with cpu). + */ + for_each_affinity_balance_step( bs ) + { + list_for_each_entry( wvc, &prv->waitq, waitq_elem ) + { + if ( bs =3D=3D BALANCE_SOFT_AFFINITY && !has_soft_affinity(wvc= ->vcpu) ) + continue; + + if ( vcpu_check_affinity(wvc->vcpu, cpu, bs) ) + { + list_del_init(&wvc->waitq_elem); + vcpu_assign(prv, wvc->vcpu, cpu); + cpu_raise_softirq(cpu, SCHEDULE_SOFTIRQ); + spin_unlock(&prv->waitq_lock); + return; + } + } + } + spin_unlock(&prv->waitq_lock); } =20 /* Change the scheduler of cpu to us (null). */ @@ -459,43 +490,6 @@ static void null_vcpu_insert(const struct scheduler *o= ps, struct vcpu *v) SCHED_STAT_CRANK(vcpu_insert); } =20 -static void _vcpu_remove(struct null_private *prv, struct vcpu *v) -{ - unsigned int bs; - unsigned int cpu =3D v->processor; - struct null_vcpu *wvc; - - ASSERT(list_empty(&null_vcpu(v)->waitq_elem)); - - vcpu_deassign(prv, v, cpu); - - spin_lock(&prv->waitq_lock); - - /* - * If v is assigned to a pCPU, let's see if there is someone waiting, - * suitable to be assigned to it (prioritizing vcpus that have - * soft-affinity with cpu). - */ - for_each_affinity_balance_step( bs ) - { - list_for_each_entry( wvc, &prv->waitq, waitq_elem ) - { - if ( bs =3D=3D BALANCE_SOFT_AFFINITY && !has_soft_affinity(wvc= ->vcpu) ) - continue; - - if ( vcpu_check_affinity(wvc->vcpu, cpu, bs) ) - { - list_del_init(&wvc->waitq_elem); - vcpu_assign(prv, wvc->vcpu, cpu); - cpu_raise_softirq(cpu, SCHEDULE_SOFTIRQ); - spin_unlock(&prv->waitq_lock); - return; - } - } - } - spin_unlock(&prv->waitq_lock); -} - static void null_vcpu_remove(const struct scheduler *ops, struct vcpu *v) { struct null_private *prv =3D null_priv(ops); @@ -519,7 +513,7 @@ static void null_vcpu_remove(const struct scheduler *op= s, struct vcpu *v) ASSERT(per_cpu(npc, v->processor).vcpu =3D=3D v); ASSERT(!cpumask_test_cpu(v->processor, &prv->cpus_free)); =20 - _vcpu_remove(prv, v); + vcpu_deassign(prv, v); =20 out: vcpu_schedule_unlock_irq(lock, v); @@ -605,7 +599,7 @@ static void null_vcpu_migrate(const struct scheduler *o= ps, struct vcpu *v, */ if ( likely(list_empty(&nvc->waitq_elem)) ) { - _vcpu_remove(prv, v); + vcpu_deassign(prv, v); SCHED_STAT_CRANK(migrate_running); } else _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel From nobody Sat May 4 11:42:48 2024 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1564122428; cv=none; d=zoho.com; s=zohoarc; b=CkkV62m/PPc7sUQSe8WufJMk6FFg1IUE57Lm7wmdCLcDpZEgaMwNN/gGcjAYmsx4hTAPHLOyn/U0K36DCX7I7XmsBHIdtnbaMfiaBIkNA0emqpaB8Wa2WCx0lVJzRdRPMvH79AJF1GRyr5EWPHIju+OUJrI+MOJ6W28NRlm1/KI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564122428; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=TNWkDdoLteBkzHPxsLdNRe2cMNUDNeKHXqupZ3JMqEI=; b=i0erBHC1V4hsfKJl8vZVY60/n1RBnRsV5CeB18JVMWGQ4zpqyLIVXdn3ohaHsukdUGOPlWx4J/36/nxjZ1C10TTUW1mO004pwpHaGSyqZDMKQYoyHImU2d+LhCE22SBi2U8ou2UvH2Ky5klhJ3hIzlZFgP7w+e/IRcIhnuc6PXM= ARC-Authentication-Results: i=1; mx.zoho.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1564122428020129.11027203932917; Thu, 25 Jul 2019 23:27:08 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtg0-00039d-J5; Fri, 26 Jul 2019 06:26:00 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtfz-00039P-J8 for xen-devel@lists.xenproject.org; Fri, 26 Jul 2019 06:25:59 +0000 Received: from mx1.suse.de (unknown [195.135.220.15]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 3a8deccd-af6e-11e9-8980-bc764e045a96; Fri, 26 Jul 2019 06:25:58 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 3B627AD31; Fri, 26 Jul 2019 06:25:57 +0000 (UTC) X-Inumbo-ID: 3a8deccd-af6e-11e9-8980-bc764e045a96 X-Virus-Scanned: by amavisd-new at test-mx.suse.de From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 26 Jul 2019 08:25:56 +0200 Message-ID: <156412235656.2385.13861979113936528474.stgit@Palanthas> In-Reply-To: <156412188377.2385.12588508835559819141.stgit@Palanthas> References: <156412188377.2385.12588508835559819141.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 2/4] xen: sched: deal with vCPUs being or becoming online or offline X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: George Dunlap , Stefano Stabellini , Roger Pau Monne Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" If a vCPU is, or is going, offline we want it to be neither assigned to a pCPU, nor in the wait list, so: - if an offline vcpu is inserted (or migrated) it must not go on a pCPU, nor in the wait list; - if an offline vcpu is removed, we are sure that it is neither on a pCPU nor in the wait list already, so we should just bail, avoiding doing any further action; - if a vCPU goes offline we need to remove it either from its pCPU or from the wait list. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: George Dunlap Cc: Stefano Stabellini Cc: Roger Pau Monne --- Changes from v1: * improved wording in changelog and comments * this patch is the result of the merge of patches 2 and 3 from v1 --- xen/common/sched_null.c | 80 +++++++++++++++++++++++++++++++++++++++----= ---- 1 file changed, 66 insertions(+), 14 deletions(-) diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index c47c1b5aae..10e96f21dd 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -339,6 +339,8 @@ static unsigned int pick_cpu(struct null_private *prv, = struct vcpu *v) static void vcpu_assign(struct null_private *prv, struct vcpu *v, unsigned int cpu) { + ASSERT(is_vcpu_online(v)); + per_cpu(npc, cpu).vcpu =3D v; v->processor =3D cpu; cpumask_clear_cpu(cpu, &prv->cpus_free); @@ -358,7 +360,8 @@ static void vcpu_assign(struct null_private *prv, struc= t vcpu *v, } } =20 -static void vcpu_deassign(struct null_private *prv, struct vcpu *v) +/* Returns true if a cpu was tickled */ +static bool vcpu_deassign(struct null_private *prv, struct vcpu *v) { unsigned int bs; unsigned int cpu =3D v->processor; @@ -403,11 +406,13 @@ static void vcpu_deassign(struct null_private *prv, s= truct vcpu *v) vcpu_assign(prv, wvc->vcpu, cpu); cpu_raise_softirq(cpu, SCHEDULE_SOFTIRQ); spin_unlock(&prv->waitq_lock); - return; + return true; } } } spin_unlock(&prv->waitq_lock); + + return false; } =20 /* Change the scheduler of cpu to us (null). */ @@ -445,8 +450,14 @@ static void null_vcpu_insert(const struct scheduler *o= ps, struct vcpu *v) ASSERT(!is_idle_vcpu(v)); =20 lock =3D vcpu_schedule_lock_irq(v); - retry: =20 + if ( unlikely(!is_vcpu_online(v)) ) + { + vcpu_schedule_unlock_irq(lock, v); + return; + } + + retry: cpu =3D v->processor =3D pick_cpu(prv, v); =20 spin_unlock(lock); @@ -500,6 +511,14 @@ static void null_vcpu_remove(const struct scheduler *o= ps, struct vcpu *v) =20 lock =3D vcpu_schedule_lock_irq(v); =20 + /* If offline, the vcpu shouldn't be assigned, nor in the waitqueue */ + if ( unlikely(!is_vcpu_online(v)) ) + { + ASSERT(per_cpu(npc, v->processor).vcpu !=3D v); + ASSERT(list_empty(&nvc->waitq_elem)); + goto out; + } + /* If v is in waitqueue, just get it out of there and bail */ if ( unlikely(!list_empty(&nvc->waitq_elem)) ) { @@ -549,11 +568,33 @@ static void null_vcpu_wake(const struct scheduler *op= s, struct vcpu *v) =20 static void null_vcpu_sleep(const struct scheduler *ops, struct vcpu *v) { + struct null_private *prv =3D null_priv(ops); + unsigned int cpu =3D v->processor; + bool tickled =3D false; + ASSERT(!is_idle_vcpu(v)); =20 + /*=20 + * Check if the vcpu is in the process of being offlined. if yes, + * we need to remove it from either its pCPU or the waitqueue. + */ + if ( unlikely(!is_vcpu_online(v)) ) + { + struct null_vcpu *nvc =3D null_vcpu(v); + + if ( unlikely(!list_empty(&nvc->waitq_elem)) ) + { + spin_lock(&prv->waitq_lock); + list_del_init(&nvc->waitq_elem); + spin_unlock(&prv->waitq_lock); + } + else if ( per_cpu(npc, cpu).vcpu =3D=3D v ) + tickled =3D vcpu_deassign(prv, v); + } + /* If v is not assigned to a pCPU, or is not running, no need to bothe= r */ - if ( curr_on_cpu(v->processor) =3D=3D v ) - cpu_raise_softirq(v->processor, SCHEDULE_SOFTIRQ); + if ( likely(!tickled && curr_on_cpu(cpu) =3D=3D v) ) + cpu_raise_softirq(cpu, SCHEDULE_SOFTIRQ); =20 SCHED_STAT_CRANK(vcpu_sleep); } @@ -589,24 +630,34 @@ static void null_vcpu_migrate(const struct scheduler = *ops, struct vcpu *v, } =20 /* - * v is either assigned to a pCPU, or in the waitqueue. - * - * In the former case, the pCPU to which it was assigned would - * become free, and we, therefore, should check whether there is - * anyone in the waitqueue that can be assigned to it. - * - * In the latter, there is just nothing to do. + * If v is assigned to a pCPU, then such pCPU becomes free, and we + * should look in the waitqueue if anyone else can be assigned to it. */ - if ( likely(list_empty(&nvc->waitq_elem)) ) + if ( likely(per_cpu(npc, v->processor).vcpu =3D=3D v) ) { vcpu_deassign(prv, v); SCHED_STAT_CRANK(migrate_running); } - else + else if ( !list_empty(&nvc->waitq_elem) ) SCHED_STAT_CRANK(migrate_on_runq); =20 SCHED_STAT_CRANK(migrated); =20 + /* + * If a vcpu is (going) offline, we want it to be neither assigned + * to a pCPU, nor in the waitqueue. + * + * If it was on a cpu, we've removed it from there above. If it is + * in the waitqueue, we remove it from there now. And then we bail. + */ + if ( unlikely(!is_vcpu_online(v)) ) + { + spin_lock(&prv->waitq_lock); + list_del_init(&nvc->waitq_elem); + spin_unlock(&prv->waitq_lock); + goto out; + } + /* * Let's now consider new_cpu, which is where v is being sent. It can = be * either free, or have a vCPU already assigned to it. @@ -646,6 +697,7 @@ static void null_vcpu_migrate(const struct scheduler *o= ps, struct vcpu *v, * at least. In case of suspend, any temporary inconsistency caused * by this, will be fixed-up during resume. */ + out: v->processor =3D new_cpu; } =20 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel From nobody Sat May 4 11:42:48 2024 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1564122425; cv=none; d=zoho.com; s=zohoarc; b=nUi566S759a/D5552pONm8SEpyrLsxJEPBumVogGzCCoqoO86ebXnhpy8D78I9lDRVf4HZWjT6eLGImzW0iXe0QRES6boOtianpi3zJqsnSaF53uKvJpnWGSrrkZEVy6UyKWuNEypK48QGwhcLyBC5rEsA0xNH41cSPYPBpmXZA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564122425; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=wwY9moQ2eLRe2GQQ3XHaJd/dhPSU5JteZSERZgS+tVs=; b=mwwX9ts4XXRRp1J/oObBtuQCulbwvg99+y3JNf6+zW/7A6jfi9OTuP/zwCqk4GMpK74HHR5WerR4S2xGwQ2llUSkjCG6CGL0hHnnIqRohPubwEfYJFdVYPMuNEYjdX0/up+VZzIjqps6y89wddYXfxus588ksrVkgOk+2Ky80J8= ARC-Authentication-Results: i=1; mx.zoho.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1564122425992746.3474058596696; Thu, 25 Jul 2019 23:27:05 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtg5-0003BM-UK; Fri, 26 Jul 2019 06:26:05 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtg4-0003Az-Rk for xen-devel@lists.xenproject.org; Fri, 26 Jul 2019 06:26:04 +0000 Received: from mx1.suse.de (unknown [195.135.220.15]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 3de65e79-af6e-11e9-8980-bc764e045a96; Fri, 26 Jul 2019 06:26:03 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id D003EAD31; Fri, 26 Jul 2019 06:26:02 +0000 (UTC) X-Inumbo-ID: 3de65e79-af6e-11e9-8980-bc764e045a96 X-Virus-Scanned: by amavisd-new at test-mx.suse.de From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 26 Jul 2019 08:26:02 +0200 Message-ID: <156412236222.2385.236340632846050170.stgit@Palanthas> In-Reply-To: <156412188377.2385.12588508835559819141.stgit@Palanthas> References: <156412188377.2385.12588508835559819141.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 3/4] xen: sched: reassign vCPUs to pCPUs, when they come back online X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Roger Pau Monne , Stefano Stabellini , George Dunlap , Dario Faggioli Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When a vcpu that was offline, comes back online, we do want it to either be assigned to a pCPU, or go into the wait list. Detecting that a vcpu is coming back online is a bit tricky. Basically, if the vcpu is waking up, and is neither assigned to a pCPU, nor in the wait list, it must be coming back from offline. When this happens, we put it in the waitqueue, and we "tickle" an idle pCPU (if any), to go pick it up. Looking at the patch, it seems that the vcpu wakeup code is getting complex, and hence that it could potentially introduce latencies. However, all this new logic is triggered only by the case of a vcpu coming online, so, basically, the overhead during normal operations is just an additional 'if()'. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- Cc: Stefano Stabellini Cc: Roger Pau Monne --- xen/common/sched_null.c | 53 +++++++++++++++++++++++++++++++++++++++++++= ++-- 1 file changed, 51 insertions(+), 2 deletions(-) diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index 10e96f21dd..1bbcaf92b9 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -542,15 +542,19 @@ static void null_vcpu_remove(const struct scheduler *= ops, struct vcpu *v) =20 static void null_vcpu_wake(const struct scheduler *ops, struct vcpu *v) { + struct null_private *prv =3D null_priv(ops); + struct null_vcpu *nvc =3D null_vcpu(v); + unsigned int cpu =3D v->processor; + ASSERT(!is_idle_vcpu(v)); =20 - if ( unlikely(curr_on_cpu(v->processor) =3D=3D v) ) + if ( unlikely(curr_on_cpu(cpu) =3D=3D v) ) { SCHED_STAT_CRANK(vcpu_wake_running); return; } =20 - if ( unlikely(!list_empty(&null_vcpu(v)->waitq_elem)) ) + if ( unlikely(!list_empty(&nvc->waitq_elem)) ) { /* Not exactly "on runq", but close enough for reusing the counter= */ SCHED_STAT_CRANK(vcpu_wake_onrunq); @@ -562,6 +566,48 @@ static void null_vcpu_wake(const struct scheduler *ops= , struct vcpu *v) else SCHED_STAT_CRANK(vcpu_wake_not_runnable); =20 + /* + * If a vcpu is neither on a pCPU nor in the waitqueue, it means it was + * offline, and that it is now coming back being online. + */ + if ( unlikely(per_cpu(npc, cpu).vcpu !=3D v && list_empty(&nvc->waitq_= elem)) ) + { + spin_lock(&prv->waitq_lock); + list_add_tail(&nvc->waitq_elem, &prv->waitq); + spin_unlock(&prv->waitq_lock); + + cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity, + cpupool_domain_cpumask(v->domain)); + + if ( !cpumask_intersects(&prv->cpus_free, cpumask_scratch_cpu(cpu)= ) ) + { + dprintk(XENLOG_G_WARNING, "WARNING: d%dv%d not assigned to any= CPU!\n", + v->domain->domain_id, v->vcpu_id); + return; + } + + /* + * Now we would want to assign the vcpu to cpu, but we can't, beca= use + * we don't have the lock. So, let's do the following: + * - try to remove cpu from the list of free cpus, to avoid races = with + * other onlining, inserting or migrating operations; + * - tickle the cpu, which will pickup work from the waitqueue, and + * assign it to itself; + * - if we're racing already, and if there still are free cpus, try + * again. + */ + while ( cpumask_intersects(&prv->cpus_free, cpumask_scratch_cpu(cp= u)) ) + { + unsigned int new_cpu =3D pick_cpu(prv, v); + + if ( test_and_clear_bit(new_cpu, &prv->cpus_free) ) + { + cpu_raise_softirq(new_cpu, SCHEDULE_SOFTIRQ); + return; + } + } + } + /* Note that we get here only for vCPUs assigned to a pCPU */ cpu_raise_softirq(v->processor, SCHEDULE_SOFTIRQ); } @@ -808,6 +854,9 @@ static struct task_slice null_schedule(const struct sch= eduler *ops, } unlock: spin_unlock(&prv->waitq_lock); + + if ( ret.task =3D=3D NULL && !cpumask_test_cpu(cpu, &prv->cpus_fre= e) ) + cpumask_set_cpu(cpu, &prv->cpus_free); } =20 if ( unlikely(ret.task =3D=3D NULL || !vcpu_runnable(ret.task)) ) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel From nobody Sat May 4 11:42:48 2024 Delivered-To: importer@patchew.org Received-SPF: none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org ARC-Seal: i=1; a=rsa-sha256; t=1564122436; cv=none; d=zoho.com; s=zohoarc; b=gkxdcDixJIJ3usJDRaGo0zFoH8FNg1R9aV8FW0IB+mdo+VfQFeWPLTiTZPr9H2DuRTZfXhDAXpWeDON8dk+qd4t4/FDW11iiwCL8otb+BcN4FJiSgh/2pGgVCHM4W8NuzC4zcQanFY0v3coyGKtqslg84MjerZQiP/wlEuGYzcU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564122436; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=vZA6PUELiVFIfQB6RiYQBXCT6+SAXTiaVvnFP+JRHew=; b=E26OFHMM2EO9qJlkGwpc3MUHTZNHpae/T9yeIAUR54H+Y/N+bimwV4rhFFjuTZnu1uvU7jqzobfaXMvKn13TTYVFR+bUQVNrQtWSWaiYihyj6NU6BxFNzQ3e+dlb3jEoXEuHmuTsjOuYyzUAqMg6CjIzWYNxyUe+g7cSS7y7tz8= ARC-Authentication-Results: i=1; mx.zoho.com; spf=none (zoho.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1564122436417202.03833259071666; Thu, 25 Jul 2019 23:27:16 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtgB-0003DW-7w; Fri, 26 Jul 2019 06:26:11 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hqtgA-0003D8-9K for xen-devel@lists.xenproject.org; Fri, 26 Jul 2019 06:26:10 +0000 Received: from mx1.suse.de (unknown [195.135.220.15]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 413df3ca-af6e-11e9-8980-bc764e045a96; Fri, 26 Jul 2019 06:26:09 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 73B98AD31; Fri, 26 Jul 2019 06:26:08 +0000 (UTC) X-Inumbo-ID: 413df3ca-af6e-11e9-8980-bc764e045a96 X-Virus-Scanned: by amavisd-new at test-mx.suse.de From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Fri, 26 Jul 2019 08:26:07 +0200 Message-ID: <156412236781.2385.9110155201477198899.stgit@Palanthas> In-Reply-To: <156412188377.2385.12588508835559819141.stgit@Palanthas> References: <156412188377.2385.12588508835559819141.stgit@Palanthas> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 4/4] xen: sched: refactor the ASSERTs around vcpu_deassing() X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: George Dunlap Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" It is all the time that we call vcpu_deassing() that the vcpu _must_ be assigned to a pCPU, and hence that such pCPU can't be free. Therefore, move the ASSERT-s which check for these properties in that function, where they belong better. Signed-off-by: Dario Faggioli Reviewed-by: George Dunlap --- xen/common/sched_null.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/xen/common/sched_null.c b/xen/common/sched_null.c index 1bbcaf92b9..c72335e5fa 100644 --- a/xen/common/sched_null.c +++ b/xen/common/sched_null.c @@ -368,6 +368,8 @@ static bool vcpu_deassign(struct null_private *prv, str= uct vcpu *v) struct null_vcpu *wvc; =20 ASSERT(list_empty(&null_vcpu(v)->waitq_elem)); + ASSERT(per_cpu(npc, v->processor).vcpu =3D=3D v); + ASSERT(!cpumask_test_cpu(v->processor, &prv->cpus_free)); =20 per_cpu(npc, cpu).vcpu =3D NULL; cpumask_set_cpu(cpu, &prv->cpus_free); @@ -529,9 +531,6 @@ static void null_vcpu_remove(const struct scheduler *op= s, struct vcpu *v) goto out; } =20 - ASSERT(per_cpu(npc, v->processor).vcpu =3D=3D v); - ASSERT(!cpumask_test_cpu(v->processor, &prv->cpus_free)); - vcpu_deassign(prv, v); =20 out: _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel