From nobody Mon Feb 9 12:26:36 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1607530225; cv=none; d=zohomail.com; s=zohoarc; b=jyDxPCBOow/6J63w4CgwNE5/RM6+Hijls2VYQRPyh7q0RibNf0kDmtHtYf++coSrzcjlYPWMLLwSF4WPrVVbX4Y5QV7lpNeeoOxcIzZhpb/cLcOqXghyyDs596ZfPaovmHiQj6miOHSbxeYRLyBOy8BznP01R0f0dqY0geFgNfI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1607530225; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=708vSf4hdDLmhqVFIbpBu/E6M/WDwL53lyQ6wvfCs8c=; b=dEi//6J+exKOLXCF9Cd4GNtxyXv9IcxwLszuwMKd//rxY2n7L6dMAjs1fcizH4CMudWWhn49jFcMCGv10ppE5MRcpY5puGE9+x+IQaJIu1+NxyETqzHTYqG9xaiySg7SMmRsQQ0GHMLpRkRQij0CgLxRu0TGQ8C9EFyVw0NmfUY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1607530225812516.7385117814449; Wed, 9 Dec 2020 08:10:25 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.48427.85638 (Exim 4.92) (envelope-from ) id 1kn22V-0002Xh-0u; Wed, 09 Dec 2020 16:10:03 +0000 Received: by outflank-mailman (output) from mailman id 48427.85638; Wed, 09 Dec 2020 16:10:02 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kn22U-0002XU-Ox; Wed, 09 Dec 2020 16:10:02 +0000 Received: by outflank-mailman (input) for mailman id 48427; Wed, 09 Dec 2020 16:10:01 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kn22T-0002Oq-Ad for xen-devel@lists.xenproject.org; Wed, 09 Dec 2020 16:10:01 +0000 Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5098c72f-d137-4a1d-b8ac-fcbb34d1c47c; Wed, 09 Dec 2020 16:09:59 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 05E57AF8D; Wed, 9 Dec 2020 16:09:59 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5098c72f-d137-4a1d-b8ac-fcbb34d1c47c X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1607530199; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=708vSf4hdDLmhqVFIbpBu/E6M/WDwL53lyQ6wvfCs8c=; b=GOdn9DLkeQ5NgCKeDecYY+yJJtj3HJFe6Oc9MX2QEujxoe4TiN54CGBJ2DVq/UG0dELW7g LAfkuDAtm383RkycI3vY0SI382b/5VWBxpP0uvAXu5MYJdZCEGglQ+WzpRuN4wtPFY3xZ1 Z0ODRWmMQByUCN185u3/rFNn0tIGwrg= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , George Dunlap , Dario Faggioli Subject: [PATCH v3 1/8] xen/cpupool: support moving domain between cpupools with different granularity Date: Wed, 9 Dec 2020 17:09:49 +0100 Message-Id: <20201209160956.32456-2-jgross@suse.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201209160956.32456-1-jgross@suse.com> References: <20201209160956.32456-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) Content-Type: text/plain; charset="utf-8" When moving a domain between cpupools with different scheduling granularity the sched_units of the domain need to be adjusted. Do that by allocating new sched_units and throwing away the old ones in sched_move_domain(). Signed-off-by: Juergen Gross Reviewed-by: Dario Faggioli --- xen/common/sched/core.c | 121 ++++++++++++++++++++++++++++++---------- 1 file changed, 90 insertions(+), 31 deletions(-) diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c index a429fc7640..2a61c879b3 100644 --- a/xen/common/sched/core.c +++ b/xen/common/sched/core.c @@ -613,17 +613,45 @@ static void sched_move_irqs(const struct sched_unit *= unit) vcpu_move_irqs(v); } =20 +/* + * Move a domain from one cpupool to another. + * + * A domain with any vcpu having temporary affinity settings will be denied + * to move. Hard and soft affinities will be reset. + * + * In order to support cpupools with different scheduling granularities all + * scheduling units are replaced by new ones. + * + * The complete move is done in the following steps: + * - check prerequisites (no vcpu with temporary affinities) + * - allocate all new data structures (scheduler specific domain data, unit + * memory, scheduler specific unit data) + * - pause domain + * - temporarily move all (old) units to the same scheduling resource (this + * makes the final resource assignment easier in case the new cpupool has + * a larger granularity than the old one, as the scheduling locks for all + * vcpus must be held for that operation) + * - remove old units from scheduling + * - set new cpupool and scheduler domain data pointers in struct domain + * - switch all vcpus to new units, still assigned to the old scheduling + * resource + * - migrate all new units to scheduling resources of the new cpupool + * - unpause the domain + * - free the old memory (scheduler specific domain data, unit memory, + * scheduler specific unit data) + */ int sched_move_domain(struct domain *d, struct cpupool *c) { struct vcpu *v; - struct sched_unit *unit; + struct sched_unit *unit, *old_unit; + struct sched_unit *new_units =3D NULL, *old_units; + struct sched_unit **unit_ptr =3D &new_units; unsigned int new_p, unit_idx; - void **unit_priv; void *domdata; - void *unitdata; - struct scheduler *old_ops; + struct scheduler *old_ops =3D dom_scheduler(d); void *old_domdata; unsigned int gran =3D cpupool_get_granularity(c); + unsigned int n_units =3D DIV_ROUND_UP(d->max_vcpus, gran); int ret =3D 0; =20 for_each_vcpu ( d, v ) @@ -641,53 +669,78 @@ int sched_move_domain(struct domain *d, struct cpupoo= l *c) goto out; } =20 - unit_priv =3D xzalloc_array(void *, DIV_ROUND_UP(d->max_vcpus, gran)); - if ( unit_priv =3D=3D NULL ) + for ( unit_idx =3D 0; unit_idx < n_units; unit_idx++ ) { - sched_free_domdata(c->sched, domdata); - ret =3D -ENOMEM; - goto out; - } + unit =3D sched_alloc_unit_mem(); + if ( unit ) + { + /* Initialize unit for sched_alloc_udata() to work. */ + unit->domain =3D d; + unit->unit_id =3D unit_idx * gran; + unit->vcpu_list =3D d->vcpu[unit->unit_id]; + unit->priv =3D sched_alloc_udata(c->sched, unit, domdata); + *unit_ptr =3D unit; + } =20 - unit_idx =3D 0; - for_each_sched_unit ( d, unit ) - { - unit_priv[unit_idx] =3D sched_alloc_udata(c->sched, unit, domdata); - if ( unit_priv[unit_idx] =3D=3D NULL ) + if ( !unit || !unit->priv ) { - for ( unit_idx =3D 0; unit_priv[unit_idx]; unit_idx++ ) - sched_free_udata(c->sched, unit_priv[unit_idx]); - xfree(unit_priv); - sched_free_domdata(c->sched, domdata); + old_units =3D new_units; + old_domdata =3D domdata; ret =3D -ENOMEM; - goto out; + goto out_free; } - unit_idx++; + + unit_ptr =3D &unit->next_in_list; } =20 domain_pause(d); =20 - old_ops =3D dom_scheduler(d); old_domdata =3D d->sched_priv; =20 + new_p =3D cpumask_first(d->cpupool->cpu_valid); for_each_sched_unit ( d, unit ) { + spinlock_t *lock; + + /* + * Temporarily move all units to same processor to make locking + * easier when moving the new units to the new processors. + */ + lock =3D unit_schedule_lock_irq(unit); + sched_set_res(unit, get_sched_res(new_p)); + spin_unlock_irq(lock); + sched_remove_unit(old_ops, unit); } =20 + old_units =3D d->sched_unit_list; + d->cpupool =3D c; d->sched_priv =3D domdata; =20 + unit =3D new_units; + for_each_vcpu ( d, v ) + { + old_unit =3D v->sched_unit; + if ( unit->unit_id + gran =3D=3D v->vcpu_id ) + unit =3D unit->next_in_list; + + unit->state_entry_time =3D old_unit->state_entry_time; + unit->runstate_cnt[v->runstate.state]++; + /* Temporarily use old resource assignment */ + unit->res =3D get_sched_res(new_p); + + v->sched_unit =3D unit; + } + + d->sched_unit_list =3D new_units; + new_p =3D cpumask_first(c->cpu_valid); - unit_idx =3D 0; for_each_sched_unit ( d, unit ) { spinlock_t *lock; unsigned int unit_p =3D new_p; =20 - unitdata =3D unit->priv; - unit->priv =3D unit_priv[unit_idx]; - for_each_sched_unit_vcpu ( unit, v ) { migrate_timer(&v->periodic_timer, new_p); @@ -713,8 +766,6 @@ int sched_move_domain(struct domain *d, struct cpupool = *c) =20 sched_insert_unit(c->sched, unit); =20 - sched_free_udata(old_ops, unitdata); - unit_idx++; } =20 @@ -722,11 +773,19 @@ int sched_move_domain(struct domain *d, struct cpupoo= l *c) =20 domain_unpause(d); =20 - sched_free_domdata(old_ops, old_domdata); + out_free: + for ( unit =3D old_units; unit; ) + { + if ( unit->priv ) + sched_free_udata(c->sched, unit->priv); + old_unit =3D unit; + unit =3D unit->next_in_list; + xfree(old_unit); + } =20 - xfree(unit_priv); + sched_free_domdata(old_ops, old_domdata); =20 -out: + out: rcu_read_unlock(&sched_res_rculock); =20 return ret; --=20 2.26.2