From nobody Mon Feb 9 01:16:22 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1606810930; cv=none; d=zohomail.com; s=zohoarc; b=EUHgGAzJX2nSVOQ25MUbkWKf1hcMI+vamr3rimjZcZcmt2wh23qB2OJupV9ydy/XAATGHhm5BuFu47oZ84dqMK4Qmzz+IhrN0IHSYqC7ORiTlccOWFfAWcp+byeyHTWk6noN5lFUlzm53TUpNrHGy9Af0QEwGKaen0xWbtO0nAQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1606810930; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=708vSf4hdDLmhqVFIbpBu/E6M/WDwL53lyQ6wvfCs8c=; b=jJIwSVlCwT65zub+pCbarK34veE+MK52y8AILFtHchDTNGahAVIzTdGwcNDnA2/n8I8OVau3KiVIcp6NgGxYvDAkLEKr/7Yv1jr+gHskzoAUIPzh5STKqXySg2WSFfLHETbXc99/KRgPGePiepc6rC/AD0O5BokwCct7gou+i98= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1606810930823308.51024464309774; Tue, 1 Dec 2020 00:22:10 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.41640.75012 (Exim 4.92) (envelope-from ) id 1kk0v4-00055m-Vp; Tue, 01 Dec 2020 08:21:55 +0000 Received: by outflank-mailman (output) from mailman id 41640.75012; Tue, 01 Dec 2020 08:21:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kk0v4-00055R-Ka; Tue, 01 Dec 2020 08:21:54 +0000 Received: by outflank-mailman (input) for mailman id 41640; Tue, 01 Dec 2020 08:21:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1kk0v2-0004Uj-JJ for xen-devel@lists.xenproject.org; Tue, 01 Dec 2020 08:21:52 +0000 Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id d5433781-6557-465d-9090-c775c13054ae; Tue, 01 Dec 2020 08:21:33 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 2DC4CAEFF; Tue, 1 Dec 2020 08:21:32 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d5433781-6557-465d-9090-c775c13054ae X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1606810892; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=708vSf4hdDLmhqVFIbpBu/E6M/WDwL53lyQ6wvfCs8c=; b=EEKGCdbaUJZM5ne+c6o2Igd6i7OnsH1A4VTVrMWIIJ8haSEigJ728u0IGOeBVBvrGYgD0G sZf9D/plSzZkk4iAqQKZVxKeKecfF4J/jpq2Tz6BeS2ijXbTAaGnfnE2i4avbbshJxq0pS iekW+Jec6R8Lugat7JGWYuvemBryrL8= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , George Dunlap , Dario Faggioli Subject: [PATCH v2 07/17] xen/cpupool: support moving domain between cpupools with different granularity Date: Tue, 1 Dec 2020 09:21:18 +0100 Message-Id: <20201201082128.15239-8-jgross@suse.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201201082128.15239-1-jgross@suse.com> References: <20201201082128.15239-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) Content-Type: text/plain; charset="utf-8" When moving a domain between cpupools with different scheduling granularity the sched_units of the domain need to be adjusted. Do that by allocating new sched_units and throwing away the old ones in sched_move_domain(). Signed-off-by: Juergen Gross --- xen/common/sched/core.c | 121 ++++++++++++++++++++++++++++++---------- 1 file changed, 90 insertions(+), 31 deletions(-) diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c index a429fc7640..2a61c879b3 100644 --- a/xen/common/sched/core.c +++ b/xen/common/sched/core.c @@ -613,17 +613,45 @@ static void sched_move_irqs(const struct sched_unit *= unit) vcpu_move_irqs(v); } =20 +/* + * Move a domain from one cpupool to another. + * + * A domain with any vcpu having temporary affinity settings will be denied + * to move. Hard and soft affinities will be reset. + * + * In order to support cpupools with different scheduling granularities all + * scheduling units are replaced by new ones. + * + * The complete move is done in the following steps: + * - check prerequisites (no vcpu with temporary affinities) + * - allocate all new data structures (scheduler specific domain data, unit + * memory, scheduler specific unit data) + * - pause domain + * - temporarily move all (old) units to the same scheduling resource (this + * makes the final resource assignment easier in case the new cpupool has + * a larger granularity than the old one, as the scheduling locks for all + * vcpus must be held for that operation) + * - remove old units from scheduling + * - set new cpupool and scheduler domain data pointers in struct domain + * - switch all vcpus to new units, still assigned to the old scheduling + * resource + * - migrate all new units to scheduling resources of the new cpupool + * - unpause the domain + * - free the old memory (scheduler specific domain data, unit memory, + * scheduler specific unit data) + */ int sched_move_domain(struct domain *d, struct cpupool *c) { struct vcpu *v; - struct sched_unit *unit; + struct sched_unit *unit, *old_unit; + struct sched_unit *new_units =3D NULL, *old_units; + struct sched_unit **unit_ptr =3D &new_units; unsigned int new_p, unit_idx; - void **unit_priv; void *domdata; - void *unitdata; - struct scheduler *old_ops; + struct scheduler *old_ops =3D dom_scheduler(d); void *old_domdata; unsigned int gran =3D cpupool_get_granularity(c); + unsigned int n_units =3D DIV_ROUND_UP(d->max_vcpus, gran); int ret =3D 0; =20 for_each_vcpu ( d, v ) @@ -641,53 +669,78 @@ int sched_move_domain(struct domain *d, struct cpupoo= l *c) goto out; } =20 - unit_priv =3D xzalloc_array(void *, DIV_ROUND_UP(d->max_vcpus, gran)); - if ( unit_priv =3D=3D NULL ) + for ( unit_idx =3D 0; unit_idx < n_units; unit_idx++ ) { - sched_free_domdata(c->sched, domdata); - ret =3D -ENOMEM; - goto out; - } + unit =3D sched_alloc_unit_mem(); + if ( unit ) + { + /* Initialize unit for sched_alloc_udata() to work. */ + unit->domain =3D d; + unit->unit_id =3D unit_idx * gran; + unit->vcpu_list =3D d->vcpu[unit->unit_id]; + unit->priv =3D sched_alloc_udata(c->sched, unit, domdata); + *unit_ptr =3D unit; + } =20 - unit_idx =3D 0; - for_each_sched_unit ( d, unit ) - { - unit_priv[unit_idx] =3D sched_alloc_udata(c->sched, unit, domdata); - if ( unit_priv[unit_idx] =3D=3D NULL ) + if ( !unit || !unit->priv ) { - for ( unit_idx =3D 0; unit_priv[unit_idx]; unit_idx++ ) - sched_free_udata(c->sched, unit_priv[unit_idx]); - xfree(unit_priv); - sched_free_domdata(c->sched, domdata); + old_units =3D new_units; + old_domdata =3D domdata; ret =3D -ENOMEM; - goto out; + goto out_free; } - unit_idx++; + + unit_ptr =3D &unit->next_in_list; } =20 domain_pause(d); =20 - old_ops =3D dom_scheduler(d); old_domdata =3D d->sched_priv; =20 + new_p =3D cpumask_first(d->cpupool->cpu_valid); for_each_sched_unit ( d, unit ) { + spinlock_t *lock; + + /* + * Temporarily move all units to same processor to make locking + * easier when moving the new units to the new processors. + */ + lock =3D unit_schedule_lock_irq(unit); + sched_set_res(unit, get_sched_res(new_p)); + spin_unlock_irq(lock); + sched_remove_unit(old_ops, unit); } =20 + old_units =3D d->sched_unit_list; + d->cpupool =3D c; d->sched_priv =3D domdata; =20 + unit =3D new_units; + for_each_vcpu ( d, v ) + { + old_unit =3D v->sched_unit; + if ( unit->unit_id + gran =3D=3D v->vcpu_id ) + unit =3D unit->next_in_list; + + unit->state_entry_time =3D old_unit->state_entry_time; + unit->runstate_cnt[v->runstate.state]++; + /* Temporarily use old resource assignment */ + unit->res =3D get_sched_res(new_p); + + v->sched_unit =3D unit; + } + + d->sched_unit_list =3D new_units; + new_p =3D cpumask_first(c->cpu_valid); - unit_idx =3D 0; for_each_sched_unit ( d, unit ) { spinlock_t *lock; unsigned int unit_p =3D new_p; =20 - unitdata =3D unit->priv; - unit->priv =3D unit_priv[unit_idx]; - for_each_sched_unit_vcpu ( unit, v ) { migrate_timer(&v->periodic_timer, new_p); @@ -713,8 +766,6 @@ int sched_move_domain(struct domain *d, struct cpupool = *c) =20 sched_insert_unit(c->sched, unit); =20 - sched_free_udata(old_ops, unitdata); - unit_idx++; } =20 @@ -722,11 +773,19 @@ int sched_move_domain(struct domain *d, struct cpupoo= l *c) =20 domain_unpause(d); =20 - sched_free_domdata(old_ops, old_domdata); + out_free: + for ( unit =3D old_units; unit; ) + { + if ( unit->priv ) + sched_free_udata(c->sched, unit->priv); + old_unit =3D unit; + unit =3D unit->next_in_list; + xfree(old_unit); + } =20 - xfree(unit_priv); + sched_free_domdata(old_ops, old_domdata); =20 -out: + out: rcu_read_unlock(&sched_res_rculock); =20 return ret; --=20 2.26.2