From nobody Thu May 2 05:25:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1659448310; cv=none; d=zohomail.com; s=zohoarc; b=eLn7eSBZSmShHFsgT0fNrn/gA+QOmk4Hcdve9TmKh1dSYNDeM0fisjFUfU4ataQj63nLV25s9CRbuktI40Vi9403L34AppxFTVJycV9vUvG6Fe7/EkAUHL5qzd/pHXtuxE9w0rsGfy1ytLVYuq0JUj9YwOai59TsedfbL4HUsCc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1659448310; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=D9GBZJCBCLFyKSdywcRnXHi8pEehjBe3e89BGugt4mI=; b=E2K0cEr2LdIPhl+9RE9zkMfUxkgDYiC1vBaTm6NRxCCtG7a1fnYDgnjTutXRJus/4eGG7SK3e20YpWtToGAR44GgGeyqTET5j2j7iP6y8ZvlbUfmZ9OEHOmXN8yQmCorbj7leYjoDKBkN4jpMfN6ZhOTz6InGRRiM9gDf5K21Ds= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1659448310085478.11725884449856; Tue, 2 Aug 2022 06:51:50 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.379353.612748 (Exim 4.92) (envelope-from ) id 1oIsJ3-00019s-1O; Tue, 02 Aug 2022 13:51:33 +0000 Received: by outflank-mailman (output) from mailman id 379353.612748; Tue, 02 Aug 2022 13:51:32 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oIsJ2-00019l-UN; Tue, 02 Aug 2022 13:51:32 +0000 Received: by outflank-mailman (input) for mailman id 379353; Tue, 02 Aug 2022 13:51:32 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oIsJ2-0000sI-82 for xen-devel@lists.xenproject.org; Tue, 02 Aug 2022 13:51:32 +0000 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 36cfecab-126a-11ed-bd2d-47488cf2e6aa; Tue, 02 Aug 2022 15:51:31 +0200 (CEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 4F47B3787F; Tue, 2 Aug 2022 13:51:31 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 221C51345B; Tue, 2 Aug 2022 13:51:31 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id a/0JBuMr6WK7ZAAAMHmgww (envelope-from ); Tue, 02 Aug 2022 13:51:31 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 36cfecab-126a-11ed-bd2d-47488cf2e6aa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1659448291; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D9GBZJCBCLFyKSdywcRnXHi8pEehjBe3e89BGugt4mI=; b=rPgkKs4Mqn2AxRAy9Bec7nMeoMBA5TvQrt4teTMAu7A+qp+FqjElfaa6bENNtYy1RWQ8tJ bhrKVmOyLzZXnWzcRK7tpvVa+Eed/sFNL0TDGqbtl6RM/OtyPOa54Sd8q3Ow4zbiqbiQDt 4L+xKzBIxew0Agy5qy4SAQ1tlNBo3oQ= Subject: [PATCH 1/2] xen: sched: dom0_vcpus_pin should only affect dom0 From: Dario Faggioli To: xen-devel@lists.xenproject.org Cc: Jan Beulich , George Dunlap Date: Tue, 02 Aug 2022 15:51:30 +0200 Message-ID: <165944829054.3531.818489021551433260.stgit@tumbleweed.Wayrath> In-Reply-To: <165944775952.3531.14547627248752689801.stgit@tumbleweed.Wayrath> References: <165944775952.3531.14547627248752689801.stgit@tumbleweed.Wayrath> User-Agent: StGit/1.5 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1659448311904100003 If dom0_vcpus_pin is used, make sure the pinning is only done for dom0 vcpus, instead of for the hardware domain (which might not be dom0 at all!). Suggested-by: Jan Beulich Signed-off-by: Dario Faggioli Reviewed-by: Jan Beulich --- Cc: George Dunlap --- Difference from "RFC" [1]: - new patch [1] https://lore.kernel.org/xen-devel/e061a647cd77a36834e2085a96a07caa785c5= 066.camel@suse.com/ --- xen/common/sched/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c index f689b55783..379a791d62 100644 --- a/xen/common/sched/core.c +++ b/xen/common/sched/core.c @@ -575,7 +575,7 @@ int sched_init_vcpu(struct vcpu *v) * Initialize affinity settings. The idler, and potentially * domain-0 VCPUs, are pinned onto their respective physical CPUs. */ - if ( is_idle_domain(d) || (is_hardware_domain(d) && opt_dom0_vcpus_pin= ) ) + if ( is_idle_domain(d) || (is_control_domain(d) && opt_dom0_vcpus_pin)= ) sched_set_affinity(unit, cpumask_of(processor), &cpumask_all); else sched_set_affinity(unit, &cpumask_all, &cpumask_all); From nobody Thu May 2 05:25:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1659448317; cv=none; d=zohomail.com; s=zohoarc; b=mSdFaSUCgGal8NcQn+FPpibDuGlwdgW74NRj7VwpHder2cQNQ/emJaoTtWut8nNqNGS91dQXvY2btpmRV321/yWS3drwRQbGFEDZB5MyvqgS0JGi10uzI+vx65RoHNsS0EvTsc4IzX9x5EdbUCBpGMM7pgnbG+Vu5dLxnOB884E= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1659448317; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=lFJ2vrWywI4K8SwWHvAOn/VBLvMvx10JLU0y4NpmUyU=; b=c3Ge8DMXthxxyDWfIOgWtsFEVSN1YSMVRKGKxQi8NNKwstSc/YpLEjs12ZSuCbOZvCDnZDFrev8ooby+6pzMmBHL5ycW+5Fu6+sD+o7QEZ5Bb6Pmvi7MYd0Ioxn3YgSw9vNf16JmVSzQWf4uYMVi3AiFF5xze1wZE0uZctJzRIw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1659448317475415.6780923532626; Tue, 2 Aug 2022 06:51:57 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.379354.612759 (Exim 4.92) (envelope-from ) id 1oIsJA-0001VR-8J; Tue, 02 Aug 2022 13:51:40 +0000 Received: by outflank-mailman (output) from mailman id 379354.612759; Tue, 02 Aug 2022 13:51:40 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oIsJA-0001VE-5C; Tue, 02 Aug 2022 13:51:40 +0000 Received: by outflank-mailman (input) for mailman id 379354; Tue, 02 Aug 2022 13:51:39 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oIsJ9-0001U5-3b for xen-devel@lists.xenproject.org; Tue, 02 Aug 2022 13:51:39 +0000 Received: from smtp-out1.suse.de (smtp-out1.suse.de [2001:67c:2178:6::1c]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id 3a4f54a1-126a-11ed-924f-1f966e50362f; Tue, 02 Aug 2022 15:51:38 +0200 (CEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 30CA837402; Tue, 2 Aug 2022 13:51:37 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EA5081345B; Tue, 2 Aug 2022 13:51:36 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 887fNugr6WLKZAAAMHmgww (envelope-from ); Tue, 02 Aug 2022 13:51:36 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 3a4f54a1-126a-11ed-924f-1f966e50362f DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1659448297; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lFJ2vrWywI4K8SwWHvAOn/VBLvMvx10JLU0y4NpmUyU=; b=hojRg2OZKTmpKjq43U17Zi2j66Z0i2iPAEOjdiPww6ap97PA4Kucbx0ShHBd1/GsQL+DQb iVIIbBBWs5fuP5LN2Dv/OoUTYImbIh6Twelj9MMoPVjYAIb0rf3nk37W+WtqNkPcHx3MvL aRDqf7+FUTB5t6atnq5BfgO87+Eprpo= Subject: [PATCH 2/2] xen/sched: setup dom0 vCPUs affinity only once From: Dario Faggioli To: xen-devel@lists.xenproject.org Cc: Olaf Hering , Jan Beulich , George Dunlap Date: Tue, 02 Aug 2022 15:51:36 +0200 Message-ID: <165944829637.3531.11115034950872470305.stgit@tumbleweed.Wayrath> In-Reply-To: <165944775952.3531.14547627248752689801.stgit@tumbleweed.Wayrath> References: <165944775952.3531.14547627248752689801.stgit@tumbleweed.Wayrath> User-Agent: StGit/1.5 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1659448317969100001 Right now, affinity for dom0 vCPUs is setup in two steps. This is a problem as, at least in Credit2, unit_insert() sees and uses the "intermediate" affinity, and place the vCPUs on CPUs where they cannot be run. And this in turn results in boot hangs, if the "dom0_nodes" parameter is used. Fix this by setting up the affinity properly once and for all, in sched_init_vcpu() called by create_vcpu(). Note that, unless a soft-affinity is explicitly specified for dom0 (by using the relaxed mode of "dom0_nodes") we set it to the default, which is all CPUs, instead of computing it basing on hard affinity (if any). This is because hard and soft affinity should be considered as independent user controlled properties. In fact, if we dor derive dom0's soft-affinity from its boot-time hard-affinity, such computed value will continue to be used even if later the user changes the hard-affinity. And this could result in the vCPUs behaving differently than what the user wanted and expects. Fixes: dafd936ddd ("Make credit2 the default scheduler") Reported-by: Olaf Hering Signed-off-by: Dario Faggioli Reviewed-by: Jan Beulich --- Cc: Jan Beulich Cc: George Dunlap --- Changes from "RFC" [1]: - Moved handling of the shim case - Added some more explanation (in particular, about why we stick to all CPUs for the soft affinity) in both commit message and comment - Remove spurious (and non-necessary) credit2 change [1] https://lore.kernel.org/xen-devel/e061a647cd77a36834e2085a96a07caa785c5= 066.camel@suse.com/ --- xen/common/sched/core.c | 63 +++++++++++++++++++++++++++++--------------= ---- 1 file changed, 39 insertions(+), 24 deletions(-) diff --git a/xen/common/sched/core.c b/xen/common/sched/core.c index 379a791d62..0585c643a5 100644 --- a/xen/common/sched/core.c +++ b/xen/common/sched/core.c @@ -571,12 +571,46 @@ int sched_init_vcpu(struct vcpu *v) return 1; } =20 - /* - * Initialize affinity settings. The idler, and potentially - * domain-0 VCPUs, are pinned onto their respective physical CPUs. - */ - if ( is_idle_domain(d) || (is_control_domain(d) && opt_dom0_vcpus_pin)= ) + if ( is_idle_domain(d) ) + { + /* Idle vCPUs are always pinned onto their respective pCPUs */ + sched_set_affinity(unit, cpumask_of(processor), &cpumask_all); + } + else if ( pv_shim && v->vcpu_id =3D=3D 0 ) + { + /* + * PV-shim: vcpus are pinned 1:1. Initially only 1 cpu is online, + * others will be dealt with when onlining them. This avoids pinni= ng + * a vcpu to a not yet online cpu here. + */ + sched_set_affinity(unit, cpumask_of(0), cpumask_of(0)); + } + else if ( is_control_domain(d) && opt_dom0_vcpus_pin ) + { + /* + * If dom0_vcpus_pin is specified, dom0 vCPUs are pinned 1:1 to + * their respective pCPUs too. + */ sched_set_affinity(unit, cpumask_of(processor), &cpumask_all); + } +#ifdef CONFIG_X86 + else if ( is_control_domain(d) ) + { + /* + * In absence of dom0_vcpus_pin instead, the hard and soft affinit= y of + * dom0 is controlled by the (x86 only) dom0_nodes parameter. At t= his + * point it has been parsed and decoded into the dom0_cpus mask. + * + * Note that we always honor what user explicitly requested, for both + * hard and soft affinity, without doing any dynamic computation of + * either of them. + */ + if ( !dom0_affinity_relaxed ) + sched_set_affinity(unit, &dom0_cpus, &cpumask_all); + else + sched_set_affinity(unit, &cpumask_all, &dom0_cpus); + } +#endif else sched_set_affinity(unit, &cpumask_all, &cpumask_all); =20 @@ -3402,29 +3436,10 @@ void wait(void) void __init sched_setup_dom0_vcpus(struct domain *d) { unsigned int i; - struct sched_unit *unit; =20 for ( i =3D 1; i < d->max_vcpus; i++ ) vcpu_create(d, i); =20 - /* - * PV-shim: vcpus are pinned 1:1. - * Initially only 1 cpu is online, others will be dealt with when - * onlining them. This avoids pinning a vcpu to a not yet online cpu h= ere. - */ - if ( pv_shim ) - sched_set_affinity(d->vcpu[0]->sched_unit, - cpumask_of(0), cpumask_of(0)); - else - { - for_each_sched_unit ( d, unit ) - { - if ( !opt_dom0_vcpus_pin && !dom0_affinity_relaxed ) - sched_set_affinity(unit, &dom0_cpus, NULL); - sched_set_affinity(unit, NULL, &dom0_cpus); - } - } - domain_update_node_affinity(d); } #endif