From nobody Wed May 15 10:10:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1620223619; cv=none; d=zohomail.com; s=zohoarc; b=b27qsthyZpeLRS6f3QP9FG9gKrSkwCUFm9QRa0YxyM7rYxh4MjQiMoAhpqjKjnowtussHpwQolY/m0c1+7A0JzRUuVtk12xNQWsYHqA1sue/2sBqztXHGU9ZbCJTYWv1oKAukFYo2ueedbcdpOo8Gbebjc8f8HACjcsTRheURCU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1620223619; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=GrQ/vxcz7z+CM6IUCdxavvfcR1kIO1xA+J+bsNLTzrA=; b=B/rY6VeJMik7QdKTmmLVGAg1Zb1CpCzyg3L8iZ1d8MKJ+O3GQjfDxPHrNvHZ3l4lfNLNIf6ytYi/Xi0tgw2993sPtarySabysvI9JxYqXNmIyo74N69QMBas+9UF6te3AREA56MRkF3ZtVhcUV8Gsc+Buu3xSwiiY3HJxFyzYTI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1620223619890270.8416856111079; Wed, 5 May 2021 07:06:59 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-2-uOLhoS70MkmHD4EmbdA0Hg-1; Wed, 05 May 2021 10:06:55 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6AEBE64157; Wed, 5 May 2021 14:06:50 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7775219C46; Wed, 5 May 2021 14:06:49 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 9EB545534C; Wed, 5 May 2021 14:06:46 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 145E6ikV015916 for ; Wed, 5 May 2021 10:06:44 -0400 Received: by smtp.corp.redhat.com (Postfix) id 593C510728F; Wed, 5 May 2021 14:06:44 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast04.extmail.prod.ext.rdu2.redhat.com [10.11.55.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 546D3106264 for ; Wed, 5 May 2021 14:06:41 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 789731066685 for ; Wed, 5 May 2021 14:06:41 +0000 (UTC) Received: from mo4-p00-ob.smtp.rzone.de (mo4-p00-ob.smtp.rzone.de [81.169.146.219]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-F9v2tuXhPj-gsYaacGyzqQ-1; Wed, 05 May 2021 10:06:37 -0400 Received: from sender by smtp.strato.de (RZmta 47.25.6 AUTH) with ESMTPSA id L05296x45E6Y1Vr (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate); Wed, 5 May 2021 16:06:34 +0200 (CEST) X-MC-Unique: uOLhoS70MkmHD4EmbdA0Hg-1 X-MC-Unique: F9v2tuXhPj-gsYaacGyzqQ-1 X-RZG-AUTH: ":P2EQZWCpfu+qG7CngxMFH1J+3q8wa/QXkBR9MXjAuzpIG0mv9coXAgtl+1b1FMstFZvCqIQN5N7TvWFg4vzhFVdoKAuQ" X-RZG-CLASS-ID: mo00 From: Olaf Hering To: libvir-list@redhat.com Subject: [PATCH v2] libxl: set vcpu affinity during domain creation Date: Wed, 5 May 2021 16:06:32 +0200 Message-Id: <20210505140632.15241-1-olaf@aepfle.de> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-loop: libvir-list@redhat.com Cc: Olaf Hering X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since Xen 4.5 libxl allows to set affinities during domain creation. This enables Xen to allocate the domain memory on NUMA systems close to the specified pcpus. Libvirt can now handle in domU.xml correctly. Without this change, Xen will create the domU and assign NUMA memory and vcpu affinities on its own. Later libvirt will adjust the affinity, which may move the vcpus away from the assigned NUMA node. Signed-off-by: Olaf Hering Reviewed-by: Daniel Henrique Barboza --- src/libxl/libxl_conf.c | 53 ++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_domain.c | 46 ---------------------------------- src/libxl/libxl_domain.h | 4 --- 3 files changed, 53 insertions(+), 50 deletions(-) diff --git a/src/libxl/libxl_conf.c b/src/libxl/libxl_conf.c index 6392d7795d..2a99626f92 100644 --- a/src/libxl/libxl_conf.c +++ b/src/libxl/libxl_conf.c @@ -283,6 +283,56 @@ libxlMakeChrdevStr(virDomainChrDef *def, char **buf) return 0; } =20 +static int +libxlSetVcpuAffinities(virDomainDef *def, + libxl_ctx *ctx, + libxl_domain_build_info *b_info) +{ + libxl_bitmap *vcpu_affinity_array; + unsigned int vcpuid; + unsigned int vcpu_idx =3D 0; + virDomainVcpuDef *vcpu; + bool has_vcpu_pin =3D false; + + /* Get highest vcpuid with cpumask */ + for (vcpuid =3D 0; vcpuid < b_info->max_vcpus; vcpuid++) { + vcpu =3D virDomainDefGetVcpu(def, vcpuid); + if (!vcpu) + continue; + if (!vcpu->cpumask) + continue; + vcpu_idx =3D vcpuid; + has_vcpu_pin =3D true; + } + /* Nothing to do */ + if (!has_vcpu_pin) + return 0; + + /* Adjust index */ + vcpu_idx++; + + b_info->num_vcpu_hard_affinity =3D vcpu_idx; + /* Will be released by libxl_domain_config_dispose */ + b_info->vcpu_hard_affinity =3D g_new0(libxl_bitmap, vcpu_idx); + vcpu_affinity_array =3D b_info->vcpu_hard_affinity; + + for (vcpuid =3D 0; vcpuid < vcpu_idx; vcpuid++) { + libxl_bitmap *map =3D &vcpu_affinity_array[vcpuid]; + libxl_bitmap_init(map); + /* libxl owns the bitmap */ + if (libxl_cpu_bitmap_alloc(ctx, map, 0)) + return -1; + vcpu =3D virDomainDefGetVcpu(def, vcpuid); + /* Apply the given mask, or allow unhandled vcpus to run anywhere = */ + if (vcpu && vcpu->cpumask) + virBitmapToDataBuf(vcpu->cpumask, map->map, map->size); + else + libxl_bitmap_set_any(map); + } + libxl_defbool_set(&b_info->numa_placement, false); + return 0; +} + static int libxlMakeDomBuildInfo(virDomainDef *def, libxlDriverConfig *cfg, @@ -320,6 +370,9 @@ libxlMakeDomBuildInfo(virDomainDef *def, for (i =3D 0; i < virDomainDefGetVcpus(def); i++) libxl_bitmap_set((&b_info->avail_vcpus), i); =20 + if (libxlSetVcpuAffinities(def, ctx, b_info)) + return -1; + switch ((virDomainClockOffsetType) clock.offset) { case VIR_DOMAIN_CLOCK_OFFSET_VARIABLE: if (clock.data.variable.basis =3D=3D VIR_DOMAIN_CLOCK_BASIS_LOCALT= IME) diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c index d78765ad84..625e04a9b0 100644 --- a/src/libxl/libxl_domain.c +++ b/src/libxl/libxl_domain.c @@ -967,49 +967,6 @@ libxlDomainAutoCoreDump(libxlDriverPrivate *driver, return 0; } =20 -int -libxlDomainSetVcpuAffinities(libxlDriverPrivate *driver, virDomainObj *vm) -{ - g_autoptr(libxlDriverConfig) cfg =3D libxlDriverConfigGet(driver); - virDomainVcpuDef *vcpu; - libxl_bitmap map; - virBitmap *cpumask =3D NULL; - size_t i; - int ret =3D -1; - - libxl_bitmap_init(&map); - - for (i =3D 0; i < virDomainDefGetVcpus(vm->def); ++i) { - vcpu =3D virDomainDefGetVcpu(vm->def, i); - - if (!vcpu->online) - continue; - - if (!(cpumask =3D vcpu->cpumask)) - cpumask =3D vm->def->cpumask; - - if (!cpumask) - continue; - - if (virBitmapToData(cpumask, &map.map, (int *)&map.size) < 0) - goto cleanup; - - if (libxl_set_vcpuaffinity(cfg->ctx, vm->def->id, i, &map, NULL) != =3D 0) { - virReportError(VIR_ERR_INTERNAL_ERROR, - _("Failed to pin vcpu '%zu' with libxenlight"),= i); - goto cleanup; - } - - libxl_bitmap_dispose(&map); /* Also returns to freshly-init'd stat= e */ - } - - ret =3D 0; - - cleanup: - libxl_bitmap_dispose(&map); - return ret; -} - static int libxlDomainFreeMem(libxl_ctx *ctx, libxl_domain_config *d_config) { @@ -1460,9 +1417,6 @@ libxlDomainStart(libxlDriverPrivate *driver, goto destroy_dom; } =20 - if (libxlDomainSetVcpuAffinities(driver, vm) < 0) - goto destroy_dom; - if (!start_paused) { libxlDomainUnpauseWrapper(cfg->ctx, domid); virDomainObjSetState(vm, VIR_DOMAIN_RUNNING, VIR_DOMAIN_RUNNING_BO= OTED); diff --git a/src/libxl/libxl_domain.h b/src/libxl/libxl_domain.h index cbe7ba19ba..928ee8f5cd 100644 --- a/src/libxl/libxl_domain.h +++ b/src/libxl/libxl_domain.h @@ -124,10 +124,6 @@ int libxlDomainAutoCoreDump(libxlDriverPrivate *driver, virDomainObj *vm); =20 -int -libxlDomainSetVcpuAffinities(libxlDriverPrivate *driver, - virDomainObj *vm); - int libxlDomainStartNew(libxlDriverPrivate *driver, virDomainObj *vm,