From nobody Fri Mar 29 12:04:17 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1684836430; cv=none; d=zohomail.com; s=zohoarc; b=fppqi2CsuONVDDWBkEWCAYhi7Ob5NgL8no5c6varhDedu730Vs5l54xLs5O+N8wWcZnnIBkhUEKb+h2t/O7lYFTa8hEl+5JNuhRTapOo3XYPphcVSgzox9zp7S8vIwNhv4Nj4wOm2o77C5VQvRmLER14TfRUWXgG0VExH+jGtK4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1684836430; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=G2lXJ++WQVlIVN3RsHlpYz8QULagw9vBoLtcMisfcSI=; b=MK25Fhx1zo6MLLy9YS6yFoDDJuKR0BVDV9AhI9XHgSv6F2lphXbRFcGkegBqCG0NXolcWjCYK78iyLoCqY4NIB+zs2LPjwNsFSmgGr2+mS1Fdek/ZL+SIQ5XFVScLZyngudVWNto2epDknazLVD+WTNyNosgQGyYk2180iG9lk0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1684836430339842.1136761443588; Tue, 23 May 2023 03:07:10 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-76-bma4YQ1CPriIbcbE8oDBWg-1; Tue, 23 May 2023 06:07:06 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 400912A59575; Tue, 23 May 2023 10:07:04 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E72820296C6; Tue, 23 May 2023 10:07:04 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 93BB319451CE; Tue, 23 May 2023 10:06:36 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id DC30119465A8 for ; Tue, 23 May 2023 10:06:28 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id BD5EF1121314; Tue, 23 May 2023 10:06:23 +0000 (UTC) Received: from localhost.localdomain (unknown [10.43.2.39]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6504D1121319 for ; Tue, 23 May 2023 10:06:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684836429; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=G2lXJ++WQVlIVN3RsHlpYz8QULagw9vBoLtcMisfcSI=; b=J32dtJp8EiaFm6aj9H4xXhpYYbUuFirxGp6+usm/Z6xGQq3ahjJ8kHvE2PPhciFu1EE1SQ m/pxkLefk7XU6GxgLVvTmbZzGQhTeuDEA+htqFn3e5KPrT95vnU9yBE31UwWIshNyD7+WV 6FXMldjwQZhXuioNwiapMy/7p/CnU9o= X-MC-Unique: bma4YQ1CPriIbcbE8oDBWg-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 1/4] qemuProcessSetupPid: Use @numatune variable more Date: Tue, 23 May 2023 12:06:18 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1684836431195100001 Content-Type: text/plain; charset="utf-8"; x-default="true" Inside of qemuProcessSetupPid() there's @numatune variable which is set to vm->def->numa, but it lives only in one block. In the rest of places the expanded form (vm->def->numa) is used instead. Move the variable declaration at the beginning of the function and use it instead of the expanded form. Signed-off-by: Michal Privoznik Reviewed-by: Martin Kletzander --- src/qemu/qemu_process.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index ce5ed6389a..57c3ea2dbf 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -2553,6 +2553,7 @@ qemuProcessSetupPid(virDomainObj *vm, virDomainThreadSchedParam *sched) { qemuDomainObjPrivate *priv =3D vm->privateData; + virDomainNuma *numatune =3D vm->def->numa; virDomainNumatuneMemMode mem_mode; virCgroup *cgroup =3D NULL; virBitmap *use_cpumask =3D NULL; @@ -2589,10 +2590,10 @@ qemuProcessSetupPid(virDomainObj *vm, if (virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPU) || virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPUSET)= ) { =20 - if (virDomainNumatuneGetMode(vm->def->numa, -1, &mem_mode) =3D=3D = 0 && + if (virDomainNumatuneGetMode(numatune, -1, &mem_mode) =3D=3D 0 && (mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_STRICT || mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) && - virDomainNumatuneMaybeFormatNodeset(vm->def->numa, + virDomainNumatuneMaybeFormatNodeset(numatune, priv->autoNodeset, &mem_mask, -1) < 0) goto cleanup; @@ -2601,8 +2602,6 @@ qemuProcessSetupPid(virDomainObj *vm, * threads based on the node they are in as there is nothing else = uses * for such restriction (e.g. numa_set_membind). */ if (nameval =3D=3D VIR_CGROUP_THREAD_VCPU) { - virDomainNuma *numatune =3D vm->def->numa; - /* Look for the guest NUMA node of this vCPU */ for (i =3D 0; i < virDomainNumaGetNodeCount(numatune); i++) { virBitmap *node_cpus =3D virDomainNumaGetNodeCpumask(numat= une, i); --=20 2.39.3 From nobody Fri Mar 29 12:04:17 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1684836399; cv=none; d=zohomail.com; s=zohoarc; b=loBIkAAdiWUe7dvCc2frn6Nx2oXKjb95ktYd6dMhdOLt3mglwNHy0K33aYMVxLmEQaKE0617fnUM1Qhny3RkuvAxCqtJAafc1U19cBpGM2MszXIWkXMsAjQon7Rr/kKJKJyWu3jwtoNQ9RxE2HrFwHf43ZHr/IitGOBEvr5TFTY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1684836399; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pw9azVqRiSCNxRBo6NkBH8AuY8FSvPJzMY1T7m6uOdQ=; b=UaR260NEslmMqYet2daQqT8tu46wzRnNPms4v8hmnQdZhPAC5aB6rsHNDF6GJnWNaj5txeBiOCVwH0/xwFA01Ap7FZllSsOxo7peEfDz65SPgun2rRiWECzoyf8OvL20PuInjR/iV5AYbz6HH7JIDBwJmrmTSagF/a5uKI8rEPc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1684836399207253.55126677341764; Tue, 23 May 2023 03:06:39 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-617-IygvSD_YORapE4N8dXcNUw-1; Tue, 23 May 2023 06:06:36 -0400 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 39CC7185A793; Tue, 23 May 2023 10:06:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 26964401026; Tue, 23 May 2023 10:06:34 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id E91FF19465B6; Tue, 23 May 2023 10:06:30 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 8A1ED19465A0 for ; Tue, 23 May 2023 10:06:24 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 586BA1121319; Tue, 23 May 2023 10:06:24 +0000 (UTC) Received: from localhost.localdomain (unknown [10.43.2.39]) by smtp.corp.redhat.com (Postfix) with ESMTP id 001C4112131B for ; Tue, 23 May 2023 10:06:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684836398; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=pw9azVqRiSCNxRBo6NkBH8AuY8FSvPJzMY1T7m6uOdQ=; b=AsiJRA7fscMUAtuyccwUY1iZGwnZlfzx6eDy7i4rxnxAvIsWinJRLDnELl9oX0LHopeL5L gEHPiS1KUvb8K8wvxNXMJX69c7U8NEcuj+HeK2ycWqWu3v4ivNu1dyryhucy4NNamkWH7A peHRMEK5hTVXXQdPTZ1py8Qdzf3lm8U= X-MC-Unique: IygvSD_YORapE4N8dXcNUw-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 2/4] qemu: Start emulator thread with more generous cpuset.mems Date: Tue, 23 May 2023 12:06:19 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.9 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1684836401215100003 Content-Type: text/plain; charset="utf-8"; x-default="true" Consider a domain with two guest NUMA nodes and the following setting : What this means is the emulator thread is pinned onto host NUMA node #0 (by setting corresponding cpuset.mems to "0"), and two memory-backend-* objects are created: -object '{"qom-type":"memory-backend-ram","id":"ram-node0", .., "host-nod= es":[1],"policy":"bind"}' \ -numa node,nodeid=3D0,cpus=3D0-1,memdev=3Dram-node0 \ -object '{"qom-type":"memory-backend-ram","id":"ram-node1", .., "host-nod= es":[0],"policy":"bind"}' \ -numa node,nodeid=3D1,cpus=3D2-3,memdev=3Dram-node1 \ Note, the emulator thread is pinned well before QEMU is even exec()-ed. Now, the way memory allocation works in QEMU is: the emulator thread calls mmap() followed by mbind() (which is sane, that's how everybody should do it). BUT, because the thread is already restricted by CGroups to just NUMA node #0, calling: mbind(host-nodes:[1]); /* made up syntax (TM) */ fails. This is expected though. Kernel was instructed to place the memory at NUMA node "0" and yet, process is trying to place it elsewhere. We used to solve this by not restricting emulator thread at all initially, and only after it's done initializing (i.e. we got the QMP greeting) we placed it onto desired nodes. But this had its own problems (e.g. QEMU might have locked pieces of its memory which were then unable to migrate onto different NUMA nodes). Therefore, in v5.1.0-rc1~282 we've changed this and set CGropus upfront (even before exec()-ing QEMU). And this used to work, but something has changed (I can't really put my finger anywhere). Therefore, for the initialization start the thread with union of all configured host NUMA nodes ("0-1" in our example) and fix the placement only after QEMU is started. NB, the memory hotplug suffers the same problem, but that might be fixed later. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=3D2138150 Signed-off-by: Michal Privoznik Reviewed-by: Martin Kletzander --- src/qemu/qemu_domain.c | 26 +++++++++++++++++++++++++ src/qemu/qemu_domain.h | 5 +++++ src/qemu/qemu_process.c | 42 ++++++++++++++++++++++++++++++----------- 3 files changed, 62 insertions(+), 11 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 35d03515c7..d97a0ce24e 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -12674,3 +12674,29 @@ qemuDomainEvaluateCPUMask(const virDomainDef *def, =20 return NULL; } + + +void +qemuDomainNumatuneMaybeFormatNodesetUnion(virDomainObj *vm, + virBitmap **nodeset, + char **nodesetStr) +{ + virDomainNuma *numatune =3D vm->def->numa; + qemuDomainObjPrivate *priv =3D vm->privateData; + g_autoptr(virBitmap) unionMask =3D virBitmapNew(0); + ssize_t i; + + for (i =3D -1; i < (ssize_t)virDomainNumaGetNodeCount(numatune); i++) { + virBitmap *tmp; + + tmp =3D virDomainNumatuneGetNodeset(numatune, priv->autoNodeset, i= ); + if (tmp) + virBitmapUnion(unionMask, tmp); + } + + if (nodesetStr) + *nodesetStr =3D virBitmapFormat(unionMask); + + if (nodeset) + *nodeset =3D g_steal_pointer(&unionMask); +} diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index ec9ae75bce..999190e381 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -1140,3 +1140,8 @@ virBitmap * qemuDomainEvaluateCPUMask(const virDomainDef *def, virBitmap *cpumask, virBitmap *autoCpuset); + +void +qemuDomainNumatuneMaybeFormatNodesetUnion(virDomainObj *vm, + virBitmap **nodeset, + char **nodesetStr); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 57c3ea2dbf..6b85b7cee7 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -2550,7 +2550,8 @@ qemuProcessSetupPid(virDomainObj *vm, virBitmap *cpumask, unsigned long long period, long long quota, - virDomainThreadSchedParam *sched) + virDomainThreadSchedParam *sched, + bool unionMems) { qemuDomainObjPrivate *priv =3D vm->privateData; virDomainNuma *numatune =3D vm->def->numa; @@ -2592,11 +2593,22 @@ qemuProcessSetupPid(virDomainObj *vm, =20 if (virDomainNumatuneGetMode(numatune, -1, &mem_mode) =3D=3D 0 && (mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_STRICT || - mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) && - virDomainNumatuneMaybeFormatNodeset(numatune, - priv->autoNodeset, - &mem_mask, -1) < 0) - goto cleanup; + mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE)) { + + /* QEMU allocates its memory from the emulator thread. Thus it + * needs to access union of all host nodes configured. This is + * going to be replaced with proper value later in the startup + * process. */ + if (unionMems && + nameval =3D=3D VIR_CGROUP_THREAD_EMULATOR) { + qemuDomainNumatuneMaybeFormatNodesetUnion(vm, NULL, &mem_m= ask); + } else { + if (virDomainNumatuneMaybeFormatNodeset(numatune, + priv->autoNodeset, + &mem_mask, -1) < 0) + goto cleanup; + } + } =20 /* For restrictive numatune mode we need to set cpuset.mems for vC= PU * threads based on the node they are in as there is nothing else = uses @@ -2689,13 +2701,15 @@ qemuProcessSetupPid(virDomainObj *vm, =20 =20 static int -qemuProcessSetupEmulator(virDomainObj *vm) +qemuProcessSetupEmulator(virDomainObj *vm, + bool unionMems) { return qemuProcessSetupPid(vm, vm->pid, VIR_CGROUP_THREAD_EMULATOR, 0, vm->def->cputune.emulatorpin, vm->def->cputune.emulator_period, vm->def->cputune.emulator_quota, - vm->def->cputune.emulatorsched); + vm->def->cputune.emulatorsched, + unionMems); } =20 =20 @@ -5891,7 +5905,8 @@ qemuProcessSetupVcpu(virDomainObj *vm, vcpuid, vcpu->cpumask, vm->def->cputune.period, vm->def->cputune.quota, - &vcpu->sched) < 0) + &vcpu->sched, + false) < 0) return -1; =20 if (schedCore && @@ -6046,7 +6061,8 @@ qemuProcessSetupIOThread(virDomainObj *vm, iothread->cpumask, vm->def->cputune.iothread_period, vm->def->cputune.iothread_quota, - &iothread->sched); + &iothread->sched, + false); } =20 =20 @@ -7746,7 +7762,7 @@ qemuProcessLaunch(virConnectPtr conn, goto cleanup; =20 VIR_DEBUG("Setting emulator tuning/settings"); - if (qemuProcessSetupEmulator(vm) < 0) + if (qemuProcessSetupEmulator(vm, true) < 0) goto cleanup; =20 VIR_DEBUG("Setting cgroup for external devices (if required)"); @@ -7809,6 +7825,10 @@ qemuProcessLaunch(virConnectPtr conn, if (qemuConnectAgent(driver, vm) < 0) goto cleanup; =20 + VIR_DEBUG("Fixing up emulator tuning/settings"); + if (qemuProcessSetupEmulator(vm, false) < 0) + goto cleanup; + VIR_DEBUG("setting up hotpluggable cpus"); if (qemuDomainHasHotpluggableStartupVcpus(vm->def)) { if (qemuDomainRefreshVcpuInfo(vm, asyncJob, false) < 0) --=20 2.39.3 From nobody Fri Mar 29 12:04:17 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1684836396; cv=none; d=zohomail.com; s=zohoarc; b=Y9IBY4k0VkIKPhH9E9GvHmDJT1+MHkIWymBO7k1qC1NaT9R+66mT9NL7B78WDuRRvtaqnZllAzQ/d9CBwLOrzt7yV2j731Nq13afJ+Ofu1s+kyzkvodwBIbgantKE/lEi6Zzqnq0oyXvoyw9XuPwC5xk0M14iXatvW3xQnvhw6o= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1684836396; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TjmX7UV9rjKvB/9aFv+6zSAu5Dc+jVyMi4zNAOhoh+8=; b=O5zWbGjkT1uN3ECuIpADPBBfe1QZ26T87xH+F/aglu6edwPt6mHyiSrwsZKrfeDFJZxV0aXAbIIOccFetcvewfDzwKgXnZFZ09LjK0X1oYQy8BDNv/Kqlgwq+X6qaoDZ3PhgvrI0AbbcghRd5/9QjZG0Bss7p/aga7+9ISo/7Zc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1684836396242414.0746897530596; Tue, 23 May 2023 03:06:36 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-427-D_on32WcMGuxgvMiQJ0cVw-1; Tue, 23 May 2023 06:06:30 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 635912814247; Tue, 23 May 2023 10:06:28 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4CF3840D1B64; Tue, 23 May 2023 10:06:26 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id F175419465A8; Tue, 23 May 2023 10:06:25 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 0F03B19465A0 for ; Tue, 23 May 2023 10:06:25 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id E71791121319; Tue, 23 May 2023 10:06:24 +0000 (UTC) Received: from localhost.localdomain (unknown [10.43.2.39]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8EE48112131B for ; Tue, 23 May 2023 10:06:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684836395; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=TjmX7UV9rjKvB/9aFv+6zSAu5Dc+jVyMi4zNAOhoh+8=; b=Qq1E+LBc7JZVN3nAk6VJRVm47WJotzELROobwIw3VLqc73lrI/ut4s6QVwaG+rWRdl3RT6 HJ6DlKqq3DtXp+3Rt/RI4uKCexNO4pYrh+7CIQZsEiuJ9QTwW3WWK5T8fxPwbKu4shqH+W lj7Pj06p2OT8OiCz16BCwBNVDzO+VRE= X-MC-Unique: D_on32WcMGuxgvMiQJ0cVw-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 3/4] qemu: Start emulator thread with more generous mbind() Date: Tue, 23 May 2023 12:06:20 +0200 Message-Id: <0891634ec35410430e23892c043facc40491255c.1684836360.git.mprivozn@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1684836397125100001 Content-Type: text/plain; charset="utf-8"; x-default="true" This is very similar to the previous commit, but fixes the case where there are no CGroups (or are disabled by admin) and thus libvirt calls mbind() before exec()-ing QEMU. Signed-off-by: Michal Privoznik --- src/qemu/qemu_process.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 6b85b7cee7..0fafda0beb 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3106,10 +3106,8 @@ struct qemuProcessHookData { static int qemuProcessHook(void *data) { struct qemuProcessHookData *h =3D data; - qemuDomainObjPrivate *priv =3D h->vm->privateData; int ret =3D -1; int fd; - virBitmap *nodeset =3D NULL; virDomainNumatuneMemMode mode; =20 /* This method cannot use any mutexes, which are not @@ -3154,8 +3152,9 @@ static int qemuProcessHook(void *data) * and enabled in the config) rather than virNuma*. */ VIR_DEBUG("Relying on CGroups for memory binding"); } else { - nodeset =3D virDomainNumatuneGetNodeset(h->vm->def->numa, - priv->autoNodeset, -1); + g_autoptr(virBitmap) nodeset =3D NULL; + + qemuDomainNumatuneMaybeFormatNodesetUnion(h->vm, &nodeset, NUL= L); =20 if (virNumaSetupMemoryPolicy(mode, nodeset) < 0) goto cleanup; --=20 2.39.3 From nobody Fri Mar 29 12:04:17 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1684836426; cv=none; d=zohomail.com; s=zohoarc; b=GbCJRnNmc0EioA9CeNW7OHhzbMmfl/v44F6VEWUbs4FrN5E61o7bdy2xc1DuDWDawWDXCwfCwrDWVKeaCIdCVtGInKYuiwtQnaY/6HGnmJsdKup3UKvjCqTLVLJPQst8fPypzA0CyrBBuIXICXlT3cUaA2u41+Ldv0ZKWtrAH/Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1684836426; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=PdY2q/WadXp/wF5hKjLC/PWsiiY50AT5ae8sAEYMM7k=; b=EEayjoaxwm1QC2P5aymPggdHXUNgW57cJFLMKwrmXhRB+lD1RnNYvHBfKwMbd0Ko2YO85CHWzVAhRAtAoJ6O+e1EJDwDH8GfJ5Cq+5tYjNBMGPE28GuOL29utZ8ucjI1afWdqTiwhmA8KTOOAZ/uZEIsZdK/UXk45EkBC3jhWxs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1684836426031876.7658091473006; Tue, 23 May 2023 03:07:06 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-441-qGL4ChK9M8KVVdTxs7qeSQ-1; Tue, 23 May 2023 06:06:44 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1EADC185A78B; Tue, 23 May 2023 10:06:42 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0DD6720296C6; Tue, 23 May 2023 10:06:42 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 27A4119452C7; Tue, 23 May 2023 10:06:31 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 86CD919465A0 for ; Tue, 23 May 2023 10:06:25 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 820BA112131E; Tue, 23 May 2023 10:06:25 +0000 (UTC) Received: from localhost.localdomain (unknown [10.43.2.39]) by smtp.corp.redhat.com (Postfix) with ESMTP id 299051121319 for ; Tue, 23 May 2023 10:06:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684836425; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=PdY2q/WadXp/wF5hKjLC/PWsiiY50AT5ae8sAEYMM7k=; b=EvyXvCugouH7B90C8SUbaNH2wnUUpJSwGRYicY2ILR5yvqv/18uxaIgAwRqT9TEwnSEl0u 64tX+enLTFCRDqBFFUuCQLPi+yH2dC/8b/Vsfqb1lgiUCfrer+gyvtC3TMiOo4E7crTQcs m54W4ud0Q6d5TyJEYcGWXutdxBe/q5s= X-MC-Unique: qGL4ChK9M8KVVdTxs7qeSQ-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 4/4] qemu_hotplug: Temporarily allow emulator thread to access other NUMA nodes during mem hotplug Date: Tue, 23 May 2023 12:06:21 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1684836427205100005 Content-Type: text/plain; charset="utf-8"; x-default="true" Again, this fixes the same problem as one of previous commits, but this time for memory hotplug. Long story short, if there's a domain running and the emulator thread is restricted to a subset of host NUMA nodes, but the memory that's about to be hotplugged requires memory from a host NUMA node that's not in the set we need to allow emulator thread to access the node, temporarily. Signed-off-by: Michal Privoznik Reviewed-by: Martin Kletzander --- src/qemu/qemu_hotplug.c | 8 ++++++++ src/qemu/qemu_process.c | 2 +- src/qemu/qemu_process.h | 3 +++ 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index d5148f5815..972df572a7 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -2247,6 +2247,7 @@ qemuDomainAttachMemory(virQEMUDriver *driver, bool teardownlabel =3D false; bool teardowncgroup =3D false; bool teardowndevice =3D false; + bool restoreemulatorcgroup =3D false; g_autoptr(virJSONValue) props =3D NULL; virObjectEvent *event; int id; @@ -2294,6 +2295,10 @@ qemuDomainAttachMemory(virQEMUDriver *driver, if (qemuDomainAdjustMaxMemLock(vm) < 0) goto removedef; =20 + if (qemuProcessSetupEmulator(vm, true) < 0) + goto removedef; + restoreemulatorcgroup =3D true; + qemuDomainObjEnterMonitor(vm); if (qemuMonitorAddObject(priv->mon, &props, NULL) < 0) goto exit_monitor; @@ -2333,6 +2338,9 @@ qemuDomainAttachMemory(virQEMUDriver *driver, qemuDomainReleaseMemoryDeviceSlot(vm, mem); } =20 + if (restoreemulatorcgroup) + qemuProcessSetupEmulator(vm, false); + virDomainMemoryDefFree(mem); return ret; =20 diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 0fafda0beb..803bf9b0a3 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -2700,7 +2700,7 @@ qemuProcessSetupPid(virDomainObj *vm, } =20 =20 -static int +int qemuProcessSetupEmulator(virDomainObj *vm, bool unionMems) { diff --git a/src/qemu/qemu_process.h b/src/qemu/qemu_process.h index e0cbafda1d..1c4c0678ab 100644 --- a/src/qemu/qemu_process.h +++ b/src/qemu/qemu_process.h @@ -235,3 +235,6 @@ void qemuProcessCleanupMigrationJob(virQEMUDriver *driv= er, =20 void qemuProcessRefreshDiskProps(virDomainDiskDef *disk, struct qemuDomainDiskInfo *info); + +int qemuProcessSetupEmulator(virDomainObj *vm, + bool unionMems); --=20 2.39.3