From nobody Tue Nov 26 12:19:00 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1686149003; cv=none; d=zohomail.com; s=zohoarc; b=Zdzvy3pQy89Oj8lyMeJRPIcnpjg2WGU5w4YhDvQomQbuAIS+C8xleiVHNThy6p9cZV85uTP65B9D+65ZiKIDtwNgLvyE/FuZ7m5RDL0hSz4xhPiuelKV+LQiv1zoTjo+k08JpL027fLOkW8qWNVeJLLSUuZO31uu/0ElBvYobqs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1686149003; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dcBrlr02oh0Mx6nyVyiTjACQJRGIs8wfVjho6VULIoY=; b=Z5jFsFdScNtRKJSQvbxl8tpfaG7G/0aC0LKiwNv+tlTeHOo+GLtpx9zy0T2Wxb+qlhc8yq2zAXlBZUTgNSUSlEjTJdHrirUFNWq9CrVSXqXrb1GBFCnXIE8hOeVo8T10biw7yOuLQFRbUQtI9ZWhltkEleYkF9+NJVzLDYeuTFI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 168614900393910.428563017439274; Wed, 7 Jun 2023 07:43:23 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-353-51GXJDUsOKyY0IiXobtBsw-1; Wed, 07 Jun 2023 10:43:19 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48BD6803902; Wed, 7 Jun 2023 14:43:16 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id CAD332026D6A; Wed, 7 Jun 2023 14:43:13 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 907EF19451C4; Wed, 7 Jun 2023 14:43:06 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 1D30B19451E0 for ; Wed, 7 Jun 2023 14:41:09 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 3E8C8403375; Wed, 7 Jun 2023 14:41:04 +0000 (UTC) Received: from localhost.localdomain (unknown [10.43.2.36]) by smtp.corp.redhat.com (Postfix) with ESMTP id DAC4D403374 for ; Wed, 7 Jun 2023 14:41:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686149002; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=dcBrlr02oh0Mx6nyVyiTjACQJRGIs8wfVjho6VULIoY=; b=LHSr9gWPXPPIrz0P49H3ST6NBsWvOqe5Zm63nh0O+t+vSKhTq5EcQMiNtIX13Gv10iLthv YaxigasnoJqt9pTpST5UwWPL6R6brkaoHvssFBfvQUEZe42uD4nj3pNuBCoh1bwlEY3pXw +JO7I9oDz5fmGugUlrWD5YwjCIHr3OY= X-MC-Unique: 51GXJDUsOKyY0IiXobtBsw-1 X-Original-To: libvir-list@listman.corp.redhat.com From: Michal Privoznik To: libvir-list@redhat.com Subject: [PATCH 1/3] qemu: Don't try to 'fix up' cpuset.mems after QEMU's memory allocation Date: Wed, 7 Jun 2023 16:40:59 +0200 Message-Id: <68e317c0430e348f80efd6dbf787e7688d42b3d5.1686148404.git.mprivozn@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1686149004808100001 Content-Type: text/plain; charset="utf-8"; x-default="true" In ideal world, my plan was perfect. We allow union of all host nodes in cpuset.mems and once QEMU has allocated its memory, we 'fix up' restriction of its emulator thread by writing the original value we wanted to set all along. But in fact, we can't do it because that triggers memory movement. For instance, consider the following : This is meant to create 1:1 mapping between guest and host NUMA nodes. So we start QEMU with cpuset.mems set to "0-1" (so that it can allocate memory even for guest node #1 and have the memory come fro host node #1) and then, set cpuset.mems to "0" (because that's where we wanted emulator thread to live). But this in turn triggers movement of all memory (even the allocated one) to host NUMA node #0. Therefore, we have to just keep cpuset.mems untouched and rely on .host-nodes passed on the QEMU cmd line. The placement still suffers because of cpuset.mems set for vcpus or iothreads, but that's fixed in next commit. Fixes: 3ec6d586bc3ec7a8cf406b1b6363e87d50aa159c Signed-off-by: Michal Privoznik Reviewed-by: Martin Kletzander --- src/qemu/qemu_hotplug.c | 7 +++---- src/qemu/qemu_process.c | 15 ++++----------- src/qemu/qemu_process.h | 3 +-- 3 files changed, 8 insertions(+), 17 deletions(-) diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 806aecb29d..ba9e44945b 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -2295,7 +2295,7 @@ qemuDomainAttachMemory(virQEMUDriver *driver, if (qemuDomainAdjustMaxMemLock(vm) < 0) goto removedef; =20 - if (qemuProcessSetupEmulator(vm, true) < 0) + if (qemuProcessSetupEmulator(vm) < 0) goto removedef; restoreemulatorcgroup =3D true; =20 @@ -2336,11 +2336,10 @@ qemuDomainAttachMemory(virQEMUDriver *driver, VIR_WARN("Unable to remove memory device from /dev"); if (releaseaddr) qemuDomainReleaseMemoryDeviceSlot(vm, mem); + if (restoreemulatorcgroup) + qemuProcessSetupEmulator(vm); } =20 - if (restoreemulatorcgroup) - qemuProcessSetupEmulator(vm, false); - virDomainMemoryDefFree(mem); return ret; =20 diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index a4a4a17a9b..1d3cdeff9a 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -2596,9 +2596,7 @@ qemuProcessSetupPid(virDomainObj *vm, mem_mode =3D=3D VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE)) { =20 /* QEMU allocates its memory from the emulator thread. Thus it - * needs to access union of all host nodes configured. This is - * going to be replaced with proper value later in the startup - * process. */ + * needs to access union of all host nodes configured. */ if (unionMems && nameval =3D=3D VIR_CGROUP_THREAD_EMULATOR && mem_mode !=3D VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) { @@ -2702,15 +2700,14 @@ qemuProcessSetupPid(virDomainObj *vm, =20 =20 int -qemuProcessSetupEmulator(virDomainObj *vm, - bool unionMems) +qemuProcessSetupEmulator(virDomainObj *vm) { return qemuProcessSetupPid(vm, vm->pid, VIR_CGROUP_THREAD_EMULATOR, 0, vm->def->cputune.emulatorpin, vm->def->cputune.emulator_period, vm->def->cputune.emulator_quota, vm->def->cputune.emulatorsched, - unionMems); + true); } =20 =20 @@ -7764,7 +7761,7 @@ qemuProcessLaunch(virConnectPtr conn, goto cleanup; =20 VIR_DEBUG("Setting emulator tuning/settings"); - if (qemuProcessSetupEmulator(vm, true) < 0) + if (qemuProcessSetupEmulator(vm) < 0) goto cleanup; =20 VIR_DEBUG("Setting cgroup for external devices (if required)"); @@ -7827,10 +7824,6 @@ qemuProcessLaunch(virConnectPtr conn, if (qemuConnectAgent(driver, vm) < 0) goto cleanup; =20 - VIR_DEBUG("Fixing up emulator tuning/settings"); - if (qemuProcessSetupEmulator(vm, false) < 0) - goto cleanup; - VIR_DEBUG("setting up hotpluggable cpus"); if (qemuDomainHasHotpluggableStartupVcpus(vm->def)) { if (qemuDomainRefreshVcpuInfo(vm, asyncJob, false) < 0) diff --git a/src/qemu/qemu_process.h b/src/qemu/qemu_process.h index 1c4c0678ab..cae1b49756 100644 --- a/src/qemu/qemu_process.h +++ b/src/qemu/qemu_process.h @@ -236,5 +236,4 @@ void qemuProcessCleanupMigrationJob(virQEMUDriver *driv= er, void qemuProcessRefreshDiskProps(virDomainDiskDef *disk, struct qemuDomainDiskInfo *info); =20 -int qemuProcessSetupEmulator(virDomainObj *vm, - bool unionMems); +int qemuProcessSetupEmulator(virDomainObj *vm); --=20 2.39.3