From nobody Sun Feb 8 15:46:36 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1569514975; cv=none; d=zoho.com; s=zohoarc; b=PNwBqGRnPhUUittbAX1VPzRfait3Jt2fU1rzTSXKOqI0KFbiJXQ+fBraDw8bO1x+2Qm8l7U+R1d91K1iYAN6RFeCYadQAUJJujnODK0rEKis//PxVxnE8u6EUhtcPXqWwaQQwSKdYI+yLyIA3U1ze5ynCT7d59tN+mZncLybH74= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1569514975; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=uVdXQEebrIcfvm6zZoTOznx+kqNAJWLSxv1yzr/xMIo=; b=G+mlCpvpJgQisWtVi6Jtvv7qRXq9vRgELFFt9xoyVo/sVxzfi2Y4855OvulrpKf65j4ofgkikjXJQE3z0CQDJAhmptfhFCP/wbbIIDhBid/s4PyBOblMqgWW/9YlZVYdh/fExgPwwTz3wqXhDNXIA7YbbNUKkGa43RtZSU/o19o= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 15695149750591015.3525408441596; Thu, 26 Sep 2019 09:22:55 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 8918518C4299; Thu, 26 Sep 2019 16:22:53 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 61FB75D6B0; Thu, 26 Sep 2019 16:22:53 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 2327F1803517; Thu, 26 Sep 2019 16:22:53 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x8QGF3vO003950 for ; Thu, 26 Sep 2019 12:15:03 -0400 Received: by smtp.corp.redhat.com (Postfix) id 81CFA5D9E2; Thu, 26 Sep 2019 16:15:03 +0000 (UTC) Received: from moe.brq.redhat.com (unknown [10.43.2.30]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0A3645D9D5 for ; Thu, 26 Sep 2019 16:14:57 +0000 (UTC) From: Michal Privoznik To: libvir-list@redhat.com Date: Thu, 26 Sep 2019 18:12:33 +0200 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH v2 37/39] qemu: Allow forcing VFIO when computing memlock limit X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.6.2 (mx1.redhat.com [10.5.110.62]); Thu, 26 Sep 2019 16:22:54 +0000 (UTC) Content-Type: text/plain; charset="utf-8" With NVMe disks, one can start a blockjob with a NVMe disk that is not visible in domain XML (at least right away). Usually, it's fairly easy to override this limitation of qemuDomainGetMemLockLimitBytes() - for instance for hostdevs we temporarily add the device to domain def, let the function calculate the limit and then remove the device. But it's not so easy with virStorageSourcePtr - in some cases they don't necessarily are attached to a disk. And even if they are it's done later in the process and frankly, I find it too complicated to be able to use the simple trick we use with hostdevs. Signed-off-by: Michal Privoznik --- src/qemu/qemu_command.c | 2 +- src/qemu/qemu_domain.c | 46 ++++++++++++++++++++++++++--------------- src/qemu/qemu_domain.h | 6 ++++-- src/qemu/qemu_hotplug.c | 12 +++++------ tests/qemumemlocktest.c | 2 +- 5 files changed, 41 insertions(+), 27 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index ad6c1e7e8a..fa2394378a 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -10511,7 +10511,7 @@ qemuBuildCommandLine(virQEMUDriverPtr driver, =20 /* In some situations, eg. VFIO passthrough, QEMU might need to lock a * significant amount of memory, so we need to set the limit according= ly */ - virCommandSetMaxMemLock(cmd, qemuDomainGetMemLockLimitBytes(def)); + virCommandSetMaxMemLock(cmd, qemuDomainGetMemLockLimitBytes(def, false= )); =20 if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_MSG_TIMESTAMP) && cfg->logTimestamp) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index e4e9d4e361..3fe0004cab 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -11843,12 +11843,14 @@ ppc64VFIODeviceIsNV2Bridge(const char *device) /** * getPPC64MemLockLimitBytes: * @def: domain definition + * @forceVFIO: force VFIO usage * * A PPC64 helper that calculates the memory locking limit in order for * the guest to operate properly. */ static unsigned long long -getPPC64MemLockLimitBytes(virDomainDefPtr def) +getPPC64MemLockLimitBytes(virDomainDefPtr def, + bool forceVFIO) { unsigned long long memKB =3D 0; unsigned long long baseLimit =3D 0; @@ -11939,7 +11941,7 @@ getPPC64MemLockLimitBytes(virDomainDefPtr def) passthroughLimit =3D maxMemory + 128 * (1ULL<<30) / 512 * nPCIHostBridges + 8192; - } else if (usesVFIO) { + } else if (usesVFIO || forceVFIO) { /* For regular (non-NVLink2 present) VFIO passthrough, the value * of passthroughLimit is: * @@ -11977,16 +11979,20 @@ getPPC64MemLockLimitBytes(virDomainDefPtr def) /** * qemuDomainGetMemLockLimitBytes: * @def: domain definition + * @forceVFIO: force VFIO calculation * * Calculate the memory locking limit that needs to be set in order for * the guest to operate properly. The limit depends on a number of factors, * including certain configuration options and less immediately apparent o= nes * such as the guest architecture or the use of certain devices. + * The @forceVFIO argument can be used to tell this function will use VFIO= even + * though @def doesn't indicates so right now. * * Returns: the memory locking limit, or 0 if setting the limit is not nee= ded */ unsigned long long -qemuDomainGetMemLockLimitBytes(virDomainDefPtr def) +qemuDomainGetMemLockLimitBytes(virDomainDefPtr def, + bool forceVFIO) { unsigned long long memKB =3D 0; bool usesVFIO =3D false; @@ -12007,7 +12013,7 @@ qemuDomainGetMemLockLimitBytes(virDomainDefPtr def) return VIR_DOMAIN_MEMORY_PARAM_UNLIMITED; =20 if (ARCH_IS_PPC64(def->os.arch) && def->virtType =3D=3D VIR_DOMAIN_VIR= T_KVM) - return getPPC64MemLockLimitBytes(def); + return getPPC64MemLockLimitBytes(def, forceVFIO); =20 /* For device passthrough using VFIO the guest memory and MMIO memory * regions need to be locked persistent in order to allow DMA. @@ -12027,18 +12033,20 @@ qemuDomainGetMemLockLimitBytes(virDomainDefPtr de= f) * * Note that this may not be valid for all platforms. */ - for (i =3D 0; i < def->nhostdevs; i++) { - if (virHostdevIsVFIODevice(def->hostdevs[i]) || - virHostdevIsMdevDevice(def->hostdevs[i])) { - usesVFIO =3D true; - break; + if (!forceVFIO) { + for (i =3D 0; i < def->nhostdevs; i++) { + if (virHostdevIsVFIODevice(def->hostdevs[i]) || + virHostdevIsMdevDevice(def->hostdevs[i])) { + usesVFIO =3D true; + break; + } } + + if (virDomainDefHasNVMeDisk(def)) + usesVFIO =3D true; } =20 - if (virDomainDefHasNVMeDisk(def)) - usesVFIO =3D true; - - if (usesVFIO) + if (usesVFIO || forceVFIO) memKB =3D virDomainDefGetMemoryTotal(def) + 1024 * 1024; =20 done: @@ -12049,9 +12057,12 @@ qemuDomainGetMemLockLimitBytes(virDomainDefPtr def) /** * qemuDomainAdjustMaxMemLock: * @vm: domain + * @forceVFIO: apply VFIO requirements even if vm's def doesn't require it * * Adjust the memory locking limit for the QEMU process associated to @vm,= in - * order to comply with VFIO or architecture requirements. + * order to comply with VFIO or architecture requirements. If @forceVFIO is + * true then the limit is changed even if nothing in @vm's definition indi= cates + * so. * * The limit will not be changed unless doing so is needed; the first time * the limit is changed, the original (default) limit is stored in @vm and @@ -12061,12 +12072,13 @@ qemuDomainGetMemLockLimitBytes(virDomainDefPtr de= f) * Returns: 0 on success, <0 on failure */ int -qemuDomainAdjustMaxMemLock(virDomainObjPtr vm) +qemuDomainAdjustMaxMemLock(virDomainObjPtr vm, + bool forceVFIO) { unsigned long long bytes =3D 0; int ret =3D -1; =20 - bytes =3D qemuDomainGetMemLockLimitBytes(vm->def); + bytes =3D qemuDomainGetMemLockLimitBytes(vm->def, forceVFIO); =20 if (bytes) { /* If this is the first time adjusting the limit, save the current @@ -12115,7 +12127,7 @@ qemuDomainAdjustMaxMemLockHostdev(virDomainObjPtr v= m, int ret =3D 0; =20 vm->def->hostdevs[vm->def->nhostdevs++] =3D hostdev; - if (qemuDomainAdjustMaxMemLock(vm) < 0) + if (qemuDomainAdjustMaxMemLock(vm, false) < 0) ret =3D -1; =20 vm->def->hostdevs[--(vm->def->nhostdevs)] =3D NULL; diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index de6479f8e0..7802a8b98b 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -962,8 +962,10 @@ bool qemuDomainSupportsPCI(virDomainDefPtr def, =20 void qemuDomainUpdateCurrentMemorySize(virDomainObjPtr vm); =20 -unsigned long long qemuDomainGetMemLockLimitBytes(virDomainDefPtr def); -int qemuDomainAdjustMaxMemLock(virDomainObjPtr vm); +unsigned long long qemuDomainGetMemLockLimitBytes(virDomainDefPtr def, + bool forceVFIO); +int qemuDomainAdjustMaxMemLock(virDomainObjPtr vm, + bool forceVFIO); int qemuDomainAdjustMaxMemLockHostdev(virDomainObjPtr vm, virDomainHostdevDefPtr hostdev); =20 diff --git a/src/qemu/qemu_hotplug.c b/src/qemu/qemu_hotplug.c index 75ffc099e1..f5a672a195 100644 --- a/src/qemu/qemu_hotplug.c +++ b/src/qemu/qemu_hotplug.c @@ -1648,7 +1648,7 @@ qemuDomainAttachHostPCIDevice(virQEMUDriverPtr driver, if (teardowndevice && qemuDomainNamespaceTeardownHostdev(vm, hostdev) < 0) VIR_WARN("Unable to remove host device from /dev"); - if (teardownmemlock && qemuDomainAdjustMaxMemLock(vm) < 0) + if (teardownmemlock && qemuDomainAdjustMaxMemLock(vm, false) < 0) VIR_WARN("Unable to reset maximum locked memory on hotplug fail"); =20 if (releaseaddr) @@ -2387,7 +2387,7 @@ qemuDomainAttachMemory(virQEMUDriverPtr driver, if (virDomainMemoryInsert(vm->def, mem) < 0) goto cleanup; =20 - if (qemuDomainAdjustMaxMemLock(vm) < 0) + if (qemuDomainAdjustMaxMemLock(vm, false) < 0) goto removedef; =20 qemuDomainObjEnterMonitor(driver, vm); @@ -2458,7 +2458,7 @@ qemuDomainAttachMemory(virQEMUDriverPtr driver, =20 /* reset the mlock limit */ virErrorPreserveLast(&orig_err); - ignore_value(qemuDomainAdjustMaxMemLock(vm)); + ignore_value(qemuDomainAdjustMaxMemLock(vm, false)); virErrorRestore(&orig_err); =20 goto audit; @@ -2856,7 +2856,7 @@ qemuDomainAttachMediatedDevice(virQEMUDriverPtr drive= r, ret =3D 0; cleanup: if (ret < 0) { - if (teardownmemlock && qemuDomainAdjustMaxMemLock(vm) < 0) + if (teardownmemlock && qemuDomainAdjustMaxMemLock(vm, false) < 0) VIR_WARN("Unable to reset maximum locked memory on hotplug fai= l"); if (teardowncgroup && qemuTeardownHostdevCgroup(vm, hostdev) < 0) VIR_WARN("Unable to remove host device cgroup ACL on hotplug f= ail"); @@ -4378,7 +4378,7 @@ qemuDomainRemoveMemoryDevice(virQEMUDriverPtr driver, ignore_value(qemuProcessRefreshBalloonState(driver, vm, QEMU_ASYNC_JOB= _NONE)); =20 /* decrease the mlock limit after memory unplug if necessary */ - ignore_value(qemuDomainAdjustMaxMemLock(vm)); + ignore_value(qemuDomainAdjustMaxMemLock(vm, false)); =20 return 0; } @@ -4505,7 +4505,7 @@ qemuDomainRemoveHostDevice(virQEMUDriverPtr driver, qemuDomainRemovePCIHostDevice(driver, vm, hostdev); /* QEMU might no longer need to lock as much memory, eg. we just * detached the last VFIO device, so adjust the limit here */ - if (qemuDomainAdjustMaxMemLock(vm) < 0) + if (qemuDomainAdjustMaxMemLock(vm, false) < 0) VIR_WARN("Failed to adjust locked memory limit"); break; case VIR_DOMAIN_HOSTDEV_SUBSYS_TYPE_USB: diff --git a/tests/qemumemlocktest.c b/tests/qemumemlocktest.c index c9484ac9cb..1d52498e25 100644 --- a/tests/qemumemlocktest.c +++ b/tests/qemumemlocktest.c @@ -43,7 +43,7 @@ testCompareMemLock(const void *data) goto cleanup; } =20 - ret =3D virTestCompareToULL(info->memlock, qemuDomainGetMemLockLimitBy= tes(def)); + ret =3D virTestCompareToULL(info->memlock, qemuDomainGetMemLockLimitBy= tes(def, false)); =20 cleanup: virDomainDefFree(def); --=20 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list