From nobody Mon Mar 23 23:18:37 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass(p=none dis=none) header.from=gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1772651411; cv=none; d=zohomail.com; s=zohoarc; b=KTCcSymQvIZ3NcB/zC3Dv6ahA443C6t6stuvrMEFCo940qiAsJeKuWDpbGVCPEZLEx9XnxR51+Y7PmIHKx3Hp6SFqnd8GzHj8TiCYCj16+P1XIN/4WMbYCuvi48ZLdLmojPL4Eyl3+1XRXQQ1ar60m8WQg2SWKlIdd3S820ubY4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1772651411; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:List-Subscribe:List-Post:List-Owner:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Subject:Subject:To:To:Message-Id:Reply-To; bh=sIgyjfkzAcXwuLxjYqEhJKv71Alw5p9sPsVG1WDrd7Q=; b=JubDdiNpKzeb/J3dufHF5yzaCidOLXz7xCRcw8lY4vz9LrHYcnik3kyCF2SsmeYQWrwii+T0/jUzQ0SeAjoZd95U2RBeOmAdyuZF5VzWBcpeWWxoZjhQrhgnl9UiwHm5bHe5t564vgRZyhThWA9tMY2JXtC0pAkfHN2bf6vGSMU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 177265141169346.46559550515542; Wed, 4 Mar 2026 11:10:11 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 993) id 23E9E4193F; Wed, 4 Mar 2026 14:10:10 -0500 (EST) Received: from [172.19.199.12] (lists.libvirt.org [8.43.85.245]) by lists.libvirt.org (Postfix) with ESMTP id 30A9441A2F; Wed, 4 Mar 2026 14:09:10 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 993) id D81103F8A1; Wed, 4 Mar 2026 14:09:05 -0500 (EST) Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (3072 bits) server-digest SHA256) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 02EC13F311 for ; Wed, 4 Mar 2026 14:09:05 -0500 (EST) Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-b904e1cd038so965668266b.1 for ; Wed, 04 Mar 2026 11:09:04 -0800 (PST) Received: from tulp.my.domain (2001-1c02-1a15-3000-c218-03ff-feb5-6cc4.cable.dynamic.v6.ziggo.nl. [2001:1c02:1a15:3000:c218:3ff:feb5:6cc4]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-b935aec39fdsm773702866b.54.2026.03.04.11.09.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Mar 2026 11:09:02 -0800 (PST) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,HELO_MISC_IP,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_VALIDITY_CERTIFIED_BLOCKED, RCVD_IN_VALIDITY_RPBL_BLOCKED,RCVD_IN_VALIDITY_SAFE_BLOCKED,SPF_PASS autolearn=unavailable autolearn_force=no version=4.0.1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772651343; x=1773256143; darn=lists.libvirt.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=sIgyjfkzAcXwuLxjYqEhJKv71Alw5p9sPsVG1WDrd7Q=; b=OguXJ8TVqA7hZF9O0By9+JhL5+yjDRSd6QARo6v70HNEHnZ9u92p5fWDn/A7d5SzYX weYBVH0DGmguSNLH6GRu7IpvI8O7YwkHHqd4w783aMWfzCaGqh6BNLdCgBNyE7WJ/822 TKLRYVS12RN6ZELOQIMSUyouQJwitl6DvO4bmKZrrF98M/nHxcbkaFS9BpllZ8DGPAxo Ly3Aswgz9OmQe3AivzT6WQvxIB9fS0lryxz+eBjSNTta7dVCSPMBRNitxfp/qpkPkQVY HwvLDb4d3d5ecjcz/uAYLJXkKtnGREweeoiFHQU7++2qFWQyBuvFJlLVvcHqUZg6WbrN 95iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772651343; x=1773256143; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=sIgyjfkzAcXwuLxjYqEhJKv71Alw5p9sPsVG1WDrd7Q=; b=fmNxOBYs0w3zWthuSSYslaMOvm0XIxhJntqLZj+EtJNJ+0KXME0pP5aFwsdc/wiSIg eUPn7YUGBy4I/B8pAp5kqEoPC1pObBe3aPDSOvqvmyQ1cEha7gdPgefSCo6pT3U8z7az vyYvrsi363jJZAwMtaWjAFAZ8FeVneEsMx3gA4pDm7o3EKcJzf2LJe8Rj2MMqjiQI9Cv VTvf+0oqDVY1F/JnpZhgjZA0DJ/M9xC5aMZL+wiEFCGINyivWX9VBtvzRo3DQL2Vr7Xn mloNzPXphFP28r7BeaGEPn1MCEALRnGO8AXCIwq657nIiOiw7dL7f8hNemqGM96OSb6C 3dEQ== X-Gm-Message-State: AOJu0Yx/6t5x/N4pAOKnmWZ+yiAkA8Q+nQJ0BJJqlq6SgmDQWGBay9bN eu52Pv9/dflx1WtkR+fZRGQi6LRFuaph0GcUChKJTE0TDdhQV1lRxVZVKlOPA43V X-Gm-Gg: ATEYQzytFBOxqJoJaGwxnK5K6pGxE2fUJ+PBQ3x8E9Wr9CE5Z13tvxktXpVAxnG0GjX ni448wtre6myyr1IisXTltzahxUBXh4qWOIMLafR81i7uGEegBmW7+tvN1OGRafuax0a41gjWna OzDO+A5B4A9oPVcf3jeU1/XyxJa66jjYmMv57NDb/liWFsYXCQja1/BUjlVW1YkrWzEuGIw6eXb TnEcv+IwqHRbCCFMspjR8YRoKnL4AAl4lt53rb+aR0dD0sWvMGbShFL9op8j5ZSRfOUE1VIHJyK BbMwY+ChnpP90NdVk5rfMNyhN6TpzHaBBTdgEYYnXZ0akJZJgN8cV4ROhgk/cYGVzO9v0dU2u3/ utM9jUM6cu9AjkUyTx72+uY8Qq+UzEPp7ho735ejPwPLrvDz/cUBFpd9QcZIiWNUgPHH1Y7Z9h1 tH0izEqx/UhHNirNgqbuEd7dCyrWv4eTOu49htBYDyy4/AJSgAN81WH6S440SIJ7rMHJMCJ2bXu odiA9/FowKNmUb6aDskPlyxrSDsQ2j1itE9 X-Received: by 2002:a17:907:9625:b0:b8f:deff:a019 with SMTP id a640c23a62f3a-b93f0e8017bmr214428566b.0.1772651343045; Wed, 04 Mar 2026 11:09:03 -0800 (PST) From: Roman Bogorodskiy To: devel@lists.libvirt.org Subject: [PATCH] bhyve: support NUMA configuration for domains Date: Wed, 4 Mar 2026 20:08:01 +0100 Message-ID: <20260304190801.95520-1-bogorodskiy@gmail.com> X-Mailer: git-send-email 2.52.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 4RAXQKBOEEMEITHPMMIS2YQTTVTLKZTU X-Message-ID-Hash: 4RAXQKBOEEMEITHPMMIS2YQTTVTLKZTU X-MailFrom: bogorodskiy@gmail.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-devel.lists.libvirt.org-0; emergency; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Roman Bogorodskiy X-Mailman-Version: 3.3.10 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-ZohoMail-DKIM: pass (identity @gmail.com) X-ZM-MESSAGEID: 1772651412941158500 Content-Type: text/plain; charset="utf-8" Bhyve supports NUMA domains configuration using the '-n' command line argument: -n id,size,cpus[,domain_policy] Here, "id" is a numeric NUMA domain id, "size" is the total VM memory size with units format similar to the "-m" switch, "cpus" is a cpuset, and "domain_policy" is an optional domainset(9) memory allocation policy. The "domain_policy" is currently not used by the libvirt driver. This argument is repeated for every NUMA domain to be configured, e.g.: bhyve \ ... -n id=3D0,size=3D107,cpus=3D0-3 -n id=3D1,size=3D107,cpus=3D4-7 To support that: * Add a corresponding capability; it is considered supported if the bhyve binary has the '-n' command line switch. * Generate command line arguments for NUMA from .. domain configuration. Additionally, validate that: * NUMA domains can be only configured with the UEFI loaders. * No more than 8 domains configured per VM as limited by bhyve. Signed-off-by: Roman Bogorodskiy Reviewed-by: Michal Privoznik --- src/bhyve/bhyve_capabilities.c | 3 ++ src/bhyve/bhyve_capabilities.h | 1 + src/bhyve/bhyve_command.c | 26 ++++++++++++ src/bhyve/bhyve_domain.c | 19 +++++++++ .../bhyvexml2argv-numa-empty-cpuset.xml | 29 +++++++++++++ .../bhyvexml2argv-numa-too-many-domains.xml | 36 ++++++++++++++++ .../x86_64/bhyvexml2argv-numa.args | 14 +++++++ .../x86_64/bhyvexml2argv-numa.ldargs | 1 + .../x86_64/bhyvexml2argv-numa.xml | 29 +++++++++++++ tests/bhyvexml2argvtest.c | 10 ++++- .../x86_64/bhyvexml2xmlout-numa.xml | 42 +++++++++++++++++++ tests/bhyvexml2xmltest.c | 1 + 12 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-empty= -cpuset.xml create mode 100644 tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-too-m= any-domains.xml create mode 100644 tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.args create mode 100644 tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.ldargs create mode 100644 tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.xml create mode 100644 tests/bhyvexml2xmloutdata/x86_64/bhyvexml2xmlout-numa.x= ml diff --git a/src/bhyve/bhyve_capabilities.c b/src/bhyve/bhyve_capabilities.c index d2a48ed30c..c3fb88fe9f 100644 --- a/src/bhyve/bhyve_capabilities.c +++ b/src/bhyve/bhyve_capabilities.c @@ -258,6 +258,9 @@ bhyveProbeCapsFromHelp(unsigned int *caps, char *binary) if (strstr(help, "-A:") !=3D NULL) *caps |=3D BHYVE_CAP_ACPI; =20 + if (strstr(help, "-n:") !=3D NULL) + *caps |=3D BHYVE_CAP_NUMA; + return 0; } =20 diff --git a/src/bhyve/bhyve_capabilities.h b/src/bhyve/bhyve_capabilities.h index d5346df7ba..31fd9ab86a 100644 --- a/src/bhyve/bhyve_capabilities.h +++ b/src/bhyve/bhyve_capabilities.h @@ -56,6 +56,7 @@ typedef enum { BHYVE_CAP_VIRTIO_RND =3D 1 << 10, BHYVE_CAP_NVME =3D 1 << 11, BHYVE_CAP_ACPI =3D 1 << 12, + BHYVE_CAP_NUMA =3D 1 << 13, } virBhyveCapsFlags; =20 int virBhyveProbeGrubCaps(virBhyveGrubCapsFlags *caps); diff --git a/src/bhyve/bhyve_command.c b/src/bhyve/bhyve_command.c index 37618812bc..931d7dd551 100644 --- a/src/bhyve/bhyve_command.c +++ b/src/bhyve/bhyve_command.c @@ -905,6 +905,7 @@ virBhyveProcessBuildBhyveCmd(struct _bhyveConn *driver,= virDomainDef *def, unsigned nusbcontrollers =3D 0; unsigned nisacontrollers =3D 0; unsigned nvcpus =3D virDomainDefGetVcpus(def); + size_t ncells =3D virDomainNumaGetNodeCount(def->numa); =20 /* CPUs */ virCommandAddArg(cmd, "-c"); @@ -955,6 +956,31 @@ virBhyveProcessBuildBhyveCmd(struct _bhyveConn *driver= , virDomainDef *def, } } =20 + /* NUMA */ + if (ncells) { + if (!(bhyveDriverGetBhyveCaps(driver) & BHYVE_CAP_NUMA)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Installed bhyve binary does not support NUMA= configuration")); + return NULL; + } + + if (def->os.bootloader || !def->os.loader) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("NUMA configuration is only supported when bo= oting using UEFI")); + return NULL; + } + + for (i =3D 0; i < ncells; i++) { + unsigned long long memSize =3D virDomainNumaGetNodeMemorySize(= def->numa, i); + virBitmap *cpus =3D virDomainNumaGetNodeCpumask(def->numa, i); + g_autofree char *cpumask =3D virBitmapFormat(cpus); + + virCommandAddArg(cmd, "-n"); + virCommandAddArgFormat(cmd, "id=3D%zu,size=3D%llu,cpus=3D%s", = i, VIR_DIV_UP(memSize, 1024), + cpumask); + } + } + /* Memory */ virCommandAddArg(cmd, "-m"); virCommandAddArgFormat(cmd, "%llu", diff --git a/src/bhyve/bhyve_domain.c b/src/bhyve/bhyve_domain.c index 85960c6e12..4594d7673f 100644 --- a/src/bhyve/bhyve_domain.c +++ b/src/bhyve/bhyve_domain.c @@ -411,6 +411,7 @@ bhyveDomainDefValidate(const virDomainDef *def, void *parseOpaque G_GNUC_UNUSED) { size_t i; + size_t ncells; virStorageSource *src =3D NULL; g_autoptr(GHashTable) nvme_controllers =3D g_hash_table_new(g_direct_h= ash, g_direct_equ= al); @@ -445,6 +446,24 @@ bhyveDomainDefValidate(const virDomainDef *def, return -1; } =20 + ncells =3D virDomainNumaGetNodeCount(def->numa); + if (ncells) { + if (ncells > 8) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", + _("Only up to 8 NUMA domains are supported")); + return -1; + } + + for (i =3D 0; i < ncells; i++) { + if (!virDomainNumaGetNodeCpumask(def->numa, i)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("NUMA domain id %1$zu: empty cpusets are = not allowed"), + i); + return -1; + } + } + } + if (!def->os.loader) return 0; =20 diff --git a/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-empty-cpuset= .xml b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-empty-cpuset.xml new file mode 100644 index 0000000000..9a5fc282ba --- /dev/null +++ b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-empty-cpuset.xml @@ -0,0 +1,29 @@ + + bhyve + df3be7e7-a104-11e3-aeb0-50e5492bd3dc + 219136 + 8 + + hvm + + + + + + + + + + + + +
+ + + + + +
+ + + diff --git a/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-too-many-dom= ains.xml b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-too-many-domai= ns.xml new file mode 100644 index 0000000000..bcabe5cd85 --- /dev/null +++ b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa-too-many-domains.xml @@ -0,0 +1,36 @@ + + bhyve + df3be7e7-a104-11e3-aeb0-50e5492bd3dc + 876544 + 9 + + hvm + + + + + + + + + + + + + + + + + + + +
+ + + + + +
+ + + diff --git a/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.args b/tests= /bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.args new file mode 100644 index 0000000000..15efd1c357 --- /dev/null +++ b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.args @@ -0,0 +1,14 @@ +bhyve \ +-c 8 \ +-n id=3D0,size=3D107,cpus=3D0-3 \ +-n id=3D1,size=3D107,cpus=3D4-7 \ +-m 214 \ +-u \ +-H \ +-P \ +-s 0:0,hostbridge \ +-l bootrom,fakefirmwaredir/BHYVE_UEFI.fd,fakenvramdir/bhyve_VARS.fd \ +-s 1:0,lpc \ +-s 2:0,ahci,hd:/tmp/freebsd.img \ +-s 3:0,virtio-net,faketapdev,mac=3D52:54:00:b9:94:02 \ +bhyve diff --git a/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.ldargs b/tes= ts/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.ldargs new file mode 100644 index 0000000000..421376db9e --- /dev/null +++ b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.ldargs @@ -0,0 +1 @@ +dummy diff --git a/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.xml b/tests/= bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.xml new file mode 100644 index 0000000000..8a0da7830d --- /dev/null +++ b/tests/bhyvexml2argvdata/x86_64/bhyvexml2argv-numa.xml @@ -0,0 +1,29 @@ + + bhyve + df3be7e7-a104-11e3-aeb0-50e5492bd3dc + 219136 + 8 + + hvm + + + + + + + + + + + + +
+ + + + + +
+ + + diff --git a/tests/bhyvexml2argvtest.c b/tests/bhyvexml2argvtest.c index 2330e70bbf..b7749fec6f 100644 --- a/tests/bhyvexml2argvtest.c +++ b/tests/bhyvexml2argvtest.c @@ -202,7 +202,7 @@ mymain(void) BHYVE_CAP_FBUF | BHYVE_CAP_XHCI | \ BHYVE_CAP_CPUTOPOLOGY | BHYVE_CAP_SOUND_HDA | \ BHYVE_CAP_VNC_PASSWORD | BHYVE_CAP_VIRTIO_9P | \ - BHYVE_CAP_NVME; + BHYVE_CAP_NVME | BHYVE_CAP_NUMA; =20 DO_TEST("base"); DO_TEST("wired"); @@ -254,6 +254,11 @@ mymain(void) DO_TEST("isa-controller"); DO_TEST_FAILURE("isa-multiple-controllers"); DO_TEST("firmware-efi"); + DO_TEST("numa"); + DO_TEST_FAILURE("numa-empty-cpuset"); + DO_TEST_FAILURE("numa-too-many-domains"); + driver.bhyvecaps &=3D ~BHYVE_CAP_NUMA; + DO_TEST_FAILURE("numa"); fakefirmwaredir =3D g_steal_pointer(&driver.config->firmwareDir); driver.config->firmwareDir =3D g_steal_pointer(&fakefirmwareemptydir); DO_TEST_PREPARE_ERROR("firmware-efi"); @@ -345,10 +350,13 @@ mymain(void) driver.caps =3D virBhyveCapsBuild(); /* bhyve does not support UTC clock on ARM */ driver.bhyvecaps ^=3D BHYVE_CAP_RTC_UTC; + /* bhyve does not support NUMA on ARM */ + driver.bhyvecaps &=3D ~BHYVE_CAP_NUMA; =20 DO_TEST("base"); DO_TEST("console"); DO_TEST("bootloader"); + DO_TEST_FAILURE("numa"); =20 virObjectUnref(driver.caps); virObjectUnref(driver.xmlopt); diff --git a/tests/bhyvexml2xmloutdata/x86_64/bhyvexml2xmlout-numa.xml b/te= sts/bhyvexml2xmloutdata/x86_64/bhyvexml2xmlout-numa.xml new file mode 100644 index 0000000000..ecc147db78 --- /dev/null +++ b/tests/bhyvexml2xmloutdata/x86_64/bhyvexml2xmlout-numa.xml @@ -0,0 +1,42 @@ + + bhyve + df3be7e7-a104-11e3-aeb0-50e5492bd3dc + 219136 + 219136 + 8 + + hvm + + + + + + + + + + destroy + restart + destroy + + + + + +
+ + + +
+ + +
+ + + + + +
+ + + diff --git a/tests/bhyvexml2xmltest.c b/tests/bhyvexml2xmltest.c index 7f9de2bc36..950aaea672 100644 --- a/tests/bhyvexml2xmltest.c +++ b/tests/bhyvexml2xmltest.c @@ -132,6 +132,7 @@ mymain(void) DO_TEST_DIFFERENT("passthru-multiple-devs"); DO_TEST_DIFFERENT("slirp"); DO_TEST_DIFFERENT("virtio-scsi"); + DO_TEST_DIFFERENT("numa"); =20 /* Address allocation tests */ DO_TEST_DIFFERENT("addr-single-sata-disk"); --=20 2.52.0