From nobody Mon Feb 9 10:29:11 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1656020604032242.49930744444464; Thu, 23 Jun 2022 14:43:24 -0700 (PDT) Received: from localhost ([::1]:40276 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o4Ubh-0005S3-U6 for importer@patchew.org; Thu, 23 Jun 2022 17:43:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54994) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o4UTr-0004UF-Pw; Thu, 23 Jun 2022 17:35:17 -0400 Received: from wout5-smtp.messagingengine.com ([64.147.123.21]:37471) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o4UTp-0006df-RN; Thu, 23 Jun 2022 17:35:15 -0400 Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id EDBE43200974; Thu, 23 Jun 2022 17:35:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Thu, 23 Jun 2022 17:35:12 -0400 Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 23 Jun 2022 17:35:08 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=irrelevant.dk; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; t=1656020110; x= 1656106510; bh=oUsR2Oq1gDWwLmgxFMFxwSUd8p7vpsc4CW1xrXDQ7yE=; b=f DQBAlgs6wnQZMIT337O7aTnRdzVK5EigJ+uDShZ+sAAqkLTk2lx8WL09rlyb6GXU HuQBFPrd7Ih4Xl1h8qOXNzcU+t3WpQc94ytiL/Kn45OW8qbxXVZGHSTRRdFwMv1d qVHdNk6r8IZhOtAVySPtHiFUUcpQ/4wBhNKVyfCus5zvVcyz7CwmsmJHBgLuo6au uH4IV4bg+s8deuX2HuRJRiEpK+8n1WiaV9YV2/krpo/IsClXRSCButjhl00mqy9i cjriiPhW3/xypzdCFVNVn2q94AWPH/722AeyTsqBHfHkD0cFzbVnEUYbK1hkFwxg 0KfRhjgDMTlnTKGB+nv5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1656020110; x= 1656106510; bh=oUsR2Oq1gDWwLmgxFMFxwSUd8p7vpsc4CW1xrXDQ7yE=; b=c Lx3RHbfnVFjX9FfZq4K5l54QhyRSXULAqV+4xMNUSyS3zgg8tseUumtAor/n5/bV gyMhlBSHe9fp2jDNdsN6AlsxLIKS6zWrlG9FWv0uYWqacJLbHDAHlSahAWkcQMOu xr4GQFmpB9Q+aZLlV/Ko6XbUJcAUxwkvcvhlzqpdCR1x4RqCo9TPNIlG3CbmI13S UonCK+yBT/FpnQ+7eBiBn+YFTR1e9Iaz+7Dihaq9IBIzrNqxRmRw5OeogjimJkY9 GlweF+DDcDtDY4gmtXyyVDUVlBKKYFkdcT1USIkgyhNB0LEx/x5mbxkN1NwdB4KK sihtDkA+OGkTwKxMSy3Vw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrudefjedgudeifecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvfevufffkffojghfgggtgfesthekredtredtjeenucfhrhhomhepmfhl rghushculfgvnhhsvghnuceoihhtshesihhrrhgvlhgvvhgrnhhtrdgukheqnecuggftrf grthhtvghrnhepfeevtdeuteeuudffvefggfdtfedtueelfffhieegffekgeefjeefffet jeeihfdvnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epihhtshesihhrrhgvlhgvvhgrnhhtrdgukh X-ME-Proxy: Feedback-ID: idc91472f:Fastmail From: Klaus Jensen To: Peter Maydell , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Igor Mammedov , Ani Sinha , Hanna Reitz , Kevin Wolf , "Michael S. Tsirkin" , Klaus Jensen , qemu-block@nongnu.org, Keith Busch , Fam Zheng , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Marcel Apfelbaum , =?UTF-8?q?=C5=81ukasz=20Gieryk?= , Klaus Jensen Subject: [PULL 05/15] hw/nvme: Make max_ioqpairs and msix_qsize configurable in runtime Date: Thu, 23 Jun 2022 23:34:32 +0200 Message-Id: <20220623213442.67789-6-its@irrelevant.dk> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220623213442.67789-1-its@irrelevant.dk> References: <20220623213442.67789-1-its@irrelevant.dk> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=64.147.123.21; envelope-from=its@irrelevant.dk; helo=wout5-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZM-MESSAGEID: 1656020604824100003 From: =C5=81ukasz Gieryk The NVMe device defines two properties: max_ioqpairs, msix_qsize. Having them as constants is problematic for SR-IOV support. SR-IOV introduces virtual resources (queues, interrupts) that can be assigned to PF and its dependent VFs. Each device, following a reset, should work with the configured number of queues. A single constant is no longer sufficient to hold the whole state. This patch tries to solve the problem by introducing additional variables in NvmeCtrl=E2=80=99s state. The variables for, e.g., managing qu= eues are therefore organized as: - n->params.max_ioqpairs =E2=80=93 no changes, constant set by the user - n->(mutable_state) =E2=80=93 (not a part of this patch) user-configurabl= e, specifies number of queues available _after_ reset - n->conf_ioqpairs - (new) used in all the places instead of the =E2=80=98= old=E2=80=99 n->params.max_ioqpairs; initialized in realize() and updated during reset() to reflect user=E2=80=99s changes to the mutable state Since the number of available i/o queues and interrupts can change in runtime, buffers for sq/cqs and the MSIX-related structures are allocated big enough to handle the limits, to completely avoid the complicated reallocation. A helper function (nvme_update_msixcap_ts) updates the corresponding capability register, to signal configuration changes. Signed-off-by: =C5=81ukasz Gieryk Reviewed-by: Klaus Jensen Acked-by: Michael S. Tsirkin Signed-off-by: Klaus Jensen --- hw/nvme/ctrl.c | 52 ++++++++++++++++++++++++++++++++++---------------- hw/nvme/nvme.h | 2 ++ 2 files changed, 38 insertions(+), 16 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 5ae80f114003..e970234a2c87 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -448,12 +448,12 @@ static bool nvme_nsid_valid(NvmeCtrl *n, uint32_t nsi= d) =20 static int nvme_check_sqid(NvmeCtrl *n, uint16_t sqid) { - return sqid < n->params.max_ioqpairs + 1 && n->sq[sqid] !=3D NULL ? 0 = : -1; + return sqid < n->conf_ioqpairs + 1 && n->sq[sqid] !=3D NULL ? 0 : -1; } =20 static int nvme_check_cqid(NvmeCtrl *n, uint16_t cqid) { - return cqid < n->params.max_ioqpairs + 1 && n->cq[cqid] !=3D NULL ? 0 = : -1; + return cqid < n->conf_ioqpairs + 1 && n->cq[cqid] !=3D NULL ? 0 : -1; } =20 static void nvme_inc_cq_tail(NvmeCQueue *cq) @@ -4295,8 +4295,7 @@ static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_err_invalid_create_sq_cqid(cqid); return NVME_INVALID_CQID | NVME_DNR; } - if (unlikely(!sqid || sqid > n->params.max_ioqpairs || - n->sq[sqid] !=3D NULL)) { + if (unlikely(!sqid || sqid > n->conf_ioqpairs || n->sq[sqid] !=3D NULL= )) { trace_pci_nvme_err_invalid_create_sq_sqid(sqid); return NVME_INVALID_QID | NVME_DNR; } @@ -4648,8 +4647,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_create_cq(prp1, cqid, vector, qsize, qflags, NVME_CQ_FLAGS_IEN(qflags) !=3D 0); =20 - if (unlikely(!cqid || cqid > n->params.max_ioqpairs || - n->cq[cqid] !=3D NULL)) { + if (unlikely(!cqid || cqid > n->conf_ioqpairs || n->cq[cqid] !=3D NULL= )) { trace_pci_nvme_err_invalid_create_cq_cqid(cqid); return NVME_INVALID_QID | NVME_DNR; } @@ -4665,7 +4663,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_err_invalid_create_cq_vector(vector); return NVME_INVALID_IRQ_VECTOR | NVME_DNR; } - if (unlikely(vector >=3D n->params.msix_qsize)) { + if (unlikely(vector >=3D n->conf_msix_qsize)) { trace_pci_nvme_err_invalid_create_cq_vector(vector); return NVME_INVALID_IRQ_VECTOR | NVME_DNR; } @@ -5263,13 +5261,12 @@ defaults: =20 break; case NVME_NUMBER_OF_QUEUES: - result =3D (n->params.max_ioqpairs - 1) | - ((n->params.max_ioqpairs - 1) << 16); + result =3D (n->conf_ioqpairs - 1) | ((n->conf_ioqpairs - 1) << 16); trace_pci_nvme_getfeat_numq(result); break; case NVME_INTERRUPT_VECTOR_CONF: iv =3D dw11 & 0xffff; - if (iv >=3D n->params.max_ioqpairs + 1) { + if (iv >=3D n->conf_ioqpairs + 1) { return NVME_INVALID_FIELD | NVME_DNR; } =20 @@ -5425,10 +5422,10 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeR= equest *req) =20 trace_pci_nvme_setfeat_numq((dw11 & 0xffff) + 1, ((dw11 >> 16) & 0xffff) + 1, - n->params.max_ioqpairs, - n->params.max_ioqpairs); - req->cqe.result =3D cpu_to_le32((n->params.max_ioqpairs - 1) | - ((n->params.max_ioqpairs - 1) << 16)= ); + n->conf_ioqpairs, + n->conf_ioqpairs); + req->cqe.result =3D cpu_to_le32((n->conf_ioqpairs - 1) | + ((n->conf_ioqpairs - 1) << 16)); break; case NVME_ASYNCHRONOUS_EVENT_CONF: n->features.async_config =3D dw11; @@ -5903,8 +5900,24 @@ static void nvme_process_sq(void *opaque) } } =20 +static void nvme_update_msixcap_ts(PCIDevice *pci_dev, uint32_t table_size) +{ + uint8_t *config; + + if (!msix_present(pci_dev)) { + return; + } + + assert(table_size > 0 && table_size <=3D pci_dev->msix_entries_nr); + + config =3D pci_dev->config + pci_dev->msix_cap; + pci_set_word_by_mask(config + PCI_MSIX_FLAGS, PCI_MSIX_FLAGS_QSIZE, + table_size - 1); +} + static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetType rst) { + PCIDevice *pci_dev =3D &n->parent_obj; NvmeNamespace *ns; int i; =20 @@ -5934,15 +5947,17 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) g_free(event); } =20 - if (!pci_is_vf(&n->parent_obj) && n->params.sriov_max_vfs) { + if (!pci_is_vf(pci_dev) && n->params.sriov_max_vfs) { if (rst !=3D NVME_RESET_CONTROLLER) { - pcie_sriov_pf_disable_vfs(&n->parent_obj); + pcie_sriov_pf_disable_vfs(pci_dev); } } =20 n->aer_queued =3D 0; n->outstanding_aers =3D 0; n->qs_created =3D false; + + nvme_update_msixcap_ts(pci_dev, n->conf_msix_qsize); } =20 static void nvme_ctrl_shutdown(NvmeCtrl *n) @@ -6653,6 +6668,9 @@ static void nvme_init_state(NvmeCtrl *n) NvmeSecCtrlEntry *sctrl; int i; =20 + n->conf_ioqpairs =3D n->params.max_ioqpairs; + n->conf_msix_qsize =3D n->params.msix_qsize; + /* add one to max_ioqpairs to account for the admin queue pair */ n->reg_size =3D pow2ceil(sizeof(NvmeBar) + 2 * (n->params.max_ioqpairs + 1) * NVME_DB_SIZE= ); @@ -6814,6 +6832,8 @@ static int nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_= dev, Error **errp) } } =20 + nvme_update_msixcap_ts(pci_dev, n->conf_msix_qsize); + if (n->params.cmb_size_mb) { nvme_init_cmb(n, pci_dev); } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 7b317d3dc469..aab4962fb857 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -439,6 +439,8 @@ typedef struct NvmeCtrl { uint64_t starttime_ms; uint16_t temperature; uint8_t smart_critical_warning; + uint32_t conf_msix_qsize; + uint32_t conf_ioqpairs; =20 struct { MemoryRegion mem; --=20 2.36.1