From nobody Fri Sep 26 16:53:05 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail header.i=@intel.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=linux.intel.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1645121096573859.2310094710065; Thu, 17 Feb 2022 10:04:56 -0800 (PST) Received: from localhost ([::1]:59404 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nKl9D-0005Eq-RK for importer@patchew.org; Thu, 17 Feb 2022 13:04:55 -0500 Received: from eggs.gnu.org ([209.51.188.92]:36070) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKksH-0002BK-16; Thu, 17 Feb 2022 12:47:25 -0500 Received: from mga12.intel.com ([192.55.52.136]:31713) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nKksE-0007HU-Jt; Thu, 17 Feb 2022 12:47:24 -0500 Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2022 09:47:08 -0800 Received: from lmaniak-dev.igk.intel.com ([10.55.249.72]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Feb 2022 09:47:06 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1645120042; x=1676656042; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Uj2YMh9ir1Vxf9iPOgrxN9E7Up1+bw6MwsSbpnopV90=; b=O7s2BJ9ZZ/uYAmYByJRSBXpoYIYCDyKPayJNMIS3SIl+ewsp1N35asSy Ke1nEUJxkTEVX+nRqxL2wDnRB6tX3L6dmgxvmv+Q8hPJJJcxp8ADIo00W lMU8rDWg4R4DsENvOEF+toEum8tEB9CimFk7Ceq8n7gjtX5f8MRjBfXib Re+iddsFauif8rWpSY6ALcn6OUgf72WYE6JiX2J+dBoA6jX2uCTDwY+Yj zyIOtw9myeb3wMzj3uzHJwZNZkGMZzjTwRQ0dJMsAqBZn1D5B0BpO5M2p xKPM4DQCqbQBmMp22DALzOA5c9YKTrL/V27KP0qL+ptoBMUUEY+n4UIZ0 w==; X-IronPort-AV: E=McAfee;i="6200,9189,10261"; a="230894089" X-IronPort-AV: E=Sophos;i="5.88,376,1635231600"; d="scan'208";a="230894089" X-IronPort-AV: E=Sophos;i="5.88,376,1635231600"; d="scan'208";a="545746189" From: Lukasz Maniak To: qemu-devel@nongnu.org Subject: [PATCH v5 09/15] hw/nvme: Make max_ioqpairs and msix_qsize configurable in runtime Date: Thu, 17 Feb 2022 18:44:58 +0100 Message-Id: <20220217174504.1051716-10-lukasz.maniak@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220217174504.1051716-1-lukasz.maniak@linux.intel.com> References: <20220217174504.1051716-1-lukasz.maniak@linux.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: none client-ip=192.55.52.136; envelope-from=lukasz.maniak@linux.intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Klaus Jensen , Keith Busch , Lukasz Maniak , qemu-block@nongnu.org, =?UTF-8?q?=C5=81ukasz=20Gieryk?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1645121098175100001 From: =C5=81ukasz Gieryk The NVMe device defines two properties: max_ioqpairs, msix_qsize. Having them as constants is problematic for SR-IOV support. SR-IOV introduces virtual resources (queues, interrupts) that can be assigned to PF and its dependent VFs. Each device, following a reset, should work with the configured number of queues. A single constant is no longer sufficient to hold the whole state. This patch tries to solve the problem by introducing additional variables in NvmeCtrl=E2=80=99s state. The variables for, e.g., managing qu= eues are therefore organized as: - n->params.max_ioqpairs =E2=80=93 no changes, constant set by the user - n->(mutable_state) =E2=80=93 (not a part of this patch) user-configurabl= e, specifies number of queues available _after_ reset - n->conf_ioqpairs - (new) used in all the places instead of the =E2=80=98= old=E2=80=99 n->params.max_ioqpairs; initialized in realize() and updated during reset() to reflect user=E2=80=99s changes to the mutable state Since the number of available i/o queues and interrupts can change in runtime, buffers for sq/cqs and the MSIX-related structures are allocated big enough to handle the limits, to completely avoid the complicated reallocation. A helper function (nvme_update_msixcap_ts) updates the corresponding capability register, to signal configuration changes. Signed-off-by: =C5=81ukasz Gieryk --- hw/nvme/ctrl.c | 52 ++++++++++++++++++++++++++++++++++---------------- hw/nvme/nvme.h | 2 ++ 2 files changed, 38 insertions(+), 16 deletions(-) diff --git a/hw/nvme/ctrl.c b/hw/nvme/ctrl.c index 7c1dd80f21d..f1b4026e4f8 100644 --- a/hw/nvme/ctrl.c +++ b/hw/nvme/ctrl.c @@ -445,12 +445,12 @@ static bool nvme_nsid_valid(NvmeCtrl *n, uint32_t nsi= d) =20 static int nvme_check_sqid(NvmeCtrl *n, uint16_t sqid) { - return sqid < n->params.max_ioqpairs + 1 && n->sq[sqid] !=3D NULL ? 0 = : -1; + return sqid < n->conf_ioqpairs + 1 && n->sq[sqid] !=3D NULL ? 0 : -1; } =20 static int nvme_check_cqid(NvmeCtrl *n, uint16_t cqid) { - return cqid < n->params.max_ioqpairs + 1 && n->cq[cqid] !=3D NULL ? 0 = : -1; + return cqid < n->conf_ioqpairs + 1 && n->cq[cqid] !=3D NULL ? 0 : -1; } =20 static void nvme_inc_cq_tail(NvmeCQueue *cq) @@ -4188,8 +4188,7 @@ static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_err_invalid_create_sq_cqid(cqid); return NVME_INVALID_CQID | NVME_DNR; } - if (unlikely(!sqid || sqid > n->params.max_ioqpairs || - n->sq[sqid] !=3D NULL)) { + if (unlikely(!sqid || sqid > n->conf_ioqpairs || n->sq[sqid] !=3D NULL= )) { trace_pci_nvme_err_invalid_create_sq_sqid(sqid); return NVME_INVALID_QID | NVME_DNR; } @@ -4541,8 +4540,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_create_cq(prp1, cqid, vector, qsize, qflags, NVME_CQ_FLAGS_IEN(qflags) !=3D 0); =20 - if (unlikely(!cqid || cqid > n->params.max_ioqpairs || - n->cq[cqid] !=3D NULL)) { + if (unlikely(!cqid || cqid > n->conf_ioqpairs || n->cq[cqid] !=3D NULL= )) { trace_pci_nvme_err_invalid_create_cq_cqid(cqid); return NVME_INVALID_QID | NVME_DNR; } @@ -4558,7 +4556,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeReque= st *req) trace_pci_nvme_err_invalid_create_cq_vector(vector); return NVME_INVALID_IRQ_VECTOR | NVME_DNR; } - if (unlikely(vector >=3D n->params.msix_qsize)) { + if (unlikely(vector >=3D n->conf_msix_qsize)) { trace_pci_nvme_err_invalid_create_cq_vector(vector); return NVME_INVALID_IRQ_VECTOR | NVME_DNR; } @@ -5155,13 +5153,12 @@ defaults: =20 break; case NVME_NUMBER_OF_QUEUES: - result =3D (n->params.max_ioqpairs - 1) | - ((n->params.max_ioqpairs - 1) << 16); + result =3D (n->conf_ioqpairs - 1) | ((n->conf_ioqpairs - 1) << 16); trace_pci_nvme_getfeat_numq(result); break; case NVME_INTERRUPT_VECTOR_CONF: iv =3D dw11 & 0xffff; - if (iv >=3D n->params.max_ioqpairs + 1) { + if (iv >=3D n->conf_ioqpairs + 1) { return NVME_INVALID_FIELD | NVME_DNR; } =20 @@ -5316,10 +5313,10 @@ static uint16_t nvme_set_feature(NvmeCtrl *n, NvmeR= equest *req) =20 trace_pci_nvme_setfeat_numq((dw11 & 0xffff) + 1, ((dw11 >> 16) & 0xffff) + 1, - n->params.max_ioqpairs, - n->params.max_ioqpairs); - req->cqe.result =3D cpu_to_le32((n->params.max_ioqpairs - 1) | - ((n->params.max_ioqpairs - 1) << 16)= ); + n->conf_ioqpairs, + n->conf_ioqpairs); + req->cqe.result =3D cpu_to_le32((n->conf_ioqpairs - 1) | + ((n->conf_ioqpairs - 1) << 16)); break; case NVME_ASYNCHRONOUS_EVENT_CONF: n->features.async_config =3D dw11; @@ -5757,8 +5754,24 @@ static void nvme_process_sq(void *opaque) } } =20 +static void nvme_update_msixcap_ts(PCIDevice *pci_dev, uint32_t table_size) +{ + uint8_t *config; + + if (!msix_present(pci_dev)) { + return; + } + + assert(table_size > 0 && table_size <=3D pci_dev->msix_entries_nr); + + config =3D pci_dev->config + pci_dev->msix_cap; + pci_set_word_by_mask(config + PCI_MSIX_FLAGS, PCI_MSIX_FLAGS_QSIZE, + table_size - 1); +} + static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetType rst) { + PCIDevice *pci_dev =3D &n->parent_obj; NvmeNamespace *ns; int i; =20 @@ -5788,15 +5801,17 @@ static void nvme_ctrl_reset(NvmeCtrl *n, NvmeResetT= ype rst) g_free(event); } =20 - if (!pci_is_vf(&n->parent_obj) && n->params.sriov_max_vfs) { + if (!pci_is_vf(pci_dev) && n->params.sriov_max_vfs) { if (rst !=3D NVME_RESET_CONTROLLER) { - pcie_sriov_pf_disable_vfs(&n->parent_obj); + pcie_sriov_pf_disable_vfs(pci_dev); } } =20 n->aer_queued =3D 0; n->outstanding_aers =3D 0; n->qs_created =3D false; + + nvme_update_msixcap_ts(pci_dev, n->conf_msix_qsize); } =20 static void nvme_ctrl_shutdown(NvmeCtrl *n) @@ -6507,6 +6522,9 @@ static void nvme_init_state(NvmeCtrl *n) NvmeSecCtrlEntry *sctrl; int i; =20 + n->conf_ioqpairs =3D n->params.max_ioqpairs; + n->conf_msix_qsize =3D n->params.msix_qsize; + /* add one to max_ioqpairs to account for the admin queue pair */ n->reg_size =3D pow2ceil(sizeof(NvmeBar) + 2 * (n->params.max_ioqpairs + 1) * NVME_DB_SIZE= ); @@ -6668,6 +6686,8 @@ static int nvme_init_pci(NvmeCtrl *n, PCIDevice *pci_= dev, Error **errp) } } =20 + nvme_update_msixcap_ts(pci_dev, n->conf_msix_qsize); + if (n->params.cmb_size_mb) { nvme_init_cmb(n, pci_dev); } diff --git a/hw/nvme/nvme.h b/hw/nvme/nvme.h index 5ba07b62dff..314a2894759 100644 --- a/hw/nvme/nvme.h +++ b/hw/nvme/nvme.h @@ -434,6 +434,8 @@ typedef struct NvmeCtrl { uint64_t starttime_ms; uint16_t temperature; uint8_t smart_critical_warning; + uint32_t conf_msix_qsize; + uint32_t conf_ioqpairs; =20 struct { MemoryRegion mem; --=20 2.25.1