From nobody Tue Dec 16 18:34:36 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F9DAC10F05 for ; Mon, 11 Dec 2023 17:39:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344933AbjLKRjd (ORCPT ); Mon, 11 Dec 2023 12:39:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344792AbjLKRj2 (ORCPT ); Mon, 11 Dec 2023 12:39:28 -0500 Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0B3C93; Mon, 11 Dec 2023 09:39:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cisco.com; i=@cisco.com; l=15141; q=dns/txt; s=iport; t=1702316374; x=1703525974; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cDVNd5/6nHGLx8yPVJU8kVqgncHWoByE84K2bDAD1DE=; b=Gep9K/MLeSVkpzqJNxpdd+bDj+6G9dyQ3MEaWuRazlJydFkLrleGkvhq X28Du7nuW4EarFVZcz/heb5nqEGVt0iFg//MFug3jcSmHe40nK2602h1D cpPPcEIRj7VjTjj0XytmZOM1svYMcQu+DyvfRGmXTKeBLXcNY3fkl8cuP k=; X-CSE-ConnectionGUID: /3oLVJ0mQSesEyUKTx1QBg== X-CSE-MsgGUID: PV/LhXTjQw6XJ6IZ4r2f0w== X-IronPort-AV: E=Sophos;i="6.04,268,1695686400"; d="scan'208";a="194303971" Received: from rcdn-core-1.cisco.com ([173.37.93.152]) by alln-iport-5.cisco.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Dec 2023 17:39:33 +0000 Received: from localhost.cisco.com ([10.193.101.253]) (authenticated bits=0) by rcdn-core-1.cisco.com (8.15.2/8.15.2) with ESMTPSA id 3BBHaKr1009547 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Dec 2023 17:39:32 GMT From: Karan Tilak Kumar To: sebaddel@cisco.com Cc: arulponn@cisco.com, djhawar@cisco.com, gcboffa@cisco.com, mkai2@cisco.com, satishkh@cisco.com, jejb@linux.ibm.com, martin.petersen@oracle.com, linux-scsi@vger.kernel.org, linux-kernel@vger.kernel.org, Karan Tilak Kumar , kernel test robot Subject: [PATCH v6 07/13] scsi: fnic: Modify ISRs to support multiqueue(MQ) Date: Mon, 11 Dec 2023 09:36:11 -0800 Message-Id: <20231211173617.932990-8-kartilak@cisco.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231211173617.932990-1-kartilak@cisco.com> References: <20231211173617.932990-1-kartilak@cisco.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Authenticated-User: kartilak@cisco.com X-Outbound-SMTP-Client: 10.193.101.253, [10.193.101.253] X-Outbound-Node: rcdn-core-1.cisco.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Modify interrupt service routines for INTx, MSI, and MSI-x to support multiqueue. Modify parameter list of fnic_wq_copy_cmpl_handler to take cq_index. Modify fnic_cleanup function to use the new function call of fnic_wq_copy_cmpl_handler. Refactor code to set interrupt mode to MSI-x to a new function. Add a new stat for intx_dummy. Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202310251847.4T8BVZAZ-lkp@int= el.com/ Reviewed-by: Sesidhar Baddela Reviewed-by: Arulprabhu Ponnusamy Signed-off-by: Karan Tilak Kumar --- Changes between v4 and v5: Incorporate review comments from Martin: Modify patch commits to include a "---" separator. Changes between v2 and v3: Incorporate the following review comments from Hannes: Replace cpy_wq_base with copy_wq_base. Remove C99 style comment. Extend review comments of FNIC_MAIN_DBG and FNIC_SCSI_DBG to FNIC_ISR_DBG: Use fnic_num as an argument to FNIC_ISR_DBG. Modify definition of FNIC_ISR_DBG. Host number is still used as an argument to FNIC_ISR_DBG since it in turn uses shost_printk. Removed reviewed by tag from Hannes due to additional modifications. Changes between v1 and v2: Suppress warning from kernel test bot. --- drivers/scsi/fnic/fnic.h | 9 +- drivers/scsi/fnic/fnic_isr.c | 166 ++++++++++++++++++++++++--------- drivers/scsi/fnic/fnic_main.c | 4 +- drivers/scsi/fnic/fnic_scsi.c | 38 +++----- drivers/scsi/fnic/fnic_stats.h | 1 + 5 files changed, 144 insertions(+), 74 deletions(-) diff --git a/drivers/scsi/fnic/fnic.h b/drivers/scsi/fnic/fnic.h index 47173f1eec1e..8b3e4fe1ce92 100644 --- a/drivers/scsi/fnic/fnic.h +++ b/drivers/scsi/fnic/fnic.h @@ -160,9 +160,11 @@ do { \ FNIC_CHECK_LOGGING(FNIC_SCSI_LOGGING, \ shost_printk(kern_level, host, fmt, ##args);) =20 -#define FNIC_ISR_DBG(kern_level, host, fmt, args...) \ +#define FNIC_ISR_DBG(kern_level, host, fnic_num, fmt, args...) \ FNIC_CHECK_LOGGING(FNIC_ISR_LOGGING, \ - shost_printk(kern_level, host, fmt, ##args);) + shost_printk(kern_level, host, \ + "fnic<%d>: %s: %d: " fmt, fnic_num,\ + __func__, __LINE__, ##args);) =20 #define FNIC_MAIN_NOTE(kern_level, host, fmt, args...) \ shost_printk(kern_level, host, fmt, ##args) @@ -349,6 +351,7 @@ extern const struct attribute_group *fnic_host_groups[]; =20 void fnic_clear_intr_mode(struct fnic *fnic); int fnic_set_intr_mode(struct fnic *fnic); +int fnic_set_intr_mode_msix(struct fnic *fnic); void fnic_free_intr(struct fnic *fnic); int fnic_request_intr(struct fnic *fnic); =20 @@ -375,7 +378,7 @@ void fnic_scsi_cleanup(struct fc_lport *); void fnic_scsi_abort_io(struct fc_lport *); void fnic_empty_scsi_cleanup(struct fc_lport *); void fnic_exch_mgr_reset(struct fc_lport *, u32, u32); -int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int); +int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do, unsi= gned int cq_index); int fnic_wq_cmpl_handler(struct fnic *fnic, int); int fnic_flogi_reg_handler(struct fnic *fnic, u32); void fnic_wq_copy_cleanup_handler(struct vnic_wq_copy *wq, diff --git a/drivers/scsi/fnic/fnic_isr.c b/drivers/scsi/fnic/fnic_isr.c index dff9689023e4..ff85441c6cea 100644 --- a/drivers/scsi/fnic/fnic_isr.c +++ b/drivers/scsi/fnic/fnic_isr.c @@ -38,8 +38,13 @@ static irqreturn_t fnic_isr_legacy(int irq, void *data) fnic_log_q_error(fnic); } =20 + if (pba & (1 << FNIC_INTX_DUMMY)) { + atomic64_inc(&fnic->fnic_stats.misc_stats.intx_dummy); + vnic_intr_return_all_credits(&fnic->intr[FNIC_INTX_DUMMY]); + } + if (pba & (1 << FNIC_INTX_WQ_RQ_COPYWQ)) { - work_done +=3D fnic_wq_copy_cmpl_handler(fnic, io_completions); + work_done +=3D fnic_wq_copy_cmpl_handler(fnic, io_completions, FNIC_MQ_C= Q_INDEX); work_done +=3D fnic_wq_cmpl_handler(fnic, -1); work_done +=3D fnic_rq_cmpl_handler(fnic, -1); =20 @@ -60,7 +65,7 @@ static irqreturn_t fnic_isr_msi(int irq, void *data) fnic->fnic_stats.misc_stats.last_isr_time =3D jiffies; atomic64_inc(&fnic->fnic_stats.misc_stats.isr_count); =20 - work_done +=3D fnic_wq_copy_cmpl_handler(fnic, io_completions); + work_done +=3D fnic_wq_copy_cmpl_handler(fnic, io_completions, FNIC_MQ_CQ= _INDEX); work_done +=3D fnic_wq_cmpl_handler(fnic, -1); work_done +=3D fnic_rq_cmpl_handler(fnic, -1); =20 @@ -109,12 +114,22 @@ static irqreturn_t fnic_isr_msix_wq_copy(int irq, voi= d *data) { struct fnic *fnic =3D data; unsigned long wq_copy_work_done =3D 0; + int i; =20 fnic->fnic_stats.misc_stats.last_isr_time =3D jiffies; atomic64_inc(&fnic->fnic_stats.misc_stats.isr_count); =20 - wq_copy_work_done =3D fnic_wq_copy_cmpl_handler(fnic, io_completions); - vnic_intr_return_credits(&fnic->intr[FNIC_MSIX_WQ_COPY], + i =3D irq - fnic->msix[0].irq_num; + if (i >=3D fnic->wq_copy_count + fnic->copy_wq_base || + i < 0 || fnic->msix[i].irq_num !=3D irq) { + for (i =3D fnic->copy_wq_base; i < fnic->wq_copy_count + fnic->copy_wq_b= ase ; i++) { + if (fnic->msix[i].irq_num =3D=3D irq) + break; + } + } + + wq_copy_work_done =3D fnic_wq_copy_cmpl_handler(fnic, io_completions, i); + vnic_intr_return_credits(&fnic->intr[i], wq_copy_work_done, 1 /* unmask intr */, 1 /* reset intr timer */); @@ -128,7 +143,7 @@ static irqreturn_t fnic_isr_msix_err_notify(int irq, vo= id *data) fnic->fnic_stats.misc_stats.last_isr_time =3D jiffies; atomic64_inc(&fnic->fnic_stats.misc_stats.isr_count); =20 - vnic_intr_return_all_credits(&fnic->intr[FNIC_MSIX_ERR_NOTIFY]); + vnic_intr_return_all_credits(&fnic->intr[fnic->err_intr_offset]); fnic_log_q_error(fnic); fnic_handle_link_event(fnic); =20 @@ -186,26 +201,30 @@ int fnic_request_intr(struct fnic *fnic) fnic->msix[FNIC_MSIX_WQ].isr =3D fnic_isr_msix_wq; fnic->msix[FNIC_MSIX_WQ].devid =3D fnic; =20 - sprintf(fnic->msix[FNIC_MSIX_WQ_COPY].devname, - "%.11s-scsi-wq", fnic->name); - fnic->msix[FNIC_MSIX_WQ_COPY].isr =3D fnic_isr_msix_wq_copy; - fnic->msix[FNIC_MSIX_WQ_COPY].devid =3D fnic; + for (i =3D fnic->copy_wq_base; i < fnic->wq_copy_count + fnic->copy_wq_b= ase; i++) { + sprintf(fnic->msix[i].devname, + "%.11s-scsi-wq-%d", fnic->name, i-FNIC_MSIX_WQ_COPY); + fnic->msix[i].isr =3D fnic_isr_msix_wq_copy; + fnic->msix[i].devid =3D fnic; + } =20 - sprintf(fnic->msix[FNIC_MSIX_ERR_NOTIFY].devname, + sprintf(fnic->msix[fnic->err_intr_offset].devname, "%.11s-err-notify", fnic->name); - fnic->msix[FNIC_MSIX_ERR_NOTIFY].isr =3D + fnic->msix[fnic->err_intr_offset].isr =3D fnic_isr_msix_err_notify; - fnic->msix[FNIC_MSIX_ERR_NOTIFY].devid =3D fnic; + fnic->msix[fnic->err_intr_offset].devid =3D fnic; =20 - for (i =3D 0; i < ARRAY_SIZE(fnic->msix); i++) { - err =3D request_irq(pci_irq_vector(fnic->pdev, i), - fnic->msix[i].isr, 0, - fnic->msix[i].devname, - fnic->msix[i].devid); + for (i =3D 0; i < fnic->intr_count; i++) { + fnic->msix[i].irq_num =3D pci_irq_vector(fnic->pdev, i); + + err =3D request_irq(fnic->msix[i].irq_num, + fnic->msix[i].isr, 0, + fnic->msix[i].devname, + fnic->msix[i].devid); if (err) { - shost_printk(KERN_ERR, fnic->lport->host, - "MSIX: request_irq" - " failed %d\n", err); + FNIC_ISR_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "request_irq failed with error: %d\n", + err); fnic_free_intr(fnic); break; } @@ -220,44 +239,99 @@ int fnic_request_intr(struct fnic *fnic) return err; } =20 -int fnic_set_intr_mode(struct fnic *fnic) +int fnic_set_intr_mode_msix(struct fnic *fnic) { unsigned int n =3D ARRAY_SIZE(fnic->rq); unsigned int m =3D ARRAY_SIZE(fnic->wq); unsigned int o =3D ARRAY_SIZE(fnic->hw_copy_wq); + unsigned int min_irqs =3D n + m + 1 + 1; /*rq, raw wq, wq, err*/ =20 /* - * Set interrupt mode (INTx, MSI, MSI-X) depending - * system capabilities. - * - * Try MSI-X first - * * We need n RQs, m WQs, o Copy WQs, n+m+o CQs, and n+m+o+1 INTRs * (last INTR is used for WQ/RQ errors and notification area) */ - if (fnic->rq_count >=3D n && - fnic->raw_wq_count >=3D m && - fnic->wq_copy_count >=3D o && - fnic->cq_count >=3D n + m + o) { - int vecs =3D n + m + o + 1; - - if (pci_alloc_irq_vectors(fnic->pdev, vecs, vecs, - PCI_IRQ_MSIX) =3D=3D vecs) { + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "rq-array size: %d wq-array size: %d copy-wq array size: %d\n", + n, m, o); + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "rq_count: %d raw_wq_count: %d wq_copy_count: %d cq_count: %d\n", + fnic->rq_count, fnic->raw_wq_count, + fnic->wq_copy_count, fnic->cq_count); + + if (fnic->rq_count <=3D n && fnic->raw_wq_count <=3D m && + fnic->wq_copy_count <=3D o) { + int vec_count =3D 0; + int vecs =3D fnic->rq_count + fnic->raw_wq_count + fnic->wq_copy_count += 1; + + vec_count =3D pci_alloc_irq_vectors(fnic->pdev, min_irqs, vecs, + PCI_IRQ_MSIX | PCI_IRQ_AFFINITY); + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "allocated %d MSI-X vectors\n", + vec_count); + + if (vec_count > 0) { + if (vec_count < vecs) { + FNIC_ISR_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "interrupts number mismatch: vec_count: %d vecs: %d\n", + vec_count, vecs); + if (vec_count < min_irqs) { + FNIC_ISR_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "no interrupts for copy wq\n"); + return 1; + } + } + fnic->rq_count =3D n; fnic->raw_wq_count =3D m; - fnic->wq_copy_count =3D o; - fnic->wq_count =3D m + o; - fnic->cq_count =3D n + m + o; - fnic->intr_count =3D vecs; - fnic->err_intr_offset =3D FNIC_MSIX_ERR_NOTIFY; - - FNIC_ISR_DBG(KERN_DEBUG, fnic->lport->host, - "Using MSI-X Interrupts\n"); - vnic_dev_set_intr_mode(fnic->vdev, - VNIC_DEV_INTR_MODE_MSIX); + fnic->copy_wq_base =3D fnic->rq_count + fnic->raw_wq_count; + fnic->wq_copy_count =3D vec_count - n - m - 1; + fnic->wq_count =3D fnic->raw_wq_count + fnic->wq_copy_count; + if (fnic->cq_count !=3D vec_count - 1) { + FNIC_ISR_DBG(KERN_ERR, fnic->lport->host, fnic->fnic_num, + "CQ count: %d does not match MSI-X vector count: %d\n", + fnic->cq_count, vec_count); + fnic->cq_count =3D vec_count - 1; + } + fnic->intr_count =3D vec_count; + fnic->err_intr_offset =3D fnic->rq_count + fnic->wq_count; + + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "rq_count: %d raw_wq_count: %d copy_wq_base: %d\n", + fnic->rq_count, + fnic->raw_wq_count, fnic->copy_wq_base); + + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "wq_copy_count: %d wq_count: %d cq_count: %d\n", + fnic->wq_copy_count, + fnic->wq_count, fnic->cq_count); + + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "intr_count: %d err_intr_offset: %u", + fnic->intr_count, + fnic->err_intr_offset); + + vnic_dev_set_intr_mode(fnic->vdev, VNIC_DEV_INTR_MODE_MSIX); + FNIC_ISR_DBG(KERN_INFO, fnic->lport->host, fnic->fnic_num, + "fnic using MSI-X\n"); return 0; } } + return 1; +} + +int fnic_set_intr_mode(struct fnic *fnic) +{ + int ret_status =3D 0; + + /* + * Set interrupt mode (INTx, MSI, MSI-X) depending + * system capabilities. + * + * Try MSI-X first + */ + ret_status =3D fnic_set_intr_mode_msix(fnic); + if (ret_status =3D=3D 0) + return ret_status; =20 /* * Next try MSI @@ -277,7 +351,7 @@ int fnic_set_intr_mode(struct fnic *fnic) fnic->intr_count =3D 1; fnic->err_intr_offset =3D 0; =20 - FNIC_ISR_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_ISR_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Using MSI Interrupts\n"); vnic_dev_set_intr_mode(fnic->vdev, VNIC_DEV_INTR_MODE_MSI); =20 @@ -303,7 +377,7 @@ int fnic_set_intr_mode(struct fnic *fnic) fnic->cq_count =3D 3; fnic->intr_count =3D 3; =20 - FNIC_ISR_DBG(KERN_DEBUG, fnic->lport->host, + FNIC_ISR_DBG(KERN_DEBUG, fnic->lport->host, fnic->fnic_num, "Using Legacy Interrupts\n"); vnic_dev_set_intr_mode(fnic->vdev, VNIC_DEV_INTR_MODE_INTX); =20 diff --git a/drivers/scsi/fnic/fnic_main.c b/drivers/scsi/fnic/fnic_main.c index dacefbeb44f0..5726c0fb6f0f 100644 --- a/drivers/scsi/fnic/fnic_main.c +++ b/drivers/scsi/fnic/fnic_main.c @@ -476,6 +476,7 @@ static int fnic_cleanup(struct fnic *fnic) { unsigned int i; int err; + int raw_wq_rq_counts; =20 vnic_dev_disable(fnic->vdev); for (i =3D 0; i < fnic->intr_count; i++) @@ -495,10 +496,11 @@ static int fnic_cleanup(struct fnic *fnic) err =3D vnic_wq_copy_disable(&fnic->hw_copy_wq[i]); if (err) return err; + raw_wq_rq_counts =3D fnic->raw_wq_count + fnic->rq_count; + fnic_wq_copy_cmpl_handler(fnic, -1, i + raw_wq_rq_counts); } =20 /* Clean up completed IOs and FCS frames */ - fnic_wq_copy_cmpl_handler(fnic, io_completions); fnic_wq_cmpl_handler(fnic, -1); fnic_rq_cmpl_handler(fnic, -1); =20 diff --git a/drivers/scsi/fnic/fnic_scsi.c b/drivers/scsi/fnic/fnic_scsi.c index b7dc304446f8..c7f3bfc650d1 100644 --- a/drivers/scsi/fnic/fnic_scsi.c +++ b/drivers/scsi/fnic/fnic_scsi.c @@ -1319,10 +1319,8 @@ static int fnic_fcpio_cmpl_handler(struct vnic_dev *= vdev, * fnic_wq_copy_cmpl_handler * Routine to process wq copy */ -int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do) +int fnic_wq_copy_cmpl_handler(struct fnic *fnic, int copy_work_to_do, unsi= gned int cq_index) { - unsigned int wq_work_done =3D 0; - unsigned int i, cq_index; unsigned int cur_work_done; struct misc_stats *misc_stats =3D &fnic->fnic_stats.misc_stats; u64 start_jiffies =3D 0; @@ -1330,28 +1328,20 @@ int fnic_wq_copy_cmpl_handler(struct fnic *fnic, in= t copy_work_to_do) u64 delta_jiffies =3D 0; u64 delta_ms =3D 0; =20 - for (i =3D 0; i < fnic->wq_copy_count; i++) { - cq_index =3D i + fnic->raw_wq_count + fnic->rq_count; - - start_jiffies =3D jiffies; - cur_work_done =3D vnic_cq_copy_service(&fnic->cq[cq_index], - fnic_fcpio_cmpl_handler, - copy_work_to_do); - end_jiffies =3D jiffies; - - wq_work_done +=3D cur_work_done; - delta_jiffies =3D end_jiffies - start_jiffies; - if (delta_jiffies > - (u64) atomic64_read(&misc_stats->max_isr_jiffies)) { - atomic64_set(&misc_stats->max_isr_jiffies, - delta_jiffies); - delta_ms =3D jiffies_to_msecs(delta_jiffies); - atomic64_set(&misc_stats->max_isr_time_ms, delta_ms); - atomic64_set(&misc_stats->corr_work_done, - cur_work_done); - } + start_jiffies =3D jiffies; + cur_work_done =3D vnic_cq_copy_service(&fnic->cq[cq_index], + fnic_fcpio_cmpl_handler, + copy_work_to_do); + end_jiffies =3D jiffies; + delta_jiffies =3D end_jiffies - start_jiffies; + if (delta_jiffies > (u64) atomic64_read(&misc_stats->max_isr_jiffies)) { + atomic64_set(&misc_stats->max_isr_jiffies, delta_jiffies); + delta_ms =3D jiffies_to_msecs(delta_jiffies); + atomic64_set(&misc_stats->max_isr_time_ms, delta_ms); + atomic64_set(&misc_stats->corr_work_done, cur_work_done); } - return wq_work_done; + + return cur_work_done; } =20 static bool fnic_cleanup_io_iter(struct scsi_cmnd *sc, void *data) diff --git a/drivers/scsi/fnic/fnic_stats.h b/drivers/scsi/fnic/fnic_stats.h index bdf639eef8cf..07d1556e3c32 100644 --- a/drivers/scsi/fnic/fnic_stats.h +++ b/drivers/scsi/fnic/fnic_stats.h @@ -103,6 +103,7 @@ struct misc_stats { atomic64_t rport_not_ready; atomic64_t frame_errors; atomic64_t current_port_speed; + atomic64_t intx_dummy; }; =20 struct fnic_stats { --=20 2.31.1