From nobody Sun Apr 6 23:10:11 2025
Delivered-To: importer@patchew.org
Authentication-Results: mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass(p=reject dis=none) header.from=linux.ibm.com
ARC-Seal: i=1; a=rsa-sha256; t=1743528294; cv=none;
d=zohomail.com; s=zohoarc;
b=nH+9Ge6lguVxwGimeok/7oHACc9mM0Jlid77KOK9+87iYgWdPZ96gSPK0zO6wXL+oHLHS6YhENkuNgkruNdi/YVxQmm322SactLYf+TCM1QX2Tg0JDUQeMTqf0x0QyLJLeJSzoWoUrfBuX5OISiJj/09O6FG0WEgqnjyKW6ZuKc=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
s=zohoarc;
t=1743528294;
h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To;
bh=nC7FB1pwbm5octQaaoSTqyZZdYWIxjXLrstXWTcH8iQ=;
b=AyaOeQWB4T3ZEDDCV3Y1ShlZaXBhwhmDrpYHwb+QsLWMwsOPANYq1/Y/8nr1QYFoRJ4VKvZsvf0uW1sFmv9vBz6TmioTVReYRL4Enw5UGp5b4bxymG254f7Jat9hoU1oJJ6ikj9A3RuPfgk7CGZdFafHmxOzTVtGo1S4jVpQbdg=
ARC-Authentication-Results: i=1; mx.zohomail.com;
dkim=pass;
spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as
permitted sender)
smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org;
dmarc=pass header.from=<alifm@linux.ibm.com> (p=reject dis=none)
Return-Path: <qemu-devel-bounces+importer=patchew.org@nongnu.org>
Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by
mx.zohomail.com
with SMTPS id 1743528294165985.243871727986;
Tue, 1 Apr 2025 10:24:54 -0700 (PDT)
Received: from localhost ([::1] helo=lists1p.gnu.org)
by lists.gnu.org with esmtp (Exim 4.90_1)
(envelope-from <qemu-devel-bounces@nongnu.org>)
id 1tzfKa-0004jr-KC; Tue, 01 Apr 2025 13:23:20 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <alifm@linux.ibm.com>)
id 1tzfKI-0004gb-8A; Tue, 01 Apr 2025 13:23:03 -0400
Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <alifm@linux.ibm.com>)
id 1tzfKF-0005w0-Ce; Tue, 01 Apr 2025 13:23:01 -0400
Received: from pps.filterd (m0353725.ppops.net [127.0.0.1])
by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id
531Gf9Xw002667;
Tue, 1 Apr 2025 17:22:53 GMT
Received: from ppma22.wdc07v.mail.ibm.com
(5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92])
by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 45r290n2mf-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 01 Apr 2025 17:22:52 +0000 (GMT)
Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1])
by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id
531GQhea004829;
Tue, 1 Apr 2025 17:22:52 GMT
Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5])
by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 45pujyunds-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT);
Tue, 01 Apr 2025 17:22:52 +0000
Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com
[10.241.53.100])
by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id
531HMpdZ19857988
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK);
Tue, 1 Apr 2025 17:22:51 GMT
Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1])
by IMSVA (Postfix) with ESMTP id E302E58058;
Tue, 1 Apr 2025 17:22:50 +0000 (GMT)
Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1])
by IMSVA (Postfix) with ESMTP id EF1AD5805D;
Tue, 1 Apr 2025 17:22:49 +0000 (GMT)
Received: from IBM-D32RQW3.ibm.com (unknown [9.61.254.127])
by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP;
Tue, 1 Apr 2025 17:22:49 +0000 (GMT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc
:content-transfer-encoding:content-type:date:from:in-reply-to
:message-id:mime-version:references:subject:to; s=pp1; bh=nC7FB1
pwbm5octQaaoSTqyZZdYWIxjXLrstXWTcH8iQ=; b=iABUfT7TaVghQrkAoQZlQJ
/Xl/4Q63x6F15yrU06xJbhWqupQAcBo8VSkE/k+RoIwTE5ZOp4+Jie1/gh/Fh67G
AEoapkmgHfF1n7oGugOo31sEQlRDtiyptgRp+tdE07WMmcNdOKioAYVxOEjeec3D
zFfSB+l6sJQp7CqCbdwh2vbpZtvz0v5g2hs/yQW6c1XtN7YmAe9Cpqr798SqVvpE
werYQ8VCTUbOAoTn/CSIIU5/JRNDSKx3QOvvHH7fHcn1OjAyfSApHPkTCheg0yDn
0HqEmCBf3pA3codRIR5FU+OfyfDw03hS+uuih2eW76F31si2gGZVbITLAq+ivRGg
==
From: Farhan Ali <alifm@linux.ibm.com>
To: qemu-devel@nongnu.org
Cc: qemu-block@nongnu.org, qemu-s390x@nongnu.org, stefanha@redhat.com,
fam@euphon.net, philmd@linaro.org, kwolf@redhat.com, hreitz@redhat.com,
thuth@redhat.com, alifm@linux.ibm.com, mjrosato@linux.ibm.com,
schnelle@linux.ibm.com
Subject: [PATCH v3 3/3] block/nvme: Use host PCI MMIO API
Date: Tue, 1 Apr 2025 10:22:46 -0700
Message-ID: <20250401172246.2688-4-alifm@linux.ibm.com>
X-Mailer: git-send-email 2.43.0
In-Reply-To: <20250401172246.2688-1-alifm@linux.ibm.com>
References: <20250401172246.2688-1-alifm@linux.ibm.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
X-TM-AS-GCONF: 00
X-Proofpoint-ORIG-GUID: ouiuaVic1suWTdaC7VFd9gvw0ttw8QTT
X-Proofpoint-GUID: ouiuaVic1suWTdaC7VFd9gvw0ttw8QTT
X-Proofpoint-Virus-Version: vendor=baseguard
engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34
definitions=2025-04-01_06,2025-04-01_01,2024-11-22_01
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0
malwarescore=0
priorityscore=1501 bulkscore=0 clxscore=1015 adultscore=0 spamscore=0
lowpriorityscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999
impostorscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx
scancount=1 engine=8.19.0-2502280000 definitions=main-2504010104
Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17
as permitted sender) client-ip=209.51.188.17;
envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org;
helo=lists.gnu.org;
Received-SPF: pass client-ip=148.163.158.5; envelope-from=alifm@linux.ibm.com;
helo=mx0b-001b2d01.pphosted.com
X-Spam_score_int: -26
X-Spam_score: -2.7
X-Spam_bar: --
X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7,
RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001,
RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001,
RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001,
SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-BeenThere: qemu-devel@nongnu.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <https://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org
Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org
X-ZohoMail-DKIM: pass (identity @ibm.com)
X-ZM-MESSAGEID: 1743528295494019000
Use the host PCI MMIO functions to read/write
to NVMe registers, rather than directly accessing
them.
Reviewed-by: Philippe Mathieu-Daud=C3=A9 <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Farhan Ali <alifm@linux.ibm.com>
---
block/nvme.c | 37 +++++++++++++++++++++----------------
1 file changed, 21 insertions(+), 16 deletions(-)
diff --git a/block/nvme.c b/block/nvme.c
index bbf7c23dcd..ba66fbc93a 100644
--- a/block/nvme.c
+++ b/block/nvme.c
@@ -18,6 +18,7 @@
#include "qobject/qstring.h"
#include "qemu/defer-call.h"
#include "qemu/error-report.h"
+#include "qemu/host-pci-mmio.h"
#include "qemu/main-loop.h"
#include "qemu/module.h"
#include "qemu/cutils.h"
@@ -60,7 +61,7 @@ typedef struct {
uint8_t *queue;
uint64_t iova;
/* Hardware MMIO register */
- volatile uint32_t *doorbell;
+ uint32_t *doorbell;
} NVMeQueue;
=20
typedef struct {
@@ -100,7 +101,7 @@ struct BDRVNVMeState {
QEMUVFIOState *vfio;
void *bar0_wo_map;
/* Memory mapped registers */
- volatile struct {
+ struct {
uint32_t sq_tail;
uint32_t cq_head;
} *doorbells;
@@ -292,7 +293,7 @@ static void nvme_kick(NVMeQueuePair *q)
assert(!(q->sq.tail & 0xFF00));
/* Fence the write to submission queue entry before notifying the devi=
ce. */
smp_wmb();
- *q->sq.doorbell =3D cpu_to_le32(q->sq.tail);
+ host_pci_mmio_write_32(q->sq.doorbell, q->sq.tail);
q->inflight +=3D q->need_kick;
q->need_kick =3D 0;
}
@@ -441,7 +442,7 @@ static bool nvme_process_completion(NVMeQueuePair *q)
if (progress) {
/* Notify the device so it can post more completions. */
smp_mb_release();
- *q->cq.doorbell =3D cpu_to_le32(q->cq.head);
+ host_pci_mmio_write_32(q->cq.doorbell, q->cq.head);
nvme_wake_free_req_locked(q);
}
=20
@@ -460,7 +461,7 @@ static void nvme_process_completion_bh(void *opaque)
* so notify the device that it has space to fill in more completions =
now.
*/
smp_mb_release();
- *q->cq.doorbell =3D cpu_to_le32(q->cq.head);
+ host_pci_mmio_write_32(q->cq.doorbell, q->cq.head);
nvme_wake_free_req_locked(q);
=20
nvme_process_completion(q);
@@ -749,9 +750,10 @@ static int nvme_init(BlockDriverState *bs, const char =
*device, int namespace,
int ret;
uint64_t cap;
uint32_t ver;
+ uint32_t cc;
uint64_t timeout_ms;
uint64_t deadline, now;
- volatile NvmeBar *regs =3D NULL;
+ NvmeBar *regs =3D NULL;
=20
qemu_co_mutex_init(&s->dma_map_lock);
qemu_co_queue_init(&s->dma_flush_queue);
@@ -779,7 +781,7 @@ static int nvme_init(BlockDriverState *bs, const char *=
device, int namespace,
/* Perform initialize sequence as described in NVMe spec "7.6.1
* Initialization". */
=20
- cap =3D le64_to_cpu(regs->cap);
+ cap =3D host_pci_mmio_read_64(®s->cap);
trace_nvme_controller_capability_raw(cap);
trace_nvme_controller_capability("Maximum Queue Entries Supported",
1 + NVME_CAP_MQES(cap));
@@ -805,16 +807,17 @@ static int nvme_init(BlockDriverState *bs, const char=
*device, int namespace,
bs->bl.request_alignment =3D s->page_size;
timeout_ms =3D MIN(500 * NVME_CAP_TO(cap), 30000);
=20
- ver =3D le32_to_cpu(regs->vs);
+ ver =3D host_pci_mmio_read_32(®s->vs);
trace_nvme_controller_spec_version(extract32(ver, 16, 16),
extract32(ver, 8, 8),
extract32(ver, 0, 8));
=20
/* Reset device to get a clean state. */
- regs->cc =3D cpu_to_le32(le32_to_cpu(regs->cc) & 0xFE);
+ cc =3D host_pci_mmio_read_32(®s->cc);
+ host_pci_mmio_write_32(®s->cc, cc & 0xFE);
/* Wait for CSTS.RDY =3D 0. */
deadline =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + timeout_ms * SCA=
LE_MS;
- while (NVME_CSTS_RDY(le32_to_cpu(regs->csts))) {
+ while (NVME_CSTS_RDY(host_pci_mmio_read_32(®s->csts))) {
if (qemu_clock_get_ns(QEMU_CLOCK_REALTIME) > deadline) {
error_setg(errp, "Timeout while waiting for device to reset (%"
PRId64 " ms)",
@@ -843,19 +846,21 @@ static int nvme_init(BlockDriverState *bs, const char=
*device, int namespace,
s->queues[INDEX_ADMIN] =3D q;
s->queue_count =3D 1;
QEMU_BUILD_BUG_ON((NVME_QUEUE_SIZE - 1) & 0xF000);
- regs->aqa =3D cpu_to_le32(((NVME_QUEUE_SIZE - 1) << AQA_ACQS_SHIFT) |
- ((NVME_QUEUE_SIZE - 1) << AQA_ASQS_SHIFT));
- regs->asq =3D cpu_to_le64(q->sq.iova);
- regs->acq =3D cpu_to_le64(q->cq.iova);
+ host_pci_mmio_write_32(®s->aqa,
+ ((NVME_QUEUE_SIZE - 1) << AQA_ACQS_SHIFT) |
+ ((NVME_QUEUE_SIZE - 1) << AQA_ASQS_SHIFT));
+ host_pci_mmio_write_64(®s->asq, q->sq.iova);
+ host_pci_mmio_write_64(®s->acq, q->cq.iova);
=20
/* After setting up all control registers we can enable device now. */
- regs->cc =3D cpu_to_le32((ctz32(NVME_CQ_ENTRY_BYTES) << CC_IOCQES_SHIF=
T) |
+ host_pci_mmio_write_32(®s->cc,
+ (ctz32(NVME_CQ_ENTRY_BYTES) << CC_IOCQES_SHIFT)=
|
(ctz32(NVME_SQ_ENTRY_BYTES) << CC_IOSQES_SHIFT)=
|
CC_EN_MASK);
/* Wait for CSTS.RDY =3D 1. */
now =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
deadline =3D now + timeout_ms * SCALE_MS;
- while (!NVME_CSTS_RDY(le32_to_cpu(regs->csts))) {
+ while (!NVME_CSTS_RDY(host_pci_mmio_read_32(®s->csts))) {
if (qemu_clock_get_ns(QEMU_CLOCK_REALTIME) > deadline) {
error_setg(errp, "Timeout while waiting for device to start (%"
PRId64 " ms)",
--=20
2.43.0