From nobody Fri Apr 4 03:41:01 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=linux.ibm.com ARC-Seal: i=1; a=rsa-sha256; t=1743528259; cv=none; d=zohomail.com; s=zohoarc; b=OI/qkPbh+o5oChqbEwl4FSPeJCVo0RdBoG/ayRGBT5+Dv5nChwq3hts5mZRnGTI6ETlrQwDPwUh6mLgsrUs/zeEB03LwsOWAxVTBT4O3RYEgiZxyzcNywxMPZbTnOLpTPYX+54Pocp4tlJiOqqT+hUeErMkTxz7SHhbM7rjOHM8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1743528259; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=XbxOmOxaCGJ2wrgJRqkKjhBcMBVejLmaBs+JzNXKqsI=; b=nKmdrt+Dukw8XqhwsEZcC++CiqTV3WKdOCrArZ6gzuVODYjQPBQEmpj+5gzfAW0uVEYRFXUN3ldgcdRYqYtHY5WKOByQOEmUzCPM3x05itL9Y5ouSF+AG9fS9jfnZL2mzpJn52u3BZgC6KVWnNdeUKaHkLfADiltJaXO3ka02bc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1743528259456314.02022125269275; Tue, 1 Apr 2025 10:24:19 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tzfKZ-0004j3-5h; Tue, 01 Apr 2025 13:23:19 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKG-0004fv-Br; Tue, 01 Apr 2025 13:23:01 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKD-0005v5-FM; Tue, 01 Apr 2025 13:23:00 -0400 Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 531Gf3gm021141; Tue, 1 Apr 2025 17:22:51 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 45rhjps1q9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:51 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 531GailC004788; Tue, 1 Apr 2025 17:22:50 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 45pujyundk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:50 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 531HMmh515008436 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 1 Apr 2025 17:22:49 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A5C5A58059; Tue, 1 Apr 2025 17:22:48 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8526158061; Tue, 1 Apr 2025 17:22:47 +0000 (GMT) Received: from IBM-D32RQW3.ibm.com (unknown [9.61.254.127]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Tue, 1 Apr 2025 17:22:47 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=XbxOmOxaCGJ2wrgJR qkKjhBcMBVejLmaBs+JzNXKqsI=; b=JkNWaIJGidtrhmdMF5EjeNtC4IuEJE/AG S7BFCez9mZ91wuwo9hTnX4+OHCp0LOwyIYVJh1vbCqJZUOtjOdcX53xD6a5c8usK IXY44qdq++vz/txXljoKl7dbgohqpOfSX/k1a/16Ums75pHQI4T4Hyf6IFkZN1r5 I1zxpdwNUsxON+/PNMXnu4ykAIt/iAvcynwi8eGKQfaA34bI3+CqG7sIw6AFB7sA clYldpsT3RhVxH0+xMwh086f4BaRessvMxfZLdXPFpZ5JnUjxPJOg35Olal39Q70 eC0yx2dbaU0OnL4iopimDBVGi8wSQYDzcox6a+3ZrlzbBSWXZlkIQ== From: Farhan Ali To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, qemu-s390x@nongnu.org, stefanha@redhat.com, fam@euphon.net, philmd@linaro.org, kwolf@redhat.com, hreitz@redhat.com, thuth@redhat.com, alifm@linux.ibm.com, mjrosato@linux.ibm.com, schnelle@linux.ibm.com Subject: [PATCH v3 1/3] util: Add functions for s390x mmio read/write Date: Tue, 1 Apr 2025 10:22:44 -0700 Message-ID: <20250401172246.2688-2-alifm@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250401172246.2688-1-alifm@linux.ibm.com> References: <20250401172246.2688-1-alifm@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: wF1sj1XXzUVJei_bkVFhr-UEzVek6GyB X-Proofpoint-ORIG-GUID: wF1sj1XXzUVJei_bkVFhr-UEzVek6GyB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-01_06,2025-04-01_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=629 phishscore=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 spamscore=0 mlxscore=0 impostorscore=0 suspectscore=0 adultscore=0 malwarescore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2502280000 definitions=main-2504010104 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=148.163.156.1; envelope-from=alifm@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @ibm.com) X-ZM-MESSAGEID: 1743528261365019100 Content-Type: text/plain; charset="utf-8" Starting with z15 (or newer) we can execute mmio instructions from userspace. On older platforms where we don't have these instructions available we can fallback to using system calls to access the PCI mapped resources. This patch adds helper functions for mmio reads and writes for s390x. Reviewed-by: Stefan Hajnoczi Reviewed-by: Niklas Schnelle Signed-off-by: Farhan Ali --- include/qemu/s390x_pci_mmio.h | 24 ++++++ util/meson.build | 2 + util/s390x_pci_mmio.c | 148 ++++++++++++++++++++++++++++++++++ 3 files changed, 174 insertions(+) create mode 100644 include/qemu/s390x_pci_mmio.h create mode 100644 util/s390x_pci_mmio.c diff --git a/include/qemu/s390x_pci_mmio.h b/include/qemu/s390x_pci_mmio.h new file mode 100644 index 0000000000..c5f63ecefa --- /dev/null +++ b/include/qemu/s390x_pci_mmio.h @@ -0,0 +1,24 @@ +/* + * s390x PCI MMIO definitions + * + * Copyright 2025 IBM Corp. + * Author(s): Farhan Ali + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ +#ifndef S390X_PCI_MMIO_H +#define S390X_PCI_MMIO_H + +#ifdef __s390x__ +uint8_t s390x_pci_mmio_read_8(const void *ioaddr); +uint16_t s390x_pci_mmio_read_16(const void *ioaddr); +uint32_t s390x_pci_mmio_read_32(const void *ioaddr); +uint64_t s390x_pci_mmio_read_64(const void *ioaddr); + +void s390x_pci_mmio_write_8(void *ioaddr, uint8_t val); +void s390x_pci_mmio_write_16(void *ioaddr, uint16_t val); +void s390x_pci_mmio_write_32(void *ioaddr, uint32_t val); +void s390x_pci_mmio_write_64(void *ioaddr, uint64_t val); +#endif /* __s390x__ */ + +#endif /* S390X_PCI_MMIO_H */ diff --git a/util/meson.build b/util/meson.build index 780b5977a8..acb21592f9 100644 --- a/util/meson.build +++ b/util/meson.build @@ -131,4 +131,6 @@ elif cpu in ['ppc', 'ppc64'] util_ss.add(files('cpuinfo-ppc.c')) elif cpu in ['riscv32', 'riscv64'] util_ss.add(files('cpuinfo-riscv.c')) +elif cpu =3D=3D 's390x' + util_ss.add(files('s390x_pci_mmio.c')) endif diff --git a/util/s390x_pci_mmio.c b/util/s390x_pci_mmio.c new file mode 100644 index 0000000000..820458a026 --- /dev/null +++ b/util/s390x_pci_mmio.c @@ -0,0 +1,148 @@ +/* + * s390x PCI MMIO definitions + * + * Copyright 2025 IBM Corp. + * Author(s): Farhan Ali + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include +#include +#include "qemu/s390x_pci_mmio.h" +#include "elf.h" + +union register_pair { + unsigned __int128 pair; + struct { + uint64_t even; + uint64_t odd; + }; +}; + +static bool is_mio_supported; + +static __attribute__((constructor)) void check_is_mio_supported(void) +{ + is_mio_supported =3D !!(qemu_getauxval(AT_HWCAP) & HWCAP_S390_PCI_MIO); +} + +static uint64_t s390x_pcilgi(const void *ioaddr, size_t len) +{ + union register_pair ioaddr_len =3D { .even =3D (uint64_t)ioaddr, + .odd =3D len }; + uint64_t val; + int cc; + + asm volatile( + /* pcilgi */ + ".insn rre,0xb9d60000,%[val],%[ioaddr_len]\n" + "ipm %[cc]\n" + "srl %[cc],28\n" + : [cc] "=3Dd"(cc), [val] "=3Dd"(val), + [ioaddr_len] "+&d"(ioaddr_len.pair) :: "cc"); + + if (cc) { + val =3D -1ULL; + } + + return val; +} + +static void s390x_pcistgi(void *ioaddr, uint64_t val, size_t len) +{ + union register_pair ioaddr_len =3D {.even =3D (uint64_t)ioaddr, .odd = =3D len}; + + asm volatile ( + /* pcistgi */ + ".insn rre,0xb9d40000,%[val],%[ioaddr_len]\n" + : [ioaddr_len] "+&d" (ioaddr_len.pair) + : [val] "d" (val) + : "cc", "memory"); +} + +uint8_t s390x_pci_mmio_read_8(const void *ioaddr) +{ + uint8_t val =3D 0; + + if (is_mio_supported) { + val =3D s390x_pcilgi(ioaddr, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_read, ioaddr, &val, sizeof(val)); + } + return val; +} + +uint16_t s390x_pci_mmio_read_16(const void *ioaddr) +{ + uint16_t val =3D 0; + + if (is_mio_supported) { + val =3D s390x_pcilgi(ioaddr, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_read, ioaddr, &val, sizeof(val)); + } + return val; +} + +uint32_t s390x_pci_mmio_read_32(const void *ioaddr) +{ + uint32_t val =3D 0; + + if (is_mio_supported) { + val =3D s390x_pcilgi(ioaddr, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_read, ioaddr, &val, sizeof(val)); + } + return val; +} + +uint64_t s390x_pci_mmio_read_64(const void *ioaddr) +{ + uint64_t val =3D 0; + + if (is_mio_supported) { + val =3D s390x_pcilgi(ioaddr, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_read, ioaddr, &val, sizeof(val)); + } + return val; +} + +void s390x_pci_mmio_write_8(void *ioaddr, uint8_t val) +{ + if (is_mio_supported) { + s390x_pcistgi(ioaddr, val, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_write, ioaddr, &val, sizeof(val)); + } +} + +void s390x_pci_mmio_write_16(void *ioaddr, uint16_t val) +{ + if (is_mio_supported) { + s390x_pcistgi(ioaddr, val, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_write, ioaddr, &val, sizeof(val)); + } +} + +void s390x_pci_mmio_write_32(void *ioaddr, uint32_t val) +{ + if (is_mio_supported) { + s390x_pcistgi(ioaddr, val, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_write, ioaddr, &val, sizeof(val)); + } +} + +void s390x_pci_mmio_write_64(void *ioaddr, uint64_t val) +{ + if (is_mio_supported) { + s390x_pcistgi(ioaddr, val, sizeof(val)); + } else { + syscall(__NR_s390_pci_mmio_write, ioaddr, &val, sizeof(val)); + } +} + --=20 2.43.0 From nobody Fri Apr 4 03:41:01 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=linux.ibm.com ARC-Seal: i=1; a=rsa-sha256; t=1743528254; cv=none; d=zohomail.com; s=zohoarc; b=KNe3L2TWUM2jLllsyPLU+wkrS1Pk35975YZTDB/zjTIvNU3wXXtoHE7WzBrqiSw2kT2GiP+Fm2deZkIxus+QYDZlmQOvhpt2KbiQsmWc6D1yR92fn0Gw9wn7tbPsKkMllYxv20nwr9Cc+CFx6LG/doGiKzNs2cDYUfPsXaJkQ44= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1743528254; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=bCFobdStZf3H/XWan64B1rnOwW39hOKaIS5l8IoldU0=; b=WPqdvbGcc1/HzJep4cERNcou9LGhP4gK7k+AMqHqh0N9nKCnlvPsxsuFrr2FNR6rf/Zn9NlqM/qxcE3mFs7ke6uUPxHxhXp0McPG6TCLL1abMnF9JHFgH3Vp0GU/GCdGJvdq4uZQb0pfbUt4kTF+9JxNpAdDZuPswMJLoVki3h0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1743528254446519.3669185084681; Tue, 1 Apr 2025 10:24:14 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tzfKX-0004iZ-7C; Tue, 01 Apr 2025 13:23:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKF-0004fn-Si; Tue, 01 Apr 2025 13:23:00 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKD-0005vH-5s; Tue, 01 Apr 2025 13:22:58 -0400 Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 531Gf4G4002444; Tue, 1 Apr 2025 17:22:52 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 45r290n2ma-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:51 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 531DJqAo009999; Tue, 1 Apr 2025 17:22:51 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([172.16.1.73]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 45pv6nug9q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:51 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 531HMnAB30016014 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 1 Apr 2025 17:22:50 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C3ACC58058; Tue, 1 Apr 2025 17:22:49 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C9BAB5805D; Tue, 1 Apr 2025 17:22:48 +0000 (GMT) Received: from IBM-D32RQW3.ibm.com (unknown [9.61.254.127]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Tue, 1 Apr 2025 17:22:48 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=bCFobd StZf3H/XWan64B1rnOwW39hOKaIS5l8IoldU0=; b=s0PCyytgMbZ1K71h59q4V9 jKNTud1KeGRqqlJRMdsKMr0r49qS8WYoE3Zgj5sGjbcCNQzzK+A/pvmYvAxXppUF fk6JvEB++9daWvdBSsnfgr8aMSt3kBjA1eg9+HOcTX8AFVdKoHggQS2IPkMP4+nJ MUGKB36dvYOiUAcws+Fs4PRAPC/m5K9p02DHwX27lImFEy/4GVayibEAlSN/M2Cy cFuSEqbUBSVhRdr5uV74dljt3S1z6XIcubfSnoY8ZMHT4px6eiqX6FWhszYz2bwh mkvtLET2bAgP3SE0iBbVLyXgOr66ijpaL3eQGkXg9O5RI9B8+sx5lE4cSFfh/wIg == From: Farhan Ali To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, qemu-s390x@nongnu.org, stefanha@redhat.com, fam@euphon.net, philmd@linaro.org, kwolf@redhat.com, hreitz@redhat.com, thuth@redhat.com, alifm@linux.ibm.com, mjrosato@linux.ibm.com, schnelle@linux.ibm.com Subject: [PATCH v3 2/3] include: Add a header to define host PCI MMIO functions Date: Tue, 1 Apr 2025 10:22:45 -0700 Message-ID: <20250401172246.2688-3-alifm@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250401172246.2688-1-alifm@linux.ibm.com> References: <20250401172246.2688-1-alifm@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: UFtx_qgfqYEixo8eeXe9DJywsySy5gTp X-Proofpoint-GUID: UFtx_qgfqYEixo8eeXe9DJywsySy5gTp X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-01_06,2025-04-01_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 bulkscore=0 clxscore=1015 adultscore=0 spamscore=0 lowpriorityscore=0 suspectscore=0 mlxscore=0 mlxlogscore=823 impostorscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2502280000 definitions=main-2504010104 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=148.163.158.5; envelope-from=alifm@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @ibm.com) X-ZM-MESSAGEID: 1743528255444019000 Add a generic API for host PCI MMIO reads/writes (e.g. Linux VFIO BAR accesses). The functions access little endian memory and returns the result in host cpu endianness. Reviewed-by: Philippe Mathieu-Daud=C3=A9 Signed-off-by: Farhan Ali Reviewed-by: Stefan Hajnoczi --- include/qemu/host-pci-mmio.h | 116 +++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 include/qemu/host-pci-mmio.h diff --git a/include/qemu/host-pci-mmio.h b/include/qemu/host-pci-mmio.h new file mode 100644 index 0000000000..c26426524f --- /dev/null +++ b/include/qemu/host-pci-mmio.h @@ -0,0 +1,116 @@ +/* + * API for host PCI MMIO accesses (e.g. Linux VFIO BARs) + * + * Copyright 2025 IBM Corp. + * Author(s): Farhan Ali + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef HOST_PCI_MMIO_H +#define HOST_PCI_MMIO_H + +#include "qemu/bswap.h" +#include "qemu/s390x_pci_mmio.h" + + +static inline uint8_t host_pci_mmio_read_8(const void *ioaddr) +{ + uint8_t ret =3D 0; +#ifdef __s390x__ + ret =3D s390x_pci_mmio_read_8(ioaddr); +#else + /* Prevent the compiler from optimizing away the load */ + ret =3D *((volatile uint8_t *)ioaddr); +#endif + + return ret; +} + +static inline uint16_t host_pci_mmio_read_16(const void *ioaddr) +{ + uint16_t ret =3D 0; +#ifdef __s390x__ + ret =3D s390x_pci_mmio_read_16(ioaddr); +#else + /* Prevent the compiler from optimizing away the load */ + ret =3D *((volatile uint16_t *)ioaddr); +#endif + + return le16_to_cpu(ret); +} + +static inline uint32_t host_pci_mmio_read_32(const void *ioaddr) +{ + uint32_t ret =3D 0; +#ifdef __s390x__ + ret =3D s390x_pci_mmio_read_32(ioaddr); +#else + /* Prevent the compiler from optimizing away the load */ + ret =3D *((volatile uint32_t *)ioaddr); +#endif + + return le32_to_cpu(ret); +} + +static inline uint64_t host_pci_mmio_read_64(const void *ioaddr) +{ + uint64_t ret =3D 0; +#ifdef __s390x__ + ret =3D s390x_pci_mmio_read_64(ioaddr); +#else + /* Prevent the compiler from optimizing away the load */ + ret =3D *((volatile uint64_t *)ioaddr); +#endif + + return le64_to_cpu(ret); +} + +static inline void host_pci_mmio_write_8(void *ioaddr, uint8_t val) +{ + +#ifdef __s390x__ + s390x_pci_mmio_write_8(ioaddr, val); +#else + /* Prevent the compiler from optimizing away the store */ + *((volatile uint8_t *)ioaddr) =3D val; +#endif +} + +static inline void host_pci_mmio_write_16(void *ioaddr, uint16_t val) +{ + val =3D cpu_to_le16(val); + +#ifdef __s390x__ + s390x_pci_mmio_write_16(ioaddr, val); +#else + /* Prevent the compiler from optimizing away the store */ + *((volatile uint16_t *)ioaddr) =3D val; +#endif +} + +static inline void host_pci_mmio_write_32(void *ioaddr, uint32_t val) +{ + val =3D cpu_to_le32(val); + +#ifdef __s390x__ + s390x_pci_mmio_write_32(ioaddr, val); +#else + /* Prevent the compiler from optimizing away the store */ + *((volatile uint32_t *)ioaddr) =3D val; +#endif +} + +static inline void host_pci_mmio_write_64(void *ioaddr, uint64_t val) +{ + val =3D cpu_to_le64(val); + +#ifdef __s390x__ + s390x_pci_mmio_write_64(ioaddr, val); +#else + /* Prevent the compiler from optimizing away the store */ + *((volatile uint64_t *)ioaddr) =3D val; +#endif +} + +#endif --=20 2.43.0 From nobody Fri Apr 4 03:41:01 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=linux.ibm.com ARC-Seal: i=1; a=rsa-sha256; t=1743528294; cv=none; d=zohomail.com; s=zohoarc; b=nH+9Ge6lguVxwGimeok/7oHACc9mM0Jlid77KOK9+87iYgWdPZ96gSPK0zO6wXL+oHLHS6YhENkuNgkruNdi/YVxQmm322SactLYf+TCM1QX2Tg0JDUQeMTqf0x0QyLJLeJSzoWoUrfBuX5OISiJj/09O6FG0WEgqnjyKW6ZuKc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1743528294; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=nC7FB1pwbm5octQaaoSTqyZZdYWIxjXLrstXWTcH8iQ=; b=AyaOeQWB4T3ZEDDCV3Y1ShlZaXBhwhmDrpYHwb+QsLWMwsOPANYq1/Y/8nr1QYFoRJ4VKvZsvf0uW1sFmv9vBz6TmioTVReYRL4Enw5UGp5b4bxymG254f7Jat9hoU1oJJ6ikj9A3RuPfgk7CGZdFafHmxOzTVtGo1S4jVpQbdg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1743528294165985.243871727986; Tue, 1 Apr 2025 10:24:54 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tzfKa-0004jr-KC; Tue, 01 Apr 2025 13:23:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKI-0004gb-8A; Tue, 01 Apr 2025 13:23:03 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tzfKF-0005w0-Ce; Tue, 01 Apr 2025 13:23:01 -0400 Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 531Gf9Xw002667; Tue, 1 Apr 2025 17:22:53 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 45r290n2mf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:52 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 531GQhea004829; Tue, 1 Apr 2025 17:22:52 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 45pujyunds-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 01 Apr 2025 17:22:52 +0000 Received: from smtpav01.dal12v.mail.ibm.com (smtpav01.dal12v.mail.ibm.com [10.241.53.100]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 531HMpdZ19857988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 1 Apr 2025 17:22:51 GMT Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E302E58058; Tue, 1 Apr 2025 17:22:50 +0000 (GMT) Received: from smtpav01.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EF1AD5805D; Tue, 1 Apr 2025 17:22:49 +0000 (GMT) Received: from IBM-D32RQW3.ibm.com (unknown [9.61.254.127]) by smtpav01.dal12v.mail.ibm.com (Postfix) with ESMTP; Tue, 1 Apr 2025 17:22:49 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pp1; bh=nC7FB1 pwbm5octQaaoSTqyZZdYWIxjXLrstXWTcH8iQ=; b=iABUfT7TaVghQrkAoQZlQJ /Xl/4Q63x6F15yrU06xJbhWqupQAcBo8VSkE/k+RoIwTE5ZOp4+Jie1/gh/Fh67G AEoapkmgHfF1n7oGugOo31sEQlRDtiyptgRp+tdE07WMmcNdOKioAYVxOEjeec3D zFfSB+l6sJQp7CqCbdwh2vbpZtvz0v5g2hs/yQW6c1XtN7YmAe9Cpqr798SqVvpE werYQ8VCTUbOAoTn/CSIIU5/JRNDSKx3QOvvHH7fHcn1OjAyfSApHPkTCheg0yDn 0HqEmCBf3pA3codRIR5FU+OfyfDw03hS+uuih2eW76F31si2gGZVbITLAq+ivRGg == From: Farhan Ali To: qemu-devel@nongnu.org Cc: qemu-block@nongnu.org, qemu-s390x@nongnu.org, stefanha@redhat.com, fam@euphon.net, philmd@linaro.org, kwolf@redhat.com, hreitz@redhat.com, thuth@redhat.com, alifm@linux.ibm.com, mjrosato@linux.ibm.com, schnelle@linux.ibm.com Subject: [PATCH v3 3/3] block/nvme: Use host PCI MMIO API Date: Tue, 1 Apr 2025 10:22:46 -0700 Message-ID: <20250401172246.2688-4-alifm@linux.ibm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250401172246.2688-1-alifm@linux.ibm.com> References: <20250401172246.2688-1-alifm@linux.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: ouiuaVic1suWTdaC7VFd9gvw0ttw8QTT X-Proofpoint-GUID: ouiuaVic1suWTdaC7VFd9gvw0ttw8QTT X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1095,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-04-01_06,2025-04-01_01,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 bulkscore=0 clxscore=1015 adultscore=0 spamscore=0 lowpriorityscore=0 suspectscore=0 mlxscore=0 mlxlogscore=999 impostorscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2502280000 definitions=main-2504010104 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=148.163.158.5; envelope-from=alifm@linux.ibm.com; helo=mx0b-001b2d01.pphosted.com X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @ibm.com) X-ZM-MESSAGEID: 1743528295494019000 Use the host PCI MMIO functions to read/write to NVMe registers, rather than directly accessing them. Reviewed-by: Philippe Mathieu-Daud=C3=A9 Reviewed-by: Stefan Hajnoczi Signed-off-by: Farhan Ali --- block/nvme.c | 37 +++++++++++++++++++++---------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/block/nvme.c b/block/nvme.c index bbf7c23dcd..ba66fbc93a 100644 --- a/block/nvme.c +++ b/block/nvme.c @@ -18,6 +18,7 @@ #include "qobject/qstring.h" #include "qemu/defer-call.h" #include "qemu/error-report.h" +#include "qemu/host-pci-mmio.h" #include "qemu/main-loop.h" #include "qemu/module.h" #include "qemu/cutils.h" @@ -60,7 +61,7 @@ typedef struct { uint8_t *queue; uint64_t iova; /* Hardware MMIO register */ - volatile uint32_t *doorbell; + uint32_t *doorbell; } NVMeQueue; =20 typedef struct { @@ -100,7 +101,7 @@ struct BDRVNVMeState { QEMUVFIOState *vfio; void *bar0_wo_map; /* Memory mapped registers */ - volatile struct { + struct { uint32_t sq_tail; uint32_t cq_head; } *doorbells; @@ -292,7 +293,7 @@ static void nvme_kick(NVMeQueuePair *q) assert(!(q->sq.tail & 0xFF00)); /* Fence the write to submission queue entry before notifying the devi= ce. */ smp_wmb(); - *q->sq.doorbell =3D cpu_to_le32(q->sq.tail); + host_pci_mmio_write_32(q->sq.doorbell, q->sq.tail); q->inflight +=3D q->need_kick; q->need_kick =3D 0; } @@ -441,7 +442,7 @@ static bool nvme_process_completion(NVMeQueuePair *q) if (progress) { /* Notify the device so it can post more completions. */ smp_mb_release(); - *q->cq.doorbell =3D cpu_to_le32(q->cq.head); + host_pci_mmio_write_32(q->cq.doorbell, q->cq.head); nvme_wake_free_req_locked(q); } =20 @@ -460,7 +461,7 @@ static void nvme_process_completion_bh(void *opaque) * so notify the device that it has space to fill in more completions = now. */ smp_mb_release(); - *q->cq.doorbell =3D cpu_to_le32(q->cq.head); + host_pci_mmio_write_32(q->cq.doorbell, q->cq.head); nvme_wake_free_req_locked(q); =20 nvme_process_completion(q); @@ -749,9 +750,10 @@ static int nvme_init(BlockDriverState *bs, const char = *device, int namespace, int ret; uint64_t cap; uint32_t ver; + uint32_t cc; uint64_t timeout_ms; uint64_t deadline, now; - volatile NvmeBar *regs =3D NULL; + NvmeBar *regs =3D NULL; =20 qemu_co_mutex_init(&s->dma_map_lock); qemu_co_queue_init(&s->dma_flush_queue); @@ -779,7 +781,7 @@ static int nvme_init(BlockDriverState *bs, const char *= device, int namespace, /* Perform initialize sequence as described in NVMe spec "7.6.1 * Initialization". */ =20 - cap =3D le64_to_cpu(regs->cap); + cap =3D host_pci_mmio_read_64(®s->cap); trace_nvme_controller_capability_raw(cap); trace_nvme_controller_capability("Maximum Queue Entries Supported", 1 + NVME_CAP_MQES(cap)); @@ -805,16 +807,17 @@ static int nvme_init(BlockDriverState *bs, const char= *device, int namespace, bs->bl.request_alignment =3D s->page_size; timeout_ms =3D MIN(500 * NVME_CAP_TO(cap), 30000); =20 - ver =3D le32_to_cpu(regs->vs); + ver =3D host_pci_mmio_read_32(®s->vs); trace_nvme_controller_spec_version(extract32(ver, 16, 16), extract32(ver, 8, 8), extract32(ver, 0, 8)); =20 /* Reset device to get a clean state. */ - regs->cc =3D cpu_to_le32(le32_to_cpu(regs->cc) & 0xFE); + cc =3D host_pci_mmio_read_32(®s->cc); + host_pci_mmio_write_32(®s->cc, cc & 0xFE); /* Wait for CSTS.RDY =3D 0. */ deadline =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME) + timeout_ms * SCA= LE_MS; - while (NVME_CSTS_RDY(le32_to_cpu(regs->csts))) { + while (NVME_CSTS_RDY(host_pci_mmio_read_32(®s->csts))) { if (qemu_clock_get_ns(QEMU_CLOCK_REALTIME) > deadline) { error_setg(errp, "Timeout while waiting for device to reset (%" PRId64 " ms)", @@ -843,19 +846,21 @@ static int nvme_init(BlockDriverState *bs, const char= *device, int namespace, s->queues[INDEX_ADMIN] =3D q; s->queue_count =3D 1; QEMU_BUILD_BUG_ON((NVME_QUEUE_SIZE - 1) & 0xF000); - regs->aqa =3D cpu_to_le32(((NVME_QUEUE_SIZE - 1) << AQA_ACQS_SHIFT) | - ((NVME_QUEUE_SIZE - 1) << AQA_ASQS_SHIFT)); - regs->asq =3D cpu_to_le64(q->sq.iova); - regs->acq =3D cpu_to_le64(q->cq.iova); + host_pci_mmio_write_32(®s->aqa, + ((NVME_QUEUE_SIZE - 1) << AQA_ACQS_SHIFT) | + ((NVME_QUEUE_SIZE - 1) << AQA_ASQS_SHIFT)); + host_pci_mmio_write_64(®s->asq, q->sq.iova); + host_pci_mmio_write_64(®s->acq, q->cq.iova); =20 /* After setting up all control registers we can enable device now. */ - regs->cc =3D cpu_to_le32((ctz32(NVME_CQ_ENTRY_BYTES) << CC_IOCQES_SHIF= T) | + host_pci_mmio_write_32(®s->cc, + (ctz32(NVME_CQ_ENTRY_BYTES) << CC_IOCQES_SHIFT)= | (ctz32(NVME_SQ_ENTRY_BYTES) << CC_IOSQES_SHIFT)= | CC_EN_MASK); /* Wait for CSTS.RDY =3D 1. */ now =3D qemu_clock_get_ns(QEMU_CLOCK_REALTIME); deadline =3D now + timeout_ms * SCALE_MS; - while (!NVME_CSTS_RDY(le32_to_cpu(regs->csts))) { + while (!NVME_CSTS_RDY(host_pci_mmio_read_32(®s->csts))) { if (qemu_clock_get_ns(QEMU_CLOCK_REALTIME) > deadline) { error_setg(errp, "Timeout while waiting for device to start (%" PRId64 " ms)", --=20 2.43.0