From nobody Sat Nov 15 12:50:30 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=nongnu.org ARC-Seal: i=1; a=rsa-sha256; t=1752513310; cv=none; d=zohomail.com; s=zohoarc; b=Paf7MSCHPjvRJ/rUwY9PLyZO8dMMpMueUQwJP3yOFw2i06pxfEdC9hZQHybkpXb3JO+zK5fBtsEhWW6q6vmKaEaSaxH4Bwq3vuEk1H0dGKwvELioxo1F3cebIaeD+FOXtkpEANYLrdRqO1aQxgZrHCU2iAiDLU/ROpMX12FjvBU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1752513310; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=QM6lOrrGaQ8jFIl4uzzSMmGCHVP9RcXgCV7sFI3lM5Q=; b=RongfQSA+oifak8xlXio+twuONyfzgVlRFFBgtuHj7/pM9NtknxAHwOki0I9SN2X7zgU2K3NtXXvbQMk3IgZTeMXuTjjvEDPOBO9HPcg99ZmUFA/8O6Ji6xKNMfzZPYmXfkilju+kCgiIGqsJPztP1OgsemoiewrZwA4ImRIAPM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1752513310119509.48922544874733; Mon, 14 Jul 2025 10:15:10 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ubMlX-0001t4-QZ; Mon, 14 Jul 2025 13:15:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ubLcv-0005Yt-5X; Mon, 14 Jul 2025 12:02:05 -0400 Received: from [185.176.79.56] (helo=frasgout.his.huawei.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ubLct-0008PQ-0Q; Mon, 14 Jul 2025 12:02:00 -0400 Received: from mail.maildlp.com (unknown [172.18.186.31]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4bgn9G1MDrz6L5d4; Tue, 15 Jul 2025 00:00:50 +0800 (CST) Received: from frapeml500008.china.huawei.com (unknown [7.182.85.71]) by mail.maildlp.com (Postfix) with ESMTPS id 1C1BB1402FB; Tue, 15 Jul 2025 00:01:55 +0800 (CST) Received: from A2303104131.china.huawei.com (10.203.177.241) by frapeml500008.china.huawei.com (7.182.85.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.39; Mon, 14 Jul 2025 18:01:46 +0200 To: , CC: , , , , , , , , , , , , , , , Subject: [RFC PATCH v3 11/15] hw/pci/pci: Introduce optional get_msi_address_space() callback. Date: Mon, 14 Jul 2025 16:59:37 +0100 Message-ID: <20250714155941.22176-12-shameerali.kolothum.thodi@huawei.com> X-Mailer: git-send-email 2.12.0.windows.1 In-Reply-To: <20250714155941.22176-1-shameerali.kolothum.thodi@huawei.com> References: <20250714155941.22176-1-shameerali.kolothum.thodi@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.203.177.241] X-ClientProxiedBy: kwepems500002.china.huawei.com (7.221.188.17) To frapeml500008.china.huawei.com (7.182.85.71) X-Host-Lookup-Failed: Reverse DNS lookup failed for 185.176.79.56 (deferred) Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=185.176.79.56; envelope-from=shameerali.kolothum.thodi@huawei.com; helo=frasgout.his.huawei.com X-Spam_score_int: -33 X-Spam_score: -3.4 X-Spam_bar: --- X-Spam_report: (-3.4 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RDNS_NONE=0.793, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-to: Shameer Kolothum From: Shameer Kolothum via Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZM-MESSAGEID: 1752513312281116600 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" On ARM, when a device is behind an IOMMU, its MSI doorbell address is subject to translation by the IOMMU. This behavior affects vfio-pci passthrough devices assigned to guests using an accelerated SMMUv3. In this setup, we configure the host SMMUv3 in nested mode, where VFIO sets up the Stage-2 (S2) mappings for guest RAM, while the guest controls Stage-1 (S1). To allow VFIO to correctly configure S2 mappings, we currently return the system address space via the get_address_space() callback for vfio-pci devices. However, QEMU/KVM also uses this same callback path when resolving the address space for MSI doorbells: kvm_irqchip_add_msi_route() kvm_arch_fixup_msi_route() pci_device_iommu_address_space() This leads to problems when MSI doorbells need to be translated. To fix this, introduce an optional get_msi_address_space() callback. In the SMMUv3 accelerated case, this callback returns the IOMMU address space if the guest has set up S1 translations for the vfio-pci device. Otherwise, it returns the system address space. Suggested-by: Nicolin Chen Signed-off-by: Shameer Kolothum --- hw/arm/smmuv3-accel.c | 25 +++++++++++++++++++++++++ hw/pci/pci.c | 19 +++++++++++++++++++ include/hw/pci/pci.h | 16 ++++++++++++++++ target/arm/kvm.c | 2 +- 4 files changed, 61 insertions(+), 1 deletion(-) diff --git a/hw/arm/smmuv3-accel.c b/hw/arm/smmuv3-accel.c index f1584dd775..04c665ccf5 100644 --- a/hw/arm/smmuv3-accel.c +++ b/hw/arm/smmuv3-accel.c @@ -346,6 +346,30 @@ static void smmuv3_accel_unset_iommu_device(PCIBus *bu= s, void *opaque, } } =20 +static AddressSpace *smmuv3_accel_find_msi_as(PCIBus *bus, void *opaque, + int devfn) +{ + SMMUState *bs =3D opaque; + SMMUPciBus *sbus; + SMMUv3AccelDevice *accel_dev; + SMMUDevice *sdev; + + sbus =3D smmu_get_sbus(bs, bus); + accel_dev =3D smmuv3_accel_get_dev(bs, sbus, bus, devfn); + sdev =3D &accel_dev->sdev; + + /* + * If the assigned vfio-pci dev has S1 translation enabled by + * Guest, return IOMMU address space for MSI translation. + * Otherwise, return system address space. + */ + if (accel_dev->s1_hwpt) { + return &sdev->as; + } else { + return &accel_dev->as_sysmem; + } +} + static bool smmuv3_accel_pdev_allowed(PCIDevice *pdev, bool *vfio_pci) { =20 @@ -407,6 +431,7 @@ static const PCIIOMMUOps smmuv3_accel_ops =3D { .get_viommu_cap =3D smmuv3_accel_get_viommu_cap, .set_iommu_device =3D smmuv3_accel_set_iommu_device, .unset_iommu_device =3D smmuv3_accel_unset_iommu_device, + .get_msi_address_space =3D smmuv3_accel_find_msi_as, }; =20 void smmuv3_accel_init(SMMUv3State *s) diff --git a/hw/pci/pci.c b/hw/pci/pci.c index 13de0e2809..404aeb643d 100644 --- a/hw/pci/pci.c +++ b/hw/pci/pci.c @@ -2957,6 +2957,25 @@ AddressSpace *pci_device_iommu_address_space(PCIDevi= ce *dev) return &address_space_memory; } =20 +AddressSpace *pci_device_iommu_msi_address_space(PCIDevice *dev) +{ + PCIBus *bus; + PCIBus *iommu_bus; + int devfn; + + pci_device_get_iommu_bus_devfn(dev, &iommu_bus, &bus, &devfn); + if (iommu_bus) { + if (iommu_bus->iommu_ops->get_msi_address_space) { + return iommu_bus->iommu_ops->get_msi_address_space(bus, + iommu_bus->iommu_opaque, devfn); + } else { + return iommu_bus->iommu_ops->get_address_space(bus, + iommu_bus->iommu_opaque, devfn); + } + } + return &address_space_memory; +} + int pci_iommu_init_iotlb_notifier(PCIDevice *dev, IOMMUNotifier *n, IOMMUNotify fn, void *opaque) { diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h index d1d43e9fb9..55138c406e 100644 --- a/include/hw/pci/pci.h +++ b/include/hw/pci/pci.h @@ -639,12 +639,28 @@ typedef struct PCIIOMMUOps { uint32_t pasid, bool priv_req, bool exec_req, hwaddr addr, bool lpig, uint16_t prgi, bool is= _read, bool is_write); + /** + * @get_msi_address_space: get the address space for MSI doorbell addr= ess + * for devices + * + * Optional callback which returns a pointer to an #AddressSpace. This + * is required if MSI doorbell also gets translated through IOMMU(eg: = ARM) + * + * @bus: the #PCIBus being accessed. + * + * @opaque: the data passed to pci_setup_iommu(). + * + * @devfn: device and function number + */ + AddressSpace * (*get_msi_address_space)(PCIBus *bus, void *opaque, + int devfn); } PCIIOMMUOps; =20 AddressSpace *pci_device_iommu_address_space(PCIDevice *dev); bool pci_device_set_iommu_device(PCIDevice *dev, HostIOMMUDevice *hiod, Error **errp); void pci_device_unset_iommu_device(PCIDevice *dev); +AddressSpace *pci_device_iommu_msi_address_space(PCIDevice *dev); =20 /** * pci_device_get_viommu_cap: get vIOMMU capabilities. diff --git a/target/arm/kvm.c b/target/arm/kvm.c index 6672344855..c78d0d59bb 100644 --- a/target/arm/kvm.c +++ b/target/arm/kvm.c @@ -1535,7 +1535,7 @@ int kvm_arm_set_irq(int cpu, int irqtype, int irq, in= t level) int kvm_arch_fixup_msi_route(struct kvm_irq_routing_entry *route, uint64_t address, uint32_t data, PCIDevice *d= ev) { - AddressSpace *as =3D pci_device_iommu_address_space(dev); + AddressSpace *as =3D pci_device_iommu_msi_address_space(dev); hwaddr xlat, len, doorbell_gpa; MemoryRegionSection mrs; MemoryRegion *mr; --=20 2.34.1