From nobody Sat Nov 30 01:48:17 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1618388595; cv=none; d=zohomail.com; s=zohoarc; b=J91AUmMH08JAefjFOMrMwrDX60c7j3LdgbJHtqQYOhV8gDX6VeKi+t3xCqWv3voQv8cgor/m3zklMRVHfnbLDS8qgK2cUdtQBBaYreftcKO2aLpnUsHJ0jgIv++gF+E4u4X91VxNs2TfpBvmjW+EZ9asY5Uw560xKMZxiI4dyfQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1618388595; h=Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=f6X1ntobvTVs6a12PXKDxBmBWJQ9QqXIOY1jNZcVkjw=; b=Kp8L5Ortffo9NOzDPj3u8Ufd+YQvYHLNI5vq9QLA4Zr8ajs/od7vpAEicxlwOsTx/ONHmYuUFGJykwkdtD+AUkjyUUke77R4lYQQyrbIurEaZajpbh3ijgCcMOt17urusD1+ELzmYqJuquuGmyqTA/jZ1kfjzHWatm98i51lqhA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1618388595732356.3004001326019; Wed, 14 Apr 2021 01:23:15 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.110353.210613 (Exim 4.92) (envelope-from ) id 1lWanW-0003c6-Ul; Wed, 14 Apr 2021 08:22:54 +0000 Received: by outflank-mailman (output) from mailman id 110353.210613; Wed, 14 Apr 2021 08:22:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lWanW-0003bz-Qi; Wed, 14 Apr 2021 08:22:54 +0000 Received: by outflank-mailman (input) for mailman id 110353; Wed, 14 Apr 2021 08:22:53 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lWanV-0003bu-Me for xen-devel@lists.xenproject.org; Wed, 14 Apr 2021 08:22:53 +0000 Received: from mga02.intel.com (unknown [134.134.136.20]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id a9bee840-1f62-4592-8592-b75c6df97c02; Wed, 14 Apr 2021 08:22:50 +0000 (UTC) Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 01:22:49 -0700 Received: from unknown (HELO hyperv-sh4.sh.intel.com) ([10.239.48.33]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Apr 2021 01:22:46 -0700 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: a9bee840-1f62-4592-8592-b75c6df97c02 IronPort-SDR: pvkxHMBbtyE44qGu8d6Nhbc0mArhfvwKoXxtl9X1sd45oADEFoW3yhZTgsB5bTONii4SY1pPNZ V67DAbxloVrQ== X-IronPort-AV: E=McAfee;i="6200,9189,9953"; a="181715370" X-IronPort-AV: E=Sophos;i="5.82,221,1613462400"; d="scan'208";a="181715370" IronPort-SDR: dI9AmB6jIeR57Bskm5ThJiCmxAseeWAVFROpB20vOGLjevnoTZoWNckk1rcXLa0T2DQP5TGnwC xaUxAwHaArLA== X-IronPort-AV: E=Sophos;i="5.82,221,1613462400"; d="scan'208";a="418232839" From: Chao Gao To: xen-devel@lists.xenproject.org Cc: Chao Gao , Andrew Cooper , George Dunlap , Ian Jackson , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu , Kevin Tian Subject: [RFC PATCH] VT-d: Don't assume register-based invalidation is always supported Date: Wed, 14 Apr 2021 08:55:26 +0800 Message-Id: <20210414005526.36760-1-chao.gao@intel.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" According to Intel VT-d SPEC rev3.3 Section 6.5, Register-based Invalidation isn't supported by Intel VT-d version 6 and beyond. This hardware change impacts following two scenarios: admin can disable queued invalidation via 'qinval' cmdline and use register-based interface; VT-d switches to register-based invalidation when queued invalidation needs to be disabled, for example, during disabling x2apic or during system suspension. To deal with this hardware change, if register-based invalidation isn't supported, queued invalidation cannot be disabled through Xen cmdline; and if queued invalidation has to be disabled temporarily in some scenarios, VT-d won't switch to register-based interface but use some dummy functions to catch errors in case there is any invalidation request issued when queued invalidation is disabled. Signed-off-by: Chao Gao --- I only tested Xen boot with qinval/no-qinval. I also want to do system suspension and resumption to see if any unexpected error. But I don't know how to trigger them. Any recommendation? --- docs/misc/xen-command-line.pandoc | 4 ++- xen/drivers/passthrough/vtd/iommu.c | 40 +++++++++++++++++++++++++--- xen/drivers/passthrough/vtd/iommu.h | 7 +++++ xen/drivers/passthrough/vtd/qinval.c | 33 +++++++++++++++++++++-- 4 files changed, 77 insertions(+), 7 deletions(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line= .pandoc index deef6d0b4c..4ff4a87844 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -1442,7 +1442,9 @@ The following options are specific to Intel VT-d hard= ware: * The `qinval` boolean controls the Queued Invalidation sub-feature, and= is active by default on compatible hardware. Queued Invalidation is a feature in second-generation IOMMUs and is a functional prerequisite f= or - Interrupt Remapping. + Interrupt Remapping. Note that Xen disregards this setting for Intel V= T-d + version 6 and greater as Registered-Based Invalidation isn't supported + by them. =20 * The `igfx` boolean is active by default, and controls whether the IOMM= U in front of an Intel Graphics Device is enabled or not. diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/= vtd/iommu.c index 6428c8fe3e..e738d04543 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -1193,6 +1193,14 @@ int __init iommu_alloc(struct acpi_drhd_unit *drhd) =20 iommu->cap =3D dmar_readq(iommu->reg, DMAR_CAP_REG); iommu->ecap =3D dmar_readq(iommu->reg, DMAR_ECAP_REG); + iommu->version =3D dmar_readl(iommu->reg, DMAR_VER_REG); + + if ( !iommu_qinval && !has_register_based_invalidation(iommu) ) + { + printk(XENLOG_WARNING VTDPREFIX "IOMMU %d: cannot disable Queued I= nvalidation.\n", + iommu->index); + iommu_qinval =3D true; + } =20 if ( iommu_verbose ) { @@ -2231,6 +2239,8 @@ static int __init vtd_setup(void) struct acpi_drhd_unit *drhd; struct vtd_iommu *iommu; int ret; + bool queued_inval_supported =3D true; + bool reg_inval_supported =3D true; =20 if ( list_empty(&acpi_drhd_units) ) { @@ -2252,8 +2262,8 @@ static int __init vtd_setup(void) } =20 /* We enable the following features only if they are supported by all = VT-d - * engines: Snoop Control, DMA passthrough, Queued Invalidation, Inter= rupt - * Remapping, and Posted Interrupt + * engines: Snoop Control, DMA passthrough, Register-based Invalidatio= n, + * Queued Invalidation, Interrupt Remapping, and Posted Interrupt. */ for_each_drhd_unit ( drhd ) { @@ -2272,8 +2282,11 @@ static int __init vtd_setup(void) if ( iommu_hwdom_passthrough && !ecap_pass_thru(iommu->ecap) ) iommu_hwdom_passthrough =3D false; =20 - if ( iommu_qinval && !ecap_queued_inval(iommu->ecap) ) - iommu_qinval =3D 0; + if ( reg_inval_supported && !has_register_based_invalidation(iommu= ) ) + reg_inval_supported =3D false; + + if ( queued_inval_supported && !ecap_queued_inval(iommu->ecap) ) + queued_inval_supported =3D false; =20 if ( iommu_intremap && !ecap_intr_remap(iommu->ecap) ) iommu_intremap =3D iommu_intremap_off; @@ -2301,6 +2314,25 @@ static int __init vtd_setup(void) =20 softirq_tasklet_init(&vtd_fault_tasklet, do_iommu_page_fault, NULL); =20 + if ( !queued_inval_supported && !reg_inval_supported ) + { + dprintk(XENLOG_ERR VTDPREFIX, "No available invalidation interface= .\n"); + ret =3D -ENODEV; + goto error; + } + + /* + * We cannot have !iommu_qinval && !reg_inval_supported here since + * iommu_qinval is set in iommu_alloc() if any iommu doesn't support + * Register-based Invalidation. + */ + if ( iommu_qinval && !queued_inval_supported ) + { + dprintk(XENLOG_WARNING VTDPREFIX, "Disable Queued Invalidation as " + "it isn't supported.\n"); + iommu_qinval =3D false; + } + if ( !iommu_qinval && iommu_intremap ) { iommu_intremap =3D iommu_intremap_off; diff --git a/xen/drivers/passthrough/vtd/iommu.h b/xen/drivers/passthrough/= vtd/iommu.h index 216791b3d6..644224051a 100644 --- a/xen/drivers/passthrough/vtd/iommu.h +++ b/xen/drivers/passthrough/vtd/iommu.h @@ -540,6 +540,7 @@ struct vtd_iommu { struct list_head ats_devices; unsigned long *domid_bitmap; /* domain id bitmap */ u16 *domid_map; /* domain id mapping array */ + u32 version; }; =20 #define INTEL_IOMMU_DEBUG(fmt, args...) \ @@ -549,4 +550,10 @@ struct vtd_iommu { dprintk(XENLOG_WARNING VTDPREFIX, fmt, ## args); \ } while(0) =20 +/* Register-based invalidation isn't supported by VT-d version 6 and beyon= d. */ +static inline bool has_register_based_invalidation(struct vtd_iommu *iommu) +{ + return VER_MAJOR(iommu->version) < 6; +} + #endif diff --git a/xen/drivers/passthrough/vtd/qinval.c b/xen/drivers/passthrough= /vtd/qinval.c index 764ef9f0fc..1e90f50ff8 100644 --- a/xen/drivers/passthrough/vtd/qinval.c +++ b/xen/drivers/passthrough/vtd/qinval.c @@ -442,6 +442,23 @@ int enable_qinval(struct vtd_iommu *iommu) return 0; } =20 +static int vtd_flush_context_noop(struct vtd_iommu *iommu, uint16_t did, + uint16_t source_id, uint8_t function_mas= k, + uint64_t type, bool flush_non_present_en= try) +{ + dprintk(XENLOG_ERR VTDPREFIX, "IOMMU: Cannot flush CONTEXT.\n"); + return -EIO; +} + +static int vtd_flush_iotlb_noop(struct vtd_iommu *iommu, uint16_t did, + uint64_t addr, unsigned int size_order, + uint64_t type, bool flush_non_present_entr= y, + bool flush_dev_iotlb) +{ + dprintk(XENLOG_ERR VTDPREFIX, "IOMMU: Cannot flush IOTLB.\n"); + return -EIO; +} + void disable_qinval(struct vtd_iommu *iommu) { u32 sts; @@ -463,6 +480,18 @@ void disable_qinval(struct vtd_iommu *iommu) out: spin_unlock_irqrestore(&iommu->register_lock, flags); =20 - iommu->flush.context =3D vtd_flush_context_reg; - iommu->flush.iotlb =3D vtd_flush_iotlb_reg; + /* + * Assign callbacks to noop to catch errors if register-based invalida= tion + * isn't supported. + */ + if ( has_register_based_invalidation(iommu) ) + { + iommu->flush.context =3D vtd_flush_context_reg; + iommu->flush.iotlb =3D vtd_flush_iotlb_reg; + } + else + { + iommu->flush.context =3D vtd_flush_context_noop; + iommu->flush.iotlb =3D vtd_flush_iotlb_noop; + } } --=20 2.25.1