From nobody Sun Nov 2 12:40:38 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass header.i=teddy.astie@vates.tech; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=vates.tech ARC-Seal: i=1; a=rsa-sha256; t=1739787540; cv=none; d=zohomail.com; s=zohoarc; b=kPIiXID5DxPbut/MdzGALbiFvtWQiu939McEyx3QFrsVEeeafm9Jo3AX5vDw8voLaQSjL+CCizXcZj9lwJ9H1jZ/TcCG9mZMmdtSjqUlzHJhInMkfO+tOkCTLLMIenSkOVtWshr/dHBnmLMZRfDjeLVFrSzDftJwYO63LogHws8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1739787540; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=bzfNvWE5kbPAbUkvS+oqP/fh38Hyfkhqt9wTC3oGwMA=; b=KOFIH6O68m2WXGN99CZLOurGb/7Zi226p3HEMhxauXN1j95UWHvafeJVtunVppIx2Zgno4GtUT2KL3r+m0bmLm/Y0SQNw1XVhlFISvi6FCsMyETUqVsRqoYrJcwFZctab+/stjQRJKlZ/M9cpTTdJKaNy3dNF0aMKko2sDIU7Io= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=teddy.astie@vates.tech; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1739787540349372.3294304907166; Mon, 17 Feb 2025 02:19:00 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.890010.1299106 (Exim 4.92) (envelope-from ) id 1tjyCt-0001u7-2D; Mon, 17 Feb 2025 10:18:31 +0000 Received: by outflank-mailman (output) from mailman id 890010.1299106; Mon, 17 Feb 2025 10:18:30 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tjyCs-0001qL-GC; Mon, 17 Feb 2025 10:18:30 +0000 Received: by outflank-mailman (input) for mailman id 890010; Mon, 17 Feb 2025 10:18:29 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1tjyCr-0008Nl-9f for xen-devel@lists.xenproject.org; Mon, 17 Feb 2025 10:18:29 +0000 Received: from mail13.wdc04.mandrillapp.com (mail13.wdc04.mandrillapp.com [205.201.139.13]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 852e5ae5-ed18-11ef-9aa6-95dc52dad729; Mon, 17 Feb 2025 11:18:24 +0100 (CET) Received: from pmta16.mandrill.prod.suw01.rsglab.com (localhost [127.0.0.1]) by mail13.wdc04.mandrillapp.com (Mailchimp) with ESMTP id 4YxJWz6N4FzNCdCf0 for ; Mon, 17 Feb 2025 10:18:23 +0000 (GMT) Received: from [37.26.189.201] by mandrillapp.com id a3bc10c83c634101bf8dbbdbe1fbb292; Mon, 17 Feb 2025 10:18:23 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 852e5ae5-ed18-11ef-9aa6-95dc52dad729 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mandrillapp.com; s=mte1; t=1739787503; x=1740057503; bh=bzfNvWE5kbPAbUkvS+oqP/fh38Hyfkhqt9wTC3oGwMA=; h=From:Subject:To:Cc:Message-Id:In-Reply-To:References:Feedback-ID: Date:MIME-Version:Content-Type:Content-Transfer-Encoding:CC:Date: Subject:From; b=P4pJcXtqIZb8rjypb3pSidHMF8UD8INqTrYrMJSrQl9wIUyuDlj75Yaz9fPRkOeV7 AtdSBnWYyHD4kuKtNDohL/MyaOd4OFwdSBCy99Q4gYzOfg2/FDyJ7OFda7hLh3oeMr 992fKEERMmk8s+GKgCaAOVd3KdE4NeBBzDNfUokdksPm/1fDwNhLC5X4pSYx+zbI84 qIx49jS4ky5CIXAZwbTE0IT/Tu91YBd6Ty9Oz2Q3r0AuqZp7ytPlqJX4VC959h5oF5 55RHP6O3GiAVBFASY6Lqu9NcLqnuxPt3TaSS1w5ud0uWyAx2ma/8InNOeGMTewFJER zLHJfprVjkrtA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vates.tech; s=mte1; t=1739787503; x=1740048003; i=teddy.astie@vates.tech; bh=bzfNvWE5kbPAbUkvS+oqP/fh38Hyfkhqt9wTC3oGwMA=; h=From:Subject:To:Cc:Message-Id:In-Reply-To:References:Feedback-ID: Date:MIME-Version:Content-Type:Content-Transfer-Encoding:CC:Date: Subject:From; b=rJatvqD04TbD9W1Ss7x6UBVWoVwBH8KBw0VueBTnpOTaWYd6OeyADqBs/3BtLd72i /f+yrxBFsGsIndLYAoPKMZCF429PVAvn6oSKU5I1MnkcnvWzKrb2E1iyQXHSEebmpx JSABY4RvZXoulbZcGxO/NnNlisBBpG2oRsdFEFLPii32MoAUG+QuO+kk0r33Kzim01 Z9Fn9WwJ8lKNI2HKRADAj7Q8wiCAvUTuTlvKbnAIHtMP26QvUyt7KZZrTeBPZX+3g9 gtNwSruJwgNOjCQKxTh9usHbkNVi/HJ6AaFp1WISikPaiiWYttAXEjMN1g6QB34d69 aZF8JqmEyt7MA== From: "Teddy Astie" Subject: =?utf-8?Q?[XEN=20RFC=20PATCH=20v6=2004/11]=20iommu:=20Move=20IOMMU=20domain=20related=20structures=20to=20(arch=5F)iommu=5Fcontext?= X-Mailer: git-send-email 2.47.2 X-Bm-Disclaimer: Yes X-Bm-Milter-Handled: 4ffbd6c1-ee69-4e1b-aabd-f977039bd3e2 X-Bm-Transport-Timestamp: 1739787500484 To: xen-devel@lists.xenproject.org Cc: "Teddy Astie" , "Stefano Stabellini" , "Julien Grall" , "Bertrand Marquis" , "Michal Orzel" , "Volodymyr Babchuk" , "Shawn Anastasio" , "Jan Beulich" , "Andrew Cooper" , "=?utf-8?Q?Roger=20Pau=20Monn=C3=A9?=" , "Lukasz Hawrylko" , "Daniel P. Smith" , "=?utf-8?Q?Mateusz=20M=C3=B3wka?=" Message-Id: <0cd25b4114458dc957c0fb818d01162dfab9548b.1739785339.git.teddy.astie@vates.tech> In-Reply-To: References: X-Native-Encoded: 1 X-Report-Abuse: =?UTF-8?Q?Please=20forward=20a=20copy=20of=20this=20message,=20including=20all=20headers,=20to=20abuse@mandrill.com.=20You=20can=20also=20report=20abuse=20here:=20https://mandrillapp.com/contact/abuse=3Fid=3D30504962.a3bc10c83c634101bf8dbbdbe1fbb292?= X-Mandrill-User: md_30504962 Feedback-ID: 30504962:30504962.20250217:md Date: Mon, 17 Feb 2025 10:18:23 +0000 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity teddy.astie@vates.tech) (identity @mandrillapp.com) X-ZM-MESSAGEID: 1739787540878019000 Content-Type: text/plain; charset="utf-8" Preparatory work for IOMMU redesign. Introduce a new structure (arch_)iommu_context that will hold all per-IOMMU context related informations for the IOMMU drivers. Signed-off-by: Teddy Astie --- xen/arch/arm/include/asm/iommu.h | 4 + xen/arch/ppc/include/asm/iommu.h | 3 + xen/arch/x86/domain.c | 4 +- xen/arch/x86/include/asm/iommu.h | 50 +++-- xen/arch/x86/tboot.c | 3 +- xen/drivers/passthrough/amd/iommu.h | 5 +- xen/drivers/passthrough/amd/iommu_init.c | 8 +- xen/drivers/passthrough/amd/iommu_map.c | 102 +++++----- xen/drivers/passthrough/amd/pci_amd_iommu.c | 81 ++++---- xen/drivers/passthrough/iommu.c | 6 + xen/drivers/passthrough/vtd/extern.h | 4 +- xen/drivers/passthrough/vtd/iommu.c | 208 +++++++++++--------- xen/drivers/passthrough/vtd/quirks.c | 3 +- xen/drivers/passthrough/x86/iommu.c | 62 +++--- xen/include/xen/iommu.h | 10 + 15 files changed, 320 insertions(+), 233 deletions(-) diff --git a/xen/arch/arm/include/asm/iommu.h b/xen/arch/arm/include/asm/io= mmu.h index d57bd8a38c..5ca56cc663 100644 --- a/xen/arch/arm/include/asm/iommu.h +++ b/xen/arch/arm/include/asm/iommu.h @@ -20,6 +20,10 @@ struct arch_iommu void *priv; }; =20 +struct arch_iommu_context +{ +}; + const struct iommu_ops *iommu_get_ops(void); void iommu_set_ops(const struct iommu_ops *ops); =20 diff --git a/xen/arch/ppc/include/asm/iommu.h b/xen/arch/ppc/include/asm/io= mmu.h index 024ead3473..8367505de2 100644 --- a/xen/arch/ppc/include/asm/iommu.h +++ b/xen/arch/ppc/include/asm/iommu.h @@ -5,4 +5,7 @@ struct arch_iommu { }; =20 +struct arch_iommu_context { +}; + #endif /* __ASM_PPC_IOMMU_H__ */ diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c index 48bf9625e2..26729c879c 100644 --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -678,7 +678,7 @@ int arch_sanitise_domain_config(struct xen_domctl_creat= edomain *config) if ( nested_virt && !hvm_nested_virt_supported() ) { dprintk(XENLOG_INFO, "Nested virt requested but not available\n"); - return -EINVAL; =20 + return -EINVAL; } =20 if ( nested_virt && !hap ) @@ -2392,7 +2392,7 @@ int domain_relinquish_resources(struct domain *d) =20 PROGRESS(iommu_pagetables): =20 - ret =3D iommu_free_pgtables(d); + ret =3D iommu_free_pgtables(d, iommu_default_context(d)); if ( ret ) return ret; =20 diff --git a/xen/arch/x86/include/asm/iommu.h b/xen/arch/x86/include/asm/io= mmu.h index 8dc464fbd3..94513ba9dc 100644 --- a/xen/arch/x86/include/asm/iommu.h +++ b/xen/arch/x86/include/asm/iommu.h @@ -31,22 +31,21 @@ typedef uint64_t daddr_t; #define dfn_to_daddr(dfn) __dfn_to_daddr(dfn_x(dfn)) #define daddr_to_dfn(daddr) _dfn(__daddr_to_dfn(daddr)) =20 -struct arch_iommu -{ - spinlock_t mapping_lock; /* io page table lock */ - struct { - struct page_list_head list; - spinlock_t lock; - } pgtables; +struct iommu_context; =20 +struct arch_iommu_context +{ + struct page_list_head pgtables; struct list_head identity_maps; =20 + + spinlock_t mapping_lock; /* io page table lock */ + union { /* Intel VT-d */ struct { uint64_t pgd_maddr; /* io page directory machine address */ - unsigned int agaw; /* adjusted guest address width, 0 is level= 2 30-bit */ - unsigned long *iommu_bitmap; /* bitmap of iommu(s) that the do= main uses */ + unsigned long *iommu_bitmap; /* bitmap of iommu(s) that the co= ntext uses */ } vtd; /* AMD IOMMU */ struct { @@ -56,6 +55,24 @@ struct arch_iommu }; }; =20 +struct arch_iommu +{ + /* Queue for freeing pages */ + struct page_list_head free_queue; + + union { + /* Intel VT-d */ + struct { + unsigned int agaw; /* adjusted guest address width, 0 is level= 2 30-bit */ + } vtd; + /* AMD IOMMU */ + struct { + unsigned int paging_mode; + struct guest_iommu *g_iommu; + }; + }; +}; + extern struct iommu_ops iommu_ops; =20 # include @@ -109,10 +126,10 @@ static inline void iommu_disable_x2apic(void) iommu_vcall(&iommu_ops, disable_x2apic); } =20 -int iommu_identity_mapping(struct domain *d, p2m_access_t p2ma, - paddr_t base, paddr_t end, +int iommu_identity_mapping(struct domain *d, struct iommu_context *ctx, + p2m_access_t p2ma, paddr_t base, paddr_t end, unsigned int flag); -void iommu_identity_map_teardown(struct domain *d); +void iommu_identity_map_teardown(struct domain *d, struct iommu_context *c= tx); =20 extern bool untrusted_msi; =20 @@ -128,14 +145,19 @@ unsigned long *iommu_init_domid(domid_t reserve); domid_t iommu_alloc_domid(unsigned long *map); void iommu_free_domid(domid_t domid, unsigned long *map); =20 -int __must_check iommu_free_pgtables(struct domain *d); +int __must_check iommu_free_pgtables(struct domain *d, struct iommu_contex= t *ctx); struct domain_iommu; struct page_info *__must_check iommu_alloc_pgtable(struct domain_iommu *hd, + struct iommu_context *c= tx, uint64_t contig_mask); -void iommu_queue_free_pgtable(struct domain_iommu *hd, struct page_info *p= g); +void iommu_queue_free_pgtable(struct domain *d, struct iommu_context *ctx, + struct page_info *pg); =20 /* Check [start, end] unity map range for correctness. */ bool iommu_unity_region_ok(const char *prefix, mfn_t start, mfn_t end); +int arch_iommu_context_init(struct domain *d, struct iommu_context *ctx, u= 32 flags); +int arch_iommu_context_teardown(struct domain *d, struct iommu_context *ct= x, u32 flags); +int arch_iommu_flush_free_queue(struct domain *d); =20 #endif /* !__ARCH_X86_IOMMU_H__ */ /* diff --git a/xen/arch/x86/tboot.c b/xen/arch/x86/tboot.c index d5db60d335..0a5aee8b92 100644 --- a/xen/arch/x86/tboot.c +++ b/xen/arch/x86/tboot.c @@ -220,7 +220,8 @@ static void tboot_gen_domain_integrity(const uint8_t ke= y[TB_KEY_SIZE], { const struct domain_iommu *dio =3D dom_iommu(d); =20 - update_iommu_mac(&ctx, dio->arch.vtd.pgd_maddr, + update_iommu_mac(&ctx, + iommu_default_context(d)->arch.vtd.pgd_maddr, agaw_to_level(dio->arch.vtd.agaw)); } } diff --git a/xen/drivers/passthrough/amd/iommu.h b/xen/drivers/passthrough/= amd/iommu.h index c32e9e9a16..6095bc6a21 100644 --- a/xen/drivers/passthrough/amd/iommu.h +++ b/xen/drivers/passthrough/amd/iommu.h @@ -26,6 +26,7 @@ #include #include #include +#include =20 #include #include @@ -199,10 +200,10 @@ int __must_check cf_check amd_iommu_unmap_page( struct domain *d, dfn_t dfn, unsigned int order, unsigned int *flush_flags); int __must_check amd_iommu_alloc_root(struct domain *d); -int amd_iommu_reserve_domain_unity_map(struct domain *d, +int amd_iommu_reserve_domain_unity_map(struct domain *d, struct iommu_cont= ext *ctx, const struct ivrs_unity_map *map, unsigned int flag); -int amd_iommu_reserve_domain_unity_unmap(struct domain *d, +int amd_iommu_reserve_domain_unity_unmap(struct domain *d, struct iommu_co= ntext *ctx, const struct ivrs_unity_map *map); int cf_check amd_iommu_get_reserved_device_memory( iommu_grdm_t *func, void *ctxt); diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthr= ough/amd/iommu_init.c index 3023625020..41e241ccc8 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -604,7 +604,6 @@ static void iommu_check_event_log(struct amd_iommu *iom= mu) sizeof(event_entry_t), parse_event_log_entry); =20 spin_lock_irqsave(&iommu->lock, flags); - =20 /* Check event overflow. */ entry =3D readl(iommu->mmio_base + IOMMU_STATUS_MMIO_OFFSET); if ( entry & IOMMU_STATUS_EVENT_LOG_OVERFLOW ) @@ -660,9 +659,8 @@ static void iommu_check_ppr_log(struct amd_iommu *iommu) =20 iommu_read_log(iommu, &iommu->ppr_log, sizeof(ppr_entry_t), parse_ppr_log_entry); - =20 - spin_lock_irqsave(&iommu->lock, flags); =20 + spin_lock_irqsave(&iommu->lock, flags); /* Check event overflow. */ entry =3D readl(iommu->mmio_base + IOMMU_STATUS_MMIO_OFFSET); if ( entry & IOMMU_STATUS_PPR_LOG_OVERFLOW ) @@ -1545,7 +1543,7 @@ static void invalidate_all_domain_pages(void) static int cf_check _invalidate_all_devices( u16 seg, struct ivrs_mappings *ivrs_mappings) { - unsigned int bdf;=20 + unsigned int bdf; u16 req_id; struct amd_iommu *iommu; =20 @@ -1595,7 +1593,7 @@ void cf_check amd_iommu_resume(void) for_each_amd_iommu ( iommu ) { /* - * To make sure that iommus have not been touched=20 + * To make sure that iommus have not been touched * before re-enablement */ disable_iommu(iommu); diff --git a/xen/drivers/passthrough/amd/iommu_map.c b/xen/drivers/passthro= ugh/amd/iommu_map.c index dde393645a..7514384789 100644 --- a/xen/drivers/passthrough/amd/iommu_map.c +++ b/xen/drivers/passthrough/amd/iommu_map.c @@ -18,6 +18,7 @@ */ =20 #include +#include =20 #include "iommu.h" =20 @@ -264,9 +265,9 @@ void __init iommu_dte_add_device_entry(struct amd_iommu= _dte *dte, * {Re, un}mapping super page frames causes re-allocation of io * page tables. */ -static int iommu_pde_from_dfn(struct domain *d, unsigned long dfn, - unsigned int target, unsigned long *pt_mfn, - unsigned int *flush_flags, bool map) +static int iommu_pde_from_dfn(struct domain *d, struct iommu_context *ctx, + unsigned long dfn, unsigned int target, + unsigned long *pt_mfn, unsigned int *flush_f= lags, bool map) { union amd_iommu_pte *pde, *next_table_vaddr; unsigned long next_table_mfn; @@ -274,8 +275,8 @@ static int iommu_pde_from_dfn(struct domain *d, unsigne= d long dfn, struct page_info *table; struct domain_iommu *hd =3D dom_iommu(d); =20 - table =3D hd->arch.amd.root_table; - level =3D hd->arch.amd.paging_mode; + table =3D ctx->arch.amd.root_table; + level =3D ctx->arch.amd.paging_mode; =20 if ( !table || target < 1 || level < target || level > 6 ) { @@ -311,7 +312,7 @@ static int iommu_pde_from_dfn(struct domain *d, unsigne= d long dfn, mfn =3D next_table_mfn; =20 /* allocate lower level page table */ - table =3D iommu_alloc_pgtable(hd, IOMMU_PTE_CONTIG_MASK); + table =3D iommu_alloc_pgtable(hd, ctx, IOMMU_PTE_CONTIG_MASK); if ( table =3D=3D NULL ) { AMD_IOMMU_ERROR("cannot allocate I/O page table\n"); @@ -346,7 +347,7 @@ static int iommu_pde_from_dfn(struct domain *d, unsigne= d long dfn, =20 if ( next_table_mfn =3D=3D 0 ) { - table =3D iommu_alloc_pgtable(hd, IOMMU_PTE_CONTIG_MASK); + table =3D iommu_alloc_pgtable(hd, ctx, IOMMU_PTE_CONTIG_MA= SK); if ( table =3D=3D NULL ) { AMD_IOMMU_ERROR("cannot allocate I/O page table\n"); @@ -376,7 +377,8 @@ static int iommu_pde_from_dfn(struct domain *d, unsigne= d long dfn, return 0; } =20 -static void queue_free_pt(struct domain_iommu *hd, mfn_t mfn, unsigned int= level) +static void queue_free_pt(struct domain *d, struct iommu_context *ctx, mfn= _t mfn, + unsigned int level) { if ( level > 1 ) { @@ -387,13 +389,13 @@ static void queue_free_pt(struct domain_iommu *hd, mf= n_t mfn, unsigned int level if ( pt[i].pr && pt[i].next_level ) { ASSERT(pt[i].next_level < level); - queue_free_pt(hd, _mfn(pt[i].mfn), pt[i].next_level); + queue_free_pt(d, ctx, _mfn(pt[i].mfn), pt[i].next_level); } =20 unmap_domain_page(pt); } =20 - iommu_queue_free_pgtable(hd, mfn_to_page(mfn)); + iommu_queue_free_pgtable(d, ctx, mfn_to_page(mfn)); } =20 int cf_check amd_iommu_map_page( @@ -401,6 +403,7 @@ int cf_check amd_iommu_map_page( unsigned int *flush_flags) { struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); unsigned int level =3D (IOMMUF_order(flags) / PTE_PER_TABLE_SHIFT) + 1; bool contig; int rc; @@ -410,7 +413,7 @@ int cf_check amd_iommu_map_page( ASSERT((hd->platform_ops->page_sizes >> IOMMUF_order(flags)) & PAGE_SIZE_4K); =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); =20 /* * IOMMU mapping request can be safely ignored when the domain is dyin= g. @@ -420,24 +423,24 @@ int cf_check amd_iommu_map_page( */ if ( d->is_dying ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); return 0; } =20 rc =3D amd_iommu_alloc_root(d); if ( rc ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); AMD_IOMMU_ERROR("root table alloc failed, dfn =3D %"PRI_dfn"\n", dfn_x(dfn)); domain_crash(d); return rc; } =20 - if ( iommu_pde_from_dfn(d, dfn_x(dfn), level, &pt_mfn, flush_flags, tr= ue) || + if ( iommu_pde_from_dfn(d, ctx, dfn_x(dfn), level, &pt_mfn, flush_flag= s, true) || !pt_mfn ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); AMD_IOMMU_ERROR("invalid IO pagetable entry dfn =3D %"PRI_dfn"\n", dfn_x(dfn)); domain_crash(d); @@ -449,12 +452,12 @@ int cf_check amd_iommu_map_page( flags & IOMMUF_writable, flags & IOMMUF_readable, &contig); =20 - while ( unlikely(contig) && ++level < hd->arch.amd.paging_mode ) + while ( unlikely(contig) && ++level < ctx->arch.amd.paging_mode ) { struct page_info *pg =3D mfn_to_page(_mfn(pt_mfn)); unsigned long next_mfn; =20 - if ( iommu_pde_from_dfn(d, dfn_x(dfn), level, &pt_mfn, flush_flags, + if ( iommu_pde_from_dfn(d, ctx, dfn_x(dfn), level, &pt_mfn, flush_= flags, false) ) BUG(); BUG_ON(!pt_mfn); @@ -464,11 +467,11 @@ int cf_check amd_iommu_map_page( flags & IOMMUF_writable, flags & IOMMUF_readable, &contig); *flush_flags |=3D IOMMU_FLUSHF_modified | IOMMU_FLUSHF_all; - iommu_queue_free_pgtable(hd, pg); + iommu_queue_free_pgtable(d, ctx, pg); perfc_incr(iommu_pt_coalesces); } =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); =20 *flush_flags |=3D IOMMU_FLUSHF_added; if ( old.pr ) @@ -476,7 +479,7 @@ int cf_check amd_iommu_map_page( *flush_flags |=3D IOMMU_FLUSHF_modified; =20 if ( IOMMUF_order(flags) && old.next_level ) - queue_free_pt(hd, _mfn(old.mfn), old.next_level); + queue_free_pt(d, ctx, _mfn(old.mfn), old.next_level); } =20 return 0; @@ -487,6 +490,7 @@ int cf_check amd_iommu_unmap_page( { unsigned long pt_mfn =3D 0; struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); unsigned int level =3D (order / PTE_PER_TABLE_SHIFT) + 1; union amd_iommu_pte old =3D {}; =20 @@ -496,17 +500,17 @@ int cf_check amd_iommu_unmap_page( */ ASSERT((hd->platform_ops->page_sizes >> order) & PAGE_SIZE_4K); =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); =20 - if ( !hd->arch.amd.root_table ) + if ( !ctx->arch.amd.root_table ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); return 0; } =20 - if ( iommu_pde_from_dfn(d, dfn_x(dfn), level, &pt_mfn, flush_flags, fa= lse) ) + if ( iommu_pde_from_dfn(d, ctx, dfn_x(dfn), level, &pt_mfn, flush_flag= s, false) ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); AMD_IOMMU_ERROR("invalid IO pagetable entry dfn =3D %"PRI_dfn"\n", dfn_x(dfn)); domain_crash(d); @@ -520,30 +524,30 @@ int cf_check amd_iommu_unmap_page( /* Mark PTE as 'page not present'. */ old =3D clear_iommu_pte_present(pt_mfn, dfn_x(dfn), level, &free); =20 - while ( unlikely(free) && ++level < hd->arch.amd.paging_mode ) + while ( unlikely(free) && ++level < ctx->arch.amd.paging_mode ) { struct page_info *pg =3D mfn_to_page(_mfn(pt_mfn)); =20 - if ( iommu_pde_from_dfn(d, dfn_x(dfn), level, &pt_mfn, + if ( iommu_pde_from_dfn(d, ctx, dfn_x(dfn), level, &pt_mfn, flush_flags, false) ) BUG(); BUG_ON(!pt_mfn); =20 clear_iommu_pte_present(pt_mfn, dfn_x(dfn), level, &free); *flush_flags |=3D IOMMU_FLUSHF_all; - iommu_queue_free_pgtable(hd, pg); + iommu_queue_free_pgtable(d, ctx, pg); perfc_incr(iommu_pt_coalesces); } } =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); =20 if ( old.pr ) { *flush_flags |=3D IOMMU_FLUSHF_modified; =20 if ( order && old.next_level ) - queue_free_pt(hd, _mfn(old.mfn), old.next_level); + queue_free_pt(d, ctx, _mfn(old.mfn), old.next_level); } =20 return 0; @@ -646,7 +650,7 @@ int cf_check amd_iommu_flush_iotlb_pages( return 0; } =20 -int amd_iommu_reserve_domain_unity_map(struct domain *d, +int amd_iommu_reserve_domain_unity_map(struct domain *d, struct iommu_cont= ext *ctx, const struct ivrs_unity_map *map, unsigned int flag) { @@ -664,14 +668,14 @@ int amd_iommu_reserve_domain_unity_map(struct domain = *d, if ( map->write ) p2ma |=3D p2m_access_w; =20 - rc =3D iommu_identity_mapping(d, p2ma, map->addr, + rc =3D iommu_identity_mapping(d, ctx, p2ma, map->addr, map->addr + map->length - 1, flag); } =20 return rc; } =20 -int amd_iommu_reserve_domain_unity_unmap(struct domain *d, +int amd_iommu_reserve_domain_unity_unmap(struct domain *d, struct iommu_co= ntext *ctx, const struct ivrs_unity_map *map) { int rc; @@ -681,7 +685,7 @@ int amd_iommu_reserve_domain_unity_unmap(struct domain = *d, =20 for ( rc =3D 0; map; map =3D map->next ) { - int ret =3D iommu_identity_mapping(d, p2m_access_x, map->addr, + int ret =3D iommu_identity_mapping(d, ctx, p2m_access_x, map->addr, map->addr + map->length - 1, 0); =20 if ( ret && ret !=3D -ENOENT && !rc ) @@ -771,6 +775,7 @@ static int fill_qpt(union amd_iommu_pte *this, unsigned= int level, struct page_info *pgs[IOMMU_MAX_PT_LEVELS]) { struct domain_iommu *hd =3D dom_iommu(dom_io); + struct iommu_context *ctx =3D iommu_default_context(dom_io); unsigned int i; int rc =3D 0; =20 @@ -787,7 +792,7 @@ static int fill_qpt(union amd_iommu_pte *this, unsigned= int level, * page table pages, and the resulting allocations are alw= ays * zeroed. */ - pgs[level] =3D iommu_alloc_pgtable(hd, 0); + pgs[level] =3D iommu_alloc_pgtable(hd, ctx, 0); if ( !pgs[level] ) { rc =3D -ENOMEM; @@ -823,14 +828,15 @@ static int fill_qpt(union amd_iommu_pte *this, unsign= ed int level, int cf_check amd_iommu_quarantine_init(struct pci_dev *pdev, bool scratch_= page) { struct domain_iommu *hd =3D dom_iommu(dom_io); - unsigned int level =3D hd->arch.amd.paging_mode; + struct iommu_context *ctx =3D iommu_default_context(dom_io); + unsigned int level =3D ctx->arch.amd.paging_mode; unsigned int req_id =3D get_dma_requestor_id(pdev->seg, pdev->sbdf.bdf= ); const struct ivrs_mappings *ivrs_mappings =3D get_ivrs_mappings(pdev->= seg); int rc; =20 ASSERT(pcidevs_locked()); - ASSERT(!hd->arch.amd.root_table); - ASSERT(page_list_empty(&hd->arch.pgtables.list)); + ASSERT(!ctx->arch.amd.root_table); + ASSERT(page_list_empty(&ctx->arch.pgtables)); =20 if ( !scratch_page && !ivrs_mappings[req_id].unity_map ) return 0; @@ -843,19 +849,19 @@ int cf_check amd_iommu_quarantine_init(struct pci_dev= *pdev, bool scratch_page) return 0; } =20 - pdev->arch.amd.root_table =3D iommu_alloc_pgtable(hd, 0); + pdev->arch.amd.root_table =3D iommu_alloc_pgtable(hd, ctx, 0); if ( !pdev->arch.amd.root_table ) return -ENOMEM; =20 /* Transiently install the root into DomIO, for iommu_identity_mapping= (). */ - hd->arch.amd.root_table =3D pdev->arch.amd.root_table; + ctx->arch.amd.root_table =3D pdev->arch.amd.root_table; =20 - rc =3D amd_iommu_reserve_domain_unity_map(dom_io, + rc =3D amd_iommu_reserve_domain_unity_map(dom_io, ctx, ivrs_mappings[req_id].unity_ma= p, 0); =20 - iommu_identity_map_teardown(dom_io); - hd->arch.amd.root_table =3D NULL; + iommu_identity_map_teardown(dom_io, ctx); + ctx->arch.amd.root_table =3D NULL; =20 if ( rc ) AMD_IOMMU_WARN("%pp: quarantine unity mapping failed\n", &pdev->sb= df); @@ -871,7 +877,7 @@ int cf_check amd_iommu_quarantine_init(struct pci_dev *= pdev, bool scratch_page) pdev->arch.leaf_mfn =3D page_to_mfn(pgs[0]); } =20 - page_list_move(&pdev->arch.pgtables_list, &hd->arch.pgtables.list); + page_list_move(&pdev->arch.pgtables_list, &ctx->arch.pgtables); =20 if ( rc ) amd_iommu_quarantine_teardown(pdev); @@ -881,16 +887,16 @@ int cf_check amd_iommu_quarantine_init(struct pci_dev= *pdev, bool scratch_page) =20 void amd_iommu_quarantine_teardown(struct pci_dev *pdev) { - struct domain_iommu *hd =3D dom_iommu(dom_io); + struct iommu_context *ctx =3D iommu_default_context(dom_io); =20 ASSERT(pcidevs_locked()); =20 if ( !pdev->arch.amd.root_table ) return; =20 - ASSERT(page_list_empty(&hd->arch.pgtables.list)); - page_list_move(&hd->arch.pgtables.list, &pdev->arch.pgtables_list); - while ( iommu_free_pgtables(dom_io) =3D=3D -ERESTART ) + ASSERT(page_list_empty(&ctx->arch.pgtables)); + page_list_move(&ctx->arch.pgtables, &pdev->arch.pgtables_list); + while ( iommu_free_pgtables(dom_io, ctx) =3D=3D -ERESTART ) /* nothing */; pdev->arch.amd.root_table =3D NULL; } diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/pass= through/amd/pci_amd_iommu.c index f96f59440b..a3815d71be 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -19,6 +19,7 @@ =20 #include #include +#include =20 #include =20 @@ -86,12 +87,12 @@ int get_dma_requestor_id(uint16_t seg, uint16_t bdf) =20 static int __must_check allocate_domain_resources(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); int rc; =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); rc =3D amd_iommu_alloc_root(d); - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); =20 return rc; } @@ -125,7 +126,7 @@ static bool use_ats( } =20 static int __must_check amd_iommu_setup_domain_device( - struct domain *domain, struct amd_iommu *iommu, + struct domain *domain, struct iommu_context *ctx, struct amd_iommu *io= mmu, uint8_t devfn, struct pci_dev *pdev) { struct amd_iommu_dte *table, *dte; @@ -133,7 +134,6 @@ static int __must_check amd_iommu_setup_domain_device( unsigned int req_id, sr_flags; int rc; u8 bus =3D pdev->bus; - struct domain_iommu *hd =3D dom_iommu(domain); const struct ivrs_mappings *ivrs_dev; const struct page_info *root_pg; domid_t domid; @@ -141,7 +141,7 @@ static int __must_check amd_iommu_setup_domain_device( if ( QUARANTINE_SKIP(domain, pdev) ) return 0; =20 - BUG_ON(!hd->arch.amd.paging_mode || !iommu->dev_table.buffer); + BUG_ON(!ctx->arch.amd.paging_mode || !iommu->dev_table.buffer); =20 rc =3D allocate_domain_resources(domain); if ( rc ) @@ -161,7 +161,7 @@ static int __must_check amd_iommu_setup_domain_device( =20 if ( domain !=3D dom_io ) { - root_pg =3D hd->arch.amd.root_table; + root_pg =3D ctx->arch.amd.root_table; domid =3D domain->domain_id; } else @@ -177,7 +177,7 @@ static int __must_check amd_iommu_setup_domain_device( /* bind DTE to domain page-tables */ rc =3D amd_iommu_set_root_page_table( dte, page_to_maddr(root_pg), domid, - hd->arch.amd.paging_mode, sr_flags); + ctx->arch.amd.paging_mode, sr_flags); if ( rc ) { ASSERT(rc < 0); @@ -219,7 +219,7 @@ static int __must_check amd_iommu_setup_domain_device( else rc =3D amd_iommu_set_root_page_table( dte, page_to_maddr(root_pg), domid, - hd->arch.amd.paging_mode, sr_flags); + ctx->arch.amd.paging_mode, sr_flags); if ( rc < 0 ) { spin_unlock_irqrestore(&iommu->lock, flags); @@ -270,7 +270,7 @@ static int __must_check amd_iommu_setup_domain_device( "root table =3D %#"PRIx64", " "domain =3D %d, paging mode =3D %d\n", req_id, pdev->type, page_to_maddr(root_pg), - domid, hd->arch.amd.paging_mode); + domid, ctx->arch.amd.paging_mode); =20 ASSERT(pcidevs_locked()); =20 @@ -352,11 +352,12 @@ static int cf_check iov_enable_xt(void) int amd_iommu_alloc_root(struct domain *d) { struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 - if ( unlikely(!hd->arch.amd.root_table) && d !=3D dom_io ) + if ( unlikely(!ctx->arch.amd.root_table) && d !=3D dom_io ) { - hd->arch.amd.root_table =3D iommu_alloc_pgtable(hd, 0); - if ( !hd->arch.amd.root_table ) + ctx->arch.amd.root_table =3D iommu_alloc_pgtable(hd, ctx, 0); + if ( !ctx->arch.amd.root_table ) return -ENOMEM; } =20 @@ -368,7 +369,7 @@ int __read_mostly amd_iommu_min_paging_mode =3D 1; =20 static int cf_check amd_iommu_domain_init(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); int pglvl =3D amd_iommu_get_paging_mode( 1UL << (domain_max_paddr_bits(d) - PAGE_SHIFT)); =20 @@ -379,7 +380,7 @@ static int cf_check amd_iommu_domain_init(struct domain= *d) * Choose the number of levels for the IOMMU page tables, taking into * account unity maps. */ - hd->arch.amd.paging_mode =3D max(pglvl, amd_iommu_min_paging_mode); + ctx->arch.amd.paging_mode =3D max(pglvl, amd_iommu_min_paging_mode); =20 return 0; } @@ -455,7 +456,7 @@ static void amd_iommu_disable_domain_device(const struc= t domain *domain, AMD_IOMMU_DEBUG("Disable: device id =3D %#x, " "domain =3D %d, paging mode =3D %d\n", req_id, dte->domain_id, - dom_iommu(domain)->arch.amd.paging_mode); + iommu_default_context(domain)->arch.amd.paging_mod= e); } else spin_unlock_irqrestore(&iommu->lock, flags); @@ -466,6 +467,8 @@ static int cf_check reassign_device( struct pci_dev *pdev) { struct amd_iommu *iommu; + struct iommu_context *target_ctx =3D iommu_default_context(target); + struct iommu_context *source_ctx =3D iommu_default_context(source); int rc; =20 iommu =3D find_iommu_for_device(pdev->seg, pdev->sbdf.bdf); @@ -478,7 +481,7 @@ static int cf_check reassign_device( =20 if ( !QUARANTINE_SKIP(target, pdev) ) { - rc =3D amd_iommu_setup_domain_device(target, iommu, devfn, pdev); + rc =3D amd_iommu_setup_domain_device(target, target_ctx, iommu, de= vfn, pdev); if ( rc ) return rc; } @@ -509,7 +512,7 @@ static int cf_check reassign_device( unsigned int bdf =3D PCI_BDF(pdev->bus, devfn); =20 rc =3D amd_iommu_reserve_domain_unity_unmap( - source, + source, source_ctx, ivrs_mappings[get_dma_requestor_id(pdev->seg, bdf)].unity= _map); if ( rc ) return rc; @@ -528,7 +531,8 @@ static int cf_check amd_iommu_assign_device( unsigned int bdf =3D PCI_BDF(pdev->bus, devfn); int req_id =3D get_dma_requestor_id(pdev->seg, bdf); int rc =3D amd_iommu_reserve_domain_unity_map( - d, ivrs_mappings[req_id].unity_map, flag); + d, iommu_default_context(d), + ivrs_mappings[req_id].unity_map, flag); =20 if ( !rc ) rc =3D reassign_device(pdev->domain, d, devfn, pdev); @@ -536,7 +540,8 @@ static int cf_check amd_iommu_assign_device( if ( rc && !is_hardware_domain(d) ) { int ret =3D amd_iommu_reserve_domain_unity_unmap( - d, ivrs_mappings[req_id].unity_map); + d, iommu_default_context(d), + ivrs_mappings[req_id].unity_map); =20 if ( ret ) { @@ -553,22 +558,25 @@ static int cf_check amd_iommu_assign_device( =20 static void cf_check amd_iommu_clear_root_pgtable(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 - spin_lock(&hd->arch.mapping_lock); - hd->arch.amd.root_table =3D NULL; - spin_unlock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); + ctx->arch.amd.root_table =3D NULL; + spin_unlock(&ctx->arch.mapping_lock); } =20 static void cf_check amd_iommu_domain_destroy(struct domain *d) { - iommu_identity_map_teardown(d); - ASSERT(!dom_iommu(d)->arch.amd.root_table); + struct iommu_context *ctx =3D iommu_default_context(d); + + iommu_identity_map_teardown(d, ctx); + ASSERT(!ctx->arch.amd.root_table); } =20 static int cf_check amd_iommu_add_device(u8 devfn, struct pci_dev *pdev) { struct amd_iommu *iommu; + struct iommu_context *ctx; u16 bdf; struct ivrs_mappings *ivrs_mappings; bool fresh_domid =3D false; @@ -577,6 +585,8 @@ static int cf_check amd_iommu_add_device(u8 devfn, stru= ct pci_dev *pdev) if ( !pdev->domain ) return -EINVAL; =20 + ctx =3D iommu_default_context(pdev->domain); + for_each_amd_iommu(iommu) if ( pdev->seg =3D=3D iommu->seg && pdev->sbdf.bdf =3D=3D iommu->b= df ) return is_hardware_domain(pdev->domain) ? 0 : -ENODEV; @@ -633,7 +643,7 @@ static int cf_check amd_iommu_add_device(u8 devfn, stru= ct pci_dev *pdev) } =20 if ( amd_iommu_reserve_domain_unity_map( - pdev->domain, + pdev->domain, ctx, ivrs_mappings[ivrs_mappings[bdf].dte_requestor_id].unity_map, 0) ) AMD_IOMMU_WARN("%pd: unity mapping failed for %pp\n", @@ -647,7 +657,7 @@ static int cf_check amd_iommu_add_device(u8 devfn, stru= ct pci_dev *pdev) fresh_domid =3D true; } =20 - ret =3D amd_iommu_setup_domain_device(pdev->domain, iommu, devfn, pdev= ); + ret =3D amd_iommu_setup_domain_device(pdev->domain, ctx, iommu, devfn,= pdev); if ( ret && fresh_domid ) { iommu_free_domid(pdev->arch.pseudo_domid, iommu->domid_map); @@ -660,12 +670,15 @@ static int cf_check amd_iommu_add_device(u8 devfn, st= ruct pci_dev *pdev) static int cf_check amd_iommu_remove_device(u8 devfn, struct pci_dev *pdev) { struct amd_iommu *iommu; + struct iommu_context *ctx; u16 bdf; struct ivrs_mappings *ivrs_mappings; =20 if ( !pdev->domain ) return -EINVAL; =20 + ctx =3D iommu_default_context(pdev->domain); + iommu =3D find_iommu_for_device(pdev->seg, pdev->sbdf.bdf); if ( !iommu ) { @@ -680,7 +693,7 @@ static int cf_check amd_iommu_remove_device(u8 devfn, s= truct pci_dev *pdev) bdf =3D PCI_BDF(pdev->bus, devfn); =20 if ( amd_iommu_reserve_domain_unity_unmap( - pdev->domain, + pdev->domain, ctx, ivrs_mappings[ivrs_mappings[bdf].dte_requestor_id].unity_map)= ) AMD_IOMMU_WARN("%pd: unity unmapping failed for %pp\n", pdev->domain, &PCI_SBDF(pdev->seg, bdf)); @@ -755,14 +768,14 @@ static void amd_dump_page_table_level(struct page_inf= o *pg, int level, =20 static void cf_check amd_dump_page_tables(struct domain *d) { - const struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 - if ( !hd->arch.amd.root_table ) + if ( !ctx->arch.amd.root_table ) return; =20 - printk("AMD IOMMU %pd table has %u levels\n", d, hd->arch.amd.paging_m= ode); - amd_dump_page_table_level(hd->arch.amd.root_table, - hd->arch.amd.paging_mode, 0, 0); + printk("AMD IOMMU %pd table has %u levels\n", d, ctx->arch.amd.paging_= mode); + amd_dump_page_table_level(ctx->arch.amd.root_table, + ctx->arch.amd.paging_mode, 0, 0); } =20 static const struct iommu_ops __initconst_cf_clobber _iommu_ops =3D { diff --git a/xen/drivers/passthrough/iommu.c b/xen/drivers/passthrough/iomm= u.c index 9e74a1fc72..662da49766 100644 --- a/xen/drivers/passthrough/iommu.c +++ b/xen/drivers/passthrough/iommu.c @@ -403,12 +403,15 @@ long iommu_unmap(struct domain *d, dfn_t dfn0, unsign= ed long page_count, unsigned long i; unsigned int order, j =3D 0; int rc =3D 0; + struct iommu_context *ctx; =20 if ( !is_iommu_enabled(d) ) return 0; =20 ASSERT(!(flags & ~IOMMUF_preempt)); =20 + ctx =3D iommu_default_context(d); + for ( i =3D 0; i < page_count; i +=3D 1UL << order ) { dfn_t dfn =3D dfn_add(dfn0, i); @@ -468,10 +471,13 @@ int iommu_lookup_page(struct domain *d, dfn_t dfn, mf= n_t *mfn, unsigned int *flags) { const struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx; =20 if ( !is_iommu_enabled(d) || !hd->platform_ops->lookup_page ) return -EOPNOTSUPP; =20 + ctx =3D iommu_default_context(d); + return iommu_call(hd->platform_ops, lookup_page, d, dfn, mfn, flags); } =20 diff --git a/xen/drivers/passthrough/vtd/extern.h b/xen/drivers/passthrough= /vtd/extern.h index c16583c951..3dcb77c711 100644 --- a/xen/drivers/passthrough/vtd/extern.h +++ b/xen/drivers/passthrough/vtd/extern.h @@ -80,8 +80,8 @@ uint64_t alloc_pgtable_maddr(unsigned long npages, nodeid= _t node); void free_pgtable_maddr(u64 maddr); void *map_vtd_domain_page(u64 maddr); void unmap_vtd_domain_page(const void *va); -int domain_context_mapping_one(struct domain *domain, struct vtd_iommu *io= mmu, - uint8_t bus, uint8_t devfn, +int domain_context_mapping_one(struct domain *domain, struct iommu_context= *ctx, + struct vtd_iommu *iommu, uint8_t bus, uint8= _t devfn, const struct pci_dev *pdev, domid_t domid, paddr_t pgd_maddr, unsigned int mode); int domain_context_unmap_one(struct domain *domain, struct vtd_iommu *iomm= u, diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/= vtd/iommu.c index 9d7a9977a6..f60f39ee1d 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -54,7 +54,7 @@ #define DEVICE_DOMID(d, pdev) ((d) !=3D dom_io ? (d)->domain_id \ : (pdev)->arch.pseudo_domid) #define DEVICE_PGTABLE(d, pdev) ((d) !=3D dom_io \ - ? dom_iommu(d)->arch.vtd.pgd_maddr \ + ? iommu_default_context(d)->arch.vtd.pgd_= maddr \ : (pdev)->arch.vtd.pgd_maddr) =20 bool __read_mostly iommu_igfx =3D true; @@ -227,7 +227,7 @@ static void check_cleanup_domid_map(const struct domain= *d, =20 if ( !found ) { - clear_bit(iommu->index, dom_iommu(d)->arch.vtd.iommu_bitmap); + clear_bit(iommu->index, iommu_default_context(d)->arch.vtd.iommu_b= itmap); cleanup_domid_map(d->domain_id, iommu); } } @@ -315,8 +315,9 @@ static u64 bus_to_context_maddr(struct vtd_iommu *iommu= , u8 bus) * PTE for the requested address, * - for target =3D=3D 0 the full PTE contents below PADDR_BITS limit. */ -static uint64_t addr_to_dma_page_maddr(struct domain *domain, daddr_t addr, - unsigned int target, +static uint64_t addr_to_dma_page_maddr(struct domain *domain, + struct iommu_context *ctx, + daddr_t addr, unsigned int target, unsigned int *flush_flags, bool all= oc) { struct domain_iommu *hd =3D dom_iommu(domain); @@ -326,10 +327,10 @@ static uint64_t addr_to_dma_page_maddr(struct domain = *domain, daddr_t addr, u64 pte_maddr =3D 0; =20 addr &=3D (((u64)1) << addr_width) - 1; - ASSERT(spin_is_locked(&hd->arch.mapping_lock)); + ASSERT(spin_is_locked(&ctx->arch.mapping_lock)); ASSERT(target || !alloc); =20 - if ( !hd->arch.vtd.pgd_maddr ) + if ( !ctx->arch.vtd.pgd_maddr ) { struct page_info *pg; =20 @@ -337,13 +338,13 @@ static uint64_t addr_to_dma_page_maddr(struct domain = *domain, daddr_t addr, goto out; =20 pte_maddr =3D level; - if ( !(pg =3D iommu_alloc_pgtable(hd, 0)) ) + if ( !(pg =3D iommu_alloc_pgtable(hd, ctx, 0)) ) goto out; =20 - hd->arch.vtd.pgd_maddr =3D page_to_maddr(pg); + ctx->arch.vtd.pgd_maddr =3D page_to_maddr(pg); } =20 - pte_maddr =3D hd->arch.vtd.pgd_maddr; + pte_maddr =3D ctx->arch.vtd.pgd_maddr; parent =3D map_vtd_domain_page(pte_maddr); while ( level > target ) { @@ -379,7 +380,7 @@ static uint64_t addr_to_dma_page_maddr(struct domain *d= omain, daddr_t addr, } =20 pte_maddr =3D level - 1; - pg =3D iommu_alloc_pgtable(hd, DMA_PTE_CONTIG_MASK); + pg =3D iommu_alloc_pgtable(hd, ctx, DMA_PTE_CONTIG_MASK); if ( !pg ) break; =20 @@ -431,13 +432,12 @@ static uint64_t addr_to_dma_page_maddr(struct domain = *domain, daddr_t addr, return pte_maddr; } =20 -static paddr_t domain_pgd_maddr(struct domain *d, paddr_t pgd_maddr, - unsigned int nr_pt_levels) +static paddr_t domain_pgd_maddr(struct domain *d, struct iommu_context *ct= x, + paddr_t pgd_maddr, unsigned int nr_pt_leve= ls) { - struct domain_iommu *hd =3D dom_iommu(d); unsigned int agaw; =20 - ASSERT(spin_is_locked(&hd->arch.mapping_lock)); + ASSERT(spin_is_locked(&ctx->arch.mapping_lock)); =20 if ( pgd_maddr ) /* nothing */; @@ -449,19 +449,19 @@ static paddr_t domain_pgd_maddr(struct domain *d, pad= dr_t pgd_maddr, } else { - if ( !hd->arch.vtd.pgd_maddr ) + if ( !ctx->arch.vtd.pgd_maddr ) { /* * Ensure we have pagetables allocated down to the smallest * level the loop below may need to run to. */ - addr_to_dma_page_maddr(d, 0, min_pt_levels, NULL, true); + addr_to_dma_page_maddr(d, ctx, 0, min_pt_levels, NULL, true); =20 - if ( !hd->arch.vtd.pgd_maddr ) + if ( !ctx->arch.vtd.pgd_maddr ) return 0; } =20 - pgd_maddr =3D hd->arch.vtd.pgd_maddr; + pgd_maddr =3D ctx->arch.vtd.pgd_maddr; } =20 /* Skip top level(s) of page tables for less-than-maximum level DRHDs.= */ @@ -734,7 +734,7 @@ static int __must_check cf_check iommu_flush_iotlb(stru= ct domain *d, dfn_t dfn, unsigned long page_coun= t, unsigned int flush_flag= s) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); struct acpi_drhd_unit *drhd; struct vtd_iommu *iommu; bool flush_dev_iotlb; @@ -762,7 +762,7 @@ static int __must_check cf_check iommu_flush_iotlb(stru= ct domain *d, dfn_t dfn, =20 iommu =3D drhd->iommu; =20 - if ( !test_bit(iommu->index, hd->arch.vtd.iommu_bitmap) ) + if ( !test_bit(iommu->index, ctx->arch.vtd.iommu_bitmap) ) continue; =20 flush_dev_iotlb =3D !!find_ats_dev_drhd(iommu); @@ -790,7 +790,8 @@ static int __must_check cf_check iommu_flush_iotlb(stru= ct domain *d, dfn_t dfn, return ret; } =20 -static void queue_free_pt(struct domain_iommu *hd, mfn_t mfn, unsigned int= level) +static void queue_free_pt(struct domain *d, struct iommu_context *ctx, mfn= _t mfn, + unsigned int level) { if ( level > 1 ) { @@ -799,13 +800,13 @@ static void queue_free_pt(struct domain_iommu *hd, mf= n_t mfn, unsigned int level =20 for ( i =3D 0; i < PTE_NUM; ++i ) if ( dma_pte_present(pt[i]) && !dma_pte_superpage(pt[i]) ) - queue_free_pt(hd, maddr_to_mfn(dma_pte_addr(pt[i])), + queue_free_pt(d, ctx, maddr_to_mfn(dma_pte_addr(pt[i])), level - 1); =20 unmap_domain_page(pt); } =20 - iommu_queue_free_pgtable(hd, mfn_to_page(mfn)); + iommu_queue_free_pgtable(d, ctx, mfn_to_page(mfn)); } =20 static int iommu_set_root_entry(struct vtd_iommu *iommu) @@ -1435,10 +1436,11 @@ void __init iommu_free(struct acpi_drhd_unit *drhd) static int cf_check intel_iommu_domain_init(struct domain *d) { struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 - hd->arch.vtd.iommu_bitmap =3D xzalloc_array(unsigned long, + ctx->arch.vtd.iommu_bitmap =3D xzalloc_array(unsigned long, BITS_TO_LONGS(nr_iommus)); - if ( !hd->arch.vtd.iommu_bitmap ) + if ( !ctx->arch.vtd.iommu_bitmap ) return -ENOMEM; =20 hd->arch.vtd.agaw =3D width_to_agaw(DEFAULT_DOMAIN_ADDRESS_WIDTH); @@ -1479,11 +1481,11 @@ static void __hwdom_init cf_check intel_iommu_hwdom= _init(struct domain *d) */ int domain_context_mapping_one( struct domain *domain, + struct iommu_context *ctx, struct vtd_iommu *iommu, uint8_t bus, uint8_t devfn, const struct pci_dev *pdev, domid_t domid, paddr_t pgd_maddr, unsigned int mode) { - struct domain_iommu *hd =3D dom_iommu(domain); struct context_entry *context, *context_entries, lctxt; __uint128_t res, old; uint64_t maddr; @@ -1531,12 +1533,12 @@ int domain_context_mapping_one( { paddr_t root; =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); =20 - root =3D domain_pgd_maddr(domain, pgd_maddr, iommu->nr_pt_levels); + root =3D domain_pgd_maddr(domain, ctx, pgd_maddr, iommu->nr_pt_lev= els); if ( !root ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); spin_unlock(&iommu->lock); unmap_vtd_domain_page(context_entries); if ( prev_dom ) @@ -1550,7 +1552,7 @@ int domain_context_mapping_one( else context_set_translation_type(lctxt, CONTEXT_TT_MULTI_LEVEL); =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); } =20 rc =3D context_set_domain_id(&lctxt, domid, iommu); @@ -1624,7 +1626,7 @@ int domain_context_mapping_one( if ( rc > 0 ) rc =3D 0; =20 - set_bit(iommu->index, hd->arch.vtd.iommu_bitmap); + set_bit(iommu->index, ctx->arch.vtd.iommu_bitmap); =20 unmap_vtd_domain_page(context_entries); =20 @@ -1642,7 +1644,7 @@ int domain_context_mapping_one( (prev_dom =3D=3D dom_io && !pdev) ) ret =3D domain_context_unmap_one(domain, iommu, bus, devfn); else - ret =3D domain_context_mapping_one(prev_dom, iommu, bus, devfn= , pdev, + ret =3D domain_context_mapping_one(prev_dom, ctx, iommu, bus, = devfn, pdev, DEVICE_DOMID(prev_dom, pdev), DEVICE_PGTABLE(prev_dom, pdev= ), (mode & MAP_WITH_RMRR) | @@ -1661,8 +1663,8 @@ int domain_context_mapping_one( static const struct acpi_drhd_unit *domain_context_unmap( struct domain *d, uint8_t devfn, struct pci_dev *pdev); =20 -static int domain_context_mapping(struct domain *domain, u8 devfn, - struct pci_dev *pdev) +static int domain_context_mapping(struct domain *domain, struct iommu_cont= ext *ctx, + u8 devfn, struct pci_dev *pdev) { const struct acpi_drhd_unit *drhd =3D acpi_find_matched_drhd_unit(pdev= ); const struct acpi_rmrr_unit *rmrr; @@ -1731,7 +1733,7 @@ static int domain_context_mapping(struct domain *doma= in, u8 devfn, if ( iommu_debug ) printk(VTDPREFIX "%pd:PCIe: map %pp\n", domain, &PCI_SBDF(seg, bus, devfn)); - ret =3D domain_context_mapping_one(domain, drhd->iommu, bus, devfn= , pdev, + ret =3D domain_context_mapping_one(domain, ctx, drhd->iommu, bus, = devfn, pdev, DEVICE_DOMID(domain, pdev), pgd_m= addr, mode); if ( ret > 0 ) @@ -1757,7 +1759,7 @@ static int domain_context_mapping(struct domain *doma= in, u8 devfn, printk(VTDPREFIX "%pd:PCI: map %pp\n", domain, &PCI_SBDF(seg, bus, devfn)); =20 - ret =3D domain_context_mapping_one(domain, drhd->iommu, bus, devfn, + ret =3D domain_context_mapping_one(domain, ctx, drhd->iommu, bus, = devfn, pdev, DEVICE_DOMID(domain, pdev), pgd_maddr, mode); if ( ret < 0 ) @@ -1788,7 +1790,7 @@ static int domain_context_mapping(struct domain *doma= in, u8 devfn, * their owner would be the wrong one. Pass NULL instead. */ if ( ret >=3D 0 ) - ret =3D domain_context_mapping_one(domain, drhd->iommu, bus, d= evfn, + ret =3D domain_context_mapping_one(domain, ctx, drhd->iommu, b= us, devfn, NULL, DEVICE_DOMID(domain, pd= ev), pgd_maddr, mode); =20 @@ -1804,7 +1806,7 @@ static int domain_context_mapping(struct domain *doma= in, u8 devfn, */ if ( !ret && pdev_type(seg, bus, devfn) =3D=3D DEV_TYPE_PCIe2PCI_B= RIDGE && (secbus !=3D pdev->bus || pdev->devfn !=3D 0) ) - ret =3D domain_context_mapping_one(domain, drhd->iommu, secbus= , 0, + ret =3D domain_context_mapping_one(domain, ctx, drhd->iommu, s= ecbus, 0, NULL, DEVICE_DOMID(domain, pd= ev), pgd_maddr, mode); =20 @@ -1813,7 +1815,7 @@ static int domain_context_mapping(struct domain *doma= in, u8 devfn, if ( !prev_present ) domain_context_unmap(domain, devfn, pdev); else if ( pdev->domain !=3D domain ) /* Avoid infinite recursi= on. */ - domain_context_mapping(pdev->domain, devfn, pdev); + domain_context_mapping(pdev->domain, ctx, devfn, pdev); } =20 break; @@ -2001,44 +2003,44 @@ static const struct acpi_drhd_unit *domain_context_= unmap( =20 static void cf_check iommu_clear_root_pgtable(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 - spin_lock(&hd->arch.mapping_lock); - hd->arch.vtd.pgd_maddr =3D 0; - spin_unlock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); + ctx->arch.vtd.pgd_maddr =3D 0; + spin_unlock(&ctx->arch.mapping_lock); } =20 static void cf_check iommu_domain_teardown(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); const struct acpi_drhd_unit *drhd; =20 if ( list_empty(&acpi_drhd_units) ) return; =20 - iommu_identity_map_teardown(d); + iommu_identity_map_teardown(d, ctx); =20 - ASSERT(!hd->arch.vtd.pgd_maddr); + ASSERT(!ctx->arch.vtd.pgd_maddr); =20 for_each_drhd_unit ( drhd ) cleanup_domid_map(d->domain_id, drhd->iommu); =20 - XFREE(hd->arch.vtd.iommu_bitmap); + XFREE(ctx->arch.vtd.iommu_bitmap); } =20 static void quarantine_teardown(struct pci_dev *pdev, const struct acpi_drhd_unit *drhd) { - struct domain_iommu *hd =3D dom_iommu(dom_io); + struct iommu_context *ctx =3D iommu_default_context(dom_io); =20 ASSERT(pcidevs_locked()); =20 if ( !pdev->arch.vtd.pgd_maddr ) return; =20 - ASSERT(page_list_empty(&hd->arch.pgtables.list)); - page_list_move(&hd->arch.pgtables.list, &pdev->arch.pgtables_list); - while ( iommu_free_pgtables(dom_io) =3D=3D -ERESTART ) + ASSERT(page_list_empty(&ctx->arch.pgtables)); + page_list_move(&ctx->arch.pgtables, &pdev->arch.pgtables_list); + while ( iommu_free_pgtables(dom_io, ctx) =3D=3D -ERESTART ) /* nothing */; pdev->arch.vtd.pgd_maddr =3D 0; =20 @@ -2051,6 +2053,7 @@ static int __must_check cf_check intel_iommu_map_page( unsigned int *flush_flags) { struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); struct dma_pte *page, *pte, old, new =3D {}; u64 pg_maddr; unsigned int level =3D (IOMMUF_order(flags) / LEVEL_STRIDE) + 1; @@ -2067,7 +2070,7 @@ static int __must_check cf_check intel_iommu_map_page( if ( iommu_hwdom_passthrough && is_hardware_domain(d) ) return 0; =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); =20 /* * IOMMU mapping request can be safely ignored when the domain is dyin= g. @@ -2077,15 +2080,15 @@ static int __must_check cf_check intel_iommu_map_pa= ge( */ if ( d->is_dying ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); return 0; } =20 - pg_maddr =3D addr_to_dma_page_maddr(d, dfn_to_daddr(dfn), level, flush= _flags, + pg_maddr =3D addr_to_dma_page_maddr(d, ctx, dfn_to_daddr(dfn), level, = flush_flags, true); if ( pg_maddr < PAGE_SIZE ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); return -ENOMEM; } =20 @@ -2106,7 +2109,7 @@ static int __must_check cf_check intel_iommu_map_page( =20 if ( !((old.val ^ new.val) & ~DMA_PTE_CONTIG_MASK) ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); unmap_vtd_domain_page(page); return 0; } @@ -2135,7 +2138,7 @@ static int __must_check cf_check intel_iommu_map_page( new.val &=3D ~(LEVEL_MASK << level_to_offset_bits(level)); dma_set_pte_superpage(new); =20 - pg_maddr =3D addr_to_dma_page_maddr(d, dfn_to_daddr(dfn), ++level, + pg_maddr =3D addr_to_dma_page_maddr(d, ctx, dfn_to_daddr(dfn), ++l= evel, flush_flags, false); BUG_ON(pg_maddr < PAGE_SIZE); =20 @@ -2145,11 +2148,11 @@ static int __must_check cf_check intel_iommu_map_pa= ge( iommu_sync_cache(pte, sizeof(*pte)); =20 *flush_flags |=3D IOMMU_FLUSHF_modified | IOMMU_FLUSHF_all; - iommu_queue_free_pgtable(hd, pg); + iommu_queue_free_pgtable(d, ctx, pg); perfc_incr(iommu_pt_coalesces); } =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); unmap_vtd_domain_page(page); =20 *flush_flags |=3D IOMMU_FLUSHF_added; @@ -2158,7 +2161,7 @@ static int __must_check cf_check intel_iommu_map_page( *flush_flags |=3D IOMMU_FLUSHF_modified; =20 if ( IOMMUF_order(flags) && !dma_pte_superpage(old) ) - queue_free_pt(hd, maddr_to_mfn(dma_pte_addr(old)), + queue_free_pt(d, ctx, maddr_to_mfn(dma_pte_addr(old)), IOMMUF_order(flags) / LEVEL_STRIDE); } =20 @@ -2169,6 +2172,7 @@ static int __must_check cf_check intel_iommu_unmap_pa= ge( struct domain *d, dfn_t dfn, unsigned int order, unsigned int *flush_f= lags) { struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); daddr_t addr =3D dfn_to_daddr(dfn); struct dma_pte *page =3D NULL, *pte =3D NULL, old; uint64_t pg_maddr; @@ -2188,12 +2192,12 @@ static int __must_check cf_check intel_iommu_unmap_= page( if ( iommu_hwdom_passthrough && is_hardware_domain(d) ) return 0; =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); /* get target level pte */ - pg_maddr =3D addr_to_dma_page_maddr(d, addr, level, flush_flags, false= ); + pg_maddr =3D addr_to_dma_page_maddr(d, ctx, addr, level, flush_flags, = false); if ( pg_maddr < PAGE_SIZE ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); return pg_maddr ? -ENOMEM : 0; } =20 @@ -2202,7 +2206,7 @@ static int __must_check cf_check intel_iommu_unmap_pa= ge( =20 if ( !dma_pte_present(*pte) ) { - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); unmap_vtd_domain_page(page); return 0; } @@ -2220,7 +2224,7 @@ static int __must_check cf_check intel_iommu_unmap_pa= ge( =20 unmap_vtd_domain_page(page); =20 - pg_maddr =3D addr_to_dma_page_maddr(d, addr, level, flush_flags, f= alse); + pg_maddr =3D addr_to_dma_page_maddr(d, ctx, addr, level, flush_fla= gs, false); BUG_ON(pg_maddr < PAGE_SIZE); =20 page =3D map_vtd_domain_page(pg_maddr); @@ -2229,18 +2233,18 @@ static int __must_check cf_check intel_iommu_unmap_= page( iommu_sync_cache(pte, sizeof(*pte)); =20 *flush_flags |=3D IOMMU_FLUSHF_all; - iommu_queue_free_pgtable(hd, pg); + iommu_queue_free_pgtable(d, ctx, pg); perfc_incr(iommu_pt_coalesces); } =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); =20 unmap_vtd_domain_page(page); =20 *flush_flags |=3D IOMMU_FLUSHF_modified; =20 if ( order && !dma_pte_superpage(old) ) - queue_free_pt(hd, maddr_to_mfn(dma_pte_addr(old)), + queue_free_pt(d, ctx, maddr_to_mfn(dma_pte_addr(old)), order / LEVEL_STRIDE); =20 return 0; @@ -2249,7 +2253,7 @@ static int __must_check cf_check intel_iommu_unmap_pa= ge( static int cf_check intel_iommu_lookup_page( struct domain *d, dfn_t dfn, mfn_t *mfn, unsigned int *flags) { - struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); uint64_t val; =20 /* @@ -2260,11 +2264,11 @@ static int cf_check intel_iommu_lookup_page( (iommu_hwdom_passthrough && is_hardware_domain(d)) ) return -EOPNOTSUPP; =20 - spin_lock(&hd->arch.mapping_lock); + spin_lock(&ctx->arch.mapping_lock); =20 - val =3D addr_to_dma_page_maddr(d, dfn_to_daddr(dfn), 0, NULL, false); + val =3D addr_to_dma_page_maddr(d, ctx, dfn_to_daddr(dfn), 0, NULL, fal= se); =20 - spin_unlock(&hd->arch.mapping_lock); + spin_unlock(&ctx->arch.mapping_lock); =20 if ( val < PAGE_SIZE ) return -ENOENT; @@ -2285,7 +2289,7 @@ static bool __init vtd_ept_page_compatible(const stru= ct vtd_iommu *iommu) =20 /* EPT is not initialised yet, so we must check the capability in * the MSR explicitly rather than use cpu_has_vmx_ept_*() */ - if ( rdmsr_safe(MSR_IA32_VMX_EPT_VPID_CAP, ept_cap) !=3D 0 )=20 + if ( rdmsr_safe(MSR_IA32_VMX_EPT_VPID_CAP, ept_cap) !=3D 0 ) return false; =20 return (ept_has_2mb(ept_cap) && opt_hap_2mb) <=3D @@ -2297,6 +2301,7 @@ static bool __init vtd_ept_page_compatible(const stru= ct vtd_iommu *iommu) static int cf_check intel_iommu_add_device(u8 devfn, struct pci_dev *pdev) { struct acpi_rmrr_unit *rmrr; + struct iommu_context *ctx; u16 bdf; int ret, i; =20 @@ -2305,6 +2310,8 @@ static int cf_check intel_iommu_add_device(u8 devfn, = struct pci_dev *pdev) if ( !pdev->domain ) return -EINVAL; =20 + ctx =3D iommu_default_context(pdev->domain); + for_each_rmrr_device ( rmrr, bdf, i ) { if ( rmrr->segment =3D=3D pdev->seg && bdf =3D=3D PCI_BDF(pdev->bu= s, devfn) ) @@ -2315,7 +2322,7 @@ static int cf_check intel_iommu_add_device(u8 devfn, = struct pci_dev *pdev) * Since RMRRs are always reserved in the e820 map for the har= dware * domain, there shouldn't be a conflict. */ - ret =3D iommu_identity_mapping(pdev->domain, p2m_access_rw, + ret =3D iommu_identity_mapping(pdev->domain, ctx, p2m_access_r= w, rmrr->base_address, rmrr->end_add= ress, 0); if ( ret ) @@ -2324,7 +2331,7 @@ static int cf_check intel_iommu_add_device(u8 devfn, = struct pci_dev *pdev) } } =20 - ret =3D domain_context_mapping(pdev->domain, devfn, pdev); + ret =3D domain_context_mapping(pdev->domain, ctx, devfn, pdev); if ( ret ) dprintk(XENLOG_ERR VTDPREFIX, "%pd: context mapping failed\n", pdev->domain); @@ -2353,10 +2360,13 @@ static int cf_check intel_iommu_remove_device(u8 de= vfn, struct pci_dev *pdev) struct acpi_rmrr_unit *rmrr; u16 bdf; unsigned int i; + struct iommu_context *ctx; =20 if ( !pdev->domain ) return -EINVAL; =20 + ctx =3D iommu_default_context(pdev->domain); + drhd =3D domain_context_unmap(pdev->domain, devfn, pdev); if ( IS_ERR(drhd) ) return PTR_ERR(drhd); @@ -2370,7 +2380,7 @@ static int cf_check intel_iommu_remove_device(u8 devf= n, struct pci_dev *pdev) * Any flag is nothing to clear these mappings but here * its always safe and strict to set 0. */ - iommu_identity_mapping(pdev->domain, p2m_access_x, rmrr->base_addr= ess, + iommu_identity_mapping(pdev->domain, ctx, p2m_access_x, rmrr->base= _address, rmrr->end_address, 0); } =20 @@ -2389,7 +2399,9 @@ static int cf_check intel_iommu_remove_device(u8 devf= n, struct pci_dev *pdev) static int __hwdom_init cf_check setup_hwdom_device( u8 devfn, struct pci_dev *pdev) { - return domain_context_mapping(pdev->domain, devfn, pdev); + struct iommu_context *ctx =3D iommu_default_context(pdev->domain); + + return domain_context_mapping(pdev->domain, ctx, devfn, pdev); } =20 void clear_fault_bits(struct vtd_iommu *iommu) @@ -2483,7 +2495,7 @@ static int __must_check init_vtd_hw(bool resume) =20 /* * Enable queue invalidation - */ =20 + */ for_each_drhd_unit ( drhd ) { iommu =3D drhd->iommu; @@ -2504,7 +2516,7 @@ static int __must_check init_vtd_hw(bool resume) =20 /* * Enable interrupt remapping - */ =20 + */ if ( iommu_intremap !=3D iommu_intremap_off ) { int apic; @@ -2561,6 +2573,7 @@ static int __must_check init_vtd_hw(bool resume) =20 static void __hwdom_init setup_hwdom_rmrr(struct domain *d) { + struct iommu_context *ctx =3D iommu_default_context(d); struct acpi_rmrr_unit *rmrr; u16 bdf; int ret, i; @@ -2574,7 +2587,7 @@ static void __hwdom_init setup_hwdom_rmrr(struct doma= in *d) * domain, there shouldn't be a conflict. So its always safe and * strict to set 0. */ - ret =3D iommu_identity_mapping(d, p2m_access_rw, rmrr->base_addres= s, + ret =3D iommu_identity_mapping(d, ctx, p2m_access_rw, rmrr->base_a= ddress, rmrr->end_address, 0); if ( ret ) dprintk(XENLOG_ERR VTDPREFIX, @@ -2739,6 +2752,8 @@ static int cf_check reassign_device_ownership( =20 if ( !QUARANTINE_SKIP(target, pdev->arch.vtd.pgd_maddr) ) { + struct iommu_context *target_ctx =3D iommu_default_context(target); + if ( !has_arch_pdevs(target) ) vmx_pi_hooks_assign(target); =20 @@ -2753,7 +2768,7 @@ static int cf_check reassign_device_ownership( untrusted_msi =3D true; #endif =20 - ret =3D domain_context_mapping(target, devfn, pdev); + ret =3D domain_context_mapping(target, target_ctx, devfn, pdev); =20 if ( !ret && pdev->devfn =3D=3D devfn && !QUARANTINE_SKIP(source, pdev->arch.vtd.pgd_maddr) ) @@ -2802,6 +2817,7 @@ static int cf_check reassign_device_ownership( if ( !is_hardware_domain(source) ) { const struct acpi_rmrr_unit *rmrr; + struct iommu_context *ctx =3D iommu_default_context(source); u16 bdf; unsigned int i; =20 @@ -2813,7 +2829,7 @@ static int cf_check reassign_device_ownership( * Any RMRR flag is always ignored when remove a device, * but its always safe and strict to set 0. */ - ret =3D iommu_identity_mapping(source, p2m_access_x, + ret =3D iommu_identity_mapping(source, ctx, p2m_access_x, rmrr->base_address, rmrr->end_address, 0); if ( ret && ret !=3D -ENOENT ) @@ -2828,6 +2844,7 @@ static int cf_check intel_iommu_assign_device( struct domain *d, u8 devfn, struct pci_dev *pdev, u32 flag) { struct domain *s =3D pdev->domain; + struct iommu_context *ctx =3D iommu_default_context(d); struct acpi_rmrr_unit *rmrr; int ret =3D 0, i; u16 bdf, seg; @@ -2875,7 +2892,7 @@ static int cf_check intel_iommu_assign_device( { if ( rmrr->segment =3D=3D seg && bdf =3D=3D PCI_BDF(bus, devfn) ) { - ret =3D iommu_identity_mapping(d, p2m_access_rw, rmrr->base_ad= dress, + ret =3D iommu_identity_mapping(d, ctx, p2m_access_rw, rmrr->ba= se_address, rmrr->end_address, flag); if ( ret ) { @@ -2898,7 +2915,7 @@ static int cf_check intel_iommu_assign_device( { if ( rmrr->segment =3D=3D seg && bdf =3D=3D PCI_BDF(bus, devfn) ) { - int rc =3D iommu_identity_mapping(d, p2m_access_x, + int rc =3D iommu_identity_mapping(d, ctx, p2m_access_x, rmrr->base_address, rmrr->end_address, 0); =20 @@ -3071,10 +3088,11 @@ static void vtd_dump_page_table_level(paddr_t pt_ma= ddr, int level, paddr_t gpa, static void cf_check vtd_dump_page_tables(struct domain *d) { const struct domain_iommu *hd =3D dom_iommu(d); + struct iommu_context *ctx =3D iommu_default_context(d); =20 printk(VTDPREFIX" %pd table has %d levels\n", d, agaw_to_level(hd->arch.vtd.agaw)); - vtd_dump_page_table_level(hd->arch.vtd.pgd_maddr, + vtd_dump_page_table_level(ctx->arch.vtd.pgd_maddr, agaw_to_level(hd->arch.vtd.agaw), 0, 0); } =20 @@ -3082,6 +3100,7 @@ static int fill_qpt(struct dma_pte *this, unsigned in= t level, struct page_info *pgs[6]) { struct domain_iommu *hd =3D dom_iommu(dom_io); + struct iommu_context *ctx =3D iommu_default_context(dom_io); unsigned int i; int rc =3D 0; =20 @@ -3098,7 +3117,7 @@ static int fill_qpt(struct dma_pte *this, unsigned in= t level, * page table pages, and the resulting allocations are alw= ays * zeroed. */ - pgs[level] =3D iommu_alloc_pgtable(hd, 0); + pgs[level] =3D iommu_alloc_pgtable(hd, ctx, 0); if ( !pgs[level] ) { rc =3D -ENOMEM; @@ -3132,6 +3151,7 @@ static int cf_check intel_iommu_quarantine_init(struc= t pci_dev *pdev, bool scratch_page) { struct domain_iommu *hd =3D dom_iommu(dom_io); + struct iommu_context *ctx =3D iommu_default_context(dom_io); struct page_info *pg; unsigned int agaw =3D hd->arch.vtd.agaw; unsigned int level =3D agaw_to_level(agaw); @@ -3142,8 +3162,8 @@ static int cf_check intel_iommu_quarantine_init(struc= t pci_dev *pdev, int rc; =20 ASSERT(pcidevs_locked()); - ASSERT(!hd->arch.vtd.pgd_maddr); - ASSERT(page_list_empty(&hd->arch.pgtables.list)); + ASSERT(!ctx->arch.vtd.pgd_maddr); + ASSERT(page_list_empty(&ctx->arch.pgtables)); =20 if ( pdev->arch.vtd.pgd_maddr ) { @@ -3155,14 +3175,14 @@ static int cf_check intel_iommu_quarantine_init(str= uct pci_dev *pdev, if ( !drhd ) return -ENODEV; =20 - pg =3D iommu_alloc_pgtable(hd, 0); + pg =3D iommu_alloc_pgtable(hd, ctx, 0); if ( !pg ) return -ENOMEM; =20 rc =3D context_set_domain_id(NULL, pdev->arch.pseudo_domid, drhd->iomm= u); =20 /* Transiently install the root into DomIO, for iommu_identity_mapping= (). */ - hd->arch.vtd.pgd_maddr =3D page_to_maddr(pg); + ctx->arch.vtd.pgd_maddr =3D page_to_maddr(pg); =20 for_each_rmrr_device ( rmrr, bdf, i ) { @@ -3173,7 +3193,7 @@ static int cf_check intel_iommu_quarantine_init(struc= t pci_dev *pdev, { rmrr_found =3D true; =20 - rc =3D iommu_identity_mapping(dom_io, p2m_access_rw, + rc =3D iommu_identity_mapping(dom_io, ctx, p2m_access_rw, rmrr->base_address, rmrr->end_addr= ess, 0); if ( rc ) @@ -3183,8 +3203,8 @@ static int cf_check intel_iommu_quarantine_init(struc= t pci_dev *pdev, } } =20 - iommu_identity_map_teardown(dom_io); - hd->arch.vtd.pgd_maddr =3D 0; + iommu_identity_map_teardown(dom_io, ctx); + ctx->arch.vtd.pgd_maddr =3D 0; pdev->arch.vtd.pgd_maddr =3D page_to_maddr(pg); =20 if ( !rc && scratch_page ) @@ -3199,7 +3219,7 @@ static int cf_check intel_iommu_quarantine_init(struc= t pci_dev *pdev, pdev->arch.leaf_mfn =3D page_to_mfn(pgs[0]); } =20 - page_list_move(&pdev->arch.pgtables_list, &hd->arch.pgtables.list); + page_list_move(&pdev->arch.pgtables_list, &ctx->arch.pgtables); =20 if ( rc || (!scratch_page && !rmrr_found) ) quarantine_teardown(pdev, drhd); diff --git a/xen/drivers/passthrough/vtd/quirks.c b/xen/drivers/passthrough= /vtd/quirks.c index dc3dac749c..7937eb8c2b 100644 --- a/xen/drivers/passthrough/vtd/quirks.c +++ b/xen/drivers/passthrough/vtd/quirks.c @@ -422,7 +422,8 @@ static int __must_check map_me_phantom_function(struct = domain *domain, =20 /* map or unmap ME phantom function */ if ( !(mode & UNMAP_ME_PHANTOM_FUNC) ) - rc =3D domain_context_mapping_one(domain, drhd->iommu, 0, + rc =3D domain_context_mapping_one(domain, iommu_default_context(do= main), + drhd->iommu, 0, PCI_DEVFN(dev, 7), NULL, domid, pgd_maddr, mode); else diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/= x86/iommu.c index 8b1e0596b8..4a3fe059cb 100644 --- a/xen/drivers/passthrough/x86/iommu.c +++ b/xen/drivers/passthrough/x86/iommu.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -185,26 +186,31 @@ void __hwdom_init arch_iommu_check_autotranslated_hwd= om(struct domain *d) =20 int arch_iommu_domain_init(struct domain *d) { - struct domain_iommu *hd =3D dom_iommu(d); + INIT_PAGE_LIST_HEAD(&dom_iommu(d)->arch.free_queue); + return 0; +} =20 - spin_lock_init(&hd->arch.mapping_lock); +int arch_iommu_context_init(struct domain *d, struct iommu_context *ctx, u= 32 flags) +{ + spin_lock_init(&ctx->arch.mapping_lock); =20 - INIT_PAGE_LIST_HEAD(&hd->arch.pgtables.list); - spin_lock_init(&hd->arch.pgtables.lock); - INIT_LIST_HEAD(&hd->arch.identity_maps); + INIT_PAGE_LIST_HEAD(&ctx->arch.pgtables); + INIT_LIST_HEAD(&ctx->arch.identity_maps); + + return 0; +} + +int arch_iommu_context_teardown(struct domain *d, struct iommu_context *ct= x, u32 flags) +{ + /* Cleanup all page tables */ + while ( iommu_free_pgtables(d, ctx) =3D=3D -ERESTART ) + /* nothing */; =20 return 0; } =20 void arch_iommu_domain_destroy(struct domain *d) { - /* - * There should be not page-tables left allocated by the time the - * domain is destroyed. Note that arch_iommu_domain_destroy() is - * called unconditionally, so pgtables may be uninitialized. - */ - ASSERT(!dom_iommu(d)->platform_ops || - page_list_empty(&dom_iommu(d)->arch.pgtables.list)); } =20 struct identity_map { @@ -214,14 +220,13 @@ struct identity_map { unsigned int count; }; =20 -int iommu_identity_mapping(struct domain *d, p2m_access_t p2ma, - paddr_t base, paddr_t end, +int iommu_identity_mapping(struct domain *d, struct iommu_context *ctx, + p2m_access_t p2ma, paddr_t base, paddr_t end, unsigned int flag) { unsigned long base_pfn =3D base >> PAGE_SHIFT_4K; unsigned long end_pfn =3D PAGE_ALIGN_4K(end) >> PAGE_SHIFT_4K; struct identity_map *map; - struct domain_iommu *hd =3D dom_iommu(d); =20 ASSERT(pcidevs_locked()); ASSERT(base < end); @@ -230,7 +235,7 @@ int iommu_identity_mapping(struct domain *d, p2m_access= _t p2ma, * No need to acquire hd->arch.mapping_lock: Both insertion and removal * get done while holding pcidevs_lock. */ - list_for_each_entry( map, &hd->arch.identity_maps, list ) + list_for_each_entry( map, &ctx->arch.identity_maps, list ) { if ( map->base =3D=3D base && map->end =3D=3D end ) { @@ -280,7 +285,7 @@ int iommu_identity_mapping(struct domain *d, p2m_access= _t p2ma, * Insert into list ahead of mapping, so the range can be found when * trying to clean up. */ - list_add_tail(&map->list, &hd->arch.identity_maps); + list_add_tail(&map->list, &ctx->arch.identity_maps); =20 for ( ; base_pfn < end_pfn; ++base_pfn ) { @@ -300,12 +305,11 @@ int iommu_identity_mapping(struct domain *d, p2m_acce= ss_t p2ma, return 0; } =20 -void iommu_identity_map_teardown(struct domain *d) +void iommu_identity_map_teardown(struct domain *d, struct iommu_context *c= tx) { - struct domain_iommu *hd =3D dom_iommu(d); struct identity_map *map, *tmp; =20 - list_for_each_entry_safe ( map, tmp, &hd->arch.identity_maps, list ) + list_for_each_entry_safe ( map, tmp, &ctx->arch.identity_maps, list ) { list_del(&map->list); xfree(map); @@ -603,7 +607,7 @@ void iommu_free_domid(domid_t domid, unsigned long *map) BUG(); } =20 -int iommu_free_pgtables(struct domain *d) +int iommu_free_pgtables(struct domain *d, struct iommu_context *ctx) { struct domain_iommu *hd =3D dom_iommu(d); struct page_info *pg; @@ -613,7 +617,7 @@ int iommu_free_pgtables(struct domain *d) return 0; =20 /* After this barrier, no new IOMMU mappings can be inserted. */ - spin_barrier(&hd->arch.mapping_lock); + spin_barrier(&ctx->arch.mapping_lock); =20 /* * Pages will be moved to the free list below. So we want to @@ -621,7 +625,7 @@ int iommu_free_pgtables(struct domain *d) */ iommu_vcall(hd->platform_ops, clear_root_pgtable, d); =20 - while ( (pg =3D page_list_remove_head(&hd->arch.pgtables.list)) ) + while ( (pg =3D page_list_remove_head(&ctx->arch.pgtables)) ) { free_domheap_page(pg); =20 @@ -633,6 +637,7 @@ int iommu_free_pgtables(struct domain *d) } =20 struct page_info *iommu_alloc_pgtable(struct domain_iommu *hd, + struct iommu_context *ctx, uint64_t contig_mask) { unsigned int memflags =3D 0; @@ -677,9 +682,7 @@ struct page_info *iommu_alloc_pgtable(struct domain_iom= mu *hd, =20 unmap_domain_page(p); =20 - spin_lock(&hd->arch.pgtables.lock); - page_list_add(pg, &hd->arch.pgtables.list); - spin_unlock(&hd->arch.pgtables.lock); + page_list_add(pg, &ctx->arch.pgtables); =20 return pg; } @@ -718,13 +721,12 @@ static void cf_check free_queued_pgtables(void *arg) } } =20 -void iommu_queue_free_pgtable(struct domain_iommu *hd, struct page_info *p= g) +void iommu_queue_free_pgtable(struct domain *d, struct iommu_context *ctx, + struct page_info *pg) { unsigned int cpu =3D smp_processor_id(); =20 - spin_lock(&hd->arch.pgtables.lock); - page_list_del(pg, &hd->arch.pgtables.list); - spin_unlock(&hd->arch.pgtables.lock); + page_list_del(pg, &ctx->arch.pgtables); =20 page_list_add_tail(pg, &per_cpu(free_pgt_list, cpu)); =20 diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index b928c67e19..11d23cdafb 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -343,9 +343,18 @@ extern int iommu_get_extra_reserved_device_memory(iomm= u_grdm_t *func, # define iommu_vcall iommu_call #endif =20 +struct iommu_context { + #ifdef CONFIG_HAS_PASSTHROUGH + u16 id; /* Context id (0 means default context) */ + + struct arch_iommu_context arch; + #endif +}; + struct domain_iommu { #ifdef CONFIG_HAS_PASSTHROUGH struct arch_iommu arch; + struct iommu_context default_ctx; #endif =20 /* iommu_ops */ @@ -380,6 +389,7 @@ struct domain_iommu { #define dom_iommu(d) (&(d)->iommu) #define iommu_set_feature(d, f) set_bit(f, dom_iommu(d)->features) #define iommu_clear_feature(d, f) clear_bit(f, dom_iommu(d)->features) +#define iommu_default_context(d) (&dom_iommu(d)->default_ctx) /* does not = lock ! */ =20 /* Are we using the domain P2M table as its IOMMU pagetable? */ #define iommu_use_hap_pt(d) (IS_ENABLED(CONFIG_HVM) && \ --=20 2.47.2 Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech