From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16712155385 for ; Mon, 25 Mar 2024 02:18:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333087; cv=none; b=uQYrP660vCQ0yVw/aDs95DPxkQyZ4egSj1C2zpuA34MtvLlkUgpxJE/pIPbmMWZais9/Zx2uPkTUdUoDQMqoHfXSE5pNyMqYcfND8LqQohye+jGJrbH5H3/mVz37lW3u1YYtSQ4wi38R3MNHkucl0fDKvWIjlSl6Wz930oaN6yo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333087; c=relaxed/simple; bh=aRhkwlvzw+S3wIT3VW4mtSxCNmP3iuVaiUGPcoZs1qA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=RSZITnNQ6TPtFekU2FubFpMuUuf+IBZXCxVLNxJd68POeL9KR6nA9HZb/wsdAfT5lDhhDpwfTG5Pfq9MelddPcX/Uqxpndym7+FvRXrnb6N4JbpD3nSv/8vv7uBE3/DBaYbo26gfoiZ5eefFX+ivLdIMoTitYQ08zdCt0nXeV5U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=TUYBqLCZ; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="TUYBqLCZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333085; x=1742869085; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aRhkwlvzw+S3wIT3VW4mtSxCNmP3iuVaiUGPcoZs1qA=; b=TUYBqLCZRHBH9UlGd2QGZSFOBaAV/TtsyNCelVYflajUrMCZZA84S+Js dLRhJ4bdWIf9xemvCvo8l98i9v4UbCaTRzvmOLE4D7fNY0HetW8It6S7E MYBJya2gk0lhBIfysK2sWLkpHig22OLI8KXbm6i3yYbbcq5u6AvcM32la tVQg8sBstpjb0VnyMGjyzo//zB3MVFI8KfIxLnN8UuVPzE7z+UfsO5mc8 REIYn00jwWvzMhWNzjgJM1rpc5E4xKYDMazSoA8Ri8Tk2p35V3zEXKC2e CaQcLhY6NQ4GJHosFnZ0AOtDLfkNTlmybo6hgjuMPQLDx3LPFakwlfMLU g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271320" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271320" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353901" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:00 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 01/12] iommu/vt-d: Add cache tag assignment interface Date: Mon, 25 Mar 2024 10:16:54 +0800 Message-Id: <20240325021705.249769-2-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations. Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock. In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 25 +++++ drivers/iommu/intel/cache.c | 192 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 31 +++++- drivers/iommu/intel/nested.c | 21 +++- drivers/iommu/intel/svm.c | 12 ++- drivers/iommu/intel/Makefile | 2 +- 6 files changed, 274 insertions(+), 9 deletions(-) create mode 100644 drivers/iommu/intel/cache.c diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 404d2476a877..e3723b7a0b31 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -607,6 +607,9 @@ struct dmar_domain { struct list_head devices; /* all devices' list */ struct list_head dev_pasids; /* all attached pasids */ =20 + spinlock_t cache_lock; /* Protect the cache tag list */ + struct list_head cache_tags; /* Cache tag list */ + int iommu_superpage;/* Level of superpages supported: 0 =3D=3D 4KiB (no superpages), 1 =3D=3D 2MiB, 2 =3D=3D 1GiB, 3 =3D=3D 512GiB, 4 =3D=3D 1TiB */ @@ -1092,6 +1095,28 @@ struct iommu_domain *intel_nested_domain_alloc(struc= t iommu_domain *parent, const struct iommu_user_data *user_data); struct device *device_rbtree_find(struct intel_iommu *iommu, u16 rid); =20 +enum cache_tag_type { + CACHE_TAG_TYPE_IOTLB, + CACHE_TAG_TYPE_DEVTLB, + CACHE_TAG_TYPE_PARENT_IOTLB, + CACHE_TAG_TYPE_PARENT_DEVTLB, +}; + +struct cache_tag { + struct list_head node; + enum cache_tag_type type; + struct intel_iommu *iommu; + struct device *dev; + u16 domain_id; + ioasid_t pasid; + int users; +}; + +int cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid); +void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid); + #ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check(struct intel_iommu *iommu); int intel_svm_enable_prq(struct intel_iommu *iommu); diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c new file mode 100644 index 000000000000..5a4e12e494b6 --- /dev/null +++ b/drivers/iommu/intel/cache.c @@ -0,0 +1,192 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * cache.c - Intel VT-d cache invalidation + * + * Copyright (C) 2024 Intel Corporation + * + * Author: Lu Baolu + */ + +#define pr_fmt(fmt) "DMAR: " fmt + +#include +#include +#include +#include + +#include "iommu.h" +#include "pasid.h" + +/* Checks if an existing cache tag can be reused for a new association. */ +static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id, + struct intel_iommu *iommu, struct device *dev, + ioasid_t pasid, enum cache_tag_type type) +{ + if (tag->type !=3D type) + return false; + + if (tag->domain_id !=3D domain_id || tag->pasid !=3D pasid) + return false; + + if (type =3D=3D CACHE_TAG_TYPE_IOTLB) + return tag->iommu =3D=3D iommu; + + if (type =3D=3D CACHE_TAG_TYPE_DEVTLB) + return tag->dev =3D=3D dev; + + return false; +} + +/* Assign a cache tag with specified type to domain. */ +static int cache_tag_assign(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid, + enum cache_tag_type type) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct intel_iommu *iommu =3D info->iommu; + struct cache_tag *tag, *temp; + unsigned long flags; + + tag =3D kzalloc(sizeof(*tag), GFP_KERNEL); + if (!tag) + return -ENOMEM; + + tag->type =3D type; + tag->iommu =3D iommu; + tag->dev =3D dev; + tag->domain_id =3D did; + tag->pasid =3D pasid; + tag->users =3D 1; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(temp, &domain->cache_tags, node) { + if (cache_tag_reusable(temp, did, iommu, dev, pasid, type)) { + temp->users++; + spin_unlock_irqrestore(&domain->cache_lock, flags); + kfree(tag); + return 0; + } + } + list_add_tail(&tag->node, &domain->cache_tags); + spin_unlock_irqrestore(&domain->cache_lock, flags); + + return 0; +} + +/* Unassign a cache tag with specified type from domain. */ +static void cache_tag_unassign(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid, + enum cache_tag_type type) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct intel_iommu *iommu =3D info->iommu; + struct cache_tag *tag; + unsigned long flags; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + if (cache_tag_reusable(tag, did, iommu, dev, pasid, type)) { + if (--tag->users =3D=3D 0) { + list_del(&tag->node); + kfree(tag); + } + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +static int __cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + int ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_IOTLB); + if (ret || !info->ats_enabled) + return ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_DEVTLB); + if (ret) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_IOTLB); + + return ret; +} + +static void __cache_tag_unassign_domain(struct dmar_domain *domain, u16 di= d, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_IOTLB); + + if (info->ats_enabled) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_DEVTLB); +} + +static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u1= 6 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + int ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_I= OTLB); + if (ret || !info->ats_enabled) + return ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_D= EVTLB); + if (ret) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_IOTLB); + + return ret; +} + +static void __cache_tag_unassign_parent_domain(struct dmar_domain *domain,= u16 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_IOTLB); + + if (info->ats_enabled) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_TYPE_PARENT_DEVTLB= ); +} + +/* + * Assigns cache tags to a domain when it's associated with a device's + * PASID using a specific domain ID. + * + * On success (return value of 0), cache tags are created and added to the + * domain's cache tag list. On failure (negative return value), an error + * code is returned indicating the reason for the failure. + */ +int cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + int ret; + + ret =3D __cache_tag_assign_domain(domain, did, dev, pasid); + if (ret || domain->domain.type !=3D IOMMU_DOMAIN_NESTED) + return ret; + + ret =3D __cache_tag_assign_parent_domain(domain->s2_domain, did, dev, pas= id); + if (ret) + __cache_tag_unassign_domain(domain, did, dev, pasid); + + return ret; +} + +/* + * Removes the cache tags associated with a device's PASID when the domain= is + * detached from the device. + * + * The cache tags must be previously assigned to the domain by calling the + * assign interface. + */ +void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + __cache_tag_unassign_domain(domain, did, dev, pasid); + if (domain->domain.type =3D=3D IOMMU_DOMAIN_NESTED) + __cache_tag_unassign_parent_domain(domain->s2_domain, did, dev, pasid); +} diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 50eb9aed47cc..b4efbdedccce 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1750,7 +1750,9 @@ static struct dmar_domain *alloc_domain(unsigned int = type) domain->has_iotlb_device =3D false; INIT_LIST_HEAD(&domain->devices); INIT_LIST_HEAD(&domain->dev_pasids); + INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->lock); + spin_lock_init(&domain->cache_lock); xa_init(&domain->iommu_array); =20 return domain; @@ -2322,11 +2324,20 @@ static int dmar_domain_attach_device(struct dmar_do= main *domain, struct device_domain_info *info =3D dev_iommu_priv_get(dev); struct intel_iommu *iommu =3D info->iommu; unsigned long flags; + u16 did; int ret; =20 ret =3D domain_attach_iommu(domain, iommu); if (ret) return ret; + + did =3D domain_id_iommu(domain, iommu); + ret =3D cache_tag_assign_domain(domain, did, dev, IOMMU_NO_PASID); + if (ret) { + domain_detach_iommu(domain, iommu); + return ret; + } + info->domain =3D domain; spin_lock_irqsave(&domain->lock, flags); list_add(&info->link, &domain->devices); @@ -3798,6 +3809,7 @@ void device_block_translation(struct device *dev) struct device_domain_info *info =3D dev_iommu_priv_get(dev); struct intel_iommu *iommu =3D info->iommu; unsigned long flags; + u16 did; =20 iommu_disable_pci_caps(info); if (!dev_is_real_dma_subdevice(dev)) { @@ -3815,6 +3827,8 @@ void device_block_translation(struct device *dev) list_del(&info->link); spin_unlock_irqrestore(&info->domain->lock, flags); =20 + did =3D domain_id_iommu(info->domain, iommu); + cache_tag_unassign_domain(info->domain, did, dev, IOMMU_NO_PASID); domain_detach_iommu(info->domain, iommu); info->domain =3D NULL; } @@ -4595,10 +4609,12 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) struct dmar_domain *dmar_domain; struct iommu_domain *domain; unsigned long flags; + u16 did; =20 domain =3D iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON_ONCE(!domain)) goto out_tear_down; + dmar_domain =3D to_dmar_domain(domain); =20 /* * The SVA implementation needs to handle its own stuffs like the mm @@ -4607,10 +4623,11 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) */ if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { intel_svm_remove_dev_pasid(dev, pasid); + cache_tag_unassign_domain(dmar_domain, + FLPT_DEFAULT_DID, dev, pasid); goto out_tear_down; } =20 - dmar_domain =3D to_dmar_domain(domain); spin_lock_irqsave(&dmar_domain->lock, flags); list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) { if (curr->dev =3D=3D dev && curr->pasid =3D=3D pasid) { @@ -4622,6 +4639,8 @@ static void intel_iommu_remove_dev_pasid(struct devic= e *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 + did =3D domain_id_iommu(dmar_domain, iommu); + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); domain_detach_iommu(dmar_domain, iommu); intel_iommu_debugfs_remove_dev_pasid(dev_pasid); kfree(dev_pasid); @@ -4638,6 +4657,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, struct intel_iommu *iommu =3D info->iommu; struct dev_pasid_info *dev_pasid; unsigned long flags; + u16 did; int ret; =20 if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev)) @@ -4661,6 +4681,11 @@ static int intel_iommu_set_dev_pasid(struct iommu_do= main *domain, if (ret) goto out_free; =20 + did =3D domain_id_iommu(dmar_domain, iommu); + ret =3D cache_tag_assign_domain(dmar_domain, did, dev, pasid); + if (ret) + goto out_detach_iommu; + if (domain_type_is_si(dmar_domain)) ret =3D intel_pasid_setup_pass_through(iommu, dev, pasid); else if (dmar_domain->use_first_level) @@ -4670,7 +4695,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, ret =3D intel_pasid_setup_second_level(iommu, dmar_domain, dev, pasid); if (ret) - goto out_detach_iommu; + goto out_unassign_tag; =20 dev_pasid->dev =3D dev; dev_pasid->pasid =3D pasid; @@ -4682,6 +4707,8 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, intel_iommu_debugfs_create_dev_pasid(dev_pasid); =20 return 0; +out_unassign_tag: + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); out_detach_iommu: domain_detach_iommu(dmar_domain, iommu); out_free: diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index a7d68f3d518a..85c744099558 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -26,6 +26,7 @@ static int intel_nested_attach_dev(struct iommu_domain *d= omain, struct intel_iommu *iommu =3D info->iommu; unsigned long flags; int ret =3D 0; + u16 did; =20 if (info->domain) device_block_translation(dev); @@ -52,13 +53,15 @@ static int intel_nested_attach_dev(struct iommu_domain = *domain, return ret; } =20 + did =3D domain_id_iommu(dmar_domain, iommu); + ret =3D cache_tag_assign_domain(dmar_domain, did, dev, IOMMU_NO_PASID); + if (ret) + goto detach_iommu; + ret =3D intel_pasid_setup_nested(iommu, dev, IOMMU_NO_PASID, dmar_domain); - if (ret) { - domain_detach_iommu(dmar_domain, iommu); - dev_err_ratelimited(dev, "Failed to setup pasid entry\n"); - return ret; - } + if (ret) + goto unassign_tag; =20 info->domain =3D dmar_domain; spin_lock_irqsave(&dmar_domain->lock, flags); @@ -68,6 +71,12 @@ static int intel_nested_attach_dev(struct iommu_domain *= domain, domain_update_iotlb(dmar_domain); =20 return 0; +unassign_tag: + cache_tag_unassign_domain(dmar_domain, did, dev, IOMMU_NO_PASID); +detach_iommu: + domain_detach_iommu(dmar_domain, iommu); + + return ret; } =20 static void intel_nested_domain_free(struct iommu_domain *domain) @@ -206,7 +215,9 @@ struct iommu_domain *intel_nested_domain_alloc(struct i= ommu_domain *parent, domain->domain.type =3D IOMMU_DOMAIN_NESTED; INIT_LIST_HEAD(&domain->devices); INIT_LIST_HEAD(&domain->dev_pasids); + INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->lock); + spin_lock_init(&domain->cache_lock); xa_init(&domain->iommu_array); =20 spin_lock(&s2_domain->s1_lock); diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index c1bed89b1026..d706226e84ee 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -366,17 +366,25 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, sdev->qdep =3D 0; } =20 + ret =3D cache_tag_assign_domain(to_dmar_domain(domain), + FLPT_DEFAULT_DID, dev, pasid); + if (ret) + goto free_sdev; + /* Setup the pasid table: */ sflags =3D cpu_feature_enabled(X86_FEATURE_LA57) ? PASID_FLAG_FL5LP : 0; ret =3D intel_pasid_setup_first_level(iommu, dev, mm->pgd, pasid, FLPT_DEFAULT_DID, sflags); if (ret) - goto free_sdev; + goto unassign_tag; =20 list_add_rcu(&sdev->list, &svm->devs); =20 return 0; =20 +unassign_tag: + cache_tag_unassign_domain(to_dmar_domain(domain), + FLPT_DEFAULT_DID, dev, pasid); free_sdev: kfree(sdev); free_svm: @@ -795,6 +803,8 @@ struct iommu_domain *intel_svm_domain_alloc(void) if (!domain) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; + INIT_LIST_HEAD(&domain->cache_tags); + spin_lock_init(&domain->cache_lock); =20 return &domain->domain; } diff --git a/drivers/iommu/intel/Makefile b/drivers/iommu/intel/Makefile index 5402b699a122..c8beb0281559 100644 --- a/drivers/iommu/intel/Makefile +++ b/drivers/iommu/intel/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DMAR_TABLE) +=3D dmar.o -obj-$(CONFIG_INTEL_IOMMU) +=3D iommu.o pasid.o nested.o +obj-$(CONFIG_INTEL_IOMMU) +=3D iommu.o pasid.o nested.o cache.o obj-$(CONFIG_DMAR_TABLE) +=3D trace.o cap_audit.o obj-$(CONFIG_DMAR_PERF) +=3D perf.o obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) +=3D debugfs.o --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 600D8153837 for ; Mon, 25 Mar 2024 02:18:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333088; cv=none; b=MXE+ig+lzX6sxMubIT8rRb3owLBWoO6Z2P2/tobmUjA9dWpmn2vfDMLy5JJqxaw7xumIWbtd+1ufu5wOqDIcQkw/5P0KNzbuMo3voYAXEgNFdJvxqkO+2AkTwP8vKamvLBeKLR+OE8tuyrnUi3S5xFra/+v70XbKrm+kluKjDw4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333088; c=relaxed/simple; bh=ftJbBlWpjwZNe/1FS4EnOcju++knLjq8w7xu9aQ/9RQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=l0+4+WdSaMy4N8WQ7+KJBokkBCt6Sh11pmVeKSPYZqa6yFogHjEu68zz2BUCuMJpAYVbJoxHoEjnNcD2jkbzxHsQzz3fIC1AHIe1HHxMNAr5KrRCIg97ZBDHZ4ACs3TI86qKOElJw9R7jMToKnl8j+kHR5ZnhDY8b88+Ra2BbdE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=c3gEufIu; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="c3gEufIu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333086; x=1742869086; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ftJbBlWpjwZNe/1FS4EnOcju++knLjq8w7xu9aQ/9RQ=; b=c3gEufIubf4BjdXjQNFdsK9N4sY+8zq5DHsXMum4VcDOirU0LbTsK7s4 BsvTkhgGd/xhb1YXXzSF1I0Mg44NStUdTRY1x5nGKOpNaW6kW4PPq4tU1 jd/ibd0BWGGZuvJvN65X5VcBuvbtXJYKy2XyBU0gIq6EgJ5+hWL3TS7Cc vAE7ibe5m2G9DHIDMr4Vs57yJz1md5etf3RptZDnkP1w5r5sgNDhOWJK5 zm1s1OuVhVAtn4q+criX+6HwbJFJIEhD8ZGc3zvW3J8AgbS0foBnnRsa5 0BriarfXN6kUMHVet0qVuWlnCp3mZLBcEuq+285a84fHRoE67upTez36D Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271326" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271326" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:06 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353914" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:03 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 02/12] iommu/vt-d: Add cache tag invalidation helpers Date: Mon, 25 Mar 2024 10:16:55 +0800 Message-Id: <20240325021705.249769-3-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add several helpers to invalidate the caches after mappings in the affected domain are changed. - cache_tag_flush_range() invalidates a range of caches after mappings within this range are changed. It uses the page-selective cache invalidation methods. - cache_tag_flush_all() invalidates all caches tagged by a domain ID. It uses the domain-selective cache invalidation methods. - cache_tag_flush_cm_range() invalidates a range of caches if IOMMU is working in the caching mode and second-only translation is used for the affected domain. It is called when non-present to present mappings are created. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 7 ++ drivers/iommu/intel/cache.c | 189 ++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 5 - 3 files changed, 196 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index e3723b7a0b31..d05fa0122d65 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -35,6 +35,8 @@ #define VTD_PAGE_MASK (((u64)-1) << VTD_PAGE_SHIFT) #define VTD_PAGE_ALIGN(addr) (((addr) + VTD_PAGE_SIZE - 1) & VTD_PAGE_MASK) =20 +#define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) + #define VTD_STRIDE_SHIFT (9) #define VTD_STRIDE_MASK (((u64)-1) << VTD_STRIDE_SHIFT) =20 @@ -1116,6 +1118,11 @@ int cache_tag_assign_domain(struct dmar_domain *doma= in, u16 did, struct device *dev, ioasid_t pasid); void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, struct device *dev, ioasid_t pasid); +void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, + unsigned long end, int ih); +void cache_tag_flush_all(struct dmar_domain *domain); +void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long st= art, + unsigned long end); =20 #ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check(struct intel_iommu *iommu); diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 5a4e12e494b6..4c245d39faf2 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -12,6 +12,7 @@ #include #include #include +#include #include =20 #include "iommu.h" @@ -190,3 +191,191 @@ void cache_tag_unassign_domain(struct dmar_domain *do= main, u16 did, if (domain->domain.type =3D=3D IOMMU_DOMAIN_NESTED) __cache_tag_unassign_parent_domain(domain->s2_domain, did, dev, pasid); } + +static unsigned long calculate_psi_aligned_address(unsigned long start, + unsigned long end, + unsigned long *_pages, + unsigned long *_mask) +{ + unsigned long pages =3D (end - start + VTD_PAGE_SIZE - 1) >> VTD_PAGE_SHI= FT; + unsigned long aligned_pages =3D __roundup_pow_of_two(pages); + unsigned long bitmask =3D aligned_pages - 1; + unsigned long mask =3D ilog2(aligned_pages); + unsigned long pfn =3D IOVA_PFN(start); + + /* + * PSI masks the low order bits of the base address. If the + * address isn't aligned to the mask, then compute a mask value + * needed to ensure the target range is flushed. + */ + if (unlikely(bitmask & pfn)) { + unsigned long end_pfn =3D pfn + pages - 1, shared_bits; + + /* + * Since end_pfn <=3D pfn + bitmask, the only way bits + * higher than bitmask can differ in pfn and end_pfn is + * by carrying. This means after masking out bitmask, + * high bits starting with the first set bit in + * shared_bits are all equal in both pfn and end_pfn. + */ + shared_bits =3D ~(pfn ^ end_pfn) & ~bitmask; + mask =3D shared_bits ? __ffs(shared_bits) : BITS_PER_LONG; + } + + *_pages =3D aligned_pages; + *_mask =3D mask; + + return ALIGN_DOWN(start, VTD_PAGE_SIZE); +} + +/* + * Invalidates a range of IOVA from @start (inclusive) to @end (inclusive) + * when the memory mappings in the target domain have been modified. + */ +void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, + unsigned long end, int ih) +{ + unsigned long pages, mask, addr; + struct cache_tag *tag; + unsigned long flags; + + addr =3D calculate_psi_aligned_address(start, end, &pages, &mask); + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct device_domain_info *info =3D dev_iommu_priv_get(tag->dev); + struct intel_iommu *iommu =3D tag->iommu; + u16 sid =3D PCI_DEVID(info->bus, info->devfn); + + switch (tag->type) { + case CACHE_TAG_TYPE_IOTLB: + case CACHE_TAG_TYPE_PARENT_IOTLB: + if (domain->use_first_level) { + qi_flush_piotlb(iommu, tag->domain_id, + tag->pasid, addr, pages, ih); + } else { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + addr | ih, mask, + DMA_TLB_PSI_FLUSH); + } + break; + case CACHE_TAG_TYPE_DEVTLB: + if (tag->pasid =3D=3D IOMMU_NO_PASID) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr, mask); + else + qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, + tag->pasid, info->ats_qdep, + addr, mask); + + quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); + break; + case CACHE_TAG_TYPE_PARENT_DEVTLB: + /* + * Address translation cache in device side caches the + * result of nested translation. There is no easy way + * to identify the exact set of nested translations + * affected by a change in S2. So just flush the entire + * device cache. + */ + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, + IOMMU_NO_PASID, info->ats_qdep); + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +/* + * Invalidates all ranges of IOVA when the memory mappings in the target + * domain have been modified. + */ +void cache_tag_flush_all(struct dmar_domain *domain) +{ + struct cache_tag *tag; + unsigned long flags; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct device_domain_info *info =3D dev_iommu_priv_get(tag->dev); + struct intel_iommu *iommu =3D tag->iommu; + u16 sid =3D PCI_DEVID(info->bus, info->devfn); + + switch (tag->type) { + case CACHE_TAG_TYPE_IOTLB: + case CACHE_TAG_TYPE_PARENT_IOTLB: + if (domain->use_first_level) + qi_flush_piotlb(iommu, tag->domain_id, + tag->pasid, 0, -1, 0); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + break; + case CACHE_TAG_TYPE_DEVTLB: + case CACHE_TAG_TYPE_PARENT_DEVTLB: + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, + IOMMU_NO_PASID, info->ats_qdep); + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +/* + * Invalidate a range of IOVA when IOMMU is in caching mode and new mappin= gs + * are added to the target domain. + */ +void cache_tag_flush_cm_range(struct dmar_domain *domain, unsigned long st= art, + unsigned long end) +{ + unsigned long pages, mask, addr; + struct cache_tag *tag; + unsigned long flags; + + addr =3D calculate_psi_aligned_address(start, end, &pages, &mask); + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct intel_iommu *iommu =3D tag->iommu; + + /* + * When IOMMU is enabled in caching mode some non-present + * mappings may be cached by the IOTLB if it uses second- + * only translation. + */ + if (!cap_caching_mode(iommu->cap) || domain->use_first_level) { + iommu_flush_write_buffer(iommu); + continue; + } + + if (tag->type =3D=3D CACHE_TAG_TYPE_IOTLB || + tag->type =3D=3D CACHE_TAG_TYPE_PARENT_IOTLB) { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + addr, mask, + DMA_TLB_PSI_FLUSH); + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index b4efbdedccce..93e4422c9b10 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -54,11 +54,6 @@ __DOMAIN_MAX_PFN(gaw), (unsigned long)-1)) #define DOMAIN_MAX_ADDR(gaw) (((uint64_t)__DOMAIN_MAX_PFN(gaw)) << VTD_PAG= E_SHIFT) =20 -/* IO virtual address start page frame number */ -#define IOVA_START_PFN (1) - -#define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) - static void __init check_tylersburg_isoch(void); static int rwbf_quirk; =20 --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B698814AD02 for ; Mon, 25 Mar 2024 02:18:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333094; cv=none; b=mx+gTjiDyQNw6KjYk6Ksa/1nI1BQ+Carj2s4fJYH1md23puh6nG1WSKMp8+iQo/sSVpLgnHco39h/0Piqr7M7Jk23xr83ieoD7GFVYqIrZ5wXh3k0FGhoSaTr72lv8WY6/vpxR4jLcSVIN3ZKGUXAjdzEBD3yTEx/wwn4jLYeKE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333094; c=relaxed/simple; bh=/yd76U2C/MzpuU1HznKVypbC7FLgM1XIezJ+dva0SCI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=W0W/i4Sev631nFjZuZw2bRAds867LM4sWvGNCAjNYegeR2YRpu8ATtL3lV5jCFQ6H2L1gACkr/0vQ7FNQ11rwOBPxleSeZmHA3QNOTZxg2Z2hLt2UiZUvKubJnNIH2q69SxWD2qJcHXFQ+ha+wozA8Vj8JIw0Rms0ZOCZS+kM6s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nBstBUrp; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nBstBUrp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333093; x=1742869093; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=/yd76U2C/MzpuU1HznKVypbC7FLgM1XIezJ+dva0SCI=; b=nBstBUrph536P5cgPyz3cR5MDCXDM3R/2b9Rh/9gqqGWNYwFZ+0/5Zyj eYJzqG21JD1KAF22pFZb0738/GcOmFr6Iseh0qIKHOh54gNHpOeiUG9cX obkQ/phcGP8EqiGzaxfqVL0tDDc/lhLjbZYo68sYWcE0IKd2p/KinlCCc 4QNlXHKyJChjxvkUijMf4JaOA2RMqubOXs3jATn2rfFzQR+1f56UGkSTD rhTfpypM9dUMRhb3+q583tenlUC/OvTv0ctITElP5I9Crdv3MoL6lRy8O Q1zvXiRWIDY+VxusQ5yNCBetc5+NURER1yP+IcCuYG74L+yw4dWtgG0Ya Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271335" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271335" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353931" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:06 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 03/12] iommu/vt-d: Add trace events for cache tag interface Date: Mon, 25 Mar 2024 10:16:56 +0800 Message-Id: <20240325021705.249769-4-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add trace events for cache tag assign/unassign/flush operations and trace the events in the interfaces. These trace events will improve debugging capabilities by providing detailed information about cache tag activity. A sample of the traced messages looks like below [messages have been stripped and wrapped to make the line short]. cache_tag_assign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1 cache_tag_assign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1 cache_tag_flush_all: dmar6/0000:8a:00.0 type iotlb did 7 pasid 0 ref 1 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeab00000-0xeab1afff] addr 0xeab00000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeab20000-0xeab31fff] addr 0xeab20000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeaa40000-0xeaa51fff] addr 0xeaa40000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0x98de0000-0x98de4fff] addr 0x98de0000 pages 0x8 mask 0x3 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xe9828000-0xe9828fff] addr 0xe9828000 pages 0x1 mask 0x0 cache_tag_unassign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1 cache_tag_unassign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1 Signed-off-by: Lu Baolu --- drivers/iommu/intel/trace.h | 97 +++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/cache.c | 8 +++ 2 files changed, 105 insertions(+) diff --git a/drivers/iommu/intel/trace.h b/drivers/iommu/intel/trace.h index 93d96f93a89b..b8c2ebace8af 100644 --- a/drivers/iommu/intel/trace.h +++ b/drivers/iommu/intel/trace.h @@ -89,6 +89,103 @@ TRACE_EVENT(prq_report, __entry->dw1, __entry->dw2, __entry->dw3) ) ); + +DECLARE_EVENT_CLASS(cache_tag_log, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag), + TP_STRUCT__entry( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) + __field(u32, users) + ), + TP_fast_assign( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type =3D tag->type; + __entry->domain_id =3D tag->domain_id; + __entry->pasid =3D tag->pasid; + __entry->users =3D tag->users; + ), + TP_printk("%s/%s type %s did %d pasid %d ref %d", + __get_str(iommu), __get_str(dev), + __print_symbolic(__entry->type, + { CACHE_TAG_TYPE_IOTLB, "iotlb" }, + { CACHE_TAG_TYPE_DEVTLB, "devtlb" }, + { CACHE_TAG_TYPE_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_TYPE_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->pasid, __entry->users + ) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_assign, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_unassign, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_flush_all, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DECLARE_EVENT_CLASS(cache_tag_flush, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask), + TP_STRUCT__entry( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) + __field(unsigned long, start) + __field(unsigned long, end) + __field(unsigned long, addr) + __field(unsigned long, pages) + __field(unsigned long, mask) + ), + TP_fast_assign( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type =3D tag->type; + __entry->domain_id =3D tag->domain_id; + __entry->pasid =3D tag->pasid; + __entry->start =3D start; + __entry->end =3D end; + __entry->addr =3D addr; + __entry->pages =3D pages; + __entry->mask =3D mask; + ), + TP_printk("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx pages 0x%lx = mask 0x%lx", + __get_str(iommu), __get_str(dev), __entry->pasid, + __print_symbolic(__entry->type, + { CACHE_TAG_TYPE_IOTLB, "iotlb" }, + { CACHE_TAG_TYPE_DEVTLB, "devtlb" }, + { CACHE_TAG_TYPE_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_TYPE_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->start, __entry->end, + __entry->addr, __entry->pages, __entry->mask + ) +); + +DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask) +); + +DEFINE_EVENT(cache_tag_flush, cache_tag_flush_cm_range, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask) +); #endif /* _TRACE_INTEL_IOMMU_H */ =20 /* This part must be outside protection */ diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 4c245d39faf2..b35ef2ee2aca 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -17,6 +17,7 @@ =20 #include "iommu.h" #include "pasid.h" +#include "trace.h" =20 /* Checks if an existing cache tag can be reused for a new association. */ static bool cache_tag_reusable(struct cache_tag *tag, u16 domain_id, @@ -65,11 +66,13 @@ static int cache_tag_assign(struct dmar_domain *domain,= u16 did, temp->users++; spin_unlock_irqrestore(&domain->cache_lock, flags); kfree(tag); + trace_cache_tag_assign(temp); return 0; } } list_add_tail(&tag->node, &domain->cache_tags); spin_unlock_irqrestore(&domain->cache_lock, flags); + trace_cache_tag_assign(tag); =20 return 0; } @@ -87,6 +90,7 @@ static void cache_tag_unassign(struct dmar_domain *domain= , u16 did, spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { if (cache_tag_reusable(tag, did, iommu, dev, pasid, type)) { + trace_cache_tag_unassign(tag); if (--tag->users =3D=3D 0) { list_del(&tag->node); kfree(tag); @@ -293,6 +297,8 @@ void cache_tag_flush_range(struct dmar_domain *domain, = unsigned long start, IOMMU_NO_PASID, info->ats_qdep); break; } + + trace_cache_tag_flush_range(tag, start, end, addr, pages, mask); } spin_unlock_irqrestore(&domain->cache_lock, flags); } @@ -330,6 +336,7 @@ void cache_tag_flush_all(struct dmar_domain *domain) IOMMU_NO_PASID, info->ats_qdep); break; } + trace_cache_tag_flush_all(tag); } spin_unlock_irqrestore(&domain->cache_lock, flags); } @@ -376,6 +383,7 @@ void cache_tag_flush_cm_range(struct dmar_domain *domai= n, unsigned long start, addr, mask, DMA_TLB_PSI_FLUSH); } + trace_cache_tag_flush_cm_range(tag, start, end, addr, pages, mask); } spin_unlock_irqrestore(&domain->cache_lock, flags); } --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 269E214AD1A for ; Mon, 25 Mar 2024 02:18:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333098; cv=none; b=GEK30mrrSe4zyGC+2FaeSisCDIos0STy1Wf+86yV/wElUS+hOnLyO0pb5lkUQD/xjhsh2qMB+3ZCX2AN4t1xjsjWghbxhMWC5ZWLFJ0oVnW2mG1NnU/e6pb1fZyVpBwwwkUgyHjz0tmF7ZJzB0t0wfA9NvDhxolkAx9dsm0zWjU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333098; c=relaxed/simple; bh=X91sn3qLI1hNjQ+x53CENgbWg0CtCQsOM/1KIO/sS8s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=h+35azmh0aN4QcVyXGuVgotU0krRP8w3MYUeU+R33ZdTx882kOyvDBS6juQw/urNfpwUeMDXhl66yDjYuWLDQAHA+sCh7B1FJefxJcTZzDqPL71CD802QjFowixE9iukek9KJJGaa6U0e8pCwjxY/cF7s8ibipjdxg9gg1rFMss= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=W5q4+HFk; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="W5q4+HFk" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333096; x=1742869096; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=X91sn3qLI1hNjQ+x53CENgbWg0CtCQsOM/1KIO/sS8s=; b=W5q4+HFk01ZhavriYnUlUFGCWQ9ILaRRfwU70cJdU8EMxauNufDU+6DD YgB+5VKka8FKhBq0sM3Qwo1GUBDLuG0rc+9Ksct2zksej/u2kYtbjvn0I 9PZZ98lF2CH34uMkW1HctXBRhkVoGeorUGeDy4VpSUdFmQt8qpDeYPf+d 9CMVQbqZ0EkajHJzQsr1NUYoBvIldPmo8bd0NNVMOxxcWeeIECbSD2S2i 5svKOu2I2SmXK8GCScuCOLF5aYlxdIZ2DR0qmFcqCcPNIExfwBWiGcMMQ iKz+HFKAKo6IMJU0MG5BorZ4tqbYlnnyHU1gSaID9IEQ3DNn/OA8uQAMJ g==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271347" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271347" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353941" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:09 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all Date: Mon, 25 Mar 2024 10:16:57 +0800 Message-Id: <20240325021705.249769-5-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 93e4422c9b10..4ce98f23917c 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1560,26 +1560,7 @@ static void parent_domain_flush(struct dmar_domain *= domain, =20 static void intel_flush_iotlb_all(struct iommu_domain *domain) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - struct iommu_domain_info *info; - unsigned long idx; - - xa_for_each(&dmar_domain->iommu_array, idx, info) { - struct intel_iommu *iommu =3D info->iommu; - u16 did =3D domain_id_iommu(dmar_domain, iommu); - - if (dmar_domain->use_first_level) - domain_flush_pasid_iotlb(iommu, dmar_domain, 0, -1, 0); - else - iommu->flush.flush_iotlb(iommu, did, 0, 0, - DMA_TLB_DSI_FLUSH); - - if (!cap_caching_mode(iommu->cap)) - iommu_flush_dev_iotlb(dmar_domain, 0, MAX_AGAW_PFN_WIDTH); - } - - if (dmar_domain->nested_parent) - parent_domain_flush(dmar_domain, 0, -1, 0); + cache_tag_flush_all(to_dmar_domain(domain)); } =20 static void iommu_disable_protect_mem_regions(struct intel_iommu *iommu) --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04CE914AD0C for ; Mon, 25 Mar 2024 02:18:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333096; cv=none; b=pYC+J14MtZ9QOb5+LsnRXydhrOkb82vOCGF3kDefVefuubr6XNncuPcbo5W+LXzm53UmaqtZxew+ZUIysOGx0VhEo0rJcytp1RFmoFfQl99sKFB/OO+I5GsR8oP5ywY0yTT9ZeqGSw4vQRBIM81wOd+YGAuXO+e0RLkmO5KE0P4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333096; c=relaxed/simple; bh=+OAkQ+xgV6iFnsM2Piz46SFV5ZnSYjQLnEdHmWb5riU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WCnNwRRG8Ii6U9Ne+GzMUSHRNUfd/dFxxZJlMra8WKzFcy1iIt7um+pBBwI69YzxVI+VY3iK57g+sWbuxqcSLBih09l5ihtfwRMqYeK9ngcxWTQDbS4c3ptqc+/5/4NLkydw0EjKoEwmxvX565zVRPm9mXXFYUGYlSF1RPZiH98= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=IObLNW1W; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="IObLNW1W" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333095; x=1742869095; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+OAkQ+xgV6iFnsM2Piz46SFV5ZnSYjQLnEdHmWb5riU=; b=IObLNW1WvBAJ+42wjK5KEqoPieDvcdaCnOSKKoWd1KcRHnBxvDXkc8P5 pjBM1LARWh8PEO1qiF7opp/eeyGSpzNeYn3mC7gVSvB68cJZxdOEms9+r 2Im/AHKV5aR3Z4067EbD3fVKbCnCaQEsTeJa8TSe3twPkYsfyEAGwLCPX bjvDHP5as41xu6CqKg86ZcczRUdNChkPy4eoyFPRUx9AHuox6MIM5Dqo/ YFAVbFlLWS8Y0O3MVYqyc0e+FJFBdvNKrzb0ac4DN8OPkLjBP5Di+Gakp pWv8CKGdcwR5fHwYORp49Ju78z1xyse7FzNdGxVPQFdFpYnZmC1HcvKwg A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271356" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271356" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353954" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:12 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 05/12] iommu/vt-d: Use cache_tag_flush_range() in tlb_sync Date: Mon, 25 Mar 2024 10:16:58 +0800 Message-Id: <20240325021705.249769-6-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The tlb_sync callback is called by the iommu core to flush a range of caches for the affected domain. Use cache_tag_flush_range() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 4ce98f23917c..1c03d2dafb9d 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4112,25 +4112,8 @@ static size_t intel_iommu_unmap_pages(struct iommu_d= omain *domain, static void intel_iommu_tlb_sync(struct iommu_domain *domain, struct iommu_iotlb_gather *gather) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - unsigned long iova_pfn =3D IOVA_PFN(gather->start); - size_t size =3D gather->end - gather->start; - struct iommu_domain_info *info; - unsigned long start_pfn; - unsigned long nrpages; - unsigned long i; - - nrpages =3D aligned_nrpages(gather->start, size); - start_pfn =3D mm_to_dma_pfn_start(iova_pfn); - - xa_for_each(&dmar_domain->iommu_array, i, info) - iommu_flush_iotlb_psi(info->iommu, dmar_domain, - start_pfn, nrpages, - list_empty(&gather->freelist), 0); - - if (dmar_domain->nested_parent) - parent_domain_flush(dmar_domain, start_pfn, nrpages, - list_empty(&gather->freelist)); + cache_tag_flush_range(to_dmar_domain(domain), gather->start, + gather->end, list_empty(&gather->freelist)); put_pages_list(&gather->freelist); } =20 --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC97715538B for ; Mon, 25 Mar 2024 02:18:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333102; cv=none; b=MKKLUWhe1yzcmRLGBWMqO9mZmYW6c56YJYzAe6+cL1/wWUhIB7wobCh+HYVzh9/sV4yqxt0jdJjM7KBoTCrdrFobysCgh5iME7+RMBkWBmkWP9oSwGx6pZ33AwUiiQS3AGpqIQac7RZk+EWdimd8EiJ2jUbRfoyN6GAjBxy0ha8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333102; c=relaxed/simple; bh=glq+1FyhuB1qxaNB4WrfvXWGDmm2KzfQ3M2Hznkp84E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mMVPv35wTkNMrT1u9PZumqDOjEm+N0A7tCpOnZPWWMUyATnHkKclH9Ffd/iXf0hfnxBego0kIZtQa0bfLRLzKiohodkV3uITfhEl2vBpinb55E4DDGZmRGqTB5fgcH0P6eBpIXXnqpP9NuMqOFLOxaIdAw3SZJKbMoH2HxqbzEg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=nPY6XxUU; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="nPY6XxUU" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333100; x=1742869100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=glq+1FyhuB1qxaNB4WrfvXWGDmm2KzfQ3M2Hznkp84E=; b=nPY6XxUULEQTH5y9W+lDfjExHwBk+iTBgga1v2Wo98T/3j4OevjObqkC YRUHBhF+VA+Td5H6rQxMPZK8sNCFendNfAo6LNMFmr4U/R0LrVPvgMzZd 7kGDUwm3vE/WBEhx81GkH43RxRbkJKabjssg94hlpfYF2N7d6BZfMLE5e WnOGc824lopMuLl+y50Y+X2j0FEnx7He/X24My91DD61v1UTXMPWgmLLV JiYUbHH1BkCslQDOKqCnfRHULRxglmGuL2Sp9CfVF3CHkfurCQzfyCL79 Kqh5fwxlyCMfmE0sFIB0EGphWAlbMN3rEAv2HQVJUYcf7GxeQfH1Jiimn w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271372" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271372" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353962" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:14 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 06/12] iommu/vt-d: Use cache_tag_flush_cm_range() in iotlb_sync_map Date: Mon, 25 Mar 2024 10:16:59 +0800 Message-Id: <20240325021705.249769-7-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_cm_range() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 1c03d2dafb9d..2dcab1e5cd4d 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1504,20 +1504,6 @@ static void iommu_flush_iotlb_psi(struct intel_iommu= *iommu, iommu_flush_dev_iotlb(domain, addr, mask); } =20 -/* Notification for newly created mappings */ -static void __mapping_notify_one(struct intel_iommu *iommu, struct dmar_do= main *domain, - unsigned long pfn, unsigned int pages) -{ - /* - * It's a non-present to present mapping. Only flush if caching mode - * and second level. - */ - if (cap_caching_mode(iommu->cap) && !domain->use_first_level) - iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1); - else - iommu_flush_write_buffer(iommu); -} - /* * Flush the relevant caches in nested translation if the domain * also serves as a parent @@ -4549,14 +4535,8 @@ static bool risky_device(struct pci_dev *pdev) static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain, unsigned long iova, size_t size) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - unsigned long pages =3D aligned_nrpages(iova, size); - unsigned long pfn =3D iova >> VTD_PAGE_SHIFT; - struct iommu_domain_info *info; - unsigned long i; + cache_tag_flush_cm_range(to_dmar_domain(domain), iova, iova + size - 1); =20 - xa_for_each(&dmar_domain->iommu_array, i, info) - __mapping_notify_one(info->iommu, dmar_domain, pfn, pages); return 0; } =20 --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 581DB15538E for ; Mon, 25 Mar 2024 02:18:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333104; cv=none; b=mvoAWuHpaVdmVgY3qQCWrV3AAQPnySQzdJGkC3xgG2iX7ISesQDov9P1TdG/JukFLj52BtlGKrPUYwMCFLtgTTnkLe+dXwlfJuFFPtUHVG380CyBJk9+PxAqm4z1PkgmdfOzSU3qIYTrJQbFu2Q/6GFIvsBjXD1facdEU6ZEzrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333104; c=relaxed/simple; bh=0tIbvWia2r+172vtYeoeqKPSZ4y6wUN/0xRGVLV++4s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I2rt9sAsWqUydVh3xmMwc5uGKcJaOyD5HTTUnahFvMciUig4KSiAhLKknjOSimN+CcLcnHEKwdZSOcJcw2QkGm/YT/Yw7TQFbEy486253TqLb+tRaZBiqmFa8QgM/F2Km6Vpif/eqTtBSZFvhmk4J/Z17pBxsdk3UBVrDnyA5dg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ZsmUHEbP; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ZsmUHEbP" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333102; x=1742869102; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=0tIbvWia2r+172vtYeoeqKPSZ4y6wUN/0xRGVLV++4s=; b=ZsmUHEbPUtmHlLJTw8bICxdK9uJhSQ0SviwXMhMEjm2jrp4rvm+GC+tR CVHVBJc3xABDO8mJLJi/pjbIS2ekPgzJ7trAb1lL2ClNLAcyuVSf77NIC 0CTy6MhfRzOytiglLcOpuaCwcG0jwG2gjW9eWpKoyLbIHCetF1VXgWPVy C1kdyfPQ7BuhQ6Jyg3/ZiUgFvv80pRFXjBP6UO8AfWWNCGXq7ZoaHYxqM CD8q28KUcTdyu+Jppw6JFsXt4k74kO486HqtFiGx6jpawmwRExgOwAW1W ggKdeVJQbtap6UF7XRzhCAM4y8cgCghfnsjszSKyBPl4A3LLOGgKrHt3W w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271391" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271391" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353970" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:17 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi() Date: Mon, 25 Mar 2024 10:17:00 +0800 Message-Id: <20240325021705.249769-8-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain. Clean up iommu_flush_iotlb_psi() after the last call site is removed. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 175 +----------------------------------- 1 file changed, 2 insertions(+), 173 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 2dcab1e5cd4d..6e019297843b 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1389,161 +1389,6 @@ static void __iommu_flush_dev_iotlb(struct device_d= omain_info *info, quirk_extra_dev_tlb_flush(info, addr, mask, IOMMU_NO_PASID, qdep); } =20 -static void iommu_flush_dev_iotlb(struct dmar_domain *domain, - u64 addr, unsigned mask) -{ - struct dev_pasid_info *dev_pasid; - struct device_domain_info *info; - unsigned long flags; - - if (!domain->has_iotlb_device) - return; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(info, &domain->devices, link) - __iommu_flush_dev_iotlb(info, addr, mask); - - list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { - info =3D dev_iommu_priv_get(dev_pasid->dev); - - if (!info->ats_enabled) - continue; - - qi_flush_dev_iotlb_pasid(info->iommu, - PCI_DEVID(info->bus, info->devfn), - info->pfsid, dev_pasid->pasid, - info->ats_qdep, addr, - mask); - } - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void domain_flush_pasid_iotlb(struct intel_iommu *iommu, - struct dmar_domain *domain, u64 addr, - unsigned long npages, bool ih) -{ - u16 did =3D domain_id_iommu(domain, iommu); - struct dev_pasid_info *dev_pasid; - unsigned long flags; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) - qi_flush_piotlb(iommu, did, dev_pasid->pasid, addr, npages, ih); - - if (!list_empty(&domain->devices)) - qi_flush_piotlb(iommu, did, IOMMU_NO_PASID, addr, npages, ih); - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void __iommu_flush_iotlb_psi(struct intel_iommu *iommu, u16 did, - unsigned long pfn, unsigned int pages, - int ih) -{ - unsigned int aligned_pages =3D __roundup_pow_of_two(pages); - unsigned long bitmask =3D aligned_pages - 1; - unsigned int mask =3D ilog2(aligned_pages); - u64 addr =3D (u64)pfn << VTD_PAGE_SHIFT; - - /* - * PSI masks the low order bits of the base address. If the - * address isn't aligned to the mask, then compute a mask value - * needed to ensure the target range is flushed. - */ - if (unlikely(bitmask & pfn)) { - unsigned long end_pfn =3D pfn + pages - 1, shared_bits; - - /* - * Since end_pfn <=3D pfn + bitmask, the only way bits - * higher than bitmask can differ in pfn and end_pfn is - * by carrying. This means after masking out bitmask, - * high bits starting with the first set bit in - * shared_bits are all equal in both pfn and end_pfn. - */ - shared_bits =3D ~(pfn ^ end_pfn) & ~bitmask; - mask =3D shared_bits ? __ffs(shared_bits) : BITS_PER_LONG; - } - - /* - * Fallback to domain selective flush if no PSI support or - * the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, did, 0, 0, - DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, did, addr | ih, mask, - DMA_TLB_PSI_FLUSH); -} - -static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, - struct dmar_domain *domain, - unsigned long pfn, unsigned int pages, - int ih, int map) -{ - unsigned int aligned_pages =3D __roundup_pow_of_two(pages); - unsigned int mask =3D ilog2(aligned_pages); - uint64_t addr =3D (uint64_t)pfn << VTD_PAGE_SHIFT; - u16 did =3D domain_id_iommu(domain, iommu); - - if (WARN_ON(!pages)) - return; - - if (ih) - ih =3D 1 << 6; - - if (domain->use_first_level) - domain_flush_pasid_iotlb(iommu, domain, addr, pages, ih); - else - __iommu_flush_iotlb_psi(iommu, did, pfn, pages, ih); - - /* - * In caching mode, changes of pages from non-present to present require - * flush. However, device IOTLB doesn't need to be flushed in this case. - */ - if (!cap_caching_mode(iommu->cap) || !map) - iommu_flush_dev_iotlb(domain, addr, mask); -} - -/* - * Flush the relevant caches in nested translation if the domain - * also serves as a parent - */ -static void parent_domain_flush(struct dmar_domain *domain, - unsigned long pfn, - unsigned long pages, int ih) -{ - struct dmar_domain *s1_domain; - - spin_lock(&domain->s1_lock); - list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) { - struct device_domain_info *device_info; - struct iommu_domain_info *info; - unsigned long flags; - unsigned long i; - - xa_for_each(&s1_domain->iommu_array, i, info) - __iommu_flush_iotlb_psi(info->iommu, info->did, - pfn, pages, ih); - - if (!s1_domain->has_iotlb_device) - continue; - - spin_lock_irqsave(&s1_domain->lock, flags); - list_for_each_entry(device_info, &s1_domain->devices, link) - /* - * Address translation cache in device side caches the - * result of nested translation. There is no easy way - * to identify the exact set of nested translations - * affected by a change in S2. So just flush the entire - * device cache. - */ - __iommu_flush_dev_iotlb(device_info, 0, - MAX_AGAW_PFN_WIDTH); - spin_unlock_irqrestore(&s1_domain->lock, flags); - } - spin_unlock(&domain->s1_lock); -} - static void intel_flush_iotlb_all(struct iommu_domain *domain) { cache_tag_flush_all(to_dmar_domain(domain)); @@ -1995,9 +1840,7 @@ static void switch_to_super_page(struct dmar_domain *= domain, unsigned long end_pfn, int level) { unsigned long lvl_pages =3D lvl_to_nr_pages(level); - struct iommu_domain_info *info; struct dma_pte *pte =3D NULL; - unsigned long i; =20 while (start_pfn <=3D end_pfn) { if (!pte) @@ -2009,13 +1852,8 @@ static void switch_to_super_page(struct dmar_domain = *domain, start_pfn + lvl_pages - 1, level + 1); =20 - xa_for_each(&domain->iommu_array, i, info) - iommu_flush_iotlb_psi(info->iommu, domain, - start_pfn, lvl_pages, - 0, 0); - if (domain->nested_parent) - parent_domain_flush(domain, start_pfn, - lvl_pages, 0); + cache_tag_flush_range(domain, start_pfn << VTD_PAGE_SHIFT, + end_pfn << VTD_PAGE_SHIFT, 0); } =20 pte++; @@ -3387,18 +3225,9 @@ static int intel_iommu_memory_notifier(struct notifi= er_block *nb, case MEM_OFFLINE: case MEM_CANCEL_ONLINE: { - struct dmar_drhd_unit *drhd; - struct intel_iommu *iommu; LIST_HEAD(freelist); =20 domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist); - - rcu_read_lock(); - for_each_active_iommu(iommu, drhd) - iommu_flush_iotlb_psi(iommu, si_domain, - start_vpfn, mhp->nr_pages, - list_empty(&freelist), 0); - rcu_read_unlock(); put_pages_list(&freelist); } break; --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E333155387 for ; Mon, 25 Mar 2024 02:18:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333105; cv=none; b=TzZaUjMGP7j+Wuv8sTVZiIgH+SYLooPVeNQG6bHx36ZL2ByO/FM0oKAf36gr+w9JcDhOUJ2Oe+H4Zy6Oh3iuSLa40Y+jfxCIEwxkZTvzngvX+I1e/FLBg2L5/aiHDBqoN0XkIHCanXXF/t/nVc+H9tI8IvqDkrOtxoGRe9J2RuU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333105; c=relaxed/simple; bh=GRqfeV/YKXedxQfVodmQRhJEmBymUSTAs5i+B3DOhSQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i/8yuuObfnG0VwC9w4v14bCMm5vx3IB4568j3uU0TzTEgqlOvPlJ3E4/XHubwN4J3chgaHWkhz7XIBNphECu7tvyHt8jFGYoZ/YKvnEH21f3m8n7jUIFaL7ifexJUuN/TORceaBBJl/hLdhbwUUdkCtjDOPYzQyI+vtnk4O4Hxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=GeEqZZyo; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="GeEqZZyo" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333103; x=1742869103; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GRqfeV/YKXedxQfVodmQRhJEmBymUSTAs5i+B3DOhSQ=; b=GeEqZZyotNUsS0tBN7Zix89M/XBFhO2RyxknBL49qCKPtqhKNgyRW/+e qV1AaOW+WVakmKebq2x6nu5xucea91DKQI6RotWa8Nn457qIs76iqiggN VFgfNGBMijH2Ulcuka9M4daxvjEFvW8ns5wUB6EhulWRqJIjOjSD6ZkKx 798Q6zqC9TPr6E5bIbh7Tt6pbCUfa1Jf142/Dmi0KH9a3Pjl3pD8zjszw Mn32WWOSfFC2LzYHM3w4Tca9nG+aXoaeL4kSxAPm+wEYO4jHdiIaci9hV zhxt3P8s4pQcYP8VkZrosGLEZ+7pC7DMM/wUsaAaqVZnl3OS01SjygvY9 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271409" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271409" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353986" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:20 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user Date: Mon, 25 Mar 2024 10:17:01 +0800 Message-Id: <20240325021705.249769-9-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The cache_invalidate_user callback is called to invalidate a range of caches for the affected user domain. Use cache_tag_flush_range() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/nested.c | 50 +++--------------------------------- 1 file changed, 3 insertions(+), 47 deletions(-) diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index 85c744099558..e251507cfcc0 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -90,50 +90,6 @@ static void intel_nested_domain_free(struct iommu_domain= *domain) kfree(dmar_domain); } =20 -static void nested_flush_dev_iotlb(struct dmar_domain *domain, u64 addr, - unsigned int mask) -{ - struct device_domain_info *info; - unsigned long flags; - u16 sid, qdep; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(info, &domain->devices, link) { - if (!info->ats_enabled) - continue; - sid =3D info->bus << 8 | info->devfn; - qdep =3D info->ats_qdep; - qi_flush_dev_iotlb(info->iommu, sid, info->pfsid, - qdep, addr, mask); - quirk_extra_dev_tlb_flush(info, addr, mask, - IOMMU_NO_PASID, qdep); - } - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void intel_nested_flush_cache(struct dmar_domain *domain, u64 addr, - u64 npages, bool ih) -{ - struct iommu_domain_info *info; - unsigned int mask; - unsigned long i; - - xa_for_each(&domain->iommu_array, i, info) - qi_flush_piotlb(info->iommu, - domain_id_iommu(domain, info->iommu), - IOMMU_NO_PASID, addr, npages, ih); - - if (!domain->has_iotlb_device) - return; - - if (npages =3D=3D U64_MAX) - mask =3D 64 - VTD_PAGE_SHIFT; - else - mask =3D ilog2(__roundup_pow_of_two(npages)); - - nested_flush_dev_iotlb(domain, addr, mask); -} - static int intel_nested_cache_invalidate_user(struct iommu_domain *domain, struct iommu_user_data_array *array) { @@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct io= mmu_domain *domain, break; } =20 - intel_nested_flush_cache(dmar_domain, inv_entry.addr, - inv_entry.npages, - inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); + cache_tag_flush_range(dmar_domain, inv_entry.addr, + inv_entry.npages, + inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); processed++; } =20 --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D62B2155391 for ; Mon, 25 Mar 2024 02:18:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333109; cv=none; b=P8ZXDVEmcQLFVQzWQygAPDxZrSfiaGKVmTo9upZQMPINhkyRnD6sO4S93egPIC4ZaZDGPeZA65ThoFSf/iffWTcGpN8Xdx8u+FdP5I72BEnLLUcjwqCT0YjUVb1dK3ZP/h5LBxLLIwCvb9uiXUkPoJdOO6HJoyqSFXzUTIj7RhE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333109; c=relaxed/simple; bh=UjPXkQ6AmKr1iNLH78v2nHs2xpUcx0lFy26N8z4uNc8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jP9DEHht7WyjwNUaK/cClNRh8iT8bDVGa174Ygu745OfG27VqvkVZTaal2rJ8AQcYU/TP0NU5gDMlYs9rFqxhO14HLpQM8BW5QOPcs7uI1ven9K2ZHRnt2hgfMj6Xp1jjDwEdT1P9SDN4imHGpNiHb8v0KPiXbs6VayvpQepxy0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=ivvMJAAC; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="ivvMJAAC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333107; x=1742869107; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UjPXkQ6AmKr1iNLH78v2nHs2xpUcx0lFy26N8z4uNc8=; b=ivvMJAACE6A95XhAvTR1FimfFOyWyMgaIlODixcmBt0np1pFWtmTFWvt OvikrXbv+d3vfBJYxZe5WzuBrMQdxxJHNsYjv5ogBa88+NdEaci3NnTPQ bg4NFbd74+kJrLmXR8HRNKybLeFdCguT4PVh+Z5o45wZCSGt5D6No4QJs SjSrSaFHw9iPL/BhYCu6xQf9ootaIH0I4JM+QPsd+KdTJmecQdb5LEI8w Qc1VNXYPw6pROvvR86QEzAY/icaheGCUzyA3VVaC7YWrBKihXGnAc72cC lGahwK8PG6kM2DgVSwv/Jk8j5rFsTm4oLnf/QkepxgZ7L7TpDHw16b3Zl A==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271428" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271428" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15353997" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:23 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 09/12] iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs Date: Mon, 25 Mar 2024 10:17:02 +0800 Message-Id: <20240325021705.249769-10-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The arch_invalidate_secondary_tlbs callback is called in the SVA mm notification path. It invalidates all or a range of caches after the CPU page table is modified. Use the cache tag helps in this path. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/svm.c | 76 +++---------------------------------- 2 files changed, 6 insertions(+), 71 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index d05fa0122d65..2575ded9f354 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1148,6 +1148,7 @@ struct intel_svm { struct mm_struct *mm; u32 pasid; struct list_head devs; + struct dmar_domain *domain; }; #else static inline void intel_svm_check(struct intel_iommu *iommu) {} diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index d706226e84ee..751fab476fa2 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -168,88 +168,20 @@ void intel_svm_check(struct intel_iommu *iommu) iommu->flags |=3D VTD_FLAG_SVM_CAPABLE; } =20 -static void __flush_svm_range_dev(struct intel_svm *svm, - struct intel_svm_dev *sdev, - unsigned long address, - unsigned long pages, int ih) -{ - struct device_domain_info *info =3D dev_iommu_priv_get(sdev->dev); - - if (WARN_ON(!pages)) - return; - - qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, address, pages, ih); - if (info->ats_enabled) { - qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid, - svm->pasid, sdev->qdep, address, - order_base_2(pages)); - quirk_extra_dev_tlb_flush(info, address, order_base_2(pages), - svm->pasid, sdev->qdep); - } -} - -static void intel_flush_svm_range_dev(struct intel_svm *svm, - struct intel_svm_dev *sdev, - unsigned long address, - unsigned long pages, int ih) -{ - unsigned long shift =3D ilog2(__roundup_pow_of_two(pages)); - unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + shift)); - unsigned long start =3D ALIGN_DOWN(address, align); - unsigned long end =3D ALIGN(address + (pages << VTD_PAGE_SHIFT), align); - - while (start < end) { - __flush_svm_range_dev(svm, sdev, start, align >> VTD_PAGE_SHIFT, ih); - start +=3D align; - } -} - -static void intel_flush_svm_range(struct intel_svm *svm, unsigned long add= ress, - unsigned long pages, int ih) -{ - struct intel_svm_dev *sdev; - - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) - intel_flush_svm_range_dev(svm, sdev, address, pages, ih); - rcu_read_unlock(); -} - -static void intel_flush_svm_all(struct intel_svm *svm) -{ - struct device_domain_info *info; - struct intel_svm_dev *sdev; - - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) { - info =3D dev_iommu_priv_get(sdev->dev); - - qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, 0, -1UL, 0); - if (info->ats_enabled) { - qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid, - svm->pasid, sdev->qdep, - 0, 64 - VTD_PAGE_SHIFT); - quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT, - svm->pasid, sdev->qdep); - } - } - rcu_read_unlock(); -} - /* Pages have been freed at this point */ static void intel_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end) { struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); + struct dmar_domain *domain =3D svm->domain; =20 if (start =3D=3D 0 && end =3D=3D -1UL) { - intel_flush_svm_all(svm); + cache_tag_flush_all(domain); return; } =20 - intel_flush_svm_range(svm, start, - (end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0); + cache_tag_flush_range(domain, start, end, 0); } =20 static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) @@ -336,6 +268,7 @@ static int intel_svm_set_dev_pasid(struct iommu_domain = *domain, INIT_LIST_HEAD_RCU(&svm->devs); =20 svm->notifier.ops =3D &intel_mmuops; + svm->domain =3D to_dmar_domain(domain); ret =3D mmu_notifier_register(&svm->notifier, mm); if (ret) { kfree(svm); @@ -803,6 +736,7 @@ struct iommu_domain *intel_svm_domain_alloc(void) if (!domain) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; + domain->use_first_level =3D true; INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->cache_lock); =20 --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A325E15538A for ; Mon, 25 Mar 2024 02:18:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333111; cv=none; b=lSqtT4lHx0yceMsNYB+YNZPlzOJ7ZHkRFSGF/Uybr/9niZuLwHTwTvOAZ1DrG+9siuN7XbuA7syjMALZW3NEOX+dAa2oXfOpE9T9mH96QuOrz0784CHR7XVz1Yl1T6ieiebdi8GvxSFUbage84s4WWNOiH2J6Ez6Hte8hXuVfh0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333111; c=relaxed/simple; bh=8/WQV9+3WuymPpz+pu9Xu/DO05uKb4YwYS5DTDp8Ce8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NU/8Yky5Ss4drGSqhVu9FedQ+jSwzfwwmMFeQzPKUzPfGKagLOkdLSsFSSmiJZKOvOxxtsZ8OEOSDVVZ6VoWWuwDthMPboBjHlNVdWyKC+qNAy33tHKxkActS5xYR/ALmJS0XAUdPqn0qBCHn+y/cwmtKBRC3CatMe/iqLjxoXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=YQWtVOZQ; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="YQWtVOZQ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333109; x=1742869109; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8/WQV9+3WuymPpz+pu9Xu/DO05uKb4YwYS5DTDp8Ce8=; b=YQWtVOZQMon+oC3MJ6kvd+n7uePdRkVK5WQVhQn9SdGhOvXMkzknGLy8 0UH4dUfhZ2dgSrz0MkaahJPfuXsdeZP+uKvnW/IhZGOVAOkXh7oJ4l0cq dBqljPEy+CaAOiLHWN097OO/Jksk/C9xrOOgGov3dLeMo1P76Gaqzo7vO fmYimt6esgaYsJ/CCPnGtZUKfjWB1lJkh0svyiGb6zEABoquV+4qEYOv0 zl8fydmpK6EKR5TEllakpV4d0elXRn5lpuAFmijmgg7WBykhHQB2ayj6b IiMIEm2F2pS/pM1RPGzOo4wLcqverRBpsXLp+9Qztw+e2NoPuArx0mXOt Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271445" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271445" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15354007" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:26 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 10/12] iommu/vt-d: Retire intel_svm_dev Date: Mon, 25 Mar 2024 10:17:03 +0800 Message-Id: <20240325021705.249769-11-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The intel_svm_dev data structure used in the sva implementation for the Intel IOMMU driver stores information about a device attached to an SVA domain. It is a duplicate of dev_pasid_info that serves the same purpose. Replace intel_svm_dev with dev_pasid_info and clean up the use of intel_svm_dev. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 15 +---- drivers/iommu/intel/iommu.c | 30 ++++----- drivers/iommu/intel/svm.c | 131 +++++++++++------------------------- 3 files changed, 55 insertions(+), 121 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 2575ded9f354..5dcced6f3634 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -650,6 +650,7 @@ struct dmar_domain { struct list_head s2_link; }; }; + struct intel_svm *svm; =20 struct iommu_domain domain; /* generic domain data structure for iommu core */ @@ -1131,23 +1132,13 @@ int intel_svm_finish_prq(struct intel_iommu *iommu); void intel_svm_page_response(struct device *dev, struct iopf_fault *evt, struct iommu_page_response *msg); struct iommu_domain *intel_svm_domain_alloc(void); -void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t pasid); +void intel_svm_remove_dev_pasid(struct iommu_domain *domain); void intel_drain_pasid_prq(struct device *dev, u32 pasid); =20 -struct intel_svm_dev { - struct list_head list; - struct rcu_head rcu; - struct device *dev; - struct intel_iommu *iommu; - u16 did; - u16 sid, qdep; -}; - struct intel_svm { struct mmu_notifier notifier; struct mm_struct *mm; u32 pasid; - struct list_head devs; struct dmar_domain *domain; }; #else @@ -1158,7 +1149,7 @@ static inline struct iommu_domain *intel_svm_domain_a= lloc(void) return NULL; } =20 -static inline void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t= pasid) +static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain) { } #endif diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 6e019297843b..2ac1fc041333 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4384,18 +4384,6 @@ static void intel_iommu_remove_dev_pasid(struct devi= ce *dev, ioasid_t pasid) goto out_tear_down; dmar_domain =3D to_dmar_domain(domain); =20 - /* - * The SVA implementation needs to handle its own stuffs like the mm - * notification. Before consolidating that code into iommu core, let - * the intel sva code handle it. - */ - if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { - intel_svm_remove_dev_pasid(dev, pasid); - cache_tag_unassign_domain(dmar_domain, - FLPT_DEFAULT_DID, dev, pasid); - goto out_tear_down; - } - spin_lock_irqsave(&dmar_domain->lock, flags); list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) { if (curr->dev =3D=3D dev && curr->pasid =3D=3D pasid) { @@ -4407,10 +4395,20 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 - did =3D domain_id_iommu(dmar_domain, iommu); - cache_tag_unassign_domain(dmar_domain, did, dev, pasid); - domain_detach_iommu(dmar_domain, iommu); - intel_iommu_debugfs_remove_dev_pasid(dev_pasid); + /* + * The SVA implementation needs to handle its own stuffs like the mm + * notification. Before consolidating that code into iommu core, let + * the intel sva code handle it. + */ + if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { + cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid); + intel_svm_remove_dev_pasid(domain); + } else { + did =3D domain_id_iommu(dmar_domain, iommu); + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); + domain_detach_iommu(dmar_domain, iommu); + intel_iommu_debugfs_remove_dev_pasid(dev_pasid); + } kfree(dev_pasid); out_tear_down: intel_pasid_tear_down_entry(iommu, dev, pasid, false); diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 751fab476fa2..0b767d16fb71 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -43,23 +43,6 @@ static void *pasid_private_find(ioasid_t pasid) return xa_load(&pasid_private_array, pasid); } =20 -static struct intel_svm_dev * -svm_lookup_device_by_dev(struct intel_svm *svm, struct device *dev) -{ - struct intel_svm_dev *sdev =3D NULL, *t; - - rcu_read_lock(); - list_for_each_entry_rcu(t, &svm->devs, list) { - if (t->dev =3D=3D dev) { - sdev =3D t; - break; - } - } - rcu_read_unlock(); - - return sdev; -} - int intel_svm_enable_prq(struct intel_iommu *iommu) { struct iopf_queue *iopfq; @@ -187,7 +170,10 @@ static void intel_arch_invalidate_secondary_tlbs(struc= t mmu_notifier *mn, static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct intel_svm_dev *sdev; + struct dmar_domain *domain =3D svm->domain; + struct dev_pasid_info *dev_pasid; + struct device_domain_info *info; + unsigned long flags; =20 /* This might end up being called from exit_mmap(), *before* the page * tables are cleared. And __mmu_notifier_release() will delete us from @@ -201,11 +187,13 @@ static void intel_mm_release(struct mmu_notifier *mn,= struct mm_struct *mm) * page) so that we end up taking a fault that the hardware really * *has* to handle gracefully without affecting other processes. */ - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) - intel_pasid_tear_down_entry(sdev->iommu, sdev->dev, - svm->pasid, true); - rcu_read_unlock(); + spin_lock_irqsave(&domain->lock, flags); + list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { + info =3D dev_iommu_priv_get(dev_pasid->dev); + intel_pasid_tear_down_entry(info->iommu, dev_pasid->dev, + dev_pasid->pasid, true); + } + spin_unlock_irqrestore(&domain->lock, flags); =20 } =20 @@ -214,47 +202,17 @@ static const struct mmu_notifier_ops intel_mmuops =3D= { .arch_invalidate_secondary_tlbs =3D intel_arch_invalidate_secondary_tlbs, }; =20 -static int pasid_to_svm_sdev(struct device *dev, unsigned int pasid, - struct intel_svm **rsvm, - struct intel_svm_dev **rsdev) -{ - struct intel_svm_dev *sdev =3D NULL; - struct intel_svm *svm; - - if (pasid =3D=3D IOMMU_PASID_INVALID || pasid >=3D PASID_MAX) - return -EINVAL; - - svm =3D pasid_private_find(pasid); - if (IS_ERR(svm)) - return PTR_ERR(svm); - - if (!svm) - goto out; - - /* - * If we found svm for the PASID, there must be at least one device - * bond. - */ - if (WARN_ON(list_empty(&svm->devs))) - return -EINVAL; - sdev =3D svm_lookup_device_by_dev(svm, dev); - -out: - *rsvm =3D svm; - *rsdev =3D sdev; - - return 0; -} - static int intel_svm_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid) { struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); struct intel_iommu *iommu =3D info->iommu; struct mm_struct *mm =3D domain->mm; - struct intel_svm_dev *sdev; + struct dev_pasid_info *dev_pasid; struct intel_svm *svm; unsigned long sflags; + unsigned long flags; int ret =3D 0; =20 svm =3D pasid_private_find(pasid); @@ -265,7 +223,6 @@ static int intel_svm_set_dev_pasid(struct iommu_domain = *domain, =20 svm->pasid =3D pasid; svm->mm =3D mm; - INIT_LIST_HEAD_RCU(&svm->devs); =20 svm->notifier.ops =3D &intel_mmuops; svm->domain =3D to_dmar_domain(domain); @@ -283,26 +240,19 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, } } =20 - sdev =3D kzalloc(sizeof(*sdev), GFP_KERNEL); - if (!sdev) { - ret =3D -ENOMEM; + dmar_domain->svm =3D svm; + + dev_pasid =3D kzalloc(sizeof(*dev_pasid), GFP_KERNEL); + if (!dev_pasid) goto free_svm; - } =20 - sdev->dev =3D dev; - sdev->iommu =3D iommu; - sdev->did =3D FLPT_DEFAULT_DID; - sdev->sid =3D PCI_DEVID(info->bus, info->devfn); - if (info->ats_enabled) { - sdev->qdep =3D info->ats_qdep; - if (sdev->qdep >=3D QI_DEV_EIOTLB_MAX_INVS) - sdev->qdep =3D 0; - } + dev_pasid->dev =3D dev; + dev_pasid->pasid =3D pasid; =20 ret =3D cache_tag_assign_domain(to_dmar_domain(domain), FLPT_DEFAULT_DID, dev, pasid); if (ret) - goto free_sdev; + goto free_dev_pasid; =20 /* Setup the pasid table: */ sflags =3D cpu_feature_enabled(X86_FEATURE_LA57) ? PASID_FLAG_FL5LP : 0; @@ -311,17 +261,19 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, if (ret) goto unassign_tag; =20 - list_add_rcu(&sdev->list, &svm->devs); + spin_lock_irqsave(&dmar_domain->lock, flags); + list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids); + spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 return 0; =20 unassign_tag: cache_tag_unassign_domain(to_dmar_domain(domain), FLPT_DEFAULT_DID, dev, pasid); -free_sdev: - kfree(sdev); +free_dev_pasid: + kfree(dev_pasid); free_svm: - if (list_empty(&svm->devs)) { + if (list_empty(&dmar_domain->dev_pasids)) { mmu_notifier_unregister(&svm->notifier, mm); pasid_private_remove(pasid); kfree(svm); @@ -330,26 +282,17 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, return ret; } =20 -void intel_svm_remove_dev_pasid(struct device *dev, u32 pasid) +void intel_svm_remove_dev_pasid(struct iommu_domain *domain) { - struct intel_svm_dev *sdev; - struct intel_svm *svm; - struct mm_struct *mm; + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); + struct intel_svm *svm =3D dmar_domain->svm; + struct mm_struct *mm =3D domain->mm; =20 - if (pasid_to_svm_sdev(dev, pasid, &svm, &sdev)) - return; - mm =3D svm->mm; - - if (sdev) { - list_del_rcu(&sdev->list); - kfree_rcu(sdev, rcu); - - if (list_empty(&svm->devs)) { - if (svm->notifier.ops) - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(svm->pasid); - kfree(svm); - } + if (list_empty(&dmar_domain->dev_pasids)) { + if (svm->notifier.ops) + mmu_notifier_unregister(&svm->notifier, mm); + pasid_private_remove(svm->pasid); + kfree(svm); } } =20 @@ -737,8 +680,10 @@ struct iommu_domain *intel_svm_domain_alloc(void) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; domain->use_first_level =3D true; + INIT_LIST_HEAD(&domain->dev_pasids); INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->cache_lock); + spin_lock_init(&domain->lock); =20 return &domain->domain; } --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CECE91553AC for ; Mon, 25 Mar 2024 02:18:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333114; cv=none; b=PKjt/uTurEsPMvkT2S2LYe3dBU/egVZ9CJJr0V0E2u3GWUw78eKISMErUxJ92XcxP4dXrq2xUyLiheOTJpdt0NCPAyukL9SlSXjmlwvpHPptpKDSSTmIbECvKuNfN4AmAVUuL1Z0M81IRxBWw5Vsm+cvkxfHgX9N/FZNcZk5Fbw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333114; c=relaxed/simple; bh=Z4N9WjZXB3Uq7GgVWrBbRtPl6f+yBep/k+sWPTppZwU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sfkxy37DYzcPHfjvqlJ/SdSEhoAKOVO9LpQBZZvtIbNk3U7wnZmGekrFYhetC9HBMs6CgHvc+nN2EvO9OozfZvOvPPoAPpkVkbnFrDEMB9NzEOsjEFF6+leCCwVFGim5Y14aaWTvsTFoaKMphayemuZhrdozfLxbFVAOAIxaQrE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=LnbyqRiZ; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="LnbyqRiZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333113; x=1742869113; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Z4N9WjZXB3Uq7GgVWrBbRtPl6f+yBep/k+sWPTppZwU=; b=LnbyqRiZLnPwJUiyEONEepEEkFe8mKSIH8/XfUEWeyNDrVegENWfxp7C psC1IiD9jGkuKZWHf7M4ruqV+A41Y2QbNhnU77VkNqEUzgevWKLiAqQrU seEV7z1spDcaAKv6kXMzfLhgcID66JVDZ+1pur65eHtc7JNDHhha0uyy5 Jx9+xGD7l6wnaMKmYEo4LzTuOkmUhrjKIiWHLUzhMrSVCjctcbzpsBX1Q UOIYBibGjIuzPr9Li9LWpX3PArN0ss3MAZ60UyGU2AgPFNxnO9f6bLhtX vb+vcXiK2VU2H5H8uBdDy0V0iZMd4mEXCeZ4zjvIVKVrP32edhlqogTxA w==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271461" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271461" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15354011" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:29 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Jason Gunthorpe , Vasant Hegde , Lu Baolu Subject: [PATCH 11/12] iommu: Add ops->domain_alloc_sva() Date: Mon, 25 Mar 2024 10:17:04 +0800 Message-Id: <20240325021705.249769-12-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jason Gunthorpe Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here. This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain. Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL. Signed-off-by: Jason Gunthorpe [Removed smmu3 related changes - Vasant] Signed-off-by: Vasant Hegde Reviewed-by: Tina Zhang Link: https://lore.kernel.org/r/20240311090843.133455-15-vasant.hegde@amd.c= om Signed-off-by: Lu Baolu --- include/linux/iommu.h | 3 +++ drivers/iommu/iommu-sva.c | 16 +++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 2e925b5eba53..8aabe83af8f2 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array( * Upon failure, ERR_PTR must be returned. * @domain_alloc_paging: Allocate an iommu_domain that can be used for * UNMANAGED, DMA, and DMA_FQ domain types. + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressi= ng. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IO= MMU @@ -558,6 +559,8 @@ struct iommu_ops { struct device *dev, u32 flags, struct iommu_domain *parent, const struct iommu_user_data *user_data); struct iommu_domain *(*domain_alloc_paging)(struct device *dev); + struct iommu_domain *(*domain_alloc_sva)(struct device *dev, + struct mm_struct *mm); =20 struct iommu_device *(*probe_device)(struct device *dev); void (*release_device)(struct device *dev); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 640acc804e8c..18a35e798b72 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm =20 /* Allocate a new domain and set it on device pasid. */ domain =3D iommu_sva_domain_alloc(dev, mm); - if (!domain) { - ret =3D -ENOMEM; + if (IS_ERR(domain)) { + ret =3D PTR_ERR(domain); goto out_free_handle; } =20 @@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct dev= ice *dev, const struct iommu_ops *ops =3D dev_iommu_ops(dev); struct iommu_domain *domain; =20 - domain =3D ops->domain_alloc(IOMMU_DOMAIN_SVA); - if (!domain) - return NULL; + if (ops->domain_alloc_sva) { + domain =3D ops->domain_alloc_sva(dev, mm); + if (IS_ERR(domain)) + return domain; + } else { + domain =3D ops->domain_alloc(IOMMU_DOMAIN_SVA); + if (!domain) + return ERR_PTR(-ENOMEM); + } =20 domain->type =3D IOMMU_DOMAIN_SVA; mmgrab(mm); --=20 2.34.1 From nobody Sat Feb 7 17:13:54 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 27C3E15539B for ; Mon, 25 Mar 2024 02:18:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333116; cv=none; b=M2b4DazPcazh762IRRt24Z1Qb57kJAVztpXd2AqD4xT+JpGIe78rZv7eR5WjGmNDC6CG69n2/zTsiXxiVkQHJpL6iWL3O0YFOg0rJ9wuq/JMUEMCjhtXCOE9PDgTAmZhvWI0L0le1Z4ce7R3CRrRQWuMFHh+XJ7gPbfYWk2NWy0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711333116; c=relaxed/simple; bh=jEIymX71O072SVGexRyBOj25jZMdFHRVH49ILGdL1mc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NAz/4NxtwBNyExMnl8xlKx1CGwEwo+xHs9qPPdWRkU3uLn08jVtse71PCYTSvcOLBZUOi+OqaWW6Z/8P4yWA1+37r3xDyAFFB6GYInWk+X3wmuWCPczSFv2JDpyT7kQHIGzlGENn9DMCcbFpTBTNb/CRUUnptIz/rT9R2BUI7cE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CCX9zIIi; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CCX9zIIi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1711333115; x=1742869115; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jEIymX71O072SVGexRyBOj25jZMdFHRVH49ILGdL1mc=; b=CCX9zIIi5CuY8zBKCyoimnSa6JssG9tjzaQQTBewjfrJXgV9fcSGJ4yt Ss2c7Q2laBKLa2mM+v1kBjdUbjsNRTcblEbOFDolOxLqExtSgRMtwj+wQ MYyV6yKADW2UY3A/xQnncpm1hVGrKsmJ6PrNm30bQYa6OYOhsSvbdHWJo 1HR0WdYEzcQdDACbs40OWdysihhzMqSCyyDrc3EaxzM0nvS8PvbcVEfOt 5VVNTptVp9YHnhRa9WvyeuOAtgaIywGPLkDSEsS8Z02AIbEv4ExwpCxJy yDL7rmTUi5vWUz0RNQhYknDkytn6CjkOcTTP2UzjgaK5UJYlqoygzBlAV Q==; X-IronPort-AV: E=McAfee;i="6600,9927,11023"; a="9271479" X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="9271479" Received: from orviesa010.jf.intel.com ([10.64.159.150]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Mar 2024 19:18:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,152,1708416000"; d="scan'208";a="15354019" Received: from allen-box.sh.intel.com ([10.239.159.127]) by orviesa010.jf.intel.com with ESMTP; 24 Mar 2024 19:18:32 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH 12/12] iommu/vt-d: Retire struct intel_svm Date: Mon, 25 Mar 2024 10:17:05 +0800 Message-Id: <20240325021705.249769-13-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240325021705.249769-1-baolu.lu@linux.intel.com> References: <20240325021705.249769-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The struct intel_svm was used for keeping attached devices info for sva domain. Since sva domain is a kind of iommu_domain, the struct dmar_domain should centralize all info of a sva domain, including the info of attached devices. Therefore, retire struct intel_svm to clean up the code. Besides, allocate sva domain in domain_alloc_sva() callback which allows the memory management notifier lifetime to follow the lifetime of the iommu_domain. Co-developed-by: Tina Zhang Signed-off-by: Tina Zhang Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 26 ++++------ drivers/iommu/intel/iommu.c | 9 +--- drivers/iommu/intel/svm.c | 94 +++++++++---------------------------- 3 files changed, 32 insertions(+), 97 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 5dcced6f3634..7a4645e06bb7 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -649,8 +649,12 @@ struct dmar_domain { /* link to parent domain siblings */ struct list_head s2_link; }; + + /* SVA domain */ + struct { + struct mmu_notifier notifier; + }; }; - struct intel_svm *svm; =20 struct iommu_domain domain; /* generic domain data structure for iommu core */ @@ -1131,26 +1135,16 @@ int intel_svm_enable_prq(struct intel_iommu *iommu); int intel_svm_finish_prq(struct intel_iommu *iommu); void intel_svm_page_response(struct device *dev, struct iopf_fault *evt, struct iommu_page_response *msg); -struct iommu_domain *intel_svm_domain_alloc(void); -void intel_svm_remove_dev_pasid(struct iommu_domain *domain); +struct iommu_domain *intel_svm_domain_alloc(struct device *dev, + struct mm_struct *mm); void intel_drain_pasid_prq(struct device *dev, u32 pasid); - -struct intel_svm { - struct mmu_notifier notifier; - struct mm_struct *mm; - u32 pasid; - struct dmar_domain *domain; -}; #else static inline void intel_svm_check(struct intel_iommu *iommu) {} static inline void intel_drain_pasid_prq(struct device *dev, u32 pasid) {} -static inline struct iommu_domain *intel_svm_domain_alloc(void) -{ - return NULL; -} - -static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain) +static inline struct iommu_domain *intel_svm_domain_alloc(struct device *d= ev, + struct mm_struct *mm) { + return ERR_PTR(-ENODEV); } #endif =20 diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 2ac1fc041333..cdf7882f58f3 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -3687,8 +3687,6 @@ static struct iommu_domain *intel_iommu_domain_alloc(= unsigned type) return domain; case IOMMU_DOMAIN_IDENTITY: return &si_domain->domain; - case IOMMU_DOMAIN_SVA: - return intel_svm_domain_alloc(); default: return NULL; } @@ -4395,14 +4393,8 @@ static void intel_iommu_remove_dev_pasid(struct devi= ce *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 - /* - * The SVA implementation needs to handle its own stuffs like the mm - * notification. Before consolidating that code into iommu core, let - * the intel sva code handle it. - */ if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid); - intel_svm_remove_dev_pasid(domain); } else { did =3D domain_id_iommu(dmar_domain, iommu); cache_tag_unassign_domain(dmar_domain, did, dev, pasid); @@ -4631,6 +4623,7 @@ const struct iommu_ops intel_iommu_ops =3D { .hw_info =3D intel_iommu_hw_info, .domain_alloc =3D intel_iommu_domain_alloc, .domain_alloc_user =3D intel_iommu_domain_alloc_user, + .domain_alloc_sva =3D intel_svm_domain_alloc, .probe_device =3D intel_iommu_probe_device, .probe_finalize =3D intel_iommu_probe_finalize, .release_device =3D intel_iommu_release_device, diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 0b767d16fb71..47e475f67046 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -26,23 +26,6 @@ =20 static irqreturn_t prq_event_thread(int irq, void *d); =20 -static DEFINE_XARRAY_ALLOC(pasid_private_array); -static int pasid_private_add(ioasid_t pasid, void *priv) -{ - return xa_alloc(&pasid_private_array, &pasid, priv, - XA_LIMIT(pasid, pasid), GFP_ATOMIC); -} - -static void pasid_private_remove(ioasid_t pasid) -{ - xa_erase(&pasid_private_array, pasid); -} - -static void *pasid_private_find(ioasid_t pasid) -{ - return xa_load(&pasid_private_array, pasid); -} - int intel_svm_enable_prq(struct intel_iommu *iommu) { struct iopf_queue *iopfq; @@ -156,8 +139,7 @@ static void intel_arch_invalidate_secondary_tlbs(struct= mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end) { - struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct dmar_domain *domain =3D svm->domain; + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); =20 if (start =3D=3D 0 && end =3D=3D -1UL) { cache_tag_flush_all(domain); @@ -169,8 +151,7 @@ static void intel_arch_invalidate_secondary_tlbs(struct= mmu_notifier *mn, =20 static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct dmar_domain *domain =3D svm->domain; + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); struct dev_pasid_info *dev_pasid; struct device_domain_info *info; unsigned long flags; @@ -210,41 +191,13 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, struct intel_iommu *iommu =3D info->iommu; struct mm_struct *mm =3D domain->mm; struct dev_pasid_info *dev_pasid; - struct intel_svm *svm; unsigned long sflags; unsigned long flags; int ret =3D 0; =20 - svm =3D pasid_private_find(pasid); - if (!svm) { - svm =3D kzalloc(sizeof(*svm), GFP_KERNEL); - if (!svm) - return -ENOMEM; - - svm->pasid =3D pasid; - svm->mm =3D mm; - - svm->notifier.ops =3D &intel_mmuops; - svm->domain =3D to_dmar_domain(domain); - ret =3D mmu_notifier_register(&svm->notifier, mm); - if (ret) { - kfree(svm); - return ret; - } - - ret =3D pasid_private_add(svm->pasid, svm); - if (ret) { - mmu_notifier_unregister(&svm->notifier, mm); - kfree(svm); - return ret; - } - } - - dmar_domain->svm =3D svm; - dev_pasid =3D kzalloc(sizeof(*dev_pasid), GFP_KERNEL); if (!dev_pasid) - goto free_svm; + return -ENOMEM; =20 dev_pasid->dev =3D dev; dev_pasid->pasid =3D pasid; @@ -272,30 +225,10 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, FLPT_DEFAULT_DID, dev, pasid); free_dev_pasid: kfree(dev_pasid); -free_svm: - if (list_empty(&dmar_domain->dev_pasids)) { - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(pasid); - kfree(svm); - } =20 return ret; } =20 -void intel_svm_remove_dev_pasid(struct iommu_domain *domain) -{ - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - struct intel_svm *svm =3D dmar_domain->svm; - struct mm_struct *mm =3D domain->mm; - - if (list_empty(&dmar_domain->dev_pasids)) { - if (svm->notifier.ops) - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(svm->pasid); - kfree(svm); - } -} - /* Page request queue descriptor */ struct page_req_dsc { union { @@ -663,7 +596,12 @@ void intel_svm_page_response(struct device *dev, struc= t iopf_fault *evt, =20 static void intel_svm_domain_free(struct iommu_domain *domain) { - kfree(to_dmar_domain(domain)); + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); + + if (dmar_domain->notifier.ops) + mmu_notifier_unregister(&dmar_domain->notifier, domain->mm); + + kfree(dmar_domain); } =20 static const struct iommu_domain_ops intel_svm_domain_ops =3D { @@ -671,13 +609,16 @@ static const struct iommu_domain_ops intel_svm_domain= _ops =3D { .free =3D intel_svm_domain_free }; =20 -struct iommu_domain *intel_svm_domain_alloc(void) +struct iommu_domain *intel_svm_domain_alloc(struct device *dev, + struct mm_struct *mm) { struct dmar_domain *domain; + int ret; =20 domain =3D kzalloc(sizeof(*domain), GFP_KERNEL); if (!domain) - return NULL; + return ERR_PTR(-ENOMEM); + domain->domain.ops =3D &intel_svm_domain_ops; domain->use_first_level =3D true; INIT_LIST_HEAD(&domain->dev_pasids); @@ -685,5 +626,12 @@ struct iommu_domain *intel_svm_domain_alloc(void) spin_lock_init(&domain->cache_lock); spin_lock_init(&domain->lock); =20 + domain->notifier.ops =3D &intel_mmuops; + ret =3D mmu_notifier_register(&domain->notifier, mm); + if (ret) { + kfree(domain); + return ERR_PTR(ret); + } + return &domain->domain; } --=20 2.34.1