From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F48B53BE for ; Wed, 10 Apr 2024 02:10:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715006; cv=none; b=hL0Q3I3D3CoWqGnl4+9a9YRv4ksqPK2cozwFQH/7fFsWIqfGjcx9Ul3uKFAd+d4m/rU8UkRzLA+vRXpMdtEpUynaK7hLRSFt+QfpIRoyJnZ/mvsYcMH4Rizd8bz+R+n80IYbPDDGNHkVclKaKFRCH/aXZxesEPHi8uWAnS3mg+Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715006; c=relaxed/simple; bh=SwmxmPxDMRej1aYgKsDEycIVIcKzEXYyxRg87X2L/r8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SXofi6af/ftux8GqkQv3GegdkG4+5q8XmXry5rxb/IsS7o2CMokj0lWtJ9nxWR0BFdpQTOI1oLD4FL5xOr2QQifcInKGieBkLVk5+Eb2typ8Tj/mJO8NJCicOqwnnwArtvMQ7TCvPsPk/9TCfWXQE2iWnVOUaiAcBIsuFOr0LMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=eMoSmsVZ; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="eMoSmsVZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715005; x=1744251005; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SwmxmPxDMRej1aYgKsDEycIVIcKzEXYyxRg87X2L/r8=; b=eMoSmsVZXqro+/s7xCNbF4uu7coug3OJyqsARBWSXBMQQGwu/fv6oAeS Lli+IAfPPO8VxJwiA47DJ1pemlBKzSWcw+bMOz7cCGMIHn/lKslWuZnPq WCp14pJh7gegjHHfNxuv1jckrIoMxwQq0Sfj+R22HvQeOEZXcAXZfoO0+ TiSpI8XpTet0PVHfYSOCgYf8qK1H3/V623wmPY5TiowgHvfNee5hfv4eE R9ML0Pw/oBHqxAFB4p6/yP17Wtri5TtgvGijMFbKRMtJalhzgh/909T1h cjQXDRQtNqReoZZOP5h3Jn7XXt4R68w37Z9rvaeDzB2AaqLyZkh2lAq67 Q==; X-CSE-ConnectionGUID: Og8eijmeRoma2IL3TdNEmQ== X-CSE-MsgGUID: eXHVCsYhQ1qGOhjLeB4WAQ== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918532" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918532" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:04 -0700 X-CSE-ConnectionGUID: qNPDp6isRJ+zv2EnAYlzGw== X-CSE-MsgGUID: tAoth4ETR8qPGy/4unfZzg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478882" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:01 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 01/12] iommu/vt-d: Add cache tag assignment interface Date: Wed, 10 Apr 2024 10:08:33 +0800 Message-Id: <20240410020844.253535-2-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Caching tag is a combination of tags used by the hardware to cache various translations. Whenever a mapping in a domain is changed, the IOMMU driver should invalidate the caches with the caching tags. The VT-d specification describes caching tags in section 6.2.1, Tagging of Cached Translations. Add interface to assign caching tags to an IOMMU domain when attached to a RID or PASID, and unassign caching tags when a domain is detached from a RID or PASID. All caching tags are listed in the per-domain tag list and are protected by a dedicated lock. In addition to the basic IOTLB and devTLB caching tag types, PARENT_IOTLB and PARENT_DEVTLB tag types are also introduced. These tags are used for caches that store translations for DMA accesses through a nested user domain. They are affected by changes to mappings in the parent domain. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 25 +++++ drivers/iommu/intel/cache.c | 196 +++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 31 +++++- drivers/iommu/intel/nested.c | 21 +++- drivers/iommu/intel/svm.c | 12 ++- drivers/iommu/intel/Makefile | 2 +- 6 files changed, 278 insertions(+), 9 deletions(-) create mode 100644 drivers/iommu/intel/cache.c diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 404d2476a877..6f6bffc60852 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -607,6 +607,9 @@ struct dmar_domain { struct list_head devices; /* all devices' list */ struct list_head dev_pasids; /* all attached pasids */ =20 + spinlock_t cache_lock; /* Protect the cache tag list */ + struct list_head cache_tags; /* Cache tag list */ + int iommu_superpage;/* Level of superpages supported: 0 =3D=3D 4KiB (no superpages), 1 =3D=3D 2MiB, 2 =3D=3D 1GiB, 3 =3D=3D 512GiB, 4 =3D=3D 1TiB */ @@ -1092,6 +1095,28 @@ struct iommu_domain *intel_nested_domain_alloc(struc= t iommu_domain *parent, const struct iommu_user_data *user_data); struct device *device_rbtree_find(struct intel_iommu *iommu, u16 rid); =20 +enum cache_tag_type { + CACHE_TAG_IOTLB, + CACHE_TAG_DEVTLB, + CACHE_TAG_PARENT_IOTLB, + CACHE_TAG_PARENT_DEVTLB, +}; + +struct cache_tag { + struct list_head node; + enum cache_tag_type type; + struct intel_iommu *iommu; + struct device *dev; + u16 domain_id; + ioasid_t pasid; + int users; +}; + +int cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid); +void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid); + #ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check(struct intel_iommu *iommu); int intel_svm_enable_prq(struct intel_iommu *iommu); diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c new file mode 100644 index 000000000000..debbdaeff1c4 --- /dev/null +++ b/drivers/iommu/intel/cache.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * cache.c - Intel VT-d cache invalidation + * + * Copyright (C) 2024 Intel Corporation + * + * Author: Lu Baolu + */ + +#define pr_fmt(fmt) "DMAR: " fmt + +#include +#include +#include +#include + +#include "iommu.h" +#include "pasid.h" + +/* Check if an existing cache tag can be reused for a new association. */ +static bool cache_tage_match(struct cache_tag *tag, u16 domain_id, + struct intel_iommu *iommu, struct device *dev, + ioasid_t pasid, enum cache_tag_type type) +{ + if (tag->type !=3D type) + return false; + + if (tag->domain_id !=3D domain_id || tag->pasid !=3D pasid) + return false; + + if (type =3D=3D CACHE_TAG_IOTLB || type =3D=3D CACHE_TAG_PARENT_IOTLB) + return tag->iommu =3D=3D iommu; + + if (type =3D=3D CACHE_TAG_DEVTLB || type =3D=3D CACHE_TAG_PARENT_DEVTLB) + return tag->dev =3D=3D dev; + + return false; +} + +/* Assign a cache tag with specified type to domain. */ +static int cache_tag_assign(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid, + enum cache_tag_type type) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct intel_iommu *iommu =3D info->iommu; + struct cache_tag *tag, *temp; + unsigned long flags; + + tag =3D kzalloc(sizeof(*tag), GFP_KERNEL); + if (!tag) + return -ENOMEM; + + tag->type =3D type; + tag->iommu =3D iommu; + tag->domain_id =3D did; + tag->pasid =3D pasid; + tag->users =3D 1; + + if (type =3D=3D CACHE_TAG_DEVTLB || type =3D=3D CACHE_TAG_PARENT_DEVTLB) + tag->dev =3D dev; + else + tag->dev =3D iommu->iommu.dev; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(temp, &domain->cache_tags, node) { + if (cache_tage_match(temp, did, iommu, dev, pasid, type)) { + temp->users++; + spin_unlock_irqrestore(&domain->cache_lock, flags); + kfree(tag); + return 0; + } + } + list_add_tail(&tag->node, &domain->cache_tags); + spin_unlock_irqrestore(&domain->cache_lock, flags); + + return 0; +} + +/* Unassign a cache tag with specified type from domain. */ +static void cache_tag_unassign(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid, + enum cache_tag_type type) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct intel_iommu *iommu =3D info->iommu; + struct cache_tag *tag; + unsigned long flags; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + if (cache_tage_match(tag, did, iommu, dev, pasid, type)) { + if (--tag->users =3D=3D 0) { + list_del(&tag->node); + kfree(tag); + } + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +static int __cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + int ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_IOTLB); + if (ret || !info->ats_enabled) + return ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_DEVTLB); + if (ret) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_IOTLB); + + return ret; +} + +static void __cache_tag_unassign_domain(struct dmar_domain *domain, u16 di= d, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_IOTLB); + + if (info->ats_enabled) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_DEVTLB); +} + +static int __cache_tag_assign_parent_domain(struct dmar_domain *domain, u1= 6 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + int ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_PARENT_IOTLB); + if (ret || !info->ats_enabled) + return ret; + + ret =3D cache_tag_assign(domain, did, dev, pasid, CACHE_TAG_PARENT_DEVTLB= ); + if (ret) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_PARENT_IOTLB); + + return ret; +} + +static void __cache_tag_unassign_parent_domain(struct dmar_domain *domain,= u16 did, + struct device *dev, ioasid_t pasid) +{ + struct device_domain_info *info =3D dev_iommu_priv_get(dev); + + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_PARENT_IOTLB); + + if (info->ats_enabled) + cache_tag_unassign(domain, did, dev, pasid, CACHE_TAG_PARENT_DEVTLB); +} + +/* + * Assign cache tags to a domain when it's associated with a device's + * PASID using a specific domain ID. + * + * On success (return value of 0), cache tags are created and added to the + * domain's cache tag list. On failure (negative return value), an error + * code is returned indicating the reason for the failure. + */ +int cache_tag_assign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + int ret; + + ret =3D __cache_tag_assign_domain(domain, did, dev, pasid); + if (ret || domain->domain.type !=3D IOMMU_DOMAIN_NESTED) + return ret; + + ret =3D __cache_tag_assign_parent_domain(domain->s2_domain, did, dev, pas= id); + if (ret) + __cache_tag_unassign_domain(domain, did, dev, pasid); + + return ret; +} + +/* + * Remove the cache tags associated with a device's PASID when the domain = is + * detached from the device. + * + * The cache tags must be previously assigned to the domain by calling the + * assign interface. + */ +void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, + struct device *dev, ioasid_t pasid) +{ + __cache_tag_unassign_domain(domain, did, dev, pasid); + if (domain->domain.type =3D=3D IOMMU_DOMAIN_NESTED) + __cache_tag_unassign_parent_domain(domain->s2_domain, did, dev, pasid); +} diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 681789b1258d..17b8ce4b9154 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1745,7 +1745,9 @@ static struct dmar_domain *alloc_domain(unsigned int = type) domain->has_iotlb_device =3D false; INIT_LIST_HEAD(&domain->devices); INIT_LIST_HEAD(&domain->dev_pasids); + INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->lock); + spin_lock_init(&domain->cache_lock); xa_init(&domain->iommu_array); =20 return domain; @@ -2317,11 +2319,20 @@ static int dmar_domain_attach_device(struct dmar_do= main *domain, struct device_domain_info *info =3D dev_iommu_priv_get(dev); struct intel_iommu *iommu =3D info->iommu; unsigned long flags; + u16 did; int ret; =20 ret =3D domain_attach_iommu(domain, iommu); if (ret) return ret; + + did =3D domain_id_iommu(domain, iommu); + ret =3D cache_tag_assign_domain(domain, did, dev, IOMMU_NO_PASID); + if (ret) { + domain_detach_iommu(domain, iommu); + return ret; + } + info->domain =3D domain; spin_lock_irqsave(&domain->lock, flags); list_add(&info->link, &domain->devices); @@ -3793,6 +3804,7 @@ void device_block_translation(struct device *dev) struct device_domain_info *info =3D dev_iommu_priv_get(dev); struct intel_iommu *iommu =3D info->iommu; unsigned long flags; + u16 did; =20 iommu_disable_pci_caps(info); if (!dev_is_real_dma_subdevice(dev)) { @@ -3810,6 +3822,8 @@ void device_block_translation(struct device *dev) list_del(&info->link); spin_unlock_irqrestore(&info->domain->lock, flags); =20 + did =3D domain_id_iommu(info->domain, iommu); + cache_tag_unassign_domain(info->domain, did, dev, IOMMU_NO_PASID); domain_detach_iommu(info->domain, iommu); info->domain =3D NULL; } @@ -4590,10 +4604,12 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) struct dmar_domain *dmar_domain; struct iommu_domain *domain; unsigned long flags; + u16 did; =20 domain =3D iommu_get_domain_for_dev_pasid(dev, pasid, 0); if (WARN_ON_ONCE(!domain)) goto out_tear_down; + dmar_domain =3D to_dmar_domain(domain); =20 /* * The SVA implementation needs to handle its own stuffs like the mm @@ -4602,10 +4618,11 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) */ if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { intel_svm_remove_dev_pasid(dev, pasid); + cache_tag_unassign_domain(dmar_domain, + FLPT_DEFAULT_DID, dev, pasid); goto out_tear_down; } =20 - dmar_domain =3D to_dmar_domain(domain); spin_lock_irqsave(&dmar_domain->lock, flags); list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) { if (curr->dev =3D=3D dev && curr->pasid =3D=3D pasid) { @@ -4617,6 +4634,8 @@ static void intel_iommu_remove_dev_pasid(struct devic= e *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 + did =3D domain_id_iommu(dmar_domain, iommu); + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); domain_detach_iommu(dmar_domain, iommu); intel_iommu_debugfs_remove_dev_pasid(dev_pasid); kfree(dev_pasid); @@ -4633,6 +4652,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, struct intel_iommu *iommu =3D info->iommu; struct dev_pasid_info *dev_pasid; unsigned long flags; + u16 did; int ret; =20 if (!pasid_supported(iommu) || dev_is_real_dma_subdevice(dev)) @@ -4656,6 +4676,11 @@ static int intel_iommu_set_dev_pasid(struct iommu_do= main *domain, if (ret) goto out_free; =20 + did =3D domain_id_iommu(dmar_domain, iommu); + ret =3D cache_tag_assign_domain(dmar_domain, did, dev, pasid); + if (ret) + goto out_detach_iommu; + if (domain_type_is_si(dmar_domain)) ret =3D intel_pasid_setup_pass_through(iommu, dev, pasid); else if (dmar_domain->use_first_level) @@ -4665,7 +4690,7 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, ret =3D intel_pasid_setup_second_level(iommu, dmar_domain, dev, pasid); if (ret) - goto out_detach_iommu; + goto out_unassign_tag; =20 dev_pasid->dev =3D dev; dev_pasid->pasid =3D pasid; @@ -4677,6 +4702,8 @@ static int intel_iommu_set_dev_pasid(struct iommu_dom= ain *domain, intel_iommu_debugfs_create_dev_pasid(dev_pasid); =20 return 0; +out_unassign_tag: + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); out_detach_iommu: domain_detach_iommu(dmar_domain, iommu); out_free: diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index a7d68f3d518a..85c744099558 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -26,6 +26,7 @@ static int intel_nested_attach_dev(struct iommu_domain *d= omain, struct intel_iommu *iommu =3D info->iommu; unsigned long flags; int ret =3D 0; + u16 did; =20 if (info->domain) device_block_translation(dev); @@ -52,13 +53,15 @@ static int intel_nested_attach_dev(struct iommu_domain = *domain, return ret; } =20 + did =3D domain_id_iommu(dmar_domain, iommu); + ret =3D cache_tag_assign_domain(dmar_domain, did, dev, IOMMU_NO_PASID); + if (ret) + goto detach_iommu; + ret =3D intel_pasid_setup_nested(iommu, dev, IOMMU_NO_PASID, dmar_domain); - if (ret) { - domain_detach_iommu(dmar_domain, iommu); - dev_err_ratelimited(dev, "Failed to setup pasid entry\n"); - return ret; - } + if (ret) + goto unassign_tag; =20 info->domain =3D dmar_domain; spin_lock_irqsave(&dmar_domain->lock, flags); @@ -68,6 +71,12 @@ static int intel_nested_attach_dev(struct iommu_domain *= domain, domain_update_iotlb(dmar_domain); =20 return 0; +unassign_tag: + cache_tag_unassign_domain(dmar_domain, did, dev, IOMMU_NO_PASID); +detach_iommu: + domain_detach_iommu(dmar_domain, iommu); + + return ret; } =20 static void intel_nested_domain_free(struct iommu_domain *domain) @@ -206,7 +215,9 @@ struct iommu_domain *intel_nested_domain_alloc(struct i= ommu_domain *parent, domain->domain.type =3D IOMMU_DOMAIN_NESTED; INIT_LIST_HEAD(&domain->devices); INIT_LIST_HEAD(&domain->dev_pasids); + INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->lock); + spin_lock_init(&domain->cache_lock); xa_init(&domain->iommu_array); =20 spin_lock(&s2_domain->s1_lock); diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index c1bed89b1026..d706226e84ee 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -366,17 +366,25 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, sdev->qdep =3D 0; } =20 + ret =3D cache_tag_assign_domain(to_dmar_domain(domain), + FLPT_DEFAULT_DID, dev, pasid); + if (ret) + goto free_sdev; + /* Setup the pasid table: */ sflags =3D cpu_feature_enabled(X86_FEATURE_LA57) ? PASID_FLAG_FL5LP : 0; ret =3D intel_pasid_setup_first_level(iommu, dev, mm->pgd, pasid, FLPT_DEFAULT_DID, sflags); if (ret) - goto free_sdev; + goto unassign_tag; =20 list_add_rcu(&sdev->list, &svm->devs); =20 return 0; =20 +unassign_tag: + cache_tag_unassign_domain(to_dmar_domain(domain), + FLPT_DEFAULT_DID, dev, pasid); free_sdev: kfree(sdev); free_svm: @@ -795,6 +803,8 @@ struct iommu_domain *intel_svm_domain_alloc(void) if (!domain) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; + INIT_LIST_HEAD(&domain->cache_tags); + spin_lock_init(&domain->cache_lock); =20 return &domain->domain; } diff --git a/drivers/iommu/intel/Makefile b/drivers/iommu/intel/Makefile index 5402b699a122..c8beb0281559 100644 --- a/drivers/iommu/intel/Makefile +++ b/drivers/iommu/intel/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_DMAR_TABLE) +=3D dmar.o -obj-$(CONFIG_INTEL_IOMMU) +=3D iommu.o pasid.o nested.o +obj-$(CONFIG_INTEL_IOMMU) +=3D iommu.o pasid.o nested.o cache.o obj-$(CONFIG_DMAR_TABLE) +=3D trace.o cap_audit.o obj-$(CONFIG_DMAR_PERF) +=3D perf.o obj-$(CONFIG_INTEL_IOMMU_DEBUGFS) +=3D debugfs.o --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C9C78F6C for ; Wed, 10 Apr 2024 02:10:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715009; cv=none; b=UfXJKuR73448cR7pNK3RnZnwFXXbKwJ0cQhJp422DwskPsAyXVq8PRd0FCQjvIopkCm18bYPTlvdVgHriVn+A8hcw1fnSe9WZUFt9a6vuhyYZaL2OaGk2OwtYiTZUAVJFi8D2iE5pJrjTdd7hdlU8zCxiWuu0oR09kt7uilTwtQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715009; c=relaxed/simple; bh=tgJam1AA8RhybLELFvuMJfu2ggMrfQ1btbDCF0y2Kdk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gbfRvfNpUJGbF8rDl4gkT9w5skOAUhKcJ3Eqd4os5hDDbmgb5gjE2Z6RaexYrR1Bg+Roo1Qlof0R7C7p7c0HQMfTFB2aJb4vY9URmF1N/AbhnD9MGzqmK6+qeWruLVzo5NnDqDftrBInOYH/us4ZhfTK4Y6Mb4FUpPEIVHlZtc4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Y8HsRsfq; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Y8HsRsfq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715008; x=1744251008; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tgJam1AA8RhybLELFvuMJfu2ggMrfQ1btbDCF0y2Kdk=; b=Y8HsRsfqp2ztga+oPv4NU1YPzS95IruZC5Hccz9Bl8iusQ3YnkxhnVa4 5MQ7Y52DsAmg8YPDFuXu1G+WcPcRO663LSo5ZoQziEukk4/V7rkqF1Hm9 tgT71j3J5uU/JmRppMwtKSXEgqfL2YmwE1RWMJoMSgy7kK7czzAIJCDRw FV1yjxd66W4Op+sjERIyZpBETz7UWoX4rwpfHQpJpthlyDYi+ZXmbj7Gq eNd05kGkKPEj8nUxCIxeJ3PB1Aq3jaRBq/44oB9F3Fp0UugA2LWckW46Q JfV3BNnwW2jXTr6KxImZ4rKbUso6KkQgT/U6zg5VM8ax3fyPWH83eJll1 g==; X-CSE-ConnectionGUID: xmIhYvh3SKSgNdk5+Iif1Q== X-CSE-MsgGUID: PmE94UdCSOSwo+4BCZ+BVw== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918539" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918539" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:07 -0700 X-CSE-ConnectionGUID: wVV8b4STSGO20CXD1PexxA== X-CSE-MsgGUID: MYLMYFFTScqj16e31lr/2g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478888" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:04 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 02/12] iommu/vt-d: Add cache tag invalidation helpers Date: Wed, 10 Apr 2024 10:08:34 +0800 Message-Id: <20240410020844.253535-3-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add several helpers to invalidate the caches after mappings in the affected domain are changed. - cache_tag_flush_range() invalidates a range of caches after mappings within this range are changed. It uses the page-selective cache invalidation methods. - cache_tag_flush_all() invalidates all caches tagged by a domain ID. It uses the domain-selective cache invalidation methods. - cache_tag_flush_range_np() invalidates a range of caches when new mappings are created in the domain and the corresponding page table entries change from non-present to present. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 14 +++ drivers/iommu/intel/cache.c | 195 ++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.c | 12 --- 3 files changed, 209 insertions(+), 12 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 6f6bffc60852..700574421b51 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -35,6 +35,8 @@ #define VTD_PAGE_MASK (((u64)-1) << VTD_PAGE_SHIFT) #define VTD_PAGE_ALIGN(addr) (((addr) + VTD_PAGE_SIZE - 1) & VTD_PAGE_MASK) =20 +#define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) + #define VTD_STRIDE_SHIFT (9) #define VTD_STRIDE_MASK (((u64)-1) << VTD_STRIDE_SHIFT) =20 @@ -1041,6 +1043,13 @@ static inline void context_set_sm_pre(struct context= _entry *context) context->lo |=3D BIT_ULL(4); } =20 +/* Returns a number of VTD pages, but aligned to MM page size */ +static inline unsigned long aligned_nrpages(unsigned long host_addr, size_= t size) +{ + host_addr &=3D ~PAGE_MASK; + return PAGE_ALIGN(host_addr + size) >> VTD_PAGE_SHIFT; +} + /* Convert value to context PASID directory size field coding. */ #define context_pdts(pds) (((pds) & 0x7) << 9) =20 @@ -1116,6 +1125,11 @@ int cache_tag_assign_domain(struct dmar_domain *doma= in, u16 did, struct device *dev, ioasid_t pasid); void cache_tag_unassign_domain(struct dmar_domain *domain, u16 did, struct device *dev, ioasid_t pasid); +void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, + unsigned long end, int ih); +void cache_tag_flush_all(struct dmar_domain *domain); +void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long st= art, + unsigned long end); =20 #ifdef CONFIG_INTEL_IOMMU_SVM void intel_svm_check(struct intel_iommu *iommu); diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index debbdaeff1c4..b2270dc8a765 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -12,6 +12,7 @@ #include #include #include +#include #include =20 #include "iommu.h" @@ -194,3 +195,197 @@ void cache_tag_unassign_domain(struct dmar_domain *do= main, u16 did, if (domain->domain.type =3D=3D IOMMU_DOMAIN_NESTED) __cache_tag_unassign_parent_domain(domain->s2_domain, did, dev, pasid); } + +static unsigned long calculate_psi_aligned_address(unsigned long start, + unsigned long end, + unsigned long *_pages, + unsigned long *_mask) +{ + unsigned long pages =3D aligned_nrpages(start, end - start + 1); + unsigned long aligned_pages =3D __roundup_pow_of_two(pages); + unsigned long bitmask =3D aligned_pages - 1; + unsigned long mask =3D ilog2(aligned_pages); + unsigned long pfn =3D IOVA_PFN(start); + + /* + * PSI masks the low order bits of the base address. If the + * address isn't aligned to the mask, then compute a mask value + * needed to ensure the target range is flushed. + */ + if (unlikely(bitmask & pfn)) { + unsigned long end_pfn =3D pfn + pages - 1, shared_bits; + + /* + * Since end_pfn <=3D pfn + bitmask, the only way bits + * higher than bitmask can differ in pfn and end_pfn is + * by carrying. This means after masking out bitmask, + * high bits starting with the first set bit in + * shared_bits are all equal in both pfn and end_pfn. + */ + shared_bits =3D ~(pfn ^ end_pfn) & ~bitmask; + mask =3D shared_bits ? __ffs(shared_bits) : BITS_PER_LONG; + } + + *_pages =3D aligned_pages; + *_mask =3D mask; + + return ALIGN_DOWN(start, VTD_PAGE_SIZE); +} + +/* + * Invalidates a range of IOVA from @start (inclusive) to @end (inclusive) + * when the memory mappings in the target domain have been modified. + */ +void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, + unsigned long end, int ih) +{ + unsigned long pages, mask, addr; + struct cache_tag *tag; + unsigned long flags; + + addr =3D calculate_psi_aligned_address(start, end, &pages, &mask); + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + switch (tag->type) { + case CACHE_TAG_IOTLB: + case CACHE_TAG_PARENT_IOTLB: + if (domain->use_first_level) { + qi_flush_piotlb(iommu, tag->domain_id, + tag->pasid, addr, pages, ih); + } else { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + addr | ih, mask, + DMA_TLB_PSI_FLUSH); + } + break; + case CACHE_TAG_PARENT_DEVTLB: + /* + * Address translation cache in device side caches the + * result of nested translation. There is no easy way + * to identify the exact set of nested translations + * affected by a change in S2. So just flush the entire + * device cache. + */ + addr =3D 0; + mask =3D MAX_AGAW_PFN_WIDTH; + fallthrough; + case CACHE_TAG_DEVTLB: + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + if (tag->pasid =3D=3D IOMMU_NO_PASID) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr, mask); + else + qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, + tag->pasid, info->ats_qdep, + addr, mask); + + quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +/* + * Invalidates all ranges of IOVA when the memory mappings in the target + * domain have been modified. + */ +void cache_tag_flush_all(struct dmar_domain *domain) +{ + struct cache_tag *tag; + unsigned long flags; + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + switch (tag->type) { + case CACHE_TAG_IOTLB: + case CACHE_TAG_PARENT_IOTLB: + if (domain->use_first_level) + qi_flush_piotlb(iommu, tag->domain_id, + tag->pasid, 0, -1, 0); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + break; + case CACHE_TAG_DEVTLB: + case CACHE_TAG_PARENT_DEVTLB: + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, + IOMMU_NO_PASID, info->ats_qdep); + break; + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} + +/* + * Invalidate a range of IOVA when new mappings are created in the target + * domain. + * + * - VT-d spec, Section 6.1 Caching Mode: When the CM field is reported as + * Set, any software updates to remapping structures other than first- + * stage mapping requires explicit invalidation of the caches. + * - VT-d spec, Section 6.8 Write Buffer Flushing: For hardware that requi= res + * write buffer flushing, software must explicitly perform write-buffer + * flushing, if cache invalidation is not required. + */ +void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long st= art, + unsigned long end) +{ + unsigned long pages, mask, addr; + struct cache_tag *tag; + unsigned long flags; + + addr =3D calculate_psi_aligned_address(start, end, &pages, &mask); + + spin_lock_irqsave(&domain->cache_lock, flags); + list_for_each_entry(tag, &domain->cache_tags, node) { + struct intel_iommu *iommu =3D tag->iommu; + + if (!cap_caching_mode(iommu->cap) || domain->use_first_level) { + iommu_flush_write_buffer(iommu); + continue; + } + + if (tag->type =3D=3D CACHE_TAG_IOTLB || + tag->type =3D=3D CACHE_TAG_PARENT_IOTLB) { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap)) + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + addr, mask, + DMA_TLB_PSI_FLUSH); + } + } + spin_unlock_irqrestore(&domain->cache_lock, flags); +} diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 17b8ce4b9154..4572624a275e 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -54,11 +54,6 @@ __DOMAIN_MAX_PFN(gaw), (unsigned long)-1)) #define DOMAIN_MAX_ADDR(gaw) (((uint64_t)__DOMAIN_MAX_PFN(gaw)) << VTD_PAG= E_SHIFT) =20 -/* IO virtual address start page frame number */ -#define IOVA_START_PFN (1) - -#define IOVA_PFN(addr) ((addr) >> PAGE_SHIFT) - static void __init check_tylersburg_isoch(void); static int rwbf_quirk; =20 @@ -1985,13 +1980,6 @@ domain_context_mapping(struct dmar_domain *domain, s= truct device *dev) domain_context_mapping_cb, domain); } =20 -/* Returns a number of VTD pages, but aligned to MM page size */ -static unsigned long aligned_nrpages(unsigned long host_addr, size_t size) -{ - host_addr &=3D ~PAGE_MASK; - return PAGE_ALIGN(host_addr + size) >> VTD_PAGE_SHIFT; -} - /* Return largest possible superpage level for a given mapping */ static int hardware_largepage_caps(struct dmar_domain *domain, unsigned lo= ng iov_pfn, unsigned long phy_pfn, unsigned long pages) --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C337BA2B for ; Wed, 10 Apr 2024 02:10:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715011; cv=none; b=lCkSq0dtg8KXgx4izalhXfS3jTL1o4y7nqrh4hWmfWOMc//dXaO6o/PMGLei5/xdj0ZnGu0wcb5UQDbmyqdNV5aSoxeDUi36uNiyWYo5oMS1xjmAlngGz+tkDJS6MNKdhG8scKCF2tdv2fj6sW6JXCuWllt3zxGuCISLVGc2Qp4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715011; c=relaxed/simple; bh=8teXHYBsF3B5aBTrIvC+ZuvihQv1TsbNjGe1zr9zOhQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rmYX81d/VMhVUx2bYpb2v/ecscNjANzFmuyArlPy8KzFaG1n4LD20oRdjt+wCGrp/zS/VklpxUeO7j5pmVkZ7Hdh1iEi36CfFJMf+41qNNYLXWppUnSL/UX3ChxYD9bRB7z02Nv4Wtk56usnWIIkDzIaWCsAcXWOSG/UTo1ESDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=m9SDNOK1; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="m9SDNOK1" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715010; x=1744251010; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8teXHYBsF3B5aBTrIvC+ZuvihQv1TsbNjGe1zr9zOhQ=; b=m9SDNOK1pvbu0BzL+6cf0JKBL6aEbq7WalsSha6QWJYl5brBm5TLlARm SmGu/Ux3KIQQz9SPZDDdy/DDb0X6JvgTsv61Sx8lfx7ZkYZFv6SJ7oLou b425fEccipyWNBbSdgrX34C1cLEpAKay41/Q8nRT+P2QC7UZzOsu/B2HJ dxVhaL1Zg9pHf1hZrlsnZVnM292hjBQQz/WO61kAwPuJhpC7bAohVs4Ht itLPjAm7Fp97zxRdXbU4rMjikWVYlU6LwOsDT6kOUSa1RyEJPLeT5hJ3x 15QOm6BO65e9Zr9L3Esc8ZeCnxQo4VnobJdy/1IpmjbsCTWK8ILpX1wfp g==; X-CSE-ConnectionGUID: GoG7ZsojTvuxVaNJNwmb4w== X-CSE-MsgGUID: gn0ci7z7RgCb9meyXh86AA== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918546" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918546" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:10 -0700 X-CSE-ConnectionGUID: X9eowHwPT1y6H7UagGgh9g== X-CSE-MsgGUID: l5XKhdArR9WjweJEXtSTBQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478892" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:07 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 03/12] iommu/vt-d: Add trace events for cache tag interface Date: Wed, 10 Apr 2024 10:08:35 +0800 Message-Id: <20240410020844.253535-4-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add trace events for cache tag assign/unassign/flush operations and trace the events in the interfaces. These trace events will improve debugging capabilities by providing detailed information about cache tag activity. A sample of the traced messages looks like below [messages have been stripped and wrapped to make the line short]. cache_tag_assign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1 cache_tag_assign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1 cache_tag_flush_all: dmar6/0000:8a:00.0 type iotlb did 7 pasid 0 ref 1 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeab00000-0xeab1afff] addr 0xeab00000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeab20000-0xeab31fff] addr 0xeab20000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xeaa40000-0xeaa51fff] addr 0xeaa40000 pages 0x20 mask 0x5 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0x98de0000-0x98de4fff] addr 0x98de0000 pages 0x8 mask 0x3 cache_tag_flush_range: dmar1 0000:00:1b.0[0] type iotlb did 9 [0xe9828000-0xe9828fff] addr 0xe9828000 pages 0x1 mask 0x0 cache_tag_unassign: dmar9/0000:00:01.0 type iotlb did 1 pasid 9 ref 1 cache_tag_unassign: dmar9/0000:00:01.0 type devtlb did 1 pasid 9 ref 1 Signed-off-by: Lu Baolu --- drivers/iommu/intel/trace.h | 97 +++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/cache.c | 10 ++++ 2 files changed, 107 insertions(+) diff --git a/drivers/iommu/intel/trace.h b/drivers/iommu/intel/trace.h index 93d96f93a89b..c34cef760f0b 100644 --- a/drivers/iommu/intel/trace.h +++ b/drivers/iommu/intel/trace.h @@ -89,6 +89,103 @@ TRACE_EVENT(prq_report, __entry->dw1, __entry->dw2, __entry->dw3) ) ); + +DECLARE_EVENT_CLASS(cache_tag_log, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag), + TP_STRUCT__entry( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) + __field(u32, users) + ), + TP_fast_assign( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type =3D tag->type; + __entry->domain_id =3D tag->domain_id; + __entry->pasid =3D tag->pasid; + __entry->users =3D tag->users; + ), + TP_printk("%s/%s type %s did %d pasid %d ref %d", + __get_str(iommu), __get_str(dev), + __print_symbolic(__entry->type, + { CACHE_TAG_IOTLB, "iotlb" }, + { CACHE_TAG_DEVTLB, "devtlb" }, + { CACHE_TAG_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->pasid, __entry->users + ) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_assign, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_unassign, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DEFINE_EVENT(cache_tag_log, cache_tag_flush_all, + TP_PROTO(struct cache_tag *tag), + TP_ARGS(tag) +); + +DECLARE_EVENT_CLASS(cache_tag_flush, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask), + TP_STRUCT__entry( + __string(iommu, tag->iommu->name) + __string(dev, dev_name(tag->dev)) + __field(u16, type) + __field(u16, domain_id) + __field(u32, pasid) + __field(unsigned long, start) + __field(unsigned long, end) + __field(unsigned long, addr) + __field(unsigned long, pages) + __field(unsigned long, mask) + ), + TP_fast_assign( + __assign_str(iommu, tag->iommu->name); + __assign_str(dev, dev_name(tag->dev)); + __entry->type =3D tag->type; + __entry->domain_id =3D tag->domain_id; + __entry->pasid =3D tag->pasid; + __entry->start =3D start; + __entry->end =3D end; + __entry->addr =3D addr; + __entry->pages =3D pages; + __entry->mask =3D mask; + ), + TP_printk("%s %s[%d] type %s did %d [0x%lx-0x%lx] addr 0x%lx pages 0x%lx = mask 0x%lx", + __get_str(iommu), __get_str(dev), __entry->pasid, + __print_symbolic(__entry->type, + { CACHE_TAG_IOTLB, "iotlb" }, + { CACHE_TAG_DEVTLB, "devtlb" }, + { CACHE_TAG_PARENT_IOTLB, "parent_iotlb" }, + { CACHE_TAG_PARENT_DEVTLB, "parent_devtlb" }), + __entry->domain_id, __entry->start, __entry->end, + __entry->addr, __entry->pages, __entry->mask + ) +); + +DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask) +); + +DEFINE_EVENT(cache_tag_flush, cache_tag_flush_range_np, + TP_PROTO(struct cache_tag *tag, unsigned long start, unsigned long end, + unsigned long addr, unsigned long pages, unsigned long mask), + TP_ARGS(tag, start, end, addr, pages, mask) +); #endif /* _TRACE_INTEL_IOMMU_H */ =20 /* This part must be outside protection */ diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index b2270dc8a765..63873b17f963 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -17,6 +17,7 @@ =20 #include "iommu.h" #include "pasid.h" +#include "trace.h" =20 /* Check if an existing cache tag can be reused for a new association. */ static bool cache_tage_match(struct cache_tag *tag, u16 domain_id, @@ -69,11 +70,13 @@ static int cache_tag_assign(struct dmar_domain *domain,= u16 did, temp->users++; spin_unlock_irqrestore(&domain->cache_lock, flags); kfree(tag); + trace_cache_tag_assign(temp); return 0; } } list_add_tail(&tag->node, &domain->cache_tags); spin_unlock_irqrestore(&domain->cache_lock, flags); + trace_cache_tag_assign(tag); =20 return 0; } @@ -91,6 +94,7 @@ static void cache_tag_unassign(struct dmar_domain *domain= , u16 did, spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { if (cache_tage_match(tag, did, iommu, dev, pasid, type)) { + trace_cache_tag_unassign(tag); if (--tag->users =3D=3D 0) { list_del(&tag->node); kfree(tag); @@ -298,6 +302,8 @@ void cache_tag_flush_range(struct dmar_domain *domain, = unsigned long start, quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); break; } + + trace_cache_tag_flush_range(tag, start, end, addr, pages, mask); } spin_unlock_irqrestore(&domain->cache_lock, flags); } @@ -338,6 +344,8 @@ void cache_tag_flush_all(struct dmar_domain *domain) IOMMU_NO_PASID, info->ats_qdep); break; } + + trace_cache_tag_flush_all(tag); } spin_unlock_irqrestore(&domain->cache_lock, flags); } @@ -386,6 +394,8 @@ void cache_tag_flush_range_np(struct dmar_domain *domai= n, unsigned long start, addr, mask, DMA_TLB_PSI_FLUSH); } + + trace_cache_tag_flush_range_np(tag, start, end, addr, pages, mask); } spin_unlock_irqrestore(&domain->cache_lock, flags); } --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6C72C157 for ; Wed, 10 Apr 2024 02:10:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715014; cv=none; b=aGQGWLxd1a8fUKCXmRDOkeSWDPgMKHRuOFb5JJSEXAWEEo9m9CP4sjbAfYJmHD8pqWGxJKkeeunpSjKvRCXmZ8wndNzp2+HVqMA1VAh/5zSROyBbmJGmy0Um7bpBR8zvJitIVX+p2cfcDRTuejRsfiBIaUwieieQn1heg8uR4GI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715014; c=relaxed/simple; bh=P2L4NEAsMbZ06PB/7yZB1CdCqGYXQUk6ORyaaOMFzZM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WtmLEDtZ+g6klZktm6fwJRRJhjpeEcmO5mrs70zagRj9lNiWPCj3GFiw7FkZFF4RlCe4di1FOPB2mXzUOWJj/pnn/UAyB5H21B+jn5Ialrfzn1UIHiyjpOh3jlLIqgXommCBqt613bVxBGFhQQhqGOC4ejt0oONAA69GZVDObyc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=SSoVpZdJ; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="SSoVpZdJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715013; x=1744251013; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=P2L4NEAsMbZ06PB/7yZB1CdCqGYXQUk6ORyaaOMFzZM=; b=SSoVpZdJxdxhl4OBey3vt7vKjrWu8oQCOsBzBEQ+zANfN2LG3ZlipHCh ua5hm3wkFdcG6iP31+IKXqHBk13jK5FbY5PZFvOFCrQv0fV4usZ5I9/sJ mc3W56qj6a77ypPp4Bv8W2+SI8HFpZszpZxDwK3ImxmQE0lo5PO3SxCAi TOSZ2FeJjxX+sly7az5n9n8V+SR3OCclAT5ytFPCBLPnI37mpMXWM4jE4 l+zgDMBGQFtsMHoSYqI5CIlf6KSDJo/dk8V0l8UzRFU5HwaalD2cFgpaz wGua00jx/4G1/f9pBu2yfo1x9xYo3UeO7vmKyeiDugMyWdh9prFdx6/Y4 w==; X-CSE-ConnectionGUID: ByF9A2+cT/K4MTfuP+d50w== X-CSE-MsgGUID: JOvUIH+bSpmzSpLwdiNBvw== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918562" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918562" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:13 -0700 X-CSE-ConnectionGUID: OOjbAe5lR+KiwsAFuPf28Q== X-CSE-MsgGUID: en0+Aw15RKCzc3lcG3ZwpQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478895" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:10 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 04/12] iommu/vt-d: Use cache_tag_flush_all() in flush_iotlb_all Date: Wed, 10 Apr 2024 10:08:36 +0800 Message-Id: <20240410020844.253535-5-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The flush_iotlb_all callback is called by the iommu core to flush all caches for the affected domain. Use cache_tag_flush_all() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 4572624a275e..2af3e6f54af0 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1556,25 +1556,7 @@ static void parent_domain_flush(struct dmar_domain *= domain, =20 static void intel_flush_iotlb_all(struct iommu_domain *domain) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - struct iommu_domain_info *info; - unsigned long idx; - - xa_for_each(&dmar_domain->iommu_array, idx, info) { - struct intel_iommu *iommu =3D info->iommu; - u16 did =3D domain_id_iommu(dmar_domain, iommu); - - if (dmar_domain->use_first_level) - domain_flush_pasid_iotlb(iommu, dmar_domain, 0, -1, 0); - else - iommu->flush.flush_iotlb(iommu, did, 0, 0, - DMA_TLB_DSI_FLUSH); - - iommu_flush_dev_iotlb(dmar_domain, 0, MAX_AGAW_PFN_WIDTH); - } - - if (dmar_domain->nested_parent) - parent_domain_flush(dmar_domain, 0, -1, 0); + cache_tag_flush_all(to_dmar_domain(domain)); } =20 static void iommu_disable_protect_mem_regions(struct intel_iommu *iommu) --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B64E10A01 for ; Wed, 10 Apr 2024 02:10:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715016; cv=none; b=mODEoPhufIIEYg3IrvXvW0ZEtrYWu+QvHqt6Y77eWeV54tEz3fVNmJWpBjkXaXBDi54OjOAJ8lY5hS5DwpR5jlVLcS57I9z5RW30qUxmyQjEY3+X8cwg3H8yoUJeYbCcIdR1E9YmB5xaXhSuM3n7A5e/k2K53ZexiHmm/H8BWS8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715016; c=relaxed/simple; bh=DQeusfHho9+hT8qneJdFuRLOFHrtM4f8nTwIs8LP6XM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iRYauRncWnDVwfETYZfNV2JLtzyj52MwYIJsK40GZdrbJzbd+0VVFhr07JCdCvaPEjzixjPizfJEmRgtws/lnLSNkCvFP2JdZNg1VGYvO7d7v5lJFGB9AXfQAplhm0D4pNK8mSmFCNjhFMjXQKlN7oueD3JG1Dy8PzDNQqFfFHs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Nnlm3unx; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Nnlm3unx" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715016; x=1744251016; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DQeusfHho9+hT8qneJdFuRLOFHrtM4f8nTwIs8LP6XM=; b=Nnlm3unxFdUzJFCdaAWoaES74N7RPMR6ldzSC0Qp7BfnRTHdw7WXKlAd htwF2D4ZyQg101OfBS0et8uQcdCpxKUnCjVO70dxYbRz6TI6bmKNkPU+0 8RnWbYRz91H2idnIFLtgcg9tkUv8o4keB6NmgsFGAQeDTpAqzxP9shTAG TLgbf2ywCmcbId0JYyD0XPclqRaDjapm92Btu4XQNufGSn0XhLkFqFvqU D8O5GqL9rJiTw3cu7nWpppc26mhCfZKy0//dK5gX5R0FccaxPKur4WgEU 5Q+UGl0Ug11Io0cZKHKJTWI5HPX/WU38ABbNyfS6OoEUhJw76ALzJgqxr Q==; X-CSE-ConnectionGUID: XhXL92jPTO6SI/j3Ymy3eQ== X-CSE-MsgGUID: 3dI3yYkURkKhxdG9ph2i2Q== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918579" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918579" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:15 -0700 X-CSE-ConnectionGUID: pUFUDgUPSZSy1HxI14s1vQ== X-CSE-MsgGUID: fqTHkzSPQ6aYfE/adKddAA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478899" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:12 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 05/12] iommu/vt-d: Use cache_tag_flush_range() in tlb_sync Date: Wed, 10 Apr 2024 10:08:37 +0800 Message-Id: <20240410020844.253535-6-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The tlb_sync callback is called by the iommu core to flush a range of caches for the affected domain. Use cache_tag_flush_range() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 2af3e6f54af0..8ec69f09935d 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4101,25 +4101,8 @@ static size_t intel_iommu_unmap_pages(struct iommu_d= omain *domain, static void intel_iommu_tlb_sync(struct iommu_domain *domain, struct iommu_iotlb_gather *gather) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - unsigned long iova_pfn =3D IOVA_PFN(gather->start); - size_t size =3D gather->end - gather->start; - struct iommu_domain_info *info; - unsigned long start_pfn; - unsigned long nrpages; - unsigned long i; - - nrpages =3D aligned_nrpages(gather->start, size); - start_pfn =3D mm_to_dma_pfn_start(iova_pfn); - - xa_for_each(&dmar_domain->iommu_array, i, info) - iommu_flush_iotlb_psi(info->iommu, dmar_domain, - start_pfn, nrpages, - list_empty(&gather->freelist), 0); - - if (dmar_domain->nested_parent) - parent_domain_flush(dmar_domain, start_pfn, nrpages, - list_empty(&gather->freelist)); + cache_tag_flush_range(to_dmar_domain(domain), gather->start, + gather->end, list_empty(&gather->freelist)); put_pages_list(&gather->freelist); } =20 --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD4C8134DE for ; Wed, 10 Apr 2024 02:10:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715019; cv=none; b=kxuCQ3DWqqUM1PJvEAavEr0JirOVKM/59WOLX4+O6YAlkRTRW5m6lCVhyhubGnBN0uveo3HvBdvo+pzuDZuT8vPL9eSTzaJahpCwXd4dZNclfKPmlCLN+LzVw4XA8vESoaISxwtY0erNS2KWzCigElzTgZFpLgoFBwVaeqELkfY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715019; c=relaxed/simple; bh=dO88KW9d3p1OgtVTAKsTq/85V6RpBoSSDsD9RVywS+w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nRLC3piLmRQrrOJPFMViHT9TZSkiBPS4mU6rWCeH+tKq/KtXNL9Ol/CKZ6pyqKuDCucxtzWqDx5FKVzY3B0A6CWKWoraZ+32xXekwzXBZVtJFCea9kWQ7Wt+Yg3jJ7gF6+hpbOy3XqDljxEPRVlNQxZ7Dy+pyc36qHEbs3Z1L5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OKqjgZbq; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OKqjgZbq" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715018; x=1744251018; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dO88KW9d3p1OgtVTAKsTq/85V6RpBoSSDsD9RVywS+w=; b=OKqjgZbqf4JRlx0MXZZXFSl4NdTTuC9casBc5cT27k3voNBuwUF1/0Rz 2zTnjwxw7AkDtCq1PvZVfV3AWlVNam4GqtJucVOZjqPUi6J8v3A2wjvXo +sxLTziG4HlDj7KTQGepVb6iACOQ8NVkbLX5UTpn61vz75TyduvqO3/FY RPlM+jTymlOVdFpvamhGiisOS5WebQIGEBz0r/jJW2dYQ98JlCDDpj7O8 L2R/va1G7W5eeccpPNF/uGqgh/GGdbnx52ZNyVIGNq/Gl82w68/hRwfvt YAlUzWuied1BP9hE8EFIxWH453g5CusKfoEoxskYj3ZvMOH2PH2ISHlLW g==; X-CSE-ConnectionGUID: tCLe/U9HTxaDINWafyV8fA== X-CSE-MsgGUID: +kmxrH8nRvSQfP9HFwbySw== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918585" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918585" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:18 -0700 X-CSE-ConnectionGUID: PcONbFTAQq6ptB1lvuQtGQ== X-CSE-MsgGUID: Z8KDwvKsRM+sCncRQ7g32w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478902" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:15 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 06/12] iommu/vt-d: Use cache_tag_flush_range_np() in iotlb_sync_map Date: Wed, 10 Apr 2024 10:08:38 +0800 Message-Id: <20240410020844.253535-7-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The iotlb_sync_map callback is called by the iommu core after non-present to present mappings are created. The iommu driver uses this callback to invalidate caches if IOMMU is working in caching mode and second-only translation is used for the domain. Use cache_tag_flush_range_np() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 8ec69f09935d..ee5cac6ce4af 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1500,20 +1500,6 @@ static void iommu_flush_iotlb_psi(struct intel_iommu= *iommu, iommu_flush_dev_iotlb(domain, addr, mask); } =20 -/* Notification for newly created mappings */ -static void __mapping_notify_one(struct intel_iommu *iommu, struct dmar_do= main *domain, - unsigned long pfn, unsigned int pages) -{ - /* - * It's a non-present to present mapping. Only flush if caching mode - * and second level. - */ - if (cap_caching_mode(iommu->cap) && !domain->use_first_level) - iommu_flush_iotlb_psi(iommu, domain, pfn, pages, 0, 1); - else - iommu_flush_write_buffer(iommu); -} - /* * Flush the relevant caches in nested translation if the domain * also serves as a parent @@ -4538,14 +4524,8 @@ static bool risky_device(struct pci_dev *pdev) static int intel_iommu_iotlb_sync_map(struct iommu_domain *domain, unsigned long iova, size_t size) { - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - unsigned long pages =3D aligned_nrpages(iova, size); - unsigned long pfn =3D iova >> VTD_PAGE_SHIFT; - struct iommu_domain_info *info; - unsigned long i; + cache_tag_flush_range_np(to_dmar_domain(domain), iova, iova + size - 1); =20 - xa_for_each(&dmar_domain->iommu_array, i, info) - __mapping_notify_one(info->iommu, dmar_domain, pfn, pages); return 0; } =20 --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 738551BF3D for ; Wed, 10 Apr 2024 02:10:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715022; cv=none; b=pmLTKt5VHHOB37gp+J+ju3QiIYN2VQhPtUsV1apw30cBDaLEvEyVs3Kvf5K/EgqpjuD/HLajgrXFKG6Ytq7qbXgwBkKf1yHIEEt2Dgtyhsrke20LZ8hd7jtLVKsxJk+7/7ach8aaE3ObyO8ud/+E40nrjtNyVBo272JJc23ULhA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715022; c=relaxed/simple; bh=G3DcUZn7ozqTiyOHyEuehEJXxO9W0i86Nb5MkL37c5E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ZEp6Ms6H9bW2ljt39IA5SY2J/PAYHVGB4eAXvkgROxPBUf/9IjuFjohicBpPFiU9blOvKILsz5OD/ChvLRoQuu4vesdBKZc+4Dd5tX3ztFGE8UYP+zMVJykOAMADWYZ0hg820zi78+hBF05sMFKQeAdMyOhsodMdvJ2J6jhOtvE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=QltV5eLb; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="QltV5eLb" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715021; x=1744251021; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=G3DcUZn7ozqTiyOHyEuehEJXxO9W0i86Nb5MkL37c5E=; b=QltV5eLbFmLOjFScNfpTizRRrRpIrg/y2dD0a1glWl2oTd2rEQlm1RpU Kr/7ZJ93j+15ehmZABlynrOEBRcyR4AOlJPQ3/8i5tf8kH9cfamWNZ/gz jb6aL6hNR9Yp64CTn6PyVW23xqTB1iUgtjbEXVpjU3OTOwZ/imWgtNKjZ u+rle/UhptNnpQQxcz/wlejdy1+ZhyWqLlWLVxenRoPyroCmT9A98APhW bITB7zSMFSKTSULY9SHi200+WhUQ1L/X2k+c376yMZs/YH3NnsBfccNIb OK0HGBTlHhjJuVJ6O/Hg687EFIz6O9vqbpbjbrPCZ5sPjz2sJpGeE4eGf w==; X-CSE-ConnectionGUID: yUDq0MJARAyhPc+xJZ2FrA== X-CSE-MsgGUID: w8Zryxt1RQOQwcERHnC8yg== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918591" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918591" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:21 -0700 X-CSE-ConnectionGUID: UV1te2TlQW+hoL9kfkrcmw== X-CSE-MsgGUID: YOMmPsrpRjyFycbrUNJ/Eg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478907" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:17 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 07/12] iommu/vt-d: Cleanup use of iommu_flush_iotlb_psi() Date: Wed, 10 Apr 2024 10:08:39 +0800 Message-Id: <20240410020844.253535-8-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use cache_tag_flush_range() in switch_to_super_page() to invalidate the necessary caches when switching mappings from normal to super pages. The iommu_flush_iotlb_psi() call in intel_iommu_memory_notifier() is unnecessary since there should be no cache invalidation for the identity domain. Clean up iommu_flush_iotlb_psi() after the last call site is removed. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.c | 171 +----------------------------------- 1 file changed, 2 insertions(+), 169 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index ee5cac6ce4af..586b6a6afc23 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1389,157 +1389,6 @@ static void __iommu_flush_dev_iotlb(struct device_d= omain_info *info, quirk_extra_dev_tlb_flush(info, addr, mask, IOMMU_NO_PASID, qdep); } =20 -static void iommu_flush_dev_iotlb(struct dmar_domain *domain, - u64 addr, unsigned mask) -{ - struct dev_pasid_info *dev_pasid; - struct device_domain_info *info; - unsigned long flags; - - if (!domain->has_iotlb_device) - return; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(info, &domain->devices, link) - __iommu_flush_dev_iotlb(info, addr, mask); - - list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { - info =3D dev_iommu_priv_get(dev_pasid->dev); - - if (!info->ats_enabled) - continue; - - qi_flush_dev_iotlb_pasid(info->iommu, - PCI_DEVID(info->bus, info->devfn), - info->pfsid, dev_pasid->pasid, - info->ats_qdep, addr, - mask); - } - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void domain_flush_pasid_iotlb(struct intel_iommu *iommu, - struct dmar_domain *domain, u64 addr, - unsigned long npages, bool ih) -{ - u16 did =3D domain_id_iommu(domain, iommu); - struct dev_pasid_info *dev_pasid; - unsigned long flags; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) - qi_flush_piotlb(iommu, did, dev_pasid->pasid, addr, npages, ih); - - if (!list_empty(&domain->devices)) - qi_flush_piotlb(iommu, did, IOMMU_NO_PASID, addr, npages, ih); - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void __iommu_flush_iotlb_psi(struct intel_iommu *iommu, u16 did, - unsigned long pfn, unsigned int pages, - int ih) -{ - unsigned int aligned_pages =3D __roundup_pow_of_two(pages); - unsigned long bitmask =3D aligned_pages - 1; - unsigned int mask =3D ilog2(aligned_pages); - u64 addr =3D (u64)pfn << VTD_PAGE_SHIFT; - - /* - * PSI masks the low order bits of the base address. If the - * address isn't aligned to the mask, then compute a mask value - * needed to ensure the target range is flushed. - */ - if (unlikely(bitmask & pfn)) { - unsigned long end_pfn =3D pfn + pages - 1, shared_bits; - - /* - * Since end_pfn <=3D pfn + bitmask, the only way bits - * higher than bitmask can differ in pfn and end_pfn is - * by carrying. This means after masking out bitmask, - * high bits starting with the first set bit in - * shared_bits are all equal in both pfn and end_pfn. - */ - shared_bits =3D ~(pfn ^ end_pfn) & ~bitmask; - mask =3D shared_bits ? __ffs(shared_bits) : BITS_PER_LONG; - } - - /* - * Fallback to domain selective flush if no PSI support or - * the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, did, 0, 0, - DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, did, addr | ih, mask, - DMA_TLB_PSI_FLUSH); -} - -static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, - struct dmar_domain *domain, - unsigned long pfn, unsigned int pages, - int ih, int map) -{ - unsigned int aligned_pages =3D __roundup_pow_of_two(pages); - unsigned int mask =3D ilog2(aligned_pages); - uint64_t addr =3D (uint64_t)pfn << VTD_PAGE_SHIFT; - u16 did =3D domain_id_iommu(domain, iommu); - - if (WARN_ON(!pages)) - return; - - if (ih) - ih =3D 1 << 6; - - if (domain->use_first_level) - domain_flush_pasid_iotlb(iommu, domain, addr, pages, ih); - else - __iommu_flush_iotlb_psi(iommu, did, pfn, pages, ih); - - if (!map) - iommu_flush_dev_iotlb(domain, addr, mask); -} - -/* - * Flush the relevant caches in nested translation if the domain - * also serves as a parent - */ -static void parent_domain_flush(struct dmar_domain *domain, - unsigned long pfn, - unsigned long pages, int ih) -{ - struct dmar_domain *s1_domain; - - spin_lock(&domain->s1_lock); - list_for_each_entry(s1_domain, &domain->s1_domains, s2_link) { - struct device_domain_info *device_info; - struct iommu_domain_info *info; - unsigned long flags; - unsigned long i; - - xa_for_each(&s1_domain->iommu_array, i, info) - __iommu_flush_iotlb_psi(info->iommu, info->did, - pfn, pages, ih); - - if (!s1_domain->has_iotlb_device) - continue; - - spin_lock_irqsave(&s1_domain->lock, flags); - list_for_each_entry(device_info, &s1_domain->devices, link) - /* - * Address translation cache in device side caches the - * result of nested translation. There is no easy way - * to identify the exact set of nested translations - * affected by a change in S2. So just flush the entire - * device cache. - */ - __iommu_flush_dev_iotlb(device_info, 0, - MAX_AGAW_PFN_WIDTH); - spin_unlock_irqrestore(&s1_domain->lock, flags); - } - spin_unlock(&domain->s1_lock); -} - static void intel_flush_iotlb_all(struct iommu_domain *domain) { cache_tag_flush_all(to_dmar_domain(domain)); @@ -1984,9 +1833,7 @@ static void switch_to_super_page(struct dmar_domain *= domain, unsigned long end_pfn, int level) { unsigned long lvl_pages =3D lvl_to_nr_pages(level); - struct iommu_domain_info *info; struct dma_pte *pte =3D NULL; - unsigned long i; =20 while (start_pfn <=3D end_pfn) { if (!pte) @@ -1998,13 +1845,8 @@ static void switch_to_super_page(struct dmar_domain = *domain, start_pfn + lvl_pages - 1, level + 1); =20 - xa_for_each(&domain->iommu_array, i, info) - iommu_flush_iotlb_psi(info->iommu, domain, - start_pfn, lvl_pages, - 0, 0); - if (domain->nested_parent) - parent_domain_flush(domain, start_pfn, - lvl_pages, 0); + cache_tag_flush_range(domain, start_pfn << VTD_PAGE_SHIFT, + end_pfn << VTD_PAGE_SHIFT, 0); } =20 pte++; @@ -3376,18 +3218,9 @@ static int intel_iommu_memory_notifier(struct notifi= er_block *nb, case MEM_OFFLINE: case MEM_CANCEL_ONLINE: { - struct dmar_drhd_unit *drhd; - struct intel_iommu *iommu; LIST_HEAD(freelist); =20 domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist); - - rcu_read_lock(); - for_each_active_iommu(iommu, drhd) - iommu_flush_iotlb_psi(iommu, si_domain, - start_vpfn, mhp->nr_pages, - list_empty(&freelist), 0); - rcu_read_unlock(); put_pages_list(&freelist); } break; --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 378DA20304 for ; Wed, 10 Apr 2024 02:10:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715024; cv=none; b=Sjr7PjqiyzLHcoDMXVkuNb59DAhA/tVXAUqI0YFtY+Whi5HgC2wo3Adk3kxsVbrxZ6k/2GuHYSfB0lz03Ay7OFsH9Zwtj52EPNtITxPGoXl5DAW+fwXJPrdDyM1bwCgcISthWnSXNgVRi6Gi5vlO0osu2rMqlh/TNsq/PbcYbjw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715024; c=relaxed/simple; bh=jYNsXY9ExG8M0Ob2c88oOkriVnbsWsmB8tTBDVSAhSU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=EtCYXJbwG0m2epev4Fyiz/3sxUeAfYvTpAAEnRncX7RA59M1f+Qmnvwt2z8YWLx+Lx5rCnjtik+dxefeM+SRNOERP92veAK9DBfUxeqGTjsR0fI7spHqB2p5WgEoIbBrm6AgW9jQgw2Z96Jh3x60wxLQPHpZrwbUMgF0jhGHdAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=gbzeY6cn; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="gbzeY6cn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715024; x=1744251024; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jYNsXY9ExG8M0Ob2c88oOkriVnbsWsmB8tTBDVSAhSU=; b=gbzeY6cnW/Lz8CP5dCcv7iy9/V0VorLmj4wdn2JFzhwP7OJA5fxRpTzm cPv1liE/pEVzQNxGj36VLTYUX/Q6c0kjkODEdLbW/KDwoLa1kIVx124/R EoRcO6pnWW6NFB67BMOp45ht9Ill44l8KVAxt5mFlli5TPh+vgqLmrm7y yogAWSxrD6+KjOhvGZg239fW6UhBSDxzapTVB1WU80D0qEsxaj2XAfW2a /928ZRfeTUwGGa0Ei9+GO70N4ocKcJLymJR90vdZnVPzmq/2RjmVAn878 dHo/kUpEUliYWvLxC5S9elsbwKEH4szjLyJgAg8yPzp5EMLwfx5wh7/R2 w==; X-CSE-ConnectionGUID: djnmZpSWRoCZW2CuiyhpqA== X-CSE-MsgGUID: swP7iLRCTBukrc60DsadhA== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918597" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918597" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:23 -0700 X-CSE-ConnectionGUID: 4YiX2TjaS0i9jr50F3hCPA== X-CSE-MsgGUID: mdlYYQcDTZ63xNocZDOROw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478915" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:20 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 08/12] iommu/vt-d: Use cache_tag_flush_range() in cache_invalidate_user Date: Wed, 10 Apr 2024 10:08:40 +0800 Message-Id: <20240410020844.253535-9-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The cache_invalidate_user callback is called to invalidate a range of caches for the affected user domain. Use cache_tag_flush_range() in this callback. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 6 +++++ drivers/iommu/intel/nested.c | 50 +++--------------------------------- 2 files changed, 9 insertions(+), 47 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 700574421b51..90a300665962 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1050,6 +1050,12 @@ static inline unsigned long aligned_nrpages(unsigned= long host_addr, size_t size return PAGE_ALIGN(host_addr + size) >> VTD_PAGE_SHIFT; } =20 +/* Return a size from number of VTD pages. */ +static inline unsigned long nrpages_to_size(unsigned long npages) +{ + return npages << VTD_PAGE_SHIFT; +} + /* Convert value to context PASID directory size field coding. */ #define context_pdts(pds) (((pds) & 0x7) << 9) =20 diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index 85c744099558..ffbd8d98a3b8 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -90,50 +90,6 @@ static void intel_nested_domain_free(struct iommu_domain= *domain) kfree(dmar_domain); } =20 -static void nested_flush_dev_iotlb(struct dmar_domain *domain, u64 addr, - unsigned int mask) -{ - struct device_domain_info *info; - unsigned long flags; - u16 sid, qdep; - - spin_lock_irqsave(&domain->lock, flags); - list_for_each_entry(info, &domain->devices, link) { - if (!info->ats_enabled) - continue; - sid =3D info->bus << 8 | info->devfn; - qdep =3D info->ats_qdep; - qi_flush_dev_iotlb(info->iommu, sid, info->pfsid, - qdep, addr, mask); - quirk_extra_dev_tlb_flush(info, addr, mask, - IOMMU_NO_PASID, qdep); - } - spin_unlock_irqrestore(&domain->lock, flags); -} - -static void intel_nested_flush_cache(struct dmar_domain *domain, u64 addr, - u64 npages, bool ih) -{ - struct iommu_domain_info *info; - unsigned int mask; - unsigned long i; - - xa_for_each(&domain->iommu_array, i, info) - qi_flush_piotlb(info->iommu, - domain_id_iommu(domain, info->iommu), - IOMMU_NO_PASID, addr, npages, ih); - - if (!domain->has_iotlb_device) - return; - - if (npages =3D=3D U64_MAX) - mask =3D 64 - VTD_PAGE_SHIFT; - else - mask =3D ilog2(__roundup_pow_of_two(npages)); - - nested_flush_dev_iotlb(domain, addr, mask); -} - static int intel_nested_cache_invalidate_user(struct iommu_domain *domain, struct iommu_user_data_array *array) { @@ -166,9 +122,9 @@ static int intel_nested_cache_invalidate_user(struct io= mmu_domain *domain, break; } =20 - intel_nested_flush_cache(dmar_domain, inv_entry.addr, - inv_entry.npages, - inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); + cache_tag_flush_range(dmar_domain, inv_entry.addr, + inv_entry.addr + nrpages_to_size(inv_entry.npages) - 1, + inv_entry.flags & IOMMU_VTD_INV_FLAGS_LEAF); processed++; } =20 --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B3962747D for ; Wed, 10 Apr 2024 02:10:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715028; cv=none; b=G7zZjZd8hoWy/3SQVh8+76iYWyVdv0MWBnKxvGRaUwwYqAvL6WhaXc90FNGErowCHrkEfdwojd/tPMv7i0nu5fpvFZnwIZLDuuCwU1BbiQdkfazcoqJwBh0SCqwYf57naQikkJHcclWHfdksExsM9kNZwy64Om7uJ38Uh139+8g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715028; c=relaxed/simple; bh=SMlefkTzrwr89MMQgB2wDx1oBzU0DcBlIgzd1/OsE2c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kR1/vyOC8Vm9kj2Dx88aDrI4F0p5GdA37Eh1kbLSfvQny+0N3Mv03MrURrVWSi9ORIjBMW/+c74M3Y0RbSp7mpUKkGyIm2m8W45iyDGrs5unWAALVrOmmc44G7afJSNvUFYvDhb+YygUSe9trBjZ/KZliCGc/6LvjjQvQvVaQ2A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=jMEDTuqh; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="jMEDTuqh" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715028; x=1744251028; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=SMlefkTzrwr89MMQgB2wDx1oBzU0DcBlIgzd1/OsE2c=; b=jMEDTuqhHTsyHfdl50Se37oGp5oNcbm8p5DCocd+IqZsBqN4GRjhWo7T ZcCthcFlBmb6cN2mP+95hyK/tqF2xzhCK5bFxrXJSaSDNZN/Coq9saU7f t/lw0YTB9A6urWBritmqONaNSDDapH/DJEPpRoam5tQ15ZOzkXuO/rNq5 rNYxd5I5AfR8P8sfzn1ZhLIhRSbTYNwYXrcN38S3r5e7kOvC5W+/79lJ1 a+vU4PaLieL0U0jEY7Sot4lFtUgG2IY0w7GhWqd+dgmtMvvCrv6wKqEYg FrCubiHqs/CEWWtIxSgdni6RpXKA9jWsEg5tHYGtvoNatRSDt/9Fkjojg w==; X-CSE-ConnectionGUID: 75E1xsKIR36Y7+LEMWwsDQ== X-CSE-MsgGUID: q7Kie1/aQHO2qqRFdHyCzg== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918605" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918605" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:27 -0700 X-CSE-ConnectionGUID: nQP+uigZTh2VOKVh6sh0kw== X-CSE-MsgGUID: v/xXe3YaQXexzUg+uSaQuw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478920" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:23 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 09/12] iommu/vt-d: Use cache helpers in arch_invalidate_secondary_tlbs Date: Wed, 10 Apr 2024 10:08:41 +0800 Message-Id: <20240410020844.253535-10-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The arch_invalidate_secondary_tlbs callback is called in the SVA mm notification path. It invalidates all or a range of caches after the CPU page table is modified. Use the cache tag helps in this path. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/svm.c | 76 +++---------------------------------- 2 files changed, 6 insertions(+), 71 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 90a300665962..5a42d6ee9119 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1161,6 +1161,7 @@ struct intel_svm { struct mm_struct *mm; u32 pasid; struct list_head devs; + struct dmar_domain *domain; }; #else static inline void intel_svm_check(struct intel_iommu *iommu) {} diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index d706226e84ee..751fab476fa2 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -168,88 +168,20 @@ void intel_svm_check(struct intel_iommu *iommu) iommu->flags |=3D VTD_FLAG_SVM_CAPABLE; } =20 -static void __flush_svm_range_dev(struct intel_svm *svm, - struct intel_svm_dev *sdev, - unsigned long address, - unsigned long pages, int ih) -{ - struct device_domain_info *info =3D dev_iommu_priv_get(sdev->dev); - - if (WARN_ON(!pages)) - return; - - qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, address, pages, ih); - if (info->ats_enabled) { - qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid, - svm->pasid, sdev->qdep, address, - order_base_2(pages)); - quirk_extra_dev_tlb_flush(info, address, order_base_2(pages), - svm->pasid, sdev->qdep); - } -} - -static void intel_flush_svm_range_dev(struct intel_svm *svm, - struct intel_svm_dev *sdev, - unsigned long address, - unsigned long pages, int ih) -{ - unsigned long shift =3D ilog2(__roundup_pow_of_two(pages)); - unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + shift)); - unsigned long start =3D ALIGN_DOWN(address, align); - unsigned long end =3D ALIGN(address + (pages << VTD_PAGE_SHIFT), align); - - while (start < end) { - __flush_svm_range_dev(svm, sdev, start, align >> VTD_PAGE_SHIFT, ih); - start +=3D align; - } -} - -static void intel_flush_svm_range(struct intel_svm *svm, unsigned long add= ress, - unsigned long pages, int ih) -{ - struct intel_svm_dev *sdev; - - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) - intel_flush_svm_range_dev(svm, sdev, address, pages, ih); - rcu_read_unlock(); -} - -static void intel_flush_svm_all(struct intel_svm *svm) -{ - struct device_domain_info *info; - struct intel_svm_dev *sdev; - - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) { - info =3D dev_iommu_priv_get(sdev->dev); - - qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, 0, -1UL, 0); - if (info->ats_enabled) { - qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid, - svm->pasid, sdev->qdep, - 0, 64 - VTD_PAGE_SHIFT); - quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT, - svm->pasid, sdev->qdep); - } - } - rcu_read_unlock(); -} - /* Pages have been freed at this point */ static void intel_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end) { struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); + struct dmar_domain *domain =3D svm->domain; =20 if (start =3D=3D 0 && end =3D=3D -1UL) { - intel_flush_svm_all(svm); + cache_tag_flush_all(domain); return; } =20 - intel_flush_svm_range(svm, start, - (end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0); + cache_tag_flush_range(domain, start, end, 0); } =20 static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) @@ -336,6 +268,7 @@ static int intel_svm_set_dev_pasid(struct iommu_domain = *domain, INIT_LIST_HEAD_RCU(&svm->devs); =20 svm->notifier.ops =3D &intel_mmuops; + svm->domain =3D to_dmar_domain(domain); ret =3D mmu_notifier_register(&svm->notifier, mm); if (ret) { kfree(svm); @@ -803,6 +736,7 @@ struct iommu_domain *intel_svm_domain_alloc(void) if (!domain) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; + domain->use_first_level =3D true; INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->cache_lock); =20 --=20 2.34.1 From nobody Sat Feb 7 18:15:31 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2645A29CF2 for ; Wed, 10 Apr 2024 02:10:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715033; cv=none; b=XuoswkI+zTu2rsD26fVap/dnHTpRAaqupTpc9q8UEQkn/5EP+SpOcRGVVo2EW4KhkauJaLoNyLWcLxda48Ey7c+IvhqLUdLSAfpY0TLTvkM/MBcS6mOwzalj/+nyOP1vikF4+SuJzvpPDUp1mRC3gdZN7o/oY1yHLgLrrMC7tNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715033; c=relaxed/simple; bh=8uOaaxm9hTyzO9Aw/q0vgU9R2IfJCzDzjSxHWJxDukU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QBdx8tl3JFPd2Afad4xsB0e9HrJh3Yvg0wDi8h3v5l9edq5/do9196oLBPCLQuU9NFD3P3bz1YCwBiEOIDaV8flxHkNIn+vUHYg14zEAGG/5CLLPFLFKrGOTdYDDcMKHIBur/9t8OPMh+dZDb8zEs1DHBMamcPD15zTcBFteNZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=inInR1bC; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="inInR1bC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715033; x=1744251033; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8uOaaxm9hTyzO9Aw/q0vgU9R2IfJCzDzjSxHWJxDukU=; b=inInR1bCWJryw+TC94CvYQ1B4rp/jGOPw2dygTWuouSIduxFXQwZkW+z gHQFyErMt5lfIElkZPiMR1pP9AS2jlfQ9dP2m+FwWqKVggg50UzkXJnWJ Z0Q8WN++ry8C3ItxQZJTPG1z1IWWUU86xQsQDgTHR8koxXyy+AJWPQ/XM i2dpHVUphocC8okqUASTEBEPGYLwKHWs5+L67pSYkknIz/JFtkjK3vbG/ 2OUQ/jayh6oQtIosUoZLiPh+IGz2ffYpzkAmVR5ij821Bsbq/v6NZgy1j jZPxyz8VsgK4cEIMtFtnvQ5EqYGi4lf7qoQjecYKvUFqaghodixjyDAyD Q==; X-CSE-ConnectionGUID: gzJPafmKRBuQwNHB5RnkkA== X-CSE-MsgGUID: 2LVECzpmRuOy7mETo0SDhw== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918612" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918612" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:30 -0700 X-CSE-ConnectionGUID: 5ff/sxeVT2eCB/5ogfr3Wg== X-CSE-MsgGUID: iasNmo9gScia7A0L1BKg7Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478936" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:26 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 10/12] iommu/vt-d: Retire intel_svm_dev Date: Wed, 10 Apr 2024 10:08:42 +0800 Message-Id: <20240410020844.253535-11-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The intel_svm_dev data structure used in the sva implementation for the Intel IOMMU driver stores information about a device attached to an SVA domain. It is a duplicate of dev_pasid_info that serves the same purpose. Replace intel_svm_dev with dev_pasid_info and clean up the use of intel_svm_dev. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 15 +---- drivers/iommu/intel/iommu.c | 30 ++++----- drivers/iommu/intel/svm.c | 131 +++++++++++------------------------- 3 files changed, 55 insertions(+), 121 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 5a42d6ee9119..90611ec08a7c 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -650,6 +650,7 @@ struct dmar_domain { struct list_head s2_link; }; }; + struct intel_svm *svm; =20 struct iommu_domain domain; /* generic domain data structure for iommu core */ @@ -1144,23 +1145,13 @@ int intel_svm_finish_prq(struct intel_iommu *iommu); void intel_svm_page_response(struct device *dev, struct iopf_fault *evt, struct iommu_page_response *msg); struct iommu_domain *intel_svm_domain_alloc(void); -void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t pasid); +void intel_svm_remove_dev_pasid(struct iommu_domain *domain); void intel_drain_pasid_prq(struct device *dev, u32 pasid); =20 -struct intel_svm_dev { - struct list_head list; - struct rcu_head rcu; - struct device *dev; - struct intel_iommu *iommu; - u16 did; - u16 sid, qdep; -}; - struct intel_svm { struct mmu_notifier notifier; struct mm_struct *mm; u32 pasid; - struct list_head devs; struct dmar_domain *domain; }; #else @@ -1171,7 +1162,7 @@ static inline struct iommu_domain *intel_svm_domain_a= lloc(void) return NULL; } =20 -static inline void intel_svm_remove_dev_pasid(struct device *dev, ioasid_t= pasid) +static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain) { } #endif diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 586b6a6afc23..d7f205cd0aac 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4377,18 +4377,6 @@ static void intel_iommu_remove_dev_pasid(struct devi= ce *dev, ioasid_t pasid) goto out_tear_down; dmar_domain =3D to_dmar_domain(domain); =20 - /* - * The SVA implementation needs to handle its own stuffs like the mm - * notification. Before consolidating that code into iommu core, let - * the intel sva code handle it. - */ - if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { - intel_svm_remove_dev_pasid(dev, pasid); - cache_tag_unassign_domain(dmar_domain, - FLPT_DEFAULT_DID, dev, pasid); - goto out_tear_down; - } - spin_lock_irqsave(&dmar_domain->lock, flags); list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) { if (curr->dev =3D=3D dev && curr->pasid =3D=3D pasid) { @@ -4400,10 +4388,20 @@ static void intel_iommu_remove_dev_pasid(struct dev= ice *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 - did =3D domain_id_iommu(dmar_domain, iommu); - cache_tag_unassign_domain(dmar_domain, did, dev, pasid); - domain_detach_iommu(dmar_domain, iommu); - intel_iommu_debugfs_remove_dev_pasid(dev_pasid); + /* + * The SVA implementation needs to handle its own stuffs like the mm + * notification. Before consolidating that code into iommu core, let + * the intel sva code handle it. + */ + if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { + cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid); + intel_svm_remove_dev_pasid(domain); + } else { + did =3D domain_id_iommu(dmar_domain, iommu); + cache_tag_unassign_domain(dmar_domain, did, dev, pasid); + domain_detach_iommu(dmar_domain, iommu); + intel_iommu_debugfs_remove_dev_pasid(dev_pasid); + } kfree(dev_pasid); out_tear_down: intel_pasid_tear_down_entry(iommu, dev, pasid, false); diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 751fab476fa2..0b767d16fb71 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -43,23 +43,6 @@ static void *pasid_private_find(ioasid_t pasid) return xa_load(&pasid_private_array, pasid); } =20 -static struct intel_svm_dev * -svm_lookup_device_by_dev(struct intel_svm *svm, struct device *dev) -{ - struct intel_svm_dev *sdev =3D NULL, *t; - - rcu_read_lock(); - list_for_each_entry_rcu(t, &svm->devs, list) { - if (t->dev =3D=3D dev) { - sdev =3D t; - break; - } - } - rcu_read_unlock(); - - return sdev; -} - int intel_svm_enable_prq(struct intel_iommu *iommu) { struct iopf_queue *iopfq; @@ -187,7 +170,10 @@ static void intel_arch_invalidate_secondary_tlbs(struc= t mmu_notifier *mn, static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct intel_svm_dev *sdev; + struct dmar_domain *domain =3D svm->domain; + struct dev_pasid_info *dev_pasid; + struct device_domain_info *info; + unsigned long flags; =20 /* This might end up being called from exit_mmap(), *before* the page * tables are cleared. And __mmu_notifier_release() will delete us from @@ -201,11 +187,13 @@ static void intel_mm_release(struct mmu_notifier *mn,= struct mm_struct *mm) * page) so that we end up taking a fault that the hardware really * *has* to handle gracefully without affecting other processes. */ - rcu_read_lock(); - list_for_each_entry_rcu(sdev, &svm->devs, list) - intel_pasid_tear_down_entry(sdev->iommu, sdev->dev, - svm->pasid, true); - rcu_read_unlock(); + spin_lock_irqsave(&domain->lock, flags); + list_for_each_entry(dev_pasid, &domain->dev_pasids, link_domain) { + info =3D dev_iommu_priv_get(dev_pasid->dev); + intel_pasid_tear_down_entry(info->iommu, dev_pasid->dev, + dev_pasid->pasid, true); + } + spin_unlock_irqrestore(&domain->lock, flags); =20 } =20 @@ -214,47 +202,17 @@ static const struct mmu_notifier_ops intel_mmuops =3D= { .arch_invalidate_secondary_tlbs =3D intel_arch_invalidate_secondary_tlbs, }; =20 -static int pasid_to_svm_sdev(struct device *dev, unsigned int pasid, - struct intel_svm **rsvm, - struct intel_svm_dev **rsdev) -{ - struct intel_svm_dev *sdev =3D NULL; - struct intel_svm *svm; - - if (pasid =3D=3D IOMMU_PASID_INVALID || pasid >=3D PASID_MAX) - return -EINVAL; - - svm =3D pasid_private_find(pasid); - if (IS_ERR(svm)) - return PTR_ERR(svm); - - if (!svm) - goto out; - - /* - * If we found svm for the PASID, there must be at least one device - * bond. - */ - if (WARN_ON(list_empty(&svm->devs))) - return -EINVAL; - sdev =3D svm_lookup_device_by_dev(svm, dev); - -out: - *rsvm =3D svm; - *rsdev =3D sdev; - - return 0; -} - static int intel_svm_set_dev_pasid(struct iommu_domain *domain, struct device *dev, ioasid_t pasid) { struct device_domain_info *info =3D dev_iommu_priv_get(dev); + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); struct intel_iommu *iommu =3D info->iommu; struct mm_struct *mm =3D domain->mm; - struct intel_svm_dev *sdev; + struct dev_pasid_info *dev_pasid; struct intel_svm *svm; unsigned long sflags; + unsigned long flags; int ret =3D 0; =20 svm =3D pasid_private_find(pasid); @@ -265,7 +223,6 @@ static int intel_svm_set_dev_pasid(struct iommu_domain = *domain, =20 svm->pasid =3D pasid; svm->mm =3D mm; - INIT_LIST_HEAD_RCU(&svm->devs); =20 svm->notifier.ops =3D &intel_mmuops; svm->domain =3D to_dmar_domain(domain); @@ -283,26 +240,19 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, } } =20 - sdev =3D kzalloc(sizeof(*sdev), GFP_KERNEL); - if (!sdev) { - ret =3D -ENOMEM; + dmar_domain->svm =3D svm; + + dev_pasid =3D kzalloc(sizeof(*dev_pasid), GFP_KERNEL); + if (!dev_pasid) goto free_svm; - } =20 - sdev->dev =3D dev; - sdev->iommu =3D iommu; - sdev->did =3D FLPT_DEFAULT_DID; - sdev->sid =3D PCI_DEVID(info->bus, info->devfn); - if (info->ats_enabled) { - sdev->qdep =3D info->ats_qdep; - if (sdev->qdep >=3D QI_DEV_EIOTLB_MAX_INVS) - sdev->qdep =3D 0; - } + dev_pasid->dev =3D dev; + dev_pasid->pasid =3D pasid; =20 ret =3D cache_tag_assign_domain(to_dmar_domain(domain), FLPT_DEFAULT_DID, dev, pasid); if (ret) - goto free_sdev; + goto free_dev_pasid; =20 /* Setup the pasid table: */ sflags =3D cpu_feature_enabled(X86_FEATURE_LA57) ? PASID_FLAG_FL5LP : 0; @@ -311,17 +261,19 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, if (ret) goto unassign_tag; =20 - list_add_rcu(&sdev->list, &svm->devs); + spin_lock_irqsave(&dmar_domain->lock, flags); + list_add(&dev_pasid->link_domain, &dmar_domain->dev_pasids); + spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 return 0; =20 unassign_tag: cache_tag_unassign_domain(to_dmar_domain(domain), FLPT_DEFAULT_DID, dev, pasid); -free_sdev: - kfree(sdev); +free_dev_pasid: + kfree(dev_pasid); free_svm: - if (list_empty(&svm->devs)) { + if (list_empty(&dmar_domain->dev_pasids)) { mmu_notifier_unregister(&svm->notifier, mm); pasid_private_remove(pasid); kfree(svm); @@ -330,26 +282,17 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, return ret; } =20 -void intel_svm_remove_dev_pasid(struct device *dev, u32 pasid) +void intel_svm_remove_dev_pasid(struct iommu_domain *domain) { - struct intel_svm_dev *sdev; - struct intel_svm *svm; - struct mm_struct *mm; + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); + struct intel_svm *svm =3D dmar_domain->svm; + struct mm_struct *mm =3D domain->mm; =20 - if (pasid_to_svm_sdev(dev, pasid, &svm, &sdev)) - return; - mm =3D svm->mm; - - if (sdev) { - list_del_rcu(&sdev->list); - kfree_rcu(sdev, rcu); - - if (list_empty(&svm->devs)) { - if (svm->notifier.ops) - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(svm->pasid); - kfree(svm); - } + if (list_empty(&dmar_domain->dev_pasids)) { + if (svm->notifier.ops) + mmu_notifier_unregister(&svm->notifier, mm); + pasid_private_remove(svm->pasid); + kfree(svm); } } =20 @@ -737,8 +680,10 @@ struct iommu_domain *intel_svm_domain_alloc(void) return NULL; domain->domain.ops =3D &intel_svm_domain_ops; domain->use_first_level =3D true; + INIT_LIST_HEAD(&domain->dev_pasids); INIT_LIST_HEAD(&domain->cache_tags); spin_lock_init(&domain->cache_lock); + spin_lock_init(&domain->lock); =20 return &domain->domain; } --=20 2.34.1 From nobody Sat Feb 7 18:15:32 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B31B29D08 for ; Wed, 10 Apr 2024 02:10:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715034; cv=none; b=jIbzZJrv3hrQP3+0UXhl3NaQoyEsNYOjwd1lgS0tx2WmnYufD+qCUUbqlAFArZqYn/LEk7iq1g7ivRii9UpFWc/mLVPSR5hW8sNGZDDIBWxsYABS81zwU9iLc0lceZJLmjZ/dTjcFwQbNzQi2wmjxxfJDaB8hAbfPcA2U5ZXW1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715034; c=relaxed/simple; bh=Z4N9WjZXB3Uq7GgVWrBbRtPl6f+yBep/k+sWPTppZwU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qEe3pFndf2GQR+hSuomDl7AhiF+T+pOmnLw5SSt+LPrjEg3oy2NU0C0fglCMnFj716WA++1M9rcmORndj5aJIhuQS5/4USRBReUOw6ZZnf6NYntSlQWTKbmpF6i8KOGQcXaHzs4d0vCESA7EmKxESnMJeKSGfBA3WRIIa8h3KBU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=N0DmI/LT; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="N0DmI/LT" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715034; x=1744251034; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Z4N9WjZXB3Uq7GgVWrBbRtPl6f+yBep/k+sWPTppZwU=; b=N0DmI/LTx15oPeIC2ESylrk+w2fH6hYO5Jhj2F2DbD2ly2G7e9CqPdOW b4TYsXr7UtCxIrSmh/ANP+AKCoCIgxZpAS9mQv5YhYu9pA5TgveCCZC4z Cx8c5ozUP4/QcYxMzEB2H/1ph0W7uprt1W/Ddjc38dqC7JNmbZpmDXQrb PzCy7EPIoG0sun1vDuQ73FbLrZuv7HYNysp3CpeoK8Tw4P8WYciPi+p/r SQxTZaFx0b56qCCe1LaLi1MX/AL3KhIuE+RTUHFXORfQ9f5m1rh2q7BoE jl99ZuHv/lCLhW8qAdKZZ/hDStNc3ON+6BcxvMP5WobiV7gOg0l1H1XO9 A==; X-CSE-ConnectionGUID: FfT84J3YQDyia+aQxhXTGQ== X-CSE-MsgGUID: pWAmjWc/RbqT6JcEmJwZSA== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918623" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918623" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:33 -0700 X-CSE-ConnectionGUID: WHam9GteSOefNDx1SxfR3Q== X-CSE-MsgGUID: 0FmamW2PRGChoDBl7nx6CQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478942" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:29 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Jason Gunthorpe , Vasant Hegde , Lu Baolu Subject: [PATCH v2 11/12] iommu: Add ops->domain_alloc_sva() Date: Wed, 10 Apr 2024 10:08:43 +0800 Message-Id: <20240410020844.253535-12-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jason Gunthorpe Make a new op that receives the device and the mm_struct that the SVA domain should be created for. Unlike domain_alloc_paging() the dev argument is never NULL here. This allows drivers to fully initialize the SVA domain and allocate the mmu_notifier during allocation. It allows the notifier lifetime to follow the lifetime of the iommu_domain. Since we have only one call site, upgrade the new op to return ERR_PTR instead of NULL. Signed-off-by: Jason Gunthorpe [Removed smmu3 related changes - Vasant] Signed-off-by: Vasant Hegde Reviewed-by: Tina Zhang Link: https://lore.kernel.org/r/20240311090843.133455-15-vasant.hegde@amd.c= om Signed-off-by: Lu Baolu --- include/linux/iommu.h | 3 +++ drivers/iommu/iommu-sva.c | 16 +++++++++++----- 2 files changed, 14 insertions(+), 5 deletions(-) diff --git a/include/linux/iommu.h b/include/linux/iommu.h index 2e925b5eba53..8aabe83af8f2 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -518,6 +518,7 @@ static inline int __iommu_copy_struct_from_user_array( * Upon failure, ERR_PTR must be returned. * @domain_alloc_paging: Allocate an iommu_domain that can be used for * UNMANAGED, DMA, and DMA_FQ domain types. + * @domain_alloc_sva: Allocate an iommu_domain for Shared Virtual Addressi= ng. * @probe_device: Add device to iommu driver handling * @release_device: Remove device from iommu driver handling * @probe_finalize: Do final setup work after the device is added to an IO= MMU @@ -558,6 +559,8 @@ struct iommu_ops { struct device *dev, u32 flags, struct iommu_domain *parent, const struct iommu_user_data *user_data); struct iommu_domain *(*domain_alloc_paging)(struct device *dev); + struct iommu_domain *(*domain_alloc_sva)(struct device *dev, + struct mm_struct *mm); =20 struct iommu_device *(*probe_device)(struct device *dev); void (*release_device)(struct device *dev); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 640acc804e8c..18a35e798b72 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -108,8 +108,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *= dev, struct mm_struct *mm =20 /* Allocate a new domain and set it on device pasid. */ domain =3D iommu_sva_domain_alloc(dev, mm); - if (!domain) { - ret =3D -ENOMEM; + if (IS_ERR(domain)) { + ret =3D PTR_ERR(domain); goto out_free_handle; } =20 @@ -283,9 +283,15 @@ struct iommu_domain *iommu_sva_domain_alloc(struct dev= ice *dev, const struct iommu_ops *ops =3D dev_iommu_ops(dev); struct iommu_domain *domain; =20 - domain =3D ops->domain_alloc(IOMMU_DOMAIN_SVA); - if (!domain) - return NULL; + if (ops->domain_alloc_sva) { + domain =3D ops->domain_alloc_sva(dev, mm); + if (IS_ERR(domain)) + return domain; + } else { + domain =3D ops->domain_alloc(IOMMU_DOMAIN_SVA); + if (!domain) + return ERR_PTR(-ENOMEM); + } =20 domain->type =3D IOMMU_DOMAIN_SVA; mmgrab(mm); --=20 2.34.1 From nobody Sat Feb 7 18:15:32 2026 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93EA42AD21 for ; Wed, 10 Apr 2024 02:10:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.20 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715037; cv=none; b=ntV8AInqrAlNhOBBpZrEFnUjd6h8LA1SpXqyK6RbC50GppI2wmXRsLyQWWk3nqfmIc79KNPGYZZoTZJ3g0RL6c9NlTKQT6NzGtB0oJeFPmrcnTXD8XDlfaz9k+XiKJZhtgoYDBomWSEUXiV7zxfNTV7axzPf0xQ+2aM3rXg74xM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712715037; c=relaxed/simple; bh=PpWRvG0ufJ/UB2ZXJs4tHv+Cjj7GBTUSH9Rn4/Owdn8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=iT5x32AIa0IRKTc3vtNSA6Wo9bNpuXn0A2WqTUAMUU8WkA1bb7ssZqnDnCRZEWEX2dhmO21fXnnfN1UAY2hHYz2Qe0QY184vKVigfjDu7LOLeaq9gJ7KrQ+5JbFCe4J3c9zhfCgEfseGHx+ceH4zMUJKNW4KOAHb4X6UROv96hw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com; spf=none smtp.mailfrom=linux.intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Ad3W7ixm; arc=none smtp.client-ip=198.175.65.20 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Ad3W7ixm" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1712715036; x=1744251036; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PpWRvG0ufJ/UB2ZXJs4tHv+Cjj7GBTUSH9Rn4/Owdn8=; b=Ad3W7ixmNqlTtN1mvAV99l98YNuOSVQAa0zgrzH58fzzayeMdirTP5I/ ZSre/D2E3aK6FaxEJXeg1oq/wVLkiVvJyovROnZ3y4Cn/I7FOqYrI4aZb 6gzrf8jFiVEbir0KCa9g1A39zwiFv8MTiTeveqH2e7rcmpSLa30isfsHO GILKKGt7oUalXlMWcHjaTW/N6GTwBuONicv8Hc03VkoF9xn5aK1PFgX2c lHZxQVEiBGcrKds1VSPRSxaJgGzMViZhwJKZSxW5OIZT6CGMp7nqTZnC4 69/RWmz/eZzuL1cBQ9C02Oo2FXJWaDxel/grYy3JkPR/SEYYi3R57qMKw Q==; X-CSE-ConnectionGUID: RWWgyIHRSeeNpdFNdbTImg== X-CSE-MsgGUID: 8YD8piPXT16XniRKaMwv5g== X-IronPort-AV: E=McAfee;i="6600,9927,11039"; a="7918636" X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="7918636" Received: from fmviesa007.fm.intel.com ([10.60.135.147]) by orvoesa112.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Apr 2024 19:10:36 -0700 X-CSE-ConnectionGUID: AfN4VIDxSJKd7mqzZ36GHQ== X-CSE-MsgGUID: 0blSaMAKT5mLW20WbhnUbw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.07,190,1708416000"; d="scan'208";a="20478947" Received: from unknown (HELO allen-box.sh.intel.com) ([10.239.159.127]) by fmviesa007.fm.intel.com with ESMTP; 09 Apr 2024 19:10:32 -0700 From: Lu Baolu To: Joerg Roedel , Will Deacon , Robin Murphy , Kevin Tian , Jason Gunthorpe Cc: Tina Zhang , Yi Liu , iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Lu Baolu Subject: [PATCH v2 12/12] iommu/vt-d: Retire struct intel_svm Date: Wed, 10 Apr 2024 10:08:44 +0800 Message-Id: <20240410020844.253535-13-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240410020844.253535-1-baolu.lu@linux.intel.com> References: <20240410020844.253535-1-baolu.lu@linux.intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The struct intel_svm was used for keeping attached devices info for sva domain. Since sva domain is a kind of iommu_domain, the struct dmar_domain should centralize all info of a sva domain, including the info of attached devices. Therefore, retire struct intel_svm to clean up the code. Besides, allocate sva domain in domain_alloc_sva() callback which allows the memory management notifier lifetime to follow the lifetime of the iommu_domain. Co-developed-by: Tina Zhang Signed-off-by: Tina Zhang Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 26 ++++------ drivers/iommu/intel/iommu.c | 9 +--- drivers/iommu/intel/svm.c | 94 +++++++++---------------------------- 3 files changed, 32 insertions(+), 97 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 90611ec08a7c..0951be947ff9 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -649,8 +649,12 @@ struct dmar_domain { /* link to parent domain siblings */ struct list_head s2_link; }; + + /* SVA domain */ + struct { + struct mmu_notifier notifier; + }; }; - struct intel_svm *svm; =20 struct iommu_domain domain; /* generic domain data structure for iommu core */ @@ -1144,26 +1148,16 @@ int intel_svm_enable_prq(struct intel_iommu *iommu); int intel_svm_finish_prq(struct intel_iommu *iommu); void intel_svm_page_response(struct device *dev, struct iopf_fault *evt, struct iommu_page_response *msg); -struct iommu_domain *intel_svm_domain_alloc(void); -void intel_svm_remove_dev_pasid(struct iommu_domain *domain); +struct iommu_domain *intel_svm_domain_alloc(struct device *dev, + struct mm_struct *mm); void intel_drain_pasid_prq(struct device *dev, u32 pasid); - -struct intel_svm { - struct mmu_notifier notifier; - struct mm_struct *mm; - u32 pasid; - struct dmar_domain *domain; -}; #else static inline void intel_svm_check(struct intel_iommu *iommu) {} static inline void intel_drain_pasid_prq(struct device *dev, u32 pasid) {} -static inline struct iommu_domain *intel_svm_domain_alloc(void) -{ - return NULL; -} - -static inline void intel_svm_remove_dev_pasid(struct iommu_domain *domain) +static inline struct iommu_domain *intel_svm_domain_alloc(struct device *d= ev, + struct mm_struct *mm) { + return ERR_PTR(-ENODEV); } #endif =20 diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index d7f205cd0aac..b8a0f759a24f 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -3680,8 +3680,6 @@ static struct iommu_domain *intel_iommu_domain_alloc(= unsigned type) return domain; case IOMMU_DOMAIN_IDENTITY: return &si_domain->domain; - case IOMMU_DOMAIN_SVA: - return intel_svm_domain_alloc(); default: return NULL; } @@ -4388,14 +4386,8 @@ static void intel_iommu_remove_dev_pasid(struct devi= ce *dev, ioasid_t pasid) WARN_ON_ONCE(!dev_pasid); spin_unlock_irqrestore(&dmar_domain->lock, flags); =20 - /* - * The SVA implementation needs to handle its own stuffs like the mm - * notification. Before consolidating that code into iommu core, let - * the intel sva code handle it. - */ if (domain->type =3D=3D IOMMU_DOMAIN_SVA) { cache_tag_unassign_domain(dmar_domain, FLPT_DEFAULT_DID, dev, pasid); - intel_svm_remove_dev_pasid(domain); } else { did =3D domain_id_iommu(dmar_domain, iommu); cache_tag_unassign_domain(dmar_domain, did, dev, pasid); @@ -4624,6 +4616,7 @@ const struct iommu_ops intel_iommu_ops =3D { .hw_info =3D intel_iommu_hw_info, .domain_alloc =3D intel_iommu_domain_alloc, .domain_alloc_user =3D intel_iommu_domain_alloc_user, + .domain_alloc_sva =3D intel_svm_domain_alloc, .probe_device =3D intel_iommu_probe_device, .probe_finalize =3D intel_iommu_probe_finalize, .release_device =3D intel_iommu_release_device, diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 0b767d16fb71..47e475f67046 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -26,23 +26,6 @@ =20 static irqreturn_t prq_event_thread(int irq, void *d); =20 -static DEFINE_XARRAY_ALLOC(pasid_private_array); -static int pasid_private_add(ioasid_t pasid, void *priv) -{ - return xa_alloc(&pasid_private_array, &pasid, priv, - XA_LIMIT(pasid, pasid), GFP_ATOMIC); -} - -static void pasid_private_remove(ioasid_t pasid) -{ - xa_erase(&pasid_private_array, pasid); -} - -static void *pasid_private_find(ioasid_t pasid) -{ - return xa_load(&pasid_private_array, pasid); -} - int intel_svm_enable_prq(struct intel_iommu *iommu) { struct iopf_queue *iopfq; @@ -156,8 +139,7 @@ static void intel_arch_invalidate_secondary_tlbs(struct= mmu_notifier *mn, struct mm_struct *mm, unsigned long start, unsigned long end) { - struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct dmar_domain *domain =3D svm->domain; + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); =20 if (start =3D=3D 0 && end =3D=3D -1UL) { cache_tag_flush_all(domain); @@ -169,8 +151,7 @@ static void intel_arch_invalidate_secondary_tlbs(struct= mmu_notifier *mn, =20 static void intel_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) { - struct intel_svm *svm =3D container_of(mn, struct intel_svm, notifier); - struct dmar_domain *domain =3D svm->domain; + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); struct dev_pasid_info *dev_pasid; struct device_domain_info *info; unsigned long flags; @@ -210,41 +191,13 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, struct intel_iommu *iommu =3D info->iommu; struct mm_struct *mm =3D domain->mm; struct dev_pasid_info *dev_pasid; - struct intel_svm *svm; unsigned long sflags; unsigned long flags; int ret =3D 0; =20 - svm =3D pasid_private_find(pasid); - if (!svm) { - svm =3D kzalloc(sizeof(*svm), GFP_KERNEL); - if (!svm) - return -ENOMEM; - - svm->pasid =3D pasid; - svm->mm =3D mm; - - svm->notifier.ops =3D &intel_mmuops; - svm->domain =3D to_dmar_domain(domain); - ret =3D mmu_notifier_register(&svm->notifier, mm); - if (ret) { - kfree(svm); - return ret; - } - - ret =3D pasid_private_add(svm->pasid, svm); - if (ret) { - mmu_notifier_unregister(&svm->notifier, mm); - kfree(svm); - return ret; - } - } - - dmar_domain->svm =3D svm; - dev_pasid =3D kzalloc(sizeof(*dev_pasid), GFP_KERNEL); if (!dev_pasid) - goto free_svm; + return -ENOMEM; =20 dev_pasid->dev =3D dev; dev_pasid->pasid =3D pasid; @@ -272,30 +225,10 @@ static int intel_svm_set_dev_pasid(struct iommu_domai= n *domain, FLPT_DEFAULT_DID, dev, pasid); free_dev_pasid: kfree(dev_pasid); -free_svm: - if (list_empty(&dmar_domain->dev_pasids)) { - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(pasid); - kfree(svm); - } =20 return ret; } =20 -void intel_svm_remove_dev_pasid(struct iommu_domain *domain) -{ - struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); - struct intel_svm *svm =3D dmar_domain->svm; - struct mm_struct *mm =3D domain->mm; - - if (list_empty(&dmar_domain->dev_pasids)) { - if (svm->notifier.ops) - mmu_notifier_unregister(&svm->notifier, mm); - pasid_private_remove(svm->pasid); - kfree(svm); - } -} - /* Page request queue descriptor */ struct page_req_dsc { union { @@ -663,7 +596,12 @@ void intel_svm_page_response(struct device *dev, struc= t iopf_fault *evt, =20 static void intel_svm_domain_free(struct iommu_domain *domain) { - kfree(to_dmar_domain(domain)); + struct dmar_domain *dmar_domain =3D to_dmar_domain(domain); + + if (dmar_domain->notifier.ops) + mmu_notifier_unregister(&dmar_domain->notifier, domain->mm); + + kfree(dmar_domain); } =20 static const struct iommu_domain_ops intel_svm_domain_ops =3D { @@ -671,13 +609,16 @@ static const struct iommu_domain_ops intel_svm_domain= _ops =3D { .free =3D intel_svm_domain_free }; =20 -struct iommu_domain *intel_svm_domain_alloc(void) +struct iommu_domain *intel_svm_domain_alloc(struct device *dev, + struct mm_struct *mm) { struct dmar_domain *domain; + int ret; =20 domain =3D kzalloc(sizeof(*domain), GFP_KERNEL); if (!domain) - return NULL; + return ERR_PTR(-ENOMEM); + domain->domain.ops =3D &intel_svm_domain_ops; domain->use_first_level =3D true; INIT_LIST_HEAD(&domain->dev_pasids); @@ -685,5 +626,12 @@ struct iommu_domain *intel_svm_domain_alloc(void) spin_lock_init(&domain->cache_lock); spin_lock_init(&domain->lock); =20 + domain->notifier.ops =3D &intel_mmuops; + ret =3D mmu_notifier_register(&domain->notifier, mm); + if (ret) { + kfree(domain); + return ERR_PTR(ret); + } + return &domain->domain; } --=20 2.34.1