From nobody Sun Feb 8 16:06:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED1BC17BB6 for ; Fri, 9 Aug 2024 02:56:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172195; cv=none; b=eL/SfMZWbYgVdZwrAhZl8t7nbFuACSqXmVywx+wlu17ab7GctfbDZl7bfy4KwshofwA+UPaTL+LhPEfrpv4ZcbqsUb2AzWq1wS+NAu1zLO20wSwXcnoSITz6DxR+fSUtHy8b+AvpbruiRc3tS2jPiQWqn36sZlUAIi3JmOW6RvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172195; c=relaxed/simple; bh=1ql59GPNLQi3WjGEBGQSQ0824+BH6uuhZxVOIMX4z8w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XrACrrY9vP/i05n8ZgBRH70zzX9SClUoK3LnVjNtVeRRvxngqkcHpuCMEDyHleSSBN4rhq6tJyl0Ju9iTr5NDnO32vLVOAgYI+wtyX45c/cRuop1AHF7LhyMXGwoFfPIgKH8ivF+xs3DmW8zyIWzdicyf+RphCTf9Cg9X58xXQg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=T6rYRErl; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="T6rYRErl" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723172192; x=1754708192; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1ql59GPNLQi3WjGEBGQSQ0824+BH6uuhZxVOIMX4z8w=; b=T6rYRErlhpApmsPycHAPw3Sf5O/AKsgJ2tPBLw7ewu/XJjeyValBrRy9 5yxmXCZXvlT7mG7bCzTK57dgA+JxgQfhl0nD0UzFy7oQMaHNgB5z9PETC 0jbhuB0FnTw0TAOgjGXkKx7dKY/GGOjnEPPm5UelbJwcjoGHEn5lsIs1r 9mf2aRaMVxcwJyAD7XL8pXFkupkwaI7r7eRVausD/l+ZR3fePFH547LH1 XScCiESK8LiOvJkuYk0unE3pTIdCdNf2T+oMaBKjh3ZZZ20K4gaBeqJTk AbkMbW39pP+yzRCCusTh/ZUTCg3X9aJnz7FB3Lkv4skslxD0uKYcRtuO4 Q==; X-CSE-ConnectionGUID: tgQeBHaUQ1GUZxFRUpfRYg== X-CSE-MsgGUID: 3JK1pVk9TKe+fcUwWgH0wg== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24239350" X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="24239350" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2024 19:56:30 -0700 X-CSE-ConnectionGUID: KOmL/coXRP+DrBF/G1U74Q== X-CSE-MsgGUID: eVbQj0BySNaEbS+Q4ll4lA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="62072644" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa004.fm.intel.com with ESMTP; 08 Aug 2024 19:56:28 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v2 1/5] iommu/vt-d: Refactor IOTLB/Dev-IOTLB invalidation command logic Date: Fri, 9 Aug 2024 10:54:27 +0800 Message-Id: <20240809025431.14605-2-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809025431.14605-1-tina.zhang@intel.com> References: <20240809025431.14605-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Separate assembling IOTLB/dev-IOTLB invalidation command logic from qi_flush interfaces. New qi_desc() functions are introduced for this purpose. The goal is to facilitate the reuse of these qi_desc() functions in the qi_batch interfaces which will be introduced in a subsequent patch. Signed-off-by: Tina Zhang --- drivers/iommu/intel/dmar.c | 203 +++++++++++++++++++++---------------- 1 file changed, 116 insertions(+), 87 deletions(-) diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c index 1c8d3141cb55..64724af1a618 100644 --- a/drivers/iommu/intel/dmar.c +++ b/drivers/iommu/intel/dmar.c @@ -1360,6 +1360,116 @@ static int qi_check_fault(struct intel_iommu *iommu= , int index, int wait_index) return 0; } =20 +static void qi_desc_iotlb(struct intel_iommu *iommu, u16 did, + u64 addr, unsigned int size_order, + u64 type, struct qi_desc *desc) +{ + u8 dw =3D 0, dr =3D 0; + int ih =3D 0; + + if (cap_write_drain(iommu->cap)) + dw =3D 1; + + if (cap_read_drain(iommu->cap)) + dr =3D 1; + + desc->qw0 =3D QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw) + | QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE; + desc->qw1 =3D QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih) + | QI_IOTLB_AM(size_order); + desc->qw2 =3D 0; + desc->qw3 =3D 0; +} + +static void qi_desc_dev_iotlb(u16 sid, u16 pfsid, + u16 qdep, u64 addr, + unsigned int mask, + struct qi_desc *desc) +{ + if (mask) { + addr |=3D (1ULL << (VTD_PAGE_SHIFT + mask - 1)) - 1; + desc->qw1 =3D QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE; + } else + desc->qw1 =3D QI_DEV_IOTLB_ADDR(addr); + + if (qdep >=3D QI_DEV_IOTLB_MAX_INVS) + qdep =3D 0; + + desc->qw0 =3D QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) | + QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid); + desc->qw2 =3D 0; + desc->qw3 =3D 0; +} + +static void qi_desc_piotlb(u16 did, u32 pasid, u64 addr, + unsigned long npages, bool ih, + struct qi_desc *desc) +{ + if (npages =3D=3D -1) { + desc->qw0 =3D QI_EIOTLB_PASID(pasid) | + QI_EIOTLB_DID(did) | + QI_EIOTLB_GRAN(QI_GRAN_NONG_PASID) | + QI_EIOTLB_TYPE; + desc->qw1 =3D 0; + } else { + int mask =3D ilog2(__roundup_pow_of_two(npages)); + unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + mask)); + + if (WARN_ON_ONCE(!IS_ALIGNED(addr, align))) + addr =3D ALIGN_DOWN(addr, align); + + desc->qw0 =3D QI_EIOTLB_PASID(pasid) | + QI_EIOTLB_DID(did) | + QI_EIOTLB_GRAN(QI_GRAN_PSI_PASID) | + QI_EIOTLB_TYPE; + desc->qw1 =3D QI_EIOTLB_ADDR(addr) | + QI_EIOTLB_IH(ih) | + QI_EIOTLB_AM(mask); + } +} + +static void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, + u32 pasid, u16 qdep, u64 addr, + unsigned int size_order, + struct qi_desc *desc) +{ + unsigned long mask =3D 1UL << (VTD_PAGE_SHIFT + size_order - 1); + + desc->qw0 =3D QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) | + QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE | + QI_DEV_IOTLB_PFSID(pfsid); + + /* + * If S bit is 0, we only flush a single page. If S bit is set, + * The least significant zero bit indicates the invalidation address + * range. VT-d spec 6.5.2.6. + * e.g. address bit 12[0] indicates 8KB, 13[0] indicates 16KB. + * size order =3D 0 is PAGE_SIZE 4KB + * Max Invs Pending (MIP) is set to 0 for now until we have DIT in + * ECAP. + */ + if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order)) + pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n", + addr, size_order); + + /* Take page address */ + desc->qw1 =3D QI_DEV_EIOTLB_ADDR(addr); + + if (size_order) { + /* + * Existing 0s in address below size_order may be the least + * significant bit, we must set them to 1s to avoid having + * smaller size than desired. + */ + desc->qw1 |=3D GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1, + VTD_PAGE_SHIFT); + /* Clear size_order bit to indicate size */ + desc->qw1 &=3D ~mask; + /* Set the S bit to indicate flushing more than 1 page */ + desc->qw1 |=3D QI_DEV_EIOTLB_SIZE; + } +} + /* * Function to submit invalidation descriptors of all types to the queued * invalidation interface(QI). Multiple descriptors can be submitted at a @@ -1520,24 +1630,9 @@ void qi_flush_context(struct intel_iommu *iommu, u16= did, u16 sid, u8 fm, void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, unsigned int size_order, u64 type) { - u8 dw =3D 0, dr =3D 0; - struct qi_desc desc; - int ih =3D 0; - - if (cap_write_drain(iommu->cap)) - dw =3D 1; - - if (cap_read_drain(iommu->cap)) - dr =3D 1; - - desc.qw0 =3D QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw) - | QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE; - desc.qw1 =3D QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih) - | QI_IOTLB_AM(size_order); - desc.qw2 =3D 0; - desc.qw3 =3D 0; =20 + qi_desc_iotlb(iommu, did, addr, size_order, type, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1555,20 +1650,7 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, u= 16 sid, u16 pfsid, if (!(iommu->gcmd & DMA_GCMD_TE)) return; =20 - if (mask) { - addr |=3D (1ULL << (VTD_PAGE_SHIFT + mask - 1)) - 1; - desc.qw1 =3D QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE; - } else - desc.qw1 =3D QI_DEV_IOTLB_ADDR(addr); - - if (qdep >=3D QI_DEV_IOTLB_MAX_INVS) - qdep =3D 0; - - desc.qw0 =3D QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) | - QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid); - desc.qw2 =3D 0; - desc.qw3 =3D 0; - + qi_desc_dev_iotlb(sid, pfsid, qdep, addr, mask, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1588,28 +1670,7 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 = did, u32 pasid, u64 addr, return; } =20 - if (npages =3D=3D -1) { - desc.qw0 =3D QI_EIOTLB_PASID(pasid) | - QI_EIOTLB_DID(did) | - QI_EIOTLB_GRAN(QI_GRAN_NONG_PASID) | - QI_EIOTLB_TYPE; - desc.qw1 =3D 0; - } else { - int mask =3D ilog2(__roundup_pow_of_two(npages)); - unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + mask)); - - if (WARN_ON_ONCE(!IS_ALIGNED(addr, align))) - addr =3D ALIGN_DOWN(addr, align); - - desc.qw0 =3D QI_EIOTLB_PASID(pasid) | - QI_EIOTLB_DID(did) | - QI_EIOTLB_GRAN(QI_GRAN_PSI_PASID) | - QI_EIOTLB_TYPE; - desc.qw1 =3D QI_EIOTLB_ADDR(addr) | - QI_EIOTLB_IH(ih) | - QI_EIOTLB_AM(mask); - } - + qi_desc_piotlb(did, pasid, addr, npages, ih, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1617,7 +1678,6 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 d= id, u32 pasid, u64 addr, void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, u16 sid, u16 pfsi= d, u32 pasid, u16 qdep, u64 addr, unsigned int size_order) { - unsigned long mask =3D 1UL << (VTD_PAGE_SHIFT + size_order - 1); struct qi_desc desc =3D {.qw1 =3D 0, .qw2 =3D 0, .qw3 =3D 0}; =20 /* @@ -1629,40 +1689,9 @@ void qi_flush_dev_iotlb_pasid(struct intel_iommu *io= mmu, u16 sid, u16 pfsid, if (!(iommu->gcmd & DMA_GCMD_TE)) return; =20 - desc.qw0 =3D QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) | - QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE | - QI_DEV_IOTLB_PFSID(pfsid); - - /* - * If S bit is 0, we only flush a single page. If S bit is set, - * The least significant zero bit indicates the invalidation address - * range. VT-d spec 6.5.2.6. - * e.g. address bit 12[0] indicates 8KB, 13[0] indicates 16KB. - * size order =3D 0 is PAGE_SIZE 4KB - * Max Invs Pending (MIP) is set to 0 for now until we have DIT in - * ECAP. - */ - if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order)) - pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n", - addr, size_order); - - /* Take page address */ - desc.qw1 =3D QI_DEV_EIOTLB_ADDR(addr); - - if (size_order) { - /* - * Existing 0s in address below size_order may be the least - * significant bit, we must set them to 1s to avoid having - * smaller size than desired. - */ - desc.qw1 |=3D GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1, - VTD_PAGE_SHIFT); - /* Clear size_order bit to indicate size */ - desc.qw1 &=3D ~mask; - /* Set the S bit to indicate flushing more than 1 page */ - desc.qw1 |=3D QI_DEV_EIOTLB_SIZE; - } - + qi_desc_dev_iotlb_pasid(sid, pfsid, pasid, + qdep, addr, size_order, + &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 --=20 2.43.0 From nobody Sun Feb 8 16:06:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA20A17BCC for ; Fri, 9 Aug 2024 02:56:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172194; cv=none; b=mSH3RURmFhI1zcYhe85yBFeGUZczZGDWrn0hJ8sbj/TKCVUUpbR6fSrIoLB+wgNA73PG5DNTuoYHgzTzd5b/XpN1UPwydo46OPlilyb7D59iTnMrqahLK/x9b/DTV0s/BGkw2yTwq2RPYwnj7+yk1zjlzUN6k3uWRl3LS1l3+Eo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172194; c=relaxed/simple; bh=KckKa/R5b/+837PBERqDENwklzGlb1Ks72KL9Ex+/Jg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GwvOQRuGh+OLb1GCoVnTN8WnkEMl7al4D9EM+bnehydsKCK7n7X08vkBG7uzwOX1A+ajGlQR+kIrkfDSKKzZ90lW+MeURZk9qumblog8G1c5kdS6bQooZCXZCXUdjNKOOLXP3+P3JAQdXLCQiy7ddbizo/48uooxr/YC8FFVw4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=OksdoeAi; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="OksdoeAi" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723172192; x=1754708192; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KckKa/R5b/+837PBERqDENwklzGlb1Ks72KL9Ex+/Jg=; b=OksdoeAiPzAfFIGMkeTe3TtR3N7rhc7OyjWIUBjnoEMYxN6FzVIZnUqu Bgj5N5rg/910cuyXIrPgULm/2U6CdFQNrKcc0EGa+Jpf+PGcjaLCj2Gg9 cNylRWLuPjPFFRRwvYJtAQzIVYO0Z458duUcknFm8NbZevidFGjGRQsie 2L1zXhrV1SC3rC5cADsjeVwe1BE5vJIWbqhlg/77vdI66ot0hpLPqlHCc dRtJhChl1touseuYS8be+SCFP7Im620vWp4idq1ah2BHHGSeRc95RT3XO oHS5+BuMF06pqwbU9YdM3mrR/sA+y6oZk5ub4vf7eOiUFOcpgy/PnDZXs Q==; X-CSE-ConnectionGUID: GjAvh3uJScGUDLbYayHZsA== X-CSE-MsgGUID: Mqb/jdr8SXOWpIghJoftVg== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24239355" X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="24239355" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2024 19:56:31 -0700 X-CSE-ConnectionGUID: G7GH4bExSi++4EJcY1GlvQ== X-CSE-MsgGUID: mTVerTHdRSaiAdyCjycGUA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="62072647" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa004.fm.intel.com with ESMTP; 08 Aug 2024 19:56:30 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v2 2/5] iommu/vt-d: Refactor IOTLB and Dev-IOTLB flush logic Date: Fri, 9 Aug 2024 10:54:28 +0800 Message-Id: <20240809025431.14605-3-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809025431.14605-1-tina.zhang@intel.com> References: <20240809025431.14605-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce three new helper functions, handle_iotlb_flush(), handle_dev_ tlb_flush() and handle_dev_tlb_flush_all() to encapsulate the logic for IOTLB and Dev-IOTLB invalidation commands. This refactoring aims to improve code readability and maintainability by centralizing the handling of these flush operations. Signed-off-by: Tina Zhang --- drivers/iommu/intel/cache.c | 136 +++++++++++++++++++----------------- 1 file changed, 72 insertions(+), 64 deletions(-) diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 44e92638c0cd..3ae84ccfcfa1 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -255,6 +255,72 @@ static unsigned long calculate_psi_aligned_address(uns= igned long start, return ALIGN_DOWN(start, VTD_PAGE_SIZE << mask); } =20 +static inline void handle_iotlb_flush(struct dmar_domain *domain, + struct cache_tag *tag, + unsigned long addr, + unsigned long pages, + unsigned long mask, + int ih) +{ + struct intel_iommu *iommu =3D tag->iommu; + + if (domain->use_first_level) { + qi_flush_piotlb(iommu, tag->domain_id, + tag->pasid, addr, pages, ih); + } else { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap) || + pages =3D=3D -1) + iommu->flush.flush_iotlb(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH); + else + iommu->flush.flush_iotlb(iommu, tag->domain_id, + addr | ih, mask, + DMA_TLB_PSI_FLUSH); + } +} + +static void handle_dev_tlb_flush(struct cache_tag *tag, + unsigned long addr, + unsigned long mask) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + if (tag->pasid =3D=3D IOMMU_NO_PASID) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr, mask); + else + qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, + tag->pasid, info->ats_qdep, + addr, mask); + + quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); +} + +static void handle_dev_tlb_flush_all(struct cache_tag *tag) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + 0, MAX_AGAW_PFN_WIDTH); + quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, + IOMMU_NO_PASID, info->ats_qdep); +} + /* * Invalidates a range of IOVA from @start (inclusive) to @end (inclusive) * when the memory mappings in the target domain have been modified. @@ -270,30 +336,10 @@ void cache_tag_flush_range(struct dmar_domain *domain= , unsigned long start, =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { - struct intel_iommu *iommu =3D tag->iommu; - struct device_domain_info *info; - u16 sid; - switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: - if (domain->use_first_level) { - qi_flush_piotlb(iommu, tag->domain_id, - tag->pasid, addr, pages, ih); - } else { - /* - * Fallback to domain selective flush if no - * PSI support or the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || - mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - addr | ih, mask, - DMA_TLB_PSI_FLUSH); - } + handle_iotlb_flush(domain, tag, addr, pages, mask, ih); break; case CACHE_TAG_NESTING_DEVTLB: /* @@ -307,18 +353,7 @@ void cache_tag_flush_range(struct dmar_domain *domain,= unsigned long start, mask =3D MAX_AGAW_PFN_WIDTH; fallthrough; case CACHE_TAG_DEVTLB: - info =3D dev_iommu_priv_get(tag->dev); - sid =3D PCI_DEVID(info->bus, info->devfn); - - if (tag->pasid =3D=3D IOMMU_NO_PASID) - qi_flush_dev_iotlb(iommu, sid, info->pfsid, - info->ats_qdep, addr, mask); - else - qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, - tag->pasid, info->ats_qdep, - addr, mask); - - quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); + handle_dev_tlb_flush(tag, addr, mask); break; } =20 @@ -338,29 +373,14 @@ void cache_tag_flush_all(struct dmar_domain *domain) =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { - struct intel_iommu *iommu =3D tag->iommu; - struct device_domain_info *info; - u16 sid; - switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: - if (domain->use_first_level) - qi_flush_piotlb(iommu, tag->domain_id, - tag->pasid, 0, -1, 0); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); + handle_iotlb_flush(domain, tag, 0, -1, 0, 0); break; case CACHE_TAG_DEVTLB: case CACHE_TAG_NESTING_DEVTLB: - info =3D dev_iommu_priv_get(tag->dev); - sid =3D PCI_DEVID(info->bus, info->devfn); - - qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, - 0, MAX_AGAW_PFN_WIDTH); - quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, - IOMMU_NO_PASID, info->ats_qdep); + handle_dev_tlb_flush_all(tag); break; } =20 @@ -399,20 +419,8 @@ void cache_tag_flush_range_np(struct dmar_domain *doma= in, unsigned long start, } =20 if (tag->type =3D=3D CACHE_TAG_IOTLB || - tag->type =3D=3D CACHE_TAG_NESTING_IOTLB) { - /* - * Fallback to domain selective flush if no - * PSI support or the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || - mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - addr, mask, - DMA_TLB_PSI_FLUSH); - } + tag->type =3D=3D CACHE_TAG_NESTING_IOTLB) + handle_iotlb_flush(domain, tag, addr, pages, mask, 0); =20 trace_cache_tag_flush_range_np(tag, start, end, addr, pages, mask); } --=20 2.43.0 From nobody Sun Feb 8 16:06:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A95AE1805E for ; Fri, 9 Aug 2024 02:56:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172196; cv=none; b=PdAhJ7qhZdXrL7u3MTyH9wRo98ZUwgWpLPyI/CdlNL2tqJhwHDd/SSgvR/d3+gu6baZc5Xu4qkM64PukktEbALIb43JETsTAcOWUKgjeUAxyHIWEf7XQ9UH6pdJdmNcOMcsCBTfS9TJlfqq0jJTgruX+4MDW8oUsGfU0l1TLFbQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172196; c=relaxed/simple; bh=fBlM0qLseZLN9df3ook7bpErdmbmwNQHQ19NE1NArGs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M6gqFk4lpboGykYDEVlwKmsxEVIt+8+18sHKgfZ1nkRlzAqP1bJXgbF0V0xJnkKoDMGqIplTWjgc43wtTBfYf7ZMRIpUK835pEK+LvVIi4q1UoRD6gvGArI32NQpySZnwsm/pGiJw4m3QgiiG/qlsUdBOQLZSR6X3fOC5221BJc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Wm6Mn7r5; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Wm6Mn7r5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723172194; x=1754708194; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fBlM0qLseZLN9df3ook7bpErdmbmwNQHQ19NE1NArGs=; b=Wm6Mn7r5BRs+GKOXhlFEK+CJmJRVXJ7IzuNNJG/dP3gNms9MWpbnt4K1 5jrG1pVF96OSv8yFtGTGYAQY/CCUO1g+OOJlj87uQdWZtuyjnEMyFWbOc /pQNM01wndJPbsI6Slt1sDvNjar5vH7nGkDftctAe0zk7fbmULC2Q9lOH nXVvwpwv/aDRTmYrzietZU0el2CnRZObrSvsB5vJpF5FigsPeVi7AVayf nyoX/tyhTqUQC7CnXEVweb/jBoLgVG4QuRUVOZItDhdsxnVnm2Oj17EhN 01mvd+J5ZEbnccfIQzSEcAOn0bBC+oKFpGCbDR31yNWjjIQ53+k4Yt8XU Q==; X-CSE-ConnectionGUID: +4jwmDVqStu+l4zc6Me5NA== X-CSE-MsgGUID: 8Vtl5omcSvGc2e3r3HKHfw== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24239357" X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="24239357" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2024 19:56:33 -0700 X-CSE-ConnectionGUID: nzBwAlseRAeUkivd8GNMhw== X-CSE-MsgGUID: 8bqFr1YQRXmWURAMUx+axA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="62072650" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa004.fm.intel.com with ESMTP; 08 Aug 2024 19:56:32 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v2 3/5] iommu/vt-d: Introduce interfaces for QI batching operations Date: Fri, 9 Aug 2024 10:54:29 +0800 Message-Id: <20240809025431.14605-4-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809025431.14605-1-tina.zhang@intel.com> References: <20240809025431.14605-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduces qi_batch_xxx() interfaces to the VT-d driver to enhance the efficiency of IOTLB and Dev-IOTLB invalidation command processing. By allowing these commands to be batched together before submission, the patch aims to minimize the overhead previously incurred when handling these operations individually. The addition of qi_batch_add_xxx() functions enable the accumulation of invalidation commands into a batch, while the qi_batch_flush_descs() function allows for the collective submission of these commands. Signed-off-by: Tina Zhang --- drivers/iommu/intel/dmar.c | 78 +++++++++++++++++++++++++++++++++++++ drivers/iommu/intel/iommu.h | 39 +++++++++++++++++++ 2 files changed, 117 insertions(+) diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c index 64724af1a618..8d55c49382fc 100644 --- a/drivers/iommu/intel/dmar.c +++ b/drivers/iommu/intel/dmar.c @@ -1636,6 +1636,84 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 d= id, u64 addr, qi_submit_sync(iommu, &desc, 1, 0); } =20 +static void qi_batch_increment_index(struct intel_iommu *iommu, + struct qi_batch *batch) +{ + if (++batch->index =3D=3D QI_MAX_BATCHED_DESC_COUNT) + qi_batch_flush_descs(iommu, batch); +} + +void qi_batch_flush_descs(struct intel_iommu *iommu, struct qi_batch *batc= h) +{ + if (!batch->index) + return; + + qi_submit_sync(iommu, batch->descs, batch->index, 0); + + /* Reset the index value and clean the whole batch buffer */ + memset(batch, 0, sizeof(struct qi_batch)); +} + +void qi_batch_add_iotlb_desc(struct intel_iommu *iommu, u16 did, u64 addr, + unsigned int size_order, u64 type, + struct qi_batch *batch) +{ + qi_desc_iotlb(iommu, did, addr, size_order, type, &(batch->descs[batch->i= ndex])); + qi_batch_increment_index(iommu, batch); +} + +void qi_batch_add_dev_iotlb_desc(struct intel_iommu *iommu, u16 sid, + u16 pfsid, u16 qdep, u64 addr, + unsigned int mask, + struct qi_batch *batch) +{ + /* + * According to VT-d spec, software is recommended to not submit any Devi= ce-TLB + * invalidation requests while address remapping hardware is disabled. + */ + if (!(iommu->gcmd & DMA_GCMD_TE)) + return; + + qi_desc_dev_iotlb(sid, pfsid, qdep, addr, mask, &(batch->descs[batch->ind= ex])); + qi_batch_increment_index(iommu, batch); +} + +void qi_batch_add_piotlb_desc(struct intel_iommu *iommu, u16 did, + u32 pasid, u64 addr, + unsigned long npages, bool ih, + struct qi_batch *batch) +{ + /* + * npages =3D=3D -1 means a PASID-selective invalidation, otherwise, + * a positive value for Page-selective-within-PASID invalidation. + * 0 is not a valid input. + */ + if (!npages) + return; + + qi_desc_piotlb(did, pasid, addr, npages, ih, &(batch->descs[batch->index]= )); + qi_batch_increment_index(iommu, batch); +} + +void qi_batch_add_dev_iotlb_pasid_desc(struct intel_iommu *iommu, + u16 sid, u16 pfsid, + u32 pasid, u16 qdep, + u64 addr, unsigned int size_order, + struct qi_batch *batch) +{ + /* + * According to VT-d spec, software is recommended to not submit any Devi= ce-TLB + * invalidation requests while address remapping hardware is disabled. + */ + if (!(iommu->gcmd & DMA_GCMD_TE)) + return; + + qi_desc_dev_iotlb_pasid(sid, pfsid, pasid, + qdep, addr, size_order, + &(batch->descs[batch->index])); + qi_batch_increment_index(iommu, batch); +} + void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid, u16 qdep, u64 addr, unsigned mask) { diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index b67c14da1240..cd7c1d0a01c6 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -584,6 +584,22 @@ struct iommu_domain_info { * to VT-d spec, section 9.3 */ }; =20 +/* + * The QI_MAX_BATCHED_DESC_COUNT value is determined by the two considerat= ions: + * 1) Maximizing the batching of IOTLB and Dev-IOTLB invalidation commands= , which is + * especially advantageous in virtualization environments where multipl= e devices may be + * associated with a single virtual IOMMU. + * 2) Minimizing unnecessary memory allocation for domains lacking ATS sup= port. + * + * Future enhancements could include dynamically allocating the batch buff= er based on actual + * demand, allowing for adjustments to the batch size to better accommodat= e various use cases. + */ +#define QI_MAX_BATCHED_DESC_COUNT 16 +struct qi_batch { + struct qi_desc descs[QI_MAX_BATCHED_DESC_COUNT]; + unsigned int index; +}; + struct dmar_domain { int nid; /* node id */ struct xarray iommu_array; /* Attached IOMMU array */ @@ -1098,6 +1114,29 @@ void qi_flush_pasid_cache(struct intel_iommu *iommu,= u16 did, u64 granu, =20 int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc, unsigned int count, unsigned long options); + +void qi_batch_flush_descs(struct intel_iommu *iommu, + struct qi_batch *batch); + +void qi_batch_add_iotlb_desc(struct intel_iommu *iommu, u16 did, u64 addr, + unsigned int size_order, u64 type, + struct qi_batch *batch); + +void qi_batch_add_dev_iotlb_desc(struct intel_iommu *iommu, u16 sid, + u16 pfsid, u16 qdep, u64 addr, + unsigned int mask, + struct qi_batch *batch); + +void qi_batch_add_piotlb_desc(struct intel_iommu *iommu, u16 did, + u32 pasid, u64 addr, + unsigned long npages, bool ih, + struct qi_batch *batch); + +void qi_batch_add_dev_iotlb_pasid_desc(struct intel_iommu *iommu, + u16 sid, u16 pfsid, + u32 pasid, u16 qdep, + u64 addr, unsigned int size_order, + struct qi_batch *batch); /* * Options used in qi_submit_sync: * QI_OPT_WAIT_DRAIN - Wait for PRQ drain completion, spec 6.5.2.8. --=20 2.43.0 From nobody Sun Feb 8 16:06:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44A2F18654 for ; Fri, 9 Aug 2024 02:56:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172197; cv=none; b=Tu6ZbdN7KTt5X6tfRy0UriM8de914xEQ4+zBTjVcYK80NSFDmhYnfLXCtxwa3l5HZcExJoNvWSvVettdNLFAtEnD+78k5uoTTTlpBkNxAPTm4MZDc47NvY4+W9GgLh3HkexS//B+umgvGN8QBsxDcSPS+BMDELFgQOTDjEJ0Blc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172197; c=relaxed/simple; bh=nj1nJU3aNBybLvKEXohwrftCV9iqavHOaz5vSyI1PEA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MJzbjHPj9CADQ/i9z7PSegKNYYzfDubyMZS0UQjDJwU8d9cyLLbgP3bIsvhzyliz6H05VE6Ghz7TQuazUuqCB06+53No7R/egc5mnO5oJauexAKh7kQNZu6OJmnUwA1CN4mCVaKvDFA1+ILI2M2oRvISn6gWP2vrB3DjKutP12Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Upr2ykow; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Upr2ykow" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723172195; x=1754708195; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nj1nJU3aNBybLvKEXohwrftCV9iqavHOaz5vSyI1PEA=; b=Upr2ykowZPN79sDdYCKwEzlpFdtfVrkDCLZ5jjuD6NBsvXp5BI4UkDCk IWYjIK99EkwCFdH3I9g3JwjQ2JSGT3pZohmlLC1e0wTaXmJi7bP1hT5CY IYkX775aCdEy+595g58x9QJIdceP+ktTXtn0EO3ngbSxgOlpp03wcd3Cw AMqHCXa4vmm4j2DLJEOzXv4YEYXEJNH/jLSikyYeNwWpB8jxH+T9Fy+/H fFDzB4RnTPjGPuDzdS3vWMKbcPdhSboo1Or49D/QQoNNSYWnsmgxBvXmh TWI4wbn4eHdu3e4zPFqZhZes1P/FgmKDHmtIfYlqWfWJEdo9L8G4hHXXw A==; X-CSE-ConnectionGUID: Qq/xX32bS5WniGeS+v5uyQ== X-CSE-MsgGUID: nVl6tv5oRQGAov9pVsxHFA== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24239359" X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="24239359" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2024 19:56:35 -0700 X-CSE-ConnectionGUID: lHrhUBJARRKke3w+g1WGIw== X-CSE-MsgGUID: /RHadu3dTzChHyxZlhlInw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="62072653" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa004.fm.intel.com with ESMTP; 08 Aug 2024 19:56:33 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v2 4/5] vt-d/iommu: Refactor quirk_extra_dev_tlb_flush() Date: Fri, 9 Aug 2024 10:54:30 +0800 Message-Id: <20240809025431.14605-5-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809025431.14605-1-tina.zhang@intel.com> References: <20240809025431.14605-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the core logic from quirk_extra_dev_tlb_flush() into a new helper __quirk_extra_dev_tlb_flush(). This helper is for accommodating for both individual and batched TLB invalidation commands, thereby streamlining the process for handling device-specific TLB flush quirks. Signed-off-by: Tina Zhang --- drivers/iommu/intel/iommu.c | 55 +++++++++++++++++++++++++++++-------- drivers/iommu/intel/iommu.h | 4 +++ 2 files changed, 47 insertions(+), 12 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 9ff8b83c19a3..160d569015b4 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -4875,6 +4875,41 @@ static void __init check_tylersburg_isoch(void) vtisochctrl); } =20 +static inline void __quirk_extra_dev_tlb_flush(struct device_domain_info *= info, + unsigned long address, unsigned long mask, + u32 pasid, u16 qdep, + struct qi_batch *batch) +{ + u16 sid; + + if (likely(!info->dtlb_extra_inval)) + return; + + sid =3D PCI_DEVID(info->bus, info->devfn); + if (batch =3D=3D NULL) { + if (pasid =3D=3D IOMMU_NO_PASID) + qi_flush_dev_iotlb(info->iommu, sid, info->pfsid, + qdep, address, mask); + else + qi_flush_dev_iotlb_pasid(info->iommu, sid, + info->pfsid, pasid, + qdep, address, mask); + } else { + if (pasid =3D=3D IOMMU_NO_PASID) + qi_batch_add_dev_iotlb_desc(info->iommu, sid, + info->pfsid, qdep, + address, mask, batch); + else + qi_batch_add_dev_iotlb_pasid_desc(info->iommu, + sid, + info->pfsid, + pasid, qdep, + address, + mask, + batch); + } +} + /* * Here we deal with a device TLB defect where device may inadvertently is= sue ATS * invalidation completion before posted writes initiated with translated = address @@ -4905,19 +4940,15 @@ void quirk_extra_dev_tlb_flush(struct device_domain= _info *info, unsigned long address, unsigned long mask, u32 pasid, u16 qdep) { - u16 sid; + __quirk_extra_dev_tlb_flush(info, address, mask, pasid, qdep, NULL); +} =20 - if (likely(!info->dtlb_extra_inval)) - return; - - sid =3D PCI_DEVID(info->bus, info->devfn); - if (pasid =3D=3D IOMMU_NO_PASID) { - qi_flush_dev_iotlb(info->iommu, sid, info->pfsid, - qdep, address, mask); - } else { - qi_flush_dev_iotlb_pasid(info->iommu, sid, info->pfsid, - pasid, qdep, address, mask); - } +void batch_quirk_extra_dev_tlb_flush(struct device_domain_info *info, + unsigned long address, unsigned long mask, + u32 pasid, u16 qdep, + struct qi_batch *batch) +{ + __quirk_extra_dev_tlb_flush(info, address, mask, pasid, qdep, batch); } =20 #define ecmd_get_status_code(res) (((res) & 0xff) >> 1) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index cd7c1d0a01c6..04aa1f200124 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1109,6 +1109,10 @@ void qi_flush_dev_iotlb_pasid(struct intel_iommu *io= mmu, u16 sid, u16 pfsid, void quirk_extra_dev_tlb_flush(struct device_domain_info *info, unsigned long address, unsigned long pages, u32 pasid, u16 qdep); +void batch_quirk_extra_dev_tlb_flush(struct device_domain_info *info, + unsigned long address, unsigned long mask, + u32 pasid, u16 qdep, + struct qi_batch *batch); void qi_flush_pasid_cache(struct intel_iommu *iommu, u16 did, u64 granu, u32 pasid); =20 --=20 2.43.0 From nobody Sun Feb 8 16:06:16 2026 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17FD11C6A5 for ; Fri, 9 Aug 2024 02:56:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=192.198.163.13 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172198; cv=none; b=j/FBfKlDAEyPwmbKiGOthJRABqWFoyzzcZRXH4Z3RslbG/Mvxtuc14m4bzW/n9sF3DCVd1OWKaZWvezxVaRwHztWwQaQ41zBuk4XmvGWjxRVVdkHyZWNJvIh8KMm17I8WMqLaB5biVq9nkcKFGH34HZn3VypjBkUZZxzd2HT2Ww= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723172198; c=relaxed/simple; bh=RGRzvg1dpdiE/0xoWJxcDwnO1TbWA2NedxHlvDFTfAk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Qbme/ZPU3AeI/HMsD08wIkp2PpWQ3sXf/SJAaQmnjySNDSrFf8Yt0huzSmbjc9vAMiLafM7QoCjuw1rN4fBoccnnnwCj5OlYgf1LWaBRH0m+YOFklv6WX3ZVrUk9s8fBIsB7iQCc6PqqMqIRuJdLn7Sp12vGj+SH9UtvWLBuTWA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=e/NG20CG; arc=none smtp.client-ip=192.198.163.13 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="e/NG20CG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723172197; x=1754708197; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RGRzvg1dpdiE/0xoWJxcDwnO1TbWA2NedxHlvDFTfAk=; b=e/NG20CGcUpr092VmxtK1jv3x7QLwI8Q1U8LeA37aODUFAW+VK4xKYAW 8coU18DtsMCeshxD+yFYAB8m7Nbtftuvs2XfJrnEUXy7hGDN/liyccOb3 A7M9k673P2Iuwn6MFRHoh0b//LQkPrL34GHQT0jOvjqnYIzQJYZVknAr9 OPEtEIOv4uSTjnKNloccw6wmrq3o6WT5djNJ7cYivLgdgcICyzexjPuTm nbSdNK02uWIK7T2VrnL7eAf9UCITeE6XFOuT9RnF/8FL1ONu3U6f69P2K gLw99vzopqJT8QgoO7Y7iTCLVPtVg1zM+oBHJR/RmbDPnYeqjQzwRswuN w==; X-CSE-ConnectionGUID: mI1s67LBTu6Vvclf3QLtWg== X-CSE-MsgGUID: kZ464if3Q4ibd5pzmf8XPg== X-IronPort-AV: E=McAfee;i="6700,10204,11158"; a="24239363" X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="24239363" Received: from fmviesa004.fm.intel.com ([10.60.135.144]) by fmvoesa107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2024 19:56:37 -0700 X-CSE-ConnectionGUID: 0Zbb8TqrRqanZETzp6wlwg== X-CSE-MsgGUID: VlD7KDGuRDSAvBeGDMM2kg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.09,275,1716274800"; d="scan'208";a="62072660" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa004.fm.intel.com with ESMTP; 08 Aug 2024 19:56:35 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v2 5/5] vt-d/iommu: Enable batching of IOTLB/Dev-IOTLB invalidations Date: Fri, 9 Aug 2024 10:54:31 +0800 Message-Id: <20240809025431.14605-6-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240809025431.14605-1-tina.zhang@intel.com> References: <20240809025431.14605-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable batch processing for IOTLB/Dev-IOTLB invalidation commands of SVA domains and default domains with ATS enabled. Signed-off-by: Tina Zhang --- drivers/iommu/intel/cache.c | 145 ++++++++++++++++++++++++++++++++++-- drivers/iommu/intel/iommu.c | 1 + drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/svm.c | 5 +- 4 files changed, 145 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 3ae84ccfcfa1..fabb98138760 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -59,10 +59,19 @@ static int cache_tag_assign(struct dmar_domain *domain,= u16 did, tag->pasid =3D pasid; tag->users =3D 1; =20 - if (type =3D=3D CACHE_TAG_DEVTLB || type =3D=3D CACHE_TAG_NESTING_DEVTLB) + if (type =3D=3D CACHE_TAG_DEVTLB || type =3D=3D CACHE_TAG_NESTING_DEVTLB)= { tag->dev =3D dev; - else + + if (!domain->qi_batch && iommu->qi) + /* + * It doesn't matter if domain->qi_batch is NULL, as in + * this case the commands will be submitted individually. + */ + domain->qi_batch =3D kzalloc(sizeof(struct qi_batch), + GFP_KERNEL); + } else { tag->dev =3D iommu->iommu.dev; + } =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(temp, &domain->cache_tags, node) { @@ -255,6 +264,84 @@ static unsigned long calculate_psi_aligned_address(uns= igned long start, return ALIGN_DOWN(start, VTD_PAGE_SIZE << mask); } =20 +static inline void handle_batched_iotlb_descs(struct dmar_domain *domain, + struct cache_tag *tag, + unsigned long addr, + unsigned long pages, + unsigned long mask, + int ih) +{ + struct intel_iommu *iommu =3D tag->iommu; + + if (domain->use_first_level) { + qi_batch_add_piotlb_desc(iommu, tag->domain_id, + tag->pasid, addr, pages, + ih, domain->qi_batch); + } else { + /* + * Fallback to domain selective flush if no + * PSI support or the size is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap) || + pages =3D=3D -1) + qi_batch_add_iotlb_desc(iommu, tag->domain_id, + 0, 0, DMA_TLB_DSI_FLUSH, + domain->qi_batch); + else + qi_batch_add_iotlb_desc(iommu, tag->domain_id, + addr | ih, mask, + DMA_TLB_PSI_FLUSH, + domain->qi_batch); + } + +} + +static inline void handle_batched_dev_tlb_descs(struct dmar_domain *domain, + struct cache_tag *tag, + unsigned long addr, + unsigned long mask) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + if (tag->pasid =3D=3D IOMMU_NO_PASID) + qi_batch_add_dev_iotlb_desc(iommu, sid, info->pfsid, + info->ats_qdep, addr, mask, + domain->qi_batch); + else + qi_batch_add_dev_iotlb_pasid_desc(iommu, sid, info->pfsid, + tag->pasid, info->ats_qdep, + addr, mask, domain->qi_batch); + + batch_quirk_extra_dev_tlb_flush(info, addr, mask, + tag->pasid, + info->ats_qdep, + domain->qi_batch); +} + +static void handle_batched_dev_tlb_descs_all(struct dmar_domain *domain, + struct cache_tag *tag) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + qi_batch_add_dev_iotlb_desc(iommu, sid, info->pfsid, info->ats_qdep, + 0, MAX_AGAW_PFN_WIDTH, domain->qi_batch); + batch_quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, + IOMMU_NO_PASID, info->ats_qdep, + domain->qi_batch); + +} + static inline void handle_iotlb_flush(struct dmar_domain *domain, struct cache_tag *tag, unsigned long addr, @@ -264,6 +351,12 @@ static inline void handle_iotlb_flush(struct dmar_doma= in *domain, { struct intel_iommu *iommu =3D tag->iommu; =20 + if (domain->qi_batch) { + handle_batched_iotlb_descs(domain, tag, addr, + pages, mask, ih); + return; + } + if (domain->use_first_level) { qi_flush_piotlb(iommu, tag->domain_id, tag->pasid, addr, pages, ih); @@ -284,7 +377,8 @@ static inline void handle_iotlb_flush(struct dmar_domai= n *domain, } } =20 -static void handle_dev_tlb_flush(struct cache_tag *tag, +static void handle_dev_tlb_flush(struct dmar_domain *domain, + struct cache_tag *tag, unsigned long addr, unsigned long mask) { @@ -292,6 +386,11 @@ static void handle_dev_tlb_flush(struct cache_tag *tag, struct device_domain_info *info; u16 sid; =20 + if (domain->qi_batch) { + handle_batched_dev_tlb_descs(domain, tag, addr, mask); + return; + } + info =3D dev_iommu_priv_get(tag->dev); sid =3D PCI_DEVID(info->bus, info->devfn); =20 @@ -306,12 +405,18 @@ static void handle_dev_tlb_flush(struct cache_tag *ta= g, quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); } =20 -static void handle_dev_tlb_flush_all(struct cache_tag *tag) +static void handle_dev_tlb_flush_all(struct dmar_domain *domain, + struct cache_tag *tag) { struct intel_iommu *iommu =3D tag->iommu; struct device_domain_info *info; u16 sid; =20 + if (domain->qi_batch) { + handle_batched_dev_tlb_descs_all(domain, tag); + return; + } + info =3D dev_iommu_priv_get(tag->dev); sid =3D PCI_DEVID(info->bus, info->devfn); =20 @@ -329,6 +434,7 @@ void cache_tag_flush_range(struct dmar_domain *domain, = unsigned long start, unsigned long end, int ih) { unsigned long pages, mask, addr; + struct intel_iommu *iommu =3D NULL; struct cache_tag *tag; unsigned long flags; =20 @@ -336,6 +442,17 @@ void cache_tag_flush_range(struct dmar_domain *domain,= unsigned long start, =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { + if (domain->qi_batch && iommu !=3D tag->iommu) { + /* + * If domain supports batching commands, need to flush the + * batch buffer before switching to another iommu. + */ + if (iommu) + qi_batch_flush_descs(iommu, domain->qi_batch); + + iommu =3D tag->iommu; + } + switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: @@ -353,12 +470,14 @@ void cache_tag_flush_range(struct dmar_domain *domain= , unsigned long start, mask =3D MAX_AGAW_PFN_WIDTH; fallthrough; case CACHE_TAG_DEVTLB: - handle_dev_tlb_flush(tag, addr, mask); + handle_dev_tlb_flush(domain, tag, addr, mask); break; } =20 trace_cache_tag_flush_range(tag, start, end, addr, pages, mask); } + if (domain->qi_batch && domain->qi_batch->index) + qi_batch_flush_descs(iommu, domain->qi_batch); spin_unlock_irqrestore(&domain->cache_lock, flags); } =20 @@ -368,11 +487,23 @@ void cache_tag_flush_range(struct dmar_domain *domain= , unsigned long start, */ void cache_tag_flush_all(struct dmar_domain *domain) { + struct intel_iommu *iommu =3D NULL; struct cache_tag *tag; unsigned long flags; =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { + if (domain->qi_batch && iommu !=3D tag->iommu) { + /* + * If domain supports batching commands, need to flush the + * batch buffer before switching to another iommu. + */ + if (iommu) + qi_batch_flush_descs(iommu, domain->qi_batch); + + iommu =3D tag->iommu; + } + switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: @@ -380,12 +511,14 @@ void cache_tag_flush_all(struct dmar_domain *domain) break; case CACHE_TAG_DEVTLB: case CACHE_TAG_NESTING_DEVTLB: - handle_dev_tlb_flush_all(tag); + handle_dev_tlb_flush_all(domain, tag); break; } =20 trace_cache_tag_flush_all(tag); } + if (domain->qi_batch && domain->qi_batch->index) + qi_batch_flush_descs(iommu, domain->qi_batch); spin_unlock_irqrestore(&domain->cache_lock, flags); } =20 diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 160d569015b4..5907470b9b35 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1632,6 +1632,7 @@ static void domain_exit(struct dmar_domain *domain) if (WARN_ON(!list_empty(&domain->devices))) return; =20 + kfree(domain->qi_batch); kfree(domain); } =20 diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 04aa1f200124..f16ffda48095 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -625,6 +625,7 @@ struct dmar_domain { =20 spinlock_t cache_lock; /* Protect the cache tag list */ struct list_head cache_tags; /* Cache tag list */ + struct qi_batch *qi_batch; /* QI descriptors batch */ =20 int iommu_superpage;/* Level of superpages supported: 0 =3D=3D 4KiB (no superpages), 1 =3D=3D 2MiB, diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 0e3a9b38bef2..3421813995db 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -184,7 +184,10 @@ static void intel_mm_release(struct mmu_notifier *mn, = struct mm_struct *mm) =20 static void intel_mm_free_notifier(struct mmu_notifier *mn) { - kfree(container_of(mn, struct dmar_domain, notifier)); + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); + + kfree(domain->qi_batch); + kfree(domain); } =20 static const struct mmu_notifier_ops intel_mmuops =3D { --=20 2.43.0