From nobody Fri Dec 19 17:35:43 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 643A117A592 for ; Thu, 15 Aug 2024 06:53:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704809; cv=none; b=ZmZ57cLeM6mdSXc1BeJ95eByxtYROGTFy5sBR7zgiEUbYaHBOm+5exG4bCVfvX7UboAJnUQw+ZdRgSN3ixgNRmORfc7JkZX5Tzew4iY6Oq0N8TAvw687wGFGE5a+9zaNFNm2s1iVzBUSgtkTw5W8LwIFiOkXLFK1JIev5bEMinY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704809; c=relaxed/simple; bh=XSYlWOtrIMf6in1bhc/ncpYlAxPLTtwq8F8Ok0DkSF8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=X4vBnRK8TkquPS/q1sRQzNv2Jc1m1ntOchnJkm7GE8AseOGG8aXO5pwPLOW4BcjSWv61bM7fDc7HXu2CvRZr62CsnTQR2dcqgXVCn4PWuKYUd8DKZKFk/TO+pd1c1W4Cr3QeIIUe2xjoCcr0j6lC/8EA/j6f84fTtu6W490mQZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=Dr62RS/N; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="Dr62RS/N" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723704807; x=1755240807; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XSYlWOtrIMf6in1bhc/ncpYlAxPLTtwq8F8Ok0DkSF8=; b=Dr62RS/NfM2npBJNHtXRYGqaBu2U4xAy2DlrAjFzHMh/4QTVHzdSD4pr GMXv9WBTbr8y1l9ZW3JPKO7KKTcHwmV5uUpDxFoLwybles6SleZQ1W2bR slNrby0F1auewq74G1zHdnvkdEz3mUY40Y1XBkojdrZhbBUFKL419aFza Rm/qYaUetUs825bGJ2OKk1jzgMaLnIHnrocRZTtzxp5ocgZeLRF+tg0nI Mz3HnMADB7ARGCaaUuu1C/+BnlQ4DcoaPswa/oromeOfZ3c4ZjvORG2th c8oVyLfHVbK7jxDQOvYWPQHtjw2yJy7+FVdtpl3Vlxz22NpFLAEZ37it+ Q==; X-CSE-ConnectionGUID: KtILOTDrQN6I5fMcctyqAQ== X-CSE-MsgGUID: k6G4eEj9Qlij93axSMUc7Q== X-IronPort-AV: E=McAfee;i="6700,10204,11164"; a="32528284" X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="32528284" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Aug 2024 23:53:27 -0700 X-CSE-ConnectionGUID: x1Wp0EwjR+SFWWVICeFLUw== X-CSE-MsgGUID: LMCe578xRySs4UZHvJojvw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="82471528" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa002.fm.intel.com with ESMTP; 14 Aug 2024 23:53:25 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v3 1/4] iommu/vt-d: Factor out invalidation descriptor composition Date: Thu, 15 Aug 2024 14:52:18 +0800 Message-Id: <20240815065221.50328-2-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240815065221.50328-1-tina.zhang@intel.com> References: <20240815065221.50328-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Separate the logic for constructing IOTLB and device TLB invalidation descriptors from the qi_flush interfaces. New helpers, qi_desc(), are introduced to encapsulate this common functionality. Moving descriptor composition code to new helpers enables its reuse in the upcoming qi_batch interfaces. No functional changes are intended. Signed-off-by: Tina Zhang --- drivers/iommu/intel/dmar.c | 93 ++---------------------------- drivers/iommu/intel/iommu.h | 109 ++++++++++++++++++++++++++++++++++++ 2 files changed, 115 insertions(+), 87 deletions(-) diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c index 01e157d89a163..eaf862e8dea1a 100644 --- a/drivers/iommu/intel/dmar.c +++ b/drivers/iommu/intel/dmar.c @@ -1526,24 +1526,9 @@ void qi_flush_context(struct intel_iommu *iommu, u16= did, u16 sid, u8 fm, void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, unsigned int size_order, u64 type) { - u8 dw =3D 0, dr =3D 0; - struct qi_desc desc; - int ih =3D 0; - - if (cap_write_drain(iommu->cap)) - dw =3D 1; - - if (cap_read_drain(iommu->cap)) - dr =3D 1; - - desc.qw0 =3D QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw) - | QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE; - desc.qw1 =3D QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih) - | QI_IOTLB_AM(size_order); - desc.qw2 =3D 0; - desc.qw3 =3D 0; =20 + qi_desc_iotlb(iommu, did, addr, size_order, type, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1561,20 +1546,7 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, u= 16 sid, u16 pfsid, if (!(iommu->gcmd & DMA_GCMD_TE)) return; =20 - if (mask) { - addr |=3D (1ULL << (VTD_PAGE_SHIFT + mask - 1)) - 1; - desc.qw1 =3D QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE; - } else - desc.qw1 =3D QI_DEV_IOTLB_ADDR(addr); - - if (qdep >=3D QI_DEV_IOTLB_MAX_INVS) - qdep =3D 0; - - desc.qw0 =3D QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) | - QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid); - desc.qw2 =3D 0; - desc.qw3 =3D 0; - + qi_desc_dev_iotlb(sid, pfsid, qdep, addr, mask, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1594,28 +1566,7 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 = did, u32 pasid, u64 addr, return; } =20 - if (npages =3D=3D -1) { - desc.qw0 =3D QI_EIOTLB_PASID(pasid) | - QI_EIOTLB_DID(did) | - QI_EIOTLB_GRAN(QI_GRAN_NONG_PASID) | - QI_EIOTLB_TYPE; - desc.qw1 =3D 0; - } else { - int mask =3D ilog2(__roundup_pow_of_two(npages)); - unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + mask)); - - if (WARN_ON_ONCE(!IS_ALIGNED(addr, align))) - addr =3D ALIGN_DOWN(addr, align); - - desc.qw0 =3D QI_EIOTLB_PASID(pasid) | - QI_EIOTLB_DID(did) | - QI_EIOTLB_GRAN(QI_GRAN_PSI_PASID) | - QI_EIOTLB_TYPE; - desc.qw1 =3D QI_EIOTLB_ADDR(addr) | - QI_EIOTLB_IH(ih) | - QI_EIOTLB_AM(mask); - } - + qi_desc_piotlb(did, pasid, addr, npages, ih, &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 @@ -1623,7 +1574,6 @@ void qi_flush_piotlb(struct intel_iommu *iommu, u16 d= id, u32 pasid, u64 addr, void qi_flush_dev_iotlb_pasid(struct intel_iommu *iommu, u16 sid, u16 pfsi= d, u32 pasid, u16 qdep, u64 addr, unsigned int size_order) { - unsigned long mask =3D 1UL << (VTD_PAGE_SHIFT + size_order - 1); struct qi_desc desc =3D {.qw1 =3D 0, .qw2 =3D 0, .qw3 =3D 0}; =20 /* @@ -1635,40 +1585,9 @@ void qi_flush_dev_iotlb_pasid(struct intel_iommu *io= mmu, u16 sid, u16 pfsid, if (!(iommu->gcmd & DMA_GCMD_TE)) return; =20 - desc.qw0 =3D QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) | - QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE | - QI_DEV_IOTLB_PFSID(pfsid); - - /* - * If S bit is 0, we only flush a single page. If S bit is set, - * The least significant zero bit indicates the invalidation address - * range. VT-d spec 6.5.2.6. - * e.g. address bit 12[0] indicates 8KB, 13[0] indicates 16KB. - * size order =3D 0 is PAGE_SIZE 4KB - * Max Invs Pending (MIP) is set to 0 for now until we have DIT in - * ECAP. - */ - if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order)) - pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n", - addr, size_order); - - /* Take page address */ - desc.qw1 =3D QI_DEV_EIOTLB_ADDR(addr); - - if (size_order) { - /* - * Existing 0s in address below size_order may be the least - * significant bit, we must set them to 1s to avoid having - * smaller size than desired. - */ - desc.qw1 |=3D GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1, - VTD_PAGE_SHIFT); - /* Clear size_order bit to indicate size */ - desc.qw1 &=3D ~mask; - /* Set the S bit to indicate flushing more than 1 page */ - desc.qw1 |=3D QI_DEV_EIOTLB_SIZE; - } - + qi_desc_dev_iotlb_pasid(sid, pfsid, pasid, + qdep, addr, size_order, + &desc); qi_submit_sync(iommu, &desc, 1, 0); } =20 diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 01002ae2a091a..e297a322ba2d9 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1066,6 +1066,115 @@ static inline unsigned long nrpages_to_size(unsigne= d long npages) return npages << VTD_PAGE_SHIFT; } =20 +static inline void qi_desc_iotlb(struct intel_iommu *iommu, u16 did, u64 a= ddr, + unsigned int size_order, u64 type, + struct qi_desc *desc) +{ + u8 dw =3D 0, dr =3D 0; + int ih =3D 0; + + if (cap_write_drain(iommu->cap)) + dw =3D 1; + + if (cap_read_drain(iommu->cap)) + dr =3D 1; + + desc->qw0 =3D QI_IOTLB_DID(did) | QI_IOTLB_DR(dr) | QI_IOTLB_DW(dw) + | QI_IOTLB_GRAN(type) | QI_IOTLB_TYPE; + desc->qw1 =3D QI_IOTLB_ADDR(addr) | QI_IOTLB_IH(ih) + | QI_IOTLB_AM(size_order); + desc->qw2 =3D 0; + desc->qw3 =3D 0; +} + +static inline void qi_desc_dev_iotlb(u16 sid, u16 pfsid, u16 qdep, u64 add= r, + unsigned int mask, struct qi_desc *desc) +{ + if (mask) { + addr |=3D (1ULL << (VTD_PAGE_SHIFT + mask - 1)) - 1; + desc->qw1 =3D QI_DEV_IOTLB_ADDR(addr) | QI_DEV_IOTLB_SIZE; + } else { + desc->qw1 =3D QI_DEV_IOTLB_ADDR(addr); + } + + if (qdep >=3D QI_DEV_IOTLB_MAX_INVS) + qdep =3D 0; + + desc->qw0 =3D QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) | + QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid); + desc->qw2 =3D 0; + desc->qw3 =3D 0; +} + +static inline void qi_desc_piotlb(u16 did, u32 pasid, u64 addr, + unsigned long npages, bool ih, + struct qi_desc *desc) +{ + if (npages =3D=3D -1) { + desc->qw0 =3D QI_EIOTLB_PASID(pasid) | + QI_EIOTLB_DID(did) | + QI_EIOTLB_GRAN(QI_GRAN_NONG_PASID) | + QI_EIOTLB_TYPE; + desc->qw1 =3D 0; + } else { + int mask =3D ilog2(__roundup_pow_of_two(npages)); + unsigned long align =3D (1ULL << (VTD_PAGE_SHIFT + mask)); + + if (WARN_ON_ONCE(!IS_ALIGNED(addr, align))) + addr =3D ALIGN_DOWN(addr, align); + + desc->qw0 =3D QI_EIOTLB_PASID(pasid) | + QI_EIOTLB_DID(did) | + QI_EIOTLB_GRAN(QI_GRAN_PSI_PASID) | + QI_EIOTLB_TYPE; + desc->qw1 =3D QI_EIOTLB_ADDR(addr) | + QI_EIOTLB_IH(ih) | + QI_EIOTLB_AM(mask); + } +} + +static inline void qi_desc_dev_iotlb_pasid(u16 sid, u16 pfsid, u32 pasid, + u16 qdep, u64 addr, + unsigned int size_order, + struct qi_desc *desc) +{ + unsigned long mask =3D 1UL << (VTD_PAGE_SHIFT + size_order - 1); + + desc->qw0 =3D QI_DEV_EIOTLB_PASID(pasid) | QI_DEV_EIOTLB_SID(sid) | + QI_DEV_EIOTLB_QDEP(qdep) | QI_DEIOTLB_TYPE | + QI_DEV_IOTLB_PFSID(pfsid); + + /* + * If S bit is 0, we only flush a single page. If S bit is set, + * The least significant zero bit indicates the invalidation address + * range. VT-d spec 6.5.2.6. + * e.g. address bit 12[0] indicates 8KB, 13[0] indicates 16KB. + * size order =3D 0 is PAGE_SIZE 4KB + * Max Invs Pending (MIP) is set to 0 for now until we have DIT in + * ECAP. + */ + if (!IS_ALIGNED(addr, VTD_PAGE_SIZE << size_order)) + pr_warn_ratelimited("Invalidate non-aligned address %llx, order %d\n", + addr, size_order); + + /* Take page address */ + desc->qw1 =3D QI_DEV_EIOTLB_ADDR(addr); + + if (size_order) { + /* + * Existing 0s in address below size_order may be the least + * significant bit, we must set them to 1s to avoid having + * smaller size than desired. + */ + desc->qw1 |=3D GENMASK_ULL(size_order + VTD_PAGE_SHIFT - 1, + VTD_PAGE_SHIFT); + /* Clear size_order bit to indicate size */ + desc->qw1 &=3D ~mask; + /* Set the S bit to indicate flushing more than 1 page */ + desc->qw1 |=3D QI_DEV_EIOTLB_SIZE; + } +} + /* Convert value to context PASID directory size field coding. */ #define context_pdts(pds) (((pds) & 0x7) << 9) =20 --=20 2.34.1 From nobody Fri Dec 19 17:35:43 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7DAA17920A for ; Thu, 15 Aug 2024 06:53:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704811; cv=none; b=G6hTufE5LD+lTQ9CGfz8GWfOCv7XPRg33ivIxRZ+WKanDyPpdzqqx3hxaP2hPgQx59N/Wk8aytM4muvEt/9+hvpbbssc0ljLi2Rw6nV4v1J2cCkEhxWEmMQET/Hy4y5qkmhAzHghx4QQfrphDB8BcEj/pLrqH8iB18n3ahcCGMI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704811; c=relaxed/simple; bh=NxnI69OedtlYQflM+SHnZiz78ueK/p8w3vNn+Ui1nB4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dyMTOzw5xyLSp+xRUaDO6h53j284ksSsIR7tK9+cNvj1874h1zyvu1F+GjC+P4qO1gMU3bQcVoQE0NWETMw8Yx7tHv+eIf9y6pWTZtYdN5RIdgxzJ3/YQt5VW+VerMt8yX7qoABl0ajZmTBVV2MC8Ac60wCboowAc94oVH9aU3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=B5yfCzes; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="B5yfCzes" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723704809; x=1755240809; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NxnI69OedtlYQflM+SHnZiz78ueK/p8w3vNn+Ui1nB4=; b=B5yfCzes1KBNMlA1nDSCSAZ+Tz03/EGMxHaBWIurg7bIazA4bHrugxC0 b9VqZXmSSMd8wg7huu2RYU5kQ6sXNXfMhFC9SSJIxIsdDdOxkQPRD1ZZs Lu7hRFkzuMF1rdrG2oJZbYVB/TZ8hD+po60hf/OM8hNCvp0paC6fErP/+ XiAod6GjLnRRkkh3FMxTIcBXlQa6XT3PS1VZL0bE4uhWVb7+o70X3h78W XKLL7pdNSwgZVtIvsW+/1PcJSgmYeKvB3hydygsNrIquMdUhv0nWLsZ5t IflBwQkOLkTrJ1pV6udN+4XVV/O0VOhqMGpCVZb43JeDN/V5bDtZA/7PI Q==; X-CSE-ConnectionGUID: OtmRHw9CQISErpIWGk0z8A== X-CSE-MsgGUID: yiehBA15ROey0P2GZ6Z5Xw== X-IronPort-AV: E=McAfee;i="6700,10204,11164"; a="32528288" X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="32528288" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Aug 2024 23:53:29 -0700 X-CSE-ConnectionGUID: iSHSpCD0Qt6kiPHBwihV1g== X-CSE-MsgGUID: kkYUWm2CSPqqK/VnzQJ4gQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="82471555" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa002.fm.intel.com with ESMTP; 14 Aug 2024 23:53:27 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v3 2/4] iommu/vt-d: Refactor IOTLB and Dev-IOTLB flush for batching Date: Thu, 15 Aug 2024 14:52:19 +0800 Message-Id: <20240815065221.50328-3-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240815065221.50328-1-tina.zhang@intel.com> References: <20240815065221.50328-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extracts IOTLB and Dev-IOTLB invalidation logic from cache tag flush interfaces into dedicated helper functions. It prepares the codebase for upcoming changes to support batched cache invalidations. To enable direct use of qi_flush helpers in the new functions, iommu->flush.flush_iotlb and quirk_extra_dev_tlb_flush() are opened up. No functional changes are intended. Co-developed-by: Lu Baolu Signed-off-by: Lu Baolu Signed-off-by: Tina Zhang --- drivers/iommu/intel/cache.c | 142 ++++++++++++++++++++---------------- drivers/iommu/intel/iommu.c | 5 +- drivers/iommu/intel/iommu.h | 3 + 3 files changed, 83 insertions(+), 67 deletions(-) diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 44e92638c0cd1..08f7ce2c16c3b 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -255,6 +255,78 @@ static unsigned long calculate_psi_aligned_address(uns= igned long start, return ALIGN_DOWN(start, VTD_PAGE_SIZE << mask); } =20 +static void cache_tag_flush_iotlb(struct dmar_domain *domain, struct cache= _tag *tag, + unsigned long addr, unsigned long pages, + unsigned long mask, int ih) +{ + struct intel_iommu *iommu =3D tag->iommu; + u64 type =3D DMA_TLB_PSI_FLUSH; + + if (domain->use_first_level) { + qi_flush_piotlb(iommu, tag->domain_id, tag->pasid, addr, pages, ih); + return; + } + + /* + * Fallback to domain selective flush if no PSI support or the size + * is too big. + */ + if (!cap_pgsel_inv(iommu->cap) || + mask > cap_max_amask_val(iommu->cap) || pages =3D=3D -1) { + addr =3D 0; + mask =3D 0; + ih =3D 0; + type =3D DMA_TLB_DSI_FLUSH; + } + + if (ecap_qis(iommu->ecap)) + qi_flush_iotlb(iommu, tag->domain_id, addr | ih, mask, type); + else + __iommu_flush_iotlb(iommu, tag->domain_id, addr | ih, mask, type); +} + +static void cache_tag_flush_devtlb_psi(struct dmar_domain *domain, struct = cache_tag *tag, + unsigned long addr, unsigned long mask) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + if (tag->pasid =3D=3D IOMMU_NO_PASID) { + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + addr, mask); + if (info->dtlb_extra_inval) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, + info->ats_qdep, addr, mask); + return; + } + + qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, tag->pasid, + info->ats_qdep, addr, mask); + if (info->dtlb_extra_inval) + qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, tag->pasid, + info->ats_qdep, addr, mask); +} + +static void cache_tag_flush_devtlb_all(struct dmar_domain *domain, struct = cache_tag *tag) +{ + struct intel_iommu *iommu =3D tag->iommu; + struct device_domain_info *info; + u16 sid; + + info =3D dev_iommu_priv_get(tag->dev); + sid =3D PCI_DEVID(info->bus, info->devfn); + + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, + MAX_AGAW_PFN_WIDTH); + if (info->dtlb_extra_inval) + qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, + MAX_AGAW_PFN_WIDTH); +} + /* * Invalidates a range of IOVA from @start (inclusive) to @end (inclusive) * when the memory mappings in the target domain have been modified. @@ -270,30 +342,10 @@ void cache_tag_flush_range(struct dmar_domain *domain= , unsigned long start, =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { - struct intel_iommu *iommu =3D tag->iommu; - struct device_domain_info *info; - u16 sid; - switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: - if (domain->use_first_level) { - qi_flush_piotlb(iommu, tag->domain_id, - tag->pasid, addr, pages, ih); - } else { - /* - * Fallback to domain selective flush if no - * PSI support or the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || - mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - addr | ih, mask, - DMA_TLB_PSI_FLUSH); - } + cache_tag_flush_iotlb(domain, tag, addr, pages, mask, ih); break; case CACHE_TAG_NESTING_DEVTLB: /* @@ -307,18 +359,7 @@ void cache_tag_flush_range(struct dmar_domain *domain,= unsigned long start, mask =3D MAX_AGAW_PFN_WIDTH; fallthrough; case CACHE_TAG_DEVTLB: - info =3D dev_iommu_priv_get(tag->dev); - sid =3D PCI_DEVID(info->bus, info->devfn); - - if (tag->pasid =3D=3D IOMMU_NO_PASID) - qi_flush_dev_iotlb(iommu, sid, info->pfsid, - info->ats_qdep, addr, mask); - else - qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, - tag->pasid, info->ats_qdep, - addr, mask); - - quirk_extra_dev_tlb_flush(info, addr, mask, tag->pasid, info->ats_qdep); + cache_tag_flush_devtlb_psi(domain, tag, addr, mask); break; } =20 @@ -338,29 +379,14 @@ void cache_tag_flush_all(struct dmar_domain *domain) =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { - struct intel_iommu *iommu =3D tag->iommu; - struct device_domain_info *info; - u16 sid; - switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: - if (domain->use_first_level) - qi_flush_piotlb(iommu, tag->domain_id, - tag->pasid, 0, -1, 0); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); + cache_tag_flush_iotlb(domain, tag, 0, -1, 0, 0); break; case CACHE_TAG_DEVTLB: case CACHE_TAG_NESTING_DEVTLB: - info =3D dev_iommu_priv_get(tag->dev); - sid =3D PCI_DEVID(info->bus, info->devfn); - - qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, - 0, MAX_AGAW_PFN_WIDTH); - quirk_extra_dev_tlb_flush(info, 0, MAX_AGAW_PFN_WIDTH, - IOMMU_NO_PASID, info->ats_qdep); + cache_tag_flush_devtlb_all(domain, tag); break; } =20 @@ -399,20 +425,8 @@ void cache_tag_flush_range_np(struct dmar_domain *doma= in, unsigned long start, } =20 if (tag->type =3D=3D CACHE_TAG_IOTLB || - tag->type =3D=3D CACHE_TAG_NESTING_IOTLB) { - /* - * Fallback to domain selective flush if no - * PSI support or the size is too big. - */ - if (!cap_pgsel_inv(iommu->cap) || - mask > cap_max_amask_val(iommu->cap)) - iommu->flush.flush_iotlb(iommu, tag->domain_id, - 0, 0, DMA_TLB_DSI_FLUSH); - else - iommu->flush.flush_iotlb(iommu, tag->domain_id, - addr, mask, - DMA_TLB_PSI_FLUSH); - } + tag->type =3D=3D CACHE_TAG_NESTING_IOTLB) + cache_tag_flush_iotlb(domain, tag, addr, pages, mask, 0); =20 trace_cache_tag_flush_range_np(tag, start, end, addr, pages, mask); } diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 159da629349c4..14020414af892 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1184,9 +1184,8 @@ static void __iommu_flush_context(struct intel_iommu = *iommu, raw_spin_unlock_irqrestore(&iommu->register_lock, flag); } =20 -/* return value determine if we need a write buffer flush */ -static void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, - u64 addr, unsigned int size_order, u64 type) +void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, + unsigned int size_order, u64 type) { int tlb_offset =3D ecap_iotlb_offset(iommu->ecap); u64 val =3D 0, val_iva =3D 0; diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index e297a322ba2d9..74634805abd19 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -1206,6 +1206,9 @@ void qi_flush_pasid_cache(struct intel_iommu *iommu, = u16 did, u64 granu, =20 int qi_submit_sync(struct intel_iommu *iommu, struct qi_desc *desc, unsigned int count, unsigned long options); + +void __iommu_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr, + unsigned int size_order, u64 type); /* * Options used in qi_submit_sync: * QI_OPT_WAIT_DRAIN - Wait for PRQ drain completion, spec 6.5.2.8. --=20 2.34.1 From nobody Fri Dec 19 17:35:43 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9FCA17A597 for ; Thu, 15 Aug 2024 06:53:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704813; cv=none; b=BYt0uRP2+WUEYWmtVsOfHyx/tf7rEpDPyx6MP8j0/r6KweNq34ThE8Uo4kFhao2NRDjeSUPS/Vk39zp2CVWYrTQBi0D6rBfHhGl0BShYecaUijbA/0IntxiMEKeShomm50CgiiKBrb0/XooQHYLEArKmsKs8R+j3Xtk61x9009A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704813; c=relaxed/simple; bh=LfGSL2s3OqRer5h+/wGQYpRy+Iut/M0xOkkSLN974M4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mv+z7Q2/Ke3dp1v0lIGP3nfmsFDeECnmYip3rsKyBRsfUM2Tn8L0M3Zpk9XmSlqMRRWJXaPdfFLuQgdaeBWWUWlwvDxc6pQzT8Aa+PDdkh0Wr3cr0/wIyIytJVfMDNzB8tJpXOoPyWUZD8PrWcEIYouQzhIJqVK/zqXwWjH9SQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=I8JhGG0C; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="I8JhGG0C" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723704811; x=1755240811; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LfGSL2s3OqRer5h+/wGQYpRy+Iut/M0xOkkSLN974M4=; b=I8JhGG0CymNM3irXhfmMiZEQIHvQqL4w/ob/J6f0uluirf/kZ+yGOea7 DK97PCyD4Swl9JskjS8YLxjpKv4OW4pZrDrF5WMphiwm/7N89KDQhU1Ah p+Gspi1U1mX1xFFVPC/uVTWRwGUFE+EzbK3oOSalz5IVD5nT4SZv2PS1D RRqlTMZwRPUabaYy7tm7Lb7Qot9EHKvYbZC+hqux+hxWjSrPVuZoPPAZ7 r3uBu/kpUhKVs5y1H8M5iMeKRg1uLY/7K0vwVKqyGVtwIWvKnuX+Acnje vNz0c+3ZX0C0UPsGMGjlaH8R3HTj7eHCwMGUwhmG9xeYL/F5XUpXETXYD Q==; X-CSE-ConnectionGUID: qZZGGaIrQmS7i0IXvrbsPw== X-CSE-MsgGUID: Kgr6YmibRleuakgJUG20yw== X-IronPort-AV: E=McAfee;i="6700,10204,11164"; a="32528290" X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="32528290" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Aug 2024 23:53:30 -0700 X-CSE-ConnectionGUID: bUACLD1mSDCgVXMOcQCgTQ== X-CSE-MsgGUID: 3XMDrh4bQxWgbZc3PNHQug== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="82471574" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa002.fm.intel.com with ESMTP; 14 Aug 2024 23:53:29 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v3 3/4] iommu/vt-d: Add qi_batch for dmar_domain Date: Thu, 15 Aug 2024 14:52:20 +0800 Message-Id: <20240815065221.50328-4-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240815065221.50328-1-tina.zhang@intel.com> References: <20240815065221.50328-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Lu Baolu Introduces a qi_batch structure to hold batched cache invalidation descriptors on a per-dmar_domain basis. A fixed-size descriptor array is used for simplicity. The qi_batch is allocated when the first cache tag is added to the domain and freed during iommu_free_domain(). Signed-off-by: Lu Baolu Signed-off-by: Tina Zhang --- drivers/iommu/intel/cache.c | 7 +++++++ drivers/iommu/intel/iommu.c | 1 + drivers/iommu/intel/iommu.h | 14 ++++++++++++++ drivers/iommu/intel/nested.c | 1 + drivers/iommu/intel/svm.c | 5 ++++- 5 files changed, 27 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 08f7ce2c16c3b..21485c86e7381 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -190,6 +190,13 @@ int cache_tag_assign_domain(struct dmar_domain *domain, u16 did =3D domain_get_id_for_dev(domain, dev); int ret; =20 + /* domain->qi_bach will be freed in iommu_free_domain() path. */ + if (!domain->qi_batch) { + domain->qi_batch =3D kzalloc(sizeof(struct qi_batch), GFP_KERNEL); + if (!domain->qi_batch) + return -ENOMEM; + } + ret =3D __cache_tag_assign_domain(domain, did, dev, pasid); if (ret || domain->domain.type !=3D IOMMU_DOMAIN_NESTED) return ret; diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 14020414af892..cde5ed4d0fede 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1586,6 +1586,7 @@ static void domain_exit(struct dmar_domain *domain) if (WARN_ON(!list_empty(&domain->devices))) return; =20 + kfree(domain->qi_batch); kfree(domain); } =20 diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 74634805abd19..d21eca94cb8f9 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -584,6 +584,19 @@ struct iommu_domain_info { * to VT-d spec, section 9.3 */ }; =20 +/* + * We start simply by using a fixed size for the batched descriptors. This + * size is currently sufficient for our needs. Future improvements could + * involve dynamically allocating the batch buffer based on actual demand, + * allowing us to adjust the batch size for optimal performance in differe= nt + * scenarios. + */ +#define QI_MAX_BATCHED_DESC_COUNT 16 +struct qi_batch { + struct qi_desc descs[QI_MAX_BATCHED_DESC_COUNT]; + unsigned int index; +}; + struct dmar_domain { int nid; /* node id */ struct xarray iommu_array; /* Attached IOMMU array */ @@ -608,6 +621,7 @@ struct dmar_domain { =20 spinlock_t cache_lock; /* Protect the cache tag list */ struct list_head cache_tags; /* Cache tag list */ + struct qi_batch *qi_batch; /* Batched QI descriptors */ =20 int iommu_superpage;/* Level of superpages supported: 0 =3D=3D 4KiB (no superpages), 1 =3D=3D 2MiB, diff --git a/drivers/iommu/intel/nested.c b/drivers/iommu/intel/nested.c index 36a91b1b52be3..433c58944401f 100644 --- a/drivers/iommu/intel/nested.c +++ b/drivers/iommu/intel/nested.c @@ -83,6 +83,7 @@ static void intel_nested_domain_free(struct iommu_domain = *domain) spin_lock(&s2_domain->s1_lock); list_del(&dmar_domain->s2_link); spin_unlock(&s2_domain->s1_lock); + kfree(dmar_domain->qi_batch); kfree(dmar_domain); } =20 diff --git a/drivers/iommu/intel/svm.c b/drivers/iommu/intel/svm.c index 0e3a9b38bef21..3421813995db8 100644 --- a/drivers/iommu/intel/svm.c +++ b/drivers/iommu/intel/svm.c @@ -184,7 +184,10 @@ static void intel_mm_release(struct mmu_notifier *mn, = struct mm_struct *mm) =20 static void intel_mm_free_notifier(struct mmu_notifier *mn) { - kfree(container_of(mn, struct dmar_domain, notifier)); + struct dmar_domain *domain =3D container_of(mn, struct dmar_domain, notif= ier); + + kfree(domain->qi_batch); + kfree(domain); } =20 static const struct mmu_notifier_ops intel_mmuops =3D { --=20 2.34.1 From nobody Fri Dec 19 17:35:43 2025 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CB0C17B4FA for ; Thu, 15 Aug 2024 06:53:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=198.175.65.11 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704814; cv=none; b=HRgM6ZGIhCkChjgvuRcbdrM/6mG2h8TLuNANDVKDnNVh9luXhl3KfFZ69s9Wo2W0zMG1pvKfYnNM2L8kzzRGS6pqN44jjqfZAtYnDaRu/h1fRjtttaZTSOkFMRP3Ek8qeJq/4jAaDFcCl7OSF4GbSudbDa5PbgwzqeGxupZAmbs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723704814; c=relaxed/simple; bh=EZRupiXcACIS811ijjV9HK/TmFHYK3tDinzBUIa98RM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=C9eNkDIfkDXJut4M+XdE2wBLR3Mk8d+gQKk4KeUh95m9XF6W+uAqsT5LP1EZ2z255aS/2SYR39UWYHpDR2WxMq5MmPeF+fzThBoTqGyo2q9uMOJB0tGh0bJhJd+ztmziSqmCZPJmyY6blTKCStCDM36GKmzWgmgT0DDyr5GlOw4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com; spf=pass smtp.mailfrom=intel.com; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b=CvY66DxZ; arc=none smtp.client-ip=198.175.65.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="CvY66DxZ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723704812; x=1755240812; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=EZRupiXcACIS811ijjV9HK/TmFHYK3tDinzBUIa98RM=; b=CvY66DxZkE+j5btRiw18/OlAkUVe1u/SM+O+OE3Y/BS31iJAYGrWyj5D GU5dk94HbAo3ApFC4UlW4ZXFn+jU/ViPETzr9xZF6+wn3sHSZAOmUlrkV qvrnD+qfzsuZeKkH5Um4YYJlHe1sCV0rjOoPNPUTBHIpdFbkdW2NsozSD mFFlG+jaMyfxYUODsjezPlqJkNgqx+qY/u5An8HnzmaE7gGugruT927B0 KwxG6177x478O5DDZpzkLVVhOCnrfFgr1D6d47wjpnzTFnrZ4qBcPU1G1 /4J9vwC1so6qtv+UFifYrQKQn7b3ittRyBUH6YSWB13YkX+KvvAS/xhKZ g==; X-CSE-ConnectionGUID: h/007jjQQViJnd+a0+ZV5g== X-CSE-MsgGUID: iiU9tBfpQomgospwJzgI0A== X-IronPort-AV: E=McAfee;i="6700,10204,11164"; a="32528296" X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="32528296" Received: from fmviesa002.fm.intel.com ([10.60.135.142]) by orvoesa103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Aug 2024 23:53:32 -0700 X-CSE-ConnectionGUID: Qp922m7wR+SFq2/AXOrdiw== X-CSE-MsgGUID: ZYUHu4UKQGGD0IhSTc5spA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,148,1719903600"; d="scan'208";a="82471591" Received: from desktoptgl-tinazhang.bj.intel.com ([10.238.156.103]) by fmviesa002.fm.intel.com with ESMTP; 14 Aug 2024 23:53:30 -0700 From: Tina Zhang To: Lu Baolu , Kevin Tian Cc: iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Tina Zhang Subject: [PATCH v3 4/4] iommu/vt-d: Introduce batched cache invalidation Date: Thu, 15 Aug 2024 14:52:21 +0800 Message-Id: <20240815065221.50328-5-tina.zhang@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240815065221.50328-1-tina.zhang@intel.com> References: <20240815065221.50328-1-tina.zhang@intel.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Converts IOTLB and Dev-IOTLB invalidation to a batched model. Cache tag invalidation requests for a domain are now accumulated in a qi_batch structure before being flushed in bulk. It replaces the previous per- request qi_flush approach with a more efficient batching mechanism. Co-developed-by: Lu Baolu Signed-off-by: Lu Baolu Signed-off-by: Tina Zhang --- drivers/iommu/intel/cache.c | 122 +++++++++++++++++++++++++++++++----- 1 file changed, 107 insertions(+), 15 deletions(-) diff --git a/drivers/iommu/intel/cache.c b/drivers/iommu/intel/cache.c index 21485c86e7381..983769de3bc90 100644 --- a/drivers/iommu/intel/cache.c +++ b/drivers/iommu/intel/cache.c @@ -262,6 +262,79 @@ static unsigned long calculate_psi_aligned_address(uns= igned long start, return ALIGN_DOWN(start, VTD_PAGE_SIZE << mask); } =20 +static void qi_batch_flush_descs(struct intel_iommu *iommu, struct qi_batc= h *batch) +{ + if (!iommu || !batch->index) + return; + + qi_submit_sync(iommu, batch->descs, batch->index, 0); + + /* Reset the index value and clean the whole batch buffer. */ + memset(batch, 0, sizeof(*batch)); +} + +static void qi_batch_increment_index(struct intel_iommu *iommu, struct qi_= batch *batch) +{ + if (++batch->index =3D=3D QI_MAX_BATCHED_DESC_COUNT) + qi_batch_flush_descs(iommu, batch); +} + +static void qi_batch_add_iotlb(struct intel_iommu *iommu, u16 did, u64 add= r, + unsigned int size_order, u64 type, + struct qi_batch *batch) +{ + qi_desc_iotlb(iommu, did, addr, size_order, type, &(batch->descs[batch->i= ndex])); + qi_batch_increment_index(iommu, batch); +} + +static void qi_batch_add_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16= pfsid, + u16 qdep, u64 addr, unsigned int mask, + struct qi_batch *batch) +{ + /* + * According to VT-d spec, software is recommended to not submit any Devi= ce-TLB + * invalidation requests while address remapping hardware is disabled. + */ + if (!(iommu->gcmd & DMA_GCMD_TE)) + return; + + qi_desc_dev_iotlb(sid, pfsid, qdep, addr, mask, &(batch->descs[batch->ind= ex])); + qi_batch_increment_index(iommu, batch); +} + +static void qi_batch_add_piotlb(struct intel_iommu *iommu, u16 did, u32 pa= sid, + u64 addr, unsigned long npages, bool ih, + struct qi_batch *batch) +{ + /* + * npages =3D=3D -1 means a PASID-selective invalidation, otherwise, + * a positive value for Page-selective-within-PASID invalidation. + * 0 is not a valid input. + */ + if (!npages) + return; + + qi_desc_piotlb(did, pasid, addr, npages, ih, &(batch->descs[batch->index]= )); + qi_batch_increment_index(iommu, batch); +} + +static void qi_batch_add_pasid_dev_iotlb(struct intel_iommu *iommu, u16 si= d, u16 pfsid, + u32 pasid, u16 qdep, u64 addr, + unsigned int size_order, struct qi_batch *batch) +{ + /* + * According to VT-d spec, software is recommended to not submit any + * Device-TLB invalidation requests while address remapping hardware + * is disabled. + */ + if (!(iommu->gcmd & DMA_GCMD_TE)) + return; + + qi_desc_dev_iotlb_pasid(sid, pfsid, pasid, qdep, addr, size_order, + &(batch->descs[batch->index])); + qi_batch_increment_index(iommu, batch); +} + static void cache_tag_flush_iotlb(struct dmar_domain *domain, struct cache= _tag *tag, unsigned long addr, unsigned long pages, unsigned long mask, int ih) @@ -270,7 +343,8 @@ static void cache_tag_flush_iotlb(struct dmar_domain *d= omain, struct cache_tag * u64 type =3D DMA_TLB_PSI_FLUSH; =20 if (domain->use_first_level) { - qi_flush_piotlb(iommu, tag->domain_id, tag->pasid, addr, pages, ih); + qi_batch_add_piotlb(iommu, tag->domain_id, tag->pasid, addr, + pages, ih, domain->qi_batch); return; } =20 @@ -287,7 +361,8 @@ static void cache_tag_flush_iotlb(struct dmar_domain *d= omain, struct cache_tag * } =20 if (ecap_qis(iommu->ecap)) - qi_flush_iotlb(iommu, tag->domain_id, addr | ih, mask, type); + qi_batch_add_iotlb(iommu, tag->domain_id, addr | ih, mask, type, + domain->qi_batch); else __iommu_flush_iotlb(iommu, tag->domain_id, addr | ih, mask, type); } @@ -303,19 +378,20 @@ static void cache_tag_flush_devtlb_psi(struct dmar_do= main *domain, struct cache_ sid =3D PCI_DEVID(info->bus, info->devfn); =20 if (tag->pasid =3D=3D IOMMU_NO_PASID) { - qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, - addr, mask); + qi_batch_add_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + addr, mask, domain->qi_batch); if (info->dtlb_extra_inval) - qi_flush_dev_iotlb(iommu, sid, info->pfsid, - info->ats_qdep, addr, mask); + qi_batch_add_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, + addr, mask, domain->qi_batch); return; } =20 - qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, tag->pasid, - info->ats_qdep, addr, mask); + qi_batch_add_pasid_dev_iotlb(iommu, sid, info->pfsid, tag->pasid, + info->ats_qdep, addr, mask, domain->qi_batch); if (info->dtlb_extra_inval) - qi_flush_dev_iotlb_pasid(iommu, sid, info->pfsid, tag->pasid, - info->ats_qdep, addr, mask); + qi_batch_add_pasid_dev_iotlb(iommu, sid, info->pfsid, tag->pasid, + info->ats_qdep, addr, mask, + domain->qi_batch); } =20 static void cache_tag_flush_devtlb_all(struct dmar_domain *domain, struct = cache_tag *tag) @@ -327,11 +403,11 @@ static void cache_tag_flush_devtlb_all(struct dmar_do= main *domain, struct cache_ info =3D dev_iommu_priv_get(tag->dev); sid =3D PCI_DEVID(info->bus, info->devfn); =20 - qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, - MAX_AGAW_PFN_WIDTH); + qi_batch_add_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, + MAX_AGAW_PFN_WIDTH, domain->qi_batch); if (info->dtlb_extra_inval) - qi_flush_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, - MAX_AGAW_PFN_WIDTH); + qi_batch_add_dev_iotlb(iommu, sid, info->pfsid, info->ats_qdep, 0, + MAX_AGAW_PFN_WIDTH, domain->qi_batch); } =20 /* @@ -341,6 +417,7 @@ static void cache_tag_flush_devtlb_all(struct dmar_doma= in *domain, struct cache_ void cache_tag_flush_range(struct dmar_domain *domain, unsigned long start, unsigned long end, int ih) { + struct intel_iommu *iommu =3D NULL; unsigned long pages, mask, addr; struct cache_tag *tag; unsigned long flags; @@ -349,6 +426,10 @@ void cache_tag_flush_range(struct dmar_domain *domain,= unsigned long start, =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { + if (iommu && iommu !=3D tag->iommu) + qi_batch_flush_descs(iommu, domain->qi_batch); + iommu =3D tag->iommu; + switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: @@ -372,6 +453,7 @@ void cache_tag_flush_range(struct dmar_domain *domain, = unsigned long start, =20 trace_cache_tag_flush_range(tag, start, end, addr, pages, mask); } + qi_batch_flush_descs(iommu, domain->qi_batch); spin_unlock_irqrestore(&domain->cache_lock, flags); } =20 @@ -381,11 +463,16 @@ void cache_tag_flush_range(struct dmar_domain *domain= , unsigned long start, */ void cache_tag_flush_all(struct dmar_domain *domain) { + struct intel_iommu *iommu =3D NULL; struct cache_tag *tag; unsigned long flags; =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { + if (iommu && iommu !=3D tag->iommu) + qi_batch_flush_descs(iommu, domain->qi_batch); + iommu =3D tag->iommu; + switch (tag->type) { case CACHE_TAG_IOTLB: case CACHE_TAG_NESTING_IOTLB: @@ -399,6 +486,7 @@ void cache_tag_flush_all(struct dmar_domain *domain) =20 trace_cache_tag_flush_all(tag); } + qi_batch_flush_descs(iommu, domain->qi_batch); spin_unlock_irqrestore(&domain->cache_lock, flags); } =20 @@ -416,6 +504,7 @@ void cache_tag_flush_all(struct dmar_domain *domain) void cache_tag_flush_range_np(struct dmar_domain *domain, unsigned long st= art, unsigned long end) { + struct intel_iommu *iommu =3D NULL; unsigned long pages, mask, addr; struct cache_tag *tag; unsigned long flags; @@ -424,7 +513,9 @@ void cache_tag_flush_range_np(struct dmar_domain *domai= n, unsigned long start, =20 spin_lock_irqsave(&domain->cache_lock, flags); list_for_each_entry(tag, &domain->cache_tags, node) { - struct intel_iommu *iommu =3D tag->iommu; + if (iommu && iommu !=3D tag->iommu) + qi_batch_flush_descs(iommu, domain->qi_batch); + iommu =3D tag->iommu; =20 if (!cap_caching_mode(iommu->cap) || domain->use_first_level) { iommu_flush_write_buffer(iommu); @@ -437,5 +528,6 @@ void cache_tag_flush_range_np(struct dmar_domain *domai= n, unsigned long start, =20 trace_cache_tag_flush_range_np(tag, start, end, addr, pages, mask); } + qi_batch_flush_descs(iommu, domain->qi_batch); spin_unlock_irqrestore(&domain->cache_lock, flags); } --=20 2.34.1