From nobody Mon Oct 6 01:49:55 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A79D27AC4B for ; Mon, 28 Jul 2025 17:53:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753725227; cv=none; b=aJHFMwg4bPd5FIRAxR5r/jOwDWGOc6YSXtsokizsyqlyY6dy5Xh/QTj5thNErA7R1dMM7w2RQ76AvklqQrEqCvZ2Djf6p1bK2CTQSEzJkLIeHUCCxuAlgqa0NG2D3RM80m3ORBwnJE2J1XOwGSq96alPOsy9luZQqDt4BF2eIzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1753725227; c=relaxed/simple; bh=IWszQThaLRIClyIX4i0D/ufNTnIxoyf1Y2v1vsCiCxE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=f6j18g3oQrlNLcw7VFZP2p6C+SCbEmxtvBSsGF7afkIl2cwASZjqTdWbIuXyMJ+g+VPgUkd6bEYo4VDKTm7JfKXyxdZQ370CCs3ftsev5+45YV0aCKspgVx/hRBb2KmzFx6OI+zrehqU7nYo8JRQG/2ElbtOnodKMGToZg44ovg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bpOFRmME; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bpOFRmME" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4538f375e86so36589245e9.3 for ; Mon, 28 Jul 2025 10:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1753725223; x=1754330023; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6e/2TheqBDOEi4HSiBuM8XyP32c0aQfj3qnc6fRtrsM=; b=bpOFRmMEnuYZ65xv6bvF3WLanxp+B8CtSpTF38uPfVvejjS3s3St2cXodBqX49IaFy uoieBhznst0EE7zRyDKdrqB5lHKGBvPQCFNT+UkOI4qKLnga2OWlZKqsPDoon/gj+7IW L8lkwdzPsHbQDTNW4sicFmEyvI6CiQwy0bMTf4t5nRT1Wi5pz6HGvK6XkVXRbhEQFzBe Rw5E5jvtQ9A3AyoHYW//WO2baMmyHBlXJJW1gzaDHMQGQ8tXavHJEuQzH14fDcTDjBqC qFwVnqKPXpCltSm2l56cSCk7+mu9QSeLJNAqOoGjoDufn83Ao3DVs59dluv7tW0Aj1QW x7UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753725223; x=1754330023; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6e/2TheqBDOEi4HSiBuM8XyP32c0aQfj3qnc6fRtrsM=; b=jU1ZdND2njpgj3/tfd7y86Z8k2xlUGbhpBjf2OP2keQt0H1YImgC1o5Hy8+wPywmHr SAjTacfHt8aNFDKmNbna2jUuTDlxv0NYHLW99XDrYzg7AEoD258RWZ0TEmcj26k26FUi WTyfuWC2mlEuNGg+VAd65OYwT6/dj4L4O/f7TWFcql17wh6OHkox9nuAoD5jrdbrahA7 xbK5/91aNP1idmXmtCMrUZ552ELzsjh4WDJ0o3c2R0jqBBnVkjenpA3NGIi4ltmLHZSj IdMOPJdehgeRjx5hEQEuOZUa4640gUIyfSRmFhSNCR08l1xEzMjSnCVZ3uP0bo8zjiEa /p1A== X-Gm-Message-State: AOJu0Yzt4lZBjsP0L0wfIX8tJCjBMuJuRO67qwyE1krBJEPsBu2ah7DJ 8cpPTZljJisZTx7MEXNfatm1RNC/7HkhJzPU2gDuIkWizHvP0cCImqXZMUk7VqSaOerJ2k1XCHa bkBt0co16ONBs0z5geo2RjomMeTBGqmqbg9XvNehQsJBkFiPsfvHsIb99JCrg2CT2lLpjD54TEG DzhX1wGUIBCaWWdD8l/TRU5lJf+0ac4TN5uzvj8kMdGhwVcGv4H4Xf0bk= X-Google-Smtp-Source: AGHT+IEDIjSlCUVUxCpo1rhb6F+umTy0J4Onr6Gk0yVTZfYuvmOMhy6s8JdN8rpuLmzFtQt/COWBfeqeFpc2iQ== X-Received: from wrbgv1.prod.google.com ([2002:a05:6000:4601:b0:3b7:8ac1:5f96]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4011:b0:3a5:243c:6042 with SMTP id ffacd0b85a97d-3b7765e58a1mr8332168f8f.2.1753725223461; Mon, 28 Jul 2025 10:53:43 -0700 (PDT) Date: Mon, 28 Jul 2025 17:53:00 +0000 In-Reply-To: <20250728175316.3706196-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250728175316.3706196-1-smostafa@google.com> X-Mailer: git-send-email 2.50.1.552.g942d659e1b-goog Message-ID: <20250728175316.3706196-14-smostafa@google.com> Subject: [PATCH v3 13/29] iommu/arm-smmu-v3: Move TLB range invalidation into a macro From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Range TLB invalidation has a very specific algorithm, instead of re-writing it for the hypervisor, put it in a macro so it can be re-used. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/arm-smmu-v3-common.h | 65 +++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 59 +---------------- 2 files changed, 68 insertions(+), 56 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h b/drivers/i= ommu/arm/arm-smmu-v3/arm-smmu-v3-common.h index f115ff9a547f..ab6db8b4b9f2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common.h @@ -540,4 +540,69 @@ static inline void queue_write(__le64 *dst, u64 *src, = size_t n_dwords) for (i =3D 0; i < n_dwords; ++i) *dst++ =3D cpu_to_le64(*src++); } + +/** + * arm_smmu_tlb_inv_build - Create a range invalidation command + * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid. + * @iova: Start IOVA to invalidate + * @size: Size of range + * @granule: Granule of invalidation + * @pgsize_bitmap: Page size bit map of the page table. + * @smmu: Struct for the smmu, must have ::features + * @add_cmd: Function to send/batch the invalidation command + * @cmds: Incase of batching, it includes the pointer to the batch + */ +#define arm_smmu_tlb_inv_build(cmd, iova, size, granule, pgsize_bitmap, sm= mu, add_cmd, cmds) \ +{ \ + unsigned long _iova =3D (iova); \ + size_t _size =3D (size); \ + size_t _granule =3D (granule); \ + unsigned long end =3D _iova + _size, num_pages =3D 0, tg =3D 0; \ + size_t inv_range =3D _granule; \ + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \ + /* Get the leaf page size */ \ + tg =3D __ffs(pgsize_bitmap); \ + num_pages =3D _size >> tg; \ + /* Convert page size of 12,14,16 (log2) to 1,2,3 */ \ + cmd->tlbi.tg =3D (tg - 10) / 2; \ + /* + * Determine what level the granule is at. For non-leaf, both + * io-pgtable and SVA pass a nominal last-level granule because + * they don't know what level(s) actually apply, so ignore that + * and leave TTL=3D0. However for various errata reasons we still + * want to use a range command, so avoid the SVA corner case + * where both scale and num could be 0 as well. + */ \ + if (cmd->tlbi.leaf) \ + cmd->tlbi.ttl =3D 4 - ((ilog2(_granule) - 3) / (tg - 3)); \ + else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) \ + num_pages++; \ + } \ + while (_iova < end) { \ + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \ + /* + * On each iteration of the loop, the range is 5 bits + * worth of the aligned size remaining. + * The range in pages is: + * + * range =3D (num_pages & (0x1f << __ffs(num_pages))) + */ \ + unsigned long scale, num; \ + /* Determine the power of 2 multiple number of pages */ \ + scale =3D __ffs(num_pages); \ + cmd->tlbi.scale =3D scale; \ + /* Determine how many chunks of 2^scale size we have */ \ + num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; \ + cmd->tlbi.num =3D num - 1; \ + /* range is num * 2^scale * pgsize */ \ + inv_range =3D num << (scale + tg); \ + /* Clear out the lower order bits for the next iteration */ \ + num_pages -=3D num << scale; \ + } \ + cmd->tlbi.addr =3D iova; \ + add_cmd(smmu, cmds, cmd); \ + _iova +=3D inv_range; \ + } \ +} \ + #endif /* _ARM_SMMU_V3_COMMON_H */ \ No newline at end of file diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 066d9d9e8088..245b39ddef66 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2075,68 +2075,15 @@ static void __arm_smmu_tlb_inv_range(struct arm_smm= u_cmdq_ent *cmd, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_device *smmu =3D smmu_domain->smmu; - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; - size_t inv_range =3D granule; struct arm_smmu_cmdq_batch cmds; =20 if (!size) return; =20 - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* Get the leaf page size */ - tg =3D __ffs(smmu_domain->domain.pgsize_bitmap); - - num_pages =3D size >> tg; - - /* Convert page size of 12,14,16 (log2) to 1,2,3 */ - cmd->tlbi.tg =3D (tg - 10) / 2; - - /* - * Determine what level the granule is at. For non-leaf, both - * io-pgtable and SVA pass a nominal last-level granule because - * they don't know what level(s) actually apply, so ignore that - * and leave TTL=3D0. However for various errata reasons we still - * want to use a range command, so avoid the SVA corner case - * where both scale and num could be 0 as well. - */ - if (cmd->tlbi.leaf) - cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); - else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) - num_pages++; - } - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - - while (iova < end) { - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* - * On each iteration of the loop, the range is 5 bits - * worth of the aligned size remaining. - * The range in pages is: - * - * range =3D (num_pages & (0x1f << __ffs(num_pages))) - */ - unsigned long scale, num; - - /* Determine the power of 2 multiple number of pages */ - scale =3D __ffs(num_pages); - cmd->tlbi.scale =3D scale; - - /* Determine how many chunks of 2^scale size we have */ - num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; - cmd->tlbi.num =3D num - 1; - - /* range is num * 2^scale * pgsize */ - inv_range =3D num << (scale + tg); - - /* Clear out the lower order bits for the next iteration */ - num_pages -=3D num << scale; - } - - cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, &cmds, cmd); - iova +=3D inv_range; - } + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + smmu_domain->domain.pgsize_bitmap, + smmu, arm_smmu_cmdq_batch_add, &cmds); arm_smmu_cmdq_batch_submit(smmu, &cmds); } =20 --=20 2.50.1.552.g942d659e1b-goog