From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0918847B406 for ; Thu, 15 Oct 2026 20:09:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094955; cv=none; b=i+lKu/OPl5DbBiDYBt6x+sy5DYVBe2qCrPdfXFHSJ1HYfOQ3sdFXoWP74UbqHJpTYafFE42Pg+/RJ0iqNjx8MXOMKN8UauBj77k1/5ljl8q3nL75Uz8Syj54lPabh6LxLBYS2j1442cDJENEBG6yoexB5Hcxo9v6VNKOFoyre0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094955; c=relaxed/simple; bh=YxLFkA27bOezQ/xTbhKCXlt4eRe3P5CCTBEyjjxxyRk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bgYN5Ocz/xKuY1dhlQAkFTHCaI6UAdiVVnrhk40Ht1j3Lb08USx4dOewJp/V9VtdRD7JwY0QcrxJXEIUMOMIC9yebpbZcGB/vshH/qkVeSKFzeXtOh1z5/W8+st3s4FtOHlq1apovMtgpm5EVWcNQn4KL7iJOof1f74uhWT3opc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PLICnJmB; arc=none smtp.client-ip=209.85.128.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PLICnJmB" Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4a17e77c5b1so20420895e9.1 for ; Thu, 15 Oct 2026 13:09:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094952; x=1792699752; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=IMjLsiB8JUe1zn3H8B72Qby1JQXygMn6MqjBzblWNdg=; b=PLICnJmBaS49fe6uqgrMfdDKWx+8HdJWh9SlOx79Ka+f/eWiIcUuMbi4tbnGpEPykR 3BE4jrEO/usoLxHeaO584k+90z4766t4mr4kYNd9ggF3I10ELAJl4gpADQ9tL3k62gEh AqWBxbZYktKBo8R7OZMx2S+Ail1ccKHR7bJyFPDeTRUfx7UupZ4uS77EAXAOea+scLBm AUenYCcyL3kX3S+1qrqr5FA+HNVW/klGo9IQY+fezNAL0v68Vcge2xawoTeLbfBvbnMq vI9wqKJd60JC3br3YCU4OxyjqsQPUOHznoaHXPSHpiZ7Pqcyw1X99J277Nvv5J5h/8IE YoYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094952; x=1792699752; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=IMjLsiB8JUe1zn3H8B72Qby1JQXygMn6MqjBzblWNdg=; b=v/Fb2Npixq77et7G4WppE8lTlUixAT3vEiRUU/aSHnO42u24xJ4KnxVyXMJ2VP67y6 o6oAwENdL+gKg2kRzqSFh2wrRHZYDMwWDEd4wPetTy7Ugvc5YYzE2l7iLONTfD/Vco/+ Qgr/4e0nUA+2nxv3WP6TgPyCi/355hKyV9jG8tmvZB8tPkY1LLXmofAN32ucnDDN3OdY S8XhdtOyCVpi+u65M+twNorhFZ96Voy4ldo+ZtbrAF7jaKrVec10Iz4YshgwlOE1FI1T ahBotHy9a3cJjugapPiirAK873tBgDpo4ocGYqeIgqadVNAioWlHb7U1rVw+5JIg5yDa Qjyw== X-Forwarded-Encrypted: i=1; AKwUvByorC+2hqKBobLS19YDJ7dw7ImlGklr6ictg/4lHcjcfanFIaVNo4R7Qa+HM9muttfx569fn2q5RVhE+6w=@vger.kernel.org X-Gm-Message-State: AFq9FYLgosPQDq8qiviHHTvhp77JbTMf+E8V6lscAq/b5clML/LhQO94 0kzqzUt8P1y2gNwTOn6dmh1HVkNFrmbPju1jq/wgE4TGhQm7iU0r22BHC7JQuhcwyczlVM9KDY0 r2bxjQlsynwxKzQ== X-Received: from wmbd18.prod.google.com ([2002:a05:600c:58d2:b0:4a1:7ec4:6621]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600d:17:b0:4a1:9663:7639 with SMTP id 5b1f17b1804b1-4a196637a39mr305024695e9.34.1792094952096; Thu, 15 Oct 2026 13:09:12 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:32 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-2-smostafa@google.com> Subject: [PATCH v9 01/25] KVM: arm64: Donate MMIO to the hypervisor From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add __pkvm_host_donate_hyp_mmio() so IOMMU drivers in the hypervisor can take ownership of the IOMMU MMIO. The range is mapped in the hypervisor private range and annotated as owned by the hypervisor in the host stage-2, so host accesses fault and further donations of the same range are refused. MMIO has no struct hyp_page, so the annotation is the only state, and a dedicated function is simpler than teaching the memory donation path about MMIO. Donation is only allowed before the host is de-privileged, which is when the hypervisor IOMMU drivers initialise. Make kvm_pgtable_stage2_unmap() leave invalid counted PTEs (annotations) in place. host_stage2_unmap_dev_all() unmaps all non-memory ranges to recycle page-table pages under memory pressure, and would otherwise drop the annotation and let the host fault the MMIO back in. No caller depends on the old behaviour: host_stage2_unmap_dev_all() only covers non-memory ranges, where no annotation existed until now, and the guest unmap paths check that the PTE is valid first. Returning an error for counted PTEs instead is not an option, as a non-zero visitor return stops the walk, which would abort recycling at the first hypervisor-owned MMIO page and inject an abort into the host for an unrelated address. Add __pkvm_hyp_donate_host_mmio() for the IOMMU driver error path. It unmaps the range from the hypervisor and maps it back in the host stage-2. The private VA range cannot be reclaimed and is leaked, as with every other failed private mapping. Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_pgtable.h | 2 + arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 10 ++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 99 ++++++++++++++++++- arch/arm64/kvm/hyp/pgtable.c | 8 +- 4 files changed, 112 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_pgtable.h b/arch/arm64/include/asm/= kvm_pgtable.h index 41a8687938eb..20b1adb8a058 100644 --- a/arch/arm64/include/asm/kvm_pgtable.h +++ b/arch/arm64/include/asm/kvm_pgtable.h @@ -713,6 +713,8 @@ int kvm_pgtable_stage2_annotate(struct kvm_pgtable *pgt= , u64 addr, u64 size, * freed. Unmapping a cacheable page will ensure that it is clean to the P= oC if * FWB is not supported by the CPU. * + * Invalid counted PTEs (annotations) are left in place. + * * Return: 0 on success, negative error code on failure. */ int kvm_pgtable_stage2_unmap(struct kvm_pgtable *pgt, u64 addr, u64 size); diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index ec85a9547120..c9c3e2dbbe58 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -36,6 +36,16 @@ int __pkvm_guest_share_host(struct pkvm_hyp_vcpu *vcpu, = u64 gfn); int __pkvm_guest_unshare_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); +/* + * Donate MMIO range to the hypervisor, it will be mapped in the hyperviso= r's + * private range and unmapped from the host stage-2. + */ +int __pkvm_host_donate_hyp_mmio(u64 pfn, u64 nr_pages, unsigned long *hadd= r); +/* + * Undo __pkvm_host_donate_hyp_mmio(), typically on an error path. @haddr = is + * the address returned by the donation. + */ +int __pkvm_hyp_donate_host_mmio(u64 pfn, u64 nr_pages, unsigned long haddr= ); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index a6a47c1e058b..75d250ca809c 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -385,7 +385,11 @@ static int host_stage2_unmap_dev_all(void) u64 addr =3D 0; int i, ret; =20 - /* Unmap all non-memory regions to recycle the pages */ + /* + * Unmap all non-memory regions to recycle the pages. + * That relies on kvm_pgtable_stage2_unmap() not clearing + * counted PTEs which include hypervisor MMIO. + */ for (i =3D 0; i < hyp_memblock_nr; i++, addr =3D reg->base + reg->size) { reg =3D &hyp_memory[i]; ret =3D kvm_pgtable_stage2_unmap(pgt, addr, reg->base - addr); @@ -1146,6 +1150,99 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) return ret; } =20 +int __pkvm_host_donate_hyp_mmio(u64 pfn, u64 nr_pages, unsigned long *hadd= r) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + u64 size =3D PAGE_SIZE * nr_pages; + unsigned long addr; + kvm_pte_t pte; + u64 offset; + int ret; + + /* Only before de-privilege. */ + if (static_branch_unlikely(&kvm_protected_mode_initialized)) + return -EPERM; + + if (!pfn_range_is_valid(pfn, nr_pages)) + return -EINVAL; + + host_lock_component(); + for (offset =3D 0; offset < size; offset +=3D PAGE_SIZE) { + if (addr_is_memory(phys + offset)) { + ret =3D -EINVAL; + goto unlock; + } + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, phys + offset, &pte, NULL); + if (ret) + goto unlock; + if (pte && !kvm_pte_valid(pte)) { + ret =3D -EPERM; + goto unlock; + } + } + + ret =3D __pkvm_create_private_mapping(phys, size, PAGE_HYP_DEVICE, &addr); + if (ret) + goto unlock; + + /* + * Mark the range as owned by the hypervisor in the host stage-2. + * The annotation makes host accesses fault, and is retained by + * host_stage2_unmap_dev_all() + */ + ret =3D host_stage2_try(kvm_pgtable_stage2_annotate, &host_mmu.pgt, + phys, size, &host_s2_pool, + KVM_HOST_INVALID_PTE_TYPE_DONATION, + FIELD_PREP(KVM_HOST_DONATION_PTE_OWNER_MASK, PKVM_ID_HYP)); + if (ret) { + hyp_lock_component(); + WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, addr, size) !=3D size); + hyp_unlock_component(); + } else { + *haddr =3D addr; + } +unlock: + host_unlock_component(); + return ret; +} + +int __pkvm_hyp_donate_host_mmio(u64 pfn, u64 nr_pages, unsigned long haddr) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + u64 size =3D PAGE_SIZE * nr_pages; + kvm_pte_t pte; + u64 offset; + int ret =3D 0; + + if (static_branch_unlikely(&kvm_protected_mode_initialized)) + return -EPERM; + + if (!pfn_range_is_valid(pfn, nr_pages)) + return -EINVAL; + + host_lock_component(); + hyp_lock_component(); + for (offset =3D 0; offset < size; offset +=3D PAGE_SIZE) { + if (addr_is_memory(phys + offset)) { + ret =3D -EINVAL; + goto unlock; + } + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, phys + offset, &pte, NULL); + if (ret) + goto unlock; + if (!host_stage2_pte_is_hyp_owned(pte)) { + ret =3D -EPERM; + goto unlock; + } + } + WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, haddr, size) !=3D size); + WARN_ON(host_stage2_idmap_locked(phys, size, PKVM_HOST_MMIO_PROT)); +unlock: + hyp_unlock_component(); + host_unlock_component(); + return ret; +} + int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) { u64 phys =3D hyp_pfn_to_phys(pfn); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index b74dd5ce1efd..58b74450c824 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1161,13 +1161,9 @@ static int stage2_unmap_walker(const struct kvm_pgta= ble_visit_ctx *ctx, kvm_pte_t *childp =3D NULL; bool need_flush =3D false; =20 - if (!kvm_pte_valid(ctx->old)) { - if (stage2_pte_is_counted(ctx->old)) { - kvm_clear_pte(ctx->ptep); - mm_ops->put_page(ctx->ptep); - } + /* Invalid PTEs are left in place, including annotations. */ + if (!kvm_pte_valid(ctx->old)) return 0; - } =20 if (kvm_pte_table(ctx->old, ctx->level)) { childp =3D kvm_pte_follow(ctx->old, mm_ops); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F08AF475350 for ; Thu, 15 Oct 2026 20:09:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094958; cv=none; b=DJvw49RTmocLcY32aiqnuW9kEwOEkDVGNsUfqrtK75RwNKZPDTT+oiGw1Fz6nskGcj5p6d/PRcEGqdxm56yqzSwjRs5zvOvc5FBdmyARABeDmADO67M5qEkDtL6EWrZ73KN6G98FfaDWKNMnF5BGaLgnnkNzaBVVYNyP0wGsWUU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094958; c=relaxed/simple; bh=S/OdhPySBRbmZocD6BWu21RnoGNGWr4Ep2msGIau+qY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VWPBg4/YLqm7SnuTL3PskZgWPwwR4IS48g3CeIYVHarjqgQXBRSBeuaswuHE7dFiFifdvYSFYnl6uwtfOxXys5uUtO7RMe4CauGY7YUreC97HVamBS1CeXv213WgQIqTcNrBv91P+S4Il1pf4MjMt46o4nhiSY+4BBHMNhCs6mg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Y+tvPNbk; arc=none smtp.client-ip=209.85.218.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Y+tvPNbk" Received: by mail-ej1-f72.google.com with SMTP id a640c23a62f3a-c2e2da5823aso321220866b.0 for ; Thu, 15 Oct 2026 13:09:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094954; x=1792699754; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=M5ixsAbdw/m1P57/+3HLK7jfThlRJ+tmwZshst71JoE=; b=Y+tvPNbkVlyJneFdop25sV/6Cc/8fC5jwwBfChBP0gwymk8V8umar8VkOZNf6A2P0t 0YpXBtmS/i1IgGdkRcGZoW0uiHy99bK1Z2ESe+n4AfYioa3xC6OIqaJJS9y8xefhihtO 5U5VgLGYYMj6TduK1qONgiTupeM8PmpYDwra+IXTz5n/nL5zG89ZkWeyFs4YhC/hWho2 xfzfKyze4dE737bgpuIvXmD7Mno4e4Gvdy4DBhvvI6NSNxM/AM5D1q+sFARMHb0Meo/g KZa1aSCbx/FxjbFbpz5YsmkecP+8Ys2lnJQM8tnBIDnbczMHOTYPmvJYCimnsfIkadEA qNZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094954; x=1792699754; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=M5ixsAbdw/m1P57/+3HLK7jfThlRJ+tmwZshst71JoE=; b=jAZAoJ1xH74yOQhmdA9rbF8Exry8EY/EA10c5gUuOXzNo6migWkzyUVwD8cEHiXE+X kryiSEr4ps9YG6oC+Y5agrrqjmFNpCXYvAlNk0sx+PTtPgsPXAWkMDoLF4Ce2LikW16k QIZ67GknvifuiiCzOYrn7D1eZZf3uLLhIdpmztsEHn/wBKJMOsWXY8QDnfQJDoPFPZy6 ysh6oGAzTP3ya4W4UfZJM+zUFKrUfdMgK99TPyODj6Q0jZuyqBvcXT4XUnVTWiNNN5bf JXy7PdwY+pkYun9UHLdsHxlVF7i84lj0h2+Ska853tj8F+zQxGte90cQPLyZUCDxZo9b LW3g== X-Forwarded-Encrypted: i=1; AKwUvBy78WVYIGOcR+z/IiNQaKCtI/R25P4iExE9A2aId5zZ7e1/T6Jkl0Y4H+ZZki/Hkg/uPtBV9xNtFhbyzxQ=@vger.kernel.org X-Gm-Message-State: AFq9FYJpQVz0Efw4JqQCJY3nHzLSQ9gTwgKrXOpxYBzQVbmSHtxuAbVI rGWrSi15Fo7+0eI3RQ5QM6YcqwzHUn+mb+j+NNNfxHpxoREJDrNEiYI/LN2KyKsHkk/eOY8u3Tu bkLqOeFE0p+qQzA== X-Received: from ejiq6.prod.google.com ([2002:a17:906:e46:b0:c37:a171:8eda]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:72cd:b0:c40:33d:f06f with SMTP id a640c23a62f3a-c40033df381mr136086766b.39.1792094953037; Thu, 15 Oct 2026 13:09:13 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:33 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-3-smostafa@google.com> Subject: [PATCH v9 02/25] iommu/arm-smmu-v3: Move Queue and STE functions to header From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The KVM SMMUv3 driver would re-use some of the CMDQ and STE code inside the hypervisor. Move those functions to the header file that would be shared between the host kernel and the hypervisor. Signed-off-by: Mostafa Saleh Reviewed-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 62 -------------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 63 +++++++++++++++++++++ 2 files changed, 63 insertions(+), 62 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 34e916ea339f..101ece0e38d3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -140,33 +140,6 @@ static void parse_driver_options(struct arm_smmu_devic= e *smmu) } =20 /* Low-level queue manipulation functions */ -static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) -{ - u32 space, prod, cons; - - prod =3D Q_IDX(q, q->prod); - cons =3D Q_IDX(q, q->cons); - - if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; - - return space >=3D n; -} - -static bool queue_full(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); -} - -static bool queue_empty(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); -} - static bool queue_consumed(struct arm_smmu_ll_queue *q, u32 prod) { return ((Q_WRP(q, q->cons) =3D=3D Q_WRP(q, prod)) && @@ -185,12 +158,6 @@ static void queue_sync_cons_out(struct arm_smmu_queue = *q) writel_relaxed(q->llq.cons, q->cons_reg); } =20 -static void queue_inc_cons(struct arm_smmu_ll_queue *q) -{ - u32 cons =3D Q_POS(q, q->cons) + 1; - q->cons =3D Q_OVF(q->cons) | Q_POS(q, cons); -} - static void queue_sync_cons_ovf(struct arm_smmu_queue *q) { struct arm_smmu_ll_queue *llq =3D &q->llq; @@ -221,12 +188,6 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q) return ret; } =20 -static u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) -{ - u32 prod =3D Q_POS(q, q->prod) + n; - return Q_OVF(q->prod) | Q_POS(q, prod); -} - static void queue_poll_init(struct arm_smmu_device *smmu, struct arm_smmu_queue_poll *qp) { @@ -254,14 +215,6 @@ static int queue_poll(struct arm_smmu_queue_poll *qp) return 0; } =20 -static void queue_write(__le64 *dst, u64 *src, size_t n_dwords) -{ - int i; - - for (i =3D 0; i < n_dwords; ++i) - *dst++ =3D cpu_to_le64(*src++); -} - static void queue_read(u64 *dst, __le64 *src, size_t n_dwords) { int i; @@ -1817,21 +1770,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_= master *master) } =20 /* Stream table manipulation functions */ -static void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst, - dma_addr_t l2ptr_dma) -{ - u64 val =3D 0; - - val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); - val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - - /* Ensure the new L2 table is fully visible. */ - dma_wmb(); - - /* The HW has 64 bit atomicity with stores to the L2 STE table */ - WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); -} - struct arm_smmu_ste_writer { struct arm_smmu_entry_writer writer; u32 sid; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index bb048e08d613..8c8957831286 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1284,6 +1284,69 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_devi= ce *smmu, bool sync); bool arm_smmu_erratum_repeat_tlbi_cfgi(void); =20 +/* Queue and STE functions shared between kernel and hyp. */ +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + u32 space, prod, cons; + + prod =3D Q_IDX(q, q->prod); + cons =3D Q_IDX(q, q->cons); + + if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) + space =3D (1 << q->max_n_shift) - (prod - cons); + else + space =3D cons - prod; + + return space >=3D n; +} + +static inline bool queue_full(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); +} + +static inline bool queue_empty(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); +} + +static inline u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) +{ + u32 prod =3D Q_POS(q, q->prod) + n; + return Q_OVF(q->prod) | Q_POS(q, prod); +} + +static inline void queue_inc_cons(struct arm_smmu_ll_queue *q) +{ + u32 cons =3D Q_POS(q, q->cons) + 1; + q->cons =3D Q_OVF(q->cons) | Q_POS(q, cons); +} + +static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords) +{ + int i; + + for (i =3D 0; i < n_dwords; ++i) + *dst++ =3D cpu_to_le64(*src++); +} + +static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1= *dst, + dma_addr_t l2ptr_dma) +{ + u64 val =3D 0; + + val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); + val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; + + /* Ensure the new L2 table is fully visible. */ + dma_wmb(); + + /* The HW has 64 bit atomicity with stores to the L2 STE table */ + WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); +} + #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); void arm_smmu_sva_notifier_synchronize(void); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F85C481FD6 for ; Thu, 15 Oct 2026 20:09:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094961; cv=none; b=iFYaOB/MsONACHKVaXrO1DUidSGyvE/NhV/eq9xgooe7D11WTANs7oLJkKHYumzojh7vPzux7aLUzx+y6bUyixl+cUFZk56+xWMYkmavwOZvzxuNsbpjaE9PdKlf+3qT7+maqQ3+BtU4/GCaO34n3kxBHTyF80KMS3NESLNlndk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094961; c=relaxed/simple; bh=r63L3owY7mfJ3h4aniiVL9LvB+bNIP+0KJVZO/Aww5U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JmwhcgJvcHFZKg2l4yxJKHf5B/4xXoZRTjckBd8iT7l8C0/899z/ivBQgtjKlYXhD5/mAMZpv0eIUHLFMQ7A0XMbynhNv8CIg79pK6qHA1Tnpnl6xLT/ff4APyN89aXW9v2S0Vunf8bK31yiomdbtvKYciHEWa4PA/mxyAvIUKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EtU2wX54; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EtU2wX54" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4a17741243eso15526785e9.0 for ; Thu, 15 Oct 2026 13:09:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094954; x=1792699754; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=fvOl/hxvEkk8z+uVJIZLKJKVTrO3KToYAMaC4rC2vNA=; b=EtU2wX540PnXUXVZl2+EUJYn0HS5ButN1LqF9bkQvEcKMwS9cbFe8S/Uhqou3weXiy tjGU2K6SCzbsvVzQ501w7YqbvDLSNkbWlnDzH9GZdkK0YyML84Dfrq/TKDafgxb3dM0H aXhmsmTZdT7O0fivjkcJnaxtSPHVeStwhW5w/cbbh85g0ih109xUzxgm5xZ7xq5LhsRX kcu4/cYbppx2PQgkDyuLytBzSaoDH2TQoqiQrLmujZfLWAobjFes4+7pgCvq9Gxx2WBJ qk7Eoaw3WjwwdqAJIDdkEzbV782yZe8nsCOY7Cdvp3K6g4yvrKKTY4WPhV61cgQ1Unnz ZBTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094954; x=1792699754; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=fvOl/hxvEkk8z+uVJIZLKJKVTrO3KToYAMaC4rC2vNA=; b=AvKkILZK5mUKmYd5BiacPpPVMieydI4zXaZFjWQtRsN+QTp/kIoeukJ2PPiat6phG+ bAWEIeOjTOdFI7C181UEauZpKkMmoMlaLMBcmSOgH0yrXDGq7JaW+IpA7KO/P3HUwdcU RaIGnH43m41LwyT56eM0N4HrIdTKu49Ykm5sj88cU18GqUUPJ7EuPjrdL3z/dW4Dcz9J gvEBr/sMf78G74+kFnBmGOdlh9JCwA613LaP1MLHiqmd/siAwy9k3OhL2J2BYVcRpN+C IXAiW6XlkIEundmk3CI77f19iJrEDVHR7hCdmfzPzjS63CgsfnAA07q3hicdYfM9o2J+ 3qVw== X-Forwarded-Encrypted: i=1; AKwUvBzdoqGqF2RCXWEPRhzYpyY0LBIR+n79kwIr2LjqsllNkTgCzOhM4lIes4nWG7DKJ3pNOaUNeH8VFukGOis=@vger.kernel.org X-Gm-Message-State: AFq9FYL7YvcZvCNhEGsyKl0ubLYEH5hm5l53FdQy+8flOOTkqa/8ksvp OmOaTSkGQZBpYUMRCO0H4Tp8RjUQXocfYtSmK01fG8Ini2G64nJ398K9iRgRNIzlZQ/J5L+0TXd Znk5TdY/0AftQlQ== X-Received: from wmbg12.prod.google.com ([2002:a05:600c:a40c:b0:49f:fd3d:3bd3]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e89:b0:4a0:12d7:4c88 with SMTP id 5b1f17b1804b1-4a18e4ac77amr453376685e9.30.1792094954389; Thu, 15 Oct 2026 13:09:14 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:34 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-4-smostafa@google.com> Subject: [PATCH v9 03/25] iommu/arm-smmu-v3: Move IDR parsing to common functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh , Jason Gunthorpe Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The hypervisor SMMUv3 driver needs the same IDR0, IDR3, IDR5 and IIDR parsing as the kernel driver, plus the IDR1 checks it depends on (embedded implementation, ATTR_TYPES_OVR, StreamID size). Move them to arm-smmu-v3-common-lib.c, which will be built for both, and the IIDR implementer and product IDs to arm-smmu-v3.h. The two drivers use different device structs, so the shared code is written against ARM_SMMU_OBJ, which each compilation unit defines. The kernel maps it to struct arm_smmu_device. Static asserts check that the fields the shared code uses exist with the expected types. The validity checks stay with the parsing. dev_err() and dev_info() become arm_smmu_err() and arm_smmu_info(), defined per world next to ARM_SMMU_OBJ, as the hypervisor cannot print to the console. The disable_msipolling module parameter is passed as an argument so the hypervisor can opt out, and cpus_have_cap() becomes cpus_have_final_cap(), which the hypervisor can use. No functional change intended. Suggested-by: Jason Gunthorpe Signed-off-by: Mostafa Saleh Reviewed-by: Nicolin Chen --- drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 240 ++++++++++++++++++ .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.h | 39 +++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 220 +--------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 + 5 files changed, 297 insertions(+), 210 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index 493a659cc66b..c9ce392e6d31 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ARM_SMMU_V3) +=3D arm_smmu_v3.o -arm_smmu_v3-y :=3D arm-smmu-v3.o +arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c new file mode 100644 index 000000000000..22467aaca191 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -0,0 +1,240 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2015 ARM Limited + * Arm SMMUv3 driver functions shared with hypervisor. + */ + +#include + +#include "arm-smmu-v3.h" +#include "arm-smmu-v3-common-lib.h" + +void arm_smmu_device_iidr_probe(ARM_SMMU_OBJ *smmu) +{ + u32 reg; + unsigned int implementer, productid, variant, revision; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IIDR); + implementer =3D FIELD_GET(IIDR_IMPLEMENTER, reg); + productid =3D FIELD_GET(IIDR_PRODUCTID, reg); + variant =3D FIELD_GET(IIDR_VARIANT, reg); + revision =3D FIELD_GET(IIDR_REVISION, reg); + + switch (implementer) { + case IIDR_IMPLEMENTER_ARM: + switch (productid) { + case IIDR_PRODUCTID_ARM_MMU_600: + /* Arm erratum 1076982 */ + if (variant =3D=3D 0 && revision <=3D 2) + smmu->features &=3D ~ARM_SMMU_FEAT_SEV; + /* Arm erratum 1209401 */ + if (variant < 2) + smmu->features &=3D ~ARM_SMMU_FEAT_NESTING; + break; + case IIDR_PRODUCTID_ARM_MMU_700: + /* Many errata... */ + smmu->features &=3D ~ARM_SMMU_FEAT_BTM; + if (variant < 1 || revision < 1) { + /* Arm erratum 2812531 */ + smmu->options |=3D ARM_SMMU_OPT_CMDQ_FORCE_SYNC; + /* Arm errata 2268618, 2812531 */ + smmu->features &=3D ~ARM_SMMU_FEAT_NESTING; + } + /* Arm errata 3777127 */ + smmu->options |=3D ARM_SMMU_OPT_FULL_CONT_RANGE_INV; + break; + case IIDR_PRODUCTID_ARM_MMU_L1: + /* Arm errata 3878312 */ + smmu->features &=3D ~ARM_SMMU_FEAT_BTM; + break; + case IIDR_PRODUCTID_ARM_MMU_S3: + /* Arm errata 3995052 */ + smmu->features &=3D ~ARM_SMMU_FEAT_BTM; + /* Arm errata 3673557 */ + if (variant < 1 || (variant =3D=3D 1 && revision < 1)) + smmu->options |=3D + ARM_SMMU_OPT_FULL_CONT_RANGE_INV; + break; + } + break; + } +} + +int arm_smmu_device_idr0_probe(ARM_SMMU_OBJ *smmu, bool disable_msipolling, + u32 *idr0) +{ + bool coherent =3D smmu->features & ARM_SMMU_FEAT_COHERENCY; + u32 reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); + + /* 2-level structures */ + if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) + smmu->features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; + + if (reg & IDR0_CD2L) + smmu->features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; + + /* + * Translation table endianness. + * We currently require the same endianness as the CPU, but this + * could be changed later by adding a new IO_PGTABLE_QUIRK. + */ + switch (FIELD_GET(IDR0_TTENDIAN, reg)) { + case IDR0_TTENDIAN_MIXED: + smmu->features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; + break; +#ifdef __BIG_ENDIAN + case IDR0_TTENDIAN_BE: + smmu->features |=3D ARM_SMMU_FEAT_TT_BE; + break; +#else + case IDR0_TTENDIAN_LE: + smmu->features |=3D ARM_SMMU_FEAT_TT_LE; + break; +#endif + default: + arm_smmu_err(smmu, "unknown/unsupported TT endianness!\n"); + return -ENXIO; + } + + /* Boolean feature flags */ + if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) + smmu->features |=3D ARM_SMMU_FEAT_PRI; + + if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) + smmu->features |=3D ARM_SMMU_FEAT_ATS; + + if (reg & IDR0_SEV) + smmu->features |=3D ARM_SMMU_FEAT_SEV; + + if (reg & IDR0_MSI) { + smmu->features |=3D ARM_SMMU_FEAT_MSI; + if (coherent && !disable_msipolling) + smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; + } + + if (reg & IDR0_HYP) { + smmu->features |=3D ARM_SMMU_FEAT_HYP; + if (cpus_have_final_cap(ARM64_HAS_VIRT_HOST_EXTN)) + smmu->features |=3D ARM_SMMU_FEAT_E2H; + } + + switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { + case IDR0_STALL_MODEL_FORCE: + smmu->features |=3D ARM_SMMU_FEAT_STALL_FORCE; + fallthrough; + case IDR0_STALL_MODEL_STALL: + smmu->features |=3D ARM_SMMU_FEAT_STALLS; + } + + if (reg & IDR0_S1P) + smmu->features |=3D ARM_SMMU_FEAT_TRANS_S1; + + if (reg & IDR0_S2P) + smmu->features |=3D ARM_SMMU_FEAT_TRANS_S2; + + if (!(reg & (IDR0_S1P | IDR0_S2P))) { + arm_smmu_err(smmu, "no translation support!\n"); + return -ENXIO; + } + + /* We only support the AArch64 table format at present */ + if (!(FIELD_GET(IDR0_TTF, reg) & IDR0_TTF_AARCH64)) { + arm_smmu_err(smmu, "AArch64 table format not supported!\n"); + return -ENXIO; + } + + if ((smmu->features & ARM_SMMU_FEAT_TRANS_S1) && + (smmu->features & ARM_SMMU_FEAT_TRANS_S2)) + smmu->features |=3D ARM_SMMU_FEAT_NESTING; + + *idr0 =3D reg; + return 0; +} + +int arm_smmu_device_idr1_probe(ARM_SMMU_OBJ *smmu, u32 *idr1) +{ + u32 reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); + + if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) { + arm_smmu_err(smmu, "embedded implementation not supported\n"); + return -ENXIO; + } + + if (reg & IDR1_ATTR_TYPES_OVR) + smmu->features |=3D ARM_SMMU_FEAT_ATTR_TYPES_OVR; + + smmu->sid_bits =3D FIELD_GET(IDR1_SIDSIZE, reg); + + /* + * If the SMMU supports fewer bits than would fill a single L2 stream + * table, use a linear table instead. + */ + if (smmu->sid_bits <=3D STRTAB_SPLIT) + smmu->features &=3D ~ARM_SMMU_FEAT_2_LVL_STRTAB; + + *idr1 =3D reg; + return 0; +} + +void arm_smmu_device_idr3_probe(ARM_SMMU_OBJ *smmu) +{ + u32 reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); + + if (FIELD_GET(IDR3_RIL, reg)) + smmu->features |=3D ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_FWB, reg)) + smmu->features |=3D ARM_SMMU_FEAT_S2FWB; + if (FIELD_GET(IDR3_BBM, reg) =3D=3D 2) + smmu->features |=3D ARM_SMMU_FEAT_BBML2; +} + +u32 arm_smmu_device_idr5_probe(ARM_SMMU_OBJ *smmu) +{ + u32 reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); + + if (reg & IDR5_DS) + smmu->features |=3D ARM_SMMU_FEAT_DS; + + /* Page sizes */ + if (reg & IDR5_GRAN64K) + smmu->pgsize_bitmap |=3D SZ_64K | SZ_512M; + if (reg & IDR5_GRAN16K) + smmu->pgsize_bitmap |=3D SZ_16K | SZ_32M; + if (reg & IDR5_GRAN4K) + smmu->pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + + /* Input address size */ + if (FIELD_GET(IDR5_VAX, reg) =3D=3D IDR5_VAX_52_BIT) + smmu->features |=3D ARM_SMMU_FEAT_VAX; + + /* Output address size */ + switch (FIELD_GET(IDR5_OAS, reg)) { + case IDR5_OAS_32_BIT: + smmu->oas =3D 32; + break; + case IDR5_OAS_36_BIT: + smmu->oas =3D 36; + break; + case IDR5_OAS_40_BIT: + smmu->oas =3D 40; + break; + case IDR5_OAS_42_BIT: + smmu->oas =3D 42; + break; + case IDR5_OAS_44_BIT: + smmu->oas =3D 44; + break; + case IDR5_OAS_52_BIT: + smmu->oas =3D 52; + smmu->pgsize_bitmap |=3D 1ULL << 42; /* 4TB */ + break; + default: + arm_smmu_info(smmu, + "unknown output address size. Truncating to 48-bit\n"); + fallthrough; + case IDR5_OAS_48_BIT: + smmu->oas =3D 48; + } + + return reg; +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h new file mode 100644 index 000000000000..926a84b11a82 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h @@ -0,0 +1,39 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __ARM_SMMU_V3_COMMON_LIB_H +#define __ARM_SMMU_V3_COMMON_LIB_H + +#include +#include +#include + +/* + * The IDR probe functions are shared by the kernel and the hypervisor dri= vers, + * which define ARM_SMMU_OBJ, arm_smmu_err() and arm_smmu_info() different= ly. + * Ensure the fields used by them are defined and have the correct types. + */ +#ifndef __KVM_NVHE_HYPERVISOR__ +#include + +#include "arm-smmu-v3.h" + +typedef struct arm_smmu_device ARM_SMMU_OBJ; + +#define arm_smmu_err(smmu, fmt, ...) dev_err((smmu)->dev, fmt, ##__VA_ARGS= __) +#define arm_smmu_info(smmu, fmt, ...) dev_info((smmu)->dev, fmt, ##__VA_AR= GS__) +#endif + +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, features), u32)); +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, options), u32)); +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, oas), unsigned long)= ); +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, pgsize_bitmap), unsi= gned long)); +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, base), void __iomem = *)); +static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, sid_bits), unsigned = int)); + +void arm_smmu_device_iidr_probe(ARM_SMMU_OBJ *smmu); +int arm_smmu_device_idr0_probe(ARM_SMMU_OBJ *smmu, bool disable_msipolling, + u32 *idr0); +int arm_smmu_device_idr1_probe(ARM_SMMU_OBJ *smmu, u32 *idr1); +void arm_smmu_device_idr3_probe(ARM_SMMU_OBJ *smmu); +u32 arm_smmu_device_idr5_probe(ARM_SMMU_OBJ *smmu); + +#endif /* __ARM_SMMU_V3_COMMON_LIB_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 101ece0e38d3..8cbf1e7354e2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -33,6 +33,7 @@ #include =20 #include "arm-smmu-v3.h" +#include "arm-smmu-v3-common-lib.h" #include "../../dma-iommu.h" =20 static bool disable_msipolling; @@ -5168,63 +5169,6 @@ static int arm_smmu_device_reset(struct arm_smmu_dev= ice *smmu) return 0; } =20 -#define IIDR_IMPLEMENTER_ARM 0x43b -#define IIDR_PRODUCTID_ARM_MMU_600 0x483 -#define IIDR_PRODUCTID_ARM_MMU_700 0x487 -#define IIDR_PRODUCTID_ARM_MMU_L1 0x48a -#define IIDR_PRODUCTID_ARM_MMU_S3 0x498 - -static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu) -{ - u32 reg; - unsigned int implementer, productid, variant, revision; - - reg =3D readl_relaxed(smmu->base + ARM_SMMU_IIDR); - implementer =3D FIELD_GET(IIDR_IMPLEMENTER, reg); - productid =3D FIELD_GET(IIDR_PRODUCTID, reg); - variant =3D FIELD_GET(IIDR_VARIANT, reg); - revision =3D FIELD_GET(IIDR_REVISION, reg); - - switch (implementer) { - case IIDR_IMPLEMENTER_ARM: - switch (productid) { - case IIDR_PRODUCTID_ARM_MMU_600: - /* Arm erratum 1076982 */ - if (variant =3D=3D 0 && revision <=3D 2) - smmu->features &=3D ~ARM_SMMU_FEAT_SEV; - /* Arm erratum 1209401 */ - if (variant < 2) - smmu->features &=3D ~ARM_SMMU_FEAT_NESTING; - break; - case IIDR_PRODUCTID_ARM_MMU_700: - /* Many errata... */ - smmu->features &=3D ~ARM_SMMU_FEAT_BTM; - if (variant < 1 || revision < 1) { - /* Arm erratum 2812531 */ - smmu->options |=3D ARM_SMMU_OPT_CMDQ_FORCE_SYNC; - /* Arm errata 2268618, 2812531 */ - smmu->features &=3D ~ARM_SMMU_FEAT_NESTING; - } - /* Arm errata 3777127 */ - smmu->options |=3D ARM_SMMU_OPT_FULL_CONT_RANGE_INV; - break; - case IIDR_PRODUCTID_ARM_MMU_L1: - /* Arm errata 3878312 */ - smmu->features &=3D ~ARM_SMMU_FEAT_BTM; - break; - case IIDR_PRODUCTID_ARM_MMU_S3: - /* Arm errata 3995052 */ - smmu->features &=3D ~ARM_SMMU_FEAT_BTM; - /* Arm errata 3673557 */ - if (variant < 1 || (variant =3D=3D 1 && revision < 1)) - smmu->options |=3D - ARM_SMMU_OPT_FULL_CONT_RANGE_INV; - break; - } - break; - } -} - static void arm_smmu_get_httu(struct arm_smmu_device *smmu, u32 reg) { u32 fw_features =3D smmu->features & (ARM_SMMU_FEAT_HA | ARM_SMMU_FEAT_HD= | @@ -5255,61 +5199,12 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) { u32 reg; bool coherent =3D smmu->features & ARM_SMMU_FEAT_COHERENCY; + int ret; =20 /* IDR0 */ - reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); - - /* 2-level structures */ - if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; - - if (reg & IDR0_CD2L) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; - - /* - * Translation table endianness. - * We currently require the same endianness as the CPU, but this - * could be changed later by adding a new IO_PGTABLE_QUIRK. - */ - switch (FIELD_GET(IDR0_TTENDIAN, reg)) { - case IDR0_TTENDIAN_MIXED: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; - break; -#ifdef __BIG_ENDIAN - case IDR0_TTENDIAN_BE: - smmu->features |=3D ARM_SMMU_FEAT_TT_BE; - break; -#else - case IDR0_TTENDIAN_LE: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE; - break; -#endif - default: - dev_err(smmu->dev, "unknown/unsupported TT endianness!\n"); - return -ENXIO; - } - - /* Boolean feature flags */ - if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) - smmu->features |=3D ARM_SMMU_FEAT_PRI; - - if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) - smmu->features |=3D ARM_SMMU_FEAT_ATS; - - if (reg & IDR0_SEV) - smmu->features |=3D ARM_SMMU_FEAT_SEV; - - if (reg & IDR0_MSI) { - smmu->features |=3D ARM_SMMU_FEAT_MSI; - if (coherent && !disable_msipolling) - smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; - } - - if (reg & IDR0_HYP) { - smmu->features |=3D ARM_SMMU_FEAT_HYP; - if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) - smmu->features |=3D ARM_SMMU_FEAT_E2H; - } + ret =3D arm_smmu_device_idr0_probe(smmu, disable_msipolling, ®); + if (ret) + return ret; =20 arm_smmu_get_httu(smmu, reg); =20 @@ -5321,44 +5216,14 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n", str_true_false(coherent)); =20 - switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { - case IDR0_STALL_MODEL_FORCE: - smmu->features |=3D ARM_SMMU_FEAT_STALL_FORCE; - fallthrough; - case IDR0_STALL_MODEL_STALL: - smmu->features |=3D ARM_SMMU_FEAT_STALLS; - } - - if (reg & IDR0_S1P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S1; - - if (reg & IDR0_S2P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S2; - - if (!(reg & (IDR0_S1P | IDR0_S2P))) { - dev_err(smmu->dev, "no translation support!\n"); - return -ENXIO; - } - - /* We only support the AArch64 table format at present */ - if (!(FIELD_GET(IDR0_TTF, reg) & IDR0_TTF_AARCH64)) { - dev_err(smmu->dev, "AArch64 table format not supported!\n"); - return -ENXIO; - } - /* ASID/VMID sizes */ smmu->asid_bits =3D reg & IDR0_ASID16 ? 16 : 8; smmu->vmid_bits =3D reg & IDR0_VMID16 ? 16 : 8; =20 /* IDR1 */ - reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); - if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) { - dev_err(smmu->dev, "embedded implementation not supported\n"); - return -ENXIO; - } - - if (reg & IDR1_ATTR_TYPES_OVR) - smmu->features |=3D ARM_SMMU_FEAT_ATTR_TYPES_OVR; + ret =3D arm_smmu_device_idr1_probe(smmu, ®); + if (ret) + return ret; =20 if ((smmu->options & ARM_SMMU_OPT_OVR_INSTCFG_DATA) && !(reg & IDR1_ATTR_PERMS_OVR)) { @@ -5385,87 +5250,24 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) smmu->priq.q.llq.max_n_shift =3D arm_smmu_priq_max_n_shift(FIELD_GET(IDR1_PRIQS, reg)); =20 - /* SID/SSID sizes */ + /* SSID size */ smmu->ssid_bits =3D FIELD_GET(IDR1_SSIDSIZE, reg); - smmu->sid_bits =3D FIELD_GET(IDR1_SIDSIZE, reg); smmu->iommu.max_pasids =3D 1UL << smmu->ssid_bits; =20 - /* - * If the SMMU supports fewer bits than would fill a single L2 stream - * table, use a linear table instead. - */ - if (smmu->sid_bits <=3D STRTAB_SPLIT) - smmu->features &=3D ~ARM_SMMU_FEAT_2_LVL_STRTAB; - /* IDR3 */ - reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); - if (FIELD_GET(IDR3_RIL, reg)) - smmu->features |=3D ARM_SMMU_FEAT_RANGE_INV; - if (FIELD_GET(IDR3_FWB, reg)) - smmu->features |=3D ARM_SMMU_FEAT_S2FWB; - - if (FIELD_GET(IDR3_BBM, reg) =3D=3D 2) - smmu->features |=3D ARM_SMMU_FEAT_BBML2; + arm_smmu_device_idr3_probe(smmu); =20 /* IDR5 */ - reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); + reg =3D arm_smmu_device_idr5_probe(smmu); =20 /* Maximum number of outstanding stalls */ smmu->evtq.max_stalls =3D FIELD_GET(IDR5_STALL_MAX, reg); =20 - if (reg & IDR5_DS) - smmu->features |=3D ARM_SMMU_FEAT_DS; - - /* Page sizes */ - if (reg & IDR5_GRAN64K) - smmu->pgsize_bitmap |=3D SZ_64K | SZ_512M; - if (reg & IDR5_GRAN16K) - smmu->pgsize_bitmap |=3D SZ_16K | SZ_32M; - if (reg & IDR5_GRAN4K) - smmu->pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; - - /* Input address size */ - if (FIELD_GET(IDR5_VAX, reg) =3D=3D IDR5_VAX_52_BIT) - smmu->features |=3D ARM_SMMU_FEAT_VAX; - - /* Output address size */ - switch (FIELD_GET(IDR5_OAS, reg)) { - case IDR5_OAS_32_BIT: - smmu->oas =3D 32; - break; - case IDR5_OAS_36_BIT: - smmu->oas =3D 36; - break; - case IDR5_OAS_40_BIT: - smmu->oas =3D 40; - break; - case IDR5_OAS_42_BIT: - smmu->oas =3D 42; - break; - case IDR5_OAS_44_BIT: - smmu->oas =3D 44; - break; - case IDR5_OAS_52_BIT: - smmu->oas =3D 52; - smmu->pgsize_bitmap |=3D 1ULL << 42; /* 4TB */ - break; - default: - dev_info(smmu->dev, - "unknown output address size. Truncating to 48-bit\n"); - fallthrough; - case IDR5_OAS_48_BIT: - smmu->oas =3D 48; - } - /* Set the DMA mask for our table walker */ if (dma_set_mask_and_coherent(smmu->dev, DMA_BIT_MASK(smmu->oas))) dev_warn(smmu->dev, "failed to set DMA mask for table walker\n"); =20 - if ((smmu->features & ARM_SMMU_FEAT_TRANS_S1) && - (smmu->features & ARM_SMMU_FEAT_TRANS_S2)) - smmu->features |=3D ARM_SMMU_FEAT_NESTING; - arm_smmu_device_iidr_probe(smmu); =20 if (arm_smmu_sva_supported(smmu)) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 8c8957831286..bc4c25421835 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -87,6 +87,12 @@ struct arm_vsmmu; #define IIDR_REVISION GENMASK(15, 12) #define IIDR_IMPLEMENTER GENMASK(11, 0) =20 +#define IIDR_IMPLEMENTER_ARM 0x43b +#define IIDR_PRODUCTID_ARM_MMU_600 0x483 +#define IIDR_PRODUCTID_ARM_MMU_700 0x487 +#define IIDR_PRODUCTID_ARM_MMU_L1 0x48a +#define IIDR_PRODUCTID_ARM_MMU_S3 0x498 + #define ARM_SMMU_AIDR 0x1C =20 #define ARM_SMMU_CR0 0x20 --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C6B648C8DC for ; Thu, 15 Oct 2026 20:09:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094963; cv=none; b=DB3NUZI+DnImPyW5ZJB5jp3VF8pW4GUAHYsIog2kVdkHlKW4PWGvXQNPMIUODcBLP3aofxjnwHJzZkJdK6xvWXji1MD5liK2Spzx1fuwkWaU716qgF1sI8Wfnl9Gd30wEyasZDIw7DFeTohtaAguKjWH44ElAZkjdlybVWTmeXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094963; c=relaxed/simple; bh=fUgza7SLLFb5anuHt2xluUBMUoBTWGDhr1TBJyyaCbk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ICyLqJ4fFNbMRLKlzgdVI+QyDy4E0i4Ry6trjmpU4d+Bn1jLl3c6O3r+06khefKB6thTd8RvH3hx8FkwrQeZfi37b3fbV47Ipy8yF3vGDWocGnHwcQ5V9S26xB4iIoPj7FaLdS5Rg/6AS5kxc+GU6Wt1viRj3BHUEBWKsuTxkzE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OMO8uL5s; arc=none smtp.client-ip=209.85.218.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OMO8uL5s" Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-c29466736e3so689567466b.2 for ; Thu, 15 Oct 2026 13:09:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094956; x=1792699756; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=wic91XhpkzCA7vTTouSB42jSaYhrjWojUPYtPDmPpuk=; b=OMO8uL5sXyK0OKorJODq4w0OjsQA83pLwea3y7WaLBDLf1S3zLurNy/XicGk+DiwYX QohzBrUxYBcE1jHBjc7d+ereAcCp8EBd61IB47YIAOB/3F6a29ywP+X1j+XtThzTf8qk hH8Tq14P/45ASXs5pXlAQXiWTErYXwcrlvOuzUkRCgWsU5CTn++mNMIOmAFk4CRVXrOx BNC1cfkTwm2Z0f5DrwckQMFpsJHzpFVssqXQ2kG+p8iPkyGROZl2P9o2kM5GcAErz84M OVUDM534JeQ4BowQPN+767xwac5N5TFBlU2cCg3popjeq7aU4oPL3Y+rgKXi5zZXDKkM fzeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094956; x=1792699756; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wic91XhpkzCA7vTTouSB42jSaYhrjWojUPYtPDmPpuk=; b=Z1cmHYyWUvBCAA4MzLrkzyUu+vFEDn8rRIV+lCbbMl+vj6ul3V9HE7aQKz+Rw80GJ3 HpwvtHaCzCygQg89cHed58zaMeFOvFqv4GWqIK0Z2cnzNRh9w0wZaI47LgErC7s+Mbuh aGyBIroCHY7lwz0JgL1bpSMq6De7a8qLnuXUXurKY6Oer4jEbiPxbUNXMmsVPSJ6W9fj sZ6rOnmTaHQuu8ars1SYrLK27rDNFmAA9km/T8xCT62kCIV9o7RrPFfXV3j3RTQ5tJnN Wb/BC26LyMuBru5voH4Gev/vKxS31uTj4VPonaoOTt7o4v+1kWjPAukcu4C86R/GLzKm N07w== X-Forwarded-Encrypted: i=1; AKwUvBy8hOfDPH8VCx7/JGVL8RGhynWqiZWiIAK7jgG8i+G48/Gb6iuFPwoJ4x1jJgtRj9b3MDK1U5egMU7DPI0=@vger.kernel.org X-Gm-Message-State: AFq9FYITAVXpslxL9/QN6oWWpHG5OvRghoCmr15/5s8RU9KD5ASrpC4U /gHSm9QszQn+tYSqONoEimDXa9aqgGCnw0luQvGVt9p6pHzCdckjAUcVq1P5PpP7RMElKNA/U6B SgP9Qs6mKWnvJDw== X-Received: from ejcb24.prod.google.com ([2002:a17:906:36d8:b0:c3f:7f5a:30f9]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:906:c106:b0:c2d:bf61:24d1 with SMTP id a640c23a62f3a-c31a9a2919bmr1774150966b.1.1792094955429; Thu, 15 Oct 2026 13:09:15 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:35 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-5-smostafa@google.com> Subject: [PATCH v9 04/25] iommu/arm-smmu-v3: Move hitless machinery to common code From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the hitless STE functions to the common file so they can be reused by the hypervisor. No functional change. Signed-off-by: Mostafa Saleh Reviewed-by: Nicolin Chen --- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 246 +++++++++++++++++ .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.h | 6 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c | 2 +- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 249 ------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 +- 5 files changed, 254 insertions(+), 255 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c index 22467aaca191..cf9287cdbd19 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -5,6 +5,7 @@ */ =20 #include +#include =20 #include "arm-smmu-v3.h" #include "arm-smmu-v3-common-lib.h" @@ -238,3 +239,248 @@ u32 arm_smmu_device_idr5_probe(ARM_SMMU_OBJ *smmu) =20 return reg; } + +/* + * Based on the value of ent report which bits of the STE the HW will acce= ss. It + * would be nice if this was complete according to the spec, but minimally= it + * has to capture the bits this driver uses. + */ +void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) +{ + unsigned int cfg =3D FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); + + used_bits[0] =3D cpu_to_le64(STRTAB_STE_0_V); + if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V))) + return; + + used_bits[0] |=3D cpu_to_le64(STRTAB_STE_0_CFG); + + /* S1 translates */ + if (cfg & BIT(0)) { + used_bits[0] |=3D cpu_to_le64(STRTAB_STE_0_S1FMT | + STRTAB_STE_0_S1CTXPTR_MASK | + STRTAB_STE_0_S1CDMAX); + used_bits[1] |=3D + cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | + STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | + STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | + STRTAB_STE_1_EATS | STRTAB_STE_1_MEV | + STRTAB_STE_1_INSTCFG); + used_bits[2] |=3D cpu_to_le64(STRTAB_STE_2_S2VMID); + + /* + * See 13.5 Summary of attribute/permission configuration fields + * for the SHCFG behavior. + */ + if (FIELD_GET(STRTAB_STE_1_S1DSS, le64_to_cpu(ent[1])) =3D=3D + STRTAB_STE_1_S1DSS_BYPASS) + used_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_SHCFG); + } + + /* S2 translates */ + if (cfg & BIT(1)) { + used_bits[1] |=3D + cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS | + STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV | + STRTAB_STE_1_INSTCFG); + used_bits[2] |=3D + cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | + STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | + STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2S | + STRTAB_STE_2_S2R); + used_bits[3] |=3D cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); + } + + if (cfg =3D=3D STRTAB_STE_0_CFG_BYPASS) + used_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_SHCFG); +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_get_ste_used); + +void arm_smmu_get_ste_update_safe(const __le64 *cur, const __le64 *target, + __le64 *safe_bits) +{ + const u64 eats_s1chk =3D + FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_S1CHK); + const u64 eats_trans =3D + FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_TRANS); + + /* + * When an STE changes EATS_TRANS, the sequencing code in the attach + * logic already will have the PCI cap for ATS disabled. Thus at this + * moment we can expect that the device will not generate ATS queries + * and so we don't care about the sequencing of EATS. The purpose of + * EATS_TRANS is to protect the system from hostile untrusted devices + * that issue ATS when the PCI config space is disabled. However, if + * EATS_TRANS is being changed, then we must have already trusted the + * device as the EATS_TRANS security block is being disabled. + * + * Note: now the EATS_TRANS update is moved to the first entry_set(). + * Changing S2S and EATS might transiently result in S2S=3D1 and EATS=3D1 + * which is a bad STE (see "5.2 Stream Table Entry"). In such a case, + * we can't do a hitless update. Also, it should not be added to the + * safe bits with STRTAB_STE_1_EATS_S1CHK, because EATS=3D0b11 would be + * effectively an errant 0b00 configuration. + */ + if (!((cur[1] | target[1]) & cpu_to_le64(eats_s1chk)) && + !((cur[2] | target[2]) & cpu_to_le64(STRTAB_STE_2_S2S))) + safe_bits[1] |=3D cpu_to_le64(eats_trans); + + /* + * MEV does not meaningfully impact the operation of the HW, it only + * changes how many fault events are generated, thus we can relax it + * when computing the ordering. The spec notes the device can act like + * MEV=3D1 anyhow: + * + * Note: Software must expect, and be able to deal with, coalesced + * fault records even when MEV =3D=3D 0. + */ + safe_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_MEV); +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_get_ste_update_safe); + +/* + * Figure out if we can do a hitless update of entry to become target. Ret= urns a + * bit mask where 1 indicates that qword needs to be set disruptively. + * unused_update is an intermediate value of entry that has unused bits se= t to + * their new values. + */ +static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, + const __le64 *entry, const __le64 *target, + __le64 *unused_update) +{ + __le64 target_used[NUM_ENTRY_QWORDS] =3D {}; + __le64 cur_used[NUM_ENTRY_QWORDS] =3D {}; + __le64 safe[NUM_ENTRY_QWORDS] =3D {}; + u8 used_qword_diff =3D 0; + unsigned int i; + + writer->ops->get_used(entry, cur_used); + writer->ops->get_used(target, target_used); + if (writer->ops->get_update_safe) + writer->ops->get_update_safe(entry, target, safe); + + for (i =3D 0; i !=3D NUM_ENTRY_QWORDS; i++) { + /* + * Safe is only used for bits that are used by both entries, + * otherwise it is sequenced according to the unused entry. + */ + safe[i] &=3D target_used[i] & cur_used[i]; + + /* + * Check that masks are up to date, the make functions are not + * allowed to set a bit to 1 if the used function doesn't say it + * is used. + */ + WARN_ON_ONCE(target[i] & ~target_used[i]); + + /* Bits can change because they are not currently being used */ + cur_used[i] &=3D ~safe[i]; + unused_update[i] =3D (entry[i] & cur_used[i]) | + (target[i] & ~cur_used[i]); + /* + * Each bit indicates that a used bit in a qword needs to be + * changed after unused_update is applied. + */ + if ((unused_update[i] & target_used[i]) !=3D target[i]) + used_qword_diff |=3D 1 << i; + } + return used_qword_diff; +} + +static void entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, + const __le64 *target, unsigned int start, + unsigned int len) +{ + bool changed =3D false; + unsigned int i; + + for (i =3D start; len !=3D 0; len--, i++) { + if (entry[i] !=3D target[i]) { + WRITE_ONCE(entry[i], target[i]); + changed =3D true; + } + } + + if (changed) + writer->ops->sync(writer); +} + +/* + * Update the STE/CD to the target configuration. The transition from the + * current entry to the target entry takes place over multiple steps that + * attempts to make the transition hitless if possible. This function take= s care + * not to create a situation where the HW can perceive a corrupted entry. = HW is + * only required to have a 64 bit atomicity with stores from the CPU, while + * entries are many 64 bit values big. + * + * The difference between the current value and the target value is analyz= ed to + * determine which of three updates are required - disruptive, hitless or = no + * change. + * + * In the most general disruptive case we can make any update in three ste= ps: + * - Disrupting the entry (V=3D0) + * - Fill now unused qwords, execpt qword 0 which contains V + * - Make qword 0 have the final value and valid (V=3D1) with a single 64 + * bit store + * + * However this disrupts the HW while it is happening. There are several + * interesting cases where a STE/CD can be updated without disturbing the = HW + * because only a small number of bits are changing (S1DSS, CONFIG, etc) or + * because the used bits don't intersect. We can detect this by calculatin= g how + * many 64 bit values need update after adjusting the unused bits and skip= the + * V=3D0 process. This relies on the IGNORED behavior described in the + * specification. + */ +void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *en= try, + const __le64 *target) +{ + __le64 unused_update[NUM_ENTRY_QWORDS]; + u8 used_qword_diff; + + /* + * Many of the entry structures have pointers to other structures that + * need to have their updates be visible before any writes of the entry + * happen. + */ + dma_wmb(); + + used_qword_diff =3D + arm_smmu_entry_qword_diff(writer, entry, target, unused_update); + if (hweight8(used_qword_diff) =3D=3D 1) { + /* + * Only one qword needs its used bits to be changed. This is a + * hitless update, update all bits the current STE/CD is + * ignoring to their new values, then update a single "critical + * qword" to change the STE/CD and finally 0 out any bits that + * are now unused in the target configuration. + */ + unsigned int critical_qword_index =3D ffs(used_qword_diff) - 1; + + /* + * Skip writing unused bits in the critical qword since we'll be + * writing it in the next step anyways. This can save a sync + * when the only change is in that qword. + */ + unused_update[critical_qword_index] =3D + entry[critical_qword_index]; + entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS); + entry_set(writer, entry, target, critical_qword_index, 1); + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); + } else if (used_qword_diff) { + /* + * At least two qwords need their inuse bits to be changed. This + * requires a breaking update, zero the V bit, write all qwords + * but 0, then set qword 0 + */ + unused_update[0] =3D 0; + entry_set(writer, entry, unused_update, 0, 1); + entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1); + entry_set(writer, entry, target, 0, 1); + } else { + /* + * No inuse bit changed, though safe bits may have changed. + */ + entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); + } +} +EXPORT_SYMBOL_IF_KUNIT(arm_smmu_write_entry); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h index 926a84b11a82..55ffa880b093 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h @@ -36,4 +36,10 @@ int arm_smmu_device_idr1_probe(ARM_SMMU_OBJ *smmu, u32 *= idr1); void arm_smmu_device_idr3_probe(ARM_SMMU_OBJ *smmu); u32 arm_smmu_device_idr5_probe(ARM_SMMU_OBJ *smmu); =20 +void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits); +void arm_smmu_get_ste_update_safe(const __le64 *cur, const __le64 *target, + __le64 *safe_bits); +void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *cu= r, + const __le64 *target); + #endif /* __ARM_SMMU_V3_COMMON_LIB_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c b/drivers/iom= mu/arm/arm-smmu-v3/arm-smmu-v3-test.c index 5a4e45e15125..d9df9adc14ea 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c @@ -6,6 +6,7 @@ #include =20 #include "arm-smmu-v3.h" +#include "arm-smmu-v3-common-lib.h" =20 struct arm_smmu_test_writer { struct arm_smmu_entry_writer writer; @@ -18,7 +19,6 @@ struct arm_smmu_test_writer { unsigned int num_syncs; }; =20 -#define NUM_ENTRY_QWORDS 8 #define NUM_EXPECTED_SYNCS(x) x =20 static struct arm_smmu_ste bypass_ste; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 8cbf1e7354e2..5d7de893865e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -64,7 +64,6 @@ enum arm_smmu_msi_index { ARM_SMMU_MAX_MSIS, }; =20 -#define NUM_ENTRY_QWORDS 8 static_assert(sizeof(struct arm_smmu_ste) =3D=3D NUM_ENTRY_QWORDS * sizeof= (u64)); static_assert(sizeof(struct arm_smmu_cd) =3D=3D NUM_ENTRY_QWORDS * sizeof(= u64)); =20 @@ -1216,254 +1215,6 @@ EXPORT_SYMBOL_IF_KUNIT(arm_smmu_invs_purge); =20 /* Context descriptor manipulation functions */ =20 -/* - * Based on the value of ent report which bits of the STE the HW will acce= ss. It - * would be nice if this was complete according to the spec, but minimally= it - * has to capture the bits this driver uses. - */ -VISIBLE_IF_KUNIT -void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits) -{ - unsigned int cfg =3D FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ent[0])); - - used_bits[0] =3D cpu_to_le64(STRTAB_STE_0_V); - if (!(ent[0] & cpu_to_le64(STRTAB_STE_0_V))) - return; - - used_bits[0] |=3D cpu_to_le64(STRTAB_STE_0_CFG); - - /* S1 translates */ - if (cfg & BIT(0)) { - used_bits[0] |=3D cpu_to_le64(STRTAB_STE_0_S1FMT | - STRTAB_STE_0_S1CTXPTR_MASK | - STRTAB_STE_0_S1CDMAX); - used_bits[1] |=3D - cpu_to_le64(STRTAB_STE_1_S1DSS | STRTAB_STE_1_S1CIR | - STRTAB_STE_1_S1COR | STRTAB_STE_1_S1CSH | - STRTAB_STE_1_S1STALLD | STRTAB_STE_1_STRW | - STRTAB_STE_1_EATS | STRTAB_STE_1_MEV | - STRTAB_STE_1_INSTCFG); - used_bits[2] |=3D cpu_to_le64(STRTAB_STE_2_S2VMID); - - /* - * See 13.5 Summary of attribute/permission configuration fields - * for the SHCFG behavior. - */ - if (FIELD_GET(STRTAB_STE_1_S1DSS, le64_to_cpu(ent[1])) =3D=3D - STRTAB_STE_1_S1DSS_BYPASS) - used_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_SHCFG); - } - - /* S2 translates */ - if (cfg & BIT(1)) { - used_bits[1] |=3D - cpu_to_le64(STRTAB_STE_1_S2FWB | STRTAB_STE_1_EATS | - STRTAB_STE_1_SHCFG | STRTAB_STE_1_MEV | - STRTAB_STE_1_INSTCFG); - used_bits[2] |=3D - cpu_to_le64(STRTAB_STE_2_S2VMID | STRTAB_STE_2_VTCR | - STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2ENDI | - STRTAB_STE_2_S2PTW | STRTAB_STE_2_S2S | - STRTAB_STE_2_S2R); - used_bits[3] |=3D cpu_to_le64(STRTAB_STE_3_S2TTB_MASK); - } - - if (cfg =3D=3D STRTAB_STE_0_CFG_BYPASS) - used_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_SHCFG); -} -EXPORT_SYMBOL_IF_KUNIT(arm_smmu_get_ste_used); - -VISIBLE_IF_KUNIT -void arm_smmu_get_ste_update_safe(const __le64 *cur, const __le64 *target, - __le64 *safe_bits) -{ - const u64 eats_s1chk =3D - FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_S1CHK); - const u64 eats_trans =3D - FIELD_PREP(STRTAB_STE_1_EATS, STRTAB_STE_1_EATS_TRANS); - - /* - * When an STE changes EATS_TRANS, the sequencing code in the attach - * logic already will have the PCI cap for ATS disabled. Thus at this - * moment we can expect that the device will not generate ATS queries - * and so we don't care about the sequencing of EATS. The purpose of - * EATS_TRANS is to protect the system from hostile untrusted devices - * that issue ATS when the PCI config space is disabled. However, if - * EATS_TRANS is being changed, then we must have already trusted the - * device as the EATS_TRANS security block is being disabled. - * - * Note: now the EATS_TRANS update is moved to the first entry_set(). - * Changing S2S and EATS might transiently result in S2S=3D1 and EATS=3D1 - * which is a bad STE (see "5.2 Stream Table Entry"). In such a case, - * we can't do a hitless update. Also, it should not be added to the - * safe bits with STRTAB_STE_1_EATS_S1CHK, because EATS=3D0b11 would be - * effectively an errant 0b00 configuration. - */ - if (!((cur[1] | target[1]) & cpu_to_le64(eats_s1chk)) && - !((cur[2] | target[2]) & cpu_to_le64(STRTAB_STE_2_S2S))) - safe_bits[1] |=3D cpu_to_le64(eats_trans); - - /* - * MEV does not meaningfully impact the operation of the HW, it only - * changes how many fault events are generated, thus we can relax it - * when computing the ordering. The spec notes the device can act like - * MEV=3D1 anyhow: - * - * Note: Software must expect, and be able to deal with, coalesced - * fault records even when MEV =3D=3D 0. - */ - safe_bits[1] |=3D cpu_to_le64(STRTAB_STE_1_MEV); -} -EXPORT_SYMBOL_IF_KUNIT(arm_smmu_get_ste_update_safe); - -/* - * Figure out if we can do a hitless update of entry to become target. Ret= urns a - * bit mask where 1 indicates that qword needs to be set disruptively. - * unused_update is an intermediate value of entry that has unused bits se= t to - * their new values. - */ -static u8 arm_smmu_entry_qword_diff(struct arm_smmu_entry_writer *writer, - const __le64 *entry, const __le64 *target, - __le64 *unused_update) -{ - __le64 target_used[NUM_ENTRY_QWORDS] =3D {}; - __le64 cur_used[NUM_ENTRY_QWORDS] =3D {}; - __le64 safe[NUM_ENTRY_QWORDS] =3D {}; - u8 used_qword_diff =3D 0; - unsigned int i; - - writer->ops->get_used(entry, cur_used); - writer->ops->get_used(target, target_used); - if (writer->ops->get_update_safe) - writer->ops->get_update_safe(entry, target, safe); - - for (i =3D 0; i !=3D NUM_ENTRY_QWORDS; i++) { - /* - * Safe is only used for bits that are used by both entries, - * otherwise it is sequenced according to the unused entry. - */ - safe[i] &=3D target_used[i] & cur_used[i]; - - /* - * Check that masks are up to date, the make functions are not - * allowed to set a bit to 1 if the used function doesn't say it - * is used. - */ - WARN_ON_ONCE(target[i] & ~target_used[i]); - - /* Bits can change because they are not currently being used */ - cur_used[i] &=3D ~safe[i]; - unused_update[i] =3D (entry[i] & cur_used[i]) | - (target[i] & ~cur_used[i]); - /* - * Each bit indicates that a used bit in a qword needs to be - * changed after unused_update is applied. - */ - if ((unused_update[i] & target_used[i]) !=3D target[i]) - used_qword_diff |=3D 1 << i; - } - return used_qword_diff; -} - -static void entry_set(struct arm_smmu_entry_writer *writer, __le64 *entry, - const __le64 *target, unsigned int start, - unsigned int len) -{ - bool changed =3D false; - unsigned int i; - - for (i =3D start; len !=3D 0; len--, i++) { - if (entry[i] !=3D target[i]) { - WRITE_ONCE(entry[i], target[i]); - changed =3D true; - } - } - - if (changed) - writer->ops->sync(writer); -} - -/* - * Update the STE/CD to the target configuration. The transition from the - * current entry to the target entry takes place over multiple steps that - * attempts to make the transition hitless if possible. This function take= s care - * not to create a situation where the HW can perceive a corrupted entry. = HW is - * only required to have a 64 bit atomicity with stores from the CPU, while - * entries are many 64 bit values big. - * - * The difference between the current value and the target value is analyz= ed to - * determine which of three updates are required - disruptive, hitless or = no - * change. - * - * In the most general disruptive case we can make any update in three ste= ps: - * - Disrupting the entry (V=3D0) - * - Fill now unused qwords, execpt qword 0 which contains V - * - Make qword 0 have the final value and valid (V=3D1) with a single 64 - * bit store - * - * However this disrupts the HW while it is happening. There are several - * interesting cases where a STE/CD can be updated without disturbing the = HW - * because only a small number of bits are changing (S1DSS, CONFIG, etc) or - * because the used bits don't intersect. We can detect this by calculatin= g how - * many 64 bit values need update after adjusting the unused bits and skip= the - * V=3D0 process. This relies on the IGNORED behavior described in the - * specification. - */ -VISIBLE_IF_KUNIT -void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *en= try, - const __le64 *target) -{ - __le64 unused_update[NUM_ENTRY_QWORDS]; - u8 used_qword_diff; - - /* - * Many of the entry structures have pointers to other structures that - * need to have their updates be visible before any writes of the entry - * happen. - */ - dma_wmb(); - - used_qword_diff =3D - arm_smmu_entry_qword_diff(writer, entry, target, unused_update); - if (hweight8(used_qword_diff) =3D=3D 1) { - /* - * Only one qword needs its used bits to be changed. This is a - * hitless update, update all bits the current STE/CD is - * ignoring to their new values, then update a single "critical - * qword" to change the STE/CD and finally 0 out any bits that - * are now unused in the target configuration. - */ - unsigned int critical_qword_index =3D ffs(used_qword_diff) - 1; - - /* - * Skip writing unused bits in the critical qword since we'll be - * writing it in the next step anyways. This can save a sync - * when the only change is in that qword. - */ - unused_update[critical_qword_index] =3D - entry[critical_qword_index]; - entry_set(writer, entry, unused_update, 0, NUM_ENTRY_QWORDS); - entry_set(writer, entry, target, critical_qword_index, 1); - entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); - } else if (used_qword_diff) { - /* - * At least two qwords need their inuse bits to be changed. This - * requires a breaking update, zero the V bit, write all qwords - * but 0, then set qword 0 - */ - unused_update[0] =3D 0; - entry_set(writer, entry, unused_update, 0, 1); - entry_set(writer, entry, target, 1, NUM_ENTRY_QWORDS - 1); - entry_set(writer, entry, target, 0, 1); - } else { - /* - * No inuse bit changed, though safe bits may have changed. - */ - entry_set(writer, entry, target, 0, NUM_ENTRY_QWORDS); - } -} -EXPORT_SYMBOL_IF_KUNIT(arm_smmu_write_entry); - static void arm_smmu_sync_cd(struct arm_smmu_master *master, int ssid, bool leaf) { diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index bc4c25421835..d07fca3d1cf5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -226,6 +226,7 @@ struct arm_vsmmu; #define STRTAB_L1_DESC_L2PTR_MASK GENMASK_ULL(51, 6) =20 #define STRTAB_STE_DWORDS 8 +#define NUM_ENTRY_QWORDS 8 =20 struct arm_smmu_ste { __le64 data[STRTAB_STE_DWORDS]; @@ -1140,11 +1141,6 @@ void arm_smmu_make_s2_domain_ste(struct arm_smmu_ste= *target, bool ats_enabled); =20 #if IS_ENABLED(CONFIG_KUNIT) -void arm_smmu_get_ste_used(const __le64 *ent, __le64 *used_bits); -void arm_smmu_get_ste_update_safe(const __le64 *cur, const __le64 *target, - __le64 *safe_bits); -void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *cu= r, - const __le64 *target); void arm_smmu_get_cd_used(const __le64 *ent, __le64 *used_bits); void arm_smmu_make_bypass_ste(struct arm_smmu_device *smmu, struct arm_smmu_ste *target); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B9A66497B79 for ; Thu, 15 Oct 2026 20:09:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094965; cv=none; b=Ed3SLDRikQSFEWJgNu/+mXOfUbg0wHiG9tOryhW+hmbEL/bWFXv0MfslY9yUBFbzy4NswBUnRSS2BYuyAEHxrhxnvPYSmY5cSnMIMv1saTk5Z/mW1+XBpLtYc0funbHUD7/dvKfRExfiSWTGKMfgGWN3d4pqrD/EzM7gvS/KNVA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094965; c=relaxed/simple; bh=jGofXRdQWlPPjp83OqunXoF5r7Bgh7n8Ru/myPqaSqo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=J9v+7XsRjM7uXvSQp72BJtdv1zjfitoFwDUZ9zVoVX/PRn7yaJEHvXqgWGFoiSCnatxC2dIVF2JqU8u3pFsIsp+s51OQx0pcj3XNuytSO4oE3fg6SOoAzHFUAaK224Y63qJUX4PkZXeyu2J+sq2sxyS/1IuDdNYSauFVTsdriRI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FlX+DHjP; arc=none smtp.client-ip=209.85.221.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FlX+DHjP" Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-48c437ae81eso1354224f8f.1 for ; Thu, 15 Oct 2026 13:09:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094957; x=1792699757; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=z8Ak1AjyYrSkoTMkJRUpVV8BvgfNONKVLHUTW1w7Yqc=; b=FlX+DHjP0i6r1oedYocqIylboWX92Cy3FHm+klLH+fDociPrQxW8ijF2Boko7hyhQC Bc8Nq1gVzn+coc0i1Foa6NV/o722jLD0LFKXDpsm5UGJuYNWEFQTpnPXU9YnU60fpHqq w27dkQexjGwkyuNCvLDtdD3Rf1JK7CPYpXZ8ACNTW8w431VWYPeaIGFVfkl9W5Oyd04t Nph0t4xTiodUO7hsrIJZT/UliaS4l6wPUySyf19YnnJ7DR+TlIpIA/hbt2/BdJBWKaTl OgYlNJjP7aMTovS6A8yTB2pCBSzpmZbnp0eUJAeLQaxA2DKmjZrkg0Rd1by78M4jGQe2 l9PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094957; x=1792699757; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=z8Ak1AjyYrSkoTMkJRUpVV8BvgfNONKVLHUTW1w7Yqc=; b=ON+cBdYA2sbkKIRtuxU4VYbaVqrxefT96qeA/Az638AVUaI1B06hTCVWRJUjeFLMSf zFpOdG86CYJfd9CQDvUraDFBa0pezQFOTOWWea3N0M82GNqKMja3V0Ytut1hKWx58hr0 DLl5heWt/J7Vk2ILiotrRsQGTOmyszX+9f0UZ1HPy3AnHpMrNeF/+VGZTK+WF7hs/gva kZYr+b8q4ymZigfNMqMFsUabmdUkbi8ImHp/wp/W/mJskm1OU0NpDRsYmsswY3SXfoPh +50KE7vl64pUQ4fyf1uaI4cqpkoZU595JXA4chvB6BcWWL3ZQsRdMSymr6yzxGdXL4RS 2bkQ== X-Forwarded-Encrypted: i=1; AKwUvByoPvGaDDzMCdncRLCYQFH8MUfdUSAElvO//z4qI79y7f4RGyfOu3fbg39526oXWWw48BMU5LrbbYlb/88=@vger.kernel.org X-Gm-Message-State: AFq9FYJHZEJzdgmqy3AbG9XXulUiOowqRp5HXD82mUsymBVyJl0fvgS+ Qjb0iw/UmeBiGl38i1n03K7yBTmLjabQbwZ4OTHzvZPfci6Kc8hL4AYOjjMt7vT5nNhzhtxg7rO j5eaYrp3E5IB4yQ== X-Received: from wrvs12.prod.google.com ([2002:a5d:4ecc:0:b0:48d:ed95:6614]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:644a:0:b0:486:f42f:6860 with SMTP id ffacd0b85a97d-48dbaaefc28mr30540044f8f.56.1792094956572; Thu, 15 Oct 2026 13:09:16 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:36 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-6-smostafa@google.com> Subject: [PATCH v9 05/25] iommu/arm-smmu-v3: Move RangeInval to common code From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move arm_smmu_tlbi_calc_range() to the common code shared with the hypervisor as it will use it later to populate TLBIs for the SMMUv3 stage-2 page table. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 160 ++++++++++++++++++ .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.h | 2 + drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 160 ------------------ 3 files changed, 162 insertions(+), 160 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c index cf9287cdbd19..8c15ed04ccdb 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -484,3 +484,163 @@ void arm_smmu_write_entry(struct arm_smmu_entry_write= r *writer, __le64 *entry, } } EXPORT_SYMBOL_IF_KUNIT(arm_smmu_write_entry); + +/* + * Check address alignment for TTL hint per SMMUv3 H.a Section 4.4.1. Addr= ess + * bits below the alignment must be zero, otherwise UNPREDICTABLE. + */ +static bool arm_smmu_ttl_addr_aligned(u64 address, unsigned int tg, + unsigned int ttl) +{ + unsigned int pgsz_lg2 =3D arm_smmu_pt_level_to_lg2sz(tg, 3 - ttl); + + return !(address & GENMASK_U64(pgsz_lg2 - 1, 0)); +} + +struct arm_smmu_range_inv { + u64 start_tg; + /* Normal integer, not encoded. 0 means 0.*/ + u64 num; + unsigned int scale; +}; + +static unsigned int arm_smmu_range_inv_calc_scale(u64 num_tg) +{ + return fls64((num_tg - 1) / (CMDQ_TLBI_RANGE_NUM_MAX + 1)); +} + +static u64 arm_smmu_range_inv_calc_num(u64 num_tg, unsigned int scale) +{ + return DIV_ROUND_UP_ULL(num_tg, 1ULL << scale); +} + +/* + * Initialize the smallest range invalidation covering num_tg and ending at + * last_tg. + */ +static struct arm_smmu_range_inv arm_smmu_range_inv_init_end(u64 last_tg, + u64 num_tg) +{ + struct arm_smmu_range_inv range_inv =3D {}; + + if (!num_tg) + return range_inv; + + range_inv.scale =3D arm_smmu_range_inv_calc_scale(num_tg); + range_inv.num =3D arm_smmu_range_inv_calc_num(num_tg, range_inv.scale); + range_inv.start_tg =3D last_tg - ((range_inv.num << range_inv.scale) - 1); + return range_inv; +} + +static void +arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi, + const struct arm_smmu_range_inv *range_inv, u8 ttl, + u8 tg_enc) +{ + struct arm_smmu_cmd *cmd =3D &tlbi->range.cmds[tlbi->range.num_cmds++]; + unsigned int tgsz_lg2 =3D tg_enc * 2 + 10; + u64 iova =3D range_inv->start_tg << tgsz_lg2; + unsigned int num =3D range_inv->num - 1; + + /* 16K granule TTL=3D1 is reserved (Section 4.4.1) */ + if (WARN_ON(tgsz_lg2 =3D=3D 14 && ttl =3D=3D 1)) + ttl =3D 0; + + /* Verify address alignment for the TTL hint */ + if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl)) + ttl =3D 0; + + /* + * SMMUv3 H.a Section 4.4.1: TG!=3D0, NUM=3D=3D0, SCALE=3D=3D0, TTL=3D=3D= 0 is Reserved + * and causes CERROR_ILL. Single tg uses NUM=3D0, SCALE=3D0 with a TTL hi= nt + * to target only the exact leaf entry. + * + * For a single tg invalidation a 0 TTL can come from places like the + * SVA path that don't have enough information to get a TTL, or as a + * side of effect of the splitting. + * + * A single-TG invalidation cannot reach this point if it is part of a + * CONT group, so it is safe to transform it into a single invalidation. + * The ARM_SMMU_OPT_FULL_CONT_RANGE_INV errata does not apply. + */ + if (!num && !range_inv->scale && !ttl) + tg_enc =3D 0; + + cmd->data[0] =3D FIELD_PREP(CMDQ_TLBI_0_NUM, num) | + FIELD_PREP(CMDQ_TLBI_0_SCALE, range_inv->scale); + cmd->data[1] =3D FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) | + FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) | + FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova; +} + +/* + * Generate up to two range TLBI command payloads covering [iova, iova+siz= e). + * Sets use_full_inv if the range is too large to represent. + * + * Normally the first range invalidation is the largest representable span= which + * does not exceed the requested range. If necessary, the second range + * invalidation is the smallest representable range covering the remainder= and + * is anchored at the end. Any excess coverage from the second range + * invalidation overlaps the first instead of exceeding the requested rang= e. + * + * For SVA on an invs containing an SMMU with ARM_SMMU_OPT_FULL_CONT_RANGE= _INV, + * produce only a single range invalidation and overinvalidate so any pote= ntial CONT is + * covered by one command. + */ +void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi, + bool single_range_inv) +{ + u8 tgsz_lg2 =3D tlbi->tgsz_lg2; + struct arm_smmu_range_inv first =3D { .start_tg =3D tlbi->iova >> + tgsz_lg2 }; + u64 last_tg =3D (tlbi->iova + tlbi->size - 1) >> tgsz_lg2; + u64 num_tg =3D last_tg - first.start_tg + 1; + u8 tg_enc =3D (tgsz_lg2 - 10) / 2; + struct arm_smmu_range_inv trail; + u8 ttl =3D 0; + + /* + * Determine what level the granule is at. For non-leaf, both io-pgtable + * and SVA pass a nominal last-level granule because they don't know + * what level(s) actually apply, so leave TTL=3D0. + */ + if (tlbi->leaf_only) + ttl =3D 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3)); + + /* + * The spec defines the invalidated range as: + * Range =3D ((NUM+1) * 2^SCALE) * Translation_Granule_Size + * NUM is 5 bits, so (NUM+1) covers 1..32 granules. Find the smallest + * SCALE at which a single command could cover num_tg. + * + * Unlike other IOMMUs the spec has no alignment requirement on the + * address beyond alignment to tg (so long as TTL=3D0). + */ + first.scale =3D arm_smmu_range_inv_calc_scale(num_tg); + if (first.scale > 31) { + /* Range too large for a single command do full invalidation */ + tlbi->range.use_full_inv =3D true; + return; + } + + if (single_range_inv) { + /* + * Produce a single invalidation by rounding up and disabling + * the trailer. + */ + first.num =3D arm_smmu_range_inv_calc_num(num_tg, first.scale); + trail.num =3D 0; + } else { + /* + * Produce two invalidations by rounding down and adding a + * second trailing range invalidation anchored at the end. + */ + first.num =3D num_tg >> first.scale; + trail =3D arm_smmu_range_inv_init_end( + last_tg, num_tg - ((u64)first.num << first.scale)); + } + arm_smmu_tlbi_add_range_cmd(tlbi, &first, ttl, tg_enc); + + if (trail.num) + arm_smmu_tlbi_add_range_cmd(tlbi, &trail, ttl, tg_enc); +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h index 55ffa880b093..9f6949a60fe8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h @@ -42,4 +42,6 @@ void arm_smmu_get_ste_update_safe(const __le64 *cur, cons= t __le64 *target, void arm_smmu_write_entry(struct arm_smmu_entry_writer *writer, __le64 *cu= r, const __le64 *target); =20 +void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi, bool single_rang= e_inv); + #endif /* __ARM_SMMU_V3_COMMON_LIB_H */ diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 5d7de893865e..c2ec33c763a0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2182,166 +2182,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) arm_smmu_domain_inv(smmu_domain); } =20 -/* - * Check address alignment for TTL hint per SMMUv3 H.a Section 4.4.1. Addr= ess - * bits below the alignment must be zero, otherwise UNPREDICTABLE. - */ -static bool arm_smmu_ttl_addr_aligned(u64 address, unsigned int tg, - unsigned int ttl) -{ - unsigned int pgsz_lg2 =3D arm_smmu_pt_level_to_lg2sz(tg, 3 - ttl); - - return !(address & GENMASK_U64(pgsz_lg2 - 1, 0)); -} - -struct arm_smmu_range_inv { - u64 start_tg; - /* Normal integer, not encoded. 0 means 0.*/ - u64 num; - unsigned int scale; -}; - -static unsigned int arm_smmu_range_inv_calc_scale(u64 num_tg) -{ - return fls64((num_tg - 1) / (CMDQ_TLBI_RANGE_NUM_MAX + 1)); -} - -static u64 arm_smmu_range_inv_calc_num(u64 num_tg, unsigned int scale) -{ - return DIV_ROUND_UP_ULL(num_tg, 1ULL << scale); -} - -/* - * Initialize the smallest range invalidation covering num_tg and ending at - * last_tg. - */ -static struct arm_smmu_range_inv arm_smmu_range_inv_init_end(u64 last_tg, - u64 num_tg) -{ - struct arm_smmu_range_inv range_inv =3D {}; - - if (!num_tg) - return range_inv; - - range_inv.scale =3D arm_smmu_range_inv_calc_scale(num_tg); - range_inv.num =3D arm_smmu_range_inv_calc_num(num_tg, range_inv.scale); - range_inv.start_tg =3D last_tg - ((range_inv.num << range_inv.scale) - 1); - return range_inv; -} - -static void -arm_smmu_tlbi_add_range_cmd(struct arm_smmu_tlbi *tlbi, - const struct arm_smmu_range_inv *range_inv, u8 ttl, - u8 tg_enc) -{ - struct arm_smmu_cmd *cmd =3D &tlbi->range.cmds[tlbi->range.num_cmds++]; - unsigned int tgsz_lg2 =3D tg_enc * 2 + 10; - u64 iova =3D range_inv->start_tg << tgsz_lg2; - unsigned int num =3D range_inv->num - 1; - - /* 16K granule TTL=3D1 is reserved (Section 4.4.1) */ - if (WARN_ON(tgsz_lg2 =3D=3D 14 && ttl =3D=3D 1)) - ttl =3D 0; - - /* Verify address alignment for the TTL hint */ - if (ttl && !arm_smmu_ttl_addr_aligned(iova, tgsz_lg2, ttl)) - ttl =3D 0; - - /* - * SMMUv3 H.a Section 4.4.1: TG!=3D0, NUM=3D=3D0, SCALE=3D=3D0, TTL=3D=3D= 0 is Reserved - * and causes CERROR_ILL. Single tg uses NUM=3D0, SCALE=3D0 with a TTL hi= nt - * to target only the exact leaf entry. - * - * For a single tg invalidation a 0 TTL can come from places like the - * SVA path that don't have enough information to get a TTL, or as a - * side of effect of the splitting. - * - * A single-TG invalidation cannot reach this point if it is part of a - * CONT group, so it is safe to transform it into a single invalidation. - * The ARM_SMMU_OPT_FULL_CONT_RANGE_INV errata does not apply. - */ - if (!num && !range_inv->scale && !ttl) - tg_enc =3D 0; - - cmd->data[0] =3D FIELD_PREP(CMDQ_TLBI_0_NUM, num) | - FIELD_PREP(CMDQ_TLBI_0_SCALE, range_inv->scale); - cmd->data[1] =3D FIELD_PREP(CMDQ_TLBI_1_LEAF, tlbi->leaf_only) | - FIELD_PREP(CMDQ_TLBI_1_TTL, ttl) | - FIELD_PREP(CMDQ_TLBI_1_TG, tg_enc) | iova; -} - -/* - * Generate up to two range TLBI command payloads covering [iova, iova+siz= e). - * Sets use_full_inv if the range is too large to represent. - * - * Normally the first range invalidation is the largest representable span= which - * does not exceed the requested range. If necessary, the second range - * invalidation is the smallest representable range covering the remainder= and - * is anchored at the end. Any excess coverage from the second range - * invalidation overlaps the first instead of exceeding the requested rang= e. - * - * For SVA on an invs containing an SMMU with ARM_SMMU_OPT_FULL_CONT_RANGE= _INV, - * produce only a single range invalidation and overinvalidate so any pote= ntial CONT is - * covered by one command. - */ -static void arm_smmu_tlbi_calc_range(struct arm_smmu_tlbi *tlbi, - bool single_range_inv) -{ - u8 tgsz_lg2 =3D tlbi->tgsz_lg2; - struct arm_smmu_range_inv first =3D { .start_tg =3D tlbi->iova >> - tgsz_lg2 }; - u64 last_tg =3D (tlbi->iova + tlbi->size - 1) >> tgsz_lg2; - u64 num_tg =3D last_tg - first.start_tg + 1; - u8 tg_enc =3D (tgsz_lg2 - 10) / 2; - struct arm_smmu_range_inv trail; - u8 ttl =3D 0; - - /* - * Determine what level the granule is at. For non-leaf, both io-pgtable - * and SVA pass a nominal last-level granule because they don't know - * what level(s) actually apply, so leave TTL=3D0. - */ - if (tlbi->leaf_only) - ttl =3D 4 - ((ilog2(tlbi->iopte_size) - 3) / (tgsz_lg2 - 3)); - - /* - * The spec defines the invalidated range as: - * Range =3D ((NUM+1) * 2^SCALE) * Translation_Granule_Size - * NUM is 5 bits, so (NUM+1) covers 1..32 granules. Find the smallest - * SCALE at which a single command could cover num_tg. - * - * Unlike other IOMMUs the spec has no alignment requirement on the - * address beyond alignment to tg (so long as TTL=3D0). - */ - first.scale =3D arm_smmu_range_inv_calc_scale(num_tg); - if (first.scale > 31) { - /* Range too large for a single command do full invalidation */ - tlbi->range.use_full_inv =3D true; - return; - } - - if (single_range_inv) { - /* - * Produce a single invalidation by rounding up and disabling - * the trailer. - */ - first.num =3D arm_smmu_range_inv_calc_num(num_tg, first.scale); - trail.num =3D 0; - } else { - /* - * Produce two invalidations by rounding down and adding a - * second trailing range invalidation anchored at the end. - */ - first.num =3D num_tg >> first.scale; - trail =3D arm_smmu_range_inv_init_end( - last_tg, num_tg - ((u64)first.num << first.scale)); - } - arm_smmu_tlbi_add_range_cmd(tlbi, &first, ttl, tg_enc); - - if (trail.num) - arm_smmu_tlbi_add_range_cmd(tlbi, &trail, ttl, tg_enc); -} - /* * One TLBI command per IOTLB entry, assuming the entries are all at least * iopte_granule sized. Sets use_full_inv if too many commands would be ne= eded --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 144734749E2 for ; Thu, 15 Oct 2026 20:09:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; cv=none; b=Af8Bq+ILZ5pzAcuX51syBKJdm1IWeTByouh1FcMSqzOwynFUvfQYp2Vu/VzaQxQLr7kz+3tC9pf0n4AWZlm8KkTnqesrDaXW3ZwS5aOiJzwfyTWZV2VCQlr17ynsphLfEtClKnxPl7VJn6wvw+vt7ZpDv7rC6bZzZe0IJEqyzu0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; c=relaxed/simple; bh=OEBJ9jBwoh3u5ibQYjsm3jERzX/nBiVFHAkhW2sQFQs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KfjcZZUSKSObTEznITDh0natpwI9o0HRpo5Kq6gZcgxP56XNjZTpwY44eIHTC0B7FuYMrOxqXykBXyRjDUJIdILvbsJKNGG23sSPWxCsnTkWYZ7F4FaHkipUAJSJfhmMWU1ysOXQ8YpDGezzIaUo57yD4myyRdk8s/oiQ/MFewI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=c9zRUdRt; arc=none smtp.client-ip=209.85.208.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="c9zRUdRt" Received: by mail-ed1-f71.google.com with SMTP id 4fb4d7f45d1cf-6b1b5ea276bso3007739a12.2 for ; Thu, 15 Oct 2026 13:09:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094958; x=1792699758; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=SPgM8pFySMpBzoU0tQSRtWXbtV3WXjZpxKdn14a2X+Q=; b=c9zRUdRt5/JpvrdYBEh2jzseMWFz/3xR0DihF2MLyd5RGFjcym0ZfifMc4+PgorBuW dOQg/6DBKwawC8pnLIS8ouxuFmgakdhG4D8BzC1NZPalw2tOfacJnt6ERmqvd37TeYeq 2j0uZTqgYyuXAf3mOaafS329EQuvWZCNfly7m98HggNWDXd8n36ELmU8uPJXcEnGY9M/ uhc1aH0ZMTfBzG18OnObbNlCp2kSnN6BwhCslYJLHCKnmONjEZTLQkwcIyXclk6wwYc3 9s64xjpbMVR0rWc61dbO7lsoK7iv/ljgZHyTAKYAa1SoJs4kpzQj1V8fF/hnNSIApUAl 0pOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094958; x=1792699758; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=SPgM8pFySMpBzoU0tQSRtWXbtV3WXjZpxKdn14a2X+Q=; b=mmJ96y8fNRXg7g2gQ8i1k9CYXG8gj2tHZG0yQr+tHJOaFCZ0YS5WgaGvZ2gza+kX8p 0zT7CQmd8qpjnvv4zMSu6pX5bVqIyRdwyYQeq4dTwflKa1PEHxKTbDgP0rtW5rg5dBVk JV8P/G23lqahRjK1Dj+dER6cze5uR/twv06OpwkkNaGo47GDadZqHZWigIATeu1Rk5Ic INpSXRmBm7Fz0S3LOXvvjZdNPRCFnBJm0zuBjeNeqG7/7lb/FLdwJ6QiN+34aeCxszv0 MJTF7uBmQi2tSu44itrZj2CCRA7NMSRRU/M93qkokEWN9v+NaKXD2VdsBQI8z9GNneNL BlLw== X-Forwarded-Encrypted: i=1; AKwUvByz9vXk3dpOkb+Z9QPH1TMVxJ4llluTpA5J/nSPEiCtEfVroxpyWY2eE/gOyHZZ6fS2VT/UOXMu5n0h3Oc=@vger.kernel.org X-Gm-Message-State: AFq9FYJ94jFGO+Em4L6qMLj7XK8+IfqBskBcQD4aloAvjrhbCuzv4lCh ynFG9UBc9VREIwHu9xGihExTCYoAB06eOBBCIFErR2hGRwxOnnNKVvwCS2wjXWpv+5DgpAuY3tu bou1FDhniB3Vsww== X-Received: from edv13.prod.google.com ([2002:a05:6402:15cd:b0:6ac:bd3a:5051]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:2489:b0:6ac:8dce:5a8a with SMTP id 4fb4d7f45d1cf-6b17c25d1dbmr16505883a12.9.1792094957800; Thu, 15 Oct 2026 13:09:17 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:37 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-7-smostafa@google.com> Subject: [PATCH v9 06/25] KVM: arm64: iommu: Introduce IOMMU driver infrastructure From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh , Jean-Philippe Brucker Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To establish DMA isolation, pKVM needs an IOMMU driver that provides ops implemented at EL2. Only one driver can be used. It is registered before KVM initialises with pkvm_iommu_register_driver(). Registration must come from an initcall between core_initcall() and fs_initcall(): earlier and the CPU capabilities are not finalised, later and kvm_arm_init() at device_initcall level has already consumed the ops. The hypervisor calls the driver's init last in __pkvm_init_finalise(), so that no step that can fail runs after it, as drivers have no teardown. DMA isolation stays optional: without a registered driver pKVM initialises as before. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_host.h | 7 +++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 13 +++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu.c | 19 ++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 10 +++++++ arch/arm64/kvm/iommu.c | 35 +++++++++++++++++++++++++ 7 files changed, 87 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/iommu.h create mode 100644 arch/arm64/kvm/hyp/nvhe/iommu.c create mode 100644 arch/arm64/kvm/iommu.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index cd9b9d2462f9..0c7fb6fa24f5 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1717,4 +1717,11 @@ static __always_inline enum fgt_group_id __fgt_reg_t= o_group_id(enum vcpu_sysreg =20 long kvm_get_cap_for_kvm_ioctl(unsigned int ioctl, long *ext); =20 +struct pkvm_iommu_ops; +extern struct pkvm_iommu_ops *kvm_nvhe_sym(pkvm_iommu_ops); + +#ifndef __KVM_NVHE_HYPERVISOR__ +int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops); +#endif + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 59612d2f277c..0ddef54f7434 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -24,7 +24,7 @@ kvm-y +=3D arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.= o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ - vgic/vgic-v5.o + vgic/vgic-v5.o iommu.o =20 kvm-$(CONFIG_HW_PERF_EVENTS) +=3D pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) +=3D pauth.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h new file mode 100644 index 000000000000..df3d0cc5d4db --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NVHE_IOMMU_H__ +#define __ARM64_KVM_NVHE_IOMMU_H__ + +#include + +struct pkvm_iommu_ops { + int (*init)(void); +}; + +int pkvm_iommu_init(void); + +#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index f57450ebcb49..7cb79db9bc47 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o +=3D -Wno-override-init =20 hyp-obj-y :=3D timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o= host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \ + iommu.o hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../en= try.o \ ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o hyp-obj-y +=3D ../../../kernel/smccc-call.o diff --git a/arch/arm64/kvm/hyp/nvhe/iommu.c b/arch/arm64/kvm/hyp/nvhe/iomm= u.c new file mode 100644 index 000000000000..f4572b2dba76 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/iommu.c @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * IOMMU operations for pKVM + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +/* Only one set of ops supported */ +struct pkvm_iommu_ops *pkvm_iommu_ops __ro_after_init; + +int pkvm_iommu_init(void) +{ + /* Keep DMA isolation optional. */ + if (!pkvm_iommu_ops || !pkvm_iommu_ops->init) + return 0; + + return pkvm_iommu_ops->init(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index 45ac5f2ba4f7..620b05c2799a 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -342,6 +343,15 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 + /* + * IOMMU init is done at the end, as it sets up and initializes many + * resources that would be leaked if the hypervisor fails after as there + * is no remove_iommu_driver() at the moment. + */ + ret =3D pkvm_iommu_init(); + if (ret) + goto out; + pkvm_hyp_vm_table_init(vm_table_base); =20 pkvm_ownership_selftest(selftest_base); diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c new file mode 100644 index 000000000000..ab3f37d61579 --- /dev/null +++ b/arch/arm64/kvm/iommu.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 Google LLC + * Author: Mostafa Saleh + */ + +#include + +#include + +static DEFINE_MUTEX(pkvm_iommu_reg_lock); + +/** + * pkvm_iommu_register_driver() - Register the hypervisor IOMMU driver + * @hyp_ops: Kernel address of the ops, which live in the hypervisor image, + * i.e. &kvm_nvhe_sym(ops). + * + * Only one driver can be registered. Call from an initcall between + * core_initcall() and fs_initcall(). + * + * Return: 0 on success. + */ +int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops) +{ + guard(mutex)(&pkvm_iommu_reg_lock); + + if (!is_protected_kvm_enabled() || is_kvm_arm_initialised()) + return -EPERM; + + if (kvm_nvhe_sym(pkvm_iommu_ops)) + return -EBUSY; + + kvm_nvhe_sym(pkvm_iommu_ops) =3D kern_hyp_va(kvm_ksym_ref(hyp_ops)); + return 0; +} --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1195549E127 for ; Thu, 15 Oct 2026 20:09:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; cv=none; b=T12gIWT/m4A6flRosy/LkOSzvIfNJrr1Q96+wgO5czjCv50IBlN40xh7KnFPu0USgnfOdGoEpxbRwZykEYdPXido5Y8RNai+U7a/Ms8k8BmUeZxhb4JukXCQPymOqn1vPbZZe2zo8FjozPrQNu/FFZ0YdT9Nar64mX5PhnS4G30= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; c=relaxed/simple; bh=nN7afVMv9gWA7IKhNgInm6BMrfDGvDudnzhw9n7dBpY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=oKLDn+aNUSMSGnnxr9yXf2T4Fh7iRgDveEZbVNLdWHeR5M9goXG6xKeqpStqBbW+Cf3iUeFeCdc5+OCz1IWcK9o8EtzYfh8El1YFGFfPD+amZo+nPeCuxInL1x0VE20eqm5K3g4pE4bC4JYtGV05tooF5HnlWTycZIWXN0FSmiE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cOgg8Jni; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cOgg8Jni" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4a009f82bedso44806385e9.0 for ; Thu, 15 Oct 2026 13:09:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094959; x=1792699759; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=WtEQd+Mbv2A+20oTamxspD6VLu1dz9WsK6wgSsf9W6U=; b=cOgg8JniyZ5CY0XY5Es8D3FpRp3u6Jyq/Wgb8z3HetUM7byTTNF6UrxKcwXMHJVGel CgtVpd8T8JioYAJ3yhsvl/pz0/Q5I2c/fxgOzWHCXU7Cj+JibEApozOQ0Ur4jarjCcTz 5FOzFc969bS/Dge/sxv09S0vyBxipdII2sDK5kqmkJ84/Jat8RsvnsubhHRsPi5n8d44 rEOGapRDD/tjemNp1t8FdKGxPxXnlZtUZuQd9vdFwiPzfHDrOQ6eQpfzHPwwQPkTg49j GxRy2zmEXL3+eoS6EbcXIW3Hob3nZOn1lRkFN+xbskkb/znCgrz1d6+xlbE2DOqBsDs4 PyAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094959; x=1792699759; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=WtEQd+Mbv2A+20oTamxspD6VLu1dz9WsK6wgSsf9W6U=; b=n+cGr10eI3gAOHh4Bom6poNn+sxT6fZl2oIWR9pHZaCOvM2szEGD5OOrVvp4x0Iof7 Xn1YdhFooAs7oO8jH6N72/5ruTlHCB3F3lrZ8Nk873EaEfNYPInFrbf8POaWiq/LxKCJ KobWsRztqxaQWeWJy+EvK8JLyebaoWQCZhnaLUWGSSIwdg75i+SYp95JHz6ZmnySSI9g UdLEBnPVONbI59zhZ69cUoSPvXTokGMvh9j20oQuDdjYMDnLcOE9xb2Dws3hyRqN5KmX p2HO9b12l+0AqrjD1kg4FvopziEIhi9ikDhUmwoGwedJZLaAD8uDmMAWoWHDZYN5bX89 LxNA== X-Forwarded-Encrypted: i=1; AKwUvBxJGN0hJpt/o1GtV7ZF3leCukQuOKQRxP4zIKdrGNAQpVMP65tS15fPfGEHPjLbTEkwoU9cCYyf/EYaCYI=@vger.kernel.org X-Gm-Message-State: AFq9FYLzD165G7oqpbNkG+6PKrYSvO+25FX35TM/WEm6wLPlAR75ukRx 8S29bMG0bHCwTeteIn6S9fNxP1O3bgXiIRR7u15tK6HB5xIiJsd5jnsPax/2MCMAVVL2WH2CsbZ NccwKPuP5eeh1Jw== X-Received: from wmjs9.prod.google.com ([2002:a7b:c389:0:b0:4a1:8be5:4018]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6207:b0:49f:ff73:9971 with SMTP id 5b1f17b1804b1-4a18e4aff66mr402752955e9.18.1792094959086; Thu, 15 Oct 2026 13:09:19 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:38 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-8-smostafa@google.com> Subject: [PATCH v9 07/25] KVM: arm64: iommu: Shadow host stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a page table for the IOMMU that shadows the host CPU stage-2 to establish DMA isolation. An initial snapshot is taken after the driver init, then on every ownership change a callback is called for the IOMMU driver to update its page table. The callback runs under host_mmu.lock, prot 0 unmaps, and a range may already be mapped with the same prot. There are 3 different ways to add the callback: 1) In the high level memory transitions (__pkvm_host_donate_hyp(), __pkvm_host_donate_guest(), ...). 2) In lower level functions covering all transitions: - host_stage2_set_owner_metadata_locked(), which covers __pkvm_host_donate_hyp(), __pkvm_host_donate_guest() and __pkvm_guest_unshare_host(). - host_stage2_set_owner_locked() for PKVM_ID_HOST, which covers __pkvm_hyp_donate_host(), __pkvm_host_force_reclaim_page_guest() and __pkvm_host_reclaim_page_guest(). - __pkvm_guest_share_host(). 3) In the lowest level function __host_update_page_state(), which requires only one callback. However, the page state is not enough there as the old state is needed as well. Option 2 is implemented here. IOMMUs can't handle faults, so the snapshot maps everything the host owns, including what it has not faulted in yet. Failures to update the shadow are fatal, as WARN_ON() at EL2 panics, so a driver must guarantee the update cannot fail. The SMMUv3 driver does so by mapping memory at page granularity, so ownership changes never split a block, and by sizing its page table pool accordingly. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 9 ++ arch/arm64/kvm/hyp/nvhe/iommu.c | 123 +++++++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 27 +++--- 3 files changed, 143 insertions(+), 16 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index df3d0cc5d4db..6cbcc9d65b4b 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -3,11 +3,20 @@ #define __ARM64_KVM_NVHE_IOMMU_H__ =20 #include +#include =20 struct pkvm_iommu_ops { int (*init)(void); + /* + * Mirror [start, end) of the host stage-2 with IOMMU_* @prot, where 0 + * unmaps. Called with host_mmu.lock held. A range may already be mapped + * with the same @prot, which must succeed. + */ + int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 int pkvm_iommu_init(void); =20 +int pkvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu.c b/arch/arm64/kvm/hyp/nvhe/iomm= u.c index f4572b2dba76..662b1ed2330f 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu.c @@ -4,16 +4,131 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include +#include + #include +#include +#include =20 /* Only one set of ops supported */ struct pkvm_iommu_ops *pkvm_iommu_ops __ro_after_init; =20 -int pkvm_iommu_init(void) +/* Protected by host_mmu.lock */ +static bool pkvm_idmap_initialized; + +static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) { - /* Keep DMA isolation optional. */ - if (!pkvm_iommu_ops || !pkvm_iommu_ops->init) + int iommu_prot =3D 0; + + if (prot & KVM_PGTABLE_PROT_R) + iommu_prot |=3D IOMMU_READ; + if (prot & KVM_PGTABLE_PROT_W) + iommu_prot |=3D IOMMU_WRITE; + + /* We don't understand that, might be dangerous. */ + WARN_ON(prot & ~PKVM_HOST_MEM_PROT); + return iommu_prot; +} + +static int pkvm_iommu_snapshot_walker(const struct kvm_pgtable_visit_ctx *= ctx, + enum kvm_pgtable_walk_flags visit) +{ + bool is_memory =3D *(bool *)ctx->arg; + kvm_pte_t pte =3D ctx->old; + u64 start =3D ctx->addr; + u64 block_end, end; + int prot; + + block_end =3D ALIGN_DOWN(ctx->addr, kvm_granule_size(ctx->level)) + + kvm_granule_size(ctx->level); + end =3D min(ctx->end, block_end); + /* + * Annotated PTEs stay unmapped. Everything else is mapped, including + * PTEs the host has not faulted in yet (0) to avoid page faults. + */ + if (pte && !kvm_pte_valid(pte)) return 0; =20 - return pkvm_iommu_ops->init(); + if (kvm_pte_valid(pte)) + prot =3D pkvm_to_iommu_prot(kvm_pgtable_stage2_pte_prot(pte)); + else + prot =3D IOMMU_READ | IOMMU_WRITE; + + if (!is_memory) + prot |=3D IOMMU_MMIO; + + return pkvm_iommu_ops->host_stage2_idmap(start, end, prot); +} + +static int pkvm_iommu_snapshot_host_stage2(void) +{ + struct kvm_pgtable *pgt =3D &host_mmu.pgt; + bool is_memory; + struct kvm_pgtable_walker walker =3D { + .cb =3D pkvm_iommu_snapshot_walker, + .flags =3D KVM_PGTABLE_WALK_LEAF, + .arg =3D &is_memory, + }; + int ret =3D 0, i; + u64 start =3D 0; + + hyp_spin_lock(&host_mmu.lock); + for (i =3D 0; i < hyp_memblock_nr; i++) { + struct memblock_region *reg =3D &hyp_memory[i]; + + if (start < reg->base) { + is_memory =3D false; + ret =3D kvm_pgtable_walk(pgt, start, reg->base - start, &walker); + if (ret) + goto out_unlock; + } + + is_memory =3D true; + ret =3D kvm_pgtable_walk(pgt, reg->base, reg->size, &walker); + if (ret) + goto out_unlock; + + start =3D reg->base + reg->size; + } + + if (start < BIT(pgt->ia_bits)) { + is_memory =3D false; + ret =3D kvm_pgtable_walk(pgt, start, BIT(pgt->ia_bits) - start, &walker); + if (ret) + goto out_unlock; + } + + pkvm_idmap_initialized =3D true; + +out_unlock: + hyp_spin_unlock(&host_mmu.lock); + return ret; +} + +int pkvm_iommu_init(void) +{ + int ret; + + /* Keep DMA isolation optional. */ + if (!pkvm_iommu_ops || !pkvm_iommu_ops->init || + !pkvm_iommu_ops->host_stage2_idmap) + return 0; + + ret =3D pkvm_iommu_ops->init(); + if (ret) + return ret; + + return WARN_ON(pkvm_iommu_snapshot_host_stage2()); +} + +int pkvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot) +{ + hyp_assert_lock_held(&host_mmu.lock); + + if (!pkvm_idmap_initialized) + return 0; + + return pkvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(p= rot)); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 75d250ca809c..c14af6a4877c 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -17,6 +17,7 @@ =20 #include #include +#include #include #include #include @@ -611,16 +612,15 @@ static int host_stage2_set_owner_metadata_locked(phys= _addr_t addr, u64 size, ret =3D host_stage2_try(kvm_pgtable_stage2_annotate, &host_mmu.pgt, addr, size, &host_s2_pool, KVM_HOST_INVALID_PTE_TYPE_DONATION, annotation); - if (!ret) { - /* - * After stage2 maintenance has happened, but before the page - * owner has changed. - */ - pkvm_sme_dvmsync_fw_call(); - __host_update_page_state(addr, size, PKVM_NOPAGE); - } - - return ret; + if (ret) + return ret; + /* + * After stage2 maintenance has happened, but before the page + * owner has changed. + */ + pkvm_sme_dvmsync_fw_call(); + __host_update_page_state(addr, size, PKVM_NOPAGE); + return WARN_ON(pkvm_iommu_host_stage2_idmap(addr, addr + size, 0)); } =20 int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id) @@ -633,8 +633,10 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64= size, u8 owner_id) return -EPERM; =20 ret =3D host_stage2_idmap_locked(addr, size, PKVM_HOST_MEM_PROT); - if (!ret) - __host_update_page_state(addr, size, PKVM_PAGE_OWNED); + if (ret) + break; + __host_update_page_state(addr, size, PKVM_PAGE_OWNED); + ret =3D WARN_ON(pkvm_iommu_host_stage2_idmap(addr, addr + size, PKVM_HOS= T_MEM_PROT)); break; case PKVM_ID_HYP: ret =3D host_stage2_set_owner_metadata_locked(addr, size, @@ -1049,6 +1051,7 @@ int __pkvm_guest_share_host(struct pkvm_hyp_vcpu *vcp= u, u64 gfn) pkvm_mkstate(KVM_PGTABLE_PROT_RWX, PKVM_PAGE_SHARED_OWNED), &vcpu->vcpu.arch.pkvm_memcache, 0)); WARN_ON(__host_set_page_state_range(phys, PAGE_SIZE, PKVM_PAGE_SHARED_BOR= ROWED)); + WARN_ON(pkvm_iommu_host_stage2_idmap(phys, phys + PAGE_SIZE, PKVM_HOST_ME= M_PROT)); unlock: guest_unlock_component(vm); host_unlock_component(); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41B8749F10D for ; Thu, 15 Oct 2026 20:09:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094965; cv=none; b=JZG0Sde1qYVGZZ4H8K4GVezWoYcphBx1/5oyyxsYOGB/sfc/ONZV0wQ+Ajx7K9ZbQaNzib+V0SdCOQM1sZg2ntp3bWWv0ewjYArZjMRX+ToKWyc5Qsg3UvJNNFJE7WLa5yjx6kJ0KZPLNmuHELmO52bSGSPiI7u6e8OK2BzKgRY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094965; c=relaxed/simple; bh=MYIHfQGtsSCPKNTZVHBtpV7bnA6XMGFWZjWzYiFOijg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=arKykFQheK/+KkbVpyKS3rm4sEYdbd62BXSOCNmIJp4mRaZTUDxlZJc1rnEQNfstNZh5AZ5x9uaKi0pwGqfsr7wPrbBNgidIksGJM6qn43jR1QeYZ7sTV4phgYDFp/huk9zYf3q8U+22ERU7/0tYTlwy8MBaPX/Vj8YRQyoqh/4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LFWx4th+; arc=none smtp.client-ip=209.85.128.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LFWx4th+" Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4a0761224e1so21970965e9.0 for ; Thu, 15 Oct 2026 13:09:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094960; x=1792699760; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=uPXk62xgLV+hWiazYa3Dt6SbhQEq4OxwLitgfdTfwW4=; b=LFWx4th+2dA/wmFYbBg2Tt50Ea6ZwU7rNb1hF43EeYSAkt1Uf3wRKjbxN9QyReDQpw F1iDKYs7gVvc6WgeXCegSROBdIXOHJvIod/l1Y07GJvzdwe9f7EvSdj2/PvVsrj0H2eo /6Z9ek3PNc09EIUp6iYeoFm5iK5WkdmSWpTZ9CA8WSRh5jnKCUDTEq4mSh1bI0cA3SXl XpLp5CsEwH2Apwke82YT8B3OSbGy71IGikqN4pweNXS6qApSziEuBx7Ge8y5DD0SOdSu vTuZgVXNTjLEh/YaT6TJhzhRFkwzJRaMSkIPajDZXX8OT05loEXa9hmZnkCPFA8KhjFf 12tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094960; x=1792699760; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uPXk62xgLV+hWiazYa3Dt6SbhQEq4OxwLitgfdTfwW4=; b=2dhDZ1CbEwINWZj64TFti9677hf1kGjbJjZdXoyLGzPFz5VCxtXhEZRG3SAPm31kyA 5wP7Ef58fyUNJh+tgJHLJaS1eUsJTj5lTUYfXpyjTbBJZC7KPqUyP5+COci1PofDTyFZ gM+HieU5vjKth4csHpRh4qmbvGy1DGt1VWn/t+H51i2bXEVMiQAf+DWmXDD7XB3xr/9F GkhkjOZ4lEFqMNPC27EOWo0fwto5p2uffHGmOFawEJfmt1ZKtjZviGXTNoyx2zBb9D0P 1rNMQlvbZp3j2dFr/Aa5VPtk8O6LF+NzeW+7/FfB4lEUUP6/2ZtcRnXcYAlNP+BKO8iY 4iqA== X-Forwarded-Encrypted: i=1; AKwUvBxQdP/pgy9MhLYr44gtQGHv7yutlYmrBO7hPE7MRyYAFUHDC+vvf+RUoba6dW5RnfB7C3ZtRO9MH0ddaSI=@vger.kernel.org X-Gm-Message-State: AFq9FYJG/VRszmqFi24A+8JrgMguPJND3XU1SO8xAsIrTjJFZgQTA/T2 erV9RiTfIk09Fk0dho8erYDVt8F+hLMOOEnBOIq3JgcMMdAMheRbpPZ/KEdFjo/V90iYh2LsGxE U+ilzYx8PVHbkHQ== X-Received: from wmjt9.prod.google.com ([2002:a7b:c3c9:0:b0:4a1:7dc3:5e6b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5251:b0:49f:e772:6ddf with SMTP id 5b1f17b1804b1-4a18e4d5a01mr408073195e9.32.1792094960196; Thu, 15 Oct 2026 13:09:20 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:39 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-9-smostafa@google.com> Subject: [PATCH v9 08/25] KVM: arm64: iommu: Add memory pool From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" IOMMU drivers need memory for the shadow page table. As with the host stage-2 CPU page table, reserve it early from the pKVM carveout and add it to a hyp_pool that the IOMMU driver allocates from and frees to at run time. The pool is fixed for the lifetime of the hypervisor. The carveout is reserved before any driver registers, so the pool size comes from the kernel command line, "kvm-arm.iommu_pgt_mem". When the driver registers it passes the number of pages it needs: if that is more than what was reserved, registration fails and pKVM runs without DMA isolation. Otherwise the driver gets the whole reserved pool, not only what it asked for. Signed-off-by: Mostafa Saleh --- .../admin-guide/kernel-parameters.txt | 7 ++++ arch/arm64/include/asm/kvm_host.h | 3 +- arch/arm64/include/asm/kvm_pkvm.h | 10 ++++++ arch/arm64/kvm/hyp/include/nvhe/iommu.h | 8 ++++- arch/arm64/kvm/hyp/nvhe/iommu.c | 19 +++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 11 +++++- arch/arm64/kvm/iommu.c | 35 ++++++++++++++++++- arch/arm64/kvm/pkvm.c | 1 + 8 files changed, 89 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index e75344f4e0cd..96b4151f31c1 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3243,6 +3243,13 @@ Kernel parameters max_snp_asid =3D=3D min_sev_asid-1, will effectively make SEV-ES unusable. =20 + kvm-arm.iommu_pgt_mem=3Dnn[KMG] + [KVM,ARM,EARLY] + Memory allocated for the IOMMU pool from the KVM carveout + when running in protected mode (See kvm-arm.mode=3D). If + unset or smaller than the IOMMU driver requires, the driver + fails to register and pKVM runs without DMA isolation. + kvm-arm.mode=3D [KVM,ARM,EARLY] Select one of KVM/arm64's modes of operation. diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 0c7fb6fa24f5..253b4bd1f421 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1721,7 +1721,8 @@ struct pkvm_iommu_ops; extern struct pkvm_iommu_ops *kvm_nvhe_sym(pkvm_iommu_ops); =20 #ifndef __KVM_NVHE_HYPERVISOR__ -int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops); +int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops, + unsigned int nr_pages); #endif =20 #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm= _pkvm.h index cad60569f061..6027c7280e07 100644 --- a/arch/arm64/include/asm/kvm_pkvm.h +++ b/arch/arm64/include/asm/kvm_pkvm.h @@ -73,6 +73,7 @@ static inline bool kvm_pkvm_ioctl_allowed(struct kvm *kvm= , unsigned int ioctl) =20 extern struct memblock_region kvm_nvhe_sym(hyp_memory)[]; extern unsigned int kvm_nvhe_sym(hyp_memblock_nr); +extern unsigned int kvm_nvhe_sym(hyp_iommu_pages); =20 static inline unsigned long hyp_vmemmap_memblock_size(struct memblock_region *reg, size_t vmemmap_entr= y_size) @@ -185,6 +186,15 @@ static inline unsigned long hyp_ffa_proxy_pages(void) return (2 * KVM_FFA_MBOX_NR_PAGES) + DIV_ROUND_UP(desc_max, PAGE_SIZE); } =20 +/* + * Set from the kernel command line, as the carveout is reserved before the + * IOMMU driver registers. Registration fails if the driver needs more. + */ +static inline unsigned int pkvm_iommu_pages(void) +{ + return kvm_nvhe_sym(hyp_iommu_pages); +} + static inline size_t pkvm_host_sve_state_size(void) { if (!system_supports_sve()) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 6cbcc9d65b4b..acd5058ef3ed 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -15,8 +15,14 @@ struct pkvm_iommu_ops { int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 -int pkvm_iommu_init(void); +int pkvm_iommu_init(void *pool_base, unsigned int nr_pages); =20 int pkvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, enum kvm_pgtable_prot prot); + +/* Allocate pages from the IOMMU carveout, returns zeroed memory. */ +void *pkvm_iommu_alloc_pages(u8 order); +/* Free pages from pkvm_iommu_alloc_pages(). */ +void pkvm_iommu_free_pages(void *ptr); + #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu.c b/arch/arm64/kvm/hyp/nvhe/iomm= u.c index 662b1ed2330f..36432bd46c78 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu.c @@ -14,6 +14,8 @@ /* Only one set of ops supported */ struct pkvm_iommu_ops *pkvm_iommu_ops __ro_after_init; =20 +static struct hyp_pool iommu_pages_pool; + /* Protected by host_mmu.lock */ static bool pkvm_idmap_initialized; =20 @@ -106,7 +108,7 @@ static int pkvm_iommu_snapshot_host_stage2(void) return ret; } =20 -int pkvm_iommu_init(void) +int pkvm_iommu_init(void *pool_base, unsigned int nr_pages) { int ret; =20 @@ -115,6 +117,11 @@ int pkvm_iommu_init(void) !pkvm_iommu_ops->host_stage2_idmap) return 0; =20 + ret =3D hyp_pool_init(&iommu_pages_pool, hyp_virt_to_pfn(pool_base), + nr_pages, 0); + if (ret) + return ret; + ret =3D pkvm_iommu_ops->init(); if (ret) return ret; @@ -132,3 +139,13 @@ int pkvm_iommu_host_stage2_idmap(phys_addr_t start, ph= ys_addr_t end, =20 return pkvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(p= rot)); } + +void *pkvm_iommu_alloc_pages(u8 order) +{ + return hyp_alloc_pages(&iommu_pages_pool, order); +} + +void pkvm_iommu_free_pages(void *ptr) +{ + hyp_put_page(&iommu_pages_pool, ptr); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index 620b05c2799a..8259837fb6de 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -22,6 +22,8 @@ =20 unsigned long hyp_nr_cpus; =20 +unsigned int hyp_iommu_pages __ro_after_init; + #define hyp_percpu_size ((unsigned long)__per_cpu_end - \ (unsigned long)__per_cpu_start) =20 @@ -33,6 +35,7 @@ static void *selftest_base; static void *ffa_proxy_pages; static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; static struct hyp_pool hpool; +static void *iommu_base; =20 static int divide_memory_pool(void *virt, unsigned long size) { @@ -70,6 +73,12 @@ static int divide_memory_pool(void *virt, unsigned long = size) if (!ffa_proxy_pages) return -ENOMEM; =20 + if (hyp_iommu_pages) { + iommu_base =3D hyp_early_alloc_contig(hyp_iommu_pages); + if (!iommu_base) + return -ENOMEM; + } + return 0; } =20 @@ -348,7 +357,7 @@ void __noreturn __pkvm_init_finalise(void) * resources that would be leaked if the hypervisor fails after as there * is no remove_iommu_driver() at the moment. */ - ret =3D pkvm_iommu_init(); + ret =3D pkvm_iommu_init(iommu_base, hyp_iommu_pages); if (ret) goto out; =20 diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c index ab3f37d61579..4e81878192a0 100644 --- a/arch/arm64/kvm/iommu.c +++ b/arch/arm64/kvm/iommu.c @@ -7,6 +7,7 @@ #include =20 #include +#include =20 static DEFINE_MUTEX(pkvm_iommu_reg_lock); =20 @@ -14,13 +15,16 @@ static DEFINE_MUTEX(pkvm_iommu_reg_lock); * pkvm_iommu_register_driver() - Register the hypervisor IOMMU driver * @hyp_ops: Kernel address of the ops, which live in the hypervisor image, * i.e. &kvm_nvhe_sym(ops). + * @nr_pages: Number of pages the driver needs from the IOMMU pool, see + * kvm-arm.iommu_pgt_mem. * * Only one driver can be registered. Call from an initcall between * core_initcall() and fs_initcall(). * * Return: 0 on success. */ -int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops) +int __init pkvm_iommu_register_driver(struct pkvm_iommu_ops *hyp_ops, + unsigned int nr_pages) { guard(mutex)(&pkvm_iommu_reg_lock); =20 @@ -30,6 +34,35 @@ int __init pkvm_iommu_register_driver(struct pkvm_iommu_= ops *hyp_ops) if (kvm_nvhe_sym(pkvm_iommu_ops)) return -EBUSY; =20 + if (!nr_pages) + return -EINVAL; + + if (nr_pages > pkvm_iommu_pages()) { + kvm_err("IOMMU pool needs kvm-arm.iommu_pgt_mem=3D%luK or more\n", + nr_pages * PAGE_SIZE / SZ_1K); + return -ENOMEM; + } + kvm_nvhe_sym(pkvm_iommu_ops) =3D kern_hyp_va(kvm_ksym_ref(hyp_ops)); return 0; } + +static int __init early_iommu_pgt_mem(char *arg) +{ + unsigned long long requested_size; + + if (!arg) + return -EINVAL; + + requested_size =3D memparse(arg, NULL); + + /* hyp_pool_init() computes nr_pages << PAGE_SHIFT in an unsigned int. */ + if (requested_size > (UINT_MAX & PAGE_MASK)) { + kvm_err("kvm-arm.iommu_pgt_mem is too large\n"); + return -EINVAL; + } + + kvm_nvhe_sym(hyp_iommu_pages) =3D DIV_ROUND_UP(requested_size, PAGE_SIZE); + return 0; +} +early_param("kvm-arm.iommu_pgt_mem", early_iommu_pgt_mem); diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 8e4c6e4bec12..b6cf01e00f6b 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -63,6 +63,7 @@ void __init kvm_hyp_reserve(void) hyp_mem_pages +=3D hyp_vmemmap_pages(STRUCT_HYP_PAGE_SIZE); hyp_mem_pages +=3D pkvm_selftest_pages(); hyp_mem_pages +=3D hyp_ffa_proxy_pages(); + hyp_mem_pages +=3D pkvm_iommu_pages(); =20 /* * Try to allocate a PMD-aligned region to reduce TLB pressure once --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93E804A015A for ; Thu, 15 Oct 2026 20:09:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; cv=none; b=KuFZzC4KVJFSrfO4FfNL67iJGvpcPoq2w49I/7D0y/2DijaOstMJ8FA+R/NsPbQFmY6f+Hod+chTnhgWpKTn1oL3iSlEiwhn+b0qTpTsDXsjpCsc3K4oCTey3wv/RaotonfR3su3PGGE/bHRe+7KQy6oy27bJyFZb/q409R37Oo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094966; c=relaxed/simple; bh=3vT4+pcdhJcVD4Z3oSn11IvY9+bAtQf3X3KAWQIbuE0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nb+Ub0WgZVih79JiksexF2/OrnRm3wN4KH2lT6Jo+QAWUp5ic0DXzjX/9PeZguoqhYB2WBReAjN69xRnbZwnI2CkWh4DOKiNGCub5tHG1Pi3DhmvsUfTZNr/6bYep0ptyzCwmVnTeD9yoQv7HRHPVenXnmXurrkGWRzqHiJDvK8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pTrEhjjZ; arc=none smtp.client-ip=209.85.221.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pTrEhjjZ" Received: by mail-wr1-f72.google.com with SMTP id ffacd0b85a97d-48dfa054f26so440276f8f.2 for ; Thu, 15 Oct 2026 13:09:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094961; x=1792699761; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=laM6+wZiNuYMQDtQzAoP4s+131ogUMmRLEJwdwwKDsc=; b=pTrEhjjZkao/gkr8ZRRphcdXpdGCCP/7CDO/PHIyaoqD9FdruInUs3+1M9usdVLA6A GJAhMMJs2K4yCp1cxWlt/8yhwarEds9LzTcxkIBYd+dyV/kHIqYNVnxrgYCYZyx43fPT ZoIXpHrOteHb1DO1u3cirX4oChm73WtVTxS4tfUz/FQ34Ct5mSi6fHuHz5OjVBZs08ZJ 9g/p0Vkak3dAikCbMI7TRzZ0YPDhg4tSF79X5pmfMLhUxBXfQ5DBwSab9UgFrXT4jl0G PPLOzM4t9XzfP32McXAGGf4cDLZTJnyKYR6FMyZLiQngegp+z3KTcOe5dRWJ68ZyIy6l tzSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094962; x=1792699762; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=laM6+wZiNuYMQDtQzAoP4s+131ogUMmRLEJwdwwKDsc=; b=nUwdTZAObK3XdhC8+lcMaRvrHCw7aJYbEEptu5NEbtaqJ+3lAIqS+e4pnwyvxSbtd9 bKX8fa5ARwvszakNryL2l4a1vP9l4V9rmumFOSwASITMlGMT72N3Rq3HzlFR8sW4z5NE bGsFVHj6jKzrYa5POys0N7xGM8k2KO1VK3igMfTNst/5p8ctLoozWxvXnRuwEcDAmsqM u0IFdmIM14jWHeUVkglmGPJMmmG40tuLW5mxirEhx53U4lXE9Dlg0w9N5TCe+ti/y7yY d3EvSKuK7RHVJKoDUUlQVo2QDvQQUVHPLoHfjcan8GdO2iHpWR3FyUxt2TWU4WiLhwiI N0IA== X-Forwarded-Encrypted: i=1; AKwUvBxs4P8krpwDRXJrKW/TW/wFxo4i5uGYHCoPLw7NBkJoBdWV4yHp+IExlF8K8U/E/HoVb+wQeG2IyjWO9lE=@vger.kernel.org X-Gm-Message-State: AFq9FYJtZ94/nhJDTLpqTJ25NkMm0R25kM0msbQ5NLWFJMJ+hZy+y/4B 4iUr/WCaCLJ3d9ipGCFtResxW9Up/16X22tL7BZKaib0FcfhtmOxRtgrvsdrPZbQEP6i+jdHJQL vASn5Q6T7DxiTeA== X-Received: from wmbd8.prod.google.com ([2002:a05:600c:58c8:b0:4a1:8444:ff6d]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e4a:b0:4a1:ac66:6ac8 with SMTP id 5b1f17b1804b1-4a1c7fa065fmr12093425e9.4.1792094961525; Thu, 15 Oct 2026 13:09:21 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:40 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-10-smostafa@google.com> Subject: [PATCH v9 09/25] KVM: arm64: iommu: Support DABT for IOMMU From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pKVM SMMUv3 driver emulates the SMMU registers for the host: the MMIO is donated to the hypervisor so that host accesses trap, and the hypervisor performs or rejects each access on the host's behalf. Add a dabt_handler op and call it from the host data abort path for aborts with a valid instruction syndrome on non-memory addresses. If the driver emulates the access, skip the faulting instruction, otherwise take the normal path, where hyp-owned MMIO is denied. Only aborts with a valid instruction syndrome can be emulated, so the host must access emulated registers with single-register loads and stores (readl/writel and friends). Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 9 +++++++++ arch/arm64/kvm/hyp/nvhe/iommu.c | 6 ++++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 17 +++++++++++++++++ 3 files changed, 32 insertions(+) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index acd5058ef3ed..bd6604f194b5 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -13,6 +13,13 @@ struct pkvm_iommu_ops { * with the same @prot, which must succeed. */ int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); + /* + * Emulate a host data abort on MMIO @addr + * @esr has a valid instruction syndrome and @regs holds the host x0-x30, + * SRT 31 is the zero register. Return true if the access was emulated, + * which would skip the instruction. + */ + bool (*dabt_handler)(struct user_pt_regs *regs, u64 esr, u64 addr); }; =20 int pkvm_iommu_init(void *pool_base, unsigned int nr_pages); @@ -25,4 +32,6 @@ void *pkvm_iommu_alloc_pages(u8 order); /* Free pages from pkvm_iommu_alloc_pages(). */ void pkvm_iommu_free_pages(void *ptr); =20 +bool pkvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 = addr); + #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu.c b/arch/arm64/kvm/hyp/nvhe/iomm= u.c index 36432bd46c78..cdf6e87bb68e 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu.c @@ -149,3 +149,9 @@ void pkvm_iommu_free_pages(void *ptr) { hyp_put_page(&iommu_pages_pool, ptr); } + +bool pkvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 = addr) +{ + return pkvm_iommu_ops && pkvm_iommu_ops->dabt_handler && + pkvm_iommu_ops->dabt_handler(regs, esr, addr); +} diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index c14af6a4877c..523bb6bd793b 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -13,6 +13,7 @@ #include #include =20 +#include #include =20 #include @@ -778,6 +779,12 @@ static void host_inject_mem_abort(struct kvm_cpu_conte= xt *host_ctxt) inject_host_exception(esr); } =20 +static bool is_dabt_with_isv(u64 esr) +{ + return (ESR_ELx_EC(esr) =3D=3D ESR_ELx_EC_DABT_LOW) && + (esr & ESR_ELx_ISV); +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault; @@ -800,6 +807,16 @@ void handle_host_mem_abort(struct kvm_cpu_context *hos= t_ctxt) BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS)); addr =3D FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12; =20 + /* + * Let the IOMMU driver emulate accesses to its MMIO. + */ + if (is_dabt_with_isv(esr) && !addr_is_memory(addr) && + pkvm_iommu_host_dabt_handler(&host_ctxt->regs, esr, + addr | FAR_TO_FIPA_OFFSET(fault.far_el2))) { + kvm_skip_host_instr(); + return; + } + switch (host_stage2_idmap(addr)) { case -EPERM: host_inject_mem_abort(host_ctxt); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72344413796 for ; Thu, 15 Oct 2026 20:09:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094969; cv=none; b=ZCrbn50+RZYZrnFIpW8TwOLHBvXXmhZYw3joz0hgFLP7F6cBUWkRQeTA3+YjV2GcUuTzIiHVGtj2CmLCjzkySI4XBUcXwWgw1mhj0+qCS2DwY7K+Jh2Wld8a844oR0lK0wDWutE/NLIi1Nd2ljAe3k96bBVo8UxISyglvi+KBbo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094969; c=relaxed/simple; bh=AuMop6iv85sEbIDnkg7iHftbM8s83ZooxPppN5Nau7U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=larOuag9UobHdvIZ/Rr+LSSya908/xZIwOSkIlvOu/y9nqprksB06Ef1kVnuWURfAiQVXTcI953E/dCe6UttYX0XrxZIiObwuyAtCF6nnEWhEYjqBKYqRM7fDifLB/kgthcHw291G2iCykTOse+y9cJTike7I7oWuRXJO6HXXfA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gytamOUe; arc=none smtp.client-ip=209.85.221.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gytamOUe" Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-48df02b91f9so173336f8f.1 for ; Thu, 15 Oct 2026 13:09:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094962; x=1792699762; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=8UA46w466axNh+PriFGrOQFaeqWdmA8tYPa6OW//CU8=; b=gytamOUeaBlcSnXJs2Ts4LTliT9Sb6ajcJa6UcYM9CglN0kmD9S9/tuw6cuf2SL3/b lI/rhN1M/YHwUNp1vH1ppaF5FLbhvnmYgCxgPIcs3/qaWt5h4SD9QsyoeTGcPaMSgMF6 A2OBalQ9s1EETzSJ20YTv6LeBjjUIxpAS0QlsSjSjR6J9rZqI6jjRMlEk0xRv6NPXS5+ Z1Q5LbI2G2yD6p8P/2ryJRFdA7or/0F+lNaFDnnrP+IwRXF+1aKg3GV/vHsMCfffAUTt EZVRc3Ub0H9K1Nb4AG+AU7E338iuFqxhKxuLWy+sfGXRa9qlYXmVUD5WNqpUB2wg+f/W YK6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094962; x=1792699762; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=8UA46w466axNh+PriFGrOQFaeqWdmA8tYPa6OW//CU8=; b=I5uCPSnHGoM+WxncgqbTSMrIgj30PzHLEJExOU4TJ+BWZX/aPWfJbVEt6j/SEliLIV 08Vd8apqH1Qbq8+z57AiJ82n7fXg3E94SzweOELIZyMdYR9LbPFCGW3ogC4Cmy+Ld0PY KSr/yh3L0D+xyD3m7B0jiRpIsKI81g7q9yyiPM8QmIuRDIXHzuPVCRv3p70u+Ap07S9R wpVdRg/Fl/wHUl+slAB4xM+aIRo62aAsCgZCx+Q2Q7xJXtc/RGr1wZxoyl5ZGef7/JEw ofaLJmyQQgSPiVlex3w3VQ9BJT8GkIIGsvQ1LjAtxjTYRS2Ji2WiCK6IkQxxrXDL8AuH 1pHg== X-Forwarded-Encrypted: i=1; AKwUvBw2HcOcA7wL/aGaWuoRdXo5lmP1KH8hHYEG4Se8z6ElBR7KOszhj9EH4nw87QJbuBf+7OqMx73n24IHWnc=@vger.kernel.org X-Gm-Message-State: AFq9FYL8IM+6WGSBgfOYrT+2tN5hkAt/oFwuU77vnMz8rKmco00zng0T tOfIJaG6W7GbuvHFJgWTUekLYQO1AhRB2jP3zfI1kcuQtz3AnOAxm3ox8Wan4441aTeAt4RZXH7 l+JaThSIvNKsnUQ== X-Received: from wmbgz3.prod.google.com ([2002:a05:600c:8883:b0:4a1:b6f1:344b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a407:b0:49f:c5aa:9ef4 with SMTP id 5b1f17b1804b1-4a1c665f687mr21139025e9.8.1792094962450; Thu, 15 Oct 2026 13:09:22 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:41 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-11-smostafa@google.com> Subject: [PATCH v9 10/25] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Jean-Philippe Brucker , Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jean-Philippe Brucker Add the skeleton of the Arm SMMUv3 driver at EL2, and build the shared IDR/STE code and lib/hweight.o into the hypervisor. The kernel driver, added next, fills an array describing the SMMUv3s in the system and hands it to the hypervisor. At init the hypervisor takes ownership of the array and of the SMMU MMIO regions, so the host cannot change either after de-privilege. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 5 + drivers/iommu/arm/Kconfig | 12 +++ .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.h | 7 ++ .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 96 +++++++++++++++++++ .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 41 ++++++++ 5 files changed, 161 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 7cb79db9bc47..ee86e4b3436f 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -33,6 +33,11 @@ hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o hyp-obj-$(CONFIG_NVHE_EL2_TRACING) +=3D clock.o trace.o events.o hyp-obj-y +=3D $(lib-objs) =20 +HYP_SMMU_V3_DRV_PATH :=3D ../../../../../drivers/iommu/arm/arm-smmu-v3 + +hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3-hyp.o \ + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o ../../../../../lib/hweig= ht.o + # Path to simple_ring_buffer.c CFLAGS_trace.nvhe.o +=3D -I$(srctree)/kernel/trace/ =20 diff --git a/drivers/iommu/arm/Kconfig b/drivers/iommu/arm/Kconfig index 5fac08b89dee..82fbc729563e 100644 --- a/drivers/iommu/arm/Kconfig +++ b/drivers/iommu/arm/Kconfig @@ -128,6 +128,18 @@ config TEGRA241_CMDQV =20 Say Y here if your system is NVIDIA Tegra241 (Grace) or it has the same CMDQ-V extension. + +config ARM_SMMU_V3_PKVM + bool "ARM SMMUv3 support for protected Virtual Machines" + depends on KVM && ARM_SMMU_V3=3Dy && OF + help + Enable SMMUv3 driver in the KVM hypervisor, to protect VMs against + memory accesses from devices owned by the host. + This works alongside ARM_SMMU_V3 where the kernel driver manages the + SMMUv3 stage-1, and the hypervisor driver manages the stage-2 + transparently through trap and emulate. + + Say Y here if you intend to enable KVM in protected mode. endif =20 config QCOM_IOMMU diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h index 9f6949a60fe8..74b6c05b8f8f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.h @@ -20,6 +20,13 @@ typedef struct arm_smmu_device ARM_SMMU_OBJ; =20 #define arm_smmu_err(smmu, fmt, ...) dev_err((smmu)->dev, fmt, ##__VA_ARGS= __) #define arm_smmu_info(smmu, fmt, ...) dev_info((smmu)->dev, fmt, ##__VA_AR= GS__) +#else +#include "pkvm/arm-smmu-v3-hyp.h" + +typedef struct hyp_arm_smmu_v3_device ARM_SMMU_OBJ; + +#define arm_smmu_err(smmu, fmt, ...) +#define arm_smmu_info(smmu, fmt, ...) #endif =20 static_assert(__same_type(typeof_member(ARM_SMMU_OBJ, features), u32)); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c new file mode 100644 index 000000000000..ba209951039f --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -0,0 +1,96 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM hyp driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + * Copyright (C) 2025 Google LLC + */ +#include + +#include +#include + +#include "arm-smmu-v3-hyp.h" +#include "../arm-smmu-v3.h" +#include "../arm-smmu-v3-common-lib.h" + +size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; +struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; + +#define for_each_smmu(smmu) \ + for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ + (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ + (smmu)++) + +/* Put the device in a state that can be probed by the host driver. */ +static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(__pkvm_hyp_donate_host_mmio(hyp_phys_to_pfn(smmu->mmio_addr), + smmu->mmio_size >> PAGE_SHIFT, + (unsigned long)smmu->base)); + smmu->base =3D NULL; +} + +static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) +{ + unsigned long haddr; + int ret; + + if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) + return -EINVAL; + + ret =3D __pkvm_host_donate_hyp_mmio(hyp_phys_to_pfn(smmu->mmio_addr), + smmu->mmio_size >> PAGE_SHIFT, &haddr); + if (ret) + return ret; + + smmu->base =3D (void __iomem *)haddr; + + return 0; +} + +/* Called while the host is still trusted. */ +static int smmu_init(void) +{ + size_t smmu_arr_size =3D PAGE_ALIGN(sizeof(*kvm_hyp_arm_smmu_v3_smmus) * + kvm_hyp_arm_smmu_v3_count); + struct hyp_arm_smmu_v3_device *smmu; + u64 pfn, nr_pages; + int ret; + + if (!PAGE_ALIGNED(kvm_hyp_arm_smmu_v3_smmus)) + return -EINVAL; + + kvm_hyp_arm_smmu_v3_smmus =3D kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus); + pfn =3D hyp_virt_to_pfn(kvm_hyp_arm_smmu_v3_smmus); + nr_pages =3D smmu_arr_size >> PAGE_SHIFT; + + ret =3D __pkvm_host_donate_hyp(pfn, nr_pages); + if (ret) + return ret; + + for_each_smmu(smmu) { + ret =3D smmu_init_device(smmu); + if (ret) + goto out_reclaim_smmu; + } + + return 0; + +out_reclaim_smmu: + while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) + smmu_deinit_device(--smmu); + WARN_ON(__pkvm_hyp_donate_host(pfn, nr_pages)); + return ret; +} + +static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) +{ + return 0; +} + +/* Shared with the kernel driver in EL1 */ +struct pkvm_iommu_ops smmu_ops =3D { + .init =3D smmu_init, + .host_stage2_idmap =3D smmu_host_stage2_idmap, +}; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h new file mode 100644 index 000000000000..c2630397f57a --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -0,0 +1,41 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_ARM_SMMU_V3_HYP_H +#define __KVM_ARM_SMMU_V3_HYP_H + +#include + +/** + * struct hyp_arm_smmu_v3_device - SMMUv3 instance owned by the hypervisor + * @mmio_addr: Physical address of the SMMU registers + * @mmio_size: Size of the register region + * @base: Hypervisor virtual address of the registers + * @features: ARM_SMMU_FEAT_* bits + * @options: ARM_SMMU_OPT_* bits + * @oas: Output address size + * @pgsize_bitmap: Supported page sizes + * @sid_bits: Number of StreamID bits + * + * @mmio_addr, @mmio_size and ARM_SMMU_FEAT_COHERENCY are filled by the ho= st + * before de-privilege, the rest by the hypervisor driver. + */ +struct hyp_arm_smmu_v3_device { + phys_addr_t mmio_addr; + size_t mmio_size; + void __iomem *base; + u32 features; + u32 options; + unsigned long oas; + unsigned long pgsize_bitmap; + unsigned int sid_bits; +}; + +extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); +#define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count) + +extern struct hyp_arm_smmu_v3_device *kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smm= us); +#define kvm_hyp_arm_smmu_v3_smmus kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus) + +struct pkvm_iommu_ops; +extern struct pkvm_iommu_ops kvm_nvhe_sym(smmu_ops); + +#endif /* __KVM_ARM_SMMU_V3_HYP_H */ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96AB549E5FC for ; Thu, 15 Oct 2026 20:09:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094968; cv=none; b=cxwgrVwVx159zZmg1h8drYjcs7lePEKVAAhMJfuamnOiD5TWMaXgkl5bnHh60qJtiMxJ4G/cBGW1MM1u1wBXcvXCGWUSwPePaNqSvwpxzwZOxiUmMAT8qXsIUd02GYvKJ9elAz3jz0KTc0+eLLcE7rkhKQSyGWf6Uy5aTC2ELnk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094968; c=relaxed/simple; bh=rtZeA6wy9gVzOWw8LH32j4S+p5Q9x9daJYQGBlaMFIQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=i7ktDJaqy2mknz8HwxnxIDc/osyy98atSoG0JLxd4FahtkaWPiir7Vr2/+SStxb9BvaVlxSrPlO0bW8LjpfbREOAXuDzV9iCrNT+jvupqjMWbvpF0JgNp3o8aSI+Xcq6yNBq5LZORK5Jo96v0cJhODhXU2dd23tfkququN0q0PY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=K3aQhMml; arc=none smtp.client-ip=209.85.208.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="K3aQhMml" Received: by mail-ed1-f71.google.com with SMTP id 4fb4d7f45d1cf-6a178080182so6824127a12.2 for ; Thu, 15 Oct 2026 13:09:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094964; x=1792699764; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=5ln7kKe9smd/hPeerr96XakAp8DXZ4YEDMpg01bk0tQ=; b=K3aQhMmlgvxjyASRqSmEBDPKF4iJOIpahx44pJ8qUFCCAlbN/3JmYDfVkoWaXr+qQA jcVDZpPNwpK1ba5WICuROOXMiMqDqwgcS5eiR6YRMNTcRBiHyoJCNImvYkcXPJqu0puA f/EEv6c3OjfmwE/lGCpzcDjLc0zS12C4DnTDQ8cbjdY+jsTG+RwTZaHSPG3FySAVx/D3 F+paNCrQcX3yTmEwCshcAKyco+ak8THBxDVzZrmaw2fillfHeKtMX2tf0hoQttZFQtut 35mNDj7MTgvJKarr5fO3vBEEhJFM+7RTCJc7w/ZHsb7fJ7eD4RsTHLY3ceyH0EcDcjAX VcjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094964; x=1792699764; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=5ln7kKe9smd/hPeerr96XakAp8DXZ4YEDMpg01bk0tQ=; b=J/x802yD1ReCgxirrgv7NgOQ7qBF0u33ZxrW1P6pRGNsiTM1e70A5A2SLmjXjzPhvM HIBfvPjvEJ5XJvsfIHJeDqahLj4zQvyNT761l6xAr4TydMt6D3wLL2we6/tnXEJngqOF VemcLLFiB2rlqPi4KlZoh6DNnR/+GTOwylY14eTeavraUzErw8gYFQGLMb2fFuAWhPta fdJlYpiE9afMLthfObza7JeYTt/taFfs7NSLGCl3eVwqb56+LJdgecVFgnXknV9sPKsr o3i2+hxf+4LTEjR8RUJDZfGNpCiOHyD/i+MxbDqbaeZasRHXks4lAeXljWp6mqjtnMOd D2Mw== X-Forwarded-Encrypted: i=1; AKwUvBzrXIyo0FtW5pSf9lJGwkHdEV4NTb5b9MaSL03kIRAMsnbdAysgxN3x0tzzQUE0+B9gN+jDln1+lMwezTc=@vger.kernel.org X-Gm-Message-State: AFq9FYLMvZMeO1T9sHZwurBbzrV25GDHhIr6ZfwToPdW23olpMvoWRwi 0CMCDSPNMUbzRNrQWkxoLslmgtXPIXf3fjhXAReWzfBN5Wf1xA4Ywt+iCh3Ti+TzEviGzm10oJj jSe6xP2cHDiY2Jg== X-Received: from edcnu6-n2.prod.google.com ([2002:a05:6402:c506:20b0:6b1:c3e6:d08]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:2744:b0:6b1:ba86:5f48 with SMTP id 4fb4d7f45d1cf-6b38032f753mr1614433a12.36.1792094963468; Thu, 15 Oct 2026 13:09:23 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:42 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-12-smostafa@google.com> Subject: [PATCH v9 11/25] iommu/arm-smmu-v3-kvm: Add the kernel driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When KVM runs in protected mode with CONFIG_ARM_SMMU_V3_PKVM, the hypervisor owns the SMMUv3 hardware and presents emulated SMMUs to the host through trap and emulate. Add the host side of the driver. At subsys_initcall, before the hypervisor is initialised, it claims the "arm,smmu-v3" platform devices, records their MMIO regions in an array handed to the hypervisor, and registers the hypervisor ops with pKVM, requesting a page table pool the size of the host stage-2 one. After KVM initialises, it creates an auxiliary device for each SMMU. The SMMUv3 driver probes those through a small auxiliary driver that keeps using the platform device for the DMA API, IRQs and devres. The emulation complies with the architecture, so the rest of the driver is unchanged. Suggested-by: Jason Gunthorpe Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 1 + .../arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c | 183 ++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 +++++++ 3 files changed, 249 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index c9ce392e6d31..b782d1fda5fd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -4,5 +4,6 @@ arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o +arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D arm-smmu-v3-pkvm-init.o =20 obj-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) +=3D arm-smmu-v3-test.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c b/driver= s/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c new file mode 100644 index 000000000000..5c5f0ff455e3 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c @@ -0,0 +1,183 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM host driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + * Copyright (C) 2025 Google LLC + */ +#include +#include + +#include +#include +#include +#include + +#include "arm-smmu-v3.h" +#include "pkvm/arm-smmu-v3-hyp.h" + +static size_t kvm_arm_smmu_count; +static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; +static size_t kvm_arm_smmu_cur; + +static void __init kvm_arm_smmu_array_free(void) +{ + int order; + + order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); + free_pages((unsigned long)kvm_arm_smmu_array, order); +} + +static int __init kvm_arm_smmu_array_alloc(void) +{ + int smmu_order; + struct device_node *np; + + for_each_compatible_node(np, NULL, "arm,smmu-v3") + kvm_arm_smmu_count++; + + if (!kvm_arm_smmu_count) + return -ENODEV; + smmu_order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)= ); + kvm_arm_smmu_array =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, = smmu_order); + if (!kvm_arm_smmu_array) + return -ENOMEM; + return 0; +} + +static unsigned int __init smmu_hyp_pgt_pages(void) +{ + struct device_node *np =3D of_find_compatible_node(NULL, NULL, "arm,smmu-= v3"); + + if (!np) + return 0; + of_node_put(np); + + /* + * Minimum for mirroring the host stage-2 at page granularity. The pool + * itself is sized by kvm-arm.iommu_pgt_mem, which must also cover the + * stream tables. + */ + return host_s2_pgtable_pages(); +} + +static struct platform_driver smmuv3_nesting_driver; +static int __init smmuv3_nesting_probe(struct platform_device *pdev) +{ + struct hyp_arm_smmu_v3_device *smmu; + struct device *dev =3D &pdev->dev; + struct resource *res; + + /* Only device tree, ACPI not supported. */ + if (!dev->of_node) + return -EINVAL; + + if (WARN_ON(kvm_arm_smmu_cur >=3D kvm_arm_smmu_count)) + return -ENOSPC; + + res =3D platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (!res) + return -ENODEV; + + if (of_property_read_bool(dev->of_node, "cavium,cn9900-broken-page1-regsp= ace")) { + dev_err(dev, "cavium,cn9900-broken-page1-regspace is not supported\n"); + return -EINVAL; + } + + if (resource_size(res) < SZ_128K || + !PAGE_ALIGNED(res->start | resource_size(res))) { + dev_err(dev, "unsupported MMIO region %pr\n", res); + return -EINVAL; + } + + smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_cur]; + smmu->mmio_addr =3D res->start; + smmu->mmio_size =3D resource_size(res); + + if (of_dma_is_coherent(dev->of_node)) + smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; + + kvm_arm_smmu_cur++; + return 0; +} + +static int __init kvm_arm_smmu_v3_register(void) +{ + size_t nr_pages =3D smmu_hyp_pgt_pages(); + int ret; + + if (!is_protected_kvm_enabled() || !nr_pages) + return 0; + + ret =3D kvm_arm_smmu_array_alloc(); + if (ret) + goto out_err; + + ret =3D platform_driver_probe(&smmuv3_nesting_driver, smmuv3_nesting_prob= e); + if (ret) + goto out_free; + + ret =3D pkvm_iommu_register_driver(&kvm_nvhe_sym(smmu_ops), nr_pages); + if (ret) + goto out_unregister; + + /* + * These variables are stored in the nVHE image, and won't be accessible + * after KVM initialization. Ownership of kvm_arm_smmu_array will be + * transferred to the hypervisor as well. + */ + kvm_hyp_arm_smmu_v3_smmus =3D kvm_arm_smmu_array; + kvm_hyp_arm_smmu_v3_count =3D kvm_arm_smmu_cur; + return ret; + +out_unregister: + platform_driver_unregister(&smmuv3_nesting_driver); +out_free: + kvm_arm_smmu_array_free(); +out_err: + kvm_arm_smmu_count =3D 0; + kvm_arm_smmu_array =3D NULL; + return ret; +} + +static int __init smmu_create_aux_device(struct device *dev, void *data) +{ + static int dev_id; + struct auxiliary_device *auxdev; + + auxdev =3D __devm_auxiliary_device_create(dev, "protected_kvm", + "smmu_v3_emu", NULL, dev_id++); + if (!auxdev) + return -ENODEV; + return 0; +} + +static int __init kvm_arm_smmu_v3_post_init(void) +{ + if (!kvm_arm_smmu_count) + return 0; + + /* + * If KVM failed to initialise, the host stage-2 is never enabled and + * the SMMUs are driven natively through the same path. + */ + WARN_ON(driver_for_each_device(&smmuv3_nesting_driver.driver, NULL, + NULL, smmu_create_aux_device)); + + return 0; +} + +static const struct of_device_id smmuv3_nested_of_match[] =3D { + { .compatible =3D "arm,smmu-v3", }, + { }, +}; + +static struct platform_driver smmuv3_nesting_driver =3D { + .driver =3D { + .name =3D "smmuv3-nesting", + .of_match_table =3D smmuv3_nested_of_match, + .suppress_bind_attrs =3D true, + }, +}; +late_initcall(kvm_arm_smmu_v3_post_init); +subsys_initcall(kvm_arm_smmu_v3_register); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index c2ec33c763a0..66f38c54742b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -11,6 +11,7 @@ =20 #include #include +#include #include #include #include @@ -5264,6 +5265,70 @@ static struct platform_driver arm_smmu_driver =3D { module_driver(arm_smmu_driver, platform_driver_register, arm_smmu_driver_unregister); =20 +#ifdef CONFIG_ARM_SMMU_V3_PKVM +/* + * Two devices are involved: the aux device bound to this driver, and the + * physical platform device bound to the pKVM init driver, which only uses= it + * for discovery. This driver keeps using the platform device, as it carri= es + * the fwnode the IOMMU core and consumers link to, the DMA configuration = and + * the IRQs. + * + * So arm_smmu_device_probe() allocates its devm resources on the platform + * device, where an aux unbind would not free them. The devres group relea= ses + * them if the probe fails, so a deferred probe can retry. The platform de= vice + * is never unbound: both drivers set suppress_bind_attrs and + * ARM_SMMU_V3_PKVM requires ARM_SMMU_V3=3Dy. + */ +static int arm_smmu_device_probe_emu(struct auxiliary_device *auxdev, + const struct auxiliary_device_id *id) +{ + struct device *parent =3D auxdev->dev.parent; + void *group; + int ret; + + dev_info(&auxdev->dev, "Probing from %s\n", dev_name(parent)); + group =3D devres_open_group(parent, NULL, GFP_KERNEL); + if (!group) + return -ENOMEM; + + ret =3D arm_smmu_device_probe(to_platform_device(parent)); + if (ret) + devres_release_group(parent, group); + else + devres_close_group(parent, group); + return ret; +} + +static void arm_smmu_device_remove_emu(struct auxiliary_device *auxdev) +{ + arm_smmu_device_remove(to_platform_device(auxdev->dev.parent)); +} + +static void arm_smmu_device_shutdown_emu(struct auxiliary_device *auxdev) +{ + arm_smmu_device_shutdown(to_platform_device(auxdev->dev.parent)); +} + +static const struct auxiliary_device_id arm_smmu_aux_table[] =3D { + { .name =3D "protected_kvm.smmu_v3_emu" }, + { }, +}; +MODULE_DEVICE_TABLE(auxiliary, arm_smmu_aux_table); + +static struct auxiliary_driver arm_smmu_driver_emu =3D { + .driver =3D { + .suppress_bind_attrs =3D true, + }, + .name =3D "arm-smmu-v3-emu", + .id_table =3D arm_smmu_aux_table, + .probe =3D arm_smmu_device_probe_emu, + .remove =3D arm_smmu_device_remove_emu, + .shutdown =3D arm_smmu_device_shutdown_emu, +}; + +module_auxiliary_driver(arm_smmu_driver_emu); +#endif + MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); MODULE_ALIAS("platform:arm-smmu-v3"); --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ed1-f71.google.com (mail-ed1-f71.google.com [209.85.208.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E9434A6892 for ; Thu, 15 Oct 2026 20:09:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094969; cv=none; b=lUVWJA/0JGrAqAd2LfCUXtLfrxj7mWYbl5gEwPkzTfOWCpe6MRBgiJalBJ4LFnxJAOOHqigDUWU+Tjm/id/OWqwa3Tmu+1qF88VdTjZWtmAK72vlpnwVqsMpfFx1KWaECJMi/XVSrDOL8DVHLB5oi1JkJdL4pODRH/xP0W4A/a0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094969; c=relaxed/simple; bh=v96PErYEvvh2sXqT5nhrigxoy5ACRjUIgyH+xbGOyoU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=APqxl2hmTDoukR3SwNVlspSNuzhYoGHnqenanN+yXlMUTNs/IJHBsG9ZjUp0m8OKMtXtxAioZ+amYUGB5JafmQvY4LS68mA7sLMp2GxkAv6zcozuB3xSFw1Gr0LpjZrF/GGzRnIAmGPoEDWpR8RyhREoPO9q7ou6xJ/ExkQoK9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=suHdEnS0; arc=none smtp.client-ip=209.85.208.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="suHdEnS0" Received: by mail-ed1-f71.google.com with SMTP id 4fb4d7f45d1cf-6b1d6c10725so4994461a12.3 for ; Thu, 15 Oct 2026 13:09:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094965; x=1792699765; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=20qKmJOpxWxZsYx6berl9FrHJn60ekVP6T05FryarsM=; b=suHdEnS0yrBvXeY7qOneV4ApVJZnO9jXDkxXBKcHpQ9UR4wVc0lYhdqkXbIR6XVQnE LkN2s/TRofDZB8oCpmThQ3x9mORcVRv7Kncs6JuD2iSshc86rEmfHe5MFEnr/GYOqh4R 9jmEVaWkwY/t7RflKMhLuiOz/upYAY8mm/eOQfoWQmwgga1WFeqEUc4XwOUEOIZb4Fw8 1XVjUrHtXugxUlozSIV9iFtTo8E/OdOkhukoUg0vQpU332cQR/qFOpwJFbNJ+k58OEbc eJ8G1yV9Elo6OSacl+DqPV3N9rhipOtPbj+TyUGyRWHIHfhZXGZriT/P/XhOoCXloWBx CbGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094965; x=1792699765; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=20qKmJOpxWxZsYx6berl9FrHJn60ekVP6T05FryarsM=; b=zOjmTwjM4iO2nK0Xjr2j555W/kwr+e4tMTU6l/lObZ28/yIRvnf2lJeYhKNIXVCm1s 6U8KyTcmgQPOR01CJm6NcEW6V2FWbG7Ylef77ozYdxfiI2hPwUDnP8LWV8uS/uADqsov G7wPHny1rdcczVZ4xkbH+zQ9lYqtAE6CcXpxuTJ054VxdZfkyBQULMHCgmcB2yNCzVm9 s3AoM43lltTe/YF51B1Rp/dOGRVb0+Way7T+MocMI/iErVckyiHDhvLclqLfFEvPM1YY h1DnrNKZNvC91UaHAr/BMkpU1m/2emy0l6vq+ZM7PldPU6Icr1aKiPr8s2UPFBGS6YkI 9AoA== X-Forwarded-Encrypted: i=1; AKwUvBySE8j6sYq3cGl0RQN4xuD/yyvsTpa/0wsDAlUSc1YRy+3zUEnaoaVjfhMdfPhx2J1q6Xmb3+NzrixVpPg=@vger.kernel.org X-Gm-Message-State: AFq9FYLb00kBbONWCzrigpWOtKiUk3kn61N/qqtFebCogDdlilzCiic5 CetXTMdd5Zubaaj3QSydP8PUE5ejC4RDkX2EAglV+OTNYuxgU/HMf0HX0Gt9f4N5ys7PyAlMvtZ Ph+/CWcPk0xKboA== X-Received: from edai29.prod.google.com ([2002:a05:6402:f1d:b0:6b1:d940:d8b8]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:4554:b0:6b1:ced2:dfd4 with SMTP id 4fb4d7f45d1cf-6b1ced2e477mr7043030a12.21.1792094964923; Thu, 15 Oct 2026 13:09:24 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:43 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-13-smostafa@google.com> Subject: [PATCH v9 12/25] iommu/arm-smmu-v3-kvm: Probe SMMU HW From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Probe the SMMU features from the IDR registers with the functions shared with the kernel driver. SubstreamID bits and queue sizes are not needed here: CDs are not shadowed and the command queue is set up later. The hypervisor requires two things the kernel driver does not: - Coherent table walks, reported by both firmware and IDR0.COHACC, as it does no cache maintenance on its tables. - Nested translation, as the host keeps stage-1 and the hypervisor adds stage-2. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 46 ++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index ba209951039f..2b4a0f961a66 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -31,6 +31,47 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_de= vice *smmu) smmu->base =3D NULL; } =20 +/* + * Mini-probe and validation for the hypervisor. + */ +static int smmu_probe(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 reg; + int ret; + + /* Not supported yet. */ + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + return -ENXIO; + + ret =3D arm_smmu_device_idr0_probe(smmu, true, ®); + if (ret) + return ret; + + /* + * Core code clears FEAT_ATS with !CONFIG_PCI_ATS, but the hypervisor + * must know if it is actually supported for CR0 emulation. + */ + if (reg & IDR0_ATS) + smmu->features |=3D ARM_SMMU_FEAT_ATS; + + /* Firmware says coherent, the hardware must agree. */ + if (!(reg & IDR0_COHACC)) + return -ENXIO; + + ret =3D arm_smmu_device_idr1_probe(smmu, ®); + if (ret) + return ret; + + arm_smmu_device_idr3_probe(smmu); + arm_smmu_device_idr5_probe(smmu); + arm_smmu_device_iidr_probe(smmu); + + if (!(smmu->features & ARM_SMMU_FEAT_NESTING)) + return -ENXIO; + + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -45,8 +86,11 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devic= e *smmu) return ret; =20 smmu->base =3D (void __iomem *)haddr; + ret =3D smmu_probe(smmu); + if (ret) + smmu_deinit_device(smmu); =20 - return 0; + return ret; } =20 /* Called while the host is still trusted. */ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E914548562 for ; Thu, 15 Oct 2026 20:09:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094971; cv=none; b=VvE7Ouh70NQdoO21tvlMHKAayuDR43+JmFEAAriLyvimz6EW/+bJZhpSwuq5d4tJsvnJQQvmeRIs39liq33rZm1i/J3p2sYCrhrh3FN6sM1AFtVZWZYGYVQb3qiWQJ4ajUzmzJURY+A8hSwzaHv8tLha2toDJXm6ZfNXpQDQNzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094971; c=relaxed/simple; bh=hMNpCYihSvV4R6sPhQV6qnrtxA6hcUizJETpMZs37M0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gAmfqu/mUKl1xE1IDX8HxuAOVH+pZW1U7aDEktg2+bh83b/qKYGn3yuPQBrYJvnjbDqYKk6o7WWSKZzvDg3xFsBV15SgDY7NgLlQ620lqGaxuaXlEhjQ8a5HHgl1vSmW7tBs5K+JtsrVwxiZ+Bc9bwhwdt84ZPN/H3SfvrjbSqE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pY87Jo4q; arc=none smtp.client-ip=209.85.218.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pY87Jo4q" Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-c3f88d2ff28so81592766b.0 for ; Thu, 15 Oct 2026 13:09:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094966; x=1792699766; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=3u/Wwfq4O359uWiEJsBsS49XBNpLbBfsGX6z5I3CMkU=; b=pY87Jo4qqFjxZd+axX2FoIOJM297mh+KP2oUOvoc8Qo9HiDtN1hsp+5n1R5qUU9rMI HSzMzO3EIH8LJwQGRG3x9Xhyf5e6ZW2v6Og2fQ+Sc1q/nGn8cpT9AafD8puOsOfmQdq9 MJr5uaS4Q+MKc5/6j5qonEwbCxx1DsR4QQqCxkK864hHUhX0WDlGILnQVijEemB9qXzp gZaTeKNnmS4+WjnlxYBanjuo9nUfzqJzn9ewiSyUICEGtnON7zKwWTUkSlRikGiq9Ifd 4Y/WpffwhZtCnQGu8+oKJV8WrexDQ3Uh4dTmK4lgRtUYU/XPgPbDFfiDqnFeuIAIiGRP PvoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094966; x=1792699766; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=3u/Wwfq4O359uWiEJsBsS49XBNpLbBfsGX6z5I3CMkU=; b=y4lz3pNcmCxn5hIuEGRlgACN6v6H9rge+QFbx3OM5b62w8AhfsnLqgsDApSX6kEv0/ R8Q+JYt6OrE3E8lBxMzYUvQcsxe+kLsukC2rie1StSQpLthS9N2EJhhnaRcIsDWJr5fF VVKCS5k4oM7m/E+gm23eCyvsxhxSqBLDEeK779E4m1emy8R38hh8hPmOALyc+N7hMf9N 3jINAWpgkG1pPQ1nGW5xbjvarfhYH5yaXiIY80kOwTOKpP/V8THgX9/CdUKJIb6BWeJA 1rIckyjniqU8wUGYAxjLY/VVWsfedjnmJ2MZhYU2HK0C+KSbTk5IaqMbph2Cvl3aAIck yxPA== X-Forwarded-Encrypted: i=1; AKwUvBxye8qQBfLeGUHFcvgRM8ygV4vhta4DaNsVxKoBYUMFbMVk5hSeirotS9RCY5LwI3Nj1A3PnK/0j5ynb/Q=@vger.kernel.org X-Gm-Message-State: AFq9FYK0B0WVAZpzLNgUYMEZ7Q8QW8hqhw3C5g4K/7q0rjlvghr7y6tn wl72QM2szkXYjvFmmDSOw6WNMmnZEC1CiIJzxT3YNH9eD2xfouYjATl+fleWeM+Assx7BOmdc+w uIcCznqBI2CyLYQ== X-Received: from edfz6.prod.google.com ([2002:a05:6402:a406:b0:6b1:b15e:791f]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:9720:b0:c26:337e:7e05 with SMTP id a640c23a62f3a-c31a9a29d50mr1794237266b.4.1792094966321; Thu, 15 Oct 2026 13:09:26 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:44 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-14-smostafa@google.com> Subject: [PATCH v9 13/25] iommu/arm-smmu-v3-kvm: Add MMIO emulation From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add data abort handler for the SMMUs, at the moment most registers are just passthrough. In the next patches CMDQ/STE emulation will be added which inserts logic to some register access. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 169 ++++++++++++++++++ .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 18 ++ 2 files changed, 187 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 2b4a0f961a66..d95b459a796a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -9,6 +9,7 @@ =20 #include #include +#include =20 #include "arm-smmu-v3-hyp.h" #include "../arm-smmu-v3.h" @@ -80,6 +81,9 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_device= *smmu) if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) return -EINVAL; =20 + hyp_spin_lock_init(&smmu->lock); + hyp_spin_lock_init(&smmu->hw_lock); + ret =3D __pkvm_host_donate_hyp_mmio(hyp_phys_to_pfn(smmu->mmio_addr), smmu->mmio_size >> PAGE_SHIFT, &haddr); if (ret) @@ -128,6 +132,170 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, + struct user_pt_regs *regs, + u64 esr, u32 off) +{ + bool is_write =3D esr & ESR_ELx_WNR; + unsigned int len =3D BIT((esr & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); + int rd =3D (esr & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; + const u64 read_write =3D -1ULL; + const u64 no_access =3D 0; + u64 mask =3D no_access; + const u64 read_only =3D is_write ? no_access : read_write; + bool is_xzr =3D (rd =3D=3D 31); + u64 val =3D is_xzr ? 0 : regs->regs[rd]; + + switch (off) { + case ARM_SMMU_IDR0: + if (len !=3D sizeof(u32)) + break; + /* + * Clear stage-2 support, hide MSI to avoid write back to cmdq, + * and hide ATS as it is not emulated or handled in TLB invalidation. + */ + mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP | ID= R0_ATS); + break; + /* Passthrough the register access for bisectability, handled later */ + case ARM_SMMU_CMDQ_BASE: + case ARM_SMMU_CMDQ_PROD: + case ARM_SMMU_CMDQ_CONS: + case ARM_SMMU_STRTAB_BASE: + case ARM_SMMU_STRTAB_BASE_CFG: + case ARM_SMMU_GBPA: + mask =3D read_write; + break; + case ARM_SMMU_CR0: + if (len !=3D sizeof(u32)) + break; + + if (is_write) { + /* Enforce ATS checks. */ + if (smmu->features & ARM_SMMU_FEAT_ATS) + val |=3D CR0_ATSCHK; + mask =3D read_write; + } else { + mask =3D read_write & ~CR0_ATSCHK; + } + break; + case ARM_SMMU_CR0ACK: + if (len !=3D sizeof(u32)) + break; + mask =3D read_only & ~CR0_ATSCHK; + break; + case ARM_SMMU_CR1: + if (len !=3D sizeof(u32)) + break; + /* Don't mess with shareability/cacheability. */ + if (is_write && + (val !=3D (FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB)))) + break; + + mask =3D read_write; + break; + /* Allowed 32 bit registers. */ + case ARM_SMMU_EVTQ_IRQ_CFG1: + case ARM_SMMU_EVTQ_IRQ_CFG2: + case ARM_SMMU_GERROR_IRQ_CFG1: + case ARM_SMMU_GERROR_IRQ_CFG2: + case ARM_SMMU_PRIQ_IRQ_CFG1: + case ARM_SMMU_PRIQ_IRQ_CFG2: + /* These are RES0 as MSI support is hidden. */ + val =3D 0; + if (!is_write) + goto out_update_regs; + fallthrough; + case ARM_SMMU_EVTQ_PROD + SZ_64K: + case ARM_SMMU_EVTQ_CONS + SZ_64K: + case ARM_SMMU_PRIQ_PROD + SZ_64K: + case ARM_SMMU_PRIQ_CONS + SZ_64K: + case ARM_SMMU_GERRORN: + case ARM_SMMU_IRQ_CTRL: + case ARM_SMMU_CR2: + if (len !=3D sizeof(u32)) + break; + mask =3D read_write; + break; + /* Allowed 64 bit registers. */ + case ARM_SMMU_EVTQ_IRQ_CFG0: + case ARM_SMMU_PRIQ_IRQ_CFG0: + case ARM_SMMU_GERROR_IRQ_CFG0: + /* These are RES0 as MSI support is hidden. */ + val =3D 0; + if (!is_write) + goto out_update_regs; + fallthrough; + case ARM_SMMU_EVTQ_BASE: + case ARM_SMMU_PRIQ_BASE: + if (len !=3D sizeof(u64)) + break; + mask =3D read_write; + break; + /* Allowed RO 32 bit registers. */ + case ARM_SMMU_IIDR: + case ARM_SMMU_IDR5: + case ARM_SMMU_IDR3: + case ARM_SMMU_IDR1: + case ARM_SMMU_IRQ_CTRLACK: + case ARM_SMMU_GERROR: + if (len !=3D sizeof(u32)) + break; + mask =3D read_only; + break; + } + + if (WARN_ON(!mask)) + goto out_ret; + + hyp_spin_lock(&smmu->hw_lock); + if (is_write) { + if (len =3D=3D sizeof(u64)) + writeq_relaxed(val & mask, smmu->base + off); + else + writel_relaxed(val & mask, smmu->base + off); + + hyp_spin_unlock(&smmu->hw_lock); + return true; + } + + if (len =3D=3D sizeof(u64)) + val =3D readq_relaxed(smmu->base + off) & mask; + else + val =3D readl_relaxed(smmu->base + off) & mask; + hyp_spin_unlock(&smmu->hw_lock); +out_update_regs: + /* + * Device might be read sensitive, so do it but ignore writing + * back for xzr. + */ + if (!is_xzr) + regs->regs[rd] =3D val; + +out_ret: + return true; +} + +static bool smmu_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 addr) +{ + struct hyp_arm_smmu_v3_device *smmu; + bool ret; + + for_each_smmu(smmu) { + if (addr < smmu->mmio_addr || addr >=3D smmu->mmio_addr + smmu->mmio_siz= e) + continue; + hyp_spin_lock(&smmu->lock); + ret =3D smmu_dabt_device(smmu, regs, esr, addr - smmu->mmio_addr); + hyp_spin_unlock(&smmu->lock); + return ret; + } + return false; +} + static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) { return 0; @@ -137,4 +305,5 @@ static int smmu_host_stage2_idmap(phys_addr_t start, ph= ys_addr_t end, int prot) struct pkvm_iommu_ops smmu_ops =3D { .init =3D smmu_init, .host_stage2_idmap =3D smmu_host_stage2_idmap, + .dabt_handler =3D smmu_dabt_handler, }; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h index c2630397f57a..afc3ca791583 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -4,6 +4,10 @@ =20 #include =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +#include +#endif + /** * struct hyp_arm_smmu_v3_device - SMMUv3 instance owned by the hypervisor * @mmio_addr: Physical address of the SMMU registers @@ -14,6 +18,9 @@ * @oas: Output address size * @pgsize_bitmap: Supported page sizes * @sid_bits: Number of StreamID bits + * @lock: Lock to protect SMMU emulation + * @hw_lock: Lock to protect SMMU HW (e.g. CMDQ) + * Order smmu.lock =3D> host_mmu.lock =3D> smmu.hw_lock * * @mmio_addr, @mmio_size and ARM_SMMU_FEAT_COHERENCY are filled by the ho= st * before de-privilege, the rest by the hypervisor driver. @@ -27,8 +34,19 @@ struct hyp_arm_smmu_v3_device { unsigned long oas; unsigned long pgsize_bitmap; unsigned int sid_bits; +#ifdef __KVM_NVHE_HYPERVISOR__ + hyp_spinlock_t lock; + hyp_spinlock_t hw_lock; +#else + u32 lock; + u32 hw_lock; +#endif }; =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +static_assert(sizeof(hyp_spinlock_t) =3D=3D sizeof(u32)); +#endif + extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); #define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count) =20 --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 941B249E5DA for ; Thu, 15 Oct 2026 20:09:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094973; cv=none; b=NueemYQEYxp+NA6ACXc+gS3JXko8sBjlwDCHIKm94Ghgmf+AXlVLtadUwMSAmrRNm5wM187NOCPVpkJY9IOAsB3Aw8wZJv8YK5Osmys6opFS3rja3Gjg/p/x+Vi4DA3aIy960t/xsWUp/RfArOro3x0163SdkVV9fjhn7CkR9Wc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094973; c=relaxed/simple; bh=wzECFTgv0gbRiroiT6ZEgspnN/KjS2YQoxW9c4Mnl1s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=K3w0Mpg81C6Y9B7ZRRnA9/hSmtkkbIK01yGvq124I0W4vaXwdfZx4Ti4uY/2UpDkoJ3/ugn/05SB8IvrPLLTH2vh9vGfZGdDMecz4QZb7KClyewzXznsMZk8cyhHGXKkDtSNxmD0pCrlQUSaje87S6ghpJRgoViBKhtUfGVIrVU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=d2kXgC+/; arc=none smtp.client-ip=209.85.128.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="d2kXgC+/" Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4a1612c4b80so58662895e9.0 for ; Thu, 15 Oct 2026 13:09:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094968; x=1792699768; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Tp66Frxalg0sY1/x9PE5w46OtuySfbUHfMRz44KKCn8=; b=d2kXgC+/z7kNXtA5PmqFtDND9iDgv4dscFyD/Glwv8T60iUQcF7wC/oafe7PrHr9GE 9qLxvRP2uTdkn6A3gGasYazOppGn9tU8l1MWkVUk32hg7f9P0gUusija4Z3y4TuflncA +i3XF0OuPtAFacj65OKSViApSy4PvlOD0HBpTHAecMWYYxsapWnnmi3CI7fHgXA43EhH 9eWCaHIy2lCWyWAiMoF5lLgHz+KA0KQO6Q+c0YYnLjkZL9e14JpBnZHt82iEolvxZS1Y WatWAtc6+LhnJQ1yb+WqJ3M18XAIRNtpNZGPUc9CZRy8zot70kQA6UHZsPnKAQ4q8G5x lmHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094968; x=1792699768; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Tp66Frxalg0sY1/x9PE5w46OtuySfbUHfMRz44KKCn8=; b=Qh5JvfdhT5ul9SksMV9IBZhsIOaBQJT69XNgZ7vgbC40q4qGm9NeU+OeD8GNgSPDpV jGyk7xTvNV0KageFBYcieuMdybwIB7ruVUzC/pCffYR7xruvQtfptV9mOKR9Nw76AwqN 0PCUZokaFsV6P9uEhfqldILm+LVwGP1nEm3wjC+ryuh6PnBVcRVa+RdbA26PRsiCNnyt hb0UJF9W6M32acWA1bBkRGEFSdcUKgz1R+FMERWuO+QhRn4gSarF6z2Vz6PWv+FVcZws sYMK7a7U163ejTTDSaefG3e3qcc5qNw6vpKNtHamIbaW36AJhZZTkGk/KEfAzzGfyxf1 Av2Q== X-Forwarded-Encrypted: i=1; AKwUvBwuHUXXruH2XL0r3ntsRJipHKjNyHOAXm87cZ2r5XPtpG6yJxpHLrwRiKcNYmPPghVRZg885UqblBivijE=@vger.kernel.org X-Gm-Message-State: AFq9FYLPhTjO81y3YqYdHSYhNfmNP28XaPnEIL81+OVL2ADd9VYZ93Jv SHkJQz7YzOgjHnEUF4HZUNAPCrzIzuNdz0KpcX8/yYb5ymWZIRvSY6DM5VhN1jZHuevGKPmdf0q J8P86lJa3pN5UBw== X-Received: from wmbz11.prod.google.com ([2002:a05:600c:c08b:b0:4a1:88e2:2dde]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4689:b0:4a0:1cfd:321 with SMTP id 5b1f17b1804b1-4a18e4a9b31mr433958165e9.12.1792094967596; Thu, 15 Oct 2026 13:09:27 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:45 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-15-smostafa@google.com> Subject: [PATCH v9 14/25] iommu/arm-smmu-v3-kvm: Shadow the command queue From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At boot, allocate a command queue per SMMU which is used as a shadow by the hypervisor. The command queue size is 64K which is more than enough, as the hypervisor would consume all the entries per a command queue prod write, which means it can handle up to 4096 at a time. Then, the host command queue needs to be pinned in a shared state, so it can't be donated to VMs, and avoid tricking the hypervisor into accessing them. This is done each time the command queue is enabled, and undone each time the command queue is disabled. The hypervisor won't access the host command queue when it is disabled from the host. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c | 25 ++++ .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 127 +++++++++++++++++- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 11 +- 3 files changed, 161 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c b/driver= s/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c index 5c5f0ff455e3..d7f2e73add7f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c @@ -16,6 +16,8 @@ #include "arm-smmu-v3.h" #include "pkvm/arm-smmu-v3-hyp.h" =20 +#define SMMU_KVM_CMDQ_ORDER get_order(MAX(SZ_64K, PAGE_SIZE)) + static size_t kvm_arm_smmu_count; static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; static size_t kvm_arm_smmu_cur; @@ -23,6 +25,15 @@ static size_t kvm_arm_smmu_cur; static void __init kvm_arm_smmu_array_free(void) { int order; + int i; + + for (i =3D 0 ; i < kvm_arm_smmu_cur ; ++i) { + struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[i]; + + if (smmu->cmdq.base_dma) + free_pages((unsigned long)phys_to_virt(smmu->cmdq.base_dma), + SMMU_KVM_CMDQ_ORDER); + } =20 order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); free_pages((unsigned long)kvm_arm_smmu_array, order); @@ -67,6 +78,7 @@ static int __init smmuv3_nesting_probe(struct platform_de= vice *pdev) struct hyp_arm_smmu_v3_device *smmu; struct device *dev =3D &pdev->dev; struct resource *res; + void *cmdq_base; =20 /* Only device tree, ACPI not supported. */ if (!dev->of_node) @@ -94,6 +106,19 @@ static int __init smmuv3_nesting_probe(struct platform_= device *pdev) smmu->mmio_addr =3D res->start; smmu->mmio_size =3D resource_size(res); =20 + /* + * Allocate the shadow command queue, it doesn't have to be the same + * size as the host. + * Only populate base_dma and llq.max_n_shift, the hypervisor will init + * the rest. + */ + cmdq_base =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_= CMDQ_ORDER); + if (!cmdq_base) + return -ENOMEM; + + smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); + smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; + if (of_dma_is_coherent(dev->of_node)) smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index d95b459a796a..02e7c6175832 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -23,12 +23,69 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmu= s; (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ (smmu)++) =20 +#define cmdq_size(cmdq) ((1UL << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWO= RDS * 8) + +static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_CMDQEN, smmu->cr0); +} + +/* + * Host structures the hypervisor reads (CMDQ, STE) are shared and pinned: + * - Prevent the host from donating this memory. + * - Map them in the hypervisor s1 page tables. + */ +static int smmu_share_pages(phys_addr_t addr, size_t size) +{ + size_t nr_pages =3D PAGE_ALIGN(size + (addr & ~PAGE_MASK)) >> PAGE_SHIFT; + phys_addr_t base =3D addr & PAGE_MASK; + int i, ret; + + for (i =3D 0; i < nr_pages; ++i) { + if (__pkvm_host_share_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT)) { + while (i--) + __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + return -EPERM; + } + } + + ret =3D hyp_pin_shared_mem(hyp_phys_to_virt(base), + hyp_phys_to_virt(base + nr_pages * PAGE_SIZE)); + if (ret) { + for (i =3D 0; i < nr_pages; ++i) + __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + } + + return ret; +} + +static int smmu_unshare_pages(phys_addr_t addr, size_t size) +{ + size_t nr_pages =3D PAGE_ALIGN(size + (addr & ~PAGE_MASK)) >> PAGE_SHIFT; + phys_addr_t base =3D addr & PAGE_MASK; + int i, ret; + + hyp_unpin_shared_mem(hyp_phys_to_virt(base), + hyp_phys_to_virt(base + nr_pages * PAGE_SIZE)); + + for (i =3D 0; i < nr_pages; ++i) { + ret =3D __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + if (ret) + return ret; + } + + return 0; +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { WARN_ON(__pkvm_hyp_donate_host_mmio(hyp_phys_to_pfn(smmu->mmio_addr), smmu->mmio_size >> PAGE_SHIFT, (unsigned long)smmu->base)); + if (smmu->cmdq.base) + WARN_ON(__pkvm_hyp_donate_host(smmu->cmdq.base_dma >> PAGE_SHIFT, + PAGE_ALIGN(cmdq_size(&smmu->cmdq)) >> PAGE_SHIFT)); smmu->base =3D NULL; } =20 @@ -62,6 +119,7 @@ static int smmu_probe(struct hyp_arm_smmu_v3_device *smm= u) ret =3D arm_smmu_device_idr1_probe(smmu, ®); if (ret) return ret; + smmu->cmdq_max_shift =3D FIELD_GET(IDR1_CMDQS, reg); =20 arm_smmu_device_idr3_probe(smmu); arm_smmu_device_idr5_probe(smmu); @@ -73,6 +131,33 @@ static int smmu_probe(struct hyp_arm_smmu_v3_device *sm= mu) return 0; } =20 +/* + * The kernel part of the driver allocates and zeroes the shadow cmdq. Take + * ownership of it and program it into the hardware. + */ +static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_nr_pages; + int ret; + + smmu->cmdq.llq.max_n_shift =3D min(smmu->cmdq.llq.max_n_shift, smmu->cmdq= _max_shift); + cmdq_nr_pages =3D PAGE_ALIGN(cmdq_size(&smmu->cmdq)) >> PAGE_SHIFT; + ret =3D __pkvm_host_donate_hyp(smmu->cmdq.base_dma >> PAGE_SHIFT, cmdq_nr= _pages); + if (ret) + return ret; + + smmu->cmdq.base =3D hyp_phys_to_virt(smmu->cmdq.base_dma); + smmu->cmdq.prod_reg =3D smmu->base + ARM_SMMU_CMDQ_PROD; + smmu->cmdq.cons_reg =3D smmu->base + ARM_SMMU_CMDQ_CONS; + smmu->cmdq.q_base =3D smmu->cmdq.base_dma | + FIELD_PREP(Q_BASE_LOG2SIZE, smmu->cmdq.llq.max_n_shift); + smmu->cmdq.ent_dwords =3D CMDQ_ENT_DWORDS; + writel_relaxed(0, smmu->cmdq.prod_reg); + writel_relaxed(0, smmu->cmdq.cons_reg); + writeq_relaxed(smmu->cmdq.q_base, smmu->base + ARM_SMMU_CMDQ_BASE); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -92,8 +177,16 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) smmu->base =3D (void __iomem *)haddr; ret =3D smmu_probe(smmu); if (ret) - smmu_deinit_device(smmu); + goto out_deinit; =20 + ret =3D smmu_init_cmdq(smmu); + if (ret) + goto out_deinit; + + return 0; + +out_deinit: + smmu_deinit_device(smmu); return ret; } =20 @@ -132,6 +225,23 @@ static int smmu_init(void) return ret; } =20 +static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; + + smmu->cmdq_host.llq.max_n_shift =3D min(shift, smmu->cmdq_max_shift); + smmu->cmdq_host.base_dma =3D smmu->cmdq_host.q_base & Q_BASE_ADDR_MASK; + smmu->cmdq_host.base_dma &=3D ~(cmdq_size(&smmu->cmdq_host) - 1); + WARN_ON(smmu_share_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + +static void smmu_emulate_cmdq_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(smmu_unshare_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -158,6 +268,14 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, break; /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_CMDQ_BASE: + if (is_write) { + /* Not allowed by the architecture */ + if (is_cmdq_enabled(smmu)) + break; + smmu->cmdq_host.q_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_CMDQ_PROD: case ARM_SMMU_CMDQ_CONS: case ARM_SMMU_STRTAB_BASE: @@ -170,9 +288,16 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, break; =20 if (is_write) { + bool last_cmdq_en =3D is_cmdq_enabled(smmu); + /* Enforce ATS checks. */ if (smmu->features & ARM_SMMU_FEAT_ATS) val |=3D CR0_ATSCHK; + smmu->cr0 =3D val; + if (!last_cmdq_en && is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_enable(smmu); + else if (last_cmdq_en && !is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_disable(smmu); mask =3D read_write; } else { mask =3D read_write & ~CR0_ATSCHK; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h index afc3ca791583..3903ae358555 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -8,6 +8,8 @@ #include #endif =20 +#include "../arm-smmu-v3.h" + /** * struct hyp_arm_smmu_v3_device - SMMUv3 instance owned by the hypervisor * @mmio_addr: Physical address of the SMMU registers @@ -21,7 +23,10 @@ * @lock: Lock to protect SMMU emulation * @hw_lock: Lock to protect SMMU HW (e.g. CMDQ) * Order smmu.lock =3D> host_mmu.lock =3D> smmu.hw_lock - * + * @cmdq: CMDQ as observed by HW + * @cmdq_host: Host view of the CMDQ, only q_base, base_dma and llq used + * @cmdq_max_shift: Max shift for the CMDQ probed from HW + * @cr0: Last value of CR0 * @mmio_addr, @mmio_size and ARM_SMMU_FEAT_COHERENCY are filled by the ho= st * before de-privilege, the rest by the hypervisor driver. */ @@ -41,6 +46,10 @@ struct hyp_arm_smmu_v3_device { u32 lock; u32 hw_lock; #endif + struct arm_smmu_queue cmdq; + struct arm_smmu_queue cmdq_host; + u32 cmdq_max_shift; + u32 cr0; }; =20 #ifdef __KVM_NVHE_HYPERVISOR__ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6D4D58B6B4 for ; Thu, 15 Oct 2026 20:09:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094973; cv=none; b=BX30nQ2V6aIcTDBH9XlcLTlwFiRBV1MKUSkhzR8VmY1MbnpXCGrXDmq+m/FwrCNDrXCVSKeDRiEoV4MPIQtmKc6MoS9vysmovcw6/neUpRKTIdzUAgwAJLAiUP2EiGvMm5SKYevX6Iu0hqq5yVe7BUnrXIHTD8NCH55rW+/iXDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094973; c=relaxed/simple; bh=NcIQJYvuGHWznbFTUVkvtGXTi7lEtb2z7ScpZwFVY3s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=D7rCOXqC04o1I1FBEvEzP2FU4ekHh3UPa42BBccFhiKgBMxmT2r1i35KOC44HCic/6v+HP2wJSU/C+Q6aKMvfsP4y6AOtnRDiFFbIi1nPP9Crml/Z4dBgvVN2N6PmNZp3UhpgraRN5Jc6iQAAnVCDmVmC61YzvLZWqSHK4uBuIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FJBeMtO/; arc=none smtp.client-ip=209.85.128.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FJBeMtO/" Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-4a1c7025d8dso1579425e9.0 for ; Thu, 15 Oct 2026 13:09:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094969; x=1792699769; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=TaAoLgM/CtoTiNdadIKWG+p+UJsFKFHvL5NWzPeT3Ac=; b=FJBeMtO/yNTXflp5K3A+hqH9enOdk6UUjpqHr7Z1Tc4gWwvn68Z9CoaOIUpIjgp08L 3ae9D27QT5EubvTxRfkvHg4qCr1Wv1Tv9AJtxWsv9oNQdUyTTHR2Mk7lAATQAsawcoYU gTfgs/XsZnWuGNj0aAjLVDsO/W2NA+rkCcjHLF0A1cez+lIRfLsOgzXgjT4xis7X0Nso AO8CQnXl2PbQGEqkPtmKJXKUfgCrnmv2vm5MfEbae7l2BlPXS2IdPaY3J+NyOoytROKT XmskpwpQ1LY53X5CCXVER/JXpFgjloNZKGT3So6JEKdegSVn2LKJH2CD3f40tmT5LymN batQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094969; x=1792699769; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=TaAoLgM/CtoTiNdadIKWG+p+UJsFKFHvL5NWzPeT3Ac=; b=GnPDiuSVMdeGYfUeeLReN2Jt46HFLqvjSuDDuAwZuSyJ0V5DeX0tOPqutDN969ixeT TIjMHhK/Sxdq7xr8XF1GpYj96jFftUsLeywKcYD+Un2UoP/9VNXaRxLF5BoU/05YuoGd P4uoT+vORKRGx3cBeoCzz27cr5AS6sKkzfkHU9yhYOD8Axw/SRbpd7YAejY9LB9aWIWZ 9JcnwKfV4mx7n5+ce830taZktJFkyh2R2nbyLGpEs/qXl6oFxVux7v54jNByU0AneiXG hSEp34NARavHf82XGYaNxNJEtVJezogBEcJ+RPL1pNz3JBDiLaKCQNZUR6NUg8+Bk0i0 ee0w== X-Forwarded-Encrypted: i=1; AKwUvBzMkQacbxCahmm7NbqSDpzhLumXq6vLrpKdwHNnYDh7UG7h6oWLusAhcmNOTCJtfK6izkROqqYur0xXNEI=@vger.kernel.org X-Gm-Message-State: AFq9FYJ8oqXOBp+gLD9IrYQH+mFlaFc1cCHhtxWjkeJ4iMy5p8j9liwa rBoS/DEiy3DZwtZeOb0J72na7/Uu073VtM89dn5sxgPztgByaKigaKYEO7nLAGzjlxPwtODd5lB o4dvzyU3bcqw7zA== X-Received: from wmqh18.prod.google.com ([2002:a05:600c:3512:b0:4a1:93c0:d3c2]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6215:b0:4a0:eae0:b30e with SMTP id 5b1f17b1804b1-4a18e4ca157mr369276785e9.22.1792094968875; Thu, 15 Oct 2026 13:09:28 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:46 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-16-smostafa@google.com> Subject: [PATCH v9 15/25] iommu/arm-smmu-v3-kvm: Add CMDQ functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the command queue accessors. The hypervisor's own commands, such as TLB invalidations, go through smmu_send_cmd(), which queues the command followed by a CMD_SYNC and waits for completion. Filtered host commands are copied into the shadow queue with smmu_add_cmd_raw(). Polling is bounded by a 1 ms timeout measured with the architected counter, and sleeps in WFE when the SMMU supports SEV. Split queue_space() out of queue_has_space(), as the emulation will check the space in the host's queue. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 ++- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 119 ++++++++++++++++++ 2 files changed, 127 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index d07fca3d1cf5..bc2e7e10200a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1287,19 +1287,21 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_dev= ice *smmu, bool arm_smmu_erratum_repeat_tlbi_cfgi(void); =20 /* Queue and STE functions shared between kernel and hyp. */ -static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +static inline u32 queue_space(struct arm_smmu_ll_queue *q) { - u32 space, prod, cons; + u32 prod, cons; =20 prod =3D Q_IDX(q, q->prod); cons =3D Q_IDX(q, q->cons); =20 if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; + return (1 << q->max_n_shift) - (prod - cons); + return cons - prod; +} =20 - return space >=3D n; +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + return queue_space(q) >=3D n; } =20 static inline bool queue_full(struct arm_smmu_ll_queue *q) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 02e7c6175832..c9fc21e405e7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -25,6 +25,42 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 #define cmdq_size(cmdq) ((1UL << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWO= RDS * 8) =20 +static u32 timer_freq; + +static inline u64 hyp_clock_ns(void) +{ + return mul_u64_u32_div(__arch_counter_get_cntvct(), NSEC_PER_SEC, timer_f= req); +} + +#define ARM_SMMU_EL2_POLL_TIMEOUT_US 1000 +/* + * Wait until @__cond is true. With @__use_wfe, sleep in WFE between polls: + * interrupts are masked at EL2, so only the SMMU's SEV or the timer event + * stream wakes the CPU, and the timeout is only checked once awake. + * Return 0 on success, or -ETIMEDOUT. + */ +#define smmu_wait(__use_wfe, __cond) \ +({ \ + int __ret =3D 0; \ + u64 __start =3D hyp_clock_ns(); \ + u64 __timeout =3D ARM_SMMU_EL2_POLL_TIMEOUT_US * NSEC_PER_USEC; \ + \ + while (!(__cond)) { \ + if (__use_wfe) { \ + wfe(); \ + } else { \ + cpu_relax(); \ + } \ + if ((hyp_clock_ns() - __start) >=3D __timeout) { \ + if (__cond) \ + break; \ + __ret =3D -ETIMEDOUT; \ + break; \ + } \ + } \ + __ret; \ +}) + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -77,6 +113,85 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t = size) return 0; } =20 +__maybe_unused +static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_has_space(llq, n); +} + +static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_full(llq); +} + +static bool smmu_cmdq_empty(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_empty(llq); +} + +static void smmu_add_cmd_raw(struct hyp_arm_smmu_v3_device *smmu, + u64 *cmd) +{ + struct arm_smmu_queue *q =3D &smmu->cmdq; + struct arm_smmu_ll_queue *llq =3D &q->llq; + + queue_write(Q_ENT(q, llq->prod), cmd, CMDQ_ENT_DWORDS); + llq->prod =3D queue_inc_prod_n(llq, 1); +} + +static int smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmd *cmd) +{ + int ret; + + hyp_assert_lock_held(&smmu->hw_lock); + ret =3D smmu_wait(smmu->features & ARM_SMMU_FEAT_SEV, + !smmu_cmdq_full(&smmu->cmdq)); + if (ret) + return ret; + + smmu_add_cmd_raw(smmu, cmd->data); + writel(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + return 0; +} + +static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + struct arm_smmu_cmd cmd; + + hyp_assert_lock_held(&smmu->hw_lock); + cmd =3D arm_smmu_make_cmd_sync(CMDQ_SYNC_0_CS_SEV, 0); + ret =3D smmu_add_cmd(smmu, &cmd); + if (ret) + return ret; + + return smmu_wait(false, smmu_cmdq_empty(&smmu->cmdq)); +} + +__maybe_unused +static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmd *cmd) +{ + int ret; + + hyp_assert_lock_held(&smmu->hw_lock); + ret =3D smmu_add_cmd(smmu, cmd); + if (ret) + return ret; + + return smmu_sync_cmd(smmu); +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -202,6 +317,10 @@ static int smmu_init(void) if (!PAGE_ALIGNED(kvm_hyp_arm_smmu_v3_smmus)) return -EINVAL; =20 + timer_freq =3D read_sysreg(cntfrq_el0); + if (!timer_freq || timer_freq > NSEC_PER_SEC) + return -EINVAL; + kvm_hyp_arm_smmu_v3_smmus =3D kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus); pfn =3D hyp_virt_to_pfn(kvm_hyp_arm_smmu_v3_smmus); nr_pages =3D smmu_arr_size >> PAGE_SHIFT; --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33A1358B6DF for ; Thu, 15 Oct 2026 20:09:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094974; cv=none; b=PkDzKZxhWgtusw/+tPhsYxHGF5725XYh2rPuisyUA/2Sbjr+AWM5yjevVtyfW3OZl3m+XU9LgW4EgylRizuoG9E5apE4Zed13vLVnc/M88ejU4IqVnBi+oxND39WHAhM2ZtU8W6rczNuBEogghNr6ol+2n48qJ7/GT8OKzgksyM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094974; c=relaxed/simple; bh=ZJK1kmjG/RFwnOjvDQYvKcQ3fO2aN12GQvMf9Dol6VY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LDoYHL5FnaT56TGHAdZ2ZnXxzUVRPsU4BfQ7DeDV8NTFTIPx0/QSLjNlCVhGayIWoaf1gdUzuOCgBpS1PIG76B518VSrixIMpQZq3rjEIs0RyI6lC72tbswJFs48ZCCx1Rj6P2Zeen2M3n/DO2nPsRV4+LPkSXS2cD31T5juC5A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gWpzKqV9; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gWpzKqV9" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4a1813fea7dso82123225e9.2 for ; Thu, 15 Oct 2026 13:09:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094970; x=1792699770; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=0Bi/zP2WEVGciETaGfBszben9Iy5rJdDBhJW2dD5Omo=; b=gWpzKqV9uOCM7mVQ6jNS9AilgMk0SvdObQ4VWTVDfGms7mT64PC5h3q41U78bj7Ied p+Qz3HHpE4rOHIW63zQ+h4oEI+kp47EFmBK/rzE/9CEq6eYxPBISjXKrUikfgYykXM5p sJiZRyC2o8Xl69Y3GLGNJBlMJ5KjIdVH665FoQVr3XGxx/NlOO0S1A0rvORHxkMwuHwz JGpvDWIJEc0Tz7AFra0/YD+MGuv6cSkle2+JgkInmwO3rtoeUlhlaNZwPvd4yOLFqXTh yCS7Rqlj1RMnyIbFGgBG7EXrElWW4y1i+Q9pJzkxOlhSp2Jrd83j8po6oWRLOG54i57w OxJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094970; x=1792699770; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=0Bi/zP2WEVGciETaGfBszben9Iy5rJdDBhJW2dD5Omo=; b=Hn5/NZuK/FfOqh4UkARUt+69uziZyN6kuTc5NXG6UHPgD/bVNdBLkOZYurSUQGf45Q mm4MUFUvcw1WfwPe1qGh88SaIMVRMz4+V5kDttbf0FlXvX78mP8YKZHphqGrY5mgLoMk 2pMlqo5tdSj1eRC3YaIf7+xS/XdAMbuQ22Q52USd7poQ/DKZMmoasiN5MTzdyaobrbMj vGZ5h1R6nN2998ZoAaQRnRm3QK18OVySNoRAAviGuXnf+42a31IeLcSHdtLvGBRX9BGu VU0mGtU9W2srcfVL8tb+LZWlWrra06YwtokkQp/AX3U5+OcU+us0/qIMEefRcKSnQdIV Mc6g== X-Forwarded-Encrypted: i=1; AKwUvByhyipHSdwbpmaZLZfCijfPo+jOlzWeJijOSLgIF5nO9+mZM8GZrvlx7HaLLV5qfMGFn+wjxdCByTu0X9k=@vger.kernel.org X-Gm-Message-State: AFq9FYKtJO92B2YbtHghbRqS4gxnCt0NHqLwMAobrQ1OiACjqmuFwpCv L4qwAaJ/wHlo4hJSBKmHvwkF3o0kcCH2YBuo5fZ/Kk+l88oFYDjfe+FQd5IAlecVOjeqJ16DyX5 7rHnLJObOrW9KLg== X-Received: from wmsp12.prod.google.com ([2002:a05:600c:1d8c:b0:4a1:c745:6233]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:19c8:b0:4a1:7bae:d80a with SMTP id 5b1f17b1804b1-4a18e47ad49mr419007385e9.9.1792094970227; Thu, 15 Oct 2026 13:09:30 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:47 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-17-smostafa@google.com> Subject: [PATCH v9 16/25] iommu/arm-smmu-v3-kvm: Emulate CMDQ for host From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Take the command queue away from the host: - ARM_SMMU_CMDQ_BASE: writes are only accepted while the command queue is disabled and record the host queue base. Reads return the saved value. - ARM_SMMU_CMDQ_PROD: writes run the emulation, which copies, checks and filters every pending command into the shadow queue, ends the batch with a CMD_SYNC and waits for the hardware to drain it, so the host's view of CONS never runs ahead of completion. Reads return the host copy. - ARM_SMMU_CMDQ_CONS: writes update the software copy and are only accepted while the queue is disabled, so the host cannot skip submitted commands. Reads return the emulated value plus the error bits of the hardware CONS. Commands the filter rejects are fatal, as no well-behaved host issues them. Also emulate IDR1.CMDQS to advertise the shadow queue size, which can be smaller than the hardware's. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 150 +++++++++++++++++- 1 file changed, 145 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index c9fc21e405e7..dc9b1e659a42 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -113,7 +113,6 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t = size) return 0; } =20 -__maybe_unused static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -344,6 +343,104 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_filter_command(struct hyp_arm_smmu_v3_device *smmu, u64 *= command) +{ + u64 type =3D FIELD_GET(CMDQ_0_OP, command[0]); + + switch (type) { + case CMDQ_OP_CFGI_STE: + /* TBD: SHADOW_STE*/ + break; + case CMDQ_OP_CFGI_ALL: + { + /* + * Linux doesn't use range STE invalidation, and only use this + * for CFGI_ALL, which is done on reset and not on an new STE + * being used. + * Although, this is not architectural we rely on the current Linux + * implementation. + */ + if ((FIELD_GET(CMDQ_CFGI_1_RANGE, command[1]) !=3D 31)) + return true; + break; + } + case CMDQ_OP_TLBI_NH_ASID: + case CMDQ_OP_TLBI_NH_VA: + case CMDQ_OP_TLBI_NH_ALL: + case 0x13: /* CMD_TLBI_NH_VAA: Not used by Linux */ + { + /* Only allow VMID =3D 0 */ + if (FIELD_GET(CMDQ_TLBI_0_VMID, command[0]) !=3D 0) + return true; + break; + } + case CMDQ_OP_PREFETCH_CFG: + case CMDQ_OP_CFGI_CD: + case CMDQ_OP_CFGI_CD_ALL: + case CMDQ_OP_TLBI_NSNH_ALL: + case CMDQ_OP_PRI_RESP: + case CMDQ_OP_RESUME: + break; + case CMDQ_OP_CMD_SYNC: + if (FIELD_GET(CMDQ_SYNC_0_CS, command[0]) =3D=3D CMDQ_SYNC_0_CS_IRQ) { + /* Do not allow MSI */ + command[0] &=3D ~CMDQ_SYNC_0_CS; + command[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + command[1] &=3D ~CMDQ_SYNC_1_MSIADDR_MASK; + } + break; + default: + /* Deny unknown commands */ + return true; + } + + return false; +} + +static int smmu_emulate_cmdq_insert(struct hyp_arm_smmu_v3_device *smmu) +{ + u64 *host_cmdq =3D hyp_phys_to_virt(smmu->cmdq_host.base_dma); + u64 cmd[CMDQ_ENT_DWORDS]; + int idx, ret; + u32 pending; + bool skip; + + if (!is_cmdq_enabled(smmu)) + return 0; + + pending =3D (1 << (smmu->cmdq_host.llq.max_n_shift)) - queue_space(&smmu-= >cmdq_host.llq); + + hyp_spin_lock(&smmu->hw_lock); + /* Wait for the command queue to have some space. */ + ret =3D smmu_wait(false, smmu_cmdq_has_space(&smmu->cmdq, pending)); + if (ret) { + hyp_spin_unlock(&smmu->hw_lock); + return ret; + } + + while (pending--) { + int i; + + idx =3D Q_IDX(&smmu->cmdq_host.llq, smmu->cmdq_host.llq.cons); + queue_inc_cons(&smmu->cmdq_host.llq); + + /* Copy the command to local buffer avoiding TOCTOU */ + for (i =3D 0; i < CMDQ_ENT_DWORDS; ++i) + cmd[i] =3D le64_to_cpu(READ_ONCE(host_cmdq[idx * CMDQ_ENT_DWORDS + i])); + + skip =3D smmu_filter_command(smmu, cmd); + if (WARN_ON(skip)) + continue; + smmu_add_cmd_raw(smmu, cmd); + } + + writel(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + + ret =3D smmu_wait(false, smmu_cmdq_empty(&smmu->cmdq)); + hyp_spin_unlock(&smmu->hw_lock); + return ret; +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; @@ -385,18 +482,51 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, */ mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP | ID= R0_ATS); break; - /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_CMDQ_BASE: + /* + * Although allowed to use smaller size, we rely on the SMMUv3 driver + * using 64-bit store instruction for simplicity. + */ + if (len !=3D sizeof(u64)) + break; if (is_write) { /* Not allowed by the architecture */ if (is_cmdq_enabled(smmu)) break; smmu->cmdq_host.q_base =3D val; + goto out_ret; + } else { + val =3D smmu->cmdq_host.q_base; + goto out_update_regs; } - mask =3D read_write; - break; case ARM_SMMU_CMDQ_PROD: + if (len !=3D sizeof(u32)) + break; + if (is_write) { + smmu->cmdq_host.llq.prod =3D val; + WARN_ON(smmu_emulate_cmdq_insert(smmu)); + goto out_ret; + } else { + val =3D smmu->cmdq_host.llq.prod; + goto out_update_regs; + } case ARM_SMMU_CMDQ_CONS: + if (len !=3D sizeof(u32)) + break; + if (is_write) { + if (WARN_ON(is_cmdq_enabled(smmu))) + break; + + smmu->cmdq_host.llq.cons =3D val; + goto out_ret; + } else { + /* Propagate errors back to the host.*/ + u32 cons =3D readl_relaxed(smmu->base + ARM_SMMU_CMDQ_CONS); + + val =3D smmu->cmdq_host.llq.cons | (CMDQ_CONS_ERR & cons); + goto out_update_regs; + } + /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_STRTAB_BASE: case ARM_SMMU_STRTAB_BASE_CFG: case ARM_SMMU_GBPA: @@ -481,10 +611,20 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, mask =3D read_write; break; /* Allowed RO 32 bit registers. */ + case ARM_SMMU_IDR1: + if (len !=3D sizeof(u32)) + break; + /* Cap CMDQS to the shadow queue size. */ + if (!is_write) { + val =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); + val &=3D ~IDR1_CMDQS; + val |=3D FIELD_PREP(IDR1_CMDQS, smmu->cmdq.llq.max_n_shift); + goto out_update_regs; + } + fallthrough; case ARM_SMMU_IIDR: case ARM_SMMU_IDR5: case ARM_SMMU_IDR3: - case ARM_SMMU_IDR1: case ARM_SMMU_IRQ_CTRLACK: case ARM_SMMU_GERROR: if (len !=3D sizeof(u32)) --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C938649F10D for ; Thu, 15 Oct 2026 20:09:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094978; cv=none; b=MntsX14PL+0e1If3VuZ/niPiKdsTmmu4agdpQb2qTq9zWFJmnmJXIl0fonFH0iP5dvZrZg5ymFfLHUmxdMpxVW4L8ixJjbesneO5Ox92xs7GnsJukjDr5QdNVSNOatrSJ4k4FqTjLcY8JLPkKUDmN5qG9n67AD8CSsR645Fsndk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094978; c=relaxed/simple; bh=H1XJZkFETNlFizGY4gN5Rz/C+FYzCAV4WD+qe29jC30=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GnxQwwe5SRPSWJZFiSx8dxUsPtKEz3Q5MxUqjq7xFjp8t3alfxhBAXw4Bz/7bUechPTXorTyMUpLJPDmuz2IrlmDqnKtyE3ZJ9ZPV+a4Cai0j23Di13eeWiwoh8+JVNU2WBp8PQlRnLLiD9czK8j0fhRFbOu/1dkJQLx4SNceBE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vS7E7EVX; arc=none smtp.client-ip=209.85.208.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vS7E7EVX" Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-6b1d35e3c0fso2662814a12.3 for ; Thu, 15 Oct 2026 13:09:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094972; x=1792699772; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9TkbtrZkecnnvik8qD/P9yBzmRooaT/6w/h40hH+Qfk=; b=vS7E7EVXR1jbz7/D845w9mtEcBfAHfCAiekXPNUjIPmhE2kO7s3CXQVOgstJc2XdNO 00eGgk5iTQfGoeizjXNBOqu8cnD+nyrCqLuI+Sj9dIuf2PNgR+YHzZb/wleAbH6BWiut 9gK3+rY0b4cvoU3ftfTkpIjMhChlrk7xvDowGEmUDP6Co8PoOsub78QQp1Wht2WXYoYX 7WARqyeCc6PDdlc6KWUaLArm+C4YY6AkMW5VhYvwELN+1nTZjDSLG/ATHHihdf84aBob njNQ1v6LpoZZhwxRdf4puEdIWTp3B0M0aja8Jl9rvYyEXHWbkWe/mB4lDp9HBAw0kaBs 1oug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094972; x=1792699772; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9TkbtrZkecnnvik8qD/P9yBzmRooaT/6w/h40hH+Qfk=; b=kt+JX6FD+Mv5b8BRgAAo7d/dpo7ZON54THM5oy0V07naA+5k5wl02P5WbCNBtGxTD4 Mqs3d3P+gNrW0/Uep7ifcRY92WK5EJjVxAvAlcH3DUKdyvjLN9mqn4QFLszL51v+A00R qZ9+BFisitbKpVSrzrAvJKIaVPyNPzvShMyVY52btU0NSXTP9N91TatnyMceX5w9hApS lJT6UNQ/Oz12RsdpS58Hha7CCVyhSeOuPD3W7368mBGWbSzNPHujqypl/ZmoOAtnmi2I 7SfZrmUBVq8MdgQlOAf2ZZirzmK9exgedMEwqt8E0uGuJRH4wT5mWqI/NHWIsFaPCufx v50Q== X-Forwarded-Encrypted: i=1; AKwUvBx8mF7lA7Dze5j30FzuLSKi2CZtKXB4A7O8s52/u4cysMqAUcqad0G4iELKObMUSmmT0/zG4LkdIpBnt18=@vger.kernel.org X-Gm-Message-State: AFq9FYJLB3iNlBx0iiXcNoIDnDQNE2RUYPAAYAGX0fUXzPJHQ2PRs7Th Z5ckRTNZUbeiT8KK4+gL7NcqVi+bcELBZV17HTWtKuTMINuWnfEQ3W+pw2g4rmLq6p3k4wHJ9dm 6fYUjLM4d61C58w== X-Received: from edro22.prod.google.com ([2002:aa7:d3d6:0:b0:6ac:b91c:a379]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:42c3:b0:6a9:a3df:e4f1 with SMTP id 4fb4d7f45d1cf-6b17c2c4b13mr16130875a12.25.1792094971599; Thu, 15 Oct 2026 13:09:31 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:48 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-18-smostafa@google.com> Subject: [PATCH v9 17/25] iommu/arm-smmu-v3-kvm: Shadow stream table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allocate the shadow stream table per SMMU. We choose the size of that table to be 1MB which is the max size used by host in the case of 2 levels. All the host writes are still passthrough for bisectability, that is changed next where CFGI commands will be trapped and used to update the shadow copy that will be used by the HW. Similar to the command queue, the host stream table is shared/unshared each time the SMMU is enabled/disabled. Handling of L2 tables is also done in the next patch when the shadowing is added. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c | 21 ++- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 122 ++++++++++++++++++ .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 11 ++ 3 files changed, 153 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c b/driver= s/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c index d7f2e73add7f..09078a6c39b0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-pkvm-init.c @@ -17,6 +17,13 @@ #include "pkvm/arm-smmu-v3-hyp.h" =20 #define SMMU_KVM_CMDQ_ORDER get_order(MAX(SZ_64K, PAGE_SIZE)) +/* + * Use the max value of L1 the kernel uses, that also covers the worst case + * for linear tables as it is mandatory according to the spec to support 2 + * lvl tables if SIDSIZE >=3D 7 + */ +#define SMMU_KVM_STRTAB_ORDER (get_order(STRTAB_MAX_L1_ENTRIES * \ + sizeof(struct arm_smmu_strtab_l1))) =20 static size_t kvm_arm_smmu_count; static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; @@ -33,6 +40,9 @@ static void __init kvm_arm_smmu_array_free(void) if (smmu->cmdq.base_dma) free_pages((unsigned long)phys_to_virt(smmu->cmdq.base_dma), SMMU_KVM_CMDQ_ORDER); + if (smmu->strtab_dma) + free_pages((unsigned long)phys_to_virt(smmu->strtab_dma), + SMMU_KVM_STRTAB_ORDER); } =20 order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); @@ -77,8 +87,8 @@ static int __init smmuv3_nesting_probe(struct platform_de= vice *pdev) { struct hyp_arm_smmu_v3_device *smmu; struct device *dev =3D &pdev->dev; + void *cmdq_base, *strtab; struct resource *res; - void *cmdq_base; =20 /* Only device tree, ACPI not supported. */ if (!dev->of_node) @@ -119,6 +129,15 @@ static int __init smmuv3_nesting_probe(struct platform= _device *pdev) smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; =20 + strtab =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_STR= TAB_ORDER); + if (!strtab) { + free_pages((unsigned long)cmdq_base, SMMU_KVM_CMDQ_ORDER); + return -ENOMEM; + } + + smmu->strtab_dma =3D virt_to_phys(strtab); + smmu->strtab_size =3D PAGE_SIZE << SMMU_KVM_STRTAB_ORDER; + if (of_dma_is_coherent(dev->of_node)) smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index dc9b1e659a42..0fa98915e00d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -18,6 +18,14 @@ size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +/* strtab accessors */ +#define strtab_log2size(smmu) (FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, (smmu)-= >host_ste_cfg)) +#define strtab_size(smmu) ((1UL << strtab_log2size(smmu)) * STRTAB_STE_DWO= RDS * 8) +#define strtab_host_base(smmu) ((smmu)->host_ste_base & STRTAB_BASE_ADDR_M= ASK) +#define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) +#define strtab_l1_size(smmu) ((1UL << (strtab_log2size(smmu) - strtab_spli= t(smmu))) * \ + (sizeof(struct arm_smmu_strtab_l1))) + #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ @@ -66,6 +74,11 @@ static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_devic= e *smmu) return FIELD_GET(CR0_CMDQEN, smmu->cr0); } =20 +static bool is_smmu_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_SMMUEN, smmu->cr0); +} + /* * Host structures the hypervisor reads (CMDQ, STE) are shared and pinned: * - Prevent the host from donating this memory. @@ -200,6 +213,11 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_= device *smmu) if (smmu->cmdq.base) WARN_ON(__pkvm_hyp_donate_host(smmu->cmdq.base_dma >> PAGE_SHIFT, PAGE_ALIGN(cmdq_size(&smmu->cmdq)) >> PAGE_SHIFT)); + + if (smmu->strtab_cfg.linear.table || + smmu->strtab_cfg.l2.l1tab) + WARN_ON(__pkvm_hyp_donate_host(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT)); smmu->base =3D NULL; } =20 @@ -272,6 +290,45 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + int ret; + u32 reg; + + ret =3D __pkvm_host_donate_hyp(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT); + if (ret) + return ret; + + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + unsigned int last_sid_idx =3D + arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1); + + cfg->l2.l1tab =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->l2.l1_dma =3D smmu->strtab_dma; + cfg->l2.num_l1_ents =3D min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES); + + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_2LVL) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, + ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) | + FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT); + } else { + cfg->linear.table =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->linear.ste_dma =3D smmu->strtab_dma; + cfg->linear.num_ents =3D 1UL << smmu->sid_bits; + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_LINEAR) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits); + } + + writeq_relaxed((smmu->strtab_dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_R= A, + smmu->base + ARM_SMMU_STRTAB_BASE); + writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -297,6 +354,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_deinit; =20 + ret =3D smmu_init_strtab(smmu); + if (ret) + goto out_deinit; + return 0; =20 out_deinit: @@ -441,6 +502,46 @@ static int smmu_emulate_cmdq_insert(struct hyp_arm_smm= u_v3_device *smmu) return ret; } =20 +static int smmu_update_ste_shadow(struct hyp_arm_smmu_v3_device *smmu, boo= l enabled) +{ + size_t size; + u32 fmt =3D FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->host_ste_cfg); + + /* Linux doesn't change the fmt nor size of the strtab in the run time. */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + if ((fmt !=3D STRTAB_BASE_CFG_FMT_2LVL) || + (strtab_split(smmu) !=3D STRTAB_SPLIT) || + (strtab_log2size(smmu) > (ilog2(STRTAB_MAX_L1_ENTRIES) + STRTAB_SPLI= T)) || + (strtab_split(smmu) >=3D strtab_log2size(smmu))) + return -EINVAL; + size =3D strtab_l1_size(smmu); + } else { + if ((fmt !=3D STRTAB_BASE_CFG_FMT_LINEAR) || + (strtab_log2size(smmu) > smmu->sid_bits)) + return -EINVAL; + size =3D strtab_size(smmu); + } + + if (enabled) + return smmu_share_pages(strtab_host_base(smmu), size); + + return smmu_unshare_pages(strtab_host_base(smmu), size); +} + +static void smmu_emulate_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + /* Enabling SMMU without CMDQ, means TLB invalidation won't work. */ + if (WARN_ON(!is_cmdq_enabled(smmu))) + return; + + WARN_ON(smmu_update_ste_shadow(smmu, true)); +} + +static void smmu_emulate_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(smmu_update_ste_shadow(smmu, false)); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; @@ -528,7 +629,23 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, } /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_STRTAB_BASE: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0. */ + if (is_smmu_enabled(smmu)) + break; + smmu->host_ste_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_STRTAB_BASE_CFG: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0. */ + if (is_smmu_enabled(smmu)) + break; + smmu->host_ste_cfg =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_GBPA: mask =3D read_write; break; @@ -538,6 +655,7 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_dev= ice *smmu, =20 if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); + bool last_smmu_en =3D is_smmu_enabled(smmu); =20 /* Enforce ATS checks. */ if (smmu->features & ARM_SMMU_FEAT_ATS) @@ -547,6 +665,10 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + if (!last_smmu_en && is_smmu_enabled(smmu)) + smmu_emulate_enable(smmu); + else if (last_smmu_en && !is_smmu_enabled(smmu)) + smmu_emulate_disable(smmu); mask =3D read_write; } else { mask =3D read_write & ~CR0_ATSCHK; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h index 3903ae358555..31ab7a0335a8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -27,6 +27,12 @@ * @cmdq_host: Host view of the CMDQ, only q_base, base_dma and llq used * @cmdq_max_shift: Max shift for the CMDQ probed from HW * @cr0: Last value of CR0 + * @host_ste_cfg: Host stream table config + * @host_ste_base: Host stream table base + * @strtab_cfg: Stream table config as seen by HW + * @strtab_dma: Physical address of the shadow stream table + * @strtab_size: Size of the shadow stream table + * * @mmio_addr, @mmio_size and ARM_SMMU_FEAT_COHERENCY are filled by the ho= st * before de-privilege, the rest by the hypervisor driver. */ @@ -50,6 +56,11 @@ struct hyp_arm_smmu_v3_device { struct arm_smmu_queue cmdq_host; u32 cmdq_max_shift; u32 cr0; + dma_addr_t strtab_dma; + size_t strtab_size; + u64 host_ste_cfg; + u64 host_ste_base; + struct arm_smmu_strtab_cfg strtab_cfg; }; =20 #ifdef __KVM_NVHE_HYPERVISOR__ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3CBE3595735 for ; Thu, 15 Oct 2026 20:09:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094978; cv=none; b=FPPUMKiPWfsOEhP/MSj8gsoq3ZkyuJpELULspPvO0sO72ID1I/66NOOQbMPQChNtwlvO77N+U3mtOnxdwWT1sRqGJNKX3LY9mtmdBiC5Y6L1GTKHXxwVDI1nrcBeOSgrAUZ8S5LpvZSd2xw3EpNO3wSRNSiePXEYC3slzewpgP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094978; c=relaxed/simple; bh=iTalrg1xW1kAOUo6QghDdcHthHRFMoeDKbdR5Syc7gg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=G0v1XQSCdIknLn14UuHnFoi5ZinfqdZbkNd/cCk4tEsYO0JUaqxBHA0e3FGW5Tp50+z/amI45AXYKe543Rc1ov1XgnGKSp62ToWBd0wgLbvHP+ux7Lx5bj9joja/U7Vppisyab028khqpm2Y2X9pLhWVc0o8cGw5MkL+lZMmqOg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eVrDqJ4I; arc=none smtp.client-ip=209.85.221.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eVrDqJ4I" Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-48b01a74792so3724747f8f.0 for ; Thu, 15 Oct 2026 13:09:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094973; x=1792699773; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=FI4iSn8zAFr7RU3MPIgQJ9gzmtZlTDgEvk+laO2rSqk=; b=eVrDqJ4I+ZSpZzYndFSM6XYAS1ty14taBNbi7iQZ6QW4JiAEtrGpUNObEdU9NlNqfk GVeIMZilKSDiCcDE/nQRzSsjMA1zz5gK4cLNHqKuAaqdCSn/srfw8wVut7UVncHNYaSd RkdrjJjY+d8LvJO9ifg50K4acOQGO+GD4mXfLGMMoyPsz5fzw+sQPNlHOTnOnzjHTft5 98p3tl6e4BGm0Vdld7eovTWPxlFR/N1nRtPlW7YAea0K3SM/CA+dyD8g3IQjpHoPhhO0 a+GjfGTeldRFRe6VxSqD78Z+xySbM118Gix9o+FfNuyoTQpthTqzB9GDolnjv1TwVyBa rJRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094973; x=1792699773; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=FI4iSn8zAFr7RU3MPIgQJ9gzmtZlTDgEvk+laO2rSqk=; b=1UiLUz2pbaj9vd6T1kMI/gdc81foO25ULIuNA5Mja9kef9my74SKf5uXIvc5ME/QS1 U1nVMJslbtzGIzAbFKoTKQKr2IEyv2+yBAxA1f9QxR0CLuDaEFcXBwAkm6noVNMa/mO/ SubAEPDrqw7sjBu5C4anTFjS8/tmHjyERcaM5f3OcF4YBDci8MgkzDEPnDJHQvNck1km X5YsMB3DPLpLSYPCOTQ7lFD4A9crSOlpJLm7wIiJpWF+31JTQ/L+dq1NUx+lHVkNIsKl TgWWrhLEXGodVSxdVoFejjNqG3G0i8LeyssB5URugOchfnxKjacDArqmraRxKC2OXkPO c2cw== X-Forwarded-Encrypted: i=1; AKwUvByZvfrRH0XfUwCKfdo+voti2pETOU5tHo7rn3xd6BFdFS3mfG4WPq7BXt/azKGFAUAjyUGWIpXYW4agHnc=@vger.kernel.org X-Gm-Message-State: AFq9FYJfq16Sbz8KbD22nuWsavnXDTLDNy1fEZH75uVX/WBs9oHQr8YW oMxbjHUoyzToKd+mTnPzoqIP9jEj8XgbwMc++STaxC7T0MUqTOIsMjkVjsHv6iJgATuhNgN6vHE aQ6rRWPrtB58/sw== X-Received: from wrqr3.prod.google.com ([2002:a5d:4983:0:b0:48b:9d:e37d]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2dc1:b0:488:8850:50ec with SMTP id ffacd0b85a97d-48de2b4d7e5mr19390320f8f.46.1792094973106; Thu, 15 Oct 2026 13:09:33 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:49 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-19-smostafa@google.com> Subject: [PATCH v9 18/25] iommu/arm-smmu-v3-kvm: Shadow STEs From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Shadow the STE when the host sends CFGI_STE: copy the host entry as is into the hypervisor-owned stream table, which is the one the hardware uses. For 2-level tables the host L2 table is shared and pinned while the entry is read, and the hypervisor L2 table is allocated from the IOMMU pool the first time its L1 slot is used. STRTAB_BASE and STRTAB_BASE_CFG are no longer passed through: writes are recorded while the SMMU is disabled and reads return the recorded values. The next patch attaches the stage-2 page table to the shadow entry. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 124 +++++++++++++++++- 1 file changed, 118 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 0fa98915e00d..132f3a9ba308 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -25,6 +25,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; #define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) #define strtab_l1_size(smmu) ((1UL << (strtab_log2size(smmu) - strtab_spli= t(smmu))) * \ (sizeof(struct arm_smmu_strtab_l1))) +#define strtab_hyp_base(smmu) ((smmu)->features & ARM_SMMU_FEAT_2_LVL_STRT= AB ? \ + (u64 *)(smmu)->strtab_cfg.l2.l1tab :\ + (u64 *)(smmu)->strtab_cfg.linear.table) =20 #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ @@ -290,6 +293,91 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v3_device *smmu, u32 s= id, + struct arm_smmu_ste *host_ste_out) +{ + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + struct arm_smmu_strtab_l1 host_l1_desc; + struct arm_smmu_strtab_l2 *l2ptr; + phys_addr_t host_l2_tab; + int ret, i; + + host_l1_desc.l2ptr =3D READ_ONCE(host_ste_base[arm_smmu_strtab_l1_idx(sid= )]); + if (!(le64_to_cpu(host_l1_desc.l2ptr) & STRTAB_L1_DESC_SPAN)) + return -EINVAL; + + host_l2_tab =3D le64_to_cpu(host_l1_desc.l2ptr) & STRTAB_L1_DESC_L2PTR_MA= SK; + /* Share and pin the table before accessing it. */ + ret =3D smmu_share_pages(host_l2_tab, sizeof(struct arm_smmu_strtab_l2)); + if (ret) + return ret; + + l2ptr =3D hyp_phys_to_virt(host_l2_tab); + + for (i =3D 0; i < STRTAB_STE_DWORDS; ++i) + host_ste_out->data[i] =3D + READ_ONCE(l2ptr->stes[arm_smmu_strtab_l2_idx(sid)].data[i]); + + WARN_ON(smmu_unshare_pages(host_l2_tab, sizeof(struct arm_smmu_strtab_l2)= )); + return 0; +} + +static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid,= bool leaf) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + struct arm_smmu_ste *hyp_ste_ptr, *host_ste_ptr, host_ste_copy; + u64 *hyp_ste_base =3D strtab_hyp_base(smmu); + int ret, i; + + /* Leaf =3D 0 is not supported, as the kernel driver. */ + if (!leaf || !is_smmu_enabled(smmu) || + (sid >=3D (1UL << strtab_log2size(smmu)))) + return -EINVAL; + + if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)) { + struct arm_smmu_ste *hyp_table =3D (struct arm_smmu_ste *)hyp_ste_base; + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + struct arm_smmu_ste *host_table =3D (struct arm_smmu_ste *)host_ste_base; + + if (sid >=3D cfg->linear.num_ents) + return -E2BIG; + + hyp_ste_ptr =3D &hyp_table[sid]; + host_ste_ptr =3D &host_table[sid]; + } else { + struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)hyp_st= e_base; + u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); + struct arm_smmu_strtab_l2 *l2ptr; + + if (l1_idx >=3D cfg->l2.num_l1_ents) + return -E2BIG; + + host_ste_ptr =3D &host_ste_copy; + ret =3D smmu_get_host_l2_ste(smmu, sid, host_ste_ptr); + if (ret) + return ret; + + /* No hypervisor entry, first time the L2 is populated. */ + if (!l1tab[l1_idx].l2ptr) { + struct arm_smmu_strtab_l2 *l2table; + + l2table =3D pkvm_iommu_alloc_pages(get_order(sizeof(*l2table))); + if (!l2table) + return -ENOMEM; + /* Ensure L2 is observed before writing the ptr. */ + dma_wmb(); + arm_smmu_write_strtab_l1_desc(&l1tab[l1_idx], hyp_virt_to_phys(l2table)= ); + } + l2ptr =3D hyp_phys_to_virt(le64_to_cpu(l1tab[l1_idx].l2ptr) & + STRTAB_L1_DESC_L2PTR_MASK); + hyp_ste_ptr =3D &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; + } + + for (i =3D 0; i < STRTAB_STE_DWORDS; ++i) + WRITE_ONCE(hyp_ste_ptr->data[i], host_ste_ptr->data[i]); + return 0; +} + static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) { struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; @@ -410,8 +498,24 @@ static bool smmu_filter_command(struct hyp_arm_smmu_v3= _device *smmu, u64 *comman =20 switch (type) { case CMDQ_OP_CFGI_STE: - /* TBD: SHADOW_STE*/ + { + u32 leaf =3D FIELD_GET(CMDQ_CFGI_1_LEAF, command[1]); + u32 sid =3D FIELD_GET(CMDQ_CFGI_0_SID, command[0]); + int ret; + + /* + * If STE update is required, flush the CMDQ and drop the lock as that + * might require to update the host page table and acquire its lock. + */ + writel(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + hyp_spin_unlock(&smmu->hw_lock); + ret =3D smmu_reshadow_ste(smmu, sid, leaf); + hyp_spin_lock(&smmu->hw_lock); + if (ret) + return true; + break; + } case CMDQ_OP_CFGI_ALL: { /* @@ -627,25 +731,33 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, val =3D smmu->cmdq_host.llq.cons | (CMDQ_CONS_ERR & cons); goto out_update_regs; } - /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_STRTAB_BASE: + if (len !=3D sizeof(u64)) + break; if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0. */ if (is_smmu_enabled(smmu)) break; smmu->host_ste_base =3D val; + goto out_ret; + } else { + val =3D smmu->host_ste_base; + goto out_update_regs; } - mask =3D read_write; - break; case ARM_SMMU_STRTAB_BASE_CFG: + if (len !=3D sizeof(u32)) + break; if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0. */ if (is_smmu_enabled(smmu)) break; smmu->host_ste_cfg =3D val; + goto out_ret; + } else { + val =3D smmu->host_ste_cfg; + goto out_update_regs; } - mask =3D read_write; - break; + /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_GBPA: mask =3D read_write; break; --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AE76595750 for ; Thu, 15 Oct 2026 20:09:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094981; cv=none; b=LJT+StkFyPG3PWGODn5ZymKHWP94V1HU7l1p2/mwoXFaLEvdStyvfHYRZbWlNJiARPJaEhE8Z6aa4OX0sT4jCuhlFFVziJm4KG+eqxgFyfI/i2K9jjphy5tAYSQ33Bec0Pe9A2enHjTXPQ0hZ+LpOT6gFX30W4JyzA3zLdOt/jY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094981; c=relaxed/simple; bh=hu8Ca3NAu0Ymk1meYbnbfYAPWUvoT1ANrBH1t0G+/hU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lcbqJ4vglwNkhn1whgPRgyAW/6gIXaOVfrzQAZcXEQnGOSZAI/lomxSSC6G8RrISZ3zQxjzCYVOe7oS3i3xGLCSwYBXfUkK4sUwY+8I5LcIu0hKomTA1VUy6MFMa2jDCgfNY2Egk5yJK0wsWHa6HjvAfSZ+yPkiwbgSZHPUXLJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=f+pgb33H; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="f+pgb33H" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4a0761224e1so21972035e9.0 for ; Thu, 15 Oct 2026 13:09:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094974; x=1792699774; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=OrEOEPSGJdrT+r8ut9ZMnt1QbUsQgQecPDrsSjFG450=; b=f+pgb33Hxkp+aLH7JXzlUVd3+M2+IQ3Gb2AXWNA8Ohu+SEza2dWLNKTqjRCSZpiWIF Qb9hqFY3o+uwy4U628F+bDPdsS3RWgYxXpvhnXzWtFJxTNB2PyvqmJ/o1SPAblneCnqS mKGHEQKSFsFGvxJOuZVNgxXhWdW/HDYgFXtVmHgxs8DE2cy+2X7LPsGXj0klvFESxgu9 avy9fV+1kKXZCKBbvcTQNDqADVS+sTMssYhekfv/EOdillorQL2G9ehynYr7AJg5389i Gu8zR2M4JugfabFPWLZggIv0Io5Dy2v6n04l40icq7dwbhNFdF5tZqGoXTS5jL4mv4zn lD/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094974; x=1792699774; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OrEOEPSGJdrT+r8ut9ZMnt1QbUsQgQecPDrsSjFG450=; b=18H8FDJjDLAlicj2ewkaWjIrXQDxEWt1FNxJkacjTPqcLGraotAadPlG3fg3moOm/k O76CtvyqquHZyXaNHu0OQ7nar4nmiMkvTaqxOAOrsd2gVsJDZ5+db/N7/TZf4FbqB4jF iKmKgb4W55SANPNkKxokIejJ9QwL/IVHLEzzkSaL1Qpl77cZs1CEyG10wMXm3kwxWyg/ AEOJ58mGBIx9f/9JxhEXQ171HA6MmvhKKpQ3nguEaXkqFs+aLWDUjVO5iCzrLJPU9ICE tRt2e8mCVcd8ilZkokW1Y189FWPPmThNdQKRwQyczu0vbFbaHugHVZVX3eukDJYHpdwg aBEA== X-Forwarded-Encrypted: i=1; AKwUvBzvRGn6Ja5goKyZ13MxD9O7DnSANO668Fwp7my7fUTB2T3ZGuJnWI5iO5vgxTbBfMgriY9lM8Y1a2lrLCk=@vger.kernel.org X-Gm-Message-State: AFq9FYL0hamxYwiOn53oinig+Xb9sITCuC1Nzp+4MTstHakscCEYF1+K AthzhkCtlymreV7OHkk8cJ4GIdpVBess3XlVIi17Uo3OOwaaaw3KwJC5VxC1srNhEyFvzJMaU5X zSCBzBRPCE0M1TA== X-Received: from wmbf10.prod.google.com ([2002:a05:600c:594a:b0:4a1:b4c7:1ef2]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3e07:b0:4a1:8627:aec7 with SMTP id 5b1f17b1804b1-4a18e4987abmr362157685e9.5.1792094974491; Thu, 15 Oct 2026 13:09:34 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:50 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-20-smostafa@google.com> Subject: [PATCH v9 19/25] iommu/arm-smmu-v3-kvm: Share other queues From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The event and PRI queues are written by the SMMU, not read by the hypervisor, so they are not shadowed. They still must not be pointed at protected memory: when the host enables a queue, share and pin its memory, which fails unless the host owns it and keeps the host from donating it afterwards. The queues stay shared once enabled, as the SMMU may keep writing until CR0ACK reflects the disable, and the base may not change once shared. The kernel driver programs each base once and never changes it. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 81 +++++++++++++++++-- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 8 ++ 2 files changed, 84 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 132f3a9ba308..1972df1d2a8d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -82,6 +82,16 @@ static bool is_smmu_enabled(struct hyp_arm_smmu_v3_devic= e *smmu) return FIELD_GET(CR0_SMMUEN, smmu->cr0); } =20 +static bool is_evtq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_EVTQEN, smmu->cr0); +} + +static bool is_priq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_PRIQEN, smmu->cr0); +} + /* * Host structures the hypervisor reads (CMDQ, STE) are shared and pinned: * - Prevent the host from donating this memory. @@ -663,6 +673,20 @@ static void smmu_emulate_cmdq_disable(struct hyp_arm_s= mmu_v3_device *smmu) cmdq_size(&smmu->cmdq_host))); } =20 +static void smmu_share_queue(struct hyp_arm_smmu_v3_device *smmu, + unsigned long q_base, size_t ent_size_shift) +{ + /* Q_BASE_ADDR_MASK is not enough as the SMMU also ignores bits > OAS */ + phys_addr_t base =3D q_base & Q_BASE_ADDR_MASK & ((1ULL << smmu->oas) - 1= ); + size_t size =3D 1UL << (FIELD_GET(Q_BASE_LOG2SIZE, q_base) + ent_size_shi= ft); + + /* The host must program the base first and it must be aligned to size. */ + if (WARN_ON(!q_base || (base & (size - 1)))) + return; + + WARN_ON(smmu_share_pages(base, size)); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -768,6 +792,8 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_dev= ice *smmu, if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); bool last_smmu_en =3D is_smmu_enabled(smmu); + bool last_evtq_en =3D is_evtq_enabled(smmu); + bool last_priq_en =3D is_priq_enabled(smmu); =20 /* Enforce ATS checks. */ if (smmu->features & ARM_SMMU_FEAT_ATS) @@ -777,6 +803,26 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + + /* + * Share PRI and EVTQ to avoid the host using them to write to + * protected memory. However, do not unshare the queues at disable + * as that is more complicated, unsharing from here can lead to + * use-after-unshare issues, and requires ordering with cr0ack. + * The host can disable those queues during shutdown, but it never + * changes the base address (even with RPM), so leave the queue + * shared and assert that multiple host writes do not change it. + */ + if (!last_evtq_en && is_evtq_enabled(smmu) && !smmu->evtq_shared) { + smmu_share_queue(smmu, smmu->evtq_base, EVTQ_ENT_SZ_SHIFT); + smmu->evtq_shared =3D true; + } + + if (!last_priq_en && is_priq_enabled(smmu) && !smmu->priq_shared) { + smmu_share_queue(smmu, smmu->priq_base, PRIQ_ENT_SZ_SHIFT); + smmu->priq_shared =3D true; + } + if (!last_smmu_en && is_smmu_enabled(smmu)) smmu_emulate_enable(smmu); else if (last_smmu_en && !is_smmu_enabled(smmu)) @@ -806,6 +852,34 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, =20 mask =3D read_write; break; + case ARM_SMMU_EVTQ_BASE: + if (len !=3D sizeof(u64)) + break; + + if (is_write) { + /* See ARM_SMMU_CR0 */ + if (is_evtq_enabled(smmu) || + (smmu->evtq_shared && smmu->evtq_base !=3D val)) + break; + smmu->evtq_base =3D val; + } + mask =3D read_write; + break; + + case ARM_SMMU_PRIQ_BASE: + if (len !=3D sizeof(u64)) + break; + + if (is_write) { + /* See ARM_SMMU_CR0 */ + if (is_priq_enabled(smmu) || + (smmu->priq_shared && smmu->priq_base !=3D val)) + break; + smmu->priq_base =3D val; + } + mask =3D read_write; + break; + /* Allowed 32 bit registers. */ case ARM_SMMU_EVTQ_IRQ_CFG1: case ARM_SMMU_EVTQ_IRQ_CFG2: @@ -833,15 +907,12 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, case ARM_SMMU_EVTQ_IRQ_CFG0: case ARM_SMMU_PRIQ_IRQ_CFG0: case ARM_SMMU_GERROR_IRQ_CFG0: + if (len !=3D sizeof(u64)) + break; /* These are RES0 as MSI support is hidden. */ val =3D 0; if (!is_write) goto out_update_regs; - fallthrough; - case ARM_SMMU_EVTQ_BASE: - case ARM_SMMU_PRIQ_BASE: - if (len !=3D sizeof(u64)) - break; mask =3D read_write; break; /* Allowed RO 32 bit registers. */ diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h index 31ab7a0335a8..2fb478e9233a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -32,6 +32,10 @@ * @strtab_cfg: Stream table config as seen by HW * @strtab_dma: Physical address of the shadow stream table * @strtab_size: Size of the shadow stream table + * @evtq_base: Host EVTQ_BASE value + * @priq_base: Host PRIQ_BASE value + * @evtq_shared: Whether the EVTQ was shared + * @priq_shared: Whether the PRIQ was shared * * @mmio_addr, @mmio_size and ARM_SMMU_FEAT_COHERENCY are filled by the ho= st * before de-privilege, the rest by the hypervisor driver. @@ -61,6 +65,10 @@ struct hyp_arm_smmu_v3_device { u64 host_ste_cfg; u64 host_ste_base; struct arm_smmu_strtab_cfg strtab_cfg; + u64 evtq_base; + u64 priq_base; + bool evtq_shared; + bool priq_shared; }; =20 #ifdef __KVM_NVHE_HYPERVISOR__ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71459595747 for ; Thu, 15 Oct 2026 20:09:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094981; cv=none; b=cJhl4RW0LfvXSU3m+R8kgySiQsjZ0nxubcGwpg+qZ2nuB9TOF+yNhdu3StRIb6SUeFOWcMzir8cnIhQMrvnexFLiQv0H4jgcW2iGYC8MvoncHQmmdsnNTyeWQTFt7H7tGpyFkE5t8DGgy1Qhzzzu2vArMkitkildSe0HqlaxZCw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094981; c=relaxed/simple; bh=esGTvS6DZ/ClMQbU119hVRZc93nVCLe8LifqfkQfW5A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=p3cCNpHsIt8fV2zxZDtZRR/zbsHbHP2gmBBplROexV0YcWX4ZVNvfhL7pmXWn5ZUj6RtjBLuSumUm7LRtSImVSowpk+va9kAAk1QpeDThN7Q9Ye/iTm0O3zSkgON4iIYTGoyEohStmnrDQLPcASe8JxoLTmuQmO7dt76KbQnckY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=O/DE1dTI; arc=none smtp.client-ip=209.85.218.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="O/DE1dTI" Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-c25501f4aa2so42476166b.1 for ; Thu, 15 Oct 2026 13:09:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094975; x=1792699775; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ACz9N/uLV4ni8vIh9HTqb065WbDM683/OQmKXL1iQNQ=; b=O/DE1dTIb979jcCiV8zfPazR5KxHehC70/iq3d6I/jNsWANt51RT9WdF3Fxuo90PbY HBa/AGauf3nZHN8LKNPbdPEJNkx82F4FmbOs1khF0tsYoCG3yiYDKxpP5dKaREJ6eMxS yd8ocLDkQtYikYDpg/6ePAoiplmrJli4NeqFmgn8Mubl65wi4lV+pwB/e7PTgGjynPXU xyTnEz2dZIB79h7zTUDIJy7f7S1xKaZzsC6xWxy152B08HWnTXj1ycCG8jnUklBYuesE 3KC/h4E+drIKKGI60RbDyYHOxPdCg9y6RfYL35QeOCE22PJSBAi25LSZCsIxru2Pt530 QpBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094975; x=1792699775; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ACz9N/uLV4ni8vIh9HTqb065WbDM683/OQmKXL1iQNQ=; b=rdwM0/bu12mHp6z0ZFChkf11XzEC3aFJ2HW0Bj7fF9Isa+zzbAVJYbwG7tN0UgEpFj 1iIt0GnaGTcvMy0Ee205rtr8cD7/Fpfb4R9zIDAcBnxvvhQPakKVUlVA5ZEfWny9jQ7i jja14r1zd/eMrni8qpUE0QI8PjB8Lb3UlPel0m7lWx2gRJyjgzoRO2nfjXSlfasDAIRz 4fPZPuBE48SvZ6pZHPaCldPkMW1+4YX9ZmbLv7I+QPjAtK/J0Sl2WQ0gMFURDTfD549I kkCp+9a9hOtrgwNXbp6OJJY4Rmg1zd8QM2sNfqAcCR0jjKpKNj/cOjVgvNya11W3bdKJ y0SA== X-Forwarded-Encrypted: i=1; AKwUvBzGBWyxZfFTHwuJv0PbTdFpk+U4gLxWj6J1YDa0UL02rNs21/B5wuqah+GflgVBWROBcUOtECBik5JChEo=@vger.kernel.org X-Gm-Message-State: AFq9FYIyksWY3mVFEvu2FVDQUq7JmTYsymLJFKc4qUDkKTpUTayevM9R EMZk1gBkVrrLfFeqXaaNg12m3NkTgHsbITbusoku2fCko+B0vCbA42/CeWgME8F1FSGTgfJE6BF AU1OElS+AlFfTJw== X-Received: from ejsh10.prod.google.com ([2002:a17:906:584a:b0:c3f:cc01:4b72]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:2d11:b0:c29:519b:3aee with SMTP id a640c23a62f3a-c402349d091mr109107066b.33.1792094975445; Thu, 15 Oct 2026 13:09:35 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:51 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-21-smostafa@google.com> Subject: [PATCH v9 20/25] iommu/arm-smmu-v3-kvm: Emulate GBPA From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The last bit of emulation is GBPA. it must be always set to ABORT, as when the SMMU is disabled it's not allowed for the host to bypass the SMMU. That's is done by setting the GBPA to ABORT at init time, and host writes are always ignored and host reads always return ABORT. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 32 +++++++++++++++++-- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 1972df1d2a8d..943ffef0bb4a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -217,6 +217,22 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device= *smmu, return smmu_sync_cmd(smmu); } =20 +static int smmu_abort_gbpa(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + u32 reg; + + ret =3D smmu_wait(false, + (readl_relaxed(smmu->base + ARM_SMMU_GBPA) & GBPA_UPDATE) =3D=3D 0); + if (ret) + return ret; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_GBPA); + writel_relaxed(GBPA_UPDATE | GBPA_ABORT | reg, smmu->base + ARM_SMMU_GBPA= ); + return smmu_wait(false, + (readl_relaxed(smmu->base + ARM_SMMU_GBPA) & GBPA_UPDATE) =3D=3D 0); +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -456,6 +472,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_deinit; =20 + ret =3D smmu_abort_gbpa(smmu); + if (ret) + goto out_deinit; + return 0; =20 out_deinit: @@ -781,10 +801,16 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, val =3D smmu->host_ste_cfg; goto out_update_regs; } - /* Passthrough the register access for bisectability, handled later */ case ARM_SMMU_GBPA: - mask =3D read_write; - break; + if (len !=3D sizeof(u32)) + break; + + /* Ignore write, always read to abort. */ + if (!is_write) { + val =3D GBPA_ABORT; + goto out_update_regs; + } + goto out_ret; case ARM_SMMU_CR0: if (len !=3D sizeof(u32)) break; --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EC9D58E2B6 for ; Thu, 15 Oct 2026 20:09:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094983; cv=none; b=lPQ0PYTUfemJISDS2u7oD/iBMeAynOCVvRCzruCaTKk/xrAMt9VY5gTe/FDTtDXHon8mbNOVcXfgG9E5yzyEGCwf3o8Cvk/PAxh6O26m3XFGwtoOQDMGMfKjzFLLCw1fZ3Yb+T5I+mNvmdbb4wgVYh/srFfm3xKkVtTatyy67cA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094983; c=relaxed/simple; bh=imFEZKcawpde6wneHyOMm60Ni0qSyDkx3BQ2dv1g5HY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UNzP/NdKaufzc6uPh0JCZmEw4ZKdHTOS9VUPXClqhog+QWFCNnwW8NKR8uSaCZDh3Yx0PLPAjFTd745DRP3dJH7u5l9nlskWdBaXyX/WYZk5NSBdffp0hrMxG9pFtFXVIVQwOG7Wn5l4n4efcC1CKry/xbqntoSJWeEkz3rEd2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WogQEUcz; arc=none smtp.client-ip=209.85.221.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WogQEUcz" Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-48df02b91f9so173483f8f.1 for ; Thu, 15 Oct 2026 13:09:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094977; x=1792699777; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=l+MxF80V9nS6Nyq45ubbKvlA2fryYNHeqrMVD/2LTb0=; b=WogQEUcz0SqptzZsX68m1Ss19KrsnVJdm+iNJgRuIZRwcwNgLcfW74gtXzaiA/XMRJ xsaTjMw8yoAx2Mnrhi5gRFT9KgzQYY+bqMA88J7rJWuhHpxpcD0R0M308NOnNEhy6rXT v7mxIRtLrWu0JUO1NLOknGsYPFc846uTss07euUB01E0gdgCxKjyVW/jcQa4emQgPGb8 XIT3KbtY59lAaA29jiHXgTgOxFl2PNMiNiU0jWbwMqZJZzUjtjgCZl3bIy1IPzzlDiOx S/cSkkoeYCzlGJ1DSy229d6Bn2mu8WblKhnU7DKx7YsrabrJXlOk2+XB147gwAUOkvPz vM7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094977; x=1792699777; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l+MxF80V9nS6Nyq45ubbKvlA2fryYNHeqrMVD/2LTb0=; b=AP0Tzcsr/AYsEDZ0/V2iQ3iQNOWfZKjrQPlezyRVb9DyOJNKbUjl3hRDucy7FeJV0L Y2r5p5yfkAtv290WJaOGYX14G/CB4KaHit2Z2VUS1Z47oG/vKQqqJzW5Cc5UT4cv+zZz OHG5lK095wpu1hM3mM4TYCcSJzxRId7TRQqphxhYaVSQ2Y5w1RHgV3zmvJmdKvThmgpg AFdI1u2njdijBnFfPagLL5kuAqrbjBMjW3bbtxAko4A9Ss8sHW+ZqF3YXoikX8hze7My T52EFuHGS+8R7Pv07uP7zydV1uMe8mvZG9P4PTzvGoPE9DUk+PhpGTr2KXPr/wuIlG1H WXuA== X-Forwarded-Encrypted: i=1; AKwUvBwZNSnbJo+83FLqtXxFXdk7EbS2h5dgWZPlWj00nesrkZunQ9ckeewFfjIg9AMExgbOvZKn7Ij80S1JBwc=@vger.kernel.org X-Gm-Message-State: AFq9FYL4AXcKE6jdEW9VJLJeg+vTnKeI+YWt4pxXkgINjQsAvF3Wp0ac Qu4BHTwRv6tFXySXiFxOs7S+BAN49Qb6hg5Shtu/zyXDmFCMml8lrLdTXKRhTJ8k9gNdvpQe2Zw 0FbXbZSwtNZpZRw== X-Received: from wrzh2.prod.google.com ([2002:a5d:6e02:0:b0:48d:f498:dec8]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:6292:0:b0:48c:6756:dab3 with SMTP id ffacd0b85a97d-48dfacf53famr2128690f8f.3.1792094976702; Thu, 15 Oct 2026 13:09:36 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:52 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-22-smostafa@google.com> Subject: [PATCH v9 21/25] iommu/io-pgtable-arm: Support io-pgtable-arm in the hypervisor From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To be able to populate the shadow stage-2 in the hypervisor, compile io-pgtable-arm object for the hypervisor. That requires the hypervisor to provide the iommu-pages allocator, redefine the address conversion helpers, and stub out the DMA API. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 3 +- .../arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h | 88 +++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 14 +-- drivers/iommu/io-pgtable-arm.h | 9 ++ 4 files changed, 106 insertions(+), 8 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index ee86e4b3436f..08da05ddda5d 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -36,7 +36,8 @@ hyp-obj-y +=3D $(lib-objs) HYP_SMMU_V3_DRV_PATH :=3D ../../../../../drivers/iommu/arm/arm-smmu-v3 =20 hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3-hyp.o \ - $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o ../../../../../lib/hweig= ht.o + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o ../../../../../lib/hweig= ht.o \ + $(HYP_SMMU_V3_DRV_PATH)/../../io-pgtable-arm.o =20 # Path to simple_ring_buffer.c CFLAGS_trace.nvhe.o +=3D -I$(srctree)/kernel/trace/ diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h b/driv= ers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h new file mode 100644 index 000000000000..d278c6788fb1 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef IO_PGTABLE_ARM_HYP_H_ +#define IO_PGTABLE_ARM_HYP_H_ + +#include +#include + +#undef WARN_ONCE +#define WARN_ONCE(condition, format...) WARN_ON(condition) + +static inline void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t s= ize) +{ + return pkvm_iommu_alloc_pages(get_order(size)); +} + +static inline void iommu_free_pages(void *virt) +{ + pkvm_iommu_free_pages(virt); +} + +#define arm_lpae_alloc_data(obj) pkvm_iommu_alloc_pages(get_order(sizeof(o= bj))) +#define arm_lpae_free_data(p) pkvm_iommu_free_pages(p) + +#undef phys_to_virt +#define phys_to_virt(x) hyp_phys_to_virt(x) + +#undef virt_to_phys +#define virt_to_phys(x) hyp_virt_to_phys(x) + +/* Stubs used by io-pgtable-arm but no relevant for hypervisor object. */ +#undef dma_map_single +#define dma_map_single(d, p, s, dir) \ + ({ (void)(d); (void)(s); (void)(dir); virt_to_phys(p); }) + +#undef dma_unmap_single +#define dma_unmap_single(d, a, s, r) \ + do { (void)(d); (void)(a); (void)(s); (void)(r); } while (0) + +#define dma_sync_single_for_device(d, a, s, r) \ + do { (void)(d); (void)(a); (void)(s); (void)(r); } while (0) + +#undef dma_mapping_error +#define dma_mapping_error(...) (0) + +#undef dev_err +#define dev_err(dev, format, ...) \ + ({ (void)(dev); no_printk(format, ##__VA_ARGS__); }) + +#undef iommu_dirty_bitmap_record +#define iommu_dirty_bitmap_record(d, i, l) \ + do { (void)(d); (void)(i); (void)(l); } while (0) + +static inline int iommu_pages_start_incoherent(void *virt, struct device *= dma_dev) +{ + return 0; +} + +static inline void iommu_pages_free_incoherent(void *virt, + struct device *dma_dev) +{ +} + +static inline void iommu_pages_flush_incoherent(struct device *dma_dev, + void *virt, size_t offset, + size_t len) +{ +} + +static inline struct io_pgtable_ops *pkvm_alloc_io_pgtable_ops(enum io_pgt= able_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct io_pgtable *iop; + + if (fmt !=3D ARM_64_LPAE_S2) + return NULL; + + iop =3D io_pgtable_arm_64_lpae_s2_init_fns.alloc(cfg, cookie); + if (!iop) + return NULL; + + iop->fmt =3D fmt; + iop->cookie =3D cookie; + iop->cfg =3D *cfg; + + return &iop->ops; +} +#endif /* IO_PGTABLE_ARM_HYP_H_ */ diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 476c0e25631a..6b2668c2ef44 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -20,7 +20,6 @@ #include =20 #include "io-pgtable-arm.h" -#include "iommu-pages.h" =20 #define ARM_LPAE_MAX_ADDR_BITS 52 #define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 @@ -301,6 +300,7 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t = gfp, { struct device *dev =3D cfg->iommu_dev; void *pages; + int nid =3D dev ? dev_to_node(dev) : NUMA_NO_NODE; =20 /* * For very small starting-level translation tables the HW requires a @@ -311,7 +311,7 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t = gfp, if (cfg->alloc) return __arm_lpae_cfg_alloc(size, gfp, cfg, cookie); =20 - pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, size); + pages =3D iommu_alloc_pages_node_sz(nid, gfp, size); if (!pages) return NULL; =20 @@ -650,7 +650,7 @@ static void arm_lpae_free_pgtable(struct io_pgtable *io= p) struct arm_lpae_io_pgtable *data =3D io_pgtable_to_data(iop); =20 __arm_lpae_free_pgtable(data, data->start_level, data->pgd); - kfree(data); + arm_lpae_free_data(data); } =20 static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, @@ -964,7 +964,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg) if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS) return NULL; =20 - data =3D kmalloc_obj(*data); + data =3D arm_lpae_alloc_data(*data); if (!data) return NULL; =20 @@ -1091,7 +1091,7 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *c= fg, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + arm_lpae_free_data(data); return NULL; } =20 @@ -1187,7 +1187,7 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *c= fg, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + arm_lpae_free_data(data); return NULL; } =20 @@ -1266,7 +1266,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + arm_lpae_free_data(data); return NULL; } =20 diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index ba7cfdf7afa0..4278da41e406 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -27,4 +27,13 @@ #define ARM_LPAE_TCR_PS_48_BIT 0x5ULL #define ARM_LPAE_TCR_PS_52_BIT 0x6ULL =20 +#ifndef __KVM_NVHE_HYPERVISOR__ +#include "iommu-pages.h" +#define arm_lpae_alloc_data(obj) kmalloc_obj(obj) +#define arm_lpae_free_data(p) kfree(p) + +#else +#include "arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.h" +#endif + #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC7D4598BFA for ; Thu, 15 Oct 2026 20:09:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094984; cv=none; b=AV/4NjoeUzZ0hKalqHHMWZPjnAws79XGbkl8EduyiSVv0AUCmrSN6hgMFUglYEwhUMRReXONsb9k52A8pNQCXud1ht7KbcuvZcF6xh1hsLQ/aEVXSaYT1g7QPoONF/2qL+4f/9qp3Nx9DMfku7HjhAhCoTqJYxue/t/fHTLiUgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094984; c=relaxed/simple; bh=8tg9o3qv8IiFhlcVCCrsusGvG3Ox1GG+zDGl5YrpDKQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mL7xDwWyr6ORoVAsU9R2T6lxMXhnn801Z0EjC7CEQ7EgdayJOkNiKXeuqsLMDvxiuAQKxOT16V7/xPVCA0Zzbxql1/JX7o7L7CP9n0LI59Jc62ojizpwILz6fwxCtWEA+nfExw6Ue+w9cwZSlgtjYiswsVnFznfU3ylDsnlvUMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A84as8Vz; arc=none smtp.client-ip=209.85.128.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A84as8Vz" Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4a01c621dc2so43865895e9.2 for ; Thu, 15 Oct 2026 13:09:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094978; x=1792699778; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=uOWI3MWzFM4E6XsnBaoMm2oO1l7gBBooy2wESuq6Qac=; b=A84as8VzYLhj0WTUE+g7j6Ny3JtdbLC808KJSY+eSzPEb3JMxwSEdXfX9uhuKhpaUd d6RGloXCapwQOEuEUSqQGnlsd9GeQ5yc9seWI6O6F5CT9eyPNF9oIA5rTObG/C115mkE j7smvO7TywSkx91gyJZR2zZlowp1o7iD+gbEc1UvsC/BfNXtlhqP4f8XLF+iCu3t3a64 fTD0tL1Lpey7L2pnGY3XJK2sTiU3cddgw9sRba1BHKPMgGpEmMUgQ2sNmuxYYpi9o9Ke aw7QsqRyUGb3Cvh0p3hkyF3isABfhqOoI25ND4LtJyDwcDJManEU0c6PhrHG3o9x7BJb Ip2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094978; x=1792699778; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uOWI3MWzFM4E6XsnBaoMm2oO1l7gBBooy2wESuq6Qac=; b=n7Emx4k7qZX0gQs+FllMWkOyaeVime0Bfmk/qlKKR5i8hjPU62o9Et3gXzrATlsXjv QoMYK35qPEeRHB+IAng2jtzABtJWOKiuQ5XeARVBy2/uMkZfiJI/IGp1JBCCcXhweA+s qOkdGkMiV+RsBYWnWmMeSxpmWrq8zap6yivMLgShTBA3qHDIk2MSKt/ikc13srlfxXwN NFLTPoT7/rE00xKf1S/ePl6NxcGgRiNB5TTjeHbf4D7F81+AbzQyQ6i1I+MI9R1JCxv6 KL8KBgQsfjuFwHtAI7E9squHSbObciMlmmV6WnA5P9UoyfdR1lgxE7zkmy301dVQAzJO 3IKg== X-Forwarded-Encrypted: i=1; AKwUvBwDxLpHNsZQv5SSWKslOWmdjsiAgNtA9zX8DihIVhhY8szeDapnImnXFvumz1uwswCiNfBv1S7I+YknLpU=@vger.kernel.org X-Gm-Message-State: AFq9FYJSs/1i82SWUtAdGQqhvefW7r+FadO9VrQGf9obGun3LuskaXmj u+dWhLQ6DcnFcDuBVX2/27241sQrFGP3jE7ypd21EldkH1OL26iYRDKfcXpSqDDR33nq8/xGDQF oVny2IC5bKeX9ng== X-Received: from wmjs9.prod.google.com ([2002:a7b:c389:0:b0:4a1:8be5:4018]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3e0a:b0:4a0:1c10:1419 with SMTP id 5b1f17b1804b1-4a18e4cd667mr404934305e9.26.1792094977861; Thu, 15 Oct 2026 13:09:37 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:53 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-23-smostafa@google.com> Subject: [PATCH v9 22/25] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The hypervisor calls back into the driver on every change to the host stage-2 page table. Mirror those changes into an identity mapped stage-2 for the SMMUv3, That would be attached to all active SIDs. Difference between memory and MMIO handling: - Memory is always mapped with PAGE_SIZE. io-pgtable-arm no longer supports split_blk_unmap, so a block cannot be broken into a table once it is mapped, and pages are donated back and forth at PAGE_SIZE granularity. The page table pool is sized to cover all of memory at that granularity. - MMIO is mapped with the largest block that fits, as it is assumed to cover the whole IAS that is not memory while pKVM only reserves 1G for the page table. MMIO is never donated at runtime, so it is never unmapped and never needs a block to be split. The TLB maintenance ops are left as stubs, they are implemented in the next patch. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 153 +++++++++++++++++- 1 file changed, 152 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 943ffef0bb4a..75b2b1477a4f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -15,6 +15,9 @@ #include "../arm-smmu-v3.h" #include "../arm-smmu-v3-common-lib.h" =20 +#include +#include "../../../io-pgtable-arm.h" + size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 @@ -72,6 +75,9 @@ static inline u64 hyp_clock_ns(void) __ret; \ }) =20 +/* Protected by host_mmu.lock from core code. */ +static struct io_pgtable *idmap_pgtable; + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -217,6 +223,24 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device= *smmu, return smmu_sync_cmd(smmu); } =20 +static void smmu_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + /* TBD: Invalidate the range in all the SMMUs. */ +} + +static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ + /* TBD: Invalidate the granule in all the SMMUs. */ +} + +static const struct iommu_flush_ops smmu_tlb_ops =3D { + .tlb_flush_walk =3D smmu_tlb_flush_walk, + .tlb_add_page =3D smmu_tlb_add_page, +}; + static int smmu_abort_gbpa(struct hyp_arm_smmu_v3_device *smmu) { int ret; @@ -483,6 +507,38 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) return ret; } =20 +static int smmu_init_pgt(void) +{ + /* Default values overridden based on SMMUs common features. */ + struct io_pgtable_cfg cfg =3D (struct io_pgtable_cfg) { + .tlb =3D &smmu_tlb_ops, + .pgsize_bitmap =3D ~0UL, + .ias =3D 48, + .oas =3D 48, + .coherent_walk =3D true, + .quirks =3D IO_PGTABLE_QUIRK_NO_WARN, + }; + struct hyp_arm_smmu_v3_device *smmu; + struct io_pgtable_ops *ops; + + for_each_smmu(smmu) { + cfg.ias =3D min(cfg.ias, smmu->oas); + cfg.oas =3D min(cfg.oas, smmu->oas); + cfg.pgsize_bitmap &=3D smmu->pgsize_bitmap; + cfg.coherent_walk &=3D !!(smmu->features & ARM_SMMU_FEAT_COHERENCY); + } + + /* At least PAGE_SIZE must be supported by all SMMUs */ + if ((cfg.pgsize_bitmap & PAGE_SIZE) =3D=3D 0) + return -EINVAL; + + ops =3D pkvm_alloc_io_pgtable_ops(ARM_64_LPAE_S2, &cfg, NULL); + if (!ops) + return -ENOMEM; + idmap_pgtable =3D io_pgtable_ops_to_pgtable(ops); + return 0; +} + /* Called while the host is still trusted. */ static int smmu_init(void) { @@ -513,7 +569,10 @@ static int smmu_init(void) goto out_reclaim_smmu; } =20 - return 0; + ret =3D smmu_init_pgt(); + if (ret) + goto out_reclaim_smmu; + return ret; =20 out_reclaim_smmu: while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) @@ -1011,8 +1070,100 @@ static bool smmu_dabt_handler(struct user_pt_regs *= regs, u64 esr, u64 addr) return false; } =20 +static size_t smmu_pgsize_idmap(size_t size, u64 paddr, size_t pgsize_bitm= ap) +{ + size_t pgsizes; + + /* Remove page sizes that are larger than the current size */ + pgsizes =3D pgsize_bitmap & GENMASK_ULL(__fls(size), 0); + + /* Remove page sizes that the address is not aligned to. */ + if (likely(paddr)) + pgsizes &=3D GENMASK_ULL(__ffs(paddr), 0); + + WARN_ON(!pgsizes); + + /* Return the largest page size that fits. */ + return BIT(__fls(pgsizes)); +} + static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) { + size_t pgsize =3D PAGE_SIZE, pgcount, size; + struct io_pgtable *pgtable =3D idmap_pgtable; + int ret =3D 0; + + end =3D min(end, BIT(pgtable->cfg.oas)); + if (start >=3D end) + return 0; + + size =3D end - start; + if (prot) { + size_t mapped; + + if (!(prot & IOMMU_MMIO)) + prot |=3D IOMMU_CACHE; + + while (size) { + mapped =3D 0; + /* + * We handle pages size for memory and MMIO differently: + * - memory: Map everything with PAGE_SIZE, that is guaranteed to + * find memory as we allocated enough pages to cover the entire + * memory, we do that as io-pgtable-arm doesn't support + * split_blk_unmap logic any more, so we can't break blocks once + * mapped to tables. + * - MMIO: Unlike memory, pKVM allocates 1G for all MMIO, while + * the MMIO space can be large, as it is assumed to cover the + * whole IAS that is not memory, we have to use block mappings, + * that is fine for MMIO as it is never donated at the moment, + * so we never need to unmap MMIO at the run time triggering + * split block logic. + */ + if (prot & IOMMU_MMIO) + pgsize =3D smmu_pgsize_idmap(size, start, pgtable->cfg.pgsize_bitmap); + + pgcount =3D size / pgsize; + ret =3D pgtable->ops.map_pages(&pgtable->ops, start, start, + pgsize, pgcount, prot, 0, &mapped); + size -=3D mapped; + start +=3D mapped; + + if (ret =3D=3D -EEXIST) { + /* + * It is possible to get EEXIST when a VM dies with pages + * in a shared state. + */ + ret =3D 0; + size -=3D pgsize; + start +=3D pgsize; + continue; + } + if (!mapped || ret) + break; + } + } else { + struct iommu_iotlb_gather gather; + size_t unmapped; + + while (size) { + pgcount =3D size / pgsize; + iommu_iotlb_gather_init(&gather); + unmapped =3D pgtable->ops.unmap_pages(&pgtable->ops, start, + pgsize, pgcount, &gather); + size -=3D unmapped; + start +=3D unmapped; + if (!unmapped) + break; + } + } + + if (ret) + return ret; + + if (WARN_ON(size)) + return -EINVAL; + return 0; } =20 --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-ed1-f72.google.com (mail-ed1-f72.google.com [209.85.208.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71EAA598C15 for ; Thu, 15 Oct 2026 20:09:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094986; cv=none; b=rf7DHIf6ex16qOBys1c/rw7Ny+HEXWxoFFa3AwgbsQIH0+j6QoPy3i89j/fANfi91I983MYt/fjOuFKb/k0UbXUe9k4sTNHxyggZqsgYdu+NuYILIV8rAmfCMAHTRdKKcJHUryOtWI0HdLd9SxdcBItB05gXKfTJF50eTr+kDYU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094986; c=relaxed/simple; bh=pax5OLPSUo+ZBv6oi2uayg8Bdmkx6QpeOtfiynqJo64=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rOtgydWF0rKfmpZzaxXGzAKTbXM73d09aalHoryPZ7lRRiJrHkjmyC9iWn0WCy20nvjLaBbMqAhJI7/WHrGONiuvEhTM04qgtsRExocuGhOg3MS9g1UruRzzb61s4UD2YuGkwg+bKfXlx1l74luOUO/aW0WwjPD6RNstQt9BrRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=arjh15ux; arc=none smtp.client-ip=209.85.208.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="arjh15ux" Received: by mail-ed1-f72.google.com with SMTP id 4fb4d7f45d1cf-6b1a56fd429so2812782a12.3 for ; Thu, 15 Oct 2026 13:09:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094979; x=1792699779; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=9VVZGlsZIqsR8XCbbkxUjFMNMhFH8B9fEpFvm5P7Qik=; b=arjh15ux3Ylh33eX1uwwhkNo4tvLSzq/XzdO8gG8aGiZ0jTWrk+boELy2gZKY5ej4D 65Lcwq+TGX/nfUP3UeZRMRr3tIwaAvaYAhgZfSCTB4dj/5ky/lNEYObAlfRdrf/JKl6b yoKRnv860Ho7Zcu1YfxFxY1Y3RiVZbN4lAxZGtSerL8WOXDnp4ErGoc46Ydfj0nIGaYc 1Lv4nAJ9DX2bzoDG7IXJ+gPXSW4lmwj1VVpvIqQF5a+ELGsBinJDvnIsPe4Uf9hnEYEi u1mflVUzW3hjXqvumCOoMzphv8fQLFvWUpTJcYfPJrexiSc27v3p4+qkFKBx2Fmzze1l jMIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094979; x=1792699779; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=9VVZGlsZIqsR8XCbbkxUjFMNMhFH8B9fEpFvm5P7Qik=; b=zXTuEUCGMN+zrTadvVYjtfuxT3+6q7W92OnVRVS/WDNK/LWBbdvQrrWEY8zUjXyvqE Ulw6NCNwR3nzRz0K2E7cBDZ7akrKNwnpk0mfu9EoXY9tmuZC7LEUaBYQJNOocDXbWDPP iF6ReX0p6Bc+DIDM/z6QnUJcT7GD2Yio08AksYlLwZHDjAKeygAMiiMpMjCkKa2bVcOX phvhYaa4vckQ6wLdwCEQJ1DMr9G+7idMnrxULTjn3Yl6ZE7mMALDKUyFvP1KMMntikep uGB8ULxu3vgN1w/I+ZIUDpwcwRU0tGaHFDNJWBrRdzJLsSJIBZB0sLvr42MzHhweUF6C +bNw== X-Forwarded-Encrypted: i=1; AKwUvBy1mJBStGeOSBZBZ2YXoLOToU1jdWor+Dpk8Hvci51Q9txsTDFsSS0H/keXW3clUYp0iQT56t1LhgUsnO8=@vger.kernel.org X-Gm-Message-State: AFq9FYJLzDuMdvBo4rBXBZqnBklAm3DAE6A7G3aRQSNHVfV2hKZE6sCh GYZ8PkdzeD/pd/X5kHXO5JGuF8vHKmIAi8U/Rc06dCvDyi26aLVh4R6Aa+Vo1n2Tj/yyND46Zig eV1IonuxFfzgI3Q== X-Received: from edze9.prod.google.com ([2002:a05:6402:1909:b0:6ae:201f:ff33]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:2498:b0:6aa:ee55:a650 with SMTP id 4fb4d7f45d1cf-6b17c27c6e1mr15665392a12.37.1792094979040; Thu, 15 Oct 2026 13:09:39 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:54 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-24-smostafa@google.com> Subject: [PATCH v9 23/25] iommu/arm-smmu-v3-kvm: Invalidate the SMMU TLBs From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement the io-pgtable tlb ops, which invalidate the shadow stage-2 in every SMMU. The range invalidation commands come from arm_smmu_tlbi_calc_range() in the common lib, SMMUs without range invalidation get one command per entry. The hypervisor also needs to invaldiate all of S1 after S2 invalidations by IPA to invalidate combined entries. Invalidation needs the command queue, so track whether it is enabled in cmdq_active. It is updated with hw_lock held once the CR0 write has been acknowledged, so it cannot change under a concurrent invalidation. For the same reason the host is not allowed to disable the command queue while the SMMU is enabled, as the hypervisor would no longer be able to invalidate. As invalidation is skipped while the command queue is off, flush the TLBs every time the SMMU is enabled. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 156 +++++++++++++++++- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h | 2 + 2 files changed, 155 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 75b2b1477a4f..01f23fce6be5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -209,7 +209,6 @@ static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device = *smmu) return smmu_wait(false, smmu_cmdq_empty(&smmu->cmdq)); } =20 -__maybe_unused static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, struct arm_smmu_cmd *cmd) { @@ -223,17 +222,118 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_devi= ce *smmu, return smmu_sync_cmd(smmu); } =20 +static int smmu_tlb_inv_vmid(struct hyp_arm_smmu_v3_device *smmu) +{ + struct arm_smmu_cmd cmd =3D arm_smmu_make_cmd_op(CMDQ_OP_TLBI_S12_VMALL); + + return smmu_send_cmd(smmu, &cmd); +} + +static int __smmu_tlb_inv_range(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmd *cmd, + unsigned long iova, size_t size, size_t granule, + bool leaf) +{ + struct arm_smmu_tlbi tlbi =3D { + .iova =3D iova, + .size =3D size, + .iopte_size =3D granule, + .tgsz_lg2 =3D __ffs(idmap_pgtable->cfg.pgsize_bitmap), + .leaf_only =3D leaf, + }; + unsigned int i; + int ret; + + arm_smmu_tlbi_calc_range(&tlbi, false); + if (tlbi.range.use_full_inv) + return smmu_tlb_inv_vmid(smmu); + + for (i =3D 0; i < tlbi.range.num_cmds; i++) { + struct arm_smmu_cmd range_cmd =3D tlbi.range.cmds[i]; + + range_cmd.data[0] |=3D cmd->data[0]; + ret =3D smmu_add_cmd(smmu, &range_cmd); + if (ret) + return ret; + } + + return smmu_sync_cmd(smmu); +} + +static int __smmu_tlb_inv_range_pages(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmd *cmd, + unsigned long iova, size_t size, size_t granule, + bool leaf) +{ + unsigned long end =3D iova + size; + int ret; + + /* Same threshold as arm_smmu_tlbi_calc_single(). */ + if (size / granule > 512) + return smmu_tlb_inv_vmid(smmu); + + for (; iova < end; iova +=3D granule) { + cmd->data[1] =3D FIELD_PREP(CMDQ_TLBI_1_LEAF, leaf) | + (iova & ~GENMASK_U64(11, 0)); + ret =3D smmu_add_cmd(smmu, cmd); + if (ret) + return ret; + } + + return smmu_sync_cmd(smmu); +} + +static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu, + unsigned long iova, size_t size, size_t granule, + bool leaf) +{ + struct arm_smmu_cmd cmd_s1 =3D arm_smmu_make_cmd_op(CMDQ_OP_TLBI_NH_ALL); + struct arm_smmu_cmd cmd =3D arm_smmu_make_cmd_op(CMDQ_OP_TLBI_S2_IPA); + int ret; + + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) + ret =3D __smmu_tlb_inv_range(smmu, &cmd, iova, size, granule, leaf); + else + ret =3D __smmu_tlb_inv_range_pages(smmu, &cmd, iova, size, granule, leaf= ); + if (ret) + return ret; + + return smmu_send_cmd(smmu, &cmd_s1); +} + +static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t gra= nule, + bool leaf) +{ + struct hyp_arm_smmu_v3_device *smmu; + + for_each_smmu(smmu) { + hyp_spin_lock(&smmu->hw_lock); + /* + * Don't bother if CMDQ is disabled, this would be useful for the case + * when RPM is supported to avoid touching the SMMU MMIO when disabled. + * The hypervisor also asserts CMDQEN is enabled before the SMMU is + * enabled. As otherwise the host can prevent the hypervisor from doing + * TLB invalidations. + * When the SMMU is re-enabled the hypervisor cleans the TLBs. + */ + if (smmu->cmdq_active) + WARN_ON(smmu_tlb_inv_range_smmu(smmu, iova, size, + granule, leaf)); + hyp_spin_unlock(&smmu->hw_lock); + } +} + static void smmu_tlb_flush_walk(unsigned long iova, size_t size, size_t granule, void *cookie) { - /* TBD: Invalidate the range in all the SMMUs. */ + smmu_tlb_inv_range(iova, size, granule, false); } =20 static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather, unsigned long iova, size_t granule, void *cookie) { - /* TBD: Invalidate the granule in all the SMMUs. */ + smmu_tlb_inv_range(iova, granule, granule, true); } =20 static const struct iommu_flush_ops smmu_tlb_ops =3D { @@ -721,6 +821,33 @@ static int smmu_update_ste_shadow(struct hyp_arm_smmu_= v3_device *smmu, bool enab return smmu_unshare_pages(strtab_host_base(smmu), size); } =20 +static int smmu_flush_all_tlb(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + u32 cr0; + struct arm_smmu_cmd cmd =3D arm_smmu_make_cmd_op(CMDQ_OP_TLBI_NSNH_ALL); + + hyp_spin_lock(&smmu->hw_lock); + /* + * This must be called when the SMMU is getting enabled. + * First enable the cmdq and then invalidate the TLB. + */ + cr0 =3D readl_relaxed(smmu->base + ARM_SMMU_CR0); + if (!(cr0 & CR0_CMDQEN)) { + cr0 |=3D CR0_CMDQEN; + writel_relaxed(cr0, smmu->base + ARM_SMMU_CR0); + ret =3D smmu_wait(false, readl_relaxed(smmu->base + ARM_SMMU_CR0ACK) =3D= =3D cr0); + if (ret) { + hyp_spin_unlock(&smmu->hw_lock); + return ret; + } + } + + ret =3D smmu_send_cmd(smmu, &cmd); + hyp_spin_unlock(&smmu->hw_lock); + return ret; +} + static void smmu_emulate_enable(struct hyp_arm_smmu_v3_device *smmu) { /* Enabling SMMU without CMDQ, means TLB invalidation won't work. */ @@ -728,6 +855,8 @@ static void smmu_emulate_enable(struct hyp_arm_smmu_v3_= device *smmu) return; =20 WARN_ON(smmu_update_ste_shadow(smmu, true)); + /* Clean the TLBs each time the SMMU is enabled. */ + WARN_ON(smmu_flush_all_tlb(smmu)); } =20 static void smmu_emulate_disable(struct hyp_arm_smmu_v3_device *smmu) @@ -748,6 +877,13 @@ static void smmu_emulate_cmdq_enable(struct hyp_arm_sm= mu_v3_device *smmu) =20 static void smmu_emulate_cmdq_disable(struct hyp_arm_smmu_v3_device *smmu) { + /* + * We can not enable the SMMU if the CMDQ is disabled and similarly + * we can not disable the CMDQ if the SMMU is enabled, as that can + * lead to stale TLBs. + */ + WARN_ON(is_smmu_enabled(smmu)); + WARN_ON(smmu_unshare_pages(smmu->cmdq_host.base_dma, cmdq_size(&smmu->cmdq_host))); } @@ -1033,6 +1169,20 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_= device *smmu, else writel_relaxed(val & mask, smmu->base + off); =20 + /* + * Make sure writes to CR0 are immediately observed, that is important + * when synchronizing with TLB invalidation as reading CR0 is not enough + * to deduce the SMMU state, and we have to enforce the ack with the + * hw_lock acquired. + */ + if (off =3D=3D ARM_SMMU_CR0) { + u32 cr0 =3D val; + + WARN_ON(smmu_wait(false, + readl_relaxed(smmu->base + ARM_SMMU_CR0ACK) =3D=3D cr0)); + smmu->cmdq_active =3D !!(cr0 & CR0_CMDQEN); + } + hyp_spin_unlock(&smmu->hw_lock); return true; } diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h index 2fb478e9233a..9ccee7fb710f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.h @@ -27,6 +27,7 @@ * @cmdq_host: Host view of the CMDQ, only q_base, base_dma and llq used * @cmdq_max_shift: Max shift for the CMDQ probed from HW * @cr0: Last value of CR0 + * @cmdq_active: CMDQEN as acknowledged by the hardware, under hw_lock * @host_ste_cfg: Host stream table config * @host_ste_base: Host stream table base * @strtab_cfg: Stream table config as seen by HW @@ -60,6 +61,7 @@ struct hyp_arm_smmu_v3_device { struct arm_smmu_queue cmdq_host; u32 cmdq_max_shift; u32 cr0; + bool cmdq_active; dma_addr_t strtab_dma; size_t strtab_size; u64 host_ste_cfg; --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F6FC599A24 for ; Thu, 15 Oct 2026 20:09:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094989; cv=none; b=kRa9ZiSjxW9aagWK37YieQiQKlhn88Nlmgw9OXgnfzGtfggr9MB+M9kdzw81ityyxYwBWJE/8jl2/l9EqJpqy4/QWSrDqoSqXIEcD8jecPcm7Ycp6g9fClgze9KTJlSnGfS/i2pz70+gtOfQ/Fc6aM5R71oVx5KBQp/GOh+Vf5Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094989; c=relaxed/simple; bh=tBz432yZOVanKRViYqmo2mhM8D8MdHrMDagSLSsS7Ac=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gCRlwi/C4uEV3073FnHJmw/HNL6xAQjf4MjCApLbRShKuUPxO3mVIS/M1nHelSR1qpNpH3920S7MUrrlVSbUbogGYw4/sMAI98JwwWf9ytuYMcLqI4qSD4azkE5nqeQnU0rPd6WrqlT18bPC70IzTOX/VzA2W6YwkGsWZIwTRrc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=r7x7ztuU; arc=none smtp.client-ip=209.85.221.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="r7x7ztuU" Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-48af5a07314so3622099f8f.2 for ; Thu, 15 Oct 2026 13:09:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094980; x=1792699780; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=oXSBGll6V5AX0HAYiKE1Rqp8ZLXo/cvuVZr8HoLMmEg=; b=r7x7ztuUiOQNNtrRJ2IpLQc+yP2MgVRM3mxDHCaEZWehLsPNviB/WIB7WvUxyzd9wx iJ6rSyaJlj3Ax8YjhL9GyCDEmI49GxYzb3mCErY5QtQdPGpghA33RRE5omDmLnY0RKIg VYwTicPfX6YYUJSVjzZzqAd1R3EuZJ/FY46lBuVjPrE6VEyLqXzxdRoVL9GBe8ucfPeC sxtTKZOE1/Rv6z7G1R/ZDMEc6Ynd63SewGejHqQtGInYTwavACrYTeXccA6Gr2YqIPLX tYQOcCCQ4jMUx69wAZHRqWw/U3PnhGU6rsyjcXWGAN8dv5p3ni/5JMdJuHu8AmC+E0Br EFHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094980; x=1792699780; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=oXSBGll6V5AX0HAYiKE1Rqp8ZLXo/cvuVZr8HoLMmEg=; b=OcjoR1sh8/PdFBt3ifWsY69hsR1RtVeFsFhKFMf7XxvjKcl++1sGm20incd0DMXKNP j/QORS8jp2LNq4RR7GytM7wli5j5/+7sR/wo4LDtM6KBlH2Fu/0XFi/Pxg1nP1Zbrj+p +QEHWqijmBMvTdF5/rKNwJ6pjecE7h1eknHnvGPayxinxMI+jgpR+CCy69VXfLeWoARt wWNZdgndWaTJMKoEGX9hhiEWlrGVoYPxyXwb2DPlafKZ6D2HZsr1HVtnW01QGCU0K3HH QOJFiTsI4kqPndIF9JNNYDG1IqSifUkqhG3tJJ0M+jKbY10IXc3ofwOL687KJC7h2aJp NiwQ== X-Forwarded-Encrypted: i=1; AKwUvBzhf45eoZzVjzYYTGL02Gwj1MEBSUkzMFbq93S/Wpc5aUUQ8/8CkTKsNpqk5WyojMjnjuWoVNDiTfuCLv0=@vger.kernel.org X-Gm-Message-State: AFq9FYI9Y50XJ/87qTJIEl3PtEKArRtcBA/ntdYpn5/oOZlqroAPUH95 RxjG+koIJCrUR2EZ7/UauteW4PCDCxMhu3WlxMtnoPnSFQA92yM8xexvMhzmxa3TCyE9EoELuQt ow+8fYq3x7RqlPA== X-Received: from wmjs1.prod.google.com ([2002:a7b:c381:0:b0:4a1:ae53:2feb]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4fcb:b0:4a0:1fe2:d4f1 with SMTP id 5b1f17b1804b1-4a18e4d1686mr397679435e9.28.1792094980042; Thu, 15 Oct 2026 13:09:40 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:55 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-25-smostafa@google.com> Subject: [PATCH v9 24/25] iommu/arm-smmu-v3-kvm: Enable nesting From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As now the hypervisor controls the command queue, stream table, and shadows the stage-2 page table, we enable stage-2 translation in case the host puts an STE in bypass or stage-1. The hypervisor will control the S2 part of the STE and sanitize it. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c | 112 +++++++++++++++++- 1 file changed, 106 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c b/drivers= /iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c index 01f23fce6be5..3562bb222f53 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3-hyp.c @@ -443,6 +443,60 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_attach_stage_2(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_ste *ste) +{ + unsigned long vttbr, cfg; + unsigned long ts, sl, ic, oc, sh, tg, ps; + struct io_pgtable_cfg *pgt_cfg =3D &idmap_pgtable->cfg; + + cfg =3D FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ste->data[0])); + if (!FIELD_GET(STRTAB_STE_0_V, le64_to_cpu(ste->data[0])) || + (cfg =3D=3D STRTAB_STE_0_CFG_ABORT)) { + return 0; + } + /* S2 is not advertised, that should never be attempted. */ + if ((cfg =3D=3D STRTAB_STE_0_CFG_S2_TRANS) || + (cfg =3D=3D STRTAB_STE_0_CFG_NESTED)) + return -EINVAL; + vttbr =3D pgt_cfg->arm_lpae_s2_cfg.vttbr; + ps =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.ps; + tg =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tg; + sh =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sh; + oc =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.orgn; + ic =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.irgn; + sl =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sl; + ts =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tsz; + + if (smmu->features & ARM_SMMU_FEAT_ATTR_TYPES_OVR) { + ste->data[1] &=3D ~cpu_to_le64(STRTAB_STE_1_SHCFG); + ste->data[1] |=3D cpu_to_le64(FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_= 1_SHCFG_INCOMING)); + } + + ste->data[1] &=3D ~cpu_to_le64(STRTAB_STE_1_EATS | STRTAB_STE_1_S2FWB | S= TRTAB_STE_1_STRW); + + /* The host shouldn't write dwords 2 and 3, overwrite them. */ + ste->data[2] =3D cpu_to_le64(FIELD_PREP(STRTAB_STE_2_VTCR, + FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, ps) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, tg) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SH0, sh) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2OR0, oc) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2IR0, ic) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, sl) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, ts)) | + FIELD_PREP(STRTAB_STE_2_S2VMID, 0) | + STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R | +#ifdef __BIG_ENDIAN + STRTAB_STE_2_S2ENDI | +#endif + STRTAB_STE_2_S2PTW); + + ste->data[3] =3D cpu_to_le64(vttbr & STRTAB_STE_3_S2TTB_MASK); + /* Convert S1 =3D> nested and bypass =3D> S2 */ + ste->data[0] |=3D cpu_to_le64(FIELD_PREP(STRTAB_STE_0_CFG, cfg | BIT(1))); + return 0; +} + static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v3_device *smmu, u32 s= id, struct arm_smmu_ste *host_ste_out) { @@ -472,12 +526,40 @@ static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v= 3_device *smmu, u32 sid, return 0; } =20 +struct smmu_ste_writer { + struct arm_smmu_entry_writer writer; + struct hyp_arm_smmu_v3_device *smmu; + u32 sid; +}; + +static void smmu_ste_writer_sync_entry(struct arm_smmu_entry_writer *write= r) +{ + struct smmu_ste_writer *ste_writer =3D + container_of(writer, struct smmu_ste_writer, writer); + struct arm_smmu_cmd cfgi_cmd =3D arm_smmu_make_cmd_cfgi_ste(ste_writer->s= id, true); + + WARN_ON(smmu_send_cmd(ste_writer->smmu, &cfgi_cmd)); +} + +static const struct arm_smmu_entry_writer_ops smmu_ste_writer_ops =3D { + .get_used =3D arm_smmu_get_ste_used, + .get_update_safe =3D arm_smmu_get_ste_update_safe, + .sync =3D smmu_ste_writer_sync_entry, +}; + static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid,= bool leaf) { struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; - struct arm_smmu_ste *hyp_ste_ptr, *host_ste_ptr, host_ste_copy; + struct arm_smmu_ste *hyp_ste_ptr; u64 *hyp_ste_base =3D strtab_hyp_base(smmu); + __le64 used[STRTAB_STE_DWORDS] =3D {}; + struct arm_smmu_ste target =3D {}; int ret, i; + struct smmu_ste_writer ste_writer =3D { + .writer.ops =3D &smmu_ste_writer_ops, + .smmu =3D smmu, + .sid =3D sid, + }; =20 /* Leaf =3D 0 is not supported, as the kernel driver. */ if (!leaf || !is_smmu_enabled(smmu) || @@ -493,7 +575,7 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_dev= ice *smmu, u32 sid, bool return -E2BIG; =20 hyp_ste_ptr =3D &hyp_table[sid]; - host_ste_ptr =3D &host_table[sid]; + memcpy(target.data, host_table[sid].data, STRTAB_STE_DWORDS << 3); } else { struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)hyp_st= e_base; u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); @@ -502,8 +584,7 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_dev= ice *smmu, u32 sid, bool if (l1_idx >=3D cfg->l2.num_l1_ents) return -E2BIG; =20 - host_ste_ptr =3D &host_ste_copy; - ret =3D smmu_get_host_l2_ste(smmu, sid, host_ste_ptr); + ret =3D smmu_get_host_l2_ste(smmu, sid, &target); if (ret) return ret; =20 @@ -523,8 +604,27 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_de= vice *smmu, u32 sid, bool hyp_ste_ptr =3D &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; } =20 - for (i =3D 0; i < STRTAB_STE_DWORDS; ++i) - WRITE_ONCE(hyp_ste_ptr->data[i], host_ste_ptr->data[i]); + /* + * Summary of each host emulated state vs real HW. + * | Host | HW | + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D + * | V=3D0 | V=3D0 | + * | Abort | Abort | + * | Bypass | S2 | + * | S1 | S1+S2 | + */ + ret =3D smmu_attach_stage_2(smmu, &target); + if (ret) + return ret; + + /* The HW ignores unused fields, don't let the host set them. */ + arm_smmu_get_ste_used(target.data, used); + for (i =3D 0; i < STRTAB_STE_DWORDS; i++) + target.data[i] &=3D used[i]; + + hyp_spin_lock(&smmu->hw_lock); + arm_smmu_write_entry(&ste_writer.writer, hyp_ste_ptr->data, target.data); + hyp_spin_unlock(&smmu->hw_lock); return 0; } =20 --=20 2.56.0.176.g3575e65682-goog From nobody Fri Oct 23 03:28:21 2026 Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72980599A35 for ; Thu, 15 Oct 2026 20:09:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094987; cv=none; b=qJ3wdRwaerJCRmUmHOJ7bB3kLjizg8DGB3bRk2tNAWLFBvas/Irujdv03VC1EThtirx+3GG1sU4HCPK2q44Vot0dY/+7sUOuBnCcrEEKc2pXfOUGZFQ9F4ENKNLHAfMedC51dkAM/KPV8ECtQC7STzpEeu2+jOqFro8wltZIruY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1792094987; c=relaxed/simple; bh=vuQ82iuTyv6dFRtcM5SHwsBOiusydyoSCEgHa3W9AvQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YP3Rrphzy9sdh7AN7+ra2BF5Xff5H7QILuJb/8+2/qVyHfZxCq9ddykDyegO1OAUEeNDrJGvXhxYv6KvzEZ/xD/Jd2pDjfb1Q6QSIO1+li1gpWZVtsYF4pQW+XhpCgYGZfw/vbwPreUyu3x4ufU9rWEFmtcB0UfTU7Ax7sN5EME= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GSOgFTB6; arc=none smtp.client-ip=209.85.128.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GSOgFTB6" Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4a166a9ea21so47935875e9.2 for ; Thu, 15 Oct 2026 13:09:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1792094981; x=1792699781; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=LIrVQ92JIOcBfxb2Vgbbd+S0vgGCTt9XwUnEmjZAhJs=; b=GSOgFTB6y+tgRbG1gC38sDOczJu0Iv0gozuE90plo+cSR5pmxlQRkHAtzUYMs/Wl07 60Xh++SOuwQxMhix/dXVxA0VR39G6stvQLZz5m2Bh//I1CoY6y5k0x70nhTzTwySiGnf +dLnfVqOXwbN39MLLhvVNAGhWAcvlomi+qhVFMTBhK3f0bGkEDyx6iZNNxBg3NiYvahy 8Ke1Aq+H6Db86aC60sqe+bysikf83LXaZsbBh+dUtZ2vlbJJkE7H3f9Nlp55LLQReS/6 5/fpft5WAoR4yzdQOOUBY/VmOlPoR+U2/A26Jbb6EUvISN6Xr49SzRWF01GvFaDjDZe6 uPQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1792094981; x=1792699781; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LIrVQ92JIOcBfxb2Vgbbd+S0vgGCTt9XwUnEmjZAhJs=; b=vvDm9I9K9FVaJMmyOZDiiz6bEgyzDyG0YVRK2+lkXrRHloEuiYAqAvymmHUsodoYRk klK3Azqv94m8kBu/3rccv4OhuItgN6wY/Ysw7hbe3AAKC2iTdw7LDXzHtfGqSS6t8+bz ixwPIusBfSVh+SCDnhUQV40H6xV6yvADwAEGLIfQhNj1qj/WfuOE4Eu/BtlQE4Cj1so8 GzTZ0NC1KM1M0MIkTl4c6zCWkebEbbe+o8tTOE6nvBA2rcZU8OuZNztAZHFuCQfX6wAa saP6eKVAtnoaAr657IKLGPRdeZKoT7tWZuSkDbsIDQ61fN0hHphMe8U7liez1uCY7jOl ukyQ== X-Forwarded-Encrypted: i=1; AKwUvBxqWn2AUL3xIJrMawIuDury8dRzwh4y+fu3AQLAkxYEzzURFhprhAzqsGitRm/rEOAtPhO3Px23YJHpB4A=@vger.kernel.org X-Gm-Message-State: AFq9FYJvosZq/FSgOQbaoRb7uAcOGRyTednIQtZRnYRu2Kfxa3n10Fn2 +5NJOyk/pTDf/pHqIhKKZhPpI++4G8/oE9/pHwDUK+wjEKqFMUzMRx2ezv08DiiUFvm0/h5rxeS Ynm6L8RbdqjbDXg== X-Received: from wmoo13.prod.google.com ([2002:a05:600d:10d:b0:4a0:4b7:a33f]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1c07:b0:4a1:b044:9ccd with SMTP id 5b1f17b1804b1-4a1b0449d00mr214328195e9.28.1792094981436; Thu, 15 Oct 2026 13:09:41 -0700 (PDT) Date: Thu, 15 Oct 2026 20:08:56 +0000 In-Reply-To: <20261015200903.2735565-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20261015200903.2735565-1-smostafa@google.com> X-Mailer: git-send-email 2.56.0.176.g3575e65682-goog Message-ID: <20261015200903.2735565-26-smostafa@google.com> Subject: [PATCH v9 25/25] KVM: arm64: Add documentation for pKVM DMA isolation From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, nicolinc@nvidia.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Populate the section for DMA isolation in pKVM with the newly added KVM IOMMU and pKVM SMMUv3 driver details. Signed-off-by: Mostafa Saleh --- Documentation/virt/kvm/arm/pkvm.rst | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/Documentation/virt/kvm/arm/pkvm.rst b/Documentation/virt/kvm/a= rm/pkvm.rst index 514992a79a83..ce67e2ae1f9a 100644 --- a/Documentation/virt/kvm/arm/pkvm.rst +++ b/Documentation/virt/kvm/arm/pkvm.rst @@ -77,7 +77,28 @@ Status: **Unimplemented.** DMA isolation using an IOMMU ---------------------------- =20 -Status: **Unimplemented.** +Status: Supported for devices behind SMMUv3 supporting dual stages +of translation. + +With ``CONFIG_ARM_SMMU_V3_PKVM``, the hypervisor will take over the SMMUs +on the system and provide an architectural emulation to the kernel SMMUv3 +driver. + +If some devices are not behind an IOMMU or behind another IOMMU architectu= re, +DMA isolation is not supported, as a driver must be provided for that. + +DMA isolation is enforced by dual stages of translation; similar to the CPU +where a driver can register their ops through ``kvm_iommu_register_driver`` +and implement ``host_stage2_idmap`` to shadow the CPU page table. + +The page tables for the stage-2 SMMUv3 will be allocated from a pool creat= ed +at boot with size configured from the command line "kvm-arm.iommu_pgt_mem" +The size must cover the worst case scenario for leaf mappings of system me= mory. + +This implementation trusts the system firmware not to allow the untrusted +host kernel to bypass the SMMUv3. +For example, by resetting the power. In that case, it is the firmware's +responsibility to save/restore the SMMUv3 state. =20 Proxying of Trustzone services ------------------------------ --=20 2.56.0.176.g3575e65682-goog