From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EE6232C33B for ; Mon, 17 Nov 2025 18:48:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405310; cv=none; b=HOoRUBTQ1ogN246aQkwY8KZZe9p0WvIUtb7HhE7WDfL8VeShcxG5sES9S0AnP4TIPu2GOLY9RwYUBuy8Y4gXmZn5qlR95ESG6hato7VzKTvWrzxfn3Jx5lltT1nSiqEip2wBffCxAm1Vt/XQOr6sfz+XOyYAXe/6MT9GLuKglsk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405310; c=relaxed/simple; bh=4IqE8xy28CzdZK+AF2L6odntz8pjMNLJKCMqqsIJDbk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LsZuPO6STBAYO/uVi3WiQ+tw2SrolIfsWXh5LC+btwhV79iY3Fa0m31tjJfZTHi2ChtTSPKJl6sWFLbxt9Rj/RE5uApkOv6Z4qyFoPRb6yZg2+huVLV6R2whbNHC9zBlieHa2Y/Bo9j309uSLlVnhT9s5D6FRLSkL6iNh4xu5P0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aFsyeSLR; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aFsyeSLR" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-47106720618so58530765e9.1 for ; Mon, 17 Nov 2025 10:48:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405307; x=1764010107; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gey9r3BX8+q00LlBjzU+MZugMloP0I19eNAirMDKs/Y=; b=aFsyeSLRNDP7KrGEQAqdH7goCCYYR01gZMv1OsAGEwdZDCCCHFzlqDtZ8V6AuU8xXD 0gznZpc/CwznqmvoAaG29qZmgLbvcRlBh9ZzjJB+D4o1tDVZ8V+BDRsS6xRHszxVflmZ iX+fRN6zVDqYm/BZZw4UjAMv5KAG1e3DrqPKo700TKDBkBRQdgMh8WnW1rjS2aV079Zm ETa1SH5z0CRcqWyrGTG5OCCgZmDUX6/yOeQAipRj3JMDcLNYnKs/nqxs6ldRUGw2a+tg Kl65FRGn+sUFqgYuhHCq1ikDhksy+0dD0aWqg57WTcBafUh0R1zNlTwZyVNmazBbhgu2 K7PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405307; x=1764010107; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gey9r3BX8+q00LlBjzU+MZugMloP0I19eNAirMDKs/Y=; b=cOxlFEX6qqtD5LWOn1qF1Oxoi3vbjNP3K0hcsBqpFMocw5anbCMzndNjCLklHzKVoM IeZ5noSDP1TwCTxJq64gGyGvKauEy3JQ8ZMUHCVfqRVfU0TYpmzqLbb0PaMF13Wwy2ZW QDNovMKJ1Vug/VkRtLYz/lPrXUd9Bl2Tp8OcbFxXv0aKptBn/nSXxOpoXA3HrpBEoyoL lwupj5YCW9tG21Ho1nUGYS0j8fFrq222JWSJHXdI9165UzkjF4IZfXTDJFl2wM6hzrt5 /PNyd68UpPyyEe87ZKzZGP1iJx5br90npSlQ1nHLOUO7fbJqaTAC11EObnbK6VLUYGHU VUlg== X-Forwarded-Encrypted: i=1; AJvYcCV5zipybGww6efzDC1AQl7UVo5BsfGFc3qoIewbhUl5kxAMo9OtsF+k73JC2NYrByhj3XlZ+HINdifZ+W8=@vger.kernel.org X-Gm-Message-State: AOJu0YxhEM8fwVU1MGSipvC0cZkxSCeije0FC1WjBRizHNcE/tTML7yF 376I4yFS+e63r3GiIPMJUfcadBjKvpYPIrjuxvLhB06qGiuGpwMzhjrNsLZEqZdBHQAQ9t0jT7Y ZtRamu+gfwINl/Q== X-Google-Smtp-Source: AGHT+IELZ+q3N1lhluRpg0RwaEHfI9/MWh02IHmoZQ//qg9i+LZm5h9SGh+IdmhWZuUmmhd2BHNm41tOgYJ4fA== X-Received: from wmqt6.prod.google.com ([2002:a05:600c:1986:b0:46e:37d6:c1a6]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f93:b0:46e:7247:cbc0 with SMTP id 5b1f17b1804b1-4778fe6654dmr122626225e9.18.1763405307068; Mon, 17 Nov 2025 10:48:27 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:48 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-2-smostafa@google.com> Subject: [PATCH v5 01/27] KVM: arm64: Add a new function to donate memory with prot From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Soon, IOMMU drivers running in the hypervisor might interact with non-coherent devices, so it needs a mechanism to map memory as non cacheable. Add ___pkvm_host_donate_hyp() which accepts a new argument for prot, so the driver can add KVM_PGTABLE_PROT_NORMAL_NC. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index 5f9d56754e39..52d7ee91e18c 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -36,6 +36,7 @@ int __pkvm_prot_finalize(void); int __pkvm_host_share_hyp(u64 pfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); +int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index ddc8beb55eee..434b1d6aa49e 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -769,13 +769,15 @@ int __pkvm_host_unshare_hyp(u64 pfn) return ret; } =20 -int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) +int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot) { u64 phys =3D hyp_pfn_to_phys(pfn); u64 size =3D PAGE_SIZE * nr_pages; void *virt =3D __hyp_va(phys); int ret; =20 + WARN_ON((prot & KVM_PGTABLE_PROT_RWX) !=3D KVM_PGTABLE_PROT_RW); + host_lock_component(); hyp_lock_component(); =20 @@ -787,7 +789,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) goto unlock; =20 __hyp_set_page_state_range(phys, size, PKVM_PAGE_OWNED); - WARN_ON(pkvm_create_mappings_locked(virt, virt + size, PAGE_HYP)); + WARN_ON(pkvm_create_mappings_locked(virt, virt + size, prot)); WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HYP)); =20 unlock: @@ -797,6 +799,11 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) return ret; } =20 +int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) +{ + return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP); +} + int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) { u64 phys =3D hyp_pfn_to_phys(pfn); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 75EC132C33E for ; Mon, 17 Nov 2025 18:48:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405311; cv=none; b=u4/6UeXywZEO2oVMKLKFgkvxrFfYEsrEDadQ09eI5i/SkPApjRI+z52Fikqz730RxCffFPoUpvtEx5E5BRfmCWVW2Q1P8bMxIDqeRgScdmJtnTZ1ClirHSd0EaLBzfsWaj+5saFPE3q+Nz4Z1mrcJurUhjXGrlC5OPYR9SmtGq8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405311; c=relaxed/simple; bh=2Y9643ZngljQl47Ad+xEoXXZjIjmeGyFMzazpt4UT+Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aJhky5zMxPpNAEjzooiMgmYX8Ljmy7hP2yZLZ3PO1d0dfohoKArqVtHRZBxCaiAzzM5rdHbzIHrCdU519UYmgn7RWTw34CinUzAwr7H2JNhXJQiGPR9WdqwNXL2ET8cJD3987+IzIt7AtJHAz+I+SDlesqOtAU39SFsOjU2R2AY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A8SGNvXz; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A8SGNvXz" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-477563e531cso38246055e9.1 for ; Mon, 17 Nov 2025 10:48:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405308; x=1764010108; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=ud78eB0arPLfokP7b6+iOXM9DvD6Lviy96UJdE9HiHk=; b=A8SGNvXz+17Eq4Vla6FkcVJoPsBlPQFSjfcK/WDK7BgNW7Yv+HVCbVUNpWrCE8ARGw WwzCuq9FoE4uPm6Qj2zP6v8njuEMXPLESOv6ylyxkTIZxWkbQCQaffy4f2IiIkty1f4g N5lEhruidOC5Ndn0zI7v26D5xKv8H1a7w1KLBsh9i5nJ4jnqzCtopsixj2Rb5B4a++ap ZCI+7Kzm+bMlWZ1Ko4fsajZs9PNSXPNqmX+q3CcJFWyf7SDWZFRB0ttQMs2lpD44C+W9 By1DtlZfarmkL7Styl+zSS6au7q0hUhAPM9Z8N4QN9IIwbxHH81m/PL3NJnTHi513HPP qUPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405308; x=1764010108; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=ud78eB0arPLfokP7b6+iOXM9DvD6Lviy96UJdE9HiHk=; b=RVt1I0+SJZCvI24zE2wrt68ns9FYtWqLHjgSv7kntzJefZmKBtRw7j2NW/u61ptabT KzYHclppZ8NH+8R9WhCgerO7ZYey+1Zy4/7gSk8d/BrHiwTk3CL+/6PSofQ7wImbfPf2 +TLASqO1gYItEqP2CrNunFzaudRVGHVmgRPOLnGwjtaTd6JoIbBe29gjhR81XZivrzka tmBCBSx3YDh7orXX1vPuw6ekfFqIoDAjFqiMFgaymBvqghPL0N/8RmCGAFHo/NYpEiDs gjHUD+jZM43vb3EVTBzw1IxAsfXRDgvuT+4hnuWnoDrSXoCLeAvfai7JBGNET2dkj0T2 1TLg== X-Forwarded-Encrypted: i=1; AJvYcCVtOXFSMlMU+4n3HMMmxUgPpND1Hzr6C+VlXLvpnUy9LE4q0DMriSQgsuqAxlPjpxL0Q2N2ZhkHHvE8NiM=@vger.kernel.org X-Gm-Message-State: AOJu0YxBV/pyoZrQw88V5AzEGzukv50l+9tJUN4ftfBU+/By3UEZ5DnJ cI5ZPJedBhCw/tq85Ij0hueXLEafe0x+SDPtKab2sWFm7hhelI2k1tmNc/jLADE/coP8bXoZJEB FKD60Q58syNzH1w== X-Google-Smtp-Source: AGHT+IFtQM3E60fdG09G4db3NhGt1PCpbNSce99vQEOknynP4HtzybGxEOyhlaEXa2/I2J7fKVIRDAwQImwldQ== X-Received: from wmbgx25.prod.google.com ([2002:a05:600c:8599:b0:477:a338:214b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4513:b0:477:2f7c:314f with SMTP id 5b1f17b1804b1-4778fe5c820mr145661085e9.10.1763405307962; Mon, 17 Nov 2025 10:48:27 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:49 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-3-smostafa@google.com> Subject: [PATCH v5 02/27] KVM: arm64: Donate MMIO to the hypervisor From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a function to donate MMIO to the hypervisor so IOMMU hypervisor drivers can use that to protect the MMIO of IOMMU. The initial attempt to implement this was to have a new flag to "___pkvm_host_donate_hyp" to accept MMIO. However that had many problems, it was quite intrusive for host/hyp to check/set page state to make it aware of MMIO and to encode the state in the page table in that case. Which is called in paths that can be sensitive to performance (FFA, VMs..) As donating MMIO is very rare, and we don=E2=80=99t need to encode the full= state, it=E2=80=99s reasonable to have a separate function to do this. It will init the host s2 page table with an invalid leaf with the owner ID to prevent the host from mapping the page on faults. Also, prevent kvm_pgtable_stage2_unmap() from removing owner ID from stage-2 PTEs, as this can be triggered from recycle logic under memory pressure. There is no code relying on this, as all ownership changes is done via kvm_pgtable_stage2_set_owner() For error path in IOMMU drivers, add a function to donate MMIO back from hyp to host. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 90 +++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 9 +- 3 files changed, 94 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index 52d7ee91e18c..98e173da0f9b 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -37,6 +37,8 @@ int __pkvm_host_share_hyp(u64 pfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot); +int __pkvm_host_donate_hyp_mmio(u64 pfn); +int __pkvm_hyp_donate_host_mmio(u64 pfn); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 434b1d6aa49e..c3eac0da7cbe 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -799,6 +799,96 @@ int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enu= m kvm_pgtable_prot prot) return ret; } =20 +int __pkvm_host_donate_hyp_mmio(u64 pfn) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + void *virt =3D __hyp_va(phys); + int ret; + kvm_pte_t pte; + + if (addr_is_memory(phys)) + return -EINVAL; + + host_lock_component(); + hyp_lock_component(); + + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL); + if (ret) + goto unlock; + + if (pte && !kvm_pte_valid(pte)) { + ret =3D -EPERM; + goto unlock; + } + + ret =3D kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)virt, &pte, NULL); + if (ret) + goto unlock; + if (pte) { + ret =3D -EBUSY; + goto unlock; + } + + ret =3D pkvm_create_mappings_locked(virt, virt + PAGE_SIZE, PAGE_HYP_DEVI= CE); + if (ret) + goto unlock; + /* + * We set HYP as the owner of the MMIO pages in the host stage-2, for: + * - host aborts: host_stage2_adjust_range() would fail for invalid non z= ero PTEs. + * - recycle under memory pressure: host_stage2_unmap_dev_all() would call + * kvm_pgtable_stage2_unmap() which will not clear non zero invalid pte= s (counted). + * - other MMIO donation: Would fail as we check that the PTE is valid or= empty. + */ + WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys, + PAGE_SIZE, &host_s2_pool, PKVM_ID_HYP)); +unlock: + hyp_unlock_component(); + host_unlock_component(); + + return ret; +} + +int __pkvm_hyp_donate_host_mmio(u64 pfn) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + u64 virt =3D (u64)__hyp_va(phys); + size_t size =3D PAGE_SIZE; + int ret; + kvm_pte_t pte; + + if (addr_is_memory(phys)) + return -EINVAL; + + host_lock_component(); + hyp_lock_component(); + + ret =3D kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)virt, &pte, NULL); + if (ret) + goto unlock; + if (!kvm_pte_valid(pte)) { + ret =3D -ENOENT; + goto unlock; + } + + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL); + if (ret) + goto unlock; + + if (FIELD_GET(KVM_INVALID_PTE_OWNER_MASK, pte) !=3D PKVM_ID_HYP) { + ret =3D -EPERM; + goto unlock; + } + + WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, virt, size) !=3D size); + WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys, + PAGE_SIZE, &host_s2_pool, PKVM_ID_HOST)); +unlock: + hyp_unlock_component(); + host_unlock_component(); + + return ret; +} + int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) { return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c351b4abd5db..ba06b0c21d5a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1095,13 +1095,8 @@ static int stage2_unmap_walker(const struct kvm_pgta= ble_visit_ctx *ctx, kvm_pte_t *childp =3D NULL; bool need_flush =3D false; =20 - if (!kvm_pte_valid(ctx->old)) { - if (stage2_pte_is_counted(ctx->old)) { - kvm_clear_pte(ctx->ptep); - mm_ops->put_page(ctx->ptep); - } - return 0; - } + if (!kvm_pte_valid(ctx->old)) + return stage2_pte_is_counted(ctx->old) ? -EPERM : 0; =20 if (kvm_pte_table(ctx->old, ctx->level)) { childp =3D kvm_pte_follow(ctx->old, mm_ops); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E21832C943 for ; Mon, 17 Nov 2025 18:48:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405312; cv=none; b=UJWWOwDhIFEBDkNOnqRqzl0Ve/U+GbSwuLnRJ4roKr39Ej/bVFcbD2mRLSJQC1vUE1ieU+8IphXJTzrwvcelFj0E45PBcf6TTIt1iekzYFsyNLcXXJKaNjOGR36yiXbIZFV7H1VWWh25b8/goCUupiuYD27QWvw3QahMTPeJEbA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405312; c=relaxed/simple; bh=S/1TCoG4iWrrYeaePo4/lgi9pJbViEDsshc6JJsQ2aA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZdcRn0colQ7JbCwNO74Et1wmJlUo7m+LfDehxL7seL1xnO6GMc1BKqi5s9ztoLE9GXUUyXJnG9VJhk5kHMhFyVLRrGYvziQiKy82bM1zZLVlj9WBQp4jYpZx3Ad1zAB4d4ys0n5RlvGR+NaBO4L8RpbyUl0mM561rrTCnn5xqSY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lsLg3Zth; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lsLg3Zth" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-47777158a85so41835515e9.3 for ; Mon, 17 Nov 2025 10:48:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405308; x=1764010108; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mdiIv9AzHH9LJUmk1x5XNuh2D17ItySFVQ+lpR2xjRw=; b=lsLg3Zth84ipCefndg5yrTft/9rK1cojfyimWfVYcyCjWqSL9S11iw/2jXCfqln84y Xawd8iEcP4Y6gL0uJTZ0JZfZd6c39HIIx3zGnMB7NL5Ah9eXwz3UD4RqYxDcd5+nuCzR fCumRYgS0+pmIG/utmCMvBUq3ZayhlMGwdC6S9Ppq7oIOlmJPjEImG8rV68DP8jJ+JwZ zWRk0DD3Ds16fbb08I4DylIjbh8NXyIXegtE0wZqW0FSRm8Rf0b4z6Px3SgEQ9fG1ILs vv8TEnTiYH6DbCKkjWFgVDiLanfxkmINs4H9HMDCF3ei2EYDrdWO1+t2uFfh0I57sGu7 Wjmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405308; x=1764010108; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mdiIv9AzHH9LJUmk1x5XNuh2D17ItySFVQ+lpR2xjRw=; b=mhjC3YwEjXUdGqmXyGbkfq1Eeq4cqEGzD/PykaxXl6ySTXPF9gQGpGHOOoWphurKGb pe7WzFXJrh2BQr2frh6a8rOZocbYq+8VZYwQ9U2v3EFiMk3Mb5pupb0yBcc2rzTnTKRk Tt9KH6gvJ0m1QiP2LsxvorZlxtwB6m8yR/xL3zdcbORKw3tE4aajIxyOARY91iw/X4nM CxK8S8As1MoYKzsiqSArk+RQNQzGTRXzI3D9jddFtg2x5ah3qpeQUw3ly8YmjiN21cxh AeBsL4gVOz2Me0M2GLDHBqy+DCMP+JO3NIp6gDkKOtdN4UkdInrOvRH3p7/jjL52iGsx PAKw== X-Forwarded-Encrypted: i=1; AJvYcCXI9JB4aF/J05KYrSxSGHHXyt4wXSy1LkEjP2GGemxy50vY6OXYgUlpm1pbbdEFwS86bEfFfvDUJ23KoWA=@vger.kernel.org X-Gm-Message-State: AOJu0YzTj4kWGdH0/m3pGMP4XHuSmzlvUrjLp7PtmtfvC3sS/9M4VM1U aoK8n3s/2ztAndT3i9cD8Kqhznm2MFWeArysXp3G1IkKMuvrU0cRnCAP8gOzMZ4SAOxaNBftzRV OP0UG0E2Qnmzxmw== X-Google-Smtp-Source: AGHT+IEIO6NZXXPWg9+hR4+SPtTpCKpBv3Ky3OdW45sXPemfU0CwllktSYR7niEb66Ks+RCqsCJfiR5agNLU5w== X-Received: from wmbgx23.prod.google.com ([2002:a05:600c:8597:b0:477:a0a1:eae8]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8b4b:b0:477:639d:bca2 with SMTP id 5b1f17b1804b1-4778fe51eb3mr117363365e9.4.1763405308662; Mon, 17 Nov 2025 10:48:28 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:50 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-4-smostafa@google.com> Subject: [PATCH v5 03/27] KVM: arm64: pkvm: Add pkvm_time_get() From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a function to return time in us. This can be used from IOMMU drivers while waiting for conditions as for SMMUv3 TLB invalidation waiting for sync. Signed-off-by: Mostafa Saleh Signed-off-by: Jean-Philippe Brucker --- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 2 ++ arch/arm64/kvm/hyp/nvhe/setup.c | 4 ++++ arch/arm64/kvm/hyp/nvhe/timer-sr.c | 32 ++++++++++++++++++++++++++ 3 files changed, 38 insertions(+) diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/in= clude/nvhe/pkvm.h index 184ad7a39950..2b065e048a35 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h @@ -89,4 +89,6 @@ bool kvm_handle_pvm_restricted(struct kvm_vcpu *vcpu, u64= *exit_code); void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu); int kvm_check_pvm_sysreg_table(void); =20 +int pkvm_timer_init(void); +u64 pkvm_time_get(void); #endif /* __ARM64_KVM_NVHE_PKVM_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index 90bd014e952f..eff76be89329 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -312,6 +312,10 @@ void __noreturn __pkvm_init_finalise(void) }; pkvm_pgtable.mm_ops =3D &pkvm_pgtable_mm_ops; =20 + ret =3D pkvm_timer_init(); + if (ret) + goto out; + ret =3D fix_host_ownership(); if (ret) goto out; diff --git a/arch/arm64/kvm/hyp/nvhe/timer-sr.c b/arch/arm64/kvm/hyp/nvhe/t= imer-sr.c index ff176f4ce7de..ce91719c876d 100644 --- a/arch/arm64/kvm/hyp/nvhe/timer-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/timer-sr.c @@ -11,6 +11,10 @@ #include #include =20 +#include + +static u32 timer_freq; + void __kvm_timer_set_cntvoff(u64 cntvoff) { write_sysreg(cntvoff, cntvoff_el2); @@ -68,3 +72,31 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu) =20 sysreg_clear_set(cnthctl_el2, clr, set); } + +static u64 pkvm_ticks_get(void) +{ + return __arch_counter_get_cntvct(); +} + +#define SEC_TO_US 1000000 + +int pkvm_timer_init(void) +{ + timer_freq =3D read_sysreg(cntfrq_el0); + + /* + * KVM will not initialize if FW didn't set cntfrq_el0, that is already + * part of the boot protocol. + */ + if (!timer_freq || timer_freq < SEC_TO_US) + return -ENODEV; + return 0; +} + +#define pkvm_time_ticks_to_us(ticks) ((u64)(ticks) * SEC_TO_US / timer_fre= q) + +/* Return time in us. */ +u64 pkvm_time_get(void) +{ + return pkvm_time_ticks_to_us(pkvm_ticks_get()); +} --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6DB132E6AD for ; Mon, 17 Nov 2025 18:48:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405314; cv=none; b=NaD48T9gwAJm5RuTfJ3/DWFhbB6xpa45hG/FT8eIHEha6jjlTGm3yPOL3ZBeaUJr3vwenjws+GuFuUJNegz8Z688W1inqerK4DwVlowhntlD+ERRv3Xa9T2SKy3K2m72b14WZqK24AyM25LVU3F3g7DCO0XvA9w5qzNk6OIYL1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405314; c=relaxed/simple; bh=vQaTJ6UbtFrpH+abzCIlTBUfgMGn+5ViTneidf9fDrA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dCI4zPNwmmSH5sS/F2PRK48Swq3e/cU8TrQs84Uoh+i+1BPkWVA01ANO+lP6IZ0CZn7EgBoct51ZvI+4Y+jNgmEdpbq5SBe53CMcQqSu+eAM791gQC+8qOw9zWr9TQctTDlDGkaIAgNvvMGdgfpFAiMK/ZRP1MHCPbG5ImPApo8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=CdIHFon9; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="CdIHFon9" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-429c95fdba8so237248f8f.0 for ; Mon, 17 Nov 2025 10:48:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405310; x=1764010110; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=pa06yX889DYuT+oZK4r6IpdtcQ56yZdpVOQD3ivoTx4=; b=CdIHFon9ACNfknbH8swjqSionJuYdPU6ECiDZybO8M1NB5c0aAK+iCWXick4nTa9Mh puvZfksHUGyDTgiY/9quu82bOWXkbOlRXT86Ce/MEDbpGJXfc2gLdh1R4A6VzSg73L9a MqhvLMPi/h2M7J/EaOJcgDMiBJtP12NX7hJp31YbA6urD6orU2wtAropoOqVwt6r6+Yl bGAn84Kn9yAsih8S10sNkQYS0iVuB8+vjo5CXIS2OrdZ4se1g4O965n74uBLi3PmFIny EskXf0LzIVg11r6NSAd+Xc5t1vQIqGUa97JRKBlBoGfAR4hiF5RAMmq4LALUgMTO2bIH sn6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405310; x=1764010110; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=pa06yX889DYuT+oZK4r6IpdtcQ56yZdpVOQD3ivoTx4=; b=Vk+t92h+Rz5JHoYOVKgW1pkHswY5Bvxucs+2QMsEc0yxDmBnghanf8q/a3yGQZGF0E 3GkhQk8JH1DkcZRMyJAsNoNj0sfwxlnzHNQbZ5wfZ+C3guqhmxqrmKc151xaEzXj8Djr SNNCQAqWbTcdN3Dds08/VtAwJXMtKiPUq5iRXmR9mezkMbdP92LNi63lS6D+IlR3uvWQ DBf5jfqzswncMtzA9oXnCE9qGMgX+kSjO0lcXuJe+VM+meE+zTZULpb9CEtPUi8mxIdB A5qyi81/64O9v8qwR00xVyTB+Dxu/YIXbKmvyT2MDBEzimqmYW/k0NTXyksfRiCH2i6u rZvg== X-Forwarded-Encrypted: i=1; AJvYcCXkGy6MiSV4crP/M18Z1ZE2vo3yBXveefzXc7f4OQqSwCMIu/dGeWVqKx4ZrZmu7jKCK/uTxXJvTwCCxQg=@vger.kernel.org X-Gm-Message-State: AOJu0Ywg88iqcVebXstH9DbOtwMrhwTIkHER7E7s9jCf2mKq20YQH3Sr 9em177AHBRTn7pcNbbw9Wbwxc+tjRxzED491iMVVENHHwOivbJwEPi7C1rwYuEAK3i8dk9dY3Sl 8O1L4iqRT5owpNA== X-Google-Smtp-Source: AGHT+IEpCDIon+7uG37hSzh8xEnVSPrSqapIC6AAiKmu0f1Bo4efhhsle4hTqBMXTR+Cl+gfspw5gNrrIxHwpg== X-Received: from wrao26.prod.google.com ([2002:adf:8b9a:0:b0:42b:4c63:868a]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:3108:b0:42b:30d4:e3f0 with SMTP id ffacd0b85a97d-42b5933b5c7mr14452962f8f.22.1763405309761; Mon, 17 Nov 2025 10:48:29 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:51 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-5-smostafa@google.com> Subject: [PATCH v5 04/27] iommu/io-pgtable-arm: Factor kernel specific code out From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some of the used APIs are only part of the kernel and are not available in the hypervisor, factor those out: - alloc/free memory - CMOs - virt/phys conversions Which are implemented by the kernel in io-pgtable-arm-kernel.c and similarly for the hypervisor later in this series. va/pa conversion kept as macros. Signed-off-by: Mostafa Saleh --- drivers/iommu/Makefile | 2 +- drivers/iommu/io-pgtable-arm-kernel.c | 103 ++++++++++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 101 +++---------------------- drivers/iommu/io-pgtable-arm.h | 19 +++++ 4 files changed, 133 insertions(+), 92 deletions(-) create mode 100644 drivers/iommu/io-pgtable-arm-kernel.c diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 8e8843316c4b..439431fd4bc5 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -12,7 +12,7 @@ obj-$(CONFIG_IOMMU_DEBUGFS) +=3D iommu-debugfs.o obj-$(CONFIG_IOMMU_DMA) +=3D dma-iommu.o obj-$(CONFIG_IOMMU_IO_PGTABLE) +=3D io-pgtable.o obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) +=3D io-pgtable-arm-v7s.o -obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) +=3D io-pgtable-arm.o +obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) +=3D io-pgtable-arm.o io-pgtable-arm-k= ernel.o obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE_KUNIT_TEST) +=3D io-pgtable-arm-selftes= ts.o obj-$(CONFIG_IOMMU_IO_PGTABLE_DART) +=3D io-pgtable-dart.o obj-$(CONFIG_IOMMU_IOVA) +=3D iova.o diff --git a/drivers/iommu/io-pgtable-arm-kernel.c b/drivers/iommu/io-pgtab= le-arm-kernel.c new file mode 100644 index 000000000000..d025f7c180f9 --- /dev/null +++ b/drivers/iommu/io-pgtable-arm-kernel.c @@ -0,0 +1,103 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * CPU-agnostic ARM page table allocator. + * + * Copyright (C) 2014 ARM Limited + * + * Author: Will Deacon + */ +#include + +#include +#include +#include + +#include "io-pgtable-arm.h" +#include "iommu-pages.h" + +static dma_addr_t __arm_lpae_dma_addr(void *pages) +{ + return (dma_addr_t)virt_to_phys(pages); +} + +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct device *dev =3D cfg->iommu_dev; + size_t alloc_size; + dma_addr_t dma; + void *pages; + + /* + * For very small starting-level translation tables the HW requires a + * minimum alignment of at least 64 to cover all cases. + */ + alloc_size =3D max(size, 64); + if (cfg->alloc) + pages =3D cfg->alloc(cookie, alloc_size, gfp); + else + pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, + alloc_size); + + if (!pages) + return NULL; + + if (!cfg->coherent_walk) { + dma =3D dma_map_single(dev, pages, size, DMA_TO_DEVICE); + if (dma_mapping_error(dev, dma)) + goto out_free; + /* + * We depend on the IOMMU being able to work with any physical + * address directly, so if the DMA layer suggests otherwise by + * translating or truncating them, that bodes very badly... + */ + if (dma !=3D virt_to_phys(pages)) + goto out_unmap; + } + + return pages; + +out_unmap: + dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n"= ); + dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); + +out_free: + if (cfg->free) + cfg->free(cookie, pages, size); + else + iommu_free_pages(pages); + + return NULL; +} + +void __arm_lpae_free_pages(void *pages, size_t size, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + if (!cfg->coherent_walk) + dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages), + size, DMA_TO_DEVICE); + + if (cfg->free) + cfg->free(cookie, pages, size); + else + iommu_free_pages(pages); +} + +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg) +{ + dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep), + sizeof(*ptep) * num_entries, DMA_TO_DEVICE); +} + +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp) +{ + return kmalloc(size, gfp); +} + +void __arm_lpae_free_data(void *p) +{ + return kfree(p); +} diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index e6626004b323..377c15bc8350 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -15,12 +15,10 @@ #include #include #include -#include =20 #include =20 #include "io-pgtable-arm.h" -#include "iommu-pages.h" =20 #define ARM_LPAE_MAX_ADDR_BITS 52 #define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 @@ -143,7 +141,7 @@ #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL =20 /* IOPTE accessors */ -#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d)) +#define iopte_deref(pte,d) __arm_lpae_phys_to_virt(iopte_to_paddr(pte, d)) =20 #define iopte_type(pte) \ (((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK) @@ -164,8 +162,6 @@ struct arm_lpae_io_pgtable { void *pgd; }; =20 -typedef u64 arm_lpae_iopte; - static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl, enum io_pgtable_fmt fmt) { @@ -243,83 +239,6 @@ static inline bool arm_lpae_concat_mandatory(struct io= _pgtable_cfg *cfg, (data->start_level =3D=3D 1) && (oas =3D=3D 40); } =20 -static dma_addr_t __arm_lpae_dma_addr(void *pages) -{ - return (dma_addr_t)virt_to_phys(pages); -} - -static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, - struct io_pgtable_cfg *cfg, - void *cookie) -{ - struct device *dev =3D cfg->iommu_dev; - size_t alloc_size; - dma_addr_t dma; - void *pages; - - /* - * For very small starting-level translation tables the HW requires a - * minimum alignment of at least 64 to cover all cases. - */ - alloc_size =3D max(size, 64); - if (cfg->alloc) - pages =3D cfg->alloc(cookie, alloc_size, gfp); - else - pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, - alloc_size); - - if (!pages) - return NULL; - - if (!cfg->coherent_walk) { - dma =3D dma_map_single(dev, pages, size, DMA_TO_DEVICE); - if (dma_mapping_error(dev, dma)) - goto out_free; - /* - * We depend on the IOMMU being able to work with any physical - * address directly, so if the DMA layer suggests otherwise by - * translating or truncating them, that bodes very badly... - */ - if (dma !=3D virt_to_phys(pages)) - goto out_unmap; - } - - return pages; - -out_unmap: - dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n"= ); - dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); - -out_free: - if (cfg->free) - cfg->free(cookie, pages, size); - else - iommu_free_pages(pages); - - return NULL; -} - -static void __arm_lpae_free_pages(void *pages, size_t size, - struct io_pgtable_cfg *cfg, - void *cookie) -{ - if (!cfg->coherent_walk) - dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages), - size, DMA_TO_DEVICE); - - if (cfg->free) - cfg->free(cookie, pages, size); - else - iommu_free_pages(pages); -} - -static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, - struct io_pgtable_cfg *cfg) -{ - dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep), - sizeof(*ptep) * num_entries, DMA_TO_DEVICE); -} - static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_c= fg *cfg, int num_entries) { for (int i =3D 0; i < num_entries; i++) @@ -395,7 +314,7 @@ static arm_lpae_iopte arm_lpae_install_table(arm_lpae_i= opte *table, arm_lpae_iopte old, new; struct io_pgtable_cfg *cfg =3D &data->iop.cfg; =20 - new =3D paddr_to_iopte(__pa(table), data) | ARM_LPAE_PTE_TYPE_TABLE; + new =3D paddr_to_iopte(__arm_lpae_virt_to_phys(table), data) | ARM_LPAE_P= TE_TYPE_TABLE; if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS) new |=3D ARM_LPAE_PTE_NSTABLE; =20 @@ -616,7 +535,7 @@ static void arm_lpae_free_pgtable(struct io_pgtable *io= p) struct arm_lpae_io_pgtable *data =3D io_pgtable_to_data(iop); =20 __arm_lpae_free_pgtable(data, data->start_level, data->pgd); - kfree(data); + __arm_lpae_free_data(data); } =20 static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, @@ -930,7 +849,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg) if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS) return NULL; =20 - data =3D kmalloc(sizeof(*data), GFP_KERNEL); + data =3D __arm_lpae_alloc_data(sizeof(*data), GFP_KERNEL); if (!data) return NULL; =20 @@ -1053,11 +972,11 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *= cfg, void *cookie) wmb(); =20 /* TTBR */ - cfg->arm_lpae_s1_cfg.ttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s1_cfg.ttbr =3D __arm_lpae_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 @@ -1149,11 +1068,11 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg = *cfg, void *cookie) wmb(); =20 /* VTTBR */ - cfg->arm_lpae_s2_cfg.vttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s2_cfg.vttbr =3D __arm_lpae_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 @@ -1223,7 +1142,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) /* Ensure the empty pgd is visible before TRANSTAB can be written */ wmb(); =20 - cfg->arm_mali_lpae_cfg.transtab =3D virt_to_phys(data->pgd) | + cfg->arm_mali_lpae_cfg.transtab =3D __arm_lpae_virt_to_phys(data->pgd) | ARM_MALI_LPAE_TTBR_READ_INNER | ARM_MALI_LPAE_TTBR_ADRMODE_TABLE; if (cfg->coherent_walk) @@ -1232,7 +1151,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index ba7cfdf7afa0..62d127dae1c2 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -2,6 +2,8 @@ #ifndef IO_PGTABLE_ARM_H_ #define IO_PGTABLE_ARM_H_ =20 +#include + #define ARM_LPAE_TCR_TG0_4K 0 #define ARM_LPAE_TCR_TG0_64K 1 #define ARM_LPAE_TCR_TG0_16K 2 @@ -27,4 +29,21 @@ #define ARM_LPAE_TCR_PS_48_BIT 0x5ULL #define ARM_LPAE_TCR_PS_52_BIT 0x6ULL =20 +typedef u64 arm_lpae_iopte; + +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg); +void __arm_lpae_free_pages(void *pages, size_t size, + struct io_pgtable_cfg *cfg, + void *cookie); +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, + void *cookie); +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp); +void __arm_lpae_free_data(void *p); +#ifndef __KVM_NVHE_HYPERVISOR__ +#define __arm_lpae_virt_to_phys __pa +#define __arm_lpae_phys_to_virt __va +#endif /* !__KVM_NVHE_HYPERVISOR__ */ + #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE1AB32F74D for ; Mon, 17 Nov 2025 18:48:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405315; cv=none; b=UTfGh5aM4WfFIDIe5zpmmBB9uNlfzBhYCDSNh5UViUXMkRCdOJwaampqaibIosAMhL85S/wCeADlgwo6f3RWab9qxMd6tZ+rBElH+cyCueHctKqZ8bImteWI6VBRrOWzeIqU/W5DM3NiIyES6AxIwVeD9jWE8slMEVGnVoaADso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405315; c=relaxed/simple; bh=m0eyTKHGEnJ+4t3vyjA7YY2Y9ideYxvZxlB9Ql41BYc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=n6ZjScEpGEfYjlMzR1zCBPFbzITEOXeMmcazTGi/ufYiiPdOFSWk7l5oV+BtbGw/QJnkrALBK7+0W+0l6V2CFUDayuwOFp9eCNYYUf6H6Gf2TyPGQ25P7tuSzvFwLiSu4r1/LHXpwEvUSZpkiYDZ2NOpt9rhzmb3ifVhxMvpTic= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=17TVSGGn; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="17TVSGGn" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b739b3d8997so134080566b.2 for ; Mon, 17 Nov 2025 10:48:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405311; x=1764010111; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3riz6/OzFPZPFTsti7c+9X4sXSVC+UYKQKi4yNQptVI=; b=17TVSGGnylLiSJ8ifbQNa3G5/xlo2r96fhKXc5FboC6dwsGJFicjyCB3GR+7I8vYUY ndwSdIfGTf9OzmwgEEphA3tnIIqtYKAOSEHvO9/LAY4ESOHCgLlKbJf+ycXc0jatCSgz E1X19DRIjvtWggC2SRrc35k+QBXskJgX1gAxrmrB2TlCnZij9fJabdXDqCU9tfHJ7Req Al2svhlZvBVT1s9pUKkE1ObCTLlqDJ3NmupYoKblGRUAHlKBQya7qRCZyZ8gMQinVSsC gx/EoLyf6b1NBPXW9i7ilX9hJCFadauN5Mk0kjxfcP897gtsn6FPYV15MWQTn1jSjdBA AvIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405311; x=1764010111; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3riz6/OzFPZPFTsti7c+9X4sXSVC+UYKQKi4yNQptVI=; b=GajFgRw107LpKozU6whYMEEf1p6V1vaOBlAAEj81eS/HwB865Oz5Brwd4634LYYv/Z bAVQiJYVA8EpqE3weeU21MPKErQOpx2C9weQk2ZIiXsHI6x47akG7EYEZo4ceJMgFls7 PhfoubBjoEJkYg1YFONt8v6lAbcHtXY4VujLGMFH+y4TTMJcQlYZZ9G20U2MLzNE3J0Q gbUO5BCY5avsDMUeO1AtFR8OjPKLydtoX9ELV0kmkz0iLnezBfB3mEgkEnLImmm5zOKU P4N6HMp+wSwICuGIPh+PqUcde3rPIlN2YwYTptcgqcIL2D3YXbwOY3N8ScM44V5KVftz jOrA== X-Forwarded-Encrypted: i=1; AJvYcCXTPY+HD/WiyvwJ4dsSch4keSBU+Ku75xEkd6Jg3oYua5nHxtqv3pNar4ZsWjgom/RvCXe7YoqQexZp5uA=@vger.kernel.org X-Gm-Message-State: AOJu0YyVvNwgEFzEtibldeFJ5LAwi1mXusEpZW2D7wJbMakaYa/XoMDz ZcydY68E0oRxjVFsGWN+sBSoupeACavOpDddAa9BBox/Az/VqVRCpziIgoNGO39h0XHGP6BB8uN 2xmEHjEQB6cx3IA== X-Google-Smtp-Source: AGHT+IHMHkpsTJge4qLXOrIez+lKfkrvSIzQtWv7dToHUpC0p8gFukjOOv6/08P6LsO19cn+GeMm7M+u5A4siw== X-Received: from ejchw17.prod.google.com ([2002:a17:907:a0d1:b0:b73:7374:2f96]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:787:b0:b73:5f48:6159 with SMTP id a640c23a62f3a-b73678088e2mr1486327366b.5.1763405311192; Mon, 17 Nov 2025 10:48:31 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:52 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-6-smostafa@google.com> Subject: [PATCH v5 05/27] iommu/arm-smmu-v3: Split code with hyp From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The KVM SMMUv3 driver would re-use some of the cmdq code inside the hypervisor, move these functions to a new common c file that is shared between the host kernel and the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 114 +++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 161 ------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 61 +++++++ 4 files changed, 176 insertions(+), 162 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index 493a659cc66b..c9ce392e6d31 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ARM_SMMU_V3) +=3D arm_smmu_v3.o -arm_smmu_v3-y :=3D arm-smmu-v3.o +arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c new file mode 100644 index 000000000000..62744c8548a8 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2015 ARM Limited + * + * Author: Will Deacon + * Arm SMMUv3 driver functions shared with hypervisor. + */ + +#include "arm-smmu-v3.h" +#include + +#include + +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) +{ + memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); + cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); + + switch (ent->opcode) { + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_NSNH_ALL: + break; + case CMDQ_OP_PREFETCH_CFG: + cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); + break; + case CMDQ_OP_CFGI_CD: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); + fallthrough; + case CMDQ_OP_CFGI_STE: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); + break; + case CMDQ_OP_CFGI_CD_ALL: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + break; + case CMDQ_OP_CFGI_ALL: + /* Cover the entire SID range */ + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); + break; + case CMDQ_OP_TLBI_NH_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + fallthrough; + case CMDQ_OP_TLBI_EL2_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; + break; + case CMDQ_OP_TLBI_S2_IPA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; + break; + case CMDQ_OP_TLBI_NH_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + fallthrough; + case CMDQ_OP_TLBI_NH_ALL: + case CMDQ_OP_TLBI_S12_VMALL: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + break; + case CMDQ_OP_TLBI_EL2_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + break; + case CMDQ_OP_ATC_INV: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); + cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; + break; + case CMDQ_OP_PRI_RESP: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); + switch (ent->pri.resp) { + case PRI_RESP_DENY: + case PRI_RESP_FAIL: + case PRI_RESP_SUCC: + break; + default: + return -EINVAL; + } + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); + break; + case CMDQ_OP_RESUME: + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); + cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); + break; + case CMDQ_OP_CMD_SYNC: + if (ent->sync.msiaddr) { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); + cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; + } else { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + } + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); + break; + default: + return -ENOENT; + } + + return 0; +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index a33fbd12a0dd..1497ffcd4555 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -123,33 +123,6 @@ static void parse_driver_options(struct arm_smmu_devic= e *smmu) } =20 /* Low-level queue manipulation functions */ -static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) -{ - u32 space, prod, cons; - - prod =3D Q_IDX(q, q->prod); - cons =3D Q_IDX(q, q->cons); - - if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; - - return space >=3D n; -} - -static bool queue_full(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); -} - -static bool queue_empty(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); -} - static bool queue_consumed(struct arm_smmu_ll_queue *q, u32 prod) { return ((Q_WRP(q, q->cons) =3D=3D Q_WRP(q, prod)) && @@ -168,12 +141,6 @@ static void queue_sync_cons_out(struct arm_smmu_queue = *q) writel_relaxed(q->llq.cons, q->cons_reg); } =20 -static void queue_inc_cons(struct arm_smmu_ll_queue *q) -{ - u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; - q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); -} - static void queue_sync_cons_ovf(struct arm_smmu_queue *q) { struct arm_smmu_ll_queue *llq =3D &q->llq; @@ -205,12 +172,6 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q) return ret; } =20 -static u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) -{ - u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; - return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); -} - static void queue_poll_init(struct arm_smmu_device *smmu, struct arm_smmu_queue_poll *qp) { @@ -238,14 +199,6 @@ static int queue_poll(struct arm_smmu_queue_poll *qp) return 0; } =20 -static void queue_write(__le64 *dst, u64 *src, size_t n_dwords) -{ - int i; - - for (i =3D 0; i < n_dwords; ++i) - *dst++ =3D cpu_to_le64(*src++); -} - static void queue_read(u64 *dst, __le64 *src, size_t n_dwords) { int i; @@ -266,108 +219,6 @@ static int queue_remove_raw(struct arm_smmu_queue *q,= u64 *ent) } =20 /* High-level queue accessors */ -static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) -{ - memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); - cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); - - switch (ent->opcode) { - case CMDQ_OP_TLBI_EL2_ALL: - case CMDQ_OP_TLBI_NSNH_ALL: - break; - case CMDQ_OP_PREFETCH_CFG: - cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); - break; - case CMDQ_OP_CFGI_CD: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); - fallthrough; - case CMDQ_OP_CFGI_STE: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); - break; - case CMDQ_OP_CFGI_CD_ALL: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - break; - case CMDQ_OP_CFGI_ALL: - /* Cover the entire SID range */ - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); - break; - case CMDQ_OP_TLBI_NH_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - fallthrough; - case CMDQ_OP_TLBI_EL2_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; - break; - case CMDQ_OP_TLBI_S2_IPA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; - break; - case CMDQ_OP_TLBI_NH_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - fallthrough; - case CMDQ_OP_TLBI_NH_ALL: - case CMDQ_OP_TLBI_S12_VMALL: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - break; - case CMDQ_OP_TLBI_EL2_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - break; - case CMDQ_OP_ATC_INV: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); - cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; - break; - case CMDQ_OP_PRI_RESP: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); - switch (ent->pri.resp) { - case PRI_RESP_DENY: - case PRI_RESP_FAIL: - case PRI_RESP_SUCC: - break; - default: - return -EINVAL; - } - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); - break; - case CMDQ_OP_RESUME: - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); - cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); - break; - case CMDQ_OP_CMD_SYNC: - if (ent->sync.msiaddr) { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); - cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; - } else { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); - } - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); - break; - default: - return -ENOENT; - } - - return 0; -} - static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smm= u, struct arm_smmu_cmdq_ent *ent) { @@ -1508,18 +1359,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_= master *master) } =20 /* Stream table manipulation functions */ -static void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst, - dma_addr_t l2ptr_dma) -{ - u64 val =3D 0; - - val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); - val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - - /* The HW has 64 bit atomicity with stores to the L2 STE table */ - WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); -} - struct arm_smmu_ste_writer { struct arm_smmu_entry_writer writer; u32 sid; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ae23aacc3840..4aaf93945ee3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1010,6 +1010,67 @@ void arm_smmu_install_ste_for_dev(struct arm_smmu_ma= ster *master, int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq, u64 *cmds, int n, bool sync); +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent); + +/* Queue functions shared between kernel and hyp. */ +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + u32 space, prod, cons; + + prod =3D Q_IDX(q, q->prod); + cons =3D Q_IDX(q, q->cons); + + if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) + space =3D (1 << q->max_n_shift) - (prod - cons); + else + space =3D cons - prod; + + return space >=3D n; +} + +static inline bool queue_full(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); +} + +static inline bool queue_empty(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); +} + +static inline u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) +{ + u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; + return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); +} + +static inline void queue_inc_cons(struct arm_smmu_ll_queue *q) +{ + u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; + q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); +} + +static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords) +{ + int i; + + for (i =3D 0; i < n_dwords; ++i) + *dst++ =3D cpu_to_le64(*src++); +} + +static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1= *dst, + dma_addr_t l2ptr_dma) +{ + u64 val =3D 0; + + val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); + val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; + + /* The HW has 64 bit atomicity with stores to the L2 STE table */ + WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); +} =20 #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4F3C331A6C for ; Mon, 17 Nov 2025 18:48:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405316; cv=none; b=Ev+Ne3QxKwO4kccAxDrIL5yLqRdJct5twmIchOiOIEjJGXTeAZ5q9tiBuIhtx0RA3RfvN3FGJv0sYCDVm04Nhg60jQc9Y2GTNBtNmPkoQsFPyZA2KOEkq4AdiUqG6rO0krx9ijHccIlI8G35t3TwE4/0/aPX3VzS74Oe8dHTCQk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405316; c=relaxed/simple; bh=Vclyp15KhjSu44V08/F6eiOMM27jme7DCCOu/Buxcyw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VxvdBXz8T25NgTQVzW07G2/WyI6y3o9EsHB1hRQ0DtIgLwD5Ufx2I8BB8alBMOFtuV8z6K4/x1AisnMmi+SXrpZbB8rH8+E+oBqcSSMGH3XrnDSzAA/lmilbQJ4ezA76FxVVUrwb2yAQoywYVKUlEua8+0+4Xj1gnZeawYlA7uU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Zu6fx6I+; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Zu6fx6I+" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-47788165c97so2428095e9.0 for ; Mon, 17 Nov 2025 10:48:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405312; x=1764010112; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Fa/YvcJ5tNlhBRUyXop3BZTS83Exwp3fcbT0/9k9OOI=; b=Zu6fx6I+kX3989ZjnSCFtb7gBvqv8PiIu9gJ7Weqv91Z1eSZjqEqb7WH8XoxK6Ni/a UJonC7ElrIBldnmvN4uzD/jyZ82O6lfM+3G83tBbGhUIHJA7g1Oeib3gbLQzvr4xSnkH aG56TQWKpizN+DYz4xmcEr5CkNerDEgLWkiOdkjHlgpLBFRlcxZihNPM8POxxwbjitZM tBdrQdvYtbBG3rcNzpBh7Lht8p+MsRAHpdWdQPrlLSYWJ/uAP7YGlRXWS1YRQFa6z380 bQQ3ya0XlUfcWIzd8mLsKkRByYAqmJeyt9VOb7g2ellxwRQ/vFM6DvH0GREeVq3HRegP 5wYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405312; x=1764010112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Fa/YvcJ5tNlhBRUyXop3BZTS83Exwp3fcbT0/9k9OOI=; b=K81nghswS740OYTccE2iooh6f2fzmJUCoRfQ2W+Nae2kdftHOY4ojMDk1GMzdEwT/+ ptnplgG4pJ1S1i6ssioG4MvrStXaEbFtvcv2u+O7q5ba4b2iWhVFdtER6D4qYE0fvONF vy2u7pFfxSdsICTt+l9oAIZ7qVgwVrP9+Qi1OPBc0j/t/72+SiuEVAYO0VAtgFg/JYxo AOugyRVNMswV/N0Ylo+0JuW1WwRuR/MDmDu95L5oIa9fJoKyjM6ApfPkvez1V/oHbztb +v/LpkWGWeO8k9m2nWHkw+72rvLyDVmFT/Azfz2ijUGou0LXwSqKyBAL41sJB4TTA6kk Sm4g== X-Forwarded-Encrypted: i=1; AJvYcCWrZHCneGhWs1v7ryvZNB50OObqRgl2qBuJgvG0bUDFnaHe5+rx3IGHF/yH0i3ghfbXsS2juhLx/0gyjNc=@vger.kernel.org X-Gm-Message-State: AOJu0YyVoWFQ+3Q7jSxamqeWBX4LyuR+pdlIlw6d2f/xGyp/vsHEbvu8 o9ejpQUJ30o8lFhzvXM+LNc8qrS8vEcIR0riZiVtyEjk3ie71BLcH/UHhvpO8n9Wl3lJGA3hbcJ jLZ02iOBV3CxxUA== X-Google-Smtp-Source: AGHT+IFGfjLxAYeewB8UtexvjesLcgKTw4qvbkwbmE1jIA9t3QqiMcAfRLlAcExJxKbgHrtdx0lO1xzwSnlrcA== X-Received: from wmgf6.prod.google.com ([2002:a05:600c:1546:b0:477:9fba:50e4]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1f87:b0:471:9da:524c with SMTP id 5b1f17b1804b1-4778fe69a76mr122513705e9.12.1763405312411; Mon, 17 Nov 2025 10:48:32 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:53 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-7-smostafa@google.com> Subject: [PATCH v5 06/27] iommu/arm-smmu-v3: Move TLB range invalidation into common code From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Range TLB invalidation has a very specific algorithm, instead of re-writing it for the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 69 ++++-------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 79 +++++++++++++++++++++ 2 files changed, 92 insertions(+), 56 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 1497ffcd4555..f6c3eeb4ecea 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2105,74 +2105,31 @@ static void arm_smmu_tlb_inv_context(void *cookie) arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } =20 +static void __arm_smmu_cmdq_batch_add(void *__opaque, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd) +{ + struct arm_smmu_device *smmu =3D (struct arm_smmu_device *)__opaque; + + arm_smmu_cmdq_batch_add(smmu, cmds, cmd); +} + static void __arm_smmu_tlb_inv_range(struct arm_smmu_cmdq_ent *cmd, unsigned long iova, size_t size, size_t granule, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_device *smmu =3D smmu_domain->smmu; - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; - size_t inv_range =3D granule; struct arm_smmu_cmdq_batch cmds; =20 if (!size) return; =20 - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* Get the leaf page size */ - tg =3D __ffs(smmu_domain->domain.pgsize_bitmap); - - num_pages =3D size >> tg; - - /* Convert page size of 12,14,16 (log2) to 1,2,3 */ - cmd->tlbi.tg =3D (tg - 10) / 2; - - /* - * Determine what level the granule is at. For non-leaf, both - * io-pgtable and SVA pass a nominal last-level granule because - * they don't know what level(s) actually apply, so ignore that - * and leave TTL=3D0. However for various errata reasons we still - * want to use a range command, so avoid the SVA corner case - * where both scale and num could be 0 as well. - */ - if (cmd->tlbi.leaf) - cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); - else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) - num_pages++; - } - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - - while (iova < end) { - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* - * On each iteration of the loop, the range is 5 bits - * worth of the aligned size remaining. - * The range in pages is: - * - * range =3D (num_pages & (0x1f << __ffs(num_pages))) - */ - unsigned long scale, num; - - /* Determine the power of 2 multiple number of pages */ - scale =3D __ffs(num_pages); - cmd->tlbi.scale =3D scale; - - /* Determine how many chunks of 2^scale size we have */ - num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; - cmd->tlbi.num =3D num - 1; - - /* range is num * 2^scale * pgsize */ - inv_range =3D num << (scale + tg); - - /* Clear out the lower order bits for the next iteration */ - num_pages -=3D num << scale; - } - - cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, &cmds, cmd); - iova +=3D inv_range; - } + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + smmu_domain->domain.pgsize_bitmap, + smmu->features & ARM_SMMU_FEAT_RANGE_INV, + smmu, __arm_smmu_cmdq_batch_add, &cmds); arm_smmu_cmdq_batch_submit(smmu, &cmds); } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 4aaf93945ee3..4a59b4d39c4f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1072,6 +1072,85 @@ static inline void arm_smmu_write_strtab_l1_desc(str= uct arm_smmu_strtab_l1 *dst, WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); } =20 +/** + * arm_smmu_tlb_inv_build - Create a range invalidation command + * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid. + * @iova: Start IOVA to invalidate + * @size: Size of range + * @granule: Granule of invalidation + * @pgsize_bitmap: Page size bit map of the page table. + * @is_range: Use range invalidation commands. + * @opaque: Pointer to pass to add_cmd + * @add_cmd: Function to send/batch the invalidation command + * @cmds: Incase of batching, it includes the pointer to the batch + */ +static inline void arm_smmu_tlb_inv_build(struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, unsigned long pgsize_bitmap, + bool is_range, void *opaque, + void (*add_cmd)(void *_opaque, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd), + struct arm_smmu_cmdq_batch *cmds) +{ + unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; + size_t inv_range =3D granule; + + if (is_range) { + /* Get the leaf page size */ + tg =3D __ffs(pgsize_bitmap); + + num_pages =3D size >> tg; + + /* Convert page size of 12,14,16 (log2) to 1,2,3 */ + cmd->tlbi.tg =3D (tg - 10) / 2; + + /* + * Determine what level the granule is at. For non-leaf, both + * io-pgtable and SVA pass a nominal last-level granule because + * they don't know what level(s) actually apply, so ignore that + * and leave TTL=3D0. However for various errata reasons we still + * want to use a range command, so avoid the SVA corner case + * where both scale and num could be 0 as well. + */ + if (cmd->tlbi.leaf) + cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); + else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) + num_pages++; + } + + while (iova < end) { + if (is_range) { + /* + * On each iteration of the loop, the range is 5 bits + * worth of the aligned size remaining. + * The range in pages is: + * + * range =3D (num_pages & (0x1f << __ffs(num_pages))) + */ + unsigned long scale, num; + + /* Determine the power of 2 multiple number of pages */ + scale =3D __ffs(num_pages); + cmd->tlbi.scale =3D scale; + + /* Determine how many chunks of 2^scale size we have */ + num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; + cmd->tlbi.num =3D num - 1; + + /* range is num * 2^scale * pgsize */ + inv_range =3D num << (scale + tg); + + /* Clear out the lower order bits for the next iteration */ + num_pages -=3D num << scale; + } + + cmd->tlbi.addr =3D iova; + add_cmd(opaque, cmds, cmd); + iova +=3D inv_range; + } +} + #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); void arm_smmu_sva_notifier_synchronize(void); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C96433345F for ; Mon, 17 Nov 2025 18:48:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405317; cv=none; b=IRS8xtD2moByXjhbm+RNhUyUonjwCSs5E3WSzNCvjyl8q2B3eN8kXnO8zC2GYIEYIyQtiFpdxkbaABisP2ZaV3Ov+QwQ+1bbWUQd3mpcBijNCUL6l2pkZd646JPj+o4NCRXTQ++lhSsvyLgolSkudHFYsBLmorhS/r6uPUq556Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405317; c=relaxed/simple; bh=oi0/lM+aBLYJf/HQK6DC0wmk8f5QQK92GHXZnx6nFfk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FCwZAOukKSojCzyEGNe+jIEkPiAxTUx72zZgqgCb9tDE8qNLqFGUp9Ci+jJctGwYTSMI/edkWEdijvxsS7P0tnMjJ73N8B1YgeEo/6KnTX+YxX7D0IFUEFyV1wIwJ8vfB6QKqFkNp/Hcri88FKo0o4nMVr2ViDmwI+b1ydewUdo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aLu9Kj5x; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aLu9Kj5x" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b7369635423so523946066b.2 for ; Mon, 17 Nov 2025 10:48:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405313; x=1764010113; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0hiT3u+aBwqtTyR01nuzLNZMiaS7VflcHmUp2NLVpzk=; b=aLu9Kj5xsn6ToImJu9DreCjFkijTd/xLdduypMi8oa8upG1ZRp/3483R2FMkebWHZ5 Zu7Wv2sjdpL+AWBDIOqysnq8vt9HYZaqn3Op5+BMdSP53BqLjQrswolAU+VyNpKPIpEQ YDTex4j28LSe1i47SeHrpVmY7BomtwxXs+L/pg6mmgIujt7tIZKwErTbtz09remkihc2 QMi471Qa8mtKimTkdVebtgRsQRY93dEDvz1uhxH/DY/Q6twxglDYKqQwtZGJeLViS9ZT sBNn/3bsSziVkRi5fnW/8DISmQRs9nwYLJE98k0IKWP+eikKDZnOUMrq5V64vSPmOd48 quOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405313; x=1764010113; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0hiT3u+aBwqtTyR01nuzLNZMiaS7VflcHmUp2NLVpzk=; b=RynqON3RwjKrf6Yw+2XyHiX37oSyVQTbB3Ar0VGQubYnN/5Lu801dCjCgylz/Fa6JH GUBMYf6e5GehC3MYsHjhFAPwomZNrxGC+OpZJHzSxnYKqKBacjY1FwnWR/Yq8AWGcVri fEwos18j8B2crfJT3gxNKK56w6wDePfwOIDfX/TzQNO3v5UgUvila+XnrdVpzx/rwS5E X798jY/V9Jz+JUdFbCKIqxxb5qUI1//gQB8cwZBYLy5HUZLZuCzg+QBzN/veXHcd6L9n nk9zb4OI5xUyupkk6c8EzAQB+PtPEu2tr1HBFK8LJ8cfRTXF7Xfma+9zorv0nWbr1SmS e/Lg== X-Forwarded-Encrypted: i=1; AJvYcCUnmXSw87uTcpzhu36BVvZvHbN0cLqjURVmZ5iKSN1ZbvMGVNiaxmPU8OAL5O2nrGAxwdLxHcFTT+5AA7g=@vger.kernel.org X-Gm-Message-State: AOJu0YxTFTCB1kApalhqTlzkFde0Y6O7y/bzg5NpomGUFKFX/hUYkGB8 PnDlrhE6e0LvBToMAon1oHdUye+D1Fkmx09Ipj1+0S79KsvEwheyQBDCLR+UaqTKOI1PTNOeWVl wLLGrE2YPCpj7iA== X-Google-Smtp-Source: AGHT+IFYRIXgsXNF82x3u/Gk1+ZyU3/lN7ti3VErqwwIPcz3ByOxhGpblAT8LR3u4vT/vVdR5di2euXFrcpalg== X-Received: from ejclk7.prod.google.com ([2002:a17:907:1787:b0:b73:7130:388f]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:94ca:b0:b73:709e:a20a with SMTP id a640c23a62f3a-b73709ea300mr1129061466b.47.1763405313254; Mon, 17 Nov 2025 10:48:33 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:54 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-8-smostafa@google.com> Subject: [PATCH v5 07/27] iommu/arm-smmu-v3: Move IDR parsing to common functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move parsing of IDRs to functions so that it can be re-used from the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 112 +++----------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 111 +++++++++++++++++++ 2 files changed, 126 insertions(+), 97 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index f6c3eeb4ecea..7b1bd0658910 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4109,57 +4109,17 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) /* IDR0 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); =20 - /* 2-level structures */ - if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; - - if (reg & IDR0_CD2L) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; - - /* - * Translation table endianness. - * We currently require the same endianness as the CPU, but this - * could be changed later by adding a new IO_PGTABLE_QUIRK. - */ - switch (FIELD_GET(IDR0_TTENDIAN, reg)) { - case IDR0_TTENDIAN_MIXED: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; - break; -#ifdef __BIG_ENDIAN - case IDR0_TTENDIAN_BE: - smmu->features |=3D ARM_SMMU_FEAT_TT_BE; - break; -#else - case IDR0_TTENDIAN_LE: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE; - break; -#endif - default: + smmu->features |=3D smmu_idr0_features(reg); + if (FIELD_GET(IDR0_TTENDIAN, reg) =3D=3D IDR0_TTENDIAN_RESERVED) { dev_err(smmu->dev, "unknown/unsupported TT endianness!\n"); return -ENXIO; } - - /* Boolean feature flags */ - if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) - smmu->features |=3D ARM_SMMU_FEAT_PRI; - - if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) - smmu->features |=3D ARM_SMMU_FEAT_ATS; - - if (reg & IDR0_SEV) - smmu->features |=3D ARM_SMMU_FEAT_SEV; - - if (reg & IDR0_MSI) { - smmu->features |=3D ARM_SMMU_FEAT_MSI; - if (coherent && !disable_msipolling) - smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; - } - - if (reg & IDR0_HYP) { - smmu->features |=3D ARM_SMMU_FEAT_HYP; - if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) - smmu->features |=3D ARM_SMMU_FEAT_E2H; - } + if (coherent && !disable_msipolling && + smmu->features & ARM_SMMU_FEAT_MSI) + smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; + if (smmu->features & ARM_SMMU_FEAT_HYP && + cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + smmu->features |=3D ARM_SMMU_FEAT_E2H; =20 arm_smmu_get_httu(smmu, reg); =20 @@ -4171,21 +4131,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n", str_true_false(coherent)); =20 - switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { - case IDR0_STALL_MODEL_FORCE: - smmu->features |=3D ARM_SMMU_FEAT_STALL_FORCE; - fallthrough; - case IDR0_STALL_MODEL_STALL: - smmu->features |=3D ARM_SMMU_FEAT_STALLS; - } - - if (reg & IDR0_S1P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S1; - - if (reg & IDR0_S2P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S2; - - if (!(reg & (IDR0_S1P | IDR0_S2P))) { + if (!(smmu->features & (ARM_SMMU_FEAT_TRANS_S1 | ARM_SMMU_FEAT_TRANS_S2))= ) { dev_err(smmu->dev, "no translation support!\n"); return -ENXIO; } @@ -4250,10 +4196,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) =20 /* IDR3 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); - if (FIELD_GET(IDR3_RIL, reg)) - smmu->features |=3D ARM_SMMU_FEAT_RANGE_INV; - if (FIELD_GET(IDR3_FWB, reg)) - smmu->features |=3D ARM_SMMU_FEAT_S2FWB; + smmu->features |=3D smmu_idr3_features(reg); =20 if (FIELD_GET(IDR3_BBM, reg) =3D=3D 2) smmu->features |=3D ARM_SMMU_FEAT_BBML2; @@ -4265,43 +4208,18 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) smmu->evtq.max_stalls =3D FIELD_GET(IDR5_STALL_MAX, reg); =20 /* Page sizes */ - if (reg & IDR5_GRAN64K) - smmu->pgsize_bitmap |=3D SZ_64K | SZ_512M; - if (reg & IDR5_GRAN16K) - smmu->pgsize_bitmap |=3D SZ_16K | SZ_32M; - if (reg & IDR5_GRAN4K) - smmu->pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); =20 /* Input address size */ if (FIELD_GET(IDR5_VAX, reg) =3D=3D IDR5_VAX_52_BIT) smmu->features |=3D ARM_SMMU_FEAT_VAX; =20 - /* Output address size */ - switch (FIELD_GET(IDR5_OAS, reg)) { - case IDR5_OAS_32_BIT: - smmu->oas =3D 32; - break; - case IDR5_OAS_36_BIT: - smmu->oas =3D 36; - break; - case IDR5_OAS_40_BIT: - smmu->oas =3D 40; - break; - case IDR5_OAS_42_BIT: - smmu->oas =3D 42; - break; - case IDR5_OAS_44_BIT: - smmu->oas =3D 44; - break; - case IDR5_OAS_52_BIT: - smmu->oas =3D 52; + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) smmu->pgsize_bitmap |=3D 1ULL << 42; /* 4TB */ - break; - default: + else if (!smmu->oas) { dev_info(smmu->dev, - "unknown output address size. Truncating to 48-bit\n"); - fallthrough; - case IDR5_OAS_48_BIT: + "unknown output address size. Truncating to 48-bit\n"); smmu->oas =3D 48; } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 4a59b4d39c4f..309194ceebe7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -27,6 +27,7 @@ struct arm_vsmmu; #define IDR0_STALL_MODEL_FORCE 2 #define IDR0_TTENDIAN GENMASK(22, 21) #define IDR0_TTENDIAN_MIXED 0 +#define IDR0_TTENDIAN_RESERVED 1 #define IDR0_TTENDIAN_LE 2 #define IDR0_TTENDIAN_BE 3 #define IDR0_CD2L (1 << 19) @@ -1072,6 +1073,116 @@ static inline void arm_smmu_write_strtab_l1_desc(st= ruct arm_smmu_strtab_l1 *dst, WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); } =20 +static inline u32 smmu_idr0_features(u32 reg) +{ + u32 features =3D 0; + + /* 2-level structures */ + if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) + features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; + + if (reg & IDR0_CD2L) + features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; + + /* + * Translation table endianness. + * We currently require the same endianness as the CPU, but this + * could be changed later by adding a new IO_PGTABLE_QUIRK. + */ + switch (FIELD_GET(IDR0_TTENDIAN, reg)) { + case IDR0_TTENDIAN_MIXED: + features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; + break; +#ifdef __BIG_ENDIAN + case IDR0_TTENDIAN_BE: + features |=3D ARM_SMMU_FEAT_TT_BE; + break; +#else + case IDR0_TTENDIAN_LE: + features |=3D ARM_SMMU_FEAT_TT_LE; + break; +#endif + } + + /* Boolean feature flags */ + if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) + features |=3D ARM_SMMU_FEAT_PRI; + + if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) + features |=3D ARM_SMMU_FEAT_ATS; + + if (reg & IDR0_SEV) + features |=3D ARM_SMMU_FEAT_SEV; + + if (reg & IDR0_MSI) + features |=3D ARM_SMMU_FEAT_MSI; + + if (reg & IDR0_HYP) + features |=3D ARM_SMMU_FEAT_HYP; + + switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { + case IDR0_STALL_MODEL_FORCE: + features |=3D ARM_SMMU_FEAT_STALL_FORCE; + fallthrough; + case IDR0_STALL_MODEL_STALL: + features |=3D ARM_SMMU_FEAT_STALLS; + } + + if (reg & IDR0_S1P) + features |=3D ARM_SMMU_FEAT_TRANS_S1; + + if (reg & IDR0_S2P) + features |=3D ARM_SMMU_FEAT_TRANS_S2; + + return features; +} + +static inline u32 smmu_idr3_features(u32 reg) +{ + u32 features =3D 0; + + if (FIELD_GET(IDR3_RIL, reg)) + features |=3D ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_FWB, reg)) + features |=3D ARM_SMMU_FEAT_S2FWB; + + return features; +} + +static inline u32 smmu_idr5_to_oas(u32 reg) +{ + switch (FIELD_GET(IDR5_OAS, reg)) { + case IDR5_OAS_32_BIT: + return 32; + case IDR5_OAS_36_BIT: + return 36; + case IDR5_OAS_40_BIT: + return 40; + case IDR5_OAS_42_BIT: + return 42; + case IDR5_OAS_44_BIT: + return 44; + case IDR5_OAS_48_BIT: + return 48; + case IDR5_OAS_52_BIT: + return 52; + } + return 0; +} + +static inline unsigned long smmu_idr5_to_pgsize(u32 reg) +{ + unsigned long pgsize_bitmap =3D 0; + + if (reg & IDR5_GRAN64K) + pgsize_bitmap |=3D SZ_64K | SZ_512M; + if (reg & IDR5_GRAN16K) + pgsize_bitmap |=3D SZ_16K | SZ_32M; + if (reg & IDR5_GRAN4K) + pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + return pgsize_bitmap; +} + /** * arm_smmu_tlb_inv_build - Create a range invalidation command * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid. --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77153330314 for ; Mon, 17 Nov 2025 18:48:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405319; cv=none; b=JSpbfLXo1uVj1YhxDGW8eqUtfOwMKcLA7bqHsBKG/BLh2JZE8SuGfWGtIGWH4i+F8O3pfKW0/pNhqB6lKOxzGNOurW9/gA+upZgCOXKidNSFLfROjADjwB/cDZYPy03BGsc1F0RVd+TfSIgMqf0vrL6YJxl6d4B5Fs6grVrEUq8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405319; c=relaxed/simple; bh=9zeMjcd2L1k8BuaEQ9HDOIEDJgUOeJvw5uMmbl2pBbA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=A6wIoFHIAhvHVMV8cr2yOGWyzMi/iaKz/6Z65Gw2YcAQyOlVM4lsaMNBkbp7tgws5aVPqwYC2EtqPbWJyrqedVZBK7nHSJ0JyJWWHsDc4OP+5zoVHDumBzAMNZwqMjo3JBPwPbLgzUkkywXNBjfMCQEEZhBDeBX7IHAv3WUdZbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=f07RFCDl; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="f07RFCDl" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-47775585257so37314035e9.1 for ; Mon, 17 Nov 2025 10:48:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405314; x=1764010114; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=55AGVVkYakMhSKq7jdoT4Ydc3YlHaQUlYhRGsZGie9E=; b=f07RFCDl5b3GtXGOMUhSdgRSzQYueXgjQ0c85pm2PAmsGpos+9kNNaivxz22OI9eGg WZwWEaLeh0bMQtNtahfwthJ1OdrCFSKjxuvUK+lYWdt1qZL96WCBrE4hajLUnQX4StB0 Cc4BW/MNWX6hlGTnnesCMqIyeZ/ce0bXf5g08O8in5qudmCrOkG4UVb8mgABWDgIWwCq S9SdDYnjdVq+36vm72dVTPBBa3tfjIzOJYUmtMnuQk4YEYTEJVag2SRj/2UqELI3JeTP pLuIPaa588fu6F4+d6UgXv/483kWmD2dejvEsE2bi8H4zKbyDCfIcE43muIu0OAGqq1A I+Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405314; x=1764010114; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=55AGVVkYakMhSKq7jdoT4Ydc3YlHaQUlYhRGsZGie9E=; b=p0JQA6pig+ydYjRIOPm7o7rHQTSd8+AysAX89qYFPfdjQ+U1ysP46S2o+yFUQ++11C CPrPE3wXaR1YkwWfsUqDebZIOXYL6324+GYcmu9Xf2Gw0FDIudzTEH7fVx+VcACR5lav /hGiljwG42TsLHwJrhmeEvFHel0aCPfC+mp+9UD61+9mBuQx4vpDJaWqU6/rkTpmx01e UUYA2ahB9kW9q8fIxFfu3xzZYbLPQ/NaW2DsVzLuN+4CK0kGBJzYRpOz9zRqKsEhlx0/ UQdikp6eRDTGw64ZCEHSsAsCmTkU3t4nbxCBewoSudi+eHYKpiILQqdD4cEUpJHqzDLV l79Q== X-Forwarded-Encrypted: i=1; AJvYcCXhwFweiLcOr5+ilsr0VMTYjl/Var5pkc/BKGsylkT2zNdJm7XfXvVir6XpvyIZd1jKA2rSPUa0p75AQSo=@vger.kernel.org X-Gm-Message-State: AOJu0YxtYFNSWiWD1692Vs1EflJBx/J7GT2rLNvnjpNovXEdyH8BKObr o2BjRafdQKQgO9QU76alADjs8kh1SWud8BRwU1FMJeeNvW58siOVmK7OwNZ/O1U2HOe0EK6gjlG iQqh4AK80d0yyZA== X-Google-Smtp-Source: AGHT+IETdIxY/B60kJGZ2vxQfTV572Z6M9F03+Q3FJ5EaaTwe2O+lpK+vSDpNGyLrCO0Jqd9FsK+JMhkS2KL/g== X-Received: from wmkz11.prod.google.com ([2002:a7b:c7cb:0:b0:477:98b9:1e26]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:630d:b0:471:1765:839c with SMTP id 5b1f17b1804b1-4778fea2bf7mr111910515e9.20.1763405314472; Mon, 17 Nov 2025 10:48:34 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:55 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-9-smostafa@google.com> Subject: [PATCH v5 08/27] KVM: arm64: iommu: Introduce IOMMU driver infrastructure From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To establish DMA isolation, KVM needs an IOMMU driver which provide ops implemented at EL2. Only one driver can be used and is registered with kvm_iommu_register_driver() by passing pointer to the ops. This must be called before module_init() which is the point KVM initializes. Signed-off-by: Mostafa Saleh Signed-off-by: Jean-Philippe Brucker --- arch/arm64/include/asm/kvm_host.h | 5 +++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 18 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 5 +++++ arch/arm64/kvm/iommu.c | 15 +++++++++++++++ 7 files changed, 59 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/iommu.h create mode 100644 arch/arm64/kvm/hyp/nvhe/iommu/iommu.c create mode 100644 arch/arm64/kvm/iommu.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 64302c438355..fb2551ba8798 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1652,4 +1652,9 @@ static __always_inline enum fgt_group_id __fgt_reg_to= _group_id(enum vcpu_sysreg p; \ }) =20 +#ifndef __KVM_NVHE_HYPERVISOR__ +struct kvm_iommu_ops; +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); +#endif + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 3ebc0570345c..66959c048492 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -24,7 +24,7 @@ kvm-y +=3D arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.= o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ - vgic/vgic-v5.o + vgic/vgic-v5.o iommu.o =20 kvm-$(CONFIG_HW_PERF_EVENTS) +=3D pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) +=3D pauth.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h new file mode 100644 index 000000000000..1ac70cc28a9e --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NVHE_IOMMU_H__ +#define __ARM64_KVM_NVHE_IOMMU_H__ + +#include + +struct kvm_iommu_ops { + int (*init)(void); +}; + +int kvm_iommu_init(void); + +#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index a244ec25f8c5..8210788d6f88 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o +=3D -Wno-override-init =20 hyp-obj-y :=3D timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o= host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \ + iommu/iommu.o hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../en= try.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o hyp-obj-y +=3D ../../../kernel/smccc-call.o diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c new file mode 100644 index 000000000000..a01c036c55be --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * IOMMU operations for pKVM + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +/* Only one set of ops supported */ +struct kvm_iommu_ops *kvm_iommu_ops; + +int kvm_iommu_init(void) +{ + if (!kvm_iommu_ops || !kvm_iommu_ops->init) + return -ENODEV; + + return kvm_iommu_ops->init(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index eff76be89329..de79803e7439 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -328,6 +329,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 + ret =3D kvm_iommu_init(); + if (ret) + goto out; + ret =3D hyp_ffa_init(ffa_proxy_pages); if (ret) goto out; diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c new file mode 100644 index 000000000000..c9041dcb6c57 --- /dev/null +++ b/arch/arm64/kvm/iommu.c @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 Google LLC + * Author: Mostafa Saleh + */ + +#include + +extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); + +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) +{ + kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; + return 0; +} --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C4903321BC for ; Mon, 17 Nov 2025 18:48:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405319; cv=none; b=X5dyaZBiKAGjhbnOZMaZHeQTAHZtfmUgXnAMkzjTXkqIQPejHl9m+8X+s8aaWQkHGTFbdhHhBBqDHkRabLMeKCR9uiM4jpLSk00ntWI2NhoFDBqUhRpEeodLsKC01K4CjvO923R+VF5ZyNJdk3pH3dOdJkFQUHQt3+V8MFSvDak= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405319; c=relaxed/simple; bh=pOVN4YIln6H9/psIOPh7ht/smthij5U8tZ78Nxr8DpI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OyX/R5Gh0xGvitzAe1q0pFWeK3vFMPvZw1zU+nF84aN2NKwOr5Wzgrc4ihub9D7xbpSIr8UD3KnIrECflTJp8Vav5eBYHLopth89hBN8FKx4P3FqZmTLD8vnekI6UuNHiqobCFPtlsNNuhGY286TWd2w2j6GfF6DM/n3voRddu8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=I+jVEAcx; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="I+jVEAcx" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-477a60a23adso6833395e9.2 for ; Mon, 17 Nov 2025 10:48:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405315; x=1764010115; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=86k/mgvI8zrVMpw0/0P0jY4VXuRBlWrQXsQcBMGtbfo=; b=I+jVEAcxv0kkgWr0CZEto/CzKnINcPD+npakUWCSHfCyM9sHQeYsLOWV5fcHTM1o9Y Hi/KwtKlHPfHfkXV7FGCN47Rh1CvzhzzM5a3p6Vhj+rvRWVMCmyaqwfYhAbJoztFQnpG 4cZqtykb/wiuETuAaRB/Fg+qzGQjyk4gpKHOKNv2ASKHa4UQnoTmyG4wQGnMQVLJQDiG Q313M1XY7eAiDYfHKVWGp8SOXttkWheL5e+t4AblKVb06SU2io5jnDaX7AGj1DzTM3Pe IUmS4uBJzqpWthB0Hd8gPLX0j4dA69tSwXZnX8tabJjTBF2RiDMu08C3QxoLENaZHbIY oFOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405315; x=1764010115; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=86k/mgvI8zrVMpw0/0P0jY4VXuRBlWrQXsQcBMGtbfo=; b=o5rtgy3IcRNnZMQjST6mCUdbziMzRT+ZZGyh5ww7DIHmPmvrGIuNZaXEuIw9W7r0MN aAcrlWvPLeJbLqi3cUZDgl3/wSyuA1kpF+tf99tdv5cOWzRLVInHJS/5MUXTC471MtVw LNaeB3GGBv4fYIM07iJV3uKG0AjqS6Os1BZspZuECIxcADf9V60RGpvBNQG/DKvxwiye tly+F7pmCKHdOpUNXL969whFt9MS031ZzLd7u8gQ8s4drseQdGRvrjySySnpX1koLUhi t5s0bF8MkhPGdPTJMN04HJLSOEzjiCWjmzehVLePM81pNLSTEHt04+Uby4ztr0saaF9D TF5Q== X-Forwarded-Encrypted: i=1; AJvYcCXLuB08yx454hBZ3xoSKwPjlAifcOcTna/38/BpiVQ+rqkCd34k8BGntAWie1aRs7yiPZ9w7EOho7oOnjY=@vger.kernel.org X-Gm-Message-State: AOJu0YyN1d5GZgKWR7WIrYP4URSSpmrLSvssxmzGUkX8ONCumePUV5q9 ujoys6/tT0ZUDc4E0YMs3R7EizpcpycFpnnzGvaKBHHB86b5lIczNNjxQpKpw0/7Q2VmqOrWVqN 7M7cpFlkhnLsTFw== X-Google-Smtp-Source: AGHT+IEj4z2+t+yDF5un/88u8JO0bKDq8uupoiOB9c833JgKVOtSjewqPTAdRZkxIX1uyr5HoS4xTfM/25eAYQ== X-Received: from wmbz8.prod.google.com ([2002:a05:600c:c088:b0:475:df4b:95a9]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4753:b0:477:333a:f71f with SMTP id 5b1f17b1804b1-4778fe60658mr132994865e9.17.1763405315623; Mon, 17 Nov 2025 10:48:35 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:56 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-10-smostafa@google.com> Subject: [PATCH v5 09/27] KVM: arm64: iommu: Shadow host stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a shadow page table for the IOMMU that shadows the host CPU stage-2 into the IOMMUs to establish DMA isolation. An initial snapshot is created after the driver init, then on every permission change a callback would be called for the IOMMU driver to update the page table. For some cases, an SMMUv3 may be able to share the same page table used with the host CPU stage-2 directly. However, this is too strict and requires changes to the core hypervisor page table code, plus it would require the hypervisor to handle IOMMU page faults. This can be added later as an optimization for SMMUV3. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 4 ++ arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 83 ++++++++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 5 ++ 3 files changed, 90 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 1ac70cc28a9e..219363045b1c 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -3,11 +3,15 @@ #define __ARM64_KVM_NVHE_IOMMU_H__ =20 #include +#include =20 struct kvm_iommu_ops { int (*init)(void); + void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 int kvm_iommu_init(void); =20 +void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index a01c036c55be..414bd4c97690 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,15 +4,94 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + #include +#include +#include =20 /* Only one set of ops supported */ struct kvm_iommu_ops *kvm_iommu_ops; =20 +/* Protected by host_mmu.lock */ +static bool kvm_idmap_initialized; + +static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) +{ + int iommu_prot =3D 0; + + if (prot & KVM_PGTABLE_PROT_R) + iommu_prot |=3D IOMMU_READ; + if (prot & KVM_PGTABLE_PROT_W) + iommu_prot |=3D IOMMU_WRITE; + if (prot =3D=3D PKVM_HOST_MMIO_PROT) + iommu_prot |=3D IOMMU_MMIO; + + /* We don't understand that, might be dangerous. */ + WARN_ON(prot & ~PKVM_HOST_MEM_PROT); + return iommu_prot; +} + +static int __snapshot_host_stage2(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags visit) +{ + u64 start =3D ctx->addr; + kvm_pte_t pte =3D *ctx->ptep; + u32 level =3D ctx->level; + u64 end =3D start + kvm_granule_size(level); + int prot =3D IOMMU_READ | IOMMU_WRITE; + + /* Keep unmapped. */ + if (pte && !kvm_pte_valid(pte)) + return 0; + + if (kvm_pte_valid(pte)) + prot =3D pkvm_to_iommu_prot(kvm_pgtable_stage2_pte_prot(pte)); + else if (!addr_is_memory(start)) + prot |=3D IOMMU_MMIO; + + kvm_iommu_ops->host_stage2_idmap(start, end, prot); + return 0; +} + +static int kvm_iommu_snapshot_host_stage2(void) +{ + int ret; + struct kvm_pgtable_walker walker =3D { + .cb =3D __snapshot_host_stage2, + .flags =3D KVM_PGTABLE_WALK_LEAF, + }; + struct kvm_pgtable *pgt =3D &host_mmu.pgt; + + hyp_spin_lock(&host_mmu.lock); + ret =3D kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker); + /* Start receiving calls to host_stage2_idmap. */ + kvm_idmap_initialized =3D !ret; + hyp_spin_unlock(&host_mmu.lock); + + return ret; +} + int kvm_iommu_init(void) { - if (!kvm_iommu_ops || !kvm_iommu_ops->init) + int ret; + + if (!kvm_iommu_ops || !kvm_iommu_ops->init || + !kvm_iommu_ops->host_stage2_idmap) return -ENODEV; =20 - return kvm_iommu_ops->init(); + ret =3D kvm_iommu_ops->init(); + if (ret) + return ret; + return kvm_iommu_snapshot_host_stage2(); +} + +void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot) +{ + hyp_assert_lock_held(&host_mmu.lock); + + if (!kvm_idmap_initialized) + return; + kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot)); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index c3eac0da7cbe..f60acfb868d0 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -15,6 +15,7 @@ #include =20 #include +#include #include #include #include @@ -529,6 +530,7 @@ static void __host_update_page_state(phys_addr_t addr, = u64 size, enum pkvm_page_ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id) { int ret; + enum kvm_pgtable_prot prot; =20 if (!range_is_memory(addr, addr + size)) return -EPERM; @@ -538,6 +540,9 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 = size, u8 owner_id) if (ret) return ret; =20 + prot =3D owner_id =3D=3D PKVM_ID_HOST ? PKVM_HOST_MEM_PROT : 0; + kvm_iommu_host_stage2_idmap(addr, addr + size, prot); + /* Don't forget to update the vmemmap tracking for the host */ if (owner_id =3D=3D PKVM_ID_HOST) __host_update_page_state(addr, size, PKVM_PAGE_OWNED); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 495AE33344D for ; Mon, 17 Nov 2025 18:48:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405321; cv=none; b=N08NubAiRyF8oGz1KHKDJMf4x2tskYC3VGktuGiI3K3h+RrQjEKIwtMLyotdDYi8qe81TyGGIueheSlebhU0dfrtExxxHqwo/1KgDlbWCZ1kJ+x3ep+mzit6oznu0N0Z1EA+FYD6bjKXJqRXYdVTr5E42pzVJkee2TwWtiq+F+U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405321; c=relaxed/simple; bh=NAMroDrXjFET8e/V1jEPHTmEuoxT2nPqbJIpZ7VUQF4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dfXxLe5w/brHhX/pDTUgBzRgCzPEyYl6YVaTv+c9r6x9ohSf6qVmw1zsBukdI9K1X7VK5N3BrSjagalQsSpQDrDVX6AchE3O3pSSHBhel9fJzn98/JtnpxJpZck9epbBXHmxLeTKDSwt3FED2B9LjhMeIrniGSwNHd1FqYmFcsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=S2VB/vK5; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="S2VB/vK5" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-b726a3c3214so396789066b.2 for ; Mon, 17 Nov 2025 10:48:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405316; x=1764010116; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Rk1fl88JutZZYCoKcXxRTtxtyt/5+rf5iGOfbus1BTk=; b=S2VB/vK5mh2Yn1+3ALzfnJKVVsSvEzR2V8K7B/tbaRb3g1YJoQ9VN9gydSAqd651FV 2LmEja+4sEQ//8B7/u/G4t80pL+vWmQQD6ji6yh8ppzA7mLtmm1a7fwow8QlLUwaWAWq v/euv4YuktyJbieUfpt8UN9A1/Yac/rN4HD7oTfVObOBQ7dunW8QOnTX1LfjqUaBQJ8M OEtOA7Z1jHqbuEcgOXainXjFYs1DiT0M/SzdAK2XDrRBthg5y/hA+MeGMgQxDoFSBEGW egbmP5FfS2+mLrLP3OOPDcdkIrP9fSa3nBDePJhY7CmuGblVW/+q0ei/r4rEJ7HE58OD yPfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405316; x=1764010116; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rk1fl88JutZZYCoKcXxRTtxtyt/5+rf5iGOfbus1BTk=; b=RwjmyNVVFxdh5by2gGQWCfRFM1C83QgPudUFWgKpZQXqPfQki33w0M2ttJHLr/WORu rd5hP27drlzaz+lyBSRKqSxxJLFcErmFKZsGvQKa4D6s1yB7PMC6xNetB6Jn2GIAGuC1 S5Fp4QalFPA082S4Rlxoxr/9Y8TPtYkaSzyDx09qhvnA/z3lh65wVkVQJXUduNdf4E14 uTJv1gwAhHA4wlS8DTnn1kP6tmP2QlSnRKWyLXAjgoM6Eym9wcqOzfUG/jZ2Po6DvTha z42yllDe+6qWMWn7csUXDzwUZVREdIlYRWYBmHclBJmdE1M/27WN4neJFIA4IYxV55k6 y46Q== X-Forwarded-Encrypted: i=1; AJvYcCWysI0n0Uyager09t7/QYlAIs5NYBYfPRHhZdmJf5C87vm4aL/obt5XZbjs3o5l8v7thVcTviHao70qpVM=@vger.kernel.org X-Gm-Message-State: AOJu0Yxu+8vLkHcDcEuRGTyStcOlrgOkvlgVkFK4efFaS+K/ktqomRCn qOzEgYVitADL2gOTrS6hyeilJYDfVZmwaa4x+CVnOWA1cKr8RsLMNC4PKesvK73nrR5Ssp4P68b CWzB1TLdskiAnFQ== X-Google-Smtp-Source: AGHT+IFK6mFtucBRLiWaaV+YANl4dQaIe6ZXNzcjcMCQe/1qsUgD5bMWkFpnXRoT7wZjLhUKC6CSylQU/X+dYw== X-Received: from ejcvd5.prod.google.com ([2002:a17:907:d205:b0:b73:7a81:20af]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:3c82:b0:b73:16fc:d469 with SMTP id a640c23a62f3a-b736793dcdcmr1369485766b.51.1763405316436; Mon, 17 Nov 2025 10:48:36 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:57 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-11-smostafa@google.com> Subject: [PATCH v5 10/27] KVM: arm64: iommu: Add memory pool From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" IOMMU drivers would require to allocate memory for the shadow page table. Similar to the host stage-2 CPU page table, the IOMMU pool is allocated early from the carveout and it's memory is added in a pool which the IOMMU driver can allocate from and reclaim at run time. As this is too early for drivers to use init calls, a default value can be set in the kernel config through IOMMU_POOL_PAGES, which then can be overridden later from the kernel command line: "kvm-arm.hyp_iommu_pages". Later when the driver registers, it will pass how many pages it needs, and if it was less than what was allocated, it will fail to register. Signed-off-by: Mostafa Saleh --- .../admin-guide/kernel-parameters.txt | 4 +++ arch/arm64/include/asm/kvm_host.h | 3 +- arch/arm64/kvm/Kconfig | 7 +++++ arch/arm64/kvm/hyp/include/nvhe/iommu.h | 5 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 20 +++++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 16 +++++++++- arch/arm64/kvm/iommu.c | 31 ++++++++++++++++++- arch/arm64/kvm/pkvm.c | 1 + 8 files changed, 82 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 6c42061ca20e..f843d10a3dfc 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3059,6 +3059,10 @@ trap: set WFI instruction trap =20 notrap: clear WFI instruction trap + kvm-arm.hyp_iommu_pages=3D + [KVM, ARM, EARLY] + Number of pages allocated for the IOMMU pool from the + KVM carveout when running in protected mode. =20 kvm_cma_resv_ratio=3Dn [PPC,EARLY] Reserves given percentage from system memory area for diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index fb2551ba8798..5496c52d0163 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1654,7 +1654,8 @@ static __always_inline enum fgt_group_id __fgt_reg_to= _group_id(enum vcpu_sysreg =20 #ifndef __KVM_NVHE_HYPERVISOR__ struct kvm_iommu_ops; -int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops, size_t pool_p= ages); +size_t kvm_iommu_pages(void); #endif =20 #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 4f803fd1c99a..6a1bd82a0d07 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -83,4 +83,11 @@ config PTDUMP_STAGE2_DEBUGFS =20 If in doubt, say N. =20 +config IOMMU_POOL_PAGES + hex "Number of pages reserved for IOMMU pool" + depends on KVM && IOMMU_SUPPORT + default 0x0 + help + IOMMU pool is used with protected mode to allocated IOMMU drivers page = tables. + endif # VIRTUALIZATION diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 219363045b1c..9f4906c6dcc9 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -10,8 +10,11 @@ struct kvm_iommu_ops { void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 -int kvm_iommu_init(void); +int kvm_iommu_init(void *pool_base, size_t nr_pages); =20 void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, enum kvm_pgtable_prot prot); +void *kvm_iommu_donate_pages(u8 order); +void kvm_iommu_reclaim_pages(void *ptr); + #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index 414bd4c97690..a0df34ecf6b0 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -15,6 +15,7 @@ struct kvm_iommu_ops *kvm_iommu_ops; =20 /* Protected by host_mmu.lock */ static bool kvm_idmap_initialized; +static struct hyp_pool iommu_pages_pool; =20 static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) { @@ -72,7 +73,7 @@ static int kvm_iommu_snapshot_host_stage2(void) return ret; } =20 -int kvm_iommu_init(void) +int kvm_iommu_init(void *pool_base, size_t nr_pages) { int ret; =20 @@ -80,6 +81,13 @@ int kvm_iommu_init(void) !kvm_iommu_ops->host_stage2_idmap) return -ENODEV; =20 + if (nr_pages) { + ret =3D hyp_pool_init(&iommu_pages_pool, hyp_virt_to_pfn(pool_base), + nr_pages, 0); + if (ret) + return ret; + } + ret =3D kvm_iommu_ops->init(); if (ret) return ret; @@ -95,3 +103,13 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phy= s_addr_t end, return; kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot)); } + +void *kvm_iommu_donate_pages(u8 order) +{ + return hyp_alloc_pages(&iommu_pages_pool, order); +} + +void kvm_iommu_reclaim_pages(void *ptr) +{ + hyp_put_page(&iommu_pages_pool, ptr); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index de79803e7439..c245ea88c480 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -22,6 +22,13 @@ =20 unsigned long hyp_nr_cpus; =20 +/* See kvm_iommu_pages() */ +#ifdef CONFIG_IOMMU_POOL_PAGES +size_t hyp_kvm_iommu_pages =3D CONFIG_IOMMU_POOL_PAGES; +#else +size_t hyp_kvm_iommu_pages; +#endif + #define hyp_percpu_size ((unsigned long)__per_cpu_end - \ (unsigned long)__per_cpu_start) =20 @@ -33,6 +40,7 @@ static void *selftest_base; static void *ffa_proxy_pages; static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; static struct hyp_pool hpool; +static void *iommu_base; =20 static int divide_memory_pool(void *virt, unsigned long size) { @@ -70,6 +78,12 @@ static int divide_memory_pool(void *virt, unsigned long = size) if (!ffa_proxy_pages) return -ENOMEM; =20 + if (hyp_kvm_iommu_pages) { + iommu_base =3D hyp_early_alloc_contig(hyp_kvm_iommu_pages); + if (!iommu_base) + return -ENOMEM; + } + return 0; } =20 @@ -329,7 +343,7 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 - ret =3D kvm_iommu_init(); + ret =3D kvm_iommu_init(iommu_base, hyp_kvm_iommu_pages); if (ret) goto out; =20 diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c index c9041dcb6c57..6143fd3e1de3 100644 --- a/arch/arm64/kvm/iommu.c +++ b/arch/arm64/kvm/iommu.c @@ -7,9 +7,38 @@ #include =20 extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); +extern size_t kvm_nvhe_sym(hyp_kvm_iommu_pages); =20 -int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops, size_t pool_p= ages) { + /* See kvm_iommu_pages() */ + if (pool_pages > kvm_nvhe_sym(hyp_kvm_iommu_pages)) { + kvm_err("Missing memory for the IOMMU pool, need 0x%zx pages, check kvm-= arm.hyp_iommu_pages", + pool_pages); + return -ENOMEM; + } + kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; return 0; } + +size_t kvm_iommu_pages(void) +{ + /* + * This is called very early during setup_arch() where no initcalls, + * so this has to call specific functions per each KVM driver. + * So we allow a config option that can set the defaul value for + * the IOMMU pool that can overridden by a command line option. + * When the driver registers it will pass the number pages needed + * for it's page tables, if less that what the system has already + * allocated we fail. + */ + return kvm_nvhe_sym(hyp_kvm_iommu_pages); +} + +/* Number of pages to reserve for iommu pool*/ +static int __init early_hyp_iommu_pages(char *arg) +{ + return kstrtoul(arg, 10, &kvm_nvhe_sym(hyp_kvm_iommu_pages)); +} +early_param("kvm-arm.hyp_iommu_pages", early_hyp_iommu_pages); diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 24f0f8a8c943..b9d212b48c04 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -63,6 +63,7 @@ void __init kvm_hyp_reserve(void) hyp_mem_pages +=3D hyp_vmemmap_pages(STRUCT_HYP_PAGE_SIZE); hyp_mem_pages +=3D pkvm_selftest_pages(); hyp_mem_pages +=3D hyp_ffa_proxy_pages(); + hyp_mem_pages +=3D kvm_iommu_pages(); =20 /* * Try to allocate a PMD-aligned region to reduce TLB pressure once --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62A4933710A for ; Mon, 17 Nov 2025 18:48:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405323; cv=none; b=sseRw2epwEqL/iSeb+6JI7ZbLpeDo96PVuw5nxN3mVFU9CK7o5ONtOu2oSJ52+mV00IynHWcOL+NnxkikPNm1CC+BOVj+LvR37v3AmXniuIPAGNJIkX/ezoQY/yWV0u3caQKXdX/nWmFbWn3GlwIYz2zonTlDRKiQjzvyPKgxvc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405323; c=relaxed/simple; bh=6nFSpBu3ZwObAoKoHCayRp3PVnE6EITEZuEZzaiIGGQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OXnYziWxs2xEmcuffCfXYtUIQR4aV8Hh9M1tgLQXTGxFxxwBJe9iUlH48hy1LN4NMTmVB6YSkNvNdkNSNpxd34bCfqypa99NRCjKyK5u0E75ATH51d98kysRUlW2887zm5iDZcgMFaQg4gSBKqr+QCDrDdqfmmV0oIOYxOdIEhE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ry8QedJ4; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ry8QedJ4" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-477a11d9f89so1924275e9.3 for ; Mon, 17 Nov 2025 10:48:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405317; x=1764010117; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UGnO/Bwz/Bdpjz8RxXVYUEJKHIK5XJgWwQocmBNRacI=; b=ry8QedJ4P5bt7XXIOvkBr6SRNXPsrOIHgMwsGqnP+NNQ9Fl1NDxi0YuW+yrHR16sO8 CksCZqJhk9BGBiXSbJMRZC9+J7DIyvgDGu8PVjVitQnwfaBsheByzs/sLCypMMPvO0Dj JuebT3z09opMROvmpFTk9V6uuYhoqEkRx7sDLISR3wLiluc5fso2F3G3ONnmCyUJGMIV qy+WZLL2TPnj/LV3qFDxYO5aKGZIMLqnigKPyqhp1ciSBaO5pTLfoVK1QvBULuoCR/W8 zNvf5E9ux+oITq462qLlMYtVEft4KpYBbUTE+K3c9hN53gp2qfaah58EactLdpelwWIH XI2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405317; x=1764010117; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UGnO/Bwz/Bdpjz8RxXVYUEJKHIK5XJgWwQocmBNRacI=; b=Fdse4//EumV//xEQNJ36uBa1WF1AvF4dkIbD2IijP362ZslZerKAkHA3sr8+Hw5wSp 25XHBgr7Mojv5D6ktJekkXyuBS18iciur8d1w9C6fcWMo1gmF0HyeXbQ/e8QfxLfeJlt 6iiksgFSa3MwLLBh3+KnmgHbkVFujUgMXqV5riQA/fsTB9ij80m33o6xiVVRZ8xlk1bP HITfvR8hx98Tt6TIs2DUW9DqEtAYHedY1zfOTogw86yRNd1g0jmvnrO+0QamjpR8yq2c PCu+04vk9rkCEGhbBUb1qvhcGXXFyORkeWo7JsZA+auNe2WhkzSKosRn2O1iEAbi+TU5 7lGQ== X-Forwarded-Encrypted: i=1; AJvYcCWYT+Ok0vz4CJ0hqbdTfUqSp/3JOIvZKR9tINRM/KhjDu5wcYcuzEqqeIGZI3Ai88MmECOCQaBhHMmLu0o=@vger.kernel.org X-Gm-Message-State: AOJu0YzH3vCNLtp7gd8AycN+WVeZN27vS3DZIOJrCwGmIb7LaL430MDi ckpRobwQaqb7u52ZkeGhhlBDXXeBOVjrOFxw80/zonFTaRJNSHyMR5ErJgg0Rhf06EwZFhFsiHd 5HtkEiajpw0EUVA== X-Google-Smtp-Source: AGHT+IFrvnE51rjqhZgZikSAId4eBRwM0rBdtcUofGQG56OV6eJr8JyqvovBQ6X7CtmMQHN+xu1fxfhfh3qxIg== X-Received: from wmbgz10.prod.google.com ([2002:a05:600c:888a:b0:477:afa:d217]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8b23:b0:477:7bd2:693f with SMTP id 5b1f17b1804b1-4778fe60615mr126251305e9.6.1763405317263; Mon, 17 Nov 2025 10:48:37 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:58 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-12-smostafa@google.com> Subject: [PATCH v5 11/27] KVM: arm64: iommu: Support DABT for IOMMU From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SMMUv3 driver need to trap and emulate access to the MMIO space to provide emulation for the kernel. Add a handler for DABTs for IOMMU drivers to be able to do so, in case the host fault in page, check if it's part of IOMMU emulation first. Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_arm.h | 2 ++ arch/arm64/kvm/hyp/include/nvhe/iommu.h | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 15 +++++++++++++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 ++++++++++ 4 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_= arm.h index 1da290aeedce..8d63308ccd5c 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -331,6 +331,8 @@ =20 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */ #define HPFAR_MASK (~UL(0xf)) + +#define FAR_MASK GENMASK_ULL(11, 0) /* * We have * PAR [PA_Shift - 1 : 12] =3D PA [PA_Shift - 1 : 12] diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 9f4906c6dcc9..10fe4fbf7424 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -8,6 +8,7 @@ struct kvm_iommu_ops { int (*init)(void); void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); + bool (*dabt_handler)(struct user_pt_regs *regs, u64 esr, u64 addr); }; =20 int kvm_iommu_init(void *pool_base, size_t nr_pages); @@ -16,5 +17,5 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_= addr_t end, enum kvm_pgtable_prot prot); void *kvm_iommu_donate_pages(u8 order); void kvm_iommu_reclaim_pages(void *ptr); - +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index a0df34ecf6b0..c9f2103a25fe 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,6 +4,10 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + +#include + #include =20 #include @@ -113,3 +117,14 @@ void kvm_iommu_reclaim_pages(void *ptr) { hyp_put_page(&iommu_pages_pool, ptr); } + +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr) +{ + if (kvm_iommu_ops && kvm_iommu_ops->dabt_handler && + kvm_iommu_ops->dabt_handler(regs, esr, addr)) { + /* DABT handled by the driver, skip to next instruction. */ + kvm_skip_host_instr(); + return true; + } + return false; +} diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index f60acfb868d0..4c47df895579 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -595,6 +595,11 @@ static int host_stage2_idmap(u64 addr) return ret; } =20 +static bool is_dabt(u64 esr) +{ + return ESR_ELx_EC(esr) =3D=3D ESR_ELx_EC_DABT_LOW; +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault; @@ -617,6 +622,11 @@ void handle_host_mem_abort(struct kvm_cpu_context *hos= t_ctxt) */ BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS)); addr =3D FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12; + addr |=3D fault.far_el2 & FAR_MASK; + + if (is_dabt(esr) && !addr_is_memory(addr) && + kvm_iommu_host_dabt_handler(&host_ctxt->regs, esr, addr)) + return; =20 ret =3D host_stage2_idmap(addr); BUG_ON(ret && ret !=3D -EAGAIN); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA867329E55 for ; Mon, 17 Nov 2025 18:48:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405324; cv=none; b=qZOcV1c2o2CHDgXzEdGVSPE2u7BcVyURovbVlX3hnsvMW4/PilZ22vrdIT+/QbNC8Ws1yMidII7yB3jbUtZdXjM/bTElUUyupIRgLeBQ8Oj1Eb/mv1MWaEyRKGRhPnBYaZGhrcDfdN3MoQsV2eFPFjWcXRGhgR+tWMpKEAVVMvk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405324; c=relaxed/simple; bh=X1SB8xBsyDP9CObrHvadpvuvmKMLHDjQeJHpDgXwBrU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lTWbGVYzzzPCkjmFk/L/WEM+YysayW/+sgq540p/QWRz34SGcht7oyB+WtZZTopK860rykMuW14NQDg+qkl83oVvLzirLvRcvvIp4U99RKYEgGVmQ+W7XhdyU98fhAVYdwW5s4Oq4y/q6R/ujQWPrBE+gWs9kdjL+ViNoQvuUyI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JkQQe+0O; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JkQQe+0O" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-42b3c965ce5so3210024f8f.2 for ; Mon, 17 Nov 2025 10:48:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405318; x=1764010118; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VXBWv8Dw5hE8mlU4Sm4VTohDwotOri7T9rWgww1xmos=; b=JkQQe+0OMx78LAbxq7nTwU/i55+XSkrNeQOkbZJK+/j/2Qc4wA4PeKfn4nYVDqrWvB 6i86i0yXnywz8f94bV/7/hPRcnJYn4Eb2wgmEuAWmDWS9YXdn3yLAQ+PuxMnqw4t7HUZ E+cRzoHkirdyBqZq9UOPVjSngYYh2vk3eqr42HTpDXaURdN5sxXX1d212yBdEYF/zGBg LVqJhGThMuvGXIMM8hE4CJqdkgTJu8qKvsCIAkXlJHTvbbduNZ5XRpnH+ZIcP+eD58lQ gzTfiuxLscl6WvE/E9xHd7ytLzUvV14Xei39Ch9Q954To6IhjBK5HfbNxWZhNG9ZZVp6 bqug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405318; x=1764010118; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VXBWv8Dw5hE8mlU4Sm4VTohDwotOri7T9rWgww1xmos=; b=Wszu2mBKZ0/CdGsQVetbRT3B0cu1eR9qq9Umzd2SwzXG7gxkCmDikgvLkd8oxA0/LS F4TVcijusrfiWPcD8+gXOy0vw4ET0fsEcmeRisaUH+aBnWpmsn+lfCfJWXJqEEK3h0JT 8FHjl4vI+NmEBUHQzKWiDqd/CwTq7Q/xh0Fjp/jC14CGMqEz6t8DzvnzTkQxQQBJ59Co W8P4aF0JlWVApdpIY6BPJmqZoP64oIf3uQMw1nHHeq6wIUg+qVx97YyVdgz+iahZx7Ae txa7aYlRStMTjLiTjtqPU4FbXamdGnRrl4b8iXtD735cu2TJ2szEbVzgYW9RV1b1gmHF bYGg== X-Forwarded-Encrypted: i=1; AJvYcCVPXp4IVfaV2imAqJldsnFyi8FEBhMwWn0j0+1GnoJq8/7NZLYcjwlwlXWP7G5uGLembyLRGHfrYcr5UUo=@vger.kernel.org X-Gm-Message-State: AOJu0YxaeIM6FP06xOmLYc4R0Hp7/PyARAvepu4vh4SE7T/xAPrJQDbN tG3hr5LBPU8CPX3Y9zB8D9wLVhkVhIYfBi1GLbYJf1uIe1TXnqC1pOilMOPxLO2OLj7+S4BefI+ Woj+WbOMI8khbmA== X-Google-Smtp-Source: AGHT+IHQBDWwjO/s1zZNuEFY1oL5kAjhY98F1o68vfkU7yeDw2zqyTvhCAFTbYHvFaunhrv4os5cxfn9oBJJZA== X-Received: from wrxe9.prod.google.com ([2002:a05:6000:1209:b0:428:563f:ce93]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:615:b0:42b:3a1b:f71a with SMTP id ffacd0b85a97d-42b5934e2b6mr13388046f8f.23.1763405318003; Mon, 17 Nov 2025 10:48:38 -0800 (PST) Date: Mon, 17 Nov 2025 18:47:59 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-13-smostafa@google.com> Subject: [PATCH v5 12/27] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jean-Philippe Brucker Add the skeleton for an Arm SMMUv3 driver at EL2. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 5 ++++ drivers/iommu/arm/Kconfig | 9 ++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 29 +++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 16 ++++++++++ 4 files changed, 59 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 8210788d6f88..197685817546 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -32,6 +32,11 @@ hyp-obj-y +=3D ../../../kernel/smccc-call.o hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o hyp-obj-y +=3D $(lib-objs) =20 +HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 + +hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o + ## ## Build rules for compiling nVHE hyp code ## Output of this folder is `kvm_nvhe.o`, a partially linked object diff --git a/drivers/iommu/arm/Kconfig b/drivers/iommu/arm/Kconfig index ef42bbe07dbe..7eeb94d2499d 100644 --- a/drivers/iommu/arm/Kconfig +++ b/drivers/iommu/arm/Kconfig @@ -142,3 +142,12 @@ config QCOM_IOMMU select ARM_DMA_USE_IOMMU help Support for IOMMU on certain Qualcomm SoCs. + +config ARM_SMMU_V3_PKVM + bool "ARM SMMUv3 support for protected Virtual Machines" + depends on KVM && ARM64 && ARM_SMMU_V3=3Dy + help + Enable a SMMUv3 driver in the KVM hypervisor, to protect VMs against + memory accesses from devices owned by the host. + + Say Y here if you intend to enable KVM in protected mode. diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c new file mode 100644 index 000000000000..fa8b71152560 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM hyp driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +#include + +#include "arm_smmu_v3.h" + +size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; +struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; + +static int smmu_init(void) +{ + return -ENOSYS; +} + +static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) +{ +} + +/* Shared with the kernel driver in EL1 */ +struct kvm_iommu_ops smmu_ops =3D { + .init =3D smmu_init, + .host_stage2_idmap =3D smmu_host_stage2_idmap, +}; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h new file mode 100644 index 000000000000..f6ad91d3fb85 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_ARM_SMMU_V3_H +#define __KVM_ARM_SMMU_V3_H + +#include + +struct hyp_arm_smmu_v3_device { +}; + +extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); +#define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count) + +extern struct hyp_arm_smmu_v3_device *kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smm= us); +#define kvm_hyp_arm_smmu_v3_smmus kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus) + +#endif /* __KVM_ARM_SMMU_V3_H */ --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96481338596 for ; Mon, 17 Nov 2025 18:48:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405323; cv=none; b=i9OoCPeOUYZtOd1DP5ib6sAgBIFCIlZV+14dppNiDX9to+K1v13mQR3tcYvAS43EVGMsHdZ4TPYdraIPyevIMPnprB0eLPuJrCA0Dybpqsxs6D/IZl9pYOXwuZfaU+MzyO7nV7F+KjhwYCuG6Hi3StUkryse1QrPDTJ9hH3VQ9U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405323; c=relaxed/simple; bh=INcTJzS700s05ZGwDrQtXEsdidTABqC6bRW5Y2ydOoI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sdLF40tqBEtF5TZsIv+p7jQ0e49faAI1kXdaT76Tzv2BmEBewgg1ZuHjqsc8LaSXkSXplcpF7fNXkVEKzr2URpmXlx0U65J0zfmNmJ67EerJY64kxU5/8MkVRaztPYRmioaKru4WXwF5ayJEvDWN+IhoSQhUiDSAJSjkM2XPKeE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YA4Len0v; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YA4Len0v" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b70b974b818so540101366b.1 for ; Mon, 17 Nov 2025 10:48:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405319; x=1764010119; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6trFz58Gu4UJEu1ThwWz2XtrxsLtl476+uXL3Pe7jcQ=; b=YA4Len0vtKnkhkzMnD1Ar5vuOQtTfMbE+7Wy9wS3SYHCZ9BjOJln/sRNyKab1V70EB YARwXwawRlOpJTYKWBz8vPYJhEk937jHcjoQ1M+eL06H7skZLuFPpL3Acu8PtigdYCkE arsRxDv1s8Ria0a92ifUsjrMQ6R+IOSN9tGkfY6Ciq5oOa3kvfLNW/cBvICmMJpbSp0V 02DIHAmu7zsmmpVNoxaGGCDiGHy19SucxddaRd0OUX/eAc3qjDFYx2Gb7OSeRYZAkYHr R6lJXS4GHNQi0Hb8E2GjKYl99OOV+anPT35s0L9kNtW8/C645ZyTEYQ5HSvWnMbtGPvC 2oNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405319; x=1764010119; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6trFz58Gu4UJEu1ThwWz2XtrxsLtl476+uXL3Pe7jcQ=; b=ro3VlzSj8yY3vwXJQ3p7yAvohbbQejuXXvXSzjrpBEbYaQYjMscRNfJdAXjEqycqkm Gyxb5SaP8h/caQQnee62PFumj3alFVg2dqDg6ufb+DIALoMUnvBbE50uktS6XTD5EGun aDtNI4QTcoPRs5ef4NbomQ9rOXhP3YTXYUEgioXAsp7Lc+ODWVy513JM7qZ3Q1pV4NSA J1nIDa3tXVU5Kii8+CSd6mlEaHCPo9cMnGBzA5NK1WpfaBuI5Y82PKVNOMRcf1AfXcoT d0irmayjUGqMu+1oxI2sYpR4uh0i2RADoIl9edeWg2DHNbqDEzYT8iHNzHcWJy2iFUU8 31Ww== X-Forwarded-Encrypted: i=1; AJvYcCWm8JGOkNLjqMgT0HZCIW/R6oxI8lhsBV5Lx1RqRVYuCw48LJN2k5BnLLnqv5GhXlNLmRedW7wQIk39Gvg=@vger.kernel.org X-Gm-Message-State: AOJu0YzgLEHWbXZMYClueXapekf1b8tJIL/JeHPqB62ECnZrVuykrErX ixmxyQ5Ju6qGrr2HCz87rsfkYXHN9GU/jtU8vKGkgyXKEaUTtgzNauJRbU343inNs/WH6oNw8m3 B6kih9veUUygSMw== X-Google-Smtp-Source: AGHT+IE3+viO3srJHAcHB3c5RRKlLg8klxOmp1l9jNQ0CtxcTZZi09dcfnPoyGFkFEx9s/uAX82FyaMdOVjXZg== X-Received: from ejsr3.prod.google.com ([2002:a17:906:7fc3:b0:b6d:5546:e475]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:3c90:b0:b70:b077:b94b with SMTP id a640c23a62f3a-b7367b99c79mr1398956966b.43.1763405318837; Mon, 17 Nov 2025 10:48:38 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:00 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-14-smostafa@google.com> Subject: [PATCH v5 13/27] iommu/arm-smmu-v3-kvm: Add the kernel driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A platform driver is created to probe the SMMUs, which then creates aux devices for the emulated SMMUs. Then the driver registers with KVM. Next, the SMMUv3 driver will probed the aux devices. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 91 +++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index c9ce392e6d31..c3fc5c4a4a1e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -4,5 +4,6 @@ arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o +arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D arm-smmu-v3-kvm.o =20 obj-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) +=3D arm-smmu-v3-test.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c new file mode 100644 index 000000000000..ca12560639c5 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM host driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include +#include + +#include +#include +#include + +#include "arm-smmu-v3.h" +#include "pkvm/arm_smmu_v3.h" + +extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); + +static size_t smmu_hyp_pgt_pages(void) +{ + /* + * SMMUv3 uses the same format as stage-2 and hence have the same memory + * requirements, we add extra 500 pages for L2 ste. + */ + if (of_find_compatible_node(NULL, NULL, "arm,smmu-v3")) + return host_s2_pgtable_pages() + 500; + return 0; +} + +static struct platform_driver smmuv3_nesting_driver; +static int smmuv3_nesting_probe(struct platform_device *pdev) +{ + return 0; +} + +static int kvm_arm_smmu_v3_register(void) +{ + size_t nr_pages =3D smmu_hyp_pgt_pages(); + int ret; + + if (!is_protected_kvm_enabled() || !nr_pages) + return 0; + + ret =3D platform_driver_probe(&smmuv3_nesting_driver, smmuv3_nesting_prob= e); + if (ret) + return ret; + + return kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_= ops))), + nr_pages); +}; + +static int smmu_create_aux_device(struct device *dev, void *data) +{ + static int dev_id; + struct auxiliary_device *auxdev; + + auxdev =3D __devm_auxiliary_device_create(dev, "protected_kvm", + "smmu_v3_emu", NULL, dev_id++); + if (!auxdev) + return -ENODEV; + + auxdev->dev.parent =3D dev; + + return 0; +} + +static struct platform_driver smmuv3_nesting_driver; +static int kvm_arm_smmu_v3_post_init(void) +{ + if (!is_protected_kvm_enabled()) + return 0; + + WARN_ON(driver_for_each_device(&smmuv3_nesting_driver.driver, NULL, + NULL, smmu_create_aux_device)); + + return 0; +} + +static const struct of_device_id smmuv3_nested_of_match[] =3D { + { .compatible =3D "arm,smmu-v3", }, + { }, +}; + +static struct platform_driver smmuv3_nesting_driver =3D { + .driver =3D { + .name =3D "smmuv3-nesting", + .of_match_table =3D smmuv3_nested_of_match, + }, +}; +late_initcall(kvm_arm_smmu_v3_post_init); +subsys_initcall(kvm_arm_smmu_v3_register); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 94DAD33971E for ; Mon, 17 Nov 2025 18:48:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405324; cv=none; b=rTv7x2kbNupd/iQ98F1FiINM1nOrokBvIdT1I5NkTgAK1fWwfx8hdjlCpmOHbUIaT8LWnZKvKyyRjmniXAeNMG9gLoPPzHY2rce53qpzWkpEihJjko7MQ7F/VX1uHAi+j++2XVSbRrnEbDlJyjRCzPDc8lT6M7NaNfDuFXzg9ws= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405324; c=relaxed/simple; bh=Tcqv38/05N/B44jMwcSYWvtznjIegtRBbmE71X1CBgw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BshQYIpf6/qyvpAcZYZ2GZO6fxjr7xJdV+X8X6yOQ3oGRszFe4ueuMTyw7+rXaPkpn2+D7zNEsrN0/g8z+LTFDzqiARCu67KAZdPR0ckxMfutYlZxm2b0Xwb3njH1MBQscmUvWLcA2t8gGMnVfB0bwy5sB0u6hf6rij4wbtPg2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dIf8b61t; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dIf8b61t" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-42b2ad29140so2137452f8f.0 for ; Mon, 17 Nov 2025 10:48:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405319; x=1764010119; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=C8whZ0pZrNIRwagAm0ORPy5mFVOpkEuU2Ow6ExYwaLg=; b=dIf8b61tU/FnUSqbKfFNZRXyTGYlml5jywrucIFzvMV99UJkQgMVobflMXaZqGWmF0 zplbGdQ6P6J1DkAhHsH6BXL3i/Id9yWaQxQtlEFLASNC0A01W+AK34Y0nj6waC0SMJew N4Y3sPt4teV3ve44VidZ3I8+5Hplwztk2+chOp7RlKQQv1PJbqhLhQFE4nlGCfmCvOAo P44NW7yxhnpv7cY8GfSO+sWI8qe0Mjwqf9skCzBz+xWcgHvCZ4vwXeR3f1ppOfoVB9Bg iITkTBch9wsnhYYebseVeITsPWnAqTXeTM67tUNn2AcpFhfWe5JTRmBEUdXYjnqRUsD8 nG/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405319; x=1764010119; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=C8whZ0pZrNIRwagAm0ORPy5mFVOpkEuU2Ow6ExYwaLg=; b=LMT/rHYcSvaWNXAA6mImIMM4ZUz1xC3E1BNQUdUwk1phKJiWKNoICe1hAhcekuCgYc WdlG8qz9eF4FuONV/NKuhVB8nYOn5iIhSBroID05E9CYEy3ScvpW2/i7j5VJ/ZUWor60 4VKj6ruqsGHB9Btrqz3Yw+DyPMjBXpxvz63Apga0+m3N4uCQWFT3J6PgygBy7j0BKmtK wSDO5hbjpzspLrwy4VU7huTrJKm0d3cbLLM1ZHXRQZdSyT7+sDbRIz+z72BuFlW1dTYV Yj8A2VPRXlYftqQ+1n6/g09JFNfQNdFGBAPv2yu/0Kx+uwVJ7KiXMMOyAemH24RLaFS3 vH/Q== X-Forwarded-Encrypted: i=1; AJvYcCUa/V1YGdrJj7Dm8fEkvETeQhbi459WKEYUai7tR/raTiWwZP/L1A/k1bitIMAZjKCN2g5sSMB5Y5GeaBY=@vger.kernel.org X-Gm-Message-State: AOJu0YyA9aT/ij/7tGWrCNEie07XDdZoAjvVNkyn3Bq4U+COdUBb5Onz tASyudxcbAemzUGVNIzgYyuuuZdukvB7Mc6NHn0MICyrxqUQpYGAdtaTQO75/Zxz0alvjDG9QDd JL5TpU4xjOd8urQ== X-Google-Smtp-Source: AGHT+IGsLY0ihnfe+d9VivrIlxJpw79fEdNZ7d3k+PXAdZb1HiZsb1oiiqepnVn54b8AHulaH6k3LpFgrXdMyA== X-Received: from wrmf14.prod.google.com ([2002:adf:e90e:0:b0:42b:b28a:6748]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2c0c:b0:42b:3806:2bb4 with SMTP id ffacd0b85a97d-42b59342e50mr13284074f8f.27.1763405319568; Mon, 17 Nov 2025 10:48:39 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:01 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-15-smostafa@google.com> Subject: [PATCH v5 14/27] iommu/arm-smmu-v3: Support probing KVM emulated devices From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When KVM runs in protected mode, and CONFIG_ARM_SMMU_V3_PKVM is enabled, it will manage the SMMUv3 HW using trap and emulate and present emulated SMMUs to the host kernel. In that case, those SMMUs will be on the aux bus, so make it possibly to the driver to probe those devices. Otherwise, everything else is the same as the KVM emulation complies with the architecutre, so the driver doesn't need to be modified. Suggested-by: Jason Gunthorpe Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 58 +++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 7b1bd0658910..851d47bedae6 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -11,6 +11,7 @@ =20 #include #include +#include #include #include #include @@ -4604,6 +4605,63 @@ static struct platform_driver arm_smmu_driver =3D { module_driver(arm_smmu_driver, platform_driver_register, arm_smmu_driver_unregister); =20 +#ifdef CONFIG_ARM_SMMU_V3_PKVM +/* + * Now we have 2 devices, the aux device bound to this driver, and pdev + * which is the physical platform device. + * This part is a bit hairy but it works due to the fact that + * CONFIG_ARM_SMMU_V3_PKVM forces both drivers to be built in. + * The struct device for the SMMU is used in the following cases: + * 1) Printing using dev_*() + * 2) DMA memory alloc (dmam_alloc_coherent, devm_*) + * 3) Requesting resources (iomem, sysfs) + * 4) Probing firmware info (of_node, fwnode...) + * 5) Dealing with abstracted HW resources (irqs, MSIs, RPM) + * 6) Saving/reading driver data + * For point 4) and 5) we must use the platform device. + * For, 1) pdev is better for debuggability. + * For 2), 3), 6) it's better to use the bound device. + * However that doesn't really work: + * For 2) The DMA allocation using the aux device will fail, as + * we need to setup some device DMA attrs (mask), to match the + * platform. + * For 6) Some contexts from the pdev as MSI, it needs to use the + * drvdata. + * Based on the following: + * 1- Both drivers must be built-in to enable this (enforced by Kconfig), + * which means that none of them can be removed. + * 2- The KVM driver doesn't do anythng at runtime and doesn't use drvdata. + * We can keep the driver simple and to claim the platform device in all c= ases. + */ +static int arm_smmu_device_probe_emu(struct auxiliary_device *auxdev, + const struct auxiliary_device_id *id) +{ + struct device *parent =3D auxdev->dev.parent; + + dev_info(&auxdev->dev, "Probing from %s\n", dev_name(parent)); + return arm_smmu_device_probe(to_platform_device(parent)); +} + +static void arm_smmu_device_shutdown_emu(struct auxiliary_device *auxdev) +{ + arm_smmu_device_shutdown(to_platform_device(auxdev->dev.parent)); +} + +const struct auxiliary_device_id arm_smmu_aux_table[] =3D { + { .name =3D "protected_kvm.smmu_v3_emu" }, + { }, +}; + +struct auxiliary_driver arm_smmu_driver_emu =3D { + .name =3D "arm-smmu-v3-emu", + .id_table =3D arm_smmu_aux_table, + .probe =3D arm_smmu_device_probe_emu, + .shutdown =3D arm_smmu_device_shutdown_emu, +}; + +module_auxiliary_driver(arm_smmu_driver_emu); +#endif + MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); MODULE_ALIAS("platform:arm-smmu-v3"); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AB9633BBD7 for ; Mon, 17 Nov 2025 18:48:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405325; cv=none; b=IyogUCDYWpgNC/V7j+WPGf41TAqeNBX5sfOO9i9X+tBucToJS6aOSL9CCJfJ4qemwXZcn9aWAcUUm1BgK0Hg6vecbZ0xfH8w5YkOVf6UpElP5EvOW/lcrJXU9IjdJlBPefLZkSejNjYpMBFuJuIxphVyt5JYIVQy/GJky9H5tDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405325; c=relaxed/simple; bh=i72KEyIIF7BldyepX+BF2QCGkIH25YglohZoplhDZ30=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=r9QwNnl2GhvkhsUYIQlgQ5+bxd9eEGhA6RP7ky57/ZloLYlfXyGS5Dno3FzHIjk7d81ld8ywu7BquZeASHi+HL4CV01+XAZWOgw0UdrchWpuja7BGfNNSOOsioEvG3XmjZzG9FmCLtpxBsUpIgNgpChmK2JcGvTwQq51riFkubc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VILsXl9J; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VILsXl9J" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b7343d2a0fbso493762366b.3 for ; Mon, 17 Nov 2025 10:48:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405321; x=1764010121; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Pg6/d7tYH7Fa8SuxEA+vck9sHqkvG7ZrFQ8YnS0s4pY=; b=VILsXl9J4UrgPoWi7bMtUPCRmToZfSqNnkuMxuYtF460GA9r8RhtcEekZSncoPbVKf Rszu0vVRmU6A8T6faWVmYjgg8U7k4Jujwr3F/A2IRvnm67fm9C9U5OVC5sIUqzyArE7T ZJqp3IqrVmITHFSqLyX7CLHMoqju3+6TTA//zerErQvGeQBMH+ixhroRn075RwsLfYSL Kps9aYBkJLFRJ6d5uwhPOHT5h72i6CRlHMP5xPeboaemYFAoRa3IKIZ9k6x5W5S6QcFt 4jP20O22xndpThiwtAFhgULupW/IkNiVq6znuUvTQ1R9L1zpkChxVnGYiXjj8rPT/ucP mUDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405321; x=1764010121; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Pg6/d7tYH7Fa8SuxEA+vck9sHqkvG7ZrFQ8YnS0s4pY=; b=dXAHyv81OfD6Cpe5vcYCkHWLeCVWfizwn3mNRR5r9i5gK0m5D+uXmofZNm7H4PkQXh lv0QEAfLHVFx9J02RYoJDgcYRLNdXDW6CnyLCNG6yRA9jOchyRA7E2+GKhEkzpSDbzQ2 8rmioT47i6BbqzOzqsaGR5RWZVuFoPrhWgo+1RvED+Wpxc3KhJnY5qNThDkx4H8VQFan llZ1PniEgdTF8FbyrS9NSWn5h4Omb2+Y/cqPUgBpTmBivRlzLX/+AE8W+uRnuZ2fpLSN JdPn0WTS/0P62nt9NmAncVdrOgl5Q3Yf9TEJSW7nEQopyQOZps2bxQjV2Gfrs/fLFNQh qHIw== X-Forwarded-Encrypted: i=1; AJvYcCV4kJInG0wBCPf4UytPJwqTKKnNF8x2+eL/yPjKtiJZ0oB3rEMErNTOOobggUcVDn0qSc7BYW29wIhwiTE=@vger.kernel.org X-Gm-Message-State: AOJu0YwdZjAbBCaP3IJO+Pag6q5iQOaLmCbt4eqhoseoy/vFiGoauOJ9 3Bo6Jj7tkAz3FfwyXXQlw+g9g57i5ozNdnnw1qdHKUybhAsuYTPrsPakr9dXu2AlTOnzmVUPKyR SsvKExm4nDyD4Zw== X-Google-Smtp-Source: AGHT+IEDmtdye0WfWxlkb3rtc6UM9OGL+b9K1JFhVnvxv97zVFh9YFE5Gri7gLnInPVmef/IeMxDtKAYgxWvdw== X-Received: from ejczi1.prod.google.com ([2002:a17:907:e981:b0:b70:b2b1:8ca]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:97c9:b0:b73:5db5:b6d9 with SMTP id a640c23a62f3a-b7367c16560mr1552133566b.55.1763405320730; Mon, 17 Nov 2025 10:48:40 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:02 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-16-smostafa@google.com> Subject: [PATCH v5 15/27] iommu/arm-smmu-v3-kvm: Create array for hyp SMMUv3 From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As the hypervisor has no access to firmware tables, the device discovery is done from the kernel, where it parses firmware tables and populates a list of devices to the hypervisor, which later takes over. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 82 ++++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 13 +++ 2 files changed, 91 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index ca12560639c5..1d72951b7b53 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -8,6 +8,7 @@ #include =20 #include +#include #include #include =20 @@ -16,6 +17,45 @@ =20 extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 +static size_t kvm_arm_smmu_count; +static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; +static size_t kvm_arm_smmu_cur; + +static void kvm_arm_smmu_array_free(void) +{ + int order; + + order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); + free_pages((unsigned long)kvm_arm_smmu_array, order); +} + +/* + * The hypervisor have to know the basic information about the SMMUs + * from the firmware. + * This has to be done before the SMMUv3 probes and does anything meaningf= ul + * with the hardware, otherwise it becomes harder to reason about the SMMU + * state and we'd require to hand-off the state to the hypervisor at certa= in point + * while devices are live, which is complicated and dangerous. + * Instead, the hypervisor is interested in a very small part of the probe= path, + * so just add a separate logic for it. + */ +static int kvm_arm_smmu_array_alloc(void) +{ + int smmu_order; + struct device_node *np; + + for_each_compatible_node(np, NULL, "arm,smmu-v3") + kvm_arm_smmu_count++; + + if (!kvm_arm_smmu_count) + return -ENODEV; + smmu_order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)= ); + kvm_arm_smmu_array =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, = smmu_order); + if (!kvm_arm_smmu_array) + return -ENOMEM; + return 0; +} + static size_t smmu_hyp_pgt_pages(void) { /* @@ -30,6 +70,21 @@ static size_t smmu_hyp_pgt_pages(void) static struct platform_driver smmuv3_nesting_driver; static int smmuv3_nesting_probe(struct platform_device *pdev) { + struct resource *res; + struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; + + res =3D platform_get_resource(pdev, IORESOURCE_MEM, 0); + smmu->mmio_addr =3D res->start; + smmu->mmio_size =3D resource_size(res); + if (smmu->mmio_size < SZ_128K) { + dev_err(&pdev->dev, "MMIO region too small(%pr)\n", &res); + return -EINVAL; + } + + if (of_dma_is_coherent(pdev->dev.of_node)) + smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; + + kvm_arm_smmu_cur++; return 0; } =20 @@ -41,12 +96,31 @@ static int kvm_arm_smmu_v3_register(void) if (!is_protected_kvm_enabled() || !nr_pages) return 0; =20 - ret =3D platform_driver_probe(&smmuv3_nesting_driver, smmuv3_nesting_prob= e); + ret =3D kvm_arm_smmu_array_alloc(); if (ret) return ret; =20 - return kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_= ops))), - nr_pages); + ret =3D platform_driver_probe(&smmuv3_nesting_driver, smmuv3_nesting_prob= e); + if (ret) + goto out_err; + + ret =3D kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu= _ops))), + nr_pages); + if (ret) + goto out_err; + + /* + * These variables are stored in the nVHE image, and won't be accessible + * after KVM initialization. Ownership of kvm_arm_smmu_array will be + * transferred to the hypervisor as well. + */ + kvm_hyp_arm_smmu_v3_smmus =3D kvm_arm_smmu_array; + kvm_hyp_arm_smmu_v3_count =3D kvm_arm_smmu_count; + return ret; + +out_err: + kvm_arm_smmu_array_free(); + return ret; }; =20 static int smmu_create_aux_device(struct device *dev, void *data) @@ -67,7 +141,7 @@ static int smmu_create_aux_device(struct device *dev, vo= id *data) static struct platform_driver smmuv3_nesting_driver; static int kvm_arm_smmu_v3_post_init(void) { - if (!is_protected_kvm_enabled()) + if (!is_protected_kvm_enabled() || !kvm_arm_smmu_cur) return 0; =20 WARN_ON(driver_for_each_device(&smmuv3_nesting_driver.driver, NULL, diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index f6ad91d3fb85..744ee2b7f0b4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -4,7 +4,20 @@ =20 #include =20 +/* + * Parameters from the trusted host: + * @mmio_addr base address of the SMMU registers + * @mmio_size size of the registers resource + * @features Features of SMMUv3, subset of the main driver + * + * Other members are filled and used at runtime by the SMMU driver. + * @base Virtual address of SMMU registers + */ struct hyp_arm_smmu_v3_device { + phys_addr_t mmio_addr; + size_t mmio_size; + void __iomem *base; + u32 features; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D682333CE8B for ; Mon, 17 Nov 2025 18:48:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405326; cv=none; b=tAbyE1X8EhdfE/BuG4S3AzPrdFn1voCIXEGJXzRqDHypDiaqSG3ASuVWGQnvFGBHWLGHLVxAeDD/uKiHwbetkQYQ9Z9tqVpjwI9GO7CiF5y5ei/ydBWmDkJUruUDgMr0QyuypVM9FBKkC2eHLWq8GuQEHchhokLKsMMxi7MKgFU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405326; c=relaxed/simple; bh=iLqOj/XaV65ljxTgcZoZCOXuA7GIKuDSXUnmXUF7ZCw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dE08QsXvIzwHBYNAdoIa8HTvXMCgnUN4FAJafGNiwtKA2PvLImT5Pbg2p8S1GC60aqssrmx2xdg5itk44mf+w+nwcvUA+bFiv2gcP7lFKPPwRnnymLdpGjutc3VzxzsFNscR28bod/D5VZAiWjS2u5BakGxKcajiFZUcVEiAIBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PXVb2rZE; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PXVb2rZE" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-42b2b642287so2698888f8f.2 for ; Mon, 17 Nov 2025 10:48:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405322; x=1764010122; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HZRHy3svuNchmJFYToVRnx2mDpnN1m/iPQ4+pK/3fxU=; b=PXVb2rZEqLBWomzeEa6eF+ZDmXf6G7TunEECYZwQ9+ncAyEz1eFRPz80eW2n57b1XE EvSt7sIMqIHbKwSZWdWSyW0stAVgnurlgDYVy7ENW4aW8XtZvE5WFx38oWZCkXTqMwUG cHLuhLuB1tG/jt/eg5Aorf0tLAfE3iX0XosZlAgOAmXRFjzzBT77O2hWHPQE2eFAl0Y0 O/g36vK/cCgcUj/SVyb0V+P6SWsjViojccvSrVBclKD9xasOz9h4nU6sn5g3PcG7FDYO nRJHvv8/8NxQ82fBObn54Py+Yhbrn6Hl8O6NPwFz/NbZnW+O+C3yOQuN28FcoL1FMoQd Asog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405322; x=1764010122; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HZRHy3svuNchmJFYToVRnx2mDpnN1m/iPQ4+pK/3fxU=; b=MH4yeVnQnnHbVm60KWRUq2qDAo+0qxuduIqAJTBwGXojeexM1qMatWTdFOWm0iVavx S2cpLNipjgx4jNYLpYtWWJzt6bCAvHYJeWuGva20/KJhEy4TmbVyizYwrEy+uadSb2Na KN76UCrJ8qlTlR6GVZgqApiFFr4XjXrOCNNmjcCFbaucPvDqvY0AAwxCoqKBdJGUZ9Cm 1+SWqClVfoMxUmapxEbwAt1cnBfXiMFAu304X9wOkIPIUtdljOUXthgZ2GyG4L3Xe4K2 p9yJtCyfeaUJya2biXPVrKcAzzCex3ybotY0GzwnRabtJJmtRBpHdt7aSzPfYPL7JmjN L+hA== X-Forwarded-Encrypted: i=1; AJvYcCUDFKP2RZHBZrsilv52D1kOSiLw/Qj/guufDJ6vXveTFO+5GWJpxZLru4lQO24y3f0vH6+PdPypIci55rw=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6PPMo1ZSS7CfMQsXI4Qpluvsvofiwgs8xaal3YKBSpTh2nxqh 1b39D20xCT/LkrriU3LbgsD9w+am+ULsWjD9QuzHu/Rb2q1LhpZmXStwx3JvN0R8vrslAzfvLIe o5C7bZxBKDW+POQ== X-Google-Smtp-Source: AGHT+IEzcJDMw53+AVSQNytxPGimAg5Et4Wx8256qZ8eJwNoP5G/5wvwA6REx65+VnhZk9Fj+izitvg/zOxMdw== X-Received: from wrpz8.prod.google.com ([2002:adf:f748:0:b0:42b:32f2:9092]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2509:b0:3fb:aca3:d5d9 with SMTP id ffacd0b85a97d-42b5933916bmr12787870f8f.1.1763405321905; Mon, 17 Nov 2025 10:48:41 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:03 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-17-smostafa@google.com> Subject: [PATCH v5 16/27] iommu/arm-smmu-v3-kvm: Take over SMMUs From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Donate the array with SMMU description to the hypervisor as it can't be changed by the host after de-privileges. Also, donate the SMMU resources to the hypervisor. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 81 ++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index fa8b71152560..b56feae81dda 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -7,15 +7,94 @@ #include =20 #include +#include =20 #include "arm_smmu_v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +#define for_each_smmu(smmu) \ + for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ + (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ + (smmu)++) + +/* Transfer ownership of memory */ +static int smmu_take_pages(u64 phys, size_t size) +{ + WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size)); + return __pkvm_host_donate_hyp(phys >> PAGE_SHIFT, size >> PAGE_SHIFT); +} + +static void smmu_reclaim_pages(u64 phys, size_t size) +{ + WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size)); + WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT)); +} + +/* Put the device in a state that can be probed by the host driver. */ +static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) +{ + int i; + size_t nr_pages =3D smmu->mmio_size >> PAGE_SHIFT; + + for (i =3D 0 ; i < nr_pages ; ++i) { + u64 pfn =3D (smmu->mmio_addr >> PAGE_SHIFT) + i; + + WARN_ON(__pkvm_hyp_donate_host_mmio(pfn)); + } +} + +static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) +{ + int i; + size_t nr_pages; + + if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) + return -EINVAL; + + nr_pages =3D smmu->mmio_size >> PAGE_SHIFT; + for (i =3D 0 ; i < nr_pages ; ++i) { + u64 pfn =3D (smmu->mmio_addr >> PAGE_SHIFT) + i; + + /* + * This should never happen, so it's fine to be strict to avoid + * complicated error handling. + */ + WARN_ON(__pkvm_host_donate_hyp_mmio(pfn)); + } + smmu->base =3D hyp_phys_to_virt(smmu->mmio_addr); + + return 0; +} + static int smmu_init(void) { - return -ENOSYS; + int ret; + struct hyp_arm_smmu_v3_device *smmu; + size_t smmu_arr_size =3D PAGE_ALIGN(sizeof(*kvm_hyp_arm_smmu_v3_smmus) * + kvm_hyp_arm_smmu_v3_count); + + kvm_hyp_arm_smmu_v3_smmus =3D kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus); + ret =3D smmu_take_pages(hyp_virt_to_phys(kvm_hyp_arm_smmu_v3_smmus), + smmu_arr_size); + if (ret) + return ret; + + for_each_smmu(smmu) { + ret =3D smmu_init_device(smmu); + if (ret) + goto out_reclaim_smmu; + } + + return 0; + +out_reclaim_smmu: + while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) + smmu_deinit_device(--smmu); + smmu_reclaim_pages(hyp_virt_to_phys(kvm_hyp_arm_smmu_v3_smmus), + smmu_arr_size); + return ret; } =20 static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CB99533A01E for ; Mon, 17 Nov 2025 18:48:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405327; cv=none; b=Af/vdOYZ7L6XpSfH6Up22vfK3+7Itewj70ZL9KnUSPdQRiSoHQvh3Xjj9Fi8BTxM7uzc0+IXf1HD7OesOeF1Ib3UytF7qbwYWcQ3Pr2IFdR//giioggYIcxxTX6IQRV7bUp7OMwUbUejZcTYiOuVgp6VzgpMZfsC+MaiF0Hk0sE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405327; c=relaxed/simple; bh=WWfCCBvddxM5OBSZYoCRyfY75yEJKzutFES+Qny/bnk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FTLLIG665bzABGKawP4/jek2fhK+RTKFFYpeHvjxiIBXnxO5rJN4Bi6AZFWuBmj6kHY+LQVVGgIQ8GoxBkyIG60iXuspCa4NeU+WcPqgkzLBWYPolWLmeQ36P4OHXJ0WaZfrpZF8ONfviKMmqhtf/kwX1zZrehqsUJkdlm5mE/k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dYJ51Frk; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dYJ51Frk" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b73720562ceso257216466b.0 for ; Mon, 17 Nov 2025 10:48:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405323; x=1764010123; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NDHGRVWxr8tiMMJHgFkthzaCbmxDX2W8q2KLEFUMQ0M=; b=dYJ51FrkmdABQ2T0a9MIrK3CAjYwugwFz1YSMgEs+lt+Rlbgfry2WTdbtoqR0uatcr SZt+VDACKmiflSj+HMHXNjeIDyyDdfUpGG6PJZYYEfrEpIZwN2nmA+gqAnj63iIhnmv+ bwenB4f5FJHPAAt4WySlALUVMZHOesCD4LV0YDn8SZWKKHW7bEPZiNF8d7aME5urMNJ/ K94IP+cwnfFonunVzWdG5ciwHfunI2SFr9YX6tFPCl9QYwpsuVDrLdYxywbDZwsB0RhD gNTxlwhf0SmVJfqd/MMyxPKmu2WPD5NEbxyeAc4BX2R3FUGmenHs2CqP1u2NaRQxTrFj Pdbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405323; x=1764010123; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NDHGRVWxr8tiMMJHgFkthzaCbmxDX2W8q2KLEFUMQ0M=; b=mXdHJs395Nhdvdl74TmpLpQoSI6dbOz4yE3ZphJI9WCko3JFv+bEufk880noW33LWO RfXnVRgabu/f1p5toY87uCXmrWMjOEmMO+Khb+q5IIA0PCz9alE1MM89MF7j/nKO8Sb7 jQzArsXKlVVxBwwEFgzQ3LYWw+CRTHYInF0xrhS8kMXfQcv2+Z1wZofHpx2wXe+5097j i2ZbxxtGKIySX3fSOrUqMO2E92uQLkyM2bAVVvzvnIXYkQBech5hEnwlUqkW8tJ/hKFH B1D84XtGhsXx65A+N/Qq7/u3r/v41eokLDUudqRl2eqKYMsCb3QtF3AFFMYsF/YpVjng Z5xw== X-Forwarded-Encrypted: i=1; AJvYcCUM6nimkS+WYuJKJgkU6JLG+htR3uIJKgVAk5+XKFY/JBv0WW0QcDjraYFjkHrR+ktYuuECYGwNy8nZi6Y=@vger.kernel.org X-Gm-Message-State: AOJu0YwkgDq12zAN630rFXdhPfwXDmiVAUzJxugzfwmUa7PIVT7dKvhJ dimMFfKZjDQBraXsk28T27mh+hLf1chx5bREIVR/Bc7VZDehZvVkb3oq6QFJXXFw6XN0E0P8ogG +KZEmY9kQpsVSnw== X-Google-Smtp-Source: AGHT+IG0TrsHTOFqlfcoz8fH0quPptO87OaYvkyHV4a6ntExQ0SYHVEwf58bnjdfGxOgO/BBPbDmIMTbefbdcw== X-Received: from ejclk13.prod.google.com ([2002:a17:907:178d:b0:b73:69d2:27b7]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:97cc:b0:b73:7b5d:e76a with SMTP id a640c23a62f3a-b737b5dea0bmr1018806066b.27.1763405322837; Mon, 17 Nov 2025 10:48:42 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:04 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-18-smostafa@google.com> Subject: [PATCH v5 17/27] iommu/arm-smmu-v3-kvm: Probe SMMU HW From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Probe SMMU features from the IDR register space, most of the logic is common with the kernel. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 57 ++++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 8 +++ 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 309194ceebe7..1d552efdc4ae 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -49,6 +49,7 @@ struct arm_vsmmu; #define IDR0_S2P (1 << 0) =20 #define ARM_SMMU_IDR1 0x4 +#define IDR1_ECMDQ (1 << 31) #define IDR1_TABLES_PRESET (1 << 30) #define IDR1_QUEUES_PRESET (1 << 29) #define IDR1_REL (1 << 28) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index b56feae81dda..e45b4e50b1e4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -10,6 +10,7 @@ #include =20 #include "arm_smmu_v3.h" +#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -45,9 +46,56 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_de= vice *smmu) } } =20 +/* + * Mini-probe and validation for the hypervisor. + */ +static int smmu_probe(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 reg; + + /* IDR0 */ + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); + smmu->features =3D smmu_idr0_features(reg); + + /* + * Some MMU600 and MMU700 have errata that prevent them from using nestin= g, + * not sure how can we identify those, so it's recommended not to enable = this + * drivers on such systems. + * And preventing any of those will be too restrictive + */ + if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1) || + !(smmu->features & ARM_SMMU_FEAT_TRANS_S2)) + return -ENXIO; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); + if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL | IDR1_ECMD= Q)) + return -EINVAL; + + smmu->sid_bits =3D FIELD_GET(IDR1_SIDSIZE, reg); + /* Follows the kernel logic */ + if (smmu->sid_bits <=3D STRTAB_SPLIT) + smmu->features &=3D ~ARM_SMMU_FEAT_2_LVL_STRTAB; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); + smmu->features |=3D smmu_idr3_features(reg); + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); + + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) + smmu->pgsize_bitmap |=3D 1ULL << 42; + else if (!smmu->oas) + smmu->oas =3D 48; + + smmu->ias =3D 64; + smmu->ias =3D min(smmu->ias, smmu->oas); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { - int i; + int i, ret; size_t nr_pages; =20 if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) @@ -64,8 +112,13 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) WARN_ON(__pkvm_host_donate_hyp_mmio(pfn)); } smmu->base =3D hyp_phys_to_virt(smmu->mmio_addr); - + ret =3D smmu_probe(smmu); + if (ret) + goto out_ret; return 0; +out_ret: + smmu_deinit_device(smmu); + return ret; } =20 static int smmu_init(void) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 744ee2b7f0b4..3550fa695539 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -12,12 +12,20 @@ * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers + * @ias IPA size + * @oas PA size + * @pgsize_bitmap Supported page sizes + * @sid_bits Max number of SID bits supported */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; size_t mmio_size; void __iomem *base; u32 features; + unsigned long ias; + unsigned long oas; + unsigned long pgsize_bitmap; + unsigned int sid_bits; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 06B5F33DEF0 for ; Mon, 17 Nov 2025 18:48:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405328; cv=none; b=tYT05RwkjdT8Uja2yBEbpMHCPBEazZqEW+jo3AFCKW3lB76CvpwsSMZh4DCYUHx4FJmHnsvGWhm2ei9GLOVkVWrUT6gnMFihguVtox4b8KmuR/z2a3+JfAF/Apyo9Poyn3lCmmtZTERnthgh3KPtemIfM1boz51CkPSmKV6KANY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405328; c=relaxed/simple; bh=qwh5bWqrvxWib+fkY1bKqGNWl7AdpDhlBehg1j6Y/Qc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H9Ks4bjJusVRNxAXfPRNiVBlvn/tSAFP+Yo/CvvLvMzqolRbVj1PikKxrOqgQGQR4UbnrkgURVetxl+jrS8iTv5FA9en4Eqg8wuaev4U/AtJSxhW5aFeqzME7VAzQLCitrMLA0kSnrpjwCPhGy7qF/KK/jlyStr9hDcUHWfiPw4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=llgRW1aO; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="llgRW1aO" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-429cd1d0d98so3273762f8f.3 for ; Mon, 17 Nov 2025 10:48:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405324; x=1764010124; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p3xOR8x6QzbVlWlZrKYLBDo4QOwlksJXf7F4qOhAA+w=; b=llgRW1aOVvJeDD6PAWIpfayYGHe10CF2OELFeSOcCRgG+U5taxmkmR69ouQK36Soom V+z3KgN1r0+BGK57L2S5YnkihAd7VEEkK3mZ1SvasPtn5Ag+28mDdtTZ2/FyGNNI0fux GCOp/vlGaZ4FT8lplCBcPwAOMZ11NX/o39wdfNTMKDvdD1lbuT9oWvciTKVnCuQ4gNCf ieNr6Ah5nWFRaGgJRNuni2Y6VuFeFxoZBVBSrCWnh/Z7UJ/QVDTYyPQ9u+e1VENmjGw3 2faGvIxfKqSGmJmIcXIdKKOJMwH6c8BjPf1ptHPpW67EeaPQxBXnQIvkXj2Iv5YCF5qk Wo0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405324; x=1764010124; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p3xOR8x6QzbVlWlZrKYLBDo4QOwlksJXf7F4qOhAA+w=; b=Ar8ZiCFx54Ttv1iXutYaw7VanLOd73Cf6rcC6VOJqIsxYLivnjRIg7a0pZ3BoLlGLP ClXjQZcOWzX34Agg4Cdi+ueun8JCpWmEYNKzQf7fpHb6FY39OL6QDO9ZVj5I4jk5eLVo yoCRNyKprK8xbiuaIJ7Gv12oVMu1Qxa3x1zfJUKP4iF7SPuuS4glZM0CKdrf3v2F80/e gGI3znqc8GXopZTTSIv28NioCWEtUryqqq0EwxKX2U/hFQrwe1Ea6CzYd6Q+fdVLGpj5 2I9t/G6bnSlonUFnss008IKMb/MZh/Njr3aevZ9DelIpVAIl1Qdv608mRWuoiYXC//zi SYxA== X-Forwarded-Encrypted: i=1; AJvYcCWiPHmu8VNlAeyWab2vOZA4j7Hg8LmKlv7VrmbQqWnLR65cq3VF5Gm5x5vJePeAXYTEMNtmLDPb+BjI90Y=@vger.kernel.org X-Gm-Message-State: AOJu0YwYAOAFuladNGcdTzos3imu96UmwfBA28MxSDT3HrfPKONRVdv1 KzMUY5RU6dPaHnNm3KHL/WZk/RCeyM77+iZLOPs1yZcw5zagvFQkNDIR6B5YMclllUe9qqdxkh1 cn0Dd/VTPNCFoRw== X-Google-Smtp-Source: AGHT+IHCWbQlr8o940tPX5ysnGCiRUYmATJoia86/TFJSTwquzFFs3l4nOHlPBVKTiqXidY7DwnVkD6YGitCIA== X-Received: from wrio1.prod.google.com ([2002:a5d:6481:0:b0:429:bf18:9f1]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:230d:b0:42b:2e94:5a8f with SMTP id ffacd0b85a97d-42b5939dc13mr12310824f8f.52.1763405324058; Mon, 17 Nov 2025 10:48:44 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:05 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-19-smostafa@google.com> Subject: [PATCH v5 18/27] iommu/arm-smmu-v3-kvm: Add MMIO emulation From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At the moment most registers are just passthrough, then in the next patches CMDQ/STE emulation will be added which inserts logic to some register access. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 126 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 2 files changed, 136 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index e45b4e50b1e4..f0dae94daf89 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -8,6 +8,7 @@ =20 #include #include +#include =20 #include "arm_smmu_v3.h" #include "../arm-smmu-v3.h" @@ -115,6 +116,7 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) ret =3D smmu_probe(smmu); if (ret) goto out_ret; + hyp_spin_lock_init(&smmu->lock); return 0; out_ret: smmu_deinit_device(smmu); @@ -140,6 +142,8 @@ static int smmu_init(void) goto out_reclaim_smmu; } =20 + BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); + return 0; =20 out_reclaim_smmu: @@ -150,6 +154,127 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, + struct user_pt_regs *regs, + u64 esr, u32 off) +{ + bool is_write =3D esr & ESR_ELx_WNR; + unsigned int len =3D BIT((esr & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); + int rd =3D (esr & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; + const u64 read_write =3D -1ULL; + const u64 no_access =3D 0; + u64 mask =3D no_access; + const u64 read_only =3D is_write ? no_access : read_write; + u64 val =3D regs->regs[rd]; + + switch (off) { + case ARM_SMMU_IDR0: + /* Clear stage-2 support, hide MSI to avoid write back to cmdq */ + mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); + WARN_ON(len !=3D sizeof(u32)); + break; + /* Passthrough the register access for bisectiblity, handled later */ + case ARM_SMMU_CMDQ_BASE: + case ARM_SMMU_CMDQ_PROD: + case ARM_SMMU_CMDQ_CONS: + case ARM_SMMU_STRTAB_BASE: + case ARM_SMMU_STRTAB_BASE_CFG: + case ARM_SMMU_GBPA: + mask =3D read_write; + break; + case ARM_SMMU_CR0: + mask =3D read_write; + WARN_ON(len !=3D sizeof(u32)); + break; + case ARM_SMMU_CR1: { + /* Based on Linux implementation */ + u64 cr2_template =3D FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB); + /* Don't mess with shareability/cacheability. */ + if (is_write) + WARN_ON(val !=3D cr2_template); + mask =3D read_write; + WARN_ON(len !=3D sizeof(u32)); + break; + } + /* + * These should be safe, just enforce RO or RW and size according to arch= itecture. + * There are some other registers that are not used by Linux as IDR2, IDR4 + * that won't be allowed. + */ + case ARM_SMMU_EVTQ_PROD + SZ_64K: + case ARM_SMMU_EVTQ_CONS + SZ_64K: + case ARM_SMMU_EVTQ_IRQ_CFG1: + case ARM_SMMU_EVTQ_IRQ_CFG2: + case ARM_SMMU_PRIQ_PROD + SZ_64K: + case ARM_SMMU_PRIQ_CONS + SZ_64K: + case ARM_SMMU_PRIQ_IRQ_CFG1: + case ARM_SMMU_PRIQ_IRQ_CFG2: + case ARM_SMMU_GERRORN: + case ARM_SMMU_GERROR_IRQ_CFG1: + case ARM_SMMU_GERROR_IRQ_CFG2: + case ARM_SMMU_IRQ_CTRLACK: + case ARM_SMMU_IRQ_CTRL: + case ARM_SMMU_CR0ACK: + case ARM_SMMU_CR2: + /* These are 32 bit registers. */ + WARN_ON(len !=3D sizeof(u32)); + fallthrough; + case ARM_SMMU_EVTQ_BASE: + case ARM_SMMU_EVTQ_IRQ_CFG0: + case ARM_SMMU_PRIQ_BASE: + case ARM_SMMU_PRIQ_IRQ_CFG0: + case ARM_SMMU_GERROR_IRQ_CFG0: + mask =3D read_write; + break; + case ARM_SMMU_IIDR: + case ARM_SMMU_IDR5: + case ARM_SMMU_IDR3: + case ARM_SMMU_IDR1: + case ARM_SMMU_GERROR: + WARN_ON(len !=3D sizeof(u32)); + mask =3D read_only; + }; + + if (WARN_ON(!mask)) + goto out_ret; + + if (is_write) { + if (len =3D=3D sizeof(u64)) + writeq_relaxed(regs->regs[rd] & mask, smmu->base + off); + else + writel_relaxed(regs->regs[rd] & mask, smmu->base + off); + } else { + if (len =3D=3D sizeof(u64)) + regs->regs[rd] =3D readq_relaxed(smmu->base + off) & mask; + else + regs->regs[rd] =3D readl_relaxed(smmu->base + off) & mask; + } + +out_ret: + return true; +} + +static bool smmu_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 addr) +{ + struct hyp_arm_smmu_v3_device *smmu; + bool ret; + + for_each_smmu(smmu) { + if (addr < smmu->mmio_addr || addr >=3D smmu->mmio_addr + smmu->mmio_siz= e) + continue; + hyp_spin_lock(&smmu->lock); + ret =3D smmu_dabt_device(smmu, regs, esr, addr - smmu->mmio_addr); + hyp_spin_unlock(&smmu->lock); + return ret; + } + return false; +} + static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) { } @@ -158,4 +283,5 @@ static void smmu_host_stage2_idmap(phys_addr_t start, p= hys_addr_t end, int prot) struct kvm_iommu_ops smmu_ops =3D { .init =3D smmu_init, .host_stage2_idmap =3D smmu_host_stage2_idmap, + .dabt_handler =3D smmu_dabt_handler, }; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 3550fa695539..dfeaed728982 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -4,6 +4,10 @@ =20 #include =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +#include +#endif + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -16,6 +20,7 @@ * @oas PA size * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported + * @lock Lock to protect SMMU */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -26,6 +31,11 @@ struct hyp_arm_smmu_v3_device { unsigned long oas; unsigned long pgsize_bitmap; unsigned int sid_bits; +#ifdef __KVM_NVHE_HYPERVISOR__ + hyp_spinlock_t lock; +#else + u32 lock; +#endif }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F34B533EB0E for ; Mon, 17 Nov 2025 18:48:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405331; cv=none; b=jWb6zYZUTIJipbl26M7QeFa7RCYQs0KfPUKyF54ExWBa5HuswL24keE0tBiDFtKHKYfjfHCUwiaN7HNCo1LDlt1glCg5Phgdo1cJ9sNrxHZiffDnxRBTsLPnijELAvOWCrChndTyjU5VyONOQfno6lRXAhQIzcsf93RN/IYepcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405331; c=relaxed/simple; bh=GwOPg6FlFCx/BY8EpD+/tKl7DeNEXmYEAlOuY72ewsg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mEv6mXoOuV7cwQxfLik8hKRtAuo9u+0xVDQ+cSLgNuuHGYnvg8ehIArDL1sN0RLljejpB33LUWAMMUSQlGAOzmWIx7sOw48hq7i5q+dpaz+sYnfE2+HDiaOT2Qe8H8pnMFgFTC6EbvQ4cxI6ASRpeLllsMv4ByDC57BcyFnPdtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hIDdJuAQ; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hIDdJuAQ" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-6409c803b1aso5830035a12.3 for ; Mon, 17 Nov 2025 10:48:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405325; x=1764010125; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=KaEfzU6yZVtxcuFF1g0xre8mMX/xz4F/cTBqkb/2nDs=; b=hIDdJuAQIW7ofLICmK9DVUsUSLrcomKK3+y+v1RrMy3GhaeMrKFZvGkiJGQHCJR5mL usbFvIGtNG5Kz7tZ7L1n/orpUKgiPx8fjmGUkD81XYq93R62+fJ9qT6M+d4m6Clijo5P 19IIeEwa/7Vnshy4UWpuPZ1uRsBM2mbs19kGJJLz1hDVIy8eVLiSojXcNoMaussSzHI9 j3e9RJF+D92yILHnwBNga3DtCatFRdZiyVEp3zK3BK5Kh40+63pyRd6FjQcnqJDg4P/8 qZzuS7egWRWhe90d5OvMZPUOoItVOwvUvos0yxQ0kGmNovTZlH1dkUjVEjKQYkZT/+vW kqTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405325; x=1764010125; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=KaEfzU6yZVtxcuFF1g0xre8mMX/xz4F/cTBqkb/2nDs=; b=hy57Eyq2gM6cyoFIohx6NqJtx0u/lAIHoVPsYJVsXSY9uWdScpDtq5HXL6s+b3oL3A st47dxphbngEulAcVLUNhxVIrN7dR4ypnOi+HwYTO105TCuLumKWYLKZku8CBavnLFLl gm4vfwNY0VQvdiQCD9st9ZcaCOj2Jf6Yk//ZCdr6RcV0sei60Bo8o+DQOsHbsKjs5jAt bB7eShx2wmMoMysN52pj9FXvLzikWcjG0kWHjPwszmODSQQ/odEkvbyRgpErsMSXFXmZ Wkoak/jFb0MD1y/6xrhCAp5fYGkpNQcjxjo0zSlsI+TUvSi9d5p3TQ56X012ZzbIVVEK gd+w== X-Forwarded-Encrypted: i=1; AJvYcCULZky17blSr2KK62vqvdQQcBUlRgSLTvSUZia5MwSwmlgMRebP0uMQJWE4OyrXqF86Ljl+m6G38+jBbXU=@vger.kernel.org X-Gm-Message-State: AOJu0YzFRvSN2KupTOiEVN7FY183nWjoPsmkuSZEE5txq6Ku8RKKko+M KPUzgh6elJb8H735jjJ0fOd2mkd4rmYzws6hfdSq9KGJVd0wGU6dRqYsi+LylRyOdufyP2531I5 wWrTsuttTmopHFA== X-Google-Smtp-Source: AGHT+IGYWSQvx/vSf5yAHHEgbkBONoL9bq/wnyu6GlN5XLTrHCKyyCf6BcTG/d+4znbhK0TqerfxI9YTZPVTXA== X-Received: from edh26.prod.google.com ([2002:a05:6402:505a:b0:641:8ac2:5031]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:3489:b0:641:9e1c:ba89 with SMTP id 4fb4d7f45d1cf-64350dee91dmr12404365a12.7.1763405324917; Mon, 17 Nov 2025 10:48:44 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:06 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-20-smostafa@google.com> Subject: [PATCH v5 19/27] iommu/arm-smmu-v3-kvm: Shadow the command queue From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At boot allocate a command queue per SMMU which is used as a shadow by the hypervisor. The command queue size is 64K which is more than enough, as the hypervisor would consume all the entries per a command queue prod write, which means it can handle up to 4096 at a time. Then, the host command queue needs to be pinned in a shared state, so it can't be donated to VMs, and avoid tricking the hypervisor into accessing them. This is done each time the command queue is enabled, and undone each time the command queue is disabled. The hypervisor won=E2=80=99t access the host command queue when it is disab= led from the host. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 16 +++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 103 +++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 8 ++ 3 files changed, 126 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index 1d72951b7b53..87376f615798 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -15,6 +15,8 @@ #include "arm-smmu-v3.h" #include "pkvm/arm_smmu_v3.h" =20 +#define SMMU_KVM_CMDQ_ORDER 4 + extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 static size_t kvm_arm_smmu_count; @@ -71,6 +73,7 @@ static struct platform_driver smmuv3_nesting_driver; static int smmuv3_nesting_probe(struct platform_device *pdev) { struct resource *res; + void *cmdq_base; struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; =20 res =3D platform_get_resource(pdev, IORESOURCE_MEM, 0); @@ -84,6 +87,19 @@ static int smmuv3_nesting_probe(struct platform_device *= pdev) if (of_dma_is_coherent(pdev->dev.of_node)) smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; =20 + /* + * Allocate the shadow command queue, it doesn't have to be the same + * size as the host. + * Only populate base_dma and llq.max_n_shift, the hypervisor will init + * the rest. + */ + cmdq_base =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_= CMDQ_ORDER); + if (!cmdq_base) + return -ENOMEM; + + smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); + smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; + kvm_arm_smmu_cur++; return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index f0dae94daf89..bcb3f99fdcd2 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -11,7 +11,6 @@ #include =20 #include "arm_smmu_v3.h" -#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -21,6 +20,13 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ (smmu)++) =20 +#define cmdq_size(cmdq) ((1 << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWORD= S * 8) + +static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_CMDQEN, smmu->cr0); +} + /* Transfer ownership of memory */ static int smmu_take_pages(u64 phys, size_t size) { @@ -34,6 +40,35 @@ static void smmu_reclaim_pages(u64 phys, size_t size) WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT)); } =20 +/* + * CMDQ, STE host copies are accessed by the hypervisor, we share them to + * - Prevent the host from passing protected VM memory. + * - Having them mapped in the hyp page table. + */ +static int smmu_share_pages(phys_addr_t addr, size_t size) +{ + int i; + size_t nr_pages =3D PAGE_ALIGN(size) >> PAGE_SHIFT; + + for (i =3D 0 ; i < nr_pages ; ++i) + WARN_ON(__pkvm_host_share_hyp((addr + i * PAGE_SIZE) >> PAGE_SHIFT)); + + return hyp_pin_shared_mem(hyp_phys_to_virt(addr), hyp_phys_to_virt(addr += size)); +} + +static int smmu_unshare_pages(phys_addr_t addr, size_t size) +{ + int i; + size_t nr_pages =3D PAGE_ALIGN(size) >> PAGE_SHIFT; + + hyp_unpin_shared_mem(hyp_phys_to_virt(addr), hyp_phys_to_virt(addr + size= )); + + for (i =3D 0 ; i < nr_pages ; ++i) + WARN_ON(__pkvm_host_unshare_hyp((addr + i * PAGE_SIZE) >> PAGE_SHIFT)); + + return 0; +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -94,6 +129,36 @@ static int smmu_probe(struct hyp_arm_smmu_v3_device *sm= mu) return 0; } =20 +/* + * The kernel part of the driver will allocate the shadow cmdq, + * and zero it. This function only donates it. + */ +static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_nr_pages =3D cmdq_size(&smmu->cmdq) >> PAGE_SHIFT; + int ret; + enum kvm_pgtable_prot prot =3D PAGE_HYP; + + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + prot |=3D KVM_PGTABLE_PROT_NORMAL_NC; + + ret =3D ___pkvm_host_donate_hyp(smmu->cmdq.base_dma >> PAGE_SHIFT, + cmdq_nr_pages, prot); + if (ret) + return ret; + + smmu->cmdq.base =3D hyp_phys_to_virt(smmu->cmdq.base_dma); + smmu->cmdq.prod_reg =3D smmu->base + ARM_SMMU_CMDQ_PROD; + smmu->cmdq.cons_reg =3D smmu->base + ARM_SMMU_CMDQ_CONS; + smmu->cmdq.q_base =3D smmu->cmdq.base_dma | + FIELD_PREP(Q_BASE_LOG2SIZE, smmu->cmdq.llq.max_n_shift); + smmu->cmdq.ent_dwords =3D CMDQ_ENT_DWORDS; + writel_relaxed(0, smmu->cmdq.prod_reg); + writel_relaxed(0, smmu->cmdq.cons_reg); + writeq_relaxed(smmu->cmdq.q_base, smmu->base + ARM_SMMU_CMDQ_BASE); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { int i, ret; @@ -117,7 +182,13 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; hyp_spin_lock_init(&smmu->lock); + + ret =3D smmu_init_cmdq(smmu); + if (ret) + goto out_ret; + return 0; + out_ret: smmu_deinit_device(smmu); return ret; @@ -154,6 +225,20 @@ static int smmu_init(void) return ret; } =20 +static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + smmu->cmdq_host.llq.max_n_shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2S= IZE; + smmu->cmdq_host.base_dma =3D smmu->cmdq_host.q_base & Q_BASE_ADDR_MASK; + WARN_ON(smmu_share_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + +static void smmu_emulate_cmdq_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(smmu_unshare_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -175,6 +260,13 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, break; /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: + if (is_write) { + /* Not allowed by the architecture */ + WARN_ON(is_cmdq_enabled(smmu)); + smmu->cmdq_host.q_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_CMDQ_PROD: case ARM_SMMU_CMDQ_CONS: case ARM_SMMU_STRTAB_BASE: @@ -183,6 +275,15 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, mask =3D read_write; break; case ARM_SMMU_CR0: + if (is_write) { + bool last_cmdq_en =3D is_cmdq_enabled(smmu); + + smmu->cr0 =3D val; + if (!last_cmdq_en && is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_enable(smmu); + else if (last_cmdq_en && !is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_disable(smmu); + } mask =3D read_write; WARN_ON(len !=3D sizeof(u32)); break; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index dfeaed728982..2fb4c0cab47c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -8,6 +8,8 @@ #include #endif =20 +#include "../arm-smmu-v3.h" + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -21,6 +23,9 @@ * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported * @lock Lock to protect SMMU + * @cmdq CMDQ as observed by HW + * @cmdq_host Host view of the CMDQ, only q_base and llq used. + * @cr0 Last value of CR0 */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -36,6 +41,9 @@ struct hyp_arm_smmu_v3_device { #else u32 lock; #endif + struct arm_smmu_queue cmdq; + struct arm_smmu_queue cmdq_host; + u32 cr0; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C87F733EAE7 for ; Mon, 17 Nov 2025 18:48:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405330; cv=none; b=g7Dj8RoZXNUSZ10iUJxxhM2MS4rs1Prqrffl1pmnDyAKuBvcs3YGHgukxh4b+bfX6UuqbWrsRTiqWgve0sOcDn2qvKf8OHcDsa45uiMQlWMAXm32heGCh/o6f18JeRw2E5lr2rivGF7qCYg/zkOdOpbvGTMQnIX/OBU5/HROgk0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405330; c=relaxed/simple; bh=e3EIrVNRd9t3QHVp59OzFLDa7p4DxlL7+5kVQLVXEbQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=EkdNCdIOiVLusmllas6GaGmhWueTyYIUdqx963ILktsU8A3tPWIkSgwBsD0RHHNfd/ZELz3RBI/DlzmdR9fsBIm28xEUCjd9PhK0iK1VX0yoXuOWh++V5o2dQgiayGkljQMqxzY09zEWrTrgQQzW1UQy/50bBtakjK5YRh8s8E4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LhHEhl/+; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LhHEhl/+" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-b72a95dc686so365682166b.3 for ; Mon, 17 Nov 2025 10:48:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405326; x=1764010126; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gthqCq5o6yeE89UPMEuJgIME9dtn3tyMj4iiO/+WnDw=; b=LhHEhl/+6OzewqMw5gqzJEXBtfyBbq9P5YSVv9tM3aBbwM3JS8+w6IHqqXA5ys2/1J SlIT7gVTSdR0AKb9Ped+NHo8DgxsCvISCjthFtgKyvyjJL0w1ha0nAvS/gJJg6ENLrfp ydHsJSBUI4GBR3l1ih0rnQFeQ69BXoMQyoRREL+bFs1EFJJeNQxX79yVDH1Q26m+ZN52 IE8O5GZr+GEb750EE+6vk2xxL8g/CJc4CvAaHkCVBi7gCa6bIjLzykVjuDIhmBBKmj8I ZcNgmj7Y+AP/PS/isNuS4/VICGN/vNYicJHH0PSFjrUXcegfE5gxZN3dzF/8tuBxB/jW aw5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405326; x=1764010126; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gthqCq5o6yeE89UPMEuJgIME9dtn3tyMj4iiO/+WnDw=; b=uewCx/KFsOfrq92rqvTpIYV6sRcb7wKzk74wA5OqflhAVIWeps7x+z1H3pkfm8r89N tzjTf3zrgtb2Yk6AT6A1J/0n4QNp86l+JyjhS0z2wQnuCsIWU+3UjjQananaW2aCe+qZ AwrTIlDU8tSb4ykTpfwSRmNsASdbt5iZnVQQJ3DPjYyr/pd5YHcT7iHqWB3WZzIA90Xg xIQ4HmniooXHe7O2fFu1araR1t735HNYlG5xhB9A7JH5P4IdBPW/1u7lOeb81A87SnmP VThgumX4WacryQ/34Mp4UZB+T7ZwHVQNg7mCeWmY+k2LKOLkW3sF+jpBG5guM1EUVDWa CTJQ== X-Forwarded-Encrypted: i=1; AJvYcCUdoYqx5qd7TNcosEkWUfe2wCrIHfQNs4UtkOMXrO4elyaK9XY1UfZPEAFiDEGziDMF7kqALKxHsSShT3A=@vger.kernel.org X-Gm-Message-State: AOJu0YzPOuKDyr8xU2L6N03svfGa4Y9ZSHfIeg08UTnQ4bIfHFHd1+Wh o1IO9isoxc9M3RZk7gWIxCVu3EKSsXpa1a0S7qJO9gUg8AUlJpcCQ/di0fgo1SMi3KaPj6t3P9C XZlsRqes1KbP+5A== X-Google-Smtp-Source: AGHT+IE5KRl2mOZHi8gtVKjABXvz8EzQH63tM2/gsVZz7+LwCgoA43ubS78h95GU7o7UGQNIlpHqH+cFHQPaLQ== X-Received: from ejcqo4.prod.google.com ([2002:a17:907:8744:b0:b73:7130:3892]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:849:b0:b72:598:2f32 with SMTP id a640c23a62f3a-b73678ed1d0mr1476075666b.42.1763405325882; Mon, 17 Nov 2025 10:48:45 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:07 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-21-smostafa@google.com> Subject: [PATCH v5 20/27] iommu/arm-smmu-v3-kvm: Add CMDQ functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add functions to access the command queue, there are 2 main usage: - Hypervisor's own commands, as TLB invalidation, would use functions as smmu_send_cmd(), which creates and sends a command. - Add host commands to the shadow command queue, after being filtered, these will be added with smmu_add_cmd_raw. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 102 ++++++++++++++++++ 2 files changed, 110 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 1d552efdc4ae..d909014baad3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1015,19 +1015,21 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_dev= ice *smmu, int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent); =20 /* Queue functions shared between kernel and hyp. */ -static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +static inline u32 queue_space(struct arm_smmu_ll_queue *q) { - u32 space, prod, cons; + u32 prod, cons; =20 prod =3D Q_IDX(q, q->prod); cons =3D Q_IDX(q, q->cons); =20 if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; + return (1 << q->max_n_shift) - (prod - cons); + return cons - prod; +} =20 - return space >=3D n; +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + return queue_space(q) >=3D n; } =20 static inline bool queue_full(struct arm_smmu_ll_queue *q) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index bcb3f99fdcd2..a970b43e6a7e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -22,6 +22,26 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 #define cmdq_size(cmdq) ((1 << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWORD= S * 8) =20 +/* + * Wait until @cond is true. + * Return 0 on success, or -ETIMEDOUT + */ +#define smmu_wait(use_wfe, _cond) \ +({ \ + int __ret =3D 0; \ + u64 delay =3D pkvm_time_get() + ARM_SMMU_POLL_TIMEOUT_US; \ + \ + while (!(_cond)) { \ + if (use_wfe) \ + wfe(); \ + if (pkvm_time_get() >=3D delay) { \ + __ret =3D -ETIMEDOUT; \ + break; \ + } \ + } \ + __ret; \ +}) + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -69,6 +89,88 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t s= ize) return 0; } =20 +__maybe_unused +static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_has_space(llq, n); +} + +static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_full(llq); +} + +static bool smmu_cmdq_empty(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_empty(llq); +} + +static void smmu_add_cmd_raw(struct hyp_arm_smmu_v3_device *smmu, + u64 *cmd) +{ + struct arm_smmu_queue *q =3D &smmu->cmdq; + struct arm_smmu_ll_queue *llq =3D &q->llq; + + queue_write(Q_ENT(q, llq->prod), cmd, CMDQ_ENT_DWORDS); + llq->prod =3D queue_inc_prod_n(llq, 1); +} + +static int smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *ent) +{ + int ret; + u64 cmd[CMDQ_ENT_DWORDS]; + + ret =3D smmu_wait(smmu->features & ARM_SMMU_FEAT_SEV, + !smmu_cmdq_full(&smmu->cmdq)); + if (ret) + return ret; + + ret =3D arm_smmu_cmdq_build_cmd(cmd, ent); + if (ret) + return ret; + + smmu_add_cmd_raw(smmu, cmd); + writel_relaxed(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + return 0; +} + +static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_CMD_SYNC, + }; + + ret =3D smmu_add_cmd(smmu, &cmd); + if (ret) + return ret; + + return smmu_wait(smmu->features & ARM_SMMU_FEAT_SEV, + smmu_cmdq_empty(&smmu->cmdq)); +} + +__maybe_unused +static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd) +{ + int ret =3D smmu_add_cmd(smmu, cmd); + + if (ret) + return ret; + + return smmu_sync_cmd(smmu); +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8C1733F36C for ; Mon, 17 Nov 2025 18:48:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405331; cv=none; b=fMvigZesr1eayqAKqRfMxApEIIBl/p2YBsxgsBPfQ3UDKD9p/WUPU3MCJabjioa/A3sJ4rPLoFyKCUVHLOaXaEL8n+8ooOXrQgSbKEZNBeDYbXkHuWIa0VqfkTbgClqgOIzp6EzBFgVmMuW0wTcTsISRUSpz1w9MUx5HruY26zE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405331; c=relaxed/simple; bh=mulobR960QYfKaphz0wYlGrjMp0Ox8WLfgC9zyBpQpU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Vu6jpW61uW25fPjjL9u2SQzGf1jfV1Jyp8cijAbYuEwt2rRqtTE81S2MXgkILPdS9pNjHwD2TYLH9+3eaGHcGtjHqyRCAI2F/y68yk+6/2px7bW/dPVuaty4szD4ORkJw+MqBsnUbsC2uSyzvkxRdhVLvaTivPWJ+6Pp7Z9ZyGQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Lw71SIB/; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Lw71SIB/" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4779b3749a8so3691835e9.1 for ; Mon, 17 Nov 2025 10:48:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405327; x=1764010127; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=M5WVE3Hl8yIdruGNt8XIVgNufhaprbpT9QA5T5hpVt8=; b=Lw71SIB/CFromm/YZZTpxLcqYlJ9Q/ZhyPfTskzRzCQqlfLDta9EiRnW6XZGkIlY3j p/MQO05eoFNnz3Yev0LWPtYq98bOld2iKFAyvjkWqP060Crt6/XntHy87z6d/ARBbf69 ghYkDXAvxuZDZVl1N0nPDJmzm09FH/X55J76AiqNSWCKesjumEzWcgLBlc16yisnyxAX ZqtigVCEpKOwONDWf6yZEVsRIhvdN9D90uJXDS8GQCldEyICOBtPGMBCNQHsUwUYaxIy wlSwHjco8Gk3mZTYjv5EUm8OKOFRAwoa436yu5PU6ZBegsmFMyYouCjTWWr2536O7Haq DjxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405327; x=1764010127; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=M5WVE3Hl8yIdruGNt8XIVgNufhaprbpT9QA5T5hpVt8=; b=jOwo4tgQ+FPybaGI1DahnjMzR75T/+JXF9CblslZhqODLh2BA+ZhTDDWxnewjRd0he sCaWM0ad5/hfDCkOP9RmyAl9y3UYukNRHPkxEhxly3pni1qpgTLxdHsERqh6HoYyTrBg vuQI/BFpYAadoErB3CdOGBqAFC+TyUTZQ9ubv7o2ujsZKiLmUyURmW/yoc9VROALrtRD s8B6uRKp/CE4PNG0VcGvqqztNBYv/DvedXGeVx7FKPyJCv8mYxvMfvLhMKNslv8LoIVS lxoc9RA5OR+qfS7a7MKXwMfuS0p5mTTl/8HcnjyOrOwJpivd5ZNuANGhSC8KF2wyYUI3 phhQ== X-Forwarded-Encrypted: i=1; AJvYcCVDTSBfiK1eL1c9YzM17nnIq7cTEDHkO3ytXAtfaVX29g22Z+FqBvjlrAg8QmHqhfoeX92KjYM6HdELD2I=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7pTRLIg7022F2m01G8L4cEMojbijpe+5LM7ethDn/0DZi8oRc fLljZUW6CGGfuEM+I0+K/fHvA51+/oQIfcni4RFAayJ5PJzFgXcHEGq/DJdQWzu2AUri0ligWZy jOi9GVcU7ncIyJg== X-Google-Smtp-Source: AGHT+IHvvN75ZRrmEK6mbR78xTnUzHtBzNs3L5jIM2mnz88x+TZ3qPg9z7gWLJIqDbGhhpJDNO5GQC2qpVQ4ZQ== X-Received: from wmgg23.prod.google.com ([2002:a05:600d:17:b0:477:7aa2:99cb]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4452:b0:477:8b2e:aa8f with SMTP id 5b1f17b1804b1-4778fe6a42fmr114864735e9.15.1763405326933; Mon, 17 Nov 2025 10:48:46 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:08 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-22-smostafa@google.com> Subject: [PATCH v5 21/27] iommu/arm-smmu-v3-kvm: Emulate CMDQ for host From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don=E2=80=99t allow access to the command queue from the host: - ARM_SMMU_CMDQ_BASE: Only allowed to be written when CMDQ is disabled, we use it to keep track of the host command queue base. Reads return the saved value. - ARM_SMMU_CMDQ_PROD: Writes trigger command queue emulation which sanitise and filters the whole range. Reads returns the host copy. - ARM_SMMU_CMDQ_CONS: Writes move the sw copy of the cons, but the host can= =E2=80=99t skip commands once submitted. Reads return the emulated value and the err= or bits in the actual cons. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 119 +++++++++++++++++- 1 file changed, 115 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index a970b43e6a7e..746ffc4b0a70 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -60,6 +60,16 @@ static void smmu_reclaim_pages(u64 phys, size_t size) WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT)); } =20 +static void smmu_copy_from_host(struct hyp_arm_smmu_v3_device *smmu, + void *dst_hyp_va, void *src_hyp_va, + size_t size) +{ + /* Clean and inval DC as the kernel uses NC mapping. */ + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + kvm_flush_dcache_to_poc(src_hyp_va, size); + memcpy(dst_hyp_va, src_hyp_va, size); +} + /* * CMDQ, STE host copies are accessed by the hypervisor, we share them to * - Prevent the host from passing protected VM memory. @@ -89,7 +99,6 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t si= ze) return 0; } =20 -__maybe_unused static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -327,6 +336,88 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_filter_command(struct hyp_arm_smmu_v3_device *smmu, u64 *= command) +{ + u64 type =3D FIELD_GET(CMDQ_0_OP, command[0]); + + switch (type) { + case CMDQ_OP_CFGI_STE: + /* TBD: SHADOW_STE*/ + break; + case CMDQ_OP_CFGI_ALL: + { + /* + * Linux doesn't use range STE invalidation, and only use this + * for CFGI_ALL, which is done on reset and not on an new STE + * being used. + * Although, this is not architectural we rely on the current Linux + * implementation. + */ + WARN_ON((FIELD_GET(CMDQ_CFGI_1_RANGE, command[1]) !=3D 31)); + break; + } + case CMDQ_OP_TLBI_NH_ASID: + case CMDQ_OP_TLBI_NH_VA: + case 0x13: /* CMD_TLBI_NH_VAA: Not used by Linux */ + { + /* Only allow VMID =3D 0*/ + if (FIELD_GET(CMDQ_TLBI_0_VMID, command[0]) =3D=3D 0) + break; + break; + } + case 0x10: /* CMD_TLBI_NH_ALL: Not used by Linux */ + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_EL2_VA: + case CMDQ_OP_TLBI_EL2_ASID: + case CMDQ_OP_TLBI_S12_VMALL: + case 0x23: /* CMD_TLBI_EL2_VAA: Not used by Linux */ + return WARN_ON(true); + case CMDQ_OP_CMD_SYNC: + if (FIELD_GET(CMDQ_SYNC_0_CS, command[0]) =3D=3D CMDQ_SYNC_0_CS_IRQ) { + /* Allow it, but let the host timeout, as this should never happen. */ + command[0] &=3D ~CMDQ_SYNC_0_CS; + command[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + command[1] &=3D ~CMDQ_SYNC_1_MSIADDR_MASK; + } + break; + } + + return false; +} + +static void smmu_emulate_cmdq_insert(struct hyp_arm_smmu_v3_device *smmu) +{ + u64 *host_cmdq =3D hyp_phys_to_virt(smmu->cmdq_host.q_base & Q_BASE_ADDR_= MASK); + int idx; + u64 cmd[CMDQ_ENT_DWORDS]; + bool skip; + u32 space; + bool use_wfe =3D smmu->features & ARM_SMMU_FEAT_SEV; + + if (!is_cmdq_enabled(smmu)) + return; + + space =3D (1 << (smmu->cmdq_host.llq.max_n_shift)) - queue_space(&smmu->c= mdq_host.llq); + /* Wait for the command queue to have some space. */ + WARN_ON(smmu_wait(use_wfe, smmu_cmdq_has_space(&smmu->cmdq, space))); + + while (space--) { + idx =3D Q_IDX(&smmu->cmdq_host.llq, smmu->cmdq_host.llq.cons); + queue_inc_cons(&smmu->cmdq_host.llq); + + smmu_copy_from_host(smmu, cmd, &host_cmdq[idx * CMDQ_ENT_DWORDS], + CMDQ_ENT_DWORDS << 3); + skip =3D smmu_filter_command(smmu, cmd); + if (skip) + continue; + smmu_add_cmd_raw(smmu, cmd); + } + + writel_relaxed(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + + WARN_ON(smmu_wait(use_wfe, smmu_cmdq_empty(&smmu->cmdq))); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { smmu->cmdq_host.llq.max_n_shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2S= IZE; @@ -360,17 +451,37 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); WARN_ON(len !=3D sizeof(u32)); break; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: if (is_write) { /* Not allowed by the architecture */ WARN_ON(is_cmdq_enabled(smmu)); smmu->cmdq_host.q_base =3D val; + } else { + regs->regs[rd] =3D smmu->cmdq_host.q_base; } - mask =3D read_write; - break; + goto out_ret; case ARM_SMMU_CMDQ_PROD: + if (is_write) { + smmu->cmdq_host.llq.prod =3D val; + smmu_emulate_cmdq_insert(smmu); + } else { + regs->regs[rd] =3D smmu->cmdq_host.llq.prod; + } + goto out_ret; case ARM_SMMU_CMDQ_CONS: + if (is_write) { + /* Not allowed by the architecture */ + WARN_ON(is_cmdq_enabled(smmu)); + smmu->cmdq_host.llq.cons =3D val; + } else { + /* Propagate errors back to the host.*/ + u32 cons =3D readl_relaxed(smmu->base + ARM_SMMU_CMDQ_CONS); + u32 err =3D CMDQ_CONS_ERR & cons; + + regs->regs[rd] =3D smmu->cmdq_host.llq.cons | err; + } + goto out_ret; + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: case ARM_SMMU_STRTAB_BASE_CFG: case ARM_SMMU_GBPA: --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3031E33F8C6 for ; Mon, 17 Nov 2025 18:48:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405334; cv=none; b=T8cgm2v+7PgiIOIaqEZt+kb3z0MpY0VoChxzi47DLWJY/BWHtnld2P9urapHSbbTUwok/537VmXqwCS7merZVW0hZs8YUgEbZa3djvWE82uoWsud6tQV223uyR6lgL0WxPpBVbYv8eKTi59HNgjgxh3YhTQIwxEJ0aUpPSlfe8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405334; c=relaxed/simple; bh=McTMZ04nUPWuX1adTdf753MDynTmdT9+Yvl+v890ioQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UoU4tiUlexbGITC6M/wDnTL7+Cj23rG9/tErNpfh8qN4OGxPr6RoY5yyrYNrfnKo1hpPY2y2Q1jnHqw0Vya+OSyqvxwmqVnOKfYcESLw5h/r5bblHrT1HQeVc4irZ/5bL+LB9s8P11CIabR7RXjN+/w83yKxClC4wSiFancCE/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zwOGiTpl; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zwOGiTpl" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-b739b3d8997so134096766b.2 for ; Mon, 17 Nov 2025 10:48:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405328; x=1764010128; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rb8/GtH372WJ0qxy4TkHlGTwt2qL/Zq2TNdTmFCsamI=; b=zwOGiTplhwSqb9bkt8VOlok2YmdpaXAXjJS+lNUjfXowhwBDte/BNgdlzUGuxvhvMR cbdiWl8oVS6e6O7rXyHONDVaY7+8XCmPSsoiAs8o/+DnTxOmT4RcSkTFr8aAfQmLOMGX 2P0rDwh5WZZdIzSvS1m5RpcM0qyMuHM0mgPLaewK9Un0ey/LiTKAusLdjSuulSBOdrwc yiknMh19gEiO35jdoWV6A9cdQIoe8J0o1r6nwpQOkNpjUE5a1XASspcRc+EXIIobMmVn sP2EPOj4khm6FSHIJmNMPkjkxCu3A3BUcOt/utIR2oafEnq+0E5l8idath6zXQhmtTua pnDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405328; x=1764010128; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rb8/GtH372WJ0qxy4TkHlGTwt2qL/Zq2TNdTmFCsamI=; b=bRXCok7J58H5iVRgfvn+hqVRZDywcc13eTGWxmtfnB1J1KtmNrSTbN8zxEXSI4hF58 jTFeoIqAyrEYot00KzJQiOP8rgdyoZk9Dgr9n1FnnQuLtJELAw6l6tvUEBJ0lqmwxv5f xn57YjIIe/4A1tKSyfLVm91/Tn67lWbdEkQuDf51MOZsq2z/w+XLd0PmxFTOVObkIQkS o+hwUx/2pN7ZiWvmCmfNbdlT6oLDdkc3nb4jxw9BtNwjy0XIxT2ac2mLW7ObSUBYSwsI 3S/YBLqZ4OmeqUU9IyuOiPr0yyDAgHx8nOBlp8PmtnXsecUT49V+utXeoCkfskKVto7x 4Qsw== X-Forwarded-Encrypted: i=1; AJvYcCWuBQ9ehy98MLudb96cydeUHmL8+pWHdPYQMiY32LS7zqq+Q5ANMY07bwGgvliZeJYD+kFaJ4Ouy8wF6WY=@vger.kernel.org X-Gm-Message-State: AOJu0YzKBUBIFNh041sDpOtpTKRCLngXyNjq1phr5PmW+OMdGtd78T3W DN0/EQ4rdok8UsoiwQkPVSi5KiGcE3+WsocdX/kmawJUCI6CVKKSSmniBbLp/kZM74LSjrvTFWk KvSs7B9/U2wqoXA== X-Google-Smtp-Source: AGHT+IGu0kjXqesw4Vlk33i3V4Q6Afh7tCka5IjsceWQYSdkeKorvnxtzP7ZJLMV5a7eguSrQpC56lnDAuMk9Q== X-Received: from ejcuy7.prod.google.com ([2002:a17:907:d107:b0:b73:4e7b:9fe3]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:25c9:b0:b72:8e31:4354 with SMTP id a640c23a62f3a-b736782c35cmr1540477366b.25.1763405327824; Mon, 17 Nov 2025 10:48:47 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:09 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-23-smostafa@google.com> Subject: [PATCH v5 22/27] iommu/arm-smmu-v3-kvm: Shadow stream table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch allocates the shadow stream table per SMMU. We choose the size of that table to be 1MB which is the max size used by host in the case of 2 levels. In this patch all the host writes are still paththrough for bisectibility, that is changed next where CFGI commands will be trapped and used to update the shadow copy hypervisor that will be used by HW. Similar to the command queue, the host stream table is shared/unshared each time the SMMU is enabled/disabled. Handling of L2 tables is also done in the next patch when the shadowing is added. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 11 +- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 114 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 3 files changed, 134 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index 87376f615798..82626e052a2f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -16,6 +16,8 @@ #include "pkvm/arm_smmu_v3.h" =20 #define SMMU_KVM_CMDQ_ORDER 4 +#define SMMU_KVM_STRTAB_ORDER (get_order(STRTAB_MAX_L1_ENTRIES * \ + sizeof(struct arm_smmu_strtab_l1))) =20 extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 @@ -73,7 +75,7 @@ static struct platform_driver smmuv3_nesting_driver; static int smmuv3_nesting_probe(struct platform_device *pdev) { struct resource *res; - void *cmdq_base; + void *cmdq_base, *strtab; struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; =20 res =3D platform_get_resource(pdev, IORESOURCE_MEM, 0); @@ -100,6 +102,13 @@ static int smmuv3_nesting_probe(struct platform_device= *pdev) smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; =20 + strtab =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_STR= TAB_ORDER); + if (!strtab) + return -ENOMEM; + + smmu->strtab_dma =3D virt_to_phys(strtab); + smmu->strtab_size =3D PAGE_SIZE << SMMU_KVM_STRTAB_ORDER; + kvm_arm_smmu_cur++; return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 746ffc4b0a70..9e515a130711 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -15,6 +15,14 @@ size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +/* strtab accessors */ +#define strtab_log2size(smmu) (FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, (smmu)-= >host_ste_cfg)) +#define strtab_size(smmu) ((1 << strtab_log2size(smmu)) * STRTAB_STE_DWORD= S * 8) +#define strtab_host_base(smmu) ((smmu)->host_ste_base & STRTAB_BASE_ADDR_M= ASK) +#define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) +#define strtab_l1_size(smmu) ((1 << (strtab_log2size(smmu) - strtab_split(= smmu))) * \ + (sizeof(struct arm_smmu_strtab_l1))) + #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ @@ -47,6 +55,11 @@ static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_devic= e *smmu) return FIELD_GET(CR0_CMDQEN, smmu->cr0); } =20 +static bool is_smmu_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_SMMUEN, smmu->cr0); +} + /* Transfer ownership of memory */ static int smmu_take_pages(u64 phys, size_t size) { @@ -270,6 +283,49 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + u32 reg; + enum kvm_pgtable_prot prot =3D PAGE_HYP; + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + prot |=3D KVM_PGTABLE_PROT_NORMAL_NC; + + ret =3D ___pkvm_host_donate_hyp(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT, prot); + if (ret) + return ret; + + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + unsigned int last_sid_idx =3D + arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1); + + cfg->l2.l1tab =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->l2.l1_dma =3D smmu->strtab_dma; + cfg->l2.num_l1_ents =3D min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES); + + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_2LVL) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, + ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) | + FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT); + } else { + cfg->linear.table =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->linear.ste_dma =3D smmu->strtab_dma; + cfg->linear.num_ents =3D 1UL << smmu->sid_bits; + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_LINEAR) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits); + } + + writeq_relaxed((smmu->strtab_dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_R= A, + smmu->base + ARM_SMMU_STRTAB_BASE); + writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { int i, ret; @@ -298,6 +354,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_init_strtab(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -418,6 +478,41 @@ static void smmu_emulate_cmdq_insert(struct hyp_arm_sm= mu_v3_device *smmu) WARN_ON(smmu_wait(use_wfe, smmu_cmdq_empty(&smmu->cmdq))); } =20 +static void smmu_update_ste_shadow(struct hyp_arm_smmu_v3_device *smmu, bo= ol enabled) +{ + size_t strtab_size; + u32 fmt =3D FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->host_ste_cfg); + + /* Linux doesn't change the fmt nor size of the strtab in the run time. */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + strtab_size =3D strtab_l1_size(smmu); + WARN_ON(fmt !=3D STRTAB_BASE_CFG_FMT_2LVL); + WARN_ON((strtab_split(smmu) !=3D STRTAB_SPLIT)); + } else { + strtab_size =3D strtab_size(smmu); + WARN_ON(fmt !=3D STRTAB_BASE_CFG_FMT_LINEAR); + WARN_ON(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->host_ste_cfg) > + smmu->sid_bits); + } + + if (enabled) + WARN_ON(smmu_share_pages(strtab_host_base(smmu), strtab_size)); + else + WARN_ON(smmu_unshare_pages(strtab_host_base(smmu), strtab_size)); +} + +static void smmu_emulate_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + /* Enabling SMMU without CMDQ, means TLB invalidation won't work. */ + WARN_ON(!is_cmdq_enabled(smmu)); + smmu_update_ste_shadow(smmu, true); +} + +static void smmu_emulate_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + smmu_update_ste_shadow(smmu, false); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { smmu->cmdq_host.llq.max_n_shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2S= IZE; @@ -483,19 +578,38 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, goto out_ret; /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + WARN_ON(is_smmu_enabled(smmu)); + smmu->host_ste_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_STRTAB_BASE_CFG: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + WARN_ON(is_smmu_enabled(smmu)); + smmu->host_ste_cfg =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_GBPA: mask =3D read_write; break; case ARM_SMMU_CR0: if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); + bool last_smmu_en =3D is_smmu_enabled(smmu); =20 smmu->cr0 =3D val; if (!last_cmdq_en && is_cmdq_enabled(smmu)) smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + if (!last_smmu_en && is_smmu_enabled(smmu)) + smmu_emulate_enable(smmu); + else if (last_smmu_en && !is_smmu_enabled(smmu)) + smmu_emulate_disable(smmu); } mask =3D read_write; WARN_ON(len !=3D sizeof(u32)); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 2fb4c0cab47c..8efa9273b194 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -15,6 +15,8 @@ * @mmio_addr base address of the SMMU registers * @mmio_size size of the registers resource * @features Features of SMMUv3, subset of the main driver + * @strtab_dma Phys address of stream table + * @strtab_size Stream table size * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers @@ -26,6 +28,9 @@ * @cmdq CMDQ as observed by HW * @cmdq_host Host view of the CMDQ, only q_base and llq used. * @cr0 Last value of CR0 + * @host_ste_cfg Host stream table config + * @host_ste_base Host stream table base + * @strtab_cfg Stream table as seen by HW */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -44,6 +49,11 @@ struct hyp_arm_smmu_v3_device { struct arm_smmu_queue cmdq; struct arm_smmu_queue cmdq_host; u32 cr0; + dma_addr_t strtab_dma; + size_t strtab_size; + u64 host_ste_cfg; + u64 host_ste_base; + struct arm_smmu_strtab_cfg strtab_cfg; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEA8432B996 for ; Mon, 17 Nov 2025 18:48:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405333; cv=none; b=gX7U/rolD/ehMWVsZWxzkszdD8UQt7jWV1k8+VhDGm7EBDAUqsrPtsMAslCPDpKtHwP+o/63IkaZ+xcNLf4t9z2H6eql5+EHR8EYOFnwi0UvbWaCvqTBgEaUuo1HKffU8lQqnOTfhLDghqJSEFnYYKYCcy/qZPUHGS0aVIAG39c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405333; c=relaxed/simple; bh=OZ4uQ0yDoACRmNdHRBQeRNzIXtXWO2EVRqqs9TQAI90=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AgJAfbi4bRzqWlhxd0NEWqSNNy3Kf0Hh2dyMSXuACE4mytHiSisDPy4LEZ924iUnffAbQ32cMS6xvz8MbAA0BJ2JtK0Zad5t89bq8acIjijOk9FIy7Yun2D6zthawfIGQAbj+pOMYO2+hqkHcd+gdRS9+MPeeAkXUKqtlP5guPY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ig3fR3Aa; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ig3fR3Aa" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-42b2ffbba05so2450778f8f.0 for ; Mon, 17 Nov 2025 10:48:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405329; x=1764010129; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qs8S7LI/sbSmCau3G8bqaka/RHnKzp9VmZFW3fnA4G8=; b=ig3fR3AamJErpHQUa7Dl1rc86bEc446rOC75zHg7WPa9hzVs6xs88tWOJ+pAsM9Mhz YJZKyNkVaQUWVSveQx8NNl2sm3B/ExHDjNoHMVxWif86KLptnEkP8u4inIFGnVAa9kfm DjqyE0ybH2F8SxfwC//tyRL3/gfI5SvS1OnakAgSCItf4iKVcQgjYhHlXVO+ASZ58zZU 0g1tzUAFRxClYCVK1NNZSFeayU6iQLeHPtM3ddAlhn/qS2rwv9OiQoM7G6DlcJWeKv7Z SkUpLuBBj7BiJ3quENwMK7RL6tqB5ikvQDfIKU6Nx+nTa41wf2CIn0XUuH7mQfY4pRxy m8Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405329; x=1764010129; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qs8S7LI/sbSmCau3G8bqaka/RHnKzp9VmZFW3fnA4G8=; b=eZXNR2UgyluFBWs0o8UhaNCPLBwwZabwBQFzJCWYg9N52/yfC2ijN5C4cic19v+akW Lq54DOun9k19bMFy/ZG1Pi3nwT69CD8d25cH7Fgv61fah/Wu+SID6G3bnmYz/ugHONUz WtS2YkuGzF1xRWtCfCdJijsMArcElLH530yBb+o4AHUUkmyfrA3CwYPMR8J1E0jqyUI/ NPgAlcJUuLx7AWV7XnxlqDjKVVXOXr5TMWAORIi1c0AHDnk2El7d7gWakyaP8MyakmhO Rc+63jKeZDOQo6L0DjGX+bJ7XkilIz2axmzAWI4MAIMANa6Dq+Ig7hrQMjJXQdlZazPt pKpg== X-Forwarded-Encrypted: i=1; AJvYcCWOGJpWwIrecc441zXOMGoqpWIZe1oaMQYLGX1GQlFhpcQHJ/67qSHxgX3dL3PxmhRlFAvwoWG1Jkw1D+s=@vger.kernel.org X-Gm-Message-State: AOJu0Yy+BE9ge5ILHgNZGIRxVKztukaNmTaKmydRowc83avlwUQwYRZ0 5IFu7AyJ7vu5odHeljdDh6TC0xKC2wWQuYEyrnjPmaHoqCplPaZVnIveliJF6FHudcOf9HxEu9U kxLzzdhRQ9IljYQ== X-Google-Smtp-Source: AGHT+IEaVN1RJqQh1usVWtaU2+GmutZ2HuXAEHxOLo10Rm858igk+GLC7xcyxVVFvzJzEDXSgQJrFWO0SiMrKA== X-Received: from wrbfq12.prod.google.com ([2002:a05:6000:2a0c:b0:42b:5250:ee1b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:3106:b0:42b:3b4c:f411 with SMTP id ffacd0b85a97d-42b5938a9e3mr13616468f8f.36.1763405328637; Mon, 17 Nov 2025 10:48:48 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:10 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-24-smostafa@google.com> Subject: [PATCH v5 23/27] iommu/arm-smmu-v3-kvm: Shadow STEs From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch adds STE emulation, this is done when the host sends the CFGI_STE command. In this patch we copy the STE as is to the shadow owned by the hypervisor, in the next patch, stage-2 page table will be attached. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 92 +++++++++++++++++-- 1 file changed, 86 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 9e515a130711..fbe1e13fc15d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -22,6 +22,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; #define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) #define strtab_l1_size(smmu) ((1 << (strtab_log2size(smmu) - strtab_split(= smmu))) * \ (sizeof(struct arm_smmu_strtab_l1))) +#define strtab_hyp_base(smmu) ((smmu)->features & ARM_SMMU_FEAT_2_LVL_STRT= AB ? \ + (u64 *)(smmu)->strtab_cfg.l2.l1tab :\ + (u64 *)(smmu)->strtab_cfg.linear.table) =20 #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ @@ -283,6 +286,80 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +/* Get an STE for a stream table base. */ +static struct arm_smmu_ste *smmu_get_ste_ptr(struct hyp_arm_smmu_v3_device= *smmu, + u32 sid, u64 *strtab) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + struct arm_smmu_ste *table =3D (struct arm_smmu_ste *)strtab; + + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)strtab; + u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); + struct arm_smmu_strtab_l2 *l2ptr; + + if (WARN_ON(l1_idx >=3D cfg->l2.num_l1_ents) || + !(l1tab[l1_idx].l2ptr & STRTAB_L1_DESC_SPAN)) + return NULL; + + l2ptr =3D hyp_phys_to_virt(l1tab[l1_idx].l2ptr & STRTAB_L1_DESC_L2PTR_MA= SK); + /* Two-level walk */ + return &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; + } + if (WARN_ON(sid >=3D cfg->linear.num_ents)) + return NULL; + return &table[sid]; +} + +static int smmu_shadow_l2_strtab(struct hyp_arm_smmu_v3_device *smmu, u32 = sid) +{ + u32 idx =3D arm_smmu_strtab_l1_idx(sid); + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + struct arm_smmu_strtab_l1 *l1_desc =3D &smmu->strtab_cfg.l2.l1tab[idx]; + u64 l1_desc_host; + struct arm_smmu_strtab_l2 *l2table; + + l2table =3D kvm_iommu_donate_pages(get_order(sizeof(*l2table))); + if (!l2table) + return -ENOMEM; + + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + kvm_flush_dcache_to_poc(&host_ste_base[idx], sizeof(*l1_desc)); + l1_desc_host =3D host_ste_base[idx]; + + arm_smmu_write_strtab_l1_desc(l1_desc, hyp_virt_to_phys(l2table)); + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + kvm_flush_dcache_to_poc(l1_desc, sizeof(*l1_desc)); + + smmu_share_pages(l1_desc_host & STRTAB_L1_DESC_L2PTR_MASK, sizeof(*l2tabl= e)); + return 0; +} + +static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid= , bool leaf) +{ + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + u64 *hyp_ste_base =3D strtab_hyp_base(smmu); + struct arm_smmu_ste *host_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, host_st= e_base); + struct arm_smmu_ste *hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_= base); + + /* + * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this + * is a 2 level table and reshadow of l2. + * Also Linux never clears l1 ptr, that needs to free the old shadow. + */ + if (WARN_ON(!leaf || !host_ste_ptr)) + return; + + /* If host is valid and hyp is not, means a new L1 installed. */ + if (!hyp_ste_ptr) { + WARN_ON(smmu_shadow_l2_strtab(smmu, sid)); + hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_base); + } + + smmu_copy_from_host(smmu, hyp_ste_ptr->data, host_ste_ptr->data, + STRTAB_STE_DWORDS << 3); +} + static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) { int ret; @@ -402,8 +479,13 @@ static bool smmu_filter_command(struct hyp_arm_smmu_v3= _device *smmu, u64 *comman =20 switch (type) { case CMDQ_OP_CFGI_STE: - /* TBD: SHADOW_STE*/ + { + u32 sid =3D FIELD_GET(CMDQ_CFGI_0_SID, command[0]); + u32 leaf =3D FIELD_GET(CMDQ_CFGI_1_LEAF, command[1]); + + smmu_reshadow_ste(smmu, sid, leaf); break; + } case CMDQ_OP_CFGI_ALL: { /* @@ -576,23 +658,21 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, regs->regs[rd] =3D smmu->cmdq_host.llq.cons | err; } goto out_ret; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ WARN_ON(is_smmu_enabled(smmu)); smmu->host_ste_base =3D val; } - mask =3D read_write; - break; + goto out_ret; case ARM_SMMU_STRTAB_BASE_CFG: if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ WARN_ON(is_smmu_enabled(smmu)); smmu->host_ste_cfg =3D val; } - mask =3D read_write; - break; + goto out_ret; + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: mask =3D read_write; break; --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4298B32E12B for ; Mon, 17 Nov 2025 18:48:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405335; cv=none; b=sJRpEOAe72IihvkZuIoMtt1JPE9JoMPeBs3TbFT6zFAjWLqUV3dKw1nXTP8ImJsaGijFC0IFZTGNG0v18vNcSyr7lh1vHXHN9MKm9vw52Wnof/CQgTEN1wCRwY6kr/pPRtfMItFX6QPjMgnOHZUUhwiM7VqSsj998frZIZRdhj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405335; c=relaxed/simple; bh=hrcmq+7BKS+7RyD2MLzXmh7hMuGNu7NQQpndrNeBtvg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pHN4QwLxSTy6JFJyt5ilRqQ6Xhgs/mTNYjSjZ4elD9Bm1XRQMWQghagR+UJ+ZMENpdfWOIZ9Nj2js4gRKC0i+oLnSxh3Hv1Fzwva7BWVJ59gKFriYtiAxuYNAmjEsP7AmKXXQzTKiQk/rZBTg90t5SxUCVoj6iJ2Oiug16xAh1I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sQHkDu10; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sQHkDu10" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-42b56125e77so2137904f8f.3 for ; Mon, 17 Nov 2025 10:48:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405330; x=1764010130; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=7Pr/AhyDI18gHyudCTe0PajcLVV2Ma7cRe8uP5zP/9Y=; b=sQHkDu10vS1g1VjR8CPENx16vQeAHGCFnZlVFC3KeS6ckf/KC+UDDqygCS7XQ0Bn3m 8rZpHPWvJYTRA2zdylFlHxevUwiaGGpW+tLUH0THHCAHnYrEQQ1Tp/wXams5Y2ytXII9 ++Icl+TQKXAZEqNsgKHsLKwbqGxz0Bn7xPJ9Ig6PBV5brcb1Pf5aHuiZCwEmFecZBRCk EZIIBYqfCQtxgUsKHZlv95I93edo+UmwHf4P1G1cLKGwA+t8ZIikzM3L5r415brpRM0+ X97mRjSaPzNCUkJyK8aMqNZPiEDl9nB0RACLw2W0ipM99+foyIVLle1Fy/oqHUzyCpJ2 VNYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405330; x=1764010130; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=7Pr/AhyDI18gHyudCTe0PajcLVV2Ma7cRe8uP5zP/9Y=; b=gN1oP3pctr/1tjPEolmNdn7E0wuYdnj1YQupeFJgJRV7M9gJ46WxnJO8Lh9GZpkbub QFkhEOUAF1xybLjqOzMNBqAi/FZeuwYmQz95+e5dXTzeD33hYI1Ad/HVXxNGJpQDzIpX 8eorPTcZmeW9WcprvwQJxskO4DMIYq8oGcKW1xNhIByLt6CSa1PMEEEL0Zjbu2qqUCb9 BEUPIZJGovNQa+YC558yszwCTKYFDCnX6Z4w503OnI3VEfQ42r38n8kkWJjBYCr5UZBB eV3UDMxBy7lO4itmFnKQPUmBCqlOVz57vOHqq5mJj6gMN9+RlrDJWh2lIVR5KA0nJQ8X r5eQ== X-Forwarded-Encrypted: i=1; AJvYcCVT95KaS+p+A4dmT9uTie3DPyB55cUDl0oFXrPvmZrusFLUfkabvrjDrIH8baUnEvVGymKKJnUeccO41oo=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/I0gCKFB0k7bG3fVt4HHaoQcuG/WngrwFhmxF0eFuRYRTmrNI ebyGGdvbTJZ+A0nwkCkEc9n3Kx4GWW2FzHsAxZj/0hywGZoZs3jSLXyruJt0w00AU4jJNcVmbRS dz8Kb2p92ohAQZg== X-Google-Smtp-Source: AGHT+IEp1GJrifJg25A7HBsu430z3IudVIoooo816bFplXawdroOYHu2KCfVFavBNdBdQ4MlmLiDzZJPVbDY1w== X-Received: from wrbee4.prod.google.com ([2002:a05:6000:2104:b0:42b:3308:563]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1849:b0:429:d725:4125 with SMTP id ffacd0b85a97d-42b5959e88dmr13794450f8f.54.1763405329830; Mon, 17 Nov 2025 10:48:49 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:11 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-25-smostafa@google.com> Subject: [PATCH v5 24/27] iommu/arm-smmu-v3-kvm: Emulate GBPA From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The last bit of emulation is GBPA. it must be always set to ABORT, as when the SMMU is disabled it=E2=80=99s not allowed for the host to bypass the SMMU. That =E2=80=98s is done by setting the GBPA to ABORT at init time, and host writes are always ignored and host reads always return ABORT. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index fbe1e13fc15d..0a2ce6c06f4f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -115,6 +115,14 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t= size) return 0; } =20 +static int smmu_abort_gbpa(struct hyp_arm_smmu_v3_device *smmu) +{ + writel_relaxed(GBPA_UPDATE | GBPA_ABORT, smmu->base + ARM_SMMU_GBPA); + /* Wait till UPDATE is cleared. */ + return smmu_wait(false, + readl_relaxed(smmu->base + ARM_SMMU_GBPA) =3D=3D GBPA_ABORT); +} + static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -435,6 +443,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_abort_gbpa(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -672,10 +684,13 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, smmu->host_ste_cfg =3D val; } goto out_ret; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: - mask =3D read_write; - break; + /* Ignore write, always read to abort. */ + if (!is_write) + regs->regs[rd] =3D GBPA_ABORT; + + WARN_ON(len !=3D sizeof(u32)); + goto out_ret; case ARM_SMMU_CR0: if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B70232D193F for ; Mon, 17 Nov 2025 18:48:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405336; cv=none; b=TRfh/89Wfq3MJq3wxxZQkOq6sRw5Dk3ZKnpG/EpgfO0stAhbAJiC91mOpOs5g3YhA4Zg3VwGrhwTf8fCjeSJq1DG8kbfQ/5cCRcw154YJjsWRIOV4X/GkVDdxHv3t2Vvp4NegsWP97/BbIlvI6mieMCaPlBZIf2vRR5foAQrwVw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405336; c=relaxed/simple; bh=OIllo30ZkV3wRlt33j6CE/EE04FNRPaHVxJuJzubPrY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=iUlV5pWSs6ZgVGRWzPfjSpxZUXGyEed+3raL+iM9LA4u4W/hEMy3xs6kkgsik6DAkOirw7w/ZpbTVUoRRawNeYF7tMXh9k3E7PjDNWGdMdwQyG7yd0C03w/4fG3X7Z3VZPW/O7zQrOIghRATkMLuSZad2opErR2km+8SFd/r7TQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pri/duwk; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pri/duwk" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-477771366cbso28638155e9.0 for ; Mon, 17 Nov 2025 10:48:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405331; x=1764010131; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2e8AJ/3qpxCOMqh4v7dKF/r5F0keF6BFA14sDMtgseA=; b=pri/duwk4FW/n6KIJs+Nc/j0EPeU2XwR3tN4xCMChwEtGBqrrCKyJ0UI4+4lon8RiH Irr16D5gu0dpFdLkIhboKPLyaeaOgU3Xf56rjUSHOGb29rp3aj/0dKqoRTm5j0c5AKyM EerpzzZ+prrz74Z0DuEgSIvhsqaFG0cyY0aznE1eEOyx9VmUtDTMr67X7mOe7Qqwe7Oy A5fmJqPmFaVQbKBh6mz7vpXUJ2nomsy0HXnO0B/YZBFqDjVKOUY/f2OwndL7b6d69YSB 85Ca0QRL0xolqbQwCm+brqKX5L9X9kvCoIXasrZoxOkHdNnnspyTiM09JTYB4uMIRxI+ l68w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405331; x=1764010131; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2e8AJ/3qpxCOMqh4v7dKF/r5F0keF6BFA14sDMtgseA=; b=Z/c5DTdY3YmbYb6+QWz+074vEt7RHFW+Y7MvfiyKVXL8HgdbPfmw4PKhjYcJFXu78a qhpRfyNNGgRO7rbEsPVpTAxPZtJ9MTEWFIlf5H7aHKTqOZEjtlzD0746iddO2BalWJsh TjFfN9MJZvohtSu5vNyg2cwC81mO3DXSfEU34HsBzsmjRvI64RvAjxyHPZdzvPoMTUSv a6y3dG15FqwhM01o490UEdO+Y43Yb9ZDLcFkzyh+bA0fFl4UqwTZKkJgvHnCK81fV3cz lPW4LFnBtr/UnQZeyFXoM834ODiU5+cawa8CxgTCoMSqpmXJdRImI1zwKWXZHzYJzxt/ fLAQ== X-Forwarded-Encrypted: i=1; AJvYcCWv7dk0hoY3h9FoPvUvDTtbijFKtIn64e4P/Sv/YvMTVJzcn0rWfNJ0VybbfW+8r6o6qvMjQP/9udBQK8E=@vger.kernel.org X-Gm-Message-State: AOJu0YzSSozc4V52HLSAKBObrjGXJ4mUzu1VYlU1C1UuqhYVpa6YHwj/ +5gUktOkRswMWqGEmLNy4FcfUru0aVIFBWZDLDou+pEa3n4uX5NPiK9Gs7DN8p7vKKOpJc4r3/T ey7mtsZAnBy0AKA== X-Google-Smtp-Source: AGHT+IFYxGMytoDZZlTDHtwywr4/Pi+/xR6UqlI0u6DlCyyeGT+QjWCV1nogGfL6thKzA2YoXGlkE2OMke9X7g== X-Received: from wrqa7.prod.google.com ([2002:adf:f7c7:0:b0:42b:33e5:eba1]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:240d:b0:3ea:80ec:8552 with SMTP id ffacd0b85a97d-42b59387fbfmr14696008f8f.57.1763405331039; Mon, 17 Nov 2025 10:48:51 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:12 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-26-smostafa@google.com> Subject: [PATCH v5 25/27] iommu/arm-smmu-v3-kvm: Support io-pgtable From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add hooks needed to support io-pgtable-arm, mostly about memory allocation. Also add a function to allocate s2 64 bit page table. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 4 +- .../arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c | 68 +++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 2 +- drivers/iommu/io-pgtable-arm.h | 11 +++ 4 files changed, 83 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 197685817546..99c2ce941146 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -35,7 +35,9 @@ hyp-obj-y +=3D $(lib-objs) HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 =20 hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ - $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o \ + $(HYP_SMMU_V3_DRV_PATH)/pkvm/io-pgtable-arm-hyp.o \ + $(HYP_SMMU_V3_DRV_PATH)/../../io-pgtable-arm.o =20 ## ## Build rules for compiling nVHE hyp code diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c b/driv= ers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c new file mode 100644 index 000000000000..fc2006dc0b82 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 Arm Ltd. + */ +#include + +#include +#include "../../../io-pgtable-arm.h" + +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct io_pgtable *iop; + + if (fmt !=3D ARM_64_LPAE_S2) + return NULL; + + iop =3D arm_64_lpae_alloc_pgtable_s2(cfg, cookie); + if (!iop) + return NULL; + + iop->fmt =3D fmt; + iop->cookie =3D cookie; + iop->cfg =3D *cfg; + + return &iop->ops; +} + +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, void *cookie) +{ + void *addr; + + addr =3D kvm_iommu_donate_pages(get_order(size)); + + if (addr && !cfg->coherent_walk) + kvm_flush_dcache_to_poc(addr, size); + + return addr; +} + +void __arm_lpae_free_pages(void *addr, size_t size, struct io_pgtable_cfg = *cfg, + void *cookie) +{ + if (!cfg->coherent_walk) + kvm_flush_dcache_to_poc(addr, size); + + kvm_iommu_reclaim_pages(addr); +} + +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg) +{ + if (!cfg->coherent_walk) + kvm_flush_dcache_to_poc(ptep, sizeof(*ptep) * num_entries); +} + +/* At the moment this is only used once, so rounding up to a page is not r= eally a problem. */ +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp) +{ + return kvm_iommu_donate_pages(get_order(size)); +} + +void __arm_lpae_free_data(void *p) +{ + return kvm_iommu_reclaim_pages(p); +} diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 377c15bc8350..dbb1a58a5ce6 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -980,7 +980,7 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg= , void *cookie) return NULL; } =20 -static struct io_pgtable * +struct io_pgtable * arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie) { u64 sl; diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index 62d127dae1c2..742a6ed9ae3c 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -41,9 +41,20 @@ void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, void *cookie); void *__arm_lpae_alloc_data(size_t size, gfp_t gfp); void __arm_lpae_free_data(void *p); +struct io_pgtable * +arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie); #ifndef __KVM_NVHE_HYPERVISOR__ #define __arm_lpae_virt_to_phys __pa #define __arm_lpae_phys_to_virt __va +#else +#include +#define __arm_lpae_virt_to_phys hyp_virt_to_phys +#define __arm_lpae_phys_to_virt hyp_phys_to_virt +#undef WARN_ONCE +#define WARN_ONCE(condition, format...) WARN_ON(1) +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie); #endif /* !__KVM_NVHE_HYPERVISOR__ */ =20 #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA7EB33F8C7 for ; Mon, 17 Nov 2025 18:48:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405338; cv=none; b=m7KWI0s37VlAvz/cixgnp2D9tBWoVD0s1DpSGA1Z70/qQKD/UfIXdGl1XgAOvN+thCgZSxEOvizeJ81x8AKPSI+G6+rTPuNqsjnjtk3RalhklqDcHMXpJoSasFEWP5+ph8hZvGyTo+vR20OyXsa8zc9vEYdKlGli5JPGbb8k+pI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405338; c=relaxed/simple; bh=yCEYZh2AGSeMppy6W4pQzDAJmahJxTs69c+HgyMSqIs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jWypQOhTqihRCEtC2CtxpgIDrKQtpcWdiFankGvsT+lPuPD64TA5ymTyadJPA+NwihuYvAgopzs7nhGNPCntYkLZ34JiUKTlExXtgOI0XXJvb1bFqQLqpiAuRk6FIB7yw87YVwS6p+j+gzPXXcfyd5piL4FwlC2E3LmqF9IT77Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cHG3/u5o; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cHG3/u5o" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-640bae7d83aso4714280a12.2 for ; Mon, 17 Nov 2025 10:48:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405332; x=1764010132; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=31DOoq9hxxOVuOmnZkKoNNzWo0+aj1CFvIYXfA+a1tI=; b=cHG3/u5osAW0z3CdEe5ZxYkA4Nzv6xizBSR8pFt2uKTTMAUU/qvZCQ9xoN480rJiEl pgmYid8oHua50ncNgP+MHLe/wDqn5+dec5EpTK8Z05/V1/gB5rpnUInZHt/4q4B66lKh KUOaIf4ixgBJPxoRiSyioPj94Wk0gytkst61tvBW71FBX0SPv2ZW1plXv6l8JLfJegrN AyMJ6rvV2XgiwEMNvW4RmDHN3xOGY4UBpOsc8X5eQRn1lggGmlo2u/lTdZVhrduLyJdF PWG5sK3HLAt5UL8Sunz5MqLcXzipSCB91YuGISqxMTPEpW/U1EjqhxtjfNMY7Oq0x+DO FEoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405332; x=1764010132; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=31DOoq9hxxOVuOmnZkKoNNzWo0+aj1CFvIYXfA+a1tI=; b=JT+yTpDXietfr86OjrHUqsOc138eLbH6n5D1U9pg6rRhbjrnTbqd74B87sg069dcpx e7yvLjQoQOqr/LSI1chTIeeXvS1+VMNIVhPDvapdjOQQOoO97XRnU3p+VszpH/rV3+Qy M3i33VZuLkv1GqZPAmQ+PGKe3Ojkkbw9syleGrbHVVPDLw6zi8THn+7CkVLhcCm60cXv RvZ6u+0SJBN7//aIwB9s6PlkLBd38O2j6jYvqpi94PJyAys2tyDLa0035WhTpXoT/GKB Me3WxZKhufjBNe89JCdzK+y2UV8pYcoNHb5Pa8WehnNxEwKIdqzzF5LEE8LSmFYrtOcA P4EA== X-Forwarded-Encrypted: i=1; AJvYcCXuvpOGWir9p8whtPpR7jjB3FAJP72jcoDOMEkWb/1e5gDiUZaQUNYj/yLwuIZTe0LIbSEIxjX43Srr2gE=@vger.kernel.org X-Gm-Message-State: AOJu0YxPwQYOx+XIi95O35dspFBzk9C7oeVkpXpuFrWLFEYtXdIr4WWP bPvwViz77NX6Z+cuXR0xQPX2xcOdHWOXj/tT8LSYJS5iDTvXlgCK7eAdzTrrDgpivxPBPXtRJs9 Kdp8B9gnv4DIJig== X-Google-Smtp-Source: AGHT+IHvu5Pm7T3ulD8svZ9/ewILArBTgbOobiFYwmLzMkQ3ahrrnLUgoZ1utD9YIAk9Ft4XjCqCnrq1nS8Y+Q== X-Received: from edy24.prod.google.com ([2002:a05:6402:5358:b0:641:76b8:383a]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:5213:b0:641:72a8:c91c with SMTP id 4fb4d7f45d1cf-64350e9b333mr11728047a12.27.1763405332073; Mon, 17 Nov 2025 10:48:52 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:13 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-27-smostafa@google.com> Subject: [PATCH v5 26/27] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Based on the callbacks from the hypervisor, update the SMMUv3 Identity mapped page table. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 189 +++++++++++++++++- 1 file changed, 187 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 0a2ce6c06f4f..f0075f9a0947 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -12,6 +12,9 @@ =20 #include "arm_smmu_v3.h" =20 +#include +#include "../../../io-pgtable-arm.h" + size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 @@ -53,6 +56,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; __ret; \ }) =20 +/* Protected by host_mmu.lock from core code. */ +static struct io_pgtable *idmap_pgtable; + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -192,7 +198,6 @@ static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device = *smmu) smmu_cmdq_empty(&smmu->cmdq)); } =20 -__maybe_unused static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, struct arm_smmu_cmdq_ent *cmd) { @@ -204,6 +209,78 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device= *smmu, return smmu_sync_cmd(smmu); } =20 +static void __smmu_add_cmd(void *__opaque, struct arm_smmu_cmdq_batch *unu= sed, + struct arm_smmu_cmdq_ent *cmd) +{ + struct hyp_arm_smmu_v3_device *smmu =3D (struct hyp_arm_smmu_v3_device *)= __opaque; + + WARN_ON(smmu_add_cmd(smmu, cmd)); +} + +static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, size_t granule) +{ + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + idmap_pgtable->cfg.pgsize_bitmap, + smmu->features & ARM_SMMU_FEAT_RANGE_INV, + smmu, __smmu_add_cmd, NULL); + return smmu_sync_cmd(smmu); +} + +static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t gra= nule, + bool leaf) +{ + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_TLBI_S2_IPA, + .tlbi =3D { + .leaf =3D leaf, + .vmid =3D 0, + }, + }; + struct arm_smmu_cmdq_ent cmd_s1 =3D { + .opcode =3D CMDQ_OP_TLBI_NH_ALL, + .tlbi =3D { + .vmid =3D 0, + }, + }; + struct hyp_arm_smmu_v3_device *smmu; + + for_each_smmu(smmu) { + hyp_spin_lock(&smmu->lock); + /* + * Don't bother if SMMU is disabled, this would be useful for the case + * when RPM is supported to avoid thouching the SMMU MMIO when disabled. + * The hypervisor also asserts CMDQEN is enabled before the SMMU is + * enabled. As otherwise the host can prevent the hypervisor from doing + * TLB invalidations. + */ + if (is_smmu_enabled(smmu)) { + WARN_ON(smmu_tlb_inv_range_smmu(smmu, &cmd, iova, size, granule)); + WARN_ON(smmu_send_cmd(smmu, &cmd_s1)); + } + hyp_spin_unlock(&smmu->lock); + } +} + +static void smmu_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + smmu_tlb_inv_range(iova, size, granule, false); +} + +static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ + smmu_tlb_inv_range(iova, granule, granule, true); +} + +static const struct iommu_flush_ops smmu_tlb_ops =3D { + .tlb_flush_walk =3D smmu_tlb_flush_walk, + .tlb_add_page =3D smmu_tlb_add_page, +}; + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -454,6 +531,40 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) return ret; } =20 +static int smmu_init_pgt(void) +{ + /* Default values overridden based on SMMUs common features. */ + struct io_pgtable_cfg cfg =3D (struct io_pgtable_cfg) { + .tlb =3D &smmu_tlb_ops, + .pgsize_bitmap =3D -1, + .ias =3D 48, + .oas =3D 48, + .coherent_walk =3D true, + }; + struct hyp_arm_smmu_v3_device *smmu; + struct io_pgtable_ops *ops; + + for_each_smmu(smmu) { + cfg.ias =3D min(cfg.ias, smmu->ias); + cfg.oas =3D min(cfg.oas, smmu->oas); + cfg.pgsize_bitmap &=3D smmu->pgsize_bitmap; + cfg.coherent_walk &=3D !!(smmu->features & ARM_SMMU_FEAT_COHERENCY); + } + + /* Avoid larger input size as this is identity mapped. */ + cfg.ias =3D min(cfg.ias, cfg.oas); + + /* At least PAGE_SIZE must be supported by all SMMUs*/ + if ((cfg.pgsize_bitmap & PAGE_SIZE) =3D=3D 0) + return -EINVAL; + + ops =3D kvm_alloc_io_pgtable_ops(ARM_64_LPAE_S2, &cfg, NULL); + if (!ops) + return -ENOMEM; + idmap_pgtable =3D io_pgtable_ops_to_pgtable(ops); + return 0; +} + static int smmu_init(void) { int ret; @@ -475,7 +586,7 @@ static int smmu_init(void) =20 BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); =20 - return 0; + return smmu_init_pgt(); =20 out_reclaim_smmu: while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) @@ -798,8 +909,82 @@ static bool smmu_dabt_handler(struct user_pt_regs *reg= s, u64 esr, u64 addr) return false; } =20 +static size_t smmu_pgsize_idmap(size_t size, u64 paddr, size_t pgsize_bitm= ap) +{ + size_t pgsizes; + + /* Remove page sizes that are larger than the current size */ + pgsizes =3D pgsize_bitmap & GENMASK_ULL(__fls(size), 0); + + /* Remove page sizes that the address is not aligned to. */ + if (likely(paddr)) + pgsizes &=3D GENMASK_ULL(__ffs(paddr), 0); + + WARN_ON(!pgsizes); + + /* Return the larget page size that fits. */ + return BIT(__fls(pgsizes)); +} + static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) { + size_t size =3D end - start; + size_t pgsize =3D PAGE_SIZE, pgcount; + size_t mapped, unmapped; + int ret; + struct io_pgtable *pgtable =3D idmap_pgtable; + struct iommu_iotlb_gather gather; + + end =3D min(end, BIT(pgtable->cfg.oas)); + if (start >=3D end) + return; + + if (prot) { + if (!(prot & IOMMU_MMIO)) + prot |=3D IOMMU_CACHE; + + while (size) { + mapped =3D 0; + /* + * We handle pages size for memory and MMIO differently: + * - memory: Map everything with PAGE_SIZE, that is guaranteed to + * find memory as we allocated enough pages to cover the entire + * memory, we do that as io-pgtable-arm doesn't support + * split_blk_unmap logic any more, so we can't break blocks once + * mapped to tables. + * - MMIO: Unlike memory, pKVM allocate 1G to for all MMIO, while + * the MMIO space can be large, as it is assumed to cover the + * whole IAS that is not memory, we have to use block mappings, + * that is fine for MMIO as it is never donated at the moment, + * so we never need to unmap MMIO at the run time triggereing + * split block logic. + */ + if (prot & IOMMU_MMIO) + pgsize =3D smmu_pgsize_idmap(size, start, pgtable->cfg.pgsize_bitmap); + + pgcount =3D size / pgsize; + ret =3D pgtable->ops.map_pages(&pgtable->ops, start, start, + pgsize, pgcount, prot, 0, &mapped); + size -=3D mapped; + start +=3D mapped; + if (!mapped || ret) + return; + } + } else { + /* Shouldn't happen. */ + WARN_ON(prot & IOMMU_MMIO); + while (size) { + pgcount =3D size / pgsize; + unmapped =3D pgtable->ops.unmap_pages(&pgtable->ops, start, + pgsize, pgcount, &gather); + size -=3D unmapped; + start +=3D unmapped; + if (!unmapped) + break; + } + /* Some memory were not unmapped. */ + WARN_ON(size); + } } =20 /* Shared with the kernel driver in EL1 */ --=20 2.52.0.rc1.455.g30608eb744-goog From nobody Tue Dec 2 02:51:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1D20342519 for ; Mon, 17 Nov 2025 18:48:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405338; cv=none; b=UEdpUMHzcyJScHcX1KLdyDm845cylt14A7OeTch4x2p0Hd+Hu4UEy0MIeFnzhknFzYMCZ4HNFzj5MhJgHCM/LNKYsQomsZYZ4DrLBbVrzzjv++zZpGlHsJNUIPzM7d4LKV9ykZzeuAo9wMxXt1TNiKQG4X0XrlW77omg+JtBP+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763405338; c=relaxed/simple; bh=U01zuPmHBR4NpjwfbFt/lTc1f2okLAeIr+T5ztn4mi4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=rFFzxHzkPksYze0xNMSbhZ7hoz0lBxPVEmpzSOERlFp6h6iu6YZDxrCyCmZ6GxLeI1ur/NK3GME6dd3YQ7VpwU+X0RiOnRLvG0MW3A1AZ50CgH1biFJv8y/oCYEu0KFg4A4Y4fiyS/c55rmKB16GfXJls3cMUtgxE5JRRxiN4RU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vSFvZAq6; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vSFvZAq6" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4775f51ce36so40130195e9.1 for ; Mon, 17 Nov 2025 10:48:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1763405333; x=1764010133; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yqv9vt6rYdUcUmbSADKPrtDVgWOjgzH7G7qmk1O24d8=; b=vSFvZAq6YpxQKJUUoOTKSuVci3aEJwpWmIpX2hcZmI/YgKuDF9wSxiQJtqbYAX0yVG /P7DruV+unBjErFFdocFPJafIyZTEc0lyYqnP7dpOlGKDb2aelyTkA4JXEXK+Ejao4fy ciXgXQB45j5V9M018AisB/X/JDpN5U6c6ojwvcQgOqHXrY44pVgdNzHvp+EFxvCJ5LmE 2FlbkzDQx6Drok2PosS+ceL/k2c77xsilEAnKYve0EiodppaR3I+RmxsTv6ehYkZN+Cm 9vfJN9FG9+4JLLOkExeoIb9kW2hAFO1W3XgSa1m1KU18tkn1HUZmasaKig01dvp6KY7g KaCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763405333; x=1764010133; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yqv9vt6rYdUcUmbSADKPrtDVgWOjgzH7G7qmk1O24d8=; b=wWc0H3dfOHKq5xa3YM0f7SQ1GwZRxiZvdlVa71aXlsiXxD7yZRMWqiV4oLxjlX52Kx iYrvm7g32MO4dU8jCUmIAyaHBoFo5OMnyAIre8t7sSA3Pmvgu3ccngAJi7CobUsC+OJs cuBsAfUMGMUSWLW6988P7DBfe0lhovvM9NUC6xNrv5RLSXUyO+IZmsAinL4vF5f9r+GM 6/Tfw8/uObb0eKo7TsMzoUjQ9g/faFT82VJ2qcelRAORwFNFjZnRVQZzoXgBDGSu8uN8 rXHsgsqfsw/NuwXYi8Fz3SEUarSqHAN2Jg765ZafhAKygw9pCNns6xjmWJ2CQU6VU9uz 5zgQ== X-Forwarded-Encrypted: i=1; AJvYcCUaDaLg8ueuV3h2w3w7gKAL1qmXy/oGHp37Fh3B+OfLsL60i6wHylNMgeSWFk5KlyeJv7MRp38GFgiJ7js=@vger.kernel.org X-Gm-Message-State: AOJu0YynOGB3dpSQT3ViTMmfOftxSPjqQBpBtVQL4z8dy6BF4B9zf8L9 D26m7RCm4OniprbEuEePlqu65kgPbuoXFNSr45JO8XbIZ2EoQelgBbBZAau+xARJeJQfjwC7daJ uaPX9oNmutZo+JQ== X-Google-Smtp-Source: AGHT+IFKuKQmQwETqVMo5V4rZDWAjfD2WAPxG29C33FYX0labdGr2WvGuWsAPiA9DtRuNjEGIDoqy8Foar2wTg== X-Received: from wmqy12.prod.google.com ([2002:a05:600c:364c:b0:477:9aa2:7d50]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d15:b0:477:9bfc:dcb6 with SMTP id 5b1f17b1804b1-4779bfcde09mr72288665e9.14.1763405333101; Mon, 17 Nov 2025 10:48:53 -0800 (PST) Date: Mon, 17 Nov 2025 18:48:14 +0000 In-Reply-To: <20251117184815.1027271-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20251117184815.1027271-1-smostafa@google.com> X-Mailer: git-send-email 2.52.0.rc1.455.g30608eb744-goog Message-ID: <20251117184815.1027271-28-smostafa@google.com> Subject: [PATCH v5 27/27] iommu/arm-smmu-v3-kvm: Enable nesting From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, praan@google.com, danielmentz@google.com, mark.rutland@arm.com, qperret@google.com, tabba@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now, as the hypervisor controls the command queue, stream table, and shadows the stage-2 page table. Enable stage-2 in case the host puts an STE in bypass or stage-1. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 75 ++++++++++++++++++- 1 file changed, 74 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index f0075f9a0947..3e451cef937c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -371,6 +371,46 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static void smmu_attach_stage_2(struct arm_smmu_ste *ste) +{ + unsigned long vttbr; + unsigned long ts, sl, ic, oc, sh, tg, ps; + unsigned long cfg; + struct io_pgtable_cfg *pgt_cfg =3D &idmap_pgtable->cfg; + + cfg =3D FIELD_GET(STRTAB_STE_0_CFG, ste->data[0]); + if (!FIELD_GET(STRTAB_STE_0_V, ste->data[0]) || + (cfg =3D=3D STRTAB_STE_0_CFG_ABORT)) + return; + /* S2 is not advertised, that should never be attempted. */ + if (WARN_ON(cfg =3D=3D STRTAB_STE_0_CFG_NESTED)) + return; + vttbr =3D pgt_cfg->arm_lpae_s2_cfg.vttbr; + ps =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.ps; + tg =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tg; + sh =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sh; + oc =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.orgn; + ic =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.irgn; + sl =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sl; + ts =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tsz; + + ste->data[1] |=3D FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOM= ING); + /* The host shouldn't write dwords 2 and 3, overwrite them. */ + ste->data[2] =3D FIELD_PREP(STRTAB_STE_2_VTCR, + FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, ps) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, tg) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SH0, sh) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2OR0, oc) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2IR0, ic) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, sl) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, ts)) | + FIELD_PREP(STRTAB_STE_2_S2VMID, 0) | + STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R; + ste->data[3] =3D vttbr & STRTAB_STE_3_S2TTB_MASK; + /* Convert S1 =3D> nested and bypass =3D> S2 */ + ste->data[0] |=3D FIELD_PREP(STRTAB_STE_0_CFG, cfg | BIT(1)); +} + /* Get an STE for a stream table base. */ static struct arm_smmu_ste *smmu_get_ste_ptr(struct hyp_arm_smmu_v3_device= *smmu, u32 sid, u64 *strtab) @@ -426,6 +466,15 @@ static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_d= evice *smmu, u32 sid, bool u64 *hyp_ste_base =3D strtab_hyp_base(smmu); struct arm_smmu_ste *host_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, host_st= e_base); struct arm_smmu_ste *hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_= base); + struct arm_smmu_ste target =3D {}; + struct arm_smmu_cmdq_ent cfgi_cmd =3D { + .opcode =3D CMDQ_OP_CFGI_STE, + .cfgi =3D { + .sid =3D sid, + .leaf =3D true, + }, + }; + int i; =20 /* * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this @@ -441,8 +490,32 @@ static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_d= evice *smmu, u32 sid, bool hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_base); } =20 - smmu_copy_from_host(smmu, hyp_ste_ptr->data, host_ste_ptr->data, + smmu_copy_from_host(smmu, target.data, host_ste_ptr->data, STRTAB_STE_DWORDS << 3); + /* + * Typically, STE update is done as the following + * 1- Write last 7 dwords, while STE is invalid + * 2- CFGI + * 3- Write first dword, making STE valid + * 4- CFGI + * As the SMMU MUST at least load 64 bits atomically + * that gurantees that there is no race between writing + * the STE and the CFGI where the SMMU observes parts + * of the STE. + * In the shadow we update the STE to enable nested translation, + * which requires updating first 4 dwords. + * That is only done if the STE is valid and not in abort. + * Which means it happens at step 4) + * So we need to also write the last 7 dwords and send CFGI + * before writing the first dword. + * There is no need for last CFGI as it's done next. + */ + smmu_attach_stage_2(&target); + for (i =3D 1; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], target.data[i]); + + WARN_ON(smmu_send_cmd(smmu, &cfgi_cmd)); + WRITE_ONCE(hyp_ste_ptr->data[0], target.data[0]); } =20 static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) --=20 2.52.0.rc1.455.g30608eb744-goog