From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C110031194A for ; Tue, 19 Aug 2025 21:52:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640347; cv=none; b=XFB3Xx+dw7Yx6TAzf1Y6/1VbFHTs0rf+CaplSVJJGc6zpVNiZIk6wyEOnPCfedF6/khhA3e03YDk39Qu7lGA4z36fvQ3m1k85bZNpw/BtlzvpwrfGBu3ahujr/uwpKYROcU7KQaEpjg6k4v/kYH9HTeNDUn0noPoYiEAVfkSiGE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640347; c=relaxed/simple; bh=1DQlyjp3Ypyw67ZtM4RckxS+tfIALxj62mzUclwfeTQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dMYYK+l8z+UaG3SQILD6Kha3Ct9cbaM3320q2Z8GpyvT0w2Q5BfdAqBaAtZel/IRPP1ix5Ws2yV+lLnew/6aC7/mVGNIgx6a+01n7wJ/0ZpL5pmN8D4qI8fXv9X5kk08AIq0oUZyzmVDFuvJTcdM1HDNNaMQqorsNh+qEJj+2vE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jdU005YG; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jdU005YG" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3b9e4146aa2so3250074f8f.2 for ; Tue, 19 Aug 2025 14:52:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640344; x=1756245144; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ioxZ9+DNy86koGENgFZw7NoVqbAVt0aWywoB+DgYjJQ=; b=jdU005YGA9PDzDLNZCOWr7PRvYKlVSlmUsEoIVQgZMRtwIM9Ku9beF3Nti8u14C5a9 w58FznsppJT9vP3FRupRcuU2F/Hu+QHwZeZDbRM7GFI4B1kvHktCOU1fic5EhBOd26+o sotNmJODGgTkWZrKpC8lz2+Jd1Dc51GMxrKTIGH4bgilcLxq5Uq3LJ387CApwt8XxO3q TY3NlMuLLrHoP8E1P18LFyCK6tZ2aevlPrYK1WE/A16VWaIfIRiLNXtC39qNkrb1ibz2 F2XAuhfMsdobklh9MEEMcTp5M98z5y8ZcITn2j68Oa1E2bC/BtIy0UaBs5jliZyhVwms m86Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640344; x=1756245144; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ioxZ9+DNy86koGENgFZw7NoVqbAVt0aWywoB+DgYjJQ=; b=JeT3s+ZvOiYziHMGcJqA6gKw6+Th08VmLzBNLMNQMyLgvr9EU71tYH/vHZSw6bgnRX Q7pB3Q5e+q8DCIwBGA8zCcEkRDwkOsGqWY/pg60OeDud2IreyQ6JVkvnEkmniajTRCfk pLPZwt5UPM5pCSLu8jSL+9M3VTteEyZLRIAsxVq4J8c9CmF8JfqinXR11gIqrH7aaC4E DIK6CkBnKo5r12MriP1awbMsbHyGzIiUVXv8DXxK5aPKNipE2lUDbS/J/Rdp2OFy9FMv qp7PQ7FBJGqWBqJznydHtyq1etB4bcFmUrORHFJveePbCUlFCddc9Ydd/p/lVJW9e1TR QA2w== X-Gm-Message-State: AOJu0YymvtcDTVwdyn0VyLQHTRItkuqAz5CGwkEWrDej5hX/ZJtm4nIm jrJ8YVqyZViBCjAzsmiNxQKU53uRtnfSsPd7TJdDfTQ7q8W/r/YaK19T6MmN8GuM288SYVvoIY1 mupd2Sid18PSHoRcVHEw5JWcIixgQQHbQj6rUU/YTY9JrOAGnG4pRlFp1YPG8pJGaYXjDwnVvMz xUuA3t8+ZOZuAgCklOKaa0lX6OplB1GZEscSPl2JqdlsyGhgA0oTgofxw= X-Google-Smtp-Source: AGHT+IEHXpWg/f+rEUO60O9qrX4vZsfY84WtfBBV4+NcwjtcLonZ2Ky7JZEiBzijF5XkyOmNGYqUgyc+J4F+Sw== X-Received: from wmbfa5.prod.google.com ([2002:a05:600c:5185:b0:45a:2855:e836]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:438a:b0:3b7:b3f2:f8c3 with SMTP id ffacd0b85a97d-3c32ecce5bfmr288271f8f.57.1755640344145; Tue, 19 Aug 2025 14:52:24 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:29 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-2-smostafa@google.com> Subject: [PATCH v4 01/28] KVM: arm64: Add a new function to donate memory with prot From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Soon, IOMMU drivers running in the hypervisor might interact with non-coherent devices, so it needs a mechanism to map memory as non cacheable. Add ___pkvm_host_donate_hyp() which accepts a new argument for prot, so the driver can add KVM_PGTABLE_PROT_NORMAL_NC. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 1 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index 5f9d56754e39..52d7ee91e18c 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -36,6 +36,7 @@ int __pkvm_prot_finalize(void); int __pkvm_host_share_hyp(u64 pfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); +int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 8957734d6183..861e448183fd 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -769,13 +769,15 @@ int __pkvm_host_unshare_hyp(u64 pfn) return ret; } =20 -int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) +int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot) { u64 phys =3D hyp_pfn_to_phys(pfn); u64 size =3D PAGE_SIZE * nr_pages; void *virt =3D __hyp_va(phys); int ret; =20 + WARN_ON(prot & KVM_PGTABLE_PROT_X); + host_lock_component(); hyp_lock_component(); =20 @@ -787,7 +789,7 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) goto unlock; =20 __hyp_set_page_state_range(phys, size, PKVM_PAGE_OWNED); - WARN_ON(pkvm_create_mappings_locked(virt, virt + size, PAGE_HYP)); + WARN_ON(pkvm_create_mappings_locked(virt, virt + size, prot)); WARN_ON(host_stage2_set_owner_locked(phys, size, PKVM_ID_HYP)); =20 unlock: @@ -797,6 +799,11 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) return ret; } =20 +int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) +{ + return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP); +} + int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) { u64 phys =3D hyp_pfn_to_phys(pfn); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D76731AF26 for ; Tue, 19 Aug 2025 21:52:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640348; cv=none; b=sI3wP9OImOf6toVZWpX6fyH1NZevD8N+9IrsMguEE+9a2ZHqggsNvEDx6OLqhSyPdkHWty7JVyLA1nR6Q4tg0xXakWVQd0Msk3quLferJbYRAovCoe77Oxj2W7BzA4qcqeFrkE7bIiIgjPBk0WJ/OIcf3exErgIa4cK8AwGrO5I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640348; c=relaxed/simple; bh=BD3OB//GZsI6wyE5tIICFFT0lSi+PCLS/mLJZoc5JA0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mRp7ZMPX9yxV9RnM0yiZugGA/42AEzxBxNGVo9WnIZPsBGNhvnk9yKBDQk1vUfOXn76gzPiNGYiKzdZZDeOitVZ9IWnV7/7opCHAn3UtbbElIAOMJGFOOY60GTZUuxCm5resd0N01LO86kEaMV7kw8a2cRPjuZs/OkhKn0y2eN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=byvFxikl; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="byvFxikl" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0d221aso24417035e9.3 for ; Tue, 19 Aug 2025 14:52:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640345; x=1756245145; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=vVz3Y9VpnYE4PVj2+x4ppZYRh/7u9Tfs4/CSD5zsJnw=; b=byvFxikl7obK3H6zXDdaCSQX4hZYNsPQzRY2X9OAHET7S58sM9wW3/WCLbHaej3wd3 ksZSDjVTZPSl2mwKrK+kNJQv7MFzx2dPi3/C4G0rPVq4+T98YlqV8Usl9WGxhUkNYjpA Hy7fk4qboTGlS2lnKCycYGovm+Jj4CZ/Hli/mRYUNulxD7RdNG4HnRUqkircgYxaH3Up Wa+mmPhkztTqiCqIq6wLUWHi5HQDDtiXlmV9TvtFtc232g7lnMvijUy2Bf6qQc937K92 7YrFryiPoa87w0w26lgS/MTyhsDpWpH/mIJd9zQMMVDZbnfPV7Z3oxEtDAYqZohC8DV6 mXVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640345; x=1756245145; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=vVz3Y9VpnYE4PVj2+x4ppZYRh/7u9Tfs4/CSD5zsJnw=; b=KXCmFkCoDHbXLB62yEep1K4ODFMrNmIGzFbcVfomt2qOSS8Vd2dXmgWxLn4JcLPOiO FqA5JCZif9aTHmTznkfu6Oifru8N5D4GpQFO5+m1LcYJIfsgIvXGbNxZq5NdXGOrS95x HH3u4MW/9e2+MbUDCXAVz5xkOJc6UDmUCZvd9rbbNp6ydbN1BbgfzDjqkBsniD0ded3R m/XgJC6Dn70nMy6fc5MQpr/BjpmMylazoXikg16DBtDOvgbKQw8krh4VZ3a4ltx1zzw4 xeQU+G7Uwm+RKnNLs1Hm7cmi6EF2nUFdUL66Ihi6ADTRn489ed0DtvytOQi8YCwHLW4T 1vFQ== X-Gm-Message-State: AOJu0YxFdPo4OMBPYLbSdsYLlf5VK2ibR6V2xtbRqHcsnzoC1HbUsM03 PMpTWnNTvVoFpSvlhRwnLaPxu+jxchr9ksnsEoaIX223SqAkt6+igIphMxZ2FAP2nH4Kg7vxoBg 7WbTK1w1EXgzUH5NHLl/6kIGRu0Ubb2shBng4F/cX/ZxdJtNU0rymTwacpO/lpn6DSDkXBmfgLz GFuSJ38iguXJvAWAp0wK3fAe/ZWOY203fxo3qdGe4eso4hIwNqCWAGdOk= X-Google-Smtp-Source: AGHT+IE7z3JoHU46VhERaG2rUhWS+WRB2JYInik0I1QmJUrgMcHdJZz9Xytw3QMikJfteQ5I/RRwqWO+5o1F0Q== X-Received: from wmbem12.prod.google.com ([2002:a05:600c:820c:b0:458:c0cd:291c]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f0c:b0:43d:4e9:27ff with SMTP id 5b1f17b1804b1-45b4798a7bdmr3379925e9.7.1755640344849; Tue, 19 Aug 2025 14:52:24 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:30 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-3-smostafa@google.com> Subject: [PATCH v4 02/28] KVM: arm64: Donate MMIO to the hypervisor From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a function to donate MMIO to the hypervisor so IOMMU hypervisor drivers can use that to protect the MMIO of IOMMU. The initial attempt to implement this was to have a new flag to "___pkvm_host_donate_hyp" to accept MMIO. However that had many problems, it was quite intrusive for host/hyp to check/set page state to make it aware of MMIO and to encode the state in the page table in that case. Which is called in paths that can be sensitive to performance (FFA, VMs..) As donating MMIO is very rare, and we don=E2=80=99t need to encode the full= state, it=E2=80=99s reasonable to have a separate function to do this. It will init the host s2 page table with an invalid leaf with the owner ID to prevent the host from mapping the page on faults. Also, prevent kvm_pgtable_stage2_unmap() from removing owner ID from stage-2 PTEs, as this can be triggered from recycle logic under memory pressure. There is no code relying on this, as all ownership changes is done via kvm_pgtable_stage2_set_owner() For error path in IOMMU drivers, add a function to donate MMIO back from hyp to host. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 64 +++++++++++++++++++ arch/arm64/kvm/hyp/pgtable.c | 9 +-- 3 files changed, 68 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index 52d7ee91e18c..98e173da0f9b 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -37,6 +37,8 @@ int __pkvm_host_share_hyp(u64 pfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enum kvm_pgtable_prot p= rot); +int __pkvm_host_donate_hyp_mmio(u64 pfn); +int __pkvm_hyp_donate_host_mmio(u64 pfn); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 861e448183fd..c9a15ef6b18d 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -799,6 +799,70 @@ int ___pkvm_host_donate_hyp(u64 pfn, u64 nr_pages, enu= m kvm_pgtable_prot prot) return ret; } =20 +int __pkvm_host_donate_hyp_mmio(u64 pfn) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + void *virt =3D __hyp_va(phys); + int ret; + kvm_pte_t pte; + + host_lock_component(); + hyp_lock_component(); + + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, phys, &pte, NULL); + if (ret) + goto unlock; + + if (pte && !kvm_pte_valid(pte)) { + ret =3D -EPERM; + goto unlock; + } + + ret =3D kvm_pgtable_get_leaf(&pkvm_pgtable, (u64)virt, &pte, NULL); + if (ret) + goto unlock; + if (pte) { + ret =3D -EBUSY; + goto unlock; + } + + ret =3D pkvm_create_mappings_locked(virt, virt + PAGE_SIZE, PAGE_HYP_DEVI= CE); + if (ret) + goto unlock; + /* + * We set HYP as the owner of the MMIO pages in the host stage-2, for: + * - host aborts: host_stage2_adjust_range() would fail for invalid non z= ero PTEs. + * - recycle under memory pressure: host_stage2_unmap_dev_all() would call + * kvm_pgtable_stage2_unmap() which will not clear non zero invalid pte= s (counted). + * - other MMIO donation: Would fail as we check that the PTE is valid or= empty. + */ + WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys, + PAGE_SIZE, &host_s2_pool, PKVM_ID_HYP)); +unlock: + hyp_unlock_component(); + host_unlock_component(); + + return ret; +} + +int __pkvm_hyp_donate_host_mmio(u64 pfn) +{ + u64 phys =3D hyp_pfn_to_phys(pfn); + u64 virt =3D (u64)__hyp_va(phys); + size_t size =3D PAGE_SIZE; + + host_lock_component(); + hyp_lock_component(); + + WARN_ON(kvm_pgtable_hyp_unmap(&pkvm_pgtable, virt, size) !=3D size); + WARN_ON(host_stage2_try(kvm_pgtable_stage2_set_owner, &host_mmu.pgt, phys, + PAGE_SIZE, &host_s2_pool, PKVM_ID_HOST)); + hyp_unlock_component(); + host_unlock_component(); + + return 0; +} + int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) { return ___pkvm_host_donate_hyp(pfn, nr_pages, PAGE_HYP); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index c351b4abd5db..ba06b0c21d5a 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1095,13 +1095,8 @@ static int stage2_unmap_walker(const struct kvm_pgta= ble_visit_ctx *ctx, kvm_pte_t *childp =3D NULL; bool need_flush =3D false; =20 - if (!kvm_pte_valid(ctx->old)) { - if (stage2_pte_is_counted(ctx->old)) { - kvm_clear_pte(ctx->ptep); - mm_ops->put_page(ctx->ptep); - } - return 0; - } + if (!kvm_pte_valid(ctx->old)) + return stage2_pte_is_counted(ctx->old) ? -EPERM : 0; =20 if (kvm_pte_table(ctx->old, ctx->level)) { childp =3D kvm_pte_follow(ctx->old, mm_ops); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89EF52E2298 for ; Tue, 19 Aug 2025 21:52:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640349; cv=none; b=Yddhn5HNufYIwA1DYHfEno965r6c/9uhzIE9w5Z/58veYxYAJxCe07taEL/sJQkpxGgMZ7rX2B18q3tHEgo6PTIek66ZMtpk/eZDBrGhVs99JshkMbtnYV1dOXv9E6LwjgXmTTLg0MR5tTdfKxa2l3qyb31gL8bbTgzleeJPUCc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640349; c=relaxed/simple; bh=iu6gKl2CIH0Do5wVBqGs+FISYJXVfUyFEZp0ZLhFP2w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=thn/2sp/HL+15RDjrGQPcf7Wp8uRbW4LbOxuPdhPB31aHH2PeUUGSb+QfUn2iS/lYWt+QAwHhLHvQLOXF3i3KAkk0jE3z+U9QhSNSTtkvDnMM4d6oHendw6gOOftLqhn2E3Ig0Ond1asEKqK9Jyiy61VXMiBrSSEeZUFIYFHEIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=U9VqVqey; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="U9VqVqey" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-afcb5f59d9fso22738066b.0 for ; Tue, 19 Aug 2025 14:52:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640346; x=1756245146; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=W+DG7jBXGZxGQ4XuCOpRWmsZ2ZfXC/o78dYhqpKpX0Y=; b=U9VqVqeyxO39vlQMeIQ4a5OsNrkkZUj7jDLghHQxCr1SbcqTLld0WZBHqW4eKUivT8 EkVbfCcaEcpqQmFBsP1QDA5MDc764jxT501R1GOre+AMbq5l5VPSaWGDvCa1K0xmY1FD 2tDkAbyV3mj0dVtWGSUbnzaP/Sup4RTkIPXniZw6rHbYiNzz4UBT5QZZLqFLwuy+HFNq G+ZYIXbl8nx8PdRZTKm1uxKG88RjpGk6V1PPAZXF/iqZzDVLw0rAvHYscutpvwVjhS9g 6nnI+JhcR6bP97koT63s8v5yjNWW48qGAArkZu/r4Di5wW8YT0T5/Uc37Bkh58Vs0+WM Ee1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640346; x=1756245146; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W+DG7jBXGZxGQ4XuCOpRWmsZ2ZfXC/o78dYhqpKpX0Y=; b=nrVgMWKOaheSHvbue1WQ8cWfheUDhRVNuTlLfWzz9vkRwwt66OUukR0jC9NodRMmSr FFGH8Bk2je47twQxY2yPZD6c8oIzgyQvLnnxAxZlypPgsRmzii0Q36DN7YPr2zxR9Yo2 BxgHjKRqXaOGgGRcTcluUobop9FkFw31QyF7P0S3J/XaYxHu2Vidpfh1RvfrV1upWJ1r jkt/+HUTYp/k9HiymMi+6X7UZfLA2GoX2tFhLf3rIcIlnG1DiRBdT1O8MMZzlCxKBMsb DfhYAL8uQIdfg7fKnW3JzgUdUHqf3KzQApo+5zf1U5lGdzUyrzveYk4AzGyLSPlkntza PWlg== X-Gm-Message-State: AOJu0YxRfTIqSAoB3K9r82cU8qfzP+LE9lAHuaEQy2In86Mx1FY78uCx jvhR3mavakiFmf27Si4plf645tR/K/JcffBEd8l8ANffhFkh85IsH3bAREJOOo10bNt12yjHV3J hLwwR1E8jICBtDlnKJyhe52Ml4a2CwdAWRygxGcHVm7iPsV96PbbGnkML3yjGWaquw/03b9r2bd yC+hDAdBRT1w50B0y7Tdyxl1H1fZw8N1lI0cr92mBKodX4kMqu6NKfmN0= X-Google-Smtp-Source: AGHT+IH9p4RnQonIqO2IvBstcQzhGwFgVqersbPBUPhqv5wCNXWBDT/LoPiy2B89sQst64Mbf5itUgGSgohDDQ== X-Received: from edc12.prod.google.com ([2002:a05:6402:460c:b0:61a:3e52:115a]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:7fa3:b0:af9:23b:9f9 with SMTP id a640c23a62f3a-afdded25839mr293959566b.24.1755640345755; Tue, 19 Aug 2025 14:52:25 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:31 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-4-smostafa@google.com> Subject: [PATCH v4 03/28] KVM: arm64: pkvm: Add pkvm_time_get() From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a function to return time in us. This can be used from IOMMU drivers while waiting for conditions as for SMMUv3 TLB invalidation waiting for sync. Signed-off-by: Mostafa Saleh Signed-off-by: Jean-Philippe Brucker --- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 2 ++ arch/arm64/kvm/hyp/nvhe/setup.c | 4 ++++ arch/arm64/kvm/hyp/nvhe/timer-sr.c | 33 ++++++++++++++++++++++++++ 3 files changed, 39 insertions(+) diff --git a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h b/arch/arm64/kvm/hyp/in= clude/nvhe/pkvm.h index ce31d3b73603..6c19691720cd 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/pkvm.h +++ b/arch/arm64/kvm/hyp/include/nvhe/pkvm.h @@ -87,4 +87,6 @@ bool kvm_handle_pvm_restricted(struct kvm_vcpu *vcpu, u64= *exit_code); void kvm_init_pvm_id_regs(struct kvm_vcpu *vcpu); int kvm_check_pvm_sysreg_table(void); =20 +int pkvm_timer_init(void); +u64 pkvm_time_get(void); #endif /* __ARM64_KVM_NVHE_PKVM_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index a48d3f5a5afb..ee6435473204 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -304,6 +304,10 @@ void __noreturn __pkvm_init_finalise(void) }; pkvm_pgtable.mm_ops =3D &pkvm_pgtable_mm_ops; =20 + ret =3D pkvm_timer_init(); + if (ret) + goto out; + ret =3D fix_host_ownership(); if (ret) goto out; diff --git a/arch/arm64/kvm/hyp/nvhe/timer-sr.c b/arch/arm64/kvm/hyp/nvhe/t= imer-sr.c index ff176f4ce7de..e166cd5a56b8 100644 --- a/arch/arm64/kvm/hyp/nvhe/timer-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/timer-sr.c @@ -11,6 +11,10 @@ #include #include =20 +#include + +static u32 timer_freq; + void __kvm_timer_set_cntvoff(u64 cntvoff) { write_sysreg(cntvoff, cntvoff_el2); @@ -68,3 +72,32 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu) =20 sysreg_clear_set(cnthctl_el2, clr, set); } + +static u64 pkvm_ticks_get(void) +{ + return __arch_counter_get_cntvct(); +} + +#define SEC_TO_US 1000000 + +int pkvm_timer_init(void) +{ + timer_freq =3D read_sysreg(cntfrq_el0); + /* + * TODO: The highest privileged level is supposed to initialize this + * register. But on some systems (which?), this information is only + * contained in the device-tree, so we'll need to find it out some other + * way. + */ + if (!timer_freq || timer_freq < SEC_TO_US) + return -ENODEV; + return 0; +} + +#define pkvm_time_ticks_to_us(ticks) ((u64)(ticks) * SEC_TO_US / timer_fre= q) + +/* Return time in us. */ +u64 pkvm_time_get(void) +{ + return pkvm_time_ticks_to_us(pkvm_ticks_get()); +} --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EF653469E3 for ; Tue, 19 Aug 2025 21:52:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640351; cv=none; b=AqhoG7EKjNnam9COI52mwHqy1RCXJZKfMbtd3KMGfuN1sIYT2kLGFrEvo1+4xSFb+ak89K2voJ6KA66SoqgEhcZCCKwP6EaGVzy0fDCmZwtQuR6QbfbKs8mJhp/b+DzbGlBXO0qk1BzccLR0RiYkxIcb4PnVl0zKzD7La7kLBkg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640351; c=relaxed/simple; bh=hXDOvZIpu6yyJHFw/mqRtW1+koiyAV4owG5DPpyR3nE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ljOyWGfTnGi+ACN0w8xM2tS6qHuWaNhzzymD62lZOWOdru5o+tRTnOW9AgS2mNfiE6qiUgM9ojllwOWBTvuW1JFe+S3gnI5KgBgM4/16OgxTpEsgLteqORTJhy+6FJ5uRgEyxXPGg39+9iKuDkbMkmMZyRciBtlXY9xXwwA/+Xg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WhZehyrU; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WhZehyrU" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0045a0so31009725e9.0 for ; Tue, 19 Aug 2025 14:52:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640347; x=1756245147; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UN5Lx0TpQ9JBjLBscgxu6am7pN7RbulKEGMCKcFuzuI=; b=WhZehyrUxCJYJUKE0j13tgPTOGQiDzv/p217ykpQOkF/J5umJyvtVg49Hukrw/B8B6 +rerd4QDR/6Ld53/dJBgaoVnxVnse5rqRXkiUrZbjvsA3ITqGhaInFAxNNEgL1Ian15G 6XQJFEs8mmsIqsEEFgkOS/wJY4hEz8/32C2EnjktRbxHDryk04i3nISpaLWOend7tWTS ZLE3Pmf2wZWxwQjmeNuHOZvA30T3Fx6DFo557j3RhaY0QpnoJ5jDCzU0j5RGQb21B1oF lhlNZVFhZK/+vGRyUQohLZSN16XFo0f73ILTzsVO+EZYzKGYBQsFgXvrHH8l+F6tiTgA IQ5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640347; x=1756245147; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UN5Lx0TpQ9JBjLBscgxu6am7pN7RbulKEGMCKcFuzuI=; b=A70CKMG0nBMKT+vUN2U0bY8XKGgAQmH2UvEfCX03J+L+kUKn66xkMNobycILkwtSF9 MCxNBwoov9gExcDs4OnYf6ODeK6P/V6+WajoaE2wOE/pXd3VUe/5mLBP2rTeu17JGte2 sXxDaAMnyRRkikUzGq3j1JxeQkF1j+rQyJUMOkHO1fEIiLvjOM6Qx2Eg62wmVuiOOoxm HzjJz7slIbTSKCs7HgNdhXUI7NfXDty23Zi47aTq9Gynj4gCGUfCGkVjhq5AOjnyp57x WguQyIKDKS5wCWeA7E+uB+e/xCC88lyFArdCFzp/e5qzXckbg/FUHTozi3e5mfu+MLEh aEMQ== X-Gm-Message-State: AOJu0Yzw+++tLYcuZYvh4TAkb67Gg/lECn9Q83SWinN0JZLlNp7Niq8g O58wTDAD4SIam3tSgvslN3nbbuRC61uOXpg7M1qYF/sMjBceXObRkNe4TiQfMNHQc3f9pW4NYEV GksM9lNNv8EGfeegFsy4P0aeuTuaeizHQynOWPvJTJ3q9cJmBgljdnI2jzITcYKj4/zRX7rZvaO jQ7NxF0xUnY7qsxnuwsKxRojYJGZ7+pePFRDQH2Q4wDEP44AZJDBqynX0= X-Google-Smtp-Source: AGHT+IFfGGhQp/LMSHGNHUFDTsK3KwMtLRsoFmkddiawwjZDD2qQVNA+kxHOOymS+x1LtUNJ/bavT/Nny02iqw== X-Received: from wmbeb11.prod.google.com ([2002:a05:600c:678b:b0:459:d5f6:df30]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:47c9:b0:458:bfe1:4a81 with SMTP id 5b1f17b1804b1-45b47a2042dmr2041815e9.17.1755640346929; Tue, 19 Aug 2025 14:52:26 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:32 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-5-smostafa@google.com> Subject: [PATCH v4 04/28] iommu/io-pgtable-arm: Move selftests to a separate file From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Soon, io-pgtable-arm.c will be compiled as part of the KVM/arm64 in the hypervisor object, which doesn't have many of the kernel APIs, as faux devices, printk... We would need to factor this things outside of this file, this patch moves the selftests outside, which remove many of the kernel dependencies, which also is not needed by the hypervisor. Create io-pgtable-arm-kernel.c for that, and in the next patch the rest of the code is factored out. Signed-off-by: Mostafa Saleh Reviewed-by: Pranjal Shrivastava --- drivers/iommu/Makefile | 2 +- drivers/iommu/io-pgtable-arm-kernel.c | 216 +++++++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 245 -------------------------- drivers/iommu/io-pgtable-arm.h | 41 +++++ 4 files changed, 258 insertions(+), 246 deletions(-) create mode 100644 drivers/iommu/io-pgtable-arm-kernel.c diff --git a/drivers/iommu/Makefile b/drivers/iommu/Makefile index 355294fa9033..d601b0e25ef5 100644 --- a/drivers/iommu/Makefile +++ b/drivers/iommu/Makefile @@ -11,7 +11,7 @@ obj-$(CONFIG_IOMMU_DEBUGFS) +=3D iommu-debugfs.o obj-$(CONFIG_IOMMU_DMA) +=3D dma-iommu.o obj-$(CONFIG_IOMMU_IO_PGTABLE) +=3D io-pgtable.o obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) +=3D io-pgtable-arm-v7s.o -obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) +=3D io-pgtable-arm.o +obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) +=3D io-pgtable-arm.o io-pgtable-arm-k= ernel.o obj-$(CONFIG_IOMMU_IO_PGTABLE_DART) +=3D io-pgtable-dart.o obj-$(CONFIG_IOMMU_IOVA) +=3D iova.o obj-$(CONFIG_OF_IOMMU) +=3D of_iommu.o diff --git a/drivers/iommu/io-pgtable-arm-kernel.c b/drivers/iommu/io-pgtab= le-arm-kernel.c new file mode 100644 index 000000000000..f3b869310964 --- /dev/null +++ b/drivers/iommu/io-pgtable-arm-kernel.c @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * CPU-agnostic ARM page table allocator. + * + * Copyright (C) 2014 ARM Limited + * + * Author: Will Deacon + */ +#define pr_fmt(fmt) "arm-lpae io-pgtable: " fmt + +#include +#include +#include + +#include "io-pgtable-arm.h" + +#ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST + +static struct io_pgtable_cfg *cfg_cookie __initdata; + +static void __init dummy_tlb_flush_all(void *cookie) +{ + WARN_ON(cookie !=3D cfg_cookie); +} + +static void __init dummy_tlb_flush(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + WARN_ON(cookie !=3D cfg_cookie); + WARN_ON(!(size & cfg_cookie->pgsize_bitmap)); +} + +static void __init dummy_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ + dummy_tlb_flush(iova, granule, granule, cookie); +} + +static const struct iommu_flush_ops dummy_tlb_ops __initconst =3D { + .tlb_flush_all =3D dummy_tlb_flush_all, + .tlb_flush_walk =3D dummy_tlb_flush, + .tlb_add_page =3D dummy_tlb_add_page, +}; + +static void __init arm_lpae_dump_ops(struct io_pgtable_ops *ops) +{ + struct arm_lpae_io_pgtable *data =3D io_pgtable_ops_to_data(ops); + struct io_pgtable_cfg *cfg =3D &data->iop.cfg; + + pr_err("cfg: pgsize_bitmap 0x%lx, ias %u-bit\n", + cfg->pgsize_bitmap, cfg->ias); + pr_err("data: %d levels, 0x%zx pgd_size, %u pg_shift, %u bits_per_level, = pgd @ %p\n", + ARM_LPAE_MAX_LEVELS - data->start_level, ARM_LPAE_PGD_SIZE(data), + ilog2(ARM_LPAE_GRANULE(data)), data->bits_per_level, data->pgd); +} + +#define __FAIL(ops, i) ({ \ + WARN(1, "selftest: test failed for fmt idx %d\n", (i)); \ + arm_lpae_dump_ops(ops); \ + -EFAULT; \ +}) + +static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg) +{ + static const enum io_pgtable_fmt fmts[] __initconst =3D { + ARM_64_LPAE_S1, + ARM_64_LPAE_S2, + }; + + int i, j; + unsigned long iova; + size_t size, mapped; + struct io_pgtable_ops *ops; + + for (i =3D 0; i < ARRAY_SIZE(fmts); ++i) { + cfg_cookie =3D cfg; + ops =3D alloc_io_pgtable_ops(fmts[i], cfg, cfg); + if (!ops) { + pr_err("selftest: failed to allocate io pgtable ops\n"); + return -ENOMEM; + } + + /* + * Initial sanity checks. + * Empty page tables shouldn't provide any translations. + */ + if (ops->iova_to_phys(ops, 42)) + return __FAIL(ops, i); + + if (ops->iova_to_phys(ops, SZ_1G + 42)) + return __FAIL(ops, i); + + if (ops->iova_to_phys(ops, SZ_2G + 42)) + return __FAIL(ops, i); + + /* + * Distinct mappings of different granule sizes. + */ + iova =3D 0; + for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) { + size =3D 1UL << j; + + if (ops->map_pages(ops, iova, iova, size, 1, + IOMMU_READ | IOMMU_WRITE | + IOMMU_NOEXEC | IOMMU_CACHE, + GFP_KERNEL, &mapped)) + return __FAIL(ops, i); + + /* Overlapping mappings */ + if (!ops->map_pages(ops, iova, iova + size, size, 1, + IOMMU_READ | IOMMU_NOEXEC, + GFP_KERNEL, &mapped)) + return __FAIL(ops, i); + + if (ops->iova_to_phys(ops, iova + 42) !=3D (iova + 42)) + return __FAIL(ops, i); + + iova +=3D SZ_1G; + } + + /* Full unmap */ + iova =3D 0; + for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) { + size =3D 1UL << j; + + if (ops->unmap_pages(ops, iova, size, 1, NULL) !=3D size) + return __FAIL(ops, i); + + if (ops->iova_to_phys(ops, iova + 42)) + return __FAIL(ops, i); + + /* Remap full block */ + if (ops->map_pages(ops, iova, iova, size, 1, + IOMMU_WRITE, GFP_KERNEL, &mapped)) + return __FAIL(ops, i); + + if (ops->iova_to_phys(ops, iova + 42) !=3D (iova + 42)) + return __FAIL(ops, i); + + iova +=3D SZ_1G; + } + + /* + * Map/unmap the last largest supported page of the IAS, this can + * trigger corner cases in the concatednated page tables. + */ + mapped =3D 0; + size =3D 1UL << __fls(cfg->pgsize_bitmap); + iova =3D (1UL << cfg->ias) - size; + if (ops->map_pages(ops, iova, iova, size, 1, + IOMMU_READ | IOMMU_WRITE | + IOMMU_NOEXEC | IOMMU_CACHE, + GFP_KERNEL, &mapped)) + return __FAIL(ops, i); + if (mapped !=3D size) + return __FAIL(ops, i); + if (ops->unmap_pages(ops, iova, size, 1, NULL) !=3D size) + return __FAIL(ops, i); + + free_io_pgtable_ops(ops); + } + + return 0; +} + +static int __init arm_lpae_do_selftests(void) +{ + static const unsigned long pgsize[] __initconst =3D { + SZ_4K | SZ_2M | SZ_1G, + SZ_16K | SZ_32M, + SZ_64K | SZ_512M, + }; + + static const unsigned int address_size[] __initconst =3D { + 32, 36, 40, 42, 44, 48, + }; + + int i, j, k, pass =3D 0, fail =3D 0; + struct faux_device *dev; + struct io_pgtable_cfg cfg =3D { + .tlb =3D &dummy_tlb_ops, + .coherent_walk =3D true, + .quirks =3D IO_PGTABLE_QUIRK_NO_WARN, + }; + + dev =3D faux_device_create("io-pgtable-test", NULL, 0); + if (!dev) + return -ENOMEM; + + cfg.iommu_dev =3D &dev->dev; + + for (i =3D 0; i < ARRAY_SIZE(pgsize); ++i) { + for (j =3D 0; j < ARRAY_SIZE(address_size); ++j) { + /* Don't use ias > oas as it is not valid for stage-2. */ + for (k =3D 0; k <=3D j; ++k) { + cfg.pgsize_bitmap =3D pgsize[i]; + cfg.ias =3D address_size[k]; + cfg.oas =3D address_size[j]; + pr_info("selftest: pgsize_bitmap 0x%08lx, IAS %u OAS %u\n", + pgsize[i], cfg.ias, cfg.oas); + if (arm_lpae_run_tests(&cfg)) + fail++; + else + pass++; + } + } + } + + pr_info("selftest: completed with %d PASS %d FAIL\n", pass, fail); + faux_device_destroy(dev); + + return fail ? -EFAULT : 0; +} +subsys_initcall(arm_lpae_do_selftests); +#endif diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 96425e92f313..791a2c4ecb83 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -7,15 +7,10 @@ * Author: Will Deacon */ =20 -#define pr_fmt(fmt) "arm-lpae io-pgtable: " fmt - #include #include #include -#include -#include #include -#include #include #include =20 @@ -24,33 +19,6 @@ #include "io-pgtable-arm.h" #include "iommu-pages.h" =20 -#define ARM_LPAE_MAX_ADDR_BITS 52 -#define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 -#define ARM_LPAE_MAX_LEVELS 4 - -/* Struct accessors */ -#define io_pgtable_to_data(x) \ - container_of((x), struct arm_lpae_io_pgtable, iop) - -#define io_pgtable_ops_to_data(x) \ - io_pgtable_to_data(io_pgtable_ops_to_pgtable(x)) - -/* - * Calculate the right shift amount to get to the portion describing level= l - * in a virtual address mapped by the pagetable in d. - */ -#define ARM_LPAE_LVL_SHIFT(l,d) \ - (((ARM_LPAE_MAX_LEVELS - (l)) * (d)->bits_per_level) + \ - ilog2(sizeof(arm_lpae_iopte))) - -#define ARM_LPAE_GRANULE(d) \ - (sizeof(arm_lpae_iopte) << (d)->bits_per_level) -#define ARM_LPAE_PGD_SIZE(d) \ - (sizeof(arm_lpae_iopte) << (d)->pgd_bits) - -#define ARM_LPAE_PTES_PER_TABLE(d) \ - (ARM_LPAE_GRANULE(d) >> ilog2(sizeof(arm_lpae_iopte))) - /* * Calculate the index at level l used to map virtual address a using the * pagetable in d. @@ -163,18 +131,6 @@ #define iopte_set_writeable_clean(ptep) \ set_bit(ARM_LPAE_PTE_AP_RDONLY_BIT, (unsigned long *)(ptep)) =20 -struct arm_lpae_io_pgtable { - struct io_pgtable iop; - - int pgd_bits; - int start_level; - int bits_per_level; - - void *pgd; -}; - -typedef u64 arm_lpae_iopte; - static inline bool iopte_leaf(arm_lpae_iopte pte, int lvl, enum io_pgtable_fmt fmt) { @@ -1274,204 +1230,3 @@ struct io_pgtable_init_fns io_pgtable_arm_mali_lpae= _init_fns =3D { .alloc =3D arm_mali_lpae_alloc_pgtable, .free =3D arm_lpae_free_pgtable, }; - -#ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST - -static struct io_pgtable_cfg *cfg_cookie __initdata; - -static void __init dummy_tlb_flush_all(void *cookie) -{ - WARN_ON(cookie !=3D cfg_cookie); -} - -static void __init dummy_tlb_flush(unsigned long iova, size_t size, - size_t granule, void *cookie) -{ - WARN_ON(cookie !=3D cfg_cookie); - WARN_ON(!(size & cfg_cookie->pgsize_bitmap)); -} - -static void __init dummy_tlb_add_page(struct iommu_iotlb_gather *gather, - unsigned long iova, size_t granule, - void *cookie) -{ - dummy_tlb_flush(iova, granule, granule, cookie); -} - -static const struct iommu_flush_ops dummy_tlb_ops __initconst =3D { - .tlb_flush_all =3D dummy_tlb_flush_all, - .tlb_flush_walk =3D dummy_tlb_flush, - .tlb_add_page =3D dummy_tlb_add_page, -}; - -static void __init arm_lpae_dump_ops(struct io_pgtable_ops *ops) -{ - struct arm_lpae_io_pgtable *data =3D io_pgtable_ops_to_data(ops); - struct io_pgtable_cfg *cfg =3D &data->iop.cfg; - - pr_err("cfg: pgsize_bitmap 0x%lx, ias %u-bit\n", - cfg->pgsize_bitmap, cfg->ias); - pr_err("data: %d levels, 0x%zx pgd_size, %u pg_shift, %u bits_per_level, = pgd @ %p\n", - ARM_LPAE_MAX_LEVELS - data->start_level, ARM_LPAE_PGD_SIZE(data), - ilog2(ARM_LPAE_GRANULE(data)), data->bits_per_level, data->pgd); -} - -#define __FAIL(ops, i) ({ \ - WARN(1, "selftest: test failed for fmt idx %d\n", (i)); \ - arm_lpae_dump_ops(ops); \ - -EFAULT; \ -}) - -static int __init arm_lpae_run_tests(struct io_pgtable_cfg *cfg) -{ - static const enum io_pgtable_fmt fmts[] __initconst =3D { - ARM_64_LPAE_S1, - ARM_64_LPAE_S2, - }; - - int i, j; - unsigned long iova; - size_t size, mapped; - struct io_pgtable_ops *ops; - - for (i =3D 0; i < ARRAY_SIZE(fmts); ++i) { - cfg_cookie =3D cfg; - ops =3D alloc_io_pgtable_ops(fmts[i], cfg, cfg); - if (!ops) { - pr_err("selftest: failed to allocate io pgtable ops\n"); - return -ENOMEM; - } - - /* - * Initial sanity checks. - * Empty page tables shouldn't provide any translations. - */ - if (ops->iova_to_phys(ops, 42)) - return __FAIL(ops, i); - - if (ops->iova_to_phys(ops, SZ_1G + 42)) - return __FAIL(ops, i); - - if (ops->iova_to_phys(ops, SZ_2G + 42)) - return __FAIL(ops, i); - - /* - * Distinct mappings of different granule sizes. - */ - iova =3D 0; - for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) { - size =3D 1UL << j; - - if (ops->map_pages(ops, iova, iova, size, 1, - IOMMU_READ | IOMMU_WRITE | - IOMMU_NOEXEC | IOMMU_CACHE, - GFP_KERNEL, &mapped)) - return __FAIL(ops, i); - - /* Overlapping mappings */ - if (!ops->map_pages(ops, iova, iova + size, size, 1, - IOMMU_READ | IOMMU_NOEXEC, - GFP_KERNEL, &mapped)) - return __FAIL(ops, i); - - if (ops->iova_to_phys(ops, iova + 42) !=3D (iova + 42)) - return __FAIL(ops, i); - - iova +=3D SZ_1G; - } - - /* Full unmap */ - iova =3D 0; - for_each_set_bit(j, &cfg->pgsize_bitmap, BITS_PER_LONG) { - size =3D 1UL << j; - - if (ops->unmap_pages(ops, iova, size, 1, NULL) !=3D size) - return __FAIL(ops, i); - - if (ops->iova_to_phys(ops, iova + 42)) - return __FAIL(ops, i); - - /* Remap full block */ - if (ops->map_pages(ops, iova, iova, size, 1, - IOMMU_WRITE, GFP_KERNEL, &mapped)) - return __FAIL(ops, i); - - if (ops->iova_to_phys(ops, iova + 42) !=3D (iova + 42)) - return __FAIL(ops, i); - - iova +=3D SZ_1G; - } - - /* - * Map/unmap the last largest supported page of the IAS, this can - * trigger corner cases in the concatednated page tables. - */ - mapped =3D 0; - size =3D 1UL << __fls(cfg->pgsize_bitmap); - iova =3D (1UL << cfg->ias) - size; - if (ops->map_pages(ops, iova, iova, size, 1, - IOMMU_READ | IOMMU_WRITE | - IOMMU_NOEXEC | IOMMU_CACHE, - GFP_KERNEL, &mapped)) - return __FAIL(ops, i); - if (mapped !=3D size) - return __FAIL(ops, i); - if (ops->unmap_pages(ops, iova, size, 1, NULL) !=3D size) - return __FAIL(ops, i); - - free_io_pgtable_ops(ops); - } - - return 0; -} - -static int __init arm_lpae_do_selftests(void) -{ - static const unsigned long pgsize[] __initconst =3D { - SZ_4K | SZ_2M | SZ_1G, - SZ_16K | SZ_32M, - SZ_64K | SZ_512M, - }; - - static const unsigned int address_size[] __initconst =3D { - 32, 36, 40, 42, 44, 48, - }; - - int i, j, k, pass =3D 0, fail =3D 0; - struct faux_device *dev; - struct io_pgtable_cfg cfg =3D { - .tlb =3D &dummy_tlb_ops, - .coherent_walk =3D true, - .quirks =3D IO_PGTABLE_QUIRK_NO_WARN, - }; - - dev =3D faux_device_create("io-pgtable-test", NULL, 0); - if (!dev) - return -ENOMEM; - - cfg.iommu_dev =3D &dev->dev; - - for (i =3D 0; i < ARRAY_SIZE(pgsize); ++i) { - for (j =3D 0; j < ARRAY_SIZE(address_size); ++j) { - /* Don't use ias > oas as it is not valid for stage-2. */ - for (k =3D 0; k <=3D j; ++k) { - cfg.pgsize_bitmap =3D pgsize[i]; - cfg.ias =3D address_size[k]; - cfg.oas =3D address_size[j]; - pr_info("selftest: pgsize_bitmap 0x%08lx, IAS %u OAS %u\n", - pgsize[i], cfg.ias, cfg.oas); - if (arm_lpae_run_tests(&cfg)) - fail++; - else - pass++; - } - } - } - - pr_info("selftest: completed with %d PASS %d FAIL\n", pass, fail); - faux_device_destroy(dev); - - return fail ? -EFAULT : 0; -} -subsys_initcall(arm_lpae_do_selftests); -#endif diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index ba7cfdf7afa0..a06a23543cff 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -2,6 +2,8 @@ #ifndef IO_PGTABLE_ARM_H_ #define IO_PGTABLE_ARM_H_ =20 +#include + #define ARM_LPAE_TCR_TG0_4K 0 #define ARM_LPAE_TCR_TG0_64K 1 #define ARM_LPAE_TCR_TG0_16K 2 @@ -27,4 +29,43 @@ #define ARM_LPAE_TCR_PS_48_BIT 0x5ULL #define ARM_LPAE_TCR_PS_52_BIT 0x6ULL =20 +/* Struct accessors */ +#define io_pgtable_to_data(x) \ + container_of((x), struct arm_lpae_io_pgtable, iop) + +#define io_pgtable_ops_to_data(x) \ + io_pgtable_to_data(io_pgtable_ops_to_pgtable(x)) + +struct arm_lpae_io_pgtable { + struct io_pgtable iop; + + int pgd_bits; + int start_level; + int bits_per_level; + + void *pgd; +}; + +#define ARM_LPAE_MAX_ADDR_BITS 52 +#define ARM_LPAE_S2_MAX_CONCAT_PAGES 16 +#define ARM_LPAE_MAX_LEVELS 4 + +/* + * Calculate the right shift amount to get to the portion describing level= l + * in a virtual address mapped by the pagetable in d. + */ +#define ARM_LPAE_LVL_SHIFT(l,d) \ + (((ARM_LPAE_MAX_LEVELS - (l)) * (d)->bits_per_level) + \ + ilog2(sizeof(arm_lpae_iopte))) + +#define ARM_LPAE_GRANULE(d) \ + (sizeof(arm_lpae_iopte) << (d)->bits_per_level) +#define ARM_LPAE_PGD_SIZE(d) \ + (sizeof(arm_lpae_iopte) << (d)->pgd_bits) + +#define ARM_LPAE_PTES_PER_TABLE(d) \ + (ARM_LPAE_GRANULE(d) >> ilog2(sizeof(arm_lpae_iopte))) + +typedef u64 arm_lpae_iopte; + #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97B40342CB5 for ; Tue, 19 Aug 2025 21:52:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640351; cv=none; b=Iufd8jPxFgoYF2OnzsG7nz878R38B70t7VvZcrSxG9Ogb/LQtzObRvr7Ahy5tmydKj+uIHqiWAoGKMcbL1tBU74qWVwnlBzYTWVHG98TTS7SD9UzyUUnJ63zoSeLT5bdGR9/QgfXA/+D+jsttXcq3mRSLkbvC4kRGH5msjVVSGA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640351; c=relaxed/simple; bh=f1zhYYpnAEfMBFW4F8zFMkjpoIuuoFRXw/JRH6L83UM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=k7WNgNsuyUs0jD9xzr5EmX/9f5oAmPMoz7KuW9j2shN2OAQ6IJ/WwyZTLLjOlhpB9B8yJXYvNL5ILtrv17iVgSQElYKNB/1q/slcu0LJ3Eleup1BQLclI8f+jcZ0uyZqusev8EIq7r4ex9yYcC0arj9lFC0Zp/A/OwHCqV7BwaI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=stT/zhCb; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="stT/zhCb" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0b14daso21688695e9.2 for ; Tue, 19 Aug 2025 14:52:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640348; x=1756245148; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3pO2YE+QLHcZDr5ERfg64RTVx9rRsQJCpks4Jpbfd5g=; b=stT/zhCbTcJ0hrFpSYT7hJpIAbq5EtC1VXmUkuAa8MG+YxVFVOp6pA3AdtEhYw1bit AZoZRd1/N/M/e0ZoZN5oMWoyWpJ8oUk+vs4rIT0bL5pTffR/buTXW7rRymL7wv4rxrW6 FqhFR/cWpC8UbS6ialJcXM+PugcCAHw91AvZjvt1ybd7UZ2aM+2KsDAZDZ9PgsAarslL QGQ5QgCj+eOcDFpowAmae7VLBbg9me5JDTUr0dSko/D70wQ1MFz2LqAbK753fHtp9qvF wSqHM2MdauhIql1m/R7OlOKf7tbcmHo4esTwOL7850e6N6Jng2BzPBMC3/LMabusrg4/ TleQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640348; x=1756245148; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3pO2YE+QLHcZDr5ERfg64RTVx9rRsQJCpks4Jpbfd5g=; b=JXB+t9sVP3YoZW9b5ARl1pAKqaQrckgl5sVDdm9gGtJDqKKo3zYB2v/WGcEdvQiUl7 5xoRcBCxd3/rJ0KAHH3iLvFIZe3TU19ZK96lBuncNmSrQkL9szC+SwFgfgLQIpWY5dgW LEj3NJ7wPyq8aERXTF5vN0LT4KUU+Drk8ZbpNF2JmZGP8YzCAzCZs/3uF4mtTEeFTHvJ 9+DCUx65U0CwpMFAwnx9hIas7wZ0OFVZ5ea7nr34rByN8W9RhONqWGkCpFVKgKymAcMv UxN9y9kgEabNLWaEjm3cy91abM5TEfXWWhrJbWSpO63Z9tGLZwNMoizKXVcVovbGnjI9 mAJw== X-Gm-Message-State: AOJu0Ywgx2xcA8kAovVUheXSENunDN8GSobsj6Bt59AMcXWp3WzAwCJ5 s6JiGqB9UAUNHhya2Q1hV813V/+/u8MD9zTVdNJ5mgAHhzYta5cDnqudKJgg4+WPqiId4BvX0As cVJOPhZ9j0dosq4wl3d8hFTOt1CChVfBUDpL0UMNQoB59X6l5nCfkY5dKkpAuBNdhnd+jT7B0dU T7qdb0tkq4p8T3pNmWBrYDmnJj0erfjxdp3OSRhsgl2cbkLmW5Kr6kNDM= X-Google-Smtp-Source: AGHT+IGJ3zxP4tZSJCnPpnLfo8kuY9/ChMlhp2Vc3i11mcX5tsN9jlzR3HCoS4DbN8uykW/2HzhGSH0eYEjOYg== X-Received: from wmbhh7.prod.google.com ([2002:a05:600c:5307:b0:458:bfa7:dd5b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6216:b0:456:1b93:76b with SMTP id 5b1f17b1804b1-45b47b50207mr2162775e9.4.1755640348094; Tue, 19 Aug 2025 14:52:28 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:33 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-6-smostafa@google.com> Subject: [PATCH v4 05/28] iommu/io-pgtable-arm: Factor kernel specific code out From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Some of the currently used APIs are only part of the kernel and not available in the hypervisor, factor those out of the common file: - alloc/free memory - CMOs - virt/phys conversions Which is implemented by the kernel in io-pgtable-arm-kernel.c and similarly for the hypervisor later in this series. va/pa conversion kept as macros. Signed-off-by: Mostafa Saleh --- drivers/iommu/io-pgtable-arm-kernel.c | 89 ++++++++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 99 +++------------------------ drivers/iommu/io-pgtable-arm.h | 14 ++++ 3 files changed, 113 insertions(+), 89 deletions(-) diff --git a/drivers/iommu/io-pgtable-arm-kernel.c b/drivers/iommu/io-pgtab= le-arm-kernel.c index f3b869310964..d3056487b0f6 100644 --- a/drivers/iommu/io-pgtable-arm-kernel.c +++ b/drivers/iommu/io-pgtable-arm-kernel.c @@ -9,10 +9,99 @@ #define pr_fmt(fmt) "arm-lpae io-pgtable: " fmt =20 #include +#include #include #include =20 #include "io-pgtable-arm.h" +#include "iommu-pages.h" + +static dma_addr_t __arm_lpae_dma_addr(void *pages) +{ + return (dma_addr_t)virt_to_phys(pages); +} + +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct device *dev =3D cfg->iommu_dev; + size_t alloc_size; + dma_addr_t dma; + void *pages; + + /* + * For very small starting-level translation tables the HW requires a + * minimum alignment of at least 64 to cover all cases. + */ + alloc_size =3D max(size, 64); + if (cfg->alloc) + pages =3D cfg->alloc(cookie, alloc_size, gfp); + else + pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, + alloc_size); + + if (!pages) + return NULL; + + if (!cfg->coherent_walk) { + dma =3D dma_map_single(dev, pages, size, DMA_TO_DEVICE); + if (dma_mapping_error(dev, dma)) + goto out_free; + /* + * We depend on the IOMMU being able to work with any physical + * address directly, so if the DMA layer suggests otherwise by + * translating or truncating them, that bodes very badly... + */ + if (dma !=3D virt_to_phys(pages)) + goto out_unmap; + } + + return pages; + +out_unmap: + dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n"= ); + dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); + +out_free: + if (cfg->free) + cfg->free(cookie, pages, size); + else + iommu_free_pages(pages); + + return NULL; +} + +void __arm_lpae_free_pages(void *pages, size_t size, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + if (!cfg->coherent_walk) + dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages), + size, DMA_TO_DEVICE); + + if (cfg->free) + cfg->free(cookie, pages, size); + else + iommu_free_pages(pages); +} + +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg) +{ + dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep), + sizeof(*ptep) * num_entries, DMA_TO_DEVICE); +} + +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp) +{ + return kmalloc(size, gfp); +} + +void __arm_lpae_free_data(void *p) +{ + return kfree(p); +} =20 #ifdef CONFIG_IOMMU_IO_PGTABLE_LPAE_SELFTEST =20 diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 791a2c4ecb83..2ca09081c3b0 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -12,12 +12,10 @@ #include #include #include -#include =20 #include =20 #include "io-pgtable-arm.h" -#include "iommu-pages.h" =20 /* * Calculate the index at level l used to map virtual address a using the @@ -118,7 +116,7 @@ #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL =20 /* IOPTE accessors */ -#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d)) +#define iopte_deref(pte,d) __arm_lpae_phys_to_virt(iopte_to_paddr(pte, d)) =20 #define iopte_type(pte) \ (((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK) @@ -208,83 +206,6 @@ static inline bool arm_lpae_concat_mandatory(struct io= _pgtable_cfg *cfg, (data->start_level =3D=3D 1) && (oas =3D=3D 40); } =20 -static dma_addr_t __arm_lpae_dma_addr(void *pages) -{ - return (dma_addr_t)virt_to_phys(pages); -} - -static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, - struct io_pgtable_cfg *cfg, - void *cookie) -{ - struct device *dev =3D cfg->iommu_dev; - size_t alloc_size; - dma_addr_t dma; - void *pages; - - /* - * For very small starting-level translation tables the HW requires a - * minimum alignment of at least 64 to cover all cases. - */ - alloc_size =3D max(size, 64); - if (cfg->alloc) - pages =3D cfg->alloc(cookie, alloc_size, gfp); - else - pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, - alloc_size); - - if (!pages) - return NULL; - - if (!cfg->coherent_walk) { - dma =3D dma_map_single(dev, pages, size, DMA_TO_DEVICE); - if (dma_mapping_error(dev, dma)) - goto out_free; - /* - * We depend on the IOMMU being able to work with any physical - * address directly, so if the DMA layer suggests otherwise by - * translating or truncating them, that bodes very badly... - */ - if (dma !=3D virt_to_phys(pages)) - goto out_unmap; - } - - return pages; - -out_unmap: - dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n"= ); - dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); - -out_free: - if (cfg->free) - cfg->free(cookie, pages, size); - else - iommu_free_pages(pages); - - return NULL; -} - -static void __arm_lpae_free_pages(void *pages, size_t size, - struct io_pgtable_cfg *cfg, - void *cookie) -{ - if (!cfg->coherent_walk) - dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages), - size, DMA_TO_DEVICE); - - if (cfg->free) - cfg->free(cookie, pages, size); - else - iommu_free_pages(pages); -} - -static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, - struct io_pgtable_cfg *cfg) -{ - dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep), - sizeof(*ptep) * num_entries, DMA_TO_DEVICE); -} - static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_c= fg *cfg, int num_entries) { for (int i =3D 0; i < num_entries; i++) @@ -360,7 +281,7 @@ static arm_lpae_iopte arm_lpae_install_table(arm_lpae_i= opte *table, arm_lpae_iopte old, new; struct io_pgtable_cfg *cfg =3D &data->iop.cfg; =20 - new =3D paddr_to_iopte(__pa(table), data) | ARM_LPAE_PTE_TYPE_TABLE; + new =3D paddr_to_iopte(__arm_lpae_virt_to_phys(table), data) | ARM_LPAE_P= TE_TYPE_TABLE; if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS) new |=3D ARM_LPAE_PTE_NSTABLE; =20 @@ -581,7 +502,7 @@ static void arm_lpae_free_pgtable(struct io_pgtable *io= p) struct arm_lpae_io_pgtable *data =3D io_pgtable_to_data(iop); =20 __arm_lpae_free_pgtable(data, data->start_level, data->pgd); - kfree(data); + __arm_lpae_free_data(data); } =20 static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, @@ -895,7 +816,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg) if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS) return NULL; =20 - data =3D kmalloc(sizeof(*data), GFP_KERNEL); + data =3D __arm_lpae_alloc_data(sizeof(*data), GFP_KERNEL); if (!data) return NULL; =20 @@ -1018,11 +939,11 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *= cfg, void *cookie) wmb(); =20 /* TTBR */ - cfg->arm_lpae_s1_cfg.ttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s1_cfg.ttbr =3D __arm_lpae_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 @@ -1114,11 +1035,11 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg = *cfg, void *cookie) wmb(); =20 /* VTTBR */ - cfg->arm_lpae_s2_cfg.vttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s2_cfg.vttbr =3D __arm_lpae_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 @@ -1188,7 +1109,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) /* Ensure the empty pgd is visible before TRANSTAB can be written */ wmb(); =20 - cfg->arm_mali_lpae_cfg.transtab =3D virt_to_phys(data->pgd) | + cfg->arm_mali_lpae_cfg.transtab =3D __arm_lpae_virt_to_phys(data->pgd) | ARM_MALI_LPAE_TTBR_READ_INNER | ARM_MALI_LPAE_TTBR_ADRMODE_TABLE; if (cfg->coherent_walk) @@ -1197,7 +1118,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + __arm_lpae_free_data(data); return NULL; } =20 diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index a06a23543cff..7d9f0b759275 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -68,4 +68,18 @@ struct arm_lpae_io_pgtable { =20 typedef u64 arm_lpae_iopte; =20 +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg); +void __arm_lpae_free_pages(void *pages, size_t size, + struct io_pgtable_cfg *cfg, + void *cookie); +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, + void *cookie); +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp); +void __arm_lpae_free_data(void *p); +#ifndef __KVM_NVHE_HYPERVISOR__ +#define __arm_lpae_virt_to_phys __pa +#define __arm_lpae_phys_to_virt __va +#endif /* !__KVM_NVHE_HYPERVISOR__ */ #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04F5234AB03 for ; Tue, 19 Aug 2025 21:52:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640353; cv=none; b=LWx2j7HBWc16MJoImY/+EB6wXJ3g+DPU4Q2FTy2KSH7wNHQkwTcce14e/W6quOzX9mw2+DzPrzWtQoWDtQhVkvTsUYGIZuW6SFLn2lx2f8txaGrzBE9S8jOtk3ai8u+jS5HrvztWyEnumC0vErCnLeG/GcbZOVVDwegOMgVnAUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640353; c=relaxed/simple; bh=K2rLoWFHCceXuAx8BZDuqA7LSYY9eSN7nJ8Kr254ba0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jA5pSOZmvAjVbnSyAAqqoqTQmkocL+YDPFWtlEVioDFaUhNFwzJbg5EmT29dDdcscoVS+1WliRojvbFMmnQQHLI49O+tZbpZjbeft/q4B1r0Vo4PvavxjK2u3omSG3KIHNELqejuqWk6Qjmt0XMyiY324C9rXQD+PIhjr8Q+V3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZAr+llmq; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZAr+llmq" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0071c1so21301805e9.0 for ; Tue, 19 Aug 2025 14:52:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640349; x=1756245149; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+dGtbLdVT9MKEfcSMncVaN02ha99d5hmmU4GzF6sl48=; b=ZAr+llmqq6VYOv9qlSTfiS2SfefGrBuzdg/OVEljUZE16NgEWH74nMkiKaBCAWzGUy X00bjcmQ0dbpxWtj6Hzfc76QV5fCy1wZxPTzye8PoPFiMobzp+dhMXjV0J4/RtZ9HLFQ KmU0ZdGTOgv/KKjiGzl/k4Mqvx/g21ieBkwtqGOk3DK5yW4LhjLPc0DqJtM6TqF7Joiq foep1KwhSi2JWph1g3kDCWpgaAjA45Nlt68DlLIt3dZ6g8svQnZ49U83OYydm9UOyIwZ IEFv4NDxSrxtHt17X2h+I+mIx6wNUNi6G1sppTmFEqbtNUBF8cRYT80DVF6JrIfjhWDY /hoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640349; x=1756245149; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+dGtbLdVT9MKEfcSMncVaN02ha99d5hmmU4GzF6sl48=; b=EQDKrtbcj9C+rl6mjTcURBOmOwXGBDm+XP/zVBp+Xt0W4jQ5IOVe0XUtc6WrIm91eE eyfDdvmhckEY/uFHZ6jh6AI79Ko3WMM/rrPHilEeguXGkOQhO5Zmr2in0R+Pz8X+I3mL fFDBzQWCaC7e/3TX+kJHaa3TjRz072Cq3RnYnGFOvAaM4GBuTm1wggeCBXz+63XnsnBN LsapyT+MCW137KH9xJEHb5xMHl2eV+gZ8x9hQUT0sWRY4QDybwsvmrJcUFQSR6nhRZC6 DcawL+frebKW1vmLoRKxhgARK2KyGrzJnrW35LbitdaCK1xI6dtslq6vnWn+xvDFqDmd t2Tg== X-Gm-Message-State: AOJu0YzP5ASDgrjIIY5XGBQ+obPaJDXCIiOUmy9SV2tdyCsL5H7FEs25 oodH32vYbMZL+pA2PMPGwuP8wiyogj8DKE+6QCLdwrduqclYOLguCW8hCJy4spc4EJTVruxmfzP QoyKXZqKEpSwWhIRzwZPVL0z5+J3VsfuK/JNkZR7lrovq66/soyR5CNCzKG6KbgcpfqBSSa2mmY x1THpemsK2GLWVMIGQTvqABb+OhbL6QUNYCMa/zbLuB6JQSmwhtLpJ2TI= X-Google-Smtp-Source: AGHT+IEgfdf8docIf6HWBBYKV6+o57DCMa9xUPjyluTHGVKx6KOjoNpXqO6AwvMOQH1ekPl2WnYagt2sYt/zBA== X-Received: from wmtg2.prod.google.com ([2002:a05:600c:8b52:b0:459:d4b5:52cc]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4ecf:b0:456:18f3:b951 with SMTP id 5b1f17b1804b1-45b479b5818mr4007305e9.15.1755640349178; Tue, 19 Aug 2025 14:52:29 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:34 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-7-smostafa@google.com> Subject: [PATCH v4 06/28] iommu/arm-smmu-v3: Split code with hyp From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The KVM SMMUv3 driver would re-use some of the cmdq code inside the hypervisor, move these functions to a new common c file that is shared between the host kernel and the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- .../arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c | 114 ++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 146 ------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 44 ++++++ 4 files changed, 159 insertions(+), 147 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index 493a659cc66b..1918b4a64cb0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ARM_SMMU_V3) +=3D arm_smmu_v3.o -arm_smmu_v3-y :=3D arm-smmu-v3.o +arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-hyp.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c new file mode 100644 index 000000000000..62744c8548a8 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-hyp.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2015 ARM Limited + * + * Author: Will Deacon + * Arm SMMUv3 driver functions shared with hypervisor. + */ + +#include "arm-smmu-v3.h" +#include + +#include + +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) +{ + memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); + cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); + + switch (ent->opcode) { + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_NSNH_ALL: + break; + case CMDQ_OP_PREFETCH_CFG: + cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); + break; + case CMDQ_OP_CFGI_CD: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); + fallthrough; + case CMDQ_OP_CFGI_STE: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); + break; + case CMDQ_OP_CFGI_CD_ALL: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + break; + case CMDQ_OP_CFGI_ALL: + /* Cover the entire SID range */ + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); + break; + case CMDQ_OP_TLBI_NH_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + fallthrough; + case CMDQ_OP_TLBI_EL2_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; + break; + case CMDQ_OP_TLBI_S2_IPA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; + break; + case CMDQ_OP_TLBI_NH_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + fallthrough; + case CMDQ_OP_TLBI_NH_ALL: + case CMDQ_OP_TLBI_S12_VMALL: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + break; + case CMDQ_OP_TLBI_EL2_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + break; + case CMDQ_OP_ATC_INV: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); + cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; + break; + case CMDQ_OP_PRI_RESP: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); + switch (ent->pri.resp) { + case PRI_RESP_DENY: + case PRI_RESP_FAIL: + case PRI_RESP_SUCC: + break; + default: + return -EINVAL; + } + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); + break; + case CMDQ_OP_RESUME: + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); + cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); + break; + case CMDQ_OP_CMD_SYNC: + if (ent->sync.msiaddr) { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); + cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; + } else { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + } + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); + break; + default: + return -ENOENT; + } + + return 0; +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 10cc6dc26b7b..1f765b4e36fa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -138,18 +138,6 @@ static bool queue_has_space(struct arm_smmu_ll_queue *= q, u32 n) return space >=3D n; } =20 -static bool queue_full(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); -} - -static bool queue_empty(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); -} - static bool queue_consumed(struct arm_smmu_ll_queue *q, u32 prod) { return ((Q_WRP(q, q->cons) =3D=3D Q_WRP(q, prod)) && @@ -168,12 +156,6 @@ static void queue_sync_cons_out(struct arm_smmu_queue = *q) writel_relaxed(q->llq.cons, q->cons_reg); } =20 -static void queue_inc_cons(struct arm_smmu_ll_queue *q) -{ - u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; - q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); -} - static void queue_sync_cons_ovf(struct arm_smmu_queue *q) { struct arm_smmu_ll_queue *llq =3D &q->llq; @@ -205,12 +187,6 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q) return ret; } =20 -static u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) -{ - u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; - return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); -} - static void queue_poll_init(struct arm_smmu_device *smmu, struct arm_smmu_queue_poll *qp) { @@ -238,14 +214,6 @@ static int queue_poll(struct arm_smmu_queue_poll *qp) return 0; } =20 -static void queue_write(__le64 *dst, u64 *src, size_t n_dwords) -{ - int i; - - for (i =3D 0; i < n_dwords; ++i) - *dst++ =3D cpu_to_le64(*src++); -} - static void queue_read(u64 *dst, __le64 *src, size_t n_dwords) { int i; @@ -266,108 +234,6 @@ static int queue_remove_raw(struct arm_smmu_queue *q,= u64 *ent) } =20 /* High-level queue accessors */ -static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) -{ - memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); - cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); - - switch (ent->opcode) { - case CMDQ_OP_TLBI_EL2_ALL: - case CMDQ_OP_TLBI_NSNH_ALL: - break; - case CMDQ_OP_PREFETCH_CFG: - cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); - break; - case CMDQ_OP_CFGI_CD: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); - fallthrough; - case CMDQ_OP_CFGI_STE: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); - break; - case CMDQ_OP_CFGI_CD_ALL: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - break; - case CMDQ_OP_CFGI_ALL: - /* Cover the entire SID range */ - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); - break; - case CMDQ_OP_TLBI_NH_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - fallthrough; - case CMDQ_OP_TLBI_EL2_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; - break; - case CMDQ_OP_TLBI_S2_IPA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; - break; - case CMDQ_OP_TLBI_NH_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - fallthrough; - case CMDQ_OP_TLBI_NH_ALL: - case CMDQ_OP_TLBI_S12_VMALL: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - break; - case CMDQ_OP_TLBI_EL2_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - break; - case CMDQ_OP_ATC_INV: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); - cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; - break; - case CMDQ_OP_PRI_RESP: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); - switch (ent->pri.resp) { - case PRI_RESP_DENY: - case PRI_RESP_FAIL: - case PRI_RESP_SUCC: - break; - default: - return -EINVAL; - } - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); - break; - case CMDQ_OP_RESUME: - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); - cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); - break; - case CMDQ_OP_CMD_SYNC: - if (ent->sync.msiaddr) { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); - cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; - } else { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); - } - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); - break; - default: - return -ENOENT; - } - - return 0; -} - static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smm= u, struct arm_smmu_cmdq_ent *ent) { @@ -1508,18 +1374,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_= master *master) } =20 /* Stream table manipulation functions */ -static void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst, - dma_addr_t l2ptr_dma) -{ - u64 val =3D 0; - - val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); - val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - - /* The HW has 64 bit atomicity with stores to the L2 STE table */ - WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); -} - struct arm_smmu_ste_writer { struct arm_smmu_entry_writer writer; u32 sid; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ea41d790463e..2698438cd35c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -997,6 +997,50 @@ void arm_smmu_install_ste_for_dev(struct arm_smmu_mast= er *master, int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq, u64 *cmds, int n, bool sync); +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent); + +/* Queue functions shared between kernel and hyp. */ +static inline bool queue_full(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); +} +static inline bool queue_empty(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); +} +static inline u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) +{ + u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; + return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); +} + +static inline void queue_inc_cons(struct arm_smmu_ll_queue *q) +{ + u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; + q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); +} + + +static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords) +{ + int i; + for (i =3D 0; i < n_dwords; ++i) + *dst++ =3D cpu_to_le64(*src++); +} + +static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1= *dst, + dma_addr_t l2ptr_dma) +{ + u64 val =3D 0; + + val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); + val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; + + /* The HW has 64 bit atomicity with stores to the L2 STE table */ + WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); +} =20 #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11F7835082A for ; Tue, 19 Aug 2025 21:52:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640354; cv=none; b=hBXPvIVUYiOpcHf0/70FtlFXhfJv048BSTr3H8HejGNiJkcESNZigEI6Q7eJbp/7AzPCxQu3UpnkwGpzo3oDD4RHDfBrEwNx3/Gle7zcE4880Z1mtGRb7q7wPRcH5Ri5cJZFcln16PDIkRyijj3RnVqH2/1tER9pv6iS+loQuBc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640354; c=relaxed/simple; bh=3TG4Z5+XXX9K3RELM+h/fWcPxGfI20T9a5XvTmfpW3Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ozjhRdi/Q9+MTMPlVEofmQ7/rIKHQw6NYs39LTr3uVuXfP/lSTheKEZ38HPqHa4sWlJX4vANUZ2S0VGxxpNG/WpA1COXCUHNZwvMRl4dEdjEYP9UFh0BCAIMEyVd9OIvZ1+Jqhy8FHZIiPESBC0gPIdB9uTZxNkq3x+ffefwyGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hBjO66Eb; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hBjO66Eb" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3b9edf5b346so3938827f8f.3 for ; Tue, 19 Aug 2025 14:52:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640350; x=1756245150; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1QLGehsG9TWs7vYu5jUxb2yk8sP6DWLxxs21XC/HL6E=; b=hBjO66EbDSo1L7ECIEMtOWrwtv46zJUsKiVZz7shU9vgVOnB++IT999thRlFoN9b9H 3dzroUJsp5J5mkqrcj11ZtgO1t+y8y4LESSKLrlbqCAPHS+X/Ed9GNDysIiWhV+yFlv5 EGsX7P973iKJwjXA4HhSYZ8QeaYpMow1Z0H6SV3oglyQ+i/Z37fLkNRBVAXzjCXVns40 QUeWBYYFx3+7bbZgsag0I1pdX6KM02f8FIlqgBLqq5/qhj3HjGjyfF7J/PpAanPaAp3+ gpaVRxwHHIsknYgDEUQEBDaBW+iMBZPQ8msdqdneVeTjCKDfFW2nqfP+I0eQDrWbj5EZ uJ9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640350; x=1756245150; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1QLGehsG9TWs7vYu5jUxb2yk8sP6DWLxxs21XC/HL6E=; b=u/hddBp2Ml/gbxaDm7A4TWpo+iOcDeFIICVmOUZJN3rr9gYxPofg985dkggHkVGmfv qrOq4gs9nZf1y5b9MPrRpwdz44iQmZL6vkk6YbXMwpNUUf4iz+ob7mA+qxgi39Dhwt37 Du9igMlMdY+V8dEbBzqWYIUNvbocxKhM5j8pnvwbgkPoh+34hKtK2lMrmGoBXaxZs6RD JM5b+aQJ1a7kANM6zASIY9BVwun6wnwZbwgRWQ0LiPq8BBvYU30kslF/3UYGyvhbG6vM AGIve9GPQK70c5361I0hQv4SNHqyQx6R69fh6M5osBDbhslsZnJUg0BwM9DOnfa/yy5a FxlQ== X-Gm-Message-State: AOJu0YxM0ZvcKcuUU0jFFtAgn/QzVMJpTcrTmM4l84bCezahxAfOubGm h7WgCri2wSmuaHs+F5ysVLQnKWAlnz1QbwdVjE7ODfVQn/Lljq20XO/NNEyODB8pzIP153DyElq iH6MU0Ltim3cD6VRijYpwVcVktaAiYbukK8qj65bSvLt1VjJy52dT4VUPRGPriaLoe+1RRg49AL 1Ikin//3PFgOzayqLoc/hOr8w/07jTsnZqrczvQRsU7l7J+95cmLpZ0xY= X-Google-Smtp-Source: AGHT+IEyv3kVrYIlmgHeAnLq3AxRjEMrVoo+46FdGKld4xfeBpJx0bEncQb2Yqr4jiUVqXM4bZQzPz6PEoFmag== X-Received: from wrvb7.prod.google.com ([2002:a5d:5507:0:b0:3b9:c5f:d58b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5f52:0:b0:3b7:994b:840f with SMTP id ffacd0b85a97d-3c32eccdad0mr317497f8f.57.1755640350354; Tue, 19 Aug 2025 14:52:30 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:35 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-8-smostafa@google.com> Subject: [PATCH v4 07/28] iommu/arm-smmu-v3: Move TLB range invalidation into a macro From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Range TLB invalidation has a very specific algorithm, instead of re-writing it for the hypervisor, put it in a macro so it can be re-used. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 59 +------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 64 +++++++++++++++++++++ 2 files changed, 67 insertions(+), 56 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 1f765b4e36fa..41820a9180f4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2126,68 +2126,15 @@ static void __arm_smmu_tlb_inv_range(struct arm_smm= u_cmdq_ent *cmd, struct arm_smmu_domain *smmu_domain) { struct arm_smmu_device *smmu =3D smmu_domain->smmu; - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D 0; - size_t inv_range =3D granule; struct arm_smmu_cmdq_batch cmds; =20 if (!size) return; =20 - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* Get the leaf page size */ - tg =3D __ffs(smmu_domain->domain.pgsize_bitmap); - - num_pages =3D size >> tg; - - /* Convert page size of 12,14,16 (log2) to 1,2,3 */ - cmd->tlbi.tg =3D (tg - 10) / 2; - - /* - * Determine what level the granule is at. For non-leaf, both - * io-pgtable and SVA pass a nominal last-level granule because - * they don't know what level(s) actually apply, so ignore that - * and leave TTL=3D0. However for various errata reasons we still - * want to use a range command, so avoid the SVA corner case - * where both scale and num could be 0 as well. - */ - if (cmd->tlbi.leaf) - cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); - else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) - num_pages++; - } - arm_smmu_cmdq_batch_init(smmu, &cmds, cmd); - - while (iova < end) { - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* - * On each iteration of the loop, the range is 5 bits - * worth of the aligned size remaining. - * The range in pages is: - * - * range =3D (num_pages & (0x1f << __ffs(num_pages))) - */ - unsigned long scale, num; - - /* Determine the power of 2 multiple number of pages */ - scale =3D __ffs(num_pages); - cmd->tlbi.scale =3D scale; - - /* Determine how many chunks of 2^scale size we have */ - num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; - cmd->tlbi.num =3D num - 1; - - /* range is num * 2^scale * pgsize */ - inv_range =3D num << (scale + tg); - - /* Clear out the lower order bits for the next iteration */ - num_pages -=3D num << scale; - } - - cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, &cmds, cmd); - iova +=3D inv_range; - } + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + smmu_domain->domain.pgsize_bitmap, + smmu, arm_smmu_cmdq_batch_add, &cmds); arm_smmu_cmdq_batch_submit(smmu, &cmds); } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 2698438cd35c..a222fb7ef2ec 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1042,6 +1042,70 @@ static inline void arm_smmu_write_strtab_l1_desc(str= uct arm_smmu_strtab_l1 *dst, WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); } =20 +/** + * arm_smmu_tlb_inv_build - Create a range invalidation command + * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid. + * @iova: Start IOVA to invalidate + * @size: Size of range + * @granule: Granule of invalidation + * @pgsize_bitmap: Page size bit map of the page table. + * @smmu: Struct for the smmu, must have ::features + * @add_cmd: Function to send/batch the invalidation command + * @cmds: Incase of batching, it includes the pointer to the batch + */ +#define arm_smmu_tlb_inv_build(cmd, iova, size, granule, pgsize_bitmap, sm= mu, add_cmd, cmds) \ +{ \ + unsigned long _iova =3D (iova); \ + size_t _size =3D (size); \ + size_t _granule =3D (granule); \ + unsigned long end =3D _iova + _size, num_pages =3D 0, tg =3D 0; \ + size_t inv_range =3D _granule; \ + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \ + /* Get the leaf page size */ \ + tg =3D __ffs(pgsize_bitmap); \ + num_pages =3D _size >> tg; \ + /* Convert page size of 12,14,16 (log2) to 1,2,3 */ \ + cmd->tlbi.tg =3D (tg - 10) / 2; \ + /* + * Determine what level the granule is at. For non-leaf, both + * io-pgtable and SVA pass a nominal last-level granule because + * they don't know what level(s) actually apply, so ignore that + * and leave TTL=3D0. However for various errata reasons we still + * want to use a range command, so avoid the SVA corner case + * where both scale and num could be 0 as well. + */ \ + if (cmd->tlbi.leaf) \ + cmd->tlbi.ttl =3D 4 - ((ilog2(_granule) - 3) / (tg - 3)); \ + else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) \ + num_pages++; \ + } \ + while (_iova < end) { \ + if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { \ + /* + * On each iteration of the loop, the range is 5 bits + * worth of the aligned size remaining. + * The range in pages is: + * + * range =3D (num_pages & (0x1f << __ffs(num_pages))) + */ \ + unsigned long scale, num; \ + /* Determine the power of 2 multiple number of pages */ \ + scale =3D __ffs(num_pages); \ + cmd->tlbi.scale =3D scale; \ + /* Determine how many chunks of 2^scale size we have */ \ + num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; \ + cmd->tlbi.num =3D num - 1; \ + /* range is num * 2^scale * pgsize */ \ + inv_range =3D num << (scale + tg); \ + /* Clear out the lower order bits for the next iteration */ \ + num_pages -=3D num << scale; \ + } \ + cmd->tlbi.addr =3D _iova; \ + add_cmd(smmu, cmds, cmd); \ + _iova +=3D inv_range; \ + } \ +} \ + #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); void arm_smmu_sva_notifier_synchronize(void); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4493B350843 for ; Tue, 19 Aug 2025 21:52:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640356; cv=none; b=guof7T5LxM0rephTd2ey1eFCxvkjdwHP6V3WnbRDgZ6Hx0PKnSfdUPSKx+bop4mMFyxpw89tAVbEbYaZ8W2xElcYwOerbP9NokhAVvXLLCKVTFpG+KUgX39Z/IKIxY5X8PJNivl4syUbOc0pO8Chq5bZIUDXp1JYu47HFxcQzHk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640356; c=relaxed/simple; bh=gY7xXY2zsrLp/8by2WZLyKzu8vPZW0iFJI4dE5BSgNE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KreCWvRKBbKg356E9Nt8qGf0JQEO4X+Xy4AOCsDqc4Tw3yScCZbTGGPU+CbD0mqXNK7Ru2wXoqHFGhgN35jiZt4EJ3KfEhBxuDKa+8SHrMOtMHMkVu+UKIgXIU9uiDJ3gwyFl7fNwC3gPea8aMnz5HkX/x5InD27UzyYAttfpeE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=W5+Bx7eD; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="W5+Bx7eD" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-afcb7a4132eso406015566b.3 for ; Tue, 19 Aug 2025 14:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640351; x=1756245151; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=aD+G81Z9oxnwqUIpqodF/OdO7i2K2AUH5qGZ1B3RDBI=; b=W5+Bx7eDxPSgfEttfRGNfHR/ZJdyKRqGT662sYEhVKIXh38yOjBQTLaIDCmoPgOza6 fTlHhqyBg0k+mep7j1Ort5bnavUwbMZi/WkVdLl3XxyzLfjFj28ZFBmOU8mCZT0RQ+Yl j2KGKutkNv4v3jDIWvAigBCbeK3iU54ZVyGeW2K5KX3wgEQb5IvG2EWrTwzgF2rDHMsH dfqjciRMVJ83Am6R1KfsZ95xEkQptq3yBAcGK17Dl506qo7SjHze4BMmUC0T4oSPezh+ 0+JsxY39aOzEKhgR2bvU29GPc4qXShw5fxPwMlji+N5xT+F1lDLU2qQbc2dP64Jr7FE7 vyKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640351; x=1756245151; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aD+G81Z9oxnwqUIpqodF/OdO7i2K2AUH5qGZ1B3RDBI=; b=q1N1PcDGxl0eNxV+aAw9z4+maB37b90jvBMuUfOTkbh52q7eXvB1ioMsqeW8h5rRft /w+PEhXD89Gyl/ilIdBja9clJh46rtAprGiTuIG8pEKzH1qY3dmjinNSE32vyJawNmxx 2dqvT946HgMVAgo6xbMKqp41UxnQeudaRdXoJDpGeS7EjUuQHJBf3YKhxtI8wvpTLU5Y +X/qmM4vMx/4zKA4+dLCXDKmMmCT11y+iJZ2rJNHeRUgvwj72iPCXNLONuXrRWcc8WR4 yaF+PtXcGdhhnWkMYOV8Q7bD70sT9veQx5R9+GOQ6Nj4xrF6i8vjaefnXpcFrXdejowI c8yw== X-Gm-Message-State: AOJu0YzY4UzKQIcMVWKkoBwSVq8aYIfXnWu5wC+YMQqBzdQeIlYbhYOL gYcu9WezGKWbQmNAyE3uy49oW+TiJRjTyAcBq8rUfgpzB4NG3LQZgeNMGoLy/NbdSHaZUFK0lt3 7nZYtcWAX9GXSQaDh9+bZGvG5PJZP5VE360K+7/7RqtoE533B/37rPFjpwCs2JZTEUWwBTXAKN0 ntUCxyPEW2rryfuca6xWb1KN/wTBa5lhTzuPgTUWnd+ti6QmllBcaIvCc= X-Google-Smtp-Source: AGHT+IER6S3IHefQc1j2cX6131Dk1oXUA4oF1GAkycdxrs/phTvQWDjEc8Kuq1XUg+nr2Cpuhr0CZkidIrgNsA== X-Received: from ejcvh5.prod.google.com ([2002:a17:907:d385:b0:ae0:b29b:51a2]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:906:8459:b0:af9:a2cb:f1a1 with SMTP id a640c23a62f3a-afdf01a8374mr29365866b.38.1755640351521; Tue, 19 Aug 2025 14:52:31 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:36 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-9-smostafa@google.com> Subject: [PATCH v4 08/28] iommu/arm-smmu-v3: Move IDR parsing to common functions From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move parsing of IDRs to functions so that it can be re-used from the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 112 +++----------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 111 +++++++++++++++++++ 2 files changed, 126 insertions(+), 97 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 41820a9180f4..10ca07c6dbe9 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4112,57 +4112,17 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) /* IDR0 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); =20 - /* 2-level structures */ - if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; - - if (reg & IDR0_CD2L) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; - - /* - * Translation table endianness. - * We currently require the same endianness as the CPU, but this - * could be changed later by adding a new IO_PGTABLE_QUIRK. - */ - switch (FIELD_GET(IDR0_TTENDIAN, reg)) { - case IDR0_TTENDIAN_MIXED: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; - break; -#ifdef __BIG_ENDIAN - case IDR0_TTENDIAN_BE: - smmu->features |=3D ARM_SMMU_FEAT_TT_BE; - break; -#else - case IDR0_TTENDIAN_LE: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE; - break; -#endif - default: + smmu->features |=3D smmu_idr0_features(reg); + if (FIELD_GET(IDR0_TTENDIAN, reg) =3D=3D IDR0_TTENDIAN_RESERVED) { dev_err(smmu->dev, "unknown/unsupported TT endianness!\n"); return -ENXIO; } - - /* Boolean feature flags */ - if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) - smmu->features |=3D ARM_SMMU_FEAT_PRI; - - if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) - smmu->features |=3D ARM_SMMU_FEAT_ATS; - - if (reg & IDR0_SEV) - smmu->features |=3D ARM_SMMU_FEAT_SEV; - - if (reg & IDR0_MSI) { - smmu->features |=3D ARM_SMMU_FEAT_MSI; - if (coherent && !disable_msipolling) - smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; - } - - if (reg & IDR0_HYP) { - smmu->features |=3D ARM_SMMU_FEAT_HYP; - if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) - smmu->features |=3D ARM_SMMU_FEAT_E2H; - } + if (coherent && !disable_msipolling && + smmu->features & ARM_SMMU_FEAT_MSI) + smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; + if (smmu->features & ARM_SMMU_FEAT_HYP && + cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + smmu->features |=3D ARM_SMMU_FEAT_E2H; =20 arm_smmu_get_httu(smmu, reg); =20 @@ -4174,21 +4134,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n", str_true_false(coherent)); =20 - switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { - case IDR0_STALL_MODEL_FORCE: - smmu->features |=3D ARM_SMMU_FEAT_STALL_FORCE; - fallthrough; - case IDR0_STALL_MODEL_STALL: - smmu->features |=3D ARM_SMMU_FEAT_STALLS; - } - - if (reg & IDR0_S1P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S1; - - if (reg & IDR0_S2P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S2; - - if (!(reg & (IDR0_S1P | IDR0_S2P))) { + if (!(smmu->features & (ARM_SMMU_FEAT_TRANS_S1 | ARM_SMMU_FEAT_TRANS_S2))= ) { dev_err(smmu->dev, "no translation support!\n"); return -ENXIO; } @@ -4253,10 +4199,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) =20 /* IDR3 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); - if (FIELD_GET(IDR3_RIL, reg)) - smmu->features |=3D ARM_SMMU_FEAT_RANGE_INV; - if (FIELD_GET(IDR3_FWB, reg)) - smmu->features |=3D ARM_SMMU_FEAT_S2FWB; + smmu->features |=3D smmu_idr3_features(reg); =20 /* IDR5 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); @@ -4265,43 +4208,18 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) smmu->evtq.max_stalls =3D FIELD_GET(IDR5_STALL_MAX, reg); =20 /* Page sizes */ - if (reg & IDR5_GRAN64K) - smmu->pgsize_bitmap |=3D SZ_64K | SZ_512M; - if (reg & IDR5_GRAN16K) - smmu->pgsize_bitmap |=3D SZ_16K | SZ_32M; - if (reg & IDR5_GRAN4K) - smmu->pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); =20 /* Input address size */ if (FIELD_GET(IDR5_VAX, reg) =3D=3D IDR5_VAX_52_BIT) smmu->features |=3D ARM_SMMU_FEAT_VAX; =20 - /* Output address size */ - switch (FIELD_GET(IDR5_OAS, reg)) { - case IDR5_OAS_32_BIT: - smmu->oas =3D 32; - break; - case IDR5_OAS_36_BIT: - smmu->oas =3D 36; - break; - case IDR5_OAS_40_BIT: - smmu->oas =3D 40; - break; - case IDR5_OAS_42_BIT: - smmu->oas =3D 42; - break; - case IDR5_OAS_44_BIT: - smmu->oas =3D 44; - break; - case IDR5_OAS_52_BIT: - smmu->oas =3D 52; + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) smmu->pgsize_bitmap |=3D 1ULL << 42; /* 4TB */ - break; - default: + else if (!smmu->oas) { dev_info(smmu->dev, - "unknown output address size. Truncating to 48-bit\n"); - fallthrough; - case IDR5_OAS_48_BIT: + "unknown output address size. Truncating to 48-bit\n"); smmu->oas =3D 48; } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index a222fb7ef2ec..8ffcc2e32474 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -26,6 +26,7 @@ struct arm_smmu_device; #define IDR0_STALL_MODEL_FORCE 2 #define IDR0_TTENDIAN GENMASK(22, 21) #define IDR0_TTENDIAN_MIXED 0 +#define IDR0_TTENDIAN_RESERVED 1 #define IDR0_TTENDIAN_LE 2 #define IDR0_TTENDIAN_BE 3 #define IDR0_CD2L (1 << 19) @@ -1042,6 +1043,116 @@ static inline void arm_smmu_write_strtab_l1_desc(st= ruct arm_smmu_strtab_l1 *dst, WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); } =20 +static inline u32 smmu_idr0_features(u32 reg) +{ + u32 features =3D 0; + + /* 2-level structures */ + if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) + features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; + + if (reg & IDR0_CD2L) + features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; + + /* + * Translation table endianness. + * We currently require the same endianness as the CPU, but this + * could be changed later by adding a new IO_PGTABLE_QUIRK. + */ + switch (FIELD_GET(IDR0_TTENDIAN, reg)) { + case IDR0_TTENDIAN_MIXED: + features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; + break; +#ifdef __BIG_ENDIAN + case IDR0_TTENDIAN_BE: + features |=3D ARM_SMMU_FEAT_TT_BE; + break; +#else + case IDR0_TTENDIAN_LE: + features |=3D ARM_SMMU_FEAT_TT_LE; + break; +#endif + } + + /* Boolean feature flags */ + if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) + features |=3D ARM_SMMU_FEAT_PRI; + + if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) + features |=3D ARM_SMMU_FEAT_ATS; + + if (reg & IDR0_SEV) + features |=3D ARM_SMMU_FEAT_SEV; + + if (reg & IDR0_MSI) + features |=3D ARM_SMMU_FEAT_MSI; + + if (reg & IDR0_HYP) + features |=3D ARM_SMMU_FEAT_HYP; + + switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { + case IDR0_STALL_MODEL_FORCE: + features |=3D ARM_SMMU_FEAT_STALL_FORCE; + fallthrough; + case IDR0_STALL_MODEL_STALL: + features |=3D ARM_SMMU_FEAT_STALLS; + } + + if (reg & IDR0_S1P) + features |=3D ARM_SMMU_FEAT_TRANS_S1; + + if (reg & IDR0_S2P) + features |=3D ARM_SMMU_FEAT_TRANS_S2; + + return features; +} + +static inline u32 smmu_idr3_features(u32 reg) +{ + u32 features =3D 0; + + if (FIELD_GET(IDR3_RIL, reg)) + features |=3D ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_FWB, reg)) + features |=3D ARM_SMMU_FEAT_S2FWB; + + return features; +} + +static inline u32 smmu_idr5_to_oas(u32 reg) +{ + switch (FIELD_GET(IDR5_OAS, reg)) { + case IDR5_OAS_32_BIT: + return 32; + case IDR5_OAS_36_BIT: + return 36; + case IDR5_OAS_40_BIT: + return 40; + case IDR5_OAS_42_BIT: + return 42; + case IDR5_OAS_44_BIT: + return 44; + case IDR5_OAS_48_BIT: + return 48; + case IDR5_OAS_52_BIT: + return 52; + } + return 0; +} + +static inline unsigned long smmu_idr5_to_pgsize(u32 reg) +{ + unsigned long pgsize_bitmap =3D 0; + + if (reg & IDR5_GRAN64K) + pgsize_bitmap |=3D SZ_64K | SZ_512M; + if (reg & IDR5_GRAN16K) + pgsize_bitmap |=3D SZ_16K | SZ_32M; + if (reg & IDR5_GRAN4K) + pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + return pgsize_bitmap; +} + /** * arm_smmu_tlb_inv_build - Create a range invalidation command * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid. --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90D83322C79 for ; Tue, 19 Aug 2025 21:52:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640356; cv=none; b=MIAPfiUYZnS/HaxTx97io8ty0aUR5bG052ZR668pUyBssDx9RV5KrNHxm+1MkkluvMXPxMpiL7bWwz9bfGK4zemPQILNlHi2mInK8jCeyNLmgW+AT7Z40qHz2FCIb8sjuoqi1sWDncRnhOfrB8V1qIdoR+8Daos0l5l5lCc6ZBM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640356; c=relaxed/simple; bh=OOUKzEIINv62+Nr69ONVix62Znh1FqiJp+D0GMg+n1Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kaMssaDd7NhiyFDGNFJgbhPQ9JdvP43b6oFfH1hYdIqAleRa2WcIReR2Y9sgcBALgfEMPyiP84iUx7jEaMA+5BlPT2s2SH6UG/Ir0jIzJRCkhBnaJ0g90KqFsY+HG2Nt2zfB8CYBI8+ntbyAwtkwXHlh0yMDqEXBN27DYe/z4YQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sWVraoJP; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sWVraoJP" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-afcb7aa89a3so492715866b.3 for ; Tue, 19 Aug 2025 14:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640353; x=1756245153; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uZD73D+JnG1A66Ldv7Bw9STsd96Djsc0CQybPNL1eTI=; b=sWVraoJPCH4YaP0F6nUeNkwrkU67+tAOJPQGKoUi+tpxLjBVruaU+YvXHkQC+vQbUy E8+Vc8xliBcUdxBVgD2xYeJmUxoZyqgf0f3KIiKmfWAvFBc4wZCIP2jk/G3ABv2h8WbK lhGF8pYsEotz3XLl4nbKNB//wajpJRtjQ9BekdZLbKjL9xAlb5WcevwaNebZTVigbW9N MOfhrlBuatE1l9/edn/VTAML5+ptSndvKYVg9hgSvXDpXpOn77v/7v5rUymVVCVvVSVu 9ij8eWkSENTr6ClAkdPuVJYin+/kFItVoUAxr+I87MfZCiGB1JgSTw2kJm4r70nVkt60 CZlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640353; x=1756245153; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uZD73D+JnG1A66Ldv7Bw9STsd96Djsc0CQybPNL1eTI=; b=Y9vRF5B51EhUQ1feknbrlrpQUvKaFu1z1mBwEmFRrMw5FesM69+iAyAi6i/pXCvGCb EFijVK16zfZdWPAQ2ev/fZzIam7ZajcPSCc9In6fJF4xHhX/I2B1HnZq1+KJ+NeHk5os GzggNB0175xMJI/eZKsppgh+Fm+o/vpHyycsm3PZ8UzkDJs4tXfn7dM0FxLDxVjwaA8m 08aI2KHoMvuENWLNhZr2Hz0ttGGSxwQecYIuEQ4otW9UzYjth+c7ydwRqsE8MwXpoeHM fVptOwSF/e/12XSkdStPZokYLAEQzc3I0hVLNO4mrm1Q4HQmzHFHDFeGZjrJt1m7Onr2 2Jww== X-Gm-Message-State: AOJu0Yy9jKxVyE7SViYZ6Tqr5BAQnRtJzzf7Zy2twpKCHkYKVkL6eW28 7RPW80PNlFCKxjrQsR3xz91IDGR32W0H5W8QREPtwshPxb3HAihA9/EYQR56ZZnVwReqItJzmm+ eaOHrZ/r21f85hdcJXlYKrWEF3VEsi7v0jd/HIUbwd7zhv+B1sHA8SGr/jTc87bWSsdb9LGVqNp et1gGavGudhcGEI5MR5rmn64IquUrckU50o113V65LRj7BpSuuSMP0QN8= X-Google-Smtp-Source: AGHT+IFhsc1uRSTsRRSTrrAPTUS/dsYG6n2qlV2ToNbhyhl/dxgu7ubhyCt6NPAyXCFtBe4NKVRTeRDOa40fsg== X-Received: from edo2.prod.google.com ([2002:a05:6402:52c2:b0:618:7f6:4f8a]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:9621:b0:afa:f1:ff98 with SMTP id a640c23a62f3a-afdf01bfe1amr42142466b.37.1755640352647; Tue, 19 Aug 2025 14:52:32 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:37 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-10-smostafa@google.com> Subject: [PATCH v4 09/28] KVM: arm64: iommu: Introduce IOMMU driver infrastructure From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To establish DMA isolation, KVM needs an IOMMU driver which provide ops implemented at EL2. Only one driver can be used and is registered with kvm_iommu_register_driver() by passing pointer to the ops. This must be called before module_init() which is the point KVM initializes. Signed-off-by: Mostafa Saleh Signed-off-by: Jean-Philippe Brucker --- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/Makefile | 3 ++- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 18 ++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 5 +++++ arch/arm64/kvm/iommu.c | 15 +++++++++++++++ 7 files changed, 57 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/iommu.h create mode 100644 arch/arm64/kvm/hyp/nvhe/iommu/iommu.c create mode 100644 arch/arm64/kvm/iommu.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 3e41a880b062..1a08066eaf7e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1674,5 +1674,7 @@ void compute_fgu(struct kvm *kvm, enum fgt_group_id f= gt); void get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg, u64 *res0, = u64 *res1); void check_feature_map(void); =20 +struct kvm_iommu_ops; +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); =20 #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 7c329e01c557..5528704bfd72 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -23,7 +23,8 @@ kvm-y +=3D arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.= o \ vgic/vgic-v3.o vgic/vgic-v4.o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ - vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o + vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ + iommu.o =20 kvm-$(CONFIG_HW_PERF_EVENTS) +=3D pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) +=3D pauth.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h new file mode 100644 index 000000000000..1ac70cc28a9e --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NVHE_IOMMU_H__ +#define __ARM64_KVM_NVHE_IOMMU_H__ + +#include + +struct kvm_iommu_ops { + int (*init)(void); +}; + +int kvm_iommu_init(void); + +#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index a76522d63c3e..393ff143f0be 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o +=3D -Wno-override-init =20 hyp-obj-y :=3D timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o= host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \ + iommu/iommu.o hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../en= try.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c new file mode 100644 index 000000000000..a01c036c55be --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * IOMMU operations for pKVM + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +/* Only one set of ops supported */ +struct kvm_iommu_ops *kvm_iommu_ops; + +int kvm_iommu_init(void) +{ + if (!kvm_iommu_ops || !kvm_iommu_ops->init) + return -ENODEV; + + return kvm_iommu_ops->init(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index ee6435473204..bdbc77395e03 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -320,6 +321,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 + ret =3D kvm_iommu_init(); + if (ret) + goto out; + ret =3D hyp_ffa_init(ffa_proxy_pages); if (ret) goto out; diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c new file mode 100644 index 000000000000..926a1a94698f --- /dev/null +++ b/arch/arm64/kvm/iommu.c @@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 Google LLC + * Author: Mostafa Saleh + */ + +#include + +extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); + +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) +{ + kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; + return 0; +} --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 305E43570A4 for ; Tue, 19 Aug 2025 21:52:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640357; cv=none; b=s3c35bGXq7mdw0Xi0JucMh783l4IFdZaPNOZYfyOuHyuFWTRn2WsdcHuWMwE/CAEtWHNRMEWu7IZZw7XzG379dbSOQbcHWOUd1T0xtlj5YUQK2dJMQuYv3T1B/vyQAdhKQmfX0hRethLvWfW945Hk5wtMGNoXHKVStgSvZ66X5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640357; c=relaxed/simple; bh=HSd950jJ2t4GJC22m7W741IHNilb7Crd6j1IkBxAl2E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QCunWkMaqQgN7kKMb7yvqtf77z+mMrvI/l+OqUCB+SYPAYiqFtzFiXUQSOphIMl2jsYZq+ITQy8+joq43My0vOT+zX0232nwpYgBXcmkY/qTpu/F0yu2F0VBeQFYhc61p/T4xuIkZEAiuju3Njv9tVibnuL/aEhTBufOWJMxQFA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vktyHfN0; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vktyHfN0" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45a1b0d0feaso34565205e9.2 for ; Tue, 19 Aug 2025 14:52:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640353; x=1756245153; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FTW2UGpkj+rMJnUPmb+I3Uc3pZGMzoyVmbU3iR5h6JE=; b=vktyHfN0KRINQj90AcXKe7Lsha2rmZ8EL3eXvAh+CD/Y+2rq1cH2dUUB9RpXeQbVeQ dp1Z5BC2x02dX0C3UeBLoajIGa+00sSZwUgOM9rmuBRCST3sj779d2pIWxbSTL+L8ol5 Oo1TIxnePGrUwd62NOUzBhn38Ap7umfuLBFvSHWzKoVb+pvp1532+u0yWZNrc8Brzkfa G94I8PD69a2xnNpFEx2rhTJxIsAYy59kLwY086tY1mUlpRj556f8TpscpypbGyqMHTp/ MU0zTyLuCidUdr19cesvkzoYU6k+1PSloGloDr7pKCX11eS7HQI0jo/+TywhaTKWN/yH Ztqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640353; x=1756245153; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FTW2UGpkj+rMJnUPmb+I3Uc3pZGMzoyVmbU3iR5h6JE=; b=TZ+vj7Ywu48HhJPWJrw+pqYhGuqSOnXaP5Ob/7jnfdPwitDbMAP/EQsYG2PpYNBnQK LV3kwMLYsoPD+1jBWOcxXUJZUNRLbIYD13mUwEvtl+L55hdHFJ2jMdXH9elGwdv49ql3 ses3Ekj4j+duCGvyqEp3M2gcfPfq3PQMVsWHx/KS5EvcG5YMGkxDC+WIk1RT30604lA/ ETC1+s56CCQiEkXLeg3swXI1fhEQ64GFvPgt9HVgIRBlGadvLzu5M3sn4iypbuOxSjwc fi6jxqTG8Th7e9KeMkXFk/UckUp+14BMedPbh4DdhMTaeSC0AF9W5xGK1l2adCHcXsDZ iHTw== X-Gm-Message-State: AOJu0YytB+adgNXIZ9dYwvxdzYiHNdi34ZzPnKVsaDnQ6QOWYVcmHjE3 iCAp/oHJcV55y7E9afitfqO5s5wuO7AZ7F8P0/4/jjHgf7s5zU/xj2oM7zowRZblJyXuzQhsHR9 AgtL2hTU+jEwrnKIex46yC/zlRjV/2wOlGtpaZTR75xgM+89WdNsuZKYt+bJO1ZiDwCoWRvIhbr J56WKXqxHsYM8JCd5S8STZNh2+D3YhynkAttfD11TDVNKW1olKl4BrA4E= X-Google-Smtp-Source: AGHT+IFC4f9TFYtJcHeYguMFJJzghlWK9KU6mi0KIys1MVGWduWRRdB37rOMzmuqxanQUm8EohCMZm6hqx7hmQ== X-Received: from wmpz12.prod.google.com ([2002:a05:600c:a0c:b0:459:de2a:4d58]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:154b:b0:459:dc35:dc05 with SMTP id 5b1f17b1804b1-45b479a393dmr3326805e9.9.1755640353566; Tue, 19 Aug 2025 14:52:33 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:38 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-11-smostafa@google.com> Subject: [PATCH v4 10/28] KVM: arm64: iommu: Shadow host stage-2 page table From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a shadow page table for the IOMMU that shadows the host CPU stage-2 into the IOMMUs to establish DMA isolation. An initial snapshot is created after the driver init, then on every permission change a callback would be called for the IOMMU driver to update the page table. For some cases, an SMMUv3 may be able to share the same page table used with the host CPU stage-2 directly. However, this is too strict and requires changes to the core hypervisor page table code, plus it would require the hypervisor to handle IOMMU page faults. This can be added later as an optimization for SMMUV3. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 4 ++ arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 83 ++++++++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 5 ++ 3 files changed, 90 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 1ac70cc28a9e..219363045b1c 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -3,11 +3,15 @@ #define __ARM64_KVM_NVHE_IOMMU_H__ =20 #include +#include =20 struct kvm_iommu_ops { int (*init)(void); + void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 int kvm_iommu_init(void); =20 +void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index a01c036c55be..f7d1c8feb358 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,15 +4,94 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + #include +#include +#include =20 /* Only one set of ops supported */ struct kvm_iommu_ops *kvm_iommu_ops; =20 +/* Protected by host_mmu.lock */ +static bool kvm_idmap_initialized; + +static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) +{ + int iommu_prot =3D 0; + + if (prot & KVM_PGTABLE_PROT_R) + iommu_prot |=3D IOMMU_READ; + if (prot & KVM_PGTABLE_PROT_W) + iommu_prot |=3D IOMMU_WRITE; + if (prot =3D=3D PKVM_HOST_MMIO_PROT) + iommu_prot |=3D IOMMU_MMIO; + + /* We don't understand that, might be dangerous. */ + WARN_ON(prot & ~PKVM_HOST_MEM_PROT); + return iommu_prot; +} + +static int __snapshot_host_stage2(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags visit) +{ + u64 start =3D ctx->addr; + kvm_pte_t pte =3D *ctx->ptep; + u32 level =3D ctx->level; + u64 end =3D start + kvm_granule_size(level); + int prot =3D IOMMU_READ | IOMMU_WRITE; + + /* Keep unmapped. */ + if (pte && !kvm_pte_valid(pte)) + return 0; + + if (kvm_pte_valid(pte)) + prot =3D pkvm_to_iommu_prot(kvm_pgtable_stage2_pte_prot(pte)); + else if (!addr_is_memory(start)) + prot |=3D IOMMU_MMIO; + + kvm_iommu_ops->host_stage2_idmap(start, end, prot); + return 0; +} + +static int kvm_iommu_snapshot_host_stage2(void) +{ + int ret; + struct kvm_pgtable_walker walker =3D { + .cb =3D __snapshot_host_stage2, + .flags =3D KVM_PGTABLE_WALK_LEAF, + }; + struct kvm_pgtable *pgt =3D &host_mmu.pgt; + + hyp_spin_lock(&host_mmu.lock); + ret =3D kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker); + /* Start receiving calls to host_stage2_idmap. */ + kvm_idmap_initialized =3D !!ret; + hyp_spin_unlock(&host_mmu.lock); + + return ret; +} + int kvm_iommu_init(void) { - if (!kvm_iommu_ops || !kvm_iommu_ops->init) + int ret; + + if (!kvm_iommu_ops || !kvm_iommu_ops->init || + !kvm_iommu_ops->host_stage2_idmap) return -ENODEV; =20 - return kvm_iommu_ops->init(); + ret =3D kvm_iommu_ops->init(); + if (ret) + return ret; + return kvm_iommu_snapshot_host_stage2(); +} + +void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot) +{ + hyp_assert_lock_held(&host_mmu.lock); + + if (!kvm_idmap_initialized) + return; + kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot)); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index c9a15ef6b18d..bce6690f29c0 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -15,6 +15,7 @@ #include =20 #include +#include #include #include #include @@ -529,6 +530,7 @@ static void __host_update_page_state(phys_addr_t addr, = u64 size, enum pkvm_page_ int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id) { int ret; + enum kvm_pgtable_prot prot; =20 if (!range_is_memory(addr, addr + size)) return -EPERM; @@ -538,6 +540,9 @@ int host_stage2_set_owner_locked(phys_addr_t addr, u64 = size, u8 owner_id) if (ret) return ret; =20 + prot =3D owner_id =3D=3D PKVM_ID_HOST ? PKVM_HOST_MEM_PROT : 0; + kvm_iommu_host_stage2_idmap(addr, addr + size, prot); + /* Don't forget to update the vmemmap tracking for the host */ if (owner_id =3D=3D PKVM_ID_HOST) __host_update_page_state(addr, size, PKVM_PAGE_OWNED); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 757A935A281 for ; Tue, 19 Aug 2025 21:52:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640359; cv=none; b=UlPKyEVxTaaqQShq8uFMgcjR/tTH9AqsX/BfSlz67KCC0h/JYUTLqMCjTd04REILOhQoH3qd4q8r0FZT0UYJ5cxhIJKLUcHgA0zh3ZazmcW9X3lFkKVs7bPePYht3wvTjD3BOb3joPapMjv+G1Nm9ZEDSoa+7dBYkUv7JF52ktM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640359; c=relaxed/simple; bh=vXwQDqElLwmdFXH6atI73BgGkGDD39YTucq302WuvLs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YorD5kIaABeJn632BqeHS648kNq1yfvh9Pj0iDZXw0Y4+hfLQ67+VyCfp+ujjF1fXm2y2AvbVrC5kd3noHZqHcnuhQRxHsWyBbn8WVTNAMrXHtFIwbQ46vj4TjYmkHSwji4wxGl/be9FdCR8XF9/VoXRzUuO1TddiaFCRuNk4eQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a+E170kc; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a+E170kc" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3b9d41b88ffso2609393f8f.0 for ; Tue, 19 Aug 2025 14:52:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640355; x=1756245155; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bHyXp0c9r8a6Z3kqM6MYogJzQkCF/sRbjU6o05FpAFo=; b=a+E170kcRHESTgF8KYS2+YedhptexBklgcrhhWSp1WX7kEw3ckG0EGARBuPWGIeyAs n6/j1+VMqpZlJgkgnF8cXTjQy2zY38DK02re6qRCB20Hx9HqRs1zDDaV7wHBzwSXVrrC /vND0T2kA1fi5QgxTDo3MXG64TRlAmE2e+hfMQQVgj2E89B3DzC+ISU33nTNQfJjyxRR vAVYI59KlLL8RZVSv0O255VpW5uQSyxMkMCutSTgs8aH5Sf1N/+qU8JUsW863l9AsVv5 qlFqj5QQySHp7T70+KemVNaVkiBd2Oe93G8feba0+PDRHF4hIi9asUMvRTxLxvnwV59e FbCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640355; x=1756245155; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bHyXp0c9r8a6Z3kqM6MYogJzQkCF/sRbjU6o05FpAFo=; b=SI5Avcfh4PQAWkprnSPG6VZPuVBryfThkxkwAJHHzG3RGIY9YWK5QE2vPLlcI8aj2C 5mD5Wa3aCkqBz1XKZ8wVJ6W0FMAUDCkjz/AihivI54UZvCSiK6o7mlMYfiyYlmxfjqt3 vMZuZbnSQcKh+cHz7hrf1SvkpQt7Mdpfce6fqd6k1Qh0jQ4ox+OsGzVd4qgCHVB+qllj UNNTo86W54rN7sryVHP25Lg/jkTnxeAN9x2FsPhk0LQ/4j2MzAXv/Wfah5d3/jf9hLk9 NDEGShQ9N4Z70o1WYwNiXu4uc0cPo1riv1hUU/wbLwZSFxDV5Rn6G9RbDGr/gZFQbReP Zunw== X-Gm-Message-State: AOJu0Yx/R+/GoBMRBpM8+7SiU4KKMfRiyr8kXLPJ447ZSe+I/rpQpKfr qLe8QvpXIJ/MchRLcgqIvJZzg5Cns3QMLCACK9rMDixx5f+IAvfvTXhJiqYdV6pQ3uNlYIaNGBX TjXboDxHHrdAZlXhFoEjvQIJbx8zknFY7kOMFCvV2Z6atWumbCGm113zrKzaL/8tT48N9ct9kL2 JgT6tIQMbYy2+CUvChhaE7gbIl8v3ae4EPeRzPQKHlntzkUoujAryDrkY= X-Google-Smtp-Source: AGHT+IF09z/zChrY/XKW2p3rPVo1I7WRsoRzioqDNTUyJHzYESu4rlz8VOrrsAe/v0WNmek1lpwYcIduC6GS/A== X-Received: from wmth10.prod.google.com ([2002:a05:600c:8b6a:b0:459:e347:8a6e]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4284:b0:3a6:d145:e2cc with SMTP id ffacd0b85a97d-3c32e03e447mr337072f8f.15.1755640354609; Tue, 19 Aug 2025 14:52:34 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:39 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-12-smostafa@google.com> Subject: [PATCH v4 11/28] KVM: arm64: iommu: Add memory pool From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" IOMMU drivers would require to allocate memory for the shadow page table. Similar to the host stage-2 CPU page table, the IOMMU pool is allocated early from the carveout and it's memory is added in a pool which the IOMMU driver can allocate from and reclaim at run time. At this point the nr_pages is 0 as there are no driver, in the next patches when the SMMUv3 driver is added, it will add it's own function to return the number of pages needed in kvm/iommu.c. Unfortunately, this part has 2 leak into kvm/iommu as this happens too early before drivers can have any init calls. Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/hyp/include/nvhe/iommu.h | 5 ++++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 20 +++++++++++++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 10 +++++++++- arch/arm64/kvm/iommu.c | 11 +++++++++++ arch/arm64/kvm/pkvm.c | 1 + 6 files changed, 45 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 1a08066eaf7e..fcb4b26072f7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1676,5 +1676,6 @@ void check_feature_map(void); =20 struct kvm_iommu_ops; int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); +size_t kvm_iommu_pages(void); =20 #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 219363045b1c..9f4906c6dcc9 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -10,8 +10,11 @@ struct kvm_iommu_ops { void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 -int kvm_iommu_init(void); +int kvm_iommu_init(void *pool_base, size_t nr_pages); =20 void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, enum kvm_pgtable_prot prot); +void *kvm_iommu_donate_pages(u8 order); +void kvm_iommu_reclaim_pages(void *ptr); + #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index f7d1c8feb358..1673165c7330 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -15,6 +15,7 @@ struct kvm_iommu_ops *kvm_iommu_ops; =20 /* Protected by host_mmu.lock */ static bool kvm_idmap_initialized; +static struct hyp_pool iommu_pages_pool; =20 static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) { @@ -72,7 +73,7 @@ static int kvm_iommu_snapshot_host_stage2(void) return ret; } =20 -int kvm_iommu_init(void) +int kvm_iommu_init(void *pool_base, size_t nr_pages) { int ret; =20 @@ -80,6 +81,13 @@ int kvm_iommu_init(void) !kvm_iommu_ops->host_stage2_idmap) return -ENODEV; =20 + if (nr_pages) { + ret =3D hyp_pool_init(&iommu_pages_pool, hyp_virt_to_pfn(pool_base), + nr_pages, 0); + if (ret) + return ret; + } + ret =3D kvm_iommu_ops->init(); if (ret) return ret; @@ -95,3 +103,13 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phy= s_addr_t end, return; kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(prot)); } + +void *kvm_iommu_donate_pages(u8 order) +{ + return hyp_alloc_pages(&iommu_pages_pool, order); +} + +void kvm_iommu_reclaim_pages(void *ptr) +{ + hyp_put_page(&iommu_pages_pool, ptr); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index bdbc77395e03..09ecee2cd864 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -21,6 +21,7 @@ #include =20 unsigned long hyp_nr_cpus; +size_t hyp_kvm_iommu_pages; =20 #define hyp_percpu_size ((unsigned long)__per_cpu_end - \ (unsigned long)__per_cpu_start) @@ -33,6 +34,7 @@ static void *selftest_base; static void *ffa_proxy_pages; static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; static struct hyp_pool hpool; +static void *iommu_base; =20 static int divide_memory_pool(void *virt, unsigned long size) { @@ -70,6 +72,12 @@ static int divide_memory_pool(void *virt, unsigned long = size) if (!ffa_proxy_pages) return -ENOMEM; =20 + if (hyp_kvm_iommu_pages) { + iommu_base =3D hyp_early_alloc_contig(hyp_kvm_iommu_pages); + if (!iommu_base) + return -ENOMEM; + } + return 0; } =20 @@ -321,7 +329,7 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 - ret =3D kvm_iommu_init(); + ret =3D kvm_iommu_init(iommu_base, hyp_kvm_iommu_pages); if (ret) goto out; =20 diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c index 926a1a94698f..5460b1bd44a6 100644 --- a/arch/arm64/kvm/iommu.c +++ b/arch/arm64/kvm/iommu.c @@ -7,9 +7,20 @@ #include =20 extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); +extern size_t kvm_nvhe_sym(hyp_kvm_iommu_pages); =20 int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) { kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; return 0; } + +size_t kvm_iommu_pages(void) +{ + /* + * This is called very early during setup_arch() where no initcalls, + * so this has to call specific functions per each KVM driver. + */ + kvm_nvhe_sym(hyp_kvm_iommu_pages) =3D 0; + return 0; +} diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index fcd70bfe44fb..6098beda36fa 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -63,6 +63,7 @@ void __init kvm_hyp_reserve(void) hyp_mem_pages +=3D hyp_vmemmap_pages(STRUCT_HYP_PAGE_SIZE); hyp_mem_pages +=3D pkvm_selftest_pages(); hyp_mem_pages +=3D hyp_ffa_proxy_pages(); + hyp_mem_pages +=3D kvm_iommu_pages(); =20 /* * Try to allocate a PMD-aligned region to reduce TLB pressure once --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5CCBE35A2B9 for ; Tue, 19 Aug 2025 21:52:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640360; cv=none; b=pHXDmZotDXJxLkKzcpK9gxSa/nkOkafnKLFR+FcUmP+ax/S4Sxn7F/F78DHPlBAY2fbrFtinlxDOaF5vo9NE4cARZjS2NXSMNXfeWKl2ZC72mGqNByMEOrC7OEBmTAGxqe4pCYM6zQXSg9Dln7MViTuret6ziydlnEFQHJRCmdc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640360; c=relaxed/simple; bh=ZStA5L9CRyv7ZGIOd1ez63lJfoDR2LVWfDOZLq7FmHc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NnemI8XZcGkcT98Nvsr6ZFepx7QZJFvzCaQJPf9FbVmCgAKXd+Z9Q/K7be761vSzEMO373e5EInlvrFGa85DNCfwhQFyMwxYy1SpZLURlvMFg0OzdFI90x12D32Mom0CShl9M4n2nt/TknEwpUZRsAfmhK3AnzyO9O9w/Fp2ejg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a94wrXlR; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a94wrXlR" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3b9e4157303so4677820f8f.2 for ; Tue, 19 Aug 2025 14:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640355; x=1756245155; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=gBsLyy3u3rOhplnfl3DMV6GQeOwFjCQ0fFDaMLqFvas=; b=a94wrXlRoiGB8lzHiXUE11dWoxIl7mK1MXL5fIXLyYnSAdSlAQIkf48vvqU6WYVT0J fOcrFGA7KaKvztxR1CQvrIN50dPD2zUX3gfaqpfuS7cASWzLiBX6awuwD70KjfDaKOKx EsOeG686+4oMNURr8fiRGPKUiUZylFxOO/6I2iQiTEVV4xojiMjUZCR4OILQtqBhUtBR ZluNn2qhAwMg79UQ177ZSn5gu0sc8NnE5YXBqmX4czcyrH3diKROFUA5wdbz6EB4vlTn o9x0iWr80s4sTdEH/1bVpYs9x19K0swSdvNgtrXAO/Pj3yys6XlgeKRMAP5DZ2K3tYRm hffA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640355; x=1756245155; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gBsLyy3u3rOhplnfl3DMV6GQeOwFjCQ0fFDaMLqFvas=; b=OWGw9Gyq88ZGeBs4rQG9welOzPDWQuy3eoaS/+7aU09kL81D8e+vSAgsACj2L7uMxW ELetW7OUv2BlnsouSeVNhjeNuLh38uqKksEn/4Nu74Zq4Vx4NejI3XNepyaHZqCl5CCX 17yFsdwhU/LhFopksqsaMwbGVlX4Yd2IRuzB2WvE7lrCte3LxiDQaakeS5NmaKAnf2Li f9Hqa0lqBVR0q2kgQXrREQAlgNfdZG0ifmY8DJa1y5GFyyy4NELzvuFKkGXRT/ecUE9P GqI1jsirPzEwf49sYRLj6ycmdLsFdJA9jL8yK6NlzjpKvNOp2tppU+h+rGq6VGM7vIS5 8/2g== X-Gm-Message-State: AOJu0YzOxriIJBV1hqUGv25K9x4LOpyU603SKun8Jq4CCTNYLRPGvoJB zy9RjpGLLnPVQwrWWY09KPWqwRnVFUbE+2ZEd6jFyIzlQ7Cqj1GW/JEfKPa/HjV7n4kjMpgRnk3 pZ4qst4ZfV3WO5o6QMWO9sqSgJrX4uC2isHOueiJxtKsGSY2vLB+fdtopND43IpkfOkaO4Xb8zC 5tHsnYU4l5dBYlccp6/buW7ZsJhQE4r9YoOciC//9Q5A9EcWgRyKhkA9E= X-Google-Smtp-Source: AGHT+IGi+MrTYS9vOAkhE2MwbT5in2zuzEYUdbygOz38UpjYqK2AvcuVjizybf9/jK1Tu7dGcGgyI44X4YgECw== X-Received: from wmsr19.prod.google.com ([2002:a05:600c:8b13:b0:459:8c48:6c34]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:22c4:b0:3a4:eb92:39b6 with SMTP id ffacd0b85a97d-3c32eeb8077mr359066f8f.54.1755640355498; Tue, 19 Aug 2025 14:52:35 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:40 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-13-smostafa@google.com> Subject: [PATCH v4 12/28] KVM: arm64: iommu: Support DABT for IOMMU From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SMMUv3 driver need to trap and emulate access to the MMIO space as part of the nested implementation. Add a handler for DABTs for IOMMU drivers to be able to do so, in case the host fault in page, check if it's part of IOMMU emulation first. Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_arm.h | 2 ++ arch/arm64/kvm/hyp/include/nvhe/iommu.h | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 15 +++++++++++++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 10 ++++++++++ 4 files changed, 29 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_= arm.h index 1da290aeedce..8d63308ccd5c 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -331,6 +331,8 @@ =20 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */ #define HPFAR_MASK (~UL(0xf)) + +#define FAR_MASK GENMASK_ULL(11, 0) /* * We have * PAR [PA_Shift - 1 : 12] =3D PA [PA_Shift - 1 : 12] diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 9f4906c6dcc9..10fe4fbf7424 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -8,6 +8,7 @@ struct kvm_iommu_ops { int (*init)(void); void (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); + bool (*dabt_handler)(struct user_pt_regs *regs, u64 esr, u64 addr); }; =20 int kvm_iommu_init(void *pool_base, size_t nr_pages); @@ -16,5 +17,5 @@ void kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_= addr_t end, enum kvm_pgtable_prot prot); void *kvm_iommu_donate_pages(u8 order); void kvm_iommu_reclaim_pages(void *ptr); - +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index 1673165c7330..376b30f557a2 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,6 +4,10 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + +#include + #include =20 #include @@ -113,3 +117,14 @@ void kvm_iommu_reclaim_pages(void *ptr) { hyp_put_page(&iommu_pages_pool, ptr); } + +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr) +{ + if (kvm_iommu_ops && kvm_iommu_ops->dabt_handler && + kvm_iommu_ops->dabt_handler(regs, esr, addr)) { + /* DABT handled by the driver, skip to next instruction. */ + kvm_skip_host_instr(); + return true; + } + return false; +} diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index bce6690f29c0..7371b2183e1e 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -595,6 +595,11 @@ static int host_stage2_idmap(u64 addr) return ret; } =20 +static bool is_dabt(u64 esr) +{ + return ESR_ELx_EC(esr) =3D=3D ESR_ELx_EC_DABT_LOW; +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault; @@ -617,6 +622,11 @@ void handle_host_mem_abort(struct kvm_cpu_context *hos= t_ctxt) */ BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS)); addr =3D FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12; + addr |=3D fault.far_el2 & FAR_MASK; + + if (is_dabt(esr) && !addr_is_memory(addr) && + kvm_iommu_host_dabt_handler(&host_ctxt->regs, esr, addr)) + return; =20 ret =3D host_stage2_idmap(addr); BUG_ON(ret && ret !=3D -EAGAIN); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33D0C35A2A4 for ; Tue, 19 Aug 2025 21:52:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640360; cv=none; b=HLJ12LPGBrJCgFltx0+PLaiyAAq6HPWJg3+2iUo9OiHNogbLyQLHFNcmzpeG4HJC0uCd6wxO0UYG+qZjbgXZStUQpOqmYNfVJK0l/bEq24suh0+RaYszS+DvVh30I1s0SP8L+BlfL/Od1u2s0gn8ZqcfTYsh4DSkUo46JOKLK/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640360; c=relaxed/simple; bh=/Unve+csthHtV/Kg7izjSh135002Cudi01KNNjtlOpY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=O8CLZlMhMVFejdwO8EM4scGcMl4dfr0o5AkopsvGcTXq+hf8mJCOOm7KNwiPvFCH3TexxFwJLC+WJWAPwmO8mbL7wlrvmwdrY5EiRYDGmx/T2AGZRfumAN0cGzsVg0ikGD9N8W6JmtRjVRZZ/4q83V7GtHd/zAbaFMSXmX2eBYY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LqyuAjDM; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LqyuAjDM" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-afcb733834bso548513766b.0 for ; Tue, 19 Aug 2025 14:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640356; x=1756245156; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=e98f0DNE5CRlyU4NM0kgE4ri91mOXmN3Zo8dZtJNlTc=; b=LqyuAjDMmj+gt7Tw5dRZsRNwW06FWcMtv9WVHoXl5IIAbbhJUK90mSQqrV59kVWBWN SzzISb7oSwDPsVHhSBhIYwEcbtUl203RqN9mRuA4IBEH66TylvcGtrDS/Zw9COAyW6qu la57u47v3YX0lRVUmhMPiAHH4EdPyHNtpp5D229zEi8tGCTOMLhmaqKhyub9WcbcqphP qJV45yz1GsRWtLg9hUIaeLSosAQcoSxhyBSIVN9wokIirgkydLtyvSLjFocieBzJzmBE JNn1bGMNls7KbxJDAhiVYHgW3b/EBll1yYmssEZrnNvalCGJpFM4ccVpD05adJjSVqQo bi0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640356; x=1756245156; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e98f0DNE5CRlyU4NM0kgE4ri91mOXmN3Zo8dZtJNlTc=; b=tX8e3LpLUwXy/4AaE5rxkOKr3X/whJfZ2ZLqFRG3uCq4xMOCElNLm5ngbXKlzHTZga 8fe3hmG5bmqsaycEQINHGwbiVA3SaHmPSaKv4EPXr4plvtKWQ5BggZKfyKb6mlM9scnr 5CQXavwulO1koiFkktl+topIS8JQoL+zdK0OjHmxOyLsFfNhOll5XRLWwVOT7XbXVGyN lp3EXq+Zh5bYGl0oX2NCLSrtxWnfj98PVSROmP0+1hwSXf6572wVi9UYU7+/XMefs6Ei b0O/qHz/rIdG8qy4jDMg3Jcc5HjfRG9na6JmxuQSVHEsEf4OsAlkbGWT9jeGaKnTa9Wc UDyw== X-Gm-Message-State: AOJu0YzYNc76S1+LCwzP7aW+q2XsGggH09xfOYEH0MefiSgVhH9F0z/m 0XaePKfe7GLimgu4HQ8jlkjEMQX0tI0six7j/XbIozHaFMJTNxz8n5ocQYoHIqR5FzIWKDVgca1 KBfw5iqjwy1CrmlsUxXzA4c9kP0LWiSfz4TkSAUWU45XLxy0KxF3w6pZ9sA7qvf0aPd2/v5Sfxw zE4H2BxCWIYtVFWmb+1fL2r535MKjb6HT0fdCfSgZvYCiqhtgFi3WwBLE= X-Google-Smtp-Source: AGHT+IHCyvZOecVyOHzeGqtTcsp9WMNtt0BHBZO+OkO4882nr2LShFVO1rZjvNy7Rv+QSkccbtheqDZoV50swQ== X-Received: from edbek21.prod.google.com ([2002:a05:6402:3715:b0:61a:1a2c:d08e]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:7f8e:b0:af9:1122:6af0 with SMTP id a640c23a62f3a-afdf01cfa9dmr47797666b.30.1755640356474; Tue, 19 Aug 2025 14:52:36 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:41 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-14-smostafa@google.com> Subject: [PATCH v4 13/28] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jean-Philippe Brucker Add the skeleton for an Arm SMMUv3 driver at EL2. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 5 ++++ drivers/iommu/arm/Kconfig | 9 ++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 29 +++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 16 ++++++++++ 4 files changed, 59 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 393ff143f0be..c71c96262378 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -31,6 +31,11 @@ hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-c= puif-proxy.o ../entry.o \ hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o hyp-obj-y +=3D $(lib-objs) =20 +HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 + +hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o + ## ## Build rules for compiling nVHE hyp code ## Output of this folder is `kvm_nvhe.o`, a partially linked object diff --git a/drivers/iommu/arm/Kconfig b/drivers/iommu/arm/Kconfig index ef42bbe07dbe..7eeb94d2499d 100644 --- a/drivers/iommu/arm/Kconfig +++ b/drivers/iommu/arm/Kconfig @@ -142,3 +142,12 @@ config QCOM_IOMMU select ARM_DMA_USE_IOMMU help Support for IOMMU on certain Qualcomm SoCs. + +config ARM_SMMU_V3_PKVM + bool "ARM SMMUv3 support for protected Virtual Machines" + depends on KVM && ARM64 && ARM_SMMU_V3=3Dy + help + Enable a SMMUv3 driver in the KVM hypervisor, to protect VMs against + memory accesses from devices owned by the host. + + Say Y here if you intend to enable KVM in protected mode. diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c new file mode 100644 index 000000000000..fa8b71152560 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM hyp driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +#include + +#include "arm_smmu_v3.h" + +size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; +struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; + +static int smmu_init(void) +{ + return -ENOSYS; +} + +static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) +{ +} + +/* Shared with the kernel driver in EL1 */ +struct kvm_iommu_ops smmu_ops =3D { + .init =3D smmu_init, + .host_stage2_idmap =3D smmu_host_stage2_idmap, +}; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h new file mode 100644 index 000000000000..f6ad91d3fb85 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -0,0 +1,16 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_ARM_SMMU_V3_H +#define __KVM_ARM_SMMU_V3_H + +#include + +struct hyp_arm_smmu_v3_device { +}; + +extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); +#define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count) + +extern struct hyp_arm_smmu_v3_device *kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smm= us); +#define kvm_hyp_arm_smmu_v3_smmus kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus) + +#endif /* __KVM_ARM_SMMU_V3_H */ --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E86C93054CA for ; Tue, 19 Aug 2025 21:52:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640361; cv=none; b=EHqO9FmYzRqoL2jY3HWRZB/8HkAxakrf9p1SbRO5OblJZOrIUVNZdSfBMs8u2hbkT/3dw3n46WLTGMTPDKkTGM1LlEivxlAr6T+0Z9NqAbdrSdEDt0WMQ3zA+mtRy5t0OEElWr4lnqJ+ZtYM5cvyfGD5C0y1LxINI3jLcshqI6A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640361; c=relaxed/simple; bh=3r8oXH7nsfm/ONpYF9TIZSokbPzVQMcEKf/AvnlgvDw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=plnCUZnsEadGnzMHynkO3h1mfSAMYzSH7NJ01oWNUrKHqKec+glgWMVSHu0Z0EgQO8zbEjmoap/oIn2VfA3yg74a3Bx5dhb/u5hr1XnF29Z3RaPE7nN8lGgoXL098IS2+0eUAni4sRXwXDIwsSmbJdal9bmibTTGFEhFhJv4Njw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kRdtOuyN; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kRdtOuyN" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45a1b00352eso22034225e9.0 for ; Tue, 19 Aug 2025 14:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640357; x=1756245157; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=X4PaN1FBVh+xdnfphbKwqZhoaGiNxq0bOXNDJ6vx+5U=; b=kRdtOuyNL7+JmopG/IhWNHex8pKqAONtnWh7OLIUmBQzFBuCGXasY0xxh1KlMy3lCI np38Z+ItsUZzSqFcn91NPkOtDPH91Mw1RT+4IwWaBGyMkeM/cpuWePlrUWd2KC7+rSjc qyv4az4vy8UdIIs4w8g/jWnA7v2JSz13cNRTUOZmRYqCEQdWerW4IQAgH7PcYMLgPnMP Dj6PcUWaeWbwKOFIypGHN9eCCKAQi2EZPILiB2ggHgjxcuH82uIuA0gxZKm8QavJQuaZ r4cXNk4fD3i16Hgpz52+YR9sWeXGTZjxF8NZ+wvoXoF++U2MOyAb1O1jqaYVzrMugQuJ hr/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640357; x=1756245157; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X4PaN1FBVh+xdnfphbKwqZhoaGiNxq0bOXNDJ6vx+5U=; b=IfI/H9ppJ4m3ARqBSATLb/rb2mU9W0nPNvI7a4bCw4qnLVrqVsNewmZFrwgs8y65tt l6bFqiT+ipRqS8BaBren+vQuSFfMKk40HTUwBUBX1u5f+DiZHkHcEgsTgAaOGKfjGX1C jw77n2+o9wJlIX+FyCcCGjlNCpRpf1Jr4rgdI1eTzu2XJxKWPYmvOgA8djHBz/JY3NEU x0u3knHtel6raK/2fS6DcmLS+mc0Fhfq5YdnU0/CO5m6qt3PXKgP8pyiluVI5Z/SXMYm ezSUDVx/ildkNRKYsYBmAicm0fhYB7ZRX2c9AOB6kpWFIg+KG1C5+L8E7I1qFZ0KF8lS 7mnQ== X-Gm-Message-State: AOJu0YyMNCrAn8La00AWwfSaOWMMuNayJ3G9yMzKfnmGQbei2QcvYGKx 7ye3ulHeNYZby5wP1tSib02e5bxC++GkaTkvVGYqTewIzbygvnfZ+054DqnTKGV3mlsL1Zzoklo zf2HU5Fxd0nCQLcP6RhzqFPLr5kdXwhodPmkDTLV+aAWlw8w7GN/R6z5ZweNQj5z/j+tMDW362v vH398PI0287h6Qe+Pmyd8VWR8enzUiiLNB3tXY0K/2z48N/45vIBTPy0Q= X-Google-Smtp-Source: AGHT+IFsd9NA8cMc7rDL27Ho1qYY/f8xjo3m+B7JdKk4faCMk1aV/KdUXjy/UILcCgJqjvIHpGxiAh/dHHuItw== X-Received: from wmcn3-n2.prod.google.com ([2002:a05:600c:c0c3:20b0:45b:4777:8063]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1e89:b0:459:e398:ed80 with SMTP id 5b1f17b1804b1-45b47a0700bmr2494445e9.32.1755640357365; Tue, 19 Aug 2025 14:52:37 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:42 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-15-smostafa@google.com> Subject: [PATCH v4 14/28] iommu/arm-smmu-v3: Add KVM mode in the driver From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a file only compiled for KVM mode. At the moment it registers the driver with KVM, and add the hook needed for memory allocation. Next, it will create the array with available SMMUs and their description. Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_host.h | 4 +++ arch/arm64/kvm/iommu.c | 10 ++++-- drivers/iommu/arm/arm-smmu-v3/Makefile | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 36 +++++++++++++++++++ 4 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index fcb4b26072f7..52212c0f2e9c 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1678,4 +1678,8 @@ struct kvm_iommu_ops; int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); size_t kvm_iommu_pages(void); =20 +#ifdef CONFIG_ARM_SMMU_V3_PKVM +size_t smmu_hyp_pgt_pages(void); +#endif + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c index 5460b1bd44a6..0475f7c95c6c 100644 --- a/arch/arm64/kvm/iommu.c +++ b/arch/arm64/kvm/iommu.c @@ -17,10 +17,16 @@ int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp= _ops) =20 size_t kvm_iommu_pages(void) { + size_t nr_pages =3D 0; + /* * This is called very early during setup_arch() where no initcalls, * so this has to call specific functions per each KVM driver. */ - kvm_nvhe_sym(hyp_kvm_iommu_pages) =3D 0; - return 0; +#ifdef CONFIG_ARM_SMMU_V3_PKVM + nr_pages =3D smmu_hyp_pgt_pages(); +#endif + + kvm_nvhe_sym(hyp_kvm_iommu_pages) =3D nr_pages; + return nr_pages; } diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index 1918b4a64cb0..284ad71c5282 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -4,5 +4,6 @@ arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-hyp.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o +arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D arm-smmu-v3-kvm.o =20 obj-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) +=3D arm-smmu-v3-test.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c new file mode 100644 index 000000000000..ac4eac6d567f --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM host driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include +#include + +#include + +#include "arm-smmu-v3.h" +#include "pkvm/arm_smmu_v3.h" + +extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); + +size_t smmu_hyp_pgt_pages(void) +{ + /* + * SMMUv3 uses the same format as stage-2 and hence have the same memory + * requirements, we add extra 500 pages for L2 ste. + */ + if (of_find_compatible_node(NULL, NULL, "arm,smmu-v3")) + return host_s2_pgtable_pages() + 500; + return 0; +} + +static int kvm_arm_smmu_v3_register(void) +{ + if (!is_protected_kvm_enabled()) + return 0; + + return kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_= ops)))); +}; + +core_initcall(kvm_arm_smmu_v3_register); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 60C263054EB for ; Tue, 19 Aug 2025 21:52:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640362; cv=none; b=Y6icnk6VcKUG7xWQ8v5uzoRqghjNRXVkVxB0j5m09wmIl6EQicpCbp+m2V8Zw/KcRkFGdYnTmtsSDpcvQCQObvh4B3SVMm1vG8+36wMQQO7jnivy74e6Rxb9JTda/Qw4m5lBGWCgEutj6OG1xAhtSWKYyMiV5Cx6FnYQ2KkP68Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640362; c=relaxed/simple; bh=cMrRujJwddEPtHhK0JXsWsznkc7gKRpUZGwRSJ6bEwY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IGesdwJfv7PPzA8snz4QsJeX6CFmZPkn7TP/fdFRb0CTwo4fwc1+zr7lJXLMzKZNRH+N6ny+AVChC2lNRPWY0i62Ft6IxO+mEbVR/P91ZgW6313ppefy2LnaJ0Ud1yPPIcqPReFKOngOWM7j3BTxuc1bsHJycwND8e771jqlqZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BhsqjdB/; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BhsqjdB/" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0ccb6cso29972475e9.3 for ; Tue, 19 Aug 2025 14:52:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640358; x=1756245158; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=YNj1S5Tjh+MTuuSgivTU6GhE64woFcS5FBFIpdALKxk=; b=BhsqjdB/Csl6HZd1/ovKouxJ4papOadhDLCHcQsGKZ7biJKA9loiSLPoX8XCMYdBlq 0WRwRjEz3klc5L8m1Ks2ejqD62LzCJetjbPjw0tqStgr++D0ooSzlmP+vOT+v/tqiczt 7PNu0oP6tA5YXRSAi5mDpQmuYUTWai6llgecke4dj7ExlGIZBsVQuJBxzoZEP0wtmlql F49S2UTT9dLtFFAiGkVWps4JSWhWOq9eVnwdNteH7lp5ncZOqxtg4CBe/WI+Q6gGTTuI UCeKozkRtjcD0iRJ/Fu+qOUZ77olWb0IYLaYjjG0P1gZnKRPZ/FevUrpvF478KmvuROb 4mcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640358; x=1756245158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YNj1S5Tjh+MTuuSgivTU6GhE64woFcS5FBFIpdALKxk=; b=Q+5IUG8qZ+m+rZmCQAMlnT8Q6MtEVqhCCOF7BV8uyEQS8l+yddHpyglgswEQUgeMq7 3lzgyMt2b+GSxzXviDrWKn4PHVCjBb4Uzvcm3ikoOBQJWi3frULDcI0dWQhpeHruo0CR vB1qnmPTm0ZLNFLt2n39dJjPhvXFGlLAM2jH6LyeZGXlLx2lXdEUa1NQn83DR8/JcnJS 1vgQWqoIt+FL/c6dNvrQ7LmJP8nbgFoji1izX/pcaiWBA219BBqEVz2Bt/jmPWxPk2h5 Psk0Fwge6kpZlR+5sBZB5YxjW31MeP5Qd/EF0M0ZGO6xGtF6kWJ5bQjuh6SZ9LSwIN1W tZRw== X-Gm-Message-State: AOJu0YzKcTZvGDjVnN5xn3S85LZ1Z6inkxSpb1JbvESlz3sHfh4k0Hbe G+nvOkl/PT6Z5crmMdWMRtHZptir7sgZqKTYyy1xc7rYejZjKAhfXTY9zdTC3QsRo3dhELQR01W wwhZtiiQZ2CyzwVQbqLvq5Wp3t/nxZ3J1Mfv6W3bjOKtVBJTPSTv9eiBgvEcb1d6qBScwmfuiXG fcU63tVAyi3XOWyBblIlMgJnKxZPCj07GZeDijRSdxBP7jqNSrC1rbBOA= X-Google-Smtp-Source: AGHT+IHhXE5eFe6Mvs1Uc4cVRo8SUsVH79pNU31b0EWqV1qHu+8OuogNqLM1q4nApTxhxyHGYIGINP4HjbEypA== X-Received: from wmsd13.prod.google.com ([2002:a05:600c:3acd:b0:459:10dd:2e55]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8b4b:b0:456:161c:3d6f with SMTP id 5b1f17b1804b1-45b479b8ed5mr2712295e9.11.1755640358514; Tue, 19 Aug 2025 14:52:38 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:43 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-16-smostafa@google.com> Subject: [PATCH v4 15/28] iommu/arm-smmu-v3: Load the driver later in KVM mode From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" While in KVM mode, the driver must be loaded after the hypervisor initializes. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 25 ++++++++++++++++----- 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 10ca07c6dbe9..a04730b5fe41 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4576,12 +4576,6 @@ static const struct of_device_id arm_smmu_of_match[]= =3D { }; MODULE_DEVICE_TABLE(of, arm_smmu_of_match); =20 -static void arm_smmu_driver_unregister(struct platform_driver *drv) -{ - arm_smmu_sva_notifier_synchronize(); - platform_driver_unregister(drv); -} - static struct platform_driver arm_smmu_driver =3D { .driver =3D { .name =3D "arm-smmu-v3", @@ -4592,8 +4586,27 @@ static struct platform_driver arm_smmu_driver =3D { .remove =3D arm_smmu_device_remove, .shutdown =3D arm_smmu_device_shutdown, }; + +#ifndef CONFIG_ARM_SMMU_V3_PKVM +static void arm_smmu_driver_unregister(struct platform_driver *drv) +{ + arm_smmu_sva_notifier_synchronize(); + platform_driver_unregister(drv); +} + module_driver(arm_smmu_driver, platform_driver_register, arm_smmu_driver_unregister); +#else +/* + * Must be done after the hypervisor initializes at module_init() + * No need for unregister as this is a built in driver. + */ +static int arm_smmu_driver_register(void) +{ + return platform_driver_register(&arm_smmu_driver); +} +device_initcall_sync(arm_smmu_driver_register); +#endif /* !CONFIG_ARM_SMMU_V3_PKVM */ =20 MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DB7B31DD83 for ; Tue, 19 Aug 2025 21:52:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640363; cv=none; b=VBxeJgnsQ28UwPBntRaFbcNPAzhqrcUo7V/WBmswu1eBa+7ZWCh362XzOZFHyjYmZHh9d6lfhU8FBHZzlzmzq+vQL0JL/fLPpEoZJ77WhPLohiEZIPOLzEvQKJlo2m1hoWfxrO3QiO/LeqbD6Figsc6LF82YClGeEVVrE1hKBuM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640363; c=relaxed/simple; bh=XwAPL7cCuKQZyzoDsXxWaKOvvw7cbqN7O6Uc5m8Oi5Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MCsF9J8uh/dISgiLb0gGyixvAP6w/yM7QZe6Qqjel0+rAfZUfUTTPUknwZkcPzXwecmbasIfVt868qLvqVuOoqX8dvxjVx1OCoMU3anqIU0ngMsmtzRB1lPsCSjBqOq7rufJ3YvW0OgmWIUbObj57tflB9pK/QclJEAZkHnUzRI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tvIzT4R4; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tvIzT4R4" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b05d8d0so35806835e9.1 for ; Tue, 19 Aug 2025 14:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640359; x=1756245159; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xwu1qRO+eOJjyAwUn9pi1Q+O90cBvLjGAkOSG+zhKow=; b=tvIzT4R4DHc3wEsnYQWfzs7N7FeZ7NaTtsRz0QqGv86AgGXzbvHdlKs3u2H6dK4VrL JN393EN3m7bGEC4+embf12eYEMt4Z5g0xuNL02WT5dCxYUibk2908RLT/4O7izs5hqJ3 TjAo+xsMiGdXP7AzcMRHWn266edS6la2SLoFXlwKVi9gINgeOMTgSvhCzZcwtMp2i4kQ PTkrU/aUNSosozh5ue7fJWRZFvFCJRVoO+CDgdPLa+SCeu4IEoW8840ICOzDtrBEorkJ sHwq+1+dCUEQt8WS7YSJ4y62Nq4IqxcGtdmn8qZ136a/8noeuelfjTFtmbRROqv8DqHm O/Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640359; x=1756245159; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xwu1qRO+eOJjyAwUn9pi1Q+O90cBvLjGAkOSG+zhKow=; b=P9N60kmmxARTPRcVQzzr7LZlh5EPV0jcPplIbZggsK5soo7MUuxYUGS9OzGJoDn9av 0Owy1gS9T1mf691YLZ0A30osIGhGnLYNIGyNeNUdk9i4pyg96et7YBsM2SPsUpDLipwY uaawQN0JeXatOEBa3gOkSbE0g3rRmuJgAQeSU9Ak+k65D5JkM8Wd3XU80uDCfPfMbwFD x3phjOqipe0snKb8pEGFgwg5MxJgB7jsoSwl13jqX3B260pamlYMFDZZWSHHgfJUeXNr ZW5jx19xV8n4ea7Yc9RAEcurUZOgkAglyNTQz+XjtlctrN0Dlkxz5UbO/YMPGZDf9NwG m8Xw== X-Gm-Message-State: AOJu0YwUb+XqnQwQwcxuExH4sL/zB+BbN1elBH/NVbQEAhuC43E2+R9U a7RKReqine0Pm2ExgpAobWTs16cM8O6EwElecL4skMgzsHB1GGZ2iP0eGtHfGNmQHunx0BEAfh8 lSCtVrdFTM6iwkACh4B4KSk7uMchK1DrFcvRJ2+43sJaaXVa5p+VMXckrJU2+gvEJaWcIHVcl7B Zr/ZhB09MHABgUL1IONSK0xKYzNeiDu67POQdxOTpTJhsiA3mcLX2Bz0o= X-Google-Smtp-Source: AGHT+IEhkGtNHGU6UMmeB0BAIZ4sLir5Qlb/i2T9JqWqyDJv4XQTeVZhQ/vtD+ljWCO9zlGmXGV9Ovms10Lr6A== X-Received: from wmbhc5.prod.google.com ([2002:a05:600c:8705:b0:458:6799:d779]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3b05:b0:43d:745a:5a50 with SMTP id 5b1f17b1804b1-45b479ea5a9mr2802145e9.19.1755640359305; Tue, 19 Aug 2025 14:52:39 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:44 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-17-smostafa@google.com> Subject: [PATCH v4 16/28] iommu/arm-smmu-v3-kvm: Create array for hyp SMMUv3 From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" As the hypervisor has no access to firmware tables, the device discovery is done from the kernel, where it parses firmware tables and populates a list of devices to the hypervisor, which later takes over. At the moment only the device tree is supported. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 93 ++++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 13 +++ 2 files changed, 105 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index ac4eac6d567f..27ea39c0fb1f 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -7,6 +7,7 @@ #include #include =20 +#include #include =20 #include "arm-smmu-v3.h" @@ -14,6 +15,75 @@ =20 extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 +static size_t kvm_arm_smmu_count; +static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; + +static void kvm_arm_smmu_array_free(void) +{ + int order; + + order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); + free_pages((unsigned long)kvm_arm_smmu_array, order); +} + +/* + * The hypervisor have to know the basic information about the SMMUs + * from the firmware. + * This has to be done before the SMMUv3 probes and does anything meaningf= ul + * with the hardware, otherwise it becomes harder to reason about the SMMU + * state and we'd require to hand-off the state to the hypervisor at certa= in point + * while devices are live, which is complicated and dangerous. + * Instead, the hypervisor is interested in a very small part of the probe= path, + * so just add a separate logic for it. + */ +static int kvm_arm_smmu_array_alloc(void) +{ + int smmu_order; + struct device_node *np; + int ret; + int i =3D 0; + + kvm_arm_smmu_count =3D 0; + for_each_compatible_node(np, NULL, "arm,smmu-v3") + kvm_arm_smmu_count++; + + if (!kvm_arm_smmu_count) + return -ENODEV; + + smmu_order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)= ); + kvm_arm_smmu_array =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, = smmu_order); + if (!kvm_arm_smmu_array) + return -ENOMEM; + + /* Basic device tree parsing. */ + for_each_compatible_node(np, NULL, "arm,smmu-v3") { + struct resource res; + + ret =3D of_address_to_resource(np, 0, &res); + if (ret) + goto out_err; + kvm_arm_smmu_array[i].mmio_addr =3D res.start; + kvm_arm_smmu_array[i].mmio_size =3D resource_size(&res); + if (kvm_arm_smmu_array[i].mmio_size < SZ_128K) { + pr_err("SMMUv3(%s) has unsupported size(0x%lx)\n", np->name, + kvm_arm_smmu_array[i].mmio_size); + ret =3D -EINVAL; + goto out_err; + } + + if (of_dma_is_coherent(np)) + kvm_arm_smmu_array[i].features |=3D ARM_SMMU_FEAT_COHERENCY; + + i++; + } + + return 0; + +out_err: + kvm_arm_smmu_array_free(); + return ret; +} + size_t smmu_hyp_pgt_pages(void) { /* @@ -27,10 +97,31 @@ size_t smmu_hyp_pgt_pages(void) =20 static int kvm_arm_smmu_v3_register(void) { + int ret; + if (!is_protected_kvm_enabled()) return 0; =20 - return kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu_= ops)))); + ret =3D kvm_arm_smmu_array_alloc(); + if (ret) + return ret; + + ret =3D kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu= _ops)))); + if (ret) + goto out_err; + + /* + * These variables are stored in the nVHE image, and won't be accessible + * after KVM initialization. Ownership of kvm_arm_smmu_array will be + * transferred to the hypervisor as well. + */ + kvm_hyp_arm_smmu_v3_smmus =3D kvm_arm_smmu_array; + kvm_hyp_arm_smmu_v3_count =3D kvm_arm_smmu_count; + return ret; + +out_err: + kvm_arm_smmu_array_free(); + return ret; }; =20 core_initcall(kvm_arm_smmu_v3_register); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index f6ad91d3fb85..744ee2b7f0b4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -4,7 +4,20 @@ =20 #include =20 +/* + * Parameters from the trusted host: + * @mmio_addr base address of the SMMU registers + * @mmio_size size of the registers resource + * @features Features of SMMUv3, subset of the main driver + * + * Other members are filled and used at runtime by the SMMU driver. + * @base Virtual address of SMMU registers + */ struct hyp_arm_smmu_v3_device { + phys_addr_t mmio_addr; + size_t mmio_size; + void __iomem *base; + u32 features; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BD8C31DDAD for ; Tue, 19 Aug 2025 21:52:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640364; cv=none; b=mW3iQJeH+zka2ZwfSMgUIdrOP4tr9Rj0o3gB6DQkLUxWfFBBATh/Vl6wy88B/HXFINEi/GOjuijUcQqdVQ7fdy2nIfgVC/5zFeJd1w+yYbFhimtwaku2KKn95+BblMD6L7s+JmTd1FsUSPOIGPQxQYO+mdlLLeaDJN/Y0Xvw/+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640364; c=relaxed/simple; bh=3hKxHuU/XBxBW6Fz378wf7Rbaz+dVXKNYo3PuthQO9w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YpcSdCHWV3Q63/j0Wz45RZoibylDWG42psw/aNaDmbLQ/+lRf6kX7FwSA7/1594V/wL15923MkPQ73pggtkyjMSRocIPWTjlUUf9ScDTYnremXXx4s2t/YfsVMP0JOHEeJPBlrY8CjnqJukm8zd8FFanIFF7+j5gwmdQPE+ZMLU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gnL3s3Bj; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gnL3s3Bj" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3bb2fb3a489so2712236f8f.1 for ; Tue, 19 Aug 2025 14:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640361; x=1756245161; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=No5rZzETEpiSYrnAp/3rx5tnKsAcr1mN2J0hRF7T4gc=; b=gnL3s3BjUW9rL9hW/oo0fzhXa2d4CvS1OLnvLVU/ZhXaqzLA4LUbRS5bKRbqecUJta Gvc6HBsKpE79KVDqbFYns3AP1wkgKP27mq+CtPKYdBmdwoC85rWY9gO2515cZPZMS8UJ VWOoLWq7nq4rYCwCrM/ovgqqj7IFcFJTI5ymg/SkGMqp7FVszhiiubxK+Gt+wwFEEl3E tSB1gVpJK1621LjTnqthMZPb+fN3pFiVRiaMs2e4g5pZg6Y2qkcYXFiwK8izDtoihneP WzYQ1f8zTzFJ84A7UFxJfDJ6TVRXZqIvjCDueV57adt3HCBZ6pzaFwmRJZ5+XQvHSedY IAdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640361; x=1756245161; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=No5rZzETEpiSYrnAp/3rx5tnKsAcr1mN2J0hRF7T4gc=; b=jca6psYF52xCiqkKN3Ed538e7tpUa9HSQlSG7zXMjsgyZbAWd6pv1BOgpU5SkbGFJb niT/e8AfMRmwmb029imn5/DHu/ynpPTTlLgST1Z+w/OtDAiACHfNp+/0Ouv3CnLVrSXf GwILqDaY3ANCqrxOd1bO/Dp02PxpgQmKzozRD5hUZz8bcQTxrpp1PSppxnt6KQ9cWxaz lfvEWDT25VFKe9kdBA02uwVZa3mE6Ku88I5y+PveyqyvczxkG0fOFu6QLZOlvlc2Gyns qRytfHFTrqRMCJ94NXs8qs2R+/oVVbes5ycGOmFMVutG/ujla/TB48ubH5qbkZo0coLs 6dyw== X-Gm-Message-State: AOJu0Yw6aXsrRE9uluTsCdvK52ZIQ0/I9bXxaQTbkJAfK8x/AhXgsr1i QfWFJ+/lStEpwoN05HLs7ulYQUDDHUPyqe7rg+z39zqQoR7BBXjK9q1//1uY6ZXHspqqgPzLdVn SjTgzwn7zTt0B2g0O+fQ8iG5djZ3JMtUScbuXIMUTnfQNK7X9UpT7aBCPdyGGcewZ9Ip9BfkbX/ dkfQbz9JtL3exHmmsJxRsEtIU2pqDsAzCjXqiJJB/J2Nkkq1IB3wHmJNI= X-Google-Smtp-Source: AGHT+IFejZ2I69Eqlg88g8PwiPvQhSLUEBHZrSmUeIH84olhnRxAOXDRbUsOIXtbc95r4ewB/WJLIdwuVFgT6Q== X-Received: from wmbdw4.prod.google.com ([2002:a05:600c:6384:b0:459:dcf5:1b54]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:22c5:b0:3b8:f863:672c with SMTP id ffacd0b85a97d-3c32e13359bmr316604f8f.33.1755640360470; Tue, 19 Aug 2025 14:52:40 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:45 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-18-smostafa@google.com> Subject: [PATCH v4 17/28] iommu/arm-smmu-v3-kvm: Take over SMMUs From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Donate the array with SMMU description to the hypervisor as it can't be changed by the host after de-privileges. Also, donate the SMMU resources to the hypervisor. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 81 ++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index fa8b71152560..b56feae81dda 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -7,15 +7,94 @@ #include =20 #include +#include =20 #include "arm_smmu_v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +#define for_each_smmu(smmu) \ + for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ + (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ + (smmu)++) + +/* Transfer ownership of memory */ +static int smmu_take_pages(u64 phys, size_t size) +{ + WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size)); + return __pkvm_host_donate_hyp(phys >> PAGE_SHIFT, size >> PAGE_SHIFT); +} + +static void smmu_reclaim_pages(u64 phys, size_t size) +{ + WARN_ON(!PAGE_ALIGNED(phys) || !PAGE_ALIGNED(size)); + WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT)); +} + +/* Put the device in a state that can be probed by the host driver. */ +static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) +{ + int i; + size_t nr_pages =3D smmu->mmio_size >> PAGE_SHIFT; + + for (i =3D 0 ; i < nr_pages ; ++i) { + u64 pfn =3D (smmu->mmio_addr >> PAGE_SHIFT) + i; + + WARN_ON(__pkvm_hyp_donate_host_mmio(pfn)); + } +} + +static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) +{ + int i; + size_t nr_pages; + + if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) + return -EINVAL; + + nr_pages =3D smmu->mmio_size >> PAGE_SHIFT; + for (i =3D 0 ; i < nr_pages ; ++i) { + u64 pfn =3D (smmu->mmio_addr >> PAGE_SHIFT) + i; + + /* + * This should never happen, so it's fine to be strict to avoid + * complicated error handling. + */ + WARN_ON(__pkvm_host_donate_hyp_mmio(pfn)); + } + smmu->base =3D hyp_phys_to_virt(smmu->mmio_addr); + + return 0; +} + static int smmu_init(void) { - return -ENOSYS; + int ret; + struct hyp_arm_smmu_v3_device *smmu; + size_t smmu_arr_size =3D PAGE_ALIGN(sizeof(*kvm_hyp_arm_smmu_v3_smmus) * + kvm_hyp_arm_smmu_v3_count); + + kvm_hyp_arm_smmu_v3_smmus =3D kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus); + ret =3D smmu_take_pages(hyp_virt_to_phys(kvm_hyp_arm_smmu_v3_smmus), + smmu_arr_size); + if (ret) + return ret; + + for_each_smmu(smmu) { + ret =3D smmu_init_device(smmu); + if (ret) + goto out_reclaim_smmu; + } + + return 0; + +out_reclaim_smmu: + while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) + smmu_deinit_device(--smmu); + smmu_reclaim_pages(hyp_virt_to_phys(kvm_hyp_arm_smmu_v3_smmus), + smmu_arr_size); + return ret; } =20 static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3D70B3570A4 for ; Tue, 19 Aug 2025 21:52:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640365; cv=none; b=b6VoKHEFX9bnZTM6CMWkR1y1D8AcKwDILblocN1YGMhYTvuH3mlvSojcXjF9gP5iFZLMoge6LOaUAkmNuq0suSMbXVdlwT+YcIh+V5B7ECwHGI3ObfZwh+5+27fzQoV3zCtOXpt05dCi6dTwgGyVOa1sK8s7GMV0JrmICpVEDhE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640365; c=relaxed/simple; bh=cLjOanwFCYbBZ+DeQcNx28zYJugfOvEDf7zTDH1asV4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lThF9TWJE8akN8mQHN1961EENrc1N1yM2Yxe5gmJpR1b46z08GVp3e0rr8WOCDlFkDq6DQTcFwiyT/g0Rks5BuDRPBv5H1ny826eq6erKe0oko45vl9m+Viy/ouI3ufXDqdqIRmB5mMoKr2Hz4aP1ULWmqq2ZucxNdJN1Lwct7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aS/LaW+9; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aS/LaW+9" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3b9e418aab7so2448251f8f.3 for ; Tue, 19 Aug 2025 14:52:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640362; x=1756245162; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=y1XMrJFUsZlHqUSwfaqqxufwXrUFJ8lXJIFDET8NUKg=; b=aS/LaW+9jY+USyVRl2CiCUNy8cJs/8afqFCdSAB/t9ppzKlHdlj8AtYi7nDcrA99fC tV+pymMtc2naQrNNsu6nvpIjhcVDQAIR6PCDw/JD6LsOINJTo23VmDHKpRLBnuGwBc5/ xdL6rMl8QUYIzoqZd01YdVhx/zm6IjHB3QDwEpXFgYwIAnEd0Gf1w53q9uPbHG607cxO wckRJx22SnaEaWWqLUVwxt3M7hWkTBAe0jCh2gH34dq3Bb70IAU1l6WXjRdTwD9eEyB6 NTyc4IDcp8jmWo9ReZI8nT/PsdwqqqPXl8IILbPkyI68idGq2sjsfrqoVgFYzKQ3Js14 AN5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640362; x=1756245162; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=y1XMrJFUsZlHqUSwfaqqxufwXrUFJ8lXJIFDET8NUKg=; b=ZNL4Hwz882kEXvsDGV+8NIs5hsBFfTxWBgfdNbs+4DQZmljgxpVyE0I512Rws02hQv A3k0QWbY3UKigJTAGZe/pTJ0jkn84iPeOgqvNUbILjWs2eiFBVW19eV2j4ba1OLowVMz 2jqUAJofHx/2ySRX3QAvkq6zgh6FnrDajeBCUtAgWany5Uc0FFjaDf5viT9uBWN93JVc LDRz8lrzgocPqWElC0TIXks9rts4c5ByejBD1qEgxJ8XgXGJFhszMz6YrE/jrS88dSzt d2bZbnpf3EKX/nZJ+C/OmG4zyie915Uk/sq7LbLD06OMFuIcwJJOl7j//0ISni6aleHr m/Mw== X-Gm-Message-State: AOJu0YyBrdKsDrmqTpnM6agHIvTOZj39Cd/5LNUIE5YVkjBrN9Jh0i92 SzxhjlbyuwAppUY8PU3Y+b/iSzGbSPldI3EBNHpL4g/n1+PP98NtQ9D/iX4c+eH8pl2xuR/0LXM u4tdWT0OK1z761p58E/iMAKXwy5+5rGwM/JpXhWjk6Joh+IOg2haXlUwDU+cgwn2ZwtbtGAGEIn 4njF7WEkdiPVxYwvBnfZx36lpWt6CXnnQmNhR6CJQPI9ft7hDynwAa9AQ= X-Google-Smtp-Source: AGHT+IGRK4bp6sesJCXzi77ty+REwj3D43YiESI0ZMsE4l0swwRaEoRodh4XIVE78li6aGNDnGWJRk469Etvfw== X-Received: from wrqf13.prod.google.com ([2002:adf:f8cd:0:b0:3b7:8aed:a63d]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2309:b0:3b7:54b3:c512 with SMTP id ffacd0b85a97d-3c32e7fe680mr338963f8f.58.1755640361697; Tue, 19 Aug 2025 14:52:41 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:46 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-19-smostafa@google.com> Subject: [PATCH v4 18/28] iommu/arm-smmu-v3-kvm: Probe SMMU HW From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Probe SMMU features from the IDR register space, most of the logic is common with the kernel. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 1 + .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 57 ++++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 8 +++ 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 8ffcc2e32474..f0e1feee8a49 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -48,6 +48,7 @@ struct arm_smmu_device; #define IDR0_S2P (1 << 0) =20 #define ARM_SMMU_IDR1 0x4 +#define IDR1_ECMDQ (1 << 31) #define IDR1_TABLES_PRESET (1 << 30) #define IDR1_QUEUES_PRESET (1 << 29) #define IDR1_REL (1 << 28) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index b56feae81dda..e45b4e50b1e4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -10,6 +10,7 @@ #include =20 #include "arm_smmu_v3.h" +#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -45,9 +46,56 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_de= vice *smmu) } } =20 +/* + * Mini-probe and validation for the hypervisor. + */ +static int smmu_probe(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 reg; + + /* IDR0 */ + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); + smmu->features =3D smmu_idr0_features(reg); + + /* + * Some MMU600 and MMU700 have errata that prevent them from using nestin= g, + * not sure how can we identify those, so it's recommended not to enable = this + * drivers on such systems. + * And preventing any of those will be too restrictive + */ + if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1) || + !(smmu->features & ARM_SMMU_FEAT_TRANS_S2)) + return -ENXIO; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); + if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL | IDR1_ECMD= Q)) + return -EINVAL; + + smmu->sid_bits =3D FIELD_GET(IDR1_SIDSIZE, reg); + /* Follows the kernel logic */ + if (smmu->sid_bits <=3D STRTAB_SPLIT) + smmu->features &=3D ~ARM_SMMU_FEAT_2_LVL_STRTAB; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); + smmu->features |=3D smmu_idr3_features(reg); + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); + + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) + smmu->pgsize_bitmap |=3D 1ULL << 42; + else if (!smmu->oas) + smmu->oas =3D 48; + + smmu->ias =3D 64; + smmu->ias =3D min(smmu->ias, smmu->oas); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { - int i; + int i, ret; size_t nr_pages; =20 if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) @@ -64,8 +112,13 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) WARN_ON(__pkvm_host_donate_hyp_mmio(pfn)); } smmu->base =3D hyp_phys_to_virt(smmu->mmio_addr); - + ret =3D smmu_probe(smmu); + if (ret) + goto out_ret; return 0; +out_ret: + smmu_deinit_device(smmu); + return ret; } =20 static int smmu_init(void) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 744ee2b7f0b4..3550fa695539 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -12,12 +12,20 @@ * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers + * @ias IPA size + * @oas PA size + * @pgsize_bitmap Supported page sizes + * @sid_bits Max number of SID bits supported */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; size_t mmio_size; void __iomem *base; u32 features; + unsigned long ias; + unsigned long oas; + unsigned long pgsize_bitmap; + unsigned int sid_bits; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50C543277AA for ; Tue, 19 Aug 2025 21:52:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640366; cv=none; b=qRqf4uhkgrOjxoHibcwvSncw7og9CJfI6OuESz77ymJgcdpTLFrHMLzjVil9LMmDNCSi6bDg7LNgiIuwo4WPAlk78u9xxERkCRGozj40x0cmPK+FPtNgS/IO8TDqOwdyasrdcD560wPEDOqbEFE/ppyVTykgHNqMPtAFaxKU1e8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640366; c=relaxed/simple; bh=XhZYr4PdM87HvMy3LEorV9A6JwTdgzf6sEBGmHDWIMA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pawz7Vn/r2QeunSqm+aoBmv4d211kE7zbsaF4hz9i2EQ6MxYqvfKakTW0xC8TVbDCffn11qx6MMlf1Pi5jp10oBFY5naIKPwfW7fym5akHvtzoCd4dd7kGfmp+naQkrAt5eIwME5sTzAjD6iL1Jp097Ti6ZFcW6hfIuJ21Ijec8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yxY+NeyN; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yxY+NeyN" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-afcb7a2ba1dso474665766b.2 for ; Tue, 19 Aug 2025 14:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640363; x=1756245163; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PqV10pLXwwjj2OAipIDyVAMMO+u2/mFHIv6eR2zAaWM=; b=yxY+NeyN1pOd8voLfL1Kb1okCIgfeJnv4dLti386yzuBWwF63vkF+n5nqb6jX71Zoe dK/4uE2x6P03EA/mbGe8GbSkky7i9Mkktt4oT5AgOlPuXIuHcV82MAK2vwqEWO2i0C0w K9bHBuhz33KCU0Cmo3prELXJTRoSJe6p1P/9Kspw4gRaaMkJt63RyvWaoyj96VXyqsAu jVJ8OIvXlIvnUufbzZkwneO07uw0JASoTI/VYR77378QwHdowwgt8T2D7HouhMpeUvBh Gukfg7CuN7fwXwyVgH3DQe8xQd5cmXKWh4KheCgBvOj1SwJi1d3b2RYGQ2kuAnq2PYdy hKsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640363; x=1756245163; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PqV10pLXwwjj2OAipIDyVAMMO+u2/mFHIv6eR2zAaWM=; b=lkcdRsn8NSw9M2ARMYiAiLjAEXoNU/mSYUDYool91NkBdeMsIOjCnAree2USG6nz/p dTN/qWVMl7KIqcKRYQl+ppW9WGN9BcNDXMsgZlgFAoKjptkeOY/jel6aMDBNy7Qc2R/4 ug0xK+gnbxT1EJuPJjzHKHNDbvP5mbVZZNu+jK3XT4YpW5WJJ6Ph3OgfbDLY795sKPAn VUowVkeo9nUVjYOrLVh88p+J2kZNZZsDo01w5jU0+h7yZ0HGo4stofmEnop7u7SJy3GJ 5B+/8+nWEIOFbRtYBG9XRcQEIFuOjRTU0U0DvfKP3JkwnRRTiK3pG5Nmnp/TQ5Qwk6It d9Hw== X-Gm-Message-State: AOJu0Yz5DEQSClycYUTynko4SnfYq+pqf0PErBOJqBjU4B3dB5ansHmO RXkCJkNgm6OsM3fWUKUR9ExaGuLhhpNJUCR7IWHF6DavJ2hhHZ8QV7pfNTKdGEfWbEBYYI3+ADk SqYfsEsYudPzgG/r5TKjHXv3OjycJXJExLVJuxbfo3/iEvxg82L7E+urfUp4GslMY9eQHCvIemx Geqq9Gb3DHRvV3ViBj+X5CUQfNzQXRT/LQC97nu+68PCz3tI1bY+xtQHM= X-Google-Smtp-Source: AGHT+IGtl/M2mX2DFxMKjU0G/SGsKcUwRPA08PYakKSf6zJpbc9gAmUuIUtlYOT263WzT4l5+3VnMXOmobBQ9w== X-Received: from ejctl25.prod.google.com ([2002:a17:907:c319:b0:ae3:6f1c:e0e1]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:2d23:b0:ae6:e25b:2413 with SMTP id a640c23a62f3a-afdf0201e20mr32340066b.44.1755640362537; Tue, 19 Aug 2025 14:52:42 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:47 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-20-smostafa@google.com> Subject: [PATCH v4 19/28] iommu/arm-smmu-v3-kvm: Add MMIO emulation From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At the moment most registers are just passthrough, then in the next patches CMDQ/STE emulation will be added which inserts logic to some register access. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 125 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 2 files changed, 135 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index e45b4e50b1e4..32f199aeec9e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -8,6 +8,7 @@ =20 #include #include +#include =20 #include "arm_smmu_v3.h" #include "../arm-smmu-v3.h" @@ -140,6 +141,8 @@ static int smmu_init(void) goto out_reclaim_smmu; } =20 + BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); + return 0; =20 out_reclaim_smmu: @@ -150,6 +153,127 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, + struct user_pt_regs *regs, + u64 esr, u32 off) +{ + bool is_write =3D esr & ESR_ELx_WNR; + unsigned int len =3D BIT((esr & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); + int rd =3D (esr & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; + const u64 read_write =3D -1ULL; + const u64 no_access =3D 0; + u64 mask =3D no_access; + const u64 read_only =3D is_write ? no_access : read_write; + u64 val =3D regs->regs[rd]; + + switch (off) { + case ARM_SMMU_IDR0: + /* Clear stage-2 support, hide MSI to avoid write back to cmdq */ + mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); + WARN_ON(len !=3D sizeof(u32)); + break; + /* Passthrough the register access for bisectiblity, handled later */ + case ARM_SMMU_CMDQ_BASE: + case ARM_SMMU_CMDQ_PROD: + case ARM_SMMU_CMDQ_CONS: + case ARM_SMMU_STRTAB_BASE: + case ARM_SMMU_STRTAB_BASE_CFG: + case ARM_SMMU_GBPA: + mask =3D read_write; + break; + case ARM_SMMU_CR0: + mask =3D read_write; + WARN_ON(len !=3D sizeof(u32)); + break; + case ARM_SMMU_CR1: { + /* Based on Linux implementation */ + u64 cr2_template =3D FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB); + /* Don't mess with shareability/cacheability. */ + if (is_write) + WARN_ON(val !=3D cr2_template); + mask =3D read_write; + WARN_ON(len !=3D sizeof(u32)); + break; + } + /* + * These should be safe, just enforce RO or RW and size according to arch= itecture. + * There are some other registers that are not used by Linux as IDR2, IDR4 + * that won't be allowed. + */ + case ARM_SMMU_EVTQ_PROD + SZ_64K: + case ARM_SMMU_EVTQ_CONS + SZ_64K: + case ARM_SMMU_EVTQ_IRQ_CFG1: + case ARM_SMMU_EVTQ_IRQ_CFG2: + case ARM_SMMU_PRIQ_PROD + SZ_64K: + case ARM_SMMU_PRIQ_CONS + SZ_64K: + case ARM_SMMU_PRIQ_IRQ_CFG1: + case ARM_SMMU_PRIQ_IRQ_CFG2: + case ARM_SMMU_GERRORN: + case ARM_SMMU_GERROR_IRQ_CFG1: + case ARM_SMMU_GERROR_IRQ_CFG2: + case ARM_SMMU_IRQ_CTRLACK: + case ARM_SMMU_IRQ_CTRL: + case ARM_SMMU_CR0ACK: + case ARM_SMMU_CR2: + /* These are 32 bit registers. */ + WARN_ON(len !=3D sizeof(u32)); + fallthrough; + case ARM_SMMU_EVTQ_BASE: + case ARM_SMMU_EVTQ_IRQ_CFG0: + case ARM_SMMU_PRIQ_BASE: + case ARM_SMMU_PRIQ_IRQ_CFG0: + case ARM_SMMU_GERROR_IRQ_CFG0: + mask =3D read_write; + break; + case ARM_SMMU_IIDR: + case ARM_SMMU_IDR5: + case ARM_SMMU_IDR3: + case ARM_SMMU_IDR1: + case ARM_SMMU_GERROR: + WARN_ON(len !=3D sizeof(u32)); + mask =3D read_only; + }; + + if (WARN_ON(!mask)) + goto out_ret; + + if (is_write) { + if (len =3D=3D sizeof(u64)) + writeq_relaxed(regs->regs[rd] & mask, smmu->base + off); + else + writel_relaxed(regs->regs[rd] & mask, smmu->base + off); + } else { + if (len =3D=3D sizeof(u64)) + regs->regs[rd] =3D readq_relaxed(smmu->base + off) & mask; + else + regs->regs[rd] =3D readl_relaxed(smmu->base + off) & mask; + } + +out_ret: + return true; +} + +static bool smmu_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 addr) +{ + struct hyp_arm_smmu_v3_device *smmu; + bool ret; + + for_each_smmu(smmu) { + if (addr < smmu->mmio_addr || addr >=3D smmu->mmio_addr + smmu->mmio_siz= e) + continue; + hyp_spin_lock(&smmu->lock); + ret =3D smmu_dabt_device(smmu, regs, esr, addr - smmu->mmio_addr); + hyp_spin_unlock(&smmu->lock); + return ret; + } + return false; +} + static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) { } @@ -158,4 +282,5 @@ static void smmu_host_stage2_idmap(phys_addr_t start, p= hys_addr_t end, int prot) struct kvm_iommu_ops smmu_ops =3D { .init =3D smmu_init, .host_stage2_idmap =3D smmu_host_stage2_idmap, + .dabt_handler =3D smmu_dabt_handler, }; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 3550fa695539..dfeaed728982 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -4,6 +4,10 @@ =20 #include =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +#include +#endif + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -16,6 +20,7 @@ * @oas PA size * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported + * @lock Lock to protect SMMU */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -26,6 +31,11 @@ struct hyp_arm_smmu_v3_device { unsigned long oas; unsigned long pgsize_bitmap; unsigned int sid_bits; +#ifdef __KVM_NVHE_HYPERVISOR__ + hyp_spinlock_t lock; +#else + u32 lock; +#endif }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58E48334369 for ; Tue, 19 Aug 2025 21:52:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640367; cv=none; b=Zh9YPn55UzAIWmKzB13F5PC+HpuczwNQ0rAasO6lZl8mDobBbQpWOCTBDHCPpDRq+soxuNUkJUxRmzpVopzZ5qaT751gZ7hfHraHFacCQUD0AfoGQskVTBfoYg0bAQtkygN/PbPAmBr8zBWczCKAKksE3SWtiqLtZNRuOyj9WvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640367; c=relaxed/simple; bh=IDQoEPFCey8Q5YZAxs3yoAHr98Lqnt32fDIqn3IVCCE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=liaCt7KegIdJVVDVNaPLTYCrkw60kxuR22EqgKCGQITMVNYLRYvO7MeCE8NjTf9ZxGbZYzSVRnz0ZkKeXptDZTrcelRYTm6vntLR+k8XrGcE2r9bLjyNxDZNyte5vrpp3tF77AToS9ynDt0MzYw7kzUW8vjRGZjIrurkBgz+fbc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=OgBuaSPW; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="OgBuaSPW" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3b9edf41d07so3952815f8f.3 for ; Tue, 19 Aug 2025 14:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640363; x=1756245163; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=4/fU42/CqOBV85U7agBN+amVl2Ghkx92fsFwG9kBi+E=; b=OgBuaSPWeCdAQbskQwcdb4ua2l9NnFxAZEN25o9AlqlYofR01AZiTTVJYY4GvvmhM2 9qRkakJjqglD97qPaj6I+uof7QRvSxkTpjceV0cCZzKM7jzxCgUAwCLhnbMUgtldMha9 LdRDwsRAv8HPPi05OzQvo1b8W8hCLR6BB7p535v+2SRToAkYAYTcbYzmm/1xtJvtdHD+ c9KTpNbNxs+31pfQnbgopwC6cka78cgO1gqYeqTynaHJJKJRa+NCsckZna42uHRrqR+M yu6yu1f2e/5v5EpUQXNWMoROkVdED2e4/jKaYag7SYZ6h8kjFFhWX1o4TMeZvR7NuH3U P1fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640363; x=1756245163; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=4/fU42/CqOBV85U7agBN+amVl2Ghkx92fsFwG9kBi+E=; b=WpC0qC1vJQH4uBhoR50Zdu8dwoyza6/Vy14cLvA9lgZFlzyN4BG2ls0gJFl4FDuPUy BdpePCd9qf9VLri880/o8SO/EmzpW5Csne7lCmWwSctfK0TsPCEHKRFqlHzbL0ek/P6J ykvSJZP7FA3DsEiB5qF5QLEj0pcDPjPXtkAWuLRru8UJt/1EMW0kOfkCnNhtDreiKLMX WCpDn19pOrh7hF//xhEJ2N0sLfgf6Bw68MUm7ALMQdnr4wvlhKUu5m4WMNhVN03tQiCU qWxft80dYsG8SfIZCi6IwDqXOeGeGg4nZhki93Mx76yf8TiBWGpjxFnRkMRfun/0T2uB 5Qgw== X-Gm-Message-State: AOJu0Yw1m2MbhHlaYdS/30f1m0XyoFaassbwOaCFQ0/EF3CNrjwZe6pV coKfVKuer6tiNOezwoh1VZNmC/UXEeMTErnSQFjULoMNJzVsOJy/0PVyML5U0pHAHxw1XPRm759 yRO789zMSnPwkElOh7P6Bw24Ix49VJ9PQCb1TULIsPYi366BPrEO72eR1kJYUzs7eSIccFDrH3/ rYnDGxQBH5o0dRN+33znX0BAFVsDDIms5HOmawVGJJstFBAAJe/+IM+zg= X-Google-Smtp-Source: AGHT+IHPR2TOMegxsTQof55IYyVi+Xg9i8ImL2L9AnGlc4nKnoOPBojWzhoAgagvHESAt69mQX/N31I7JbKjMQ== X-Received: from wmbeb11.prod.google.com ([2002:a05:600c:678b:b0:459:e068:b510]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:64e7:0:b0:3a5:783f:528a with SMTP id ffacd0b85a97d-3c32eeb80e4mr358475f8f.59.1755640363546; Tue, 19 Aug 2025 14:52:43 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:48 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-21-smostafa@google.com> Subject: [PATCH v4 20/28] iommu/arm-smmu-v3-kvm: Shadow the command queue From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At boot allocate a command queue per SMMU which is used as a shadow by the hypervisor. The command queue size is 64K which is more than enough, as the hypervisor would consume all the entries per a command queue prod write, which means it can handle up to 4096 at a time. Then, the host command queue needs to be pinned in a shared state, so it can't be donated to VMs, and avoid tricking the hypervisor into accessing them. This is done each time the command queue is enabled, and undone each time the command queue is disabled. The hypervisor won=E2=80=99t access the host command queue when it is disab= led from the host. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 20 ++++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 108 +++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 8 ++ 3 files changed, 135 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index 27ea39c0fb1f..86e6c68aad4e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -13,6 +13,8 @@ #include "arm-smmu-v3.h" #include "pkvm/arm_smmu_v3.h" =20 +#define SMMU_KVM_CMDQ_ORDER 4 + extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 static size_t kvm_arm_smmu_count; @@ -58,6 +60,7 @@ static int kvm_arm_smmu_array_alloc(void) /* Basic device tree parsing. */ for_each_compatible_node(np, NULL, "arm,smmu-v3") { struct resource res; + void *cmdq_base; =20 ret =3D of_address_to_resource(np, 0, &res); if (ret) @@ -74,6 +77,23 @@ static int kvm_arm_smmu_array_alloc(void) if (of_dma_is_coherent(np)) kvm_arm_smmu_array[i].features |=3D ARM_SMMU_FEAT_COHERENCY; =20 + /* + * Allocate shadow for the command queue, it doesn't have to be the same + * size as the host. + * Only populate base_dma and llq.max_n_shift, the hypervisor will init + * the rest. + * We don't what size the host will choose at this point, the shadow copy + * will 64K which is a reasonable size. + */ + cmdq_base =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM= _CMDQ_ORDER); + if (!cmdq_base) { + ret =3D -ENOMEM; + goto out_err; + } + + kvm_arm_smmu_array[i].cmdq.base_dma =3D virt_to_phys(cmdq_base); + kvm_arm_smmu_array[i].cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAG= E_SHIFT - + CMDQ_ENT_SZ_SHIFT; i++; } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 32f199aeec9e..d3ab4b814be4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -11,7 +11,6 @@ #include =20 #include "arm_smmu_v3.h" -#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -34,6 +33,35 @@ static void smmu_reclaim_pages(u64 phys, size_t size) WARN_ON(__pkvm_hyp_donate_host(phys >> PAGE_SHIFT, size >> PAGE_SHIFT)); } =20 +/* + * CMDQ, STE host copies are accessed by the hypervisor, we share them to + * - Prevent the host from passing protected VM memory. + * - Having them mapped in the hyp page table. + */ +static int smmu_share_pages(phys_addr_t addr, size_t size) +{ + int i; + size_t nr_pages =3D PAGE_ALIGN(size) >> PAGE_SHIFT; + + for (i =3D 0 ; i < nr_pages ; ++i) + WARN_ON(__pkvm_host_share_hyp((addr + i * PAGE_SIZE) >> PAGE_SHIFT)); + + return hyp_pin_shared_mem(hyp_phys_to_virt(addr), hyp_phys_to_virt(addr += size)); +} + +static int smmu_unshare_pages(phys_addr_t addr, size_t size) +{ + int i; + size_t nr_pages =3D PAGE_ALIGN(size) >> PAGE_SHIFT; + + hyp_unpin_shared_mem(hyp_phys_to_virt(addr), hyp_phys_to_virt(addr + size= )); + + for (i =3D 0 ; i < nr_pages ; ++i) + WARN_ON(__pkvm_host_unshare_hyp((addr + i * PAGE_SIZE) >> PAGE_SHIFT)); + + return 0; +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -94,6 +122,41 @@ static int smmu_probe(struct hyp_arm_smmu_v3_device *sm= mu) return 0; } =20 +/* + * The kernel part of the driver will allocate the shadow cmdq, + * which is different from the one used by the driver. + * This function only donates it. + */ +static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_size; + int ret; + enum kvm_pgtable_prot prot =3D PAGE_HYP; + + cmdq_size =3D (1 << (smmu->cmdq.llq.max_n_shift)) * + CMDQ_ENT_DWORDS * 8; + + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + prot |=3D KVM_PGTABLE_PROT_NORMAL_NC; + + ret =3D ___pkvm_host_donate_hyp(smmu->cmdq.base_dma >> PAGE_SHIFT, + PAGE_ALIGN(cmdq_size) >> PAGE_SHIFT, prot); + if (ret) + return ret; + + smmu->cmdq.base =3D hyp_phys_to_virt(smmu->cmdq.base_dma); + smmu->cmdq.prod_reg =3D smmu->base + ARM_SMMU_CMDQ_PROD; + smmu->cmdq.cons_reg =3D smmu->base + ARM_SMMU_CMDQ_CONS; + smmu->cmdq.q_base =3D smmu->cmdq.base_dma | + FIELD_PREP(Q_BASE_LOG2SIZE, smmu->cmdq.llq.max_n_shift); + smmu->cmdq.ent_dwords =3D CMDQ_ENT_DWORDS; + memset(smmu->cmdq.base, 0, cmdq_size); + writel_relaxed(0, smmu->cmdq.prod_reg); + writel_relaxed(0, smmu->cmdq.cons_reg); + writeq_relaxed(smmu->cmdq.q_base, smmu->base + ARM_SMMU_CMDQ_BASE); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { int i, ret; @@ -116,7 +179,13 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) ret =3D smmu_probe(smmu); if (ret) goto out_ret; + + ret =3D smmu_init_cmdq(smmu); + if (ret) + goto out_ret; + return 0; + out_ret: smmu_deinit_device(smmu); return ret; @@ -153,6 +222,27 @@ static int smmu_init(void) return ret; } =20 +static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_CMDQEN, smmu->cr0); +} + +static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_size; + + smmu->cmdq_host.llq.max_n_shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2S= IZE; + cmdq_size =3D (1 << smmu->cmdq_host.llq.max_n_shift) * CMDQ_ENT_DWORDS * = 8; + WARN_ON(smmu_share_pages(smmu->cmdq_host.q_base & Q_BASE_ADDR_MASK, cmdq_= size)); +} + +static void smmu_emulate_cmdq_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_size =3D cmdq_size =3D (1 << smmu->cmdq_host.llq.max_n_shift)= * CMDQ_ENT_DWORDS * 8; + + WARN_ON(smmu_unshare_pages(smmu->cmdq_host.q_base & Q_BASE_ADDR_MASK, cmd= q_size)); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -174,6 +264,13 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, break; /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: + + /* Not allowed by the architecture */ + WARN_ON(is_cmdq_enabled(smmu)); + if (is_write) + smmu->cmdq_host.q_base =3D val; + mask =3D read_write; + break; case ARM_SMMU_CMDQ_PROD: case ARM_SMMU_CMDQ_CONS: case ARM_SMMU_STRTAB_BASE: @@ -182,6 +279,15 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, mask =3D read_write; break; case ARM_SMMU_CR0: + if (is_write) { + bool last_cmdq_en =3D is_cmdq_enabled(smmu); + + smmu->cr0 =3D val; + if (!last_cmdq_en && is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_enable(smmu); + else if (last_cmdq_en && !is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_disable(smmu); + } mask =3D read_write; WARN_ON(len !=3D sizeof(u32)); break; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index dfeaed728982..330da53f80d0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -8,6 +8,8 @@ #include #endif =20 +#include "../arm-smmu-v3.h" + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -21,6 +23,9 @@ * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported * @lock Lock to protect SMMU + * @cmdq CMDQ as observed by HW + * @cmdq_host Host view of the command queue + * @cr0 Last value of CR0 */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -36,6 +41,9 @@ struct hyp_arm_smmu_v3_device { #else u32 lock; #endif + struct arm_smmu_queue cmdq; + struct arm_smmu_queue cmdq_host; + u32 cr0; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3979D334381 for ; Tue, 19 Aug 2025 21:52:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640368; cv=none; b=hbek/Zs3uebxW6g3LLsKjcNPma2sCBUsaSI5JR69OgxXuxHkX1/fQxVKdTV8O8fMBdHjm/sPZ4kRQLzLkJTOxdmIe2AiqMlWSCxlmloMyzZHdLN5+pQJbNhENPdj0ZyuXVcbsm8FgVqKhiSjvPowNCTuFxBjtZUoSeLhStr9RV4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640368; c=relaxed/simple; bh=BdXTwmydT3BtN8HebN1Pb6HB/oKVhfmvfEyBuFMb0uM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BOAj7KxZzr7fXZ5hMYYLY378r0pFWWByOgwTEZBfId+CFh+MB5l0j1ZUig5RJNmioBtZA+R6GroVtuRs5utN5tMIg+1vtgPhJUbZO0vWqMaeDbiFRiY+/Q2yPH8VMUTtF1ynHmTIIfpxgzKFdUiRyNez4dUr+iuppoOx7kfgQwg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Px0wOzM8; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Px0wOzM8" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0c5377so25525015e9.3 for ; Tue, 19 Aug 2025 14:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640364; x=1756245164; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZyRtXnhnhppM44IFifDXYrWtilG94lpK59HvMcWGonE=; b=Px0wOzM8IVtiZ1ZqfqewqgtOC6HOfYvH/HU7ILKM6M1OhUDas5OXz/7TGXF5BmakUy vEyX/XrqwF69gUiVcRr6ty0tdIi+x4m6fohVJidk6JdecEDLKM6LDsK4C6o2u0S3hPcY 5NnPrnYLGIO6OHVEnuJA0lJ4eBYifmLdN1AFmxSP+IXavnPmrxXeuPO9uwKVgS9UB8u8 qhfN2IGPBysLvRlJOBW8AJEJ1pIpSWRxo197xOKgaInTyQ5GV8WgxLsSqGJiGS1LC/2o h97p5PGuRlg+dPnMLLETLoN0ir9qzXQ3kB/ZyIdLlY4sIIn6VWvkI8G2PhayFqtEWYks nHkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640364; x=1756245164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZyRtXnhnhppM44IFifDXYrWtilG94lpK59HvMcWGonE=; b=pQycM16uRIixMHJ2jpdY4PjR7Tg4ig5ApBLXEinym4qrOLJ5fMt1t6O+aR5wM0pXce MgrtAKd4g3A5K4cupt+axXNnuh6MWx6IHDgWQyuLEkCMlU6qkhW+moDjaGdh9f4GwgMe 0+E1ht9LH06XYCidXOTj2cFRhceBtr2ntdLlEYQr4wRNF9fsVFU3cZhbhX+dtuQ432JO +z/RYiDWraect8nUYEhW03A+k3ZNY4O17y+gbHJ0A2hWmWebBFUe89c99XyUXQBe67Xa 1lhCS6ZK/AvDi8qZLWoIGkNDezwyQFujytSl1yx9U+AM0JrR1bxeZoY1RQ7PuwTQ9mjC /cZQ== X-Gm-Message-State: AOJu0Yw2JBkpea2fD2zSDW+588n2a2tt76AU9BbJ5wpEo3faQ1be9vLl 6QIyjM2HFWt+vG9IK0Fv/vVjbgR8ZA6WThIG//nswN5yDILTbgkdb4sZzna0p8bz8+uvzPlQDh3 k5ez3KxWeIxrOQtPwRbn29c7JESWJa1JcPtMxNA7t7tiY4MDA4hjq1AgPkgK0K2K2GC2XGwSmN/ 0fGuBm/Yt3dAzf2InnO6N6jRGCmpbso1QdSTKvs6T8BnjbNkCVk2rze6U= X-Google-Smtp-Source: AGHT+IGr5ioioVJcsD/j/B2rJ42P01JBGG14bJdGLQMTv0vagaiysEKhVuqLi/9sRGavTW3nFD47MLqZZDL7iQ== X-Received: from wmqd6.prod.google.com ([2002:a05:600c:34c6:b0:458:c02c:3ccd]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:b86:b0:456:eb9:5236 with SMTP id 5b1f17b1804b1-45b479bdf64mr3584515e9.15.1755640364362; Tue, 19 Aug 2025 14:52:44 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:49 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-22-smostafa@google.com> Subject: [PATCH v4 21/28] iommu/arm-smmu-v3-kvm: Add CMDQ functions From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add functions to access the command queue, there are 2 main usage: - Hypervisor's own commands, as TLB invalidation, would use functions as smmu_send_cmd(), which creates and sends a command. - Add host commands to the shadow command queue, after being filtered, these will be added with smmu_add_cmd_raw. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 98 +++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index d3ab4b814be4..554229e466f3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -20,6 +20,33 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ (smmu)++) =20 +/* + * Wait until @cond is true. + * Return 0 on success, or -ETIMEDOUT + */ +#define smmu_wait(_cond) \ +({ \ + int __ret =3D 0; \ + u64 delay =3D pkvm_time_get() + ARM_SMMU_POLL_TIMEOUT_US; \ + \ + while (!(_cond)) { \ + if (pkvm_time_get() >=3D delay) { \ + __ret =3D -ETIMEDOUT; \ + break; \ + } \ + } \ + __ret; \ +}) + +#define smmu_wait_event(_smmu, _cond) \ +({ \ + if ((_smmu)->features & ARM_SMMU_FEAT_SEV) { \ + while (!(_cond)) \ + wfe(); \ + } \ + smmu_wait(_cond); \ +}) + /* Transfer ownership of memory */ static int smmu_take_pages(u64 phys, size_t size) { @@ -62,6 +89,77 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t s= ize) return 0; } =20 +static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_full(llq); +} + +static bool smmu_cmdq_empty(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_empty(llq); +} + +static void smmu_add_cmd_raw(struct hyp_arm_smmu_v3_device *smmu, + u64 *cmd) +{ + struct arm_smmu_queue *q =3D &smmu->cmdq; + struct arm_smmu_ll_queue *llq =3D &q->llq; + + queue_write(Q_ENT(q, llq->prod), cmd, CMDQ_ENT_DWORDS); + llq->prod =3D queue_inc_prod_n(llq, 1); + writel_relaxed(llq->prod, q->prod_reg); +} + +static int smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *ent) +{ + int ret; + u64 cmd[CMDQ_ENT_DWORDS]; + + ret =3D smmu_wait_event(smmu, !smmu_cmdq_full(&smmu->cmdq)); + if (ret) + return ret; + + ret =3D arm_smmu_cmdq_build_cmd(cmd, ent); + if (ret) + return ret; + + smmu_add_cmd_raw(smmu, cmd); + return 0; +} + +static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_CMD_SYNC, + }; + + ret =3D smmu_add_cmd(smmu, &cmd); + if (ret) + return ret; + + return smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq)); +} + +__maybe_unused +static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd) +{ + int ret =3D smmu_add_cmd(smmu, cmd); + + if (ret) + return ret; + + return smmu_sync_cmd(smmu); +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C49933439B for ; Tue, 19 Aug 2025 21:52:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640369; cv=none; b=HvQIsYUIFZKGZXkEQevsL3U/aNfhlqHRee8OyFbJKI418Hzy8yELFEN7UrCTEING0FwY36+SPZRa9gGSMBPWBrvSEs9u5GADGQ2S4pYxyB7edwroYA+ExODFiIVjw0gQtUJy+m87lZ3FtOeyncuS0lWK0WcJ0QPoNGeEwGcG9P8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640369; c=relaxed/simple; bh=x6+sOLq4FPZ4CMFUSWd3Y8U0nD5a7Aqh64+XSIM/UM8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ig1r4UG+I8i3pPSv2DHhIjRKm4BdS7+TZiljk9sqefs+4Y8eecHXkCLpJdU2qGJtidOQ9EHJGDXCvRmSEHacJmmKQwVZ1moNE7O0fNrYJtoj11UtKX8TP1oa//3dew+FMDT+IiCZpkyaW3ulEwen2UGnEE+IlZP/MaZwgy/2T4A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JPE6G6gP; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JPE6G6gP" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-afcb7338319so416396166b.0 for ; Tue, 19 Aug 2025 14:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640365; x=1756245165; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=s2OiNO3KzQO8B0w4ffRojG8RRfuYkjFyK2nm+fAlxiU=; b=JPE6G6gP6019p8hLhpjNUoZjXTziIKld8Wu5Q/a4mxE4NtYt4GiWSiiq/kioBJ89HW IaX2zzlXqPIX9iYmNo4igxAtnYNlzRQzu6acW7nAYa43throeoydZ6NtRuOnhcq3Zi0L uZ8qGePAs1+v0TTfJ+lYf9xNjEX2ZQYSe3KXlo3jMpYZXyz4ad85P6sN/wrGlO33OdFk qtl0y2ViDmFyvg6mr47abLJtOjVRCQMOcp6MkRn7pvQZNDFpTeHQq5rMf1/paSD/7Anv CnW25A/kc3EhTaA4LafOzr1bysena40E1lkgtXV9A3MmlvesNo/hJzkel2kPK3O/o8VE 042w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640365; x=1756245165; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=s2OiNO3KzQO8B0w4ffRojG8RRfuYkjFyK2nm+fAlxiU=; b=QDuGUWNVIZopLtVA68Z5Tu6irrUpwoZt3svpt9/0Rwd31gvFEzGuBgffDHa+HOMoiu ZvCZNC0Qcka/HoJZ0Q6C/I+Xt/nGnRxnhQLJGr26J4tCps+ScIl10sRIRkWTbjUFUuWe FJzIIYw0RldT6dXdUZOpfCY5g2GocMb8vFlALraDoLEkxIpJiQUg9RGekkGU3QxADLto s+kH4YmWlASEutSBHrUaKZso/tFzP5scc8X3WqqGHI1WFP93t/M/gKVThM5LatkwsJoq 1WZQ/CGW9W2XFK0Y3KRSfDOjxaB9+Y2Ya6/wx6Da+tQcHMmhNLzKwwGlH1S250SbG67S JdZQ== X-Gm-Message-State: AOJu0YzczCZVdLq4kzCd8HTYGoCeGmQFNoj1NKnBvZmTfdpdSPv4k8T6 b9rrdYvsskVtmdjdfSOXLrFZwlgfnfdxL5uCoZcrV/4m31qHB8P+Bo2RWlsIo0o4H+nLB+Kxlfv 2MFs7Q1FE2Es/7oe2Ga1+MYgJFdPD3bBDYc54pPUXpyJKQQyICSN+0oTM+er1Jfxb3lSBL5oxDy XGTpLjy48DpuQogLKeVbuKYTFkH8j3LOacoSaCTYRibNFmmas/G71sK/0= X-Google-Smtp-Source: AGHT+IHd99Hbl/EO+Fv3HJ9l/C88pz7JGJRqrSKU6LiBDjnLkGYw7fqHFm99BcFANhnvyjy+geOwjTDrJ/MdzA== X-Received: from ejcuy5.prod.google.com ([2002:a17:907:d105:b0:ae9:c7c1:c2a2]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:7ba3:b0:ae0:d9f3:9131 with SMTP id a640c23a62f3a-afdf00bda1bmr33581766b.6.1755640365290; Tue, 19 Aug 2025 14:52:45 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:50 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-23-smostafa@google.com> Subject: [PATCH v4 22/28] iommu/arm-smmu-v3-kvm: Emulate CMDQ for host From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don=E2=80=99t allow access to the command queue from the host: - ARM_SMMU_CMDQ_BASE: Only allowed to be written when CMDQ is disabled, we use it to keep track of the host command queue base. Reads return the saved value. - ARM_SMMU_CMDQ_PROD: Writes trigger command queue emulation which sanitises and filters the whole range. Reads returns the host copy. - ARM_SMMU_CMDQ_CONS: Writes move the sw copy of the cons, but the host can= =E2=80=99t skip commands once submitted. Reads return the emulated value and the err= or bits in the actual cons. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 108 +++++++++++++++++- 1 file changed, 105 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 554229e466f3..10c6461bbf12 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -325,6 +325,88 @@ static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_dev= ice *smmu) return FIELD_GET(CR0_CMDQEN, smmu->cr0); } =20 +static bool smmu_filter_command(struct hyp_arm_smmu_v3_device *smmu, u64 *= command) +{ + u64 type =3D FIELD_GET(CMDQ_0_OP, command[0]); + + switch (type) { + case CMDQ_OP_CFGI_STE: + /* TBD: SHADOW_STE*/ + break; + case CMDQ_OP_CFGI_ALL: + { + /* + * Linux doesn't use range STE invalidation, and only use this + * for CFGI_ALL, which is done on reset and not on an new STE + * being used. + * Although, this is not architectural we rely on the current Linux + * implementation. + */ + WARN_ON((FIELD_GET(CMDQ_CFGI_1_RANGE, command[1]) !=3D 31)); + break; + } + case CMDQ_OP_TLBI_NH_ASID: + case CMDQ_OP_TLBI_NH_VA: + case 0x13: /* CMD_TLBI_NH_VAA: Not used by Linux */ + { + /* Only allow VMID =3D 0*/ + if (FIELD_GET(CMDQ_TLBI_0_VMID, command[0]) =3D=3D 0) + break; + break; + } + case 0x10: /* CMD_TLBI_NH_ALL: Not used by Linux */ + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_EL2_VA: + case CMDQ_OP_TLBI_EL2_ASID: + case CMDQ_OP_TLBI_S12_VMALL: + case 0x23: /* CMD_TLBI_EL2_VAA: Not used by Linux */ + /* Malicous host */ + return WARN_ON(true); + case CMDQ_OP_CMD_SYNC: + if (FIELD_GET(CMDQ_SYNC_0_CS, command[0]) =3D=3D CMDQ_SYNC_0_CS_IRQ) { + /* Allow it, but let the host timeout, as this should never happen. */ + command[0] &=3D ~CMDQ_SYNC_0_CS; + command[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + command[1] &=3D ~CMDQ_SYNC_1_MSIADDR_MASK; + } + break; + } + + return false; +} + +static void smmu_emulate_cmdq_insert(struct hyp_arm_smmu_v3_device *smmu) +{ + u64 *host_cmdq =3D hyp_phys_to_virt(smmu->cmdq_host.q_base & Q_BASE_ADDR_= MASK); + int idx; + u64 cmd[CMDQ_ENT_DWORDS]; + bool skip; + + if (!is_cmdq_enabled(smmu)) + return; + + while (!queue_empty(&smmu->cmdq_host.llq)) { + /* Wait for the command queue to have some space. */ + WARN_ON(smmu_wait_event(smmu, !smmu_cmdq_full(&smmu->cmdq))); + + idx =3D Q_IDX(&smmu->cmdq_host.llq, smmu->cmdq_host.llq.cons); + /* Avoid TOCTOU */ + memcpy(cmd, &host_cmdq[idx * CMDQ_ENT_DWORDS], CMDQ_ENT_DWORDS << 3); + skip =3D smmu_filter_command(smmu, cmd); + if (!skip) + smmu_add_cmd_raw(smmu, cmd); + queue_inc_cons(&smmu->cmdq_host.llq); + } + + /* + * Wait till consumed, this can be improved a bit by returning to the host + * while flagging the current offset in the command queue with the host, + * this would be maintained from the hyp entering command or when the + * host issuing another read to cons. + */ + WARN_ON(smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq))); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { size_t cmdq_size; @@ -360,17 +442,37 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); WARN_ON(len !=3D sizeof(u32)); break; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: =20 /* Not allowed by the architecture */ WARN_ON(is_cmdq_enabled(smmu)); if (is_write) smmu->cmdq_host.q_base =3D val; - mask =3D read_write; - break; + else + regs->regs[rd] =3D smmu->cmdq_host.q_base; + goto out_ret; case ARM_SMMU_CMDQ_PROD: + if (is_write) { + smmu->cmdq_host.llq.prod =3D val; + smmu_emulate_cmdq_insert(smmu); + } else { + regs->regs[rd] =3D smmu->cmdq_host.llq.prod; + } + goto out_ret; case ARM_SMMU_CMDQ_CONS: + if (is_write) { + /* Not allowed by the architecture */ + WARN_ON(is_cmdq_enabled(smmu)); + smmu->cmdq_host.llq.cons =3D val; + } else { + /* Propagate errors back to the host.*/ + u32 cons =3D readl_relaxed(smmu->base + ARM_SMMU_CMDQ_CONS); + u32 err =3D CMDQ_CONS_ERR & cons; + + regs->regs[rd] =3D smmu->cmdq_host.llq.cons | err; + } + goto out_ret; + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: case ARM_SMMU_STRTAB_BASE_CFG: case ARM_SMMU_GBPA: --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C5AF35A281 for ; Tue, 19 Aug 2025 21:52:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640370; cv=none; b=J7FE6fJGMd1toSxzv4XWM4U6fskMQUL1H9IZf3qKCoDBWiSGm7iJRsajBXgjA6Y0jCe/TdfnJmf2alYrD52ADVJjD5hYP7FLnqVCpiZ39HTPz/CKp0a6qUad50oRVGmSHDLe3EIg/EMtBoSIRDJzP4+IiNR3/YmSCpm5bJyZ3Xg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640370; c=relaxed/simple; bh=v+RLrZmyOSZ5aJXqdEVrQV5WjWOHlbStlf6eN2dzLn8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=G8zFvCi6FYMqBbm7IcmQnx6g6GhJzRtrmoehJQDh91U1LuPFki052JOhEFp4bXsQpmdtFkLLSJFHYQghdFvuYPCQ8om+W/ZmWZKZb1dKVUJm7yWTwTrOP99ndPDRaUcN3WuJT6SlPKQHgQ4eVVDWB0nE+teRbYXT41eUtTvi1Ys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MrRRrHxi; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MrRRrHxi" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3b9e418aab7so2448276f8f.3 for ; Tue, 19 Aug 2025 14:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640366; x=1756245166; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GpTNTvVpyesRrKSyybEzym6wQoCbw5IvodwWn1PTcVU=; b=MrRRrHxiW881VSQpk0HCnQmOYwI4V4cBgC6VbTdhHAq+RtH5MlH4o4kZurHuMzkox9 96zCuvi0lgRCF9id3KgbktWbbDU8uaOMvTSTFYXPNiiOBL3mHn/6waBYLYT4tqbpDycr gYaqLHO1UF8Jo+vz/m2wJfNAJV76JnaT/jtlh/K10Tzw7sj7MrOXdUUA0M6lmSuJ0kb5 +c8EN2pWpl8KQC9v4CVFed2qA4Eavno8xxjEMy3BXSbEvfR2FtsnN52blbqHKeuYtfu9 7UKz/Owkie7TzLLuQYtEaQ0po6LHAVegQG8yipfchpKSMScFQYHWLN+gy3o4M3uz9Sdb lFQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640366; x=1756245166; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GpTNTvVpyesRrKSyybEzym6wQoCbw5IvodwWn1PTcVU=; b=HbWrkwOEv3ttg0A54/TDxDzuUiK9DSTtTX0odhmF5LLW7agSqKxn9pzFzta2C+EL7U Nlwkkc9GrmKC/5vfgrk5xi6y1kFmMjwMSO3CId2sU/R4XCFI1zsa0CK4txls6/urluDZ gzRNFoVI+s3ZiVlxrvorV14uVqmNbZoNj3sAk4HLWYqitEZw7RHUehFrDJTh9mGeWlM1 lfBsxEkXuhbuxkXIaliW0FnOj5K3rv1QiGDDGk+8zDR8c1NCzA75x137krfGKz7UJRj/ 5lYaevOQTajGhwJz1D3C7ZcR6iIK3CCGkAYTw9mmrqXBlAAXvWrjiEgCBNP03ccgx/ID sjeA== X-Gm-Message-State: AOJu0Yw3k8SLUTsBCb7IFUb9RM3ZyRlmbLBJkX8IhHRVUP9lWEsg297Q J3W/6YPuAZVHU9fGEt1JmdaGv+F1XAqNa9d/Zdvj7zRvt2nSaqV0jG9fTAk3Y0gmCVTqYFsLM0G 3eATxPC8lqOlWEi8tLPZJt2AG2zA4SWivqRT8BhKPgrjg4a1uPTrF2f+VwjsT0Ld43hRerKrHAH KAACTAMLHhVKj+BHIHylnIiuDg8ZOAVEqrI/Kh6RK2BhhaERwN+6rXKjg= X-Google-Smtp-Source: AGHT+IHbIVYXrILomuPPQlVOuNyID7IU9oeIvKa7J2aqvl3fa71ziBavxg/PMhhybuVw4KlLKOLHJjlemW2pCg== X-Received: from wrbee12.prod.google.com ([2002:a05:6000:210c:b0:3b9:1c6e:dc37]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:200d:b0:3b4:9b82:d42c with SMTP id ffacd0b85a97d-3c32c52f02dmr296609f8f.17.1755640366147; Tue, 19 Aug 2025 14:52:46 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:51 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-24-smostafa@google.com> Subject: [PATCH v4 23/28] iommu/arm-smmu-v3-kvm: Shadow stream table From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch allocates the shadow stream table per SMMU. We choose the size of that table to be 1MB which is the max size used by host in the case of 2 levels. In this patch all the host writes are still paththrough for bisectibility, that is changed next where CFGI commands will be trapped and used to update the shadow copy hypervisor that will be used by HW. Similar to the command queue, the host stream table is shared/unshared each time the SMMU is enabled/disabled. Handling of L2 tables is also done in the next patch when the shadowing is added. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 13 +- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 111 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 3 files changed, 133 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index 86e6c68aad4e..821190abac5a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -14,6 +14,8 @@ #include "pkvm/arm_smmu_v3.h" =20 #define SMMU_KVM_CMDQ_ORDER 4 +#define SMMU_KVM_STRTAB_ORDER (get_order(STRTAB_MAX_L1_ENTRIES * \ + sizeof(struct arm_smmu_strtab_l1))) =20 extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 @@ -60,7 +62,7 @@ static int kvm_arm_smmu_array_alloc(void) /* Basic device tree parsing. */ for_each_compatible_node(np, NULL, "arm,smmu-v3") { struct resource res; - void *cmdq_base; + void *cmdq_base, *strtab; =20 ret =3D of_address_to_resource(np, 0, &res); if (ret) @@ -94,6 +96,15 @@ static int kvm_arm_smmu_array_alloc(void) kvm_arm_smmu_array[i].cmdq.base_dma =3D virt_to_phys(cmdq_base); kvm_arm_smmu_array[i].cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAG= E_SHIFT - CMDQ_ENT_SZ_SHIFT; + + strtab =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_ST= RTAB_ORDER); + if (!strtab) { + ret =3D -ENOMEM; + goto out_err; + } + kvm_arm_smmu_array[i].strtab_dma =3D virt_to_phys(strtab); + kvm_arm_smmu_array[i].strtab_size =3D PAGE_SIZE << SMMU_KVM_STRTAB_ORDER; + i++; } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 10c6461bbf12..d722f8ce0635 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -15,6 +15,14 @@ size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +/* strtab accessors */ +#define strtab_log2size(smmu) (FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, (smmu)-= >host_ste_cfg)) +#define strtab_size(smmu) ((1 << strtab_log2size(smmu)) * STRTAB_STE_DWORD= S * 8) +#define strtab_host_base(smmu) ((smmu)->host_ste_base & STRTAB_BASE_ADDR_M= ASK) +#define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) +#define strtab_l1_size(smmu) ((1 << (strtab_log2size(smmu) - strtab_split(= smmu))) * \ + (sizeof(struct arm_smmu_strtab_l1))) + #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ @@ -255,6 +263,48 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + u32 reg; + enum kvm_pgtable_prot prot =3D PAGE_HYP; + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + prot |=3D KVM_PGTABLE_PROT_NORMAL_NC; + + ret =3D ___pkvm_host_donate_hyp(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT, prot); + if (ret) + return ret; + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + unsigned int last_sid_idx =3D + arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1); + + cfg->l2.l1tab =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->l2.l1_dma =3D smmu->strtab_dma; + cfg->l2.num_l1_ents =3D min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES); + + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_2LVL) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, + ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) | + FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT); + } else { + cfg->linear.table =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->linear.ste_dma =3D smmu->strtab_dma; + cfg->linear.num_ents =3D 1UL << smmu->sid_bits; + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_LINEAR) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits); + } + + writeq_relaxed((smmu->strtab_dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_R= A, + smmu->base + ARM_SMMU_STRTAB_BASE); + writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { int i, ret; @@ -282,6 +332,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_init_strtab(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -320,6 +374,11 @@ static int smmu_init(void) return ret; } =20 +static bool is_smmu_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_SMMUEN, smmu->cr0); +} + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -407,6 +466,39 @@ static void smmu_emulate_cmdq_insert(struct hyp_arm_sm= mu_v3_device *smmu) WARN_ON(smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq))); } =20 +static void smmu_update_ste_shadow(struct hyp_arm_smmu_v3_device *smmu, bo= ol enabled) +{ + size_t strtab_size; + u32 fmt =3D FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->host_ste_cfg); + + /* Linux doesn't change the fmt nor size of the strtab in the run time. */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + strtab_size =3D strtab_l1_size(smmu); + WARN_ON(fmt !=3D STRTAB_BASE_CFG_FMT_2LVL); + WARN_ON((strtab_split(smmu) !=3D STRTAB_SPLIT)); + } else { + strtab_size =3D strtab_size(smmu); + WARN_ON(fmt !=3D STRTAB_BASE_CFG_FMT_LINEAR); + WARN_ON(FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, smmu->host_ste_cfg) > + smmu->sid_bits); + } + + if (enabled) + WARN_ON(smmu_share_pages(strtab_host_base(smmu), strtab_size)); + else + WARN_ON(smmu_unshare_pages(strtab_host_base(smmu), strtab_size)); +} + +static void smmu_emulate_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + smmu_update_ste_shadow(smmu, true); +} + +static void smmu_emulate_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + smmu_update_ste_shadow(smmu, false); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { size_t cmdq_size; @@ -474,19 +566,38 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, goto out_ret; /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + WARN_ON(is_smmu_enabled(smmu)); + smmu->host_ste_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_STRTAB_BASE_CFG: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + WARN_ON(is_smmu_enabled(smmu)); + smmu->host_ste_cfg =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_GBPA: mask =3D read_write; break; case ARM_SMMU_CR0: if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); + bool last_smmu_en =3D is_smmu_enabled(smmu); =20 smmu->cr0 =3D val; if (!last_cmdq_en && is_cmdq_enabled(smmu)) smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + if (!last_smmu_en && is_smmu_enabled(smmu)) + smmu_emulate_enable(smmu); + else if (last_smmu_en && !is_smmu_enabled(smmu)) + smmu_emulate_disable(smmu); } mask =3D read_write; WARN_ON(len !=3D sizeof(u32)); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 330da53f80d0..cf85e5efdd9e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -15,6 +15,8 @@ * @mmio_addr base address of the SMMU registers * @mmio_size size of the registers resource * @features Features of SMMUv3, subset of the main driver + * @strtab_dma Phys address of stream table + * @strtab_size Stream table size * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers @@ -26,6 +28,9 @@ * @cmdq CMDQ as observed by HW * @cmdq_host Host view of the command queue * @cr0 Last value of CR0 + * @host_ste_cfg Host stream table config + * @host_ste_base Host stream table base + * @strtab_cfg Stream table as seen by HW */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -44,6 +49,11 @@ struct hyp_arm_smmu_v3_device { struct arm_smmu_queue cmdq; struct arm_smmu_queue cmdq_host; u32 cr0; + dma_addr_t strtab_dma; + size_t strtab_size; + u64 host_ste_cfg; + u64 host_ste_base; + struct arm_smmu_strtab_cfg strtab_cfg; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF4CB341659 for ; Tue, 19 Aug 2025 21:52:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640370; cv=none; b=gVP7nMUySDRg5S28l9kzTfCIXQlrbzz5fXaw03xRViq4oNKX2L6c4SsagZXn1CL/t9NOjEKNRkzNx6VEeTPU2yujkIrszBRbDmwepLY/TunTl4+l0qePnnfZ9zt5G2jJdkxQu1skInM3Dslxqjn/1CnJFCi0HayKqYWGzhd5Brw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640370; c=relaxed/simple; bh=9B+IUAI36ouSms7t4Ms79PVbPypli/gAzLkEi+fAwyI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fh/ypokh8uY0ItJEknI5DFGBZ+jxSyL3Jaa68GB7WVsdrOF2j4E+mx/1s5h/72kJYXP2hQ3xT1CphyRB67xGH93Ybg3oId2hLlqHU2AGs9O1D2uoQ8b3+KghMB+N4G3atKoegHFiODV21a8WcUZfiq96MwL2afk+m7DSvJKyDLQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zJHRB2q3; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zJHRB2q3" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b05a59cso35208925e9.1 for ; Tue, 19 Aug 2025 14:52:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640367; x=1756245167; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7sk4h0o5FTOh9oFFMoej2yRTt0xkrjtRUSeSqNCk5ts=; b=zJHRB2q3ZX+9d7kIA3mdk61U2NYTaDHAranQRSZNDl/askCQ1zcm8wvXgZDQmHKYBi +ZaA8PoJpkfW7gbunnMU7jXkOqPa3N5exse3rkXP06xWuhiAGXBCq3dS+rSMe7T+w6X+ zJE27nh1of/n4Je2SUEwqz8WFO7UYXFmn+n4sNkPw3PYvJtSVO5dXQQ4CVvtQLXUXtye S2oFofnR0j9RrtpK+jDQPecpN9/M73TH5QCE2L2VKSR5HlbBVr5lbFJ+mkaHjmhZAO8g s+qr3NF3SbcY9Mip5Q90Of/hJdVunqLZ28tsC8tNDzMoUf46EDk8KMWmypJv8gLLey2t iGrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640367; x=1756245167; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7sk4h0o5FTOh9oFFMoej2yRTt0xkrjtRUSeSqNCk5ts=; b=A67Ljx+h9NR53MyaYp6Sfx3hmcJJxiN8GtciqvIY18jeTtkFYEpGJWp6Blq+wMQHMP N1r6UF3JpZCzwUKYcY9XAYscjRYru+fmcNX7hkDDgzdefFahpDiE40/IpcnB6j6A486+ g3DBPYi4lXn0x45BkQ6xNtk/euMPCwWOpS0T9xWAhkMdAAtsNv115uT36dMFtwn0oOUW D69tKFDYv5L/hZqbPzURXg9G5uMzXa4p9f5N1xyAYS5P/83CsNrc12GSWD33Tpk91T6H Sa1lI73GlkoJ7aJmwfx1kVm62hIKif39cd8Su23pmXhrYYanRz7e3hyy5C2nPbgaT7o+ MceA== X-Gm-Message-State: AOJu0YzfZaC2qeLTTN0mzPzgYVvSSFzkwnrQAr1yNDA+djh47WgvOEXk NqY5Cs/EQcA4n98VE5OrQwJNWJvAsrPUFM2TrUSG3jnDKGSh2RIUR+tgsAQIowjc+/ygmoQZQmU t1rpCL0nr0ufHRpcYNJTlS9pelk7bmrys9wxAHxpeyx/GQ3CpEOqR4xN2nOhV+y5zWp71qFhciI 7uvDt4aTMi0ETdfncGtvZ7ckA+yTCzwopTjAnARISyp4HENRTeuuukYOA= X-Google-Smtp-Source: AGHT+IFUpQvsGMxlMZxN5e8l+1BLUo+tGugwis/W3Zc75woWAnm0Ggc8nFVugp0gt+/gCvYhg6NiMTal3YfIkg== X-Received: from wmbhc18.prod.google.com ([2002:a05:600c:8712:b0:458:bb2b:d73e]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2dc6:b0:3bd:148e:212 with SMTP id ffacd0b85a97d-3c32ca0701emr371514f8f.14.1755640366978; Tue, 19 Aug 2025 14:52:46 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:52 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-25-smostafa@google.com> Subject: [PATCH v4 24/28] iommu/arm-smmu-v3-kvm: Shadow STEs From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch adds STE emulation, this is done when the host sends the CFGI_STE command. In this patch we copy the STE as is to the shadow owned by the hypervisor, in the next patch, stage-2 page table will be attached. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 95 +++++++++++++++++-- 1 file changed, 89 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index d722f8ce0635..0f890a7d8db3 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -22,6 +22,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; #define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) #define strtab_l1_size(smmu) ((1 << (strtab_log2size(smmu) - strtab_split(= smmu))) * \ (sizeof(struct arm_smmu_strtab_l1))) +#define strtab_hyp_base(smmu) ((smmu)->features & ARM_SMMU_FEAT_2_LVL_STRT= AB ? \ + (u64 *)(smmu)->strtab_cfg.l2.l1tab :\ + (u64 *)(smmu)->strtab_cfg.linear.table) =20 #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ @@ -263,6 +266,83 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +/* Get an STE for a stream table base. */ +static struct arm_smmu_ste *smmu_get_ste_ptr(struct hyp_arm_smmu_v3_device= *smmu, + u32 sid, u64 *strtab) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + struct arm_smmu_ste *table =3D (struct arm_smmu_ste *)strtab; + + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)strtab; + u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); + struct arm_smmu_strtab_l2 *l2ptr; + + if (WARN_ON(l1_idx >=3D cfg->l2.num_l1_ents) || + !(l1tab[l1_idx].l2ptr & STRTAB_L1_DESC_SPAN)) + return NULL; + + l2ptr =3D hyp_phys_to_virt(l1tab[l1_idx].l2ptr & STRTAB_L1_DESC_L2PTR_MA= SK); + /* Two-level walk */ + return &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; + } + if (WARN_ON(sid >=3D cfg->linear.num_ents)) + return NULL; + return &table[sid]; +} + +static int smmu_shadow_l2_strtab(struct hyp_arm_smmu_v3_device *smmu, u32 = sid) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + struct arm_smmu_strtab_l2 *l2table; + u32 idx =3D arm_smmu_strtab_l1_idx(sid); + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + u64 l1_desc_host =3D host_ste_base[idx]; + struct arm_smmu_strtab_l1 *l1_desc =3D &cfg->l2.l1tab[idx]; + + l2table =3D kvm_iommu_donate_pages(get_order(sizeof(*l2table))); + if (!l2table) + return -ENOMEM; + arm_smmu_write_strtab_l1_desc(l1_desc, hyp_virt_to_phys(l2table)); + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + kvm_flush_dcache_to_poc(l1_desc, sizeof(*l1_desc)); + + /* + * Now set the hyp l1 to a shared state. + * As mentioned in smmu_reshadow_ste() Linux never clears L1 ptrs, + * so no need to handle that case. Otherwise, we need to unshare + * the tables and emulate STE clear. + */ + smmu_share_pages(l1_desc_host & STRTAB_L1_DESC_L2PTR_MASK, sizeof(*l2tabl= e)); + return 0; +} + +static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid= , bool leaf) +{ + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + u64 *hyp_ste_base =3D strtab_hyp_base(smmu); + struct arm_smmu_ste *host_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, host_st= e_base); + struct arm_smmu_ste *hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_= base); + int i; + + /* + * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this + * is a 2 level table and reshadow of l2. + * Also Linux never clears l1 ptr, that needs to free the old shadow. + */ + if (WARN_ON(!leaf || !host_ste_ptr)) + return; + + /* If host is valid and hyp is not, means a new L1 installed. */ + if (!hyp_ste_ptr) { + WARN_ON(smmu_shadow_l2_strtab(smmu, sid)); + hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_base); + } + + for (i =3D 0; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], host_ste_ptr->data[i]); +} + static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) { int ret; @@ -390,8 +470,13 @@ static bool smmu_filter_command(struct hyp_arm_smmu_v3= _device *smmu, u64 *comman =20 switch (type) { case CMDQ_OP_CFGI_STE: - /* TBD: SHADOW_STE*/ + { + u32 sid =3D FIELD_GET(CMDQ_CFGI_0_SID, command[0]); + u32 leaf =3D FIELD_GET(CMDQ_CFGI_1_LEAF, command[1]); + + smmu_reshadow_ste(smmu, sid, leaf); break; + } case CMDQ_OP_CFGI_ALL: { /* @@ -564,23 +649,21 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, regs->regs[rd] =3D smmu->cmdq_host.llq.cons | err; } goto out_ret; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ WARN_ON(is_smmu_enabled(smmu)); smmu->host_ste_base =3D val; } - mask =3D read_write; - break; + goto out_ret; case ARM_SMMU_STRTAB_BASE_CFG: if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ WARN_ON(is_smmu_enabled(smmu)); smmu->host_ste_cfg =3D val; } - mask =3D read_write; - break; + goto out_ret; + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: mask =3D read_write; break; --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72F7634166B for ; Tue, 19 Aug 2025 21:52:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640371; cv=none; b=p2KTkUorHsiSfmT8pONdPn14Sa+0M/iY2grqPG5BUEQz1DnX7twuDgzRWKNyvodWwJ7orh5UDTwcbMKRYxXmRzfFEIe4sW7YzOtJTq5x3Y0g9AGFRmxDYjc6xxkxic06ZMXOEpp4FuT/hE0b4QHUtenSqIjr8/VpjyPuj/HpKZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640371; c=relaxed/simple; bh=zmVhO1Fx//55azUTuwuZDUNjyqH7HeyRTi3vIAgq+T8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RenQWfMnFAz20iLLJLAP1RUCPLC4H9HLpePzgYylulCk9ZilFylcQ/SQPoiCzkb/FtxuyaJOxLHJs/N1BabHMNzUDs+jEc5H5Fwgpo3PXoOet7v753dnjFMoNwJpwvUyF7F6FmiDSXRjiwgLHnQH0nfk8Z6MtjIGMy+Z2zsj1/Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Yb7N7ANf; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Yb7N7ANf" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0ccb6cso29972755e9.3 for ; Tue, 19 Aug 2025 14:52:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640368; x=1756245168; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=w5Xrv+yZchFUzMTWX2SO30dKLg2MrQIxC8g2iyFC05M=; b=Yb7N7ANfQdl5j86gjVcUYci1d7MgNzZ8w0kDghryd+CkVOnRmTqTOrqBosBGT8Wz95 +kDA7IToZFAd1M5BDZ9v6q3qSXv2xevJz0QBn3JRMTQMgObbzcaq9R+Vy6h47VBCn/1r wKfvb4FVy98N8kGRxWlGJDpmEmAUMbvLkPG7Fxu639qqo7EJAIKSGJoAe6gVyhbybkfV Al7piyAvSS64fJlnKBs3ycajlq9l9zFv7W0M4x9WlQpkkK20p9HXz2JGJlwTPhW0eB8W EZxtQoUNYEfHSDWUMFssw+QcftGdmrEEPChSl0X+d+tbJIQls1FwIKzM+ZZzPJeEHhMY v4oQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640368; x=1756245168; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=w5Xrv+yZchFUzMTWX2SO30dKLg2MrQIxC8g2iyFC05M=; b=Z6YnvdtxOXTFXiT78GNKn/ZQWNe/GV22oSPNf9lB5VXi+r3jghyc4L2+198oLIrzg9 5RGNCGbvHJ3izEXPNSdr6u9/HArTZblw1CfaVQJSvr5No3fFlv2djO4VWgrA8kRlRLbD JpwkFjlHjTmW5Syu+n5LyeF5WUUHYbghUR9xcqe0unL9IVdBhEfvLi1q+aKVbvgFh2I7 pAwyA7TOj3XH18cDCjUlQOJhm9dH6Q8Pc2aGBYPCywq1uGVNPdEATSJ+nesZ4lItxYwX BYj/672q87aEr5U2aymcVTA0et7wcnEzcZULSBqmsyp86zmvvTYcXMfEnQdrpKQKnK7A QNng== X-Gm-Message-State: AOJu0YySTdJeGKAFnkCeWZU/zjwgdpuEFWutLCMFraE0V8NXE2ac1jnq Czr6tEH0GopGK1zntaNQ4M39j6AcSQ5Yzw4NI7QedUJ26sMDAThd1EMhX5Roftv+EcwSOXoAftb CtSiCrc1XVoeP7nrGUPzEh8vcPRQzUsXp4kCRoHD90RX8Mun++0abpwC9yAa7165O6SRqGR6cfl yyabZoBWUw0qIFC3bf3Z4FRO/u9mxKS4BUAz1NimPGPIg8y8cI+RdXA1Q= X-Google-Smtp-Source: AGHT+IEkIYLt8jKXZ6jlGQreyOBP5MPpq4IR1pMtFairXlGsN9A2lAviNYTKCG1j1G3gzMwAxgyEa1tGajcbqg== X-Received: from wmqa4.prod.google.com ([2002:a05:600c:3484:b0:459:dbaa:93b0]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3ba7:b0:458:bb0e:4181 with SMTP id 5b1f17b1804b1-45b47d904a1mr1114035e9.10.1755640367876; Tue, 19 Aug 2025 14:52:47 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:53 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-26-smostafa@google.com> Subject: [PATCH v4 25/28] iommu/arm-smmu-v3-kvm: Emulate GBPA From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The last bit of emulation is GBPA. it must be always set to ABORT, as when the SMMU is disabled it=E2=80=99s not allowed for the host to bypass the SMMU. That =E2=80=98s is done by setting the GBPA to ABORT at init time, when the= host: - Writes, we ignore the write and save the value without the UPDATE bit. - Reads, return the saved value. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 21 ++++++++++++++++--- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 2 ++ 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 0f890a7d8db3..db9d9caaca2c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -100,6 +100,13 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t= size) return 0; } =20 +static int smmu_abort_gbpa(struct hyp_arm_smmu_v3_device *smmu) +{ + writel_relaxed(GBPA_UPDATE | GBPA_ABORT, smmu->base + ARM_SMMU_GBPA); + /* Wait till UPDATE is cleared. */ + return smmu_wait(readl_relaxed(smmu->base + ARM_SMMU_GBPA) =3D=3D GBPA_AB= ORT); +} + static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -416,6 +423,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_abort_gbpa(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -663,10 +674,14 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, smmu->host_ste_cfg =3D val; } goto out_ret; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: - mask =3D read_write; - break; + if (is_write) + smmu->gbpa =3D val & ~GBPA_UPDATE; + else + regs->regs[rd] =3D smmu->gbpa; + + WARN_ON(len !=3D sizeof(u32)); + goto out_ret; case ARM_SMMU_CR0: if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index cf85e5efdd9e..aab585dd9fd8 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -31,6 +31,7 @@ * @host_ste_cfg Host stream table config * @host_ste_base Host stream table base * @strtab_cfg Stream table as seen by HW + * @gbpa Last value of GBPA from the host */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -54,6 +55,7 @@ struct hyp_arm_smmu_v3_device { u64 host_ste_cfg; u64 host_ste_base; struct arm_smmu_strtab_cfg strtab_cfg; + u32 gbpa; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B5D031DDAD for ; Tue, 19 Aug 2025 21:52:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640372; cv=none; b=Z+8PFwlRS1XhHKGkGHn1DT9NWCTPVUPhxX4QIRUi6isLmtonDxb5DOceUg1u1vN702WyVhV/w88TCQvHudc4K+enQq9Tw1uOaxeE0u1jNGMOdZBUBnzPNxm4RjKq2CZx8uqRHM76pws+HXZB+zEGny11tat40dzZjD9bsmji7l4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640372; c=relaxed/simple; bh=5TTl062DHH7yHuyRacgTvFvIS4u56vNnv9ZfS0wBre8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=M0tljLFrZQ8QQQFeENgfrivj3hH6EU2E1fIg3i+TamWG888Ws809Ji8cUq3rQo5OyJD4BHYhIyKJ6pdqYXOKAr5H9FoDFu64W6k2P3/mEZNVfhjQBJ8rwsRmB8iquww8PnYRsK0K7JP8TRk1SKuqBkpZwNeFi1/XNZ9m3m5d/9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xXzjiuqm; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xXzjiuqm" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45b467f5173so4286085e9.3 for ; Tue, 19 Aug 2025 14:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640369; x=1756245169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kQKSjbAOLoARkZeCpW60ga+O1M1bwmJjgvA8RqrWhDQ=; b=xXzjiuqm8zQiMeM2j9o5+ReV6d9s/k+602rQusVBbkStvT81j4CnJZdXEnbcaRL1Dz ycHesIU256QDRu0ASIeJbjQ36DvMCxK+/BxwGfmkLZKFFPVft5Q899Rid4X5MsBbhYZu DQz/ZiPsz9kyXfvz9ERIouzyZqvkfCiSQG9HqgF7WEpxmc46YM0zyo+MLXqGihEeHBbp cjYzI/u4ojsXXYu3JmAKECrL7y1XD1wfzmBoCwz6Flghaq0eLQSvFbqCfVmoCpuZ9vYH a2CQ9jxV311aLhquO9oAI9QcnxqVGXz2O5gCijjg3+SHpkGCT7ipAWAVWyB6CEiEOvla EAfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640369; x=1756245169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kQKSjbAOLoARkZeCpW60ga+O1M1bwmJjgvA8RqrWhDQ=; b=LgxYxyUUeufpG42xiHBshWsuEfmCVwRbXdEi7GsUhh3wn0WtWbm/P4mWSrV9iVJTY2 IAWxNpnbDseixNTE2vGADCZcK17wQxGuIo4rYww1/XBELa229mqBkHI3hVrjVOBAc5e+ tFtOz/mjtwqLa9cHAAEwX3Erny5uiad6yo0v7ygbThOriI2NWZn1ubA/xHV20/kgAV8+ K2wzQG8mle+0zukiNW3qg/5pJIOH/5NEjg4uHG7ApE9xLhBsA6+mxKciefK2maR1PFhd 0/AV1E3uWJALzC3IJeYJzSH1I6g1TdO+0ujt2DHsdKFh9TCsav3zFTiQedvsXoaxxASD rtfw== X-Gm-Message-State: AOJu0YxHRE9nSn7Kd7hr0zJ0TMlysLwE/8Wk7ZZVgNhCEu40Kx2422ib H3O5096zWFlkj+eD4MoQF6S0HDneGahJBXSVZhp3IxdRzmsJl1sx/Z7BiN+BvdlF2rjpGtHn9+P 19l24Fj2oJI4Zc8PzNFBWnXZNdSD10w5o5q7ac0h4w66qGCmH4SlH93OWU379TVqJpm/TvDTr9h 0MD0ltvBxPLrdoPRS4gFh6v+4KhHw3rpdVWbMRcSWqv+lUzIiRCnzykM0= X-Google-Smtp-Source: AGHT+IEeSFl8aQhXA74KHxoebV5C+2+fcJhttjMY5RrNdpsd+ZTDbX53WFbbWrnver3BkDgaNxReEk7xnsBkcQ== X-Received: from wmsr18.prod.google.com ([2002:a05:600c:8b12:b0:459:dcaa:ec9e]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5250:b0:459:10de:551e with SMTP id 5b1f17b1804b1-45b479fd300mr2516095e9.27.1755640368689; Tue, 19 Aug 2025 14:52:48 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:54 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-27-smostafa@google.com> Subject: [PATCH v4 26/28] iommu/arm-smmu-v3-kvm: Support io-pgtable From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add hooks needed to support io-pgtable-arm, mostly about memory allocation. Also add a function to allocate s2 64 bit page table. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 4 +- .../arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c | 64 +++++++++++++++++++ drivers/iommu/io-pgtable-arm.c | 2 +- drivers/iommu/io-pgtable-arm.h | 11 ++++ 4 files changed, 79 insertions(+), 2 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index c71c96262378..10090be6b067 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -34,7 +34,9 @@ hyp-obj-y +=3D $(lib-objs) HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 =20 hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ - $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-hyp.o \ + $(HYP_SMMU_V3_DRV_PATH)/pkvm/io-pgtable-arm-hyp.o \ + $(HYP_SMMU_V3_DRV_PATH)/../../io-pgtable-arm.o =20 ## ## Build rules for compiling nVHE hyp code diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c b/driv= ers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c new file mode 100644 index 000000000000..6cf9e5bb76e7 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/io-pgtable-arm-hyp.c @@ -0,0 +1,64 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2022 Arm Ltd. + */ +#include + +#include "../../../io-pgtable-arm.h" + +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct io_pgtable *iop; + + if (fmt !=3D ARM_64_LPAE_S2) + return NULL; + + iop =3D arm_64_lpae_alloc_pgtable_s2(cfg, cookie); + iop->fmt =3D fmt; + iop->cookie =3D cookie; + iop->cfg =3D *cfg; + + return &iop->ops; +} + +void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, + struct io_pgtable_cfg *cfg, void *cookie) +{ + void *addr; + + addr =3D kvm_iommu_donate_pages(get_order(size)); + + if (addr && !cfg->coherent_walk) + kvm_flush_dcache_to_poc(addr, size); + + return addr; +} + +void __arm_lpae_free_pages(void *addr, size_t size, struct io_pgtable_cfg = *cfg, + void *cookie) +{ + if (!cfg->coherent_walk) + kvm_flush_dcache_to_poc(addr, size); + + kvm_iommu_reclaim_pages(addr); +} + +void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, + struct io_pgtable_cfg *cfg) +{ + if (!cfg->coherent_walk) + kvm_flush_dcache_to_poc(ptep, sizeof(*ptep) * num_entries); +} + +/* At the moment this is only used once, so rounding up to a page is not r= eally a problem. */ +void *__arm_lpae_alloc_data(size_t size, gfp_t gfp) +{ + return kvm_iommu_donate_pages(get_order(size)); +} + +void __arm_lpae_free_data(void *p) +{ + return kvm_iommu_reclaim_pages(p); +} diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 2ca09081c3b0..211f6d54b902 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -947,7 +947,7 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg *cfg= , void *cookie) return NULL; } =20 -static struct io_pgtable * +struct io_pgtable * arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie) { u64 sl; diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index 7d9f0b759275..194c3e975288 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -78,8 +78,19 @@ void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, void *cookie); void *__arm_lpae_alloc_data(size_t size, gfp_t gfp); void __arm_lpae_free_data(void *p); +struct io_pgtable * +arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg *cfg, void *cookie); #ifndef __KVM_NVHE_HYPERVISOR__ #define __arm_lpae_virt_to_phys __pa #define __arm_lpae_phys_to_virt __va +#else +#include +#define __arm_lpae_virt_to_phys hyp_virt_to_phys +#define __arm_lpae_phys_to_virt hyp_phys_to_virt +#undef WARN_ONCE +#define WARN_ONCE(condition, format...) WARN_ON(1) +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie); #endif /* !__KVM_NVHE_HYPERVISOR__ */ #endif /* IO_PGTABLE_ARM_H_ */ --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4742B34DCD1 for ; Tue, 19 Aug 2025 21:52:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640373; cv=none; b=g3aqhdJzSuu3MlRuklXsJl40/InNTKeDj6igXl0IweA9Uiz9Mx0krLt1TsJ7jpaSh6Au+i8YmF+fzZj8fKv2qPpQzBuS/3kZZoNXJNuSd2gMViYo8cxMvXWkP85IOW9+eqQYbOh0Q2vZo8HgMhtIsq6PiPuZwRHOUcvABg12gK0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640373; c=relaxed/simple; bh=pfRmcOYtcTv/5f7EQKmMO5NvR2S3xc1JUwdDGpRDJlc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ss5MxuFJqMyjOF1uKvtN2PYFMtvI649IBt6AiJDEBjzrPmBuZC80phNKPeFXuVVxvXtyf4mBjyZWTnj+NDjqQGac7m5q34ikGJbzciENwghAnIctyCnDni53CqY0CR5UnM9f4DhYBlBb2E8JV50r9gU/VC1J+rpufCRGmcJa0F8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GjwPFrnQ; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GjwPFrnQ" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45a1b0015aaso22034355e9.0 for ; Tue, 19 Aug 2025 14:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640369; x=1756245169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ykEL2Fz+GpEcyIy6HAU7e6ae/ECc9Mn8DWNDC0CC2vc=; b=GjwPFrnQUK4ftvdnmmkw1TceVNb/Obmjh3mseEXH4yH//9rOaQiNMswk73dhhPSdC4 PiRzancW8kpTCCcRpIBAGVhUAm3sgBSAOqWLkI8hHtfOep9VF7lhFTZZy7DWjQnkp0J/ n4hHYzk5eJQOwbQ9Ztob/AcoxRd4zkmwkVCQtSL3x85X+mtDn6oZpKTOeijnSgYZZOyZ R/gBJTniuusyKHNQE5zaSVYMHbYi3eTrF+86uPoXKvUR6J1ebDoSK7JvUt0kF4hMsPZ2 27BJEE7zrLd4M8UlGV3FFz6HTOOlrjhCckHJmGa3hmRwVW0EroZkNH5ndKbsEh+PvJoW x3jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640369; x=1756245169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ykEL2Fz+GpEcyIy6HAU7e6ae/ECc9Mn8DWNDC0CC2vc=; b=UtzSQ3xI1EqeJEQjUaegsGw/9f63ZbKEQstXrs1jXXDzw7+J3EVi+mne+JNSOfVltW NDWKNmpt5skJCsDllp9PvJ0Y9j0Xr66ayQliyqtI5Wev3tOFJXiLk+v+o6E8NUnXKUk+ x/kzIb+GZYAvCBz+s2PQQl7vHcov5DptwFDyPAFFFn02oSPLdExOh/Rle/msqSU7U2i5 sFFFz7uTYudvJop6TOB0PCnhOAV8Jnd2LMCpmV28oPcfvIfYNRpR0B8GNFy8ylsjApJ9 RC8TU/xCgHDMgy2PJit1KF/w3L6CRWGZfWP5UGGbzvGxmtdcbSlZicnUyywEFqbzG89g wJsA== X-Gm-Message-State: AOJu0Yz9uUGMu+Ggzu5O4S+QAGLGnh+OAjCnWtGjHFRYRZnI/LSrGPRY h3VhwxFA1N4VdK1ZpXg69KM4Zrl7ozO7dnV+Dh+oPzKXUZpLV4PGLNUhhRHPIH2UGGtpVs2WM+L Nek1oB92EB/2zErTvwu+lWoqlX3X6UaT8aJz58cCuQmE8esYhGbMxp7iD9o4yYObbOxpLX6k/Ep Bs0air0phff/vfcNziVxHgHfx9tAY8zm7ZRumXP/8N93ZLu41UX0rTUkw= X-Google-Smtp-Source: AGHT+IFhJzb4rLm89vMz9Kb+NMtcA3enxs1xJ/ZH3zd93j7HhNxOeMD+CCA4otMjkpxLYE9Tct7Cp4AlPuNvpA== X-Received: from wmbhj15.prod.google.com ([2002:a05:600c:528f:b0:459:db93:61f0]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e56:b0:458:c059:7da0 with SMTP id 5b1f17b1804b1-45b47caca71mr1829455e9.2.1755640369505; Tue, 19 Aug 2025 14:52:49 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:55 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-28-smostafa@google.com> Subject: [PATCH v4 27/28] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Based on the callbacks from the hypervisor, update the SMMUv3 Identity mapped page table. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 171 +++++++++++++++++- 1 file changed, 169 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index db9d9caaca2c..2d4ff21f83f9 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -11,6 +11,7 @@ #include =20 #include "arm_smmu_v3.h" +#include "../../../io-pgtable-arm.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -58,6 +59,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; smmu_wait(_cond); \ }) =20 +/* Protected by host_mmu.lock from core code. */ +static struct io_pgtable *idmap_pgtable; + /* Transfer ownership of memory */ static int smmu_take_pages(u64 phys, size_t size) { @@ -166,7 +170,6 @@ static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device = *smmu) return smmu_wait_event(smmu, smmu_cmdq_empty(&smmu->cmdq)); } =20 -__maybe_unused static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, struct arm_smmu_cmdq_ent *cmd) { @@ -178,6 +181,66 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device= *smmu, return smmu_sync_cmd(smmu); } =20 +static void __smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, void *unus= ed, + struct arm_smmu_cmdq_ent *cmd) +{ + WARN_ON(smmu_add_cmd(smmu, cmd)); +} + +static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, size_t granule) +{ + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + idmap_pgtable->cfg.pgsize_bitmap, smmu, + __smmu_add_cmd, NULL); + return smmu_sync_cmd(smmu); +} + +static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t gra= nule, + bool leaf) +{ + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_TLBI_S2_IPA, + .tlbi =3D { + .leaf =3D leaf, + .vmid =3D 0, + }, + }; + struct arm_smmu_cmdq_ent cmd_s1 =3D { + .opcode =3D CMDQ_OP_TLBI_NH_ALL, + .tlbi =3D { + .vmid =3D 0, + }, + }; + struct hyp_arm_smmu_v3_device *smmu; + + for_each_smmu(smmu) { + hyp_spin_lock(&smmu->lock); + WARN_ON(smmu_tlb_inv_range_smmu(smmu, &cmd, iova, size, granule)); + WARN_ON(smmu_send_cmd(smmu, &cmd_s1)); + hyp_spin_unlock(&smmu->lock); + } +} + +static void smmu_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + smmu_tlb_inv_range(iova, size, granule, false); +} + +static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ + smmu_tlb_inv_range(iova, granule, granule, true); +} + +static const struct iommu_flush_ops smmu_tlb_ops =3D { + .tlb_flush_walk =3D smmu_tlb_flush_walk, + .tlb_add_page =3D smmu_tlb_add_page, +}; + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -434,6 +497,37 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) return ret; } =20 +static int smmu_init_pgt(void) +{ + /* Default values overridden based on SMMUs common features. */ + struct io_pgtable_cfg cfg =3D (struct io_pgtable_cfg) { + .tlb =3D &smmu_tlb_ops, + .pgsize_bitmap =3D -1, + .ias =3D 48, + .oas =3D 48, + .coherent_walk =3D true, + }; + struct hyp_arm_smmu_v3_device *smmu; + struct io_pgtable_ops *ops; + + for_each_smmu(smmu) { + cfg.ias =3D min(cfg.ias, smmu->ias); + cfg.oas =3D min(cfg.oas, smmu->oas); + cfg.pgsize_bitmap &=3D smmu->pgsize_bitmap; + cfg.coherent_walk &=3D !!(smmu->features & ARM_SMMU_FEAT_COHERENCY); + } + + /* At least PAGE_SIZE must be supported by all SMMUs*/ + if ((cfg.pgsize_bitmap & PAGE_SIZE) =3D=3D 0) + return -EINVAL; + + ops =3D kvm_alloc_io_pgtable_ops(ARM_64_LPAE_S2, &cfg, NULL); + if (!ops) + return -ENOMEM; + idmap_pgtable =3D io_pgtable_ops_to_pgtable(ops); + return 0; +} + static int smmu_init(void) { int ret; @@ -455,7 +549,7 @@ static int smmu_init(void) =20 BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); =20 - return 0; + return smmu_init_pgt(); =20 out_reclaim_smmu: while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) @@ -789,8 +883,81 @@ static bool smmu_dabt_handler(struct user_pt_regs *reg= s, u64 esr, u64 addr) return false; } =20 +static size_t smmu_pgsize_idmap(size_t size, u64 paddr, size_t pgsize_bitm= ap) +{ + size_t pgsizes; + + /* Remove page sizes that are larger than the current size */ + pgsizes =3D pgsize_bitmap & GENMASK_ULL(__fls(size), 0); + + /* Remove page sizes that the address is not aligned to. */ + if (likely(paddr)) + pgsizes &=3D GENMASK_ULL(__ffs(paddr), 0); + + WARN_ON(!pgsizes); + + /* Return the larget page size that fits. */ + return BIT(__fls(pgsizes)); +} + static void smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int= prot) { + size_t size =3D end - start; + size_t pgsize =3D PAGE_SIZE, pgcount; + size_t mapped, unmapped; + int ret; + struct io_pgtable *pgtable =3D idmap_pgtable; + + end =3D min(end, BIT(pgtable->cfg.oas)); + if (start >=3D end) + return; + + if (prot) { + if (!(prot & IOMMU_MMIO)) + prot |=3D IOMMU_CACHE; + + while (size) { + mapped =3D 0; + /* + * We handle pages size for memory and MMIO differently: + * - memory: Map everything with PAGE_SIZE, that is guaranteed to + * find memory as we allocated enough pages to cover the entire + * memory, we do that as io-pgtable-arm doesn't support + * split_blk_unmap logic any more, so we can't break blocks once + * mapped to tables. + * - MMIO: Unlike memory, pKVM allocate 1G to for all MMIO, while + * the MMIO space can be large, as it is assumed to cover the + * whole IAS that is not memory, we have to use block mappings, + * that is fine for MMIO as it is never donated at the moment, + * so we never need to unmap MMIO at the run time triggereing + * split block logic. + */ + if (prot & IOMMU_MMIO) + pgsize =3D smmu_pgsize_idmap(size, start, pgtable->cfg.pgsize_bitmap); + + pgcount =3D size / pgsize; + ret =3D pgtable->ops.map_pages(&pgtable->ops, start, start, + pgsize, pgcount, prot, 0, &mapped); + size -=3D mapped; + start +=3D mapped; + if (!mapped || ret) + return; + } + } else { + /* Shouldn't happen. */ + WARN_ON(prot & IOMMU_MMIO); + while (size) { + pgcount =3D size / pgsize; + unmapped =3D pgtable->ops.unmap_pages(&pgtable->ops, start, + pgsize, pgcount, NULL); + size -=3D unmapped; + start +=3D unmapped; + if (!unmapped) + return; + } + /* Some memory were not unmapped. */ + WARN_ON(size); + } } =20 /* Shared with the kernel driver in EL1 */ --=20 2.51.0.rc1.167.g924127e9c0-goog From nobody Sat Oct 4 06:37:13 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8665C34DCFE for ; Tue, 19 Aug 2025 21:52:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640374; cv=none; b=EEDekfaelVgQDW6xcK9mHLeCkXyO48RRZtt84k6GqihnQh5UrFuSsrGA1THg8v0jwhSm4gew+NjW0iIXLiY7moUFDV+mkj0l5wDD7a94pmHt9PDjFgtrapmrZHXQJKL+1PzkutF7j5MhneOLLl1SLEHIh5mV7bJi2clsE2ceGzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755640374; c=relaxed/simple; bh=ouFg05ZUoCgKsBBLYpEzl0x2/Hx613KSJzCk/sH774M=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nNskOmsOme10ZlawofkOtDlmpo89uA5QAf0is3qtaZW+AfeRlpZJleBAPXjckF1VIsW/7bBMyq1W5caWu49+mXUrVzIsRzFt7SNSHgXMuZqFCrIsQHwrhfUMlEKIhf9Q5S9Al3LrC3N7w5UOrwlAGaFAzwdzu8Boir61QiKIiGk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sgOe/R86; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sgOe/R86" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-45a1b04f817so21173385e9.1 for ; Tue, 19 Aug 2025 14:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1755640371; x=1756245171; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NAhjHEW7JiGbHkZ3+MEbEcd94kVTPfsHqLQCKJs7pUU=; b=sgOe/R86QJmNDeSgtiuR/mFaHxYuEaGLBTYbdrP8fYn2y5aJeMWTWIOybFIoP53ErX e1qjwAXsmE8/JfsHhIiihcsC7z4IPBlRVMcTRl7DhAxu9iXVZJ49ND2Za72tM/nIpwQK mFXYe1zS6NYBlutTeo9iQAFA29+jOzyoO0udDQjey7yipa+pjPTko2WRwUiXa4dWM7l9 MxkjVEEzBBP4g9mPJhK9fwfoslJP5RUCi5k25yd3QI0/0sC3YBblgM1yU0UEOHdmqAS5 XmpAqzxtbvkare4psB0sT+DKqj+ZFpF7tF8BgvWG/hJqP7X4ZqjMNMeKOI6hPgDyeAE4 u24Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755640371; x=1756245171; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NAhjHEW7JiGbHkZ3+MEbEcd94kVTPfsHqLQCKJs7pUU=; b=aIVGXeql636wwpXojR8kKZaRh1tzxGlGspSCwh/DVbTgT2dEsfhj5sanl5XDypJ0sc qAcZneMXkuzvLrqV/LCozZqnKEoI007nVJhtzcwCEUsthNP+GpgyhiBRqEmtJOlaV45j tTLe7MB/f1YCSTOxyfwiRcZqzHVQHqTSc7wGPNpbEXcBt3GSRs+NaStFcXorapPAigC7 5qB1kgj9Ag6XgQ2o0cyzgn3e6w1AuHTqTQFlEQROtlLxdRDRCXgFbY8n/ZVFe7/YmtNZ N0fzmDTtLDGzVEgJtDdEOe/RHfISnNVwLvUsqkue/+9sapbqCglQXVxLtkJyuvl0xG/J rERQ== X-Gm-Message-State: AOJu0YzjEZL1LfZijNOvaZ1WTPqNYpb3An5fEbLdVJt3K9ZCH0/Zp568 BK2KBGPFvuksPzljQ8yr9HoA7tnxYkTqPTrimdXB5HM2qwRq+mSVvAMwclI7R9V9eDR40+L9KJJ jqrhV0ZISlI6m11qKDiYizGm4bYhMUIK63sbqXa8JwzbY3GnTWeXWZvqEi8xP9CDOhSqZeDc3qx dnJlX0NkdRET94pfnH8aaCUH/DixNXLZ2KjKGuYLn7YDSS2TaSs33ai18= X-Google-Smtp-Source: AGHT+IEZystQ29Iwydry030qs2Gr/OWu/XIzN2opOZePf7XxWhuKurUI07dQ/jVoWuLSREf7SL8Xi1ULimkXvw== X-Received: from wmbhj7.prod.google.com ([2002:a05:600c:5287:b0:456:111e:4717]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f4d:b0:459:d5d1:d602 with SMTP id 5b1f17b1804b1-45b479a3f4dmr3809755e9.3.1755640370667; Tue, 19 Aug 2025 14:52:50 -0700 (PDT) Date: Tue, 19 Aug 2025 21:51:56 +0000 In-Reply-To: <20250819215156.2494305-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250819215156.2494305-1-smostafa@google.com> X-Mailer: git-send-email 2.51.0.rc1.167.g924127e9c0-goog Message-ID: <20250819215156.2494305-29-smostafa@google.com> Subject: [PATCH v4 28/28] iommu/arm-smmu-v3-kvm: Enable nesting From: Mostafa Saleh To: linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Cc: maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org, qperret@google.com, tabba@google.com, jgg@ziepe.ca, mark.rutland@arm.com, praan@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now, as the hypervisor controls the command queue, stream table, and shadows the stage-2 page table. Enable stage-2 in case the host puts an STE in bypass or stage-1. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 72 ++++++++++++++++++- 1 file changed, 70 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 2d4ff21f83f9..5be44a37d581 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -336,6 +336,46 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static void smmu_attach_stage_2(struct hyp_arm_smmu_v3_device *smmu, struc= t arm_smmu_ste *ste) +{ + unsigned long vttbr; + unsigned long ts, sl, ic, oc, sh, tg, ps; + unsigned long cfg; + struct io_pgtable_cfg *pgt_cfg =3D &idmap_pgtable->cfg; + + cfg =3D FIELD_GET(STRTAB_STE_0_CFG, ste->data[0]); + if (!FIELD_GET(STRTAB_STE_0_V, ste->data[0]) || + (cfg =3D=3D STRTAB_STE_0_CFG_ABORT)) + return; + /* S2 is not advertised, that should never be attempted. */ + if (WARN_ON(cfg =3D=3D STRTAB_STE_0_CFG_NESTED)) + return; + vttbr =3D pgt_cfg->arm_lpae_s2_cfg.vttbr; + ps =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.ps; + tg =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tg; + sh =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sh; + oc =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.orgn; + ic =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.irgn; + sl =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sl; + ts =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tsz; + + ste->data[1] |=3D FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_1_SHCFG_INCOM= ING); + /* The host shouldn't write dwords 2 and 3, overwrite them. */ + ste->data[2] =3D FIELD_PREP(STRTAB_STE_2_VTCR, + FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, ps) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, tg) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SH0, sh) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2OR0, oc) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2IR0, ic) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, sl) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, ts)) | + FIELD_PREP(STRTAB_STE_2_S2VMID, 0) | + STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R; + ste->data[3] =3D vttbr & STRTAB_STE_3_S2TTB_MASK; + /* Convert S1 =3D> nested and bypass =3D> S2 */ + ste->data[0] |=3D FIELD_PREP(STRTAB_STE_0_CFG, cfg | BIT(1)); +} + /* Get an STE for a stream table base. */ static struct arm_smmu_ste *smmu_get_ste_ptr(struct hyp_arm_smmu_v3_device= *smmu, u32 sid, u64 *strtab) @@ -394,6 +434,10 @@ static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_d= evice *smmu, u32 sid, bool struct arm_smmu_ste *host_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, host_st= e_base); struct arm_smmu_ste *hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_= base); int i; + struct arm_smmu_ste target =3D {}; + struct arm_smmu_cmdq_ent cfgi_cmd =3D { + .opcode =3D CMDQ_OP_CFGI_ALL, + }; =20 /* * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this @@ -409,8 +453,32 @@ static void smmu_reshadow_ste(struct hyp_arm_smmu_v3_d= evice *smmu, u32 sid, bool hyp_ste_ptr =3D smmu_get_ste_ptr(smmu, sid, hyp_ste_base); } =20 - for (i =3D 0; i < STRTAB_STE_DWORDS; i++) - WRITE_ONCE(hyp_ste_ptr->data[i], host_ste_ptr->data[i]); + memcpy(target.data, host_ste_ptr->data, STRTAB_STE_DWORDS << 3); + + /* + * Typically, STE update is done as the following + * 1- Write last 7 dwords, while STE is invalid + * 2- CFGI + * 3- Write first dword, making STE valid + * 4- CFGI + * As the SMMU has to least to load 64 bits atomically + * that gurantees that there is no race between writing + * the STE and the CFGI where the SMMU observes parts + * of the STE. + * In the shadow we update the STE to enable nested translation, + * which requires updating first 3 dwords. + * That is only done if the STE is valid and not in abort. + * Which means it happens at step 4) + * So we need to also write the last 7 dwords and send CFGI + * before writing the first dword. + * There is no need for last CFGI as the host will do that. + */ + smmu_attach_stage_2(smmu, &target); + for (i =3D 1; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], target.data[i]); + + WARN_ON(smmu_send_cmd(smmu, &cfgi_cmd)); + WRITE_ONCE(hyp_ste_ptr->data[0], target.data[0]); } =20 static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) --=20 2.51.0.rc1.167.g924127e9c0-goog