From nobody Sun Jun 14 07:34:45 2026 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31E4839E18C for ; Fri, 1 May 2026 11:19:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634393; cv=none; b=YmIIR7i/fEQKZmly2/4M5hBM5G7ZbD2/ednZr4PAJryeQpzk3dOM+g/GVklL6UOwMX5FqMqISPxeW5RLDDGdnSYiGd4+dMLLBkx/LXmFaRwoOIm3nvmG6x4mqeTqO5FXnha9pvbkXwE6srCDFc6lJ6RUNeW0Q4HC/ch/QXV3nak= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634393; c=relaxed/simple; bh=hQvMJPmlY2Hj6FmtUqvk5LfujDesEr9peo98uTnWvUw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gZLL9VkK8bSA1FNFzd1MFL63LbRNACp23RF52MJIZJLRqe3qqEdw747DGOzSA+03SulKFWuu8ehcXh5/4+agYkUx2h1r2Lc/s7uh7m13Ob1KH3SDAbQfwtsI4M1dpx54K9sa0m3FQpnVOMJ0Ct0Ohm7bseeQMiOaQpECle+vOcA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=G79gI5qk; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="G79gI5qk" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-6729caff479so1912096a12.2 for ; Fri, 01 May 2026 04:19:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634390; x=1778239190; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ycktJLz1KBPXSpg6NVU24LQdR/O3V/jOKSZRApslagY=; b=G79gI5qkTjLIJ2F9NwfFpPJ08nWFW/r56B17m6es6A5uyELxEu8Ula9P+JrnjTB+kn BPRhFhzUZK0Ncy1SiS2YfQb+wd2ORhYWYJGLXqSW601Juw+IENNDCt9M/3mKoEtZfmLa hPdfM9Aftfa6YcO9ehhwN7DDHNWpAu2uD4dAhNd17nVZe6Fq6dKj3eedcdhmULvrX153 4Ul12pqimpXNnJARdiF14wTLmzlUkyCSHhvyC4eieQ5WOxnsrizA+Fab9ndGzqVQ9a3j z0Jyr/HhefFfZTdvNw94a2Z0BckJZYKUPOkLqNJ/yGd8umZruPLyY+YzJWQ7cMopH7Zp Zxhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634390; x=1778239190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ycktJLz1KBPXSpg6NVU24LQdR/O3V/jOKSZRApslagY=; b=WotM9q5GntF1Sudtv2I3iMXe94LFvgE3EBXSWYOqf7mTS82ODMX3HHDgbgMALQWazY ypVYPRch1PO7rxcgLO+BReLeMDegK+S6VmnIhQ2ZrqGrUYBSULftFszysFasgBP1NiG+ cy8HshqMiiDEt7CSKGSSqTGBeT+gv3woK2gygK+U5o7Ymoyr4ZmiGcF05wzErarTKEsY JMqvNdIzmXo7jR4B6sSRX5NoSj6IYqUtwLWekNWZjTcBS0LF4GweOvKOiR+VggkhzKOO q5N+pKUfukh2DpqjU6PljL+1LJWpkGu2mujlv4nzHZqg5gsfZC3K8tOe6zb6se/xh8z2 tznw== X-Forwarded-Encrypted: i=1; AFNElJ9+/F/JI1XIxvgo8CJOa5OIkY5a9Gfp2bXxBb0lSdLby2aci1BhRZpi1PsO2UX6fG3/jJigj4QVcU6ANtk=@vger.kernel.org X-Gm-Message-State: AOJu0Yww/P7k2cwBpOn+4TWZpGN0EGZMiZfoL4x/FRlbTH7zpHrUyLeK YX6uLpG+KJeOxGLYQSn5W/RnmP4KicYmV2O2uJpL6upuxepD5NP0VmdCMQAt0K1oFRDYud/idzC zEnIUDzLY5dpnqA== X-Received: from edag36.prod.google.com ([2002:a05:6402:3224:b0:66e:76b4:2912]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:46d8:b0:678:a507:e81b with SMTP id 4fb4d7f45d1cf-67b5d97defdmr2810112a12.21.1777634390386; Fri, 01 May 2026 04:19:50 -0700 (PDT) Date: Fri, 1 May 2026 11:19:03 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-2-smostafa@google.com> Subject: [PATCH v6 01/25] KVM: arm64: Generalize trace clock From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" IOMMU drivers need to track time, mainly for timeouts. Generalize the tracing clock functions in the hypervsior, so they can be used from IOMMU drivers. 1) Make the compilation independent from tracing. 2) As drivers might need to use that quite early, provide default values for the clock data based on cntfrq_el0, the driver can keep using these values without calling hyp_clock_update() as they don't need to sync with the host timers. This is mainly used for timeouts, so a malicious host can DoS the system or cause premature timeouts which likely end up in hyp panic, that should be acceptable as neither of those would undermine the security guarantees. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/clock.h | 11 ++----- arch/arm64/kvm/hyp/nvhe/Makefile | 4 +-- arch/arm64/kvm/hyp/nvhe/clock.c | 44 ++++++++++++++++++++++--- arch/arm64/kvm/hyp/nvhe/setup.c | 5 +++ arch/arm64/kvm/hyp/nvhe/trace.c | 4 +-- 5 files changed, 51 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/clock.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/clock.h index 9f429f5c0664..e6a0e43af88d 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/clock.h +++ b/arch/arm64/kvm/hyp/include/nvhe/clock.h @@ -5,12 +5,7 @@ =20 #include =20 -#ifdef CONFIG_NVHE_EL2_TRACING -void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc); -u64 trace_clock(void); -#else -static inline void -trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { } -static inline u64 trace_clock(void) { return 0; } -#endif +void hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc); +u64 hyp_clock_ns(void); +int hyp_clock_init(void); #endif diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 62cdfbff7562..89d0533921f9 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -26,10 +26,10 @@ hyp-obj-y :=3D timer-sr.o sysreg-sr.o debug-sr.o switch= .o tlb.o hyp-init.o host.o hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../en= try.o \ - ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o + ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o c= lock.o hyp-obj-y +=3D ../../../kernel/smccc-call.o hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o -hyp-obj-$(CONFIG_NVHE_EL2_TRACING) +=3D clock.o trace.o events.o +hyp-obj-$(CONFIG_NVHE_EL2_TRACING) +=3D trace.o events.o hyp-obj-y +=3D $(lib-objs) =20 # Path to simple_ring_buffer.c diff --git a/arch/arm64/kvm/hyp/nvhe/clock.c b/arch/arm64/kvm/hyp/nvhe/cloc= k.c index 32fc4313fe43..53d0bd55e866 100644 --- a/arch/arm64/kvm/hyp/nvhe/clock.c +++ b/arch/arm64/kvm/hyp/nvhe/clock.c @@ -18,7 +18,41 @@ static struct clock_data { u64 cyc_overflow64; } data[2]; u64 cur; -} trace_clock_data; +} clock_data; + +#define HYP_CLK_SEC_TO_NS 1000000000UL + +int hyp_clock_init(void) +{ + u32 timer_freq =3D read_sysreg(cntfrq_el0); + u32 shift =3D 32; + u64 mult; + + /* + * KVM will not initialize if FW didn't set cntfrq_el0, that is already + * part of the boot protocol. + */ + if (!timer_freq) + return -ENODEV; + + /* Timer freq can't be larger than 1Ghz by spec. */ + if (timer_freq > HYP_CLK_SEC_TO_NS) + return -EINVAL; + + /* Simplified logic from clocks_calc_mult_shift() */ + do { + mult =3D (HYP_CLK_SEC_TO_NS << shift); + mult =3D div_u64(mult, timer_freq); + if (mult <=3D (~0U)) + break; + shift--; + } while (shift > 0); + + clock_data.data[0].shift =3D shift; + clock_data.data[0].mult =3D mult; + clock_data.data[0].cyc_overflow64 =3D ULONG_MAX / mult; + return 0; +} =20 static u64 __clock_mult_uint128(u64 cyc, u32 mult, u32 shift) { @@ -30,9 +64,9 @@ static u64 __clock_mult_uint128(u64 cyc, u32 mult, u32 sh= ift) } =20 /* Does not guarantee no reader on the modified bank. */ -void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) +void hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { - struct clock_data *clock =3D &trace_clock_data; + struct clock_data *clock =3D &clock_data; u64 bank =3D clock->cur ^ 1; =20 clock->data[bank].mult =3D mult; @@ -45,9 +79,9 @@ void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns= , u64 epoch_cyc) } =20 /* Use untrusted host data */ -u64 trace_clock(void) +u64 hyp_clock_ns(void) { - struct clock_data *clock =3D &trace_clock_data; + struct clock_data *clock =3D &clock_data; u64 bank =3D smp_load_acquire(&clock->cur); u64 cyc, ns; =20 diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index d8e5b563fd3d..8041f6e80cd1 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -10,6 +10,7 @@ #include #include =20 +#include #include #include #include @@ -312,6 +313,10 @@ void __noreturn __pkvm_init_finalise(void) }; pkvm_pgtable.mm_ops =3D &pkvm_pgtable_mm_ops; =20 + ret =3D hyp_clock_init(); + if (ret) + goto out; + ret =3D fix_host_ownership(); if (ret) goto out; diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trac= e.c index a6ca27b18e15..e30de840e6c2 100644 --- a/arch/arm64/kvm/hyp/nvhe/trace.c +++ b/arch/arm64/kvm/hyp/nvhe/trace.c @@ -35,7 +35,7 @@ static bool hyp_trace_buffer_loaded(struct hyp_trace_buff= er *trace_buffer) void *tracing_reserve_entry(unsigned long length) { return simple_ring_buffer_reserve(this_cpu_ptr(trace_buffer.simple_rbs), = length, - trace_clock()); + hyp_clock_ns()); } =20 void tracing_commit_entry(void) @@ -285,7 +285,7 @@ void __tracing_update_clock(u32 mult, u32 shift, u64 ep= och_ns, u64 epoch_cyc) } =20 /* ...we can now override the old one and swap. */ - trace_clock_update(mult, shift, epoch_ns, epoch_cyc); + hyp_clock_update(mult, shift, epoch_ns, epoch_cyc); } =20 int __tracing_reset(unsigned int cpu) --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D8A839E6FE for ; Fri, 1 May 2026 11:19:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634397; cv=none; b=qAK0txpMsawMWiI/yjspKMmfGQHVz4EUjnfVgzi1FrzZ37parOjvrNaZJBekGbW1WNxBBGlhyxH++MQuKlnETusTWNnxbmQiukv9gOghobO3NeNFmsUyQYPLOgcJDOdjhum9y+5cv+hJhZKYF4HlTZShR5Rdaan4Q3gXPgaz9LI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634397; c=relaxed/simple; bh=BmiEfuWtlBnKnC8QbqZfTTQw11BzMKVlOkGG+YAJ6JE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=K4NDuHS+6qc92ZVy6lXLpk+zXjbzZV8PqSNQTNva3lGAkavFVoraWSwUn8T8a1ui3n++ZN+jOeCJoyraKcjdMOhDEpyBr6/AdzSiDUp/uiENEwtDE0V52JHkAGJszUgtYB9MFzoaLMw681lRECcqqk2+Z9d7CoTy/mM72Jg3whg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=H8DHbTsQ; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="H8DHbTsQ" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-44a56cf1466so476165f8f.1 for ; Fri, 01 May 2026 04:19:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634394; x=1778239194; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=MJCtNul8s1oM/6GNPCvZ51Sl0VZ6ziIdGPUB2bbn4jI=; b=H8DHbTsQ3JhdBSyFJE4VAoUCG2Iq/5GzQV1yzin/rIzO+I6aQa7zT/AN9gvLh/Mppn UW6zGs2a7i+yPA3U86+wSHXIzwI64GnzHxR+T8BMSiDQthle2XD7s80TPnOG9Q61aVg6 vywc+56UYQvM43lfO5gEbzpp6H8EVFz0bMxYLOuJWPit2ksDymQyBZAZzeU8hhOfOHDm apk3pKfmFFSVsjgY0Xc52tVMAm3ZyS9vn1emOGROW5WZDaQfJKn+hjeuYxfFpribFh8z 7jXgN3cgwgBrVq7LwrYCus2mxehFp42bfa+T2p5zelXlqcX9e2+Q/C7azMuehgR+T8eU t/iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634394; x=1778239194; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=MJCtNul8s1oM/6GNPCvZ51Sl0VZ6ziIdGPUB2bbn4jI=; b=niB4pSAcB973dkdygspAcW4sKmDhGPHh3oTbzLfBM90KrXt3KFvIFY5g7frhgcRHiJ dS35CFHAwrPCoEdeAV2OQzmLCuvgt5/mAEUQjqlp7w/E9lyVOxDJWht6XlB2VYND8KvC 8OcPPP4TkX3mah3e8yk/F4huH9eJbvdGOcQF178DOBrI1fVoaq50+9ntQm+aN+tSvsYS DQGokABlEI5u8JG01OwbmG8CgOKPVqWVHeZUu2cdzOYzCh7zgGUO/8XbdWU9glIczlJ3 PxbtWxPQk1/Ibvnr9wqOz5szr1e53btSu5oA8X3lLslfgcdTlHO6oxhBn1CRBTcdwWTD LYSw== X-Forwarded-Encrypted: i=1; AFNElJ8vsw4dU7jfWd7juyVQPejTz/HQT9zuNVowQLLwaZ6zUGaeswcstLq1GWN8uqfGTc9sJjIVPoGMmEUy1jE=@vger.kernel.org X-Gm-Message-State: AOJu0YyUFFS5xch5xoy+F6+DzV3k/gzIdmRWNblq1Ss4+PnVHIhjORrx JUwFw0ljOm4M2iVRR6LRw3IuEeHmxORlsjt1cP82ZqQCRK7L7zyXvpGVRne9xnIx2G/igWzDkiF BY9Gl9sf8ytENsA== X-Received: from wmqy6.prod.google.com ([2002:a05:600c:3646:b0:488:a6d9:e91a]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c170:b0:489:1a63:509c with SMTP id 5b1f17b1804b1-48a83d06bc3mr102554135e9.0.1777634393631; Fri, 01 May 2026 04:19:53 -0700 (PDT) Date: Fri, 1 May 2026 11:19:04 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-3-smostafa@google.com> Subject: [PATCH v6 02/25] KVM: arm64: Donate MMIO to the hypervisor From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a function to donate MMIO to the hypervisor so IOMMU hypervisor drivers can protect and access the MMIO of IOMMUs. As donating MMIO is very rare, and we don=E2=80=99t need to encode the full state, it=E2=80=99s reasonable to have a separate function to do this. It will init the host s2 page table with an invalid leaf with the owner ID to prevent the host from mapping the page on faults. Also, prevent kvm_pgtable_stage2_unmap() from removing owner ID from stage-2 PTEs, as this can be triggered from recycle logic under memory pressure. There is no code relying on this, as all ownership changes is done via kvm_pgtable_stage2_set_owner() For the error path in IOMMU drivers, add a function to donate MMIO back from hyp to host. However, that leaks the hypervisor virtual address range which should be acceptable as this is quite rare and it matches the behaviour of fix_map/block. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/mem_protect.c | 119 +++++++++++++++++- arch/arm64/kvm/hyp/pgtable.c | 9 +- 3 files changed, 121 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index 3cbfae0e3dda..ff440204d2c7 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -36,6 +36,8 @@ int __pkvm_guest_share_host(struct pkvm_hyp_vcpu *vcpu, u= 64 gfn); int __pkvm_guest_unshare_host(struct pkvm_hyp_vcpu *vcpu, u64 gfn); int __pkvm_host_unshare_hyp(u64 pfn); int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages); +int __pkvm_host_donate_hyp_mmio(phys_addr_t addr, size_t size, unsigned lo= ng *haddr); +int __pkvm_hyp_donate_host_mmio(phys_addr_t addr, size_t size); int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages); int __pkvm_host_share_ffa(u64 pfn, u64 nr_pages); int __pkvm_host_unshare_ffa(u64 pfn, u64 nr_pages); diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 28a471d1927c..2fb20a63a417 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -353,6 +353,38 @@ int __pkvm_prot_finalize(void) return 0; } =20 +/* Unmap MMIO region while skipping donated PTEs. */ +static int host_stage2_unmap_mmio_region(u64 start, u64 size) +{ + struct kvm_pgtable *pgt =3D &host_mmu.pgt; + u64 unmap_start =3D start; + u64 addr =3D start; + kvm_pte_t pte; + int ret =3D 0; + u8 level; + + while (addr < start + size) { + ret =3D kvm_pgtable_get_leaf(pgt, addr, &pte, &level); + if (ret) + return ret; + if (!kvm_pte_valid(pte) && pte !=3D 0) { + if (addr > unmap_start) { + ret =3D kvm_pgtable_stage2_unmap(pgt, unmap_start, + addr - unmap_start); + if (ret) + return ret; + } + addr +=3D kvm_granule_size(level); + unmap_start =3D addr; + } else { + addr +=3D kvm_granule_size(level); + } + } + if (addr > unmap_start) + ret =3D kvm_pgtable_stage2_unmap(pgt, unmap_start, addr - unmap_start); + return ret; +} + static int host_stage2_unmap_dev_all(void) { struct kvm_pgtable *pgt =3D &host_mmu.pgt; @@ -363,11 +395,11 @@ static int host_stage2_unmap_dev_all(void) /* Unmap all non-memory regions to recycle the pages */ for (i =3D 0; i < hyp_memblock_nr; i++, addr =3D reg->base + reg->size) { reg =3D &hyp_memory[i]; - ret =3D kvm_pgtable_stage2_unmap(pgt, addr, reg->base - addr); + ret =3D host_stage2_unmap_mmio_region(addr, reg->base - addr); if (ret) return ret; } - return kvm_pgtable_stage2_unmap(pgt, addr, BIT(pgt->ia_bits) - addr); + return host_stage2_unmap_mmio_region(addr, BIT(pgt->ia_bits) - addr); } =20 /* @@ -1087,6 +1119,89 @@ int __pkvm_host_donate_hyp(u64 pfn, u64 nr_pages) return ret; } =20 +int __pkvm_host_donate_hyp_mmio(phys_addr_t addr, size_t size, unsigned lo= ng *haddr) +{ + kvm_pte_t pte; + u64 offset; + int ret; + + /* Only before de-privilege. */ + if (static_branch_unlikely(&kvm_protected_mode_initialized)) + return -EPERM; + + if (!PAGE_ALIGNED(addr | size)) + return -EINVAL; + + ret =3D __pkvm_create_private_mapping(addr, size, PAGE_HYP_DEVICE, haddr); + if (ret) + return ret; + + host_lock_component(); + for (offset =3D 0; offset < size; offset +=3D PAGE_SIZE) { + if (addr_is_memory(addr + offset)) { + ret =3D -EINVAL; + goto unlock; + } + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, addr + offset, &pte, NULL); + if (ret) + goto unlock; + if (pte && !kvm_pte_valid(pte)) { + ret =3D -EPERM; + goto unlock; + } + } + /* + * We set HYP as the owner of the MMIO pages in the host stage-2, for: + * - host aborts: host_stage2_adjust_range() would fail for invalid non z= ero PTEs. + * - recycle under memory pressure: host_stage2_unmap_dev_all() would call + * kvm_pgtable_stage2_unmap() which will not clear non zero invalid pte= s (counted). + * - other MMIO donation: Would fail as we check that the PTE is valid or= empty. + */ + ret =3D host_stage2_try(kvm_pgtable_stage2_annotate, &host_mmu.pgt, + addr, size, &host_s2_pool, + KVM_HOST_INVALID_PTE_TYPE_DONATION, + FIELD_PREP(KVM_HOST_DONATION_PTE_OWNER_MASK, PKVM_ID_HYP)); +unlock: + host_unlock_component(); + return ret; +} + +int __pkvm_hyp_donate_host_mmio(phys_addr_t addr, size_t size) +{ + kvm_pte_t pte; + u64 offset; + int ret =3D 0; + + if (static_branch_unlikely(&kvm_protected_mode_initialized)) + return -EPERM; + + if (!PAGE_ALIGNED(addr | size)) + return -EINVAL; + + host_lock_component(); + for (offset =3D 0; offset < size; offset +=3D PAGE_SIZE) { + if (addr_is_memory(addr + offset)) { + ret =3D -EINVAL; + goto unlock; + } + ret =3D kvm_pgtable_get_leaf(&host_mmu.pgt, addr + offset, &pte, NULL); + if (ret) + goto unlock; + if (!pte || kvm_pte_valid(pte)) { + ret =3D -EINVAL; + goto unlock; + } + if (FIELD_GET(KVM_HOST_DONATION_PTE_OWNER_MASK, pte) !=3D PKVM_ID_HYP) { + ret =3D -EPERM; + goto unlock; + } + } + WARN_ON(host_stage2_idmap_locked(addr, size, PKVM_HOST_MMIO_PROT)); +unlock: + host_unlock_component(); + return ret; +} + int __pkvm_hyp_donate_host(u64 pfn, u64 nr_pages) { u64 phys =3D hyp_pfn_to_phys(pfn); diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 0c1defa5fb0f..b64a50f9bfa8 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1159,13 +1159,8 @@ static int stage2_unmap_walker(const struct kvm_pgta= ble_visit_ctx *ctx, kvm_pte_t *childp =3D NULL; bool need_flush =3D false; =20 - if (!kvm_pte_valid(ctx->old)) { - if (stage2_pte_is_counted(ctx->old)) { - kvm_clear_pte(ctx->ptep); - mm_ops->put_page(ctx->ptep); - } - return 0; - } + if (!kvm_pte_valid(ctx->old)) + return stage2_pte_is_counted(ctx->old) ? -EPERM : 0; =20 if (kvm_pte_table(ctx->old, ctx->level)) { childp =3D kvm_pte_follow(ctx->old, mm_ops); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48DAA39E6F5 for ; Fri, 1 May 2026 11:19:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634399; cv=none; b=TXaFTF9JyAY+4bWvgByr5YFNZgkPIE+Zk8C2eFy2Hc2Hzna4gQGFhAWWXlgTsNJ87DpT89sAjUow3AK3xaMXKbC54KhnsRiknpnqMxixuTP13Y3TkiK9L9lhKo4jW0BiQQeXTbfQbLrWqlq/Vy9HDgSWdB49bI7n/yqZlkn2OrY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634399; c=relaxed/simple; bh=Mc9VSadrlG3HR8bgyHW9QCg2LJjHfIHy8yI4QUWjNlM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XgZpnyvtV1x9y5QO2j1Gty/kfTLnlplOZIpVZ+tdUIPunrb73t0ItMPiSxl+q0bXOhkhpPONJDxQFF6XqHinbZD5MStdgfyip0ccipUT9LCOfaaMz8aqULaCwFmxxZao/9flS980YApALGf6c5zAi6mho0GudbLFne1JhyDksW8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Q0nC3qXM; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Q0nC3qXM" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-48a5adc12ffso11675165e9.0 for ; Fri, 01 May 2026 04:19:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634396; x=1778239196; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=O6kXYkgLH2ICtXbKKNbc9V/7v2AisF8lJzqevwz+DWw=; b=Q0nC3qXMeqVtigPzbFen5xZ8wJKhK4MJJdidA/74jf9QQRreLl1VMRlxsDa1y3+AA4 t3pmWORo50X40IyC7kAXC/be78+/S6I3RnQL4uymF9oTFVsXh7fXwvkmUa8MgT2iw/3S qrDURYNnZTieKcG2a5v0mmSvLFQFhtSpjbEQUkmmEbq2HEebJK/K+RaBQJFTD9RuW0KG KIGFRoCPmgYaaONgwH0TivSpk7vK2XJddMQQgIGXin2AgXe/dJVI6qSrx7n7rcPeJrh4 WC3vVD5Wct8nkR5HFe0CaZsE3NHI3NBWy8g1K4ESnO9SL7NH7NGQhOxn+bl3SshNNRv7 YY8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634396; x=1778239196; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O6kXYkgLH2ICtXbKKNbc9V/7v2AisF8lJzqevwz+DWw=; b=SxKTcVz7hA7G2rSQesh8qjCXGKgbJSVP2Es/kv8jaIlvmfQVb9/csVpgNeYGw+8DAc D1oDcbsvDb+9+diPwqzcP5YBtOKEYLV8+98eN20DqXBHyGL38SPSK7RAaO0YzkWsZ2HU JaRSM8pfz48eDXNk/TS/nw4ThRtUH8ciuOkzQy5EaOwIIqqpYSG1VMdENDamSB6/0MQv GUTynBQpusmPN7VSUdZqjXgYy3UAR8W++c8PgyDwjGo/8avH5/dc4COYYm433YdNu7QJ 5nBAU11I09vRLV9WNwsyNMJU36TEd0qxOZqXaa36Csk9+h+UJs6PLKaBcfenKCo/CNWd dpZA== X-Forwarded-Encrypted: i=1; AFNElJ+6hYwaN4rrSgHCdENBMOgs8bLicSepRghkb2JiASrH1htHmtY+93SPJ03tqONqZIbKlKK68rwHaYmcy0w=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1b1ye1AI9bUV7PZApnRW8W/Fgn5zq3n0gIAiIB+vG5OsrrvHr LJTiXm28gCGMBrXE3d6QQRSfHPhSyL6HkXL0Qd5SHMvAyoJi89f6K8SvTP//GkPPwBr+sZ3f1Lu NztbkjfjnZTYRhQ== X-Received: from wmxb6-n2.prod.google.com ([2002:a05:600d:8446:20b0:488:7b85:acf0]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f09:b0:48a:525b:e148 with SMTP id 5b1f17b1804b1-48a844e4a91mr99363655e9.4.1777634395587; Fri, 01 May 2026 04:19:55 -0700 (PDT) Date: Fri, 1 May 2026 11:19:05 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-4-smostafa@google.com> Subject: [PATCH v6 03/25] iommu/arm-smmu-v3: Split code with hyp From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The KVM SMMUv3 driver would re-use some of the cmdq and ste code inside the hypervisor, move these functions to a new common C file that is shared between the host kernel and the hypervisor. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 2 +- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 114 +++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 161 ------------------ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 61 +++++++ 4 files changed, 176 insertions(+), 162 deletions(-) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index 493a659cc66b..c9ce392e6d31 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0 obj-$(CONFIG_ARM_SMMU_V3) +=3D arm_smmu_v3.o -arm_smmu_v3-y :=3D arm-smmu-v3.o +arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c new file mode 100644 index 000000000000..62744c8548a8 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2015 ARM Limited + * + * Author: Will Deacon + * Arm SMMUv3 driver functions shared with hypervisor. + */ + +#include "arm-smmu-v3.h" +#include + +#include + +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) +{ + memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); + cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); + + switch (ent->opcode) { + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_NSNH_ALL: + break; + case CMDQ_OP_PREFETCH_CFG: + cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); + break; + case CMDQ_OP_CFGI_CD: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); + fallthrough; + case CMDQ_OP_CFGI_STE: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); + break; + case CMDQ_OP_CFGI_CD_ALL: + cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); + break; + case CMDQ_OP_CFGI_ALL: + /* Cover the entire SID range */ + cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); + break; + case CMDQ_OP_TLBI_NH_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + fallthrough; + case CMDQ_OP_TLBI_EL2_VA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; + break; + case CMDQ_OP_TLBI_S2_IPA: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); + cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); + cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; + break; + case CMDQ_OP_TLBI_NH_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + fallthrough; + case CMDQ_OP_TLBI_NH_ALL: + case CMDQ_OP_TLBI_S12_VMALL: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); + break; + case CMDQ_OP_TLBI_EL2_ASID: + cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); + break; + case CMDQ_OP_ATC_INV: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); + cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; + break; + case CMDQ_OP_PRI_RESP: + cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); + cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); + switch (ent->pri.resp) { + case PRI_RESP_DENY: + case PRI_RESP_FAIL: + case PRI_RESP_SUCC: + break; + default: + return -EINVAL; + } + cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); + break; + case CMDQ_OP_RESUME: + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); + cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); + cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); + break; + case CMDQ_OP_CMD_SYNC: + if (ent->sync.msiaddr) { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); + cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; + } else { + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + } + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); + cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); + break; + default: + return -ENOENT; + } + + return 0; +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index e8d7dbe495f0..cb64f88989f0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -125,33 +125,6 @@ static void parse_driver_options(struct arm_smmu_devic= e *smmu) } =20 /* Low-level queue manipulation functions */ -static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) -{ - u32 space, prod, cons; - - prod =3D Q_IDX(q, q->prod); - cons =3D Q_IDX(q, q->cons); - - if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; - - return space >=3D n; -} - -static bool queue_full(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); -} - -static bool queue_empty(struct arm_smmu_ll_queue *q) -{ - return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && - Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); -} - static bool queue_consumed(struct arm_smmu_ll_queue *q, u32 prod) { return ((Q_WRP(q, q->cons) =3D=3D Q_WRP(q, prod)) && @@ -170,12 +143,6 @@ static void queue_sync_cons_out(struct arm_smmu_queue = *q) writel_relaxed(q->llq.cons, q->cons_reg); } =20 -static void queue_inc_cons(struct arm_smmu_ll_queue *q) -{ - u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; - q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); -} - static void queue_sync_cons_ovf(struct arm_smmu_queue *q) { struct arm_smmu_ll_queue *llq =3D &q->llq; @@ -207,12 +174,6 @@ static int queue_sync_prod_in(struct arm_smmu_queue *q) return ret; } =20 -static u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) -{ - u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; - return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); -} - static void queue_poll_init(struct arm_smmu_device *smmu, struct arm_smmu_queue_poll *qp) { @@ -240,14 +201,6 @@ static int queue_poll(struct arm_smmu_queue_poll *qp) return 0; } =20 -static void queue_write(__le64 *dst, u64 *src, size_t n_dwords) -{ - int i; - - for (i =3D 0; i < n_dwords; ++i) - *dst++ =3D cpu_to_le64(*src++); -} - static void queue_read(u64 *dst, __le64 *src, size_t n_dwords) { int i; @@ -268,108 +221,6 @@ static int queue_remove_raw(struct arm_smmu_queue *q,= u64 *ent) } =20 /* High-level queue accessors */ -static int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent) -{ - memset(cmd, 0, 1 << CMDQ_ENT_SZ_SHIFT); - cmd[0] |=3D FIELD_PREP(CMDQ_0_OP, ent->opcode); - - switch (ent->opcode) { - case CMDQ_OP_TLBI_EL2_ALL: - case CMDQ_OP_TLBI_NSNH_ALL: - break; - case CMDQ_OP_PREFETCH_CFG: - cmd[0] |=3D FIELD_PREP(CMDQ_PREFETCH_0_SID, ent->prefetch.sid); - break; - case CMDQ_OP_CFGI_CD: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SSID, ent->cfgi.ssid); - fallthrough; - case CMDQ_OP_CFGI_STE: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_LEAF, ent->cfgi.leaf); - break; - case CMDQ_OP_CFGI_CD_ALL: - cmd[0] |=3D FIELD_PREP(CMDQ_CFGI_0_SID, ent->cfgi.sid); - break; - case CMDQ_OP_CFGI_ALL: - /* Cover the entire SID range */ - cmd[1] |=3D FIELD_PREP(CMDQ_CFGI_1_RANGE, 31); - break; - case CMDQ_OP_TLBI_NH_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - fallthrough; - case CMDQ_OP_TLBI_EL2_VA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_VA_MASK; - break; - case CMDQ_OP_TLBI_S2_IPA: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_NUM, ent->tlbi.num); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_SCALE, ent->tlbi.scale); - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_LEAF, ent->tlbi.leaf); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TTL, ent->tlbi.ttl); - cmd[1] |=3D FIELD_PREP(CMDQ_TLBI_1_TG, ent->tlbi.tg); - cmd[1] |=3D ent->tlbi.addr & CMDQ_TLBI_1_IPA_MASK; - break; - case CMDQ_OP_TLBI_NH_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - fallthrough; - case CMDQ_OP_TLBI_NH_ALL: - case CMDQ_OP_TLBI_S12_VMALL: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_VMID, ent->tlbi.vmid); - break; - case CMDQ_OP_TLBI_EL2_ASID: - cmd[0] |=3D FIELD_PREP(CMDQ_TLBI_0_ASID, ent->tlbi.asid); - break; - case CMDQ_OP_ATC_INV: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_GLOBAL, ent->atc.global); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SSID, ent->atc.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_ATC_0_SID, ent->atc.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_ATC_1_SIZE, ent->atc.size); - cmd[1] |=3D ent->atc.addr & CMDQ_ATC_1_ADDR_MASK; - break; - case CMDQ_OP_PRI_RESP: - cmd[0] |=3D FIELD_PREP(CMDQ_0_SSV, ent->substream_valid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SSID, ent->pri.ssid); - cmd[0] |=3D FIELD_PREP(CMDQ_PRI_0_SID, ent->pri.sid); - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_GRPID, ent->pri.grpid); - switch (ent->pri.resp) { - case PRI_RESP_DENY: - case PRI_RESP_FAIL: - case PRI_RESP_SUCC: - break; - default: - return -EINVAL; - } - cmd[1] |=3D FIELD_PREP(CMDQ_PRI_1_RESP, ent->pri.resp); - break; - case CMDQ_OP_RESUME: - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_SID, ent->resume.sid); - cmd[0] |=3D FIELD_PREP(CMDQ_RESUME_0_RESP, ent->resume.resp); - cmd[1] |=3D FIELD_PREP(CMDQ_RESUME_1_STAG, ent->resume.stag); - break; - case CMDQ_OP_CMD_SYNC: - if (ent->sync.msiaddr) { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_IRQ); - cmd[1] |=3D ent->sync.msiaddr & CMDQ_SYNC_1_MSIADDR_MASK; - } else { - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); - } - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSH, ARM_SMMU_SH_ISH); - cmd[0] |=3D FIELD_PREP(CMDQ_SYNC_0_MSIATTR, ARM_SMMU_MEMATTR_OIWB); - break; - default: - return -ENOENT; - } - - return 0; -} - static struct arm_smmu_cmdq *arm_smmu_get_cmdq(struct arm_smmu_device *smm= u, struct arm_smmu_cmdq_ent *ent) { @@ -1827,18 +1678,6 @@ static void arm_smmu_free_cd_tables(struct arm_smmu_= master *master) } =20 /* Stream table manipulation functions */ -static void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1 *dst, - dma_addr_t l2ptr_dma) -{ - u64 val =3D 0; - - val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); - val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; - - /* The HW has 64 bit atomicity with stores to the L2 STE table */ - WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); -} - struct arm_smmu_ste_writer { struct arm_smmu_entry_writer writer; u32 sid; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index ef42df4753ec..9b8c5fb7282b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1142,6 +1142,67 @@ void arm_smmu_install_ste_for_dev(struct arm_smmu_ma= ster *master, int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq, u64 *cmds, int n, bool sync); +int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent); + +/* Queue functions shared between kernel and hyp. */ +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + u32 space, prod, cons; + + prod =3D Q_IDX(q, q->prod); + cons =3D Q_IDX(q, q->cons); + + if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) + space =3D (1 << q->max_n_shift) - (prod - cons); + else + space =3D cons - prod; + + return space >=3D n; +} + +static inline bool queue_full(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) !=3D Q_WRP(q, q->cons); +} + +static inline bool queue_empty(struct arm_smmu_ll_queue *q) +{ + return Q_IDX(q, q->prod) =3D=3D Q_IDX(q, q->cons) && + Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons); +} + +static inline u32 queue_inc_prod_n(struct arm_smmu_ll_queue *q, int n) +{ + u32 prod =3D (Q_WRP(q, q->prod) | Q_IDX(q, q->prod)) + n; + return Q_OVF(q->prod) | Q_WRP(q, prod) | Q_IDX(q, prod); +} + +static inline void queue_inc_cons(struct arm_smmu_ll_queue *q) +{ + u32 cons =3D (Q_WRP(q, q->cons) | Q_IDX(q, q->cons)) + 1; + q->cons =3D Q_OVF(q->cons) | Q_WRP(q, cons) | Q_IDX(q, cons); +} + +static inline void queue_write(__le64 *dst, u64 *src, size_t n_dwords) +{ + int i; + + for (i =3D 0; i < n_dwords; ++i) + *dst++ =3D cpu_to_le64(*src++); +} + +static inline void arm_smmu_write_strtab_l1_desc(struct arm_smmu_strtab_l1= *dst, + dma_addr_t l2ptr_dma) +{ + u64 val =3D 0; + + val |=3D FIELD_PREP(STRTAB_L1_DESC_SPAN, STRTAB_SPLIT + 1); + val |=3D l2ptr_dma & STRTAB_L1_DESC_L2PTR_MASK; + + /* The HW has 64 bit atomicity with stores to the L2 STE table */ + WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); +} =20 #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9334739E184 for ; Fri, 1 May 2026 11:20:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634405; cv=none; b=bQYq2eqJ+DKwo4H764ywOsLLt/8VFxDEVLp9/wsotGdM+5iURa7QX5kQ4cGdisqDD37dlo8FeCruap6tQeWtuvKV6FZpucSu83EDBiLREVZL/MSm7hOK+PSeWNXGPTzPIblU2tC5JtcohlUgYHSA9prxaoxY7B0K67TLChsEazM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634405; c=relaxed/simple; bh=15+QeiFyLvHiPcCIubc5fSnh88MpTHwIauK/sgpB9Kk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=B2+B6mN+n/sNEghQI1d1gooMx4hjJxz5BmvVr7gUMhBZONsraNVeniR1gfwCzizb7jgcRXP37YuteePRlJOP5oUOhT2fm1UI7K+zyIA2SHG24P2Kgygr/5dWh8+1bPI+fAvRvGnvJBbByCBs5X6Uauj5OdSr1vrdB/+LQiHorag= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ebaDV6dQ; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ebaDV6dQ" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-48906aa28cbso22973565e9.0 for ; Fri, 01 May 2026 04:20:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634402; x=1778239202; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cn3lch7CcipEUf/hsRSFMjJhdn+CahR9BLaY3vu1ZIo=; b=ebaDV6dQP8zd2CeQCODSNtWqHoXxzGUHEeq1v9zQTywCJXe5dFIorIxgzfJIn/YMpo umHOjnXINKwTR8RISRbjwtU3MLPJ/ADyWluP/BT1Ve1NYWKSljTeomR1SjiC23Ktz9Cc yqhgaa6FRF6z2onbIfUNw+Uj90MORaL1GL23+wcb6PByLBTU2LDRoR6BWLvO/kNIOyxR xMG9fULZc4mD7mom8M7ZUkbsev6CLpZtBP9cre4sIwHqgCbCxVVwrImebkEHxOYIM9tF yz1HA+I7JvGJkRnodsA0Qhp042KITOoqj0hsVk1skdfQT8Ss7VIe/lsX08pxEiTCTTja 9GuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634402; x=1778239202; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cn3lch7CcipEUf/hsRSFMjJhdn+CahR9BLaY3vu1ZIo=; b=DhijAGB/5b3CMm87YpNQtsoxURXsvQ8EN8hfyO58gxsg5r56tevWYwuPCdaMLJ3LHG t31Vqgi/XAHNDSgcm7DAHRe5HrLKXbrFY4kyS6GtovzUz9cvn5j2Yj6+IcpTpCBP/Zvo IBl78iZi3RE4oF7h0Wiv2lY3/EcsjOL8slBDS5pLBqMVyMUtb/f/OtVcFjKvsdY91hoQ kOn+qTYwPL77uNCaZRJUvFCLUgMYWQbGDctqoREZvVTE8InmODaqWhesQwJBspWnF602 k/AIRQq2pbsgJsZ1iLLudBTBxrQHIa8zVGDvkFYaqoMXCgyzhWYxyGOkHVJThYEW9dUJ 3hNA== X-Forwarded-Encrypted: i=1; AFNElJ/J4tA/FAN9G/fmxRdrvuvhioQCALIg5EC2orXhAdotT7rpvbvGGiQg8uo9PSkv/i2QqpzuTZRVZdYsVpQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwWCikaCfjqj3FiKlQV6EIQhJBJ+3Uk4/7AWF9hv92eevG8hkgU 2fUI6ivaFoVirdvgImwO7XTg4J6Vz0JiH9aNCXosRablu0yPmRe9c9jen3kER+FC/PxFrx4ItIG cpiXbUSWf1fvlow== X-Received: from wrna11.prod.google.com ([2002:adf:e5cb:0:b0:449:2f8e:2b32]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:310e:b0:48a:5821:6006 with SMTP id 5b1f17b1804b1-48a93e9bb92mr17243045e9.4.1777634401907; Fri, 01 May 2026 04:20:01 -0700 (PDT) Date: Fri, 1 May 2026 11:19:06 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-5-smostafa@google.com> Subject: [PATCH v6 04/25] iommu/arm-smmu-v3: Move TLB range invalidation into common code From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Range TLB invalidation has a very specific algorithm. Instead of re-writing it for the hypervisor, move it to a function that can be re-used. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 65 ++++-------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 76 +++++++++++++++++++++ 2 files changed, 88 insertions(+), 53 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index cb64f88989f0..c22832d26495 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2362,68 +2362,27 @@ static void arm_smmu_tlb_inv_context(void *cookie) arm_smmu_domain_inv(smmu_domain); } =20 +static void __arm_smmu_cmdq_batch_add(void *__opaque, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd) +{ + struct arm_smmu_device *smmu =3D (struct arm_smmu_device *)__opaque; + + arm_smmu_cmdq_batch_add(smmu, cmds, cmd); +} + static void arm_smmu_cmdq_batch_add_range(struct arm_smmu_device *smmu, struct arm_smmu_cmdq_batch *cmds, struct arm_smmu_cmdq_ent *cmd, unsigned long iova, size_t size, size_t granule, size_t pgsize) { - unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; - size_t inv_range =3D granule; - if (WARN_ON_ONCE(!size)) return; =20 - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - num_pages =3D size >> tg; - - /* Convert page size of 12,14,16 (log2) to 1,2,3 */ - cmd->tlbi.tg =3D (tg - 10) / 2; - - /* - * Determine what level the granule is at. For non-leaf, both - * io-pgtable and SVA pass a nominal last-level granule because - * they don't know what level(s) actually apply, so ignore that - * and leave TTL=3D0. However for various errata reasons we still - * want to use a range command, so avoid the SVA corner case - * where both scale and num could be 0 as well. - */ - if (cmd->tlbi.leaf) - cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); - else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) - num_pages++; - } - - while (iova < end) { - if (smmu->features & ARM_SMMU_FEAT_RANGE_INV) { - /* - * On each iteration of the loop, the range is 5 bits - * worth of the aligned size remaining. - * The range in pages is: - * - * range =3D (num_pages & (0x1f << __ffs(num_pages))) - */ - unsigned long scale, num; - - /* Determine the power of 2 multiple number of pages */ - scale =3D __ffs(num_pages); - cmd->tlbi.scale =3D scale; - - /* Determine how many chunks of 2^scale size we have */ - num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; - cmd->tlbi.num =3D num - 1; - - /* range is num * 2^scale * pgsize */ - inv_range =3D num << (scale + tg); - - /* Clear out the lower order bits for the next iteration */ - num_pages -=3D num << scale; - } - - cmd->tlbi.addr =3D iova; - arm_smmu_cmdq_batch_add(smmu, cmds, cmd); - iova +=3D inv_range; - } + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + pgsize, smmu->features & ARM_SMMU_FEAT_RANGE_INV, + smmu, __arm_smmu_cmdq_batch_add, cmds); } =20 static bool arm_smmu_inv_size_too_big(struct arm_smmu_device *smmu, size_t= size, diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 9b8c5fb7282b..7be41dbe5aaa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1204,6 +1204,82 @@ static inline void arm_smmu_write_strtab_l1_desc(str= uct arm_smmu_strtab_l1 *dst, WRITE_ONCE(dst->l2ptr, cpu_to_le64(val)); } =20 +/** + * arm_smmu_tlb_inv_build - Create a range invalidation command + * @cmd: Base command initialized with OPCODE (S1, S2..), vmid and asid + * @iova: Start IOVA to invalidate + * @size: Size of range + * @granule: Granule of invalidation + * @pgsize: Page size of the invalidation + * @is_range: Use range invalidation commands + * @opaque: Pointer to pass to add_cmd + * @add_cmd: Function to send/batch the invalidation command + * @cmds: Incase of batching, it includes the pointer to the batch + */ +static inline void arm_smmu_tlb_inv_build(struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, + size_t granule, unsigned long pgsize, + bool is_range, void *opaque, + void (*add_cmd)(void *_opaque, + struct arm_smmu_cmdq_batch *cmds, + struct arm_smmu_cmdq_ent *cmd), + struct arm_smmu_cmdq_batch *cmds) +{ + unsigned long end =3D iova + size, num_pages =3D 0, tg =3D pgsize; + size_t inv_range =3D granule; + + if (is_range) { + num_pages =3D size >> tg; + + /* Convert page size of 12,14,16 (log2) to 1,2,3 */ + cmd->tlbi.tg =3D (tg - 10) / 2; + + /* + * Determine what level the granule is at. For non-leaf, both + * io-pgtable and SVA pass a nominal last-level granule because + * they don't know what level(s) actually apply, so ignore that + * and leave TTL=3D0. However for various errata reasons we still + * want to use a range command, so avoid the SVA corner case + * where both scale and num could be 0 as well. + */ + if (cmd->tlbi.leaf) + cmd->tlbi.ttl =3D 4 - ((ilog2(granule) - 3) / (tg - 3)); + else if ((num_pages & CMDQ_TLBI_RANGE_NUM_MAX) =3D=3D 1) + num_pages++; + } + + while (iova < end) { + if (is_range) { + /* + * On each iteration of the loop, the range is 5 bits + * worth of the aligned size remaining. + * The range in pages is: + * + * range =3D (num_pages & (0x1f << __ffs(num_pages))) + */ + unsigned long scale, num; + + /* Determine the power of 2 multiple number of pages */ + scale =3D __ffs(num_pages); + cmd->tlbi.scale =3D scale; + + /* Determine how many chunks of 2^scale size we have */ + num =3D (num_pages >> scale) & CMDQ_TLBI_RANGE_NUM_MAX; + cmd->tlbi.num =3D num - 1; + + /* range is num * 2^scale * pgsize */ + inv_range =3D num << (scale + tg); + + /* Clear out the lower order bits for the next iteration */ + num_pages -=3D num << scale; + } + + cmd->tlbi.addr =3D iova; + add_cmd(opaque, cmds, cmd); + iova +=3D inv_range; + } +} + #ifdef CONFIG_ARM_SMMU_V3_SVA bool arm_smmu_sva_supported(struct arm_smmu_device *smmu); void arm_smmu_sva_notifier_synchronize(void); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59F5739EF09 for ; Fri, 1 May 2026 11:20:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634407; cv=none; b=RsBfkdwxh60PpLDeMBQb63Aej+TS592tQbdPF7OLIG9oCHOibqKGgBhbFd/LtabqhVIRfNjiANXSOeCab3lPALBJQrPzZogN/yTe8bcZEK0W8cLuFS9ykCYl6wqAt34a/NAeMxpIhGSzwJOpct4mZsQp1jshpfg6G7Y5ibBXOrk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634407; c=relaxed/simple; bh=MJpO/a/WnHz3UIfcZbr2Jxww3GoG5+ijB8pjliAreDw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=V4mQGsTEyavq3k6ev9cq9F8gkU4bhcj0IUiZ7tmI/BWK5lydneo38I+1g0/nYL0EQewdck+Ani+bge32mdIym4MT7P7T1e+RhmZkCGdD2Mvip1NjWjySNgmi4cRycFt73OS6a9SsfX6XZUrPS63dbFxUehcGZSIYbcVBb1Hg0O4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RNQLXQjB; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RNQLXQjB" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-43d103e46c3so1109600f8f.3 for ; Fri, 01 May 2026 04:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634404; x=1778239204; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SvUMSgIU3B3ilTsGpCNLEttDMrzsuBUntqQsjjdex/k=; b=RNQLXQjBm4ZvMDeNHM+J7jD9X70TnsFBxxOlNhWSl6ZOZSPAj9o92uJnGFDxWKlPO7 v0HhepdhcNdHzguJ2KPwKS/fzMavC/VYEW0TZvpXgT9u/k33zaQNOEmDNJdeFK3GL3cP Mr42pRUZaCZJbW2sptuB8P1136OI8ZaK51lrKlk8asoRxTgoWui9SrFcdLq66BhNteRK AaXKCTyEmYo9ej1WaEvFYlQF0wkQnmKRlqA/6MOpWXug5qvOz5/956Alrrrav0DfUX/u rAbBbKX+9uUx+XyX528nDW1Q90JXjEmBv5pEOIAACwfpEA8ahfycRGCUSJ2Ps0F+Ywc6 OeLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634404; x=1778239204; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SvUMSgIU3B3ilTsGpCNLEttDMrzsuBUntqQsjjdex/k=; b=N050mMKT1ONJ5bdF9tXGWtlHNWu8XTlIFbveUsBubXjU2Nd+hqtNVyn24qZi+MV8Er iZYefI9B+wXA+d6n0mRyKBbF5Flaot9nUQC732hx2RrMyhcGOqwjRRxQs2ZQRAE3KlCO NRmvMkz1s3XcshOKnM5bkQl2xpMTSq4/VCbQzBzQaH7EfCy51lHrKZnMuOKnIvSptjSJ 37AcB5zg/DlbmPsrH7QBR/wYRc+NZTu8q4Un4N+94WhI7aucQ4CG9Lubg6VDvorpGkyi SrLb7EOsqsomw9vNTyq6X6yof/CjOfbtXAjGig6x9B9HJrn2nkv/XrBuAwlB8cfDiGqt eaqA== X-Forwarded-Encrypted: i=1; AFNElJ+l0QOgHczFpusJENxkGWeKCSaO0S51ULWPVRK2UCKsXw2CAXsLNtJTvl7FB+jiUKczERCDuUkE4xTCjoY=@vger.kernel.org X-Gm-Message-State: AOJu0YzVMyWHgmphMn82DlvjN9jLhh3+43Y6XJ4i6uoAW9kobThXsln2 J+52zugGtM6BVwiZCCVPVDLE7ImW+nFMQtvIrpNYfbjZ3FGyjK2fV/4IK2l7kkW+OrOAX3Bg0sP RjgbcUYr7HM2dgA== X-Received: from wmoo5.prod.google.com ([2002:a05:600d:105:b0:488:9c2a:aeba]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e16:b0:48a:52ce:a4b1 with SMTP id 5b1f17b1804b1-48a8451cff6mr117057225e9.15.1777634403717; Fri, 01 May 2026 04:20:03 -0700 (PDT) Date: Fri, 1 May 2026 11:19:07 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-6-smostafa@google.com> Subject: [PATCH v6 05/25] iommu/arm-smmu-v3: Move IDR parsing to common functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move parsing of IDRs to functions so that it can be re-used from the hypervisor. Signed-off-by: Mostafa Saleh --- .../arm/arm-smmu-v3/arm-smmu-v3-common-lib.c | 110 +++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 112 +++--------------- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 5 + 3 files changed, 130 insertions(+), 97 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c b/drive= rs/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c index 62744c8548a8..e6dd087e2999 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-common-lib.c @@ -112,3 +112,113 @@ int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu= _cmdq_ent *ent) =20 return 0; } + +u32 smmu_idr0_features(u32 reg) +{ + u32 features =3D 0; + + /* 2-level structures */ + if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) + features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; + + if (reg & IDR0_CD2L) + features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; + + /* + * Translation table endianness. + * We currently require the same endianness as the CPU, but this + * could be changed later by adding a new IO_PGTABLE_QUIRK. + */ + switch (FIELD_GET(IDR0_TTENDIAN, reg)) { + case IDR0_TTENDIAN_MIXED: + features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; + break; +#ifdef __BIG_ENDIAN + case IDR0_TTENDIAN_BE: + features |=3D ARM_SMMU_FEAT_TT_BE; + break; +#else + case IDR0_TTENDIAN_LE: + features |=3D ARM_SMMU_FEAT_TT_LE; + break; +#endif + } + + /* Boolean feature flags */ + if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) + features |=3D ARM_SMMU_FEAT_PRI; + + if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) + features |=3D ARM_SMMU_FEAT_ATS; + + if (reg & IDR0_SEV) + features |=3D ARM_SMMU_FEAT_SEV; + + if (reg & IDR0_MSI) + features |=3D ARM_SMMU_FEAT_MSI; + + if (reg & IDR0_HYP) + features |=3D ARM_SMMU_FEAT_HYP; + + switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { + case IDR0_STALL_MODEL_FORCE: + features |=3D ARM_SMMU_FEAT_STALL_FORCE; + fallthrough; + case IDR0_STALL_MODEL_STALL: + features |=3D ARM_SMMU_FEAT_STALLS; + } + + if (reg & IDR0_S1P) + features |=3D ARM_SMMU_FEAT_TRANS_S1; + + if (reg & IDR0_S2P) + features |=3D ARM_SMMU_FEAT_TRANS_S2; + + return features; +} + +u32 smmu_idr3_features(u32 reg) +{ + u32 features =3D 0; + + if (FIELD_GET(IDR3_RIL, reg)) + features |=3D ARM_SMMU_FEAT_RANGE_INV; + if (FIELD_GET(IDR3_FWB, reg)) + features |=3D ARM_SMMU_FEAT_S2FWB; + + return features; +} + +u32 smmu_idr5_to_oas(u32 reg) +{ + switch (FIELD_GET(IDR5_OAS, reg)) { + case IDR5_OAS_32_BIT: + return 32; + case IDR5_OAS_36_BIT: + return 36; + case IDR5_OAS_40_BIT: + return 40; + case IDR5_OAS_42_BIT: + return 42; + case IDR5_OAS_44_BIT: + return 44; + case IDR5_OAS_48_BIT: + return 48; + case IDR5_OAS_52_BIT: + return 52; + } + return 0; +} + +unsigned long smmu_idr5_to_pgsize(u32 reg) +{ + unsigned long pgsize_bitmap =3D 0; + + if (reg & IDR5_GRAN64K) + pgsize_bitmap |=3D SZ_64K | SZ_512M; + if (reg & IDR5_GRAN16K) + pgsize_bitmap |=3D SZ_16K | SZ_32M; + if (reg & IDR5_GRAN4K) + pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + return pgsize_bitmap; +} diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index c22832d26495..96d5e7f80ce7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4815,57 +4815,17 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) /* IDR0 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); =20 - /* 2-level structures */ - if (FIELD_GET(IDR0_ST_LVL, reg) =3D=3D IDR0_ST_LVL_2LVL) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_STRTAB; - - if (reg & IDR0_CD2L) - smmu->features |=3D ARM_SMMU_FEAT_2_LVL_CDTAB; - - /* - * Translation table endianness. - * We currently require the same endianness as the CPU, but this - * could be changed later by adding a new IO_PGTABLE_QUIRK. - */ - switch (FIELD_GET(IDR0_TTENDIAN, reg)) { - case IDR0_TTENDIAN_MIXED: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE; - break; -#ifdef __BIG_ENDIAN - case IDR0_TTENDIAN_BE: - smmu->features |=3D ARM_SMMU_FEAT_TT_BE; - break; -#else - case IDR0_TTENDIAN_LE: - smmu->features |=3D ARM_SMMU_FEAT_TT_LE; - break; -#endif - default: + smmu->features |=3D smmu_idr0_features(reg); + if (!(smmu->features & (ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE))) { dev_err(smmu->dev, "unknown/unsupported TT endianness!\n"); return -ENXIO; } - - /* Boolean feature flags */ - if (IS_ENABLED(CONFIG_PCI_PRI) && reg & IDR0_PRI) - smmu->features |=3D ARM_SMMU_FEAT_PRI; - - if (IS_ENABLED(CONFIG_PCI_ATS) && reg & IDR0_ATS) - smmu->features |=3D ARM_SMMU_FEAT_ATS; - - if (reg & IDR0_SEV) - smmu->features |=3D ARM_SMMU_FEAT_SEV; - - if (reg & IDR0_MSI) { - smmu->features |=3D ARM_SMMU_FEAT_MSI; - if (coherent && !disable_msipolling) - smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; - } - - if (reg & IDR0_HYP) { - smmu->features |=3D ARM_SMMU_FEAT_HYP; - if (cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) - smmu->features |=3D ARM_SMMU_FEAT_E2H; - } + if (coherent && !disable_msipolling && + smmu->features & ARM_SMMU_FEAT_MSI) + smmu->options |=3D ARM_SMMU_OPT_MSIPOLL; + if (smmu->features & ARM_SMMU_FEAT_HYP && + cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN)) + smmu->features |=3D ARM_SMMU_FEAT_E2H; =20 arm_smmu_get_httu(smmu, reg); =20 @@ -4877,21 +4837,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) dev_warn(smmu->dev, "IDR0.COHACC overridden by FW configuration (%s)\n", str_true_false(coherent)); =20 - switch (FIELD_GET(IDR0_STALL_MODEL, reg)) { - case IDR0_STALL_MODEL_FORCE: - smmu->features |=3D ARM_SMMU_FEAT_STALL_FORCE; - fallthrough; - case IDR0_STALL_MODEL_STALL: - smmu->features |=3D ARM_SMMU_FEAT_STALLS; - } - - if (reg & IDR0_S1P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S1; - - if (reg & IDR0_S2P) - smmu->features |=3D ARM_SMMU_FEAT_TRANS_S2; - - if (!(reg & (IDR0_S1P | IDR0_S2P))) { + if (!(smmu->features & (ARM_SMMU_FEAT_TRANS_S1 | ARM_SMMU_FEAT_TRANS_S2))= ) { dev_err(smmu->dev, "no translation support!\n"); return -ENXIO; } @@ -4950,10 +4896,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_= device *smmu) =20 /* IDR3 */ reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); - if (FIELD_GET(IDR3_RIL, reg)) - smmu->features |=3D ARM_SMMU_FEAT_RANGE_INV; - if (FIELD_GET(IDR3_FWB, reg)) - smmu->features |=3D ARM_SMMU_FEAT_S2FWB; + smmu->features |=3D smmu_idr3_features(reg); =20 if (FIELD_GET(IDR3_BBM, reg) =3D=3D 2) smmu->features |=3D ARM_SMMU_FEAT_BBML2; @@ -4965,43 +4908,18 @@ static int arm_smmu_device_hw_probe(struct arm_smmu= _device *smmu) smmu->evtq.max_stalls =3D FIELD_GET(IDR5_STALL_MAX, reg); =20 /* Page sizes */ - if (reg & IDR5_GRAN64K) - smmu->pgsize_bitmap |=3D SZ_64K | SZ_512M; - if (reg & IDR5_GRAN16K) - smmu->pgsize_bitmap |=3D SZ_16K | SZ_32M; - if (reg & IDR5_GRAN4K) - smmu->pgsize_bitmap |=3D SZ_4K | SZ_2M | SZ_1G; + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); =20 /* Input address size */ if (FIELD_GET(IDR5_VAX, reg) =3D=3D IDR5_VAX_52_BIT) smmu->features |=3D ARM_SMMU_FEAT_VAX; =20 - /* Output address size */ - switch (FIELD_GET(IDR5_OAS, reg)) { - case IDR5_OAS_32_BIT: - smmu->oas =3D 32; - break; - case IDR5_OAS_36_BIT: - smmu->oas =3D 36; - break; - case IDR5_OAS_40_BIT: - smmu->oas =3D 40; - break; - case IDR5_OAS_42_BIT: - smmu->oas =3D 42; - break; - case IDR5_OAS_44_BIT: - smmu->oas =3D 44; - break; - case IDR5_OAS_52_BIT: - smmu->oas =3D 52; + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) smmu->pgsize_bitmap |=3D 1ULL << 42; /* 4TB */ - break; - default: + else if (!smmu->oas) { dev_info(smmu->dev, - "unknown output address size. Truncating to 48-bit\n"); - fallthrough; - case IDR5_OAS_48_BIT: + "unknown output address size. Truncating to 48-bit\n"); smmu->oas =3D 48; } =20 diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 7be41dbe5aaa..64618299d03a 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1144,6 +1144,11 @@ int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_devi= ce *smmu, bool sync); int arm_smmu_cmdq_build_cmd(u64 *cmd, struct arm_smmu_cmdq_ent *ent); =20 +u32 smmu_idr0_features(u32 reg); +u32 smmu_idr3_features(u32 reg); +u32 smmu_idr5_to_oas(u32 reg); +unsigned long smmu_idr5_to_pgsize(u32 reg); + /* Queue functions shared between kernel and hyp. */ static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) { --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF09839DBFD for ; Fri, 1 May 2026 11:20:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634408; cv=none; b=VLf0NV6ARXRhxU5Ox8q1juLX6wIhOB22B++CFsx8H49fRe4Sz88pWSI+F9t82iFsyf/Dfg7N+ibWaEWnfo7rH5ENbCsizMd4ax1t2JMpCVR5V1fldxv2viYFTdD+iR2j+ZKU7os4mK9iebybR3HiMBaFdZzX8OBXuNxREhlmcj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634408; c=relaxed/simple; bh=s+i9FkVsDdqiDhtbenKjDnOOXgFPIjoZdSeFxYf+bdE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ERZDm6BQcWilBOh12XVXrB5IUFp4DXPfIcVtIXCgFEoaW2lxfuvXlV0UDpdzDhz1tQVL3Qzz4dCJgH/tfs/EYYvOgxvx8aPWpEu4oah1Zkl5cOo60KVo0vw/7IZz2MR6k+a7W9JbsOEUO5obFbh0Q1j5OvK4VF+47XgE7v9e/vI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=i0CvOYM0; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="i0CvOYM0" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-4411a529bc0so1474206f8f.0 for ; Fri, 01 May 2026 04:20:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634405; x=1778239205; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7C4551UfTUE9108SwsL7eQBuHqZOfGoFP6bfRW56kPQ=; b=i0CvOYM0w7x0Mn1HPxt/VWGVSvnffvcep0Trsb0L1f69cQ6GEDG62CjYHqHF4e7HHP hTHKKXKn4VWbHT1wN16OgnPmPSZgGXJP6Q0bg33VfdTB5P+W3jiwGsdk+vp5SaX7qFg7 86GTWE2RKcT5xZpAM55bQjXRNG/vPIhj7ww5UeTjH5l/TAaY/IrvQOXM66HnByCZmGkw j8OBKRUW9St++Ojdf7fIda9Hw093E6sIMZ3qXj9GL6SyDiI30SELwIzsr8ijfiPGil4m yXAo+o2dPMTYzj8g7/BSOqc9rcPfwlK0N5/bKUTpd6kRxkAy35VBM7JvTQXbn2i4eTF5 9xEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634405; x=1778239205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7C4551UfTUE9108SwsL7eQBuHqZOfGoFP6bfRW56kPQ=; b=PyvZNOk0RlJ04PSbSwPF1uAPc04vgjKMnVncuH2gu0hOK9e+m+JYl9DELwO7ccLPw/ OxCkNpVdtJ3rKRJkBrtPK+T3X+of+RnqAZyyGAMABKksIJozNDEEiwn1KToYEU3SZ0DP GX1XZBUOZgYRHxRmfYfKIRkOxLuEWpMBH6iiFcjqSlvTsGqE7YNd0Z79w6z8Lf1UMPpO sYkNp4gPidDyCOzagWpi5q+LUoSLM/c/RXIjw2NBVDuirdXe1pDDlhW9wK6Rjok2l92P v/cOB/iVHLm/F6aNFMpVd4ySfhZ+yo6zo9blWcLIojSI9H0aL7hH8xplXOxW/9RE1k9D GHtQ== X-Forwarded-Encrypted: i=1; AFNElJ/+5/agXl0DY/nEoODEaflHCTmvPB/GRHvDp0t3mU0uLbPLqA5/ykeccH2x8t45sYXd2XJOINLqrHsovMs=@vger.kernel.org X-Gm-Message-State: AOJu0YzX3/jS1VXQcomlzie5p/+ce6DZPQRbiIMY/D2NQfPQjWqlyEx+ GaNf/6xSiWh+udMaDOnOuVNdbbgLqRMIrsMkjl2krjx3g6qRur/jZxs1N5P/usxVyJyIPDSN+he z2K7WTwutv3pmcA== X-Received: from wmro23.prod.google.com ([2002:a05:600c:3797:b0:485:2f8c:9478]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c0ca:b0:488:b14f:b8ed with SMTP id 5b1f17b1804b1-48a83d1a406mr82848235e9.0.1777634405297; Fri, 01 May 2026 04:20:05 -0700 (PDT) Date: Fri, 1 May 2026 11:19:08 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-7-smostafa@google.com> Subject: [PATCH v6 06/25] iommu/io-pgtable-arm: Rework to use the iommu-pages API From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To prepare for supporting io-pgtable-arm in the pKVM hypervisor, we need to abstract away standard kernel allocations, frees, virt/phys conversions, and DMA API mapping. This patch introduces a set of generic wrappers in iommu-pages.h: - iommu_alloc_data - iommu_free_data - iommu_virt_to_phys - iommu_phys_to_virt - iommu_pages_dma_map - iommu_pages_dma_mapping_error - iommu_pages_dma_unmap The io-pgtable-arm.c code is updated to universally use these new wrappers instead of standard kernel kmalloc_obj, kfree, virt_to_phys, dma_map_single, etc. This abstraction makes it easy to replace them with hypervisor-specific implementations in a later patch. Signed-off-by: Mostafa Saleh --- drivers/iommu/io-pgtable-arm.c | 37 ++++++++++++++++------------------ drivers/iommu/iommu-pages.h | 36 +++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+), 20 deletions(-) diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index 0208e5897c29..e765021308f9 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -15,7 +15,6 @@ #include #include #include -#include =20 #include =20 @@ -143,7 +142,7 @@ #define ARM_MALI_LPAE_MEMATTR_WRITE_ALLOC 0x8DULL =20 /* IOPTE accessors */ -#define iopte_deref(pte,d) __va(iopte_to_paddr(pte, d)) +#define iopte_deref(pte, d) iommu_phys_to_virt(iopte_to_paddr(pte, d)) =20 #define iopte_type(pte) \ (((pte) >> ARM_LPAE_PTE_TYPE_SHIFT) & ARM_LPAE_PTE_TYPE_MASK) @@ -245,7 +244,7 @@ static inline bool arm_lpae_concat_mandatory(struct io_= pgtable_cfg *cfg, =20 static dma_addr_t __arm_lpae_dma_addr(void *pages) { - return (dma_addr_t)virt_to_phys(pages); + return (dma_addr_t)iommu_virt_to_phys(pages); } =20 static void *__arm_lpae_alloc_pages(size_t size, gfp_t gfp, @@ -272,15 +271,15 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_= t gfp, return NULL; =20 if (!cfg->coherent_walk) { - dma =3D dma_map_single(dev, pages, size, DMA_TO_DEVICE); - if (dma_mapping_error(dev, dma)) + dma =3D iommu_pages_dma_map(dev, pages, size); + if (iommu_pages_dma_mapping_error(dev, dma)) goto out_free; /* * We depend on the IOMMU being able to work with any physical * address directly, so if the DMA layer suggests otherwise by * translating or truncating them, that bodes very badly... */ - if (dma !=3D virt_to_phys(pages)) + if (dma !=3D iommu_virt_to_phys(pages)) goto out_unmap; } =20 @@ -288,7 +287,7 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t = gfp, =20 out_unmap: dev_err(dev, "Cannot accommodate DMA translation for IOMMU page tables\n"= ); - dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); + iommu_pages_dma_unmap(dev, dma, size); =20 out_free: if (cfg->free) @@ -304,8 +303,7 @@ static void __arm_lpae_free_pages(void *pages, size_t s= ize, void *cookie) { if (!cfg->coherent_walk) - dma_unmap_single(cfg->iommu_dev, __arm_lpae_dma_addr(pages), - size, DMA_TO_DEVICE); + iommu_pages_dma_unmap(cfg->iommu_dev, __arm_lpae_dma_addr(pages), size); =20 if (cfg->free) cfg->free(cookie, pages, size); @@ -316,8 +314,7 @@ static void __arm_lpae_free_pages(void *pages, size_t s= ize, static void __arm_lpae_sync_pte(arm_lpae_iopte *ptep, int num_entries, struct io_pgtable_cfg *cfg) { - dma_sync_single_for_device(cfg->iommu_dev, __arm_lpae_dma_addr(ptep), - sizeof(*ptep) * num_entries, DMA_TO_DEVICE); + iommu_pages_flush_incoherent(cfg->iommu_dev, ptep, 0, sizeof(*ptep) * num= _entries); } =20 static void __arm_lpae_clear_pte(arm_lpae_iopte *ptep, struct io_pgtable_c= fg *cfg, int num_entries) @@ -395,7 +392,7 @@ static arm_lpae_iopte arm_lpae_install_table(arm_lpae_i= opte *table, arm_lpae_iopte old, new; struct io_pgtable_cfg *cfg =3D &data->iop.cfg; =20 - new =3D paddr_to_iopte(__pa(table), data) | ARM_LPAE_PTE_TYPE_TABLE; + new =3D paddr_to_iopte(iommu_virt_to_phys(table), data) | ARM_LPAE_PTE_TY= PE_TABLE; if (cfg->quirks & IO_PGTABLE_QUIRK_ARM_NS) new |=3D ARM_LPAE_PTE_NSTABLE; =20 @@ -616,7 +613,7 @@ static void arm_lpae_free_pgtable(struct io_pgtable *io= p) struct arm_lpae_io_pgtable *data =3D io_pgtable_to_data(iop); =20 __arm_lpae_free_pgtable(data, data->start_level, data->pgd); - kfree(data); + iommu_free_data(data); } =20 static size_t __arm_lpae_unmap(struct arm_lpae_io_pgtable *data, @@ -930,7 +927,7 @@ arm_lpae_alloc_pgtable(struct io_pgtable_cfg *cfg) if (cfg->oas > ARM_LPAE_MAX_ADDR_BITS) return NULL; =20 - data =3D kmalloc_obj(*data); + data =3D iommu_alloc_data(sizeof(*data), GFP_KERNEL); if (!data) return NULL; =20 @@ -1053,11 +1050,11 @@ arm_64_lpae_alloc_pgtable_s1(struct io_pgtable_cfg = *cfg, void *cookie) wmb(); =20 /* TTBR */ - cfg->arm_lpae_s1_cfg.ttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s1_cfg.ttbr =3D iommu_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + iommu_free_data(data); return NULL; } =20 @@ -1149,11 +1146,11 @@ arm_64_lpae_alloc_pgtable_s2(struct io_pgtable_cfg = *cfg, void *cookie) wmb(); =20 /* VTTBR */ - cfg->arm_lpae_s2_cfg.vttbr =3D virt_to_phys(data->pgd); + cfg->arm_lpae_s2_cfg.vttbr =3D iommu_virt_to_phys(data->pgd); return &data->iop; =20 out_free_data: - kfree(data); + iommu_free_data(data); return NULL; } =20 @@ -1223,7 +1220,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) /* Ensure the empty pgd is visible before TRANSTAB can be written */ wmb(); =20 - cfg->arm_mali_lpae_cfg.transtab =3D virt_to_phys(data->pgd) | + cfg->arm_mali_lpae_cfg.transtab =3D iommu_virt_to_phys(data->pgd) | ARM_MALI_LPAE_TTBR_READ_INNER | ARM_MALI_LPAE_TTBR_ADRMODE_TABLE; if (cfg->coherent_walk) @@ -1232,7 +1229,7 @@ arm_mali_lpae_alloc_pgtable(struct io_pgtable_cfg *cf= g, void *cookie) return &data->iop; =20 out_free_data: - kfree(data); + iommu_free_data(data); return NULL; } =20 diff --git a/drivers/iommu/iommu-pages.h b/drivers/iommu/iommu-pages.h index ae9da4f571f6..e1945193ad7f 100644 --- a/drivers/iommu/iommu-pages.h +++ b/drivers/iommu/iommu-pages.h @@ -7,6 +7,7 @@ #ifndef __IOMMU_PAGES_H #define __IOMMU_PAGES_H =20 +#include #include =20 /** @@ -145,4 +146,39 @@ void iommu_pages_stop_incoherent_list(struct iommu_pag= es_list *list, void iommu_pages_free_incoherent(void *virt, struct device *dma_dev); #endif =20 +static inline void *iommu_alloc_data(size_t size, gfp_t gfp) +{ + return kmalloc(size, gfp); +} + +static inline void iommu_free_data(void *p) +{ + kfree(p); +} + +static inline phys_addr_t iommu_virt_to_phys(void *virt) +{ + return virt_to_phys(virt); +} + +static inline void *iommu_phys_to_virt(phys_addr_t phys) +{ + return phys_to_virt(phys); +} + +static inline dma_addr_t iommu_pages_dma_map(struct device *dev, void *vir= t, size_t size) +{ + return dma_map_single(dev, virt, size, DMA_TO_DEVICE); +} + +static inline int iommu_pages_dma_mapping_error(struct device *dev, dma_ad= dr_t dma) +{ + return dma_mapping_error(dev, dma); +} + +static inline void iommu_pages_dma_unmap(struct device *dev, dma_addr_t dm= a, size_t size) +{ + dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); +} + #endif /* __IOMMU_PAGES_H */ --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A171A39EF1D for ; Fri, 1 May 2026 11:20:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634410; cv=none; b=rgRcTDkam+O1dnAcylWnjGDfgi+HbHdbIwhg2aXwZd9JszcYwOzen+UALiZfouafstvW/3EXb3aaeXMxvnUS7GvSmFCT2DainNw7CFo00mEG59sORckGtXtlMUZgJJhFh3GsXerViXCR/MrCH3QT/6Nx/LW6YwhF+kTQCvNxAZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634410; c=relaxed/simple; bh=7BiQeDIdExKTrYvjjivfx/XCGpvMdX1OxrkfqO2/Rss=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ulmfE76ujgp3DOeTLQtDPYobZAcUQvCt3I1jbJLLLD+R10c/X7sUPq0Ual065V4zSPwqBKnJD3KB1N04ylTaCTX+ZEpcuat3mceaCaosc9Yq18cbqpJkOJsKy4bcUsY3L970MPCs/8phgSZe3Ev2F2PSiz8qo+dwYvGv/OeVpWs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LfhZhGxU; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LfhZhGxU" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-43ff19e54beso1245379f8f.2 for ; Fri, 01 May 2026 04:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634407; x=1778239207; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PcUXILApPYbJmZVAe9rvUD8ScMEsBCpayhLYZpcyGpA=; b=LfhZhGxU02n2MIRYReD6HaNMLjy/cUL3IQ5EyHeXXksGb7sWaxmuV/+XnVITxd0QVH S763XdaXQvZXWZh0+fjKSW9uwIz74e+EdLjLXapKk9ZjMm4IqZy4AALR8sP1vFXmZYuk 6CBkxCq7OiG4WEHZnhqczB7Hz0PGE+ZVqdZuJms9Xvfcno39IWHrmki3obfGbDuQrVca QZ8OQU07Tk+tfRqt9/LAAm+Cy0iF5v7u/vEpKBI05WIfQ2hsbav01WLeSip4lRnoCwt5 6o6B7u38J8jR7HdIIU7gVmpSASX3QBLU6xL8RwbeKDZZST2iiQcCZhN+pEvr3HCuEW0X ADDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634407; x=1778239207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PcUXILApPYbJmZVAe9rvUD8ScMEsBCpayhLYZpcyGpA=; b=UEO7zJR72zZpSZ+caU3oH7tJj6ZjLjeVZQ03QcmPHU3D7h2LYQTNvwr8gA4Dojnug6 KwszFHWvdSt/eKmaY0Sq/mRiXVNaRXs0rEGxsrm6HfqOBzsP7ZG9grPSOsnV7Skvz8Rf EFcl0WRHQIQ5biQs/mlI4vEUbYf+iStXbeIaqbvV8aixvpuQcYp1OLRA5kgA7tS4T2DL J3fwtc3Eo2Dl/imuwy4iPs9oNkfrCM25bp7RO31ZnEEB66gSiYuXv3DG3AEyMozu3sBG OaEnHQOAxJyerGLxEFvI9Pqp/FHP1e5q8Xt9dkVnXQh9+PMIRPjD7N+k3r09ewXTQqkX CfDg== X-Forwarded-Encrypted: i=1; AFNElJ82nGSKZ3Kjx7v2r54OIEXU6+ZVc3J/nlLjvryW6dGYCtS6A/saXADqnjQbQKgt8dGTyDdhM0r9aCi0Iqw=@vger.kernel.org X-Gm-Message-State: AOJu0Yy9+mqTTVgQnL7rFn+xkG1x3zEc7RLl8f4oP5gV+fxZfLYHl71M yQ9K51T9IWKChV0kx59fJwcYco6Ceferwb8ToAgxmTVtDvd9HlohHerxlR+lqvd97Hw6CQImLi/ eJc/FGwSkbVYuxQ== X-Received: from wmrk4.prod.google.com ([2002:a05:600c:b44:b0:48a:6a1b:6c3b]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600d:a:b0:489:1d74:56d with SMTP id 5b1f17b1804b1-48a84532792mr89794945e9.29.1777634406991; Fri, 01 May 2026 04:20:06 -0700 (PDT) Date: Fri, 1 May 2026 11:19:09 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-8-smostafa@google.com> Subject: [PATCH v6 07/25] KVM: arm64: iommu: Introduce IOMMU driver infrastructure From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To establish DMA isolation, KVM needs an IOMMU driver which provides ops implemented at EL2. Only one driver can be used and is registered with kvm_iommu_register_driver() by passing pointer to the ops. This must be called before module_init() which is the point KVM initializes. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/include/asm/kvm_host.h | 5 +++++ arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 13 +++++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 20 +++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/setup.c | 5 +++++ arch/arm64/kvm/iommu.c | 26 +++++++++++++++++++++++++ 7 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/nvhe/iommu.h create mode 100644 arch/arm64/kvm/hyp/nvhe/iommu/iommu.c create mode 100644 arch/arm64/kvm/iommu.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 851f6171751c..52898d2a3ec6 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1733,4 +1733,9 @@ static __always_inline enum fgt_group_id __fgt_reg_to= _group_id(enum vcpu_sysreg =20 long kvm_get_cap_for_kvm_ioctl(unsigned int ioctl, long *ext); =20 +#ifndef __KVM_NVHE_HYPERVISOR__ +struct kvm_iommu_ops; +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); +#endif + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 59612d2f277c..0ddef54f7434 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -24,7 +24,7 @@ kvm-y +=3D arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.= o \ vgic/vgic-mmio.o vgic/vgic-mmio-v2.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o vgic/vgic-v3-nested.o \ - vgic/vgic-v5.o + vgic/vgic-v5.o iommu.o =20 kvm-$(CONFIG_HW_PERF_EVENTS) +=3D pmu-emul.o pmu.o kvm-$(CONFIG_ARM64_PTR_AUTH) +=3D pauth.o diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h new file mode 100644 index 000000000000..1ac70cc28a9e --- /dev/null +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NVHE_IOMMU_H__ +#define __ARM64_KVM_NVHE_IOMMU_H__ + +#include + +struct kvm_iommu_ops { + int (*init)(void); +}; + +int kvm_iommu_init(void); + +#endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 89d0533921f9..606c0e1b7bd0 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -24,7 +24,8 @@ CFLAGS_switch.nvhe.o +=3D -Wno-override-init =20 hyp-obj-y :=3D timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o= host.o \ hyp-main.o hyp-smp.o psci-relay.o early_alloc.o page_alloc.o \ - cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o + cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o \ + iommu/iommu.o hyp-obj-y +=3D ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../en= try.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o ../vgic-v5-sr.o c= lock.o hyp-obj-y +=3D ../../../kernel/smccc-call.o diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c new file mode 100644 index 000000000000..406c8fb9b3b9 --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * IOMMU operations for pKVM + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +/* Only one set of ops supported */ +struct kvm_iommu_ops *kvm_iommu_ops; + + +int kvm_iommu_init(void) +{ + /* Keep DMA isolation optional. */ + if (!kvm_iommu_ops || !kvm_iommu_ops->init) + return 0; + + return kvm_iommu_ops->init(); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index 8041f6e80cd1..1f6b221db9a0 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -329,6 +330,10 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 + ret =3D kvm_iommu_init(); + if (ret) + goto out; + ret =3D hyp_ffa_init(ffa_proxy_pages); if (ret) goto out; diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c new file mode 100644 index 000000000000..f247384fa193 --- /dev/null +++ b/arch/arm64/kvm/iommu.c @@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2025 Google LLC + * Author: Mostafa Saleh + */ + +#include + +extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); + +static DEFINE_MUTEX(kvm_iommu_reg_lock); + +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) +{ + guard(mutex)(&kvm_iommu_reg_lock); + + /* Only protected KVM before de-privilege. */ + if (!is_protected_kvm_enabled() || is_kvm_arm_initialised()) + return -EINVAL; + + if (kvm_nvhe_sym(kvm_iommu_ops)) + return -EBUSY; + + kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; + return 0; +} --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9708039EF2B for ; Fri, 1 May 2026 11:20:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634414; cv=none; b=ddW913oPdQ9DI3TyEsessyKIFApKkkcCRu+yOOigg+5Pq7g5fveoXADZIzD4Lu6dzF3WZWM8XqPg8wHy7Y3z9L3TlFOvhdXuiyxADn9TKwrDsq0/2/eeO1+zK0EuCTYMT3NYcwv8HIU6rssyQUzfGvK6bP9MqhE/wmzP+ymKLik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634414; c=relaxed/simple; bh=4dvaQhefj/gLOS6fMysm6OEgJIPg+rENa3gunBbX4+k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jcUtk+iBKxYPL4P9JUqyI9bFIQWqERH/1h7425o1zjkLdmklBVWgedSyxQNdYLIv5yi6tkTBTpscEqfc5ATfHJ+7rSH7A1lZb0b4boLN/Lja1BLomyFkVAVfF6meuBjDUrStqluVlsm3Wp1eTrbjibGX8VJD+DvCTICpz6xfJTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sjw4L3N/; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sjw4L3N/" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-488bd1ee9e7so20148385e9.1 for ; Fri, 01 May 2026 04:20:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634409; x=1778239209; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FN4paQwo2ROc1ZHKgUwjGYJd1quRshkryUW2WW7y3qE=; b=sjw4L3N/t1MPvTg9pOzRcLV5Qn2bWJp05PXUNBOTo+VT8BzTsOyFsJOJG97yx6bjwx S62KFfxYnmJl52aGf4Dvy48kb61gdAGg2ZWE4XrVtzQCsguxY2TopKsxdp2StxsfuvJA I7a7i6Zz2JcXaq/EjYxLpE7G6CVAplF83jmbhEILOhnVsKR9jagyiBjB2FsaP6gX9yvy FwO4yyZDPyfJSKzZwsBVO73pbmneRBjmjGTlDT/Wv5GyaTg5GkIZTbPNit33F++7s/Jm Jn+bPfNfnjQppV6SMd2lUv//9Sn2M+gtEdbqWXfESd5+RJSSx5istaUeVXfzLBFJ3RK0 X2WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634409; x=1778239209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FN4paQwo2ROc1ZHKgUwjGYJd1quRshkryUW2WW7y3qE=; b=KV4v7G9NJWgHIKPW4bayMChBi8jBgxGAL60xbNkhgOliA2ea/KzP1iFNBx1osIZSiD /pclo6uBGQ6LZqwcQPK74Z0KzA1miQ4qUcLp0OgD2fm5SZWagZtBTxIJ5RSat+Z6cE++ Ik7xUMTdTLxVey2keyWumWAdpYda4RZqRNy3Ywhzu6pxGcnhUV7ln3p4oePrlXQ8FWB6 rs0Xo5hP1oPeEKFSPdmkLMbe7iDzOna1sImfVh75prQGBnikVlqGYL7Pkr2V0JvzlU/Q HRypO0RjcN2zOuoN35LAK/GxNaqOW5T/xMOYp+Xay3zPHj0NjSUAm6L4/98MufV/I9kG MlMQ== X-Forwarded-Encrypted: i=1; AFNElJ/fGmZSlxU7KDz8BI1laXakouQLBf5p/8k5GTXXq8VbbG9lnDhQRWxWVzQor8wpfP21NAgKWGLGpAiP5Pw=@vger.kernel.org X-Gm-Message-State: AOJu0YwwhoE/JwkeJAqvHn1k5u9uMseve7pAOg3k7OCjhhPZUOTimRKM zopJIXTsS/usHPaLC4t9AfOGhqEOM9P9bOehxavqbIvT7PVmZ9ViUWq8Gz5BhMpnj+hLC77+we8 XqbyO7MK06pd2XA== X-Received: from wmlc6.prod.google.com ([2002:a7b:c846:0:b0:48a:8963:49ee]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:818c:b0:48a:5c23:cab with SMTP id 5b1f17b1804b1-48a8eb985b1mr43676235e9.19.1777634408826; Fri, 01 May 2026 04:20:08 -0700 (PDT) Date: Fri, 1 May 2026 11:19:10 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-9-smostafa@google.com> Subject: [PATCH v6 08/25] KVM: arm64: iommu: Shadow host stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Create a page-table for the IOMMU that shadows the host CPU stage-2 to establish DMA isolation. An initial snapshot is created after the driver init, then on every permission change a callback would be called for the IOMMU driver to update the page table. There are 3 different ways to add the callback: 1) In the high level memory transitions: (__pkvm_host_donate_hyp(), __pkvm_host_donate_guest()... 2) In Lower level functions covering all transitions - host_stage2_set_owner_metadata_locked() which covers: - __pkvm_host_donate_hyp() - __pkvm_host_donate_guest() - __pkvm_host_donate_hyp() - __pkvm_guest_unshare_host() - host_stage2_set_owner_locked() only for ID_HOST which covers: - __pkvm_hyp_donate_host() - __pkvm_host_force_reclaim_page_guest() - __pkvm_host_reclaim_page_guest() - __pkvm_guest_share_host() 3) In the lowest level function __host_update_page_state(), which requires only one callback. However, in that case the page state is not enough as we might need to know the old state also. Option #3 was implemented here. For some cases, an SMMUv3 may be able to share the same page-table used with the host CPU stage-2 directly. However, this is too strict and requires changes to the core hypervisor page-table code, plus it would require the hypervisor to handle IOMMU page-faults. This can be added later as an optimization for SMMUV3. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 4 + arch/arm64/kvm/hyp/include/nvhe/mem_protect.h | 2 + arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 108 +++++++++++++++++- arch/arm64/kvm/hyp/nvhe/mem_protect.c | 35 ++++++ 4 files changed, 146 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 1ac70cc28a9e..6277d845cdcf 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -3,11 +3,15 @@ #define __ARM64_KVM_NVHE_IOMMU_H__ =20 #include +#include =20 struct kvm_iommu_ops { int (*init)(void); + int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 int kvm_iommu_init(void); =20 +int kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h b/arch/arm64/kvm= /hyp/include/nvhe/mem_protect.h index ff440204d2c7..f7faecc3b70a 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h +++ b/arch/arm64/kvm/hyp/include/nvhe/mem_protect.h @@ -54,6 +54,8 @@ int __pkvm_host_test_clear_young_guest(u64 gfn, u64 nr_pa= ges, bool mkold, struct int __pkvm_host_mkyoung_guest(u64 gfn, struct pkvm_hyp_vcpu *vcpu); =20 bool addr_is_memory(phys_addr_t phys); +u64 find_mem_range_from(u64 start, bool *is_memory); + int host_stage2_idmap_locked(phys_addr_t addr, u64 size, enum kvm_pgtable_= prot prot); int host_stage2_set_owner_locked(phys_addr_t addr, u64 size, u8 owner_id); int kvm_host_prepare_stage2(void *pgt_pool_base); diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index 406c8fb9b3b9..1db52bd87c38 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,17 +4,119 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + #include +#include +#include =20 /* Only one set of ops supported */ struct kvm_iommu_ops *kvm_iommu_ops; =20 +/* Protected by host_mmu.lock */ +static bool kvm_idmap_initialized; + +static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) +{ + int iommu_prot =3D 0; + + if (prot & KVM_PGTABLE_PROT_R) + iommu_prot |=3D IOMMU_READ; + if (prot & KVM_PGTABLE_PROT_W) + iommu_prot |=3D IOMMU_WRITE; + + /* We don't understand that, might be dangerous. */ + WARN_ON(prot & ~PKVM_HOST_MEM_PROT); + return iommu_prot; +} + +static int __snapshot_host_stage2(const struct kvm_pgtable_visit_ctx *ctx, + enum kvm_pgtable_walk_flags visit) +{ + u64 start =3D ctx->addr; + u64 end =3D start + kvm_granule_size(ctx->level); + kvm_pte_t pte =3D *ctx->ptep; + bool is_memory; + u64 region_end; + int prot; + int ret; + + /* + * Keep annotated PTEs unmapped, and map everything else even lazily + * mapped MMIO with pte =3D=3D 0, as the IOMMU can't handle page faults. + * That maps the whole address space which can be large, but that doesn't + * use a lot of memory as it will be mostly large block (1 GB with 4kb pa= ges) + */ + if (pte && !kvm_pte_valid(pte)) + return 0; + + if (kvm_pte_valid(pte)) { + prot =3D pkvm_to_iommu_prot(kvm_pgtable_stage2_pte_prot(pte)); + /* If the range is mapped in a single PTE, it must be the same type.*/ + if (!addr_is_memory(start)) + prot |=3D IOMMU_MMIO; + + return kvm_iommu_ops->host_stage2_idmap(start, end, prot); + } + + /* In case of invalid PTE, we need to figure out which part of it is MMIO= */ + do { + prot =3D IOMMU_READ | IOMMU_WRITE; + region_end =3D find_mem_range_from(start, &is_memory); + region_end =3D min(end, region_end); + if (!is_memory) + prot |=3D IOMMU_MMIO; + + ret =3D kvm_iommu_ops->host_stage2_idmap(start, region_end, prot); + if (ret) + return ret; + + start =3D region_end; + } while (start < end); + + return 0; +} + +static int kvm_iommu_snapshot_host_stage2(void) +{ + int ret; + struct kvm_pgtable_walker walker =3D { + .cb =3D __snapshot_host_stage2, + .flags =3D KVM_PGTABLE_WALK_LEAF, + }; + struct kvm_pgtable *pgt =3D &host_mmu.pgt; + + hyp_spin_lock(&host_mmu.lock); + ret =3D kvm_pgtable_walk(pgt, 0, BIT(pgt->ia_bits), &walker); + /* Start receiving calls to host_stage2_idmap. */ + kvm_idmap_initialized =3D !ret; + hyp_spin_unlock(&host_mmu.lock); + + return ret; +} =20 int kvm_iommu_init(void) { - /* Keep DMA isolation optional. */ - if (!kvm_iommu_ops || !kvm_iommu_ops->init) + int ret; + + if (!kvm_iommu_ops || !kvm_iommu_ops->init || + !kvm_iommu_ops->host_stage2_idmap) + return 0; + + ret =3D kvm_iommu_ops->init(); + if (ret) + return ret; + + return kvm_iommu_snapshot_host_stage2(); +} + +int kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, + enum kvm_pgtable_prot prot) +{ + hyp_assert_lock_held(&host_mmu.lock); + + if (!kvm_idmap_initialized) return 0; =20 - return kvm_iommu_ops->init(); + return kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(pr= ot)); } diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index 2fb20a63a417..b54cb72ed88c 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -15,6 +15,7 @@ #include =20 #include +#include #include #include #include @@ -481,6 +482,14 @@ static int check_range_allowed_memory(u64 start, u64 e= nd) return 0; } =20 +u64 find_mem_range_from(u64 start, bool *is_memory) +{ + struct kvm_mem_range r; + + *is_memory =3D !!find_mem_range(start, &r); + return r.end; +} + static bool range_is_memory(u64 start, u64 end) { struct kvm_mem_range r; @@ -577,8 +586,34 @@ int host_stage2_idmap_locked(phys_addr_t addr, u64 siz= e, =20 static void __host_update_page_state(phys_addr_t addr, u64 size, enum pkvm= _page_state state) { + enum pkvm_page_state old =3D get_host_state(hyp_phys_to_page(addr)); + enum kvm_pgtable_prot prot =3D 0; + for_each_hyp_page(page, addr, size) set_host_state(page, state); + + /* + * Any transition to PKVM_NOPAGE, unmaps the page from the host + * Any transition to PKVM_PAGE_SHARED_BORROWED, maps the page in the host + * Any transition to PKVM_PAGE_SHARED_OWNED is ignored as page is already= mapped. + * Transitions to PKVM_PAGE_OWNED from anything but PKVM_NOPAGE are ignor= ed. + * Transitions to PKVM_PAGE_OWNED from PKVM_NOPAGE will map the page. + */ + if ((state =3D=3D PKVM_PAGE_SHARED_OWNED) || + ((state =3D=3D PKVM_PAGE_OWNED) && (old !=3D PKVM_NOPAGE))) + return; + + if ((state =3D=3D PKVM_PAGE_SHARED_BORROWED) || + (state =3D=3D PKVM_PAGE_OWNED)) + prot =3D PKVM_HOST_MEM_PROT; + + /* + * Only update the IOMMU from here, as MMIO can't transition after + * de-privilege, that will need to change when device assignment + * is supported. + * And WARN on failure as we can't unroll at this point. + */ + WARN_ON(kvm_iommu_host_stage2_idmap(addr, addr + size, prot)); } =20 #define KVM_HOST_DONATION_PTE_OWNER_MASK GENMASK(3, 1) --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F7D939F176 for ; Fri, 1 May 2026 11:20:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634415; cv=none; b=CMmQXh9+9Ufcb3v1dHWqyUH+TeFFbFiKBodegVzfz7D6GdEKbaLA33YbZccga9AbgI0lUJpWA1ZqxelUyFoBJQjpVU3jas8UvRIKL2OtUZh28L/G/Fh0GtKGst5kPs/+S+vl6pJtGFBVxvRec84wkFiZzv0ZMyH/fbxZulHovbU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634415; c=relaxed/simple; bh=qxqKnqxfNJ7XYWVGc81dtZPlQ0S/fgywNS1fFSbtZeI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WsUeHon8GSGZgtausU3IDH3oRTVmI3uYJ1Tc7IEF6WvA5la2gQcGsbYklXH/CoeQStm7aQvs5iWgCyJWykgqlwGF0AZ7qoZrEdZGZx87sOAF7GJ3wo10ayPSWbalaMqd02De8GjXpiX5nmG4R64UtDfquoEl0LRAtJRw5Svl1k4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=D2pAjV5X; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="D2pAjV5X" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4836abfc742so14273395e9.0 for ; Fri, 01 May 2026 04:20:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634410; x=1778239210; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Rodgv7A+aYOiGQUhnQ9CHMotwYGD2z3moCBzLHWH+C4=; b=D2pAjV5Xj7KDoXkkmGCsPgSbgCJ9WeGgD8TsciTuWYF+UUUMIyAge3eFx9ACSZJ8HX pXunYcPJgQGHzn9O6UJVZEyEzh7TnstLIMDicK+7/SUade8P69lGiw2zkScXh7bqhP+8 ujKqfB0j+jXzPYPyuQXrN4ro9UVM4jvqUtouz65W0fkXIy849roZXz8yhyXENR6C33Rz SBWGYOr41DovTkuHFWhZNskFbb+IP1wz1OxekT/pCfJm0mDHLN04ETX3zkz1+DleLhHf 5FHd8n3SzPtwBKgT172FD9veN66I5pJjqga7diRz0NDvFreQwcbteoWwmWCfWRXvaoa0 5z/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634410; x=1778239210; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rodgv7A+aYOiGQUhnQ9CHMotwYGD2z3moCBzLHWH+C4=; b=b/iojFW+oF/oT0h+FzgI7BFpqshi4QvLmTdrfAzwOG2U0xAldNpAYI+UmNu772/MY2 JnHXvdNzMPsID1NnXcsNX//uRegyvIm1B0BW+F58CjCx5KJVshSSbx5O8duVr7zr9slt u5YkMkMtAQxwwGDZDxzEMkuyi0gi2kdIVVxLPQE1SCncVlXEA9HeVZi3E3Z3OTDTpXiq qO/B+79gPKtKbiFn5GsV7Dzk7Bhonj+8IZZC/CYYNRvbqqHAQGT4HtdlIlmUg+ZIK3Ya VtlRaBSbZjbjXUmtDbUtQ40kdTVZn4wYUkQTrWLW/jlUd81Aiv0B7Q976bBAIY8tZ1HC 2cgQ== X-Forwarded-Encrypted: i=1; AFNElJ8ACLncdlt5X8Iw2RDKHR/jt59/QKE/fyOCfuY8gkJg5249PTRQ2nuyIhd4qdQC98PnLY0RbP+dzg1BHyc=@vger.kernel.org X-Gm-Message-State: AOJu0YzUVSYgVdTpfi22RuPRzu8kVpfgn3CprIGCsYzzZa3c8TWZfFdu M3grv6Tr/+kgt5yJLV/bSD56jUfmJC0xwDs8ZBK63zjlLPKKrdeB0pmOrl0RLGX7zHd/eI74e/f TKvWd4Lkc66Po2Q== X-Received: from wmbf9.prod.google.com ([2002:a05:600c:5949:b0:48a:79a9:335c]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3e10:b0:488:ffb1:494c with SMTP id 5b1f17b1804b1-48a84456732mr106489765e9.12.1777634410490; Fri, 01 May 2026 04:20:10 -0700 (PDT) Date: Fri, 1 May 2026 11:19:11 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-10-smostafa@google.com> Subject: [PATCH v6 09/25] KVM: arm64: iommu: Add memory pool From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" IOMMU drivers would require to allocate memory for the shadow page table. Similar to the host stage-2 CPU page table, the IOMMU pool is allocated early from the carveout and it's memory is added in a pool which the IOMMU driver can allocate from and reclaim at run time. As this is too early for drivers to use init calls, set the number of page allocated from the kernel command line "kvm-arm.hyp_iommu_pages". Later when the driver registers, it will pass how many pages it needs, and if it was less than what was allocated, it will fail to register. Signed-off-by: Mostafa Saleh --- .../admin-guide/kernel-parameters.txt | 4 +++ arch/arm64/include/asm/kvm_host.h | 3 +- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 7 +++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 21 +++++++++++- arch/arm64/kvm/hyp/nvhe/setup.c | 12 ++++++- arch/arm64/kvm/iommu.c | 33 ++++++++++++++++++- arch/arm64/kvm/pkvm.c | 1 + 7 files changed, 76 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index cf3807641d89..5e49946ff7ed 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3283,6 +3283,10 @@ Kernel parameters trap: set WFI instruction trap =20 notrap: clear WFI instruction trap + kvm-arm.hyp_iommu_pages=3D + [KVM, ARM, EARLY] + Number of pages allocated for the IOMMU pool from the + KVM carveout when running in protected mode. =20 kvm_cma_resv_ratio=3Dn [PPC,EARLY] Reserves given percentage from system memory area for diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm= _host.h index 52898d2a3ec6..17f4cce86ec3 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1735,7 +1735,8 @@ long kvm_get_cap_for_kvm_ioctl(unsigned int ioctl, lo= ng *ext); =20 #ifndef __KVM_NVHE_HYPERVISOR__ struct kvm_iommu_ops; -int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops); +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops, unsigned int = pool_pages); +unsigned int kvm_iommu_pages(void); #endif =20 #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index 6277d845cdcf..eba94b4f6050 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -10,8 +10,13 @@ struct kvm_iommu_ops { int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); }; =20 -int kvm_iommu_init(void); +int kvm_iommu_init(void *pool_base, unsigned int nr_pages); =20 int kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, enum kvm_pgtable_prot prot); + +/* Returns zeroed memory. */ +void *kvm_iommu_donate_pages(u8 order); +void kvm_iommu_reclaim_pages(void *ptr); + #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index 1db52bd87c38..53cb5e4b0aac 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -15,6 +15,7 @@ struct kvm_iommu_ops *kvm_iommu_ops; =20 /* Protected by host_mmu.lock */ static bool kvm_idmap_initialized; +static struct hyp_pool iommu_pages_pool; =20 static inline int pkvm_to_iommu_prot(enum kvm_pgtable_prot prot) { @@ -95,7 +96,7 @@ static int kvm_iommu_snapshot_host_stage2(void) return ret; } =20 -int kvm_iommu_init(void) +int kvm_iommu_init(void *pool_base, unsigned int nr_pages) { int ret; =20 @@ -103,6 +104,14 @@ int kvm_iommu_init(void) !kvm_iommu_ops->host_stage2_idmap) return 0; =20 + if (!nr_pages) + return -ENOMEM; + + ret =3D hyp_pool_init(&iommu_pages_pool, hyp_virt_to_pfn(pool_base), + nr_pages, 0); + if (ret) + return ret; + ret =3D kvm_iommu_ops->init(); if (ret) return ret; @@ -120,3 +129,13 @@ int kvm_iommu_host_stage2_idmap(phys_addr_t start, phy= s_addr_t end, =20 return kvm_iommu_ops->host_stage2_idmap(start, end, pkvm_to_iommu_prot(pr= ot)); } + +void *kvm_iommu_donate_pages(u8 order) +{ + return hyp_alloc_pages(&iommu_pages_pool, order); +} + +void kvm_iommu_reclaim_pages(void *ptr) +{ + hyp_put_page(&iommu_pages_pool, ptr); +} diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setu= p.c index 1f6b221db9a0..215014e42c27 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -23,6 +23,9 @@ =20 unsigned long hyp_nr_cpus; =20 +/* See kvm_iommu_pages() */ +unsigned int hyp_kvm_iommu_pages; + #define hyp_percpu_size ((unsigned long)__per_cpu_end - \ (unsigned long)__per_cpu_start) =20 @@ -34,6 +37,7 @@ static void *selftest_base; static void *ffa_proxy_pages; static struct kvm_pgtable_mm_ops pkvm_pgtable_mm_ops; static struct hyp_pool hpool; +static void *iommu_base; =20 static int divide_memory_pool(void *virt, unsigned long size) { @@ -71,6 +75,12 @@ static int divide_memory_pool(void *virt, unsigned long = size) if (!ffa_proxy_pages) return -ENOMEM; =20 + if (hyp_kvm_iommu_pages) { + iommu_base =3D hyp_early_alloc_contig(hyp_kvm_iommu_pages); + if (!iommu_base) + return -ENOMEM; + } + return 0; } =20 @@ -330,7 +340,7 @@ void __noreturn __pkvm_init_finalise(void) if (ret) goto out; =20 - ret =3D kvm_iommu_init(); + ret =3D kvm_iommu_init(iommu_base, hyp_kvm_iommu_pages); if (ret) goto out; =20 diff --git a/arch/arm64/kvm/iommu.c b/arch/arm64/kvm/iommu.c index f247384fa193..213429ceb549 100644 --- a/arch/arm64/kvm/iommu.c +++ b/arch/arm64/kvm/iommu.c @@ -7,10 +7,11 @@ #include =20 extern struct kvm_iommu_ops *kvm_nvhe_sym(kvm_iommu_ops); +extern unsigned int kvm_nvhe_sym(hyp_kvm_iommu_pages); =20 static DEFINE_MUTEX(kvm_iommu_reg_lock); =20 -int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops) +int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_ops, unsigned int = pool_pages) { guard(mutex)(&kvm_iommu_reg_lock); =20 @@ -21,6 +22,36 @@ int kvm_iommu_register_driver(struct kvm_iommu_ops *hyp_= ops) if (kvm_nvhe_sym(kvm_iommu_ops)) return -EBUSY; =20 + /* See kvm_iommu_pages() */ + if (pool_pages > kvm_nvhe_sym(hyp_kvm_iommu_pages)) { + kvm_err("Not enough memory for the IOMMU pool, need 0x%x pages, check kv= m-arm.hyp_iommu_pages", + pool_pages); + return -ENOMEM; + } + kvm_nvhe_sym(kvm_iommu_ops) =3D hyp_ops; return 0; } + +unsigned int kvm_iommu_pages(void) +{ + /* + * This is used very early during setup_arch() before any initcalls + * or any drivers are registered. + * This value is set by a command line option. + * Later, when the driver is registered, it will pass the number + * pages needed for it's page tables, if it was less that what + * the system has already allocated, the registration will fail. + */ + return kvm_nvhe_sym(hyp_kvm_iommu_pages); +} + +/* Number of pages to reserve for iommu pool*/ +static int __init early_hyp_iommu_pages(char *arg) +{ + if (!arg) + return -EINVAL; + + return kstrtouint(arg, 0, &kvm_nvhe_sym(hyp_kvm_iommu_pages)); +} +early_param("kvm-arm.hyp_iommu_pages", early_hyp_iommu_pages); diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 053e4f733e4b..79dd14db4919 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -63,6 +63,7 @@ void __init kvm_hyp_reserve(void) hyp_mem_pages +=3D hyp_vmemmap_pages(STRUCT_HYP_PAGE_SIZE); hyp_mem_pages +=3D pkvm_selftest_pages(); hyp_mem_pages +=3D hyp_ffa_proxy_pages(); + hyp_mem_pages +=3D kvm_iommu_pages(); =20 /* * Try to allocate a PMD-aligned region to reduce TLB pressure once --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AC1B39EF1D for ; Fri, 1 May 2026 11:20:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634417; cv=none; b=u2aynsMchb86txf0+rCbmYwx1xR+von2tMheduThzQaeaFnRgt4BZ6LJCfC9CRoP4WFmTN8ujW2guUAy38U5TrjvXeyoL+85YvFgsLAJNPWxCxjc0sIr5IyIa2F66B9rPlbgw//TKO1E6xYi54pcCaYlkRinne7XYASMrD6UH3w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634417; c=relaxed/simple; bh=La4+9+sPCS/4eXA8ja7XO9KK/olvOoUWDQbSdswhUu0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gxknYi1xUA4tflblvXX1fplrbliXSL3xIX2GFi8Bi7mxZ9ceITZ0uRU3yqAP40oxecxixp9ZrehbEW8JLfOIcKhgPlFSTVbwld9YOa8JjJhgis+3k4a+TftcnoEUY07ZKiofiqczuuOTCY8fpbeJIIE2DHngl8oXgOxtGIVoRtY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=tf0jtpQ3; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="tf0jtpQ3" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-488bd1ee9e7so20148785e9.1 for ; Fri, 01 May 2026 04:20:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634412; x=1778239212; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9kXrhx388xhTRL3MvJdHy6WWnE8EmOsHEooLjon0268=; b=tf0jtpQ38Br8MVdG6EKNIYHLr4d3nznvLq8Oz0C9AmC9cnC4UgVUPJXVY/kkmKmguj 9I1plYKplanL3hxIzzBTMcNTk2SK2dAXSkWStVRtSYi0wzdCqK3sPBTbFcPwZHBIzeTa YS2UqkjSVfAXHeehIJIq0xKa5aLQ6AIWMZzbmx+cWnn+bCGIU+vnDRzfPg5soPU5zIts Dp7aOejhx+nbtBsJWirFE2pA4aEcRRIZ6DrkAwb6e30TZcVhjgvtPngVrlg5DzGYnxlM BKvZ51wvJazy5PiE0LH9WKPuKFvGN41Fc+RdmTQy/IJ7htNkpemwrafMtXiG4ACoh5D2 sWgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634412; x=1778239212; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9kXrhx388xhTRL3MvJdHy6WWnE8EmOsHEooLjon0268=; b=Wm5v23hgSnPnBl0Neo7mfIgE4O907a/wCi7o2Z+pVTFjVA0nHm/o2YbOKShrdYyf7G l7zybF9U41I+u3EeCRyqoYlcWzWB96Nje52KRynVrvR8wNxi9/JCEdn8gZ44xvYOmFK6 Mk9byzktim+jKY/JT+3BKr2yeXH9uVaOIvlkGmhTzD/09hiWyeLo9KKz8V4/pVMMFbQf 1PrT4rmYG1OPXLwY7d44fnvHwhTAv1zLnEOckAWihCNJXsUz0muqT4CE0hrsokXeRKrL oRd5vdJDiIZAdrijbmDMO6VbaLAUFFN6uLo26csIjHUtqakKr0iiR5fONvyhw+MEd6Jn lUQA== X-Forwarded-Encrypted: i=1; AFNElJ+JbF8G0oQdbhdRQUYqI6t5zsDRX0xdIft6ZG64EOrnidCAMeQPYllKHBbjbDTzfnUHHIrQlDKUPDMG4mA=@vger.kernel.org X-Gm-Message-State: AOJu0YwknfrvPE4o5q5Q372Ip/VUJgSBFIdetgajj3KUQi5iekI4aNll M9PKymUfDWU6uAas8P4h9N1HCh2KTJZ295exwu+/gaiZ93Q1JCBKeAFzEFcR7UVjGk2rqwSoCgt oiZVdFcrQxeJWFQ== X-Received: from wmat2.prod.google.com ([2002:a05:600c:6d02:b0:485:33e8:92af]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c108:b0:485:40db:d40c with SMTP id 5b1f17b1804b1-48a8eb61ddfmr29338485e9.3.1777634412089; Fri, 01 May 2026 04:20:12 -0700 (PDT) Date: Fri, 1 May 2026 11:19:12 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-11-smostafa@google.com> Subject: [PATCH v6 10/25] KVM: arm64: iommu: Support DABT for IOMMU From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The pKVM SMMUv3 driver needs to trap and emulate access to the MMIO space of the SMMUv3 to provide emulation for the kernel driver. Add a handler for DABTs for IOMMU drivers to be able to do so. In case the host causes a data abort, check if it's part of IOMMU emulation first. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/include/nvhe/iommu.h | 3 ++- arch/arm64/kvm/hyp/nvhe/iommu/iommu.c | 15 +++++++++++++++ arch/arm64/kvm/hyp/nvhe/mem_protect.c | 15 +++++++++++++++ 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/include/nvhe/iommu.h b/arch/arm64/kvm/hyp/i= nclude/nvhe/iommu.h index eba94b4f6050..e1b6f16391cc 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/iommu.h +++ b/arch/arm64/kvm/hyp/include/nvhe/iommu.h @@ -8,6 +8,7 @@ struct kvm_iommu_ops { int (*init)(void); int (*host_stage2_idmap)(phys_addr_t start, phys_addr_t end, int prot); + bool (*dabt_handler)(struct user_pt_regs *regs, u64 esr, u64 addr); }; =20 int kvm_iommu_init(void *pool_base, unsigned int nr_pages); @@ -18,5 +19,5 @@ int kvm_iommu_host_stage2_idmap(phys_addr_t start, phys_a= ddr_t end, /* Returns zeroed memory. */ void *kvm_iommu_donate_pages(u8 order); void kvm_iommu_reclaim_pages(void *ptr); - +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr); #endif /* __ARM64_KVM_NVHE_IOMMU_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c b/arch/arm64/kvm/hyp/nvh= e/iommu/iommu.c index 53cb5e4b0aac..b1474db016e5 100644 --- a/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c +++ b/arch/arm64/kvm/hyp/nvhe/iommu/iommu.c @@ -4,6 +4,10 @@ * * Copyright (C) 2022 Linaro Ltd. */ +#include + +#include + #include =20 #include @@ -139,3 +143,14 @@ void kvm_iommu_reclaim_pages(void *ptr) { hyp_put_page(&iommu_pages_pool, ptr); } + +bool kvm_iommu_host_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 a= ddr) +{ + if (kvm_iommu_ops && kvm_iommu_ops->dabt_handler && + kvm_iommu_ops->dabt_handler(regs, esr, addr)) { + /* DABT handled by the driver, skip to next instruction. */ + kvm_skip_host_instr(); + return true; + } + return false; +} diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvh= e/mem_protect.c index b54cb72ed88c..5c64007dba4d 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -788,6 +788,12 @@ static void host_inject_mem_abort(struct kvm_cpu_conte= xt *host_ctxt) inject_host_exception(esr); } =20 +static bool is_mmio_dabt(u64 esr) +{ + return (ESR_ELx_EC(esr) =3D=3D ESR_ELx_EC_DABT_LOW) && + (esr & ESR_ELx_ISV); +} + void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt) { struct kvm_vcpu_fault_info fault; @@ -810,6 +816,15 @@ void handle_host_mem_abort(struct kvm_cpu_context *hos= t_ctxt) BUG_ON(!(fault.hpfar_el2 & HPFAR_EL2_NS)); addr =3D FIELD_GET(HPFAR_EL2_FIPA, fault.hpfar_el2) << 12; =20 + /* + * Emulate data aborts for IOMMU drivers, other access will be denied + * by host_stage2_adjust_range() + */ + if (is_mmio_dabt(esr) && !addr_is_memory(addr) && + kvm_iommu_host_dabt_handler(&host_ctxt->regs, + esr, addr | FAR_TO_FIPA_OFFSET(fault.far_el2))) + return; + switch (host_stage2_idmap(addr)) { case -EPERM: host_inject_mem_abort(host_ctxt); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBF7A39E6F5 for ; Fri, 1 May 2026 11:20:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634420; cv=none; b=lE7IiYYnqJeuTzKuXKK1gLFCpxlnVbq1XxuBssqc0y7n4mE0T8Q2sovSzICRCq8snx4iPr9k0FUphmLR1EtX30FVbYdLmCeBKs5oYnKfY593r02e7rjC1RuxUUnmarNSqfj007qU5HSR8i4GusdMuSfeiDdpqjnXIkO+0gF/dkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634420; c=relaxed/simple; bh=OJF405qqToDaQYlnDrTfIPaZ3ERCsQQsEZlpDMFs1BM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hJ+NTk0dpbEZg7/fi+8T8ocdaDlMJ7WqoohQ3PyH0ROx/Z7Od0le+A2ODuKnCKUcS73jhmfqx2R9gD9+QM8Zg5cvDWJvn0ZoQM8EDP0NMjMw2V75yqbKySczlDmkl7aNGPQPT/GMwQbM17GvEahE5XkkAGHQDG3WZAnRmmY6HPA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Om0OX6rd; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Om0OX6rd" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-48a55ecc32cso16294705e9.1 for ; Fri, 01 May 2026 04:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634413; x=1778239213; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TzYlVYNpWMosnl9oou+4ebeG+lgxbNyORXDEfv3bAUM=; b=Om0OX6rd8tMduPgajazaUJq7yB9UKfkE0BIAdv+sNF7AvnXhCg1haWel4OpiTvmZ4M JJwZVEHs8WT3ErVhvutf09kvZFUs40DRGu/8mQcLo+22T2z0TUUsSmP3IvfOFdZPx7mZ UieCVvyddCID8vW7IAtXS3ftKRDKzbe8iVOEj/khK5PCWGoSIHKg3MuNinbTjOpBsOr5 yCu6iRaCPoWIHa/0Y3Eiv1R2VsGvtRh/erQq2JX+Nvq5LwboTEreCwv7gMow5hE1UCAz hUC+IbN5503W4F0YQK/dQ4WFJVUcUUDo731BIcEgBpUpohaINsRPu+A+cVW8wFWp+Riz cBVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634413; x=1778239213; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TzYlVYNpWMosnl9oou+4ebeG+lgxbNyORXDEfv3bAUM=; b=Xct34RKzqXMdAGUwrlcCk+2bJaj/ft7wl/KuMJrAlLo0ytd3cCFgoya3llRMe3P6Pz jEZ5jHMg5wOXW8zq8L3NgUTa/7aBHm7OtJFrwsC+UqwTUEEJ0VnJA6HTXaXWYw9QDP72 MBJ0O7/VyTRiXVZIaYcvK83sOIn4cVdID/Soh0sbEDc0EmPev5NxBj+Zd/KSoB3Ovugu InfuA2sVT2aiVSdVw7R8rcB/rwz0iRU/Lpakhq06v0YLmI9qZDgimbCQWn8nRSUY7Ee+ Jl+ACatjor2c9RKAckQlpoRIAVsbJFprWMeGs5BXyydIWvrr6wNnhMDA5OVWoGJiEvFf tpgA== X-Forwarded-Encrypted: i=1; AFNElJ9tnwXsplKRn8lt8dTIH4h3sD87uWBzUeABd/2XShcgOkEwVxOYvjdHuJq4mdd8IZIkwLEdXy6y8zIXraI=@vger.kernel.org X-Gm-Message-State: AOJu0YzWa7c6nA3QxQm5d7KHvrvkpQSK6oSQQP5i+sYxDk8SX1zAJyz4 MzbZIw5zd/vUQCmHk3dntHU37v7JLhJaxoH//epAdulPmhcJNyc4tx61yaCYnVGWcoZm1yaqTvu PwA487C51SEmohw== X-Received: from wrut11.prod.google.com ([2002:a5d:690b:0:b0:445:168d:28b6]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c10b:b0:488:8840:e5ae with SMTP id 5b1f17b1804b1-48a8445876dmr93780755e9.24.1777634413385; Fri, 01 May 2026 04:20:13 -0700 (PDT) Date: Fri, 1 May 2026 11:19:13 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-12-smostafa@google.com> Subject: [PATCH v6 11/25] iommu/arm-smmu-v3-kvm: Add SMMUv3 driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Jean-Philippe Brucker Add the skeleton for an Arm SMMUv3 driver at EL2. The driver rely on an array of SMMUv3s on the system, where at init it will donate the array and the resources of the SMMUv3s so they can't be changed by the host after de-privilege. This array will be populated in the next patch. Signed-off-by: Jean-Philippe Brucker Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 5 ++ drivers/iommu/arm/Kconfig | 9 ++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 87 +++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 27 ++++++ 4 files changed, 128 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c create mode 100644 drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 606c0e1b7bd0..8a75739db947 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -33,6 +33,11 @@ hyp-obj-$(CONFIG_LIST_HARDENED) +=3D list_debug.o hyp-obj-$(CONFIG_NVHE_EL2_TRACING) +=3D trace.o events.o hyp-obj-y +=3D $(lib-objs) =20 +HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 + +hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o + # Path to simple_ring_buffer.c CFLAGS_trace.nvhe.o +=3D -I$(srctree)/kernel/trace/ =20 diff --git a/drivers/iommu/arm/Kconfig b/drivers/iommu/arm/Kconfig index 5fac08b89dee..916f4723238d 100644 --- a/drivers/iommu/arm/Kconfig +++ b/drivers/iommu/arm/Kconfig @@ -141,3 +141,12 @@ config QCOM_IOMMU select ARM_DMA_USE_IOMMU help Support for IOMMU on certain Qualcomm SoCs. + +config ARM_SMMU_V3_PKVM + bool "ARM SMMUv3 support for protected Virtual Machines" + depends on KVM && ARM64 && ARM_SMMU_V3=3Dy + help + Enable a SMMUv3 driver in the KVM hypervisor, to protect VMs against + memory accesses from devices owned by the host. + + Say Y here if you intend to enable KVM in protected mode. diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c new file mode 100644 index 000000000000..9afc314d0acc --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM hyp driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include + +#include +#include + +#include "arm_smmu_v3.h" + +size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; +struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; + +#define for_each_smmu(smmu) \ + for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ + (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ + (smmu)++) + +/* Put the device in a state that can be probed by the host driver. */ +static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(__pkvm_hyp_donate_host_mmio(smmu->mmio_addr, smmu->mmio_size)); + smmu->base =3D NULL; +} + +static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) +{ + unsigned long haddr; + int ret; + + if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) + return -EINVAL; + + ret =3D __pkvm_host_donate_hyp_mmio(smmu->mmio_addr, smmu->mmio_size, &ha= ddr); + if (ret) + return ret; + + smmu->base =3D (void __iomem *)haddr; + + return 0; +} + +/* Called while is the host is still trusted. */ +static int smmu_init(void) +{ + size_t smmu_arr_size =3D PAGE_ALIGN(sizeof(*kvm_hyp_arm_smmu_v3_smmus) * + kvm_hyp_arm_smmu_v3_count); + struct hyp_arm_smmu_v3_device *smmu; + u64 pfn, nr_pages; + int ret; + + kvm_hyp_arm_smmu_v3_smmus =3D kern_hyp_va(kvm_hyp_arm_smmu_v3_smmus); + pfn =3D hyp_virt_to_pfn(kvm_hyp_arm_smmu_v3_smmus); + nr_pages =3D smmu_arr_size >> PAGE_SHIFT; + + ret =3D __pkvm_host_donate_hyp(pfn, nr_pages); + if (ret) + return ret; + + for_each_smmu(smmu) { + ret =3D smmu_init_device(smmu); + if (ret) + goto out_reclaim_smmu; + } + + return 0; + +out_reclaim_smmu: + while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) + smmu_deinit_device(--smmu); + WARN_ON(__pkvm_hyp_donate_host(pfn, nr_pages)); + return ret; +} + +static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) +{ + return 0; +} + +/* Shared with the kernel driver in EL1 */ +struct kvm_iommu_ops smmu_ops =3D { + .init =3D smmu_init, + .host_stage2_idmap =3D smmu_host_stage2_idmap, +}; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h new file mode 100644 index 000000000000..0d9e48b201f5 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -0,0 +1,27 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_ARM_SMMU_V3_H +#define __KVM_ARM_SMMU_V3_H + +#include + +/* + * Parameters from the trusted host: + * @mmio_addr base address of the SMMU registers + * @mmio_size size of the registers resource + * + * Other members are filled and used at runtime by the SMMU driver. + * @base Virtual address of SMMU registers + */ +struct hyp_arm_smmu_v3_device { + phys_addr_t mmio_addr; + size_t mmio_size; + void __iomem *base; +}; + +extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); +#define kvm_hyp_arm_smmu_v3_count kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count) + +extern struct hyp_arm_smmu_v3_device *kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smm= us); +#define kvm_hyp_arm_smmu_v3_smmus kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_smmus) + +#endif /* __KVM_ARM_SMMU_V3_H */ --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B687439F195 for ; Fri, 1 May 2026 11:20:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634421; cv=none; b=Ny80ZVdbit+UAGZpmRBCyINJTRpdeiAbBejV6FM/Oiw32UQi3JKu/JSZhg8cEQw8t2z32EO9j6ECXFBI8jrDPqh9HCK4UC8TTwNjniXevYX0wPffqLUqDK96rjsyuig0taj8qmRsNKUN/9wWc32dvDnNmsPuQ1oe1N1CKZ3c2t0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634421; c=relaxed/simple; bh=9s2Fj5rL9PAArhF9yi+Z2hB0zXt9fNq+q0TUsuAuCm0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=TReY10/bE98BnbnmlxjNmTvjWksGIy8yzZsQtv33jLRYlu3C1/2NpCDlfgcJAafT7oRiIsnhCOQNzcaCZmPpSrXHXzG8yax6+0Mx/RDyl/ki1N1u1THVfqVIHZC/Z1BbcEA64BoB0QeLEfrQ2gwp/UvXmxS07Sl9HpjIE9ZGk0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=mMnv/tnb; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="mMnv/tnb" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-449c8f24067so878034f8f.1 for ; Fri, 01 May 2026 04:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634415; x=1778239215; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5e9Zs3MwDJ2yjIhn7su11zoqJKlrKCutAbPHqtuih2E=; b=mMnv/tnb+R8V6KXi1EnxWyxWkhgnWP/hFfdn6l0KzQPJ0UJf2vdlRwmYVCe60nDny/ VX/g/xVfZEHPOtw9Vh0uYRu4R5bRe4AhNWI5R2/IFc9DjhFX2cBAk1VFdh0qddstqRrW kxiRK2xICkIzYoDDpOmqKBMOBs+ThJ3K3EyizCwM1MkDxCSwBWy00ate4iQb/rFx9/pf 6CZ6lO+/64zeEadXXg2Agppvl0zdpOqt7q/WsYNSF1jFC1A8LxAy/auvParNlX6N8Btm WmxH4FRj8sSEObHVEmxugZBmFL9kg1/hn+e/9kVnybt1t52oFAOAuNerdLlUiKSKDg1l uGfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634415; x=1778239215; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5e9Zs3MwDJ2yjIhn7su11zoqJKlrKCutAbPHqtuih2E=; b=IL/LXouWhQiZnInehqoiIu46oYx2w/Q29B/tllMWs44XOgIOXrojHP495L3AptfnCB +eAUd5dWmjoYP9bTBtXdlG+UYMS5D/lXZDWltzorIsWgxiWxsfkjVhZtV/SJK+Fmnrz/ 3uo8rq8SqC+BhHD2EZLyL5z022AKTOuFTDSkGmpkTfy2BBrItZamHHNg27t4K0cESQug c4b3z4jY9RJyRAfMxj/RNUInn+vlRuLwjtCEpsOJoNsw07xW+5QwTmwvuoBF4dWG7b0u Slxm/LWVG362k3Jear0AlPm0M53fuN2k8ozNUfYJyJblR50Nna7a3eRIEf8Jdlo+iPwu L8vw== X-Forwarded-Encrypted: i=1; AFNElJ++1agiTBOdrukhF52HDX6iK4Lz9psHVKMk7rDYG3HaeFvrglLeWwQUof3PP/P5Lsdyn8mqCiFC99hZ9RI=@vger.kernel.org X-Gm-Message-State: AOJu0YyWQGLmIK1EYlThznkWmOq77v2EhNt/Ib41Khw7z+NUq1EEtVI8 YXQsFFxiEtSZ2yPGXdKEqasgM68lAOYAXBaJ4PmjmKFEvqytFgtBQKFtCyRmF5o2kN5s8jcIgqx xzp1R1rM6wuy1nw== X-Received: from wrrb6.prod.google.com ([2002:adf:f906:0:b0:43f:e4ba:3a2c]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:e11:b0:449:acdb:3009 with SMTP id ffacd0b85a97d-449acdb3074mr6505269f8f.6.1777634414928; Fri, 01 May 2026 04:20:14 -0700 (PDT) Date: Fri, 1 May 2026 11:19:14 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-13-smostafa@google.com> Subject: [PATCH v6 12/25] iommu/arm-smmu-v3-kvm: Add the kernel driver From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When KVM runs in protected mode, and CONFIG_ARM_SMMU_V3_PKVM is enabled, it will manage the SMMUv3 HW using trap and emulate and present emulated SMMUs to the host kernel. In that case, those SMMUs will be on the aux bus, so make it possible to the driver to probe those devices. Otherwise, everything else is the same as the KVM emulation complies with the architecture,so the driver doesn't need to be modified. Suggested-by: Jason Gunthorpe Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/Makefile | 1 + .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 188 ++++++++++++++++++ drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 43 ++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 2 + 4 files changed, 234 insertions(+) create mode 100644 drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c diff --git a/drivers/iommu/arm/arm-smmu-v3/Makefile b/drivers/iommu/arm/arm= -smmu-v3/Makefile index c9ce392e6d31..c3fc5c4a4a1e 100644 --- a/drivers/iommu/arm/arm-smmu-v3/Makefile +++ b/drivers/iommu/arm/arm-smmu-v3/Makefile @@ -4,5 +4,6 @@ arm_smmu_v3-y :=3D arm-smmu-v3.o arm-smmu-v3-common-lib.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_IOMMUFD) +=3D arm-smmu-v3-iommufd.o arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_SVA) +=3D arm-smmu-v3-sva.o arm_smmu_v3-$(CONFIG_TEGRA241_CMDQV) +=3D tegra241-cmdqv.o +arm_smmu_v3-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D arm-smmu-v3-kvm.o =20 obj-$(CONFIG_ARM_SMMU_V3_KUNIT_TEST) +=3D arm-smmu-v3-test.o diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c new file mode 100644 index 000000000000..9765d3d636d7 --- /dev/null +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * pKVM host driver for the Arm SMMUv3 + * + * Copyright (C) 2022 Linaro Ltd. + */ +#include +#include + +#include +#include +#include +#include + +#include "arm-smmu-v3.h" +#include "pkvm/arm_smmu_v3.h" + +extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); + +static size_t kvm_arm_smmu_count; +static struct hyp_arm_smmu_v3_device *kvm_arm_smmu_array; +static size_t kvm_arm_smmu_cur; + +static void kvm_arm_smmu_array_free(void) +{ + int order; + + order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); + free_pages((unsigned long)kvm_arm_smmu_array, order); +} + +static int kvm_arm_smmu_array_alloc(void) +{ + int smmu_order; + struct device_node *np; + + for_each_compatible_node(np, NULL, "arm,smmu-v3") + kvm_arm_smmu_count++; + + if (!kvm_arm_smmu_count) + return -ENODEV; + smmu_order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)= ); + kvm_arm_smmu_array =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, = smmu_order); + if (!kvm_arm_smmu_array) + return -ENOMEM; + return 0; +} + +static unsigned int smmu_hyp_pgt_pages(void) +{ + struct device_node *np =3D of_find_compatible_node(NULL, NULL, "arm,smmu-= v3"); + + /* + * SMMUv3 uses the same format as the CPU stage-2 and hence have the same= memory + * requirements, we add extra 500 pages for L2 STEs. + * Only one set of memory is allocated as the page table is shared betwee= n all + * the SMMUs. + */ + if (np) { + of_node_put(np); + return host_s2_pgtable_pages() + 500; + } + + return 0; +} + +static struct platform_driver smmuv3_nesting_driver; +static int smmuv3_nesting_probe(struct platform_device *pdev) +{ + struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; + struct device *dev =3D &pdev->dev; + struct resource *res; + + /* Only device tree, ACPI not supported. */ + if (!dev->of_node) + return -EINVAL; + + if (kvm_arm_smmu_cur >=3D kvm_arm_smmu_count) + return -ENOSPC; + + res =3D platform_get_resource(pdev, IORESOURCE_MEM, 0); + if (!res) + return -ENODEV; + + if (of_property_read_bool(dev->of_node, "cavium,cn9900-broken-page1-regsp= ace")) + return -EINVAL; + + smmu->mmio_addr =3D res->start; + smmu->mmio_size =3D resource_size(res); + if (smmu->mmio_size < SZ_128K) { + dev_err(dev, "MMIO region too small(%pr)\n", res); + return -EINVAL; + } + + if (of_dma_is_coherent(dev->of_node)) + smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; + + kvm_arm_smmu_cur++; + return 0; +} + +static int kvm_arm_smmu_v3_register(void) +{ + size_t nr_pages =3D smmu_hyp_pgt_pages(); + int ret; + + if (!is_protected_kvm_enabled() || !nr_pages) + return 0; + + ret =3D kvm_arm_smmu_array_alloc(); + if (ret) + goto out_err; + + ret =3D platform_driver_probe(&smmuv3_nesting_driver, smmuv3_nesting_prob= e); + if (ret) + goto out_free; + + ret =3D kvm_iommu_register_driver(kern_hyp_va(lm_alias(&kvm_nvhe_sym(smmu= _ops))), + nr_pages); + if (ret) + goto out_unregister; + + /* + * These variables are stored in the nVHE image, and won't be accessible + * after KVM initialization. Ownership of kvm_arm_smmu_array will be + * transferred to the hypervisor as well. + */ + kvm_hyp_arm_smmu_v3_smmus =3D kvm_arm_smmu_array; + kvm_hyp_arm_smmu_v3_count =3D kvm_arm_smmu_cur; + return ret; + +out_unregister: + platform_driver_unregister(&smmuv3_nesting_driver); +out_free: + kvm_arm_smmu_array_free(); +out_err: + kvm_arm_smmu_count =3D 0; + kvm_arm_smmu_array =3D NULL; + return ret; +}; + +static int smmu_create_aux_device(struct device *dev, void *data) +{ + static int dev_id; + struct auxiliary_device *auxdev; + + auxdev =3D __devm_auxiliary_device_create(dev, "protected_kvm", + "smmu_v3_emu", NULL, dev_id++); + if (!auxdev) + return -ENODEV; + + auxdev->dev.parent =3D dev; + return 0; +} + +static int kvm_arm_smmu_v3_post_init(void) +{ + if (!kvm_arm_smmu_count) + return 0; + + /* + * If the hypervisor part of the driver fails, KVM will not initialise. + */ + if (!is_kvm_arm_initialised()) { + kvm_arm_smmu_array_free(); + return 0; + } + + WARN_ON(driver_for_each_device(&smmuv3_nesting_driver.driver, NULL, + NULL, smmu_create_aux_device)); + + return 0; +} + +static const struct of_device_id smmuv3_nested_of_match[] =3D { + { .compatible =3D "arm,smmu-v3", }, + { }, +}; + +static struct platform_driver smmuv3_nesting_driver =3D { + .driver =3D { + .name =3D "smmuv3-nesting", + .of_match_table =3D smmuv3_nested_of_match, + .suppress_bind_attrs =3D true, + }, +}; +late_initcall(kvm_arm_smmu_v3_post_init); +subsys_initcall(kvm_arm_smmu_v3_register); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 96d5e7f80ce7..61e6ab364086 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -11,6 +11,7 @@ =20 #include #include +#include #include #include #include @@ -5335,6 +5336,48 @@ static struct platform_driver arm_smmu_driver =3D { module_driver(arm_smmu_driver, platform_driver_register, arm_smmu_driver_unregister); =20 +#ifdef CONFIG_ARM_SMMU_V3_PKVM +/* + * Now we have 2 devices, the aux device bound to this driver, and pdev + * which is the physical platform device bound to the KVM driver but not u= sed. + * However, this driver keeps using the platform device for 2 reasons: + * 1) Simplicity: Avoiding changing big parts of the code assuming + * the underlying device is a platform device. + * 2) Dealing with DMA-API, irqs(MSIs), RPM... requires the physical devic= e. + */ + +static int arm_smmu_device_probe_emu(struct auxiliary_device *auxdev, + const struct auxiliary_device_id *id) +{ + struct device *parent =3D auxdev->dev.parent; + + dev_info(&auxdev->dev, "Probing from %s\n", dev_name(parent)); + return arm_smmu_device_probe(to_platform_device(parent)); +} + +static void arm_smmu_device_remove_emu(struct auxiliary_device *auxdev) +{ + arm_smmu_device_remove(to_platform_device(auxdev->dev.parent)); +} + +const struct auxiliary_device_id arm_smmu_aux_table[] =3D { + { .name =3D "protected_kvm.smmu_v3_emu" }, + { }, +}; + +struct auxiliary_driver arm_smmu_driver_emu =3D { + .driver =3D { + .suppress_bind_attrs =3D true, + }, + .name =3D "arm-smmu-v3-emu", + .id_table =3D arm_smmu_aux_table, + .probe =3D arm_smmu_device_probe_emu, + .remove =3D arm_smmu_device_remove_emu, +}; + +module_auxiliary_driver(arm_smmu_driver_emu); +#endif + MODULE_DESCRIPTION("IOMMU API for ARM architected SMMUv3 implementations"); MODULE_AUTHOR("Will Deacon "); MODULE_ALIAS("platform:arm-smmu-v3"); diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 0d9e48b201f5..744ee2b7f0b4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -8,6 +8,7 @@ * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers * @mmio_size size of the registers resource + * @features Features of SMMUv3, subset of the main driver * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers @@ -16,6 +17,7 @@ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; size_t mmio_size; void __iomem *base; + u32 features; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F14739EF21 for ; Fri, 1 May 2026 11:20:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634421; cv=none; b=LegjiM4bjiT9jjxAtXMalhVolRJYF/J0cH62D0Cq9EXeNvbdp9wrKXUIVrhnEOL/dtmP5WNiKE/H2t5wKLo6z6BFLzoz6AYzQsHPD6ZpkJ2ahgj9GBSHTYkQKwF4QNEMAy6wr2MT3q2JGnN5vyjxyJJrNmALFxLE2N6R1+hn2MU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634421; c=relaxed/simple; bh=dtlBVLOo14YOqCmNNvP1AMUdNnEW7NiQxpiFymghyvU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IjanYYRxTeXBB1CpwP5jAKUXByOFJ1GqgDsFGKyGqWIr3u3PfzlBRJpCbjUVulTcFCZ5YPldGsjtICSmGRQOu88Ccmhln1lkIg6J5cXpb2KZDxnN6hsieheqJ8AaGB60ncmRNLbBeSh7X2+qT9/4daWNjyg88nG0tK6IGNme0YY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=oU0do3WC; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oU0do3WC" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-488bd1ee9e7so20149655e9.1 for ; Fri, 01 May 2026 04:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634417; x=1778239217; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UU83Oeb4MDmTny3jalsI86VqzD8OKMnbSWGrcxMKTpA=; b=oU0do3WCj3zTDey3L+Eyx7w14nRqYsMjx3Li9lwzrnf62No5YbSQ0Lihu9C6Rg+wDa lUR5BHSqHRjtEdZaQDw+pjXOIBsGauxZJ/RmGrE1Sl5tKutJ82akfFd7vgVRMKzOIm23 1aLxaOLA2PjLCIlz9Gg98MEIXG3gfj0xh1y+2lL8PkzKN+oYs1Cd6BQFEkLyY1PxaeV0 /ff22Zhp9MpJpYKjVKbvuGff0J3CxmnHeUm+StIRZKsStp/eI7RbqZ3ol15jbavCTvkp /ujfIwvd2MH6J+YrOYwcBnpgzBGKTqW4jELPe4av6DX5b1NjadptrLtvPDJkB4nW3Tzc TJeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634417; x=1778239217; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UU83Oeb4MDmTny3jalsI86VqzD8OKMnbSWGrcxMKTpA=; b=ZFwu528/TD9QZ5UbcsESwqW9tslxM/AZTpoclcW0mkVdC06KTEFeIq/bifqS4cU0j0 odaRHGUIfNg84+W9mQXSl8LI4FGwZFHKEHBXMlfqdiPp9EW6KLdMhIqmHaEveeGmWNDx fw7rKe7Xx28ICga1zo5IYl9Y/ydn/KHUWnydZvrXWwydVQJXeiXXFAKeZH52GQydLoFn UDKwHLQZkQHxSma3qxWw8cy9UtuYIOCn0xqvT+6pf4wIqlD2nWHRN5x7f+KeSh+FAsAy Fs+QSNimoFsQaR1qZNKCVRpJUD8JaxyvE4ZmeWt4dioCoYnnK1qj6xGnRFZ7zukcCeTZ ANXg== X-Forwarded-Encrypted: i=1; AFNElJ8EmHxOkHf0bZo4Ha8VNs4WZM5J8c1XwaSGYLLqNCO8O7dvOWrG9MgBXVUsVIw1oECMecyoqIVweSYyi9g=@vger.kernel.org X-Gm-Message-State: AOJu0Yysnadbi8hIGd2DbPSHdT1zI7f4+IyzgjKMhBsLhiZLxbKgPQ2q uruBP4Q4yJCQW6baEf8viarufQ389uAwh7/Aio6YjMR7Oqi+EuN6RxWgjqLIqnQrkAHzAPh6EnX BECcF34r/nBFfDw== X-Received: from wrtx8.prod.google.com ([2002:a5d:60c8:0:b0:43d:7331:91da]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6995:b0:483:709e:f238 with SMTP id 5b1f17b1804b1-48a8ebab77emr40681985e9.29.1777634416769; Fri, 01 May 2026 04:20:16 -0700 (PDT) Date: Fri, 1 May 2026 11:19:15 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-14-smostafa@google.com> Subject: [PATCH v6 13/25] iommu/arm-smmu-v3-kvm: Probe SMMU HW From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Probe SMMU features from the IDR register space, most of the logic is common with the kernel. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 6 -- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 6 ++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 78 +++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 6 ++ 4 files changed, 90 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.c index 61e6ab364086..157acde0436d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -4738,12 +4738,6 @@ static int arm_smmu_device_reset(struct arm_smmu_dev= ice *smmu) return 0; } =20 -#define IIDR_IMPLEMENTER_ARM 0x43b -#define IIDR_PRODUCTID_ARM_MMU_600 0x483 -#define IIDR_PRODUCTID_ARM_MMU_700 0x487 -#define IIDR_PRODUCTID_ARM_MMU_L1 0x48a -#define IIDR_PRODUCTID_ARM_MMU_S3 0x498 - static void arm_smmu_device_iidr_probe(struct arm_smmu_device *smmu) { u32 reg; diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index 64618299d03a..f904f4d19609 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -84,6 +84,12 @@ struct arm_vsmmu; #define IIDR_REVISION GENMASK(15, 12) #define IIDR_IMPLEMENTER GENMASK(11, 0) =20 +#define IIDR_IMPLEMENTER_ARM 0x43b +#define IIDR_PRODUCTID_ARM_MMU_600 0x483 +#define IIDR_PRODUCTID_ARM_MMU_700 0x487 +#define IIDR_PRODUCTID_ARM_MMU_L1 0x48a +#define IIDR_PRODUCTID_ARM_MMU_S3 0x498 + #define ARM_SMMU_AIDR 0x1C =20 #define ARM_SMMU_CR0 0x20 diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 9afc314d0acc..d9945db9e102 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -10,6 +10,7 @@ #include =20 #include "arm_smmu_v3.h" +#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -26,6 +27,77 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_de= vice *smmu) smmu->base =3D NULL; } =20 +static bool smmu_nesting_supported(struct hyp_arm_smmu_v3_device *smmu) +{ + unsigned int implementer, productid, variant, revision; + u32 reg; + + if (!(smmu->features & ARM_SMMU_FEAT_TRANS_S1) || + !(smmu->features & ARM_SMMU_FEAT_TRANS_S2)) + return false; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IIDR); + implementer =3D FIELD_GET(IIDR_IMPLEMENTER, reg); + productid =3D FIELD_GET(IIDR_PRODUCTID, reg); + variant =3D FIELD_GET(IIDR_VARIANT, reg); + revision =3D FIELD_GET(IIDR_REVISION, reg); + + if (implementer !=3D IIDR_IMPLEMENTER_ARM) + return true; + + if (productid =3D=3D IIDR_PRODUCTID_ARM_MMU_600) + return variant >=3D 2; + else if (productid =3D=3D IIDR_PRODUCTID_ARM_MMU_700) + return !(variant < 1 || revision < 1); + + return true; +} + +/* + * Mini-probe and validation for the hypervisor. + */ +static int smmu_probe(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 reg; + + /* Similar to the kernel, rely on firmware override. */ + if (!(smmu->features & ARM_SMMU_FEAT_COHERENCY)) + return -EINVAL; + + /* IDR0 */ + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR0); + + smmu->features |=3D smmu_idr0_features(reg); + if (!smmu_nesting_supported(smmu)) + return -ENXIO; + + if (!(smmu->features & (ARM_SMMU_FEAT_TT_LE | ARM_SMMU_FEAT_TT_BE))) + return -ENXIO; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR1); + if (reg & (IDR1_TABLES_PRESET | IDR1_QUEUES_PRESET | IDR1_REL)) + return -EINVAL; + + smmu->sid_bits =3D FIELD_GET(IDR1_SIDSIZE, reg); + /* Follows the kernel logic */ + if (smmu->sid_bits <=3D STRTAB_SPLIT) + smmu->features &=3D ~ARM_SMMU_FEAT_2_LVL_STRTAB; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR3); + smmu->features |=3D smmu_idr3_features(reg); + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_IDR5); + smmu->pgsize_bitmap =3D smmu_idr5_to_pgsize(reg); + + smmu->oas =3D smmu_idr5_to_oas(reg); + if (smmu->oas =3D=3D 52) + smmu->pgsize_bitmap |=3D 1ULL << 42; + else if (!smmu->oas) + smmu->oas =3D 48; + + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -39,8 +111,14 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) return ret; =20 smmu->base =3D (void __iomem *)haddr; + ret =3D smmu_probe(smmu); + if (ret) + goto out_ret; =20 return 0; +out_ret: + smmu_deinit_device(smmu); + return ret; } =20 /* Called while is the host is still trusted. */ diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 744ee2b7f0b4..82b84673e85b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -12,12 +12,18 @@ * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers + * @oas PA size + * @pgsize_bitmap Supported page sizes + * @sid_bits Max number of SID bits supported */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; size_t mmio_size; void __iomem *base; u32 features; + unsigned long oas; + unsigned long pgsize_bitmap; + unsigned int sid_bits; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 910DC39FCB3 for ; Fri, 1 May 2026 11:20:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634423; cv=none; b=kqRVJC+6ZT8Oz65Sxqk+GxRX2Ojj9W8sZ+J1vgjqN53Jkbnt8uBCSD3reRVnbjQUOhpxxW9Wvkfvp+Nz7mkSS+T6PNGaKhB10Q/6atdbpNMmEZhKwt1+0HnDl83ZON2rSgoNVYNgkW6HtPsr6V9opvTmcRGfxUSdCpNtCblrZWc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634423; c=relaxed/simple; bh=o14JzmXYNahFrdFmiylYn3Zxk6zxXvzNvdkjVQ1daIo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H545lCN56aV5VXCdNttN6Mi6LLCBVNSk2ybhIm6/FTAhYL/G6laos3Qj0zzZVRNpsw3GICYjeK9rteUmpnARJaBFF0aOIrQ9T1cvPs0h6ZHHC+VfZ6EvGDCRAmNBM5t7jKg9B5NG+kJyAJSF3s2z705NXFdq+QJJabvBZB9n2oM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=G0cTfDgZ; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="G0cTfDgZ" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-b8704795d25so137295966b.2 for ; Fri, 01 May 2026 04:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634418; x=1778239218; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TWp0aeThUsgvfFBxJWuWRQdGQ3dxhvfuaZARmJjLR/k=; b=G0cTfDgZ+lBJXxRJIkThyAHGAQ5PJebTy+MWuq5c3nbQ9R31QA5ClcCAyAHRmW5TPc MnhQf9xp3OG6Av7UVhvvbWA8QuiRDx1aF4ByzdpvlkOkgoasjHINdERoYos4xPx/uGBD v6OB2Y3TMxaa7/y5w8ncTWFn2zXHJwxlFulelxaAYhhhKc3kCgS0V1GQV7T+42zCza2H m+PrFf7mmQWXhJiz6/xspT2ylUbf6jPsMcKh8s7mG17qRRgk5wdViDeuUXzxQgBAJkiw 58hGuiSEg657zuywXYsL5K2GsqIO/YvYzRp/4scUPcnIiY7slFYQNu1sXn15BENXjO0b 6lGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634418; x=1778239218; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TWp0aeThUsgvfFBxJWuWRQdGQ3dxhvfuaZARmJjLR/k=; b=dfRksC9VpLAFRtV+LwuxWcmoJ+pDK6Kfob6+IkrdNaaSl5SX/31nbI+0m+225ii/TU TiB7SMhW7jDRlfXQVDWJYXtQO4/yu1Gjc7B6cesfMn2C/7aztHngHj28JZYKKb5dGcCC D94MBr/9CbZaLn3R50hE2EFkXk9lTEX+4NMtMGWkc8qS+bLA2ZBsdTeqwR9n7icd19x3 lCdnSPFIKRBF1Qy9Ak8nI1/0ihJfn81TJtUkPGBbmx05CpQx/8x5cqDnCWYJjzs4w40g 1gkZk/Fs7WfIIU5+aI/mVKh1aXVW0et0uExPYukwwIuZJ46BZQiV/aAyEoE906w5kkT9 7elg== X-Forwarded-Encrypted: i=1; AFNElJ8Lya8AXvDMB9oe5/8G2GStsh/gcuN+UNz99SOetYflCkNte42uW9mJqDK2IWxnTAB3bpeAbNcxK1DLaKk=@vger.kernel.org X-Gm-Message-State: AOJu0YyU2Y6lg01YBFmyIDDTkpLBcxE96/9eFOhoslCrVNy07G0N54li DT2ENgpU3THijwa1hL+94+6QbwY2GCtvVKryFNCbTMKQNeGpPc39GfPyTJYTXVnCwT0MzHwgBgC jmlLv3iRKlcF6jQ== X-Received: from ejcjs18.prod.google.com ([2002:a17:907:97d2:b0:b97:b7aa:b4]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:72d1:b0:ba5:2f0a:2f40 with SMTP id a640c23a62f3a-bbac4bd9fa1mr416440366b.4.1777634417769; Fri, 01 May 2026 04:20:17 -0700 (PDT) Date: Fri, 1 May 2026 11:19:16 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-15-smostafa@google.com> Subject: [PATCH v6 14/25] iommu/arm-smmu-v3-kvm: Add MMIO emulation From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add data abort handler for the SMMUs, at the moment most registers are just passthrough. In the next patches CMDQ/STE emulation will be added which inserts logic to some register access. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 143 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 2 files changed, 153 insertions(+) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index d9945db9e102..cce5a51b4656 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -8,6 +8,7 @@ =20 #include #include +#include =20 #include "arm_smmu_v3.h" #include "../arm-smmu-v3.h" @@ -106,6 +107,7 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_devi= ce *smmu) if (!PAGE_ALIGNED(smmu->mmio_addr | smmu->mmio_size)) return -EINVAL; =20 + hyp_spin_lock_init(&smmu->lock); ret =3D __pkvm_host_donate_hyp_mmio(smmu->mmio_addr, smmu->mmio_size, &ha= ddr); if (ret) return ret; @@ -144,6 +146,8 @@ static int smmu_init(void) goto out_reclaim_smmu; } =20 + BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); + return 0; =20 out_reclaim_smmu: @@ -153,6 +157,144 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, + struct user_pt_regs *regs, + u64 esr, u32 off) +{ + bool is_write =3D esr & ESR_ELx_WNR; + unsigned int len =3D BIT((esr & ESR_ELx_SAS) >> ESR_ELx_SAS_SHIFT); + int rd =3D (esr & ESR_ELx_SRT_MASK) >> ESR_ELx_SRT_SHIFT; + const u64 read_write =3D -1ULL; + const u64 no_access =3D 0; + u64 mask =3D no_access; + const u64 read_only =3D is_write ? no_access : read_write; + bool is_xzr =3D (rd =3D=3D 31); + u64 val =3D is_xzr ? 0 : regs->regs[rd]; + + switch (off) { + case ARM_SMMU_IDR0: + if (len !=3D sizeof(u32)) + break; + /* Clear stage-2 support, hide MSI to avoid write back to cmdq */ + mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); + break; + /* Passthrough the register access for bisectiblity, handled later */ + case ARM_SMMU_CMDQ_BASE: + case ARM_SMMU_CMDQ_PROD: + case ARM_SMMU_CMDQ_CONS: + case ARM_SMMU_STRTAB_BASE: + case ARM_SMMU_STRTAB_BASE_CFG: + case ARM_SMMU_GBPA: + mask =3D read_write; + break; + case ARM_SMMU_CR0: + if (len !=3D sizeof(u32)) + break; + mask =3D read_write; + break; + case ARM_SMMU_CR1: { + /* Based on Linux implementation */ + u64 cr1_template =3D FIELD_PREP(CR1_TABLE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_TABLE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_TABLE_IC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_SH, ARM_SMMU_SH_ISH) | + FIELD_PREP(CR1_QUEUE_OC, CR1_CACHE_WB) | + FIELD_PREP(CR1_QUEUE_IC, CR1_CACHE_WB); + if (len !=3D sizeof(u32)) + break; + /* Don't mess with shareability/cacheability. */ + if (is_write) { + WARN_ON(val !=3D cr1_template); + val =3D cr1_template; + } + mask =3D read_write; + break; + } + + /* Allowed 32 bit registers. */ + case ARM_SMMU_EVTQ_PROD + SZ_64K: + case ARM_SMMU_EVTQ_CONS + SZ_64K: + case ARM_SMMU_EVTQ_IRQ_CFG1: + case ARM_SMMU_EVTQ_IRQ_CFG2: + case ARM_SMMU_PRIQ_PROD + SZ_64K: + case ARM_SMMU_PRIQ_CONS + SZ_64K: + case ARM_SMMU_PRIQ_IRQ_CFG1: + case ARM_SMMU_PRIQ_IRQ_CFG2: + case ARM_SMMU_GERRORN: + case ARM_SMMU_GERROR_IRQ_CFG1: + case ARM_SMMU_GERROR_IRQ_CFG2: + case ARM_SMMU_IRQ_CTRLACK: + case ARM_SMMU_IRQ_CTRL: + case ARM_SMMU_CR0ACK: + case ARM_SMMU_CR2: + if (len !=3D sizeof(u32)) + break; + mask =3D read_write; + break; + /* Allowed 64 bit registers. */ + case ARM_SMMU_EVTQ_BASE: + case ARM_SMMU_EVTQ_IRQ_CFG0: + case ARM_SMMU_PRIQ_BASE: + case ARM_SMMU_PRIQ_IRQ_CFG0: + case ARM_SMMU_GERROR_IRQ_CFG0: + if (len !=3D sizeof(u64)) + break; + mask =3D read_write; + break; + /* Allowed RO 32 bit registers. */ + case ARM_SMMU_IIDR: + case ARM_SMMU_IDR5: + case ARM_SMMU_IDR3: + case ARM_SMMU_IDR1: + case ARM_SMMU_GERROR: + if (len !=3D sizeof(u32)) + break; + mask =3D read_only; + }; + + if (WARN_ON(!mask)) + goto out_ret; + + if (is_write) { + if (len =3D=3D sizeof(u64)) + writeq_relaxed(val & mask, smmu->base + off); + else + writel_relaxed(val & mask, smmu->base + off); + + return true; + } + + if (len =3D=3D sizeof(u64)) + val =3D readq_relaxed(smmu->base + off) & mask; + else + val =3D readl_relaxed(smmu->base + off) & mask; + /* + * Device might be read senstive, so do it but ignore writing + * back for xzr. + */ + if (!is_xzr) + regs->regs[rd] =3D val; + +out_ret: + return true; +} + +static bool smmu_dabt_handler(struct user_pt_regs *regs, u64 esr, u64 addr) +{ + struct hyp_arm_smmu_v3_device *smmu; + bool ret; + + for_each_smmu(smmu) { + if (addr < smmu->mmio_addr || addr >=3D smmu->mmio_addr + smmu->mmio_siz= e) + continue; + hyp_spin_lock(&smmu->lock); + ret =3D smmu_dabt_device(smmu, regs, esr, addr - smmu->mmio_addr); + hyp_spin_unlock(&smmu->lock); + return ret; + } + return false; +} + static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) { return 0; @@ -162,4 +304,5 @@ static int smmu_host_stage2_idmap(phys_addr_t start, ph= ys_addr_t end, int prot) struct kvm_iommu_ops smmu_ops =3D { .init =3D smmu_init, .host_stage2_idmap =3D smmu_host_stage2_idmap, + .dabt_handler =3D smmu_dabt_handler, }; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 82b84673e85b..263b0fef262d 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -4,6 +4,10 @@ =20 #include =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +#include +#endif + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -15,6 +19,7 @@ * @oas PA size * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported + * @lock Lock to protect SMMU */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -24,6 +29,11 @@ struct hyp_arm_smmu_v3_device { unsigned long oas; unsigned long pgsize_bitmap; unsigned int sid_bits; +#ifdef __KVM_NVHE_HYPERVISOR__ + hyp_spinlock_t lock; +#else + u32 lock; +#endif }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 614B53A0B28 for ; Fri, 1 May 2026 11:20:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634424; cv=none; b=oEfI2zGwcWeaE4OgAQeGG0FDM2sZKfriR8Kvyh9+slN1l7h+UHw/szmbLkgj8bN0RVD9nmNZdjfVKL4uNYoow4KCKr1mV0xMHV+Y3960NZc3SHEtha+PwWcVOqlNmubr329MIIncGTup8Flx4QDl5Ek0DcbJ1kkPOyfDgYbD3qc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634424; c=relaxed/simple; bh=+Qu6Xr8SZccDkGbo59oLqz3v9y141GWTZZp2UsNaiPg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HO1GMxV0/uoimZt+SyVTVH9WYvehNm7oCJXRdalnZ8XvScZXRRHUkaHQPvf8ROhxcMmiHftWlcV4kP9q6yoS60ppWPF9QKN2rktfeuYWn2WX7bnkjWJfRSdB8jpbfychSUEydiECXY2/FIzGyDDyDNtueWWJs6iLdd6djTCHRYI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RboRtzI1; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RboRtzI1" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-4411a1f9601so1370781f8f.0 for ; Fri, 01 May 2026 04:20:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634421; x=1778239221; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=aHDjZzOY7Hx/XA6TloE0Km6NFJhjbYET1ob/3IzkCuE=; b=RboRtzI1oOxws65Oyk2mdbbvhF62ua8bcG3M63Xa2LJXt/8DrRt7R0p/yA/rMUHncf xqotfbafeA6IjdZGBLeuOTrPHn9yGf4g8DXu42HNd4gA2obtjyPSlNQBFHvm6TTZzjJk EdjKP5QeSznJ+tFY/gqxtBSXb7+DTB3dghseq613tH29ee6flSXASE2O1OIuZ3Ddnu5z ue7t+fKcX7jn3MAAAaIZ6QWgPRGsKFQICUuiCnnf5BT79wPjA6fNFl40aqT+n1ltLwXC 6/zNdy3c5fTQvQ9cZt4FEfvYeW4TUS4JdK+TmQNzhRN2DmOmARhsQULuNwoNFsF7milA wnGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634421; x=1778239221; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=aHDjZzOY7Hx/XA6TloE0Km6NFJhjbYET1ob/3IzkCuE=; b=JLr6bxpftOlMe4Wj2KTKNJiDCBPVHW0yGUkSdRERx5M4dbDokB9pZIvypP2RkJp+k4 rG++EH1Pjqzf9BayPEPtmCrHMHIWIZeoDa6FfCDsKPKLiCBddiwuC2YjLV+hlNQPKWfN gF/l7pNEfdBjBOVSAXEvQBxrUZswcIE3HgGuThx5a8xB5GgFgR9yZLzRRLkfE6FrkZ7E NzaGO6cGAN5RKTHWQyKkDY2FwM2wts//ZX5nHsz4V7xrjL9CUJBAeC6aT/0MrFIe7w6W z9G3ogJbFlcqvGnMPYcrpolf5+J9bAFKIMyzWqByIIghhm6LwigsLn7gTIezLvueHBdB FGJw== X-Forwarded-Encrypted: i=1; AFNElJ8yEk5FTxtBkxcxF/2bUSezEED8vBm8XLxZEw5+P218A/UODMeYhdmwzZ6CCiZcHrMu4RpZdn2bwXgWaqU=@vger.kernel.org X-Gm-Message-State: AOJu0YyEJRksXO6sCgPjuTaIXegaTLtZSmh4Lvf377Tnw5JpLot+uRna VGNs6Fuc7OuwFcxqCCG78jDAFk75Of5/JSsNmdBabl8956Lvn4ndmhPGRilEaBWwRWTSmGAydsI OlOK/P+SUqXKg/g== X-Received: from wrxl7.prod.google.com ([2002:a05:6000:12c7:b0:43b:4453:39d8]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2212:b0:43d:3004:5fef with SMTP id ffacd0b85a97d-4493d7fa8ccmr10517288f8f.7.1777634420432; Fri, 01 May 2026 04:20:20 -0700 (PDT) Date: Fri, 1 May 2026 11:19:17 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-16-smostafa@google.com> Subject: [PATCH v6 15/25] iommu/arm-smmu-v3-kvm: Shadow the command queue From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" At boot allocate a command queue per SMMU which is used as a shadow by the hypervisor. The command queue size is 64K which is more than enough, as the hypervisor would consume all the entries per a command queue prod write, which means it can handle up to 4096 at a time. Then, the host command queue needs to be pinned in a shared state, so it can't be donated to VMs, and avoid tricking the hypervisor into accessing them. This is done each time the command queue is enabled, and undone each time the command queue is disabled. The hypervisor won=E2=80=99t access the host command queue when it is disab= led from the host. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 25 ++++ .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 122 +++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 8 ++ 3 files changed, 154 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index 9765d3d636d7..fccbc34de087 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -15,6 +15,8 @@ #include "arm-smmu-v3.h" #include "pkvm/arm_smmu_v3.h" =20 +#define SMMU_KVM_CMDQ_ORDER 4 + extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 static size_t kvm_arm_smmu_count; @@ -24,6 +26,15 @@ static size_t kvm_arm_smmu_cur; static void kvm_arm_smmu_array_free(void) { int order; + int i; + + for (i =3D 0 ; i < kvm_arm_smmu_cur ; ++i) { + struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[i]; + + if (smmu->cmdq.base_dma) + free_pages((unsigned long)phys_to_virt(smmu->cmdq.base_dma), + SMMU_KVM_CMDQ_ORDER); + } =20 order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); free_pages((unsigned long)kvm_arm_smmu_array, order); @@ -70,6 +81,7 @@ static int smmuv3_nesting_probe(struct platform_device *p= dev) struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; struct device *dev =3D &pdev->dev; struct resource *res; + void *cmdq_base; =20 /* Only device tree, ACPI not supported. */ if (!dev->of_node) @@ -95,6 +107,19 @@ static int smmuv3_nesting_probe(struct platform_device = *pdev) if (of_dma_is_coherent(dev->of_node)) smmu->features |=3D ARM_SMMU_FEAT_COHERENCY; =20 + /* + * Allocate the shadow command queue, it doesn't have to be the same + * size as the host. + * Only populate base_dma and llq.max_n_shift, the hypervisor will init + * the rest. + */ + cmdq_base =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_= CMDQ_ORDER); + if (!cmdq_base) + return -ENOMEM; + + smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); + smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; + kvm_arm_smmu_cur++; return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index cce5a51b4656..3b77796dafc7 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -11,7 +11,6 @@ #include =20 #include "arm_smmu_v3.h" -#include "../arm-smmu-v3.h" =20 size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; @@ -21,10 +20,68 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmu= s; (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ (smmu)++) =20 +#define cmdq_size(cmdq) ((1 << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWORD= S * 8) + +static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_CMDQEN, smmu->cr0); +} + +/* + * CMDQ, STE host copies are accessed by the hypervisor, we share them to + * - Prevent the host from passing protected VM memory. + * - Having them mapped in the hyp page table. + */ +static int smmu_share_pages(phys_addr_t addr, size_t size) +{ + size_t nr_pages =3D PAGE_ALIGN(size + (addr & ~PAGE_MASK)) >> PAGE_SHIFT; + phys_addr_t base =3D addr & PAGE_MASK; + int i, ret; + + for (i =3D 0 ; i < nr_pages ; ++i) { + if (__pkvm_host_share_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT)) { + while (i--) + __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + return -EPERM; + } + } + + ret =3D hyp_pin_shared_mem(hyp_phys_to_virt(base), + hyp_phys_to_virt(base + nr_pages * PAGE_SIZE)); + if (ret) { + for (i =3D 0 ; i < nr_pages ; ++i) + __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + } + + return ret; +} + +static int smmu_unshare_pages(phys_addr_t addr, size_t size) +{ + size_t nr_pages =3D PAGE_ALIGN(size + (addr & ~PAGE_MASK)) >> PAGE_SHIFT; + phys_addr_t base =3D addr & PAGE_MASK; + int i, ret; + + hyp_unpin_shared_mem(hyp_phys_to_virt(base), + hyp_phys_to_virt(base + nr_pages * PAGE_SIZE)); + + for (i =3D 0 ; i < nr_pages ; ++i) { + ret =3D __pkvm_host_unshare_hyp((base + i * PAGE_SIZE) >> PAGE_SHIFT); + if (ret) + return ret; + } + + return 0; +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { WARN_ON(__pkvm_hyp_donate_host_mmio(smmu->mmio_addr, smmu->mmio_size)); + + if (smmu->cmdq.base) + WARN_ON(__pkvm_hyp_donate_host(smmu->cmdq.base_dma >> PAGE_SHIFT, + cmdq_size(&smmu->cmdq) >> PAGE_SHIFT)); smmu->base =3D NULL; } =20 @@ -99,6 +156,31 @@ static int smmu_probe(struct hyp_arm_smmu_v3_device *sm= mu) return 0; } =20 +/* + * The kernel part of the driver will allocate the shadow cmdq, + * and zero it. This function only donates it. + */ +static int smmu_init_cmdq(struct hyp_arm_smmu_v3_device *smmu) +{ + size_t cmdq_nr_pages =3D cmdq_size(&smmu->cmdq) >> PAGE_SHIFT; + int ret; + + ret =3D __pkvm_host_donate_hyp(smmu->cmdq.base_dma >> PAGE_SHIFT, cmdq_nr= _pages); + if (ret) + return ret; + + smmu->cmdq.base =3D hyp_phys_to_virt(smmu->cmdq.base_dma); + smmu->cmdq.prod_reg =3D smmu->base + ARM_SMMU_CMDQ_PROD; + smmu->cmdq.cons_reg =3D smmu->base + ARM_SMMU_CMDQ_CONS; + smmu->cmdq.q_base =3D smmu->cmdq.base_dma | + FIELD_PREP(Q_BASE_LOG2SIZE, smmu->cmdq.llq.max_n_shift); + smmu->cmdq.ent_dwords =3D CMDQ_ENT_DWORDS; + writel_relaxed(0, smmu->cmdq.prod_reg); + writel_relaxed(0, smmu->cmdq.cons_reg); + writeq_relaxed(smmu->cmdq.q_base, smmu->base + ARM_SMMU_CMDQ_BASE); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -117,7 +199,12 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_init_cmdq(smmu); + if (ret) + goto out_ret; + return 0; + out_ret: smmu_deinit_device(smmu); return ret; @@ -157,6 +244,22 @@ static int smmu_init(void) return ret; } =20 +static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; + + smmu->cmdq_host.llq.max_n_shift =3D min(shift, 19); + smmu->cmdq_host.base_dma =3D smmu->cmdq_host.q_base & Q_BASE_ADDR_MASK; + WARN_ON(smmu_share_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + +static void smmu_emulate_cmdq_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(smmu_unshare_pages(smmu->cmdq_host.base_dma, + cmdq_size(&smmu->cmdq_host))); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -180,6 +283,14 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, break; /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: + if (is_write) { + /* Not allowed by the architecture */ + if (WARN_ON(is_cmdq_enabled(smmu))) + break; + smmu->cmdq_host.q_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_CMDQ_PROD: case ARM_SMMU_CMDQ_CONS: case ARM_SMMU_STRTAB_BASE: @@ -190,6 +301,15 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, case ARM_SMMU_CR0: if (len !=3D sizeof(u32)) break; + if (is_write) { + bool last_cmdq_en =3D is_cmdq_enabled(smmu); + + smmu->cr0 =3D val; + if (!last_cmdq_en && is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_enable(smmu); + else if (last_cmdq_en && !is_cmdq_enabled(smmu)) + smmu_emulate_cmdq_disable(smmu); + } mask =3D read_write; break; case ARM_SMMU_CR1: { diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 263b0fef262d..cc1ad4c19845 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -8,6 +8,8 @@ #include #endif =20 +#include "../arm-smmu-v3.h" + /* * Parameters from the trusted host: * @mmio_addr base address of the SMMU registers @@ -20,6 +22,9 @@ * @pgsize_bitmap Supported page sizes * @sid_bits Max number of SID bits supported * @lock Lock to protect SMMU + * @cmdq CMDQ as observed by HW + * @cmdq_host Host view of the CMDQ, only q_base and llq used. + * @cr0 Last value of CR0 */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -34,6 +39,9 @@ struct hyp_arm_smmu_v3_device { #else u32 lock; #endif + struct arm_smmu_queue cmdq; + struct arm_smmu_queue cmdq_host; + u32 cr0; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E84C83A1A21 for ; Fri, 1 May 2026 11:20:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634426; cv=none; b=GSN2UKm/lZwOkJRxckkKTKW+B/CqqsYc6Uf8bQI85flTw3YVXWIVMs25i623T7uIimmbS5J2Cn3xYJs5t7uZoEEtBaxzRDzu+BsWg+2BdLi1lkn3ydrsj1cXFqSZmzh0wUSiU2mpCfuN+QqHJcmBda7VdRJM0hHcyf6AEhaHm/s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634426; c=relaxed/simple; bh=ht3+eoYVAH5LRRYEvRvfYwLs0rEwiSQ92hy2jk6fn44=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FF/u7NgFqp7815Xh5Stc4Ekim20v9p30x+CCQqNr36cpYGjApHyXl+dB6I3JjvLxjn1CyWKvnh4cLCYZ4lpw5k/wzk8suNI8zrIWF4NJUJkGRnz9aHyULisXWbPhFRiCBcYEsayx6wH5bWFlA130jv73G7QocyNt6hGNWeesGys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LJ8SxihL; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LJ8SxihL" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-6763c34cf60so2283234a12.0 for ; Fri, 01 May 2026 04:20:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634422; x=1778239222; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DQ1p7g2Tvq/hKfuZPoTWdpsu5XMV18xbkA26lNofyBM=; b=LJ8SxihLXHJ8QzE4lxPI+zD6KcX4pykl92Hv8RiPpcYzfoz08kb1SiD6jSKM/XdgY+ 54/Q9pg4WNOY+7Tj+WSrLbPWEM+54O/VM5HRQvwM+iZWb7kzAvGtUL8N5Xs0kzpVI4F8 frTk9DZtZnGnc3zj4DRdMP1DSnHh3domUfjFOVmYYsWs9s1M5jU7E8DPnUfj4LFd0Ygl MIQgAGKKJrNrezvA44GzV8t/TK9h4lHEYISx5fGS8JQZxl5LmPRFXee+buVIs65c8A9k AQC4sQA6AigOsUb4QC9lStk9BpONIPZkO2zWqVE0VRr8bW48GrV0qYsIFTuv63lEhsLz yogA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634422; x=1778239222; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DQ1p7g2Tvq/hKfuZPoTWdpsu5XMV18xbkA26lNofyBM=; b=AzNtoFmJsuw4Tg18LXD/Ej3uqa9tNBewgJVOSoE7CWSZ+86uvXzai1mFDSh4midK63 RyHZpYPhmNXJtKFFYvvw6nP6iEs+zj52kq0os/0ffJso8yL862UJ+GzgeV40XP0vxe5j J1wFCCFt0l60RbDR16EfrtiVdcgy9vHXuuqnpZTrknUXq89yx7ehafuyjm2tCcAx1auM 8ddzz+cSCcBEQCQvF2u5J6E427GzEEVwmLC5kfGPugXno0Hwk2I07vA4CwcT7gJzyIj2 t1U9buZSm4l+lmeyk6HmDcRLbYuHMACGvx69KvHhLvtl8D0Yocpwm6ej5kS/jHJ0/oQr lFQA== X-Forwarded-Encrypted: i=1; AFNElJ+h8DKIJFZ3hBZ4POalwgtS5IlsYRvAJ4uDWIIpJk0dcaBOj7NpIfOZkX7rymeqcyTPT5xq/05kQ/VFwrI=@vger.kernel.org X-Gm-Message-State: AOJu0Yzf5seVL9sOdvyfrqMlNET/QagI1eFpeTVkuhiXMBKWL+62TVSO 5XJykrMkCIEnk03IViQFzOgzC9qVYeOxxCDAvDR49Viohp/YraJ0wAgPsZe5l3eyQQDK3KkDDcq hve5MxHEOpdxcfg== X-Received: from edj4.prod.google.com ([2002:a05:6402:3244:b0:672:bfdb:8b72]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:aa7:d94f:0:b0:67b:acf5:f39e with SMTP id 4fb4d7f45d1cf-67bacf5f78amr800846a12.10.1777634422049; Fri, 01 May 2026 04:20:22 -0700 (PDT) Date: Fri, 1 May 2026 11:19:18 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-17-smostafa@google.com> Subject: [PATCH v6 16/25] iommu/arm-smmu-v3-kvm: Add CMDQ functions From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add functions to access the command queue, there are 2 main usage: - Hypervisor's own commands, as TLB invalidation, would use functions as smmu_send_cmd(), which creates and sends a command. - Add host commands to the shadow command queue, after being filtered, these will be added with smmu_add_cmd_raw. Signed-off-by: Mostafa Saleh --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 ++- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 107 ++++++++++++++++++ 2 files changed, 115 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h b/drivers/iommu/ar= m/arm-smmu-v3/arm-smmu-v3.h index f904f4d19609..3fc499608d76 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h @@ -1156,19 +1156,21 @@ u32 smmu_idr5_to_oas(u32 reg); unsigned long smmu_idr5_to_pgsize(u32 reg); =20 /* Queue functions shared between kernel and hyp. */ -static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +static inline u32 queue_space(struct arm_smmu_ll_queue *q) { - u32 space, prod, cons; + u32 prod, cons; =20 prod =3D Q_IDX(q, q->prod); cons =3D Q_IDX(q, q->cons); =20 if (Q_WRP(q, q->prod) =3D=3D Q_WRP(q, q->cons)) - space =3D (1 << q->max_n_shift) - (prod - cons); - else - space =3D cons - prod; + return (1 << q->max_n_shift) - (prod - cons); + return cons - prod; +} =20 - return space >=3D n; +static inline bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) +{ + return queue_space(q) >=3D n; } =20 static inline bool queue_full(struct arm_smmu_ll_queue *q) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 3b77796dafc7..aac455599728 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -6,6 +6,7 @@ */ #include =20 +#include #include #include #include @@ -22,6 +23,31 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 #define cmdq_size(cmdq) ((1 << ((cmdq)->llq.max_n_shift)) * CMDQ_ENT_DWORD= S * 8) =20 +/* + * Wait until @cond is true. + * Return 0 on success, or -ETIMEDOUT + */ +#define smmu_wait(use_wfe, _cond) \ +({ \ + int __ret =3D 0; \ + u64 delay =3D hyp_clock_ns() + ARM_SMMU_POLL_TIMEOUT_US * 1000; \ + \ + while (!(_cond)) { \ + if (use_wfe) { \ + wfe(); \ + if ((_cond)) \ + break; \ + } else { \ + cpu_relax(); \ + } \ + if (hyp_clock_ns() >=3D delay) { \ + __ret =3D -ETIMEDOUT; \ + break; \ + } \ + } \ + __ret; \ +}) + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -74,6 +100,87 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t = size) return 0; } =20 +__maybe_unused +static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_has_space(llq, n); +} + +static bool smmu_cmdq_full(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_full(llq); +} + +static bool smmu_cmdq_empty(struct arm_smmu_queue *cmdq) +{ + struct arm_smmu_ll_queue *llq =3D &cmdq->llq; + + WRITE_ONCE(llq->cons, readl_relaxed(cmdq->cons_reg)); + return queue_empty(llq); +} + +static void smmu_add_cmd_raw(struct hyp_arm_smmu_v3_device *smmu, + u64 *cmd) +{ + struct arm_smmu_queue *q =3D &smmu->cmdq; + struct arm_smmu_ll_queue *llq =3D &q->llq; + + queue_write(Q_ENT(q, llq->prod), cmd, CMDQ_ENT_DWORDS); + llq->prod =3D queue_inc_prod_n(llq, 1); +} + +static int smmu_add_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *ent) +{ + int ret; + u64 cmd[CMDQ_ENT_DWORDS]; + + ret =3D smmu_wait(false, !smmu_cmdq_full(&smmu->cmdq)); + if (ret) + return ret; + + ret =3D arm_smmu_cmdq_build_cmd(cmd, ent); + if (ret) + return ret; + + smmu_add_cmd_raw(smmu, cmd); + writel(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + return 0; +} + +static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_CMD_SYNC, + }; + + ret =3D smmu_add_cmd(smmu, &cmd); + if (ret) + return ret; + + return smmu_wait(smmu->features & ARM_SMMU_FEAT_SEV, + smmu_cmdq_empty(&smmu->cmdq)); +} + +__maybe_unused +static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd) +{ + int ret =3D smmu_add_cmd(smmu, cmd); + + if (ret) + return ret; + + return smmu_sync_cmd(smmu); +} + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E4B33A2568 for ; Fri, 1 May 2026 11:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634427; cv=none; b=g3YeB1KjpbnJbMEh1TULEd5e9uLKJnX/+pDbLWVHMFHNHzOHN02adI2iCFusG4liDyzG7KfygHt0i4tSGL20f00lHDbmoAaw/wHpXnlWc0WK/A0m1fGd/bBKS1KI9sqkH3mCtq5JNq6FK8/1E1p2ZWjDGtH6gt/GaZRFQe/jWPU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634427; c=relaxed/simple; bh=EUQFrcZsstiOOzdonmOjv2zocRlqE5cRVJsvTbS1jvs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=uUw3JHn/H4z//aypo9yh5ugYnJHBgmfLZLGGFIlLi8xNB+aVOCzgDT9G2e2mur+rH1mCLoG8+fxlaro4KLmaxZp8xvzwULqXvZfY6oRKVJglhZhvAZGKmt8QlrMAnXWarrj4AIWoxlnM71vvqrU9LP0wKJXZIMZ/irIivNhRj1s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=E/Bgj7nR; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="E/Bgj7nR" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-43ff0eb2b2aso1405362f8f.2 for ; Fri, 01 May 2026 04:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634424; x=1778239224; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=9dQBaHVEA8R39nVXfDncsPbJ/+uAVI3QDhXt4O/6+ME=; b=E/Bgj7nRv/MOP8CaRFy/4RPgB0wk88Yh61O1troLeHwshSz2tXJQmPKWmW188zdITI 7fKkrR7ZSbliJAvh5fwHVddZtemAO8+ZyaRzyGUkAJH3Fb0PrTdlk9xl4XZ6uGfBLW9P 8ClcroJeBHcnQUT0SMCu+5MMi0Gm6KU0MKhuRAg+JjgB/zpN+heriuzhiZ7ctL2mXgY0 KMKmkSUsc2GOouOOn0N4u53+g0r20xkWwr4oqdfC+1/otNdPCaJ0GHQIDfskwtafV0XE z0TSXET1AuhPhpphnVnLi5tM5Mxqfitry3ra73rZq5bRMvGX2sLm77PHjU/q1ZbO8d5b eWvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634424; x=1778239224; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=9dQBaHVEA8R39nVXfDncsPbJ/+uAVI3QDhXt4O/6+ME=; b=KYws27bzgWOEGTs90iXyCuhvWv2W+LAy0lyXKO5kpyUFEb/VblwzWjk9WcQ5EnkPoa Rh+3UsB3FdU9yR70tFPXAqb1O3ZFIsXIG9DI38u9W6pVyDf9AmUzjAIoABH4GrHznEJg NfpVSwKUoTE6HfcDSEpEn2DEOKm4ood/LNKtgUAR4JM6UHXvnHFlH/tErJgLJyfYFl67 VNZk5WZ3Cx+swv6c2lfn6oJ6zsmabD18b5IVKC6+ubiquWR9bWCOSs4zfe6ByvgZUiPY W/MLKPXPfmUif5uOWKpp7/arzlaTHiy/gX3vVZGK1ti03q4hEK8TcjMI2D0Ia/unw0FQ Ptyg== X-Forwarded-Encrypted: i=1; AFNElJ9T89FLfjcG1OK1C+rtLhAqPIxTm0nG1708GZ/uuAvT8Ria7YNPSVel5NWKr13WgjllTBYnvMDc9ZT7WIA=@vger.kernel.org X-Gm-Message-State: AOJu0YwNlzslW3CRHUsUBla/7+UxHvwdJTNkyXWet9r0h9NJIHhwkwMc tLofpe9AmH4M33tEs4w1uIHhlEsgcebSSr09bdD5uevaCKe4vvResvicOf4filVnOdvnlkT6VQC MiWI5zZqckSK+cg== X-Received: from wroj14.prod.google.com ([2002:adf:f00e:0:b0:440:62c9:5fd4]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:adf:fc50:0:b0:449:e8c0:fd58 with SMTP id ffacd0b85a97d-449e8c0fe50mr5501396f8f.27.1777634423506; Fri, 01 May 2026 04:20:23 -0700 (PDT) Date: Fri, 1 May 2026 11:19:19 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-18-smostafa@google.com> Subject: [PATCH v6 17/25] iommu/arm-smmu-v3-kvm: Emulate CMDQ for host From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don=E2=80=99t allow access to the command queue from the host: - ARM_SMMU_CMDQ_BASE: Only allowed to be written when CMDQ is disabled, we use it to keep track of the host command queue base. Reads return the saved value. - ARM_SMMU_CMDQ_PROD: Writes trigger command queue emulation which sanitise and filters the whole range. Reads returns the host copy. - ARM_SMMU_CMDQ_CONS: Writes move the sw copy of the cons, but the host can=E2=80=99t skip commands once submitted. Reads return the emulated val= ue and the error bits in the actual cons. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 128 +++++++++++++++++- 1 file changed, 124 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index aac455599728..1633a3cf8a3b 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -100,7 +100,6 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t = size) return 0; } =20 -__maybe_unused static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -351,6 +350,92 @@ static int smmu_init(void) return ret; } =20 +static bool smmu_filter_command(struct hyp_arm_smmu_v3_device *smmu, u64 *= command) +{ + u64 command0 =3D le64_to_cpu(command[0]); + u64 command1 =3D le64_to_cpu(command[1]); + u64 type =3D FIELD_GET(CMDQ_0_OP, command0); + + switch (type) { + case CMDQ_OP_CFGI_STE: + /* TBD: SHADOW_STE*/ + break; + case CMDQ_OP_CFGI_ALL: + { + /* + * Linux doesn't use range STE invalidation, and only use this + * for CFGI_ALL, which is done on reset and not on an new STE + * being used. + * Although, this is not architectural we rely on the current Linux + * implementation. + */ + if ((FIELD_GET(CMDQ_CFGI_1_RANGE, command1) !=3D 31)) + return true; + break; + } + case CMDQ_OP_TLBI_NH_ASID: + case CMDQ_OP_TLBI_NH_VA: + case 0x13: /* CMD_TLBI_NH_VAA: Not used by Linux */ + { + /* Only allow VMID =3D 0 */ + if (FIELD_GET(CMDQ_TLBI_0_VMID, command0) !=3D 0) + return true; + break; + } + case 0x10: /* CMD_TLBI_NH_ALL: Not used by Linux */ + case CMDQ_OP_TLBI_EL2_ALL: + case CMDQ_OP_TLBI_EL2_VA: + case CMDQ_OP_TLBI_EL2_ASID: + case CMDQ_OP_TLBI_S12_VMALL: + case CMDQ_OP_TLBI_S2_IPA: + case 0x23: /* CMD_TLBI_EL2_VAA: Not used by Linux */ + return true; + case CMDQ_OP_CMD_SYNC: + if (FIELD_GET(CMDQ_SYNC_0_CS, command0) =3D=3D CMDQ_SYNC_0_CS_IRQ) { + /* Allow it, but let the host timeout, as this should never happen. */ + command0 &=3D ~CMDQ_SYNC_0_CS; + command0 |=3D FIELD_PREP(CMDQ_SYNC_0_CS, CMDQ_SYNC_0_CS_SEV); + command1 &=3D ~CMDQ_SYNC_1_MSIADDR_MASK; + } + break; + } + + return false; +} + +static int smmu_emulate_cmdq_insert(struct hyp_arm_smmu_v3_device *smmu) +{ + u64 *host_cmdq =3D hyp_phys_to_virt(smmu->cmdq_host.q_base & Q_BASE_ADDR_= MASK); + bool use_wfe =3D smmu->features & ARM_SMMU_FEAT_SEV, skip; + u64 cmd[CMDQ_ENT_DWORDS]; + int idx, ret; + u32 space; + + if (!is_cmdq_enabled(smmu)) + return 0; + + space =3D (1 << (smmu->cmdq_host.llq.max_n_shift)) - queue_space(&smmu->c= mdq_host.llq); + /* Wait for the command queue to have some space. */ + ret =3D smmu_wait(use_wfe, smmu_cmdq_has_space(&smmu->cmdq, space)); + if (ret) + return ret; + + while (space--) { + idx =3D Q_IDX(&smmu->cmdq_host.llq, smmu->cmdq_host.llq.cons); + queue_inc_cons(&smmu->cmdq_host.llq); + + memcpy(cmd, &host_cmdq[idx * CMDQ_ENT_DWORDS], CMDQ_ENT_DWORDS << 3); + skip =3D smmu_filter_command(smmu, cmd); + if (WARN_ON(skip)) + continue; + smmu_add_cmd_raw(smmu, cmd); + } + + writel(smmu->cmdq.llq.prod, smmu->cmdq.prod_reg); + + return smmu_wait(use_wfe, smmu_cmdq_empty(&smmu->cmdq)); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; @@ -388,18 +473,51 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, /* Clear stage-2 support, hide MSI to avoid write back to cmdq */ mask =3D read_only & ~(IDR0_S2P | IDR0_VMID16 | IDR0_MSI | IDR0_HYP); break; - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_CMDQ_BASE: + /* + * Although allowed to use smaller size, we rely on the SMMUv3 driver + * using 64-bit store instruction for simplicity. + */ + if (len !=3D sizeof(u64)) + break; if (is_write) { /* Not allowed by the architecture */ if (WARN_ON(is_cmdq_enabled(smmu))) break; smmu->cmdq_host.q_base =3D val; + goto out_ret; + } else { + val =3D smmu->cmdq_host.q_base; + goto out_update_regs; } - mask =3D read_write; - break; case ARM_SMMU_CMDQ_PROD: + if (len !=3D sizeof(u32)) + break; + if (is_write) { + smmu->cmdq_host.llq.prod =3D val; + WARN_ON(smmu_emulate_cmdq_insert(smmu)); + goto out_ret; + } else { + val =3D smmu->cmdq_host.llq.prod; + goto out_update_regs; + } case ARM_SMMU_CMDQ_CONS: + if (len !=3D sizeof(u32)) + break; + if (is_write) { + if (WARN_ON(is_cmdq_enabled(smmu))) + break; + + smmu->cmdq_host.llq.cons =3D val; + goto out_ret; + } else { + /* Propagate errors back to the host.*/ + u32 cons =3D readl_relaxed(smmu->base + ARM_SMMU_CMDQ_CONS); + + val =3D smmu->cmdq_host.llq.cons | (CMDQ_CONS_ERR & cons); + goto out_update_regs; + } + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: case ARM_SMMU_STRTAB_BASE_CFG: case ARM_SMMU_GBPA: @@ -495,6 +613,8 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_dev= ice *smmu, val =3D readq_relaxed(smmu->base + off) & mask; else val =3D readl_relaxed(smmu->base + off) & mask; + +out_update_regs: /* * Device might be read senstive, so do it but ignore writing * back for xzr. --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B27E3A3817 for ; Fri, 1 May 2026 11:20:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634428; cv=none; b=U8/smkGWQhXnlSyhy1zCksPQjN5yEeLJrqgoY35X0xSEGtlifff8Y9o95QhL+FoUXJZz3VgXLo8E2QudPgCu7MVQ/5wh2rTseBxG6x6TgYZw+2Tl+f3/pa49sP6HaampUM8KZhvRUs/CgOsooAWubtWtxbHYFwkgzwjcJj+Uzr8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634428; c=relaxed/simple; bh=fY8MgosuIzxAf/kMmWlM8XbQqE2bmppPhcnH3qooiwk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=LNTEfUuNzipiz0zyh/NPpz4ij06eXivLjo8LAxJpjJe7KqxbpKczc3fNGrxT0qGzV/aZoUd+uN0x3aSew4Lu5kYiEb5C1l9/iN2K07HpvM5WqgyfJjGqRImFj1A6TDYxLCimFfMB7M9WGcgrwvGOnH6YGnp4n+rR9/j/H916Sck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uq52qXuz; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uq52qXuz" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-48a55ecc32cso16296995e9.1 for ; Fri, 01 May 2026 04:20:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634425; x=1778239225; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5m1KpsLfHL+l1yTb9rqM+ybYvsBgDjrRbLz7Mm8Rnis=; b=uq52qXuz7J2KGefSWPTGNon+hSVd0z4OdCAPqBDFHG3zeeMum59F/5zv0BWC7q670h KlWLk/tnmwH9ELmHzyLb7oiu7QDhcBZYfqKodtQBeNwYPxVIWm+qyb4NfY/Gnkpgg9g4 3fCfAt7Rkx/jD5+V96gApTQXiQ+MuyfxgYEVoZ9xC0vOpeuMFtHvaUKJZdLxJxrqHYt3 OImnw1c5h5/uRenNeVJ/2H5yuc97KTopkaGrs3ao9A5F47miz7MIK3pGo7vDHDzqAplK 38MNOwr5o6Mf9AAN8HYL5OR675eVZIb5f6/ZghiI7ZgVMrVR+tyVHztRGuCysncsclRi 3KDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634425; x=1778239225; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5m1KpsLfHL+l1yTb9rqM+ybYvsBgDjrRbLz7Mm8Rnis=; b=ldkSA+y+BSf7UmpIxEMksc01dB70x4iRtiY2PcwcmdWKaS7DTeZg9I3SjG8kexhC9w wvqw8akVp/ECrGy9B0GKNQlCSUw3HPGIHFompQxwSwGVOKkLTy1A3hiTJwGfH4bbg8sq X9GQA/AvsZeTvnUK3pCCHY79xOcqj3xc2giZQ6XRMf7nn3jIOm9NcSjkKiJJPmJ9anij HDZwKHkHlnys1g+8xhmdQBxZUYEezRoEbJlgU1tHON8NDcWEbEEdBCo51z/qwbY5O+Cv xS/0GW/1gthXhxut/Oi/6+TsqTEw/J9zlCDWhGCQ6m8CLDDpokMO90IFZqYZPtj0Zl7S 3kww== X-Forwarded-Encrypted: i=1; AFNElJ+E+XQ23KcCe0bvV1LC9uo4BM9RPtan2d6OaN9AdePCpNX+WTGQnvxJ4u5k7neIuPg8vtnu+dlsFI5fyMQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxpC9Pi45nbpn10SOt+bUJgUdH80x3bcMI0s3sQ96SCLpdJjcVO stx0h5TfrCEAUwGI8aysqW52RssZIwbs7XCqDb7wPHVywCMU12ghUptfIw5048QPrVWhsMvBCna bxSKa4gy4mINGqw== X-Received: from wmbjl23.prod.google.com ([2002:a05:600c:6a97:b0:48a:79c3:d04c]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:198b:b0:488:a2ac:a334 with SMTP id 5b1f17b1804b1-48a83d66cb9mr112477685e9.3.1777634424524; Fri, 01 May 2026 04:20:24 -0700 (PDT) Date: Fri, 1 May 2026 11:19:20 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-19-smostafa@google.com> Subject: [PATCH v6 18/25] iommu/arm-smmu-v3-kvm: Shadow stream table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allocate the shadow stream table per SMMU. We choose the size of that table to be 1MB which is the max size used by host in the case of 2 levels. All the host writes are still paththrough for bisectibility, that is changed next where CFGI commands will be trapped and used to update the shadow copy hypervisor that will be used by HW. Similar to the command queue, the host stream table is shared/unshared each time the SMMU is enabled/disabled. Handling of L2 tables is also done in the next patch when the shadowing is added. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c | 21 ++- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 122 ++++++++++++++++++ .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 10 ++ 3 files changed, 152 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c b/drivers/iomm= u/arm/arm-smmu-v3/arm-smmu-v3-kvm.c index fccbc34de087..7aec558eea29 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-kvm.c @@ -16,6 +16,13 @@ #include "pkvm/arm_smmu_v3.h" =20 #define SMMU_KVM_CMDQ_ORDER 4 +/* + * Use the max value of L1 the kernel uses, that also covers the worst case + * for linear tables as it is mandatory according to the spec to support 2 + * lvl tables if SIDSIZE >=3D 7 + */ +#define SMMU_KVM_STRTAB_ORDER (get_order(STRTAB_MAX_L1_ENTRIES * \ + sizeof(struct arm_smmu_strtab_l1))) =20 extern struct kvm_iommu_ops kvm_nvhe_sym(smmu_ops); =20 @@ -34,6 +41,9 @@ static void kvm_arm_smmu_array_free(void) if (smmu->cmdq.base_dma) free_pages((unsigned long)phys_to_virt(smmu->cmdq.base_dma), SMMU_KVM_CMDQ_ORDER); + if (smmu->strtab_dma) + free_pages((unsigned long)phys_to_virt(smmu->strtab_dma), + SMMU_KVM_STRTAB_ORDER); } =20 order =3D get_order(kvm_arm_smmu_count * sizeof(*kvm_arm_smmu_array)); @@ -80,8 +90,8 @@ static int smmuv3_nesting_probe(struct platform_device *p= dev) { struct hyp_arm_smmu_v3_device *smmu =3D &kvm_arm_smmu_array[kvm_arm_smmu_= cur]; struct device *dev =3D &pdev->dev; + void *cmdq_base, *strtab; struct resource *res; - void *cmdq_base; =20 /* Only device tree, ACPI not supported. */ if (!dev->of_node) @@ -120,6 +130,15 @@ static int smmuv3_nesting_probe(struct platform_device= *pdev) smmu->cmdq.base_dma =3D virt_to_phys(cmdq_base); smmu->cmdq.llq.max_n_shift =3D SMMU_KVM_CMDQ_ORDER + PAGE_SHIFT - CMDQ_EN= T_SZ_SHIFT; =20 + strtab =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, SMMU_KVM_STR= TAB_ORDER); + if (!strtab) { + free_pages((unsigned long)cmdq_base, SMMU_KVM_CMDQ_ORDER); + return -ENOMEM; + } + + smmu->strtab_dma =3D virt_to_phys(strtab); + smmu->strtab_size =3D PAGE_SIZE << SMMU_KVM_STRTAB_ORDER; + kvm_arm_smmu_cur++; return 0; } diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 1633a3cf8a3b..d15c9e5aa998 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -16,6 +16,14 @@ size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 +/* strtab accessors */ +#define strtab_log2size(smmu) (FIELD_GET(STRTAB_BASE_CFG_LOG2SIZE, (smmu)-= >host_ste_cfg)) +#define strtab_size(smmu) ((1UL << strtab_log2size(smmu)) * STRTAB_STE_DWO= RDS * 8) +#define strtab_host_base(smmu) ((smmu)->host_ste_base & STRTAB_BASE_ADDR_M= ASK) +#define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) +#define strtab_l1_size(smmu) ((1UL << (strtab_log2size(smmu) - strtab_spli= t(smmu))) * \ + (sizeof(struct arm_smmu_strtab_l1))) + #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ (smmu) !=3D &kvm_hyp_arm_smmu_v3_smmus[kvm_hyp_arm_smmu_v3_count]; \ @@ -53,6 +61,11 @@ static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_devic= e *smmu) return FIELD_GET(CR0_CMDQEN, smmu->cr0); } =20 +static bool is_smmu_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_SMMUEN, smmu->cr0); +} + /* * CMDQ, STE host copies are accessed by the hypervisor, we share them to * - Prevent the host from passing protected VM memory. @@ -188,6 +201,11 @@ static void smmu_deinit_device(struct hyp_arm_smmu_v3_= device *smmu) if (smmu->cmdq.base) WARN_ON(__pkvm_hyp_donate_host(smmu->cmdq.base_dma >> PAGE_SHIFT, cmdq_size(&smmu->cmdq) >> PAGE_SHIFT)); + + if (smmu->strtab_cfg.linear.table || + smmu->strtab_cfg.l2.l1tab) + WARN_ON(__pkvm_hyp_donate_host(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT)); smmu->base =3D NULL; } =20 @@ -287,6 +305,45 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + int ret; + u32 reg; + + ret =3D __pkvm_host_donate_hyp(hyp_phys_to_pfn(smmu->strtab_dma), + smmu->strtab_size >> PAGE_SHIFT); + if (ret) + return ret; + + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + unsigned int last_sid_idx =3D + arm_smmu_strtab_l1_idx((1ULL << smmu->sid_bits) - 1); + + cfg->l2.l1tab =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->l2.l1_dma =3D smmu->strtab_dma; + cfg->l2.num_l1_ents =3D min(last_sid_idx + 1, STRTAB_MAX_L1_ENTRIES); + + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_2LVL) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, + ilog2(cfg->l2.num_l1_ents) + STRTAB_SPLIT) | + FIELD_PREP(STRTAB_BASE_CFG_SPLIT, STRTAB_SPLIT); + } else { + cfg->linear.table =3D hyp_phys_to_virt(smmu->strtab_dma); + cfg->linear.ste_dma =3D smmu->strtab_dma; + cfg->linear.num_ents =3D 1UL << smmu->sid_bits; + reg =3D FIELD_PREP(STRTAB_BASE_CFG_FMT, + STRTAB_BASE_CFG_FMT_LINEAR) | + FIELD_PREP(STRTAB_BASE_CFG_LOG2SIZE, smmu->sid_bits); + } + + writeq_relaxed((smmu->strtab_dma & STRTAB_BASE_ADDR_MASK) | STRTAB_BASE_R= A, + smmu->base + ARM_SMMU_STRTAB_BASE); + writel_relaxed(reg, smmu->base + ARM_SMMU_STRTAB_BASE_CFG); + return 0; +} + static int smmu_init_device(struct hyp_arm_smmu_v3_device *smmu) { unsigned long haddr; @@ -309,6 +366,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_init_strtab(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -436,6 +497,46 @@ static int smmu_emulate_cmdq_insert(struct hyp_arm_smm= u_v3_device *smmu) return smmu_wait(use_wfe, smmu_cmdq_empty(&smmu->cmdq)); } =20 +static int smmu_update_ste_shadow(struct hyp_arm_smmu_v3_device *smmu, boo= l enabled) +{ + size_t strtab_size; + u32 fmt =3D FIELD_GET(STRTAB_BASE_CFG_FMT, smmu->host_ste_cfg); + + /* Linux doesn't change the fmt nor size of the strtab in the run time. */ + if (smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB) { + if ((fmt !=3D STRTAB_BASE_CFG_FMT_2LVL) || + (strtab_split(smmu) !=3D STRTAB_SPLIT) || + (strtab_log2size(smmu) > (ilog2(STRTAB_MAX_L1_ENTRIES) + STRTAB_SPL= IT)) || + (strtab_split(smmu) >=3D strtab_log2size(smmu))) + return -EINVAL; + strtab_size =3D strtab_l1_size(smmu); + } else { + if ((fmt !=3D STRTAB_BASE_CFG_FMT_LINEAR) || + (strtab_log2size(smmu) > smmu->sid_bits)) + return -EINVAL; + strtab_size =3D strtab_size(smmu); + } + + if (enabled) + return smmu_share_pages(strtab_host_base(smmu), strtab_size); + + return smmu_unshare_pages(strtab_host_base(smmu), strtab_size); +} + +static void smmu_emulate_enable(struct hyp_arm_smmu_v3_device *smmu) +{ + /* Enabling SMMU without CMDQ, means TLB invalidation won't work. */ + if (WARN_ON(!is_cmdq_enabled(smmu))) + return; + + WARN_ON(smmu_update_ste_shadow(smmu, true)); +} + +static void smmu_emulate_disable(struct hyp_arm_smmu_v3_device *smmu) +{ + WARN_ON(smmu_update_ste_shadow(smmu, false)); +} + static void smmu_emulate_cmdq_enable(struct hyp_arm_smmu_v3_device *smmu) { u32 shift =3D smmu->cmdq_host.q_base & Q_BASE_LOG2SIZE; @@ -519,7 +620,23 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, } /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + if (is_smmu_enabled(smmu)) + break; + smmu->host_ste_base =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_STRTAB_BASE_CFG: + if (is_write) { + /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ + if (is_smmu_enabled(smmu)) + break; + smmu->host_ste_cfg =3D val; + } + mask =3D read_write; + break; case ARM_SMMU_GBPA: mask =3D read_write; break; @@ -528,12 +645,17 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, break; if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); + bool last_smmu_en =3D is_smmu_enabled(smmu); =20 smmu->cr0 =3D val; if (!last_cmdq_en && is_cmdq_enabled(smmu)) smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + if (!last_smmu_en && is_smmu_enabled(smmu)) + smmu_emulate_enable(smmu); + else if (last_smmu_en && !is_smmu_enabled(smmu)) + smmu_emulate_disable(smmu); } mask =3D read_write; break; diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index cc1ad4c19845..6a73cf6b8873 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -15,6 +15,8 @@ * @mmio_addr base address of the SMMU registers * @mmio_size size of the registers resource * @features Features of SMMUv3, subset of the main driver + * @strtab_dma Phys address of stream table + * @strtab_size Stream table size * * Other members are filled and used at runtime by the SMMU driver. * @base Virtual address of SMMU registers @@ -25,6 +27,9 @@ * @cmdq CMDQ as observed by HW * @cmdq_host Host view of the CMDQ, only q_base and llq used. * @cr0 Last value of CR0 + * @host_ste_cfg Host stream table config + * @host_ste_base Host stream table base + * @strtab_cfg Stream table as seen by HW */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -42,6 +47,11 @@ struct hyp_arm_smmu_v3_device { struct arm_smmu_queue cmdq; struct arm_smmu_queue cmdq_host; u32 cr0; + dma_addr_t strtab_dma; + size_t strtab_size; + u64 host_ste_cfg; + u64 host_ste_base; + struct arm_smmu_strtab_cfg strtab_cfg; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A7483A3E6D for ; Fri, 1 May 2026 11:20:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634429; cv=none; b=EW2mSISqoVlcZAFScvY59Q356/zzK0qR5vtwghH4iktE7uGw6DDJ61X5vc9xN1REF/WmxuMMXYtUzToHb1MvN9WBIFjIt1bTr9mWKKQf1PYneFoNNNxgflF0gvd5J71C8k/uxjmlwHg96RWYxofbe5S/pgNGUeRq0m8qeOOwyIs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634429; c=relaxed/simple; bh=quyqCx4BwvJd/so0F7ehZxJ2/HYxSOs09XGpwJ1CpE8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hBVZfUnL0ChtJNq7YlqRk0hJXDPiTrLWKVXEX7f5+StQ6cWYuFh6ULPxtaJf3WfVV+OLzGkgHAtJXaRDHQeKSxJnYYd3sDF2/BunKZjLTpfgb7otwUZlQtKX0W2oKfIS0C8p7MgUTI1m4xFjNJy+2Ui/VpD6zLk8na9ZlQtMxzk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GHexR+4N; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GHexR+4N" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-44b186b715aso175105f8f.0 for ; Fri, 01 May 2026 04:20:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634426; x=1778239226; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dc4s6xiUZWCpOIz9PUIb3W8M/9lmR8q/3KHesYxyGwI=; b=GHexR+4NDLcCjBxQRnTb0tXPQzio9ZYyJ5LJDfHcDfNuHqxG6pxTV9YMaXjXfbkYZj fDnzRzJjkN+D8DxxxtQbgWVV9RJa9TPkzI5lJwCW+rSzjNaoWxzGU3w0e2rhmLCZCJ1V iGLLWd5VYZqW6AsSuH0qXLCZABjm4cFrgTuneKlcDoAHzc0Aa/vZIAObktXIWMxcz8MV RgRfdSrBmCQtt4hrNiIVrNcshxIxcujntAvSMRFADIxRLpxHPwYY+WnDQu+8kYW6eG6L LhJw8+bUJhBUe/7TmnPk8xawRIQD//Hz4JEKayV+3Qo+2Ea9lbU6GRx95O/ybVuFK8mU Zt2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634426; x=1778239226; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dc4s6xiUZWCpOIz9PUIb3W8M/9lmR8q/3KHesYxyGwI=; b=KherhoaTAtlFZTHnC2MOLypHMjHFJh89ZEFXicueWFcm+LMkk/GuyV5s9faL63X1E/ H2EogDTzff5hdJnr16u7+xURJHiAh/0nY4vENMCeaBQbvKnP3pSmcLkokjq9/v2zGRBU lHmtOEDgSbc54isJ+bdvr7w6DXURbpuIqzgi1/uQoYiK/EPx5Dt8gctz1D72DDMH0DeK TPVEmb4kxz5Hjv8rzwpn6017OOsbh9Q4eiNCyW1NwWLgT2YzdmgBCeto1tAfTg33K4++ 9ZGRGNQQlBTqmTNIt9baAgEDzTBg32JHu6EgVfOmhEPcgfH+TkkVeuodSpOXQGXqYQul hGyQ== X-Forwarded-Encrypted: i=1; AFNElJ9uX0s2VKTNi0zHBeD+gtKEKHB1zOM2Bb3uGH03k4M257OjHsPpn+Rx6rrShGlp7dRgpFsrOEHm2aQREFM=@vger.kernel.org X-Gm-Message-State: AOJu0Yzbbh/Usikyjewsr7eseLRbzMm0z/VKVwypcXOxCNk7cLN8gx5H 4YGWZQB68KfVa30bsmoTziIRHhIV32C7IY28kWaciuxMLHvfFU5pcb6j2H8kyIbdBv0HYDozolt f4gz/Y2C4vuJO0Q== X-Received: from wrnd1.prod.google.com ([2002:adf:e841:0:b0:43d:76ee:764]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:e11:b0:449:acdb:3009 with SMTP id ffacd0b85a97d-449acdb3074mr6506108f8f.6.1777634425669; Fri, 01 May 2026 04:20:25 -0700 (PDT) Date: Fri, 1 May 2026 11:19:21 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-20-smostafa@google.com> Subject: [PATCH v6 19/25] iommu/arm-smmu-v3-kvm: Shadow STEs From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add STE emulation, when the host sends the CFGI_STE command. Copy the STE as is to the shadow owned by the hypervisor, in the next patch, stage-2 page table will be attached. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 114 +++++++++++++++++- 1 file changed, 108 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index d15c9e5aa998..d92811ef2af5 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -23,6 +23,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; #define strtab_split(smmu) (FIELD_GET(STRTAB_BASE_CFG_SPLIT, (smmu)->host_= ste_cfg)) #define strtab_l1_size(smmu) ((1UL << (strtab_log2size(smmu) - strtab_spli= t(smmu))) * \ (sizeof(struct arm_smmu_strtab_l1))) +#define strtab_hyp_base(smmu) ((smmu)->features & ARM_SMMU_FEAT_2_LVL_STRT= AB ? \ + (u64 *)(smmu)->strtab_cfg.l2.l1tab :\ + (u64 *)(smmu)->strtab_cfg.linear.table) =20 #define for_each_smmu(smmu) \ for ((smmu) =3D kvm_hyp_arm_smmu_v3_smmus; \ @@ -305,6 +308,91 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v3_device *smmu, u32 s= id, + struct arm_smmu_ste *host_ste_out) +{ + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + struct arm_smmu_strtab_l1 host_l1_desc; + struct arm_smmu_strtab_l2 *l2ptr; + phys_addr_t host_l2_tab; + int ret; + + host_l1_desc.l2ptr =3D le64_to_cpu(READ_ONCE(host_ste_base[arm_smmu_strta= b_l1_idx(sid)])); + if (!(host_l1_desc.l2ptr & STRTAB_L1_DESC_SPAN)) + return -EINVAL; + + host_l2_tab =3D host_l1_desc.l2ptr & STRTAB_L1_DESC_L2PTR_MASK; + /* Share and pin the table before accessing it. */ + ret =3D smmu_share_pages(host_l2_tab, sizeof(struct arm_smmu_strtab_l2)); + if (ret) + return ret; + + l2ptr =3D hyp_phys_to_virt(host_l2_tab); + memcpy(host_ste_out, &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)], + STRTAB_STE_DWORDS << 3); + WARN_ON(smmu_unshare_pages(host_l2_tab, sizeof(struct arm_smmu_strtab_l2)= )); + return 0; +} + +static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid,= bool leaf) +{ + struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; + struct arm_smmu_ste *hyp_ste_ptr, *host_ste_ptr, host_ste_copy; + u64 *hyp_ste_base =3D strtab_hyp_base(smmu); + int ret; + + /* + * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this + * is a 2 level table and reshadow of l2. + * Also, we rely on Linux only issuing CFGI_STE to attach a device when + * the SMMU is enabled. + */ + if (!leaf || !is_smmu_enabled(smmu) || + (sid >=3D (1UL << strtab_log2size(smmu)))) + return -EINVAL; + + if (!(smmu->features & ARM_SMMU_FEAT_2_LVL_STRTAB)) { + struct arm_smmu_ste *hyp_table =3D (struct arm_smmu_ste *)hyp_ste_base; + u64 *host_ste_base =3D hyp_phys_to_virt(strtab_host_base(smmu)); + struct arm_smmu_ste *host_table =3D (struct arm_smmu_ste *)host_ste_base; + + if (sid >=3D cfg->linear.num_ents) + return -E2BIG; + + hyp_ste_ptr =3D &hyp_table[sid]; + host_ste_ptr =3D &host_table[sid]; + } else { + struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)hyp_st= e_base; + u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); + struct arm_smmu_strtab_l2 *l2ptr; + + if (l1_idx >=3D cfg->l2.num_l1_ents) + return -E2BIG; + + host_ste_ptr =3D &host_ste_copy; + ret =3D smmu_get_host_l2_ste(smmu, sid, host_ste_ptr); + if (ret) + return ret; + + if (!l1tab[l1_idx].l2ptr) { + struct arm_smmu_strtab_l2 *l2table; + + /* No hypervisor entry, first time the L2 is populated. */ + l2table =3D kvm_iommu_donate_pages(get_order(sizeof(*l2table))); + if (!l2table) + return -ENOMEM; + arm_smmu_write_strtab_l1_desc(&l1tab[l1_idx], hyp_virt_to_phys(l2table)= ); + } + l2ptr =3D hyp_phys_to_virt(le64_to_cpu(l1tab[l1_idx].l2ptr) & + STRTAB_L1_DESC_L2PTR_MASK); + hyp_ste_ptr =3D &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; + } + + memcpy(hyp_ste_ptr->data, host_ste_ptr->data, STRTAB_STE_DWORDS << 3); + + return 0; +} + static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) { struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; @@ -419,8 +507,14 @@ static bool smmu_filter_command(struct hyp_arm_smmu_v3= _device *smmu, u64 *comman =20 switch (type) { case CMDQ_OP_CFGI_STE: - /* TBD: SHADOW_STE*/ + { + u32 sid =3D FIELD_GET(CMDQ_CFGI_0_SID, command[0]); + u32 leaf =3D FIELD_GET(CMDQ_CFGI_1_LEAF, command[1]); + + if (smmu_reshadow_ste(smmu, sid, leaf)) + return true; break; + } case CMDQ_OP_CFGI_ALL: { /* @@ -618,25 +712,33 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, val =3D smmu->cmdq_host.llq.cons | (CMDQ_CONS_ERR & cons); goto out_update_regs; } - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_STRTAB_BASE: + if (len !=3D sizeof(u64)) + break; if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ if (is_smmu_enabled(smmu)) break; smmu->host_ste_base =3D val; + goto out_ret; + } else { + val =3D smmu->host_ste_base; + goto out_update_regs; } - mask =3D read_write; - break; case ARM_SMMU_STRTAB_BASE_CFG: + if (len !=3D sizeof(u32)) + break; if (is_write) { /* Must only be written when SMMU_CR0.SMMUEN =3D=3D 0.*/ if (is_smmu_enabled(smmu)) break; smmu->host_ste_cfg =3D val; + goto out_ret; + } else { + val =3D smmu->host_ste_cfg; + goto out_update_regs; } - mask =3D read_write; - break; + /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: mask =3D read_write; break; --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A27AA3A3E6C for ; Fri, 1 May 2026 11:20:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634431; cv=none; b=mpGoa8n2b2vsLPRv5iICycvDSqW+HWDVJ8Wqb0t/UnltldqpaolAOkwXveQZobyXTFu6pS6TRkMKdEPDInw4sCJFdZXPELvLHgon29PKWiD+1bfD34pbm7H1qw3CwAa01f8XuUsCmUPOrjZGiDwKINqKZNwbF7UTJcZ7MnJqZ/E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634431; c=relaxed/simple; bh=0JvbB+uG8UeC6ETWOsWx833yVnlDygSPCyX5gmo8vcA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=GpBY7xf6+ntDykzWXJwsirJAGrJo/8aGACclPNRwp8COEAJRimhGDU+6022TMwMp6Qav8kCnyBdTMAnY+RsQgxWto6CE9KSMGL5IoIYjNl5WyRoHK4Y8zGeKY/cg+GtyiOP0JP69/+FByYhrNsmGAFXmAUaa5T0daSErEmhtVUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=uxVJJ/cK; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="uxVJJ/cK" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-488c2a4e257so13589045e9.3 for ; Fri, 01 May 2026 04:20:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634428; x=1778239228; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LHCyI4M39cq38I/WoB1TwD8PIWXzXb414TVCe9luOcI=; b=uxVJJ/cKjNbGdsnwuF5N0JRvtN58/bcoNBbXySScXyFLc0FMx5D4JncczUzFRneaF3 Xevu9MFVzwgwO2SJ42BhAxghYfCQa0t0e/dClrQ15UDVgQUpuZeQJpRDiiZ5M4fc+gqI IbtaqMrDZImhtIDaT8g96LoGHesi17K8vyMVkeTIOGH2yeczUf5oLPYdrJYBJ1M7uWhp sf4RKUoJfGPouZt79hgfqDGQRCkn1V8F3hH44v6RaFFvv6ckn262ln2GFvRsSsWblxSn QnoSAZ9OJ1msxVnv9MogTXwdMEkDUS6OQT9iLNiTHYCnU3JQBBJlSD7ryYTw1PuHRoBJ jp+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634428; x=1778239228; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LHCyI4M39cq38I/WoB1TwD8PIWXzXb414TVCe9luOcI=; b=cGlj/7eIXlaxKq7WNeb3ArDb0QYfKj6av8PMAJv9pMesXSJ6FCysCLSQ9fKynN9h61 F/Fja+Sw/9JMCKfhlp4z/tp9wTbBIR11YrBf+r2AFdcpedomGnq3CaMhlSsDULN0q1Rk mNB2d2uO+j6tm5ZxdKaSEhy4BxkObzrxKYip0i4S2ks5iKSXij41GHOGO4z/aZ1ow0n+ fU/XKq5jIhoA59mqYqqSnsJFWgBRB97c4JrH3MPIlqtFDhPxkfe5bTKhZDTUizsszSFv LyBisFgn3oSXKGvEEqWq4y+NOjYxQCbUOFXDinsmE8zd8XTE7H5Sny0atfFfraF9Hb0y xDRg== X-Forwarded-Encrypted: i=1; AFNElJ8BRb36ZuEEYURX932oSmxrZv8068uFNdyXYSGYsWxZrrcBDwzxs4JAduY/MpXoRhfAqVa2qeVQQDJng2A=@vger.kernel.org X-Gm-Message-State: AOJu0Yz6aMhdfeg6TcWFxkEoZqpqQqwRFCOBS/KWLR7EA7tBvdaC/E3s b56+NRmaVZPxPn+D9JdJqb04RCBebr5iiIdoEuJncJQLUvs+x7pjNQK3GVOE6GDL+tFpw6gJ7MC XVN9gCNJBGGIHNw== X-Received: from wmxa17-n2.prod.google.com ([2002:a05:600d:6451:20b0:48a:5d27:4655]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e55:b0:48a:554d:b9a2 with SMTP id 5b1f17b1804b1-48a8eb616a7mr42752255e9.6.1777634427898; Fri, 01 May 2026 04:20:27 -0700 (PDT) Date: Fri, 1 May 2026 11:19:22 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-21-smostafa@google.com> Subject: [PATCH v6 20/25] iommu/arm-smmu-v3-kvm: Share other queues From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Other queues as PRIQ and EVTQ doesn't need to be shadowed. However, we need to make sure they are in a state that disallow them to be donated to the hypervisor or guests. So, keep track of those and share them when they get enabled. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 62 ++++++++++++++++++- .../iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h | 4 ++ 2 files changed, 64 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index d92811ef2af5..e258690384f4 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -69,6 +69,16 @@ static bool is_smmu_enabled(struct hyp_arm_smmu_v3_devic= e *smmu) return FIELD_GET(CR0_SMMUEN, smmu->cr0); } =20 +static bool is_evtq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_EVTQEN, smmu->cr0); +} + +static bool is_priq_enabled(struct hyp_arm_smmu_v3_device *smmu) +{ + return FIELD_GET(CR0_PRIQEN, smmu->cr0); +} + /* * CMDQ, STE host copies are accessed by the hypervisor, we share them to * - Prevent the host from passing protected VM memory. @@ -647,6 +657,14 @@ static void smmu_emulate_cmdq_disable(struct hyp_arm_s= mmu_v3_device *smmu) cmdq_size(&smmu->cmdq_host))); } =20 +static void smmu_emulate_queue(unsigned long q_base, size_t ent_size_shift) +{ + phys_addr_t base =3D q_base & Q_BASE_ADDR_MASK; + size_t size =3D 1UL << (FIELD_GET(Q_BASE_LOG2SIZE, q_base) + ent_size_shi= ft); + + WARN_ON(smmu_share_pages(base ,size)); +} + static bool smmu_dabt_device(struct hyp_arm_smmu_v3_device *smmu, struct user_pt_regs *regs, u64 esr, u32 off) @@ -748,12 +766,31 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, if (is_write) { bool last_cmdq_en =3D is_cmdq_enabled(smmu); bool last_smmu_en =3D is_smmu_enabled(smmu); + bool last_evtq_en =3D is_evtq_enabled(smmu); + bool last_priq_en =3D is_priq_enabled(smmu); =20 smmu->cr0 =3D val; if (!last_cmdq_en && is_cmdq_enabled(smmu)) smmu_emulate_cmdq_enable(smmu); else if (last_cmdq_en && !is_cmdq_enabled(smmu)) smmu_emulate_cmdq_disable(smmu); + + /* + * Share PRI and EVTQ to avoid the host using them to write to + * protected memory. However, panic on disable for those queues + * as that is more complicated, unsharing from here can lead to + * use-after-unshare issues, and requires ordering with cr0ack. + * As the host never disable those queues, don't support that. + */ + if (!last_evtq_en && is_evtq_enabled(smmu)) + smmu_emulate_queue(smmu->evtq_base, EVTQ_ENT_SZ_SHIFT); + else if (last_evtq_en && !is_evtq_enabled(smmu)) + WARN_ON(1); + if (!last_priq_en && is_priq_enabled(smmu)) + smmu_emulate_queue(smmu->priq_base, PRIQ_ENT_SZ_SHIFT); + else if (last_priq_en && !is_priq_enabled(smmu)) + WARN_ON(1); + if (!last_smmu_en && is_smmu_enabled(smmu)) smmu_emulate_enable(smmu); else if (last_smmu_en && !is_smmu_enabled(smmu)) @@ -779,6 +816,29 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_de= vice *smmu, mask =3D read_write; break; } + case ARM_SMMU_EVTQ_BASE: + if (len !=3D sizeof(u64)) + break; + + if (is_write) { + if (is_evtq_enabled(smmu)) + break; + smmu->evtq_base =3D val; + } + mask =3D read_write; + break; + + case ARM_SMMU_PRIQ_BASE: + if (len !=3D sizeof(u64)) + break; + + if (is_write) { + if (is_priq_enabled(smmu)) + break; + smmu->priq_base =3D val; + } + mask =3D read_write; + break; =20 /* Allowed 32 bit registers. */ case ARM_SMMU_EVTQ_PROD + SZ_64K: @@ -801,9 +861,7 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_dev= ice *smmu, mask =3D read_write; break; /* Allowed 64 bit registers. */ - case ARM_SMMU_EVTQ_BASE: case ARM_SMMU_EVTQ_IRQ_CFG0: - case ARM_SMMU_PRIQ_BASE: case ARM_SMMU_PRIQ_IRQ_CFG0: case ARM_SMMU_GERROR_IRQ_CFG0: if (len !=3D sizeof(u64)) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h index 6a73cf6b8873..e811d51bdfaa 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm_smmu_v3.h @@ -30,6 +30,8 @@ * @host_ste_cfg Host stream table config * @host_ste_base Host stream table base * @strtab_cfg Stream table as seen by HW + * @evtq_base Host evtq base reg + * @priq_base Host priq base reg */ struct hyp_arm_smmu_v3_device { phys_addr_t mmio_addr; @@ -52,6 +54,8 @@ struct hyp_arm_smmu_v3_device { u64 host_ste_cfg; u64 host_ste_base; struct arm_smmu_strtab_cfg strtab_cfg; + unsigned long evtq_base; + unsigned long priq_base; }; =20 extern size_t kvm_nvhe_sym(kvm_hyp_arm_smmu_v3_count); --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB11139E166 for ; Fri, 1 May 2026 11:20:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634432; cv=none; b=uHjUfyC0dAIk/T8woALA0dGNyxrcQD4Bktyls1WdnPaCgsiWbj7x3CYeMwPkNNCVTzcqM47DQunvdMF8clM5gwfvJflHh1DkrqXEuLrwHJVDZWaF7vsv64qoMsZsypRBNcAl9YNKjO1vulMXkJPPwYfUjm6mtXSpWrsnspS+OhE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634432; c=relaxed/simple; bh=4IbK2AkELhYHNt+kY7TsOF+nTpzv9kfiFGq+yMByoyA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MYbX6Kb552BAiZeVgE5xx5OuHbQ4gGygYprn7ecDfsBbgFROec5PbciMmOxseAb47VpwG1CDQWRY23VAHOCT9t43stCHTlPJ0LtiyQVZllXcyLaF5WLlRyVqzDFKvRzkC7SWQHR2OPJqdFl8N+baS+TpFZRz7K8QSGw/2RjzqDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z2ls4e0N; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z2ls4e0N" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-678a526f24aso1967397a12.3 for ; Fri, 01 May 2026 04:20:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634429; x=1778239229; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=OvQ7rT8sfpUuWMPN37mpWwK5CZ/GStRVkhZPxjRKMSU=; b=Z2ls4e0NVbWz+dsMMluHhOOePefYvuQFezKYZcjWua8odiholVUnDHXD8B23J6OWX5 nNoUGFUr4No/NZgztnIGl0CN+/1Hz0aCbAEuUZrsG6eMgFBknAQkqXVzgp+WJEdT0bdE OvsPnZyEbjaJpGJiUeEzC8dyuQ4aPIwTKBmFiFbjLnY6hJLdRSx9wTBw5vJ6nG+n1LrX dwW14X/iETUxnEyoTlR3GKJuD7Ldu6uvi4qFBKtcPrLi+5yvbNX9wPgBIAUTzksEPpyQ CPISe0HcV7F4kNYha0UddE70pHDL8BFvfo4iNimdXzUzftTx8LyWoJj4GPk0QQmQwmLm j9Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634429; x=1778239229; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=OvQ7rT8sfpUuWMPN37mpWwK5CZ/GStRVkhZPxjRKMSU=; b=GnZuX897a2Pp/HoBRzubklDvT0cKifqKo26R8W5gylqRvC9eklVtCG2rcQKMCCEm5l KNU8xItRTW45bhYSFwEy6eFspEgfBxtd2sA9t/14K4O9De2qsZmVoGPx0KXFq46efPNU ypqD9wyQdc62RBJXgo8/TAp6EKsZyisDHGb1xOF0vQ2aglPfFY708DcVzu3DUVa3l4MN I6ihEc9fCNgjR0+0cxYiadGLIJSQrxuxZKYLenP21wyjgWOCmyJQs3By1L0ey3uO5imi 9R2WGRm6sdeXmwpHO57Ff7GJukApvg7lGQemmpWTnGf1uUWpenJGxcHxyUuJJD0PO6IY 719g== X-Forwarded-Encrypted: i=1; AFNElJ8NziRBI+G7IBYJbzImNjsYa2swxw+u0mQ+K8gCZ4vNrT21n+owlQoF3U4R74fjID4w9EYT1qu7iaKH4lQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yz+g8c2KYotqW11h9/nQ2QCy0M2gxDLFNQ84Yq2lSbNjCa6TUc8 ba5e0anThGHtECnVDyyXI6oyDcC8P/A7dyGosvmgxJ6RLrO9otvH5b70QaZmrP41KsvxK+jimQk n9N85SSsBIRBvCg== X-Received: from edbeo10.prod.google.com ([2002:a05:6402:530a:b0:66f:bef2:d9de]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:1588:b0:670:8b38:5717 with SMTP id 4fb4d7f45d1cf-67b5d84910cmr3588586a12.24.1777634428940; Fri, 01 May 2026 04:20:28 -0700 (PDT) Date: Fri, 1 May 2026 11:19:23 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-22-smostafa@google.com> Subject: [PATCH v6 21/25] iommu/arm-smmu-v3-kvm: Emulate GBPA From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The last bit of emulation is GBPA. it must be always set to ABORT, as when the SMMU is disabled it=E2=80=99s not allowed for the host to bypass the SMMU. That's is done by setting the GBPA to ABORT at init time, and host writes are always ignored and host reads always return ABORT. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 32 +++++++++++++++++-- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index e258690384f4..1ed5ccce7849 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -126,6 +126,22 @@ static int smmu_unshare_pages(phys_addr_t addr, size_t= size) return 0; } =20 +static int smmu_abort_gbpa(struct hyp_arm_smmu_v3_device *smmu) +{ + int ret; + u32 reg; + + ret =3D smmu_wait(false, + (readl_relaxed(smmu->base + ARM_SMMU_GBPA) & GBPA_UPDATE) =3D=3D 0); + if (ret) + return ret; + + reg =3D readl_relaxed(smmu->base + ARM_SMMU_GBPA); + writel_relaxed(GBPA_UPDATE | GBPA_ABORT | reg, smmu->base + ARM_SMMU_GBPA= ); + return smmu_wait(false, + (readl_relaxed(smmu->base + ARM_SMMU_GBPA) & GBPA_UPDATE) =3D=3D 0); +} + static bool smmu_cmdq_has_space(struct arm_smmu_queue *cmdq, u32 n) { struct arm_smmu_ll_queue *llq =3D &cmdq->llq; @@ -468,6 +484,10 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) if (ret) goto out_ret; =20 + ret =3D smmu_abort_gbpa(smmu); + if (ret) + goto out_ret; + return 0; =20 out_ret: @@ -756,10 +776,16 @@ static bool smmu_dabt_device(struct hyp_arm_smmu_v3_d= evice *smmu, val =3D smmu->host_ste_cfg; goto out_update_regs; } - /* Passthrough the register access for bisectiblity, handled later */ case ARM_SMMU_GBPA: - mask =3D read_write; - break; + if (len !=3D sizeof(u32)) + break; + + /* Ignore write, always read to abort. */ + if (!is_write) { + val =3D GBPA_ABORT; + goto out_update_regs; + } + goto out_ret; case ARM_SMMU_CR0: if (len !=3D sizeof(u32)) break; --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D739C3A4F34 for ; Fri, 1 May 2026 11:20:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634433; cv=none; b=l/JFzT81q0n7cOh8YIVhboKB7JyOpNZ/pPbLP2aI1Ovj/hCVaJ3DKgVx46qG9rzKb01PcNrubzv9KdxkizF7UJx9KVvp+w1D+U7EHO1L/ECNWRuCp7DofHee6Ngex0jeuMzU1FJWMHUodgtb4zo/tWeBGn3htcnBoQbBxtpvUp8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634433; c=relaxed/simple; bh=/cJ7AkplAzft5yBV8ZxNUAFnFWochsKTITaa04WSWs4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nj8sCWFQlWIeNYZbYBHsiV0GFaEchkojsk+RqH8Rjhiqzx7a2xpQLDuuKNPM/Ub8c70XhZBI6Bbenbk/0uliUIdWKl1QSTBn6a2/+nY3J0ekF4HG4OqD27+t2A4boG3k14a/spbUasq1LLD8rCHrjAYpXnXD2XK5EFrak6iX/u4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=e6S/a2jr; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="e6S/a2jr" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-43d780757eeso1296490f8f.1 for ; Fri, 01 May 2026 04:20:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634430; x=1778239230; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yTbFP/vzGSoWzHV54z7Mcha0ge1NRm8j7joYQ/l7s4Q=; b=e6S/a2jrNORQmjX2uPQ3cQ4gFUfr3D6gJZa2PHEMTfRS+oN2Jgji35VQEqqA/XvdV4 Ys/jTlEVC5LQkj9g8jAL/iRs48v8X2JQf4VPccDIbd6owBAOgvNtiJGsr8Ne8ECsfIfb GdioLrw9Nc3ANvgLrCeyEVnc2JL82D+srwiYS6hgcOFLao5dl9F/UXiKttWc0zKC0Scn 24GJMdqqmO6q8gSu++PnitCVSZV8xVesVr/h4jyXPSeaEcNGX8I3yWpPVPfTr7MGzquE gbasTZ6tZTnoIw3cdxFAIAire6b9Ul5kR/OBtry5RLyQyk+zOtjXzLafW9hygAChHY+a hzJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634430; x=1778239230; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yTbFP/vzGSoWzHV54z7Mcha0ge1NRm8j7joYQ/l7s4Q=; b=SKF58HO+hwCd923amudB/UohrBGvpQNnbUP/eHerjCATfKKx8vObgaPO/kFtKzcDja M2hp+xkwI624ymLXEI86ps9d9Epwf7oaQVTxqsrhwbbCeKUFeTn5UUncmnrWpicEik0J 8ySXbaZHzh7t/CA1iAAv5joP5oHDe4oP8MFaQYxxWRqByOkkXz5iR1TINOemPWGdsDmu IG/Png9s7COPRsA4SE8NquOV5kT9r7iSTzP5RAwVj0uvR/NPaigZorQSOdcyw7G78pXZ wKJAbQ6ZJr/BEXqR2zWAIZhWaPOXuwKf/RUYC3yccFHostasIqSNUPraOoau5oHqgTAn ufvA== X-Forwarded-Encrypted: i=1; AFNElJ9c3N9SNX27EfZmO1w+KeXu3GJiDbBK1BY8ZE2hdlS7LXm+7TDS2e1lzzDCItds4Ts4s18nYDq/lFjFQ34=@vger.kernel.org X-Gm-Message-State: AOJu0YzQwkqk6cFeVMIfeZTb99nMe9ORabOByLPc3jjVimROxXYap9hQ DIID4GxOiWw5uMoSMfDYZiTGBZE9RCE7g/Ru4pcOACa1xWE3KvY7DeOVwaXhNIo+Z+4WsjIgqNd l0qJeSd2XaKGOxg== X-Received: from wrxp8.prod.google.com ([2002:a05:6000:188:b0:449:9a82:a60d]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1862:b0:43d:779e:afa8 with SMTP id ffacd0b85a97d-4494f9a1430mr11063527f8f.16.1777634430023; Fri, 01 May 2026 04:20:30 -0700 (PDT) Date: Fri, 1 May 2026 11:19:24 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-23-smostafa@google.com> Subject: [PATCH v6 22/25] iommu/io-pgtable-arm: Support io-pgtable-arm in the hypervisor From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To support DMA isolation in pKVM through SMMUv3 nested trap and emulate, io-pgtable-arm needs to be compiled for the hypervisor to create the SMMUs page tables. Instead of factoring out the kernel-specific code and providing parallel implementations for the hypervisor, we directly utilize and abstract the differences within the iommu-pages API. This introduces a set of hypervisor-specific wrappers in iommu-pages.h when compiled under __KVM_NVHE_HYPERVISOR__, routing allocations, frees, virt/phys conversions, and DMA API mapping to the appropriate pKVM hypervisor functions (kvm_iommu_donate_pages, etc). The generic kernel definitions are now appropriately excluded in this case. It also introduces kvm_alloc_io_pgtable_ops at the end of io-pgtable-arm.c to instantiate the page table for pKVM and adds the io-pgtable-arm.o object to the hypervisor Makefile. Signed-off-by: Mostafa Saleh --- arch/arm64/kvm/hyp/nvhe/Makefile | 3 +- drivers/iommu/io-pgtable-arm.c | 31 +++++++++++++++- drivers/iommu/io-pgtable-arm.h | 6 +++ drivers/iommu/iommu-pages.h | 63 ++++++++++++++++++++++++++++++++ 4 files changed, 100 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Mak= efile index 8a75739db947..4e9e0f1ed2b5 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -36,7 +36,8 @@ hyp-obj-y +=3D $(lib-objs) HYP_SMMU_V3_DRV_PATH =3D ../../../../../drivers/iommu/arm/arm-smmu-v3 =20 hyp-obj-$(CONFIG_ARM_SMMU_V3_PKVM) +=3D $(HYP_SMMU_V3_DRV_PATH)/pkvm/arm-s= mmu-v3.o \ - $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o + $(HYP_SMMU_V3_DRV_PATH)/arm-smmu-v3-common-lib.o \ + $(HYP_SMMU_V3_DRV_PATH)/../../io-pgtable-arm.o =20 # Path to simple_ring_buffer.c CFLAGS_trace.nvhe.o +=3D -I$(srctree)/kernel/trace/ diff --git a/drivers/iommu/io-pgtable-arm.c b/drivers/iommu/io-pgtable-arm.c index e765021308f9..8a0ffea3ae2c 100644 --- a/drivers/iommu/io-pgtable-arm.c +++ b/drivers/iommu/io-pgtable-arm.c @@ -252,10 +252,14 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_= t gfp, void *cookie) { struct device *dev =3D cfg->iommu_dev; + int nid =3D NUMA_NO_NODE; size_t alloc_size; dma_addr_t dma; void *pages; =20 + if (dev) + nid =3D dev_to_node(dev); + /* * For very small starting-level translation tables the HW requires a * minimum alignment of at least 64 to cover all cases. @@ -264,8 +268,7 @@ static void *__arm_lpae_alloc_pages(size_t size, gfp_t = gfp, if (cfg->alloc) pages =3D cfg->alloc(cookie, alloc_size, gfp); else - pages =3D iommu_alloc_pages_node_sz(dev_to_node(dev), gfp, - alloc_size); + pages =3D iommu_alloc_pages_node_sz(nid, gfp, alloc_size); =20 if (!pages) return NULL; @@ -1262,3 +1265,27 @@ struct io_pgtable_init_fns io_pgtable_arm_mali_lpae_= init_fns =3D { .alloc =3D arm_mali_lpae_alloc_pgtable, .free =3D arm_lpae_free_pgtable, }; + +#ifdef __KVM_NVHE_HYPERVISOR__ +#include + +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie) +{ + struct io_pgtable *iop; + + if (fmt !=3D ARM_64_LPAE_S2) + return NULL; + + iop =3D arm_64_lpae_alloc_pgtable_s2(cfg, cookie); + if (!iop) + return NULL; + + iop->fmt =3D fmt; + iop->cookie =3D cookie; + iop->cfg =3D *cfg; + + return &iop->ops; +} +#endif diff --git a/drivers/iommu/io-pgtable-arm.h b/drivers/iommu/io-pgtable-arm.h index ba7cfdf7afa0..af3a3f1e765e 100644 --- a/drivers/iommu/io-pgtable-arm.h +++ b/drivers/iommu/io-pgtable-arm.h @@ -27,4 +27,10 @@ #define ARM_LPAE_TCR_PS_48_BIT 0x5ULL #define ARM_LPAE_TCR_PS_52_BIT 0x6ULL =20 +#ifdef __KVM_NVHE_HYPERVISOR__ +struct io_pgtable_ops *kvm_alloc_io_pgtable_ops(enum io_pgtable_fmt fmt, + struct io_pgtable_cfg *cfg, + void *cookie); +#endif + #endif /* IO_PGTABLE_ARM_H_ */ diff --git a/drivers/iommu/iommu-pages.h b/drivers/iommu/iommu-pages.h index e1945193ad7f..749f0f4f870c 100644 --- a/drivers/iommu/iommu-pages.h +++ b/drivers/iommu/iommu-pages.h @@ -10,6 +10,7 @@ #include #include =20 +#ifndef __KVM_NVHE_HYPERVISOR__ /** * struct ioptdesc - Memory descriptor for IOMMU page tables * @iopt_freelist_elm: List element for a struct iommu_pages_list @@ -181,4 +182,66 @@ static inline void iommu_pages_dma_unmap(struct device= *dev, dma_addr_t dma, siz dma_unmap_single(dev, dma, size, DMA_TO_DEVICE); } =20 +#else /* __KVM_NVHE_HYPERVISOR__ */ + +#include +#include + +static inline void *iommu_alloc_pages_node_sz(int nid, gfp_t gfp, size_t s= ize) +{ + return kvm_iommu_donate_pages(get_order(size)); +} + +static inline void iommu_free_pages(void *virt) +{ + kvm_iommu_reclaim_pages(virt); +} + +static inline void *iommu_alloc_data(size_t size, gfp_t gfp) +{ + return kvm_iommu_donate_pages(get_order(size)); +} + +static inline void iommu_free_data(void *p) +{ + kvm_iommu_reclaim_pages(p); +} + +#undef WARN_ONCE +#define WARN_ONCE(condition, format...) WARN_ON(condition) + +static inline phys_addr_t iommu_virt_to_phys(void *virt) +{ + return hyp_virt_to_phys(virt); +} + +static inline void *iommu_phys_to_virt(phys_addr_t phys) +{ + return hyp_phys_to_virt(phys); +} + +static inline void iommu_pages_flush_incoherent(struct device *dma_dev, + void *virt, size_t offset, + size_t len) +{ + kvm_flush_dcache_to_poc(virt + offset, len); +} + +static inline dma_addr_t iommu_pages_dma_map(struct device *dev, void *vir= t, size_t size) +{ + kvm_flush_dcache_to_poc(virt, size); + return (dma_addr_t)hyp_virt_to_phys(virt); +} + +static inline int iommu_pages_dma_mapping_error(struct device *dev, dma_ad= dr_t dma) +{ + return 0; +} + +static inline void iommu_pages_dma_unmap(struct device *dev, dma_addr_t dm= a, size_t size) +{ +} + +#endif /* __KVM_NVHE_HYPERVISOR__ */ + #endif /* __IOMMU_PAGES_H */ --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33CE43A63E3 for ; Fri, 1 May 2026 11:20:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634435; cv=none; b=ICsFq/uOcQA7KU1O/kMbn9Q9Psu0drzIJMoGV9VLK9CW9t1/xON2bZWuC0F3XUoIO0g15ZlsAwVdNXRJBN5LBOZtPAPyNelMFYXknlBCcX2wuLiBRs07KwOZZxQ9GoLZkubrPlXh7YwQp2D4LZuOD9rmoz1VXXIzUq6sFgvlniA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634435; c=relaxed/simple; bh=mlHokx8sO2Xw5g9TqHe+AADr1SoayKwnTHlfGZA9zeM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ljtgQYFO6lJ5klogiSfrWik4KtVysqdVZiISZy966x5gDj4CB++I+DcUL3CG5S1F2ZOC+h6lMZeBFMlwCRGE14wbvh0wGwcuGKp9VlnN7Y4+EYCPyhD/rU2BAR0DJLORLvGLc28isrJOsGojGvzo4GmrCEt8KbClZu6qRB5V9jg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QQHPmF1e; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QQHPmF1e" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-44a044cb7d9so783211f8f.1 for ; Fri, 01 May 2026 04:20:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634431; x=1778239231; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=w0w7aacCAKh5xDmIAde6/c4ma74UnnFyw9cMZFzY2vk=; b=QQHPmF1ejcLxKx2l4hLb0mjx5JJE0hH6zjbAg2OY42emDkEpmNeaGS21a3xcy9zGfS eacfLAjyNykKRErogKzIpxeFNW077UZrxfAnqeW9EcpWDY1r+5w1WvGuWJpl2ft+nFMS n41Zg+LnQkfZ+wuTSdOkq248urgd+7GKdTuqnXbKhPuy8J2Wm2o3UpegQ2kgTz/kafvF Fkcc53rtMjWyRMymZuc3fxM1pl41AE5FTvDcUvMCT9l+R9VWt1KvOvdikTUrfWTMOLj6 25P+VNjGm73hl21M+E+Ouk4xTgfaeC4kxJWuTtn5dcmCMbTrkl24rO7PpeXYeyONDAMD k6qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634431; x=1778239231; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=w0w7aacCAKh5xDmIAde6/c4ma74UnnFyw9cMZFzY2vk=; b=I3WuvrgB0s+krfosvvz3TjnRMKy6LwExAhznH/g/q8Z9mQq/LxG3/Sx12YJ2hyDF/p lrqieuCSUrdXNb0Kxud4reTHIZpvrkK/wp+qI0DTsih/n4Gr7KtGmF33PhD/NSTBCWZO zNMy8gxqwA+gQCR9Yrn97+3coiM2TihIX6os6Agu1ThA9nLHo3cA4qcaQgTc+wVNyjCe ee0v0vC+eRN+7Ytv8qLuMBBylohfrx9nK5+MwH449uDLBt+YNaEkP5PuuE+CwGszIoXG grkHM+F0h6yqYskcA+TwkK2MbJT1kg9X89sm4+kd7G3mGtIlbKsGhal0EmMixWOyQKhL sdsg== X-Forwarded-Encrypted: i=1; AFNElJ8oNjy0L0XwYqU2V9DeMNnFHt2qaQlGjiDVgw3gkCp0L77NBvln3qTfgsKK96YLR0l7ofGwbwTItRcwuAY=@vger.kernel.org X-Gm-Message-State: AOJu0Yy6+0bK46KgyKiW7adWsfHDBBN53I1dCU4WDVuTKnbNPF/jQjgB vvy0OhIXpz59bSHFcxw1oGKj5hrnbpTLN6Dg0G5SSpPPmYtAz+1KbMWF4v2AtcR2/iGlgwmrksw POWjJexiq9LCvoA== X-Received: from wroq15.prod.google.com ([2002:adf:f50f:0:b0:44a:c22c:e636]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a316:b0:48a:58ae:9933 with SMTP id 5b1f17b1804b1-48a8eb8b706mr26555135e9.18.1777634431501; Fri, 01 May 2026 04:20:31 -0700 (PDT) Date: Fri, 1 May 2026 11:19:25 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-24-smostafa@google.com> Subject: [PATCH v6 23/25] iommu/arm-smmu-v3-kvm: Shadow the CPU stage-2 page table From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Based on the callbacks from the hypervisor, update the SMMUv3 Identity mapped page table. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 197 +++++++++++++++++- 1 file changed, 195 insertions(+), 2 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index 1ed5ccce7849..b73a2462f0dd 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -13,6 +13,9 @@ =20 #include "arm_smmu_v3.h" =20 +#include +#include "../../../io-pgtable-arm.h" + size_t __ro_after_init kvm_hyp_arm_smmu_v3_count; struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; =20 @@ -59,6 +62,9 @@ struct hyp_arm_smmu_v3_device *kvm_hyp_arm_smmu_v3_smmus; __ret; \ }) =20 +/* Protected by host_mmu.lock from core code. */ +static struct io_pgtable *idmap_pgtable; + static bool is_cmdq_enabled(struct hyp_arm_smmu_v3_device *smmu) { return FIELD_GET(CR0_CMDQEN, smmu->cr0); @@ -210,7 +216,6 @@ static int smmu_sync_cmd(struct hyp_arm_smmu_v3_device = *smmu) smmu_cmdq_empty(&smmu->cmdq)); } =20 -__maybe_unused static int smmu_send_cmd(struct hyp_arm_smmu_v3_device *smmu, struct arm_smmu_cmdq_ent *cmd) { @@ -222,6 +227,78 @@ static int smmu_send_cmd(struct hyp_arm_smmu_v3_device= *smmu, return smmu_sync_cmd(smmu); } =20 +static void __smmu_add_cmd(void *__opaque, struct arm_smmu_cmdq_batch *unu= sed, + struct arm_smmu_cmdq_ent *cmd) +{ + struct hyp_arm_smmu_v3_device *smmu =3D (struct hyp_arm_smmu_v3_device *)= __opaque; + + WARN_ON(smmu_add_cmd(smmu, cmd)); +} + +static int smmu_tlb_inv_range_smmu(struct hyp_arm_smmu_v3_device *smmu, + struct arm_smmu_cmdq_ent *cmd, + unsigned long iova, size_t size, size_t granule) +{ + arm_smmu_tlb_inv_build(cmd, iova, size, granule, + PAGE_SHIFT, smmu->features & ARM_SMMU_FEAT_RANGE_INV, + smmu, __smmu_add_cmd, NULL); + return smmu_sync_cmd(smmu); +} + +static void smmu_tlb_inv_range(unsigned long iova, size_t size, size_t gra= nule, + bool leaf) +{ + struct arm_smmu_cmdq_ent cmd_s1 =3D { + .opcode =3D CMDQ_OP_TLBI_NH_ALL, + .tlbi =3D { + .vmid =3D 0, + }, + }; + struct hyp_arm_smmu_v3_device *smmu; + + for_each_smmu(smmu) { + struct arm_smmu_cmdq_ent cmd =3D { + .opcode =3D CMDQ_OP_TLBI_S2_IPA, + .tlbi =3D { + .leaf =3D leaf, + .vmid =3D 0, + }, + }; + + hyp_spin_lock(&smmu->lock); + /* + * Don't bother if SMMU is disabled, this would be useful for the case + * when RPM is supported to avoid touching the SMMU MMIO when disabled. + * The hypervisor also asserts CMDQEN is enabled before the SMMU is + * enabled. As otherwise the host can prevent the hypervisor from doing + * TLB invalidations. + */ + if (is_smmu_enabled(smmu)) { + WARN_ON(smmu_tlb_inv_range_smmu(smmu, &cmd, iova, size, granule)); + WARN_ON(smmu_send_cmd(smmu, &cmd_s1)); + } + hyp_spin_unlock(&smmu->lock); + } +} + +static void smmu_tlb_flush_walk(unsigned long iova, size_t size, + size_t granule, void *cookie) +{ + smmu_tlb_inv_range(iova, size, granule, false); +} + +static void smmu_tlb_add_page(struct iommu_iotlb_gather *gather, + unsigned long iova, size_t granule, + void *cookie) +{ + smmu_tlb_inv_range(iova, granule, granule, true); +} + +static const struct iommu_flush_ops smmu_tlb_ops =3D { + .tlb_flush_walk =3D smmu_tlb_flush_walk, + .tlb_add_page =3D smmu_tlb_add_page, +}; + /* Put the device in a state that can be probed by the host driver. */ static void smmu_deinit_device(struct hyp_arm_smmu_v3_device *smmu) { @@ -495,6 +572,37 @@ static int smmu_init_device(struct hyp_arm_smmu_v3_dev= ice *smmu) return ret; } =20 +static int smmu_init_pgt(void) +{ + /* Default values overridden based on SMMUs common features. */ + struct io_pgtable_cfg cfg =3D (struct io_pgtable_cfg) { + .tlb =3D &smmu_tlb_ops, + .pgsize_bitmap =3D -1, + .ias =3D 48, + .oas =3D 48, + .coherent_walk =3D true, + }; + struct hyp_arm_smmu_v3_device *smmu; + struct io_pgtable_ops *ops; + + for_each_smmu(smmu) { + cfg.ias =3D min(cfg.ias, smmu->oas); + cfg.oas =3D min(cfg.oas, smmu->oas); + cfg.pgsize_bitmap &=3D smmu->pgsize_bitmap; + cfg.coherent_walk &=3D !!(smmu->features & ARM_SMMU_FEAT_COHERENCY); + } + + /* At least PAGE_SIZE must be supported by all SMMUs*/ + if ((cfg.pgsize_bitmap & PAGE_SIZE) =3D=3D 0) + return -EINVAL; + + ops =3D kvm_alloc_io_pgtable_ops(ARM_64_LPAE_S2, &cfg, NULL); + if (!ops) + return -ENOMEM; + idmap_pgtable =3D io_pgtable_ops_to_pgtable(ops); + return 0; +} + /* Called while is the host is still trusted. */ static int smmu_init(void) { @@ -520,7 +628,10 @@ static int smmu_init(void) =20 BUILD_BUG_ON(sizeof(hyp_spinlock_t) !=3D sizeof(u32)); =20 - return 0; + ret =3D smmu_init_pgt(); + if (ret) + goto out_reclaim_smmu; + return ret; =20 out_reclaim_smmu: while (smmu !=3D kvm_hyp_arm_smmu_v3_smmus) @@ -950,8 +1061,90 @@ static bool smmu_dabt_handler(struct user_pt_regs *re= gs, u64 esr, u64 addr) return false; } =20 +static size_t smmu_pgsize_idmap(size_t size, u64 paddr, size_t pgsize_bitm= ap) +{ + size_t pgsizes; + + /* Remove page sizes that are larger than the current size */ + pgsizes =3D pgsize_bitmap & GENMASK_ULL(__fls(size), 0); + + /* Remove page sizes that the address is not aligned to. */ + if (likely(paddr)) + pgsizes &=3D GENMASK_ULL(__ffs(paddr), 0); + + WARN_ON(!pgsizes); + + /* Return the largest page size that fits. */ + return BIT(__fls(pgsizes)); +} + static int smmu_host_stage2_idmap(phys_addr_t start, phys_addr_t end, int = prot) { + size_t pgsize =3D PAGE_SIZE, pgcount, size; + struct io_pgtable *pgtable =3D idmap_pgtable; + int ret =3D 0; + + end =3D min(end, BIT(pgtable->cfg.oas)); + if (start >=3D end) + return 0; + + size =3D end - start; + if (prot) { + size_t mapped; + + if (!(prot & IOMMU_MMIO)) + prot |=3D IOMMU_CACHE; + + while (size) { + mapped =3D 0; + /* + * We handle pages size for memory and MMIO differently: + * - memory: Map everything with PAGE_SIZE, that is guaranteed to + * find memory as we allocated enough pages to cover the entire + * memory, we do that as io-pgtable-arm doesn't support + * split_blk_unmap logic any more, so we can't break blocks once + * mapped to tables. + * - MMIO: Unlike memory, pKVM allocate 1G to for all MMIO, while + * the MMIO space can be large, as it is assumed to cover the + * whole IAS that is not memory, we have to use block mappings, + * that is fine for MMIO as it is never donated at the moment, + * so we never need to unmap MMIO at the run time triggereing + * split block logic. + */ + if (prot & IOMMU_MMIO) + pgsize =3D smmu_pgsize_idmap(size, start, pgtable->cfg.pgsize_bitmap); + + pgcount =3D size / pgsize; + ret =3D pgtable->ops.map_pages(&pgtable->ops, start, start, + pgsize, pgcount, prot, 0, &mapped); + size -=3D mapped; + start +=3D mapped; + /* Map failures doesn't impact security, tolerate it. */ + if (!mapped || ret) + break; + } + } else { + struct iommu_iotlb_gather gather; + size_t unmapped; + + while (size) { + pgcount =3D size / pgsize; + iommu_iotlb_gather_init(&gather); + unmapped =3D pgtable->ops.unmap_pages(&pgtable->ops, start, + pgsize, pgcount, &gather); + size -=3D unmapped; + start +=3D unmapped; + if (!unmapped) + break; + } + } + + if (ret) + return ret; + + if (WARN_ON(size)) + return -EINVAL; + return 0; } =20 --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFC643A6F15 for ; Fri, 1 May 2026 11:20:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634438; cv=none; b=TSu4yq9FsAQd5EpAXE26vrQtNHcAbmP4ZVVdBUN0UeaItlYN5t6NkwLZZ5juHhumxGmXY5Y0HKWBj0NXKMziJwiaVyix16tM5+6V+umFEHPirWbMXNwC8hsmUxgNL+S6+EiCeeOyk/f+O52rn0kvJWXITdVcNrflzdGsVGzkw6U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634438; c=relaxed/simple; bh=K0ptwM46SUnUhZWwfceVEOB6BIJZjEKLOxtcjcGQS/U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kbVY4uHOowiHRejrVNnCod+DVHooMsD6zGIvV+FFdAL9KLCCquY2Gb2boFDPK3HAvVbBSbi5dWbHxnDZiDNl3bzDMnPHebZZBl9TxTdNdBsl/3/W65ncypZi/pvJuI02TZgbI1JqKAh23jqF7L1nwHHvj23ROz9kERVybh2VHs8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZptM7gQ9; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZptM7gQ9" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4837b6f6b93so21239985e9.3 for ; Fri, 01 May 2026 04:20:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634434; x=1778239234; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cBYFLa+FC4uonW8niogdkcZuL7rcf529F6haDy+QJc8=; b=ZptM7gQ9DQvKX0kG1Jj+Y6RyAnWQMqzPgi4el7rmPKmC/ScqrwNqwTWC7jbZyJwbiI XU/AeYwsojbXQhtQxN+LVzG0Q794I934rzFwj7taHPs0Prrv41HrfF5i6VYPK4Ia8Ng3 8yp50uk+IS7QmG7sbHHynmGtt/6My9bF4bAG0mb8IIPcDikfvioU4qBzwcwmKp4cJT8G HPybqVDDNIgaEnDHHDT+DrXlS/q8w91O0ST9T6vRlsdkg8uN2ROpNwlxYGCMCpny1O3f CVFGHt+yMXwjvHKIg0Bf2tmkCW/u2qBP0dPIaOWfh04AGG4jmu7iTs3F+OIXJSyA0j/Y ORFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634434; x=1778239234; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cBYFLa+FC4uonW8niogdkcZuL7rcf529F6haDy+QJc8=; b=IREvIcKBfu5eq1xvGCjuGtLSQSbxiGLVGFwD/27RDmywXJ21V7y6gmmXW3PM0mmqyZ RGI5M1VE0MSCZLexBmDBy4spJhBHYBtVFjm1e6jarogfYDIxgRDlzRF1XZ1iH96CDTBu uce1x1wb5JbVhpU8mZOZcvp46VXG0+pxomcOJ7+Asibo0UQJZCeQLqPyTOD2oIbfUJ/D nNPIDPddluMxelVXIWySa1Q8yt6Ty0CKPajac9J/I5ViKZ/MTWfyhKE4Wkq5olW7HCSl SzvmDB7HuFoEqShp8q/qWlGMU/HVPXX5O/2tQM3mpKfkNfKse9bZeznX1NdBS8TOOSzD vSIA== X-Forwarded-Encrypted: i=1; AFNElJ82/E1n8GQ8ylrwXGWlTMRvXDMBBWfSiGl5bmgiRnRNvGZynxfJa3s+JgqcYPntDdTADsC4wigSYAHVYAY=@vger.kernel.org X-Gm-Message-State: AOJu0YyJh9P1QrH0paudR35g62gHQPkIQIAVG4qPGr0yBRBEJL+TLiHi aWiYkHhgEWp/Va7yWpkfQXdFdQldJv/cBNnQ9QNusDdAqAw7aYovJ0MvBY0yT8Ea5UJYFFHUkUz FSrpbionLWM5JNQ== X-Received: from wmrk12.prod.google.com ([2002:a05:600c:b4c:b0:489:1dcf:1b38]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4588:b0:488:ab1d:dcc5 with SMTP id 5b1f17b1804b1-48a8447f4e9mr111938855e9.27.1777634434360; Fri, 01 May 2026 04:20:34 -0700 (PDT) Date: Fri, 1 May 2026 11:19:26 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-25-smostafa@google.com> Subject: [PATCH v6 24/25] iommu/arm-smmu-v3-kvm: Enable nesting From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now, as the hypervisor controls the command queue, stream table, and shadows the stage-2 page table. Enable stage-2 in case the host puts an STE in bypass or stage-1. Signed-off-by: Mostafa Saleh --- .../iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c | 108 ++++++++++++++++-- 1 file changed, 101 insertions(+), 7 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c b/drivers/iom= mu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c index b73a2462f0dd..3d727d6dfbf0 100644 --- a/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/pkvm/arm-smmu-v3.c @@ -411,6 +411,57 @@ static int smmu_init_cmdq(struct hyp_arm_smmu_v3_devic= e *smmu) return 0; } =20 +static int smmu_attach_stage_2(struct arm_smmu_ste *ste) +{ + unsigned long vttbr; + unsigned long ts, sl, ic, oc, sh, tg, ps; + unsigned long cfg; + struct io_pgtable_cfg *pgt_cfg =3D &idmap_pgtable->cfg; + + cfg =3D FIELD_GET(STRTAB_STE_0_CFG, le64_to_cpu(ste->data[0])); + if (!FIELD_GET(STRTAB_STE_0_V, le64_to_cpu(ste->data[0])) || + (cfg =3D=3D STRTAB_STE_0_CFG_ABORT)) { + ste->data[2] =3D 0; + ste->data[3] =3D 0; + return 0; + } + /* S2 is not advertised, that should never be attempted. */ + if (cfg =3D=3D STRTAB_STE_0_CFG_NESTED) + return -EINVAL; + vttbr =3D pgt_cfg->arm_lpae_s2_cfg.vttbr; + ps =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.ps; + tg =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tg; + sh =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sh; + oc =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.orgn; + ic =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.irgn; + sl =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.sl; + ts =3D pgt_cfg->arm_lpae_s2_cfg.vtcr.tsz; + + ste->data[1] &=3D ~cpu_to_le64(STRTAB_STE_1_SHCFG); + ste->data[1] |=3D cpu_to_le64(FIELD_PREP(STRTAB_STE_1_SHCFG, STRTAB_STE_1= _SHCFG_INCOMING)); + + /* The host shouldn't write dwords 2 and 3, overwrite them. */ + ste->data[2] =3D cpu_to_le64(FIELD_PREP(STRTAB_STE_2_VTCR, + FIELD_PREP(STRTAB_STE_2_VTCR_S2PS, ps) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2TG, tg) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SH0, sh) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2OR0, oc) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2IR0, ic) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2SL0, sl) | + FIELD_PREP(STRTAB_STE_2_VTCR_S2T0SZ, ts)) | + FIELD_PREP(STRTAB_STE_2_S2VMID, 0) | + STRTAB_STE_2_S2AA64 | STRTAB_STE_2_S2R | + #ifdef __BIG_ENDIAN + STRTAB_STE_2_S2ENDI | +#endif + STRTAB_STE_2_S2PTW); + + ste->data[3] =3D cpu_to_le64(vttbr & STRTAB_STE_3_S2TTB_MASK); + /* Convert S1 =3D> nested and bypass =3D> S2 */ + ste->data[0] |=3D cpu_to_le64(FIELD_PREP(STRTAB_STE_0_CFG, cfg | BIT(1))); + return 0; +} + static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v3_device *smmu, u32 s= id, struct arm_smmu_ste *host_ste_out) { @@ -440,9 +491,18 @@ static int smmu_get_host_l2_ste(struct hyp_arm_smmu_v3= _device *smmu, u32 sid, static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_device *smmu, u32 sid,= bool leaf) { struct arm_smmu_strtab_cfg *cfg =3D &smmu->strtab_cfg; - struct arm_smmu_ste *hyp_ste_ptr, *host_ste_ptr, host_ste_copy; + struct arm_smmu_ste *hyp_ste_ptr; u64 *hyp_ste_base =3D strtab_hyp_base(smmu); - int ret; + struct arm_smmu_ste target =3D {}; + struct arm_smmu_cmdq_ent cfgi_cmd =3D { + .opcode =3D CMDQ_OP_CFGI_STE, + .cfgi =3D { + .sid =3D sid, + .leaf =3D true, + }, + }; + bool cur_valid, target_valid; + int i, ret; =20 /* * Linux only uses leaf =3D 1, when leaf is 0, we need to verify that this @@ -463,7 +523,7 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_dev= ice *smmu, u32 sid, bool return -E2BIG; =20 hyp_ste_ptr =3D &hyp_table[sid]; - host_ste_ptr =3D &host_table[sid]; + memcpy(target.data, host_table[sid].data, STRTAB_STE_DWORDS << 3); } else { struct arm_smmu_strtab_l1 *l1tab =3D (struct arm_smmu_strtab_l1 *)hyp_st= e_base; u32 l1_idx =3D arm_smmu_strtab_l1_idx(sid); @@ -472,8 +532,7 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_dev= ice *smmu, u32 sid, bool if (l1_idx >=3D cfg->l2.num_l1_ents) return -E2BIG; =20 - host_ste_ptr =3D &host_ste_copy; - ret =3D smmu_get_host_l2_ste(smmu, sid, host_ste_ptr); + ret =3D smmu_get_host_l2_ste(smmu, sid, &target); if (ret) return ret; =20 @@ -491,9 +550,44 @@ static int smmu_reshadow_ste(struct hyp_arm_smmu_v3_de= vice *smmu, u32 sid, bool hyp_ste_ptr =3D &l2ptr->stes[arm_smmu_strtab_l2_idx(sid)]; } =20 - memcpy(hyp_ste_ptr->data, host_ste_ptr->data, STRTAB_STE_DWORDS << 3); =20 - return 0; + /* + * Summary of each host emulated state vs real HW. + * | Host | HW | + * =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D + * | V=3D0 | V=3D0 | + * | Abort | Abort | + * | Bypass | S2 | + * | S1 | S1+S2 | + * + * For the host, any V=3D0 transition is not hitless, all other permutati= ons of + * (abort, bypass, S1) transitions are hitless. + * For the HW state, any V=3D0 transition is not hitless, as all the S2 c= onfig is + * always the same (ttbr, vtcr...), all other transitions should be hitle= ss too. + * However, the host is not trusted, which means that any V=3D0 <=3D> V= =3D1 transitions + * we need to enforce writing order of the STE and add CFGI. + */ + cur_valid =3D FIELD_GET(STRTAB_STE_0_V, le64_to_cpu(hyp_ste_ptr->data[0])= ); + ret =3D smmu_attach_stage_2(&target); + if (ret) + return ret; + target_valid =3D FIELD_GET(STRTAB_STE_0_V, le64_to_cpu(target.data[0])); + if (cur_valid && !target_valid) { + WRITE_ONCE(hyp_ste_ptr->data[0], target.data[0]); + WARN_ON(smmu_send_cmd(smmu, &cfgi_cmd)); + for (i =3D 1; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], target.data[i]); + } else if (!cur_valid && target_valid) { + for (i =3D 1; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], target.data[i]); + WARN_ON(smmu_send_cmd(smmu, &cfgi_cmd)); + WRITE_ONCE(hyp_ste_ptr->data[0], target.data[0]); + } else { + for (i =3D 0; i < STRTAB_STE_DWORDS; i++) + WRITE_ONCE(hyp_ste_ptr->data[i], target.data[i]); + } + + return smmu_send_cmd(smmu, &cfgi_cmd); } =20 static int smmu_init_strtab(struct hyp_arm_smmu_v3_device *smmu) --=20 2.54.0.545.g6539524ca2-goog From nobody Sun Jun 14 07:34:45 2026 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C96D63A7F5E for ; Fri, 1 May 2026 11:20:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634439; cv=none; b=Bofp6qiFgBqku+sRYtRjr7/5nMfv49zvAC/rKTIXmFXWpVIkkJ6h8g9z0hg7CjT4UPCOX1j1WhcQ7gZhznBN6bgTgDdJeLNbNaGDEeM1Wor91yFCpTfOeMlvMCoV60TgQnIolPwdICoikbTPFba+ecTWxs/ASw8omCAa+qarevk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777634439; c=relaxed/simple; bh=hDWBK5NoJKy+6Kx7m6iUjizTKOjBwxg/U8Mh0br2RZY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VcAcPslipsKSAF0WBGdFdfz8C1J9ymGcR6gonklXPQxZxw8XAYGYuGmpELkOUXcjxmaH9PR4AVq6iohO6oM8rgdmAz+8OsbKpBWgwYto2uN3+6mFOnk++qjlKn2HoNmh+CQ49vkEJtYIGilOkb89YaR0aB/deWal7cMyxxeKA04= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=v34sG9AW; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--smostafa.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="v34sG9AW" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-48a5523fd53so8377275e9.3 for ; Fri, 01 May 2026 04:20:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1777634436; x=1778239236; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vwSpZfMSAme8K1/Tu2WzVQA/LKpXv4XDkU/OYSLP8lY=; b=v34sG9AWSMziAr9If+EIs9W35Yqw+0S7iSiNv/AwuraTIkL3cMbWrC4br/0rfZZxSU waohtGOMDUnZr1FHwSQdCXbFUgD86IbOEEmvPSC5Z/MTpgeGvIPSTZ1A6K8ugCp8PzRl B2ATmC/HN1Et5nS6Si3YvUCr0ilOodpTARcq/o6RICcVPlqHzdFUwCZlkXjpy1AyPgRC rEzRIKMUTQb5Uhe+IUTpZQSMGAsqIDaBNuy2qVLdJxpWtBNqmMUgxJf0tGWTkyuh+wXH ro94qN3c8ReZEplSJl7/rsVyxVaKdPv+jDdjOuIVdZSjWB5cWjYJHLVW2q0WBk8JKTO3 REpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1777634436; x=1778239236; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vwSpZfMSAme8K1/Tu2WzVQA/LKpXv4XDkU/OYSLP8lY=; b=gVkGDUFSy4wj8ndJQVJkns5+bXsabnJi+hw+/54Tn2kCiZ8RQTjIaJXCYhBEPu+3Eq CoLdxxmV8ClKBe6PkBTRJjSaecKMWJiDYUPVxhWCCKrj/xUZw5Py23G1kSTSDpl4Ihtv b8IQOKOmabYX9SJrcvlvvfFaGH5GDZKETZIsMVepHRfp1dOOj6rHEzJMEAGfkV+YYSdi oOQRHSCsSBm1TSe228gKi3oYOvaTwhRyAi4GfWfQVWkBEQqXoua0Ts5AEj7mPl1jazmN Mopoy+T+75QgydNMpfyUi/vIr9ijzKJ+h4SWmWSMjbnFm5h0eNKAoyLO6u0f8NQREswD BDhg== X-Forwarded-Encrypted: i=1; AFNElJ8SNL02+jSqUDh6mRWsXjrYWU2Fa+V4Uzg/f3TyFruHV2ALhlFEB5KQs9GblHaBx1nb8WW0iy1dgReUSyY=@vger.kernel.org X-Gm-Message-State: AOJu0Yx+6vPVR9k4lzCtq55QsWtxFsZE/jKP6uDf+XXH6FTOYz1UqSet +DSRfYCHB5ppMTA+eyFYFhvwl40Gwkv2nLbeYV+dU+G2Li8RI8V5asffhV37mWkli4lDMhtM6dm /V7uyhLVjqPNJ1Q== X-Received: from wmbjn2.prod.google.com ([2002:a05:600c:6b02:b0:489:d74d:f20]) (user=smostafa job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:64cd:b0:487:243f:dc3e with SMTP id 5b1f17b1804b1-48a83d6e17cmr115589325e9.6.1777634436116; Fri, 01 May 2026 04:20:36 -0700 (PDT) Date: Fri, 1 May 2026 11:19:27 +0000 In-Reply-To: <20260501111928.259252-1-smostafa@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260501111928.259252-1-smostafa@google.com> X-Mailer: git-send-email 2.54.0.545.g6539524ca2-goog Message-ID: <20260501111928.259252-26-smostafa@google.com> Subject: [PATCH v6 25/25] KVM: arm64: Add documentation for pKVM DMA isolation From: Mostafa Saleh To: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, iommu@lists.linux.dev Cc: catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, joro@8bytes.org, jean-philippe@linaro.org, jgg@ziepe.ca, mark.rutland@arm.com, qperret@google.com, tabba@google.com, vdonnefort@google.com, sebastianene@google.com, keirf@google.com, Mostafa Saleh Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Populate the section for DMA isolation in pKVM with the newly added KVM IOMMU and pKVM SMMUv3 driver details. Signed-off-by: Mostafa Saleh --- Documentation/virt/kvm/arm/pkvm.rst | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/Documentation/virt/kvm/arm/pkvm.rst b/Documentation/virt/kvm/a= rm/pkvm.rst index 514992a79a83..46e5c553646b 100644 --- a/Documentation/virt/kvm/arm/pkvm.rst +++ b/Documentation/virt/kvm/arm/pkvm.rst @@ -77,7 +77,24 @@ Status: **Unimplemented.** DMA isolation using an IOMMU ---------------------------- =20 -Status: **Unimplemented.** +Status: Supported for devices behind SMMUv3 supporting dual stages +of translation. + +With ``CONFIG_ARM_SMMU_V3_PKVM``, the hypervisor will take over the SMMUs +on the system and provide an architectural emulation to the kernel SMMUv3 +driver transparently. + +If some devices are not behind an IOMMU or behind another IOMMU architectu= re, +DMA isolation is not supported, as a driver must be provided for that. + +DMA isolation is enforced by dual stage of translation; similar to the CPU +where a driver can register their ops through ``kvm_iommu_register_driver`` +and implement ``host_stage2_idmap`` to shadow the CPU page table. + +This implementation trusts the systems firmware not to allow the untrusted +host kernel to bypass the SMMUv3. +For example by resetting the power. In that case, it is the firmware +responsibility to save/restore the SMMUv3 state. =20 Proxying of Trustzone services ------------------------------ --=20 2.54.0.545.g6539524ca2-goog