From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24CE33002B6; Fri, 17 Jul 2026 18:05:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311514; cv=none; b=aqpOt8XZ1qrqjVQ4Yb3cZnHpz+BpukJ1o8b6Y5hFTLLEMfQbj6klAit5ykIDtyPlumU8PSX6zlg6JxteKnF5WmrC1GQxFrqVc/lg16IENFkICJLhKwwzDTxvHntI/GEp4m5eJijSdT+l078BSh7FbWX7Icex4I3vBGXExPqFbiM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311514; c=relaxed/simple; bh=moMl3E65h89Fm072BiQDDw0MZgkDGYQCCZHgVYMQCWg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eSSNoEI9BESC8+OOazYCQxk8cyAyNyzyP614mcOUMx998vnpYXRgAMEAGNDKttQM7aNhXTaJwREnr5K7keNcN5GDG1+FO4t3Xd/GM8D8CbsFYeUFdrVVz8S1SEvZrbqfA+vo3oPXpma/cNcz04tO60GMyNmabe9JinBvDgcJEKc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MzFNHvro; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MzFNHvro" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A7CF91F00A3A; Fri, 17 Jul 2026 18:05:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311512; bh=bAt2B2DVBX4/RIpHNlO5/JDoLwNz0B6Ol+c72/5t3UA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=MzFNHvroNpxLssZqtDE/XzisYnpfZTw7q6hhmN2FsH0HM/1K/Ck03+b4yToNC8BFM e2nA1/X53VCAdYAxXw0SpVBiFy+pf7EIeTE2TpOdwjEH/ygihfaxBJCDyTzPtLrb1g YolAHpsYonN2KOcRj8XToi826MiewgqkOV/JdJjq4YkJKobLWq1dhIwERvSGWisdNG 9ybxIL7Jy3VwIv4BiWyZXwMg6tO3CS6lg5VMd8zQqB+TOKMAJLfYmHSNXlnX77aLLN gyP+uY2DSl4GnAX+BGqUfp6M3HsonmcxJW7+SCrE4u9yROwK9SY8UzlKT2Ca0XN6rk s4/02TpbIKBsg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, stable@vger.kernel.org, Michael Kelley , Jason Gunthorpe Subject: [PATCH v8 01/23] dma-direct: return struct page from dma_direct_alloc_from_pool() Date: Fri, 17 Jul 2026 23:34:19 +0530 Message-ID: <20260717180442.110954-2-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool helper") changed dma_direct_alloc_from_pool() to return the CPU address from dma_alloc_from_pool(). That fits dma_direct_alloc(), but dma_direct_alloc_pages() also uses the helper and expects a struct page *. Fix this by making dma_direct_alloc_from_pool() return the struct page * again, and pass the CPU address back through an out-parameter for the dma_direct_alloc() caller. Fixes: 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool h= elper") Cc: stable@vger.kernel.org Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Jason Gunthorpe Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index d8219efe3273..363d984d90e7 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -164,22 +164,21 @@ static bool dma_direct_use_pool(struct device *dev, g= fp_t gfp) return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev); } =20 -static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp) +static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t = size, + dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp) { struct page *page; u64 phys_limit; - void *ret; =20 if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL))) return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok); + page =3D dma_alloc_from_pool(dev, size, cpu_addr, gfp, dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); - return ret; + return page; } =20 static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size, @@ -247,8 +246,11 @@ void *dma_direct_alloc(struct device *dev, size_t size, * the atomic pools instead if we aren't allowed block. */ if ((remap || force_dma_unencrypted(dev)) && - dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + dma_direct_use_pool(dev, gfp)) { + page =3D dma_direct_alloc_from_pool(dev, size, dma_handle, + &ret, gfp); + return page ? ret : NULL; + } =20 /* we always manually zero the memory once we are done */ page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); @@ -357,7 +359,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, void *ret; =20 if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); =20 page =3D __dma_direct_alloc_pages(dev, size, gfp, false); if (!page) --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA005340417; Fri, 17 Jul 2026 18:05:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311525; cv=none; b=g2bwfRUXL5yNvOilnHCOtSHcIwdiXXA0wwJwCJYMSjy5paxKk7gmoemLhAsr3xeqfYnBumwXuQ18jIx2piTGfEScU8cQ1n4vr1Op2Opg5QwRerYAOTxh2N/BDGKCoUl6Fr6f6+WKP7z5j9UIPsh2q2LgQizzi/HshVUPjt+VvF8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311525; c=relaxed/simple; bh=wDumvzkXnSiFdlhTUSh4W5hc8/0cvR4MLjYrq2D2y5I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Un0mK7+sk6eb8WQqFn+K/IHdAWMRwT1KlcP3rdpqUJ2TPmFvO1KVKw9KfKoCA6wE+G0/EsmKBvnW1vO26/FUcv7AwK0+BZUQQ4lsQd5ft/ca8rwJuIUviWwQU79b+rNUT2rb1kmPdFoZ7/nXQK4YdEMV+oUb/uh+dLJxhaAwSKo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TuVQu0Md; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TuVQu0Md" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A20961F000E9; Fri, 17 Jul 2026 18:05:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311524; bh=Ss9PM/006K6SV5h+g0Mm+5y5zthxbL2BqyOfIndeCeE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=TuVQu0MdtCse0JkcTEXH2Lq3N8GnmxJYSHQ6CoRTHDltevRiIULT5pnqPM4+8+BAQ /JYVDAwN6m05+4yZnuZwY+K+Uvh7axJ4AmMsAJCMR7dGUeQuyQp75bJT0fW4GwPxsr r2m0krA+7xFAwrpSKW3D1wGN0ak4798EK1c2pnF1nE1AxAqM5JU7p2Y+uwKAtlzr/3 0SP8yoL0OntLzCWzdi5O4R036tEdejgeaZDyzHmy7KzUiLH+vr0e7bLicKBgrzgUFT QWcdjpVZw+U9E+w0lbFwqX/3IGUvZJvLJDX+7Ctr3NP0ikMrI2SChaP0sOzNHVVMfQ j3ClE0BUZz12A== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 02/23] dma-pool: fix page leak in atomic_pool_expand() cleanup Date: Fri, 17 Jul 2026 23:34:20 +0530 Message-ID: <20260717180442.110954-3-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" atomic_pool_expand() frees the allocated pages from the remove_mapping error path only when CONFIG_DMA_DIRECT_REMAP is enabled. When CONFIG_DMA_DIRECT_REMAP is disabled, failures after page allocation, such as gen_pool_add_virt(), jump to remove_mapping and return without freeing the pages. Move __free_pages(page, order) out of the CONFIG_DMA_DIRECT_REMAP block so that cleanup paths always release the allocation. Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/pool.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index 2b2fbb709242..b0303efbc153 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -81,6 +81,7 @@ static int atomic_pool_expand(struct gen_pool *pool, size= _t pool_size, { unsigned int order; struct page *page =3D NULL; + bool leak_pages =3D false; void *addr; int ret =3D -ENOMEM; =20 @@ -115,8 +116,10 @@ static int atomic_pool_expand(struct gen_pool *pool, s= ize_t pool_size, */ ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), 1 << order); - if (ret) + if (ret) { + leak_pages =3D true; goto remove_mapping; + } ret =3D gen_pool_add_virt(pool, (unsigned long)addr, page_to_phys(page), pool_size, NUMA_NO_NODE); if (ret) @@ -130,14 +133,15 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, 1 << order); if (WARN_ON_ONCE(ret)) { /* Decrypt succeeded but encrypt failed, purposely leak */ - goto out; + leak_pages =3D true; } remove_mapping: #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); free_page: - __free_pages(page, order); #endif + if (!leak_pages) + __free_pages(page, order); out: return ret; } --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA6EE32570D; Fri, 17 Jul 2026 18:05:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311537; cv=none; b=elneCjz5ese6NNQIJoJ9czTUuLl0hTgzjNSae9RUdPmdx33A5HDc0nSjYsTd95Pw6I4BSHfk4cCSaVeo3GmGBdaMIYRlThGVq4TF5URt2sAbRCiHL3cbxXLpikfFHGBFSeUwtNuIT2CncljIxVTlLiuX+9uXQpaj/RnysrH92HQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311537; c=relaxed/simple; bh=I+x2QPZqGU6QLJFjyfra/KkgEvRsmXqnm/8TIga5+kQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=U/PnGAlRLV2vLLHZn2z7QbsPaKbDqW+N3ub01sc1bq6nbiSVc8caORy4Aa3gNMV/X/QYVhvk86p8+XXZl0AMAIj/LiXQZHT85gTOxyfYxyTurQeO5cgdMkK7tdZGkXmMakHqBMMXLR0GC+riGKowePj3ipq44BJdA1ErVhtxo6Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PiVA0z2H; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PiVA0z2H" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 85BB81F00A3D; Fri, 17 Jul 2026 18:05:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311536; bh=6JAWb8OTjOFId3JccbNdiGr1f8/BydsRdPfwaJAJRw8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=PiVA0z2H4Snzd6lFwW/aQE3N8yGaK1f96qr9raHxxnqHLOyMqLswOfAV3OmODRrA7 KWUigEa1nQENHkk2fpPUawIZcoNGzZNukyqTGX742GZmzNW6aE5IMprZ53p8Gl0ReC PMiHzCvvyGrqH1+lS0m8KzVFNwiaw7wxbBglg+5j0c4X9d5VMAV2b8SiIO55sh8C1J rRegbQR7pFr11wRS4d3YTLrh7NmJTjul0aBRX04PN8ZcVlAgfLB053IG6duvKqH9/Q QbgfQb2XG57vWlQvnpcQYOqY+7NpdXzDTtJ0ZtvQWUXoQjjlNO9YmGnJeWLKkp7rqQ O0E8L88PGpcHA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 03/23] iommu/dma: Check atomic pool allocation result directly Date: Fri, 17 Jul 2026 23:34:21 +0530 Message-ID: <20260717180442.110954-4-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The non-blocking, non-coherent allocation path uses dma_alloc_from_pool(), which returns the allocated page and fills cpu_addr only on success. Do not rely on cpu_addr to detect allocation failure in this path. Check the returned page directly before using it for the IOMMU mapping. Fixes: 9420139f516d ("dma-pool: fix coherent pool allocations for IOMMU map= pings") Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 5598ed4bff72..6a45acbbfb0c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1671,13 +1671,16 @@ void *iommu_dma_alloc(struct device *dev, size_t si= ze, dma_addr_t *handle, } =20 if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && - !gfpflags_allow_blocking(gfp) && !coherent) + !gfpflags_allow_blocking(gfp) && !coherent) { page =3D dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, - gfp, NULL); - else + gfp, NULL); + if (!page) + return NULL; + } else { cpu_addr =3D iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); - if (!cpu_addr) - return NULL; + if (!cpu_addr) + return NULL; + } =20 *handle =3D __iommu_dma_map(dev, page_to_phys(page), size, ioprot, dev->coherent_dma_mask); --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8174D31714A; Fri, 17 Jul 2026 18:05:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311549; cv=none; b=RG02ag8iUEqDoAWLL4fHycSXrg6Ulpkg9Od034OGwhFvf00Zll9+4gxDew1mv5l43B9WZ7zOJ0iM9GSDoPKxKyM6Bc7oh6Dh0bIXm2sEHVAx4KCBOzDiDYTh9qqJZ6KLbN8PA4FffRSjdvL2G0eONDg2kE4bJ2utPMlnWx2E/M8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311549; c=relaxed/simple; bh=ooE2r7JwM/HGQIw48oIFNvbdsX7HgZSRQubrOtylgUM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ojZAG2j2P9nU1T+x5ylER6apQDZG7u48QFiFLBCzcl0cxfxqo6E8MUqZI9GgD9OAdjW9AqhhBNfnjX3SxrAESZZCg/yehWpMaK6MOVvAi4Vmzbk5Oz40is5euQIkBG411fyPZ9YGcaa83ivGKGxzmA5XAQnIBuAGvbKiW0dZrM8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oeF+Ifxp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oeF+Ifxp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 813691F000E9; Fri, 17 Jul 2026 18:05:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311548; bh=z6dDgE7JkOObtE2Hn1HKt3eeloSTK5VxIseLNnZOFlg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=oeF+IfxpYmSZu1u63a32cjs/L6r2e1rcv4bFhtQBcuR85N5csGMHhS2Zv4clRRRfo 166/0G3NoQm3IkkzhttzLiZor3pQyQuRBeIi7AUb6eNG/m9T26xDOD0U7IBM5q3JD2 KDj0cfreenJRdo5kM018PQi+WeqxvoKAsj6IjfGXWjwZFhBuo0Ovvq3OR5H73JZoBS AnPoFylIrnK6Oo6qmRifNa4hFKTXFTTxF0DVCZ4aRukA0AGbk+98JPvjtojnbw7dnm 4d3c38FgZhROoLkjtF1V6KXLi89DhnzsYwwQZ+rHP1+zvJyKLm4edWgJzrrUnBBTBT mpGs1kqSePKhA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v8 04/23] dma: free atomic pool pages by physical address Date: Fri, 17 Jul 2026 23:34:22 +0530 Message-ID: <20260717180442.110954-5-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" dma_direct_alloc_pages() may satisfy atomic allocations from the coherent atomic pools. The pool allocation is keyed by the virtual address stored in the gen_pool, but the pages API returns only the backing struct page. On architectures with CONFIG_DMA_DIRECT_REMAP, atomic pool chunks are added to the gen_pool using their remapped virtual address. dma_direct_free_pages() reconstructs a linear-map address with page_address(page) and passes that to dma_free_from_pool(). That address does not match the gen_pool virtual range, so the pool lookup can fail and the code can fall through to freeing a pool-owned page through the normal page allocator path. Add a page-based pool free helper that looks up the owning pool chunk by physical address, translates it back to the gen_pool virtual address, and frees that address to the pool. Use it from dma_direct_free_pages() while keeping the existing virtual-address helper for coherent allocation frees. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/dma-map-ops.h | 1 + kernel/dma/direct.c | 4 +-- kernel/dma/pool.c | 61 +++++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+), 2 deletions(-) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index bcb5b5428aea..137e015c1750 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -215,6 +215,7 @@ struct page *dma_alloc_from_pool(struct device *dev, si= ze_t size, void **cpu_addr, gfp_t flags, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)); bool dma_free_from_pool(struct device *dev, void *start, size_t size); +bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size); =20 int dma_direct_set_offset(struct device *dev, phys_addr_t cpu_start, dma_addr_t dma_start, u64 size); diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 363d984d90e7..a81597f8c3ed 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -381,9 +381,9 @@ void dma_direct_free_pages(struct device *dev, size_t s= ize, { void *vaddr =3D page_address(page); =20 - /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ + /* If page is not from an atomic pool, dma_free_from_pool_page() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && - dma_free_from_pool(dev, vaddr, size)) + dma_free_from_pool_page(dev, page, size)) return; =20 if (dma_set_encrypted(dev, vaddr, size)) diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index b0303efbc153..e981c1faaadf 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -311,3 +311,64 @@ bool dma_free_from_pool(struct device *dev, void *star= t, size_t size) =20 return false; } + +struct dma_pool_phys_match { + phys_addr_t phys; + size_t size; + unsigned long addr; + bool found; +}; + +static void dma_pool_find_phys(struct gen_pool *pool, struct gen_pool_chun= k *chunk, + void *data) +{ + struct dma_pool_phys_match *match =3D data; + phys_addr_t end =3D match->phys + match->size - 1; + phys_addr_t chunk_end; + + if (match->found) + return; + + chunk_end =3D chunk->phys_addr + (chunk->end_addr - chunk->start_addr); + if (match->phys < chunk->phys_addr || end > chunk_end) + return; + + match->addr =3D chunk->start_addr + (match->phys - chunk->phys_addr); + match->found =3D true; +} + +static bool dma_free_from_pool_phys(struct gen_pool *pool, phys_addr_t phy= s, + size_t size) +{ + struct dma_pool_phys_match match =3D { + .phys =3D phys, + .size =3D size, + }; + + gen_pool_for_each_chunk(pool, dma_pool_find_phys, &match); + if (!match.found) + return false; + + gen_pool_free(pool, match.addr, size); + return true; +} + +/* + * FIXME: We could avoid this by storing the remapped virtual address in + * struct page and using that for lookup. + */ +bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size) +{ + struct gen_pool *pool =3D NULL; + phys_addr_t phys =3D page_to_phys(page); + + if (!IS_ENABLED(CONFIG_DMA_DIRECT_REMAP)) + return dma_free_from_pool(dev, page_address(page), size); + + while ((pool =3D dma_guess_pool(pool, 0))) { + if (dma_free_from_pool_phys(pool, phys, size)) + return true; + } + + return false; +} --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3542A2E091E; Fri, 17 Jul 2026 18:06:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311561; cv=none; b=cYlKsO1l5O5E4ssI1Yw3wLpVhjY5JW6kNwOXdPfTHRc+3BdjHcYS1OZTTxqvBl4ylMM8Y5FnOuU3WOnOkF7VrDlQjJn4vlyCT3/FiHnUHgV62zMTle9UPPribfRXAndhgXEYLV7FVgTy+KZth6Px7ssH6OvD64j0mJNoiMCohtU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311561; c=relaxed/simple; bh=D0lIpCSFgTkUhg0uRhn6FKrfCBOuu3cAUsq0HEbsL0A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ENsKuBjYnOSIwxnaKvXSEYLWPTe9aRtKqIBn8RrixszErCTJ/3Iy1/JGeuvsiOlZhV20KMo/oGV9xjlzDZYTJhMdGzLR1MNSH9DPf6OIcSt2xLcTKEvl4HE3rzsiyAE96fmfeom9/2JYfC6hGL41/yL7bqi6DN10K1U2fNcg36I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SF5vvtJO; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SF5vvtJO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DA8261F00A3A; Fri, 17 Jul 2026 18:05:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311559; bh=A2xG7pWlX3DyXnh9sUcAESvn/UCw3OvXFeAcACc1di8=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=SF5vvtJO3IbG6SWDEmipvnsY5zIIKPP50kIEmreUNc+etmS+ZMoAWrFL1m2vhzKBE GmWBbg/bud2Wyo7Dwf91o5Z6k9jjSkd+KsHSejHMJhS0Pcxujm96j6y5ZE5UGVP2KW vcfvSblOJ7LEhbymOLpl8R+gkDSNRXUbzP9z2bh1zvMwVrCnHpHh8a/qXonVUZaQbl 7QBzqSiG+5JS8w9tuw2nAwGIG5napwdWBIvOCgjdD8Hdosi3yMghH+A4I6QazTK/Xg J/pBNma9rPOPgdR0nX1wj9vU9ODaSSv3pXdBbXng+bf/Yv+oPoEaTms2QqmDGqaG8l Zfum5zzJ3Sd8w== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 05/23] swiotlb: Preserve allocation virtual address for dynamic pools Date: Fri, 17 Jul 2026 23:34:23 +0530 Message-ID: <20260717180442.110954-6-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swiotlb_alloc_tlb() can allocate from the DMA atomic pool when a decrypted pool is needed from atomic context. With CONFIG_DMA_DIRECT_REMAP, the atomic pool is backed by remapped virtual addresses, which are not the same as the direct-map addresses returned by phys_to_virt(). swiotlb_init_io_tlb_pool() currently reconstructs the pool virtual address from the physical start address. For atomic-pool backed allocations this stores the wrong address in pool->vaddr. Later, swiotlb_free_tlb() passes that address to dma_free_from_pool(), which will fail to recognize the chunk Pass the virtual address returned by the allocation path into swiotlb_init_io_tlb_pool(), and store that address in pool->vaddr. This keeps the pool free path using the same virtual address as the allocator. Fixes: 79636caad361 ("swiotlb: if swiotlb is full, fall back to a transient= memory pool") Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/swiotlb.c | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1abd3e6146f4..6e8db52866bf 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -266,9 +266,9 @@ void __init swiotlb_update_mem_attributes(void) } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, - unsigned long nslabs, bool late_alloc, unsigned int nareas) + void *vaddr, unsigned long nslabs, bool late_alloc, + unsigned int nareas) { - void *vaddr =3D phys_to_virt(start); unsigned long bytes =3D nslabs << IO_TLB_SHIFT, i; =20 mem->nslabs =3D nslabs; @@ -409,7 +409,7 @@ void __init swiotlb_init_remap(bool addressing_limit, u= nsigned int flags, return; } =20 - swiotlb_init_io_tlb_pool(mem, __pa(tlb), nslabs, false, nareas); + swiotlb_init_io_tlb_pool(mem, __pa(tlb), tlb, nslabs, false, nareas); add_mem_pool(&io_tlb_default_mem, mem); =20 if (flags & SWIOTLB_VERBOSE) @@ -507,7 +507,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, =20 set_memory_decrypted((unsigned long)vstart, (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); - swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); =20 @@ -605,25 +605,26 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. * @gfp: GFP flags for the allocation. + * @vaddr: Receives the virtual address for the allocated buffer. * * Return: Allocated pages, or %NULL on allocation failure. */ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, - u64 phys_limit, gfp_t gfp) + u64 phys_limit, gfp_t gfp, void **vaddr) { struct page *page; =20 + *vaddr =3D NULL; + /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) { - void *vaddr; - if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 - return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, + return dma_alloc_from_pool(dev, bytes, vaddr, gfp, dma_coherent_ok); } =20 @@ -645,6 +646,8 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, return NULL; } =20 + if (page) + *vaddr =3D phys_to_virt(page_to_phys(page)); return page; } =20 @@ -685,6 +688,7 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, { struct io_tlb_pool *pool; unsigned int slot_order; + void *tlb_vaddr; struct page *tlb; size_t pool_size; size_t tlb_size; @@ -701,7 +705,8 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, pool->areas =3D (void *)pool + sizeof(*pool); =20 tlb_size =3D nslabs << IO_TLB_SHIFT; - while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp))) { + while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp, + &tlb_vaddr))) { if (nslabs <=3D minslabs) goto error_tlb; nslabs =3D ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -715,11 +720,12 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct = device *dev, if (!pool->slots) goto error_slots; =20 - swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), nslabs, true, nareas); + swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), tlb_vaddr, nslabs, + true, nareas); return pool; =20 error_slots: - swiotlb_free_tlb(page_address(tlb), tlb_size); + swiotlb_free_tlb(tlb_vaddr, tlb_size); error_tlb: kfree(pool); error: @@ -1851,7 +1857,8 @@ static int rmem_swiotlb_device_init(struct reserved_m= em *rmem, =20 set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), rmem->size >> PAGE_SHIFT); - swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs, + swiotlb_init_io_tlb_pool(pool, rmem->base, phys_to_virt(rmem->base), + nslabs, false, nareas); mem->force_bounce =3D true; mem->for_alloc =3D true; --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98F7527AC4C; Fri, 17 Jul 2026 18:06:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311573; cv=none; b=cLF4UCmWfgmZFKFoorqRMJZVbiB26DdXQJLjoOfa2IAbS1BMv5BRoZPip+OjiTCYZeWfZQ3VyTsAiOF7ybreAK5iMbIwW0t/5gA/j/lQbX6yPYMXGDqOPVftCVNAlkIva1BtCam49yTnzhI6PMtDTh750K44xpOV4B6doB3VWUU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311573; c=relaxed/simple; bh=j1TDEzVVIIA8q2Uvy+R53B4H2W7vI89z2yjA/W5mds4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WnWkdqgghxAB3lqbhZFGwf2+WNwzOcooUoGMSp+fR/cmtQiPLoBu8NzDR45/g6IFCBRqmTLETK/r1XgBLSq9P86H9CIPMhAg/gPBbRK8WuPDIQdmBNx5CAmMKVFeAzNILagRO7xuEgyh/Tm8DkAMwWNqaGjGlOfBvS3XX3f0MAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EERFj5zc; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EERFj5zc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BCEE21F000E9; Fri, 17 Jul 2026 18:06:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311571; bh=Rrxw0zEy8zZMWEqkMIDa1FzZShUbfKRclWdKlq0ShMQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=EERFj5zc2d1jA+hIVXVcVJggI1VJBh6HdI8oHQug+TwsCjCPDhkj8fgqKZNT5UKmQ skT7ficUTZLNY6S5Q3WuT+J3gDPyb5GnOK0eE4+99RLNKTOi0lmWjkSlC/FxYVNHnn q+TDBxiQ5oDyN2mWpC+pdJKRIjdGTidnIJ2DRnbkb/Wwweh43EfdUgHvUnqdvibl/B e/zfr+LG1u2pOOInDppAvmMm2gA8NijvsUlHIklzuhUedsUdhgRC1xjDdCSCqYxnZL whMwkwGTaERO5lLJjICrZdh5y2RbN8VXWq6eAWyMjpTj/NNgKByqrALdpvYGXDfxBD AzsJQx65teoug== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Halil Pasic , Matthew Rosato , Jaehoon Kim Subject: [PATCH v8 06/23] s390: Expose protected virtualization through cc_platform_has() Date: Fri, 17 Jul 2026 23:34:24 +0530 Message-ID: <20260717180442.110954-7-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protected virtualization guests use memory encryption, so advertise that to the rest of the kernel through cc_platform_has(CC_ATTR_MEM_ENCRYPT). s390 already forces DMA mappings to be unencrypted for protected virtualization guests through force_dma_unencrypted(). Add ARCH_HAS_CC_PLATFORM and provide the matching cc_platform_has() implementation Cc: Halil Pasic Cc: Matthew Rosato Cc: Jaehoon Kim Tested-by: Jaehoon Kim Signed-off-by: Aneesh Kumar K.V (Arm) --- arch/s390/Kconfig | 1 + arch/s390/mm/init.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 84404e6778d5..ab1bf55d197a 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -84,6 +84,7 @@ config S390 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CC_CAN_LINK + select ARCH_HAS_CC_PLATFORM select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index f07168a0d3dd..c29326c44af0 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -50,6 +50,7 @@ #include #include #include +#include =20 pgd_t swapper_pg_dir[PTRS_PER_PGD] __section(".bss..swapper_pg_dir"); pgd_t invalid_pg_dir[PTRS_PER_PGD] __section(".bss..invalid_pg_dir"); @@ -142,6 +143,19 @@ bool force_dma_unencrypted(struct device *dev) return is_prot_virt_guest(); } =20 + +bool cc_platform_has(enum cc_attr attr) +{ + switch (attr) { + case CC_ATTR_MEM_ENCRYPT: + return is_prot_virt_guest(); + + default: + return false; + } +} +EXPORT_SYMBOL_GPL(cc_platform_has); + /* protected virtualization */ static void __init pv_init(void) { --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8364A360ED1; Fri, 17 Jul 2026 18:06:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311584; cv=none; b=CgOjemObxks6F22YoX7R/Y1I3Yeoi1Zt0WZyuR+fj3ailFzp6vhBI0+lKbBMkjQOfEwksnQJuMXFOZsYj58zeoBfZV5kTggqpKHT6BX/KPGVTMfyPOuOaK+mxI5lhgqjkEbEoCpzStpqeS1az6RqE8jZj0EeCWgGQH8sRvEJQaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311584; c=relaxed/simple; bh=uUqTM7Ie/BfqNrHjzj/oZoRU0kK3FqMHnCCsb4Z75I8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=REx/m3HP9REVbiZ40h1zhFw1R9R/D+DQRg8zNvRiC0djd+W7WW2BEEFyVmSmobg319iRGh6QYgiS0+DlkDbLuFZ9BRKQQkua05sWGLYl3Mxd7CfG2rOQ0TxXXLh4fGHrfPfcuMOkmhUZx18vXCSfB8JMrWR4hGp/mayd/J4IH2w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VCzBaeJ2; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VCzBaeJ2" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F5DA1F00A3A; Fri, 17 Jul 2026 18:06:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311583; bh=R7yPnRK77euSH/3yJIYBaXB5NtWw567DUBYzFmJsmfU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=VCzBaeJ25/9yYTzNufWNBmVaP8pobM4k1dm5orkRszRYhZtCF/8/B4NznSrg5Hob7 NzY1RwSsCSrVOVIsh4pfBONit5MrW0k5jXEhaCsQYbmwfPBH1yVE7OFUFp7Ig2rXvm 0ehzuJyhqQyhIOW8K6gJrmQ/tALA+Hv1pemElQ++zsnqpqNk01YxTzGrH44OuqKt/v D5gFBBn7Zdm3+UA98L3NPk4R8VVSHYNc0c3NdsWgFUygakRaVuNYBVl9NLeMNUN/dA aJOq4lTrBSyxctOpuEQu6pKbweuEVd66ReKL/jZtYuhXFCTFAICQEhehBOUFdIt7cF 9DUyq0YZBfpYg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 07/23] dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages Date: Fri, 17 Jul 2026 23:34:25 +0530 Message-ID: <20260717180442.110954-8-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in dma_direct_alloc() / dma_direct_alloc_pages(). This is needed for follow-up changes that simplify the handling of memory encryption/decryption based on the DMA attribute flags. swiotlb backing pages are already mapped decrypted by swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so dma-direct should not call dma_set_decrypted() on allocation nor dma_set_encrypted() on free for swiotlb-backed memory. Update alloc/free paths to detect swiotlb-backed pages and skip encrypt/decrypt transitions for those paths. Keep the existing highmem rejection in dma_direct_alloc_pages() for swiotlb allocations. Only for "restricted-dma-pool", we currently set `for_alloc =3D true`, while rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is typically used together with "shared-dma-pool", where the shared region is accessed after remap/ioremap and the returned address is suitable for decrypted memory access. So existing code paths remain valid. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 6 ++++ kernel/dma/direct.c | 71 ++++++++++++++++++++++++++++++----------- kernel/dma/swiotlb.c | 6 ++++ 3 files changed, 65 insertions(+), 18 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 3dae0f592063..c92ff6791595 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -284,6 +284,8 @@ extern void swiotlb_print_info(void); #ifdef CONFIG_DMA_RESTRICTED_POOL struct page *swiotlb_alloc(struct device *dev, size_t size); bool swiotlb_free(struct device *dev, struct page *page, size_t size); +void swiotlb_free_from_pool(struct device *dev, + phys_addr_t tlb_addr, struct io_tlb_pool *pool); =20 static inline bool is_swiotlb_for_alloc(struct device *dev) { @@ -299,6 +301,10 @@ static inline bool swiotlb_free(struct device *dev, st= ruct page *page, { return false; } +static inline void swiotlb_free_from_pool(struct device *dev, + phys_addr_t tlb_addr, struct io_tlb_pool *pool) +{ +} static inline bool is_swiotlb_for_alloc(struct device *dev) { return false; diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index a81597f8c3ed..171c89fe99c9 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -96,14 +96,6 @@ static int dma_set_encrypted(struct device *dev, void *v= addr, size_t size) return ret; } =20 -static void __dma_direct_free_pages(struct device *dev, struct page *page, - size_t size) -{ - if (swiotlb_free(dev, page, size)) - return; - dma_free_contiguous(dev, page, size); -} - static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze) { struct page *page =3D swiotlb_alloc(dev, size); @@ -125,9 +117,6 @@ static struct page *__dma_direct_alloc_pages(struct dev= ice *dev, size_t size, =20 WARN_ON_ONCE(!PAGE_ALIGNED(size)); =20 - if (is_swiotlb_for_alloc(dev)) - return dma_direct_alloc_swiotlb(dev, size); - gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); page =3D dma_alloc_contiguous(dev, size, gfp); if (page) { @@ -203,6 +192,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; + bool mark_mem_decrypt =3D true; struct page *page; void *ret; =20 @@ -252,11 +242,21 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, return page ? ret : NULL; } =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (page) { + mark_mem_decrypt =3D false; + goto setup_page; + } + return NULL; + } + /* we always manually zero the memory once we are done */ page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); if (!page) return NULL; =20 +setup_page: /* * dma_alloc_contiguous can return highmem pages depending on a * combination the cma=3D arguments and per-arch setup. These need to be @@ -283,7 +283,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, goto out_free_pages; } else { ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) goto out_leak_pages; } =20 @@ -300,10 +300,11 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, return ret; =20 out_encrypt_pages: - if (dma_set_encrypted(dev, page_address(page), size)) + if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) return NULL; out_free_pages: - __dma_direct_free_pages(dev, page, size); + if (!swiotlb_free(dev, page, size)) + dma_free_contiguous(dev, page, size); return NULL; out_leak_pages: return NULL; @@ -312,6 +313,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { + phys_addr_t phys; + bool mark_mem_encrypted =3D true; + struct io_tlb_pool *swiotlb_pool; unsigned int page_order =3D get_order(size); =20 if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && @@ -340,16 +344,25 @@ void dma_direct_free(struct device *dev, size_t size, dma_free_from_pool(dev, cpu_addr, PAGE_ALIGN(size))) return; =20 + phys =3D dma_to_phys(dev, dma_addr); + swiotlb_pool =3D swiotlb_find_pool(dev, phys); + if (swiotlb_pool) + /* Swiotlb doesn't need a page attribute update on free */ + mark_mem_encrypted =3D false; + if (is_vmalloc_addr(cpu_addr)) { vunmap(cpu_addr); } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (dma_set_encrypted(dev, cpu_addr, size)) + if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) return; } =20 - __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); + if (swiotlb_pool) + swiotlb_free_from_pool(dev, phys, swiotlb_pool); + else + dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); } =20 struct page *dma_direct_alloc_pages(struct device *dev, size_t size, @@ -361,6 +374,15 @@ struct page *dma_direct_alloc_pages(struct device *dev= , size_t size, if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (!page) + return NULL; + + ret =3D page_address(page); + goto setup_page; + } + page =3D __dma_direct_alloc_pages(dev, size, gfp, false); if (!page) return NULL; @@ -368,6 +390,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, ret =3D page_address(page); if (dma_set_decrypted(dev, ret, size)) goto out_leak_pages; +setup_page: memset(ret, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return page; @@ -379,16 +402,28 @@ void dma_direct_free_pages(struct device *dev, size_t= size, struct page *page, dma_addr_t dma_addr, enum dma_data_direction dir) { + phys_addr_t phys; void *vaddr =3D page_address(page); + struct io_tlb_pool *swiotlb_pool; + bool mark_mem_encrypted =3D true; =20 /* If page is not from an atomic pool, dma_free_from_pool_page() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool_page(dev, page, size)) return; =20 - if (dma_set_encrypted(dev, vaddr, size)) + phys =3D page_to_phys(page); + swiotlb_pool =3D swiotlb_find_pool(dev, phys); + if (swiotlb_pool) + mark_mem_encrypted =3D false; + + if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size)) return; - __dma_direct_free_pages(dev, page, size); + + if (swiotlb_pool) + swiotlb_free_from_pool(dev, phys, swiotlb_pool); + else + dma_free_contiguous(dev, page, size); } =20 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 6e8db52866bf..d54154c165e5 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1815,6 +1815,12 @@ bool swiotlb_free(struct device *dev, struct page *p= age, size_t size) return true; } =20 +void swiotlb_free_from_pool(struct device *dev, + phys_addr_t tlb_addr, struct io_tlb_pool *pool) +{ + swiotlb_release_slots(dev, tlb_addr, pool); +} + static int rmem_swiotlb_device_init(struct reserved_mem *rmem, struct device *dev) { --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B28A36F902; Fri, 17 Jul 2026 18:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311595; cv=none; b=WonxHgiLQ6Qm1lx+edLu9aIm93Iug6Ozw7UIW5I2IgVH4NAgI1qnoBXyzAbrfJNlY0o4VcM+NzeiDykWybxW7J1JyoKJ9gV+m4XwXCd01zlZdBB+wswiXJc4MPov8T2SzvbEq44Nc6CcVIZmpT54KZpZ3UV1ru3NMljfhQjTgp0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311595; c=relaxed/simple; bh=exFLkpW/HMwkjq9ifI43aaD8fINmyz2L4L3/XDVnMJQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VVrO6YYJH9vEoWjNyzq1B1alQlabprMA2wOVjfWvmoUJAA54EJHIdkyOO9CvRVxdb2mmGeVMr8cpr/YZutVu0qTIlmGFzmZLjUsqW9j2oPPZ1r+hsluVsvf58K6nF4Mv+rDE2G2fvqrykXdXv6YA17uCePYXEONE7PBltS8o/dI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W4D94t/w; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W4D94t/w" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E00181F000E9; Fri, 17 Jul 2026 18:06:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311594; bh=9dRlfhm2GJvO0NKET2nomIv2slIBL0lQ9SwCkEKvtM0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=W4D94t/wneRBeRYLAxxxPKoeAyp9CDNhgy+7bpaY5yTYbVT8ophnINPXzQyM9OHYN uagA6UPpU0Lgd0flDoEB5rlssvKZ3tyRPfFbpPV2sf4zB7a3+52/Uwmoy4G5i14rxU R6qttKEJdgr3zR0OdDRKEWhPzw3UvFgaBhsrJTAssxanyZ8F5+nRj1wmIDiSTDvFfT C9XrJ8Uks6+0EVkTEMzI/n52dg18W/W/0DljfnwfNv/RbZ5MkYOz1a6E2e7S53d5Rv /7mUns0bXxC78eCQaFFkVK6EawRz39RdB16ufoyCKINyYamNITiKhCj3jV2gzhZkpL a4YscX7FQeceg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v8 08/23] coco: arm64: s390: powerpc: Mark secure guests with CC_ATTR_GUEST_MEM_ENCRYPT Date: Fri, 17 Jul 2026 23:34:26 +0530 Message-ID: <20260717180442.110954-9-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" arm64 Realms, powerpc secure guests and s390 protected virtualization guests currently report CC_ATTR_MEM_ENCRYPT, but not CC_ATTR_GUEST_MEM_ENCRYPT. These environments are all secure guest configurations. Their force_dma_unencrypted() implementations also treat the same secure guest state as requiring unencrypted DMA. Report CC_ATTR_GUEST_MEM_ENCRYPT for the same condition so generic confidential-computing code can distinguish guest memory encryption from host memory encryption instead of relying only on CC_ATTR_MEM_ENCRYPT. Signed-off-by: Aneesh Kumar K.V (Arm) --- arch/arm64/mm/init.c | 1 + arch/powerpc/platforms/pseries/cc_platform.c | 1 + arch/s390/mm/init.c | 1 + 3 files changed, 3 insertions(+) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index edcc305ec883..48d69e390158 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -419,6 +419,7 @@ bool cc_platform_has(enum cc_attr attr) { switch (attr) { case CC_ATTR_MEM_ENCRYPT: + case CC_ATTR_GUEST_MEM_ENCRYPT: return is_realm_world() || is_protected_kvm_guest(); default: return false; diff --git a/arch/powerpc/platforms/pseries/cc_platform.c b/arch/powerpc/pl= atforms/pseries/cc_platform.c index e8021af83a19..46e110a5a717 100644 --- a/arch/powerpc/platforms/pseries/cc_platform.c +++ b/arch/powerpc/platforms/pseries/cc_platform.c @@ -17,6 +17,7 @@ bool cc_platform_has(enum cc_attr attr) { switch (attr) { case CC_ATTR_MEM_ENCRYPT: + case CC_ATTR_GUEST_MEM_ENCRYPT: return is_secure_guest(); =20 default: diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index c29326c44af0..6b1c5a4fa9ce 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -148,6 +148,7 @@ bool cc_platform_has(enum cc_attr attr) { switch (attr) { case CC_ATTR_MEM_ENCRYPT: + case CC_ATTR_GUEST_MEM_ENCRYPT: return is_prot_virt_guest(); =20 default: --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97CD8379990; Fri, 17 Jul 2026 18:06:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311606; cv=none; b=Ti+SXOsW4mi4uiZeOM8sUyFUHEsZepLP/haNuTCVg3SbDNtWwD4pdqwR7ZJ4Pm1XeIGVpCwAB6fZp9jjzhc/IKO6ALKiY2rkxKvYAnxDtVYVExws5PvLRrQ9WOkw1BTae8u+8dcr76wWHoC2MVKRlsKTwnoc5F5GTiGCw1qaiJU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311606; c=relaxed/simple; bh=9nw5aIYzatCEe3Flqvct3jQReC+9+/hhq53ePIM21jc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FHC7GuW1ZMKhkjjF+F/tph1nOaI/BlxSqP0GdUq8LJ/k5jHJkYcgw7n56LF5PrDcgPwBiyeFyADU8noZCuSenYYhEyu3DT8+suamHHgX7GYx1fUYgZacPXzYltxWF6APYPLjs51lvetZaQX2XaxqjivsKXn6Nhtvgri47Kg0dUw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lKWnTwZk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lKWnTwZk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC53D1F00A3A; Fri, 17 Jul 2026 18:06:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311605; bh=+jYpPT4yoOmUsd5LmqpXbLSNFTvE7U1R4IX5MV5qOnI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=lKWnTwZkBJNfcxr3yoDEkHJjczN3UyUzlyushdGlL83xmNsEfAsye+3iSTLFWqXxn ztR40aAk0Ouu7NhezKGDg8Yq/ZKbkImDSf7CjwI5Faed67v3qLop+8qv/RtesJx3dH 5KcFGx8I8rLW5vCkTVp22N/WPJhUXtQ6ucnhEX0yvmX/ANam6WaHYPXw9Y4ktnNGVN an1FQEwPkTP2UaTyK7zboDbZXDXqZe/JGi4LaRUxWH5w7hCUgw+hrNHbgWMzwHa1IX qRe7IykVDvj00+/8wQAsJdKZ9mz9aHH37o+m3erTp0QnML1B5sOqtNISiGwUxk2sqW qOTGbKgbYOisQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe Subject: [PATCH v8 09/23] dma-mapping: Add internal shared allocation attribute Date: Fri, 17 Jul 2026 23:34:27 +0530 Message-ID: <20260717180442.110954-10-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" DMA_ATTR_CC_SHARED describes an existing DMA mapping whose backing memory is already shared, or decrypted, for confidential computing. It is a mapping attribute: callers use it to request a shared DMA address encoding for memory that has already been prepared for shared DMA. Allocation paths need a related but different state. Once the DMA core decides that an allocation must use shared backing pages, the lower-level allocation helpers need to select shared pools, decrypt newly allocated pages, derive the DMA address with the shared-memory translation and restore encryption on free. That state is internal to the DMA-mapping implementation and should not be passed by drivers to dma_alloc_attrs(). Reviewed-by: Jason Gunthorpe Signed-off-by: Aneesh Kumar K.V (Arm) --- Documentation/core-api/dma-attributes.rst | 29 +++++++++++++++++++++++ include/linux/dma-mapping.h | 8 +++++++ include/trace/events/dma.h | 3 ++- 3 files changed, 39 insertions(+), 1 deletion(-) diff --git a/Documentation/core-api/dma-attributes.rst b/Documentation/core= -api/dma-attributes.rst index 123c8468d58f..eee743184acd 100644 --- a/Documentation/core-api/dma-attributes.rst +++ b/Documentation/core-api/dma-attributes.rst @@ -179,3 +179,32 @@ interface when building their uAPIs, when possible. =20 It must never be used in an in-kernel driver that only works with kernel memory. + +DMA_ATTR_CC_SHARED +------------------ + +This attribute indicates that a DMA mapping is shared, or decrypted, for +confidential computing guests. For normal system memory, the caller must +already have marked the memory decrypted with set_memory_decrypted(). CPU +PTEs for the mapping must use pgprot_decrypted(), and the same shared +semantic may be passed to a vIOMMU when it sets up the IOPTE. + +This attribute describes an existing mapping. It does not allocate shared +backing pages and must not be passed to dma_alloc_attrs(). For MMIO, use +this together with DMA_ATTR_MMIO to indicate shared MMIO. Unless +DMA_ATTR_MMIO is provided, the mapping requires a struct page. + +__DMA_ATTR_ALLOC_CC_SHARED +-------------------------- + +This is an internal DMA-mapping attribute for confidential computing guest= s. +It is used by allocation paths after the DMA core has determined that the +backing pages must be shared, or decrypted. For example, the direct DMA and +SWIOTLB allocation paths use it to select shared DMA pools, decrypt newly +allocated pages, derive DMA addresses using the shared-memory translation,= and +restore encryption on free. + +__DMA_ATTR_ALLOC_CC_SHARED differs from DMA_ATTR_CC_SHARED in that it is n= ot +a caller-visible DMA API attribute. DMA_ATTR_CC_SHARED describes an +already-shared mapping and requires the caller to have prepared normal +system memory before mapping it. diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index cc0823a99cfd..a3e880649fa4 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -103,6 +103,14 @@ */ #define DMA_ATTR_CC_SHARED (1UL << 13) =20 +/* + * __DMA_ATTR_ALLOC_CC_SHARED: Internal DMA-mapping attribute used by + * allocation paths that create shared (decrypted) backing pages for + * confidential computing guests. Drivers must not pass this attribute to + * dma_alloc_attrs(). + */ +#define __DMA_ATTR_ALLOC_CC_SHARED (1UL << 14) + /* * A dma_addr_t can hold any valid DMA or bus address for the platform. I= t can * be given to a device to use as a DMA source or target. It is specific = to a diff --git a/include/trace/events/dma.h b/include/trace/events/dma.h index 31c9ddf72c9d..9df02c1511de 100644 --- a/include/trace/events/dma.h +++ b/include/trace/events/dma.h @@ -35,7 +35,8 @@ TRACE_DEFINE_ENUM(DMA_NONE); { DMA_ATTR_MMIO, "MMIO" }, \ { DMA_ATTR_DEBUGGING_IGNORE_CACHELINES, "CACHELINES_OVERLAP" }, \ { DMA_ATTR_REQUIRE_COHERENT, "REQUIRE_COHERENT" }, \ - { DMA_ATTR_CC_SHARED, "CC_SHARED" }) + { DMA_ATTR_CC_SHARED, "CC_SHARED" }, \ + { __DMA_ATTR_ALLOC_CC_SHARED, "ALLOC_CC_SHARED" }) =20 DECLARE_EVENT_CLASS(dma_map, TP_PROTO(struct device *dev, phys_addr_t phys_addr, dma_addr_t dma_addr, --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0892381AF4; Fri, 17 Jul 2026 18:06:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311620; cv=none; b=Ky+UYoW61yxf90Ie25x79D1JlSox50v64zC/nPi/q2+ll/d08SM7cNwSgp0tm/y8eR+mJ1BVVdp9qLbdCxnb3buMfmpelu3WU7pWxcr6QyDgULisRfZ0AwPZeeucAni+q9dNhAwp6NAwOmm6LzapHaVNcq/HBkvu7rpwY1UyyT8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311620; c=relaxed/simple; bh=8FdRmSBOTYCRf9YarFwod2kTaD8MCcCdoydVh2lmXNQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tPZMXHT1wYcLlrteZw9XgOuTBwBbwKJy94SBevhv52E7JJqJXTh6wIK9wCkhNOpFh0ri7DlH8TPrCvpl9lem+Ii8ufOPUpB0XSrUol3cZrfn1wyW8y49x5fdErUoNAp/FWjvhX5aSbKOCHWbRZRKAjckIECOFgUxIUMOuiDvStk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Fi0KoA/e; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Fi0KoA/e" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 048FD1F00A3E; Fri, 17 Jul 2026 18:06:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311617; bh=vhd8ARWkryxrWwIm7KhnUETWTJknCTz38d8nlJNnprQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Fi0KoA/ePSpIRE6p1P5b8WFj1HHt45EONpGIMYarmdrr1D5rMvl72euUoRDCMuAM5 zbeOwFDh1wXeyO8ALlxZFBuERXeseFibb2X7SfQmBMZZu0aN+qj3jemlAp3kGIiUKe mcLRmeyxuqKJYqs2KS0bLnr0BsTahuIrZZ6vcCiEzJaUeJ4buInrCfJUxPSBq5Bsgs Pi2IbRpiAZtnRvzumFiAOIBzWPl9t2T4Pxb3s+4HEiFSeWSZTMdAfR9k1jfLRXWiTx xC3pkQeeojY0PZQppL0wtTp5QF0B6dswPMnrADubzvnEBSGIECjumBnzUR10MFNUFS KJzLvycPv1HOA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 10/23] dma-direct: use __DMA_ATTR_ALLOC_CC_SHARED in alloc/free paths Date: Fri, 17 Jul 2026 23:34:28 +0530 Message-ID: <20260717180442.110954-11-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Propagate force_dma_unencrypted() into __DMA_ATTR_ALLOC_CC_SHARED in the dma-direct allocation path and use the attribute to drive the related decisions. This updates dma_direct_alloc(), dma_direct_free(), and dma_direct_alloc_pages() to fold the forced unencrypted case into attrs. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 42 +++++++++++++++++++++++++++++++++--------- kernel/dma/mapping.c | 9 +++++++++ 2 files changed, 42 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 171c89fe99c9..1b49ba5e0a60 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -192,16 +192,22 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; - bool mark_mem_decrypt =3D true; + bool mark_mem_decrypt =3D false; struct page *page; void *ret; =20 + if (force_dma_unencrypted(dev)) + attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; + + if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) + mark_mem_decrypt =3D true; + size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) gfp |=3D __GFP_NOWARN; =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | __DMA_ATTR_ALLOC_CC_SHARED)) = =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp); =20 if (!dev_is_dma_coherent(dev)) { @@ -235,7 +241,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, * Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ - if ((remap || force_dma_unencrypted(dev)) && + if ((remap || (attrs & __DMA_ATTR_ALLOC_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) { page =3D dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); @@ -314,12 +320,22 @@ void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { phys_addr_t phys; - bool mark_mem_encrypted =3D true; + bool mark_mem_encrypted =3D false; struct io_tlb_pool *swiotlb_pool; unsigned int page_order =3D get_order(size); =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) { + /* + * If the allocation used decrypted/shared backing pages, restore + * the encryption state on free. + */ + if (force_dma_unencrypted(dev)) + attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; + + if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) + mark_mem_encrypted =3D true; + + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | __DMA_ATTR_ALLOC_CC_SHARED)) = =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) { /* cpu_addr is a struct page cookie, not a kernel address */ dma_free_contiguous(dev, cpu_addr, size); return; @@ -368,10 +384,14 @@ void dma_direct_free(struct device *dev, size_t size, struct page *dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp) { + unsigned long attrs =3D 0; struct page *page; void *ret; =20 - if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) + if (force_dma_unencrypted(dev)) + attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; + + if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); =20 if (is_swiotlb_for_alloc(dev)) { @@ -405,7 +425,11 @@ void dma_direct_free_pages(struct device *dev, size_t = size, phys_addr_t phys; void *vaddr =3D page_address(page); struct io_tlb_pool *swiotlb_pool; - bool mark_mem_encrypted =3D true; + /* + * if the device had requested for an unencrypted buffer, + * convert it to encrypted on free + */ + bool mark_mem_encrypted =3D force_dma_unencrypted(dev); =20 /* If page is not from an atomic pool, dma_free_from_pool_page() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 4fe04669e5e6..d2f70b6ccd0f 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -638,6 +638,15 @@ void *dma_alloc_attrs(struct device *dev, size_t size,= dma_addr_t *dma_handle, if (WARN_ON_ONCE(flag & __GFP_COMP)) return NULL; =20 + if (attrs & (DMA_ATTR_CC_SHARED | __DMA_ATTR_ALLOC_CC_SHARED)) { + trace_dma_alloc(dev, NULL, 0, size, DMA_BIDIRECTIONAL, flag, + attrs); + return NULL; + } + + if (force_dma_unencrypted(dev)) + attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; + if (dma_alloc_from_dev_coherent(dev, size, dma_handle, &cpu_addr)) { trace_dma_alloc(dev, cpu_addr, *dma_handle, size, DMA_BIDIRECTIONAL, flag, attrs); --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3454376A10; Fri, 17 Jul 2026 18:07:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311631; cv=none; b=RXZ4WfSWY1TTS46oYw9Kuz8Z2/evbazGRUZdJWerECwpgj33bB1YnB+H1D45gtQXotpowgwfS+A47wRw1dsCYN7sAGsLec2DmvE6cvTEC/yYrC1swWOY8vUVTVD+75MzJ3mlfXg+GrWjWsuwyF3u7sBg1SPx5QbA8PBBnp68A7E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311631; c=relaxed/simple; bh=cAFZioD4AMqqSP52hQFgVt1KAj+aPXZcAUrSxBKhWMA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tpQfrE3R2/yWKnrA/VqRVJKzfY0NC5CGaSCDR3KS2/DOy5CnyPGQTNWrnDdtjMjt41AzjAmrf2/tVFRZ2CM48l1wjfyY6vDpAks/nr86dO7zAvUA7uYHsaAKHsFLmS/qZZZNPCFF/Zbpoi1RpP7TuJv8BSZHSqqPrTX1k5tC9b4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=W0+VA3JA; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="W0+VA3JA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E08B21F00A3A; Fri, 17 Jul 2026 18:06:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311629; bh=zhoVNQSfeQOGKOCtTTE1JWqlBu+3klDpNUZDVpGU7EQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=W0+VA3JAWbdPoQbmXdHAnN7Sv+U3kKQpLHcPqeUfp+S9evEFyzUdhcfTDpFNXOjD9 P9+F3EkKGXxjHMrJiS1uJYkGabvlQvQpRP/ELqNyIgwG98qHAY+3IUF9mMbG98w80s d8X2E4U1c18+TF1WK6LAe76qxSGxqM/Rge8yGopiua76qhfkDRZOymmjvwh3qmWFmj BhjdqVY4Kl9eRfvujqJ69fNHZKD0poZnESHrvy63gOpXtTnHr3qVNhAj+YVdN+EbP5 adSJLvL6OrqDlYdgovZ7JyjmCUyRA50YKZo1XWojFLQKf9MnPN4r9lP//aj8leL6Wy cbQvWXYVHaljg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v8 11/23] dma-pool: track decrypted atomic pools and select them via attrs Date: Fri, 17 Jul 2026 23:34:29 +0530 Message-ID: <20260717180442.110954-12-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach the atomic DMA pool code to distinguish between encrypted and unencrypted pools, and make pool allocation select the matching pool based on DMA attributes. Introduce a dma_gen_pool wrapper that records whether a pool is unencrypted, initialize that state when the atomic pools are created, and use it when expanding and resizing the pools. Update dma_alloc_from_pool() to take attrs and skip pools whose encrypted state does not match __DMA_ATTR_ALLOC_CC_SHARED. Update dma_free_from_pool() accordingly. Also pass __DMA_ATTR_ALLOC_CC_SHARED from the swiotlb atomic allocation path so decrypted swiotlb allocations are taken from the correct atomic pool. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 7 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/direct.c | 14 ++- kernel/dma/pool.c | 182 +++++++++++++++++++++++------------- kernel/dma/swiotlb.c | 8 +- 5 files changed, 139 insertions(+), 74 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 6a45acbbfb0c..83bf15ae9777 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1660,9 +1660,14 @@ void *iommu_dma_alloc(struct device *dev, size_t siz= e, dma_addr_t *handle, { bool coherent =3D dev_is_dma_coherent(dev); int ioprot =3D dma_info_to_prot(DMA_BIDIRECTIONAL, coherent, attrs); + bool is_alloc_cc_shared =3D attrs & __DMA_ATTR_ALLOC_CC_SHARED; struct page *page =3D NULL; void *cpu_addr; =20 + /* Not yet supported */ + if (is_alloc_cc_shared) + return NULL; + gfp |=3D __GFP_ZERO; =20 if (gfpflags_allow_blocking(gfp) && @@ -1673,7 +1678,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size= , dma_addr_t *handle, if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !gfpflags_allow_blocking(gfp) && !coherent) { page =3D dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, - gfp, NULL); + gfp, attrs, NULL); if (!page) return NULL; } else { diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 137e015c1750..8fae2b7deb20 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -212,7 +212,7 @@ void *dma_common_pages_remap(struct page **pages, size_= t size, pgprot_t prot, void dma_common_free_remap(void *cpu_addr, size_t size); =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t flags, + void **cpu_addr, gfp_t flags, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)); bool dma_free_from_pool(struct device *dev, void *start, size_t size); bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size); diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 1b49ba5e0a60..6960ee1cfefe 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -154,7 +154,8 @@ static bool dma_direct_use_pool(struct device *dev, gfp= _t gfp) } =20 static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t = size, - dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp) + dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp, + unsigned long attrs) { struct page *page; u64 phys_limit; @@ -163,7 +164,8 @@ static struct page *dma_direct_alloc_from_pool(struct d= evice *dev, size_t size, return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, cpu_addr, gfp, dma_coherent_ok); + page =3D dma_alloc_from_pool(dev, size, cpu_addr, gfp, attrs, + dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); @@ -240,11 +242,14 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, /* * Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. + * FIXME: With CONFIG_DMA_DIRECT_REMAP, the pool is also mapped as + * DMA-coherent (non-cacheable). We may want to create a separate pool + * dedicated to CC_SHARED atomic allocations. */ if ((remap || (attrs & __DMA_ATTR_ALLOC_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) { page =3D dma_direct_alloc_from_pool(dev, size, dma_handle, - &ret, gfp); + &ret, gfp, attrs); return page ? ret : NULL; } =20 @@ -392,7 +397,8 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; =20 if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, + &ret, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size); diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index e981c1faaadf..00f422a1e896 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -12,12 +12,18 @@ #include #include #include +#include =20 -static struct gen_pool *atomic_pool_dma __ro_after_init; +struct dma_gen_pool { + bool cc_shared; + struct gen_pool *pool; +}; + +static struct dma_gen_pool atomic_pool_dma __ro_after_init; static unsigned long pool_size_dma; -static struct gen_pool *atomic_pool_dma32 __ro_after_init; +static struct dma_gen_pool atomic_pool_dma32 __ro_after_init; static unsigned long pool_size_dma32; -static struct gen_pool *atomic_pool_kernel __ro_after_init; +static struct dma_gen_pool atomic_pool_kernel __ro_after_init; static unsigned long pool_size_kernel; =20 /* Size can be defined by the coherent_pool command line */ @@ -76,7 +82,7 @@ static bool cma_in_zone(gfp_t gfp) return true; } =20 -static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size, +static int atomic_pool_expand(struct dma_gen_pool *dma_pool, size_t pool_s= ize, gfp_t gfp) { unsigned int order; @@ -84,6 +90,7 @@ static int atomic_pool_expand(struct gen_pool *pool, size= _t pool_size, bool leak_pages =3D false; void *addr; int ret =3D -ENOMEM; + pgprot_t prot __maybe_unused; =20 /* Cannot allocate larger than MAX_PAGE_ORDER */ order =3D min(get_order(pool_size), MAX_PAGE_ORDER); @@ -102,8 +109,12 @@ static int atomic_pool_expand(struct gen_pool *pool, s= ize_t pool_size, arch_dma_prep_coherent(page, pool_size); =20 #ifdef CONFIG_DMA_DIRECT_REMAP - addr =3D dma_common_contiguous_remap(page, pool_size, - pgprot_decrypted(pgprot_dmacoherent(PAGE_KERNEL)), + if (dma_pool->cc_shared) + prot =3D pgprot_decrypted(pgprot_dmacoherent(PAGE_KERNEL)); + else + prot =3D pgprot_dmacoherent(PAGE_KERNEL); + + addr =3D dma_common_contiguous_remap(page, pool_size, prot, __builtin_return_address(0)); if (!addr) goto free_page; @@ -114,14 +125,17 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, * Memory in the atomic DMA pools must be unencrypted, the pools do not * shrink so no re-encryption occurs in dma_direct_free(). */ - ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), - 1 << order); - if (ret) { - leak_pages =3D true; - goto remove_mapping; + if (dma_pool->cc_shared) { + ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), + 1 << order); + if (ret) { + leak_pages =3D true; + goto remove_mapping; + } } - ret =3D gen_pool_add_virt(pool, (unsigned long)addr, page_to_phys(page), - pool_size, NUMA_NO_NODE); + + ret =3D gen_pool_add_virt(dma_pool->pool, (unsigned long)addr, + page_to_phys(page), pool_size, NUMA_NO_NODE); if (ret) goto encrypt_mapping; =20 @@ -129,12 +143,10 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, return 0; =20 encrypt_mapping: - ret =3D set_memory_encrypted((unsigned long)page_to_virt(page), - 1 << order); - if (WARN_ON_ONCE(ret)) { - /* Decrypt succeeded but encrypt failed, purposely leak */ + if (dma_pool->cc_shared && + set_memory_encrypted((unsigned long)page_to_virt(page), 1 << order)) leak_pages =3D true; - } + remove_mapping: #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); @@ -146,46 +158,52 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, return ret; } =20 -static void atomic_pool_resize(struct gen_pool *pool, gfp_t gfp) +static void atomic_pool_resize(struct dma_gen_pool *dma_pool, gfp_t gfp) { - if (pool && gen_pool_avail(pool) < atomic_pool_size) - atomic_pool_expand(pool, gen_pool_size(pool), gfp); + if (dma_pool->pool && gen_pool_avail(dma_pool->pool) < atomic_pool_size) + atomic_pool_expand(dma_pool, gen_pool_size(dma_pool->pool), gfp); } =20 static void atomic_pool_work_fn(struct work_struct *work) { if (IS_ENABLED(CONFIG_ZONE_DMA)) - atomic_pool_resize(atomic_pool_dma, + atomic_pool_resize(&atomic_pool_dma, GFP_KERNEL | GFP_DMA); if (IS_ENABLED(CONFIG_ZONE_DMA32)) - atomic_pool_resize(atomic_pool_dma32, + atomic_pool_resize(&atomic_pool_dma32, GFP_KERNEL | GFP_DMA32); - atomic_pool_resize(atomic_pool_kernel, GFP_KERNEL); + atomic_pool_resize(&atomic_pool_kernel, GFP_KERNEL); } =20 -static __init struct gen_pool *__dma_atomic_pool_init(size_t pool_size, - gfp_t gfp) +static __init struct dma_gen_pool *__dma_atomic_pool_init(struct dma_gen_p= ool *dma_pool, + size_t pool_size, gfp_t gfp) { - struct gen_pool *pool; int ret; =20 - pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); - if (!pool) + dma_pool->pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); + if (!dma_pool->pool) return NULL; =20 - gen_pool_set_algo(pool, gen_pool_first_fit_order_align, NULL); + gen_pool_set_algo(dma_pool->pool, gen_pool_first_fit_order_align, NULL); =20 - ret =3D atomic_pool_expand(pool, pool_size, gfp); + /* if platform is using memory encryption atomic pools are by default sha= red. */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + dma_pool->cc_shared =3D true; + else + dma_pool->cc_shared =3D false; + + ret =3D atomic_pool_expand(dma_pool, pool_size, gfp); if (ret) { - gen_pool_destroy(pool); + gen_pool_destroy(dma_pool->pool); + dma_pool->pool =3D NULL; pr_err("DMA: failed to allocate %zu KiB %pGg pool for atomic allocation\= n", pool_size >> 10, &gfp); return NULL; } =20 pr_info("DMA: preallocated %zu KiB %pGg pool for atomic allocations\n", - gen_pool_size(pool) >> 10, &gfp); - return pool; + gen_pool_size(dma_pool->pool) >> 10, &gfp); + return dma_pool; } =20 #ifdef CONFIG_ZONE_DMA32 @@ -211,21 +229,22 @@ static int __init dma_atomic_pool_init(void) =20 /* All memory might be in the DMA zone(s) to begin with */ if (has_managed_zone(ZONE_NORMAL)) { - atomic_pool_kernel =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL); - if (!atomic_pool_kernel) + __dma_atomic_pool_init(&atomic_pool_kernel, atomic_pool_size, GFP_KERNEL= ); + if (!atomic_pool_kernel.pool) ret =3D -ENOMEM; } + if (has_managed_dma()) { - atomic_pool_dma =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA); - if (!atomic_pool_dma) + __dma_atomic_pool_init(&atomic_pool_dma, atomic_pool_size, + GFP_KERNEL | GFP_DMA); + if (!atomic_pool_dma.pool) ret =3D -ENOMEM; } + if (has_managed_dma32) { - atomic_pool_dma32 =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA32); - if (!atomic_pool_dma32) + __dma_atomic_pool_init(&atomic_pool_dma32, atomic_pool_size, + GFP_KERNEL | GFP_DMA32); + if (!atomic_pool_dma32.pool) ret =3D -ENOMEM; } =20 @@ -234,19 +253,44 @@ static int __init dma_atomic_pool_init(void) } postcore_initcall(dma_atomic_pool_init); =20 -static inline struct gen_pool *dma_guess_pool(struct gen_pool *prev, gfp_t= gfp) +static inline struct dma_gen_pool *__dma_guess_pool(struct dma_gen_pool *f= irst, + struct dma_gen_pool *second, struct dma_gen_pool *third) { - if (prev =3D=3D NULL) { + if (first->pool) + return first; + if (second && second->pool) + return second; + if (third && third->pool) + return third; + return NULL; +} + +static inline struct dma_gen_pool *dma_guess_pool(struct dma_gen_pool *pre= v, + gfp_t gfp) +{ + if (!prev) { if (gfp & GFP_DMA) - return atomic_pool_dma ?: atomic_pool_dma32 ?: atomic_pool_kernel; + return __dma_guess_pool(&atomic_pool_dma, + &atomic_pool_dma32, + &atomic_pool_kernel); + if (gfp & GFP_DMA32) - return atomic_pool_dma32 ?: atomic_pool_dma ?: atomic_pool_kernel; - return atomic_pool_kernel ?: atomic_pool_dma32 ?: atomic_pool_dma; + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, + &atomic_pool_kernel); + + return __dma_guess_pool(&atomic_pool_kernel, + &atomic_pool_dma32, + &atomic_pool_dma); } - if (prev =3D=3D atomic_pool_kernel) - return atomic_pool_dma32 ? atomic_pool_dma32 : atomic_pool_dma; - if (prev =3D=3D atomic_pool_dma32) - return atomic_pool_dma; + + if (prev =3D=3D &atomic_pool_kernel) + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, NULL); + + if (prev =3D=3D &atomic_pool_dma32) + return __dma_guess_pool(&atomic_pool_dma, NULL, NULL); + return NULL; } =20 @@ -276,16 +320,20 @@ static struct page *__dma_alloc_from_pool(struct devi= ce *dev, size_t size, } =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t gfp, + void **cpu_addr, gfp_t gfp, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; struct page *page; bool pool_found =3D false; =20 - while ((pool =3D dma_guess_pool(pool, gfp))) { + while ((dma_pool =3D dma_guess_pool(dma_pool, gfp))) { + + if (dma_pool->cc_shared !=3D !!(attrs & __DMA_ATTR_ALLOC_CC_SHARED)) + continue; + pool_found =3D true; - page =3D __dma_alloc_from_pool(dev, size, pool, cpu_addr, + page =3D __dma_alloc_from_pool(dev, size, dma_pool->pool, cpu_addr, phys_addr_ok); if (page) return page; @@ -300,12 +348,14 @@ struct page *dma_alloc_from_pool(struct device *dev, = size_t size, =20 bool dma_free_from_pool(struct device *dev, void *start, size_t size) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; + + while ((dma_pool =3D dma_guess_pool(dma_pool, 0))) { =20 - while ((pool =3D dma_guess_pool(pool, 0))) { - if (!gen_pool_has_addr(pool, (unsigned long)start, size)) + if (!gen_pool_has_addr(dma_pool->pool, (unsigned long)start, size)) continue; - gen_pool_free(pool, (unsigned long)start, size); + + gen_pool_free(dma_pool->pool, (unsigned long)start, size); return true; } =20 @@ -337,7 +387,7 @@ static void dma_pool_find_phys(struct gen_pool *pool, s= truct gen_pool_chunk *chu match->found =3D true; } =20 -static bool dma_free_from_pool_phys(struct gen_pool *pool, phys_addr_t phy= s, +static bool dma_free_from_pool_phys(struct dma_gen_pool *dma_pool, phys_ad= dr_t phys, size_t size) { struct dma_pool_phys_match match =3D { @@ -345,11 +395,11 @@ static bool dma_free_from_pool_phys(struct gen_pool *= pool, phys_addr_t phys, .size =3D size, }; =20 - gen_pool_for_each_chunk(pool, dma_pool_find_phys, &match); + gen_pool_for_each_chunk(dma_pool->pool, dma_pool_find_phys, &match); if (!match.found) return false; =20 - gen_pool_free(pool, match.addr, size); + gen_pool_free(dma_pool->pool, match.addr, size); return true; } =20 @@ -359,14 +409,14 @@ static bool dma_free_from_pool_phys(struct gen_pool *= pool, phys_addr_t phys, */ bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; phys_addr_t phys =3D page_to_phys(page); =20 if (!IS_ENABLED(CONFIG_DMA_DIRECT_REMAP)) return dma_free_from_pool(dev, page_address(page), size); =20 - while ((pool =3D dma_guess_pool(pool, 0))) { - if (dma_free_from_pool_phys(pool, phys, size)) + while ((dma_pool =3D dma_guess_pool(dma_pool, 0))) { + if (dma_free_from_pool_phys(dma_pool, phys, size)) return true; } =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index d54154c165e5..908de28aceb2 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -613,9 +613,9 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, u64 phys_limit, gfp_t gfp, void **vaddr) { struct page *page; + unsigned long attrs =3D 0; =20 *vaddr =3D NULL; - /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. @@ -624,8 +624,12 @@ static struct page *swiotlb_alloc_tlb(struct device *d= ev, size_t bytes, if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 + /* swiotlb considered decrypted by default */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + attrs =3D __DMA_ATTR_ALLOC_CC_SHARED; + return dma_alloc_from_pool(dev, bytes, vaddr, gfp, - dma_coherent_ok); + attrs, dma_coherent_ok); } =20 gfp &=3D ~GFP_ZONEMASK; --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2E599363C75; Fri, 17 Jul 2026 18:07:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311642; cv=none; b=RxnZ5biEqykrsNs6jHJ3f/hAbQeUR6Dx4mFFbay+QxRRDi1kr7TTGHA5567H8TXPjBSjMSyWSW0hy+GtH7YRXhuj3eSytGF9WNdASDM8KOpxDLH/IQZDvT/kgvimeiCcxfu5stW63BjgHjOa46WLDATjaNZelKUNnKxnjOlbtWM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311642; c=relaxed/simple; bh=hjk54hwCeXFuHOgGlL08cCYy5A8Di3d+CJ09zT+uMFs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Vr2o2lxQNtqRGLWUiF6cf9Rj3paSPEzZ7RxD1EG1mdjIegxOSP1oHFW14qGQRs9b9LDEkiZ6+lNBldulopVLP30bQDgud5BFq4x5gf1Zna7l0igFm9B/Auc/jAQoTtrtSd9injWH1gJT+Xnn5N1GxcruIPcD0zjMuLydgSAFJn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=az13Ithd; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="az13Ithd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F37B11F000E9; Fri, 17 Jul 2026 18:07:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311641; bh=GmV+DLQ7tYIWroxvAExoKmvbkkmdzgJAVMErN4og4x4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=az13IthdkXyuHXIkezoBcf/c1Fq7gOmr7pJ1N3ScoTlVliZ+mKyhugHc9IoFT/+ps mx/VWltvB/OE4cP7XVfJT7AqIaVa7vuI1ROM488yX1sON079TaNz/XxE+xmHpKkt8M /hZFLyvrERm9g9N7xKjKO9NUJrnHEBcwHHBg7sH5PBNQyieyFGG7wCvdZngIL5m3j+ ael+cAywtofFkarpwGqmsZLoLfjzo12yL3TkTj8rc0hI4v+RYuvJ7cnoHuhB0vZPUz Qziyito1Btp7hGtyoEZ9uNsobXFeDPhhKVJQYbd306KM6UCfmPQpZagHXHUAo/FDXE oax4sLpMKdbDA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 12/23] dma: swiotlb: pass mapping attributes by reference Date: Fri, 17 Jul 2026 23:34:30 +0530 Message-ID: <20260717180442.110954-13-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Change swiotlb_tbl_map_single() to take the DMA mapping attributes by reference and update the direct callers accordingly. This is a preparatory change for a follow-up patch which updates the attributes based on the selected swiotlb pool. Keeping the signature change separate makes the follow-up patch easier to review. No functional change in this patch. Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/swiotlb.h | 2 +- kernel/dma/swiotlb.c | 6 +++--- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 83bf15ae9777..c42e40e60575 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1180,7 +1180,7 @@ static phys_addr_t iommu_dma_map_swiotlb(struct devic= e *dev, phys_addr_t phys, trace_swiotlb_bounced(dev, phys, size); =20 phys =3D swiotlb_tbl_map_single(dev, phys, size, iova_mask(iovad), dir, - attrs); + &attrs); =20 /* * Untrusted devices should not see padding areas with random leftover diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 2cbf2b588f5b..8c4abe65cd49 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -243,7 +243,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, */ trace_swiotlb_bounced(dev, dev_addr, size); =20 - map =3D swiotlb_tbl_map_single(dev, phys, size, 0, dir, attrs); + map =3D swiotlb_tbl_map_single(dev, phys, size, 0, dir, &attrs); if (map =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index c92ff6791595..ea4c0a292dea 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -238,7 +238,7 @@ static inline phys_addr_t default_swiotlb_limit(void) =20 phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys, size_t mapping_size, unsigned int alloc_aligned_mask, - enum dma_data_direction dir, unsigned long attrs); + enum dma_data_direction dir, unsigned long *attrs); dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs); =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 908de28aceb2..046ae92c4832 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1396,7 +1396,7 @@ static unsigned long mem_used(struct io_tlb_mem *mem) */ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_ad= dr, size_t mapping_size, unsigned int alloc_align_mask, - enum dma_data_direction dir, unsigned long attrs) + enum dma_data_direction dir, unsigned long *attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; unsigned int offset; @@ -1430,7 +1430,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev= , phys_addr_t orig_addr, size =3D ALIGN(mapping_size + offset, alloc_align_mask + 1); index =3D swiotlb_find_slots(dev, orig_addr, size, alloc_align_mask, &poo= l); if (index =3D=3D -1) { - if (!(attrs & DMA_ATTR_NO_WARN)) + if (!(*attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slo= ts)\n", size, mem->nslabs, mem_used(mem)); @@ -1609,7 +1609,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_= t paddr, size_t size, =20 trace_swiotlb_bounced(dev, phys_to_dma(dev, paddr), size); =20 - swiotlb_addr =3D swiotlb_tbl_map_single(dev, paddr, size, 0, dir, attrs); + swiotlb_addr =3D swiotlb_tbl_map_single(dev, paddr, size, 0, dir, &attrs); if (swiotlb_addr =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29294378D7F; Fri, 17 Jul 2026 18:07:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311655; cv=none; b=s8yvYDww0MhE+L1ZrNVH3YHKNg07MgDMzajBSTwTp4z7yywQs6AxwgX4w4wuME5CWKisC8U9R61K6lwsOO7xDyKzaSquEtuAILa45b5IRpL4Wbt5CvPjr9TO6ev6eesm7uC3G2W5ijSSvAjFilJpG8ODqb8vm6r3CuICbUFCYMY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311655; c=relaxed/simple; bh=sGfQCA5jyO3DxDQvgLT5NKctUdqrbJS0gWz4iCcF9UQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MAKrz+tDPZefpDL1xhWgp7TM1VdmKR+PwAmMUeNyRz5YByEcZZap5MKVhpRstGTLf1REJD3lhxrDxDR4epLXIjfLvXMkvZdc0GvO8UEG2gE4RTspTxO6GLBZloIWZFyPwK43bu0QWyxrRJGrlh0TLqVh0RQ7P4RrOQQPp/iRCSo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oAPG1n5e; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oAPG1n5e" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D610B1F00A3A; Fri, 17 Jul 2026 18:07:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311652; bh=jY52qK7bnI8mIvfepyYFG9piQPAbkRue6lCaRwQnx1c=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=oAPG1n5e0sbwIV93ixqQeah8zx/EAd/G08hBuDD86OfVbzBwnKb+cprfDkVo4mi3q rGQGhJwGw8hQ1ercm8sx0l61BnXDeE+Gocjk8cbNKGthJ12M9RBCR4L0oy5Ii8jbcV LeY86pXT90vyLK89fARmd/6zSZSW6AleoRVWgjL9zp/hJV3HmB54+gk4e+p3+Gin+w /k+HnWMZavLdXpdAkX93DDDwuuakkHrRSeunn4E0q0joH0dOBajY8im0atvHWTh/LQ hB6Rh4kfiXQenFUIZ4PYaR6hAZFaG40ifVrRht/CHcbKUEjaZ8yH6GpoqUT6hadcUO lUXqPER7WVE+Q== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 13/23] dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED Date: Fri, 17 Jul 2026 23:34:31 +0530 Message-ID: <20260717180442.110954-14-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach swiotlb to distinguish between encrypted and decrypted bounce buffer pools, and make allocation and mapping paths select a pool whose state matches the requested DMA attributes. Add a cc_shared flag to io_tlb_mem, initialize it for the default and restricted pools, and propagate __DMA_ATTR_ALLOC_CC_SHARED into swiotlb pool allocation. Reject swiotlb alloc/map requests when the selected pool does not match the required encrypted/decrypted state. Also return DMA addresses with the matching phys_to_dma_{encrypted, unencrypted} helper so the DMA address encoding stays consistent with the chosen pool. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/dma-direct.h | 10 +++ include/linux/swiotlb.h | 10 ++- kernel/dma/direct.c | 13 ++- kernel/dma/swiotlb.c | 177 ++++++++++++++++++++++++++++--------- 4 files changed, 162 insertions(+), 48 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index c249912456f9..94fad4e7c11e 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -77,6 +77,10 @@ static inline dma_addr_t dma_range_map_max(const struct = bus_dma_region *map) #ifndef phys_to_dma_unencrypted #define phys_to_dma_unencrypted phys_to_dma #endif + +#ifndef phys_to_dma_encrypted +#define phys_to_dma_encrypted phys_to_dma +#endif #else static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t pad= dr) { @@ -90,6 +94,12 @@ static inline dma_addr_t phys_to_dma_unencrypted(struct = device *dev, { return dma_addr_unencrypted(__phys_to_dma(dev, paddr)); } + +static inline dma_addr_t phys_to_dma_encrypted(struct device *dev, + phys_addr_t paddr) +{ + return dma_addr_encrypted(__phys_to_dma(dev, paddr)); +} /* * If memory encryption is supported, phys_to_dma will set the memory encr= yption * bit in the DMA address, and dma_to_phys will clear it. diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index ea4c0a292dea..ee42f7588847 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -66,6 +66,7 @@ extern void __init swiotlb_update_mem_attributes(void); * @node: Member of the IO TLB memory pool list. * @rcu: RCU head for swiotlb_dyn_free(). * @transient: %true if transient memory pool. + * @cc_shared: %true if the pool memory is shared for confidential computi= ng. */ struct io_tlb_pool { phys_addr_t start; @@ -81,6 +82,7 @@ struct io_tlb_pool { struct list_head node; struct rcu_head rcu; bool transient; + bool cc_shared; #endif }; =20 @@ -92,6 +94,7 @@ struct io_tlb_pool { * @debugfs: The dentry to debugfs. * @force_bounce: %true if swiotlb bouncing is forced * @for_alloc: %true if the pool is used for memory allocation + * @cc_shared: %true if the pool memory is shared for confidential computi= ng. * @can_grow: %true if more pools can be allocated dynamically. * @phys_limit: Maximum allowed physical address. * @lock: Lock to synchronize changes to the list. @@ -111,6 +114,7 @@ struct io_tlb_mem { struct dentry *debugfs; bool force_bounce; bool for_alloc; + bool cc_shared; #ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow; u64 phys_limit; @@ -282,7 +286,8 @@ static inline void swiotlb_sync_single_for_cpu(struct d= evice *dev, extern void swiotlb_print_info(void); =20 #ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc(struct device *dev, size_t size); +struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs); bool swiotlb_free(struct device *dev, struct page *page, size_t size); void swiotlb_free_from_pool(struct device *dev, phys_addr_t tlb_addr, struct io_tlb_pool *pool); @@ -292,7 +297,8 @@ static inline bool is_swiotlb_for_alloc(struct device *= dev) return dev->dma_io_tlb_mem->for_alloc; } #else -static inline struct page *swiotlb_alloc(struct device *dev, size_t size) +static inline struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs) { return NULL; } diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 6960ee1cfefe..9f013ff8e0f6 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -96,9 +96,10 @@ static int dma_set_encrypted(struct device *dev, void *v= addr, size_t size) return ret; } =20 -static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze) +static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze, + unsigned long attrs) { - struct page *page =3D swiotlb_alloc(dev, size); + struct page *page =3D swiotlb_alloc(dev, size, attrs); =20 if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { swiotlb_free(dev, page, size); @@ -254,8 +255,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, } =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (page) { + /* + * swiotlb allocations comes from pool already marked + * decrypted + */ mark_mem_decrypt =3D false; goto setup_page; } @@ -401,7 +406,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, &ret, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (!page) return NULL; =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 046ae92c4832..335e27bc1e1f 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -259,10 +259,21 @@ void __init swiotlb_update_mem_attributes(void) struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; unsigned long bytes; =20 + /* + * if platform support memory encryption, swiotlb buffers are + * shared by default. + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + io_tlb_default_mem.cc_shared =3D true; + else + io_tlb_default_mem.cc_shared =3D false; + if (!mem->nslabs || mem->late_alloc) return; bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); - set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); + + if (io_tlb_default_mem.cc_shared) + set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, @@ -505,8 +516,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, if (!mem->slots) goto error_slots; =20 - set_memory_decrypted((unsigned long)vstart, - (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (io_tlb_default_mem.cc_shared) + set_memory_decrypted((unsigned long)vstart, + (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); @@ -539,7 +552,9 @@ void __init swiotlb_exit(void) tbl_size =3D PAGE_ALIGN(mem->end - mem->start); slots_size =3D PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs)); =20 - set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (io_tlb_default_mem.cc_shared) + set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (mem->late_alloc) { area_order =3D get_order(array_size(sizeof(*mem->areas), mem->nareas)); @@ -563,6 +578,7 @@ void __init swiotlb_exit(void) * @gfp: GFP flags for the allocation. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. + * @attrs: DMA attributes for the allocation. * * Allocate pages from the buddy allocator. If successful, make the alloca= ted * pages decrypted that they can be used for DMA. @@ -570,9 +586,11 @@ void __init swiotlb_exit(void) * Return: Decrypted pages, %NULL on allocation failure, or ERR_PTR(-EAGAI= N) * if the allocated physical address was above @phys_limit. */ -static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limi= t) +static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, + u64 phys_limit, unsigned long attrs) { unsigned int order =3D get_order(bytes); + bool cc_shared =3D attrs & __DMA_ATTR_ALLOC_CC_SHARED; struct page *page; phys_addr_t paddr; void *vaddr; @@ -588,13 +606,13 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) } =20 vaddr =3D phys_to_virt(paddr); - if (set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (cc_shared && set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes)= )) goto error; return page; =20 error: /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (cc_shared && !set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes= ))) __free_pages(page, order); return NULL; } @@ -602,6 +620,7 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t b= ytes, u64 phys_limit) /** * swiotlb_alloc_tlb() - allocate a dynamic IO TLB buffer * @dev: Device for which a memory pool is allocated. + * @mem: SWIOTLB allocator for the pool. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. * @gfp: GFP flags for the allocation. @@ -609,25 +628,23 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) * * Return: Allocated pages, or %NULL on allocation failure. */ -static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, +static struct page *swiotlb_alloc_tlb(struct device *dev, + struct io_tlb_mem *mem, size_t bytes, u64 phys_limit, gfp_t gfp, void **vaddr) { struct page *page; - unsigned long attrs =3D 0; + unsigned long attrs =3D mem->cc_shared ? __DMA_ATTR_ALLOC_CC_SHARED : 0; =20 *vaddr =3D NULL; /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ - if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) { + if (!gfpflags_allow_blocking(gfp) && dev && mem->cc_shared) { + if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 - /* swiotlb considered decrypted by default */ - if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) - attrs =3D __DMA_ATTR_ALLOC_CC_SHARED; - return dma_alloc_from_pool(dev, bytes, vaddr, gfp, attrs, dma_coherent_ok); } @@ -638,7 +655,7 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, else if (phys_limit <=3D DMA_BIT_MASK(32)) gfp |=3D __GFP_DMA32; =20 - while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit))) { + while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit, attrs))) { if (IS_ENABLED(CONFIG_ZONE_DMA32) && phys_limit < DMA_BIT_MASK(64) && !(gfp & (__GFP_DMA32 | __GFP_DMA))) @@ -659,21 +676,25 @@ static struct page *swiotlb_alloc_tlb(struct device *= dev, size_t bytes, * swiotlb_free_tlb() - free a dynamically allocated IO TLB buffer * @vaddr: Virtual address of the buffer. * @bytes: Size of the buffer. + * @cc_shared: true if @vaddr was allocated decrypted and must be + * re-encrypted before being freed */ -static void swiotlb_free_tlb(void *vaddr, size_t bytes) +static void swiotlb_free_tlb(void *vaddr, size_t bytes, bool cc_shared) { if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool(NULL, vaddr, bytes)) return; =20 /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (!cc_shared || + !set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) __free_pages(virt_to_page(vaddr), get_order(bytes)); } =20 /** * swiotlb_alloc_pool() - allocate a new IO TLB memory pool * @dev: Device for which a memory pool is allocated. + * @mem: SWIOTLB allocator for the pool. * @minslabs: Minimum number of slabs. * @nslabs: Desired (maximum) number of slabs. * @nareas: Number of areas. @@ -687,8 +708,9 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes) * Return: New memory pool, or %NULL on allocation failure. */ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, - unsigned long minslabs, unsigned long nslabs, - unsigned int nareas, u64 phys_limit, gfp_t gfp) + struct io_tlb_mem *mem, unsigned long minslabs, + unsigned long nslabs, unsigned int nareas, u64 phys_limit, + gfp_t gfp) { struct io_tlb_pool *pool; unsigned int slot_order; @@ -707,10 +729,11 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct = device *dev, if (!pool) goto error; pool->areas =3D (void *)pool + sizeof(*pool); + pool->cc_shared =3D mem->cc_shared; =20 tlb_size =3D nslabs << IO_TLB_SHIFT; - while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp, - &tlb_vaddr))) { + while (!(tlb =3D swiotlb_alloc_tlb(dev, mem, tlb_size, + phys_limit, gfp, &tlb_vaddr))) { if (nslabs <=3D minslabs) goto error_tlb; nslabs =3D ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -729,7 +752,7 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, return pool; =20 error_slots: - swiotlb_free_tlb(tlb_vaddr, tlb_size); + swiotlb_free_tlb(tlb_vaddr, tlb_size, mem->cc_shared); error_tlb: kfree(pool); error: @@ -746,7 +769,7 @@ static void swiotlb_dyn_alloc(struct work_struct *work) container_of(work, struct io_tlb_mem, dyn_alloc); struct io_tlb_pool *pool; =20 - pool =3D swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs, + pool =3D swiotlb_alloc_pool(NULL, mem, IO_TLB_MIN_SLABS, default_nslabs, default_nareas, mem->phys_limit, GFP_KERNEL); if (!pool) { pr_warn_ratelimited("Failed to allocate new pool"); @@ -767,7 +790,7 @@ static void swiotlb_dyn_free(struct rcu_head *rcu) size_t tlb_size =3D pool->end - pool->start; =20 free_pages((unsigned long)pool->slots, get_order(slots_size)); - swiotlb_free_tlb(pool->vaddr, tlb_size); + swiotlb_free_tlb(pool->vaddr, tlb_size, pool->cc_shared); kfree(pool); } =20 @@ -1031,6 +1054,7 @@ static void dec_transient_used(struct io_tlb_mem *mem= , unsigned int nslots) * @pool: Memory pool to be searched. * @area_index: Index of the IO TLB memory area to be searched. * @orig_addr: Original (non-bounced) IO buffer address. + * @tbl_dma_addr: DMA address of the bounce buffer. * @alloc_size: Total requested size of the bounce buffer, * including initial alignment padding. * @alloc_align_mask: Required alignment of the allocated buffer. @@ -1042,13 +1066,11 @@ static void dec_transient_used(struct io_tlb_mem *m= em, unsigned int nslots) * Return: Index of the first allocated slot, or -1 on error. */ static int swiotlb_search_pool_area(struct device *dev, struct io_tlb_pool= *pool, - int area_index, phys_addr_t orig_addr, size_t alloc_size, - unsigned int alloc_align_mask) + int area_index, phys_addr_t orig_addr, dma_addr_t tbl_dma_addr, + size_t alloc_size, unsigned int alloc_align_mask) { struct io_tlb_area *area =3D pool->areas + area_index; unsigned long boundary_mask =3D dma_get_seg_boundary(dev); - dma_addr_t tbl_dma_addr =3D - phys_to_dma_unencrypted(dev, pool->start) & boundary_mask; unsigned long max_slots =3D get_max_slots(boundary_mask); unsigned int iotlb_align_mask =3D dma_get_min_align_mask(dev); unsigned int nslots =3D nr_slots(alloc_size), stride; @@ -1061,6 +1083,8 @@ static int swiotlb_search_pool_area(struct device *de= v, struct io_tlb_pool *pool BUG_ON(!nslots); BUG_ON(area_index >=3D pool->nareas); =20 + tbl_dma_addr &=3D boundary_mask; + /* * Historically, swiotlb allocations >=3D PAGE_SIZE were guaranteed to be * page-aligned in the absence of any other alignment requirements. @@ -1172,6 +1196,7 @@ static int swiotlb_search_area(struct device *dev, in= t start_cpu, { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; int area_index; int index =3D -1; =20 @@ -1180,9 +1205,15 @@ static int swiotlb_search_area(struct device *dev, i= nt start_cpu, if (cpu_offset >=3D pool->nareas) continue; area_index =3D (start_cpu + cpu_offset) & (pool->nareas - 1); + + if (mem->cc_shared) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + index =3D swiotlb_search_pool_area(dev, pool, area_index, - orig_addr, alloc_size, - alloc_align_mask); + orig_addr, tbl_dma_addr, + alloc_size, alloc_align_mask); if (index >=3D 0) { *retpool =3D pool; break; @@ -1212,6 +1243,7 @@ static int swiotlb_find_slots(struct device *dev, phy= s_addr_t orig_addr, { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; unsigned long nslabs; unsigned long flags; u64 phys_limit; @@ -1236,12 +1268,17 @@ static int swiotlb_find_slots(struct device *dev, p= hys_addr_t orig_addr, =20 nslabs =3D nr_slots(alloc_size); phys_limit =3D min_not_zero(*dev->dma_mask, dev->bus_dma_limit); - pool =3D swiotlb_alloc_pool(dev, nslabs, nslabs, 1, phys_limit, + pool =3D swiotlb_alloc_pool(dev, mem, nslabs, nslabs, 1, phys_limit, GFP_NOWAIT); if (!pool) return -1; =20 - index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, + if (mem->cc_shared) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + + index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, tbl_dma_addr, alloc_size, alloc_align_mask); if (index < 0) { swiotlb_dyn_free(&pool->rcu); @@ -1286,15 +1323,23 @@ static int swiotlb_find_slots(struct device *dev, p= hys_addr_t orig_addr, size_t alloc_size, unsigned int alloc_align_mask, struct io_tlb_pool **retpool) { + struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; int start, i; int index; =20 - *retpool =3D pool =3D &dev->dma_io_tlb_mem->defpool; + *retpool =3D pool =3D &mem->defpool; + if (mem->cc_shared) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + i =3D start =3D raw_smp_processor_id() & (pool->nareas - 1); do { index =3D swiotlb_search_pool_area(dev, pool, i, orig_addr, - alloc_size, alloc_align_mask); + tbl_dma_addr, alloc_size, + alloc_align_mask); if (index >=3D 0) return index; if (++i >=3D pool->nareas) @@ -1377,9 +1422,19 @@ static unsigned long mem_used(struct io_tlb_mem *mem) * any pre- or post-padding for alignment * @alloc_align_mask: Required start and end alignment of the allocated bu= ffer * @dir: DMA direction - * @attrs: Optional DMA attributes for the map operation + * @attrs: Optional DMA attributes for the map operation, updated + * to match the selected SWIOTLB pool * * Find and allocate a suitable sequence of IO TLB slots for the request. + * The device's SWIOTLB pool must match the device's current DMA encryption + * requirements. If the device requires decrypted DMA, bouncing is done th= rough + * an unencrypted pool and the mapping is marked shared. If the device can= DMA + * to encrypted memory, bouncing is done through an encrypted pool even wh= en the + * original DMA address was unencrypted. Enabling encrypted DMA for a devi= ce is + * therefore expected to update its default io_tlb_mem to an encrypted poo= l, so + * later bounce mappings for both encrypted and decrypted original memory = use + * that encrypted pool. + * * The allocated space starts at an alignment specified by alloc_align_mas= k, * and the size of the allocated space is rounded up so that the total amo= unt * of allocated space is a multiple of (alloc_align_mask + 1). If @@ -1416,6 +1471,30 @@ phys_addr_t swiotlb_tbl_map_single(struct device *de= v, phys_addr_t orig_addr, if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) pr_warn_once("Memory encryption is active and system is using DMA bounce= buffers\n"); =20 + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { + + /* swiotlb pool is incorrect for this device */ + if (unlikely(mem->cc_shared !=3D force_dma_unencrypted(dev))) + return (phys_addr_t)DMA_MAPPING_ERROR; + + } else if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) { + /* + * On hosts with memory encryption, SWIOTLB-backed memory is + * unencrypted. DMA addresses returned for bounce buffers must + * therefore be marked unencrypted, even for devices that can + * address encrypted memory. This also preserves swiotlb=3Dforce + * behavior for those devices. + */ + if (unlikely(!mem->cc_shared)) + return (phys_addr_t)DMA_MAPPING_ERROR; + } + + /* Force attrs to match the kind of memory in the pool */ + if (mem->cc_shared) + *attrs |=3D DMA_ATTR_CC_SHARED; + else + *attrs &=3D ~DMA_ATTR_CC_SHARED; + /* * The default swiotlb memory pool is allocated with PAGE_SIZE * alignment. If a mapping is requested with larger alignment, @@ -1613,8 +1692,11 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr= _t paddr, size_t size, if (swiotlb_addr =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 - /* Ensure that the address returned is DMA'ble */ - dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + if (attrs & DMA_ATTR_CC_SHARED) + dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + else + dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, @@ -1778,7 +1860,7 @@ static inline void swiotlb_create_debugfs_files(struc= t io_tlb_mem *mem, =20 #ifdef CONFIG_DMA_RESTRICTED_POOL =20 -struct page *swiotlb_alloc(struct device *dev, size_t size) +struct page *swiotlb_alloc(struct device *dev, size_t size, unsigned long = attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; @@ -1789,6 +1871,9 @@ struct page *swiotlb_alloc(struct device *dev, size_t= size) if (!mem) return NULL; =20 + if (mem->cc_shared !=3D !!(attrs & __DMA_ATTR_ALLOC_CC_SHARED)) + return NULL; + align =3D (1 << (get_order(size) + PAGE_SHIFT)) - 1; index =3D swiotlb_find_slots(dev, 0, size, align, &pool); if (index =3D=3D -1) @@ -1864,12 +1949,20 @@ static int rmem_swiotlb_device_init(struct reserved= _mem *rmem, kfree(mem); return -ENOMEM; } + /* + * if platform supports memory encryption, + * restricted mem pool is shared by default + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + mem->cc_shared =3D true; + set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), + rmem->size >> PAGE_SHIFT); + } else { + mem->cc_shared =3D false; + } =20 - set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), - rmem->size >> PAGE_SHIFT); swiotlb_init_io_tlb_pool(pool, rmem->base, phys_to_virt(rmem->base), - nslabs, - false, nareas); + nslabs, false, nareas); mem->force_bounce =3D true; mem->for_alloc =3D true; #ifdef CONFIG_SWIOTLB_DYNAMIC --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFEEA37EFFD; Fri, 17 Jul 2026 18:07:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311666; cv=none; b=sunIS2CluwYAONQ44I4bp1dBDNQy9dk+Y1PvkvrgZNYkp0jpAEEzk5cI6LY58AWa1ulUnP/d0ISyUCcmYnfws9Jut56RGgkpU2+SKgp3bV9i5dewNZ7maO3XAZc3nbbSVDwL6Mj2asjt1HO6PyO8HQCMrUTxo2vUB2n2mVdJF3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311666; c=relaxed/simple; bh=znpDaT8XBbyKxa5j02rof9vGu9nD8Neg7ktKUih5OAg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g3rmOg0nf7QJdEVpbodQxZHdAkywnY2ur78cs6d2IOmGvP67VsUntMOhW9pDMyJ943skj3qlO0Lcs1YLHLJHz7eVO0lZVnf9BNVw2HTHv4RWItZ5jGT09PGh13OgQDhBdj34i2MaggP+ffWEKRpq++8Y+gNP1doeyBZaeRODk7k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jpXbh1c5; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jpXbh1c5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 863641F000E9; Fri, 17 Jul 2026 18:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311664; bh=IkFBlHM9TbaSw6EdZs7ctVNZjEKNsaSG21JkeuDmlSA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=jpXbh1c5NN9ni0L+tVVrdZw70rzNXRJeifg5Rs/lMab41T8Ylgnz8PO1aCzPc4UHg zHAQUYGizCjDNOd5OxUjZjeAC2NAP1v7Wv0ZyBAxekfXyYIj4PYrz6UkHpGaudfU27 QpYd7iiOSfMwgDy+5KmPMPzceGNno1Yh8HhdcxaYv7fTs+WmnqQs267gcGh3z0okPF yahaiX2oPFpc0Ny5bn0+THR1w9Euxp/wm+CfdIDuEaLQZCw+QecQhh29k/AUSOq3R8 S6tjDMuVdv9DmKCpR3AWQqqc0MHV/1S84d8XGGn8rbRmNu9QEpdMaxKkPwyP4snICT A3qnwGSNszIyQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 14/23] dma-mapping: make dma_pgprot() honor __DMA_ATTR_ALLOC_CC_SHARED Date: Fri, 17 Jul 2026 23:34:32 +0530 Message-ID: <20260717180442.110954-15-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold encrypted/decrypted pgprot selection into dma_pgprot() so callers do not need to adjust the page protection separately. Update dma_pgprot() to apply pgprot_decrypted() when DMA_ATTR_CC_SHARED or __DMA_ATTR_ALLOC_CC_SHARED is set and pgprot_encrypted() otherwise Convert the dma-direct mmap paths to pass DMA_ATTR_CC_SHARED instead of open-coding force_dma_unencrypted() handling around dma_pgprot(). Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 8 +++----- kernel/dma/mapping.c | 16 ++++++++++++---- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 9f013ff8e0f6..f54c7929b97f 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -286,9 +286,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 - if (force_dma_unencrypted(dev)) - prot =3D pgprot_decrypted(prot); - /* remove any dirty cache lines on the kernel alias */ arch_dma_prep_coherent(page, size); =20 @@ -607,9 +604,10 @@ int dma_direct_mmap(struct device *dev, struct vm_area= _struct *vma, unsigned long pfn =3D PHYS_PFN(dma_to_phys(dev, dma_addr)); int ret =3D -ENXIO; =20 - vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); if (force_dma_unencrypted(dev)) - vma->vm_page_prot =3D pgprot_decrypted(vma->vm_page_prot); + attrs |=3D DMA_ATTR_CC_SHARED; + + vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); =20 if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) return ret; diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index d2f70b6ccd0f..a628820fd10e 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -537,13 +537,21 @@ EXPORT_SYMBOL(dma_get_sgtable_attrs); */ pgprot_t dma_pgprot(struct device *dev, pgprot_t prot, unsigned long attrs) { + pgprot_t dma_prot; + if (dev_is_dma_coherent(dev)) - return prot; + dma_prot =3D prot; #ifdef CONFIG_ARCH_HAS_DMA_WRITE_COMBINE - if (attrs & DMA_ATTR_WRITE_COMBINE) - return pgprot_writecombine(prot); + else if (attrs & DMA_ATTR_WRITE_COMBINE) + dma_prot =3D pgprot_writecombine(prot); #endif - return pgprot_dmacoherent(prot); + else + dma_prot =3D pgprot_dmacoherent(prot); + + if (attrs & (DMA_ATTR_CC_SHARED | __DMA_ATTR_ALLOC_CC_SHARED)) + return pgprot_decrypted(dma_prot); + else + return pgprot_encrypted(dma_prot); } #endif /* CONFIG_MMU */ =20 --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C613A37D126; Fri, 17 Jul 2026 18:07:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311678; cv=none; b=dxSTYDTcl22v1SejaU3eQ5w1nKnvNkZnhSO72aDt8MsNetVXeFQF83H8UBe5tpYbfUq1XU9Exn2logf+DRe5Jr7Dnug3mBtJ73xlNFBCRrXiuqOhwoLqA2RAjVO3buT+SBJ87ifHKZbRfq6DgKgFWG3ezOqs5WqL+Q9PjCrH/AQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311678; c=relaxed/simple; bh=0RUOPeLGCgxi8j6Zn+fAmTTgkNTa20monc8btpgm3AA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GK26Wbkvghg1/COsjjBHCweuMakfEb5C2Pa7WBDGGYZeV+5c5J8U8qAzVi2ogcsT0Vh05ylNPDPlczGRyRP7lfvouS4u5mHcGbF8Gd7d9XF1nFa+5MYdYv3uYe0a0JHFlpNEL3T0YqbjTmQTep72yUWvcdLnJ47vI0p5T+Ir0ME= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hTfBgcLf; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hTfBgcLf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 819A91F00A3A; Fri, 17 Jul 2026 18:07:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311676; bh=ooR93KHObwoAe5YKSjF58LPYSw/01Lnf6Ipp2w5o86o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=hTfBgcLfrLrE2QoJnVMKz1ivhWK6Hjx0E/u9Xf2wdvQFnMM+oIAwFuo/zFBluhaUz rg08HvWaUpCBCLC25KCdX6MDl9rCszkmsV+DMLezXA5JkoD1iMKlOa8MrCr6ay0KHr JRivx1jYAbhDJZM5DujEc/zNd8mN21wVQmEB6k8FNI4t/p8nmxU75qqOqjCg4i7nAg TizkOymHDI+uWyVd/yO+8Q3CgPglCzpYPJFV35WAqIbL27Y7S0BJt6NMUV2J1n2bVd Iiew//l2+W7chPSl5/q3WUiPu+S9DHhLLwpf20b8RPWqXHKhF8/FU4IwQgMQdkDIGW L+gIO4n5/WSMQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 15/23] dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks Date: Fri, 17 Jul 2026 23:34:33 +0530 Message-ID: <20260717180442.110954-16-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability check can reject encrypted DMA addresses for devices that require unencrypted/shared DMA. Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected SWIOTLB pool is decrypted so the capability check sees the correct DMA address attribute. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- arch/x86/kernel/amd_gart_64.c | 30 ++++++++++++++++-------------- drivers/xen/swiotlb-xen.c | 6 +++--- include/linux/dma-direct.h | 10 +++++++++- kernel/dma/direct.h | 6 +++--- kernel/dma/swiotlb.c | 2 +- 5 files changed, 32 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index e8000a56732e..b5f1f031d45b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -180,22 +180,23 @@ static void iommu_full(struct device *dev, size_t siz= e, int dir) } =20 static inline int -need_iommu(struct device *dev, unsigned long addr, size_t size) +need_iommu(struct device *dev, unsigned long addr, size_t size, unsigned l= ong attrs) { - return force_iommu || !dma_capable(dev, addr, size, true); + return force_iommu || !dma_capable(dev, addr, size, true, attrs); } =20 static inline int -nonforced_iommu(struct device *dev, unsigned long addr, size_t size) +nonforced_iommu(struct device *dev, unsigned long addr, size_t size, + unsigned long attrs) { - return !dma_capable(dev, addr, size, true); + return !dma_capable(dev, addr, size, true, attrs); } =20 /* Map a single continuous physical area into the IOMMU. * Caller needs to check if the iommu is needed and flush. */ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem, - size_t size, int dir, unsigned long align_mask) + size_t size, int dir, unsigned long align_mask, unsigned long attrs) { unsigned long npages =3D iommu_num_pages(phys_mem, size, PAGE_SIZE); unsigned long iommu_page; @@ -206,7 +207,7 @@ static dma_addr_t dma_map_area(struct device *dev, dma_= addr_t phys_mem, =20 iommu_page =3D alloc_iommu(dev, npages, align_mask); if (iommu_page =3D=3D -1) { - if (!nonforced_iommu(dev, phys_mem, size)) + if (!nonforced_iommu(dev, phys_mem, size, attrs)) return phys_mem; if (panic_on_overflow) panic("dma_map_area overflow %lu bytes\n", size); @@ -231,10 +232,10 @@ static dma_addr_t gart_map_phys(struct device *dev, p= hys_addr_t paddr, if (unlikely(attrs & DMA_ATTR_MMIO)) return DMA_MAPPING_ERROR; =20 - if (!need_iommu(dev, paddr, size)) + if (!need_iommu(dev, paddr, size, attrs)) return paddr; =20 - bus =3D dma_map_area(dev, paddr, size, dir, 0); + bus =3D dma_map_area(dev, paddr, size, dir, 0, attrs); flush_gart(); =20 return bus; @@ -289,7 +290,7 @@ static void gart_unmap_sg(struct device *dev, struct sc= atterlist *sg, int nents, =20 /* Fallback for dma_map_sg in case of overflow */ static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg, - int nents, int dir) + int nents, int dir, unsigned long attrs) { struct scatterlist *s; int i; @@ -301,8 +302,8 @@ static int dma_map_sg_nonforce(struct device *dev, stru= ct scatterlist *sg, for_each_sg(sg, s, nents, i) { unsigned long addr =3D sg_phys(s); =20 - if (nonforced_iommu(dev, addr, s->length)) { - addr =3D dma_map_area(dev, addr, s->length, dir, 0); + if (nonforced_iommu(dev, addr, s->length, attrs)) { + addr =3D dma_map_area(dev, addr, s->length, dir, 0, attrs); if (addr =3D=3D DMA_MAPPING_ERROR) { if (i > 0) gart_unmap_sg(dev, sg, i, dir, 0); @@ -401,7 +402,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, s->dma_address =3D addr; BUG_ON(s->length =3D=3D 0); =20 - nextneed =3D need_iommu(dev, addr, s->length); + nextneed =3D need_iommu(dev, addr, s->length, attrs); =20 /* Handle the previous not yet processed entries */ if (i > start) { @@ -449,7 +450,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, =20 /* When it was forced or merged try again in a dumb way */ if (force_iommu || iommu_merge) { - out =3D dma_map_sg_nonforce(dev, sg, nents, dir); + out =3D dma_map_sg_nonforce(dev, sg, nents, dir, attrs); if (out > 0) return out; } @@ -473,7 +474,8 @@ gart_alloc_coherent(struct device *dev, size_t size, dm= a_addr_t *dma_addr, return vaddr; =20 *dma_addr =3D dma_map_area(dev, virt_to_phys(vaddr), size, - DMA_BIDIRECTIONAL, (1UL << get_order(size)) - 1); + DMA_BIDIRECTIONAL, + (1UL << get_order(size)) - 1, attrs); flush_gart(); if (unlikely(*dma_addr =3D=3D DMA_MAPPING_ERROR)) goto out_free; diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 8c4abe65cd49..e2538824ef52 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -212,7 +212,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, BUG_ON(dir =3D=3D DMA_NONE); =20 if (attrs & DMA_ATTR_MMIO) { - if (unlikely(!dma_capable(dev, phys, size, false))) { + if (unlikely(!dma_capable(dev, phys, size, false, attrs))) { dev_err_once( dev, "DMA addr %pa+%zu overflow (mask %llx, bus limit %llx).\n", @@ -231,7 +231,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, * we can safely return the device addr and not worry about bounce * buffering it. */ - if (dma_capable(dev, dev_addr, size, true) && + if (dma_capable(dev, dev_addr, size, true, attrs) && !dma_kmalloc_needs_bounce(dev, size, dir) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && @@ -253,7 +253,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, /* * Ensure that the address returned is DMA'ble */ - if (unlikely(!dma_capable(dev, dev_addr, size, true))) { + if (unlikely(!dma_capable(dev, dev_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, map, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, map)); diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 94fad4e7c11e..daa31a1adf7b 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -135,12 +135,20 @@ static inline bool force_dma_unencrypted(struct devic= e *dev) #endif /* CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED */ =20 static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t= size, - bool is_ram) + bool is_ram, unsigned long attrs) { dma_addr_t end =3D addr + size - 1; =20 if (addr =3D=3D DMA_MAPPING_ERROR) return false; + /* + * The DMA address was derived from encrypted RAM, but this device + * requires unencrypted DMA addresses. Treat it as not DMA-capable + * so the caller can fall back to a suitable SWIOTLB pool. + */ + if (!(attrs & DMA_ATTR_CC_SHARED) && force_dma_unencrypted(dev)) + return false; + if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) && min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn))) return false; diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index 7140c208c123..e05dc7649366 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -101,15 +101,15 @@ static inline dma_addr_t dma_direct_map_phys(struct d= evice *dev, =20 if (attrs & DMA_ATTR_MMIO) { dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else if (attrs & DMA_ATTR_CC_SHARED) { dma_addr =3D phys_to_dma_unencrypted(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else { dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_swiotlb_active(dev) && !(attrs & DMA_ATTR_REQUIRE_COHERENT)) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 335e27bc1e1f..b5960c2e98d8 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1697,7 +1697,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_= t paddr, size_t size, else dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); =20 - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, swiotlb_addr)); --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B4FA2374A1D; Fri, 17 Jul 2026 18:08:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311688; cv=none; b=OTtJrQfyFRn34CTJW3FrsS6HaM+coSOkq+J1z0cTTbMTKSMTj8MTPkr1BPF4Jzhhwcu/ty/gCUumUnMdxIT3FSBiFLDrmPqgh1fDEJtEbNTFig8TcbvHmK3UOoyJuMc8CcBvDOXuLrYxJx7AbQ553CIC5A1iCvWEt5spbJcAOzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311688; c=relaxed/simple; bh=nMgLq/UbqOEm2Qt5oKdXfCYamVKavI4EwsnV2gmHiug=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KmCw7hXsfBq+JWcSDalcL8L5WvK3LaG+cy8A7MyKb9iRisM80llNKO24ygdRo0mjSkVqXtTk/g1ivITVFoLj2BmrszcL5orW4gXhKxoO2Rqw9G2XED7pa3IaHGVMVOjaM819fL+NbzbYxNAjT1d9ZLMQsoI2tJMdrKaT0bAxsqE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BP5iTTw7; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BP5iTTw7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4ABC81F000E9; Fri, 17 Jul 2026 18:07:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311687; bh=PwGwo2v1bc/tfRKKTB7zzFn/AHGX4mQVp7Ym9P3mvNc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=BP5iTTw71m+JzUVT4oQamNjA8tRrSdlrhJHPabvG4jvbiWwSchW6LscApjzvzfSYm MQCa8o0m1ipdDX7SdMuLHgTZ4IWXQxJktMmbiNLIQ+gJUPsFyi0XLXhJwYwxMpWaiL SWcUDH7wvsith+5+bko8GxVl0wq3lBLtnfCOZfDXmaHKztP/qHL3IowfvzKQ5EzoBP h4t+FDvFCW6cSzB18rA8aSxbkoiIIlsu9iUi8pNYrKPB6S3FtoxcP4so3dP5UOSKr3 fNoA1XnRHK/caUQEAWeczcbWRQAFkU5QeVq6grwcH/uSyIyJ/o5fqvAYN618S/gMjP dLx40xNRBhQtw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v8 16/23] dma-direct: Move dma_direct_map_phys() to dma/direct.c Date: Fri, 17 Jul 2026 23:34:34 +0530 Message-ID: <20260717180442.110954-17-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" dma_direct_map_phys() is too large to benefit from being inlined. Move its implementation to direct.c and leave the declaration in direct.h. No functional change in this patch Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 53 ++++++++++++++++++++++++++++++++++++++++++ kernel/dma/direct.h | 56 +++------------------------------------------ 2 files changed, 56 insertions(+), 53 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f54c7929b97f..2413235c50f0 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -620,6 +620,59 @@ int dma_direct_mmap(struct device *dev, struct vm_area= _struct *vma, user_count << PAGE_SHIFT, vma->vm_page_prot); } =20 +dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys, + size_t size, enum dma_data_direction dir, + unsigned long attrs, bool flush) +{ + dma_addr_t dma_addr; + + if (is_swiotlb_force_bounce(dev)) { + if (!(attrs & DMA_ATTR_CC_SHARED)) { + if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) + return DMA_MAPPING_ERROR; + + return swiotlb_map(dev, phys, size, dir, attrs); + } + } else if (attrs & DMA_ATTR_CC_SHARED) { + return DMA_MAPPING_ERROR; + } + + if (attrs & DMA_ATTR_MMIO) { + dma_addr =3D phys; + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) + goto err_overflow; + } else if (attrs & DMA_ATTR_CC_SHARED) { + dma_addr =3D phys_to_dma_unencrypted(dev, phys); + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) + goto err_overflow; + } else { + dma_addr =3D phys_to_dma(dev, phys); + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { + if (is_swiotlb_active(dev) && + !(attrs & DMA_ATTR_REQUIRE_COHERENT)) + return swiotlb_map(dev, phys, size, dir, attrs); + + goto err_overflow; + } + } + + if (!dev_is_dma_coherent(dev) && + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { + arch_sync_dma_for_device(phys, size, dir); + if (flush) + arch_sync_dma_flush(); + } + return dma_addr; + +err_overflow: + dev_WARN_ONCE( + dev, 1, + "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); + return DMA_MAPPING_ERROR; +} + int dma_direct_supported(struct device *dev, u64 mask) { u64 min_mask =3D ((u64)max_pfn << PAGE_SHIFT) - 1; diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e05dc7649366..a7adadb1b2a5 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -17,6 +17,9 @@ bool dma_direct_can_mmap(struct device *dev); int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); +dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys, + size_t size, enum dma_data_direction dir, + unsigned long attrs, bool flush); bool dma_direct_need_sync(struct device *dev, dma_addr_t dma_addr); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nen= ts, enum dma_data_direction dir, unsigned long attrs); @@ -82,59 +85,6 @@ static inline void dma_direct_sync_single_for_cpu(struct= device *dev, swiotlb_sync_single_for_cpu(dev, paddr, size, dir); } =20 -static inline dma_addr_t dma_direct_map_phys(struct device *dev, - phys_addr_t phys, size_t size, enum dma_data_direction dir, - unsigned long attrs, bool flush) -{ - dma_addr_t dma_addr; - - if (is_swiotlb_force_bounce(dev)) { - if (!(attrs & DMA_ATTR_CC_SHARED)) { - if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) - return DMA_MAPPING_ERROR; - - return swiotlb_map(dev, phys, size, dir, attrs); - } - } else if (attrs & DMA_ATTR_CC_SHARED) { - return DMA_MAPPING_ERROR; - } - - if (attrs & DMA_ATTR_MMIO) { - dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) - goto err_overflow; - } else if (attrs & DMA_ATTR_CC_SHARED) { - dma_addr =3D phys_to_dma_unencrypted(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) - goto err_overflow; - } else { - dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || - dma_kmalloc_needs_bounce(dev, size, dir)) { - if (is_swiotlb_active(dev) && - !(attrs & DMA_ATTR_REQUIRE_COHERENT)) - return swiotlb_map(dev, phys, size, dir, attrs); - - goto err_overflow; - } - } - - if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { - arch_sync_dma_for_device(phys, size, dir); - if (flush) - arch_sync_dma_flush(); - } - return dma_addr; - -err_overflow: - dev_WARN_ONCE( - dev, 1, - "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); - return DMA_MAPPING_ERROR; -} - static inline void dma_direct_unmap_phys(struct device *dev, dma_addr_t ad= dr, size_t size, enum dma_data_direction dir, unsigned long attrs, bool flush) --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4939340417; Fri, 17 Jul 2026 18:08:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311700; cv=none; b=bLr/3E3wSfEtX/dynhywlijTi1NVAwNBLdjT4alFS23xkl5CJhk0Mlfi5qrXOtlj7oH1x6EhQ5AdDx4VjTZ0xDXsYvMxkymAqgLW9bRSxBPBH+yCsA9Y8VwdCFUSwvqHjUpO5aNzJRTeHzdt1DdzPDkPZoYjhNRKksV9RerYHqA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311700; c=relaxed/simple; bh=Xq24ApDCDZEF81VDksq1HvzM/P+yTjCmGsP2m4R0Shk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YDo/kdtzRCevpei3AaNijP20shpJfdHA5awdXzOLOuroWkQz8AsMQ8z4rPwIepbTcMZ5fxl/Un1jt9NJ+DbRDM/PEtHU4djLfaf0h+UDCSr/rpJoOSRI6yCuJuQv2tagvwuispsB3vVsFt7l8caspTuXuo9C/glmuyDe+bl1LSs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UcA7rv5C; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UcA7rv5C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4003F1F00A3A; Fri, 17 Jul 2026 18:08:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311699; bh=CV8QBh8RHI5mzqb2fm7QecW+bWIQzre8IVIoWdK1GKs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=UcA7rv5CVRknaqmMkLhCn+0BkGdRgLuDGal93rx/bnxSs87OrIsgv6UlyCzhge900 WWGpujZyRUd0jzlt2kT+uztP/P8EU2SyAW2q2Bv0GNfVZnPyvq0sEzP6RkV/cQUhx/ dGB0nDNgVJaJFd8xnO1rKvZiRXdHK4j3a7bymP/ZQxLhXPqsOcB7GfOiMYybllVRrV EQLBKLPCShaco9snoCCdH+AjuFbu9LdqrsP3UqrA9e81PNcWmoG9ipa4T+ixJFfis9 R91MEf+DW2egIu2h7wMcSFTjBITZjCj6owfXUXnOzdtRu8QGy/gTniAoG1ozoiSx3o 04NmOFLPiPWGA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v8 17/23] dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED Date: Fri, 17 Jul 2026 23:34:35 +0530 Message-ID: <20260717180442.110954-18-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_direct_map_phys() to select the DMA address encoding based on DMA_ATTR_CC_SHARED. Use phys_to_dma_unencrypted() for decrypted mappings and phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA but the source physical address is still encrypted, force the mapping through swiotlb so the DMA address and backing memory attributes remain consistent. Update the arm64, x86, s390 and powerpc secure-guest setup to not use swiotlb force option Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- Changes from v3: * Handle DMA_ATTR_MMIO --- arch/arm64/mm/init.c | 4 +-- arch/powerpc/platforms/pseries/svm.c | 2 +- arch/s390/mm/init.c | 2 +- arch/x86/kernel/pci-dma.c | 4 +-- kernel/dma/direct.c | 52 ++++++++++++++++------------ 5 files changed, 34 insertions(+), 30 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 48d69e390158..3c390ef7b41b 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -340,9 +340,7 @@ void __init arch_mm_preinit(void) unsigned int flags =3D SWIOTLB_VERBOSE; bool limited_addressing =3D max_pfn > PFN_DOWN(arm64_dma_phys_limit); =20 - if (is_realm_world() || is_protected_kvm_guest()) { - flags |=3D SWIOTLB_FORCE; - } else if (!limited_addressing) { + if (!limited_addressing) { /* * If no bouncing needed for ZONE_DMA, reduce the swiotlb * buffer for kmalloc() bouncing to 1MB per 1GB of RAM. diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/= pseries/svm.c index 384c9dc1899a..7a403dbd35ee 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -29,7 +29,7 @@ static int __init init_svm(void) * need to use the SWIOTLB buffer for DMA even if dma_capable() says * otherwise. */ - ppc_swiotlb_flags |=3D SWIOTLB_ANY | SWIOTLB_FORCE; + ppc_swiotlb_flags |=3D SWIOTLB_ANY; =20 /* Share the SWIOTLB buffer with the host. */ swiotlb_update_mem_attributes(); diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index 6b1c5a4fa9ce..8d1de5a2e554 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -166,7 +166,7 @@ static void __init pv_init(void) virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc); =20 /* make sure bounce buffers are shared */ - swiotlb_init(true, SWIOTLB_FORCE | SWIOTLB_VERBOSE); + swiotlb_init(true, SWIOTLB_VERBOSE); swiotlb_update_mem_attributes(); } =20 diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index 6267363e0189..75cf8f6ae8cd 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -59,10 +59,8 @@ static void __init pci_swiotlb_detect(void) * bounce buffers as the hypervisor can't access arbitrary VM memory * that is not explicitly shared with it. */ - if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) x86_swiotlb_enable =3D true; - x86_swiotlb_flags |=3D SWIOTLB_FORCE; - } } #else static inline void __init pci_swiotlb_detect(void) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 2413235c50f0..fed901c0224e 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -14,6 +14,8 @@ #include #include #include +#include + #include "direct.h" =20 /* @@ -626,37 +628,41 @@ dma_addr_t dma_direct_map_phys(struct device *dev, ph= ys_addr_t phys, { dma_addr_t dma_addr; =20 + if (attrs & DMA_ATTR_MMIO) { + /* + * For host memory encryption treat MMIO memory as shared + */ + if (cc_platform_has(CC_ATTR_HOST_MEM_ENCRYPT)) + attrs |=3D DMA_ATTR_CC_SHARED; + } + if (is_swiotlb_force_bounce(dev)) { - if (!(attrs & DMA_ATTR_CC_SHARED)) { - if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) - return DMA_MAPPING_ERROR; + if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) + return DMA_MAPPING_ERROR; =20 - return swiotlb_map(dev, phys, size, dir, attrs); - } - } else if (attrs & DMA_ATTR_CC_SHARED) { - return DMA_MAPPING_ERROR; + return swiotlb_map(dev, phys, size, dir, attrs); } =20 - if (attrs & DMA_ATTR_MMIO) { - dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) - goto err_overflow; - } else if (attrs & DMA_ATTR_CC_SHARED) { + if (attrs & DMA_ATTR_CC_SHARED) dma_addr =3D phys_to_dma_unencrypted(dev, phys); + else + dma_addr =3D phys_to_dma_encrypted(dev, phys); + + if (attrs & DMA_ATTR_MMIO) { if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; - } else { - dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || - dma_kmalloc_needs_bounce(dev, size, dir)) { - if (is_swiotlb_active(dev) && - !(attrs & DMA_ATTR_REQUIRE_COHERENT)) - return swiotlb_map(dev, phys, size, dir, attrs); + goto dma_mapped; + } =20 - goto err_overflow; - } + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { + if (is_swiotlb_active(dev) && + !(attrs & DMA_ATTR_REQUIRE_COHERENT)) + return swiotlb_map(dev, phys, size, dir, attrs); + goto err_overflow; } =20 +dma_mapped: if (!dev_is_dma_coherent(dev) && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); @@ -748,8 +754,10 @@ size_t dma_direct_max_mapping_size(struct device *dev) { /* If SWIOTLB is active, use its maximum mapping size */ if (is_swiotlb_active(dev) && - (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev))) + (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev) || + force_dma_unencrypted(dev))) return swiotlb_max_mapping_size(dev); + return SIZE_MAX; } =20 --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BBAE378D76; Fri, 17 Jul 2026 18:08:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311712; cv=none; b=LiOegMGxPH8VnmAoeWDjF+JObdVnrGBF4hTyLK+u3FjgqQtOnhvrWOIMYAFoC3u3AYHapOi+CCwTW8lDkOvvg2u+w0sz2+n4VLrAsf2ts2U7LG8cLykuMTbmL8PKBYrhT/Zc/ip4850SKumcvDUafhIa4cOP3FWFD09x6FeaLiI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311712; c=relaxed/simple; bh=kG2SziB+23H+voX0slrPvRYv1LbMNlaZ2eAnEj/gHr4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d/djY4Bz9T/JTypasc3ysEiTi2dAOHHxFLDKB39nglL+2RoAHwYp2mie25c4MTnddPohuCh9gfmOjp5gHHvboPX84XCPF9vg7WTQOITV1lddflFyXy3efBrp8oKVa/40v/yQxG6YwsGsqJ70Lb6fX6rpdNYRkBsPClCRfxRXthA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Q33giytp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Q33giytp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 099D01F000E9; Fri, 17 Jul 2026 18:08:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311711; bh=gZrGl+jtrdO3STdBAPtI9YcrqokEMnK381jHPPxX4QM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Q33giytp3Pgs15qPJ1uZX7Rj8NcL8zDMsdhdsUMfr0vCKfMSuT9Tet9T2sH4Fvtdx RVYNPIJsmev0Bviu6bDXydJJ6XRIwsRJ6+4NoLaVoFGNN8V1B9G2C6xvTEPbUYmcEx 3qjf2tvr5H6hjTN2aJ4+BAycTPse4r+/+bfvf9r3rV8LUYuvqv6lrjdSRm26w5lUsB eT/9KpcWPKmdW2ypdaJm1YEauyXbohdQePWmzwPFF+tEr+EMyeekRReFWj54UC7V1J zCQfT0waCLsxriu+oXFree30DZaF1SLCRyMJTrZUZ212KRje67qrdih8qmWBNGP5ON IaaSbzUWq9fMA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 18/23] dma-direct: set decrypted flag for remapped DMA allocations Date: Fri, 17 Jul 2026 23:34:36 +0530 Message-ID: <20260717180442.110954-19-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Devices that are DMA non-coherent and require a remap were skipping dma_set_decrypted(), leaving DMA buffers encrypted even when the device requires unencrypted access. Move the call after the if (remap) branch so that both the direct and remapped allocation paths correctly mark the allocation as decrypted (or fail cleanly) before use. Fix dma_direct_alloc() and dma_direct_free() to apply set_memory_*() to the linear-map alias of the backing pages instead of the remapped CPU address. Also disallow highmem pages for __DMA_ATTR_ALLOC_CC_SHARED, because highmem buffers do not provide a usable linear-map address. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 56 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 44 insertions(+), 12 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index fed901c0224e..f7f064323bd9 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -198,14 +198,23 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, { bool remap =3D false, set_uncached =3D false; bool mark_mem_decrypt =3D false; + bool allow_highmem =3D true; struct page *page; void *ret; =20 if (force_dma_unencrypted(dev)) attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; =20 - if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) + if (attrs & __DMA_ATTR_ALLOC_CC_SHARED) { + /* + * Unencrypted/shared DMA requires a linear-mapped buffer + * address to look up the PFN and set architecture-required PFN + * attributes. This is not possible with HighMem. Avoid HighMem + * allocation. + */ + allow_highmem =3D false; mark_mem_decrypt =3D true; + } =20 size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) @@ -270,7 +279,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, } =20 /* we always manually zero the memory once we are done */ - page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); + page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, allow_hig= hmem); if (!page) return NULL; =20 @@ -285,6 +294,14 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached =3D false; } =20 + if (mark_mem_decrypt) { + void *lm_addr; + + lm_addr =3D page_address(page); + if (set_memory_decrypted((unsigned long)lm_addr, PFN_UP(size))) + goto out_leak_pages; + } + if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 @@ -295,29 +312,36 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, ret =3D dma_common_contiguous_remap(page, size, prot, __builtin_return_address(0)); if (!ret) - goto out_free_pages; + goto out_encrypt_pages; } else { ret =3D page_address(page); - if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) - goto out_leak_pages; } =20 memset(ret, 0, size); =20 if (set_uncached) { + void *uncached_cpu_addr; + arch_dma_prep_coherent(page, size); - ret =3D arch_dma_set_uncached(ret, size); - if (IS_ERR(ret)) - goto out_encrypt_pages; + uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + if (IS_ERR(uncached_cpu_addr)) + goto out_free_remap_pages; + ret =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return ret; =20 + +out_free_remap_pages: + if (remap) + dma_common_free_remap(ret, size); + out_encrypt_pages: - if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) - return NULL; -out_free_pages: + if (mark_mem_decrypt && + dma_set_encrypted(dev, page_address(page), size)) + goto out_leak_pages; + if (!swiotlb_free(dev, page, size)) dma_free_contiguous(dev, page, size); return NULL; @@ -380,8 +404,16 @@ void dma_direct_free(struct device *dev, size_t size, } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) + } + + if (mark_mem_encrypted) { + void *lm_addr; + + lm_addr =3D phys_to_virt(phys); + if (set_memory_encrypted((unsigned long)lm_addr, PFN_UP(size))) { + pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); return; + } } =20 if (swiotlb_pool) --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93142376A10; Fri, 17 Jul 2026 18:08:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311724; cv=none; b=Y/VyTe3lkJgSNzeNr9GhvrzESvlbdt0LXz9dTHfddUAeg725plH6chA/sgR0mitjkm11h6jcYTubPgW3OGHrj5fEgYNhAWv/50eJ0CbH0cCVPx7EvYAB7G0rJFC9OtgysV7wNiMLyfOOuElMZqKf86vv8EocO17Y6HMtSdS12HY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311724; c=relaxed/simple; bh=IWt4DT8zbxEVNr7Hsx2WH1ZhX/xIxKzGuVhjRtq3j+8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TYttpz5q4EPZ8t2q8SN3Y4pF+7zeolDrquS2902IXRUi5TdWIyFdVlgreZu6ALdRevfuqS+s1wvh6B3ChBIH5jad22dFWY9vyxlfRB+l1chm/enMMJyyf4nc4NKDQileFSdls6ciX5meaNhQ7CV5R4TrE/1lnyX22fLqRBENgNw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=m8fJtpWw; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="m8fJtpWw" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DC221F00A3A; Fri, 17 Jul 2026 18:08:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311723; bh=5bS+J6Rwxh/WXwnquOaDBNcfrARTnQLlbkZ3NkMBgPI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=m8fJtpWwBWHB9Ahirb07ZBZx03wpyr0HvEFiaOWqchFPaIHlOfOF8nIxp255RlCqc SA5lvDtSPSORHGBbXXe4ESMuP1HJ9TN2IktHo7lqFU4dEvpGz6SdLKKWBU3Jqy3dX8 dgDnw9Z4VAp5LeDo7M07SWRQ0ucTOCo4445a2InOiCz+dHLvzFSjW+KIWHYr6MTTid HCl/6afcZD5n5aVyJfVbbvZmRaMv9fI/ZA6mzhnlGUuldBmAZMvl8HImPa6Kb2BaTj aXPintlcklxZ6GlqObb+V/NV2vZBAQxpaENAkHBkzy/kvt74XPCY9FobR3QT/5MGXi u4DoaX4q93Pqg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Jiri Pirko , Michael Kelley Subject: [PATCH v8 19/23] dma-direct: select DMA address encoding from __DMA_ATTR_ALLOC_CC_SHARED Date: Fri, 17 Jul 2026 23:34:37 +0530 Message-ID: <20260717180442.110954-20-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make the dma-direct helpers derive the DMA address encoding from __DMA_ATTR_ALLOC_CC_SHARED instead of implicitly relying on force_dma_unencrypted() inside phys_to_dma_direct() Pass an explicit unencrypted/decrypted state into phys_to_dma_direct(), make the alloc paths return DMA addresses that match the requested buffer encryption state. Also only call dma_set_decrypted() when __DMA_ATTR_ALLOC_CC_SHARED is actually set. Reviewed-by: Jason Gunthorpe Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f7f064323bd9..eb0ce0787885 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -26,11 +26,11 @@ u64 zone_dma_limit __ro_after_init =3D DMA_BIT_MASK(24); =20 static inline dma_addr_t phys_to_dma_direct(struct device *dev, - phys_addr_t phys) + phys_addr_t phys, bool unencrypted) { - if (force_dma_unencrypted(dev)) + if (unencrypted) return phys_to_dma_unencrypted(dev, phys); - return phys_to_dma(dev, phys); + return phys_to_dma_encrypted(dev, phys); } =20 static inline struct page *dma_direct_to_page(struct device *dev, @@ -41,8 +41,9 @@ static inline struct page *dma_direct_to_page(struct devi= ce *dev, =20 u64 dma_direct_get_required_mask(struct device *dev) { + bool require_decrypted =3D force_dma_unencrypted(dev); phys_addr_t phys =3D ((phys_addr_t)max_pfn << PAGE_SHIFT) - 1; - u64 max_dma =3D phys_to_dma_direct(dev, phys); + u64 max_dma =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 return (1ULL << (fls64(max_dma) - 1)) * 2 - 1; } @@ -71,7 +72,8 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *d= ev, u64 *phys_limit) =20 bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) { - dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys); + bool require_decrypted =3D force_dma_unencrypted(dev); + dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 if (dma_addr =3D=3D DMA_MAPPING_ERROR) return false; @@ -81,17 +83,18 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t ph= ys, size_t size) =20 static int dma_set_decrypted(struct device *dev, void *vaddr, size_t size) { - if (!force_dma_unencrypted(dev)) - return 0; - return set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + int ret; + + ret =3D set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + if (ret) + pr_warn_ratelimited("leaking DMA memory that can't be decrypted\n"); + return ret; } =20 static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) { int ret; =20 - if (!force_dma_unencrypted(dev)) - return 0; ret =3D set_memory_encrypted((unsigned long)vaddr, PFN_UP(size)); if (ret) pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); @@ -171,7 +174,8 @@ static struct page *dma_direct_alloc_from_pool(struct d= evice *dev, size_t size, dma_coherent_ok); if (!page) return NULL; - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + attrs & __DMA_ATTR_ALLOC_CC_SHARED); return page; } =20 @@ -187,9 +191,11 @@ static void *dma_direct_alloc_no_mapping(struct device= *dev, size_t size, /* remove any dirty cache lines on the kernel alias */ if (!PageHighMem(page)) arch_dma_prep_coherent(page, size); - - /* return the page pointer as the opaque cookie */ - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + /* + * return the page pointer as the opaque cookie. + * Never used for unencrypted allocation + */ + *dma_handle =3D phys_to_dma_encrypted(dev, page_to_phys(page)); return page; } =20 @@ -329,7 +335,8 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =3D uncached_cpu_addr; } =20 - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + attrs & __DMA_ATTR_ALLOC_CC_SHARED); return ret; =20 =20 @@ -450,11 +457,13 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, return NULL; =20 ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && + dma_set_decrypted(dev, ret, size)) goto out_leak_pages; setup_page: memset(ret, 0, size); - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + attrs & __DMA_ATTR_ALLOC_CC_SHARED); return page; out_leak_pages: return NULL; --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C28C35C1A9; Fri, 17 Jul 2026 18:08:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311736; cv=none; b=i4CnOIAggpZOfguskWg0ZU2OqlPTBSPukKXE0AHbYshxO+SwuFqgv7NdEYnf5b6HiI2ROhGf5ZGNPpg11eE2beZfB98FXuKuJcPMsn1Vxto0lbtDGS8VauuXLFPWyBVFxACVfuPbGPTXCBzU6ogCvqga1MYEU1eiWyNvTWNcrrE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311736; c=relaxed/simple; bh=2kRYTQsjVRDuCAhUYtz4n8Rcw0radgP2kMH6vESjQ50=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sjuAM1oK9+mMaiwOSQA0aqynaPqmOy/5MJReIgXk2QWvnid8waFVtT6jBMqmkJPWCqXP5y0OSJvx8E5TGFWJXdEFqz+Mf8FPzWw84h2o7F6Gef5oQyWx0YXVlJZgxjcHybknM2nZeoDCb0NYWiK6pQYQJVHd9cofivLzgDIo+pg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=P+so4Cn9; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="P+so4Cn9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB10D1F000E9; Fri, 17 Jul 2026 18:08:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311734; bh=EwNHrZwSusGthQDiVN8gv3gZ/NqcRGqEVZjtKRv5t3M=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=P+so4Cn9Bp/lktmtgwcfowHs6E4slI6aZAzD2tpNiBuY1Zm9/zccXXr2TLAjVxfUD q143dSuzlsOH/opFm3EBoMxg8xypyOsYyPtjXLRif5H2C70sQyZ9xZFA02T2W/K162 bIcOqD+oqZbRbtNXiQsShnjDhr7+GY9m2Sxw1FgcymZInV0hfCkqzznGb5TpFsdd2u 555J/pjq6j7dtU3oHQjF5mnuRzCwg/T1Lo1bl9egA+1w3ZP+C6Izeu9onXFXcfpMbR xxjyhcVwpdILCOnQqzb0d4clRjUm5i0m3TJr3PDmjB3mpGwYCZxyk3hmScCXTY1vyj vM8LlSDWutWsA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 20/23] dma-direct: rename ret to cpu_addr in alloc helpers Date: Fri, 17 Jul 2026 23:34:38 +0530 Message-ID: <20260717180442.110954-21-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ret in dma_direct_alloc() and dma_direct_alloc_pages() holds the returned CPU mapping, not a generic return value. Rename it to cpu_addr and update the remaining uses to match. This makes the allocation paths easier to follow and keeps the local naming consistent with what the variable actually represents. Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index eb0ce0787885..50a8335f3830 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -206,7 +206,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, bool mark_mem_decrypt =3D false; bool allow_highmem =3D true; struct page *page; - void *ret; + void *cpu_addr; =20 if (force_dma_unencrypted(dev)) attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; @@ -266,9 +266,10 @@ void *dma_direct_alloc(struct device *dev, size_t size, */ if ((remap || (attrs & __DMA_ATTR_ALLOC_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) { - page =3D dma_direct_alloc_from_pool(dev, size, dma_handle, - &ret, gfp, attrs); - return page ? ret : NULL; + page =3D dma_direct_alloc_from_pool(dev, size, + dma_handle, &cpu_addr, + gfp, attrs); + return page ? cpu_addr : NULL; } =20 if (is_swiotlb_for_alloc(dev)) { @@ -315,34 +316,33 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, arch_dma_prep_coherent(page, size); =20 /* create a coherent mapping */ - ret =3D dma_common_contiguous_remap(page, size, prot, - __builtin_return_address(0)); - if (!ret) + cpu_addr =3D dma_common_contiguous_remap(page, size, prot, + __builtin_return_address(0)); + if (!cpu_addr) goto out_encrypt_pages; } else { - ret =3D page_address(page); + cpu_addr =3D page_address(page); } =20 - memset(ret, 0, size); + memset(cpu_addr, 0, size); =20 if (set_uncached) { void *uncached_cpu_addr; =20 arch_dma_prep_coherent(page, size); - uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + uncached_cpu_addr =3D arch_dma_set_uncached(cpu_addr, size); if (IS_ERR(uncached_cpu_addr)) goto out_free_remap_pages; - ret =3D uncached_cpu_addr; + cpu_addr =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), attrs & __DMA_ATTR_ALLOC_CC_SHARED); - return ret; - + return cpu_addr; =20 out_free_remap_pages: if (remap) - dma_common_free_remap(ret, size); + dma_common_free_remap(cpu_addr, size); =20 out_encrypt_pages: if (mark_mem_decrypt && @@ -434,21 +434,21 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, { unsigned long attrs =3D 0; struct page *page; - void *ret; + void *cpu_addr; =20 if (force_dma_unencrypted(dev)) attrs |=3D __DMA_ATTR_ALLOC_CC_SHARED; =20 if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, - &ret, gfp, attrs); + &cpu_addr, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (!page) return NULL; =20 - ret =3D page_address(page); + cpu_addr =3D page_address(page); goto setup_page; } =20 @@ -456,12 +456,12 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, if (!page) return NULL; =20 - ret =3D page_address(page); + cpu_addr =3D page_address(page); if ((attrs & __DMA_ATTR_ALLOC_CC_SHARED) && - dma_set_decrypted(dev, ret, size)) + dma_set_decrypted(dev, cpu_addr, size)) goto out_leak_pages; setup_page: - memset(ret, 0, size); + memset(cpu_addr, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), attrs & __DMA_ATTR_ALLOC_CC_SHARED); return page; --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8225E37CD59; Fri, 17 Jul 2026 18:09:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311747; cv=none; b=VALvrDSpKuVjM8mPRAmt9JXMqzmwxL+cIwPxLT3lXXwI4Q1e01S6IZrtE7DasPoeYqn2VMIPtqsVtq2DUElyZwPJMRJLBPd1a3ysWMQInKCpN2p6RoqT4HqTUDOS5W++hvWJcJyXhL3DIoYT1N3MLpHHVPBXjuZtsLI9zplGGoc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311747; c=relaxed/simple; bh=dhLi/o3VkwA5aaQ5w6A5H2fjW31AoKvHQoSVrBdLIpQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mbeddha4lqb92t4KqmT4nRqCJAwKDqAWP4rREED+ArzFpCCdKuYW2AVvoRjFKWJKRdiNlvCs7fusOj1qehkpVrgssf0ulCHp5WrWF0/71C3AIz6vSp6LpNOOUQmW+tFnD5LoP/M4xXCo+9i/m4Tv+edNZ9Fyc6MYVugIsUWqxHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=erlKBMkz; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="erlKBMkz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8BF531F00A3A; Fri, 17 Jul 2026 18:08:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311746; bh=+N7VVODChZxFmZKynUomGOW/L3aQ1CGwQqBrr7RD6DA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=erlKBMkz1EMPtuxKULszUZBtrczla6utr3w+JR9UPS0JZYc/631OIU6rHE1bQNnR9 rDzLAFikuLpQuBzp/UpQ1q7+zlF8qpwDmjX2BlpI2ppP1N8cQ8dpFkwA93/JNDC/k8 rOkds6i8kkbUOxBrU4x57wByfum6nlvm663KOcoEOgeuiN0St0dom+CyeDWeQ8qk6e Cmjd5c5OYS3BeZLvM1imffUCpXm4jDWcVG8ygGuHUvj2Lav3rKMtVl1/g15nqNxanJ l7nA7NmfQYeSGexEg7xnquhCz6keUfNbyRuqqCktI+CsomvKTAhw9AQbjn9WNCzNI8 pAB0Sesg7BhGQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v8 21/23] dma: swiotlb: free dynamic pools from process context Date: Fri, 17 Jul 2026 23:34:39 +0530 Message-ID: <20260717180442.110954-22-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swiotlb_dyn_free() is used after removing a dynamic swiotlb pool from RCU-protected lists. It can call swiotlb_free_tlb(), which may need to restore the encryption state of an unencrypted pool with set_memory_encrypted() before freeing the pages. RCU callbacks run in atomic context, but set_memory_encrypted() is not guaranteed to be atomic-safe on all architectures. For example, page attribute updates may allocate page tables or take sleeping locks. Use queue_rcu_work() for dynamic pool freeing instead. This keeps the RCU grace period before freeing a published pool, while running the actual pool teardown from workqueue context. Use the same helper for the transient-pool error path, since that path may also be reached from atomic DMA mapping context. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 4 ++-- kernel/dma/swiotlb.c | 19 +++++++++++-------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index ee42f7588847..c3bf7ed6f7a6 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -64,7 +64,7 @@ extern void __init swiotlb_update_mem_attributes(void); * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. * @node: Member of the IO TLB memory pool list. - * @rcu: RCU head for swiotlb_dyn_free(). + * @dyn_free: RCU work item used to free the pool from process context. * @transient: %true if transient memory pool. * @cc_shared: %true if the pool memory is shared for confidential computi= ng. */ @@ -80,7 +80,7 @@ struct io_tlb_pool { struct io_tlb_slot *slots; #ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node; - struct rcu_head rcu; + struct rcu_work dyn_free; bool transient; bool cc_shared; #endif diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index b5960c2e98d8..4d0f2c04d891 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -779,13 +779,10 @@ static void swiotlb_dyn_alloc(struct work_struct *wor= k) add_mem_pool(mem, pool); } =20 -/** - * swiotlb_dyn_free() - RCU callback to free a memory pool - * @rcu: RCU head in the corresponding struct io_tlb_pool. - */ -static void swiotlb_dyn_free(struct rcu_head *rcu) +static void swiotlb_dyn_free_work(struct work_struct *work) { - struct io_tlb_pool *pool =3D container_of(rcu, struct io_tlb_pool, rcu); + struct io_tlb_pool *pool =3D + container_of(to_rcu_work(work), struct io_tlb_pool, dyn_free); size_t slots_size =3D array_size(sizeof(*pool->slots), pool->nslabs); size_t tlb_size =3D pool->end - pool->start; =20 @@ -794,6 +791,12 @@ static void swiotlb_dyn_free(struct rcu_head *rcu) kfree(pool); } =20 +static void swiotlb_schedule_dyn_free(struct io_tlb_pool *pool) +{ + INIT_RCU_WORK(&pool->dyn_free, swiotlb_dyn_free_work); + queue_rcu_work(system_wq, &pool->dyn_free); +} + /** * __swiotlb_find_pool() - find the IO TLB pool for a physical address * @dev: Device which has mapped the DMA buffer. @@ -840,7 +843,7 @@ static void swiotlb_del_pool(struct device *dev, struct= io_tlb_pool *pool) list_del_rcu(&pool->node); spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags); =20 - call_rcu(&pool->rcu, swiotlb_dyn_free); + swiotlb_schedule_dyn_free(pool); } =20 #endif /* CONFIG_SWIOTLB_DYNAMIC */ @@ -1281,7 +1284,7 @@ static int swiotlb_find_slots(struct device *dev, phy= s_addr_t orig_addr, index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, tbl_dma_addr, alloc_size, alloc_align_mask); if (index < 0) { - swiotlb_dyn_free(&pool->rcu); + swiotlb_schedule_dyn_free(pool); return -1; } =20 --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D859837CD59; Fri, 17 Jul 2026 18:09:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311759; cv=none; b=iVzpCpfv2VIqmrCrDRkM/FMYCfSNJby8eHxKyOE6WcxaOndzxu9TB6REgjZjZ1ueSGqslw34fX/vfFzRMuw2VR8OKeBleFf/FVir0xcXidmVHTeG/wRywlEQcvt8PsiXp25Vb25aJdlcrkXK79hOqnotiUNHWugVLQskwRU0b+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311759; c=relaxed/simple; bh=mbzUwBV/dDnmXR/IMr7kPaFlWRolI/J6cxOiVlMIEUk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FwFRKrnmkkgsBSBOL8Lrn4DKR8uw50LEUumpQ/afug1k/oQHOiTqxUg8Y4yllby4lEBCC7Jqky0YwdTC8sFqGlqCtdmLYe80ptAEDhLEpWssQ+6fEh9+Ol7PA4koW36ulAHPyEm2RHt6pVlmpfUyalLy1bdBF6DFtMYUglkIBzs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GXAYsOaH; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GXAYsOaH" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C834D1F000E9; Fri, 17 Jul 2026 18:09:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311757; bh=wekj/NDLmWCmZxzHsUwUUJzzNTnKgpVzqdkg0KE+9sA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=GXAYsOaHOvwy0ZJpRZO5dho+WvfBB9pgk11RR+R9CcBsTOa5l02MR708hIyxwV3Wa KL17plk3f2oFf6I7XXRAYeLfbuk0WW0XF4KJ+SdRhqp0bQecjj2Uq5qkPyyiDP5QyN nLw0k2c/iBxOQByS8wqfcaMKe4gKNPlmKEgFl+pl/DrXRZ/usfEMrTXvp9Jbo3RZPI SzkblS2p02DDN+XmOjvOBWEaLmm7ZFx4KSl4f20ZL0vNVgrINurAAMJo5m9+P92PuJ iqrot+YS6pcDf5mAApMj4m/PGzRry1hrvyKjdo9kXjQPr8FcQ1ue12sspd3O0hnAme SaVV9tqbcS7Tg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe , Michael Kelley Subject: [PATCH v8 22/23] dma: swiotlb: handle set_memory_decrypted() failures Date: Fri, 17 Jul 2026 23:34:40 +0530 Message-ID: <20260717180442.110954-23-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Check the return value when converting swiotlb pools between encrypted and decrypted mappings. If the default pool cannot be decrypted after early initialization, mark the pool fully used so it cannot satisfy future bounce allocations. For late initialization, return the `set_memory_decrypted()` failure. For restricted DMA pools, fail device initialization if the reserved pool cannot be decrypted. This prevents swiotlb from using pools whose encryption attributes do not match their metadata, and avoids returning pages with uncertain encryption state back to the allocator. Reviewed-by: Jason Gunthorpe Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Petr Tesarik Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/swiotlb.c | 80 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 15 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 4d0f2c04d891..8b7e47504304 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -248,6 +248,23 @@ static inline unsigned long nr_slots(u64 val) return DIV_ROUND_UP(val, IO_TLB_SIZE); } =20 +static void swiotlb_mark_pool_used(struct io_tlb_pool *pool) +{ + unsigned long i; + + for (i =3D 0; i < pool->nareas; i++) { + pool->areas[i].index =3D 0; + pool->areas[i].used =3D pool->area_nslabs; + } + + for (i =3D 0; i < pool->nslabs; i++) { + pool->slots[i].list =3D 0; + pool->slots[i].orig_addr =3D INVALID_PHYS_ADDR; + pool->slots[i].alloc_size =3D 0; + pool->slots[i].pad_slots =3D 0; + } +} + /* * Early SWIOTLB allocation may be too early to allow an architecture to * perform the desired operations. This function allows the architecture = to @@ -272,8 +289,16 @@ void __init swiotlb_update_mem_attributes(void) return; bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); =20 - if (io_tlb_default_mem.cc_shared) - set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); + if (io_tlb_default_mem.cc_shared) { + int ret; + + ret =3D set_memory_decrypted((unsigned long)mem->vaddr, + bytes >> PAGE_SHIFT); + if (ret) { + pr_warn("Failed to decrypt default memory pool, disabling it\n"); + swiotlb_mark_pool_used(mem); + } + } } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, @@ -442,9 +467,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, { struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; unsigned long nslabs =3D ALIGN(size >> IO_TLB_SHIFT, IO_TLB_SEGSIZE); + unsigned int order, area_order, slot_order; + bool leak_pages =3D false; unsigned int nareas; unsigned char *vstart =3D NULL; - unsigned int order, area_order; bool retried =3D false; int rc =3D 0; =20 @@ -504,6 +530,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, (PAGE_SIZE << order) >> 20); } =20 + rc =3D -ENOMEM; nareas =3D limit_nareas(default_nareas, nslabs); area_order =3D get_order(array_size(sizeof(*mem->areas), nareas)); mem->areas =3D (struct io_tlb_area *) @@ -511,14 +538,20 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, if (!mem->areas) goto error_area; =20 + slot_order =3D get_order(array_size(sizeof(*mem->slots), nslabs)); mem->slots =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, - get_order(array_size(sizeof(*mem->slots), nslabs))); + slot_order); if (!mem->slots) goto error_slots; =20 - if (io_tlb_default_mem.cc_shared) - set_memory_decrypted((unsigned long)vstart, - (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (io_tlb_default_mem.cc_shared) { + rc =3D set_memory_decrypted((unsigned long)vstart, + (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (rc) { + leak_pages =3D true; + goto error_decrypt; + } + } =20 swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true, nareas); @@ -527,16 +560,20 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, swiotlb_print_info(); return 0; =20 +error_decrypt: + free_pages((unsigned long)mem->slots, slot_order); error_slots: free_pages((unsigned long)mem->areas, area_order); error_area: - free_pages((unsigned long)vstart, order); - return -ENOMEM; + if (!leak_pages) + free_pages((unsigned long)vstart, order); + return rc; } =20 void __init swiotlb_exit(void) { struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; + bool leak_pages =3D false; unsigned long tbl_vaddr; size_t tbl_size, slots_size; unsigned int area_order; @@ -552,19 +589,23 @@ void __init swiotlb_exit(void) tbl_size =3D PAGE_ALIGN(mem->end - mem->start); slots_size =3D PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs)); =20 - if (io_tlb_default_mem.cc_shared) - set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (io_tlb_default_mem.cc_shared) { + if (set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT)) + leak_pages =3D true; + } =20 if (mem->late_alloc) { area_order =3D get_order(array_size(sizeof(*mem->areas), mem->nareas)); free_pages((unsigned long)mem->areas, area_order); - free_pages(tbl_vaddr, get_order(tbl_size)); + if (!leak_pages) + free_pages(tbl_vaddr, get_order(tbl_size)); free_pages((unsigned long)mem->slots, get_order(slots_size)); } else { memblock_free(mem->areas, array_size(sizeof(*mem->areas), mem->nareas)); - memblock_phys_free(mem->start, tbl_size); + if (!leak_pages) + memblock_phys_free(mem->start, tbl_size); memblock_free(mem->slots, slots_size); } =20 @@ -1957,9 +1998,18 @@ static int rmem_swiotlb_device_init(struct reserved_= mem *rmem, * restricted mem pool is shared by default */ if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + int ret; + mem->cc_shared =3D true; - set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), - rmem->size >> PAGE_SHIFT); + ret =3D set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), + rmem->size >> PAGE_SHIFT); + if (ret) { + dev_err(dev, "Failed to decrypt restricted DMA pool\n"); + kfree(pool->areas); + kfree(pool->slots); + kfree(mem); + return ret; + } } else { mem->cc_shared =3D false; } --=20 2.43.0 From nobody Sat Jul 25 04:53:34 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D64D737D126; Fri, 17 Jul 2026 18:09:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311770; cv=none; b=UKCLuTWZjeL16bzjsMjf8wWaguvrDRzR92Yz4QSIAXPPt/beIp5ZJ/wELe962EeohWA5dyImyGz1qQKqs+HAVx+4cZRhJv4aWjuw3+arVi9ZqbLlgEBx7XqcwFywheR31gWPDsOr716/E4vr22ZUJOCFInIjvac/QFocsYkYD2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784311770; c=relaxed/simple; bh=pGo1yPgsER01aQ8A4oxLe7N9Xn4S7xozVAJYzXGd0dQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZDKdxwts/GWJixJfOteVWo2FLbbw7WqfCXPoEggxeaNUXJuTfoYZ7/YV/E+KtLDjQg/wCkOuqRD2IvcRygzTdOEmcuqKeeo558NZodjuOClgbGR/+CoRxouiTNmHnji9kk4r3OWL7+on4M6yAs/V5sAAuwNZdxNwzacRMeLnejI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZKEaTtkj; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZKEaTtkj" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63CFE1F00A3A; Fri, 17 Jul 2026 18:09:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1784311768; bh=tkXR0ipZ7ma+5T3tic3bVYqHfLRJ5/MS0548pdfa6Q4=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ZKEaTtkjlsYTecPUwUZYY5boRz8Hl5p9D2dk5Y5OHE5V+KrdaCWiMHICIVvyeZo8l bfDENN6CkL/X2Nhd1Dhikwp79Ii7RZhp6+TdbxOWmncy5evwDeUC0PrecgKYvOfJ2i z5t5X7pIAq2lJxusYJVvtvqityS5Yt1NuT1GCEYBv/ZX+VxYwD85Oz7Abz4+1TQJdX wcai+O3Dp9MiHid7sAM+sdbkz+TE4Ihrj2BOXgkzKovblm2cKclw8oi9o7WFLwiD3J OUQsZBxyG2lJSEe2eHv3+KJZZXFXBOwVqSpI2S4/WrMB2IqHdqF6IqWNhGaswGM9hg XgoRYSziMoMAQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jason Gunthorpe Subject: [PATCH v8 23/23] swiotlb: remove unused SWIOTLB_FORCE flag Date: Fri, 17 Jul 2026 23:34:41 +0530 Message-ID: <20260717180442.110954-24-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260717180442.110954-1-aneesh.kumar@kernel.org> References: <20260717180442.110954-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SWIOTLB_FORCE has no remaining in-tree users. Forced bouncing is now controlled through the swiotlb=3Dforce command line option via swiotlb_force_bounce. Remove the unused flag and simplify the force_bounce initialization. Reviewed-by: Jason Gunthorpe Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 3 +-- kernel/dma/swiotlb.c | 3 +-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index c3bf7ed6f7a6..9caca923c380 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -15,8 +15,7 @@ struct page; struct scatterlist; =20 #define SWIOTLB_VERBOSE (1 << 0) /* verbose initialization */ -#define SWIOTLB_FORCE (1 << 1) /* force bounce buffering */ -#define SWIOTLB_ANY (1 << 2) /* allow any memory for the buffer */ +#define SWIOTLB_ANY (1 << 1) /* allow any memory for the buffer */ =20 /* * Maximum allowable number of contiguous slabs to map, diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 8b7e47504304..897aba538c5b 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -400,8 +400,7 @@ void __init swiotlb_init_remap(bool addressing_limit, u= nsigned int flags, if (swiotlb_force_disable) return; =20 - io_tlb_default_mem.force_bounce =3D - swiotlb_force_bounce || (flags & SWIOTLB_FORCE); + io_tlb_default_mem.force_bounce =3D swiotlb_force_bounce; =20 #ifdef CONFIG_SWIOTLB_DYNAMIC if (!remap) --=20 2.43.0