From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 350813E5A15; Thu, 4 Jun 2026 08:40:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562429; cv=none; b=IeQuYRsrmIcE74uzwsp6MD7kTPRtZJSQsmLmpl1lBI3XTyFYfkVBVd/f7RJSHEwyNMjU8rSXLbwb6xWOT/TQPxbpdIA5TH9jJVJnnG3cClqJkUEJrO7rZdL9BTz5cLPQf0dCq9DoNZt4W2wPDsCvfhCUmpqX/HC4WXP8tcUFsNU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562429; c=relaxed/simple; bh=iylfyXBNO6S14YbSeAPeNtbQXHgcnAOZOy6z6gxK0Ig=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=stLKpNxYue4lxPbhI8fZjvofA5BbSRvKzYGyyqIUb0UMO5lQo2J/5APx064Kb1cjdxQUGgJWxR7oSwMWLQ8ig2Qhe1qrVCM1XrT2ZFsCCGYxJCwGQKZrzRh40xbabs4yvyjSUlJPkkVsAzwUh+u+wZ5oozteCHMCV/dDP/iLdZ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZSAcfFlh; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZSAcfFlh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC06B1F00898; Thu, 4 Jun 2026 08:40:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562422; bh=4aF9+d8jwgDAX56PY9Oecf4wOZ9YOQdZwhZO6X/dmDA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=ZSAcfFlhwVoXj+bvE9YfbowEqQBV1/lgoSIeAFRJcpkKHwSs/j6JzCILJWkoZrYhQ kFn6+KM7tJBsYQQBgnfTV2opqNsAG0x3D+sFC7a+6IWdM24Wvu579IiQJr5KPHuhpk +Q9cG9POofUpPzLkKXLr2AZMA1JBJtglWRqpYpO92nmXndSknhtqkGaYmPxAYJnxBB L1c6KC8V5RRssQOFW3NZC1aoelPWsOtMhR8NSBtEEqbQMOE0yOwQoHtOgjbBM/1abX 9MjBsnZeCbQt1ioCBWhXC62eBjVL0V2x4ezbsIA4JafSB7JeyX+7WS922MqP2zcwpf bk1DLeIniUmLw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Halil Pasic , Matthew Rosato , Jaehoon Kim Subject: [PATCH v6 01/20] s390: Expose protected virtualization through cc_platform_has() Date: Thu, 4 Jun 2026 14:09:40 +0530 Message-ID: <20260604083959.1265923-2-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Protected virtualization guests use memory encryption, so advertise that to the rest of the kernel through cc_platform_has(CC_ATTR_MEM_ENCRYPT). s390 already forces DMA mappings to be unencrypted for protected virtualization guests through force_dma_unencrypted(). Add ARCH_HAS_CC_PLATFORM and provide the matching cc_platform_has() implementation Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jaehoon Kim --- Cc: Halil Pasic Cc: Matthew Rosato Cc: Jaehoon Kim --- arch/s390/Kconfig | 1 + arch/s390/mm/init.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+) diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ecbcbb781e40..9b5e6029e043 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -87,6 +87,7 @@ config S390 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CC_CAN_LINK + select ARCH_HAS_CC_PLATFORM select ARCH_HAS_CPU_FINALIZE_INIT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index 1f72efc2a579..ad3c6d92b801 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -50,6 +50,7 @@ #include #include #include +#include =20 pgd_t swapper_pg_dir[PTRS_PER_PGD] __section(".bss..swapper_pg_dir"); pgd_t invalid_pg_dir[PTRS_PER_PGD] __section(".bss..invalid_pg_dir"); @@ -140,6 +141,19 @@ bool force_dma_unencrypted(struct device *dev) return is_prot_virt_guest(); } =20 + +bool cc_platform_has(enum cc_attr attr) +{ + switch (attr) { + case CC_ATTR_MEM_ENCRYPT: + return is_prot_virt_guest(); + + default: + return false; + } +} +EXPORT_SYMBOL_GPL(cc_platform_has); + /* protected virtualization */ static void __init pv_init(void) { --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BFAF3EBF07; Thu, 4 Jun 2026 08:40:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562442; cv=none; b=JPP3VYHOUW6q39YGWLuOaMsoDs8znHNgU7QNBm6yStKxASaZzu15io4Yajro+kj3Q3+yRl4XOkzO0jR3CNjHSFiHfM0ritvxAKeUX5ujBKwS6jtDQzmkyINrp1vSbDtSwBPn8Qy8RKE288fU3FD8gNg+87gdfDLFQw6T0sEIh40= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562442; c=relaxed/simple; bh=hUzEjBt2CN4lvcmmnF6q4DdaDW2TUw243P2pjhqN5MI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nuJBQK79/nA/VJwvV4ECEtZJaJuOPv6gbgP9IHdt1Ao/PE2v0slsSMCZPa6VCaImGQvgFLjGtOGP8jQfTwwabuAHWiZFWL0eul+5XL4OISKCvYj9Mggaz0gh+eFjlIYvWHVjbd2Rvlekb1A8c4tvKcGZI4hTc8RWvdPCODsPCyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Rcu6QwQ9; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Rcu6QwQ9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4186B1F00893; Thu, 4 Jun 2026 08:40:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562431; bh=06/l/ie87JUAu214M+kEjiVedXrxQg5tsimMv1iyt+s=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Rcu6QwQ9robwXxBRcRoz7H6FcgSpQ+z2a9MiWQDQnkLRiP/1XvzL4YOAhZ2DCiYLy ZRX8sqVN6RaqxwZ9gJtRFVE7kOsFapaZmGluem+X+y9YxXgyItgZ0ByYJyOOj7yz9j h2wWznu29IfIuiumuAI7FOsblzm4Al8VE9NQ9R9jfoNsLvr7WOtMemAeavTYTr+Pwy BSMhuu0S0GQH9kE/I1/FXmWZz5M1lm/uwE1u+oyiuTHKo/nw28v5JkxQOYoiauKTHc M98C2fXvV7hcknQfWMHtIitHI+keJxH8mN4Pjgxx5r1GFPvik/MMwn5MCxowoNw9Zy 6+S6j7i4QbIdg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 02/20] dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages Date: Thu, 4 Jun 2026 14:09:41 +0530 Message-ID: <20260604083959.1265923-3-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in dma_direct_alloc() / dma_direct_alloc_pages(). This is needed for follow-up changes that simplify the handling of memory encryption/decryption based on the DMA attribute flags. swiotlb backing pages are already mapped decrypted by swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so dma-direct should not call dma_set_decrypted() on allocation nor dma_set_encrypted() on free for swiotlb-backed memory. Update alloc/free paths to detect swiotlb-backed pages and skip encrypt/decrypt transitions for those paths. Keep the existing highmem rejection in dma_direct_alloc_pages() for swiotlb allocations. Only for "restricted-dma-pool", we currently set `for_alloc =3D true`, while rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is typically used together with "shared-dma-pool", where the shared region is accessed after remap/ioremap and the returned address is suitable for decrypted memory access. So existing code paths remain valid. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 6 ++++ kernel/dma/direct.c | 71 ++++++++++++++++++++++++++++++----------- kernel/dma/swiotlb.c | 6 ++++ 3 files changed, 65 insertions(+), 18 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 3dae0f592063..133bb8ca9032 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -284,6 +284,8 @@ extern void swiotlb_print_info(void); #ifdef CONFIG_DMA_RESTRICTED_POOL struct page *swiotlb_alloc(struct device *dev, size_t size); bool swiotlb_free(struct device *dev, struct page *page, size_t size); +void swiotlb_free_from_pool(struct device *dev, phys_addr_t tlb_addr, + size_t size, struct io_tlb_pool *pool); =20 static inline bool is_swiotlb_for_alloc(struct device *dev) { @@ -299,6 +301,10 @@ static inline bool swiotlb_free(struct device *dev, st= ruct page *page, { return false; } +static inline void swiotlb_free_from_pool(struct device *dev, phys_addr_t = tlb_addr, + size_t size, struct io_tlb_pool *pool) +{ +} static inline bool is_swiotlb_for_alloc(struct device *dev) { return false; diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 583c5922bca2..a741c8a2ee66 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -96,14 +96,6 @@ static int dma_set_encrypted(struct device *dev, void *v= addr, size_t size) return ret; } =20 -static void __dma_direct_free_pages(struct device *dev, struct page *page, - size_t size) -{ - if (swiotlb_free(dev, page, size)) - return; - dma_free_contiguous(dev, page, size); -} - static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze) { struct page *page =3D swiotlb_alloc(dev, size); @@ -125,9 +117,6 @@ static struct page *__dma_direct_alloc_pages(struct dev= ice *dev, size_t size, =20 WARN_ON_ONCE(!PAGE_ALIGNED(size)); =20 - if (is_swiotlb_for_alloc(dev)) - return dma_direct_alloc_swiotlb(dev, size); - gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); page =3D dma_alloc_contiguous(dev, size, gfp); if (page) { @@ -204,6 +193,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; + bool mark_mem_decrypt =3D true; struct page *page; void *ret; =20 @@ -250,11 +240,21 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (page) { + mark_mem_decrypt =3D false; + goto setup_page; + } + return NULL; + } + /* we always manually zero the memory once we are done */ page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); if (!page) return NULL; =20 +setup_page: /* * dma_alloc_contiguous can return highmem pages depending on a * combination the cma=3D arguments and per-arch setup. These need to be @@ -281,7 +281,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, goto out_free_pages; } else { ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) goto out_leak_pages; } =20 @@ -298,10 +298,11 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, return ret; =20 out_encrypt_pages: - if (dma_set_encrypted(dev, page_address(page), size)) + if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) return NULL; out_free_pages: - __dma_direct_free_pages(dev, page, size); + if (!swiotlb_free(dev, page, size)) + dma_free_contiguous(dev, page, size); return NULL; out_leak_pages: return NULL; @@ -310,6 +311,9 @@ void *dma_direct_alloc(struct device *dev, size_t size, void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { + phys_addr_t phys; + bool mark_mem_encrypted =3D true; + struct io_tlb_pool *swiotlb_pool; unsigned int page_order =3D get_order(size); =20 if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && @@ -338,16 +342,25 @@ void dma_direct_free(struct device *dev, size_t size, dma_free_from_pool(dev, cpu_addr, PAGE_ALIGN(size))) return; =20 + phys =3D dma_to_phys(dev, dma_addr); + swiotlb_pool =3D swiotlb_find_pool(dev, phys); + if (swiotlb_pool) + /* Swiotlb doesn't need a page attribute update on free */ + mark_mem_encrypted =3D false; + if (is_vmalloc_addr(cpu_addr)) { vunmap(cpu_addr); } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (dma_set_encrypted(dev, cpu_addr, size)) + if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) return; } =20 - __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); + if (swiotlb_pool) + swiotlb_free_from_pool(dev, phys, size, swiotlb_pool); + else + dma_free_contiguous(dev, dma_direct_to_page(dev, dma_addr), size); } =20 struct page *dma_direct_alloc_pages(struct device *dev, size_t size, @@ -359,6 +372,15 @@ struct page *dma_direct_alloc_pages(struct device *dev= , size_t size, if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (!page) + return NULL; + + ret =3D page_address(page); + goto setup_page; + } + page =3D __dma_direct_alloc_pages(dev, size, gfp, false); if (!page) return NULL; @@ -366,6 +388,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, ret =3D page_address(page); if (dma_set_decrypted(dev, ret, size)) goto out_leak_pages; +setup_page: memset(ret, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return page; @@ -377,16 +400,28 @@ void dma_direct_free_pages(struct device *dev, size_t= size, struct page *page, dma_addr_t dma_addr, enum dma_data_direction dir) { + phys_addr_t phys; void *vaddr =3D page_address(page); + struct io_tlb_pool *swiotlb_pool; + bool mark_mem_encrypted =3D true; =20 /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool(dev, vaddr, size)) return; =20 - if (dma_set_encrypted(dev, vaddr, size)) + phys =3D page_to_phys(page); + swiotlb_pool =3D swiotlb_find_pool(dev, phys); + if (swiotlb_pool) + mark_mem_encrypted =3D false; + + if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size)) return; - __dma_direct_free_pages(dev, page, size); + + if (swiotlb_pool) + swiotlb_free_from_pool(dev, phys, size, swiotlb_pool); + else + dma_free_contiguous(dev, page, size); } =20 #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1abd3e6146f4..ac03a6856c2e 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1809,6 +1809,12 @@ bool swiotlb_free(struct device *dev, struct page *p= age, size_t size) return true; } =20 +void swiotlb_free_from_pool(struct device *dev, phys_addr_t tlb_addr, size= _t size, + struct io_tlb_pool *pool) +{ + swiotlb_release_slots(dev, tlb_addr, pool); +} + static int rmem_swiotlb_device_init(struct reserved_mem *rmem, struct device *dev) { --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE6703E51FD; Thu, 4 Jun 2026 08:40:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562451; cv=none; b=JXy1Pf+KaqlfPt1aTOUzlPPiZEs/Iln++bvPA921SM2uT7z/egnseyf3FDbX/6i2Q5t0acGIkFxv+NDQWiqxH6ztWv2E1RdDR2TejdXp00VARFRYIuTjxH6dEyQmCMbf1YWdKLbzVH9f7CZNjwPVAHC0LuI1VcRBBLJ1Y5MtWkk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562451; c=relaxed/simple; bh=e0zeMHGPIsebiML/DvYRgUnWvNGpu/wJhPgDz5wk8PA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GyQraqz7qHN7B9sqaOojNE3Z6JOvJ7OfjlajGOxvbJmxFt6G+4rmSauumKg+EBECz6hAOEYuMpQdjwRcgCWmTsJzPTiM5tsK1AV8Xsh1WjxpSQhuKSr7MLwaI8aoX6iBSaDcIuDRG1uKybHbE/xA7D8Pnv4rNhvUKLeV0F+I89I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dzQj+Xv1; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dzQj+Xv1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8164C1F00898; Thu, 4 Jun 2026 08:40:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562441; bh=+B5f91SDYjAMs4Y9geXO5xScntRrhAje95uzpPpXQjE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=dzQj+Xv1YAFhivBTK7CKWkM+0NQuzr0RN2f9LB9ULjUJFlIfftky7+KoKyQ6ybxAb kZq+RqQ2asniAHfXZsuPLq5Cl8ip0oTmOAqQn9C7X+nvS1coYG67x7Cpm3bSVL+bL5 vrepqvziucoNCPihmTW6n77YoFQWBwZ2w0MV4E2d9QIzNbohyIR9tfS9i0iaosR/eU Ipx7IIKwdgW1G7X8FTC8/8X8E5CF3blsI7qoIrOeeLUmOcsQfBnyhE14/6nLKxglOR QcTn9ne7mZ3pscR/8jBpwRPIxIH/OO/KhbSOctArSt0uYE7p88CBCpdkjgJ+ciwNBz 2iqe8OEs0HMTQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 03/20] dma-direct: use DMA_ATTR_CC_SHARED in alloc/free paths Date: Thu, 4 Jun 2026 14:09:42 +0530 Message-ID: <20260604083959.1265923-4-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Propagate force_dma_unencrypted() into DMA_ATTR_CC_SHARED in the dma-direct allocation path and use the attribute to drive the related decisions. This updates dma_direct_alloc(), dma_direct_free(), and dma_direct_alloc_pages() to fold the forced unencrypted case into attrs. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 53 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index a741c8a2ee66..90dc5057a0c0 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -193,16 +193,31 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; - bool mark_mem_decrypt =3D true; + bool mark_mem_decrypt =3D false; struct page *page; void *ret; =20 + /* + * DMA_ATTR_CC_SHARED is not a caller-visible dma_alloc_*() + * attribute. The direct allocator uses it internally after it has + * decided that the backing pages must be shared/decrypted, so the + * rest of the allocation path can consistently select DMA addresses, + * choose compatible pools and restore encryption on free. + */ + if (attrs & DMA_ATTR_CC_SHARED) + return NULL; + + if (force_dma_unencrypted(dev)) { + attrs |=3D DMA_ATTR_CC_SHARED; + mark_mem_decrypt =3D true; + } + size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) gfp |=3D __GFP_NOWARN; =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | DMA_ATTR_CC_SHARED)) =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp); =20 if (!dev_is_dma_coherent(dev)) { @@ -236,7 +251,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, * Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ - if ((remap || force_dma_unencrypted(dev)) && + if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 @@ -312,12 +327,24 @@ void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { phys_addr_t phys; - bool mark_mem_encrypted =3D true; + bool mark_mem_encrypted =3D false; struct io_tlb_pool *swiotlb_pool; unsigned int page_order =3D get_order(size); =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) { + /* see dma_direct_alloc() for details */ + WARN_ON(attrs & DMA_ATTR_CC_SHARED); + + /* + * if the device had requested for an unencrypted buffer, + * convert it to encrypted on free + */ + if (force_dma_unencrypted(dev)) { + attrs |=3D DMA_ATTR_CC_SHARED; + mark_mem_encrypted =3D true; + } + + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | DMA_ATTR_CC_SHARED)) =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) { /* cpu_addr is a struct page cookie, not a kernel address */ dma_free_contiguous(dev, cpu_addr, size); return; @@ -366,10 +393,14 @@ void dma_direct_free(struct device *dev, size_t size, struct page *dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp) { + unsigned long attrs =3D 0; struct page *page; void *ret; =20 - if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) + if (force_dma_unencrypted(dev)) + attrs |=3D DMA_ATTR_CC_SHARED; + + if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 if (is_swiotlb_for_alloc(dev)) { @@ -403,7 +434,11 @@ void dma_direct_free_pages(struct device *dev, size_t = size, phys_addr_t phys; void *vaddr =3D page_address(page); struct io_tlb_pool *swiotlb_pool; - bool mark_mem_encrypted =3D true; + /* + * if the device had requested for an unencrypted buffer, + * convert it to encrypted on free + */ + bool mark_mem_encrypted =3D force_dma_unencrypted(dev); =20 /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CF153E8333; Thu, 4 Jun 2026 08:40:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562460; cv=none; b=T2GBW00ZnbaB3/X1HVIkN8UF9UKv+fVHU1tV4QMzkjbKDiqCOCrMdeXafxHFc7Y9Ns7dATJ0DnCNwY29s3YE0KSXdsAEedKOZTxlimtd+s69p629HQDHPDFnx42jF3m/pUEHHWB5GP+sCYx3pR6SY/zd44h/T2ogjzxAmlx54zU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562460; c=relaxed/simple; bh=gCQiDtQ3gUOroN6tQaB4mcoDR/CnhellP5R1eFcJOTo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EtYh91L8pHFP20Q2B5G1BJFxEZpe1O6w+/ESLKjEics+n5eJ2qZuAnXFpwgBEAjuCFcX5cod5PsGeIcABobBccA9d4BA7mWgushSdxz+TGXkhBwCOqrvk3KNIRgjlqDfkQy9T3HYMjFR2iNiwcHc9egF7gyuhPRmlTYZ7IHPhCI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nirNpdm6; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nirNpdm6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C217E1F00899; Thu, 4 Jun 2026 08:40:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562450; bh=UoA06AF2jFoOpEty6iEpghCPkBEXBL7lN2LetCBuQvA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=nirNpdm61GhI9uD/QaPnGzj3UXVrWlFQ7rWYOt5n1qZeeoMvcXQKN0YuKdV6+fuUH NpXVYpyJS1Mr4ALgQKc/Dj07O0z9tbRA1LpbpoZwIOO0WXIOelpMyXTGf1l2UXDAra G+0dW7jmRmVlfzdF3HKV93gwA9cncAciPg/VH0ClInhTHzShDghVvBvemEhKquhTBy 4pgaTRiZs3nXLifjc6BTW5tPhZxMqKlMf7AUPMuCP5HCaeCDaP3g3bskKPXpZGYt8p iauNVO0jGJanSIGl1pz50kw9nbS+p0dhFut2dez9+wYClgOQZp/IvjAD6so6SEwm8n b1t2Mm9uYwNkA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 04/20] dma-pool: track decrypted atomic pools and select them via attrs Date: Thu, 4 Jun 2026 14:09:43 +0530 Message-ID: <20260604083959.1265923-5-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach the atomic DMA pool code to distinguish between encrypted and unencrypted pools, and make pool allocation select the matching pool based on DMA attributes. Introduce a dma_gen_pool wrapper that records whether a pool is unencrypted, initialize that state when the atomic pools are created, and use it when expanding and resizing the pools. Update dma_alloc_from_pool() to take attrs and skip pools whose encrypted state does not match DMA_ATTR_CC_SHARED. Update dma_free_from_pool() accordingly. Also pass DMA_ATTR_CC_SHARED from the swiotlb atomic allocation path so decrypted swiotlb allocations are taken from the correct atomic pool. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Reviewed-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/direct.c | 11 ++- kernel/dma/pool.c | 167 +++++++++++++++++++++++------------- kernel/dma/swiotlb.c | 7 +- 5 files changed, 123 insertions(+), 66 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 54d96e847f16..c2595bee3d41 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1673,7 +1673,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size= , dma_addr_t *handle, if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !gfpflags_allow_blocking(gfp) && !coherent) page =3D dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, - gfp, NULL); + gfp, attrs, NULL); else cpu_addr =3D iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); if (!cpu_addr) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 6a1832a73cad..696b2c3a2305 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -212,7 +212,7 @@ void *dma_common_pages_remap(struct page **pages, size_= t size, pgprot_t prot, void dma_common_free_remap(void *cpu_addr, size_t size); =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t flags, + void **cpu_addr, gfp_t flags, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)); bool dma_free_from_pool(struct device *dev, void *start, size_t size); =20 diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 90dc5057a0c0..681f16a984ab 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -154,7 +154,7 @@ static bool dma_direct_use_pool(struct device *dev, gfp= _t gfp) } =20 static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp) + dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { struct page *page; u64 phys_limit; @@ -164,7 +164,8 @@ static void *dma_direct_alloc_from_pool(struct device *= dev, size_t size, return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok); + page =3D dma_alloc_from_pool(dev, size, &ret, gfp, attrs, + dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); @@ -253,7 +254,8 @@ void *dma_direct_alloc(struct device *dev, size_t size, */ if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, + gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size); @@ -401,7 +403,8 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, attrs |=3D DMA_ATTR_CC_SHARED; =20 if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, + gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size); diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index 2b2fbb709242..be78474a6c49 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -12,12 +12,18 @@ #include #include #include +#include =20 -static struct gen_pool *atomic_pool_dma __ro_after_init; +struct dma_gen_pool { + bool unencrypted; + struct gen_pool *pool; +}; + +static struct dma_gen_pool atomic_pool_dma __ro_after_init; static unsigned long pool_size_dma; -static struct gen_pool *atomic_pool_dma32 __ro_after_init; +static struct dma_gen_pool atomic_pool_dma32 __ro_after_init; static unsigned long pool_size_dma32; -static struct gen_pool *atomic_pool_kernel __ro_after_init; +static struct dma_gen_pool atomic_pool_kernel __ro_after_init; static unsigned long pool_size_kernel; =20 /* Size can be defined by the coherent_pool command line */ @@ -76,11 +82,12 @@ static bool cma_in_zone(gfp_t gfp) return true; } =20 -static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size, +static int atomic_pool_expand(struct dma_gen_pool *dma_pool, size_t pool_s= ize, gfp_t gfp) { unsigned int order; struct page *page =3D NULL; + bool leak_pages =3D false; void *addr; int ret =3D -ENOMEM; =20 @@ -113,12 +120,17 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, * Memory in the atomic DMA pools must be unencrypted, the pools do not * shrink so no re-encryption occurs in dma_direct_free(). */ - ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), - 1 << order); - if (ret) - goto remove_mapping; - ret =3D gen_pool_add_virt(pool, (unsigned long)addr, page_to_phys(page), - pool_size, NUMA_NO_NODE); + if (dma_pool->unencrypted) { + ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), + 1 << order); + if (ret) { + leak_pages =3D true; + goto remove_mapping; + } + } + + ret =3D gen_pool_add_virt(dma_pool->pool, (unsigned long)addr, + page_to_phys(page), pool_size, NUMA_NO_NODE); if (ret) goto encrypt_mapping; =20 @@ -126,62 +138,67 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, return 0; =20 encrypt_mapping: - ret =3D set_memory_encrypted((unsigned long)page_to_virt(page), - 1 << order); - if (WARN_ON_ONCE(ret)) { - /* Decrypt succeeded but encrypt failed, purposely leak */ - goto out; - } + if (dma_pool->unencrypted && + set_memory_encrypted((unsigned long)page_to_virt(page), 1 << order)) + leak_pages =3D true; + remove_mapping: #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); free_page: - __free_pages(page, order); + if (!leak_pages) + __free_pages(page, order); #endif out: return ret; } =20 -static void atomic_pool_resize(struct gen_pool *pool, gfp_t gfp) +static void atomic_pool_resize(struct dma_gen_pool *dma_pool, gfp_t gfp) { - if (pool && gen_pool_avail(pool) < atomic_pool_size) - atomic_pool_expand(pool, gen_pool_size(pool), gfp); + if (dma_pool->pool && gen_pool_avail(dma_pool->pool) < atomic_pool_size) + atomic_pool_expand(dma_pool, gen_pool_size(dma_pool->pool), gfp); } =20 static void atomic_pool_work_fn(struct work_struct *work) { if (IS_ENABLED(CONFIG_ZONE_DMA)) - atomic_pool_resize(atomic_pool_dma, + atomic_pool_resize(&atomic_pool_dma, GFP_KERNEL | GFP_DMA); if (IS_ENABLED(CONFIG_ZONE_DMA32)) - atomic_pool_resize(atomic_pool_dma32, + atomic_pool_resize(&atomic_pool_dma32, GFP_KERNEL | GFP_DMA32); - atomic_pool_resize(atomic_pool_kernel, GFP_KERNEL); + atomic_pool_resize(&atomic_pool_kernel, GFP_KERNEL); } =20 -static __init struct gen_pool *__dma_atomic_pool_init(size_t pool_size, - gfp_t gfp) +static __init struct dma_gen_pool *__dma_atomic_pool_init(struct dma_gen_p= ool *dma_pool, + size_t pool_size, gfp_t gfp) { - struct gen_pool *pool; int ret; =20 - pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); - if (!pool) + dma_pool->pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); + if (!dma_pool->pool) return NULL; =20 - gen_pool_set_algo(pool, gen_pool_first_fit_order_align, NULL); + gen_pool_set_algo(dma_pool->pool, gen_pool_first_fit_order_align, NULL); + + /* if platform is using memory encryption atomic pools are by default dec= rypted. */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + dma_pool->unencrypted =3D true; + else + dma_pool->unencrypted =3D false; =20 - ret =3D atomic_pool_expand(pool, pool_size, gfp); + ret =3D atomic_pool_expand(dma_pool, pool_size, gfp); if (ret) { - gen_pool_destroy(pool); + gen_pool_destroy(dma_pool->pool); + dma_pool->pool =3D NULL; pr_err("DMA: failed to allocate %zu KiB %pGg pool for atomic allocation\= n", pool_size >> 10, &gfp); return NULL; } =20 pr_info("DMA: preallocated %zu KiB %pGg pool for atomic allocations\n", - gen_pool_size(pool) >> 10, &gfp); - return pool; + gen_pool_size(dma_pool->pool) >> 10, &gfp); + return dma_pool; } =20 #ifdef CONFIG_ZONE_DMA32 @@ -207,21 +224,22 @@ static int __init dma_atomic_pool_init(void) =20 /* All memory might be in the DMA zone(s) to begin with */ if (has_managed_zone(ZONE_NORMAL)) { - atomic_pool_kernel =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL); - if (!atomic_pool_kernel) + __dma_atomic_pool_init(&atomic_pool_kernel, atomic_pool_size, GFP_KERNEL= ); + if (!atomic_pool_kernel.pool) ret =3D -ENOMEM; } + if (has_managed_dma()) { - atomic_pool_dma =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA); - if (!atomic_pool_dma) + __dma_atomic_pool_init(&atomic_pool_dma, atomic_pool_size, + GFP_KERNEL | GFP_DMA); + if (!atomic_pool_dma.pool) ret =3D -ENOMEM; } + if (has_managed_dma32) { - atomic_pool_dma32 =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA32); - if (!atomic_pool_dma32) + __dma_atomic_pool_init(&atomic_pool_dma32, atomic_pool_size, + GFP_KERNEL | GFP_DMA32); + if (!atomic_pool_dma32.pool) ret =3D -ENOMEM; } =20 @@ -230,19 +248,44 @@ static int __init dma_atomic_pool_init(void) } postcore_initcall(dma_atomic_pool_init); =20 -static inline struct gen_pool *dma_guess_pool(struct gen_pool *prev, gfp_t= gfp) +static inline struct dma_gen_pool *__dma_guess_pool(struct dma_gen_pool *f= irst, + struct dma_gen_pool *second, struct dma_gen_pool *third) { - if (prev =3D=3D NULL) { + if (first->pool) + return first; + if (second && second->pool) + return second; + if (third && third->pool) + return third; + return NULL; +} + +static inline struct dma_gen_pool *dma_guess_pool(struct dma_gen_pool *pre= v, + gfp_t gfp) +{ + if (!prev) { if (gfp & GFP_DMA) - return atomic_pool_dma ?: atomic_pool_dma32 ?: atomic_pool_kernel; + return __dma_guess_pool(&atomic_pool_dma, + &atomic_pool_dma32, + &atomic_pool_kernel); + if (gfp & GFP_DMA32) - return atomic_pool_dma32 ?: atomic_pool_dma ?: atomic_pool_kernel; - return atomic_pool_kernel ?: atomic_pool_dma32 ?: atomic_pool_dma; + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, + &atomic_pool_kernel); + + return __dma_guess_pool(&atomic_pool_kernel, + &atomic_pool_dma32, + &atomic_pool_dma); } - if (prev =3D=3D atomic_pool_kernel) - return atomic_pool_dma32 ? atomic_pool_dma32 : atomic_pool_dma; - if (prev =3D=3D atomic_pool_dma32) - return atomic_pool_dma; + + if (prev =3D=3D &atomic_pool_kernel) + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, NULL); + + if (prev =3D=3D &atomic_pool_dma32) + return __dma_guess_pool(&atomic_pool_dma, NULL, NULL); + return NULL; } =20 @@ -272,16 +315,20 @@ static struct page *__dma_alloc_from_pool(struct devi= ce *dev, size_t size, } =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t gfp, + void **cpu_addr, gfp_t gfp, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; struct page *page; bool pool_found =3D false; =20 - while ((pool =3D dma_guess_pool(pool, gfp))) { + while ((dma_pool =3D dma_guess_pool(dma_pool, gfp))) { + + if (dma_pool->unencrypted !=3D !!(attrs & DMA_ATTR_CC_SHARED)) + continue; + pool_found =3D true; - page =3D __dma_alloc_from_pool(dev, size, pool, cpu_addr, + page =3D __dma_alloc_from_pool(dev, size, dma_pool->pool, cpu_addr, phys_addr_ok); if (page) return page; @@ -296,12 +343,14 @@ struct page *dma_alloc_from_pool(struct device *dev, = size_t size, =20 bool dma_free_from_pool(struct device *dev, void *start, size_t size) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; + + while ((dma_pool =3D dma_guess_pool(dma_pool, 0))) { =20 - while ((pool =3D dma_guess_pool(pool, 0))) { - if (!gen_pool_has_addr(pool, (unsigned long)start, size)) + if (!gen_pool_has_addr(dma_pool->pool, (unsigned long)start, size)) continue; - gen_pool_free(pool, (unsigned long)start, size); + + gen_pool_free(dma_pool->pool, (unsigned long)start, size); return true; } =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index ac03a6856c2e..be4d418d92ac 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -612,6 +612,7 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, u64 phys_limit, gfp_t gfp) { struct page *page; + unsigned long attrs =3D 0; =20 /* * Allocate from the atomic pools if memory is encrypted and @@ -623,8 +624,12 @@ static struct page *swiotlb_alloc_tlb(struct device *d= ev, size_t bytes, if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 + /* swiotlb considered decrypted by default */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + attrs =3D DMA_ATTR_CC_SHARED; + return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, - dma_coherent_ok); + attrs, dma_coherent_ok); } =20 gfp &=3D ~GFP_ZONEMASK; --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D11D3ED5A4; Thu, 4 Jun 2026 08:41:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562468; cv=none; b=qegST2OregjKnCCbB1aB0o09UQ+6AnbVkCZTSVbGIFpT0fJiwuFsIiZhKpyi5LdM/i4cxwOD3nlx9EY02MVkz7s2fCcOveWlktfYarmHtWCw72ruVewWUQP6B8dkiqtVu8l9hdw6LqUe/iYtQCsCUmXOTijyAqxXPAAgFisZnXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562468; c=relaxed/simple; bh=ZyAaao2IvZfCimPsjpUYzsJcOcQj4Vm8+chekynOTPg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=dg/T/p2MAIoq+KpihRx9c6CWfT9gVyzjAYNnZ+q+JsoiTw9crvKHdr4m2WjwO1HXuZN4A7ZEPHGQ6m6VXaiEMuMoQ4AbG2i5nUFyl3w/nqzSzO+wv+GYZEBASMy9CC+c1sghGo/Td6Uy2HRJAw4jbPaN/YASlTwiaUQn+627z8o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XVt6K0F0; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XVt6K0F0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4BD071F00893; Thu, 4 Jun 2026 08:40:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562459; bh=gJgN3Nk/k+tFUL+0/WMv/cq1G7XOyV8nwz1Z5AKbeXc=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=XVt6K0F0E+WZUeaVFrUGdVPJ3+NN/+MzYiEblTOgqmiQT5fnZpleh7w06dYvthsq9 JlIEFOyF/BQPCgA7554RUz5ctqo2iB6CyFUPyVjxOTr2o6yK8p/adqd2E67MgCbt4S +EXYbFb5JLqG9aNfxzVC6T0vJXhsQzoO92Z0Gi4ytITnmxcWe3q2nIsQvkE4X0lo2g h5cfhnXZ/MK4LI9zBn5rA+q15KN5+ErbWg0JFpKPiNZ1rIAEGLHSyFDvSyf7F+84Ln W8GP3VxY/EYOGfFhOc73Bs5SUJf2yT9A5ubOLvT2F8uKkrL3tyaTsC6yC5T0BWHokt K+vk0Pe6Vz7KA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 05/20] dma: swiotlb: pass mapping attributes by reference Date: Thu, 4 Jun 2026 14:09:44 +0530 Message-ID: <20260604083959.1265923-6-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Change swiotlb_tbl_map_single() to take the DMA mapping attributes by reference and update the direct callers accordingly. This is a preparatory change for a follow-up patch which updates the attributes based on the selected swiotlb pool. Keeping the signature change separate makes the follow-up patch easier to review. No functional change in this patch. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 2 +- drivers/xen/swiotlb-xen.c | 2 +- include/linux/swiotlb.h | 2 +- kernel/dma/swiotlb.c | 6 +++--- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index c2595bee3d41..725c7adb0a8d 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1180,7 +1180,7 @@ static phys_addr_t iommu_dma_map_swiotlb(struct devic= e *dev, phys_addr_t phys, trace_swiotlb_bounced(dev, phys, size); =20 phys =3D swiotlb_tbl_map_single(dev, phys, size, iova_mask(iovad), dir, - attrs); + &attrs); =20 /* * Untrusted devices should not see padding areas with random leftover diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 2cbf2b588f5b..8c4abe65cd49 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -243,7 +243,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, */ trace_swiotlb_bounced(dev, dev_addr, size); =20 - map =3D swiotlb_tbl_map_single(dev, phys, size, 0, dir, attrs); + map =3D swiotlb_tbl_map_single(dev, phys, size, 0, dir, &attrs); if (map =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 133bb8ca9032..29187cec90d8 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -238,7 +238,7 @@ static inline phys_addr_t default_swiotlb_limit(void) =20 phys_addr_t swiotlb_tbl_map_single(struct device *hwdev, phys_addr_t phys, size_t mapping_size, unsigned int alloc_aligned_mask, - enum dma_data_direction dir, unsigned long attrs); + enum dma_data_direction dir, unsigned long *attrs); dma_addr_t swiotlb_map(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs); =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index be4d418d92ac..78ce05857c00 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1391,7 +1391,7 @@ static unsigned long mem_used(struct io_tlb_mem *mem) */ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_ad= dr, size_t mapping_size, unsigned int alloc_align_mask, - enum dma_data_direction dir, unsigned long attrs) + enum dma_data_direction dir, unsigned long *attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; unsigned int offset; @@ -1425,7 +1425,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev= , phys_addr_t orig_addr, size =3D ALIGN(mapping_size + offset, alloc_align_mask + 1); index =3D swiotlb_find_slots(dev, orig_addr, size, alloc_align_mask, &poo= l); if (index =3D=3D -1) { - if (!(attrs & DMA_ATTR_NO_WARN)) + if (!(*attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slo= ts)\n", size, mem->nslabs, mem_used(mem)); @@ -1604,7 +1604,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_= t paddr, size_t size, =20 trace_swiotlb_bounced(dev, phys_to_dma(dev, paddr), size); =20 - swiotlb_addr =3D swiotlb_tbl_map_single(dev, paddr, size, 0, dir, attrs); + swiotlb_addr =3D swiotlb_tbl_map_single(dev, paddr, size, 0, dir, &attrs); if (swiotlb_addr =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A28653CEBB0; Thu, 4 Jun 2026 08:41:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562480; cv=none; b=uOyG1D3YXLgsQv5koYtH1B2o5R6pp0pzY56in0seNtCOxvv/X/vJLvACWgBOpMVW7YvIRB+5lJGJqIU6hg+B7oE6+Kh+nhkGe9jG+KPhczE/juTkl827kdjqvQu1TIH4Ej35xZaxwvy+a8UDPRjxaMQMDeDHMTjKCjlFGASQQN0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562480; c=relaxed/simple; bh=6HETLAt32CBbeEc4GAU5rHrvV27bMsoj5a4NMg17Sp0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=olf1ICkX4ursQ/IBi5R994wVQF8VbQ/k+CkiAjFOKUUV4BGpanrxG1zVpHDtS80RoK6F/jd4ZAFgslx1a5yIBT/o1YuF4qmo3i8O2Ja1Bn/aVoXhdAp5lBaM/k5nbZBIhZGTTiQvEbA+Ncw9bf9sqzzlKzuwtVJnqlk8voQ3DXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=K6VahbK6; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="K6VahbK6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4EE051F00898; Thu, 4 Jun 2026 08:41:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562469; bh=URo6DlnMpbtavFM7tRdTFhpFHPXTlgZ6/DhDkraf8VQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=K6VahbK6+J+Q76eYsnHHEyhjg9QaXu227M3xPkrMB2CAsrvGEmAgG6CStw1jVAA3/ i3I8j1afhgTuDT1+F7KukkLLV+gTaruspi2DS3o3T6EepbrAqEcXJLZaLzHaQfSaRU RfMl56/LZRFkDm3ann0u/1J4q6r3GffD7SSciLVgb22nzDicFgqlh0B5tTGDbVH5GU M67B4Y69Ybzn3ge+8RRJQX0lz1jjCArmG1KPWHvv5tW3efSxVawYfxS6NJ6a+m4s9y 70pDL+Q3FIKaLYXjuZy+GmHTDkvFgVx7S0cnEk6AnALdT+8sq1fQurJZLTL1+KWkko 83EdzX3vr0S4Q== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 06/20] dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED Date: Thu, 4 Jun 2026 14:09:45 +0530 Message-ID: <20260604083959.1265923-7-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach swiotlb to distinguish between encrypted and decrypted bounce buffer pools, and make allocation and mapping paths select a pool whose state matches the requested DMA attributes. Add a unencrypted flag to io_tlb_mem, initialize it for the default and restricted pools, and propagate DMA_ATTR_CC_SHARED into swiotlb pool allocation. Reject swiotlb alloc/map requests when the selected pool does not match the required encrypted/decrypted state. Also return DMA addresses with the matching phys_to_dma_{encrypted, unencrypted} helper so the DMA address encoding stays consistent with the chosen pool. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/dma-direct.h | 10 +++ include/linux/swiotlb.h | 8 +- kernel/dma/direct.c | 13 +++- kernel/dma/swiotlb.c | 154 ++++++++++++++++++++++++++++--------- 4 files changed, 142 insertions(+), 43 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index c249912456f9..94fad4e7c11e 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -77,6 +77,10 @@ static inline dma_addr_t dma_range_map_max(const struct = bus_dma_region *map) #ifndef phys_to_dma_unencrypted #define phys_to_dma_unencrypted phys_to_dma #endif + +#ifndef phys_to_dma_encrypted +#define phys_to_dma_encrypted phys_to_dma +#endif #else static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t pad= dr) { @@ -90,6 +94,12 @@ static inline dma_addr_t phys_to_dma_unencrypted(struct = device *dev, { return dma_addr_unencrypted(__phys_to_dma(dev, paddr)); } + +static inline dma_addr_t phys_to_dma_encrypted(struct device *dev, + phys_addr_t paddr) +{ + return dma_addr_encrypted(__phys_to_dma(dev, paddr)); +} /* * If memory encryption is supported, phys_to_dma will set the memory encr= yption * bit in the DMA address, and dma_to_phys will clear it. diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 29187cec90d8..4dcbf3931be1 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -81,6 +81,7 @@ struct io_tlb_pool { struct list_head node; struct rcu_head rcu; bool transient; + bool unencrypted; #endif }; =20 @@ -111,6 +112,7 @@ struct io_tlb_mem { struct dentry *debugfs; bool force_bounce; bool for_alloc; + bool unencrypted; #ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow; u64 phys_limit; @@ -282,7 +284,8 @@ static inline void swiotlb_sync_single_for_cpu(struct d= evice *dev, extern void swiotlb_print_info(void); =20 #ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc(struct device *dev, size_t size); +struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs); bool swiotlb_free(struct device *dev, struct page *page, size_t size); void swiotlb_free_from_pool(struct device *dev, phys_addr_t tlb_addr, size_t size, struct io_tlb_pool *pool); @@ -292,7 +295,8 @@ static inline bool is_swiotlb_for_alloc(struct device *= dev) return dev->dma_io_tlb_mem->for_alloc; } #else -static inline struct page *swiotlb_alloc(struct device *dev, size_t size) +static inline struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs) { return NULL; } diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 681f16a984ab..0b4a26c6b6fd 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -96,9 +96,10 @@ static int dma_set_encrypted(struct device *dev, void *v= addr, size_t size) return ret; } =20 -static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze) +static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze, + unsigned long attrs) { - struct page *page =3D swiotlb_alloc(dev, size); + struct page *page =3D swiotlb_alloc(dev, size, attrs); =20 if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { swiotlb_free(dev, page, size); @@ -258,8 +259,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (page) { + /* + * swiotlb allocations comes from pool already marked + * decrypted + */ mark_mem_decrypt =3D false; goto setup_page; } @@ -407,7 +412,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (!page) return NULL; =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 78ce05857c00..2bf3981db35d 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -259,10 +259,21 @@ void __init swiotlb_update_mem_attributes(void) struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; unsigned long bytes; =20 + /* + * if platform support memory encryption, swiotlb buffers are + * decrypted by default. + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + io_tlb_default_mem.unencrypted =3D true; + else + io_tlb_default_mem.unencrypted =3D false; + if (!mem->nslabs || mem->late_alloc) return; bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); - set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); + + if (io_tlb_default_mem.unencrypted) + set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, @@ -505,8 +516,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, if (!mem->slots) goto error_slots; =20 - set_memory_decrypted((unsigned long)vstart, - (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) + set_memory_decrypted((unsigned long)vstart, + (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); @@ -539,7 +552,9 @@ void __init swiotlb_exit(void) tbl_size =3D PAGE_ALIGN(mem->end - mem->start); slots_size =3D PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs)); =20 - set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) + set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (mem->late_alloc) { area_order =3D get_order(array_size(sizeof(*mem->areas), mem->nareas)); @@ -563,6 +578,7 @@ void __init swiotlb_exit(void) * @gfp: GFP flags for the allocation. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. + * @unencrypted: true to allocate unencrypted memory, false for encrypted = memory * * Allocate pages from the buddy allocator. If successful, make the alloca= ted * pages decrypted that they can be used for DMA. @@ -570,7 +586,8 @@ void __init swiotlb_exit(void) * Return: Decrypted pages, %NULL on allocation failure, or ERR_PTR(-EAGAI= N) * if the allocated physical address was above @phys_limit. */ -static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limi= t) +static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, + u64 phys_limit, bool unencrypted) { unsigned int order =3D get_order(bytes); struct page *page; @@ -588,13 +605,13 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) } =20 vaddr =3D phys_to_virt(paddr); - if (set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (unencrypted && set_memory_decrypted((unsigned long)vaddr, PFN_UP(byte= s))) goto error; return page; =20 error: /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (unencrypted && !set_memory_encrypted((unsigned long)vaddr, PFN_UP(byt= es))) __free_pages(page, order); return NULL; } @@ -604,30 +621,26 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) * @dev: Device for which a memory pool is allocated. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. + * @attrs: DMA attributes for the allocation. * @gfp: GFP flags for the allocation. * * Return: Allocated pages, or %NULL on allocation failure. */ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, - u64 phys_limit, gfp_t gfp) + u64 phys_limit, unsigned long attrs, gfp_t gfp) { struct page *page; - unsigned long attrs =3D 0; =20 /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ - if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) { + if (!gfpflags_allow_blocking(gfp) && (attrs & DMA_ATTR_CC_SHARED)) { void *vaddr; =20 if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 - /* swiotlb considered decrypted by default */ - if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) - attrs =3D DMA_ATTR_CC_SHARED; - return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, attrs, dma_coherent_ok); } @@ -638,7 +651,8 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, else if (phys_limit <=3D DMA_BIT_MASK(32)) gfp |=3D __GFP_DMA32; =20 - while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit))) { + while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit, + !!(attrs & DMA_ATTR_CC_SHARED)))) { if (IS_ENABLED(CONFIG_ZONE_DMA32) && phys_limit < DMA_BIT_MASK(64) && !(gfp & (__GFP_DMA32 | __GFP_DMA))) @@ -657,15 +671,18 @@ static struct page *swiotlb_alloc_tlb(struct device *= dev, size_t bytes, * swiotlb_free_tlb() - free a dynamically allocated IO TLB buffer * @vaddr: Virtual address of the buffer. * @bytes: Size of the buffer. + * @unencrypted: true if @vaddr was allocated decrypted and must be + * re-encrypted before being freed */ -static void swiotlb_free_tlb(void *vaddr, size_t bytes) +static void swiotlb_free_tlb(void *vaddr, size_t bytes, bool unencrypted) { if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool(NULL, vaddr, bytes)) return; =20 /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (!unencrypted || + !set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) __free_pages(virt_to_page(vaddr), get_order(bytes)); } =20 @@ -676,6 +693,7 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes) * @nslabs: Desired (maximum) number of slabs. * @nareas: Number of areas. * @phys_limit: Maximum DMA buffer physical address. + * @attrs: DMA attributes for the allocation. * @gfp: GFP flags for the allocations. * * Allocate and initialize a new IO TLB memory pool. The actual number of @@ -686,7 +704,8 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes) */ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, unsigned long minslabs, unsigned long nslabs, - unsigned int nareas, u64 phys_limit, gfp_t gfp) + unsigned int nareas, u64 phys_limit, + unsigned long attrs, gfp_t gfp) { struct io_tlb_pool *pool; unsigned int slot_order; @@ -704,9 +723,10 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct d= evice *dev, if (!pool) goto error; pool->areas =3D (void *)pool + sizeof(*pool); + pool->unencrypted =3D !!(attrs & DMA_ATTR_CC_SHARED); =20 tlb_size =3D nslabs << IO_TLB_SHIFT; - while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp))) { + while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, attrs, gfp)= )) { if (nslabs <=3D minslabs) goto error_tlb; nslabs =3D ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -724,7 +744,8 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, return pool; =20 error_slots: - swiotlb_free_tlb(page_address(tlb), tlb_size); + swiotlb_free_tlb(page_address(tlb), tlb_size, + !!(attrs & DMA_ATTR_CC_SHARED)); error_tlb: kfree(pool); error: @@ -742,7 +763,9 @@ static void swiotlb_dyn_alloc(struct work_struct *work) struct io_tlb_pool *pool; =20 pool =3D swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs, - default_nareas, mem->phys_limit, GFP_KERNEL); + default_nareas, mem->phys_limit, + mem->unencrypted ? DMA_ATTR_CC_SHARED : 0, + GFP_KERNEL); if (!pool) { pr_warn_ratelimited("Failed to allocate new pool"); return; @@ -762,7 +785,7 @@ static void swiotlb_dyn_free(struct rcu_head *rcu) size_t tlb_size =3D pool->end - pool->start; =20 free_pages((unsigned long)pool->slots, get_order(slots_size)); - swiotlb_free_tlb(pool->vaddr, tlb_size); + swiotlb_free_tlb(pool->vaddr, tlb_size, pool->unencrypted); kfree(pool); } =20 @@ -1037,13 +1060,11 @@ static void dec_transient_used(struct io_tlb_mem *m= em, unsigned int nslots) * Return: Index of the first allocated slot, or -1 on error. */ static int swiotlb_search_pool_area(struct device *dev, struct io_tlb_pool= *pool, - int area_index, phys_addr_t orig_addr, size_t alloc_size, - unsigned int alloc_align_mask) + int area_index, phys_addr_t orig_addr, dma_addr_t tbl_dma_addr, + size_t alloc_size, unsigned int alloc_align_mask) { struct io_tlb_area *area =3D pool->areas + area_index; unsigned long boundary_mask =3D dma_get_seg_boundary(dev); - dma_addr_t tbl_dma_addr =3D - phys_to_dma_unencrypted(dev, pool->start) & boundary_mask; unsigned long max_slots =3D get_max_slots(boundary_mask); unsigned int iotlb_align_mask =3D dma_get_min_align_mask(dev); unsigned int nslots =3D nr_slots(alloc_size), stride; @@ -1056,6 +1077,8 @@ static int swiotlb_search_pool_area(struct device *de= v, struct io_tlb_pool *pool BUG_ON(!nslots); BUG_ON(area_index >=3D pool->nareas); =20 + tbl_dma_addr &=3D boundary_mask; + /* * Historically, swiotlb allocations >=3D PAGE_SIZE were guaranteed to be * page-aligned in the absence of any other alignment requirements. @@ -1167,6 +1190,7 @@ static int swiotlb_search_area(struct device *dev, in= t start_cpu, { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; int area_index; int index =3D -1; =20 @@ -1175,9 +1199,15 @@ static int swiotlb_search_area(struct device *dev, i= nt start_cpu, if (cpu_offset >=3D pool->nareas) continue; area_index =3D (start_cpu + cpu_offset) & (pool->nareas - 1); + + if (mem->unencrypted) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + index =3D swiotlb_search_pool_area(dev, pool, area_index, - orig_addr, alloc_size, - alloc_align_mask); + orig_addr, tbl_dma_addr, + alloc_size, alloc_align_mask); if (index >=3D 0) { *retpool =3D pool; break; @@ -1207,6 +1237,7 @@ static int swiotlb_find_slots(struct device *dev, phy= s_addr_t orig_addr, { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; unsigned long nslabs; unsigned long flags; u64 phys_limit; @@ -1232,11 +1263,17 @@ static int swiotlb_find_slots(struct device *dev, p= hys_addr_t orig_addr, nslabs =3D nr_slots(alloc_size); phys_limit =3D min_not_zero(*dev->dma_mask, dev->bus_dma_limit); pool =3D swiotlb_alloc_pool(dev, nslabs, nslabs, 1, phys_limit, + mem->unencrypted ? DMA_ATTR_CC_SHARED : 0, GFP_NOWAIT); if (!pool) return -1; =20 - index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, + if (mem->unencrypted) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + + index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, tbl_dma_addr, alloc_size, alloc_align_mask); if (index < 0) { swiotlb_dyn_free(&pool->rcu); @@ -1281,15 +1318,23 @@ static int swiotlb_find_slots(struct device *dev, p= hys_addr_t orig_addr, size_t alloc_size, unsigned int alloc_align_mask, struct io_tlb_pool **retpool) { + struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; + dma_addr_t tbl_dma_addr; int start, i; int index; =20 - *retpool =3D pool =3D &dev->dma_io_tlb_mem->defpool; + *retpool =3D pool =3D &mem->defpool; + if (mem->unencrypted) + tbl_dma_addr =3D phys_to_dma_unencrypted(dev, pool->start); + else + tbl_dma_addr =3D phys_to_dma_encrypted(dev, pool->start); + i =3D start =3D raw_smp_processor_id() & (pool->nareas - 1); do { index =3D swiotlb_search_pool_area(dev, pool, i, orig_addr, - alloc_size, alloc_align_mask); + tbl_dma_addr, alloc_size, + alloc_align_mask); if (index >=3D 0) return index; if (++i >=3D pool->nareas) @@ -1372,9 +1417,19 @@ static unsigned long mem_used(struct io_tlb_mem *mem) * any pre- or post-padding for alignment * @alloc_align_mask: Required start and end alignment of the allocated bu= ffer * @dir: DMA direction - * @attrs: Optional DMA attributes for the map operation + * @attrs: Optional DMA attributes for the map operation, updated + * to match the selected SWIOTLB pool * * Find and allocate a suitable sequence of IO TLB slots for the request. + * The device's SWIOTLB pool must match the device's current DMA encryption + * requirements. If the device requires decrypted DMA, bouncing is done th= rough + * an unencrypted pool and the mapping is marked shared. If the device can= DMA + * to encrypted memory, bouncing is done through an encrypted pool even wh= en the + * original DMA address was unencrypted. Enabling encrypted DMA for a devi= ce is + * therefore expected to update its default io_tlb_mem to an encrypted poo= l, so + * later bounce mappings for both encrypted and decrypted original memory = use + * that encrypted pool. + * * The allocated space starts at an alignment specified by alloc_align_mas= k, * and the size of the allocated space is rounded up so that the total amo= unt * of allocated space is a multiple of (alloc_align_mask + 1). If @@ -1411,6 +1466,16 @@ phys_addr_t swiotlb_tbl_map_single(struct device *de= v, phys_addr_t orig_addr, if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) pr_warn_once("Memory encryption is active and system is using DMA bounce= buffers\n"); =20 + /* swiotlb pool is incorrect for this device */ + if (unlikely(mem->unencrypted !=3D force_dma_unencrypted(dev))) + return (phys_addr_t)DMA_MAPPING_ERROR; + + /* Force attrs to match the kind of memory in the pool */ + if (mem->unencrypted) + *attrs |=3D DMA_ATTR_CC_SHARED; + else + *attrs &=3D ~DMA_ATTR_CC_SHARED; + /* * The default swiotlb memory pool is allocated with PAGE_SIZE * alignment. If a mapping is requested with larger alignment, @@ -1608,8 +1673,11 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr= _t paddr, size_t size, if (swiotlb_addr =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 - /* Ensure that the address returned is DMA'ble */ - dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + if (attrs & DMA_ATTR_CC_SHARED) + dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + else + dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, @@ -1773,7 +1841,7 @@ static inline void swiotlb_create_debugfs_files(struc= t io_tlb_mem *mem, =20 #ifdef CONFIG_DMA_RESTRICTED_POOL =20 -struct page *swiotlb_alloc(struct device *dev, size_t size) +struct page *swiotlb_alloc(struct device *dev, size_t size, unsigned long = attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; @@ -1784,6 +1852,9 @@ struct page *swiotlb_alloc(struct device *dev, size_t= size) if (!mem) return NULL; =20 + if (mem->unencrypted !=3D !!(attrs & DMA_ATTR_CC_SHARED)) + return NULL; + align =3D (1 << (get_order(size) + PAGE_SHIFT)) - 1; index =3D swiotlb_find_slots(dev, 0, size, align, &pool); if (index =3D=3D -1) @@ -1859,9 +1930,18 @@ static int rmem_swiotlb_device_init(struct reserved_= mem *rmem, kfree(mem); return -ENOMEM; } + /* + * if platform supports memory encryption, + * restricted mem pool is decrypted by default + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + mem->unencrypted =3D true; + set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), + rmem->size >> PAGE_SHIFT); + } else { + mem->unencrypted =3D false; + } =20 - set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), - rmem->size >> PAGE_SHIFT); swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs, false, nareas); mem->force_bounce =3D true; --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E98473F54AD; Thu, 4 Jun 2026 08:41:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562485; cv=none; b=dNQxTYQNsKGtFVF1WbhBHpEEFVoDufbyE7Q4wLzJ+MCtUL9QIen+aMyTK/PhEOGzhCs/aGQ9hArkAP5dkh8zzZce35EvxRFHiAPnepBOmj9nnR0mhH8uNSjPzp+u89yBGjC5T8nAYW0pyoKUJsE2NZTTqOs4j0FKH1hkPLlEPj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562485; c=relaxed/simple; bh=nvh8fIDNVPfMC0ICTaQnFFRAmvBIOsOrYj/UqKcPxjs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pyXv4lLDifI8WVdGH7ivBaYsRHhbcjkuUraHN5DS6SKyY03RHJezFDIJsoYNBKlWn+bHPTVE6UbbwzMwn53dunl7q09tDMGAOBC8ECMilacKCIbNvrOvqBA7n1RBoZsyh7MxJENa+AiwONamgmVjmqlUQhzFBhL4U+vkrCat/sM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dGHCOI2B; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dGHCOI2B" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91CED1F00893; Thu, 4 Jun 2026 08:41:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562478; bh=7TH+Qx6Oz0KBtBZSC9IyhaEGjXmJSkpAjhqTot9QjuU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=dGHCOI2Bz9um/GE0RCmyC1Qk8mhygoFuyVx4f9RWLcknMIrnOeO1o89nB7cD6S1eF HCwS2uugMipB1BZZH5MOmysdZMPH0ywJ3v9XoAj1yRJIpnDZO+zVuVaxmS7xdumvUp NhS3o2stcEn3JDpf4ibdzQtQIEtCrc8xyj9TzgZgAX2LAxwHSvOZoGAOydxesY33LJ w/TE2fgsn7CR7xJMRQ2Yv8WjUDeo1bXqs/XKP2MCtX+ZKF4fT7bhuDsxbjYKpoWdVf WEkG7JikC58CakVgsl1VdHpKsKTWdOMBv770v1GQXd2lrC7+ywI2slHBfh2xQzbqhw voPqTvIfAquqw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 07/20] dma-mapping: make dma_pgprot() honor DMA_ATTR_CC_SHARED Date: Thu, 4 Jun 2026 14:09:46 +0530 Message-ID: <20260604083959.1265923-8-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold encrypted/decrypted pgprot selection into dma_pgprot() so callers do not need to adjust the page protection separately. Update dma_pgprot() to apply pgprot_decrypted() when DMA_ATTR_CC_SHARED is set and pgprot_encrypted() otherwise Convert the dma-direct allocation and mmap paths to pass DMA_ATTR_CC_SHARED instead of open-coding force_dma_unencrypted() handling around dma_pgprot(). Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 8 +++----- kernel/dma/mapping.c | 16 ++++++++++++---- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 0b4a26c6b6fd..e4cba322386d 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -290,9 +290,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 - if (force_dma_unencrypted(dev)) - prot =3D pgprot_decrypted(prot); - /* remove any dirty cache lines on the kernel alias */ arch_dma_prep_coherent(page, size); =20 @@ -614,9 +611,10 @@ int dma_direct_mmap(struct device *dev, struct vm_area= _struct *vma, unsigned long pfn =3D PHYS_PFN(dma_to_phys(dev, dma_addr)); int ret =3D -ENXIO; =20 - vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); if (force_dma_unencrypted(dev)) - vma->vm_page_prot =3D pgprot_decrypted(vma->vm_page_prot); + attrs |=3D DMA_ATTR_CC_SHARED; + + vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); =20 if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) return ret; diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index e6b07f160d20..3f4ae283c466 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -539,13 +539,21 @@ EXPORT_SYMBOL(dma_get_sgtable_attrs); */ pgprot_t dma_pgprot(struct device *dev, pgprot_t prot, unsigned long attrs) { + pgprot_t dma_prot; + if (dev_is_dma_coherent(dev)) - return prot; + dma_prot =3D prot; #ifdef CONFIG_ARCH_HAS_DMA_WRITE_COMBINE - if (attrs & DMA_ATTR_WRITE_COMBINE) - return pgprot_writecombine(prot); + else if (attrs & DMA_ATTR_WRITE_COMBINE) + dma_prot =3D pgprot_writecombine(prot); #endif - return pgprot_dmacoherent(prot); + else + dma_prot =3D pgprot_dmacoherent(prot); + + if (attrs & DMA_ATTR_CC_SHARED) + return pgprot_decrypted(dma_prot); + else + return pgprot_encrypted(dma_prot); } #endif /* CONFIG_MMU */ =20 --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C06C3F58F5; Thu, 4 Jun 2026 08:41:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562497; cv=none; b=ltKasDVvHiVP6KjgegHO6xGfiV3E0yIojHaF6fIGJC8RcQlJSZWpg9tGQjjocChXEYzWwc5ODkxl4UsDXBHZ8i9MAIyJQPejFliGgoSPZ/dnV4n7VEuC77p5ckw90zTOTKCEm5OGQhMsJGbKzxuIWeHIbPnWBebzHdEvBFzt+4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562497; c=relaxed/simple; bh=8DA2l/aD5rZoWlAk0Rc10Wg1tBCJuJ0FRh2pOhb8PDc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oOyi24fAQ834pOt6GMku8zDx2GQMAohPsfx0Gi3+kGpwhQPTMAZIRWvGPKahqrNeuSNCcBJLrZ1sEs67520O6XAv7NKLXt22zF947IEHjW1tKhKH4hX7qEdKvAVbgCKuI7oKMbv5EGt145jpHSnwcatA1cMUZquI6mdjB6+oYIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PowogtMA; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PowogtMA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D35B71F00899; Thu, 4 Jun 2026 08:41:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562487; bh=3mRSiMYV0JgpP11Zy/vXlKZg4TGLhucRNZEepjj5feU=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=PowogtMAiU4yc/pAcLLbanP1alNGyy3UF/k0wvtkWglA1XL/srhFQ1i1oAtyu7cZ6 JOvTpPDTBS2kdmVDvGJrFJXD0QawTZc67omT2YMJoretOhp1koW/NoLDWzTq3dabwm ZVK452dMu72fo7x206w5k5495kkiMIOjicP5z2o5ghxdpyj0KmjQJ0lK+x+EHnMAm+ aouHaQuJGEkyjmi26R0nJto8wf19+k98nGOGxfr2L1xgRtZRd0DVeI/MXPPgMARG5S nteoJmokfM4g9cK/iRqAAo0vp9IP405Eh4Ziyfeakq6JwWprCNZ3D5FcAYENVIYDlC nfVbQ9zgyB8cA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 08/20] dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks Date: Thu, 4 Jun 2026 14:09:47 +0530 Message-ID: <20260604083959.1265923-9-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability check can reject encrypted DMA addresses for devices that require unencrypted/shared DMA. Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected SWIOTLB pool is decrypted so the capability check sees the correct DMA address attribute. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- arch/x86/kernel/amd_gart_64.c | 30 ++++++++++++++++-------------- drivers/xen/swiotlb-xen.c | 6 +++--- include/linux/dma-direct.h | 10 +++++++++- kernel/dma/direct.h | 6 +++--- kernel/dma/swiotlb.c | 2 +- 5 files changed, 32 insertions(+), 22 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index e8000a56732e..b5f1f031d45b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -180,22 +180,23 @@ static void iommu_full(struct device *dev, size_t siz= e, int dir) } =20 static inline int -need_iommu(struct device *dev, unsigned long addr, size_t size) +need_iommu(struct device *dev, unsigned long addr, size_t size, unsigned l= ong attrs) { - return force_iommu || !dma_capable(dev, addr, size, true); + return force_iommu || !dma_capable(dev, addr, size, true, attrs); } =20 static inline int -nonforced_iommu(struct device *dev, unsigned long addr, size_t size) +nonforced_iommu(struct device *dev, unsigned long addr, size_t size, + unsigned long attrs) { - return !dma_capable(dev, addr, size, true); + return !dma_capable(dev, addr, size, true, attrs); } =20 /* Map a single continuous physical area into the IOMMU. * Caller needs to check if the iommu is needed and flush. */ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem, - size_t size, int dir, unsigned long align_mask) + size_t size, int dir, unsigned long align_mask, unsigned long attrs) { unsigned long npages =3D iommu_num_pages(phys_mem, size, PAGE_SIZE); unsigned long iommu_page; @@ -206,7 +207,7 @@ static dma_addr_t dma_map_area(struct device *dev, dma_= addr_t phys_mem, =20 iommu_page =3D alloc_iommu(dev, npages, align_mask); if (iommu_page =3D=3D -1) { - if (!nonforced_iommu(dev, phys_mem, size)) + if (!nonforced_iommu(dev, phys_mem, size, attrs)) return phys_mem; if (panic_on_overflow) panic("dma_map_area overflow %lu bytes\n", size); @@ -231,10 +232,10 @@ static dma_addr_t gart_map_phys(struct device *dev, p= hys_addr_t paddr, if (unlikely(attrs & DMA_ATTR_MMIO)) return DMA_MAPPING_ERROR; =20 - if (!need_iommu(dev, paddr, size)) + if (!need_iommu(dev, paddr, size, attrs)) return paddr; =20 - bus =3D dma_map_area(dev, paddr, size, dir, 0); + bus =3D dma_map_area(dev, paddr, size, dir, 0, attrs); flush_gart(); =20 return bus; @@ -289,7 +290,7 @@ static void gart_unmap_sg(struct device *dev, struct sc= atterlist *sg, int nents, =20 /* Fallback for dma_map_sg in case of overflow */ static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg, - int nents, int dir) + int nents, int dir, unsigned long attrs) { struct scatterlist *s; int i; @@ -301,8 +302,8 @@ static int dma_map_sg_nonforce(struct device *dev, stru= ct scatterlist *sg, for_each_sg(sg, s, nents, i) { unsigned long addr =3D sg_phys(s); =20 - if (nonforced_iommu(dev, addr, s->length)) { - addr =3D dma_map_area(dev, addr, s->length, dir, 0); + if (nonforced_iommu(dev, addr, s->length, attrs)) { + addr =3D dma_map_area(dev, addr, s->length, dir, 0, attrs); if (addr =3D=3D DMA_MAPPING_ERROR) { if (i > 0) gart_unmap_sg(dev, sg, i, dir, 0); @@ -401,7 +402,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, s->dma_address =3D addr; BUG_ON(s->length =3D=3D 0); =20 - nextneed =3D need_iommu(dev, addr, s->length); + nextneed =3D need_iommu(dev, addr, s->length, attrs); =20 /* Handle the previous not yet processed entries */ if (i > start) { @@ -449,7 +450,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, =20 /* When it was forced or merged try again in a dumb way */ if (force_iommu || iommu_merge) { - out =3D dma_map_sg_nonforce(dev, sg, nents, dir); + out =3D dma_map_sg_nonforce(dev, sg, nents, dir, attrs); if (out > 0) return out; } @@ -473,7 +474,8 @@ gart_alloc_coherent(struct device *dev, size_t size, dm= a_addr_t *dma_addr, return vaddr; =20 *dma_addr =3D dma_map_area(dev, virt_to_phys(vaddr), size, - DMA_BIDIRECTIONAL, (1UL << get_order(size)) - 1); + DMA_BIDIRECTIONAL, + (1UL << get_order(size)) - 1, attrs); flush_gart(); if (unlikely(*dma_addr =3D=3D DMA_MAPPING_ERROR)) goto out_free; diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 8c4abe65cd49..e2538824ef52 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -212,7 +212,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, BUG_ON(dir =3D=3D DMA_NONE); =20 if (attrs & DMA_ATTR_MMIO) { - if (unlikely(!dma_capable(dev, phys, size, false))) { + if (unlikely(!dma_capable(dev, phys, size, false, attrs))) { dev_err_once( dev, "DMA addr %pa+%zu overflow (mask %llx, bus limit %llx).\n", @@ -231,7 +231,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, * we can safely return the device addr and not worry about bounce * buffering it. */ - if (dma_capable(dev, dev_addr, size, true) && + if (dma_capable(dev, dev_addr, size, true, attrs) && !dma_kmalloc_needs_bounce(dev, size, dir) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && @@ -253,7 +253,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, /* * Ensure that the address returned is DMA'ble */ - if (unlikely(!dma_capable(dev, dev_addr, size, true))) { + if (unlikely(!dma_capable(dev, dev_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, map, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, map)); diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 94fad4e7c11e..daa31a1adf7b 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -135,12 +135,20 @@ static inline bool force_dma_unencrypted(struct devic= e *dev) #endif /* CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED */ =20 static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t= size, - bool is_ram) + bool is_ram, unsigned long attrs) { dma_addr_t end =3D addr + size - 1; =20 if (addr =3D=3D DMA_MAPPING_ERROR) return false; + /* + * The DMA address was derived from encrypted RAM, but this device + * requires unencrypted DMA addresses. Treat it as not DMA-capable + * so the caller can fall back to a suitable SWIOTLB pool. + */ + if (!(attrs & DMA_ATTR_CC_SHARED) && force_dma_unencrypted(dev)) + return false; + if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) && min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn))) return false; diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index 7140c208c123..e05dc7649366 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -101,15 +101,15 @@ static inline dma_addr_t dma_direct_map_phys(struct d= evice *dev, =20 if (attrs & DMA_ATTR_MMIO) { dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else if (attrs & DMA_ATTR_CC_SHARED) { dma_addr =3D phys_to_dma_unencrypted(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else { dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_swiotlb_active(dev) && !(attrs & DMA_ATTR_REQUIRE_COHERENT)) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 2bf3981db35d..f4e8b241a1c4 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1678,7 +1678,7 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr_= t paddr, size_t size, else dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); =20 - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, swiotlb_addr)); --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 238403F1ADB; Thu, 4 Jun 2026 08:41:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562505; cv=none; b=l/C8WKWrPO1IM3ljq0iMrZ7P6zkWl/pwVjlJcUuz7/wJUqbSW3yf13FxNB7QdTk70jSSH5q3aiAcB91asmddOaPap81KPvU1S/XwUeNf3QjIRviZmBjDwSNJH72TzZPlJ0Fk+gp5O23WD6eWxRcHU8pfFcmjaeXc9FxGn+qvTIU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562505; c=relaxed/simple; bh=BRJxEn4t8YCZ7/EXDNEDlEd+dpxx7F2DSPDX8TEl/vk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FYeGutVA6YRkW6SR90MjG5J1DdLvxjSr1fixvjDOg5dSFZufJ1NblWvnacFFPbdYv3pKIUflj17IxwHbN4ZHuPe6KisJXKoBDDkW0kfGgMPLPMJ28yJOf7m9fdxglt72+VgP3aL9UDZdyL/+pVnjVoobjpuQ0dNp35ywNYujqYo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Xvdh0MQe; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Xvdh0MQe" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 204E51F00898; Thu, 4 Jun 2026 08:41:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562496; bh=3CiqjWgPjIETOFd/RHaVKoSJDsdZciSeCPQBjeeboj0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Xvdh0MQeJ/b/48NOUsnmuAsauPP0E6/ujqJYsTLG4sz/54ny/xrxp7VdtMBxRDDvJ Ud49deqI2YWvvZ6K0b4OOGEha2+OqCAboobMdaZ3ApmnNHeCLAVz7W56qPKsE7CCvg F/OsO3tIBsb0UkyEDnsmj/rcbZF0dVRP4syf8Iq7cEzw/EDtqcJCHtpyfeZ7G7+dou RCkeNRqWKGooUbundlUU9qj6HaVFahiRLHqgPuaY9ih2sghXuPzRAJTFW8OKsNDizx VZLV+T0MCQRjG/XVjuqLqcLaUZ+qalbzcKfEkSMo0JhFJnSiZyUdLsyRv2XEDHdK9q YkgH52QYueURg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 09/20] dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED Date: Thu, 4 Jun 2026 14:09:48 +0530 Message-ID: <20260604083959.1265923-10-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_direct_map_phys() to select the DMA address encoding based on DMA_ATTR_CC_SHARED. Use phys_to_dma_unencrypted() for decrypted mappings and phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA but the source physical address is still encrypted, force the mapping through swiotlb so the DMA address and backing memory attributes remain consistent. Update the arm64, x86, s390 and powerpc secure-guest setup to not use swiotlb force option Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- Changes from v3: * Handle DMA_ATTR_MMIO --- arch/arm64/mm/init.c | 4 +-- arch/powerpc/platforms/pseries/svm.c | 2 +- arch/s390/mm/init.c | 2 +- arch/x86/kernel/pci-dma.c | 4 +-- kernel/dma/direct.c | 4 ++- kernel/dma/direct.h | 45 +++++++++++++++------------- 6 files changed, 31 insertions(+), 30 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 97987f850a33..acf67c7064db 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -338,10 +338,8 @@ void __init arch_mm_preinit(void) unsigned int flags =3D SWIOTLB_VERBOSE; bool swiotlb =3D max_pfn > PFN_DOWN(arm64_dma_phys_limit); =20 - if (is_realm_world()) { + if (is_realm_world()) swiotlb =3D true; - flags |=3D SWIOTLB_FORCE; - } =20 if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) { /* diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/= pseries/svm.c index 384c9dc1899a..7a403dbd35ee 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -29,7 +29,7 @@ static int __init init_svm(void) * need to use the SWIOTLB buffer for DMA even if dma_capable() says * otherwise. */ - ppc_swiotlb_flags |=3D SWIOTLB_ANY | SWIOTLB_FORCE; + ppc_swiotlb_flags |=3D SWIOTLB_ANY; =20 /* Share the SWIOTLB buffer with the host. */ swiotlb_update_mem_attributes(); diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index ad3c6d92b801..581af1483c42 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -163,7 +163,7 @@ static void __init pv_init(void) virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc); =20 /* make sure bounce buffers are shared */ - swiotlb_init(true, SWIOTLB_FORCE | SWIOTLB_VERBOSE); + swiotlb_init(true, SWIOTLB_VERBOSE); swiotlb_update_mem_attributes(); } =20 diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index 6267363e0189..75cf8f6ae8cd 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -59,10 +59,8 @@ static void __init pci_swiotlb_detect(void) * bounce buffers as the hypervisor can't access arbitrary VM memory * that is not explicitly shared with it. */ - if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) x86_swiotlb_enable =3D true; - x86_swiotlb_flags |=3D SWIOTLB_FORCE; - } } #else static inline void __init pci_swiotlb_detect(void) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index e4cba322386d..6d0ce3cfd8cc 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -702,8 +702,10 @@ size_t dma_direct_max_mapping_size(struct device *dev) { /* If SWIOTLB is active, use its maximum mapping size */ if (is_swiotlb_active(dev) && - (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev))) + (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev) || + force_dma_unencrypted(dev))) return swiotlb_max_mapping_size(dev); + return SIZE_MAX; } =20 diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e05dc7649366..f3fc28f352ba 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -88,37 +88,40 @@ static inline dma_addr_t dma_direct_map_phys(struct dev= ice *dev, { dma_addr_t dma_addr; =20 + /* + * For a device requiring unencrypted DMA, MMIO memory is treated + * as shared by default. + */ + if (force_dma_unencrypted(dev) && (attrs & DMA_ATTR_MMIO)) + attrs |=3D DMA_ATTR_CC_SHARED; + if (is_swiotlb_force_bounce(dev)) { - if (!(attrs & DMA_ATTR_CC_SHARED)) { - if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) - return DMA_MAPPING_ERROR; + if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) + return DMA_MAPPING_ERROR; =20 - return swiotlb_map(dev, phys, size, dir, attrs); - } - } else if (attrs & DMA_ATTR_CC_SHARED) { - return DMA_MAPPING_ERROR; + return swiotlb_map(dev, phys, size, dir, attrs); } =20 - if (attrs & DMA_ATTR_MMIO) { - dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) - goto err_overflow; - } else if (attrs & DMA_ATTR_CC_SHARED) { + if (attrs & DMA_ATTR_CC_SHARED) dma_addr =3D phys_to_dma_unencrypted(dev, phys); + else + dma_addr =3D phys_to_dma_encrypted(dev, phys); + + if (attrs & DMA_ATTR_MMIO) { if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; - } else { - dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || - dma_kmalloc_needs_bounce(dev, size, dir)) { - if (is_swiotlb_active(dev) && - !(attrs & DMA_ATTR_REQUIRE_COHERENT)) - return swiotlb_map(dev, phys, size, dir, attrs); + goto dma_mapped; + } =20 - goto err_overflow; - } + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { + if (is_swiotlb_active(dev) && + !(attrs & DMA_ATTR_REQUIRE_COHERENT)) + return swiotlb_map(dev, phys, size, dir, attrs); + goto err_overflow; } =20 +dma_mapped: if (!dev_is_dma_coherent(dev) && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A9C093E9F8D; Thu, 4 Jun 2026 08:41:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562514; cv=none; b=ZbO14pO77nnEcf1DUAklnnddUjEegB1H2eJvS5hkyN5Gt8TUk1J4hLPujzIJX1/KSLFIZy7gw3rTDql5RFXEo3nFkc+txuCrCX5nqD3mPItSKvuSzCzvJediC37ojbl5EMhATy8JKcMcKUT8mhf0yMz+OnDvSWyYuiBeTMXlSOU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562514; c=relaxed/simple; bh=crKlpKNpllKbKyJKWabyznvoxMX6Gz7vZyOFMfck1Yg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EsTV7XqFN7UAb8d0LxleDZAf3EXa7p8Ayp+RTrAGPfvzXyKtyDo3oHpVsCL3DmdTyganlnhfcALs0r7tCCn21V/wZtUeavS9tPdtDjbJ6bMsTe+Em02d6CJvMbGChSEp6yXLqKdUp1rtNVlLeBTnU9zTCEtFnHcqPSX/lka82YU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gaePA6lJ; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gaePA6lJ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 604A91F0089B; Thu, 4 Jun 2026 08:41:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562506; bh=iwy17gOiLcUnZL3dPCs29Ky8/+57kmk9bpomHZfp9xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=gaePA6lJZzg8wblozCtFlJ89jmAQkxm8vDSyCSgNwKS+Z+IgjmpztM6ihYK9J/IpW 2Aqn6VLDs/f0qSLgH2fcA19sL5QDvKhEMoeDRt065XT6+HUc/lKKbavrVhFXr+F/pl xtTXgNC3B1kNTaTNqqs+1IkjpD4+XMKAaK7xmg5Xdujfy2iDC49ICt4cW3NzeMJGgt E4DXiRLy+rnXuMIiNJORZ8RbVXTG1Gm+DCO65fBGbgMxB1gY8ixWVx/RRmKXRA3Q+Y RJKOOsHiJm7pjBdCRo3TI8oyizE7Vg4+q9UHsAmSAtCy64aWnBla25GJVEYWTHv1Qp d9HEGrWJjIlag== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 10/20] dma-direct: set decrypted flag for remapped DMA allocations Date: Thu, 4 Jun 2026 14:09:49 +0530 Message-ID: <20260604083959.1265923-11-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Devices that are DMA non-coherent and require a remap were skipping dma_set_decrypted(), leaving DMA buffers encrypted even when the device requires unencrypted access. Move the call after the if (remap) branch so that both the direct and remapped allocation paths correctly mark the allocation as decrypted (or fail cleanly) before use. Fix dma_direct_alloc() and dma_direct_free() to apply set_memory_*() to the linear-map alias of the backing pages instead of the remapped CPU address. Also disallow highmem pages for DMA_ATTR_CC_SHARED, because highmem buffers do not provide a usable linear-map address. Fixes: f3c962226dbe ("dma-direct: clean up the remapping checks in dma_dire= ct_alloc") Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 6d0ce3cfd8cc..9ce4fff6c112 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -196,6 +196,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, { bool remap =3D false, set_uncached =3D false; bool mark_mem_decrypt =3D false; + bool allow_highmem =3D true; struct page *page; void *ret; =20 @@ -214,6 +215,15 @@ void *dma_direct_alloc(struct device *dev, size_t size, mark_mem_decrypt =3D true; } =20 + if (attrs & DMA_ATTR_CC_SHARED) + /* + * Unencrypted/shared DMA requires a linear-mapped buffer + * address to look up the PFN and set architecture-required PFN + * attributes. This is not possible with HighMem. Avoid HighMem + * allocation. + */ + allow_highmem =3D false; + size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) gfp |=3D __GFP_NOWARN; @@ -272,7 +282,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, } =20 /* we always manually zero the memory once we are done */ - page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); + page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, allow_hig= hmem); if (!page) return NULL; =20 @@ -287,6 +297,14 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached =3D false; } =20 + if (mark_mem_decrypt) { + void *lm_addr; + + lm_addr =3D page_address(page); + if (set_memory_decrypted((unsigned long)lm_addr, PFN_UP(size))) + goto out_leak_pages; + } + if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 @@ -297,29 +315,36 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, ret =3D dma_common_contiguous_remap(page, size, prot, __builtin_return_address(0)); if (!ret) - goto out_free_pages; + goto out_encrypt_pages; } else { ret =3D page_address(page); - if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) - goto out_leak_pages; } =20 memset(ret, 0, size); =20 if (set_uncached) { + void *uncached_cpu_addr; + arch_dma_prep_coherent(page, size); - ret =3D arch_dma_set_uncached(ret, size); - if (IS_ERR(ret)) - goto out_encrypt_pages; + uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + if (IS_ERR(uncached_cpu_addr)) + goto out_free_remap_pages; + ret =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return ret; =20 + +out_free_remap_pages: + if (remap) + dma_common_free_remap(ret, size); + out_encrypt_pages: - if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) - return NULL; -out_free_pages: + if (mark_mem_decrypt && + dma_set_encrypted(dev, page_address(page), size)) + goto out_leak_pages; + if (!swiotlb_free(dev, page, size)) dma_free_contiguous(dev, page, size); return NULL; @@ -384,8 +409,16 @@ void dma_direct_free(struct device *dev, size_t size, } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) + } + + if (mark_mem_encrypted) { + void *lm_addr; + + lm_addr =3D phys_to_virt(phys); + if (set_memory_encrypted((unsigned long)lm_addr, PFN_UP(size))) { + pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); return; + } } =20 if (swiotlb_pool) --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 703053CEBB0; Thu, 4 Jun 2026 08:41:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562525; cv=none; b=kG3r5ypKjD68W7sGwFl69iO3GixEjn2McIns5fwg61v7hX5+Cyhid+58OsYQ+cCmbw7nKyL6QmJaNC8ST8h8aBXea8wvwflSQHebDUTSZJNwrp5iDOJ9ZIBCT1PxqrIc4TOGUOv63LM8d5PdNORnkRvNpiO6HkdGZqOXZD+sFJ0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562525; c=relaxed/simple; bh=a79uAugO8jCrNGo5kjYnBji5yN0tvrHm/LgknICs6vM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LKMDI4fZGgnXbF4OZggPle0YqXvhMi0P7X1OXvteov4ldePXjxa6Z95t1uO12UQdo4vDSkUnzGqv9Y8FxITjiJVuLeB6PygBkxZmLVdR/xsVtIgzstpQ7Q/F6VO6VvAg/KwK5Mlovkwrz0IRZVjyJPxh5v6OQiPm0Xd1+xYe20w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RaYvx39T; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RaYvx39T" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0C661F00899; Thu, 4 Jun 2026 08:41:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562515; bh=VVvebO+W7fx9D2IfL+crQhesHqFusXbqI+VwgiWtJ48=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=RaYvx39TjVmbeoSK/kZjnLcTw6JXTrnumJD1Sy0TdQouE5NGrkNQ6cXppAC+lbyAv WupoLrOqwBPQTXuAWdLS4gUGo1ZGYNVF8A8flV4k16rk5xcLL+FABacY3w9cgCbG6e MpnimWXs7lnhMPWsNvD+QpsvKH0to7KO2ZczUH1oHx0fIbPfsqVXVL75tK34qBYPUZ AxyUXLcFsdJGQ+yH27c45lZcPHcv0fhYhzDQqe37MZE3V6hfvd2Ns1HPmy5FHfSRgM fctF0rPauJx076NvtSD6hV9ev8RJA7Kq09f4evhXptl7oC5XwBG2ynLhdJ2vELiC3Y vjgK8T8jzXFGg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Jiri Pirko , Michael Kelley Subject: [PATCH v6 11/20] dma-direct: select DMA address encoding from DMA_ATTR_CC_SHARED Date: Thu, 4 Jun 2026 14:09:50 +0530 Message-ID: <20260604083959.1265923-12-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make the dma-direct helpers derive the DMA address encoding from DMA_ATTR_CC_SHARED instead of implicitly relying on force_dma_unencrypted() inside phys_to_dma_direct() Pass an explicit unencrypted/decrypted state into phys_to_dma_direct(), make the alloc paths return DMA addresses that match the requested buffer encryption state. Also only call dma_set_decrypted() when DMA_ATTR_CC_SHARED is actually set. Tested-by: Jiri Pirko Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 42 +++++++++++++++++++++++++----------------- 1 file changed, 25 insertions(+), 17 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 9ce4fff6c112..aa3489aa10a0 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -24,11 +24,11 @@ u64 zone_dma_limit __ro_after_init =3D DMA_BIT_MASK(24); =20 static inline dma_addr_t phys_to_dma_direct(struct device *dev, - phys_addr_t phys) + phys_addr_t phys, bool unencrypted) { - if (force_dma_unencrypted(dev)) + if (unencrypted) return phys_to_dma_unencrypted(dev, phys); - return phys_to_dma(dev, phys); + return phys_to_dma_encrypted(dev, phys); } =20 static inline struct page *dma_direct_to_page(struct device *dev, @@ -39,8 +39,9 @@ static inline struct page *dma_direct_to_page(struct devi= ce *dev, =20 u64 dma_direct_get_required_mask(struct device *dev) { + bool require_decrypted =3D force_dma_unencrypted(dev); phys_addr_t phys =3D ((phys_addr_t)max_pfn << PAGE_SHIFT) - 1; - u64 max_dma =3D phys_to_dma_direct(dev, phys); + u64 max_dma =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 return (1ULL << (fls64(max_dma) - 1)) * 2 - 1; } @@ -69,7 +70,8 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *d= ev, u64 *phys_limit) =20 bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) { - dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys); + bool require_decrypted =3D force_dma_unencrypted(dev); + dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 if (dma_addr =3D=3D DMA_MAPPING_ERROR) return false; @@ -79,17 +81,18 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t ph= ys, size_t size) =20 static int dma_set_decrypted(struct device *dev, void *vaddr, size_t size) { - if (!force_dma_unencrypted(dev)) - return 0; - return set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + int ret; + + ret =3D set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + if (ret) + pr_warn_ratelimited("leaking DMA memory that can't be decrypted\n"); + return ret; } =20 static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) { int ret; =20 - if (!force_dma_unencrypted(dev)) - return 0; ret =3D set_memory_encrypted((unsigned long)vaddr, PFN_UP(size)); if (ret) pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); @@ -169,7 +172,8 @@ static void *dma_direct_alloc_from_pool(struct device *= dev, size_t size, dma_coherent_ok); if (!page) return NULL; - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return ret; } =20 @@ -185,9 +189,11 @@ static void *dma_direct_alloc_no_mapping(struct device= *dev, size_t size, /* remove any dirty cache lines on the kernel alias */ if (!PageHighMem(page)) arch_dma_prep_coherent(page, size); - - /* return the page pointer as the opaque cookie */ - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + /* + * return the page pointer as the opaque cookie. + * Never used for unencrypted allocation + */ + *dma_handle =3D phys_to_dma_encrypted(dev, page_to_phys(page)); return page; } =20 @@ -332,7 +338,8 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =3D uncached_cpu_addr; } =20 - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return ret; =20 =20 @@ -455,11 +462,12 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, return NULL; =20 ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, ret, size)) goto out_leak_pages; setup_page: memset(ret, 0, size); - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return page; out_leak_pages: return NULL; --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09C893FB7FA; Thu, 4 Jun 2026 08:42:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562532; cv=none; b=BC3BwLln/JbZrmLqwE2xm6dLcgiU5ccgug8ynksjfcxTGAml2Q7rGwJjifRzHhDYQvYnE1d/3ntWjCuk9I09knGe6sdXwS0THZsjauxvYLua15Qf1f5A5jVxbcrYsn+umssD7IiCmxTseLAzDfPPDUw6RLgO9dLa4r9g2ZBepqY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562532; c=relaxed/simple; bh=42pKzQV5QRjKs8guM0b0nbr1bq1Ft61lzzlCLkX5RsE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FJuNnvBOYYUAmbhoHWJnAu14n2VcNh/wrRtyBgSHze9fEyKdxlKIg0UBXOAS+r9B3f5zU1QDXqbQAADbdzVP0Vj1Zi1+503Vjm6Z5LdtKRbPAEVslOphbpZMC5AiTclOeGGNyq1j9xI1NQqXSdNUX/K22SA6M6VmfEtwXIvWsgU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PR2wINvc; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PR2wINvc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DF9321F00893; Thu, 4 Jun 2026 08:41:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562524; bh=OmMSb/sy1rEjM9SAjzcclksqW5ZMYzL/SObxAZ/44zQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=PR2wINvcuAOCRRLqDiFxasw3MjrjE8q1zZJKgb6GlACxxyI7AggfIZ1AUHukout58 WFjBXmrh6RwbkLBn6pSMkJEaQopuR45jlhfYS3oahFNCnzXAqapiJ393CO2ZDBFk4E VYBiXWP1I0u8vF03YGx5un0/AFc9L1A14lgEAy0GOt2Fd29lB7wucGQQU8IUjtRlm5 wFcSHwaELzMOYASOyvnArsQplSlvlQjzzbjusg4Ltyj1pPMnwYWaQ6yqYbTDLXPI0w k0OfdwRk+Aqa3jCdEnMpKEHGvUB6CZX/l7uFNqMZxzFd477Q8eWUY7nQQFufeSVeL1 WX93aDw8L+vRQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 12/20] dma-pool: fix page leak in atomic_pool_expand() cleanup Date: Thu, 4 Jun 2026 14:09:51 +0530 Message-ID: <20260604083959.1265923-13-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" atomic_pool_expand() frees the allocated pages from the remove_mapping error path only when CONFIG_DMA_DIRECT_REMAP is enabled. When CONFIG_DMA_DIRECT_REMAP is disabled, failures after page allocation, such as gen_pool_add_virt(), jump to remove_mapping and return without freeing the pages. Move __free_pages(page, order) out of the CONFIG_DMA_DIRECT_REMAP block so that cleanup paths always release the allocation. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index be78474a6c49..e7df8d279e75 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -146,9 +146,9 @@ static int atomic_pool_expand(struct dma_gen_pool *dma_= pool, size_t pool_size, #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); free_page: +#endif if (!leak_pages) __free_pages(page, order); -#endif out: return ret; } --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5C8A3E9F60; Thu, 4 Jun 2026 08:42:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562540; cv=none; b=ntPGLl4mA4skAbAGUR0Qic2kdFj21wTzKru9FBZ1J/qGO9rkEXwYKryTvXFQyAP/pcLPHgEdUYuEIv5j5vHQHAOsfbnVaZUrZkz4uAy7XnPFI+W6ybOcJyWq34DKZKYeFHrUGmNnrszNTBPPZIPKrNPdRtFGUfUqyICDeHpB+QM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562540; c=relaxed/simple; bh=Hs3J2Seso9OliD0BIWjaglJJsqy9xLncdXwariektdM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=T//mBXZbcF/+cClKHU8Rjfe6G2O2osg4bI7XJkbsFE4p/EAojIONpEdpiUL+iD/rSdiIHEQYUMUFKC1QvqdvjKd/2zkKc+E5RzxYA4QiIeCsUnahrKbgdeiTHjwsUFsl98DPmXqfQayyPIhfQZAVWct59xJIWnatlHp4MB+97mI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JMg19boI; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JMg19boI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E50521F00898; Thu, 4 Jun 2026 08:42:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562533; bh=zZ74WE3NPJcGhukL6BHrevm7m6Mzo9c9NrhYN47Aay0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JMg19boI0cLz/7ejQT9QzBE9Im2xv53R2G16c5U/p5BuEcJSqYgf0U1zEliZFa6+C eNRR0hp91nfODNozHPMR4LbGZeSLnEpft46UimZUCkmRL8+9aD/b9Rp86uXEYDzPyD re+5opN/jBK5GhvJF4YckHXOJaU6pof4ksDfUDeLQTpyPbWxgnqEpYAI/aJiiEbFCI p+KC+RS+ukCf2WuHbJ5mw/tI0sFilqtSgqP9f2GMe80M2kjNQ7uYXY+ko8szGo/nJB PZThVarr8r6zhCgVXB0ZRCgiJyR0HJquVz/o66+yDuenfY9z7ekPE1wHRZ5TesHSwm NVWsugctx9cfA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 13/20] dma-direct: rename ret to cpu_addr in alloc helpers Date: Thu, 4 Jun 2026 14:09:52 +0530 Message-ID: <20260604083959.1265923-14-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ret in dma_direct_alloc() and dma_direct_alloc_pages() holds the returned CPU mapping, not a generic return value. Rename it to cpu_addr and update the remaining uses to match. This makes the allocation paths easier to follow and keeps the local naming consistent with what the variable actually represents. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index aa3489aa10a0..4e446aa4130e 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -204,7 +204,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, bool mark_mem_decrypt =3D false; bool allow_highmem =3D true; struct page *page; - void *ret; + void *cpu_addr; =20 /* * DMA_ATTR_CC_SHARED is not a caller-visible dma_alloc_*() @@ -318,34 +318,33 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, arch_dma_prep_coherent(page, size); =20 /* create a coherent mapping */ - ret =3D dma_common_contiguous_remap(page, size, prot, - __builtin_return_address(0)); - if (!ret) + cpu_addr =3D dma_common_contiguous_remap(page, size, prot, + __builtin_return_address(0)); + if (!cpu_addr) goto out_encrypt_pages; } else { - ret =3D page_address(page); + cpu_addr =3D page_address(page); } =20 - memset(ret, 0, size); + memset(cpu_addr, 0, size); =20 if (set_uncached) { void *uncached_cpu_addr; =20 arch_dma_prep_coherent(page, size); - uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + uncached_cpu_addr =3D arch_dma_set_uncached(cpu_addr, size); if (IS_ERR(uncached_cpu_addr)) goto out_free_remap_pages; - ret =3D uncached_cpu_addr; + cpu_addr =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); - return ret; - + return cpu_addr; =20 out_free_remap_pages: if (remap) - dma_common_free_remap(ret, size); + dma_common_free_remap(cpu_addr, size); =20 out_encrypt_pages: if (mark_mem_decrypt && @@ -439,7 +438,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, { unsigned long attrs =3D 0; struct page *page; - void *ret; + void *cpu_addr; =20 if (force_dma_unencrypted(dev)) attrs |=3D DMA_ATTR_CC_SHARED; @@ -453,7 +452,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, if (!page) return NULL; =20 - ret =3D page_address(page); + cpu_addr =3D page_address(page); goto setup_page; } =20 @@ -461,11 +460,11 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, if (!page) return NULL; =20 - ret =3D page_address(page); - if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, ret, size)) + cpu_addr =3D page_address(page); + if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, cpu_addr, size= )) goto out_leak_pages; setup_page: - memset(ret, 0, size); + memset(cpu_addr, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); return page; --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 481033D6CB5; Thu, 4 Jun 2026 08:42:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562546; cv=none; b=NJc/FBmFyHoMt1XD+uvv2Iv5DDXD+JYI+HZxDewAJpo7XOPBXi9R96aMZJP7KPsvdimIGLgJPDIBZ4wtry5QvHQYs7rGKg3gW2uauD39QO97WfXtgBeFkuP93IQZqZ95MbNJhyek+WfvttlEXCdSzGsXjX+NMMZoH4ZDUUbFiJ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562546; c=relaxed/simple; bh=rcg0Q+6+2FAtZMgWgfjQda2Drx/FcFQqdoAZCTdZ9wI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cs4qnHGVVCce5qPlQNe5gZqtI9GiXnAWJC4ZYOBy+vfpVdQQGaDP3tLnWXYHhmm3/ZEPBmpo9M2Ds+/JyTQmraHyaXQH2opYK1pAt+rXr9WZ7ziA+TLhVp0Kkq0nPCQSYNSUG2uL2lf01WKhjcZbBuWUFnXv16zEMK9AKYWF/Ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Yn5AXRgp; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Yn5AXRgp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E9C031F0089A; Thu, 4 Jun 2026 08:42:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562542; bh=na5iA7JJF5EbHB1DYhHq0zpzWxfHieUed4f5owflzmA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Yn5AXRgpAXhRhafyxaaxRkC67nV6/1y/+hBVV4GhrE+hn6mZ914QVIlT0cp2sEANU 31krwTQNaQEGArg+jE8pLIZkcV3lRKxHmCqg4rcunxn9SFCQnW57tpDUoTy0SNTkgu ug+9vIb5NLvIr86ZUT042mohsW/DLHJv+3rBczDVcJUdLS7DDks0QbxpRqZE4ibDtC HMKeGTAd4ndhTvuwabYB5tD3VDr6Dks37RULt/QS8FLqWMEA94m+73lteArVYW9JX2 DrxMbFoSPumqXO6By77oyJZRIcyY7l+Csbe6fG6Z2+SORBo3XY6fEaYwqzhXw5lzKR xdLBKCIH+4CSQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, stable@vger.kernel.org, Michael Kelley Subject: [PATCH v6 14/20] dma-direct: return struct page from dma_direct_alloc_from_pool() Date: Thu, 4 Jun 2026 14:09:53 +0530 Message-ID: <20260604083959.1265923-15-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool helper") changed dma_direct_alloc_from_pool() to return the CPU address from dma_alloc_from_pool(). That fits dma_direct_alloc(), but dma_direct_alloc_pages() also uses the helper and expects a struct page *. Fix this by making dma_direct_alloc_from_pool() return the struct page * again, and pass the CPU address back through an out-parameter for the dma_direct_alloc() caller. Fixes: 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool h= elper") Cc: stable@vger.kernel.org Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/direct.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 4e446aa4130e..e0ab9ff3f1d6 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -157,24 +157,24 @@ static bool dma_direct_use_pool(struct device *dev, g= fp_t gfp) return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev); } =20 -static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) +static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t = size, + dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp, + unsigned long attrs) { struct page *page; u64 phys_limit; - void *ret; =20 if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL))) return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, &ret, gfp, attrs, + page =3D dma_alloc_from_pool(dev, size, cpu_addr, gfp, attrs, dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); - return ret; + return page; } =20 static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size, @@ -270,9 +270,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, * the atomic pools instead if we aren't allowed block. */ if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && - dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, - gfp, attrs); + dma_direct_use_pool(dev, gfp)) { + page =3D dma_direct_alloc_from_pool(dev, size, + dma_handle, &cpu_addr, + gfp, attrs); + return page ? cpu_addr : NULL; + } =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size, attrs); @@ -445,7 +448,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, =20 if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, - gfp, attrs); + &cpu_addr, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size, attrs); --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1D963F0AA8; Thu, 4 Jun 2026 08:42:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562554; cv=none; b=WwIVc5y/NCKVfSUmytd1ErDgvbj/pHUkvgs37Uvk8NXDhNhqZ6oLiUkIEG/EacnZtLXIzD6vY5K0PLbFpKWV0EbjXBhbe2m/dBdOzNlwmPQOkXMHtTG8GH7sBqePI9hiQLQjczYWuSg4fMd6XjklaGT6myS2G8eJOrdjJxU3k90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562554; c=relaxed/simple; bh=iczgo5wSbem2q/StVcpJPf4dCmkL4x7I0g8WRdVDUFQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h3qU2iKz0iV39dx26lN7zaBFNPRwpbb1OgfhKdm4Dm08oT6I98WfhkOG0Sur/Mp0A09fKxylyMWNJuCX8xZ5ODm5txUzNbaOVnOZzB/acxcLw/kDOJZIghEMGdV0olDVdj/RMdzyieRpGIo7hA0VjHp8G/Pp7igreWo9p4pUXnI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IVwvBUGl; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IVwvBUGl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 361A01F00898; Thu, 4 Jun 2026 08:42:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562551; bh=8RqOZoZWEO6SeOB2Dx6VDRiy1MZJt2V2zJgzbuqAzxg=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=IVwvBUGl4YDJfsvKSMOpRRLu9z169sOpVXGOlzsJjuX3Lja/6J7Ym0aDhuyJpk9q/ Pwj8t+ZfHxry2X9UyTDAOecWvmygXTI7WYYPOORCoCIK6usXlD5ajDVD50sd+t2g+G ykNn9OVAw+Cb+87vzbKgAL4tCIA3er6EHBms9OADHr5V7I0Uzij2bpY6SVxY4xK5sn wnQwdH2eTCJpj0cdObPeW8Be7wcAxXJPicFwBeSmnY2tlfE7KhDJZ1ot5WB6r/UdHg UttmA0BLx44bicDZJj4IMIg9+YfMzqXZvrAYKDqaSKYLTTC9sW3wT1O46HhnUxY/im PKU7uTrrkRrKQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 15/20] iommu/dma: Check atomic pool allocation result directly Date: Thu, 4 Jun 2026 14:09:54 +0530 Message-ID: <20260604083959.1265923-16-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The non-blocking, non-coherent allocation path uses dma_alloc_from_pool(), which returns the allocated page and fills cpu_addr only on success. Do not rely on cpu_addr to detect allocation failure in this path. Check the returned page directly before using it for the IOMMU mapping. Fixes: 9420139f516d ("dma-pool: fix coherent pool allocations for IOMMU map= pings") Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- drivers/iommu/dma-iommu.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 725c7adb0a8d..52c599f4472c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1671,13 +1671,16 @@ void *iommu_dma_alloc(struct device *dev, size_t si= ze, dma_addr_t *handle, } =20 if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && - !gfpflags_allow_blocking(gfp) && !coherent) + !gfpflags_allow_blocking(gfp) && !coherent) { page =3D dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, gfp, attrs, NULL); - else + if (!page) + return NULL; + } else { cpu_addr =3D iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); - if (!cpu_addr) - return NULL; + if (!cpu_addr) + return NULL; + } =20 *handle =3D __iommu_dma_map(dev, page_to_phys(page), size, ioprot, dev->coherent_dma_mask); --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 722373F9F39; Thu, 4 Jun 2026 08:42:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562564; cv=none; b=H3uMAkRkuzY3P8uwRblpuH4Zi6Mh90Z6pJF+GdjlHbNnLAWLFMbmjZg+zwsc8kfp2A1Q/PBB/zdimWu+dRYF78De6hE1k6XJrn5aCJidhdzsgIR8+/43ly0xApMYIybatcm+lc71TI6yOn6SM7ppndvHG791GWPg5dbaDNRe8p4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562564; c=relaxed/simple; bh=eVdmJY+o9XiwPJc3bN3dXicUHefCCr6TR47swKmpRAA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XNXtwS48WI5ZuKyjbcakZI8jzLjqSTFTQp+bMhiUSt4+hXPe6RFE0F0Uw6k9SfDCYxixYMjfGtDIUCLQmxkKAiENyGQ8SD+/KuJ2XApXg7fEawYKthrVCvu+eWoI3n8tJ3rFrXrLByqiLfTa7wcJ2w6BwmUr8rEhZeKGHAyNibM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QN+wRW6W; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QN+wRW6W" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 38C371F00899; Thu, 4 Jun 2026 08:42:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562560; bh=YDcUplBTtF+vjaqPzZkpVvKrTSP2VS59GGyhIg5eOfI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=QN+wRW6W1coJMn2xiTdq9mWM6rsXHpmvncy0qsPD4rdSpGgX+NENrmQ72N3SfGQbk 0kg8ZSMlQtYHIoCtmrnJf+HuQ3RHHKyocPGAyQKdsvLEt+y2mO44UEZ9SX9jnyumoL b+TS/30qXbbX0Oa1eJh56EyW3Xd2LSXb1fqr32U96RSITU6S9J8zJqhdqiZvkp8n2Z 56PVxxuPpMRxyhsK7Al3YUaVC9LOIEDjQZWTHFO9NoqZMWTokUNouWJC8wC364rAp3 tq7FtMnOS5RBdjSImDCUYUGabL6F495efTYC7iyDranN2c1vzkWYZVoOoGq3r/QqYx hiyQkOBe6hT6w== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 16/20] dma: swiotlb: free dynamic pools from process context Date: Thu, 4 Jun 2026 14:09:55 +0530 Message-ID: <20260604083959.1265923-17-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swiotlb_dyn_free() is used after removing a dynamic swiotlb pool from RCU-protected lists. It can call swiotlb_free_tlb(), which may need to restore the encryption state of an unencrypted pool with set_memory_encrypted() before freeing the pages. RCU callbacks run in atomic context, but set_memory_encrypted() is not guaranteed to be atomic-safe on all architectures. For example, page attribute updates may allocate page tables or take sleeping locks. Use queue_rcu_work() for dynamic pool freeing instead. This keeps the RCU grace period before freeing a published pool, while running the actual pool teardown from workqueue context. Use the same helper for the transient-pool error path, since that path may also be reached from atomic DMA mapping context. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 4 ++-- kernel/dma/swiotlb.c | 19 +++++++++++-------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 4dcbf3931be1..526f82e9da45 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -64,7 +64,7 @@ extern void __init swiotlb_update_mem_attributes(void); * @areas: Array of memory area descriptors. * @slots: Array of slot descriptors. * @node: Member of the IO TLB memory pool list. - * @rcu: RCU head for swiotlb_dyn_free(). + * @dyn_free: RCU work item used to free the pool from process context. * @transient: %true if transient memory pool. */ struct io_tlb_pool { @@ -79,7 +79,7 @@ struct io_tlb_pool { struct io_tlb_slot *slots; #ifdef CONFIG_SWIOTLB_DYNAMIC struct list_head node; - struct rcu_head rcu; + struct rcu_work dyn_free; bool transient; bool unencrypted; #endif diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index f4e8b241a1c4..4c56f64602ea 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -774,13 +774,10 @@ static void swiotlb_dyn_alloc(struct work_struct *wor= k) add_mem_pool(mem, pool); } =20 -/** - * swiotlb_dyn_free() - RCU callback to free a memory pool - * @rcu: RCU head in the corresponding struct io_tlb_pool. - */ -static void swiotlb_dyn_free(struct rcu_head *rcu) +static void swiotlb_dyn_free_work(struct work_struct *work) { - struct io_tlb_pool *pool =3D container_of(rcu, struct io_tlb_pool, rcu); + struct io_tlb_pool *pool =3D + container_of(to_rcu_work(work), struct io_tlb_pool, dyn_free); size_t slots_size =3D array_size(sizeof(*pool->slots), pool->nslabs); size_t tlb_size =3D pool->end - pool->start; =20 @@ -789,6 +786,12 @@ static void swiotlb_dyn_free(struct rcu_head *rcu) kfree(pool); } =20 +static void swiotlb_schedule_dyn_free(struct io_tlb_pool *pool) +{ + INIT_RCU_WORK(&pool->dyn_free, swiotlb_dyn_free_work); + queue_rcu_work(system_wq, &pool->dyn_free); +} + /** * __swiotlb_find_pool() - find the IO TLB pool for a physical address * @dev: Device which has mapped the DMA buffer. @@ -835,7 +838,7 @@ static void swiotlb_del_pool(struct device *dev, struct= io_tlb_pool *pool) list_del_rcu(&pool->node); spin_unlock_irqrestore(&dev->dma_io_tlb_lock, flags); =20 - call_rcu(&pool->rcu, swiotlb_dyn_free); + swiotlb_schedule_dyn_free(pool); } =20 #endif /* CONFIG_SWIOTLB_DYNAMIC */ @@ -1276,7 +1279,7 @@ static int swiotlb_find_slots(struct device *dev, phy= s_addr_t orig_addr, index =3D swiotlb_search_pool_area(dev, pool, 0, orig_addr, tbl_dma_addr, alloc_size, alloc_align_mask); if (index < 0) { - swiotlb_dyn_free(&pool->rcu); + swiotlb_schedule_dyn_free(pool); return -1; } =20 --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 342DB3F5BDC; Thu, 4 Jun 2026 08:42:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562572; cv=none; b=RS84TNc5nBZaDHQZlJMCbrozAzJqhi/Q+sxWQz/b2BaRlFMy87mBZgSrix839237jiRtZgHPSCo2VqKDrvaPrTL1xkdUgXgo0NKy5BHvogJ5JzLOIOlYr1Vtds3jfvx2ic90uFSRmaAHwqyjiI6ZU6oOgumOU6JFO6sfCMaN3OY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562572; c=relaxed/simple; bh=yHRr1lp15U7RGZX8O931NyqoDVO5pUyfmQrFGuGSZKY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nVzU9b1Tq4QUE+ncQ0TXynZ/LpK+sjbptmLi0zhfbt6DjSppb6JUXwe2aytKrgS38UTnxfXnD5NksJ9P0AFU2DYMkmrb+rW0oOfhkeuw0juZOchWHWK7BmddfVN4iH01SKXCTit28hx6mvddoiMs/Jy4Tkdnr7psG5H3SHiYJ1I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JMeQpZtx; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JMeQpZtx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3DE9C1F00893; Thu, 4 Jun 2026 08:42:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562569; bh=gIe+3PDvKixXDpYuwkSkDBkDRMcsfcae67VvbDIqkSI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=JMeQpZtxD/t4LMxT6bBk+abdMJwCuKqMUOI6LvMDUxWqbk+0W2lPCQfnGRvNn3zEQ TnhynEd0A/26RsZBz5Wi/aXFtvcyKim0I0BgjVd41z0HT6Q6gyfgRtfAnix5YH85wK MptdDvq+fGlCB1kxIMuGgqi+Fp1PJdHzxWsb4W4zztFWd8e46Oul8/Y4fketumkxtU TIVLEkVW+udRm4S6dxNrYVlOtARRFNIt+IE9hJR9xUin0J+piQQaLPZmmukIendJMh Il0qYnbhukAyR37QM4G67V3ADzyJXlSyMEOL16DX6Y3vkmLnThquWduHSfZsVbFft2 /5M5gp/Mvw0lA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 17/20] dma: swiotlb: handle set_memory_decrypted() failures Date: Thu, 4 Jun 2026 14:09:56 +0530 Message-ID: <20260604083959.1265923-18-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Check the return value when converting swiotlb pools between encrypted and decrypted mappings. If the default pool cannot be decrypted after early initialization, mark the pool fully used so it cannot satisfy future bounce allocations. For late initialization, return the `set_memory_decrypted()` failure. For restricted DMA pools, fail device initialization if the reserved pool cannot be decrypted. This prevents swiotlb from using pools whose encryption attributes do not match their metadata, and avoids returning pages with uncertain encryption state back to the allocator. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/swiotlb.c | 80 +++++++++++++++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 15 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 4c56f64602ea..14d834ca298b 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -248,6 +248,23 @@ static inline unsigned long nr_slots(u64 val) return DIV_ROUND_UP(val, IO_TLB_SIZE); } =20 +static void swiotlb_mark_pool_used(struct io_tlb_pool *pool) +{ + unsigned long i; + + for (i =3D 0; i < pool->nareas; i++) { + pool->areas[i].index =3D 0; + pool->areas[i].used =3D pool->area_nslabs; + } + + for (i =3D 0; i < pool->nslabs; i++) { + pool->slots[i].list =3D 0; + pool->slots[i].orig_addr =3D INVALID_PHYS_ADDR; + pool->slots[i].alloc_size =3D 0; + pool->slots[i].pad_slots =3D 0; + } +} + /* * Early SWIOTLB allocation may be too early to allow an architecture to * perform the desired operations. This function allows the architecture = to @@ -272,8 +289,16 @@ void __init swiotlb_update_mem_attributes(void) return; bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); =20 - if (io_tlb_default_mem.unencrypted) - set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) { + int ret; + + ret =3D set_memory_decrypted((unsigned long)mem->vaddr, + bytes >> PAGE_SHIFT); + if (ret) { + pr_warn("Failed to decrypt default memory pool, disabling it\n"); + swiotlb_mark_pool_used(mem); + } + } } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, @@ -442,9 +467,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, { struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; unsigned long nslabs =3D ALIGN(size >> IO_TLB_SHIFT, IO_TLB_SEGSIZE); + unsigned int order, area_order, slot_order; + bool leak_pages =3D false; unsigned int nareas; unsigned char *vstart =3D NULL; - unsigned int order, area_order; bool retried =3D false; int rc =3D 0; =20 @@ -504,6 +530,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, (PAGE_SIZE << order) >> 20); } =20 + rc =3D -ENOMEM; nareas =3D limit_nareas(default_nareas, nslabs); area_order =3D get_order(array_size(sizeof(*mem->areas), nareas)); mem->areas =3D (struct io_tlb_area *) @@ -511,14 +538,20 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, if (!mem->areas) goto error_area; =20 + slot_order =3D get_order(array_size(sizeof(*mem->slots), nslabs)); mem->slots =3D (void *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, - get_order(array_size(sizeof(*mem->slots), nslabs))); + slot_order); if (!mem->slots) goto error_slots; =20 - if (io_tlb_default_mem.unencrypted) - set_memory_decrypted((unsigned long)vstart, - (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) { + rc =3D set_memory_decrypted((unsigned long)vstart, + (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (rc) { + leak_pages =3D true; + goto error_decrypt; + } + } =20 swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, nareas); @@ -527,16 +560,20 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, swiotlb_print_info(); return 0; =20 +error_decrypt: + free_pages((unsigned long)mem->slots, slot_order); error_slots: free_pages((unsigned long)mem->areas, area_order); error_area: - free_pages((unsigned long)vstart, order); - return -ENOMEM; + if (!leak_pages) + free_pages((unsigned long)vstart, order); + return rc; } =20 void __init swiotlb_exit(void) { struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; + bool leak_pages =3D false; unsigned long tbl_vaddr; size_t tbl_size, slots_size; unsigned int area_order; @@ -552,19 +589,23 @@ void __init swiotlb_exit(void) tbl_size =3D PAGE_ALIGN(mem->end - mem->start); slots_size =3D PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs)); =20 - if (io_tlb_default_mem.unencrypted) - set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) { + if (set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT)) + leak_pages =3D true; + } =20 if (mem->late_alloc) { area_order =3D get_order(array_size(sizeof(*mem->areas), mem->nareas)); free_pages((unsigned long)mem->areas, area_order); - free_pages(tbl_vaddr, get_order(tbl_size)); + if (!leak_pages) + free_pages(tbl_vaddr, get_order(tbl_size)); free_pages((unsigned long)mem->slots, get_order(slots_size)); } else { memblock_free(mem->areas, array_size(sizeof(*mem->areas), mem->nareas)); - memblock_phys_free(mem->start, tbl_size); + if (!leak_pages) + memblock_phys_free(mem->start, tbl_size); memblock_free(mem->slots, slots_size); } =20 @@ -1938,9 +1979,18 @@ static int rmem_swiotlb_device_init(struct reserved_= mem *rmem, * restricted mem pool is decrypted by default */ if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + int ret; + mem->unencrypted =3D true; - set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), - rmem->size >> PAGE_SHIFT); + ret =3D set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), + rmem->size >> PAGE_SHIFT); + if (ret) { + dev_err(dev, "Failed to decrypt restricted DMA pool\n"); + kfree(pool->areas); + kfree(pool->slots); + kfree(mem); + return ret; + } } else { mem->unencrypted =3D false; } --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 08C033E7BC5; Thu, 4 Jun 2026 08:42:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562580; cv=none; b=pPHm3zIkQbhvawyJAw+YASV7kSoaopR4sShASeiMDnpHNN7+df6f56FvQDml86AR/Sc77GYsY3riZIU6fbgMqx9NkdzC7Rbhep20ZoabPgQ+3YsjyUFBTpPXO+w+rXQU8cnTiEVOZiW30eooU7bDEHKmCe8HKu2yptxjnAoIi1w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562580; c=relaxed/simple; bh=g/wtgfIecl6o8kwHeLkDvFzrvI6n8z3n5Cw9liWsHGo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Hv/k16on6i0zdDBOu3sJa6+IhIFCZv8oU4PsczDWICH/mI9AoxMytKuOql+eDywaeIgLKGeW2AXtGiFHKa6DjcOc0G4CnvokxmDL+UYS3+SZRLjkylnosIzeAhKwQMB4Yy43db5sy3g1k6LYpsNW2nBf4liddruF0hKWifhPVEc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kPwkUEOk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kPwkUEOk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 415181F00898; Thu, 4 Jun 2026 08:42:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562578; bh=5WeykkaLRY84/gMTdzzVPplQJSrHkGjIEUjp2FUb45E=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=kPwkUEOk1WgnrF0lOV0PcG8AQaiVhngfx/W0TVtjGwpDNk1Baj72OsUUlUxaxIdxt GCMc3uO9dZ0y+4DmEajexLfTCy0njgJdDKWFY9hNA6cHq5Mr0VzYb075IVKnz09ylZ mhURdS/1AaP9I7PUlOb/20/S5Qac8flUQ4Ku/7TmUPowi2LXf/qpYsGBY67dQRdlSr BJppQVtzWfkLsFwBGMpoLv4SBMYJXdg+a+LtnxbzH90+JUMUfoODtJ8bcesSA/7/Q7 ep7tfYzuBiwuLbY57SGVqmXpr1NYNfK8KsDo5ZI/oY1hIt+e9hbCAbrkFpKYImCFdg JlZGreIpvlfMA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 18/20] dma: free atomic pool pages by physical address Date: Thu, 4 Jun 2026 14:09:57 +0530 Message-ID: <20260604083959.1265923-19-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" dma_direct_alloc_pages() may satisfy atomic allocations from the coherent atomic pools. The pool allocation is keyed by the virtual address stored in the gen_pool, but the pages API returns only the backing struct page. On architectures with CONFIG_DMA_DIRECT_REMAP, atomic pool chunks are added to the gen_pool using their remapped virtual address. dma_direct_free_pages() reconstructs a linear-map address with page_address(page) and passes that to dma_free_from_pool(). That address does not match the gen_pool virtual range, so the pool lookup can fail and the code can fall through to freeing a pool-owned page through the normal page allocator path. Add a page-based pool free helper that looks up the owning pool chunk by physical address, translates it back to the gen_pool virtual address, and frees that address to the pool. Use it from dma_direct_free_pages() while keeping the existing virtual-address helper for coherent allocation frees. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/dma-map-ops.h | 1 + kernel/dma/direct.c | 4 +-- kernel/dma/pool.c | 54 +++++++++++++++++++++++++++++++++++++ 3 files changed, 57 insertions(+), 2 deletions(-) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 696b2c3a2305..8be059e69935 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -215,6 +215,7 @@ struct page *dma_alloc_from_pool(struct device *dev, si= ze_t size, void **cpu_addr, gfp_t flags, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)); bool dma_free_from_pool(struct device *dev, void *start, size_t size); +bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size); =20 int dma_direct_set_offset(struct device *dev, phys_addr_t cpu_start, dma_addr_t dma_start, u64 size); diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index e0ab9ff3f1d6..58f7ea1be963 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -488,9 +488,9 @@ void dma_direct_free_pages(struct device *dev, size_t s= ize, */ bool mark_mem_encrypted =3D force_dma_unencrypted(dev); =20 - /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ + /* If page is not from an atomic pool, dma_free_from_pool_page() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && - dma_free_from_pool(dev, vaddr, size)) + dma_free_from_pool_page(dev, page, size)) return; =20 phys =3D page_to_phys(page); diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index e7df8d279e75..43b8101d860f 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -356,3 +356,57 @@ bool dma_free_from_pool(struct device *dev, void *star= t, size_t size) =20 return false; } + +struct dma_pool_phys_match { + phys_addr_t phys; + size_t size; + unsigned long addr; + bool found; +}; + +static void dma_pool_find_phys(struct gen_pool *pool, struct gen_pool_chun= k *chunk, + void *data) +{ + struct dma_pool_phys_match *match =3D data; + phys_addr_t end =3D match->phys + match->size - 1; + phys_addr_t chunk_end; + + if (match->found) + return; + + chunk_end =3D chunk->phys_addr + (chunk->end_addr - chunk->start_addr); + if (match->phys < chunk->phys_addr || end > chunk_end) + return; + + match->addr =3D chunk->start_addr + (match->phys - chunk->phys_addr); + match->found =3D true; +} + +static bool dma_free_from_pool_phys(struct dma_gen_pool *dma_pool, phys_ad= dr_t phys, + size_t size) +{ + struct dma_pool_phys_match match =3D { + .phys =3D phys, + .size =3D size, + }; + + gen_pool_for_each_chunk(dma_pool->pool, dma_pool_find_phys, &match); + if (!match.found) + return false; + + gen_pool_free(dma_pool->pool, match.addr, size); + return true; +} + +bool dma_free_from_pool_page(struct device *dev, struct page *page, size_t= size) +{ + struct dma_gen_pool *dma_pool =3D NULL; + phys_addr_t phys =3D page_to_phys(page); + + while ((dma_pool =3D dma_guess_pool(dma_pool, 0))) { + if (dma_free_from_pool_phys(dma_pool, phys, size)) + return true; + } + + return false; +} --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 279D53E1224; Thu, 4 Jun 2026 08:43:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562589; cv=none; b=GkYm74GArQmJp01MI3GgWlSNUyyRqsEA3onQHV2TYlKiKL6hQt0aWDOKowTItNml6VZ3zKR4zjAcFMRoiTlBYFp77ere8cSb3eljOyRHn2v4yq7ZAa5B6FfW7klNC10UtuCQlTA2ld7/isj1ix5T119AMF7eGVCEyTWRGUydsFU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562589; c=relaxed/simple; bh=TegZPjM3rQ+HL/D+iEWYjVCfP3sBZRjsMLXnOkhdCHY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=O777Bcy3Ontk1Xj8ti+nlfzLvfhvKFzKFZl8NAu71souscijAsvw3QWlhE5lJOdSJSRz4bPpdbkB+1rBkULSA3y0c2Qckbm5pTdkRHjv5CVZVYK6kQJ79HiOlShZx41/qvmDE8+yvAARbaNRIbiEExUozyYFqecuxClhebN7RB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nB9gQ4aF; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nB9gQ4aF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44D721F00893; Thu, 4 Jun 2026 08:42:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562587; bh=28gAGxAT6yyFPOrZbknmeVeC1tPuyb3t0ezhz06WVCM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=nB9gQ4aFlKUpGUcihtsmI/cFkRNvSLQlBMVfFfId6za3U9oIZifeElqfvJ/gMMYLV kK9nLDYi/76JeTYI0eOMh6Ev87PICHvhy0d1ZKgpW4V7V0HhFFF5pltK8QUwBgbaaU AfTbyxMYiMbryQfwJ65A2G2jc3RkvI/TeFKscmT4OobGaUuPuVSaPsf+Y9qMoAzQ+w qQShsxWWYTx1w+YWqV2DLGX6UxvSmK9OjRoCDsr0peBDdamrU4PWYXtZy55sxBwaSo WpS//rd8RA5c1c8l+zl23/OHY/f2k3XfRfr004sW3x9c754aQDpKx39tz9vKXx6uZO kzFmq6SY4lRqw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, Michael Kelley Subject: [PATCH v6 19/20] swiotlb: Preserve allocation virtual address for dynamic pools Date: Thu, 4 Jun 2026 14:09:58 +0530 Message-ID: <20260604083959.1265923-20-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" swiotlb_alloc_tlb() can allocate from the DMA atomic pool when a decrypted pool is needed from atomic context. With CONFIG_DMA_DIRECT_REMAP, the atomic pool is backed by remapped virtual addresses, which are not the same as the direct-map addresses returned by phys_to_virt(). swiotlb_init_io_tlb_pool() currently reconstructs the pool virtual address from the physical start address. For atomic-pool backed allocations this stores the wrong address in pool->vaddr. Later, swiotlb_free_tlb() passes that address to dma_free_from_pool(), which will fail to recognize the chunk Pass the virtual address returned by the allocation path into swiotlb_init_io_tlb_pool(), and store that address in pool->vaddr. This keeps the pool free path using the same virtual address as the allocator. Tested-by: Michael Kelley Tested-by: Mostafa Saleh Signed-off-by: Aneesh Kumar K.V (Arm) --- kernel/dma/swiotlb.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 14d834ca298b..e4bd8c9eaeda 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -302,9 +302,9 @@ void __init swiotlb_update_mem_attributes(void) } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, - unsigned long nslabs, bool late_alloc, unsigned int nareas) + void *vaddr, unsigned long nslabs, bool late_alloc, + unsigned int nareas) { - void *vaddr =3D phys_to_virt(start); unsigned long bytes =3D nslabs << IO_TLB_SHIFT, i; =20 mem->nslabs =3D nslabs; @@ -445,7 +445,7 @@ void __init swiotlb_init_remap(bool addressing_limit, u= nsigned int flags, return; } =20 - swiotlb_init_io_tlb_pool(mem, __pa(tlb), nslabs, false, nareas); + swiotlb_init_io_tlb_pool(mem, __pa(tlb), tlb, nslabs, false, nareas); add_mem_pool(&io_tlb_default_mem, mem); =20 if (flags & SWIOTLB_VERBOSE) @@ -553,7 +553,7 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, } } =20 - swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), vstart, nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); =20 @@ -664,25 +664,26 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, * @phys_limit: Maximum allowed physical address of the buffer. * @attrs: DMA attributes for the allocation. * @gfp: GFP flags for the allocation. + * @vaddr: Receives the virtual address for the allocated buffer. * * Return: Allocated pages, or %NULL on allocation failure. */ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, - u64 phys_limit, unsigned long attrs, gfp_t gfp) + u64 phys_limit, unsigned long attrs, gfp_t gfp, void **vaddr) { struct page *page; =20 + *vaddr =3D NULL; + /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ if (!gfpflags_allow_blocking(gfp) && (attrs & DMA_ATTR_CC_SHARED)) { - void *vaddr; - if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 - return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, + return dma_alloc_from_pool(dev, bytes, vaddr, gfp, attrs, dma_coherent_ok); } =20 @@ -705,6 +706,8 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, return NULL; } =20 + if (page) + *vaddr =3D phys_to_virt(page_to_phys(page)); return page; } =20 @@ -750,6 +753,7 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, { struct io_tlb_pool *pool; unsigned int slot_order; + void *tlb_vaddr; struct page *tlb; size_t pool_size; size_t tlb_size; @@ -767,7 +771,8 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, pool->unencrypted =3D !!(attrs & DMA_ATTR_CC_SHARED); =20 tlb_size =3D nslabs << IO_TLB_SHIFT; - while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, attrs, gfp)= )) { + while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, attrs, gfp, + &tlb_vaddr))) { if (nslabs <=3D minslabs) goto error_tlb; nslabs =3D ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -781,12 +786,12 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct = device *dev, if (!pool->slots) goto error_slots; =20 - swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), nslabs, true, nareas); + swiotlb_init_io_tlb_pool(pool, page_to_phys(tlb), tlb_vaddr, nslabs, + true, nareas); return pool; =20 error_slots: - swiotlb_free_tlb(page_address(tlb), tlb_size, - !!(attrs & DMA_ATTR_CC_SHARED)); + swiotlb_free_tlb(tlb_vaddr, tlb_size, !!(attrs & DMA_ATTR_CC_SHARED)); error_tlb: kfree(pool); error: @@ -1995,7 +2000,8 @@ static int rmem_swiotlb_device_init(struct reserved_m= em *rmem, mem->unencrypted =3D false; } =20 - swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs, + swiotlb_init_io_tlb_pool(pool, rmem->base, phys_to_virt(rmem->base), + nslabs, false, nareas); mem->force_bounce =3D true; mem->for_alloc =3D true; --=20 2.43.0 From nobody Mon Jun 8 09:48:10 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DF5683F7A83; Thu, 4 Jun 2026 08:43:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562597; cv=none; b=n9fsCKe7gpUg0pFITlNschJKetZU2Uh5iUJ2jXEWBxEA1fL45iafrWRjvvltRQMxYhH60eCB6yncB9Eai1EA0eE81N4Yuz91sBI24PJ1gSsK5Z0cLcCpwPds7I8msXa4xsMJfKNAgdnQsQkTF3vcG/zXRhmQIbG15iO1wX0BENM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780562597; c=relaxed/simple; bh=FQaMnL+ddLrG1vPODW9NwFl0dcAcGQZGEyYQRkwIpHU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LfeS2vMz0GdRok6IBSHeUKd9w0Al87yscDAGWf5YN1kH+LjyQeqDjfss+rx7qZmB0J5D8K83j2uHFrUOmbypGl6kEV4Ky3aHpLjs1TRzRtjBDbmu0ldO3PGOoLsZ+rgX2eBz7kS2SNGyXrG6cpZfHrqxCHgE0Nu32ojGH1HSuDs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Cgae1E98; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Cgae1E98" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 480651F00898; Thu, 4 Jun 2026 08:43:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780562596; bh=ePeZTaXNJBx/2RYYA/AuIc3DdpRwUBlOsN24dG2IECQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=Cgae1E98GtHsFcI4+E6f2e3cq2gB5ntC7mDmNNAmlcu5ofgb9Rubm286CJNZMNO10 qSFJCmn7ZFC61rYD6tfmuyK2Pzcg2VxIatIJe4OAZ5ves1nEMvLkEV/62k1e+xJ//b nXAAvIdPZbsJnOB0bCNBUOXCMcnl51t8XBadLEDJSbPxMoUwt9hpY2bJyOQRq7k+Tb ZXEo7XRzcFBuF2TBZpyM78U0taYt44WBdBG2MnDNlV5a6W/7EMKvsx0Rsg20fl8Xkz RFARCUaz4S8GDwUqoxWryHd722PxfNT5E8SHmtzPEgPvhIKK5O3G0JsyBaIGYbWz+8 DaVLcJHMYLEGQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v6 20/20] swiotlb: remove unused SWIOTLB_FORCE flag Date: Thu, 4 Jun 2026 14:09:59 +0530 Message-ID: <20260604083959.1265923-21-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260604083959.1265923-1-aneesh.kumar@kernel.org> References: <20260604083959.1265923-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" SWIOTLB_FORCE has no remaining in-tree users. Forced bouncing is now controlled through the swiotlb=3Dforce command line option via swiotlb_force_bounce. Remove the unused flag and simplify the force_bounce initialization. Signed-off-by: Aneesh Kumar K.V (Arm) --- include/linux/swiotlb.h | 1 - kernel/dma/swiotlb.c | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 526f82e9da45..af88ca7182f4 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -15,7 +15,6 @@ struct page; struct scatterlist; =20 #define SWIOTLB_VERBOSE (1 << 0) /* verbose initialization */ -#define SWIOTLB_FORCE (1 << 1) /* force bounce buffering */ #define SWIOTLB_ANY (1 << 2) /* allow any memory for the buffer */ =20 /* diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index e4bd8c9eaeda..81cc4928e949 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -400,8 +400,7 @@ void __init swiotlb_init_remap(bool addressing_limit, u= nsigned int flags, if (swiotlb_force_disable) return; =20 - io_tlb_default_mem.force_bounce =3D - swiotlb_force_bounce || (flags & SWIOTLB_FORCE); + io_tlb_default_mem.force_bounce =3D swiotlb_force_bounce; =20 #ifdef CONFIG_SWIOTLB_DYNAMIC if (!remap) --=20 2.43.0