From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEBB33B27C9; Tue, 12 May 2026 09:04:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576682; cv=none; b=rDZlwXnHZuCI2dwQp97ebYHbsx2WK1AL+g4JpTPpMvL/OlbJSHkNN808EpSoVlPvEmfDTP1C4XAyYwHBAoRexhUWa010GGTTu0JrGDZHjVNV6vocYZsu2juK5K3e5pCCLt9aGjsWr9ARCrGH6cbRvf4WJt72wZ7vTzgc18YDOdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576682; c=relaxed/simple; bh=3swRvyeNc5AeLtSgBuVCbdTez2sp8DKcDsxRQ7clE/A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rySYa2VUlGLxa3RgqdhgO22YH7K5y26LmjoymtO5EK94dTiqzS9fbf6e1uMqgdqP7kTRFDmknZcjKoXXmNp9Zz5KEj632OI3aQAU+xs25wRLBod3S3EJbQPKnZuSkD2gVhrtJ6yhVnWLJYEcKgDiVUMsET8ToMtPX6/nY2jf1hM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=I8zLsCKL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="I8zLsCKL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6D65C2BCB0; Tue, 12 May 2026 09:04:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576682; bh=3swRvyeNc5AeLtSgBuVCbdTez2sp8DKcDsxRQ7clE/A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=I8zLsCKLzKlKTVVx8IB04PaLldjzb5ngR2LXJqUMpFvzrdIt4lBh5JXYMV5eBvDYq s3ELYOzDsqknbNli4Bm6EE68d/EHzctMG2P/USM9ChrbSfHMXpu7ViYlmzLDd2ZUTg 2fvI0vczmwGT8r5US5wE3XfhR4f2YPdsvGAF28PX7TOhGexLPdVAEm5QKPbsTHawzS awaaC/tkWzbCz84M5u4+oGK4eaaNMwvocUqa+YjjajD6gJtNA/lUJioxPivRN8XgEJ tE7tjV3C+MsIIqZdrRHskQgXdcTuzhkUomtp2F4lW6m8CTPLaQs7TzEtKxbyJE48VJ JQEHPsu6OCNEA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 01/13] dma-direct: swiotlb: handle swiotlb alloc/free outside __dma_direct_alloc_pages Date: Tue, 12 May 2026 14:33:56 +0530 Message-ID: <20260512090408.794195-2-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move swiotlb allocation out of __dma_direct_alloc_pages() and handle it in dma_direct_alloc() / dma_direct_alloc_pages(). This is needed for follow-up changes that simplify the handling of memory encryption/decryption based on the DMA attribute flags. swiotlb backing pages are already mapped decrypted by swiotlb_update_mem_attributes() and rmem_swiotlb_device_init(), so dma-direct should not call dma_set_decrypted() on allocation nor dma_set_encrypted() on free for swiotlb-backed memory. Update alloc/free paths to detect swiotlb-backed pages and skip encrypt/decrypt transitions for those paths. Keep the existing highmem rejection in dma_direct_alloc_pages() for swiotlb allocations. Only for "restricted-dma-pool", we currently set `for_alloc =3D true`, while rmem_swiotlb_device_init() decrypts the whole pool up front. This pool is typically used together with "shared-dma-pool", where the shared region is accessed after remap/ioremap and the returned address is suitable for decrypted memory access. So existing code paths remain valid. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 44 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index ec887f443741..b958f150718a 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -125,9 +125,6 @@ static struct page *__dma_direct_alloc_pages(struct dev= ice *dev, size_t size, =20 WARN_ON_ONCE(!PAGE_ALIGNED(size)); =20 - if (is_swiotlb_for_alloc(dev)) - return dma_direct_alloc_swiotlb(dev, size); - gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); page =3D dma_alloc_contiguous(dev, size, gfp); if (page) { @@ -204,6 +201,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; + bool mark_mem_decrypt =3D true; struct page *page; void *ret; =20 @@ -250,11 +248,21 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (page) { + mark_mem_decrypt =3D false; + goto setup_page; + } + return NULL; + } + /* we always manually zero the memory once we are done */ page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); if (!page) return NULL; =20 +setup_page: /* * dma_alloc_contiguous can return highmem pages depending on a * combination the cma=3D arguments and per-arch setup. These need to be @@ -281,7 +289,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, goto out_free_pages; } else { ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) goto out_leak_pages; } =20 @@ -298,7 +306,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, return ret; =20 out_encrypt_pages: - if (dma_set_encrypted(dev, page_address(page), size)) + if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) return NULL; out_free_pages: __dma_direct_free_pages(dev, page, size); @@ -310,6 +318,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { + bool mark_mem_encrypted =3D true; unsigned int page_order =3D get_order(size); =20 if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && @@ -338,12 +347,15 @@ void dma_direct_free(struct device *dev, size_t size, dma_free_from_pool(dev, cpu_addr, PAGE_ALIGN(size))) return; =20 + if (swiotlb_find_pool(dev, dma_to_phys(dev, dma_addr))) + mark_mem_encrypted =3D false; + if (is_vmalloc_addr(cpu_addr)) { vunmap(cpu_addr); } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (dma_set_encrypted(dev, cpu_addr, size)) + if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) return; } =20 @@ -359,6 +371,19 @@ struct page *dma_direct_alloc_pages(struct device *dev= , size_t size, if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 + if (is_swiotlb_for_alloc(dev)) { + page =3D dma_direct_alloc_swiotlb(dev, size); + if (!page) + return NULL; + + if (PageHighMem(page)) { + swiotlb_free(dev, page, size); + return NULL; + } + ret =3D page_address(page); + goto setup_page; + } + page =3D __dma_direct_alloc_pages(dev, size, gfp, false); if (!page) return NULL; @@ -366,6 +391,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, ret =3D page_address(page); if (dma_set_decrypted(dev, ret, size)) goto out_leak_pages; +setup_page: memset(ret, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return page; @@ -378,13 +404,17 @@ void dma_direct_free_pages(struct device *dev, size_t= size, enum dma_data_direction dir) { void *vaddr =3D page_address(page); + bool mark_mem_encrypted =3D true; =20 /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool(dev, vaddr, size)) return; =20 - if (dma_set_encrypted(dev, vaddr, size)) + if (swiotlb_find_pool(dev, page_to_phys(page))) + mark_mem_encrypted =3D false; + + if (mark_mem_encrypted && dma_set_encrypted(dev, vaddr, size)) return; __dma_direct_free_pages(dev, page, size); } --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D57F639D6DA; Tue, 12 May 2026 09:04:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576694; cv=none; b=Nu3+4mt2JkLd158gjASQj1EVMSjLzu20UwITbH+WsvCnE4mAQggGGOaUVZMXTa5NV9bsvnqgwslazmPfMbO/K+GSEZHCTDtgIzpUqe+jza90Z8A7/qeiL+bkCojPvvN1PX1iNwSOxUAXf5cAbg1/zIH6NNkQxjPAbOhNsgYim9c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576694; c=relaxed/simple; bh=4MW7wMvDoqITfLwkJTW6/lwo+jU4h0pFXHWOV1R6rJM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tsaQowLKWSf+hRHDDNCrIklfMaG4LHPbLcb4afYUR7MltTNTyTHAOxKUu8LM8jtT/wiQI+lvB7y6r6xwos2yxuCnGaeV89rkgsOYtmKlJHU/43AwUrFIS5LSmDwWzbYh957frYCq7ltrq01cUmwAhjTC7IZUhZKOOmg1pGNMclc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KbwCoE0A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KbwCoE0A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8431C2BCF5; Tue, 12 May 2026 09:04:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576694; bh=4MW7wMvDoqITfLwkJTW6/lwo+jU4h0pFXHWOV1R6rJM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KbwCoE0ASwdaQ0sDqenX3eV2dOoxqZ1/TFckfyJW6wsl78OCEz7fdBRLlIWH2WVy/ sFRV8gkQD9L/Mqs80GkVY2hI9N/Zl7IpCNaaBGUabr02N0g8a8xHQVKvE27qFMc22Y Moc5qKLCRlFqPVQFD6o8OfprtKqgn6hObtzcYyV/ja3ahXjLP/J7BSn41O6emV6XZM j44eIFpfSCPhdsNwc6gVoAwh43+ZTWY1hvO0qfOUkcIY8lyuB3mKF2isu0H2bj7lcx 0qMWtvyB5JjqGX+H3+qQAlmugtaUALG3nubVqCuniv+CCUgcdEkzGirzgWBjcgdr1o Hvr5pzU6q77Eg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 02/13] dma-direct: use DMA_ATTR_CC_SHARED in alloc/free paths Date: Tue, 12 May 2026 14:33:57 +0530 Message-ID: <20260512090408.794195-3-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Propagate force_dma_unencrypted() into DMA_ATTR_CC_SHARED in the dma-direct allocation path and use the attribute to drive the related decisions. This updates dma_direct_alloc(), dma_direct_free(), and dma_direct_alloc_pages() to fold the forced unencrypted case into attrs. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 44 ++++++++++++++++++++++++++++++++++++-------- 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index b958f150718a..0c2e1f8436ce 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -201,16 +201,31 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { bool remap =3D false, set_uncached =3D false; - bool mark_mem_decrypt =3D true; + bool mark_mem_decrypt =3D false; struct page *page; void *ret; =20 + /* + * DMA_ATTR_CC_SHARED is not a caller-visible dma_alloc_*() + * attribute. The direct allocator uses it internally after it has + * decided that the backing pages must be shared/decrypted, so the + * rest of the allocation path can consistently select DMA addresses, + * choose compatible pools and restore encryption on free. + */ + if (attrs & DMA_ATTR_CC_SHARED) + return NULL; + + if (force_dma_unencrypted(dev)) { + attrs |=3D DMA_ATTR_CC_SHARED; + mark_mem_decrypt =3D true; + } + size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) gfp |=3D __GFP_NOWARN; =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | DMA_ATTR_CC_SHARED)) =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) return dma_direct_alloc_no_mapping(dev, size, dma_handle, gfp); =20 if (!dev_is_dma_coherent(dev)) { @@ -244,7 +259,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, * Remapping or decrypting memory may block, allocate the memory from * the atomic pools instead if we aren't allowed block. */ - if ((remap || force_dma_unencrypted(dev)) && + if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 @@ -318,11 +333,20 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, void dma_direct_free(struct device *dev, size_t size, void *cpu_addr, dma_addr_t dma_addr, unsigned long attrs) { - bool mark_mem_encrypted =3D true; + bool mark_mem_encrypted =3D false; unsigned int page_order =3D get_order(size); =20 - if ((attrs & DMA_ATTR_NO_KERNEL_MAPPING) && - !force_dma_unencrypted(dev) && !is_swiotlb_for_alloc(dev)) { + /* + * if the device had requested for an unencrypted buffer, + * convert it to encrypted on free + */ + if (force_dma_unencrypted(dev)) { + attrs |=3D DMA_ATTR_CC_SHARED; + mark_mem_encrypted =3D true; + } + + if (((attrs & (DMA_ATTR_NO_KERNEL_MAPPING | DMA_ATTR_CC_SHARED)) =3D=3D + DMA_ATTR_NO_KERNEL_MAPPING) && !is_swiotlb_for_alloc(dev)) { /* cpu_addr is a struct page cookie, not a kernel address */ dma_free_contiguous(dev, cpu_addr, size); return; @@ -365,10 +389,14 @@ void dma_direct_free(struct device *dev, size_t size, struct page *dma_direct_alloc_pages(struct device *dev, size_t size, dma_addr_t *dma_handle, enum dma_data_direction dir, gfp_t gfp) { + unsigned long attrs =3D 0; struct page *page; void *ret; =20 - if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) + if (force_dma_unencrypted(dev)) + attrs |=3D DMA_ATTR_CC_SHARED; + + if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); =20 if (is_swiotlb_for_alloc(dev)) { --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AF8D3B0AD7; Tue, 12 May 2026 09:05:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576706; cv=none; b=WMoAQnjmEkVMzRrVvcbgUbxrsfTXDKDaoHIbpm0gQSzd/m+GBgrCAC+XOKkGt+pnG4E5qyBXpgwf95/AKWOrNvf88qAFD8ha/CNUMPF85l8WrS89pUZMEUOAPALfPDcXPtdwmNebxXmLDGilOFr0LtQWjphjxDMAy+Ti+w7WiR8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576706; c=relaxed/simple; bh=j8HH2YzaomJYgS4p7VtC/iThZopaDRn/08EzGlHXEEw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=daAbgOzZZYheByBh8A+bR8KqoyjXCC6TV1f1SBstuOdpdpT7+VdR+m0uERNnl0iHFqxCAWY4jvn3bPq35sf7bKdteprwgUgEbHY5LYODqTxY7uRwtXabWHC11O+WHRXgHM58ATTVndSnE6u1TJpchi91+7r2/yyeGZdA+B+sajc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dCIUKZx5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dCIUKZx5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E350CC2BCB0; Tue, 12 May 2026 09:04:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576706; bh=j8HH2YzaomJYgS4p7VtC/iThZopaDRn/08EzGlHXEEw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dCIUKZx5FHpZbnYu2ZPfKVOqJZ+fkFIsBUG9DxgRuQG5gaGNPzVBSOE1hdDSIWe5R +893PZlWBf+VpZ5T+qNbml78oyVSe4kGadZYaz1LvJ31tLa8hoWof4vxN2XsTS+D+B +va99+Wsl/kTizQKCCfBMXDp0Bn+GrV880iqHg5B2+tu1LUJhcJJr5VT3e6vR6vM1r r+s5srPky/dOW8AJ2YjutWePcMgys+hMxfggRvRP5XSVedWG17u5wX38DR9ktR+rt9 5Ylwl2DSV4WxfZX0eSCokAMXtbSheJeWDnWczW7+5dNWOSBRx7s8J0wbTtrbTSFBg6 EpH3TviyPbApQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 03/13] dma-pool: track decrypted atomic pools and select them via attrs Date: Tue, 12 May 2026 14:33:58 +0530 Message-ID: <20260512090408.794195-4-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach the atomic DMA pool code to distinguish between encrypted and decrypted pools, and make pool allocation select the matching pool based on DMA attributes. Introduce a dma_gen_pool wrapper that records whether a pool is decrypted, initialize that state when the atomic pools are created, and use it when expanding and resizing the pools. Update dma_alloc_from_pool() to take attrs and skip pools whose encrypted/decrypted state does not match DMA_ATTR_CC_SHARED. Update dma_free_from_pool() accordingly. Also pass DMA_ATTR_CC_SHARED from the swiotlb atomic allocation path so decrypted swiotlb allocations are taken from the correct atomic pool. Signed-off-by: Aneesh Kumar K.V (Arm) Reviewed-by: Mostafa Saleh Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- drivers/iommu/dma-iommu.c | 2 +- include/linux/dma-map-ops.h | 2 +- kernel/dma/direct.c | 11 ++- kernel/dma/pool.c | 163 +++++++++++++++++++++++------------- kernel/dma/swiotlb.c | 7 +- 5 files changed, 122 insertions(+), 63 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 54d96e847f16..c2595bee3d41 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1673,7 +1673,7 @@ void *iommu_dma_alloc(struct device *dev, size_t size= , dma_addr_t *handle, if (IS_ENABLED(CONFIG_DMA_DIRECT_REMAP) && !gfpflags_allow_blocking(gfp) && !coherent) page =3D dma_alloc_from_pool(dev, PAGE_ALIGN(size), &cpu_addr, - gfp, NULL); + gfp, attrs, NULL); else cpu_addr =3D iommu_dma_alloc_pages(dev, size, &page, gfp, attrs); if (!cpu_addr) diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 6a1832a73cad..696b2c3a2305 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -212,7 +212,7 @@ void *dma_common_pages_remap(struct page **pages, size_= t size, pgprot_t prot, void dma_common_free_remap(void *cpu_addr, size_t size); =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t flags, + void **cpu_addr, gfp_t flags, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)); bool dma_free_from_pool(struct device *dev, void *start, size_t size); =20 diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 0c2e1f8436ce..dc2907439b3d 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -162,7 +162,7 @@ static bool dma_direct_use_pool(struct device *dev, gfp= _t gfp) } =20 static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp) + dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) { struct page *page; u64 phys_limit; @@ -172,7 +172,8 @@ static void *dma_direct_alloc_from_pool(struct device *= dev, size_t size, return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok); + page =3D dma_alloc_from_pool(dev, size, &ret, gfp, attrs, + dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); @@ -261,7 +262,8 @@ void *dma_direct_alloc(struct device *dev, size_t size, */ if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, + gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size); @@ -397,7 +399,8 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, attrs |=3D DMA_ATTR_CC_SHARED; =20 if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, + gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size); diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index 2b2fbb709242..75f0eba48a23 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -12,12 +12,18 @@ #include #include #include +#include =20 -static struct gen_pool *atomic_pool_dma __ro_after_init; +struct dma_gen_pool { + bool unencrypted; + struct gen_pool *pool; +}; + +static struct dma_gen_pool atomic_pool_dma __ro_after_init; static unsigned long pool_size_dma; -static struct gen_pool *atomic_pool_dma32 __ro_after_init; +static struct dma_gen_pool atomic_pool_dma32 __ro_after_init; static unsigned long pool_size_dma32; -static struct gen_pool *atomic_pool_kernel __ro_after_init; +static struct dma_gen_pool atomic_pool_kernel __ro_after_init; static unsigned long pool_size_kernel; =20 /* Size can be defined by the coherent_pool command line */ @@ -76,7 +82,7 @@ static bool cma_in_zone(gfp_t gfp) return true; } =20 -static int atomic_pool_expand(struct gen_pool *pool, size_t pool_size, +static int atomic_pool_expand(struct dma_gen_pool *dma_pool, size_t pool_s= ize, gfp_t gfp) { unsigned int order; @@ -113,12 +119,15 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, * Memory in the atomic DMA pools must be unencrypted, the pools do not * shrink so no re-encryption occurs in dma_direct_free(). */ - ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), + if (dma_pool->unencrypted) { + ret =3D set_memory_decrypted((unsigned long)page_to_virt(page), 1 << order); - if (ret) - goto remove_mapping; - ret =3D gen_pool_add_virt(pool, (unsigned long)addr, page_to_phys(page), - pool_size, NUMA_NO_NODE); + if (ret) + goto remove_mapping; + } + + ret =3D gen_pool_add_virt(dma_pool->pool, (unsigned long)addr, + page_to_phys(page), pool_size, NUMA_NO_NODE); if (ret) goto encrypt_mapping; =20 @@ -126,11 +135,15 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, return 0; =20 encrypt_mapping: - ret =3D set_memory_encrypted((unsigned long)page_to_virt(page), - 1 << order); - if (WARN_ON_ONCE(ret)) { - /* Decrypt succeeded but encrypt failed, purposely leak */ - goto out; + if (dma_pool->unencrypted) { + int rc; + + rc =3D set_memory_encrypted((unsigned long)page_to_virt(page), + 1 << order); + if (WARN_ON_ONCE(rc)) { + /* Decrypt succeeded but encrypt failed, purposely leak */ + goto out; + } } remove_mapping: #ifdef CONFIG_DMA_DIRECT_REMAP @@ -142,46 +155,52 @@ static int atomic_pool_expand(struct gen_pool *pool, = size_t pool_size, return ret; } =20 -static void atomic_pool_resize(struct gen_pool *pool, gfp_t gfp) +static void atomic_pool_resize(struct dma_gen_pool *dma_pool, gfp_t gfp) { - if (pool && gen_pool_avail(pool) < atomic_pool_size) - atomic_pool_expand(pool, gen_pool_size(pool), gfp); + if (dma_pool->pool && gen_pool_avail(dma_pool->pool) < atomic_pool_size) + atomic_pool_expand(dma_pool, gen_pool_size(dma_pool->pool), gfp); } =20 static void atomic_pool_work_fn(struct work_struct *work) { if (IS_ENABLED(CONFIG_ZONE_DMA)) - atomic_pool_resize(atomic_pool_dma, + atomic_pool_resize(&atomic_pool_dma, GFP_KERNEL | GFP_DMA); if (IS_ENABLED(CONFIG_ZONE_DMA32)) - atomic_pool_resize(atomic_pool_dma32, + atomic_pool_resize(&atomic_pool_dma32, GFP_KERNEL | GFP_DMA32); - atomic_pool_resize(atomic_pool_kernel, GFP_KERNEL); + atomic_pool_resize(&atomic_pool_kernel, GFP_KERNEL); } =20 -static __init struct gen_pool *__dma_atomic_pool_init(size_t pool_size, - gfp_t gfp) +static __init struct dma_gen_pool *__dma_atomic_pool_init(struct dma_gen_p= ool *dma_pool, + size_t pool_size, gfp_t gfp) { - struct gen_pool *pool; int ret; =20 - pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); - if (!pool) + dma_pool->pool =3D gen_pool_create(PAGE_SHIFT, NUMA_NO_NODE); + if (!dma_pool->pool) return NULL; =20 - gen_pool_set_algo(pool, gen_pool_first_fit_order_align, NULL); + gen_pool_set_algo(dma_pool->pool, gen_pool_first_fit_order_align, NULL); + + /* if platform is using memory encryption atomic pools are by default dec= rypted. */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + dma_pool->unencrypted =3D true; + else + dma_pool->unencrypted =3D false; =20 - ret =3D atomic_pool_expand(pool, pool_size, gfp); + ret =3D atomic_pool_expand(dma_pool, pool_size, gfp); if (ret) { - gen_pool_destroy(pool); + gen_pool_destroy(dma_pool->pool); + dma_pool->pool =3D NULL; pr_err("DMA: failed to allocate %zu KiB %pGg pool for atomic allocation\= n", pool_size >> 10, &gfp); return NULL; } =20 pr_info("DMA: preallocated %zu KiB %pGg pool for atomic allocations\n", - gen_pool_size(pool) >> 10, &gfp); - return pool; + gen_pool_size(dma_pool->pool) >> 10, &gfp); + return dma_pool; } =20 #ifdef CONFIG_ZONE_DMA32 @@ -207,21 +226,22 @@ static int __init dma_atomic_pool_init(void) =20 /* All memory might be in the DMA zone(s) to begin with */ if (has_managed_zone(ZONE_NORMAL)) { - atomic_pool_kernel =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL); - if (!atomic_pool_kernel) + __dma_atomic_pool_init(&atomic_pool_kernel, atomic_pool_size, GFP_KERNEL= ); + if (!atomic_pool_kernel.pool) ret =3D -ENOMEM; } + if (has_managed_dma()) { - atomic_pool_dma =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA); - if (!atomic_pool_dma) + __dma_atomic_pool_init(&atomic_pool_dma, atomic_pool_size, + GFP_KERNEL | GFP_DMA); + if (!atomic_pool_dma.pool) ret =3D -ENOMEM; } + if (has_managed_dma32) { - atomic_pool_dma32 =3D __dma_atomic_pool_init(atomic_pool_size, - GFP_KERNEL | GFP_DMA32); - if (!atomic_pool_dma32) + __dma_atomic_pool_init(&atomic_pool_dma32, atomic_pool_size, + GFP_KERNEL | GFP_DMA32); + if (!atomic_pool_dma32.pool) ret =3D -ENOMEM; } =20 @@ -230,19 +250,44 @@ static int __init dma_atomic_pool_init(void) } postcore_initcall(dma_atomic_pool_init); =20 -static inline struct gen_pool *dma_guess_pool(struct gen_pool *prev, gfp_t= gfp) +static inline struct dma_gen_pool *__dma_guess_pool(struct dma_gen_pool *f= irst, + struct dma_gen_pool *second, struct dma_gen_pool *third) +{ + if (first->pool) + return first; + if (second && second->pool) + return second; + if (third && third->pool) + return third; + return NULL; +} + +static inline struct dma_gen_pool *dma_guess_pool(struct dma_gen_pool *pre= v, + gfp_t gfp) { - if (prev =3D=3D NULL) { + if (!prev) { if (gfp & GFP_DMA) - return atomic_pool_dma ?: atomic_pool_dma32 ?: atomic_pool_kernel; + return __dma_guess_pool(&atomic_pool_dma, + &atomic_pool_dma32, + &atomic_pool_kernel); + if (gfp & GFP_DMA32) - return atomic_pool_dma32 ?: atomic_pool_dma ?: atomic_pool_kernel; - return atomic_pool_kernel ?: atomic_pool_dma32 ?: atomic_pool_dma; + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, + &atomic_pool_kernel); + + return __dma_guess_pool(&atomic_pool_kernel, + &atomic_pool_dma32, + &atomic_pool_dma); } - if (prev =3D=3D atomic_pool_kernel) - return atomic_pool_dma32 ? atomic_pool_dma32 : atomic_pool_dma; - if (prev =3D=3D atomic_pool_dma32) - return atomic_pool_dma; + + if (prev =3D=3D &atomic_pool_kernel) + return __dma_guess_pool(&atomic_pool_dma32, + &atomic_pool_dma, NULL); + + if (prev =3D=3D &atomic_pool_dma32) + return __dma_guess_pool(&atomic_pool_dma, NULL, NULL); + return NULL; } =20 @@ -272,16 +317,20 @@ static struct page *__dma_alloc_from_pool(struct devi= ce *dev, size_t size, } =20 struct page *dma_alloc_from_pool(struct device *dev, size_t size, - void **cpu_addr, gfp_t gfp, + void **cpu_addr, gfp_t gfp, unsigned long attrs, bool (*phys_addr_ok)(struct device *, phys_addr_t, size_t)) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; struct page *page; bool pool_found =3D false; =20 - while ((pool =3D dma_guess_pool(pool, gfp))) { + while ((dma_pool =3D dma_guess_pool(dma_pool, gfp))) { + + if (dma_pool->unencrypted !=3D !!(attrs & DMA_ATTR_CC_SHARED)) + continue; + pool_found =3D true; - page =3D __dma_alloc_from_pool(dev, size, pool, cpu_addr, + page =3D __dma_alloc_from_pool(dev, size, dma_pool->pool, cpu_addr, phys_addr_ok); if (page) return page; @@ -296,12 +345,14 @@ struct page *dma_alloc_from_pool(struct device *dev, = size_t size, =20 bool dma_free_from_pool(struct device *dev, void *start, size_t size) { - struct gen_pool *pool =3D NULL; + struct dma_gen_pool *dma_pool =3D NULL; + + while ((dma_pool =3D dma_guess_pool(dma_pool, 0))) { =20 - while ((pool =3D dma_guess_pool(pool, 0))) { - if (!gen_pool_has_addr(pool, (unsigned long)start, size)) + if (!gen_pool_has_addr(dma_pool->pool, (unsigned long)start, size)) continue; - gen_pool_free(pool, (unsigned long)start, size); + + gen_pool_free(dma_pool->pool, (unsigned long)start, size); return true; } =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 1abd3e6146f4..ab4eccbaa076 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -612,6 +612,7 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, u64 phys_limit, gfp_t gfp) { struct page *page; + unsigned long attrs =3D 0; =20 /* * Allocate from the atomic pools if memory is encrypted and @@ -623,8 +624,12 @@ static struct page *swiotlb_alloc_tlb(struct device *d= ev, size_t bytes, if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 + /* swiotlb considered decrypted by default */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + attrs =3D DMA_ATTR_CC_SHARED; + return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, - dma_coherent_ok); + attrs, dma_coherent_ok); } =20 gfp &=3D ~GFP_ZONEMASK; --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 601803B6C1F; Tue, 12 May 2026 09:05:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576718; cv=none; b=mUamEq5k3qxjHeyuj2ULa3Z1QpzWKWbU1pG+rZ//mYwkUwgD0SMITIfQFlQe7S2kkmQQ1J8TIDMX7p6oOb2tMeWKwuQeX1MieFdt0OLH37h43x3qkyeLy7iv1vU+mNVXbUWGWqQnstiQywieuQg9qEEUa5G2RUoXVNUlenAFcos= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576718; c=relaxed/simple; bh=7Y8s+GhOMqBiueS9KyGb7zFob1fGnzzgUGMYkKRYdxE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XbwkPqnG6unu33YzfTp/3yiuCQHTO5ZJJMHmxlsxWQmzAkcAa/MgoSTaoRAwLsHZNDZoNz/tphztacGiqe1NUhd+lvg/9CgoRPfV5IwrzYaRcGtdkqaJY5lAY0TxLC2a5CpwC7kie+h6NRoINLAYfpgtRU4uhxOLpOIz+StTwUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=OoAZ4Vz8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="OoAZ4Vz8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id A911FC2BCC7; Tue, 12 May 2026 09:05:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576717; bh=7Y8s+GhOMqBiueS9KyGb7zFob1fGnzzgUGMYkKRYdxE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OoAZ4Vz87oZEIXE2YbnjuIt+cX9zBzE13NhTiGFtD+PF0Q05t0+Yvmvu0sUo7kEd5 9MwvztFO6sgT6eeQOkABzSf+cC3c0PnkZYEbtzVap7UBzcQesxS1U5U6nSk155S4uG k0tPNDm6AZKzUGObEC8wTojHCfbqA5NCsD3wtnFv0gy/azcC3+I04d46iMF4bOGYVI 2vVLTzNLDUIog9RPtlT8UtdZMn0YCbfLwzuU6RkaLCRSaG+7+hDNERbOPeEW348DWH KJ7hBt/iFQ/lB6VJLXhJ9blAZoNA/ownQEUn9voG1H3UTsYj884xltDNcmQPWO5VQY fwZ7i5u4pVMvw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 04/13] dma: swiotlb: track pool encryption state and honor DMA_ATTR_CC_SHARED Date: Tue, 12 May 2026 14:33:59 +0530 Message-ID: <20260512090408.794195-5-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach swiotlb to distinguish between encrypted and decrypted bounce buffer pools, and make allocation and mapping paths select a pool whose state matches the requested DMA attributes. Add a decrypted flag to io_tlb_mem, initialize it for the default and restricted pools, and propagate DMA_ATTR_CC_SHARED into swiotlb pool allocation. Reject swiotlb alloc/map requests when the selected pool does not match the required encrypted/decrypted state. Also return DMA addresses with the matching phys_to_dma_{encrypted, unencrypted} helper so the DMA address encoding stays consistent with the chosen pool. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- include/linux/dma-direct.h | 10 ++++ include/linux/swiotlb.h | 8 ++- kernel/dma/direct.c | 14 +++-- kernel/dma/swiotlb.c | 108 +++++++++++++++++++++++++++---------- 4 files changed, 107 insertions(+), 33 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index c249912456f9..94fad4e7c11e 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -77,6 +77,10 @@ static inline dma_addr_t dma_range_map_max(const struct = bus_dma_region *map) #ifndef phys_to_dma_unencrypted #define phys_to_dma_unencrypted phys_to_dma #endif + +#ifndef phys_to_dma_encrypted +#define phys_to_dma_encrypted phys_to_dma +#endif #else static inline dma_addr_t __phys_to_dma(struct device *dev, phys_addr_t pad= dr) { @@ -90,6 +94,12 @@ static inline dma_addr_t phys_to_dma_unencrypted(struct = device *dev, { return dma_addr_unencrypted(__phys_to_dma(dev, paddr)); } + +static inline dma_addr_t phys_to_dma_encrypted(struct device *dev, + phys_addr_t paddr) +{ + return dma_addr_encrypted(__phys_to_dma(dev, paddr)); +} /* * If memory encryption is supported, phys_to_dma will set the memory encr= yption * bit in the DMA address, and dma_to_phys will clear it. diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 3dae0f592063..b3fa3c6e0169 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -81,6 +81,7 @@ struct io_tlb_pool { struct list_head node; struct rcu_head rcu; bool transient; + bool unencrypted; #endif }; =20 @@ -111,6 +112,7 @@ struct io_tlb_mem { struct dentry *debugfs; bool force_bounce; bool for_alloc; + bool unencrypted; #ifdef CONFIG_SWIOTLB_DYNAMIC bool can_grow; u64 phys_limit; @@ -282,7 +284,8 @@ static inline void swiotlb_sync_single_for_cpu(struct d= evice *dev, extern void swiotlb_print_info(void); =20 #ifdef CONFIG_DMA_RESTRICTED_POOL -struct page *swiotlb_alloc(struct device *dev, size_t size); +struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs); bool swiotlb_free(struct device *dev, struct page *page, size_t size); =20 static inline bool is_swiotlb_for_alloc(struct device *dev) @@ -290,7 +293,8 @@ static inline bool is_swiotlb_for_alloc(struct device *= dev) return dev->dma_io_tlb_mem->for_alloc; } #else -static inline struct page *swiotlb_alloc(struct device *dev, size_t size) +static inline struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs) { return NULL; } diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index dc2907439b3d..97ae4fa10521 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -104,9 +104,10 @@ static void __dma_direct_free_pages(struct device *dev= , struct page *page, dma_free_contiguous(dev, page, size); } =20 -static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze) +static struct page *dma_direct_alloc_swiotlb(struct device *dev, size_t si= ze, + unsigned long attrs) { - struct page *page =3D swiotlb_alloc(dev, size); + struct page *page =3D swiotlb_alloc(dev, size, attrs); =20 if (page && !dma_coherent_ok(dev, page_to_phys(page), size)) { swiotlb_free(dev, page, size); @@ -266,8 +267,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (page) { + /* + * swiotlb allocations comes from pool already marked + * decrypted + */ mark_mem_decrypt =3D false; goto setup_page; } @@ -374,6 +379,7 @@ void dma_direct_free(struct device *dev, size_t size, return; =20 if (swiotlb_find_pool(dev, dma_to_phys(dev, dma_addr))) + /* Swiotlb doesn't need a page attribute update on free */ mark_mem_encrypted =3D false; =20 if (is_vmalloc_addr(cpu_addr)) { @@ -403,7 +409,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { - page =3D dma_direct_alloc_swiotlb(dev, size); + page =3D dma_direct_alloc_swiotlb(dev, size, attrs); if (!page) return NULL; =20 diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index ab4eccbaa076..065663be282c 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -259,10 +259,21 @@ void __init swiotlb_update_mem_attributes(void) struct io_tlb_pool *mem =3D &io_tlb_default_mem.defpool; unsigned long bytes; =20 + /* + * if platform support memory encryption, swiotlb buffers are + * decrypted by default. + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) + io_tlb_default_mem.unencrypted =3D true; + else + io_tlb_default_mem.unencrypted =3D false; + if (!mem->nslabs || mem->late_alloc) return; bytes =3D PAGE_ALIGN(mem->nslabs << IO_TLB_SHIFT); - set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); + + if (io_tlb_default_mem.unencrypted) + set_memory_decrypted((unsigned long)mem->vaddr, bytes >> PAGE_SHIFT); } =20 static void swiotlb_init_io_tlb_pool(struct io_tlb_pool *mem, phys_addr_t = start, @@ -505,8 +516,10 @@ int swiotlb_init_late(size_t size, gfp_t gfp_mask, if (!mem->slots) goto error_slots; =20 - set_memory_decrypted((unsigned long)vstart, - (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) + set_memory_decrypted((unsigned long)vstart, + (nslabs << IO_TLB_SHIFT) >> PAGE_SHIFT); + swiotlb_init_io_tlb_pool(mem, virt_to_phys(vstart), nslabs, true, nareas); add_mem_pool(&io_tlb_default_mem, mem); @@ -539,7 +552,9 @@ void __init swiotlb_exit(void) tbl_size =3D PAGE_ALIGN(mem->end - mem->start); slots_size =3D PAGE_ALIGN(array_size(sizeof(*mem->slots), mem->nslabs)); =20 - set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (io_tlb_default_mem.unencrypted) + set_memory_encrypted(tbl_vaddr, tbl_size >> PAGE_SHIFT); + if (mem->late_alloc) { area_order =3D get_order(array_size(sizeof(*mem->areas), mem->nareas)); @@ -563,6 +578,7 @@ void __init swiotlb_exit(void) * @gfp: GFP flags for the allocation. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. + * @unencrypted: true to allocate unencrypted memory, false for encrypted = memory * * Allocate pages from the buddy allocator. If successful, make the alloca= ted * pages decrypted that they can be used for DMA. @@ -570,7 +586,8 @@ void __init swiotlb_exit(void) * Return: Decrypted pages, %NULL on allocation failure, or ERR_PTR(-EAGAI= N) * if the allocated physical address was above @phys_limit. */ -static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, u64 phys_limi= t) +static struct page *alloc_dma_pages(gfp_t gfp, size_t bytes, + u64 phys_limit, bool unencrypted) { unsigned int order =3D get_order(bytes); struct page *page; @@ -588,13 +605,13 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) } =20 vaddr =3D phys_to_virt(paddr); - if (set_memory_decrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (unencrypted && set_memory_decrypted((unsigned long)vaddr, PFN_UP(byte= s))) goto error; return page; =20 error: /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (unencrypted && !set_memory_encrypted((unsigned long)vaddr, PFN_UP(byt= es))) __free_pages(page, order); return NULL; } @@ -604,30 +621,26 @@ static struct page *alloc_dma_pages(gfp_t gfp, size_t= bytes, u64 phys_limit) * @dev: Device for which a memory pool is allocated. * @bytes: Size of the buffer. * @phys_limit: Maximum allowed physical address of the buffer. + * @attrs: DMA attributes for the allocation. * @gfp: GFP flags for the allocation. * * Return: Allocated pages, or %NULL on allocation failure. */ static struct page *swiotlb_alloc_tlb(struct device *dev, size_t bytes, - u64 phys_limit, gfp_t gfp) + u64 phys_limit, unsigned long attrs, gfp_t gfp) { struct page *page; - unsigned long attrs =3D 0; =20 /* * Allocate from the atomic pools if memory is encrypted and * the allocation is atomic, because decrypting may block. */ - if (!gfpflags_allow_blocking(gfp) && dev && force_dma_unencrypted(dev)) { + if (!gfpflags_allow_blocking(gfp) && (attrs & DMA_ATTR_CC_SHARED)) { void *vaddr; =20 if (!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)) return NULL; =20 - /* swiotlb considered decrypted by default */ - if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) - attrs =3D DMA_ATTR_CC_SHARED; - return dma_alloc_from_pool(dev, bytes, &vaddr, gfp, attrs, dma_coherent_ok); } @@ -638,7 +651,8 @@ static struct page *swiotlb_alloc_tlb(struct device *de= v, size_t bytes, else if (phys_limit <=3D DMA_BIT_MASK(32)) gfp |=3D __GFP_DMA32; =20 - while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit))) { + while (IS_ERR(page =3D alloc_dma_pages(gfp, bytes, phys_limit, + !!(attrs & DMA_ATTR_CC_SHARED)))) { if (IS_ENABLED(CONFIG_ZONE_DMA32) && phys_limit < DMA_BIT_MASK(64) && !(gfp & (__GFP_DMA32 | __GFP_DMA))) @@ -657,15 +671,18 @@ static struct page *swiotlb_alloc_tlb(struct device *= dev, size_t bytes, * swiotlb_free_tlb() - free a dynamically allocated IO TLB buffer * @vaddr: Virtual address of the buffer. * @bytes: Size of the buffer. + * @unencrypted: true if @vaddr was allocated decrypted and must be + * re-encrypted before being freed */ -static void swiotlb_free_tlb(void *vaddr, size_t bytes) +static void swiotlb_free_tlb(void *vaddr, size_t bytes, bool unencrypted) { if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && dma_free_from_pool(NULL, vaddr, bytes)) return; =20 /* Intentional leak if pages cannot be encrypted again. */ - if (!set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) + if (!unencrypted || + !set_memory_encrypted((unsigned long)vaddr, PFN_UP(bytes))) __free_pages(virt_to_page(vaddr), get_order(bytes)); } =20 @@ -676,6 +693,7 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes) * @nslabs: Desired (maximum) number of slabs. * @nareas: Number of areas. * @phys_limit: Maximum DMA buffer physical address. + * @attrs: DMA attributes for the allocation. * @gfp: GFP flags for the allocations. * * Allocate and initialize a new IO TLB memory pool. The actual number of @@ -686,7 +704,8 @@ static void swiotlb_free_tlb(void *vaddr, size_t bytes) */ static struct io_tlb_pool *swiotlb_alloc_pool(struct device *dev, unsigned long minslabs, unsigned long nslabs, - unsigned int nareas, u64 phys_limit, gfp_t gfp) + unsigned int nareas, u64 phys_limit, unsigned long attrs, + gfp_t gfp) { struct io_tlb_pool *pool; unsigned int slot_order; @@ -704,9 +723,10 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct d= evice *dev, if (!pool) goto error; pool->areas =3D (void *)pool + sizeof(*pool); + pool->unencrypted =3D !!(attrs & DMA_ATTR_CC_SHARED); =20 tlb_size =3D nslabs << IO_TLB_SHIFT; - while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, gfp))) { + while (!(tlb =3D swiotlb_alloc_tlb(dev, tlb_size, phys_limit, attrs, gfp)= )) { if (nslabs <=3D minslabs) goto error_tlb; nslabs =3D ALIGN(nslabs >> 1, IO_TLB_SEGSIZE); @@ -724,7 +744,8 @@ static struct io_tlb_pool *swiotlb_alloc_pool(struct de= vice *dev, return pool; =20 error_slots: - swiotlb_free_tlb(page_address(tlb), tlb_size); + swiotlb_free_tlb(page_address(tlb), tlb_size, + !!(attrs & DMA_ATTR_CC_SHARED)); error_tlb: kfree(pool); error: @@ -742,7 +763,9 @@ static void swiotlb_dyn_alloc(struct work_struct *work) struct io_tlb_pool *pool; =20 pool =3D swiotlb_alloc_pool(NULL, IO_TLB_MIN_SLABS, default_nslabs, - default_nareas, mem->phys_limit, GFP_KERNEL); + default_nareas, mem->phys_limit, + mem->unencrypted ? DMA_ATTR_CC_SHARED : 0, + GFP_KERNEL); if (!pool) { pr_warn_ratelimited("Failed to allocate new pool"); return; @@ -762,7 +785,7 @@ static void swiotlb_dyn_free(struct rcu_head *rcu) size_t tlb_size =3D pool->end - pool->start; =20 free_pages((unsigned long)pool->slots, get_order(slots_size)); - swiotlb_free_tlb(pool->vaddr, tlb_size); + swiotlb_free_tlb(pool->vaddr, tlb_size, pool->unencrypted); kfree(pool); } =20 @@ -1232,6 +1255,7 @@ static int swiotlb_find_slots(struct device *dev, phy= s_addr_t orig_addr, nslabs =3D nr_slots(alloc_size); phys_limit =3D min_not_zero(*dev->dma_mask, dev->bus_dma_limit); pool =3D swiotlb_alloc_pool(dev, nslabs, nslabs, 1, phys_limit, + mem->unencrypted ? DMA_ATTR_CC_SHARED : 0, GFP_NOWAIT); if (!pool) return -1; @@ -1394,6 +1418,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev= , phys_addr_t orig_addr, enum dma_data_direction dir, unsigned long attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; + bool require_decrypted =3D false; unsigned int offset; struct io_tlb_pool *pool; unsigned int i; @@ -1411,6 +1436,16 @@ phys_addr_t swiotlb_tbl_map_single(struct device *de= v, phys_addr_t orig_addr, if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) pr_warn_once("Memory encryption is active and system is using DMA bounce= buffers\n"); =20 + /* + * if we are trying to swiotlb map a decrypted paddr or the paddr is encr= ypted + * but the device is forcing decryption, use decrypted io_tlb_mem + */ + if ((attrs & DMA_ATTR_CC_SHARED) || force_dma_unencrypted(dev)) + require_decrypted =3D true; + + if (require_decrypted !=3D mem->unencrypted) + return (phys_addr_t)DMA_MAPPING_ERROR; + /* * The default swiotlb memory pool is allocated with PAGE_SIZE * alignment. If a mapping is requested with larger alignment, @@ -1608,8 +1643,14 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr= _t paddr, size_t size, if (swiotlb_addr =3D=3D (phys_addr_t)DMA_MAPPING_ERROR) return DMA_MAPPING_ERROR; =20 - /* Ensure that the address returned is DMA'ble */ - dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + /* + * Use the allocated io_tlb_mem encryption type to determine dma addr. + */ + if (dev->dma_io_tlb_mem->unencrypted) + dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); + else + dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, @@ -1773,7 +1814,8 @@ static inline void swiotlb_create_debugfs_files(struc= t io_tlb_mem *mem, =20 #ifdef CONFIG_DMA_RESTRICTED_POOL =20 -struct page *swiotlb_alloc(struct device *dev, size_t size) +struct page *swiotlb_alloc(struct device *dev, size_t size, + unsigned long attrs) { struct io_tlb_mem *mem =3D dev->dma_io_tlb_mem; struct io_tlb_pool *pool; @@ -1784,6 +1826,9 @@ struct page *swiotlb_alloc(struct device *dev, size_t= size) if (!mem) return NULL; =20 + if (mem->unencrypted !=3D !!(attrs & DMA_ATTR_CC_SHARED)) + return NULL; + align =3D (1 << (get_order(size) + PAGE_SHIFT)) - 1; index =3D swiotlb_find_slots(dev, 0, size, align, &pool); if (index =3D=3D -1) @@ -1853,9 +1898,18 @@ static int rmem_swiotlb_device_init(struct reserved_= mem *rmem, kfree(mem); return -ENOMEM; } + /* + * if platform supports memory encryption, + * restricted mem pool is decrypted by default + */ + if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) { + mem->unencrypted =3D true; + set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), + rmem->size >> PAGE_SHIFT); + } else { + mem->unencrypted =3D false; + } =20 - set_memory_decrypted((unsigned long)phys_to_virt(rmem->base), - rmem->size >> PAGE_SHIFT); swiotlb_init_io_tlb_pool(pool, rmem->base, nslabs, false, nareas); mem->force_bounce =3D true; --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 568EA3A16BC; Tue, 12 May 2026 09:05:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576730; cv=none; b=Cr6nGnfkXd/QiHDgtPgRCJROpOh+Fe2eMPol0EDKCOgKcv6OWQxU5J6WhklzpkjgPnsu0iqvSTB+f6d1l2NpzYlnzpXAEziU7YVk4ZVeKI0LKBmGjDhJQWRxjnpdd0a+C6rtPo8s5NkJb2uMMCsz+HkTL7BzluFl7Vx+YnuvSBY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576730; c=relaxed/simple; bh=9Mg/60FkAPiuc5fK1clRWn/J5QY6mHmOM4Gl+OW5I34=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=s40INk5RPVlJVEWPZ3dO4bQ3UUQ1T9ghWTj8/0QRp4AkMGqqoiM2n6stUoGMVFQ+YqgjNJQnNtFtCGzppjoh+4wDQWao0A2ep6ITnqR8TC9PWrbHaDyfGs73wh3wVfzGRtNnCd3KoHciL8O3cFgFaRupALXt3HS1XS4bagg4I+U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PshDt4XM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PshDt4XM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 907ECC2BCC7; Tue, 12 May 2026 09:05:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576729; bh=9Mg/60FkAPiuc5fK1clRWn/J5QY6mHmOM4Gl+OW5I34=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PshDt4XM1mvyyVNCfQLjWwIf61GnnlnERZolrSawcYkDpIfioObQLoAfIjMw/7M9Q YTz8/cGsgsVdmSVFE/GDDmGWLQM5F4mbClisOqcvomviQfO7wGGvr66P5s7Yiwte+5 fybLHXIDwBOPobE+QScCdJAI7IZ0kpk0ywCc2LZ19NFx0yfONadhppdaHsCn2eAjh8 GG2JzF/vxM9xMkbcFyz8zC2oyR5p+10XySzCrgJhMLuYJFT1/jmjORHauK5wfh1zjO /VYUPHV3+vGyJTDhUvmUU0I3RSZ8c0/PMsyqyLxd7jCDFGv0lm7o8rsAl39LxXz082 XDhYX9Ue6FRQA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 05/13] dma-mapping: make dma_pgprot() honor DMA_ATTR_CC_SHARED Date: Tue, 12 May 2026 14:34:00 +0530 Message-ID: <20260512090408.794195-6-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold encrypted/decrypted pgprot selection into dma_pgprot() so callers do not need to adjust the page protection separately. Update dma_pgprot() to apply pgprot_decrypted() when DMA_ATTR_CC_SHARED is set and pgprot_encrypted() otherwise Convert the dma-direct allocation and mmap paths to pass DMA_ATTR_CC_SHARED instead of open-coding force_dma_unencrypted() handling around dma_pgprot(). Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 8 +++----- kernel/dma/mapping.c | 16 ++++++++++++---- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 97ae4fa10521..ac315dd046c4 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -298,9 +298,6 @@ void *dma_direct_alloc(struct device *dev, size_t size, if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 - if (force_dma_unencrypted(dev)) - prot =3D pgprot_decrypted(prot); - /* remove any dirty cache lines on the kernel alias */ arch_dma_prep_coherent(page, size); =20 @@ -603,9 +600,10 @@ int dma_direct_mmap(struct device *dev, struct vm_area= _struct *vma, unsigned long pfn =3D PHYS_PFN(dma_to_phys(dev, dma_addr)); int ret =3D -ENXIO; =20 - vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); if (force_dma_unencrypted(dev)) - vma->vm_page_prot =3D pgprot_decrypted(vma->vm_page_prot); + attrs |=3D DMA_ATTR_CC_SHARED; + + vma->vm_page_prot =3D dma_pgprot(dev, vma->vm_page_prot, attrs); =20 if (dma_mmap_from_dev_coherent(dev, vma, cpu_addr, size, &ret)) return ret; diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 23ed8eb9233e..44f715f3aa2d 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -543,13 +543,21 @@ EXPORT_SYMBOL(dma_get_sgtable_attrs); */ pgprot_t dma_pgprot(struct device *dev, pgprot_t prot, unsigned long attrs) { + pgprot_t dma_prot; + if (dev_is_dma_coherent(dev)) - return prot; + dma_prot =3D prot; #ifdef CONFIG_ARCH_HAS_DMA_WRITE_COMBINE - if (attrs & DMA_ATTR_WRITE_COMBINE) - return pgprot_writecombine(prot); + else if (attrs & DMA_ATTR_WRITE_COMBINE) + dma_prot =3D pgprot_writecombine(prot); #endif - return pgprot_dmacoherent(prot); + else + dma_prot =3D pgprot_dmacoherent(prot); + + if (attrs & DMA_ATTR_CC_SHARED) + return pgprot_decrypted(dma_prot); + else + return pgprot_encrypted(dma_prot); } #endif /* CONFIG_MMU */ =20 --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDADD3A1E6D; Tue, 12 May 2026 09:05:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576743; cv=none; b=p0U54BPiPq3OCMv7uAKzj2BoLW5gvlPwqQ+JOf8hEGw4688HoVP1TXuxWlz1xyJ5ng2yus0V+XwzcYUmTMDXgDKjL7zCB31Ye8jRqbuZtUDfmqeWwZpoy3S6ZW5FPMPWZ6m1pOCG1GZKRGidSfT8RWIWrHYJBZOBtQFFzP5mkKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576743; c=relaxed/simple; bh=vPHU0CUqjPrs0bZjgUyqtv1tValYQko8vPrLTx/4JMg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t9qPHwZD16r2zefS2IMuzSAXr58nw9CHaQ4yICS7ggjp1+6kATYKoKAriq5a7fhtlUt8rUSovz427FOAcvg2LVACQ80OERQAeBzbqk8WdQXiPEEKd7lispQidgoDBZc5m4WdW9FmXkyF7fCcGsTXaVaR9D3+i/5wLngHwXmWdZs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KsG60OzG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KsG60OzG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7EB18C2BCB0; Tue, 12 May 2026 09:05:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576742; bh=vPHU0CUqjPrs0bZjgUyqtv1tValYQko8vPrLTx/4JMg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KsG60OzGz/+9kKCcBfATi/vPnDwLfvzwQ0HksoC7AxmNtRct3Hck1Lu+OfdMifRIf VsNcOtnZ4S4ZRrvU1n2q5TFFJRrpXFMsifTDvgJWfy7RIUy6otlmFmntvbJzahcAZA 90YfSX9NYbILfiIdcA4CY4hvJ1s2+ay85Ec8PMVcTTxqhaXER2IFGVYAeJ38lNcSlz nR4eqlr8lPI+3dTkfZvkFTf598tyl2rT8iOs+blBJVylB4AavhZjoVPi/NhDiSJnsT H6MJAqq7KXFu0IY9wfDEuORqHajjOqiQbXM22dolWPLqdT2fXopJLCj9yh9QIJt5lI pVpWS0L4Hy+0Q== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 06/13] dma-direct: pass attrs to dma_capable() for DMA_ATTR_CC_SHARED checks Date: Tue, 12 May 2026 14:34:01 +0530 Message-ID: <20260512090408.794195-7-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_capable() about DMA_ATTR_CC_SHARED so the capability check can reject encrypted DMA addresses for devices that require unencrypted/shared DMA. Also propagate DMA_ATTR_CC_SHARED in swiotlb_map() when the selected SWIOTLB pool is decrypted so the capability check sees the correct DMA address attribute. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- arch/x86/kernel/amd_gart_64.c | 30 ++++++++++++++++-------------- drivers/xen/swiotlb-xen.c | 10 +++++++--- include/linux/dma-direct.h | 9 ++++++++- kernel/dma/direct.h | 6 +++--- kernel/dma/swiotlb.c | 8 +++++--- 5 files changed, 39 insertions(+), 24 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index e8000a56732e..b5f1f031d45b 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -180,22 +180,23 @@ static void iommu_full(struct device *dev, size_t siz= e, int dir) } =20 static inline int -need_iommu(struct device *dev, unsigned long addr, size_t size) +need_iommu(struct device *dev, unsigned long addr, size_t size, unsigned l= ong attrs) { - return force_iommu || !dma_capable(dev, addr, size, true); + return force_iommu || !dma_capable(dev, addr, size, true, attrs); } =20 static inline int -nonforced_iommu(struct device *dev, unsigned long addr, size_t size) +nonforced_iommu(struct device *dev, unsigned long addr, size_t size, + unsigned long attrs) { - return !dma_capable(dev, addr, size, true); + return !dma_capable(dev, addr, size, true, attrs); } =20 /* Map a single continuous physical area into the IOMMU. * Caller needs to check if the iommu is needed and flush. */ static dma_addr_t dma_map_area(struct device *dev, dma_addr_t phys_mem, - size_t size, int dir, unsigned long align_mask) + size_t size, int dir, unsigned long align_mask, unsigned long attrs) { unsigned long npages =3D iommu_num_pages(phys_mem, size, PAGE_SIZE); unsigned long iommu_page; @@ -206,7 +207,7 @@ static dma_addr_t dma_map_area(struct device *dev, dma_= addr_t phys_mem, =20 iommu_page =3D alloc_iommu(dev, npages, align_mask); if (iommu_page =3D=3D -1) { - if (!nonforced_iommu(dev, phys_mem, size)) + if (!nonforced_iommu(dev, phys_mem, size, attrs)) return phys_mem; if (panic_on_overflow) panic("dma_map_area overflow %lu bytes\n", size); @@ -231,10 +232,10 @@ static dma_addr_t gart_map_phys(struct device *dev, p= hys_addr_t paddr, if (unlikely(attrs & DMA_ATTR_MMIO)) return DMA_MAPPING_ERROR; =20 - if (!need_iommu(dev, paddr, size)) + if (!need_iommu(dev, paddr, size, attrs)) return paddr; =20 - bus =3D dma_map_area(dev, paddr, size, dir, 0); + bus =3D dma_map_area(dev, paddr, size, dir, 0, attrs); flush_gart(); =20 return bus; @@ -289,7 +290,7 @@ static void gart_unmap_sg(struct device *dev, struct sc= atterlist *sg, int nents, =20 /* Fallback for dma_map_sg in case of overflow */ static int dma_map_sg_nonforce(struct device *dev, struct scatterlist *sg, - int nents, int dir) + int nents, int dir, unsigned long attrs) { struct scatterlist *s; int i; @@ -301,8 +302,8 @@ static int dma_map_sg_nonforce(struct device *dev, stru= ct scatterlist *sg, for_each_sg(sg, s, nents, i) { unsigned long addr =3D sg_phys(s); =20 - if (nonforced_iommu(dev, addr, s->length)) { - addr =3D dma_map_area(dev, addr, s->length, dir, 0); + if (nonforced_iommu(dev, addr, s->length, attrs)) { + addr =3D dma_map_area(dev, addr, s->length, dir, 0, attrs); if (addr =3D=3D DMA_MAPPING_ERROR) { if (i > 0) gart_unmap_sg(dev, sg, i, dir, 0); @@ -401,7 +402,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, s->dma_address =3D addr; BUG_ON(s->length =3D=3D 0); =20 - nextneed =3D need_iommu(dev, addr, s->length); + nextneed =3D need_iommu(dev, addr, s->length, attrs); =20 /* Handle the previous not yet processed entries */ if (i > start) { @@ -449,7 +450,7 @@ static int gart_map_sg(struct device *dev, struct scatt= erlist *sg, int nents, =20 /* When it was forced or merged try again in a dumb way */ if (force_iommu || iommu_merge) { - out =3D dma_map_sg_nonforce(dev, sg, nents, dir); + out =3D dma_map_sg_nonforce(dev, sg, nents, dir, attrs); if (out > 0) return out; } @@ -473,7 +474,8 @@ gart_alloc_coherent(struct device *dev, size_t size, dm= a_addr_t *dma_addr, return vaddr; =20 *dma_addr =3D dma_map_area(dev, virt_to_phys(vaddr), size, - DMA_BIDIRECTIONAL, (1UL << get_order(size)) - 1); + DMA_BIDIRECTIONAL, + (1UL << get_order(size)) - 1, attrs); flush_gart(); if (unlikely(*dma_addr =3D=3D DMA_MAPPING_ERROR)) goto out_free; diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index 2cbf2b588f5b..fa6734461d4c 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -212,7 +212,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, BUG_ON(dir =3D=3D DMA_NONE); =20 if (attrs & DMA_ATTR_MMIO) { - if (unlikely(!dma_capable(dev, phys, size, false))) { + if (unlikely(!dma_capable(dev, phys, size, false, attrs))) { dev_err_once( dev, "DMA addr %pa+%zu overflow (mask %llx, bus limit %llx).\n", @@ -231,7 +231,7 @@ static dma_addr_t xen_swiotlb_map_phys(struct device *d= ev, phys_addr_t phys, * we can safely return the device addr and not worry about bounce * buffering it. */ - if (dma_capable(dev, dev_addr, size, true) && + if (dma_capable(dev, dev_addr, size, true, attrs) && !dma_kmalloc_needs_bounce(dev, size, dir) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && @@ -248,12 +248,16 @@ static dma_addr_t xen_swiotlb_map_phys(struct device = *dev, phys_addr_t phys, return DMA_MAPPING_ERROR; =20 phys =3D map; + /* This always return an encrypted addr */ dev_addr =3D xen_phys_to_dma(dev, map); =20 + if (WARN_ON(dev->dma_io_tlb_mem->unencrypted)) + attrs |=3D DMA_ATTR_CC_SHARED; + /* * Ensure that the address returned is DMA'ble */ - if (unlikely(!dma_capable(dev, dev_addr, size, true))) { + if (unlikely(!dma_capable(dev, dev_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, map, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, map)); diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 94fad4e7c11e..9dbe198b2c4a 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -135,12 +135,19 @@ static inline bool force_dma_unencrypted(struct devic= e *dev) #endif /* CONFIG_ARCH_HAS_FORCE_DMA_UNENCRYPTED */ =20 static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t= size, - bool is_ram) + bool is_ram, unsigned long attrs) { dma_addr_t end =3D addr + size - 1; =20 if (addr =3D=3D DMA_MAPPING_ERROR) return false; + /* + * if phys addr attribute is encrypted but the + * device is forcing an unencrypted dma addr + */ + if (!(attrs & DMA_ATTR_CC_SHARED) && force_dma_unencrypted(dev)) + return false; + if (is_ram && !IS_ENABLED(CONFIG_ARCH_DMA_ADDR_T_64BIT) && min(addr, end) < phys_to_dma(dev, PFN_PHYS(min_low_pfn))) return false; diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index 7140c208c123..e05dc7649366 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -101,15 +101,15 @@ static inline dma_addr_t dma_direct_map_phys(struct d= evice *dev, =20 if (attrs & DMA_ATTR_MMIO) { dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else if (attrs & DMA_ATTR_CC_SHARED) { dma_addr =3D phys_to_dma_unencrypted(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, false))) + if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; } else { dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true)) || + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || dma_kmalloc_needs_bounce(dev, size, dir)) { if (is_swiotlb_active(dev) && !(attrs & DMA_ATTR_REQUIRE_COHERENT)) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 065663be282c..9f87ebe42797 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1646,12 +1646,14 @@ dma_addr_t swiotlb_map(struct device *dev, phys_add= r_t paddr, size_t size, /* * Use the allocated io_tlb_mem encryption type to determine dma addr. */ - if (dev->dma_io_tlb_mem->unencrypted) + if (dev->dma_io_tlb_mem->unencrypted) { dma_addr =3D phys_to_dma_unencrypted(dev, swiotlb_addr); - else + attrs |=3D DMA_ATTR_CC_SHARED; + } else { dma_addr =3D phys_to_dma_encrypted(dev, swiotlb_addr); + } =20 - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs))) { __swiotlb_tbl_unmap_single(dev, swiotlb_addr, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC, swiotlb_find_pool(dev, swiotlb_addr)); --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03C623B19AE; Tue, 12 May 2026 09:05:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576754; cv=none; b=SvtzVrU4+1qn8A9YwxoInHVz/Fpecfw3rAeqYE5qNTTIqmljL5eLhvJ8WPBq9lyhU7iHLevmtpcMOKTg7xKJWWQfdTXfc2SnBPzzGsxNni2s3orE6LiMs27IoCW3nHxG8LERBcOJngNz0v5Y0UTkCtpuNlOh/IkCsA2Aexc/LeA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576754; c=relaxed/simple; bh=djoVlq64riIKyUlBW+OecFjb5IHhvE+yjLiRQdlK7Xo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jz7nXMoUWNz/xD1oeCVx9FVoJA8XXwxqzPjsbBl9tT7p0Ln+j+BEPIocFwsbcUpUqdhwps1/rC3ik7yYSJ+T3z6uGdhBF5tqsMaIKg5lp87RXFE99sOUKgdvTes3qx+zoN5BnATKam0ggc6beyxCYYcUQQyd2kbiDE+plcFfQ9g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fuh+zcQs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fuh+zcQs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0682DC2BCC7; Tue, 12 May 2026 09:05:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576753; bh=djoVlq64riIKyUlBW+OecFjb5IHhvE+yjLiRQdlK7Xo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fuh+zcQsaxfk2wQhFIIRp8/7gcaQLxZWOwrFiHLLxW5CnQkq0T/KSl1gp/vKApyP9 R6DmxYhpxc2zRk2urFfPqZWxXnjmoH0MofSbKYSmgdbZB4P25Neb+k+78sIifLEU6e 4w9X+UpIjBdI03TEPGNu6hiiYU2LzruavDT7RqYhhsafudue7rWC/M7yyh/AkAmu+A arg0xANxjpzXi8B9p5TT0Vu/fc7RmGkaEP3LnZP70W4DH6ZKE4PHX40DaupNv2eJe6 3fBtIwklzh6LmOj/oEur30apoK6OLSNqPw4q32I6AUAghRWN6c98HJZ1M6PzXenUfZ GnfiBePVDQ7BA== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 07/13] dma-direct: make dma_direct_map_phys() honor DMA_ATTR_CC_SHARED Date: Tue, 12 May 2026 14:34:02 +0530 Message-ID: <20260512090408.794195-8-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Teach dma_direct_map_phys() to select the DMA address encoding based on DMA_ATTR_CC_SHARED. Use phys_to_dma_unencrypted() for decrypted mappings and phys_to_dma_encrypted() otherwise. If a device requires unencrypted DMA but the source physical address is still encrypted, force the mapping through swiotlb so the DMA address and backing memory attributes remain consistent. Update the arm64, x86, s390 and powerpc secure-guest setup to not use swiotlb force option Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- Changes from v3: * Handle DMA_ATTR_MMIO --- arch/arm64/mm/init.c | 4 +-- arch/powerpc/platforms/pseries/svm.c | 2 +- arch/s390/mm/init.c | 2 +- arch/x86/kernel/pci-dma.c | 4 +-- kernel/dma/direct.c | 4 ++- kernel/dma/direct.h | 38 +++++++++++++--------------- 6 files changed, 24 insertions(+), 30 deletions(-) diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index 97987f850a33..acf67c7064db 100644 --- a/arch/arm64/mm/init.c +++ b/arch/arm64/mm/init.c @@ -338,10 +338,8 @@ void __init arch_mm_preinit(void) unsigned int flags =3D SWIOTLB_VERBOSE; bool swiotlb =3D max_pfn > PFN_DOWN(arm64_dma_phys_limit); =20 - if (is_realm_world()) { + if (is_realm_world()) swiotlb =3D true; - flags |=3D SWIOTLB_FORCE; - } =20 if (IS_ENABLED(CONFIG_DMA_BOUNCE_UNALIGNED_KMALLOC) && !swiotlb) { /* diff --git a/arch/powerpc/platforms/pseries/svm.c b/arch/powerpc/platforms/= pseries/svm.c index 384c9dc1899a..7a403dbd35ee 100644 --- a/arch/powerpc/platforms/pseries/svm.c +++ b/arch/powerpc/platforms/pseries/svm.c @@ -29,7 +29,7 @@ static int __init init_svm(void) * need to use the SWIOTLB buffer for DMA even if dma_capable() says * otherwise. */ - ppc_swiotlb_flags |=3D SWIOTLB_ANY | SWIOTLB_FORCE; + ppc_swiotlb_flags |=3D SWIOTLB_ANY; =20 /* Share the SWIOTLB buffer with the host. */ swiotlb_update_mem_attributes(); diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index 1f72efc2a579..843dbd445124 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -149,7 +149,7 @@ static void __init pv_init(void) virtio_set_mem_acc_cb(virtio_require_restricted_mem_acc); =20 /* make sure bounce buffers are shared */ - swiotlb_init(true, SWIOTLB_FORCE | SWIOTLB_VERBOSE); + swiotlb_init(true, SWIOTLB_VERBOSE); swiotlb_update_mem_attributes(); } =20 diff --git a/arch/x86/kernel/pci-dma.c b/arch/x86/kernel/pci-dma.c index 6267363e0189..75cf8f6ae8cd 100644 --- a/arch/x86/kernel/pci-dma.c +++ b/arch/x86/kernel/pci-dma.c @@ -59,10 +59,8 @@ static void __init pci_swiotlb_detect(void) * bounce buffers as the hypervisor can't access arbitrary VM memory * that is not explicitly shared with it. */ - if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) { + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) x86_swiotlb_enable =3D true; - x86_swiotlb_flags |=3D SWIOTLB_FORCE; - } } #else static inline void __init pci_swiotlb_detect(void) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index ac315dd046c4..5aaa813c5509 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -691,8 +691,10 @@ size_t dma_direct_max_mapping_size(struct device *dev) { /* If SWIOTLB is active, use its maximum mapping size */ if (is_swiotlb_active(dev) && - (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev))) + (dma_addressing_limited(dev) || is_swiotlb_force_bounce(dev) || + force_dma_unencrypted(dev))) return swiotlb_max_mapping_size(dev); + return SIZE_MAX; } =20 diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index e05dc7649366..4e35264ab6f8 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -89,36 +89,32 @@ static inline dma_addr_t dma_direct_map_phys(struct dev= ice *dev, dma_addr_t dma_addr; =20 if (is_swiotlb_force_bounce(dev)) { - if (!(attrs & DMA_ATTR_CC_SHARED)) { - if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) - return DMA_MAPPING_ERROR; + if (attrs & (DMA_ATTR_MMIO | DMA_ATTR_REQUIRE_COHERENT)) + return DMA_MAPPING_ERROR; =20 - return swiotlb_map(dev, phys, size, dir, attrs); - } - } else if (attrs & DMA_ATTR_CC_SHARED) { - return DMA_MAPPING_ERROR; + return swiotlb_map(dev, phys, size, dir, attrs); } =20 - if (attrs & DMA_ATTR_MMIO) { - dma_addr =3D phys; - if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) - goto err_overflow; - } else if (attrs & DMA_ATTR_CC_SHARED) { + if (attrs & DMA_ATTR_CC_SHARED) dma_addr =3D phys_to_dma_unencrypted(dev, phys); + else + dma_addr =3D phys_to_dma_encrypted(dev, phys); + + if (attrs & DMA_ATTR_MMIO) { if (unlikely(!dma_capable(dev, dma_addr, size, false, attrs))) goto err_overflow; - } else { - dma_addr =3D phys_to_dma(dev, phys); - if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || - dma_kmalloc_needs_bounce(dev, size, dir)) { - if (is_swiotlb_active(dev) && - !(attrs & DMA_ATTR_REQUIRE_COHERENT)) - return swiotlb_map(dev, phys, size, dir, attrs); + goto dma_mapped; + } =20 - goto err_overflow; - } + if (unlikely(!dma_capable(dev, dma_addr, size, true, attrs)) || + dma_kmalloc_needs_bounce(dev, size, dir)) { + if (is_swiotlb_active(dev) && + !(attrs & DMA_ATTR_REQUIRE_COHERENT)) + return swiotlb_map(dev, phys, size, dir, attrs); + goto err_overflow; } =20 +dma_mapped: if (!dev_is_dma_coherent(dev) && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19A873B6350; Tue, 12 May 2026 09:06:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576766; cv=none; b=e7Twl01MhKy/CLNvj0P/CWNIItwA03rXix9K+Nh0BsNt//6IqZ6zZm0ztDhFKQQwgdRnUQZ9VqcPoOTR4buBFy1rAxUsJFqVV5ztRy32jrsUMJJD/N1K2SWR8F41AOkOSnbn/Wwi6ix3Ey443GUSX9GKn4sRaO64YhU//6uwtmg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576766; c=relaxed/simple; bh=16IbbDmBI894AMGJNFhh69uBka8eAlp5FTjDwJZdlnM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N9N//SoXCIoFk9Gb9TDVeYQMZpkd/hWci5c+djk7zYb3GsIQoHZw+5CymFFYuvXWf/uYLcHPBfEOZ/RSLsmVayI3RlQ9Bqt2v6NgYTHooABP5ZAH1BxsTGdI3AeFbyYol5fBS084tT3EO46+VzSAHLhej3xFcb4piDOlWTcyEiM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nVAK40Lf; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nVAK40Lf" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6C74EC2BCF5; Tue, 12 May 2026 09:05:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576765; bh=16IbbDmBI894AMGJNFhh69uBka8eAlp5FTjDwJZdlnM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nVAK40LfYZrBtDvRtjA7tMJs0cMnULnsZSnRoWepqQHZmIqmuO9je8PQQVCyNkkNg Rc+KO9UbCW2vOCU+lW7aFNJLcd/+Sf/EeaMZhPN9SJXYOtncLxjbWoeYv2YtTFEoJZ 2APIaWChN5x4ArYIiPZjEy1HEkXcsFQ2nTfz6nFccGEeC66p0So8BhQr2Yen03LZLP NRrZ4OT0eqMTdqRr6bc/hMMZl4e2PRbobg2h/udFtsQ45xPYs0LqE1tUFgXuKqgnoX KgDFxMAbVhQoS4MVXkW9Rh8gLamKNOy++DEQYXPcgIddKiT5EDUMlaaRqAorqYLBhO LK9KOABLNjAsg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 08/13] dma-direct: set decrypted flag for remapped DMA allocations Date: Tue, 12 May 2026 14:34:03 +0530 Message-ID: <20260512090408.794195-9-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Devices that are DMA non-coherent and require a remap were skipping dma_set_decrypted(), leaving DMA buffers encrypted even when the device requires unencrypted access. Move the call after the if (remap) branch so that both the direct and remapped allocation paths correctly mark the allocation as decrypted (or fail cleanly) before use. Fix dma_direct_alloc() and dma_direct_free() to apply set_memory_*() to the linear-map alias of the backing pages instead of the remapped CPU address. Also disallow highmem pages for DMA_ATTR_CC_SHARED, because highmem buffers do not provide a usable linear-map address. Fixes: f3c962226dbe ("dma-direct: clean up the remapping checks in dma_dire= ct_alloc") Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 56 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 11 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 5aaa813c5509..f5da6e992d83 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -204,6 +204,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, { bool remap =3D false, set_uncached =3D false; bool mark_mem_decrypt =3D false; + bool allow_highmem =3D true; struct page *page; void *ret; =20 @@ -222,6 +223,15 @@ void *dma_direct_alloc(struct device *dev, size_t size, mark_mem_decrypt =3D true; } =20 + if (attrs & DMA_ATTR_CC_SHARED) + /* + * Unencrypted/shared DMA requires a linear-mapped buffer + * address to look up the PFN and set architecture-required PFN + * attributes. This is not possible with HighMem. Avoid HighMem + * allocation. + */ + allow_highmem =3D false; + size =3D PAGE_ALIGN(size); if (attrs & DMA_ATTR_NO_WARN) gfp |=3D __GFP_NOWARN; @@ -280,7 +290,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, } =20 /* we always manually zero the memory once we are done */ - page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); + page =3D __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, allow_hig= hmem); if (!page) return NULL; =20 @@ -295,6 +305,14 @@ void *dma_direct_alloc(struct device *dev, size_t size, set_uncached =3D false; } =20 + if (mark_mem_decrypt) { + void *lm_addr; + + lm_addr =3D page_address(page); + if (set_memory_decrypted((unsigned long)lm_addr, PFN_UP(size))) + goto out_leak_pages; + } + if (remap) { pgprot_t prot =3D dma_pgprot(dev, PAGE_KERNEL, attrs); =20 @@ -305,31 +323,39 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, ret =3D dma_common_contiguous_remap(page, size, prot, __builtin_return_address(0)); if (!ret) - goto out_free_pages; + goto out_encrypt_pages; } else { ret =3D page_address(page); - if (mark_mem_decrypt && dma_set_decrypted(dev, ret, size)) - goto out_leak_pages; } =20 memset(ret, 0, size); =20 if (set_uncached) { + void *uncached_cpu_addr; + arch_dma_prep_coherent(page, size); - ret =3D arch_dma_set_uncached(ret, size); - if (IS_ERR(ret)) - goto out_encrypt_pages; + uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + if (IS_ERR(uncached_cpu_addr)) + goto out_free_remap_pages; + ret =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); return ret; =20 + +out_free_remap_pages: + if (remap) + dma_common_free_remap(ret, size); + out_encrypt_pages: - if (mark_mem_decrypt && dma_set_encrypted(dev, page_address(page), size)) - return NULL; -out_free_pages: + if (mark_mem_decrypt && + dma_set_encrypted(dev, page_address(page), size)) + goto out_leak_pages; + __dma_direct_free_pages(dev, page, size); return NULL; + out_leak_pages: return NULL; } @@ -384,8 +410,16 @@ void dma_direct_free(struct device *dev, size_t size, } else { if (IS_ENABLED(CONFIG_ARCH_HAS_DMA_CLEAR_UNCACHED)) arch_dma_clear_uncached(cpu_addr, size); - if (mark_mem_encrypted && dma_set_encrypted(dev, cpu_addr, size)) + } + + if (mark_mem_encrypted) { + void *lm_addr; + + lm_addr =3D phys_to_virt(dma_to_phys(dev, dma_addr)); + if (set_memory_encrypted((unsigned long)lm_addr, PFN_UP(size))) { + pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); return; + } } =20 __dma_direct_free_pages(dev, dma_direct_to_page(dev, dma_addr), size); --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AA383BA22C; Tue, 12 May 2026 09:06:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576777; cv=none; b=qMKu4G+2h7jm5V5m8IZOLJA9BECvdYDfVxkPJ2fBnQXd1Y0s89P81gkGJEDGJ2vpJpqUkDmrIvr/6pwl5HtkxXdmLScsNux/fNE2EMAfVetYWdhpsl6D3eRSbyX4H0xWG9GafiaSFzPb3ZzzluA5xP6QqHbDpOONEfzSzwsUJOI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576777; c=relaxed/simple; bh=Q+boUmp5z1kUVbYWMstIjuxd/izdJGdKK16SPwpnnDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WOE6NMyEJ8rGKMuxzXoDTHxdVgoVc5ItI7ephPjl29VRxNs3bEnSRoIWwvXOyRkesOUk5qjNxBJrVf7b4+/qup9cHvh7QC4ZNI3ks6NK1n9Xxc5DuGGbt/A+u5d7cp6MXRRgxkBGI50/DddjODFI+AaCPNoJNw26yfthY5e5wxE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nuQjFhmv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nuQjFhmv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C608C2BCB0; Tue, 12 May 2026 09:06:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576777; bh=Q+boUmp5z1kUVbYWMstIjuxd/izdJGdKK16SPwpnnDo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nuQjFhmvoLcL41GEj42JtD4I7DEN67oML0MlCcrk2S1rMkLr5JeJ2PT3R0eGsmfIQ GMQ1CC5+xev+2epOEXrCN6tMG58pcduD4Z3cbaZS96o/SRm2Aog8gjuXHOmXUdz109 Bbk4SMLLQUbd6VcsuviVsfF8WrWdF7OIdCjoQxUFc70B2hbwHaPEeCWQWz+krXJy5J 4TSQStB+FP7ScGbhVA34Byj+NEGmXSzV1kfjIq3wajiJHZjocepy7sQCqerEFHPXPL fJefKGR8G7UgYxesmWnRKkwTTiIH6r7oWjssvO48G1gceUeFMuSLkaaFHRFXCSgcmN M7lc3XBDFIvng== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 09/13] dma-direct: select DMA address encoding from DMA_ATTR_CC_SHARED Date: Tue, 12 May 2026 14:34:04 +0530 Message-ID: <20260512090408.794195-10-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make the dma-direct helpers derive the DMA address encoding from DMA_ATTR_CC_SHARED instead of implicitly relying on force_dma_unencrypted() inside phys_to_dma_direct() Pass an explicit unencrypted/decrypted state into phys_to_dma_direct(), make the alloc paths return DMA addresses that match the requested buffer encryption state. Also only call dma_set_decrypted() DMA_ATTR_CC_SHARED is actually set. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 48 ++++++++++++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f5da6e992d83..1e9f9ff7b9d3 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -24,11 +24,11 @@ u64 zone_dma_limit __ro_after_init =3D DMA_BIT_MASK(24); =20 static inline dma_addr_t phys_to_dma_direct(struct device *dev, - phys_addr_t phys) + phys_addr_t phys, bool unencrypted) { - if (force_dma_unencrypted(dev)) + if (unencrypted) return phys_to_dma_unencrypted(dev, phys); - return phys_to_dma(dev, phys); + return phys_to_dma_encrypted(dev, phys); } =20 static inline struct page *dma_direct_to_page(struct device *dev, @@ -39,8 +39,9 @@ static inline struct page *dma_direct_to_page(struct devi= ce *dev, =20 u64 dma_direct_get_required_mask(struct device *dev) { + bool require_decrypted =3D force_dma_unencrypted(dev); phys_addr_t phys =3D (phys_addr_t)(max_pfn - 1) << PAGE_SHIFT; - u64 max_dma =3D phys_to_dma_direct(dev, phys); + u64 max_dma =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 return (1ULL << (fls64(max_dma) - 1)) * 2 - 1; } @@ -69,7 +70,8 @@ static gfp_t dma_direct_optimal_gfp_mask(struct device *d= ev, u64 *phys_limit) =20 bool dma_coherent_ok(struct device *dev, phys_addr_t phys, size_t size) { - dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys); + bool require_decrypted =3D force_dma_unencrypted(dev); + dma_addr_t dma_addr =3D phys_to_dma_direct(dev, phys, require_decrypted); =20 if (dma_addr =3D=3D DMA_MAPPING_ERROR) return false; @@ -79,17 +81,18 @@ bool dma_coherent_ok(struct device *dev, phys_addr_t ph= ys, size_t size) =20 static int dma_set_decrypted(struct device *dev, void *vaddr, size_t size) { - if (!force_dma_unencrypted(dev)) - return 0; - return set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + int ret; + + ret =3D set_memory_decrypted((unsigned long)vaddr, PFN_UP(size)); + if (ret) + pr_warn_ratelimited("leaking DMA memory that can't be decrypted\n"); + return ret; } =20 static int dma_set_encrypted(struct device *dev, void *vaddr, size_t size) { int ret; =20 - if (!force_dma_unencrypted(dev)) - return 0; ret =3D set_memory_encrypted((unsigned long)vaddr, PFN_UP(size)); if (ret) pr_warn_ratelimited("leaking DMA memory that can't be re-encrypted\n"); @@ -177,7 +180,8 @@ static void *dma_direct_alloc_from_pool(struct device *= dev, size_t size, dma_coherent_ok); if (!page) return NULL; - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return ret; } =20 @@ -193,9 +197,11 @@ static void *dma_direct_alloc_no_mapping(struct device= *dev, size_t size, /* remove any dirty cache lines on the kernel alias */ if (!PageHighMem(page)) arch_dma_prep_coherent(page, size); - - /* return the page pointer as the opaque cookie */ - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + /* + * return the page pointer as the opaque cookie. + * Never used for unencrypted allocation + */ + *dma_handle =3D phys_to_dma_encrypted(dev, page_to_phys(page)); return page; } =20 @@ -340,7 +346,8 @@ void *dma_direct_alloc(struct device *dev, size_t size, ret =3D uncached_cpu_addr; } =20 - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return ret; =20 =20 @@ -457,11 +464,12 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, return NULL; =20 ret =3D page_address(page); - if (dma_set_decrypted(dev, ret, size)) + if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, ret, size)) goto out_leak_pages; setup_page: memset(ret, 0, size); - *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page)); + *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), + !!(attrs & DMA_ATTR_CC_SHARED)); return page; out_leak_pages: return NULL; @@ -471,8 +479,12 @@ void dma_direct_free_pages(struct device *dev, size_t = size, struct page *page, dma_addr_t dma_addr, enum dma_data_direction dir) { + /* + * if the device had requested for an unencrypted buffer, + * convert it to encrypted on free + */ + bool mark_mem_encrypted =3D force_dma_unencrypted(dev); void *vaddr =3D page_address(page); - bool mark_mem_encrypted =3D true; =20 /* If cpu_addr is not from an atomic pool, dma_free_from_pool() fails */ if (IS_ENABLED(CONFIG_DMA_COHERENT_POOL) && --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E02B36F900; Tue, 12 May 2026 09:06:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576789; cv=none; b=XS2QPbLwN6vQ2qQq82+MomoJi4M59cMzNKRxfirjulUKnSpoHnIHtlry8FAUuZpJykWRiIeFbS1gzdr6yGsempsTwebmT4qoSiCveCIoZcyH8/3qnmpqmnv+57ReBPFyYzQYRpGWiCkRMtZsy5zjou/xVYcaXT4m+pbpnqsEx7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576789; c=relaxed/simple; bh=veSEmK9Ug4szH3v0qgbFOODgpY5EJGxq5OFtSmQEZoQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EzYXPRcZcQ5l5rsqZOyOGbrBJynjGyuxp2z5fk5zWbvWAl1JtOurHWa+51BrTnX/iqj4n+W3ehAhMrUiPlzN8x6q429/gAzQCJMyT3eBwbmhyqTKt477nQcABslctb65ZDL2ENCTwPxXWUGvVVMZywrk6vrTdOaVn57tvn3BMQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=hfEmbaD1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="hfEmbaD1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C140BC2BCC7; Tue, 12 May 2026 09:06:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576788; bh=veSEmK9Ug4szH3v0qgbFOODgpY5EJGxq5OFtSmQEZoQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hfEmbaD1vx0OtPyt/2zEwcfXvzz2enHIUHpU/ddgBvzPmsG0FKkM8cGvnL6yBHpxk 4jHpHHoYShzlZH9rztYvxJBggVXmDlGdmaU2VTP1Twa68ti+ABZKBLBjOyGE2Glmva 4eLsOWHe3HEgbKtQeVTPtBwyxACwv0llrWmj024jJKEohKuYvakIwLhtsSCD/y7/Z+ 0Alxpd03yhdikbAyeOllS8BT/tFi2PamwbpCUvLLoPmv81EyPHdPMxORvnuulZP/8B v3ozOATeqYxIv/FKeuK5uxsHLgCPQeZgin3CqA4AOi9eOLu3iXFg7ZWbifu1AU0/gT cmtPif0ZHnFIw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 10/13] dma-pool: fix page leak in atomic_pool_expand() cleanup Date: Tue, 12 May 2026 14:34:05 +0530 Message-ID: <20260512090408.794195-11-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" atomic_pool_expand() frees the allocated pages from the remove_mapping error path only when CONFIG_DMA_DIRECT_REMAP is enabled. When CONFIG_DMA_DIRECT_REMAP is disabled, failures after page allocation, such as gen_pool_add_virt(), jump to remove_mapping and return without freeing the pages. Move __free_pages(page, order) out of the CONFIG_DMA_DIRECT_REMAP block so that cleanup paths always release the allocation. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/pool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/dma/pool.c b/kernel/dma/pool.c index 75f0eba48a23..5abd30c5119f 100644 --- a/kernel/dma/pool.c +++ b/kernel/dma/pool.c @@ -149,8 +149,8 @@ static int atomic_pool_expand(struct dma_gen_pool *dma_= pool, size_t pool_size, #ifdef CONFIG_DMA_DIRECT_REMAP dma_common_free_remap(addr, pool_size); free_page: - __free_pages(page, order); #endif + __free_pages(page, order); out: return ret; } --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFD3C3B6C1F; Tue, 12 May 2026 09:06:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576800; cv=none; b=jF7J0tG6UlZj4dOC7mJ9cLPMDhT4UBhRT7GXbZnaLAlFYDP5EYvkxtzA8penqjr/ms6BV9mayRqluu0QinnRaaA+fWGB7rbAf93mbvHM3sVQR5IifeS94aWNEEfV3DRTotP7ktdOOrOh3p/YBgfMYaRaCz8vPMS639ra+SJEFik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576800; c=relaxed/simple; bh=mTlzwGmKaksgYc3F/GyUj4XB9AG9GATAkfL4j43fQ40=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bANJKA6R8mPj0DAnRM39AW9ZBVfNybeR1QfSUBFA19tDfWodKuP2qpTRupai8sK/cLy0bYMLgWXCjuxUxY61L2tRcbTqTBNV148l0MY6OFL7Gnexnny+gFGhcNakvsc6IQo1bCrKSPmqUMOhDMK0tTuV68b9rrlqmF/N72tto20= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Gg9NhAp7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Gg9NhAp7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3AC84C2BCF7; Tue, 12 May 2026 09:06:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576799; bh=mTlzwGmKaksgYc3F/GyUj4XB9AG9GATAkfL4j43fQ40=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Gg9NhAp7QAL+VjZbnyz/0jlu7gA+E880rsW1LbQ4Tm+kjB/qLh83dBTqQCogFkMnk 6ldiTPnH1BZAHckzOmx/zFVXOvdywPpPf75pIgEpalHO5cqsgAyIFMRGo1Pp+77GEO MM0gflFTCnuDTyh7C+CWAabFjFk801in1C2uepCZveaoORNYDG2BAqk4lzaTwLJQO6 msRKi6DGIBt9MapjDJ/C6gK56qwGC2e+Q3E3tpjbe3FWjBA4z4aPUCiQI43o+zkFJu mUPkM5VjW/y2kXSQuOj2NG74UZEsprayRa/8m7pL2NMYQAR7xaHdfQY+GzIyu7EPvK ce/+WEjgnU5BQ== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 11/13] dma-direct: rename ret to cpu_addr in alloc helpers Date: Tue, 12 May 2026 14:34:06 +0530 Message-ID: <20260512090408.794195-12-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ret in dma_direct_alloc() and dma_direct_alloc_pages() holds the returned CPU mapping, not a generic return value. Rename it to cpu_addr and update the remaining uses to match. This makes the allocation paths easier to follow and keeps the local naming consistent with what the variable actually represents. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 1e9f9ff7b9d3..902008e40ea1 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -212,7 +212,7 @@ void *dma_direct_alloc(struct device *dev, size_t size, bool mark_mem_decrypt =3D false; bool allow_highmem =3D true; struct page *page; - void *ret; + void *cpu_addr; =20 /* * DMA_ATTR_CC_SHARED is not a caller-visible dma_alloc_*() @@ -326,34 +326,33 @@ void *dma_direct_alloc(struct device *dev, size_t siz= e, arch_dma_prep_coherent(page, size); =20 /* create a coherent mapping */ - ret =3D dma_common_contiguous_remap(page, size, prot, - __builtin_return_address(0)); - if (!ret) + cpu_addr =3D dma_common_contiguous_remap(page, size, prot, + __builtin_return_address(0)); + if (!cpu_addr) goto out_encrypt_pages; } else { - ret =3D page_address(page); + cpu_addr =3D page_address(page); } =20 - memset(ret, 0, size); + memset(cpu_addr, 0, size); =20 if (set_uncached) { void *uncached_cpu_addr; =20 arch_dma_prep_coherent(page, size); - uncached_cpu_addr =3D arch_dma_set_uncached(ret, size); + uncached_cpu_addr =3D arch_dma_set_uncached(cpu_addr, size); if (IS_ERR(uncached_cpu_addr)) goto out_free_remap_pages; - ret =3D uncached_cpu_addr; + cpu_addr =3D uncached_cpu_addr; } =20 *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); - return ret; - + return cpu_addr; =20 out_free_remap_pages: if (remap) - dma_common_free_remap(ret, size); + dma_common_free_remap(cpu_addr, size); =20 out_encrypt_pages: if (mark_mem_decrypt && @@ -437,7 +436,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, { unsigned long attrs =3D 0; struct page *page; - void *ret; + void *cpu_addr; =20 if (force_dma_unencrypted(dev)) attrs |=3D DMA_ATTR_CC_SHARED; @@ -455,7 +454,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, swiotlb_free(dev, page, size); return NULL; } - ret =3D page_address(page); + cpu_addr =3D page_address(page); goto setup_page; } =20 @@ -463,11 +462,11 @@ struct page *dma_direct_alloc_pages(struct device *de= v, size_t size, if (!page) return NULL; =20 - ret =3D page_address(page); - if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, ret, size)) + cpu_addr =3D page_address(page); + if ((attrs & DMA_ATTR_CC_SHARED) && dma_set_decrypted(dev, cpu_addr, size= )) goto out_leak_pages; setup_page: - memset(ret, 0, size); + memset(cpu_addr, 0, size); *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); return page; --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D69AA39901A; Tue, 12 May 2026 09:06:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576812; cv=none; b=Qy6q18UhW9WyxarBQkQmMocDXbmtaVbBmnIjGusY6JEXkBnr5vP/jkTl+K3uCZ6LNbs56UKf/HmoxE8XDPrR6/Jlkd8J1FYbGB6sqevbHpsxiq2OX8Nj6hvDa7X09MvnTbsnazCb5yZ8lIzPYBA+2fUaJTBk4BEVaJInFq6o8j4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576812; c=relaxed/simple; bh=EFbwf4neRkinC/ATvnrHtnHJ+BgFTSGwE4EsQ470+pU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HSM++DHrwUNnrUhlsYaIvOiKE/ysVvSyMtSYN0Wxv8h9NODrkASTYhXkuUxPVcy1wfzeARzPYP8egBwzJxKQI4+Gvx1IM0S6QQHP50ZkaDVqVTtlZav1SIDd1d8L6twdGijgW9/YYb2SzzaUQbj1o9zfPPzCQEZb71FQwZ/AiW0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eeM9Ag0N; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eeM9Ag0N" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6BDA7C2BCB0; Tue, 12 May 2026 09:06:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576811; bh=EFbwf4neRkinC/ATvnrHtnHJ+BgFTSGwE4EsQ470+pU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eeM9Ag0NUZG3/Alld9F85jKroRaeGZLjnhdQgGf7eBXyj/HG02jgnYVbGzp1bWIRN vsBf49P35alkP64j8E8vIkjsK9QXygnkp6lnXZ7hp0nTSIMncyar9LV5hPjSmCuGij J2hzJEiK6xajbQWH8kGIKhH13tq1geY23fbh9pN708IqrnHi6E8tyGq3gJLqo4AoC2 CP/7dc9AbgaPx+Zi6cK4+KsnhCKfye6n5JfF+g0V6btS6/fkjFDIyiH6XLs9HmhoI4 dlBCrXXur1ypXtp92XRZx+w2F8IXqyaU7iiGtua5IjzaSyD8e/sDzCrC8SSTK0g4rT MCZlBt21G/GGg== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org, stable@vger.kernel.org Subject: [PATCH v4 12/13] dma-direct: return struct page from dma_direct_alloc_from_pool() Date: Tue, 12 May 2026 14:34:07 +0530 Message-ID: <20260512090408.794195-13-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool helper") changed dma_direct_alloc_from_pool() to return the CPU address from dma_alloc_from_pool(). That fits dma_direct_alloc(), but dma_direct_alloc_pages() also uses the helper and expects a struct page *. Fix this by making dma_direct_alloc_from_pool() return the struct page * again, and pass the CPU address back through an out-parameter for the dma_direct_alloc() caller. Fixes: 5b138c534fda ("dma-direct: factor out a dma_direct_alloc_from_pool h= elper") Cc: stable@vger.kernel.org Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- kernel/dma/direct.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 902008e40ea1..5103a04df99f 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -165,24 +165,24 @@ static bool dma_direct_use_pool(struct device *dev, g= fp_t gfp) return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev); } =20 -static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp, unsigned long attrs) +static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t = size, + dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp, + unsigned long attrs) { struct page *page; u64 phys_limit; - void *ret; =20 if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL))) return NULL; =20 gfp |=3D dma_direct_optimal_gfp_mask(dev, &phys_limit); - page =3D dma_alloc_from_pool(dev, size, &ret, gfp, attrs, + page =3D dma_alloc_from_pool(dev, size, cpu_addr, gfp, attrs, dma_coherent_ok); if (!page) return NULL; *dma_handle =3D phys_to_dma_direct(dev, page_to_phys(page), !!(attrs & DMA_ATTR_CC_SHARED)); - return ret; + return page; } =20 static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size, @@ -278,9 +278,12 @@ void *dma_direct_alloc(struct device *dev, size_t size, * the atomic pools instead if we aren't allowed block. */ if ((remap || (attrs & DMA_ATTR_CC_SHARED)) && - dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, - gfp, attrs); + dma_direct_use_pool(dev, gfp)) { + page =3D dma_direct_alloc_from_pool(dev, size, + dma_handle, &cpu_addr, + gfp, attrs); + return page ? cpu_addr : NULL; + } =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size, attrs); @@ -443,7 +446,7 @@ struct page *dma_direct_alloc_pages(struct device *dev,= size_t size, =20 if ((attrs & DMA_ATTR_CC_SHARED) && dma_direct_use_pool(dev, gfp)) return dma_direct_alloc_from_pool(dev, size, dma_handle, - gfp, attrs); + &cpu_addr, gfp, attrs); =20 if (is_swiotlb_for_alloc(dev)) { page =3D dma_direct_alloc_swiotlb(dev, size, attrs); --=20 2.43.0 From nobody Fri Jun 12 22:30:58 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8CD7A3B9601; Tue, 12 May 2026 09:07:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576822; cv=none; b=IgbJkLjJz91NEFAzmDrvxlhwMkdBUhMADnYa077HRlUD2lPKjPMXcZJnaw7SWr0BRK9RLi9aYDq8Zp6N4wQBdF2jEJ0CeGMNc+kb9jPzXm8rHztdkojl9m/8uiIM6gq3i4/ZX7TD8lEoR5tebP5OZp0Rss+siOPzLlqlKcpN5FY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778576822; c=relaxed/simple; bh=SSIb9AffiyCb4tbfwh2yXw5+hoFTxTK0yIF3uaiiYAE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=euVGwUeuiVX9S2I9gOzTUpAfiO83ROhDJkrMtYJbR3QKOVRlDQlqA6WXOU3CZnoHg1olfEs3bEV7XhOZo7JZxJBaEe8Uar7jAbkUf1Y3OHOvO1PhBUQvK07Zy69FEyaAisvU9WFa7ZFksNPRgA34XFVEPGvZ0H8iqrv0iQUes7w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GYt8EqYI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GYt8EqYI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B33B4C4AF0B; Tue, 12 May 2026 09:06:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1778576822; bh=SSIb9AffiyCb4tbfwh2yXw5+hoFTxTK0yIF3uaiiYAE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GYt8EqYIRUIaavvPuaPJBda78vHlfhOtAKKCeYDhg/engtvJHFapVeF+Ah9i0/FJE fmqDPRJJt7SSe8N9iKzSLfWg1ueFG/uoakkMNldaxV0NBL8W795piGnbSbOjFxer3x tHzfEvWrtUMhKxo/JyVufOYkwYsug6srjyXOEIjelTyvy9oZ/2taZiPlIsn2rzbK3O RiLHESdmu1hBj05ocJaxuEA7CLneYuZT5I0BkcjWHL1PfZiP2VuP1VgdVY++Z4b7Lu bhpIZ1gADFwea0mdzaZWhQa/8g6VijT0S7COzlnLvOSszVAynSv150Z4ZnbbeAHNjR D6fbBzbPj/3lw== From: "Aneesh Kumar K.V (Arm)" To: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-coco@lists.linux.dev Cc: "Aneesh Kumar K.V (Arm)" , Robin Murphy , Marek Szyprowski , Will Deacon , Marc Zyngier , Steven Price , Suzuki K Poulose , Catalin Marinas , Jiri Pirko , Jason Gunthorpe , Mostafa Saleh , Petr Tesarik , Alexey Kardashevskiy , Dan Williams , Xu Yilun , linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, Madhavan Srinivasan , Michael Ellerman , Nicholas Piggin , "Christophe Leroy (CS GROUP)" , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , x86@kernel.org Subject: [PATCH v4 13/13] x86/amd-gart: preserve the direct DMA address until GART mapping succeeds Date: Tue, 12 May 2026 14:34:08 +0530 Message-ID: <20260512090408.794195-14-aneesh.kumar@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260512090408.794195-1-aneesh.kumar@kernel.org> References: <20260512090408.794195-1-aneesh.kumar@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" gart_alloc_coherent() first allocates memory through dma_direct_alloc(), which returns a direct-mapped DMA address in dma_addr. When force_iommu is enabled, the buffer is then remapped. Do not overwrite dma_addr before dma_map_area() has succeeded. Keep the dma_map_area result in a temporary variable so the direct DMA address remains available for dma_direct_free() on the error path. Signed-off-by: Aneesh Kumar K.V (Arm) Tested-by: Jiri Pirko Tested-by: Jiri Pirko --- arch/x86/kernel/amd_gart_64.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/kernel/amd_gart_64.c b/arch/x86/kernel/amd_gart_64.c index b5f1f031d45b..a109649c5649 100644 --- a/arch/x86/kernel/amd_gart_64.c +++ b/arch/x86/kernel/amd_gart_64.c @@ -467,18 +467,20 @@ gart_alloc_coherent(struct device *dev, size_t size, = dma_addr_t *dma_addr, gfp_t flag, unsigned long attrs) { void *vaddr; + dma_addr_t dma_map_addr; =20 vaddr =3D dma_direct_alloc(dev, size, dma_addr, flag, attrs); if (!vaddr || !force_iommu || dev->coherent_dma_mask <=3D DMA_BIT_MASK(24)) return vaddr; =20 - *dma_addr =3D dma_map_area(dev, virt_to_phys(vaddr), size, - DMA_BIDIRECTIONAL, - (1UL << get_order(size)) - 1, attrs); + dma_map_addr =3D dma_map_area(dev, virt_to_phys(vaddr), size, + DMA_BIDIRECTIONAL, + (1UL << get_order(size)) - 1, attrs); flush_gart(); - if (unlikely(*dma_addr =3D=3D DMA_MAPPING_ERROR)) + if (unlikely(dma_map_addr =3D=3D DMA_MAPPING_ERROR)) goto out_free; + *dma_addr =3D dma_map_addr; return vaddr; out_free: dma_direct_free(dev, size, vaddr, *dma_addr, attrs); --=20 2.43.0