From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 445D31E3DF2 for ; Fri, 26 Dec 2025 22:53:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789607; cv=none; b=Ik/Br4fqVxTGeOjC0sufVHQYs/H+BnAUeYowYtAj+COm0GcJvMHoo2q/e7BebEqel4bq2iCd8v3CE7ZrzjaKqIBIimOmu5T8aQfJB/5gzzbFEfIv7lH9PZE+/ZCTUnHt6ROFR9mcjqmSTXw83mCqoA2gCQG4fQG5emn/SlJKOwg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789607; c=relaxed/simple; bh=H9oEURFppzzQh4ad5CY+gHDo/cE8xRpGzvsht30sTGY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JO1EkYBLO0TnVwlYIUCIdaEBdz+thR/3IqP1bpszFiKDmmOb4ZEVG/gBWX8fY/z1tzPtwkJF6qx65AGwIXYlwHSQHigW7/WsipQKAbwxRQXuJmuHG/xWHWXrz73f0XMh/BHZJ/JowvpVPsS0cXUabjuespp/9Dp7t4lXAkfjbVc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Xx9FrvZM; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xx9FrvZM" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2a07f8dd9cdso80579725ad.1 for ; Fri, 26 Dec 2025 14:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789605; x=1767394405; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6YZZU/iwW88zjNDDGdCYX7IlRTUynqMqN8pEJHwT790=; b=Xx9FrvZM+Nx9UJLresHMiIGCErd3ssYr5ZrfAYFm7LtQWpV4PbC5JiOiTu9ROAtkfY OmFufGczklGPL/19GGxYlvvAQ+ekEpD6O0PdMwNJRr8g5jJ+nGJDGt56DUoxWDGTqmnf J6UNSN/pLGL9ANsh3sic8VZU01B5+4iUpKNQ+uo47+oKeTH4w6P281UYBYmK1E9gdC7P ns0F8mtvYRjEW9zZ8VCYjveCEIjkzcfmZehAAEu8SfCh+CMUf0n0GPY7iyjomuctaHJN 0CKRhGA/RkwvLQNyyoHibimDnj+miEUAAj9qx8VzCsBiwG0gIPkPrm4QKhysNpRn/1pd o4Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789605; x=1767394405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=6YZZU/iwW88zjNDDGdCYX7IlRTUynqMqN8pEJHwT790=; b=C6WXx9ADwBm7UHyEWz0gUzOh0Pw65aUiIU0NGzMjFBPUZE2BD+/b9MLZ2OcTO/OT5A MCvymA7eONDcw1+4Ewbk7In6qBbC1yXMklJZ67f8P0PmctLASphpSeARbRi5bel/EDXH qiiMF2Ek8KpF8Vc4B2Uyh3Y845Fm51XMcdzC1bgbtKbPTjt4KJx8yxaqF39WHZdxZW9Y iip/q5HJS6CYfBjhQFNEe/7PG5U6UdZ99xAik/jDhRYkvZwrvF0Ja+G+cPD2/DvCoh0E /xIx/lKoyDBHnL1ZI9cALqY9wmct4T3ibzzo8CpHZQ5yvcSyHl+CYhXOEb1+CZ9VKxhV a7hA== X-Gm-Message-State: AOJu0YxPxOtym1sltQC+dJmNt1+oKBtY836+woSa76u3+5lVf3cL759A 9qnNJImRm10pZIEJvrDF7UJipWUxiEx8rE/Rp9CHGbz0XH34jGcJhw3L X-Gm-Gg: AY/fxX5kPfrX3E7bS4vdcut+BaN3Xcg6W3bvO3zYHK/r+F6Bt7/Kv9qx2U9ddmC4b8F 9ww9+CbuZmH0E+ktbIp16Nqhh6mzw2pOp0iia7Z3BKMWw/XAEaO1tzS//XuGw8rf+vli1ty++Hy Hfb18rH0eficUnqseKUzFDbTqnJhuRcZdKECUrp9rUVhyAusu74ALIMiPGJvLuK5Tox6q98o3Rb iIrIVhx/csx4MLKRs7dSG5Qko414ERONve9dEi5Y52eoB/5uNAoChh/pdcN1kficS2YOGGsHQwu 6SAdJ4WnuIZOMosSXDvle71/ceK1cCb3Prl1OtP2WQaQiU53LKLwbxmuy4MauwmDp+w3MOjQo1V QB/cndC7iyoenbSV7u2ElWIrFagviCuVCySFuqPc3hJAH2E6zmXbK94jWFbLArTofYjPibaQgRR KB38n2LfS+bSyE X-Google-Smtp-Source: AGHT+IEPpw6k3H5B3GfQJ5gfoJrvfI68uwFervPMjcjEQdlc2KG+bnnPwgqGQJuYg9SuR5JsgGZOMA== X-Received: by 2002:a17:902:da81:b0:2a1:c72:a41d with SMTP id d9443c01a7336-2a2f2940678mr262079485ad.40.1766789605397; Fri, 26 Dec 2025 14:53:25 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:53:24 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Tangquan Zheng Subject: [PATCH v2 1/8] arm64: Provide dcache_by_myline_op_nosync helper Date: Sat, 27 Dec 2025 11:52:41 +1300 Message-ID: <20251226225254.46197-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song dcache_by_myline_op ensures completion of the data cache operations for a region, while dcache_by_myline_op_nosync only issues them without waiting. This enables deferred synchronization so completion for multiple regions can be handled together later. Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/assembler.h | 24 +++++++++++++++++++----- arch/arm64/kernel/relocate_kernel.S | 3 ++- 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/as= sembler.h index f0ca7196f6fa..b408ed61866f 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -371,14 +371,13 @@ alternative_endif * [start, end) with dcache line size explicitly provided. * * op: operation passed to dc instruction - * domain: domain used in dsb instruction * start: starting virtual address of the region * end: end virtual address of the region * linesz: dcache line size * fixup: optional label to branch to on user fault * Corrupts: start, end, tmp */ - .macro dcache_by_myline_op op, domain, start, end, linesz, tmp, fixup + .macro raw_dcache_by_myline_op op, start, end, linesz, tmp, fixup sub \tmp, \linesz, #1 bic \start, \start, \tmp .Ldcache_op\@: @@ -402,14 +401,13 @@ alternative_endif add \start, \start, \linesz cmp \start, \end b.lo .Ldcache_op\@ - dsb \domain =20 _cond_uaccess_extable .Ldcache_op\@, \fixup .endm =20 /* * Macro to perform a data cache maintenance for the interval - * [start, end) + * [start, end) and wait for completion * * op: operation passed to dc instruction * domain: domain used in dsb instruction @@ -420,7 +418,23 @@ alternative_endif */ .macro dcache_by_line_op op, domain, start, end, tmp1, tmp2, fixup dcache_line_size \tmp1, \tmp2 - dcache_by_myline_op \op, \domain, \start, \end, \tmp1, \tmp2, \fixup + raw_dcache_by_myline_op \op, \start, \end, \tmp1, \tmp2, \fixup + dsb \domain + .endm + +/* + * Macro to perform a data cache maintenance for the interval + * [start, end) without waiting for completion + * + * op: operation passed to dc instruction + * start: starting virtual address of the region + * end: end virtual address of the region + * fixup: optional label to branch to on user fault + * Corrupts: start, end, tmp1, tmp2 + */ + .macro dcache_by_line_op_nosync op, start, end, tmp1, tmp2, fixup + dcache_line_size \tmp1, \tmp2 + raw_dcache_by_myline_op \op, \start, \end, \tmp1, \tmp2, \fixup .endm =20 /* diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/reloca= te_kernel.S index 413f899e4ac6..71938eb3a3a3 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -64,7 +64,8 @@ SYM_CODE_START(arm64_relocate_new_kernel) mov x19, x13 copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 add x1, x19, #PAGE_SIZE - dcache_by_myline_op civac, sy, x19, x1, x15, x20 + raw_dcache_by_myline_op civac, x19, x1, x15, x20 + dsb sy b .Lnext .Ltest_indirection: tbz x16, IND_INDIRECTION_BIT, .Ltest_destination --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BB167279907 for ; Fri, 26 Dec 2025 22:53:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789615; cv=none; b=dUAhhF3aU5+PRW/erTaZXfIl4pLjkZpN8dGiJDpV9NjB1TTnMB8yaHnIq/o/ZaoL3Dto/yAqEr/t15Nvb4R8TReeR11iVybMIg4Iiz6a7jwTO7/v/o1daGCcg7buGFCKvi6dpC96AS7MJuILvRK3AhKM+4QjWfLL2wGEHRPSo9Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789615; c=relaxed/simple; bh=SYPssGIhYatri/BaiGbT+Fj74/Zx3I/RX0AzqBEd12w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NJZnDDzjY7TBpwu16JcFgpltxDB3WpEL4YwU4ozXG1VCLrKojJDiGTQBPWdH9lnj0nNBM38/XQWQiICE4iweX1iDXOnQ8xGHwNw6dOom8pCLAyOlPXZ4dBXnJklsS2Ms6SqtqhX/ITKHo1nT8nQZvqIfUpSRypANtVxsWOLf/54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=C47F9wxH; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="C47F9wxH" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2a0eaf55d58so49804635ad.1 for ; Fri, 26 Dec 2025 14:53:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789613; x=1767394413; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ietD9HqJ7J3E+f0491dnZq0rlaTb/buis2UihkMu59I=; b=C47F9wxHMlsze+ihB03uaQMRlCwuX3Z6pc6F0AZU3PQiXMeuOHXCErnkNQHxOT3O5F 7J+WHi40SlLxr1V+QtlyB2zKptH4aVmXJDlYBIQ7jHA/ukDRfG4ckZu216tIMvZXVLuz ey+yQuJHnKe694x642KNMx++t9F3hX2y8oEW9Kwzv6z8vTLVsIanh3lltuZRpcgLFm3m unddWnhpFiQXgLIPv9LgB00+GKWn3nzmX7/DoiYZQqUNHNffH3z3Mtpgw/W/WC8D7TGy rCZYaH+1o8OQnBqDdYnoLTuPxl/1TIO0yJQvYbpTTrwl5IIwX+HOHERSQ9iH8P+BZ1sK 0ecA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789613; x=1767394413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ietD9HqJ7J3E+f0491dnZq0rlaTb/buis2UihkMu59I=; b=f1lERM7ddOoK7WA+16f/JFbsy/a/3jkHwX3L+tSOJAT2sFgQbxL+NTnpPui8hsJLzm 1rbqjiVXWd5+1H3pTdB9Sv0xA+JNyUyYFFdFcZs5r3juL7S0tivoGFAj+A3HrlH3y1eZ QhAzWMDo86x+ywTXbNXRwn5uY2LJwvHOCYPjcQXgSlMtT2cGEnBOxpVvNI5S8vIufCqN kXry75LSXM61gWGbdtlEMl36Ahf0WNkLCYohTX2Udud7jMwhr3Jqpc+QaWLU+dfK2c+M z5gWayH2/FsykAcqFJtT/UjTyL/dy1TL2763+sB316NtZ2nUtcVjpC5Cw8W58aJnGFhP zdwQ== X-Gm-Message-State: AOJu0Yz5FU3K5zVLIQ5ISpOkCQos6xmDCltznkpJoGQPgdeCZnbIu/lD P9wqoxyNSrcxcUybVHIdRT6FMJRbh/UJbUeV2xGrEl32wg3Yeu/Wy9DF X-Gm-Gg: AY/fxX72dWewnp1b1rJoFqiu/6EKr6NC9zw7+BGQK48NYZMbKV2XyCMJbQ1neiORJcm rnHlfbAfEZRTS8gSpjjm2qBDTJvq3YgYcU/AQ81PXIIKZLUyvw846W0JomH4TxOBliDAOHS3B5a AC1VJdZ82wo5kYsDVXvdLbYmwDpWmyOyqO7+AVDB/ovjVVbRlCdCsuMPbuqXdb6n4eFmQGN2hqU UKFiovKrLwKOWjsKD3v43rAWJmqYJx9Gu5JIW9E266K5Eh9xNVh6V43SmJt+/rRpfHFfz20FVMU f265cARkQmn+3SwdUjjjOrWlXiwqnqVyk4k7nIOIBmCcjox8g2r5Gn+dJfqYZhF9ACP1HboF+iq Qboz+GQg9lcieZPMLiJa6cI11OOai5HrUvT4VbErfibqxKYiMqcrjRA9NK07/seXqNcv7fDo55n 8NVI2RFw91fnKC X-Google-Smtp-Source: AGHT+IE9oZQXckIJDcXvuEcMulX/ZaTRdgyK2lgHN5sr4Zh2/8KW3byPESPfAHlQDRzMXfnr9Ddibw== X-Received: by 2002:a17:90b:2b46:b0:34a:be93:72ee with SMTP id 98e67ed59e1d1-34e90d6d322mr21171859a91.8.1766789612867; Fri, 26 Dec 2025 14:53:32 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:53:32 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Tangquan Zheng Subject: [PATCH v2 2/8] arm64: Provide dcache_clean_poc_nosync helper Date: Sat, 27 Dec 2025 11:52:42 +1300 Message-ID: <20251226225254.46197-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song dcache_clean_poc_nosync does not wait for the data cache clean to complete. Later, we wait for completion of all scatter-gather entries together. Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/cacheflush.h | 1 + arch/arm64/mm/cache.S | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/c= acheflush.h index 28ab96e808ef..9b6d0a62cf3d 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -74,6 +74,7 @@ extern void icache_inval_pou(unsigned long start, unsigne= d long end); extern void dcache_clean_inval_poc(unsigned long start, unsigned long end); extern void dcache_inval_poc(unsigned long start, unsigned long end); extern void dcache_clean_poc(unsigned long start, unsigned long end); +extern void dcache_clean_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_pop(unsigned long start, unsigned long end); extern void dcache_clean_pou(unsigned long start, unsigned long end); extern long caches_clean_inval_user_pou(unsigned long start, unsigned long= end); diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 503567c864fd..4a7c7e03785d 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -178,6 +178,21 @@ SYM_FUNC_START(__pi_dcache_clean_poc) SYM_FUNC_END(__pi_dcache_clean_poc) SYM_FUNC_ALIAS(dcache_clean_poc, __pi_dcache_clean_poc) =20 +/* + * dcache_clean_poc_nosync(start, end) + * + * Issue the instructions of D-cache lines for the interval [start, end). + * not necessarily cleaned to the PoC till an explicit dsb sy afterward. + * + * - start - virtual start address of region + * - end - virtual end address of region + */ +SYM_FUNC_START(__pi_dcache_clean_poc_nosync) + dcache_by_line_op_nosync cvac, x0, x1, x2, x3 + ret +SYM_FUNC_END(__pi_dcache_clean_poc_nosync) +SYM_FUNC_ALIAS(dcache_clean_poc_nosync, __pi_dcache_clean_poc_nosync) + /* * dcache_clean_pop(start, end) * --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59BEA243951 for ; Fri, 26 Dec 2025 22:53:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789623; cv=none; b=BbojM3uNWkJoMM2ttVKnfBlcRQXYDxDQX6cWWCCloOzrl/QCZBx2FyJKvCFxi+bva/bv41BJoszUhQuhrOYvVBbI9FJxkgjBL7z/x9GZ0lbCriVUkUfbGcNpyAGrAWHkAYqRtlRwTVGFHpVZrqaZpXno6fvFKGei4yPMREDGDLw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789623; c=relaxed/simple; bh=5FIlQ7B2g8aH8yOFAn3F/4ED5Pk7hQam/ZJDBiZWQVk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Wip2r9lUmvHhPBC1f7BBEqgE+O2NmCx7g6rcDYYkrtBnQhVsVB27ONMZqMj9swu2TX6gXO31P47OkoFlzKD04NIEAx/3rh/X7XYSJID35pW7KP9B4ZUSavaPi4bW94VHWyfsKMpppjEA6cMcSKGS+GqM6Iz4NS+6AYPzpGmsXV0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JS919s4o; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JS919s4o" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-34c9edf63a7so8171862a91.1 for ; Fri, 26 Dec 2025 14:53:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789620; x=1767394420; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XrpYuFgGuOqDarimfgNWi+H5QYEBWUzUVIbcnmljV6w=; b=JS919s4oWHlDzk+iJojSmssqHVgYb2IYBcC6rrfIWOEXBwGmLvNK2ruNSTX1j75Rlo dzWFGVK6aLPpLsRf1DTOgpJi2lznx4kw/Q+ZKVmDJROeybQr1/x3OIotGTmLbdeSv1K4 6kvDuItKioJfJe0zEPIdTV3WVqTE62m0glGE+9+pqJkhqU+2o5OOpslD1rbquPwHMtNc vf92zdnIry6porsvu72243tAn2ukvwdWfWIIFdMDSvMIUSUJG7dNLH14rzeZWpUfU+zp toezmBZ0uSzhc/xy+bV4Ioy7NmeHnCV6f+BOEJ2868ttUDOo5Xsrma0Yom3jzdCOkSB9 9H2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789620; x=1767394420; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XrpYuFgGuOqDarimfgNWi+H5QYEBWUzUVIbcnmljV6w=; b=T37HnDtwYLKdyVKj20/+u0YQSkYglL9ZqbWe4/igt/Fd4mTIcUYtYrGxOwCa7w7bjD EUqCsfYe0gZKzwhzQSylUARLg4sWZl9W7VcQN75wphnaLxqitpK8qpkOuA+OHQR5CsOu euVNuGAubmpRmup+qOeoSGr4+jUM8B2/hGEQxss36XdLzZQWGPEx72irYartBAefUCw8 H1afWE/Xvw3SC7ug9N6YsnQ1D22cExmQ6SptxsOR15A6cggbA2D8br63U1ct+90qatqQ PD9mSr8CqyL1GZDTqqdfDD55d4+TNmOK//pyy96gk+SDWD26yuv10YappN0X/JhGPQE5 U3WQ== X-Gm-Message-State: AOJu0YwGexZFx/lySFcJFTtKSJTPP8xUV6nj+DORb5UsmOaI3sszBpuS Qjy6w6zod0DtxfAoVM/8+9Z6jV2OwKjrGAz2H3eqatIxfqqTSUUawJwv X-Gm-Gg: AY/fxX7EL/yQb1T76n0J9Hnrv9Tz+Pn1iHTJPEmWvcPa+QOShB4KjBtnVPxBPxrncNL 0RKumudyXo5nNL8tpE+TQ3YI9j2M4TPKrf4VyS/BUFMCmTLiGAneh71eUFi8J/a30th3uawiz2N zNNg7cWIeihxC/z2n0xgUldGD5cVDhTcaJkN1pq2ZmE9unv7iAdc3qLXi061rXMj4xDYmS2C/bQ QvG3odmVsrSqMpDOGDGIAivtFYD4bhjTPiwYckExMBjryBy30OuH7IHPMXWjbESg8OmeUbQolJW SrB1NvO2C3DoAdDqOeGVKPjcvgsOmBTHcdKGSYbLnDfgxBn+iMDmpJCEu1OWyD6g2u5r4+JyXgR 2DY7QgpkwK6wM3ljkOt93MdDbwsB8UUzrRIo9gveDzHwnwxBAcFcdwCM1HeQuhePl+eWcC5f81P QE+937dtscLXpf X-Google-Smtp-Source: AGHT+IGZAwVjw6Q14Ldbj2NnyfuDwpis8NstpDAoaBbm3hUZYJbQhE2SG8A3iUeExaHFKxkFLgGP9A== X-Received: by 2002:a17:90b:528a:b0:330:6d5e:f174 with SMTP id 98e67ed59e1d1-34e921b05f6mr21341646a91.20.1766789620383; Fri, 26 Dec 2025 14:53:40 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:53:39 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Tangquan Zheng Subject: [PATCH v2 3/8] arm64: Provide dcache_inval_poc_nosync helper Date: Sat, 27 Dec 2025 11:52:43 +1300 Message-ID: <20251226225254.46197-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song dcache_inval_poc_nosync does not wait for the data cache invalidation to complete. Later, we defer the synchronization so we can wait for all SG entries together. Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/cacheflush.h | 1 + arch/arm64/mm/cache.S | 42 +++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/c= acheflush.h index 9b6d0a62cf3d..382b4ac3734d 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -74,6 +74,7 @@ extern void icache_inval_pou(unsigned long start, unsigne= d long end); extern void dcache_clean_inval_poc(unsigned long start, unsigned long end); extern void dcache_inval_poc(unsigned long start, unsigned long end); extern void dcache_clean_poc(unsigned long start, unsigned long end); +extern void dcache_inval_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_pop(unsigned long start, unsigned long end); extern void dcache_clean_pou(unsigned long start, unsigned long end); diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 4a7c7e03785d..99a093d3aecb 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -132,17 +132,7 @@ alternative_else_nop_endif ret SYM_FUNC_END(dcache_clean_pou) =20 -/* - * dcache_inval_poc(start, end) - * - * Ensure that any D-cache lines for the interval [start, end) - * are invalidated. Any partial lines at the ends of the interval are - * also cleaned to PoC to prevent data loss. - * - * - start - kernel start address of region - * - end - kernel end address of region - */ -SYM_FUNC_START(__pi_dcache_inval_poc) +.macro raw_dcache_inval_poc_macro dcache_line_size x2, x3 sub x3, x2, #1 tst x1, x3 // end cache line aligned? @@ -158,11 +148,41 @@ SYM_FUNC_START(__pi_dcache_inval_poc) 3: add x0, x0, x2 cmp x0, x1 b.lo 2b +.endm + +/* + * dcache_inval_poc(start, end) + * + * Ensure that any D-cache lines for the interval [start, end) + * are invalidated. Any partial lines at the ends of the interval are + * also cleaned to PoC to prevent data loss. + * + * - start - kernel start address of region + * - end - kernel end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_poc) + raw_dcache_inval_poc_macro dsb sy ret SYM_FUNC_END(__pi_dcache_inval_poc) SYM_FUNC_ALIAS(dcache_inval_poc, __pi_dcache_inval_poc) =20 +/* + * dcache_inval_poc_nosync(start, end) + * + * Issue the instructions of D-cache lines for the interval [start, end) + * for invalidation. Not necessarily cleaned to PoC till an explicit dsb + * sy is issued later + * + * - start - kernel start address of region + * - end - kernel end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_poc_nosync) + raw_dcache_inval_poc_macro + ret +SYM_FUNC_END(__pi_dcache_inval_poc_nosync) +SYM_FUNC_ALIAS(dcache_inval_poc_nosync, __pi_dcache_inval_poc_nosync) + /* * dcache_clean_poc(start, end) * --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41937279907 for ; Fri, 26 Dec 2025 22:53:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789632; cv=none; b=lqmqDfH9XO0xyb3/Ijdf8jo8FNUPGv9xO9eEmP8bhQuziBW0RKlo8Sdcp/Nrx1YiUlu8zcAnZlSWTTWlCgnV5G+YgjzEKu6bXP0cALZ/3LuFZ2b4kF+PrYwUhyRVczKQEmajxLDAzztqCXI5yLGdYZkah8EWZnWavT6o7kHi7tA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789632; c=relaxed/simple; bh=H+JcZ3ALFKjMEK+ridC5ARPUqoSBVPZTR86r1nheB4A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gOUWt/wt5e1Bg5dETbzjTEWsx05FAwqNWrMIymKLEymVrydo3zdTcNl0t4/QTBlJf/Exm/+4ftvFXd3J7xqUag/+3hzXUUjD59N3Von83g3NDEj5fLRcb3fvx6rK0evUEk4IiEhw5wJz1YLiqmiV2HvSYoK+mGl9Bsd//V/HhXY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yr5Cw/y/; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yr5Cw/y/" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-34a4078f669so7787372a91.1 for ; Fri, 26 Dec 2025 14:53:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789629; x=1767394429; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9CBlUJRZIbYM1prFWy/0G8KDccWD/YxTB/AJqWrZ/Ys=; b=Yr5Cw/y/devJP426FsG6TE6rRdQOhibXqPOfCNy6UAHA2QtPzSdVCoCfqaIWY6a+23 3ke37qty3xEmLFG0SRFznnmh29/UKP4hu71jJBbPgGqrlkWlY25CCwIjR0t4cZkTY2Qf 22SFmv64NSx0Pm0uJwauZ2zXfkiq2PQeuosTjxJ1F0mpMWeWaVKGDAZyJHU6gxP5p1fO kaiogMdg9PTJc0rpG4LNoOTTkXvJSv4Rjeh3B/FPYs2HoA8+SNYIu9KOe+pIQ1B9U6nN 6anKQwNwq+8LCbVsSM+rOZqXdPzt1kEOmvgwJx4yJZ+Hw13Rf/I9tbeClNIgLzsGehhY 0v0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789629; x=1767394429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9CBlUJRZIbYM1prFWy/0G8KDccWD/YxTB/AJqWrZ/Ys=; b=rTW2l8xxY2BSzVtYK2iCP6szCoGkClsxGYmpoAOLHquHyZPCmbtEB7puHEIhFcKv6z yQLXmrI8tsd4lNnb/PY23Fog4ksCJZ0mc+utqTMBwwlzhcIJFBYLEFDf14yM5GbcBC8T pbxGRKBfWcnR1Awn8eEGGtc6both+woc1BUverfGBcO/qfYG7DBgvWQrVyg62rY3p6Q/ G3+/dcUOm9JlmU3HkIxVCbv0hYUsdPN9AjzpBhRkukS7Prka0kIGveWMkqliB5TSkCZr VlnQfKoxN7LUMe84j3FETPjPbrohQY/ZabfJXekRlAfylmgxtrJXHKfqZLJ+vVaqMT1W hsIw== X-Gm-Message-State: AOJu0Yz2RyveKgO6HRoDhppwSl9w3X/oaz9GqPTttn3OqEGZrDgukB3M 7MV4YDpzeR3i5EV5vsBct+qmlQXOGgnIMKO2hjoy8s5r7DSiDTryRs2A X-Gm-Gg: AY/fxX5wcJefM9qEWl4P8Zy4a4TJyM7Cf4PzWhuNEFBNb8J4ZIfUqqFwPP+DsRQmtJF tBuhPnXeFdc1MzrTIAeHa7RSRtJDRu4/XIT9zT/xbXJpVgJdU60gdt1c7L5qpJg0A4Xo+GXXtii Vmok/50cD1MCgyyqeOnzNgJWhYulL3MfNiClB45hlY8Xd4Wl7B55n1r6VNUf+WWkKnLpLSgPR1b T9A+ElSNHK1R8wAf3nAxORYwnMcyEH9O6kwkf2IPMp0T4HDKPTxPz5LaK3fu+sRPzrOLOJbnLwt qgMb50rRlQealo6jmL+K5yeDfCmKEEfo3WWfWdp8z8LGCU43Qa+Jeis3VlGLfmqpeoik6qGfKOq IrxGwLTR/ZCYBWHE53PM/PizP1CZLoSxl0CskkuBfvs2E/r4/O5hNS/2uuOMvM5zinlfggGDSBt Ka2AvIWQdp3BC+ X-Google-Smtp-Source: AGHT+IF6GTCerKOqMOKSYSeLfAiimvUN3+7D1SxAsR14bBFsurGjwmGev9rAORcjp7S0YIZTn6CThQ== X-Received: by 2002:a17:90b:560c:b0:32d:e780:e9d5 with SMTP id 98e67ed59e1d1-34e921c3003mr16987619a91.22.1766789629137; Fri, 26 Dec 2025 14:53:49 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:53:48 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Joerg Roedel , Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , Tangquan Zheng Subject: [PATCH v2 4/8] dma-mapping: Separate DMA sync issuing and completion waiting Date: Sat, 27 Dec 2025 11:52:44 +1300 Message-ID: <20251226225254.46197-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Currently, arch_sync_dma_for_cpu and arch_sync_dma_for_device always wait for the completion of each DMA buffer. That is, issuing the DMA sync and waiting for completion is done in a single API call. For scatter-gather lists with multiple entries, this means issuing and waiting is repeated for each entry, which can hurt performance. Architectures like ARM64 may be able to issue all DMA sync operations for all entries first and then wait for completion together. To address this, arch_sync_dma_for_* now issues DMA operations in batch, followed by a flush. On ARM64, the flush is implemented using a dsb instruction within arch_sync_dma_flush(). For now, add arch_sync_dma_flush() after each arch_sync_dma_for_*() call. arch_sync_dma_flush() is defined as a no-op on all architectures except arm64, so this patch does not change existing behavior. Subsequent patches will introduce true batching for SG DMA buffers. Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Joerg Roedel Cc: Juergen Gross Cc: Stefano Stabellini Cc: Oleksandr Tyshchenko Cc: Tangquan Zheng Signed-off-by: Barry Song Reviewed-by: Juergen Gross # drivers/xen/swiotlb-xen.c Reviewed-by: Leon Romanovsky --- arch/arm64/include/asm/cache.h | 6 ++++++ arch/arm64/mm/dma-mapping.c | 4 ++-- drivers/iommu/dma-iommu.c | 37 +++++++++++++++++++++++++--------- drivers/xen/swiotlb-xen.c | 24 ++++++++++++++-------- include/linux/dma-map-ops.h | 6 ++++++ kernel/dma/direct.c | 8 ++++++-- kernel/dma/direct.h | 9 +++++++-- kernel/dma/swiotlb.c | 4 +++- 8 files changed, 73 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/cache.h b/arch/arm64/include/asm/cache.h index dd2c8586a725..487fb7c355ed 100644 --- a/arch/arm64/include/asm/cache.h +++ b/arch/arm64/include/asm/cache.h @@ -87,6 +87,12 @@ int cache_line_size(void); =20 #define dma_get_cache_alignment cache_line_size =20 +static inline void arch_sync_dma_flush(void) +{ + dsb(sy); +} +#define arch_sync_dma_flush arch_sync_dma_flush + /* Compress a u64 MPIDR value into 32 bits. */ static inline u64 arch_compact_of_hwid(u64 id) { diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index b2b5792b2caa..ae1ae0280eef 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -17,7 +17,7 @@ void arch_sync_dma_for_device(phys_addr_t paddr, size_t s= ize, { unsigned long start =3D (unsigned long)phys_to_virt(paddr); =20 - dcache_clean_poc(start, start + size); + dcache_clean_poc_nosync(start, start + size); } =20 void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size, @@ -28,7 +28,7 @@ void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t size, if (dir =3D=3D DMA_TO_DEVICE) return; =20 - dcache_inval_poc(start, start + size); + dcache_inval_poc_nosync(start, start + size); } =20 void arch_dma_prep_coherent(struct page *page, size_t size) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index c92088855450..6827763a3877 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1095,8 +1095,10 @@ void iommu_dma_sync_single_for_cpu(struct device *de= v, dma_addr_t dma_handle, return; =20 phys =3D iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu(phys, size, dir); + arch_sync_dma_flush(); + } =20 swiotlb_sync_single_for_cpu(dev, phys, size, dir); } @@ -1112,8 +1114,10 @@ void iommu_dma_sync_single_for_device(struct device = *dev, dma_addr_t dma_handle, phys =3D iommu_iova_to_phys(iommu_get_dma_domain(dev), dma_handle); swiotlb_sync_single_for_device(dev, phys, size, dir); =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_device(phys, size, dir); + arch_sync_dma_flush(); + } } =20 void iommu_dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, @@ -1122,13 +1126,16 @@ void iommu_dma_sync_sg_for_cpu(struct device *dev, = struct scatterlist *sgl, struct scatterlist *sg; int i; =20 - if (sg_dma_is_swiotlb(sgl)) + if (sg_dma_is_swiotlb(sgl)) { for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_cpu(dev, sg_dma_address(sg), sg->length, dir); - else if (!dev_is_dma_coherent(dev)) - for_each_sg(sgl, sg, nelems, i) + } else if (!dev_is_dma_coherent(dev)) { + for_each_sg(sgl, sg, nelems, i) { arch_sync_dma_for_cpu(sg_phys(sg), sg->length, dir); + arch_sync_dma_flush(); + } + } } =20 void iommu_dma_sync_sg_for_device(struct device *dev, struct scatterlist *= sgl, @@ -1143,8 +1150,10 @@ void iommu_dma_sync_sg_for_device(struct device *dev= , struct scatterlist *sgl, sg_dma_address(sg), sg->length, dir); else if (!dev_is_dma_coherent(dev)) - for_each_sg(sgl, sg, nelems, i) + for_each_sg(sgl, sg, nelems, i) { arch_sync_dma_for_device(sg_phys(sg), sg->length, dir); + arch_sync_dma_flush(); + } } =20 static phys_addr_t iommu_dma_map_swiotlb(struct device *dev, phys_addr_t p= hys, @@ -1219,8 +1228,10 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, ph= ys_addr_t phys, size_t size, return DMA_MAPPING_ERROR; } =20 - if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); + arch_sync_dma_flush(); + } =20 iova =3D __iommu_dma_map(dev, phys, size, prot, dma_mask); if (iova =3D=3D DMA_MAPPING_ERROR && !(attrs & DMA_ATTR_MMIO)) @@ -1242,8 +1253,10 @@ void iommu_dma_unmap_phys(struct device *dev, dma_ad= dr_t dma_handle, if (WARN_ON(!phys)) return; =20 - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && !dev_is_dma_coherent(dev)) + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) && !dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu(phys, size, dir); + arch_sync_dma_flush(); + } =20 __iommu_dma_unmap(dev, dma_handle, size); =20 @@ -1836,8 +1849,10 @@ static int __dma_iova_link(struct device *dev, dma_a= ddr_t addr, bool coherent =3D dev_is_dma_coherent(dev); int prot =3D dma_info_to_prot(dir, coherent, attrs); =20 - if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); + arch_sync_dma_flush(); + } =20 return iommu_map_nosync(iommu_get_dma_domain(dev), addr, phys, size, prot, GFP_ATOMIC); @@ -2008,8 +2023,10 @@ static void iommu_dma_iova_unlink_range_slow(struct = device *dev, end - addr, iovad->granule - iova_start_pad); =20 if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_cpu(phys, len, dir); + arch_sync_dma_flush(); + } =20 swiotlb_tbl_unmap_single(dev, phys, len, dir, attrs); =20 diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c index ccf25027bec1..b79917e785a5 100644 --- a/drivers/xen/swiotlb-xen.c +++ b/drivers/xen/swiotlb-xen.c @@ -262,10 +262,12 @@ static dma_addr_t xen_swiotlb_map_phys(struct device = *dev, phys_addr_t phys, =20 done: if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) { - if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dev_addr)))) + if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dev_addr)))) { arch_sync_dma_for_device(phys, size, dir); - else + arch_sync_dma_flush(); + } else { xen_dma_sync_for_device(dev, dev_addr, size, dir); + } } return dev_addr; } @@ -287,10 +289,12 @@ static void xen_swiotlb_unmap_phys(struct device *hwd= ev, dma_addr_t dev_addr, BUG_ON(dir =3D=3D DMA_NONE); =20 if (!dev_is_dma_coherent(hwdev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) { - if (pfn_valid(PFN_DOWN(dma_to_phys(hwdev, dev_addr)))) + if (pfn_valid(PFN_DOWN(dma_to_phys(hwdev, dev_addr)))) { arch_sync_dma_for_cpu(paddr, size, dir); - else + arch_sync_dma_flush(); + } else { xen_dma_sync_for_cpu(hwdev, dev_addr, size, dir); + } } =20 /* NOTE: We use dev_addr here, not paddr! */ @@ -308,10 +312,12 @@ xen_swiotlb_sync_single_for_cpu(struct device *dev, d= ma_addr_t dma_addr, struct io_tlb_pool *pool; =20 if (!dev_is_dma_coherent(dev)) { - if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dma_addr)))) + if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dma_addr)))) { arch_sync_dma_for_cpu(paddr, size, dir); - else + arch_sync_dma_flush(); + } else { xen_dma_sync_for_cpu(dev, dma_addr, size, dir); + } } =20 pool =3D xen_swiotlb_find_pool(dev, dma_addr); @@ -331,10 +337,12 @@ xen_swiotlb_sync_single_for_device(struct device *dev= , dma_addr_t dma_addr, __swiotlb_sync_single_for_device(dev, paddr, size, dir, pool); =20 if (!dev_is_dma_coherent(dev)) { - if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dma_addr)))) + if (pfn_valid(PFN_DOWN(dma_to_phys(dev, dma_addr)))) { arch_sync_dma_for_device(paddr, size, dir); - else + arch_sync_dma_flush(); + } else { xen_dma_sync_for_device(dev, dma_addr, size, dir); + } } } =20 diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4809204c674c..e7dd8a63b40e 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -361,6 +361,12 @@ static inline void arch_sync_dma_for_cpu(phys_addr_t p= addr, size_t size, } #endif /* ARCH_HAS_SYNC_DMA_FOR_CPU */ =20 +#ifndef arch_sync_dma_flush +static inline void arch_sync_dma_flush(void) +{ +} +#endif + #ifdef CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL void arch_sync_dma_for_cpu_all(void); #else diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 50c3fe2a1d55..a219911c7b90 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -402,9 +402,11 @@ void dma_direct_sync_sg_for_device(struct device *dev, =20 swiotlb_sync_single_for_device(dev, paddr, sg->length, dir); =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_device(paddr, sg->length, dir); + arch_sync_dma_flush(); + } } } #endif @@ -421,8 +423,10 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, for_each_sg(sgl, sg, nents, i) { phys_addr_t paddr =3D dma_to_phys(dev, sg_dma_address(sg)); =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu(paddr, sg->length, dir); + arch_sync_dma_flush(); + } =20 swiotlb_sync_single_for_cpu(dev, paddr, sg->length, dir); =20 diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index da2fadf45bcd..a69326eed266 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -60,8 +60,10 @@ static inline void dma_direct_sync_single_for_device(str= uct device *dev, =20 swiotlb_sync_single_for_device(dev, paddr, size, dir); =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_device(paddr, size, dir); + arch_sync_dma_flush(); + } } =20 static inline void dma_direct_sync_single_for_cpu(struct device *dev, @@ -71,6 +73,7 @@ static inline void dma_direct_sync_single_for_cpu(struct = device *dev, =20 if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu(paddr, size, dir); + arch_sync_dma_flush(); arch_sync_dma_for_cpu_all(); } =20 @@ -109,8 +112,10 @@ static inline dma_addr_t dma_direct_map_phys(struct de= vice *dev, } =20 if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); + arch_sync_dma_flush(); + } return dma_addr; =20 err_overflow: diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index a547c7693135..7cdbfcdfef86 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1595,8 +1595,10 @@ dma_addr_t swiotlb_map(struct device *dev, phys_addr= _t paddr, size_t size, return DMA_MAPPING_ERROR; } =20 - if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) { arch_sync_dma_for_device(swiotlb_addr, size, dir); + arch_sync_dma_flush(); + } return dma_addr; } =20 --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48E6E29E0E1 for ; Fri, 26 Dec 2025 22:53:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789645; cv=none; b=uhgglob70CNaTT4EMbuuO7+NyzPoO/fAPSmZEw0G75kfd7w1WnmgUTBp+pZDRL9TKDsUPIRqOpWdeZrWjMcfe6CtrEi0gMj24mI/iPCizrZe1FANiKj6hXbdGuVcCJiGtRzTqS06BLDUqF0bC33HLOxfDAvT6Ukhyk8Pu15zFSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789645; c=relaxed/simple; bh=oh176zWXXfoQMYsnxwVsxRFdbjtJrXMltLaE7iC2HWY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G8I3IZq+4zs4GWt4TjQMeQTSw/rvMn3hg424O2Y31/7G60aeNHpRrm3xTuUoXyqV0HfcfYMvn9asxDrAq2l5Wg+M5QjY8jTOlgbDN/Lc7Xm4uq6crxYKiOCgwyGUEyC//7ZIKICqBePilYjuuXiMEuKRJxHCfd0XcnEbthtU3L4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=juDIhDU7; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="juDIhDU7" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2a0c20ee83dso95950195ad.2 for ; Fri, 26 Dec 2025 14:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789637; x=1767394437; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6SwUY+FTc0pZhvaABGOEZvH4DFh2qP62P2nENr8WNe8=; b=juDIhDU7J5FzhvjCj6YaNXuiCLGg8CjiWay1o6RShobF8HDHI5H8CCAy8BrMJ21kXN Y46DSb3BC/oDilcUtTpUSRbZ6GLkH3cB8bnrFaYuvr4KDDr52IoF8zW9HcUaXIZN03MF tzlDnKpDGJymF2XzK6p6Ral8iBy7FnfiwvBgxRa3EZOrwXb+DMgDCTU8sVSoFh3VyGru bViWfmxYWx13SR9IvL3LfTiSHlQaR3TEmnvkVE56pz4tSxevqnDEEiQYVmJ85UoGrB/J SnBeMKrqgDobcu5cesCC5E7iqsxErgjPAUjnGHppfplEN4vmg+uZs+w3ZLqVviSq+ld1 ef5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789637; x=1767394437; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=6SwUY+FTc0pZhvaABGOEZvH4DFh2qP62P2nENr8WNe8=; b=Mtv2haUVCemZvcKq5fBHWSyZN2iAURqmOpC71WF4mo/v4L3F2KYgTQz1eYhCE35RjR w1ueBx2nf+whJD5kV2xkbD7l1/Bs7EiS8qIKCnvwmplwn1N/hkcGF8r62i2mQtRs8t0H s1CEcMsFTesIN/ghJMqewAlW9pAEpIpQylQexvI+WDH8HwdCAg8eeuC2EGhssILEh6Sa kIeox4tGP3svVlEnmDOG4L+iuuSl2n+fhjtmmjwPbch9C5S2EawMb9g0Q0D8F6vhdYnM cq89gcm/M14Pprdh4PP5hPE5P/Y5JlZz+zCsoEqeAg5NIu3pdpV6k93843/2Py8Te9dZ 1dVw== X-Gm-Message-State: AOJu0YxYHnWEPgLea/IRMzwzHePxaFlII0cA3Z2MUNC6YyFXvEsq/vMm eij+PKBvySEgPaH00iJDg1DNIHlgF0jPCFPnthiP4hcY/uptRDlHX4qD X-Gm-Gg: AY/fxX6FcvuRaXR3WNsfADQFLKuDe3mLXBIAYtCZdyHZLK4BkKMiY0gHf5vhAHuMjZO uqnnQDhu3Zoi0kJsuYHRWJwqu+gn4NX2QUtaFT1PfmLyUvQCmMZW4LnriuahSm52iyRCtqJi1lp Ny4Dvpu6RfjhGT4EKToaABolZRGBaSU4E5C+aJUVqOAp3wZ7phH66Qx0NZlevyzlo+SVEXYbm1I vHECTT1bprY8wcgWG3Guhx+v39Ofb6GisNNamHDY2H/h+ltnTSGkT+ze252q+lZhJ53S9iZ00Ef IogHB1qYWpFkX1qsDyXRWE6+02eDxMDsCXWo6kE9VuC8YMt3lsoV3dMoI3vIwdMIIvrnvwZKy7f mhfIcC1ts8LdVwawCKGWDFvjVVGbDVeRsGUbtOK85Zlrh5AAHYKGb/bbjvYlNakXX8oKcTESiXK tYCipWyLPkS+87rsTuNosHVZ0= X-Google-Smtp-Source: AGHT+IHzo7zg63L6IUIixvYNdgE1EhLj43E4LWUqmZi99iaSKAA/lUr+ts9Nayt1nnGWbam/Z3t7Yw== X-Received: by 2002:a17:903:1670:b0:2a2:bff6:42f2 with SMTP id d9443c01a7336-2a2f222b664mr247962585ad.6.1766789636697; Fri, 26 Dec 2025 14:53:56 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:53:55 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Tangquan Zheng Subject: [PATCH v2 5/8] dma-mapping: Support batch mode for dma_direct_sync_sg_for_* Date: Sat, 27 Dec 2025 11:52:45 +1300 Message-ID: <20251226225254.46197-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Instead of performing a flush per SG entry, issue all cache operations first and then flush once. This ultimately benefits __dma_sync_sg_for_cpu() and __dma_sync_sg_for_device(). Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- kernel/dma/direct.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index a219911c7b90..98bacf562ca1 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -402,12 +402,12 @@ void dma_direct_sync_sg_for_device(struct device *dev, =20 swiotlb_sync_single_for_device(dev, paddr, sg->length, dir); =20 - if (!dev_is_dma_coherent(dev)) { + if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_device(paddr, sg->length, dir); - arch_sync_dma_flush(); - } } + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_flush(); } #endif =20 @@ -423,10 +423,8 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, for_each_sg(sgl, sg, nents, i) { phys_addr_t paddr =3D dma_to_phys(dev, sg_dma_address(sg)); =20 - if (!dev_is_dma_coherent(dev)) { + if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_cpu(paddr, sg->length, dir); - arch_sync_dma_flush(); - } =20 swiotlb_sync_single_for_cpu(dev, paddr, sg->length, dir); =20 @@ -434,8 +432,10 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_dma_mark_clean(paddr, sg->length); } =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { + arch_sync_dma_flush(); arch_sync_dma_for_cpu_all(); + } } =20 /* --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AB9027CCE0 for ; Fri, 26 Dec 2025 22:54:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789649; cv=none; b=RVHsbYRrHwtMGVwiZG8kOTpinBxm1TFM+8sSoz8H/SPwpNMw388YxLmQmMEbBabtS8qPvjDFmO5xKznAKuUePzWX3IXUgI4KdH/DZheZBWj4rv6lMndlHa2GZplKOnjAP1sZV6ZcQpX+XmsTbaJFLQb3nGVpn7sYwNPME6oSh7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789649; c=relaxed/simple; bh=y0GAlTukBhrwG28eorWp2ff1UapqeGDpOXnThHTS++U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XZvq+iOjPoUAcy9SuES3m4udfLBuxSJiy7LYkIwbFPNHJVjh1+A388MKE5HOIip8WE5/sUsL645CBJHYKeXXmblAdDpLbFURV8ga3/r7mx1Xi5p46/oB9H2Y9Fd1eej/ne7dY9cadcLgNzAWupW/jaDeE0lSh+OuXR/Hc4Lu7WM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QIrXQYVv; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QIrXQYVv" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-34c2f335681so5787221a91.1 for ; Fri, 26 Dec 2025 14:54:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789645; x=1767394445; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pcGdIc/9DJZ3V/5WVf7vUUbl5wTlCyMzFQWvUiTQ7kw=; b=QIrXQYVvSYtGXv5Y0DdvxVGajEHwy61NH/2Rt1UPnY4iwBAzadsIy9guUbStWgC1xN BoMucDZ4xr+lC6MW4b+pKwwCiwUTNzIKoufco+mSC/vqyiflGdDBLIeu/TFRCGUSKJMd 2G0BfwjVp7+DSGoQAmWCAvXGL81HlfrD3S/whZN2AizTztHULJFM12uBzqwA4JEwXyC7 Xctp2qba2IeVQlLwyhg9XmDmyqEF6gfgiBSsDbp418AJnQqUacKZKd4yGtsWlqHaLhzG 90J/I/4ZOAS3VS05K63IFQldtE9y9W3ouHD9z0K89SHEX8x5fU8j/q8yEWvVp1Ayj1pE WUlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789645; x=1767394445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=pcGdIc/9DJZ3V/5WVf7vUUbl5wTlCyMzFQWvUiTQ7kw=; b=gpwdoSh7jZUEPzZApeWrzSHgK4PhIDCFJ1CPIibQWoLcEDpahuSjl7CslwqiZbj4uK jONp8n7JliJHeYEZpDI8ZPiDu6CTna9c24a/AwvFz2Wx+oWLuNC5ALnDtn2WLp1ODW0O k4kYqfgdlBOaKsj9qvmz/GT5MaFkLYF8vD6Mf0TD1/es9xSH1vMPFkFyXMLfZmq+rFyJ 3+cCw41JpTtqXhTA1Q8rz9AQn2WmGcVtEIBGOMegNTmdq1BadLBNniPrA+fk1w+2g1LP ERAGxi2x8vCHipDK8ysHqf3zkEF/+GrnisNfVf3p48w7cx+hJ8xEENvwIPEojXwrVuaz L+UQ== X-Gm-Message-State: AOJu0YyyOPEmeb71bqontPSa3/iIbR4qUyz1LopMtdVTKjZGfChmsHwF /wGDTIZFt5vsnpc5HKcK+eaL8CaGBTZH4zeJd5gnVRsneAYA/+AEmVzG X-Gm-Gg: AY/fxX6QGTESOiFnHn+g0Nr/QcOxaqzAnXYFYddv2cI7c0DdlTkAhfe6gOui+OLFR/b NU9XT2kS2r6Htu2uEV57BpcdpGCKwBIQ8HCQhN6rz1yW46c8Da7hKuRMMXw19RpUulc/KbcFbAJ 6F12kVGxFq9GKLUKio+v8ozjAI6OrjwIgw9KtScuk93isEniRwjAifvLTqdoV0K0bLHv7q4YVNq YPYIGT8Js9VnStNYctSwcjdbswqT1QqQc+sSsuvzZPcToWYp1BiBls1xs4jkwlWHH6a7VbF1pFx psyr/I5EAtce5ZCvr+LpVF7L0yjUA5HeA2hf8eMHnpogZmZs1r8FPvw5uPtravAmr2ks5Ye/kSn ZP1qDgDIYDq3ySy96QF/7As9EUJReD+MaBaaViXbjcey2IMXN7IWdVZRPPKZ0as8l/kARbgbkK3 kPOZ1WfLcn45k8 X-Google-Smtp-Source: AGHT+IE+oJ1Ft1XOQ1XU6OeA4rerw4thrh1Pz4yYg+rxMS7HW6pD48/QipbV6W5OVknyxXIG+nvg1w== X-Received: by 2002:a17:90b:5804:b0:34a:b459:bd10 with SMTP id 98e67ed59e1d1-34e921bf172mr21139744a91.24.1766789644533; Fri, 26 Dec 2025 14:54:04 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:54:03 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Tangquan Zheng Subject: [PATCH v2 6/8] dma-mapping: Support batch mode for dma_direct_{map,unmap}_sg Date: Sat, 27 Dec 2025 11:52:46 +1300 Message-ID: <20251226225254.46197-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Leon suggested extending a flush argument to dma_direct_unmap_phys(), dma_direct_map_phys(), and dma_direct_sync_single_for_cpu(). For single-buffer cases, this would use flush=3Dtrue, while for SG cases flush=3Dfalse would be used, followed by a single flush after all cache operations are issued in dma_direct_{map,unmap}_sg(). This ultimately benefits dma_map_sg() and dma_unmap_sg(). Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song Reviewed-by: Leon Romanovsky --- kernel/dma/direct.c | 17 +++++++++++++---- kernel/dma/direct.h | 16 ++++++++++------ kernel/dma/mapping.c | 6 +++--- 3 files changed, 26 insertions(+), 13 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 98bacf562ca1..550a1a13148d 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -447,14 +447,19 @@ void dma_direct_unmap_sg(struct device *dev, struct s= catterlist *sgl, { struct scatterlist *sg; int i; + bool need_sync =3D false; =20 for_each_sg(sgl, sg, nents, i) { - if (sg_dma_is_bus_address(sg)) + if (sg_dma_is_bus_address(sg)) { sg_dma_unmark_bus_address(sg); - else + } else { + need_sync =3D true; dma_direct_unmap_phys(dev, sg->dma_address, - sg_dma_len(sg), dir, attrs); + sg_dma_len(sg), dir, attrs, false); + } } + if (need_sync && !dev_is_dma_coherent(dev)) + arch_sync_dma_flush(); } #endif =20 @@ -464,6 +469,7 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, struct pci_p2pdma_map_state p2pdma_state =3D {}; struct scatterlist *sg; int i, ret; + bool need_sync =3D false; =20 for_each_sg(sgl, sg, nents, i) { switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(sg))) { @@ -475,8 +481,9 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, */ break; case PCI_P2PDMA_MAP_NONE: + need_sync =3D true; sg->dma_address =3D dma_direct_map_phys(dev, sg_phys(sg), - sg->length, dir, attrs); + sg->length, dir, attrs, false); if (sg->dma_address =3D=3D DMA_MAPPING_ERROR) { ret =3D -EIO; goto out_unmap; @@ -495,6 +502,8 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, sg_dma_len(sg) =3D sg->length; } =20 + if (need_sync && !dev_is_dma_coherent(dev)) + arch_sync_dma_flush(); return nents; =20 out_unmap: diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index a69326eed266..d4ad79828090 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -67,13 +67,15 @@ static inline void dma_direct_sync_single_for_device(st= ruct device *dev, } =20 static inline void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) + dma_addr_t addr, size_t size, enum dma_data_direction dir, + bool flush) { phys_addr_t paddr =3D dma_to_phys(dev, addr); =20 if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu(paddr, size, dir); - arch_sync_dma_flush(); + if (flush) + arch_sync_dma_flush(); arch_sync_dma_for_cpu_all(); } =20 @@ -85,7 +87,7 @@ static inline void dma_direct_sync_single_for_cpu(struct = device *dev, =20 static inline dma_addr_t dma_direct_map_phys(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, - unsigned long attrs) + unsigned long attrs, bool flush) { dma_addr_t dma_addr; =20 @@ -114,7 +116,8 @@ static inline dma_addr_t dma_direct_map_phys(struct dev= ice *dev, if (!dev_is_dma_coherent(dev) && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { arch_sync_dma_for_device(phys, size, dir); - arch_sync_dma_flush(); + if (flush) + arch_sync_dma_flush(); } return dma_addr; =20 @@ -127,7 +130,8 @@ static inline dma_addr_t dma_direct_map_phys(struct dev= ice *dev, } =20 static inline void dma_direct_unmap_phys(struct device *dev, dma_addr_t ad= dr, - size_t size, enum dma_data_direction dir, unsigned long attrs) + size_t size, enum dma_data_direction dir, unsigned long attrs, + bool flush) { phys_addr_t phys; =20 @@ -137,7 +141,7 @@ static inline void dma_direct_unmap_phys(struct device = *dev, dma_addr_t addr, =20 phys =3D dma_to_phys(dev, addr); if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); + dma_direct_sync_single_for_cpu(dev, addr, size, dir, flush); =20 swiotlb_tbl_unmap_single(dev, phys, size, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 37163eb49f9f..d8cfa56a3cbb 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -166,7 +166,7 @@ dma_addr_t dma_map_phys(struct device *dev, phys_addr_t= phys, size_t size, =20 if (dma_map_direct(dev, ops) || (!is_mmio && arch_dma_map_phys_direct(dev, phys + size))) - addr =3D dma_direct_map_phys(dev, phys, size, dir, attrs); + addr =3D dma_direct_map_phys(dev, phys, size, dir, attrs, true); else if (use_dma_iommu(dev)) addr =3D iommu_dma_map_phys(dev, phys, size, dir, attrs); else if (ops->map_phys) @@ -207,7 +207,7 @@ void dma_unmap_phys(struct device *dev, dma_addr_t addr= , size_t size, BUG_ON(!valid_dma_direction(dir)); if (dma_map_direct(dev, ops) || (!is_mmio && arch_dma_unmap_phys_direct(dev, addr + size))) - dma_direct_unmap_phys(dev, addr, size, dir, attrs); + dma_direct_unmap_phys(dev, addr, size, dir, attrs, true); else if (use_dma_iommu(dev)) iommu_dma_unmap_phys(dev, addr, size, dir, attrs); else if (ops->unmap_phys) @@ -373,7 +373,7 @@ void __dma_sync_single_for_cpu(struct device *dev, dma_= addr_t addr, size_t size, =20 BUG_ON(!valid_dma_direction(dir)); if (dma_map_direct(dev, ops)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); + dma_direct_sync_single_for_cpu(dev, addr, size, dir, true); else if (use_dma_iommu(dev)) iommu_dma_sync_single_for_cpu(dev, addr, size, dir); else if (ops->sync_single_for_cpu) --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C21CF30F953 for ; Fri, 26 Dec 2025 22:54:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789658; cv=none; b=i7jFKWwRyASr4b/rrkoPhQtqwSOZwJM3zazxA8lvUkVC/7nhjyNesr3YGOol0p+8fS82jIpcmWmlHUi8HgXhdlHyyYAHUEeuid8Tv1MuHDvUTp5H+Lhdk+OSqwPktowBPJH7Y1FZLQFVaH39uqcCOvZeIuQcEl7hT3MyI7awlbE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789658; c=relaxed/simple; bh=tT7PtF41A77X6yZ13NHHIhgroMHXFQkhSKzdCIl4WmQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XrB+YDkXmOrHmTeBc3BBZnCCaxK4P+zjZcLjpUXruQpOj7Sex/87Aq4XuxoQ9KdYbndA1XVVxOFYErt9lKXUybz/kMDUogwaGKPMZuRD8yChFyKnHZbETZaveU9b6mCi/dX9ZjeZUYxsMREzT7/cXHQtglH8RMUehqtPvxHfqGQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Dpj0wSFS; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Dpj0wSFS" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2a0b4320665so115250365ad.1 for ; Fri, 26 Dec 2025 14:54:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789652; x=1767394452; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9mp0y7L70LQWZn3MNrpl8AtI1agZhiyYjI11ja2HdtI=; b=Dpj0wSFSKcOWJBGPOKjR+auzyhV86uA9t07CvqbDGVFufly0lvoylgT7MbIgNrkiXU QojNhx5X3E7odtiXbhJIbm20FnwYZJKmJVeEcNtFyl1L2w0z+eFiccNCDIktFvnE1tsT QsOsQL/mZ3pEsaNhSN+fT/q7LfOyCgkbuz/HTlu5hxH6wZOApsvE3uV7blU20FSwcJAs TgyDQ4rDJVd7SIYMZQYZrGl5qvplimStBgaIBp29xTbEocq5+mnS9sfjy0tWolRwBTCz c+coxnAJnEjIoN9PnEVaWvIsuDn7mBwNUzeMDikHM9QWkpWCdlVJdFv7gLg6DfU99SrC 2mpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789652; x=1767394452; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9mp0y7L70LQWZn3MNrpl8AtI1agZhiyYjI11ja2HdtI=; b=q1Z0W5XZLvQwIcO/9immmDMWdLsTYnsuXlFQ6vk4TRVPO1dOAoaRYtZVLrJxp7TtsW 8q1qZar1YaOWEbnnfju3AqAo1K7sT5cb4XPJun2zs+tA+h8TrhrNvpsORvDZu4JTX5Wo B1oX+GuL+eh2KkbZHgSywvJChHBwhsRTipp85lZLYua2PqOQ3l0DpthpHr/6bEV15Cr3 7Ds9nmpfv68uTXfaNoCJvbWEwY2NwOt4WZYAJypZHf+h0pSWGdtyovhA+n93i38UBlkQ pzXlvo9LQtiVbRTXvKa7thAt5uuqLFrJVJTbeLSPxpedAOnVlKmi9zmBFNbTYSFPtBqh 3h+g== X-Gm-Message-State: AOJu0YzKCxk1m27EdXr5xHR+z4/FybDfTUH+lRgHSXgP9f/jMf/Jq3Vm 8JmriIkJsNxxB0wY6ehZDMO+yqxllZPnnA8mwa2DyZyHjLVN2v5yzdkl X-Gm-Gg: AY/fxX7XfBlxifZubQSbjH1y65JmpPEfM+r09r7rbTo7DigxoLDi5AhP1aukynnkMum qjkCGVEsulbP2tgTxNjqQ6gtbXAAptqsPHtAIxh5tN7Aip86+xMSuPKgXnERxaysy6NY3j1jM7H NYAfn/BoUNdMWTSgX19D7H9PBC/vSEFgllwVjt4w2K7KWBxTcBs+yjCpzaBOBGKF7qKh0fh3T0X VNx11nSUoyCczxY9fR7MF0RR/AJmD9ciVvwecKkRzFyPBPejhsvxeQVKRr7yT7ED0/NZ8GG4WwD jUouBpxQMo3Hr7opeI0Lt0X08IKQrxbAznC+hc1/AJnZxMfpW2f4RyWshutUy4e9bcAahX/ZueL jr4iaKus5j0rM2H30dkj339qXw8HWvGlz8ZEt+czkDIKtR3dJLyWIVro0IKnoHp6TKGTc6KRw1b Nx3k7aNuwNPxwS X-Google-Smtp-Source: AGHT+IH3Qyivahvz1Re89nhjkOGapFSbuTfRVGKZtZJZOQKn2pFT9vJfuh0AHXHFbf1hI+2p00nHuQ== X-Received: by 2002:a17:902:f607:b0:2a1:4c31:335 with SMTP id d9443c01a7336-2a2f2717b88mr227808145ad.26.1766789652012; Fri, 26 Dec 2025 14:54:12 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:54:11 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Joerg Roedel , Tangquan Zheng Subject: [PATCH RFC v2 7/8] dma-iommu: Support DMA sync batch mode for IOVA link and unlink Date: Sat, 27 Dec 2025 11:52:47 +1300 Message-ID: <20251226225254.46197-8-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Apply batched DMA synchronization to __dma_iova_link() and iommu_dma_iova_unlink_range_slow(). For multiple sync_dma_for_device() and sync_dma_for_cpu() calls, we only need to wait once for the completion of all sync operations, rather than waiting for each one individually. I do not have the hardware to test this, so it is marked as RFC. I would greatly appreciate it if someone could test it. Suggested-by: Marek Szyprowski Cc: Leon Romanovsky Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Robin Murphy Cc: Joerg Roedel Cc: Tangquan Zheng Signed-off-by: Barry Song --- drivers/iommu/dma-iommu.c | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 6827763a3877..ffa940bdbbaf 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1849,10 +1849,8 @@ static int __dma_iova_link(struct device *dev, dma_a= ddr_t addr, bool coherent =3D dev_is_dma_coherent(dev); int prot =3D dma_info_to_prot(dir, coherent, attrs); =20 - if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { + if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) arch_sync_dma_for_device(phys, size, dir); - arch_sync_dma_flush(); - } =20 return iommu_map_nosync(iommu_get_dma_domain(dev), addr, phys, size, prot, GFP_ATOMIC); @@ -1995,6 +1993,8 @@ int dma_iova_sync(struct device *dev, struct dma_iova= _state *state, dma_addr_t addr =3D state->addr + offset; size_t iova_start_pad =3D iova_offset(iovad, addr); =20 + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_flush(); return iommu_sync_map(domain, addr - iova_start_pad, iova_align(iovad, size + iova_start_pad)); } @@ -2008,6 +2008,8 @@ static void iommu_dma_iova_unlink_range_slow(struct d= evice *dev, struct iommu_dma_cookie *cookie =3D domain->iova_cookie; struct iova_domain *iovad =3D &cookie->iovad; size_t iova_start_pad =3D iova_offset(iovad, addr); + bool need_sync_dma =3D !dev_is_dma_coherent(dev) && + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO)); dma_addr_t end =3D addr + size; =20 do { @@ -2023,16 +2025,17 @@ static void iommu_dma_iova_unlink_range_slow(struct= device *dev, end - addr, iovad->granule - iova_start_pad); =20 if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) { + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) arch_sync_dma_for_cpu(phys, len, dir); - arch_sync_dma_flush(); - } =20 swiotlb_tbl_unmap_single(dev, phys, len, dir, attrs); =20 addr +=3D len; iova_start_pad =3D 0; } while (addr < end); + + if (need_sync_dma) + arch_sync_dma_flush(); } =20 static void __iommu_dma_iova_unlink(struct device *dev, --=20 2.43.0 From nobody Sat Feb 7 23:11:03 2026 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6660C30F93A for ; Fri, 26 Dec 2025 22:54:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789662; cv=none; b=c9j+uR+kg9yZ5OAxayisLx/f+BFpO/AoVweWBPA2WNF9hgNQqmE+W4qObhKvwL6nsKGRI+KyXHO4mfKXibeocyosFC1vcMBAxfoZIFgxROA0rtMT5vI9KEXonWHZ3zMmj5DRkVYuw52O3ArpsKktKK7rfpf7xP7sAb53ZTE3NwA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766789662; c=relaxed/simple; bh=ShBsoH34IqplDCbKkd/jSxuIHIIWI8ySxMBj3nK3wpU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mN6JmVCus28lAdECh8Oy3qJwcF2CKwBOrmVCaZsll2Bv8aJ3U5CtFBs3bwaz+FTUIDhdK7h5FX6wXT7N4oMnKJoQSaeL1hRxG4wsC6Kr2isdF819DxUcXJ1MKfyOm3VVfXDXyjVGcO1MdoVeK44anYq6qGQaipaBABOxXTMokHw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Jz9HL1TB; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jz9HL1TB" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-34b75f7a134so5644132a91.0 for ; Fri, 26 Dec 2025 14:54:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766789659; x=1767394459; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=beONI3EcXYpecg+Q1bsEm+B77vz1JdIzHQgAsq5N5yU=; b=Jz9HL1TBMl9Wxg2200JL+i/dDMiV8QfG6NRUf9/H22ItkwzTxyvogQtRNPl2XF1awe agRnUtm5VoCvqrcTJ35IUM3f1Z4xnldYZrINekTeaJ/NGuQ0R1sybgu1RwQcGfvh+gMA GQtHfkIkKzcGPL05hCsmEUhOSXB3CM5YwwU484qrEtRvHObfD1RXkafxhbrpoEqMCJlj tUHcosO56hoLPo5Ya/2f13CE5fwBGgsh542QdSK4TMQKzItA4pNafagcKVuM4kfGVmTc 0gZncF/Yh6LCz7avBUx0Qw1Zxqng7kzhXGpYnPIl6mMRjkMoeNIQRPE1a1YAwn1YVdqU Hl7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766789659; x=1767394459; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=beONI3EcXYpecg+Q1bsEm+B77vz1JdIzHQgAsq5N5yU=; b=YtgH8mAMXhKzqyk2P0gQviRa9/LbX1Kfa5Mjz733jiPYk5z1THILJx1GLn3W18GYAP J7rYVwAotE5svA+DKJfsx+8RMbwN1I1wAe14yEG98SDccwc0CvNpVnbjoHa+WlRnYUxJ dCmp3EiZ0/3te5bZ3AW953H4jxC0D/oR8BckD6hKGEyqK8YUqqqnudHooCbtz09yUAD2 aDE8/QdLQJD3NosrP6hbuPJV1ZpaK9AVt9AEtHPPWi3+9TqTpvUedsUzbA16MX3pbQOy 04qONR2KW4TyBtq2SXNVSp0JAV0ovcQ2K5KEnKXRGDSONyRWVE2mxFBOL9JsfIGJS0K5 e/mQ== X-Gm-Message-State: AOJu0YwAceE33VNPU5wYHx6KFzfjSZs6V/JdmWmzmsqnLYkhtUy8bvBk aWLJ/TIJF5D6fsJus8hOEu//O8CNfgHXYT0dgfS9U8kt5CP4M1MOyHFN X-Gm-Gg: AY/fxX6MhZDf7uecowN1hzJsrcuiIRShY41YvaHmgbEM9dKd1yeB1n+tCuQ0nBYYSSe 9zwYFU31tfB+17RUzFzksK0P9rdlqZ9WHvR8bo9pqJPSK5xN9QI2epWhKA43ezd49ufubfExDUy Bau7u+ImPrkicAqUGSz30aNZRrNSxlFHdImCC1fAvBvQke6hgNIr5cHc6QkTHVfctu+PaiY5o8q gW8dKJo76pCkLjYbmBUvS9Wa1gaOdf73ZDHXUbuoTIn34AO7W5rxmqQsFGeDf4+GgUhlqv+QhvV 8Npxg6YgJ89OOk3/UBWZNdIZHeB/VgnVp9DI1L3QYAhE/nq3pJREuaUA09qjLBQ4Q+FIOwyJ/Vz ZWxRbwF/lhSW/EXRs+p6/uDwF0KZNKor7PbTpccjvLOpvhHtZEcjFuONLG5i+83pKMd95F3/wLw nq+jJspItpTvNT X-Google-Smtp-Source: AGHT+IF3FBOgnhAE+C4yK80l6tVlXmB4Rfi7fJf1frV3RXQh1+Ry7oysljXY9g9IJMnKqHJpP4IhrA== X-Received: by 2002:a17:90b:2787:b0:33b:cbb2:31ed with SMTP id 98e67ed59e1d1-34e91f6398cmr20025228a91.0.1766789659345; Fri, 26 Dec 2025 14:54:19 -0800 (PST) Received: from barry-desktop.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34e772ac1acsm9981428a91.9.2025.12.26.14.54.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Dec 2025 14:54:18 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org, xen-devel@lists.xenproject.org, Barry Song , Leon Romanovsky , Ada Couprie Diaz , Ard Biesheuvel , Marc Zyngier , Anshuman Khandual , Ryan Roberts , Suren Baghdasaryan , Joerg Roedel , Tangquan Zheng Subject: [PATCH RFC v2 8/8] dma-iommu: Support DMA sync batch mode for iommu_dma_sync_sg_for_{cpu, device} Date: Sat, 27 Dec 2025 11:52:48 +1300 Message-ID: <20251226225254.46197-9-21cnbao@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20251226225254.46197-1-21cnbao@gmail.com> References: <20251226225254.46197-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Apply batched DMA synchronization to iommu_dma_sync_sg_for_cpu() and iommu_dma_sync_sg_for_device(). For all buffers in an SG list, only a single flush operation is needed. I do not have the hardware to test this, so the patch is marked as RFC. I would greatly appreciate any testing feedback. Cc: Leon Romanovsky Cc: Marek Szyprowski Cc: Catalin Marinas Cc: Will Deacon Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Robin Murphy Cc: Joerg Roedel Cc: Tangquan Zheng Signed-off-by: Barry Song --- drivers/iommu/dma-iommu.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index ffa940bdbbaf..b68dbfcb7846 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1131,10 +1131,9 @@ void iommu_dma_sync_sg_for_cpu(struct device *dev, s= truct scatterlist *sgl, iommu_dma_sync_single_for_cpu(dev, sg_dma_address(sg), sg->length, dir); } else if (!dev_is_dma_coherent(dev)) { - for_each_sg(sgl, sg, nelems, i) { + for_each_sg(sgl, sg, nelems, i) arch_sync_dma_for_cpu(sg_phys(sg), sg->length, dir); - arch_sync_dma_flush(); - } + arch_sync_dma_flush(); } } =20 @@ -1144,16 +1143,16 @@ void iommu_dma_sync_sg_for_device(struct device *de= v, struct scatterlist *sgl, struct scatterlist *sg; int i; =20 - if (sg_dma_is_swiotlb(sgl)) + if (sg_dma_is_swiotlb(sgl)) { for_each_sg(sgl, sg, nelems, i) iommu_dma_sync_single_for_device(dev, sg_dma_address(sg), sg->length, dir); - else if (!dev_is_dma_coherent(dev)) - for_each_sg(sgl, sg, nelems, i) { + } else if (!dev_is_dma_coherent(dev)) { + for_each_sg(sgl, sg, nelems, i) arch_sync_dma_for_device(sg_phys(sg), sg->length, dir); - arch_sync_dma_flush(); - } + arch_sync_dma_flush(); + } } =20 static phys_addr_t iommu_dma_map_swiotlb(struct device *dev, phys_addr_t p= hys, --=20 2.43.0