From nobody Sun Feb 8 09:12:43 2026 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28BDD1F5F6 for ; Fri, 19 Dec 2025 05:37:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122638; cv=none; b=j+h+nHd7hG6UHZyI/Qm28YrIxH37P9o13ytIND1iYf50WdjowepJXothYEwocBsys8YaMLTr+sd5n7AIZnInS56GVFKweHIZ4jspu+qCO9Yh5ePcThltvT1/9o16W2H6nyb0fLta6sMIxH38s3B3JDTF2CVzhZ9VcoC1GMHxi3Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122638; c=relaxed/simple; bh=oXHMuHXV30gXK0zo04y1oVqzgP5bQsiQpukq82bkvqo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version:Content-Type; b=JAq4A2/OUiaxxhvbK/1C6BCLAoZc3qNxOuhb1DXh6C7KQSsvquUIHjZPz48tnbscp1rPLvFj29GRufs5TlcmSQyvYO7KLfm8tFfk5YuzEWm3A263RUtOZATMc/ZzymjHJpOc2tZjOcLqLFpQchy8MX8rIcIcOookxH38kKFhw7M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=L49GxZgR; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="L49GxZgR" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2a0d67f1877so17058225ad.2 for ; Thu, 18 Dec 2025 21:37:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122636; x=1766727436; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/NkbhnSjmQqWyentKViHSvnUd2t4Anb4J6Vl3+26IzI=; b=L49GxZgR/udXIBxm0qL/iLFe6nEGLnEHuOf1smhFvXSVChamREmFNiGlLh4EgQdK0t u4U7FaIs1+N6ahVxKvoRd+P4Us+yrwfnE0lzo17YMHo4pPOu3itj7ghjO6RzacbIgBVm 4OID9IrPJYWwGXmuGS+VGeWlUq4IpWW1zrg9BN5rxS7KvRG2pqHHjMDUC7Dr7F6wJDdu D2x8g9Jr7HGXjmVrIb44Lu5iJ4biSSWJxlg0hCAW98yrJb64KcQ84s6/xcM4ktWottt/ pgikJvs+983xVkOA5+XbrZX/h+kjZCYflkGQe+fhOGaB4yiL6zHRlADqc/XPyua0RNfW OkqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122636; x=1766727436; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=/NkbhnSjmQqWyentKViHSvnUd2t4Anb4J6Vl3+26IzI=; b=cq8m7c+rlKKS48Cvdfl1KTkvHnQk0tJah4cvAkneNMzq2YPqTcKXo3wdNS5ERw9Ig8 kVdyo+uYymgvlXf7RUh77ioZ0vJChKgvBACV4zpnyyppNrQIkQJjuvlnXziGMHLprQWw lBGGsjmx8GT5IyJOXZprzElW+zAh0vZgcovn2JT3yM+leRsDanRuvboEiqgeg5oHgvYq gv82mviKtJAKD38EXsBerigigybwthQFXtGO1qzXmFoJgOFUaow19VDlDNc8+2jIkyzj xg6eawWdRNbzyQObn2EWrdHrPWaCTTF8MqOifCgy0FU/dUzyaPSVrED2D5ytbakoAJc7 tylQ== X-Forwarded-Encrypted: i=1; AJvYcCWD6+Z6cs+WS7z+dpeik2CWfwOFnKIH66URtgzoQtpOkF9D7ewppDgN/CLXYwbzfCRvoipHeWt3//aqvuM=@vger.kernel.org X-Gm-Message-State: AOJu0YweJMhjHawuPYbN4E19m9nqtW/ZEg5dMltLTDAB5whj0zvbrLRg nfK+uYrheI5kBdTIIk+635Tr6pO7tW/2QdOUMyc7cnDBUMcK+FkMQca0 X-Gm-Gg: AY/fxX7GQF5jl1u26UVjyEUIMeM33CuUXi86Ucoyy3WWtOXRGCt5i527/JOwPTAmxFP 4gyet54JgsSFGaExNlWdxk2radVyvUAZQYWOCpKgV2raU87cwZBY3TSdV7Im++IiqCvlCJ65hTI 6iqQeKgMLsv0I+6gdgb6oB4BcIe/6qrFH1hRog2Bul1cYKDT8jEfeatjRfZpgtbQn6QI2RTo/EQ mE3kNb9+L8F/dc2RfhIifpHUvc4B6yaGq/ZH83+POIhwXjeuXKFA+KMlBA6pqVdElgK7FENEmnM PtHvNKc34p0A3zdpTru80sLwHzf8GsBYoELOw6OfUS9whNXlPhLD5149O+hNQE6xu0HUDnwIbQ3 22oQFPXKw74UjZwMmmCDrt1wS4XHHsB//001ModAb5WIDhPhHKsflGYD62EIz8Yws4f6e30Puko 5LLqo7/aLReSsnAdluhGas0ie12XK7dw== X-Google-Smtp-Source: AGHT+IFcyO2/N6WJqTTUC8Xpz+F+HK73QiRxiWGBBRL+WIr6e3loB6zfy831gEjAtlMYHTHv9HUDPQ== X-Received: by 2002:a17:903:2290:b0:2a1:3cd9:a739 with SMTP id d9443c01a7336-2a2f2830ba1mr17129295ad.43.1766122636315; Thu, 18 Dec 2025 21:37:16 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:15 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com Subject: [PATCH 1/6] arm64: Provide dcache_by_myline_op_nosync helper Date: Fri, 19 Dec 2025 13:36:53 +0800 Message-Id: <20251219053658.84978-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable From: Barry Song dcache_by_myline_op ensures completion of the data cache operations for a region, while dcache_by_myline_op_nosync only issues them without waiting. This enables deferred synchronization so completion for multiple regions can be handled together later. Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/assembler.h | 79 ++++++++++++++++++++++-------- 1 file changed, 59 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/as= sembler.h index f0ca7196f6fa..7d84a9ca7880 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -366,22 +366,7 @@ alternative_else alternative_endif .endm =20 -/* - * Macro to perform a data cache maintenance for the interval - * [start, end) with dcache line size explicitly provided. - * - * op: operation passed to dc instruction - * domain: domain used in dsb instruction - * start: starting virtual address of the region - * end: end virtual address of the region - * linesz: dcache line size - * fixup: optional label to branch to on user fault - * Corrupts: start, end, tmp - */ - .macro dcache_by_myline_op op, domain, start, end, linesz, tmp, fixup - sub \tmp, \linesz, #1 - bic \start, \start, \tmp -.Ldcache_op\@: + .macro __dcache_op_line op, start .ifc \op, cvau __dcache_op_workaround_clean_cache \op, \start .else @@ -399,14 +384,54 @@ alternative_endif .endif .endif .endif - add \start, \start, \linesz - cmp \start, \end - b.lo .Ldcache_op\@ - dsb \domain + .endm + +/* + * Macro to perform a data cache maintenance for the interval + * [start, end) with dcache line size explicitly provided. + * + * op: operation passed to dc instruction + * domain: domain used in dsb instruction + * start: starting virtual address of the region + * end: end virtual address of the region + * linesz: dcache line size + * fixup: optional label to branch to on user fault + * Corrupts: start, end, tmp + */ + .macro dcache_by_myline_op op, domain, start, end, linesz, tmp, fixup + sub \tmp, \linesz, #1 + bic \start, \start, \tmp +.Ldcache_op\@: + __dcache_op_line \op, \start + add \start, \start, \linesz + cmp \start, \end + b.lo .Ldcache_op\@ =20 + dsb \domain _cond_uaccess_extable .Ldcache_op\@, \fixup .endm =20 +/* + * Macro to perform a data cache maintenance for the interval + * [start, end) with dcache line size explicitly provided. + * It won't wait for the completion of the dc operation. + * + * op: operation passed to dc instruction + * start: starting virtual address of the region + * end: end virtual address of the region + * linesz: dcache line size + * Corrupts: start, end, tmp + */ + .macro dcache_by_myline_op_nosync op, start, end, linesz, tmp + sub \tmp, \linesz, #1 + bic \start, \start, \tmp +.Ldcache_op\@: + __dcache_op_line \op, \start + add \start, \start, \linesz + cmp \start, \end + b.lo .Ldcache_op\@ + .endm + /* * Macro to perform a data cache maintenance for the interval * [start, end) @@ -423,6 +448,20 @@ alternative_endif dcache_by_myline_op \op, \domain, \start, \end, \tmp1, \tmp2, \fixup .endm =20 +/* + * Macro to perform a data cache maintenance for the interval + * [start, end). It won=E2=80=99t wait for the dc operation to complete. + * + * op: operation passed to dc instruction + * start: starting virtual address of the region + * end: end virtual address of the region + * Corrupts: start, end, tmp1, tmp2 + */ + .macro dcache_by_line_op_nosync op, start, end, tmp1, tmp2 + dcache_line_size \tmp1, \tmp2 + dcache_by_myline_op_nosync \op, \start, \end, \tmp1, \tmp2 + .endm + /* * Macro to perform an instruction cache maintenance for the interval * [start, end) --=20 2.39.3 (Apple Git-146) From nobody Sun Feb 8 09:12:44 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B1F62C0287 for ; Fri, 19 Dec 2025 05:37:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122645; cv=none; b=l1OcNfSCTUAHQUj/twgcT+TC5hUalyJuU6lsGTjN0MaLxNSIpJxs/KjHz36reKQ4k12hCfDgkvJlKx5WazT1F1e59dvX0f9S1UOJl8X6Hh2eWLAltFIIGzP8yG9b+Sgx9LNCxU3sw4kXBrx2HOq5SGUvZZoMCK1D31R+wL6qKms= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122645; c=relaxed/simple; bh=xKWaFiF4Stck1c6IC9qPKTLGzX1CaZt20WVNbuWFfjk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DKzcigvL8HfK8fSoS7x9kE/c0qFAgGbJbBVmFZuOVOkKpuzOlobD/q9ufqu2EHTsofoPEUVrOmtRQ83M9UjYyWqy4NN5DCdPe/qsULWIClwRXwFFZIcbrNmZqAUs6gc43O07QP41zq3Dr8CTi0bXhN0D78i29PRlnBbi6Nz072s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AIzxvAQV; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AIzxvAQV" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2a2ea96930cso12635555ad.2 for ; Thu, 18 Dec 2025 21:37:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122643; x=1766727443; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cDodzdvyFbknC8FIp1mljLdhQ9A1IRI/HhNeyykhxII=; b=AIzxvAQVZnq3o5zBocA7qPBczYpDlgkkpQELuTFP+ese03ksM69CwGc3i6WTH3vUzr 5/fWWOu+mJj2L2pUn6V5PVQL1AVu9rQ7ZVDqenu2L6pA3IgeN2kzALtiHewaJhc/Rf8d y4OUTtyOL3KmUj6vtqDJDI4sF5ZrX6FyQsirEIev5G+cKa35ft9Dx+4PrHFSSR7kyRXj uKZEzwIzaBL80Zapi/l85/S3P2cGtAaUiJmwhTQBMaAQhNbSXx7AfsS3X7fCQebzbWa5 Ul1JCu/jJV98jDerbnLcwkX2ZFjK7V4D/JeISfJ7ftaq6Lkm2ycJHyKAQJSTv1GRCX/y bfRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122643; x=1766727443; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=cDodzdvyFbknC8FIp1mljLdhQ9A1IRI/HhNeyykhxII=; b=eiQlygPvVHniEzTo6pGwvd1+ig3Grc0x4Yff8BVe4zlchvhC1+zO72JguV/Tfw65SN MlEeHaaTwve4jSR+yDjsmYHc1QNTNaRPr/xXdXbSCJPL/oEg7k/7yMN28FyuILbBjbB+ cCZqZRmXB3Qqv556hoFD40SSfkHC3IK4CQTcnbCMUbm2xCU9mVIACI2wjhdq3fgng1RL OrKCpjVHS9vzc2djuuI69xr4JKftYAdHxyH0AqXPkhIfl5oKb4p+dhyxXuectJE7qKq7 W2dgDNG1PXu59m//OAnZa+RnmvDt2zc1pDDUPyUwXOhsxS/hQ2/2w/IJCn0VMeeFwE1C h5JQ== X-Forwarded-Encrypted: i=1; AJvYcCUl99Ls/Nx/f3iUxdYT7+XtqbMbD8iV1c3eN8MNT6OTL3OqyHVnyf2dErosW8WCoGbjMK7BuhLAVFyxakE=@vger.kernel.org X-Gm-Message-State: AOJu0YzTYVy3XYBG6j+dKyStAcbTB5aOkrSCbVjOESjwMHSEaYEySMns oXDqzcnujv7VBhwoxUlo0HWQBFgT5saN9RiUZzUAt1qbkPSccSZQ/QE+ X-Gm-Gg: AY/fxX5qJn3oALyx6x39OgycuDBztEHYUombpCh+L9Fu8PtAw56mn6vDSwfIcnBr7Io nCjK3HToteiaEBWnxKXtx00o8x/ZROmnaicG0bUfZ9KmfPsVdi3L6X8BKIKlVUP3o52/eQXmP36 4tDrj0fAEVSGYMs4JPI6rOyvXgRJ5KjfA615aJa7wySOB9rGGcLdjsKIdRTBdFZuvtEN4kIfKrx QTLeBgTK6UYg5SMnguemPMlTx1nXtZj1seLPp5yJBQ8l+h4jxYdsANYw8HNUCUXaZGgPA6pxM/H XYOWcXeaJ4PWl3swVVb+aKUYiK4DEbUEOANtiO6u8mjAHVcv+bJVmCAlW7AbzuDfRsH9V1uFfLi uFuAwvD5+kQYiDJI6cpyeSm92UX38flm7N0IKgh7tKuUDrBwCMYnRQycg6V/5i1EYeeDbUN9p+h yxirnVnMXwPfC/JmfiOi0= X-Google-Smtp-Source: AGHT+IGzuZJ82GUInMpHHsz5Jh8P3tFch3LI3d67HGTpBsqXEpuwzDvEBW/+nSfCKMrEedniTwbWSg== X-Received: by 2002:a17:903:1a67:b0:29d:9755:9bd8 with SMTP id d9443c01a7336-2a2f23265b7mr14068335ad.22.1766122642560; Thu, 18 Dec 2025 21:37:22 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:21 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com Subject: [PATCH 2/6] arm64: Provide dcache_clean_poc_nosync helper Date: Fri, 19 Dec 2025 13:36:54 +0800 Message-Id: <20251219053658.84978-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song dcache_clean_poc_nosync does not wait for the data cache clean to complete. Later, we wait for completion of all scatter-gather entries together. Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/cacheflush.h | 1 + arch/arm64/mm/cache.S | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/c= acheflush.h index 28ab96e808ef..9b6d0a62cf3d 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -74,6 +74,7 @@ extern void icache_inval_pou(unsigned long start, unsigne= d long end); extern void dcache_clean_inval_poc(unsigned long start, unsigned long end); extern void dcache_inval_poc(unsigned long start, unsigned long end); extern void dcache_clean_poc(unsigned long start, unsigned long end); +extern void dcache_clean_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_pop(unsigned long start, unsigned long end); extern void dcache_clean_pou(unsigned long start, unsigned long end); extern long caches_clean_inval_user_pou(unsigned long start, unsigned long= end); diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 503567c864fd..4a7c7e03785d 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -178,6 +178,21 @@ SYM_FUNC_START(__pi_dcache_clean_poc) SYM_FUNC_END(__pi_dcache_clean_poc) SYM_FUNC_ALIAS(dcache_clean_poc, __pi_dcache_clean_poc) =20 +/* + * dcache_clean_poc_nosync(start, end) + * + * Issue the instructions of D-cache lines for the interval [start, end). + * not necessarily cleaned to the PoC till an explicit dsb sy afterward. + * + * - start - virtual start address of region + * - end - virtual end address of region + */ +SYM_FUNC_START(__pi_dcache_clean_poc_nosync) + dcache_by_line_op_nosync cvac, x0, x1, x2, x3 + ret +SYM_FUNC_END(__pi_dcache_clean_poc_nosync) +SYM_FUNC_ALIAS(dcache_clean_poc_nosync, __pi_dcache_clean_poc_nosync) + /* * dcache_clean_pop(start, end) * --=20 2.39.3 (Apple Git-146) From nobody Sun Feb 8 09:12:44 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AFEEC2C0287 for ; Fri, 19 Dec 2025 05:37:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122651; cv=none; b=FQvmxjK/7lg5Mu6bPqIcjVi9rqRSsjrBI4cIZ+ZNfuG5OTfduYstE5mbeyVIAzzm6sDSeONOiq8/hWCnHNHDB8VVVeVc1/TwjrMDguHbeQAhS0Um9nbn7MJUR556o0Lrbe3f5eKA4CGPMbY4Av6VeX9jLMdnFCsa1G3ACYZ8VpM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122651; c=relaxed/simple; bh=hlJGkag1/m+BajHKxwSoDZk8vWfVJUTYJ+ASwnoAf04=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rLJtFipGoCzuJHGgStIgWIErxHXGPMi0T247vlUVklFY6+MmMLmo/hlVBE2COsOj2OEIoc1saN81J1DgJL8qh7FinidUguI30c5fdivwRbef0igWy4JkX3PrZZvGvdVdP6YS1BLUC56/FVszklAzyYDgegLWppDpWQNrDrA+IZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=SUqR0xLz; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SUqR0xLz" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-29f0f875bc5so19075865ad.3 for ; Thu, 18 Dec 2025 21:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122649; x=1766727449; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+BUB05Xw0z/21o0FdpEbUVLloFvEcORujjQKbT7R+Vg=; b=SUqR0xLzLSBiFcvJMSTZDzNK7u+XCnGICkXG3KQCF41vAt4HFklJmcSlKoTM1G6xW+ nr8YTfWMTDPFBVVmfKJI+FjDxrN3FvYBfvXch2FXwIZF5FIoROUSqIEN2LKJjEEm6yPx Jpr0B3lsgnZNXsmaUOg9eSBLpgilxHONkBxyXePTfyG6Dwn8PTxNMO+laXlxMboSvWEP s5WmwUpLHbn6JTYJ3c4qAWsn6PD1nDQg/arqV5f51tbDtUejuJjsob7WnRRN0BKQJceh azfBbJE+7HK2V7SJpOMGVy8ErzHBWOHsGe5XkMRDnmcsbiHVGjvrz28UCGBER1spbTOB efJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122649; x=1766727449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=+BUB05Xw0z/21o0FdpEbUVLloFvEcORujjQKbT7R+Vg=; b=OXQqChWNGqk9dU7bIXxs2cX+fpbonwS/ZYN0fiIzG1bIk6p8qtftvRNn49LvYxA2QQ U3ge5QOwLqTc8ZoLkIiCZ0fLYj2AjKlBDl6dfOjWeCRGng43lsJHKeRJrfWYvYjR5Vzh 0+p2+wb2wZqTRhnSfrO6GnZHGZ9JSxjPtlgP7MS1sxaalSuLzyPN7A36E+72hiHAj2UN Ed8jXRB1TqejH3Q0Huc+eUd6gDdl7rlyRK5RySgPhc3wWhfeinMIxjkXhFVxCdWenhoS Zcea6d3ZydgtjdCgV3SUmZ2N/0tpbKJwik4QUczDsMPQvVfCTXTXMZZ/OyPxlP7UqIcB 798w== X-Forwarded-Encrypted: i=1; AJvYcCU/v1aZU4tEtQIiUeYQm7DolOl3njvw0kEEaBkCUZqEpDgiaz3FL2n5swjX4MSPV5OvrHIczv46kLXof/8=@vger.kernel.org X-Gm-Message-State: AOJu0YyPQR/lXqrO8oxUBGVLvkVUk3YsX8jokL4c6OgfXzH+y6fVAOKw FW+GiTMe77nyPSp0WwniURdlqf74M3XlcTmY95puoqBkeig1oqQxXB2H X-Gm-Gg: AY/fxX4x0hjR0GEoGdRVwTE+lAfcJ7SRnkDrk13Ach6+5TpdHCD+KrAMPlkpJCB7Cdt Y29GeSN32tTywO8iXN04f2vyEAwvNz02DfPqKNFjEEN3tHgAnhq5Zks2XVgwX+R1ljjOS4vUosy 8fQ6QAqyXGqnNxIuXFNHxashBUL5vZPxLOxnt47wKmSDWMnf+O3kzzdHb0kjAKf23DpGM+BjD+x XE1pQyI2L04PfljJYh/lhuA7A3aSIzBOg1qk50yu827iPv2s/n1xqnfzzmPF49iSzxpuzACXa92 qtSb44rWDBWA3/AA4K9DPclB7uI44TElJ0dZpyvxoK/kh2vrUgK5rAMhWGXQMGNeJ/J7104CM68 MGBDzgBVGK8FxD6birYl4HXQfVgxOdbW7mpuLnz/hDbx7bPukhvT3FN+S1Nid2UuwicqTmj/fbr OBLgqgBIwyF4dIikayKNA= X-Google-Smtp-Source: AGHT+IGkQWfCACbqbbuCCFNQ2qKU8A9EFKMEKsAn+SUo7Bb1nABCAXucZJ/1Sbvypne34EZsM9fjtA== X-Received: by 2002:a17:902:f552:b0:297:c71d:851c with SMTP id d9443c01a7336-2a2f2736bc0mr15326595ad.36.1766122648538; Thu, 18 Dec 2025 21:37:28 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.22 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:28 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com Subject: [PATCH 3/6] arm64: Provide dcache_inval_poc_nosync helper Date: Fri, 19 Dec 2025 13:36:55 +0800 Message-Id: <20251219053658.84978-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song dcache_inval_poc_nosync does not wait for the data cache invalidation to complete. Later, we defer the synchronization so we can wait for all SG entries together. Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/include/asm/cacheflush.h | 1 + arch/arm64/mm/cache.S | 43 +++++++++++++++++++++-------- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/cacheflush.h b/arch/arm64/include/asm/c= acheflush.h index 9b6d0a62cf3d..382b4ac3734d 100644 --- a/arch/arm64/include/asm/cacheflush.h +++ b/arch/arm64/include/asm/cacheflush.h @@ -74,6 +74,7 @@ extern void icache_inval_pou(unsigned long start, unsigne= d long end); extern void dcache_clean_inval_poc(unsigned long start, unsigned long end); extern void dcache_inval_poc(unsigned long start, unsigned long end); extern void dcache_clean_poc(unsigned long start, unsigned long end); +extern void dcache_inval_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_poc_nosync(unsigned long start, unsigned long end= ); extern void dcache_clean_pop(unsigned long start, unsigned long end); extern void dcache_clean_pou(unsigned long start, unsigned long end); diff --git a/arch/arm64/mm/cache.S b/arch/arm64/mm/cache.S index 4a7c7e03785d..8c1043c9b9e5 100644 --- a/arch/arm64/mm/cache.S +++ b/arch/arm64/mm/cache.S @@ -132,17 +132,7 @@ alternative_else_nop_endif ret SYM_FUNC_END(dcache_clean_pou) =20 -/* - * dcache_inval_poc(start, end) - * - * Ensure that any D-cache lines for the interval [start, end) - * are invalidated. Any partial lines at the ends of the interval are - * also cleaned to PoC to prevent data loss. - * - * - start - kernel start address of region - * - end - kernel end address of region - */ -SYM_FUNC_START(__pi_dcache_inval_poc) +.macro _dcache_inval_poc_impl, do_sync dcache_line_size x2, x3 sub x3, x2, #1 tst x1, x3 // end cache line aligned? @@ -158,11 +148,42 @@ SYM_FUNC_START(__pi_dcache_inval_poc) 3: add x0, x0, x2 cmp x0, x1 b.lo 2b +.if \do_sync dsb sy +.endif ret +.endm + +/* + * dcache_inval_poc(start, end) + * + * Ensure that any D-cache lines for the interval [start, end) + * are invalidated. Any partial lines at the ends of the interval are + * also cleaned to PoC to prevent data loss. + * + * - start - kernel start address of region + * - end - kernel end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_poc) + _dcache_inval_poc_impl 1 SYM_FUNC_END(__pi_dcache_inval_poc) SYM_FUNC_ALIAS(dcache_inval_poc, __pi_dcache_inval_poc) =20 +/* + * dcache_inval_poc_nosync(start, end) + * + * Issue the instructions of D-cache lines for the interval [start, end) + * for invalidation. Not necessarily cleaned to PoC till an explicit dsb + * sy later + * + * - start - kernel start address of region + * - end - kernel end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_poc_nosync) + _dcache_inval_poc_impl 0 +SYM_FUNC_END(__pi_dcache_inval_poc_nosync) +SYM_FUNC_ALIAS(dcache_inval_poc_nosync, __pi_dcache_inval_poc_nosync) + /* * dcache_clean_poc(start, end) * --=20 2.39.3 (Apple Git-146) From nobody Sun Feb 8 09:12:44 2026 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CBA02C1581 for ; Fri, 19 Dec 2025 05:37:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122658; cv=none; b=JT9X7u6MtASfATYe9Wfdk2leMiOb7Ow6Z3LEH6CAlTV5DtCSLtpEy09BpJslB95kKg9aEgbp8+YZqJ/L8V2X1fp4vl1mkAxSF4MKfFNtL6yW1ndTsJuAsd30vwfPWdjLYABEnN+39s6GcMzpQyEB0nXJCrniC3pR9jGvHlcTmm4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122658; c=relaxed/simple; bh=VRHBseV8q+/R/B0v4LxS/if/aoC+7Nb6x15H0rpCpJo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JH2RrEaPyUIqJV3O/WPUvSd11+Ny7uEgbKRHpPLqfq/ZlCJaUpfy5k2OparC0I/d+4nGmT7JauBBbE3pO+ieaKmsGA7f8bFN4vhhRiY9rzHt1Iacjgh6PGJebx4CcvFR14eVLBAZ1oQP5jiFvnhMEgBfslywaVwqoHmAaQrI78Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=aNG46r7O; arc=none smtp.client-ip=209.85.214.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aNG46r7O" Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2a12ebe4b74so21353965ad.0 for ; Thu, 18 Dec 2025 21:37:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122655; x=1766727455; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nVc7S6MGRFGsCbCRv42PvmwxMMf3ng26LOmJbRXLMRA=; b=aNG46r7O6E7A0ocEpk0rYAz/I9q82GOGQw6LAY1nQ6ayc9CcGmz+3aueA+NJDxwXQR 5eQVcVBd6AoIgKkatZfsVfWwz3/BZclFQ3An2WNsGR2ipDtkwKunmbrzBdkrWqQemA+h paz8SKfHEEQ7CoH/e1zjUo9g8ZQe561VucCDAI813c/nfeeIsSQrytO19kk0mxneZDUJ mDOqE2Rv5nesfJrgnZTxkQrOQIHxFNF3aV9bK/ktfBRqhTKIhTVhTbjxwavCKkn3i+ZO U727h+0kKL503cW1kWnaa73C+PvNB27v3YLCIRbMPjgV6O9FWSboRj3x4eQC/edku8E6 a3MA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122655; x=1766727455; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=nVc7S6MGRFGsCbCRv42PvmwxMMf3ng26LOmJbRXLMRA=; b=T9jXz8rFgXrgffyq4+8k3sDQRbmz3VKRYsihuBUQSSIvfLckE5I0bdrkmINWXLhIym ZOA25YwGB9rw5MnaEA6jB2UkHCMui5m45mLGqKvTeNY0d5vJGeMmo6Ko8/vJ+weOmM9x s2AjrYtgnZIhtvL7jHb/73OZT/LhUyV1uOmtECJsx80j4JWOychDHKn7LpZEl4XYIqiS 2XskdIftlmuUHi7/yfy2Nq8TmsD1C2xhati1Hfd+Q9Sj1gPfCve/ZDi6zhP/Twi7uNis un5LWS5eP5ztqotPLTPM7IELQ3Q/699xGPmyECpZW7L9m9mQhYLKtGmqkEu0IAqvsBS8 Sdag== X-Forwarded-Encrypted: i=1; AJvYcCXgiCseI9i4yT5dZArDNiRhpoZWqHXp6/uvet9l7U5BwyH91H6cFbqzXeISN2SLMtQiLu1GB6XZk7Xlbo8=@vger.kernel.org X-Gm-Message-State: AOJu0Yz7fr7G4SFvEJ9jovt5LG5C2HSduS84K2OIDAOdTaQ25AyplOQ8 0nmT8qn1T1gDUzrsaxpe1sNIydDuurvBQ8EbUZisiLu7M+3CMGgBvnry X-Gm-Gg: AY/fxX7l2weWe4Nsw+MZhy0yrt+Rsj5M+AhDrME8+/KLT2745cl2YJQPDd7fErunmX5 O1rkP6fD2Yr57SFQVs1Owc2jkccxjP17AS4BT52qB+DfkgisxK08TkL9z9KMllm/lOJ4MTePOUT 6bS6HaYUx17qcEBuqyW+B0z8w24HXbPeX+55iKjaN33YI225/Fqlct7zLaIlaVaqYg47bpv7NyB xN6dq7soU9ZQmTb/hNeAPreaoV1mA65OAr37lJWD2yoUEDEgNxvURwlCcd1vf5S64WuAlY8weuS oV/6FCtvwRGGTyMFYOJeqlShEHtk8zaogS99R/lem7VU9DXbyrjfXLZbwrY5/2l966jarb110i3 41Hb0ACxLGbWOkWRkYM+pgI7a3u9ZtTUAxnNS8VIwnapcjmWJGjnCIp7oKU+q5OAYIw+e9j46kV WRJkxzHbyI735sEuu5j9E= X-Google-Smtp-Source: AGHT+IEgIXUAtoilrAdOFUVheL04wY8Hn1GTTg9fp3Bh9Uns3qP+YxFWsnf+ZhkJ6ZTgBwxYZdARhA== X-Received: by 2002:a17:903:37d0:b0:295:8da5:c634 with SMTP id d9443c01a7336-2a2f2206b52mr14068935ad.9.1766122654548; Thu, 18 Dec 2025 21:37:34 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.28 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:34 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com Subject: [PATCH 4/6] arm64: Provide arch_sync_dma_ batched helpers Date: Fri, 19 Dec 2025 13:36:56 +0800 Message-Id: <20251219053658.84978-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song arch_sync_dma_for_device_batch_add() and arch_sync_dma_for_cpu_batch_add() batch DMA sync operations, while arch_sync_dma_batch_flush() waits for their completion as a group. On architectures that do not support batching, arch_sync_dma_for_device_batch_add() and arch_sync_dma_for_cpu_batch_add() fall back to the non-batched implementations, and arch_sync_dma_batch_flush() is a no-op. Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song --- arch/arm64/Kconfig | 1 + arch/arm64/mm/dma-mapping.c | 24 ++++++++++++++++++++++++ include/linux/dma-map-ops.h | 22 ++++++++++++++++++++++ kernel/dma/Kconfig | 3 +++ 4 files changed, 50 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 93173f0a09c7..c8adbf21b7bf 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -112,6 +112,7 @@ config ARM64 select ARCH_SUPPORTS_SCHED_CLUSTER select ARCH_SUPPORTS_SCHED_MC select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH + select ARCH_WANT_BATCHED_DMA_SYNC select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c index b2b5792b2caa..9ac1ddd1bb9c 100644 --- a/arch/arm64/mm/dma-mapping.c +++ b/arch/arm64/mm/dma-mapping.c @@ -31,6 +31,30 @@ void arch_sync_dma_for_cpu(phys_addr_t paddr, size_t siz= e, dcache_inval_poc(start, start + size); } =20 +void arch_sync_dma_for_device_batch_add(phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ + unsigned long start =3D (unsigned long)phys_to_virt(paddr); + + dcache_clean_poc_nosync(start, start + size); +} + +void arch_sync_dma_for_cpu_batch_add(phys_addr_t paddr, size_t size, + enum dma_data_direction dir) +{ + unsigned long start =3D (unsigned long)phys_to_virt(paddr); + + if (dir =3D=3D DMA_TO_DEVICE) + return; + + dcache_inval_poc_nosync(start, start + size); +} + +void arch_sync_dma_batch_flush(void) +{ + dsb(sy); +} + void arch_dma_prep_coherent(struct page *page, size_t size) { unsigned long start =3D (unsigned long)page_address(page); diff --git a/include/linux/dma-map-ops.h b/include/linux/dma-map-ops.h index 4809204c674c..5ee92c410e3c 100644 --- a/include/linux/dma-map-ops.h +++ b/include/linux/dma-map-ops.h @@ -361,6 +361,28 @@ static inline void arch_sync_dma_for_cpu(phys_addr_t p= addr, size_t size, } #endif /* ARCH_HAS_SYNC_DMA_FOR_CPU */ =20 +#ifdef CONFIG_ARCH_WANT_BATCHED_DMA_SYNC +void arch_sync_dma_for_device_batch_add(phys_addr_t paddr, size_t size, + enum dma_data_direction dir); +void arch_sync_dma_for_cpu_batch_add(phys_addr_t paddr, size_t size, + enum dma_data_direction dir); +void arch_sync_dma_batch_flush(void); +#else +static inline void arch_sync_dma_for_device_batch_add(phys_addr_t paddr, s= ize_t size, + enum dma_data_direction dir) +{ + arch_sync_dma_for_device(paddr, size, dir); +} +static inline void arch_sync_dma_for_cpu_batch_add(phys_addr_t paddr, size= _t size, + enum dma_data_direction dir) +{ + arch_sync_dma_for_cpu(paddr, size, dir); +} +static inline void arch_sync_dma_batch_flush(void) +{ +} +#endif + #ifdef CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL void arch_sync_dma_for_cpu_all(void); #else diff --git a/kernel/dma/Kconfig b/kernel/dma/Kconfig index 31cfdb6b4bc3..2785099b2fa0 100644 --- a/kernel/dma/Kconfig +++ b/kernel/dma/Kconfig @@ -78,6 +78,9 @@ config ARCH_HAS_DMA_PREP_COHERENT config ARCH_HAS_FORCE_DMA_UNENCRYPTED bool =20 +config ARCH_WANT_BATCHED_DMA_SYNC + bool + # # Select this option if the architecture assumes DMA devices are coherent # by default. --=20 2.39.3 (Apple Git-146) From nobody Sun Feb 8 09:12:44 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 92A012D0C82 for ; Fri, 19 Dec 2025 05:37:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122663; cv=none; b=ivJWIVqFPxxqgjc1sLf8qKmwmkndrOLzoe0rzjgQt4Y2aQA0dUkfo+e5nG/Cr/mZWXSmzVQIxn1u1XyNSq0ujZ9SnWKrbPenainqm62YHd2hw/g563pxw/AfNIDG3njGzkvnONPNOagvtxvEF1bpCTNhi3YTk3Llz13CGW1LkrM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122663; c=relaxed/simple; bh=6qIKRywSnWWq7hy1HRk8ynBQ/UoE1yw8azJnlduW0L0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pKmet+Ba+ADrFaMToUOV4bOgfZWKDCWTJ+lEsyLlqhoJzWubKiysnIe7dYExHYB2ieCBYSCyNLXJ5PmmL3B0RpkjA+YiuK4V8KNFDhaE+jhSX2ChWKXV8nPPbsERfkGOOUyq2E0ulJ1hvSo79Wv4XwCInInl7qTghZ33yUcQcFQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZjDVbAyl; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZjDVbAyl" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2a0834769f0so12925335ad.2 for ; Thu, 18 Dec 2025 21:37:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122661; x=1766727461; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eenPutgBr0kvR5xknWuBlG56Mzp4JXW/11o/+DFN/Mk=; b=ZjDVbAylGFTevWR5ktOcGZLuQgJOROOpDsk36MO4dZi5sZcQG9l83JyCy+DKRnHN6m u9ugGagIvEknMvZoar3IVC8hl1W9YD/tBge69AxqqYNiQKr5p3WmOzi9+cWcd0kC/o3d bzAgLsM5PIY8nu01vuPZGFzBgWHpHVKewPjEuGlWaBHgCgv2mh3zltR5dk8gWqI2V5tG DZ1v/CDUi17UT/alN7r9YUOqEXIcBp1Q7TDQmmE57ZSApwM/wy3a6PJWpiO+xcwDVUbB yiwmq6YjgGQoeRRFm9tIl8CeCMB4I/f1ZCiIAXqmbSmI3oV20wj1pKyXI/2Om5Rs/smo 0LSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122661; x=1766727461; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=eenPutgBr0kvR5xknWuBlG56Mzp4JXW/11o/+DFN/Mk=; b=r3g0cJyC6OLXvKuHUUYzj15G/JNWvtgyZo+qB8A8Za5WN3cfRco6OyfxQsfWkEwOUn 6aVXuIBVpwiKw3CYspvG7zhAsA0+6ymGgwwEPjeVxki9lslYwt8e1sjesWHWWAbdVD6+ MUaueV/MtluSvWBOnVkZW8584FLmIayHo907qo2wVrLoK6xH0ZBFWUdUcg+ZHsJPwZDs /Pj6L1ec0aG7FAlsWPGrWgflWle+Z2HjSgexkr8plt/fs0SZj+07kmKOA5ObpfgIkxfA 3jaq5FAOMlJFoRISQ6rTiryCNzbVWegBA3mcNwG7tcSnlGjg6l+h1H2sUdqBkiscZJFo vjwA== X-Forwarded-Encrypted: i=1; AJvYcCVwuQHxDhW+/qsobr+i1QclU6edy7+9W6Ap9hV6W2hopHWno8j6CQNoA/EzYoDlZWw1FLWgHFniprTOngI=@vger.kernel.org X-Gm-Message-State: AOJu0YxmHF2gUc82a+AXJ5+tucz8qaROa8v6tlGLW4pL5JrNMhLVJjLF JfwENqU8uUtkG1dGebQKPrJ2HUhsd5ND5K1VgvnLNGi/wvP7Hv/7awIf X-Gm-Gg: AY/fxX7TTb3J2oZpySRRnfxJ3LfxG8dPJJUt1byJpyrsZkGpLb6UnzGi9Wk6OPQAa2g gRQDQyl2onBryFMHoVGWYx4SDYe6xwE7pPEfPT9VbN8679bLOSXdw4tPE/kX2C7HBd3qzaWV2li YsdaQfMAa+VYXIdJiC+hQaEs32WS7/s1vXHD++Iwv95InVBp25Kl1lI7HXrdzMsXFWT4SRNiTn6 Dd2U6QaQp2mXjaVNeQ3H0YyXSAtrsvQQr9txZLK8Ew5sPQd0m4cUFu6QhBrHr3v5EqlkCNYHiff SlLmJ+PoIAXlFS0wof//EyQ37/fe0HZKJddeYiHEHP0q1vppK52s8Pu1ck6+5R6nHx+BrNB8p02 x0mQ/H0VGW58U5zKjPCcVA3Gr8q3oppTvEJFy13qBQQc/B6lpwSSAfwUbD+A2mM3ZBDBowLizg+ rWCO3/nbb59vBmobUxjkE= X-Google-Smtp-Source: AGHT+IHOj/tomowDEiB20GWpwm1Eafm2xs2aEyUzFu0xX3zTdDmt2EBoWVTLaYigoPnj18tyH/xwXQ== X-Received: by 2002:a17:902:d548:b0:2a1:325d:821a with SMTP id d9443c01a7336-2a2f2a4f0c5mr13846805ad.60.1766122660716; Thu, 18 Dec 2025 21:37:40 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:40 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com Subject: [PATCH 5/6] dma-mapping: Allow batched DMA sync operations if supported by the arch Date: Fri, 19 Dec 2025 13:36:57 +0800 Message-Id: <20251219053658.84978-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song This enables dma_direct_sync_sg_for_device, dma_direct_sync_sg_for_cpu, dma_direct_map_sg, and dma_direct_unmap_sg to use batched DMA sync operations when possible. This significantly improves performance on devices without hardware cache coherence. Tangquan's initial results show that batched synchronization can reduce dma_map_sg() time by 64.61% and dma_unmap_sg() time by 66.60% on an MTK phone platform (MediaTek Dimensity 9500). The tests were performed by pinning the task to CPU7 and fixing the CPU frequency at 2.6 GHz, running dma_map_sg() and dma_unmap_sg() on 10 MB buffers (10 MB / 4 KB sg entries per buffer) for 200 iterations and then averaging the results. Cc: Catalin Marinas Cc: Will Deacon Cc: Marek Szyprowski Cc: Robin Murphy Cc: Ada Couprie Diaz Cc: Ard Biesheuvel Cc: Marc Zyngier Cc: Anshuman Khandual Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Tangquan Zheng Signed-off-by: Barry Song Reported-by: kernel test robot --- kernel/dma/direct.c | 28 ++++++++++----- kernel/dma/direct.h | 86 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 95 insertions(+), 19 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 50c3fe2a1d55..ed2339b0c5e7 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -403,9 +403,10 @@ void dma_direct_sync_sg_for_device(struct device *dev, swiotlb_sync_single_for_device(dev, paddr, sg->length, dir); =20 if (!dev_is_dma_coherent(dev)) - arch_sync_dma_for_device(paddr, sg->length, - dir); + arch_sync_dma_for_device_batch_add(paddr, sg->length, dir); } + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_batch_flush(); } #endif =20 @@ -422,7 +423,7 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, phys_addr_t paddr =3D dma_to_phys(dev, sg_dma_address(sg)); =20 if (!dev_is_dma_coherent(dev)) - arch_sync_dma_for_cpu(paddr, sg->length, dir); + arch_sync_dma_for_cpu_batch_add(paddr, sg->length, dir); =20 swiotlb_sync_single_for_cpu(dev, paddr, sg->length, dir); =20 @@ -430,8 +431,10 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_dma_mark_clean(paddr, sg->length); } =20 - if (!dev_is_dma_coherent(dev)) + if (!dev_is_dma_coherent(dev)) { arch_sync_dma_for_cpu_all(); + arch_sync_dma_batch_flush(); + } } =20 /* @@ -443,14 +446,19 @@ void dma_direct_unmap_sg(struct device *dev, struct s= catterlist *sgl, { struct scatterlist *sg; int i; + bool need_sync =3D false; =20 for_each_sg(sgl, sg, nents, i) { - if (sg_dma_is_bus_address(sg)) + if (sg_dma_is_bus_address(sg)) { sg_dma_unmark_bus_address(sg); - else - dma_direct_unmap_phys(dev, sg->dma_address, + } else { + need_sync =3D true; + dma_direct_unmap_phys_batch_add(dev, sg->dma_address, sg_dma_len(sg), dir, attrs); + } } + if (need_sync && !dev_is_dma_coherent(dev)) + arch_sync_dma_batch_flush(); } #endif =20 @@ -460,6 +468,7 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, struct pci_p2pdma_map_state p2pdma_state =3D {}; struct scatterlist *sg; int i, ret; + bool need_sync =3D false; =20 for_each_sg(sgl, sg, nents, i) { switch (pci_p2pdma_state(&p2pdma_state, dev, sg_page(sg))) { @@ -471,7 +480,8 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, */ break; case PCI_P2PDMA_MAP_NONE: - sg->dma_address =3D dma_direct_map_phys(dev, sg_phys(sg), + need_sync =3D true; + sg->dma_address =3D dma_direct_map_phys_batch_add(dev, sg_phys(sg), sg->length, dir, attrs); if (sg->dma_address =3D=3D DMA_MAPPING_ERROR) { ret =3D -EIO; @@ -491,6 +501,8 @@ int dma_direct_map_sg(struct device *dev, struct scatte= rlist *sgl, int nents, sg_dma_len(sg) =3D sg->length; } =20 + if (need_sync && !dev_is_dma_coherent(dev)) + arch_sync_dma_batch_flush(); return nents; =20 out_unmap: diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index da2fadf45bcd..a211bab26478 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -64,15 +64,11 @@ static inline void dma_direct_sync_single_for_device(st= ruct device *dev, arch_sync_dma_for_device(paddr, size, dir); } =20 -static inline void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) +static inline void __dma_direct_sync_single_for_cpu(struct device *dev, + phys_addr_t paddr, size_t size, enum dma_data_direction dir) { - phys_addr_t paddr =3D dma_to_phys(dev, addr); - - if (!dev_is_dma_coherent(dev)) { - arch_sync_dma_for_cpu(paddr, size, dir); + if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_cpu_all(); - } =20 swiotlb_sync_single_for_cpu(dev, paddr, size, dir); =20 @@ -80,7 +76,31 @@ static inline void dma_direct_sync_single_for_cpu(struct= device *dev, arch_dma_mark_clean(paddr, size); } =20 -static inline dma_addr_t dma_direct_map_phys(struct device *dev, +#ifdef CONFIG_ARCH_WANT_BATCHED_DMA_SYNC +static inline void dma_direct_sync_single_for_cpu_batch_add(struct device = *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr =3D dma_to_phys(dev, addr); + + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_for_cpu_batch_add(paddr, size, dir); + + __dma_direct_sync_single_for_cpu(dev, paddr, size, dir); +} +#endif + +static inline void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr =3D dma_to_phys(dev, addr); + + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_for_cpu(paddr, size, dir); + + __dma_direct_sync_single_for_cpu(dev, paddr, size, dir); +} + +static inline dma_addr_t __dma_direct_map_phys(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { @@ -108,9 +128,6 @@ static inline dma_addr_t dma_direct_map_phys(struct dev= ice *dev, } } =20 - if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) - arch_sync_dma_for_device(phys, size, dir); return dma_addr; =20 err_overflow: @@ -121,6 +138,53 @@ static inline dma_addr_t dma_direct_map_phys(struct de= vice *dev, return DMA_MAPPING_ERROR; } =20 +#ifdef CONFIG_ARCH_WANT_BATCHED_DMA_SYNC +static inline dma_addr_t dma_direct_map_phys_batch_add(struct device *dev, + phys_addr_t phys, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + dma_addr_t dma_addr =3D __dma_direct_map_phys(dev, phys, size, dir, attrs= ); + + if (dma_addr !=3D DMA_MAPPING_ERROR && !dev_is_dma_coherent(dev) && + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + arch_sync_dma_for_device_batch_add(phys, size, dir); + + return dma_addr; +} +#endif + +static inline dma_addr_t dma_direct_map_phys(struct device *dev, + phys_addr_t phys, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + dma_addr_t dma_addr =3D __dma_direct_map_phys(dev, phys, size, dir, attrs= ); + + if (dma_addr !=3D DMA_MAPPING_ERROR && !dev_is_dma_coherent(dev) && + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + arch_sync_dma_for_device(phys, size, dir); + + return dma_addr; +} + +#ifdef CONFIG_ARCH_WANT_BATCHED_DMA_SYNC +static inline void dma_direct_unmap_phys_batch_add(struct device *dev, dma= _addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + phys_addr_t phys; + + if (attrs & DMA_ATTR_MMIO) + /* nothing to do: uncached and no swiotlb */ + return; + + phys =3D dma_to_phys(dev, addr); + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + dma_direct_sync_single_for_cpu_batch_add(dev, addr, size, dir); + + swiotlb_tbl_unmap_single(dev, phys, size, dir, + attrs | DMA_ATTR_SKIP_CPU_SYNC); +} +#endif + static inline void dma_direct_unmap_phys(struct device *dev, dma_addr_t ad= dr, size_t size, enum dma_data_direction dir, unsigned long attrs) { --=20 2.39.3 (Apple Git-146) From nobody Sun Feb 8 09:12:44 2026 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57C022C0287 for ; Fri, 19 Dec 2025 05:37:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122669; cv=none; b=hzgLZdvG3p0gpQedNtpc4RS6IEEzFOY00+Gtx7EtIYpLz0BN4Nn17lHg0EWksQprCLdVR+MqMmQYDed4YR9o9RpNrcmjfrRJiDs5DSHJVow72nviX2x5Db8K9Dsaammi9YjGoVCGNkkpMqpZJ1X8ILNiyn+MFSsNML2XMIygAOw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1766122669; c=relaxed/simple; bh=+jyi5gd3AIMhjz4J7lVMgs4zhdEBtbx6wjVxjxON7rU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TKMibFWy8kdkgIpi/eLucH8z8qP+b1ADvtGByRjH0nbajMv3G9QEA/N5l3Wv5HH1YMtqOhQgMe4XT5aGlGp2CW/E0xeFY8Ru8a8FTYnq2jwNZnPiyQq5/sSAzu6azsL0ht2NDGHdMAb8NcxFceCOfCuanKEIx7HYbt8Up8IZv8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=G3e+qvQj; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G3e+qvQj" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2a0bb2f093aso14246465ad.3 for ; Thu, 18 Dec 2025 21:37:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1766122667; x=1766727467; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cJO21xU2H+bRt1oLrdkCFshk8QlbaOYxvZtIbsr9n2g=; b=G3e+qvQjJAxEFjIJjHosQA29F7sciKx6YSbgiWi7f+buj3cLLzldEUg9XepA7GlWcE D4cc9efxDFZQPRQRUaAk+s6Sox9kUWAL2pt++3b75Ju5KVaTfUHFj8wjRu1SNPK3ePxN w1Lyq8WwSGzS49w4PqzVKfAc/zqb6CSo2cJAdI81g1JJY2QbQ9UxACPiWkthD6B2zSR9 Yhd1AZUXTT16SDsV5hMOxJm7kAnmYvvVxbwYAK9aMe72f+1c2YtlEXjenYfoNI77LmDB Sxzdn7xb4uSHUJYXFymiT2J1Sx4UScbjfa1kyfkEkAfA2tUJoiXmBxxow0HsFXlQIWJJ D0lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1766122667; x=1766727467; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=cJO21xU2H+bRt1oLrdkCFshk8QlbaOYxvZtIbsr9n2g=; b=TOGBgA2bpnqJMwlNK2toQzRMmQFvH0f1+1bTwmlcqabioY2vymNqLEGJovp8P1mjQX 6PGKMoztrCBJ3YAm51dIGQuRG6WT4dbgK0w558WoirhgTx+rjaHtetnSC+R1HbrhBm5+ W8856vlPHem8K+54iO9KxBkwKRUfRMj2GrGGeCbagGvoycqFHvpD9BnyF8I3kj0T0YH7 T5H98hTCfJ3FFSbCb1/RKY3WxrLyMI1ndKJBxYs7AaG1MZYcRDRmD8RP8p/hY0GvfmvQ 8ZuWGXlX6NkKOOT7oOwmrTXBBUApV/pMb1Vl79i7dyZsS+qKyUV6cPLwNw60JUweEnLa EiSg== X-Forwarded-Encrypted: i=1; AJvYcCWJjYzxaw8biF79kQ/57nNsxTmWHhNgh2d/1a2fQ2+MarRBKaEUOkUSaXL/ef0gvrSyGhc/4sgAL5FOv9E=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+j63KTCW4q/NGj1KHtAWA9SdUqzre6meB4nWZzp1gddjf6PNO Kiz3eg4GoP3UVKY4fS8B4RM3MCTyeP3nJyta7f0s9ggOfmaKOJ+wo2fk X-Gm-Gg: AY/fxX64u5FuVeZ23FLiw9jZtQOquO+IFAk5XzP5Twn41kSffDhp8bB9TMc4ReP6u6w jNRPZYqVuaxE5Effo70N+VFWbxP1hIbUrBmdKdCkx5hYbroUqPBwIuLXxQIoMVkbsS3ZYUzWRF0 NiYHJ3zTQmn/5OxFAWMAHUEnqRmneKTHy4O6oL4AxTcUiIT6DGi6nXELWVC03vNwHrHnjp4CAxC aWoHeUHPVrOr3tp3PJQIELx9eMdVETdaZwWpQyijCXQ+G/tMjXd29AH17Q+r6rh7qh77sAn4Fsc i3hz5mBm5uzKzWbGMgSMtrQE4l6SJNZPx7zmGE6492MTWgzfRO86D4DeQrZklT9QdHr+jLtNCkG 2/pvkKkRhnO8x00o1MFEHc3f9XfSYDYKvBGSA4xTQMzxpS7RgqKrCObiHrq7ovmMjoxFV1xSyo0 SxN2MpnWeuCEMlNvdKWRU= X-Google-Smtp-Source: AGHT+IGDMYqEG2oDqA96bVWjmxK1/Wde940fAntilRs7/x6PwBG31jTbdPi/FUUfggX13zuKgo2NiQ== X-Received: by 2002:a17:903:230c:b0:29d:a26c:34b6 with SMTP id d9443c01a7336-2a2f293d4f7mr14844825ad.50.1766122667279; Thu, 18 Dec 2025 21:37:47 -0800 (PST) Received: from Barrys-MBP.hub ([47.72.129.29]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-2a2f3d4d895sm9930215ad.54.2025.12.18.21.37.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Dec 2025 21:37:46 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: catalin.marinas@arm.com, m.szyprowski@samsung.com, robin.murphy@arm.com, will@kernel.org Cc: ada.coupriediaz@arm.com, anshuman.khandual@arm.com, ardb@kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, maz@kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, zhengtangquan@oppo.com, Joerg Roedel Subject: [PATCH RFC 6/6] dma-iommu: Allow DMA sync batching for IOVA link/unlink Date: Fri, 19 Dec 2025 13:36:58 +0800 Message-Id: <20251219053658.84978-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20251219053658.84978-1-21cnbao@gmail.com> References: <20251219053658.84978-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Apply batched DMA synchronization to __dma_iova_link() and iommu_dma_iova_unlink_range_slow(). For multiple sync_dma_for_device() and sync_dma_for_cpu() calls, we only need to wait once for the completion of all sync operations, rather than waiting for each one individually. I do not have the hardware to test this, so it is marked as RFC. I would greatly appreciate it if someone could test it. Suggested-by: Marek Szyprowski Cc: Robin Murphy Cc: Joerg Roedel Cc: Will Deacon Signed-off-by: Barry Song --- drivers/iommu/dma-iommu.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index c92088855450..95432bdc364f 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -1837,7 +1837,7 @@ static int __dma_iova_link(struct device *dev, dma_ad= dr_t addr, int prot =3D dma_info_to_prot(dir, coherent, attrs); =20 if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) - arch_sync_dma_for_device(phys, size, dir); + arch_sync_dma_for_device_batch_add(phys, size, dir); =20 return iommu_map_nosync(iommu_get_dma_domain(dev), addr, phys, size, prot, GFP_ATOMIC); @@ -1980,6 +1980,8 @@ int dma_iova_sync(struct device *dev, struct dma_iova= _state *state, dma_addr_t addr =3D state->addr + offset; size_t iova_start_pad =3D iova_offset(iovad, addr); =20 + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_batch_flush(); return iommu_sync_map(domain, addr - iova_start_pad, iova_align(iovad, size + iova_start_pad)); } @@ -1993,6 +1995,8 @@ static void iommu_dma_iova_unlink_range_slow(struct d= evice *dev, struct iommu_dma_cookie *cookie =3D domain->iova_cookie; struct iova_domain *iovad =3D &cookie->iovad; size_t iova_start_pad =3D iova_offset(iovad, addr); + bool need_sync_dma =3D !dev_is_dma_coherent(dev) && + !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO)); dma_addr_t end =3D addr + size; =20 do { @@ -2007,8 +2011,7 @@ static void iommu_dma_iova_unlink_range_slow(struct d= evice *dev, len =3D min_t(size_t, end - addr, iovad->granule - iova_start_pad); =20 - if (!dev_is_dma_coherent(dev) && - !(attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO))) + if (need_sync_dma) arch_sync_dma_for_cpu(phys, len, dir); =20 swiotlb_tbl_unmap_single(dev, phys, len, dir, attrs); @@ -2016,6 +2019,9 @@ static void iommu_dma_iova_unlink_range_slow(struct d= evice *dev, addr +=3D len; iova_start_pad =3D 0; } while (addr < end); + + if (need_sync_dma) + arch_sync_dma_batch_flush(); } =20 static void __iommu_dma_iova_unlink(struct device *dev, --=20 2.39.3 (Apple Git-146)