This series prevents a DMA_FROM_DEVICE map from discarding CPU-written
buffer contents on non-coherent 32-bit ARM. It is based on
v7.3-rc1-324-g986c24e0fe44.

ARM currently invalidates these buffers before the device writes them.
If the device writes only part of a buffer, discarded dirty cache lines
can expose older memory contents in the untouched bytes. A stock USB
webcam demonstrated this through usbfs. Short isochronous packets left
gaps, and usbfs returned non-zero data to userspace from bytes it had
cleared.

arm64 changed this handoff from invalidate to clean in 2022 with commit
c50f11c6196f ("arm64: mm: Don't invalidate FROM_DEVICE buffers at start
of DMA transfer"). Arnd Bergmann's 2023 ARM32 cache-maintenance series
left DMA_FROM_DEVICE preservation unresolved [1]. This series keeps the
existing ownership hooks.

Patch 1 covers v6, v7, v7-M and the common outer-cache path. These paths
already invalidate when ownership returns to the CPU. Patch 2 makes the
legacy write-back backends use their existing clean-and-invalidate
operation at device handoff. Their no-op ownership-return hooks remain
unchanged, avoiding the additional completion traversal raised during
review [2].

[1] https://lore.kernel.org/r/20230327121317.4081816-1-arnd@kernel.org
[2] https://lists.infradead.org/pipermail/linux-riscv/2023-March/029740.html

Testing:

  - An ARM11 MPCore (ARMv6) in a New Nintendo 2DS XL lost 402,080 of
    409,600 CPU-written bytes in a mapping-only test. With patch 1, none
    were lost. The UP kernel used out-of-tree platform support and the
    same map-time invalidate as mainline.

  - On an ARM926EJ-S (ARMv5TEJ) SAM9X75, eleven mapping-only runs lost
    339,936 to 351,744 of 409,600 bytes with the existing code. Five
    usbfs runs checked 245,708 unwritten gap bytes and exposed 29,403 of
    them as non-zero. With this series, five fresh 100-iteration
    mapping-only runs on the same board each lost 0 of 409,600 bytes.

  - A Cortex-A72 running AArch32/LPAE did not reproduce the loss in 22
    existing-kernel runs. This does not establish whether other v7 cores
    are affected.

The one-pass series passed focused W=1 object builds for ARM926,
Feroceon, XScale, ARM940 and v7. The remaining legacy backends have
source review and compile coverage from the previous two-pass revision.

Neither patch adds a cache traversal. Patch 2 changes the legacy
DMA_FROM_DEVICE operation from invalidate to clean-and-invalidate. This
can add per-line work and memory writes on dirty buffers. Throughput and
CPU cost have not been measured.

Karl Mehltretter (2):
  ARM: dma-mapping: preserve DMA_FROM_DEVICE buffer contents
  ARM: dma-mapping: flush FROM_DEVICE buffers on legacy backends

 arch/arm/mm/cache-fa.S          | 1 -
 arch/arm/mm/cache-v4wb.S        | 1 -
 arch/arm/mm/cache-v6.S          | 2 --
 arch/arm/mm/cache-v7.S          | 2 --
 arch/arm/mm/cache-v7m.S         | 2 --
 arch/arm/mm/dma-mapping-nommu.c | 5 +----
 arch/arm/mm/dma-mapping.c       | 8 +-------
 arch/arm/mm/proc-arm1020.S      | 1 -
 arch/arm/mm/proc-arm1020e.S     | 1 -
 arch/arm/mm/proc-arm1022.S      | 1 -
 arch/arm/mm/proc-arm1026.S      | 1 -
 arch/arm/mm/proc-arm920.S       | 1 -
 arch/arm/mm/proc-arm922.S       | 1 -
 arch/arm/mm/proc-arm925.S       | 1 -
 arch/arm/mm/proc-arm926.S       | 1 -
 arch/arm/mm/proc-arm940.S       | 1 -
 arch/arm/mm/proc-arm946.S       | 1 -
 arch/arm/mm/proc-feroceon.S     | 2 --
 arch/arm/mm/proc-mohawk.S       | 1 -
 arch/arm/mm/proc-xsc3.S         | 1 -
 arch/arm/mm/proc-xscale.S       | 1 -
 21 files changed, 2 insertions(+), 34 deletions(-)

-- 
2.53.0

On Thu, Sep 10, 2026, at 08:36, Karl Mehltretter wrote:
> This series prevents a DMA_FROM_DEVICE map from discarding CPU-written
> buffer contents on non-coherent 32-bit ARM. It is based on
> v7.3-rc1-324-g986c24e0fe44.
>
> ARM currently invalidates these buffers before the device writes them.
> If the device writes only part of a buffer, discarded dirty cache lines
> can expose older memory contents in the untouched bytes. A stock USB
> webcam demonstrated this through usbfs. Short isochronous packets left
> gaps, and usbfs returned non-zero data to userspace from bytes it had
> cleared.
>
> arm64 changed this handoff from invalidate to clean in 2022 with commit
> c50f11c6196f ("arm64: mm: Don't invalidate FROM_DEVICE buffers at start
> of DMA transfer"). Arnd Bergmann's 2023 ARM32 cache-maintenance series
> left DMA_FROM_DEVICE preservation unresolved [1]. This series keeps the
> existing ownership hooks.

Hi Karl,

I think the main problem here is that we remain inconsistent about the
rules across CPU architectures, and changing Arm on its own does not
mean we have a solution if another architecture decides to change it
in the opposite direction at some point.

I see this as a tradeoff that can go either way:

- the current 32-bit Arm approach (also arc, hexagon, microblaze, mips,
  nios2, openrisc, powerpc32, sh) is obviously faster as it avoids
  the writeback, but it relies on device drivers to ensure no stale
  data can leak back into userspace.

- Will's patch changed arm64 (later copied into riscv) to avoid that
  risk by adding the overhead out of caution, and avoid having to
  audit and fix all drivers.

Clearly the current state is suboptimal, as most drivers are shared
across architectures and should expect a clear interface. Portable
drivers now get extra overhead on arm64/riscv for doing both the
zero-pad and writeback.

If we decide to align with arm64/riscv and take your series, I think
we need two more parts:

- actually measure the performance overhead: you already did the
  work to test this on three separate arm implementations but did
  not share performance numbers.
  Can you quantify how much this costs us on the hardware you used?

- change the remaining architectures the same way: right now, both
  variants are common enough across supported embedded systems
  on all architectures, but changing over arm32 means that all
  only a vanishingly small set of users gets the invalidate-only
  version and we're much more likely to miss future driver bugs
  when driver writes assume the arm/riscv behavior is universal.

       Arnd

On Thu, Sep 10, 2026 at 11:14:28AM +0100, Arnd Bergmann wrote:

Hi Arnd,

> - actually measure the performance overhead: you already did the
>   work to test this on three separate arm implementations but did
>   not share performance numbers.
>   Can you quantify how much this costs us on the hardware you used?
> 

I measured DMA_FROM_DEVICE map/unmap round trips without a device transfer.
The 1 MiB results were unchanged within noise on Pi 400 and SAM9X75, and
faster on ARM11. The only reproducible increase was about 22 ns for a
256 B round trip on Pi 400.

These are the mean times per round trip. I wrote to the dirty buffers
before each iteration and left the clean buffers untouched.

                                         control     patched       change
  Cortex-A72, v7, 256 B dirty           353 ns         376 ns       +6.3%
  Cortex-A72, v7, 1 MiB dirty        73.840 us      73.994 us       +0.21%
  Cortex-A72, v7, 1 MiB clean        73.108 us      73.115 us       +0.01%

  ARM926EJ-S, legacy, 256 B dirty      1.208 us       1.203 us       -0.4%
  ARM926EJ-S, legacy, 1 MiB dirty    656.888 us     656.827 us       -0.009%
  ARM926EJ-S, legacy, 1 MiB clean    656.608 us     656.601 us       -0.001%

  ARM11 MPCore, v6, 256 B dirty        4.639 us       3.095 us      -33.3%
  ARM11 MPCore, v6, 1 MiB dirty       16.380 ms      15.550 ms       -5.1%
  ARM11 MPCore, v6, 1 MiB clean       16.380 ms      15.560 ms       -5.0%

Pi 400 and SAM9X75 used 786262be6048 and GCC 13.3, with three boots per
kernel. On Pi 400, only the v7 map change from patch 1 takes effect.
On SAM9X75, patch 2 changes map-time invalidate to clean-and-invalidate.
Unmap remains a no-op.

ARM11 used a 5.11-based tree and Clang/LLD 22.1.8 on both sides. Its clock
has 4 ms resolution, so I timed batches and subtracted a separate
memset-only batch for dirty buffers. I alternated kernels for five boots
each, with five batches per scenario per boot and 500,000 iterations per
256 B batch. Every patched boot's mean was below every control boot's
mean in all three scenarios.

Both ARM11 kernels invalidate at unmap, as current mainline does.
They differ only in whether they invalidate or clean at map time.

Below is the benchmark source used on Pi 400 and SAM9X75.

----- dma_from_device_benchmark.c -----
// SPDX-License-Identifier: GPL-2.0-only
/*
 * Streaming DMA_FROM_DEVICE map/unmap cost benchmark.
 *
 * Times dma_map_single()+dma_unmap_single() round trips for a few
 * buffer size / dirty-state scenarios, to quantify the cost of the
 * architecture's cache-maintenance choice for DMA_FROM_DEVICE (e.g.
 * invalidate-at-map vs clean-at-map). No real DMA hardware is used or
 * required: only the architecture's cache-maintenance side effects of
 * the streaming DMA API are measured, against a throwaway platform
 * device, so this runs unmodified on any architecture.
 *
 * insmod dma_from_device_benchmark.ko and read dmesg for one line per
 * scenario: mean/min/max nanoseconds per round trip and ns per KiB.
 */
#include <linux/dma-mapping.h>
#include <linux/interrupt.h>
#include <linux/ktime.h>
#include <linux/math64.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/slab.h>

struct scenario {
	const char *name;
	size_t size;
	unsigned int reps;
	bool dirty;
};

static const struct scenario scenarios[] = {
	{ "small-dirty-256B", 256,      5000, true  },
	{ "large-dirty-1MiB", 1 << 20,  300,  true  },
	{ "large-clean-1MiB", 1 << 20,  300,  false },
};

static struct platform_device *pdev;

static int run_scenario(struct device *dev, const struct scenario *sc)
{
	u64 sum = 0, min = U64_MAX, max = 0;
	unsigned int i;
	u8 *buf;
	int ret = 0;

	buf = kmalloc(sc->size, GFP_KERNEL);
	if (!buf)
		return -ENOMEM;

	for (i = 0; i < sc->reps; i++) {
		unsigned long flags;
		dma_addr_t dma;
		u64 t0, t1, d;

		/* Re-dirty every line each pass; a "clean" scenario never
		 * writes buf at all, so it stays whatever the allocator left
		 * it as (steady-state after the first map/unmap invalidates
		 * it out of cache).
		 */
		if (sc->dirty)
			memset(buf, (u8)(i | 1), sc->size);

		local_irq_save(flags);
		t0 = ktime_get_ns();
		dma = dma_map_single(dev, buf, sc->size, DMA_FROM_DEVICE);
		if (!dma_mapping_error(dev, dma))
			dma_unmap_single(dev, dma, sc->size, DMA_FROM_DEVICE);
		t1 = ktime_get_ns();
		local_irq_restore(flags);

		if (dma_mapping_error(dev, dma)) {
			ret = -EIO;
			break;
		}

		d = t1 - t0;
		sum += d;
		min = min_t(u64, min, d);
		max = max_t(u64, max, d);
	}

	if (!ret) {
		u64 mean = sum, ns_per_kib;

		do_div(mean, sc->reps);
		ns_per_kib = mean * 1024;
		do_div(ns_per_kib, sc->size);
		pr_info("dmabench: %-16s size=%8zu reps=%u mean_ns=%llu min_ns=%llu max_ns=%llu ns_per_KiB=%llu\n",
			sc->name, sc->size, sc->reps, mean, min, max, ns_per_kib);
	} else
		pr_err("dmabench: %s: dma_map_single failed\n", sc->name);

	kfree(buf);
	return ret;
}

static int __init dmabench_init(void)
{
	struct device *dev;
	unsigned int i;
	int ret;

	pdev = platform_device_register_simple("dmabench", -1, NULL, 0);
	if (IS_ERR(pdev))
		return PTR_ERR(pdev);
	dev = &pdev->dev;

	ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(32));
	if (ret) {
		platform_device_unregister(pdev);
		return ret;
	}

	pr_info("dmabench: START\n");
	for (i = 0; i < ARRAY_SIZE(scenarios); i++)
		run_scenario(dev, &scenarios[i]);
	pr_info("dmabench: DONE\n");
	return 0;
}

static void __exit dmabench_exit(void)
{
	platform_device_unregister(pdev);
}

module_init(dmabench_init);
module_exit(dmabench_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Streaming DMA_FROM_DEVICE map/unmap cost benchmark");

-- 
Karl

On Thu, Sep 10, 2026 at 11:14:28AM +0200, Arnd Bergmann wrote:
> On Thu, Sep 10, 2026, at 08:36, Karl Mehltretter wrote:
> > This series prevents a DMA_FROM_DEVICE map from discarding CPU-written
> > buffer contents on non-coherent 32-bit ARM. It is based on
> > v7.3-rc1-324-g986c24e0fe44.
> >
> > ARM currently invalidates these buffers before the device writes them.
> > If the device writes only part of a buffer, discarded dirty cache lines
> > can expose older memory contents in the untouched bytes. A stock USB
> > webcam demonstrated this through usbfs. Short isochronous packets left
> > gaps, and usbfs returned non-zero data to userspace from bytes it had
> > cleared.
> >
> > arm64 changed this handoff from invalidate to clean in 2022 with commit
> > c50f11c6196f ("arm64: mm: Don't invalidate FROM_DEVICE buffers at start
> > of DMA transfer"). Arnd Bergmann's 2023 ARM32 cache-maintenance series
> > left DMA_FROM_DEVICE preservation unresolved [1]. This series keeps the
> > existing ownership hooks.
> 
> I think the main problem here is that we remain inconsistent about the
> rules across CPU architectures, and changing Arm on its own does not
> mean we have a solution if another architecture decides to change it
> in the opposite direction at some point.
> 
> I see this as a tradeoff that can go either way:
> 
> - the current 32-bit Arm approach (also arc, hexagon, microblaze, mips,
>   nios2, openrisc, powerpc32, sh) is obviously faster as it avoids
>   the writeback, but it relies on device drivers to ensure no stale
>   data can leak back into userspace.
> 
> - Will's patch changed arm64 (later copied into riscv) to avoid that
>   risk by adding the overhead out of caution, and avoid having to
>   audit and fix all drivers.

How would you envisage fixing a driver for this? There were two issues
I tried to address by moving from invalidate to clean on arm64:

1. If the DMA transfer didn't write every cacheline in the buffer, then
   we could expose stale data in the gaps.

2. If the buffer has a pre-existing userspace mapping, then we expose
   stale data during the window between the DMA map() call and the DMA
   itself.

Fixing (1) in the driver would presumably require it to walk through the
buffer after the transfer and zero all the gaps, with an appreciation
for the cache writeback granule (!= cacheline size) and then (somehow)
clean those parts back to the PoC. Is that something any drivers attempt
today?

Fixing (2) in the driver would presumably require ruling out the
possibility of a user alias, which sounds hard and possibly ABI breaking
for some drivers (depending on how they manage their buffers).

> If we decide to align with arm64/riscv and take your series, I think
> we need two more parts:
> 
> - actually measure the performance overhead: you already did the
>   work to test this on three separate arm implementations but did
>   not share performance numbers.
>   Can you quantify how much this costs us on the hardware you used?

It's worth noting that many Arm CPUs upgrade invalidate to
clean+invalidate (either due to the micro-architecture, errata or because
of virtualisation).

Will

On Thu, Sep 10, 2026, at 12:55, Will Deacon wrote:
> On Thu, Sep 10, 2026 at 11:14:28AM +0200, Arnd Bergmann wrote:
>> I see this as a tradeoff that can go either way:
>> 
>> - the current 32-bit Arm approach (also arc, hexagon, microblaze, mips,
>>   nios2, openrisc, powerpc32, sh) is obviously faster as it avoids
>>   the writeback, but it relies on device drivers to ensure no stale
>>   data can leak back into userspace.
>> 
>> - Will's patch changed arm64 (later copied into riscv) to avoid that
>>   risk by adding the overhead out of caution, and avoid having to
>>   audit and fix all drivers.
>
> How would you envisage fixing a driver for this? There were two issues
> I tried to address by moving from invalidate to clean on arm64:
>
> 1. If the DMA transfer didn't write every cacheline in the buffer, then
>    we could expose stale data in the gaps.
>
> 2. If the buffer has a pre-existing userspace mapping, then we expose
>    stale data during the window between the DMA map() call and the DMA
>    itself.
>
> Fixing (1) in the driver would presumably require it to walk through the
> buffer after the transfer and zero all the gaps, with an appreciation
> for the cache writeback granule (!= cacheline size) and then (somehow)
> clean those parts back to the PoC. Is that something any drivers attempt
> today?

I'm not aware of any driver doing this, but also haven't tried looking
for them. I think the usual assumption is that a driver asking for
a variable-length reply should ensure that it doesn't access of the
data that was not returned, and that the driver understands which
parts were received.

One thing that the arm32 implementation (but not any others as far
as IIRC) does is to do a writeback+invalidate for any partial
cache lines passed into dma_sync_*(), but this of course does
not handle short transfers.

We had at some point discussed using KASAN to debug these better:
mark any memory that is passed to a device as unaccessible through
the DMA mapping API (rounded up to full cache lines), and then mark
the data as accessible again during the sync to the CPU (not rounding
up). As long as the driver only passes the actually received size
into dma_sync_single_for_cpu(), any later access would trigger
a KASAN assertion.

> Fixing (2) in the driver would presumably require ruling out the
> possibility of a user alias, which sounds hard and possibly ABI breaking
> for some drivers (depending on how they manage their buffers).

There are not that many subsystems that do streaming DMA into
user-mapped buffers, so I also can't think of any good example
here where things would actually go wrong in practice. Have you
been able to find an example that runs into this scenario?

Block drivers always transfer entire pages, and I don't think you
can access a page until a transfer from userspace has completed.
GPU and media drivers might be affected, but it looks like those
usually use coherent mappings.

>> - actually measure the performance overhead: you already did the
>>   work to test this on three separate arm implementations but did
>>   not share performance numbers.
>>   Can you quantify how much this costs us on the hardware you used?
>
> It's worth noting that many Arm CPUs upgrade invalidate to
> clean+invalidate (either due to the micro-architecture, errata or because
> of virtualisation).

Right.

    Arnd

On Thu, Sep 10, 2026 at 08:36:18AM +0100, Karl Mehltretter wrote:
> This series prevents a DMA_FROM_DEVICE map from discarding CPU-written
> buffer contents on non-coherent 32-bit ARM. It is based on
> v7.3-rc1-324-g986c24e0fe44.

> Karl Mehltretter (2):
>   ARM: dma-mapping: preserve DMA_FROM_DEVICE buffer contents
>   ARM: dma-mapping: flush FROM_DEVICE buffers on legacy backends
> 

Adding Linus at his current email address.

Sorry for the noise.

Karl