[PATCH 0/2] sunvdc: fix silent data loss under LDC congestion

Stian Halseth posted 2 patches 3 weeks, 3 days ago
drivers/block/sunvdc.c | 26 ++++++++++++++++++++++++--
1 file changed, 25 insertions(+), 1 deletion(-)
[PATCH 0/2] sunvdc: fix silent data loss under LDC congestion
Posted by Stian Halseth 3 weeks, 3 days ago
Since commit a11f6ca9aef9 ("sunvdc: Do not spin in an infinite loop
when vio_ldc_send() returns EAGAIN"), Linux guests in LDOMs see
occasional spurious write failures that manifest as filesystem
corruption. The failure signature is:

    sunvdc: vdc_tx_trigger() failure, err=-11
    I/O error, dev vdiskc, sector ... op 0x1:(WRITE) ...
    vdiskc: writeback error on inode ...

The buffered writeback data is silently dropped. A revert was
rejected [1]; Jens proposed requeuing instead [2], but that patch,
tested in October 2025, appeared to leak memory and the effort
stalled [3].

The leak is real and diagnosed: __send_request() maps the request's
pages into the LDC channel's fixed-size map table (ldc_map_sg())
before ringing the doorbell, and the trigger-failure path never
unmaps them. A descriptor whose doorbell was never sent never
completes, and since dr->prod is not advanced, vdc_requeue_inflight()
never visits it either. With the requeue approach every retry remaps
the same descriptor, orphaning the previous mapping; the 8192-entry
per-channel table drains (contiguous allocation makes large requests
fail first as it fragments) and the disk eventually dies with
-ENOMEM. Mainline leaks the same way, once per spurious EIO.

Patch 1 fixes the leak (also a standalone mainline fix). Patch 2 is
Jens' requeue patch, unchanged, on top.

Why the bounded busy-loop cannot be "tuned" instead: the LDC tx
queue holds 128 doorbell messages while the dring holds 512
in-flight descriptors. Any writeback burst of >128 requests
(ordinary for XFS multi-stream writeback) overflows the message
queue whenever the vds service thread in the primary domain is busy
for a few milliseconds - e.g. one txg sync on the backing pool.
This is normal flow control, not an error condition; OpenBoot's own
disk driver retries it indefinitely.

Testing on an UltraSPARC T4 LDOM (Debian sid guest, 8 vcpu, disk on
a zvol served by a Solaris 11.4 primary), workload = 8-way parallel
kernel-tree copies plus periodic 512-file sync bursts on XFS:

 - Unpatched (7.2.2 + nothing): three failure events in 3.6 hours,
   each losing writeback data, on an otherwise near-idle host.
 - Patched (7.2.2 + this series), identical workload: the same
   trigger fired (vdc_tx_trigger message logged) and was absorbed by
   the requeue - no I/O error, no data loss, no checksum mismatch in
   the verified copies.

Independently, the tester who found the October 2025 memory leak has
re-run his original harness (parallel tree copies, SPARC T4-2, 6.18
plus this series): 20 hours, ~130M IOPS across six cores, 861
requests requeued after EAGAIN timeouts, no memory leak observed [3].
In the original run of that harness, those events surfaced as EIO
and the leaked mappings eventually took the LDom down.

A note on semantics: with this series a permanently unresponsive
backend parks I/O indefinitely (as pre-a11f6ca9aef9 kernels and
Solaris guests do) instead of failing after ~10ms. The per-vdisk
"timeout" property (ldm set-vdisk timeout=N) remains the bounding
knob for deployments that prefer failure over waiting.

[1] https://lore.kernel.org/all/20251006100226.4246-2-glaubitz@physik.fu-berlin.de/
[2] https://lore.kernel.org/all/418310b3-2b77-4534-b2fd-27dcc11e333c@kernel.dk/
[3] https://github.com/sparclinux/issues/issues/2

Stian Halseth (1):
  sunvdc: unmap LDC cookies when the descriptor send fails

Jens Axboe (1):
  sunvdc: fix -EIO issue due to lack of retries

 drivers/block/sunvdc.c | 26 ++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 1 deletion(-)

--
2.53.0
Re: [PATCH 0/2] sunvdc: fix silent data loss under LDC congestion
Posted by Jens Axboe 2 weeks, 1 day ago
On Tue, 01 Sep 2026 19:39:44 +0200, Stian Halseth wrote:
> Since commit a11f6ca9aef9 ("sunvdc: Do not spin in an infinite loop
> when vio_ldc_send() returns EAGAIN"), Linux guests in LDOMs see
> occasional spurious write failures that manifest as filesystem
> corruption. The failure signature is:
> 
>     sunvdc: vdc_tx_trigger() failure, err=-11
>     I/O error, dev vdiskc, sector ... op 0x1:(WRITE) ...
>     vdiskc: writeback error on inode ...
> 
> [...]

Applied, thanks!

[1/2] sunvdc: unmap LDC cookies when the descriptor send fails
      commit: 0c6da21fa35e03fc74f09895433ccd6d4a9c3530
[2/2] sunvdc: fix -EIO issue due to lack of retries
      commit: 5067d4ba713961d8ccea1e06cd4c453793f3121e

Best regards,
-- 
Jens Axboe
Re: [PATCH 0/2] sunvdc: fix silent data loss under LDC congestion
Posted by John Paul Adrian Glaubitz 2 weeks, 2 days ago
Hi Jens,

any chance you could take a look at this series?

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913