md treats an unsupported P2PDMA transfer to a member as success, and
nvme-rdma retries one forever. Whether peer memory can be DMA-mapped
depends on the PCIe topology between the two devices, and since v6.17
the failure completes as BLK_STS_INVAL: retryable for multipath and
deliberately ignored by md for member failures, so mirrors silently
diverge.
v1-v3 restored BLK_STS_TARGET for these failures and tagged P2PDMA
bios in md. Logan Gunthorpe proposed a dedicated BLK_STS_P2PDMA
instead, so no consumer needs to know whether an I/O was P2P, and
wrote the core patches; v5 is built on them. Patches 1 and 5 are
his and keep his authorship.
1 block: add BLK_STS_P2PDMA (stable, v6.17)
2 md: ensure REQ_NOMERGE is set on P2PDMA bios
3 md/raid1: serialize non-write-behind writes on CollisionCheck
rdevs (pre-existing bug patch 4 would widen; stable)
4 md/raid1: no write-behind for P2PDMA bios
5 md/raid1,raid10: factor out raid1_write_error() helper
6 md/raid1,raid10: keep REQ_NOMERGE on narrow_write_error()
retry clones
7 md/raid1,raid10: skip futile retries on P2PDMA mapping
failures
8 md/raid1,raid10: set IO_BLOCKED in case of BLK_STS_P2PDMA
9 nvme-rdma: use ib_dma_map_sgtable_attrs() (stable, v7.1;
dependency of patch 10)
10 nvme-rdma: return BLK_STS_P2PDMA (stable, v7.1; needs patches
1 and 9)
Routing: patch 1 block, 2-8 md, 9-10 nvme; 7, 8 and 10 use the new
status. Patch 1 alone stops the divergence and the dm-multipath
requeue loop; the nvme-rdma loop also needs 9-10; everything else
refines the fallout, so the series bisects safely. The md patches
fix a v7.2-rc1 regression (02666132403a), except 3 (older bug,
stable) and 5 (refactor).
Tested with the QEMU raid1/raid10 error matrix and a ConnectX-4 Lx.
After patch 1, block-layer error injection accepts status=P2PDMA, so
the md completion paths can be exercised on any member with no P2P
hardware. The patches were developed with AI assistance (see the
Assisted-by trailers); all code was human-reviewed and tested.
Against v7.2-rc4 (Logan's base).
Changes in v5:
- patches 7, 8: From: is now Mykola (Logan's request)
Logan keeps Signed-off-by on both, Co-developed-by on 8
- patch 8: raid10 declares bio up front and bi_status sits in a
blk_status_t local in both files.
- old patch 9 split: 9 is the mechanical sgtable conversion,
10 the -EREMOTEIO -> BLK_STS_P2PDMA translation; both stable
since 10 needs 9
Changes in v4:
- new BLK_STS_P2PDMA (Logan) replaces restoring BLK_STS_TARGET; the
R1BIO_P2PDMA/R10BIO_P2PDMA state bits are gone and the md changes
shrink to completion-time status tests
- blk_dma_map_iter_start() gets the explicit
PCI_P2PDMA_MAP_NOT_SUPPORTED case (Logan's patch; v3 kept
default: only)
- write completions record the whole failed range as bad blocks
instead of narrow_write_error()'s per-block retries; v3's coarse
retry is gone -- the new status is deterministic, nothing to
re-attempt
- read path: bi_status is evaluated before bio_put()
Link: https://lore.kernel.org/linux-raid/20260718162547.448892-1-mykola@meshstor.io/ [v1]
Link: https://lore.kernel.org/linux-raid/20260719105327.864949-1-mykola@meshstor.io/ [v2]
Link: https://lore.kernel.org/linux-raid/20260721174502.111503-1-mykola@meshstor.io/ [v3]
Link: https://lore.kernel.org/linux-raid/20260722185841.449934-1-mykola@meshstor.io/ [v4]
Link: https://lore.kernel.org/linux-raid/9f846c85-f427-433a-b87d-8e9e3a4406e7@deltatee.com/ [Logan's proposal]
Logan Gunthorpe (2):
block: add BLK_STS_P2PDMA for unsupported peer-to-peer transfers
md/raid1,raid10: factor out raid1_write_error() helper
Mykola Marzhan (8):
md: ensure REQ_NOMERGE is set on P2PDMA bios
md/raid1: serialize non-write-behind writes on CollisionCheck rdevs
md/raid1: don't use write-behind for P2PDMA bios
md/raid1,raid10: keep REQ_NOMERGE on narrow_write_error() retry clones
md/raid1,raid10: skip futile retries on P2PDMA mapping failures
md/raid1,raid10: set IO_BLOCKED in case of BLK_STS_P2PDMA
nvme-rdma: use ib_dma_map_sgtable_attrs()
nvme-rdma: return BLK_STS_P2PDMA for unsupported P2P transfers
block/blk-core.c | 3 +++
block/blk-mq-dma.c | 3 ++-
drivers/md/md.c | 10 +++++++--
drivers/md/md.h | 15 +++++++++++++
drivers/md/raid1-10.c | 16 +++++++++++++
drivers/md/raid1.c | 47 +++++++++++++++++++++++----------------
drivers/md/raid10.c | 27 +++++++++++-----------
drivers/nvme/host/rdma.c | 38 ++++++++++++++++++-------------
include/linux/blk_types.h | 10 +++++++++
9 files changed, 118 insertions(+), 51 deletions(-)
base-commit: 1590cf0329716306e948a8fc29f1d3ee87d3989f
--
2.52.0