The PIO_CRED case of hfi1_file_mmap() hands user space the credit-return
page for its send context. On a two-socket host with a translating IOMMU
it instead returns a page mapped from a frame above MAXPHYADDR, and the
first user read takes "Corrupted page table" / "Oops: Bad pagetable".
Correcting only the offset arithmetic replaces the Oops with a silent
wrong-page mapping, so every transfer that uses send PIO hangs instead.
The failure is intermittent: it depends on which of the credit-return
pages the context's entry lands on, which follows the hardware send
context index and so varies from boot to boot.
Patch 1 fixes the node used to resolve the credit-return buffer, which
has been wrong since the driver was merged. Patch 2 fixes the byte
offset applied to a typed pointer and the use of dma_mmap_coherent(),
both introduced when this case was converted to the DMA API. They are
separate patches because they have different Fixes: tags, but only the
two together make the mapping correct: patch 2 without patch 1 turns the
hang into an -ENXIO from iommu_dma_mmap()'s bounds check, because the
cross-node offset exceeds the buffer.
Found while debugging psm2_ep_open() Oopsing a Dell T7610 (Xeon E5-2650
v2, Intel IOMMU in DMA-FQ mode) talking to a Threadripper PRO 3995WX,
both Omni-Path 100. With the series applied to both hosts,
psm2_ep_open() succeeds, send PIO, send DMA and the default mixed mode
all work, and MPI over the PSM2 MTL sustains 95 Gb/s. Forcing send PIO
only (PSM2_SDMA=0), which hung indefinitely before, now completes.
Based on rdma/for-rc at 31b7c700670830a0e8a4cdcd451c88a13cc5dc48.
Shuhei Takeshita (2):
IB/hfi1: Resolve the credit-return buffer through the send context's
node
IB/hfi1: Fix the PIO_CRED credit-return mmap
drivers/infiniband/hw/hfi1/file_ops.c | 23 +++++++++++++++++------
1 file changed, 17 insertions(+), 6 deletions(-)
--
2.43.0