From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8447440A20 for ; Thu, 23 Jul 2026 20:43:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839388; cv=none; b=JuMwcPSJEQiDV5x7gCR3f5u4BiWfoRG9Z0x01bG3U7QV7QznjOu+x6ydLiA4hmhN2HlmSjLlhiuAMjzxYWaIryR/rpw2tQJ9Kx1xx6kX7Y2BtsTNGLpS8XoHryz4QYhlL68cjzWUHuTILcGqM9Luot96mC8NFu+HDq5Hq+DHqN4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839388; c=relaxed/simple; bh=cRH7W6syefYxFA3VhhN4QvzFBoE55r52A/E6t9wrD1M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HeNU9axSdKFGm3MLxkwClLTdFkjj40zUcmoRcgP+zuPxv2ot7Tb6TaVg1dWPF/emICEf2QmewHMdKQEQqCz0gGmAD3LAodb1p+y3Hy1F3b1fpNLQo+tpYdxU6TN1zplCTeEir7jjPGaorfTTN/GElvQaprC1O1xhUDGqlYvQH5w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=SesB9aK6; arc=none smtp.client-ip=209.85.221.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="SesB9aK6" Received: by mail-wr1-f43.google.com with SMTP id ffacd0b85a97d-47362928f65so1026825f8f.2 for ; Thu, 23 Jul 2026 13:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839375; x=1785444175; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LNWXk+rlZiNRYyXRDGVV7qCvY4AZPHPO/XYCzCCpSWs=; b=SesB9aK6CHrwPAFEI1B193p8OPva428cDKaNqpizhHiUhewUqHwqrmjD/xRwufSFok C0QIDvLbdC9pPZ+CEdXlIOyILZKCt6tsMZk99YsEvzEVMMXsNcXjvcShGE7KV9h62HlJ oyLFTnuqUYUO/StJAprXzTVNjytYoEhdeRDI21Aw4klritgm4+4yPAYSIaUwQVxtIrAr U+vpIpCu+rXzhzxyPyWR2tBGmf/yyfgr1Pw3Vt60fpaT+bjvDrn5gmUldtn9AlpfwEOR YUhDPe/enCYSdSa5Bm/9KVvO/O4FfEjHycGze1EOP2z7Zz1ZeqgPV1ph5RSwUjsi9PbK eQQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839375; x=1785444175; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LNWXk+rlZiNRYyXRDGVV7qCvY4AZPHPO/XYCzCCpSWs=; b=ItM9+dnPlNEqh+PR0uYz13Sn9Wu8Ms6zQSBONTVMycYj/hOakSzacNufozkA2pQBPu W59a4WoUHeJfxnf/XLr1QdG9+1hNozq2vbPvP86NjRPqjVcBbrphavUU1+RM0r7/jElz Y4e3SdGzWNRzFKP6ojcj8/oXMJjlWYjsF055YKptvrpvOKR1au225uA5NSMQZf4rLmRM M/cmaZD0h1Qjcf46gkRQnzT+AaTjCDyroXDDIcjNexdeizin/lQV3H9R1GsiW5PeB13z NCgOawLMdqbKcqOsTcQ9bxw+3ROQa06w20puTk8ppogOhAIwSCmdeWAHsM5Zer1fYzS7 fXIg== X-Forwarded-Encrypted: i=1; AHgh+RpvhP69ui4e3vX9JLZzL88f7QO73aDQeQpq8aE6HCMAs9wPCNvHsl9K6c4oHRfBAN6N8KkCIY4GVelqBYc=@vger.kernel.org X-Gm-Message-State: AOJu0YyQGbxGqBIUcA923tl6wXbWziUrAxcvqlqq/Qko1qoOlXjeQCrB 4gfTh3D6LYLzEQctb4PH5WU040t2fG9tl0voytAR2kB2GGG9n0p98m87rytm/reVNg== X-Gm-Gg: AR+sD12pr0u/hpquatresx3H90MZsPtzw0ERnwdQ+gftxtl/K/C2+MuS+Lol86hKhAK wxNKGZ0oW1autHB7vkmrNzM7QklrvZCIhBq3kMI3FfBBtWkWhtIJaCWefUtMKzaSxDbUPM1SN4G Bp0WFWBy5+PU4O2VWJwlpTyNpf97XyBIR/9e+l4IOtX6YYnUZAwzKxy+gt/zXtWGQAiObRCGwTW X1cbFfIeRV89kjvyBS4UlVutem0JVeukhJ55iq/LoPfMFq+zFlOb3O5K3Wx2BXn6WQnhty/5XoV hqpfSKznIGJWvYIbMekGG3+239qEni+/6P54oV9VcEAE6rqkO6VuoddqgfKf83bZShltAQIiYFB Dt1g1UEvgZx9VJLXs/qJtYbOmsynF4RBn5Gz+u6eIpaGTGkMnvHK6l95iI5gyA2s2MS4aeh/fot OC+pV8LYqCpFJ8nurO6YUUUxLf8Ngf4I3uPL4zg8o/Z0RZeMwhG1525RwHGAf8ACIOxSa1PXo9Z 1ifP3Hp0Eh0YB5uV8975wGNFV9Yrff5YYUboDGaEw== X-Received: by 2002:a05:6000:240b:b0:47f:87f9:6d06 with SMTP id ffacd0b85a97d-47f8da5e25bmr5949800f8f.61.1784839375286; Thu, 23 Jul 2026 13:42:55 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.42.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:42:54 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 01/10] block: add BLK_STS_P2PDMA for unsupported peer-to-peer transfers Date: Thu, 23 Jul 2026 22:41:57 +0200 Message-ID: <20260723204206.76930-2-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe blk_dma_map_iter_start() reports BLK_STS_INVAL when a P2PDMA transfer is attempted between two devices whose PCIe topology cannot route it. blk_path_error() treats INVAL as retryable, so multipath requeues the I/O forever, and callers cannot tell it apart from an invalid request. md also ignores it for member failures, so a mirror silently diverges. Restoring the old BLK_STS_TARGET would stay ambiguous -- nvme maps NS_NOT_READY, LBA_RANGE and CMD_INTERRUPTED to it too. Add a dedicated BLK_STS_P2PDMA, non-retryable in blk_path_error(), and return it from blk_dma_map_iter_start(). Fixes: 858299dc6160 ("block: add scatterlist-less DMA mapping helpers") Fixes: 7ce3c1dd78fc ("nvme-pci: convert the data mapping to blk_rq_dma_map") Cc: stable@vger.kernel.org # v6.17 Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan --- block/blk-core.c | 3 +++ block/blk-mq-dma.c | 3 ++- include/linux/blk_types.h | 10 ++++++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index 365641266c9e..83cb5464265f 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -177,6 +177,9 @@ static const struct { =20 /* Command duration limit device-side timeout */ ENT(DURATION_LIMIT, -ETIME, "duration limit exceeded"), + + /* no PCIe P2PDMA route between initiator and target */ + ENT(P2PDMA, -EREMOTEIO, "peer-to-peer transfer unroutable"), ENT(INVAL, -EINVAL, "invalid"), =20 /* everything else not covered above: */ diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c index bfdb9ed70741..e88d6b3524e6 100644 --- a/block/blk-mq-dma.c +++ b/block/blk-mq-dma.c @@ -189,8 +189,9 @@ static bool blk_dma_map_iter_start(struct request *req,= struct device *dma_dev, */ case PCI_P2PDMA_MAP_NONE: break; + case PCI_P2PDMA_MAP_NOT_SUPPORTED: default: - iter->status =3D BLK_STS_INVAL; + iter->status =3D BLK_STS_P2PDMA; return false; } =20 diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 8808ee76e73c..c8de7809320a 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -165,6 +165,15 @@ typedef u16 blk_short_t; */ #define BLK_STS_DURATION_LIMIT ((__force blk_status_t)17) =20 +/* + * BLK_STS_P2PDMA is returned when a peer-to-peer DMA transfer cannot be + * mapped (typically because there is no route between the initiator and t= he + * target that the platform's PCIe topology and ACS settings allow). This + * is a property of the initiator/target pair, not of the target device's + * health. This error should never be retried. + */ +#define BLK_STS_P2PDMA ((__force blk_status_t)18) + /* * Invalid size or alignment. */ @@ -188,6 +197,7 @@ static inline bool blk_path_error(blk_status_t error) case BLK_STS_NOTSUPP: case BLK_STS_NOSPC: case BLK_STS_TARGET: + case BLK_STS_P2PDMA: case BLK_STS_RESV_CONFLICT: case BLK_STS_MEDIUM: case BLK_STS_PROTECTION: --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5F2941B8CB for ; Thu, 23 Jul 2026 20:42:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839385; cv=none; b=GC8dX99syL7tsQTJ3XFxELVvd1u1Bb/lVSStJitxwA+485rgerOVet+4oZm1TdMKo1GvP8lhiIMBj5t7ygpfOCvtduu4RB5rGl+gnaon2AO7ImgSD3XO8UHDyZijtMc20ESn3JyfX/B1pDMlvWGGxWvx1pYE/ttBvEW2MUv9HfM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839385; c=relaxed/simple; bh=UmHDQ8Avh88213YFsBTKp7WY9yAPzXpx6PsDprJSvWw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TTdbO0giG7L6uB8MaGq9Ev+dAr4fwzwE42FGsA8jOB64qUf3hRUdjU15gpMpPi+QRXDuDkY5YLDQ6U16fWmRnfLLM6ZeNIcrDzcqxc175GznsyADDGuahBZ01jKJEXcFP8SM68V5fDLNJxtOEbxK0Aq9MBcfxL5j6yKWGk2k2NQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=C1ciT6M8; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="C1ciT6M8" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-47f3b39f2a1so1007046f8f.2 for ; Thu, 23 Jul 2026 13:42:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839377; x=1785444177; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=V5hliKyCAPWFZqHS5mW2T1dJRPt4J/vTka7mcNCHq10=; b=C1ciT6M83/HGZQhgGwKFZG1M8g06jwXG8b8SpyZRp6tX/vwlKnhVAXGgDf1qHqNpqQ bqjGUOBgB0Y3+3Ng/7CsHYOhB59s/nnFmWKDWObGXB0f7cHQyNK/08IVOvMpodeTywav 7p7H2AZ1G51wZrkcapsGj8Bjr8x/1tftZhYVsDbAHMsO2AtTRNrRmgZGmaCu3ebPJBnw NFFMuCNvwAb4ba+Z8zafhO/h3+NMMxHT2nWdbdeedj8Pj9Wv/rUwglTmi3U3NzvNvF+0 E4Zw3lU+0sid2q6loyN+AZ9oAivVppyVhIDO3ZmHa2tfeGd21EXa8mgwJRpOhASeK1x1 GUcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839377; x=1785444177; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=V5hliKyCAPWFZqHS5mW2T1dJRPt4J/vTka7mcNCHq10=; b=G9Z3kn8ZTO0NQtEnRmdzAUKapNGGdqTxc7VZ2CJLFRpgBz94oLlCbznmDw1UiLEhma u2RaBubLOZM7KuvktWsfWzEupdmKx5VRjK058D+D2+rO9TZyoJpqzQy27mNZ4U7s7X3k pth8eMwvQYyljCdHzn2oL/WuAYog8XowpjKUBwIs96p3fpi5nqk6qORGWn1ITdTxa8pu RukDBdUoZCuIh68m7FYsj9DXEebQz8vHPy3104WcC31DYZjCvo7oMTw47S2vS03OLzDV 4x0vPrVBRTFu7q6sjbIudpfa/Xlsp+f9XSjxR6pbpM140Neog1cxcHr/cbER0PSkfYjT ZlOA== X-Forwarded-Encrypted: i=1; AHgh+RoU6DxSTv9bGiM+zl2cUmEja609Jk/ok9gb2fxyrfYbPawW6SijDIJsk+MIF7LSIrOdI5cPSjuLSoRrEQQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxiBXo536pTZNOdaQOXuAK4MQcph1++EwtlT0jQlbF8Bkwjwhcy mbBzQjrcznZgZdldpuWIIHHOC554JkD3BTc5QxL2ORYkEoxYn7cIqiQ4LY1r41qOzQ== X-Gm-Gg: AR+sD13jet/FxorB2d8VTAKciAYswNRd0TcClGuYQmF1ru27C3++u3YlPtNV15YHpmi Yd7/DX5as8vDJNQ+7pVUOmMS9PF0h3/ZHYgouA1hREU1mec5iDln2/5WJRGonq/O27/XgpBVXII B1A2EKCbhbGhtlNzg06vGj9AsBjkx18uPe0OO7Qanl3qSIN5DivCNpfL1YeSR8QsNA2CB5WkwaR 6s+OxqR4Ba5JFykBwrXLSxgbKSqJK78Jx7EpRquFKm6WnlJdnz03MUhQFm/TC5vENo31fsTkxdh uH8eCIR9Nukv26nnZglvKghh3Ro9UHFgo4SmMKN2eXifR0ni3+CUB7rKAfNTkqNPNLDIBLj3bG7 PK2GaQH19/kcIaR/aBBOqpXjK84/Guc+bMbFNmTxtfXGTBFL8YJFLh59OfxktEWP8S8CRXagQqW sLLG6PLfFFKFO3KSGpBjxa5vo9tOUpEexGlWNw8SpEctEuaijLqnLyAPcjM2KEW+oUIOAm7G5/6 fIg2uEyN0nyuQxqIo8BE2EFXgFSqdwqsVjCdUVxxA== X-Received: by 2002:a05:6000:603:b0:47f:5a97:2a63 with SMTP id ffacd0b85a97d-47f8d6177bcmr6091080f8f.9.1784839377410; Thu, 23 Jul 2026 13:42:57 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.42.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:42:56 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 02/10] md: ensure REQ_NOMERGE is set on P2PDMA bios Date: Thu, 23 Jul 2026 22:41:58 +0200 Message-ID: <20260723204206.76930-3-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" md_submit_bio() strips REQ_NOMERGE from every bio. For P2PDMA bios the flag is what keeps requests single-provider (see __bio_add_page()): merging a P2PDMA bio with one over a different pgmap, or over host memory, maps the merged request with the wrong bus address. Set the flag on P2PDMA bios -- bios built via bio_iov_bvec_set() arrive without it -- and keep stripping it otherwise. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/md.c | 10 ++++++++-- drivers/md/md.h | 15 +++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index d1465bcd86c8..3770bdb4d4b2 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -451,8 +451,14 @@ static void md_submit_bio(struct bio *bio) return; } =20 - /* bio could be mergeable after passing to underlayer */ - bio->bi_opf &=3D ~REQ_NOMERGE; + /* + * A bio md split may merge again below md -- except P2PDMA bios, + * which must stay single-provider (see __bio_add_page()). + */ + if (md_bio_is_p2pdma(bio)) + bio->bi_opf |=3D REQ_NOMERGE; + else + bio->bi_opf &=3D ~REQ_NOMERGE; =20 md_handle_request(mddev, bio); } diff --git a/drivers/md/md.h b/drivers/md/md.h index d8daf0f75cbb..73df1a6eccd8 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -11,8 +11,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -22,6 +24,19 @@ #include =20 #define MaxSector (~(sector_t)0) + +/* + * P2P and host pages never mix within a bio, so the first bvec is + * representative. Read bi_io_vec directly: bio_first_bvec_all() + * WARNs on the split clones md handles, and data-less bios have no + * bi_io_vec. Not valid after the bio's iterator is consumed. + */ +static inline bool md_bio_is_p2pdma(struct bio *bio) +{ + return bio_has_data(bio) && bio->bi_io_vec && + is_pci_p2pdma_page(bio->bi_io_vec->bv_page); +} + /* * Number of guaranteed raid bios in case of extreme VM load: */ --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 774D945D1A9 for ; Thu, 23 Jul 2026 20:43:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839386; cv=none; b=VB6gk8gB/2sri+SbqhmxuYx5i9fU83iIkAmkAHMsy1LaUMvR1ojGihUJ0VUL3o75O264bwx2N54rH2FpG7q2Q1mQayiAYSQ7vr5oFd75tce6gpOqlYHkVGgMy6BkImwYbP8XvXlMnP329l/UU/vetK1hYmtR0FJzFSmqysJTzS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839386; c=relaxed/simple; bh=IeQBFCH3bM4bVxRkUROK9805WYYXPr2VLMisCc8LPq4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kQjghmvK8BQ3CJPV0vVQ/9zprcLL9+LukilGQW7Uz73USAGSYBqJxgvz6hFvqezHz4f/nCIgbUKR+sGjhsSSpSqxwDCYkOeLmDMizr2xMuXRW8yErDN5uNZRSqMHQWoYECs8fOSR+5OW5/DWbiqIAzwKJ7hJOOm4jl/hVUub/+I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=mF1+onUo; arc=none smtp.client-ip=209.85.221.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="mF1+onUo" Received: by mail-wr1-f44.google.com with SMTP id ffacd0b85a97d-47db714766aso493558f8f.0 for ; Thu, 23 Jul 2026 13:43:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839380; x=1785444180; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=xY1kaJm76bL7mULPuQYN6MgHSE4hk4KEujLtUR/lx4I=; b=mF1+onUoH8g5krZlDyx9lcBqRTRrF8paWBw3zaSjWmlVE6WvKJ0hEWSwfUYbccpbQ1 t9YUnOrozr0YB0sILt9Ctur/Nz1zQ/+OXnwO4Yateq4zcnMQslQTbkXRAru/PCvt6s0v oJii8kk5FyPuMEupv9MF29Mgc1Y5C6bhNWmwe7p0TOW3xEArB6PLHLZqPweAU+ntzo1B ktrg4VEg0Gh8PWx+hQQgfsVKYPv7TvjDzrQV917gexA2Xy6m7/K6teAHHecYcbfbwQil nTYInJ/HyX7DpgOhTOvw6luyy1O0Q+pApnsCG6gVUm6a/fJMfcX/7HCemMP8nZ376T66 QUmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839380; x=1785444180; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=xY1kaJm76bL7mULPuQYN6MgHSE4hk4KEujLtUR/lx4I=; b=XZWXP5UrWv1GByT6+9kFkPAJaXftpYZeX9XmomJc3Oi7iDUg9zbOgYtv3XcpYF+q8K ZeiF1r9WMz11U8/xic5ycelnZaUDfTL/oyVjpg6uGGKTDZzbw+MythF/eO0n/o62mBa5 Xa7osCkf01YjlRo2AScpgx/l8Ls1W2clxtmghAeNo79BTckvI9OaKYlyDtnbLP8quynU miwcWZn8KxHhSE85vFybRMSiJom39y5MrVBcOw8ivQyBKCJz9mB/JqQLIijGcoiOBhfL UvkvA/ZYQnbArfkOlKlrZ/wVqSKSO1pDxm2RrdrwcpO7Fty8U2aWLLApwrNtR7apzt0+ BdnA== X-Forwarded-Encrypted: i=1; AHgh+Rqzc6cHMn3mar/F92pCJ6YyQLMTbvl4BHfjrnG2wd8Zo2R2Y3PASTLVonlpd77iCm1ywfKrEPAphYVbrOs=@vger.kernel.org X-Gm-Message-State: AOJu0YxqZulxVIcFm8TAWjzfj8XgsawBsLv4cmz3eCmoxhklMI8i/9y9 us/HR4wXrruiOSqU+ZCWpLDL6yRvRbZ6O86Yb3pfPs2u1mJC8Hul2NlZ7dY6SmqhfQ== X-Gm-Gg: AR+sD106SMZiw+vOsndp/Jm6cZdo7rlgdzJwOMgaWITTpiQFEAdEF1NsR2sVCdjQLke wibFaVKfEldb7C+WQSwGZJ8R+ArA7lug//azdAvWlWT+C8qf7JPzdavpfITCewNykn+dio9tyCg E43nvpstXxwbDq9yS3qJB89qrVhXKmgGdZ13GmCpwzBX9owOsB9FAIV4oyusPhxYVJRDwlsXMpz CP2EwnaqEEB7IAIuvBbDKKYetHQ3i1bQYdnYQ0s8WvGTX71ujW6R+Jf3HmYJrqAvj9kft4ZruZF +rpEK2QPyZV3h+Q9UM4mYq5jxJ0yBrl0KBo4YnzWqk0rCotyoCbN6cXTm8KxyzWpAiafOaf0bj9 C2jAT3516FgHMZ4jZOfxhpM3YLka6TqD9nSi2Mwd6WjoB+TVPYvh82hAtwuvmJWS3VT37mhdM6q mtuckyaVkb1SEcE6/B+PRP/KCsrbFC8R3k2Axye0dIZkVT9jLS/38mFMXAw9Xnck5sra0Y5dWFk CxUWrADy/tCGZQrHR7qhRY2csy53/DW/khOk/ILlg== X-Received: by 2002:a05:6000:2309:b0:47f:9208:9e4 with SMTP id ffacd0b85a97d-47f92080a4cmr4670984f8f.31.1784839379605; Thu, 23 Jul 2026 13:42:59 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.42.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:42:59 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 03/10] md/raid1: serialize non-write-behind writes on CollisionCheck rdevs Date: Thu, 23 Jul 2026 22:41:59 +0200 Message-ID: <20260723204206.76930-4-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A write that skips write-behind (full behind queue, a waiting reader, failed behind-bio allocation) races the in-flight behind write it overlaps: nothing orders the two on the write-mostly member, and if the older behind data lands last the member keeps stale data for sectors already acknowledged as rewritten. Serialize on CollisionCheck, which marks exactly the rdevs owning a serial tree: serialize_policy rdevs plus write-mostly members when write-behind arms serialization -- the case the MD_SERIALIZE_POLICY test misses. remove_serial() under the same condition. This also avoids the old gate's latent NULL deref for rdevs hot-added under serialize_policy (they never get a serial tree). Fixes: 69df9cfc7042 ("raid1: serialize the overlap write") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index afe2ca96ad8c..997f79c05db7 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -564,7 +564,7 @@ static void raid1_end_write_request(struct bio *bio) call_bio_endio(r1_bio); } } - } else if (test_bit(MD_SERIALIZE_POLICY, &rdev->mddev->flags)) + } else if (test_bit(CollisionCheck, &rdev->flags)) remove_serial(rdev, lo, hi); if (r1_bio->bios[mirror] =3D=3D NULL) rdev_dec_pending(rdev, conf->mddev); @@ -1677,7 +1677,11 @@ static bool raid1_write_request(struct mddev *mddev,= struct bio *bio, mbio =3D bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set); =20 - if (test_bit(MD_SERIALIZE_POLICY, &mddev->flags)) + /* + * CollisionCheck marks every rdev with a serial + * tree; order against in-flight write-behind I/O. + */ + if (test_bit(CollisionCheck, &rdev->flags)) wait_for_serialization(rdev, r1_bio); } =20 --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7040747FB01 for ; Thu, 23 Jul 2026 20:43:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839393; cv=none; b=LQ67QbzqWGQmr4scYxTsdSbqjP59xVFy4dvzxtkTKeHyry2zzQk48s9RHzv3BDQc8Job8pa44zIp53yE1a8puDPXd79XWP9/mieB113rgWuDAQKrU3f+agjyWBMuTPY5ROBQEAVRe9l4/W7woUFlBSPB0J+RBwWp0KfuiOdLYNI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839393; c=relaxed/simple; bh=tb87IS5YwxWP0/pdiRmCvUCzVsNHu9wl18O2iixHy6s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=L4JgnvrQqo0IJDEgsS2gK1ULkgqJd8H4gh/c26D6KCNYbQ5MsfFPF4rwChWUlcbL1JBrVEh1QRCHUNJPBIc/Wt+yaxO1KYLgmVsZD8zgXSyjLhBQWSbwh7N2JLXKbu49Z9dCGqyCWxT7G2q6M/C8rWaKdpw1g+tra5za9GxD+jI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=Pt9YwXgU; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="Pt9YwXgU" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-47f7854678bso401224f8f.3 for ; Thu, 23 Jul 2026 13:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839382; x=1785444182; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jvUC0Lia0it+LgLYsOXSPjeMnXEhLq3Hqn52LEDkVn4=; b=Pt9YwXgUB/n16uijEWINzVy9fytMeDArZSye7T8oKIibLMO23Fp79F6KTTDm4420OC K/c+vKpY0QebTMHsOxZFCUtSAFXNKXhWTwpsfnOA+iS6Y4gj4OfU37ft6goBgbz/WAyK khgeZ9l6ktaXAkg9fMXiilzowWyWyZpCPVceqH+Dp7XoKYKfneNV+HSAGQOA2+FgEkSb cKe21z/xgFbykL/dvmvMtEe/e4LwyNBlYR+uts3K0w0X0tkwkScWSfk1A9lpW/UMhScV kPxrYYImjXAs21aTfaVI+BwU3u4x1OTe1kV9QtHsAoWo/OGblM05ozxD7wQRl7bz+fIZ c35g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839382; x=1785444182; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=jvUC0Lia0it+LgLYsOXSPjeMnXEhLq3Hqn52LEDkVn4=; b=ckRossmEDQR33dstMo0/GQTpVxatUQWgvihBdMQFt3G5+jQlV9FUqsbPxvqMaQEb8V kDzmOMIr+S/s96iEhHr7ssBvwSEb929cRMi87zCqYSC+IrTuS8KVjWuMdyXSlw21M7Lf ytYBclRGkb1oacaBbEU8rpR3GFupb8z17vt2PPLlMQv6IDf3gLqv3PsFUssfj32S2jb4 yeEkCvwDraO/rYA8Q/qbd4DS2kDQUtEipzEyibDIqQvEaH0kupixtHDLNxlkkBIAA5k6 1pyIXCKlJTqHwUMrrI0hnQ3ttwqnB+FiellTtn7XldBMaOTNdwRfrn7M83bhohAwDBoy 0nBQ== X-Forwarded-Encrypted: i=1; AHgh+RrRf+ZLLKHWwUP+0rHF31edCQM4M/s5qFyJ8FJjGSZLATp+cBM7/I/pFV58J8zpAFHm8ocbtMhm7GQlAJg=@vger.kernel.org X-Gm-Message-State: AOJu0YwkARBqwF1gbRNpfzZ1WtsvCh67R37JmaxjfFXvCzmVI9xfUhCd GLn0+NcFiOyL6+wLEVwFkH3RuKvnAX3S/cURK5fiJJaQEZxxKNGC+S1L1F78O8yvtw== X-Gm-Gg: AR+sD12jBWenTI2k5CE3bwlTxbaKr8V3g+molGl5j3Z969TgIyKP/9aj1eHJYtuDqyg yLk7vmgo6afDI1Zj41AknTUWormMf7EqDLkk1Merm7A64qdxl9CoVgYFVTxN/x2jo6k8KU71n7c voC/Zr0ujEnbN3hpk71sgGONpslNZyAO/nktUHA6R0dFNuEotNXqz0nAJMINVZ+OSCzUxna0pac JLyE/bnlPq34BWWJhr0FvxM9Q7sBej2exTwI657zUIo1XQEek3O2nAiy4jyxFWs9LdJSWsj+A2a g8YncfhXIrZA/k3hMDABEnupnBiKTfyA2csOWisU0d0K/cajyKNpMXX095/fRNMtUiT7fPwkTac HXzWuXtllLW3Man6dm4k7rVIQPRCpXUQA/OnH7e0CmbWQrYhEIijGrATs1aiRtEoMySLvIvwXRd 2IgUR9OTIUVp6D8oAiYuh7evulwWrK8Gbp1lcEWDgczpb8Hz3TCPf0Y+W7lQe7sCVB6iSTYLZLr ZDaSHKFzf5Gdg+s7Tk1p94x7abGLUMJY58AYqWssf1fzFWOWMrz X-Received: by 2002:a05:6000:4301:b0:47f:943a:a7f4 with SMTP id ffacd0b85a97d-47f943aa945mr2275162f8f.61.1784839381805; Thu, 23 Jul 2026 13:43:01 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.42.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:01 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 04/10] md/raid1: don't use write-behind for P2PDMA bios Date: Thu, 23 Jul 2026 22:42:00 +0200 Message-ID: <20260723204206.76930-5-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" alloc_behind_master_bio() copies the bio's data with bio_copy_data(), a CPU copy. P2PDMA pages are peer device (BAR) memory; generic code must not assume CPU load/store access to them is safe or fast on every architecture, and bouncing peer memory through the CPU defeats the point of a peer-to-peer transfer. Skip write-behind for P2PDMA bios: they are written directly to all members, including write-mostly ones. Ordering against write-behind I/O in flight to overlapping sectors is preserved: the non-behind clone path serializes on CollisionCheck rdevs (see the preceding fix), which covers these bios like any other write that bypasses write-behind. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/raid1.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 997f79c05db7..7d769b6460ca 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1523,6 +1523,7 @@ static bool raid1_write_request(struct mddev *mddev, = struct bio *bio, bool write_behind =3D false; bool nowait =3D bio->bi_opf & REQ_NOWAIT; bool is_discard =3D op_is_discard(bio->bi_opf); + bool is_p2pdma =3D md_bio_is_p2pdma(bio); sector_t sector =3D bio->bi_iter.bi_sector; =20 if (mddev_is_clustered(mddev) && @@ -1575,9 +1576,12 @@ static bool raid1_write_request(struct mddev *mddev,= struct bio *bio, /* * The write-behind io is only attempted on drives marked as * write-mostly, which means we could allocate write behind - * bio later. + * bio later. P2PDMA bios are excluded: write-behind copies + * the data with bio_copy_data(), a CPU copy that cannot be + * assumed safe or fast on P2PDMA (device BAR) pages. */ - if (!is_discard && rdev && test_bit(WriteMostly, &rdev->flags)) + if (!is_discard && !is_p2pdma && rdev && + test_bit(WriteMostly, &rdev->flags)) write_behind =3D true; =20 r1_bio->bios[i] =3D NULL; --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD04F47FB14 for ; Thu, 23 Jul 2026 20:43:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839395; cv=none; b=c6QA+7voSyvL07bUq4+g0g3rZ67toB5erjd4inE2ibRMYh8a/Z6UNF4mJT73WTTYtUexJpRgQvay4ooCwGyxJn5ezDUsVdTstOVH0sgsZ2idg8Z5yBRojnMjyheWFnSd393X1vzt0MP40hZwLGpnRYBApsj6eqLGuslLHT4aiIw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839395; c=relaxed/simple; bh=B1aIegApdgScj51bEIGVKX+FENfztmE4q/fjJ31TwzI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=e/hnUV4mrRHVAcCA0ro06eIM1H3ONeQ2kzYIqpdzls26Lg1EudR/oCQXAzV8FKftJpZiEuiBBzodtTgmQY5/oBsV50fB5yR3Kaw181TmSzRz8YT308ivSmfQeEV6hIHvxr73kZpdkukOvGtkSuVxu1WUKUFJcQPkhPQggBLhlrI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=Vh+j2fYI; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="Vh+j2fYI" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-47f7027ca11so691072f8f.3 for ; Thu, 23 Jul 2026 13:43:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839384; x=1785444184; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vP39KbYqkNNRmuPyaoPhdZlQ+sTmuJQGW6DJY4lSzaU=; b=Vh+j2fYIAAlwbHOWPa/hK7YJWC5+OwIKFTcBSfP8hHCijIILORAWmxr+3P6EqtVgzQ oACyPP5VymPVgFWCa/QsQudx1SiPEkxSbl4CZCkvael5lKoYxr44301de+BaY1DHKx1Y 6IPDD2uMr7JW+vQai4Y1xq+/OsXG253cfNngZ5segD6cZdlkvG7azAk6yykodkYdvWPK RtLLvBOhOAh2gPOF3sYAwcELt5tsgEx5JZqh1Vr2AzLiV8UubU30Mj1mWCQYdLN27sgK UmUfC/LqYccptvEscUmHWmKlPg5h8kr5V0GVmAssdlPsheM+DsAHMp5wdTObvg9HvfY9 BMHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839384; x=1785444184; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vP39KbYqkNNRmuPyaoPhdZlQ+sTmuJQGW6DJY4lSzaU=; b=aTEbj7Dw0e2H5ltETpiAmmITJdANRSN5m1cmCz7LJulTnniPtj7/x/uGOeAyGTLCEt e1CCrBkGh8jsOmxlwQuTrE3GdbZRjB4vLC2OyGDa+iRqNw9GE7DJcCITsDgxpsT2VuYb tnsHdvGn+smNLUKbNLbprCibTMnN0FTLYcx6opMbZ+u9yxOsiqVra/tkWcTDpQAd/YJS 06ZwET6Uhty4udx/QUmuvYCAlbzAXPPGQ9VASYjBDBXHEc6rtJevXFkErlQJEaJcgMSp ttE8K03/JtRZw1pM8UGb4BvamAxwvH00SBx9YkzMEjKqKrPjMMUJslTOLM1RheBFk/1A cibQ== X-Forwarded-Encrypted: i=1; AHgh+RqfJy4A/w3JS+AOsSj1Qn/AJ3zTM5W8q7SQhPT8Q/u/EfbzevLPvzhiLy2oeFpcCpIEJtuDBnv7R8kR8I8=@vger.kernel.org X-Gm-Message-State: AOJu0YxPIkvTOH8s9ZT6b0rnItwiqSffmuhvDgw21GKP78975xh4Wo12 FPsGa5d7lwlWuvxXSASVr08hPi8vFNbkiCILSaXeGNBYjGHCADZvK8GYJizVVYgiyQ== X-Gm-Gg: AR+sD11D8FioWqbD8YIcJ+38hVPfnF+9GjSmzCLFFSabevDDZkFwJVy5mPc8g77Zr13 ud5s2CJvIMGhCY6xmjvwRZXvCQsung7Uq+tfFWIj1KZtskY2wPKHszac40F9s8GL7ALRLgqddO/ C2GH3/LIdNA4qKfwPRqQN9zbtZ0IobNNmvtpFwAeABOhOO4Q6FLYzWJgeguKxzbK5ng9PCAIaPl gR1iqarGJouP6oOg4GurI7gpEtjUlMFtC3BnPQH26QM4sVbKjYXWN0l1nD0X1ILkc5TrAs1CmA4 9CGwS9zi0YwAw5G+VArbdnQ7k6P1MOLKegBVL+5S/XGm83Ysv8V8zrcmQgdZOuMhBs/kxp22YkX sjuB7befx8QEDEl+Z9+aRnS10PNhSrYW8ekxSKhs4jhvivAGPQeER/mq08NjpX6KVcZPcPDsyhx /HfDMH7JQpCFoEYJgXcOJkkqx8YvTYGuw34UWy4k4RuXSeLiy2z+AloUNSU+ijuIbLHnv7WwDDm vhKJ/uR13ieUAMtw4Z6R/HeNkrswRr/n+jA39E24Q== X-Received: by 2002:a05:6000:1ac5:b0:47f:28e2:d4c4 with SMTP id ffacd0b85a97d-47f8da6e5bcmr6028034f8f.2.1784839384106; Thu, 23 Jul 2026 13:43:04 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:03 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 05/10] md/raid1,raid10: factor out raid1_write_error() helper Date: Thu, 23 Jul 2026 22:42:01 +0200 Message-ID: <20260723204206.76930-6-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe Factor out a common raid1_write_error() helper from raid1.c and raid10.c. This will help reduce code duplication in subsequent patches. Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan --- drivers/md/raid1-10.c | 13 +++++++++++++ drivers/md/raid1.c | 13 ++----------- drivers/md/raid10.c | 10 +--------- 3 files changed, 16 insertions(+), 20 deletions(-) diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c index 56a56a4da4f8..cee21452e066 100644 --- a/drivers/md/raid1-10.c +++ b/drivers/md/raid1-10.c @@ -303,3 +303,16 @@ static inline bool raid1_should_handle_error(struct bi= o *bio) return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) && bio->bi_status !=3D BLK_STS_INVAL; } + +static inline void raid1_write_error(struct mddev *mddev, struct md_rdev *= rdev, + struct bio *bio, bool allow_failfast) +{ + set_bit(WriteErrorSeen, &rdev->flags); + + if (!test_and_set_bit(WantReplacement, &rdev->flags)) + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); + + if (allow_failfast && test_bit(FailFast, &rdev->flags) && + (bio->bi_opf & MD_FAILFAST)) + md_error(mddev, rdev); +} diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 7d769b6460ca..f07edeac6e5a 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -483,17 +483,8 @@ static void raid1_end_write_request(struct bio *bio) * 'one mirror IO has finished' event handler: */ if (bio->bi_status && !ignore_error) { - set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) - set_bit(MD_RECOVERY_NEEDED, & - conf->mddev->recovery); - - if (test_bit(FailFast, &rdev->flags) && - (bio->bi_opf & MD_FAILFAST) && - /* We never try FailFast to WriteMostly devices */ - !test_bit(WriteMostly, &rdev->flags)) { - md_error(r1_bio->mddev, rdev); - } + raid1_write_error(conf->mddev, rdev, bio, + !test_bit(WriteMostly, &rdev->flags)); =20 /* * When the device is faulty, it is not necessary to diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 0a3cfdd3f5df..5b63175fb382 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -482,16 +482,8 @@ static void raid10_end_write_request(struct bio *bio) */ md_error(rdev->mddev, rdev); else { - set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) - set_bit(MD_RECOVERY_NEEDED, - &rdev->mddev->recovery); - dec_rdev =3D 0; - if (test_bit(FailFast, &rdev->flags) && - (bio->bi_opf & MD_FAILFAST)) { - md_error(rdev->mddev, rdev); - } + raid1_write_error(rdev->mddev, rdev, bio, true); =20 /* * When the device is faulty, it is not necessary to --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50EEC3F3279 for ; Thu, 23 Jul 2026 20:43:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839396; cv=none; b=cFkROB9OVLPHATrJj8iLmPVvCWpFqQb5LTbl7r8iMldnJAWLUDBsTdTfGbUL7RZ0RN0HjYFRbJlKfOBDQmBXvCQ/S6lwNrmCrlSi7GXNHNBm8QDsEHWYMTE2K4Ug6OMlwJWHYPPNnfK/rLMGK8aGjwqrNpAAQs688afvbeJyQuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839396; c=relaxed/simple; bh=6JUXonkvhqCluPq91YNslI6Xc6PHhaJJ82TMx9QRo24=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GpSA8noSSOzMoyXp7785V+qzuN3z6pafSfIEIMmc4m6VCATvIpY+Qk4AD8EwzvuFCoi4kDPD7V9BUDfCE2+Y8OcV+ltplyDeAhIkbFcf/c6EOu3jbY7ykz4ROXEhDO2jZruc1xR3qysq1rewIaJNs3z8VssCYENIFM1iufAoyOU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=b9TCH004; arc=none smtp.client-ip=209.85.221.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="b9TCH004" Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-47f7854678bso401243f8f.3 for ; Thu, 23 Jul 2026 13:43:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839387; x=1785444187; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=InzNp2nHOOp7QQLn15Ay7fcFkfBNFGx8C4/KxQQFD2c=; b=b9TCH004kM7Ijzlq2L+uYapxZbsJtgiPIoccs97iLOok/j4G0z9BoSi7OGTZzfNraM 9nBbybkLeR3kaqz8qibH9iQR2/+3QMIg8k9UtqWVXdcUsUBX4RdymARurgQLc9PBz+tX cU1oMcOcw42w0OxGq4UZ131+iaLE2py0jzPsTM1peGRNz2HSGRqdZT2XUA1zms+v/aFP RSKb39m9qHn/9kGElrHMfdLkSFCt2vA+ObV319T3/Q3iv30jBdhCVHMR7mKE0DXcTuWe tR1+ya8cQ5UblTusXMB+XkhJKb/d+KQIXTWrnGw3X47bbkfgMA0AmX7SlpGUHEbpPpM4 +F7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839387; x=1785444187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=InzNp2nHOOp7QQLn15Ay7fcFkfBNFGx8C4/KxQQFD2c=; b=R4O9psHLOY2cVa5hLYXGgiPxqnEI/ltQmirGkdwDuTX13yOQQl4BEbSAt/4OtZq/ST 2++nC81j8JNv6c7tffKRqltRerAjR8C4T775FFMjaVp27G3F2rwBceGHNMDtyMDO385A 22JhGSSbQUwQTb6iRuxPTZbVcARq9CBCpmZqLL8+OhFBCP9htUchQn/KlUCZRv+MCiXo GkJE68WgaDEgRrVs51KrEWLBG0rqXuFVrGt1qV1GYDSmRllV+DkOmKPsoUXL3Q95qSi5 9cCfGMTfO3XfyntMtHAa7JqJHRDK8XuQUM9qd8fE9n6iOGWVUazipis07MxMVsYlTdPE sdeg== X-Forwarded-Encrypted: i=1; AHgh+RqjhaZsXnx0FA1saZO+o4PC8tO+RZOnV4qopwnvD817gctVpGIG59oZZERZIAlY9AY36TMt/djiXa0nRqQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyMxw7BYqSZnar80ZZw4HALI9kgLRgWizGIZjkgaebZK7+3Ebqz oncS4ajjDjA6luxaTgkzYe7RJBUOGLSkxkQBrfoA2k/PLQ9PaYlTptLU7+YSGt3eNA== X-Gm-Gg: AR+sD13uSGrgSLn4iNakP7hitTp6DwLK5vXMEKfeqs5s3Hg/7BraXnD3ME5V5BEXG0Z sCMNrBYYSBI7rxFt6Q9THyrzFp12xwX1KnuII/lYSfA4WUNafGoJ1B1FZS3oMoxYcxOJtL8XNu0 3554sE1lGWs99YFtmYSe1iChhhWbaka1eVMFwYq6IGgprjv9UFqe1gocB9eoxzMXugQwidCSeqI ubuT7S3Tlb0H6KZ9XXlkTLOXrdYpW+H/sMAo14/7QebwQCaJbkKyG9TtvA7Xkrm5yRl/rCkGKmA tQ7VUtQTEic/NfPU5onqZlAvOXImoVEDSGyaPo5hKEJKImjN8wjSBYGeMptEeWt/+sChdKH3r7Y MbYbxicF6ZvYlMc0VkM4sKYeR6tYt4fj/LPDgH5yFqVWPSE2AuobJGSEeOMibthWtEg4q6VzX+n Y8tHrLLrvyfVfkku/TqfgMyH+XK7GQjIW6o1ob+88ObsmS39SXcIvsg/xC/YJZ4omzrVbQAMBdh xAIh/5GoFq59dXAd6SoZjQZJUfnx7Pk5mSM0HsAGg== X-Received: by 2002:a05:6000:611:b0:47f:25b1:e7b6 with SMTP id ffacd0b85a97d-47f8dc7a211mr5521629f8f.30.1784839386364; Thu, 23 Jul 2026 13:43:06 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:05 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 06/10] md/raid1,raid10: keep REQ_NOMERGE on narrow_write_error() retry clones Date: Thu, 23 Jul 2026 22:42:02 +0200 Message-ID: <20260723204206.76930-7-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" narrow_write_error() re-issues a failed write in badblock-granularity chunks, cloning from the master bio and resetting bi_opf to a bare REQ_OP_WRITE. For a P2PDMA bio that reset drops REQ_NOMERGE, which is the only request-level protection against the member queue merging P2PDMA segments across pgmaps or with host memory (see the preceding md_submit_bio() fix): the retry path would quietly reopen the hole the submission path closes. Restore the flag on P2PDMA retry clones. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/raid1.c | 3 +++ drivers/md/raid10.c | 3 +++ 2 files changed, 6 insertions(+) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index f07edeac6e5a..19c4dec450de 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2561,6 +2561,9 @@ static void narrow_write_error(struct r1bio *r1_bio, = int i) } =20 wbio->bi_opf =3D REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |=3D REQ_NOMERGE; wbio->bi_iter.bi_sector =3D r1_bio->sector; wbio->bi_iter.bi_size =3D r1_bio->sectors << 9; =20 diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 5b63175fb382..428bddddfe71 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2823,6 +2823,9 @@ static void narrow_write_error(struct r10bio *r10_bio= , int i) wbio->bi_iter.bi_sector =3D wsector + choose_data_offset(r10_bio, rdev); wbio->bi_opf =3D REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |=3D REQ_NOMERGE; =20 if (submit_bio_wait(wbio) && !rdev_set_badblocks(rdev, wsector, sectors, 0)) { --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9004374756 for ; Thu, 23 Jul 2026 20:43:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839396; cv=none; b=k/doyxN3Ruk54tlpsQuJHddth9as1zdSIa9yB/x8wsYLMzxA763gfyb3+Y9I/7nUMwno0uUTrJZ9LeDXRR5zYuHhQxowBJj+SIrbzmEaXS5FOq/0vrXwtQ15hJfqHeOn4RnbgWLVi2VGDNppuzc7heGq6Ag7BXmSKUBtG1hzAqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839396; c=relaxed/simple; bh=1E+TY7i/EaKVM6Ai4S4WruKLMZDKABVE9DkE2+RPAms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CoZ+wRYYiFl2KkGNAsH8fmNpIE64n6VOFqyAxcBUPkwN0zTlZiCw9UAARnSBy9P0g60KE06jDJvXCE+Ph1Q16+xND0Sl6Ibvzw/+Eplf/HViN3nXpeGNnZ+k8STMl9OZjuciiolrlv65Y0pLIlqq4P2BYiy6w/I5PEbBnY2ZW3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=f7HrgNst; arc=none smtp.client-ip=209.85.221.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="f7HrgNst" Received: by mail-wr1-f48.google.com with SMTP id ffacd0b85a97d-47ddf7b09e5so1146237f8f.1 for ; Thu, 23 Jul 2026 13:43:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839389; x=1785444189; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2ii4b8aBCeGQxAV9y7kovFsk89PBwma3JtjswJc/vPQ=; b=f7HrgNst7G5OrUHnt81ledYHC7cH9MuwxYocukraxqKFrRYLJ3cE4qu7RATZtJaqH6 hxzHUGAiWO56gs+RxSo2gcYhx3wcf803GyYcEMmMJd40B9sTTnCpBrlSmLZaM/kCyfmY o/qmKb9wP4IVNumAOV+s/ZrXrSBYjv2UMEH2YXHiLa6sgCJLzOWDsSn95fNWGzdM371J RmjKUeo+Rrd2eLNy+5MAN9//S1yD7feDAAxXprIteO4WNISR34g/hf3lhQSKMpc/q6Fz CCsLahqpLRdqSEeEZc88+tuyBJxQGYZIZOBBvelUnlrHUS9VxzRQy98T5B3bZDaupxcH bCLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839389; x=1785444189; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=2ii4b8aBCeGQxAV9y7kovFsk89PBwma3JtjswJc/vPQ=; b=W8WtovcWWutFHm258g3SOcr3K5GA5CiY8dI86olmEx5r1x/BKsDVohqidnCIolfGDk QU+TmrmPineI8FHx3q0owViViYftxD/psF4+C8bxFvQQrGhN7V7LtU8FyTIcHDKIzP5w viypwkJtYoLFna3OY3FDTs4GHllu6+icWEBG5LUYRPvX8Ejc/dwUz/YKDSDibnLZreHT PmN9WAOzr2Cw3NKPC9+ZYlt/fStt6uFOFvrTlzKG+lPgQhUf9DJ9Ku0gAp3LaHxewZ9w xlmYO/Vb9TMMnsnL5WT/yJ9cIwwEl4ug/Zt0ySjsI7AZ+rJW3r//7yOvRSnAf2aB3zsd 6GaA== X-Forwarded-Encrypted: i=1; AHgh+Rqunv3TTvib/rNBerA3eWVc+7CMLRd5P/xOcisa8p6I/b4vChJV7xQcdyWnmspFlXSE3So+P0VJokGmj90=@vger.kernel.org X-Gm-Message-State: AOJu0YyNgAy3NQ7PRpW+RsA6Z50P9LEvf2PwcMXzIc2/ZFOIWqI6qhmS TU2xqJZ0GmxOlvnXDmVWJkP0GVYrrxVDxj5u11EapYstQQxWYe8T17y0J4IC34z3OQ== X-Gm-Gg: AR+sD13eZnjVl5KZHzIJgK1BDsISlnLTR+LNbcR5r58v9Of8RWmjFSNT0xgcE2pSUXv qpjTtrNVgCtOuCzMeeHUgC1zJLi+hkuypcLje3PsB01W5ngnNrEiwOa0e2+uoc9BDcZ/EZlq3pm BMBsaIDpR81YuV+o3VaOReuKtyI39ACKYUrSA6XfCE2YMDN6I5c+lsD1dxFVh+kD66O1IGQ0bxZ EJXxzsf6MqOuWmO1N48khwWZwnW5oYjmxoiIhcysfN0iW6cOot1xDHa1TAgB6O+h05te4gI5ig3 khn2wS51e432pKEqexvPDTh6sj7IDJiVvGnP7VyVvqCvnWisr+/tlILG48OyG2DRgyawvlfn0an e/BqpR9lIdJ2Dvqw23PGm/kEsbfSS12Uuu6h6a793mAO5v3LBK8OdStoIb45CjAa68bf+j80wJo DH6MZgY60gAMB335P75kEvPJitNxGuJCnqXoIOLg66JtYFpndUbd2KVK7vR2aC35ev4aV2qNzNk +IqTv1Yp7CxkRkQcWiB7zfIBCqXyP9KNlnrsGabvQ== X-Received: by 2002:a05:6000:4702:b0:475:f100:35f4 with SMTP id ffacd0b85a97d-47f8dca9a60mr5832815f8f.49.1784839388604; Thu, 23 Jul 2026 13:43:08 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:08 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 07/10] md/raid1,raid10: skip futile retries on P2PDMA mapping failures Date: Thu, 23 Jul 2026 22:42:03 +0200 Message-ID: <20260723204206.76930-8-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" BLK_STS_P2PDMA reflects the peer/member pairing, not member health: retrying the same pages on the same member cannot succeed. Keep WriteErrorSeen so the range stays fenced, but skip WantReplacement and FailFast eviction, as commit f7b24c7b41f2 ("md/raid1,raid10: don't fail devices for invalid IO errors") did for BLK_STS_INVAL. The failure is whole-range and deterministic, so narrow_write_error()'s block-by-block submit_bio_wait() retries are pointless. Record the whole range as a bad block in the completion handler instead. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-fable-5 Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/raid1-10.c | 3 +++ drivers/md/raid1.c | 11 ++++++++--- drivers/md/raid10.c | 7 ++++++- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c index cee21452e066..89daf8558251 100644 --- a/drivers/md/raid1-10.c +++ b/drivers/md/raid1-10.c @@ -309,6 +309,9 @@ static inline void raid1_write_error(struct mddev *mdde= v, struct md_rdev *rdev, { set_bit(WriteErrorSeen, &rdev->flags); =20 + if (bio->bi_status =3D=3D BLK_STS_P2PDMA) + return; + if (!test_and_set_bit(WantReplacement, &rdev->flags)) set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); =20 diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 19c4dec450de..83ebf40237d5 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2624,10 +2624,15 @@ static void handle_write_finished(struct r1conf *co= nf, struct r1bio *r1_bio) * narrow down and record precise write * errors. */ + struct md_rdev *rdev =3D conf->mirrors[m].rdev; + fail =3D true; - narrow_write_error(r1_bio, m); - rdev_dec_pending(conf->mirrors[m].rdev, - conf->mddev); + if (r1_bio->bios[m]->bi_status =3D=3D BLK_STS_P2PDMA) + rdev_set_badblocks(rdev, r1_bio->sector, + r1_bio->sectors, 0); + else + narrow_write_error(r1_bio, m); + rdev_dec_pending(rdev, conf->mddev); } if (fail) { spin_lock_irq(&conf->device_lock); diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 428bddddfe71..77c7d8a59916 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2940,7 +2940,12 @@ static void handle_write_completed(struct r10conf *c= onf, struct r10bio *r10_bio) rdev_dec_pending(rdev, conf->mddev); } else if (bio !=3D NULL && bio->bi_status) { fail =3D true; - narrow_write_error(r10_bio, m); + if (bio->bi_status =3D=3D BLK_STS_P2PDMA) + rdev_set_badblocks(rdev, + r10_bio->devs[m].addr, + r10_bio->sectors, 0); + else + narrow_write_error(r10_bio, m); rdev_dec_pending(rdev, conf->mddev); } bio =3D r10_bio->devs[m].repl_bio; --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DEA04823D6 for ; Thu, 23 Jul 2026 20:43:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839400; cv=none; b=A4F9fVoNMgusvDAs68Jlnw81hjdHfMXhswAGD8PwWfACNlPADJEEUJ0+PwXbZqF/NUiG9bsysdr2WNqL7NEMrGSlVAVY/B+feYWaX6yVMxKV/A6d9h0RFpFyu9oisEcbG3KNi/+w9/LzKSSEKbqk1I/0OsedopaB4qBO1no31ow= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839400; c=relaxed/simple; bh=5VbqgoxnpZ25+DAS/Bu8wOeDLmqQ+TGKyYgc7R7RYZ8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KE+pKe0WrNZifTL2JLqn9QegAQKWwXnw/GYUeoztuPWeGD0dZ1tB41UByV+VEIZCpwuPG+sDpwEQjvvc48ApNRnOpbTiknGUe0wq/z7D8fMOS35re96gjj4wQEAXFAr75YCUlurkq5CFSHWhZkfnHfw/DMvMCP6JTwPdXHzaPB8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=pWUtozAa; arc=none smtp.client-ip=209.85.128.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="pWUtozAa" Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-4953de5be0aso7501035e9.0 for ; Thu, 23 Jul 2026 13:43:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839391; x=1785444191; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LVDrwHOShpJST2v1XpPaKOLt9RN1zMUNY9kPyJb4mps=; b=pWUtozAaJSTecCwJDXbOMORoINDMoqgo9Qoopb+fRZGbZ+1dCa8zKImQRbnzXgwtS0 4bxSTGZ+KB3kbLh8dqXEI6NqPkrJ7dpV3qr0Uhy9JLv6EOyCFQ0M7hmPKIX5uGzEwLJu WAjRRQS3RtP+TuPh9cC6966bVDVMFSZoQpJk/xLxg7RN7MSzwL9Xz3zp4XNk5qxbTPVa Ij7eQdCt2J9dRmBSjFvR0XePkWBYixyUSYzKKC3gd3hdtkBHQgGXcsohxJs9UXrqalVi bZytFv3qu5xQpBMWwy/WkbvtM7H6Yoe0LOjSNcckFsOaOnNOdz7TS0TrRWgJa7+N9MT9 8NkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839391; x=1785444191; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LVDrwHOShpJST2v1XpPaKOLt9RN1zMUNY9kPyJb4mps=; b=MejpkDpUXahv0vg9kmvqCH7G70wdVEnD4kaxNksdG4Xm271u+izslvX0kIy0YHdDg0 pkwCLLJhYQoTQnlMJnMH/Aerw5UbDqKWmXSJQsDW9h8Y+hM109gQQnRLNmIH4wcknEXT tbHHMV9xMVsBRkzF5brpue1kmLSpN4omFIQrFR4guk7FuRpQwCIiEr41U4pCM7EzU7pA GOvUH8TmwbTSquCZsovHP+lxX+iSLWWGz/tL/48rD1A9E2qTEpUnvvgskNIOUL4XWYzH TCUiAUrVfQCk3EO489twddyGi8Mp6Lullcson4DTZY+NW28DVnkrvAkZ2feJ3PQFvvM5 FVcw== X-Forwarded-Encrypted: i=1; AHgh+RqqIoHxcsA0jbiuAfQAHEFALfLjBwYWz74JaWb10emW13QmxVR3rbGA5DdAz6d7dOIuzMGNLA379dia45E=@vger.kernel.org X-Gm-Message-State: AOJu0YxvPaM4wwgYIpONBPNFcwe/Ziwr4ih62M1NVDDm6Bvd2/k2vynD C1RZ6frZtQCzzbdn/PlUWLl0f8a/DyS5SsP0wh3telJu7eLPjBk3cw9FDF2GmViI4g== X-Gm-Gg: AR+sD1234OytxjS80CCsTAewMezST/hAtFK847ezRBJ0X3ake/8SgerrU2tTNgK5qqJ d86ri0bgvM+v6PrLreIiFN0LxlRf2bc/w8b8UST0FeVSkJ6Nq7q01I+4omUqp6a7ElgLLlmUD9g +rAT5RbtNPFa4l6XasrcpDQtfGMNETLbOxVeDxDSyvYEGntIssSW8/rcF4LPVP5fMiLDrQH7uAd daEiZB9fF/HDnFV+dQiqybytFcXL5a7AgvTR0g1S8oWlLosrIvtZj720CuoRBSMTqMRDoRkAX3t mc0FQt/BkkZZ3qxj3MTtB8lVOPVXrlJO0BiSu3TlJAZypfetJn690MEwd0GHDaAIQOx/RAgu1TP A3RbMaj5iyQj1iXcJdQ0YYyqYosivavrN2500jrutof1FVClgpC/p2daJ4VWOYdYYtgZf4rwwy6 +ASGLxobkTgv1bEK5wjKtIYynOsPVO3Wx4rLPyrw5xyQrC/qmY2GTKMm6U2qAhKgp/FqdRtPO4n MRSX4084ctFMb9rD/wj45Kyro2QSGScewuhbJloKw== X-Received: by 2002:a05:600c:6a07:b0:493:a613:56b2 with SMTP id 5b1f17b1804b1-49573c8d700mr36854855e9.8.1784839390786; Thu, 23 Jul 2026 13:43:10 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:10 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 08/10] md/raid1,raid10: set IO_BLOCKED in case of BLK_STS_P2PDMA Date: Thu, 23 Jul 2026 22:42:04 +0200 Message-ID: <20260723204206.76930-9-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A read that fails with BLK_STS_P2PDMA cannot succeed against that member, so mark the leg IO_BLOCKED and let the retry redirect elsewhere. Skip the read-error machinery: there is nothing on the medium to fix, fix_read_error()'s probe reads into host pages and would "succeed", and charging the read-error budget would evict a healthy member under a P2P read workload. FailFast eviction is skipped for the same reason -- the request never reached the wire. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-fable-5 Co-developed-by: Logan Gunthorpe Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 4 +++- drivers/md/raid10.c | 7 ++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 83ebf40237d5..16a47568679f 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2657,6 +2657,8 @@ static void handle_read_error(struct r1conf *conf, st= ruct r1bio *r1_bio) { struct md_rdev *rdev =3D conf->mirrors[r1_bio->read_disk].rdev; struct bio *bio =3D r1_bio->bios[r1_bio->read_disk]; + /* evaluate before the bio_put() below */ + blk_status_t status =3D bio->bi_status; struct mddev *mddev =3D conf->mddev; sector_t sector; =20 @@ -2674,7 +2676,7 @@ static void handle_read_error(struct r1conf *conf, st= ruct r1bio *r1_bio) * read error. This is all done synchronously while the array is * frozen. */ - if (mddev->ro) { + if (mddev->ro || status =3D=3D BLK_STS_P2PDMA) { r1_bio->bios[r1_bio->read_disk] =3D IO_BLOCKED; } else if (test_bit(FailFast, &rdev->flags)) { md_error(mddev, rdev); diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 77c7d8a59916..462f622556ed 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2847,7 +2847,9 @@ static void narrow_write_error(struct r10bio *r10_bio= , int i) static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) { int slot =3D r10_bio->read_slot; - struct bio *bio; + struct bio *bio =3D r10_bio->devs[slot].bio; + /* evaluate before the bio_put() below */ + blk_status_t status =3D bio->bi_status; struct r10conf *conf =3D mddev->private; struct md_rdev *rdev =3D r10_bio->devs[slot].rdev; =20 @@ -2859,11 +2861,10 @@ static void handle_read_error(struct mddev *mddev, = struct r10bio *r10_bio) * This is all done synchronously while the array is * frozen. */ - bio =3D r10_bio->devs[slot].bio; bio_put(bio); r10_bio->devs[slot].bio =3D NULL; =20 - if (mddev->ro) + if (mddev->ro || status =3D=3D BLK_STS_P2PDMA) r10_bio->devs[slot].bio =3D IO_BLOCKED; else if (!test_bit(FailFast, &rdev->flags)) { freeze_array(conf, 1); --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f53.google.com (mail-wr1-f53.google.com [209.85.221.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 305793F3277 for ; Thu, 23 Jul 2026 20:43:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839407; cv=none; b=g7lzzZIvTtkcFzVEDmH8NJWoWRdgSjf0f5KZqjCMANmYXeQUy1E4Xi5iN/8rijhw/CpMuJ5rFM+rZQn3yeAiDr0iMDRTbSHj8HFuwcQX9oN5cW25CWRgIVFFr3wMTcD/JrbDfBRL0wxvcAhgBIlZru+Ue+uJzq0k3GpJCcwCmIs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839407; c=relaxed/simple; bh=sQcSSX/jufOA+fic/HSdeMNwwQDiSrn4ZEB0dsr0Z0g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MvrzrIZplmNh8iYJANZE21A7ekoAUNUNLx8ikaAckrnMlUVZSyajHDa7o3xhwoFV3OIUSjxr+0KATTbkNTjBUGx9I9q7eTnDvjjvIe312m5mP3t2xrzGv0aKLObucWReLk3gSvoHkjvgGG2AlP+ovpQkUKjtcCSG5OVCLnOBN94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=A0PhEHmn; arc=none smtp.client-ip=209.85.221.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="A0PhEHmn" Received: by mail-wr1-f53.google.com with SMTP id ffacd0b85a97d-4799b3f7c83so785188f8f.2 for ; Thu, 23 Jul 2026 13:43:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839393; x=1785444193; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=wlFyTwpdvsIeETeN6SJHO+G4KnBBWZplfaizFoTi1JY=; b=A0PhEHmnTJ9+QVVolXbcvu7kUytNNzkuhnHI49ebvWjAT6aQC68XJXaZrWn7EYoFv5 UIO5jsHccWyUmv7Nv/1ttGqpO6WGslxxl0LC7v3d2+kqu8O6A4jeX57CmWsLOJU5+l07 JUMPQJJdBfPptyNw5ZmEZpCs6++PzJKMUNnrMRAA6hdTyIF/T/F4TH/OpLZvYP8EtPdH axSjIMMIRo/9Qv11/Sq3CpM7Sp/JMxKM93gVOS9NxH9ur9U9e2WTf5uKUKO+kc1gb6gb DoXu81Q0TiGqj9JO1cZDqx/lBMgNGlBkDNSOWle9RU93/xX2Cd5Xm0D106GZydDPVLmQ bwfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839393; x=1785444193; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=wlFyTwpdvsIeETeN6SJHO+G4KnBBWZplfaizFoTi1JY=; b=dn2lcUppGBDmURDBPVNXAhew/9sXOJLob32Obinc8p3n0XfS7roufUNsTP7XnYGYFW SgnrkNKyYtGFDTrtwT/8P9/bVk6r00Hijy+rMg9XJ79G8W2U1HpoVOyIxaR7G1CwkMUB IruUpt35sjkoCvyGgHjrwRv3jjOEtzG778utc+yh7SFccDC/Fe0UOOCDj0A3cj8BO0Sn 4bE4xLNx//y3c8StTV3MoHvbUIYTxfmll1Wj53AOAyGMeSbe9Z14Dnp1vNJiU+sE3uBZ kNVkqhulf8iD1D4Y2QO4Sc78CG8kq+zkpKWfDhEUjR8nasCA5ftz3C6nEHBlLwme9Drg 5HGA== X-Forwarded-Encrypted: i=1; AHgh+RrbYcTI1q+UM9Ok0lNQVjA6m6bBfA0ehM2aIZHMqTDY62QKz8MiuEC0HdexCoNEpOPlCKA4Pm3MGnoVF7k=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+lkaAeQft8kHV8cW0fP9gyW8/wiAkoxjZwRELUMoDmVd9OZJC ejjxQDACmByJ7d+qfPejTKgLzPC0c8MmxyYCvyOuMjJEywlEJMNH3vDhuOmDEdHpPg== X-Gm-Gg: AR+sD10SLY/huL0kGIHtTK7kSwySUMQOetFxXwaAv/RYImBHkkNzX771wt7szLUyCHS hQAcFbS+DPlX1HO+gKxVA2qDB302uOhkn0uQVwX3jihH/I+R8GUZaVls+4YFNIHxJ/Fgm7pHQDi redS2v05pXdYBPg5XFo1Ebz04u9D+5/wWYGOm2fyxLLN/rNCFyXSoLVXlA/zgXiryH9G2aXPH9a ZqiNbeshYULR50uUeAlrEgnOOKgdkP/ueqGLcJlhQqM0l2Z/rOSRYrDDSSykcKSxuq/tSH5XX6Z NRwHUsEWCs/9dmlwmT3a9n88q45NvK32tmlPp7IeXeJET3HXUIG1PUKThswJS896dgmThRzVW8X alh2SFPm4WPOMRfRsiIj+aDL9nJ1zjOQP59l4rrDcGOUEKjMFZ53Y2j+4U0KTEJ3nwQ2jjivT0E aMmdko7MGivHWMENs/9duoBGZtHnPW+M1HGWQXCkpWzazsu9x4yFNHlZQMiTALTEpuiXSrOPO5L WrnwSoBEhsXCUwlhtP8bTH7tLI0w+JeZQnniU5KUg== X-Received: by 2002:a05:6000:4203:b0:47f:7b75:61a with SMTP id ffacd0b85a97d-47f8d71f494mr6221114f8f.15.1784839392888; Thu, 23 Jul 2026 13:43:12 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:12 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 09/10] nvme-rdma: use ib_dma_map_sgtable_attrs() Date: Thu, 23 Jul 2026 22:42:05 +0200 Message-ID: <20260723204206.76930-10-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" ib_dma_map_sg() folds every mapping error into a zero return, losing the DMA layer's error code. Switch both scatterlist mappings to ib_dma_map_sgtable_attrs(), which preserves it for the next patch to act on. No behavior change: failures are still reported as -EIO. A local sg_table is needed: dma_map_sgtable() maps orig_nents entries, and the embedded table's orig_nents is the allocated count, not the count blk_rq_map_sg() produced. rdma_rw_ctx_init() does the same. Assisted-by: Claude:claude-fable-5 Cc: stable@vger.kernel.org # v7.1 Signed-off-by: Mykola Marzhan --- drivers/nvme/host/rdma.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 6909e3542794..63830334c73e 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1469,6 +1469,7 @@ static int nvme_rdma_dma_map_req(struct ib_device *ib= dev, struct request *rq, int *count, int *pi_count) { struct nvme_rdma_request *req =3D blk_mq_rq_to_pdu(rq); + struct sg_table sgt; int ret; =20 req->data_sgl.sg_table.sgl =3D (struct scatterlist *)(req + 1); @@ -1480,12 +1481,16 @@ static int nvme_rdma_dma_map_req(struct ib_device *= ibdev, struct request *rq, =20 req->data_sgl.nents =3D blk_rq_map_sg(rq, req->data_sgl.sg_table.sgl); =20 - *count =3D ib_dma_map_sg(ibdev, req->data_sgl.sg_table.sgl, - req->data_sgl.nents, rq_dma_dir(rq)); - if (unlikely(*count <=3D 0)) { + sgt =3D (struct sg_table) { + .sgl =3D req->data_sgl.sg_table.sgl, + .orig_nents =3D req->data_sgl.nents, + }; + ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) { ret =3D -EIO; goto out_free_table; } + *count =3D sgt.nents; =20 if (blk_integrity_rq(rq)) { req->metadata_sgl->sg_table.sgl =3D @@ -1501,14 +1506,16 @@ static int nvme_rdma_dma_map_req(struct ib_device *= ibdev, struct request *rq, =20 req->metadata_sgl->nents =3D blk_rq_map_integrity_sg(rq, req->metadata_sgl->sg_table.sgl); - *pi_count =3D ib_dma_map_sg(ibdev, - req->metadata_sgl->sg_table.sgl, - req->metadata_sgl->nents, - rq_dma_dir(rq)); - if (unlikely(*pi_count <=3D 0)) { + sgt =3D (struct sg_table) { + .sgl =3D req->metadata_sgl->sg_table.sgl, + .orig_nents =3D req->metadata_sgl->nents, + }; + ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) { ret =3D -EIO; goto out_free_pi_table; } + *pi_count =3D sgt.nents; } =20 return 0; --=20 2.52.0 From nobody Fri Jul 24 21:53:36 2026 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB09F47FAF2 for ; Thu, 23 Jul 2026 20:43:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839402; cv=none; b=I0BCoAWaj2W23R1jh2ajK50GZYD9DMb1lwz7pE9NZ4QhogrluUsL24OqXjG1o1K6bWSeyjj6hjNstkPIJEY32TUX7FzcV3P5wiU/2CGM1G8F89xvqAzeVwjnJDWs6GY7Jln8DPm0rzJNfvkG6LJ95elkfJVJRhuJGvs4n2yrLoE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784839402; c=relaxed/simple; bh=eshNG+SSpxeB0R+uSz5ejlLdh8nbo3X4lM9abX04Wkw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=J/Jbd/yDL4pH/tuBBjF6Ry8FRSU9wi36V9/q4bsFd+k/z1lPfysH37pDq9c3nFozBqwq/QKBBRhjdR5M9Awx/hsXz75zjgHKzb/SQ6epjBzKMftiRKDpjHC5kpuiBfN5o47cH2KiviGSTNJRDvZbj2WsT/9Gfcn0vg6RFAZQVjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=JP/MQufB; arc=none smtp.client-ip=209.85.221.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="JP/MQufB" Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-4728c12ba97so667242f8f.0 for ; Thu, 23 Jul 2026 13:43:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784839396; x=1785444196; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Cyyjdg8Mj0TLFiupy7K/HZDk4131Cjbb/1b/0iFmC/s=; b=JP/MQufBS8sYRNFsRJke4HbFtOcpynWXtaii0Nkh9nxYLvzmUkC6+/GW1vBl/FFYJz AZE/zDqNW7APG7+P6BvOApXLNTZ3KmfN4wFnysPIK0fYliRydP5641HFHHZRnchU0HSQ gSNGgLH2DJ5n4O4XPaD+42bdczIO1eYe5oHPyVNZ8kOF8Pb6V2l46scxhzbRFw27SDI6 clHHVUIVYeoUpMokIUlcCa8ZI0YHJfA/IltHet7Geu3r0sgWp3BfePaXf6w0JiUCLTLt LUCnkQitN0ajMKyxO7mvooDh9rgQy0cPHV5auaFHzJGSoX5MigM3RBC6OsypKZy7ZWWs cTsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784839396; x=1785444196; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Cyyjdg8Mj0TLFiupy7K/HZDk4131Cjbb/1b/0iFmC/s=; b=tF5mgWu5NIfBv7C4DVycV645obCbkTPeNvwil0v24h3blEcGh/4/AQdXbZ9yD1wmfT /eEeWBcK+0BbMdL8YfTZ6/Q69k7fQ914U6PxFnRQamZFMPGqUsP7aOLptEHewVTFCTW6 wz1zRGjeXHTneYRBoC2hG8mlXkRAZdauInQF0cQ0IPgAa9fs7XdeoYzlcbJPGrK+NiZT eO1tw8XQjJE7qkRr87vdrC5mc5MMJsXrXW4d/K6wLSXY07A2lmbkRvG0yCsDVMAHO8eL PaAdTgydLLhJxMY30KYqHlPrAkI9QvoJE0vtSdNcaMdV8QsOVgS3KAAcTPRvf05Dg8Qx Ljrg== X-Forwarded-Encrypted: i=1; AHgh+Rpu8rYa6E7pu6mAc5fit8trAkSWY0+iNtAGx2xae1bCTW5QT7+maIYyQQ+N2NqfN9O9l0okEEFMwMIqW68=@vger.kernel.org X-Gm-Message-State: AOJu0YyC9Hf/xLzh5N9Zqu6Dwo4j0S537y6IbnczoT2iEZf+ivt43rIu j5dHKzGxrVPhP1YNwpQEbv38IyGKOoxfqnMTKeUlMOuAxETdzh7kiC0Psy5C005YtQ== X-Gm-Gg: AR+sD12pGXwYFiLXjRBNpuUg3exIENoHjUDVPlglUlgycwUlisIVMvUSWZiA2RHCCDx OURN4EkVyzFlOMo5TvhbetRgi0bO1bPDXqViHpoIu+uC6+xVznq6hw7VC7kFZxSElqisRjY2UeM 4Nxnp6RICxTWQsoVejDX8vBhkAcSlO0VVylfFz9J6WmEUvLz4RTeRHoa7sJAW8cumjscEqqj2B3 QVUFVF8p/EHNlQvQWgXcAYlcK7KuKCxpo3tp8aojfLafjJsoyVuGY6i9T9Ae4XJH+V0Wphbyhc3 zz3sN4d2wNJ7dnCdsRnA8nr+XD2qwUMtvBfAiA0sTSlI8EdAbqbSWbVtPupDdRI7ecfuFkw3YwK mtP4/QviVxlnRLbFqwmg7LNwI5W+bXO22xfV24SD9/BDkTpNUcuBAGAhOfBcM362FmSGoCyeG1B vC1+A56PPH70z3vIr59cnC0dzxfIZY50c7+3M6aUSZef1m1XsT6ZCG7ii2IbpRyqP99pbWNM+RU ax2J253Ecxk2XdYo7uUUqAjkwxcycXNQzovtEDSwA== X-Received: by 2002:a05:6000:25fb:b0:47f:9446:95a1 with SMTP id ffacd0b85a97d-47f944696b0mr1793960f8f.25.1784839396094; Thu, 23 Jul 2026 13:43:16 -0700 (PDT) Received: from meshstor-pc ([185.87.42.56]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-47f85c52fc0sm18268192f8f.23.2026.07.23.13.43.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Jul 2026 13:43:15 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , stable@vger.kernel.org, linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v5 10/10] nvme-rdma: return BLK_STS_P2PDMA for unsupported P2P transfers Date: Thu, 23 Jul 2026 22:42:06 +0200 Message-ID: <20260723204206.76930-11-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260723204206.76930-1-mykola@meshstor.io> References: <20260723204206.76930-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A P2P transfer the PCIe topology cannot route fails DMA mapping with -EREMOTEIO. nvme-rdma folds every mapping error into -EIO, a retryable host-path error: multipath requeues the I/O forever, a single path burns its whole retry budget. Propagate the real error code and return the unroutable case as the non-retryable BLK_STS_P2PDMA. As in nvme-pci, -ENOMEM now requeues (BLK_STS_RESOURCE) and -EINVAL fails (BLK_STS_IOERR); -EIO stays a retryable host-path error. While at it, start the request only after mapping succeeds -- nvme-pci's order -- and ratelimit the map-failure message. Fixes: 23528aa3320a ("nvme: enable PCI P2PDMA support for RDMA transport") Cc: stable@vger.kernel.org # v7.1: requires "nvme-rdma: use ib_dma_map_sgta= ble_attrs()" Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/nvme/host/rdma.c | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 63830334c73e..2f50509a7a61 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1486,10 +1486,8 @@ static int nvme_rdma_dma_map_req(struct ib_device *i= bdev, struct request *rq, .orig_nents =3D req->data_sgl.nents, }; ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); - if (unlikely(ret)) { - ret =3D -EIO; + if (unlikely(ret)) goto out_free_table; - } *count =3D sgt.nents; =20 if (blk_integrity_rq(rq)) { @@ -1511,10 +1509,8 @@ static int nvme_rdma_dma_map_req(struct ib_device *i= bdev, struct request *rq, .orig_nents =3D req->metadata_sgl->nents, }; ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); - if (unlikely(ret)) { - ret =3D -EIO; + if (unlikely(ret)) goto out_free_pi_table; - } *pi_count =3D sgt.nents; } =20 @@ -2033,8 +2029,6 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_= hw_ctx *hctx, if (ret) goto unmap_qe; =20 - nvme_start_request(rq); - if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode =3D=3D nvme_cmd_write || @@ -2046,11 +2040,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_m= q_hw_ctx *hctx, =20 err =3D nvme_rdma_map_data(queue, rq, c); if (unlikely(err < 0)) { - dev_err(queue->ctrl->ctrl.device, - "Failed to map data (%d)\n", err); + dev_err_ratelimited(queue->ctrl->ctrl.device, + "Failed to map data (%d)\n", err); goto err; } =20 + nvme_start_request(rq); + sqe->cqe.done =3D nvme_rdma_send_done; =20 ib_dma_sync_single_for_device(dev, sqe->dma, @@ -2070,6 +2066,9 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_= hw_ctx *hctx, ret =3D nvme_host_path_error(rq); else if (err =3D=3D -ENOMEM || err =3D=3D -EAGAIN) ret =3D BLK_STS_RESOURCE; + /* Peer memory unreachable from this device: don't retry. */ + else if (err =3D=3D -EREMOTEIO) + ret =3D BLK_STS_P2PDMA; else ret =3D BLK_STS_IOERR; nvme_cleanup_cmd(rq); --=20 2.52.0