From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17C86384CF6 for ; Wed, 22 Jul 2026 18:59:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746745; cv=none; b=QZKxMGJahuKU5mMfhJvHHiFRUsugoSCo0z6U6hUglYEWSLrhVf2GsOi+DfirWsWINuYKIbglFfXzjg4OHvuSPfMmlBPcIAwhzniUEDEzewuGSxW7+5NXWPP54fJ7wZVp/+bnxzRRlLYjI/rOnNxm/S+8J27ekqH/KmcXwBP99aY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746745; c=relaxed/simple; bh=cRH7W6syefYxFA3VhhN4QvzFBoE55r52A/E6t9wrD1M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qtrLx2Z/93vM3L9aZ9+M/IgI+pLWz5zA0LP3QhI57Nj2dUA/60UC0xHV8p/OKCUL6KO5AMwaSZjc8eaIsScA3/iGw55sWyLJpZoAB6bperL2L0aJpJMjcA394n+O0qKkbMQyHhEloO2VawjPzFqm7V/5okrDAgsvuNWohfzL488= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=PjXrY8sk; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="PjXrY8sk" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-4954dff6536so36841725e9.0 for ; Wed, 22 Jul 2026 11:59:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746740; x=1785351540; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LNWXk+rlZiNRYyXRDGVV7qCvY4AZPHPO/XYCzCCpSWs=; b=PjXrY8skVYXsu0ZtujioERfZepLyP1Qucn4GVHfDugrdZIMyqzKP0f+V5fm7jWQZ8g HLp++x8eJH9DsAyezDkUmpzHkQbk1hTzyR61GBUpSIzsPgI/OZtI/CEsA9yOvmTioqYZ /CRuZt6aNksbCpDhZYuIJaeUJ6K4Hc0R5olcuxheTb+V6wG/OViRjutmcg2b9S/3QHmu jaDuIAFDCkHkdZOyesapOVYEojFY26oBMqu4Epv6nn02TbXBG9sD1+lkzyqkCT5tn6it O5xlT1JYU4URKGXBMXg+rB5D8kZ46hAbEO9K9OKJgAacrlFZytEkvLV7278vvyJrp4+T bmUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746740; x=1785351540; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LNWXk+rlZiNRYyXRDGVV7qCvY4AZPHPO/XYCzCCpSWs=; b=K88C7Rf4505N5yxyQhXmg+M9l/NYzp0L0BFcUGHtkblqGcCIcDnhjOak5kM/hCI1Xt Dqget0hKaGZOuKjbMktDNvZ+t9jSrvUx1XRjgH6OS9kv1ERj2/YaXVPWwu3tSGoAKexr HVlw891X+0nzMOw9pJby8jDpLZYdPUJyt7R4C9qz+40GVn6c0LAVt+rjd3ZTswoXQ7Km 9e/Zae6O2aahLMcof7hY6HzYhzoq2u/nSIStLftRqQ5ivxkLcYAm+E03LYX8o05mo0Bq jLeyh5rDKCHcyEEYeV+5wFKkt95LFwHIZFqY+8uym0K6V7ylrDpnNy6py2XBAG//4KCw I2xg== X-Forwarded-Encrypted: i=1; AHgh+RrAXQLa3oPABLeiQOeukat65ItNlKPXlHUKh5saVL9V8x/Knk85cR8c5VH0LNC+QgkzDFOFAELzscJCtQw=@vger.kernel.org X-Gm-Message-State: AOJu0YwcHxmS3BDXYg+pIszoyWX6iCcaF63L9TDEy2RO7HVc0uDBW4ri gwae76OLadCaENGrxocrASO/4j6hdqNSUEYPHR+WMtgWfklGpY4koLTINk4AV+uG3g== X-Gm-Gg: AR+sD104APD8MxbMNyo2cmVS+q8nDgbZ/Ac5A2crhr3kh/aEuFrAyOnDCgsxTpPcRGS IDyunKp83Kx3kYR/44k/QQGbsaB9+Y0aRdVn6DlvPlGnKoCeFzwmblBj+4UZwBQsjA7rXDHZaaE R+RLf1uZB+1ragIJk6e+Ku/WvrPZIcNdhZ3no0j5br2KLthAWC2K/GkotemBJhfHVQeF8mP3XNf CDIkHclqRdBlnpKmyLicScj+djmeimSXr+ofpBvBhRNx7tPd29Sf9cx13NciCZwH1nLO4hOnNum 2eYWyN8OgL5mbP412mgbxtFxHUCLIISVg/R+IijNqjav4ReOCjZ0uKUpqlXOxBik1aREM4acaT4 JcTIUdk1geFnvT3xt4TUkN7wf4uD1+/clshc/oSPnduJBX698Gq5ySNqtYOh4Bt5j+BeJYaTcGT AZ1ZGiyl01V794lZqe5vseNvemO5gy+IBdyEK/07oDdw9XnpU= X-Received: by 2002:a05:600c:3153:b0:493:e97c:216e with SMTP id 5b1f17b1804b1-4954a41382fmr294367705e9.39.1784746740281; Wed, 22 Jul 2026 11:59:00 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.58.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:58:59 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 1/9] block: add BLK_STS_P2PDMA for unsupported peer-to-peer transfers Date: Wed, 22 Jul 2026 20:58:33 +0200 Message-ID: <20260722185841.449934-2-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe blk_dma_map_iter_start() reports BLK_STS_INVAL when a P2PDMA transfer is attempted between two devices whose PCIe topology cannot route it. blk_path_error() treats INVAL as retryable, so multipath requeues the I/O forever, and callers cannot tell it apart from an invalid request. md also ignores it for member failures, so a mirror silently diverges. Restoring the old BLK_STS_TARGET would stay ambiguous -- nvme maps NS_NOT_READY, LBA_RANGE and CMD_INTERRUPTED to it too. Add a dedicated BLK_STS_P2PDMA, non-retryable in blk_path_error(), and return it from blk_dma_map_iter_start(). Fixes: 858299dc6160 ("block: add scatterlist-less DMA mapping helpers") Fixes: 7ce3c1dd78fc ("nvme-pci: convert the data mapping to blk_rq_dma_map") Cc: stable@vger.kernel.org # v6.17 Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan --- block/blk-core.c | 3 +++ block/blk-mq-dma.c | 3 ++- include/linux/blk_types.h | 10 ++++++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/block/blk-core.c b/block/blk-core.c index 365641266c9e..83cb5464265f 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -177,6 +177,9 @@ static const struct { =20 /* Command duration limit device-side timeout */ ENT(DURATION_LIMIT, -ETIME, "duration limit exceeded"), + + /* no PCIe P2PDMA route between initiator and target */ + ENT(P2PDMA, -EREMOTEIO, "peer-to-peer transfer unroutable"), ENT(INVAL, -EINVAL, "invalid"), =20 /* everything else not covered above: */ diff --git a/block/blk-mq-dma.c b/block/blk-mq-dma.c index bfdb9ed70741..e88d6b3524e6 100644 --- a/block/blk-mq-dma.c +++ b/block/blk-mq-dma.c @@ -189,8 +189,9 @@ static bool blk_dma_map_iter_start(struct request *req,= struct device *dma_dev, */ case PCI_P2PDMA_MAP_NONE: break; + case PCI_P2PDMA_MAP_NOT_SUPPORTED: default: - iter->status =3D BLK_STS_INVAL; + iter->status =3D BLK_STS_P2PDMA; return false; } =20 diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h index 8808ee76e73c..c8de7809320a 100644 --- a/include/linux/blk_types.h +++ b/include/linux/blk_types.h @@ -165,6 +165,15 @@ typedef u16 blk_short_t; */ #define BLK_STS_DURATION_LIMIT ((__force blk_status_t)17) =20 +/* + * BLK_STS_P2PDMA is returned when a peer-to-peer DMA transfer cannot be + * mapped (typically because there is no route between the initiator and t= he + * target that the platform's PCIe topology and ACS settings allow). This + * is a property of the initiator/target pair, not of the target device's + * health. This error should never be retried. + */ +#define BLK_STS_P2PDMA ((__force blk_status_t)18) + /* * Invalid size or alignment. */ @@ -188,6 +197,7 @@ static inline bool blk_path_error(blk_status_t error) case BLK_STS_NOTSUPP: case BLK_STS_NOSPC: case BLK_STS_TARGET: + case BLK_STS_P2PDMA: case BLK_STS_RESV_CONFLICT: case BLK_STS_MEDIUM: case BLK_STS_PROTECTION: --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E2DC35C685 for ; Wed, 22 Jul 2026 18:59:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746749; cv=none; b=I8+krZDeFa2usokKNEP1NAGEuBdaKfWt5ZFwP4dYtYxMrLfdsgpiphW9I8jZKweI/s7/84dSlJy9EQdQXWyy+gUNtuBiGrDWDAtaVyfqesDoNRQpRN0x0JuH6r6cRtjtCeAWL8PekYAYKUW6mOrthvqVI0IOLcizGJMDU/1UWoo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746749; c=relaxed/simple; bh=UmHDQ8Avh88213YFsBTKp7WY9yAPzXpx6PsDprJSvWw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P+BhImr9dG1mKm5sufS6gLtWEPNEw6tZHTR6dCaVx9UNXRq08h42HzxlcuM8C4pgLM1EluFA9ZA1a58gBjrRcAVVDA8K9w+Mj7dBX808p8YblNMH68Ui7TgwMJcG8gWZn+dpFi951g/QoITk0q4dQQtgmeB04J6b+2y1rO4mxAk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=pO2ctPAC; arc=none smtp.client-ip=209.85.128.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="pO2ctPAC" Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-49571fea44eso2338565e9.2 for ; Wed, 22 Jul 2026 11:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746743; x=1785351543; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=V5hliKyCAPWFZqHS5mW2T1dJRPt4J/vTka7mcNCHq10=; b=pO2ctPAC573zzwGSfqULFeBA/5OYTJ9IkLMmUQ1RrtLSK3KfQS67DoOTO1kXRMVcya ACKo1PTnFejVlqK1mJ2B7lvLgQjhUUIz10hXfa3KGkvobunjwH6ccIGlqZtzITEUKLGo K9jTdr8MlsILiYfBr76Tfr4iOEOOXEXRgam3mpvmRrDutTw87yV0VAeAtxL7pA+B7WCZ mWoqdrMEA0gIJbX29W85S8v/qNcw3urPkU1GbNFi2T+jfYpMwqWoyuEBsCAdAh5SR3I9 OeVFWeaX4uwA6M0Nd+EQszydv0AU6zpsVxdwRx49or6g1yrsqNGgpKFlu1ORIke9WuSX GLpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746743; x=1785351543; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=V5hliKyCAPWFZqHS5mW2T1dJRPt4J/vTka7mcNCHq10=; b=NLHwsWuGskSYkuz7kqEKG5BFNUSkLv2kkz9EufIIFHG8J8hKDeIGGMNNOWh8TwN6pU 66MeCtI23Crop7il9WxvRvCEXstZFRICkhwxeEvC5h8NLp6TxhJFw386PA0UqhGkwOvq NGS0Yl5RGhWB516KsBLkD3f2mdBLREy+87Hu1AM3hlpmoNpY46v39xp9elsdjJQfvpAZ Nmwb3onZv0s8vogbLpW4lXeUKzVSvkvtwflZ0Fm/4F3BlbZvh/F3DsEyNLov9at90H3y /kORaYeraU76Bqy28VwsJqgcV/hHh8ePiiM9OeTi0fJ7jwn6PY2Fu+URALqGd+QZG1uH Wbhg== X-Forwarded-Encrypted: i=1; AHgh+Rq0XEEZikpP7f7dit8vYwl3dlvgDXr+3jO+l24R6nsTzOkQSdX+W/nEB5dYP4YBopmp6UiCXTyv3+qp14Y=@vger.kernel.org X-Gm-Message-State: AOJu0Yyn8/AgfIYQhWApN65K03sZIKPWffq2kOYzZ8VfWwMN6dQmd8dN 1X8K8/OlOWuL/vpfmCHDuiGuRC9Iaz/burwf6HLBbQneVTH4lYtYBvU3kj8wEag4hA== X-Gm-Gg: AR+sD111Psh0G1Wgwu4fZt4H8WMIqebUy9JflZQVo47QfOgHNj+rFMar0vQ5ssP5+Qm e6ou27fRMzgSaT1xlHn9y2WRs5V1nivgI1gF2MC89lfpu/OPe1kK1MaUrSQR/Nx0rKRfAfs35x2 xrpL6qzEFN99YQU+7olTkAkBQ+OYSSPVQWcMMidyRQuPF5dfFAi4VdplvlFdi81F6ada3gV3unZ VHO9pcQzNIaMUicM3cRm8/Om0GVoKlmiAc8hqWKwPAVNEplPe+tjo6ZhVrZVPVIeYSv7rTK3d1P tMLuiMM4nm9T+JTlRfMhzFk5EEyq9qHtpoHp625XAE9iK8F2gplkIvoxeW/ceiPmseEdC/ofG7y OZklSUiS6f2ZJ6irQfxs9CXac/6B0qXRderNm3nOHYQJ0O1sgucA8n98UxLpHvCy3qmvEHRYMct g8LEiqqWzGuZ2Sk3ldWSeTevuNGt4gBBr7WqIo65R9gKi25K0= X-Received: by 2002:a05:600c:c174:b0:495:63e4:7f78 with SMTP id 5b1f17b1804b1-49563e47fa7mr133877135e9.10.1784746742455; Wed, 22 Jul 2026 11:59:02 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:02 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 2/9] md: ensure REQ_NOMERGE is set on P2PDMA bios Date: Wed, 22 Jul 2026 20:58:34 +0200 Message-ID: <20260722185841.449934-3-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" md_submit_bio() strips REQ_NOMERGE from every bio. For P2PDMA bios the flag is what keeps requests single-provider (see __bio_add_page()): merging a P2PDMA bio with one over a different pgmap, or over host memory, maps the merged request with the wrong bus address. Set the flag on P2PDMA bios -- bios built via bio_iov_bvec_set() arrive without it -- and keep stripping it otherwise. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/md.c | 10 ++++++++-- drivers/md/md.h | 15 +++++++++++++++ 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index d1465bcd86c8..3770bdb4d4b2 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -451,8 +451,14 @@ static void md_submit_bio(struct bio *bio) return; } =20 - /* bio could be mergeable after passing to underlayer */ - bio->bi_opf &=3D ~REQ_NOMERGE; + /* + * A bio md split may merge again below md -- except P2PDMA bios, + * which must stay single-provider (see __bio_add_page()). + */ + if (md_bio_is_p2pdma(bio)) + bio->bi_opf |=3D REQ_NOMERGE; + else + bio->bi_opf &=3D ~REQ_NOMERGE; =20 md_handle_request(mddev, bio); } diff --git a/drivers/md/md.h b/drivers/md/md.h index d8daf0f75cbb..73df1a6eccd8 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -11,8 +11,10 @@ #include #include #include +#include #include #include +#include #include #include #include @@ -22,6 +24,19 @@ #include =20 #define MaxSector (~(sector_t)0) + +/* + * P2P and host pages never mix within a bio, so the first bvec is + * representative. Read bi_io_vec directly: bio_first_bvec_all() + * WARNs on the split clones md handles, and data-less bios have no + * bi_io_vec. Not valid after the bio's iterator is consumed. + */ +static inline bool md_bio_is_p2pdma(struct bio *bio) +{ + return bio_has_data(bio) && bio->bi_io_vec && + is_pci_p2pdma_page(bio->bi_io_vec->bv_page); +} + /* * Number of guaranteed raid bios in case of extreme VM load: */ --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2152372EE9 for ; Wed, 22 Jul 2026 18:59:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746750; cv=none; b=sgESn5ZhBPmHJbjTAIfDb8z5L7iyqai6C90HndK/C1eei3hVhiNisigBTa91lmFUVREDf5UouXBZ/fmOnQ4uEncDpKAvWKhV2kRpPgxXRA5RKImyrjCjFIzYatGNFQLWgmY9oR62LIsySuSFJFqY6ELe7rUNmnashT4wCyzkU68= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746750; c=relaxed/simple; bh=hrTCMMr70YntHE8rHfpC3UJfjfd1U5mS3A9u7ATI3Z0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mfwt/TWIMwYEDSiLP8B+QZPBNC1ddas8Be2o+rU7ZI/LDtkLnCeu6YSmJGGW7B7YhftWD3tFxMDIZ+3if3+vewiUbtVD3yfsFcmojP30wDh1eCa6XwtuVKknY9edCaZmZTn6vkECCWwIJy4+hnvlAEVTpoBTplBAgowKiwi20WE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=SCM4LtE0; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="SCM4LtE0" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-493f75f7172so100317225e9.1 for ; Wed, 22 Jul 2026 11:59:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746744; x=1785351544; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=GLcS/ETJPdW48ntvSy2VqAomcBgiATmYTCUKJMNlgIw=; b=SCM4LtE0/HpN8EcuX55CGhXLzQF6CsoQYXwzal4P7HmkEvDIZZkJr5vC/kZovKCeL5 PbzcFYqq0NevqxMdGWPlGgJNkm069JS2A0YXg+z/gRCj9mM8V8mTJ5g0OePRGhZahP2R WE1+kftU5H0VLSnCXfGtSkx5C5lJoWSsgahn9zwgXpEuuEgZuQwbsFRU9c/YuKlbGSmt ls30RVA7zLvvd4l2JiWbZKamN9xdgNZiryp5kbj9H5flxCjCwY1nLZRjB14Jy4PxFwTH wW+Mbb5hP/c+D2MkM3PGnESFAs905wfwpcYLmk/M1LL3WJLiy+rXXs9KPasSJz4m//Gf a2bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746744; x=1785351544; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=GLcS/ETJPdW48ntvSy2VqAomcBgiATmYTCUKJMNlgIw=; b=d79qdJFIRKepYdo7Kdb568dXoFx2ETAFg+Lke7ODc+mhTW+ny1rTWLY0sC6ziWrPjb GvhB7WsDwL3XPoFnk9A3PFHka1H0XpkEo0xfT18fjhMWWinP/SzIfUSd39YNcJGUAqvl 05fHiO9Lb/KrfcvVzUnuyLwHn0Np41stKnWsrl5bfoBoQKuJcnHEn5klSe3mNCzgoQCa CJaOmYsV8ALlYwLpAAbFSW+7yaE/cuIs1ztXbD8XY0tO6dumRTTvx66Be5MgfMy85m9+ zvFdzgKz4rYd8I4/EVfNiQCORaF38yv0e0V6czf0NAMp0Hd3LiTO3j3t0sLa06se778A BJ8Q== X-Forwarded-Encrypted: i=1; AHgh+RpMpqmKio1utann5fRDUUFnPJOuKoXn03uPnWaMoP5efpAadDWPykkuSd+6k/2KCbi66eaaIAg6zGV4aUY=@vger.kernel.org X-Gm-Message-State: AOJu0Yw2maBWScziDQoLSaIRPNLHx3YwyHzpUmeLnDSh9cX1IYPl/2dn qzUXckn9xGqbinbwnx2id2/aZgSecpiuqQ9jLZtvOYmom70Wo+1StbiXK31NTr/xhw== X-Gm-Gg: AR+sD13AZ5zNHpI8xA9EWlmJMIEh9aQHpsPTAPQqqOnrGAbeLnjkyp3Z4z5RotDUdFE Z1GiK6CBIWEWF2ZDouPu5AkUsB9e56BJPqvBXiVxI+6bYUJI52IkhdUjTU0Y8U5NE16Ue33MN4x HrB8GUW3ufr4IFfIpPZtw0CSmyYiN7jlZPoEhJV/0Ie7Sm8QbDGzLBIuEH6DKARq9oDWZ7HbVfn vZAl7f8PWwSO3Mb/sDsPp9ldBqVxYN+lutAc6RkaYdVVpzLQr316HsmZD+LFOqLOe7xvVN76DoE j70ubQet8j8ixMBzhPEtvX4+ToorHX/kygWlqaqNpYmFlXDQs2CtUhuTB5xSuIbS+aIgck0RRKb K5PQP7mmucgq7qaoLmWeZESx8hjuKzn+1DuukZ/GdA9IwhG5bjtGHN6WFmiXiAc3hJt3VqhN1h1 x7Dk8uMQXj/Bk9PeU4kt9A2Pdb1sc8Og+U5kCzVYN+BLZgG8k= X-Received: by 2002:a05:600c:c84:b0:495:4811:d71c with SMTP id 5b1f17b1804b1-4954a3ed4bamr271393965e9.13.1784746744468; Wed, 22 Jul 2026 11:59:04 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:03 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 3/9] md/raid1: serialize non-write-behind writes on CollisionCheck rdevs Date: Wed, 22 Jul 2026 20:58:35 +0200 Message-ID: <20260722185841.449934-4-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A write that skips write-behind (full behind queue, a waiting reader, failed behind-bio allocation) races the in-flight behind write it overlaps: nothing orders the two on the write-mostly member, and if the older behind data lands last the member keeps stale data for sectors already acknowledged as rewritten. Serialize on CollisionCheck, which marks exactly the rdevs owning a serial tree: serialize_policy rdevs plus write-mostly members when write-behind arms serialization -- the case the MD_SERIALIZE_POLICY test misses. remove_serial() under the same condition. This also avoids the old gate's latent NULL deref for rdevs hot-added under serialize_policy (they never get a serial tree). Fixes: d0d2d8ba0494 ("md/raid1: introduce wait_for_serialization") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index afe2ca96ad8c..997f79c05db7 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -564,7 +564,7 @@ static void raid1_end_write_request(struct bio *bio) call_bio_endio(r1_bio); } } - } else if (test_bit(MD_SERIALIZE_POLICY, &rdev->mddev->flags)) + } else if (test_bit(CollisionCheck, &rdev->flags)) remove_serial(rdev, lo, hi); if (r1_bio->bios[mirror] =3D=3D NULL) rdev_dec_pending(rdev, conf->mddev); @@ -1677,7 +1677,11 @@ static bool raid1_write_request(struct mddev *mddev,= struct bio *bio, mbio =3D bio_alloc_clone(rdev->bdev, bio, GFP_NOIO, &mddev->bio_set); =20 - if (test_bit(MD_SERIALIZE_POLICY, &mddev->flags)) + /* + * CollisionCheck marks every rdev with a serial + * tree; order against in-flight write-behind I/O. + */ + if (test_bit(CollisionCheck, &rdev->flags)) wait_for_serialization(rdev, r1_bio); } =20 --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35001382379 for ; Wed, 22 Jul 2026 18:59:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746757; cv=none; b=DDk1l0grZOuaHF/wHbM3Occa/ivG5ZcH4cAtfFWClm5VC+/cnUkihfHT4iTnc150eX1w+lHiV5EM4+g7io+F+gPG8rarIL2SKfka0fG5UJcPktzfeop9TiHlSnYgJPGgak07EBki7wHeq5I7qo17LL0JK9R1PmPr09+Fq7N90mM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746757; c=relaxed/simple; bh=tb87IS5YwxWP0/pdiRmCvUCzVsNHu9wl18O2iixHy6s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LDQBN76U3tLV/SzyRPc3AbsShtUXYslKhwEHdsiMLINlY1brZFdEEbscsbp0bDN4wjampUCwlxtTvhOd9e0dPEP+nZm54l+AEgY8NzbzicsftsTvtvbynVcNdUgi6wwMSKf0O8ryBAlinRO9RJIrBiR3kxvu2E3MMJRM8atc668= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=ke96zD1A; arc=none smtp.client-ip=209.85.128.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="ke96zD1A" Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-4955484387cso28196875e9.1 for ; Wed, 22 Jul 2026 11:59:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746747; x=1785351547; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=jvUC0Lia0it+LgLYsOXSPjeMnXEhLq3Hqn52LEDkVn4=; b=ke96zD1AV2KN7/+6dDmPKBnVJGHdbbiVVwXnekYtUt5RKh8NxsEXdLh4Uhrtj3OP+z Qa5Tjtwp+zgHtPjbmH3sLbPAuREYsRuyEi64W2NwBdnsmMmAk6WQ7+qwFI7D272BVQcX +4x7jTG20yYoO5cuGmCNe/+2tylFSiszgDYajx4OjtF9KyTRz0yLQqTUUYwHiHxpTAT4 RWdrCIx2RSbWF6yp2I+Adi8w+y0/CWUUFbEkOqc4fXhCR1L3uydeMAZZiLN3IZS9fQv4 qUxQzbQxifQSHcmfu3kGAYb1Sk1hGER4swGw2xx0wzQnRi+IYx0e3kS98iq0ytplswMF JBew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746747; x=1785351547; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=jvUC0Lia0it+LgLYsOXSPjeMnXEhLq3Hqn52LEDkVn4=; b=fOwiUiigtXbSyzrP/B1diGKNYckjxdiySNDEPnHjtFkEy0MifT9LHXj1+Ple4WVTzf SntCk4IRnfSYNnGdRRcxaam4XlpKWN0GkITx3d/4yhw9R8p/EVl0xwcwZrI9mldxthjZ rs9X+coSeEv+jgLIUDiqCuUSOmpITnjrNYQk5wwroIxAxqTGsiAjdkF8YzhTxxp/32dA IWemnH21mOW6NoUJIOibpLj41FBUGBZBQHxLFzZov0R/Xpe962AWPbqoYUygbrXv3tv7 ygstS3hlPYZa5cuk/ZrFqD3TVzr3GM6KX1BCK2BwEptKi5gwERd5kPTieWnWD0pFMKUN qUEA== X-Forwarded-Encrypted: i=1; AHgh+Rpch0ym7g+tfv/2SHkk1J51IKlA3NPkvsdQap3tUzCgsHFeSbipFRSZCZPobL5UkMo6ESDdGuIZJ/twDug=@vger.kernel.org X-Gm-Message-State: AOJu0Yyqsr6nKq1/nBBIFPicVf1fvRhWJkchuHiuWEQfopApiXPVveOk rTzFMwhcy0LqTlXzsWeAndmUZ/a7E8jYjahlinyPHJHQX+9deGHJJDHWGiy69wvAZA== X-Gm-Gg: AR+sD10kU7WQgbaWqJacPwmZHTvsAmYBGgUnl4882Th3K39OI+CELuXoePeeAC19xth Ruc7nQzLfsFoiQbdJNBg5AP5Jr0S+yHo5R3cVUm+nUnpAbY+WC69UPduYd7QjKv2T91Y5wv3v2V KXKmEiHhOuBAq1EOW1ZPI3tJvKx3kn4ZL4A0XuFRfF54ycEJtPsyvCR2gG71vcsVICtxwW6XbM7 JygnXimurYGz/pPWAUeVJpNFBY3gsQ5LIWugXxeUepATSEk2WD8PwqZwQOYX5ORQ7SYoWZEWyiK L6Ffriazb9daSIjCG1eb8gHe7UNTQBWaXcdqY9Mq22sgbsE/88vLDLsG9Ri6pjIe9BTxHX9SgeO Rx4+YzivS7bywnGAdWHmJuwG+ytmRcL07cGRQ05NQVwK6My2LuNqE7+bkRBUOtbgAjBRXEQwWqy rRE5KkWN1t+TKOobrDWjlVrtTFe9ais5KlzvlOmc2cBVoEqs0= X-Received: by 2002:a05:600c:2049:b0:495:6397:14b1 with SMTP id 5b1f17b1804b1-495639714bfmr108525535e9.34.1784746746655; Wed, 22 Jul 2026 11:59:06 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:06 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 4/9] md/raid1: don't use write-behind for P2PDMA bios Date: Wed, 22 Jul 2026 20:58:36 +0200 Message-ID: <20260722185841.449934-5-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" alloc_behind_master_bio() copies the bio's data with bio_copy_data(), a CPU copy. P2PDMA pages are peer device (BAR) memory; generic code must not assume CPU load/store access to them is safe or fast on every architecture, and bouncing peer memory through the CPU defeats the point of a peer-to-peer transfer. Skip write-behind for P2PDMA bios: they are written directly to all members, including write-mostly ones. Ordering against write-behind I/O in flight to overlapping sectors is preserved: the non-behind clone path serializes on CollisionCheck rdevs (see the preceding fix), which covers these bios like any other write that bypasses write-behind. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/raid1.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 997f79c05db7..7d769b6460ca 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1523,6 +1523,7 @@ static bool raid1_write_request(struct mddev *mddev, = struct bio *bio, bool write_behind =3D false; bool nowait =3D bio->bi_opf & REQ_NOWAIT; bool is_discard =3D op_is_discard(bio->bi_opf); + bool is_p2pdma =3D md_bio_is_p2pdma(bio); sector_t sector =3D bio->bi_iter.bi_sector; =20 if (mddev_is_clustered(mddev) && @@ -1575,9 +1576,12 @@ static bool raid1_write_request(struct mddev *mddev,= struct bio *bio, /* * The write-behind io is only attempted on drives marked as * write-mostly, which means we could allocate write behind - * bio later. + * bio later. P2PDMA bios are excluded: write-behind copies + * the data with bio_copy_data(), a CPU copy that cannot be + * assumed safe or fast on P2PDMA (device BAR) pages. */ - if (!is_discard && rdev && test_bit(WriteMostly, &rdev->flags)) + if (!is_discard && !is_p2pdma && rdev && + test_bit(WriteMostly, &rdev->flags)) write_behind =3D true; =20 r1_bio->bios[i] =3D NULL; --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9ACB938887C for ; Wed, 22 Jul 2026 18:59:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746757; cv=none; b=HoK8rgU6fs5JEyxd8NvxeUKZC0hvdoKH4K0J0/CPLKpPg0dedCN0U9UMepn8Yh7utKIeRg4R2uW0iSgzFW0BPbsNewwiKNKLZ0MRpG9sNcypXVcbVsNM7cS+lhlquFcJx/8cYJOV/EnfCWs2T62rKGU/oK7UkOhhimbgQ1lOWO4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746757; c=relaxed/simple; bh=B1aIegApdgScj51bEIGVKX+FENfztmE4q/fjJ31TwzI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kTk8RXMFz1bFDCI28KtSChJWW5iChRfzzLn3j5mk2D0ksgFOdirgZhX9T0PhQlRInIYTGu3ohQNgW2AmMyV6i1e4tDISytKV7Oz/LCEuT7nE2mSYhZmUUt/d5bMa4ohX6TlQX7lziGAf2n6VfCnmv3+fIoNWWYJ8Eg9YQtNSGg8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=YG3QqS7J; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="YG3QqS7J" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-490cf322ed0so94472185e9.1 for ; Wed, 22 Jul 2026 11:59:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746749; x=1785351549; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vP39KbYqkNNRmuPyaoPhdZlQ+sTmuJQGW6DJY4lSzaU=; b=YG3QqS7Jid6IqJYT0W7IiR1qx0yowOy6yEbFhcrhAtS3pKtMdicZN82yqP+/8j0Hhl hf2tGciLP7iSxi/MovmA5oJLfV4L7DaEnA+J3Xop0aDrc8Uzb53Y/lhJoR6dABf5MpGd bE80Cyj0DHk1mTrQOaGh9JRDTnFGNlk0BEkx4irU0EZq4N/8pxv9fVFGsGEQQYidwuPw 5Ka1UuYw99pGXAYaP5Z6miKfuA6guPZDTvvj0qlqK2GEoOvsV5/s052JTkFrcGYeaWjM 4ZiGgdfbhyUGN8eQk8gJafo4CwRzSAj8VaXAN/20FDG4mv3q5HZwhGyP1aPXdGxIF/Hf lhIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746749; x=1785351549; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vP39KbYqkNNRmuPyaoPhdZlQ+sTmuJQGW6DJY4lSzaU=; b=a2HLx/AvY9mT+5lbvU1ikCxVkZ89kKtieLKUQ42OSQaaCCOXhkhyqkOTPxTH2OuHFc Fg48fj42HxIZNoggOz9eIhtKkv62bwD2I0X0MdghGTJpEkV70n1TfJTI7FYUO8F4z5gj TGtedGBzXN8UfG8pyHHDDU6nL3/R7FWN3S203eMgVI76eUipHW01ClhFZhXmTdGIHorr URxGeHZH0BqK6rizwu8VHCt/Q6TMpJKxJRr9zIl/6Syt8oFy122JkNvuMVUMkxiomhu/ v1gLi5Rq6LPVTKIM6wtVQqKZEBvfq9aTRwat4rKYBmfndJmzdQ+XR34PEnZPTvHuU3by nh7A== X-Forwarded-Encrypted: i=1; AHgh+RryKLeXs4rPVzRJ25s3zG5q780HI9nDpP9YzRAAXJRsB2UxgPa2GWi0RBVepp/W8ZJGIXNIRjIONsOnyY4=@vger.kernel.org X-Gm-Message-State: AOJu0Yya2BCJB0ymmg8Q5chu0naYdQ2Z5R0vi0SvP1cdaqbjH97mTSXK MGCQb5AKfGVgRc2uzNUdqlt1N9hFhHdopnI3RfVqytfTdzelBBZBbcfGGnpdH9Ih0g== X-Gm-Gg: AR+sD10LA1TBAI60DfRCU9VmCGnhISlKdDEatH6Bzfuuu579OZX/ZRTaoMCVHroUYfZ cS+gy9uu9j1NzENyaYxweP4COjen9PyCJZepaSYgSF9QktPWsXPcJU4Avbzz/55gpKjlNQi8vgZ 9v6R9ZnvEU8fX5gQqlKlzkK8PiZFTKxL+dIWDVLVRI0HpdRBrHgo0x8yfXXU5KyoBJ4DlnuWbXw 92eekTqwCxw+RcF9X64BCWPUs5SHbw4d8DWp3xnJIGn+mAb+BjMrFb14q0PuORjXjFTRShDTVwd UIJ1KWlzMZoLWWUg1Bj2QUqsHOaKgIqB9BrxnRyXx+cAwKBSF+G/WZ1S8DCj6IpqEOEz1XambUI JecvFYYHp6F9aIMFToXhCZ99CzJJ2ZQ1Re5/7/9v88zfIfl3DR7i3MC0EzAcFxoJHDJ3B8PZ0mF uybRv5KN2kLJLHFOQZrPWhS1MDIwG+ZR4tjHGrjdEPn2BProE= X-Received: by 2002:a05:600c:5490:b0:495:6130:7805 with SMTP id 5b1f17b1804b1-49561307a51mr135357485e9.14.1784746749133; Wed, 22 Jul 2026 11:59:09 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:08 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 5/9] md/raid1,raid10: factor out raid1_write_error() helper Date: Wed, 22 Jul 2026 20:58:37 +0200 Message-ID: <20260722185841.449934-6-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe Factor out a common raid1_write_error() helper from raid1.c and raid10.c. This will help reduce code duplication in subsequent patches. Signed-off-by: Logan Gunthorpe Signed-off-by: Mykola Marzhan --- drivers/md/raid1-10.c | 13 +++++++++++++ drivers/md/raid1.c | 13 ++----------- drivers/md/raid10.c | 10 +--------- 3 files changed, 16 insertions(+), 20 deletions(-) diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c index 56a56a4da4f8..cee21452e066 100644 --- a/drivers/md/raid1-10.c +++ b/drivers/md/raid1-10.c @@ -303,3 +303,16 @@ static inline bool raid1_should_handle_error(struct bi= o *bio) return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) && bio->bi_status !=3D BLK_STS_INVAL; } + +static inline void raid1_write_error(struct mddev *mddev, struct md_rdev *= rdev, + struct bio *bio, bool allow_failfast) +{ + set_bit(WriteErrorSeen, &rdev->flags); + + if (!test_and_set_bit(WantReplacement, &rdev->flags)) + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); + + if (allow_failfast && test_bit(FailFast, &rdev->flags) && + (bio->bi_opf & MD_FAILFAST)) + md_error(mddev, rdev); +} diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 7d769b6460ca..f07edeac6e5a 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -483,17 +483,8 @@ static void raid1_end_write_request(struct bio *bio) * 'one mirror IO has finished' event handler: */ if (bio->bi_status && !ignore_error) { - set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) - set_bit(MD_RECOVERY_NEEDED, & - conf->mddev->recovery); - - if (test_bit(FailFast, &rdev->flags) && - (bio->bi_opf & MD_FAILFAST) && - /* We never try FailFast to WriteMostly devices */ - !test_bit(WriteMostly, &rdev->flags)) { - md_error(r1_bio->mddev, rdev); - } + raid1_write_error(conf->mddev, rdev, bio, + !test_bit(WriteMostly, &rdev->flags)); =20 /* * When the device is faulty, it is not necessary to diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 0a3cfdd3f5df..5b63175fb382 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -482,16 +482,8 @@ static void raid10_end_write_request(struct bio *bio) */ md_error(rdev->mddev, rdev); else { - set_bit(WriteErrorSeen, &rdev->flags); - if (!test_and_set_bit(WantReplacement, &rdev->flags)) - set_bit(MD_RECOVERY_NEEDED, - &rdev->mddev->recovery); - dec_rdev =3D 0; - if (test_bit(FailFast, &rdev->flags) && - (bio->bi_opf & MD_FAILFAST)) { - md_error(rdev->mddev, rdev); - } + raid1_write_error(rdev->mddev, rdev, bio, true); =20 /* * When the device is faulty, it is not necessary to --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAF5238D400 for ; Wed, 22 Jul 2026 18:59:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746765; cv=none; b=Lirt7jFuMQ151v8dux1pWivvNd9EEm+hWSTP/YlgFLB2n5rDyG2zAKxhlRnwVelZmI55j2Uub+2orjA+iRtfkCkT6AOgJTws2y+aGyMbsdvqH36nmywcfVyW3JXtvJ2rPU7RPCAPQUhig+jGqI3Q8z2sj/GPnd19cFDxCP6dv40= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746765; c=relaxed/simple; bh=6JUXonkvhqCluPq91YNslI6Xc6PHhaJJ82TMx9QRo24=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Dfd4z5gAqFXMLw0lkj3Ncu1g0Bv24c3RsreW1h1hRTUb/wqQkxn2ZSRHtv+MuRDra4qMFBCHi3UBMVbY28X2WOpkBSoyHv2uCuO65w+v6ew7IxbYCsK9q+WHTubAm9RJyrgPU7mlru6hi7cR0U03DhZCGrEOe1aIyGa041swK3U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=Ai0g5NKB; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="Ai0g5NKB" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-49557167508so44945985e9.1 for ; Wed, 22 Jul 2026 11:59:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746751; x=1785351551; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=InzNp2nHOOp7QQLn15Ay7fcFkfBNFGx8C4/KxQQFD2c=; b=Ai0g5NKBb/WoSwM4WCinC9gL5DuAa88xLuEC4hRIs96ghwRQmvDlMLh5VL8zWotpFo JLjmRiIu37BjI3rhd4Ubw4snBPwDFxkD6abowchkrUjNFOey/CJUVv59O8K2o34bXApT 4wCOv64ZNQn2gEcUXAlJfmTzBq/ChFOvz0KDkCMfXy+BPORDQH/yJVd8q4o1hH5eyKUb S/BWDDtTklXRRcmxmHW3gZoCTvjMSA7HKGVHZyrsR6bfdDbQLne1R7K151Fu6TuNRAWt gNTwsano4qKf9TqettAQaDaizTf9HGlyParu20USjlWCEDbIz720Sk/FeTdSzuha7whc pl+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746751; x=1785351551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=InzNp2nHOOp7QQLn15Ay7fcFkfBNFGx8C4/KxQQFD2c=; b=U/WK/UqoZoH1RRXYEQ11XH7g84W5hEpmfnqbsrNg7OT7WHstZyv9H+yCltzoftWmrn rGFH+94PWSVFnK+rQkMVYrcKLytNYK5Ty7g0FTwycMVSCGoJyuZ3Fw9iijmGk8IcFy5L ElWQQs6pFrxpPkc/yo90qMmsIaZ5ePXMu2KcjXBnxr6gI9VaA9KObqmk8N611UulvS3F VeZp6aIAoZG4XV868Og5fFuLQ+5Z7x/1T9AgWghxezkr/o4PFsKieMYJMywwE57QljMa YWsaQ47INkvxthjfauWlPMASomi9OeGW7FVHyH2kbdWN15314tML+zJT78F7BTtm9Na0 sLoA== X-Forwarded-Encrypted: i=1; AHgh+RoAo7O87Hev/4K1irQcJFJGqowQ/L/vCXSt3hI9FaMXwA8DjqFjKDbCEPIwxfYR69uyICJhdtkdLTdrMtY=@vger.kernel.org X-Gm-Message-State: AOJu0YysenG9T7yFr9nYBmpkY6rXyoBKmEUVUre1sHwtfFMH5rkjdZe4 FZ3NATWeCm4IWdpa5q6d6WmixPSqwxvzWBfrK+1WJwPrYwwD16m1cdjybO4kuqBiQuxQTV9Xkce moWp1wg== X-Gm-Gg: AR+sD11SwrWQn+wmtCeUIwsnRWTesMsu3ef7bCnIr1TWdr9VqcFfCpK8Rn7F7sh01Ej rPgeS2fPa8jSvqBQOhcSZtLucJ/u4XfFG/ctStjMHCt/1FXe/UFnkYXDyelzhhrOUHB55Zd2VB4 ono2Y62UG+oRewJwn//wrll7g3pY0ZMU4Mjb9VZ2DonZ3BGmkGhKxe7x4LB6RUJ4LE9nt2lGVE5 G0bIHoP1P6m7wQB/aEQAMe68Zhf8hqFluGuZdUT272GEFPHrVC7bnZCunB2WS7VX5xkpePKMp2M oxJDtmoBjGWK9JM0NMdDomuQc9gWXx+Q8B+7Qp/GKA+3vVsiq04+p+e/LMfZI8/qV63+4maC48k XxVuX5ukHZcrIkopvaIrJ/sT9dlaSfa8ohfnA9evtjMgJXHTU3YIy2Ji/zXYVfQeh/tj+b/tNH7 DhaP23OBQqUl3M7Vx8V/xmYh6V6KjyR7cwUXL8ngjFztxIQNM= X-Received: by 2002:a05:600c:1551:b0:495:5858:e163 with SMTP id 5b1f17b1804b1-4955858e1damr203016475e9.27.1784746751519; Wed, 22 Jul 2026 11:59:11 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:11 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 6/9] md/raid1,raid10: keep REQ_NOMERGE on narrow_write_error() retry clones Date: Wed, 22 Jul 2026 20:58:38 +0200 Message-ID: <20260722185841.449934-7-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" narrow_write_error() re-issues a failed write in badblock-granularity chunks, cloning from the master bio and resetting bi_opf to a bare REQ_OP_WRITE. For a P2PDMA bio that reset drops REQ_NOMERGE, which is the only request-level protection against the member queue merging P2PDMA segments across pgmaps or with host memory (see the preceding md_submit_bio() fix): the retry path would quietly reopen the hole the submission path closes. Restore the flag on P2PDMA retry clones. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Mykola Marzhan Reviewed-by: Logan Gunthorpe --- drivers/md/raid1.c | 3 +++ drivers/md/raid10.c | 3 +++ 2 files changed, 6 insertions(+) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index f07edeac6e5a..19c4dec450de 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2561,6 +2561,9 @@ static void narrow_write_error(struct r1bio *r1_bio, = int i) } =20 wbio->bi_opf =3D REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |=3D REQ_NOMERGE; wbio->bi_iter.bi_sector =3D r1_bio->sector; wbio->bi_iter.bi_size =3D r1_bio->sectors << 9; =20 diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 5b63175fb382..428bddddfe71 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2823,6 +2823,9 @@ static void narrow_write_error(struct r10bio *r10_bio= , int i) wbio->bi_iter.bi_sector =3D wsector + choose_data_offset(r10_bio, rdev); wbio->bi_opf =3D REQ_OP_WRITE; + /* Keep P2PDMA retry bios unmergeable, like the original */ + if (md_bio_is_p2pdma(wbio)) + wbio->bi_opf |=3D REQ_NOMERGE; =20 if (submit_bio_wait(wbio) && !rdev_set_badblocks(rdev, wsector, sectors, 0)) { --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2FD654399CE for ; Wed, 22 Jul 2026 18:59:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746762; cv=none; b=ngJBEJckiT1senyak4ZyVCfN4NKKzS27iCNx8hCiNHn+eBZibdx3STSRBjUGHYvM9vb4yT5pDib0FTkvmFW8Aztffyr+E29HfjOdByolcwHKt9aiifVWmoX24mNUcbeGclu0fLJCmf+A2M+2XHq2oySSW2mqgN0CEMqG0H2MjKI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746762; c=relaxed/simple; bh=txEBr1MoU1y1mkJ/HHXFbnluQk+VRueyunkBGuSqgtw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bL4vt1P4bZvq+bHHkNVuWpuecoGEq78bnAqLUx6feu8NwlwyShkM6r3D/0OWBBeNriwis1l9MgJMAo8q0UGISpuWtI/pTyh1trbTQM52hCfMjgR7slsSZ0FrrG6oCDPXzRcnY20f15W6FDdZ5bNlFQgMVLwvxWrYSTYbirJYT1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=H1wkgX05; arc=none smtp.client-ip=209.85.128.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="H1wkgX05" Received: by mail-wm1-f52.google.com with SMTP id 5b1f17b1804b1-493f6de72faso45305835e9.0 for ; Wed, 22 Jul 2026 11:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746754; x=1785351554; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XGlz7sxVhffzS6r4ROT44BH0oh58kUxNrwba0cAE+h4=; b=H1wkgX05RHJTUWYprRsuGf3VDGrnWdopU+EKgLmswxyGXuyAtX/JXr2Yuh5N2t/JOX Yh0EiRSzmOI3gHd3cQSaQLV4wPbLXSMrUo6vOhcssWWslM3ZDyZ1hGSA0kl7iRZpHciE +It5bRwc0temO8hUL1zEx9DO8Xq9U/LuG2bJFf1g3aI1PvTOOmamJeDG6WDSK9qarsa1 4X43Tmrn6PTMlZ6pRckVGJ+DyfLyU9DVNtTgFFMEZ3ol3K5RJAtTOxOV8K8PxW8kXID+ KARFNKDueTbGrLuXCiQl+vHl4HwyLNw8FyO8YBJRE8cY03Jz0ixbLwVZsByBskLVGcAN Pf5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746754; x=1785351554; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=XGlz7sxVhffzS6r4ROT44BH0oh58kUxNrwba0cAE+h4=; b=QCslX9zZfOSaILSssqloyZd57721xn19sx7ckH2vd131k8F4QrX3DrpdGbomE01V34 HmWMrKBe6lTHRRM6F55bVLwLLN19w+iVX13Uc+hmvAIuFEXM1R+EGmXun5ajaSzmEyKG pChdJ1RicZZ0C34r8yeyeehcG7aIMmPV7t8wSK+5mIXb85pxvGJKpoAPEhJ2trx9Zqff U+PHMBKRK5OhhQea2SZZKpp9kBXIX7uG6h99zCku6JtHC2HvqzDll6JlKDxkqk8WAUJ5 yj7NlgxjP6N/LY3S/Q3Z0VBl4JGA12hSnUMgGZ91JU4QvIMs1H6p1Z1ynP+2w+15Z5u5 wyXg== X-Forwarded-Encrypted: i=1; AHgh+RrwhFwi5+hdpR2b4ddqk8gc1SCMo8Vc64fC63aKEQukfg+o+GZ4VTDRw0Qe7CmEiona1G+Ot+hJXi6E77k=@vger.kernel.org X-Gm-Message-State: AOJu0YzBKAbPedpp7+M38iS8RpYopIezbUkSxSD0pcjDTsDWjrXr9D9U BGnnxHzze2Makk9qOKjY7mKtvi2FxjabqZr60C08+xRcKXCXZ8mXpOt47QWd8jVjlg== X-Gm-Gg: AR+sD12qMihf7SQEEU5bONPKXg92EcVByRJhGG29Usu5DBl5hUk9A06Tx1bMBXu5HLn nP9yemfq+RK0oDAzN4Akrp+Ma8uz/IDmo2o3DJDJMByhIGhwlQijYUsQ8JoJTNhrbuR5ejK1uHi 34W51+NFeyNs95j+zRUL/KvGDWXH349PCDiVhrWJTg0p7oB9oUH5MP+k27mN+zkvU64kUwYOwQY 3bp65K8ViuLswoRD4829zj988QAtsdDUvxftfESQrf7l17b+Rq7EwnWRbn37BIPUCVH9LcIcCR7 PCdM979CNbeC9KHfE8+pko7IxdR/O1bSejIOFp2Kz/kUsDuN31cXWEt6OED7UtCZSSyTJa1t6gr W/aZ0IKiMIZEvL3jsaapG7Swky8uGf432U6DeCXxnIgwTdO1xnif1G82ziyf21SLilTEnPlauCT 3fw8q17LbZU9B9AohTHKzgYiAl7x929j0kdyVAC8yAgiVEtvw= X-Received: by 2002:a05:600c:1d23:b0:495:7016:b875 with SMTP id 5b1f17b1804b1-4957016c17dmr25808785e9.13.1784746753845; Wed, 22 Jul 2026 11:59:13 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:13 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 7/9] md/raid1,raid10: skip futile retries on P2PDMA mapping failures Date: Wed, 22 Jul 2026 20:58:39 +0200 Message-ID: <20260722185841.449934-8-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe BLK_STS_P2PDMA reflects the peer/member pairing, not member health: retrying the same pages on the same member cannot succeed. Keep WriteErrorSeen so the range stays fenced, but skip WantReplacement and FailFast eviction, as commit f7b24c7b41f2 ("md/raid1,raid10: don't fail devices for invalid IO errors") did for BLK_STS_INVAL. The failure is whole-range and deterministic, so narrow_write_error()'s block-by-block submit_bio_wait() retries are pointless. Record the whole range as a bad block in the completion handler instead. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-fable-5 Signed-off-by: Logan Gunthorpe Co-developed-by: Mykola Marzhan Signed-off-by: Mykola Marzhan --- drivers/md/raid1-10.c | 3 +++ drivers/md/raid1.c | 12 +++++++++--- drivers/md/raid10.c | 8 +++++++- 3 files changed, 19 insertions(+), 4 deletions(-) diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c index cee21452e066..89daf8558251 100644 --- a/drivers/md/raid1-10.c +++ b/drivers/md/raid1-10.c @@ -309,6 +309,9 @@ static inline void raid1_write_error(struct mddev *mdde= v, struct md_rdev *rdev, { set_bit(WriteErrorSeen, &rdev->flags); =20 + if (bio->bi_status =3D=3D BLK_STS_P2PDMA) + return; + if (!test_and_set_bit(WantReplacement, &rdev->flags)) set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); =20 diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 19c4dec450de..917d694ef401 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2624,10 +2624,16 @@ static void handle_write_finished(struct r1conf *co= nf, struct r1bio *r1_bio) * narrow down and record precise write * errors. */ + struct md_rdev *rdev =3D conf->mirrors[m].rdev; + fail =3D true; - narrow_write_error(r1_bio, m); - rdev_dec_pending(conf->mirrors[m].rdev, - conf->mddev); + /* Mapping failures are whole-range and deterministic */ + if (r1_bio->bios[m]->bi_status =3D=3D BLK_STS_P2PDMA) + rdev_set_badblocks(rdev, r1_bio->sector, + r1_bio->sectors, 0); + else + narrow_write_error(r1_bio, m); + rdev_dec_pending(rdev, conf->mddev); } if (fail) { spin_lock_irq(&conf->device_lock); diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 428bddddfe71..9045a3f02dae 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2940,7 +2940,13 @@ static void handle_write_completed(struct r10conf *c= onf, struct r10bio *r10_bio) rdev_dec_pending(rdev, conf->mddev); } else if (bio !=3D NULL && bio->bi_status) { fail =3D true; - narrow_write_error(r10_bio, m); + /* Mapping failures are whole-range and deterministic */ + if (bio->bi_status =3D=3D BLK_STS_P2PDMA) + rdev_set_badblocks(rdev, + r10_bio->devs[m].addr, + r10_bio->sectors, 0); + else + narrow_write_error(r10_bio, m); rdev_dec_pending(rdev, conf->mddev); } bio =3D r10_bio->devs[m].repl_bio; --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8C2B3A9629 for ; Wed, 22 Jul 2026 18:59:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746765; cv=none; b=slkZAwmEspbWmWSOU6tIspb2AuqO7B6cQi9uGRiUBI8/l3gVNJEN1wKHVDZ63YO8OdZGcbE7L9MJCmLbwq4+v+Yq7hm4j2jL6kSw8DI6I4H4a5KBYAa34tmN2SzcwYA4NQ/twGtGXRUZjpFH3KvPUL/8W7LLgTbjp0F1TAxKXO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746765; c=relaxed/simple; bh=n8GgQxWkHVm/KdGafTPsIHfxiB0IxGBQX22H0tLv/oc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=k+NINrFfFG4F1g5pEG0gHpu7xlsuPQWoQdohew7mlnyQnSFUSSy9Kt5wRw+7K51si2eVKDVpt8/XO/F6svciNl4VOlCXoKa80MMxEOKk/Gh+UHmtbZmH9VIxwA9v7ycxabv4AEzl4FBo0imdDmPmQ/r9FP0GkKvbcuIpq3AUmBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=BUD9EGMP; arc=none smtp.client-ip=209.85.128.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="BUD9EGMP" Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-49548aebcd8so39532575e9.3 for ; Wed, 22 Jul 2026 11:59:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746756; x=1785351556; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=MJ1SD6Cu8vcIcr/kUuDg+vlUw58q6JfkUpXNh8eryGw=; b=BUD9EGMP/1U+a0nwK74clxQtwB3Fm+sF+M1EXzUJE5EBTXRCP8t/YTnJJL2fiL3DaU RGtm0KUneztislxIlc1Q+0tEucU5M0v56RHyBl0zuvRRuv1WDOINASAaw1ePnYxBHh2X yzHHdOgc6q9ZyWd+T7b9/jdCU/okasOSi02zqUIpGd4dYSA3cCJIHJPMroCbo2P/2CKF M+Z0bF0YoyByOp9jyResJ4jJUtJN3fultJElB5c6/k7p3M1BBRC3mQ0YUAjPYFBBFDOj QxCoC9mCOrsuI67i0dUaZbkkjIBb3auJio1hYn8hzbdhmR6cz2WgS5QfFyeXrYVixWba sk/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746756; x=1785351556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=MJ1SD6Cu8vcIcr/kUuDg+vlUw58q6JfkUpXNh8eryGw=; b=YSVkqk5qFcB6kFDE/YxJJ3WxsKgBV1NDBjqHqN51/+GabprfyPDim2+GzMo/zF+aMM Gh5VyZcEDgp4V5piNWWKzN9GcsHwelY5bQ/Bn378J7lT8ADLOqgrZoXUcgI0mdoWpW13 Y6hwpp+o4Ua85rfi7bWr7h1Iri8WOd6rUU+7fslrU5fdODWp9X4nI9LNASVY5MsNRFY2 AegQUzFWJNbZUJ3v6FZVn+6KzbMte43pvKsCXXfQJGFe2NL41hvuY7VyCAHfrP3ivU4w pwz++ndIQCtnXVaj4CpYG1DV9xV67OlbbtUVVWFhe1vt7mJpc7djMEC+ZGTXtp89tqZB vRsw== X-Forwarded-Encrypted: i=1; AHgh+Ro/y2637cuxtAgSDD7cOfCUKXHUX/q42SFd9KrzgWJ6sPuDMEWOoK/ppR7rjM/kDTckGl1a1idUgiidwJY=@vger.kernel.org X-Gm-Message-State: AOJu0YxHB02lLetCQp1SaRcC30xTFGQfY8ZnG3+EhfPqo24BUmtzOyKI goFlBnJ6tpEzzP7weIJ5zMPahGGpqJc6jE8WiWb8YjQAWBUKG2XAwuiajNVpnafPFQ== X-Gm-Gg: AR+sD10BZhHlASl6QyPspZjlDAfWO+WLY2wbjvclZMwmFlA4bI71u3XBf/FxadAd4Kc ZZPI1QevAdgXAqaCDxQwZ5fS74laoISW4VJkinB1kZODta6aXCqihStVR96PQaVO2vcGa4Glfrv nXL9SwNuZR8VFMzDR9GuwieRulGT425IsVMJu+zD6i+1UggUypgEW/0UfVDA9V4IrTxNr8UL5rB yt7SLh7Q0ABXbBlc9tKpnEpj1TH96VGbBBOrxZswvnrDtFozldH4aSUiM6EixgXg4CtYz4LyHem 9QFjd6FmZmnfCJFHekhCvGnXRL77lA9AhJb0d+5uGoU3zEtBsj8bfXYCaD9emLlHZDjYphdPRxY bzy0W7aZiMdsuJnNKgtMi11iUvcP6xtpt9X1PfN5aU6Mq8vdEAlDldMGlPaNX5+Dk83Reg0GzbB m72bJwSRE0rsKW9fIaWR4zYPfB+YgQoiTFS/4rDEr1Aovr/B8= X-Received: by 2002:a05:600c:4f92:b0:495:f31:7340 with SMTP id 5b1f17b1804b1-4954a3d06e4mr270795875e9.5.1784746756410; Wed, 22 Jul 2026 11:59:16 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:15 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 8/9] md/raid1,raid10: set IO_BLOCKED in case of BLK_STS_P2PDMA Date: Wed, 22 Jul 2026 20:58:40 +0200 Message-ID: <20260722185841.449934-9-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Logan Gunthorpe A read that fails with BLK_STS_P2PDMA cannot succeed against that member, so mark the leg IO_BLOCKED and let the retry redirect elsewhere. Skip the read-error machinery: there is nothing on the medium to fix, fix_read_error()'s probe reads into host pages and would "succeed", and charging the read-error budget would evict a healthy member under a P2P read workload. FailFast eviction is skipped for the same reason -- the request never reached the wire. Fixes: 02666132403a ("md: propagate BLK_FEAT_PCI_P2PDMA from member devices= to RAID device") Assisted-by: Claude:claude-fable-5 Signed-off-by: Logan Gunthorpe Co-developed-by: Mykola Marzhan Signed-off-by: Mykola Marzhan --- drivers/md/raid1.c | 4 +++- drivers/md/raid10.c | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 917d694ef401..76a1426e64f0 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2658,6 +2658,8 @@ static void handle_read_error(struct r1conf *conf, st= ruct r1bio *r1_bio) { struct md_rdev *rdev =3D conf->mirrors[r1_bio->read_disk].rdev; struct bio *bio =3D r1_bio->bios[r1_bio->read_disk]; + /* evaluate before the bio_put() below */ + bool p2pdma_error =3D bio->bi_status =3D=3D BLK_STS_P2PDMA; struct mddev *mddev =3D conf->mddev; sector_t sector; =20 @@ -2675,7 +2677,7 @@ static void handle_read_error(struct r1conf *conf, st= ruct r1bio *r1_bio) * read error. This is all done synchronously while the array is * frozen. */ - if (mddev->ro) { + if (mddev->ro || p2pdma_error) { r1_bio->bios[r1_bio->read_disk] =3D IO_BLOCKED; } else if (test_bit(FailFast, &rdev->flags)) { md_error(mddev, rdev); diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 9045a3f02dae..e35aeac29f06 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -2848,6 +2848,7 @@ static void handle_read_error(struct mddev *mddev, st= ruct r10bio *r10_bio) { int slot =3D r10_bio->read_slot; struct bio *bio; + bool p2pdma_error; struct r10conf *conf =3D mddev->private; struct md_rdev *rdev =3D r10_bio->devs[slot].rdev; =20 @@ -2860,10 +2861,12 @@ static void handle_read_error(struct mddev *mddev, = struct r10bio *r10_bio) * frozen. */ bio =3D r10_bio->devs[slot].bio; + /* evaluate before the bio_put() below */ + p2pdma_error =3D bio->bi_status =3D=3D BLK_STS_P2PDMA; bio_put(bio); r10_bio->devs[slot].bio =3D NULL; =20 - if (mddev->ro) + if (mddev->ro || p2pdma_error) r10_bio->devs[slot].bio =3D IO_BLOCKED; else if (!test_bit(FailFast, &rdev->flags)) { freeze_array(conf, 1); --=20 2.52.0 From nobody Fri Jul 24 22:55:03 2026 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49DF9442B2F for ; Wed, 22 Jul 2026 18:59:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746768; cv=none; b=rHxJbKINC7HU+W0UuRndGePsApyg6IamPmX2wVGm4K83QSTPH5uBSvsLiOGvMQimO7BLikGJlG9ybuvq3QtTIvKSlvjNYQqx0SN/THFiv4IUjAKTcY/1GixCfcwGXW5pfEN1tEZ3uyvVZ2caJtrt79xZQxQgsb++fGH3o+Km8yg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784746768; c=relaxed/simple; bh=PFyLWHE9pTJ4175QPKDcNHyfOtGQRFYQNlkqnpPL1lc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PkAiM3M28JEd944m3uQVQUihydMiy9Qz8dTlp5Qurr/Plq4oxbZbVwa66R0YQyJf/Xb6nf+xDO/6378EgwudXMjQ71WZ0G/KYnqPBWhMFyO2chQ50BtzKnaE1Ms8R0VqxNLoYINj7+gWB+ROQ3ZeOW3Ezqlhpdvj56LynZ9dbVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io; spf=pass smtp.mailfrom=meshstor.io; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b=gFkA1PYc; arc=none smtp.client-ip=209.85.128.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meshstor.io Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meshstor.io Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meshstor.io header.i=@meshstor.io header.b="gFkA1PYc" Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-495437bb891so54357465e9.1 for ; Wed, 22 Jul 2026 11:59:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meshstor.io; s=google; t=1784746759; x=1785351559; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZyX4ma10pC/frHu+Y/NUZniwvhp9EvBkdSwrv0doV3k=; b=gFkA1PYcdgJBj7Vk0LpWIXQdy5xoIr7JDJ4V6i5w+Cvel3qWwN8RjqopjcOvvtLeMM uBqArxT30CfTjh0676/kjNn2ppFN731beeq9rT71vPCEeMveVlDaadsU54BS0g55Vd9i rAYDk044Y5AnWmYPGvtkJtk3GSJ8Yr8SRmpRQK1DkmZEcGTlkXrF5dLRlooBfwJI5eLH 6fsDprOXAb51C2TyF8cYKU5NK5+jJi4SjQq5AYe0ragqMg7YykVIzzSw9P39yIym5uVr 1aK+008mvyyLPI8ftiReDk5SFbPVFZ2UnJ09yY1ZYjvjB1rVitvOviOPkHiza7yzEJeV WKRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784746759; x=1785351559; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZyX4ma10pC/frHu+Y/NUZniwvhp9EvBkdSwrv0doV3k=; b=HeM4ezo0GQR+zCTf1D3l+0OvqbKTsMYBckLHWS/M9zfZetm+ab+lOOiDaNdl8bK6Gi xiwdvkcEPP5+u0koQmOIfBUkrbFnOFXzaHItl8eOKZcLiyU7ylx1ubPydGNioOMsJhl7 Qr/A7EV0N5my2jPaHnhLH1XgGcV5zb78xBg5dCmgHNCnh1IwzXYmolNWvZJH+nL1T8Tf MqyA39OrD/I3tibRIiP0TVdtx4ngYUO5A2Pc2veNwW6TES9cRnxdqc95JAABfyKL3MNv 6dWWUS83qrUH1Q6lTiSLreKbRHvIHEjbqTAVj8UBM+4GYAe+KlAiJ20hniOLykRANc8W aFpA== X-Forwarded-Encrypted: i=1; AHgh+RrWgQv0u791S7K4CyTf8Gfsz2/nTFewXWlbbamUSRoqvBXoOG+nBg4s7FenfFtty3tDtC7GrCtcq5qGY1k=@vger.kernel.org X-Gm-Message-State: AOJu0YyEpkc7yWujOas+tsE3y1XyLwBRIE1QZ4fVJD4jgi/Kzj8GbUOC x1RRuqN6d+cn/4U4h7XhNFymvCrdA6/W4HnEbL03FoRiDKzve/pamKnvzvxdUs4lPg== X-Gm-Gg: AR+sD10uTtl16xHTERkAiJsNCnSYwYPQJdgNuggIk46mmYfHRJ+W1nCyuXqKMmxavlR c+UBG9tcWjIl82hI7XvaVeNpKn8bCpqZDC7JLUzxx9q/MyJHD7mhX8aNnX9sON8wnc0rXaj5luS vB9259irY0yUrK8wQ0eytWvv3VezCAeWWuY3P5aqtED1ZvrX6HHRsEBh3+gkwYSYOP7YOHkdyyA 2XJ6IOCSXsJ6J8cRZgM+J304Vn7LED8PNtquHRS8rWUEduGg7jGTfX/CUNCJm96SL/z8tcE7woo ClidGKlJBvQPBwFx3sBdnlpuHpGMomtx/DJry+kN2evDHqDbQ3u7sNI9h/x5BRnuMsMZhTg6TgH qK+/P+MbuO+qMXIAyUFKJGJnE3Aml1m5XkcEc8pup2QabAeTJvAowAluKDJxbnH2RTGka4xNk2v KRRPhdl7Y/EKPpP1qyROkHozbfbiDiwp3zfvCM7UZ8b63ZWFE= X-Received: by 2002:a05:600c:6610:b0:492:6f5c:fd8c with SMTP id 5b1f17b1804b1-4956a511b49mr53704175e9.15.1784746758682; Wed, 22 Jul 2026 11:59:18 -0700 (PDT) Received: from ai-pc ([46.172.241.174]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-495730353d9sm13740055e9.12.2026.07.22.11.59.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Jul 2026 11:59:18 -0700 (PDT) From: Mykola Marzhan To: Jens Axboe , Song Liu , Yu Kuai , Keith Busch , Christoph Hellwig , Sagi Grimberg , linux-block@vger.kernel.org, linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org Cc: Li Nan , Xiao Ni , Guoqing Jiang , Leon Romanovsky , Jason Gunthorpe , Kiran Kumar Modukuri , Chaitanya Kulkarni , Logan Gunthorpe , Bjorn Helgaas , Shivaji Kant , Pranjal Shrivastava , Henrique Carvalho , linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH v4 9/9] nvme-rdma: return BLK_STS_P2PDMA for unsupported P2P transfers Date: Wed, 22 Jul 2026 20:58:41 +0200 Message-ID: <20260722185841.449934-10-mykola@meshstor.io> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260722185841.449934-1-mykola@meshstor.io> References: <20260722185841.449934-1-mykola@meshstor.io> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When the PCIe topology cannot route a P2P transfer between two devices, the DMA layer fails the mapping with -EREMOTEIO, but ib_dma_map_sg() returns 0 and the error is lost. nvme-rdma then fails the I/O with a retryable status, so multipath requeues it forever and a single path wastes its whole retry budget on an I/O that can never succeed. Map the data and metadata scatterlists with ib_dma_map_sgtable_attrs(), which preserves the error, and translate -EREMOTEIO to the new BLK_STS_P2PDMA status so the request fails fast instead of being retried. While at it, call nvme_start_request() only after mapping succeeds and ratelimit the map-failure message. Fixes: 23528aa3320a ("nvme: enable PCI P2PDMA support for RDMA transport") Cc: stable@vger.kernel.org # v7.1 Assisted-by: Claude:claude-fable-5 Signed-off-by: Mykola Marzhan --- drivers/nvme/host/rdma.c | 38 ++++++++++++++++++++++---------------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index 6909e3542794..2f50509a7a61 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -1469,6 +1469,7 @@ static int nvme_rdma_dma_map_req(struct ib_device *ib= dev, struct request *rq, int *count, int *pi_count) { struct nvme_rdma_request *req =3D blk_mq_rq_to_pdu(rq); + struct sg_table sgt; int ret; =20 req->data_sgl.sg_table.sgl =3D (struct scatterlist *)(req + 1); @@ -1480,12 +1481,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *= ibdev, struct request *rq, =20 req->data_sgl.nents =3D blk_rq_map_sg(rq, req->data_sgl.sg_table.sgl); =20 - *count =3D ib_dma_map_sg(ibdev, req->data_sgl.sg_table.sgl, - req->data_sgl.nents, rq_dma_dir(rq)); - if (unlikely(*count <=3D 0)) { - ret =3D -EIO; + sgt =3D (struct sg_table) { + .sgl =3D req->data_sgl.sg_table.sgl, + .orig_nents =3D req->data_sgl.nents, + }; + ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_table; - } + *count =3D sgt.nents; =20 if (blk_integrity_rq(rq)) { req->metadata_sgl->sg_table.sgl =3D @@ -1501,14 +1504,14 @@ static int nvme_rdma_dma_map_req(struct ib_device *= ibdev, struct request *rq, =20 req->metadata_sgl->nents =3D blk_rq_map_integrity_sg(rq, req->metadata_sgl->sg_table.sgl); - *pi_count =3D ib_dma_map_sg(ibdev, - req->metadata_sgl->sg_table.sgl, - req->metadata_sgl->nents, - rq_dma_dir(rq)); - if (unlikely(*pi_count <=3D 0)) { - ret =3D -EIO; + sgt =3D (struct sg_table) { + .sgl =3D req->metadata_sgl->sg_table.sgl, + .orig_nents =3D req->metadata_sgl->nents, + }; + ret =3D ib_dma_map_sgtable_attrs(ibdev, &sgt, rq_dma_dir(rq), 0); + if (unlikely(ret)) goto out_free_pi_table; - } + *pi_count =3D sgt.nents; } =20 return 0; @@ -2026,8 +2029,6 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_= hw_ctx *hctx, if (ret) goto unmap_qe; =20 - nvme_start_request(rq); - if (IS_ENABLED(CONFIG_BLK_DEV_INTEGRITY) && queue->pi_support && (c->common.opcode =3D=3D nvme_cmd_write || @@ -2039,11 +2040,13 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_m= q_hw_ctx *hctx, =20 err =3D nvme_rdma_map_data(queue, rq, c); if (unlikely(err < 0)) { - dev_err(queue->ctrl->ctrl.device, - "Failed to map data (%d)\n", err); + dev_err_ratelimited(queue->ctrl->ctrl.device, + "Failed to map data (%d)\n", err); goto err; } =20 + nvme_start_request(rq); + sqe->cqe.done =3D nvme_rdma_send_done; =20 ib_dma_sync_single_for_device(dev, sqe->dma, @@ -2063,6 +2066,9 @@ static blk_status_t nvme_rdma_queue_rq(struct blk_mq_= hw_ctx *hctx, ret =3D nvme_host_path_error(rq); else if (err =3D=3D -ENOMEM || err =3D=3D -EAGAIN) ret =3D BLK_STS_RESOURCE; + /* Peer memory unreachable from this device: don't retry. */ + else if (err =3D=3D -EREMOTEIO) + ret =3D BLK_STS_P2PDMA; else ret =3D BLK_STS_IOERR; nvme_cleanup_cmd(rq); --=20 2.52.0