From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3D98C433FE for ; Tue, 28 Dec 2021 08:02:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235674AbhL1ICr (ORCPT ); Tue, 28 Dec 2021 03:02:47 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:10151 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235528AbhL1ICZ (ORCPT ); Tue, 28 Dec 2021 03:02:25 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AWxc+CqrtKQLbHi8b6Xft34jZtSxeBmJGZBIvgKr?= =?us-ascii?q?LsJaIsI5as4F+vmQdWGiEOKzYYmakLt91bNyx9UtVupGDzNZlQVRl+3s9QiMRo?= =?us-ascii?q?6IpJ/zDcB6oYHn6wu4v7a5fx5xHLIGGdajYd1eEzvuWGuWn/SkUOZ2gHOKmUbe?= =?us-ascii?q?eYHApHGeIdQ964f5ds79g6mJXqYjha++9kYuaT/z3YDdJ6RYtWo4nw/7rRCdUg?= =?us-ascii?q?RjHkGhwUmrSyhx8lAS2e3E9VPrzLEwqRpfyatE88uWSH44vwFwll1418SvBCvv?= =?us-ascii?q?9+lr6WkYMBLDPPwmSkWcQUK+n6vRAjnVqlP9la7xHMgEK49mKt4kZJNFlsZ2iS?= =?us-ascii?q?QYrP6TKsOoAURhECDw4NqpDkFPCCSHm4JLOkBGfKhMAxN0rVinaJ7Yw4P56CHt?= =?us-ascii?q?V8voYMD0lYRWKhubwy7W+IsF+l8YxPcuxZNtHkn5lxDDdS/0hRPjrR6TD49BH0?= =?us-ascii?q?TEoi8ZBNfbDbtUUaHxkaxGoSxlOJVoWCJs4k8+om3Dgfjweo1WQzYIz7m/V5A9?= =?us-ascii?q?8yr7gNJzSYNPibcxVl1yfoGbu+Xr4DhATcteYzFKt93iogeTPtSXlWY4THfuz8?= =?us-ascii?q?fsCqFmSwHEDTR4bT122pdGnhUOkHdFSMUoZ/mwpt6da3EiqSMTtGge0pXesoBE?= =?us-ascii?q?RQZxTHvc85QXLzbDbiy6bCWcsXD9McNFgv8ZeeNCA/jdlhPuwXXo27uLTEinbq?= =?us-ascii?q?9+pQfqJEXB9BQc/ieUsFFBtDwHfnbwO?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3Addz9L65rcqxeHRTzVwPXwCzXdLJyesId70hD?= =?us-ascii?q?6qhwISY6TiX+rbHJoB17726MtN9/YhEdcLy7VJVoBEmskKKdgrNhWotKPjOW21?= =?us-ascii?q?dARbsKheCJrgEIWReOktK1vp0AT0ERMrLN5CBB/KTHCReDYqsd6ejC4Ka1nv3f?= =?us-ascii?q?0nsoaQlrbptr5wB/Bh3zKDwMeCB2QYo+CIGH5tdK4x6peXEsZMy9AXUfG8fZod?= =?us-ascii?q?mjruOdXTc2Qw4g9BKVjS6lrJrzEx2j1B8YVD9VhZcOmFK16zDE2g=3D=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657411" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD05.g08.fujitsu.local (unknown [10.167.33.204]) by cn.fujitsu.com (Postfix) with ESMTP id 193974D15A21; Tue, 28 Dec 2021 16:01:47 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD05.g08.fujitsu.local (10.167.33.204) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:48 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:44 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 01/10] RDMA: mr: Introduce is_pmem Date: Tue, 28 Dec 2021 16:07:08 +0800 Message-ID: <20211228080717.10666-2-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 193974D15A21.AE5CF X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We can use it to indicate whether the registering mr is associated with a pmem/nvdimm or not. Currently, we only assign it in rxe driver, for other device/drivers, they should implement it if needed. RDMA FLUSH will support the persistence feature for a pmem/nvdimm. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 47 ++++++++++++++++++++++++++++++ include/rdma/ib_verbs.h | 1 + 2 files changed, 48 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe= /rxe_mr.c index 7c4cd19a9db2..bcd5e7afa475 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -162,6 +162,50 @@ void rxe_mr_init_dma(struct rxe_pd *pd, int access, st= ruct rxe_mr *mr) mr->type =3D IB_MR_TYPE_DMA; } =20 +// XXX: the logic is similar with mm/memory-failure.c +static bool page_in_dev_pagemap(struct page *page) +{ + unsigned long pfn; + struct page *p; + struct dev_pagemap *pgmap =3D NULL; + + pfn =3D page_to_pfn(page); + if (!pfn) { + pr_err("no such pfn for page %p\n", page); + return false; + } + + p =3D pfn_to_online_page(pfn); + if (!p) { + if (pfn_valid(pfn)) { + pgmap =3D get_dev_pagemap(pfn, NULL); + if (pgmap) + put_dev_pagemap(pgmap); + } + } + + return !!pgmap; +} + +static bool iova_in_pmem(struct rxe_mr *mr, u64 iova, int length) +{ + struct page *page =3D NULL; + char *vaddr =3D iova_to_vaddr(mr, iova, length); + + if (!vaddr) { + pr_err("not a valid iova %llu\n", iova); + return false; + } + + page =3D virt_to_page(vaddr); + if (!page) { + pr_err("no such page for vaddr %p\n", vaddr); + return false; + } + + return page_in_dev_pagemap(page); +} + int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova, int access, struct rxe_mr *mr) { @@ -236,6 +280,9 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 = length, u64 iova, set->va =3D start; set->offset =3D ib_umem_offset(umem); =20 + // iova_in_pmem must be called after set is updated + mr->ibmr.is_pmem =3D iova_in_pmem(mr, iova, length); + return 0; =20 err_release_umem: diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 6e9ad656ecb7..822ebb3425dc 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1807,6 +1807,7 @@ struct ib_mr { unsigned int page_size; enum ib_mr_type type; bool need_inval; + bool is_pmem; union { struct ib_uobject *uobject; /* user */ struct list_head qp_entry; /* FR */ --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B59BEC433F5 for ; Tue, 28 Dec 2021 08:02:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235490AbhL1ICU (ORCPT ); Tue, 28 Dec 2021 03:02:20 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:10151 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235464AbhL1ICS (ORCPT ); Tue, 28 Dec 2021 03:02:18 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AdK9/rKwvNA4xbTLWkDh6t+dpxyrEfRIJ4+MujC/?= =?us-ascii?q?XYbTApDor1mMDmGEZWGiPPqyNYGuhKItzat628UwA6JPQm4RnHQtv/xmBbVoQ9?= =?us-ascii?q?5OdWo7xwmQcns+qBpSaChohtq3yU/GYRCwPZiKa9kfF3oTJ9yEmjPjSHOqkUYY?= =?us-ascii?q?oBwgqLeNaYHZ44f5cs75h6mJYqYDR7zKl4bsekeWGULOW82Ic3lYv1k62gEgHU?= =?us-ascii?q?MIeF98vlgdWifhj5DcynpSOZX4VDfnZw3DQGuG4EgMmLtsvwo1V/kuBl/ssIti?= =?us-ascii?q?j1LjmcEwWWaOUNg+L4pZUc/H6xEEc+WppieBmXBYfQR4/ZzGhm9FjyNRPtJW2Y?= =?us-ascii?q?Qk0PKzQg/lbWB5de817FfQfqeGafiTn4KR/yGWDKRMA2c5GFlk7NJcD/eB3GWx?= =?us-ascii?q?m+vkRKTRLZReG78qk0bCpW+s23px7BMbuNYIb/HpnyFnxCfcvR5/cTqPS6NlX9?= =?us-ascii?q?Dctj99DHLDVYM9xQT5ucxnBYxRJNX8XFZshkebujX76GxVcpVWTjak6+W7eyEp?= =?us-ascii?q?2yreFGNPVc8aNQ8F9mFiZqmPPuW/+B3kyMdabzjGF2nSyh+POlGXwX4d6PLm58?= =?us-ascii?q?ON6xV6e3GoeDDUIWlah5/q0kEizX5RYMUN80i4vq7UisVanS9DVQRK1ujiHswQ?= =?us-ascii?q?aVt4WFPc1gCmPxaX88QeUHmVCRTcpVTCMnKfaXhRzjhnQwYyvXmcp7dWopbum3?= =?us-ascii?q?u/8hVuP1eI9dDRqifc4cDY4?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AI8ttva4u5M/IRmAdTAPXwCzXdLJyesId70hD?= =?us-ascii?q?6qhwISY6TiX+rbHJoB17726MtN9/YhEdcLy7VJVoBEmskKKdgrNhWotKPjOW21?= =?us-ascii?q?dARbsKheCJrgEIWReOktK1vp0AT0ERMrLN5CBB/KTHCReDYqsd6ejC4Ka1nv3f?= =?us-ascii?q?0nsoaQlrbptr5wB/Bh3zKDwMeCB2QYo+CIGH5tdK4x6peXEsZMy9AXUfG8fZod?= =?us-ascii?q?mjruOdXTc2Qw4g9BKVjS6lrJrzEx2j1B8YVD9VhZcOmFK16zDE2g=3D=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657409" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD06.g08.fujitsu.local (unknown [10.167.33.206]) by cn.fujitsu.com (Postfix) with ESMTP id 06CDB4D15A22; Tue, 28 Dec 2021 16:01:48 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD06.g08.fujitsu.local (10.167.33.206) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:46 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:45 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 02/10] RDMA: Allow registering MR with flush access flags Date: Tue, 28 Dec 2021 16:07:09 +0800 Message-ID: <20211228080717.10666-3-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 06CDB4D15A22.AF473 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Users can use ibv_reg_mr(3) to register flush access flags. Signed-off-by: Li Zhijian --- include/rdma/ib_verbs.h | 6 +++++- include/uapi/rdma/ib_user_ioctl_verbs.h | 2 ++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 822ebb3425dc..f04d66539879 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1444,10 +1444,14 @@ enum ib_access_flags { IB_ACCESS_ON_DEMAND =3D IB_UVERBS_ACCESS_ON_DEMAND, IB_ACCESS_HUGETLB =3D IB_UVERBS_ACCESS_HUGETLB, IB_ACCESS_RELAXED_ORDERING =3D IB_UVERBS_ACCESS_RELAXED_ORDERING, + IB_ACCESS_FLUSH_GLOBAL_VISIBILITY =3D IB_UVERBS_ACCESS_FLUSH_GLOBAL_VISIB= LITY, + IB_ACCESS_FLUSH_PERSISTENT =3D IB_UVERBS_ACCESS_FLUSH_PERSISTENT, =20 + IB_ACCESS_FLUSH =3D IB_ACCESS_FLUSH_GLOBAL_VISIBILITY | + IB_ACCESS_FLUSH_PERSISTENT, IB_ACCESS_OPTIONAL =3D IB_UVERBS_ACCESS_OPTIONAL_RANGE, IB_ACCESS_SUPPORTED =3D - ((IB_ACCESS_HUGETLB << 1) - 1) | IB_ACCESS_OPTIONAL, + ((IB_ACCESS_FLUSH_PERSISTENT << 1) - 1) | IB_ACCESS_OPTIONAL, }; =20 /* diff --git a/include/uapi/rdma/ib_user_ioctl_verbs.h b/include/uapi/rdma/ib= _user_ioctl_verbs.h index 3072e5d6b692..4763d3e3ea16 100644 --- a/include/uapi/rdma/ib_user_ioctl_verbs.h +++ b/include/uapi/rdma/ib_user_ioctl_verbs.h @@ -57,6 +57,8 @@ enum ib_uverbs_access_flags { IB_UVERBS_ACCESS_ZERO_BASED =3D 1 << 5, IB_UVERBS_ACCESS_ON_DEMAND =3D 1 << 6, IB_UVERBS_ACCESS_HUGETLB =3D 1 << 7, + IB_UVERBS_ACCESS_FLUSH_GLOBAL_VISIBLITY =3D 1 << 8, + IB_UVERBS_ACCESS_FLUSH_PERSISTENT =3D 1 << 9, =20 IB_UVERBS_ACCESS_RELAXED_ORDERING =3D IB_UVERBS_ACCESS_OPTIONAL_FIRST, IB_UVERBS_ACCESS_OPTIONAL_RANGE =3D --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83E66C433EF for ; Tue, 28 Dec 2021 08:02:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235615AbhL1ICj (ORCPT ); Tue, 28 Dec 2021 03:02:39 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47591 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235464AbhL1ICW (ORCPT ); Tue, 28 Dec 2021 03:02:22 -0500 IronPort-Data: =?us-ascii?q?A9a23=3A/yqbtaoh+6phOnsktzIlST90qlFeBmJGZBIvgKr?= =?us-ascii?q?LsJaIsI5as4F+vjYZWzuHMvnfMGv9c4oia4nn8xxUvZ+DnNNhQVA5/C9nQiMRo?= =?us-ascii?q?6IpJ/zDcB6oYHn6wu4v7a5fx5xHLIGGdajYd1eEzvuWGuWn/SkUOZ2gHOKmUbe?= =?us-ascii?q?eYHApHGeIdQ964f5ds79g6mJXqYjha++9kYuaT/z3YDdJ6RYtWo4nw/7rRCdUg?= =?us-ascii?q?RjHkGhwUmrSyhx8lAS2e3E9VPrzLEwqRpfyatE88uWSH44vwFwll1418SvBCvv?= =?us-ascii?q?9+lr6WkYMBLDPPwmSkWcQUK+n6vRAjnVqlP9la7xHMgEK49mKt4kZJNFlsZ2iS?= =?us-ascii?q?QYrP6TKsOoAURhECDw4NqpDkFPCCSHm4JLOkBGfKRMAxN0rVinaJ7Yw4P56CHt?= =?us-ascii?q?V8voYMD0lYRWKhubwy7W+IsF+l8YxPcuxZNtHkn5lxDDdS/0hRPjrR6TD49BH0?= =?us-ascii?q?TEoi8ZBNfbDbtUUaHxkaxGoSxlOJVoWCJs4k8+om3Dgfjweo1WQzYIz7m/V5A9?= =?us-ascii?q?8yr7gNJzSYNPibcxVl1yfoGbu+Xr4DhATcteYzFKt93iogeTPtSXlWY4THfuz8?= =?us-ascii?q?fsCqFmSwHEDTR4bT122pdGnhUOkHdFSMUoZ/mwpt6da3EiqSMTtGge0pXesoBE?= =?us-ascii?q?RQZxTHvc85QXLzbDbiy6bCWcsXD9McNFgv8ZeeNCA/jdlhPuwXXo27uLTEinbq?= =?us-ascii?q?9+pQfqJEXB9BQc/ieUsFFdtDwHfnbwO?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AJ1d0dKCLHqZc767lHelI55DYdb4zR+YMi2TC?= =?us-ascii?q?1yhKJyC9Ffbo8fxG/c5rrCMc5wxwZJhNo7y90ey7MBbhHP1OkO4s1NWZLWrbUQ?= =?us-ascii?q?KTRekIh+bfKn/baknDH4VmtJuIHZIQNDSJNykZsS/l2njEL/8QhMmA7LuzhfrT?= =?us-ascii?q?i1NkTQRRYalm6AtjYzzraXFedU1XA4YjDpqA6o5irzqkQ34eacO2HT0rRO7Gzu?= =?us-ascii?q?e77q7OUFoXAQI98gmSgXeN4L7+KRKR2RATSHdu7N4ZgBD4rzA=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657412" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD04.g08.fujitsu.local (unknown [10.167.33.201]) by cn.fujitsu.com (Postfix) with ESMTP id B95204D15A23; Tue, 28 Dec 2021 16:01:48 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD04.g08.fujitsu.local (10.167.33.201) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:48 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:46 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 03/10] RDMA/rxe: Allow registering FLUSH flags for supported device only Date: Tue, 28 Dec 2021 16:07:10 +0800 Message-ID: <20211228080717.10666-4-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: B95204D15A23.AE83B X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Device should enable IB_DEVICE_RDMA_FLUSH capability if it want to support RDMA FLUSH. Signed-off-by: Li Zhijian --- include/rdma/ib_verbs.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index f04d66539879..51d58b641201 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -291,6 +291,7 @@ enum ib_device_cap_flags { /* The device supports padding incoming writes to cacheline. */ IB_DEVICE_PCI_WRITE_END_PADDING =3D (1ULL << 36), IB_DEVICE_ALLOW_USER_UNREG =3D (1ULL << 37), + IB_DEVICE_RDMA_FLUSH =3D (1ULL << 38), }; =20 enum ib_atomic_cap { @@ -4319,6 +4320,10 @@ static inline int ib_check_mr_access(struct ib_devic= e *ib_dev, if (flags & IB_ACCESS_ON_DEMAND && !(ib_dev->attrs.device_cap_flags & IB_DEVICE_ON_DEMAND_PAGING)) return -EINVAL; + + if (flags & IB_ACCESS_FLUSH && + !(ib_dev->attrs.device_cap_flags & IB_DEVICE_RDMA_FLUSH)) + return -EINVAL; return 0; } =20 --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84BBDC433EF for ; Tue, 28 Dec 2021 08:02:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235610AbhL1ICa (ORCPT ); Tue, 28 Dec 2021 03:02:30 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:10151 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235494AbhL1ICV (ORCPT ); Tue, 28 Dec 2021 03:02:21 -0500 IronPort-Data: =?us-ascii?q?A9a23=3Auzrgya4WU07Bpw2qGEIyHwxRtGbGchMFZxGqfqr?= =?us-ascii?q?LsXjdYENS1zMAyTAXUDiGMvaJNjb1c4h3YYW19xwEvsPTzoRjTVY5pCpnJ55og?= =?us-ascii?q?ZCbXIzGdC8cHM8zwvXrFRsht4NHAjX5BJhcokT0+1H9YtANkVEmjfvRH+CmWLa?= =?us-ascii?q?dUsxMbVQMpBkJ2EsLd9ER0tYAbeiRW2thiPuqyyHtEAbNNw1cbgr435m+RCZH5?= =?us-ascii?q?5wejt+3UmsWPpintHeG/5Uc4Ql2yauZdxMUSaEMdgK2qnqq8V23wo/Z109F5tK?= =?us-ascii?q?NmbC9fFAIQ6LJIE6FjX8+t6qK20AE/3JtlP1gcqd0hUR/0l1lm/h1ycdNtJ6xQ?= =?us-ascii?q?AEBMLDOmfgGTl9TFCQW0ahuoeaWfyXg7ZXOp6HBWz62qxl0N2ksJYAR4P1wB2F?= =?us-ascii?q?W+NQXLTkMalaIgOfe6KCqSPt9hJ57dJHDM4YWu3UmxjbcZd4iQJnFTLrH48dV2?= =?us-ascii?q?jgYht1HAvvfIcEebFJHaBXfbhxPM1EYIJIjlfilhz/0dDgwgFaYoqwf4GXJygF?= =?us-ascii?q?1lr/3P7L9c9yDX8JTmm6euGPK/m2/CRYfXPSawz6C+XKEgPHOkSLyHokVEdWQ8?= =?us-ascii?q?v9snU3WxWEJDhASfUW0rOP/iUOkXd9bbUsO9UIGqak06VzuVNX4VjWmr3Oe+B0?= =?us-ascii?q?RQdxdF6s98g7l4qjb7C6LB2UcQ3hKYbQbWGUeLdAx/gbR2Yq3Wno06/vIIU9xP?= =?us-ascii?q?4y89VuaURX550dbNUfolTc43uQ=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AJ6kF2aOdY4ip78BcTiWjsMiBIKoaSvp037Eq?= =?us-ascii?q?v3oedfUzSL3/qynOpoVj6faaslYssR0b9exofZPwJE80lqQFhrX5X43SPzUO0V?= =?us-ascii?q?HAROoJgLcKgQeQfxEWndQ96U4PScdD4aXLfDpHZNjBkXSFOudl0N+a67qpmOub?= =?us-ascii?q?639sSDthY6Zm4xwRMHfhLmRGABlBGYEiFIeRou5Opz+bc3wRacihQlYfWeyrna?= =?us-ascii?q?ywqLvWJQ4BGwU86BSDyReh6LvBGRCe2RsEFxNjqI1SiVT4rw=3D=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657415" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD04.g08.fujitsu.local (unknown [10.167.33.201]) by cn.fujitsu.com (Postfix) with ESMTP id A464E4D15A24; Tue, 28 Dec 2021 16:01:49 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD04.g08.fujitsu.local (10.167.33.201) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:49 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:47 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 04/10] RDMA/rxe: Enable IB_DEVICE_RDMA_FLUSH for rxe device Date: Tue, 28 Dec 2021 16:07:11 +0800 Message-ID: <20211228080717.10666-5-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: A464E4D15A24.AF05D X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We will enable the RDMA FLUSH on rxe device later. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_param.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/= rxe/rxe_param.h index 918270e34a35..5c140f311aa2 100644 --- a/drivers/infiniband/sw/rxe/rxe_param.h +++ b/drivers/infiniband/sw/rxe/rxe_param.h @@ -53,7 +53,8 @@ enum rxe_device_param { | IB_DEVICE_ALLOW_USER_UNREG | IB_DEVICE_MEM_WINDOW | IB_DEVICE_MEM_WINDOW_TYPE_2A - | IB_DEVICE_MEM_WINDOW_TYPE_2B, + | IB_DEVICE_MEM_WINDOW_TYPE_2B + | IB_DEVICE_RDMA_FLUSH, RXE_MAX_SGE =3D 32, RXE_MAX_WQE_SIZE =3D sizeof(struct rxe_send_wqe) + sizeof(struct ib_sge) * RXE_MAX_SGE, --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DCCFC433EF for ; Tue, 28 Dec 2021 08:02:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235541AbhL1IC0 (ORCPT ); Tue, 28 Dec 2021 03:02:26 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47591 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235478AbhL1ICT (ORCPT ); Tue, 28 Dec 2021 03:02:19 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AhCRgkqJPaLKhLizxFE+Rj5QlxSXFcZb7ZxGrkP8?= =?us-ascii?q?bfHCxgjMh0DRSzWEdDWmOOK2DYDSnf4h+a4y0oB9SusLXx4NqS1BcGVNFFSwT8?= =?us-ascii?q?ZWfbTi6wuYcBwvLd4ubChsPA/w2MrEsF+hpCC+MzvuRGuK59yAlj/vYHuOU5NP?= =?us-ascii?q?sYUideyc1EU/Ntjozw4bVsqYw6TSIK1vlVeHa+qUzC3f5s9JACV/43orYwP9ZU?= =?us-ascii?q?FsejxtD1rA2TagjUFYzDBD5BrpHTU26ByOQroW5goeHq+j/ILGRpgs1/j8mDJW?= =?us-ascii?q?rj7T6blYXBLXVOGBiiFIPA+773EcE/Xd0j87XN9JFAatTozGIjdBwytREs7S+V?= =?us-ascii?q?AUoIrbR3u8aVnG0FgknZPwYpeCbeSfXXcu7iheun2HX6+92AUgsJooe+v56KW5?= =?us-ascii?q?L/P0cbjsKa3irlfO00qO5ELE03uwsKcDqOMUUvXQI5TXYBPApXp3FW6jM6vdYw?= =?us-ascii?q?T4vi8EIFvHbD+IdaCVibRTJZRJnOkoeF58/2uyvgxHXdzBfrnqWqLAx7myVyxZ?= =?us-ascii?q?+uJDkMNPPfdqObcNLn0qZryTN+GGRKhQQMNuUyRKD7HOgh+aJliT+MKoWFbul5?= =?us-ascii?q?rtpjUeVy2g7FhIbTx24rOO/h0r4XMhQQ2QQ+ywzve0o+EmiZsfyUgf+o3OeuBM?= =?us-ascii?q?YHd1KHIUS6g6C4rjV7h6UQGMNJgOtwvROWNQeHGRsjwHW2YiyQ2EHjVFcclrFn?= =?us-ascii?q?p/8kN94EXFORYPaWRI5cA=3D=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3ACvgY86xaWGnT48rM8r8SKrPwyL1zdoMgy1kn?= =?us-ascii?q?xilNoRw8SKKlfqeV7ZImPH7P+U8ssR4b+exoVJPtfZqYz+8R3WBzB8bEYOCFgh?= =?us-ascii?q?rKEGgK1+KLqFeMJ8S9zJ846U4KSclD4bPLYmSS9fyKgjVQDexQveWvweS5g/vE?= =?us-ascii?q?1XdxQUVPY6Fk1Q1wDQGWCSRNNXJ7LKt8BJyB/dBGujblXXwWa/6wDn4DU/OGiM?= =?us-ascii?q?bMkPvdEGQ7Li9i+A+Tlimp9bK/NxCZ2y0VWzRJzaxn0UWtqX2A2pme?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657407" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD05.g08.fujitsu.local (unknown [10.167.33.204]) by cn.fujitsu.com (Postfix) with ESMTP id 70B9C4D15A26; Tue, 28 Dec 2021 16:01:50 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD05.g08.fujitsu.local (10.167.33.204) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:51 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:47 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 05/10] RDMA/rxe: Allow registering persistent flag for pmem MR only Date: Tue, 28 Dec 2021 16:07:12 +0800 Message-ID: <20211228080717.10666-6-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 70B9C4D15A26.AF226 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Memory region should support 2 placement types: IB_ACCESS_FLUSH_PERSISTENT and IB_ACCESS_FLUSH_GLOBAL_VISIBILITY, and only pmem/nvdimm has ability to persist data(IB_ACCESS_FLUSH_PERSISTENT). Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_mr.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe= /rxe_mr.c index bcd5e7afa475..21616d058f29 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -206,6 +206,11 @@ static bool iova_in_pmem(struct rxe_mr *mr, u64 iova, = int length) return page_in_dev_pagemap(page); } =20 +static bool ib_check_flush_access_flags(struct ib_mr *mr, u32 flags) +{ + return mr->is_pmem || !(flags & IB_ACCESS_FLUSH_PERSISTENT); +} + int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64 length, u64 iova, int access, struct rxe_mr *mr) { @@ -282,6 +287,13 @@ int rxe_mr_init_user(struct rxe_pd *pd, u64 start, u64= length, u64 iova, =20 // iova_in_pmem must be called after set is updated mr->ibmr.is_pmem =3D iova_in_pmem(mr, iova, length); + if (!ib_check_flush_access_flags(&mr->ibmr, access)) { + pr_err("Cannot set IB_ACCESS_FLUSH_PERSISTENT for non-pmem memory\n"); + mr->state =3D RXE_MR_STATE_INVALID; + mr->umem =3D NULL; + err =3D -EINVAL; + goto err_release_umem; + } =20 return 0; =20 --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E84E8C433FE for ; Tue, 28 Dec 2021 08:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235587AbhL1ICb (ORCPT ); Tue, 28 Dec 2021 03:02:31 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47599 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235495AbhL1ICW (ORCPT ); Tue, 28 Dec 2021 03:02:22 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AiGnAeqAlo2CqFRVW/3fiw5YqxClBgxIJ4g17XOL?= =?us-ascii?q?fUFO9gjgj0TYOymsaXmuEP/vYYmCmfo9/b9yx8UhQu5WAx9UxeLYW3SszFioV8?= =?us-ascii?q?6IpJjg4wn/YZnrUdouaJK5ex512huLocYZkHhcwmj/3auK49CMmhfnRLlbBILW?= =?us-ascii?q?s1h5ZFFYMpBgJ2UoLd94R2uaEsPDha++/kYqaT/73ZDdJ7wVJ3lc8sMpvnv/AU?= =?us-ascii?q?MPa41v0tnRmDRxCUcS3e3M9VPrzLonpR5f0rxU9IwK0ewrD5OnREmLx9BFrBM6?= =?us-ascii?q?nk6rgbwsBRbu60Qqm0yIQAvb9xEMZ4HFaPqUTbZLwbW9TiieJntJwwdNlu4GyS?= =?us-ascii?q?BsyI+vHn+F1vxxwSngvY/AZpOabSZS4mYnJp6HcSFP22/hnFloxO40A9854BGh?= =?us-ascii?q?P8boTLzVlRgKShfCnwujjErFEicEqLc2tN4Qa0llkzDjfAukrR4jORari5cJRw?= =?us-ascii?q?zoxwMtJGJ72Y8sGZDtvZRLPSx1SM0gaCdQ1m+LArn3ydDtwq1+Po6czpW/Jw2R?= =?us-ascii?q?Z2bjkKt3TfvSMW8RZn0/erWXDl0z8CBUdP9y3zySE/nOlwOTImEvTXIMUCa399?= =?us-ascii?q?fNwhlCX7nIcBQdQVlahp/S9zEmkVLp3L00S5zprt6Q3/WS1QdTnGR61uniJulg?= =?us-ascii?q?bQdU4O+815ymfy6fM7kCSDwA5opRpADA9nJZuA2V0iRnSxJW0bQGDeYa9ERq1n?= =?us-ascii?q?op4ZxvraUD59VM/WBI=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3Adt8IOq9tVvkgwGevW2xuk+A8I+orL9Y04lQ7?= =?us-ascii?q?vn2YSXRuHPBw8Pre5cjztCWE7gr5N0tBpTntAsW9qDbnhPtICOoqTNCftWvdyQ?= =?us-ascii?q?iVxehZhOOIqVDd8m/Fh4pgPMxbEpSWZueeMbEDt7eZ3OCnKadc/PC3tLCvmfzF?= =?us-ascii?q?z2pgCSVja6Rb5Q9/DQqBe3cGPzVuNN4oEoaG/Mpbq36FcXQTVM6yAX4IRKztvN?= =?us-ascii?q?vO/aiWGyIuNlo27hWUlzO05PrfGxic5B0XVDRC2vMD3AH+4nTE2pk=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657414" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD04.g08.fujitsu.local (unknown [10.167.33.201]) by cn.fujitsu.com (Postfix) with ESMTP id 50FDC4D15A28; Tue, 28 Dec 2021 16:01:51 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD04.g08.fujitsu.local (10.167.33.201) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:50 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:48 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 06/10] RDMA/rxe: Implement RC RDMA FLUSH service in requester side Date: Tue, 28 Dec 2021 16:07:13 +0800 Message-ID: <20211228080717.10666-7-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 50FDC4D15A28.A04D9 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org a RC FLUSH packet consists of: +----+------+------+ |BTH | FETH | RETH | +----+------+------+ oA19-2: FLUSH shall be single packet message and shall have no payload. oA19-5: FLUSH BTH shall hold the Opcode =3D 0x1C FLUSH Extended Transport Header(FETH) +-----+-----------+------------------------+----------------------+ |Bits | 31-6 | 5-4 | 3-0 | +-----+-----------+------------------------+----------------------+ | | Reserved | Selectivity Level(SEL) | Placement Type(PLT) | +-----+-----------+------------------------+----------------------+ Selectivity Level (SEL) =E2=80=93 defines the memory region scope the FLUSH should apply on. Values are as follows: =E2=80=A2 b=E2=80=9900 - Memory Region Range: FLUSH applies for all precedi= ng memory updates to the RETH range on this QP. All RETH fields shall be valid in this selectivity mode. RETH:DMALen field shall be between zero and (2 31 -1) bytes (inclusive). =E2=80=A2 b=E2=80=9901 - Memory Region: FLUSH applies for all preceding mem= ory updates to RETH.R_key on this QP. RETH:DMALen and RETH:VA shall be ignored in this mode. =E2=80=A2 b'10 - Reserved. =E2=80=A2 b'11 - Reserved. Placement Type (PLT) =E2=80=93 Defines the memory placement guarantee of this FLUSH. Multiple bits may be set in this field. Values are as follows: =E2=80=A2 Bit 0 if set to '1' indicated that the FLUSH should guarantee Glo= bal Visibility. =E2=80=A2 Bit 1 if set to '1' indicated that the FLUSH should guarantee Persistence. =E2=80=A2 Bits 3:2 are reserved Signed-off-by: Li Zhijian --- drivers/infiniband/core/uverbs_cmd.c | 16 ++++++++++++++++ drivers/infiniband/sw/rxe/rxe_hdr.h | 24 ++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_opcode.c | 13 +++++++++++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 3 +++ drivers/infiniband/sw/rxe/rxe_req.c | 10 ++++++++++ include/rdma/ib_pack.h | 2 ++ include/rdma/ib_verbs.h | 9 +++++++++ include/uapi/rdma/ib_user_verbs.h | 7 +++++++ include/uapi/rdma/rdma_user_rxe.h | 6 ++++++ 9 files changed, 90 insertions(+) diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core= /uverbs_cmd.c index 6b6393176b3c..ed54fb0fd13d 100644 --- a/drivers/infiniband/core/uverbs_cmd.c +++ b/drivers/infiniband/core/uverbs_cmd.c @@ -2080,6 +2080,22 @@ static int ib_uverbs_post_send(struct uverbs_attr_bu= ndle *attrs) rdma->rkey =3D user_wr->wr.rdma.rkey; =20 next =3D &rdma->wr; + } else if (user_wr->opcode =3D=3D IB_WR_RDMA_FLUSH) { + struct ib_flush_wr *flush; + + next_size =3D sizeof(*flush); + flush =3D alloc_wr(next_size, user_wr->num_sge); + if (!flush) { + ret =3D -ENOMEM; + goto out_put; + } + + flush->remote_addr =3D user_wr->wr.flush.remote_addr; + flush->rkey =3D user_wr->wr.flush.rkey; + flush->type =3D user_wr->wr.flush.type; + flush->level =3D user_wr->wr.flush.level; + + next =3D &flush->wr; } else if (user_wr->opcode =3D=3D IB_WR_ATOMIC_CMP_AND_SWP || user_wr->opcode =3D=3D IB_WR_ATOMIC_FETCH_AND_ADD) { struct ib_atomic_wr *atomic; diff --git a/drivers/infiniband/sw/rxe/rxe_hdr.h b/drivers/infiniband/sw/rx= e/rxe_hdr.h index e432f9e37795..e37aa1944b18 100644 --- a/drivers/infiniband/sw/rxe/rxe_hdr.h +++ b/drivers/infiniband/sw/rxe/rxe_hdr.h @@ -607,6 +607,29 @@ static inline void reth_set_len(struct rxe_pkt_info *p= kt, u32 len) rxe_opcode[pkt->opcode].offset[RXE_RETH], len); } =20 +/* + * FLUSH Extended Transport Header(FETH) + * +-----+-----------+------------------------+----------------------+ + * |Bits | 31-6 | 5-4 | 3-0 | + * +-----+-----------+------------------------+----------------------+ + * | Reserved | Selectivity Level(SEL) | Placement Type(PLT) | + * +-----+-----------+------------------------+----------------------+ + */ +#define FETH_PLT_SHIFT 0UL +#define FETH_SEL_SHIFT 4UL +#define FETH_RESERVED_SHIFT 6UL +#define FETH_PLT_MASK ((1UL << FETH_SEL_SHIFT) - 1UL) +#define FETH_SEL_MASK (~FETH_PLT_MASK & ((1UL << FETH_RESERVED_SHIFT) - 1U= L)) + +static inline void feth_init(struct rxe_pkt_info *pkt, u32 type, u32 level) +{ + u32 *p =3D (u32 *)(pkt->hdr + rxe_opcode[pkt->opcode].offset[RXE_FETH]); + u32 feth =3D ((level << FETH_SEL_SHIFT) & FETH_SEL_MASK) | + ((type << FETH_PLT_SHIFT) & FETH_PLT_MASK); + + *p =3D cpu_to_be32(feth); +} + /*************************************************************************= ***** * Atomic Extended Transport Header *************************************************************************= *****/ @@ -910,6 +933,7 @@ enum rxe_hdr_length { RXE_ATMETH_BYTES =3D sizeof(struct rxe_atmeth), RXE_IETH_BYTES =3D sizeof(struct rxe_ieth), RXE_RDETH_BYTES =3D sizeof(struct rxe_rdeth), + RXE_FETH_BYTES =3D sizeof(u32), }; =20 static inline size_t header_size(struct rxe_pkt_info *pkt) diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw= /rxe/rxe_opcode.c index 3ef5a10a6efd..d61c8b354af4 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.c +++ b/drivers/infiniband/sw/rxe/rxe_opcode.c @@ -316,6 +316,19 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] =3D { + RXE_AETH_BYTES, } }, + [IB_OPCODE_RC_RDMA_FLUSH] =3D { + .name =3D "IB_OPCODE_RC_RDMA_FLUSH", + .mask =3D RXE_FETH_MASK | RXE_RETH_MASK | RXE_FLUSH_MASK + | RXE_START_MASK | RXE_END_MASK | RXE_REQ_MASK, + .length =3D RXE_BTH_BYTES + RXE_FETH_BYTES + RXE_RETH_BYTES, + .offset =3D { + [RXE_BTH] =3D 0, + [RXE_FETH] =3D RXE_BTH_BYTES, + [RXE_RETH] =3D RXE_BTH_BYTES + RXE_FETH_BYTES, + [RXE_PAYLOAD] =3D RXE_BTH_BYTES + + RXE_FETH_BYTES + RXE_RETH_BYTES, + } + }, [IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE] =3D { .name =3D "IB_OPCODE_RC_ATOMIC_ACKNOWLEDGE", .mask =3D RXE_AETH_MASK | RXE_ATMACK_MASK | RXE_ACK_MASK diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.h b/drivers/infiniband/sw= /rxe/rxe_opcode.h index 8f9aaaf260f2..dbc2eca8a92c 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.h +++ b/drivers/infiniband/sw/rxe/rxe_opcode.h @@ -48,6 +48,7 @@ enum rxe_hdr_type { RXE_DETH, RXE_IMMDT, RXE_PAYLOAD, + RXE_FETH, NUM_HDR_TYPES }; =20 @@ -63,6 +64,7 @@ enum rxe_hdr_mask { RXE_IETH_MASK =3D BIT(RXE_IETH), RXE_RDETH_MASK =3D BIT(RXE_RDETH), RXE_DETH_MASK =3D BIT(RXE_DETH), + RXE_FETH_MASK =3D BIT(RXE_FETH), RXE_PAYLOAD_MASK =3D BIT(RXE_PAYLOAD), =20 RXE_REQ_MASK =3D BIT(NUM_HDR_TYPES + 0), @@ -80,6 +82,7 @@ enum rxe_hdr_mask { RXE_END_MASK =3D BIT(NUM_HDR_TYPES + 10), =20 RXE_LOOPBACK_MASK =3D BIT(NUM_HDR_TYPES + 12), + RXE_FLUSH_MASK =3D BIT(NUM_HDR_TYPES + 13), =20 RXE_READ_OR_ATOMIC_MASK =3D (RXE_READ_MASK | RXE_ATOMIC_MASK), RXE_WRITE_OR_SEND_MASK =3D (RXE_WRITE_MASK | RXE_SEND_MASK), diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rx= e/rxe_req.c index 5eb89052dd66..a3e9351873e2 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -220,6 +220,9 @@ static int next_opcode_rc(struct rxe_qp *qp, u32 opcode= , int fits) IB_OPCODE_RC_SEND_ONLY_WITH_IMMEDIATE : IB_OPCODE_RC_SEND_FIRST; =20 + case IB_WR_RDMA_FLUSH: + return IB_OPCODE_RC_RDMA_FLUSH; + case IB_WR_RDMA_READ: return IB_OPCODE_RC_RDMA_READ_REQUEST; =20 @@ -418,6 +421,10 @@ static struct sk_buff *init_req_packet(struct rxe_qp *= qp, reth_set_len(pkt, wqe->dma.resid); } =20 + /* Fill Flush Extension Transport Header */ + if (pkt->mask & RXE_FETH_MASK) + feth_init(pkt, ibwr->wr.flush.type, ibwr->wr.flush.level); + if (pkt->mask & RXE_IMMDT_MASK) immdt_set_imm(pkt, ibwr->ex.imm_data); =20 @@ -477,6 +484,9 @@ static int finish_packet(struct rxe_qp *qp, struct rxe_= send_wqe *wqe, =20 memset(pad, 0, bth_pad(pkt)); } + } else if (pkt->mask & RXE_FLUSH_MASK) { + // oA19-2: shall have no payload. + wqe->dma.resid =3D 0; } =20 return 0; diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h index a9162f25beaf..d19edb502de6 100644 --- a/include/rdma/ib_pack.h +++ b/include/rdma/ib_pack.h @@ -84,6 +84,7 @@ enum { /* opcode 0x15 is reserved */ IB_OPCODE_SEND_LAST_WITH_INVALIDATE =3D 0x16, IB_OPCODE_SEND_ONLY_WITH_INVALIDATE =3D 0x17, + IB_OPCODE_RDMA_FLUSH =3D 0x1C, =20 /* real constants follow -- see comment about above IB_OPCODE() macro for more details */ @@ -112,6 +113,7 @@ enum { IB_OPCODE(RC, FETCH_ADD), IB_OPCODE(RC, SEND_LAST_WITH_INVALIDATE), IB_OPCODE(RC, SEND_ONLY_WITH_INVALIDATE), + IB_OPCODE(RC, RDMA_FLUSH), =20 /* UC */ IB_OPCODE(UC, SEND_FIRST), diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 51d58b641201..d336f2f8bb69 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1297,6 +1297,7 @@ struct ib_qp_attr { enum ib_wr_opcode { /* These are shared with userspace */ IB_WR_RDMA_WRITE =3D IB_UVERBS_WR_RDMA_WRITE, + IB_WR_RDMA_FLUSH =3D IB_UVERBS_WR_RDMA_FLUSH, IB_WR_RDMA_WRITE_WITH_IMM =3D IB_UVERBS_WR_RDMA_WRITE_WITH_IMM, IB_WR_SEND =3D IB_UVERBS_WR_SEND, IB_WR_SEND_WITH_IMM =3D IB_UVERBS_WR_SEND_WITH_IMM, @@ -1391,6 +1392,14 @@ struct ib_atomic_wr { u32 rkey; }; =20 +struct ib_flush_wr { + struct ib_send_wr wr; + u64 remote_addr; + u32 rkey; + u32 type; + u32 level; +}; + static inline const struct ib_atomic_wr *atomic_wr(const struct ib_send_wr= *wr) { return container_of(wr, struct ib_atomic_wr, wr); diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_= verbs.h index 7ee73a0652f1..4b7093f58259 100644 --- a/include/uapi/rdma/ib_user_verbs.h +++ b/include/uapi/rdma/ib_user_verbs.h @@ -784,6 +784,7 @@ enum ib_uverbs_wr_opcode { IB_UVERBS_WR_RDMA_READ_WITH_INV =3D 11, IB_UVERBS_WR_MASKED_ATOMIC_CMP_AND_SWP =3D 12, IB_UVERBS_WR_MASKED_ATOMIC_FETCH_AND_ADD =3D 13, + IB_UVERBS_WR_RDMA_FLUSH =3D 14, /* Review enum ib_wr_opcode before modifying this */ }; =20 @@ -797,6 +798,12 @@ struct ib_uverbs_send_wr { __u32 invalidate_rkey; } ex; union { + struct { + __aligned_u64 remote_addr; + __u32 rkey; + __u32 type; + __u32 level; + } flush; struct { __aligned_u64 remote_addr; __u32 rkey; diff --git a/include/uapi/rdma/rdma_user_rxe.h b/include/uapi/rdma/rdma_use= r_rxe.h index f09c5c9e3dd5..45f1b5b25414 100644 --- a/include/uapi/rdma/rdma_user_rxe.h +++ b/include/uapi/rdma/rdma_user_rxe.h @@ -82,6 +82,12 @@ struct rxe_send_wr { __u32 invalidate_rkey; } ex; union { + struct { + __aligned_u64 remote_addr; + __u32 rkey; + __u32 type; + __u32 level; + } flush; struct { __aligned_u64 remote_addr; __u32 rkey; --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58B93C433F5 for ; Tue, 28 Dec 2021 08:02:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235576AbhL1IC1 (ORCPT ); Tue, 28 Dec 2021 03:02:27 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47591 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235491AbhL1ICV (ORCPT ); Tue, 28 Dec 2021 03:02:21 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AH/QsvK+TKEYZjE2zE+AIDrUD3n+TJUtcMsCJ2f8?= =?us-ascii?q?bfWQNrUpz0jMCzzBKWjiOaarca2GhLd51bt609xlT6p6DxoAyQFdlrnsFo1Bi8?= =?us-ascii?q?5ScXYvDRqvT04J+FuWaFQQ/qZx2huDodKjYdVeB4Ef9WlTdhSMkj/jRHOCsULe?= =?us-ascii?q?s1h1ZHmeIdg9w0HqPpMZp2uaEsfDha++8kYuaT//3YTdJ6BYoWo4g0J9vnTs01?= =?us-ascii?q?BjEVJz0iXRlDRxDlAe2e3D4l/vzL4npR5fzatE88uJX24/+IL+FEmPxp3/BC/u?= =?us-ascii?q?ulPD1b08LXqXPewOJjxK6WYD72l4b+HN0if19aZLwam8O49mNt9Rw2tVMt525T?= =?us-ascii?q?y8nI6/NhP8AFRJfFkmSOIUfoe6ZfCfj6Jf7I0ruNiGEL+9VJFsuMIQC4eFxAXl?= =?us-ascii?q?D3fMdITEJKBuEgoqe0qO5WPhu3Jx7dOHkOYoevjdryjSxJfInSJbMXKjM/dJe0?= =?us-ascii?q?x8wm8lREPeYbM0cARJrbQvNYh1GPFg/CI83g+qpwHL4dlVwrF+So4I07nLVwQg?= =?us-ascii?q?316LiWPLcetWQQsNRtkCGp27H9iLyBRRyHNmVzT2O8lqqmO7DnCq9U4UXfJW89?= =?us-ascii?q?/h3kBiQy3YVBRk+S1S2u7+6h1S4VtYZLFYbkgIqrK4v5AmxQtz0dwO3rWTCvRM?= =?us-ascii?q?GXddUVeog52mlyKXbyxSYC3AJCDVIAOHKHudeqScCjwfPxo22Q2c09uD9dJ5Uz?= =?us-ascii?q?Z/MxRvaBMTfBTZSDcPccTY43g=3D=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3A6nAw1Kx30g05ZRlNMBjHKrPwyr1zdoMgy1kn?= =?us-ascii?q?xilNoRw8SK2lfqeV7Y0mPH7P+U4ssR4b6LO90cW7Lk80sKQFhbX5Xo3SOjUO2l?= =?us-ascii?q?HYTr2KhLGKq1aLdkHDH6xmpMBdmsNFaOEYY2IVsS+D2njcLz8/+qj6zImYwffZ?= =?us-ascii?q?02x2TRxnL4Vp7wJCAA6dFUFsLTM2f6YRJd6N4NZdvTq8dTAyZsS/PHMMWO/OvJ?= =?us-ascii?q?nlj5TjCCR2eiIP2U2fiy+y8r7mH1y91hcaaTlGxrAv6izkvmXCl96ej80=3D?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657413" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD04.g08.fujitsu.local (unknown [10.167.33.201]) by cn.fujitsu.com (Postfix) with ESMTP id 42F6E4D15A2A; Tue, 28 Dec 2021 16:01:52 +0800 (CST) Received: from G08CNEXJMPEKD02.g08.fujitsu.local (10.167.33.202) by G08CNEXMBPEKD04.g08.fujitsu.local (10.167.33.201) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:52 +0800 Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXJMPEKD02.g08.fujitsu.local (10.167.33.202) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:51 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:49 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 07/10] RDMA/rxe: Set BTH's SE to zero for FLUSH packet Date: Tue, 28 Dec 2021 16:07:14 +0800 Message-ID: <20211228080717.10666-8-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 42F6E4D15A2A.A000C X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The SPEC said: oA19-6: FLUSH BTH header field solicited event (SE) indication shall be set to zero. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_req.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rx= e/rxe_req.c index a3e9351873e2..082c5f76f29b 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -401,7 +401,9 @@ static struct sk_buff *init_req_packet(struct rxe_qp *q= p, (pkt->mask & RXE_END_MASK) && ((pkt->mask & (RXE_SEND_MASK)) || (pkt->mask & (RXE_WRITE_MASK | RXE_IMMDT_MASK)) =3D=3D - (RXE_WRITE_MASK | RXE_IMMDT_MASK)); + (RXE_WRITE_MASK | RXE_IMMDT_MASK)) && + /* oA19-6: always set SE to zero */ + !(pkt->mask & RXE_FETH_MASK); =20 qp_num =3D (pkt->mask & RXE_DETH_MASK) ? ibwr->wr.ud.remote_qpn : qp->attr.dest_qp_num; --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65024C433EF for ; Tue, 28 Dec 2021 08:02:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235655AbhL1ICf (ORCPT ); Tue, 28 Dec 2021 03:02:35 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:10151 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235499AbhL1ICX (ORCPT ); Tue, 28 Dec 2021 03:02:23 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AercpgKPcO46kp9LvrR37lsFynXyQoLVcMsFnjC/?= =?us-ascii?q?WdQi5gG8qgjUHzWUaUTvSOPrZMDfwLY9/PdzioEoG65GHm99gGjLY11k3ESsS9?= =?us-ascii?q?pCt6fd1j6vIF3rLaJWFFSqL1u1GAjX7BJ1yHi+0SiuFaOC79CAmj/HQH9IQNca?= =?us-ascii?q?fUsxPbV49IMseoUI78wIJqtYAbemRW2thi/uryyHsEAPNNwpPD44hw/nrRCWDE?= =?us-ascii?q?xjFkGhwUlQWPZintbJF/pUfJMp3yaqZdxMUTmTId9NWSdovzJnhlo/Y1xwrTN2?= =?us-ascii?q?4kLfnaVBMSbnXVeSMoiMOHfH83V4Z/Wpvuko4HKN0hUN/jzSbn9FzydxLnZKtS?= =?us-ascii?q?wY1JbCKk+MYO/VdO3gnbPMbp+OefhBTtuTWlSUqaUDE2e1jBVstOosY4utfDmR?= =?us-ascii?q?H9PheIzcIBjiRluCk0bDhErE0rssmJcjveogYvxlIyTDQC/k5TJbbTqPFzd9F1?= =?us-ascii?q?Sg9h4ZFGvO2T84YdjdubB3GbDVPJ14IBZN4l+Ct7lH7fjpegFGYv6w65y7U1gM?= =?us-ascii?q?Z+LHtOcDSfNiiQ9tUkkeR4GnB+gzRBxseM9ef4Tyb836tj6nEmiaTcIYTEqCos?= =?us-ascii?q?/1nmluewkQNBxAME1i2u/+0jgi5Qd03A0gV/Dc+6LI+8UWDUNbwRVu7rWSCsxp?= =?us-ascii?q?aXMBfe8U45w6l2KvZ+wvfDWFsc9LrQLTKr+dvHXpziADPxIivWFRSXHSuYSr13?= =?us-ascii?q?t+pQfmaYHl9wbc+WBI5?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AO5RRV6xplBwui8LOMMP0KrPwyL1zdoMgy1kn?= =?us-ascii?q?xilNoRw8SKKlfqeV7ZImPH7P+U8ssR4b+exoVJPtfZqYz+8R3WBzB8bEYOCFgh?= =?us-ascii?q?rKEGgK1+KLqFeMJ8S9zJ846U4KSclD4bPLYmSS9fyKgjVQDexQveWvweS5g/vE?= =?us-ascii?q?1XdxQUVPY6Fk1Q1wDQGWCSRNNXJ7LKt8BJyB/dBGujblXXwWa/6wDn4DU/OGiM?= =?us-ascii?q?bMkPvdEGQ7Li9i+A+Tlimp9bK/NxCZ2y0VWzRJzaxn0UWtqX2A2pme?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657417" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXMBPEKD06.g08.fujitsu.local (unknown [10.167.33.206]) by cn.fujitsu.com (Postfix) with ESMTP id DC4A84D15A2C; Tue, 28 Dec 2021 16:01:52 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD06.g08.fujitsu.local (10.167.33.206) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:51 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:50 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 08/10] RDMA/rxe: Implement flush execution in responder side Date: Tue, 28 Dec 2021 16:07:15 +0800 Message-ID: <20211228080717.10666-9-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: DC4A84D15A2C.A2D95 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In contrast to other opcodes, after a series of sanity checking, FLUSH opcode will do a Placement Type checking before it really do the FLUSH operation. Responder will also reply NAK "Remote Access Error" if it found a placement type violation. We will persist data via arch_wb_cache_pmem(), which could be architecture specific. After the execution, responder would reply a responded successfully by RDMA READ response of zero size. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_hdr.h | 28 ++++++ drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 4 +- drivers/infiniband/sw/rxe/rxe_resp.c | 131 ++++++++++++++++++++++++++- include/uapi/rdma/ib_user_verbs.h | 10 ++ 5 files changed, 169 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe_hdr.h b/drivers/infiniband/sw/rx= e/rxe_hdr.h index e37aa1944b18..cdfd393b8bd8 100644 --- a/drivers/infiniband/sw/rxe/rxe_hdr.h +++ b/drivers/infiniband/sw/rxe/rxe_hdr.h @@ -630,6 +630,34 @@ static inline void feth_init(struct rxe_pkt_info *pkt,= u32 type, u32 level) *p =3D cpu_to_be32(feth); } =20 +static inline u32 __feth_plt(void *arg) +{ + u32 *fethp =3D arg; + u32 feth =3D be32_to_cpu(*fethp); + + return (feth & FETH_PLT_MASK) >> FETH_PLT_SHIFT; +} + +static inline u32 __feth_sel(void *arg) +{ + u32 *fethp =3D arg; + u32 feth =3D be32_to_cpu(*fethp); + + return (feth & FETH_SEL_MASK) >> FETH_SEL_SHIFT; +} + +static inline u32 feth_plt(struct rxe_pkt_info *pkt) +{ + return __feth_plt(pkt->hdr + + rxe_opcode[pkt->opcode].offset[RXE_FETH]); +} + +static inline u32 feth_sel(struct rxe_pkt_info *pkt) +{ + return __feth_sel(pkt->hdr + + rxe_opcode[pkt->opcode].offset[RXE_FETH]); +} + /*************************************************************************= ***** * Atomic Extended Transport Header *************************************************************************= *****/ diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rx= e/rxe_loc.h index b1e174afb1d4..73c39ff11e28 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -80,6 +80,8 @@ int rxe_mr_copy(struct rxe_mr *mr, u64 iova, void *addr, = int length, enum rxe_mr_copy_dir dir); int copy_data(struct rxe_pd *pd, int access, struct rxe_dma_info *dma, void *addr, int length, enum rxe_mr_copy_dir dir); +void lookup_iova(struct rxe_mr *mr, u64 iova, int *m_out, int *n_out, + size_t *offset_out); void *iova_to_vaddr(struct rxe_mr *mr, u64 iova, int length); struct rxe_mr *lookup_mr(struct rxe_pd *pd, int access, u32 key, enum rxe_mr_lookup_type type); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe= /rxe_mr.c index 21616d058f29..2cb530305392 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -325,8 +325,8 @@ int rxe_mr_init_fast(struct rxe_pd *pd, int max_pages, = struct rxe_mr *mr) return err; } =20 -static void lookup_iova(struct rxe_mr *mr, u64 iova, int *m_out, int *n_ou= t, - size_t *offset_out) +void lookup_iova(struct rxe_mr *mr, u64 iova, int *m_out, int *n_out, + size_t *offset_out) { struct rxe_map_set *set =3D mr->cur_map_set; size_t offset =3D iova - set->iova + set->offset; diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/r= xe/rxe_resp.c index e8f435fa6e4d..6730336037d1 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -5,6 +5,7 @@ */ =20 #include +#include =20 #include "rxe.h" #include "rxe_loc.h" @@ -19,6 +20,7 @@ enum resp_states { RESPST_CHK_RESOURCE, RESPST_CHK_LENGTH, RESPST_CHK_RKEY, + RESPST_CHK_PLT, RESPST_EXECUTE, RESPST_READ_REPLY, RESPST_COMPLETE, @@ -35,6 +37,7 @@ enum resp_states { RESPST_ERR_TOO_MANY_RDMA_ATM_REQ, RESPST_ERR_RNR, RESPST_ERR_RKEY_VIOLATION, + RESPST_ERR_PLT_VIOLATION, RESPST_ERR_INVALIDATE_RKEY, RESPST_ERR_LENGTH, RESPST_ERR_CQ_OVERFLOW, @@ -53,6 +56,7 @@ static char *resp_state_name[] =3D { [RESPST_CHK_RESOURCE] =3D "CHK_RESOURCE", [RESPST_CHK_LENGTH] =3D "CHK_LENGTH", [RESPST_CHK_RKEY] =3D "CHK_RKEY", + [RESPST_CHK_PLT] =3D "CHK_PLACEMENT_TYPE", [RESPST_EXECUTE] =3D "EXECUTE", [RESPST_READ_REPLY] =3D "READ_REPLY", [RESPST_COMPLETE] =3D "COMPLETE", @@ -69,6 +73,7 @@ static char *resp_state_name[] =3D { [RESPST_ERR_TOO_MANY_RDMA_ATM_REQ] =3D "ERR_TOO_MANY_RDMA_ATM_REQ", [RESPST_ERR_RNR] =3D "ERR_RNR", [RESPST_ERR_RKEY_VIOLATION] =3D "ERR_RKEY_VIOLATION", + [RESPST_ERR_PLT_VIOLATION] =3D "ERR_PLACEMENT_TYPE_VIOLATION", [RESPST_ERR_INVALIDATE_RKEY] =3D "ERR_INVALIDATE_RKEY_VIOLATION", [RESPST_ERR_LENGTH] =3D "ERR_LENGTH", [RESPST_ERR_CQ_OVERFLOW] =3D "ERR_CQ_OVERFLOW", @@ -400,6 +405,30 @@ static enum resp_states check_length(struct rxe_qp *qp, } } =20 +static enum resp_states check_placement_type(struct rxe_qp *qp, + struct rxe_pkt_info *pkt) +{ + struct rxe_mr *mr =3D qp->resp.mr; + u32 plt =3D feth_plt(pkt); + + // no FLUSH access flag MR + if ((mr->access & IB_ACCESS_FLUSH) =3D=3D 0) { + pr_err("This mr isn't registered any flush access permission\n"); + return RESPST_ERR_PLT_VIOLATION; + } + + if ((plt & IB_EXT_PLT_GLB_VIS && + !(mr->access & IB_ACCESS_FLUSH_GLOBAL_VISIBILITY)) || + (plt & IB_EXT_PLT_PERSIST && + !(mr->access & IB_ACCESS_FLUSH_PERSISTENT))) { + pr_err("Target MR don't allow this placement type, is_pmem: %d, register= flag: %x, request flag: %x\n", + mr->ibmr.is_pmem, mr->access >> 8, plt); + return RESPST_ERR_PLT_VIOLATION; + } + + return RESPST_EXECUTE; +} + static enum resp_states check_rkey(struct rxe_qp *qp, struct rxe_pkt_info *pkt) { @@ -413,7 +442,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, enum resp_states state; int access; =20 - if (pkt->mask & RXE_READ_OR_WRITE_MASK) { + if (pkt->mask & (RXE_READ_OR_WRITE_MASK | RXE_FLUSH_MASK)) { if (pkt->mask & RXE_RETH_MASK) { qp->resp.va =3D reth_va(pkt); qp->resp.offset =3D 0; @@ -434,8 +463,10 @@ static enum resp_states check_rkey(struct rxe_qp *qp, } =20 /* A zero-byte op is not required to set an addr or rkey. */ + /* RXE_FETH_MASK carraies zero-byte playload */ if ((pkt->mask & RXE_READ_OR_WRITE_MASK) && (pkt->mask & RXE_RETH_MASK) && + !(pkt->mask & RXE_FETH_MASK) && reth_len(pkt) =3D=3D 0) { return RESPST_EXECUTE; } @@ -503,7 +534,7 @@ static enum resp_states check_rkey(struct rxe_qp *qp, WARN_ON_ONCE(qp->resp.mr); =20 qp->resp.mr =3D mr; - return RESPST_EXECUTE; + return pkt->mask & RXE_FETH_MASK ? RESPST_CHK_PLT : RESPST_EXECUTE; =20 err: if (mr) @@ -549,6 +580,91 @@ static enum resp_states write_data_in(struct rxe_qp *q= p, return rc; } =20 +static int nvdimm_flush_iova(struct rxe_mr *mr, u64 iova, int length) +{ + int err; + int bytes; + u8 *va; + struct rxe_map **map; + struct rxe_phys_buf *buf; + int m; + int i; + size_t offset; + + if (length =3D=3D 0) + return 0; + + if (mr->type =3D=3D IB_MR_TYPE_DMA) { + arch_wb_cache_pmem((void *)iova, length); + return 0; + } + + WARN_ON_ONCE(!mr->cur_map_set); + + err =3D mr_check_range(mr, iova, length); + if (err) { + err =3D -EFAULT; + goto err1; + } + + lookup_iova(mr, iova, &m, &i, &offset); + + map =3D mr->cur_map_set->map + m; + buf =3D map[0]->buf + i; + + while (length > 0) { + va =3D (u8 *)(uintptr_t)buf->addr + offset; + bytes =3D buf->size - offset; + + if (bytes > length) + bytes =3D length; + + arch_wb_cache_pmem(va, bytes); + + length -=3D bytes; + + offset =3D 0; + buf++; + i++; + + if (i =3D=3D RXE_BUF_PER_MAP) { + i =3D 0; + map++; + buf =3D map[0]->buf; + } + } + + return 0; + +err1: + return err; +} + +static enum resp_states process_flush(struct rxe_qp *qp, + struct rxe_pkt_info *pkt) +{ + u64 length =3D 0, start =3D qp->resp.va; + u32 sel =3D feth_sel(pkt); + u32 plt =3D feth_plt(pkt); + struct rxe_mr *mr =3D qp->resp.mr; + + if (sel =3D=3D IB_EXT_SEL_MR_RANGE) + length =3D qp->resp.length; + else if (sel =3D=3D IB_EXT_SEL_MR_WHOLE) + length =3D mr->cur_map_set->length; + + if (plt =3D=3D IB_EXT_PLT_PERSIST) { + nvdimm_flush_iova(mr, start, length); + wmb(); // clwb follows by a sfence + } else if (plt =3D=3D IB_EXT_PLT_GLB_VIS) + wmb(); // sfence is enough + + /* set RDMA READ response of zero */ + qp->resp.resid =3D 0; + + return RESPST_READ_REPLY; +} + /* Guarantee atomicity of atomic operations at the machine level. */ static DEFINE_SPINLOCK(atomic_ops_lock); =20 @@ -801,6 +917,8 @@ static enum resp_states execute(struct rxe_qp *qp, stru= ct rxe_pkt_info *pkt) err =3D process_atomic(qp, pkt); if (err) return err; + } else if (pkt->mask & RXE_FLUSH_MASK) { + return process_flush(qp, pkt); } else { /* Unreachable */ WARN_ON_ONCE(1); @@ -1059,7 +1177,7 @@ static enum resp_states duplicate_request(struct rxe_= qp *qp, /* SEND. Ack again and cleanup. C9-105. */ send_ack(qp, pkt, AETH_ACK_UNLIMITED, prev_psn); return RESPST_CLEANUP; - } else if (pkt->mask & RXE_READ_MASK) { + } else if (pkt->mask & RXE_READ_MASK || pkt->mask & RXE_FLUSH_MASK) { struct resp_res *res; =20 res =3D find_resource(qp, pkt->psn); @@ -1098,7 +1216,7 @@ static enum resp_states duplicate_request(struct rxe_= qp *qp, /* Reset the resource, except length. */ res->read.va_org =3D iova; res->read.va =3D iova; - res->read.resid =3D resid; + res->read.resid =3D pkt->mask & RXE_FLUSH_MASK ? 0 : resid; =20 /* Replay the RDMA read reply. */ qp->resp.res =3D res; @@ -1244,6 +1362,9 @@ int rxe_responder(void *arg) case RESPST_CHK_RKEY: state =3D check_rkey(qp, pkt); break; + case RESPST_CHK_PLT: + state =3D check_placement_type(qp, pkt); + break; case RESPST_EXECUTE: state =3D execute(qp, pkt); break; @@ -1298,6 +1419,8 @@ int rxe_responder(void *arg) break; =20 case RESPST_ERR_RKEY_VIOLATION: + /* oA19-13 8 */ + case RESPST_ERR_PLT_VIOLATION: if (qp_type(qp) =3D=3D IB_QPT_RC) { /* Class C */ do_class_ac_error(qp, AETH_NAK_REM_ACC_ERR, diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_= verbs.h index 4b7093f58259..efa06f53c7c6 100644 --- a/include/uapi/rdma/ib_user_verbs.h +++ b/include/uapi/rdma/ib_user_verbs.h @@ -105,6 +105,16 @@ enum { IB_USER_VERBS_EX_CMD_MODIFY_CQ }; =20 +enum ib_ext_placement_type { + IB_EXT_PLT_GLB_VIS =3D 1 << 0, + IB_EXT_PLT_PERSIST =3D 1 << 1, +}; + +enum ib_ext_selectivity_level { + IB_EXT_SEL_MR_RANGE =3D 0, + IB_EXT_SEL_MR_WHOLE, +}; + /* * Make sure that all structs defined in this file remain laid out so * that they pack the same way on 32-bit and 64-bit architectures (to --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC650C433F5 for ; Tue, 28 Dec 2021 08:02:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235670AbhL1ICp (ORCPT ); Tue, 28 Dec 2021 03:02:45 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47591 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235519AbhL1ICY (ORCPT ); Tue, 28 Dec 2021 03:02:24 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AtAOA9KB5HCWTOhVW/3fiw5YqxClBgxIJ4g17XOL?= =?us-ascii?q?fB1a6hj4q0WBWx2FJD2iFaK6OYDf3fN9+Pojn/E0E7MeAx9UxeLYW3SszFioV8?= =?us-ascii?q?6IpJjg4wn/YZnrUdouaJK5ex512huLocYZkHhcwmj/3auK49CMmhfnRLlbBILW?= =?us-ascii?q?s1h5ZFFYMpBgJ2UoLd94R2uaEsPDha++/kYqaT/73ZDdJ7wVJ3lc8sMpvnv/AU?= =?us-ascii?q?MPa41v0tnRmDRxCUcS3e3M9VPrzLonpR5f0rxU9IwK0ewrD5OnREmLx9BFrBM6?= =?us-ascii?q?nk6rgbwsBRbu60Qqm0yIQAvb9xEMZ4HFaPqUTbZLwbW9TiieJntJwwdNlu4GyS?= =?us-ascii?q?BsyI+vHn+F1vxxwSngvY/AZpOWbSZS4mYnJp6HcSFP22/hnFloxO40A9854BGh?= =?us-ascii?q?P8boTLzVlRgKShfCnwujjErFEicEqLc2tN4Qa0llkzDjfAukrR4jORari5cJRw?= =?us-ascii?q?zoxwMtJGJ72Y8sGZDtvZRLPSx1SM0gaCdQ1m+LArn3ydDtwq1+Po6czpW/Jw2R?= =?us-ascii?q?Z2bjkKt3TfvSMW8RZn0/erWXDl0z8CBUdP9y3zySE/nOlwOTImEvTXIMUCa399?= =?us-ascii?q?fNwhlCX7nIcBQdQVlahp/S9zEmkVLp3L00S5zprt6Q3/WS1QdTnGR61uniJulg?= =?us-ascii?q?bQdU4O+815ymfy6fM7kCSDwA5opRpADA9nJZuA2V0iRnSxJW0bQGDeYa9ERq1n?= =?us-ascii?q?op4ZxvoYkD59VM/WBI=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AC35bv6HTqGBBuhHKpLqEDMeALOsnbusQ8zAX?= =?us-ascii?q?P0AYc3Jom6uj5qaTdZUgpGbJYVkqOE3I9ertBEDEewK4yXcX2/h3AV7BZniEhI?= =?us-ascii?q?LAFugLhuGO/9SjIVybygc378ZdmsZFZ+EYdWIK7/oS/jPIbuoI8Z2W9ryyn+fC?= =?us-ascii?q?wzNIRQFuUatp6AB0EW+gYzZLbTgDFZwkD4Cd+8YCgzKhfE4cZsO9CmJAcPPEo7?= =?us-ascii?q?Tw5ejbSC9DFxg68xOPkD/tzLb7FiKT1hAYXygK4ZpKyxm8rzDE?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657424" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:59 +0800 Received: from G08CNEXMBPEKD04.g08.fujitsu.local (unknown [10.167.33.201]) by cn.fujitsu.com (Postfix) with ESMTP id 0CD2A4D15A20; Tue, 28 Dec 2021 16:01:54 +0800 (CST) Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXMBPEKD04.g08.fujitsu.local (10.167.33.201) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:53 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:51 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 09/10] RDMA/rxe: Implement flush completion Date: Tue, 28 Dec 2021 16:07:16 +0800 Message-ID: <20211228080717.10666-10-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 0CD2A4D15A20.AE515 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Introduce a new IB_UVERBS_WC_FLUSH code to tell userspace a FLUSH completion. Signed-off-by: Li Zhijian --- drivers/infiniband/sw/rxe/rxe_comp.c | 4 +++- include/rdma/ib_verbs.h | 1 + include/uapi/rdma/ib_user_verbs.h | 1 + 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/r= xe/rxe_comp.c index f363fe3fa414..e5b9d07eba93 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -104,6 +104,7 @@ static enum ib_wc_opcode wr_to_wc_opcode(enum ib_wr_opc= ode opcode) case IB_WR_LOCAL_INV: return IB_WC_LOCAL_INV; case IB_WR_REG_MR: return IB_WC_REG_MR; case IB_WR_BIND_MW: return IB_WC_BIND_MW; + case IB_WR_RDMA_FLUSH: return IB_WC_RDMA_FLUSH; =20 default: return 0xff; @@ -261,7 +262,8 @@ static inline enum comp_state check_ack(struct rxe_qp *= qp, */ case IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE: if (wqe->wr.opcode !=3D IB_WR_RDMA_READ && - wqe->wr.opcode !=3D IB_WR_RDMA_READ_WITH_INV) { + wqe->wr.opcode !=3D IB_WR_RDMA_READ_WITH_INV && + wqe->wr.opcode !=3D IB_WR_RDMA_FLUSH) { wqe->status =3D IB_WC_FATAL_ERR; return COMPST_ERROR; } diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index d336f2f8bb69..d528f2c4eba0 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -963,6 +963,7 @@ const char *__attribute_const__ ib_wc_status_msg(enum i= b_wc_status status); enum ib_wc_opcode { IB_WC_SEND =3D IB_UVERBS_WC_SEND, IB_WC_RDMA_WRITE =3D IB_UVERBS_WC_RDMA_WRITE, + IB_WC_RDMA_FLUSH =3D IB_UVERBS_WC_FLUSH, IB_WC_RDMA_READ =3D IB_UVERBS_WC_RDMA_READ, IB_WC_COMP_SWAP =3D IB_UVERBS_WC_COMP_SWAP, IB_WC_FETCH_ADD =3D IB_UVERBS_WC_FETCH_ADD, diff --git a/include/uapi/rdma/ib_user_verbs.h b/include/uapi/rdma/ib_user_= verbs.h index efa06f53c7c6..b9d6b3ca5708 100644 --- a/include/uapi/rdma/ib_user_verbs.h +++ b/include/uapi/rdma/ib_user_verbs.h @@ -476,6 +476,7 @@ enum ib_uverbs_wc_opcode { IB_UVERBS_WC_BIND_MW =3D 5, IB_UVERBS_WC_LOCAL_INV =3D 6, IB_UVERBS_WC_TSO =3D 7, + IB_UVERBS_WC_FLUSH =3D 8, }; =20 struct ib_uverbs_wc { --=20 2.31.1 From nobody Wed Jul 1 06:50:30 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EA46C433EF for ; Tue, 28 Dec 2021 08:02:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235551AbhL1ICm (ORCPT ); Tue, 28 Dec 2021 03:02:42 -0500 Received: from mail.cn.fujitsu.com ([183.91.158.132]:47599 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S235496AbhL1ICY (ORCPT ); Tue, 28 Dec 2021 03:02:24 -0500 IronPort-Data: =?us-ascii?q?A9a23=3AzZyyYKB7kOjqQBVW/3fiw5YqxClBgxIJ4g17XOL?= =?us-ascii?q?fAVOw1GhxhDIPn2FOWGnVOv3fZGv9ctAiPY/n800Ou5SAx9UxeLYW3SszFioV8?= =?us-ascii?q?6IpJjg4wn/YZnrUdouaJK5ex512huLocYZkHhcwmj/3auK49CMmhfnRLlbBILW?= =?us-ascii?q?s1h5ZFFYMpBgJ2UoLd94R2uaEsPDha++/kYqaT/73ZDdJ7wVJ3lc8sMpvnv/AU?= =?us-ascii?q?MPa41v0tnRmDRxCUcS3e3M9VPrzLonpR5f0rxU9IwK0ewrD5OnREmLx9BFrBM6?= =?us-ascii?q?nk6rgbwsBRbu60Qqm0yIQAvb9xEMZ4HFaPqUTbZLwbW9TiieJntJwwdNlu4GyS?= =?us-ascii?q?BsyI+vHn+F1vxxwSngvY/AZpOWcSZS4mYnJp6HcSFP22/hnFloxO40A9854BGh?= =?us-ascii?q?P8boTLzVlRgKShfCnwujjErFEicEqLc2tN4Qa0llkzDjfAukrR4jORari5cJRw?= =?us-ascii?q?zoxwMtJGJ72Y8sGZDtvZRLPSx1SM0gaCdQ1m+LArn3ydDtwq1+Po6czpW/Jw2R?= =?us-ascii?q?Z2bjkKt3TfvSMW8RZn0/erWXDl0z8CBUdP9y3zySE/nOlwOTImEvTXIMUCa399?= =?us-ascii?q?fNwhlCX7nIcBQdQVlahp/S9zEmkVLp3L00S5zprt6Q3/WS1QdTnGR61uniJulg?= =?us-ascii?q?bQdU4O+815ymfy6fM7kCSDwA5opRpADA9nJZuA2V0iRnSxJW0bQGDeYa9ERq1n?= =?us-ascii?q?op4ZxvoZUD59VM/WBI=3D?= IronPort-HdrOrdr: =?us-ascii?q?A9a23=3A9zTfxK6D+bQqeArINwPXwC7XdLJyesId70hD?= =?us-ascii?q?6qhwISY1TiX+rbHWoB17726NtN9/YgBCpTntAsa9qDbnhPpICOoqTNGftWvdyQ?= =?us-ascii?q?mVxehZhOOIqVCNJ8S9zJ876U4KSchD4bPLY2SS9fyKhTVQDexQvOWvweS5g/vE?= =?us-ascii?q?1XdxQUVPY6Fk1Q1wDQGWCSRNNXN7LKt8BJyB/dBGujblXXwWa/6wDn4DU/OGiM?= =?us-ascii?q?bMkPvdEGM7Li9i+A+Tlimp9bK/NxCZ2y0VWzRJzaxn0UWtqX2D2pme?= X-IronPort-AV: E=Sophos;i="5.88,241,1635177600"; d="scan'208";a="119657423" Received: from unknown (HELO cn.fujitsu.com) ([10.167.33.5]) by heian.cn.fujitsu.com with ESMTP; 28 Dec 2021 16:01:58 +0800 Received: from G08CNEXMBPEKD06.g08.fujitsu.local (unknown [10.167.33.206]) by cn.fujitsu.com (Postfix) with ESMTP id 0CE9D4D15A21; Tue, 28 Dec 2021 16:01:55 +0800 (CST) Received: from G08CNEXJMPEKD02.g08.fujitsu.local (10.167.33.202) by G08CNEXMBPEKD06.g08.fujitsu.local (10.167.33.206) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:53 +0800 Received: from G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.85) by G08CNEXJMPEKD02.g08.fujitsu.local (10.167.33.202) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Tue, 28 Dec 2021 16:01:54 +0800 Received: from localhost.localdomain (10.167.225.141) by G08CNEXCHPEKD09.g08.fujitsu.local (10.167.33.209) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Tue, 28 Dec 2021 16:01:52 +0800 From: Li Zhijian To: , , , , CC: , , , , , , , Li Zhijian Subject: [RFC PATCH rdma-next 10/10] RDMA/rxe: Add RD FLUSH service support Date: Tue, 28 Dec 2021 16:07:17 +0800 Message-ID: <20211228080717.10666-11-lizhijian@cn.fujitsu.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> References: <20211228080717.10666-1-lizhijian@cn.fujitsu.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-yoursite-MailScanner-ID: 0CE9D4D15A21.AF2B7 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: lizhijian@fujitsu.com Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Although the SPEC said FLUSH is supported by RC/RD/XRC services, XRC has not been supported by the rxe. So XRC FLUSH will not be supported until rxe implements XRC service. Signed-off-by: Li Zhijian --- I have not setup a RD environment to test this protocol --- drivers/infiniband/sw/rxe/rxe_opcode.c | 20 ++++++++++++++++++++ include/rdma/ib_pack.h | 1 + 2 files changed, 21 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_opcode.c b/drivers/infiniband/sw= /rxe/rxe_opcode.c index d61c8b354af4..0d5f555f5ec5 100644 --- a/drivers/infiniband/sw/rxe/rxe_opcode.c +++ b/drivers/infiniband/sw/rxe/rxe_opcode.c @@ -919,6 +919,26 @@ struct rxe_opcode_info rxe_opcode[RXE_NUM_OPCODE] =3D { + RXE_RDETH_BYTES, } }, + [IB_OPCODE_RD_RDMA_FLUSH] =3D { + .name =3D "IB_OPCODE_RD_RDMA_FLUSH", + .mask =3D RXE_RDETH_MASK | RXE_FETH_MASK | RXE_RETH_MASK + | RXE_FLUSH_MASK | RXE_START_MASK + | RXE_END_MASK | RXE_REQ_MASK, + .length =3D RXE_BTH_BYTES + RXE_FETH_BYTES + RXE_RETH_BYTES, + .offset =3D { + [RXE_BTH] =3D 0, + [RXE_RDETH] =3D RXE_BTH_BYTES, + [RXE_FETH] =3D RXE_BTH_BYTES + + RXE_RDETH_BYTES, + [RXE_RETH] =3D RXE_BTH_BYTES + + RXE_RDETH_BYTES + + RXE_FETH_BYTES, + [RXE_PAYLOAD] =3D RXE_BTH_BYTES + + RXE_RDETH_BYTES + + RXE_FETH_BYTES + + RXE_RETH_BYTES, + } + }, =20 /* UD */ [IB_OPCODE_UD_SEND_ONLY] =3D { diff --git a/include/rdma/ib_pack.h b/include/rdma/ib_pack.h index d19edb502de6..40568a33ead8 100644 --- a/include/rdma/ib_pack.h +++ b/include/rdma/ib_pack.h @@ -151,6 +151,7 @@ enum { IB_OPCODE(RD, ATOMIC_ACKNOWLEDGE), IB_OPCODE(RD, COMPARE_SWAP), IB_OPCODE(RD, FETCH_ADD), + IB_OPCODE(RD, RDMA_FLUSH), =20 /* UD */ IB_OPCODE(UD, SEND_ONLY), --=20 2.31.1