[LSF/MM/BPF RFC PATCH 00/13]

Md Haris Iqbal posted 13 patches 1 month, 1 week ago
There is a newer version of this series
drivers/block/Kconfig                      |    2 +
drivers/block/Makefile                     |    1 +
drivers/block/brmr/Kconfig                 |   28 +
drivers/block/brmr/Makefile                |   16 +
drivers/block/brmr/brmr-clt-reque.c        |  228 ++
drivers/block/brmr/brmr-clt-stats.c        |  332 ++
drivers/block/brmr/brmr-clt-sysfs.c        |  463 +++
drivers/block/brmr/brmr-clt.c              | 1222 +++++++
drivers/block/brmr/brmr-clt.h              |  299 ++
drivers/block/brmr/brmr-proto.h            |  121 +
drivers/block/brmr/brmr-srv-sysfs.c        |  707 ++++
drivers/block/brmr/brmr-srv.c              | 1402 +++++++
drivers/block/brmr/brmr-srv.h              |  133 +
drivers/infiniband/Kconfig                 |    1 +
drivers/infiniband/ulp/Makefile            |    1 +
drivers/infiniband/ulp/rmr/Kconfig         |   35 +
drivers/infiniband/ulp/rmr/Makefile        |   23 +
drivers/infiniband/ulp/rmr/rmr-clt-stats.c |   29 +
drivers/infiniband/ulp/rmr/rmr-clt-sysfs.c | 1496 ++++++++
drivers/infiniband/ulp/rmr/rmr-clt-trace.c |   11 +
drivers/infiniband/ulp/rmr/rmr-clt-trace.h |  110 +
drivers/infiniband/ulp/rmr/rmr-clt.c       | 3866 ++++++++++++++++++++
drivers/infiniband/ulp/rmr/rmr-clt.h       |  291 ++
drivers/infiniband/ulp/rmr/rmr-map-mgmt.c  |  933 +++++
drivers/infiniband/ulp/rmr/rmr-map.c       |  904 +++++
drivers/infiniband/ulp/rmr/rmr-map.h       |  246 ++
drivers/infiniband/ulp/rmr/rmr-pool.c      |  401 ++
drivers/infiniband/ulp/rmr/rmr-pool.h      |  400 ++
drivers/infiniband/ulp/rmr/rmr-proto.h     |  273 ++
drivers/infiniband/ulp/rmr/rmr-req.c       |  796 ++++
drivers/infiniband/ulp/rmr/rmr-req.h       |   65 +
drivers/infiniband/ulp/rmr/rmr-srv-md.c    |  764 ++++
drivers/infiniband/ulp/rmr/rmr-srv-sysfs.c | 1047 ++++++
drivers/infiniband/ulp/rmr/rmr-srv.c       | 3306 +++++++++++++++++
drivers/infiniband/ulp/rmr/rmr-srv.h       |  219 ++
drivers/infiniband/ulp/rmr/rmr.h           |  229 ++
36 files changed, 20400 insertions(+)
create mode 100644 drivers/block/brmr/Kconfig
create mode 100644 drivers/block/brmr/Makefile
create mode 100644 drivers/block/brmr/brmr-clt-reque.c
create mode 100644 drivers/block/brmr/brmr-clt-stats.c
create mode 100644 drivers/block/brmr/brmr-clt-sysfs.c
create mode 100644 drivers/block/brmr/brmr-clt.c
create mode 100644 drivers/block/brmr/brmr-clt.h
create mode 100644 drivers/block/brmr/brmr-proto.h
create mode 100644 drivers/block/brmr/brmr-srv-sysfs.c
create mode 100644 drivers/block/brmr/brmr-srv.c
create mode 100644 drivers/block/brmr/brmr-srv.h
create mode 100644 drivers/infiniband/ulp/rmr/Kconfig
create mode 100644 drivers/infiniband/ulp/rmr/Makefile
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-stats.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-sysfs.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-trace.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-trace.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-map-mgmt.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-map.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-map.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-pool.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-pool.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-proto.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-req.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-req.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv-md.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv-sysfs.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv.c
create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv.h
create mode 100644 drivers/infiniband/ulp/rmr/rmr.h
[LSF/MM/BPF RFC PATCH 00/13]
Posted by Md Haris Iqbal 1 month, 1 week ago
Following a conversation with Bart yesterday, I am sending the RMR+BRMR
code through patch for easier review.

The patches apply over the for-next branch of the block tree over commit
07dfa981ca3

For context,
RMR (Reliable Multicast over RTRS) is a kernel module that provides
active-active block-level replication over RDMA. It guarantees delivery
of IO to a group of storage nodes and handles resynchronization of data
directly between storage nodes without involving the compute client.

BRMR (Block device over RMR) sits on top of RMR and exposes a standard
Linux block device (/dev/brmrX) backed by an RMR pool. Together, RMR and
BRMR provide a single-hop replication and resynchronization solution for
RDMA-connected storage clusters.

My session is on Wednesday, at 12 in the storage room (Istanbul).

The modules are still in-development. Many of the parts are only
functionally complete. Some features (backend store replace) are
disabled for now.

Documentation: https://ionos-cloud.github.io/rmr.io/index.html
GitHub link: https://github.com/ionos-cloud/RMR (for kernel v6.12)

Md Haris Iqbal (13):
  RDMA/rmr: add public and private headers
  RDMA/rmr: add shared library code (pool, map, request)
  RDMA/rmr: client: main functionality
  RDMA/rmr: client: sysfs interface functions
  RDMA/rmr: server: main functionality
  RDMA/rmr: server: sysfs interface functions
  RDMA/rmr: include client and server modules into kernel compilation
  block/brmr: add private headers with brmr protocol structs and helpers
  block/brmr: client: main functionality
  block/brmr: client: sysfs interface functions
  block/brmr: server: main functionality
  block/brmr: server: sysfs interface functions
  block/brmr: include client and server modules into kernel compilation

 drivers/block/Kconfig                      |    2 +
 drivers/block/Makefile                     |    1 +
 drivers/block/brmr/Kconfig                 |   28 +
 drivers/block/brmr/Makefile                |   16 +
 drivers/block/brmr/brmr-clt-reque.c        |  228 ++
 drivers/block/brmr/brmr-clt-stats.c        |  332 ++
 drivers/block/brmr/brmr-clt-sysfs.c        |  463 +++
 drivers/block/brmr/brmr-clt.c              | 1222 +++++++
 drivers/block/brmr/brmr-clt.h              |  299 ++
 drivers/block/brmr/brmr-proto.h            |  121 +
 drivers/block/brmr/brmr-srv-sysfs.c        |  707 ++++
 drivers/block/brmr/brmr-srv.c              | 1402 +++++++
 drivers/block/brmr/brmr-srv.h              |  133 +
 drivers/infiniband/Kconfig                 |    1 +
 drivers/infiniband/ulp/Makefile            |    1 +
 drivers/infiniband/ulp/rmr/Kconfig         |   35 +
 drivers/infiniband/ulp/rmr/Makefile        |   23 +
 drivers/infiniband/ulp/rmr/rmr-clt-stats.c |   29 +
 drivers/infiniband/ulp/rmr/rmr-clt-sysfs.c | 1496 ++++++++
 drivers/infiniband/ulp/rmr/rmr-clt-trace.c |   11 +
 drivers/infiniband/ulp/rmr/rmr-clt-trace.h |  110 +
 drivers/infiniband/ulp/rmr/rmr-clt.c       | 3866 ++++++++++++++++++++
 drivers/infiniband/ulp/rmr/rmr-clt.h       |  291 ++
 drivers/infiniband/ulp/rmr/rmr-map-mgmt.c  |  933 +++++
 drivers/infiniband/ulp/rmr/rmr-map.c       |  904 +++++
 drivers/infiniband/ulp/rmr/rmr-map.h       |  246 ++
 drivers/infiniband/ulp/rmr/rmr-pool.c      |  401 ++
 drivers/infiniband/ulp/rmr/rmr-pool.h      |  400 ++
 drivers/infiniband/ulp/rmr/rmr-proto.h     |  273 ++
 drivers/infiniband/ulp/rmr/rmr-req.c       |  796 ++++
 drivers/infiniband/ulp/rmr/rmr-req.h       |   65 +
 drivers/infiniband/ulp/rmr/rmr-srv-md.c    |  764 ++++
 drivers/infiniband/ulp/rmr/rmr-srv-sysfs.c | 1047 ++++++
 drivers/infiniband/ulp/rmr/rmr-srv.c       | 3306 +++++++++++++++++
 drivers/infiniband/ulp/rmr/rmr-srv.h       |  219 ++
 drivers/infiniband/ulp/rmr/rmr.h           |  229 ++
 36 files changed, 20400 insertions(+)
 create mode 100644 drivers/block/brmr/Kconfig
 create mode 100644 drivers/block/brmr/Makefile
 create mode 100644 drivers/block/brmr/brmr-clt-reque.c
 create mode 100644 drivers/block/brmr/brmr-clt-stats.c
 create mode 100644 drivers/block/brmr/brmr-clt-sysfs.c
 create mode 100644 drivers/block/brmr/brmr-clt.c
 create mode 100644 drivers/block/brmr/brmr-clt.h
 create mode 100644 drivers/block/brmr/brmr-proto.h
 create mode 100644 drivers/block/brmr/brmr-srv-sysfs.c
 create mode 100644 drivers/block/brmr/brmr-srv.c
 create mode 100644 drivers/block/brmr/brmr-srv.h
 create mode 100644 drivers/infiniband/ulp/rmr/Kconfig
 create mode 100644 drivers/infiniband/ulp/rmr/Makefile
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-stats.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-sysfs.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-trace.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt-trace.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-clt.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-map-mgmt.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-map.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-map.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-pool.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-pool.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-proto.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-req.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-req.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv-md.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv-sysfs.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv.c
 create mode 100644 drivers/infiniband/ulp/rmr/rmr-srv.h
 create mode 100644 drivers/infiniband/ulp/rmr/rmr.h

-- 
2.43.0
Re: [LSF/MM/BPF RFC PATCH 00/13]
Posted by Leon Romanovsky 1 month ago
On Tue, May 05, 2026 at 09:46:12AM +0200, Md Haris Iqbal wrote:
> Following a conversation with Bart yesterday, I am sending the RMR+BRMR
> code through patch for easier review.
> 
> The patches apply over the for-next branch of the block tree over commit
> 07dfa981ca3
> 
> For context,
> RMR (Reliable Multicast over RTRS) is a kernel module that provides
> active-active block-level replication over RDMA. It guarantees delivery
> of IO to a group of storage nodes and handles resynchronization of data
> directly between storage nodes without involving the compute client.
> 
> BRMR (Block device over RMR) sits on top of RMR and exposes a standard
> Linux block device (/dev/brmrX) backed by an RMR pool. Together, RMR and
> BRMR provide a single-hop replication and resynchronization solution for
> RDMA-connected storage clusters.
> 
> My session is on Wednesday, at 12 in the storage room (Istanbul).

To summarize the discussion:

1. Move as much logic as possible into the block layer; RDMA should serve
   strictly as a transport.
2. Identify another in‑kernel user of this functionality, and add support for
   it if required. At least accommodate potential users elsewhere in the
   kernel.

Thanks
Re: [LSF/MM/BPF RFC PATCH 00/13]
Posted by Haris Iqbal 2 weeks, 3 days ago
On Tue, May 12, 2026 at 12:34 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Tue, May 05, 2026 at 09:46:12AM +0200, Md Haris Iqbal wrote:
> > Following a conversation with Bart yesterday, I am sending the RMR+BRMR
> > code through patch for easier review.
> >
> > The patches apply over the for-next branch of the block tree over commit
> > 07dfa981ca3
> >
> > For context,
> > RMR (Reliable Multicast over RTRS) is a kernel module that provides
> > active-active block-level replication over RDMA. It guarantees delivery
> > of IO to a group of storage nodes and handles resynchronization of data
> > directly between storage nodes without involving the compute client.
> >
> > BRMR (Block device over RMR) sits on top of RMR and exposes a standard
> > Linux block device (/dev/brmrX) backed by an RMR pool. Together, RMR and
> > BRMR provide a single-hop replication and resynchronization solution for
> > RDMA-connected storage clusters.
> >
> > My session is on Wednesday, at 12 in the storage room (Istanbul).
>
> To summarize the discussion:
>
> 1. Move as much logic as possible into the block layer; RDMA should serve
>    strictly as a transport.
> 2. Identify another in‑kernel user of this functionality, and add support for
>    it if required. At least accommodate potential users elsewhere in the
>    kernel.

Thanks for the summary Leon.

The main logic which handles multicast/replication legs, missed I/O
tracking, re-synchronization, etc are the core parts of RMR.
If we move those to a separate module, there won't be much left in
RMR. RMR already uses RTRS from the RDMA subsystem as transport.

Having said that, I am not against moving RMR out of the RDMA layer.
It can serve as a reliable replication service/library for any other
user in the kernel to use.
Which subsystem (block or something else) would be a better fit then,
can be discussed.

PS: Would this be a good candidate for a session/discussion in the upcoming LPC?

>
> Thanks
Re: [LSF/MM/BPF RFC PATCH 00/13]
Posted by Leon Romanovsky 2 days, 10 hours ago
On Wed, May 27, 2026 at 02:44:08PM +0200, Haris Iqbal wrote:
> On Tue, May 12, 2026 at 12:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Tue, May 05, 2026 at 09:46:12AM +0200, Md Haris Iqbal wrote:
> > > Following a conversation with Bart yesterday, I am sending the RMR+BRMR
> > > code through patch for easier review.
> > >
> > > The patches apply over the for-next branch of the block tree over commit
> > > 07dfa981ca3
> > >
> > > For context,
> > > RMR (Reliable Multicast over RTRS) is a kernel module that provides
> > > active-active block-level replication over RDMA. It guarantees delivery
> > > of IO to a group of storage nodes and handles resynchronization of data
> > > directly between storage nodes without involving the compute client.
> > >
> > > BRMR (Block device over RMR) sits on top of RMR and exposes a standard
> > > Linux block device (/dev/brmrX) backed by an RMR pool. Together, RMR and
> > > BRMR provide a single-hop replication and resynchronization solution for
> > > RDMA-connected storage clusters.
> > >
> > > My session is on Wednesday, at 12 in the storage room (Istanbul).
> >
> > To summarize the discussion:
> >
> > 1. Move as much logic as possible into the block layer; RDMA should serve
> >    strictly as a transport.
> > 2. Identify another in‑kernel user of this functionality, and add support for
> >    it if required. At least accommodate potential users elsewhere in the
> >    kernel.
> 
> Thanks for the summary Leon.
> 
> The main logic which handles multicast/replication legs, missed I/O
> tracking, re-synchronization, etc are the core parts of RMR.
> If we move those to a separate module, there won't be much left in
> RMR. RMR already uses RTRS from the RDMA subsystem as transport.
> 
> Having said that, I am not against moving RMR out of the RDMA layer.
> It can serve as a reliable replication service/library for any other
> user in the kernel to use.
> Which subsystem (block or something else) would be a better fit then,
> can be discussed.
> 
> PS: Would this be a good candidate for a session/discussion in the upcoming LPC?

Probably yes.

Thanks

> 
> >
> > Thanks
Re: [LSF/MM/BPF RFC PATCH 00/13]
Posted by Haris Iqbal 1 day, 12 hours ago
On Thu, Jun 11, 2026 at 1:59 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, May 27, 2026 at 02:44:08PM +0200, Haris Iqbal wrote:
> > On Tue, May 12, 2026 at 12:34 PM Leon Romanovsky <leon@kernel.org> wrote:
> > >
> > > On Tue, May 05, 2026 at 09:46:12AM +0200, Md Haris Iqbal wrote:
> > > > Following a conversation with Bart yesterday, I am sending the RMR+BRMR
> > > > code through patch for easier review.
> > > >
> > > > The patches apply over the for-next branch of the block tree over commit
> > > > 07dfa981ca3
> > > >
> > > > For context,
> > > > RMR (Reliable Multicast over RTRS) is a kernel module that provides
> > > > active-active block-level replication over RDMA. It guarantees delivery
> > > > of IO to a group of storage nodes and handles resynchronization of data
> > > > directly between storage nodes without involving the compute client.
> > > >
> > > > BRMR (Block device over RMR) sits on top of RMR and exposes a standard
> > > > Linux block device (/dev/brmrX) backed by an RMR pool. Together, RMR and
> > > > BRMR provide a single-hop replication and resynchronization solution for
> > > > RDMA-connected storage clusters.
> > > >
> > > > My session is on Wednesday, at 12 in the storage room (Istanbul).
> > >
> > > To summarize the discussion:
> > >
> > > 1. Move as much logic as possible into the block layer; RDMA should serve
> > >    strictly as a transport.
> > > 2. Identify another in‑kernel user of this functionality, and add support for
> > >    it if required. At least accommodate potential users elsewhere in the
> > >    kernel.
> >
> > Thanks for the summary Leon.
> >
> > The main logic which handles multicast/replication legs, missed I/O
> > tracking, re-synchronization, etc are the core parts of RMR.
> > If we move those to a separate module, there won't be much left in
> > RMR. RMR already uses RTRS from the RDMA subsystem as transport.
> >
> > Having said that, I am not against moving RMR out of the RDMA layer.
> > It can serve as a reliable replication service/library for any other
> > user in the kernel to use.
> > Which subsystem (block or something else) would be a better fit then,
> > can be discussed.
> >
> > PS: Would this be a good candidate for a session/discussion in the upcoming LPC?
>
> Probably yes.
>
> Thanks

Thanks Leon. I'll submit the abstract through the portal.
Do you think the topic is better suited towards Refereed track or Kernel Summit?

>
> >
> > >
> > > Thanks