Hi,

Here's another attempt at supporting user-space allocations from a
specific carved-out reserved memory region.

The initial problem we were discussing was that I'm currently working on
a platform which has a memory layout with ECC enabled. However, enabling
the ECC has a number of drawbacks on that platform: lower performance,
increased memory usage, etc. So for things like framebuffers, the
trade-off isn't great and thus there's a memory region with ECC disabled
to allocate from for such use cases.

After a suggestion from John, I chose to first start using heap
allocations flags to allow for userspace to ask for a particular ECC
setup. This is then backed by a new heap type that runs from reserved
memory chunks flagged as such, and the existing DT properties to specify
the ECC properties.

After further discussion, it was considered that flags were not the
right solution, and relying on the names of the heaps would be enough to
let userspace know the kind of buffer it deals with.

Thus, even though the uAPI part of it had been dropped in this second
version, we still needed a driver to create heaps out of carved-out memory
regions. In addition to the original usecase, a similar driver can be
found in BSPs from most vendors, so I believe it would be a useful
addition to the kernel.

Some extra discussion with Rob Herring [1] came to the conclusion that
some specific compatible for this is not great either, and as such an
new driver probably isn't called for either.

Some other discussions we had with John [2] also dropped some hints that
multiple CMA heaps might be a good idea, and some vendors seem to do
that too.

So here's another attempt that doesn't affect the device tree at all and
will just create a heap for every CMA reserved memory region.

It also falls nicely into the current plan we have to support cgroups in
DRM/KMS and v4l2, which is an additional benefit.

Let me know what you think,
Maxime

1: https://lore.kernel.org/all/20250707-cobalt-dingo-of-serenity-dbf92c@houat/
2: https://lore.kernel.org/all/CANDhNCroe6ZBtN_o=c71kzFFaWK-fF5rCdnr9P5h1sgPOWSGSw@mail.gmail.com/

Let me know what you think,
Maxime

Signed-off-by: Maxime Ripard <mripard@kernel.org>
---
Changes in v7:
- Invert the logic and register CMA heap from the reserved memory /
  dma contiguous code, instead of iterating over them from the CMA heap.
- Link to v6: https://lore.kernel.org/r/20250709-dma-buf-ecc-heap-v6-0-dac9bf80f35d@kernel.org

Changes in v6:
- Drop the new driver and allocate a CMA heap for each region now
- Dropped the binding
- Rebased on 6.16-rc5
- Link to v5: https://lore.kernel.org/r/20250617-dma-buf-ecc-heap-v5-0-0abdc5863a4f@kernel.org

Changes in v5:
- Rebased on 6.16-rc2
- Switch from property to dedicated binding
- Link to v4: https://lore.kernel.org/r/20250520-dma-buf-ecc-heap-v4-1-bd2e1f1bb42c@kernel.org

Changes in v4:
- Rebased on 6.15-rc7
- Map buffers only when map is actually called, not at allocation time
- Deal with restricted-dma-pool and shared-dma-pool
- Reword Kconfig options
- Properly report dma_map_sgtable failures
- Link to v3: https://lore.kernel.org/r/20250407-dma-buf-ecc-heap-v3-0-97cdd36a5f29@kernel.org

Changes in v3:
- Reworked global variable patch
- Link to v2: https://lore.kernel.org/r/20250401-dma-buf-ecc-heap-v2-0-043fd006a1af@kernel.org

Changes in v2:
- Add vmap/vunmap operations
- Drop ECC flags uapi
- Rebase on top of 6.14
- Link to v1: https://lore.kernel.org/r/20240515-dma-buf-ecc-heap-v1-0-54cbbd049511@kernel.org

---
Maxime Ripard (5):
      doc: dma-buf: List the heaps by name
      dma-buf: heaps: cma: Register list of CMA regions at boot
      dma: contiguous: Register reusable CMA regions at boot
      dma: contiguous: Reserve default CMA heap
      dma-buf: heaps: cma: Create CMA heap for each CMA reserved region

 Documentation/userspace-api/dma-buf-heaps.rst | 24 ++++++++------
 MAINTAINERS                                   |  1 +
 drivers/dma-buf/heaps/Kconfig                 | 10 ------
 drivers/dma-buf/heaps/cma_heap.c              | 47 +++++++++++++++++----------
 include/linux/dma-buf/heaps/cma.h             | 16 +++++++++
 kernel/dma/contiguous.c                       | 11 +++++++
 6 files changed, 72 insertions(+), 37 deletions(-)
---
base-commit: 47633099a672fc7bfe604ef454e4f116e2c954b1
change-id: 20240515-dma-buf-ecc-heap-28a311d2c94e
prerequisite-message-id: <20250610131231.1724627-1-jkangas@redhat.com>
prerequisite-patch-id: bc44be5968feb187f2bc1b8074af7209462b18e7
prerequisite-patch-id: f02a91b723e5ec01fbfedf3c3905218b43d432da
prerequisite-patch-id: e944d0a3e22f2cdf4d3b3906e5603af934696deb

Best regards,
-- 
Maxime Ripard <mripard@kernel.org>

Hi,

On Mon, Jul 21, 2025 at 01:17:29PM +0200, Maxime Ripard wrote:
> Here's another attempt at supporting user-space allocations from a
> specific carved-out reserved memory region.
> 
> The initial problem we were discussing was that I'm currently working on
> a platform which has a memory layout with ECC enabled. However, enabling
> the ECC has a number of drawbacks on that platform: lower performance,
> increased memory usage, etc. So for things like framebuffers, the
> trade-off isn't great and thus there's a memory region with ECC disabled
> to allocate from for such use cases.
> 
> After a suggestion from John, I chose to first start using heap
> allocations flags to allow for userspace to ask for a particular ECC
> setup. This is then backed by a new heap type that runs from reserved
> memory chunks flagged as such, and the existing DT properties to specify
> the ECC properties.
> 
> After further discussion, it was considered that flags were not the
> right solution, and relying on the names of the heaps would be enough to
> let userspace know the kind of buffer it deals with.
> 
> Thus, even though the uAPI part of it had been dropped in this second
> version, we still needed a driver to create heaps out of carved-out memory
> regions. In addition to the original usecase, a similar driver can be
> found in BSPs from most vendors, so I believe it would be a useful
> addition to the kernel.
> 
> Some extra discussion with Rob Herring [1] came to the conclusion that
> some specific compatible for this is not great either, and as such an
> new driver probably isn't called for either.
> 
> Some other discussions we had with John [2] also dropped some hints that
> multiple CMA heaps might be a good idea, and some vendors seem to do
> that too.
> 
> So here's another attempt that doesn't affect the device tree at all and
> will just create a heap for every CMA reserved memory region.
> 
> It also falls nicely into the current plan we have to support cgroups in
> DRM/KMS and v4l2, which is an additional benefit.
> 
> Let me know what you think,
> Maxime

Any chance we can get this merged?

Maxime

On Tue, Aug 26, 2025 at 09:36:03AM +0200, Maxime Ripard wrote:
> Hi,
> 
> On Mon, Jul 21, 2025 at 01:17:29PM +0200, Maxime Ripard wrote:
> > Here's another attempt at supporting user-space allocations from a
> > specific carved-out reserved memory region.
> > 
> > The initial problem we were discussing was that I'm currently working on
> > a platform which has a memory layout with ECC enabled. However, enabling
> > the ECC has a number of drawbacks on that platform: lower performance,
> > increased memory usage, etc. So for things like framebuffers, the
> > trade-off isn't great and thus there's a memory region with ECC disabled
> > to allocate from for such use cases.
> > 
> > After a suggestion from John, I chose to first start using heap
> > allocations flags to allow for userspace to ask for a particular ECC
> > setup. This is then backed by a new heap type that runs from reserved
> > memory chunks flagged as such, and the existing DT properties to specify
> > the ECC properties.
> > 
> > After further discussion, it was considered that flags were not the
> > right solution, and relying on the names of the heaps would be enough to
> > let userspace know the kind of buffer it deals with.
> > 
> > Thus, even though the uAPI part of it had been dropped in this second
> > version, we still needed a driver to create heaps out of carved-out memory
> > regions. In addition to the original usecase, a similar driver can be
> > found in BSPs from most vendors, so I believe it would be a useful
> > addition to the kernel.
> > 
> > Some extra discussion with Rob Herring [1] came to the conclusion that
> > some specific compatible for this is not great either, and as such an
> > new driver probably isn't called for either.
> > 
> > Some other discussions we had with John [2] also dropped some hints that
> > multiple CMA heaps might be a good idea, and some vendors seem to do
> > that too.
> > 
> > So here's another attempt that doesn't affect the device tree at all and
> > will just create a heap for every CMA reserved memory region.
> > 
> > It also falls nicely into the current plan we have to support cgroups in
> > DRM/KMS and v4l2, which is an additional benefit.
> > 
> > Let me know what you think,
> > Maxime
> 
> Any chance we can get this merged?

Guys, can we move forward on this?

Maxime

On Wed, Sep 10, 2025 at 12:33 AM Maxime Ripard <mripard@kernel.org> wrote:
>
> On Tue, Aug 26, 2025 at 09:36:03AM +0200, Maxime Ripard wrote:
> > Hi,
> >
> > On Mon, Jul 21, 2025 at 01:17:29PM +0200, Maxime Ripard wrote:
> > > Here's another attempt at supporting user-space allocations from a
> > > specific carved-out reserved memory region.
> > >
> > > The initial problem we were discussing was that I'm currently working on
> > > a platform which has a memory layout with ECC enabled. However, enabling
> > > the ECC has a number of drawbacks on that platform: lower performance,
> > > increased memory usage, etc. So for things like framebuffers, the
> > > trade-off isn't great and thus there's a memory region with ECC disabled
> > > to allocate from for such use cases.
> > >
> > > After a suggestion from John, I chose to first start using heap
> > > allocations flags to allow for userspace to ask for a particular ECC
> > > setup. This is then backed by a new heap type that runs from reserved
> > > memory chunks flagged as such, and the existing DT properties to specify
> > > the ECC properties.
> > >
> > > After further discussion, it was considered that flags were not the
> > > right solution, and relying on the names of the heaps would be enough to
> > > let userspace know the kind of buffer it deals with.
> > >
> > > Thus, even though the uAPI part of it had been dropped in this second
> > > version, we still needed a driver to create heaps out of carved-out memory
> > > regions. In addition to the original usecase, a similar driver can be
> > > found in BSPs from most vendors, so I believe it would be a useful
> > > addition to the kernel.
> > >
> > > Some extra discussion with Rob Herring [1] came to the conclusion that
> > > some specific compatible for this is not great either, and as such an
> > > new driver probably isn't called for either.
> > >
> > > Some other discussions we had with John [2] also dropped some hints that
> > > multiple CMA heaps might be a good idea, and some vendors seem to do
> > > that too.
> > >
> > > So here's another attempt that doesn't affect the device tree at all and
> > > will just create a heap for every CMA reserved memory region.
> > >
> > > It also falls nicely into the current plan we have to support cgroups in
> > > DRM/KMS and v4l2, which is an additional benefit.
> > >
> > > Let me know what you think,
> > > Maxime
> >
> > Any chance we can get this merged?
>
> Guys, can we move forward on this?
>
> Maxime

Hi Maxime,

Sorry I've been MIA the last couple of months.

The docs for the "reusable" property say, "device driver(s) owning the
region need to be able to reclaim it back", but how can a driver
reclaim memory backing a dmabuf, since pages allocated for a dmabuf
aren't necessarily movable. Couldn't a user allocate all of it, and
refuse to close those dmabufs?

I backported this to 6.6 and ran it on a Pixel. While there are
already similar out-of-tree dmabuf heap drivers that expose heaps for
these reserved regions, they do more than just cma_alloc (multiple
flavors of buffer securing, use case specific alignment and padding,
and slightly different allocation strategies) so I don't think this
series would allow us to completely drop the custom heap code, but
it's a nice start. Does the cgroup part come in because the plan is to
add charging in cma_heap.c?

Thanks,
T.J.

Hi TJ,

On Wed, Sep 10, 2025 at 01:44:45PM -0700, T.J. Mercier wrote:
> On Wed, Sep 10, 2025 at 12:33 AM Maxime Ripard <mripard@kernel.org> wrote:
> >
> > On Tue, Aug 26, 2025 at 09:36:03AM +0200, Maxime Ripard wrote:
> > > Hi,
> > >
> > > On Mon, Jul 21, 2025 at 01:17:29PM +0200, Maxime Ripard wrote:
> > > > Here's another attempt at supporting user-space allocations from a
> > > > specific carved-out reserved memory region.
> > > >
> > > > The initial problem we were discussing was that I'm currently working on
> > > > a platform which has a memory layout with ECC enabled. However, enabling
> > > > the ECC has a number of drawbacks on that platform: lower performance,
> > > > increased memory usage, etc. So for things like framebuffers, the
> > > > trade-off isn't great and thus there's a memory region with ECC disabled
> > > > to allocate from for such use cases.
> > > >
> > > > After a suggestion from John, I chose to first start using heap
> > > > allocations flags to allow for userspace to ask for a particular ECC
> > > > setup. This is then backed by a new heap type that runs from reserved
> > > > memory chunks flagged as such, and the existing DT properties to specify
> > > > the ECC properties.
> > > >
> > > > After further discussion, it was considered that flags were not the
> > > > right solution, and relying on the names of the heaps would be enough to
> > > > let userspace know the kind of buffer it deals with.
> > > >
> > > > Thus, even though the uAPI part of it had been dropped in this second
> > > > version, we still needed a driver to create heaps out of carved-out memory
> > > > regions. In addition to the original usecase, a similar driver can be
> > > > found in BSPs from most vendors, so I believe it would be a useful
> > > > addition to the kernel.
> > > >
> > > > Some extra discussion with Rob Herring [1] came to the conclusion that
> > > > some specific compatible for this is not great either, and as such an
> > > > new driver probably isn't called for either.
> > > >
> > > > Some other discussions we had with John [2] also dropped some hints that
> > > > multiple CMA heaps might be a good idea, and some vendors seem to do
> > > > that too.
> > > >
> > > > So here's another attempt that doesn't affect the device tree at all and
> > > > will just create a heap for every CMA reserved memory region.
> > > >
> > > > It also falls nicely into the current plan we have to support cgroups in
> > > > DRM/KMS and v4l2, which is an additional benefit.
> > > >
> > > > Let me know what you think,
> > > > Maxime
> > >
> > > Any chance we can get this merged?
> >
> > Guys, can we move forward on this?
> >
> > Maxime
> 
> Hi Maxime,
> 
> Sorry I've been MIA the last couple of months.
> 
> The docs for the "reusable" property say, "device driver(s) owning the
> region need to be able to reclaim it back", but how can a driver
> reclaim memory backing a dmabuf, since pages allocated for a dmabuf
> aren't necessarily movable. Couldn't a user allocate all of it, and
> refuse to close those dmabufs?

I guess, but how is that any different than what we're doing on the
default allocator already?

It also has to be reusable, and will not be able to reclaim any memory
allocated through the heap.

> I backported this to 6.6 and ran it on a Pixel. While there are
> already similar out-of-tree dmabuf heap drivers that expose heaps for
> these reserved regions, they do more than just cma_alloc (multiple
> flavors of buffer securing, use case specific alignment and padding,
> and slightly different allocation strategies) so I don't think this
> series would allow us to completely drop the custom heap code, but
> it's a nice start.

Thanks for testing, and I totally expect more heaps coming for things
like protected memory, but it should indeed reduce the number of heap
drivers needed going forward.

> Does the cgroup part come in because the plan is to add charging in
> cma_heap.c?

Yes, and the system heap as well.

Maxime

On Thu, Sep 11, 2025 at 12:01 AM Maxime Ripard <mripard@kernel.org> wrote:
>
> Hi TJ,
>
> On Wed, Sep 10, 2025 at 01:44:45PM -0700, T.J. Mercier wrote:
> > On Wed, Sep 10, 2025 at 12:33 AM Maxime Ripard <mripard@kernel.org> wrote:
> > >
> > > On Tue, Aug 26, 2025 at 09:36:03AM +0200, Maxime Ripard wrote:
> > > > Hi,
> > > >
> > > > On Mon, Jul 21, 2025 at 01:17:29PM +0200, Maxime Ripard wrote:
> > > > > Here's another attempt at supporting user-space allocations from a
> > > > > specific carved-out reserved memory region.
> > > > >
> > > > > The initial problem we were discussing was that I'm currently working on
> > > > > a platform which has a memory layout with ECC enabled. However, enabling
> > > > > the ECC has a number of drawbacks on that platform: lower performance,
> > > > > increased memory usage, etc. So for things like framebuffers, the
> > > > > trade-off isn't great and thus there's a memory region with ECC disabled
> > > > > to allocate from for such use cases.
> > > > >
> > > > > After a suggestion from John, I chose to first start using heap
> > > > > allocations flags to allow for userspace to ask for a particular ECC
> > > > > setup. This is then backed by a new heap type that runs from reserved
> > > > > memory chunks flagged as such, and the existing DT properties to specify
> > > > > the ECC properties.
> > > > >
> > > > > After further discussion, it was considered that flags were not the
> > > > > right solution, and relying on the names of the heaps would be enough to
> > > > > let userspace know the kind of buffer it deals with.
> > > > >
> > > > > Thus, even though the uAPI part of it had been dropped in this second
> > > > > version, we still needed a driver to create heaps out of carved-out memory
> > > > > regions. In addition to the original usecase, a similar driver can be
> > > > > found in BSPs from most vendors, so I believe it would be a useful
> > > > > addition to the kernel.
> > > > >
> > > > > Some extra discussion with Rob Herring [1] came to the conclusion that
> > > > > some specific compatible for this is not great either, and as such an
> > > > > new driver probably isn't called for either.
> > > > >
> > > > > Some other discussions we had with John [2] also dropped some hints that
> > > > > multiple CMA heaps might be a good idea, and some vendors seem to do
> > > > > that too.
> > > > >
> > > > > So here's another attempt that doesn't affect the device tree at all and
> > > > > will just create a heap for every CMA reserved memory region.
> > > > >
> > > > > It also falls nicely into the current plan we have to support cgroups in
> > > > > DRM/KMS and v4l2, which is an additional benefit.
> > > > >
> > > > > Let me know what you think,
> > > > > Maxime
> > > >
> > > > Any chance we can get this merged?
> > >
> > > Guys, can we move forward on this?
> > >
> > > Maxime
> >
> > Hi Maxime,
> >
> > Sorry I've been MIA the last couple of months.
> >
> > The docs for the "reusable" property say, "device driver(s) owning the
> > region need to be able to reclaim it back", but how can a driver
> > reclaim memory backing a dmabuf, since pages allocated for a dmabuf
> > aren't necessarily movable. Couldn't a user allocate all of it, and
> > refuse to close those dmabufs?
>
> I guess, but how is that any different than what we're doing on the
> default allocator already?

Yeah fair, it's not. I'm thinking that makes determining a size for a
reusable driver-specified region that's always exposed to userspace a
bit fuzzy. The requirements for the driver can probably be known, but
for potentially unrelated allocations from userspace? The default
ownership / file permissions for the heap would have to be changed to
allow those non-reclaimable allocations, so maybe that's enough of an
opt-in for such regions.

> It also has to be reusable, and will not be able to reclaim any memory
> allocated through the heap.
>
> > I backported this to 6.6 and ran it on a Pixel. While there are
> > already similar out-of-tree dmabuf heap drivers that expose heaps for
> > these reserved regions, they do more than just cma_alloc (multiple
> > flavors of buffer securing, use case specific alignment and padding,
> > and slightly different allocation strategies) so I don't think this
> > series would allow us to completely drop the custom heap code, but
> > it's a nice start.
>
> Thanks for testing, and I totally expect more heaps coming for things
> like protected memory, but it should indeed reduce the number of heap
> drivers needed going forward.
>
> > Does the cgroup part come in because the plan is to add charging in
> > cma_heap.c?
>
> Yes, and the system heap as well.
>
> Maxime

Thanks,

Reviewed-by: T.J. Mercier <tjmercier@google.com>