This is the first series of memcg_ext, proposed at [1].  It adds
bpf_memcg_ops, a struct_ops type through which memory controller policy is
attached to a cgroup.  A charge runs the policies of its cgroup and of
every ancestor, and the kernel combines what they return.  BPF picks
between things the kernel already does; it never does the work itself and
never touches a page counter.

The plan is to grow this one member at a time, and to add a member only
when there is a concrete problem it solves and a measurement showing it
does.  Nothing is added because it might be useful later.  So this first
series adds one member, for one problem.

The problem
===========

try_charge_memcg() calls __mem_cgroup_handle_over_high() before it returns,
which reclaims and can throttle the task.  That happens wherever the charge
happens, so a task holding a kernel lock can be stuck there, and everything
waiting on that lock is stuck behind it.

One concrete scenario which can be resolved by this new feature is the
kernfs notify worker. It delivers notifications with the cgroup2
kernfs_rwsem held for read, and the charge for the delivery allocation goes
to the cgroup that set the watch, usually one already under pressure. So
the worker reclaims while holding the lock, a waiting writer blocks every
later reader, and anything touching cgroupfs stalls for seconds. The patch
proposed in [2] fixes that one path in the kernel only for kernfs_rwsem.
The same path still takes kernfs_supers_rwsem.

One can make an argument that if we know the source of the issue in the
kernel, why not fix it similarly to [2] instead of having a generic
solution? The reason is that it will be an uphill and continuous battle as
the kernel keeps evolving and new sources of lock holders doing allocations
keep coming up. In addition, there will be cases where it might not be
possible to move the allocations out of locks, or where doing so would
make the code really ugly [3].

high_policy() lets a policy say where memory.high should be enforced
instead.  Its one request, BPF_MEMCG_HIGH_DEFER_INLINE, skips the inline
call, leaving the debt to be paid at the return to userspace.

Results
=======

Measured on a kernel without [2], with a policy that marks cgroupfs kernfs
lock holders.  Reproducer at [4].

                                      baseline    policy    upstream fix
  max kernfs_rwsem hold, worker        2.049 s     199 us   not taken
  max kernfs_supers_rwsem hold         2.049 s     4.8 ms   2.056 s
  max kernfs_rwsem write wait          4.096 s     6.2 ms   8.7 ms
  walker passes over /sys/fs/cgroup        352       5711   6700
  churn mkdir+rmdir ops                 13 059    364 832   415 488
  churn max latency                     30.7 s    15.1 ms   12.7 ms

The policy matches the fix on the bystander numbers, and does better on
kernfs_supers_rwsem, which the fix does not help: it drops kernfs_rwsem
from the delivery loop, while the policy stops the worker stalling at all,
so every lock it holds benefits.

The patches
===========

Patch 1 is a build fix.  __cgroup_bpf_query() only works because
CGROUP_TCP_SOCK_OPS is the one struct_ops attach type; a second one breaks
the build and makes a query for type 0 return the wrong thing.

Patch 2 adds the type and its registration, with no members, so nothing is
dispatched and behaviour does not change.  It also moves the attach type
enum out of CONFIG_CGROUP_BPF and fixes the register_bpf_struct_ops() no-op
stub, which never compiled because no caller reached it.

Patch 3 adds high_policy() and wires it into try_charge_memcg().

Patch 4 is the sample.  It tracks the two cgroupfs kernfs locks by hooking
the rwsem primitives rather than the forty-odd functions that take them.

Known open questions
====================

The semantics of memory.high for remote chargers or kernel threads is a
grey area and this series does not aim to resolve that.

Another open question is whether a bound on debt deferral is needed. At
the moment, we think that rather than putting a limit on deferral for
memory.high, it will be better to handle that through an async worker like
memcg->high_work.  We aim to introduce that later, along with the right CPU
accounting for that async work.

There is no prog_tests harness, so the numbers above come from the
reproducer and not from the test suite.

The sample's lock tracking costs about 8% throughput, because it
instruments every rwsem operation on the system.  That is fine for a sample
that has to cover every path, but it argues for a cheaper way to track
locks that cause isolation problems between unrelated workloads.

Future work
===========

Asking the kernel to reclaim on the policy's behalf, so a task that cannot
be throttled still pays. More members, each with a use case: hard-limit
policy, reclaim shaping, dynamic protection.

This builds on "bpf: A common way to attach struct_ops to a cgroup", which
supplies attach, detach, ordering, update, query and the RCU rules.
bpf_memcg_ops is its second user.

[1] https://lore.kernel.org/20260307182424.2889780-1-shakeel.butt@linux.dev/
[2] https://lore.kernel.org/20260910045406.485295-1-shakeel.butt@linux.dev/
[3] https://lore.kernel.org/20260917-wehten-achtfach-getarnt-c85a4812337d@brauner/
[4] https://github.com/shakeelb/mempressure-repros/tree/main/kernfs-notify-memcg

Shakeel Butt (4):
  bpf, cgroup: fix cgroup struct_ops query for a second attach type
  memcg_ext: add cgroup-attached bpf_memcg_ops
  memcg_ext: allow BPF to defer memory.high enforcement
  selftests/bpf: add a cgroupfs lock-holder bpf_memcg_ops sample

 MAINTAINERS                                   |   1 +
 include/linux/bpf-cgroup-defs.h               |  22 +-
 include/linux/bpf-cgroup.h                    |  14 +-
 include/linux/bpf.h                           |   2 +-
 include/linux/bpf_memcontrol.h                |  66 ++++++
 include/linux/cgroup.h                        |   7 +
 include/linux/sched.h                         |   4 +
 kernel/bpf/cgroup.c                           |   3 +-
 mm/bpf_memcontrol.c                           | 188 ++++++++++++++-
 mm/memcontrol.c                               |  31 ++-
 .../bpf/progs/memcg_ops_lockholder.c          | 222 ++++++++++++++++++
 11 files changed, 546 insertions(+), 14 deletions(-)
 create mode 100644 include/linux/bpf_memcontrol.h
 create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c


base-commit: 6e36e099b15bed1e8e5b3e3136c5e3eb56a15aa7
-- 
2.53.0-Meta

On Tue, Sep 22, 2026 at 3:30 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> This is the first series of memcg_ext, proposed at [1].  It adds
> bpf_memcg_ops, a struct_ops type through which memory controller policy is
> attached to a cgroup.  A charge runs the policies of its cgroup and of
> every ancestor, and the kernel combines what they return.  BPF picks
> between things the kernel already does; it never does the work itself and
> never touches a page counter.
>
> The plan is to grow this one member at a time, and to add a member only
> when there is a concrete problem it solves and a measurement showing it
> does.  Nothing is added because it might be useful later.  So this first
> series adds one member, for one problem.
>
> The problem
> ===========
>
> try_charge_memcg() calls __mem_cgroup_handle_over_high() before it returns,
> which reclaims and can throttle the task.  That happens wherever the charge
> happens, so a task holding a kernel lock can be stuck there, and everything
> waiting on that lock is stuck behind it.
>
> One concrete scenario which can be resolved by this new feature is the
> kernfs notify worker. It delivers notifications with the cgroup2
> kernfs_rwsem held for read, and the charge for the delivery allocation goes
> to the cgroup that set the watch, usually one already under pressure. So
> the worker reclaims while holding the lock, a waiting writer blocks every
> later reader, and anything touching cgroupfs stalls for seconds. The patch
> proposed in [2] fixes that one path in the kernel only for kernfs_rwsem.
> The same path still takes kernfs_supers_rwsem.
>
> One can make an argument that if we know the source of the issue in the
> kernel, why not fix it similarly to [2] instead of having a generic
> solution? The reason is that it will be an uphill and continuous battle as
> the kernel keeps evolving and new sources of lock holders doing allocations
> keep coming up. In addition, there will be cases where it might not be
> possible to move the allocations out of locks, or where doing so would
> make the code really ugly [3].
>
> high_policy() lets a policy say where memory.high should be enforced
> instead.  Its one request, BPF_MEMCG_HIGH_DEFER_INLINE, skips the inline
> call, leaving the debt to be paid at the return to userspace.
>
> Results
> =======
>
> Measured on a kernel without [2], with a policy that marks cgroupfs kernfs
> lock holders.  Reproducer at [4].
>
>                                       baseline    policy    upstream fix
>   max kernfs_rwsem hold, worker        2.049 s     199 us   not taken
>   max kernfs_supers_rwsem hold         2.049 s     4.8 ms   2.056 s
>   max kernfs_rwsem write wait          4.096 s     6.2 ms   8.7 ms
>   walker passes over /sys/fs/cgroup        352       5711   6700
>   churn mkdir+rmdir ops                 13 059    364 832   415 488
>   churn max latency                     30.7 s    15.1 ms   12.7 ms
>
> The policy matches the fix on the bystander numbers, and does better on
> kernfs_supers_rwsem, which the fix does not help: it drops kernfs_rwsem
> from the delivery loop, while the policy stops the worker stalling at all,
> so every lock it holds benefits.
>
> The patches
> ===========
>
> Patch 1 is a build fix.  __cgroup_bpf_query() only works because
> CGROUP_TCP_SOCK_OPS is the one struct_ops attach type; a second one breaks
> the build and makes a query for type 0 return the wrong thing.
>
> Patch 2 adds the type and its registration, with no members, so nothing is
> dispatched and behaviour does not change.  It also moves the attach type
> enum out of CONFIG_CGROUP_BPF and fixes the register_bpf_struct_ops() no-op
> stub, which never compiled because no caller reached it.
>
> Patch 3 adds high_policy() and wires it into try_charge_memcg().
>
> Patch 4 is the sample.  It tracks the two cgroupfs kernfs locks by hooking
> the rwsem primitives rather than the forty-odd functions that take them.
>
> Known open questions
> ====================
>
> The semantics of memory.high for remote chargers or kernel threads is a
> grey area and this series does not aim to resolve that.
>
> Another open question is whether a bound on debt deferral is needed. At
> the moment, we think that rather than putting a limit on deferral for
> memory.high, it will be better to handle that through an async worker like
> memcg->high_work.  We aim to introduce that later, along with the right CPU
> accounting for that async work.

Hello Shakeel,

On the open question of how deferred debt eventually gets paid: would
it make sense for the policy to also notify userspace (e.g. via
ringbuf) when it defers, and have a userspace reclaimer do the reclaim
through memory.reclaim?

I understand one of the concerns for the async worker is CPU
accounting. If the concern is that the kworker's CPU usage is not
charged to the target cgroup, the userspace reclaimer could instead be
spawned with clone3(CLONE_INTO_CGROUP) so it runs inside the target
cgroup, and both its CPU and memory usage get charged there.

One caveat: intermediate cgroups with the no-internal-process
constraint cannot take processes, so this would only work for leaf
cgroups.

What do you think?

>
> There is no prog_tests harness, so the numbers above come from the
> reproducer and not from the test suite.
>
> The sample's lock tracking costs about 8% throughput, because it
> instruments every rwsem operation on the system.  That is fine for a sample
> that has to cover every path, but it argues for a cheaper way to track
> locks that cause isolation problems between unrelated workloads.
>
> Future work
> ===========
>
> Asking the kernel to reclaim on the policy's behalf, so a task that cannot
> be throttled still pays. More members, each with a use case: hard-limit
> policy, reclaim shaping, dynamic protection.
>
> This builds on "bpf: A common way to attach struct_ops to a cgroup", which
> supplies attach, detach, ordering, update, query and the RCU rules.
> bpf_memcg_ops is its second user.
>
> [1] https://lore.kernel.org/20260307182424.2889780-1-shakeel.butt@linux.dev/
> [2] https://lore.kernel.org/20260910045406.485295-1-shakeel.butt@linux.dev/
> [3] https://lore.kernel.org/20260917-wehten-achtfach-getarnt-c85a4812337d@brauner/
> [4] https://github.com/shakeelb/mempressure-repros/tree/main/kernfs-notify-memcg
>
> Shakeel Butt (4):
>   bpf, cgroup: fix cgroup struct_ops query for a second attach type
>   memcg_ext: add cgroup-attached bpf_memcg_ops
>   memcg_ext: allow BPF to defer memory.high enforcement
>   selftests/bpf: add a cgroupfs lock-holder bpf_memcg_ops sample
>
>  MAINTAINERS                                   |   1 +
>  include/linux/bpf-cgroup-defs.h               |  22 +-
>  include/linux/bpf-cgroup.h                    |  14 +-
>  include/linux/bpf.h                           |   2 +-
>  include/linux/bpf_memcontrol.h                |  66 ++++++
>  include/linux/cgroup.h                        |   7 +
>  include/linux/sched.h                         |   4 +
>  kernel/bpf/cgroup.c                           |   3 +-
>  mm/bpf_memcontrol.c                           | 188 ++++++++++++++-
>  mm/memcontrol.c                               |  31 ++-
>  .../bpf/progs/memcg_ops_lockholder.c          | 222 ++++++++++++++++++
>  11 files changed, 546 insertions(+), 14 deletions(-)
>  create mode 100644 include/linux/bpf_memcontrol.h
>  create mode 100644 tools/testing/selftests/bpf/progs/memcg_ops_lockholder.c
>
>
> base-commit: 6e36e099b15bed1e8e5b3e3136c5e3eb56a15aa7
> --
> 2.53.0-Meta
>
>

-- 
Regards
Yafang

Hi Yafang,

On Wed, Sep 23, 2026 at 09:07:40PM +0800, Yafang Shao wrote:
> On Tue, Sep 22, 2026 at 3:30 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
> >
[...]
> >
> > Known open questions
> > ====================
> >
> > The semantics of memory.high for remote chargers or kernel threads is a
> > grey area and this series does not aim to resolve that.
> >
> > Another open question is whether a bound on debt deferral is needed. At
> > the moment, we think that rather than putting a limit on deferral for
> > memory.high, it will be better to handle that through an async worker like
> > memcg->high_work.  We aim to introduce that later, along with the right CPU
> > accounting for that async work.
> 
> Hello Shakeel,
> 
> On the open question of how deferred debt eventually gets paid: would
> it make sense for the policy to also notify userspace (e.g. via
> ringbuf) when it defers,

I think the notification through bpf programs is already possible and a bpf
program deciding to bypass memory.high can already do notification via ringbuf.

> and have a userspace reclaimer do the reclaim
> through memory.reclaim?
> 
> I understand one of the concerns for the async worker is CPU
> accounting. If the concern is that the kworker's CPU usage is not
> charged to the target cgroup, the userspace reclaimer could instead be
> spawned with clone3(CLONE_INTO_CGROUP) so it runs inside the target
> cgroup, and both its CPU and memory usage get charged there.
> 
> One caveat: intermediate cgroups with the no-internal-process
> constraint cannot take processes, so this would only work for leaf
> cgroups.
> 
> What do you think?

I think all of this is possible without additional code and with this series.
With AI, should be very easy to prototype it. Please take a stab and I will look
into it as well (time permitting).

Thanks for taking a look and also please let me know what other ways you think
memcg can be customized through BPF in a beneficial way.

On Wed, Sep 23, 2026 at 11:47 PM Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> Hi Yafang,
>
> On Wed, Sep 23, 2026 at 09:07:40PM +0800, Yafang Shao wrote:
> > On Tue, Sep 22, 2026 at 3:30 AM Shakeel Butt <shakeel.butt@linux.dev> wrote:
> > >
> [...]
> > >
> > > Known open questions
> > > ====================
> > >
> > > The semantics of memory.high for remote chargers or kernel threads is a
> > > grey area and this series does not aim to resolve that.
> > >
> > > Another open question is whether a bound on debt deferral is needed. At
> > > the moment, we think that rather than putting a limit on deferral for
> > > memory.high, it will be better to handle that through an async worker like
> > > memcg->high_work.  We aim to introduce that later, along with the right CPU
> > > accounting for that async work.
> >
> > Hello Shakeel,
> >
> > On the open question of how deferred debt eventually gets paid: would
> > it make sense for the policy to also notify userspace (e.g. via
> > ringbuf) when it defers,
>
> I think the notification through bpf programs is already possible and a bpf
> program deciding to bypass memory.high can already do notification via ringbuf.
>
> > and have a userspace reclaimer do the reclaim
> > through memory.reclaim?
> >
> > I understand one of the concerns for the async worker is CPU
> > accounting. If the concern is that the kworker's CPU usage is not
> > charged to the target cgroup, the userspace reclaimer could instead be
> > spawned with clone3(CLONE_INTO_CGROUP) so it runs inside the target
> > cgroup, and both its CPU and memory usage get charged there.
> >
> > One caveat: intermediate cgroups with the no-internal-process
> > constraint cannot take processes, so this would only work for leaf
> > cgroups.
> >
> > What do you think?
>
> I think all of this is possible without additional code and with this series.
> With AI, should be very easy to prototype it. Please take a stab and I will look
> into it as well (time permitting).

An LLM helped me quickly implement a userspace async memcg reclaimer
based on your series, and it seems to work quite well.

>
> Thanks for taking a look and also please let me know what other ways you think
> memcg can be customized through BPF in a beneficial way.

Sure. On our production servers we have been running a set of BPF
programs to tailor kernel behavior for different workloads — all of
them global programs so far — and I believe they are all good
candidates for per-cgroup BPF policies now that cgroup-attached
struct_ops is available. They have been really helpful in our
Kubernetes production environment. I have sent some of them upstream,
such as:

  - BPF-THP
    https://lwn.net/Articles/1039689/
  - BPF-auto-NUMA
    https://lwn.net/Articles/1054030/

Perhaps we can revisit both of them and turn them into per-cgroup
policies — what do you think?

We are also running some custom BPF programs that have not been sent
upstream yet, such as:

  - BPF-async-reclaimer
    We don't care about the CPU accounting of the kworker, so we just
wake up a kworker to do the async reclaiming.
  - BPF-fault-around

Both are really beneficial to our workloads, and both are global programs today.

We are planning a few more customizations to resolve painful
production issues, such as:

  - The long-standing inode::lock contention caused by dentries [0].
    We have not started implementing it yet, but we might introduce a
memcg->dentry_limit or a memcg->vfs_cache_pressure as BPF policies..
  - cgroup-level readahead.

So, to answer your question directly: for memcg itself, the beneficial
customizations for us are the reclaim policy (the async reclaimer
above), the dentry/vfs cache pressure knobs, and fault-around; the
rest are per-cgroup MM policies that would need the
struct_ops-to-cgroup mechanism generalized beyond memcg — which is why
I hope these use cases can help make the design more generic.

[0] https://lore.kernel.org/linux-fsdevel/20240511200240.6354-2-torvalds@linux-foundation.org/

-- 
Regards
Yafang