From the top of the mountain, Tejun said [1]:

  WQ_PERCPU would indicate that the wq must stay per-cpu for correctness,
  and we'd also want to allow concurrency management to workqueues which
  want to be percpu for performance reasons but can be switched into other
  affinity scopes for isolation, so it should move together with whether
  the backend needs concurrency management or not instead of WQ_PERCPU
  expressed at creation time.

This is an RFC patchset that tries to translate that into opinionated
code.

Design principles:

 * Keep wq->max_active and wq->percpu_max_active both current. The two
   backends meter work differently, and a pwq must not end up metered
   against a limit nobody set. (this is the semi-conflictual with my previous
   commit 27db9dd7f84f3a ("workqueue: Give percpu workqueues their own
   max_active")

 * Create a ->concurrency_managed field in the wq attributes, used to
   decide whether to do concurrency management or not.

 * Move WQ_PERCPU onto an UNBOUND workqueue with WQ_AFFN_CPU affinity and
   the newly created ->concurrency_managed. WQ_PERCPU is then only the
   promise that the workqueue stays on that backend.

This is not for acceptance, given this is drastic and it was not tested
enought, it is more to share what I have in mind and get comments from
the community.

Not done: the switching itself. concurrency_managed is fixed when the
workqueue is created and is not exported through sysfs, so nothing takes a
workqueue onto the backend or off it yet. WQ_BH is still its own path
rather than a third backend, the static per-cpu pools are still looked up
outside unbound_pool_hash, and callers still pass WQ_PERCPU rather than
asking for the attrs they want.

Link: https://lore.kernel.org/all/amESSqf0TMmzhFGz@slm.duckdns.org/ [1]

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (3):
      workqueue: Maintain both max_active limits
      workqueue: Add a concurrency_managed workqueue attribute
      workqueue: Back every workqueue with the unbound machinery

 include/linux/workqueue.h |  10 +++++
 kernel/workqueue.c        | 107 +++++++++++++++++++++++++++-------------------
 2 files changed, 72 insertions(+), 45 deletions(-)
---
base-commit: 9d4815c14f7faf789aeaa63515024168daf0390c
change-id: 20260914-wq_final-bcfbcd3b0f79

Best regards,
--  
Breno Leitao <leitao@debian.org>

Hello, Breno.

On Fri, Sep 18, 2026 at 07:25:32AM -0700, Breno Leitao wrote:
>  * Keep wq->max_active and wq->percpu_max_active both current. The two
>    backends meter work differently, and a pwq must not end up metered
>    against a limit nobody set. (this is the semi-conflictual with my previous
>    commit 27db9dd7f84f3a ("workqueue: Give percpu workqueues their own
>    max_active")

Both should stay current but I don't think they should be the same number.
max_active means different things in the two domains, per-CPU on one side and
across the whole workqueue on the other, so let's keep the two sets of values
separate and link them through scaling. On creation, the argument sets the
values for the domain the workqueue starts in and the other domain is derived
from it. Afterwards, each domain has its own interface, kernel and sysfs, and
adjusting one updates the other accordingly. That way a switch always lands
on a sensible limit without anyone having to think about it.

>  * Create a ->concurrency_managed field in the wq attributes, used to
>    decide whether to do concurrency management or not.
>
>  * Move WQ_PERCPU onto an UNBOUND workqueue with WQ_AFFN_CPU affinity and
>    the newly created ->concurrency_managed. WQ_PERCPU is then only the
>    promise that the workqueue stays on that backend.

I'd rather not build percpu on top of CPU scope. Unbound with strict CPU
scope and percpu are different things. The pools, the metering and how the
unbound cpumask applies all differ, and both should keep existing. So, how
about making PERCPU its own scope? Whether concurrency management is then
expressed as a flag or an attribute doesn't matter much as long as it can be
turned on and off. WQ_PERCPU would mean that the workqueue can't leave the
PERCPU scope while CM can still be toggled.

BH can be a scope value too for consistency. It's only selectable on creation
and can't be switched into or out of, but having it in the same enum keeps
things uniform.

With scope carrying the backend, nice can apply verbatim on unbound and snap
to normal or highpri when the workqueue is on percpu, picked from the current
value. No need to restrict what can be written.

> Not done: the switching itself. concurrency_managed is fixed when the
> workqueue is created and is not exported through sysfs, so nothing takes a
> workqueue onto the backend or off it yet.

For PREFER_PERCPU type workqueues, which benefit from cmwq but don't depend
on it for correctness, there's no reason to block switching in either
direction. Having them follow the unbound cpumask when picking the queueing
CPU even while on percpu, the way the last patch keys that on WQ_PERCPU
rather than on the backend, makes sense for them.

Thanks.

--
tejun

Hello Tejun,

On Fri, Sep 18, 2026 at 04:13:36PM -1000, Tejun Heo wrote:
> On Fri, Sep 18, 2026 at 07:25:32AM -0700, Breno Leitao wrote:
> >  * Keep wq->max_active and wq->percpu_max_active both current. The two
> >    backends meter work differently, and a pwq must not end up metered
> >    against a limit nobody set. (this is the semi-conflictual with my previous
> >    commit 27db9dd7f84f3a ("workqueue: Give percpu workqueues their own
> >    max_active")
> 
> Both should stay current but I don't think they should be the same number.

Ok, that means we are going to keep both as commit 27db9dd7f84f3a, and
keep both values set at a given time, and the values will be different.

> max_active means different things in the two domains, per-CPU on one side and
> across the whole workqueue on the other, so let's keep the two sets of values
> separate and link them through scaling.

Right, I will create a function that maps/scale one to another. Maybe
the following?

	 static int percpu_to_unbound_max_active(int percpu_max_active)
	 {
		 s64 max_active = (s64)percpu_max_active * num_possible_cpus();

		 return min_t(s64, max_active, WQ_MAX_ACTIVE);
	 }

	 static int unbound_to_percpu_max_active(int max_active)
	 {
		 return DIV_ROUND_UP(max_active, num_possible_cpus());
	 }

> On creation, the argument sets the
> values for the domain the workqueue starts in and the other domain is derived
> from it. Afterwards, each domain has its own interface, kernel and sysfs, and
> adjusting one updates the other accordingly. That way a switch always lands
> on a sensible limit without anyone having to think about it.

Sure, and probably call them from wq_adjust_max_active(), so,
independent of the one that is being set (percpu or unbound), it will
change the other as well according to the function above.

> >  * Create a ->concurrency_managed field in the wq attributes, used to
> >    decide whether to do concurrency management or not.
> >
> >  * Move WQ_PERCPU onto an UNBOUND workqueue with WQ_AFFN_CPU affinity and
> >    the newly created ->concurrency_managed. WQ_PERCPU is then only the
> >    promise that the workqueue stays on that backend.
> 
> I'd rather not build percpu on top of CPU scope. Unbound with strict CPU
> scope and percpu are different things. The pools, the metering and how the
> unbound cpumask applies all differ, and both should keep existing. So, how
> about making PERCPU its own scope? 

Ack! I will proceed with a newly created WQ_AFFN_PERCPU scope then.

> Whether concurrency management is then
> expressed as a flag or an attribute doesn't matter much as long as it can be
> turned on and off. WQ_PERCPU would mean that the workqueue can't leave the
> PERCPU scope while CM can still be toggled.

Oh, interesting, so, we are going to have CM toggable even for
WQ_PERCPU and also for WQ_UNBOUND+WQ_AFFN_PERCPU, is this right?

Once we have it, what will be the difference between
WQ_UNBOUND+WQ_AFFN_PERCPU+CM from WQ_PERCPU? They look exactly the same
from a user perspective, no?

> BH can be a scope value too for consistency. It's only selectable on creation
> and can't be switched into or out of, but having it in the same enum keeps
> things uniform.

Do you mean something like WQ_AFFN_BH or an entry in workqueue_attrs,
similar to affn_strict and the recently propsoed concurrency_managed?

I understand that scope means 'enum wq_affn_scope affn_scope', so, you
want a WQ_AFFN_BH, right?

> With scope carrying the backend, nice can apply verbatim on unbound and snap
> to normal or highpri when the workqueue is on percpu, picked from the current
> value. No need to restrict what can be written.

So, it means that we are going to expose the wq_sysfs_unbound_attrs to
WQ_PERCPU as well, and we are going to map nice to high priority queues
and normal queues. For instance:

  - echo -5 > nice succeeds and stores -5, whatever scope the wq is in
  - For WQ_PERCPU it runs on the highpri pool (nice -20 in practice)
  - echo 0 > nice, will turn the WQ_PERCPU workqueue into the normal
    pool.

Am I getting the idea right here?

> > Not done: the switching itself. concurrency_managed is fixed when the
> > workqueue is created and is not exported through sysfs, so nothing takes a
> > workqueue onto the backend or off it yet.
> 
> For PREFER_PERCPU type workqueues, which benefit from cmwq but don't depend
> on it for correctness, there's no reason to block switching in either
> direction. Having them follow the unbound cpumask when picking the queueing
> CPU even while on percpu, the way the last patch keys that on WQ_PERCPU
> rather than on the backend, makes sense for them.

let me back up a bit and talk about the scopes. The final state of the
scopes will be:

enum wq_affn_scope {
        WQ_AFFN_DFL,    
        WQ_AFFN_CPU,     
        WQ_AFFN_SMT,      
        WQ_AFFN_CACHE,     
        WQ_AFFN_CACHE_SHARD,
        WQ_AFFN_NUMA,       
        WQ_AFFN_SYSTEM,     
+       WQ_AFFN_PERCPU,                 /* one pod per CPU + CM + affinity */
+       WQ_AFFN_PEREFER_PERCPU,         /* one pod per CPU + CM + relaxed affinity */
+       WQ_AFFN_BH,			/* scope for WQ_BH */
}

Is this the final state you are envisioning?

Thanks for the excellent review and guidance,
--breno

Hello, Breno.

On Tue, Sep 22, 2026 at 03:34:48AM -0700, Breno Leitao wrote:
> Right, I will create a function that maps/scale one to another. Maybe
> the following?

Unbound max_active is distributed among nodes, with min_active as the
floor on each node, so I think the conversion should account for both.
One possibility, using:

  U = unbound max_active, the system-wide target
  L = unbound min_active, the floor on each node
  P = percpu_max_active, the limit on each CPU
  N = online CPUs in the effective unbound mask, at least 1

  Setting P:
    U = min(P * N, WQ_MAX_ACTIVE)
    L = P

  Setting U:
    L = min(L, U)
    P = max(DIV_ROUND_UP(U, N), L)

  Setting L:
    L = clamp(L, 0, U)
    P = max(DIV_ROUND_UP(U, N), L)

This would carry the minimum concurrency in both directions. Clipping L
when lowering U preserves the existing setter's behavior. The cap and
rounding make the conversion approximate. We could derive the other
domain on explicit configuration changes and leave scope changes and
hotplug to use the saved values, with hotplug continuing to redistribute
U through the existing mechanism.

> Sure, and probably call them from wq_adjust_max_active(), so,
> independent of the one that is being set (percpu or unbound), it will
> change the other as well according to the function above.

I think separate kernel entry points and sysfs knobs for the two domains
would be clearer. Each would have a fixed meaning regardless of the
current scope. The conversion could happen in those setters, with
wq_adjust_max_active() publishing the saved limits and handling freezing.

> Oh, interesting, so, we are going to have CM toggable even for
> WQ_PERCPU and also for WQ_UNBOUND+WQ_AFFN_PERCPU, is this right?

The attribute could be changed in any scope, with its value taking effect
only in PERCPU scope. Maybe give it a percpu_ prefix to make that clear?

> Once we have it, what will be the difference between
> WQ_UNBOUND+WQ_AFFN_PERCPU+CM from WQ_PERCPU? They look exactly the same
> from a user perspective, no?

WQ_PERCPU declares that per-CPU affinity is required for correctness, so
the workqueue cannot leave PERCPU scope. WQ_PREFER_PERCPU starts in that
scope but permits scope changes and follows the unbound cpumask when
selecting the queueing CPU.

> I understand that scope means 'enum wq_affn_scope affn_scope', so, you
> want a WQ_AFFN_BH, right?

Yes. Scope is largely selecting the backend, and BH fits there. It remains
selectable only at creation, with no switching into or out of it.

>   - echo -5 > nice succeeds and stores -5, whatever scope the wq is in
>   - For WQ_PERCPU it runs on the highpri pool (nice -20 in practice)
>   - echo 0 > nice, will turn the WQ_PERCPU workqueue into the normal
>     pool.

I'd map only -20 to the highpri pool in PERCPU scope and everything else
to normal. The requested nice value would stay stored and apply verbatim
when using an unbound backend.

> Is this the final state you are envisioning?

How about PERCPU and BH scopes, with PREFER as a modifier of PERCPU like
CM? CM controls concurrency management. PREFER allows affinity to be
relaxed. WQ_PREFER_PERCPU would set the initial scope and modifier. We'd
still need to preserve the correctness requirement declared by
WQ_PERCPU when changing attributes.

These are suggestions. If you find a better way during implementation,
please go with that.

Thanks.

-- 
tejun

Hello,

On Fri, Sep 18, 2026 at 4:25 PM Breno Leitao <leitao@debian.org> wrote:
> [...]
> Not done: the switching itself. concurrency_managed is fixed when the
> workqueue is created and is not exported through sysfs, so nothing takes a
> workqueue onto the backend or off it yet. WQ_BH is still its own path
> rather than a third backend, the static per-cpu pools are still looked up
> outside unbound_pool_hash, and callers still pass WQ_PERCPU rather than
> asking for the attrs they want.

I personally think it is better that way indeed, with WQ_PERCPU present.

I started an RFC about WQ_PREFER_PERCPU, still not published: the idea
would be an unbound workqueue that can be per-CPU (using affinities)
but without CM; so when CPUs are isolated, these WQ_PREFER_PERCPU WQs
can act as unbound. So the CM flag you added makes sense to me.

I tested my code a bit on the wq/for-7.4 branch, and it seems to be
working, but I will wait to see what happens with this series, before
sending mine.

Let's hear Tejun opinion about all of this; I'm also interested
regarding my series, so I can change if I'm not heading in the right
direction.

Thanks!
-- 

Marco Crivellari

SUSE Labs