POC for Tejun's idea to unify the per-cpu and unbound workqueues, as
suggested in http://lkml.kernel.org/r/ak569WYSm3ygKl1-@slm.duckdns.org

This series only unifies the allocation part (alloc_pwq()). It makes
alloc_pwq() a single allocator that returns a pwq backed by either a static
per-cpu pool (get_percpu_pool) or an unbound pool (get_unbound_pool), reusing
the existing static per-cpu pools. The install/link path is unchanged and all
patches are behavior-neutral. This sets things up to unify the install side
later.

Is this the right direction?

Questions to follow up:

1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
   non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
   so the unification can happen later (apply_workqueue_attrs_locked())?

2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
   treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
   accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
   (pwq->nr_active).

3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
   unified internals (single install path, per-cpu accounting special-cased)
   with WQ_UNBOUND staying for now -- or something more radical (the flags gone
   entirely, percpu purely an affinity value)?

Tests: 

 1) I've tested this on x86 and arm64, with regular tests
 2) I've hacked up a workqueue test suite, which has a bunch of tests,
    and this is what I am using to test these changes:
	https://github.com/leitao/wqtest

Thanks,
-breno

Signed-off-by: Breno Leitao <leitao@debian.org>
---
Breno Leitao (3):
      workqueue: introduce alloc_pwq()
      workqueue: allocate percpu pwqs through alloc_pwq()
      workqueue: factor out alloc_and_link_percpu_pwqs()

 kernel/workqueue.c | 96 ++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 60 insertions(+), 36 deletions(-)
---
base-commit: b9810cd75b9fb56a3425d391cba3f608502bd474
change-id: 20260709-tejun1-c2aaf36f54a4

Best regards,
--  
Breno Leitao <leitao@debian.org>

Hello,

On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
...
> 1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
>    non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
>    so the unification can happen later (apply_workqueue_attrs_locked())?

I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the
shape more be like unbound workqueue subsuming percpu workqueue?

> 2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
>    treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
>    accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
>    (pwq->nr_active).

I think it probably would be better to introduce a separate affinity scope
than modifying WQ_AFFN_CPU. Something which indicates that concurrency
management is in effect and max_active is per-cpu.

> 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
>    unified internals (single install path, per-cpu accounting special-cased)
>    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
>    entirely, percpu purely an affinity value)?

Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
to me.

Thanks.

-- 
tejun

On Thu, Jul 16, 2026 at 09:22:50AM -1000, Tejun Heo wrote:
> Hello,
> 
> On Tue, Jul 14, 2026 at 04:41:46AM -0700, Breno Leitao wrote:
> ...
> > 1) wqattrs is an unbound concept and apply_workqueue_attrs_locked() rejects
> >    non-unbound wqs. Should we leverage wqattrs in per-cpu workqueues as well,
> >    so the unification can happen later (apply_workqueue_attrs_locked())?
> 
> I'm not sure adding wqattrs to percpu workqueues makes sense. Wouldn't the
> shape more be like unbound workqueue subsuming percpu workqueue?

> > 2) If percpu becomes a WQ_AFFN_CPU affinity setting, how should max_active be
> >    treated? WQ_AFFN_CPU is unbound today, so it would inherit per-node
> >    accounting (wq_node_nr_active) and lose percpu's per-cpu max_active
> >    (pwq->nr_active).
> 
> I think it probably would be better to introduce a separate affinity scope
> than modifying WQ_AFFN_CPU. Something which indicates that concurrency
> management is in effect and max_active is per-cpu.
> 
> > 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
> >    unified internals (single install path, per-cpu accounting special-cased)
> >    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
> >    entirely, percpu purely an affinity value)?
> 
> Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
> to me.

Oh, now I see what you mean, I was heading the wrong way. Thanks for the
clarification, that makes total sense.

So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than
reusing WQ_AFFN_CPU. Something like:

    enum wq_affn_scope {
            ...
  +         WQ_AFFN_PERCPU,         /* one pod per CPU, backed by the per-cpu pool */

and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the
unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can
point a pwq at a per-cpu pool, so one mechanism serves both. Then move
all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate
WQ_PERCPU ?

I have this working as a prototype: WQ_PERCPU selects the scope and forces
strict affinity, and it boots with every percpu wq created through the
new path. 

A few things I'd like your read on:

    1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to
       WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu
       accounting falls out of the existing !WQ_UNBOUND checks. do you
       have any preference here, or should percpu become purely an
       affn_scope value with accounting decoupled from the flag?

    2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq
       context) for now?

    3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions
       (unbound_attrs allocation, and the CPU-hotplug fixups in
       workqueue_online_cpu()/workqueue_offline_cpu() that gate on
       unbound_attrs) that now have to learn about the percpu scope.

       Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would
       you rather percpu keep a lighter install path (closer to the current
       WQ_PERCPU direct path), if that's feasible?

Thanks for the guidance,
--breno

Hello, Breno.

Sorry about the delay.

On Fri, Jul 17, 2026 at 09:25:09AM -0700, Breno Leitao wrote:
> So: add a PERCPU wq_affn_scope and back it strictly per-CPU, rather than
> reusing WQ_AFFN_CPU. Something like:
> 
>     enum wq_affn_scope {
>             ...
>   +         WQ_AFFN_PERCPU,         /* one pod per CPU, backed by the per-cpu pool */
> 
> and move the per-cpu workqueue users onto WQ_AFFN_PERCPU. With that, the
> unbound install path (apply_wqattrs and the per-cpu, replaceable pwqs) can
> point a pwq at a per-cpu pool, so one mechanism serves both. Then move
> all the WQ_PERCPU users to WQ_AFFN_PERCPU, and eventually deprecate
> WQ_PERCPU ?

I don't think we'd deprecate WQ_PERCPU. It'd remain the way to specify on
creation that the workqueue has to be WQ_AFFN_PERCPU.

Also, maybe WQ_AFFN_PERCPU can be more descriptive - WQ_AFFN_CPU_CM for
concurrency-managed CPU scope? Or maybe this shouldn't be packaged into
WQ_AFFN but rather become its own mode field.

> I have this working as a prototype: WQ_PERCPU selects the scope and forces
> strict affinity, and it boots with every percpu wq created through the
> new path. 
> 
> A few things I'd like your read on:
> 
>     1) Percpu workqueues keep the WQ_PERCPU flag (I don't switch them to
>        WQ_UNBOUND when they move onto the WQ_AFFN_PERCPU scope), so per-cpu
>        accounting falls out of the existing !WQ_UNBOUND checks. do you
>        have any preference here, or should percpu become purely an
>        affn_scope value with accounting decoupled from the flag?

For concurrency management to work, it would need separate accounting (of
max_active, right?). WQ_PERCPU would indicate that the wq must stay per-cpu
for correctness, and we'd also want to allow concurrency management to
workqueues which want to be percpu for performance reasons but can be
switched into other affinity scopes for isolation, so it should move
together with whether the backend needs concurrency management or not
instead of WQ_PERCPU expressed at creation time.

It kinda sucks that max_active's meaning is different across the boundary
tho. Maybe percpu max_active should be separate into its own field, idk.

>     2) What about WQ_BH? Can we keep it on the direct per-cpu path (softirq
>        context) for now?

We can't switch workqueues on / off WQ_BH, but it'd also look a bit silly if
this becomes its own path. Code-shape-wise, I suspect it'd be cleaner if
this also becomes one of the pwq backends like the other two cases but
that's just a gut feel.

>     3) Routing percpu through apply_wqattrs pulls in unbound-only assumptions
>        (unbound_attrs allocation, and the CPU-hotplug fixups in
>        workqueue_online_cpu()/workqueue_offline_cpu() that gate on
>        unbound_attrs) that now have to learn about the percpu scope.
> 
>        Do you prefer teaching that shared path about WQ_AFFN_PERCPU, or would
>        you rather percpu keep a lighter install path (closer to the current
>        WQ_PERCPU direct path), if that's feasible?

Again, without thinking too deeploy about it, I think the code would look
better if we unify everything we can.

Thanks.

-- 
tejun

On Thu, Jul 16, 2026 at 09:22:50AM -1000, Tejun Heo wrote:
...
> > 3) What end state are you aiming for? Keep WQ_PERCPU as a thin flag over
> >    unified internals (single install path, per-cpu accounting special-cased)
> >    with WQ_UNBOUND staying for now -- or something more radical (the flags gone
> >    entirely, percpu purely an affinity value)?
> 
> Keeping WQ_PERCPU as a shorthand for specifying the percpu scope makes sense
> to me.

Oh, and please note that if a workqueue specifies WQ_PERCPU, it should be
confined to strict cpu affinity mode.

Thanks.

-- 
tejun