[v5] Implement CPU hotplug on Arm

[PATCH v5 0/5] Implement CPU hotplug on Arm

Posted by Mykyta Poturai 3 weeks, 3 days ago

This series implements support for CPU hotplug/unplug on Arm. To achieve this,
several things need to be done:

1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
2. Enabled building of xen-hptool.
3. Migration of irqs from dying CPUs implemented.

Tested on QEMU.

v4->v5:
* drop merged patches
* combine "smp: Move cpu_up/down helpers to common code" with 
  "arm/sysctl: Implement cpu hotplug ops"
* see individual patches

v3->v4:
* add irq migration patches
* see individual patches

v2->v3:
* add docs

v1->v2:
* see individual patches

Mykyta Poturai (5):
  arm/irq: Keep track of irq affinities
  arm/irq: Migrate IRQs during CPU up/down operations
  arm/sysctl: Implement cpu hotplug ops
  tools: Allow building xen-hptool without CONFIG_MIGRATE
  docs: Document CPU hotplug

 SUPPORT.md                       |  1 +
 docs/misc/cpu-hotplug.txt        | 50 +++++++++++++++++++++++++
 tools/libs/guest/Makefile.common |  2 +-
 tools/misc/Makefile              |  2 +-
 xen/arch/arm/Kconfig             |  1 +
 xen/arch/arm/gic-vgic.c          |  2 +
 xen/arch/arm/include/asm/irq.h   |  2 +
 xen/arch/arm/irq.c               | 63 +++++++++++++++++++++++++++++++-
 xen/arch/arm/smp.c               |  9 +++++
 xen/arch/arm/smpboot.c           |  6 +++
 xen/arch/arm/vgic.c              | 14 ++++++-
 xen/arch/ppc/stubs.c             |  4 ++
 xen/arch/riscv/stubs.c           |  5 +++
 xen/arch/x86/Kconfig             |  1 +
 xen/arch/x86/include/asm/smp.h   |  3 --
 xen/arch/x86/smp.c               | 33 ++---------------
 xen/arch/x86/sysctl.c            | 12 ++----
 xen/common/Kconfig               |  3 ++
 xen/common/smp.c                 | 34 +++++++++++++++++
 xen/common/sysctl.c              | 45 +++++++++++++++++++++++
 xen/include/xen/smp.h            |  4 ++
 21 files changed, 248 insertions(+), 48 deletions(-)
 create mode 100644 docs/misc/cpu-hotplug.txt

-- 
2.51.2

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

Posted by Bertrand Marquis 3 days, 13 hours ago

Hi Mykyta,

> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
> several things need to be done:
> 
> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
> 2. Enabled building of xen-hptool.
> 3. Migration of irqs from dying CPUs implemented.
> 
> Tested on QEMU.

Do you have a way to validate this on real hardware ?
Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
I would not be completely confident to have validated by only testing this in qemu.

Could you explain how you validated this (what kind of tests you did).

I would be interested to know how the system behave in the following cases:
- stopping cpu with guests pinned to specific cores
- stopping a cpu with a guest pinned and only running on that one
- checking with passthrough devices (for irq balancing)

Then a general remark i have is that most of this code (if not all) is always in Xen
even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
generate some dead code. Could we protect some of the code not used when
HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
equivalent forms ?

Cheers
Bertrand

> 
> v4->v5:
> * drop merged patches
> * combine "smp: Move cpu_up/down helpers to common code" with 
>  "arm/sysctl: Implement cpu hotplug ops"
> * see individual patches
> 
> v3->v4:
> * add irq migration patches
> * see individual patches
> 
> v2->v3:
> * add docs
> 
> v1->v2:
> * see individual patches
> 
> Mykyta Poturai (5):
>  arm/irq: Keep track of irq affinities
>  arm/irq: Migrate IRQs during CPU up/down operations
>  arm/sysctl: Implement cpu hotplug ops
>  tools: Allow building xen-hptool without CONFIG_MIGRATE
>  docs: Document CPU hotplug
> 
> SUPPORT.md                       |  1 +
> docs/misc/cpu-hotplug.txt        | 50 +++++++++++++++++++++++++
> tools/libs/guest/Makefile.common |  2 +-
> tools/misc/Makefile              |  2 +-
> xen/arch/arm/Kconfig             |  1 +
> xen/arch/arm/gic-vgic.c          |  2 +
> xen/arch/arm/include/asm/irq.h   |  2 +
> xen/arch/arm/irq.c               | 63 +++++++++++++++++++++++++++++++-
> xen/arch/arm/smp.c               |  9 +++++
> xen/arch/arm/smpboot.c           |  6 +++
> xen/arch/arm/vgic.c              | 14 ++++++-
> xen/arch/ppc/stubs.c             |  4 ++
> xen/arch/riscv/stubs.c           |  5 +++
> xen/arch/x86/Kconfig             |  1 +
> xen/arch/x86/include/asm/smp.h   |  3 --
> xen/arch/x86/smp.c               | 33 ++---------------
> xen/arch/x86/sysctl.c            | 12 ++----
> xen/common/Kconfig               |  3 ++
> xen/common/smp.c                 | 34 +++++++++++++++++
> xen/common/sysctl.c              | 45 +++++++++++++++++++++++
> xen/include/xen/smp.h            |  4 ++
> 21 files changed, 248 insertions(+), 48 deletions(-)
> create mode 100644 docs/misc/cpu-hotplug.txt
> 
> -- 
> 2.51.2

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

Posted by Mykyta Poturai 2 days, 17 hours ago

On 03.02.26 18:35, Bertrand Marquis wrote:
> Hi Mykyta,
> 
>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>
>> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
>> several things need to be done:
>>
>> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
>> 2. Enabled building of xen-hptool.
>> 3. Migration of irqs from dying CPUs implemented.
>>
>> Tested on QEMU.
> 
> Do you have a way to validate this on real hardware ?
> Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
> I would not be completely confident to have validated by only testing this in qemu.
> 
> Could you explain how you validated this (what kind of tests you did).
> 
> I would be interested to know how the system behave in the following cases:
> - stopping cpu with guests pinned to specific cores
> - stopping a cpu with a guest pinned and only running on that one
> - checking with passthrough devices (for irq balancing)
> 
> Then a general remark i have is that most of this code (if not all) is always in Xen
> even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
> generate some dead code. Could we protect some of the code not used when
> HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
> equivalent forms ?
> 
> Cheers
> Bertrand
> 

Hi Bertrand

I’ve tested various pinning scenarios, and all of them work fine, as Xen 
forcibly unpins everything that was pinned to the offlined CPU.
I’ve also explicitly checked that interrupts are getting moved to and 
from cores when needed.

I will update the next cover letter to explain testing better and also 
try to test on some hardware to be sure.

Can you please elaborate a little on what exactly you mean by “checking 
with passthrough devices”? I suspect I am not fully getting the idea.


-- 
Mykyta

Re: [PATCH v5 0/5] Implement CPU hotplug on Arm

Posted by Bertrand Marquis 2 days, 17 hours ago

Hi Mykyta,

> On 4 Feb 2026, at 13:58, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
> 
> On 03.02.26 18:35, Bertrand Marquis wrote:
>> Hi Mykyta,
>> 
>>> On 13 Jan 2026, at 09:45, Mykyta Poturai <Mykyta_Poturai@epam.com> wrote:
>>> 
>>> This series implements support for CPU hotplug/unplug on Arm. To achieve this,
>>> several things need to be done:
>>> 
>>> 1. XEN_SYSCTL_CPU_HOTPLUG_* calls implemented on Arm64.
>>> 2. Enabled building of xen-hptool.
>>> 3. Migration of irqs from dying CPUs implemented.
>>> 
>>> Tested on QEMU.
>> 
>> Do you have a way to validate this on real hardware ?
>> Doing hotplug/unplug might have side effects (caches, coherency, interrupts, etc) that
>> I would not be completely confident to have validated by only testing this in qemu.
>> 
>> Could you explain how you validated this (what kind of tests you did).
>> 
>> I would be interested to know how the system behave in the following cases:
>> - stopping cpu with guests pinned to specific cores
>> - stopping a cpu with a guest pinned and only running on that one
>> - checking with passthrough devices (for irq balancing)
>> 
>> Then a general remark i have is that most of this code (if not all) is always in Xen
>> even if CONFIG_CPU_HOTPLUG is not activated in xen config which will probably
>> generate some dead code. Could we protect some of the code not used when
>> HOTPLUG is not activated with if (is_enabled(CONFIG_CPU_HOTPLUG)) or
>> equivalent forms ?
>> 
>> Cheers
>> Bertrand
>> 
> 
> Hi Bertrand
> 
> I’ve tested various pinning scenarios, and all of them work fine, as Xen 
> forcibly unpins everything that was pinned to the offlined CPU.
> I’ve also explicitly checked that interrupts are getting moved to and 
> from cores when needed.
> 
> I will update the next cover letter to explain testing better and also 
> try to test on some hardware to be sure.

Thanks.

> 
> Can you please elaborate a little on what exactly you mean by “checking 
> with passthrough devices”? I suspect I am not fully getting the idea.

If a guest has an hardware device assigned using passthrough and one of
its core is removed, is the guest and hardware device continue to work
well after ?

Cheers
Bertrand

> 
> 
> -- 
> Mykyta