[v9] support CPU hot-unplug

[edk2-devel] [PATCH v9 00/10] support CPU hot-unplug

Posted by Ankur Arora 3 years, 1 month ago

Hi,

This series adds OVMF support for CPU hot-unplug.

QEMU secureboot hot-unplug logic corresponding to this is in upstream.
Also posted here:
  https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/

Testing (with QEMU 5.2.50):
 - Stable with randomized CPU plug/unplug (guest maxcpus=33,128)
 - Synthetic tests with simultaneous multi CPU hot-unplug

Also at:
  github.com/terminus/edk2/ hot-unplug-v9

Changelog:

v9:
  - Rebased on top of edd46cd407ea
  - Clarify comments around memory-barriers in patches 7, 8, 9
  - Address other review comments from v8

v8:
  - Fixes a couple of ECC issues in the code (in patches 7, 9)
  URL: https://patchew.org/EDK2/20210222071928.1401820-1-ankur.a.arora@oracle.com/

v7:
  - Address review comments from v6.
  - Fix ejection bug where we were using APIC ID to do the ejection
    rather than the Qemu Selector.
  - Describes safety properties and ordering needed for concurrent
    accesses to CPU_HOT_EJECT_DATA->QemuSelectorMap, and
    CPU_HOT_EJECT_DATA->Handler.
  URL: https://patchew.org/EDK2/20210219090444.1332380-1-ankur.a.arora@oracle.com/

v6:
  - addresses v5 review comments.
  URL: https://patchew.org/EDK2/20210129005950.467638-1-ankur.a.arora@oracle.com/

v5:
  - fixes ECC errors (all but one in "OvmfPkg/CpuHotplugSmm: add
    add Qemu Cpu Status helper").
  URL: https://patchew.org/EDK2/20210126064440.299596-1-ankur.a.arora@oracle.com/

v4:
  - Gets rid of unnecessary UefiCpuPkg changes
  URL: https://patchew.org/EDK2/20210118063457.358581-1-ankur.a.arora@oracle.com/

v3:
  - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
    and OvmfPkg/CpuHotplugSmm
  - Cleaner split of the hot-unplug code
  URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/

v2:
  - Do the ejection via SmmCpuFeaturesRendezvousExit()
  URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/

RFC:
  URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/

Please review.

Thanks
Ankur

Ankur Arora (10):
  OvmfPkg/CpuHotplugSmm: refactor hotplug logic
  OvmfPkg/CpuHotplugSmm: collect hot-unplug events
  OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
  OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
  OvmfPkg: define CPU_HOT_EJECT_DATA
  OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
  OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
  OvmfPkg/CpuHotplugSmm: add EjectCpu()
  OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
  OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug

 OvmfPkg/OvmfPkg.dec                                |   4 +
 OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf            |   2 +
 .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf        |   4 +
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.h                  |   7 +
 OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h  |   2 +
 OvmfPkg/Include/Pcd/CpuHotEjectData.h              |  60 +++
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c                 | 576 +++++++++++++++++----
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                  | 123 +++--
 .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c  | 111 ++++
 OvmfPkg/SmmControl2Dxe/SmiFeatures.c               |  18 +-
 10 files changed, 775 insertions(+), 132 deletions(-)
 create mode 100644 OvmfPkg/Include/Pcd/CpuHotEjectData.h

-- 
2.9.3



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72702): https://edk2.groups.io/g/devel/message/72702
Mute This Topic: https://groups.io/mt/81273600/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v9 00/10] support CPU hot-unplug

Posted by Laszlo Ersek 3 years, 1 month ago

On 03/12/21 07:26, Ankur Arora wrote:
> Hi,
>
> This series adds OVMF support for CPU hot-unplug.
>
> QEMU secureboot hot-unplug logic corresponding to this is in upstream.
> Also posted here:
>   https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
>
> Testing (with QEMU 5.2.50):
>  - Stable with randomized CPU plug/unplug (guest maxcpus=33,128)
>  - Synthetic tests with simultaneous multi CPU hot-unplug
>
> Also at:
>   github.com/terminus/edk2/ hot-unplug-v9
>
> Changelog:
>
> v9:
>   - Rebased on top of edd46cd407ea
>   - Clarify comments around memory-barriers in patches 7, 8, 9
>   - Address other review comments from v8

Merged as commit range 4751a48aeb2a..f3bdfc41866e, via
<https://github.com/tianocore/edk2/pull/1494>, with the following R-b's
/ comments / light tweaks added:

>  1:  4b4466ed7c44 =  1:  0cb242e33693 OvmfPkg/CpuHotplugSmm: refactor hotplug logic
>  2:  719828efdebe !  2:  a752dd07466c OvmfPkg/CpuHotplugSmm: collect hot-unplug events
>     @@ -20,6 +20,7 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-3-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>      --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>  3:  7d732efcb7af =  3:  2d92e052c3af OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
>  4:  64649eeeee42 =  4:  15e6ae8ea400 OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>  5:  7190a5d134a7 !  5:  8ade9d425a6e OvmfPkg: define CPU_HOT_EJECT_DATA
>     @@ -15,6 +15,7 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-6-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
>      --- a/OvmfPkg/OvmfPkg.dec
>  6:  1fd019dac476 !  6:  b6d5996706dd OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
>     @@ -22,6 +22,7 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-7-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
>      --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
>  7:  c8758d9b9764 !  7:  af9c77e151fa OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
>     @@ -19,6 +19,7 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-8-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
>      --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
>  8:  5ff8e8700678 !  8:  30c69d2cfa63 OvmfPkg/CpuHotplugSmm: add EjectCpu()
>     @@ -19,6 +19,7 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-9-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
>      --- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
>  9:  d6660f8f2f14 !  9:  f05328886302 OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
>     @@ -27,6 +27,9 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-10-ankur.a.arora@oracle.com>
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>     +    [lersek@redhat.com: unneeded inner QemuSelector declaration in EjectCpu()
>     +     triggers VS warning #4456 (local variable shadowed); remove it]
>
>      diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>      --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>     @@ -102,8 +105,6 @@
>      +    UINT32 Idx;
>      +
>      +    for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
>     -+      UINT64 QemuSelector;
>     -+
>      +      QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx];
>      +
>      +      if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) {
> 10:  0a9a1cd40b98 ! 10:  f3bdfc41866e OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
>     @@ -14,6 +14,9 @@
>          Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>          Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>          Message-Id: <20210312062656.2477515-11-ankur.a.arora@oracle.com>
>     +    [lersek@redhat.com: preserve the empty line between the ICH9_LPC_SMI_F_*
>     +     group of macro definitions and the SCRATCH_BUFFER type definition]
>     +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
>      diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
>      --- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
>     @@ -22,16 +25,15 @@
>       // "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
>       //
>       #define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
>     --
>      +//
>      +// The following bit value stands for "enable CPU hot-unplug, and inject an SMI
>      +// with control value ICH9_APM_CNT_CPU_HOTPLUG upon hot-unplug", in the
>      +// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
>      +//
>      +#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
>     +
>       //
>       // Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
>     - // for the S3 boot script fragment to write to and read from.
>      @@
>         QemuFwCfgReadBytes (sizeof mSmiFeatures, &mSmiFeatures);
>

Thanks for the contribution,
Laszlo



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72875): https://edk2.groups.io/g/devel/message/72875
Mute This Topic: https://groups.io/mt/81273600/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [PATCH v9 00/10] support CPU hot-unplug

Posted by Ankur Arora 3 years, 1 month ago

On 2021-03-16 7:07 a.m., Laszlo Ersek wrote:
> On 03/12/21 07:26, Ankur Arora wrote:
>> Hi,
>>
>> This series adds OVMF support for CPU hot-unplug.
>>
>> QEMU secureboot hot-unplug logic corresponding to this is in upstream.
>> Also posted here:
>>    https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
>>
>> Testing (with QEMU 5.2.50):
>>   - Stable with randomized CPU plug/unplug (guest maxcpus=33,128)
>>   - Synthetic tests with simultaneous multi CPU hot-unplug
>>
>> Also at:
>>    github.com/terminus/edk2/ hot-unplug-v9
>>
>> Changelog:
>>
>> v9:
>>    - Rebased on top of edd46cd407ea
>>    - Clarify comments around memory-barriers in patches 7, 8, 9
>>    - Address other review comments from v8
> 
> Merged as commit range 4751a48aeb2a..f3bdfc41866e, via
> <https://github.com/tianocore/edk2/pull/1494>, with the following R-b's
> / comments / light tweaks added:

Awesome! Thanks Laszlo.

And, thanks especially for the thorough reviews.

Ankur

> 
>>   1:  4b4466ed7c44 =  1:  0cb242e33693 OvmfPkg/CpuHotplugSmm: refactor hotplug logic
>>   2:  719828efdebe !  2:  a752dd07466c OvmfPkg/CpuHotplugSmm: collect hot-unplug events
>>      @@ -20,6 +20,7 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-3-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>>       --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>>   3:  7d732efcb7af =  3:  2d92e052c3af OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
>>   4:  64649eeeee42 =  4:  15e6ae8ea400 OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>>   5:  7190a5d134a7 !  5:  8ade9d425a6e OvmfPkg: define CPU_HOT_EJECT_DATA
>>      @@ -15,6 +15,7 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-6-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
>>       --- a/OvmfPkg/OvmfPkg.dec
>>   6:  1fd019dac476 !  6:  b6d5996706dd OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
>>      @@ -22,6 +22,7 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-7-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
>>       --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
>>   7:  c8758d9b9764 !  7:  af9c77e151fa OvmfPkg/SmmCpuFeaturesLib: call CPU hot-eject handler
>>      @@ -19,6 +19,7 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-8-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
>>       --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
>>   8:  5ff8e8700678 !  8:  30c69d2cfa63 OvmfPkg/CpuHotplugSmm: add EjectCpu()
>>      @@ -19,6 +19,7 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-9-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
>>       --- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
>>   9:  d6660f8f2f14 !  9:  f05328886302 OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject
>>      @@ -27,6 +27,9 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-10-ankur.a.arora@oracle.com>
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>      +    [lersek@redhat.com: unneeded inner QemuSelector declaration in EjectCpu()
>>      +     triggers VS warning #4456 (local variable shadowed); remove it]
>>
>>       diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>>       --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
>>      @@ -102,8 +105,6 @@
>>       +    UINT32 Idx;
>>       +
>>       +    for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
>>      -+      UINT64 QemuSelector;
>>      -+
>>       +      QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx];
>>       +
>>       +      if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) {
>> 10:  0a9a1cd40b98 ! 10:  f3bdfc41866e OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
>>      @@ -14,6 +14,9 @@
>>           Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
>>           Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
>>           Message-Id: <20210312062656.2477515-11-ankur.a.arora@oracle.com>
>>      +    [lersek@redhat.com: preserve the empty line between the ICH9_LPC_SMI_F_*
>>      +     group of macro definitions and the SCRATCH_BUFFER type definition]
>>      +    Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>>       diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
>>       --- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
>>      @@ -22,16 +25,15 @@
>>        // "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
>>        //
>>        #define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
>>      --
>>       +//
>>       +// The following bit value stands for "enable CPU hot-unplug, and inject an SMI
>>       +// with control value ICH9_APM_CNT_CPU_HOTPLUG upon hot-unplug", in the
>>       +// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
>>       +//
>>       +#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
>>      +
>>        //
>>        // Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
>>      - // for the S3 boot script fragment to write to and read from.
>>       @@
>>          QemuFwCfgReadBytes (sizeof mSmiFeatures, &mSmiFeatures);
>>
> 
> Thanks for the contribution,
> Laszlo
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72926): https://edk2.groups.io/g/devel/message/72926
Mute This Topic: https://groups.io/mt/81273600/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-