From nobody Thu Nov 14 23:04:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+72710+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+72710+1787277+3901457@groups.io; arc=fail (BodyHash is different from the expected one); dmarc=fail(p=none dis=none) header.from=oracle.com Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1615530448754886.5490639787704; Thu, 11 Mar 2021 22:27:28 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id DbmPYY1788612xcEzkrQ013V; Thu, 11 Mar 2021 22:27:26 -0800 X-Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by mx.groups.io with SMTP id smtpd.web08.4834.1615530440956790408 for ; Thu, 11 Mar 2021 22:27:21 -0800 X-Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12C6AxGW179410; Fri, 12 Mar 2021 06:27:18 GMT X-Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 37415rgyer-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Mar 2021 06:27:18 +0000 X-Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 12C6P9bv055196; Fri, 12 Mar 2021 06:27:17 GMT X-Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2109.outbound.protection.outlook.com [104.47.55.109]) by userp3020.oracle.com with ESMTP id 374kgw3bs0-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Mar 2021 06:27:17 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FP+sNkfLaEWGlUEhxfy+6nfkAgDh7+E047oYjoYDetGZ/5kC/GNSy6PD+KM8xpkcAigiOU2PO2q3/KGNMLy0ZW6YWnIzZgkq9eReIGxo0LxxAnSsCH9vmCvNErvAiIxULGWoL+AJvtYNiR7c/wT942hGFU3TxDKJToZuYu6wdNSg0VesoxQY0HvCrYgzGDlGqF+ART3OaUk9/Fs1wvW1AIe13Gb++JZmSIIh+BqP5zuQw6Z4PyI8Jcy6tKN/5AcLw191j7JTCOyNGOBZrc08t/uVCZr8+z1W6UY4vjTvF2fekR9RaVTM8l1VRgWNiNxBXfAKof48UIMRRLF0axB7MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=41U0p8zpgTKCTq//Ujgd7ZD8QZAhgkb1JaTATjLn7MA=; b=GVULVwyuHeWfKRKNrdP7wV3LM8v5y5BOKr1T44R+pGFJJjZ6IUQO+hfmCNmTcP0K1kE6afISWStO5PD/1Pa+BLyWMIYhpp3Q9hvBLAaUmNB0iSNCjEx2UJMtdr+uWqvbePkqaKzY3A8QXrAdemNt/K1tWXLWEsWyTeGridREX0BDHoMf3JcGp/UzXq+SQ11oZQh4vzw5ayJ5dtuxRfZ/2lz2dmH7Almgf4bgenZCUFqyOtjueNJMteUl3Wz490D8KykQ8CmjBx4N1/xxLq+wGHDSjRPqsa4aCENm9kbh/HWkSTMf97JBfrXmjP6L25A9ayeoFhp9NIHxloOnk0HHDA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none X-Received: from SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) by SJ0PR10MB4544.namprd10.prod.outlook.com (2603:10b6:a03:2ad::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.31; Fri, 12 Mar 2021 06:27:16 +0000 X-Received: from SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::a021:790:7ce6:6f16]) by SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::a021:790:7ce6:6f16%6]) with mapi id 15.20.3912.030; Fri, 12 Mar 2021 06:27:16 +0000 From: "Ankur Arora" To: devel@edk2.groups.io Cc: lersek@redhat.com, imammedo@redhat.com, boris.ostrovsky@oracle.com, Ankur Arora , Jordan Justen , Ard Biesheuvel , Aaron Young Subject: [edk2-devel] [PATCH v9 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject Date: Thu, 11 Mar 2021 22:26:55 -0800 Message-Id: <20210312062656.2477515-10-ankur.a.arora@oracle.com> In-Reply-To: <20210312062656.2477515-1-ankur.a.arora@oracle.com> References: <20210312062656.2477515-1-ankur.a.arora@oracle.com> X-Originating-IP: [148.87.23.12] X-ClientProxiedBy: MWHPR02CA0018.namprd02.prod.outlook.com (2603:10b6:300:4b::28) To SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 X-Received: from prion.us.oracle.com (148.87.23.12) by MWHPR02CA0018.namprd02.prod.outlook.com (2603:10b6:300:4b::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 12 Mar 2021 06:27:15 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 884db05b-4ee1-42ba-984b-08d8e51fe184 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4544: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info: 1+9BZIlsAbToHGJoVlaSxlqCE8Je07I24gdHEqNFdte/VHU1la+nRkINvpEVOzJMhwRZjoJyis560Bdnf72UxJru0WDCKFVjhjYrcqhV2Oylj5CpKvzSmZdPyTLlUv/219SyPYtApaKuJoLmicxWSz3+lXS2gtJfOn6gJJESPNjaN0fWCWyW4ApAEyLzy8UQUZK3ZI0cF1k23cHNOMTUXY1v8GXK2wVi0+jssbX5oL8qvKJh+SDNp3iTusVR5uxn8kBam4mx20Y96MLDMxCV66abkvefMYf/70xeCBZFUMere04VdYiRFgi3Mgvq3hLNtjej3eKIr26BQKBj4qqPbc3LI7Q3C8SVzaYitPCDMgDGqr1UjqJ0cZIBNpjkJGTJjDMpfNKTs3OD+DARw1LEwlb71YGtVxkR8IvbXum7wn0KL6R+AhGuuiCI/2eJgIzQUrXo1HkHmdN3S7HqLrxdNXQPo3Jb+CoOIk23n9zVspcY7sdQ1axWaJqivf6MtdqjoTMXbqmHmbd5ik/rWJQQOO9jOrLu23kafVLtU0Nsa3txsFgFOoJhAOPw4+tE+1doj845Teedx7+ZOjNUTLCIE7/S6z3V2bdqYQtlaoMDZ481ddgI3CwEftWDSEJ7OmeL8kPoI21pjunuKrfvnUz0qg== X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?qtV0letuzxAyqLGGwN1/vOCSoy5eAu/VnvmVSKAY9yXp+BEJ/dAo1sR8ggf5?= =?us-ascii?Q?vdhEHhnzEi6wlZEVzRUl54+LdPXptbNTqIMKOVN/EJUdKwNo3t3CWrFncyN2?= =?us-ascii?Q?VeFiE81PuV69rxOWYd639uOESr6eM6XxEqgKRD+3P2ksbEX346+BVYiozGs9?= =?us-ascii?Q?chAibKmcvx+iiEqYbJU4Z/umJWoTNAZtqejl4+qubaR9bqvT0IN+6uBJhhuZ?= =?us-ascii?Q?wnSPfrcwA1CT4giDP/PIP7ug9OokznAVAMmVpdtg2xUB0XfjS/O+OQCTRZ1Z?= =?us-ascii?Q?aVpPh627uf85I9/50nWIDLwOguy++G+lIcdjrct1REDRBP62MwF+U6i42+b+?= =?us-ascii?Q?ErPzTIYYhCHFiNuBuSpwzn70XMzswxzSyUf0U0rKNVDHN5JnByEDO+mTmZ+W?= =?us-ascii?Q?tqSIUkctiL+QFPz1sdTiEzm6vIhd/1/3d1943DRElFmjKA62cy1Sz2571Nhp?= =?us-ascii?Q?vIBIhatlTIzVkp5YJrDNjh58SI1La+hrS0n6S57DIV36/yikVoGG9yflWkia?= =?us-ascii?Q?jw3UMpkBIth1OtFt4VTJ7+kjE8DZbZTbf3brWLDVvMaxogS4/hk+Hw/xS8Z6?= =?us-ascii?Q?umqv5Cy31WggMPmWbZgVQK0IWzC3zUvttYMzQc3Vo7SJ+VqbxoxK7b1HXXQy?= =?us-ascii?Q?Mt0usaM47TnN7xD0CHVsOCuqyHoQYlhbt8eVn+v5wRScw4sOjgbzsXnqqxh4?= =?us-ascii?Q?UQIxfWhfLN+64ioUkYdmbs8Vs8LOqFGc+io6T57ySKTn5o7CyuCHxO5uyDqE?= =?us-ascii?Q?SBmV0z66zGibzSd7tfrxhuvCSZsD2p/QpuT2bHb+OhV1709pU/wXFQ5pnaYb?= =?us-ascii?Q?SMcTLqzWtL+a5hbsCpPKEc3kutssdw/zbGg8f9jx3tugOMUDm4W1XSS7IQqQ?= =?us-ascii?Q?Hd032WDwBYXfm7/z3zuNXbY/3lAMovXD+ZPZ3/Nb0Jyw6zpaV5NsjJfhnAHT?= =?us-ascii?Q?aBf4hESsyskj5fxHrTrF4V2A+TVhpyLX+vri1OLO+wBp/+u/EjxqRHVQTOqN?= =?us-ascii?Q?75az16j/ssWzFF9TJ4GiwxZCFBFFd9gGcoKmVdMyLorJUaDxLN8ro32Fnuu+?= =?us-ascii?Q?AKK9n6Putr3wtFXFrQoU5INB3YZlQFWv5xcbFb9S2q3+P+QpjycKXWzksq8E?= =?us-ascii?Q?aF13CSiGRYW57OdrqnCrs610vuUr+Dc/2rbjzljKNSHMp8CJtqWM2KIQL71G?= =?us-ascii?Q?i/8dSgjTf/dO9976+i5Kyf/pM1VbphIHvJm8Nd+tEKSYl4QBX3ABZwkiAdtW?= =?us-ascii?Q?JAD1V/vmH2yuS0mig6JMxa9xNSP6eljlad8zCJVYNKlEomex66gRP/lAXdXh?= =?us-ascii?Q?5hEfGvmxgl/Gcy7mtDUcdCUs?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 884db05b-4ee1-42ba-984b-08d8e51fe184 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4605.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Mar 2021 06:27:16.5057 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qch7qVbepeTmD65/V7gOEO9R6fOZOxmLG0EW02tEB4HqxY7D2Vz0mQopPiTTpVmkA5rmDrXl+qgfXFGCE9GFHPoLSKdVsY3dKC8UzkJnW5w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4544 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ankur.a.arora@oracle.com X-Gm-Message-State: xBnK8qGA78BAawNbl12w5XEtx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1615530446; bh=819qcinTe5pwQVZv8KmYvaOYd90cUPyu2gtwf3Ixc+M=; h=Cc:Content-Type:Date:From:Reply-To:Subject:To; b=ZChBibhQvlP0Tbi5aMYm4E3dUkRswDRwlCzaLXw8AxF6okZ/7Wq4oUrS0p0RssPyaxx dLqpd6BPOW8JpsZAxpTGfh/p0Q0Do+alIM4XFK6tsX9Pj41GdBjzTPmL0lEJI7HmhcZz4 iXUU0htPZGq9jnhlC03pMsvEMirPqTYnsJs= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" Add logic in EjectCpu() to do the actual the CPU ejection. On the BSP, ejection happens by first selecting the CPU via its QemuSelector and then sending the QEMU "eject" command. QEMU in-turn signals the remote VCPU thread which context-switches the CPU out of the SMI handler. Meanwhile the CPU being ejected, waits around in its holding area until it is context-switched out. Note that it is possible that a slow CPU gets ejected before it reaches the wait loop. However, this would never happen before it has executed the "AllCpusInSync" loop in SmiRendezvous(). It can mean that an ejected CPU does not execute code after that point but given that the CPU state will be destroyed by QEMU, the missed cleanup is no great loss. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3D3132 Signed-off-by: Ankur Arora Reviewed-by: Laszlo Ersek --- Notes: Addresses the following comments from v8: =20 (1a,1b) CheckIfBsp(): get rid of ProcessorNum, document retval. (2) Line up IsBsp and ApicBaseMsr (3) s/ongoing SMI iteration/ongoing SMI/ (4) Get rid of the allusions to alignment in the comment in EjectCpu(). () Also reduce some of the repetitive detail in this comment. (5) EjectCpu(): reorder logic to cleanly separate the AP and the BSP po= rtions. (6) Get rid of unnecessary MemoryFence() between QemuCpuhpWrite and clearing of the eject status. (7) Change type of QemuSelector to %Lu in DEBUG statement (8) Get rid of the repetitive comment in SmmCpuFeaturesRendezvousExit(). The necessary parts of this got moved to patch-7. OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 + OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 113 ++++++++++++++++++= ++-- 2 files changed, 108 insertions(+), 6 deletions(-) diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/In= clude/IndustryStandard/QemuCpuHotplug.h index 2ec7a107a64d..d0e83102c13f 100644 --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h +++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h @@ -34,6 +34,7 @@ #define QEMU_CPUHP_STAT_ENABLED BIT0 #define QEMU_CPUHP_STAT_INSERT BIT1 #define QEMU_CPUHP_STAT_REMOVE BIT2 +#define QEMU_CPUHP_STAT_EJECT BIT3 #define QEMU_CPUHP_STAT_FW_REMOVE BIT4 =20 #define QEMU_CPUHP_RW_CMD_DATA 0x8 diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/Cpu= Hotplug.c index 2eeb4567a262..ae3abd525900 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -18,6 +18,7 @@ #include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOC= OL +#include // MSR_IA32_APIC_BASE_REGISTER #include // EFI_STATUS =20 #include "ApicId.h" // APIC_ID @@ -193,12 +194,40 @@ RevokeNewSlot: } =20 /** + EjectCpu needs to know the BSP at SMI exit at a point when + some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn + down. + Reuse the logic from OvmfPkg::PlatformSmmBspElection() to + do that. + + @retval TRUE If the CPU executing this function is the BSP. + + @retval FALSE If the CPU executing this function is an AP. +**/ +STATIC +BOOLEAN +CheckIfBsp ( + VOID + ) +{ + MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr; + BOOLEAN IsBsp; + + ApicBaseMsr.Uint64 =3D AsmReadMsr64 (MSR_IA32_APIC_BASE); + IsBsp =3D (BOOLEAN)(ApicBaseMsr.Bits.BSP =3D=3D 1); + return IsBsp; +} + +/** CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit() on each CPU at exit from SMM. =20 - If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is neither the BSP, nor being ejected, nothing + to be done. If, the executing CPU is being ejected, wait in a halted loop until ejected. + If, the executing CPU is the BSP, set QEMU CPU status to eject + for CPUs being ejected. =20 @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM, and will be used as an index into @@ -214,6 +243,83 @@ EjectCpu ( { UINT64 QemuSelector; =20 + if (CheckIfBsp ()) { + UINT32 Idx; + + for (Idx =3D 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) { + UINT64 QemuSelector; + + QemuSelector =3D mCpuHotEjectData->QemuSelectorMap[Idx]; + + if (QemuSelector !=3D CPU_EJECT_QEMU_SELECTOR_INVALID) { + // + // This to-be-ejected-CPU has already received the BSP's SMI exit + // signal and will execute SmmCpuFeaturesRendezvousExit() + // followed by this callback or is already penned in the + // CpuSleep() loop below. + // + // Tell QEMU to context-switch it out. + // + QemuCpuhpWriteCpuSelector (mMmCpuIo, (UINT32) QemuSelector); + QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECT); + + // + // Now that we've ejected the CPU corresponding to QemuSelectorMap= [Idx], + // clear its eject status to ensure that an invalid future SMI does + // not end up trying a spurious eject or a newly hotplugged CPU do= es + // not get penned in the CpuSleep() loop. + // + // Note that the QemuCpuhpWriteCpuStatus() command above is a writ= e to + // a different address space and uses the EFI_MM_CPU_IO_PROTOCOL. + // + // This means that we are guaranteed that the following assignment + // will not be reordered before the eject. And, so we can safely + // do this write here. + // + mCpuHotEjectData->QemuSelectorMap[Idx] =3D + CPU_EJECT_QEMU_SELECTOR_INVALID; + + DEBUG ((DEBUG_INFO, "%a: Unplugged ProcessorNum %u, " + "QemuSelector %Lu\n", __FUNCTION__, Idx, QemuSelector)); + } + } + + // + // We are done until the next hot-unplug; clear the handler. + // + // mCpuHotEjectData->Handler is a NOP for any CPU not under ejection. + // So, once we are done with all the ejections, we can safely reset it + // here since any CPU dereferencing it would only see either the old + // or the new value (since it is aligned at a natural boundary.) + // + mCpuHotEjectData->Handler =3D NULL; + return; + } + + // + // Reached only on APs + // + + // + // mCpuHotEjectData->QemuSelectorMap[ProcessorNum] is updated + // on the BSP in the ongoing SMI at two places: + // + // - UnplugCpus() where the BSP determines if a CPU is under ejection + // or not. As a comment in UnplugCpus() at set-up, and in + // SmmCpuFeaturesRendezvousExit() where it is dereferenced describe, + // any such updates are guaranteed to be ordered-before the + // dereference below. + // + // - EjectCpu() on the BSP (above) updates QemuSelectorMap[ProcessorNum] + // for a CPU once it's ejected. + // + // The CPU under ejection: might be executing anywhere between the + // AllCpusInSync loop in SmiRendezvous(), to about to dereference + // QemuSelectorMap[ProcessorNum]. + // As described in the comment above where we do the reset, this + // is not a problem since the ejected CPU never sees the after value. + // CPUs not-under ejection: never see any changes so they are fine. + // QemuSelector =3D mCpuHotEjectData->QemuSelectorMap[ProcessorNum]; if (QemuSelector =3D=3D CPU_EJECT_QEMU_SELECTOR_INVALID) { return; @@ -495,11 +601,6 @@ CpuHotplugMmi ( if (EFI_ERROR (Status)) { goto Fatal; } - if (ToUnplugCount > 0) { - DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n", - __FUNCTION__)); - goto Fatal; - } =20 if (PluggedCount > 0) { Status =3D ProcessHotAddedCpus (mPluggedApicIds, PluggedCount); --=20 2.9.3 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#72710): https://edk2.groups.io/g/devel/message/72710 Mute This Topic: https://groups.io/mt/81273610/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-