From nobody Fri Oct 31 04:18:55 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1751646883; cv=none; d=zohomail.com; s=zohoarc; b=WJgUIQJ+7n/Xi6ZjPgLi8+TqrQ5niyGjWEYrZvJZ/YkvB0n9BrfrS2FduDcS3SEAPwmu793C3ZtHm8A+zuicEGGNFaQzMeh5+eEsWwzG/Qwm48E3T9bIBG2gOnY8JGDT8RSaU/sF1REzjB4qkEn1/iQZfBXY0tGjvBh1T8dfVWk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1751646883; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=UEIMY77XlHNrZg7VYSte2iM/TK0ot478AJ0j8XeENak=; b=dg7SMjxyaybzFs0ANxkYG9paxiZqd3BNRgysaMynBVhICj/ulEBV/o+wV3zumgXoBtQhuTlJKQg6AQWOhuwlvo9BiXjOSlRqVvFc98U7MJL198jCazGf0QThhid65GXQdSdWvpsXt59aSlsvgLSbF6GszUeQE25ou7IsDVisCns= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1751646883947384.83506265726476; Fri, 4 Jul 2025 09:34:43 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.1033543.1406903 (Exim 4.92) (envelope-from ) id 1uXjMj-0005w0-Hu; Fri, 04 Jul 2025 16:34:21 +0000 Received: by outflank-mailman (output) from mailman id 1033543.1406903; Fri, 04 Jul 2025 16:34:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uXjMj-0005tv-8E; Fri, 04 Jul 2025 16:34:21 +0000 Received: by outflank-mailman (input) for mailman id 1033543; Fri, 04 Jul 2025 16:34:19 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1uXjMh-0005FA-TG for xen-devel@lists.xenproject.org; Fri, 04 Jul 2025 16:34:19 +0000 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [2a00:1450:4864:20::432]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id baaa5a50-58f4-11f0-b894-0df219b8e170; Fri, 04 Jul 2025 18:34:17 +0200 (CEST) Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-3a4e742dc97so1459899f8f.0 for ; Fri, 04 Jul 2025 09:34:17 -0700 (PDT) Received: from localhost.localdomain (host-92-26-98-202.as13285.net. [92.26.98.202]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-454a9989fcesm59199345e9.16.2025.07.04.09.34.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Jul 2025 09:34:16 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: baaa5a50-58f4-11f0-b894-0df219b8e170 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1751646857; x=1752251657; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UEIMY77XlHNrZg7VYSte2iM/TK0ot478AJ0j8XeENak=; b=KGeXL3Oy6ahDyaHm/mMLRB46/hL9/sX+IPgLUJl/uePqqHceGCzx2DnF2celfh99go B4lghzIZ+iaksAD4gXnVi8oaK+apXcPytpZqA36Xft8NDVNeLOmKfGW89RixLWsiSA2q EHHU6XbVU28AwjwM9+d4UMWUsO+L+637ZHIVg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751646857; x=1752251657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UEIMY77XlHNrZg7VYSte2iM/TK0ot478AJ0j8XeENak=; b=ltmKJb40iK8eVYkSYLiTESNM+KxWofubrOJSdlzDy3dPWbvU2Hks0DtzSAHA9L2wWv LNEOAU5jP2Akkerw9XIm60mT+sGE4J1t2w9VS62TuESZ0Uc8XBq+TzsWCpW8v/A3m3Vm r6+Nt2NYHPZXHyep+Hljxgqy2QbOc93boNDWTpkC58sNMnpDuwVelckbaDn8Jnrq2Ujp 7QpeaqDBGPDGHlolkW5vTQRej6JN9NBlYk4NHfkDt3xSQjTLoQ/uPmkZO7kB3jq8v936 1P0baGR0lB2pjgAS3IlQ82IUYyRUaq2zSHD9NG3d2YflUgC2ooVxXTphibEryFFVocP8 9hhg== X-Gm-Message-State: AOJu0YzzWQUvWZvdWfIev23AP8VqEUKPy9ZTY7LMCrN+N6mc9YdEaE8G qt3gFWb1eX8w0aBB3bi4UQxNMDhUgJY8YAkZPDa6J7it8VxrK/XMyMOB5g54hZgyakZC6e7F+n9 +k+QGUhpW1Q== X-Gm-Gg: ASbGnctcvujItHbTXRzGrer6G9M9f/idX5nIljgE8o2xm8/alRfS62RUk1zzco/fdNS FFK1vxlvexephH8bIqiu8oVv/ztSlop1m+/SZO66Ustt+sON5gn/HuDLyuFtMZ88+qr4hHnpA+f IhnjrJT6an11zctR/HFaavoifwtsAuceVIHNAYhyd71sr0yDHpoaJsN34HDoKtDVqEXiikQ+jSR svEMl8biwwYfu4/+y96g/SnrTnNiHNmHoXSbXKX84q9gSTifPeh8hxoEzAXgOR+8ip8myw/pZRJ vw6CJwNO8hiicPH4DrKP5k4KZTPHkGTWmcGpHgz/Rfbft1LzfX2AfCU2KAb0Ubt6/fgW3ytUkGL z71cT9ciqWq/jEkBN8zucjItRD6URK2ptSE0= X-Google-Smtp-Source: AGHT+IG45D0AN4G/SGXixEV06FQbvAvWSyVuh3ig7WEuWYHAvIREVbV++swpxV44w5A0x0m0Gzzyyg== X-Received: by 2002:a05:6000:2a86:b0:3b2:e0ad:758c with SMTP id ffacd0b85a97d-3b492584666mr2266089f8f.3.1751646856779; Fri, 04 Jul 2025 09:34:16 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Anthony PERARD , Michal Orzel , Julien Grall , Stefano Stabellini Subject: [PATCH v2 5/6] x86/idle: Implement a new MWAIT IPI-elision algorithm Date: Fri, 4 Jul 2025 17:34:09 +0100 Message-Id: <20250704163410.1412367-6-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250704163410.1412367-1-andrew.cooper3@citrix.com> References: <20250704163410.1412367-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) X-ZM-MESSAGEID: 1751646885076116600 In order elide IPIs, we must be able to identify whether a target CPU is in MWAIT at the point it is woken up. i.e. the store to wake it up must also identify the state. Create a new in_mwait variable beside __softirq_pending, so we can use a CMPXCHG to set the softirq while also observing the status safely. Impleme= nt an x86 version of arch_pend_softirq() which does this. In mwait_idle_with_hints(), advertise in_mwait, with an explanation of precisely what it means. X86_BUG_MONITOR can be accounted for simply by not advertising in_mwait. Signed-off-by: Andrew Cooper Acked-by: Roger Pau Monn=C3=A9 --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Anthony PERARD CC: Michal Orzel CC: Julien Grall CC: Stefano Stabellini This is modelled after Linux's TIF_NEED_RESCHED (single bit equivelent of a= ll of __softirq_pending), and TIF_POLLING_NRFLAG (arch-neutral "in_mwait"). In Linux, they're both in the same flags field, which adds complexity. In Xen, __softirq_pending is already unsigned long for everything other than x= 86, so adding an arch-neutral "in_mwait" would need wider changes. v2: * Fix cmpxchg() expression. * Use BUILD_BUG_ON()s to check opencoding of in_mwait. TODO: We want to introduce try_cmpxchg() which is better at the C and code-= gen level. --- xen/arch/x86/acpi/cpu_idle.c | 20 ++++++++++++- xen/arch/x86/include/asm/hardirq.h | 14 ++++++++- xen/arch/x86/include/asm/softirq.h | 48 ++++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86/acpi/cpu_idle.c b/xen/arch/x86/acpi/cpu_idle.c index 4f69df5a7438..c5d7a6c6fe2a 100644 --- a/xen/arch/x86/acpi/cpu_idle.c +++ b/xen/arch/x86/acpi/cpu_idle.c @@ -443,7 +443,21 @@ __initcall(cpu_idle_key_init); void mwait_idle_with_hints(unsigned int eax, unsigned int ecx) { unsigned int cpu =3D smp_processor_id(); - const unsigned int *this_softirq_pending =3D &softirq_pending(cpu); + irq_cpustat_t *stat =3D &irq_stat[cpu]; + const unsigned int *this_softirq_pending =3D &stat->__softirq_pending; + + /* + * By setting in_mwait, we promise to other CPUs that we'll notice cha= nges + * to __softirq_pending without being sent an IPI. We achieve this by + * either not going to sleep, or by having hardware notice on our beha= lf. + * + * Some errata exist where MONITOR doesn't work properly, and the + * workaround is to force the use of an IPI. Cause this to happen by + * simply not advertising ourselves as being in_mwait. + */ + alternative_io("movb $1, %[in_mwait]", + "", X86_BUG_MONITOR, + [in_mwait] "=3Dm" (stat->in_mwait)); =20 monitor(this_softirq_pending, 0, 0); =20 @@ -455,6 +469,10 @@ void mwait_idle_with_hints(unsigned int eax, unsigned = int ecx) mwait(eax, ecx); spec_ctrl_exit_idle(info); } + + alternative_io("movb $0, %[in_mwait]", + "", X86_BUG_MONITOR, + [in_mwait] "=3Dm" (stat->in_mwait)); } =20 static void acpi_processor_ffh_cstate_enter(struct acpi_processor_cx *cx) diff --git a/xen/arch/x86/include/asm/hardirq.h b/xen/arch/x86/include/asm/= hardirq.h index f3e93cc9b507..1647cff04dc8 100644 --- a/xen/arch/x86/include/asm/hardirq.h +++ b/xen/arch/x86/include/asm/hardirq.h @@ -5,7 +5,19 @@ #include =20 typedef struct { - unsigned int __softirq_pending; + /* + * The layout is important. Any CPU can set bits in __softirq_pending, + * but in_mwait is a status bit owned by the CPU. softirq_mwait_raw m= ust + * cover both, and must be in a single cacheline. + */ + union { + struct { + unsigned int __softirq_pending; + bool in_mwait; + }; + uint64_t softirq_mwait_raw; + }; + unsigned int __local_irq_count; unsigned int nmi_count; unsigned int mce_count; diff --git a/xen/arch/x86/include/asm/softirq.h b/xen/arch/x86/include/asm/= softirq.h index e4b194f069fb..55b65c9747b1 100644 --- a/xen/arch/x86/include/asm/softirq.h +++ b/xen/arch/x86/include/asm/softirq.h @@ -1,6 +1,8 @@ #ifndef __ASM_SOFTIRQ_H__ #define __ASM_SOFTIRQ_H__ =20 +#include + #define NMI_SOFTIRQ (NR_COMMON_SOFTIRQS + 0) #define TIME_CALIBRATE_SOFTIRQ (NR_COMMON_SOFTIRQS + 1) #define VCPU_KICK_SOFTIRQ (NR_COMMON_SOFTIRQS + 2) @@ -9,4 +11,50 @@ #define HVM_DPCI_SOFTIRQ (NR_COMMON_SOFTIRQS + 4) #define NR_ARCH_SOFTIRQS 5 =20 +/* + * Ensure softirq @nr is pending on @cpu. Return true if an IPI can be + * skipped, false if the IPI cannot be skipped. + * + * We use a CMPXCHG covering both __softirq_pending and in_mwait, in order= to + * set softirq @nr while also observing in_mwait in a race-free way. + */ +static always_inline bool arch_set_softirq(unsigned int nr, unsigned int c= pu) +{ + uint64_t *ptr =3D &irq_stat[cpu].softirq_mwait_raw; + uint64_t prev, old, new; + unsigned int softirq =3D 1U << nr; + + old =3D ACCESS_ONCE(*ptr); + + for ( ;; ) + { + if ( old & softirq ) + /* Softirq already pending, nothing to do. */ + return true; + + new =3D old | softirq; + + prev =3D cmpxchg(ptr, old, new); + if ( prev =3D=3D old ) + break; + + old =3D prev; + } + + /* + * We have caused the softirq to become pending. If in_mwait was set,= the + * target CPU will notice the modification and act on it. + * + * We can't access the in_mwait field nicely, so use some BUILD_BUG_ON= ()'s + * to cross-check the (1UL << 32) opencoding. + */ + BUILD_BUG_ON(sizeof(irq_stat[0].softirq_mwait_raw) !=3D 8); + BUILD_BUG_ON((offsetof(irq_cpustat_t, in_mwait) - + offsetof(irq_cpustat_t, softirq_mwait_raw)) !=3D 4); + + return new & (1UL << 32) /* in_mwait */; + +} +#define arch_set_softirq arch_set_softirq + #endif /* __ASM_SOFTIRQ_H__ */ --=20 2.39.5