From nobody Sat Nov 23 16:36:44 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1724767097; cv=none; d=zohomail.com; s=zohoarc; b=YF2i/GM1vzpBSCNLo3xnT+/M19etBlclqkB6akfuwYJ2lUNCN6KPRbvnpVY3QbS9VZWQQhVjhSEgDcCQsicafrksHCtryT73V0e1RcyOJsfseeuCWpQB4Ou8uc57I5LC4DuQtgmcEic/yEt1cGh6cSmtS89wbzhU6MVUrkBferU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1724767097; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=onbkRJjonSldeX+TKHvWoalIHglLaxnF1KDpl/v4wiA=; b=haS5FR3t6Dx7ZNVLzewFgWbKCw8iYmk4VAf5VyJYPwc6YLULCezFzsRBKTbdepKjrH5z1y5zdQ4CJeCXxhQIlOQo3dzVIC5tVrLERh5CMkFW5j8AetULNFp4A+tp10rMjjUupjNG3Vv6vCTBwyREzboZwnvAnbDMO1VwmLpwDNU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1724767097649683.4622104770576; Tue, 27 Aug 2024 06:58:17 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.784071.1193466 (Exim 4.92) (envelope-from ) id 1siwhr-0004tc-ME; Tue, 27 Aug 2024 13:57:59 +0000 Received: by outflank-mailman (output) from mailman id 784071.1193466; Tue, 27 Aug 2024 13:57:59 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1siwhr-0004tL-IY; Tue, 27 Aug 2024 13:57:59 +0000 Received: by outflank-mailman (input) for mailman id 784071; Tue, 27 Aug 2024 13:57:58 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1siwhq-00047p-4h for xen-devel@lists.xenproject.org; Tue, 27 Aug 2024 13:57:58 +0000 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [2a00:1450:4864:20::62c]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 5d3fa71f-647c-11ef-a0b0-8be0dac302b0; Tue, 27 Aug 2024 15:57:57 +0200 (CEST) Received: by mail-ej1-x62c.google.com with SMTP id a640c23a62f3a-a868d7f92feso700694466b.2 for ; Tue, 27 Aug 2024 06:57:57 -0700 (PDT) Received: from andrewcoop.eng.citrite.net ([185.25.67.249]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a86e588adf2sm113898966b.173.2024.08.27.06.57.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Aug 2024 06:57:55 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 5d3fa71f-647c-11ef-a0b0-8be0dac302b0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1724767076; x=1725371876; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=onbkRJjonSldeX+TKHvWoalIHglLaxnF1KDpl/v4wiA=; b=uT8enpQLoIrW8DSrhlqvctPlFAqWtGJTBIKzU71f/nDyntq2WNJZ7P/CP9qehyD9aV wbljrgpk+YhLg38EVpJrbG0n29+BYGMb4o8YEeNEt+ypHA80jhLu+j1gHNPwantqntOO BBrocgVovvIqUgzQ1m0f+QO02ztjRdxUH/HsM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724767076; x=1725371876; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=onbkRJjonSldeX+TKHvWoalIHglLaxnF1KDpl/v4wiA=; b=TvVdn/g1X/JlsZNnfuuwGzYJh7bBL9ImJsYQxaqAU5XwI9+gJLMl9uNe7d8GqIeTu2 XYh1C0ru8iP+crKR1wshdIYmmGAq2qbbCj2EZwtrBdp/hve94NG7Btpnvq2bCJtIebLs PdsfDl7jqeMzjZF30khSzJqPHY4gAFWtTlVHnqrOiqRK/F3tNIRx4VEAPp2IhFOLZquJ 1T2bDzrukgsVcJ7JuHEPVoOf1DhK2EIZlbqCJSbErBxPBkK+I7Za1sNlYbIpJmHRbbjP 3O2yQhAQuqTwkoIlBCV0sycdgxuYiVT94UOLNdFN3JfkqrD+yh+PvIpLWrf/PL7v81C6 G18Q== X-Gm-Message-State: AOJu0YzFst2ANLV8wnjMGsnuEa+BL3L6IKsw69Erk4w7j6aL1IUeRGXQ LwywPqkW2vr6m7lQG6pooCvbznI7K0PVDhsMBtMTxDsb9+PKFGeQTIIwfi8h+No3EUN3PtWEz6/ K X-Google-Smtp-Source: AGHT+IH7hZ1uEw4kAXnd6m6eSnLzFwncr1ABIEpVvj5g0O6NferXHSInSooeNemWXtI/6AQO/5hpRg== X-Received: by 2002:a17:907:6d0f:b0:a86:7e7f:69ab with SMTP id a640c23a62f3a-a86a52b367emr931155566b.15.1724767076098; Tue, 27 Aug 2024 06:57:56 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Stefano Stabellini , Julien Grall , Volodymyr Babchuk , Bertrand Marquis , Michal Orzel , Oleksii Kurochko Subject: [PATCH v2 4/4] x86/vmx: Rewrite vmx_sync_pir_to_irr() to be more efficient Date: Tue, 27 Aug 2024 14:57:46 +0100 Message-Id: <20240827135746.1908070-5-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240827135746.1908070-1-andrew.cooper3@citrix.com> References: <20240827135746.1908070-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) X-ZM-MESSAGEID: 1724767099944116600 There are two issues. First, pi_test_and_clear_on() pulls the cache-line to the CPU and dirties it even if there's nothing outstanding, but the final for_each_set_bit() is O(256) when O(8) would do, and would avoid multiple atomic updates to the same IRR word. Rewrite it from scratch, explaining what's going on at each step. Bloat-o-meter reports 177 -> 145 (net -32), but the better aspect is the removal calls to __find_{first,next}_bit() hidden behind for_each_set_bit(). No functional change. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Stefano Stabellini CC: Julien Grall CC: Volodymyr Babchuk CC: Bertrand Marquis CC: Michal Orzel CC: Oleksii Kurochko The main purpose of this is to get rid of bitmap_for_each(). v2: * Extend the comments --- xen/arch/x86/hvm/vmx/vmx.c | 70 +++++++++++++++++++++++++++++++++----- 1 file changed, 62 insertions(+), 8 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index 12f8a66458db..1baed7e816c4 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -2317,18 +2317,72 @@ static void cf_check vmx_deliver_posted_intr(struct= vcpu *v, u8 vector) =20 static void cf_check vmx_sync_pir_to_irr(struct vcpu *v) { - struct vlapic *vlapic =3D vcpu_vlapic(v); - unsigned int group, i; - DECLARE_BITMAP(pending_intr, X86_NR_VECTORS); + struct pi_desc *desc =3D &v->arch.hvm.vmx.pi_desc; + union { + uint64_t _64[X86_NR_VECTORS / (sizeof(uint64_t) * 8)]; + uint32_t _32[X86_NR_VECTORS / (sizeof(uint32_t) * 8)]; + } vec; + uint32_t *irr; + bool on; =20 - if ( !pi_test_and_clear_on(&v->arch.hvm.vmx.pi_desc) ) + /* + * The PIR is a contended cacheline which bounces between the CPU(s) a= nd + * IOMMU(s). An IOMMU updates the entire PIR atomically, but we can't + * express the same on the CPU side, so care has to be taken. + * + * First, do a plain read of ON. If the PIR hasn't been modified, this + * will keep the cacheline Shared and not pull it Excusive on the curr= ent + * CPU. + */ + if ( !pi_test_on(desc) ) return; =20 - for ( group =3D 0; group < ARRAY_SIZE(pending_intr); group++ ) - pending_intr[group] =3D pi_get_pir(&v->arch.hvm.vmx.pi_desc, group= ); + /* + * Second, if the plain read said that ON was set, we must clear it wi= th + * an atomic action. This will bring the cachline to Exclusive on the + * current CPU. + * + * This should always succeed because noone else should be playing with + * the PIR behind our back, but assert so just in case. + */ + on =3D pi_test_and_clear_on(desc); + ASSERT(on); + + /* + * The cacheline is now Exclusive on the current CPU, and because ON w= as + * set, some other entity (an IOMMU, or Xen on another CPU) has indica= ted + * that at PIR needs re-scanning. + * + * Note: Entities which can't update the entire cacheline atomically + * (i.e. Xen on another CPU) are required to update PIR first, t= hen + * set ON. Therefore, there is a corner case where we may have + * found and processed the PIR updates "last time around" and on= ly + * found ON this time around. This is fine; the logic still + * operates correctly. + * + * Atomically read and clear the entire pending bitmap as fast as we, = to + * reduce the window where another entity may steal the cacheline back + * from us. This is a performance concern, not a correctness concern.= If + * the another entity does steal the cacheline back, we'll just wait to + * get it back again. + */ + for ( unsigned int i =3D 0; i < ARRAY_SIZE(vec._64); ++i ) + vec._64[i] =3D xchg(&desc->pir[i], 0); + + /* + * Finally, merge the pending vectors into IRR. The IRR register is + * scattered in memory, so we have to do this 32 bits at a time. + */ + irr =3D (uint32_t *)&vcpu_vlapic(v)->regs->data[APIC_IRR]; + for ( unsigned int i =3D 0; i < ARRAY_SIZE(vec._32); ++i ) + { + if ( !vec._32[i] ) + continue; =20 - bitmap_for_each ( i, pending_intr, X86_NR_VECTORS ) - vlapic_set_vector(i, &vlapic->regs->data[APIC_IRR]); + asm ( "lock or %[val], %[irr]" + : [irr] "+m" (irr[i * 0x10]) + : [val] "r" (vec._32[i]) ); + } } =20 static bool cf_check vmx_test_pir(const struct vcpu *v, uint8_t vec) --=20 2.39.2