From nobody Wed Apr 24 14:38:18 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1575027040; cv=none; d=zohomail.com; s=zohoarc; b=UE5YkIExix5mho/txZ3hMUFNue5clRiSELXO+Tne0bGfaTS2MX6wkA77XgMfaxRXzLYjESGzwUv5Q5z1phynmNN9p7ZAv+3xz95Z2R4HBsmCc5Ni08er5w+MoA+8Tf1MIYatpHPlHSMf97/7oa+u5z2haAr0YN25yOCDsKRfiYM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1575027040; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Ko1td7cSv1utd9hDZvp/b+XZcqloMFaAqpH6AyZyYaQ=; b=jbTt76qAX8uT/NZwTdCC3SB+Tn3wGwRLFjLR7MiY9ZJeIDMwwxdRRAxVyMsuAOzmRpniYA/ZQov3jesXI83HHtMy7FuFbGdklJ6csUcWBn3TUpdWDQilcfA4PH4uRsMPwWZTObBpJk09Ck9NniEHt9ormkErdckUCIECyefUKOc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1575027040180201.8155312017792; Fri, 29 Nov 2019 03:30:40 -0800 (PST) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iaeTC-0006rz-1I; Fri, 29 Nov 2019 11:29:54 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iaeTB-0006ro-M8 for xen-devel@lists.xenproject.org; Fri, 29 Nov 2019 11:29:53 +0000 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 88b44b1a-129b-11ea-a3e3-12813bfff9fa; Fri, 29 Nov 2019 11:29:41 +0000 (UTC) X-Inumbo-ID: 88b44b1a-129b-11ea-a3e3-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1575026981; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=jsBIuccbgrAsEdfwh7i25IjVofpn1BZsmhIrRdMfrQE=; b=ZqrJrvMWwNrNGqtPbLEBLuCMcROzlcWXFupodHlPtETEyrR48r7sYZuc 66Q8NgSgtbtECxmlQ2omqr+v8XWaka/PGVIPdZm10iFQBQDXFT3WiHwHv wVcXpmmzPidaX02ykMxtS+67BH3TmLGZq46txt3Hw+H1jN2rk3nibWqzy s=; Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: AYWYG/PwhFQ1bHMEllGoKRkw3cG+jLqx9DjapJQBra9xchwBJERXk6tqGvaxxyV84YcjKLCQUV kvE1TFqx4ymw4Cfd/cNRrBRPDge7t3iyqEbo/4hq/ZzPIb+ihqvUxCV4rr3taJOZF3b47oRHI2 BP5xjwZcRhx+6sx+AFIBvfaBghp64MbMSXmYqhtPMWmO3wm0ysgnEUbSmsF8Y+/jVfy4IYZIpK cuufBElraTy7VFYNkZh4v+5ogXdaYcBVNx13D9o56+7dcUjHz5uDZteN78pRTw4WyFPq0EVvL2 S6M= X-SBRS: 2.7 X-MesageID: 9338027 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,257,1571716800"; d="scan'208";a="9338027" From: Roger Pau Monne To: Date: Fri, 29 Nov 2019 12:28:51 +0100 Message-ID: <20191129112851.19273-5-roger.pau@citrix.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191129112851.19273-1-roger.pau@citrix.com> References: <20191129112851.19273-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 4/4] x86/apic: allow enabling x2APIC mode regardless of interrupt remapping X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Wei Liu , Jan Beulich , Roger Pau Monne Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) x2APIC mode doesn't mandate interrupt remapping, and hence can be enabled independently. This patch enables x2APIC when available, regardless of whether there's interrupt remapping support. This is beneficial specially when running on virtualized environments, since it reduces the amount of vmexits. For example when sending an IPI in xAPIC mode Xen performs at least 3 different accesses to the APIC MMIO region, while when using x2APIC mode a single wrmsr is used. The following numbers are from a lock profiling of a Xen PV shim running a Linux PV kernel with 32 vCPUs and xAPIC mode: (XEN) Global lock flush_lock: addr=3Dffff82d0804af1c0, lockval=3D03190319, = not locked (XEN) lock:656153(892606463454), block:602183(9495067321843) Average lock time: 1360363ns Average block time: 15767743ns While the following are from the same configuration but with the shim using x2APIC mode: (XEN) Global lock flush_lock: addr=3Dffff82d0804b01c0, lockval=3D1adb1adb, = not locked (XEN) lock:1841883(1375128998543), block:1658716(10193054890781) Average lock time: 746588ns Average block time: 6145147ns Enabling x2APIC has halved the average lock time, thus reducing contention. Signed-off-by: Roger Pau Monn=C3=A9 --- Changes since v1: - Fix error paths of iommu_enable_x2apic call in x2apic_bsp_setup. --- NB: should enabling x2APIC without interrupt remapping be limited to running on virtualized environments? The bigger performance benefit is indeed achieved when using x2APIC on virt environments, but I also don't see why we wouldn't want to try using it everywhere where it's supported. --- xen/arch/x86/apic.c | 89 +++++++++++++++++++++------------------------ 1 file changed, 42 insertions(+), 47 deletions(-) diff --git a/xen/arch/x86/apic.c b/xen/arch/x86/apic.c index a8ee18636f..eaf8924585 100644 --- a/xen/arch/x86/apic.c +++ b/xen/arch/x86/apic.c @@ -492,7 +492,8 @@ static void __enable_x2apic(void) =20 static void resume_x2apic(void) { - iommu_enable_x2apic(); + if ( iommu_supports_x2apic() ) + iommu_enable_x2apic(); __enable_x2apic(); } =20 @@ -695,7 +696,8 @@ int lapic_suspend(void) =20 local_irq_save(flags); disable_local_APIC(); - iommu_disable_x2apic(); + if ( iommu_supports_x2apic() ) + iommu_disable_x2apic(); local_irq_restore(flags); return 0; } @@ -875,56 +877,46 @@ void __init x2apic_bsp_setup(void) printk("x2APIC: Already enabled by BIOS: Ignoring cmdline disable.= \n"); } =20 - if ( !iommu_supports_x2apic() ) + if ( iommu_supports_x2apic() ) { - if ( !x2apic_enabled ) + if ( (ioapic_entries =3D alloc_ioapic_entries()) =3D=3D NULL ) { - printk("Not enabling x2APIC: depends on IOMMU support\n"); - return; + printk("Allocate ioapic_entries failed\n"); + goto out; } - panic("x2APIC: already enabled by BIOS, but no IOMMU support\n"); - } =20 - if ( (ioapic_entries =3D alloc_ioapic_entries()) =3D=3D NULL ) - { - printk("Allocate ioapic_entries failed\n"); - goto out; - } - - if ( save_IO_APIC_setup(ioapic_entries) ) - { - printk("Saving IO-APIC state failed\n"); - goto out; - } + if ( save_IO_APIC_setup(ioapic_entries) ) + { + printk("Saving IO-APIC state failed\n"); + goto out; + } =20 - mask_8259A(); - mask_IO_APIC_setup(ioapic_entries); + mask_8259A(); + mask_IO_APIC_setup(ioapic_entries); =20 - switch ( iommu_enable_x2apic() ) - { - case 0: - intremap_enabled =3D true; - break; - case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */ - if ( !x2apic_enabled ) + switch ( iommu_enable_x2apic() ) { + case 0: + intremap_enabled =3D true; + break; + + case -ENXIO: /* ACPI_DMAR_X2APIC_OPT_OUT set */ + if ( x2apic_enabled ) + panic("IOMMU requests xAPIC mode, but x2APIC already enabl= ed by firmware\n"); + printk("Not enabling x2APIC (upon firmware request)\n"); intremap_enabled =3D false; goto restore_out; + + default: + printk(XENLOG_ERR "Failed to enable Interrupt Remapping\n"); + intremap_enabled =3D false; + break; } - /* fall through */ - default: - if ( x2apic_enabled ) - panic("Interrupt remapping could not be enabled while " - "x2APIC is already enabled by BIOS\n"); - - printk(XENLOG_ERR - "Failed to enable Interrupt Remapping: Will not enable x2AP= IC.\n"); - intremap_enabled =3D false; - goto restore_out; - } =20 - force_iommu =3D 1; + if ( intremap_enabled ) + force_iommu =3D 1; + } =20 if ( !x2apic_enabled ) { @@ -938,13 +930,16 @@ void __init x2apic_bsp_setup(void) printk("Switched to APIC driver %s\n", genapic.name); =20 restore_out: - /* - * NB: do not use raw mode when restoring entries if the iommu has been - * enabled during the process, because the entries need to be translat= ed - * and added to the remapping table in that case. - */ - restore_IO_APIC_setup(ioapic_entries, !intremap_enabled); - unmask_8259A(); + if ( iommu_supports_x2apic() ) + { + /* + * NB: do not use raw mode when restoring entries if the iommu has= been + * enabled during the process, because the entries need to be tran= slated + * and added to the remapping table in that case. + */ + restore_IO_APIC_setup(ioapic_entries, !intremap_enabled); + unmask_8259A(); + } =20 out: if ( ioapic_entries ) --=20 2.24.0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel