From nobody Fri May 3 04:32:23 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1575913132; cv=none; d=zohomail.com; s=zohoarc; b=YLJrrFKEm6m5XurYPAmwFHBNi0ykTBBQN7n7kbEvZ1h8M5lhJhQIQG8TFu6ZPr5s0YF5aoI6igro/mvnZveblAXHFyNCqTb5bR+V7HvZAlfS7Fek2kaHK0/H9V7gYHuoQRAPdQpVyQIS1jlLuhTuKTuDlwAI5L8h323JGqViPuM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1575913132; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=nrvffSGm0MFnP4GkFBOZ3j2U0F/cQTn1/BgjOGSXoYg=; b=fW+T5MGeHoygwbhgGXWGaUN5lEEz+Kj6pU1OF8eOUE6V+GHVFgnJB9Ssr3yVO1nf6UfWmBFUmcPKTq/7mrwtt6LH8X9sA3FqYx2mlvxdXoQ/6pp9bwMxkcK5l5yqNdXBwWoHyGWC5IKVCOt3d3K7/lembCEo93OvbQYxW0NBEio= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1575913132104885.4103385277745; Mon, 9 Dec 2019 09:38:52 -0800 (PST) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ieMzE-0001Ub-UB; Mon, 09 Dec 2019 17:38:20 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ieMzD-0001UT-CH for xen-devel@lists.xenproject.org; Mon, 09 Dec 2019 17:38:19 +0000 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id af86e3da-1aaa-11ea-88e7-bc764e2007e4; Mon, 09 Dec 2019 17:38:18 +0000 (UTC) X-Inumbo-ID: af86e3da-1aaa-11ea-88e7-bc764e2007e4 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1575913099; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=4shP+jxSILsDmW0PdUqOApthRN+7XdBqnWgH+mKtZg4=; b=dgo+aRjfAHSkWm4j3PXU14GI3VSOE7ReSORumHs+54l2tod5gzmlnVxe NLR+tbUPxPtiojdFETa8RU0Wemv4DoDDM7VVAA5hoPCCaoTQP0S77NdAI 0awPRgezQQWYxVy53EemPgcbSg0AejhuNRL72v0j2CoOS3NCQ5dnl8sS0 4=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: none (zohomail.com: 192.237.175.120 is neither permitted nor denied by domain of lists.xenproject.org) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: M6ao4APKkmzpDj44vMcAkHIMKsAeCx7+0KTkg5wWy1B6RtNdeNfRDYkLDrbsSquz2cPCS/8FSp TZxsug0FpteUNudvO1KbREVkK2aWXpPZBlMnTEsMtgZ7kgksjMF8xgRsvMtYjhq0e1Kcg1LWhw S2kBIwPUbp99QhyNNiTKMF3iWFe8oB7YkStUWjXN1lvKgIWHMfF1AMDeCu+yyBahc7NO6d2OI6 w6dfh5afX4aXF6DnahjgL7gUCiTUoyO5WeNbjL0dUByvKio0kFNABKFhjD3V6zeM7Hoei/ReTU kms= X-SBRS: 2.7 X-MesageID: 9411222 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.69,296,1571716800"; d="scan'208";a="9411222" From: Roger Pau Monne To: Date: Mon, 9 Dec 2019 18:37:57 +0100 Message-ID: <20191209173757.46833-1-roger.pau@citrix.com> X-Mailer: git-send-email 2.24.0 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v4] x86: do not enable global pages when virtualized on AMD hardware X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Julien Grall , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Jan Beulich , Roger Pau Monne Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-ZohoMail-DKIM: fail (Header signature does not verify) When using global pages a full tlb flush can only be performed by toggling the PGE bit in CR4, which is usually quite expensive in terms of performance when running virtualized. This is specially relevant on AMD hardware, which doesn't have the ability to do selective CR4 trapping, but can also be relevant on Intel if the underlying hypervisor also traps accesses to the PGE CR4 bit. In order to avoid this performance penalty, do not use global pages when running virtualized on AMD hardware. A command line option 'global-pages' is provided in order to allow the user to select whether global pages will be enabled for PV guests. The above figures are from a PV shim running on AMD hardware with 32 vCPUs: PGE enabled, x2APIC mode: (XEN) Global lock flush_lock: addr=3Dffff82d0804b01c0, lockval=3D1adb1adb, = not locked (XEN) lock:1841883(1375128998543), block:1658716(10193054890781) Average lock time: 746588ns Average block time: 6145147ns PGE disabled, x2APIC mode: (XEN) Global lock flush_lock: addr=3Dffff82d0804af1c0, lockval=3Da8bfa8bf, = not locked (XEN) lock:2730175(657505389886), block:2039716(2963768247738) Average lock time: 240829ns Average block time: 1453029ns As seen from the above figures the lock and block time of the flush lock is reduced to approximately 1/3 of the original value. Note that XEN_MINIMAL_CR4 and mmu_cr4_features are not modified, and thus global pages are left enabled for the hypervisor. This is not an issue because the code to switch the control registers (cr3 and cr4) already takes into account such situation and performs the necessary flushes. The same already happens when using XPTI or PCIDE, as the guest cr4 doesn't have global pages enabled in that case either. Also note that the suspend and resume code is correct in writing mmu_cr4_features into cr4 on resume, since that's the cr4 used by the idle vCPU which is the context used by the suspend and resume routine. Signed-off-by: Roger Pau Monn=C3=A9 Reviewed-by: Jan Beulich --- Changes since v3: - Expand commit message. Changes since v2: - Set the default value at init if not specified by the user. - Use int8_t and read_mostly for opt_global_pages. Changes since v1: - Provide command line option to enable/disable PGE. - Only disable PGE on AMD hardware when virtualized. - Document the global-pages option. --- docs/misc/xen-command-line.pandoc | 13 +++++++++++++ xen/arch/x86/pv/domain.c | 15 ++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line= .pandoc index d9495ef6b9..7be30f2766 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -1087,6 +1087,19 @@ value settable via Xen tools. =20 Dom0 is using this value for sizing its maptrack table. =20 +### global-pages (x86) +> `=3D ` + +> Default: `true` unless running virtualized on AMD hardware + +Set whether the PGE bit in CR4 will be enabled for PV guests. This control= s the +usage of global pages, and thus the need to perform tlb flushes by writing= to +CR4. + +Note it's disabled by default when running virtualized on AMD hardware sin= ce +AMD SVM doesn't support selective trapping of CR4, so global pages are not +enabled in order to reduce the overhead of tlb flushes. + ### guest_loglvl > `=3D [/]` where level is `none | error | warn= ing | info | debug | all` =20 diff --git a/xen/arch/x86/pv/domain.c b/xen/arch/x86/pv/domain.c index 4b6f48dea2..8ff733f56b 100644 --- a/xen/arch/x86/pv/domain.c +++ b/xen/arch/x86/pv/domain.c @@ -118,6 +118,19 @@ unsigned long pv_fixup_guest_cr4(const struct vcpu *v,= unsigned long cr4) (mmu_cr4_features & PV_CR4_GUEST_VISIBLE_MASK)); } =20 +static int8_t __read_mostly opt_global_pages =3D -1; +boolean_runtime_param("global-pages", opt_global_pages); + +static int __init pge_init(void) +{ + if ( opt_global_pages =3D=3D -1 ) + opt_global_pages =3D !cpu_has_hypervisor || + boot_cpu_data.x86_vendor !=3D X86_VENDOR_AMD; + + return 0; +} +__initcall(pge_init); + unsigned long pv_make_cr4(const struct vcpu *v) { const struct domain *d =3D v->domain; @@ -130,7 +143,7 @@ unsigned long pv_make_cr4(const struct vcpu *v) */ if ( d->arch.pv.pcid ) cr4 |=3D X86_CR4_PCIDE; - else if ( !d->arch.pv.xpti ) + else if ( !d->arch.pv.xpti && opt_global_pages ) cr4 |=3D X86_CR4_PGE; =20 /* --=20 2.24.0 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel