From nobody Mon Apr 29 04:51:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1615911576; cv=none; d=zohomail.com; s=zohoarc; b=iimNyvNzzv5HCwfCtp5wLshFeBBIN+FtfiQ2nLuH0HsrqsA1OfOBF/v1D/UwhfFXvxuFrZwdyfGzbKJrApfDwY3xfjRquCkAmYo7lv5/UKGpAXlXDjwNCaEj5nhLyDMFNm7irOImDZRit45aZwPVtZ4tD1Z82S88GvKBcoxPUHY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615911576; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Od99dIcmIYhkzw5E42v9sg2tO2HUYGeDrjqlHYH/1kk=; b=kwG3/KwJdEHX3jxsCqEUQ4339Pp3y6Ro7x4dGXWCNnhMnhTDvznGYedgCiKAbARa8tDwmNxbUAbV9/2k51h6B9rhxw1Q/1izYL9xCZOITGlU3mLjJXQfMZrjK5X9maO/IS0LEA/zZ8MiBSnWm8teKvbJEpJ/8ESl8z6goZkp8iA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1615911576036318.3791130473554; Tue, 16 Mar 2021 09:19:36 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.98383.186614 (Exim 4.92) (envelope-from ) id 1lMCPZ-0002mn-As; Tue, 16 Mar 2021 16:19:13 +0000 Received: by outflank-mailman (output) from mailman id 98383.186614; Tue, 16 Mar 2021 16:19:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPZ-0002mf-5w; Tue, 16 Mar 2021 16:19:13 +0000 Received: by outflank-mailman (input) for mailman id 98383; Tue, 16 Mar 2021 16:19:11 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPX-0002kv-Rr for xen-devel@lists.xenproject.org; Tue, 16 Mar 2021 16:19:11 +0000 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 7caab44e-cc54-404a-acd4-d8a0b92f7b51; Tue, 16 Mar 2021 16:19:07 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 7caab44e-cc54-404a-acd4-d8a0b92f7b51 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1615911547; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rf/HGM31QWHWGX+1+ToUsMdeOvi8pCwabBY2/qPrlZY=; b=ULucW+b2UFs5Cgm97eEvPEVkAJHOhn5OAJWdHEmx1RGezMBoZOUjhFgt sz4x7ruozuvmq+khkV574ZVQw3UVQlvMf7rCQX6WPTpE0NLJ3AB5Wby3R zb69rXoENhnt+GdrYd0trT9R+qkQhjjXtbhYe9wG1dcJWz78rwStPfxo7 A=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 3iUKy1CtPOjR9xr4a/YNZQvVPT3nhPYif2nwlxPGZFMEyj9P5yMHvLTvGag8cK8Ak2Zp3uC4JQ YOaKh5NqUv7oMhDAeAo/cv3kv/NvK3LJ0rK7hpw1dmds+EffaE1Xrd6IXkctN+Bjn+zeOW3lTv 6qDkGnDWyjfA1EmnFRef+WF/g4SvVtOMY/nqRu1B6LaR3rkZRQlR7dKPJY1steXu3uL6cwOmfl MAEEly3DvbolK6O4gJN2xYZ9P1IAisoCntP0LCuvQ77IJECjwRkN/dn0gbdm/aWK5ImaOQMAZE mgg= X-SBRS: 5.1 X-MesageID: 39773267 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED IronPort-HdrOrdr: A9a23:EtnUN6+GQ0i5cDfOtcpuk+FtcL1zdoIgy1knxilNYDRvWIixi9 2ukPMH1RX9lTYWXzUalcqdPbSbKEm8ybdc2qNUGbu5RgHptC+TLI9k5Zb/2DGIIUPD38Zn/+ Nbf6B6YeeeMXFTh8z3+RT9Nt4mzsWO/qzAv5ag815GZ2hRGsZdxi1+DRuWFVAzYQFAC4YwGp b03Ls4mxOLf3MLYsOnQkQURuSrnayLqLvKQz4jQyQm5g6HkC+y5NfBcySw8x8CX1p0sMwf2E fflQiR3NTHj9iazVvm23bX/9BqnrLau6d+LeitruRQFTn2kAavY+1aKv2/lRQ4uvum5lpvsP SkmWZbA+1J53ncfn64rHLWsmGKultDmhySq2Owunftrdf0Qzg3EaN69P9kWyHE4EkttswU6t Ms40ultoFaBR6FvCPx68mgbWATqmOIoGEvmeNWsnpHUYF2Us4pkaUj+ipuYfM9NRO/zLpiPP hlDcna6voTW0iddWrlsm5mx8HpdmgvHz+dK3Jy+fC94nxzpjRU3kEYzMsQkjMr75QmUaRJ4O zCL+BBiKxOdMkLdqhwbd1xA/efOyjoe1bhIWiSKVPoGOUsIHTWsaP6570z+aWMdIEXyoAx3L DMSklRu2J3W0+GM7zK4LR7tjT2BEmtVzXkzc9To7JjvKfnebbtOSqfDF80lc+tpOgeH93bV/ 6/NIk+OY6lEULeXaJymyHuUZhbLncTFOcPvMwgZl6IqsXXbo3m39arNsr7Ff7IK3IJS2n/Cn wMUHzYP8Nb9H2mXXf+nVzUU3PpcUrv4IJoHMHhjrEu4blIErcJnhkeiFy/6M3OAyZFqLYKcE x3J66ilLi6q2mw9WPB9H5oJRJZE0ZQ7NzbIjd3jD5PF3mxXacIut2Zd2wX9mCAPAVDQ8TfFx Mau0564rutL5ubxTkrDtWuNm7ytQpKmFu6C7Mn3oGT78bsfZ01Sqs8UKtqDAPRClheggBxsl pObwcCW27SHj7jkr+ekZQRHe3THuMM2zuDEIpxkzb/vV/ZjdwzTnEbNgTeIPK/sEILfX5ooX Fft4UYm6GNnD6zL3BXupVJDHR8LEKNALxHCwyZYp5zgb6DQnArcU6DmSGahxYvemDj6kUVgS j7ITeJfOzQa2AtyUxww+Lk9kh5eX6aeF81YndmsZdlHWCDoXpr1/SXD5DDnFe5ex8Gz6UFKz fObSEJLg4G/aHA6DeF3DKDH24h3JMgI6jUC6kiaaja3hqWWcC1vLBDG/9f55B+Mt/y9ucNTO KEYgeQaDf1Efkg1QDQpnErPkBP2TUZuOKt3B3u926j2nEjRfLUPVR9XrkeZ8iG8HKMfYfB7L xpydYu+eegOGT4bdCLjanRcj5YMxvW5Wq7Vfshp5xYtb873YEDUaXzQH/NzjVKzR8+JMD7mA cFTKN37KvIN4Vvc8YREhgptGYBhZCKNg8mowb2CugxcRUxlHfdJcqO+KeNprw1AEGNzTGAcG W34mlY5bPCUCSC37JBVP51LmRSdUQm6HNtuOmFbJbdDQ22d+dFuFq2W0XNBIN1WeyAA/EXqB 0//tSD2+mQfCD80BrLvTR6LrlVmlzXCv+aEUaJA6pQ79e+OV6QmaOk7864kSfvRVKAGjElrJ wAcVZVc99KhTYjhpAmyyS+SqT4pUQ+jltViAsX4WLFy8yh+2fUHUZPLA3fjNFXRFBoQwq1sf g= X-IronPort-AV: E=Sophos;i="5.81,251,1610427600"; d="scan'208";a="39773267" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu , Boris Ostrovsky , Ian Jackson Subject: [PATCH 1/3] Revert "x86/msr: drop compatibility #GP handling in guest_{rd,wr}msr()" Date: Tue, 16 Mar 2021 16:18:42 +0000 Message-ID: <20210316161844.1658-2-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210316161844.1658-1-andrew.cooper3@citrix.com> References: <20210316161844.1658-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) In hindsight, this was a poor move. Some of these MSRs require probing for, causing unhelpful spew into xl dmesg, as well as spew from unit tests explicitly checking behaviour. This restores behaviour close to that of Xen 4.14. Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Wei Liu CC: Boris Ostrovsky CC: Ian Jackson For 4.15. Restoring behaviour closer to 4.14, and prereq for a bugfix need= ing backporting. --- xen/arch/x86/msr.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c index 0ebcb04259..c3a988bd11 100644 --- a/xen/arch/x86/msr.c +++ b/xen/arch/x86/msr.c @@ -175,6 +175,30 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t= *val) =20 switch ( msr ) { + /* Write-only */ + case MSR_AMD_PATCHLOADER: + case MSR_IA32_UCODE_WRITE: + case MSR_PRED_CMD: + case MSR_FLUSH_CMD: + + /* Not offered to guests. */ + case MSR_TEST_CTRL: + case MSR_CORE_CAPABILITIES: + case MSR_TSX_FORCE_ABORT: + case MSR_TSX_CTRL: + case MSR_MCU_OPT_CTRL: + case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_U_CET: + case MSR_S_CET: + case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: + case MSR_AMD64_LWP_CFG: + case MSR_AMD64_LWP_CBADDR: + case MSR_PPIN_CTL: + case MSR_PPIN: + case MSR_AMD_PPIN_CTL: + case MSR_AMD_PPIN: + goto gp_fault; + case MSR_IA32_FEATURE_CONTROL: /* * Architecturally, availability of this MSR is enumerated by the @@ -382,6 +406,30 @@ int guest_wrmsr(struct vcpu *v, uint32_t msr, uint64_t= val) { uint64_t rsvd; =20 + /* Read-only */ + case MSR_IA32_PLATFORM_ID: + case MSR_CORE_CAPABILITIES: + case MSR_INTEL_CORE_THREAD_COUNT: + case MSR_INTEL_PLATFORM_INFO: + case MSR_ARCH_CAPABILITIES: + + /* Not offered to guests. */ + case MSR_TEST_CTRL: + case MSR_TSX_FORCE_ABORT: + case MSR_TSX_CTRL: + case MSR_MCU_OPT_CTRL: + case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_U_CET: + case MSR_S_CET: + case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: + case MSR_AMD64_LWP_CFG: + case MSR_AMD64_LWP_CBADDR: + case MSR_PPIN_CTL: + case MSR_PPIN: + case MSR_AMD_PPIN_CTL: + case MSR_AMD_PPIN: + goto gp_fault; + case MSR_AMD_PATCHLEVEL: BUILD_BUG_ON(MSR_IA32_UCODE_REV !=3D MSR_AMD_PATCHLEVEL); /* --=20 2.11.0 From nobody Mon Apr 29 04:51:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1616771347; cv=none; d=zohomail.com; s=zohoarc; b=cneBXpdvUw/ymu8GMhivitz3fzmb79b1tHG2CrXsPAsKHFjywJil2RWymjcYwFpInLet588ctV+E98lMdyM5zsOuNjxR74Y6HDM2pwFXZAdYiZgusB8DDVpTItkC3+OVMbRoL0iszFzogWScUruYNDhuRkRI0ErDJkryyekqbs0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1616771347; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=bxwBoGm/HtY2XjH6TUAwMmPqJZhSB5i77uC61meo8AE=; b=YbaiyBpbVZ0WqAEcuxIeX0WXyhPTkFeoWQ6lEIdBQvMFyEqodBOI+HNh2kvZ7s6PFcjHaYFhWLSg2m3G51Jtr1yKKpPxGYkZdCyW/ebFP36PrpWYMutrLk8ecV+i2DHuw9LAsdj2WjvJ+SNb8QN9DH9AXQzPzji7ZN7hSka0vj0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 161677134774717.39604553723359; Fri, 26 Mar 2021 08:09:07 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.101897.195191 (Exim 4.92) (envelope-from ) id 1lPo4p-0002kh-3F; Fri, 26 Mar 2021 15:08:43 +0000 Received: by outflank-mailman (output) from mailman id 101897.195191; Fri, 26 Mar 2021 15:08:43 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lPo4o-0002ka-Vh; Fri, 26 Mar 2021 15:08:42 +0000 Received: by outflank-mailman (input) for mailman id 101897; Fri, 26 Mar 2021 15:08:41 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lPo4n-0002kV-HS for xen-devel@lists.xenproject.org; Fri, 26 Mar 2021 15:08:41 +0000 Received: from mx2.suse.de (unknown [195.135.220.15]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id efc78848-211b-49c9-ae4c-c99aae012304; Fri, 26 Mar 2021 15:08:40 +0000 (UTC) Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id CAE40ADFB; Fri, 26 Mar 2021 15:08:39 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: efc78848-211b-49c9-ae4c-c99aae012304 X-Virus-Scanned: by amavisd-new at test-mx.suse.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1616771319; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bxwBoGm/HtY2XjH6TUAwMmPqJZhSB5i77uC61meo8AE=; b=GmBfnL4hwNGpnb7yMgLiLDucVuLqnIV+72oVJLz8hcMkr4WwvoJ5Szkm1179gbw2b1s9E2 IuOfbuWr+0yM91qurByFdkxzl7UiabdSrgpwghsNVx8bLHZK0SltX+QMioUPdhEYAJL200 V/9qgB53KFNz6gdq/ah1G+YzDwhnz3g= Subject: [PATCH for-4.15 v1.1 1/3] Revert "x86/msr: drop compatibility #GP handling in guest_{rd,wr}msr()" To: Ian Jackson , Xen-devel Cc: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= , Wei Liu , Boris Ostrovsky , Andrew Cooper References: <20210316161844.1658-1-andrew.cooper3@citrix.com> From: Jan Beulich Message-ID: Date: Fri, 26 Mar 2021 16:08:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.9.0 MIME-Version: 1.0 In-Reply-To: <20210316161844.1658-1-andrew.cooper3@citrix.com> Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) Content-Type: text/plain; charset="utf-8" From: Andrew Cooper In hindsight, this was a poor move. Some of these MSRs require probing for, cause unhelpful spew into xl dmesg, or cause spew from unit tests explicitly checking behaviour. This restores behaviour close to that of Xen 4.14, meaning in particular that for all of the MSRs getting re-added explicitly a #GP fault will get raised irrespective of the new "msr_relaxed" setting. Signed-off-by: Andrew Cooper Signed-off-by: Jan Beulich Reviewed-by: Jan Beulich Release-Acked-by: Ian Jackson Acked-by: Ian Jackson --- v1.1: Fold in suggested description adjustments. --- For 4.15. Restoring behaviour closer to 4.14, and prereq for a bugfix need= ing backporting. --- xen/arch/x86/msr.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c index 0ebcb04259..c3a988bd11 100644 --- a/xen/arch/x86/msr.c +++ b/xen/arch/x86/msr.c @@ -175,6 +175,30 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t= *val) =20 switch ( msr ) { + /* Write-only */ + case MSR_AMD_PATCHLOADER: + case MSR_IA32_UCODE_WRITE: + case MSR_PRED_CMD: + case MSR_FLUSH_CMD: + + /* Not offered to guests. */ + case MSR_TEST_CTRL: + case MSR_CORE_CAPABILITIES: + case MSR_TSX_FORCE_ABORT: + case MSR_TSX_CTRL: + case MSR_MCU_OPT_CTRL: + case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_U_CET: + case MSR_S_CET: + case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: + case MSR_AMD64_LWP_CFG: + case MSR_AMD64_LWP_CBADDR: + case MSR_PPIN_CTL: + case MSR_PPIN: + case MSR_AMD_PPIN_CTL: + case MSR_AMD_PPIN: + goto gp_fault; + case MSR_IA32_FEATURE_CONTROL: /* * Architecturally, availability of this MSR is enumerated by the @@ -382,6 +406,30 @@ int guest_wrmsr(struct vcpu *v, uint32_t msr, uint64_t= val) { uint64_t rsvd; =20 + /* Read-only */ + case MSR_IA32_PLATFORM_ID: + case MSR_CORE_CAPABILITIES: + case MSR_INTEL_CORE_THREAD_COUNT: + case MSR_INTEL_PLATFORM_INFO: + case MSR_ARCH_CAPABILITIES: + + /* Not offered to guests. */ + case MSR_TEST_CTRL: + case MSR_TSX_FORCE_ABORT: + case MSR_TSX_CTRL: + case MSR_MCU_OPT_CTRL: + case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_U_CET: + case MSR_S_CET: + case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: + case MSR_AMD64_LWP_CFG: + case MSR_AMD64_LWP_CBADDR: + case MSR_PPIN_CTL: + case MSR_PPIN: + case MSR_AMD_PPIN_CTL: + case MSR_AMD_PPIN: + goto gp_fault; + case MSR_AMD_PATCHLEVEL: BUILD_BUG_ON(MSR_IA32_UCODE_REV !=3D MSR_AMD_PATCHLEVEL); /* From nobody Mon Apr 29 04:51:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1615911576; cv=none; d=zohomail.com; s=zohoarc; b=XheNKXEaVDOCc0rhkRsvHJO+wQpDyhZ1Pca08OktQjhbaPsjlZLs7oPO7oy269WSkDSEj3b2zrlg/evLzL2H1WcBloIahFBoh+OQzrrlQORGGbSFtvrRkElD4Ht6zhoyBJfryjvbgfHmSTBRrFe+BYuyJBU2HoudGtU64iOTtww= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615911576; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=rihiUe7amnkM+Cv3XaHDURiX+18OfjQbyZa4bxyr+ns=; b=IVf2CXnwEpIaPpdsZpLjKTymY52S67K+9ZycHvC4wS2mEShrgfYTTUgl8tE7VwGvjhQ89su05aBHlYo/UhkCBoeJ+dRTpO1w3saVIh7SCJ7qooP2j0mYlM4w/vn0nTItlpFpIfgG2eK8Iq3Ifk9I/6WwWFcEP8mFP6/dgs13qac= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1615911576463211.21730755967133; Tue, 16 Mar 2021 09:19:36 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.98382.186601 (Exim 4.92) (envelope-from ) id 1lMCPU-0002l7-T7; Tue, 16 Mar 2021 16:19:08 +0000 Received: by outflank-mailman (output) from mailman id 98382.186601; Tue, 16 Mar 2021 16:19:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPU-0002l0-Q9; Tue, 16 Mar 2021 16:19:08 +0000 Received: by outflank-mailman (input) for mailman id 98382; Tue, 16 Mar 2021 16:19:07 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPT-0002kv-1R for xen-devel@lists.xenproject.org; Tue, 16 Mar 2021 16:19:07 +0000 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c5875003-9fc3-438f-8ad6-81bc9887a3b0; Tue, 16 Mar 2021 16:19:05 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: c5875003-9fc3-438f-8ad6-81bc9887a3b0 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1615911545; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nhAX0yyRIBSeaz93pFWYK1z2uQ1J1CjJXlWF+mLEc70=; b=CBhszXhJW9KtURAl6iYjZaEUe0TbavZJcUQGu+6lkywI15gGy86ArN92 jLsxSDdlixrv1NTkmENLOw2/P+CKm4EQFheMweLPj/jwtew+oLp87/H7B bPK8OsrTia4RNIKRWO4iGi0ym5ePNCcjrmGuHcWK/Wa4GWCoDx6MVBPhd s=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: n5BQTAtLxqxafpFEdbx0CgRATm/N9FByb1BTMPGz+5IUy/d4lXv8xcSDA6HDCkCrcpUC57TnTM LWI7cd38qgdJcu03Cwj2qMqqDRe8xwBAmUxwWOxran2w3OxgQpzMbzHkG2YS3dm8+WSaJNExyH j1s57lhe4ECcZhbjmVUp/nsWFY8CHRvGOXcNid2BlXvY3T6hRQPVWf85Uy6Xxxc4x4y4MszrOr uM2J+D1kt1ui6dz1+TIIqS1jMI1YdHJpZEA7IWw6cFmlWN1muk+KUGAancjA7NcLwgNChp/pt1 8Sw= X-SBRS: 5.1 X-MesageID: 39404104 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED IronPort-HdrOrdr: A9a23:9rE3+q4hZN5th5CqIQPXwE/XdLJzesId70hD6mlaY3VuE/Cwvc aogfgdyFvIkz4XQn4tgpStP6OHTHPa+/dOkPEsFJ2lWxTrv3btEZF64eLZskTdMgD36+I179 YCT4FQE9v1ZGIUse/b502CH88k0J279smT9IPj5lNMaS0vVK169Qd+DW+gYytLbS1LH4AwGp bZxucvnUvDRV0tYs62BmYIUoH4zrWh/v+LXTc8CxEq8wWIhz+zgYSKdCSw5RsCTyhJhY4r7G mtqX2D2oyYr/q5xhXAvlWy071qnrLapOdrNYipgsgRJi6EsHfMWLhc X-IronPort-AV: E=Sophos;i="5.81,251,1610427600"; d="scan'208";a="39404104" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu , Boris Ostrovsky , Ian Jackson Subject: [PATCH 2/3] x86/msr: Forward port XSA-351 changes from 4.14 Date: Tue, 16 Mar 2021 16:18:43 +0000 Message-ID: <20210316161844.1658-3-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210316161844.1658-1-andrew.cooper3@citrix.com> References: <20210316161844.1658-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) staging was not impacted by XSA-351 at the time of release, due to c/s 322ec7c89f and 84e848fd7a which disallows read access by default. Forward port the XSA-351 changes to make the code structure consistent betw= een 4.14 and 4.15. This removes logspew for guests probing for the RAPL interface. Signed-off-by: Andrew Cooper Reviewed-by: Roger Pau Monn=C3=A9 --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Wei Liu CC: Boris Ostrovsky CC: Ian Jackson Technically this breaks Solaris/turbostat insofar as you can no longer use msr_relaxed to "fix" the guest. The subsequent patch will unbreak it differently. For 4.15. Restoring behaviour closer to 4.14, and prereq for a bugfix need= ing backporting. --- xen/arch/x86/msr.c | 19 +++++++++++++++++++ xen/include/asm-x86/msr-index.h | 39 +++++++++++++++++++++++++++++++++++++= ++ 2 files changed, 58 insertions(+) diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c index c3a988bd11..5927b6811b 100644 --- a/xen/arch/x86/msr.c +++ b/xen/arch/x86/msr.c @@ -188,6 +188,13 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t= *val) case MSR_TSX_CTRL: case MSR_MCU_OPT_CTRL: case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_RAPL_POWER_UNIT: + case MSR_PKG_POWER_LIMIT ... MSR_PKG_POWER_INFO: + case MSR_DRAM_POWER_LIMIT ... MSR_DRAM_POWER_INFO: + case MSR_PP0_POWER_LIMIT ... MSR_PP0_POLICY: + case MSR_PP1_POWER_LIMIT ... MSR_PP1_POLICY: + case MSR_PLATFORM_ENERGY_COUNTER: + case MSR_PLATFORM_POWER_LIMIT: case MSR_U_CET: case MSR_S_CET: case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: @@ -195,6 +202,8 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t = *val) case MSR_AMD64_LWP_CBADDR: case MSR_PPIN_CTL: case MSR_PPIN: + case MSR_F15H_CU_POWER ... MSR_F15H_CU_MAX_POWER: + case MSR_AMD_RAPL_POWER_UNIT ... MSR_AMD_PKG_ENERGY_STATUS: case MSR_AMD_PPIN_CTL: case MSR_AMD_PPIN: goto gp_fault; @@ -412,6 +421,7 @@ int guest_wrmsr(struct vcpu *v, uint32_t msr, uint64_t = val) case MSR_INTEL_CORE_THREAD_COUNT: case MSR_INTEL_PLATFORM_INFO: case MSR_ARCH_CAPABILITIES: + case MSR_IA32_PERF_STATUS: =20 /* Not offered to guests. */ case MSR_TEST_CTRL: @@ -419,6 +429,13 @@ int guest_wrmsr(struct vcpu *v, uint32_t msr, uint64_t= val) case MSR_TSX_CTRL: case MSR_MCU_OPT_CTRL: case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): + case MSR_RAPL_POWER_UNIT: + case MSR_PKG_POWER_LIMIT ... MSR_PKG_POWER_INFO: + case MSR_DRAM_POWER_LIMIT ... MSR_DRAM_POWER_INFO: + case MSR_PP0_POWER_LIMIT ... MSR_PP0_POLICY: + case MSR_PP1_POWER_LIMIT ... MSR_PP1_POLICY: + case MSR_PLATFORM_ENERGY_COUNTER: + case MSR_PLATFORM_POWER_LIMIT: case MSR_U_CET: case MSR_S_CET: case MSR_PL0_SSP ... MSR_INTERRUPT_SSP_TABLE: @@ -426,6 +443,8 @@ int guest_wrmsr(struct vcpu *v, uint32_t msr, uint64_t = val) case MSR_AMD64_LWP_CBADDR: case MSR_PPIN_CTL: case MSR_PPIN: + case MSR_F15H_CU_POWER ... MSR_F15H_CU_MAX_POWER: + case MSR_AMD_RAPL_POWER_UNIT ... MSR_AMD_PKG_ENERGY_STATUS: case MSR_AMD_PPIN_CTL: case MSR_AMD_PPIN: goto gp_fault; diff --git a/xen/include/asm-x86/msr-index.h b/xen/include/asm-x86/msr-inde= x.h index f2e34dd22b..49c1afdd22 100644 --- a/xen/include/asm-x86/msr-index.h +++ b/xen/include/asm-x86/msr-index.h @@ -101,6 +101,38 @@ #define MSR_RTIT_ADDR_A(n) (0x00000580 + (n) * 2) #define MSR_RTIT_ADDR_B(n) (0x00000581 + (n) * 2) =20 +/* + * Intel Runtime Average Power Limiting (RAPL) interface. Power plane base + * addresses (MSR_*_POWER_LIMIT) are model specific, but have so-far been + * consistent since their introduction in SandyBridge. + * + * Offsets of functionality from the power plane base is architectural, but + * not all power planes support all functionality. + */ +#define MSR_RAPL_POWER_UNIT 0x00000606 + +#define MSR_PKG_POWER_LIMIT 0x00000610 +#define MSR_PKG_ENERGY_STATUS 0x00000611 +#define MSR_PKG_PERF_STATUS 0x00000613 +#define MSR_PKG_POWER_INFO 0x00000614 + +#define MSR_DRAM_POWER_LIMIT 0x00000618 +#define MSR_DRAM_ENERGY_STATUS 0x00000619 +#define MSR_DRAM_PERF_STATUS 0x0000061b +#define MSR_DRAM_POWER_INFO 0x0000061c + +#define MSR_PP0_POWER_LIMIT 0x00000638 +#define MSR_PP0_ENERGY_STATUS 0x00000639 +#define MSR_PP0_POLICY 0x0000063a + +#define MSR_PP1_POWER_LIMIT 0x00000640 +#define MSR_PP1_ENERGY_STATUS 0x00000641 +#define MSR_PP1_POLICY 0x00000642 + +/* Intel Platform-wide power interface. */ +#define MSR_PLATFORM_ENERGY_COUNTER 0x0000064d +#define MSR_PLATFORM_POWER_LIMIT 0x0000065c + #define MSR_U_CET 0x000006a0 #define MSR_S_CET 0x000006a2 #define CET_SHSTK_EN (_AC(1, ULL) << 0) @@ -116,10 +148,17 @@ #define PASID_PASID_MASK 0x000fffff #define PASID_VALID (_AC(1, ULL) << 31) =20 +#define MSR_F15H_CU_POWER 0xc001007a +#define MSR_F15H_CU_MAX_POWER 0xc001007b + #define MSR_K8_VM_CR 0xc0010114 #define VM_CR_INIT_REDIRECTION (_AC(1, ULL) << 1) #define VM_CR_SVM_DISABLE (_AC(1, ULL) << 4) =20 +#define MSR_AMD_RAPL_POWER_UNIT 0xc0010299 +#define MSR_AMD_CORE_ENERGY_STATUS 0xc001029a +#define MSR_AMD_PKG_ENERGY_STATUS 0xc001029b + /* * Legacy MSR constants in need of cleanup. No new MSRs below this commen= t. */ --=20 2.11.0 From nobody Mon Apr 29 04:51:43 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1615911586; cv=none; d=zohomail.com; s=zohoarc; b=jP58OWWy1nxdI87o0c+k526vYqPyqWznceMatwaWOrxwFhLk/ANsibmRVGUZzBgF+/Hj39ITKQR6+noqGgfSVpJ9TqIc/x+54QF0M9U7WoyppkQYiQdU3wi6Gs9ROE3le1ARAgui7xi+oT8a/DooyzR/CJ9WQQGAAFEWxTi0kAA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1615911586; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hOGOTvKSWeJuqK5QIGTYfSpKPf3AHZPqaM97dBko5/Q=; b=Q0rMnUpjl+8xpNG52Z8wBv6a2NhMt3Ago3Cgto0xIlO/E1aJKxg8Z4Ndumze/k2S7/hK9FsYGKpAYQFgsMGzRRGA7ChVwlB6q8e1eNCfbH7zcbZyT7QVk/e80buETXROqPNUh3XABONaJtMPNTq46gT293lVCXCjqF6mEf6G/xM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1615911586405984.3759102393774; Tue, 16 Mar 2021 09:19:46 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.98385.186638 (Exim 4.92) (envelope-from ) id 1lMCPp-0002xO-Sl; Tue, 16 Mar 2021 16:19:29 +0000 Received: by outflank-mailman (output) from mailman id 98385.186638; Tue, 16 Mar 2021 16:19:29 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPp-0002xH-P5; Tue, 16 Mar 2021 16:19:29 +0000 Received: by outflank-mailman (input) for mailman id 98385; Tue, 16 Mar 2021 16:19:28 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1lMCPo-0002wi-LF for xen-devel@lists.xenproject.org; Tue, 16 Mar 2021 16:19:28 +0000 Received: from esa4.hc3370-68.iphmx.com (unknown [216.71.155.144]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 39508681-bf94-4099-8062-d0420bfaea3e; Tue, 16 Mar 2021 16:19:27 +0000 (UTC) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 39508681-bf94-4099-8062-d0420bfaea3e DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1615911567; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=guXIUW2Hj1C8CTUWbOZUc8Q4qeZcP62gPTH+3TYZdDc=; b=NlHSVPvkzxS3vOeqhWrxYpA7PzB86h9LBpY4vqlBjcVwQFD79cm5zr7Q rnUgog+X/ko18MHB4Zv3aPFmYztPVJCcN3qO2RqraWnBriYHCpcvUOr+m 0GnX85GXNu40Ooo+wRZV1sUUuBwl+YDpjZpb5amcLBDHZNwFVAlj3NFgt I=; Authentication-Results: esa4.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none IronPort-SDR: 5K15W2ANNPrmne+aw5lDyfY7SCi7xsJBU1A+bb8JSMJNB8HllYsedpmc5XB7uCIC27OBpOO3v1 XtmWFHXO2QGvIvvCB/AcBFCGf46AVs2AAI7n33SZLKcwpyiWZbd/YirGqOr17EubzOpwLSnRV1 Q94nvWd3T9U/hHBspaJTCK3vQ/rxTbPNexlHgHgc3bKP+6Edh56Z5YCgQJy8ywb+9+BVMbQoB+ a3DumS+HaWyuVyN4vY3swxv0yNZadUl0TUgs/AkhXnbqcf8WLlf8Cn7TdXNv4yuv2aAQOC0+7e Eqk= X-SBRS: 5.1 X-MesageID: 40833138 X-Ironport-Server: esa4.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED IronPort-HdrOrdr: A9a23:AbhGrqrtED33HaFeaWT+3t4aV5qseYIsi2QD101hICF9WObwra GTtd4c0gL5jytUZWopnsqONLLFbXTX85N05od5B8bbYCDNvmy0IIZ+qbbz2jGIIVyGysdx3b ptGpIOa+HYIkN9ia/BjzWQM9Fl+9Wf9bDtuOG29QYIcShPS4VNqzh0ERyaFEoefnghObMcGI CH7sRK4xqMEE5nDPiTPXUOU+jdq9CjrvuPCnRqayIP0wWAgSil77T3CXGjr3AjeghC3Ks49i z9mxH5j5/TyM2T8APW1GPY8v1t+OfJ990rPqKxo/lQDj3tjwqyDb4RPoG/gA== X-IronPort-AV: E=Sophos;i="5.81,251,1610427600"; d="scan'208";a="40833138" From: Andrew Cooper To: Xen-devel CC: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu , Boris Ostrovsky , Ian Jackson Subject: [PATCH 3/3] x86/msr: Fix Solaris and turbostat following XSA-351 Date: Tue, 16 Mar 2021 16:18:44 +0000 Message-ID: <20210316161844.1658-4-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210316161844.1658-1-andrew.cooper3@citrix.com> References: <20210316161844.1658-1-andrew.cooper3@citrix.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) Signed-off-by: Andrew Cooper --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Wei Liu CC: Boris Ostrovsky CC: Ian Jackson For 4.15 This wants backporting to all security trees, as it is a fix to a regression introduced in XSA-351. Also it means that users don't need msr_relaxed=3D1 to unbreak Solaris gues= ts, which is a strict useability improvement. --- xen/arch/x86/msr.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/msr.c b/xen/arch/x86/msr.c index 5927b6811b..a83a1d7fba 100644 --- a/xen/arch/x86/msr.c +++ b/xen/arch/x86/msr.c @@ -188,7 +188,6 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t = *val) case MSR_TSX_CTRL: case MSR_MCU_OPT_CTRL: case MSR_RTIT_OUTPUT_BASE ... MSR_RTIT_ADDR_B(7): - case MSR_RAPL_POWER_UNIT: case MSR_PKG_POWER_LIMIT ... MSR_PKG_POWER_INFO: case MSR_DRAM_POWER_LIMIT ... MSR_DRAM_POWER_INFO: case MSR_PP0_POWER_LIMIT ... MSR_PP0_POLICY: @@ -284,6 +283,18 @@ int guest_rdmsr(struct vcpu *v, uint32_t msr, uint64_t= *val) goto gp_fault; break; =20 + case MSR_RAPL_POWER_UNIT: + /* + * This MSR is non-architectural. However, some versions of Solar= is + * blindly reads it without a #GP guard, and some versions of + * turbostat crash after expecting a read of /proc/cpu/0/msr not to + * fail. Read as zero on Intel hardware. + */ + if ( !(cp->x86_vendor & X86_VENDOR_INTEL) ) + goto gp_fault; + *val =3D 0; + break; + /* * These MSRs are not enumerated in CPUID. They have been around * since the Pentium 4, and implemented by other vendors. --=20 2.11.0