From nobody Mon May 20 21:57:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=cloud.com ARC-Seal: i=1; a=rsa-sha256; t=1709544211; cv=none; d=zohomail.com; s=zohoarc; b=DcCOupgVqFmBweo41ZKxtZvTFWVjSjImxjnAiTDMH3L7g7Pc/kIIi5YwgiPB+FfzhWAT8vbOLneZZ/IjbAjbGH+ghYPvZ7NBIMX+2ZwAm0txRdOXFy01DMKJR9gm5AwVZ/SID9vcrRZucpuaf4tusEXRxjMcDC54rj2AQ0CnFik= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1709544211; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=ccAwOwOdhjjKU5vrkhjFv38qxkICvc3DaQURqNSfmho=; b=EhAk+2SjLF1OVP3X6vlSAVN88y7JWJhaOY0DmB3GeLy3DKh+lE03pAQqicQN26IxmdVmVohy7K0/wr8u5q8KFSoXDET6TTv4lsFUA26MlJO4s2HGAsO0Vm/kpfrIKT6TwkRJXNdqn2t9vB7vgShOCNPf6wHj4TwUzPTvplop8IM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 170954421103779.96056236839433; Mon, 4 Mar 2024 01:23:31 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.688227.1072181 (Exim 4.92) (envelope-from ) id 1rh4XT-00034I-4Y; Mon, 04 Mar 2024 09:23:15 +0000 Received: by outflank-mailman (output) from mailman id 688227.1072181; Mon, 04 Mar 2024 09:23:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XT-00034A-0p; Mon, 04 Mar 2024 09:23:15 +0000 Received: by outflank-mailman (input) for mailman id 688227; Mon, 04 Mar 2024 09:23:13 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XR-0002pl-5e for xen-devel@lists.xenproject.org; Mon, 04 Mar 2024 09:23:13 +0000 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [2a00:1450:4864:20::62e]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id d0de529d-da08-11ee-a1ee-f123f15fe8a2; Mon, 04 Mar 2024 10:23:09 +0100 (CET) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-a44f2d894b7so159549266b.1 for ; Mon, 04 Mar 2024 01:23:10 -0800 (PST) Received: from fhilly.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id tj10-20020a170907c24a00b00a444526962asm4485529ejc.128.2024.03.04.01.23.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 01:23:08 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d0de529d-da08-11ee-a1ee-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloud.com; s=cloud; t=1709544189; x=1710148989; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ccAwOwOdhjjKU5vrkhjFv38qxkICvc3DaQURqNSfmho=; b=I5HAVRVbMOTNceOHmkEz6iBdl5UcJ+ywLU8ekcKVFjVwfasZqXA4BIJZzQdgl3tqAv 4AbDg0eqMGbNsyMpSkwudeugNpJvQnkV2Do9lFsEPTYu/d+92CaFqQWzsSfE1tZl+XZQ JZ8OKqc3SRfMxXcZ4BzypRH0T2+fNHeRKxyeY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709544189; x=1710148989; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ccAwOwOdhjjKU5vrkhjFv38qxkICvc3DaQURqNSfmho=; b=sn/T6/KlKIRsWmzf2sTF6bcK3hEgygAG8mDj7LVnG6qiN3iqBsgNM9qGM09hDuqQDI pZIG3xRgOtL/2CkMgMu1n+A7T6SK/w7sWtP1WKz2ZQvlq11hkGAwpARQnFP9QYQFWVg3 Uiy0tOLQzc4OtYtuvpWqGefyRcO2FDNOzqs9NR3cYMO81d0WNBU4ekLP6TjtJfhp7efB Lro1c6nMnS2Cny/zpsYf0zjzd9ycymlieQWKrd0V2yAWd6so6bhVLaHIBD8LAZl3DEYx JlctpKSOwabKzbTdaGIYXo7Xhs3DEZ4lhmW379YbOP1Z5UTU3qbCj9cHdtcTI3NJ3boJ xxig== X-Gm-Message-State: AOJu0YzUu9y2JhXT8sfvqEzuGCwBrFs7FGVQWxBk/9lCxR5dBkMndYQH ZVnN5Bzpa2TQLHamT/9m74Y9GUaiSFhpc0LrkcC2/5bVPcfJ+UnmzlOpabPsMNcCMKTy9B2md2Q EXYM= X-Google-Smtp-Source: AGHT+IFpVknCnBeGMT58GJNWesqTQw1OSRo9PDozkmsZZ3Qk9gRuJnRSh1A0qB4Ng6ODltK6luEauw== X-Received: by 2002:a17:906:cd03:b0:a44:511d:630b with SMTP id oz3-20020a170906cd0300b00a44511d630bmr5560629ejb.24.1709544188840; Mon, 04 Mar 2024 01:23:08 -0800 (PST) From: Fouad Hilly To: Xen-devel Cc: Wei Liu , Fouad Hilly , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu Subject: [PATCH 1/3] x86: i387.c cleanup Date: Mon, 4 Mar 2024 09:13:05 +0000 Message-ID: <20240304091307.2295344-2-fouad.hilly@cloud.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240304091307.2295344-1-fouad.hilly@cloud.com> References: <20240304091307.2295344-1-fouad.hilly@cloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @cloud.com) X-ZM-MESSAGEID: 1709544213201100001 From: Wei Liu No functional change Signed-off-by: Wei Liu Signed-off-by: Fouad Hilly Acked-by: Jan Beulich --- CC: Jan Beulich CC: Andrew Cooper CC: "Roger Pau Monn=C3=A9" CC: Wei Liu --- xen/arch/x86/i387.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c index d824f2bb5294..7a4297cc921e 100644 --- a/xen/arch/x86/i387.c +++ b/xen/arch/x86/i387.c @@ -26,7 +26,7 @@ static inline void fpu_xrstor(struct vcpu *v, uint64_t ma= sk) =20 ASSERT(v->arch.xsave_area); /* - * XCR0 normally represents what guest OS set. In case of Xen itself,=20 + * XCR0 normally represents what guest OS set. In case of Xen itself, * we set the accumulated feature mask before doing save/restore. */ ok =3D set_xcr0(v->arch.xcr0_accum | XSTATE_FP_SSE); @@ -36,7 +36,7 @@ static inline void fpu_xrstor(struct vcpu *v, uint64_t ma= sk) ASSERT(ok); } =20 -/* Restor x87 FPU, MMX, SSE and SSE2 state */ +/* Restore x87 FPU, MMX, SSE and SSE2 state */ static inline void fpu_fxrstor(struct vcpu *v) { const typeof(v->arch.xsave_area->fpu_sse) *fpu_ctxt =3D v->arch.fpu_ct= xt; @@ -307,7 +307,7 @@ void save_fpu_enable(void) int vcpu_init_fpu(struct vcpu *v) { int rc; - =20 + v->arch.fully_eager_fpu =3D opt_eager_fpu; =20 if ( (rc =3D xstate_alloc_save_area(v)) !=3D 0 ) --=20 2.42.0 From nobody Mon May 20 21:57:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=cloud.com ARC-Seal: i=1; a=rsa-sha256; t=1709544212; cv=none; d=zohomail.com; s=zohoarc; b=RCWHzX9qYSJmp6arKnDxTBOXf4ozPrDo8AQbMo0b2/0eQiZmJWTVbk260NjyQgjc3Dz78YmbY4VebqeBPG2IRgb5hEqZKSRRYxEGQRqA0yzTeRh6nq/BEmQrvu5uUt++Qf2dT116Sm67UQFRM5yA+uPW3LK8nr6ItchBZK3DIz4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1709544212; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=t8576Vz0UPo+rS38MVUiaswDJj20EFHOADQjBzvq4J4=; b=bRCZnUnh5Qrvbeghcy5xP3qvAs+LIGiav0ToID56gOiCmxkZ7L45KCe/0s5F1zhNf4g6Nd9xa8mr9XdCKAIe+VTjHHpQA9kYJGQcSWTCpcEdA/hdROOWX6sx6LN+rCMZ85Lf4vzDhFOrn2K/UfMH2w3Ms9WVrIQn+YtY57ZN42g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1709544212943384.86692688183257; Mon, 4 Mar 2024 01:23:32 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.688228.1072186 (Exim 4.92) (envelope-from ) id 1rh4XT-00037D-D7; Mon, 04 Mar 2024 09:23:15 +0000 Received: by outflank-mailman (output) from mailman id 688228.1072186; Mon, 04 Mar 2024 09:23:15 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XT-00035g-8E; Mon, 04 Mar 2024 09:23:15 +0000 Received: by outflank-mailman (input) for mailman id 688228; Mon, 04 Mar 2024 09:23:13 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XR-0002pl-CR for xen-devel@lists.xenproject.org; Mon, 04 Mar 2024 09:23:13 +0000 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [2a00:1450:4864:20::632]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id d177b03e-da08-11ee-a1ee-f123f15fe8a2; Mon, 04 Mar 2024 10:23:10 +0100 (CET) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-a44d084bfe1so214999166b.1 for ; Mon, 04 Mar 2024 01:23:11 -0800 (PST) Received: from fhilly.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id tj10-20020a170907c24a00b00a444526962asm4485529ejc.128.2024.03.04.01.23.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 01:23:09 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d177b03e-da08-11ee-a1ee-f123f15fe8a2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloud.com; s=cloud; t=1709544190; x=1710148990; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=t8576Vz0UPo+rS38MVUiaswDJj20EFHOADQjBzvq4J4=; b=CPUwA8qQ93q1gT7CmMkdRFM4uQiNAY8URrf7QA8i0/6TYzLESo0DmPGaLIaEVKr8pc yousL2+uV866bx9YNR7kxzOQ95QpalI0gKu434mGS3RI6O6fiMYkt12Tzh7ovKkAN9jW M0lDPybxKeA43jyBZChsJaF6KiKacPiO4XlDA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709544190; x=1710148990; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t8576Vz0UPo+rS38MVUiaswDJj20EFHOADQjBzvq4J4=; b=lvYm2TPhIOnHHfWS+7U/2avHgwxB9NekWSlHHqPh3xCDlGjht3zU5tqdB6zLVYXO8e krQyieaWebfeOPxoZwh08DpiMiMqTOm1EuGsA4Fr6aWqGOI19r904as0806+hQ3opxch iOzaY10nKg0y/PCVDoGQHJUISJm4J4A6a4eZ090eQRzLKqXMzXNrd/WXc5s1Bad7H3p/ snBeF14Dp5r5fzgD6xTf3Z8WI5Bm8U/LI14POpe5RKVbxCZCKWgebL9dtEB2uVCHOk+v shkfKxQG2M+M95ewcf4rysUKcaDuRsLn099vXAOnaNc0CsS7Fk9mjXS4bLtfqJjSHtqn 4c3A== X-Gm-Message-State: AOJu0YwCpZmaPPznIcW3gVDRNNNd4PzqEXUx1a4CzsNFvMHSXw/1L2N1 RIQ7bq2IynLriNp5kHdcnZLxzu1UR29rXF7F6m2O4uj28iSP2aIjL8NanounCpshZyKZvB2E5rf 5hZQ= X-Google-Smtp-Source: AGHT+IE42eCZ6CjhAvzdETjVqdy8/jVPKO6w3yWfhfb1zLFXH9r02iHSEC9lZXhHAto+j08lEc2Dhg== X-Received: by 2002:a17:906:685:b0:a44:fe70:1b82 with SMTP id u5-20020a170906068500b00a44fe701b82mr2645062ejb.8.1709544190552; Mon, 04 Mar 2024 01:23:10 -0800 (PST) From: Fouad Hilly To: Xen-devel Cc: Wei Liu , Fouad Hilly , Jan Beulich , Andrew Cooper , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Wei Liu Subject: [PATCH 2/3] x86: introduce xstate_zero Date: Mon, 4 Mar 2024 09:13:06 +0000 Message-ID: <20240304091307.2295344-3-fouad.hilly@cloud.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240304091307.2295344-1-fouad.hilly@cloud.com> References: <20240304091307.2295344-1-fouad.hilly@cloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @cloud.com) X-ZM-MESSAGEID: 1709544215101100003 From: Wei Liu Factor out xrstor__ and introduce xstate_zero, which zeros all the state components specified in the mask. Signed-off-by: Wei Liu Signed-off-by: Fouad Hilly --- CC: Jan Beulich CC: Andrew Cooper CC: "Roger Pau Monn=C3=A9" CC: Wei Liu --- xen/arch/x86/include/asm/xstate.h | 1 + xen/arch/x86/xstate.c | 39 +++++++++++++++++++++++++------ 2 files changed, 33 insertions(+), 7 deletions(-) diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/x= state.h index c08c267884f0..bd767d9cd714 100644 --- a/xen/arch/x86/include/asm/xstate.h +++ b/xen/arch/x86/include/asm/xstate.h @@ -94,6 +94,7 @@ uint64_t get_msr_xss(void); uint64_t read_bndcfgu(void); void xsave(struct vcpu *v, uint64_t mask); void xrstor(struct vcpu *v, uint64_t mask); +void xstate_zero(uint64_t mask); void xstate_set_init(uint64_t mask); bool xsave_enabled(const struct vcpu *v); int __must_check validate_xstate(const struct domain *d, diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c index cf94761d0542..92a65bd8d52c 100644 --- a/xen/arch/x86/xstate.c +++ b/xen/arch/x86/xstate.c @@ -368,11 +368,12 @@ void xsave(struct vcpu *v, uint64_t mask) ptr->fpu_sse.x[FPU_WORD_SIZE_OFFSET] =3D fip_width; } =20 -void xrstor(struct vcpu *v, uint64_t mask) +/* True -> no error, false -> failed */ +static bool xrstor__(struct xsave_struct *ptr, uint64_t xcr0_accum, + uint64_t mask) { uint32_t hmask =3D mask >> 32; uint32_t lmask =3D mask; - struct xsave_struct *ptr =3D v->arch.xsave_area; unsigned int faults, prev_faults; =20 /* @@ -412,7 +413,7 @@ void xrstor(struct vcpu *v, uint64_t mask) [ptr] "D" (ptr) ) =20 #define XRSTOR(pfx) \ - if ( v->arch.xcr0_accum & XSTATE_XSAVES_ONLY ) \ + if ( xcr0_accum & XSTATE_XSAVES_ONLY ) \ { \ if ( unlikely(!(ptr->xsave_hdr.xcomp_bv & \ XSTATE_COMPACTION_ENABLED)) ) \ @@ -461,7 +462,7 @@ void xrstor(struct vcpu *v, uint64_t mask) ((mask & X86_XCR0_YMM) && !(ptr->xsave_hdr.xcomp_bv & XSTATE_COMPACTION_ENABLED))= ) ) ptr->fpu_sse.mxcsr &=3D mxcsr_mask; - if ( v->arch.xcr0_accum & XSTATE_XSAVES_ONLY ) + if ( xcr0_accum & XSTATE_XSAVES_ONLY ) { ptr->xsave_hdr.xcomp_bv &=3D this_cpu(xcr0) | this_cpu(xss= ); ptr->xsave_hdr.xstate_bv &=3D ptr->xsave_hdr.xcomp_bv; @@ -478,14 +479,35 @@ void xrstor(struct vcpu *v, uint64_t mask) case 2: /* Stage 2: Reset all state. */ ptr->fpu_sse.mxcsr =3D MXCSR_DEFAULT; ptr->xsave_hdr.xstate_bv =3D 0; - ptr->xsave_hdr.xcomp_bv =3D v->arch.xcr0_accum & XSTATE_XSAVES= _ONLY + ptr->xsave_hdr.xcomp_bv =3D xcr0_accum & XSTATE_XSAVES_ONLY ? XSTATE_COMPACTION_ENABLED : 0; continue; } =20 - domain_crash(current->domain); - return; + return false; } + + return true; +} + +void xrstor(struct vcpu *v, uint64_t mask) +{ + if ( !xrstor__(v->arch.xsave_area, v->arch.xcr0_accum, mask) ) + domain_crash(current->domain); +} + +void xstate_zero(uint64_t mask) +{ + bool ok; + struct xsave_struct tmp; + + tmp.fpu_sse.mxcsr =3D MXCSR_DEFAULT; + tmp.xsave_hdr.xstate_bv =3D 0; + tmp.xsave_hdr.xcomp_bv =3D 0; + memset(tmp.xsave_hdr.reserved, 0, sizeof(tmp.xsave_hdr.reserved)); + + ok =3D xrstor__(&tmp, mask, mask); + ASSERT(ok); } =20 bool xsave_enabled(const struct vcpu *v) @@ -731,6 +753,9 @@ int handle_xsetbv(u32 index, u64 new_bv) if ( (new_bv & ~xcr0_max) || !valid_xcr0(new_bv) ) return -EINVAL; =20 + /* Zero state components before writing new XCR0 */ + xstate_zero(get_xcr0()); + /* By this point, new_bv really should be accepted by hardware. */ if ( unlikely(!set_xcr0(new_bv)) ) { --=20 2.42.0 From nobody Mon May 20 21:57:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=cloud.com ARC-Seal: i=1; a=rsa-sha256; t=1709544216; cv=none; d=zohomail.com; s=zohoarc; b=m6LPPaeGcfpNyhc5xiFBwokxF6oSGTMpGpG2lTtWRRfp4sw5ZrXHBYhf2F4l4MsoRTFZihY/ByzB+iKiRXYqwYwYthWu1VCmX5ZMsquvme22AlMAI0OlHoa2aXmWvIZlOgUASrEuRukScBvnAwnmCu2F0MsDo8PX7D/GGS2Tua0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1709544216; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=nduCErVJMqVVQtvPr6lEcDydzA3X7giYTe88D2YcYv0=; b=DAw0WUSnFiCRIHPVmR/OHbZlzQ5U2w8TokQXYk1J7KH3hUFq4obPSVqqZgEKdxgN+cZWUKyiXeudBFZqQVUEzpLhxtrAN0fRV8D8rYrou6p9bxAngQioSxBGPn4+FtWvz7Q/ExQb+VmEWP1qbqjIx/k2poirDXqfDxHTlFAWDIU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1709544216689519.2330786298315; Mon, 4 Mar 2024 01:23:36 -0800 (PST) Received: from list by lists.xenproject.org with outflank-mailman.688229.1072200 (Exim 4.92) (envelope-from ) id 1rh4XU-0003X8-Ne; Mon, 04 Mar 2024 09:23:16 +0000 Received: by outflank-mailman (output) from mailman id 688229.1072200; Mon, 04 Mar 2024 09:23:16 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XU-0003Wg-Id; Mon, 04 Mar 2024 09:23:16 +0000 Received: by outflank-mailman (input) for mailman id 688229; Mon, 04 Mar 2024 09:23:15 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rh4XS-0002pf-QV for xen-devel@lists.xenproject.org; Mon, 04 Mar 2024 09:23:15 +0000 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [2a00:1450:4864:20::630]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id d2e682e4-da08-11ee-afda-a90da7624cb6; Mon, 04 Mar 2024 10:23:12 +0100 (CET) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-a45784702bbso26782766b.2 for ; Mon, 04 Mar 2024 01:23:12 -0800 (PST) Received: from fhilly.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id tj10-20020a170907c24a00b00a444526962asm4485529ejc.128.2024.03.04.01.23.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Mar 2024 01:23:11 -0800 (PST) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: d2e682e4-da08-11ee-afda-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cloud.com; s=cloud; t=1709544191; x=1710148991; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nduCErVJMqVVQtvPr6lEcDydzA3X7giYTe88D2YcYv0=; b=jWXs4Wq/0Q8nTI3KHNSs2GQZboZlQT1DeaBUwSzTIAVDZ/OvLlKn6w+Qi6JeInkZD1 5enoGvL18mqaFYtogjy+i29R4W0wImDfUpFZgobRJSN1zEkCmm7GARis12Q9krdg+8F0 aaJ/DmUz8brwgWPGIbRqTKgRWwqUyE5MR5Q60= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709544191; x=1710148991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nduCErVJMqVVQtvPr6lEcDydzA3X7giYTe88D2YcYv0=; b=Sc+XHgNuZK4BCwUaZa4xtJsx2F7AcnMbt5Fd7brKMMVP8CadvOom6OxtPYj3L6LcyJ 1pQRKNmW8yNEFSAXgUXWzLXdMzR4JjWl0wsjQ77wb3kxBw4NYeLSa+hfY3TzHG9NXcQm R5lUaCgOtbKX8+m1teoK8mSkI3GZn2CwlfNsEjHArMjRpn9PmAbcb/cpGx6wOZTxF8iN ifftt/W4iFeySHbsZ0vrSI6vfWm54HiOmvvzuvzwd1lMTV6BY5GKRfpXVxaPKPWwvyDo QZORX7SLqQuKLAlrNzpf4XnKtMbqI8HOlfGPntVK9HsQ1E6SPxtxiWUEBd+MrYHMJGSl 49sg== X-Gm-Message-State: AOJu0Ywb+5rC4g1t//Lmrhs9UOsOnuFzKpc2yAeQJ+aV8nrpDSfnfNvC ULZ1bVrjRAs57DznPicNkx3d4peQq/JfryS5JV+VOB3J4uPOZWnzStOKvapKLTplHoM7Et+tyE1 gFvo= X-Google-Smtp-Source: AGHT+IElbNVt8gtwOTgBUsCkmG6ilrMzz1ASszsJ4oxaKXL7Z8uxM2khjZSpQU/lSOaUzk6BKSWL1w== X-Received: by 2002:a17:906:7185:b0:a45:2091:8193 with SMTP id h5-20020a170906718500b00a4520918193mr2169268ejk.68.1709544191482; Mon, 04 Mar 2024 01:23:11 -0800 (PST) From: Fouad Hilly To: Xen-devel Cc: Wei Liu , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , Fouad Hilly , Jan Beulich , Andrew Cooper , Wei Liu , George Dunlap , Julien Grall , Stefano Stabellini , Paul Durrant , Jun Nakajima , Kevin Tian Subject: [PATCH 3/3] x86: switch to eager fpu save / restore Date: Mon, 4 Mar 2024 09:13:07 +0000 Message-ID: <20240304091307.2295344-4-fouad.hilly@cloud.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240304091307.2295344-1-fouad.hilly@cloud.com> References: <20240304091307.2295344-1-fouad.hilly@cloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @cloud.com) X-ZM-MESSAGEID: 1709544217160100001 From: Wei Liu Previously FPU is lazily switched. Due to the fact that a malicious guest can speculatively read the not yet switched out register states, we need to eagerly switch FPU states when a domain is scheduled to run. In the new world, Xen will eagerly switch FPU context in the scheduler. Xen itself won't set CR0.TS other than for the purpose of servicing a PV guest. The following things are done: 1. Xen will only set and clear CR0.TS on behalf of a PV guest. Any #NM received in Xen should only be delivered to the running PV guest. 2. Xen no longer causes vmexit for #NM for HVM guests when nested HVM is not in use. 3. When nested HVM is in use, Xen needs to trap #NM if specified by the L1 hypervisor, but all #NM handling is left to L1 hypervisor to deal with. 4. Xen saves and restores FPU states wherever it needs to. The following places are modified: 1. Scheduling in and out a guest; 2. Calling EFI runtime service; 3. ACPI reset; 4. x86 insn emulator fpu insn emulation. 5. Treat FPU as always initialised. Adjust following components: 1. HVM vcpu context save / load code; 2. arch_{get,set}_info_guest; 3. VLAPIC code. 6. Delete lazy FPU handling code. Strip XCR0 and IA32_XSS manipulation from __context_switch. We need to be able to zero out previously used state components. Push everything into fpu_xrstor as that's the most suitable place. Tested on AMD with PKU disabled and Intel, no performance degradation. Signed-off-by: Wei Liu Signed-off-by: Roger Pau Monn=C3=A9 Signed-off-by: Fouad Hilly --- CC: Jan Beulich CC: Andrew Cooper CC: "Roger Pau Monn=C3=A9" CC: Wei Liu CC: George Dunlap CC: Julien Grall CC: Stefano Stabellini CC: Paul Durrant CC: Jun Nakajima CC: Kevin Tian --- xen/arch/x86/cpu/common.c | 3 - xen/arch/x86/domain.c | 10 +- xen/arch/x86/domctl.c | 4 +- xen/arch/x86/hvm/emulate.c | 38 +---- xen/arch/x86/hvm/hvm.c | 8 +- xen/arch/x86/hvm/svm/nestedsvm.c | 62 +------ xen/arch/x86/hvm/svm/svm.c | 81 +-------- xen/arch/x86/hvm/svm/vmcb.c | 4 +- xen/arch/x86/hvm/vlapic.c | 4 - xen/arch/x86/hvm/vmx/vmcs.c | 8 +- xen/arch/x86/hvm/vmx/vmx.c | 70 +------- xen/arch/x86/hvm/vmx/vvmx.c | 15 +- xen/arch/x86/i387.c | 164 ++++--------------- xen/arch/x86/include/asm/domain.h | 3 - xen/arch/x86/include/asm/hvm/svm/nestedsvm.h | 3 - xen/arch/x86/include/asm/hvm/vmx/vmcs.h | 2 - xen/arch/x86/include/asm/i387.h | 3 +- xen/arch/x86/include/asm/xstate.h | 17 +- xen/arch/x86/pv/misc-hypercalls.c | 3 +- xen/arch/x86/traps.c | 18 +- xen/arch/x86/xstate.c | 26 --- xen/common/domain.c | 2 - xen/common/efi/runtime.c | 9 +- xen/include/xen/sched.h | 4 - 24 files changed, 72 insertions(+), 489 deletions(-) diff --git a/xen/arch/x86/cpu/common.c b/xen/arch/x86/cpu/common.c index 28d7f34c4dbe..9ea748f959af 100644 --- a/xen/arch/x86/cpu/common.c +++ b/xen/arch/x86/cpu/common.c @@ -967,9 +967,6 @@ void cpu_init(void) /* Install correct page table. */ write_ptbase(current); =20 - /* Ensure FPU gets initialised for each domain. */ - stts(); - /* Reset debug registers: */ write_debugreg(0, 0); write_debugreg(1, 0); diff --git a/xen/arch/x86/domain.c b/xen/arch/x86/domain.c index bda853e3c92b..742d69dd93bb 100644 --- a/xen/arch/x86/domain.c +++ b/xen/arch/x86/domain.c @@ -1996,15 +1996,7 @@ static void __context_switch(void) if ( !is_idle_domain(nd) ) { memcpy(stack_regs, &n->arch.user_regs, CTXT_SWITCH_STACK_BYTES); - if ( cpu_has_xsave ) - { - if ( !set_xcr0(n->arch.xcr0 ?: XSTATE_FP_SSE) ) - BUG(); - - if ( cpu_has_xsaves && is_hvm_vcpu(n) ) - set_msr_xss(n->arch.msrs->xss.raw); - } - vcpu_restore_fpu_nonlazy(n, false); + vcpu_restore_fpu(n); nd->arch.ctxt_switch->to(n); } =20 diff --git a/xen/arch/x86/domctl.c b/xen/arch/x86/domctl.c index 9a72d57333e9..d838427feb52 100644 --- a/xen/arch/x86/domctl.c +++ b/xen/arch/x86/domctl.c @@ -967,7 +967,6 @@ long arch_do_domctl( =20 v->arch.xcr0 =3D _xcr0; v->arch.xcr0_accum =3D _xcr0_accum; - v->arch.nonlazy_xstate_used =3D _xcr0_accum & XSTATE_NONLA= ZY; compress_xsave_states(v, _xsave_area, evc->size - PV_XSAVE_HDR_SIZE); =20 @@ -1347,8 +1346,7 @@ void arch_get_info_guest(struct vcpu *v, vcpu_guest_c= ontext_u c) c(flags =3D v->arch.pv.vgc_flags & ~(VGCF_i387_valid|VGCF_in_kerne= l)); else c(flags =3D 0); - if ( v->fpu_initialised ) - c(flags |=3D VGCF_i387_valid); + c(flags |=3D VGCF_i387_valid); if ( !(v->pause_flags & VPF_down) ) c(flags |=3D VGCF_online); if ( !compat ) diff --git a/xen/arch/x86/hvm/emulate.c b/xen/arch/x86/hvm/emulate.c index ab1bc516839a..8a1720008454 100644 --- a/xen/arch/x86/hvm/emulate.c +++ b/xen/arch/x86/hvm/emulate.c @@ -2361,9 +2361,7 @@ static int cf_check hvmemul_get_fpu( { struct vcpu *curr =3D current; =20 - if ( !curr->fpu_dirtied ) - alternative_vcall(hvm_funcs.fpu_dirty_intercept); - else if ( type =3D=3D X86EMUL_FPU_fpu ) + if ( type =3D=3D X86EMUL_FPU_fpu ) { const typeof(curr->arch.xsave_area->fpu_sse) *fpu_ctxt =3D curr->arch.fpu_ctxt; @@ -2372,16 +2370,8 @@ static int cf_check hvmemul_get_fpu( * Latch current register state so that we can back out changes * if needed (namely when a memory write fails after register state * has already been updated). - * NB: We don't really need the "enable" part of the called functi= on - * (->fpu_dirtied set implies CR0.TS clear), but the additional - * overhead should be low enough to not warrant introduction of yet - * another slightly different function. However, we need to undo t= he - * ->fpu_dirtied clearing the function does as well as the possible - * masking of all exceptions by FNSTENV.) */ - save_fpu_enable(); - curr->fpu_initialised =3D true; - curr->fpu_dirtied =3D true; + vcpu_save_fpu(curr); if ( (fpu_ctxt->fcw & 0x3f) !=3D 0x3f ) { uint16_t fcw; @@ -2415,12 +2405,8 @@ static void cf_check hvmemul_put_fpu( * Latch current register state so that we can replace FIP/FDP/FOP * (which have values resulting from our own invocation of the FPU * instruction during emulation). - * NB: See also the comment in hvmemul_get_fpu(); we don't need to - * set ->fpu_dirtied here as it is going to be cleared below, and - * we also don't need to reload FCW as we're forcing full state to - * be reloaded anyway. */ - save_fpu_enable(); + vcpu_save_fpu(curr); =20 if ( boot_cpu_has(X86_FEATURE_FDP_EXCP_ONLY) && !(fpu_ctxt->fsw & ~fpu_ctxt->fcw & 0x003f) ) @@ -2465,23 +2451,7 @@ static void cf_check hvmemul_put_fpu( } =20 if ( backout =3D=3D X86EMUL_FPU_fpu ) - { - /* - * To back out changes to the register file - * - in fully eager mode, restore original state immediately, - * - in lazy mode, simply adjust state such that upon next FPU insn - * use by the guest we'll reload the state saved (or freshly loa= ded) - * by hvmemul_get_fpu(). - */ - if ( curr->arch.fully_eager_fpu ) - vcpu_restore_fpu_nonlazy(curr, false); - else - { - curr->fpu_dirtied =3D false; - stts(); - alternative_vcall(hvm_funcs.fpu_leave, curr); - } - } + vcpu_restore_fpu(curr); } =20 static int cf_check hvmemul_tlb_op( diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index c75959588c0e..e532e2a2d6a0 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -899,11 +899,8 @@ static int cf_check hvm_save_cpu_ctxt(struct vcpu *v, = hvm_domain_context_t *h) ctxt.ldtr_base =3D seg.base; ctxt.ldtr_arbytes =3D seg.attr; =20 - if ( v->fpu_initialised ) - { - memcpy(ctxt.fpu_regs, v->arch.fpu_ctxt, sizeof(ctxt.fpu_regs)); - ctxt.flags =3D XEN_X86_FPU_INITIALISED; - } + memcpy(ctxt.fpu_regs, v->arch.fpu_ctxt, sizeof(ctxt.fpu_regs)); + ctxt.flags =3D XEN_X86_FPU_INITIALISED; =20 return hvm_save_entry(CPU, v->vcpu_id, h, &ctxt); } @@ -1342,7 +1339,6 @@ static int cf_check hvm_load_cpu_xsave_states( =20 v->arch.xcr0 =3D ctxt->xcr0; v->arch.xcr0_accum =3D ctxt->xcr0_accum; - v->arch.nonlazy_xstate_used =3D ctxt->xcr0_accum & XSTATE_NONLAZY; compress_xsave_states(v, &ctxt->save_area, size - offsetof(struct hvm_hw_cpu_xsave, save_ar= ea)); =20 diff --git a/xen/arch/x86/hvm/svm/nestedsvm.c b/xen/arch/x86/hvm/svm/nested= svm.c index e4e01add8c5b..9f7595ec34f6 100644 --- a/xen/arch/x86/hvm/svm/nestedsvm.c +++ b/xen/arch/x86/hvm/svm/nestedsvm.c @@ -168,58 +168,6 @@ int cf_check nsvm_vcpu_reset(struct vcpu *v) return 0; } =20 -static uint64_t nestedsvm_fpu_vmentry(uint64_t n1cr0, - struct vmcb_struct *vvmcb, - struct vmcb_struct *n1vmcb, struct vmcb_struct *n2vmcb) -{ - uint64_t vcr0; - - vcr0 =3D vvmcb->_cr0; - if ( !(n1cr0 & X86_CR0_TS) && (n1vmcb->_cr0 & X86_CR0_TS) ) - { - /* - * svm_fpu_leave() run while l1 guest was running. - * Sync FPU state with l2 guest. - */ - vcr0 |=3D X86_CR0_TS; - n2vmcb->_exception_intercepts |=3D (1U << X86_EXC_NM); - } - else if ( !(vcr0 & X86_CR0_TS) && (n2vmcb->_cr0 & X86_CR0_TS) ) - { - /* - * svm_fpu_enter() run while l1 guest was running. - * Sync FPU state with l2 guest. - */ - vcr0 &=3D ~X86_CR0_TS; - n2vmcb->_exception_intercepts &=3D ~(1U << X86_EXC_NM); - } - - return vcr0; -} - -static void nestedsvm_fpu_vmexit(struct vmcb_struct *n1vmcb, - struct vmcb_struct *n2vmcb, uint64_t n1cr0, uint64_t guest_cr0) -{ - if ( !(guest_cr0 & X86_CR0_TS) && (n2vmcb->_cr0 & X86_CR0_TS) ) - { - /* - * svm_fpu_leave() run while l2 guest was running. - * Sync FPU state with l1 guest. - */ - n1vmcb->_cr0 |=3D X86_CR0_TS; - n1vmcb->_exception_intercepts |=3D (1U << X86_EXC_NM); - } - else if ( !(n1cr0 & X86_CR0_TS) && (n1vmcb->_cr0 & X86_CR0_TS) ) - { - /* - * svm_fpu_enter() run while l2 guest was running. - * Sync FPU state with l1 guest. - */ - n1vmcb->_cr0 &=3D ~X86_CR0_TS; - n1vmcb->_exception_intercepts &=3D ~(1U << X86_EXC_NM); - } -} - static int nsvm_vcpu_hostsave(struct vcpu *v, unsigned int inst_len) { struct nestedsvm *svm =3D &vcpu_nestedsvm(v); @@ -249,7 +197,6 @@ static int nsvm_vcpu_hostsave(struct vcpu *v, unsigned = int inst_len) static int nsvm_vcpu_hostrestore(struct vcpu *v, struct cpu_user_regs *reg= s) { struct nestedvcpu *nv =3D &vcpu_nestedhvm(v); - struct nestedsvm *svm =3D &vcpu_nestedsvm(v); struct vmcb_struct *n1vmcb, *n2vmcb; int rc; =20 @@ -284,8 +231,6 @@ static int nsvm_vcpu_hostrestore(struct vcpu *v, struct= cpu_user_regs *regs) gdprintk(XENLOG_ERR, "hvm_set_cr4 failed, rc: %u\n", rc); =20 /* CR0 */ - nestedsvm_fpu_vmexit(n1vmcb, n2vmcb, - svm->ns_cr0, v->arch.hvm.guest_cr[0]); v->arch.hvm.guest_cr[0] =3D n1vmcb->_cr0 | X86_CR0_PE; n1vmcb->rflags &=3D ~X86_EFLAGS_VM; rc =3D hvm_set_cr0(n1vmcb->_cr0 | X86_CR0_PE, true); @@ -293,7 +238,6 @@ static int nsvm_vcpu_hostrestore(struct vcpu *v, struct= cpu_user_regs *regs) hvm_inject_hw_exception(X86_EXC_GP, 0); if ( rc !=3D X86EMUL_OKAY ) gdprintk(XENLOG_ERR, "hvm_set_cr0 failed, rc: %u\n", rc); - svm->ns_cr0 =3D v->arch.hvm.guest_cr[0]; =20 /* CR2 */ v->arch.hvm.guest_cr[2] =3D n1vmcb->_cr2; @@ -421,7 +365,6 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, stru= ct cpu_user_regs *regs) struct vmcb_struct *ns_vmcb, *n1vmcb, *n2vmcb; vmcbcleanbits_t clean =3D {}; int rc; - uint64_t cr0; =20 ns_vmcb =3D nv->nv_vvmcx; n1vmcb =3D nv->nv_n1vmcx; @@ -455,7 +398,6 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, stru= ct cpu_user_regs *regs) * safed here. * The overhead comes from (ordered from highest to lowest): * - svm_ctxt_switch_to (CPU context switching) - * - svm_fpu_enter, svm_fpu_leave (lazy FPU switching) * - emulated CLGI (clears VINTR intercept) * - host clears VINTR intercept * Test results show that the overhead is high enough that the @@ -554,10 +496,8 @@ static int nsvm_vmcb_prepare4vmrun(struct vcpu *v, str= uct cpu_user_regs *regs) gdprintk(XENLOG_ERR, "hvm_set_cr4 failed, rc: %u\n", rc); =20 /* CR0 */ - svm->ns_cr0 =3D v->arch.hvm.guest_cr[0]; - cr0 =3D nestedsvm_fpu_vmentry(svm->ns_cr0, ns_vmcb, n1vmcb, n2vmcb); v->arch.hvm.guest_cr[0] =3D ns_vmcb->_cr0; - rc =3D hvm_set_cr0(cr0, true); + rc =3D hvm_set_cr0(ns_vmcb->_cr0, true); if ( rc =3D=3D X86EMUL_EXCEPTION ) hvm_inject_hw_exception(X86_EXC_GP, 0); if ( rc !=3D X86EMUL_OKAY ) diff --git a/xen/arch/x86/hvm/svm/svm.c b/xen/arch/x86/hvm/svm/svm.c index b551eac8078d..7a92b432e788 100644 --- a/xen/arch/x86/hvm/svm/svm.c +++ b/xen/arch/x86/hvm/svm/svm.c @@ -104,38 +104,6 @@ static void cf_check svm_cpu_down(void) write_efer(read_efer() & ~EFER_SVME); } =20 -static void svm_fpu_enter(struct vcpu *v) -{ - struct vmcb_struct *n1vmcb =3D vcpu_nestedhvm(v).nv_n1vmcx; - - vcpu_restore_fpu_lazy(v); - vmcb_set_exception_intercepts( - n1vmcb, - vmcb_get_exception_intercepts(n1vmcb) & ~(1U << X86_EXC_NM)); -} - -static void cf_check svm_fpu_leave(struct vcpu *v) -{ - struct vmcb_struct *n1vmcb =3D vcpu_nestedhvm(v).nv_n1vmcx; - - ASSERT(!v->fpu_dirtied); - ASSERT(read_cr0() & X86_CR0_TS); - - /* - * If the guest does not have TS enabled then we must cause and handle= an - * exception on first use of the FPU. If the guest *does* have TS enab= led - * then this is not necessary: no FPU activity can occur until the gue= st - * clears CR0.TS, and we will initialise the FPU when that happens. - */ - if ( !(v->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - { - vmcb_set_exception_intercepts( - n1vmcb, - vmcb_get_exception_intercepts(n1vmcb) | (1U << X86_EXC_NM)); - vmcb_set_cr0(n1vmcb, vmcb_get_cr0(n1vmcb) | X86_CR0_TS); - } -} - static void cf_check svm_update_guest_cr( struct vcpu *v, unsigned int cr, unsigned int flags) { @@ -145,20 +113,6 @@ static void cf_check svm_update_guest_cr( switch ( cr ) { case 0: - { - unsigned long hw_cr0_mask =3D 0; - - if ( !(v->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - { - if ( v !=3D current ) - { - if ( !v->arch.fully_eager_fpu ) - hw_cr0_mask |=3D X86_CR0_TS; - } - else if ( vmcb_get_cr0(vmcb) & X86_CR0_TS ) - svm_fpu_enter(v); - } - if ( paging_mode_hap(v->domain) ) { uint32_t intercepts =3D vmcb_get_cr_intercepts(vmcb); @@ -169,12 +123,12 @@ static void cf_check svm_update_guest_cr( vmcb_set_cr_intercepts(vmcb, intercepts | CR_INTERCEPT_CR3_= WRITE); } =20 - value =3D v->arch.hvm.guest_cr[0] | hw_cr0_mask; + value =3D v->arch.hvm.guest_cr[0]; if ( !paging_mode_hap(v->domain) ) value |=3D X86_CR0_PG | X86_CR0_WP; vmcb_set_cr0(vmcb, value); break; - } + case 2: vmcb_set_cr2(vmcb, v->arch.hvm.guest_cr[2]); break; @@ -950,9 +904,6 @@ static void cf_check svm_ctxt_switch_from(struct vcpu *= v) if ( unlikely((read_efer() & EFER_SVME) =3D=3D 0) ) return; =20 - if ( !v->arch.fully_eager_fpu ) - svm_fpu_leave(v); - svm_save_dr(v); svm_tsc_ratio_save(v); =20 @@ -1709,28 +1660,6 @@ static void svm_do_nested_pgfault(struct vcpu *v, domain_crash(v->domain); } =20 -static void cf_check svm_fpu_dirty_intercept(void) -{ - struct vcpu *v =3D current; - struct vmcb_struct *vmcb =3D v->arch.hvm.svm.vmcb; - struct vmcb_struct *n1vmcb =3D vcpu_nestedhvm(v).nv_n1vmcx; - - svm_fpu_enter(v); - - if ( vmcb !=3D n1vmcb ) - { - /* Check if l1 guest must make FPU ready for the l2 guest */ - if ( v->arch.hvm.guest_cr[0] & X86_CR0_TS ) - hvm_inject_hw_exception(X86_EXC_NM, X86_EVENT_NO_EC); - else - vmcb_set_cr0(n1vmcb, vmcb_get_cr0(n1vmcb) & ~X86_CR0_TS); - return; - } - - if ( !(v->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - vmcb_set_cr0(vmcb, vmcb_get_cr0(vmcb) & ~X86_CR0_TS); -} - static void svm_vmexit_do_cr_access( struct vmcb_struct *vmcb, struct cpu_user_regs *regs) { @@ -2499,7 +2428,6 @@ static struct hvm_function_table __initdata_cf_clobbe= r svm_function_table =3D { .update_guest_cr =3D svm_update_guest_cr, .update_guest_efer =3D svm_update_guest_efer, .cpuid_policy_changed =3D svm_cpuid_policy_changed, - .fpu_leave =3D svm_fpu_leave, .set_guest_pat =3D svm_set_guest_pat, .get_guest_pat =3D svm_get_guest_pat, .set_tsc_offset =3D svm_set_tsc_offset, @@ -2509,7 +2437,6 @@ static struct hvm_function_table __initdata_cf_clobbe= r svm_function_table =3D { .get_pending_event =3D svm_get_pending_event, .invlpg =3D svm_invlpg, .wbinvd_intercept =3D svm_wbinvd_intercept, - .fpu_dirty_intercept =3D svm_fpu_dirty_intercept, .msr_read_intercept =3D svm_msr_read_intercept, .msr_write_intercept =3D svm_msr_write_intercept, .enable_msr_interception =3D svm_enable_msr_interception, @@ -2802,10 +2729,6 @@ void asmlinkage svm_vmexit_handler(void) } break; =20 - case VMEXIT_EXCEPTION_NM: - svm_fpu_dirty_intercept(); - break; - case VMEXIT_EXCEPTION_PF: { unsigned long va =3D vmcb->ei.exc.cr2; diff --git a/xen/arch/x86/hvm/svm/vmcb.c b/xen/arch/x86/hvm/svm/vmcb.c index 282fe7cdbe39..bf568d256f5e 100644 --- a/xen/arch/x86/hvm/svm/vmcb.c +++ b/xen/arch/x86/hvm/svm/vmcb.c @@ -127,9 +127,7 @@ static int construct_vmcb(struct vcpu *v) =20 paging_update_paging_modes(v); =20 - vmcb->_exception_intercepts =3D - HVM_TRAP_MASK | - (v->arch.fully_eager_fpu ? 0 : (1U << X86_EXC_NM)); + vmcb->_exception_intercepts =3D HVM_TRAP_MASK; =20 if ( paging_mode_hap(v->domain) ) { diff --git a/xen/arch/x86/hvm/vlapic.c b/xen/arch/x86/hvm/vlapic.c index dcbcf4a1feb5..2551260acd4d 100644 --- a/xen/arch/x86/hvm/vlapic.c +++ b/xen/arch/x86/hvm/vlapic.c @@ -263,7 +263,6 @@ static void vlapic_init_sipi_one(struct vcpu *target, u= int32_t icr) switch ( icr & APIC_DM_MASK ) { case APIC_DM_INIT: { - bool fpu_initialised; int rc; =20 /* No work on INIT de-assert for P4-type APIC. */ @@ -275,11 +274,8 @@ static void vlapic_init_sipi_one(struct vcpu *target, = uint32_t icr) break; hvm_vcpu_down(target); domain_lock(target->domain); - /* Reset necessary VCPU state. This does not include FPU state. */ - fpu_initialised =3D target->fpu_initialised; rc =3D vcpu_reset(target); ASSERT(!rc); - target->fpu_initialised =3D fpu_initialised; vlapic_do_init(vcpu_vlapic(target)); domain_unlock(target->domain); break; diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c index 6b4f004a3d21..25bee5e5c404 100644 --- a/xen/arch/x86/hvm/vmx/vmcs.c +++ b/xen/arch/x86/hvm/vmx/vmcs.c @@ -1242,10 +1242,7 @@ static int construct_vmcs(struct vcpu *v) __vmwrite(HOST_TR_SELECTOR, TSS_SELECTOR); =20 /* Host control registers. */ - v->arch.hvm.vmx.host_cr0 =3D read_cr0() & ~X86_CR0_TS; - if ( !v->arch.fully_eager_fpu ) - v->arch.hvm.vmx.host_cr0 |=3D X86_CR0_TS; - __vmwrite(HOST_CR0, v->arch.hvm.vmx.host_cr0); + __vmwrite(HOST_CR0, read_cr0()); __vmwrite(HOST_CR4, mmu_cr4_features); if ( cpu_has_vmx_efer ) __vmwrite(HOST_EFER, read_efer()); @@ -1325,8 +1322,7 @@ static int construct_vmcs(struct vcpu *v) __vmwrite(VMCS_LINK_POINTER, ~0UL); =20 v->arch.hvm.vmx.exception_bitmap =3D HVM_TRAP_MASK - | (paging_mode_hap(d) ? 0 : (1U << X86_EXC_PF)) - | (v->arch.fully_eager_fpu ? 0 : (1U << X86_EXC_NM)); + | (paging_mode_hap(d) ? 0 : (1U << X86_EXC_PF)); =20 if ( cpu_has_vmx_notify_vm_exiting ) __vmwrite(NOTIFY_WINDOW, vm_notify_window); diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c index 23bdf0d1957f..6e21e8b9d396 100644 --- a/xen/arch/x86/hvm/vmx/vmx.c +++ b/xen/arch/x86/hvm/vmx/vmx.c @@ -63,7 +63,6 @@ static void cf_check vmx_update_guest_cr( struct vcpu *v, unsigned int cr, unsigned int flags); static void cf_check vmx_update_guest_efer(struct vcpu *v); static void cf_check vmx_wbinvd_intercept(void); -static void cf_check vmx_fpu_dirty_intercept(void); static int cf_check vmx_msr_read_intercept( unsigned int msr, uint64_t *msr_content); static int cf_check vmx_msr_write_intercept( @@ -1087,41 +1086,6 @@ static int cf_check vmx_load_vmcs_ctxt(struct vcpu *= v, struct hvm_hw_cpu *ctxt) return 0; } =20 -static void vmx_fpu_enter(struct vcpu *v) -{ - vcpu_restore_fpu_lazy(v); - v->arch.hvm.vmx.exception_bitmap &=3D ~(1u << X86_EXC_NM); - vmx_update_exception_bitmap(v); - v->arch.hvm.vmx.host_cr0 &=3D ~X86_CR0_TS; - __vmwrite(HOST_CR0, v->arch.hvm.vmx.host_cr0); -} - -static void cf_check vmx_fpu_leave(struct vcpu *v) -{ - ASSERT(!v->fpu_dirtied); - ASSERT(read_cr0() & X86_CR0_TS); - - if ( !(v->arch.hvm.vmx.host_cr0 & X86_CR0_TS) ) - { - v->arch.hvm.vmx.host_cr0 |=3D X86_CR0_TS; - __vmwrite(HOST_CR0, v->arch.hvm.vmx.host_cr0); - } - - /* - * If the guest does not have TS enabled then we must cause and handle= an - * exception on first use of the FPU. If the guest *does* have TS enab= led - * then this is not necessary: no FPU activity can occur until the gue= st - * clears CR0.TS, and we will initialise the FPU when that happens. - */ - if ( !(v->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - { - v->arch.hvm.hw_cr[0] |=3D X86_CR0_TS; - __vmwrite(GUEST_CR0, v->arch.hvm.hw_cr[0]); - v->arch.hvm.vmx.exception_bitmap |=3D (1u << X86_EXC_NM); - vmx_update_exception_bitmap(v); - } -} - static void cf_check vmx_ctxt_switch_from(struct vcpu *v) { /* @@ -1144,8 +1108,6 @@ static void cf_check vmx_ctxt_switch_from(struct vcpu= *v) vmx_vmcs_reload(v); } =20 - if ( !v->arch.fully_eager_fpu ) - vmx_fpu_leave(v); vmx_save_guest_msrs(v); vmx_restore_host_msrs(); vmx_save_dr(v); @@ -1666,17 +1628,6 @@ static void cf_check vmx_update_guest_cr( else nvmx_set_cr_read_shadow(v, 0); =20 - if ( !(v->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - { - if ( v !=3D current ) - { - if ( !v->arch.fully_eager_fpu ) - hw_cr0_mask |=3D X86_CR0_TS; - } - else if ( v->arch.hvm.hw_cr[0] & X86_CR0_TS ) - vmx_fpu_enter(v); - } - realmode =3D !(v->arch.hvm.guest_cr[0] & X86_CR0_PE); =20 if ( !vmx_unrestricted_guest(v) && @@ -2733,7 +2684,6 @@ static struct hvm_function_table __initdata_cf_clobbe= r vmx_function_table =3D { .update_guest_cr =3D vmx_update_guest_cr, .update_guest_efer =3D vmx_update_guest_efer, .cpuid_policy_changed =3D vmx_cpuid_policy_changed, - .fpu_leave =3D vmx_fpu_leave, .set_guest_pat =3D vmx_set_guest_pat, .get_guest_pat =3D vmx_get_guest_pat, .set_tsc_offset =3D vmx_set_tsc_offset, @@ -2745,7 +2695,6 @@ static struct hvm_function_table __initdata_cf_clobbe= r vmx_function_table =3D { .cpu_up =3D vmx_cpu_up, .cpu_down =3D vmx_cpu_down, .wbinvd_intercept =3D vmx_wbinvd_intercept, - .fpu_dirty_intercept =3D vmx_fpu_dirty_intercept, .msr_read_intercept =3D vmx_msr_read_intercept, .msr_write_intercept =3D vmx_msr_write_intercept, .handle_cd =3D vmx_handle_cd, @@ -3066,20 +3015,6 @@ void update_guest_eip(void) hvm_inject_hw_exception(X86_EXC_DB, X86_EVENT_NO_EC); } =20 -static void cf_check vmx_fpu_dirty_intercept(void) -{ - struct vcpu *curr =3D current; - - vmx_fpu_enter(curr); - - /* Disable TS in guest CR0 unless the guest wants the exception too. */ - if ( !(curr->arch.hvm.guest_cr[0] & X86_CR0_TS) ) - { - curr->arch.hvm.hw_cr[0] &=3D ~X86_CR0_TS; - __vmwrite(GUEST_CR0, curr->arch.hvm.hw_cr[0]); - } -} - static void vmx_dr_access(unsigned long exit_qualification, struct cpu_user_regs *regs) { @@ -4325,10 +4260,7 @@ void asmlinkage vmx_vmexit_handler(struct cpu_user_r= egs *regs) domain_pause_for_debugger(); } break; - case X86_EXC_NM: - HVMTRACE_1D(TRAP, vector); - vmx_fpu_dirty_intercept(); - break; + case X86_EXC_PF: __vmread(EXIT_QUALIFICATION, &exit_qualification); __vmread(VM_EXIT_INTR_ERROR_CODE, &ecode); diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c index ece0aa243a73..497592969dfc 100644 --- a/xen/arch/x86/hvm/vmx/vvmx.c +++ b/xen/arch/x86/hvm/vmx/vvmx.c @@ -1236,9 +1236,6 @@ static void virtual_vmentry(struct cpu_user_regs *reg= s) regs->rsp =3D get_vvmcs(v, GUEST_RSP); regs->rflags =3D get_vvmcs(v, GUEST_RFLAGS); =20 - /* updating host cr0 to sync TS bit */ - __vmwrite(HOST_CR0, v->arch.hvm.vmx.host_cr0); - /* Setup virtual ETP for L2 guest*/ if ( nestedhvm_paging_mode_hap(v) ) /* This will setup the initial np2m for the nested vCPU */ @@ -1466,9 +1463,6 @@ static void virtual_vmexit(struct cpu_user_regs *regs) /* VM exit clears all bits except bit 1 */ regs->rflags =3D X86_EFLAGS_MBS; =20 - /* updating host cr0 to sync TS bit */ - __vmwrite(HOST_CR0, v->arch.hvm.vmx.host_cr0); - if ( cpu_has_vmx_virtual_intr_delivery ) nvmx_update_apicv(v); =20 @@ -2454,19 +2448,14 @@ int nvmx_n2_vmexit_handler(struct cpu_user_regs *re= gs, __vmread(VM_EXIT_INTR_INFO, &intr_info); vector =3D intr_info & INTR_INFO_VECTOR_MASK; /* - * decided by L0 and L1 exception bitmap, if the vetor is set by - * both, L0 has priority on #PF and #NM, L1 has priority on others + * decided by L0 and L1 exception bitmap, if the vector is set by + * both, L0 has priority on #PF, L1 has priority on others */ if ( vector =3D=3D X86_EXC_PF ) { if ( paging_mode_hap(v->domain) ) nvcpu->nv_vmexit_pending =3D 1; } - else if ( vector =3D=3D X86_EXC_NM ) - { - if ( v->fpu_dirtied ) - nvcpu->nv_vmexit_pending =3D 1; - } else if ( (intr_info & valid_mask) =3D=3D valid_mask ) { exec_bitmap =3D get_vvmcs(v, EXCEPTION_BITMAP); diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c index 7a4297cc921e..09df2c962c9a 100644 --- a/xen/arch/x86/i387.c +++ b/xen/arch/x86/i387.c @@ -20,7 +20,7 @@ /* FPU Restore Functions */ /*******************************/ /* Restore x87 extended state */ -static inline void fpu_xrstor(struct vcpu *v, uint64_t mask) +static inline void fpu_xrstor(struct vcpu *v) { bool ok; =20 @@ -28,10 +28,14 @@ static inline void fpu_xrstor(struct vcpu *v, uint64_t = mask) /* * XCR0 normally represents what guest OS set. In case of Xen itself, * we set the accumulated feature mask before doing save/restore. + * + * Combine the outgoing and incoming XCR0 before calling xrstor to make + * sure any state component used by outgoing vcpu is cleared. Rewrite = XCR0 + * to be the ones used by incoming vcpu afterwards. */ - ok =3D set_xcr0(v->arch.xcr0_accum | XSTATE_FP_SSE); + ok =3D set_xcr0(v->arch.xcr0_accum | get_xcr0() | XSTATE_FP_SSE); ASSERT(ok); - xrstor(v, mask); + xrstor(v, XSTATE_ALL); ok =3D set_xcr0(v->arch.xcr0 ?: XSTATE_FP_SSE); ASSERT(ok); } @@ -110,33 +114,11 @@ static inline void fpu_fxrstor(struct vcpu *v) /* FPU Save Functions */ /*******************************/ =20 -static inline uint64_t vcpu_xsave_mask(const struct vcpu *v) -{ - if ( v->fpu_dirtied ) - return v->arch.nonlazy_xstate_used ? XSTATE_ALL : XSTATE_LAZY; - - ASSERT(v->arch.nonlazy_xstate_used); - - /* - * The offsets of components which live in the extended region of - * compact xsave area are not fixed. Xsave area may be overwritten - * when a xsave with v->fpu_dirtied set is followed by one with - * v->fpu_dirtied clear. - * In such case, if hypervisor uses compact xsave area and guest - * has ever used lazy states (checking xcr0_accum excluding - * XSTATE_FP_SSE), vcpu_xsave_mask will return XSTATE_ALL. Otherwise - * return XSTATE_NONLAZY. - */ - return xstate_all(v) ? XSTATE_ALL : XSTATE_NONLAZY; -} - /* Save x87 extended state */ static inline void fpu_xsave(struct vcpu *v) { bool ok; - uint64_t mask =3D vcpu_xsave_mask(v); =20 - ASSERT(mask); ASSERT(v->arch.xsave_area); /* * XCR0 normally represents what guest OS set. In case of Xen itself, @@ -144,7 +126,7 @@ static inline void fpu_xsave(struct vcpu *v) */ ok =3D set_xcr0(v->arch.xcr0_accum | XSTATE_FP_SSE); ASSERT(ok); - xsave(v, mask); + xsave(v, XSTATE_ALL); ok =3D set_xcr0(v->arch.xcr0 ?: XSTATE_FP_SSE); ASSERT(ok); } @@ -202,107 +184,6 @@ static inline void fpu_fxsave(struct vcpu *v) /*******************************/ /* VCPU FPU Functions */ /*******************************/ -/* Restore FPU state whenever VCPU is schduled in. */ -void vcpu_restore_fpu_nonlazy(struct vcpu *v, bool need_stts) -{ - /* Restore nonlazy extended state (i.e. parts not tracked by CR0.TS). = */ - if ( !v->arch.fully_eager_fpu && !v->arch.nonlazy_xstate_used ) - goto maybe_stts; - - ASSERT(!is_idle_vcpu(v)); - - /* Avoid recursion */ - clts(); - - /* - * When saving full state even with !v->fpu_dirtied (see vcpu_xsave_ma= sk() - * above) we also need to restore full state, to prevent subsequently - * saving state belonging to another vCPU. - */ - if ( v->arch.fully_eager_fpu || (v->arch.xsave_area && xstate_all(v)) ) - { - if ( cpu_has_xsave ) - fpu_xrstor(v, XSTATE_ALL); - else - fpu_fxrstor(v); - - v->fpu_initialised =3D 1; - v->fpu_dirtied =3D 1; - - /* Xen doesn't need TS set, but the guest might. */ - need_stts =3D is_pv_vcpu(v) && (v->arch.pv.ctrlreg[0] & X86_CR0_TS= ); - } - else - { - fpu_xrstor(v, XSTATE_NONLAZY); - need_stts =3D true; - } - - maybe_stts: - if ( need_stts ) - stts(); -} - -/*=20 - * Restore FPU state when #NM is triggered. - */ -void vcpu_restore_fpu_lazy(struct vcpu *v) -{ - ASSERT(!is_idle_vcpu(v)); - - /* Avoid recursion. */ - clts(); - - if ( v->fpu_dirtied ) - return; - - ASSERT(!v->arch.fully_eager_fpu); - - if ( cpu_has_xsave ) - fpu_xrstor(v, XSTATE_LAZY); - else - fpu_fxrstor(v); - - v->fpu_initialised =3D 1; - v->fpu_dirtied =3D 1; -} - -/*=20 - * On each context switch, save the necessary FPU info of VCPU being switc= h=20 - * out. It dispatches saving operation based on CPU's capability. - */ -static bool _vcpu_save_fpu(struct vcpu *v) -{ - if ( !v->fpu_dirtied && !v->arch.nonlazy_xstate_used ) - return false; - - ASSERT(!is_idle_vcpu(v)); - - /* This can happen, if a paravirtualised guest OS has set its CR0.TS. = */ - clts(); - - if ( cpu_has_xsave ) - fpu_xsave(v); - else - fpu_fxsave(v); - - v->fpu_dirtied =3D 0; - - return true; -} - -void vcpu_save_fpu(struct vcpu *v) -{ - _vcpu_save_fpu(v); - stts(); -} - -void save_fpu_enable(void) -{ - if ( !_vcpu_save_fpu(current) ) - clts(); -} - /* Initialize FPU's context save area */ int vcpu_init_fpu(struct vcpu *v) { @@ -347,8 +228,6 @@ void vcpu_setup_fpu(struct vcpu *v, struct xsave_struct= *xsave_area, =20 ASSERT(!xsave_area || xsave_area =3D=3D v->arch.xsave_area); =20 - v->fpu_initialised =3D !!data; - if ( data ) { memcpy(fpu_sse, data, sizeof(*fpu_sse)); @@ -386,6 +265,33 @@ void vcpu_destroy_fpu(struct vcpu *v) xfree(v->arch.fpu_ctxt); } =20 +void vcpu_save_fpu(struct vcpu *v) +{ + ASSERT(!is_idle_vcpu(v)); + + /* This can happen, if a paravirtualised guest OS has set its CR0.TS. = */ + clts(); + + if ( cpu_has_xsave ) + fpu_xsave(v); + else + fpu_fxsave(v); +} + +void vcpu_restore_fpu(struct vcpu *v) +{ + ASSERT(!is_idle_vcpu(v)); + ASSERT(!(read_cr0() & X86_CR0_TS)); + + if ( cpu_has_xsave ) + fpu_xrstor(v); + else + fpu_fxrstor(v); + + if ( is_pv_vcpu(v) && (v->arch.pv.ctrlreg[0] & X86_CR0_TS) ) + stts(); +} + /* * Local variables: * mode: C diff --git a/xen/arch/x86/include/asm/domain.h b/xen/arch/x86/include/asm/d= omain.h index 622d22bef255..396e9998d04f 100644 --- a/xen/arch/x86/include/asm/domain.h +++ b/xen/arch/x86/include/asm/domain.h @@ -654,9 +654,6 @@ struct arch_vcpu * it explicitly enables it via xcr0. */ uint64_t xcr0_accum; - /* This variable determines whether nonlazy extended state has been us= ed, - * and thus should be saved/restored. */ - bool nonlazy_xstate_used; =20 /* Restore all FPU state (lazy and non-lazy state) on context switch? = */ bool fully_eager_fpu; diff --git a/xen/arch/x86/include/asm/hvm/svm/nestedsvm.h b/xen/arch/x86/in= clude/asm/hvm/svm/nestedsvm.h index 406fc082b107..2f84546d6b3d 100644 --- a/xen/arch/x86/include/asm/hvm/svm/nestedsvm.h +++ b/xen/arch/x86/include/asm/hvm/svm/nestedsvm.h @@ -40,9 +40,6 @@ struct nestedsvm { /* Shadow io permission map */ unsigned long *ns_iomap; =20 - uint64_t ns_cr0; /* Cached guest_cr[0] of l1 guest while l2 guest runs. - * Needed to handle FPU context switching */ - /* Cache guest cr3/host cr3 the guest sets up for the l2 guest. * Used by Shadow-on-Shadow and Nested-on-Nested. * ns_vmcb_guestcr3: in l2 guest physical address space and points to diff --git a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h b/xen/arch/x86/include= /asm/hvm/vmx/vmcs.h index a7dd2eeffcad..d84195c0426f 100644 --- a/xen/arch/x86/include/asm/hvm/vmx/vmcs.h +++ b/xen/arch/x86/include/asm/hvm/vmx/vmcs.h @@ -141,8 +141,6 @@ struct vmx_vcpu { DECLARE_BITMAP(eoi_exit_bitmap, X86_NR_VECTORS); struct pi_desc pi_desc; =20 - unsigned long host_cr0; - /* Do we need to tolerate a spurious EPT_MISCONFIG VM exit? */ bool ept_spurious_misconfig; =20 diff --git a/xen/arch/x86/include/asm/i387.h b/xen/arch/x86/include/asm/i38= 7.h index a783549db991..bba5dfe3951a 100644 --- a/xen/arch/x86/include/asm/i387.h +++ b/xen/arch/x86/include/asm/i387.h @@ -27,8 +27,7 @@ struct ix87_env { uint16_t fds, _res6; }; =20 -void vcpu_restore_fpu_nonlazy(struct vcpu *v, bool need_stts); -void vcpu_restore_fpu_lazy(struct vcpu *v); +void vcpu_restore_fpu(struct vcpu *v); void vcpu_save_fpu(struct vcpu *v); void save_fpu_enable(void); =20 diff --git a/xen/arch/x86/include/asm/xstate.h b/xen/arch/x86/include/asm/x= state.h index bd767d9cd714..eed40c94efea 100644 --- a/xen/arch/x86/include/asm/xstate.h +++ b/xen/arch/x86/include/asm/xstate.h @@ -32,11 +32,10 @@ extern uint32_t mxcsr_mask; #define XSTATE_FP_SSE (X86_XCR0_FP | X86_XCR0_SSE) #define XCNTXT_MASK (X86_XCR0_FP | X86_XCR0_SSE | X86_XCR0_YMM | \ X86_XCR0_OPMASK | X86_XCR0_ZMM | X86_XCR0_HI_ZMM |= \ - XSTATE_NONLAZY) + X86_XCR0_LWP | X86_XCR0_BNDREGS | X86_XCR0_BNDCSR = | \ + X86_XCR0_PKRU) =20 #define XSTATE_ALL (~(1ULL << 63)) -#define XSTATE_NONLAZY (X86_XCR0_BNDREGS | X86_XCR0_BNDCSR | X86_XCR0_PKRU) -#define XSTATE_LAZY (XSTATE_ALL & ~XSTATE_NONLAZY) #define XSTATE_XSAVES_ONLY 0 #define XSTATE_COMPACTION_ENABLED (1ULL << 63) =20 @@ -121,18 +120,6 @@ static inline uint64_t xgetbv(unsigned int index) return lo | ((uint64_t)hi << 32); } =20 -static inline bool xstate_all(const struct vcpu *v) -{ - /* - * XSTATE_FP_SSE may be excluded, because the offsets of XSTATE_FP_SSE - * (in the legacy region of xsave area) are fixed, so saving - * XSTATE_FP_SSE will not cause overwriting problem with XSAVES/XSAVEC. - */ - return (v->arch.xsave_area->xsave_hdr.xcomp_bv & - XSTATE_COMPACTION_ENABLED) && - (v->arch.xcr0_accum & XSTATE_LAZY & ~XSTATE_FP_SSE); -} - static inline bool __nonnull(1) xsave_area_compressed(const struct xsave_struct *xsave_area) { diff --git a/xen/arch/x86/pv/misc-hypercalls.c b/xen/arch/x86/pv/misc-hyper= calls.c index b529f00ea127..dd783e34b117 100644 --- a/xen/arch/x86/pv/misc-hypercalls.c +++ b/xen/arch/x86/pv/misc-hypercalls.c @@ -40,8 +40,7 @@ long do_fpu_taskswitch(int set) else { v->arch.pv.ctrlreg[0] &=3D ~X86_CR0_TS; - if ( v->fpu_dirtied ) - clts(); + clts(); } =20 return 0; diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c index d554c9d41edd..448a4073f589 100644 --- a/xen/arch/x86/traps.c +++ b/xen/arch/x86/traps.c @@ -1891,15 +1891,17 @@ void asmlinkage do_device_not_available(struct cpu_= user_regs *regs) } =20 #ifdef CONFIG_PV - vcpu_restore_fpu_lazy(curr); + BUG_ON(!(curr->arch.pv.ctrlreg[0] & X86_CR0_TS)); =20 - if ( curr->arch.pv.ctrlreg[0] & X86_CR0_TS ) - { - pv_inject_hw_exception(X86_EXC_NM, X86_EVENT_NO_EC); - curr->arch.pv.ctrlreg[0] &=3D ~X86_CR0_TS; - } - else - TRACE_0D(TRC_PV_MATH_STATE_RESTORE); + /* + * PV ABI QUIRK: Classic Xen kernels (2.6.18 and SLES 11 SP4's + * 3.0) rely on Xen to clear TS. PVOPS kernels (3.0, 3.16 and 4.15 + * are checked) always clear TS themselves. + */ + clts(); + + pv_inject_hw_exception(X86_EXC_NM, X86_EVENT_NO_EC); + curr->arch.pv.ctrlreg[0] &=3D ~X86_CR0_TS; #else ASSERT_UNREACHABLE(); #endif diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c index 92a65bd8d52c..19de2d570220 100644 --- a/xen/arch/x86/xstate.c +++ b/xen/arch/x86/xstate.c @@ -731,7 +731,6 @@ int handle_xsetbv(u32 index, u64 new_bv) { struct vcpu *curr =3D current; uint64_t xcr0_max =3D cpu_policy_xcr0_max(curr->domain->arch.cpuid); - u64 mask; =20 if ( index !=3D XCR_XFEATURE_ENABLED_MASK ) return -EOPNOTSUPP; @@ -766,34 +765,9 @@ int handle_xsetbv(u32 index, u64 new_bv) return -EFAULT; } =20 - mask =3D new_bv & ~curr->arch.xcr0_accum; curr->arch.xcr0 =3D new_bv; curr->arch.xcr0_accum |=3D new_bv; =20 - if ( new_bv & XSTATE_NONLAZY ) - curr->arch.nonlazy_xstate_used =3D 1; - - mask &=3D curr->fpu_dirtied ? ~XSTATE_FP_SSE : XSTATE_NONLAZY; - if ( mask ) - { - unsigned long cr0 =3D read_cr0(); - - clts(); - if ( curr->fpu_dirtied ) - asm ( "stmxcsr %0" : "=3Dm" (curr->arch.xsave_area->fpu_sse.mx= csr) ); - else if ( xstate_all(curr) ) - { - /* See the comment in i387.c:vcpu_restore_fpu_eager(). */ - mask |=3D XSTATE_LAZY; - curr->fpu_initialised =3D 1; - curr->fpu_dirtied =3D 1; - cr0 &=3D ~X86_CR0_TS; - } - xrstor(curr, mask); - if ( cr0 & X86_CR0_TS ) - write_cr0(cr0); - } - return 0; } =20 diff --git a/xen/common/domain.c b/xen/common/domain.c index f6f557499660..450a7cf1814a 100644 --- a/xen/common/domain.c +++ b/xen/common/domain.c @@ -1488,8 +1488,6 @@ int vcpu_reset(struct vcpu *v) clear_bit(v->vcpu_id, d->poll_mask); v->poll_evtchn =3D 0; =20 - v->fpu_initialised =3D 0; - v->fpu_dirtied =3D 0; v->is_initialised =3D 0; if ( v->affinity_broken & VCPU_AFFINITY_OVERRIDE ) vcpu_temporary_affinity(v, NR_CPUS, VCPU_AFFINITY_OVERRIDE); diff --git a/xen/common/efi/runtime.c b/xen/common/efi/runtime.c index d952c3ba785e..273c03d676f8 100644 --- a/xen/common/efi/runtime.c +++ b/xen/common/efi/runtime.c @@ -85,12 +85,14 @@ struct efi_rs_state efi_rs_enter(void) static const u16 fcw =3D FCW_DEFAULT; static const u32 mxcsr =3D MXCSR_DEFAULT; struct efi_rs_state state =3D { .cr3 =3D 0 }; + struct vcpu *curr =3D current; =20 if ( mfn_eq(efi_l4_mfn, INVALID_MFN) ) return state; =20 state.cr3 =3D read_cr3(); - save_fpu_enable(); + if ( !is_idle_vcpu(curr) ) + vcpu_save_fpu(curr); asm volatile ( "fnclex; fldcw %0" :: "m" (fcw) ); asm volatile ( "ldmxcsr %0" :: "m" (mxcsr) ); =20 @@ -101,7 +103,7 @@ struct efi_rs_state efi_rs_enter(void) /* prevent fixup_page_fault() from doing anything */ irq_enter(); =20 - if ( is_pv_vcpu(current) && !is_idle_vcpu(current) ) + if ( is_pv_vcpu(curr) && !is_idle_vcpu(curr) ) { struct desc_ptr gdt_desc =3D { .limit =3D LAST_RESERVED_GDT_BYTE, @@ -153,7 +155,8 @@ void efi_rs_leave(struct efi_rs_state *state) irq_exit(); efi_rs_on_cpu =3D NR_CPUS; spin_unlock(&efi_rs_lock); - vcpu_restore_fpu_nonlazy(curr, true); + if ( !is_idle_vcpu(curr) ) + vcpu_restore_fpu(curr); } =20 bool efi_rs_using_pgtables(void) diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 37f5922f3206..1da52e77dcdf 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -206,10 +206,6 @@ struct vcpu struct guest_area runstate_guest_area; unsigned int new_state; =20 - /* Has the FPU been initialised? */ - bool fpu_initialised; - /* Has the FPU been used since it was last saved? */ - bool fpu_dirtied; /* Initialization completed for this VCPU? */ bool is_initialised; /* Currently running on a CPU? */ --=20 2.42.0