From nobody Tue May 21 21:03:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=reject dis=none) header.from=citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1712227307; cv=none; d=zohomail.com; s=zohoarc; b=EJKIys6fWUQiZMKDJvNL08r/DF8a9s/quRK6AXxlkk+w9wZCM/erI+HSFT9RJ/P09VaYRAKTm117sjOhawW/4NfkaPG4Ry0z/Hm3JmC02t1M0/MZtKC3pKT5k26/A7JThCQFjh5nl8od0+PnKHt6vTei6TBCESJRMJIYzvrUpX0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1712227307; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=b856P1WpNjFZtoUruDU6tKHFRtenQoNRPl7rkOX9bvE=; b=FvgnmjzgNsz3Tji19G7BG8qKUSk5754F2dc50MXy0YMYDo7htA6Eor4yLskP5/SpqKx/wTj9ABYfO/viqlEC2/SGIuwkpoifjn45BZ1ESRH2IOpwo1P7iA2dXuOuLfa5x4OdkKO/c21MraVX9X03QWF5Y9r0ovhOAFZn+fwPfYQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1712227307527981.576492876747; Thu, 4 Apr 2024 03:41:47 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.700804.1094494 (Exim 4.92) (envelope-from ) id 1rsKXA-00015U-MR; Thu, 04 Apr 2024 10:41:28 +0000 Received: by outflank-mailman (output) from mailman id 700804.1094494; Thu, 04 Apr 2024 10:41:28 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rsKXA-00015N-JV; Thu, 04 Apr 2024 10:41:28 +0000 Received: by outflank-mailman (input) for mailman id 700804; Thu, 04 Apr 2024 10:41:27 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1rsKX9-00015H-2b for xen-devel@lists.xenproject.org; Thu, 04 Apr 2024 10:41:27 +0000 Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [2a00:1450:4864:20::12a]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id e2dd6d4c-f26f-11ee-afe5-a90da7624cb6; Thu, 04 Apr 2024 12:41:25 +0200 (CEST) Received: by mail-lf1-x12a.google.com with SMTP id 2adb3069b0e04-516c116f480so982924e87.2 for ; Thu, 04 Apr 2024 03:41:25 -0700 (PDT) Received: from andrewcoop.citrite.net (default-46-102-197-194.interdsl.co.uk. [46.102.197.194]) by smtp.gmail.com with ESMTPSA id eg49-20020a05640228b100b0056e0a5e713bsm1660387edb.93.2024.04.04.03.41.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 03:41:23 -0700 (PDT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: e2dd6d4c-f26f-11ee-afe5-a90da7624cb6 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1712227284; x=1712832084; darn=lists.xenproject.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b856P1WpNjFZtoUruDU6tKHFRtenQoNRPl7rkOX9bvE=; b=c5PgRJxJdGl5licxoClpUSLTRPzmtUk9oXeyYj60qmcfvNVrspKi0rYS2HJcrMJoFq h2NGSXBKyFLxbxKwfSyaR0AQ+ZLVLHeKndRO1SicVEc3sFsyKfLiq1TG5H4BY9SBZPpA zufVRHY83QlOoaNicCT5YwccgXQIDnW81nLi4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712227284; x=1712832084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b856P1WpNjFZtoUruDU6tKHFRtenQoNRPl7rkOX9bvE=; b=EUHvSAcZjtn38AMUfwGMzTj05w8C3/jQsV7QZTfJ/ap1BqeA8ZCDkteRvpt316rH5r fftg175ulg71Sr8ytkckvENdN+MjWj+sRtV+94O2uJ/DZoi1SW00Jww1b48HLhXE5N+g 3DUi+78V23IyA/8UkjRib8vc2RusOWR6HSFvzQRVSY0MA08JeMMhsr6ToFqHX5hxOz0F P6p1cU6UyS78pboGc0SHfE+HbwATP+4W/zavGFPP/8Xp40FDCBv4Odeo9Qu/s4jfSujs 4B8dyIYp3i1hKHPZkhS7CyamDmxa0FDUMokl1QqSYeWxdUpevrJ7v9BBR8rsya5C7PZL 3DxQ== X-Gm-Message-State: AOJu0YzAeUyWEu2LgWwnocrj1tofwHY2/0AJG2kj/lXeo0xmZzibmit2 q3rnRS3Wp53NlJ3Du+g0l/iVRPllrOHHghSFm5t8r/x1noZj9BL4LPwtK153qmn4BcOdv4wPu05 7038= X-Google-Smtp-Source: AGHT+IEdjFRdHcK8RY8SyQS8zUsOxxbxqPy2qUUNJ0ovrzCSvgzIb55ex1BiV5Gj7oGNGgbF58j8Yg== X-Received: by 2002:a05:6512:715:b0:515:8159:788d with SMTP id b21-20020a056512071500b005158159788dmr1311481lfs.64.1712227284327; Thu, 04 Apr 2024 03:41:24 -0700 (PDT) From: Andrew Cooper To: Xen-devel Cc: Andrew Cooper , Jan Beulich , =?UTF-8?q?Roger=20Pau=20Monn=C3=A9?= , =?UTF-8?q?Marek=20Marczykowski-G=C3=B3recki?= Subject: [PATCH] x86/tsx: Cope with RTM_ALWAYS_ABORT vs RTM mismatch Date: Thu, 4 Apr 2024 11:41:22 +0100 Message-Id: <20240404104122.2870129-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @citrix.com) X-ZM-MESSAGEID: 1712227309814100001 It turns out there is something wonky on some but not all CPUs with MSR_TSX_FORCE_ABORT. The presence of RTM_ALWAYS_ABORT causes Xen to think it's safe to offer HLE/RTM to guests, but in this case, XBEGIN instructions genuinely #UD. Spot this case and try to back out as cleanly as we can. Signed-off-by: Andrew Cooper Tested-by: Marek Marczykowski-G=C3=B3recki --- CC: Jan Beulich CC: Roger Pau Monn=C3=A9 CC: Marek Marczykowski-G=C3=B3recki In the meantime, I'll see if anyone at Intel knows what's going on. Because these parts are fully out of support now, it's very unlikely that we're goi= ng to get a fix. --- xen/arch/x86/tsx.c | 55 +++++++++++++++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/xen/arch/x86/tsx.c b/xen/arch/x86/tsx.c index 50d8059f23a9..41bb39d10074 100644 --- a/xen/arch/x86/tsx.c +++ b/xen/arch/x86/tsx.c @@ -1,5 +1,6 @@ #include #include +#include #include =20 /* @@ -9,6 +10,7 @@ * -1 =3D> Default, altered to 0/1 (if unspecified) by: * - TAA heuristics/settings for speculative safety * - "TSX vs PCR3" select for TSX memory ordering safety + * -2 =3D> Implicit tsx=3D0 (from RTM_ALWAYS_ABORT vs RTM mismatch) * -3 =3D> Implicit tsx=3D1 (feed-through from spec-ctrl=3D0) * * This is arranged such that the bottom bit encodes whether TSX is actual= ly @@ -114,11 +116,50 @@ void tsx_init(void) =20 if ( cpu_has_tsx_force_abort ) { + uint64_t val; + /* - * On an early TSX-enable Skylake part subject to the memory + * On an early TSX-enabled Skylake part subject to the memory * ordering erratum, with at least the March 2019 microcode. */ =20 + rdmsrl(MSR_TSX_FORCE_ABORT, val); + + /* + * At the time of writing (April 2024), it was discovered that + * some parts (e.g. CoffeeLake 8th Gen, 06-9e-0a, ucode 0xf6) + * advertise RTM_ALWAYS_ABORT, but XBEGIN instructions #UD. O= ther + * similar parts (e.g. KabyLake Xeon-E3, 06-9e-09, ucode 0xf8) + * operate as expected. + * + * In this case: + * - RTM_ALWAYS_ABORT and MSR_TSX_FORCE_ABORT are enumerated. + * - XBEGIN instructions genuinely #UD. + * - MSR_TSX_FORCE_ABORT is write-discard and fails to hold i= ts + * value. + * - HLE and RTM are not enumerated, despite + * MSR_TSX_FORCE_ABORT.TSX_CPUID_CLEAR being clear. + * + * Spot this case, and treat it as if no TSX is available at a= ll. + * This will prevent Xen from thinking it's safe to offer HLE/= RTM + * to VMs. + */ + if ( val =3D=3D 0 && cpu_has_rtm_always_abort && !cpu_has_rtm ) + { + printk(XENLOG_ERR + "FIRMWARE BUG: CPU %02x-%02x-%02x, ucode 0x%08x: RT= M_ALWAYS_ABORT vs RTM mismatch\n", + boot_cpu_data.x86, boot_cpu_data.x86_model, + boot_cpu_data.x86_mask, this_cpu(cpu_sig).rev); + + setup_clear_cpu_cap(X86_FEATURE_RTM_ALWAYS_ABORT); + setup_clear_cpu_cap(X86_FEATURE_TSX_FORCE_ABORT); + + if ( opt_tsx < 0 ) + opt_tsx =3D -2; + + goto done_setup; + } + /* * Probe for the June 2021 microcode which de-features TSX on * client parts. (Note - this is a subset of parts impacted by @@ -128,15 +169,8 @@ void tsx_init(void) * read as zero if TSX_FORCE_ABORT.ENABLE_RTM has been set bef= ore * we run. */ - if ( !has_rtm_always_abort ) - { - uint64_t val; - - rdmsrl(MSR_TSX_FORCE_ABORT, val); - - if ( val & TSX_ENABLE_RTM ) - has_rtm_always_abort =3D true; - } + if ( val & TSX_ENABLE_RTM ) + has_rtm_always_abort =3D true; =20 /* * If no explicit tsx=3D option is provided, pick a default. @@ -191,6 +225,7 @@ void tsx_init(void) setup_force_cpu_cap(X86_FEATURE_RTM); } } + done_setup: =20 /* * Note: MSR_TSX_CTRL is enumerated on TSX-enabled MDS_NO and later pa= rts. base-commit: 6117179dd99958e4ef2687617d12c9b15bdbae24 --=20 2.30.2