From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B477C433FE for ; Wed, 2 Nov 2022 07:47:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230420AbiKBHr1 (ORCPT ); Wed, 2 Nov 2022 03:47:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230419AbiKBHrX (ORCPT ); Wed, 2 Nov 2022 03:47:23 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F6D225EAA for ; Wed, 2 Nov 2022 00:47:22 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 55A521F8F6; Wed, 2 Nov 2022 07:47:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375241; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2leKddvHcFKOPECovCQoX+YzpPkMWoS3wh7BhleyL14=; b=qFgHZiO+6+gBo4apQivoOgk7d+PDAndeKnWMGSowg6KGs6QioJHppeR7vHkmYpH6KtMdgF cfzprQcBHxXtNLEUIaqOn+EWTQrTuPdKkviAL7c1c2/IGtql0fIxihe8NlKNOAHv5ul4aP J/cF3nKW98YaLLZ0NFGUupSp9zkkHtM= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1C4171376E; Wed, 2 Nov 2022 07:47:21 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id zhumBYkgYmOLcgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:21 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 01/16] x86/mtrr: add comment for set_mtrr_state() serialization Date: Wed, 2 Nov 2022 08:46:58 +0100 Message-Id: <20221102074713.21493-2-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a comment how set_mtrr_state() is needing serialization. Note that this patch has already been applied to tip.git. Suggested-by: Borislav Petkov Signed-off-by: Juergen Gross --- V3: - new patch instead of old patch 1 --- arch/x86/kernel/cpu/mtrr/generic.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index 558108296f3c..cd64eab02393 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -684,7 +684,10 @@ static u32 deftype_lo, deftype_hi; /** * set_mtrr_state - Set the MTRR state for this CPU. * - * NOTE: The CPU must already be in a safe state for MTRR changes. + * NOTE: The CPU must already be in a safe state for MTRR changes, includi= ng + * measures that only a single CPU can be active in set_mtrr_state()= in + * order to not be subject to races for usage of deftype_lo (this is + * accomplished by taking set_atomicity_lock). * RETURNS: 0 if no changes made, else a mask indicating what was changed. */ static unsigned long set_mtrr_state(void) --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3D60C433FE for ; Wed, 2 Nov 2022 07:47:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230451AbiKBHrg (ORCPT ); Wed, 2 Nov 2022 03:47:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230424AbiKBHra (ORCPT ); Wed, 2 Nov 2022 03:47:30 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6301B25EA2 for ; Wed, 2 Nov 2022 00:47:28 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 07CF61F8B0; Wed, 2 Nov 2022 07:47:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375247; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GQEtb2vL+iQO4o3v4gnSZ8HiVeg2+FFF5GuRa10reVo=; b=XcAg0758gjmv84xLkZT984Qx7dgoXbDg2VgwRDexHARVFS/H9a7bejr26a2+5BXSudfixa TGFD3ycnVDrW8J1yLkpHFUxcBiV1J0fryFHTuP+AMoiMCwWXAJEPM8Ni2kt3zvJIY5AZqG PjWf0m9dqJ1FFqOf9Z1FczZkNAX5M7s= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BB5C41376E; Wed, 2 Nov 2022 07:47:26 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id sI5FLI4gYmOccgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:26 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 02/16] x86/mtrr: remove unused cyrix_set_all() function Date: Wed, 2 Nov 2022 08:46:59 +0100 Message-Id: <20221102074713.21493-3-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The Cyrix CPU specific MTRR function cyrix_set_all() will never be called, as the struct mtrr_ops set_all() callback will only be called in the use_intel() case, which would require the use_intel_if member of struct mtrr_ops to be set, which isn't the case for Cyrix. Note that this patch has already been applied to tip.git. Signed-off-by: Juergen Gross --- V2: - new patch --- arch/x86/kernel/cpu/mtrr/cyrix.c | 34 -------------------------------- 1 file changed, 34 deletions(-) diff --git a/arch/x86/kernel/cpu/mtrr/cyrix.c b/arch/x86/kernel/cpu/mtrr/cy= rix.c index ca670919b561..c77d3b0a5bf2 100644 --- a/arch/x86/kernel/cpu/mtrr/cyrix.c +++ b/arch/x86/kernel/cpu/mtrr/cyrix.c @@ -234,42 +234,8 @@ static void cyrix_set_arr(unsigned int reg, unsigned l= ong base, post_set(); } =20 -typedef struct { - unsigned long base; - unsigned long size; - mtrr_type type; -} arr_state_t; - -static arr_state_t arr_state[8] =3D { - {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL}, - {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL}, {0UL, 0UL, 0UL} -}; - -static unsigned char ccr_state[7] =3D { 0, 0, 0, 0, 0, 0, 0 }; - -static void cyrix_set_all(void) -{ - int i; - - prepare_set(); - - /* the CCRs are not contiguous */ - for (i =3D 0; i < 4; i++) - setCx86(CX86_CCR0 + i, ccr_state[i]); - for (; i < 7; i++) - setCx86(CX86_CCR4 + i, ccr_state[i]); - - for (i =3D 0; i < 8; i++) { - cyrix_set_arr(i, arr_state[i].base, - arr_state[i].size, arr_state[i].type); - } - - post_set(); -} - static const struct mtrr_ops cyrix_mtrr_ops =3D { .vendor =3D X86_VENDOR_CYRIX, - .set_all =3D cyrix_set_all, .set =3D cyrix_set_arr, .get =3D cyrix_get_arr, .get_free_region =3D cyrix_get_free_region, --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5F61C433FE for ; Wed, 2 Nov 2022 07:47:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230458AbiKBHrk (ORCPT ); Wed, 2 Nov 2022 03:47:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229547AbiKBHrf (ORCPT ); Wed, 2 Nov 2022 03:47:35 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC43625E8B for ; Wed, 2 Nov 2022 00:47:33 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id A41383373A; Wed, 2 Nov 2022 07:47:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375252; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z6gy2tyRJJ+OFyVFudtbf06b5xT5AM3xfmnxx6MsckU=; b=kB6M6YcMniF4XqArkgMZkLvV3UMLrbzfiFvOgWVIyGfmpeJvp3hRkiAbDedxmFIYgF2q2p ghgQo5Nl3tDpFd6AX7/hovwGDFmDXsurNIaH3Gpfy5HTKatdQV6wSsvr6LK7p9Kpc1rom/ pKmdEsxvGDdJFcqRXP2Yz5Zc71NdyBo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 6B9761376E; Wed, 2 Nov 2022 07:47:32 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id T0XlGJQgYmOmcgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:32 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 03/16] x86/mtrr: replace use_intel() with a local flag Date: Wed, 2 Nov 2022 08:47:00 +0100 Message-Id: <20221102074713.21493-4-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In MTRR code use_intel() is only used in one source file, and the relevant use_intel_if member of struct mtrr_ops is set only in generic_mtrr_ops. Replace use_intel() with a single flag in cacheinfo.c, which can be set when assigning generic_mtrr_ops to mtrr_if. This allows to drop use_intel_if from mtrr_ops, while preparing to support PAT without MTRR. As another preparation for the PAT/MTRR decoupling use a bit for MTRR control and one for PAT control. For now set both bits together, this can be changed later. As the new flag will be set only if mtrr_enabled is set, the test for mtrr_enabled can be dropped at some places. Signed-off-by: Juergen Gross --- V2: - new patch V4: - rename cache_generic to memory_caching_control (Borislav Petkov) - rename CACHE_GENERIC_* to CACHE_* (Borislav Petkov) - get rid of use_generic in mtrr_bp_init() (Borislav Petkov) V5: - keep mtrr_enabled() (Borislav Petkov) --- arch/x86/include/asm/cacheinfo.h | 5 +++++ arch/x86/kernel/cpu/cacheinfo.c | 3 +++ arch/x86/kernel/cpu/mtrr/generic.c | 1 - arch/x86/kernel/cpu/mtrr/mtrr.c | 28 +++++++++++++--------------- arch/x86/kernel/cpu/mtrr/mtrr.h | 2 -- 5 files changed, 21 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index 86b2e0dcc4bf..c3873962a7cd 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -2,6 +2,11 @@ #ifndef _ASM_X86_CACHEINFO_H #define _ASM_X86_CACHEINFO_H =20 +/* Kernel controls MTRR and/or PAT MSRs. */ +extern unsigned int memory_caching_control; +#define CACHE_MTRR 0x01 +#define CACHE_PAT 0x02 + void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu); void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, int cpu); =20 diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 66556833d7af..5228fb9a3798 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -35,6 +35,9 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_llc_shared_= map); /* Shared L2 cache maps */ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_l2c_shared_map); =20 +/* Kernel controls MTRR and/or PAT MSRs. */ +unsigned int memory_caching_control __ro_after_init; + struct _cache_table { unsigned char descriptor; char cache_type; diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index cd64eab02393..81742870ecc5 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -917,7 +917,6 @@ int positive_have_wrcomb(void) * Generic structure... */ const struct mtrr_ops generic_mtrr_ops =3D { - .use_intel_if =3D 1, .set_all =3D generic_set_all, .get =3D generic_get_mtrr, .get_free_region =3D generic_get_free_region, diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index 2746cac9d8a9..4209945c4e68 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -46,6 +46,7 @@ #include #include =20 +#include #include #include #include @@ -119,11 +120,11 @@ static int have_wrcomb(void) } =20 /* This function returns the number of variable MTRRs */ -static void __init set_num_var_ranges(void) +static void __init set_num_var_ranges(bool use_generic) { unsigned long config =3D 0, dummy; =20 - if (use_intel()) + if (use_generic) rdmsr(MSR_MTRRcap, config, dummy); else if (is_cpu(AMD) || is_cpu(HYGON)) config =3D 2; @@ -756,14 +757,16 @@ void __init mtrr_bp_init(void) =20 if (mtrr_if) { __mtrr_enabled =3D true; - set_num_var_ranges(); + set_num_var_ranges(mtrr_if =3D=3D &generic_mtrr_ops); init_table(); - if (use_intel()) { + if (mtrr_if =3D=3D &generic_mtrr_ops) { /* BIOS may override */ __mtrr_enabled =3D get_mtrr_state(); =20 - if (mtrr_enabled()) + if (mtrr_enabled()) { mtrr_bp_pat_init(); + memory_caching_control |=3D CACHE_MTRR | CACHE_PAT; + } =20 if (mtrr_cleanup(phys_addr)) { changed_by_mtrr_cleanup =3D 1; @@ -786,10 +789,7 @@ void __init mtrr_bp_init(void) =20 void mtrr_ap_init(void) { - if (!mtrr_enabled()) - return; - - if (!use_intel() || mtrr_aps_delayed_init) + if (!memory_caching_control || mtrr_aps_delayed_init) return; =20 /* @@ -825,9 +825,7 @@ void mtrr_save_state(void) =20 void set_mtrr_aps_delayed_init(void) { - if (!mtrr_enabled()) - return; - if (!use_intel()) + if (!memory_caching_control) return; =20 mtrr_aps_delayed_init =3D true; @@ -838,7 +836,7 @@ void set_mtrr_aps_delayed_init(void) */ void mtrr_aps_init(void) { - if (!use_intel() || !mtrr_enabled()) + if (!memory_caching_control) return; =20 /* @@ -855,7 +853,7 @@ void mtrr_aps_init(void) =20 void mtrr_bp_restore(void) { - if (!use_intel() || !mtrr_enabled()) + if (!memory_caching_control) return; =20 mtrr_if->set_all(); @@ -866,7 +864,7 @@ static int __init mtrr_init_finialize(void) if (!mtrr_enabled()) return 0; =20 - if (use_intel()) { + if (memory_caching_control & CACHE_MTRR) { if (!changed_by_mtrr_cleanup) mtrr_state_warn(); return 0; diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.h b/arch/x86/kernel/cpu/mtrr/mtr= r.h index 2ac99e561181..88b1c4b6174a 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.h +++ b/arch/x86/kernel/cpu/mtrr/mtrr.h @@ -14,7 +14,6 @@ extern unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; =20 struct mtrr_ops { u32 vendor; - u32 use_intel_if; void (*set)(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type); void (*set_all)(void); @@ -61,7 +60,6 @@ extern u64 size_or_mask, size_and_mask; extern const struct mtrr_ops *mtrr_if; =20 #define is_cpu(vnd) (mtrr_if && mtrr_if->vendor =3D=3D X86_VENDOR_##vnd) -#define use_intel() (mtrr_if && mtrr_if->use_intel_if =3D=3D 1) =20 extern unsigned int num_var_ranges; extern u64 mtrr_tom2; --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B24CC4332F for ; Wed, 2 Nov 2022 07:47:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230450AbiKBHrp (ORCPT ); Wed, 2 Nov 2022 03:47:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229534AbiKBHrk (ORCPT ); Wed, 2 Nov 2022 03:47:40 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B93DB26AE7 for ; Wed, 2 Nov 2022 00:47:39 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5050B338F3; Wed, 2 Nov 2022 07:47:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375258; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gjmwwu1nE4qkYJE0/hC2MZUBh8G9fijNxWNwmPmBasU=; b=Loy3jBzVbq4Y8fth8Cvtw/EA441/lsaVmGenyW7/NEybkX1pS6vP7gtya6fQ6KHWA4PhpK wbeaEPqMMccegCd1zFzdo8fKmErtB89wKkH5+KPeMIJarCCpYI2XDYXS7st9zG+2aJzQm8 tCz5nHYwSsnQ78kGr7Wk9tDLkIr++Eo= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 13A0B1376E; Wed, 2 Nov 2022 07:47:38 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id qoR6A5ogYmOucgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:38 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 04/16] x86/mtrr: rename prepare_set() and post_set() Date: Wed, 2 Nov 2022 08:47:01 +0100 Message-Id: <20221102074713.21493-5-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Rename the currently MTRR specific functions prepare_set() and post_set() for preparing to move them. Make them non-static and put their prototypes into cacheinfo.h, where they will end after moving them to their final position anyway. Expand the comment before the functions with an introductory line and rename two related static variables, too. Signed-off-by: Juergen Gross --- V4: - carved out from other patch (Borislav Petkov) --- arch/x86/include/asm/cacheinfo.h | 3 +++ arch/x86/kernel/cpu/mtrr/generic.c | 43 +++++++++++++++--------------- 2 files changed, 24 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index c3873962a7cd..6159874b4183 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -10,4 +10,7 @@ extern unsigned int memory_caching_control; void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu); void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, int cpu); =20 +void cache_disable(void); +void cache_enable(void); + #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index 81742870ecc5..aebdc90a2489 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -10,6 +10,7 @@ #include =20 #include +#include #include #include #include @@ -396,9 +397,6 @@ print_fixed(unsigned base, unsigned step, const mtrr_ty= pe *types) } } =20 -static void prepare_set(void); -static void post_set(void); - static void __init print_mtrr_state(void) { unsigned int i; @@ -450,11 +448,11 @@ void __init mtrr_bp_pat_init(void) unsigned long flags; =20 local_irq_save(flags); - prepare_set(); + cache_disable(); =20 pat_init(); =20 - post_set(); + cache_enable(); local_irq_restore(flags); } =20 @@ -687,7 +685,7 @@ static u32 deftype_lo, deftype_hi; * NOTE: The CPU must already be in a safe state for MTRR changes, includi= ng * measures that only a single CPU can be active in set_mtrr_state()= in * order to not be subject to races for usage of deftype_lo (this is - * accomplished by taking set_atomicity_lock). + * accomplished by taking cache_disable_lock). * RETURNS: 0 if no changes made, else a mask indicating what was changed. */ static unsigned long set_mtrr_state(void) @@ -718,18 +716,19 @@ static unsigned long set_mtrr_state(void) return change_mask; } =20 - -static unsigned long cr4; -static DEFINE_RAW_SPINLOCK(set_atomicity_lock); - /* + * Disable and enable caches. Needed for changing MTRRs and the PAT MSR. + * * Since we are disabling the cache don't allow any interrupts, * they would run extremely slow and would only increase the pain. * * The caller must ensure that local interrupts are disabled and - * are reenabled after post_set() has been called. + * are reenabled after cache_enable() has been called. */ -static void prepare_set(void) __acquires(set_atomicity_lock) +static unsigned long saved_cr4; +static DEFINE_RAW_SPINLOCK(cache_disable_lock); + +void cache_disable(void) __acquires(cache_disable_lock) { unsigned long cr0; =20 @@ -740,7 +739,7 @@ static void prepare_set(void) __acquires(set_atomicity_= lock) * changes to the way the kernel boots */ =20 - raw_spin_lock(&set_atomicity_lock); + raw_spin_lock(&cache_disable_lock); =20 /* Enter the no-fill (CD=3D1, NW=3D0) cache mode and flush caches. */ cr0 =3D read_cr0() | X86_CR0_CD; @@ -757,8 +756,8 @@ static void prepare_set(void) __acquires(set_atomicity_= lock) =20 /* Save value of CR4 and clear Page Global Enable (bit 7) */ if (boot_cpu_has(X86_FEATURE_PGE)) { - cr4 =3D __read_cr4(); - __write_cr4(cr4 & ~X86_CR4_PGE); + saved_cr4 =3D __read_cr4(); + __write_cr4(saved_cr4 & ~X86_CR4_PGE); } =20 /* Flush all TLBs via a mov %cr3, %reg; mov %reg, %cr3 */ @@ -776,7 +775,7 @@ static void prepare_set(void) __acquires(set_atomicity_= lock) wbinvd(); } =20 -static void post_set(void) __releases(set_atomicity_lock) +void cache_enable(void) __releases(cache_disable_lock) { /* Flush TLBs (no need to flush caches - they are disabled) */ count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); @@ -790,8 +789,8 @@ static void post_set(void) __releases(set_atomicity_loc= k) =20 /* Restore value of CR4 */ if (boot_cpu_has(X86_FEATURE_PGE)) - __write_cr4(cr4); - raw_spin_unlock(&set_atomicity_lock); + __write_cr4(saved_cr4); + raw_spin_unlock(&cache_disable_lock); } =20 static void generic_set_all(void) @@ -800,7 +799,7 @@ static void generic_set_all(void) unsigned long flags; =20 local_irq_save(flags); - prepare_set(); + cache_disable(); =20 /* Actually set the state */ mask =3D set_mtrr_state(); @@ -808,7 +807,7 @@ static void generic_set_all(void) /* also set PAT */ pat_init(); =20 - post_set(); + cache_enable(); local_irq_restore(flags); =20 /* Use the atomic bitops to update the global mask */ @@ -839,7 +838,7 @@ static void generic_set_mtrr(unsigned int reg, unsigned= long base, vr =3D &mtrr_state.var_ranges[reg]; =20 local_irq_save(flags); - prepare_set(); + cache_disable(); =20 if (size =3D=3D 0) { /* @@ -858,7 +857,7 @@ static void generic_set_mtrr(unsigned int reg, unsigned= long base, mtrr_wrmsr(MTRRphysMask_MSR(reg), vr->mask_lo, vr->mask_hi); } =20 - post_set(); + cache_enable(); local_irq_restore(flags); } =20 --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2AD91C433FE for ; Wed, 2 Nov 2022 07:48:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230456AbiKBHsF (ORCPT ); Wed, 2 Nov 2022 03:48:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230468AbiKBHr5 (ORCPT ); Wed, 2 Nov 2022 03:47:57 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 41F2825EA8 for ; Wed, 2 Nov 2022 00:47:45 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id EA2E43373A; Wed, 2 Nov 2022 07:47:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375263; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Wg1ltP5S2ST2D2shywYpbJwrHqpg2nHmgpqKBsh6EzQ=; b=MPB5tr1NRV+9NntC9d7piW075UGauYq/F0wIwl3hrWXHrOLpkt91GvuDjQ/bqOTrZjDpEU QX1BhBHf7jqkHc6qAYj9wTU4zVgjLARKDa3oAWOcyUM9u1SYh2gdoCx+sKKvTRrgd+a9qB owZ9Lyo3qx/EzoklDjpwBt/Ya9F0nZs= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B225A1376E; Wed, 2 Nov 2022 07:47:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id Zk5BKp8gYmO9cgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:43 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 05/16] x86/mtrr: split MTRR specific handling from cache dis/enabling Date: Wed, 2 Nov 2022 08:47:02 +0100 Message-Id: <20221102074713.21493-6-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Split the MTRR specific actions from cache_disable() and cache_enable() into the new functions mtrr_disable() and mtrr_enable(). Signed-off-by: Juergen Gross --- V4: - carved out from other patch (Borislav Petkov) V5: - use cpu_feature_enabled() (Borislav Petkov) --- arch/x86/include/asm/mtrr.h | 4 ++++ arch/x86/kernel/cpu/mtrr/generic.c | 26 +++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index 76d726074c16..12a16caed395 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -48,6 +48,8 @@ extern void mtrr_aps_init(void); extern void mtrr_bp_restore(void); extern int mtrr_trim_uncached_memory(unsigned long end_pfn); extern int amd_special_default_mtrr(void); +void mtrr_disable(void); +void mtrr_enable(void); # else static inline u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform) { @@ -87,6 +89,8 @@ static inline void mtrr_centaur_report_mcr(int mcr, u32 l= o, u32 hi) #define set_mtrr_aps_delayed_init() do {} while (0) #define mtrr_aps_init() do {} while (0) #define mtrr_bp_restore() do {} while (0) +#define mtrr_disable() do {} while (0) +#define mtrr_enable() do {} while (0) # endif =20 #ifdef CONFIG_COMPAT diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index aebdc90a2489..4edf0827f7ee 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -716,6 +716,21 @@ static unsigned long set_mtrr_state(void) return change_mask; } =20 +void mtrr_disable(void) +{ + /* Save MTRR state */ + rdmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); + + /* Disable MTRRs, and set the default type to uncached */ + mtrr_wrmsr(MSR_MTRRdefType, deftype_lo & ~0xcff, deftype_hi); +} + +void mtrr_enable(void) +{ + /* Intel (P6) standard MTRRs */ + mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); +} + /* * Disable and enable caches. Needed for changing MTRRs and the PAT MSR. * @@ -764,11 +779,8 @@ void cache_disable(void) __acquires(cache_disable_lock) count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); flush_tlb_local(); =20 - /* Save MTRR state */ - rdmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); - - /* Disable MTRRs, and set the default type to uncached */ - mtrr_wrmsr(MSR_MTRRdefType, deftype_lo & ~0xcff, deftype_hi); + if (cpu_feature_enabled(X86_FEATURE_MTRR)) + mtrr_disable(); =20 /* Again, only flush caches if we have to. */ if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) @@ -781,8 +793,8 @@ void cache_enable(void) __releases(cache_disable_lock) count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); flush_tlb_local(); =20 - /* Intel (P6) standard MTRRs */ - mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); + if (cpu_feature_enabled(X86_FEATURE_MTRR)) + mtrr_enable(); =20 /* Enable caches */ write_cr0(read_cr0() & ~X86_CR0_CD); --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FD82C4332F for ; Wed, 2 Nov 2022 07:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230487AbiKBHsI (ORCPT ); Wed, 2 Nov 2022 03:48:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230483AbiKBHsA (ORCPT ); Wed, 2 Nov 2022 03:48:00 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB63F27170 for ; Wed, 2 Nov 2022 00:47:50 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id A5D971F8B0; Wed, 2 Nov 2022 07:47:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375269; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NhfPGIkSQGMzUIyLCC2u9IISPCoq9TUFQ1/WyCEAvPA=; b=aUuF0bLWXXy0fVLbPkMPgxknmGre45Rj0LWrTujtDYRMlIC5iGoyJdB3SMfRKClXzIeZfp O1NeIAfRF5gbasriCALR3LHJTSP3SX0ciNr9ETvPMZZTFtlhzMaEsVqFzKj4arMyWIJ8CV QyRqHfboyRR3bQyJmJ/+ivlIvvFribE= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 632D11376E; Wed, 2 Nov 2022 07:47:49 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aPWnFqUgYmPPcgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:49 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 06/16] x86: move some code out of arch/x86/kernel/cpu/mtrr Date: Wed, 2 Nov 2022 08:47:03 +0100 Message-Id: <20221102074713.21493-7-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Prepare making PAT and MTRR support independent from each other by moving some code needed by both out of the MTRR specific sources. Signed-off-by: Juergen Gross --- V2: - move code from cpu/common.c to cpu/cacheinfo.c (Borislav Petkov) V4: - carved out all non-movement modifications (Borislav Petkov) --- arch/x86/kernel/cpu/cacheinfo.c | 77 ++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/mtrr/generic.c | 74 ---------------------------- 2 files changed, 77 insertions(+), 74 deletions(-) diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 5228fb9a3798..c6a17e21301e 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -20,6 +20,8 @@ #include #include #include +#include +#include =20 #include "cpu.h" =20 @@ -1043,3 +1045,78 @@ int populate_cache_leaves(unsigned int cpu) =20 return 0; } + +/* + * Disable and enable caches. Needed for changing MTRRs and the PAT MSR. + * + * Since we are disabling the cache don't allow any interrupts, + * they would run extremely slow and would only increase the pain. + * + * The caller must ensure that local interrupts are disabled and + * are reenabled after cache_enable() has been called. + */ +static unsigned long saved_cr4; +static DEFINE_RAW_SPINLOCK(cache_disable_lock); + +void cache_disable(void) __acquires(cache_disable_lock) +{ + unsigned long cr0; + + /* + * Note that this is not ideal + * since the cache is only flushed/disabled for this CPU while the + * MTRRs are changed, but changing this requires more invasive + * changes to the way the kernel boots + */ + + raw_spin_lock(&cache_disable_lock); + + /* Enter the no-fill (CD=3D1, NW=3D0) cache mode and flush caches. */ + cr0 =3D read_cr0() | X86_CR0_CD; + write_cr0(cr0); + + /* + * Cache flushing is the most time-consuming step when programming + * the MTRRs. Fortunately, as per the Intel Software Development + * Manual, we can skip it if the processor supports cache self- + * snooping. + */ + if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) + wbinvd(); + + /* Save value of CR4 and clear Page Global Enable (bit 7) */ + if (cpu_feature_enabled(X86_FEATURE_PGE)) { + saved_cr4 =3D __read_cr4(); + __write_cr4(saved_cr4 & ~X86_CR4_PGE); + } + + /* Flush all TLBs via a mov %cr3, %reg; mov %reg, %cr3 */ + count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); + flush_tlb_local(); + + if (cpu_feature_enabled(X86_FEATURE_MTRR)) + mtrr_disable(); + + /* Again, only flush caches if we have to. */ + if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) + wbinvd(); +} + +void cache_enable(void) __releases(cache_disable_lock) +{ + /* Flush TLBs (no need to flush caches - they are disabled) */ + count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); + flush_tlb_local(); + + if (cpu_feature_enabled(X86_FEATURE_MTRR)) + mtrr_enable(); + + /* Enable caches */ + write_cr0(read_cr0() & ~X86_CR0_CD); + + /* Restore value of CR4 */ + if (cpu_feature_enabled(X86_FEATURE_PGE)) + __write_cr4(saved_cr4); + + raw_spin_unlock(&cache_disable_lock); +} diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index 4edf0827f7ee..bfe13eedaca8 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -731,80 +731,6 @@ void mtrr_enable(void) mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); } =20 -/* - * Disable and enable caches. Needed for changing MTRRs and the PAT MSR. - * - * Since we are disabling the cache don't allow any interrupts, - * they would run extremely slow and would only increase the pain. - * - * The caller must ensure that local interrupts are disabled and - * are reenabled after cache_enable() has been called. - */ -static unsigned long saved_cr4; -static DEFINE_RAW_SPINLOCK(cache_disable_lock); - -void cache_disable(void) __acquires(cache_disable_lock) -{ - unsigned long cr0; - - /* - * Note that this is not ideal - * since the cache is only flushed/disabled for this CPU while the - * MTRRs are changed, but changing this requires more invasive - * changes to the way the kernel boots - */ - - raw_spin_lock(&cache_disable_lock); - - /* Enter the no-fill (CD=3D1, NW=3D0) cache mode and flush caches. */ - cr0 =3D read_cr0() | X86_CR0_CD; - write_cr0(cr0); - - /* - * Cache flushing is the most time-consuming step when programming - * the MTRRs. Fortunately, as per the Intel Software Development - * Manual, we can skip it if the processor supports cache self- - * snooping. - */ - if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) - wbinvd(); - - /* Save value of CR4 and clear Page Global Enable (bit 7) */ - if (boot_cpu_has(X86_FEATURE_PGE)) { - saved_cr4 =3D __read_cr4(); - __write_cr4(saved_cr4 & ~X86_CR4_PGE); - } - - /* Flush all TLBs via a mov %cr3, %reg; mov %reg, %cr3 */ - count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); - flush_tlb_local(); - - if (cpu_feature_enabled(X86_FEATURE_MTRR)) - mtrr_disable(); - - /* Again, only flush caches if we have to. */ - if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) - wbinvd(); -} - -void cache_enable(void) __releases(cache_disable_lock) -{ - /* Flush TLBs (no need to flush caches - they are disabled) */ - count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); - flush_tlb_local(); - - if (cpu_feature_enabled(X86_FEATURE_MTRR)) - mtrr_enable(); - - /* Enable caches */ - write_cr0(read_cr0() & ~X86_CR0_CD); - - /* Restore value of CR4 */ - if (boot_cpu_has(X86_FEATURE_PGE)) - __write_cr4(saved_cr4); - raw_spin_unlock(&cache_disable_lock); -} - static void generic_set_all(void) { unsigned long mask, count; --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C11C4332F for ; Wed, 2 Nov 2022 07:48:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230474AbiKBHsQ (ORCPT ); Wed, 2 Nov 2022 03:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230467AbiKBHsB (ORCPT ); Wed, 2 Nov 2022 03:48:01 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C0ED25EA5 for ; Wed, 2 Nov 2022 00:47:56 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5B1923373A; Wed, 2 Nov 2022 07:47:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375275; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hfFxlGohZFf4CqpiCTKsxCFD+Xx0kKFUH3zzu8okq48=; b=nlpM0eoMXQa+ifbQPXqCnafiiovJrI9moEH/jBkqh1HjGfdpeGr28LiYWG70VhmAB2MiDu Ggyg4U4i/o8rU4Diuq2BXGJIA6kKEUweTCGXPH0MpEhTuEA1d8f20ljR8pmgxbHF9aCUiP VhOyq8U/FvwdZR524s9c7Ge3o/C25yA= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1F2691376E; Wed, 2 Nov 2022 07:47:55 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 9Zc9BqsgYmPacgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:47:55 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 07/16] x86/mtrr: Disentangle MTRR init from PAT init. Date: Wed, 2 Nov 2022 08:47:04 +0100 Message-Id: <20221102074713.21493-8-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a main cache_cpu_init() init routine which initializes MTRR and/or PAT support depending on what has been detected on the system. Leave the MTRR-specific initialization in a MTRR-specific init function where the smp_changes_mask setting happens now with caches disabled. This global mask update was done with caches enabled before probably because atomic operations while running uncached might have been quite expensive. But since only systems with a broken BIOS should ever require to set any bit in smp_changes_mask, hurting those devices with a penalty of a few microseconds during boot shouldn't be a real issue. Signed-off-by: Juergen Gross --- V2: - new patch V4: - remove some comments (Borislav Petkov) V5: - rephrase commit message (Borislav Petkov) --- arch/x86/include/asm/cacheinfo.h | 1 + arch/x86/include/asm/mtrr.h | 2 ++ arch/x86/kernel/cpu/cacheinfo.c | 17 +++++++++++++++++ arch/x86/kernel/cpu/mtrr/generic.c | 15 ++------------- 4 files changed, 22 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index 6159874b4183..978bac70fd49 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -12,5 +12,6 @@ void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, i= nt cpu); =20 void cache_disable(void); void cache_enable(void); +void cache_cpu_init(void); =20 #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index 12a16caed395..986249a2b9b6 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -50,6 +50,7 @@ extern int mtrr_trim_uncached_memory(unsigned long end_pf= n); extern int amd_special_default_mtrr(void); void mtrr_disable(void); void mtrr_enable(void); +void mtrr_generic_set_state(void); # else static inline u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform) { @@ -91,6 +92,7 @@ static inline void mtrr_centaur_report_mcr(int mcr, u32 l= o, u32 hi) #define mtrr_bp_restore() do {} while (0) #define mtrr_disable() do {} while (0) #define mtrr_enable() do {} while (0) +#define mtrr_generic_set_state() do {} while (0) # endif =20 #ifdef CONFIG_COMPAT diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index c6a17e21301e..81ab99fe92bd 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -1120,3 +1120,20 @@ void cache_enable(void) __releases(cache_disable_loc= k) =20 raw_spin_unlock(&cache_disable_lock); } + +void cache_cpu_init(void) +{ + unsigned long flags; + + local_irq_save(flags); + cache_disable(); + + if (memory_caching_control & CACHE_MTRR) + mtrr_generic_set_state(); + + if (memory_caching_control & CACHE_PAT) + pat_init(); + + cache_enable(); + local_irq_restore(flags); +} diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index bfe13eedaca8..32aebed25e3f 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -731,30 +731,19 @@ void mtrr_enable(void) mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); } =20 -static void generic_set_all(void) +void mtrr_generic_set_state(void) { unsigned long mask, count; - unsigned long flags; - - local_irq_save(flags); - cache_disable(); =20 /* Actually set the state */ mask =3D set_mtrr_state(); =20 - /* also set PAT */ - pat_init(); - - cache_enable(); - local_irq_restore(flags); - /* Use the atomic bitops to update the global mask */ for (count =3D 0; count < sizeof(mask) * 8; ++count) { if (mask & 0x01) set_bit(count, &smp_changes_mask); mask >>=3D 1; } - } =20 /** @@ -854,7 +843,7 @@ int positive_have_wrcomb(void) * Generic structure... */ const struct mtrr_ops generic_mtrr_ops =3D { - .set_all =3D generic_set_all, + .set_all =3D cache_cpu_init, .get =3D generic_get_mtrr, .get_free_region =3D generic_get_free_region, .set =3D generic_set_mtrr, --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 39BEBC4332F for ; Wed, 2 Nov 2022 07:48:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230478AbiKBHs0 (ORCPT ); Wed, 2 Nov 2022 03:48:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230437AbiKBHsF (ORCPT ); Wed, 2 Nov 2022 03:48:05 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C38625EA8 for ; Wed, 2 Nov 2022 00:48:02 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 07B191F8F6; Wed, 2 Nov 2022 07:48:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375281; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jEdYANPBbo11yEPhclA11hPNGtJyAv+5gydUG+2wLoA=; b=HNhwEaizrMBram9B1DMCvJi1mShlOtPjN9L3JRMrTMezKPmg9gEri+ki/ZVoBZ+2c1B7Cg fEFmNmizwcFlmkkfoTYL0rYYi3ZA4ZC1U6Diyvr8jyHZmhvwV3FiqS+fWX8XojEihjGhjy rIOuhc+lCXpgsb1jyFU3vsl3BxuG9/Y= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id BC24E1376E; Wed, 2 Nov 2022 07:48:00 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id PdaVLLAgYmPmcgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:00 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 08/16] x86/mtrr: remove set_all callback from struct mtrr_ops Date: Wed, 2 Nov 2022 08:47:05 +0100 Message-Id: <20221102074713.21493-9-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Instead of using an indirect call to mtrr_if->set_all just call the only possible target cache_cpu_init() directly. This enables to remove the set_all callback from struct mtrr_ops. Signed-off-by: Juergen Gross --- arch/x86/kernel/cpu/mtrr/generic.c | 1 - arch/x86/kernel/cpu/mtrr/mtrr.c | 10 +++++----- arch/x86/kernel/cpu/mtrr/mtrr.h | 2 -- 3 files changed, 5 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index 32aebed25e3f..af8422c96b92 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -843,7 +843,6 @@ int positive_have_wrcomb(void) * Generic structure... */ const struct mtrr_ops generic_mtrr_ops =3D { - .set_all =3D cache_cpu_init, .get =3D generic_get_mtrr, .get_free_region =3D generic_get_free_region, .set =3D generic_set_mtrr, diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index 4209945c4e68..a44b510ced0e 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -170,15 +170,15 @@ static int mtrr_rendezvous_handler(void *info) * saved, and we want to replicate that across all the cpus that come * online (either at the end of boot or resume or during a runtime cpu * online). If we're doing that, @reg is set to something special and on - * all the cpu's we do mtrr_if->set_all() (On the logical cpu that + * all the CPUs we do cache_cpu_init() (On the logical CPU that * started the boot/resume sequence, this might be a duplicate - * set_all()). + * cache_cpu_init()). */ if (data->smp_reg !=3D ~0U) { mtrr_if->set(data->smp_reg, data->smp_base, data->smp_size, data->smp_type); } else if (mtrr_aps_delayed_init || !cpu_online(smp_processor_id())) { - mtrr_if->set_all(); + cache_cpu_init(); } return 0; } @@ -770,7 +770,7 @@ void __init mtrr_bp_init(void) =20 if (mtrr_cleanup(phys_addr)) { changed_by_mtrr_cleanup =3D 1; - mtrr_if->set_all(); + cache_cpu_init(); } } } @@ -856,7 +856,7 @@ void mtrr_bp_restore(void) if (!memory_caching_control) return; =20 - mtrr_if->set_all(); + cache_cpu_init(); } =20 static int __init mtrr_init_finialize(void) diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.h b/arch/x86/kernel/cpu/mtrr/mtr= r.h index 88b1c4b6174a..3b1883185185 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.h +++ b/arch/x86/kernel/cpu/mtrr/mtrr.h @@ -16,8 +16,6 @@ struct mtrr_ops { u32 vendor; void (*set)(unsigned int reg, unsigned long base, unsigned long size, mtrr_type type); - void (*set_all)(void); - void (*get)(unsigned int reg, unsigned long *base, unsigned long *size, mtrr_type *type); int (*get_free_region)(unsigned long base, unsigned long size, --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A89CC4332F for ; Wed, 2 Nov 2022 07:48:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230497AbiKBHse (ORCPT ); Wed, 2 Nov 2022 03:48:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbiKBHsI (ORCPT ); Wed, 2 Nov 2022 03:48:08 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6A4E2714C for ; Wed, 2 Nov 2022 00:48:07 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9C5951F8D0; Wed, 2 Nov 2022 07:48:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375286; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dqXfly+nuyRzVNoX1Dc+OjXV+YnmKzCTWh6bGlPZ0fg=; b=hxa617INPh9fzqGr3kbyQUayFOCFuaBCZ9uLqj9xvG3+RNrZTyxB82el3RtRth37stdlHA J4i2OeSAf5G16bqL1/J+s5/YCNMwmpSK8XygJ/W+Ji3j3BqwUCziIEg/0RrIoHYebSsAmB 3W66PBNkCuWC+EDBJdyNJFZ6eLEzM0g= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 65F781376E; Wed, 2 Nov 2022 07:48:06 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id i0a+F7YgYmP3cgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:06 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 09/16] x86/mtrr: simplify mtrr_bp_init() Date: Wed, 2 Nov 2022 08:47:06 +0100 Message-Id: <20221102074713.21493-10-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In case of the generic cache interface being used (Intel CPUs or a 64-bit system), the initialization sequence of the boot CPU is more complicated than necessary: - check if MTRR enabled, if yes, call mtrr_bp_pat_init() which will disable caching, set the PAT MSR, and reenable caching - call mtrr_cleanup(), in case that changed anything, call cache_cpu_init() doing the same caching disable/enable dance as above, but this time with setting the (modified) MTRR state (even if MTRR was disabled) AND setting the PAT MSR (again even with disabled MTRR) The sequence can be simplified a lot while removing potential inconsistencies: - check if MTRR enabled, if yes, call mtrr_cleanup() and then cache_cpu_init() This ensures to: - no longer disable/enable caching more than once - avoid to set MTRRs and/or the PAT MSR on the boot processor in case of MTRR cleanups even if MTRRs meant to be disabled With that mtrr_bp_pat_init() can be removed. Signed-off-by: Juergen Gross --- V2: - new patch --- arch/x86/kernel/cpu/mtrr/generic.c | 14 -------------- arch/x86/kernel/cpu/mtrr/mtrr.c | 6 +----- arch/x86/kernel/cpu/mtrr/mtrr.h | 1 - 3 files changed, 1 insertion(+), 20 deletions(-) diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index af8422c96b92..2f2485d6657f 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -442,20 +442,6 @@ static void __init print_mtrr_state(void) pr_debug("TOM2: %016llx aka %lldM\n", mtrr_tom2, mtrr_tom2>>20); } =20 -/* PAT setup for BP. We need to go through sync steps here */ -void __init mtrr_bp_pat_init(void) -{ - unsigned long flags; - - local_irq_save(flags); - cache_disable(); - - pat_init(); - - cache_enable(); - local_irq_restore(flags); -} - /* Grab all of the MTRR state for this CPU into *state */ bool __init get_mtrr_state(void) { diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index a44b510ced0e..a468be5d778f 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -764,12 +764,8 @@ void __init mtrr_bp_init(void) __mtrr_enabled =3D get_mtrr_state(); =20 if (mtrr_enabled()) { - mtrr_bp_pat_init(); memory_caching_control |=3D CACHE_MTRR | CACHE_PAT; - } - - if (mtrr_cleanup(phys_addr)) { - changed_by_mtrr_cleanup =3D 1; + changed_by_mtrr_cleanup =3D mtrr_cleanup(phys_addr); cache_cpu_init(); } } diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.h b/arch/x86/kernel/cpu/mtrr/mtr= r.h index 3b1883185185..c98928ceee6a 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.h +++ b/arch/x86/kernel/cpu/mtrr/mtrr.h @@ -50,7 +50,6 @@ void set_mtrr_prepare_save(struct set_mtrr_context *ctxt); void fill_mtrr_var_range(unsigned int index, u32 base_lo, u32 base_hi, u32 mask_lo, u32 mask_hi); bool get_mtrr_state(void); -void mtrr_bp_pat_init(void); =20 extern void __init set_mtrr_ops(const struct mtrr_ops *ops); =20 --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50F8BC4332F for ; Wed, 2 Nov 2022 07:48:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230503AbiKBHsj (ORCPT ); Wed, 2 Nov 2022 03:48:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59158 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230464AbiKBHsO (ORCPT ); Wed, 2 Nov 2022 03:48:14 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 850F62715C for ; Wed, 2 Nov 2022 00:48:13 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 464841F90F; Wed, 2 Nov 2022 07:48:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375292; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ux8/e9Woh3s8p0UvLOFok/6CxUvIyieZjB6VEj7G4a8=; b=BIorFGSD6wNJyIneTrGhMbM4cpdb9OTImr4C9otIkeQZVL7lnzgyvPA8CsLuyN0l2YqmHM nGYCHKnnM3D3c6q7keXpI43++4UBPl2PSwnPM4alZ0kEpqRScZt9i0vYpgGtp4GVwvzrRf 5O76h1edklbFA2nKkv0ZVGlxJwBCvKM= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 071DA1376E; Wed, 2 Nov 2022 07:48:12 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id Cb9oALwgYmP/cgAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:12 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 10/16] x86/mtrr: get rid of __mtrr_enabled bool Date: Wed, 2 Nov 2022 08:47:07 +0100 Message-Id: <20221102074713.21493-11-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There is no need for keeping __mtrr_enabled, as it can easily be replaced by testing mtrr_if to be not NULL. Signed-off-by: Juergen Gross --- V4: - new patch V5: - rebase --- arch/x86/kernel/cpu/mtrr/mtrr.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index a468be5d778f..f671be9823b6 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -59,11 +59,9 @@ #define MTRR_TO_PHYS_WC_OFFSET 1000 =20 u32 num_var_ranges; -static bool __mtrr_enabled; - static bool mtrr_enabled(void) { - return __mtrr_enabled; + return !!mtrr_if; } =20 unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; @@ -755,18 +753,17 @@ void __init mtrr_bp_init(void) } } =20 - if (mtrr_if) { - __mtrr_enabled =3D true; + if (mtrr_enabled()) { set_num_var_ranges(mtrr_if =3D=3D &generic_mtrr_ops); init_table(); if (mtrr_if =3D=3D &generic_mtrr_ops) { /* BIOS may override */ - __mtrr_enabled =3D get_mtrr_state(); - - if (mtrr_enabled()) { + if (get_mtrr_state()) { memory_caching_control |=3D CACHE_MTRR | CACHE_PAT; changed_by_mtrr_cleanup =3D mtrr_cleanup(phys_addr); cache_cpu_init(); + } else { + mtrr_if =3D NULL; } } } --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E2B6C4332F for ; Wed, 2 Nov 2022 07:48:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230527AbiKBHsp (ORCPT ); Wed, 2 Nov 2022 03:48:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230492AbiKBHsU (ORCPT ); Wed, 2 Nov 2022 03:48:20 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C3AE25E83 for ; Wed, 2 Nov 2022 00:48:19 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E8F763373A; Wed, 2 Nov 2022 07:48:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375297; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Lov3KMbpEv8g7a+8daqNFwrPW1J1X3RgaiyGShrBdqQ=; b=EwIcdofAfJvCRkRZfEZ5/66EjiOF2unOYKBvHC6JnghEcPWy8/uaFXj1bS3AwK2PYKTaB9 KG4/Rfcd2UIa2c4wJV9VFGrreUVjH01W/ChLDUeA1qPItEVgvRE+WgGHVogsXIwln/WbP+ Ex/lxTQ9usFrIvpDGIGahBhBDyBg4F8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id AAF8D1376E; Wed, 2 Nov 2022 07:48:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id XoRcKMEgYmMNcwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:17 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 11/16] x86/mtrr: let cache_aps_delayed_init replace mtrr_aps_delayed_init Date: Wed, 2 Nov 2022 08:47:08 +0100 Message-Id: <20221102074713.21493-12-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In order to prepare decoupling MTRR and PAT replace the MTRR specific mtrr_aps_delayed_init flag with a more generic cache_aps_delayed_init one. Signed-off-by: Juergen Gross --- V2: - new patch V4: - reestablish function to set cache_aps_delayed_init (Borislav Petkov) V5: - make cache_aps_delayed_init static, add get accessor (Borislav Petkov) --- arch/x86/include/asm/cacheinfo.h | 2 ++ arch/x86/include/asm/mtrr.h | 2 -- arch/x86/kernel/cpu/cacheinfo.c | 12 ++++++++++++ arch/x86/kernel/cpu/mtrr/mtrr.c | 18 +++++------------- arch/x86/kernel/smpboot.c | 5 +++-- 5 files changed, 22 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index 978bac70fd49..e443fcc1f045 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -13,5 +13,7 @@ void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, i= nt cpu); void cache_disable(void); void cache_enable(void); void cache_cpu_init(void); +void set_cache_aps_delayed_init(bool val); +bool get_cache_aps_delayed_init(void); =20 #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index 986249a2b9b6..5d31219c8529 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -43,7 +43,6 @@ extern int mtrr_del(int reg, unsigned long base, unsigned= long size); extern int mtrr_del_page(int reg, unsigned long base, unsigned long size); extern void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi); extern void mtrr_ap_init(void); -extern void set_mtrr_aps_delayed_init(void); extern void mtrr_aps_init(void); extern void mtrr_bp_restore(void); extern int mtrr_trim_uncached_memory(unsigned long end_pfn); @@ -87,7 +86,6 @@ static inline void mtrr_centaur_report_mcr(int mcr, u32 l= o, u32 hi) { } #define mtrr_ap_init() do {} while (0) -#define set_mtrr_aps_delayed_init() do {} while (0) #define mtrr_aps_init() do {} while (0) #define mtrr_bp_restore() do {} while (0) #define mtrr_disable() do {} while (0) diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 81ab99fe92bd..931ba3fb1363 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -1137,3 +1137,15 @@ void cache_cpu_init(void) cache_enable(); local_irq_restore(flags); } + +static bool cache_aps_delayed_init; + +void set_cache_aps_delayed_init(bool val) +{ + cache_aps_delayed_init =3D val; +} + +bool get_cache_aps_delayed_init(void) +{ + return cache_aps_delayed_init; +} diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index f671be9823b6..15ee6d72fb1f 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -68,7 +68,6 @@ unsigned int mtrr_usage_table[MTRR_MAX_VAR_RANGES]; static DEFINE_MUTEX(mtrr_mutex); =20 u64 size_or_mask, size_and_mask; -static bool mtrr_aps_delayed_init; =20 static const struct mtrr_ops *mtrr_ops[X86_VENDOR_NUM] __ro_after_init; =20 @@ -175,7 +174,8 @@ static int mtrr_rendezvous_handler(void *info) if (data->smp_reg !=3D ~0U) { mtrr_if->set(data->smp_reg, data->smp_base, data->smp_size, data->smp_type); - } else if (mtrr_aps_delayed_init || !cpu_online(smp_processor_id())) { + } else if (get_cache_aps_delayed_init() || + !cpu_online(smp_processor_id())) { cache_cpu_init(); } return 0; @@ -782,7 +782,7 @@ void __init mtrr_bp_init(void) =20 void mtrr_ap_init(void) { - if (!memory_caching_control || mtrr_aps_delayed_init) + if (!memory_caching_control || get_cache_aps_delayed_init()) return; =20 /* @@ -816,14 +816,6 @@ void mtrr_save_state(void) smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } =20 -void set_mtrr_aps_delayed_init(void) -{ - if (!memory_caching_control) - return; - - mtrr_aps_delayed_init =3D true; -} - /* * Delayed MTRR initialization for all AP's */ @@ -837,11 +829,11 @@ void mtrr_aps_init(void) * by doing set_mtrr_aps_delayed_init(), prior to this point. If not, * then we are done. */ - if (!mtrr_aps_delayed_init) + if (!get_cache_aps_delayed_init()) return; =20 set_mtrr(~0U, 0, 0, 0); - mtrr_aps_delayed_init =3D false; + set_cache_aps_delayed_init(false); } =20 void mtrr_bp_restore(void) diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 3f3ea0287f69..13c71ab29d84 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -58,6 +58,7 @@ #include =20 #include +#include #include #include #include @@ -1428,7 +1429,7 @@ void __init native_smp_prepare_cpus(unsigned int max_= cpus) =20 uv_system_init(); =20 - set_mtrr_aps_delayed_init(); + set_cache_aps_delayed_init(true); =20 smp_quirk_init_udelay(); =20 @@ -1439,7 +1440,7 @@ void __init native_smp_prepare_cpus(unsigned int max_= cpus) =20 void arch_thaw_secondary_cpus_begin(void) { - set_mtrr_aps_delayed_init(); + set_cache_aps_delayed_init(true); } =20 void arch_thaw_secondary_cpus_end(void) --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B528C4332F for ; Wed, 2 Nov 2022 07:48:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231153AbiKBHsu (ORCPT ); Wed, 2 Nov 2022 03:48:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230457AbiKBHs0 (ORCPT ); Wed, 2 Nov 2022 03:48:26 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0727D23151; Wed, 2 Nov 2022 00:48:25 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B4911338F3; Wed, 2 Nov 2022 07:48:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375303; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jL+jY4ifaFqfpcISKGZk1rhvt1fSpsik0OwYqcIWnDU=; b=o+9eFEVP6b4Wpu8d2ZEF86V7aJi7ksBB8opTSYJWk92vj2p0mdcQg7ftFxHN+iF084KvFf mktWJ1eohNgQGinzYPOjMJ0sk4WcrmY7d7mq0TFyVu4nERDT6uKaf4renJa0W4gn6rTlEe HALPPd5OWaZyCVJxXr4axfn7iNhzHD4= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 5ED901376E; Wed, 2 Nov 2022 07:48:23 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id JprbFccgYmMkcwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:23 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org, linux-pm@vger.kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , "Rafael J. Wysocki" , Pavel Machek Subject: [PATCH v5 12/16] x86/mtrr: add a stop_machine() handler calling only cache_cpu_init() Date: Wed, 2 Nov 2022 08:47:09 +0100 Message-Id: <20221102074713.21493-13-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Instead of having a stop_machine() handler for either a specific MTRR register or all state at once, add a handler just for calling cache_cpu_init() if appropriate. Add functions for calling stop_machine() with this handler as well. Add a generic replacement for mtrr_bp_restore() and a wrapper for mtrr_bp_init(). Signed-off-by: Juergen Gross --- V2: - completely new replacement of former patch 2 V5: - add a missing mtrr_bp_init() stub (Borislav Petkov) --- arch/x86/include/asm/cacheinfo.h | 5 +- arch/x86/include/asm/mtrr.h | 8 +-- arch/x86/kernel/cpu/cacheinfo.c | 59 ++++++++++++++++++++- arch/x86/kernel/cpu/common.c | 3 +- arch/x86/kernel/cpu/mtrr/mtrr.c | 88 +------------------------------- arch/x86/kernel/setup.c | 3 +- arch/x86/kernel/smpboot.c | 4 +- arch/x86/power/cpu.c | 3 +- 8 files changed, 74 insertions(+), 99 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index e443fcc1f045..a0ef46e9f453 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -12,8 +12,11 @@ void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, = int cpu); =20 void cache_disable(void); void cache_enable(void); -void cache_cpu_init(void); void set_cache_aps_delayed_init(bool val); bool get_cache_aps_delayed_init(void); +void cache_bp_init(void); +void cache_bp_restore(void); +void cache_ap_init(void); +void cache_aps_init(void); =20 #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index 5d31219c8529..f0eeaf6e5f5f 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -25,13 +25,12 @@ =20 #include =20 -void mtrr_bp_init(void); - /* * The following functions are for use by other drivers that cannot use * arch_phys_wc_add and arch_phys_wc_del. */ # ifdef CONFIG_MTRR +void mtrr_bp_init(void); extern u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform); extern void mtrr_save_fixed_ranges(void *); extern void mtrr_save_state(void); @@ -42,8 +41,6 @@ extern int mtrr_add_page(unsigned long base, unsigned lon= g size, extern int mtrr_del(int reg, unsigned long base, unsigned long size); extern int mtrr_del_page(int reg, unsigned long base, unsigned long size); extern void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi); -extern void mtrr_ap_init(void); -extern void mtrr_aps_init(void); extern void mtrr_bp_restore(void); extern int mtrr_trim_uncached_memory(unsigned long end_pfn); extern int amd_special_default_mtrr(void); @@ -85,8 +82,7 @@ static inline int mtrr_trim_uncached_memory(unsigned long= end_pfn) static inline void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 hi) { } -#define mtrr_ap_init() do {} while (0) -#define mtrr_aps_init() do {} while (0) +#define mtrr_bp_init() do {} while (0) #define mtrr_bp_restore() do {} while (0) #define mtrr_disable() do {} while (0) #define mtrr_enable() do {} while (0) diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 931ba3fb1363..a92099569617 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -15,6 +15,7 @@ #include #include #include +#include =20 #include #include @@ -1121,7 +1122,7 @@ void cache_enable(void) __releases(cache_disable_lock) raw_spin_unlock(&cache_disable_lock); } =20 -void cache_cpu_init(void) +static void cache_cpu_init(void) { unsigned long flags; =20 @@ -1149,3 +1150,59 @@ bool get_cache_aps_delayed_init(void) { return cache_aps_delayed_init; } + +static int cache_rendezvous_handler(void *unused) +{ + if (get_cache_aps_delayed_init() || !cpu_online(smp_processor_id())) + cache_cpu_init(); + + return 0; +} + +void __init cache_bp_init(void) +{ + mtrr_bp_init(); + + if (memory_caching_control) + cache_cpu_init(); +} + +void cache_bp_restore(void) +{ + if (memory_caching_control) + cache_cpu_init(); +} + +void cache_ap_init(void) +{ + if (!memory_caching_control || get_cache_aps_delayed_init()) + return; + + /* + * Ideally we should hold mtrr_mutex here to avoid MTRR entries + * changed, but this routine will be called in CPU boot time, + * holding the lock breaks it. + * + * This routine is called in two cases: + * + * 1. very early time of software resume, when there absolutely + * isn't MTRR entry changes; + * + * 2. CPU hotadd time. We let mtrr_add/del_page hold cpuhotplug + * lock to prevent MTRR entry changes + */ + stop_machine_from_inactive_cpu(cache_rendezvous_handler, NULL, + cpu_callout_mask); +} + +/* + * Delayed cache initialization for all AP's + */ +void cache_aps_init(void) +{ + if (!memory_caching_control || !get_cache_aps_delayed_init()) + return; + + stop_machine(cache_rendezvous_handler, NULL, cpu_online_mask); + set_cache_aps_delayed_init(false); +} diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 3e508f239098..fd058b547f8d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include @@ -1948,7 +1949,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - mtrr_ap_init(); + cache_ap_init(); validate_apic_and_package_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index 15ee6d72fb1f..99b6973a69b4 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -73,9 +73,6 @@ static const struct mtrr_ops *mtrr_ops[X86_VENDOR_NUM] __= ro_after_init; =20 const struct mtrr_ops *mtrr_if; =20 -static void set_mtrr(unsigned int reg, unsigned long base, - unsigned long size, mtrr_type type); - void __init set_mtrr_ops(const struct mtrr_ops *ops) { if (ops->vendor && ops->vendor < X86_VENDOR_NUM) @@ -158,26 +155,8 @@ static int mtrr_rendezvous_handler(void *info) { struct set_mtrr_data *data =3D info; =20 - /* - * We use this same function to initialize the mtrrs during boot, - * resume, runtime cpu online and on an explicit request to set a - * specific MTRR. - * - * During boot or suspend, the state of the boot cpu's mtrrs has been - * saved, and we want to replicate that across all the cpus that come - * online (either at the end of boot or resume or during a runtime cpu - * online). If we're doing that, @reg is set to something special and on - * all the CPUs we do cache_cpu_init() (On the logical CPU that - * started the boot/resume sequence, this might be a duplicate - * cache_cpu_init()). - */ - if (data->smp_reg !=3D ~0U) { - mtrr_if->set(data->smp_reg, data->smp_base, - data->smp_size, data->smp_type); - } else if (get_cache_aps_delayed_init() || - !cpu_online(smp_processor_id())) { - cache_cpu_init(); - } + mtrr_if->set(data->smp_reg, data->smp_base, + data->smp_size, data->smp_type); return 0; } =20 @@ -247,19 +226,6 @@ static void set_mtrr_cpuslocked(unsigned int reg, unsi= gned long base, stop_machine_cpuslocked(mtrr_rendezvous_handler, &data, cpu_online_mask); } =20 -static void set_mtrr_from_inactive_cpu(unsigned int reg, unsigned long bas= e, - unsigned long size, mtrr_type type) -{ - struct set_mtrr_data data =3D { .smp_reg =3D reg, - .smp_base =3D base, - .smp_size =3D size, - .smp_type =3D type - }; - - stop_machine_from_inactive_cpu(mtrr_rendezvous_handler, &data, - cpu_callout_mask); -} - /** * mtrr_add_page - Add a memory type region * @base: Physical base address of region in pages (in units of 4 kB!) @@ -761,7 +727,6 @@ void __init mtrr_bp_init(void) if (get_mtrr_state()) { memory_caching_control |=3D CACHE_MTRR | CACHE_PAT; changed_by_mtrr_cleanup =3D mtrr_cleanup(phys_addr); - cache_cpu_init(); } else { mtrr_if =3D NULL; } @@ -780,27 +745,6 @@ void __init mtrr_bp_init(void) } } =20 -void mtrr_ap_init(void) -{ - if (!memory_caching_control || get_cache_aps_delayed_init()) - return; - - /* - * Ideally we should hold mtrr_mutex here to avoid mtrr entries - * changed, but this routine will be called in cpu boot time, - * holding the lock breaks it. - * - * This routine is called in two cases: - * - * 1. very early time of software resume, when there absolutely - * isn't mtrr entry changes; - * - * 2. cpu hotadd time. We let mtrr_add/del_page hold cpuhotplug - * lock to prevent mtrr entry changes - */ - set_mtrr_from_inactive_cpu(~0U, 0, 0, 0); -} - /** * mtrr_save_state - Save current fixed-range MTRR state of the first * cpu in cpu_online_mask. @@ -816,34 +760,6 @@ void mtrr_save_state(void) smp_call_function_single(first_cpu, mtrr_save_fixed_ranges, NULL, 1); } =20 -/* - * Delayed MTRR initialization for all AP's - */ -void mtrr_aps_init(void) -{ - if (!memory_caching_control) - return; - - /* - * Check if someone has requested the delay of AP MTRR initialization, - * by doing set_mtrr_aps_delayed_init(), prior to this point. If not, - * then we are done. - */ - if (!get_cache_aps_delayed_init()) - return; - - set_mtrr(~0U, 0, 0, 0); - set_cache_aps_delayed_init(false); -} - -void mtrr_bp_restore(void) -{ - if (!memory_caching_control) - return; - - cache_cpu_init(); -} - static int __init mtrr_init_finialize(void) { if (!mtrr_enabled()) diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index 216fee7144ee..e0e185ee0229 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -1075,7 +1076,7 @@ void __init setup_arch(char **cmdline_p) =20 /* update e820 for memory not covered by WB MTRRs */ if (IS_ENABLED(CONFIG_MTRR)) - mtrr_bp_init(); + cache_bp_init(); else pat_disable("PAT support disabled because CONFIG_MTRR is disabled in the= kernel."); =20 diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 13c71ab29d84..1b61a480c966 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1445,7 +1445,7 @@ void arch_thaw_secondary_cpus_begin(void) =20 void arch_thaw_secondary_cpus_end(void) { - mtrr_aps_init(); + cache_aps_init(); } =20 /* @@ -1488,7 +1488,7 @@ void __init native_smp_cpus_done(unsigned int max_cpu= s) =20 nmi_selftest(); impress_friends(); - mtrr_aps_init(); + cache_aps_init(); } =20 static int __initdata setup_possible_cpus =3D -1; diff --git a/arch/x86/power/cpu.c b/arch/x86/power/cpu.c index bb176c72891c..754221c9a1c3 100644 --- a/arch/x86/power/cpu.c +++ b/arch/x86/power/cpu.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -261,7 +262,7 @@ static void notrace __restore_processor_state(struct sa= ved_context *ctxt) do_fpu_end(); tsc_verify_tsc_adjust(true); x86_platform.restore_sched_clock_state(); - mtrr_bp_restore(); + cache_bp_restore(); perf_restore_debug_store(); =20 c =3D &cpu_data(smp_processor_id()); --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 618EEC4332F for ; Wed, 2 Nov 2022 07:49:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230366AbiKBHtE (ORCPT ); Wed, 2 Nov 2022 03:49:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230489AbiKBHsc (ORCPT ); Wed, 2 Nov 2022 03:48:32 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AB3A32612B for ; Wed, 2 Nov 2022 00:48:30 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 6A0B91F8B0; Wed, 2 Nov 2022 07:48:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375309; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R10C4rCtq1ivY+IhxBh2ebzergfYAXSfqeTMWrGpHUU=; b=qZQZ0JcdkHpjwpSv3z1bF5ehq6gHqILX5N8Vbq8iXlMOAa+LIawMaBw6ksJlv/nGRZE22P FSE+oowEBloJhkwhaad3tO4V18zCI1uRxrvU73huJqz7+a3Lt1hvwcWhV6F7wy2FzyH+Xw sp3IHVCmEwTHpuzXZU4CI7aZbv5DiGY= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 224DE1376E; Wed, 2 Nov 2022 07:48:29 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id /UczB80gYmM3cwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:29 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra Subject: [PATCH v5 13/16] x86: decouple PAT and MTRR handling Date: Wed, 2 Nov 2022 08:47:10 +0100 Message-Id: <20221102074713.21493-14-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Today PAT is usable only with MTRR being active, with some nasty tweaks to make PAT usable when running as Xen PV guest, which doesn't support MTRR. The reason for this coupling is, that both, PAT MSR changes and MTRR changes, require a similar sequence and so full PAT support was added using the already available MTRR handling. Xen PV PAT handling can work without MTRR, as it just needs to consume the PAT MSR setting done by the hypervisor without the ability and need to change it. This in turn has resulted in a convoluted initialization sequence and wrong decisions regarding cache mode availability due to misguiding PAT availability flags. Fix all of that by allowing to use PAT without MTRR and by reworking the current PAT initialization sequence to match better with the newly introduced generic cache initialization. This removes the need of the recently added pat_force_disabled flag, so remove the remnants of the patch adding it. Signed-off-by: Juergen Gross --- V2: - former patch 3 completely reworked V5: - rework PAT() macro - drop local pat variable (Borislav Petkov) - use cpu_feature_enabled() (Borislav Petkov) - some more minor adjustments (Borislav Petkov) --- arch/x86/include/asm/memtype.h | 5 +- arch/x86/kernel/cpu/cacheinfo.c | 3 +- arch/x86/kernel/cpu/mtrr/mtrr.c | 12 +-- arch/x86/kernel/setup.c | 13 +-- arch/x86/mm/pat/memtype.c | 152 +++++++++++--------------------- 5 files changed, 57 insertions(+), 128 deletions(-) diff --git a/arch/x86/include/asm/memtype.h b/arch/x86/include/asm/memtype.h index 9ca760e430b9..113b2fa51849 100644 --- a/arch/x86/include/asm/memtype.h +++ b/arch/x86/include/asm/memtype.h @@ -6,9 +6,8 @@ #include =20 extern bool pat_enabled(void); -extern void pat_disable(const char *reason); -extern void pat_init(void); -extern void init_cache_modes(void); +extern void pat_bp_init(void); +extern void pat_cpu_init(void); =20 extern int memtype_reserve(u64 start, u64 end, enum page_cache_mode req_pcm, enum page_cache_mode *ret_pcm); diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index a92099569617..1aaf830254df 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -1133,7 +1133,7 @@ static void cache_cpu_init(void) mtrr_generic_set_state(); =20 if (memory_caching_control & CACHE_PAT) - pat_init(); + pat_cpu_init(); =20 cache_enable(); local_irq_restore(flags); @@ -1162,6 +1162,7 @@ static int cache_rendezvous_handler(void *unused) void __init cache_bp_init(void) { mtrr_bp_init(); + pat_bp_init(); =20 if (memory_caching_control) cache_cpu_init(); diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index 99b6973a69b4..8403daf34158 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -725,7 +725,7 @@ void __init mtrr_bp_init(void) if (mtrr_if =3D=3D &generic_mtrr_ops) { /* BIOS may override */ if (get_mtrr_state()) { - memory_caching_control |=3D CACHE_MTRR | CACHE_PAT; + memory_caching_control |=3D CACHE_MTRR; changed_by_mtrr_cleanup =3D mtrr_cleanup(phys_addr); } else { mtrr_if =3D NULL; @@ -733,16 +733,8 @@ void __init mtrr_bp_init(void) } } =20 - if (!mtrr_enabled()) { + if (!mtrr_enabled()) pr_info("Disabled\n"); - - /* - * PAT initialization relies on MTRR's rendezvous handler. - * Skip PAT init until the handler can initialize both - * features independently. - */ - pat_disable("MTRRs disabled, skipping PAT initialization too."); - } } =20 /** diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index e0e185ee0229..aacaa96f0195 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1075,23 +1075,12 @@ void __init setup_arch(char **cmdline_p) max_pfn =3D e820__end_of_ram_pfn(); =20 /* update e820 for memory not covered by WB MTRRs */ - if (IS_ENABLED(CONFIG_MTRR)) - cache_bp_init(); - else - pat_disable("PAT support disabled because CONFIG_MTRR is disabled in the= kernel."); - + cache_bp_init(); if (mtrr_trim_uncached_memory(max_pfn)) max_pfn =3D e820__end_of_ram_pfn(); =20 max_possible_pfn =3D max_pfn; =20 - /* - * This call is required when the CPU does not support PAT. If - * mtrr_bp_init() invoked it already via pat_init() the call has no - * effect. - */ - init_cache_modes(); - /* * Define random base addresses for memory sections after max_pfn is * defined and before each memory section base is used. diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 66a209f7eb86..9aab17d660cd 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -43,6 +43,7 @@ #include =20 #include +#include #include #include #include @@ -60,41 +61,34 @@ #undef pr_fmt #define pr_fmt(fmt) "" fmt =20 -static bool __read_mostly pat_bp_initialized; static bool __read_mostly pat_disabled =3D !IS_ENABLED(CONFIG_X86_PAT); -static bool __initdata pat_force_disabled =3D !IS_ENABLED(CONFIG_X86_PAT); -static bool __read_mostly pat_bp_enabled; -static bool __read_mostly pat_cm_initialized; +static u64 __ro_after_init pat_msr_val; =20 /* * PAT support is enabled by default, but can be disabled for * various user-requested or hardware-forced reasons: */ -void pat_disable(const char *msg_reason) +static void __init pat_disable(const char *msg_reason) { if (pat_disabled) return; =20 - if (pat_bp_initialized) { - WARN_ONCE(1, "x86/PAT: PAT cannot be disabled after initialization\n"); - return; - } - pat_disabled =3D true; pr_info("x86/PAT: %s\n", msg_reason); + + memory_caching_control &=3D ~CACHE_PAT; } =20 static int __init nopat(char *str) { pat_disable("PAT support disabled via boot option."); - pat_force_disabled =3D true; return 0; } early_param("nopat", nopat); =20 bool pat_enabled(void) { - return pat_bp_enabled; + return !pat_disabled; } EXPORT_SYMBOL_GPL(pat_enabled); =20 @@ -192,7 +186,8 @@ enum { =20 #define CM(c) (_PAGE_CACHE_MODE_ ## c) =20 -static enum page_cache_mode pat_get_cache_mode(unsigned pat_val, char *msg) +static enum page_cache_mode __init pat_get_cache_mode(unsigned int pat_val, + char *msg) { enum page_cache_mode cache; char *cache_mode; @@ -219,14 +214,12 @@ static enum page_cache_mode pat_get_cache_mode(unsign= ed pat_val, char *msg) * configuration. * Using lower indices is preferred, so we start with highest index. */ -static void __init_cache_modes(u64 pat) +static void __init init_cache_modes(u64 pat) { enum page_cache_mode cache; char pat_msg[33]; int i; =20 - WARN_ON_ONCE(pat_cm_initialized); - pat_msg[32] =3D 0; for (i =3D 7; i >=3D 0; i--) { cache =3D pat_get_cache_mode((pat >> (i * 8)) & 7, @@ -234,34 +227,9 @@ static void __init_cache_modes(u64 pat) update_cache_mode_entry(i, cache); } pr_info("x86/PAT: Configuration [0-7]: %s\n", pat_msg); - - pat_cm_initialized =3D true; } =20 -#define PAT(x, y) ((u64)PAT_ ## y << ((x)*8)) - -static void pat_bp_init(u64 pat) -{ - u64 tmp_pat; - - if (!boot_cpu_has(X86_FEATURE_PAT)) { - pat_disable("PAT not supported by the CPU."); - return; - } - - rdmsrl(MSR_IA32_CR_PAT, tmp_pat); - if (!tmp_pat) { - pat_disable("PAT support disabled by the firmware."); - return; - } - - wrmsrl(MSR_IA32_CR_PAT, pat); - pat_bp_enabled =3D true; - - __init_cache_modes(pat); -} - -static void pat_ap_init(u64 pat) +void pat_cpu_init(void) { if (!boot_cpu_has(X86_FEATURE_PAT)) { /* @@ -271,30 +239,39 @@ static void pat_ap_init(u64 pat) panic("x86/PAT: PAT enabled, but not supported by secondary CPU\n"); } =20 - wrmsrl(MSR_IA32_CR_PAT, pat); + wrmsrl(MSR_IA32_CR_PAT, pat_msr_val); } =20 -void __init init_cache_modes(void) +/** + * pat_bp_init - Initialize the PAT MSR value and PAT table + * + * This function initializes PAT MSR value and PAT table with an OS-defined + * value to enable additional cache attributes, WC, WT and WP. + * + * This function prepares the calls of pat_cpu_init() via cache_cpu_init() + * on all CPUs. + */ +void __init pat_bp_init(void) { - u64 pat =3D 0; + struct cpuinfo_x86 *c =3D &boot_cpu_data; +#define PAT(p0, p1, p2, p3, p4, p5, p6, p7) \ + (((u64)PAT_ ## p0) | ((u64)PAT_ ## p1 << 8) | \ + ((u64)PAT_ ## p2 << 16) | ((u64)PAT_ ## p3 << 24) | \ + ((u64)PAT_ ## p4 << 32) | ((u64)PAT_ ## p5 << 40) | \ + ((u64)PAT_ ## p6 << 48) | ((u64)PAT_ ## p7 << 56)) =20 - if (pat_cm_initialized) - return; =20 - if (boot_cpu_has(X86_FEATURE_PAT)) { - /* - * CPU supports PAT. Set PAT table to be consistent with - * PAT MSR. This case supports "nopat" boot option, and - * virtual machine environments which support PAT without - * MTRRs. In specific, Xen has unique setup to PAT MSR. - * - * If PAT MSR returns 0, it is considered invalid and emulates - * as No PAT. - */ - rdmsrl(MSR_IA32_CR_PAT, pat); - } + if (!IS_ENABLED(CONFIG_X86_PAT)) + pr_info_once("x86/PAT: PAT support disabled because CONFIG_X86_PAT is di= sabled in the kernel.\n"); + + if (!cpu_feature_enabled(X86_FEATURE_PAT)) + pat_disable("PAT not supported by the CPU."); + else + rdmsrl(MSR_IA32_CR_PAT, pat_msr_val); + + if (!pat_msr_val) { + pat_disable("PAT support disabled by the firmware."); =20 - if (!pat) { /* * No PAT. Emulate the PAT table that corresponds to the two * cache bits, PWT (Write Through) and PCD (Cache Disable). @@ -313,40 +290,17 @@ void __init init_cache_modes(void) * NOTE: When WC or WP is used, it is redirected to UC- per * the default setup in __cachemode2pte_tbl[]. */ - pat =3D PAT(0, WB) | PAT(1, WT) | PAT(2, UC_MINUS) | PAT(3, UC) | - PAT(4, WB) | PAT(5, WT) | PAT(6, UC_MINUS) | PAT(7, UC); - } else if (!pat_force_disabled && cpu_feature_enabled(X86_FEATURE_HYPERVI= SOR)) { - /* - * Clearly PAT is enabled underneath. Allow pat_enabled() to - * reflect this. - */ - pat_bp_enabled =3D true; + pat_msr_val =3D PAT(WB, WT, UC_MINUS, UC, WB, WT, UC_MINUS, UC); } =20 - __init_cache_modes(pat); -} - -/** - * pat_init - Initialize the PAT MSR and PAT table on the current CPU - * - * This function initializes PAT MSR and PAT table with an OS-defined value - * to enable additional cache attributes, WC, WT and WP. - * - * This function must be called on all CPUs using the specific sequence of - * operations defined in Intel SDM. mtrr_rendezvous_handler() provides this - * procedure for PAT. - */ -void pat_init(void) -{ - u64 pat; - struct cpuinfo_x86 *c =3D &boot_cpu_data; - -#ifndef CONFIG_X86_PAT - pr_info_once("x86/PAT: PAT support disabled because CONFIG_X86_PAT is dis= abled in the kernel.\n"); -#endif - - if (pat_disabled) + /* + * Xen PV doesn't allow to set PAT MSR, but all cache modes are + * supported. + */ + if (pat_disabled || cpu_feature_enabled(X86_FEATURE_XENPV)) { + init_cache_modes(pat_msr_val); return; + } =20 if ((c->x86_vendor =3D=3D X86_VENDOR_INTEL) && (((c->x86 =3D=3D 0x6) && (c->x86_model <=3D 0xd)) || @@ -371,8 +325,7 @@ void pat_init(void) * NOTE: When WT or WP is used, it is redirected to UC- per * the default setup in __cachemode2pte_tbl[]. */ - pat =3D PAT(0, WB) | PAT(1, WC) | PAT(2, UC_MINUS) | PAT(3, UC) | - PAT(4, WB) | PAT(5, WC) | PAT(6, UC_MINUS) | PAT(7, UC); + pat_msr_val =3D PAT(WB, WC, UC_MINUS, UC, WB, WC, UC_MINUS, UC); } else { /* * Full PAT support. We put WT in slot 7 to improve @@ -400,19 +353,14 @@ void pat_init(void) * The reserved slots are unused, but mapped to their * corresponding types in the presence of PAT errata. */ - pat =3D PAT(0, WB) | PAT(1, WC) | PAT(2, UC_MINUS) | PAT(3, UC) | - PAT(4, WB) | PAT(5, WP) | PAT(6, UC_MINUS) | PAT(7, WT); + pat_msr_val =3D PAT(WB, WC, UC_MINUS, UC, WB, WP, UC_MINUS, WT); } =20 - if (!pat_bp_initialized) { - pat_bp_init(pat); - pat_bp_initialized =3D true; - } else { - pat_ap_init(pat); - } -} + memory_caching_control |=3D CACHE_PAT; =20 + init_cache_modes(pat_msr_val); #undef PAT +} =20 static DEFINE_SPINLOCK(memtype_lock); /* protects memtype accesses */ =20 --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2A69C433FE for ; Wed, 2 Nov 2022 07:49:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231127AbiKBHtI (ORCPT ); Wed, 2 Nov 2022 03:49:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230502AbiKBHsi (ORCPT ); Wed, 2 Nov 2022 03:48:38 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CB2D275E7 for ; Wed, 2 Nov 2022 00:48:36 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 236111F8D0; Wed, 2 Nov 2022 07:48:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375315; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8mPWpLwJy7yzhex6eU7j5tLklymwcAzwe+PPdkowDuM=; b=jK+xtRHuGxgvIkpvOFSz7r4XQb0A4BZmX2KszjLMCulWag7FTQL5eJPNj8vloNmCtmUGAR VD6ozlxoai+mtixWWex89re4ODBn3w+t6Si82LwdfzTxwfv9MLOyQuOKiL0IJmCbI0saFh neFoGFASCrsQr0TiitJCiVMjDmtHjd8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id CDD681376E; Wed, 2 Nov 2022 07:48:34 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id VZ/XMNIgYmNIcwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:34 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 14/16] x86: switch cache_ap_init() to hotplug callback Date: Wed, 2 Nov 2022 08:47:11 +0100 Message-Id: <20221102074713.21493-15-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Instead of explicitly calling cache_ap_init() in identify_secondary_cpu() use a CPU hotplug callback instead. By registering the callback only after having started the non-boot CPUs and initializing cache_aps_delayed_init with "true", calling set_cache_aps_delayed_init() at boot time can be dropped. It should be noted that this change results in cache_ap_init() being called a little bit later when hotplugging CPUs. By using a new hotplug slot right at the start of the low level bringup this is not problematic, as no operations requiring a specific caching mode are performed that early in CPU initialization. Suggested-by: Borislav Petkov Signed-off-by: Juergen Gross --- V4: - new patch --- arch/x86/include/asm/cacheinfo.h | 1 - arch/x86/kernel/cpu/cacheinfo.c | 18 +++++++++++++++--- arch/x86/kernel/cpu/common.c | 1 - arch/x86/kernel/smpboot.c | 2 -- include/linux/cpuhotplug.h | 1 + 5 files changed, 16 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index a0ef46e9f453..ce9685fc78d8 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -16,7 +16,6 @@ void set_cache_aps_delayed_init(bool val); bool get_cache_aps_delayed_init(void); void cache_bp_init(void); void cache_bp_restore(void); -void cache_ap_init(void); void cache_aps_init(void); =20 #endif /* _ASM_X86_CACHEINFO_H */ diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 1aaf830254df..231cf1ff0641 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -1139,7 +1140,7 @@ static void cache_cpu_init(void) local_irq_restore(flags); } =20 -static bool cache_aps_delayed_init; +static bool cache_aps_delayed_init =3D true; =20 void set_cache_aps_delayed_init(bool val) { @@ -1174,10 +1175,10 @@ void cache_bp_restore(void) cache_cpu_init(); } =20 -void cache_ap_init(void) +static int cache_ap_init(unsigned int cpu) { if (!memory_caching_control || get_cache_aps_delayed_init()) - return; + return 0; =20 /* * Ideally we should hold mtrr_mutex here to avoid MTRR entries @@ -1194,6 +1195,8 @@ void cache_ap_init(void) */ stop_machine_from_inactive_cpu(cache_rendezvous_handler, NULL, cpu_callout_mask); + + return 0; } =20 /* @@ -1207,3 +1210,12 @@ void cache_aps_init(void) stop_machine(cache_rendezvous_handler, NULL, cpu_online_mask); set_cache_aps_delayed_init(false); } + +static int __init cache_ap_register(void) +{ + cpuhp_setup_state_nocalls(CPUHP_AP_CACHECTRL_STARTING, + "x86/cachectrl:starting", + cache_ap_init, NULL); + return 0; +} +core_initcall(cache_ap_register); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index fd058b547f8d..bf4ac1cb93d7 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -1949,7 +1949,6 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c) #ifdef CONFIG_X86_32 enable_sep_cpu(); #endif - cache_ap_init(); validate_apic_and_package_id(c); x86_spec_ctrl_setup_ap(); update_srbds_msr(); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 1b61a480c966..82b311c718bc 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -1429,8 +1429,6 @@ void __init native_smp_prepare_cpus(unsigned int max_= cpus) =20 uv_system_init(); =20 - set_cache_aps_delayed_init(true); - smp_quirk_init_udelay(); =20 speculative_store_bypass_ht_init(); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f61447913db9..0d277b4b025a 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -140,6 +140,7 @@ enum cpuhp_state { */ CPUHP_AP_IDLE_DEAD, CPUHP_AP_OFFLINE, + CPUHP_AP_CACHECTRL_STARTING, CPUHP_AP_SCHED_STARTING, CPUHP_AP_RCUTREE_DYING, CPUHP_AP_CPU_PM_STARTING, --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FA89C433FE for ; Wed, 2 Nov 2022 07:49:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231135AbiKBHtM (ORCPT ); Wed, 2 Nov 2022 03:49:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230513AbiKBHsn (ORCPT ); Wed, 2 Nov 2022 03:48:43 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E6FC2610D for ; Wed, 2 Nov 2022 00:48:42 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BDBF21F8B0; Wed, 2 Nov 2022 07:48:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375320; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SkMhXh10I996ogHNk1iHONe/Tn2QI4iwdDoxFOo7ziE=; b=cb6qsJD1Rkw1pfAhLr10zuRBQukLiY/H3TeGxwjAIRapVdkuRuxjlo+tDPenQTTeASmlfP hA4a78kWEQCMmPlWT563YnpAX3ey8KBNwLCNDxzDx+w72hWxcrVogIbGch8A+KRvMz4/Xc xPLB9ZBu2e7J2pK+IU4eBCILKwXvaxw= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 8208C1376E; Wed, 2 Nov 2022 07:48:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id nX1lHtggYmNRcwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:40 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 15/16] x86: do MTRR/PAT setup on all secondary CPUs in parallel Date: Wed, 2 Nov 2022 08:47:12 +0100 Message-Id: <20221102074713.21493-16-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Instead of serializing MTRR/PAT setup on the secondary CPUs in order to avoid clobbering of static variables used by the setup process, put those variables into a structure held on the stack and drop the serialization. This speeds up the start of secondary CPUs a little bit (on a small system with 8 CPUs the time needed for starting the secondary CPUs was measured to go down from about 60 milliseconds without this patch to about 55 milliseconds with this patch applied). Signed-off-by: Juergen Gross --- V4: - new patch --- arch/x86/include/asm/cacheinfo.h | 10 ++++++-- arch/x86/include/asm/mtrr.h | 13 +++++----- arch/x86/kernel/cpu/cacheinfo.c | 28 ++++++++------------- arch/x86/kernel/cpu/mtrr/generic.c | 40 ++++++++++++++---------------- 4 files changed, 45 insertions(+), 46 deletions(-) diff --git a/arch/x86/include/asm/cacheinfo.h b/arch/x86/include/asm/cachei= nfo.h index ce9685fc78d8..f66578e1e4e1 100644 --- a/arch/x86/include/asm/cacheinfo.h +++ b/arch/x86/include/asm/cacheinfo.h @@ -10,8 +10,14 @@ extern unsigned int memory_caching_control; void cacheinfo_amd_init_llc_id(struct cpuinfo_x86 *c, int cpu); void cacheinfo_hygon_init_llc_id(struct cpuinfo_x86 *c, int cpu); =20 -void cache_disable(void); -void cache_enable(void); +struct cache_state { + unsigned long cr4; + u32 mtrr_deftype_lo; + u32 mtrr_deftype_hi; +}; + +void cache_disable(struct cache_state *state); +void cache_enable(struct cache_state *state); void set_cache_aps_delayed_init(bool val); bool get_cache_aps_delayed_init(void); void cache_bp_init(void); diff --git a/arch/x86/include/asm/mtrr.h b/arch/x86/include/asm/mtrr.h index f0eeaf6e5f5f..2ea4a9de7318 100644 --- a/arch/x86/include/asm/mtrr.h +++ b/arch/x86/include/asm/mtrr.h @@ -23,6 +23,7 @@ #ifndef _ASM_X86_MTRR_H #define _ASM_X86_MTRR_H =20 +#include #include =20 /* @@ -44,9 +45,9 @@ extern void mtrr_centaur_report_mcr(int mcr, u32 lo, u32 = hi); extern void mtrr_bp_restore(void); extern int mtrr_trim_uncached_memory(unsigned long end_pfn); extern int amd_special_default_mtrr(void); -void mtrr_disable(void); -void mtrr_enable(void); -void mtrr_generic_set_state(void); +void mtrr_disable(struct cache_state *state); +void mtrr_enable(struct cache_state *state); +void mtrr_generic_set_state(struct cache_state *state); # else static inline u8 mtrr_type_lookup(u64 addr, u64 end, u8 *uniform) { @@ -84,9 +85,9 @@ static inline void mtrr_centaur_report_mcr(int mcr, u32 l= o, u32 hi) } #define mtrr_bp_init() do {} while (0) #define mtrr_bp_restore() do {} while (0) -#define mtrr_disable() do {} while (0) -#define mtrr_enable() do {} while (0) -#define mtrr_generic_set_state() do {} while (0) +#define mtrr_disable(s) do {} while (0) +#define mtrr_enable(s) do {} while (0) +#define mtrr_generic_set_state(s) do {} while (0) # endif =20 #ifdef CONFIG_COMPAT diff --git a/arch/x86/kernel/cpu/cacheinfo.c b/arch/x86/kernel/cpu/cacheinf= o.c index 231cf1ff0641..7e370c979417 100644 --- a/arch/x86/kernel/cpu/cacheinfo.c +++ b/arch/x86/kernel/cpu/cacheinfo.c @@ -1057,10 +1057,7 @@ int populate_cache_leaves(unsigned int cpu) * The caller must ensure that local interrupts are disabled and * are reenabled after cache_enable() has been called. */ -static unsigned long saved_cr4; -static DEFINE_RAW_SPINLOCK(cache_disable_lock); - -void cache_disable(void) __acquires(cache_disable_lock) +void cache_disable(struct cache_state *state) { unsigned long cr0; =20 @@ -1071,8 +1068,6 @@ void cache_disable(void) __acquires(cache_disable_loc= k) * changes to the way the kernel boots */ =20 - raw_spin_lock(&cache_disable_lock); - /* Enter the no-fill (CD=3D1, NW=3D0) cache mode and flush caches. */ cr0 =3D read_cr0() | X86_CR0_CD; write_cr0(cr0); @@ -1088,8 +1083,8 @@ void cache_disable(void) __acquires(cache_disable_loc= k) =20 /* Save value of CR4 and clear Page Global Enable (bit 7) */ if (cpu_feature_enabled(X86_FEATURE_PGE)) { - saved_cr4 =3D __read_cr4(); - __write_cr4(saved_cr4 & ~X86_CR4_PGE); + state->cr4 =3D __read_cr4(); + __write_cr4(state->cr4 & ~X86_CR4_PGE); } =20 /* Flush all TLBs via a mov %cr3, %reg; mov %reg, %cr3 */ @@ -1097,46 +1092,45 @@ void cache_disable(void) __acquires(cache_disable_l= ock) flush_tlb_local(); =20 if (cpu_feature_enabled(X86_FEATURE_MTRR)) - mtrr_disable(); + mtrr_disable(state); =20 /* Again, only flush caches if we have to. */ if (!static_cpu_has(X86_FEATURE_SELFSNOOP)) wbinvd(); } =20 -void cache_enable(void) __releases(cache_disable_lock) +void cache_enable(struct cache_state *state) { /* Flush TLBs (no need to flush caches - they are disabled) */ count_vm_tlb_event(NR_TLB_LOCAL_FLUSH_ALL); flush_tlb_local(); =20 if (cpu_feature_enabled(X86_FEATURE_MTRR)) - mtrr_enable(); + mtrr_enable(state); =20 /* Enable caches */ write_cr0(read_cr0() & ~X86_CR0_CD); =20 /* Restore value of CR4 */ if (cpu_feature_enabled(X86_FEATURE_PGE)) - __write_cr4(saved_cr4); - - raw_spin_unlock(&cache_disable_lock); + __write_cr4(state->cr4); } =20 static void cache_cpu_init(void) { unsigned long flags; + struct cache_state state; =20 local_irq_save(flags); - cache_disable(); + cache_disable(&state); =20 if (memory_caching_control & CACHE_MTRR) - mtrr_generic_set_state(); + mtrr_generic_set_state(&state); =20 if (memory_caching_control & CACHE_PAT) pat_cpu_init(); =20 - cache_enable(); + cache_enable(&state); local_irq_restore(flags); } =20 diff --git a/arch/x86/kernel/cpu/mtrr/generic.c b/arch/x86/kernel/cpu/mtrr/= generic.c index 2f2485d6657f..cddb440f330d 100644 --- a/arch/x86/kernel/cpu/mtrr/generic.c +++ b/arch/x86/kernel/cpu/mtrr/generic.c @@ -663,18 +663,13 @@ static bool set_mtrr_var_ranges(unsigned int index, s= truct mtrr_var_range *vr) return changed; } =20 -static u32 deftype_lo, deftype_hi; - /** * set_mtrr_state - Set the MTRR state for this CPU. * - * NOTE: The CPU must already be in a safe state for MTRR changes, includi= ng - * measures that only a single CPU can be active in set_mtrr_state()= in - * order to not be subject to races for usage of deftype_lo (this is - * accomplished by taking cache_disable_lock). + * NOTE: The CPU must already be in a safe state for MTRR changes. * RETURNS: 0 if no changes made, else a mask indicating what was changed. */ -static unsigned long set_mtrr_state(void) +static unsigned long set_mtrr_state(struct cache_state *state) { unsigned long change_mask =3D 0; unsigned int i; @@ -691,38 +686,40 @@ static unsigned long set_mtrr_state(void) * Set_mtrr_restore restores the old value of MTRRdefType, * so to set it we fiddle with the saved value: */ - if ((deftype_lo & 0xff) !=3D mtrr_state.def_type - || ((deftype_lo & 0xc00) >> 10) !=3D mtrr_state.enabled) { - - deftype_lo =3D (deftype_lo & ~0xcff) | mtrr_state.def_type | - (mtrr_state.enabled << 10); + if ((state->mtrr_deftype_lo & 0xff) !=3D mtrr_state.def_type + || ((state->mtrr_deftype_lo & 0xc00) >> 10) !=3D mtrr_state.enabled) { + state->mtrr_deftype_lo =3D (state->mtrr_deftype_lo & ~0xcff) | + mtrr_state.def_type | + (mtrr_state.enabled << 10); change_mask |=3D MTRR_CHANGE_MASK_DEFTYPE; } =20 return change_mask; } =20 -void mtrr_disable(void) +void mtrr_disable(struct cache_state *state) { /* Save MTRR state */ - rdmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); + rdmsr(MSR_MTRRdefType, state->mtrr_deftype_lo, state->mtrr_deftype_hi); =20 /* Disable MTRRs, and set the default type to uncached */ - mtrr_wrmsr(MSR_MTRRdefType, deftype_lo & ~0xcff, deftype_hi); + mtrr_wrmsr(MSR_MTRRdefType, state->mtrr_deftype_lo & ~0xcff, + state->mtrr_deftype_hi); } =20 -void mtrr_enable(void) +void mtrr_enable(struct cache_state *state) { /* Intel (P6) standard MTRRs */ - mtrr_wrmsr(MSR_MTRRdefType, deftype_lo, deftype_hi); + mtrr_wrmsr(MSR_MTRRdefType, state->mtrr_deftype_lo, + state->mtrr_deftype_hi); } =20 -void mtrr_generic_set_state(void) +void mtrr_generic_set_state(struct cache_state *state) { unsigned long mask, count; =20 /* Actually set the state */ - mask =3D set_mtrr_state(); + mask =3D set_mtrr_state(state); =20 /* Use the atomic bitops to update the global mask */ for (count =3D 0; count < sizeof(mask) * 8; ++count) { @@ -747,11 +744,12 @@ static void generic_set_mtrr(unsigned int reg, unsign= ed long base, { unsigned long flags; struct mtrr_var_range *vr; + struct cache_state state; =20 vr =3D &mtrr_state.var_ranges[reg]; =20 local_irq_save(flags); - cache_disable(); + cache_disable(&state); =20 if (size =3D=3D 0) { /* @@ -770,7 +768,7 @@ static void generic_set_mtrr(unsigned int reg, unsigned= long base, mtrr_wrmsr(MTRRphysMask_MSR(reg), vr->mask_lo, vr->mask_hi); } =20 - cache_enable(); + cache_enable(&state); local_irq_restore(flags); } =20 --=20 2.35.3 From nobody Thu Apr 9 03:15:28 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F06B8C433FE for ; Wed, 2 Nov 2022 07:49:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230197AbiKBHtU (ORCPT ); Wed, 2 Nov 2022 03:49:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60214 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231145AbiKBHst (ORCPT ); Wed, 2 Nov 2022 03:48:49 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2CAF27141 for ; Wed, 2 Nov 2022 00:48:47 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 692B121F6E; Wed, 2 Nov 2022 07:48:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1667375326; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kVtAWSumNurkoytRaTQXv7/5HrCR6FvTBpLocTA2bAY=; b=OIXg+UvpJqtAzZAoGeQCK/oKEPT6ZCODpZRPETtMlMYdoMMFIyuf02xCiw/udVP0r+WVbX /tfH3wkn1vaubIbk9R1GNlReODSacP/7iWsX67dNNoyC+/kpIK1sSqBw/DvU5a7UrRphyp muhbfPXSTPd9oQ4XbvEHj8RxQYZAE0k= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2D3301376E; Wed, 2 Nov 2022 07:48:46 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id SK2UCd4gYmNecwAAMHmgww (envelope-from ); Wed, 02 Nov 2022 07:48:46 +0000 From: Juergen Gross To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Juergen Gross , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Subject: [PATCH v5 16/16] x86/mtrr: simplify mtrr_ops initialization Date: Wed, 2 Nov 2022 08:47:13 +0100 Message-Id: <20221102074713.21493-17-jgross@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20221102074713.21493-1-jgross@suse.com> References: <20221102074713.21493-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The way mtrr_if is initialized with the correct mtrr_ops structure is quite weird. Simplify that by dropping the vendor specific init functions and the mtrr_ops[] array. Replace those with direct assignments of the related vendor specific ops array to mtrr_if. Note that a direct assignment is okay even for 64-bit builds, where the symbol isn't present, as the related code will be subject to "dead code elimination" due to how cpu_feature_enabled() is implemented. Signed-off-by: Juergen Gross --- V4: - new patch V5: - drop the vendor_mtrr_ops() macro (Borislav Petkov) --- arch/x86/kernel/cpu/mtrr/amd.c | 8 +------- arch/x86/kernel/cpu/mtrr/centaur.c | 8 +------- arch/x86/kernel/cpu/mtrr/cyrix.c | 8 +------- arch/x86/kernel/cpu/mtrr/mtrr.c | 30 +++--------------------------- arch/x86/kernel/cpu/mtrr/mtrr.h | 10 ++++------ 5 files changed, 10 insertions(+), 54 deletions(-) diff --git a/arch/x86/kernel/cpu/mtrr/amd.c b/arch/x86/kernel/cpu/mtrr/amd.c index a65a0272096d..eff6ac62c0ff 100644 --- a/arch/x86/kernel/cpu/mtrr/amd.c +++ b/arch/x86/kernel/cpu/mtrr/amd.c @@ -109,7 +109,7 @@ amd_validate_add_page(unsigned long base, unsigned long= size, unsigned int type) return 0; } =20 -static const struct mtrr_ops amd_mtrr_ops =3D { +const struct mtrr_ops amd_mtrr_ops =3D { .vendor =3D X86_VENDOR_AMD, .set =3D amd_set_mtrr, .get =3D amd_get_mtrr, @@ -117,9 +117,3 @@ static const struct mtrr_ops amd_mtrr_ops =3D { .validate_add_page =3D amd_validate_add_page, .have_wrcomb =3D positive_have_wrcomb, }; - -int __init amd_init_mtrr(void) -{ - set_mtrr_ops(&amd_mtrr_ops); - return 0; -} diff --git a/arch/x86/kernel/cpu/mtrr/centaur.c b/arch/x86/kernel/cpu/mtrr/= centaur.c index f27177816569..b8a74eddde83 100644 --- a/arch/x86/kernel/cpu/mtrr/centaur.c +++ b/arch/x86/kernel/cpu/mtrr/centaur.c @@ -111,7 +111,7 @@ centaur_validate_add_page(unsigned long base, unsigned = long size, unsigned int t return 0; } =20 -static const struct mtrr_ops centaur_mtrr_ops =3D { +const struct mtrr_ops centaur_mtrr_ops =3D { .vendor =3D X86_VENDOR_CENTAUR, .set =3D centaur_set_mcr, .get =3D centaur_get_mcr, @@ -119,9 +119,3 @@ static const struct mtrr_ops centaur_mtrr_ops =3D { .validate_add_page =3D centaur_validate_add_page, .have_wrcomb =3D positive_have_wrcomb, }; - -int __init centaur_init_mtrr(void) -{ - set_mtrr_ops(¢aur_mtrr_ops); - return 0; -} diff --git a/arch/x86/kernel/cpu/mtrr/cyrix.c b/arch/x86/kernel/cpu/mtrr/cy= rix.c index c77d3b0a5bf2..173b9e01e623 100644 --- a/arch/x86/kernel/cpu/mtrr/cyrix.c +++ b/arch/x86/kernel/cpu/mtrr/cyrix.c @@ -234,7 +234,7 @@ static void cyrix_set_arr(unsigned int reg, unsigned lo= ng base, post_set(); } =20 -static const struct mtrr_ops cyrix_mtrr_ops =3D { +const struct mtrr_ops cyrix_mtrr_ops =3D { .vendor =3D X86_VENDOR_CYRIX, .set =3D cyrix_set_arr, .get =3D cyrix_get_arr, @@ -242,9 +242,3 @@ static const struct mtrr_ops cyrix_mtrr_ops =3D { .validate_add_page =3D generic_validate_add_page, .have_wrcomb =3D positive_have_wrcomb, }; - -int __init cyrix_init_mtrr(void) -{ - set_mtrr_ops(&cyrix_mtrr_ops); - return 0; -} diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.c b/arch/x86/kernel/cpu/mtrr/mtr= r.c index 8403daf34158..6432abccbf56 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.c +++ b/arch/x86/kernel/cpu/mtrr/mtrr.c @@ -69,16 +69,8 @@ static DEFINE_MUTEX(mtrr_mutex); =20 u64 size_or_mask, size_and_mask; =20 -static const struct mtrr_ops *mtrr_ops[X86_VENDOR_NUM] __ro_after_init; - const struct mtrr_ops *mtrr_if; =20 -void __init set_mtrr_ops(const struct mtrr_ops *ops) -{ - if (ops->vendor && ops->vendor < X86_VENDOR_NUM) - mtrr_ops[ops->vendor] =3D ops; -} - /* Returns non-zero if we have the write-combining memory type */ static int have_wrcomb(void) { @@ -582,20 +574,6 @@ int arch_phys_wc_index(int handle) } EXPORT_SYMBOL_GPL(arch_phys_wc_index); =20 -/* - * HACK ALERT! - * These should be called implicitly, but we can't yet until all the initc= all - * stuff is done... - */ -static void __init init_ifs(void) -{ -#ifndef CONFIG_X86_64 - amd_init_mtrr(); - cyrix_init_mtrr(); - centaur_init_mtrr(); -#endif -} - /* The suspend/resume methods are only for CPU without MTRR. CPU using gen= eric * MTRR driver doesn't require this */ @@ -653,8 +631,6 @@ void __init mtrr_bp_init(void) { u32 phys_addr; =20 - init_ifs(); - phys_addr =3D 32; =20 if (boot_cpu_has(X86_FEATURE_MTRR)) { @@ -695,21 +671,21 @@ void __init mtrr_bp_init(void) case X86_VENDOR_AMD: if (cpu_feature_enabled(X86_FEATURE_K6_MTRR)) { /* Pre-Athlon (K6) AMD CPU MTRRs */ - mtrr_if =3D mtrr_ops[X86_VENDOR_AMD]; + mtrr_if =3D &amd_mtrr_ops; size_or_mask =3D SIZE_OR_MASK_BITS(32); size_and_mask =3D 0; } break; case X86_VENDOR_CENTAUR: if (cpu_feature_enabled(X86_FEATURE_CENTAUR_MCR)) { - mtrr_if =3D mtrr_ops[X86_VENDOR_CENTAUR]; + mtrr_if =3D ¢aur_mtrr_ops; size_or_mask =3D SIZE_OR_MASK_BITS(32); size_and_mask =3D 0; } break; case X86_VENDOR_CYRIX: if (cpu_feature_enabled(X86_FEATURE_CYRIX_ARR)) { - mtrr_if =3D mtrr_ops[X86_VENDOR_CYRIX]; + mtrr_if =3D &cyrix_mtrr_ops; size_or_mask =3D SIZE_OR_MASK_BITS(32); size_and_mask =3D 0; } diff --git a/arch/x86/kernel/cpu/mtrr/mtrr.h b/arch/x86/kernel/cpu/mtrr/mtr= r.h index c98928ceee6a..02eb5871492d 100644 --- a/arch/x86/kernel/cpu/mtrr/mtrr.h +++ b/arch/x86/kernel/cpu/mtrr/mtrr.h @@ -51,8 +51,6 @@ void fill_mtrr_var_range(unsigned int index, u32 base_lo, u32 base_hi, u32 mask_lo, u32 mask_hi); bool get_mtrr_state(void); =20 -extern void __init set_mtrr_ops(const struct mtrr_ops *ops); - extern u64 size_or_mask, size_and_mask; extern const struct mtrr_ops *mtrr_if; =20 @@ -66,10 +64,10 @@ void mtrr_state_warn(void); const char *mtrr_attrib_to_str(int x); void mtrr_wrmsr(unsigned, unsigned, unsigned); =20 -/* CPU specific mtrr init functions */ -int amd_init_mtrr(void); -int cyrix_init_mtrr(void); -int centaur_init_mtrr(void); +/* CPU specific mtrr_ops vectors. */ +extern const struct mtrr_ops amd_mtrr_ops; +extern const struct mtrr_ops cyrix_mtrr_ops; +extern const struct mtrr_ops centaur_mtrr_ops; =20 extern int changed_by_mtrr_cleanup; extern int mtrr_cleanup(unsigned address_bits); --=20 2.35.3