From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2246740EB9C; Tue, 1 Sep 2026 14:58:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274694; cv=none; b=e2IIekv8L77QswseacdhtJGoVdxjXJui6tKlohfny9+muNiccdiIHvsK9JYykexJk3pxrZ2cE/4YZZZ1R5ElbrBgrCaf5W/cG8uvEtcEwBpV8lydU+20ml9zDi0teFz8x9gblu1k2yyX3wZqIa7EMpvL5cZZnIox2e/mhuZqLcg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274694; c=relaxed/simple; bh=zH06ZKSgGhEhXr0gPfiGkd8DxI44+kr5wz8zEOUWQZk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TPsUY4dOw1d2yBYI5RYaRyKZ5pdxmXb0NxrYAHPtnufcE/X0h4oBwZH4uwXsidhe5xdJmrTgyRu5mHKP+/7ebbBzM2uraSmdRtb2I27o87CLS/SP6Ak6eVg49r+OVRnVXU9cPI0LW8XoaVMhr0TbhzU+x9CdwI/lcAFdr7N3RfU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cysZZLRF; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cysZZLRF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 945BD1F00A3A; Tue, 1 Sep 2026 14:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274692; bh=56BaSgitaCOqK+plEdyRoTXW75Ul7dUQbsnofeonq2Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=cysZZLRFOedPPPO//ebfnqWoUAq3ACVHOVowkBLYsRGJLs6dfZ4GNpCIgWF8XKMl1 +j2GuQzQOQYK2TfKBNUBNFBBBDAome86zQr8Y51X0jKUXLbqT7cUPA5HtWWPlVeWYp q9mPHrT1s2gFffWPqNKVAgQzr/VaTwoUwNWRsBSMphHqgRR6Nr3HvGEVWWc4XryRgv MPKfD0cf/YATfgKkrTbxcTyyJeOhB37+rZm0meKJhRjmx/HJ7FE+qIGsBHR0i1j4Er woSJpjWbnwlL67kE3yJx7iy9nBBIo5e9Cggk0yhLyETt1PzFjhw+8uWH6XGUw+Nn7d Nd9Xn71cRSTxA== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 1/8] x86/fpu: Check for missing AVX and AVX-512 xstate bits Date: Tue, 1 Sep 2026 07:57:33 -0700 Message-ID: <20260901145740.42337-2-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the CPU declares AVX or AVX-512 support, verify that the corresponding xstate bits are also set. If not, warn and clear them. This eliminates the perceived need for AVX and AVX-512 optimized code in the kernel to call cpu_has_xfeatures(). That has never been universally done, which strongly suggests that it has never really been needed in practice, but this should remove any remaining doubt. Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- arch/x86/kernel/fpu/xstate.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index a7b6524a9dea..97cfd4fb6cc0 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -806,7 +806,7 @@ static u64 __init guest_default_mask(void) void __init fpu__init_system_xstate(unsigned int legacy_size) { unsigned int eax, ebx, ecx, edx; - u64 xfeatures; + u64 xfeatures, mask; int err; int i; =20 @@ -818,6 +818,8 @@ void __init fpu__init_system_xstate(unsigned int legacy= _size) if (!boot_cpu_has(X86_FEATURE_XSAVE)) { pr_info("x86/fpu: x87 FPU will use %s\n", boot_cpu_has(X86_FEATURE_FXSR) ? "FXSAVE" : "FSAVE"); + /* Disable all dependent flags too */ + setup_clear_cpu_cap(X86_FEATURE_XSAVE); return; } =20 @@ -833,7 +835,8 @@ void __init fpu__init_system_xstate(unsigned int legacy= _size) cpuid_count(CPUID_LEAF_XSTATE, 1, &eax, &ebx, &ecx, &edx); fpu_kernel_cfg.max_features |=3D ecx + ((u64)edx << 32); =20 - if ((fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSSE) !=3D XFEATURE_MAS= K_FPSSE) { + mask =3D XFEATURE_MASK_FPSSE; + if ((fpu_kernel_cfg.max_features & mask) !=3D mask) { /* * This indicates that something really unexpected happened * with the enumeration. Disable XSAVE and try to continue @@ -844,6 +847,24 @@ void __init fpu__init_system_xstate(unsigned int legac= y_size) goto out_disable; } =20 + mask |=3D XFEATURE_MASK_YMM; + if (boot_cpu_has(X86_FEATURE_AVX)) { + if ((fpu_kernel_cfg.max_features & mask) !=3D mask) { + pr_err(FW_BUG + "x86/fpu: Disabling AVX support due to missing xstate features\n= "); + setup_clear_cpu_cap(X86_FEATURE_AVX); + } + } + + mask |=3D XFEATURE_MASK_AVX512; + if (boot_cpu_has(X86_FEATURE_AVX512F)) { + if ((fpu_kernel_cfg.max_features & mask) !=3D mask) { + pr_err(FW_BUG + "x86/fpu: Disabling AVX-512 support due to missing xstate featur= es\n"); + setup_clear_cpu_cap(X86_FEATURE_AVX512F); + } + } + if (fpu_kernel_cfg.max_features & XFEATURE_MASK_APX && fpu_kernel_cfg.max_features & (XFEATURE_MASK_BNDREGS | XFEATURE_MASK_= BNDCSR)) { /* --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86AA837C10C; Tue, 1 Sep 2026 14:58:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274694; cv=none; b=uKwfDShxM8BMPycVib6nFUWwfGm45hEkWDJJONt174qH57bRx8BDpVyqyE37JozfaL017x/2vldgHukJx7Cd0JnotSOJCaO9wBOgtcJNefioFVRgbwQKKXDVte7WTbI5EacGZWX5rM4jVBEYXzv3TdfwfzlGe7nW53IMAA4sq3E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274694; c=relaxed/simple; bh=Qt2g2f/NZEH5KtGE2aMVUqWOlU4GnccZTM2AkpbZgCo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lNpKNoEOvId1L+tw9MZBt50RzCrWbhdovi1tTwv5atRT9Z2Z78XZ2oTF1u2ElOCP+8wc20tNwz88NFGqSKbNh466hJhs7Pfnecb27EMJt1QAEok4WwetOAai6I0zF33ggta2emCOWSWc9YyF/7xh6pIQSh5JbLS2O2Yk1KAadCc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LbmV/iL1; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LbmV/iL1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F2C491F00A3E; Tue, 1 Sep 2026 14:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274693; bh=GjHnvhTUIuOR14d0OFZbby0ThaLZzSviDPcvynF6ARQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=LbmV/iL1gmV5lbQzrEMjAHuwjKpDYu0O8yaAsMGyAEJLPpqDGy7IvC4HA4g5B2PKf 2VWwCKAOhGUW7gGkejv/Acnva8zIS1wdH2e8wV5f1OGIdilBPyBOagxCL+LUzT5EQw ni3ApPIWXnImGpAaOd8UN5+ecRYLYIIotuihFZZ0p2q68nY2qPxEQOb7pZ5rr6U7E1 lDJ5Jd3YB3MuxJ8TBaRWNzI+UrcVoJajygPSHchEDMyTUyyrk/o5VVm5QVyUcCZD0/ Y1EZipLsaqlWWTTwGbKlFsyPRlc9G39gJa7EdyXOXy2Y+41FoI3zMjiSZpE/tCz4Gc 5//xo9ymGV/Zw== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 2/8] um: Check for missing AVX and AVX-512 xstate bits Date: Tue, 1 Sep 2026 07:57:34 -0700 Message-ID: <20260901145740.42337-3-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the CPU declares AVX or AVX-512 support, verify that all the corresponding bits are set in the XCR0 register. If any are missing, warn and don't set the corresponding X86_FEATURE_* flags. This eliminates the perceived need for UML-supporting AVX and AVX-512 optimized code in the kernel (that is, lib/raid/ currently) to start checking the XCR0 bits in addition to X86_FEATURE_AVX*. This aligns UML with the vast majority of userspace programs, which check the XCR0 bits before considering AVX and AVX-512 to be supported, as per the procedure documented in Intel's CPU manual. Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- arch/um/kernel/um_arch.c | 78 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 77 insertions(+), 1 deletion(-) diff --git a/arch/um/kernel/um_arch.c b/arch/um/kernel/um_arch.c index e4ee693961e4..9d53cc6a783d 100644 --- a/arch/um/kernel/um_arch.c +++ b/arch/um/kernel/um_arch.c @@ -264,12 +264,88 @@ unsigned long brk_start; =20 #define MIN_VMALLOC (32 * 1024 * 1024) =20 +static u64 __init read_xcr0(void) +{ + u32 a, b, c, d; + + asm volatile("cpuid" + : "=3Da"(a), "=3Db"(b), "=3Dc"(c), "=3Dd"(d) + : "a"(0), "c"(0)); + if (a >=3D 1) { /* max_leaf >=3D 1 */ + asm volatile("cpuid" + : "=3Da"(a), "=3Db"(b), "=3Dc"(c), "=3Dd"(d) + : "a"(1), "c"(0)); + if (c & (1 << 27)) { /* XSAVE enabled by OS */ + asm volatile("xgetbv" : "=3Dd"(d), "=3Da"(a) : "c"(0)); + return ((u64)d << 32) | a; + } + } + return 0; +} + +static void __init validate_and_set_cpu_cap(int cap, u64 xcr0) +{ + /* + * Check for missing xstate features right away, so that there's no + * perceived need for all optimized code in the kernel to do so. + */ + switch (cap) { + case X86_FEATURE_AVX: + case X86_FEATURE_AVX2: + case X86_FEATURE_AVX_VNNI: + case X86_FEATURE_FMA: + case X86_FEATURE_VAES: + case X86_FEATURE_VPCLMULQDQ: + if ((xcr0 & 0x7) !=3D 0x7) { + static bool warned; + + if (!warned) { + os_warn("Disabling AVX support due to missing xstate features\n"); + warned =3D true; + } + return; + } + break; + case X86_FEATURE_AVX512F: + case X86_FEATURE_AVX512BW: + case X86_FEATURE_AVX512CD: + case X86_FEATURE_AVX512DQ: + case X86_FEATURE_AVX512ER: + case X86_FEATURE_AVX512IFMA: + case X86_FEATURE_AVX512PF: + case X86_FEATURE_AVX512VBMI: + case X86_FEATURE_AVX512VL: + case X86_FEATURE_AVX512_4FMAPS: + case X86_FEATURE_AVX512_4VNNIW: + case X86_FEATURE_AVX512_BF16: + case X86_FEATURE_AVX512_BITALG: + case X86_FEATURE_AVX512_FP16: + case X86_FEATURE_AVX512_VBMI2: + case X86_FEATURE_AVX512_VNNI: + case X86_FEATURE_AVX512_VP2INTERSECT: + case X86_FEATURE_AVX512_VPOPCNTDQ: + if ((xcr0 & 0xe7) !=3D 0xe7) { + static bool warned; + + if (!warned) { + os_warn("Disabling AVX-512 support due to missing xstate features\n"); + warned =3D true; + } + return; + } + break; + } + set_cpu_cap(&boot_cpu_data, cap); +} + static void __init parse_host_cpu_flags(char *line) { + u64 xcr0 =3D read_xcr0(); int i; + for (i =3D 0; i < 32*NCAPINTS; i++) { if ((x86_cap_flags[i] !=3D NULL) && strstr(line, x86_cap_flags[i])) - set_cpu_cap(&boot_cpu_data, i); + validate_and_set_cpu_cap(i, xcr0); } } =20 --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B314B40F754; Tue, 1 Sep 2026 14:58:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274695; cv=none; b=uLF9WE32Atx/rKuVWcd3yrw1DqulPHSa7uYm9TunUK6gCDPxzSZRXIWRSRCr/7H557okoQuF78Fbbi/+toJUcxpT02eUSyGzQsNhhGRTaS6MdC++c/kaCEPidf4U/UXdgm5cXXcIv3YUzTv+QkggANAcLBuU8iNzxlQYbnr9wbo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274695; c=relaxed/simple; bh=j71FQSXxVyY13IVMc20ZI/TX8wDr4Hvb0YpK4t8lc9s=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gBNMHs4x49XwtHELdlQG6wRozZ5AccAYF5oTR8Q2awMUe+wOwY5Ch4pWLAEFO5gHfe4d0oDDUF6xni6d/Anw2GGibcplam9rzPA5b/wnwn+BVM4cu5bEGqcJgbGdShU2CbkCXTe5JOL5EeOjclFd2QaXUr/LbE/OMyPNeV4//4w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SKFEdjAl; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SKFEdjAl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5D2811F00A3D; Tue, 1 Sep 2026 14:58:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274693; bh=VKmmGWRx3Pfbo+nD8y5eQpPaN0sWZSIp/Ijjeyx4vLE=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=SKFEdjAlCXoEZ0cIslffOQKP9bgXvzTqVFqpza/wGsnzrxkLJhQ7Fy6jC1JbjXGR5 qahjv6fTi9IoseAa+HVxKGxhLBxHcS8HS8X3aS9O/eYC+ubhpS+iEYAwhx/wljOi5m 0OILfM/WbVWtY6I43xKDHS9oTaWaV9xlmklPUWDnDzjBu4c21QWtEf3cS1QIM2j1mg zhWDX0tCrF74BKmTKQ6q0fDBhgwc1XSvFoOjQ4JVuHcPvQBs8dYpE2HzU/cStXUfRD JtXquemPo9RoPLqGG5mDwvFfNXaB0x3Ba1MPfaKyPhF2Da+aHNQJqjzRbioKQPAHi9 TrJjTatjvEdRw== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 3/8] crypto: x86 - Stop using cpu_has_xfeatures() Date: Tue, 1 Sep 2026 07:57:35 -0700 Message-ID: <20260901145740.42337-4-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Checking both boot_cpu_has(X86_FEATURE_AVX*) and cpu_has_xfeatures() has never really been needed in practice, and it's never been universally done (e.g., lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both x86 and UML now explicitly clear the AVX and AVX-512 flags if their xfeatures are missing, which should remove any remaining doubts. Thus, remove all the calls to cpu_has_xfeatures(), as well as the related checks of boot_cpu_has(X86_FEATURE_OSXSAVE). In a few cases there was no corresponding boot_cpu_has(X86_FEATURE_AVX*) check, so add the missing ones. Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- arch/x86/crypto/aegis128-aesni-glue.c | 3 +-- arch/x86/crypto/aesni-intel_glue.c | 7 ++----- arch/x86/crypto/aria_aesni_avx2_glue.c | 11 +---------- arch/x86/crypto/aria_aesni_avx_glue.c | 11 +---------- arch/x86/crypto/aria_gfni_avx512_glue.c | 11 +---------- arch/x86/crypto/camellia_aesni_avx2_glue.c | 11 +---------- arch/x86/crypto/camellia_aesni_avx_glue.c | 11 +---------- arch/x86/crypto/cast5_avx_glue.c | 7 ++----- arch/x86/crypto/cast6_avx_glue.c | 7 ++----- arch/x86/crypto/serpent_avx2_glue.c | 9 +-------- arch/x86/crypto/serpent_avx_glue.c | 7 ++----- arch/x86/crypto/sm4_aesni_avx2_glue.c | 11 +---------- arch/x86/crypto/sm4_aesni_avx_glue.c | 11 +---------- arch/x86/crypto/twofish_avx_glue.c | 6 ++---- 14 files changed, 19 insertions(+), 104 deletions(-) diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis1= 28-aesni-glue.c index f1adfba1a76e..09fc0b15b0e9 100644 --- a/arch/x86/crypto/aegis128-aesni-glue.c +++ b/arch/x86/crypto/aegis128-aesni-glue.c @@ -265,8 +265,7 @@ static struct aead_alg crypto_aegis128_aesni_alg =3D { static int __init crypto_aegis128_aesni_module_init(void) { if (!boot_cpu_has(X86_FEATURE_XMM4_1) || - !boot_cpu_has(X86_FEATURE_AES) || - !cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL)) + !boot_cpu_has(X86_FEATURE_AES)) return -ENODEV; =20 return crypto_register_aead(&crypto_aegis128_aesni_alg); diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-int= el_glue.c index f522fff9231e..f6f899db7482 100644 --- a/arch/x86/crypto/aesni-intel_glue.c +++ b/arch/x86/crypto/aesni-intel_glue.c @@ -1548,8 +1548,7 @@ static int __init register_avx_algs(void) if (!boot_cpu_has(X86_FEATURE_AVX2) || !boot_cpu_has(X86_FEATURE_VAES) || !boot_cpu_has(X86_FEATURE_VPCLMULQDQ) || - !boot_cpu_has(X86_FEATURE_PCLMULQDQ) || - !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) + !boot_cpu_has(X86_FEATURE_PCLMULQDQ)) return 0; err =3D crypto_register_skciphers(skcipher_algs_vaes_avx2, ARRAY_SIZE(skcipher_algs_vaes_avx2)); @@ -1562,9 +1561,7 @@ static int __init register_avx_algs(void) =20 if (!boot_cpu_has(X86_FEATURE_AVX512BW) || !boot_cpu_has(X86_FEATURE_AVX512VL) || - !boot_cpu_has(X86_FEATURE_BMI2) || - !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | - XFEATURE_MASK_AVX512, NULL)) + !boot_cpu_has(X86_FEATURE_BMI2)) return 0; =20 if (boot_cpu_has(X86_FEATURE_PREFER_YMM)) { diff --git a/arch/x86/crypto/aria_aesni_avx2_glue.c b/arch/x86/crypto/aria_= aesni_avx2_glue.c index 1487a49bfbac..371be2fb6469 100644 --- a/arch/x86/crypto/aria_aesni_avx2_glue.c +++ b/arch/x86/crypto/aria_aesni_avx2_glue.c @@ -195,22 +195,13 @@ static struct skcipher_alg aria_algs[] =3D { =20 static int __init aria_avx2_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || !boot_cpu_has(X86_FEATURE_AVX2) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX2 or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - if (boot_cpu_has(X86_FEATURE_GFNI)) { aria_ops.aria_encrypt_16way =3D aria_aesni_avx_gfni_encrypt_16way; aria_ops.aria_decrypt_16way =3D aria_aesni_avx_gfni_decrypt_16way; diff --git a/arch/x86/crypto/aria_aesni_avx_glue.c b/arch/x86/crypto/aria_a= esni_avx_glue.c index e4e3d78915a5..d23fc91c0ebd 100644 --- a/arch/x86/crypto/aria_aesni_avx_glue.c +++ b/arch/x86/crypto/aria_aesni_avx_glue.c @@ -182,21 +182,12 @@ static struct skcipher_alg aria_algs[] =3D { =20 static int __init aria_avx_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - if (boot_cpu_has(X86_FEATURE_GFNI)) { aria_ops.aria_encrypt_16way =3D aria_aesni_avx_gfni_encrypt_16way; aria_ops.aria_decrypt_16way =3D aria_aesni_avx_gfni_decrypt_16way; diff --git a/arch/x86/crypto/aria_gfni_avx512_glue.c b/arch/x86/crypto/aria= _gfni_avx512_glue.c index 363cbf4399cc..e05bbeb22d4a 100644 --- a/arch/x86/crypto/aria_gfni_avx512_glue.c +++ b/arch/x86/crypto/aria_gfni_avx512_glue.c @@ -196,24 +196,15 @@ static struct skcipher_alg aria_algs[] =3D { =20 static int __init aria_avx512_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || !boot_cpu_has(X86_FEATURE_AVX2) || !boot_cpu_has(X86_FEATURE_AVX512F) || !boot_cpu_has(X86_FEATURE_AVX512VL) || - !boot_cpu_has(X86_FEATURE_GFNI) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_GFNI)) { pr_info("AVX512/GFNI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | - XFEATURE_MASK_AVX512, &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - aria_ops.aria_encrypt_16way =3D aria_aesni_avx_gfni_encrypt_16way; aria_ops.aria_decrypt_16way =3D aria_aesni_avx_gfni_decrypt_16way; aria_ops.aria_ctr_crypt_16way =3D aria_aesni_avx_gfni_ctr_crypt_16way; diff --git a/arch/x86/crypto/camellia_aesni_avx2_glue.c b/arch/x86/crypto/c= amellia_aesni_avx2_glue.c index 2d2f4e16537c..073fa3bb8388 100644 --- a/arch/x86/crypto/camellia_aesni_avx2_glue.c +++ b/arch/x86/crypto/camellia_aesni_avx2_glue.c @@ -97,22 +97,13 @@ static struct skcipher_alg camellia_algs[] =3D { =20 static int __init camellia_aesni_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || !boot_cpu_has(X86_FEATURE_AVX2) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX2 or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - return crypto_register_skciphers(camellia_algs, ARRAY_SIZE(camellia_algs)); } diff --git a/arch/x86/crypto/camellia_aesni_avx_glue.c b/arch/x86/crypto/ca= mellia_aesni_avx_glue.c index 5c321f255eb7..872e5e07220f 100644 --- a/arch/x86/crypto/camellia_aesni_avx_glue.c +++ b/arch/x86/crypto/camellia_aesni_avx_glue.c @@ -98,21 +98,12 @@ static struct skcipher_alg camellia_algs[] =3D { =20 static int __init camellia_aesni_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - return crypto_register_skciphers(camellia_algs, ARRAY_SIZE(camellia_algs)); } diff --git a/arch/x86/crypto/cast5_avx_glue.c b/arch/x86/crypto/cast5_avx_g= lue.c index 3aca04d43b34..5de35e863370 100644 --- a/arch/x86/crypto/cast5_avx_glue.c +++ b/arch/x86/crypto/cast5_avx_glue.c @@ -92,11 +92,8 @@ static struct skcipher_alg cast5_algs[] =3D { =20 static int __init cast5_init(void) { - const char *feature_name; - - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); + if (!boot_cpu_has(X86_FEATURE_AVX)) { + pr_info("AVX instructions are not detected.\n"); return -ENODEV; } =20 diff --git a/arch/x86/crypto/cast6_avx_glue.c b/arch/x86/crypto/cast6_avx_g= lue.c index c4dd28c30303..3d7ea48007bc 100644 --- a/arch/x86/crypto/cast6_avx_glue.c +++ b/arch/x86/crypto/cast6_avx_glue.c @@ -92,11 +92,8 @@ static struct skcipher_alg cast6_algs[] =3D { =20 static int __init cast6_init(void) { - const char *feature_name; - - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); + if (!boot_cpu_has(X86_FEATURE_AVX)) { + pr_info("AVX instructions are not detected.\n"); return -ENODEV; } =20 diff --git a/arch/x86/crypto/serpent_avx2_glue.c b/arch/x86/crypto/serpent_= avx2_glue.c index f5f2121b7956..72a9e2b306d6 100644 --- a/arch/x86/crypto/serpent_avx2_glue.c +++ b/arch/x86/crypto/serpent_avx2_glue.c @@ -93,17 +93,10 @@ static struct skcipher_alg serpent_algs[] =3D { =20 static int __init serpent_avx2_init(void) { - const char *feature_name; - - if (!boot_cpu_has(X86_FEATURE_AVX2) || !boot_cpu_has(X86_FEATURE_OSXSAVE)= ) { + if (!boot_cpu_has(X86_FEATURE_AVX2)) { pr_info("AVX2 instructions are not detected.\n"); return -ENODEV; } - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } =20 return crypto_register_skciphers(serpent_algs, ARRAY_SIZE(serpent_algs)); diff --git a/arch/x86/crypto/serpent_avx_glue.c b/arch/x86/crypto/serpent_a= vx_glue.c index 9c8b3a335d5c..42c4e1569674 100644 --- a/arch/x86/crypto/serpent_avx_glue.c +++ b/arch/x86/crypto/serpent_avx_glue.c @@ -100,11 +100,8 @@ static struct skcipher_alg serpent_algs[] =3D { =20 static int __init serpent_init(void) { - const char *feature_name; - - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); + if (!boot_cpu_has(X86_FEATURE_AVX)) { + pr_info("AVX instructions are not detected.\n"); return -ENODEV; } =20 diff --git a/arch/x86/crypto/sm4_aesni_avx2_glue.c b/arch/x86/crypto/sm4_ae= sni_avx2_glue.c index fec0ab7a63dd..eef73894e777 100644 --- a/arch/x86/crypto/sm4_aesni_avx2_glue.c +++ b/arch/x86/crypto/sm4_aesni_avx2_glue.c @@ -98,22 +98,13 @@ static struct skcipher_alg sm4_aesni_avx2_skciphers[] = =3D { =20 static int __init sm4_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || !boot_cpu_has(X86_FEATURE_AVX2) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX2 or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - return crypto_register_skciphers(sm4_aesni_avx2_skciphers, ARRAY_SIZE(sm4_aesni_avx2_skciphers)); } diff --git a/arch/x86/crypto/sm4_aesni_avx_glue.c b/arch/x86/crypto/sm4_aes= ni_avx_glue.c index 88caf418a06f..ed383da5ff46 100644 --- a/arch/x86/crypto/sm4_aesni_avx_glue.c +++ b/arch/x86/crypto/sm4_aesni_avx_glue.c @@ -314,21 +314,12 @@ static struct skcipher_alg sm4_aesni_avx_skciphers[] = =3D { =20 static int __init sm4_init(void) { - const char *feature_name; - if (!boot_cpu_has(X86_FEATURE_AVX) || - !boot_cpu_has(X86_FEATURE_AES) || - !boot_cpu_has(X86_FEATURE_OSXSAVE)) { + !boot_cpu_has(X86_FEATURE_AES)) { pr_info("AVX or AES-NI instructions are not detected.\n"); return -ENODEV; } =20 - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - &feature_name)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); - return -ENODEV; - } - return crypto_register_skciphers(sm4_aesni_avx_skciphers, ARRAY_SIZE(sm4_aesni_avx_skciphers)); } diff --git a/arch/x86/crypto/twofish_avx_glue.c b/arch/x86/crypto/twofish_a= vx_glue.c index 9e20db013750..985bc54a2340 100644 --- a/arch/x86/crypto/twofish_avx_glue.c +++ b/arch/x86/crypto/twofish_avx_glue.c @@ -102,10 +102,8 @@ static struct skcipher_alg twofish_algs[] =3D { =20 static int __init twofish_init(void) { - const char *feature_name; - - if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, &feature_na= me)) { - pr_info("CPU feature '%s' is not supported.\n", feature_name); + if (!boot_cpu_has(X86_FEATURE_AVX)) { + pr_info("AVX instructions are not detected.\n"); return -ENODEV; } =20 --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EFA23C3789; Tue, 1 Sep 2026 14:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; cv=none; b=UvFgwx+y2Ro19Ub6yEZWr2EYOzO0uf4cZMlmSygzbMUtdm5RYMFwgGxIhlAus8itoBo2Cbc3oDl0cURo5VqXAcLpGuVP3mDG8JpLJhhMG5Ol+5fdlJ6D1GY6AxUc7eZck9zr2Wev61fvM/RmAAbCILgXYDhiKjua04s1Tkz4m0g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; c=relaxed/simple; bh=e0/5EyKvRR38AkedRTxpFtVqo23cOjcvx5sQexInGDU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NEzK71QLnUWhIbQhahgMlBrSv67jAiQfkr9InofBVoKEBaSTEW5X6Xx140qIA2JENj8na/MNh6inviYrKw35L7gypDFOVS4m5arGOfiIiS0qpvdaaydjwGrwkbagq0p2eeDUFW99ozFZwgr+gmYlndvTsAenjH7QUxJjwJTwCFs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UdUkd4qI; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UdUkd4qI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BC3671F00ACA; Tue, 1 Sep 2026 14:58:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274694; bh=DdqWq0+mPRqIC9aeO4PAgts2wuJAHghXz/rr2+6DDJI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=UdUkd4qI1dN9Ut6fdtoqtGveQA4rWJtp8L5fQwXWpRW9MrbcxDxKHc/kCGp9Vqk8i vQREACmTP3DT6zLtbDBbj/BUliDgZ+KywMEiu15ZpOoxWaFPZJabsjy2Ur7s8qNDyr Jbb0CDn0lZ5wM8mneU30hhyJHk8TakKyBbTpB2zLUZQJYPxmes/NTWHk+FwYui9oOq sg7AauP0fGBQsx9+1eUv4Cjw4c+L9uMYzuxbjnEA7s1boKMdusih7K0VH2JpqIgnbl /bqoqOYoNXk2lespVepAPgOe/du7hngHIP4aEebxU4QFBIyqCw1LVEOChZ+6jrHSiK l6RWQSp/Rn4zg== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 4/8] lib/crypto: x86: Stop using cpu_has_xfeatures() Date: Tue, 1 Sep 2026 07:57:36 -0700 Message-ID: <20260901145740.42337-5-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Checking both boot_cpu_has() and cpu_has_xfeatures() has never really been needed in practice, and it's never been universally done (e.g., lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both x86 and UML now explicitly clear the AVX and AVX-512 flags if their xfeatures are missing, which should remove any remaining doubts. Thus, remove all the calls to cpu_has_xfeatures(). Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- lib/crypto/x86/blake2s.h | 4 +--- lib/crypto/x86/chacha.h | 3 +-- lib/crypto/x86/nh.h | 4 +--- lib/crypto/x86/poly1305.h | 7 ++----- lib/crypto/x86/sha1.h | 4 +--- lib/crypto/x86/sha256.h | 4 +--- lib/crypto/x86/sha512.h | 3 +-- lib/crypto/x86/sm3.h | 3 +-- 8 files changed, 9 insertions(+), 23 deletions(-) diff --git a/lib/crypto/x86/blake2s.h b/lib/crypto/x86/blake2s.h index f8eed6cb042e..0f7c51f055c8 100644 --- a/lib/crypto/x86/blake2s.h +++ b/lib/crypto/x86/blake2s.h @@ -55,8 +55,6 @@ static void blake2s_mod_init_arch(void) if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) && boot_cpu_has(X86_FEATURE_AVX512F) && - boot_cpu_has(X86_FEATURE_AVX512VL) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | - XFEATURE_MASK_AVX512, NULL)) + boot_cpu_has(X86_FEATURE_AVX512VL)) static_branch_enable(&blake2s_use_avx512); } diff --git a/lib/crypto/x86/chacha.h b/lib/crypto/x86/chacha.h index 10cf8f1c569d..c79562aac56b 100644 --- a/lib/crypto/x86/chacha.h +++ b/lib/crypto/x86/chacha.h @@ -165,8 +165,7 @@ static void chacha_mod_init_arch(void) static_branch_enable(&chacha_use_simd); =20 if (boot_cpu_has(X86_FEATURE_AVX) && - boot_cpu_has(X86_FEATURE_AVX2) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) { + boot_cpu_has(X86_FEATURE_AVX2)) { static_branch_enable(&chacha_use_avx2); =20 if (boot_cpu_has(X86_FEATURE_AVX512VL) && diff --git a/lib/crypto/x86/nh.h b/lib/crypto/x86/nh.h index 83361c2e9783..342636dcb750 100644 --- a/lib/crypto/x86/nh.h +++ b/lib/crypto/x86/nh.h @@ -37,9 +37,7 @@ static void nh_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_XMM2)) { static_branch_enable(&have_sse2); - if (boot_cpu_has(X86_FEATURE_AVX2) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - NULL)) + if (boot_cpu_has(X86_FEATURE_AVX2)) static_branch_enable(&have_avx2); } } diff --git a/lib/crypto/x86/poly1305.h b/lib/crypto/x86/poly1305.h index ee92e3740a78..b061b9926fa5 100644 --- a/lib/crypto/x86/poly1305.h +++ b/lib/crypto/x86/poly1305.h @@ -143,15 +143,12 @@ static void poly1305_emit(const struct poly1305_state= *ctx, #define poly1305_mod_init_arch poly1305_mod_init_arch static void poly1305_mod_init_arch(void) { - if (boot_cpu_has(X86_FEATURE_AVX) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) + if (boot_cpu_has(X86_FEATURE_AVX)) static_branch_enable(&poly1305_use_avx); - if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) + if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2)) static_branch_enable(&poly1305_use_avx2); if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) && boot_cpu_has(X86_FEATURE_AVX512F) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | XFEATURE_MA= SK_AVX512, NULL) && /* Skylake downclocks unacceptably much when using zmm, but later gen= erations are fast. */ boot_cpu_data.x86_vfm !=3D INTEL_SKYLAKE_X) static_branch_enable(&poly1305_use_avx512); diff --git a/lib/crypto/x86/sha1.h b/lib/crypto/x86/sha1.h index c48a0131fd12..6aff433466e7 100644 --- a/lib/crypto/x86/sha1.h +++ b/lib/crypto/x86/sha1.h @@ -59,9 +59,7 @@ static void sha1_mod_init_arch(void) { if (boot_cpu_has(X86_FEATURE_SHA_NI)) { static_call_update(sha1_blocks_x86, sha1_blocks_ni); - } else if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - NULL) && - boot_cpu_has(X86_FEATURE_AVX)) { + } else if (boot_cpu_has(X86_FEATURE_AVX)) { if (boot_cpu_has(X86_FEATURE_AVX2) && boot_cpu_has(X86_FEATURE_BMI1) && boot_cpu_has(X86_FEATURE_BMI2)) diff --git a/lib/crypto/x86/sha256.h b/lib/crypto/x86/sha256.h index 0ee69d8e39fe..e98ffdaf4b14 100644 --- a/lib/crypto/x86/sha256.h +++ b/lib/crypto/x86/sha256.h @@ -104,9 +104,7 @@ static void sha256_mod_init_arch(void) boot_cpu_has(X86_FEATURE_PHE_EN) && boot_cpu_data.x86 >=3D 0x07) { static_call_update(sha256_blocks_x86, sha256_blocks_phe); - } else if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, - NULL) && - boot_cpu_has(X86_FEATURE_AVX)) { + } else if (boot_cpu_has(X86_FEATURE_AVX)) { if (boot_cpu_has(X86_FEATURE_AVX2) && boot_cpu_has(X86_FEATURE_BMI2)) static_call_update(sha256_blocks_x86, diff --git a/lib/crypto/x86/sha512.h b/lib/crypto/x86/sha512.h index 0213c70cedd0..4e177b4606bd 100644 --- a/lib/crypto/x86/sha512.h +++ b/lib/crypto/x86/sha512.h @@ -37,8 +37,7 @@ static void sha512_blocks(struct sha512_block_state *stat= e, #define sha512_mod_init_arch sha512_mod_init_arch static void sha512_mod_init_arch(void) { - if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL) && - boot_cpu_has(X86_FEATURE_AVX)) { + if (boot_cpu_has(X86_FEATURE_AVX)) { if (boot_cpu_has(X86_FEATURE_AVX2) && boot_cpu_has(X86_FEATURE_BMI2)) static_call_update(sha512_blocks_x86, diff --git a/lib/crypto/x86/sm3.h b/lib/crypto/x86/sm3.h index 3834780f2f6a..e06d4a22e4fa 100644 --- a/lib/crypto/x86/sm3.h +++ b/lib/crypto/x86/sm3.h @@ -33,7 +33,6 @@ static void sm3_blocks(struct sm3_block_state *state, #define sm3_mod_init_arch sm3_mod_init_arch static void sm3_mod_init_arch(void) { - if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_BMI2) && - cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) + if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_BMI2)) static_call_update(sm3_blocks_x86, sm3_blocks_avx); } --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA095412C0D; Tue, 1 Sep 2026 14:58:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; cv=none; b=MX/AwEjn4P3/VpBOrAPm3FQejIHjxdGAk3aTZszm2AeBrAe6/UK1jCC32KIWXlVLQksPZ0SY7A1FkyF2lmPTTE4Pf35Zhg8LCBTOvsz4bkX/ISgT1/JR99wvcJOla5Dxpcr1/En9MWX8pNs2wBRGy7fqun4zXgWnnX4CK9CznRk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; c=relaxed/simple; bh=8EtIjUyWp+Owf958VUqMsdagPce6VSl8WYVhBIP0kNU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=OjFkrE+MM3HWaWRk4aLrNhuz65SkaIcEefRS9pt8zO7DYqfaUj7TcxNtwiOn06xR071k96nVkyc1dd6cZGlq6+HevxTSTOrpcW+Lu/xuPv6SrJgOqZh4qXYXcYemLTt/5eHImlg6qnmGOVFULTyh7as8vziotMhGjIqHCh37oek= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YawVhUJA; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YawVhUJA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2676D1F00A3F; Tue, 1 Sep 2026 14:58:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274694; bh=r4lrJRkT6oIO9uz4byVKnhiX3ckXssiVruBvq7U+FfI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=YawVhUJAl4XJC+0lJg0JqnzvyLoGpclba5vFqMseu8aQdeAfRNCqF/n89C5BFnRn4 ZMgZnt00svTZ7gKmJhtJ04BvxTdeX+//rW59BNZmOABb8rXM8WE4tuFSRJOTY0O+a3 wzlzCcKF1wjNushRp8DgVQU/dXj39smX7CyBzSJH1YujqkwHIJgLd/5F0hTLEiXvwp rpILnJjIGoK4ZqXwRp1daZvUbnKwICqiVWNfsFNqVZML5vihbF6lJP71+QoQZ6nnn8 cGO6BziajnHnrbES/E803CCkdcHeqva1iZHlGmQ767J9u+QH407m0Ng6QB+EJcNiWk vNW9n5eYCeh4w== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 5/8] lib/crc: x86: Stop using cpu_has_xfeatures() Date: Tue, 1 Sep 2026 07:57:37 -0700 Message-ID: <20260901145740.42337-6-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Checking both boot_cpu_has() and cpu_has_xfeatures() has never really been needed in practice, and it's never been universally done (e.g., lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both x86 and UML now explicitly clear the AVX and AVX-512 flags if their xfeatures are missing, which should remove any remaining doubts. Thus, remove all the calls to cpu_has_xfeatures(). Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- lib/crc/x86/crc-pclmul-template.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/crc/x86/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-tem= plate.h index 02744831c6fa..893119bb7c07 100644 --- a/lib/crc/x86/crc-pclmul-template.h +++ b/lib/crc/x86/crc-pclmul-template.h @@ -27,16 +27,14 @@ DEFINE_STATIC_CALL(prefix##_pclmul, prefix##_pclmul_sse) static inline bool have_vpclmul(void) { return boot_cpu_has(X86_FEATURE_VPCLMULQDQ) && - boot_cpu_has(X86_FEATURE_AVX2) && - cpu_has_xfeatures(XFEATURE_MASK_YMM, NULL); + boot_cpu_has(X86_FEATURE_AVX2); } =20 static inline bool have_avx512(void) { return boot_cpu_has(X86_FEATURE_AVX512BW) && boot_cpu_has(X86_FEATURE_AVX512VL) && - !boot_cpu_has(X86_FEATURE_PREFER_YMM) && - cpu_has_xfeatures(XFEATURE_MASK_AVX512, NULL); + !boot_cpu_has(X86_FEATURE_PREFER_YMM); } =20 /* --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A182415B66; Tue, 1 Sep 2026 14:58:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; cv=none; b=BCcRrNfEiTOeBCEYkAkSOr3Uc33KhkJiADsLB6+NixzbcPvQM0zzyxalHx5ROBpQfBds+NOJ8Bw02+DiqneJqgdTcG8fDgmjovJsexqEplm8WlY05ICUoxK4x0a7+DMDhT+OaOnZwCVS9+IADyQSDELT5dG9Of30vt5TGYvTESs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; c=relaxed/simple; bh=mF5Si+fJ2SqCKuU/60v4aWm6TNcVfLmMHZQC50LpG1A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a7vdot73plJAxwCE6mrHeMykC2nFz56e+9Uyc3JP1uvt8H/E1rymHIegv2rZrSDlpmv5KPP1UQHW42Kp/rCK58pLA6dhtKR38bpmS7GfxuUj2qIPig/pnjlmxHAtOz83yi1QF+ZnM60gdmxrJpFZxW9N0oGL2R312lpOq0OWRZI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eAdGQvLv; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eAdGQvLv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 84E4B1F00A3A; Tue, 1 Sep 2026 14:58:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274694; bh=Q2A0VpZ3jEW63u/fB0uqkGhkDFeSh86cNwd3FvE985o=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=eAdGQvLvBEMK0T92SDuF8l/6M5p3jjfGQ1hN/UMYv76y3i+Wq000AMtQ6EUekYuOt 45QGNhIJ0JeAK/p0IDamjgCUCSSu/nSkPUs8jo5L0gbqJ7ZmsBEZmI2Tx6tMCzOj9R lbMi1bKw0gT2OkWWcHdNbH1TwjaS6U7O/aK+qeAZUduNyWKy5974h5rfaDRTxBtniq omnBPr+wL94kYdYNermWA7DThOXr3zJecYbOP1qOzQm7WmN1QD7sxhr9zocnXhGLSI k/TTRbmSpbaDSzWohB6nGNSqXre71Bv0FlaR/Fs9CjVqZLlkhLYUUn/JNiAVy8jIIc 4NsBf2dqqOvhw== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 6/8] x86/fpu: Remove cpu_has_xfeatures() Date: Tue, 1 Sep 2026 07:57:38 -0700 Message-ID: <20260901145740.42337-7-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The only remaining caller of cpu_has_xfeatures() is print_xstate_features(), which uses it only to check and get the name of a single feature. Remove it and just inline the needed code into print_xstate_features(). This also makes the "unknown xstate feature" entry at index XFEATURE_MAX of xfeature_names[] unnecessary, so remove that too. Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- arch/x86/include/asm/fpu/api.h | 9 ------- arch/x86/kernel/fpu/xstate.c | 44 +++------------------------------- 2 files changed, 3 insertions(+), 50 deletions(-) diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index 90c63fe19c0f..cfed8b24d64f 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -99,15 +99,6 @@ static inline void fpregs_assert_state_consistent(void) = { } */ extern void switch_fpu_return(void); =20 -/* - * Query the presence of one or more xfeatures. Works on any legacy CPU as= well. - * - * If 'feature_name' is set then put a human-readable description of - * the feature there as well - this can be used to print error (or success) - * messages. - */ -extern int cpu_has_xfeatures(u64 xfeatures_mask, const char **feature_name= ); - /* Trap handling */ extern int fpu__exception_code(struct fpu *fpu, int trap_nr); extern void fpu_sync_fpstate(struct fpu *fpu); diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 97cfd4fb6cc0..0b17452d1788 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -66,8 +66,8 @@ static const char *xfeature_names[] =3D "AMX Tile config", "AMX Tile data", "APX registers", - "unknown xstate feature", }; +static_assert(ARRAY_SIZE(xfeature_names) =3D=3D XFEATURE_MAX); =20 static unsigned short xsave_cpuid_features[] __initdata =3D { [XFEATURE_FP] =3D X86_FEATURE_FPU, @@ -122,44 +122,6 @@ static inline unsigned int next_xfeature_order(unsigne= d int i, u64 mask) #define XSTATE_FLAG_SUPERVISOR BIT(0) #define XSTATE_FLAG_ALIGNED64 BIT(1) =20 -/* - * Return whether the system supports a given xfeature. - * - * Also return the name of the (most advanced) feature that the caller req= uested: - */ -int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name) -{ - u64 xfeatures_missing =3D xfeatures_needed & ~fpu_kernel_cfg.max_features; - - if (unlikely(feature_name)) { - long xfeature_idx, max_idx; - u64 xfeatures_print; - /* - * So we use FLS here to be able to print the most advanced - * feature that was requested but is missing. So if a driver - * asks about "XFEATURE_MASK_SSE | XFEATURE_MASK_YMM" we'll print the - * missing AVX feature - this is the most informative message - * to users: - */ - if (xfeatures_missing) - xfeatures_print =3D xfeatures_missing; - else - xfeatures_print =3D xfeatures_needed; - - xfeature_idx =3D fls64(xfeatures_print)-1; - max_idx =3D ARRAY_SIZE(xfeature_names)-1; - xfeature_idx =3D min(xfeature_idx, max_idx); - - *feature_name =3D xfeature_names[xfeature_idx]; - } - - if (xfeatures_missing) - return 0; - - return 1; -} -EXPORT_SYMBOL_GPL(cpu_has_xfeatures); - static bool xfeature_is_aligned64(int xfeature_nr) { return xstate_flags[xfeature_nr] & XSTATE_FLAG_ALIGNED64; @@ -302,9 +264,9 @@ static void __init print_xstate_features(void) =20 for (i =3D 0; i < XFEATURE_MAX; i++) { u64 mask =3D BIT_ULL(i); - const char *name; + const char *name =3D xfeature_names[i]; =20 - if (cpu_has_xfeatures(mask, &name)) + if (fpu_kernel_cfg.max_features & mask) pr_info("x86/fpu: Supporting XSAVE feature 0x%03Lx: '%s'\n", mask, name= ); } } --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 80501418A54; Tue, 1 Sep 2026 14:58:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; cv=none; b=TZSbQvMM1d5G9gwdfybuY+C5A5GnvIjvYEdNZ4+rvT+V+4La8gekhhLN/pePJBbOiLlKUHM2360dhiuZ/7DmEKwVkItK64F5nSWIrxsfPdsauuT/DFfPYR8u/icCGFaHCS36xSDCZlunniR6+8XgpuDK4tEoL+O66EjSoZ9Ue2U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274696; c=relaxed/simple; bh=bURSDeOHD7BeXhOUO95qSmh0YND+CXXzB5NqaBliqI4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Dn68j80ML1ioIKMLKgIAryxHiLrmnJBUGSxwKYO3lJ+FAJUtoYChGCAv1FHm5oEshgsSsX6Mt+giEV+y4jUiKeOUiwrEFrucy8F0CfVAjA6AyRiO931kpHlbQNrJmoCYZ61Uwp4UqCsh6eajvV5lSg+XdyrkYvJbXiY4i/AC8RQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VhSj3+Bk; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VhSj3+Bk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E31C61F000E9; Tue, 1 Sep 2026 14:58:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274695; bh=CGFmBktDb18Ou8jdzee9BRhN2jpwroPuYWGDnSphFl0=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=VhSj3+BkQtVSyy/0a/UXLwqRnLqlZyrvWJvJ4DMKYKWfTyH44WJT6jDdHbynwgnaD itgUcs3lXcrGXsGbeu20U2B2c8yr5dwKlpIrTcTIUg65IWx+T7wNL6jJ9fxhYw72t9 N0V+CEFK7uD5kXrBPhTbQ2AowtyQdCNIZabPOBnzEIgZL1k0LiwshRwTTVGmPOvlqf dkrgnah1pNjTN2z75fEFLOXowZHGWm/M9uf+2BQy3fPxIrRgBvmie5g/8AzFbWCpXr xIg0isQ71KxHd9h8LmQZ9X6qwtH2UaMIlFXJWJKIfVOPWviFWrQ4xP+iYFQPiJqyyV zyPDQqoaEEG2A== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers Subject: [PATCH v5 7/8] xor: Remove redundant X86_FEATURE_OSXSAVE check Date: Tue, 1 Sep 2026 07:57:39 -0700 Message-ID: <20260901145740.42337-8-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" X86_FEATURE_AVX implies X86_FEATURE_OSXSAVE already. Reviewed-by: Christoph Hellwig Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- lib/raid/xor/x86/xor_arch.h | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h index 99fe85a213c6..991abe3f4bbd 100644 --- a/lib/raid/xor/x86/xor_arch.h +++ b/lib/raid/xor/x86/xor_arch.h @@ -18,8 +18,7 @@ extern struct xor_block_template xor_block_avx; */ static __always_inline void __init arch_xor_init(void) { - if (boot_cpu_has(X86_FEATURE_AVX) && - boot_cpu_has(X86_FEATURE_OSXSAVE)) { + if (boot_cpu_has(X86_FEATURE_AVX)) { xor_force(&xor_block_avx); } else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) { xor_register(&xor_block_sse); --=20 2.55.0 From nobody Sat Sep 26 12:26:30 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C40FC41D135; Tue, 1 Sep 2026 14:58:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274697; cv=none; b=qbzcY7aE8g9IyTRD6XSEe6oN93K42+CPm6EZ441n2pHrvej5LaH60Ys0DwjwIaqo4SOohAPucIsEKDhpLlwTXMn65tG2SHeQ9ATWtlHZ5TvHlfnrQ2D1M0sI2F3b4hx/P09ouTqRLyHC+BUtdak9GSEV5G7WKgE5jU/KTwwbRz4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788274697; c=relaxed/simple; bh=0NZj8dzHMvDPafeShpBt1PLgjhP7gFYn0JXz851agAI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qVcxduYfeuBd6iy5Ay/NtrE9LR8b6SOdIdGqwhnX0YcMZVcAmjXUjbBReRfWaRvB0c08cJcJf0G6d1JSNlJXlPdn3S/x1rg4aIyj1d0WlUJ3X+a25TcD88+uTRvGCfLyIKtjqkvsoYWNoSVd7eaLh7M2WAVZLYbFLJrWi+xx9bo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fHmVoYtD; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fHmVoYtD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4CF741F00ADE; Tue, 1 Sep 2026 14:58:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1788274695; bh=UTvJDIw3wcg7xi4i0kyVSun0Dn6RIuyJL6Ze7BRUb5M=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=fHmVoYtD48si7U4QBauUocNTZRD1pcYnoGOdDynX3BhCHV/UgsIB+z2JK1nYO11t8 JVRGqHItgAbcTdxPsb66Wk+Ro3HzpCxFUkarQlX9yShbQVZAejZNohwd3QUzip7HTx kOlFQBv5nTbH304V+gsJvn+5WY3k++rpVpkiiOnLMYI3i/byofcDRs/VEYNnsetO46 AXWLkz3z1eMUD06SvLIF7oLZFGEIqd2oC54Tu7qs/JVJ81jzKMbS+H5YtJFCZe4Z03 OJi+j8iP4uy6xNa0eMsl4tyPxiNMtFDlv0RJU5uAGwo0iMO7kt5OmA+k/MUp8WvYfO ioQkgoguf72Lg== From: Eric Biggers To: x86@kernel.org Cc: linux-um@lists.infradead.org, linux-raid@vger.kernel.org, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Christoph Hellwig , Andrew Morton , Eric Biggers , David Laight Subject: [PATCH v5 8/8] xor: Add AVX-512 optimized xor_gen() Date: Tue, 1 Sep 2026 07:57:40 -0700 Message-ID: <20260901145740.42337-9-ebiggers@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260901145740.42337-1-ebiggers@kernel.org> References: <20260901145740.42337-1-ebiggers@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an implementation of xor_gen() using AVX-512. It uses 512-bit vectors, i.e. ZMM registers. It also uses the vpternlogq instruction to do three-input XORs when applicable. It's enabled on x86_64 CPUs that have AVX512F && !PREFER_YMM. In practice that means: - AMD Zen 4 and later (client and server) - Intel Sapphire Rapids and later (server) - Intel Rocket Lake (client) - Intel Nova Lake and later (client) The !PREFER_YMM condition excludes the older AVX-512 implementations in Intel Skylake Server and Intel Ice Lake. They could run this code, but they're known to have overly-eager downclocking when ZMM registers are used. This is the same policy that the crypto and CRC code uses. Results from xor_benchmark on AMD Ryzen 9 9950X (Zen 5): Before: # xor_benchmark: 4096 bytes 16384 bytes # xor_benchmark: 4 disks: 256 GB/s 185 GB/s # xor_benchmark: 5 disks: 233 GB/s 165 GB/s # xor_benchmark: 6 disks: 243 GB/s 174 GB/s # xor_benchmark: 7 disks: 244 GB/s 176 GB/s # xor_benchmark: 8 disks: 265 GB/s 179 GB/s # xor_benchmark: 10 disks: 249 GB/s 175 GB/s # xor_benchmark: 12 disks: 219 GB/s 173 GB/s # xor_benchmark: 15 disks: 206 GB/s 177 GB/s # xor_benchmark: 16 disks: 212 GB/s 180 GB/s # xor_benchmark: 32 disks: 193 GB/s 177 GB/s After: # xor_benchmark: 4096 bytes 16384 bytes # xor_benchmark: 4 disks: 492 GB/s 202 GB/s # xor_benchmark: 5 disks: 436 GB/s 191 GB/s # xor_benchmark: 6 disks: 466 GB/s 193 GB/s # xor_benchmark: 7 disks: 491 GB/s 194 GB/s # xor_benchmark: 8 disks: 513 GB/s 205 GB/s # xor_benchmark: 10 disks: 454 GB/s 199 GB/s # xor_benchmark: 12 disks: 330 GB/s 204 GB/s # xor_benchmark: 15 disks: 266 GB/s 198 GB/s # xor_benchmark: 16 disks: 243 GB/s 177 GB/s # xor_benchmark: 32 disks: 187 GB/s 165 GB/s Reviewed-by: David Laight Reviewed-by: Christoph Hellwig Acked-by: Christoph Hellwig Signed-off-by: Eric Biggers --- lib/raid/xor/Makefile | 2 +- lib/raid/xor/x86/xor-avx512.c | 122 ++++++++++++++++++++++++++++++++++ lib/raid/xor/x86/xor_arch.h | 30 ++++++--- 3 files changed, 143 insertions(+), 11 deletions(-) create mode 100644 lib/raid/xor/x86/xor-avx512.c diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile index 9b0fad459cdb..e1e3455c219d 100644 --- a/lib/raid/xor/Makefile +++ b/lib/raid/xor/Makefile @@ -31,7 +31,7 @@ xor-$(CONFIG_SPARC32) +=3D sparc/xor-sparc32.o xor-$(CONFIG_SPARC64) +=3D sparc/xor-sparc64.o sparc/xor-sparc64-glue.o xor-$(CONFIG_S390) +=3D s390/xor.o xor-$(CONFIG_X86_32) +=3D x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o -xor-$(CONFIG_X86_64) +=3D x86/xor-avx.o x86/xor-sse.o +xor-$(CONFIG_X86_64) +=3D x86/xor-avx.o x86/xor-sse.o x86/xor-avx512.o obj-y +=3D tests/ =20 CFLAGS_xor-neon.o +=3D $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) diff --git a/lib/raid/xor/x86/xor-avx512.c b/lib/raid/xor/x86/xor-avx512.c new file mode 100644 index 000000000000..c11d83441875 --- /dev/null +++ b/lib/raid/xor/x86/xor-avx512.c @@ -0,0 +1,122 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * AVX-512 optimized implementation of xor_gen() + * + * Copyright 2026 Google LLC + */ + +#include +#include +#include "xor_impl.h" +#include "xor_arch.h" + +/* + * Implementation notes: + * + * Unrolling by the number of buffers (2-5) is very important. + * + * Unrolling by length is less important, especially when using register-i= ndexed + * addressing with negative indices from the end of the buffers. That app= roach + * results in just two loop control instructions being needed per iteratio= n, + * regardless of the number of buffers. + * + * In fact, benchmarks showed that the 2 and 3 buffer cases require only 2x + * unrolling by length, while the 4 and 5 buffer cases don't require any + * unrolling by length. Benchmarks also showed that the register-indexed + * addressing isn't a bottleneck either; i.e., we can't do any better by + * incrementing the pointers as we go along, even with more unrolling. + */ + +static void xor_avx512_2(long bytes, u8 *p1, const u8 *p2) +{ + long i =3D -bytes; + + asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n" + "vmovdqa64 64(%1,%0), %%zmm1\n" + "vpxorq (%2,%0), %%zmm0, %%zmm0\n" + "vpxorq 64(%2,%0), %%zmm1, %%zmm1\n" + "vmovdqa64 %%zmm0, (%1,%0)\n" + "vmovdqa64 %%zmm1, 64(%1,%0)\n" + "add $128, %0\n" + "jnz 1b\n" + : "+&r"(i) + : "r"(p1 + bytes), "r"(p2 + bytes) + : "memory", "cc"); +} + +static void xor_avx512_3(long bytes, u8 *p1, const u8 *p2, const u8 *p3) +{ + long i =3D -bytes; + + asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n" + "vmovdqa64 64(%1,%0), %%zmm1\n" + "vmovdqa64 (%2,%0), %%zmm2\n" + "vmovdqa64 64(%2,%0), %%zmm3\n" + "vpternlogq $0x96, (%3,%0), %%zmm2, %%zmm0\n" + "vpternlogq $0x96, 64(%3,%0), %%zmm3, %%zmm1\n" + "vmovdqa64 %%zmm0, (%1,%0)\n" + "vmovdqa64 %%zmm1, 64(%1,%0)\n" + "add $128, %0\n" + "jnz 1b\n" + : "+&r"(i) + : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes) + : "memory", "cc"); +} + +static void xor_avx512_4(long bytes, u8 *p1, const u8 *p2, const u8 *p3, + const u8 *p4) +{ + long i =3D -bytes; + + asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n" + "vmovdqa64 (%2,%0), %%zmm1\n" + "vpxorq (%3,%0), %%zmm0, %%zmm0\n" + "vpternlogq $0x96, (%4,%0), %%zmm1, %%zmm0\n" + "vmovdqa64 %%zmm0, (%1,%0)\n" + "add $64, %0\n" + "jnz 1b\n" + : "+&r"(i) + : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes), + "r"(p4 + bytes) + : "memory", "cc"); +} + +static void xor_avx512_5(long bytes, u8 *p1, const u8 *p2, const u8 *p3, + const u8 *p4, const u8 *p5) +{ + long i =3D -bytes; + + asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n" + "vmovdqa64 (%2,%0), %%zmm1\n" + "vpternlogq $0x96, (%3,%0), %%zmm1, %%zmm0\n" + "vmovdqa64 (%4,%0), %%zmm1\n" + "vpternlogq $0x96, (%5,%0), %%zmm1, %%zmm0\n" + "vmovdqa64 %%zmm0, (%1,%0)\n" + "add $64, %0\n" + "jnz 1b\n" + : "+&r"(i) + : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes), + "r"(p4 + bytes), "r"(p5 + bytes) + : "memory", "cc"); +} + +DO_XOR_BLOCKS(avx512_inner, xor_avx512_2, xor_avx512_3, xor_avx512_4, + xor_avx512_5); + +/* + * Preconditions: bytes is a nonzero multiple of 512, and all buffers are + * 64-byte aligned. + */ +static void xor_gen_avx512(void *dest, void **srcs, unsigned int src_cnt, + unsigned int bytes) +{ + kernel_fpu_begin(); + xor_gen_avx512_inner(dest, srcs, src_cnt, bytes); + asm volatile("vzeroupper"); + kernel_fpu_end(); +} + +struct xor_block_template xor_block_avx512 =3D { + .name =3D "avx512", + .xor_gen =3D xor_gen_avx512, +}; diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h index 991abe3f4bbd..ed5921d2e2aa 100644 --- a/lib/raid/xor/x86/xor_arch.h +++ b/lib/raid/xor/x86/xor_arch.h @@ -6,21 +6,31 @@ extern struct xor_block_template xor_block_p5_mmx; extern struct xor_block_template xor_block_sse; extern struct xor_block_template xor_block_sse_pf64; extern struct xor_block_template xor_block_avx; +extern struct xor_block_template xor_block_avx512; =20 -/* - * When SSE is available, use it as it can write around L2. We may also b= e able - * to load into the L1 only depending on how the cpu deals with a load to = a line - * that is being prefetched. - * - * When AVX2 is available, force using it as it is better by all measures. - * - * 32-bit without MMX can fall back to the generic routines. - */ static __always_inline void __init arch_xor_init(void) { - if (boot_cpu_has(X86_FEATURE_AVX)) { + if (IS_ENABLED(CONFIG_X86_64) && boot_cpu_has(X86_FEATURE_AVX512F) && + !boot_cpu_has(X86_FEATURE_PREFER_YMM)) { + /* + * Use the AVX-512 code on CPUs that support AVX-512 without + * overly-eager downclocking. On such CPUs the AVX-512 code + * should always work at least as well as the AVX code, so + * runtime selection is unnecessary. + * + * The AVX-512 code can work on X86_32. However, due to lack of + * use case for that, for now it's built only for X86_64. + */ + xor_force(&xor_block_avx512); + } else if (boot_cpu_has(X86_FEATURE_AVX)) { + /* AVX will be the best; no need to try others. */ xor_force(&xor_block_avx); } else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) { + /* + * When SSE is available, use it as it can write around L2. We + * may also be able to load into the L1 only depending on how + * the cpu deals with a load to a line that is being prefetched. + */ xor_register(&xor_block_sse); xor_register(&xor_block_sse_pf64); } else if (boot_cpu_has(X86_FEATURE_MMX)) { --=20 2.55.0