From nobody Thu Apr 9 12:15:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EDF5C433FE for ; Sun, 6 Nov 2022 09:51:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229832AbiKFJvb (ORCPT ); Sun, 6 Nov 2022 04:51:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229783AbiKFJv1 (ORCPT ); Sun, 6 Nov 2022 04:51:27 -0500 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD6F0E017 for ; Sun, 6 Nov 2022 01:51:26 -0800 (PST) Received: by mail-pg1-x536.google.com with SMTP id e129so8003643pgc.9 for ; Sun, 06 Nov 2022 01:51:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=tnyJavHPkbHwX7+XS5T05mPkmW2Gv1owKlEwtY5yB8s=; b=EGDlpI4moUq2A6m7KudPbv/MhIVZ8nGbetGSacPWCapDt+bJN5bWh0NvT2KzL05wAW dHj2IK70k94AuZwQl1YZqgHBWP8Xak6m4PV128avPnBPcH6CzVgxADpz2Yi+u3N7jsbU LoDwwNUnurtjlEap0kie2cdbx+9s3cFGUQmt+MQBVRlDjicQ7p6Ix102pirz9Oeu5/Aj EKaPKKmsO7ywKz6Fn2Fr9V16xXqSXYlCM0gTrSB+K3VFEOKAOMzUAaMidUsjO8kgylIu icFbRlz9pQFjRgoIYZJzPKg93ukIgWp9rcrUqImfhOcRMouj2duI9ViSNqsUFFRlUQOn ossA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=tnyJavHPkbHwX7+XS5T05mPkmW2Gv1owKlEwtY5yB8s=; b=5GJmTM4YO/BWRI4ZKoz0vkXDwG/vVxqeuJcrWqe9wC/mZoJJt2NUnfCtJn/Dw48Ei0 XsoEAa4pXh/4Xn83wBId8kMGmcf8h668qIJALreS4OTSkLWfVW32LnBGaK0rkE4Bw5H4 GRGKcJTHZSsTKhZEPW3hwGIRtqxDJUsm7mO+d0s5l3h60bGbeHMwbQMbQfvmfYdfGUFW GxApMmI2NvyvxyeEKRSb7eMApaQTaWqNUVsh4TR5pRiHcWJvvlAWy7be6F2+cFXKtNwv gFYaaS5AABUqBoI6xELAgUf5UO8IHBYTDvrknDcWUU6+RvRgTPeLZA00y2R+VE6RWcyC rcBQ== X-Gm-Message-State: ACrzQf2+p9qyfyuGZFyFl/pWaAQo9GC5l/WWpwhdJ0zihQJHGOV6fm50 Lh61DtNXjrBAWQwEGA/ks54= X-Google-Smtp-Source: AMsMyM5r+Qz3YMVoETdR7h+aMmGAf6v1oaUetjL66GCctHuEVrvNmFzQ3T7P6YAkiFrbVQ6k7rkG8A== X-Received: by 2002:a05:6a00:e1b:b0:537:7c74:c405 with SMTP id bq27-20020a056a000e1b00b005377c74c405mr44307387pfb.43.1667728286198; Sun, 06 Nov 2022 01:51:26 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id b5-20020aa79505000000b0056bfd4a2702sm2368677pfp.45.2022.11.06.01.51.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Nov 2022 01:51:25 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Vincent Mailhol , Borislav Petkov Subject: [PATCH v1 1/2] x86/asm/bitops: Replace __fls() by its generic builtin implementation Date: Sun, 6 Nov 2022 18:51:05 +0900 Message-Id: <20221106095106.849154-2-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Below snippet: #include unsigned int foo(unsigned long word) { return __fls(word); } produces this on GCC 12.1.0: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: 31 ff xor %edi,%edi 19: e9 00 00 00 00 jmp 1e and that on clang 14.0.6: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 50 push %rax b: 48 89 fb mov %rdi,%rbx e: e8 00 00 00 00 call 13 13: 48 89 1c 24 mov %rbx,(%rsp) 17: 48 0f bd 04 24 bsr (%rsp),%rax 1c: 48 83 c4 08 add $0x8,%rsp 20: 5b pop %rbx 21: c3 ret The implementation from [1] produces the exact same code on GCC and below code on clang: 0000000000000000 : 0: f3 0f 1e fa endbr64 4: e8 00 00 00 00 call 9 9: 53 push %rbx a: 48 89 fb mov %rdi,%rbx d: e8 00 00 00 00 call 12 12: 48 0f bd c3 bsr %rbx,%rax 16: 5b pop %rbx 17: c3 ret The builtin implementation is better for two reasons: 1/ it saves two instructions on clang (a push and a stack pointer decrement) because of a useless tentative to save rax. 2/ when used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). For those two reasons, replace the assembly implementation by its builtin counterpart. [1] https://elixir.bootlin.com/linux/v6.0/source/include/asm-generic/bitops= /builtin-__fls.h [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate c= onstant expressions") CC: Borislav Petkov CC: Nick Desaulniers CC: Yury Norov Signed-off-by: Vincent Mailhol Reviewed-by: Nick Desaulniers --- arch/x86/include/asm/bitops.h | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index 2edf68475fec..a31453d7686d 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -285,19 +285,7 @@ static __always_inline unsigned long variable_ffz(unsi= gned long word) (unsigned long)__builtin_ctzl(~word) : \ variable_ffz(word)) =20 -/* - * __fls: find last set bit in word - * @word: The word to search - * - * Undefined if no set bit exists, so code should check against 0 first. - */ -static __always_inline unsigned long __fls(unsigned long word) -{ - asm("bsr %1,%0" - : "=3Dr" (word) - : "rm" (word)); - return word; -} +#include =20 #undef ADDR =20 --=20 2.37.4 From nobody Thu Apr 9 12:15:26 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A636BC4332F for ; Sun, 6 Nov 2022 09:51:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229719AbiKFJvg (ORCPT ); Sun, 6 Nov 2022 04:51:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229829AbiKFJva (ORCPT ); Sun, 6 Nov 2022 04:51:30 -0500 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 538E9E0B2 for ; Sun, 6 Nov 2022 01:51:29 -0800 (PST) Received: by mail-pf1-x42f.google.com with SMTP id q9so8152824pfg.5 for ; Sun, 06 Nov 2022 01:51:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:from:to:cc:subject:date :message-id:reply-to; bh=ypcKLldwiqaDsOlKIEGYcVxJUId8HFUjb6S7/qxsdjE=; b=QBgr9zaTfSggETOEfREZkmAC9yEJR6EOPOcX9uReV/JQPsZ/IU1xIcz14lf13MYlNJ PfwH/wZiLZ5HW/VSUBBAnIoxDaFNoBfpgHqP0OVPOwkagdtbatdoSACMxEfkwNSKPJJa +xD8mX/j+P4RRrKXr/QJ890acISwlsaHC5764DhakcUjQq7x97TjCgS78/quvz1p6KQo HgQj+z0vW2mini8Xj7P4O+0NAk9EQpcan1J+VKPTPWs/rNorAxtHQClBp01tB0ztr1+w ts2D7OIpGVyaIsjN11sEzKqqhHOl41ldAWwnqa2GlIlGUjMuTJJLiXpNwcW5VwhZhura Bg3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=ypcKLldwiqaDsOlKIEGYcVxJUId8HFUjb6S7/qxsdjE=; b=1AcSIxkuym5G0ZfPnd6viV/bM5ZaKaCRh4Jufx8s/U6wXDY7OyusCXklLV8S/mRtAo XKaKaTeCp4lt0JJnARdRyp970aDZwMJ09P6Bpwz9sKq+4rYVNInFsUxp0l+oIWPR9OKg qsaimri9sAgTZpsTRZf/mYiWFuXLY5sQ0u/lq7apzE2QzeMoCOiuwm0M+sHi934pdWy4 JlhCaF/UtWODMhDfDhpYZ+htom8064JuA12JHGOd1iawOZbRxJxgF0RASMM80nneCkyr lmc84dDfIwkKNDzI47EBQk2CtcznXRigUZF9X7MNb/9dD02RVJCx+7y8QYmsRVA8gyjm UXlQ== X-Gm-Message-State: ACrzQf0M0em6zFr95qm2SthdvyL9/gDMM3MuBHz9QO0Nfn/pY/6dmmEk QYYjVLF9cSic04k5x3UV8oM= X-Google-Smtp-Source: AMsMyM5acENCwMU2s5DQIuUVT25ILLLvRbnoiZTTYvvIARQltZcdlyGzSNJ5Qyqntyb70XpQQ8KKrQ== X-Received: by 2002:a63:6bc5:0:b0:460:bd9a:64b8 with SMTP id g188-20020a636bc5000000b00460bd9a64b8mr39144875pgc.257.1667728288714; Sun, 06 Nov 2022 01:51:28 -0800 (PST) Received: from localhost.localdomain (124x33x176x97.ap124.ftth.ucom.ne.jp. [124.33.176.97]) by smtp.gmail.com with ESMTPSA id b5-20020aa79505000000b0056bfd4a2702sm2368677pfp.45.2022.11.06.01.51.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 06 Nov 2022 01:51:28 -0800 (PST) Sender: Vincent Mailhol From: Vincent Mailhol To: x86@kernel.org, Ingo Molnar , Borislav Petkov Cc: Nick Desaulniers , Thomas Gleixner , linux-kernel@vger.kernel.org, Yury Norov , llvm@lists.linux.dev, Vincent Mailhol , Borislav Petkov Subject: [PATCH v1 2/2] x86/asm/bitops: Use __builtin_clz*() to evaluate constant expressions Date: Sun, 6 Nov 2022 18:51:06 +0900 Message-Id: <20221106095106.849154-3-mailhol.vincent@wanadoo.fr> X-Mailer: git-send-email 2.37.4 In-Reply-To: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> References: <20221106095106.849154-1-mailhol.vincent@wanadoo.fr> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" GCC and clang offers the __builtin_clz(x) and __builtin_clzll(x) functions which return the number of leading zero bits in x. c.f. [1]. By a simple subtraction, we can derive below equivalences: * For fls: Aside of the x =3D 0 special case, fls(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clz(x). * For fls64: Aside of the x =3D 0 special case, fls64(x) is equivalent to BITS_PER_TYPE(x) - __builtin_clzll(x). __builtin_clzll() takes an unsigned long long as argument. We choose this version because BITS_PER_LONG_LONG is defined as 64 bits for all architecture making this flavor the most portable one. A BUILD_BUG_ON() safety net is added. When used on constant expressions, the compiler is only able to fold the builtin version (c.f. [2]). However, for non constant expressions, the kernel inline assembly results in better code for both GCC and clang. Use __builtin_constant_p() to select between the kernel's fls()/fls64() __builtin_clz()/__builtin_clzll() depending on whether the argument is constant or not. While renaming fls64() to variable_fls64(), change the argument type from __64 to u64 because we are not in an uapi header. [1] Built-in Functions Provided by GCC: https://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html [2] commit 146034fed6ee ("x86/asm/bitops: Use __builtin_ffs() to evaluate c= onstant expressions") CC: Borislav Petkov CC: Nick Desaulniers CC: Yury Norov Signed-off-by: Vincent Mailhol --- arch/x86/include/asm/bitops.h | 57 ++++++++++++++++++++++++----------- 1 file changed, 40 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/bitops.h b/arch/x86/include/asm/bitops.h index a31453d7686d..58fb2fc49760 100644 --- a/arch/x86/include/asm/bitops.h +++ b/arch/x86/include/asm/bitops.h @@ -333,18 +333,15 @@ static __always_inline int variable_ffs(int x) */ #define ffs(x) (__builtin_constant_p(x) ? __builtin_ffs(x) : variable_ffs(= x)) =20 -/** - * fls - find last set bit in word - * @x: the word to search - * - * This is defined in a similar way as the libc and compiler builtin - * ffs, but returns the position of the most significant set bit. - * - * fls(value) returns 0 if value is 0 or the position of the last - * set bit if value is nonzero. The last (most significant) bit is - * at position 32. - */ -static __always_inline int fls(unsigned int x) +static __always_inline int constant_fls(unsigned int x) +{ + if (!x) + return 0; + + return BITS_PER_TYPE(x) - __builtin_clz(x); +} + +static __always_inline int variable_fls(unsigned int x) { int r; =20 @@ -375,18 +372,30 @@ static __always_inline int fls(unsigned int x) } =20 /** - * fls64 - find last set bit in a 64-bit word + * fls - find last set bit in word * @x: the word to search * * This is defined in a similar way as the libc and compiler builtin - * ffsll, but returns the position of the most significant set bit. + * ffs, but returns the position of the most significant set bit. * - * fls64(value) returns 0 if value is 0 or the position of the last + * fls(value) returns 0 if value is 0 or the position of the last * set bit if value is nonzero. The last (most significant) bit is - * at position 64. + * at position 32. */ +#define fls(x) (__builtin_constant_p(x) ? constant_fls(x) : variable_fls(x= )) + #ifdef CONFIG_X86_64 -static __always_inline int fls64(__u64 x) +static __always_inline int constant_fls64(u64 x) +{ + BUILD_BUG_ON(sizeof(unsigned long long) !=3D sizeof(x)); + + if (!x) + return 0; + + return BITS_PER_TYPE(x) - __builtin_clzll(x); +} + +static __always_inline int variable_fls64(u64 x) { int bitpos =3D -1; /* @@ -399,6 +408,20 @@ static __always_inline int fls64(__u64 x) : "rm" (x)); return bitpos + 1; } + +/** + * fls64 - find last set bit in a 64-bit word + * @x: the word to search + * + * This is defined in a similar way as the libc and compiler builtin + * ffsll, but returns the position of the most significant set bit. + * + * fls64(value) returns 0 if value is 0 or the position of the last + * set bit if value is nonzero. The last (most significant) bit is + * at position 64. + */ +#define fls64(x) \ + (__builtin_constant_p(x) ? constant_fls64(x) : variable_fls64(x)) #else #include #endif --=20 2.37.4