From nobody Tue Apr 28 07:43:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE88ACCA47B for ; Thu, 2 Jun 2022 21:04:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239333AbiFBVEm (ORCPT ); Thu, 2 Jun 2022 17:04:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234017AbiFBVEl (ORCPT ); Thu, 2 Jun 2022 17:04:41 -0400 Received: from polaris.svanheule.net (polaris.svanheule.net [84.16.241.116]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D9D235261 for ; Thu, 2 Jun 2022 14:04:40 -0700 (PDT) Received: from terra.. (unknown [IPv6:2a02:a03f:eaf9:8401:aa9f:5d01:1b2a:e3cd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sander@svanheule.net) by polaris.svanheule.net (Postfix) with ESMTPSA id A97FB2E19A7; Thu, 2 Jun 2022 23:04:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svanheule.net; s=mail1707; t=1654203875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pF66QpOyvZMR2LFw2QHcaSzXeOw1CJi9PNuSQ7tymZ8=; b=O5dI7xEPkLH+jtbBoL9KLnYtwqnk4QnmWui0gpR/128I0pvRa7I34DnFlZaq2m4QOaPVVv 3/tG6NLr22ZeqYuQtKhg0tHtlvNfUaR5VCbiLftV8WWXidFla3Q2BZkPYViCase2Zp7cDC 3Z4ZZUlWINElmpK65Q4kF3wEhieIaKBSrT22LHdoihOumlvxl5J3mnFSPa/VvIKIKFQPRP VPsP4N3pMaa2MvTM1MpPShurtZSgITvwFwrUt0HftLvwsRah/ktfegXABwwDlxcP4Lshhq NK465gNjG3Bej5pO2vfpo9x6jnmYbRiqb9bYZEJs+b1zQihATuQQecvYLFmWaw== From: Sander Vanheule To: Peter Zijlstra , Yury Norov , Andrew Morton , Valentin Schneider , Thomas Gleixner , Greg Kroah-Hartman , Marco Elver , Barry Song Cc: linux-kernel@vger.kernel.org, Andy Shevchenko , Sander Vanheule Subject: [PATCH v1 1/2] cpumask: Fix invalid uniprocessor mask assumption Date: Thu, 2 Jun 2022 23:04:19 +0200 Message-Id: <017b97698ba58d33bf45d30317d5a73c5b93d2a0.1654201862.git.sander@svanheule.net> X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" On uniprocessor builds, any CPU mask is assumed to contain exactly one CPU (cpu0). This ignores the existence of empty masks, resulting in incorrect behaviour for most of the implemented optimisations. Replace the uniprocessor optimisations with implementations that also take into account empty masks. Also drop the incorrectly optimised for_each_cpu implementations and use the generic implementations in all cases. Signed-off-by: Sander Vanheule --- include/linux/cpumask.h | 35 +++++++++++++++-------------------- 1 file changed, 15 insertions(+), 20 deletions(-) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index fe29ac7cc469..ce8c7b27f6c9 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -117,51 +117,54 @@ static __always_inline unsigned int cpumask_check(uns= igned int cpu) } =20 #if NR_CPUS =3D=3D 1 -/* Uniprocessor. Assume all masks are "1". */ +/* Uniprocessor. Assume all valid masks are "1" or empty. */ static inline unsigned int cpumask_first(const struct cpumask *srcp) { - return 0; + return !(*cpumask_bits(srcp) & 1); } =20 static inline unsigned int cpumask_first_zero(const struct cpumask *srcp) { - return 0; + return *cpumask_bits(srcp) & 1; } =20 static inline unsigned int cpumask_first_and(const struct cpumask *srcp1, const struct cpumask *srcp2) { - return 0; + return !(*cpumask_bits(srcp1) & *cpumask_bits(srcp2) & 1); } =20 static inline unsigned int cpumask_last(const struct cpumask *srcp) { - return 0; + return cpumask_first(srcp); } =20 /* Valid inputs for n are -1 and 0. */ static inline unsigned int cpumask_next(int n, const struct cpumask *srcp) { - return n+1; + return !!(n + 1 + cpumask_first(srcp)); } =20 static inline unsigned int cpumask_next_zero(int n, const struct cpumask *= srcp) { - return n+1; + return !!(n + 1 + cpumask_first_zero(srcp)); } =20 static inline unsigned int cpumask_next_and(int n, const struct cpumask *srcp, const struct cpumask *andp) { - return n+1; + return !!(n + 1 + cpumask_first_and(srcp, andp)); } =20 static inline unsigned int cpumask_next_wrap(int n, const struct cpumask *= mask, int start, bool wrap) { - /* cpu0 unless stop condition, wrap and at cpu0, then nr_cpumask_bits */ - return (wrap && n =3D=3D 0); + /* + * nr_cpumask_bits at stop condition, wrap and at cpu0, or empty mask + * otherwise cpu0 + */ + return (wrap && n =3D=3D 0) || cpumask_first(mask); } =20 /* cpu must be a valid cpu, ie 0, so there's no other choice. */ @@ -186,14 +189,6 @@ static inline int cpumask_any_distribute(const struct = cpumask *srcp) return cpumask_first(srcp); } =20 -#define for_each_cpu(cpu, mask) \ - for ((cpu) =3D 0; (cpu) < 1; (cpu)++, (void)mask) -#define for_each_cpu_not(cpu, mask) \ - for ((cpu) =3D 0; (cpu) < 1; (cpu)++, (void)mask) -#define for_each_cpu_wrap(cpu, mask, start) \ - for ((cpu) =3D 0; (cpu) < 1; (cpu)++, (void)mask, (void)(start)) -#define for_each_cpu_and(cpu, mask1, mask2) \ - for ((cpu) =3D 0; (cpu) < 1; (cpu)++, (void)mask1, (void)mask2) #else /** * cpumask_first - get the first cpu in a cpumask @@ -259,11 +254,13 @@ static inline unsigned int cpumask_next_zero(int n, c= onst struct cpumask *srcp) } =20 int __pure cpumask_next_and(int n, const struct cpumask *, const struct cp= umask *); +extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start,= bool wrap); int __pure cpumask_any_but(const struct cpumask *mask, unsigned int cpu); unsigned int cpumask_local_spread(unsigned int i, int node); int cpumask_any_and_distribute(const struct cpumask *src1p, const struct cpumask *src2p); int cpumask_any_distribute(const struct cpumask *srcp); +#endif /* SMP */ =20 /** * for_each_cpu - iterate over every cpu in a mask @@ -289,7 +286,6 @@ int cpumask_any_distribute(const struct cpumask *srcp); (cpu) =3D cpumask_next_zero((cpu), (mask)), \ (cpu) < nr_cpu_ids;) =20 -extern int cpumask_next_wrap(int n, const struct cpumask *mask, int start,= bool wrap); =20 /** * for_each_cpu_wrap - iterate over every cpu in a mask, starting at a spe= cified location @@ -324,7 +320,6 @@ extern int cpumask_next_wrap(int n, const struct cpumas= k *mask, int start, bool for ((cpu) =3D -1; \ (cpu) =3D cpumask_next_and((cpu), (mask1), (mask2)), \ (cpu) < nr_cpu_ids;) -#endif /* SMP */ =20 #define CPU_BITS_NONE \ { \ --=20 2.36.1 From nobody Tue Apr 28 07:43:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9D21C43334 for ; Thu, 2 Jun 2022 21:04:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237629AbiFBVEw (ORCPT ); Thu, 2 Jun 2022 17:04:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239332AbiFBVEm (ORCPT ); Thu, 2 Jun 2022 17:04:42 -0400 Received: from polaris.svanheule.net (polaris.svanheule.net [84.16.241.116]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3B7735260 for ; Thu, 2 Jun 2022 14:04:41 -0700 (PDT) Received: from terra.. (unknown [IPv6:2a02:a03f:eaf9:8401:aa9f:5d01:1b2a:e3cd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: sander@svanheule.net) by polaris.svanheule.net (Postfix) with ESMTPSA id CA7D92E19A9; Thu, 2 Jun 2022 23:04:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svanheule.net; s=mail1707; t=1654203876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IFZErQoe8ZlF0OjWg65tXnT3I6llghGKYUzn30c9kfg=; b=1/aKNVrzxd2vONtqJF3vP2mISEf+qR8ATq+lcQK3V9mVqcO8bMg8VosuZanJpdHuIqPKVR w6sMHR8UGO3lsyCdJg1QQuVp1cOXySGJU3yY51swvpwKiQJmYMgAHHEo0wnYKkwwvzTBoc K/tB2HP9LHz74sLhdaLhqv4hcRIQjOxAhd6Obv9EDx5EB097S4b8NgjkeETDvuwP3fcOUv +w/MX30lcsgg+wxMjK/PD1jzvCiuHML1ORSKmZ+7oiC6UtUj+P6DSjUgoXY1kuVRp0LzPY dcGmifcsMZuD1syUDbwhVRrmxNzzYHiLzMXByijECbfxy+6sV1+9w6etMGySdQ== From: Sander Vanheule To: Peter Zijlstra , Yury Norov , Andrew Morton , Valentin Schneider , Thomas Gleixner , Greg Kroah-Hartman , Marco Elver , Barry Song Cc: linux-kernel@vger.kernel.org, Andy Shevchenko , Sander Vanheule Subject: [PATCH v1 2/2] cpumask: Add UP optimised for_each_*_cpu loops Date: Thu, 2 Jun 2022 23:04:20 +0200 Message-Id: X-Mailer: git-send-email 2.36.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" On uniprocessor builds, the following loops will always run over a mask that contains one enabled CPU (cpu0): - for_each_possible_cpu - for_each_online_cpu - for_each_present_cpu Provide uniprocessor-specific macros for these loops, that always run exactly once. Signed-off-by: Sander Vanheule --- include/linux/cpumask.h | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index ce8c7b27f6c9..8af37cd603e3 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -806,9 +806,16 @@ extern const DECLARE_BITMAP(cpu_all_bits, NR_CPUS); /* First bits of cpu_bit_bitmap are in fact unset. */ #define cpu_none_mask to_cpumask(cpu_bit_bitmap[0]) =20 +#if NR_CPUS =3D=3D 1 +/* Uniprocessor: the possible/online/present masks are always "1" */ +#define for_each_possible_cpu(cpu) for ((cpu) =3D 0; (cpu) < 1; (cpu)++) +#define for_each_online_cpu(cpu) for ((cpu) =3D 0; (cpu) < 1; (cpu)++) +#define for_each_present_cpu(cpu) for ((cpu) =3D 0; (cpu) < 1; (cpu)++) +#else #define for_each_possible_cpu(cpu) for_each_cpu((cpu), cpu_possible_mask) #define for_each_online_cpu(cpu) for_each_cpu((cpu), cpu_online_mask) #define for_each_present_cpu(cpu) for_each_cpu((cpu), cpu_present_mask) +#endif =20 /* Wrappers for arch boot code to manipulate normally-constant masks */ void init_cpu_present(const struct cpumask *src); --=20 2.36.1