From nobody Fri Dec 19 20:54:14 2025 Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F38981607AB for ; Mon, 5 Aug 2024 18:40:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883238; cv=none; b=EhZfAr7emErGUCXykUcjtDk1S3pa93w5ckbR/QqybLlZCDklgsrzLegxB1ynyOW9LNrxeNzH/blD7c28Ca8XKUL1szZ+U9+5gihk7V7Jy67Gn66HSxHszoGrQGybcLONVi/+03NKzjM8gKm1sVRtUMpnxZ61onkgFjuJau2Tw1s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883238; c=relaxed/simple; bh=6TKZTvnr4hDLb6Pz4mcTEssQCKkCHvMQO3oyqAbQsnw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TmkGKV7xT6Jq7OYsm2gxgzxUu0Tdqbw0K3HaibZlRUUkZsokW790Z87v4/eBmMEiTM/CcrGeyUY4kXr3jirv2kmVitCR/5UnVaGnHISMT1+kYilPbtnlvKmLLSrFCMo5DuCqh1JAALjNh/JBOIe5vU/2+vB3FNep0P/cUFsERus= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JZ4nhW2m; arc=none smtp.client-ip=209.85.218.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JZ4nhW2m" Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a7a8a4f21aeso1399411966b.2 for ; Mon, 05 Aug 2024 11:40:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722883235; x=1723488035; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kE/o4qSaUHLv/y40/aLwm+zO5SXvHFNVlO+ercnrRJY=; b=JZ4nhW2mU1mfEU7FM9StvGkOSK1CObxWpJGYe2LO5wAeO/Fm4r17V89VY+cj5EvXLZ ONY4V+48PD/PQNx8qPjD+baX3eYlL+ezM7Tt8kaPVhkaitB7K5sLIqo3u9c9CPWg28Hc 3JkjeHNdvciUigJIUTdi22KClyLxNf9AJBVyMbQ7obVRMhADuIVR6d11RE1y1UegNluP rzKfywb+hzwbdlBPSQjadiO+UEmdIFkYKnH3I8jC0/p5Bs0X8AyMYg3s9Lj2ZLz9yNpN djxwg7WrCu/pt9aa3+RuyjVNCfh+2xjyW2wOmxcvvKD3iCNnxPcnuBZzXQ80agMIs1ov Z6Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722883235; x=1723488035; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kE/o4qSaUHLv/y40/aLwm+zO5SXvHFNVlO+ercnrRJY=; b=NXIKyA3QeG1bh6le9i+aCSK/1cvV/eJN/I25h9garvbMALCm8lwEIBc7mEfrdgSGur erSNq8lUfjKXsV3g5gLFwDiErtTk4at5f8bGZkoEz3zHgFUDOblY88gOoiF9cke8w9Xy eULR5q8VA1Tw8C659VtmQhpv0AOwq/UfKj2qu7RTr2hHE0RwAPw8xogMSCG6LQRnkmIR d1POtNdFD+FobJCcksfOrjg/E3OqV9W28OhsFzyJFocvNrtlSHm9mOy0rvnlSO8GLJ0e HAHfcmBrNi2sR2YN1TVjVGkUAn5JLhWuuEQKl1ZdbLxTN9HwDNznXoSKTeCU8f+BSUfS mUVA== X-Forwarded-Encrypted: i=1; AJvYcCVfzRr7Ga7CHf5202kJnQVWURVBEw/SN2+gvL9L4P+5H7eaX/BYAHbpG8tT0UmdAL31mEMBKA5gcZECuHYZChhJcwqk1yOhgcHi9cQc X-Gm-Message-State: AOJu0YwfCgCoeSaM5LkkwlxSZ5D84AFfhDesfZee5LC+eKwQo5E1JxxN alFTm+3cO2ga9cQwOBrTmBLCIkkRG1LnQsay0HzPQ5gJzXixSeND X-Google-Smtp-Source: AGHT+IFkVV7ouGVBTfQqeCTN4nxr/EWf3RpQbZFrLGcAjto3LgX3mNyQ3drkidVkE3T2KpUQpA4GsQ== X-Received: by 2002:a17:907:7f90:b0:a7a:bece:6223 with SMTP id a640c23a62f3a-a7dc4db8bfcmr957386466b.6.1722883234971; Mon, 05 Aug 2024 11:40:34 -0700 (PDT) Received: from localhost.localdomain (93-103-32-68.dynamic.t-2.net. [93.103.32.68]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a7dc9c12ad3sm478517266b.88.2024.08.05.11.40.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Aug 2024 11:40:33 -0700 (PDT) From: Uros Bizjak To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Dennis Zhou , Tejun Heo , Christoph Lameter , Andy Lutomirski , Ingo Molnar , Nadav Amit , Brian Gerst , Denys Vlasenko , "H . Peter Anvin" , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Borislav Petkov , Luc Van Oostenryck Subject: [RFC PATCH 1/3] percpu: Define __pcpu_typeof() Date: Mon, 5 Aug 2024 20:38:59 +0200 Message-ID: <20240805184012.358023-2-ubizjak@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240805184012.358023-1-ubizjak@gmail.com> References: <20240805184012.358023-1-ubizjak@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define __pcpu_typeof() to use __typeof_unqual__() as typeof operator when available, to return unqualified type of the expression. If an arch defines __percpu variables in their own named address space, then __pcpu_typeof() returns unqualified type of the expression without named address space quialifier when CONFIG_CC_HAS_TYPEOF_UNQUAL is defined. Signed-off-by: Uros Bizjak Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Nadav Amit Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Luc Van Oostenryck --- arch/x86/include/asm/percpu.h | 38 ++++++++++++++++++++++------------ include/linux/part_stat.h | 2 +- include/linux/percpu-defs.h | 39 ++++++++++++++++++++++++++++------- init/Kconfig | 3 +++ kernel/locking/percpu-rwsem.c | 2 +- 5 files changed, 62 insertions(+), 22 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index c55a79d5feae..4d31203eb0d2 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -73,10 +73,14 @@ unsigned long tcp_ptr__ =3D raw_cpu_read_long(this_cpu_off); \ \ tcp_ptr__ +=3D (__force unsigned long)(_ptr); \ - (typeof(*(_ptr)) __kernel __force *)tcp_ptr__; \ + (__pcpu_typeof(*(_ptr)) __kernel __force *)tcp_ptr__; \ }) #else -#define arch_raw_cpu_ptr(_ptr) ({ BUILD_BUG(); (typeof(_ptr))0; }) +#define arch_raw_cpu_ptr(_ptr) \ +({ \ + BUILD_BUG(); \ + (__pcpu_typeof(*(_ptr)) __kernel __force *)0; \ +}) #endif =20 #define PER_CPU_VAR(var) %__percpu_seg:(var)__percpu_rel @@ -172,7 +176,7 @@ do { \ : [val] __pcpu_reg_##size("=3D", pfo_val__) \ : [var] "m" (__my_cpu_var(_var))); \ \ - (typeof(_var))(unsigned long) pfo_val__; \ + (__pcpu_typeof(_var))(unsigned long) pfo_val__; \ }) =20 #define __raw_cpu_write(size, qual, _var, _val) \ @@ -180,7 +184,7 @@ do { \ __pcpu_type_##size pto_val__ =3D __pcpu_cast_##size(_val); \ \ if (0) { \ - typeof(_var) pto_tmp__; \ + __pcpu_typeof(_var) pto_tmp__; \ pto_tmp__ =3D (_val); \ (void)pto_tmp__; \ } \ @@ -193,7 +197,11 @@ do { \ * The generic per-CPU infrastrucutre is not suitable for * reading const-qualified variables. */ -#define __raw_cpu_read_const(pcp) ({ BUILD_BUG(); (typeof(pcp))0; }) +#define __raw_cpu_read_const(pcp) \ +({ \ + BUILD_BUG(); \ + (__pcpu_typeof(pcp))0; \ +}) =20 #endif /* CONFIG_USE_X86_SEG_SUPPORT */ =20 @@ -205,7 +213,7 @@ do { \ : [val] __pcpu_reg_##size("=3D", pfo_val__) \ : [var] "i" (&(_var))); \ \ - (typeof(_var))(unsigned long) pfo_val__; \ + (__pcpu_typeof(_var))(unsigned long) pfo_val__; \ }) =20 #define percpu_unary_op(size, qual, op, _var) \ @@ -219,7 +227,7 @@ do { \ __pcpu_type_##size pto_val__ =3D __pcpu_cast_##size(_val); \ \ if (0) { \ - typeof(_var) pto_tmp__; \ + __pcpu_typeof(_var) pto_tmp__; \ pto_tmp__ =3D (_val); \ (void)pto_tmp__; \ } \ @@ -239,7 +247,7 @@ do { \ (int)(val) : 0; \ \ if (0) { \ - typeof(var) pao_tmp__; \ + __pcpu_typeof(var) pao_tmp__; \ pao_tmp__ =3D (val); \ (void)pao_tmp__; \ } \ @@ -263,7 +271,7 @@ do { \ : [tmp] __pcpu_reg_##size("+", paro_tmp__), \ [var] "+m" (__my_cpu_var(_var)) \ : : "memory"); \ - (typeof(_var))(unsigned long) (paro_tmp__ + _val); \ + (__pcpu_typeof(_var))(unsigned long) (paro_tmp__ + _val); \ }) =20 /* @@ -272,7 +280,7 @@ do { \ */ #define raw_percpu_xchg_op(_var, _nval) \ ({ \ - typeof(_var) pxo_old__ =3D raw_cpu_read(_var); \ + __pcpu_typeof(_var) pxo_old__ =3D raw_cpu_read(_var); \ \ raw_cpu_write(_var, _nval); \ \ @@ -286,7 +294,7 @@ do { \ */ #define this_percpu_xchg_op(_var, _nval) \ ({ \ - typeof(_var) pxo_old__ =3D this_cpu_read(_var); \ + __pcpu_typeof(_var) pxo_old__ =3D this_cpu_read(_var); \ \ do { } while (!this_cpu_try_cmpxchg(_var, &pxo_old__, _nval)); \ \ @@ -309,7 +317,7 @@ do { \ : [nval] __pcpu_reg_##size(, pco_new__) \ : "memory"); \ \ - (typeof(_var))(unsigned long) pco_old__; \ + (__pcpu_typeof(_var))(unsigned long) pco_old__; \ }) =20 #define percpu_try_cmpxchg_op(size, qual, _var, _ovalp, _nval) \ @@ -568,7 +576,11 @@ do { \ #else /* !CONFIG_X86_64: */ =20 /* There is no generic 64-bit read stable operation for 32-bit targets. */ -#define this_cpu_read_stable_8(pcp) ({ BUILD_BUG(); (typeof(pcp))0; }) +#define this_cpu_read_stable_8(pcp) \ +({ \ + BUILD_BUG(); \ + (__pcpu_typeof(pcp))0; \ +}) =20 #define raw_cpu_read_long(pcp) raw_cpu_read_4(pcp) =20 diff --git a/include/linux/part_stat.h b/include/linux/part_stat.h index ac8c44dd8237..3807bc29ba05 100644 --- a/include/linux/part_stat.h +++ b/include/linux/part_stat.h @@ -33,7 +33,7 @@ struct disk_stats { =20 #define part_stat_read(part, field) \ ({ \ - typeof((part)->bd_stats->field) res =3D 0; \ + __pcpu_typeof((part)->bd_stats->field) res =3D 0; \ unsigned int _cpu; \ for_each_possible_cpu(_cpu) \ res +=3D per_cpu_ptr((part)->bd_stats, _cpu)->field; \ diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 8efce7414fad..842d10912fdd 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -220,6 +220,21 @@ do { \ (void)__vpp_verify; \ } while (0) =20 +/* + * Define __percpu_typeof() to use __typeof_unqual__() as typeof + * operator when available, to return unqualified type of the exp. + * + * If an arch defines __percpu variables in their own named address + * space, then __pcpu_typeof() returns unqualified type of the + * expression without named address space qualifier when + * CONFIG_CC_HAS_TYPEOF_UNQUAL is defined. + */ +#ifdef CONFIG_CC_HAS_TYPEOF_UNQUAL +#define __pcpu_typeof(exp) __typeof_unqual__(exp) +#else +#define __pcpu_typeof(exp) __typeof__(exp) +#endif + #ifdef CONFIG_SMP =20 /* @@ -228,7 +243,10 @@ do { \ * pointer value. The weird cast keeps both GCC and sparse happy. */ #define SHIFT_PERCPU_PTR(__p, __offset) \ - RELOC_HIDE((typeof(*(__p)) __kernel __force *)(__p), (__offset)) + uintptr_t ptr__ =3D (__force uintptr_t)(__p); \ + \ + RELOC_HIDE((__pcpu_typeof(*(__p)) __kernel __force *)(ptr__), \ + (__offset)) =20 #define per_cpu_ptr(ptr, cpu) \ ({ \ @@ -254,13 +272,20 @@ do { \ =20 #else /* CONFIG_SMP */ =20 -#define VERIFY_PERCPU_PTR(__p) \ +#define PERCPU_PTR(__p) \ ({ \ - __verify_pcpu_ptr(__p); \ - (typeof(*(__p)) __kernel __force *)(__p); \ + uintptr_t ptr__ =3D (__force uintptr_t)(__p); \ + \ + (__pcpu_typeof(*(__p)) __kernel __force *)(ptr__); \ +}) + +#define per_cpu_ptr(ptr, cpu) \ +({ \ + __verify_pcpu_ptr(ptr); \ + (void)(cpu); \ + PERCPU_PTR(ptr); \ }) =20 -#define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); VERIFY_PERCPU_PTR(ptr); }) #define raw_cpu_ptr(ptr) per_cpu_ptr(ptr, 0) #define this_cpu_ptr(ptr) raw_cpu_ptr(ptr) =20 @@ -315,7 +340,7 @@ static __always_inline void __this_cpu_preempt_check(co= nst char *op) { } =20 #define __pcpu_size_call_return(stem, variable) \ ({ \ - typeof(variable) pscr_ret__; \ + __pcpu_typeof(variable) pscr_ret__; \ __verify_pcpu_ptr(&(variable)); \ switch(sizeof(variable)) { \ case 1: pscr_ret__ =3D stem##1(variable); break; \ @@ -330,7 +355,7 @@ static __always_inline void __this_cpu_preempt_check(co= nst char *op) { } =20 #define __pcpu_size_call_return2(stem, variable, ...) \ ({ \ - typeof(variable) pscr2_ret__; \ + __pcpu_typeof(variable) pscr2_ret__; \ __verify_pcpu_ptr(&(variable)); \ switch(sizeof(variable)) { \ case 1: pscr2_ret__ =3D stem##1(variable, __VA_ARGS__); break; \ diff --git a/init/Kconfig b/init/Kconfig index 37260d17267e..a9a04d0683b6 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -873,6 +873,9 @@ config ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH config CC_HAS_INT128 def_bool !$(cc-option,$(m64-flag) -D__SIZEOF_INT128__=3D0) && 64BIT =20 +config CC_HAS_TYPEOF_UNQUAL + def_bool $(success,echo 'int foo (int a) { __typeof_unqual__(a) b =3D a; = return b; }' | $(CC) -x c - -S -o /dev/null) + config CC_IMPLICIT_FALLTHROUGH string default "-Wimplicit-fallthrough=3D5" if CC_IS_GCC && $(cc-option,-Wimplic= it-fallthrough=3D5) diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index 6083883c4fe0..ac9b2f4bcd92 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -184,7 +184,7 @@ EXPORT_SYMBOL_GPL(__percpu_down_read); =20 #define per_cpu_sum(var) \ ({ \ - typeof(var) __sum =3D 0; \ + __pcpu_typeof(var) __sum =3D 0; \ int cpu; \ compiletime_assert_atomic_type(__sum); \ for_each_possible_cpu(cpu) \ --=20 2.45.2 From nobody Fri Dec 19 20:54:14 2025 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B937165EE7 for ; Mon, 5 Aug 2024 18:40:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883239; cv=none; b=ifHbJiBPuRtwVPd7JcthfJq6ku4ANhJhFRrSOj+H6yKwTQIBnDy065qkpw66xaOr4GpX8by+xOuL/7RgDPAcJkAcCJtbgQjpAlzk5f3erSqidqqjqZLy6ZIuNYxkt4Iohknlh1OS/M13K1MVQe8hTsRtWqVkoVRQtDiPQxIsK28= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883239; c=relaxed/simple; bh=/CUHva9R4qh40v4PXGdZpG7qraQNBpsnib80AoLfVts=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mkIUFM4YroCsW9+TI7AEH0hvYax+ooe4dK62AQxI+u+E/wbX+RmOY/WoQxf/D884dvgp6zqtdmvTpwdWtGGquGIi3xbmTYjbwbTL3VJJTXMjwjHlkzOGIfIg5C7iGgqFklQv6lJ2Y0B76mhAxRjLA/irBDEOvQ4ReoN01UYb8jU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UXVrJQHv; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UXVrJQHv" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5b8c2a61386so4129567a12.2 for ; Mon, 05 Aug 2024 11:40:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722883236; x=1723488036; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NvmD+ez0KqmrlVq3MNB02uXxyKID9fWAsFipmCLltBQ=; b=UXVrJQHvrX7hy1gtSIHY46Ew8K0GXT3ov5aXtQ/7Rgdq6CkBUxilQBNR4PkEP53t54 I5joXxfmAO56DWsUXuA6C6T+NA85Nin53I63kKteY/Aw0gWolnIWzX/FlS5sUHEi8SMj gD1+9ziBsfVI0VJG02B72Fey8bUZTh9SjSOjMygrJaYo+lBLmYhFKOUd/h7cbvatO/hn +Es/PpwVTPHh9veAIW8hHFhqoqkziKafuk+NRnWTc3SGVvXXHDxSFggb6Tyg4UmorwJS BYltEZ6iOoRj/MnR0dIVY+tw0IpBKxTgISOOd7o3o9pad0s9WJ/bur2PddVgpgIS2S+b TtTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722883236; x=1723488036; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NvmD+ez0KqmrlVq3MNB02uXxyKID9fWAsFipmCLltBQ=; b=igVKgaAaJeAYjZ+jb3A4uPYXx6Fv2EAhqBDkXhMnBNpHsjRrWsI3zQjqauRknwWDfl 5mtIP0qWhTIrIC+EbxCCl+7r1V/KTiU3fAz9aUa2ceIayWXC0mKWacP3pW8aB878XEQf JrckFhRDueOR5ujBSomZMJkYOEZvgmnsCnlQy+oA2CnbegoeYuNjTD0p8oe7idd1IXpL NlfLZsJJXyAgcnUg38QCFUxxW0bv1yEZncAXNFgA7dirJds3/9Q6o6KuI4lt2CviqDQc 2izUcqBkziNZX4Gb0JAL7VvM5/STwV25e+PB30zhfRKJXe3Dz1fn7mDymdc8osU3VW/o NLNQ== X-Forwarded-Encrypted: i=1; AJvYcCWCWgSWrZxqUuEMjj84wT2f8/j8lBPrrPkpZ8sueYo5Tcnbv8ah0LELvG06dyZa8R0yawW9r89kGL/W3XMOQblmfrbA/WvlpTjuC1vC X-Gm-Message-State: AOJu0YyaaXFZQXV9aaftBJUoTxY5vCf2nokq+wcMGbnVKfDNytsAZKdm XE5zqHmxJW12lGyW4DffHRnhKs7DeXHP7wsXRXi3YTnnjP7z8tPM X-Google-Smtp-Source: AGHT+IGT8P8MH8PUrIhAbGQwOI91FhpuTTEeBpD91PRfdIvScb2atdeUvkPb95VBGym8haoLNMyXMg== X-Received: by 2002:a17:907:c00f:b0:a7d:e956:ad51 with SMTP id a640c23a62f3a-a7de956af6dmr465965066b.21.1722883236392; Mon, 05 Aug 2024 11:40:36 -0700 (PDT) Received: from localhost.localdomain (93-103-32-68.dynamic.t-2.net. [93.103.32.68]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a7dc9c12ad3sm478517266b.88.2024.08.05.11.40.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Aug 2024 11:40:35 -0700 (PDT) From: Uros Bizjak To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Dennis Zhou , Tejun Heo , Christoph Lameter , Andy Lutomirski , Ingo Molnar , Nadav Amit , Brian Gerst , Denys Vlasenko , "H . Peter Anvin" , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Borislav Petkov , Luc Van Oostenryck Subject: [RFC PATCH 2/3] percpu: Assorted fixes found by strict percpu address space checks Date: Mon, 5 Aug 2024 20:39:00 +0200 Message-ID: <20240805184012.358023-3-ubizjak@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240805184012.358023-1-ubizjak@gmail.com> References: <20240805184012.358023-1-ubizjak@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Assorted fixes to prevent defconfig build failures when strict percpu address space checks will be enabled. These show effeciveness of strict percpu address space checks. Signed-off-by: Uros Bizjak Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Nadav Amit Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Luc Van Oostenryck --- arch/x86/kernel/head64.c | 3 ++- drivers/base/devres.c | 2 +- fs/aio.c | 2 +- include/linux/cleanup.h | 4 ++-- include/linux/prandom.h | 1 + kernel/events/hw_breakpoint.c | 4 ++-- kernel/workqueue.c | 2 +- lib/percpu_counter.c | 2 +- net/core/dev.c | 2 +- 9 files changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/x86/kernel/head64.c b/arch/x86/kernel/head64.c index a817ed0724d1..f5d6ad351cc4 100644 --- a/arch/x86/kernel/head64.c +++ b/arch/x86/kernel/head64.c @@ -560,9 +560,10 @@ void early_setup_idt(void) void __head startup_64_setup_gdt_idt(void) { void *handler =3D NULL; + struct desc_struct *gdt =3D (struct desc_struct *)(uintptr_t)init_per_cpu= _var(gdt_page.gdt); =20 struct desc_ptr startup_gdt_descr =3D { - .address =3D (unsigned long)&RIP_REL_REF(init_per_cpu_var(gdt_page.gdt)), + .address =3D (unsigned long)&RIP_REL_REF(*gdt), .size =3D GDT_SIZE - 1, }; =20 diff --git a/drivers/base/devres.c b/drivers/base/devres.c index a2ce0ead06a6..894f5a1e6d18 100644 --- a/drivers/base/devres.c +++ b/drivers/base/devres.c @@ -1231,6 +1231,6 @@ void devm_free_percpu(struct device *dev, void __perc= pu *pdata) * devm_free_pages() does. */ WARN_ON(devres_release(dev, devm_percpu_release, devm_percpu_match, - (__force void *)pdata)); + (__force void *)(uintptr_t)pdata)); } EXPORT_SYMBOL_GPL(devm_free_percpu); diff --git a/fs/aio.c b/fs/aio.c index 6066f64967b3..e8920178b50f 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -100,7 +100,7 @@ struct kioctx { =20 unsigned long user_id; =20 - struct __percpu kioctx_cpu *cpu; + struct kioctx_cpu __percpu *cpu; =20 /* * For percpu reqs_available, number of slots we move to/from global diff --git a/include/linux/cleanup.h b/include/linux/cleanup.h index d9e613803df1..f7e1158cbacf 100644 --- a/include/linux/cleanup.h +++ b/include/linux/cleanup.h @@ -154,7 +154,7 @@ static inline class_##_name##_t class_##_name##ext##_co= nstructor(_init_args) \ #define DEFINE_GUARD(_name, _type, _lock, _unlock) \ DEFINE_CLASS(_name, _type, if (_T) { _unlock; }, ({ _lock; _T; }), _type = _T); \ static inline void * class_##_name##_lock_ptr(class_##_name##_t *_T) \ - { return *_T; } + { return (void *)(uintptr_t)*_T; } =20 #define DEFINE_GUARD_COND(_name, _ext, _condlock) \ EXTEND_CLASS(_name, _ext, \ @@ -211,7 +211,7 @@ static inline void class_##_name##_destructor(class_##_= name##_t *_T) \ \ static inline void *class_##_name##_lock_ptr(class_##_name##_t *_T) \ { \ - return _T->lock; \ + return (void *)(uintptr_t)_T->lock; \ } =20 =20 diff --git a/include/linux/prandom.h b/include/linux/prandom.h index f7f1e5251c67..f2ed5b72b3d6 100644 --- a/include/linux/prandom.h +++ b/include/linux/prandom.h @@ -10,6 +10,7 @@ =20 #include #include +#include #include =20 struct rnd_state { diff --git a/kernel/events/hw_breakpoint.c b/kernel/events/hw_breakpoint.c index 6c2cb4e4f48d..d82fe78f0658 100644 --- a/kernel/events/hw_breakpoint.c +++ b/kernel/events/hw_breakpoint.c @@ -849,7 +849,7 @@ register_wide_hw_breakpoint(struct perf_event_attr *att= r, =20 cpu_events =3D alloc_percpu(typeof(*cpu_events)); if (!cpu_events) - return (void __percpu __force *)ERR_PTR(-ENOMEM); + return (void __percpu __force *)(uintptr_t)ERR_PTR(-ENOMEM); =20 cpus_read_lock(); for_each_online_cpu(cpu) { @@ -868,7 +868,7 @@ register_wide_hw_breakpoint(struct perf_event_attr *att= r, return cpu_events; =20 unregister_wide_hw_breakpoint(cpu_events); - return (void __percpu __force *)ERR_PTR(err); + return (void __percpu __force *)(uintptr_t)ERR_PTR(err); } EXPORT_SYMBOL_GPL(register_wide_hw_breakpoint); =20 diff --git a/kernel/workqueue.c b/kernel/workqueue.c index 1745ca788ede..32729a2e93af 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -377,7 +377,7 @@ struct workqueue_struct { =20 /* hot fields used during command issue, aligned to cacheline */ unsigned int flags ____cacheline_aligned; /* WQ: WQ_* flags */ - struct pool_workqueue __percpu __rcu **cpu_pwq; /* I: per-cpu pwqs */ + struct pool_workqueue __rcu * __percpu *cpu_pwq; /* I: per-cpu pwqs */ struct wq_node_nr_active *node_nr_active[]; /* I: per-node nr_active */ }; =20 diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 51bc5246986d..3d0613ac7e73 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -209,7 +209,7 @@ int __percpu_counter_init_many(struct percpu_counter *f= bc, s64 amount, INIT_LIST_HEAD(&fbc[i].list); #endif fbc[i].count =3D amount; - fbc[i].counters =3D (void *)counters + (i * counter_size); + fbc[i].counters =3D (void __percpu *)counters + (i * counter_size); =20 debug_percpu_counter_activate(&fbc[i]); } diff --git a/net/core/dev.c b/net/core/dev.c index 751d9b70e6ad..5cad88cf029c 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -10860,7 +10860,7 @@ noinline void netdev_core_stats_inc(struct net_devi= ce *dev, u32 offset) return; } =20 - field =3D (__force unsigned long __percpu *)((__force void *)p + offset); + field =3D (unsigned long __percpu *)(void __percpu *)(p + offset); this_cpu_inc(*field); } EXPORT_SYMBOL_GPL(netdev_core_stats_inc); --=20 2.45.2 From nobody Fri Dec 19 20:54:14 2025 Received: from mail-ej1-f46.google.com (mail-ej1-f46.google.com [209.85.218.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98310165F08 for ; Mon, 5 Aug 2024 18:40:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883241; cv=none; b=Ib7YwdA1abGExTt6pnZ5ZbPTzh3ByB6DfJF52UoGgMpHJs2yrMXB1Er2Uz+md746g4ZImWPyMQl3cHXGjQCMZP4byvLeAmiDfVFziX+BbuWrNJAJAkh8LVqW8yhF8jxVJsgYf3qe5FrmRTj0JHvuhiis4zqI715KelXR6Oe5bgk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722883241; c=relaxed/simple; bh=24P15Bx9ElCDUMAi4OqJDf1LJ44qOrvW6LopdQr9Ts0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kfT45csCZSnUlXJHMzNoaOydzG31HkJkJCyXvaWuF4SMwcIz7aDBpSuSQbhHwfY4sybMJJvPhcRHjtJJKQLu/QuhpqpuRWKma4FaCvyqognk94gRcuNDWR7Q4PVi0KbI56xuuqGGC9t5Ls5ymKQ5pkfN5mrZm0w+mKy5MbLwno0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LOEPiif0; arc=none smtp.client-ip=209.85.218.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LOEPiif0" Received: by mail-ej1-f46.google.com with SMTP id a640c23a62f3a-a7a8e73b29cso904813066b.3 for ; Mon, 05 Aug 2024 11:40:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1722883238; x=1723488038; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zriScYxEubY60jokktbaWFHqZ5TIRBloyPdl3CM/JiM=; b=LOEPiif06VbgwIxjUtFDlnaswSvT9s+J+gkoUurUqyR6vKbZEwaFKkLsM2PjUqXaN5 lyL/csz47YE38ml2abONr/354u0gu+j7jPMjYCaJWvnqC9gx8Aynye89IlXgUL7UAADz kFmG6PPbHVwsSPqrZ0SYDfJ+gS+4xPdTorR4PhoZeX+Yr0/Zd265AHtZB6bEjxgojdtR /aMjo3UjqCoA8EqPQiHR6rBUDcuaUViLN8TeQT6VPiWIJoJsnOfS/dzJs2NiW/OgXWDV rh1jev2Eqyako5o3CUaxXXM0WVjMTmd3+QcCLq6M7in42nsOMqechpvUsMx+koGC5Erb YVcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722883238; x=1723488038; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zriScYxEubY60jokktbaWFHqZ5TIRBloyPdl3CM/JiM=; b=TtkNNuuEiAnqQAPw9YSId5ae5TbUbcGsWfPvALnFTv2crX9Y786qrt368cwaVcEPSH 27JLMKJy9H9kKt9TwGNa/0Ly6JrhjGppUql/esaOIczLrVePcrpxK0HnaRJys0EZDQkR FIz1eOK4n1uv/iTRKbPtbJKGOUUcgDLpkcHGgAybcfJ+sGRdYrO3eA6KShRRSQnTDdSJ ph5498ZwOE+SHT1l1T+afHi25QCV2L3jDTbRgbmG/c1CNa8gCQ9w0HAZT/MqZiP3Z3si 1ODWU4LId/IUl51++1JQhKV1lcIyhTlzIrJSLzrZdlVhcGR9hzm9CYH/VGqtgQA+IfzA go9g== X-Forwarded-Encrypted: i=1; AJvYcCVWy3N0U317L5fKTBzSlPJwYozY2o5dcFa1FcjyGj28V6ommfiWpeph7pqety2E8OKE+a+A4X1GF7fr020P7OBnmsErMkw09cKzaeBV X-Gm-Message-State: AOJu0YxLquNifikDb8U+6b7/iz5pCo6Jf/OcNErgQS/s2pZ/kAl+A64U rDI161yEgwayChRd796S3XEDD1e5NRf18GSVpuTV8VPY0nzuGdTH X-Google-Smtp-Source: AGHT+IEGo7Vm5zvJp9nEVud6phcEtTDIKhjtj5nfoLZd1H8ajHYipyEgr8snqKHgp24J3b5WtV8biQ== X-Received: by 2002:a17:907:868e:b0:a7d:a29e:5c33 with SMTP id a640c23a62f3a-a7dc4fc9544mr744489266b.28.1722883237795; Mon, 05 Aug 2024 11:40:37 -0700 (PDT) Received: from localhost.localdomain (93-103-32-68.dynamic.t-2.net. [93.103.32.68]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-a7dc9c12ad3sm478517266b.88.2024.08.05.11.40.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Aug 2024 11:40:37 -0700 (PDT) From: Uros Bizjak To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Uros Bizjak , Dennis Zhou , Tejun Heo , Christoph Lameter , Andy Lutomirski , Ingo Molnar , Nadav Amit , Brian Gerst , Denys Vlasenko , "H . Peter Anvin" , Linus Torvalds , Peter Zijlstra , Thomas Gleixner , Borislav Petkov , Luc Van Oostenryck Subject: [RFC PATCH 3/3] percpu/x86: [RFH] Enable strict percpu checks via named AS qualifiers Date: Mon, 5 Aug 2024 20:39:01 +0200 Message-ID: <20240805184012.358023-4-ubizjak@gmail.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240805184012.358023-1-ubizjak@gmail.com> References: <20240805184012.358023-1-ubizjak@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch declares percpu variables in __seg_gs/__seg_fs named AS and keeps them named AS qualified until they are dereferenced with percpu accessor. This approach enables various compiler check for corss-namespace variable assignments. RFH: The patch hijacks __percpu tag and repurposes it as a named address space qualifier. While this works surprisingly well in this RFC patchset, I would really appreciate some help on how to rewrite this hack into some "production ready" code. Signed-off-by: Uros Bizjak Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Andy Lutomirski Cc: Ingo Molnar Cc: Nadav Amit Cc: Brian Gerst Cc: Denys Vlasenko Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Borislav Petkov Cc: Luc Van Oostenryck --- arch/x86/include/asm/percpu.h | 15 ++++++++++++--- include/linux/compiler_types.h | 2 +- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 4d31203eb0d2..40916bbd2f11 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -95,9 +95,18 @@ =20 #endif /* CONFIG_SMP */ =20 -#define __my_cpu_type(var) typeof(var) __percpu_seg_override -#define __my_cpu_ptr(ptr) (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr) -#define __my_cpu_var(var) (*__my_cpu_ptr(&(var))) +#if defined(CONFIG_USE_X86_SEG_SUPPORT) && defined(CONFIG_CC_HAS_TYPEOF_UN= QUAL) +# define __my_cpu_type(var) typeof(var) +# define __my_cpu_ptr(ptr) (ptr) +# define __my_cpu_var(var) (var) +# define __percpu_qual __percpu_seg_override +#else +# define __my_cpu_type(var) typeof(var) __percpu_seg_override +# define __my_cpu_ptr(ptr) (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr) +# define __my_cpu_var(var) (*__my_cpu_ptr(&(var))) +# define __percpu_qual +#endif + #define __percpu_arg(x) __percpu_prefix "%" #x #define __force_percpu_arg(x) __force_percpu_prefix "%" #x =20 diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h index f14c275950b5..47c95a06a0ec 100644 --- a/include/linux/compiler_types.h +++ b/include/linux/compiler_types.h @@ -57,7 +57,7 @@ static inline void __chk_io_ptr(const volatile void __iom= em *ptr) { } # define __user BTF_TYPE_TAG(user) # endif # define __iomem -# define __percpu BTF_TYPE_TAG(percpu) +# define __percpu __percpu_qual BTF_TYPE_TAG(percpu) # define __rcu BTF_TYPE_TAG(rcu) =20 # define __chk_user_ptr(x) (void)0 --=20 2.45.2