From nobody Sat Feb 7 18:28:40 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E59372FC890 for ; Thu, 29 Jan 2026 00:57:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648225; cv=none; b=XzP0aqQzPrJwyd+CUNoUi2mZuVMoPtHgS3NAWqbYkXofjCLJoe2awNqWCVe42pmtesWAgCK4rgGicl0VR51je1Sg6ihHpVhM0mggmmbF0HJn0GBzZdXIwxuMg9I8Zn8vkfZvmedczChn5LMsAUJPpfptlQnbwlhyZiBHXlptYxs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648225; c=relaxed/simple; bh=h+6CO9INn+2XL4U9JHeOtddCEAyJ9YJLjzkVO1qGO/c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=keY02yeEPdQs0eNL4d0jB/u0JE28xAP/iq8Q2iiIBAimObzgszoV0PWQ6Yzo/1GbruR8g7auD1iBWHGJZTtbqDBuIIr4Hybv8gewgRS98qRNgvyTltQuHuTgIpqMeLRR2TCnSqWwhjUxQQakBbqoWChDbHM/2V1eLMhwpHod5eE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QaoWXpP9; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QaoWXpP9" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-4801bceb317so3755355e9.1 for ; Wed, 28 Jan 2026 16:57:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769648221; x=1770253021; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=t0NjJX+kK82UqSM94A1K+UHi4bU7NWZzsNdcUlluZDA=; b=QaoWXpP9qSmgRYyAacsSDUClZ88ojzPy5sMFJSmfD2UcUsgJoYp5ViJFmXzvHvpG0g HRhovAkX/N+Jcf1SP/Xt0TEL51G8PGtq9WfY9rBJQE0A6jQ8Bx+bmef3/Ciom7wpJ4Xr SbYDuHQkrI+M0SvbTC5e0yKF7pkJ75699UWOmo1YV+0RhBugQxE8+ieHWoLhkCjLab4+ 6RubICnhVcbgTAYe+QNrfrlS+JKBYVF/c2vecEzOqZz2vxTX5o14omNpRq6YeQttucmk UBx/+Fw0A6k35iTFYU2WvfqOroSlCD7ZuEX/qQDvvL1zgVUgQM4wbqeHvogjo+a1uUT2 cG/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769648221; x=1770253021; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=t0NjJX+kK82UqSM94A1K+UHi4bU7NWZzsNdcUlluZDA=; b=G75QQ/MEvpP51FFv7st77EOABm+0iRZJj2Jjm+V4FXImvnZLCrU16e+RKI5grw1pb+ 0A/RzP8cicjXSjJJdXfKBEFz6zxvmVFLgmsEWkx/Ihnic0RpE22J7CqroPUjIY2KBQGD Nbl9APE87zEbT2Lpc4UquwWpkj/Jx4fiWpfTxCzMQa/niBdIR8b7ccqO+DGADSYbKIrZ NNEhjeKfocEolWbSyZEz3q4r/ZjT7vgaPQMn+ncS9E1/7FJMMj8dVWZ6kNDFU7dchnvA FK3pwmi9D4RnsMyJ75wn0usMaIrzmdZUSJ6XvvTfxkEXNLTdGt0FyimMv2rfYY42cuWI 2rqg== X-Forwarded-Encrypted: i=1; AJvYcCUARj4pUxdqCL4wD/8B33BKJjz3rOx/C92/O4OvyTSc+vfNuG5WeOPxtwcFkh5eihMQxxb3oLS2DKaiGbg=@vger.kernel.org X-Gm-Message-State: AOJu0YyygbZ9jM3lE8FiK/2kMAVHJYoGPqCkkWaWZnELAL3zdkBVBDhu MLCJwjs8nd8y5mMZvpfJC+JiN9BBDTKO3OVfopkOFVLDmDIgQ9UOIjc6nhKnpCYI21M0nsNBu1R lEg== X-Received: from wmna1.prod.google.com ([2002:a05:600c:681:b0:480:694a:dd63]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3b13:b0:480:699c:abe9 with SMTP id 5b1f17b1804b1-48069c86f58mr73598185e9.37.1769648221236; Wed, 28 Jan 2026 16:57:01 -0800 (PST) Date: Thu, 29 Jan 2026 01:52:32 +0100 In-Reply-To: <20260129005645.747680-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260129005645.747680-1-elver@google.com> X-Mailer: git-send-email 2.53.0.rc1.217.geba53bf80e-goog Message-ID: <20260129005645.747680-2-elver@google.com> Subject: [PATCH v2 1/3] arm64: Fix non-atomic __READ_ONCE() with CONFIG_LTO=y From: Marco Elver To: elver@google.com, Peter Zijlstra , Will Deacon Cc: Ingo Molnar , Thomas Gleixner , Boqun Feng , Waiman Long , Bart Van Assche , llvm@lists.linux.dev, Catalin Marinas , Arnd Bergmann , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The implementation of __READ_ONCE() under CONFIG_LTO=3Dy incorrectly qualified the fallback "once" access for types larger than 8 bytes, which are not atomic but should still happen "once" and suppress common compiler optimizations. The cast `volatile typeof(__x)` applied the volatile qualifier to the pointer type itself rather than the pointee. This created a volatile pointer to a non-volatile type, which violated __READ_ONCE() semantics. Fix this by casting to `volatile typeof(*__x) *`. With a defconfig + LTO + debug options build, we see the following functions to be affected: xen_manage_runstate_time (884 -> 944 bytes) xen_steal_clock (248 -> 340 bytes) ^-- use __READ_ONCE() to load vcpu_runstate_info structs Fixes: e35123d83ee3 ("arm64: lto: Strengthen READ_ONCE() to acquire when CO= NFIG_LTO=3Dy") Cc: Signed-off-by: Marco Elver Reviewed-by: Boqun Feng --- arch/arm64/include/asm/rwonce.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/rwonce.h b/arch/arm64/include/asm/rwonc= e.h index 78beceec10cd..fc0fb42b0b64 100644 --- a/arch/arm64/include/asm/rwonce.h +++ b/arch/arm64/include/asm/rwonce.h @@ -58,7 +58,7 @@ default: \ atomic =3D 0; \ } \ - atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(__x))__x);\ + atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(*__x) *)__x);\ }) =20 #endif /* !BUILD_VDSO */ --=20 2.53.0.rc1.217.geba53bf80e-goog From nobody Sat Feb 7 18:28:40 2026 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 078643016EE for ; Thu, 29 Jan 2026 00:57:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648227; cv=none; b=Pws/lGbqkdqlDr3h/YqPuxcBXPorqqiR2OGIHtmOUagCadvUhqOZ2FLrPacalTjBtMWXE8AKave/bNPwvOO2g+fs13jsrKK0ZpKsPR4iWLFLeogkWEBjr6TtkOECv6k15uUbfADHgmUZec3oE9L1L/E4TKyiPHvBHSAbv7zn9eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648227; c=relaxed/simple; bh=MvUOKLho2jKBfdgvboP27mM4IneJRcgFSHwRZj4mPlY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WSWASXPIv4thD/n4dHalTH8B0jnR3KNQZGW1g8PEzBOuxfTOk5Bh9ia2xDJ73t1ay1rlxsUXx8Ae2TkYesH9iQvw1+Zq2kh6TuO/iwJ5msA8cin3rIKHMvWWnOy4RhyqVvo776vUDBbER5UTCkXMlbeefgv4107uiC8JCXTX6SE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zw6HKn8V; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zw6HKn8V" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-47d4029340aso3867355e9.3 for ; Wed, 28 Jan 2026 16:57:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769648224; x=1770253024; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=39tQtLJhBsKv56fU5nhbcwIq1x9fDO/7QHDSaFgeP0Q=; b=zw6HKn8V7LMZ909gXoyjQRaRBtmfZZhjfMv143EWpY4lRefaxu893lXaKt4H5Kwdct RsSgo3vj1GmFnL8Gsa2JSjtxRU/LAYBUXoe8m/90HRpVvENuhoZCdWqTlQl6W1ukYXtJ JAFbWYoKSGbHaBx3rJTpmLYk3breGoK4p+Sm9LrIVn19pj9KE94WAdxFyGenjR5Nzblj kPSpfojxYHPfpmXNQey2m6bYGVBrztLeqBCkYKIynWIR1PYhd6OpIa+Y1ETxh8Irgg+/ PXCHA20xZMCC2sJco+3WQ+pcrnGowDjSUKKuH4U8G4/uBWPAnjsq8iccR18F2Snbvj2p 1oRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769648224; x=1770253024; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=39tQtLJhBsKv56fU5nhbcwIq1x9fDO/7QHDSaFgeP0Q=; b=k0EKTOEL/4fKMK50VuBmPPQOOkk/viZuFNPGsSf3yQ/EchItVEgV8gttHV4XI1b0Ig 98smsOMHr8rNBey3YGkJ0uUtEaaYcY74et7hr4BCGMjFNjQfF3bihD5Gi8wvfDtHuHL3 sxIlB08dPd7uEc4QYMMyW/Z5VZ6xCjagCgYy2NVgq8YqzrST8DHc+gR/bmXNplwELsT4 DYBBtaTV5xT2FIcMSjsdAMJJBEOQLvPi4wD4vYdRQNGz9nq+Q8MX1MAfDaf/Q1UUawcD vuG9zj9rpqxwKZm190UvynySWi7rkJ7dPv36sUo2L5SW+qWBZnnqGTYHvqy6rIlPkB0W RsJA== X-Forwarded-Encrypted: i=1; AJvYcCWtHLMphSFVlhBlFSJ4G1sVyzJI4Wx0opIjk63brwX1Gz06LqUevZ7dcZEerA4OE8sIjqm/YXsE0zvqYtA=@vger.kernel.org X-Gm-Message-State: AOJu0YwJ7azldm2xDK+Grg8DTkD8T53TG+kdV2RX6OnPRACmOVe4XBGB 6Br/X5N55fJj/LKi1M6R7rHJOUSrM1rnPEd4YMyB85GZBP8TCDuDgZcorPodWsKRUCNYbb6ylas 6wg== X-Received: from wmor21.prod.google.com ([2002:a05:600c:4595:b0:480:6ccb:80fd]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:8217:b0:47e:de9c:92ec with SMTP id 5b1f17b1804b1-48069c21045mr85583175e9.14.1769648224306; Wed, 28 Jan 2026 16:57:04 -0800 (PST) Date: Thu, 29 Jan 2026 01:52:33 +0100 In-Reply-To: <20260129005645.747680-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260129005645.747680-1-elver@google.com> X-Mailer: git-send-email 2.53.0.rc1.217.geba53bf80e-goog Message-ID: <20260129005645.747680-3-elver@google.com> Subject: [PATCH v2 2/3] arm64: Optimize __READ_ONCE() with CONFIG_LTO=y From: Marco Elver To: elver@google.com, Peter Zijlstra , Will Deacon Cc: Ingo Molnar , Thomas Gleixner , Boqun Feng , Waiman Long , Bart Van Assche , llvm@lists.linux.dev, Catalin Marinas , Arnd Bergmann , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rework arm64 LTO __READ_ONCE() to improve code generation as follows: 1. Replace _Generic-based __unqual_scalar_typeof() with more complete __rwonce_typeof_unqual(). This strips qualifiers from all types, not just integer types, which is required to be able to assign (must be non-const) to __u.__val in the non-atomic case (required for #2). Once our minimum compiler versions are bumped, this just becomes TYPEOF_UNQUAL() (or typeof_unqual() should we decide to adopt C23 naming). Sadly the fallback version of __rwonce_typeof_unqual() cannot be used as a general TYPEOF_UNQUAL() fallback (see code comments). One subtle point here is that non-integer types of __val could be const or volatile within the union with the old __unqual_scalar_typeof(), if the passed variable is const or volatile. This would then result in a forced load from the stack if __u.__val is volatile; in the case of const, it does look odd if the underlying storage changes, but the compiler is told said member is "const" -- it smells like UB. 2. Eliminate the atomic flag and ternary conditional expression. Move the fallback volatile load into the default case of the switch, ensuring __u is unconditionally initialized across all paths. The statement expression now unconditionally returns __u.__val. This refactoring appears to help the compiler improve (or fix) code generation. With a defconfig + LTO + debug options builds, we observe different codegen for the following functions: btrfs_reclaim_sweep (708 -> 1032 bytes) btrfs_sinfo_bg_reclaim_threshold_store (200 -> 204 bytes) check_mem_access (3652 -> 3692 bytes) [inlined bpf_map_is_rdonly] console_flush_all (1268 -> 1264 bytes) console_lock_spinning_disable_and_check (180 -> 176 bytes) igb_add_filter (640 -> 636 bytes) igb_config_tx_modes (2404 -> 2400 bytes) kvm_vcpu_on_spin (480 -> 476 bytes) map_freeze (376 -> 380 bytes) netlink_bind (1664 -> 1656 bytes) nmi_cpu_backtrace (404 -> 400 bytes) set_rps_cpu (516 -> 520 bytes) swap_cluster_readahead (944 -> 932 bytes) tcp_accecn_third_ack (328 -> 336 bytes) tcp_create_openreq_child (1764 -> 1772 bytes) tcp_data_queue (5784 -> 5892 bytes) tcp_ecn_rcv_synack (620 -> 628 bytes) xen_manage_runstate_time (944 -> 896 bytes) xen_steal_clock (340 -> 296 bytes) Increase of some functions are due to more aggressive inlining due to better codegen (in this build, e.g. bpf_map_is_rdonly is no longer present due to being inlined completely). Signed-off-by: Marco Elver --- v2: * Add __rwonce_typeof_unqual() as fallback for old compilers. --- arch/arm64/include/asm/rwonce.h | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/rwonce.h b/arch/arm64/include/asm/rwonc= e.h index fc0fb42b0b64..712de3238f9a 100644 --- a/arch/arm64/include/asm/rwonce.h +++ b/arch/arm64/include/asm/rwonce.h @@ -19,6 +19,23 @@ "ldapr" #sfx "\t" #regs, \ ARM64_HAS_LDAPR) =20 +#ifdef USE_TYPEOF_UNQUAL +#define __rwonce_typeof_unqual(x) TYPEOF_UNQUAL(x) +#else +/* + * Fallback for older compilers to infer an unqualified type. + * + * Uses the fact that auto is supposed to drop qualifiers. Unlike + * typeof_unqual(), the type must be complete (defines an unevaluated local + * variable); this must trivially hold because __READ_ONCE() returns a val= ue. + * + * Another caveat is that because of array-to-pointer decay, an array is + * inferred as a pointer type; this is fine for __READ_ONCE usage, but is + * unsuitable as a general fallback implementation for TYPEOF_UNQUAL. + */ +#define __rwonce_typeof_unqual(x) typeof(({ auto ____t =3D (x); ____t; })) +#endif + /* * When building with LTO, there is an increased risk of the compiler * converting an address dependency headed by a READ_ONCE() invocation @@ -32,8 +49,7 @@ #define __READ_ONCE(x) \ ({ \ typeof(&(x)) __x =3D &(x); \ - int atomic =3D 1; \ - union { __unqual_scalar_typeof(*__x) __val; char __c[1]; } __u; \ + union { __rwonce_typeof_unqual(*__x) __val; char __c[1]; } __u; \ switch (sizeof(x)) { \ case 1: \ asm volatile(__LOAD_RCPC(b, %w0, %1) \ @@ -56,9 +72,9 @@ : "Q" (*__x) : "memory"); \ break; \ default: \ - atomic =3D 0; \ + __u.__val =3D *(volatile typeof(*__x) *)__x; \ } \ - atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(*__x) *)__x);\ + __u.__val; \ }) =20 #endif /* !BUILD_VDSO */ --=20 2.53.0.rc1.217.geba53bf80e-goog From nobody Sat Feb 7 18:28:40 2026 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DC8182FFF9C for ; Thu, 29 Jan 2026 00:57:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648230; cv=none; b=MY5/sOVuY8LMMsUHKHuPwL+dzBLd3M+XuYdoxhDOyHuHUB7+nxy4Oyt+c+N16giDX72rAvIvRJ9/ZLV8Yc1wOeP4Au8Xqjq5ohhqykKaT2G3Vxka4X/JRblIXXhSE+qVGYShUVlIVX20cSpoxptz0fDV3hZjq+NjsqFsPRs56gE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769648230; c=relaxed/simple; bh=/t4AF7jnqhigQgZ2d4Fnqr4JkIMVuJcFMzzqPuGlCDA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=unWXtMLk3qlLIFdTLlvqXRtpCgsiVPQMLlzL4UQ7XXm309VP6rQ07jw2BCfeMf4bsZD9aXfmAKCbTTTWnie/6Ulk5B9rQ/MAa78JbnVvkHPpgvBbMVamiZTVKNZ2DDfBR90N9N/0vJ0SCYocji70iHKi9DHzP+EsjF5/ki0GGOo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HpuI5PSs; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HpuI5PSs" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b885979bfa9so32104266b.1 for ; Wed, 28 Jan 2026 16:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1769648227; x=1770253027; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mEepWOxCVSc1aX29GPbB6p8VltBEKBk11LnhK9lypjY=; b=HpuI5PSs0rYDV2HjyYvQgQUHtDl2A3VUGAJ114hRV+1/0M4LXZ2v7A/hfXUWUEQ2UG vNxi+2n0PJ3Pd+VOMKG4JyR5gm2ywFPf9gt6XBhK67lVDwcstv9MKFspx9LSUKyH1PFA 53+9HQlHn3cpMl2zk9NWElAB40cVnQuAeK+fWFIaOmQH4p89B07TnADzDtIsEByFPy9W uBQRii1MYLv7Zi0G+ugK4lybBKCJZdrX7et02pvxkr4Sq22WNOi/gmJAudFr3sFL9iq7 4AAlYbaZXUwREUp34QMl6rcPIUR0jkZzS005G9DRRuWlt7oFJ/9d76VAt1lewbsQ05b3 C04Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1769648227; x=1770253027; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mEepWOxCVSc1aX29GPbB6p8VltBEKBk11LnhK9lypjY=; b=mkKKEX98HMFgvZN5eAmT9kXPyJEGJlQTxjzXkbFGrI3SNx1SfnkwRJILvR1A2VkL3B WoHxebZGfx/17QbTq1X7p0XlqbkXDYjDnimwULQm9knRCJeAI1d3aoulT2rWYWfmNYiz NFTMOGO+50FsVkZfnuoPwbABrv8HCF3T3uLAvlEjyZc0UFSS/5s4K0TtSTjiOXLhOb+Y /UCu46HG8M1er3HnpsOhNJqTbjRizPcdZOMiy9RH6b6CfA2Kon87fWvGLMBytWGjPgbe ETEUf1/XgkwQJnbT1vsILRST1SgGpHijr8ukctoyxQRXhM4Ox3WqLz5Kt7h1hVbEEk9Q Yn6w== X-Forwarded-Encrypted: i=1; AJvYcCV9kNdO/qk8EAAZahWXf9hxtFH9XEZDqYgpa+udUA7ESFIjcf3vEEB+GZFK9QpmQsGNJQWh1QUeIHoSluU=@vger.kernel.org X-Gm-Message-State: AOJu0YyfoAWsQfGovn8VFxY15nPl6k1VU9dl5g9AFv74i6Cterqw7Lvw YSS4/6r9ZNJK8gHWR2RvMmRIo3ewd3s6JeKpZxh2/6Up8iWxtxVKJCI5BgIjwUudo/0RWGjhkU7 FFg== X-Received: from ejbdt12.prod.google.com ([2002:a17:906:b78c:b0:b87:2981:bb4b]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:97d1:b0:b87:1d79:bef4 with SMTP id a640c23a62f3a-b8dab130284mr439369266b.9.1769648227185; Wed, 28 Jan 2026 16:57:07 -0800 (PST) Date: Thu, 29 Jan 2026 01:52:34 +0100 In-Reply-To: <20260129005645.747680-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260129005645.747680-1-elver@google.com> X-Mailer: git-send-email 2.53.0.rc1.217.geba53bf80e-goog Message-ID: <20260129005645.747680-4-elver@google.com> Subject: [PATCH v2 3/3] arm64, compiler-context-analysis: Permit alias analysis through __READ_ONCE() with CONFIG_LTO=y From: Marco Elver To: elver@google.com, Peter Zijlstra , Will Deacon Cc: Ingo Molnar , Thomas Gleixner , Boqun Feng , Waiman Long , Bart Van Assche , llvm@lists.linux.dev, Catalin Marinas , Arnd Bergmann , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kernel test robot Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When enabling Clang's Context Analysis (aka. Thread Safety Analysis) on kernel/futex/core.o (see Peter's changes at [1]), in arm64 LTO builds we could see: | kernel/futex/core.c:982:1: warning: spinlock 'atomic ? __u.__val : q->loc= k_ptr' is still held at the end of function [-Wthread-safety-analysis] | 982 | } | | ^ | kernel/futex/core.c:976:2: note: spinlock acquired here | 976 | spin_lock(lock_ptr); | | ^ | kernel/futex/core.c:982:1: warning: expecting spinlock 'q->lock_ptr' to b= e held at the end of function [-Wthread-safety-analysis] | 982 | } | | ^ | kernel/futex/core.c:966:6: note: spinlock acquired here | 966 | void futex_q_lockptr_lock(struct futex_q *q) | | ^ | 2 warnings generated. Where we have: extern void futex_q_lockptr_lock(struct futex_q *q) __acquires(q->lock_ptr= ); .. void futex_q_lockptr_lock(struct futex_q *q) { spinlock_t *lock_ptr; /* * See futex_unqueue() why lock_ptr can change. */ guard(rcu)(); retry: >> lock_ptr =3D READ_ONCE(q->lock_ptr); spin_lock(lock_ptr); ... } The READ_ONCE() above is expanded to arm64's LTO __READ_ONCE(). Here, Clang Thread Safety Analysis's alias analysis resolves 'lock_ptr' to 'atomic ? __u.__val : q->lock_ptr', and considers this the identity of the context lock given it can't see through the inline assembly; however, we simply want 'q->lock_ptr' as the canonical context lock. While for code generation the compiler simplified to __u.__val for pointers (8 byte case -> atomic), TSA's analysis (a) happens much earlier on the AST, and (b) would be the wrong deduction. Now that we've gotten rid of the 'atomic' ternary comparison, we can return '__u.__val' through a pointer that we initialize with '&x', but then change with a pointer-to-pointer. When READ_ONCE()'ing a context lock pointer, TSA's alias analysis does not invalidate the initial alias when updated through the pointer-to-pointer, and we make it effectively "see through" the __READ_ONCE(). Code generation is unchanged. Link: https://lkml.kernel.org/r/20260121110704.221498346@infradead.org [1] Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-kbuild-all/202601221040.TeM0ihff-lkp@int= el.com/ Cc: Peter Zijlstra Signed-off-by: Marco Elver Tested-by: Boqun Feng --- v2: * Rebase. --- arch/arm64/include/asm/rwonce.h | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/rwonce.h b/arch/arm64/include/asm/rwonc= e.h index 712de3238f9a..3a50a1d0d17e 100644 --- a/arch/arm64/include/asm/rwonce.h +++ b/arch/arm64/include/asm/rwonce.h @@ -48,8 +48,11 @@ */ #define __READ_ONCE(x) \ ({ \ - typeof(&(x)) __x =3D &(x); \ + auto __x =3D &(x); \ + auto __ret =3D (__rwonce_typeof_unqual(*__x) *)__x; \ + auto __retp =3D &__ret; \ union { __rwonce_typeof_unqual(*__x) __val; char __c[1]; } __u; \ + *__retp =3D &__u.__val; \ switch (sizeof(x)) { \ case 1: \ asm volatile(__LOAD_RCPC(b, %w0, %1) \ @@ -74,7 +77,7 @@ default: \ __u.__val =3D *(volatile typeof(*__x) *)__x; \ } \ - __u.__val; \ + *__ret; \ }) =20 #endif /* !BUILD_VDSO */ --=20 2.53.0.rc1.217.geba53bf80e-goog