From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA8EE25DCF0 for ; Mon, 3 Mar 2025 16:53:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020796; cv=none; b=RQVTPCzbrOPI2yyi942D+ihSPo3mKXF9cvOfnBTuIjlettdioDbK6d1KcKdGAU21xZy4swvIiqFdXxhyTPeRR0vf6qQHml4j8GEtuuZNgtNg/gik+m8sUweTeFUG40oInSeS3gNA0yrJpiYo4Uv7rv1Pt+pdHn7wCGvP6Sd85SA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020796; c=relaxed/simple; bh=PiXmU6lUBB5AxY09cRXKal52ViVGG4MfYTE3SZPxqoE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AlY1UdED31e0Ty74fOLC6j6Cx8C6QIwRcYFZK5R1TbNgVFJnILQ2Q7eK/H/ez5AL0nen0o3A7bgR7W+d5K/7n0WOEL+K2nC1Ns+IHM5kN4hrchMByl0AXES0au4BHKUZdxCy4/btM8Ns52lBvwq1ti3sG6PhCMZxktm9C3ya91o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ATpBB0s5; arc=none smtp.client-ip=209.85.219.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ATpBB0s5" Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-6dd01781b56so55034586d6.0 for ; Mon, 03 Mar 2025 08:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020792; x=1741625592; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=x3glzTjf8IgiUSBtnGndwG/CjLpV6ZT+iGSPTYBmwgQ=; b=ATpBB0s5xF0/1c+udxIXo9f+zN/V7gMJwZPknMHBivZVqeQgq/w8YMuWOoAMPVjoZo kIWk9S/CSi1613gQv0Q0KtRLXyZmS1iVCg9b+b1UfuR2nSV9HpNvUke0hXY4wL0LHMLj Zsu0lol6PlI9DnYGP0iCxHjbtfb0CC4dzOp+YUPawj8kFdmiLEndEJxJ9TaJjkZQ8udH qDvSsH4D3ROdAT3AUm1keXTWPUDuRPqgvxvv86R1qkVIFTmJkBu+Rl79vnvy+Tz2Yz/V ZJfmXjDEzxsBIaOijyPOUqyDmgVSyv4TeZbb0SVd0n08Cm6r7pvL5N12MHFpqdFLYDWB VWEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020792; x=1741625592; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x3glzTjf8IgiUSBtnGndwG/CjLpV6ZT+iGSPTYBmwgQ=; b=dfoe8LciiviF4C6agExAmmFflKyr/7fUEYFQHMb+FCrBfuvjw8/ZgSL0sTUY5tlPOF 56DVPF6rkMMNmR2FjmXRFxHc0I4qQXwaRx7w3aWIEOipwDEDLKHXmVhDDHfpdOOcsDbo lhOcze6WKgEntrUvcFRQhidjB9k4+YpK9OJQfwgr9sMS7t13JnK8Yxe/poe6fQwwA6e8 3kL/BTZRQ8EILOs2xpEctMtDeo2I8VX6YN6w8Vs6uJqdC0vPH3zZLzw4kUFxdXC/QDeK RX5Zt1LAOXxYivkmz79deOYdqy/Tcq6DatmS7S9jObPnSz9yNjUijSCBt0A8qpFa1IVJ CNdA== X-Gm-Message-State: AOJu0YyS+VwHit3EginEZgmSd31rHgdYFv8tNG6SJChudbisod/NZ6Ki sKuMTcWygMViRyXvxeHMFat9EyGVJK83NE/uYUB7DtRIA4lcm3EZlkO5 X-Gm-Gg: ASbGncsSMoE5S/bj3vyqqzOMc4qteXZGlPleB2WNBKbNWXr+9yMtjc1XlJ7zHyxm9nP +RW++Els8e3ZP3BU1z6f1GhfHnRdl8684OZs+Il9hYKJ1o1je7E3geLdxdJ6TY2psnBrrU0+RaT U6+jA+rq3Vp+pOqjyb6uLSRXyF+Jm+H+H4d0gUwT4zkNLf4Y6YQL/MUuw9cTTnkXnExd0Vx8R/t k7joIokxWL06WItVjJeMi7mX8nfyMHh3787KJkjC+T1+lolDZoefemrlcI0AlLzQFZ8p+mjixZ7 a0UOUoWD4LrqucUxJysT8GnG0A== X-Google-Smtp-Source: AGHT+IFZQ88ZlYfzX2N54kSWRFQ+paaHStvVgF3kPHPVSggP3M6SGgBSnH7+SCEx/2ZbKT/ghbDPMg== X-Received: by 2002:a05:6214:21e7:b0:6e4:9b59:d9ce with SMTP id 6a1803df08f44-6e8a0c7fdf6mr213591196d6.5.1741020791886; Mon, 03 Mar 2025 08:53:11 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:10 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 01/11] percpu: Introduce percpu hot section Date: Mon, 3 Mar 2025 11:52:36 -0500 Message-ID: <20250303165246.2175811-2-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a subsection to the percpu data for frequently accessed variables that should remain cached on each processor. These varables should not be accessed from other processors to avoid cacheline bouncing. This will replace the pcpu_hot struct on x86, and open up similar functionality to other architectures and the kernel core. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- include/asm-generic/vmlinux.lds.h | 11 +++++++++++ include/linux/percpu-defs.h | 13 +++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 85fc0524f161..4ed0e6a013d0 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -385,6 +385,11 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPEL= LER_CLANG) . =3D ALIGN(PAGE_SIZE); \ __nosave_end =3D .; =20 +#define CACHE_HOT_DATA(align) \ + . =3D ALIGN(align); \ + *(SORT_BY_ALIGNMENT(.data..hot.*)) \ + . =3D ALIGN(align); + #define PAGE_ALIGNED_DATA(page_align) \ . =3D ALIGN(page_align); \ *(.data..page_aligned) \ @@ -1064,6 +1069,11 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROP= ELLER_CLANG) . =3D ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . =3D ALIGN(cacheline); \ + __per_cpu_hot_start =3D .; \ + *(SORT_BY_ALIGNMENT(.data..percpu..hot.*)) \ + __per_cpu_hot_pad =3D .; \ + . =3D ALIGN(cacheline); \ + __per_cpu_hot_end =3D .; \ *(.data..percpu..read_mostly) \ . =3D ALIGN(cacheline); \ *(.data..percpu) \ @@ -1111,6 +1121,7 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPE= LLER_CLANG) INIT_TASK_DATA(inittask) \ NOSAVE_DATA \ PAGE_ALIGNED_DATA(pagealigned) \ + CACHE_HOT_DATA(cacheline) \ CACHELINE_ALIGNED_DATA(cacheline) \ READ_MOSTLY_DATA(cacheline) \ DATA_DATA \ diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 40d34e032d5b..0fcacb909778 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -112,6 +112,19 @@ #define DEFINE_PER_CPU(type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") =20 +/* + * Declaration/definition used for per-CPU variables that are frequently + * accessed and should be in a single cacheline. + * + * For use only by architecture and core code. Only use scalar or pointer + * types to maximize density. + */ +#define DECLARE_PER_CPU_CACHE_HOT(type, name) \ + DECLARE_PER_CPU_SECTION(type, name, "..hot.." #name) + +#define DEFINE_PER_CPU_CACHE_HOT(type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..hot.." #name) + /* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of t= he --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A10825DCF7 for ; Mon, 3 Mar 2025 16:53:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020797; cv=none; b=I2Y57LkTvVnZBMq3PPEWzArHHlBl1Qu/+P6X7dhtTB2k1uFI2wj4qIQl5+j4qG1HMPN/e+wakEBbVpj1iTCzUnR/1Cpi240F5Paf7ybjwHkSEfaHUUeWybz6Eusmiyuy/wjJRcq98GI2HZBwn4myaao8gb+I9OBWT3cc4we0tRc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020797; c=relaxed/simple; bh=ie9S/8MDQErq++hnfAmXc+Lw3UbKnIj84zN+yIyykiE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ms8o/1SroNSqNQbGUgN5QbSwLg4PpFT1xK+KMP+jdEEa8wP7pd+e8oJtHY7Jo6DF138f8YHmikYXOFgAKm0sb+tCshazIm/sK98gyizaROkJPpm66lQyYOpf/HBYVDkZDBS+UH8rdKTke5rk5hr6m7exzv9MLmtJuZP3JX5+/Pw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EXsTR8l9; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EXsTR8l9" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7c0ac2f439eso571675585a.0 for ; Mon, 03 Mar 2025 08:53:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020793; x=1741625593; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aXpU/uYR4oxAkP1tzBNe+du+KzjtuWOQI3OyRLVRl8s=; b=EXsTR8l9BM3+DJxW/GeJB/wWUZsachVbJVo2NdFfM3hsHrIwch0r/YWepCNxZ1zsrL eeYFfTPSaa0xS5EBU5TVDHe2YJrDXfHQnqRrx05n3SKTbLFjtluvfZMVhzuF9ZsmmIMv Q8QJLxwmNRdeez7yWL2m4APrbZhN8sLUiPAvts/SMLNxJeUYRtcIPvE8OQbx5ggVgx/0 Cyl29zMxaWVcnP7mOIGQHycY26h1h+LMyhDDc49f6x2+T4x65SDem2Je0qEX8L8iGRac h7qkxmIZDk24yNTCx+dIRhD7Wwlo0dqLPnI5dQNoqLtUb+lksY8+PA2BhqUBwwtk6I+X oB6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020793; x=1741625593; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aXpU/uYR4oxAkP1tzBNe+du+KzjtuWOQI3OyRLVRl8s=; b=goZXXoKGPFoKV+HQc22SPbTjdU9f+/+WCTunyBUBUpfBt9E1IhhO79Tuepx5Rk+64r RUj2Jpn5QpjMx5ohbFyJ/eZk/sWY3+GIKBY2K+N/1MARDO15qIelZCAyRaTeBfRAoytA rHXHPQZLJRYhWUmrb6mIYwZp6PQyo9aasmT28KH9oruBDXcVwENvKzWjTPbPuQs9vWxR G0Np3Yd0xpJTpXmWl2mlpkccVVCBeORT5Z+eMrnjkUqm3RdGjcL+gDl7/4v9c1m5kVyU wgCZmvetxVMGQfNwvMp7Gn3XfirJa6dQYTbXPPRE9KIJ96ZqoHdMPq9PWqxsV+M0h5aA QuBQ== X-Gm-Message-State: AOJu0Yw97C5sl8+VpqKqBFpi5a7F3yX/BQ4V7gA71BXMofFWgRgWuvng upozIv7hePjUvF30IoVkPMkT/b1zLbI0ZY6YyM7mM+moplPUQ9fAuQug X-Gm-Gg: ASbGncuol35HSrlmAYHB/D9TB9d119bowVo7ME0WR/Jg0vZ9RSYWJZ81cBdnm7PimfB c+WC1rRDxKHC6ZSko/nxmzPU/rInX+KZ6iy+gDUf/jJqKcv5tBAXvR5ZUF1gnP99uInQttcZw1S iujBp52iYh1qN23Uspz1Bpd2kH5NTAuUe7/0lkt3W8WWmWv/A41V3AsFd0P4v9FCUgQTkt1x7GK GDT3tYkg2MBT8k6sNDDB2rlEal6F7qCPB9m/TrShhQBF8ZZjT2LrpMpSWyYtxRLnd94RC65XJ7W ofZgOIqZLSTA2gnFWNTMcZ+GZw== X-Google-Smtp-Source: AGHT+IEfdZpFbpSpMDgGCY6ETqFQtXvQfYD1sMc7RZZLk2OOuuz+yvpinSd1ztTERkLeFeCanmi1mg== X-Received: by 2002:a05:620a:3726:b0:7c0:a357:fe62 with SMTP id af79cd13be357-7c39c4be40fmr1871198385a.19.1741020793336; Mon, 03 Mar 2025 08:53:13 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:12 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 02/11] x86/percpu: Move pcpu_hot to percpu hot section Date: Mon, 3 Mar 2025 11:52:37 -0500 Message-ID: <20250303165246.2175811-3-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 28 +++++++++++----------------- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 3 +++ 3 files changed, 15 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index bf5953883ec3..60bc66edca83 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -13,32 +13,26 @@ struct task_struct; =20 struct pcpu_hot { - union { - struct { - struct task_struct *current_task; - int preempt_count; - int cpu_number; + struct task_struct *current_task; + int preempt_count; + int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - u64 call_depth; + u64 call_depth; #endif - unsigned long top_of_stack; - void *hardirq_stack_ptr; - u16 softirq_pending; + unsigned long top_of_stack; + void *hardirq_stack_ptr; + u16 softirq_pending; #ifdef CONFIG_X86_64 - bool hardirq_stack_inuse; + bool hardirq_stack_inuse; #else - void *softirq_stack_ptr; + void *softirq_stack_ptr; #endif - }; - u8 pad[64]; - }; }; -static_assert(sizeof(struct pcpu_hot) =3D=3D 64); =20 -DECLARE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); +DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); =20 /* const-qualified alias to pcpu_hot, aliased by linker. */ -DECLARE_PER_CPU_ALIGNED(const struct pcpu_hot __percpu_seg_override, +DECLARE_PER_CPU_CACHE_HOT(const struct pcpu_hot __percpu_seg_override, const_pcpu_hot); =20 static __always_inline struct task_struct *get_current(void) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 7d33c45ad241..c33eaca1e9f9 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2049,7 +2049,7 @@ static __init int setup_setcpuid(char *arg) } __setup("setcpuid=3D", setup_setcpuid); =20 -DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { +DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, .preempt_count =3D INIT_PREEMPT_COUNT, .top_of_stack =3D TOP_OF_INIT_STACK, diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 1769a7126224..7586a9be8c59 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -187,6 +187,8 @@ SECTIONS =20 PAGE_ALIGNED_DATA(PAGE_SIZE) =20 + CACHE_HOT_DATA(L1_CACHE_BYTES) + CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES) =20 DATA_DATA @@ -328,6 +330,7 @@ SECTIONS } =20 PERCPU_SECTION(INTERNODE_CACHE_BYTES) + ASSERT(__per_cpu_hot_end - __per_cpu_hot_start <=3D 64, "percpu cache hot= section too large") =20 RUNTIME_CONST_VARIABLES RUNTIME_CONST(ptr, USER_PTR_MAX) --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F1FF25E446 for ; Mon, 3 Mar 2025 16:53:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020799; cv=none; b=Gd8fFJix/hbfAgFITIqtThj/s58q+o/kaKzB/UQSOAkzLdLdqkNJ3saBFOgG+JKTiSW8ABPJvIfkqDE06QjYDTuI7yKajo6Z+kX+cfkjg8vk6t8pesO1tlAFnuJBTnNbpslzBnOJNnQjvxjio2mBWpjXSmhxan/F4jjJYqQ3lmQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020799; c=relaxed/simple; bh=rSUNvriQSjVIpywPkCIL0ssJzw1abFGSQqq9fcCN0Bk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I3W+CFYQ7j+mMH8EnRL3IPwjb5X99E8rsoEKIytwASat7ZoPqeY3aOT9lxOM3jppMXWP66Yk4IfXCVGxS63E1FnbIGj66ZBhh3F/xtzFfIk73puFq9J4LgCtN7lhxAcXrNjSX0YgvvNpdqn/i8IstskhS06Q8wSGrCyEARtBKCE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=YX5e86AF; arc=none smtp.client-ip=209.85.219.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YX5e86AF" Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6dd420f82e2so66099446d6.1 for ; Mon, 03 Mar 2025 08:53:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020795; x=1741625595; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9iNgkHr5bj85f6gaDanw+SgUtlzoAy3q+atablztih4=; b=YX5e86AFC6bgQYgKTZN7twHw1VRn0993YF8fLkWbzYWwjb+Wgk6KQBK8XVjT5+jw8T j4bmkp18XoUPs4slOPyQcKL4ZX2Eg3TrgUzr0Rw83lW4Ki4/qg1XQ1Icsx8pksnPaHEI WoPxD0M0259MGKo3xGA4cZ1hbNMnmLVYhPwz79cENzkFHmGp2/fkOoshYJ+Ol3pWGNEv dSuEpZ1qXj+V8Ur6sY8m+TYE/8q8Fpg5S6SahsRJbuE41Jx9Y3VEtI6i9YCz533cXAxT OoQt7tXSsJsZCj2A/ue7/k1DN1Ne5b5Jj0ociMnALsisjgVtf55DVEPAMIPmya3IJAmU Tq/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020795; x=1741625595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9iNgkHr5bj85f6gaDanw+SgUtlzoAy3q+atablztih4=; b=nIgGnZ5bttl2gQ5cU63jSrnJm3BmR9ViC/3gOr5qf5Q19AERBgRtKCtTeEV1+IXmwn /EZV9PAMIM2Env9fsEvCCI18W9Dbz9sDx4dEgTa/QEPBBAVRPOQEI0VcbnOtEJ+REBQO 8hoKHzq1f0f4EUrSDQYmivZgHwHOOoodzwj6mTE8rJEfMuPa4hMiGK/33rrxOnTMYu7F 0RbGEgw3/5b1KHB8UZbqFltyF2moSvxVD2f3r518gxoyWTHqPh0qkW0/gB+Xhhn5v9bO I6X2uyu0XSQ5hmUYrx5tCHcs93v7XbVxl/NZ5U7Yl+sR+SovUfYCiibZtU29uvLWJaju 8sTg== X-Gm-Message-State: AOJu0YxHvRY58NMiKxEWRNCFlgR1xaYRqW0B8EHX8PUfI9zlbjcDD3uz Vigp7dFEPITbaNkpRPD0RsW+Yw6c8WmO0lAHKsIiXBo5lD3iKZEf5Fc1 X-Gm-Gg: ASbGnctuf8L4HPLiuP1LhtbSOatoPcW2wl3dTygfV5Z4bNTdKhnbzKdbIKqNdfnihru b09k4upAJyIWMFSYrSeEAIi+YUb8+ZGfgwJcMfbl+vCYp0RDTvp/hDP86tZSeXZPvMe5Ys7Z7aG Q1Hhec7pcUsi8if8ToEcA1ozZhy/mancUQQlsw2sYxYCy6QJ4QraDXGKfjFKu5BpTB+RatS4XN+ QAu7DM6uVN29M7NpT77OdyVjb8IcmAk7uuB3Vfr6lAMkxAcHl2Rpt84wxxU5tse/iCHwlLnZTUH iZH+xVg4QazvwoqZT+wXTPa0Eg== X-Google-Smtp-Source: AGHT+IHy8HDzYv7FlGVafnjXASHLEtkIcUC087TOMcpwjRQRGkBS+ZNPs3S26DW1plbiHblSx/3Y3w== X-Received: by 2002:a05:6214:19ef:b0:6d8:9062:6616 with SMTP id 6a1803df08f44-6e8a0c80a4cmr233960906d6.7.1741020795433; Mon, 03 Mar 2025 08:53:15 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:14 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 03/11] x86/preempt: Move preempt count to percpu hot section Date: Mon, 3 Mar 2025 11:52:38 -0500 Message-ID: <20250303165246.2175811-4-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/preempt.h | 25 +++++++++++++------------ arch/x86/kernel/cpu/common.c | 4 +++- include/linux/preempt.h | 1 + 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 60bc66edca83..46a736d6f2ec 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - int preempt_count; int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 7e834820e030..a1748b4e5598 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -4,10 +4,11 @@ =20 #include #include -#include =20 #include =20 +DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); + /* We use the MSB mostly because its available */ #define PREEMPT_NEED_RESCHED 0x80000000 =20 @@ -23,18 +24,18 @@ */ static __always_inline int preempt_count(void) { - return raw_cpu_read_4(pcpu_hot.preempt_count) & ~PREEMPT_NEED_RESCHED; + return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED; } =20 static __always_inline void preempt_count_set(int pc) { int old, new; =20 - old =3D raw_cpu_read_4(pcpu_hot.preempt_count); + old =3D raw_cpu_read_4(__preempt_count); do { new =3D (old & PREEMPT_NEED_RESCHED) | (pc & ~PREEMPT_NEED_RESCHED); - } while (!raw_cpu_try_cmpxchg_4(pcpu_hot.preempt_count, &old, new)); + } while (!raw_cpu_try_cmpxchg_4(__preempt_count, &old, new)); } =20 /* @@ -43,7 +44,7 @@ static __always_inline void preempt_count_set(int pc) #define init_task_preempt_count(p) do { } while (0) =20 #define init_idle_preempt_count(p, cpu) do { \ - per_cpu(pcpu_hot.preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ + per_cpu(__preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ } while (0) =20 /* @@ -57,17 +58,17 @@ static __always_inline void preempt_count_set(int pc) =20 static __always_inline void set_preempt_need_resched(void) { - raw_cpu_and_4(pcpu_hot.preempt_count, ~PREEMPT_NEED_RESCHED); + raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED); } =20 static __always_inline void clear_preempt_need_resched(void) { - raw_cpu_or_4(pcpu_hot.preempt_count, PREEMPT_NEED_RESCHED); + raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED); } =20 static __always_inline bool test_preempt_need_resched(void) { - return !(raw_cpu_read_4(pcpu_hot.preempt_count) & PREEMPT_NEED_RESCHED); + return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED); } =20 /* @@ -76,12 +77,12 @@ static __always_inline bool test_preempt_need_resched(v= oid) =20 static __always_inline void __preempt_count_add(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, val); + raw_cpu_add_4(__preempt_count, val); } =20 static __always_inline void __preempt_count_sub(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, -val); + raw_cpu_add_4(__preempt_count, -val); } =20 /* @@ -91,7 +92,7 @@ static __always_inline void __preempt_count_sub(int val) */ static __always_inline bool __preempt_count_dec_and_test(void) { - return GEN_UNARY_RMWcc("decl", __my_cpu_var(pcpu_hot.preempt_count), e, + return GEN_UNARY_RMWcc("decl", __my_cpu_var(__preempt_count), e, __percpu_arg([var])); } =20 @@ -100,7 +101,7 @@ static __always_inline bool __preempt_count_dec_and_tes= t(void) */ static __always_inline bool should_resched(int preempt_offset) { - return unlikely(raw_cpu_read_4(pcpu_hot.preempt_count) =3D=3D preempt_off= set); + return unlikely(raw_cpu_read_4(__preempt_count) =3D=3D preempt_offset); } =20 #ifdef CONFIG_PREEMPTION diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c33eaca1e9f9..8db36699dc78 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2051,12 +2051,14 @@ __setup("setcpuid=3D", setup_setcpuid); =20 DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .preempt_count =3D INIT_PREEMPT_COUNT, .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); =20 +DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; +EXPORT_PER_CPU_SYMBOL(__preempt_count); + #ifdef CONFIG_X86_64 static void wrmsrl_cstar(unsigned long val) { diff --git a/include/linux/preempt.h b/include/linux/preempt.h index ca86235ac15c..4c1af9b7e28b 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -319,6 +319,7 @@ do { \ #ifdef CONFIG_PREEMPT_NOTIFIERS =20 struct preempt_notifier; +struct task_struct; =20 /** * preempt_ops - notifiers called when a task is preempted and rescheduled --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D2A6825E828 for ; Mon, 3 Mar 2025 16:53:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020802; cv=none; b=XhrHBrf67Tsu97k2MSJYpoCZcXP0CSPIBo6ZYPm61plFf2KovBEhgHuAtt7+p9QKo2jY4XN2igP7SA3ofBpmU41Me9DAsYjCFdUuZDu5p40N6qlFiiCwYZm6kfj7hpNU9Lnknk2i33g5EhXnN5FP6M+r7ZNBucNEhluPqsTnB+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020802; c=relaxed/simple; bh=52lOENL9T0/nUWxyw+B5/VLu3eMD9wCD80R7uWLaNBQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aJl9KNnMK0AHWGpTZ0gpgVQIJdZkemitGXLQJF7SR9lT8Q+/PTtpX1MMHbgiCLhQqzJSYJbn05EW4IhUlJUl8WLa3UhiVEUlgUDeo49FIbQNjbToe8yE+178IW80lwASbxCeUiU6ydoAFQYDSMY95+x0zFMou3i+y/8gfi3SZI8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WuZfK4Q3; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WuZfK4Q3" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6e6c082eac0so44381896d6.0 for ; Mon, 03 Mar 2025 08:53:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020798; x=1741625598; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZFL4zyI1qItSJ1SO+PACSRk4BcOG2Axak54Fldvg7PY=; b=WuZfK4Q3SGVL/VXZc4bDcjUht2CqBFrcTRI2YbvITYiiIM48EzDi89XrfOcav4FfJ/ 9Wbeka9iLYzWgrGbsL4eeSWm0cZqfTayc+BmnjlJJKa7jZ/CAjlpzmwTj/pySmSGzvfW fqCJkTmmTx1gR3zbtP6g2Tk8yKPwnn/XmFKl4rvcFIDegAUBDjEcebw27KP4Ue92yAIi wvrCtq1Z2akqXFr+dNN5BiOTG40MAeF8kRrdpO4YMeOlpGYa/O7pMSEdtScA20jMSmH9 CYi9jo9jcD/KgbMO96lIXD0GTy/Iz+Ruw+WQHPtvY6LGSbtS4iLzcWNQcMdzDXdVYPog faxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020798; x=1741625598; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZFL4zyI1qItSJ1SO+PACSRk4BcOG2Axak54Fldvg7PY=; b=pXabArvP9y/FoCxGNVgt/C+V31RdvJ+M1SU6dKa2wNhc5Ztnk5G7m19+9A9Ws+JvnK oZyU4U9rHP45toixtdRtubzzkFXYCu+gx2Urejk65PgXNeXEHfgj7CCOHp4UaduA8YOP LZG5FBlPi/14vaRyeTRqLanFvRN/WFYus32m89bnw90QcBUmzu7v+RUHCCj6nKLf3TyV 0ogr0wN+MkqPBhhR/XIlceExD3KNuGVRWkfohBaN3719w6EsvaVJyWcG2bjsfLuRH1wc 0oswqeHi2yDvtSRAYOMkO96Ybhhq/9kqbcNsnNxU74xzRvRifhNI79B4DEDFo0riMh3D 1Blg== X-Gm-Message-State: AOJu0YwJIdledxy/wEkd1VCRBhtQBQCk/aBhz1QAUyVLBysxJBBnFO5u Tw4z9A24Ef2BMleZ3G547MLH7J+CHmn7ajRaOJtcEX0f3DWKNDZq0yAA X-Gm-Gg: ASbGncsxRs6fQGZL8EXBJ3QEDTVMFMdoJjhKUNSWTQYFOqp9A+WvnFulbcavp75Om7+ a095T7Ncb/ccwZjuQaEHneo2CusfXQvPUp/Z72FykhE8brD/30otLTLEaWJ8KeagcTZzWFBm/2c /f66jBuvPOpwC1fnpa+al5rpKnLUqb+XwMPD2ll0kIKGL7YeDrWNwPIs3s29H6TXQ+t7zsThpnt /stilkEWUM+GtfbXnzmx/FcYXIOmqbE6xEi14kydgb1AEG3wT0AIuleIEiYF1zE2kOqWCbwcg+x jJ7tA35IZy6brpX2hmj+nmIbqQ== X-Google-Smtp-Source: AGHT+IHDnLn/Gr54Fas6oQVfUD7K0AwWpBjA1WqJc6unF9/StNKY4QS33NWtH98wHBAqsEmoTF2f+A== X-Received: by 2002:ad4:5eca:0:b0:6e5:a0fc:f663 with SMTP id 6a1803df08f44-6e8a0acbcccmr214109246d6.0.1741020798156; Mon, 03 Mar 2025 08:53:18 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:16 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 04/11] x86/smp: Move cpu number to percpu hot section Date: Mon, 3 Mar 2025 11:52:39 -0500 Message-ID: <20250303165246.2175811-5-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/smp.h | 7 ++++--- arch/x86/kernel/setup_percpu.c | 5 ++++- kernel/bpf/verifier.c | 4 ++-- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 46a736d6f2ec..f988462d8b69 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; #endif diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index c8508d78ef3e..d9d849535291 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -6,7 +6,8 @@ #include =20 #include -#include + +DECLARE_PER_CPU_CACHE_HOT(int, cpu_number); =20 DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map); @@ -134,8 +135,8 @@ __visible void smp_call_function_single_interrupt(struc= t pt_regs *r); * This function is needed by all SMP systems. It must _always_ be valid * from the initial startup. */ -#define raw_smp_processor_id() this_cpu_read(pcpu_hot.cpu_number) -#define __smp_processor_id() __this_cpu_read(pcpu_hot.cpu_number) +#define raw_smp_processor_id() this_cpu_read(cpu_number) +#define __smp_processor_id() __this_cpu_read(cpu_number) =20 #ifdef CONFIG_X86_32 extern int safe_smp_processor_id(void); diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 1e7be9409aa2..175afc3ffb12 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -23,6 +23,9 @@ #include #include =20 +DEFINE_PER_CPU_CACHE_HOT(int, cpu_number); +EXPORT_PER_CPU_SYMBOL(cpu_number); + DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 @@ -161,7 +164,7 @@ void __init setup_per_cpu_areas(void) for_each_possible_cpu(cpu) { per_cpu_offset(cpu) =3D delta + pcpu_unit_offsets[cpu]; per_cpu(this_cpu_off, cpu) =3D per_cpu_offset(cpu); - per_cpu(pcpu_hot.cpu_number, cpu) =3D cpu; + per_cpu(cpu_number, cpu) =3D cpu; setup_percpu_segment(cpu); /* * Copy data used in early init routines from the diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f4859516b190..6e604caa870c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -21702,12 +21702,12 @@ static int do_misc_fixups(struct bpf_verifier_env= *env) if (insn->imm =3D=3D BPF_FUNC_get_smp_processor_id && verifier_inlines_helper_call(env, insn->imm)) { /* BPF_FUNC_get_smp_processor_id inlining is an - * optimization, so if pcpu_hot.cpu_number is ever + * optimization, so if cpu_number is ever * changed in some incompatible and hard to support * way, it's fine to back out this inlining logic */ #ifdef CONFIG_SMP - insn_buf[0] =3D BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&pcpu_hot.= cpu_number); + insn_buf[0] =3D BPF_MOV64_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_numbe= r); insn_buf[1] =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); insn_buf[2] =3D BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); cnt =3D 3; --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AAFE925E837 for ; Mon, 3 Mar 2025 16:53:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020803; cv=none; b=PEzph+BAM4gHOq4zL4BZOh8bQj3lSTCgIJsNPhsltXyOWkKc8ADFkacAPHEw++pCP7pJItTgjgaFAHjW/i0djTojyAHLdBPvspq6wi8GgxbTkxA6zQFBEqKEc1jxwalLYWpOG7gJ7h+EDaQCUXVUNCZjz7BuqngVF7DKvWih5I4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020803; c=relaxed/simple; bh=OYJKRaVndOAwuJVwhaYRftxlVapFhnPm4xheG6tycvE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qRvP/WOSzI5+urQ/YzvrzmCXIhw/wKgiqFkgxkesgnkiA7JUxdtlBbp/ud5PUXAX4GMRDcq+PXeiAXaz1oi3ZKkU350AYkVEZO0yuFVeUUKp5qO8VSijj36QgyPRCkbnRLQahT1SVobNXOlC80SNxnzk8GAggbVvg5MvZ/adYRU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RfydUpP9; arc=none smtp.client-ip=209.85.219.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RfydUpP9" Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6e89959f631so37337056d6.3 for ; Mon, 03 Mar 2025 08:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020800; x=1741625600; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=KnEWHldBFNu/D2mybRN/4ZjcO3DTEeclYWAdvriLmXo=; b=RfydUpP9KG0SyZ/gvvm1qLQPbLbBG3HFBSltIUQopIPCUyk9Xz++RdOUlhL7+OH4jN +p3KYGZUQ5JhAokt+43M3QuOoYAz7BfPFwrJYM7MmdhfUdDcWTvFrsJbdq9DrxJwfzs+ VoVyD4mqM0Dl6RAeYKfSX92T1O3wj0b/fBDCprHsEZig6kuqPTqGHVUbDyUHwrhmOkm5 KbaQNyU9v7YurB1pqSrKW5OLiC6oO+MmCqoval36qOYwMoJcfOxWKy2WfTpBgLuGLKF9 ikEbEXy4MMl+UmsUpN4GV0daZUcQFPpIESd8sIpTn77qoA4klvFJ+PiuU8h2FzMrlteM 0q8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020800; x=1741625600; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KnEWHldBFNu/D2mybRN/4ZjcO3DTEeclYWAdvriLmXo=; b=rR99/DLyM6QeDWumwF5PwkyvUsT4dfm54AVax2eT/C+t8Pi7ILXFwheE4ep6qqBT/y RnXuDKEiiRgfykRXLDnfXhAyPBkxSnPyTryFy1kX30PEsx8Ui4rvg/v2LF+eTd9XIwIy DAHndzUwizjd0NSQ0W/jMCBOOzwTy7RGIxC/1m453x6ic1p3XVI8eP2kkeilmCMg3oFd Q/VOTKaS1FQ90CExuYvqgyX1mst7vLv/a9bDAnq76aYea2j3AV1J19An9gWdcSBloD1I wSkMU1EBT+WeHwAMcRJOmlhkEMldNthhzHra2eoDoe7VLPJOc48ouB3J36o3f6/+dTwl TLUQ== X-Gm-Message-State: AOJu0YxeOrwj4wDPBDSxXOjoB6q6lW03e0/FukI4bw7z+vRrSqtkWIan EXP1Wrz5QEj+jzy/9XTpMIAtdrJhRquF9jEtLU8/ZoGM48TG+wRPV1Pb X-Gm-Gg: ASbGncuhBX1w25riMbEdg24xCvWE978koYIHzp7Nd1CGO5ZroguAl3fuibWT1ipgkwq nOK6MePEqY2Ch3xzu2GWiFG/RBmm4eLzaFu5VZRt9Y5WWOp6lI934yumIpRseK/4phPWq2pOwVf wep5vAT9bVUyKFtk2qWqKFVFEje3GUXnuOu095yQYVeYag0TeK6gQ2hCmuAuwSYel4o8IEwPyum 7dUF6ARSSrkg0RBPYd88BHpTYkgnj8AJ8t1784ipo533QeP4x83AYVu5nfld6kP6CIJWb+gYhFJ EA2lUrEsEn9DTeRmhoUzEjVXOw== X-Google-Smtp-Source: AGHT+IGg/JwlRqTd6J4J4I9Fda9p3LYVTXmcE+NXXjiOh6O/ljL62wgmSfKtoYXIomj962qwjuhlNw== X-Received: by 2002:a05:6214:2aad:b0:6e6:6c10:76fb with SMTP id 6a1803df08f44-6e8a0d28452mr213528196d6.25.1741020799889; Mon, 03 Mar 2025 08:53:19 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:18 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 05/11] x86/retbleed: Move call depth to percpu hot section Date: Mon, 3 Mar 2025 11:52:40 -0500 Message-ID: <20250303165246.2175811-6-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 3 --- arch/x86/include/asm/nospec-branch.h | 11 ++++++----- arch/x86/kernel/asm-offsets.c | 3 --- arch/x86/kernel/cpu/common.c | 8 ++++++++ arch/x86/lib/retpoline.S | 2 +- 5 files changed, 15 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index f988462d8b69..8ba2c0f8bcaf 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,9 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - u64 call_depth; -#endif unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 99daafd5fee2..e8757d7a3582 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -12,7 +12,6 @@ #include #include #include -#include =20 /* * Call depth tracking for Intel SKL CPUs to address the RSB underflow @@ -78,21 +77,21 @@ #include =20 #define CREDIT_CALL_DEPTH \ - movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq $-1, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH \ xor %eax, %eax; \ bts $63, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq %rax, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH_FROM_CALL \ movb $0xfc, %al; \ shl $56, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + movq %rax, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #define INCREMENT_CALL_DEPTH \ - sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + sarq $5, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #else @@ -387,6 +386,8 @@ extern void call_depth_return_thunk(void); __stringify(INCREMENT_CALL_DEPTH), \ X86_FEATURE_CALL_DEPTH) =20 +DECLARE_PER_CPU_CACHE_HOT(u64, __x86_call_depth); + #ifdef CONFIG_CALL_THUNKS_DEBUG DECLARE_PER_CPU(u64, __x86_call_count); DECLARE_PER_CPU(u64, __x86_ret_count); diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index a98020bf31bb..6fae88f8ae1e 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -109,9 +109,6 @@ static void __used common(void) OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - OFFSET(X86_call_depth, pcpu_hot, call_depth); -#endif #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 8db36699dc78..f051fc90b44b 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2060,6 +2060,14 @@ DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D I= NIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 #ifdef CONFIG_X86_64 +/* + * Note: Do not make this dependant on CONFIG_MITIGATION_CALL_DEPTH_TRACKI= NG + * so that this space is reserved in the hot cache section even when the + * mitigation is disabled. + */ +DEFINE_PER_CPU_CACHE_HOT(u64, __x86_call_depth); +EXPORT_PER_CPU_SYMBOL(__x86_call_depth); + static void wrmsrl_cstar(unsigned long val) { /* diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 038f49a43ab4..a26c43abd47d 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -343,7 +343,7 @@ SYM_FUNC_START(call_depth_return_thunk) * case. */ CALL_THUNKS_DEBUG_INC_RETS - shlq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth) + shlq $5, PER_CPU_VAR(__x86_call_depth) jz 1f ANNOTATE_UNRET_SAFE ret --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f52.google.com (mail-qv1-f52.google.com [209.85.219.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0529325F781 for ; Mon, 3 Mar 2025 16:53:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020805; cv=none; b=bwuySknCg55oAxBymDH0umtm04y942OLahhIKPUoJOA26ZtvUvw9qff/8WzxYBga/Z5O8FZgDtz4QHsU12FJLh+SJxh/A+oscO68bX1b4qAuv7g33Qo+oQ+tl/FzqQP1Bn7d6akGksyGzqmzcAFxzY2oSJQiW26GXqHgQFQmFHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020805; c=relaxed/simple; bh=OuEF1PyTt3q4WbTO9qkv6fDdA58X8CinRhLfM3wEf50=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PqtH59ty2WCMvUWSX6AXu+v0Nr6jz1SEUprwFU5pgeR50d3y8sw/RFx3t7V9NmzxsAK70GCxDHO3kGo+1fdO9byHdV2M0uu4vq4zoDm5HLcKCIHZsFaSTETdRPVI0DgeX2rQwHl3kNJ6eSSeIcvAd3IULd/RsIUOOpPreMI0/IY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mGC0zpcr; arc=none smtp.client-ip=209.85.219.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mGC0zpcr" Received: by mail-qv1-f52.google.com with SMTP id 6a1803df08f44-6e894b81678so23734376d6.0 for ; Mon, 03 Mar 2025 08:53:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020801; x=1741625601; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mTWx48nOtEjOPqQVYnv4iK/yyTHbSogRVm8T0/grgxE=; b=mGC0zpcr1SQSKxqwAhr9zXU+WfndwSDJ+T2gAg32KTBybCvzGBwnqUXYH+m11AuRPw FfTVQmwyuv8w/agph5oIhpN3wpFlKgG3aCBzr5lWTx/vyfFprKAOuOiFHbrJGxSaWPKd o8Ha4x6c0Q5QYKnxwayafJFwQwRfe3CZtLT0EGEQc33NtR5DavKZ/3z2miJKSJJUQilK FkZsnoKVvdGCS5A5w3oS6TETyeGh8z0tOw/WlzNW6CBNp2yI92OEuIhuCzp4Pd5gEexT zN1W4SjZc5MeIwAzGu61IpI6uk2d95IHIvft1rWfmHnp+Wy0GKof3wyLYxCQ2SsS62YT S94A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020801; x=1741625601; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mTWx48nOtEjOPqQVYnv4iK/yyTHbSogRVm8T0/grgxE=; b=fmQ3td93SBrT1WcaWg9eiJ3Trkw+2Al2Xio2leXMXndkTilBAa6kYZmazm+uTNDQ/0 CKvrk/C+/fYWH35BKh8ivn+GurI8TsrUQebHlcXPD+RiIWQZo5PDLYUumw+03X0X2qZR TGyuKbZF7XhAY+vTiDDQHF8D5GYjNwSCFQUWTtnLNIO73YVMNUK7z9a/mW98efQafJ+o +RIYQi/eoBGdVCkGuyxohr+gN+vqGkC2vJTL/XVfV+IR6MANOdhfSjrmYUKhTkwEd8/5 iCaOG302nk6AD0lxLUa3VcioJ2XSJUkKzDukT7SVLHBDwnh293JoOLSOtUzUMjCDiioT eeMA== X-Gm-Message-State: AOJu0YxsvFq2SQgNIGS7Q2kHngYqr/F4H1Tgr2tkDk/vLzJXh6Zr6jhj EHhzMJiHQZeSi2FHiLd6fFEnSjxq8euq2qmguqra5HpNHrqhk/49KEIa X-Gm-Gg: ASbGncvi4GjhwGvbcc/YXVohuDh7/J5/LwLAyKXYGRoD2LaND6x6AAg1f/NuWDZOpla 0yrglCHHxOr7Gthkbr2iupPZ5KlHaptFqc1wLgNKB6vkxnlyyfPk7c/f/U8I4UqXRJNBk1y/cyD ka1ydwvEO5PRVgwxfODdWiJ4NzOh3QzLXOt4zcrrBCF7zOMxiK/iDngO4WpA44TNn0Xy0crjf1J h8XxC9+HV7cFSRk2iWaD4i+ilbhafpY/xUeMLviPJ14/NjRMJWiwGfAv31/TfQKbhnxzDSfy3nS bIeh2xsWWQeRtofcqJcrad6FBA== X-Google-Smtp-Source: AGHT+IGbRjMShz35NX28xx4o26X5hQbd26WoL+j2ShutjdFJUIZWKWp4HvECngvv3uECxna8Q9SCKw== X-Received: by 2002:a05:6214:d85:b0:6e8:9dd7:dfd7 with SMTP id 6a1803df08f44-6e8a0dd52e8mr225868466d6.45.1741020801634; Mon, 03 Mar 2025 08:53:21 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:20 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 06/11] x86/softirq: Move softirq_pending to percpu hot section Date: Mon, 3 Mar 2025 11:52:41 -0500 Message-ID: <20250303165246.2175811-7-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/hardirq.h | 4 ++-- arch/x86/kernel/irq.c | 3 +++ 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 8ba2c0f8bcaf..f153c77853de 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { struct task_struct *current_task; unsigned long top_of_stack; void *hardirq_stack_ptr; - u16 softirq_pending; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 6ffa8b75f4cd..f00c09ffe6a9 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -3,7 +3,6 @@ #define _ASM_X86_HARDIRQ_H =20 #include -#include =20 typedef struct { #if IS_ENABLED(CONFIG_KVM_INTEL) @@ -66,7 +65,8 @@ extern u64 arch_irq_stat_cpu(unsigned int cpu); extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat =20 -#define local_softirq_pending_ref pcpu_hot.softirq_pending +DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending); +#define local_softirq_pending_ref __softirq_pending =20 #if IS_ENABLED(CONFIG_KVM_INTEL) /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index feca4f20b06a..83a5252a473c 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -33,6 +33,9 @@ DEFINE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); EXPORT_PER_CPU_SYMBOL(irq_stat); =20 +DEFINE_PER_CPU_CACHE_HOT(u16, __softirq_pending); +EXPORT_PER_CPU_SYMBOL(__softirq_pending); + atomic_t irq_err_count; =20 /* --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 060E125F961 for ; Mon, 3 Mar 2025 16:53:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020807; cv=none; b=ktoPVWc/1RBiRCvLm2u5qVdQPg+dS6WEryWHuQie1GEMY9Jo2a/xjYxneltsTGjB3vrCPWfdBXFAzJt/l5EjRNFz3chipj3Mpx6y5JiG8skFdgM5hhQe559/DUe8FiKwIam+5A5vJ69fENuxgitPeSrT71po+DTRzXDX3kaSBV0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020807; c=relaxed/simple; bh=A+l+hoR2IeyWEqsJeLmBzlIukxl12Att0NLJaH3RYwM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iAQFKmGbwlN06DXPlBtjmcaBUoolHh9OrbJ9McxW8NS+fWJZfYS4bEUtDgg73DMHsc5oMIyOmkFaM+7TpTMsxDw2oxnQYK82JBrALgYEFT+rnqyi83xUAFg6gn9kBBE6xy/l7e4sZAvJJ72WIshD69CXzqxJilPdno1Fpyr2vwk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VpZ5PfmD; arc=none smtp.client-ip=209.85.219.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VpZ5PfmD" Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-6e8992f9a45so33356716d6.0 for ; Mon, 03 Mar 2025 08:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020804; x=1741625604; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c37x08hTKlDAT1EL3+ji6o6B0AZmhiaBFIwDXnwiew8=; b=VpZ5PfmDCwktN8nTAJ4WBMUu8EOPblP8uOHIHYLiTcXVqMu48K8mo+StbR5AZemFyi bjf80Ko0VRWuI/N+Y4HwJEgyGH1oeKJRSvtdGx9GfL7l0k+QHQlDQanbbDebSdydUuI9 6NPmX3f1CHgT/7n9Dr1X/g0to1FRu5aBS07dwcfsM84AonC1ekKDBICW8HsaY9z+twg/ D7a25vPKeHE80h/jd4sONy6HJlIBp80vUpVtrQ5cIGzbCXJLEko41Ef7OESp3tOAfzNe PTTcb3btj84dGrGMS9bT7qNwaOFH0yrFq0UHhZHvMEgNbQBFImAX1mYOMFor/RpItyPL CjNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020804; x=1741625604; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c37x08hTKlDAT1EL3+ji6o6B0AZmhiaBFIwDXnwiew8=; b=O1NMY76yfXr+CV6Jf8XodtbFaDhoT3LKoM0ckpKq5KXpuJF2whe69Gvte3+kr8UH+Z 9DnTK6QcUBoy7ZITctukGj9sihYiVjCQNqaur9m6+FXhCgNpOmvYB8XwuUy6lqfNJybw 1MgZeBKMFRDtpQgMldzsZCXsrpr8O4f2UfbZHxqrey66q1N54stVJYqcEqACy+ztJz4l LcHF1eeenTWjf9SP5k7Q2UyGOVc5dzWsOTaWc0P1YoSwCeC9UWmNOyN4cd24gtVB8kys 8rLXs/1Ekrca8okcDTgGxbdJMHhKSQJ5D6YnLZ7BzkrqRF/R2Jt5ob1WVblQKG7j3MIY Yy4Q== X-Gm-Message-State: AOJu0YwE3tc1qyiXCZKuV48tj9pAIm2KybzwUMcs/gyJ0CqAVxV2gqDd 6VENcg92qaQUqGOKI+cl1eE/Qh3e4Br5dXJl1Wx4VNK5vpIDd+QWptbn X-Gm-Gg: ASbGncu5WhLYCN5sNOarN3KdcaMnQ+eo48BtEqBvCMLCKXK7nN+7f63O5ZFa6NrbZSC qkN9y2yN0HiGGHTz8cS18dtxgBJNLx2x9QjF8ILCx6yJnXmRqJaygiM/7Kl/CqqxbMtSG4+Vz+G THgHZBY6vW7evyGTegC6T+Sbj8Jhk0XCC2zNorcP3TFFg199kBrjWjEGBRmoL2nphZgY5HD/jrC oUwAdBhfnx8rfc485f+VZhkQFSHIt/P8q2Vot8xZ8xW2INoq7fP4vmEz8srAGvb01PKD6K53W3I mHgNPPSSxoR4fmvgzmn8+g+pyg== X-Google-Smtp-Source: AGHT+IGWWZHHlQUDQC1c/ndgMPf9SnnXcmPX3NGUhVz3YvH6Kj4AmYuCSsKlALqHvpO6trM2SXv5yg== X-Received: by 2002:a05:6214:2462:b0:6e6:5aa2:4e4a with SMTP id 6a1803df08f44-6e8a0d700e5mr221809956d6.28.1741020804077; Mon, 03 Mar 2025 08:53:24 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:22 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 07/11] x86/irq: Move irq stacks to percpu hot section Date: Mon, 3 Mar 2025 11:52:42 -0500 Message-ID: <20250303165246.2175811-8-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 6 ------ arch/x86/include/asm/irq_stack.h | 12 ++++++------ arch/x86/include/asm/processor.h | 7 +++++++ arch/x86/kernel/dumpstack_32.c | 4 ++-- arch/x86/kernel/dumpstack_64.c | 2 +- arch/x86/kernel/irq.c | 2 ++ arch/x86/kernel/irq_32.c | 12 +++++++----- arch/x86/kernel/irq_64.c | 7 ++++--- arch/x86/kernel/process_64.c | 2 +- 9 files changed, 30 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index f153c77853de..6fad5a4c21d7 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -15,12 +15,6 @@ struct task_struct; struct pcpu_hot { struct task_struct *current_task; unsigned long top_of_stack; - void *hardirq_stack_ptr; -#ifdef CONFIG_X86_64 - bool hardirq_stack_inuse; -#else - void *softirq_stack_ptr; -#endif }; =20 DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_st= ack.h index 8e56a07aef70..3ce0314eaf7c 100644 --- a/arch/x86/include/asm/irq_stack.h +++ b/arch/x86/include/asm/irq_stack.h @@ -117,7 +117,7 @@ ASM_CALL_ARG2 =20 #define call_on_irqstack(func, asm_call, argconstr...) \ - call_on_stack(__this_cpu_read(pcpu_hot.hardirq_stack_ptr), \ + call_on_stack(__this_cpu_read(hardirq_stack_ptr), \ func, asm_call, argconstr) =20 /* Macros to assert type correctness for run_*_on_irqstack macros */ @@ -136,7 +136,7 @@ * User mode entry and interrupt on the irq stack do not \ * switch stacks. If from user mode the task stack is empty. \ */ \ - if (user_mode(regs) || __this_cpu_read(pcpu_hot.hardirq_stack_inuse)) { \ + if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \ irq_enter_rcu(); \ func(c_args); \ irq_exit_rcu(); \ @@ -147,9 +147,9 @@ * places. Invoke the stack switch macro with the call \ * sequence which matches the above direct invocation. \ */ \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(func, asm_call, constr); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } \ } =20 @@ -213,9 +213,9 @@ */ #define do_softirq_own_stack() \ { \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(__do_softirq, ASM_CALL_ARG0); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } =20 #endif diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index a969bea1ed07..9203cecb7695 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -420,6 +420,13 @@ struct irq_stack { char stack[IRQ_STACK_SIZE]; } __aligned(IRQ_STACK_SIZE); =20 +DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, hardirq_stack_ptr); +#ifdef CONFIG_X86_64 +DECLARE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); +#else +DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); +#endif + #ifdef CONFIG_X86_64 static inline unsigned long cpu_kernelmode_gs_base(int cpu) { diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c index b4905d5173fd..722fd712e1cf 100644 --- a/arch/x86/kernel/dumpstack_32.c +++ b/arch/x86/kernel/dumpstack_32.c @@ -37,7 +37,7 @@ const char *stack_type_name(enum stack_type type) =20 static bool in_hardirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* @@ -62,7 +62,7 @@ static bool in_hardirq_stack(unsigned long *stack, struct= stack_info *info) =20 static bool in_softirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.softirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(softirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c index f05339fee778..6c5defd6569a 100644 --- a/arch/x86/kernel/dumpstack_64.c +++ b/arch/x86/kernel/dumpstack_64.c @@ -134,7 +134,7 @@ static __always_inline bool in_exception_stack(unsigned= long *stack, struct stac =20 static __always_inline bool in_irq_stack(unsigned long *stack, struct stac= k_info *info) { - unsigned long *end =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_st= ack_ptr); + unsigned long *end =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr); unsigned long *begin; =20 /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 83a5252a473c..81f9b78e0f7b 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -36,6 +36,8 @@ EXPORT_PER_CPU_SYMBOL(irq_stat); DEFINE_PER_CPU_CACHE_HOT(u16, __softirq_pending); EXPORT_PER_CPU_SYMBOL(__softirq_pending); =20 +DEFINE_PER_CPU_CACHE_HOT(struct irq_stack *, hardirq_stack_ptr); + atomic_t irq_err_count; =20 /* diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c index dc1049c01f9b..48a27cde9635 100644 --- a/arch/x86/kernel/irq_32.c +++ b/arch/x86/kernel/irq_32.c @@ -52,6 +52,8 @@ static inline int check_stack_overflow(void) { return 0; } static inline void print_stack_overflow(void) { } #endif =20 +DEFINE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); + static void call_on_stack(void *func, void *stack) { asm volatile("xchgl %%ebx,%%esp \n" @@ -74,7 +76,7 @@ static inline int execute_on_irq_stack(int overflow, stru= ct irq_desc *desc) u32 *isp, *prev_esp, arg1; =20 curstk =3D (struct irq_stack *) current_stack(); - irqstk =3D __this_cpu_read(pcpu_hot.hardirq_stack_ptr); + irqstk =3D __this_cpu_read(hardirq_stack_ptr); =20 /* * this is where we switch to the IRQ stack. However, if we are @@ -112,7 +114,7 @@ int irq_init_percpu_irqstack(unsigned int cpu) int node =3D cpu_to_node(cpu); struct page *ph, *ps; =20 - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; =20 ph =3D alloc_pages_node(node, THREADINFO_GFP, THREAD_SIZE_ORDER); @@ -124,8 +126,8 @@ int irq_init_percpu_irqstack(unsigned int cpu) return -ENOMEM; } =20 - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D page_address(ph); - per_cpu(pcpu_hot.softirq_stack_ptr, cpu) =3D page_address(ps); + per_cpu(hardirq_stack_ptr, cpu) =3D page_address(ph); + per_cpu(softirq_stack_ptr, cpu) =3D page_address(ps); return 0; } =20 @@ -135,7 +137,7 @@ void do_softirq_own_stack(void) struct irq_stack *irqstk; u32 *isp, *prev_esp; =20 - irqstk =3D __this_cpu_read(pcpu_hot.softirq_stack_ptr); + irqstk =3D __this_cpu_read(softirq_stack_ptr); =20 /* build the stack frame on the softirq stack */ isp =3D (u32 *) ((char *)irqstk + sizeof(*irqstk)); diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index 56bdeecd8ee0..ca78dce39361 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -26,6 +26,7 @@ #include #include =20 +DEFINE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); DEFINE_PER_CPU_PAGE_ALIGNED(struct irq_stack, irq_stack_backing_store) __v= isible; =20 #ifdef CONFIG_VMAP_STACK @@ -50,7 +51,7 @@ static int map_irq_stack(unsigned int cpu) return -ENOMEM; =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #else @@ -63,14 +64,14 @@ static int map_irq_stack(unsigned int cpu) void *va =3D per_cpu_ptr(&irq_stack_backing_store, cpu); =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #endif =20 int irq_init_percpu_irqstack(unsigned int cpu) { - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; return map_irq_stack(cpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4ca73ddfb30b..3a8f967a0c79 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -614,7 +614,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) int cpu =3D smp_processor_id(); =20 WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) && - this_cpu_read(pcpu_hot.hardirq_stack_inuse)); + this_cpu_read(hardirq_stack_inuse)); =20 if (!test_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD)) switch_fpu_prepare(prev_p, cpu); --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C172C25F989 for ; Mon, 3 Mar 2025 16:53:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020809; cv=none; b=CqfaUyGjaaSkT32GP00Rm74FVf/BWD/Vv0c6vBDxkIEuFlUWZPAFPNAbSQLxrtW0Fc8WsIEeKBYDKVJX7qsMc9FRTM4xInLvEuYaNC1xHC2tpGwXKCttILGc3dUMqbNHCUYNPI2c5A4RvgpfSXEyBJoeZbAGxLknfRl2X/y6bVc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020809; c=relaxed/simple; bh=Qa9rbS11nQ3ULYiX0zFMnX0VY4BRsXfO39BzwlEKLp8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=A4RtbNLZ38WXH3pVDqB/I++LwdIFzAWi4FpTERV6azJB/Nj1PmITp+pXJ2CdNlCRqsiu8I5lg7MOx4y6S1lKlvbLALhTU56K+i0AI8sfj4IGi6jYB623v0/v2QVJmb7TPOgJ7zBLTDXy3IZ1U8ezScP94oTD9BBWZiRrEAfXnZE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=A9Quz0Br; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A9Quz0Br" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7c3cb761402so17347285a.0 for ; Mon, 03 Mar 2025 08:53:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020806; x=1741625606; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O+Ges8AFZBJaJk/QGwDEBI1qSGnch2qFY9doOlyO4a4=; b=A9Quz0BrJQthf8wpH11n4SyffavneGyEsFIeRGns7JbyrmuQq0LkRMjqqzHODJ3S32 hS7r6kQKbSOAj+w0C1vQGeE/0ZMXUZezV/2iOod5frbUkm9BAquyileU5sY/1ccxdxMC 9S2A0j3Ip7dZxiNmn1WMU0MIaQcXat59jX86xruUEHCUELyINEx108I+enRC276ECKBG qzNfPeTAY0rPp78yDFW1NkOevZroCLbkX8baIb6PVMH49zKW+EFBDCoRZJ4ipLuzleR+ bcdWPMe2df6oUXjK4gLWkRpjRHtDsfCTkf/KVVIHXEg0xLDfesHzySRVZKPEz3qSG9Js xMuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020806; x=1741625606; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O+Ges8AFZBJaJk/QGwDEBI1qSGnch2qFY9doOlyO4a4=; b=WgKPNWsk8e4epn17KMPizc8vhoshIu/I23H8RtwiY5dLnLD8bAhCWAdp8hDcN5V+17 XSiDxeyrsU01WCxd5f7yaEWZPHY6yEbWm6S2Oz5T98uVEnba04UqjuWuIYkvpfbtcaZu FLl6Zvg0qmM/EeURfcg9o1hWsv6jT77iL8/NIsPX5WY85Zab8A/JmW4AVFnv4cdlVbO5 MuMm9jCmCzpZBbx7pWRLEdKkJQhpTWFY6eB+UsEUwPGghqq5A5lawefy1asFGOOFRp0k LQCOWgaQy6RKCuhl6CdY94NNieu+s6bFhEREn9HxOJin+e9tv8N8vTjcYt9vDBE3eSC6 fUPA== X-Gm-Message-State: AOJu0YyX1VHump/duz18zzyWPUcw6boUU6BnYEHDnNNOSZXOG0k2u7jk q7dKKVhBPzi7BOpj20IPICmxA0cJPIjvyspgUOUMIT7UKCISNBuQWIiz X-Gm-Gg: ASbGnct2wy3ljzSDKvVdFEkwuP9KHlSTTDMC8PxEMAT+yQ7bv+bYaMFTcCQgv/P1ZsZ Rel5sshhXRTemF9cPSHmg21n6yxvQsYvcTD30891SSiYmEhP0IfBZ7bTqCnpNIo97H41+90Hgsb Ylw7MQ8CJdv4qbRt/horYy5zDAz7ZcE4mLVFCRPAuwDmo2o7nyjsTjJVchfQB0hUbAIvOYZiDp1 Taw8r+kMXSRXWITFRBdPDCQSe6W3RZGewVtkDqkZrY6s3mhnBvvhtBD/pqGB5C+1e/S+uDakGpi 4/HU0vuLtiO2vYnKKYaNvJ50+g== X-Google-Smtp-Source: AGHT+IEQGJjHx0IRKzF1BO7sDCFo5HAad462T1g3FuN+Ywu+ucmRpETBqLWaZGTjzKtlTSSNe8eOPQ== X-Received: by 2002:a05:620a:462c:b0:7c0:abe0:ce40 with SMTP id af79cd13be357-7c39c4992a1mr2060863985a.5.1741020805992; Mon, 03 Mar 2025 08:53:25 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:24 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 08/11] x86/percpu: Move top_of_stack to percpu hot section Date: Mon, 3 Mar 2025 11:52:43 -0500 Message-ID: <20250303165246.2175811-9-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/entry/entry_32.S | 4 ++-- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 4 ++-- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/percpu.h | 2 +- arch/x86/include/asm/processor.h | 9 +++++++-- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 3 ++- arch/x86/kernel/process_32.c | 4 ++-- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 + 12 files changed, 22 insertions(+), 17 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 20be5758c2d2..92c0b4a94e0a 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -1153,7 +1153,7 @@ SYM_CODE_START(asm_exc_nmi) * is using the thread stack right now, so it's safe for us to use it. */ movl %esp, %ebx - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esp + movl PER_CPU_VAR(cpu_current_top_of_stack), %esp call exc_nmi movl %ebx, %esp =20 @@ -1217,7 +1217,7 @@ SYM_CODE_START(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esi + movl PER_CPU_VAR(cpu_current_top_of_stack), %esi leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp =20 call make_task_dead diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 49d3b222fe99..f40bdf97d390 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -92,7 +92,7 @@ SYM_CODE_START(entry_SYSCALL_64) /* tss.sp2 is scratch space. */ movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -1168,7 +1168,7 @@ SYM_CODE_START(asm_exc_nmi) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ @@ -1486,7 +1486,7 @@ SYM_CODE_START_NOALIGN(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rax + movq PER_CPU_VAR(cpu_current_top_of_stack), %rax leaq -PTREGS_SIZE(%rax), %rsp UNWIND_HINT_REGS =20 diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index ed0a5f2dc129..a45e1125fc6c 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -57,7 +57,7 @@ SYM_CODE_START(entry_SYSENTER_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax popq %rax =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ @@ -193,7 +193,7 @@ SYM_CODE_START(entry_SYSCALL_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp =20 /* Switch to the kernel stack */ - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 6fad5a4c21d7..3d1b123c2ee3 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - unsigned long top_of_stack; }; =20 DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 60390a019ca9..1ca290ba6f77 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -549,7 +549,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and pcpu_hot.top_of_stack, both of which are + * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index 9203cecb7695..60f1c803b6c1 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -427,6 +427,11 @@ DECLARE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); #endif =20 +DECLARE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_CACHE_HOT(const unsigned long __percpu_seg_override, + const_cpu_current_top_of_stack); + #ifdef CONFIG_X86_64 static inline unsigned long cpu_kernelmode_gs_base(int cpu) { @@ -552,9 +557,9 @@ static __always_inline unsigned long current_top_of_sta= ck(void) * entry trampoline. */ if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.top_of_stack); + return this_cpu_read_const(const_cpu_current_top_of_stack); =20 - return this_cpu_read_stable(pcpu_hot.top_of_stack); + return this_cpu_read_stable(cpu_current_top_of_stack); } =20 static __always_inline bool on_thread_stack(void) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 6fae88f8ae1e..54ace808defd 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index f051fc90b44b..a771aa77f05e 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2051,7 +2051,6 @@ __setup("setcpuid=3D", setup_setcpuid); =20 DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); @@ -2059,6 +2058,8 @@ EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 +DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) =3D TOP_= OF_INIT_STACK; + #ifdef CONFIG_X86_64 /* * Note: Do not make this dependant on CONFIG_MITIGATION_CALL_DEPTH_TRACKI= NG diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 2bdab416298c..8ec44acb863b 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -190,13 +190,13 @@ __switch_to(struct task_struct *prev_p, struct task_s= truct *next_p) arch_end_context_switch(next_p); =20 /* - * Reload esp0 and pcpu_hot.top_of_stack. This changes + * Reload esp0 and cpu_current_top_of_stack. This changes * current_thread_info(). Refresh the SYSENTER configuration in * case prev or next is vm86. */ update_task_stack(next_p); refresh_sysenter_cs(next); - this_cpu_write(pcpu_hot.top_of_stack, + this_cpu_write(cpu_current_top_of_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 3a8f967a0c79..04da741dffd2 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -669,7 +669,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) * Switch the PDA and FPU contexts. */ raw_cpu_write(pcpu_hot.current_task, next_p); - raw_cpu_write(pcpu_hot.top_of_stack, task_top_of_stack(next_p)); + raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 582016967587..83ec0e90c98e 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -852,7 +852,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) =20 #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ - per_cpu(pcpu_hot.top_of_stack, cpu) =3D task_top_of_stack(idle); + per_cpu(cpu_current_top_of_stack, cpu) =3D task_top_of_stack(idle); #endif return 0; } diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 7586a9be8c59..85032c085af2 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -44,6 +44,7 @@ ENTRY(phys_startup_64) =20 jiffies =3D jiffies_64; const_pcpu_hot =3D pcpu_hot; +const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) /* --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6379A25FA03 for ; Mon, 3 Mar 2025 16:53:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020811; cv=none; b=S2ORvOCsmvpfayDgq9PRDrqXT/2XPq51UeVdmM1ngPty7Oug4TLWJzu59gmVNSZqhjuhoMCfGEMKzsDJTjhEvNVG0sxn2hP8SzU9yWQHHQ/f3Ph9wjG4pN112ih7Obv38i5RFdoFEWpQQxjg55sFsZQSCGo1SRV9sCmMtyrqm3c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020811; c=relaxed/simple; bh=sMsV0xjoxmWPqYhIhvlp1QSk3YpSjAVqBu7XTqzwgfc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ryUeyQZlqWHb0JPzIvLPTGQDbmtTFvrTtHCsATcSDUm5Ewd7GfbEyw2hsG15BKRRQ/Ou9U9j5hnS1ZAdgP6TOCp4+Sreu2gf7rY+ZUezocpj7S53tPTCMeotZ3eAZ0TXcMXynv5RYo8vLQp7z6efRKWZecZmppuX0nIyHTxAWx4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TCaeIM0q; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TCaeIM0q" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6e8c3a0d468so11461106d6.0 for ; Mon, 03 Mar 2025 08:53:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020808; x=1741625608; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pIZPQr2eOAkEsD5/zDdOJhEiDfF60twwqPhGFSJXNyk=; b=TCaeIM0qmz8ApJ2saoMVQsZw5aszyNBbt3s+EBu25zWQ2YyhVOUQCqB7FLYT339DbJ 59rkb+Jpw70a439w8JrGXyqceuB7cp7PGqfiDaCCMv0aWTezBqw2KsD3Vhj6R6rWwQVr NdrGiuBEwEioxyVSNrFq+n/XqQyRGyBKyVroQI1TIMZTcrWNjtO3/tJ8KjblfPyXylQK 3/4xscXjiOtKncho65uzboAEeqv6zgGcEhAHyaXB5zGl4LdopVKml76ovUPz3i9k5jTn HUiFdqcgvQY8PZwFncxeXrNVr3Y8ivWS7pe6FLZqjQGhOU02+0OG4UO6UFGyeEVykT3q pyCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020808; x=1741625608; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pIZPQr2eOAkEsD5/zDdOJhEiDfF60twwqPhGFSJXNyk=; b=DXz3ghuk3xwRDTeZabl9p1dA9fCxEfLcjJjk+gIuG4jXIOU9YMNcvi+SX7CarXSX/H 7tZj2814z6Iu/1T5G0wUK9uyZUGMwTu3R321Hi6XxY9gwzfYDZVzBIpbI7eroCvf/RZ6 Qwdt8AdJ1mkytieeLWHAH0H/MIXlw8mxWDimCJJlq1QKZZtmP6MvwJfDv3oyNPA/tw9n f/wa0ut3/ir0RWHXbvdtBaPAdA7synFTyr26Cv2xyWv3FozS6ABue7qyHaAOP3BHfgUy VFwFsxlYH7cN4s5wuIZfuhq1YlB/c7pn7RDG57bsPZUdCHVFNPQGWVUV+ihBrUMd2Ztf ZLnA== X-Gm-Message-State: AOJu0Yw3OuFd01JEbOcumukou4yGaMBVkR73WR8oOyeKgDPSchtPyP1e kilCl4PFz7RDK/4ORVOjui0jok7Ewr2QbQLzUIRs40NBMjSrLGy3+NTn X-Gm-Gg: ASbGnctfjSdosBtj0hDEocenZ4sIAofNKHQTGQDk1VASD75c/0updtPQSFAmYWW617/ mPpLKZQbzUQX77+otiftF/WgfBhQrqd0LCh1+2zzjb8m7K6ye1ZhUaEy6p9pE/ZGiP2T9KV9QaL 5ahcCO/JPPbEXnlPQKKxUBjrouA7zdZsKqXd3dC1cdQ4pePKSIWaEOAn/WsWYF8tdXuUOmVIW9P QOD9CVEtn1b4hkCCdTbaZAE7udK88++8bevB09TrMOUpwriQoOGkNH5rt+YCHbPk20SBcuHC5KZ K5k5VXdsjAWup5/YVHZ3KvQrcA== X-Google-Smtp-Source: AGHT+IF8twjs5LqK9Y6gPVFlK5mkR0wxG91+iQn4l+qqNcJ449loDyByN0+NfqJhh/B+RlmkWE4jIQ== X-Received: by 2002:a05:6214:f6f:b0:6e8:9c5c:5385 with SMTP id 6a1803df08f44-6e8a0c850a5mr260864196d6.5.1741020807797; Mon, 03 Mar 2025 08:53:27 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:26 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 09/11] x86/percpu: Move current_task to percpu hot section Date: Mon, 3 Mar 2025 11:52:44 -0500 Message-ID: <20250303165246.2175811-10-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/current.h | 17 ++++++----------- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 8 +++----- arch/x86/kernel/head_64.S | 4 ++-- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 2 +- scripts/gdb/linux/cpus.py | 2 +- 10 files changed, 17 insertions(+), 25 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 3d1b123c2ee3..dea7d8b854f0 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -12,22 +12,17 @@ =20 struct task_struct; =20 -struct pcpu_hot { - struct task_struct *current_task; -}; - -DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); - -/* const-qualified alias to pcpu_hot, aliased by linker. */ -DECLARE_PER_CPU_CACHE_HOT(const struct pcpu_hot __percpu_seg_override, - const_pcpu_hot); +DECLARE_PER_CPU_CACHE_HOT(struct task_struct *, current_task); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_CACHE_HOT(struct task_struct * const __percpu_seg_override, + const_current_task); =20 static __always_inline struct task_struct *get_current(void) { if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.current_task); + return this_cpu_read_const(const_current_task); =20 - return this_cpu_read_stable(pcpu_hot.current_task); + return this_cpu_read_stable(current_task); } =20 #define current get_current() diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 1ca290ba6f77..6fbb52abb594 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -549,7 +549,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are + * current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 54ace808defd..ad4ea6fb3b6c 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index a771aa77f05e..58b63fa4be6c 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2049,11 +2049,9 @@ static __init int setup_setcpuid(char *arg) } __setup("setcpuid=3D", setup_setcpuid); =20 -DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { - .current_task =3D &init_task, -}; -EXPORT_PER_CPU_SYMBOL(pcpu_hot); -EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); +DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) =3D &init_tas= k; +EXPORT_PER_CPU_SYMBOL(current_task); +EXPORT_PER_CPU_SYMBOL(const_current_task); =20 DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 2843b0a56198..fefe2a25cf02 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -322,7 +322,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) * * RDX contains the per-cpu offset */ - movq pcpu_hot + X86_current_task(%rdx), %rax + movq current_task(%rdx), %rax movq TASK_threadsp(%rax), %rsp =20 /* @@ -433,7 +433,7 @@ SYM_CODE_START(soft_restart_cpu) UNWIND_HINT_END_OF_STACK =20 /* Find the idle task stack */ - movq PER_CPU_VAR(pcpu_hot + X86_current_task), %rcx + movq PER_CPU_VAR(current_task), %rcx movq TASK_threadsp(%rcx), %rsp =20 jmp .Ljump_to_C_code diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 8ec44acb863b..4636ef359973 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -206,7 +206,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) if (prev->gs | next->gs) loadsegment(gs, next->gs); =20 - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 04da741dffd2..5016bbe1102b 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -668,7 +668,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) /* * Switch the PDA and FPU contexts. */ - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 83ec0e90c98e..95beca8044af 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -842,7 +842,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) /* Just in case we booted with a single CPU. */ alternatives_enable_smp(); =20 - per_cpu(pcpu_hot.current_task, cpu) =3D idle; + per_cpu(current_task, cpu) =3D idle; cpu_init_stack_canary(cpu, idle); =20 /* Initialize the interrupt stack(s) */ diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 85032c085af2..9ac6b42701fa 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -43,7 +43,7 @@ ENTRY(phys_startup_64) #endif =20 jiffies =3D jiffies_64; -const_pcpu_hot =3D pcpu_hot; +const_current_task =3D current_task; const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) diff --git a/scripts/gdb/linux/cpus.py b/scripts/gdb/linux/cpus.py index 13eb8b3901b8..8f7c4fb78c2c 100644 --- a/scripts/gdb/linux/cpus.py +++ b/scripts/gdb/linux/cpus.py @@ -164,7 +164,7 @@ def get_current_task(cpu): var_ptr =3D gdb.parse_and_eval("(struct task_struct *)cpu_task= s[0].task") return var_ptr.dereference() else: - var_ptr =3D gdb.parse_and_eval("&pcpu_hot.current_task") + var_ptr =3D gdb.parse_and_eval("¤t_task") return per_cpu(var_ptr, cpu).dereference() elif utils.is_target_arch("aarch64"): current_task_addr =3D gdb.parse_and_eval("(unsigned long)$SP_EL0") --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 961CA25FA2E for ; Mon, 3 Mar 2025 16:53:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020814; cv=none; b=LEmem+nFwZlJYTyySwSm4EAA69P25uiLu8Aj4z3F/80IKWtUV0PAwPbGPybgAm6Eijf7iqnpeQ/989U7gncYX2Q0sCc/9F23HSXC/rdPaz9QrXsvgLJHGDYrfpSrGsf2ShniqkQlVk/bfO1vWo+3Aq+d6NKHiO3CxTYvztIjvLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020814; c=relaxed/simple; bh=K0u6jh2WcTcw916LX50fw2JPVkRTiPhzSaSc7Ar5iDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u+7GhYJVG6GC9BBWe/1nsWxWasT9H6GDyVqypZpLOVbhgr5+lm006zgXaCgDBgtyPgBv39W6/PNW+am17VYWxH2HDnccGmt4OFSnycKOYNVLiByLO6oP+eO4QyWIQuILZ/G5KLMOho2TqwTWJO6/v4TiusafXNcdSoQ2THoEcJs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lO6AzX/p; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lO6AzX/p" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c0848d475cso616925585a.2 for ; Mon, 03 Mar 2025 08:53:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020809; x=1741625609; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OS5ROzQVZyeVGElnAwUMjPqqYx53fOprAY8iMrz7sKI=; b=lO6AzX/pu0TSlN6V60xxY2XrhOc84/LVP5H54AiVmKXs2sCU2HkD2vMBzHgdvGpMb3 9pnZn0yl6GzjIo0RKUvTq6/jrGODiMoOCWWjY74LjolJjcWHymk+MEcli4vXFK5uMt93 cUShwJujH7K40cA73JJ33HhXIgFwpRNZIy3WtVWhuvjbhGHPBbGOHmFN2ue6+rEkTo2n bgP1mmMJ9WG1TPhBLoYSzue6/i7OKTdqnmJ17/+XhL7qTZGMK750IhtY8lqT0oowvIc3 8zIzjfE159si2TxSOV4Nju8R68RcD2hJxDzZRVl741V3Kf4OA/C3Uefb2BRdd0Zk2of/ Rk0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020809; x=1741625609; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OS5ROzQVZyeVGElnAwUMjPqqYx53fOprAY8iMrz7sKI=; b=mUbA1IOKGsC5D3xO9SixuQ6nnhAaOk5HRmszrKSuqtKJriVcZVUg69QstB0rGGA7Ru YDk2H/QDSvVUrUV1fLYYi+h/hBXWmTrt3pZObhkGj1+crEkYHBrpSoNzKXeLG7tkqM84 idHJJEGzcC6ag3l+1jJxry+DJYm/LianY5c0YfyV0x/qm83GhKbSFisZRpoXX0rhlpFp EWKzZA/w0exM8153L7tkO7NMNpwWGBypyxWhiw6KQXYIDV0/klM0i7b3AZtn+1W96r/H c7HOl+zk/1hS4t80LWUvFhIuzLOswwzXjev4aqEpXHY0HFfpFLrjRPeSwxnaUldvfcLJ ClFA== X-Gm-Message-State: AOJu0Yz2NwyyIKpWvIlethxEXS77l4paqQlI3BKsM6r0l2+07F3pWj7Z fvlM5WbD4tNBMm8+cKlZvda+9oNcacAhSJ8tYJz1DnyM8W0VgP4eNc5M X-Gm-Gg: ASbGnctgodpObeb4rWt3L0zN0FDIUR6w714SQ1dOvhbzPPtN1R6afIUcMIWgjt5p2eZ BhTn4kURk9P4J3glTd3xGVk3ywpPPfrCEvmLKX1dXknSidwjos8K/6SzC8TH48Jt9k4bagNiDou 8oVB1SNMvqUEjW1I2xVPK8hj0872JT+c0k+/nj/Idm5x8aEw0HH5zMRR2AmugAD+dxiV0sN9BF5 /v4+8YH8foQytRSCohOoZ9UD488IKYMAVRfTNTr2d7aAkGT7pvl4uODV0VerrpcoxUdQlh+n5nw b7EM6VBOdmXpYx+2EAFYvacGZQ== X-Google-Smtp-Source: AGHT+IHujfcxWb0od0xpJ2ONYVMNZDtoedt9Ys8StxXBGoFZ6pN0Xc7LRNpdnzsY8M8ehNAAlIxYDg== X-Received: by 2002:a05:620a:5641:b0:7c3:9d35:2c91 with SMTP id af79cd13be357-7c39d353655mr1559305985a.5.1741020809540; Mon, 03 Mar 2025 08:53:29 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:28 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 10/11] x86/stackprotector: Move __stack_chk_guard to percpu hot section Date: Mon, 3 Mar 2025 11:52:45 -0500 Message-ID: <20250303165246.2175811-11-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/stackprotector.h | 2 +- arch/x86/kernel/cpu/common.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/s= tackprotector.h index d43fb589fcf6..cd761b14eb02 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -20,7 +20,7 @@ =20 #include =20 -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +DECLARE_PER_CPU_CACHE_HOT(unsigned long, __stack_chk_guard); =20 /* * Initialize the stackprotector canary value. diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 58b63fa4be6c..f06f7b2a0480 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2132,7 +2132,7 @@ void syscall_init(void) #endif /* CONFIG_X86_64 */ =20 #ifdef CONFIG_STACKPROTECTOR -DEFINE_PER_CPU(unsigned long, __stack_chk_guard); +DEFINE_PER_CPU_CACHE_HOT(unsigned long, __stack_chk_guard); #ifndef CONFIG_SMP EXPORT_PER_CPU_SYMBOL(__stack_chk_guard); #endif --=20 2.48.1 From nobody Sun Feb 8 02:22:12 2026 Received: from mail-qv1-f43.google.com (mail-qv1-f43.google.com [209.85.219.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6C16525FA37 for ; Mon, 3 Mar 2025 16:53:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020815; cv=none; b=ZW51zS8galFsuhxUUUH1Rcjydbhv9pN5HKZHBo15IGQt4PPbcf8TVxV6Dx7FYe9ehlX0I4BWq2WXb71PV/eSY8x1nwp3HZC76rz4AZpxcouZprV/+tWMufNy241r3MvIPj3UDJmsqqMmM7sg+9y0aWbspRleTiAqCCxw/6NXzY0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741020815; c=relaxed/simple; bh=06vTaM7HaBP9okDJnr14KC6PDjelCZttFmu/VK1ruJg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Kuij/N+6wAa0DfhfyyICfmZlHTEbRO1SHEAMeJMY8rr1R74L1Hvo5xNiHV+Nnj/b+dZfBCsvmH4Q4pVYRimVEa4POj+Weu0/aXM6KFVHodBCM69Z4lHZ/bOTqd9hbI6LSkBWsMrLOnU6CsIQL8GkmyuzB3epCSNd0+RAbgTNI6I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=cgtPtWl2; arc=none smtp.client-ip=209.85.219.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cgtPtWl2" Received: by mail-qv1-f43.google.com with SMTP id 6a1803df08f44-6dd420f82e2so66102886d6.1 for ; Mon, 03 Mar 2025 08:53:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1741020812; x=1741625612; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O4xtjojYc6P0PlWvEYiKTJXNUUZ/ZAA/9CxpmtDG08o=; b=cgtPtWl2sEkfQawG9qLvqXxma8bTmSVlVEUN7Di9et6GB+h1FHslYW/RhEb42fWBZQ fOqbLVpYVQjjVsZQUU6/JyrAXMUaDJEEywURWFKUKH2kw1e9a/WqNkgcnYXESrMJ/LgV nsmRYDFgab4uIxBeXbL+fca6wekJlD8OmLgx6Io8TK+F4sHVGKOrIVhpSQrPpPvhwPJt 1gOkJOmKJoOYAVOyoMAVMlPhCfn96jdJTHZQAJ9uz+GVTaWudCG7KzCQ+TueXDlGgJa7 rTc85s67scG6oyhTnuXCa+wtboye0+z4iejG3WvE7+v2GsVy4QafnH/4ryosTK+bY200 BTtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741020812; x=1741625612; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O4xtjojYc6P0PlWvEYiKTJXNUUZ/ZAA/9CxpmtDG08o=; b=h8kuRJgM650JhZsS0xH0YF9+Cp54aOuYqS9e5wjKB+kZQjYFSWdinzKzBo07pXo7y4 U7xUeodV7uKWh9XbYam+lTzf9kODQASm65AnBLwbMvQPB+Y6yzVl3trD9vxAUxtmTnyR LP98iTXb9TISPsBUChUco96X4gL+2Lj1Wa9o+p/Si9/+MXm63WWLTla+oEOCFubD1MkV Vs4jJy4SDv7qvJb02CjrWCAOrWAeKO3omucYwGkGwsaG/Sv5ekzuA9xAKxPRfFE7GoAa qICvQTUS2RVPfbgcjPOVFNUYUfi0oe4K80n1f7gCqESBPpnvOxfqY8aDSwelew/KiiXw SvAg== X-Gm-Message-State: AOJu0YzFkA22jlHkc3DxSixX3QVAZRDby2VhWVAg48j5i4wZto7PYqeV iqSKfFtiaNkp9ahk/bxNsjq+uaQN+YiR/JOGdwf+ctfTFsHw8HTpsx5N X-Gm-Gg: ASbGnctcewl/IF4KBWT21v+hz/b4FUaFHd6W2S/UmAkGbht3LAbzkav+omJ0ewu4ZN9 4N3hVXh1zLoPVNOBapSF1icxM5rmX0xctNYCIOKUX+Qo4zq9CmH4kR2DmxLc1YBwWF4Q5kEb+v3 cYp+FlF9oAItE/nqIE79iD5TkSA34GYJzmEIcxIIq5av+0gUPgM0Uja2FpPs8XTX4chMZu/RAdW mVyzZN1x9twEoM5UsCuBjHeR1Zvd/LbvpCj2DmAf6nReZHFiPcPb9HNaMnGuEhGbhr0bH/MBQo8 WTTRa0WPHajqOeSv0QphKadiyA== X-Google-Smtp-Source: AGHT+IEYE5cFUiLtXqat9nxpR923QMPswei83U+8MlDcXO/t+prqkhRskS787ORdMRfS4mxYxouMuw== X-Received: by 2002:ad4:5ec9:0:b0:6d4:c6d:17fe with SMTP id 6a1803df08f44-6e8a0d066f7mr248820926d6.25.1741020811718; Mon, 03 Mar 2025 08:53:31 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6e8976cc9cdsm54730936d6.88.2025.03.03.08.53.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Mar 2025 08:53:30 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v3 11/11] x86/smp: Move this_cpu_off to percpu hot section Date: Mon, 3 Mar 2025 11:52:46 -0500 Message-ID: <20250303165246.2175811-12-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250303165246.2175811-1-brgerst@gmail.com> References: <20250303165246.2175811-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst Acked-by: Uros Bizjak --- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/setup_percpu.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 6fbb52abb594..4f202cdd503d 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -584,7 +584,7 @@ do { \ #include =20 /* We can use this directly for local CPU (faster). */ -DECLARE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DECLARE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off); =20 #endif /* !__ASSEMBLY__ */ =20 diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 175afc3ffb12..bfa48e7a32a2 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -26,7 +26,7 @@ DEFINE_PER_CPU_CACHE_HOT(int, cpu_number); EXPORT_PER_CPU_SYMBOL(cpu_number); =20 -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DEFINE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init; --=20 2.48.1