From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17D0C1E7C24 for ; Sat, 22 Feb 2025 19:06:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251199; cv=none; b=uy6OZDxsdqU6wqZPL2z2DcbhUFdZzGnnKKQgqDD6IF7wVDHDhKDsue0IIiTNCJzOXYVXPAeQviBVdYcoSvqrAN/0ru5C5sgqkKg52NU4f0IhH2Ow6Zcp8R1GFLZDODvciE+aPuavmJSkBtkg0kOIQY2k3u8rQlq57CKaGfCngkg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251199; c=relaxed/simple; bh=MAYGHgbYKY62qo0XK/PUpsTFLR0EoTZHSLQDaF1ALp8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ODSs+4+ZPexqQk2LSnb/dI15pFdVAk0ez7GvFGzgRquvQpOqQmxd3DOQtofUGNJudD7ZDFQaC/GP54hqZWg41NZTWupl0uhezzb0jUuekM1Vwx2PUKBIrH4fDSkeuTiywlGA7uTPyAELn4EmIWcdVMpbk1xQcJLhLEwlj9lG7CM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PLvMBLo+; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PLvMBLo+" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7be3f230436so313323185a.3 for ; Sat, 22 Feb 2025 11:06:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251196; x=1740855996; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Xb9ZMWEswnCZqxNt4mNRr92ZbyiwfGCzO2RoU9QiMyc=; b=PLvMBLo+1fd1XYglQAXcde0fZCfV194Zn2fRy90Yt64GfX3IJwxQx5QtZdOhDyN2rt ftJm5e7f6WYq5BpyWt4Ft21hJdS7xI3eYYNDVajicH+YqvO1qi9s3u85kdDygXiNU19U SVTcHpG00RehtGr2cMi5caaWr3++dmX28fb0v/BDP8SPr8UlSfRoZnk1Xo67xdEh9vZM H9axsSr1YPgxgn61lmvaHEuJYm0wJEz2pJrsOUGbaGHQ2WCEOymxBRQxwkaGRrmBBwqu sH791ic3pHBAaL/lupinUKFdl8h6wZEwZud61ee2JR9hRRhoF7t6Bg+sTAebmcuK5Cxw bZIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251196; x=1740855996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Xb9ZMWEswnCZqxNt4mNRr92ZbyiwfGCzO2RoU9QiMyc=; b=CQfvg+M9GD7zDaIjm1sGB4jll4Qrw/ceL3dD7uxxtON4kAqXu5Ug1Alfuh5Cjf/HG3 ixs7fWZg9h+5/TQlMuNF6tIPUeUw9tq2IvGxNFMIdfg0pIX/hrsSttiNxyJ/5rwzFz5D U8Gz9YPqDsVF1Q+aBPeyHL1nBvEKXEoItDL5/dpz5ReJQFXHgDZ9vz5vVlIWxeZef/GY CsxCxu4Y3dk+rKVXNp9nYIYpYzEcAHuno/Pn4wjC38Bt0UnOa9ZtXV2fBn7nb5IOejJ0 1Yrd1MB/oitNRT5Zau/9UmTiujlT53pIS28XsdOjS/znesHOmn8TYBDn+WwUPSYoP7ID 3xDQ== X-Gm-Message-State: AOJu0YyX0xnrT7BmmY8HIqkBpgQURmMHeKyMM59OqCszkG0/WnqnrFHG SkH8HgtguGUM72hLYHIzCj0IG4Ln+4S3NIhHO2VqrHAAVb8LP7QXukyB X-Gm-Gg: ASbGncu5DefXAUZLASogIPJVF42qHQUZCv8ssh+AYbD/0vcmspzvEaWMUwVT9s45rey me+0QQ53+BIiojaJZc23cOQFIywlhxpwe79ADo9aAl9CzT4k0xH57OU8CDjyP7SmAqdlDgYloSC ahuJZGU2+QnIkJ+thuJkEQVkUrKBDh6E5RG/oasGNcYjyF5DYzgxYNDzthz6buZDoI/MOFz8cLq D+nfzgUTNbj5ZeLPpb9vBLiVgiaopCxpl3MJuvMC0P7Ye5G36QpLIHSzWk8U7z64rcKc404leef xCjAniU= X-Google-Smtp-Source: AGHT+IGfMN71LzcwED6XyaagweDEqY2fd50Igd+UD/UEpy3LK9qv1UdfCEflzEfWCcFLjbYW3Ff6cw== X-Received: by 2002:a05:620a:4148:b0:7c0:c13f:4181 with SMTP id af79cd13be357-7c0cf8adfdemr972294285a.2.1740251196445; Sat, 22 Feb 2025 11:06:36 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:35 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 01/11] percpu: Introduce percpu hot section Date: Sat, 22 Feb 2025 14:06:13 -0500 Message-ID: <20250222190623.262689-2-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a subsection to the percpu data for frequently accessed variables that should remain cached on each processor. These varables should not be accessed from other processors to avoid cacheline bouncing. This is intended to replace the pcpu_hot struct on X86, and open up similar functionality to other architectures. Signed-off-by: Brian Gerst --- arch/x86/kernel/vmlinux.lds.S | 2 ++ include/asm-generic/vmlinux.lds.h | 10 ++++++++++ include/linux/percpu-defs.h | 10 ++++++++++ 3 files changed, 22 insertions(+) diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 1769a7126224..049485513f3c 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -187,6 +187,8 @@ SECTIONS =20 PAGE_ALIGNED_DATA(PAGE_SIZE) =20 + HOT_DATA(L1_CACHE_BYTES) + CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES) =20 DATA_DATA diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 92fc06f7da74..aaa83ec3afe4 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -385,6 +385,11 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPEL= LER_CLANG) . =3D ALIGN(PAGE_SIZE); \ __nosave_end =3D .; =20 +#define HOT_DATA(page_align) \ + . =3D ALIGN(page_align); \ + *(.data..hot) *(.data..hot.*) \ + . =3D ALIGN(page_align); + #define PAGE_ALIGNED_DATA(page_align) \ . =3D ALIGN(page_align); \ *(.data..page_aligned) \ @@ -1065,6 +1070,10 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROP= ELLER_CLANG) . =3D ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . =3D ALIGN(cacheline); \ + __per_cpu_hot_start =3D .; \ + *(.data..percpu..hot) *(.data..percpu..hot.*) \ + __per_cpu_hot_end =3D .; \ + . =3D ALIGN(cacheline); \ *(.data..percpu..read_mostly) \ . =3D ALIGN(cacheline); \ *(.data..percpu) \ @@ -1112,6 +1121,7 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPE= LLER_CLANG) INIT_TASK_DATA(inittask) \ NOSAVE_DATA \ PAGE_ALIGNED_DATA(pagealigned) \ + HOT_DATA(cacheline) \ CACHELINE_ALIGNED_DATA(cacheline) \ READ_MOSTLY_DATA(cacheline) \ DATA_DATA \ diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 40d34e032d5b..7db773ae5b52 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -112,6 +112,16 @@ #define DEFINE_PER_CPU(type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") =20 +/* + * Declaration/definition used for per-CPU variables that are frequently + * accessed and should be in a single cacheline. + */ +#define DECLARE_PER_CPU_HOT(type, name) \ + DECLARE_PER_CPU_SECTION(type, name, "..hot.." #name) + +#define DEFINE_PER_CPU_HOT(type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..hot.." #name) + /* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of t= he --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C45BF212D84 for ; Sat, 22 Feb 2025 19:06:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251203; cv=none; b=A92RHGjv1ReTB4J7VAWcS9c+44eFU2iH7FPVNJSW+5Pkx+ZsuqtFEE9eLHDFvQfH1pe2uT4DmXsz14h/C3PEOZDskCDFyyZY9My5YB7iIEewk/YA9Sf1UUpxCwTER9Jpv/gpNSydSexpf2UkDNIFUoGmxkAUIL3Toh0WGUN34Go= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251203; c=relaxed/simple; bh=GCDATYO1AJw/DO6eXDNV5UGJUlAocFyLKrDA5Y9egrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Nred4citTGew+aa7mYTEStIknQCt7ByGWU4BZuMVCf5kUUIKm10JA82uJDiIAU0TZ7P7h2neHrl0jU+cbwJJldPvxU2WpmhOMLGNMB2Jfi+mL1WgkjoE1h65qQJUpKmNQhkXBKqnlcPBMk0BdfkanKlHlBwORnlg9KuZssypyrA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l/inXcaL; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l/inXcaL" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-6e67fad4671so28010196d6.1 for ; Sat, 22 Feb 2025 11:06:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251199; x=1740855999; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LjXYEbe3KPR6UiCU0EfQ6entX/O21jgJvDj0fnAodH4=; b=l/inXcaLl9noTTRka8T2pYh5oKsf2zD+tMl9x8kBHjjj4x6IVty7WldFeroAnaO72L PFi1q7DoFEOrtw+N7WqYbf/leB0Pu1WX/ZKIgtqWDk3A1xZQYb0hNYnW+G0NSiCVzFzK v+3Jw+YmU9LJN9jOMNcJ2Sj8SMb4JvLHe4uDuABpESA9BP5seOuVpfFPvjq2zcR8NsEX RZt6y8C0nHyPVF0P0Q+J+Lxhm6d3N69oyR3UENtmL5gts1rtGZ15j38WLRXC+V6EV5Bh Q/9vnpFrd/ncbcfis6Vh9LK0gq/ybF6dGyv0TWJ6S2o30B7/XXtnr4v9aiiK6VIKXH7e jPMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251199; x=1740855999; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LjXYEbe3KPR6UiCU0EfQ6entX/O21jgJvDj0fnAodH4=; b=LTSnyO8UE9a221ruPgU/2Tnid7AepGRJ3hN0s9EgBFDPNgFPmW09dgtCwo4tTJmrjG 4+euCE/+jEi9FUiM/y1r1TT5TvfxwF1zqgFlme6nF4Dzv0wTaLFl/bBrEEv5b67hCslG qy7MAcXQdfQpVmmlwIoaDVLD5UxXYheWRzhV2OCcT0vK6yMTfWFX+YfocHGFbPWg6mNI 3qrj3RI3D4FDDa3Lr+NvpV1s1wqkYWrNpmWUJnXPrR69Lj1KQdK+80URmdqCSQxjeEnV AS+uglYHxDHqdJauG7Xc8nC9s/3cyfzqJlqhAk6vLOlCj61eRAZsu4GSAnHLZukJ8frr 6dBw== X-Gm-Message-State: AOJu0YwcH/r510fL6nJnXSsL+jwJuepKikqpxNwFA0bUKPMujHRDBR84 0a7sOmkIQSOCOM9/qmzgZwjsJ0w2xrk2psv8JAHGI/jqsBJuO3IPaAK6 X-Gm-Gg: ASbGncuIs2KU9i5+vhztvqEOclSY8xBeceaG/BHfi9HEKFzKLtZ4trY+omYkRBO5P2X ybPLIzLQM5s+l9n0wIvG2QHlVJjL8L+eXHZ8N/FHjqVsCqmPM3+rgi82sg/ZpZ/s2L1p5jZkEId jnnUyYpXxfH/OnQKgaY0XThyMMZvWR3iBSTndmseqT/d10KsUTwTea942oTzrSOm5X0oRR+OS2T qzkvQ6ClHXeBIiCy63OMERAlEBxVNN0/T0STCYdH2ozXtLpSYp9QL0zdcch6YKBGpbfyyp+a52c EKTbEuA= X-Google-Smtp-Source: AGHT+IGTkdzWrpW7dzRyjdyZtx+YvQudfz8Sv8S4UaagzhZB79LJ0eOt3gP+2pjjOQiJt6D0gdKw4Q== X-Received: by 2002:a05:6214:daf:b0:6e4:4331:aad9 with SMTP id 6a1803df08f44-6e6ae7c96e2mr95206856d6.2.1740251199251; Sat, 22 Feb 2025 11:06:39 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:38 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 02/11] x86/preempt: Move preempt count to percpu hot section Date: Sat, 22 Feb 2025 14:06:14 -0500 Message-ID: <20250222190623.262689-3-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/preempt.h | 25 +++++++++++++------------ arch/x86/kernel/cpu/common.c | 4 +++- 3 files changed, 16 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index bf5953883ec3..9a2fe2fd7d74 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { union { struct { struct task_struct *current_task; - int preempt_count; int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 919909d8cb77..2f3a40cbdd76 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -4,10 +4,11 @@ =20 #include #include -#include =20 #include =20 +DECLARE_PER_CPU_HOT(int, __preempt_count); + /* We use the MSB mostly because its available */ #define PREEMPT_NEED_RESCHED 0x80000000 =20 @@ -23,18 +24,18 @@ */ static __always_inline int preempt_count(void) { - return raw_cpu_read_4(pcpu_hot.preempt_count) & ~PREEMPT_NEED_RESCHED; + return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED; } =20 static __always_inline void preempt_count_set(int pc) { int old, new; =20 - old =3D raw_cpu_read_4(pcpu_hot.preempt_count); + old =3D raw_cpu_read_4(__preempt_count); do { new =3D (old & PREEMPT_NEED_RESCHED) | (pc & ~PREEMPT_NEED_RESCHED); - } while (!raw_cpu_try_cmpxchg_4(pcpu_hot.preempt_count, &old, new)); + } while (!raw_cpu_try_cmpxchg_4(__preempt_count, &old, new)); } =20 /* @@ -43,7 +44,7 @@ static __always_inline void preempt_count_set(int pc) #define init_task_preempt_count(p) do { } while (0) =20 #define init_idle_preempt_count(p, cpu) do { \ - per_cpu(pcpu_hot.preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ + per_cpu(__preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ } while (0) =20 /* @@ -57,17 +58,17 @@ static __always_inline void preempt_count_set(int pc) =20 static __always_inline void set_preempt_need_resched(void) { - raw_cpu_and_4(pcpu_hot.preempt_count, ~PREEMPT_NEED_RESCHED); + raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED); } =20 static __always_inline void clear_preempt_need_resched(void) { - raw_cpu_or_4(pcpu_hot.preempt_count, PREEMPT_NEED_RESCHED); + raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED); } =20 static __always_inline bool test_preempt_need_resched(void) { - return !(raw_cpu_read_4(pcpu_hot.preempt_count) & PREEMPT_NEED_RESCHED); + return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED); } =20 /* @@ -76,12 +77,12 @@ static __always_inline bool test_preempt_need_resched(v= oid) =20 static __always_inline void __preempt_count_add(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, val); + raw_cpu_add_4(__preempt_count, val); } =20 static __always_inline void __preempt_count_sub(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, -val); + raw_cpu_add_4(__preempt_count, -val); } =20 /* @@ -91,7 +92,7 @@ static __always_inline void __preempt_count_sub(int val) */ static __always_inline bool __preempt_count_dec_and_test(void) { - return GEN_UNARY_RMWcc("decl", __my_cpu_var(pcpu_hot.preempt_count), e, + return GEN_UNARY_RMWcc("decl", __my_cpu_var(__preempt_count), e, __percpu_arg([var])); } =20 @@ -100,7 +101,7 @@ static __always_inline bool __preempt_count_dec_and_tes= t(void) */ static __always_inline bool should_resched(int preempt_offset) { - return unlikely(raw_cpu_read_4(pcpu_hot.preempt_count) =3D=3D preempt_off= set); + return unlikely(raw_cpu_read_4(__preempt_count) =3D=3D preempt_offset); } =20 #ifdef CONFIG_PREEMPTION diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 8b49b1338f76..519e2ec2027d 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2016,12 +2016,14 @@ __setup("clearcpuid=3D", setup_clearcpuid); =20 DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .preempt_count =3D INIT_PREEMPT_COUNT, .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); =20 +DEFINE_PER_CPU_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; +EXPORT_PER_CPU_SYMBOL(__preempt_count); + #ifdef CONFIG_X86_64 static void wrmsrl_cstar(unsigned long val) { --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EC031E7C24 for ; Sat, 22 Feb 2025 19:06:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251204; cv=none; b=hcUNRyXaODLyw9miNFyImP/t4l77g/RAB3ED74Z498ZJKTiGthFFcV7Yg9iIUCtwZlRNcziTSIvO5XoxT73NyaPb1xPivP5mgQsjCAVXWp+tAGx8jglnHaAeJ+Kk4Jhb6XDPy4rXeDHV+svV3rNPVP6kL3ur1MqTrEF5bNdToW0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251204; c=relaxed/simple; bh=pY81Pf0J2Mda6k+mwhlw8D99MebxoATiJa08HmwXH1g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jbsa8xDmIIsEteNifKldBd6bzPeh2gdI09EgzL2sP8mzRu4vCBCKLCmXG9ScWQ6MUCrQx+UuYuOD2m8P+yD9MXk+PSxCAaqGuzkmd2CH9xdKrUhZpAiBChY8mi8vI6Oh7OBV7sHa7F4d9cYvImkOna0ubloZGIU8fapcV7WPSjw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lxhrArhU; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lxhrArhU" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7c0a3d6a6e4so299915185a.1 for ; Sat, 22 Feb 2025 11:06:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251201; x=1740856001; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d8vjxJjSFTpAMGQl6LtUKE5Ud48F2ROpDpIRh9DuarI=; b=lxhrArhUpyJSGaD/YnqGKlt2Jt/MtxjVyUZ+V86Y++z97BYmljA2n1UpEjwtG66Koy TTYRlIoxS48z1IT5rmiyG03nf6N2dA/ePjukTmTuUFFkNWeqIwb/UnFu94VyhLHzwVS9 1FmsoLUHjpkdnTdZCKkKWeJetLXXADQt/0nfC75ptw6TuQDbe8QsJpTimAYCgj7k/jhr rJEfkHADmixt9iyggN/J1lFbg/hGmx83JFH++VttRGWRGALp1B37/r+OXX37SsBkOb3V w1pdKFTfywGx+9kj6WBoqDm5CUzH1i32EaClg2QV64V1U1H9GMH7HQnrkyjEUrD5EWuG IvOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251201; x=1740856001; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d8vjxJjSFTpAMGQl6LtUKE5Ud48F2ROpDpIRh9DuarI=; b=kzFDP+Y3JMLxlyIeqYWwp7tIpKp0GaTBSJR74X4mPdRWdtDRz6132M1XCrWp6idkS/ t7YVafvYN1GDsKKaN+2wvBSHcCaU3uc+vjeEQo2adOaTrqWkpppT00vOWGf+JI+w5Vja YrfEmwvd1r0gCaouUhDYxlvFh5VLwlAA+2C2BIBsT/b0uUOGGzYnyXzF3bSZx8J8klAz OGeGyBP3NIqbbfBXrrbqFXFRNIdhFDJFoHsdgyQP1ibGNreMstjspmGapylH7Quo/ykZ 6p+J7UczXolJwMh8yvhfYSxK+oV5uEtl5BRfGrwMVhAlXS1ae/KVZkfQqj1M9mAv7/4K F4jQ== X-Gm-Message-State: AOJu0Yx55DvIyXyxTkwWSz8SUPI9IgNtK/qFZgDEzUrofu/8pEIw/zPH iReBf3gkEywBZQOxunqbEHqSiwtNHkOz8DYOnZ82NEGGH8RsuQlgkg3d X-Gm-Gg: ASbGncuz13h2lrLu0+sYC8sSr86/sBI6B/C5W9fRA7zC0dQautuG6M09FIKF//82xGs g9Xzk4dZEdDMUEIokFLu+lwZnQi0FEdSIm0TOnTUBAigZK2/60Fm3seNNUT/491TaWNCn1ZsRQM omCbWZmgrguIiExf+kZNF1QBGDLfL1rnPYi5p28U2U77wpGC+zjBXz58PLv5lt8Po12OlR5LB+q 6FM0KhM/S8mogJvIuFjtng2B2pfQ67UoO/u+3VwjlM6Nizr3/VipQSkFuudL+B2+KdkpkHJ9zbp B17UacI= X-Google-Smtp-Source: AGHT+IEbJkf2NLkdpxnoYBolL3+34rvNiPGTiEKBNQzOGJnaONuN5mT3WVORaaLBpVmEcJHBeDzJBA== X-Received: by 2002:a05:620a:450c:b0:7c0:c282:702a with SMTP id af79cd13be357-7c0cef484a4mr1063643285a.39.1740251201177; Sat, 22 Feb 2025 11:06:41 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:39 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 03/11] x86/smp: Move cpu number to percpu hot section Date: Sat, 22 Feb 2025 14:06:15 -0500 Message-ID: <20250222190623.262689-4-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/smp.h | 7 ++++--- arch/x86/kernel/setup_percpu.c | 5 ++++- kernel/bpf/verifier.c | 4 ++-- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 9a2fe2fd7d74..119099431c76 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { union { struct { struct task_struct *current_task; - int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; #endif diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index ca073f40698f..a249f9991708 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -5,9 +5,10 @@ #include =20 #include -#include #include =20 +DECLARE_PER_CPU_HOT(int, cpu_number); + DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map); DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_die_map); @@ -133,8 +134,8 @@ __visible void smp_call_function_single_interrupt(struc= t pt_regs *r); * This function is needed by all SMP systems. It must _always_ be valid * from the initial startup. */ -#define raw_smp_processor_id() this_cpu_read(pcpu_hot.cpu_number) -#define __smp_processor_id() __this_cpu_read(pcpu_hot.cpu_number) +#define raw_smp_processor_id() this_cpu_read(cpu_number) +#define __smp_processor_id() __this_cpu_read(cpu_number) =20 #ifdef CONFIG_X86_32 extern int safe_smp_processor_id(void); diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 1e7be9409aa2..0ea3443433c5 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -23,6 +23,9 @@ #include #include =20 +DEFINE_PER_CPU_HOT(int, cpu_number); +EXPORT_PER_CPU_SYMBOL(cpu_number); + DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 @@ -161,7 +164,7 @@ void __init setup_per_cpu_areas(void) for_each_possible_cpu(cpu) { per_cpu_offset(cpu) =3D delta + pcpu_unit_offsets[cpu]; per_cpu(this_cpu_off, cpu) =3D per_cpu_offset(cpu); - per_cpu(pcpu_hot.cpu_number, cpu) =3D cpu; + per_cpu(cpu_number, cpu) =3D cpu; setup_percpu_segment(cpu); /* * Copy data used in early init routines from the diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9971c03adfd5..604134d33282 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -21687,12 +21687,12 @@ static int do_misc_fixups(struct bpf_verifier_env= *env) if (insn->imm =3D=3D BPF_FUNC_get_smp_processor_id && verifier_inlines_helper_call(env, insn->imm)) { /* BPF_FUNC_get_smp_processor_id inlining is an - * optimization, so if pcpu_hot.cpu_number is ever + * optimization, so if cpu_number is ever * changed in some incompatible and hard to support * way, it's fine to back out this inlining logic */ #ifdef CONFIG_SMP - insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&pcpu_hot.= cpu_number); + insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_numbe= r); insn_buf[1] =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); insn_buf[2] =3D BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); cnt =3D 3; --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D6B1213232 for ; Sat, 22 Feb 2025 19:06:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251207; cv=none; b=lAm3IedyfwttiMZ4kscpN5bX62u7t768KWEELmURmnif0EM9fLCtRUGhk2SGW4MtfDGP5ll5+pq66G2y5KetrYNL8gsnECqn7wWFG143g+q8z3jXmr/3sJ1hNBc+EYvOlWDuYOo8xHR+GrWbV8yBZO0g6PGpStK55vZC+eek8O4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251207; c=relaxed/simple; bh=MEFNrN3i/2AaKco1k/9HpXbv7F2of2F4/2qWL9sh8oM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mdIi676fnkCzOLu+KeqeuaJ5VD/lGevIH+lIxQVkHV20J2zBoZKzT5qrkwFa82Y0RP/rc4uy17YSauK0lWi/4vEkedSuRykSlwu9Q0O9XTBneWVmCH46JtutoOZ631FWGJzbQrjd4SzFf2rULV+Zx2S+1Sc+y0V1XKmJpDUNdJE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QwQ59tSI; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QwQ59tSI" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-7c0ba9825e9so201452285a.0 for ; Sat, 22 Feb 2025 11:06:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251204; x=1740856004; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=s9GIY+0lnUqZnRfDg+g6eHAlVXI9z0IE0EDIxYSYxYc=; b=QwQ59tSITRnihbFmmzmb5DRHD19VV+x6iVFCnKP6qBicXFpBdp+kJb542DvAkliQAb ydqN7xkrZRzAggG1iTGUgFmOZnBPJPhjAfh1TtY50d2JSVJHBLiDz2Mxro3f18WEojqq T5SPMIpjMhgaoPe+emCKBt/0yCmAWR3yQixegqE4EWxUZPdFbUpB3Y4r/yVLwuA6wQCx ZDPEem3kfmjND8XblGuX79WwrlB+cFnhcTNrGC+FBMixJ8MlD4OAO33I/lLmFF1rRARw shcIGuyg3sveJoByOaL9MUd5CiIphH419mA5/qTo7PyqHeon0EBGF2ZVeBx8MH72ufUs ZqrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251204; x=1740856004; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=s9GIY+0lnUqZnRfDg+g6eHAlVXI9z0IE0EDIxYSYxYc=; b=U2f8mwsN99n6js7AN330UZkWMIcQh1J6WAwsy2BHGpk6lS/0Bd/8xQJA3faaPHN/eA bEyUhroB+NNwRNBnjSkWZbGawYtk3d/n3iFYqI0JEgM606EDJSj3wVZZ8h303vIOJzqX D3Fn19HFHdj0zo2JApJJ4+Xwsqe+1H7rk+e0osg6YQnkwHZeoHOXYoT5bzMaI7ti5AJV ulmyTsWTe/2paWsfcbWkgDhhiIhrdTPufVbXjWvg1DqkBJiys7CP3P2Yl3zaQa7szQyn tlpBGnofMQprix8SJM43fyNaRIK/7uVl529V3kjPUfMVSKuYMjYbLpCEtTo5R3z4Q/gN IUFQ== X-Gm-Message-State: AOJu0YxfvrbwchPMKVVDW8Pr7oTpcyayzX4F0mzIfD6ejB6U4+wLzluK FN7Gmqt5HXVQ6V6iasx1F1KkyqlHV5XCIxZRo1QIbXiU3Xp4wEqTvJ91 X-Gm-Gg: ASbGncuF/53PAfDtKEYWrxE61/rfiNibKzDIPg8tDDL7nHFTGKUK308zYEKukAwewXH zjqP3bfJyzcIv3R9tp27GmaOwthCbM4fUcwxOpJQclrDMYfHrlIPd1bKt99KC99TeM5ZTn0D/yP sWHTzhk6hfEbRfZljJke9v0bsR0g1amwmbwngkZ2UXtuO7sr+3RRbz4BUU+2aF/U+ICH98MtP4j //8Wp8FcLSP5tU1v41KqbiJ4GMcm8j+ZTIebFZiBksyQc3RW1kLV+FSJLKWBGz+EQ0Zzmnu6gxl CP1zPHg= X-Google-Smtp-Source: AGHT+IG7+CewSHiEKQLMuuTRv8biNJ3hb9nw9N0yOAiP9fFpvFp5GbMxkkMiXBDHYxq7al086WXRxw== X-Received: by 2002:a05:620a:1787:b0:7c0:af16:b4b5 with SMTP id af79cd13be357-7c0ceeeb53dmr1275755585a.6.1740251204017; Sat, 22 Feb 2025 11:06:44 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:42 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 04/11] x86/retbleed: Move call depth to percpu hot section Date: Sat, 22 Feb 2025 14:06:16 -0500 Message-ID: <20250222190623.262689-5-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 3 --- arch/x86/include/asm/nospec-branch.h | 10 ++++++---- arch/x86/kernel/asm-offsets.c | 3 --- arch/x86/kernel/callthunks.c | 3 +++ arch/x86/lib/retpoline.S | 2 +- 5 files changed, 10 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 119099431c76..fbc7eb92adb2 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,9 +16,6 @@ struct pcpu_hot { union { struct { struct task_struct *current_task; -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - u64 call_depth; -#endif unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 7e8bf78c03d5..8441d1da5382 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -78,21 +78,21 @@ #include =20 #define CREDIT_CALL_DEPTH \ - movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq $-1, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH \ xor %eax, %eax; \ bts $63, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq %rax, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH_FROM_CALL \ movb $0xfc, %al; \ shl $56, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + movq %rax, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #define INCREMENT_CALL_DEPTH \ - sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + sarq $5, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #else @@ -388,6 +388,8 @@ extern void call_depth_return_thunk(void); __stringify(INCREMENT_CALL_DEPTH), \ X86_FEATURE_CALL_DEPTH) =20 +DECLARE_PER_CPU_HOT(u64, __x86_call_depth); + #ifdef CONFIG_CALL_THUNKS_DEBUG DECLARE_PER_CPU(u64, __x86_call_count); DECLARE_PER_CPU(u64, __x86_ret_count); diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index a98020bf31bb..6fae88f8ae1e 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -109,9 +109,6 @@ static void __used common(void) OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - OFFSET(X86_call_depth, pcpu_hot, call_depth); -#endif #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/callthunks.c b/arch/x86/kernel/callthunks.c index 8418a892d195..7032d4caf242 100644 --- a/arch/x86/kernel/callthunks.c +++ b/arch/x86/kernel/callthunks.c @@ -39,6 +39,9 @@ static int __init debug_thunks(char *str) } __setup("debug-callthunks", debug_thunks); =20 +DEFINE_PER_CPU_HOT(u64, __x86_call_depth); +EXPORT_PER_CPU_SYMBOL(__x86_call_depth); + #ifdef CONFIG_CALL_THUNKS_DEBUG DEFINE_PER_CPU(u64, __x86_call_count); DEFINE_PER_CPU(u64, __x86_ret_count); diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 391059b2c6fb..04502e843de0 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -342,7 +342,7 @@ SYM_FUNC_START(call_depth_return_thunk) * case. */ CALL_THUNKS_DEBUG_INC_RETS - shlq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth) + shlq $5, PER_CPU_VAR(__x86_call_depth) jz 1f ANNOTATE_UNRET_SAFE ret --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A55A2135B6 for ; Sat, 22 Feb 2025 19:06:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251209; cv=none; b=AWRa4q3rfT4U4Q1WYnFKVKr01xwibKlT4rWPzF/yi6YCNirysVWR0S4A6XhRXDAoGQMS7LS8x/NEupOU4US0rODqDNCKMNxECppoY6tPt6cy5YUt7Y7Pf2WtR/JQxOTAP12Ly7306+hCeGUKOKjrZ3UE0ciITCxht9DBPaCVKpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251209; c=relaxed/simple; bh=NypTYh7L8SC1KJhpPOvWL/N7u1oyIHpqB+4dt8A1OC8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tCaMuv9uOEXWL1L7U9+kCJ+T0TBf04Rvcg6EPwiP3yrpE9Ns/NikrCJx7apQ73ItTZfzIWz3gn46wlZoOGL9iLk1EdikWDPbMNL0JiU0RF3i4aUjycfv79Fv+HvzrMSla7Hmau4iYKCHacPMN3Cy1PGdNZlJv2yzDTsUdrzyzr8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=D3kaFxrk; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="D3kaFxrk" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-7c07b65efeeso313217485a.2 for ; Sat, 22 Feb 2025 11:06:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251206; x=1740856006; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wh8GkvDr8pzzxT2Gi5xX+rdR9qzvPHbFDIFqugJ5S/M=; b=D3kaFxrk2eBNSzeTM8l4skgv3kIddltlYqqMkH9BZ8KAM0gJO14C1HuDpY5GpNaaQI fE/h2171uqX3cbHm+CgkRVSh/iRgk+OmGEHVhQPhnvBEBCJ4u0UlGA5/bphpJA46BH7x j3P2QVf1ptW0fyK1M6CVH4BqZas/gQSxHdG+9oxCNlS/MhJrKeO9m9OKI5SDgxJQGCNX T3HIQLUsRUTgQO0+Xy8vU7qGbkdXNGe4/u44i9OWfvaDx18EIKBS05JfunSPNs8VYYmz C1LAXna/LneWgOGsw4B1oP0QZcmqJ1N5hWvUS4OUr9XhayAE381qER4K89vwRT8EXXNo A3hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251206; x=1740856006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wh8GkvDr8pzzxT2Gi5xX+rdR9qzvPHbFDIFqugJ5S/M=; b=arZfQjWeV1nQ4tHdrWSbgHiCqIyCRtJTj/2vZ/5O1I++6fH500GP+UeLatjSFIz6zv CthrDjmyrisJH30O5tpZPYnys3A9lZ9IFruCMI610ubIeLx05MYNN6fqSmwHVs5heI7K YYlmLsVGU7UWvPnLose2VNZ5iogcDkEOG1YthI9NJyb0DAtfWFnCPVrIN/ORiLaj5gAk CeubrhYLO6bc7PfRAzIbnoOKH9yEKX81GFHnAw+fCMrLdjPUondUAv9KFHLrsbX64n00 lEhuygLCkLIppLaN8n7Pt3HllwOVcBbM7SUVeujiAWoxzMfplihFS1n6h+DDd6twMMqx iweQ== X-Gm-Message-State: AOJu0YylYWPgHi7PZ+mgCZtXdCLyIgbjTbpfL+Igy24qJlmGnIr2azKJ R3JXlHFpsFHqMwpy+X32PGrNsGlHqCbdzGAPcVemXjwCSMKBQPPlpXSD X-Gm-Gg: ASbGnctLejlBZL3yULkuDVC/3b1OFKjmPP61BSOU6DxzuBtapgqeQNR4SC/8cxr1/7k nacDFhTZ969E5PLOaVYW7GU8kWZkjp2T0N3DpjT8b06KpKWkl1mxTBtLzlg7RWRMKM0w/Bamfpt rPxdioLarC4ktGhW+4WT6ez2eyFR4s7CfxcRODlhJnyYECPsc5LYxCmQnesyMuzinyI1pJrGazW korhxCF7djpgVvYpGNeFMu5aTTEynrlPnyYWXBrFi0+j+Cid6x6xONgwBWHIPvGBWN6STmW8+30 jR2yUqM= X-Google-Smtp-Source: AGHT+IFxiOgVdsLjMAFTywuRwNAZssF9gbcZqBt4beLn4CDQLMs1x+W9x0C1ucAkWUHvRAUM93uydA== X-Received: by 2002:a05:620a:444b:b0:7c0:c00e:5913 with SMTP id af79cd13be357-7c0cef5c360mr1322168385a.48.1740251205957; Sat, 22 Feb 2025 11:06:45 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:44 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 05/11] x86/percpu: Move top_of_stack to percpu hot section Date: Sat, 22 Feb 2025 14:06:17 -0500 Message-ID: <20250222190623.262689-6-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/entry/entry_32.S | 4 ++-- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 4 ++-- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/percpu.h | 2 +- arch/x86/include/asm/processor.h | 8 ++++++-- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 3 ++- arch/x86/kernel/process_32.c | 4 ++-- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 + 12 files changed, 21 insertions(+), 17 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 20be5758c2d2..92c0b4a94e0a 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -1153,7 +1153,7 @@ SYM_CODE_START(asm_exc_nmi) * is using the thread stack right now, so it's safe for us to use it. */ movl %esp, %ebx - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esp + movl PER_CPU_VAR(cpu_current_top_of_stack), %esp call exc_nmi movl %ebx, %esp =20 @@ -1217,7 +1217,7 @@ SYM_CODE_START(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esi + movl PER_CPU_VAR(cpu_current_top_of_stack), %esi leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp =20 call make_task_dead diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 33a955aa01d8..9baf32a7a118 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -92,7 +92,7 @@ SYM_CODE_START(entry_SYSCALL_64) /* tss.sp2 is scratch space. */ movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -1166,7 +1166,7 @@ SYM_CODE_START(asm_exc_nmi) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ @@ -1484,7 +1484,7 @@ SYM_CODE_START_NOALIGN(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rax + movq PER_CPU_VAR(cpu_current_top_of_stack), %rax leaq -PTREGS_SIZE(%rax), %rsp UNWIND_HINT_REGS =20 diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index ed0a5f2dc129..a45e1125fc6c 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -57,7 +57,7 @@ SYM_CODE_START(entry_SYSENTER_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax popq %rax =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ @@ -193,7 +193,7 @@ SYM_CODE_START(entry_SYSCALL_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp =20 /* Switch to the kernel stack */ - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index fbc7eb92adb2..8adbe0e3c5e7 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { union { struct { struct task_struct *current_task; - unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; #ifdef CONFIG_X86_64 diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 1a76eb87c5d8..cc19bd785f0e 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -560,7 +560,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and pcpu_hot.top_of_stack, both of which are + * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index b3d153730f63..1505cb1d09a8 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -420,6 +420,10 @@ struct irq_stack { char stack[IRQ_STACK_SIZE]; } __aligned(IRQ_STACK_SIZE); =20 +DECLARE_PER_CPU_HOT(unsigned long, cpu_current_top_of_stack); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_HOT(const unsigned long, const_cpu_current_top_of_stack); + #ifdef CONFIG_X86_64 static inline unsigned long cpu_kernelmode_gs_base(int cpu) { @@ -545,9 +549,9 @@ static __always_inline unsigned long current_top_of_sta= ck(void) * entry trampoline. */ if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.top_of_stack); + return this_cpu_read_const(const_cpu_current_top_of_stack); =20 - return this_cpu_read_stable(pcpu_hot.top_of_stack); + return this_cpu_read_stable(cpu_current_top_of_stack); } =20 static __always_inline bool on_thread_stack(void) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 6fae88f8ae1e..54ace808defd 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 519e2ec2027d..25a5806e15aa 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2016,7 +2016,6 @@ __setup("clearcpuid=3D", setup_clearcpuid); =20 DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); @@ -2024,6 +2023,8 @@ EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); DEFINE_PER_CPU_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 +DEFINE_PER_CPU_HOT(unsigned long, cpu_current_top_of_stack) =3D TOP_OF_INI= T_STACK; + #ifdef CONFIG_X86_64 static void wrmsrl_cstar(unsigned long val) { diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 0917c7f25720..3afb2428bedb 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -190,13 +190,13 @@ __switch_to(struct task_struct *prev_p, struct task_s= truct *next_p) arch_end_context_switch(next_p); =20 /* - * Reload esp0 and pcpu_hot.top_of_stack. This changes + * Reload esp0 and cpu_current_top_of_stack. This changes * current_thread_info(). Refresh the SYSENTER configuration in * case prev or next is vm86. */ update_task_stack(next_p); refresh_sysenter_cs(next); - this_cpu_write(pcpu_hot.top_of_stack, + this_cpu_write(cpu_current_top_of_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 226472332a70..4252b11718f2 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -669,7 +669,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) * Switch the PDA and FPU contexts. */ raw_cpu_write(pcpu_hot.current_task, next_p); - raw_cpu_write(pcpu_hot.top_of_stack, task_top_of_stack(next_p)); + raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index c10850ae6f09..15e054f4cbf6 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -851,7 +851,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) =20 #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ - per_cpu(pcpu_hot.top_of_stack, cpu) =3D task_top_of_stack(idle); + per_cpu(cpu_current_top_of_stack, cpu) =3D task_top_of_stack(idle); #endif return 0; } diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 049485513f3c..ee019c1ea859 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -44,6 +44,7 @@ ENTRY(phys_startup_64) =20 jiffies =3D jiffies_64; const_pcpu_hot =3D pcpu_hot; +const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) /* --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A95FB2135D3 for ; Sat, 22 Feb 2025 19:06:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251210; cv=none; b=hywbnLbUEWUAUy7oLgDCEiaLWJPbfsCt6xrxKvw6krysmxojmeo6D/3RAvHlgLVwkQTr9YD2fj/k54ttwXb4MJPdzQcSyy6hARhxOgps8rcWDSTx5Er/afHnRxN/sN3DYNio9niWC94pJjNCWD6oRr5lWpzJmGGgaPZJ2EIuqFI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251210; c=relaxed/simple; bh=8j7HS6KRodJDG6j7e7qkjuAA7iFjSETcH3RrEkYZ8nA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UvkvNjQ+wemSTfnvB8x8i3Jxgb65couUYc9rL29RbuMPgOpFzwY7c7qZsuht1YSsXR9HCtgnsaax+nmppf1IDghssDSyS3tB1cET6FD9BSk22tGCXsq6M2M2yhX7NUpnPxXyb/UY472XqBMWtu/rA/O9xYEptY4gqs7HBxGP+Xs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=EwF/vxFq; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EwF/vxFq" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c0818add57so334002985a.3 for ; Sat, 22 Feb 2025 11:06:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251207; x=1740856007; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0fJRQIUEeUCOVvELxBP9zDInBoFRFieHqHEjsUAYH7k=; b=EwF/vxFqKJtEBWpgBCKzX9YE0gnX88P4KBoCIsIv7SUeTcsaITyfWazn+vqIiUHYgP cu2uYe+DOMnXqjogLkM8+og0Dg6+ig6ROndSsC9Paad0+aaofEB2VnHobq3XBavPJq6c H6kTQK6kRsgl4iAZ1Rg+HcYtlW1UGDVL1CcrgB0UAzBlQ9bB+zwgVk9ByqlvEAalPQLM cXT6oNyrUEvFIaWZqO9A8PZHpShz+kC7NgwxFUVRpUFI+TONbDf9TtOCBQSJfFwhpvFf +2ZB0VT+J1sQfPrJxnH4dD8/ZH/NodL5d0mjr7BMZOskXJI+P3yEE+UyqFEdasmiP2QA Fs5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251207; x=1740856007; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0fJRQIUEeUCOVvELxBP9zDInBoFRFieHqHEjsUAYH7k=; b=kFyU9hM3jPjGwGwX5zTXrUYaDLbhIHbpSIIDncaLlskiV7lq/qigQdhbf8pmRHxO1G YAjDgC/3l8UALJs52oDZyRLgctq6yK3rp56/+1R2PhlSTfweKiieTbTuAuAnzpZUuMlB p3uyoffUSwnjPrlVlz0WUdqqF796k+gac1Akn/vaAihwl0H3ZpH/1gXpcbLoPYVRKEhh BE3h6oXgQCrQODRKa38oBun4w1Lde962w82jnnDryaJwO3dGMoh3Ve0aiaugmzNOWMkR 3u7FJ0GWPolZfBZiHLYZdJI1fEpoVBqJ73Qc4EyWCq5SaaVDXrrfRzBf/0V1O+XZpXP1 qODA== X-Gm-Message-State: AOJu0Yz+1qLfIc+QVDvB2q2Iof6BoBmg19c+1GJnBv8uDZLr+aIcwOYP 1kfhiwmCncx1hMmiK2uz1sxbwwPYrV4nd0sYjdGA83zzdNRUVySizoEN X-Gm-Gg: ASbGnctZRrIfmEYvTdXY6DH/WZh3voUv1CatfTgQJ22b7HByAvk/v+FcydYRdzYhwS6 nMhSX5UQnqW2iyhG4/L4WC+8WzlXNhWnqGnCaMD00L9qE4jQs1H3Iwes5Uyng/HDH0Pl5AZGW69 EMrl2T0Kp96lgMlEsr9WJFlsq93albO/i+NHwgJrHUOHVT7eoaTUhiYiIWgcRmF26gcNWpKMHJz bl7HBwm2rVWHynbjKgFGINrcQnshH4zWAR1cktSCcXjFjuEMHO2ZOc5H+vcAe6EnVDg3lRvyZAe dYY+ktw= X-Google-Smtp-Source: AGHT+IFzVSDtGTaKw58OMNWSJZxoGLxdJ8q5okT1JUktKE86ry7B+FjY/K8mwuGW1cs22+PRO2b3EQ== X-Received: by 2002:a05:620a:2a0f:b0:7c0:b76a:51d5 with SMTP id af79cd13be357-7c0cf7d816bmr993573185a.0.1740251207134; Sat, 22 Feb 2025 11:06:47 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:46 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 06/11] x86/percpu: Move current_task to percpu hot section Date: Sat, 22 Feb 2025 14:06:18 -0500 Message-ID: <20250222190623.262689-7-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 9 ++++++--- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 8 +++++--- arch/x86/kernel/head_64.S | 4 ++-- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 + scripts/gdb/linux/cpus.py | 2 +- 10 files changed, 19 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 8adbe0e3c5e7..d51299af6145 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -15,7 +15,6 @@ struct task_struct; struct pcpu_hot { union { struct { - struct task_struct *current_task; void *hardirq_stack_ptr; u16 softirq_pending; #ifdef CONFIG_X86_64 @@ -35,12 +34,16 @@ DECLARE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); DECLARE_PER_CPU_ALIGNED(const struct pcpu_hot __percpu_seg_override, const_pcpu_hot); =20 +DECLARE_PER_CPU_HOT(struct task_struct *, current_task); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_HOT(struct task_struct * const, const_current_task); + static __always_inline struct task_struct *get_current(void) { if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.current_task); + return this_cpu_read_const(const_current_task); =20 - return this_cpu_read_stable(pcpu_hot.current_task); + return this_cpu_read_stable(current_task); } =20 #define current get_current() diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index cc19bd785f0e..370778c55091 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -560,7 +560,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are + * current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 54ace808defd..ad4ea6fb3b6c 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 25a5806e15aa..f4ec6bcb2a5e 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2014,12 +2014,14 @@ static __init int setup_clearcpuid(char *arg) } __setup("clearcpuid=3D", setup_clearcpuid); =20 -DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { - .current_task =3D &init_task, -}; +DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); =20 +DEFINE_PER_CPU_HOT(struct task_struct *, current_task) =3D &init_task; +EXPORT_PER_CPU_SYMBOL(current_task); +EXPORT_PER_CPU_SYMBOL(const_current_task); + DEFINE_PER_CPU_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 2843b0a56198..fefe2a25cf02 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -322,7 +322,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) * * RDX contains the per-cpu offset */ - movq pcpu_hot + X86_current_task(%rdx), %rax + movq current_task(%rdx), %rax movq TASK_threadsp(%rax), %rsp =20 /* @@ -433,7 +433,7 @@ SYM_CODE_START(soft_restart_cpu) UNWIND_HINT_END_OF_STACK =20 /* Find the idle task stack */ - movq PER_CPU_VAR(pcpu_hot + X86_current_task), %rcx + movq PER_CPU_VAR(current_task), %rcx movq TASK_threadsp(%rcx), %rsp =20 jmp .Ljump_to_C_code diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 3afb2428bedb..c276dfda387f 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -206,7 +206,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) if (prev->gs | next->gs) loadsegment(gs, next->gs); =20 - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4252b11718f2..1517314da34a 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -668,7 +668,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) /* * Switch the PDA and FPU contexts. */ - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 15e054f4cbf6..c89545a61d08 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -841,7 +841,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) /* Just in case we booted with a single CPU. */ alternatives_enable_smp(); =20 - per_cpu(pcpu_hot.current_task, cpu) =3D idle; + per_cpu(current_task, cpu) =3D idle; cpu_init_stack_canary(cpu, idle); =20 /* Initialize the interrupt stack(s) */ diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index ee019c1ea859..3c87bb620434 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -44,6 +44,7 @@ ENTRY(phys_startup_64) =20 jiffies =3D jiffies_64; const_pcpu_hot =3D pcpu_hot; +const_current_task =3D current_task; const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) diff --git a/scripts/gdb/linux/cpus.py b/scripts/gdb/linux/cpus.py index 13eb8b3901b8..8f7c4fb78c2c 100644 --- a/scripts/gdb/linux/cpus.py +++ b/scripts/gdb/linux/cpus.py @@ -164,7 +164,7 @@ def get_current_task(cpu): var_ptr =3D gdb.parse_and_eval("(struct task_struct *)cpu_task= s[0].task") return var_ptr.dereference() else: - var_ptr =3D gdb.parse_and_eval("&pcpu_hot.current_task") + var_ptr =3D gdb.parse_and_eval("¤t_task") return per_cpu(var_ptr, cpu).dereference() elif utils.is_target_arch("aarch64"): current_task_addr =3D gdb.parse_and_eval("(unsigned long)$SP_EL0") --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B16F3213E71 for ; Sat, 22 Feb 2025 19:06:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251213; cv=none; b=XszLAYPJZL9Wu7/xJXrxiewLrsGpHsnDpkWRb4qiaHoT4WEolbKhBdVjsNYsEb0dIsx5W/DCY+R7ucZ3uVLzUADKEJs1oU5Ntu+YwCxRIz6sHuov2XP23NAKQGW51F+b0zRAsA9lWOQA190Gla/6GfNLwvkfcYnfNIfQi9fCc7g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251213; c=relaxed/simple; bh=T7mEmpFC1wFdy0W9hebceemBNBWGIYtmF1qKxtl3z7c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GcUXkdXzPmpWe1PQzxbh6Hi6CteZtKarE238aK9yAjBVZNDMIi9W0npvYTFUlPoBSFKMlQvzJdFXvp3PqOn86heEA3VuQb9Fj3sLt8074jCkTtau14OIicNXT8r4bg2jAYIc8Vv0602+4i7bLxlavecyAp9YLTVDlIBHArHSai8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AqeKrO7l; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AqeKrO7l" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7c0a1c4780bso328431885a.3 for ; Sat, 22 Feb 2025 11:06:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251210; x=1740856010; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UEr+HVsM6xtJa9sjUwL5+hzjYi8sZRQw9jjeFW5Hz0I=; b=AqeKrO7lWVnZQJ5+MYm72xsgxVQ3HBG4nx/zGodSlzbXGxbaHU5EUN86KH9vAxh+vg ULJB0kJjmsN+V9zl/ae6fr493Tf/CO1/BA2TliAmBumY9yfsjhNz/NnUzgV1r24pLe1i ZfuMwbDzPpoROPW3e8TLAj8m0kIZ/PQ3BJ4OuBJVsxKD1TqKbDuwJWHVoXBJMPeYa6E1 2wiM4bm/4petORt47U4mL4mzrK6ki1AJIEZxGvjXJyUm1ccOQJayc0tKykL+LqCbSyBf EHnTvokKyIpVRF8tRUxnbXMkYRR+eusiEcATzyt7Fw2BzGWsOtPfUCNri1nzqZx9CEHB XV/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251210; x=1740856010; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UEr+HVsM6xtJa9sjUwL5+hzjYi8sZRQw9jjeFW5Hz0I=; b=Kp7W/SvG29NpcxuSHZjJLXlxyopb6UaJMU0nwM1gsf3WOF+KvuDVgck2dp1fiilr+g FhU0eg6Wr4yrKQj8hTbE6aIOkYaOVGdR3UaIlhGYBtuCf4J313gDgyzIRAwsUx2xCoBM ct+7aXYkrwhJSRJbjrHSNbmPsuCEa08kuymcbEE9J1cp+2Xz1M2y/RWFI4nSjL0irQnc 1rDh3TgVhOFe9lHFRhK0jrb8lkDvTGTTHyIt4l1kY4x7dSJQpAOMECjJCwEfZANF4hsQ z3PoQvwrDb92QtPe2U1UQkuW1BfL9eaQfnmrxEd3lcsUWRfmovYYAd3JoUciRPHUkeYS EhBw== X-Gm-Message-State: AOJu0YwKPywUANUK1/JZfnHlLmaHF7CQ6x6pp1hObgT+XuCd4q8Q3/KE 1dIgP/DnRNPjz0vqVzRdSZtCNsg3ipDxltqE16pp3Ovbzs0cJmR2+vn1 X-Gm-Gg: ASbGnctjFAmK/hLDXF9m7L8oxG+6LHZWrE+/r0VJ3IMkh/z5L0yzjdomPlX1y3RVUga m78RtcklvbPYIyZwosZ4MMRB/58PbfWu6ieigOycjAhMvXBBPf/gsYNt3OgkXVD46TX9JISurpn O8dpg2xCNWCQzVv/ixnBzdTDgbRpCOnwC4qVKQ7kC0xbLjnvBpYMagt5uaBmPoDJAtvBu89Raxy shIH1pYTRJVZJvOGZ4laSAwHJNNvU8+TwQbsXN4CGtL4Hj4MtQvmcvys35PhvvSN48i8ZBgzkTa BSZs3/g= X-Google-Smtp-Source: AGHT+IG0M9k0xj06pItA+FR6Si4ubLa+8PsF+seWp5HNzoXUonYRUtnC0WRiuLDwyL8a8Q1Lw+eMiQ== X-Received: by 2002:a05:620a:444c:b0:7c0:ae2e:630d with SMTP id af79cd13be357-7c0ceefe6c0mr962405485a.16.1740251210169; Sat, 22 Feb 2025 11:06:50 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:48 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 07/11] x86/softirq: Move softirq_pending to percpu hot section Date: Sat, 22 Feb 2025 14:06:19 -0500 Message-ID: <20250222190623.262689-8-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/hardirq.h | 3 ++- arch/x86/kernel/irq.c | 3 +++ 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index d51299af6145..6de46e2ae2b6 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { union { struct { void *hardirq_stack_ptr; - u16 softirq_pending; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 6ffa8b75f4cd..fa8ae99d62dd 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -66,7 +66,8 @@ extern u64 arch_irq_stat_cpu(unsigned int cpu); extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat =20 -#define local_softirq_pending_ref pcpu_hot.softirq_pending +DECLARE_PER_CPU_HOT(u16, softirq_pending); +#define local_softirq_pending_ref softirq_pending =20 #if IS_ENABLED(CONFIG_KVM_INTEL) /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 385e3a5fc304..1b51d5c05583 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -31,6 +31,9 @@ DEFINE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); EXPORT_PER_CPU_SYMBOL(irq_stat); =20 +DEFINE_PER_CPU_HOT(u16, softirq_pending); +EXPORT_PER_CPU_SYMBOL(softirq_pending); + atomic_t irq_err_count; =20 /* --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qv1-f51.google.com (mail-qv1-f51.google.com [209.85.219.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E7692144B4 for ; Sat, 22 Feb 2025 19:06:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251216; cv=none; b=bJyG+RA0pWvU4oy4C1Q3I4L4WGLsspnK1gc1t+EgzTBEkwG0ou8l2SrJJTpTTNgQG1isS1dYLoVjsE1dXLX9qrasLf8+WdBtu7A/QQlpPTUCc6Seel4NXSvioX3MB/aOKYgtqA97BkfBBCOMfg8+E9taWeKJigz3+U0PtORmdZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251216; c=relaxed/simple; bh=rqn9vf/rFqcaqJHERZsjgjCV7XrUGnFtJMkvM0Gtn2Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IYVtONsEMjHXIQCFsaQ0Jrf26Tubik63YEOglKwvkD66DxZbqIoITMi3g6XMel/jOhz4yTe2W6H+OnKdDIllyczGNRAt9fcrSg25f8OibnDAnZewJ64xuUtc4g66r5kSQMERZUYI5o6XDcSW9bM0/cxLGTvIFYQGBcaVc714g/8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Vwb+ABQm; arc=none smtp.client-ip=209.85.219.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Vwb+ABQm" Received: by mail-qv1-f51.google.com with SMTP id 6a1803df08f44-6dd420f82e2so45510436d6.1 for ; Sat, 22 Feb 2025 11:06:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251213; x=1740856013; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eM0qCn4hMa/3nh37POynWdLdE6cWMXbQzyt+go2OAM0=; b=Vwb+ABQmbvUO5mTWZ8+E+7v1yPVu9FXvoKfIiDe1AFmZezF+chEAXqW7hFvJ4cjPq1 yPoE2dPJlPJksVSUgxmwbDydiNdyOOw0zt1poKSsKrbOfkKz4pombq9XnnpS0N4aejaO HA04yrld2r5Y5w3YjVk+2wbNxIjgbsHqkWRWzlBY5DSgzg6jDKQl40KbtETac4lMqit9 474kSTOVnsfMdyFwwvkdpp/dVGIgHfePZy3UG+YyHCoKPpdS+nZisZHEwYes8+5oZSmp NXbN2VJwVYLcRdjN7B/zBrd3Fi5iKvbtg+B+YOg+Ims964bEt/QBbb3gcsswIC19KOT2 W7Cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251213; x=1740856013; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eM0qCn4hMa/3nh37POynWdLdE6cWMXbQzyt+go2OAM0=; b=UN3VGGqCWozrc3WfzFE+ue6H79igFujsgOCmIxGFmyRZz5dCK6EREXNaOeSVWbAahp u32B0hVg7FZAS98LA9oGfOqOF1KUYQtLEcCLFyxTBs619DuQ7aZQHOTEbM7/vlRwNTI8 dqoeLSKFiBLzRKCAv1kkzltHa0oZCK6WWB+ccxFKebZ4cJ8WAZpmk8zgb3VJ38iiHvOe LYr51U3jMVik0H9xhPCwGC4w0OsUkAvrXC28QgWmyGP52wYU+hxEGa+gqpsuuGrNdRlP 6MTZELKu744L5EHp1yQR/ORQ1pvJGITJ4eA6IhS+RsgL9eFOHwZfr53251rgZMvr20wL tJVg== X-Gm-Message-State: AOJu0YysZfCunQrZAledCO7RLkGqrH92KbxfX1zZoJBc5ST28xRMrD1/ fK9jCrSn39MyROpltQaW2zArMxMMIfTEh2n6viYw3BBBRyHSaRDKW+uF X-Gm-Gg: ASbGncsdk4vDStvhu+tx68UykCzj1jxhqL6lIC1REBnl9SceO4PwAJKp13TQvHkiorN Df/9DLYA9LP2aoDzOgJq5dKJwfwv2ilaBmHqq8M8/1pszbLyz/sZ+rNc7Tcs+9vERqbrKaNsVdK DNaNnnrCWjZW1A7Ywy39qC/guad02kDSNIJzZ+TfgLfLGa+XTqyWUvaxgLGXzgdW5cBBlEB6JsW HNY/qQi9QTARvdcEgKQ7we/Zztam1E6Ujq79q4fTPtxfv1BE4coecU0XJ0IM8sFtxazydoaIyja C3i6lQA= X-Google-Smtp-Source: AGHT+IHOSVZDtWbxUzpe4yZeHv9123lo75UA34aQk0Paqi7JHlohpEpMMUA00IZx1vaueRoksz/ZUA== X-Received: by 2002:a05:6214:252c:b0:6e6:6a82:497e with SMTP id 6a1803df08f44-6e6ae7f5844mr103147806d6.10.1740251212993; Sat, 22 Feb 2025 11:06:52 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:51 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 08/11] x86/irq: Move irq stacks to percpu hot section Date: Sat, 22 Feb 2025 14:06:20 -0500 Message-ID: <20250222190623.262689-9-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 10 ---------- arch/x86/include/asm/irq_stack.h | 12 ++++++------ arch/x86/include/asm/processor.h | 7 +++++++ arch/x86/kernel/dumpstack_32.c | 4 ++-- arch/x86/kernel/dumpstack_64.c | 2 +- arch/x86/kernel/irq.c | 5 +++++ arch/x86/kernel/irq_32.c | 12 +++++++----- arch/x86/kernel/irq_64.c | 6 +++--- arch/x86/kernel/process_64.c | 2 +- 9 files changed, 32 insertions(+), 28 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 6de46e2ae2b6..043888c258bd 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -13,17 +13,7 @@ struct task_struct; =20 struct pcpu_hot { - union { - struct { - void *hardirq_stack_ptr; -#ifdef CONFIG_X86_64 - bool hardirq_stack_inuse; -#else - void *softirq_stack_ptr; -#endif - }; u8 pad[64]; - }; }; static_assert(sizeof(struct pcpu_hot) =3D=3D 64); =20 diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_st= ack.h index 562a547c29a5..735c3a491f60 100644 --- a/arch/x86/include/asm/irq_stack.h +++ b/arch/x86/include/asm/irq_stack.h @@ -116,7 +116,7 @@ ASM_CALL_ARG2 =20 #define call_on_irqstack(func, asm_call, argconstr...) \ - call_on_stack(__this_cpu_read(pcpu_hot.hardirq_stack_ptr), \ + call_on_stack(__this_cpu_read(hardirq_stack_ptr), \ func, asm_call, argconstr) =20 /* Macros to assert type correctness for run_*_on_irqstack macros */ @@ -135,7 +135,7 @@ * User mode entry and interrupt on the irq stack do not \ * switch stacks. If from user mode the task stack is empty. \ */ \ - if (user_mode(regs) || __this_cpu_read(pcpu_hot.hardirq_stack_inuse)) { \ + if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \ irq_enter_rcu(); \ func(c_args); \ irq_exit_rcu(); \ @@ -146,9 +146,9 @@ * places. Invoke the stack switch macro with the call \ * sequence which matches the above direct invocation. \ */ \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(func, asm_call, constr); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } \ } =20 @@ -212,9 +212,9 @@ */ #define do_softirq_own_stack() \ { \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(__do_softirq, ASM_CALL_ARG0); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } =20 #endif diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index 1505cb1d09a8..9dde4ffed917 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -420,6 +420,13 @@ struct irq_stack { char stack[IRQ_STACK_SIZE]; } __aligned(IRQ_STACK_SIZE); =20 +DECLARE_PER_CPU_HOT(struct irq_stack *, hardirq_stack_ptr); +#ifdef CONFIG_X86_64 +DECLARE_PER_CPU_HOT(bool, hardirq_stack_inuse); +#else +DECLARE_PER_CPU_HOT(struct irq_stack *, softirq_stack_ptr); +#endif + DECLARE_PER_CPU_HOT(unsigned long, cpu_current_top_of_stack); /* const-qualified alias provided by the linker. */ DECLARE_PER_CPU_HOT(const unsigned long, const_cpu_current_top_of_stack); diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c index b4905d5173fd..722fd712e1cf 100644 --- a/arch/x86/kernel/dumpstack_32.c +++ b/arch/x86/kernel/dumpstack_32.c @@ -37,7 +37,7 @@ const char *stack_type_name(enum stack_type type) =20 static bool in_hardirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* @@ -62,7 +62,7 @@ static bool in_hardirq_stack(unsigned long *stack, struct= stack_info *info) =20 static bool in_softirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.softirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(softirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c index f05339fee778..6c5defd6569a 100644 --- a/arch/x86/kernel/dumpstack_64.c +++ b/arch/x86/kernel/dumpstack_64.c @@ -134,7 +134,7 @@ static __always_inline bool in_exception_stack(unsigned= long *stack, struct stac =20 static __always_inline bool in_irq_stack(unsigned long *stack, struct stac= k_info *info) { - unsigned long *end =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_st= ack_ptr); + unsigned long *end =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr); unsigned long *begin; =20 /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 1b51d5c05583..262e477b4651 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -34,6 +34,11 @@ EXPORT_PER_CPU_SYMBOL(irq_stat); DEFINE_PER_CPU_HOT(u16, softirq_pending); EXPORT_PER_CPU_SYMBOL(softirq_pending); =20 +DEFINE_PER_CPU_HOT(struct irq_stack *, hardirq_stack_ptr); +#ifdef CONFIG_X86_64 +DEFINE_PER_CPU_HOT(bool, hardirq_stack_inuse); +#endif + atomic_t irq_err_count; =20 /* diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c index dc1049c01f9b..dd7d9ba87bd0 100644 --- a/arch/x86/kernel/irq_32.c +++ b/arch/x86/kernel/irq_32.c @@ -52,6 +52,8 @@ static inline int check_stack_overflow(void) { return 0; } static inline void print_stack_overflow(void) { } #endif =20 +DEFINE_PER_CPU_HOT(struct irq_stack *, softirq_stack_ptr); + static void call_on_stack(void *func, void *stack) { asm volatile("xchgl %%ebx,%%esp \n" @@ -74,7 +76,7 @@ static inline int execute_on_irq_stack(int overflow, stru= ct irq_desc *desc) u32 *isp, *prev_esp, arg1; =20 curstk =3D (struct irq_stack *) current_stack(); - irqstk =3D __this_cpu_read(pcpu_hot.hardirq_stack_ptr); + irqstk =3D __this_cpu_read(hardirq_stack_ptr); =20 /* * this is where we switch to the IRQ stack. However, if we are @@ -112,7 +114,7 @@ int irq_init_percpu_irqstack(unsigned int cpu) int node =3D cpu_to_node(cpu); struct page *ph, *ps; =20 - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; =20 ph =3D alloc_pages_node(node, THREADINFO_GFP, THREAD_SIZE_ORDER); @@ -124,8 +126,8 @@ int irq_init_percpu_irqstack(unsigned int cpu) return -ENOMEM; } =20 - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D page_address(ph); - per_cpu(pcpu_hot.softirq_stack_ptr, cpu) =3D page_address(ps); + per_cpu(hardirq_stack_ptr, cpu) =3D page_address(ph); + per_cpu(softirq_stack_ptr, cpu) =3D page_address(ps); return 0; } =20 @@ -135,7 +137,7 @@ void do_softirq_own_stack(void) struct irq_stack *irqstk; u32 *isp, *prev_esp; =20 - irqstk =3D __this_cpu_read(pcpu_hot.softirq_stack_ptr); + irqstk =3D __this_cpu_read(softirq_stack_ptr); =20 /* build the stack frame on the softirq stack */ isp =3D (u32 *) ((char *)irqstk + sizeof(*irqstk)); diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index 56bdeecd8ee0..4834e317e568 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -50,7 +50,7 @@ static int map_irq_stack(unsigned int cpu) return -ENOMEM; =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #else @@ -63,14 +63,14 @@ static int map_irq_stack(unsigned int cpu) void *va =3D per_cpu_ptr(&irq_stack_backing_store, cpu); =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #endif =20 int irq_init_percpu_irqstack(unsigned int cpu) { - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; return map_irq_stack(cpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 1517314da34a..13893ec03d85 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -614,7 +614,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) int cpu =3D smp_processor_id(); =20 WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) && - this_cpu_read(pcpu_hot.hardirq_stack_inuse)); + this_cpu_read(hardirq_stack_inuse)); =20 if (!test_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD)) switch_fpu_prepare(prev_p, cpu); --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7AAAF2147FA for ; Sat, 22 Feb 2025 19:06:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251219; cv=none; b=pljY/wAbg/APzPycD45/gWSsXtSLE8y8JGR+bprJj2UQBH3+20ww+6ITqxzUsG7vZ4ucQQ0W8iooC2Gin3WQSGUuBEmeZ+sMoM/TOX41lrISFH88SXZ3J6rkb4Xq5Evkp6oD8SETCvyWwkbfuLCh7dhPSkd1+Vk0izAdKD1WSS4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251219; c=relaxed/simple; bh=xiFG5PYCkAJ25zl9pu9erSTUhptfDwvFMoWSnA5q02w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oZlKdI+pGQtaEjTMmeeNXs2iqg0vnpJlTmU+8akfT8Ffkw7Guaq16kFkcL1fzdctulhi2NZPgkO8apbm0iS7NSGMt/Q9JzU6QaDLPwXgQIZajBkA4vqinE8bcxKa5OazjnVr36Ta+yC4RWmdtJOyz+MYVVk1Gqa28yF9VNjANqw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GnVH+XDN; arc=none smtp.client-ip=209.85.219.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GnVH+XDN" Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6e44fda56e3so31283416d6.1 for ; Sat, 22 Feb 2025 11:06:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251215; x=1740856015; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=he+DNqPCx269JMfrjm2gD6jmCJvnxgXUTS3R67Dcs3A=; b=GnVH+XDNOL3IeEmpfM3d0UbPnJ75bTKzfpLi5aQ1CzOK41ewd0Fr0oGCIYVMVcJjew tlXkEhdemG7lUsJnQ8LthiCyFafLkfLvjE+ofZpfgj/nn/ON8HS5ODiLnF1KKfwzakcH 4rynlQBE1oGcltWJzYjtVppUz+T8/frlFXnP+lzDTpenv9l5ORgjrUF27g7wCD2iYN3M 0JLVbAUsPFcZCO7ndGtqYXPat4HR1tFZZbi4AEB71/0BR2taMkSZHCAOleenfErY6TaS q2SfhREH1Iwrh6RzhXcJojcvSRlUjRMiIQmb7OazKDoHn2dpuQK/k7GS1G0uNF1hlODZ 3LdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251215; x=1740856015; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=he+DNqPCx269JMfrjm2gD6jmCJvnxgXUTS3R67Dcs3A=; b=p+gF8vFLop6NronkOcGAwiXOm43by0O+HngJmgEHilPxrrqCDXMDtb+ANwbWk9U6wL ixjCcGHtfOlD2zBHcgLhw/FJWkfi88kVC/JhBs5PlaOLHUG/CJQb+C6oT1eHU6Nw4xBU Zc0e4hibyLZtgwGoY0YbJIZXNA/0wydFtMH2NxOuQF5Cz+LvVZfUqsKaBaHH/zSx/ah9 iNFBk0IWimp+08Q4bDd/9TJbjWBWBrbD2FVbVaiep+UtkXaSra1WUoJRmCZWcFEruQHq r3pNKFH36M4fgfViOKdlhDteTliLNwSxUA8C5vxPabwUX9SvdeQmkq1dokWAvLdsEU5O EJCQ== X-Gm-Message-State: AOJu0YwFscxvxPtLyasrSELy4yrckAX6rPkZERfmffjVgzc2e01lNMUZ mjWRFh184LiZ8kcebklxKJwGChsRPfmN20kezJkmoKg+htALHEq9NuM+ X-Gm-Gg: ASbGncsm1GsMJGmg9vO/6+/yyclM756o8qYNm6+PTZgKuIPNjT806i0zMzzrq09jwB7 x+b8AFWIPMkazxJxtfWt7L6hzMh7tpZ1s14XNSkKKlF84/HY4T43SevprWgDCtH/afqjizH0xt6 PtaeBy0yv2niyg6JAdiammLH4gCj0kMkV5DWHbxtq7+7NCmtQVgOlXEEDkVyG6qxrRPR/5mOh8q xJuqNlTYiqj/o2cYMpjkY1k23uUFByWHrQqRbza5rWy/uc8bGU5uPJddolHQs5VFvdxMO5q7zGS TG7UrW0= X-Google-Smtp-Source: AGHT+IE+Zw8vbcCy8txGNkOqHiMVJV1xsVCjUTwbDKQuKaS7rCvCFZb8Fgewse/hVoiE6PIxb7GCXA== X-Received: by 2002:a05:6214:cc5:b0:6e6:5a83:dcf5 with SMTP id 6a1803df08f44-6e6a264cb20mr194861376d6.21.1740251214908; Sat, 22 Feb 2025 11:06:54 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:53 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 09/11] x86/percpu: Remove pcpu_hot Date: Sat, 22 Feb 2025 14:06:21 -0500 Message-ID: <20250222190623.262689-10-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" All fields have been moved to the percpu hot section. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 11 ----------- arch/x86/kernel/cpu/common.c | 4 ---- arch/x86/kernel/vmlinux.lds.S | 1 - 3 files changed, 16 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 043888c258bd..b6d1adb5538f 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -12,17 +12,6 @@ =20 struct task_struct; =20 -struct pcpu_hot { - u8 pad[64]; -}; -static_assert(sizeof(struct pcpu_hot) =3D=3D 64); - -DECLARE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); - -/* const-qualified alias to pcpu_hot, aliased by linker. */ -DECLARE_PER_CPU_ALIGNED(const struct pcpu_hot __percpu_seg_override, - const_pcpu_hot); - DECLARE_PER_CPU_HOT(struct task_struct *, current_task); /* const-qualified alias provided by the linker. */ DECLARE_PER_CPU_HOT(struct task_struct * const, const_current_task); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index f4ec6bcb2a5e..ba78ee8fdb21 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2014,10 +2014,6 @@ static __init int setup_clearcpuid(char *arg) } __setup("clearcpuid=3D", setup_clearcpuid); =20 -DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); -EXPORT_PER_CPU_SYMBOL(pcpu_hot); -EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); - DEFINE_PER_CPU_HOT(struct task_struct *, current_task) =3D &init_task; EXPORT_PER_CPU_SYMBOL(current_task); EXPORT_PER_CPU_SYMBOL(const_current_task); diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 3c87bb620434..0cfdaa0e05a0 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -43,7 +43,6 @@ ENTRY(phys_startup_64) #endif =20 jiffies =3D jiffies_64; -const_pcpu_hot =3D pcpu_hot; const_current_task =3D current_task; const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53FBA2147E5 for ; Sat, 22 Feb 2025 19:06:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251219; cv=none; b=D2BkBEuLAURnzSVX+99sLJAAl2YMkUnjYoXZqXMjCFkTPtxxCm1ixCB3UJHpmj9DwRHsyOYIeT9j+lhuCffiEKsKWYlJQbEswnsL2Bz447Hcw2YU7TebkPcwnj5UBZsQBzxEAPyLP6Vqib4zYtp7+wuWVI4A5U1xa/nZbnStdQU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251219; c=relaxed/simple; bh=sCJb295kfpNZJraMgWG8u5wtIixvHjWMUi3T4z6Le9c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=itaVr0i83G0DrhU0Oo/BULlE0k+lT9uSfGOnD0M918x/cwJ5GLS/DLyiyoDTVGZocarJvUOjpDIHPGsYzgRI68sXuhle8ZPIDDwZYAIWeSmFOjwZAUCQ2vlAoRbQr/viJ1POZjmtiliU+5p50zDDpMLdg6IT8M5lMF9EPtnuwuQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bvPfAtr8; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bvPfAtr8" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-7c0bb7328fbso335680885a.2 for ; Sat, 22 Feb 2025 11:06:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251217; x=1740856017; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xEdxq9t5pZYrTokyQQH+c0hLwTUa97QlsNU/pVltUII=; b=bvPfAtr8jaAR+nvCpwSGRXoN6m9demujzHatjrEsykz+gx2QtX1xMvEr3WsuvtfXi4 Dsvow+Tv3PesVe8lTgCsHyx4qaIEjCwQlM4CbyDhuyP79F//9T4IHtkj4iYdMZAjHw/i LZdPyhPC5VL6JUBsTN4OMPPFOiES/tTzG7nFTiBn9UDOOwGLOoh6HVITve85vA+BVLvw +6M/ym3J1wF6vtKPc1vPsoADMPlkv3dV/c9OuWhJKPFUtOqMoXavBTi/a5t/v28hdKyi Ki9nAUDOq99e54AbJjHIA5YboH+rCNHR1gV6LykLO/uCci0Q7hUzKQ0/LEUAD8Ahl76Z LwSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251217; x=1740856017; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xEdxq9t5pZYrTokyQQH+c0hLwTUa97QlsNU/pVltUII=; b=GwY6JVfoW7jZk/2izZyXLJs6ReqRlSwtfCsvaKkzztOMJxgEXgoAdyIbSxat5+Zejc UR7ZH+ec4vdsnrhOH43jygeEUvAeXxVJbpjU6FOhPbatmtvD5fLfZa9HTQ6+JB6nvuTo OIIXmhtr/ffhHlSs+Udtacv9lbyZv27HX+QITRswtpvhCGk5AvW2mhIAMTdfi31rWezK 9UGee0ksSfnu4yyvgsTHxhjoYVoFhiwMt1NsgO27kicp/yDV1JkmAyB7Zu773NKAY4R4 Z1RERhSNhcZkETMdnZIEUyiIoZdhspybx8IKYE0vkv5WLJzI/sEKN5VmgX5pj61lWoUe xrYw== X-Gm-Message-State: AOJu0Yw0T+KrnQke1jqS5E5VFDKAdp74XRXhai/Bk0EO3Wm8unfS0mYC X2DdfOoFgi+ULiyteT3XYvQ/gX0ytqwN5zO2uwtAGZDA5QCmWkMoXmKB X-Gm-Gg: ASbGncs9lTYpOnoaorIVstUieq+C20xAM6QDeJC7Bo/G7ehJfbZFhhQkytQPNnaKv6Z TEhW3I1KAue+aP5tXbILEMayXqaf1GSKkLg11CQxB58JBz6AFQVM60cEekThwsDt31AYzkFfZLG MOCZn6M6GuAAJZHGx6C4buy+2H4u/5fY/jca32HjHypuIRgCm9S6l8Id/1OPGz0H37x9/SoZgnp cqm4IzdWfm7sch8Fqqww09wZUudKn8Cvk4RATEaa2a1aU6ruX1o4QJXKulimRHBzh79H2KOX0wi 6phhCDU= X-Google-Smtp-Source: AGHT+IFluJWdccE5UzTWh9K4kVeSSBzncztfk/7KSRsf76mtVD8rCM62hDMMa6Xbkur7pRLcp9SE4Q== X-Received: by 2002:a05:620a:1995:b0:7c0:a5bc:3ac0 with SMTP id af79cd13be357-7c0ceee98cemr993549285a.12.1740251216925; Sat, 22 Feb 2025 11:06:56 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:55 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 10/11] x86/stackprotector: Move __stack_chk_guard to percpu hot section Date: Sat, 22 Feb 2025 14:06:22 -0500 Message-ID: <20250222190623.262689-11-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/stackprotector.h | 2 +- arch/x86/kernel/cpu/common.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/s= tackprotector.h index d43fb589fcf6..f08304ac262b 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -20,7 +20,7 @@ =20 #include =20 -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +DECLARE_PER_CPU_HOT(unsigned long, __stack_chk_guard); =20 /* * Initialize the stackprotector canary value. diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index ba78ee8fdb21..eb7ac92e8565 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2089,7 +2089,7 @@ void syscall_init(void) #endif /* CONFIG_X86_64 */ =20 #ifdef CONFIG_STACKPROTECTOR -DEFINE_PER_CPU(unsigned long, __stack_chk_guard); +DEFINE_PER_CPU_HOT(unsigned long, __stack_chk_guard); #ifndef CONFIG_SMP EXPORT_PER_CPU_SYMBOL(__stack_chk_guard); #endif --=20 2.48.1 From nobody Sat Feb 7 16:06:09 2026 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B6D8215076 for ; Sat, 22 Feb 2025 19:07:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251222; cv=none; b=PP12En1pQoomIPEB2gnX3qX8Vihnc0v8FGMFVKNIfCCw5LjtG4MRxuqHs9zLiwFXGCsbZuqqSxPIQEwExGPYzK07z+Mej8vC1KH+bWt3nYqu3nZg2fuAwahsEEpl3P6uFtBYzXTQOpNb+pviVzl+KNmt0mY2pVG7l5/Rls5ZkJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740251222; c=relaxed/simple; bh=X8MAwkoRH0yXAmQO/36vaJ4i6hRN9Fj5GOq28mCWY5g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=j0K+zPuKFEq2snQHtlH4C9fwRJp+oO3VIcNGiJrmFIJl7ecfp45h+JRMG+68cAnCYYvDheoT4Nt/6dZDLGzv+7IHMjSBRcvBPQ02ypwj6+vi6ZZgKXyUND6R8M6QoX04yXVwl7UGh9hIkCE7WpvbtiRGcMNH5opQgOmdGlJfNjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Eh6AU6Nw; arc=none smtp.client-ip=209.85.222.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eh6AU6Nw" Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7c0892e4b19so381121885a.3 for ; Sat, 22 Feb 2025 11:07:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740251220; x=1740856020; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5o/duRurpHnQEbVsb8P75VsPCfHKRQsuOowLrKQw3bQ=; b=Eh6AU6NwEH/auz7h1KyxZYqxddDpPvi07gAmUZX2d9KAEJZG8HSIf55jLrloG1inEH /wIBmVqConzyaiHOQNdNe51lJdXnOSTz6FMqyLtLfniQ0eJO8lJo6RFYp1bfp9seSoEK 5EvjAi4jp1Ul+FnsNdW2OhD7Tm1HeVmIRN9zs7LY+cAa0q2FYUGF7cc+L2OqneE7KFSO tZlRxZO2l0oQqJQcnWOV38Kufy/dgXyzKAn8YG6vsgcOrqL7CLL7L6BoM/K9N0FuYu7W G09jXp6QCgt+oRZHoq0E/6h/wP02vpXnMWeEzrbrCnrF5O5+MCQFWM3Xb7wdQ58tnCbu JPeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740251220; x=1740856020; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5o/duRurpHnQEbVsb8P75VsPCfHKRQsuOowLrKQw3bQ=; b=MLn4g/uSkVowCIi27F/l+erIzyzyN4IhouCUIapWgN3gCiUJmi1SoDwmTr7b44PZ0z ZxhEBD7AcRRkz59JjQSc5p+HwfJJ2cB6qRqf9YQo5GE5SqINIapI1/XbsIVcbM6F/b6H x3XHf6sTltwXprOj4Cu4qRCrHz1/9zUyph5728ql2LAEdQGF0PFdLG3G1Y6LsTIBF/30 orycMVSs1hDRheZjoKzvHGIxjc/4rr10nJoPQWAHzYDFU/Qr7nyjrkbL5AfZPubhXq32 T+4Tg0oWcbaj0Rn2xuh4sxAR7UNVb7LJQWs0SIoSOWspBPDeEK8DZ0/NHJfugegUBsrU QAvw== X-Gm-Message-State: AOJu0YycWnywNGSblnw+sZu6BonZKDyfGgctPztlJ/3jvbqIhWGEoPrL AGIRn6CJ2rSfmP+4TXNjiLKVayHUU1Yhx63Bx6OETNa1YQ+s2WonUGXx X-Gm-Gg: ASbGncsq/MIFRMww6dAxYXOQUI9eL+K0F6hSbarf01Hib5oBpGb11LP0bhLbVhh4VVS LsdZ4CMvCfmB1cU2y1pzozyJPJNWOzsevTxNtfmFUBl/a4+svn+y0W2FzU9Y3RVsR4vGlKrhhgx 3QEfPEGQvGQNkbudSoo6JipEYpX7iqVsRLxLg9FTknqaVTQkH4hh2ZnI6iGjJH5VZltzgbNQZxl 8h11YG+1rLVzCvmAfRLTtbKaDDdQ3nazXB4/O8uBXAeb4wAsUag8OJl7DNRHJ85g2qY6Gvnlsrr ysS8B+w= X-Google-Smtp-Source: AGHT+IEN9Oft8opTgV3t5xrlwmgByvZQSwsi59N/jdqt8UilfHYnK/teAp5I297RXJw4NsWcEV1eKg== X-Received: by 2002:a05:620a:4891:b0:7c0:a1ca:93b6 with SMTP id af79cd13be357-7c0cef7b052mr1176808485a.50.1740251219662; Sat, 22 Feb 2025 11:06:59 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7c0b1ef73b8sm550185285a.65.2025.02.22.11.06.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Feb 2025 11:06:58 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Brian Gerst Subject: [RFC PATCH 11/11] x86/smp: Move this_cpu_off to percpu hot section Date: Sat, 22 Feb 2025 14:06:23 -0500 Message-ID: <20250222190623.262689-12-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250222190623.262689-1-brgerst@gmail.com> References: <20250222190623.262689-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/setup_percpu.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 370778c55091..e8034fe81ec1 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -595,7 +595,7 @@ do { \ #include =20 /* We can use this directly for local CPU (faster). */ -DECLARE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DECLARE_PER_CPU_HOT(unsigned long, this_cpu_off); =20 #endif /* !__ASSEMBLY__ */ =20 diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 0ea3443433c5..11a81e2a9675 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -26,7 +26,7 @@ DEFINE_PER_CPU_HOT(int, cpu_number); EXPORT_PER_CPU_SYMBOL(cpu_number); =20 -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DEFINE_PER_CPU_HOT(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init; --=20 2.48.1