From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A50C2063FD for ; Wed, 26 Feb 2025 18:05:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593147; cv=none; b=SdSoZOUEV7u1821xMx6YUuH9PXAI4PrG+lGZ+UjUFf1fml8x37tOCmNXeNTOC9qdluQ1ZG70DZaIUd7nxH+/JCqS+vLuvMJe3dI3j51VMM2kgxppVK6kBvb7LKXZCvX+bERDWcO27hY5H7FlLbkl1GwGS45o4+dyb2X39g6fbBg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593147; c=relaxed/simple; bh=UQtlk10D5PbqASuqCZBhI2xw+GvBFK3TWkMH16CPook=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aYm3ima0KipjOnZDS/vzYSbjwKx3E/rp2CmjfPraVs9NFEephhTj+pbkW1DMfgF8HkAXCNEI/4VGK5Cv+8mXnHzi1IllOwFyvdOW0HHHELYOCRBY+/PwcMHiw2GiW6/t/9ijirXuJQ+pySkIqMrHgEc6AnzBxzn4xilnChL2WPc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=X9d1ykPI; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X9d1ykPI" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3f40b8358dcso72626b6e.3 for ; Wed, 26 Feb 2025 10:05:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593144; x=1741197944; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mh5QCRSwNUxHFOsHYRm2HTz8oiL+OJA191UzYywUmTc=; b=X9d1ykPIkjBZytKCd5nleAjnmNeitHzWUh5hPt3A11bCtJJRnpBPxMfhx4pCrRiwRa 7628cG4BY0zALGkY8xLzchV2NDKJLhqV7FnuDOD5r0/jdtbst++yo++a89YhKCP/JX4w TODja8vDKc8LODy5uRV57GgGrgCYCSa8kHLzq5PFfvcth3To89STweUnvPMWEBWKNxtI zKwGM1Il7135tLEzlzx2NZUbYkwxw7OnE958IqZqrQ6D7ZItIuMQA5xOXHx+/YvGRtNP X/xvbamkis0ONOXPPk4hI+osNxq40GzSSYGisg2ho7/I5WdPPyNZA99uH59pMK4XH6w2 UhzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593144; x=1741197944; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mh5QCRSwNUxHFOsHYRm2HTz8oiL+OJA191UzYywUmTc=; b=JPU3CxtvytiA+xDcI+5L6J/BaeQeu8BvQIeqq3+Sjv2IOeRf8IVSX1+VMo/5ok+ClG Ov+qrP7jK1mGPNSmCEDLb5gCa29myZYa/3153Xqxivf/lPpkS41EH3w5mPzCg+r32in1 SvsqYp5kVCLYISgplsiHFJvuHwwfx5yf+pS7Oi+Mghk16Wju51yiYF06QYfWkzIDqF9W xEF4rpZK697l/qWzhU5UEWaunEfV2CCwOaDT9jyEVNIH8MkOjBV5UP/6yK9wsfxOxAjC 9Wke1FzWHbYnEjNolni23DylUGbhA9KiNDyObJEZWtvCFz/dry5ZAo75eAhh9xKCdONc 1Gmw== X-Gm-Message-State: AOJu0YzjZUvzQNMKnozRsq+i28ZpQpUBWaViP9Tg7WkSIkYldO+glZCJ ahrMKBvbtTSbnT+1MG7F2aw+jusQv233qnZvXYAr/BFRjuQ9etDF70Q5 X-Gm-Gg: ASbGnctAMa21tLu5WWrPcI72GmJPIdLF+Yxoj7N3rNkfRT7s4VHAWfwMRpTYB7l7lbG kIzG+wqSGnVhQ6NjuxTYEXKXcVOJeRzQw6auTDhv2TZnfffKOK42fuMKMYBGtbnRCCSOx5UGI27 hIKOCTkLilkSvyUmSYbc3nEiPMNrr4TSB1SjGiUmGYS43LtqZ/7Y7Q6hku3ONlvp91ePczJAyOo B6VUEduvCQ23/7bcc3eSW4VRlPeImQngCNkhiafHtu0J1JYF4Agrs9przLk0QPga6rLTNZyj9YJ vXludVY= X-Google-Smtp-Source: AGHT+IEoPG9jrqJr5ixSU63WTOykbmDB7N1zOonRAcE2uVu3tibPzhJ+cBpV3QfPT2ofHhZwl6TR+w== X-Received: by 2002:a05:6808:2f1b:b0:3f4:159e:1199 with SMTP id 5614622812f47-3f540ee6ae3mr4712349b6e.20.1740593144163; Wed, 26 Feb 2025 10:05:44 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:43 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 01/11] percpu: Introduce percpu hot section Date: Wed, 26 Feb 2025 13:05:20 -0500 Message-ID: <20250226180531.1242429-2-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a subsection to the percpu data for frequently accessed variables that should remain cached on each processor. These varables should not be accessed from other processors to avoid cacheline bouncing. This will replace the pcpu_hot struct on x86, and open up similar functionality to other architectures and the kernel core. Signed-off-by: Brian Gerst --- include/asm-generic/vmlinux.lds.h | 10 ++++++++++ include/linux/percpu-defs.h | 12 ++++++++++++ 2 files changed, 22 insertions(+) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinu= x.lds.h index 92fc06f7da74..92dd6065fd0a 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -385,6 +385,11 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPEL= LER_CLANG) . =3D ALIGN(PAGE_SIZE); \ __nosave_end =3D .; =20 +#define CACHE_HOT_DATA(align) \ + . =3D ALIGN(align); \ + *(SORT_BY_ALIGNMENT(.data..hot.*)) \ + . =3D ALIGN(align); + #define PAGE_ALIGNED_DATA(page_align) \ . =3D ALIGN(page_align); \ *(.data..page_aligned) \ @@ -1065,6 +1070,10 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROP= ELLER_CLANG) . =3D ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ . =3D ALIGN(cacheline); \ + __per_cpu_hot_start =3D .; \ + *(SORT_BY_ALIGNMENT(.data..percpu..hot.*)) \ + . =3D ALIGN(cacheline); \ + __per_cpu_hot_end =3D .; \ *(.data..percpu..read_mostly) \ . =3D ALIGN(cacheline); \ *(.data..percpu) \ @@ -1112,6 +1121,7 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPE= LLER_CLANG) INIT_TASK_DATA(inittask) \ NOSAVE_DATA \ PAGE_ALIGNED_DATA(pagealigned) \ + CACLE_HOT_DATA(cacheline) \ CACHELINE_ALIGNED_DATA(cacheline) \ READ_MOSTLY_DATA(cacheline) \ DATA_DATA \ diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h index 40d34e032d5b..eb3393f96e5a 100644 --- a/include/linux/percpu-defs.h +++ b/include/linux/percpu-defs.h @@ -112,6 +112,18 @@ #define DEFINE_PER_CPU(type, name) \ DEFINE_PER_CPU_SECTION(type, name, "") =20 +/* + * Declaration/definition used for per-CPU variables that are frequently + * accessed and should be in a single cacheline. + * + * For use only by architecture and core code. + */ +#define DECLARE_PER_CPU_CACHE_HOT(type, name) \ + DECLARE_PER_CPU_SECTION(type, name, "..hot.." #name) + +#define DEFINE_PER_CPU_CACHE_HOT(type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..hot.." #name) + /* * Declaration/definition used for per-CPU variables that must be cacheline * aligned under SMP conditions so that, whilst a particular instance of t= he --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E25C623373C for ; Wed, 26 Feb 2025 18:05:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593150; cv=none; b=NPxeGZutFazLu+Ud97KiZL7Pg0QZ2va4n3VbA/WFlWcKkAVTLdg9G9+cnm68ok9gFWvXM2q7d5wq6NdHJtT9WiPoiwJgbG+/bwxTSMeCTx63643bnPIVZhZYeetrFinwd43sJyqnrubi+4oBi4cytukRVxKsMgjRoMOE1upS4aQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593150; c=relaxed/simple; bh=724XS3imcQB+rHD2VPy5MwoQNp0+HKF7rZ7lgx3gaQg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=deG+1V7pDtZPh1n85qFWveKuNIpcXXsHXI+XGrE1J1aIPAZHs5f+iN4Ij5zHLhaZAOmygGyMXO8OGZwQMtVx9igGkzTY3eIREQiX9w+Rfuh5R1RTD7WFhNLFxHOwrZzvNsPFboLaHRd5rkOyVkdabclDhGvX4JbnH/YyFj6AmII= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=iQVhB6jA; arc=none smtp.client-ip=209.85.167.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iQVhB6jA" Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3f3ff03c89dso59949b6e.2 for ; Wed, 26 Feb 2025 10:05:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593147; x=1741197947; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SCMIoyMeCBuhFoe1IXPIQM39sgN1witXCp5jUX2wbmg=; b=iQVhB6jAt4wqOj9zVhCXb5Pfqsm5xVl+tnkfqjGJq+F0+WEdcSzOgWsfKuObDyM2C4 0pVqRsVhnE4W1pIsMn3wYT4TshmagYgfVNU0Xd0j8B6tzniNFXTB5uNghoNp9v5jxrrW UkVvXAOOU1hgCW7EjmdCPAkxAre0XZSlon38GctSCEBflfgjI4Cp58wVCX8HPB/3bXV4 8tlozk1Q13jg8pIKhuFE4c3bMurv88KMmraGl/82ujBW/PPc16Qww2FyWMlXsyvW+Z5J I52nRokKymSRbUbMrBwBa/PZ5I+HdodgBbYUAqkyFf3ugutmUoO01HRDvKo34CotH6mb 06sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593147; x=1741197947; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SCMIoyMeCBuhFoe1IXPIQM39sgN1witXCp5jUX2wbmg=; b=HWJx+Kq42WwRAaTKMzYKOqAjEmwByXFPUOH7W9IVqnD3hGSBKtq5rZ95++jSbHxliF 7We9gqXg37TpCTd943u00uGVaZuuTGh0jsUegdFA1yFwhwr9pXYuvKzUwyrDopTMaxXS 9aPo8mq8A7g+UXxTRzD4lbHZKnEDIILHOVzGEVpgNJQLnv8aQl/KdnoeS4+D2E1Ny3WV FQIhuKG9MWa7Pd7TKFAkjRhmOEZ66t/xPABL7jy9XpqIUC2jkaT0arbao5D2YP7g+OG4 KbrUUCsUZyA7qXQ8FtfR8TyonlzF4D9h8pBieivSgBPlGrUxBMWIlOblpScr4Ur/Irge dwig== X-Gm-Message-State: AOJu0YzVb+TZBg2wuSGLkA2ME4lsY7OtYRx6nKuqiq6ABX4h2C712VlH CzXxGohDxL//721sY72FoT2/DY4KRYCDSWNMI/Vy0U9QBegAIz3RdwFi X-Gm-Gg: ASbGncug88kRwew1CvOEJkN40JueJPm2cUBkNfGXteq8Gx5s7aSTc5XOq9VzH6/oASX X5p8aG4I8jTsIv3x19wfaAaoChOxJgxQv30isuff/G502bf00aXaXPW2gYTRiwNsLk1j8jcCmht SrweLoAMLZ6sdiYY1rjB+SPTZOMr595//WrgJ8QrrwWga+4y8RUQBpbXp1rTQPfHWwRbCP/wTqA n2uOAywrz8CR7IAVIwlW0TNnKMzcRfrtZ4nXH7cWQW2P2e/KAnx66Q9zqrJrBL6JE7uJAQu5gJ6 DNop8SE= X-Google-Smtp-Source: AGHT+IGjwmWsqHP3kkBCauZVFGgTI0KmOgXycZFrLwCX1DPoHJ88Elv0S1i08c3US7YpBT9HdOcK0Q== X-Received: by 2002:a05:6808:2e95:b0:3f4:1197:29bc with SMTP id 5614622812f47-3f540ec2d02mr5558570b6e.10.1740593147450; Wed, 26 Feb 2025 10:05:47 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:46 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 02/11] x86/percpu: Move pcpu_hot to percpu hot section Date: Wed, 26 Feb 2025 13:05:21 -0500 Message-ID: <20250226180531.1242429-3-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 28 +++++++++++----------------- arch/x86/kernel/cpu/common.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 3 +++ 3 files changed, 15 insertions(+), 18 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index bf5953883ec3..60bc66edca83 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -13,32 +13,26 @@ struct task_struct; =20 struct pcpu_hot { - union { - struct { - struct task_struct *current_task; - int preempt_count; - int cpu_number; + struct task_struct *current_task; + int preempt_count; + int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - u64 call_depth; + u64 call_depth; #endif - unsigned long top_of_stack; - void *hardirq_stack_ptr; - u16 softirq_pending; + unsigned long top_of_stack; + void *hardirq_stack_ptr; + u16 softirq_pending; #ifdef CONFIG_X86_64 - bool hardirq_stack_inuse; + bool hardirq_stack_inuse; #else - void *softirq_stack_ptr; + void *softirq_stack_ptr; #endif - }; - u8 pad[64]; - }; }; -static_assert(sizeof(struct pcpu_hot) =3D=3D 64); =20 -DECLARE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot); +DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); =20 /* const-qualified alias to pcpu_hot, aliased by linker. */ -DECLARE_PER_CPU_ALIGNED(const struct pcpu_hot __percpu_seg_override, +DECLARE_PER_CPU_CACHE_HOT(const struct pcpu_hot __percpu_seg_override, const_pcpu_hot); =20 static __always_inline struct task_struct *get_current(void) diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 8b49b1338f76..9b8bf43019e8 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2014,7 +2014,7 @@ static __init int setup_clearcpuid(char *arg) } __setup("clearcpuid=3D", setup_clearcpuid); =20 -DEFINE_PER_CPU_ALIGNED(struct pcpu_hot, pcpu_hot) =3D { +DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, .preempt_count =3D INIT_PREEMPT_COUNT, .top_of_stack =3D TOP_OF_INIT_STACK, diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 1769a7126224..7586a9be8c59 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -187,6 +187,8 @@ SECTIONS =20 PAGE_ALIGNED_DATA(PAGE_SIZE) =20 + CACHE_HOT_DATA(L1_CACHE_BYTES) + CACHELINE_ALIGNED_DATA(L1_CACHE_BYTES) =20 DATA_DATA @@ -328,6 +330,7 @@ SECTIONS } =20 PERCPU_SECTION(INTERNODE_CACHE_BYTES) + ASSERT(__per_cpu_hot_end - __per_cpu_hot_start <=3D 64, "percpu cache hot= section too large") =20 RUNTIME_CONST_VARIABLES RUNTIME_CONST(ptr, USER_PTR_MAX) --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5ABEA23535E for ; Wed, 26 Feb 2025 18:05:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593152; cv=none; b=B4lE42Jqo7c3e5ALxOltayjAruGiAx5QWg78QPPkB5lGlAAzWKrKTZ8MgX7Tuc/SykX+nOmvGLnjOS7uQzYzd0BqCDC7uZ+olEUav16DoScmyJNBbcEpub/OOmW+p3uSFPtWA4bVa7LWOix7/wgv1ZbOj7pl+1UFMHe3J92dh8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593152; c=relaxed/simple; bh=EqJavPMpXmCpA8qSqAJ44MdPREeuZfT+T6RTPcdaOXM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o8QTOiwU5907U5kKqEXFazLvXx2uFwJVkZHwu1w4dg5v1eu9VSn0sY0eD9Cbwr/xIFDBrg96axYQQdNPGioNzHW5Ha8VpQGMaRR+OFvwdcleBaMJMKWTZwo2GJcW34OgfHkqn7o8z3UEFA8+Muqxmm0jdmj7cI17dp+uLF+A5ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nHE8TKrs; arc=none smtp.client-ip=209.85.167.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nHE8TKrs" Received: by mail-oi1-f173.google.com with SMTP id 5614622812f47-3f40ad1574fso109091b6e.0 for ; Wed, 26 Feb 2025 10:05:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593150; x=1741197950; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=436B8vpLTNU8Oi703zHdG5tjEBpnGGK8nyxnE2IzJMU=; b=nHE8TKrs32KXhOo6wz1dGqH1APvdWg16k1JO/PIDpcNR4hCBszxXLx/WV2SUkE0VW+ 2qql2DMhof5CHkV5Ruw4/VDhCkHzgFSKlL8BLJT4vmPYiMtpsrKZadPgLC3LoOeAkShx ZrWgOLhAJwYzSPsESsz8NsP1T3Du8dkIXK70EtrnGkIWzUGTilxUU8PGqnD3r0eEmtEt /CqXGQ8q4dgGYeeDUm8i7j1wJuqevgw/v1O2iah4DibIiAE/p9wgxnWsMy+/Uoplu2WV EO/cO++NuQzgFvKl1uIFYsUN5bqo1u7UsuzLAjikhUer68UaFIXoGy83GuIn8XMSyzc3 bBHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593150; x=1741197950; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=436B8vpLTNU8Oi703zHdG5tjEBpnGGK8nyxnE2IzJMU=; b=WAu1+iOR9UPH/od2IxV+yvpe2NoTtQX9Ly+kSlDworUT0mrVjAEFypqbXx/BhyR4tY TbxN/GTc2zqfp00CxUxqU9mZgzxKPgyovk2mg/bNECrMvXLC9pXkrF1Ft851DyONV2Nr VZs8Hv7c0BqqpExfBeiLKUx8ojQPU3AoFnEwrY42VIbiejMtUv3ckhpBiIrsB54D8jqb nA62EMZorQERv4j1o7zybkvPFIdNtezwqtVbJwhpr0vTSWQ0gcapTC/PcTstE6fvtPro sBXrZF4qsP4j39LvWJZxgxJFHYg7dVeynvrLZPjjzS3mc7boxYDFu3KWB9FBofGzEUDZ hxMA== X-Gm-Message-State: AOJu0YwrUbpap1yRwTZ2j36Ibqd92ZNwLZhHNb+DIjrvUS5dqauqd+q7 K0DeLN/4jhOEsP6yO6IDBgiwx7pPyiTGm888JJGnX15ZEgiMkUMQkblB X-Gm-Gg: ASbGnctxpp/9KXx2jwKgCElQ0DoB826f25reZudNkkp67pzm2RqrJHflNt+nBqbU8jJ /d5VQMJFw2gs1sRpGLBlYXlWmmlDJXEl0bAwN8Na12CjHwMWIpfndn0Z6mAuFchBvUWJKvivUdA vEboJ7TyrIqTs4GAuheSaa9FuiDt2G8bJvxzcha3mSHcgG4jacj/Ah5Wj7vjyD9hqjZJC+Joa0b Ct7fq94W2wJhdecZwHygVlEZY5C6rFayNXT7c04c4yGTc37uBjCSMqn87tnSdXCt54Sq28YBkiZ sL0GGdk= X-Google-Smtp-Source: AGHT+IHBoZzpk5ImyC5zzaHAekmEnhUZpbt3EjDP8LKX859d6FgFlKgg2K8qSx307vOhxAe3FHhv7A== X-Received: by 2002:a05:6808:3209:b0:3f4:400:e5f3 with SMTP id 5614622812f47-3f547e3981fmr2319169b6e.24.1740593149857; Wed, 26 Feb 2025 10:05:49 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:48 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 03/11] x86/preempt: Move preempt count to percpu hot section Date: Wed, 26 Feb 2025 13:05:22 -0500 Message-ID: <20250226180531.1242429-4-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/preempt.h | 25 +++++++++++++------------ arch/x86/kernel/cpu/common.c | 4 +++- include/linux/preempt.h | 1 + 4 files changed, 17 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 60bc66edca83..46a736d6f2ec 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - int preempt_count; int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; diff --git a/arch/x86/include/asm/preempt.h b/arch/x86/include/asm/preempt.h index 919909d8cb77..578441db09f0 100644 --- a/arch/x86/include/asm/preempt.h +++ b/arch/x86/include/asm/preempt.h @@ -4,10 +4,11 @@ =20 #include #include -#include =20 #include =20 +DECLARE_PER_CPU_CACHE_HOT(int, __preempt_count); + /* We use the MSB mostly because its available */ #define PREEMPT_NEED_RESCHED 0x80000000 =20 @@ -23,18 +24,18 @@ */ static __always_inline int preempt_count(void) { - return raw_cpu_read_4(pcpu_hot.preempt_count) & ~PREEMPT_NEED_RESCHED; + return raw_cpu_read_4(__preempt_count) & ~PREEMPT_NEED_RESCHED; } =20 static __always_inline void preempt_count_set(int pc) { int old, new; =20 - old =3D raw_cpu_read_4(pcpu_hot.preempt_count); + old =3D raw_cpu_read_4(__preempt_count); do { new =3D (old & PREEMPT_NEED_RESCHED) | (pc & ~PREEMPT_NEED_RESCHED); - } while (!raw_cpu_try_cmpxchg_4(pcpu_hot.preempt_count, &old, new)); + } while (!raw_cpu_try_cmpxchg_4(__preempt_count, &old, new)); } =20 /* @@ -43,7 +44,7 @@ static __always_inline void preempt_count_set(int pc) #define init_task_preempt_count(p) do { } while (0) =20 #define init_idle_preempt_count(p, cpu) do { \ - per_cpu(pcpu_hot.preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ + per_cpu(__preempt_count, (cpu)) =3D PREEMPT_DISABLED; \ } while (0) =20 /* @@ -57,17 +58,17 @@ static __always_inline void preempt_count_set(int pc) =20 static __always_inline void set_preempt_need_resched(void) { - raw_cpu_and_4(pcpu_hot.preempt_count, ~PREEMPT_NEED_RESCHED); + raw_cpu_and_4(__preempt_count, ~PREEMPT_NEED_RESCHED); } =20 static __always_inline void clear_preempt_need_resched(void) { - raw_cpu_or_4(pcpu_hot.preempt_count, PREEMPT_NEED_RESCHED); + raw_cpu_or_4(__preempt_count, PREEMPT_NEED_RESCHED); } =20 static __always_inline bool test_preempt_need_resched(void) { - return !(raw_cpu_read_4(pcpu_hot.preempt_count) & PREEMPT_NEED_RESCHED); + return !(raw_cpu_read_4(__preempt_count) & PREEMPT_NEED_RESCHED); } =20 /* @@ -76,12 +77,12 @@ static __always_inline bool test_preempt_need_resched(v= oid) =20 static __always_inline void __preempt_count_add(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, val); + raw_cpu_add_4(__preempt_count, val); } =20 static __always_inline void __preempt_count_sub(int val) { - raw_cpu_add_4(pcpu_hot.preempt_count, -val); + raw_cpu_add_4(__preempt_count, -val); } =20 /* @@ -91,7 +92,7 @@ static __always_inline void __preempt_count_sub(int val) */ static __always_inline bool __preempt_count_dec_and_test(void) { - return GEN_UNARY_RMWcc("decl", __my_cpu_var(pcpu_hot.preempt_count), e, + return GEN_UNARY_RMWcc("decl", __my_cpu_var(__preempt_count), e, __percpu_arg([var])); } =20 @@ -100,7 +101,7 @@ static __always_inline bool __preempt_count_dec_and_tes= t(void) */ static __always_inline bool should_resched(int preempt_offset) { - return unlikely(raw_cpu_read_4(pcpu_hot.preempt_count) =3D=3D preempt_off= set); + return unlikely(raw_cpu_read_4(__preempt_count) =3D=3D preempt_offset); } =20 #ifdef CONFIG_PREEMPTION diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 9b8bf43019e8..1470f687f8d6 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2016,12 +2016,14 @@ __setup("clearcpuid=3D", setup_clearcpuid); =20 DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .preempt_count =3D INIT_PREEMPT_COUNT, .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); =20 +DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; +EXPORT_PER_CPU_SYMBOL(__preempt_count); + #ifdef CONFIG_X86_64 static void wrmsrl_cstar(unsigned long val) { diff --git a/include/linux/preempt.h b/include/linux/preempt.h index ca86235ac15c..4c1af9b7e28b 100644 --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -319,6 +319,7 @@ do { \ #ifdef CONFIG_PREEMPT_NOTIFIERS =20 struct preempt_notifier; +struct task_struct; =20 /** * preempt_ops - notifiers called when a task is preempted and rescheduled --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AABC2356D1 for ; Wed, 26 Feb 2025 18:05:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593156; cv=none; b=BtCU1BXF/1xwIFQYawpAYmd6kIfIif2Y8Gp8xCrK3ZYBS9JIy7MM6tpyAyKPNwih5L6iyaaLIeWHEp+KS9ifP4caFTDz41fPfz4gKILId+tEboSM0IsivY1EfEZ4VNMoVetnd45Gobxkf8C3uw97pjvlMq2mBqsPf/zuItO2JxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593156; c=relaxed/simple; bh=+R4AE2vH5/O6elwe4lTeR/IxdlRNPhicCE/pDetofCk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WOSXW1q5u5/NZMgTuu3n5VDExQ7M6EjWkQfoYSMcczSNndTnpCq/JAYfi951S1WXq7p9aEnL8Yu3g+Dx8XR4B22kU3DkQCgHjPaQpJGW/5Dbg7pyvuBN42TYmu838JEYmA0rC6F3UGson+GEKebXfpvxjGgJnzMsUO/OCAE+lVY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BlIPOyFh; arc=none smtp.client-ip=209.85.161.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BlIPOyFh" Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5fcd7d306f3so42777eaf.3 for ; Wed, 26 Feb 2025 10:05:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593153; x=1741197953; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ifmVptM9mxgQ/UQ1wElHFd5KU5UJQ5Cl2dkvmNdyxqo=; b=BlIPOyFhxFs+TQl73Ph9VnfbzfyF4LvZ6LZGWBxxe325eQsRQcBDWCR/ucpMbe+Xec Izcv68j6EQK5/Tp9xbIWDSScrhJHMwp49uTsS/fJtWcL3RpIXiwZcT2MqR7t6iK9X3eu DX0p8eicUnxrlGJvxXTBIkdBbGNj6aJQry6hRsqcziAL9UHVWcp7RbOusXoIkz993CSm lNyqynKANrsuZyoYEGqwcBb7a3zPMouAXFoXTPJ6nAz05v6sd+SVnUD3Jhd16+jPZD8t Yi1mqA+RQ65FJu9RvrSWdqKq0QdUEf5+Oh91FPBBq6Ozrqt+QRHANlgwetYMStvE1vv1 eDaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593153; x=1741197953; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ifmVptM9mxgQ/UQ1wElHFd5KU5UJQ5Cl2dkvmNdyxqo=; b=cQPfAZUtc+hGw+G9xT04cQWJNm0ZxM/G+PmplSmWWMzY38O/r4Jrh6P7QAk72/KMjh m/rp3TNtrWQfyF5Bv4bK/QZUxfIsJ95PEotOJipN4CUhiNm/liNRK11c17rCrEjIpUbF 1SYUi34NXd7wCk/l3blHGwzvZLSzgiygPm0nAV2rGReqiaXV9s5d6cKqGQDyfnSRtz71 NQ1xQ6KyqidokX0Xl8rCIkrUStjtwrU39ZcEFbJHok4yZhECsmx/EIwut7zyih0zL2wy H7vvqbXts8GA/798lnK/NL8WJal0qVSXNBqwaS/dceMg/++X1g9Ts5EKRlgwrx1DG3tw CHSw== X-Gm-Message-State: AOJu0YzXSXDTyVhk8+mr6SfpAu/OmPI9VvHvr7sahmSsBOpSsOQlQIOu k4sA8ZZV+Id4CkzEY0QdHwtphlwrakVrlxH9b/jSe1TPbmPX70jI/DLL X-Gm-Gg: ASbGnctytFx/vEmj0pbjfpK8tb7hssrI1ZQL6toL2ibwR74Bum1pwRQhihCY5PMGAjr D6cNvgqIqRElaVvuUO5k3EAf2AJCr4mEJbh94rv7nKBFjHeZ9k8ibnM1Io2fzmfhIXZ0+kd0/He sZsqlUUwmITS/vfQpddf23v2iDGeav3hbmECO9aTyfMXHr5gqfoISVBKfrHyf3brKsMLNMwgpc5 ubLdCfjOuQTTZMnzRU/jPviw/FVpboEnHWPcClLwuxqaus1CDDX6+AMPWYdwn5/jUe/S+TawDRb bQMZpQQ= X-Google-Smtp-Source: AGHT+IF/ZgHy4vtLZAvsawnyeMz7UU5Be3jh32wQjYh6McKl2h1yO+IqNJydP8JJwsvPil3Dzyt5kw== X-Received: by 2002:a05:6820:997:b0:5fe:87b7:6ecf with SMTP id 006d021491bc7-5fe9d4a9f2dmr2377092eaf.0.1740593153145; Wed, 26 Feb 2025 10:05:53 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:51 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 04/11] x86/smp: Move cpu number to percpu hot section Date: Wed, 26 Feb 2025 13:05:23 -0500 Message-ID: <20250226180531.1242429-5-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/smp.h | 7 ++++--- arch/x86/kernel/setup_percpu.c | 5 ++++- kernel/bpf/verifier.c | 4 ++-- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 46a736d6f2ec..f988462d8b69 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - int cpu_number; #ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING u64 call_depth; #endif diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h index ca073f40698f..d1db2c131b1d 100644 --- a/arch/x86/include/asm/smp.h +++ b/arch/x86/include/asm/smp.h @@ -5,9 +5,10 @@ #include =20 #include -#include #include =20 +DECLARE_PER_CPU_CACHE_HOT(int, cpu_number); + DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map); DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_map); DECLARE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_die_map); @@ -133,8 +134,8 @@ __visible void smp_call_function_single_interrupt(struc= t pt_regs *r); * This function is needed by all SMP systems. It must _always_ be valid * from the initial startup. */ -#define raw_smp_processor_id() this_cpu_read(pcpu_hot.cpu_number) -#define __smp_processor_id() __this_cpu_read(pcpu_hot.cpu_number) +#define raw_smp_processor_id() this_cpu_read(cpu_number) +#define __smp_processor_id() __this_cpu_read(cpu_number) =20 #ifdef CONFIG_X86_32 extern int safe_smp_processor_id(void); diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 1e7be9409aa2..175afc3ffb12 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -23,6 +23,9 @@ #include #include =20 +DEFINE_PER_CPU_CACHE_HOT(int, cpu_number); +EXPORT_PER_CPU_SYMBOL(cpu_number); + DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 @@ -161,7 +164,7 @@ void __init setup_per_cpu_areas(void) for_each_possible_cpu(cpu) { per_cpu_offset(cpu) =3D delta + pcpu_unit_offsets[cpu]; per_cpu(this_cpu_off, cpu) =3D per_cpu_offset(cpu); - per_cpu(pcpu_hot.cpu_number, cpu) =3D cpu; + per_cpu(cpu_number, cpu) =3D cpu; setup_percpu_segment(cpu); /* * Copy data used in early init routines from the diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9971c03adfd5..604134d33282 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -21687,12 +21687,12 @@ static int do_misc_fixups(struct bpf_verifier_env= *env) if (insn->imm =3D=3D BPF_FUNC_get_smp_processor_id && verifier_inlines_helper_call(env, insn->imm)) { /* BPF_FUNC_get_smp_processor_id inlining is an - * optimization, so if pcpu_hot.cpu_number is ever + * optimization, so if cpu_number is ever * changed in some incompatible and hard to support * way, it's fine to back out this inlining logic */ #ifdef CONFIG_SMP - insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&pcpu_hot.= cpu_number); + insn_buf[0] =3D BPF_MOV32_IMM(BPF_REG_0, (u32)(unsigned long)&cpu_numbe= r); insn_buf[1] =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); insn_buf[2] =3D BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_0, 0); cnt =3D 3; --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E51DD235BE8 for ; Wed, 26 Feb 2025 18:05:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593159; cv=none; b=qW8bCMrb3vHTaalunYmT6BurPWaQ45BMgcm5/+1fGiaFBU/G6GEqlEQkHVm5Jzik7EAhqcRNmGxbL7Xw6T2pDeUcDPxMVF1Xnytm3u6g5zmsvQixVDEqQ1L28EyA99TM0l12OAhpMehX5IDQKRfgOesEzfIolpJrCuOEiHHidsE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593159; c=relaxed/simple; bh=y7qU9z7lgGcLUfd27np6xyc5Fks+vH2TibzUjRjdRHk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XTmt3jrlQxgP9wwHG0oAl0OOBIdm64tsWwcYS7ztuujegxpsz+sP1d9+tFixfQ8OtPq+uplYb5Ip7/FisWye+GoLSyU5e/LdGcfFGQekGGXna4Ar67OWqhMBCLjTFUXT/RHWPcTASE0FUGgF4xNmC8MFy21WpMGgtprWdCpZJrU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=GNnQu1MO; arc=none smtp.client-ip=209.85.161.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GNnQu1MO" Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-5fcd665af4eso23579eaf.2 for ; Wed, 26 Feb 2025 10:05:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593156; x=1741197956; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aCuVEC58STAtilTMaj0CWifdxvlJ14gwFXexzm17na8=; b=GNnQu1MOm+icmQ/V469Cesj76KN/aSCIENIgGaEgyVgmC8oMEVM+VMc4RAE8qwYvOS b0Jh/jP+Wuj3M/USzSSyg8/9HBB2D/iSSooSR6gh2S6m5Kgx7vpNBlt+21kwpBv3Ml6d taXXJRE2JadI4Gn5VN308hzVEJhFRaeMZu5FnQFkS4ComUEMxcaXbO39R5Rl2SlFHJ2M jTFaIb6kNQXG2BGybJWlYhu88YzxNMy4OW/jFvNkQgb994+UEE6476/kErIDMCJ9ue10 X0n3DY+uBhChihRRu9EEgS0pnQoNUzbhPfK07xBvRTpMKaAN2vL5AHHm1cFm7x2WEq1K uwgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593156; x=1741197956; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aCuVEC58STAtilTMaj0CWifdxvlJ14gwFXexzm17na8=; b=oUcJHh6Nky3T1r8IN35Q7q+vPcf145SrOmS1eeIdnkR72xAh/0n9+IfWfc6jEiRaCG mDbeFFZUUmwHSu9F8QuHBHp1BeubjsaoBF2W8zakbQy1/4oVR+YdORIEoh/3vLsaF4a+ jbf9+omh5+Ar8SixYNmd/cYboXWi07pQOAkRzWeGOBQVMPgXDYGwptyKDlDjfALqsaoz ZdudDpAbtV1ciWuUX57bLdDiYkyp4/9s5XeSULnunrt8HB4Grde2/UM9sJpUgtOucXLr s+x8VZ1IBAwQdNJu3TppLfYRhSaOyoCWq4FAvLhqet9W5khAnil43qMHKlJ0zK73zxDn S6fg== X-Gm-Message-State: AOJu0Ywx8T4AsP8tUHj1nMWxXuPy49TJCwX+B4g05Y1aRW8CAeqz5iBG LAacUi87BCm3+dWcOURRfa+SiXtfHzSjoiKuvg31PIHPueSPbi2a3uAP X-Gm-Gg: ASbGncufpkUgFQNcc3CXbpupoqjJOc2JswLTEtxQCpWlzTuHeN9XmQ2lpFkBOzqIcdE co4VClYgBhcgpShfTJujyoWlBUKiz0RpoNGC8y8fueVX3sk7W6HY9rrItVbyAS3y3z9Pq/XajMA tUs+lQvDkU08nAyYn4Kdqzx12ELQcoMegNQOYDw1rjJoG8FwtB5Bj37J+4E3jIirPS6zrfRJpjx 9JoFZaBbFhFY9V1wgGlRAW0Nj5sbsNQfJSFN1LZdqWzoAtVWMf5qeZ28IDkbzQIpiiEqHXdMNia 3u3Nvgs= X-Google-Smtp-Source: AGHT+IH6mblwDIuISIBCJFBGnvcZ8OHaO+whcezMj+njDJfvbpYtOVoYKoE3kTgPGbUg495gGQPiXA== X-Received: by 2002:a05:6820:1c93:b0:5fd:d5b:f46c with SMTP id 006d021491bc7-5fd1a456ca4mr13897942eaf.0.1740593156508; Wed, 26 Feb 2025 10:05:56 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:55 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 05/11] x86/retbleed: Move call depth to percpu hot section Date: Wed, 26 Feb 2025 13:05:24 -0500 Message-ID: <20250226180531.1242429-6-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 3 --- arch/x86/include/asm/nospec-branch.h | 11 ++++++----- arch/x86/kernel/asm-offsets.c | 3 --- arch/x86/kernel/cpu/common.c | 8 ++++++++ arch/x86/lib/retpoline.S | 2 +- 5 files changed, 15 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index f988462d8b69..8ba2c0f8bcaf 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,9 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - u64 call_depth; -#endif unsigned long top_of_stack; void *hardirq_stack_ptr; u16 softirq_pending; diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/no= spec-branch.h index 7e8bf78c03d5..a5602055bfc9 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -12,7 +12,6 @@ #include #include #include -#include =20 /* * Call depth tracking for Intel SKL CPUs to address the RSB underflow @@ -78,21 +77,21 @@ #include =20 #define CREDIT_CALL_DEPTH \ - movq $-1, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq $-1, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH \ xor %eax, %eax; \ bts $63, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); + movq %rax, PER_CPU_VAR(__x86_call_depth); =20 #define RESET_CALL_DEPTH_FROM_CALL \ movb $0xfc, %al; \ shl $56, %rax; \ - movq %rax, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + movq %rax, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #define INCREMENT_CALL_DEPTH \ - sarq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth); \ + sarq $5, PER_CPU_VAR(__x86_call_depth); \ CALL_THUNKS_DEBUG_INC_CALLS =20 #else @@ -388,6 +387,8 @@ extern void call_depth_return_thunk(void); __stringify(INCREMENT_CALL_DEPTH), \ X86_FEATURE_CALL_DEPTH) =20 +DECLARE_PER_CPU_CACHE_HOT(u64, __x86_call_depth); + #ifdef CONFIG_CALL_THUNKS_DEBUG DECLARE_PER_CPU(u64, __x86_call_count); DECLARE_PER_CPU(u64, __x86_ret_count); diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index a98020bf31bb..6fae88f8ae1e 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -109,9 +109,6 @@ static void __used common(void) OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); -#ifdef CONFIG_MITIGATION_CALL_DEPTH_TRACKING - OFFSET(X86_call_depth, pcpu_hot, call_depth); -#endif #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 1470f687f8d6..01f33fb86f05 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2025,6 +2025,14 @@ DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D I= NIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 #ifdef CONFIG_X86_64 +/* + * Note: Do not make this dependant on CONFIG_MITIGATION_CALL_DEPTH_TRACKI= NG + * so that this space is reserved in the hot cache section even when the + * mitigation is disabled. + */ +DEFINE_PER_CPU_CACHE_HOT(u64, __x86_call_depth); +EXPORT_PER_CPU_SYMBOL(__x86_call_depth); + static void wrmsrl_cstar(unsigned long val) { /* diff --git a/arch/x86/lib/retpoline.S b/arch/x86/lib/retpoline.S index 391059b2c6fb..04502e843de0 100644 --- a/arch/x86/lib/retpoline.S +++ b/arch/x86/lib/retpoline.S @@ -342,7 +342,7 @@ SYM_FUNC_START(call_depth_return_thunk) * case. */ CALL_THUNKS_DEBUG_INC_RETS - shlq $5, PER_CPU_VAR(pcpu_hot + X86_call_depth) + shlq $5, PER_CPU_VAR(__x86_call_depth) jz 1f ANNOTATE_UNRET_SAFE ret --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f46.google.com (mail-oo1-f46.google.com [209.85.161.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 10D39236A66 for ; Wed, 26 Feb 2025 18:06:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593162; cv=none; b=V7tuIndqmE/xSg7l561702V5aDvbtAdTUT8hKFzQKMwxLlWrI4TDsSgZe+9Rs4AcVC6lQa6PDuGdkwzchZ33jZJEebmgY+ZzE7DwLv/f6yhbQVM+Z6MZ3q1HM/WT3I7CgusFw6O6mN1Ki7pvm8XMvX0gbAdHCnf/93j0y7kqoG4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593162; c=relaxed/simple; bh=ZOVzoaSmhcjrh/GBlAxtX+PZyTOVsF7qh2/MNwXFnDs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XXQes2UdW4Regj6fiWe0h2Z6r+1S6J+9fIFAiLsOQ+7txsT/3RlemPytNAG400CMyGqcTd2v2MB11iej2LNiN/lJ2JUmqLoYF1ApUyvTfvzIqi76EV/GaZB7VxDBXOqe5LOZw4DkPfGsmfhOdaTFLwHYKH6cAKIPWaEVlVvBb10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=TqMByGRD; arc=none smtp.client-ip=209.85.161.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TqMByGRD" Received: by mail-oo1-f46.google.com with SMTP id 006d021491bc7-5fd19b469b0so20935eaf.3 for ; Wed, 26 Feb 2025 10:06:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593160; x=1741197960; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMJT51D6J+8ddUG6Tvu8PMUF0tVoTnLdtTaq9cp5nVI=; b=TqMByGRDxHJhWrqev9FYQEWwE9/IZbIGqoE4WohGVsR8Mj5qoGRk6IQyZIiiYeKFwh 8S6pNHXdMBKw6WJnstFWlL37CHz/SKjareXBDdLK5xM+nwZQYvggcXnVO1eo4Aj+dVBK WIK2VtIzmzOkxfoW2BxFzIGXDdPaeOpOKuFDZOMTNDNHHLophqzy7vsi3bCR0PdGHbbg y6Fh8dw4G4PqCx/weBPaYqLP+5t18T7NXjUojhoS5zs4bqGERCeMMw/vpORl+CNa83Z8 4daCr30tPjsxRoKTmqMcz5FS5bkIS8IHnTNVFTig+OEzpKdiVQJoxjkrCRKSrt+ya/7G uygA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593160; x=1741197960; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMJT51D6J+8ddUG6Tvu8PMUF0tVoTnLdtTaq9cp5nVI=; b=ELF6DGET5mhnAbDLGtMY/4hanBvJFyjIxzigWHNDOcFbD8xjTzgVsyNRZ4gZFfWkn3 ct3utwXHhn/5lzkF20+iTyGNkqMM2yy+2skVOa4pj+ZNcdrn19o8A4bR8ehGnDEXW46y iOklQyOpqgW4TSWTPhTKNxoWpGxaViPkSrRuEnxK/tIvkRdDIb+087oCSW2CputQ6lhx Gj/zKDLjdPCkuWH/tQ9e3d7ToO/2NYHsqO+Y44ZMPlMkN45UIohnBPZIfXUO6uTuBRSd tcRP4/vQW52iqtIIdlNx0F4ysdA0a5LOjPwzXi7RrUb6Epg46N9TP4lZTr+lawnNs6bt qmRg== X-Gm-Message-State: AOJu0YyvqZRx9KPt4RnwjXGlGFU19L5POJ3o6R2Zua+2tZMuVEURj2Tb 6p/E2pkgndR6eKbnN4Uzrz8bWDxx7xxM47xkyrucqlGQBnTzmIZDMB0G X-Gm-Gg: ASbGncvloOHk9QDXNCPi7gybJm1+EFj3o7IWo9o76NblACcP/MGaGyl8LDGRan3/AGF saDbMR62bP4aHR/8SabX3wM6AF47ND+QMQC/0Js6En1fVA5lQH1Z3V4bmVZ7+sEtVXqKPkWz1fo aC7OtJqrCybEg+l/2ZuH2G6jZEpnhXDaIKgLY+dC5KlM3bfLS7IS04LzqO86gD52DfykZMo8H0z E+bKsrgpMY5rYy4baU+nf+yd6rE5Idb/leVVY5/TclsCwzz4YGYGFRcdzbY3ZbOoqdOE+YXqZOY Z9GHjAE= X-Google-Smtp-Source: AGHT+IHr9DMJKqlbTPlfd+NbNgwh28HNT/sX+1cFkWdHXRAm5MpFyWtzzGZ2a4JtB5gChaumfktONg== X-Received: by 2002:a05:6820:812:b0:5fe:9769:d1b9 with SMTP id 006d021491bc7-5fe9769e2ddmr2688301eaf.5.1740593159785; Wed, 26 Feb 2025 10:05:59 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:05:58 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 06/11] x86/softirq: Move softirq_pending to percpu hot section Date: Wed, 26 Feb 2025 13:05:25 -0500 Message-ID: <20250226180531.1242429-7-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/hardirq.h | 4 ++-- arch/x86/kernel/irq.c | 3 +++ 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 8ba2c0f8bcaf..f153c77853de 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -16,7 +16,6 @@ struct pcpu_hot { struct task_struct *current_task; unsigned long top_of_stack; void *hardirq_stack_ptr; - u16 softirq_pending; #ifdef CONFIG_X86_64 bool hardirq_stack_inuse; #else diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h index 6ffa8b75f4cd..f00c09ffe6a9 100644 --- a/arch/x86/include/asm/hardirq.h +++ b/arch/x86/include/asm/hardirq.h @@ -3,7 +3,6 @@ #define _ASM_X86_HARDIRQ_H =20 #include -#include =20 typedef struct { #if IS_ENABLED(CONFIG_KVM_INTEL) @@ -66,7 +65,8 @@ extern u64 arch_irq_stat_cpu(unsigned int cpu); extern u64 arch_irq_stat(void); #define arch_irq_stat arch_irq_stat =20 -#define local_softirq_pending_ref pcpu_hot.softirq_pending +DECLARE_PER_CPU_CACHE_HOT(u16, __softirq_pending); +#define local_softirq_pending_ref __softirq_pending =20 #if IS_ENABLED(CONFIG_KVM_INTEL) /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 385e3a5fc304..474af15ae017 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -31,6 +31,9 @@ DEFINE_PER_CPU_SHARED_ALIGNED(irq_cpustat_t, irq_stat); EXPORT_PER_CPU_SYMBOL(irq_stat); =20 +DEFINE_PER_CPU_CACHE_HOT(u16, __softirq_pending); +EXPORT_PER_CPU_SYMBOL(__softirq_pending); + atomic_t irq_err_count; =20 /* --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0FB1233156 for ; Wed, 26 Feb 2025 18:06:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593165; cv=none; b=oxpXxPbLCL+3lLyTusWzr7y0HVwpcKNcRow5w+G/jGPNauUxjymt/Yvw9fIuIAVAiDfDyvXLH926piq17IuyRY1MNT0NKZ0RlfoNMvrhyk6WpNuDIbnTWJ75bClM3INnGvMU1j55xJ+4kVj5ktOeSUzNRjcyFkQT/SOijMlBpw8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593165; c=relaxed/simple; bh=sa1XYqYkQjBHHrGwa2+ng3XNpCyhBC9elduphKiYq+E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=vC0MyLvOnuKz6oJ16eV/HnFWJXaPGPZvr3lZEH6sRHYiO5YjgqPeknd88eNqvqmcSqj9MHHctZawkgDPjA3ovo7WJAMcI9/hCnDue5abQBLPCzXCPVhwgrKmcYMpkJCOKUVCFzZ8/om9M0o9xEXLN1IkwsWn0tDyJfLzIg9ErxI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QNU/ELfL; arc=none smtp.client-ip=209.85.167.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QNU/ELfL" Received: by mail-oi1-f170.google.com with SMTP id 5614622812f47-3f41091f7c5so66861b6e.3 for ; Wed, 26 Feb 2025 10:06:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593161; x=1741197961; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0xVVRQ21On+BgakJYPRhPKh5ZN4qUIiROQsAFkbaQCk=; b=QNU/ELfLjIUJRb7dq8mrfVqtTEwb5rrliawq6fcWmr6Sk0LpLq/qdpyyxP9vUe6ljp M3LMAKhcb4+aS9lvZmuV1KA2weZ58l7CotL2vrWHTzJe6/AwRRlphzsZCkk809R0aJtV K5aP12D8IcjclpnUYoEhBwLUqvnouP954huMm71tqs8AMGJEiHIdWX/RWto0XIfWTvY9 lcj9S7ExGOkKxjrTjYJf/0GEyt7GDMH5buV9Phlzc3X2JK+sUdLr7ssoHwwop0ApGXKU GuzIc9luX36RRGo3sxhYH79R96HlY10xZrGGr5Ogzr2tLwS9KfrisEEcskfkQk1K1GbU CJAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593161; x=1741197961; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0xVVRQ21On+BgakJYPRhPKh5ZN4qUIiROQsAFkbaQCk=; b=fFfhXreX3Hhk2p4aChIzxI1mLn8BrDA76Wf/jTe34qV+2cMrI7v7Xd8kHlJ4YEw3hY lmRkh/o96Ta7jF8kq4pRJOFzooeqFBdT82z6orFuIZVlKo2H8O0N8m5iLy5hlaKNL4Ge gpLW/+vzmK2Vb8iK6GF9520Gxb9k8oIlKfnULkdyGeYoSfBx0huG5sCtUjWoNiVWvKqE pRC0P6WlPP8uoXZLaviPD2HOHfS0Q8D5fHRA8O2h8PchAxWanoibqmqOqazApDGQoQQI UUccl7G5EECEFiw6UfENsp/Jn6Ip24FRo9xZMiokTja0/Jov35nDWuqg1PI5elobn8gi JCcw== X-Gm-Message-State: AOJu0YzXPA1IdbZexqk03gtc6bNDQV4PxgUN6jFoX0xS5qFQbeSWc9Z/ qCHp4gLzr7vPkQTk8FVb+iyx+OnU4S/5xUMjJLS14cR6fK/YLYnnIeQz X-Gm-Gg: ASbGncvYOddSLR3aaD+7pdMtZBiUd9+mGGet/sV1uBYyZ9vnL76neT0iW6iuIUWxkgL GQrixN1+zozjrOv+9cqZPAhssJIdpD7QvF7i+GeDv8op7oKNydQIO82ypqaRxbBsYHT6xYyREn6 WyoU8t+Oe1o9ebRhaOomSRvJyrWr2X428yAayc1X2dzLUuZ51X6PaALyO7qMsMDhx+MS0AsobP3 2c/x7Ch91ggUe8ziVL2764q0bWPQWXaaZ1AAFdGeF9QdFQBbUN/TDrcEKfzIVSTyF6X26uh1zOy ksq0fxA= X-Google-Smtp-Source: AGHT+IF/OyQpb5vYBiNBBumEKTg0F3GGpztYbLdZjY2Lx0rIKNaUvukD5+tMDHcH0hmapPA2QRaxjQ== X-Received: by 2002:a05:6808:2e89:b0:3f4:2274:3cbb with SMTP id 5614622812f47-3f425a5a56emr13475742b6e.3.1740593161194; Wed, 26 Feb 2025 10:06:01 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.05.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:06:00 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 07/11] x86/irq: Move irq stacks to percpu hot section Date: Wed, 26 Feb 2025 13:05:26 -0500 Message-ID: <20250226180531.1242429-8-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 6 ------ arch/x86/include/asm/irq_stack.h | 12 ++++++------ arch/x86/include/asm/processor.h | 7 +++++++ arch/x86/kernel/dumpstack_32.c | 4 ++-- arch/x86/kernel/dumpstack_64.c | 2 +- arch/x86/kernel/irq.c | 5 +++++ arch/x86/kernel/irq_32.c | 12 +++++++----- arch/x86/kernel/irq_64.c | 6 +++--- arch/x86/kernel/process_64.c | 2 +- 9 files changed, 32 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index f153c77853de..6fad5a4c21d7 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -15,12 +15,6 @@ struct task_struct; struct pcpu_hot { struct task_struct *current_task; unsigned long top_of_stack; - void *hardirq_stack_ptr; -#ifdef CONFIG_X86_64 - bool hardirq_stack_inuse; -#else - void *softirq_stack_ptr; -#endif }; =20 DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); diff --git a/arch/x86/include/asm/irq_stack.h b/arch/x86/include/asm/irq_st= ack.h index 562a547c29a5..735c3a491f60 100644 --- a/arch/x86/include/asm/irq_stack.h +++ b/arch/x86/include/asm/irq_stack.h @@ -116,7 +116,7 @@ ASM_CALL_ARG2 =20 #define call_on_irqstack(func, asm_call, argconstr...) \ - call_on_stack(__this_cpu_read(pcpu_hot.hardirq_stack_ptr), \ + call_on_stack(__this_cpu_read(hardirq_stack_ptr), \ func, asm_call, argconstr) =20 /* Macros to assert type correctness for run_*_on_irqstack macros */ @@ -135,7 +135,7 @@ * User mode entry and interrupt on the irq stack do not \ * switch stacks. If from user mode the task stack is empty. \ */ \ - if (user_mode(regs) || __this_cpu_read(pcpu_hot.hardirq_stack_inuse)) { \ + if (user_mode(regs) || __this_cpu_read(hardirq_stack_inuse)) { \ irq_enter_rcu(); \ func(c_args); \ irq_exit_rcu(); \ @@ -146,9 +146,9 @@ * places. Invoke the stack switch macro with the call \ * sequence which matches the above direct invocation. \ */ \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(func, asm_call, constr); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } \ } =20 @@ -212,9 +212,9 @@ */ #define do_softirq_own_stack() \ { \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, true); \ + __this_cpu_write(hardirq_stack_inuse, true); \ call_on_irqstack(__do_softirq, ASM_CALL_ARG0); \ - __this_cpu_write(pcpu_hot.hardirq_stack_inuse, false); \ + __this_cpu_write(hardirq_stack_inuse, false); \ } =20 #endif diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index b3d153730f63..54fce8d7504d 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -420,6 +420,13 @@ struct irq_stack { char stack[IRQ_STACK_SIZE]; } __aligned(IRQ_STACK_SIZE); =20 +DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, hardirq_stack_ptr); +#ifdef CONFIG_X86_64 +DECLARE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); +#else +DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); +#endif + #ifdef CONFIG_X86_64 static inline unsigned long cpu_kernelmode_gs_base(int cpu) { diff --git a/arch/x86/kernel/dumpstack_32.c b/arch/x86/kernel/dumpstack_32.c index b4905d5173fd..722fd712e1cf 100644 --- a/arch/x86/kernel/dumpstack_32.c +++ b/arch/x86/kernel/dumpstack_32.c @@ -37,7 +37,7 @@ const char *stack_type_name(enum stack_type type) =20 static bool in_hardirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* @@ -62,7 +62,7 @@ static bool in_hardirq_stack(unsigned long *stack, struct= stack_info *info) =20 static bool in_softirq_stack(unsigned long *stack, struct stack_info *info) { - unsigned long *begin =3D (unsigned long *)this_cpu_read(pcpu_hot.softirq_= stack_ptr); + unsigned long *begin =3D (unsigned long *)this_cpu_read(softirq_stack_ptr= ); unsigned long *end =3D begin + (THREAD_SIZE / sizeof(long)); =20 /* diff --git a/arch/x86/kernel/dumpstack_64.c b/arch/x86/kernel/dumpstack_64.c index f05339fee778..6c5defd6569a 100644 --- a/arch/x86/kernel/dumpstack_64.c +++ b/arch/x86/kernel/dumpstack_64.c @@ -134,7 +134,7 @@ static __always_inline bool in_exception_stack(unsigned= long *stack, struct stac =20 static __always_inline bool in_irq_stack(unsigned long *stack, struct stac= k_info *info) { - unsigned long *end =3D (unsigned long *)this_cpu_read(pcpu_hot.hardirq_st= ack_ptr); + unsigned long *end =3D (unsigned long *)this_cpu_read(hardirq_stack_ptr); unsigned long *begin; =20 /* diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c index 474af15ae017..2cd2064457b1 100644 --- a/arch/x86/kernel/irq.c +++ b/arch/x86/kernel/irq.c @@ -34,6 +34,11 @@ EXPORT_PER_CPU_SYMBOL(irq_stat); DEFINE_PER_CPU_CACHE_HOT(u16, __softirq_pending); EXPORT_PER_CPU_SYMBOL(__softirq_pending); =20 +DEFINE_PER_CPU_CACHE_HOT(struct irq_stack *, hardirq_stack_ptr); +#ifdef CONFIG_X86_64 +DEFINE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); +#endif + atomic_t irq_err_count; =20 /* diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c index dc1049c01f9b..48a27cde9635 100644 --- a/arch/x86/kernel/irq_32.c +++ b/arch/x86/kernel/irq_32.c @@ -52,6 +52,8 @@ static inline int check_stack_overflow(void) { return 0; } static inline void print_stack_overflow(void) { } #endif =20 +DEFINE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); + static void call_on_stack(void *func, void *stack) { asm volatile("xchgl %%ebx,%%esp \n" @@ -74,7 +76,7 @@ static inline int execute_on_irq_stack(int overflow, stru= ct irq_desc *desc) u32 *isp, *prev_esp, arg1; =20 curstk =3D (struct irq_stack *) current_stack(); - irqstk =3D __this_cpu_read(pcpu_hot.hardirq_stack_ptr); + irqstk =3D __this_cpu_read(hardirq_stack_ptr); =20 /* * this is where we switch to the IRQ stack. However, if we are @@ -112,7 +114,7 @@ int irq_init_percpu_irqstack(unsigned int cpu) int node =3D cpu_to_node(cpu); struct page *ph, *ps; =20 - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; =20 ph =3D alloc_pages_node(node, THREADINFO_GFP, THREAD_SIZE_ORDER); @@ -124,8 +126,8 @@ int irq_init_percpu_irqstack(unsigned int cpu) return -ENOMEM; } =20 - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D page_address(ph); - per_cpu(pcpu_hot.softirq_stack_ptr, cpu) =3D page_address(ps); + per_cpu(hardirq_stack_ptr, cpu) =3D page_address(ph); + per_cpu(softirq_stack_ptr, cpu) =3D page_address(ps); return 0; } =20 @@ -135,7 +137,7 @@ void do_softirq_own_stack(void) struct irq_stack *irqstk; u32 *isp, *prev_esp; =20 - irqstk =3D __this_cpu_read(pcpu_hot.softirq_stack_ptr); + irqstk =3D __this_cpu_read(softirq_stack_ptr); =20 /* build the stack frame on the softirq stack */ isp =3D (u32 *) ((char *)irqstk + sizeof(*irqstk)); diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c index 56bdeecd8ee0..4834e317e568 100644 --- a/arch/x86/kernel/irq_64.c +++ b/arch/x86/kernel/irq_64.c @@ -50,7 +50,7 @@ static int map_irq_stack(unsigned int cpu) return -ENOMEM; =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #else @@ -63,14 +63,14 @@ static int map_irq_stack(unsigned int cpu) void *va =3D per_cpu_ptr(&irq_stack_backing_store, cpu); =20 /* Store actual TOS to avoid adjustment in the hotpath */ - per_cpu(pcpu_hot.hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; + per_cpu(hardirq_stack_ptr, cpu) =3D va + IRQ_STACK_SIZE - 8; return 0; } #endif =20 int irq_init_percpu_irqstack(unsigned int cpu) { - if (per_cpu(pcpu_hot.hardirq_stack_ptr, cpu)) + if (per_cpu(hardirq_stack_ptr, cpu)) return 0; return map_irq_stack(cpu); } diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 226472332a70..93de583c05d1 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -614,7 +614,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) int cpu =3D smp_processor_id(); =20 WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) && - this_cpu_read(pcpu_hot.hardirq_stack_inuse)); + this_cpu_read(hardirq_stack_inuse)); =20 if (!test_tsk_thread_flag(prev_p, TIF_NEED_FPU_LOAD)) switch_fpu_prepare(prev_p, cpu); --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 620BA2376F8 for ; Wed, 26 Feb 2025 18:06:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593170; cv=none; b=hftmd81krXHSjFnQj5S0kPsvzBZk8YgTH/R7MvhdKrH8HWdygP+ZSCGxDUjzZJm/EHUxmGahXtGwrgWH3jXua6V245+D1g7EkBXbAfRBjt1a9ecKfDFPc74crwv7JGfJ+RYN0jpU/Vd3MFhS/XobrRsg2XiTOxouKpMnrkh5YYE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593170; c=relaxed/simple; bh=psXXdozQ8cmMNoyhe6WfyX2b0xDHLMpW1n/cO6IYXms=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RsA+HRh/LHMZgzYyEceHW4crJNtd3LeilXgQYLKyy39v4UrL65myLfxcZ1s+/C10q5Q2JdOgY9zFOtzsL4hhx/xeBWAO+8LciFBIfp0tNgvXEt6hCjQAL4J1iDCmXgpeV/aGvbRtURBSyJ5bJ6ditw2VnQf4SiHj4Lx8/pG/WL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Y14BTrLI; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y14BTrLI" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5fcd124dd1cso45653eaf.1 for ; Wed, 26 Feb 2025 10:06:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593164; x=1741197964; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i67t7Mkp4mM65QsMQr6CC8tvD3rsmoWgTylULSSsfnA=; b=Y14BTrLI/+NNRF26y+PcBafGjlgHKUESxhnYNTFKznxvOVDfgVys4XQhhDpbzH3hwg WRyp3XR83+F72up6PNfR9QgudIcs20vmVrYgyE/QLg15qB08yjnVBERimsgah5vGQ7x3 f84WE2BjJusk9eBQqb8mlhFeP2xeOyqlxt8RmMXD4TGKbL0XRnwL/ejW55d4oHjVvQhg C3PhDWlnvAhbYQ7BNXvaXqc09RmGCUhipYKUMsw90fhKkaPnkSvX2YBXtIGXESEVVTk1 gPm8FcvPG2+ZX8kCDH6Yj6IOJylkzna3MN67rCoCmKyC9nnfyfZJ9Ai8ZJriQIH4OGoI SLcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593164; x=1741197964; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i67t7Mkp4mM65QsMQr6CC8tvD3rsmoWgTylULSSsfnA=; b=qZPngj4fEsFdDfeZYltotJ3riR8jGGhor/TP2FSpJoWoBoAscc12DVsIT1+fB7I6Gq V0C6mnnWPJQJ/x96imsBkAIJcOY0vWjxIWM5q1f9pk6wNC6WiJrct5271uxaoQoybIDY CWuLkYSlaz+3T61VswL02LSnLbXd2wfJlShgo0ycExD/cXW3t2jMnFxS60v7SqcTmMQ5 HnwzqlHcdTpZKODRUBQ8T2VEdCeijTNvPKZJUAQABzlTC4bk4sUdISaDYlkxMx0jo9+5 R+kstmDdPrTuTR0jw2DAfIg5ZCswa3WmEFZf8VoUFVjIrsG8q8PHcJH0gqsZ19btuCw/ mByQ== X-Gm-Message-State: AOJu0YzK5Xc/pvAuFBv/wnhf4yd4a0vdbxaMSG4R1Wg7zz4q//5vbIPD sGhzk/KPYYk0vRxaEn7p7iWIReqX6Ums08/So/stBRnu9zQtYeMTt0bZ X-Gm-Gg: ASbGncsBY3imjh7s0qNlhFVnFYbt549wn7ZRg1AuwgEJCI9Afc7u3hbX2YmcoLazfgR crezLWEPzkFUC3NgCk4kYiKjXq+vYjqW09I0MGEzDy7Ob9x5NRnJcHB0RxU1g1wkQg56WZ/9M2O 2kYvvLFFqKWCl6LeBw23SHjwQ0oQIB/Mz+4pQHBhVp9upnf8raR9rdjFpQO3+oLFbaoY7IoXrxV Pkmq+AUITUDuZ5IffcC1uwB+WfBrLh7Pqlm1XJjz1BiDw2z53YTbtgozSnG959amj2D2UX6HeGI 46gfWpU= X-Google-Smtp-Source: AGHT+IGfB0j8B30jno4/9489+2Ky3OB7yEVBxj0NNkwtfe1ygCCxVdKI6KpFLN8URNPsn1c0wk6oeg== X-Received: by 2002:a05:6820:1b83:b0:5fc:a89b:a33c with SMTP id 006d021491bc7-5fe9d676f86mr1945491eaf.4.1740593163953; Wed, 26 Feb 2025 10:06:03 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.06.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:06:03 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 08/11] x86/percpu: Move top_of_stack to percpu hot section Date: Wed, 26 Feb 2025 13:05:27 -0500 Message-ID: <20250226180531.1242429-9-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/entry/entry_32.S | 4 ++-- arch/x86/entry/entry_64.S | 6 +++--- arch/x86/entry/entry_64_compat.S | 4 ++-- arch/x86/include/asm/current.h | 1 - arch/x86/include/asm/percpu.h | 2 +- arch/x86/include/asm/processor.h | 8 ++++++-- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 3 ++- arch/x86/kernel/process_32.c | 4 ++-- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 1 + 12 files changed, 21 insertions(+), 17 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 20be5758c2d2..92c0b4a94e0a 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -1153,7 +1153,7 @@ SYM_CODE_START(asm_exc_nmi) * is using the thread stack right now, so it's safe for us to use it. */ movl %esp, %ebx - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esp + movl PER_CPU_VAR(cpu_current_top_of_stack), %esp call exc_nmi movl %ebx, %esp =20 @@ -1217,7 +1217,7 @@ SYM_CODE_START(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movl PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %esi + movl PER_CPU_VAR(cpu_current_top_of_stack), %esi leal -TOP_OF_KERNEL_STACK_PADDING-PTREGS_SIZE(%esi), %esp =20 call make_task_dead diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 33a955aa01d8..9baf32a7a118 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -92,7 +92,7 @@ SYM_CODE_START(entry_SYSCALL_64) /* tss.sp2 is scratch space. */ movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_64_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -1166,7 +1166,7 @@ SYM_CODE_START(asm_exc_nmi) FENCE_SWAPGS_USER_ENTRY SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rdx movq %rsp, %rdx - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp UNWIND_HINT_IRET_REGS base=3D%rdx offset=3D8 pushq 5*8(%rdx) /* pt_regs->ss */ pushq 4*8(%rdx) /* pt_regs->rsp */ @@ -1484,7 +1484,7 @@ SYM_CODE_START_NOALIGN(rewind_stack_and_make_dead) /* Prevent any naive code from trying to unwind to our caller. */ xorl %ebp, %ebp =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rax + movq PER_CPU_VAR(cpu_current_top_of_stack), %rax leaq -PTREGS_SIZE(%rax), %rsp UNWIND_HINT_REGS =20 diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_com= pat.S index ed0a5f2dc129..a45e1125fc6c 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -57,7 +57,7 @@ SYM_CODE_START(entry_SYSENTER_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rax popq %rax =20 - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 /* Construct struct pt_regs on stack */ pushq $__USER_DS /* pt_regs->ss */ @@ -193,7 +193,7 @@ SYM_CODE_START(entry_SYSCALL_compat) SWITCH_TO_KERNEL_CR3 scratch_reg=3D%rsp =20 /* Switch to the kernel stack */ - movq PER_CPU_VAR(pcpu_hot + X86_top_of_stack), %rsp + movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp =20 SYM_INNER_LABEL(entry_SYSCALL_compat_safe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 6fad5a4c21d7..3d1b123c2ee3 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -14,7 +14,6 @@ struct task_struct; =20 struct pcpu_hot { struct task_struct *current_task; - unsigned long top_of_stack; }; =20 DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 7cb4f64b2e60..044410462d36 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -554,7 +554,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and pcpu_hot.top_of_stack, both of which are + * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index 54fce8d7504d..b4d51de071f2 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -427,6 +427,10 @@ DECLARE_PER_CPU_CACHE_HOT(bool, hardirq_stack_inuse); DECLARE_PER_CPU_CACHE_HOT(struct irq_stack *, softirq_stack_ptr); #endif =20 +DECLARE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_CACHE_HOT(const unsigned long __percpu_seg_override, const= _cpu_current_top_of_stack); + #ifdef CONFIG_X86_64 static inline unsigned long cpu_kernelmode_gs_base(int cpu) { @@ -552,9 +556,9 @@ static __always_inline unsigned long current_top_of_sta= ck(void) * entry trampoline. */ if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.top_of_stack); + return this_cpu_read_const(const_cpu_current_top_of_stack); =20 - return this_cpu_read_stable(pcpu_hot.top_of_stack); + return this_cpu_read_stable(cpu_current_top_of_stack); } =20 static __always_inline bool on_thread_stack(void) diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 6fae88f8ae1e..54ace808defd 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_top_of_stack, pcpu_hot, top_of_stack); OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index 01f33fb86f05..fc059e9c8867 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2016,7 +2016,6 @@ __setup("clearcpuid=3D", setup_clearcpuid); =20 DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { .current_task =3D &init_task, - .top_of_stack =3D TOP_OF_INIT_STACK, }; EXPORT_PER_CPU_SYMBOL(pcpu_hot); EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); @@ -2024,6 +2023,8 @@ EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); =20 +DEFINE_PER_CPU_CACHE_HOT(unsigned long, cpu_current_top_of_stack) =3D TOP_= OF_INIT_STACK; + #ifdef CONFIG_X86_64 /* * Note: Do not make this dependant on CONFIG_MITIGATION_CALL_DEPTH_TRACKI= NG diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 0917c7f25720..3afb2428bedb 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -190,13 +190,13 @@ __switch_to(struct task_struct *prev_p, struct task_s= truct *next_p) arch_end_context_switch(next_p); =20 /* - * Reload esp0 and pcpu_hot.top_of_stack. This changes + * Reload esp0 and cpu_current_top_of_stack. This changes * current_thread_info(). Refresh the SYSENTER configuration in * case prev or next is vm86. */ update_task_stack(next_p); refresh_sysenter_cs(next); - this_cpu_write(pcpu_hot.top_of_stack, + this_cpu_write(cpu_current_top_of_stack, (unsigned long)task_stack_page(next_p) + THREAD_SIZE); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 93de583c05d1..f68da7b7e50c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -669,7 +669,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) * Switch the PDA and FPU contexts. */ raw_cpu_write(pcpu_hot.current_task, next_p); - raw_cpu_write(pcpu_hot.top_of_stack, task_top_of_stack(next_p)); + raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index c10850ae6f09..15e054f4cbf6 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -851,7 +851,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) =20 #ifdef CONFIG_X86_32 /* Stack for startup_32 can be just as for start_secondary onwards */ - per_cpu(pcpu_hot.top_of_stack, cpu) =3D task_top_of_stack(idle); + per_cpu(cpu_current_top_of_stack, cpu) =3D task_top_of_stack(idle); #endif return 0; } diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 7586a9be8c59..85032c085af2 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -44,6 +44,7 @@ ENTRY(phys_startup_64) =20 jiffies =3D jiffies_64; const_pcpu_hot =3D pcpu_hot; +const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) /* --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f42.google.com (mail-oo1-f42.google.com [209.85.161.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA38023815F for ; Wed, 26 Feb 2025 18:06:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593170; cv=none; b=Nife9aYUWdeyF2nANDcnS5FMSJbMfBXhTFcR1EGn4LF+CERF4/NXKJ746km/314QqVHGNjcR0kxwrvwoX/a/KqY1+i3gkLJW5VGDk+2jSzBB9LRZAgTnDGGLGWDsJAWnSQ+jIWZ7dLOH1j7LGmiBkTMEoy7c3szt/OweZ7qlhIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593170; c=relaxed/simple; bh=ZFzEnDq8KgjuURenm9ik6dY5TfGXOxRURJTNWlmo0+4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nk7icyx/GMceamFT542xnwaj/UXMCBhh2mrsjP4JRBTCOAejuWykf4RpWsTpcdYaExvI4nyrXAkKG6ndCWSpHd/a6JDsI9lapcd6ifMFcwGfg9sLmaJvOmjEK46uHA3u4NNNDrwdjTEHUv/tlxSxD+WeJ/W702d1ufZLarMiiSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LLY2O273; arc=none smtp.client-ip=209.85.161.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LLY2O273" Received: by mail-oo1-f42.google.com with SMTP id 006d021491bc7-5fce03395d4so36606eaf.2 for ; Wed, 26 Feb 2025 10:06:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593167; x=1741197967; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vy1ju9SebX5v1mWl0+k0224cFXdsKGZNI8ymJ4KND2o=; b=LLY2O273i/+mYTUrp+zKj//tPOJZQYIg7K6atdN41odO+W7mkOgyFCM28Lz9/q9jWS zLE7oiHBwtCI3an06y+5iNz2rxMohcH58mh3e90ubxHX4SDAUDsFmfbZcEYr+5VJCVk+ WK7zhbwC73H/UtHbymQ6fNM6d0fQDp0rUNmRCptIzM2+S/7ZRqIaD/XaR0s5rfxRm3U3 AryLuhcwm49L71wlY25W2AGIe6MIy41tMZHqXJ7nYIlfzqY8kc7BO+GX4Cl74v6hBLH4 imQADLEQj9CzM3K6qbnQOyEziWkBWq9uQmqG5Etxh8VslBzH0/+goU+GXefx0CEUxz2B WocQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593167; x=1741197967; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vy1ju9SebX5v1mWl0+k0224cFXdsKGZNI8ymJ4KND2o=; b=Uzv9nfzvMZ9qZksfEpvzY3y7u8TNytnRRMaiCJ+uQIdJiJn8vbzwjU+oTjEYjJcxLt gTvI6jVDUKBpU1pwB8T1XBYvJddh7K869+gX4g1GngNvW+TEyaFjaps0vRq1fzBg2TPJ eydwD3+N9z1NCYg/S1qxIYKlgwkGFoFfhMJMP9OtiJfjhNJj9+gBX3kVJzYpmkWk1JJA N5smcaKv5CXIVDGtQTUtP3N88I6KWf+BNlm+wHOY/QO/ndAeZDsx9lRwdiAegBJPSFTf MdK5SG63O/e3taSe+30Q4RUa9mFJI2inYgvKUPCf2ARn+LXvSfPr8dBzN+GgSkoQUZjG ao1g== X-Gm-Message-State: AOJu0YwbOlM/8mXSY1GQKaZScP4kWKjtq1/726oPRsPgKD3W2+Iaesuy 0JDp9FJyScTFV0w/5WAL+J0rjA2TPf8zTphSJ5IuX1tZJpSfD3cGk2+1 X-Gm-Gg: ASbGncvxYohAr83IVmQEk8OJbO0v49aL9uJd/UjrfpXwUYJ9E65Z2RNdhbiLaBINxnE VhTWf6bSAdXR9Q7F029svp+iDEmx4y+Au+lv4ewFEe0SmzSH3SohZ3fFdCyqDlUQQDd6poUtBrg LrlDtwWu/luUj9agCqL2t+D5bgftTaYIC6C3DPCaV5UD1UhrG7LSM+td7sbmfUF2baC/+E1Baeo Eqxa+rkysVH9X7ACU/rGQSFXBksQ8+AC4RHUc66qFfMYm0w4a+yj59mrSygbjJmoUPyOpjaL9nD T2Tjz5g= X-Google-Smtp-Source: AGHT+IFUnBI5a7MiiktlMTCWimF3hQEAHEbEDDvLuwmkHUGlozp+ySgd/sKZIK34KglEqEaii0rq9g== X-Received: by 2002:a05:6820:1611:b0:5fc:e3b3:3c10 with SMTP id 006d021491bc7-5fe9d5d390cmr2129145eaf.2.1740593167435; Wed, 26 Feb 2025 10:06:07 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.06.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:06:05 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 09/11] x86/percpu: Move current_task to percpu hot section Date: Wed, 26 Feb 2025 13:05:28 -0500 Message-ID: <20250226180531.1242429-10-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/current.h | 17 ++++++----------- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/asm-offsets.c | 1 - arch/x86/kernel/cpu/common.c | 8 +++----- arch/x86/kernel/head_64.S | 4 ++-- arch/x86/kernel/process_32.c | 2 +- arch/x86/kernel/process_64.c | 2 +- arch/x86/kernel/smpboot.c | 2 +- arch/x86/kernel/vmlinux.lds.S | 2 +- scripts/gdb/linux/cpus.py | 2 +- 10 files changed, 17 insertions(+), 25 deletions(-) diff --git a/arch/x86/include/asm/current.h b/arch/x86/include/asm/current.h index 3d1b123c2ee3..dea7d8b854f0 100644 --- a/arch/x86/include/asm/current.h +++ b/arch/x86/include/asm/current.h @@ -12,22 +12,17 @@ =20 struct task_struct; =20 -struct pcpu_hot { - struct task_struct *current_task; -}; - -DECLARE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot); - -/* const-qualified alias to pcpu_hot, aliased by linker. */ -DECLARE_PER_CPU_CACHE_HOT(const struct pcpu_hot __percpu_seg_override, - const_pcpu_hot); +DECLARE_PER_CPU_CACHE_HOT(struct task_struct *, current_task); +/* const-qualified alias provided by the linker. */ +DECLARE_PER_CPU_CACHE_HOT(struct task_struct * const __percpu_seg_override, + const_current_task); =20 static __always_inline struct task_struct *get_current(void) { if (IS_ENABLED(CONFIG_USE_X86_SEG_SUPPORT)) - return this_cpu_read_const(const_pcpu_hot.current_task); + return this_cpu_read_const(const_current_task); =20 - return this_cpu_read_stable(pcpu_hot.current_task); + return this_cpu_read_stable(current_task); } =20 #define current get_current() diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index 044410462d36..e347c6656ce8 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -554,7 +554,7 @@ do { \ * it is accessed while this_cpu_read_stable() allows the value to be cach= ed. * this_cpu_read_stable() is more efficient and can be used if its value * is guaranteed to be valid across CPUs. The current users include - * pcpu_hot.current_task and cpu_current_top_of_stack, both of which are + * current_task and cpu_current_top_of_stack, both of which are * actually per-thread variables implemented as per-CPU variables and * thus stable for the duration of the respective task. */ diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 54ace808defd..ad4ea6fb3b6c 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -107,7 +107,6 @@ static void __used common(void) OFFSET(TSS_sp0, tss_struct, x86_tss.sp0); OFFSET(TSS_sp1, tss_struct, x86_tss.sp1); OFFSET(TSS_sp2, tss_struct, x86_tss.sp2); - OFFSET(X86_current_task, pcpu_hot, current_task); #if IS_ENABLED(CONFIG_CRYPTO_ARIA_AESNI_AVX_X86_64) /* Offset for fields in aria_ctx */ BLANK(); diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index fc059e9c8867..ac8721a0eb3a 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2014,11 +2014,9 @@ static __init int setup_clearcpuid(char *arg) } __setup("clearcpuid=3D", setup_clearcpuid); =20 -DEFINE_PER_CPU_CACHE_HOT(struct pcpu_hot, pcpu_hot) =3D { - .current_task =3D &init_task, -}; -EXPORT_PER_CPU_SYMBOL(pcpu_hot); -EXPORT_PER_CPU_SYMBOL(const_pcpu_hot); +DEFINE_PER_CPU_CACHE_HOT(struct task_struct *, current_task) =3D &init_tas= k; +EXPORT_PER_CPU_SYMBOL(current_task); +EXPORT_PER_CPU_SYMBOL(const_current_task); =20 DEFINE_PER_CPU_CACHE_HOT(int, __preempt_count) =3D INIT_PREEMPT_COUNT; EXPORT_PER_CPU_SYMBOL(__preempt_count); diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S index 2843b0a56198..fefe2a25cf02 100644 --- a/arch/x86/kernel/head_64.S +++ b/arch/x86/kernel/head_64.S @@ -322,7 +322,7 @@ SYM_INNER_LABEL(common_startup_64, SYM_L_LOCAL) * * RDX contains the per-cpu offset */ - movq pcpu_hot + X86_current_task(%rdx), %rax + movq current_task(%rdx), %rax movq TASK_threadsp(%rax), %rsp =20 /* @@ -433,7 +433,7 @@ SYM_CODE_START(soft_restart_cpu) UNWIND_HINT_END_OF_STACK =20 /* Find the idle task stack */ - movq PER_CPU_VAR(pcpu_hot + X86_current_task), %rcx + movq PER_CPU_VAR(current_task), %rcx movq TASK_threadsp(%rcx), %rsp =20 jmp .Ljump_to_C_code diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 3afb2428bedb..c276dfda387f 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -206,7 +206,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) if (prev->gs | next->gs) loadsegment(gs, next->gs); =20 - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); =20 switch_fpu_finish(next_p); =20 diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index f68da7b7e50c..13893ec03d85 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -668,7 +668,7 @@ __switch_to(struct task_struct *prev_p, struct task_str= uct *next_p) /* * Switch the PDA and FPU contexts. */ - raw_cpu_write(pcpu_hot.current_task, next_p); + raw_cpu_write(current_task, next_p); raw_cpu_write(cpu_current_top_of_stack, task_top_of_stack(next_p)); =20 switch_fpu_finish(next_p); diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c index 15e054f4cbf6..c89545a61d08 100644 --- a/arch/x86/kernel/smpboot.c +++ b/arch/x86/kernel/smpboot.c @@ -841,7 +841,7 @@ int common_cpu_up(unsigned int cpu, struct task_struct = *idle) /* Just in case we booted with a single CPU. */ alternatives_enable_smp(); =20 - per_cpu(pcpu_hot.current_task, cpu) =3D idle; + per_cpu(current_task, cpu) =3D idle; cpu_init_stack_canary(cpu, idle); =20 /* Initialize the interrupt stack(s) */ diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S index 85032c085af2..9ac6b42701fa 100644 --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -43,7 +43,7 @@ ENTRY(phys_startup_64) #endif =20 jiffies =3D jiffies_64; -const_pcpu_hot =3D pcpu_hot; +const_current_task =3D current_task; const_cpu_current_top_of_stack =3D cpu_current_top_of_stack; =20 #if defined(CONFIG_X86_64) diff --git a/scripts/gdb/linux/cpus.py b/scripts/gdb/linux/cpus.py index 13eb8b3901b8..8f7c4fb78c2c 100644 --- a/scripts/gdb/linux/cpus.py +++ b/scripts/gdb/linux/cpus.py @@ -164,7 +164,7 @@ def get_current_task(cpu): var_ptr =3D gdb.parse_and_eval("(struct task_struct *)cpu_task= s[0].task") return var_ptr.dereference() else: - var_ptr =3D gdb.parse_and_eval("&pcpu_hot.current_task") + var_ptr =3D gdb.parse_and_eval("¤t_task") return per_cpu(var_ptr, cpu).dereference() elif utils.is_target_arch("aarch64"): current_task_addr =3D gdb.parse_and_eval("(unsigned long)$SP_EL0") --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f51.google.com (mail-oo1-f51.google.com [209.85.161.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 259C2238D38 for ; Wed, 26 Feb 2025 18:06:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593172; cv=none; b=HHMw09vqMqA1T/3Q3NslB+lI2M+eS+0rPsWcIAiBVETCLsUlWYWcs/yYJ9zd6GmaAtFZXSm90ezK+y9IqDeeE5LlOSo3zPDQvMl+kPpwSV5Xbvwa2EtAcvLDTGpP41KL8t8LVG52GbGfuwQH/e/1pFMuZArChWQbNNqmchjJUO0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593172; c=relaxed/simple; bh=+NITKJToEUXuaXK8zS47p3zttszR09n9fFSVaLvRHdk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=u9UYnRBPjSMmbcUoqpi8xYM0IlPoxVmt9VaV9a2AaRrOCe0S3wLSEOzJEMxGMtjELMDyguysQ3VX/qx6zNfAuo1sdkDci6PLUoT9Wd8v1gmy9yIR/PNr0+QXDsSPzEi/D2RZr8ISIQKTcrZ8Rd3LrZ0tmXOf0m/ymYJ67vdXtXU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Yn9mxMVo; arc=none smtp.client-ip=209.85.161.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Yn9mxMVo" Received: by mail-oo1-f51.google.com with SMTP id 006d021491bc7-5fe944a4243so17833eaf.0 for ; Wed, 26 Feb 2025 10:06:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593170; x=1741197970; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AzASOhbn/TdUnUq2BoNH4zzSfvXYoXG0ehJxyB+XGAg=; b=Yn9mxMVoVEencS7lXzRJU1YL+j7md8nXLHoMXxlzKghYcO1Taf6Zg9S8ujTo7a6aJj yEsFeBhR6HbfiOT6kvUtnpgCRSPR6LdC1On6m9+ulWoeAqtPChdF7Sfa9LdB+sWE9665 MYx7/pa4sPQu+98BD7qfnfZOiQ/Gu68g96I8lyYzScqN/1HXEmXz855Nqr+uVEeWpyNN VjVUUlmlMFfZI4XX3WzG8optFFLm4y+UsPFz1luD0iL58wgz9UhZy71JSDaCEx50yBP7 RC/o5bYOiKtIKT1nEDLvXHXOD1S/1Uro/cxDrcdhOjky+Cjt7F3N3Dpnc7t/5KsOmhP0 q1RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593170; x=1741197970; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AzASOhbn/TdUnUq2BoNH4zzSfvXYoXG0ehJxyB+XGAg=; b=QL41eUK493Rm6ZquMEprrDSI0G46kVigdGLpPNFgPO8ANGHlZL/XI0hulQSCxr0KJ3 ujJG4/E9oanLcfCuIpykOjkGkyZ/7JQGdM65f6HRxyQJDHZZundbHlD61tC35ttU95uA bRiHA8RYrsQpjiclPxJmTMRtiIR55haZd0/MtBmW1YiOmZOEMgKdE7m3hDH0ezBrNpLa 7E721KvdHdtlzjeig7eq+GdLPBE2BtNu9CtnjS5514jRfIohgt2kNOWPmNH/doZo4pU6 OAM+1eyzyscG6wXUFhb0XRidGOs3M964dWbx9BP3K4ZbP1Fvxxl/z/HA/r74DQUq4qRP arsA== X-Gm-Message-State: AOJu0YxoS2AUZHbPcS5HrdgYwQ4KvIcD4kanNR2vXPQJzmMYa0wcT7rV a5YZmsQFOX5AIYhs3YSeNpNf3o7ftcUnAHgCGrKS8sVttGSq5Zm6pGZL X-Gm-Gg: ASbGnctmSAhsIlcE6t92uygGkICKUSubWKCWwRtQbAtY+r3dBTiOCWyfEh4g46h+SxW TccbA5gbkP+RcRmebLtOe30i5dd6vJwTiR5uitxOoDJuW4zI7t36P7OImw92ERCQNtuVaNnu2TH 13wVpKlcs6Tb38wqND0RNMlxr/9PuBZDNNlF7V9G12jGLxGCERO6+tqmK3xLTe3hOL7bkbgly4B LNzKgUJbbQ4+36oXN3jOp2rFjR5XWirldiFszMoeEfC4x7Bek6K3l9fubw1aOpDNN4/pAu0yavp Ht0vI9Q= X-Google-Smtp-Source: AGHT+IFii8Y/uCMqIDFr/IoOM4XP5abwoUhsViZfg/Vzn8cm5Kj85HSMWVmn3VxiKZKd22QeLpQM+A== X-Received: by 2002:a05:6820:506:b0:5fd:b85:9b47 with SMTP id 006d021491bc7-5fe937185camr3833158eaf.1.1740593169862; Wed, 26 Feb 2025 10:06:09 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.06.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:06:09 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 10/11] x86/stackprotector: Move __stack_chk_guard to percpu hot section Date: Wed, 26 Feb 2025 13:05:29 -0500 Message-ID: <20250226180531.1242429-11-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/stackprotector.h | 2 +- arch/x86/kernel/cpu/common.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/stackprotector.h b/arch/x86/include/asm/s= tackprotector.h index d43fb589fcf6..cd761b14eb02 100644 --- a/arch/x86/include/asm/stackprotector.h +++ b/arch/x86/include/asm/stackprotector.h @@ -20,7 +20,7 @@ =20 #include =20 -DECLARE_PER_CPU(unsigned long, __stack_chk_guard); +DECLARE_PER_CPU_CACHE_HOT(unsigned long, __stack_chk_guard); =20 /* * Initialize the stackprotector canary value. diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index ac8721a0eb3a..62472b8f798a 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -2097,7 +2097,7 @@ void syscall_init(void) #endif /* CONFIG_X86_64 */ =20 #ifdef CONFIG_STACKPROTECTOR -DEFINE_PER_CPU(unsigned long, __stack_chk_guard); +DEFINE_PER_CPU_CACHE_HOT(unsigned long, __stack_chk_guard); #ifndef CONFIG_SMP EXPORT_PER_CPU_SYMBOL(__stack_chk_guard); #endif --=20 2.48.1 From nobody Sun Feb 8 02:21:34 2026 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8319E23A564 for ; Wed, 26 Feb 2025 18:06:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593175; cv=none; b=oWZ7f8RapSjih/RHEhgZRpIrUrOFW2IyQy547GLIBrfeg5znfCdXWHtyzTRheU3BLDXFfHGHhsN88NCwJ6KBpzSNmU5nJEZC39BXMnttXHsvnR/OVst+ATMAZ75NdARWBjQZjIYDS9as1ST8LVgvjOOYVp7FrbtyR6Lcz9I9vW8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1740593175; c=relaxed/simple; bh=gbeR1LC6epTxL5hjM57IzLXNWDQ+GaAp9Yh449sHR1o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=g1x5E1E3nwHIlqDaLoNZP8q4OaSQdrKJ2w2UICAU36OYAzSamF5BhrnkKRsaXjA+7syYhPKwRnFQ175wCgXpH4WpRknvwuQLyBVOmV9rgegsp0BRrhyEieR1F8kHKR5t33YjKjS9J5fTG0Ko0VnXJTOEVrDsM6JPMscdcJAxoFA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=RErttnN9; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RErttnN9" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5f6b65c89c4so21317eaf.2 for ; Wed, 26 Feb 2025 10:06:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1740593172; x=1741197972; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ib/up2lq4vie8y203Us1LjVuEcmMH4a7XcIoSxmd8Ww=; b=RErttnN9u742rYFY4uL2Lh61sCFEb8ipMca0FsN9bi2Ld8bSVw7ReIVjPXG3K3mJUB U7Qo32BxqmEOAdOcacMeYftfNm3UapzHoO1irInnwni8KUqvoPAazigdCVpgJ9HJagvS DHlH/l5COHtL5+Q4//Lp1xr0O7gI+f8JwWPy6FxpCGhB5QcD8vBm4b2Q2O5rD7j5V5rz +J83DIs4Ijj0JLrL+dBxWplDXVeiE08kzy43l8uwT0bhfJfnXuiZ085tWB2U3JDTJorw KPTol96p6ILrFsMhMSrKJl7jjp2dPXyUWpi0o865/FtYtw49DyF9i+q+1cyl1SEIi4Z0 Hwmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740593172; x=1741197972; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ib/up2lq4vie8y203Us1LjVuEcmMH4a7XcIoSxmd8Ww=; b=lpQ/niTzOuUAxFEidKq1bpW4PjOv4KyB5qwtuCyhI4Ge/7VAfC0dX/8syNKZZGFDTK aWuTM96+w9ijz5Txrwg01kZgOfpcn6dN3RDQZsNFJAl+EHQrMpoEeDe/NPkXUaXrnWqB 419MGpWYvXxtIFeLAr/Rtb05ydPwEzasdL6KdQqCyXgahLDiN/Z58UrPzh1pstbfPJvQ NqKBMTlTpRtOYZiFCNNg2oS97n3bM8fRq89MsXFbkW4lCrpefgoYB4qtrv2Oy73KdrJd iXVJRIR1Q1hgfdb0GkiDtJXEP+2oRaz73pUtpidS1wEDRH+4vwyTbCbE5fPp+cQhPldY xOdA== X-Gm-Message-State: AOJu0Yze+VBYiEY5PNmavwUbp+ZZaUsYgqnim5WjyaS/KtOATm388Lp6 GenTKsxZ3N/pvuaV8IGMyh9c170/V4CUr+X5exgNOldYHI8b+GG112Is X-Gm-Gg: ASbGncuHPmEbZf8AnSEcadINnlTp6GXO2LgFsOr3Tl4LmZiwTeHa+OP+uFBG0JomBKl IxS4IEclYB+80XuDamZoHlobaT11l/fbnz0F/fuETLiQ0VEVIQsDVdLzODGh1gA5zolN2caELZb 6QFCB1eTnbW26bwGQfLLQLGZKUrzfurMDbfYkX7SObj1TRJ61YaKl11u769JTJGeACD9I/rP1wx S2BB8RjKgOKGRRB85jzWhufyqC8vyRcqE99UVOme4X1ryU/loM0B+AOz+d6oaw1T8vSKSs0+ywF tmEQsSw= X-Google-Smtp-Source: AGHT+IEEw2uHIFBgv6fuSgOXXKvyHfckRf/gCuwJonZJVz2UKrz+YgfxDShvuBLOs6wsHI4meNd+hA== X-Received: by 2002:a05:6808:2105:b0:3f4:af3:74c0 with SMTP id 5614622812f47-3f540fbdf5fmr5020591b6e.29.1740593172172; Wed, 26 Feb 2025 10:06:12 -0800 (PST) Received: from citadel.lan ([2600:6c4a:4d3f:6d5c::1019]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5fe9428c8b3sm755069eaf.39.2025.02.26.10.06.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 10:06:11 -0800 (PST) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Ingo Molnar , "H . Peter Anvin" , Thomas Gleixner , Borislav Petkov , Ard Biesheuvel , Uros Bizjak , Linus Torvalds , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Brian Gerst Subject: [PATCH v2 11/11] x86/smp: Move this_cpu_off to percpu hot section Date: Wed, 26 Feb 2025 13:05:30 -0500 Message-ID: <20250226180531.1242429-12-brgerst@gmail.com> X-Mailer: git-send-email 2.48.1 In-Reply-To: <20250226180531.1242429-1-brgerst@gmail.com> References: <20250226180531.1242429-1-brgerst@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Brian Gerst --- arch/x86/include/asm/percpu.h | 2 +- arch/x86/kernel/setup_percpu.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h index e347c6656ce8..f29c85a0abf4 100644 --- a/arch/x86/include/asm/percpu.h +++ b/arch/x86/include/asm/percpu.h @@ -589,7 +589,7 @@ do { \ #include =20 /* We can use this directly for local CPU (faster). */ -DECLARE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DECLARE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off); =20 #endif /* !__ASSEMBLY__ */ =20 diff --git a/arch/x86/kernel/setup_percpu.c b/arch/x86/kernel/setup_percpu.c index 175afc3ffb12..bfa48e7a32a2 100644 --- a/arch/x86/kernel/setup_percpu.c +++ b/arch/x86/kernel/setup_percpu.c @@ -26,7 +26,7 @@ DEFINE_PER_CPU_CACHE_HOT(int, cpu_number); EXPORT_PER_CPU_SYMBOL(cpu_number); =20 -DEFINE_PER_CPU_READ_MOSTLY(unsigned long, this_cpu_off); +DEFINE_PER_CPU_CACHE_HOT(unsigned long, this_cpu_off); EXPORT_PER_CPU_SYMBOL(this_cpu_off); =20 unsigned long __per_cpu_offset[NR_CPUS] __ro_after_init; --=20 2.48.1