From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C0C6307486 for ; Wed, 24 Sep 2025 15:00:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726011; cv=none; b=p22+82HXKpXXHEBWCWOtcZaNCmRaSuHzw7jfSFGcl1tP4d/+IoP1VsujZR011zeBPUcy3Gayu37fv/be+IQMeM6WucPNRqRku7Ve8w23Y3fs6fiZKu7IrKtCe1JfHvYq3t3lZxt0myE7cMuLW8gW9o4Uo8hcDO2wu35URH9CLqk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726011; c=relaxed/simple; bh=CPBxBC4PANAWEu0yh8A8UOXRCzjxsDvq4N6WzMLVkto=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PgLRg6xHQC4h77/HrOel5m6gtqPa/UjK2SF0m5mYgZoTXECcSaz8OBa7STf+WhExJ9vCcN/jLcmgP4OFC+tSJ3/fLgMH0ahamehxGMvuN7SvDkxT7hOd7yFex4YIbf0rEMY+QUug5Qv84QGheXhQ/FAJTElUaK+xgjI8FCfqL7s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=q3i9fctR; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="q3i9fctR" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3ecdb10a612so2511077f8f.2 for ; Wed, 24 Sep 2025 08:00:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726008; x=1759330808; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RoJVCKwzHTvI9L15TyOM+QFTR+4mknbbnsqq6MMe4q4=; b=q3i9fctRbRplmi8BDCmGl4Chro62AauockN0mrUaFlSnFs7JUsxklkZ3vCNGQmynIB Jsz0AbNyfSScu6mtlROblAwILmsoL9MAYwxl/eAYsEUMonkouaMvaYK6EaWaaNjwte4b 8VtHzsStsnb6B8Ae7v2snRSRFW4LwbJ0p7B23A4HznSedH2TR02hE/gvZFw3HwgX+9oT bCAv/5pFXUBIKyNmjp8b4FGHTuN4Lea/4OUgUIQykNhoH4HUuSmiUVrVJ+kvmImJ9+cJ SuyjDA5bgeIFD+OwbcVRenjU7iuM/JxkEDOj6qe8kMAvBZ1OlKhM8PtXf6sM7EKJMYmD +y7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726008; x=1759330808; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RoJVCKwzHTvI9L15TyOM+QFTR+4mknbbnsqq6MMe4q4=; b=CN+1hX3Je+mMlCZFofwew/dKCHEZ2+HU3AOpjhxmB9cLxcZG1tyNFaFUrnpVCuW7hU sAaFnRmOba2AtQ5j86YIctzofOPVO+Irg2XrBl1I1Vx8tVk4+FAPt1X2YzWUID4jK9q9 pPyukNPZWtAgeIlISPInTSx1UyXXLJ26eRt9zu/ukgjxXB2KgxHMogJxcbfdPt+F0j4N 56nSTupsUbtHmVZEFhTKgELH65vNg57uSGuAPKZDaycd3td8S7obZiv2TDRdWuLuFLBw lc1boWJ+HsS5bx4rEgjbB1bIjFO1HyaCaq/UtUjZN/9BdBmJLczs6/4SWL4z/3FjApzS u1Sg== X-Forwarded-Encrypted: i=1; AJvYcCUG7AgeWvTjzGZTyT4QmffnmKCszjcRX9ayzt2lFdOe+R1L4MbXl1vqo7s7bg4Zuh7oW7cQjvM+io6llW0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw7DVwVTX6j3prz1Q5NtCezozMLxCLqYlemyg/RUxC7VBtFROaz 1LDQhsgyUHx9ZOKNqIQH1XwlyXpa8ZQ9BGsHn3gAzCGzr0agRcV3pj1LMmQKGszcOwo1Y7O1rj4 BPrANeiCFSkadMw== X-Google-Smtp-Source: AGHT+IFYCM81xG8NnhOZnthImfsZs9y4y2Mpes3SZt0+UDE3H8DpBmBpiCVfYNrs7AMA4PJb8IINbQbGi/I5sw== X-Received: from wmbej13.prod.google.com ([2002:a05:600c:3e8d:b0:46e:2291:a3a4]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2484:b0:405:1925:4972 with SMTP id ffacd0b85a97d-40e46515005mr243422f8f.1.1758726007585; Wed, 24 Sep 2025 08:00:07 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:36 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-1-2d861768041f@google.com> Subject: [PATCH 01/21] x86/mm/asi: Add CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This long awkward name is for consistency with CONFIG_MITIGATION_PAGE_TABLE_ISOLATION. In the short term, there isn't much arch code. In the medium term, it will mostly be x86 code. So, put the code where it will need to go instead of just having to move it soon. In the long term, it should probably include other archs too, so things should be as arch-specific as necessary, but not more so. Follow the proposal by Mike Rapoport[0]: a generic header includes NOP stubs for ASI definitions. If CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION is defined then the asm/ tree must have asi.h, and that gets included instead of the stubs. Signed-off-by: Brendan Jackman --- arch/Kconfig | 13 +++++++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/asi.h | 5 +++++ include/linux/asi.h | 10 ++++++++++ 4 files changed, 29 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index bae871976d36f7b6b2af0be40a067ca2b3fd3d14..ad99637630406e5a484173f5207= bbd5a64b2bf1f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -17,6 +17,19 @@ config CPU_MITIGATIONS def_bool y endif =20 +config ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION + bool + +config MITIGATION_ADDRESS_SPACE_ISOLATION + bool "Allow code to run with a reduced kernel address space" + default n + depends on ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION + help + This feature provides the ability to run some kernel code + with a reduced kernel address space. This can be used to + mitigate some speculative execution attacks. + + ASI is not yet ready for use. # # Selected by architectures that need custom DMA operations for e.g. legacy # IOMMUs not handled by dma-iommu. Drivers must never select this symbol. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 1fd698311bc1dba134a8e14dd551d2390e752cda..cb874c3857cf443c6235e05bc3f= 070b0ea2686f0 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -38,6 +38,7 @@ config X86_64 select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE select ACPI_MRRM if ACPI + select ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION =20 config FORCE_DYNAMIC_FTRACE def_bool y diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..53acdf22fe33efc6ccedbae52b2= 62a904868459a --- /dev/null +++ b/arch/x86/include/asm/asi.h @@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ASI_H +#define _ASM_X86_ASI_H + +#endif /* _ASM_X86_ASI_H */ diff --git a/include/linux/asi.h b/include/linux/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..ef640c8e79369a9ada2881067f0= c1d78093293f7 --- /dev/null +++ b/include/linux/asi.h @@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _INCLUDE_ASI_H +#define _INCLUDE_ASI_H + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#include +#else + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +#endif /* _INCLUDE_ASI_H */ --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1E3BB30AAC2 for ; Wed, 24 Sep 2025 15:00:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726013; cv=none; b=hXFdsYHde+3+YMC87qtHRkQPp8pneLRY3I3iZANjl1OoCzJjr0nOMyqIPvCql+m0eTP0+xlFHyWwOjHegN0H22SkUP3YDAGXgScdQtx9aXalCl59Yt59yglM51Uo6S3iT/iCishVbWzvGIF+KCC+Jw6qtbVLkINKT35l3hyAyNY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726013; c=relaxed/simple; bh=Tf59i56w6rf3fyuV0YtKRnkamzqq05gjx3hewdkc3Eo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Zi+G6LWmh6IL2d1JzyfTWSOL5X4+XuHV1PW54eL6TXbW63lQIqa+cRxU3bX9TmOSK9Ge5rh+kKWQRP/QG5BFgIFFTDGKUA48WjS8Hisdz2RnI1HGMPGvNEkS6/Bie2FB9mLfum+x5nKL2NWyYQqG7BsHoVmnfdNiRJP3W6wVuL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FJyramqS; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FJyramqS" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-b2c76083936so389192966b.2 for ; Wed, 24 Sep 2025 08:00:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726009; x=1759330809; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5r90z2IR0Gemmlb3xWRzXWwMPPH2fiEvwZqfzBajcPU=; b=FJyramqSxigCS2oVgxj7qtudCJeiFOX9jhnTAxiugedDUZgUqLRi/COzQFNdljLrAB Y4mXOedqxnMdqqPMOJE/7AELv+59WkzSH/39bXJpZklteQdP9cIhsyHqJdi450/HQ1b6 ZXumyrXDQqTaSwzZJbJrnEpxXZxNPrUHsY+9qYHXjk5AQlklwikWW+tLPSAmkOAL6Frl jOlNjPyjdOFBIAcCn/uZsFhl2vrU3N8HM3jp6p/HiELgrwHZhGi68Ce2E/KGICrMoAB2 6d3S9oLOrQN8FByzrkhLuy3+s+GGn9Deod9quEnRuAKWL9hMI7ISTmZYdY+O3CAAAByB USmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726009; x=1759330809; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5r90z2IR0Gemmlb3xWRzXWwMPPH2fiEvwZqfzBajcPU=; b=wusXfll5wUkFllLqVmz4x6BrJUHFe2VhzC0LqwFPxShIEsDcKKXHk+RnF0kSzWJ39s s8L2ObPVu1QsUIDZijaSGWIiIWgZpMP+NIr8RKe2Pv6t4BQpV3nbX9kGck/egep2BUu8 lTNFOhjxeLcXbCG5eEiMy6etTmhJnjNUuOtLWjiovmOxavb3x6Ij9Fr29MCUl27qiWvg u/w8oNrGaPphYxw65pz4u6LwW/VWJylmzmY9b+99r228T4dgVmomW77GWvLXjwd5oYlx SraxCLA3xThnxlAAAU6ARasfswQBebrl3wwNxRgGZDicuKyQDO8YL1D0Bl06KOpYI9oD 44ig== X-Forwarded-Encrypted: i=1; AJvYcCUz+G9LSnlb9rQpoK8nYFi1clkaouyro+0gtBnB380CtSFNxrH7JHY71EgaYosJMnPpes7vcZtUgCObdIk=@vger.kernel.org X-Gm-Message-State: AOJu0YzLrzkndLjozJQ0ivqZ0UA9kU0AX2SyzCMiZc5xYcannwCmVDiM P9sqtw7Xfy42d3VbLv5BgF4BHSQLwW5euFNdlyWtRJE3GP/jwziOtL3HJLNKzuYsvpsD3HjRRgN mkJdd1kqV9uptcw== X-Google-Smtp-Source: AGHT+IHX44fYosJyVsS9pPKUxPF4QsPRR2phTVeYFYZ8rkhn5hEuqP4aCDJC7cWbgd2p6O99Y3BBHqDWpl1tVg== X-Received: from ejcti11.prod.google.com ([2002:a17:907:c20b:b0:b2c:9261:eb0f]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:86a6:b0:b28:6440:a85c with SMTP id a640c23a62f3a-b34bc9720a6mr11837866b.50.1758726009211; Wed, 24 Sep 2025 08:00:09 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:37 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-2-2d861768041f@google.com> Subject: [PATCH 02/21] x86/mm/asi: add X86_FEATURE_ASI and asi= From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Add a CPU feature to enable ASI, and a command-line flag to enable that feature. At present, the feature doesn't do anything, but adding it early helps to avoid unnecessary code churn later. The cmdline arg will eventually need an "auto" behaviour, but since this would be equivalent to "off", don't define it yet. Just define what's necessary to be able to test the code. Co-developed-by: Junaid Shahid Signed-off-by: Junaid Shahid Co-developed-by: Yosry Ahmed Signed-off-by: Yosry Ahmed Signed-off-by: Brendan Jackman --- Documentation/admin-guide/kernel-parameters.txt | 8 +++++++ arch/x86/include/asm/asi.h | 10 +++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/mm/Makefile | 1 + arch/x86/mm/asi.c | 28 +++++++++++++++++++++= ++++ arch/x86/mm/init.c | 3 +++ include/linux/asi.h | 5 +++++ 7 files changed, 56 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentatio= n/admin-guide/kernel-parameters.txt index 6c42061ca20e581b5192b66c6f25aba38d4f8ff8..9b8330fc1fe31721af39b08b58b= 729ced78ba803 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -5324,6 +5324,14 @@ =20 Not specifying this option is equivalent to pti=3Dauto. =20 + asi=3D [X86-64] Control Address Space Isolation (ASI), a + technology for mitigating CPU vulnerabilities. ASI is + not yet ready to provide security guarantees but can be + enabled for evaluation. + + on - unconditionally enable + off - unconditionally disable + pty.legacy_count=3D [KNL] Number of legacy pty's. Overwrites compiled-in default number. diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 53acdf22fe33efc6ccedbae52b262a904868459a..32a4c04c4be0f6f425c7cbcff4c= 58f1827a4b4c4 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -2,4 +2,14 @@ #ifndef _ASM_X86_ASI_H #define _ASM_X86_ASI_H =20 +#include + +void asi_check_boottime_disable(void); + +/* Helper for generic code. Arch code just uses cpu_feature_enabled(). */ +static inline bool asi_enabled_static(void) +{ + return cpu_feature_enabled(X86_FEATURE_ASI); +} + #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpuf= eatures.h index 4091a776e37aaed67ca93b0a0cd23cc25dbc33d4..3eee24a4cabf3b2131c34596236= d8bc8eec05b3b 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -499,6 +499,7 @@ #define X86_FEATURE_IBPB_EXIT_TO_USER (21*32+14) /* Use IBPB on exit-to-us= erspace, see VMSCAPE bug */ #define X86_FEATURE_ABMC (21*32+15) /* Assignable Bandwidth Monitoring Co= unters */ #define X86_FEATURE_MSR_IMM (21*32+16) /* MSR immediate form instructions= */ +#define X86_FEATURE_ASI (21*32+17) /* Kernel Address Space Isolation */ =20 /* * BUG word(s) diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 5b9908f13dcfd092897f3778ee56ea4d45bdb868..5ecbff70964f61a903ac96cec37= 36a7cec1221fd 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -52,6 +52,7 @@ obj-$(CONFIG_ACPI_NUMA) +=3D srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) +=3D pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) +=3D kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) +=3D pti.o +obj-$(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION) +=3D asi.o =20 obj-$(CONFIG_X86_MEM_ENCRYPT) +=3D mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) +=3D mem_encrypt_amd.o diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c new file mode 100644 index 0000000000000000000000000000000000000000..8c907f3c84f43f66e412ecbfa99= e67390d31a66f --- /dev/null +++ b/arch/x86/mm/asi.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include + +#include +#include + +void __init asi_check_boottime_disable(void) +{ + bool enabled =3D false; + char arg[4]; + int ret; + + ret =3D cmdline_find_option(boot_command_line, "asi", arg, sizeof(arg)); + if (ret =3D=3D 3 && !strncmp(arg, "off", 3)) { + enabled =3D false; + pr_info("ASI explicitly disabled by kernel cmdline.\n"); + } else if (ret =3D=3D 2 && !strncmp(arg, "on", 2)) { + enabled =3D true; + pr_info("ASI enabled.\n"); + } else if (ret) { + pr_err("Unknown asi=3D flag '%s', try 'off' or 'on'\n", arg); + } + + if (enabled) + setup_force_cpu_cap(X86_FEATURE_ASI); +} diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 8bf6ad4b9400e7a04e9dc4e341e20a4a67ddb7ab..b877a41fc291284eb271ebe764a= 52730d51da3fc 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -1,3 +1,5 @@ +// SPDX-License-Identifier: GPL-2.0 +#include #include #include #include @@ -761,6 +763,7 @@ void __init init_mem_mapping(void) unsigned long end; =20 pti_check_boottime_disable(); + asi_check_boottime_disable(); probe_page_size_mask(); setup_pcid(); =20 diff --git a/include/linux/asi.h b/include/linux/asi.h index ef640c8e79369a9ada2881067f0c1d78093293f7..1832feb1b14d63f05bbfa3f87dd= 07753338ed70b 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -6,5 +6,10 @@ #include #else =20 +#include + +static inline void asi_check_boottime_disable(void) { } +static inline bool asi_enabled_static(void) { return false; } + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1336830AAC4 for ; Wed, 24 Sep 2025 15:00:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726013; cv=none; b=oY9IXi5IvUlYCaWWQQ3af+bs/zrPiBIDlPIcY6mgc63GsDlX7I/NDdq4zV0l1t3KkKfDyj7IEBO5+wmrTm962cJkUFRrM+JWII3sbpvy4G5SvBqTsQpiQGVGU7GRlJ1SEPWreJoVWc1X0O9Erqwa/aIKvjKrLX6s0amy7qu+ho0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726013; c=relaxed/simple; bh=EYLPQH5oehnbYPc5sxBDHhm4ClGMnbLmbBqQann3GQY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jMh9crv5/cHmvZymftbGtwLRH6LY7jZnfUdJL/ryhD4egIxrmMNHbgvcbr8fSE8lGlmixqNO3d3abtlNThg3bSgJynx+nD9SmSPos3cOQY+U649cEFQX/JrXt7QN2MpXNDp48q2aiqm0vSelAN+MPOHqJ9YYlwuxlbBO/iEiego= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vx0UKi/A; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vx0UKi/A" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3ee888281c3so3986265f8f.3 for ; Wed, 24 Sep 2025 08:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726010; x=1759330810; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9sQ+0/NTgmoYJ7zvD18U+frXGZZeEnJJzFJ7cE/2yig=; b=vx0UKi/AOg4yDr0YyRqDSrnXMTTtLjGNWG1r79ZqqnnpTH7CmNrSPWvsqN8PtPaGe8 4JjQEQ8lVj1rF87HCv2ndCjY5V+um83lMKoZgb6Q/d4a1bYnFYRaC67EtF4dy/9uiaHS 1d4+ATfBiTIYiKcqSFetp3d9afGyvFy5yIvzJb1566amx6N0yz9hk1F858uo91sDsYAj zU0LSZsP6VVGL4ywzgx/7UROLc42SpPDppp4a6qXHJt9mfOuz9f5R5fSD7pabRma7ynX aLgvTytEYQNZy0VNyPIBK9rRKJn1z0ukNfC6p5+tI9MZUcOazzwXIpg4dmobUxoSYvZB d4ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726010; x=1759330810; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9sQ+0/NTgmoYJ7zvD18U+frXGZZeEnJJzFJ7cE/2yig=; b=GyyI/4gwoSy40uetIzZtYlJa+cBRedwTgVQjOugunzaxgSz/ph4olOrORgacxPygRb twAUqWTJ8FJ8NK3YwOHwl1/k5eR8HP7gtjLfyPdz2YRCf+fGWIlptTwZ4OCFzAHP9UVK xruR1Fb72t77jRsamTZ/AcoCODJ3NaYDiBCNRwr/qczX9Novqr0Mys2NPcLc8iN3xctZ utM0ktdJxTU2rz/RiFqJ5OqDBhbAHSD3pKhxCXrN/Rf48AmCxvincoQXJ2tbhylL59au ju6xfdMaY9VVsi0MhkZXjLzbSlBAeUPH7UfR+dpmdNKh3gM0KoqbEk+Ggm1QqvKYDGsu vEXA== X-Forwarded-Encrypted: i=1; AJvYcCUgS843G+I5fU2eC4CqxRhEHekba9pQ5NwioUheEnhPZVOl2hGKzpTI9ZWoZQFLUgmYuwIhMVspeBAsjXs=@vger.kernel.org X-Gm-Message-State: AOJu0YwvLwMnxVtFN9SUN15SBdEi3tRO9SzH+bSbfY9a3ozDYAc6Neq+ PFaMmL+BMYttKq2eVm0hucBkKDGG+GAOHjD+HaGRx4/FGYr1ZlnLNFkAsTQAHw2m9m9s+UekGXT OBAsoNEm0fKdpAw== X-Google-Smtp-Source: AGHT+IFqsEQb7cIi2yGxxJ+MfZzjIIfAO9RpfOWtkftlHXz5t7D+GW6T5K74X995LeYOztbN6pZKuqk/SB6tcQ== X-Received: from wrty13.prod.google.com ([2002:a5d:614d:0:b0:402:ec26:76f8]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5f85:0:b0:3e9:ad34:2b2e with SMTP id ffacd0b85a97d-40e4cb6e211mr232236f8f.46.1758726010290; Wed, 24 Sep 2025 08:00:10 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:38 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-3-2d861768041f@google.com> Subject: [PATCH 03/21] x86/mm: factor out phys_pgd_init() From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable __kernel_physical_mapping_init() will soon need to work on multiple PGDs, so factor out something similar to phys_p4d_init() and friends, which takes the base of the PGD as an argument. Signed-off-by: Brendan Jackman --- arch/x86/mm/init_64.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 0e4270e20fadb578c7fd6bf5c5e4762027c36c45..e98e85cf15f42db669696ba8195= d8fc633351b26 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -741,21 +741,20 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, u= nsigned long paddr_end, } =20 static unsigned long __meminit -__kernel_physical_mapping_init(unsigned long paddr_start, - unsigned long paddr_end, - unsigned long page_size_mask, - pgprot_t prot, bool init) +phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_start, unsigned long pa= ddr_end, + unsigned long page_size_mask, pgprot_t prot, bool init, bool *pgd_c= hanged) { - bool pgd_changed =3D false; unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; =20 + *pgd_changed =3D false; + paddr_last =3D paddr_end; vaddr =3D (unsigned long)__va(paddr_start); vaddr_end =3D (unsigned long)__va(paddr_end); vaddr_start =3D vaddr; =20 for (; vaddr < vaddr_end; vaddr =3D vaddr_next) { - pgd_t *pgd =3D pgd_offset_k(vaddr); + pgd_t *pgd =3D pgd_offset_pgd(pgd_page, vaddr); p4d_t *p4d; =20 vaddr_next =3D (vaddr & PGDIR_MASK) + PGDIR_SIZE; @@ -781,15 +780,29 @@ __kernel_physical_mapping_init(unsigned long paddr_st= art, (pud_t *) p4d, init); =20 spin_unlock(&init_mm.page_table_lock); - pgd_changed =3D true; + *pgd_changed =3D true; } =20 - if (pgd_changed) - sync_global_pgds(vaddr_start, vaddr_end - 1); - return paddr_last; } =20 +static unsigned long __meminit +__kernel_physical_mapping_init(unsigned long paddr_start, + unsigned long paddr_end, + unsigned long page_size_mask, + pgprot_t prot, bool init) +{ + bool pgd_changed; + unsigned long paddr_last; + + paddr_last =3D phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_si= ze_mask, + prot, init, &pgd_changed); + if (pgd_changed) + sync_global_pgds((unsigned long)__va(paddr_start), + (unsigned long)__va(paddr_end) - 1); + + return paddr_last; +} =20 /* * Create page table mapping for the physical memory for specific physical --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5ED0730BF63 for ; Wed, 24 Sep 2025 15:00:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726016; cv=none; b=RQgv8tcMsknSyGJryFWTtnU97av1sOawhToZ/0osHE4YYQh0y395RQMMb6jHK7aY99i7r++ZiuMtQzoLBcO4TrBl1l3dIvcL3XVCb8cfXzsqmmu5szI+94wU5xQMwaOmSskyhGbQeL+GeT9iCoapBuvJw8n7LDVJJqZZa5abFWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726016; c=relaxed/simple; bh=iVEfUkagpTd8ShTdeYE6IfpxRs8e6FjbaJNwIS6zNps=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YAPXh0WJnVnwnadU8e6dYLoglcRK4W1Vsh+4X7mk0pTHv8ebtZeTSH++yZNdH4MVRp4ZuilJ8UuMq9lhQAp8y5jjtnk3I9NAwgHRuHV+HA4nX6TI+SmGGiRdqHlDruPbbtc0gg41HCOWgHcd9ESRXywLkvJPDB6qyrQpDQEzGQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VKRkWjD4; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VKRkWjD4" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-62fce1f3fa8so4716688a12.3 for ; Wed, 24 Sep 2025 08:00:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726013; x=1759330813; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4lMKL0dag2cTvl5zrGDAXdNMfc9wzwq11ZbZ3BSylYk=; b=VKRkWjD4bdD+WZ5XD8bUleo3jJN1gZRXPTB78N/BD5UX3QzuUyQ4cPROrBoEiYDl3U nbZNTPwNU6j9mkXFK7YfhSUwiSs4vqUU8As7d52JV0LVB4UGx/0vO0KvyGeArkgh+c8c IPe+ktbffXMi00/TWrLmvgMUmIzmZCG53qbT1czJe1MXnQrBGtlnTwWHO/iyGizoOdA5 vKSNW2z5VQpi4tfsuHu0iZcnRl1TcBz1JSbfq2b1PTYr1f+2AL0VJUzrEEcZ49J7qlG6 MXAboNgPWv6u0PhQjAZTBOIplFWxESZTlcYv4Sezv+lzxv8vmj/8dV9MsGprnCkfLZK+ ry4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726013; x=1759330813; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4lMKL0dag2cTvl5zrGDAXdNMfc9wzwq11ZbZ3BSylYk=; b=soo4jQwVRm/ZLDbGVs9KfgthQG+42Zx17Ls6MIc5573YFPFw2d/QJP/rzu6ZlMy6Jk Iew0nuqEnIc0xhSAc9zY1CwfuOvJZvX8YbN12BuTHDhIu62Z6p+Z9aUz0/0n2I3wPZCT k2RhqiC8ulnnq0XTbel4HpZ+Lz5mafMwvcEcf7o1GX1U9JbFuPChFMQ0X8zGjNJBQ5P2 MO6NApiUHHy2QDVzPAlu5n26S4xuiSHCmvVu/EbXo9nE85WbyyJnOdomBdsz2qgQ6XDv 5Hq8sx59nIzIJnMJNt/vXuwa7gGJ3O0pM3eOTSO8uu5lMD7VOiCGVfCXHb3U5/oYYh2F vW1g== X-Forwarded-Encrypted: i=1; AJvYcCX7cN4TdtbUlpvbTWKTMfT9prMSGcY/QIWy4l+sEhdi+TGaBJt23h/9sq/nWSo/EkjIUl1zZwpnQ+O6uuc=@vger.kernel.org X-Gm-Message-State: AOJu0Yx/o0CqK1GuL2dh5OINjedS+me0G3V+h3R77dYZmcuw+HGzgtaU rnzsXBpPHt3aLoyE/yta7HWNq7gLfhW8HDdQVMzDBQMQFz9DaNj+3xqv3LafQ4Y7vsnuTM16C3m hvpIe3PaApKTxxw== X-Google-Smtp-Source: AGHT+IFVSQFxUV/mTSePnmrlINV4I944f6eUP8KCpi3pssIRknrIWSkYfW/GfKVJFIh3a3hgc0TE+MDy6SWe2w== X-Received: from edbm26.prod.google.com ([2002:a50:999a:0:b0:634:9b3d:9588]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:1d24:b0:634:5722:cc3f with SMTP id 4fb4d7f45d1cf-6346779876dmr4825674a12.16.1758726011350; Wed, 24 Sep 2025 08:00:11 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:39 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-4-2d861768041f@google.com> Subject: [PATCH 04/21] x86/mm/asi: set up asi_nonsensitive_pgd From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Create the initial shared pagetable to hold all the mappings that will be shared among ASI domains. Mirror the physmap into the ASI pagetables, but with a maximum granularity that's guaranteed to allow changing pageblock sensitivity without having to allocate pagetables, and with everything as non-present. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 4 ++++ arch/x86/mm/asi.c | 19 +++++++++++++++++++ arch/x86/mm/init.c | 2 ++ arch/x86/mm/init_64.c | 25 +++++++++++++++++++++++-- include/linux/asi.h | 4 ++++ init/main.c | 1 + 6 files changed, 53 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 32a4c04c4be0f6f425c7cbcff4c58f1827a4b4c4..85062f2a23e127c736a92bb0d49= e54f6fdcc2a5b 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -12,4 +12,8 @@ static inline bool asi_enabled_static(void) return cpu_feature_enabled(X86_FEATURE_ASI); } =20 +void asi_init(void); + +extern pgd_t *asi_nonsensitive_pgd; + #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 8c907f3c84f43f66e412ecbfa99e67390d31a66f..7225f6aec936eedf98cd263d791= dd62263d62575 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,11 +1,20 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include =20 #include #include =20 +#include "mm_internal.h" + +/* + * This is a bit like init_mm.pgd, it holds mappings shared among all ASI + * domains. + */ +pgd_t *asi_nonsensitive_pgd; + void __init asi_check_boottime_disable(void) { bool enabled =3D false; @@ -26,3 +35,13 @@ void __init asi_check_boottime_disable(void) if (enabled) setup_force_cpu_cap(X86_FEATURE_ASI); } + +void __init asi_init(void) +{ + if (!cpu_feature_enabled(X86_FEATURE_ASI)) + return; + + asi_nonsensitive_pgd =3D alloc_low_page(); + if (WARN_ON(!asi_nonsensitive_pgd)) + setup_clear_cpu_cap(X86_FEATURE_ASI); +} diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index b877a41fc291284eb271ebe764a52730d51da3fc..8fd34475af7ccd49d0124e13a87= 342d3bfef3e05 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -773,6 +773,8 @@ void __init init_mem_mapping(void) end =3D max_low_pfn << PAGE_SHIFT; #endif =20 + asi_init(); + /* the ISA range is always mapped regardless of memory holes */ init_memory_mapping(0, ISA_END_ADDRESS, PAGE_KERNEL); =20 diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index e98e85cf15f42db669696ba8195d8fc633351b26..7e0471d46767c63ceade479ae0d= 1bf738f14904a 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -7,6 +7,7 @@ * Copyright (C) 2002,2003 Andi Kleen */ =20 +#include #include #include #include @@ -746,7 +747,8 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_star= t, unsigned long paddr_en { unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; =20 - *pgd_changed =3D false; + if (pgd_changed) + *pgd_changed =3D false; =20 paddr_last =3D paddr_end; vaddr =3D (unsigned long)__va(paddr_start); @@ -780,7 +782,8 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_star= t, unsigned long paddr_en (pud_t *) p4d, init); =20 spin_unlock(&init_mm.page_table_lock); - *pgd_changed =3D true; + if (pgd_changed) + *pgd_changed =3D true; } =20 return paddr_last; @@ -797,6 +800,24 @@ __kernel_physical_mapping_init(unsigned long paddr_sta= rt, =20 paddr_last =3D phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_si= ze_mask, prot, init, &pgd_changed); + + /* + * Set up ASI's unrestricted physmap. This needs to mapped at minimum 2M + * size so that regions can be mapped and unmapped at pageblock + * granularity without requiring allocations. + */ + if (asi_nonsensitive_pgd) { + /* + * Since most memory is expected to end up sensitive, start with + * everything unmapped in this pagetable. + */ + pgprot_t prot_np =3D __pgprot(pgprot_val(prot) & ~_PAGE_PRESENT); + + VM_BUG_ON((PAGE_SHIFT + pageblock_order) < page_level_shift(PG_LEVEL_2M)= ); + phys_pgd_init(asi_nonsensitive_pgd, paddr_start, paddr_end, 1 << PG_LEVE= L_2M, + prot_np, init, NULL); + } + if (pgd_changed) sync_global_pgds((unsigned long)__va(paddr_start), (unsigned long)__va(paddr_end) - 1); diff --git a/include/linux/asi.h b/include/linux/asi.h index 1832feb1b14d63f05bbfa3f87dd07753338ed70b..cc4bc957274dbf92ce5bf6185a4= 18d0a8d1b7748 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -11,5 +11,9 @@ static inline void asi_check_boottime_disable(void) { } static inline bool asi_enabled_static(void) { return false; } =20 +#define asi_nonsensitive_pgd NULL + +static inline void asi_init(void) { }; + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ diff --git a/init/main.c b/init/main.c index 07a3116811c5d72cbab48410493b3d0f89d1f1b2..0ec230ba123613c89c4dfbede27= e0441207b2f88 100644 --- a/init/main.c +++ b/init/main.c @@ -12,6 +12,7 @@ =20 #define DEBUG /* Enable initcall_debug */ =20 +#include #include #include #include --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A5C430BF7D for ; Wed, 24 Sep 2025 15:00:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726016; cv=none; b=gRvKKHzpnwnk7AThQHmfEMluvIKJ8aisXaKtFv8NWLN/IdBiAAMTJuwM7v0M/xR9ip54aMiDvcCzBGSPXvWZplaA4YVe5Cx2/lrQvueGKGbpBYNg+Af8Vau8JF3qhgyx3RQvWvPQGlc0NnzUH/id/so+5/WGqFUBl5IWuXkuRXY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726016; c=relaxed/simple; bh=DcKwexSwfJbBE/7qSmcz3XCIn9Hk5DZnrMstN2LvKSA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XiMU4Ih1YKC6O7cqQQtIoUvKiCJKiHai5yP+CG4GE77T2MTAf8bfdvzH+duaLRzi1rkhxx3kznOg1sTifycc3+i4En8K8l0teXlPxo2jboNk1xC8AX9J011fQk0SJKZqs1Xy2/rvXBtNTWvRxV07H8EFF2FVnW73h10YBe1+ogQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YL2r0N9D; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YL2r0N9D" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46d8ef3526dso21388745e9.1 for ; Wed, 24 Sep 2025 08:00:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726013; x=1759330813; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Z2K58t/vrOefSYqdR7POZfHM/E5LlRhe2fvwwEfqIfU=; b=YL2r0N9DtZ3IWR5HSqH7mZ/KwJWoOWN+Qh0H7gfr4kwacukGaKoWh6F391EXMfUQj/ 4JtiApQMlxM35VkzYPRr/QCAw85/UlihBiOKWdgNmJ8jLaZoivsGrvzB3VpQMfWHWHb8 fKIb28r/UfUVCHZk9uesxPyT3BgF621BGQ2mLRugM1rYqjkSsudtI0W2MuL62kIcikCt DrgGh5tjxYKkB0Tw6q6Wzpzadc6PFZSbYZoC0D00fFSswXEagiNynx7fGuhxAibXOAar eD+Yy/Z+JWrg63TEHImWve2odTW1DLLiYbFMhhjg0/WhEOM3N3odt+5iIpNNLhwqJAPu kHmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726013; x=1759330813; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Z2K58t/vrOefSYqdR7POZfHM/E5LlRhe2fvwwEfqIfU=; b=Q4/8xEY0pzIXOGqBlwvOcjG87BpueOuBeC2ixw9xmDNzvzXQ28fCX0wZ3yZ1iAmGpi f4uO7tVQyVTQHZu+ZBpkZDsMHqUGWnqeX4ZnwfOz8mrqrDS7AOcCEhda6W8y+82dw0z1 vziXlqgUn2BpGUbJ/+Bv/GQhcDQVt3H8DrAWAHgiS6vDwgy7tsj0w0cX3jE6Q8LcJ8eE FUA+xdNxGKOhLOGQk1zD/j2Ka0a/pCbU09weshFAcz6qpHpOXY6QrMXd6RsF9tqcPnmi 2xfeongRUWb7Ets9ZbHg9zvrEJsaMTOKFIdpYetEso1F2cGt47EjWP0oKJDI3vwGvu9Y CijQ== X-Forwarded-Encrypted: i=1; AJvYcCWcVPurjqx7CpFHzwNWZQajRBv0dk0K3nJzL3PJLkNk0oKQKI4e6BNqdnFMiCfXBeHi/ohm+bzOFvsh/a0=@vger.kernel.org X-Gm-Message-State: AOJu0YwgOY6vufRc5pw/gXF8W1F/vDijwOLBTgiTUzlD7yVpL2VR9ykG FYVrRyHZKL0gjYJdE7NY5NSj+6EodwiLDOrrSbvmB8o1RKhkUaZl8tHO/+vtWNMBbs0jySZNANH JH71WWA5FZgMmbw== X-Google-Smtp-Source: AGHT+IFLbqLP4y9VzvHzeY0NK/ko0qJcgJDnQDoVhaZ1NLWwdjq/QRcEPRrnluk2M64Siud++fEyGmKNZKOPkA== X-Received: from wmqb8.prod.google.com ([2002:a05:600c:4e08:b0:45d:d60a:8d7e]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:609b:b0:45d:e4ff:b642 with SMTP id 5b1f17b1804b1-46e32a04a4emr1138645e9.25.1758726012269; Wed, 24 Sep 2025 08:00:12 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:40 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-5-2d861768041f@google.com> Subject: [PATCH 05/21] x86/mm/pat: mirror direct map changes to ASI From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable ASI has a separate PGD for the physmap, which needs to be kept in sync with the unrestricted physmap with respect to permissions. Since only the direct map is currently populated in that address space, just ignore everything else. Handling of holes in that map is left behaving the same as the unrestricted pagetables. Co-developed-by: Yosry Ahmed Signed-off-by: Yosry Ahmed Signed-off-by: Brendan Jackman --- arch/x86/mm/pat/set_memory.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index d2d54b8c4dbb04cf276d074ddee3ffde2f48e381..53c3ac0ba55d6b6992db6f6761f= fdfbd52bf3688 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -3,6 +3,7 @@ * Copyright 2002 Andi Kleen, SuSE Labs. * Thanks to Ben LaHaise for precious feedback. */ +#include #include #include #include @@ -1780,6 +1781,11 @@ static int populate_pgd(struct cpa_data *cpa, unsign= ed long addr) cpa->numpages =3D ret; return 0; } +static inline bool is_direct_map(unsigned long vaddr) +{ + return within(vaddr, PAGE_OFFSET, + PAGE_OFFSET + (max_pfn_mapped << PAGE_SHIFT)); +} =20 static int __cpa_process_fault(struct cpa_data *cpa, unsigned long vaddr, int primary) @@ -1808,8 +1814,7 @@ static int __cpa_process_fault(struct cpa_data *cpa, = unsigned long vaddr, * one virtual address page and its pfn. TBD: numpages can be set based * on the initial value and the level returned by lookup_address(). */ - if (within(vaddr, PAGE_OFFSET, - PAGE_OFFSET + (max_pfn_mapped << PAGE_SHIFT))) { + if (is_direct_map(vaddr)) { cpa->numpages =3D 1; cpa->pfn =3D __pa(vaddr) >> PAGE_SHIFT; return 0; @@ -1981,6 +1986,27 @@ static int cpa_process_alias(struct cpa_data *cpa) return 0; } =20 +/* + * Having updated the unrestricted PGD, reflect this change in the ASI + * restricted address space too. + */ +static inline int mirror_asi_direct_map(struct cpa_data *cpa, int primary) +{ + struct cpa_data asi_cpa =3D *cpa; + + if (!asi_enabled_static()) + return 0; + + /* Only need to do this for the real unrestricted direct map. */ + if ((cpa->pgd && cpa->pgd !=3D init_mm.pgd) || !is_direct_map(*cpa->vaddr= )) + return 0; + VM_WARN_ON_ONCE(!is_direct_map(*cpa->vaddr + (cpa->numpages * PAGE_SIZE))= ); + + asi_cpa.pgd =3D asi_nonsensitive_pgd; + asi_cpa.curpage =3D 0; + return __change_page_attr(cpa, primary); +} + static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary) { unsigned long numpages =3D cpa->numpages; @@ -2007,6 +2033,8 @@ static int __change_page_attr_set_clr(struct cpa_data= *cpa, int primary) if (!debug_pagealloc_enabled()) spin_lock(&cpa_lock); ret =3D __change_page_attr(cpa, primary); + if (!ret) + ret =3D mirror_asi_direct_map(cpa, primary); if (!debug_pagealloc_enabled()) spin_unlock(&cpa_lock); if (ret) --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01EFC30C60B for ; Wed, 24 Sep 2025 15:00:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; cv=none; b=CRmhhOdBMwjHZ1IXKs2jZADsjabDrC5vzv6rRdDr0KeNPdeOvFqMB+ya+AMOw3WsDlEttoqa5+fz9xwvvQ0f9DHyUZRwYVuI+/VtCLTyQ7Q5evT0A0NuiQjQX9/HadRY06p5WRIm0mfrIgu2l5nuw/VWikkEx+86K4eGJgSJIms= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; c=relaxed/simple; bh=L1VgNfoC71xNyfRLvBsWv8HKOJkfszOB/L2DYbQfD+0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AFKc9Vv3waHQxwP1D2RYrI28eBpruqbqk/dAUDDfK2tIL1L+8x7B3d1Tz8gReqLiSCehAYkI32C8s9AG6u+vBlkO5rY0/EiVIePtqxmXG/Hvyok+cGAToYJh2FUffk4lh4KUbRcWB0M48f4J1a3ay76rg2EljMfctmANBFuCK/I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=JnCeeov7; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JnCeeov7" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46cde0b2226so16296655e9.2 for ; Wed, 24 Sep 2025 08:00:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726014; x=1759330814; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=CX5Q5YH/plIL1ST5QoPko2JUmjdnIY0LBvORLR/qltw=; b=JnCeeov7+l02sCp1bnxwwkvYRbmLI6MSvypGUtX+3+JxbQLKd5KPf/pGMdxTWGgjXw MXAKDySriu+snT6NpKMuuTEJ1XcYoOuZVoTgnCeV05xzoC0e7xD8uMVNyw98nVnTG5Fo X/h1AFMe39VWpFFo6PVded0B+EznsReytc8kPH8u/JJzhx3Vr+NJorwmOSh9eu4d0sF4 Ktd9w1lu4okjy1ROBljkLMttIlTs4wOhIIZUH1OQWzY0AOrkpeDsDYnFEhtwXMXa8T0x vMaHuF+Y5mLhJgyD7vDspfPlzBJPuFXohy/X95FN3M/NIlcgMjJh5jBU60i/1vMQmz0K talA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726014; x=1759330814; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=CX5Q5YH/plIL1ST5QoPko2JUmjdnIY0LBvORLR/qltw=; b=HaEeaY3DuFwMZ4164kySPS4fhsrYh+v0ZiHjWJAIHLLO0EZ9XzWSIVIGUbIoCQGDDi 0I0JZNFlvwAyNDf4s3XFzp8t0xFCzwE2K/eL4JDBFtIlC/zlBlAfRkdztcMoAqUD+Hj/ clBUenRc6p9fOF8RaZAZgdGeMzbhoJM5IzZx1nYyggJU5oYIh3cVd4MYv06D1Ya982YY GGwIcY1V/qAQkewyuJZP7anhtN5THgob3Msmz44jRhajYwKkB06rYyCI/xA6RmZWRehH Qzn+zeGf/WnW4MUVwOlJ/WIFSVJ+PrwiPoTTQhodtHTTk0d7IjAZp+zgK24aHGRCKXYy HrEA== X-Forwarded-Encrypted: i=1; AJvYcCUiIGceLQzWqqrIOj8lyKwJ0aPIXyLmY5ZAyl4ioH6twtvjTy/HszqdLGHGXo9B8v8XZZOHtXGim/TeU40=@vger.kernel.org X-Gm-Message-State: AOJu0Yxz3u1rN0fjkfoSQW5SNVthZg44DnV9A8v96QwxQ9KHgGanGzrV UKD4xNx6gFbju6KmhS6iTFgNKeXUlrH6E0xqtEH9pWEEK9Ay4cFgFsp2ggmOaeqUUNDOLE8i8Ax sywPsMXaOXY4VgQ== X-Google-Smtp-Source: AGHT+IH1z5403qKBJ7o7GKoFaP/RUEDwuiU+FQ2BI25kf1Pv6Ap4eL3k4PLg1ZIY6q/Lrj8v/DOCur38GOqArA== X-Received: from wmbdr13.prod.google.com ([2002:a05:600c:608d:b0:45c:b682:883b]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:17d0:b0:45c:b53f:ad9 with SMTP id 5b1f17b1804b1-46e32a1919dmr978105e9.33.1758726013525; Wed, 24 Sep 2025 08:00:13 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:41 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-6-2d861768041f@google.com> Subject: [PATCH 06/21] mm/page_alloc: add __GFP_SENSITIVE and always set it From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable __GFP_SENSITIVE represents that a page should not be mapped into the ASI restricted address space. This is added as a GFP flag instead of via some contextual hint, because its presence is not ultimately expected to correspond to any such existing context. If necessary, it should be possible to instead achieve this optionality with something like __alloc_pages_sensitive(), but this would be much more invasive to the overall kernel. On startup, all pages are sensitive. Since there is currently no way to create nonsensitive pages, temporarily set the flag unconditionally at the top of the allocator. __GFP_SENSITIVE is also added to GFP_USER since that's the most important data that ASI needs to protect. Signed-off-by: Brendan Jackman --- include/linux/gfp_types.h | 15 ++++++++++++++- include/trace/events/mmflags.h | 1 + mm/page_alloc.c | 7 +++++++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f9053c701e24bdcf1dfe6afbf1278a2d..5147dbd53eafdccc32cfd506569= b04d5c082d1b2 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -58,6 +58,7 @@ enum { #ifdef CONFIG_SLAB_OBJ_EXT ___GFP_NO_OBJ_EXT_BIT, #endif + ___GFP_SENSITIVE_BIT, ___GFP_LAST_BIT }; =20 @@ -103,6 +104,11 @@ enum { #else #define ___GFP_NO_OBJ_EXT 0 #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ___GFP_SENSITIVE BIT(___GFP_SENSITIVE_BIT) +#else +#define ___GFP_SENSITIVE 0 +#endif =20 /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -299,6 +305,12 @@ enum { /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) =20 +/* + * Allocate sensitive memory, i.e. do not map it into ASI's restricted add= ress + * space. + */ +#define __GFP_SENSITIVE ((__force gfp_t)___GFP_SENSITIVE) + /* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) @@ -380,7 +392,8 @@ enum { #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM | __GFP_NOWARN) #define GFP_NOIO (__GFP_RECLAIM) #define GFP_NOFS (__GFP_RECLAIM | __GFP_IO) -#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL) +#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | \ + __GFP_HARDWALL | __GFP_SENSITIVE) #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 #define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index aa441f593e9a6b537d02189add91eb77bebc6a97..425385b7f073d05e9d660ad19cb= 7497f045adfb7 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -100,6 +100,7 @@ TRACE_DEFINE_ENUM(___GFP_LAST_BIT); gfpflag_string(GFP_DMA), \ gfpflag_string(GFP_DMA32), \ gfpflag_string(__GFP_RECLAIM), \ + gfpflag_string(__GFP_SENSITIVE), \ TRACE_GFP_FLAGS \ { 0, NULL } =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 600d9e981c23d75fdd4aec118e34f3f49d3de2e0..0d1c28decd57b4a5e250acc0efc= 41669b7f67f5b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5152,6 +5152,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, = unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac =3D { }; =20 + /* + * Temporary hack: Allocation of nonsensitive pages is not possible yet, + * allocate everything sensitive. The restricted address space is never + * actually entered yet so this is fine. + */ + gfp |=3D __GFP_SENSITIVE; + /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0200630C620 for ; Wed, 24 Sep 2025 15:00:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; cv=none; b=r7rNPyZ5ze8J+zdpqXocY5cZmrV4DLhfhtYGmQKzpH8uK9dk7KLMrpujLZadvpi7svB8p7ITHt7SluapNfusJQ1reyFDsLkWnShyTrH3zALasFFRVnxkjaKYlPIngTPLelBZK+QLnj7e0krhzAglhz4/n2iGt+Tg7zI5TGqBPKc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; c=relaxed/simple; bh=WAlqU5MUhsOddpJHILI64wsY53VfyKgPLsAmAN+Sb+U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NrPB9WG5ramvjWIIrfGB0gLHjdcq2iqdAQN0ANsXp0nDpCBVngG2u0dqXFStlL75TZOgTVwBnpMpmOZRu4M3fIRVR5CsFUFDwA3q5wD7E+uUqLQZdX4T+VMALvXCtDVoabM2lGqG7hpKY5qHB+h/0lKW+MAdB7KL5d8YSOdGZXM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SMla2dgR; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SMla2dgR" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45f2a1660fcso54922485e9.1 for ; Wed, 24 Sep 2025 08:00:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726015; x=1759330815; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=WTuaOxSTe3tk5TxQgOGSdVxjdMn4okuwwVefzm7d4xQ=; b=SMla2dgRZNa4sEWqHj8dm9tnFVf8J+ZiJGa3iGcCMSyPQgq3TBtiEtJhgk2SLhpFqM utLlGXSMdx0NjSX0R+gCwY8gODVcqRDz73+vWaFluczNGJpKHls+NdQ+yZHw4VKjYyHO 3RydyaajLnegbB0nR0WiZAP5ClFfTP3CgPGF4MZ7Ds/h8tZdTXui4/ugf8CkmMfPrb+m Z+TEsxExrWDLAe91Meeur7630VZ/HLKfqh91gp3wK26cxRjl4tns2uSOOdi5FzlwsopJ cgv7w7auUURspndp9WXDGf9xc8yiT9vmuZjc/TMZfrt/ILIuS0xHK2C/TUMnl/NoSYXZ igbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726015; x=1759330815; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WTuaOxSTe3tk5TxQgOGSdVxjdMn4okuwwVefzm7d4xQ=; b=UIFn2/GducBpyejgoYkAOK0DaJKLYnqyQeDEe6U8Q24c2WGLtE1buMRFAKwmp55CrQ YYN14Mj3QvtBPPq2hAbbMXxCfhCTWTfObHjRfd2in20QeOKT5KwWEsOTDU62K9DsahJf Vs9TMo81mR/NjXgLE6ArKjvGW73gOIgnD17MyrspTAcRNkckOqi07FqtwAwJJw9Jf5lo n8fgoPh/i/BzVYs4nGqR4DA10uxzj8rjGWdYjjKIciZV53zh+xiWSm6rHbsJeiX6DzSJ ryyF0AKykm+mmNs71fUBi5ix/hhJeZJdGb/rF+i14z5AdBE6Puvf+gu2bJGTT+Lyn85T GeNw== X-Forwarded-Encrypted: i=1; AJvYcCVisTX+ysWazmmKRv3JTudtYa7NjzTACLEIt1As8++Z2Dw1L7Y0+12q2ppw1N5abSTiBF/Esr27NB25rOM=@vger.kernel.org X-Gm-Message-State: AOJu0YyN6zVTyznyisdgSSyI1Eg1Wbr6Z1Q7ImAWNeV7gkZw4CcVQzD3 uU/iUzT1XhmZY+UrJktLcOAyHDBAQwXK86jAt0HphKMjUYKTzopp8jjojPoO63J53L7/yRuylsb JwFX19nKkYCVUkw== X-Google-Smtp-Source: AGHT+IGt9WJr9xVKvOK+GVvGrHodYbNw3zG+dtfOGTSv7cY+ZO1ThruN4S0CmXEkR5GVWAqvIrk4jx4s1l+ufA== X-Received: from wrvw4.prod.google.com ([2002:a5d:5444:0:b0:3eb:7d34:3d1d]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:5d07:0:b0:3fe:1d64:1871 with SMTP id ffacd0b85a97d-40e4911408fmr239735f8f.43.1758726014660; Wed, 24 Sep 2025 08:00:14 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:42 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-7-2d861768041f@google.com> Subject: [PATCH 07/21] mm: introduce for_each_free_list() From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Later patches will rearrange the free areas, but there are a couple of places that iterate over them with the assumption that they have the current structure. It seems ideally, code outside of mm should not be directly aware of struct free_area in the first place, but that awareness seems relatively harmless so just make the minimal change. Now instead of letting users manually iterate over the free lists, just provide a macro to do that. Then adopt that macro in a couple of places. Signed-off-by: Brendan Jackman --- include/linux/mmzone.h | 9 ++++++--- kernel/power/snapshot.c | 7 +++---- mm/mm_init.c | 11 +++++++---- 3 files changed, 16 insertions(+), 11 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 7fb7331c57250782a464a9583c6ea4867f4ffdab..02f5e8cc40c78ac8b81bb5c6f9a= f8718b1ffb316 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -123,9 +123,12 @@ static inline bool migratetype_is_mergeable(int mt) return mt < MIGRATE_PCPTYPES; } =20 -#define for_each_migratetype_order(order, type) \ - for (order =3D 0; order < NR_PAGE_ORDERS; order++) \ - for (type =3D 0; type < MIGRATE_TYPES; type++) +#define for_each_free_list(list, zone) \ + for (unsigned int order =3D 0; order < NR_PAGE_ORDERS; order++) \ + for (unsigned int type =3D 0; \ + list =3D &zone->free_area[order].free_list[type], \ + type < MIGRATE_TYPES; \ + type++) \ =20 extern int page_group_by_mobility_disabled; =20 diff --git a/kernel/power/snapshot.c b/kernel/power/snapshot.c index 645f42e404789286ffa751f083e97e52a4e4cf7e..40a7064eb6b247f47ca02211f83= 47cbd605af590 100644 --- a/kernel/power/snapshot.c +++ b/kernel/power/snapshot.c @@ -1244,8 +1244,8 @@ unsigned int snapshot_additional_pages(struct zone *z= one) static void mark_free_pages(struct zone *zone) { unsigned long pfn, max_zone_pfn, page_count =3D WD_PAGE_COUNT; + struct list_head *free_list; unsigned long flags; - unsigned int order, t; struct page *page; =20 if (zone_is_empty(zone)) @@ -1269,9 +1269,8 @@ static void mark_free_pages(struct zone *zone) swsusp_unset_page_free(page); } =20 - for_each_migratetype_order(order, t) { - list_for_each_entry(page, - &zone->free_area[order].free_list[t], buddy_list) { + for_each_free_list(free_list, zone) { + list_for_each_entry(page, free_list, buddy_list) { unsigned long i; =20 pfn =3D page_to_pfn(page); diff --git a/mm/mm_init.c b/mm/mm_init.c index 3db2dea7db4c57c81f3fc3b71f0867025edda655..9554b79d0946a4a1a2ac5c934c1= f80d2dc91b087 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1435,11 +1435,14 @@ static void __meminit zone_init_internals(struct zo= ne *zone, enum zone_type idx, =20 static void __meminit zone_init_free_lists(struct zone *zone) { - unsigned int order, t; - for_each_migratetype_order(order, t) { - INIT_LIST_HEAD(&zone->free_area[order].free_list[t]); + struct list_head *list; + unsigned int order; + + for_each_free_list(list, zone) + INIT_LIST_HEAD(list); + + for (order =3D 0; order < NR_PAGE_ORDERS; order++) zone->free_area[order].nr_free =3D 0; - } =20 #ifdef CONFIG_UNACCEPTED_MEMORY INIT_LIST_HEAD(&zone->unaccepted_pages); --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A5F91C8630 for ; Wed, 24 Sep 2025 15:00:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; cv=none; b=O4+JqjGLVHshBoo8zubtyDtNSvcBk1nkCWQ5kxlfewAFsYMVXrfl+2nEMsjAzsOb5MlHAOdeF8+nDg+xF3txqeDbpkShswSKMU5Z5iLqeYOhgHcbRxNidCp/W6L9/0720iSfp8lMTyzIhj1QJVmrIm7pZipBtZMfdu6QACH2Z3w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726019; c=relaxed/simple; bh=ehl3w1CyI570eR6ZXmEHxOSAvE6+tVJmDVrz20BaSIc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=jo4geaYTa5O7bAdPitl11rIOu0yR5KR+KbOdNXcTUndvs9DMi5ry5qvuCmVoRxqK+cpROwyrQkBrgajJ60G9YA5JxBd/uTJuFTNX+VMZqJN+juNcH+j644C2BPCQ0fPl05ztq9ERbzIzz35YieilpH7G7tp4fldOiVndrOKHkMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=h7luoj/O; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="h7luoj/O" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46e1b906bf3so23560355e9.1 for ; Wed, 24 Sep 2025 08:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726016; x=1759330816; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bKJT4vpnX5xVBMK5mB1lNcuXbTDuRuXg8xoq8hiioZw=; b=h7luoj/O9Fl5hauvrdTtmu3TNEC+Wfuf2f275Qog3D0Gg5DHt7lqDyi7jKZ20W52Y1 YgYu8SpivOPis/AtJe+yvvi2oVcz8F/QkW7ivwgaL7okGo9xYyYhQYd9WIgQTyRNt95k LK7ydQZIsS1BJ0j0VDIyMOy453N3gCFcvfr2pxlII5hILRH0R4CbqoEg0PAaiehl09zx 7Tf1uj2CYK1h5M5JXZ8CBdxrpmJi21Y+SKLbIj+Rm3w2Ohl3jbJ5KH3Eys/Ca3zIW7Iv 2x5iiYMXGs7imqdB7xGSg4f0otYR1JQUenemWmU4wL3shKg6bDhwqNvTPdpguqD/2QgA 3gBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726016; x=1759330816; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bKJT4vpnX5xVBMK5mB1lNcuXbTDuRuXg8xoq8hiioZw=; b=QSmq+6nB03Ty/NhmPb/jxLkXMcGDAHOS6AE2IgTUJ7AOY4Ku7GtmU9wiMqC0bdfuuQ 5SAuKBSmY8ehCInjPtUzSrhgwjeG2HYJjxyJ1Oeo9bb5Wd9jCW99D7Dpv83tWE9UR+qj kt/Ly5Ix4jDH+Iko9UxiXYib1R0mq7uYKsl0+9X8zQq/z19c/PAScIl4W/LfzMtbnIKg muhNnpBDNMEiWp7K3N6kN1tRLOlgL+ynqBTfKCiCA7OAH5KMFkIuFxmag+9VHSRQwH9c +hIs4/jsHBt+M8VVtxzMfBqneCHIv6xiGMeicKzflLkfSAfdY33HDyxyyeGYU1Y8Hub6 4qPA== X-Forwarded-Encrypted: i=1; AJvYcCUQs4Cygt/lstqzkUbNAh5SbEJ3AixzUqeYin5ucs5g3k10sUt960NwdpAwMYqX+hwJsKH4OPjudMuQf8I=@vger.kernel.org X-Gm-Message-State: AOJu0YxwmS3J+T0tDXvojEG1oQ4/FDkOYKTTmnAKEnIpCQD+9PQmAcg2 2kNcOvXfUvug1MgjC8op5/30nAGCpAlSxucF6EoeAu3gO7pwkB1YRy4oSQBOI+EK0ewJ/Ua7bSO GLorkM9L4vdwC6A== X-Google-Smtp-Source: AGHT+IEwpH5Oza9iuA/SA1ZMp2dUWj5FnIP9hJFno7WJD+Pwhp72Tuy9jz82MKe39ZlHCf09lf2CtXpfs4Ad2A== X-Received: from wmim22.prod.google.com ([2002:a7b:cb96:0:b0:46d:712:e422]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3b9e:b0:45f:2f0f:6649 with SMTP id 5b1f17b1804b1-46e329d474fmr2189815e9.8.1758726015769; Wed, 24 Sep 2025 08:00:15 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:43 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-8-2d861768041f@google.com> Subject: [PATCH 08/21] mm: rejig pageblock mask definitions From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable A later patch will complicate the definition of these masks, this is a preparatory patch to make that patch easier to review. - More masks will be needed, so add a PAGEBLOCK_ prefix to the names to avoid polluting the "global namespace" too much. - Move the CONFIG_MEMORY_ISOLATION ifdeffery into a separate block, this allows the various conditionally-defined masks to be combined cleanly. - This makes MIGRATETYPE_AND_ISO_MASK start to look pretty long. Well, that global mask only exists for quite a specific purpose so just drop it and take advantage of the newly-defined PAGEBLOCK_ISO_MASK. Signed-off-by: Brendan Jackman --- include/linux/pageblock-flags.h | 6 +++--- mm/page_alloc.c | 18 +++++++++--------- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flag= s.h index e046278a01fa8c37d898df94114d088933b6747f..13457e920e892c1c5083e0dc63e= 2ecfbed88f60e 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -36,12 +36,12 @@ enum pageblock_bits { =20 #define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS)) =20 -#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migra= te_2)) +#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BI= T(PB_migrate_2)) =20 #ifdef CONFIG_MEMORY_ISOLATION -#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolat= e)) +#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate) #else -#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK +#define PAGEBLOCK_ISO_MASK 0 #endif =20 #if defined(CONFIG_HUGETLB_PAGE) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0d1c28decd57b4a5e250acc0efc41669b7f67f5b..a1db87488296a6d2d91a1be8d4d= 202f1841c4dfd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -370,7 +370,7 @@ get_pfnblock_bitmap_bitidx(const struct page *page, uns= igned long pfn, #else BUILD_BUG_ON(NR_PAGEBLOCK_BITS !=3D 4); #endif - BUILD_BUG_ON(__MIGRATE_TYPE_END > MIGRATETYPE_MASK); + BUILD_BUG_ON(__MIGRATE_TYPE_END > PAGEBLOCK_MIGRATETYPE_MASK); VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); =20 bitmap =3D get_pageblock_bitmap(page, pfn); @@ -443,7 +443,7 @@ bool get_pfnblock_bit(const struct page *page, unsigned= long pfn, __always_inline enum migratetype get_pfnblock_migratetype(const struct page *page, unsigned long pfn) { - unsigned long mask =3D MIGRATETYPE_AND_ISO_MASK; + unsigned long mask =3D PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK; unsigned long flags; =20 flags =3D __get_pfnblock_flags_mask(page, pfn, mask); @@ -452,7 +452,7 @@ get_pfnblock_migratetype(const struct page *page, unsig= ned long pfn) if (flags & BIT(PB_migrate_isolate)) return MIGRATE_ISOLATE; #endif - return flags & MIGRATETYPE_MASK; + return flags & PAGEBLOCK_MIGRATETYPE_MASK; } =20 /** @@ -540,11 +540,11 @@ static void set_pageblock_migratetype(struct page *pa= ge, } VM_WARN_ONCE(get_pageblock_isolate(page), "Use clear_pageblock_isolate() to unisolate pageblock"); - /* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */ + /* PAGEBLOCK_ISO_MASK clears PB_migrate_isolate if it is set */ #endif __set_pfnblock_flags_mask(page, page_to_pfn(page), (unsigned long)migratetype, - MIGRATETYPE_AND_ISO_MASK); + PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } =20 void __meminit init_pageblock_migratetype(struct page *page, @@ -570,7 +570,7 @@ void __meminit init_pageblock_migratetype(struct page *= page, flags |=3D BIT(PB_migrate_isolate); #endif __set_pfnblock_flags_mask(page, page_to_pfn(page), flags, - MIGRATETYPE_AND_ISO_MASK); + PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } =20 #ifdef CONFIG_DEBUG_VM @@ -2122,15 +2122,15 @@ static bool __move_freepages_block_isolate(struct z= one *zone, } =20 move: - /* Use MIGRATETYPE_MASK to get non-isolate migratetype */ + /* Use PAGEBLOCK_MIGRATETYPE_MASK to get non-isolate migratetype */ if (isolate) { from_mt =3D __get_pfnblock_flags_mask(page, page_to_pfn(page), - MIGRATETYPE_MASK); + PAGEBLOCK_MIGRATETYPE_MASK); to_mt =3D MIGRATE_ISOLATE; } else { from_mt =3D MIGRATE_ISOLATE; to_mt =3D __get_pfnblock_flags_mask(page, page_to_pfn(page), - MIGRATETYPE_MASK); + PAGEBLOCK_MIGRATETYPE_MASK); } =20 __move_freepages_block(zone, start_pfn, from_mt, to_mt); --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD25C30EF99 for ; Wed, 24 Sep 2025 15:00:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726022; cv=none; b=U9ol4un+Bhtk/TziMBBq/D06mT1xquTEiUjvrs9FJ8N3MWJRdUVtnNWJB0jkKRw+ktxKgHmCYfwyDU9pkLIXVH3hZh0KVDlKdOXWB7kL52Q5UXEcJe2/RxTDBFYVK0og1pTb9AkJn2eEX30gZA+ssnDHethjsOp5GBkLqwh9KAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726022; c=relaxed/simple; bh=KLFMQaBWl73ZWJXS226eqBexHzAr7X49UHgW7C8dm18=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Yrj5JHCnW7YIVcO48XIBIQCOhkfAA0JlNDy4MDR7jr3aQVYrRjNZtAdoWxrZTlJTp4ITFuuBKqVEXTKJW4AViSiL+KIrISwsa7f8oK2YyUimyJ4b9EQyuvfx3UGKgZjEpc6MsK7cB4quS935TiJ7KMz5KNBc13DvVsECmsdNo7s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WREefWG3; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WREefWG3" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45f2a1660fcso54922975e9.1 for ; Wed, 24 Sep 2025 08:00:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726017; x=1759330817; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=cxm1+BttfzZabXPoYExS0LpEIkdqq5ptW85BH8SIzIM=; b=WREefWG3WFkiH469MlE9hhepBcPy28lUNMckXDR/1be7G7fEwGsRbafn30PYBx5ngL lX4CvW5fKoHUKOrpCpcUEyhJd4f2k/q5qAtf2XGvUtcy12CRFu/889FK+Mi1VRcNQC+c rszjbElU+9tkUe64SDlk4640Pr2xgVVET7eTntnB9Z0NZacAdI6AHPQLcUUa2qbZLP3h aW7YCLYjEutJorLdoCLBbrw+ZQSopREWDanWOygFiE3eCtZ9n7DffjqA+8MHbwlq9Xl7 aYQqRPt3WdVX0KUgI1+bOdmOKO4Zk3mLIimLVZR451L0q3R1PmQSa1mw8qca2XAhzmZp fEvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726017; x=1759330817; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cxm1+BttfzZabXPoYExS0LpEIkdqq5ptW85BH8SIzIM=; b=PaSzQeZgFue579vI9UQFedrFlHpKETEUb02kEjEAh4lxZBbmonqA0FbdzxW1mEqSCv PnwX6iJ3OibRcT9pho4amXo/CbHafoR0yFDwRmB9vC6RnrKboI8ZK9i37N7zv6LYa89J 1229Ao/X+3Z6xLahWcWmsK5WySPQmw8BHyyuhk8wwBzBTUlVO3KpHN9izxskLWHMhOhL 1Adqx2ygRMjchbbGc+zrC2+dTQD2bsAgHi7DWLEMSsh3aF+8BEuPWa+AZrRAFFqKm8Ay BSEF28fXAOTo3i80pPityAKO5vf/OMr1jPwZXNXgtk2eGWv2DK4BNnf5/A6HdTem2Pwx lCzw== X-Forwarded-Encrypted: i=1; AJvYcCU9StF/ItKdLHgADExDcK1L2xKOVc7DUFM6w41qjijsyzEsYVZ9iPRDHEhMZUu1X/Hod5y3mu9xS10Tn1g=@vger.kernel.org X-Gm-Message-State: AOJu0YxkzBtyBTjfl2ZlpPTxf8i6KdBZ6jdV0cJwwk6MB5Q/GtOm3KR6 XMjsqOejtwpL/xKj0tQ4EGwLfrLUDz7Vkgb1yn0QMRFS+/YU0bTjopuCpLo9Mxo4wGZ8gDRGTZs a34+uZYn0IRvElw== X-Google-Smtp-Source: AGHT+IFbV4XCppcQcBh4YT0+YYYGbMDe4to2Az87MctcSzey34CxXXjmAG4jYevOjLMuvS5zxWssC7rPfCkOlw== X-Received: from wmpz21.prod.google.com ([2002:a05:600c:a15:b0:46e:2f78:5910]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:420e:b0:3e3:e7a0:1fec with SMTP id ffacd0b85a97d-40e43b08a86mr265628f8f.16.1758726016966; Wed, 24 Sep 2025 08:00:16 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:44 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-9-2d861768041f@google.com> Subject: [PATCH 09/21] mm/page_alloc: Invert is_check_pages_enabled() check From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable A later patch will expand this function, making it ugly that the whole body sits inside a conditional. In preparation, invert it to de-indent the main logic. Separate commit to make review easier. Signed-off-by: Brendan Jackman --- mm/page_alloc.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a1db87488296a6d2d91a1be8d4d202f1841c4dfd..10757410da2127b0488c99c5933= 422fc649f9a1d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1752,13 +1752,14 @@ static bool check_new_page(struct page *page) =20 static inline bool check_new_pages(struct page *page, unsigned int order) { - if (is_check_pages_enabled()) { - for (int i =3D 0; i < (1 << order); i++) { - struct page *p =3D page + i; + if (!is_check_pages_enabled()) + return false; =20 - if (check_new_page(p)) - return true; - } + for (int i =3D 0; i < (1 << order); i++) { + struct page *p =3D page + i; + + if (check_new_page(p)) + return true; } =20 return false; --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE77A30DEC4 for ; Wed, 24 Sep 2025 15:00:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726021; cv=none; b=bPQDaSAIvdZbYykbuOtvrzLAexmzChNSle8L6cYVgBIabshRqj4QdodOqwALbzh0YHBJx33w5EgmOJSen8mnn5BZk9hDlvMezezyI6Ncevi0tASBN57KEU2Ud7cP96RGhuAiDG3vibG0Tm6ic9njgtVkhqDzgOsK/yZZ/8nQvUU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726021; c=relaxed/simple; bh=jxWvtMYDjx6V4MJnRkjEO23tCw+cOjtftYS5Dsn4+ag=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CXM+pP/P2lx91Z96NFJdu8JPMXE7iD9N/rgADYGsels+BoIhPV73X3ryuhNy82mbXLgt+GdkUfL2yEwzVbRjtnKcC7XrowzqbFLJqVBb4RGvygy0NBJBERYFJMQe7dRGHHIWRcJxZ7nwJ4ql4R8zrF6ajXpZonM8SdrgUUDK9KQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fSk4GrjQ; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fSk4GrjQ" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45df7e734e0so56609135e9.0 for ; Wed, 24 Sep 2025 08:00:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726018; x=1759330818; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=8QF2cRHh8FytPgxr8dpYOGpJvCeadM1546zJy7s8ewQ=; b=fSk4GrjQqYOf+m6TMI5UlU3RtEE2tz4RbcOkoP8pfVAPAFH86dJ0ylXy1Le8u97wKS 2smiN+4Sf6344So2tiuQXVVE4J7Z1fcl4+TwZdDn9nf3iXlYqUVNbu2OsjLekQ4ntgWt tVlmbI0puFQdX2ndfFgdxpuqACv/RBJF/K11k0eV3Gvs7qzXXSFKrR8/1B1mQTN69Is0 f4a699YjQBvBRVvO/lUUY8B/XSQYSL/pPSwnQHTdlZOOZyt0znfuLJzsibwNfSRdOG2I 8rKGz/SFt6dFC2zierrR4KYTQU1PE+IM9FnwAkZsZfQtIp0e2RpjZUJyHkFXhMYFJJc6 UTRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726018; x=1759330818; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=8QF2cRHh8FytPgxr8dpYOGpJvCeadM1546zJy7s8ewQ=; b=b+eL6GDhC5KF9ZtmAbAzSN4U+KsTqhykbeZPGYFbPSgCmA7cUjQeePzL1TlPF7gHyl NGZjPw/T5e9Jc4kL5c1fgsT66YWEMa6DfEgUEZ1SgKDCTY1y5AXoMW/7XvjfhWNYytBd z73xAIZ8pWLnfwYiDzLRRcTodbtXcutodQjybLdimOr9boZ+Y7mwCsa5ch6Q9TqvyM6i 9YnLsdgmN2K9I/sGdtMrbj3DQfWSD3EW26QVsn7RCGPBw+IUIq/mSX5r0eAzwZ+knlDA hS5GaR8cnK0xseZDMpvJVP41XlHnke6hagbLtTJoPBogECwIoMn30KiAASj1sV4Sr5kS qZiw== X-Forwarded-Encrypted: i=1; AJvYcCUUe1NigPuk7Ewp9CNgwFVLyiDMtdogcz9Cfciv4Tqu/4PR+GjvoklzeZ/R9kZLt9lgkd8QaHIgj62Ia5Y=@vger.kernel.org X-Gm-Message-State: AOJu0YxftDx2+KYdknb6SDZX+ZpZmO4bxVBQ+HR/msyhC92NDITc35So SyKW/TcYYMx9moIYO+ZexbuXGh7Q3NLZY40LnfsNjPLZX4gIWuhtdmv4uGdtNhDHpAw4mUZlxlW DMvfGmh8U5cQxUQ== X-Google-Smtp-Source: AGHT+IHJnBZBPS1nMKaaZcgp71cKjbeNO0YRmTqlXBuWFLXJWbiztOh0lMCy2GikS/TH3xlbidCnHScZxYe/Iw== X-Received: from wmpd40.prod.google.com ([2002:a05:600c:4c28:b0:45f:2be0:b17e]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:198f:b0:45d:d197:fecf with SMTP id 5b1f17b1804b1-46e329229ffmr2453735e9.0.1758726018015; Wed, 24 Sep 2025 08:00:18 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:45 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-10-2d861768041f@google.com> Subject: [PATCH 10/21] mm/page_alloc: remove ifdefs from pindex helpers From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The ifdefs are not technically needed here, everything used here is always defined. They aren't doing much harm right now but a following patch will complicate these functions. Switching to IS_ENABLED() makes the code a bit less tiresome to read. Signed-off-by: Brendan Jackman --- mm/page_alloc.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 10757410da2127b0488c99c5933422fc649f9a1d..08e0faab992fcf3c426d4783da0= 41f930075d903 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -656,19 +656,17 @@ static void bad_page(struct page *page, const char *r= eason) =20 static inline unsigned int order_to_pindex(int migratetype, int order) { + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + bool movable =3D migratetype =3D=3D MIGRATE_MOVABLE; =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - bool movable; - if (order > PAGE_ALLOC_COSTLY_ORDER) { - VM_BUG_ON(order !=3D HPAGE_PMD_ORDER); + if (order > PAGE_ALLOC_COSTLY_ORDER) { + VM_BUG_ON(order !=3D HPAGE_PMD_ORDER); =20 - movable =3D migratetype =3D=3D MIGRATE_MOVABLE; - - return NR_LOWORDER_PCP_LISTS + movable; + return NR_LOWORDER_PCP_LISTS + movable; + } + } else { + VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); } -#else - VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); -#endif =20 return (MIGRATE_PCPTYPES * order) + migratetype; } @@ -677,12 +675,12 @@ static inline int pindex_to_order(unsigned int pindex) { int order =3D pindex / MIGRATE_PCPTYPES; =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (pindex >=3D NR_LOWORDER_PCP_LISTS) - order =3D HPAGE_PMD_ORDER; -#else - VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); -#endif + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + if (pindex >=3D NR_LOWORDER_PCP_LISTS) + order =3D HPAGE_PMD_ORDER; + } else { + VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); + } =20 return order; } --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD2CE30EF9B for ; Wed, 24 Sep 2025 15:00:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726024; cv=none; b=jXpfTbp/I1KgpSlFz7ZHA0/vomWcZ99f2r3sckI5Lz790tRcghwERy7Q9pnZDkTwSO+SdWu9oUGV8arPNox4m4kdaCc5rW9lQlkJT9Lq9+vhcBKKplHQEZWQQ673KiKP5Yc0gTaJYObtG5vu2EvfPYwhFWaoki35Sp2LclYvodg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726024; c=relaxed/simple; bh=//CyeL17FbZStVm/tqSyHS2yfoI5x37acNVnXVn2oh4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=br7poVQlu7LRJ/kiGG6TeLGz23l5Y6a+ZYyV9w56Etv+b3+By6OyVWX9zpDBCpIDNRHi7rKSnA43hrE9lm32GKiRjLxT+J2ep9XTWln9ZDxx1sqYRjzJhHDBP817wGBReAjdMJyfGB5B22k6IV/WqJhemdj1Za0DjE8nyBZzPb0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Exym9yCH; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Exym9yCH" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46dfd711001so22293425e9.0 for ; Wed, 24 Sep 2025 08:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726019; x=1759330819; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=umKKDzHwdYkxD5fzEq4OswTg4T0LYreqV7DonTKBHhY=; b=Exym9yCHEKETn95iNBnQJ2GTeSUlyvi+HKQexfAEU4eJqm4PtbG/A4xI1qZw3Zouoo gzN+FQrGZ7flnFpOMmm37pmEkeTBZ1HmkloGUJbzL1D2O54Ezl0a10nubWMst3OdP9pC 2FpLEesEI+ZlW9swYoRi+JSev5tLdVvilLKdYYNAU9Oad81BM1QyEJbtL9dJZSgsX9Tf Il/Nch3GPXqq+ClLvW1ffN/xWoC8CQcgCZ/ZSakkvlpQLExL4oTeQ2QZjmPaQypZyEFS XWrEiw/Ra++7AhY//B9+gbfr/Ec9KUhpfEj0Ap0vu/buOpQiyhUROUJ+VwrRYQ12U3YT lqtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726019; x=1759330819; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=umKKDzHwdYkxD5fzEq4OswTg4T0LYreqV7DonTKBHhY=; b=rI+nyAi16XRC2zjHPxj2IUVbAmpq1oj8ogjXl4XCPzHb4pkVW6ckJyC2YElccgNDZS QE+tBd2Hv4kk8hBUDyBDlTZxLvXCP7ZkiH+VlDm8/UUnvOqm4QMrCP9sAYiVFYusX6cG YuwW5I7pdoFoHk+b72BOuq/rEr93qgvu/u+75c0Te/nlaE18OhgzSRsHdclyRk+1E3ti iR8cFTWwdHNNKblx8Bs9hElalU+pl4m50QZeXMVGO8DRAValYG5Ou57VBn9smUMjF4cN s45SbGQXfwkV4zCinkOmVU0OjYLOF297brhunV+Dh83y64mnnMhMs0WdZovLwXgZ/ulp WhPA== X-Forwarded-Encrypted: i=1; AJvYcCWeUG4VjNjw1tp+siJgirzZinS3E3oJjJ7orh/OlxvAs7Ptg50EZl1NV/2cg41ykRqpaUOWaTW7hd9I7QA=@vger.kernel.org X-Gm-Message-State: AOJu0Yyvffz9NTlrz86Wj924q8uffXgoZIETzDi1aPreAQUYlHrWOZ+Y aSwWuvKti0PB4fDxesr5J3khB7TvrhezB9dRKepZ5E+hB52HKYYawLtYGpfhRq+C6XZ1bIkz7wT 01Xx88sry053tdA== X-Google-Smtp-Source: AGHT+IG3cFtJV3DePXPwZsRWDDlgSW6BHlRPO0JWm3MMiIcolz1ebfjg+4gpN4H3IjkHB7jt40XE6UpsIoIiNQ== X-Received: from wmbfa13.prod.google.com ([2002:a05:600c:518d:b0:45b:89e7:5038]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:35cc:b0:46d:9d28:fb78 with SMTP id 5b1f17b1804b1-46e329a54e5mr1658455e9.8.1758726018825; Wed, 24 Sep 2025 08:00:18 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:46 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-11-2d861768041f@google.com> Subject: [PATCH 11/21] mm: introduce freetype_t From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This is preparation for making the page allocator aware of ASI sensitivity. To do that, certain properties will be highly desirable: - A "pool" of pages with each sensitivity is usually available, so that pages can be provided with the correct sensitivity without zeroing/TLB flushing. - Pages are physically grouped by sensitivity, so that large allocations rarely have to alter the pagetables due to ASI. - ASI sensitivity varies only at a certain fixed address granularity, so that the pagetables can all be pre-allocated. This is desirable because the page allocator will be changing mappings: pre-allocation is a straightforward way to avoid recursive allocations (of pagetables). It seems that the existing infrastructure for grouping pages by mobility, i.e. pageblocks and migratetypes, serves this purpose pretty nicely. Early prototypes of this approach took advantage of it by just overloading enum migratetype to encode not only mobility but also sensitivity. This overloading is OK if you can apply constraints like "only movable pages are ever sensitive", but in the real world such constraints don't actually exist, so overloading migratetype gets rather ugly. Therefore, introduce a new higher-level concept, called "freetype" (because it is used to index "free"lists) that can encode sensitivity orthogonally to mobility. This is the "more invasive changes" mentioned in [0]. Since freetypes and migratetypes would be very easy to mix up, freetypes are (at least for now) stored in a struct typedef similar to atomic_t. This provides type-safety, but comes at the expense of being pretty annoying to code with. For instance, freetype_t cannot be compared with the =3D=3D operator. Once this code matures, if the freetype/migratetype distinction gets less confusing, it might be wise to drop this struct and just use ints. To try and reduce review pain for such a churny patch, first introduce freetypes as nothing but an indirection over migratetypes. The helpers concerned with sensitivity are defined, but only as stubs. Convert everything over to using freetypes wherever they are needed to index freelists, but maintain references to migratetypes in code that really only cares specifically about mobility. A later patch will add the encoding of sensitivity into freetype_t. For this patch, no functional change is intended. [0]: https://lore.kernel.org/lkml/20250313-asi-page-alloc-v1-9-04972e046cea= @google.com/ Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 16 ++- include/linux/mmzone.h | 68 +++++++++- mm/compaction.c | 32 ++--- mm/internal.h | 20 ++- mm/page_alloc.c | 346 ++++++++++++++++++++++++++++++---------------= ---- mm/page_isolation.c | 2 +- mm/page_owner.c | 7 +- mm/page_reporting.c | 4 +- mm/show_mem.c | 2 +- 9 files changed, 330 insertions(+), 167 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 0ceb4e09306c4a7098d5a61645396e3b82a1ca30..a275171c5a6aecafd7783e57ce7= d4316c5e56655 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -16,8 +16,10 @@ struct mempolicy; #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) #define GFP_MOVABLE_SHIFT 3 =20 -static inline int gfp_migratetype(const gfp_t gfp_flags) +static inline freetype_t gfp_freetype(const gfp_t gfp_flags) { + int migratetype; + VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) =3D=3D GFP_MOVABLE_MASK); BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) !=3D ___GFP_MOVABLE); BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) !=3D MIGRATE_MOVABLE); @@ -25,11 +27,15 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) BUILD_BUG_ON(((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> GFP_MOVABLE_SHIFT) !=3D MIGRATE_HIGHATOMIC); =20 - if (unlikely(page_group_by_mobility_disabled)) - return MIGRATE_UNMOVABLE; + if (unlikely(page_group_by_mobility_disabled)) { + migratetype =3D MIGRATE_UNMOVABLE; + } else { + /* Group based on mobility */ + migratetype =3D (__force unsigned long)(gfp_flags & GFP_MOVABLE_MASK) + >> GFP_MOVABLE_SHIFT; + } =20 - /* Group based on mobility */ - return (__force unsigned long)(gfp_flags & GFP_MOVABLE_MASK) >> GFP_MOVAB= LE_SHIFT; + return migrate_to_freetype(migratetype, false); } #undef GFP_MOVABLE_MASK #undef GFP_MOVABLE_SHIFT diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 02f5e8cc40c78ac8b81bb5c6f9af8718b1ffb316..56310722f38b788154ee15845b6= 877ed7e70d6b7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -123,6 +123,55 @@ static inline bool migratetype_is_mergeable(int mt) return mt < MIGRATE_PCPTYPES; } =20 +#define NR_SENSITIVITIES 1 + +/* + * A freetype is the index used to identify free lists (free area lists and + * pcplists). On non-ASI this is the same thing as a migratetype, on ASI i= t also + * encodes sensitivity. To avoid accidentally mixing the two identifiers, + * freetypes are a struct in the style of atomic_t. + */ +typedef struct { + int type; +} freetype_t; + +#define NR_FREETYPES (MIGRATE_TYPES * NR_SENSITIVITIES) + +static inline freetype_t migrate_to_freetype(enum migratetype mt, bool sen= sitive) +{ + freetype_t freetype; + + freetype.type =3D mt; + return freetype; +} + +static inline enum migratetype free_to_migratetype(freetype_t freetype) +{ + return freetype.type; +} + +static inline bool freetype_sensitive(freetype_t freetype) +{ + return false; +} + +/* Convenience helper, return the freetype modified to have the migratetyp= e. */ +static inline freetype_t freetype_with_migrate(freetype_t freetype, + enum migratetype migratetype) +{ + return migrate_to_freetype(migratetype, freetype_sensitive(freetype)); +} + +static inline bool freetypes_equal(freetype_t a, freetype_t b) +{ + return a.type =3D=3D b.type; +} + +#define for_each_sensitivity(sensitive) \ + for (int _s =3D 0; \ + sensitive =3D (bool)_s, _s < NR_SENSITIVITIES; \ + _s++) + #define for_each_free_list(list, zone) \ for (unsigned int order =3D 0; order < NR_PAGE_ORDERS; order++) \ for (unsigned int type =3D 0; \ @@ -132,17 +181,30 @@ static inline bool migratetype_is_mergeable(int mt) =20 extern int page_group_by_mobility_disabled; =20 +freetype_t get_pfnblock_freetype(const struct page *page, unsigned long pf= n); + #define get_pageblock_migratetype(page) \ get_pfnblock_migratetype(page, page_to_pfn(page)) =20 +#define get_pageblock_freetype(page) \ + get_pfnblock_freetype(page, page_to_pfn(page)) + #define folio_migratetype(folio) \ get_pageblock_migratetype(&folio->page) =20 struct free_area { - struct list_head free_list[MIGRATE_TYPES]; + struct list_head free_list[NR_FREETYPES]; unsigned long nr_free; }; =20 +static inline +struct list_head *free_area_list(struct free_area *area, freetype_t type) +{ + VM_BUG_ON(type.type < 0 || type.type >=3D ARRAY_SIZE(area->free_list)); + VM_BUG_ON(!area); + return &area->free_list[type.type]; +} + struct pglist_data; =20 #ifdef CONFIG_NUMA @@ -726,8 +788,10 @@ enum zone_watermarks { #else #define NR_PCP_THP 0 #endif +/* Note this is the number per sensitivity. */ #define NR_LOWORDER_PCP_LISTS (MIGRATE_PCPTYPES * (PAGE_ALLOC_COSTLY_ORDER= + 1)) -#define NR_PCP_LISTS (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS_PER_SENSITIVITY (NR_LOWORDER_PCP_LISTS + NR_PCP_THP) +#define NR_PCP_LISTS (NR_PCP_LISTS_PER_SENSITIVITY * NR_SENSITIVITIES) =20 /* * Flags used in pcp->flags field. diff --git a/mm/compaction.c b/mm/compaction.c index 1e8f8eca318c6844c27682677a0a9ea552316828..64a2c88a66a92f5c87169fbc11f= 87e8ae822af99 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -1359,7 +1359,7 @@ isolate_migratepages_range(struct compact_control *cc= , unsigned long start_pfn, static bool suitable_migration_source(struct compact_control *cc, struct page *page) { - int block_mt; + freetype_t block_ft; =20 if (pageblock_skip_persistent(page)) return false; @@ -1367,12 +1367,12 @@ static bool suitable_migration_source(struct compac= t_control *cc, if ((cc->mode !=3D MIGRATE_ASYNC) || !cc->direct_compaction) return true; =20 - block_mt =3D get_pageblock_migratetype(page); + block_ft =3D get_pageblock_freetype(page); =20 - if (cc->migratetype =3D=3D MIGRATE_MOVABLE) - return is_migrate_movable(block_mt); + if (free_to_migratetype(cc->freetype) =3D=3D MIGRATE_MOVABLE) + return is_migrate_movable(free_to_migratetype(block_ft)); else - return block_mt =3D=3D cc->migratetype; + return freetypes_equal(block_ft, cc->freetype); } =20 /* Returns true if the page is within a block suitable for migration to */ @@ -1963,7 +1963,8 @@ static unsigned long fast_find_migrateblock(struct co= mpact_control *cc) * reduces the risk that a large movable pageblock is freed for * an unmovable/reclaimable small allocation. */ - if (cc->direct_compaction && cc->migratetype !=3D MIGRATE_MOVABLE) + if (cc->direct_compaction && + free_to_migratetype(cc->freetype) !=3D MIGRATE_MOVABLE) return pfn; =20 /* @@ -2234,7 +2235,7 @@ static bool should_proactive_compact_node(pg_data_t *= pgdat) static enum compact_result __compact_finished(struct compact_control *cc) { unsigned int order; - const int migratetype =3D cc->migratetype; + const freetype_t freetype =3D cc->freetype; int ret; =20 /* Compaction run completes if the migrate and free scanner meet */ @@ -2309,24 +2310,25 @@ static enum compact_result __compact_finished(struc= t compact_control *cc) for (order =3D cc->order; order < NR_PAGE_ORDERS; order++) { struct free_area *area =3D &cc->zone->free_area[order]; =20 - /* Job done if page is free of the right migratetype */ - if (!free_area_empty(area, migratetype)) + /* Job done if page is free of the right freetype */ + if (!free_area_empty(area, freetype)) return COMPACT_SUCCESS; =20 #ifdef CONFIG_CMA /* MIGRATE_MOVABLE can fallback on MIGRATE_CMA */ if (migratetype =3D=3D MIGRATE_MOVABLE && - !free_area_empty(area, MIGRATE_CMA)) + !free_areas_empty(area, MIGRATE_CMA)) return COMPACT_SUCCESS; #endif /* * Job done if allocation would steal freepages from - * other migratetype buddy lists. + * other freetype buddy lists. */ - if (find_suitable_fallback(area, order, migratetype, true) >=3D 0) + if (find_suitable_fallback(area, order, freetype, true) >=3D 0) /* - * Movable pages are OK in any pageblock. If we are - * stealing for a non-movable allocation, make sure + * Movable pages are OK in any pageblock of the right + * sensitivity. If we are * stealing for a + * non-movable allocation, make sure * we finish compacting the current pageblock first * (which is assured by the above migrate_pfn align * check) so it is as free as possible and we won't @@ -2531,7 +2533,7 @@ compact_zone(struct compact_control *cc, struct captu= re_control *capc) INIT_LIST_HEAD(&cc->freepages[order]); INIT_LIST_HEAD(&cc->migratepages); =20 - cc->migratetype =3D gfp_migratetype(cc->gfp_mask); + cc->freetype =3D gfp_freetype(cc->gfp_mask); =20 if (!is_via_compact_memory(cc->order)) { ret =3D compaction_suit_allocation_order(cc->zone, cc->order, diff --git a/mm/internal.h b/mm/internal.h index c8d6998bd6b223c62405fd54419282453fc40b9e..50ff6671f19d38a59c9f07e66d3= 47baf85ddf085 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -589,7 +590,7 @@ struct alloc_context { struct zonelist *zonelist; nodemask_t *nodemask; struct zoneref *preferred_zoneref; - int migratetype; + freetype_t freetype; =20 /* * highest_zoneidx represents highest usable zone index of @@ -740,8 +741,8 @@ static inline void clear_zone_contiguous(struct zone *z= one) } =20 extern int __isolate_free_page(struct page *page, unsigned int order); -extern void __putback_isolated_page(struct page *page, unsigned int order, - int mt); +void __putback_isolated_page(struct page *page, unsigned int order, + freetype_t freetype); extern void memblock_free_pages(struct page *page, unsigned long pfn, unsigned int order); extern void __free_pages_core(struct page *page, unsigned int order, @@ -893,7 +894,7 @@ struct compact_control { short search_order; /* order to start a fast search at */ const gfp_t gfp_mask; /* gfp mask of a direct compactor */ int order; /* order a direct compactor needs */ - int migratetype; /* migratetype of direct compactor */ + freetype_t freetype; /* freetype of direct compactor */ const unsigned int alloc_flags; /* alloc flags of a direct compactor */ const int highest_zoneidx; /* zone index of a direct compactor */ enum migrate_mode mode; /* Async or sync migration mode */ @@ -950,11 +951,16 @@ static inline void init_cma_pageblock(struct page *pa= ge) =20 =20 int find_suitable_fallback(struct free_area *area, unsigned int order, - int migratetype, bool claimable); + freetype_t freetype, bool claimable); =20 -static inline bool free_area_empty(struct free_area *area, int migratetype) +static inline bool free_area_empty(struct free_area *area, freetype_t free= type) { - return list_empty(&area->free_list[migratetype]); + return list_empty(free_area_list(area, freetype)); +} + +static inline bool free_areas_empty(struct free_area *area, int migratetyp= e) +{ + return free_area_empty(area, migrate_to_freetype(migratetype, false)); } =20 /* mm/util.c */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 08e0faab992fcf3c426d4783da041f930075d903..4ce81f8d4e59966b7c0c2902e24= aa2f4639a0e59 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -430,6 +430,37 @@ bool get_pfnblock_bit(const struct page *page, unsigne= d long pfn, return test_bit(bitidx + pb_bit, bitmap_word); } =20 +/** + * __get_pfnblock_freetype - Return the freetype of a pageblock, optionally + * ignoring the fact that it's currently isolated. + * @page: The page within the block of interest + * @pfn: The target page frame number + * @ignore_iso: If isolated, return the migratetype that the block had bef= ore + * isolation. + */ +__always_inline freetype_t +__get_pfnblock_freetype(const struct page *page, unsigned long pfn, + bool ignore_iso) +{ + int mt =3D get_pfnblock_migratetype(page, pfn); + + return migrate_to_freetype(mt, false); +} + +/** + * get_pfnblock_migratetype - Return the freetype of a pageblock + * @page: The page within the block of interest + * @pfn: The target page frame number + * + * Return: The freetype of the pageblock + */ +__always_inline freetype_t +get_pfnblock_freetype(const struct page *page, unsigned long pfn) +{ + return __get_pfnblock_freetype(page, pfn, false); +} + + /** * get_pfnblock_migratetype - Return the migratetype of a pageblock * @page: The page within the block of interest @@ -739,8 +770,11 @@ static inline struct capture_control *task_capc(struct= zone *zone) =20 static inline bool compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) + int order, freetype_t freetype) { + enum migratetype migratetype =3D free_to_migratetype(freetype); + enum migratetype capc_mt; + if (!capc || order !=3D capc->cc->order) return false; =20 @@ -749,6 +783,8 @@ compaction_capture(struct capture_control *capc, struct= page *page, is_migrate_isolate(migratetype)) return false; =20 + capc_mt =3D free_to_migratetype(capc->cc->freetype); + /* * Do not let lower order allocations pollute a movable pageblock * unless compaction is also requesting movable pages. @@ -757,12 +793,12 @@ compaction_capture(struct capture_control *capc, stru= ct page *page, * have trouble finding a high-order free page. */ if (order < pageblock_order && migratetype =3D=3D MIGRATE_MOVABLE && - capc->cc->migratetype !=3D MIGRATE_MOVABLE) + capc_mt !=3D MIGRATE_MOVABLE) return false; =20 - if (migratetype !=3D capc->cc->migratetype) + if (migratetype !=3D capc_mt) trace_mm_page_alloc_extfrag(page, capc->cc->order, order, - capc->cc->migratetype, migratetype); + capc_mt, migratetype); =20 capc->page =3D page; return true; @@ -776,7 +812,7 @@ static inline struct capture_control *task_capc(struct = zone *zone) =20 static inline bool compaction_capture(struct capture_control *capc, struct page *page, - int order, int migratetype) + int order, freetype_t freetype) { return false; } @@ -801,23 +837,23 @@ static inline void account_freepages(struct zone *zon= e, int nr_pages, =20 /* Used for pages not on another list */ static inline void __add_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int migratetype, + unsigned int order, freetype_t freetype, bool tail) { struct free_area *area =3D &zone->free_area[order]; int nr_pages =3D 1 << order; =20 - VM_WARN_ONCE(get_pageblock_migratetype(page) !=3D migratetype, + VM_WARN_ONCE(!freetypes_equal(get_pageblock_freetype(page), freetype), "page type is %d, passed migratetype is %d (nr=3D%d)\n", - get_pageblock_migratetype(page), migratetype, nr_pages); + get_pageblock_freetype(page).type, freetype.type, nr_pages); =20 if (tail) - list_add_tail(&page->buddy_list, &area->free_list[migratetype]); + list_add_tail(&page->buddy_list, free_area_list(area, freetype)); else - list_add(&page->buddy_list, &area->free_list[migratetype]); + list_add(&page->buddy_list, free_area_list(area, freetype)); area->nr_free++; =20 - if (order >=3D pageblock_order && !is_migrate_isolate(migratetype)) + if (order >=3D pageblock_order && !is_migrate_isolate(free_to_migratetype= (freetype))) __mod_zone_page_state(zone, NR_FREE_PAGES_BLOCKS, nr_pages); } =20 @@ -827,17 +863,20 @@ static inline void __add_to_free_list(struct page *pa= ge, struct zone *zone, * allocation again (e.g., optimization for memory onlining). */ static inline void move_to_free_list(struct page *page, struct zone *zone, - unsigned int order, int old_mt, int new_mt) + unsigned int order, + freetype_t old_ft, freetype_t new_ft) { struct free_area *area =3D &zone->free_area[order]; + int old_mt =3D free_to_migratetype(old_ft); + int new_mt =3D free_to_migratetype(new_ft); int nr_pages =3D 1 << order; =20 /* Free page moving can fail, so it happens before the type update */ - VM_WARN_ONCE(get_pageblock_migratetype(page) !=3D old_mt, - "page type is %d, passed migratetype is %d (nr=3D%d)\n", - get_pageblock_migratetype(page), old_mt, nr_pages); + VM_WARN_ONCE(!freetypes_equal(get_pageblock_freetype(page), old_ft), + "page type is %d, passed freetype is %d (nr=3D%d)\n", + get_pageblock_freetype(page).type, old_ft.type, nr_pages); =20 - list_move_tail(&page->buddy_list, &area->free_list[new_mt]); + list_move_tail(&page->buddy_list, free_area_list(area, new_ft)); =20 account_freepages(zone, -nr_pages, old_mt); account_freepages(zone, nr_pages, new_mt); @@ -880,9 +919,9 @@ static inline void del_page_from_free_list(struct page = *page, struct zone *zone, } =20 static inline struct page *get_page_from_free_area(struct free_area *area, - int migratetype) + freetype_t freetype) { - return list_first_entry_or_null(&area->free_list[migratetype], + return list_first_entry_or_null(free_area_list(area, freetype), struct page, buddy_list); } =20 @@ -938,9 +977,10 @@ buddy_merge_likely(unsigned long pfn, unsigned long bu= ddy_pfn, static inline void __free_one_page(struct page *page, unsigned long pfn, struct zone *zone, unsigned int order, - int migratetype, fpi_t fpi_flags) + freetype_t freetype, fpi_t fpi_flags) { struct capture_control *capc =3D task_capc(zone); + int migratetype =3D free_to_migratetype(freetype); unsigned long buddy_pfn =3D 0; unsigned long combined_pfn; struct page *buddy; @@ -949,16 +989,17 @@ static inline void __free_one_page(struct page *page, VM_BUG_ON(!zone_is_initialized(zone)); VM_BUG_ON_PAGE(page->flags.f & PAGE_FLAGS_CHECK_AT_PREP, page); =20 - VM_BUG_ON(migratetype =3D=3D -1); + VM_BUG_ON(freetype.type =3D=3D -1); VM_BUG_ON_PAGE(pfn & ((1 << order) - 1), page); VM_BUG_ON_PAGE(bad_range(zone, page), page); =20 account_freepages(zone, 1 << order, migratetype); =20 while (order < MAX_PAGE_ORDER) { - int buddy_mt =3D migratetype; + freetype_t buddy_ft =3D freetype; + enum migratetype buddy_mt =3D free_to_migratetype(buddy_ft); =20 - if (compaction_capture(capc, page, order, migratetype)) { + if (compaction_capture(capc, page, order, freetype)) { account_freepages(zone, -(1 << order), migratetype); return; } @@ -974,7 +1015,8 @@ static inline void __free_one_page(struct page *page, * pageblock isolation could cause incorrect freepage or CMA * accounting or HIGHATOMIC accounting. */ - buddy_mt =3D get_pfnblock_migratetype(buddy, buddy_pfn); + buddy_ft =3D get_pfnblock_freetype(buddy, buddy_pfn); + buddy_mt =3D free_to_migratetype(buddy_ft); =20 if (migratetype !=3D buddy_mt && (!migratetype_is_mergeable(migratetype) || @@ -1016,7 +1058,7 @@ static inline void __free_one_page(struct page *page, else to_tail =3D buddy_merge_likely(pfn, buddy_pfn, page, order); =20 - __add_to_free_list(page, zone, order, migratetype, to_tail); + __add_to_free_list(page, zone, order, freetype, to_tail); =20 /* Notify page reporting subsystem of freed page */ if (!(fpi_flags & FPI_SKIP_REPORT_NOTIFY)) @@ -1471,19 +1513,20 @@ static void free_pcppages_bulk(struct zone *zone, i= nt count, nr_pages =3D 1 << order; do { unsigned long pfn; - int mt; + freetype_t ft; =20 page =3D list_last_entry(list, struct page, pcp_list); pfn =3D page_to_pfn(page); - mt =3D get_pfnblock_migratetype(page, pfn); + ft =3D get_pfnblock_freetype(page, pfn); =20 /* must delete to avoid corrupting pcp list */ list_del(&page->pcp_list); count -=3D nr_pages; pcp->count -=3D nr_pages; =20 - __free_one_page(page, pfn, zone, order, mt, FPI_NONE); - trace_mm_page_pcpu_drain(page, order, mt); + __free_one_page(page, pfn, zone, order, ft, FPI_NONE); + trace_mm_page_pcpu_drain(page, order, + free_to_migratetype(ft)); } while (count > 0 && !list_empty(list)); } =20 @@ -1504,9 +1547,9 @@ static void split_large_buddy(struct zone *zone, stru= ct page *page, order =3D pageblock_order; =20 do { - int mt =3D get_pfnblock_migratetype(page, pfn); + freetype_t ft =3D get_pfnblock_freetype(page, pfn); =20 - __free_one_page(page, pfn, zone, order, mt, fpi); + __free_one_page(page, pfn, zone, order, ft, fpi); pfn +=3D 1 << order; if (pfn =3D=3D end) break; @@ -1684,7 +1727,7 @@ struct page *__pageblock_pfn_to_page(unsigned long st= art_pfn, * -- nyc */ static inline unsigned int expand(struct zone *zone, struct page *page, in= t low, - int high, int migratetype) + int high, freetype_t freetype) { unsigned int size =3D 1 << high; unsigned int nr_added =3D 0; @@ -1703,7 +1746,7 @@ static inline unsigned int expand(struct zone *zone, = struct page *page, int low, if (set_page_guard(zone, &page[size], high)) continue; =20 - __add_to_free_list(&page[size], zone, high, migratetype, false); + __add_to_free_list(&page[size], zone, high, freetype, false); set_buddy_order(&page[size], high); nr_added +=3D size; } @@ -1713,12 +1756,13 @@ static inline unsigned int expand(struct zone *zone= , struct page *page, int low, =20 static __always_inline void page_del_and_expand(struct zone *zone, struct page *page, int low, - int high, int migratetype) + int high, freetype_t freetype) { + enum migratetype migratetype =3D free_to_migratetype(freetype); int nr_pages =3D 1 << high; =20 __del_page_from_free_list(page, zone, high, migratetype); - nr_pages -=3D expand(zone, page, low, high, migratetype); + nr_pages -=3D expand(zone, page, low, high, freetype); account_freepages(zone, -nr_pages, migratetype); } =20 @@ -1877,7 +1921,7 @@ static void prep_new_page(struct page *page, unsigned= int order, gfp_t gfp_flags */ static __always_inline struct page *__rmqueue_smallest(struct zone *zone, unsigned int order, - int migratetype) + freetype_t freetype) { unsigned int current_order; struct free_area *area; @@ -1885,13 +1929,15 @@ struct page *__rmqueue_smallest(struct zone *zone, = unsigned int order, =20 /* Find a page of the appropriate size in the preferred list */ for (current_order =3D order; current_order < NR_PAGE_ORDERS; ++current_o= rder) { + enum migratetype migratetype =3D free_to_migratetype(freetype); + area =3D &(zone->free_area[current_order]); - page =3D get_page_from_free_area(area, migratetype); + page =3D get_page_from_free_area(area, freetype); if (!page) continue; =20 page_del_and_expand(zone, page, order, current_order, - migratetype); + freetype); trace_mm_page_alloc_zone_locked(page, order, migratetype, pcp_allowed_order(order) && migratetype < MIGRATE_PCPTYPES); @@ -1916,13 +1962,18 @@ static int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPT= YPES - 1] =3D { =20 #ifdef CONFIG_CMA static __always_inline struct page *__rmqueue_cma_fallback(struct zone *zo= ne, - unsigned int order) + unsigned int order, bool sensitive) { - return __rmqueue_smallest(zone, order, MIGRATE_CMA); + freetype_t freetype =3D migrate_to_freetype(MIGRATE_CMA, sensitive); + + return __rmqueue_smallest(zone, order, freetype); } #else static inline struct page *__rmqueue_cma_fallback(struct zone *zone, - unsigned int order) { return NULL; } + unsigned int order, bool sensitive) +{ + return NULL; +} #endif =20 /* @@ -1930,7 +1981,7 @@ static inline struct page *__rmqueue_cma_fallback(str= uct zone *zone, * change the block type. */ static int __move_freepages_block(struct zone *zone, unsigned long start_p= fn, - int old_mt, int new_mt) + freetype_t old_ft, freetype_t new_ft) { struct page *page; unsigned long pfn, end_pfn; @@ -1953,7 +2004,7 @@ static int __move_freepages_block(struct zone *zone, = unsigned long start_pfn, =20 order =3D buddy_order(page); =20 - move_to_free_list(page, zone, order, old_mt, new_mt); + move_to_free_list(page, zone, order, old_ft, new_ft); =20 pfn +=3D 1 << order; pages_moved +=3D 1 << order; @@ -2013,7 +2064,7 @@ static bool prep_move_freepages_block(struct zone *zo= ne, struct page *page, } =20 static int move_freepages_block(struct zone *zone, struct page *page, - int old_mt, int new_mt) + freetype_t old_ft, freetype_t new_ft) { unsigned long start_pfn; int res; @@ -2021,8 +2072,11 @@ static int move_freepages_block(struct zone *zone, s= truct page *page, if (!prep_move_freepages_block(zone, page, &start_pfn, NULL, NULL)) return -1; =20 - res =3D __move_freepages_block(zone, start_pfn, old_mt, new_mt); - set_pageblock_migratetype(pfn_to_page(start_pfn), new_mt); + VM_BUG_ON(freetype_sensitive(old_ft) !=3D freetype_sensitive(new_ft)); + + res =3D __move_freepages_block(zone, start_pfn, old_ft, new_ft); + set_pageblock_migratetype(pfn_to_page(start_pfn), + free_to_migratetype(new_ft)); =20 return res; =20 @@ -2090,8 +2144,7 @@ static bool __move_freepages_block_isolate(struct zon= e *zone, struct page *page, bool isolate) { unsigned long start_pfn, buddy_pfn; - int from_mt; - int to_mt; + freetype_t block_ft, from_ft, to_ft; struct page *buddy; =20 if (isolate =3D=3D get_pageblock_isolate(page)) { @@ -2247,14 +2300,15 @@ static bool should_try_claim_block(unsigned int ord= er, int start_mt) =20 /* * Check whether there is a suitable fallback freepage with requested orde= r. - * If claimable is true, this function returns fallback_mt only if + * If claimable is true, this function returns a fallback only if * we would do this whole-block claiming. This would help to reduce * fragmentation due to mixed migratetype pages in one pageblock. */ int find_suitable_fallback(struct free_area *area, unsigned int order, - int migratetype, bool claimable) + freetype_t freetype, bool claimable) { int i; + enum migratetype migratetype =3D free_to_migratetype(freetype); =20 if (claimable && !should_try_claim_block(order, migratetype)) return -2; @@ -2264,9 +2318,11 @@ int find_suitable_fallback(struct free_area *area, u= nsigned int order, =20 for (i =3D 0; i < MIGRATE_PCPTYPES - 1 ; i++) { int fallback_mt =3D fallbacks[migratetype][i]; + freetype_t fallback_ft =3D migrate_to_freetype(fallback_mt, + freetype_sensitive(freetype)); =20 - if (!free_area_empty(area, fallback_mt)) - return fallback_mt; + if (!free_area_empty(area, fallback_ft)) + return fallback_ft.type; } =20 return -1; @@ -2281,20 +2337,22 @@ int find_suitable_fallback(struct free_area *area, = unsigned int order, */ static struct page * try_to_claim_block(struct zone *zone, struct page *page, - int current_order, int order, int start_type, - int block_type, unsigned int alloc_flags) + int current_order, int order, freetype_t start_type, + freetype_t block_type, unsigned int alloc_flags) { int free_pages, movable_pages, alike_pages; + int block_mt =3D free_to_migratetype(block_type); + int start_mt =3D free_to_migratetype(start_type); unsigned long start_pfn; =20 /* Take ownership for orders >=3D pageblock_order */ if (current_order >=3D pageblock_order) { unsigned int nr_added; =20 - del_page_from_free_list(page, zone, current_order, block_type); - change_pageblock_range(page, current_order, start_type); + del_page_from_free_list(page, zone, current_order, block_mt); + change_pageblock_range(page, current_order, start_mt); nr_added =3D expand(zone, page, order, current_order, start_type); - account_freepages(zone, nr_added, start_type); + account_freepages(zone, nr_added, start_mt); return page; } =20 @@ -2316,7 +2374,7 @@ try_to_claim_block(struct zone *zone, struct page *pa= ge, * For movable allocation, it's the number of movable pages which * we just obtained. For other types it's a bit more tricky. */ - if (start_type =3D=3D MIGRATE_MOVABLE) { + if (start_mt =3D=3D MIGRATE_MOVABLE) { alike_pages =3D movable_pages; } else { /* @@ -2326,7 +2384,7 @@ try_to_claim_block(struct zone *zone, struct page *pa= ge, * vice versa, be conservative since we can't distinguish the * exact migratetype of non-movable pages. */ - if (block_type =3D=3D MIGRATE_MOVABLE) + if (block_mt =3D=3D MIGRATE_MOVABLE) alike_pages =3D pageblock_nr_pages - (free_pages + movable_pages); else @@ -2339,7 +2397,7 @@ try_to_claim_block(struct zone *zone, struct page *pa= ge, if (free_pages + alike_pages >=3D (1 << (pageblock_order-1)) || page_group_by_mobility_disabled) { __move_freepages_block(zone, start_pfn, block_type, start_type); - set_pageblock_migratetype(pfn_to_page(start_pfn), start_type); + set_pageblock_migratetype(pfn_to_page(start_pfn), start_mt); return __rmqueue_smallest(zone, order, start_type); } =20 @@ -2355,14 +2413,14 @@ try_to_claim_block(struct zone *zone, struct page *= page, * condition simpler. */ static __always_inline struct page * -__rmqueue_claim(struct zone *zone, int order, int start_migratetype, +__rmqueue_claim(struct zone *zone, int order, freetype_t start_freetype, unsigned int alloc_flags) { struct free_area *area; int current_order; int min_order =3D order; struct page *page; - int fallback_mt; + int fallback; =20 /* * Do not steal pages from freelists belonging to other pageblocks @@ -2379,25 +2437,29 @@ __rmqueue_claim(struct zone *zone, int order, int s= tart_migratetype, */ for (current_order =3D MAX_PAGE_ORDER; current_order >=3D min_order; --current_order) { + int start_mt =3D free_to_migratetype(start_freetype); + freetype_t fallback_ft; + area =3D &(zone->free_area[current_order]); - fallback_mt =3D find_suitable_fallback(area, current_order, - start_migratetype, true); + fallback =3D find_suitable_fallback(area, current_order, + start_freetype, true); =20 /* No block in that order */ - if (fallback_mt =3D=3D -1) + if (fallback =3D=3D -1) continue; =20 /* Advanced into orders too low to claim, abort */ - if (fallback_mt =3D=3D -2) + if (fallback =3D=3D -2) break; =20 - page =3D get_page_from_free_area(area, fallback_mt); + fallback_ft.type =3D fallback; + page =3D get_page_from_free_area(area, fallback_ft); page =3D try_to_claim_block(zone, page, current_order, order, - start_migratetype, fallback_mt, + start_freetype, fallback_ft, alloc_flags); if (page) { trace_mm_page_alloc_extfrag(page, order, current_order, - start_migratetype, fallback_mt); + start_mt, free_to_migratetype(fallback_ft)); return page; } } @@ -2410,24 +2472,26 @@ __rmqueue_claim(struct zone *zone, int order, int s= tart_migratetype, * the block as its current migratetype, potentially causing fragmentation. */ static __always_inline struct page * -__rmqueue_steal(struct zone *zone, int order, int start_migratetype) +__rmqueue_steal(struct zone *zone, int order, freetype_t start_freetype) { struct free_area *area; int current_order; struct page *page; - int fallback_mt; =20 for (current_order =3D order; current_order < NR_PAGE_ORDERS; current_ord= er++) { + freetype_t fallback_ft; + area =3D &(zone->free_area[current_order]); - fallback_mt =3D find_suitable_fallback(area, current_order, - start_migratetype, false); - if (fallback_mt =3D=3D -1) + fallback_ft.type =3D find_suitable_fallback(area, current_order, + start_freetype, false); + if (fallback_ft.type =3D=3D -1) continue; =20 - page =3D get_page_from_free_area(area, fallback_mt); - page_del_and_expand(zone, page, order, current_order, fallback_mt); + page =3D get_page_from_free_area(area, fallback_ft); + page_del_and_expand(zone, page, order, current_order, fallback_ft); trace_mm_page_alloc_extfrag(page, order, current_order, - start_migratetype, fallback_mt); + free_to_migratetype(start_freetype), + free_to_migratetype(fallback_ft)); return page; } =20 @@ -2446,7 +2510,7 @@ enum rmqueue_mode { * Call me with the zone->lock already held. */ static __always_inline struct page * -__rmqueue(struct zone *zone, unsigned int order, int migratetype, +__rmqueue(struct zone *zone, unsigned int order, freetype_t freetype, unsigned int alloc_flags, enum rmqueue_mode *mode) { struct page *page; @@ -2460,7 +2524,8 @@ __rmqueue(struct zone *zone, unsigned int order, int = migratetype, if (alloc_flags & ALLOC_CMA && zone_page_state(zone, NR_FREE_CMA_PAGES) > zone_page_state(zone, NR_FREE_PAGES) / 2) { - page =3D __rmqueue_cma_fallback(zone, order); + page =3D __rmqueue_cma_fallback(zone, order, + freetype_sensitive(freetype)); if (page) return page; } @@ -2477,13 +2542,14 @@ __rmqueue(struct zone *zone, unsigned int order, in= t migratetype, */ switch (*mode) { case RMQUEUE_NORMAL: - page =3D __rmqueue_smallest(zone, order, migratetype); + page =3D __rmqueue_smallest(zone, order, freetype); if (page) return page; fallthrough; case RMQUEUE_CMA: if (alloc_flags & ALLOC_CMA) { - page =3D __rmqueue_cma_fallback(zone, order); + page =3D __rmqueue_cma_fallback(zone, order, + freetype_sensitive(freetype)); if (page) { *mode =3D RMQUEUE_CMA; return page; @@ -2491,7 +2557,7 @@ __rmqueue(struct zone *zone, unsigned int order, int = migratetype, } fallthrough; case RMQUEUE_CLAIM: - page =3D __rmqueue_claim(zone, order, migratetype, alloc_flags); + page =3D __rmqueue_claim(zone, order, freetype, alloc_flags); if (page) { /* Replenished preferred freelist, back to normal mode. */ *mode =3D RMQUEUE_NORMAL; @@ -2500,7 +2566,7 @@ __rmqueue(struct zone *zone, unsigned int order, int = migratetype, fallthrough; case RMQUEUE_STEAL: if (!(alloc_flags & ALLOC_NOFRAGMENT)) { - page =3D __rmqueue_steal(zone, order, migratetype); + page =3D __rmqueue_steal(zone, order, freetype); if (page) { *mode =3D RMQUEUE_STEAL; return page; @@ -2517,7 +2583,7 @@ __rmqueue(struct zone *zone, unsigned int order, int = migratetype, */ static int rmqueue_bulk(struct zone *zone, unsigned int order, unsigned long count, struct list_head *list, - int migratetype, unsigned int alloc_flags) + freetype_t freetype, unsigned int alloc_flags) { enum rmqueue_mode rmqm =3D RMQUEUE_NORMAL; unsigned long flags; @@ -2530,7 +2596,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned i= nt order, spin_lock_irqsave(&zone->lock, flags); } for (i =3D 0; i < count; ++i) { - struct page *page =3D __rmqueue(zone, order, migratetype, + struct page *page =3D __rmqueue(zone, order, freetype, alloc_flags, &rmqm); if (unlikely(page =3D=3D NULL)) break; @@ -2815,8 +2881,8 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, str= uct zone *zone, } =20 static void free_frozen_page_commit(struct zone *zone, - struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order, fpi_t fpi_flags) + struct per_cpu_pages *pcp, struct page *page, + freetype_t freetype, unsigned int order, fpi_t fpi_flags) { int high, batch; int pindex; @@ -2829,7 +2895,7 @@ static void free_frozen_page_commit(struct zone *zone, */ pcp->alloc_factor >>=3D 1; __count_vm_events(PGFREE, 1 << order); - pindex =3D order_to_pindex(migratetype, order); + pindex =3D order_to_pindex(free_to_migratetype(freetype), order); list_add(&page->pcp_list, &pcp->lists[pindex]); pcp->count +=3D 1 << order; =20 @@ -2896,6 +2962,7 @@ static void __free_frozen_pages(struct page *page, un= signed int order, struct zone *zone; unsigned long pfn =3D page_to_pfn(page); int migratetype; + freetype_t freetype; =20 if (!pcp_allowed_order(order)) { __free_pages_ok(page, order, fpi_flags); @@ -2913,13 +2980,14 @@ static void __free_frozen_pages(struct page *page, = unsigned int order, * excessively into the page allocator */ zone =3D page_zone(page); - migratetype =3D get_pfnblock_migratetype(page, pfn); + freetype =3D get_pfnblock_freetype(page, pfn); + migratetype =3D free_to_migratetype(freetype); if (unlikely(migratetype >=3D MIGRATE_PCPTYPES)) { if (unlikely(is_migrate_isolate(migratetype))) { free_one_page(zone, page, pfn, order, fpi_flags); return; } - migratetype =3D MIGRATE_MOVABLE; + freetype =3D freetype_with_migrate(freetype, MIGRATE_MOVABLE); } =20 if (unlikely((fpi_flags & FPI_TRYLOCK) && IS_ENABLED(CONFIG_PREEMPT_RT) @@ -2930,7 +2998,7 @@ static void __free_frozen_pages(struct page *page, un= signed int order, pcp_trylock_prepare(UP_flags); pcp =3D pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_frozen_page_commit(zone, pcp, page, migratetype, order, fpi_flags); + free_frozen_page_commit(zone, pcp, page, freetype, order, fpi_flags); pcp_spin_unlock(pcp); } else { free_one_page(zone, page, pfn, order, fpi_flags); @@ -2982,10 +3050,12 @@ void free_unref_folios(struct folio_batch *folios) struct zone *zone =3D folio_zone(folio); unsigned long pfn =3D folio_pfn(folio); unsigned int order =3D (unsigned long)folio->private; + freetype_t freetype; int migratetype; =20 folio->private =3D NULL; - migratetype =3D get_pfnblock_migratetype(&folio->page, pfn); + freetype =3D get_pfnblock_freetype(&folio->page, pfn); + migratetype =3D free_to_migratetype(freetype); =20 /* Different zone requires a different pcp lock */ if (zone !=3D locked_zone || @@ -3027,10 +3097,11 @@ void free_unref_folios(struct folio_batch *folios) * to the MIGRATE_MOVABLE pcp list. */ if (unlikely(migratetype >=3D MIGRATE_PCPTYPES)) - migratetype =3D MIGRATE_MOVABLE; + freetype =3D freetype_with_migrate(freetype, + MIGRATE_MOVABLE); =20 trace_mm_page_free_batched(&folio->page); - free_frozen_page_commit(zone, pcp, &folio->page, migratetype, + free_frozen_page_commit(zone, pcp, &folio->page, freetype, order, FPI_NONE); } =20 @@ -3091,14 +3162,16 @@ int __isolate_free_page(struct page *page, unsigned= int order) if (order >=3D pageblock_order - 1) { struct page *endpage =3D page + (1 << order) - 1; for (; page < endpage; page +=3D pageblock_nr_pages) { - int mt =3D get_pageblock_migratetype(page); + freetype_t old_ft =3D get_pageblock_freetype(page); + freetype_t new_ft =3D freetype_with_migrate(old_ft, + MIGRATE_MOVABLE); + /* * Only change normal pageblocks (i.e., they can merge * with others) */ if (migratetype_is_mergeable(mt)) - move_freepages_block(zone, page, mt, - MIGRATE_MOVABLE); + move_freepages_block(zone, page, old_ft, new_ft); } } =20 @@ -3114,7 +3187,8 @@ int __isolate_free_page(struct page *page, unsigned i= nt order) * This function is meant to return a page pulled from the free lists via * __isolate_free_page back to the free lists they were pulled from. */ -void __putback_isolated_page(struct page *page, unsigned int order, int mt) +void __putback_isolated_page(struct page *page, unsigned int order, + freetype_t freetype) { struct zone *zone =3D page_zone(page); =20 @@ -3122,7 +3196,7 @@ void __putback_isolated_page(struct page *page, unsig= ned int order, int mt) lockdep_assert_held(&zone->lock); =20 /* Return isolated page to tail of freelist. */ - __free_one_page(page, page_to_pfn(page), zone, order, mt, + __free_one_page(page, page_to_pfn(page), zone, order, freetype, FPI_SKIP_REPORT_NOTIFY | FPI_TO_TAIL); } =20 @@ -3155,10 +3229,12 @@ static inline void zone_statistics(struct zone *pre= ferred_zone, struct zone *z, static __always_inline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags, - int migratetype) + freetype_t freetype) { struct page *page; unsigned long flags; + freetype_t ft_high =3D freetype_with_migrate(freetype, + MIGRATE_HIGHATOMIC); =20 do { page =3D NULL; @@ -3169,11 +3245,11 @@ struct page *rmqueue_buddy(struct zone *preferred_z= one, struct zone *zone, spin_lock_irqsave(&zone->lock, flags); } if (alloc_flags & ALLOC_HIGHATOMIC) - page =3D __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); + page =3D __rmqueue_smallest(zone, order, ft_high); if (!page) { enum rmqueue_mode rmqm =3D RMQUEUE_NORMAL; =20 - page =3D __rmqueue(zone, order, migratetype, alloc_flags, &rmqm); + page =3D __rmqueue(zone, order, freetype, alloc_flags, &rmqm); =20 /* * If the allocation fails, allow OOM handling and @@ -3182,7 +3258,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zon= e, struct zone *zone, * high-order atomic allocation in the future. */ if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) - page =3D __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); + page =3D __rmqueue_smallest(zone, order, ft_high); =20 if (!page) { spin_unlock_irqrestore(&zone->lock, flags); @@ -3251,7 +3327,7 @@ static int nr_pcp_alloc(struct per_cpu_pages *pcp, st= ruct zone *zone, int order) /* Remove page from the per-cpu list, caller must protect the list */ static inline struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, - int migratetype, + freetype_t freetype, unsigned int alloc_flags, struct per_cpu_pages *pcp, struct list_head *list) @@ -3265,7 +3341,7 @@ struct page *__rmqueue_pcplist(struct zone *zone, uns= igned int order, =20 alloced =3D rmqueue_bulk(zone, order, batch, list, - migratetype, alloc_flags); + freetype, alloc_flags); =20 pcp->count +=3D alloced << order; if (unlikely(list_empty(list))) @@ -3283,7 +3359,7 @@ struct page *__rmqueue_pcplist(struct zone *zone, uns= igned int order, /* Lock and remove page from the per-cpu list */ static struct page *rmqueue_pcplist(struct zone *preferred_zone, struct zone *zone, unsigned int order, - int migratetype, unsigned int alloc_flags) + freetype_t freetype, unsigned int alloc_flags) { struct per_cpu_pages *pcp; struct list_head *list; @@ -3304,8 +3380,8 @@ static struct page *rmqueue_pcplist(struct zone *pref= erred_zone, * frees. */ pcp->free_count >>=3D 1; - list =3D &pcp->lists[order_to_pindex(migratetype, order)]; - page =3D __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, li= st); + list =3D &pcp->lists[order_to_pindex(free_to_migratetype(freetype), order= )]; + page =3D __rmqueue_pcplist(zone, order, freetype, alloc_flags, pcp, list); pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); if (page) { @@ -3331,19 +3407,19 @@ static inline struct page *rmqueue(struct zone *preferred_zone, struct zone *zone, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags, - int migratetype) + freetype_t freetype) { struct page *page; =20 if (likely(pcp_allowed_order(order))) { page =3D rmqueue_pcplist(preferred_zone, zone, order, - migratetype, alloc_flags); + freetype, alloc_flags); if (likely(page)) goto out; } =20 page =3D rmqueue_buddy(preferred_zone, zone, order, alloc_flags, - migratetype); + freetype); =20 out: /* Separate test+clear to avoid unnecessary atomics */ @@ -3365,7 +3441,7 @@ struct page *rmqueue(struct zone *preferred_zone, static void reserve_highatomic_pageblock(struct page *page, int order, struct zone *zone) { - int mt; + freetype_t ft, ft_high; unsigned long max_managed, flags; =20 /* @@ -3387,13 +3463,14 @@ static void reserve_highatomic_pageblock(struct pag= e *page, int order, goto out_unlock; =20 /* Yoink! */ - mt =3D get_pageblock_migratetype(page); + ft =3D get_pageblock_freetype(page); /* Only reserve normal pageblocks (i.e., they can merge with others) */ - if (!migratetype_is_mergeable(mt)) + if (!migratetype_is_mergeable(free_to_migratetype(ft))) goto out_unlock; =20 + ft_high =3D freetype_with_migrate(ft, MIGRATE_HIGHATOMIC); if (order < pageblock_order) { - if (move_freepages_block(zone, page, mt, MIGRATE_HIGHATOMIC) =3D=3D -1) + if (move_freepages_block(zone, page, ft, ft_high) =3D=3D -1) goto out_unlock; zone->nr_reserved_highatomic +=3D pageblock_nr_pages; } else { @@ -3438,9 +3515,11 @@ static bool unreserve_highatomic_pageblock(const str= uct alloc_context *ac, spin_lock_irqsave(&zone->lock, flags); for (order =3D 0; order < NR_PAGE_ORDERS; order++) { struct free_area *area =3D &(zone->free_area[order]); + freetype_t ft_high =3D freetype_with_migrate(ac->freetype, + MIGRATE_HIGHATOMIC); unsigned long size; =20 - page =3D get_page_from_free_area(area, MIGRATE_HIGHATOMIC); + page =3D get_page_from_free_area(area, ft_high); if (!page) continue; =20 @@ -3467,14 +3546,14 @@ static bool unreserve_highatomic_pageblock(const st= ruct alloc_context *ac, */ if (order < pageblock_order) ret =3D move_freepages_block(zone, page, - MIGRATE_HIGHATOMIC, - ac->migratetype); + ft_high, + ac->freetype); else { move_to_free_list(page, zone, order, - MIGRATE_HIGHATOMIC, - ac->migratetype); + ft_high, + ac->freetype); change_pageblock_range(page, order, - ac->migratetype); + free_to_migratetype(ac->freetype)); ret =3D 1; } /* @@ -3580,18 +3659,18 @@ bool __zone_watermark_ok(struct zone *z, unsigned i= nt order, unsigned long mark, continue; =20 for (mt =3D 0; mt < MIGRATE_PCPTYPES; mt++) { - if (!free_area_empty(area, mt)) + if (!free_areas_empty(area, mt)) return true; } =20 #ifdef CONFIG_CMA if ((alloc_flags & ALLOC_CMA) && - !free_area_empty(area, MIGRATE_CMA)) { + !free_areas_empty(area, MIGRATE_CMA)) { return true; } #endif if ((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_OOM)) && - !free_area_empty(area, MIGRATE_HIGHATOMIC)) { + !free_areas_empty(area, MIGRATE_HIGHATOMIC)) { return true; } } @@ -3715,7 +3794,7 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp= _t gfp_mask, unsigned int alloc_flags) { #ifdef CONFIG_CMA - if (gfp_migratetype(gfp_mask) =3D=3D MIGRATE_MOVABLE) + if (free_to_migratetype(gfp_freetype(gfp_mask)) =3D=3D MIGRATE_MOVABLE) alloc_flags |=3D ALLOC_CMA; #endif return alloc_flags; @@ -3878,7 +3957,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int o= rder, int alloc_flags, =20 try_this_zone: page =3D rmqueue(zonelist_zone(ac->preferred_zoneref), zone, order, - gfp_mask, alloc_flags, ac->migratetype); + gfp_mask, alloc_flags, ac->freetype); if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); =20 @@ -4644,6 +4723,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, unsigned int cpuset_mems_cookie; unsigned int zonelist_iter_cookie; int reserve_flags; + enum migratetype migratetype; =20 if (unlikely(nofail)) { /* @@ -4714,6 +4794,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, if (page) goto got_pg; =20 + migratetype =3D free_to_migratetype(ac->freetype); + /* * For costly allocations, try direct compaction first, as it's likely * that we have enough base pages and don't need to reclaim. For non- @@ -4725,7 +4807,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, */ if (can_direct_reclaim && can_compact && (costly_order || - (order > 0 && ac->migratetype !=3D MIGRATE_MOVABLE)) + (order > 0 && migratetype !=3D MIGRATE_MOVABLE)) && !gfp_pfmemalloc_allowed(gfp_mask)) { page =3D __alloc_pages_direct_compact(gfp_mask, order, alloc_flags, ac, @@ -4933,7 +5015,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask= , unsigned int order, ac->highest_zoneidx =3D gfp_zone(gfp_mask); ac->zonelist =3D node_zonelist(preferred_nid, gfp_mask); ac->nodemask =3D nodemask; - ac->migratetype =3D gfp_migratetype(gfp_mask); + ac->freetype =3D gfp_freetype(gfp_mask); =20 if (cpusets_enabled()) { *alloc_gfp |=3D __GFP_HARDWALL; @@ -5094,7 +5176,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int = preferred_nid, goto failed_irq; =20 /* Attempt the batch allocation */ - pcp_list =3D &pcp->lists[order_to_pindex(ac.migratetype, 0)]; + pcp_list =3D &pcp->lists[order_to_pindex(free_to_migratetype(ac.freetype)= , 0)]; while (nr_populated < nr_pages) { =20 /* Skip existing pages */ @@ -5103,7 +5185,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int = preferred_nid, continue; } =20 - page =3D __rmqueue_pcplist(zone, 0, ac.migratetype, alloc_flags, + page =3D __rmqueue_pcplist(zone, 0, ac.freetype, alloc_flags, pcp, pcp_list); if (unlikely(!page)) { /* Try and allocate at least one page */ @@ -5208,7 +5290,8 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, u= nsigned int order, page =3D NULL; } =20 - trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + trace_mm_page_alloc(page, order, alloc_gfp, + free_to_migratetype(ac.freetype)); kmsan_alloc_page(page, order, alloc_gfp); =20 return page; @@ -7607,7 +7690,8 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t g= fp_flags, int nid, unsigned __free_frozen_pages(page, order, FPI_TRYLOCK); page =3D NULL; } - trace_mm_page_alloc(page, order, alloc_gfp, ac.migratetype); + trace_mm_page_alloc(page, order, alloc_gfp, + free_to_migratetype(ac.freetype)); kmsan_alloc_page(page, order, alloc_gfp); return page; } diff --git a/mm/page_isolation.c b/mm/page_isolation.c index f72b6cd38b958be97edcea9ce20154ff43131a4a..572128767a34d87cfc7ba856e78= 860e06706730d 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -265,7 +265,7 @@ static void unset_migratetype_isolate(struct page *page) WARN_ON_ONCE(!pageblock_unisolate_and_move_free_pages(zone, page)); } else { clear_pageblock_isolate(page); - __putback_isolated_page(page, order, get_pageblock_migratetype(page)); + __putback_isolated_page(page, order, get_pageblock_freetype(page)); } zone->nr_isolate_pageblock--; out: diff --git a/mm/page_owner.c b/mm/page_owner.c index c3ca21132c2c18e77cd4b6b3edb586fc1ac3cba7..355012309cea1373759e81aa1e4= 5220160250801 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -469,7 +469,8 @@ void pagetypeinfo_showmixedcount_print(struct seq_file = *m, goto ext_put_continue; =20 page_owner =3D get_page_owner(page_ext); - page_mt =3D gfp_migratetype(page_owner->gfp_mask); + page_mt =3D free_to_migratetype( + gfp_freetype(page_owner->gfp_mask)); if (pageblock_mt !=3D page_mt) { if (is_migrate_cma(pageblock_mt)) count[MIGRATE_MOVABLE]++; @@ -554,7 +555,7 @@ print_page_owner(char __user *buf, size_t count, unsign= ed long pfn, =20 /* Print information relevant to grouping pages by mobility */ pageblock_mt =3D get_pageblock_migratetype(page); - page_mt =3D gfp_migratetype(page_owner->gfp_mask); + page_mt =3D free_to_migratetype(gfp_freetype(page_owner->gfp_mask)); ret +=3D scnprintf(kbuf + ret, count - ret, "PFN 0x%lx type %s Block %lu type %s Flags %pGp\n", pfn, @@ -605,7 +606,7 @@ void __dump_page_owner(const struct page *page) =20 page_owner =3D get_page_owner(page_ext); gfp_mask =3D page_owner->gfp_mask; - mt =3D gfp_migratetype(gfp_mask); + mt =3D free_to_migratetype(gfp_freetype(gfp_mask)); =20 if (!test_bit(PAGE_EXT_OWNER, &page_ext->flags)) { pr_alert("page_owner info is not present (never set?)\n"); diff --git a/mm/page_reporting.c b/mm/page_reporting.c index e4c428e61d8c1765ae00ee22818cfadfd27f324c..faf0347e795cabf8a52ba3a35b5= 483ea0a8e1934 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -113,10 +113,10 @@ page_reporting_drain(struct page_reporting_dev_info *= prdev, */ do { struct page *page =3D sg_page(sg); - int mt =3D get_pageblock_migratetype(page); + freetype_t ft =3D get_pageblock_freetype(page); unsigned int order =3D get_order(sg->length); =20 - __putback_isolated_page(page, order, mt); + __putback_isolated_page(page, order, ft); =20 /* If the pages were not reported due to error skip flagging */ if (!reported) diff --git a/mm/show_mem.c b/mm/show_mem.c index 3a4b5207635da8e5224ace454badb99ca017170d..f0d9cbd37b0c0297cfb57349ce2= 3df7d52b98d97 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -374,7 +374,7 @@ static void show_free_areas(unsigned int filter, nodema= sk_t *nodemask, int max_z =20 types[order] =3D 0; for (type =3D 0; type < MIGRATE_TYPES; type++) { - if (!free_area_empty(area, type)) + if (!free_areas_empty(area, type)) types[order] |=3D 1 << type; } } --=20 2.50.1 From nobody Thu Oct 2 00:49:29 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABDBA30FC1C for ; Wed, 24 Sep 2025 15:00:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; cv=none; b=XUgo4k3g5OPeW4zTMjhzFujwm/sxhy1PTaSfgtvaC0DmEK+o0dBKimzt1CcrUBTIFfmqRXY+G1+w07Igroy6/X2jiT9T6FrbM3fW91ulxGZCOTu4e9XiArGmQCRyBHW+UesnzM6p9KDaVJBJxf+WGo1vtR4ysnPHcR6YbO/Abvo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; c=relaxed/simple; bh=zww4py4rn1HpG/CMzPogS9MTCdLxOtkn2Lwj7BfMBcc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BtZcYDjRjJAQV25Pb+Alw3JskXS3jbhlfoedTBqHxe0IavXcW9W9A2YwusiEiKUuyBQox7X9d17PW8cREAivzNh0yx40g0TztI+/imq6N0xfUD08Z7oWnMw4E4Cdpbjh8Ypi7jgsVlCFu/FMBkyLn4g07fzelQM6XQwMkOyyCSA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=acP6UjTO; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="acP6UjTO" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e23a66122so11668785e9.0 for ; Wed, 24 Sep 2025 08:00:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726020; x=1759330820; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+FAWLg0cAZSQEdsXqkfb3D+Y+KS6zrWexbvzTaWhCuA=; b=acP6UjTOJy6Y8w7ua7U295KGUf4rD7MOJHo5n9L9Jmoj4CFMRL8h5Wn/9RgyMGHyHL TT3IkERl0VgrbFaZ6xGvOc5pB0x/RqmlXo6JqEpGNDRByzhZ8MwgmssI1pcngwKfGbXc o3KDVRSM08jevRvHGc30vKfxNy6M+mlbWFXRYt7RW+t/KyAx0mGU5g5nwwavWqQig9wa 0j8/8UZg+pZtPjM3RZzkm7Z1s2YsQ5JVBOFHesjPYkQe/xmPgdg9J4uhZ00utiEmSOGP 2EfuiUH2g2Hr6vGfi5tMbZmyf75WMubMNKDp2YtEGYkPJDoF46VvAwzWsbEakzyoxF7D FpUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726020; x=1759330820; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+FAWLg0cAZSQEdsXqkfb3D+Y+KS6zrWexbvzTaWhCuA=; b=OD6+q54vcgBgnO8JGpfMLwElgN/V+htH32SS5lKjweYcRKxEWlfbTsLlFqM3bUo6z3 e9XyYBI3dwiZFzfGpTAtjiQdtg0BHw6MSTn52ZOeOYCCd+tPYYj+SVk2I9baka2UbuuP rKMbrWcrmENStC9F3DecHGfZXPwRDBSz8NBNJFul9WKtDtDOLdd8jccpCvLlhkx9TQoH ydiZ9HiOwPV32BHHjcOCOeGtRmE9iH8/nrUr0HN+CXlxdSYOP29yzZJTcu6uczi8/Sx/ T1MnFtjigQhL79CPceyy3XPWbqzREky7/0IFiPzl4ebkqyscfC/QHdgOq9LP43ppRGjC GG4w== X-Forwarded-Encrypted: i=1; AJvYcCW45fR7fuUGHV+HoXfT7qG/GVDNKqOJgySS4Vn4qkBdtWk/bAOPOHOjnqPEJs27xR/hlFclNPdKDFRkHrY=@vger.kernel.org X-Gm-Message-State: AOJu0YyC9S2BZ4ZfijhYsu9uSeme2lPxAphA0l/CXaD75w5HwFkofpm7 ZFRWg390Zm28TgTzk8uusm39+LF8LdKxGkJBVt07jwCzqFIGRN47v0IiSbWUKV+IuYgmph1WINu MwR6CIgkmNm0vtA== X-Google-Smtp-Source: AGHT+IEd3Rjl3nI9f3Je8VbfFMCNjxpjDcsgVNyf8yRf3esq/kDU/77UhI5eXqehOFhjda6lFEvLmXxGBzJK+Q== X-Received: from wmrm14.prod.google.com ([2002:a05:600c:37ce:b0:46e:2f1b:4ceb]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4353:b0:46e:2c3c:d6b with SMTP id 5b1f17b1804b1-46e329f8ac5mr968665e9.22.1758726019835; Wed, 24 Sep 2025 08:00:19 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:47 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-12-2d861768041f@google.com> Subject: [PATCH 12/21] mm/asi: encode sensitivity in freetypes and pageblocks From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Now that there is a higher-level concept for encoding the "type" of a collection of freelists, use this to encode sensitivity, i.e. whether pages are currently mapped into ASI restricted address spaces. Just like with migratetypes, the sensitivity of a page needs to be looked up from the pageblock flags when it is freed, so add a bit for that too. Increase the number of freelists and update the pcplist index-mapping logic to encode sensitivity as another dimension. Then update NR_SENSITIVITIES and code that iterates over freelists to be aware of the new set of lists. Blocks of differing sensitivity cannot be merged, so update __free_one_page() to reflect that. Finally, update __move_freepages_block_isolate() to be aware that it of which sensitivity's freelists it needs to manipulate. Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 2 +- include/linux/mmzone.h | 31 ++++++++++-- include/linux/pageblock-flags.h | 18 +++++++ mm/internal.h | 10 +++- mm/page_alloc.c | 104 ++++++++++++++++++++++++++----------= ---- 5 files changed, 123 insertions(+), 42 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index a275171c5a6aecafd7783e57ce7d4316c5e56655..a186a932f19e7c450e6e6b9a5f6= e592f6e8f2bed 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -35,7 +35,7 @@ static inline freetype_t gfp_freetype(const gfp_t gfp_fla= gs) >> GFP_MOVABLE_SHIFT; } =20 - return migrate_to_freetype(migratetype, false); + return migrate_to_freetype(migratetype, gfp_flags & __GFP_SENSITIVE); } #undef GFP_MOVABLE_MASK #undef GFP_MOVABLE_SHIFT diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 56310722f38b788154ee15845b6877ed7e70d6b7..c16e2c1581c8ec0cb241ab340f8= e8f65717b0cdb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -5,6 +5,7 @@ #ifndef __ASSEMBLY__ #ifndef __GENERATING_BOUNDS_H =20 +#include #include #include #include @@ -123,7 +124,11 @@ static inline bool migratetype_is_mergeable(int mt) return mt < MIGRATE_PCPTYPES; } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define NR_SENSITIVITIES 2 +#else #define NR_SENSITIVITIES 1 +#endif =20 /* * A freetype is the index used to identify free lists (free area lists and @@ -141,18 +146,30 @@ static inline freetype_t migrate_to_freetype(enum mig= ratetype mt, bool sensitive { freetype_t freetype; =20 - freetype.type =3D mt; + /* + * When ASI is off, .sensitive is meaningless. Set it to false so that + * freetype values are the same when asi=3Doff as when ASI is + * compiled out. + */ + if (!asi_enabled_static()) + sensitive =3D false; + + freetype.type =3D (MIGRATE_TYPES * sensitive) + mt; return freetype; } =20 static inline enum migratetype free_to_migratetype(freetype_t freetype) { - return freetype.type; + VM_WARN_ON_ONCE(!asi_enabled_static() && freetype.type >=3D MIGRATE_TYPES= ); + return freetype.type % MIGRATE_TYPES; } =20 static inline bool freetype_sensitive(freetype_t freetype) { - return false; + bool sensitive =3D freetype.type / MIGRATE_TYPES; + + VM_WARN_ON_ONCE(!asi_enabled_static() && sensitive); + return sensitive; } =20 /* Convenience helper, return the freetype modified to have the migratetyp= e. */ @@ -174,10 +191,14 @@ static inline bool freetypes_equal(freetype_t a, free= type_t b) =20 #define for_each_free_list(list, zone) \ for (unsigned int order =3D 0; order < NR_PAGE_ORDERS; order++) \ - for (unsigned int type =3D 0; \ - list =3D &zone->free_area[order].free_list[type], \ + for (unsigned int type =3D 0;\ type < MIGRATE_TYPES; \ type++) \ + for (int sensitive =3D 0; \ + list =3D free_area_list(&zone->free_area[order], \ + migrate_to_freetype(type, sensitive)), \ + sensitive < NR_SENSITIVITIES; \ + sensitive++) =20 extern int page_group_by_mobility_disabled; =20 diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flag= s.h index 13457e920e892c1c5083e0dc63e2ecfbed88f60e..289542ce027ca937cbad8dfed37= cd2b35e5f3ab5 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -18,6 +18,14 @@ enum pageblock_bits { PB_migrate_0, PB_migrate_1, PB_migrate_2, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + /* + * Block is mapped into restricted address spaces. Having a + * "nonsensitive" flag instead of a "sensitive" flag is convenient + * so that the initial value of 0 is correct at boot. + */ + PB_nonsensitive, +#endif PB_compact_skip,/* If set the block is skipped by compaction */ =20 #ifdef CONFIG_MEMORY_ISOLATION @@ -44,6 +52,16 @@ enum pageblock_bits { #define PAGEBLOCK_ISO_MASK 0 #endif =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define PAGEBLOCK_NONSENSITIVE_MASK BIT(PB_nonsensitive) +#else +#define PAGEBLOCK_NONSENSITIVE_MASK 0 +#endif + +#define PAGEBLOCK_FREETYPE_MASK (PAGEBLOCK_MIGRATETYPE_MASK | \ + PAGEBLOCK_ISO_MASK | \ + PAGEBLOCK_NONSENSITIVE_MASK) + #if defined(CONFIG_HUGETLB_PAGE) =20 #ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE diff --git a/mm/internal.h b/mm/internal.h index 50ff6671f19d38a59c9f07e66d347baf85ddf085..0401412220a76a233e14a7ee7d6= 4c1194fc3759d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -960,7 +960,15 @@ static inline bool free_area_empty(struct free_area *a= rea, freetype_t freetype) =20 static inline bool free_areas_empty(struct free_area *area, int migratetyp= e) { - return free_area_empty(area, migrate_to_freetype(migratetype, false)); + bool sensitive; + + for_each_sensitivity(sensitive) { + freetype_t ft =3D migrate_to_freetype(migratetype, sensitive); + + if (!free_area_empty(area, ft)) + return false; + } + return true; } =20 /* mm/util.c */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 4ce81f8d4e59966b7c0c2902e24aa2f4639a0e59..5943b821089b72fd148bd93ee03= 5c0e70e45ec91 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -365,11 +365,8 @@ get_pfnblock_bitmap_bitidx(const struct page *page, un= signed long pfn, unsigned long *bitmap; unsigned long word_bitidx; =20 -#ifdef CONFIG_MEMORY_ISOLATION - BUILD_BUG_ON(NR_PAGEBLOCK_BITS !=3D 8); -#else - BUILD_BUG_ON(NR_PAGEBLOCK_BITS !=3D 4); -#endif + /* NR_PAGEBLOCK_BITS must divide word size. */ + BUILD_BUG_ON(NR_PAGEBLOCK_BITS !=3D 4 && NR_PAGEBLOCK_BITS !=3D 8); BUILD_BUG_ON(__MIGRATE_TYPE_END > PAGEBLOCK_MIGRATETYPE_MASK); VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); =20 @@ -442,9 +439,18 @@ __always_inline freetype_t __get_pfnblock_freetype(const struct page *page, unsigned long pfn, bool ignore_iso) { - int mt =3D get_pfnblock_migratetype(page, pfn); + unsigned long mask =3D PAGEBLOCK_FREETYPE_MASK; + enum migratetype migratetype; + unsigned long flags; =20 - return migrate_to_freetype(mt, false); + flags =3D __get_pfnblock_flags_mask(page, pfn, mask); + + migratetype =3D flags & PAGEBLOCK_MIGRATETYPE_MASK; +#ifdef CONFIG_MEMORY_ISOLATION + if (!ignore_iso && flags & BIT(PB_migrate_isolate)) + migratetype =3D MIGRATE_ISOLATE; +#endif + return migrate_to_freetype(migratetype, !(flags & PAGEBLOCK_NONSENSITIVE_= MASK)); } =20 /** @@ -601,7 +607,7 @@ void __meminit init_pageblock_migratetype(struct page *= page, flags |=3D BIT(PB_migrate_isolate); #endif __set_pfnblock_flags_mask(page, page_to_pfn(page), flags, - PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); + PAGEBLOCK_FREETYPE_MASK); } =20 #ifdef CONFIG_DEBUG_VM @@ -685,29 +691,39 @@ static void bad_page(struct page *page, const char *r= eason) add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); } =20 -static inline unsigned int order_to_pindex(int migratetype, int order) +static inline unsigned int order_to_pindex(freetype_t freetype, int order) { - if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { + int migratetype =3D free_to_migratetype(freetype); + /* pindex if the freetype is nonsensitive */ + int pindex_ns; + + VM_BUG_ON(!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + order > PAGE_ALLOC_COSTLY_ORDER); + + if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + order > PAGE_ALLOC_COSTLY_ORDER) { bool movable =3D migratetype =3D=3D MIGRATE_MOVABLE; =20 - if (order > PAGE_ALLOC_COSTLY_ORDER) { - VM_BUG_ON(order !=3D HPAGE_PMD_ORDER); - - return NR_LOWORDER_PCP_LISTS + movable; - } + VM_BUG_ON(order !=3D HPAGE_PMD_ORDER); + pindex_ns =3D NR_LOWORDER_PCP_LISTS + movable; } else { - VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); + pindex_ns =3D (MIGRATE_PCPTYPES * order) + migratetype; } =20 - return (MIGRATE_PCPTYPES * order) + migratetype; + return (NR_PCP_LISTS_PER_SENSITIVITY * freetype_sensitive(freetype)) + + pindex_ns; } =20 -static inline int pindex_to_order(unsigned int pindex) +inline int pindex_to_order(unsigned int pindex) { - int order =3D pindex / MIGRATE_PCPTYPES; + /* pindex if the freetype is nonsensitive */ + int pindex_ns =3D (pindex % NR_PCP_LISTS_PER_SENSITIVITY); + int order =3D pindex_ns / MIGRATE_PCPTYPES; + + VM_BUG_ON(pindex >=3D NR_PCP_LISTS); =20 if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { - if (pindex >=3D NR_LOWORDER_PCP_LISTS) + if (pindex_ns >=3D NR_LOWORDER_PCP_LISTS) order =3D HPAGE_PMD_ORDER; } else { VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER); @@ -950,6 +966,26 @@ buddy_merge_likely(unsigned long pfn, unsigned long bu= ddy_pfn, NULL) !=3D NULL; } =20 +/* + * Can pages of these two freetypes be combined into a single higher-order= free + * page? + */ +static inline bool can_merge_freetypes(freetype_t a, freetype_t b) +{ + if (freetypes_equal(a, b)) + return true; + + if (!migratetype_is_mergeable(free_to_migratetype(a)) || + !migratetype_is_mergeable(free_to_migratetype(b))) + return false; + + /* + * Mustn't merge differing sensitivities, changing the sensitivity + * requires changing pagetables. + */ + return freetype_sensitive(a) =3D=3D freetype_sensitive(b); +} + /* * Freeing function for a buddy system allocator. * @@ -1018,9 +1054,7 @@ static inline void __free_one_page(struct page *page, buddy_ft =3D get_pfnblock_freetype(buddy, buddy_pfn); buddy_mt =3D free_to_migratetype(buddy_ft); =20 - if (migratetype !=3D buddy_mt && - (!migratetype_is_mergeable(migratetype) || - !migratetype_is_mergeable(buddy_mt))) + if (!can_merge_freetypes(freetype, buddy_ft)) goto done_merging; } =20 @@ -1037,7 +1071,9 @@ static inline void __free_one_page(struct page *page, /* * Match buddy type. This ensures that an * expand() down the line puts the sub-blocks - * on the right freelists. + * on the right freelists. Sensitivity is + * already set correctly because of + * can_merge_freetypes(). */ set_pageblock_migratetype(buddy, migratetype); } @@ -2174,18 +2210,16 @@ static bool __move_freepages_block_isolate(struct z= one *zone, } =20 move: - /* Use PAGEBLOCK_MIGRATETYPE_MASK to get non-isolate migratetype */ + block_ft =3D __get_pfnblock_freetype(page, page_to_pfn(page), true); if (isolate) { - from_mt =3D __get_pfnblock_flags_mask(page, page_to_pfn(page), - PAGEBLOCK_MIGRATETYPE_MASK); - to_mt =3D MIGRATE_ISOLATE; + from_ft =3D block_ft; + to_ft =3D freetype_with_migrate(block_ft, MIGRATE_ISOLATE); } else { - from_mt =3D MIGRATE_ISOLATE; - to_mt =3D __get_pfnblock_flags_mask(page, page_to_pfn(page), - PAGEBLOCK_MIGRATETYPE_MASK); + from_ft =3D freetype_with_migrate(block_ft, MIGRATE_ISOLATE); + to_ft =3D block_ft; } =20 - __move_freepages_block(zone, start_pfn, from_mt, to_mt); + __move_freepages_block(zone, start_pfn, from_ft, to_ft); toggle_pageblock_isolate(pfn_to_page(start_pfn), isolate); =20 return true; @@ -2895,7 +2929,7 @@ static void free_frozen_page_commit(struct zone *zone, */ pcp->alloc_factor >>=3D 1; __count_vm_events(PGFREE, 1 << order); - pindex =3D order_to_pindex(free_to_migratetype(freetype), order); + pindex =3D order_to_pindex(freetype, order); list_add(&page->pcp_list, &pcp->lists[pindex]); pcp->count +=3D 1 << order; =20 @@ -3380,7 +3414,7 @@ static struct page *rmqueue_pcplist(struct zone *pref= erred_zone, * frees. */ pcp->free_count >>=3D 1; - list =3D &pcp->lists[order_to_pindex(free_to_migratetype(freetype), order= )]; + list =3D &pcp->lists[order_to_pindex(freetype, order)]; page =3D __rmqueue_pcplist(zone, order, freetype, alloc_flags, pcp, list); pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); @@ -5176,7 +5210,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int = preferred_nid, goto failed_irq; =20 /* Attempt the batch allocation */ - pcp_list =3D &pcp->lists[order_to_pindex(free_to_migratetype(ac.freetype)= , 0)]; + pcp_list =3D &pcp->lists[order_to_pindex(ac.freetype, 0)]; while (nr_populated < nr_pages) { =20 /* Skip existing pages */ --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9912230FC11 for ; Wed, 24 Sep 2025 15:00:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; cv=none; b=VgYdB82OMjvx7MR76yFu9euTpOorcNlNLUHLN/cMxsSUVFbdazAYxCJIFpLZszzKxCG75Yo0QS88oxo7w7c/8XXlWfBNr/VpCwYIrnKAu0tn7syZIQVQyiVKuub+Dh1xPhOusKSRkswcBpXUOW5AESG+vBnjd/og9RI+tzuKdP8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; c=relaxed/simple; bh=ToEJOWiUgA3tqyi4V/GGuiGh+drLHiNu1T9bHrSLgNM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=EugKD+80NhPlL1dBZnCPkk8btPGRdYnrNqZRXtgYUG09KAJIGgKPyWhwnDx2ngulWrfXr72qmkWPsNnX7YNJACG2hoV6YAyUbw6L2sDACkrvjBEOeDtOt54i/+7D5Pu0LLW9o4na2UAavohdSE3408SfncHPBMkDvnfTf2AKlMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=V/klGdsS; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="V/klGdsS" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3eff3936180so2017272f8f.1 for ; Wed, 24 Sep 2025 08:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726021; x=1759330821; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/bFXVx3R0RyJR3UqZnAzI+0uj7yd2IK0QOGE6ezKlUY=; b=V/klGdsSh99pvfvkv+iUHFqlGuP5PtyLTS9xisHgwqPkOzfDiehajfeePh2vU/N5Sg 3ORYdwHb2VDHNAgRfdBkwscmd2NOsq6W/I/f7l68RYPTQDh1CBEOKAvd7JM4p+N+1RW8 Rxgjz69xJJ6VonAUlZZjyEZ2+lIAJMPKJpq4YGzJvTRNJm15IwmXr5mJUCZwpi47iXtA Nm3UCVmlORxwhj25R0aEUGycMGNMKzFpE6b/0TSpp4sW2Xakbq/G0fRDsvFhbQWJh4iA 8xuk/5xkg/aKaTivTL9KXe0tF8JGXy+puJS/MlOgVzu3mtHgrG97AqUtvjaIUwhntQvd 6ftw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726021; x=1759330821; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/bFXVx3R0RyJR3UqZnAzI+0uj7yd2IK0QOGE6ezKlUY=; b=HytEhS+qUNb1r1tA2YzCNrqUq3cMhNkDVmQbJKripRiRUhUaqE5TMIBrkHAavRFTBN H6o9NIE0ZsvJWymVDQG7UE2FO5Q+ZUXK9RdgfhZrfGN1VCEBxPkeHL7pTuI7ALnuy9bC pM2QEsUzyGi9LDJlBldz5srK+0OQmjUd8QCTYapsY39m13SKaCg1HSKM75BkHn0hrhng wHU8qrAFzjn2Wr/TeC4vORTzhFeq6FAKAc9tryfJ5VSsbcY9ttYooHirk1/RcJIyB2vA pMU5y4FhsdMcl0txiY521JeQ/8QWP4CkmN/JTTpCZ+S5imlqhy7ypqqjUzermMMLlFX9 UssQ== X-Forwarded-Encrypted: i=1; AJvYcCUdJCMw6mFhuLC++NL2mFHkKhtafVM5csDC2f/m8HLlg5P/ZwLIQcSgVD2Pqs4WVzNvNawb4npfJhCfDDA=@vger.kernel.org X-Gm-Message-State: AOJu0YwG4wNkiN6/W8gSDhpRJ1XY9OPIDw4SuxC5gspGYLYsUa2sX2fX +6KDewGGK2m2FLMPTMD0ZOTg0e8114PAOeHEliXAV7j4rWOa3Fry5los/WvHGukq/ECP4jdr41H r6wyGO4dYbBNvnQ== X-Google-Smtp-Source: AGHT+IHFVakseZTMWKw/YOwvxPSBppIbseG235GM/Mq7WzEAVq/9qbzbdj/TNAOCE62qZnIbK4YhRn+KuowqWA== X-Received: from wrwu11.prod.google.com ([2002:a5d:6acb:0:b0:402:78b6:a467]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:2586:b0:407:77f9:949d with SMTP id ffacd0b85a97d-40e4b945911mr255186f8f.40.1758726020780; Wed, 24 Sep 2025 08:00:20 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:48 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-13-2d861768041f@google.com> Subject: [PATCH 13/21] mm/page_alloc_test: unit test pindex helpers From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The author struggles with really basic arithmetic. This test checks for errors in the helpers that are used to map to and from pcplist indices. This can be run via a basic kunit.py invocation: tools/testing/kunit/kunit.py run "page_alloc.*" That will run it via UML which means no THP or ASI. If you want to test with those enabled you can set the --arch flag to run it via QEMU: tools/testing/kunit/kunit.py run --arch=3Dx86_64 \ --kconfig_add CONFIG_TRANSPARENT_HUGEPAGE=3Dy "page_alloc.*" tools/testing/kunit/kunit.py run --arch=3Dx86_64 \ --kconfig_add CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION=3Dy "page_alloc.*" tools/testing/kunit/kunit.py run --arch=3Dx86_64 \ --kconfig_add CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION=3Dy \ --kconfig_add CONFIG_TRANSPARENT_HUGEPAGE=3Dy \ "page_alloc.*" Signed-off-by: Brendan Jackman fix --- mm/Kconfig | 5 ++++ mm/Makefile | 1 + mm/internal.h | 6 +++++ mm/page_alloc.c | 10 +++++--- mm/page_alloc_test.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++= ++++ 5 files changed, 89 insertions(+), 3 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 034a1662d8c1af320b2262ebcb0cb51d4622e6b0..e25451c1adbd6e079f2d00e3eb8= a28affcedab7e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1375,4 +1375,9 @@ config FIND_NORMAL_PAGE =20 source "mm/damon/Kconfig" =20 +config PAGE_ALLOC_KUNIT_TEST + tristate "KUnit Tests for page_alloc code" if !KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + endmenu diff --git a/mm/Makefile b/mm/Makefile index 21abb3353550153a7a477640e4fa6dc6df327541..c6ce46a2abf144f2e62df96ec7f= 606f90affc5f0 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -65,6 +65,7 @@ page-alloc-$(CONFIG_SHUFFLE_PAGE_ALLOCATOR) +=3D shuffle.o memory-hotplug-$(CONFIG_MEMORY_HOTPLUG) +=3D memory_hotplug.o =20 obj-y +=3D page-alloc.o +obj-$(CONFIG_PAGE_ALLOC_KUNIT_TEST) +=3D page_alloc_test.o obj-y +=3D page_frag_cache.o obj-y +=3D init-mm.o obj-y +=3D memblock.o diff --git a/mm/internal.h b/mm/internal.h index 0401412220a76a233e14a7ee7d64c1194fc3759d..6006cfb2b9c7e771a0c647c4719= 01dc7fcdad242 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1693,4 +1693,10 @@ static inline int io_remap_pfn_range_complete(struct= vm_area_struct *vma, return remap_pfn_range_complete(vma, addr, pfn, size, prot); } =20 +#ifdef CONFIG_KUNIT +unsigned int order_to_pindex(freetype_t freetype, int order); +int pindex_to_order(unsigned int pindex); +bool pcp_allowed_order(unsigned int order); +#endif /* CONFIG_KUNIT */ + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 5943b821089b72fd148bd93ee035c0e70e45ec91..0b205aefd27e188c492c32754db= 08a4488317bd8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -56,6 +56,7 @@ #include #include #include +#include #include "internal.h" #include "shuffle.h" #include "page_reporting.h" @@ -691,7 +692,7 @@ static void bad_page(struct page *page, const char *rea= son) add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); } =20 -static inline unsigned int order_to_pindex(freetype_t freetype, int order) +VISIBLE_IF_KUNIT inline unsigned int order_to_pindex(freetype_t freetype, = int order) { int migratetype =3D free_to_migratetype(freetype); /* pindex if the freetype is nonsensitive */ @@ -713,8 +714,9 @@ static inline unsigned int order_to_pindex(freetype_t f= reetype, int order) return (NR_PCP_LISTS_PER_SENSITIVITY * freetype_sensitive(freetype)) + pindex_ns; } +EXPORT_SYMBOL_IF_KUNIT(order_to_pindex); =20 -inline int pindex_to_order(unsigned int pindex) +VISIBLE_IF_KUNIT inline int pindex_to_order(unsigned int pindex) { /* pindex if the freetype is nonsensitive */ int pindex_ns =3D (pindex % NR_PCP_LISTS_PER_SENSITIVITY); @@ -731,8 +733,9 @@ inline int pindex_to_order(unsigned int pindex) =20 return order; } +EXPORT_SYMBOL_IF_KUNIT(pindex_to_order); =20 -static inline bool pcp_allowed_order(unsigned int order) +VISIBLE_IF_KUNIT inline bool pcp_allowed_order(unsigned int order) { if (order <=3D PAGE_ALLOC_COSTLY_ORDER) return true; @@ -742,6 +745,7 @@ static inline bool pcp_allowed_order(unsigned int order) #endif return false; } +EXPORT_SYMBOL_IF_KUNIT(pcp_allowed_order); =20 /* * Higher-order pages are called "compound pages". They are structured th= usly: diff --git a/mm/page_alloc_test.c b/mm/page_alloc_test.c new file mode 100644 index 0000000000000000000000000000000000000000..1cc615ce90d95c47ecae206a87f= 2af3fab3a5581 --- /dev/null +++ b/mm/page_alloc_test.c @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include + +#include + +#include "internal.h" + +/* This just checks for basic arithmetic errors. */ +static void test_pindex_helpers(struct kunit *test) +{ + unsigned long bitmap[bitmap_size(NR_PCP_LISTS)]; + + /* Bit means "pindex not yet used". */ + bitmap_fill(bitmap, NR_PCP_LISTS); + + for (unsigned int order =3D 0; order < NR_PAGE_ORDERS; order++) { + for (unsigned int mt =3D 0; mt < MIGRATE_PCPTYPES; mt++) { + if (!pcp_allowed_order(order)) + continue; + + for (int sensitive =3D 0; sensitive < NR_SENSITIVITIES; sensitive++) { + freetype_t ft =3D migrate_to_freetype(mt, sensitive); + unsigned int pindex =3D order_to_pindex(ft, order); + int got_order; + + KUNIT_ASSERT_LT_MSG(test, pindex, NR_PCP_LISTS, + "invalid pindex %d (order %d mt %d sensitive %d)", + pindex, order, mt, sensitive); + KUNIT_EXPECT_TRUE_MSG(test, test_bit(pindex, bitmap), + "pindex %d reused (order %d mt %d sensitive %d)", + pindex, order, mt, sensitive); + + /* + * For THP, two migratetypes map to the + * same pindex, just manually exclude one + * of those cases. + */ + if (!(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + order =3D=3D HPAGE_PMD_ORDER && + mt =3D=3D min(MIGRATE_UNMOVABLE, MIGRATE_RECLAIMABLE))) + clear_bit(pindex, bitmap); + + got_order =3D pindex_to_order(pindex); + KUNIT_EXPECT_EQ_MSG(test, order, got_order, + "roundtrip failed, got %d want %d (pindex %d mt %d sensitive %d)", + got_order, order, pindex, mt, sensitive); + + } + } + } + + KUNIT_EXPECT_TRUE_MSG(test, bitmap_empty(bitmap, NR_PCP_LISTS), + "unused pindices: %*pbl", NR_PCP_LISTS, bitmap); +} + +static struct kunit_case page_alloc_test_cases[] =3D { + KUNIT_CASE(test_pindex_helpers), + {} +}; + +static struct kunit_suite page_alloc_test_suite =3D { + .name =3D "page_alloc", + .test_cases =3D page_alloc_test_cases, +}; + +kunit_test_suite(page_alloc_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING"); --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5035930FF25 for ; Wed, 24 Sep 2025 15:00:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; cv=none; b=dUvJzVetjJ3QqptdcTamglTPGAPzuQ6rdCSrjwUoQ5PTUuZympnkmWW49FvXqitaQhbH0oKeDgiYbFNeALJY/fS0Mr+oHddS0JMOZGLcy0MnDN2jU89fO54J3hh5BDPTz+TcttgJJjzu1YCROuxUEB20ZKrEiZcnrJXuKrhV3Wo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726025; c=relaxed/simple; bh=BlPWJVLTizBu3rBmLZteoElQSh8cUGz0SgfTUTNxP+U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aRYJrWawqcvmmrtBvdQzoOmXe3BnfBEPWc9rOXM/YZ00AJdkNlLaHF1Ad/PAkFG8qzEB60Jpew/f7an+o/wSKxICRRkP32AQNUfh+bQhoGplOieGEKsZmq6JdWsI113Xd/4eSy2YcNSppArirRc5rXYa5jS05p4t2XS/LPLmX4w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Xtau7mql; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Xtau7mql" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-45b98de0e34so65918275e9.0 for ; Wed, 24 Sep 2025 08:00:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726022; x=1759330822; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Dzh5+kDID1KhkCxx5TAU7iczEzF1E7tPp3j9MouQ4wI=; b=Xtau7mql6ix214wG0kWx5jZUnNgUoqpyTDkjFKKMMT0VJPqVA4+le3UpDtiBKuJjRE Ndmk9St9VNV/bXUgx1eALzppooG3r/izYY/8PumXqoad8AuTDCROMSAzIV+sUNWZZr7g Uvc3HlL7Dhk1PI7r+aVGrW6+vwj98+HMrJZyGGT7E1xpkiz29oaxBmHcLv3hIxrWzl9X 1RnopMwjX+gZ/9o+ozNbFA8cLdN70TSn7tuC4Q7brFKc5lSzRb4LdtJfOV8LPs8X8Vwt 1F3mizOhKxR7EHAxLquO7KxyKLQI8ju1Pjn8UK29t7RZ1KYITMtWCizzXWQBdxGNMSZl Hk0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726022; x=1759330822; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Dzh5+kDID1KhkCxx5TAU7iczEzF1E7tPp3j9MouQ4wI=; b=uoepEC8WSStPLRQs41IOhDvjTNQbS8/nm2R5DPqlkbh/TQFQKAyyXwLIDFdj+1IIIU XS5Lsk9qvytXldm8lbXa+6pho/qS/ZMuXLrzwO0JbbVMqlzBtuN2YGEc0jJcqJr5vQvm 2KsYgGFiNxLz+7II+O1FsnVZjkVLYG8VI9jWLsn2FifbQ+qKgVEUqeX0fxRzHD00jCDE RpFIxDL43um6pdV5OrN/7ztoJ5itJluQdP7+fxabxdNLfrCliF9PWOOgUcR/N2bA6KB2 jYYzMo5n4yPbAJ3Lhdc02RBVpdkSTHEIDlbVjYFIc+lgbwwtTmmQfatvM/bxXORz4UFq 9A8A== X-Forwarded-Encrypted: i=1; AJvYcCXUtOztPd/5EZf0UH5dvjajFfcJmvU4TS7p+v9pJ6SveVhL9CHq83SF8BaZey0PAMo626nX/R/m4K0sU3Y=@vger.kernel.org X-Gm-Message-State: AOJu0YxNU6/LwkqSwZfBZK96eAsr8q34buRyHTpO2cCmJdpt1tm/kloS P6hUk/nxcSZnF7y8qfbzzLfsdwbnBB2j08KAAx213bOhp6nAQKjr/o8LOULl1pa6XjemEvRI+ha Udq/WJYJtOzzklQ== X-Google-Smtp-Source: AGHT+IEsHROXBlX8gPQ5wff3adn8/PrmanibVw/L6EsgdA0zSkoUR9p6G5MTbXSwhHIl6BOnzg1VAmDD3yV6Wg== X-Received: from wmbeu9.prod.google.com ([2002:a05:600c:81c9:b0:45f:2859:5428]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:608c:b0:46e:2f74:2b9a with SMTP id 5b1f17b1804b1-46e32a1a097mr1793855e9.30.1758726021991; Wed, 24 Sep 2025 08:00:21 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:49 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-14-2d861768041f@google.com> Subject: [PATCH 14/21] x86/mm/pat: introduce cpa_fault option From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Different usecases for the CPA code have different needs for the behaviour when encountering an unmapped address. Currently this is encoded by using the presence of the .pgd as a side-channel. Subsequent ASI changes won't get the correct behaviour based on this side-channel, so add an explicit enum to request the different behaviours that might be needed. Note this is now making explicit a couple of causes that populate the pagetables when encountering holes, until now this was implicit: 1. kernel_unmap_pages_in_pgd() Calling this function without a corresponding kernel_map_pages_in_pgd() seems like a bug, so the "correct" behaviour here might actually be CPA_FAULT_ERROR. 2. Ditto for __set_memory_enc_pgtable(). It seems the comment in __cpa_process_fault() (deleted in this patch) may have been stale with regard to the coco usecases here (including point 2). Anyway, if these need to be updated that will be a separate patch, no functional change is intended with this one. Signed-off-by: Brendan Jackman --- arch/x86/mm/pat/set_memory.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 53c3ac0ba55d6b6992db6f6761ffdfbd52bf3688..2a50844515e81913fed32d5b6d1= ec19e8e249533 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -36,6 +36,16 @@ =20 #include "../mm_internal.h" =20 +/* What should CPA do if encountering an unmapped address? */ +enum cpa_fault { + /* Default depending on address. */ + CPA_FAULT_DEFAULT =3D 0, + /* Populate cpa_data.pgd using cpa_data.pfn. */ + CPA_FAULT_POPULATE, + /* Warn and return an error. */ + CPA_FAULT_ERROR, +}; + /* * The current flushing context - we pass it instead of 5 arguments: */ @@ -51,6 +61,7 @@ struct cpa_data { unsigned int force_split : 1, force_static_prot : 1, force_flush_all : 1; + enum cpa_fault on_fault : 2; struct page **pages; }; =20 @@ -1790,14 +1801,13 @@ static inline bool is_direct_map(unsigned long vadd= r) static int __cpa_process_fault(struct cpa_data *cpa, unsigned long vaddr, int primary) { - if (cpa->pgd) { - /* - * Right now, we only execute this code path when mapping - * the EFI virtual memory map regions, no other users - * provide a ->pgd value. This may change in the future. - */ + if (cpa->on_fault =3D=3D CPA_FAULT_POPULATE) return populate_pgd(cpa, vaddr); - } + + if (WARN_ON(cpa->on_fault =3D=3D CPA_FAULT_ERROR)) + return -EFAULT; + + /* CPA_FAULT_DEFAULT: */ =20 /* * Ignore all non primary paths. @@ -2417,6 +2427,7 @@ static int __set_memory_enc_pgtable(unsigned long add= r, int numpages, bool enc) cpa.mask_set =3D enc ? pgprot_encrypted(empty) : pgprot_decrypted(empty); cpa.mask_clr =3D enc ? pgprot_decrypted(empty) : pgprot_encrypted(empty); cpa.pgd =3D init_mm.pgd; + cpa.on_fault =3D CPA_FAULT_POPULATE; =20 /* Must avoid aliasing mappings in the highmem code */ kmap_flush_unused(); @@ -2743,6 +2754,7 @@ int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pf= n, unsigned long address, .mask_set =3D __pgprot(0), .mask_clr =3D __pgprot(~page_flags & (_PAGE_NX|_PAGE_RW|_PAGE_DIRTY)), .flags =3D CPA_NO_CHECK_ALIAS, + .on_fault =3D CPA_FAULT_POPULATE, }; =20 WARN_ONCE(num_online_cpus() > 1, "Don't call after initializing SMP"); @@ -2786,6 +2798,7 @@ int __init kernel_unmap_pages_in_pgd(pgd_t *pgd, unsi= gned long address, .mask_set =3D __pgprot(0), .mask_clr =3D __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY), .flags =3D CPA_NO_CHECK_ALIAS, + .on_fault =3D CPA_FAULT_POPULATE, }; =20 WARN_ONCE(num_online_cpus() > 1, "Don't call after initializing SMP"); --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA22D3112A4 for ; Wed, 24 Sep 2025 15:00:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726026; cv=none; b=Ye8VcAPqccI3xrjtJncD5RqkltGz77Y7IITjm93MKFEA6Xvo1ndhegfbDC7hQ2jKE4lsnCweCWOegT1Lh18JKy75tCSFYJMGp9j8JeR8qnB0LSHTniD4xCfLa8lUXVxLAgtLLd8N+oJzhyi2Z18RLI7yMlASMxsanjpidi6rFHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726026; c=relaxed/simple; bh=0dP5NvAwWbmZXGXCdGyVFXlOGRGhP7dtkT+spo1UHNc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=u4ZSv02qzaDMXneEAaYZGbyEzKRm/vwCT9HI5bD6gw49dn7YD9RgJlerGNI6BK95BsyX+HX/O6xvpDLN1Vxh2WS/BWpzpl3Lsjsn7x9/4hv64lhAoTwdzSK1hDkTgQXS+y1dK2WTkTrWIRdbZe/SdD0Xx9u/gRU/ZF77fmruBVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=G+Eiyk3Z; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="G+Eiyk3Z" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3f924ae2a89so3319225f8f.3 for ; Wed, 24 Sep 2025 08:00:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726023; x=1759330823; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EzrycjTVpnWNKj7lM2bs2OjeckNX2GUUhGCkamMVlMg=; b=G+Eiyk3Z5m+Wq/3isv1APAn8/+Qe+OWs3s8twD6pa9aJ6/IB9EomsvApbpDx6kcbkL 94+OyTNjLXXMZbXZPCwLIBPMVDCuTbiM0Xs8+Ho4jK8uJYrpj52ma6Ait+w9coWZFkER 2+6pEVyGD3R8oBUxQN9lMtUoDaqutWDszgkVtNwoLX5Q8WxLXRIn5qDplPztEWeK634A P8xd2gjQZeS01KBVIf5xxcxwFh3qiVDOB1KvX00jcxU/fka0PNl2+iXq2gOsmTIQi2HV fYao7efqOTCoGMVdac7wt6NG6EkBk0V5+8PIVEJjUEpBRNO6Mm1kRTxtqtBS1a7L56eU m0EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726023; x=1759330823; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EzrycjTVpnWNKj7lM2bs2OjeckNX2GUUhGCkamMVlMg=; b=n7u154kNdtU9AT3l6hk/3KdjWOHPORTVcihd/52FwtmVfUqG4262DK4rrfE1Wjwsf4 GQVsYjTql/g50TBqoJX166YbeRLyNitZBiWqb5w+E3b1EPYJ18KguYdXSQ5xtwzJXnqF JlAl186+KLT8X7SI/qm1p55q+7rzL42b5qc95vr1xLcX6UHYO6WfUtKEWkqPPBeK9C5+ CYKWmRAccF/VdTcFqOF3xnHW+mfqJ4dEh2fhawhBsqhLxS9Govs4FlG++BpS7DOLClAP kIjVdDdQi4/sChX7/KK7oNuFyBtYMSuXOgnYnEzWOp9TwGjrtpx/vh+OjZ0o98TZWRym UrhQ== X-Forwarded-Encrypted: i=1; AJvYcCUSS72IdWR4wEl8r9uAS/UkrgMW2GZWF4fhYKTKf9p8PK1psb3Cqxnu98vCntkscwjoBUMRkPWCcnKeSyw=@vger.kernel.org X-Gm-Message-State: AOJu0YwW3/DKCjR15NEdJ3nzzrTYT7L2gLrxhwWd1KmdW8Nz3QlZYKSs jDDXkhvA7VjlxZrmvd4bZLsBIk1jg9RybenccD/KmY0P23fbgaKWM5qbIOLVAqYRxeVo122Wq1H f8zMlyLhQNh0Jcw== X-Google-Smtp-Source: AGHT+IG4WxCVm2OtQGcna9zHZFJt3Y1WAwbtDQRLq8Re2vHQBN3oIS6DUJScN8kIjtixgF9x+8sRx5iaUapcaw== X-Received: from wrxo18.prod.google.com ([2002:a05:6000:112:b0:3ea:af7:bb94]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:381:b0:3e0:34f4:3225 with SMTP id ffacd0b85a97d-40e4ce4baeamr250578f8f.45.1758726022897; Wed, 24 Sep 2025 08:00:22 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:50 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-15-2d861768041f@google.com> Subject: [PATCH 15/21] mm/page_alloc: rename ALLOC_NON_BLOCK back to _HARDER From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Commit 1ebbb21811b7 ("mm/page_alloc: explicitly define how __GFP_HIGH non-blocking allocations accesses reserves") renamed ALLOC_HARDER to ALLOC_NON_BLOCK because the former is "a vague description". However, vagueness is accurate here, this is a vague flag. It is not set for __GFP_NOMEMALLOC. It doesn't really mean "allocate without blocking" but rather "allow dipping into atomic reserves, _because_ of the need not to block". A later commit will need an alloc flag that really means "don't block here", so go back to the flag's old name and update the commentary to try and give it a slightly clearer meaning. Signed-off-by: Brendan Jackman --- mm/internal.h | 9 +++++---- mm/page_alloc.c | 8 ++++---- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 6006cfb2b9c7e771a0c647c471901dc7fcdad242..513aba6c00bed813c9e38464aec= 5a15e65edaa58 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1297,9 +1297,10 @@ unsigned int reclaim_clean_pages_from_list(struct zo= ne *zone, #define ALLOC_OOM ALLOC_NO_WATERMARKS #endif =20 -#define ALLOC_NON_BLOCK 0x10 /* Caller cannot block. Allow access - * to 25% of the min watermark or - * 62.5% if __GFP_HIGH is set. +#define ALLOC_HARDER 0x10 /* Because the caller cannot block, + * allow access * to 25% of the min + * watermark or 62.5% if __GFP_HIGH is + * set. */ #define ALLOC_MIN_RESERVE 0x20 /* __GFP_HIGH set. Allow access to 50% * of the min watermark. @@ -1316,7 +1317,7 @@ unsigned int reclaim_clean_pages_from_list(struct zon= e *zone, #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAI= M set */ =20 /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC= |ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|AL= LOC_OOM) =20 enum ttu_flags; struct tlbflush_unmap_batch; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0b205aefd27e188c492c32754db08a4488317bd8..cd47cfaae820ce696d2e6e0c474= 36e00d3feef60 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3295,7 +3295,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zon= e, struct zone *zone, * reserves as failing now is worse than failing a * high-order atomic allocation in the future. */ - if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) + if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_HARDER))) page =3D __rmqueue_smallest(zone, order, ft_high); =20 if (!page) { @@ -3662,7 +3662,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int= order, unsigned long mark, * or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get * access to the min reserve. */ - if (alloc_flags & ALLOC_NON_BLOCK) + if (alloc_flags & ALLOC_HARDER) min -=3D min / 4; } =20 @@ -4546,7 +4546,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will - * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). + * set both ALLOC_HARDER and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |=3D (__force int) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); @@ -4557,7 +4557,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |=3D ALLOC_NON_BLOCK; + alloc_flags |=3D ALLOC_HARDER; =20 if (order > 0 && (alloc_flags & ALLOC_MIN_RESERVE)) alloc_flags |=3D ALLOC_HIGHATOMIC; --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C15953115A0 for ; Wed, 24 Sep 2025 15:00:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726027; cv=none; b=RE9ZbiezzkhHQCqDKmxjgBWN8vpS4QOidHBCr9lYQvo9jzgRDbe8qX9TQYCqefRyRnkdH5cJ8/Lm8CudqF4ay+MEKf0MN7J2uYJJLX1m5dHFlY+q6DmH0qnD/zYetQ6HLdKao8spmrktZKmWuxNiM+0Fy+Zo4OLv37RwB38hWmQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726027; c=relaxed/simple; bh=DpKQYYOwhhNiFVGoyRG8dxQpnsEucQPZHGf7AUa02ls=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pUgCgNiWBrYi0WjwKJb9rK9pqmxQ5GJvvKnVSjkvHa0JoDMzAM8wsNoZGrq3zThg/q4uPHgQHoGOBaeh8nHsXlJtadfB3ruzqu86QgSAnAFK5vmcy+AgpyN7eqj2qkmqCZMiqRnFSGyrw1txhZtw6kkWAMnV7qyKp6hXQnfGUfU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WaQ7Y9qd; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WaQ7Y9qd" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3ece14b9231so3547817f8f.0 for ; Wed, 24 Sep 2025 08:00:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726024; x=1759330824; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=41pQw++XrTQuPmXHdjyzKZvwkMfko68cyk5qRol2Kg0=; b=WaQ7Y9qdlkB79Lu50kIIOUvwYumy/dP1277iGbJgrvsS0q+bI4cjXYAxQ4Qauzd1u8 8JXT8u9PCPuttcKF5sqGpcP5fO5lIq/q0EwgSo0v+U+uiu1P3sFnuMBvppeADSjaVQbe WQCha+TxdxBMifJhL6jD6UDdeiIgb03oX4J7JHaWbRphuOyqtq45co/iXjzKPiBf1CwV iBZT3DmxxpFgKZACE6q6CccEyDStktcIiKahgluSa3uzc7NiGuPaBB6FfY3zmNIJVszA SJvo/P/xwd0bkBCZ48w2q0k0tleS3KlIokUCkxpmT+ImuG4/LOMBq6CejQzq0+zNc3QJ a/6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726024; x=1759330824; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=41pQw++XrTQuPmXHdjyzKZvwkMfko68cyk5qRol2Kg0=; b=U7Zw9k1KhM8wmeDrYVoNCQRNzo9j79ZOIjEJ0ySTa/2S6tDz7U+zbBBIyHi19Lmfdo o9ZqveQVVd2BkIQ3rWJNF26M0pgorU/qEvWaTnRuXmVnD8lW410QYPW5u8SCSr2E53Ai v/Vxmgikudbjl7dBYEHxdg38u7Be/Vt3gpo+wvzo1TjHZKZUAsJqyPoMS8QawhsGQsao wtjzjWQ8J65E3jFSyF8mQUWWgzdV5fBaIgSchk34g89ZheTWutzjrzqCZ/Jvx3COknis dChyBhnv2xupY9JQbDCNxAIY0ihuHST+3KS1dFl04Ld5SkeEpi6EJNz8ZNjrY21Z1wsB ZFgw== X-Forwarded-Encrypted: i=1; AJvYcCXHEvBpknJdX1RPGOxlOpNKJh/qsIvOeT1Xkb+2hXzKgHYPuKzUpUydjx3qHbOy62WiKPD0eJ6zzEymc+4=@vger.kernel.org X-Gm-Message-State: AOJu0YxNbdZ3TgP88UNg05RVwSg/grJyDdP9Qd1s6dO81v8VSU74sRjc ySAPxmIlFLxQlJxw4iBomsjm1lw/tKpbRcAUAQIEUwJGBIbkPNi5AHAMxmytxzIO8nEkV0rZmEg gvFAeypkmHkj14A== X-Google-Smtp-Source: AGHT+IHkwdtdIH8CI6uyF/rnsDuP+ou/4VjkKWWzkjAqCtb5L9y32bbpEud8KTzQ+39EVfVnPp60ULCATUlEug== X-Received: from wrbl12.prod.google.com ([2002:adf:a38c:0:b0:3ec:a590:990]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:4210:b0:3e4:5717:368e with SMTP id ffacd0b85a97d-40e498b7705mr228614f8f.2.1758726023909; Wed, 24 Sep 2025 08:00:23 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:51 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-16-2d861768041f@google.com> Subject: [PATCH 16/21] mm/page_alloc: introduce ALLOC_NOBLOCK From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This flag is set unless we can be sure the caller isn't in an atomic context. The allocator will soon start needing to call set_direct_map_* APIs which cannot be called with IRQs off. It will need to do this even before direct reclaim is possible. Despite the fact that, in principle, ALLOC_NOBLOCK is distinct from __GFP_DIRECT_RECLAIM, in order to avoid introducing a GFP flag, just infer the former based on whether the caller set the latter. This means that, in practice, ALLOC_NOBLOCK is just !__GFP_DIRECT_RECLAIM, except that it is not influenced by gfp_allowed_mask. Call it ALLOC_NOBLOCK in order to try and mitigate confusion vs the recently-removed ALLOC_NON_BLOCK, which meant something different. Signed-off-by: Brendan Jackman --- mm/internal.h | 1 + mm/page_alloc.c | 29 ++++++++++++++++++++++------- 2 files changed, 23 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 513aba6c00bed813c9e38464aec5a15e65edaa58..c697ed35a8ca3376445d1e4249e= 9ce03097f15b8 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1315,6 +1315,7 @@ unsigned int reclaim_clean_pages_from_list(struct zon= e *zone, #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_TRYLOCK 0x400 /* Only use spin_trylock in allocation path */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAI= M set */ +#define ALLOC_NOBLOCK 0x1000 /* Caller may be atomic */ =20 /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|AL= LOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cd47cfaae820ce696d2e6e0c47436e00d3feef60..b0aeb97baa13af038fff0edae33= affbbf49e825c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4552,6 +4552,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); =20 if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { + alloc_flags |=3D ALLOC_NOBLOCK; + /* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. @@ -4745,14 +4747,13 @@ check_retry_cpuset(int cpuset_mems_cookie, struct a= lloc_context *ac) =20 static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, - struct alloc_context *ac) + struct alloc_context *ac, unsigned int alloc_flags) { bool can_direct_reclaim =3D gfp_mask & __GFP_DIRECT_RECLAIM; bool can_compact =3D gfp_compaction_allowed(gfp_mask); bool nofail =3D gfp_mask & __GFP_NOFAIL; const bool costly_order =3D order > PAGE_ALLOC_COSTLY_ORDER; struct page *page =3D NULL; - unsigned int alloc_flags; unsigned long did_some_progress; enum compact_priority compact_priority; enum compact_result compact_result; @@ -4795,7 +4796,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, * kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags =3D gfp_to_alloc_flags(gfp_mask, order); + alloc_flags |=3D gfp_to_alloc_flags(gfp_mask, order); =20 /* * We need to recalculate the starting point for the zonelist iterator @@ -5045,6 +5046,19 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int = order, return page; } =20 +static inline unsigned int init_alloc_flags(gfp_t gfp_mask, unsigned int f= lags) +{ + /* + * If the caller allowed __GFP_DIRECT_RECLAIM, they can't be atomic. + * Note this is a separate determination from whether direct + * reclaim is actually allowed, it must happen before applying + * gfp_allowed_mask. + */ + if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) + flags |=3D ALLOC_NOBLOCK; + return flags; +} + static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid, nodemask_t *nodemask, struct alloc_context *ac, gfp_t *alloc_gfp, @@ -5121,7 +5135,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int = preferred_nid, struct list_head *pcp_list; struct alloc_context ac; gfp_t alloc_gfp; - unsigned int alloc_flags =3D ALLOC_WMARK_LOW; + unsigned int alloc_flags =3D init_alloc_flags(gfp, ALLOC_WMARK_LOW); int nr_populated =3D 0, nr_account =3D 0; =20 /* @@ -5267,7 +5281,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, u= nsigned int order, int preferred_nid, nodemask_t *nodemask) { struct page *page; - unsigned int alloc_flags =3D ALLOC_WMARK_LOW; + unsigned int alloc_flags =3D init_alloc_flags(gfp, ALLOC_WMARK_LOW); gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac =3D { }; =20 @@ -5319,7 +5333,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, u= nsigned int order, */ ac.nodemask =3D nodemask; =20 - page =3D __alloc_pages_slowpath(alloc_gfp, order, &ac); + page =3D __alloc_pages_slowpath(alloc_gfp, order, &ac, alloc_flags); =20 out: if (memcg_kmem_online() && (gfp & __GFP_ACCOUNT) && page && @@ -7684,10 +7698,11 @@ struct page *alloc_frozen_pages_nolock_noprof(gfp_t= gfp_flags, int nid, unsigned */ gfp_t alloc_gfp =3D __GFP_NOWARN | __GFP_ZERO | __GFP_NOMEMALLOC | __GFP_= COMP | gfp_flags; - unsigned int alloc_flags =3D ALLOC_TRYLOCK; + unsigned int alloc_flags =3D init_alloc_flags(gfp_flags, ALLOC_TRYLOCK); struct alloc_context ac =3D { }; struct page *page; =20 + VM_WARN_ON_ONCE(!(alloc_flags & ALLOC_NOBLOCK)); VM_WARN_ON_ONCE(gfp_flags & ~__GFP_ACCOUNT); /* * In PREEMPT_RT spin_trylock() will call raw_spin_lock() which is --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C619F311950 for ; Wed, 24 Sep 2025 15:00:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726028; cv=none; b=lu3cvy+JLMec2ClDxcVGmRFjnPgvEfJWuNkLP7kByDj3w4GO8e0FbbHGpc53V8OqCgyBeNBiLqcQaZ3LdkcaaejMNgNrIfQJhBErn3UYVEvPUZT4iplBx8duWGZ0vtAr9KUBhDuuKxm34e34lOM9I/lFjU/hhJeNlXjyZNBw4Mc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726028; c=relaxed/simple; bh=nvFdwqm8o9QTQz3CNfWRVh+M1QsyCyS3UTW9abSr1Ag=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Mzq+0R98ccqQCzWPWL5sWS6vdYAMz3GwQJalNujSXZ+RRwo+PcTIFIZTYfvRp5wES7rWFQtrFZyn5ZBrzgAfQV7z8XPVZWQcFo4s4HGwLIY+/FPQN1/ZcG6rHBSzAKQN4ReX2MlBYl8dPIlOaL4gf2rhdTTRXYKrvUOUUdJaud8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NGfnh/pg; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NGfnh/pg" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e1b05b42fso19989635e9.2 for ; Wed, 24 Sep 2025 08:00:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726025; x=1759330825; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ymn7hQz/pKJzXD8tPM+aagOUbUEwxLcJswgCIwaE8Js=; b=NGfnh/pghilGeWk1Ok2KZeKlk29iGOwU6YR0gc7ZXgTJVHQRAN9jXQpfQu2C39aBi2 4xQsWHr/W0pgHkfRcpfZkBvlrNc7WswsRuVbhEilXX5HulYvRptTE+/K7Q77shZZi/Zi uBmOUclVzTBTXXuY4nnYXSJfTPuwX012UXVDUoMnuPfhSvYgxqxzDdXvLYdLk/BuWnTO efN+FjnsJ/tDdSZHRTpTvBMXDcvjp3YHGHAfDOuIRnbFQNdXn+Qa9euWV0Q1VQRHKNmU RP974AeyPjbCqnXm5R9/762bUU24Tlbl3sdDvuQRTuqwiqwKOJwgwls6f8iG/QiFK78r 5KaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726025; x=1759330825; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ymn7hQz/pKJzXD8tPM+aagOUbUEwxLcJswgCIwaE8Js=; b=eNklOFgozNY2NA6edsRJc9sI88VEH9svtJX+xyv0eGw7gpCc1FA3v2C7NJnDiQksLk tl6CDRF+vtsgzDPAxbKN84+LMFsMJGwtmtXpSn8WyZsgDwe1h0wAV3dTLidRdXll3F1T 1amThpJeN/RBJ8TYPhHXRQ5N6qVRYwUqeKbTAWzsfK2mh3JPwfTbXX+DxiXK0fkA6MFn ULzIpIFSRvdApaxGA4cCSPE4jGv0fIgwDCF6SSMYbsds+/iR7fOBqm5uyL/TwyenfoPy /ko2iBdn2ZPwVLBqWD9i0OXpUR3EQuFYV319fc2bnw9jhq5RVPaQXU3A6gksahNl2UWW 3yHQ== X-Forwarded-Encrypted: i=1; AJvYcCUoSlSdxGBpLJtl2jebu4/OEYhWOgqeQSv7awUhJeuyclwuF396gUwkxzANGAt7JigDDiEaWW73U2HwT70=@vger.kernel.org X-Gm-Message-State: AOJu0YzbLgsZu8BgRr//QmhT74G9jrK1TgVToEFoImnGGIg4Yr2NhQYv QMYM2L80SmXQfylKggu6uo2h5vDiJzKlkvd7R93xh3pjseZK3ZC92elKUVrvc+SeZH9xrBJ3oGC CK/mw+DPIo+cJdA== X-Google-Smtp-Source: AGHT+IEpWYX5czTdErtD0ucu84grAEj2tMZftfqJQYBrWaTJqn3fb8Jtio2hFIBWofbJn4+DEL2AmWmtTqntmQ== X-Received: from wmbgw9.prod.google.com ([2002:a05:600c:8509:b0:46c:8b6c:6cc2]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:46cc:b0:45d:7d88:edcd with SMTP id 5b1f17b1804b1-46e32a030cbmr1610275e9.30.1758726025028; Wed, 24 Sep 2025 08:00:25 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:52 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-17-2d861768041f@google.com> Subject: [PATCH 17/21] mm/slub: defer application of gfp_allowed_mask From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The page allocator will take care of doing this so SLUB doesn't need to do this before calling into it. The page allocator will soon start using the GFP bits as a proxy to infer if it can do blocking stuff (separately from whether it can do actual reclaim), hence SLUB will benefit from leaving __GFP_DIRECT_RECLAIM set even when it's forbidden by gfp_allowed_mask. Signed-off-by: Brendan Jackman --- mm/slub.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 867a07260acf9e3c0f92de66e2d25f081ae51dcb..0f8724af4ce63f6e2a32e889f64= 90be7a25823eb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3177,8 +3177,6 @@ static struct slab *allocate_slab(struct kmem_cache *= s, gfp_t flags, int node) int idx; bool shuffle; =20 - flags &=3D gfp_allowed_mask; - flags |=3D s->allocflags; =20 /* @@ -3212,7 +3210,7 @@ static struct slab *allocate_slab(struct kmem_cache *= s, gfp_t flags, int node) slab->frozen =3D 0; init_slab_obj_exts(slab); =20 - account_slab(slab, oo_order(oo), s, flags); + account_slab(slab, oo_order(oo), s, flags & gfp_allowed_mask); =20 slab->slab_cache =3D s; =20 --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E277A30B516 for ; Wed, 24 Sep 2025 15:00:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726030; cv=none; b=do/48EE6s/kcNmu86kkrEeg3X0caau6MgzQOwVLA5E97HSufeWCrc0hnOD6bDgp9z+CosjiYKY0xYIcv3qEiALjHS2fTjMI6OVd0Q4HLQqdncEhRbzLkxSFNgDeCTmqptIVvKIFdLCqb+5ULSEdufm7AZN7hJcK5rtx+LR8cTKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726030; c=relaxed/simple; bh=wUbWX91BQp6M0/mnNhy3/4WzdN/SwErhAcw4rqPjQcw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Nv5HwWXNibfW//LkazLQnSxIeuxor/YEwOCWru+Tn2/jj/V/usb3MShZr/skYgq+cG/J2Ho2TTOkn6l0BD1KFRNxpBiJcGRjpPeZDqChZ7Awc/YGNExbfIYs5jqzC/2CEaaUTJzpLCSx1WYxK3ThPuMWQGF9bJHlqsgHQzi/j7Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kM4LIAFz; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kM4LIAFz" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-46d8ef3526dso21392255e9.1 for ; Wed, 24 Sep 2025 08:00:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726026; x=1759330826; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yt3Q1XTnbMGvlZgCdZ/CgJSleEFjvQwQBSgYepLM/TE=; b=kM4LIAFzTWVpBHx9Ns+zBbNWRLg/vhqXIJD4if55rS0vM6o8GoT92t6hE+qicxxU31 4jfPth1WO8JirPu5nk5mh0Di8q/kIKJ412sJvjFa2fwxxfqbtd16eO//a2J5MCTAcAkF wpKVbgvvyzBN+t9QqYBVn7WQbxOOCFo30J43cs6Tw2/fsJybJX0Q4l53j26Z3KSxTrJ/ Cl6fqw2ISYUI/YKqV0qjrcQbZdlZOT4O4LDyUK2yaWRzboTanvjZGpK+ixWBtrw3UpWN +5Ntlrjx3MQWhGmQxTwpma1QV8ED8BHAF/s5EoD+QulKFaugDOhjwP+XtvwABT4itd/2 Crug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726026; x=1759330826; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yt3Q1XTnbMGvlZgCdZ/CgJSleEFjvQwQBSgYepLM/TE=; b=lTBCMcHrYn12KiRxCCtgGJV41np+/8nIbDogQ9WTqmA/bhXwSpvwf65mjVgMTVDC9s nCOz1D1GQgO1VEkvBJ80yu3klP3EKuEuEuCidtbCK/YzEtiXB/eLqhmoSBOKFDiB2eF6 aPdX7mrXWbeWfhL6x4qRv08u3s/KT9FdNBim2ll5kF/CmO4qecyqGt4m8PUII2i5R2dQ KGHF1qCrZqnTLeLpQd0Asp72KenIX25BWgaqG3m55deYo2vzy8pbSG3n5dlmQtuld6Mh 2zMnU1AJ0sD4mRKKTdJrz+k5MhMTqgCoLJVsjinjNi8BIqVza0PH31EwHR5J5pcMTReY RXbA== X-Forwarded-Encrypted: i=1; AJvYcCVqpHhD6aM4ustY9z9ESvmk0+t+JTsI+GueMpWJiDppOhfq9iOner+prQxfLFhrEl5b5IqOsifxG7O9Nd8=@vger.kernel.org X-Gm-Message-State: AOJu0YxU+3zMn8ObegHdZvSRK79/03QjQQsbvQhjZg5GYC+NMAWDgP8M ZZXTPX6quqMbBDKPjfAbKF58mI+mM7+OX2x6kHHPjvL7cfPwXqV/sMxvoC6i7JLHWL5xUaAU3Ne MaFUDt6K3jIqEJw== X-Google-Smtp-Source: AGHT+IEOgWk7seh/trUQUGpJ6ZsEL7UmOVpxpeIq06m/4rtBPuJiGJmuPe07oAhOUg5xOwd6Wn79YD2LQK3VBw== X-Received: from wmbhe13.prod.google.com ([2002:a05:600c:540d:b0:46d:2f48:35f9]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:6306:b0:46d:9d28:fb5e with SMTP id 5b1f17b1804b1-46e329ab29cmr2315835e9.5.1758726026374; Wed, 24 Sep 2025 08:00:26 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:53 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-18-2d861768041f@google.com> Subject: [PATCH 18/21] mm/asi: support changing pageblock sensitivity From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Currently all pages are sensitive and there's no way to change that. This patch introduces one. Changing sensitivity has some requirements: - It can only be done at pageblock granularity. This means that there's never a need to allocate pagetables to do it (since the restricted direct map is always pre-allocated down to pageblock granularity). - Flipping pages from nonsensitive to sensitive (unmapping) requires a TLB shootdown, meaning IRQs must be enabled. - Flipping from sensitive to nonsensitive requires zeroing pages, which seems like an undesirable thing to do with a spinlock held. This makes allocations that need to change sensitivity _somewhat_ similar to those that need to fallback to a different migratetype. But, the locking requirements mean that this can't just be squashed into the existing "fallback" allocator logic, instead a new allocator path just for this purpose is needed. The new path is assumed to be much cheaper than the really heavyweight stuff like compaction and reclaim. But at present it is treated as less desirable than the mobility-related "fallback" and "stealing" logic. This might turn out to need revision (in particular, maybe it's a problem that __rmqueue_steal(), which causes fragmentation, happens before __rmqueue_asi()), but that should be treated as a subsequent optimisation project. Now that !__GFP_SENSITIVE allocations are no longer doomed to fail, stop hard-coding it at the top of the allocator. Note that this does loads of unnecessary TLB flushes and IPIs. The design goal here is that the transitions are rare enough that this doesn't matter a huge amount, but it should still be addressed in later patches. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/set_memory.h | 10 ++++ arch/x86/mm/pat/set_memory.c | 28 ++++++++++ include/linux/set_memory.h | 6 +++ mm/page_alloc.c | 106 +++++++++++++++++++++++++++++++++-= ---- 4 files changed, 138 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_m= emory.h index 61f56cdaccb5af18e36790677b635b4ab6f5e24d..396580693e7d1317537148c0c21= 9296e2b7c13fd 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -92,6 +92,16 @@ int set_direct_map_default_noflush(struct page *page); int set_direct_map_valid_noflush(struct page *page, unsigned nr, bool vali= d); bool kernel_page_present(struct page *page); =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +int set_direct_map_sensitive(struct page *page, int num_pageblocks, bool s= ensitive); +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +static inline +int set_direct_map_sensitive(struct page *page, int num_pageblocks, bool s= ensitive) +{ + return 0; +} +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + extern int kernel_set_to_readonly; =20 #endif /* _ASM_X86_SET_MEMORY_H */ diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 2a50844515e81913fed32d5b6d1ec19e8e249533..88fb65574d4fa0089fa31a9a06f= e096c408991e6 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -4,6 +4,7 @@ * Thanks to Ben LaHaise for precious feedback. */ #include +#include #include #include #include @@ -2695,6 +2696,33 @@ int set_direct_map_valid_noflush(struct page *page, = unsigned nr, bool valid) return __set_pages_np(page, nr); } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +/* + * Map/unmap a set of contiguous pageblocks into all ASI restricted address + * spaces. All pagetables are pre-allocated so this can be called anywhere. + * This should not be called on pages that may be mapped elsewhere. + */ +int set_direct_map_sensitive(struct page *page, int num_pageblocks, bool s= ensitive) +{ + if (WARN_ON_ONCE(!IS_ALIGNED(page_to_pfn(page), 1 << pageblock_order))) + return -EINVAL; + + unsigned long tempaddr =3D (unsigned long)page_address(page); + struct cpa_data cpa =3D { .vaddr =3D &tempaddr, + .pgd =3D asi_nonsensitive_pgd, + .numpages =3D num_pageblocks << pageblock_order, + .flags =3D CPA_NO_CHECK_ALIAS, + .on_fault =3D CPA_FAULT_ERROR, }; + + if (sensitive) + cpa.mask_clr =3D __pgprot(_PAGE_PRESENT); + else + cpa.mask_set =3D __pgprot(_PAGE_PRESENT); + + return __change_page_attr_set_clr(&cpa, 1); +} +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + #ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index 3030d9245f5ac8a35b27e249c6d8b9539f148635..db4225c046c47c114293af8b504= 886b103dc94ce 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -44,6 +44,12 @@ static inline bool kernel_page_present(struct page *page) { return true; } + +static inline int set_direct_map_sensitive(struct page *page, + int num_pageblocks, bool sensitive) { + return 0; +} + #else /* CONFIG_ARCH_HAS_SET_DIRECT_MAP */ /* * Some architectures, e.g. ARM64 can disable direct map modifications at diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b0aeb97baa13af038fff0edae33affbbf49e825c..a8e3556643b0ff2fe1d35a67893= 7270356006d34 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -44,6 +44,7 @@ #include #include #include +#include #include #include #include @@ -585,6 +586,13 @@ static void set_pageblock_migratetype(struct page *pag= e, PAGEBLOCK_MIGRATETYPE_MASK | PAGEBLOCK_ISO_MASK); } =20 +static inline void set_pageblock_sensitive(struct page *page, bool sensiti= ve) +{ + __set_pfnblock_flags_mask(page, page_to_pfn(page), + sensitive ? 0 : PAGEBLOCK_NONSENSITIVE_MASK, + PAGEBLOCK_NONSENSITIVE_MASK); +} + void __meminit init_pageblock_migratetype(struct page *page, enum migratetype migratetype, bool isolate) @@ -3264,6 +3272,85 @@ static inline void zone_statistics(struct zone *pref= erred_zone, struct zone *z, #endif } =20 +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +static inline struct page *__rmqueue_asi(struct zone *zone, unsigned int r= equest_order, + unsigned int alloc_flags, freetype_t freetype) +{ + freetype_t freetype_other =3D migrate_to_freetype( + free_to_migratetype(freetype), !freetype_sensitive(freetype)); + unsigned long flags; + struct page *page; + int alloc_order; + enum rmqueue_mode rmqm =3D RMQUEUE_NORMAL; + int nr_pageblocks; + + if (!asi_enabled_static()) + return NULL; + + /* + * Might need a TLB shootdown. Even if IRQs are on this isn't + * safe if the caller holds a lock (in case the other CPUs need that + * lock to handle the shootdown IPI). + */ + if (alloc_flags & ALLOC_NOBLOCK) + return NULL; + lockdep_assert(!irqs_disabled() || unlikely(early_boot_irqs_disabled)); + + /* + * Need to [un]map a whole pageblock (otherwise it might require + * allocating pagetables). First allocate it. + */ + alloc_order =3D max(request_order, pageblock_order); + nr_pageblocks =3D 1 << (alloc_order - pageblock_order); + spin_lock_irqsave(&zone->lock, flags); + page =3D __rmqueue(zone, alloc_order, freetype_other, alloc_flags, &rmqm); + spin_unlock_irqrestore(&zone->lock, flags); + if (!page) + return NULL; + + if (!freetype_sensitive(freetype)) { + /* + * These pages were formerly sensitive so we need to clear them + * out before exposing them to CPU attacks. Doing this with the + * zone lock held would have been undesirable. + */ + kernel_init_pages(page, 1 << alloc_order); + } + + /* + * Now that IRQs are on it's safe to do a TLB shootdown, so change + * mapping and update pageblock flags. + */ + set_direct_map_sensitive(page, nr_pageblocks, freetype_sensitive(freetype= )); + for (int i =3D 0; i < nr_pageblocks; i++) { + struct page *block_page =3D page + (pageblock_nr_pages * i); + + set_pageblock_sensitive(block_page, freetype_sensitive(freetype)); + } + + if (request_order >=3D alloc_order) + return page; + + /* Free any remaining pages in the block. */ + spin_lock_irqsave(&zone->lock, flags); + for (unsigned int i =3D request_order; i < alloc_order; i++) { + struct page *page_to_free =3D page + (1 << i); + + __free_one_page(page_to_free, page_to_pfn(page_to_free), zone, + i, freetype, FPI_SKIP_REPORT_NOTIFY); + } + spin_unlock_irqrestore(&zone->lock, flags); + + return page; +} +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +static inline struct page *__rmqueue_asi(struct zone *zone, unsigned int r= equest_order, + unsigned int alloc_flags, freetype_t freetype) +{ + return NULL; +} +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + static __always_inline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags, @@ -3297,13 +3384,15 @@ struct page *rmqueue_buddy(struct zone *preferred_z= one, struct zone *zone, */ if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_HARDER))) page =3D __rmqueue_smallest(zone, order, ft_high); - - if (!page) { - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; - } } spin_unlock_irqrestore(&zone->lock, flags); + + /* Try changing sensitivity, now we've released the zone lock */ + if (!page) + page =3D __rmqueue_asi(zone, order, alloc_flags, freetype); + if (!page) + return NULL; + } while (check_new_pages(page, order)); =20 __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -5285,13 +5374,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, = unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac =3D { }; =20 - /* - * Temporary hack: Allocation of nonsensitive pages is not possible yet, - * allocate everything sensitive. The restricted address space is never - * actually entered yet so this is fine. - */ - gfp |=3D __GFP_SENSITIVE; - /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3550531281C for ; Wed, 24 Sep 2025 15:00:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726031; cv=none; b=X7ubR3wQOE3bjE1zBdyHpaCkcBxUrrfKF/pVpY7DmB3JcQGoPTRP865hk96t7hoyjZLuYEBeTwIuk8WRFnXpAmo1AFLrV0vF9/7VkwMfJuWLLT6wadwGgOPDt11yH7+HpCF8u7V7KTzSrHq+qBqfTI06rh4YQH6BqAPPKUD+ayU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726031; c=relaxed/simple; bh=Nc9wGi4sa/SseUd8+DrYdDIywk30Jds65N0aHDFnlcI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nXidndxorV1pTkrmocKM3tKwBUg9VWZ1tpA2hdGw2vywhnmV2cLto6lc8IwGvxL/5fOGNz5jzt6qoDpB8OyPcFFHCMV5gcO/7AZcw4TyItkG3WCEuq4wsFHkfMBze12OitqsE4WsLjf1S2z1BDtbzwNSsgpkAoVvZzCSt50LUcs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yKG3pQaR; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yKG3pQaR" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-46e19f00bc2so27320005e9.3 for ; Wed, 24 Sep 2025 08:00:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726027; x=1759330827; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xK9DdWzysI7MLQZmTS4EgkDsy0LNYBhIo28kzW1svBA=; b=yKG3pQaRLLxcI44g9iEtm5NkcrrCpzQLaCVX6hABdIsfg9VUaJ3/z7uCzFJ+M37NYU 5n5h07vn4FmLDqtvTmAH3SLg4QRagqtu4JUgBw2g29ujW8/PzTYbUtudbJ/XgH92E+tq QcKo/OAca8SLqONvu0lphWY1rgRRo3shKINzSuOVduyV2A+31UQEtkTty+mwMHjmF45T 4DdOIretKHQsc3nVZ0eUG3/W/A6M7JRHYzgR8kW86//0wzhmn+63wd/c2zmfl+JjT/Hg FRbSV30mck8f64sU9KUwl2kEDPZJF+qH6L1ZB5e0yO1Z7LLzL6L6f43faBnqc6Rx1Nti 3WKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726027; x=1759330827; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xK9DdWzysI7MLQZmTS4EgkDsy0LNYBhIo28kzW1svBA=; b=WjCGcEChl4XAoc9s1Ah2xgSvLNf/F7dYNns8ZJB/LJwO7dPr8KsLKtPKCTtnF7Soxb pv/w4d52T+Ij2vnR/D9ZPwcAbnM1OkbqJQZXXnpzg+c5lnftg/nQdJLYzTjaUv+G4pyA NLqpOxKAuy7BI3HK8l87FuP/gyUams4DT2RBFZURAWtZ4+rwsCgaFf5gOalQlEVEsdUD 920f0w4NqVKW1GJ0LDpiHg422GMSOsxIBOlxYRMtAiA2gY6zJDjJzwpNRVSo9Engqsqy 55VEN6X9LG67YQ2LT5cRKeCqq+dYObOa9TYtU/H2clzvcJlLnsrGCOXj7OPXJfqRrMJ7 USuQ== X-Forwarded-Encrypted: i=1; AJvYcCW/IHyYvx1V/bJoA85s8NK3PcEVpvxlPPyngr42Cq5Xs5+5vQF7Z8UVaGfRjlMc2MY3Is0ByTkxCebjAjQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yx6CHiJ9pTT9xsowc90Fma6yYbiO5VC5HxILuMRR58ZN1ru8iWA qPa81NdnZZgeqSHJuZzaW5wAZ7yitXQF0yPnpcuHpgrKQqMDi6JT8d+ssYjF9VboeGr1BSJ6gRJ CWfTk0HpIUTVu7A== X-Google-Smtp-Source: AGHT+IHRBeDpUtgF245mJUc5ipAYXCdMHfAb0w1ma3Resd3ql66vcD7C0WlHnlWorox4xP6DwHfNcUDqpn/NkQ== X-Received: from wmrn38.prod.google.com ([2002:a05:600c:5026:b0:46c:f222:92c9]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:c163:b0:45d:cf5f:cff8 with SMTP id 5b1f17b1804b1-46e329b9b49mr1939205e9.11.1758726027581; Wed, 24 Sep 2025 08:00:27 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:54 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-19-2d861768041f@google.com> Subject: [PATCH 19/21] mm/asi: bad_page() when ASI mappings are wrong From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Add bad_page() checks that fire when the page allocator thinks a page is mapped/unmapped in ASI restricted address space, but the pagetables disagree. This requires adding an accessor for set_memory.c to walk the page tables and report the state. This is implemented with the assumption that the mapping is at pageblock granularity. That means it doesn't need to be repeated for each order-0 page. As a result of this special order-awareness, it can't go into free_page_is_bad() and needs to be separately integrated into free_pages_prepare(). The alloc side is easier - there it just goes into check_new_pages(). Signed-off-by: Brendan Jackman --- arch/x86/include/asm/set_memory.h | 3 +++ arch/x86/mm/pat/set_memory.c | 31 +++++++++++++++++++++++++++ include/linux/set_memory.h | 2 ++ mm/page_alloc.c | 45 ++++++++++++++++++++++++++++++++++-= ---- 4 files changed, 76 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/set_memory.h b/arch/x86/include/asm/set_m= emory.h index 396580693e7d1317537148c0c219296e2b7c13fd..3870fa8cf51c0ece0dedf4d7876= c4d14111deffd 100644 --- a/arch/x86/include/asm/set_memory.h +++ b/arch/x86/include/asm/set_memory.h @@ -94,12 +94,15 @@ bool kernel_page_present(struct page *page); =20 #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION int set_direct_map_sensitive(struct page *page, int num_pageblocks, bool s= ensitive); +bool direct_map_sensitive(struct page *page); #else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ static inline int set_direct_map_sensitive(struct page *page, int num_pageblocks, bool s= ensitive) { return 0; } + +static inline bool direct_map_sensitive(struct page *page) { return false;= } #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ =20 extern int kernel_set_to_readonly; diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 88fb65574d4fa0089fa31a9a06fe096c408991e6..d4c3219374f889f9a60c459f055= 9e5ffb472073d 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -2721,6 +2721,37 @@ int set_direct_map_sensitive(struct page *page, int = num_pageblocks, bool sensiti =20 return __change_page_attr_set_clr(&cpa, 1); } + +/* + * Walk the pagetable to check if the page is mapped into all ASI restrict= ed + * address spaces. + */ +bool direct_map_sensitive(struct page *page) +{ + unsigned long addr =3D (unsigned long)page_address(page); + pgd_t *pgd =3D pgd_offset_pgd(asi_nonsensitive_pgd, addr); + unsigned int level; + bool nx, rw; + pte_t *pte =3D lookup_address_in_pgd_attr(pgd, addr, &level, &nx, &rw); + + switch (level) { + case PG_LEVEL_4K: + /* + * lookup_address_in_pgd_attr() still returns the PTE for + * non-present 4K pages. + */ + return !pte_present(*pte); + case PG_LEVEL_2M: + /* + * pmd_present() checks PSE to deal with some hugetlb + * logic. That's not relevant for the direct map so just + * explicitly check the real P bit. + */ + return !(pmd_flags(*(pmd_t *)pte) & _PAGE_PRESENT); + default: + return !pte; + } +} #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ =20 #ifdef CONFIG_DEBUG_PAGEALLOC diff --git a/include/linux/set_memory.h b/include/linux/set_memory.h index db4225c046c47c114293af8b504886b103dc94ce..6f42d6a35feceeae4623c2da50c= fac54e3533228 100644 --- a/include/linux/set_memory.h +++ b/include/linux/set_memory.h @@ -50,6 +50,8 @@ static inline int set_direct_map_sensitive(struct page *p= age, return 0; } =20 +static inline bool direct_map_sensitive(struct page *page) { return false;= } + #else /* CONFIG_ARCH_HAS_SET_DIRECT_MAP */ /* * Some architectures, e.g. ARM64 can disable direct map modifications at diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a8e3556643b0ff2fe1d35a678937270356006d34..68bc3cc5ed7e7f1adb8dda90edc= 2e001f9a1c3c5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -15,6 +15,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ =20 +#include #include #include #include @@ -1161,6 +1162,33 @@ static const char *page_bad_reason(struct page *page= , unsigned long flags) return bad_reason; } =20 +static bool page_asi_mapping_bad(struct page *page, unsigned int order, bo= ol sensitive) +{ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + if (asi_enabled_static()) { + struct page *block_page =3D page; + + /* + * ASI mappings are at pageblock granularity. Check they match + * the requested sensitivity. + */ + while (block_page < page + (1 << order)) { + if (direct_map_sensitive(block_page) !=3D sensitive) { + bad_page(page, + sensitive ? + "page unexpectedly nonsensitive" : + "page unexpectedly sensitive"); + return true; + } + + block_page +=3D pageblock_nr_pages; + } + } +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + + return false; +} + static inline bool free_page_is_bad(struct page *page) { if (likely(page_expected_state(page, PAGE_FLAGS_CHECK_AT_FREE))) @@ -1471,8 +1499,14 @@ __always_inline bool free_pages_prepare(struct page = *page, page->page_type =3D UINT_MAX; =20 if (is_check_pages_enabled()) { + freetype_t ft =3D get_pageblock_freetype(page); + if (free_page_is_bad(page)) bad++; + + if (!bad) + bad +=3D page_asi_mapping_bad(page, order, + freetype_sensitive(ft)); if (bad) return false; } @@ -1840,7 +1874,8 @@ static bool check_new_page(struct page *page) return true; } =20 -static inline bool check_new_pages(struct page *page, unsigned int order) +static inline bool check_new_pages(struct page *page, unsigned int order, + bool sensitive) { if (!is_check_pages_enabled()) return false; @@ -1852,7 +1887,7 @@ static inline bool check_new_pages(struct page *page,= unsigned int order) return true; } =20 - return false; + return page_asi_mapping_bad(page, order, sensitive); } =20 static inline bool should_skip_kasan_unpoison(gfp_t flags) @@ -3393,7 +3428,7 @@ struct page *rmqueue_buddy(struct zone *preferred_zon= e, struct zone *zone, if (!page) return NULL; =20 - } while (check_new_pages(page, order)); + } while (check_new_pages(page, order, freetype_sensitive(freetype))); =20 __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); zone_statistics(preferred_zone, zone, 1); @@ -3478,7 +3513,7 @@ struct page *__rmqueue_pcplist(struct zone *zone, uns= igned int order, page =3D list_first_entry(list, struct page, pcp_list); list_del(&page->pcp_list); pcp->count -=3D 1 << order; - } while (check_new_pages(page, order)); + } while (check_new_pages(page, order, freetype_sensitive(freetype))); =20 return page; } @@ -7231,7 +7266,7 @@ int alloc_contig_range_noprof(unsigned long start, un= signed long end, } else if (start =3D=3D outer_start && end =3D=3D outer_end && is_power_o= f_2(end - start)) { struct page *head =3D pfn_to_page(start); =20 - check_new_pages(head, order); + check_new_pages(head, order, gfp_mask & __GFP_SENSITIVE); prep_new_page(head, order, gfp_mask, 0); set_page_refcounted(head); } else { --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 29C403128B1 for ; Wed, 24 Sep 2025 15:00:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726032; cv=none; b=D0EeoCF67bxoKRka3eOroKq/6TpyP+09KKj+9/uxZTmV4nYSLrgOAmBAc5pW4kAmaaLgzWv4nRMEiloUKchzr0UW6IP8oyKiDUK77r523VNZwk+5qW07Uy2SkOZ8sd+AwzcIKTyNQc42/HNACWoROe+CnYxv7/js1IGy43mtoJA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726032; c=relaxed/simple; bh=qDMukE+vGFaZgPAnzYH5a2YY1R/PccU0PwTyvnM0oxg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=I/Ewj4Fmx6tRDJqWy4e2X8VdN4CAAUSyUfnibgKrsa/fBFdKmVoRoMsA98+8dZkTY+TLxYzkba9+Z44SqOHlzbHTU7rXh5Rdo1tsjVqqRhusJ2jz4J5WI56nnNSxbMg9NPGVltOl/zXn46wwoRHphzQO8mPwQWWbbOxQ5fr0c3s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GDcp2xYN; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GDcp2xYN" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-40657e09e85so826715f8f.1 for ; Wed, 24 Sep 2025 08:00:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726029; x=1759330829; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=oP8jWp4MkINOH7Jswzd6ZODnFl0qB6e1xzS3CEKeQl0=; b=GDcp2xYNYT0h1c/L6kwFXyvgWnk/38WWVtqaheANUHHs6d7S8leklwmrq6nAUpdAFb BVuP1ynePhqlCyCeaxElyoiiIGJFzCULiqIlxZ2hd2kdPoitelijiAE6G3O0UFyVB2o0 HnJNMHxInqVKPecnBxYz3jLQDtZoq6fRbfL0g/Arcxetj6hFcnsEBuvwn0CRj/na7Crd T/VcizgT+btSrUNmKPCskfwMGR6zEFS6j5M7T1nmrLOnn6r2kLBkG7qOgKbFz0g709kN tHNR/KKZiuhD2UmOs/uk5CnawkDsFJ8GsMZE1kLEiYt5JckqNARMCsy9Y8gRc0ldGG5X qBjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726029; x=1759330829; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=oP8jWp4MkINOH7Jswzd6ZODnFl0qB6e1xzS3CEKeQl0=; b=o+AGTN7ghHDq64COic2MNca21JQm2QcGQVTWzkgTExY1d8JICvw8QOoa2/Rz45PalB WE4y8R44E3BakIzQ+t1MmM+lHqeB2i0yv9eaFWvXPbCxZdFevurNJ0zWV6CoHORvQqTy C6y4gsKrFq2BP05MkSkBmYB1f1h2ntsN87hytb2dI4aj6wgKXi2P4MDaeDvxvxkHSjpA oJwab5NexS7rDPuKGPGLclaocIffJb1HuytyJTVwhZj46FpJ39sQV6Mrx9y8AvZhkdU2 scpsoA7ABt5VqslwMkVH0S4hSeK0OyvIpXjxHWeEtTIfkZDp84y/gZFw9vTzO8dgjaAD cZMA== X-Forwarded-Encrypted: i=1; AJvYcCVXk4/erDpR0L6J4JUsmcfhT4RlVAugApasxPfRohVofEnIsc8XrWyKMQDzKGkHEJW5TjpR0+vmC7cyttw=@vger.kernel.org X-Gm-Message-State: AOJu0Yz18RceoowOTRNrthLbG7wGoowgHrDHPrw5/AXzAtrdcXA9GQxN jeILeUTHjT/n0MxQ1j+kJWKvK1NQeIwUsb6ORBs+mJ3xQ56kuInV9MjAcMDHf/fMibCmSfAtu/w n10OirrGWkiCqaw== X-Google-Smtp-Source: AGHT+IHodywAlVICR2SlEwjI/nDMWsb7c+Q0rCIs295IEGPrhzAjMAXo4iCffE/QGARjzAt8FOBx8w/TD0aqww== X-Received: from wrvh13.prod.google.com ([2002:a5d:548d:0:b0:3ec:e12a:e26]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:240a:b0:3e2:ac0:8c55 with SMTP id ffacd0b85a97d-40e50f5651fmr215557f8f.55.1758726028587; Wed, 24 Sep 2025 08:00:28 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:55 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-20-2d861768041f@google.com> Subject: [PATCH 20/21] x86/mm/asi: don't use global pages when ASI enabled From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable For the same reason as with PTI, ASI means kernel pages can't be global: TLB entries from the unrestricted address space must not carry over into the restricted address space. Signed-off-by: Brendan Jackman --- arch/x86/mm/init.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 8fd34475af7ccd49d0124e13a87342d3bfef3e05..45bbe764ca4e0abc13d41590dc4= ff466180cca31 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -249,8 +249,9 @@ static void __init probe_page_size_mask(void) =20 /* By the default is everything supported: */ __default_kernel_pte_mask =3D __supported_pte_mask; - /* Except when with PTI where the kernel is mostly non-Global: */ - if (cpu_feature_enabled(X86_FEATURE_PTI)) + /* Except when with PTI/ASI where the kernel is mostly non-Global: */ + if (cpu_feature_enabled(X86_FEATURE_PTI) || + cpu_feature_enabled(X86_FEATURE_ASI)) __default_kernel_pte_mask &=3D ~_PAGE_GLOBAL; =20 /* Enable 1 GB linear kernel mappings if available: */ --=20 2.50.1 From nobody Thu Oct 2 00:49:30 2025 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B774230BBBD for ; Wed, 24 Sep 2025 15:00:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726034; cv=none; b=ETpNQvMWfpzWBT3sKeDJ1T25C3XQZq1c6mnqrVaKy6q3neLxKd6/miozrgbQW1oFKZDlVapNnNPHi4dHduErDsqUUiwFLsqlBTXVNgt5HIvgytZR7pmifa5z5EvVZHkHGaf6VFMfIPwGOzo8NK/6UYR3J2Ovjnj3PG632eWU+JA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758726034; c=relaxed/simple; bh=qcUdbQm9f0IEEs//mdY+wa1apR9KiQeDm5678H0bicc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=JTQPW7pQnjsk3hugdZnRPHsEoFHleaouiPK51XxbaVVC9Ihej165if+E8jVGY3USfWuMbe7V42VKpFPK9kbGPlkhJMgYdA1HeXyil0CKBdIWmt1/7Lb+KfuCbitJjkVvM4lVdJQOdGJhK6oR444Un2KiGC2+FoUrn74aFz2tn3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZFhr6UCO; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZFhr6UCO" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-63049fca047so4240733a12.2 for ; Wed, 24 Sep 2025 08:00:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1758726030; x=1759330830; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5L6wwf2TI57XEfLhcaZXMRQ+L/0zQiVmk9x1OVnFFiI=; b=ZFhr6UCOq6AmMkQMh3hxuDH16+PwR/ebFFtT/+bWT6YCQT9+VLY6EXHdCl/TTop7Ps bRqtMaO38ftXCN+Z5Nfm8y+vDCAkaa8OG5OfvpxaN/HUjZ2h3YaiO0vYKxaT2U9vHdJN A5UgKocdrijqfA0cfP4ZWFm7ZjOGYk1Lry6XJKyKFOQBmxT/bYTFa2ySQb9F9gMqzoDB xqOa2rtp3NlmnHK5CWDpt1aUzltsgA/7iIZkQzt7YRHNpUFF5qqSHqepjjwba28NQ9GE wf13nDzLnUAr1UPnX0vudsomsoTMn+uQ4yTidf3wfxi7lSl0/nMU7GC4+DSmDI71D+OY Dk0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1758726030; x=1759330830; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5L6wwf2TI57XEfLhcaZXMRQ+L/0zQiVmk9x1OVnFFiI=; b=P8BXfSvHF1EYJBTuUrgYjCJFKrzyyqjixphO/D/CVvM/D2Jegoo+pp8lzks+cgHFCe Wz3axxF8BF4uL2X5rMy0EAke0FvwvN54Z2eu1nrOspCt0auImCQ2xxmL1DEE3XFg6kmt WvlpJObTo9NRMYvkQV/ubmMF+Atpmj/5T2eXDfitEyI8dFC9gswHfTFieX8wdWYx08S3 IZxH2saV19AeibL5x0jd72eXAnwA0V0LXQfOm+KTz04Js4nLVew4QpTH9lDV8D9HZRe5 zyVrhSax2lDgKb0G/yeJJb4et3Auxcy798gsGguDz6+hUXEla6m4mvSsj8KkIf+m1v0t bZ2Q== X-Forwarded-Encrypted: i=1; AJvYcCWR1uDd3FhKLzW3yZa0jmVU9lGmhp+ZO4uODxGpgB98hp+YQXv9mSKtk3VS14MdXcr6p07ioeNcSc2uj9s=@vger.kernel.org X-Gm-Message-State: AOJu0YyhvXFHtHFi7AXkCD9guW5V17zWaxEsPVEni5mIhakVG74cngQq DLPCmtEFs1GM1cQLD47ps6FHxhzNitdZijZQMW4vVMnvPvIOcYPCKdQmwBK4YnlXlRNNTJTA1pH DggG74Gph0Y5mYQ== X-Google-Smtp-Source: AGHT+IHyYgPrnvJDNnriMm16XcgI4Z+GtksLauuH+Ta7oWXXcU6rl19T9F/meDi87vNPoKDDvgmVTSxx+6DjQQ== X-Received: from eddp6.prod.google.com ([2002:a05:6402:46c6:b0:634:824b:21e0]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:1ecd:b0:634:66c8:9e7d with SMTP id 4fb4d7f45d1cf-63467a209bbmr6324885a12.36.1758726029633; Wed, 24 Sep 2025 08:00:29 -0700 (PDT) Date: Wed, 24 Sep 2025 14:59:56 +0000 In-Reply-To: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250924-b4-asi-page-alloc-v1-0-2d861768041f@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250924-b4-asi-page-alloc-v1-21-2d861768041f@google.com> Subject: [PATCH 21/21] mm: asi_test: smoke test for [non]sensitive page allocs From: Brendan Jackman To: jackmanb@google.com, Andy Lutomirski , Lorenzo Stoakes , "Liam R. Howlett" , Suren Baghdasaryan , Michal Hocko , Johannes Weiner , Zi Yan , Axel Rasmussen , Yuanchu Xie , Roman Gushchin Cc: peterz@infradead.org, bp@alien8.de, dave.hansen@linux.intel.com, mingo@redhat.com, tglx@linutronix.de, akpm@linux-foundation.org, david@redhat.com, derkling@google.com, junaids@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, reijiw@google.com, rientjes@google.com, rppt@kernel.org, vbabka@suse.cz, x86@kernel.org, yosry.ahmed@linux.dev Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Add a simple smoke test for allocating pages of different sensitivities. Since KUnit doesn't (yet) have infrastructure for this, add custom .init and .exit hooks to our kunit_suite that detect WARNs by checking the kernel's taint flags before and after running the tests. Since ASI is disabled by default, whatever command people currently use to run KUnit tests probably won't run these tests. Therefore add a new .kunitconfig file for the x86 tree that explicitly enables ASI. It should be possible to delete this again when ASI is the default. So the most straightforward way to run this test is: tools/testing/kunit/kunit.py run --arch=3Dx86_64 \ --kunitconfig=3Darch/x86/.kunitconfig --kernel_args asi=3Don The more long-winded way, which lets you customize the kernel config, is: mkdir -p .kunit cp arch/x86/.kunitconfig .kunit tools/testing/kunit/kunit.py config --arch=3Dx86_64 make O=3D.kunit menuconfig # Or debug.config or whatever tools/testing/kunit/kunit.py run --arch=3Dx86_64 --kernel_args asi=3Don Signed-off-by: Brendan Jackman --- arch/x86/.kunitconfig | 7 +++ arch/x86/Kconfig | 7 +++ arch/x86/mm/Makefile | 2 + arch/x86/mm/asi_test.c | 145 +++++++++++++++++++++++++++++++++++++++++++++= ++++ kernel/panic.c | 2 + mm/init-mm.c | 3 + 6 files changed, 166 insertions(+) diff --git a/arch/x86/.kunitconfig b/arch/x86/.kunitconfig new file mode 100644 index 0000000000000000000000000000000000000000..83219e6ecca8d2064aba71fab1f= 15d57161fa2e4 --- /dev/null +++ b/arch/x86/.kunitconfig @@ -0,0 +1,7 @@ +CONFIG_PCI=3Dy +CONFIG_MMU=3Dy +CONFIG_64BIT=3Dy +CONFIG_X86_64=3Dy +CONFIG_KUNIT=3Dy +CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION=3Dy +CONFIG_ASI_KUNIT_TESTS=3Dy \ No newline at end of file diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index cb874c3857cf443c6235e05bc3f070b0ea2686f0..a7b5658ecb1203458e06a0a065b= cc7aa7dca8538 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2458,6 +2458,13 @@ config MITIGATION_PAGE_TABLE_ISOLATION =20 See Documentation/arch/x86/pti.rst for more details. =20 +config ASI_KUNIT_TESTS + tristate "KUnit tests for ASI" if !KUNIT_ALL_TESTS + depends on MITIGATION_ADDRESS_SPACE_ISOLATION && KUNIT + default KUNIT_ALL_TESTS + help + Builds the KUnit tests for ASI. + config MITIGATION_RETPOLINE bool "Avoid speculative indirect branches in kernel" select OBJTOOL if HAVE_OBJTOOL diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 5ecbff70964f61a903ac96cec3736a7cec1221fd..7c36ec7f24ebb285fcfc0100042= 06a57536fc990 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -58,3 +58,5 @@ obj-$(CONFIG_X86_MEM_ENCRYPT) +=3D mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) +=3D mem_encrypt_amd.o =20 obj-$(CONFIG_AMD_MEM_ENCRYPT) +=3D mem_encrypt_boot.o + +obj-$(CONFIG_ASI_KUNIT_TESTS) +=3D asi_test.o diff --git a/arch/x86/mm/asi_test.c b/arch/x86/mm/asi_test.c new file mode 100644 index 0000000000000000000000000000000000000000..6076a61980ed9daea63113a30e9= 90eb02a7b08d5 --- /dev/null +++ b/arch/x86/mm/asi_test.c @@ -0,0 +1,145 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +struct free_pages_ctx { + unsigned int order; + struct list_head pages; +}; + +static void action_many__free_pages(void *context) +{ + struct free_pages_ctx *ctx =3D context; + struct page *page, *tmp; + + list_for_each_entry_safe(page, tmp, &ctx->pages, lru) + __free_pages(page, ctx->order); +} + +/* + * Allocate a bunch of pages with the same order and GFP flags, transparen= tly + * take care of error handling and cleanup. Does this all via a single KUn= it + * resource, i.e. has a fixed memory overhead. + */ +static struct free_pages_ctx *do_many_alloc_pages(struct kunit *test, gfp_= t gfp, + unsigned int order, unsigned int count) +{ + struct free_pages_ctx *ctx =3D kunit_kzalloc( + test, sizeof(struct free_pages_ctx), GFP_KERNEL); + + KUNIT_ASSERT_NOT_NULL(test, ctx); + INIT_LIST_HEAD(&ctx->pages); + ctx->order =3D order; + + for (int i =3D 0; i < count; i++) { + struct page *page =3D alloc_pages(gfp, order); + + if (!page) { + struct page *page, *tmp; + + list_for_each_entry_safe(page, tmp, &ctx->pages, lru) + __free_pages(page, order); + + KUNIT_FAIL_AND_ABORT(test, + "Failed to alloc order %d page (GFP *%pG) iter %d", + order, &gfp, i); + } + list_add(&page->lru, &ctx->pages); + } + + KUNIT_ASSERT_EQ(test, + kunit_add_action_or_reset(test, action_many__free_pages, ctx), 0); + return ctx; +} + +/* + * Do some allocations that force the allocator to change the sensitivity = of + * some blocks. + */ +static void test_alloc_sensitive_nonsensitive(struct kunit *test) +{ + unsigned long page_majority; + struct free_pages_ctx *ctx; + gfp_t gfp =3D GFP_KERNEL | __GFP_THISNODE; + struct page *page; + + if (!cpu_feature_enabled(X86_FEATURE_ASI)) + kunit_skip(test, "ASI off. Set asi=3Don in kernel cmdline\n"); + + /* No cleanup here - assuming kthread "belongs" to this test. */ + set_cpus_allowed_ptr(current, cpumask_of_node(numa_node_id())); + + /* + * First allocate more than half of the memory in the node as + * nonsensitive. Assuming the memory starts out unmapped, this should + * exercise the sensitive->nonsensitive flip already. + */ + page_majority =3D (node_present_pages(numa_node_id()) / 2) + 1; + ctx =3D do_many_alloc_pages(test, gfp, 0, page_majority); + + /* Check pages are mapped */ + list_for_each_entry(page, &ctx->pages, lru) { + /* + * Logically it should be an EXPECT, but that would cause heavy + * log spam on failure so use ASSERT for concision. + */ + KUNIT_ASSERT_FALSE(test, direct_map_sensitive(page)); + } + + /* + * Now free them again and allocate the same amount as sensitive. + * This will exercise the nonsensitive->sensitive flip. + */ + kunit_release_action(test, action_many__free_pages, ctx); + gfp |=3D __GFP_SENSITIVE; + ctx =3D do_many_alloc_pages(test, gfp, 0, page_majority); + + /* Check pages are unmapped */ + list_for_each_entry(page, &ctx->pages, lru) + KUNIT_ASSERT_TRUE(test, direct_map_sensitive(page)); +} + +static struct kunit_case asi_test_cases[] =3D { + KUNIT_CASE(test_alloc_sensitive_nonsensitive), + {} +}; + +static unsigned long taint_pre; + +static int store_taint_pre(struct kunit *test) +{ + taint_pre =3D get_taint(); + return 0; +} + +static void check_taint_post(struct kunit *test) +{ + unsigned long new_taint =3D get_taint() & ~taint_pre; + + KUNIT_EXPECT_EQ_MSG(test, new_taint, 0, + "Kernel newly tainted after test. Maybe a WARN?"); +} + +static struct kunit_suite asi_test_suite =3D { + .name =3D "asi", + .init =3D store_taint_pre, + .exit =3D check_taint_post, + .test_cases =3D asi_test_cases, +}; + +kunit_test_suite(asi_test_suite); + +MODULE_LICENSE("GPL"); +MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING"); diff --git a/kernel/panic.c b/kernel/panic.c index d9c7cd09aeb9fe22f05e0b05d26555e20e502d2f..6aa79c5192520af55cd473912d2= ac802de687304 100644 --- a/kernel/panic.c +++ b/kernel/panic.c @@ -39,6 +39,7 @@ #include #include #include +#include =20 #define PANIC_TIMER_STEP 100 #define PANIC_BLINK_SPD 18 @@ -737,6 +738,7 @@ unsigned long get_taint(void) { return tainted_mask; } +EXPORT_SYMBOL_IF_KUNIT(get_taint); =20 /** * add_taint: add a taint flag if not already set. diff --git a/mm/init-mm.c b/mm/init-mm.c index 4600e7605cab43b4bce24b85ec1667db8b92dc80..456b8f7d2ab3bd7963a51908dff= 76713a4e65ab5 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -13,6 +13,8 @@ #include #include =20 +#include + #ifndef INIT_MM_CONTEXT #define INIT_MM_CONTEXT(name) #endif @@ -47,6 +49,7 @@ struct mm_struct init_mm =3D { .cpu_bitmap =3D CPU_BITS_NONE, INIT_MM_CONTEXT(init_mm) }; +EXPORT_SYMBOL_IF_KUNIT(init_mm); =20 void setup_initial_init_mm(void *start_code, void *end_code, void *end_data, void *brk) --=20 2.50.1