From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D13EF1E8353 for ; Thu, 13 Mar 2025 18:11:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889505; cv=none; b=jw9OCdJj0/JQYqkrRhqa3zBv7iAnxzebswEXA5lo2fKONRApaf7mJn/IesRF2Db295pUjeYWWIrXmoYcfIkfvKngaq9H/NX3YUArvWvEz2yfo2bMkJWZjdNZQ725LH70Pj4edNqy2IUatrad9PaxPJXCW/pdWz4TLv3nYgthkJg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889505; c=relaxed/simple; bh=ySyHX086OhKMjV+LGusFiK1XjR1ttUWVQkQVY652QBo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=nGkUUgviWE8I7nWnjhUF57LwVUTonyNQ4BPOYfrJDtrMzWwRjPRYifQH+aWLV3C94YU3PUH6xNxhu85QMjhUhkqJDq/gNDfD9YA7qh7Rg8BOi8qriX5QnwoXbtJy2gcws4egF8JxUxHvKUNKTUF0MZVfyrBXfxVkBj7k4J3x6PA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=VidZKwKu; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="VidZKwKu" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43cec217977so6872045e9.0 for ; Thu, 13 Mar 2025 11:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889502; x=1742494302; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0fZcxWQbpwDGRa6fhxDYyz+3EEc+Y2ZKaTPRMrOqTnk=; b=VidZKwKu6QnKNGVQ43zvHn+n1OZYMJLQh54WWUyXTanyRrw46HJAmmZNJo45P1XgGA WLsvVyn4+M9L035fyLnea4tjidSL0jj/zthS0Y+gx/wJ5z8MWNSY8nmVMQfJ96AgfxTM lnnrqzc9K+h2NS2vqpYyRtLpTB2z3DdfBS71QwIqCtcTrGVb5O4lFlujmnEIf0hvuhKr i17/M9OyWUSLdXmaI9dVc98JQH1SGS9D8YPbSU7oLTidD7kUpF+hqSZjTrFw57qdmnlI N5E9geFu9MeQ5Z3xp69pD58UDFlqPBBk7r3uv9+6Cy9YfPh6vyVldp0+KnlVkUqoYxKi mBcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889502; x=1742494302; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0fZcxWQbpwDGRa6fhxDYyz+3EEc+Y2ZKaTPRMrOqTnk=; b=BffRXzb8YrKW8X20X2KkroAVGQKivsldZmCZ6NNaIsomIFK+HCUa+oCP9eLFZ4xjRs atr7l6VnRKuaGlfzKBKOQ1Kl1IxxjZy9A2c5Fa+J2SSgzxjS6aUd0PRtBypteZrJtcbN 0HjauqqTG6NZUVYVMX2gWjUMoX1ZB/+uU1b/StoOu0sBVdAUnGmq0nDo11rLviWtl6o2 C2DTG2nfWqc+xSB8vgciw9FAibG2gPozUjSiB1wb9KcEW8f9Fq5BrfKloJPG/f08BD69 L+m7nPh5xbJQ+8wdSABQ9gHn4cJ6G4u2osL+VlF8vEbdJcNRH9N3kE8tgQndWG4scIAx eHWQ== X-Gm-Message-State: AOJu0YyiWtxcGaAaq2IoR6Q1OAhf6WL9SGCseqeGlYgzgy+l7A1zD+Nr uF1Bc3violb7lGCq4ZZDnl0h232ThDKOdpuP5gucZHvWs1Mh8Vn5FKT1HdNLWpBuq/O25aOXIvJ GB2aeoOaLMQ== X-Google-Smtp-Source: AGHT+IHiDt1oL77dvdJgK/87oo1nyciMshPbOYF45qDzwy03t12Mx9YK7JD3EmFk7CS3zUEi1Qz1ZwU+UaX2dg== X-Received: from wmbbd19.prod.google.com ([2002:a05:600c:1f13:b0:43c:eb09:3784]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4593:b0:43c:fe15:41dd with SMTP id 5b1f17b1804b1-43d1d87fd9bmr6539185e9.6.1741889502176; Thu, 13 Mar 2025 11:11:42 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:20 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-1-04972e046cea@google.com> Subject: [PATCH RFC 01/11] x86/mm: Bare minimum ASI API for page_alloc integration From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This commit serves to provide a minimal framework to present an ASI integration into the page allocator, without getting distracted by irrelevant details. There's no need to review this actively, just refer back to it as-needed when reading the later patches. In a real [PATCH] series this should be several separate commits. Aside from missing the actual core address-space switching and security logic, this is missing runtime-disablement of ASI. If you enable it in Kconfig, ASI's mm logic gets run unconditionally. That isn't what we want in the real implementation (certainly not in the initial version, anyway). - Add CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION. Attempt to follow the proposal by Mike Rapoport here: https://lore.kernel.org/linux-mm/Z8K2B3WJoICVbDj3@kernel.org/ In this RFC, there's only a small amount of x86-specific logic, perhaps it's possible to implement this logic without any arch/ dependency. But, this is absolutely not true of the full ASI implementation. So that's already reflected in the Kconfig stuff here. - Introduce struct asi, which is an "ASI domain", i.e. an address space. For now this is nothing but a wrapper for a PGD. - Introduce the "global nonsensitive" ASI domain. This contains all the mappings that do not need to be protected from any attacker. Maintaining these mappings is the subject of this RFC. Signed-off-by: Brendan Jackman --- arch/Kconfig | 14 ++++++++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/asi.h | 28 ++++++++++++++++++++++++++++ arch/x86/mm/Makefile | 1 + arch/x86/mm/asi.c | 8 ++++++++ arch/x86/mm/init.c | 3 ++- include/linux/asi.h | 18 ++++++++++++++++++ 7 files changed, 72 insertions(+), 1 deletion(-) diff --git a/arch/Kconfig b/arch/Kconfig index b8a4ff36558228240080a5677f702d37f4f8d547..871ad0987c8740205ceec675a6b= 7304c644f28e1 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -17,6 +17,20 @@ config CPU_MITIGATIONS def_bool y endif =20 +config ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION + bool + +config MITIGATION_ADDRESS_SPACE_ISOLATION + bool "Allow code to run with a reduced kernel address space" + default n + depends on ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION && !PARAVIRT + help + This feature provides the ability to run some kernel code + with a reduced kernel address space. This can be used to + mitigate some speculative execution attacks. + + !PARAVIRT dependency is a temporary hack while ASI has custom + pagetable manipulation code. # # Selected by architectures that need custom DMA operations for e.g. legacy # IOMMUs not handled by dma-iommu. Drivers must never select this symbol. diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 0e27ebd7e36a9e3d69ad3e77c8db5dcf11ae3016..19ceecf5978bbe62e0742072c19= 2c8ee952082dc 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -36,6 +36,7 @@ config X86_64 select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 select EXECMEM if DYNAMIC_FTRACE + select ARCH_HAS_MITIGATION_ADDRESS_SPACE_ISOLATION =20 config FORCE_DYNAMIC_FTRACE def_bool y diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..b8f604df6a36508acbc10710f82= 1d5f95e8cdceb --- /dev/null +++ b/arch/x86/include/asm/asi.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ASI_H +#define _ASM_X86_ASI_H + +#include + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +extern struct asi __asi_global_nonsensitive; +#define ASI_GLOBAL_NONSENSITIVE (&__asi_global_nonsensitive) + +/* + * An ASI domain (struct asi) represents a restricted address space. The + * unrestricted address space (and user address space under PTI) are not + * represented as a domain. + */ +struct asi { + pgd_t *pgd; +}; + +static __always_inline pgd_t *asi_pgd(struct asi *asi) +{ + return asi ? asi->pgd : NULL; +} + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +#endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index 690fbf48e8538b62a176ce838820e363575b7897..89ade7363798cc20d5e5643526e= ba7378174baa0 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -61,6 +61,7 @@ obj-$(CONFIG_ACPI_NUMA) +=3D srat.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) +=3D pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) +=3D kaslr.o obj-$(CONFIG_MITIGATION_PAGE_TABLE_ISOLATION) +=3D pti.o +obj-$(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION) +=3D asi.o =20 obj-$(CONFIG_X86_MEM_ENCRYPT) +=3D mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) +=3D mem_encrypt_amd.o diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c new file mode 100644 index 0000000000000000000000000000000000000000..e5a981a7b3192655cd981633514= fbf945b92c9b6 --- /dev/null +++ b/arch/x86/mm/asi.c @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD= ]; + +struct asi __asi_global_nonsensitive =3D { + .pgd =3D asi_global_nonsensitive_pgd, +}; diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c index 62aa4d66a032d59191e79d34fc0cdaa4f32f88db..44d3dc574881dd23bb48f9af3f6= 191be309405ef 100644 --- a/arch/x86/mm/init.c +++ b/arch/x86/mm/init.c @@ -250,7 +250,8 @@ static void __init probe_page_size_mask(void) /* By the default is everything supported: */ __default_kernel_pte_mask =3D __supported_pte_mask; /* Except when with PTI where the kernel is mostly non-Global: */ - if (cpu_feature_enabled(X86_FEATURE_PTI)) + if (cpu_feature_enabled(X86_FEATURE_PTI) || + IS_ENABLED(CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION)) __default_kernel_pte_mask &=3D ~_PAGE_GLOBAL; =20 /* Enable 1 GB linear kernel mappings if available: */ diff --git a/include/linux/asi.h b/include/linux/asi.h new file mode 100644 index 0000000000000000000000000000000000000000..2d3049d5fe423e139dcce8f3d68= cdffcc0ec0bfe --- /dev/null +++ b/include/linux/asi.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _INCLUDE_ASI_H +#define _INCLUDE_ASI_H + +#include + +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#include +#else + +#define ASI_GLOBAL_NONSENSITIVE NULL + +struct asi {}; + +static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +#endif /* _INCLUDE_ASI_H */ --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0C331EDA0B for ; Thu, 13 Mar 2025 18:11:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889507; cv=none; b=DImIOeshHs9n6Vai0PPj6wKEar8guSPLQu2qpqpqBI5Kw4XnITuNmq6Hp6IALOGqC5bD6qYvbSGjpdAj2sByI5vLJv43FB0fxkR1D1wOdUxZdvlPeKQLxccvkJEqYS6Lwf8fNJDKaIsJe8XzpHbu2UsxNuNi81zBXzqMPE7Gnmo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889507; c=relaxed/simple; bh=QoFHdORegggk6C1oj4h2wdr/nyVe2zV7VnrZowp+f2o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=flSuz76KghgyaikbeieG78q9fRyej4YuEIHrGBIH1WIOWuKT7GXivIbMXZaok2g63GREObcWtj2k1VmfUX1lrEm02sxjQFyVrSgvP9id9SpiWJczLS7CE7lsnWMwK7i0RB7gagv9UhAJXLocLlSdI0k87I/melRcavygR1hHwSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=NNYbY42D; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="NNYbY42D" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3914bc0cc4aso715436f8f.3 for ; Thu, 13 Mar 2025 11:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889504; x=1742494304; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Yd3MTUJa5EaKKcsOqm6EUTHveqQqPDqzenzbNHF0zNE=; b=NNYbY42DBvqILpRNc4ZmVT1AWj+z3pnxHvCazzJFSu8MQw+ygRVnPJeHQ4/Xkw3W7s AzDfm5ziIK654UUOWbq1YV3E914d2LWTlLXXznhd8nAOdeai7+T1YX5a6qh6LOypVdsy X3IUDaSzKbt852Qlkckvf591OU3ed1yzb5M2gbdwgsQ2NH6ztff+Zx8BhMfE+rSnO7IL fwZD91qJ0LaL0W/c+HDADBSNEgEwHbGxq/9K1Ari3LlvzPapXO0jy5BDs00ov/jL3hqV jq5lYl2yghm4UVw74HObrWD/yed+uwZ3GCSMOwakUS+3TVcqmiRtwYGJyCdctp2/gSKZ GGaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889504; x=1742494304; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Yd3MTUJa5EaKKcsOqm6EUTHveqQqPDqzenzbNHF0zNE=; b=lDrLHFW/2V7qs/tcpezg4Zpevvz49SSNNdHh7wWn2IFov1z+8Ndtg9BxPM1QFvqh9C 2zIx3Nqa3ZsxnrkiYvWDMRgl2QXomFkneUD2sK+NJ9U/Yw06hVoZ98zId/iWWQRLJcV8 jTzBwfUQmfju9lxfnJXBY6f8hJdB/knzZt1Io+rM+Esx+SO+uFrlag1VOrd7ZfRiEAAi w9ByFukthOhpeDDE8ADP7oB3JF8t9IthIzEc0/cVxzSca/z15beNuDeN6Z94Y/aMRwIR vQWOBQ5/uMGEpRrpiMu6dSTs18Ozpj2MxkIg7B1djeqWqq9im80MTsOw7lrA3jsLV16x GJSg== X-Gm-Message-State: AOJu0Yz6JBH3tWEJnBsNPH8+k+MONJTxwtvkJKkxtjbjk37IFgjmhFRc 2ciOP5wkvmkHS6i/oKnL17kSuZgLQ/ZdKlpk9JL4NquRX/tKiR3ubW0lAA8aVHs4uVnoqXpGcCb i7amaMbnX+A== X-Google-Smtp-Source: AGHT+IFsOBeq83+HmwcjVeSsK8GiNED4UzJeglwlfgEFKTnF1muToCLx1a+QuwroI+KWJKQsz/7UlqSE3IjxUQ== X-Received: from wmbfp13.prod.google.com ([2002:a05:600c:698d:b0:43c:fb36:c419]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:adf:8b45:0:b0:391:9b2:f49a with SMTP id ffacd0b85a97d-396c3e16b07mr499843f8f.55.1741889504104; Thu, 13 Mar 2025 11:11:44 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:21 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-2-04972e046cea@google.com> Subject: [PATCH RFC 02/11] x86/mm: Factor out phys_pgd_init() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable __kernel_physical_mapping_init() will soon need to work on multiple PGDs, so factor out something similar to phys_p4d_init() and friends, which takes the base of the PGD as an argument. Signed-off-by: Brendan Jackman --- arch/x86/mm/init_64.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 01ea7c6df3036bd185cdb3f54ddf244b79cbce8c..8f75274fddd96b8285aff48493e= bad93e30daebe 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -731,21 +731,20 @@ phys_p4d_init(p4d_t *p4d_page, unsigned long paddr, u= nsigned long paddr_end, } =20 static unsigned long __meminit -__kernel_physical_mapping_init(unsigned long paddr_start, - unsigned long paddr_end, - unsigned long page_size_mask, - pgprot_t prot, bool init) +phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_start, unsigned long pa= ddr_end, + unsigned long page_size_mask, pgprot_t prot, bool init, bool *pgd_c= hanged) { - bool pgd_changed =3D false; unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; =20 + *pgd_changed =3D false; + paddr_last =3D paddr_end; vaddr =3D (unsigned long)__va(paddr_start); vaddr_end =3D (unsigned long)__va(paddr_end); vaddr_start =3D vaddr; =20 for (; vaddr < vaddr_end; vaddr =3D vaddr_next) { - pgd_t *pgd =3D pgd_offset_k(vaddr); + pgd_t *pgd =3D pgd_offset_pgd(pgd_page, vaddr); p4d_t *p4d; =20 vaddr_next =3D (vaddr & PGDIR_MASK) + PGDIR_SIZE; @@ -771,15 +770,29 @@ __kernel_physical_mapping_init(unsigned long paddr_st= art, (pud_t *) p4d, init); =20 spin_unlock(&init_mm.page_table_lock); - pgd_changed =3D true; + *pgd_changed =3D true; } =20 - if (pgd_changed) - sync_global_pgds(vaddr_start, vaddr_end - 1); - return paddr_last; } =20 +static unsigned long __meminit +__kernel_physical_mapping_init(unsigned long paddr_start, + unsigned long paddr_end, + unsigned long page_size_mask, + pgprot_t prot, bool init) +{ + bool pgd_changed; + unsigned long paddr_last; + + paddr_last =3D phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_si= ze_mask, + prot, init, &pgd_changed); + if (pgd_changed) + sync_global_pgds((unsigned long)__va(paddr_start), + (unsigned long)__va(paddr_end) - 1); + + return paddr_last; +} =20 /* * Create page table mapping for the physical memory for specific physical --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C59FC1EF0B5 for ; Thu, 13 Mar 2025 18:11:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889509; cv=none; b=Hi20YWlGnQ2XlHNICpIyrhFUeXUsK4SVJGzwD+vwTkuCb14dgc6R+njKoEF1g6x706K1NmotY0c+qD1LJ5HmlxqZR72paQIoFD+VcU0cSUF61xM9AdICutDNNU3sd+pUpyyJHwLmiX1PXIEHU87Tn2zkma5sYL8r1vzWlVx0VFo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889509; c=relaxed/simple; bh=W4sw0yWx5RkXrhCwq5NJ/AictZZr0NkR62X+pTU0PNo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=BR5Y0e2a960vdIVS/nw30Otv6JTpFY6s5LiND1ZgWPDP9G6A7WmDbyyThRmCbnkxrxJ1WEoZ96uQXgVpJH1f9r7zs+5iH57a24qPyUZciXpQfFSqRNI4hKYQX05t30KlOyYyg1d6VLOPSv5sZl//h9kjclWHAG8W1GizfGBKYkc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=dnjWuWnk; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dnjWuWnk" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-438e4e9a53fso9452735e9.1 for ; Thu, 13 Mar 2025 11:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889506; x=1742494306; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SFFEGXNq9QK4O4fqhmQmuGwjZkzkfGXtfnqX8VsiIbA=; b=dnjWuWnkEFu89uIVNUPc8MmayWJxvZFpqJrfh4aQVOhnvRQ5hmrYrEWSip21TPUc2G mqbY6Dhztp75ljnRHtIS8XIvBpcqKzmK8pxOcpZ+sPFi7E4pTJC8ah5BwQn5V0s+u4bR HW3hMFHkNWwzBvpJ7gxsjSW8oEpscFktsnBexcQoXnGc/mpywFzfQItW3qRr9dRKMveP 0ygdAOvbia+kHuv3tNTHHZ1zsG0XvVfMwd5QfWkzDCGBPxCmmRgOmMpMVI54u/phEZFc 6hPkq1gcZbrHH4clPdL4iCX/CFqc5jSMIAtWFElhY3l/x/345lveGHqykcHX//WHyd2F BA+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889506; x=1742494306; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SFFEGXNq9QK4O4fqhmQmuGwjZkzkfGXtfnqX8VsiIbA=; b=T7oyi1ftrhgEId00cvoXe4K3D9l2nIm235wkZoAc6jfoYyW1aUEadgllwzhuBzs9+g GRqiaplqwVdTnEkhdhFbHg5A+onXro9FNN/pImxRdksolB4Pfi0+rtPcTWKhwvNoejX7 DXJWHkL17U69VuT6At7t+McEJr10og8ZMZnCVRWfSfd+qP/bqesQ6oq8liRFipNGgUHj XEiF2YSlBInLygJvpeBwtPrbTHl8p5XT/4V+XFdCNdfYDSljMgKF4mrJFYpPLTB8VySY qrmOVYggiqANGwW+1FNmkfVqHUJqrYx1rq6fjsRajeCbrtjap5K4+kG+g4DmpSu2deCj /SIQ== X-Gm-Message-State: AOJu0Yzyk9DI7oDPVeAt4ap9ZF66esE3L5JLuak+yQ9MaD5NyQQQ6ZDO ugeZDWIK/nbKN9/8Wvt9P0ACraWn8l8Sh1ucM2qLB/jbljbZyeRiy1P0lMREexxa2NddwVq56BN Rf0S+001lvg== X-Google-Smtp-Source: AGHT+IH1VGtpk3xtUCPvw+qm8+MH2zctSvi1XXF/gGWDnzmEetB35farCcixJ7scWW3mkfOJxNfCCFjLdwFb9g== X-Received: from wmbg20.prod.google.com ([2002:a05:600c:a414:b0:43b:b74b:9350]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:46cb:b0:43c:f470:7605 with SMTP id 5b1f17b1804b1-43d01bdd4a4mr133112715e9.12.1741889506406; Thu, 13 Mar 2025 11:11:46 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:22 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-3-04972e046cea@google.com> Subject: [PATCH RFC 03/11] x86/mm: Add lookup_pgtable_in_pgd() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This is the same thing as lookup_address_in_pgd(), but it returns the pagetable unconditionally instead of returning NULL when the pagetable is none. This will be used for looking up and modifying pages that are *_none() in order to map memory into the ASI restricted address space. For a [PATCH], if this logic is needed, the surrounding code should probably first be somewhat refactored. It now looks pretty repetitive, and it's confusing that lookup_address_in_pgd() returns NULL when pmd_none() but note when pte_none(). For now here's something that works. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/pgtable_types.h | 2 ++ arch/x86/mm/pat/set_memory.c | 34 ++++++++++++++++++++++++++++++++= ++ 2 files changed, 36 insertions(+) diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pg= table_types.h index 4b804531b03c3ce5cc48f0a75cb75d58b985777a..e09b509e525534f31c986d705e0= 7b25dd9c04cb7 100644 --- a/arch/x86/include/asm/pgtable_types.h +++ b/arch/x86/include/asm/pgtable_types.h @@ -572,6 +572,8 @@ extern pte_t *lookup_address_in_pgd(pgd_t *pgd, unsigne= d long address, unsigned int *level); pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, unsigned int *level, bool *nx, bool *rw); +extern pte_t *lookup_pgtable_in_pgd(pgd_t *pgd, unsigned long address, + unsigned int *level); extern pmd_t *lookup_pmd_address(unsigned long address); extern phys_addr_t slow_virt_to_phys(void *__address); extern int __init kernel_map_pages_in_pgd(pgd_t *pgd, u64 pfn, diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index ef4514d64c0524e5854fa106e3f37ff1e1ba10a2..d066bf2c9e93e126757bd32a7a6= 66db89b2488b6 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -658,6 +658,40 @@ static inline pgprot_t verify_rwx(pgprot_t old, pgprot= _t new, unsigned long star return new; } =20 +/* + * Lookup the page table entry for a virtual address in a specific pgd. Re= turn + * the pointer to the entry, without implying that any mapping actually ex= ists + * (the returned value may be zero). + */ +pte_t *lookup_pgtable_in_pgd(pgd_t *pgd, unsigned long address, unsigned i= nt *level) +{ + p4d_t *p4d; + pud_t *pud; + pmd_t *pmd; + + *level =3D PG_LEVEL_256T; + if (pgd_none(*pgd)) + return (pte_t *)pgd; + + *level =3D PG_LEVEL_512G; + p4d =3D p4d_offset(pgd, address); + if (p4d_none(*p4d) || p4d_leaf(*p4d) || !p4d_present(*p4d)) + return (pte_t *)p4d; + + *level =3D PG_LEVEL_1G; + pud =3D pud_offset(p4d, address); + if (pud_none(*pud) || pud_leaf(*pud) || !pud_present(*pud)) + return (pte_t *)pud; + + *level =3D PG_LEVEL_2M; + pmd =3D pmd_offset(pud, address); + if (pmd_none(*pmd) || pmd_leaf(*pmd) || !pmd_present(*pmd)) + return (pte_t *)pmd; + + *level =3D PG_LEVEL_4K; + return pte_offset_kernel(pmd, address); +} + /* * Lookup the page table entry for a virtual address in a specific pgd. * Return a pointer to the entry (or NULL if the entry does not exist), --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11FCA1F0990 for ; Thu, 13 Mar 2025 18:11:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889511; cv=none; b=u4n68U2TIc+TXusYlKf/BKeUSht4i+ZBcV6ptc2MgkVMdCUczIgOzE4qncpn/m/s5HIiR4FVoMSsYJJRIF7TRpbzC9eK15Ewfqdks1MCRc06qu+UtaA+XIO5UnoeDGSgbUWUbFwG8jKajFtim2bvQbAVNfZIkV1wZ6MqfiIX83g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889511; c=relaxed/simple; bh=fGEeQXrjwKQkQP57EyE86cUhUZbL2LxsT8EcaJSEXxg=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=T7x0EWr3KvvROl6uu0LWM2kAszctaOdJfx6ooTNvNnrcn+mB/QMnYUU5o0MTsp3iDw/uOqncwg+78US+dQSVsw4tXuWeVzx9vyU22g8ZRtNAz0h1u3v3soC3mTqmX2cn7f5Rsz0jxX1fRee2hIK8wmDnDNSHimETj+TTF4fZr3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=yh+Wev4o; arc=none smtp.client-ip=209.85.221.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="yh+Wev4o" Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-3913aea90b4so554826f8f.2 for ; Thu, 13 Mar 2025 11:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889508; x=1742494308; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9PPqJkUj7Pibb+TYCrq8XdoSiIxYoCw22sHO21dNMJ4=; b=yh+Wev4orPFstZVXbA4aeH68SGxxOxexddQYOPGq6MpT1OwPaT9ABnOi6aOpNpXywp odLP5iJz26RddTE/vwcGVVHn+Cusme+rzF2VFi1nkwELRvJferVWacVpW5hugCSdTwmd Ai0Ci1gS9FSv0cCZAcgjHj190/EbOb3FO9dz3qO/mbFLPbkEv/uyJdGkKjCuDx7rUDyF dpMIvzsMZRhSXzuZnwNysI0kYGX1ykhrRisA6pFuArAwOzXkoN41dBeYecZlOKK60XKd sOQ1BHvmd0nw5E4k7jcXQqogPJxL9BlZYgUp4PXtAFdxcXgchlubDyEDwT+Ji5RNoszl efpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889508; x=1742494308; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9PPqJkUj7Pibb+TYCrq8XdoSiIxYoCw22sHO21dNMJ4=; b=Q9miIolT7vjrRAjlv9OAooYmIW5HcT8DC0HdXw4E+hQqMLdfFoQkHnBGwKdqeJKO1x jrY/QbNUt4AZN/WuBrnhA1QQXama4ELZex4y6+3DvFf/vRMpwxlIM1OKRZZ0nfhwOV1/ bDbthuQX1CkLdM1BxkFlseTjhCA80Zz7l6T5lz7/WcwYTqe0ytNbyYODUD7ax7M2NfV5 uJnJFBaZRWiUAsCz3/j9fzbc+7uc1qvrsoXhsximywI3sFn6XyB0gD4aXnqTxbppC9nr grrejZE0Q1QyzUlsfwjJATycoL/8RrEdfIzkVOjmN/2w4Sps+BHRyQL9VnrnwajPcpVp E4aA== X-Gm-Message-State: AOJu0Yw+nHtBkMeseTXpD06+Myr3sRJJI9rotCQk8fBSsNmip4l68MDK K0zOovB2nX2tfLn14Kvl4UbX/jxp9P2LVlvjjJuhFzxaavhTMVOP6trFzrQesuXOZAICHRiZpHs 0UXFTZM3MEg== X-Google-Smtp-Source: AGHT+IF8ERnTvg3YNcVI2WgDobZeTnOtswaGwtLGXdYxevgaK+phqTml6l4yAXJ9Dw4d5FjayQ84d9+Hw+R0XA== X-Received: from wmbfm8.prod.google.com ([2002:a05:600c:c08:b0:43b:c9cc:b9b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a5d:598c:0:b0:391:3988:1c7c with SMTP id ffacd0b85a97d-396c2105ea6mr522202f8f.24.1741889508450; Thu, 13 Mar 2025 11:11:48 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:23 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-4-04972e046cea@google.com> Subject: [PATCH RFC 04/11] x86/mm/asi: Sync physmap into ASI_GLOBAL_NONSENSITIVE From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Mirror the physmap into the ASI pagetables, but with a maximum granularity that's guaranteed to allow changing pageblock sensitivity without having to allocate pagetables. Signed-off-by: Brendan Jackman --- arch/x86/mm/init_64.c | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 8f75274fddd96b8285aff48493ebad93e30daebe..4ca6bb419b9643b0e72cb5b6da6= d905f2b2be84b 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -7,6 +7,7 @@ * Copyright (C) 2002,2003 Andi Kleen */ =20 +#include #include #include #include @@ -736,7 +737,8 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_star= t, unsigned long paddr_en { unsigned long vaddr, vaddr_start, vaddr_end, vaddr_next, paddr_last; =20 - *pgd_changed =3D false; + if (pgd_changed) + *pgd_changed =3D false; =20 paddr_last =3D paddr_end; vaddr =3D (unsigned long)__va(paddr_start); @@ -770,9 +772,13 @@ phys_pgd_init(pgd_t *pgd_page, unsigned long paddr_sta= rt, unsigned long paddr_en (pud_t *) p4d, init); =20 spin_unlock(&init_mm.page_table_lock); - *pgd_changed =3D true; + if (pgd_changed) + *pgd_changed =3D true; } =20 + if (pgd_changed) + sync_global_pgds(vaddr_start, vaddr_end - 1); + return paddr_last; } =20 @@ -784,9 +790,29 @@ __kernel_physical_mapping_init(unsigned long paddr_sta= rt, { bool pgd_changed; unsigned long paddr_last; + pgd_t *pgd_asi =3D asi_pgd(ASI_GLOBAL_NONSENSITIVE); =20 paddr_last =3D phys_pgd_init(init_mm.pgd, paddr_start, paddr_end, page_si= ze_mask, prot, init, &pgd_changed); + + /* + * Set up ASI's global-nonsensitive physmap. This needs to mapped at max + * 2M size so that regions can be mapped and unmapped at pageblock + * granularity without requiring allocations. + */ + if (pgd_asi) { + /* + * Since most memory is expected to end up sensitive, start with + * everything unmapped in this pagetable. The page allocator + * assumes that's the case. + */ + pgprot_t prot_np =3D __pgprot(pgprot_val(prot) & ~_PAGE_PRESENT); + + VM_BUG_ON((PAGE_SHIFT + pageblock_order) < page_level_shift(PG_LEVEL_2M)= ); + phys_pgd_init(pgd_asi, paddr_start, paddr_end, 1 << PG_LEVEL_2M, + prot_np, init, NULL); + } + if (pgd_changed) sync_global_pgds((unsigned long)__va(paddr_start), (unsigned long)__va(paddr_end) - 1); --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 698BF1F12E7 for ; Thu, 13 Mar 2025 18:11:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889514; cv=none; b=PTRtuJH3m1kunr0+Gqx189uzi0bEaDS2wZT9qWGazx+ZvPtSTE5TFfcArFkt2kET0RoKnlKETlA0JynNYBWXkxpPI9dPqRT+ZCEmZBAvkV66/WrzbLcD8rkbqEo/r9M0NI01xDRC0Hc015sEBIXq94S6YBeiBZ74ZzbYWT4UfXQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889514; c=relaxed/simple; bh=R7Z7ffLemwCnPpH5COhJObTeiUx2wzTsUCtrICT8b20=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=UxFQpwXHhJzIUnAL3HRnLCsOkIF54eCWHyQkx1X8vMEZTYmsz008KIhz2L36DNGkR8p5Z1W97vIxgsdKAvOlPUduIrYaq7ytFL9p7zmmPjRZq0xHp10/wGJZtlqSFg9AbHvGEfIE85Rhs6u5tJ3U9BUzT5I/Dsj14u+Jx85QRaY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T4iPb9+3; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T4iPb9+3" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43d007b2c79so9403545e9.2 for ; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889511; x=1742494311; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=T4iPb9+35pSq3olPvxKQJXiTNWbvZ/oXKHfHCzsKHEAiWr/yIX63F9RmDEpUfnNkvP rNAXLjSzBA+mg3Lq3UnCMwRgYJjy43yynxBNtoPpmHIS2EqxH08lx5gRykfCb4wq/tJQ tosUxhstUa9Tcua4nhCPQCbItn6ef36okHsdSVPK6P3Z+fPWftfj1yd8udv/hf6ZepZC s27xDXihuCdf520FtLoUOSsOYXh3PvWVVosAe/Z0pt1ZkfC7ORrixNuublwqOPCEZFgN YiCIzEostx5ofHkJjyLtsq4dcLSoT3Y6r6QBNiYEs8j2vBglshnhBrOcv4RiiEfT9Nf2 X1kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889511; x=1742494311; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6vGPxFkxCcqyivuTgzW117dDJKBNuPeXqpXAI/2K5I4=; b=B1gsB+NBjzKkun6duRp1ZyKyzm680AhV0O9aT1RhocAa8s8nnS6H5kqcjZ7Vg4/626 EJN4kqjIvZ8Tct7xBx9UX7NoYArvpaH9+0uICIAlpZot4IoY00ZSk2T1SaU7yoXR00mj osxc4Jzjqxml+pSvd8cqkIgWdoZ/ADGSChNP97prEpj50H12XCZlyQhfM4uBtArUjK74 ZbAwaXpy4ic1TPm2tVGoQ4CklqHlGxDI2OW1h83CxRvqmx2RqKPk4d5rs87QeRYnzTdm qis6N3IGKX/NSoNvjKLzcciQkTRxi50ztJPYrNbMohuBL9rEYcLAKB0tEDCekRX23UN1 GsMQ== X-Gm-Message-State: AOJu0YynvhC8+hRDBECl++v1fRCMChbMrt9BeAXDiD1omFpG0fsCk6do bif6TcPHjOuaTdNKfobRsCoebdOMLf+hAgJrK1TgrBGG07p2F/Ou07dwem2EaG9a347/9wqXsxr +dk0aMuKCrg== X-Google-Smtp-Source: AGHT+IF292MH6JqAkiIN8jtyln5hVYHbTRuF9S/Wm78jwx44s0FdKYuBZ0MucR5gttxIvSprRDkRwKMLpsnBdw== X-Received: from wmbfj6.prod.google.com ([2002:a05:600c:c86:b0:43c:f256:f4b3]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1d12:b0:43c:eec7:eab7 with SMTP id 5b1f17b1804b1-43d01bdd224mr152891025e9.11.1741889510788; Thu, 13 Mar 2025 11:11:50 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:24 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-5-04972e046cea@google.com> Subject: [PATCH RFC HACKS 05/11] Add asi_map() and asi_unmap() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Do not worry too much about the implementation of this. I am trying to implement this more neatly using the x86 PAT logic but haven't managed to get it working in time for the RFC. To enable testing & review, I'm very hastily throwing something together that basically works, based on a simplified version of what was used for the latest RFC [0]. [0] https://lore.kernel.org/linux-mm/20250110-asi-rfc-v2-v2-0-8419288bc805@= google.com/ Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 3 ++ arch/x86/mm/asi.c | 77 ++++++++++++++++++++++++++++++++++++++++++= ++++ include/linux/asi.h | 2 ++ include/linux/vmalloc.h | 4 +++ mm/vmalloc.c | 32 +++++++++++-------- 5 files changed, 105 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index b8f604df6a36508acbc10710f821d5f95e8cdceb..cf8be544de8b108190b765e3eb3= 37089866207a2 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -23,6 +23,9 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } =20 +void asi_map(struct page *page, int numpages); +void asi_unmap(struct page *page, int numpages); + #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ =20 #endif /* _ASM_X86_ASI_H */ diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index e5a981a7b3192655cd981633514fbf945b92c9b6..570233224789631352891f47ac2= f0453a7adc06e 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -1,8 +1,85 @@ // SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include + #include +#include =20 static __aligned(PAGE_SIZE) pgd_t asi_global_nonsensitive_pgd[PTRS_PER_PGD= ]; =20 struct asi __asi_global_nonsensitive =3D { .pgd =3D asi_global_nonsensitive_pgd, }; + +/* + * Map the given pages into the ASI nonsensitive physmap. The source of the + * mapping is the regular unrestricted page tables. Only supports mapping = at + * pageblock granularity. Does no synchronization. + */ +void asi_map(struct page *page, int numpages) +{ + unsigned long virt; + unsigned long start =3D (size_t)(page_to_virt(page)); + unsigned long end =3D start + PAGE_SIZE * numpages; + unsigned long page_size; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + for (virt =3D start; virt < end; virt =3D ALIGN(virt + 1, page_size)) { + pte_t *pte, *pte_asi; + int level, level_asi; + pgd_t *pgd =3D pgd_offset_pgd(asi_global_nonsensitive_pgd, virt); + + pte_asi =3D lookup_pgtable_in_pgd(pgd, virt, &level_asi); + page_size =3D page_level_size(level_asi); + + pte =3D lookup_address(virt, &level); + if (!pte || pte_none(*pte)) + continue; + + /* + * Physmap should already be setup by PAT code, with no pages + * smaller than 2M. This function should only be called at + * pageblock granularity. Thus it should never be required to + * break up pages here. + */ + if (WARN_ON_ONCE(!pte_asi) || + WARN_ON_ONCE(ALIGN_DOWN(virt, page_size) < virt) || + ALIGN(virt, page_size) > end) + continue; + + /* + * Existing mappings should already match the structure of the + * unrestricted physmap. + */ + if (WARN_ON_ONCE(level !=3D level_asi)) + continue; + + set_pte(pte_asi, *pte); + } +} + +/* + * Unmap pages previously mapped via asi_map(). + * + * Interrupts must be enabled as this does a TLB shootdown. + */ +void asi_unmap(struct page *page, int numpages) +{ + size_t start =3D (size_t)page_to_virt(page); + size_t end =3D start + (PAGE_SIZE * numpages); + pgtbl_mod_mask mask =3D 0; + + VM_BUG_ON(!IS_ALIGNED(page_to_pfn(page), pageblock_nr_pages)); + VM_BUG_ON(!IS_ALIGNED(numpages, pageblock_nr_pages)); + + vunmap_pgd_range(asi_pgd(ASI_GLOBAL_NONSENSITIVE), start, end, &mask); + + flush_tlb_kernel_range(start, end - 1); +} diff --git a/include/linux/asi.h b/include/linux/asi.h index 2d3049d5fe423e139dcce8f3d68cdffcc0ec0bfe..ee9811f04a417556cf2e930644e= af05f3c9bfee3 100644 --- a/include/linux/asi.h +++ b/include/linux/asi.h @@ -13,6 +13,8 @@ struct asi {}; =20 static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_map(struct page *page, int numpages) { } +static inline void asi_unmap(struct page *page, int numpages) { } =20 #endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ #endif /* _INCLUDE_ASI_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 31e9ffd936e39334ddaff910222d4751c18da5e7..c498ba127b4a511b5a6f10afa2a= ae535509fc153 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -11,6 +11,7 @@ #include /* pgprot_t */ #include #include +#include =20 #include =20 @@ -324,4 +325,7 @@ bool vmalloc_dump_obj(void *object); static inline bool vmalloc_dump_obj(void *object) { return false; } #endif =20 +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long = end, + pgtbl_mod_mask *mask); + #endif /* _LINUX_VMALLOC_H */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 61981ee1c9d2f769d4a06ab542fc84334c1b0cbd..ffeb823398809388c0599f51929= a7f3506ed035f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -427,6 +427,24 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long= addr, unsigned long end, } while (p4d++, addr =3D next, addr !=3D end); } =20 +void vunmap_pgd_range(pgd_t *pgd_table, unsigned long addr, unsigned long = end, + pgtbl_mod_mask *mask) +{ + unsigned long next; + pgd_t *pgd =3D pgd_offset_pgd(pgd_table, addr); + + BUG_ON(addr >=3D end); + + do { + next =3D pgd_addr_end(addr, end); + if (pgd_bad(*pgd)) + *mask |=3D PGTBL_PGD_MODIFIED; + if (pgd_none_or_clear_bad(pgd)) + continue; + vunmap_p4d_range(pgd, addr, next, mask); + } while (pgd++, addr =3D next, addr !=3D end); +} + /* * vunmap_range_noflush is similar to vunmap_range, but does not * flush caches or TLBs. @@ -441,21 +459,9 @@ static void vunmap_p4d_range(pgd_t *pgd, unsigned long= addr, unsigned long end, */ void __vunmap_range_noflush(unsigned long start, unsigned long end) { - unsigned long next; - pgd_t *pgd; - unsigned long addr =3D start; pgtbl_mod_mask mask =3D 0; =20 - BUG_ON(addr >=3D end); - pgd =3D pgd_offset_k(addr); - do { - next =3D pgd_addr_end(addr, end); - if (pgd_bad(*pgd)) - mask |=3D PGTBL_PGD_MODIFIED; - if (pgd_none_or_clear_bad(pgd)) - continue; - vunmap_p4d_range(pgd, addr, next, &mask); - } while (pgd++, addr =3D next, addr !=3D end); + vunmap_pgd_range(init_mm.pgd, start, end, &mask); =20 if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4CF0A1F130B for ; Thu, 13 Mar 2025 18:11:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889516; cv=none; b=uFtZ2oGm/W6aVvwWuOA1ySA/BST7OVhahByPeT8HcIHDvxcDZ1G7WGSeeSLQvA1yF4qY5Cbbi4BVGB45KzXf5ev10CNw2BCiKS8FziFMWbXdxPgqA/Gl5wYBiIDkRx0ipXCWKGKJPH4z7Gcb6XjTV9etfzlhh81/NwEDoOW6unw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889516; c=relaxed/simple; bh=MsTi0WCWVUenFgw7wL5/5lBWJN3wIK7wvteAaC3Fi60=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=B5cOq3GIByudbsEweg2ND9w9kp8Z97R80VsSAvRnPusqWN+IfPezrHAwzPU/Bw2LEANOEMq/8uQ7zUxHXlpBP9EEq51gUoRmeJxhgq8sctVLWW7zBHrsyFlL9LmxhRo9OSZZIQLSaGviLT+l8vnFs9LRac0vWjctcWsI9IOxKpo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TFWvTbdF; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TFWvTbdF" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43bd8fd03a2so7350695e9.3 for ; Thu, 13 Mar 2025 11:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889512; x=1742494312; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6roQ0y9HMVWb0fLuDGnNUraXTJD3yygGf4+oRkUuB18=; b=TFWvTbdFq9oqlwun6t2NWRXsClii6rGePwCzD0ZWuP+hGNGBZqvzyAnKgNgThOxPxO i/BwOesT9gweuFXEIq5kjoOPMfsBTpcPmw5F466t4PPzCjrVtmzHiwr9CHxQ0qIhKo0M jpEwTHBD6yl7cshuTxk6cTy5czKrMMfEqalRfjSWmg6BsmCJRIZ+A+NbY+du7CSIN33/ DHNJPzj5S7aroILiTT7zKlTf6VQWJDxpom8bRr5BryxzjAbpKYFEScUQtcRVYsCz0Vdc QNjmQKNAJnT5mK/O6+wba0u+a5xBsljIoLmwHiUxuaKyDiqDVEhrrfDptoRTZ0V9Q3E5 RYgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889512; x=1742494312; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6roQ0y9HMVWb0fLuDGnNUraXTJD3yygGf4+oRkUuB18=; b=U85E50nWc6ewihlS4xEJQ0IYEcpXxmsoCAAbA/6gzVZb1n7AwfCTSA6q9RdC6nrDCD KOGi/dmBZbfrZqUOH68hEIIyvwM3geX1hqzQZay9LNVy69wGG4na/Zfer44e7YxEUA9h WYzAduhYsu6QJLkLTAZpGWwjZly5S6K4KWDDTf0JbqLMh//jzui+dMMGx23KqkeSQJKG zsp2MCqWuET+2wIg9fxbN0OgRxBy3cvZEsv6ElxLnTeV8z0+dlfa7t0qf0un/9ecIF2J MP3g7+kZTKkkdWI871x2FNO/krwISbJBUYXtoBBOQI5fchmeuqN+7HRWUCQT464WsHL5 AtxQ== X-Gm-Message-State: AOJu0YzjaTm9Fo+E8eiIoSSa9JwvT6GFps6RPE9hsewfIf9PY6IMHjFO tUKZenUtk9iWF+NYXZTYTAF7+XoSnbf5OM0uITdq4LgGdeTPG8kpIY7uBTKiil9Hcw6h+XgkGbw FntlCTz3jdA== X-Google-Smtp-Source: AGHT+IEIusjvqc+xKBD+wnnYFxl05hffn8LYeM0xBLqlMTC3EbbSvCojjd4lwP+OgtLOTfROmKsVm430j1MBCA== X-Received: from wmrs15.prod.google.com ([2002:a05:600c:384f:b0:43d:c77:3fd8]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e8b:b0:43b:c5a3:2e1a with SMTP id 5b1f17b1804b1-43d1d88c5ddmr6267935e9.2.1741889512673; Thu, 13 Mar 2025 11:11:52 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:25 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-6-04972e046cea@google.com> Subject: [PATCH RFC 06/11] mm/page_alloc: Add __GFP_SENSITIVE and always set it From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable __GFP_SENSITIVE represents that a page should not be mapped into the ASI restricted address space. This is added as a GFP flag instead of via some contextual hint, because its presence is not ultimately expected to correspond to any such existing context. If necessary, it should be possible to instead achieve this optionality with something like __alloc_pages_sensitive(), but this would be much more invasive to the overall kernel. On startup, all pages are sensitive. Since there is currently no way to create nonsensitive pages, temporarily set the flag unconditionally at the top of the allocator. __GFP_SENSITIVE is also added to GFP_USER since that's the most important data that ASI needs to protect. Signed-off-by: Brendan Jackman --- include/linux/gfp_types.h | 15 ++++++++++++++- mm/page_alloc.c | 7 +++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 65db9349f9053c701e24bdcf1dfe6afbf1278a2d..5147dbd53eafdccc32cfd506569= b04d5c082d1b2 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -58,6 +58,7 @@ enum { #ifdef CONFIG_SLAB_OBJ_EXT ___GFP_NO_OBJ_EXT_BIT, #endif + ___GFP_SENSITIVE_BIT, ___GFP_LAST_BIT }; =20 @@ -103,6 +104,11 @@ enum { #else #define ___GFP_NO_OBJ_EXT 0 #endif +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ___GFP_SENSITIVE BIT(___GFP_SENSITIVE_BIT) +#else +#define ___GFP_SENSITIVE 0 +#endif =20 /* * Physical address zone modifiers (see linux/mmzone.h - low four bits) @@ -299,6 +305,12 @@ enum { /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) =20 +/* + * Allocate sensitive memory, i.e. do not map it into ASI's restricted add= ress + * space. + */ +#define __GFP_SENSITIVE ((__force gfp_t)___GFP_SENSITIVE) + /* Room for N __GFP_FOO bits */ #define __GFP_BITS_SHIFT ___GFP_LAST_BIT #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) @@ -380,7 +392,8 @@ enum { #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM | __GFP_NOWARN) #define GFP_NOIO (__GFP_RECLAIM) #define GFP_NOFS (__GFP_RECLAIM | __GFP_IO) -#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | __GFP_HARDWALL) +#define GFP_USER (__GFP_RECLAIM | __GFP_IO | __GFP_FS | \ + __GFP_HARDWALL | __GFP_SENSITIVE) #define GFP_DMA __GFP_DMA #define GFP_DMA32 __GFP_DMA32 #define GFP_HIGHUSER (GFP_USER | __GFP_HIGHMEM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index c9a1936c3070836a3633ebd32d423cd1a27b37a3..aa54ef95233a052b5e79b4994b2= 879a72ff9acfd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4686,6 +4686,13 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, = unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac =3D { }; =20 + /* + * Temporary hack: Allocation of nonsensitive pages is not possible yet, + * allocate everything sensitive. The restricted address space is never + * actually entered yet so this is fine. + */ + gfp |=3D __GFP_SENSITIVE; + /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42B141F180F for ; Thu, 13 Mar 2025 18:11:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889517; cv=none; b=qvGK45pVXRz4Kilcb5Tm/FiHtDrMOm6h4PfiF83LxCV13+wwm2OO4IzacQh2kCfwx+7sgS0zVKR2XE3Y6En0mU8otY0rJLDVzRrVJ7CTe3IhjXc5MRJqkniRakYZFNyvoj18cDjfstQkyWxgBlcmbrQM260+WDMLEnoQ+uuqa7k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889517; c=relaxed/simple; bh=S1m2mc+yGnN4a5PigzsimvcF9PVJTalTzczZMp/6AK8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hgUl44l+EoToLRQODlAVpvjkZviVbgkAvOi9aBmfun13ARByRvHaDIsUCQJtPiJi7tYD6xDFn0MGYDWHbdn+/1x/TVeBlWqRGH6lhcqqQ+fL3fUhOr5Qi9cwxu3fWB0FHwXVd0fhAp6NzeMbCASh2GwFCEnoArJ0za5MQvMTEQ8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xZmFUlHD; arc=none smtp.client-ip=209.85.128.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xZmFUlHD" Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-43ced8c2eb7so9317505e9.1 for ; Thu, 13 Mar 2025 11:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889514; x=1742494314; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+pdfFaB/k81zEw2QO02syAUWJ0A/YbE7SLXQbmysWgY=; b=xZmFUlHDDz7rC/so6v03PHagsW0e4S800gW8qdIH8JMEmWoQKYGtKSjLv51mSX3hKM CwZlD6sAT8oVYpNSpgN4S3UXwx+kNUB5cE8t033WiOZ8jqa4uoiLyCRSlnl8lnTUpa4/ 0QTVJw8Tx2K/ptckBDgEMUZX6yZwHSYhPK3fyUfwR77JMvGTeIPEysEoL3QoBudqjZKl qtGJ15eiXKhJ4kBkEq/XtVGoSCi7D+7D6Fj51TjTQi6tbRNvq1dkct4i3Td6Ora1HY7Q VL+4EGeTMagOzEDTwRJ3auI3Cz6NAWtVPhOOLxlz2v2izN8m9bjINM/A1WIis54Givj4 ARmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889514; x=1742494314; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+pdfFaB/k81zEw2QO02syAUWJ0A/YbE7SLXQbmysWgY=; b=hEplSOqpLJUdq+dvPZZdOY8R+Yg3Bpq0HcCNSgwjyYeBf1YaNvrrJ74froMwXNQpyE /x8pqi+S84fzIKn13CU23Daang1Ak8TNzXByLzd7UA3BtTfMZtvmwbd8oMsq4KIb8r0G RRIMt1z8hJX3rCyda/5UszKGWJqWu56Olg+vcdIT0uYPjuEt1ykzCDDEronHM2cCrQuy Un0u560VW9uoHnrJrFjOsRXFD42pXAETBcERS3OAzgnhgcgKKXsPYZrzmtxVUGy25xxD 2BDC912w4HZ4lruyR9GJpr3zMdTlUB4GfdISYYWm/uWCyPwlhg/FTe9ijhk8LxIUYRWh knqg== X-Gm-Message-State: AOJu0Yx/u/PhvwgOXB/yVRQR6eLJTrGqGqv1XTNymFo5demi+nuJCA7J HmWp2wAVyd846wSPMKdJ0J8aj3D3kFQ587YF12qWXAL9elKGi0B2gofoOojmhe/9EBQ4SuJmnrf qQf2hfExbRA== X-Google-Smtp-Source: AGHT+IHRjd714WqfAw15oFXCnMwvJ6lJg1SUh1wqrNovE6dSYYDgAsuTLSBaeJXNp59r6szjiAIaZ5iRK8TFZQ== X-Received: from wmbez8.prod.google.com ([2002:a05:600c:83c8:b0:43c:fc61:959c]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4e8f:b0:43c:fb95:c76f with SMTP id 5b1f17b1804b1-43d1d897497mr5845225e9.9.1741889514676; Thu, 13 Mar 2025 11:11:54 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:26 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-7-04972e046cea@google.com> Subject: [PATCH RFC HACKS 07/11] mm/slub: Set __GFP_SENSITIVE for reclaimable slabs From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable It's not currently possible allocate reclaimable, nonsensitive pages. For the moment, just add __GFP_SENSITIVE. This will need to be fixed before this can be a [PATCH]. Signed-off-by: Brendan Jackman --- mm/slub.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 1f50129dcfb3cd1fc76ac9398fa7718cedb42385..132e894e96df20f2e2d69d0b602= b4719cdc072f5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -5559,7 +5559,11 @@ static int calculate_sizes(struct kmem_cache_args *a= rgs, struct kmem_cache *s) s->allocflags |=3D GFP_DMA32; =20 if (s->flags & SLAB_RECLAIM_ACCOUNT) - s->allocflags |=3D __GFP_RECLAIMABLE; + /* + * TODO: Cannot currently allocate reclaimable, nonsensitive + * pages. For the moment, just add __GFP_SENSITIVE. + */ + s->allocflags |=3D __GFP_RECLAIMABLE | __GFP_SENSITIVE; =20 /* * Determine the number of objects per slab --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7927C1F2BBB for ; Thu, 13 Mar 2025 18:11:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889520; cv=none; b=IovoB5Ge4qAdu2fUX00fm1uHwOFrCjZzpgxSA4+9dTrWGkT3quYTT1sbdgJklweQEJNCtYrdGT+0EK52ml6dXIoeJZB79UX3OvQhbOzaEqxor4BhZPb2iZcXiisJiobGz0EVXhP3Xkhk564wYGYipqMIA5M1rYQIQGMdzvTHkk0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889520; c=relaxed/simple; bh=I6idEbcATjINv9XdShmp78j79HzBX3Z2G8brrJmIrc8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cDXdXOHfbPROw+kto/aOSvdBXxBTzsJeT7bO8FqjeBYtvRyEnRKksumXQu2WD/NvBl1sZ99uFsd9qQEnbXtUDAZnVy9Tkgj+GnoctROdWU12fC5KTxBaj+MklDFVE/CpmfCD7Vm7gP+m5/6k9DqKWJBcXUElzfR5QxrF4ZguSFA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rV6V5l1Y; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rV6V5l1Y" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43ceed237efso9521515e9.0 for ; Thu, 13 Mar 2025 11:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889517; x=1742494317; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tRsJowh9EEZA56z2uUOhJ3rtTOtbuzhctlmDmyeE3Lo=; b=rV6V5l1YAyjRZSBpjbfpM1p4vN6TLo68K1WekOHm10y89m9u9KvEp3S4o397bnYQZh s67O0rfCzrKFILNpcIoohxPPPh4fgfJG13tq2J8FvCfjZI+qou+GCgkYnWVjhiJ0V5A6 zZoSN9gZkW618zp6WS0zsaxobYaaIfuwZhCdQBEesLZfJ8tC6m/ipqNgpbG4co7l0f/D ZtcO3aLmSYS/8L1RUQyi4Eea5nO5BXiWoHX3vyj15HWBELS+KGqusWwSI/Dka05dTDUt WwZxGD32r1I7GnUgm4c0DJP1m8zBrYKUR6+LpHq15czX61rMLKoAGri1lcm8eTbNVG/M Hx+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889517; x=1742494317; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tRsJowh9EEZA56z2uUOhJ3rtTOtbuzhctlmDmyeE3Lo=; b=QfouvjDiNLKL8+mBJReKCzU0po0cMqnMAEYa6V3kC7UevjVzIR+IID2O+JUG6jKrVA MPsy9coG12mcbvoXfikdKdnO5TZL0FCMO1DxTGBPLe4FREjtLdC8js2RvlGeawql4JvV JlUo5Z6MfDPNsD5/Qgop/s66ELY9U+hN4THayYdrrVAGexl0/N3f5MZd73hvDr5OwY6j 01tYBM6jM5OMcM9u8IUruayEDKcftSuZsMz8ni+ZH37XfbsaRt6iOEmMgiGPf+f9UC0r oOyOGxxG84LvAZ014XPyDKkhVC+IsORYeKZVN+otIGqFwid/KOBSYG1+SUXQOdhWCdnz LkBg== X-Gm-Message-State: AOJu0Yws8cIj4UPC59EMFnmFlWI7nu0V6RZeLVpV6dNxxCALLAAJtiDn Q1DTWbbk0rJiPItl9f83126dK/N1m7OYtOcCymVoAZfX1S5wY+cfPjyun2nPxriKeT+ygs6SdLQ 2fX9vmT1IiA== X-Google-Smtp-Source: AGHT+IFfyCsrs7ccNNldH6c3NwbPOVylewQWVrKq/G7/S6WFMNKzAcRUB4lJBTdyYKD3llafwzpnB+NDVR+YSw== X-Received: from wmbeq4.prod.google.com ([2002:a05:600c:8484:b0:43b:c967:2f53]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4f14:b0:43d:7a:471f with SMTP id 5b1f17b1804b1-43d01be7e4bmr134665765e9.18.1741889516996; Thu, 13 Mar 2025 11:11:56 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:27 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-8-04972e046cea@google.com> Subject: [PATCH RFC HACKS 08/11] mm/page_alloc: Simplify gfp_migratetype() From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable This currently uses optimised bit-hacks to avoid conditional branches etc. For the purposes of the RFC, let's not get bogged down in those details - temporarily just drop the bit hacking. Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 6bb1a5a7a4ae3392c1cd39cb79271e05512adbeb..23289aa54b6c38a71a908e5a6e0= 34828a75a3b66 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -14,25 +14,20 @@ struct mempolicy; =20 /* Convert GFP flags to their corresponding migrate type */ #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE) -#define GFP_MOVABLE_SHIFT 3 =20 static inline int gfp_migratetype(const gfp_t gfp_flags) { VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) =3D=3D GFP_MOVABLE_MASK); - BUILD_BUG_ON((1UL << GFP_MOVABLE_SHIFT) !=3D ___GFP_MOVABLE); - BUILD_BUG_ON((___GFP_MOVABLE >> GFP_MOVABLE_SHIFT) !=3D MIGRATE_MOVABLE); - BUILD_BUG_ON((___GFP_RECLAIMABLE >> GFP_MOVABLE_SHIFT) !=3D MIGRATE_RECLA= IMABLE); - BUILD_BUG_ON(((___GFP_MOVABLE | ___GFP_RECLAIMABLE) >> - GFP_MOVABLE_SHIFT) !=3D MIGRATE_HIGHATOMIC); =20 if (unlikely(page_group_by_mobility_disabled)) return MIGRATE_UNMOVABLE; =20 - /* Group based on mobility */ - return (__force unsigned long)(gfp_flags & GFP_MOVABLE_MASK) >> GFP_MOVAB= LE_SHIFT; + switch (gfp_flags & GFP_MOVABLE_MASK) { + case __GFP_RECLAIMABLE: return MIGRATE_RECLAIMABLE; + case __GFP_MOVABLE: return MIGRATE_MOVABLE; + default: return MIGRATE_UNMOVABLE; + } } -#undef GFP_MOVABLE_MASK -#undef GFP_MOVABLE_SHIFT =20 static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) { --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A4F6C1F3B83 for ; Thu, 13 Mar 2025 18:12:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889522; cv=none; b=dziKqViFNg+ambLTVOSIllgXeyWsZmvhdxIOpGq5l0hVoWp01s8xlskM5MqBbqCwKU7DlwkYoLYumWbOo6wB0wEVvlcXH81zO735SDvYuTI+4IlKjKYFSWp7wCkWUhUEi5BlAgc6qD8015mN4SBxwuHs1eHWqrCs5vAb+E3VcWk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889522; c=relaxed/simple; bh=4U2OQp6CokKOk0d8UXBcXGW+WFDNfme7rxCCHcR0p8k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=a5zfRoWaaj3P4cKId0g1WgeIQYBqqB0y+CTyc+3905ac3umy5u6zUukS8DtbtUfeU1VgzA/cY5bNiSyhIP8ct4L2JjZGr5A6Doqkx3bEQhrPnP+bhWQl8UqDw6NW/CCACKxJXLFwbXUexxDGkIvEXL1wqZYgSHSm0BLwFwSZCdA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=fehhMPoC; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="fehhMPoC" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43ceeaf1524so6821705e9.1 for ; Thu, 13 Mar 2025 11:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889519; x=1742494319; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GhuPyoh4JQENDoXkPgzWqVfgnUM5Erl7lymo551VcRQ=; b=fehhMPoCF0hPLBDlPoyCfhYkyIoujTwBA/4yo9Z0Qi54WGiMJhI93GSeuJB+m1RdMY n2ULLtYGyvQpi2rwGzirbjyop13eYxCAj1EaBhj+7T1CTy2BKt3aOnw4ypHCS3KuAH+e eJixFpYdKpp9/NQ50x8eULnjHbVMpf2BXyk8PBxX/l7eRH9mFYCEKzTGFgaCMIsFu9rv DTNZFnjwkDIIDMM58T++GXL9UW4okz5S2cgQ9EUKWvBUO4mm4Yf3UDgbLcvvxYu690vF kpAPV9qmNlsLeK+WR5loHdrfiS/qqWNML+CkzCy1xEzDVU/q5n+91xiT1Fmbz5RPLU27 ZLvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889519; x=1742494319; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GhuPyoh4JQENDoXkPgzWqVfgnUM5Erl7lymo551VcRQ=; b=OZQlDMqLtLNCs9XWAkDOkgsbviyMb65qwzwGxA7kmycSf+CES8gpSYnUJ3792+i1eG sDWPtPU9Q0DQkv4fLPkmYGYgGsIuszqOkq61kw6kZGjNwooJqoNSDU2adPs4yu9nnoTT Tq/YkOrcSUAva6JaWXcCrd/idTwgvHSsnmexcuvl6PQSPj4fk80582cCr/PJZitPn0ZL lybSNHDruIUnj5JPW8IJMEU15T5uZiKobww0sL9BXMrRZ4R/nmPoE1Jd5vl0K7r5qc80 MeZyrCuHPY6tbRiRE+CcPH+/x67E+X132QP6L1QYIfVkaGFuaV6DrJCgdgdlbwTlOFta y2lQ== X-Gm-Message-State: AOJu0YxVGdm/ur0irTbEYlRRubxHkKSZD7RSpKLzH4fhLqGeuv7oMAr3 TLuGNMV562u5/8ABf//XvoTmWh0NMIta3nWZcI+rt5GPR/ODttMbEW6YNb2MjG+VT3M2g6Asi4f GfiBcFClSvw== X-Google-Smtp-Source: AGHT+IF61pDIZp4JV1y+yCI+Y7/rxQbZzIOQPlfA/NYp0rFg9DJMnWUvBtDooARG1ee4zqcfgN9USmHD1j/7QA== X-Received: from wmsp2.prod.google.com ([2002:a05:600c:1d82:b0:43c:f517:ac4e]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3143:b0:439:4c1e:d810 with SMTP id 5b1f17b1804b1-43d1808593dmr31744305e9.9.1741889518918; Thu, 13 Mar 2025 11:11:58 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:28 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-9-04972e046cea@google.com> Subject: [PATCH RFC 09/11] mm/page_alloc: Split MIGRATE_UNMOVABLE by sensitivity From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable When ASI is compiled in, create two separate unmovable migratetypes. MIGRATE_UNMOVABLE_NONSENSITIVE represents blocks that are mapped into ASI's restricted address space. MIGRATE_UNMOVABLE becomes MIGRATE_UNMOVABLE_SENSITIVE. All other migratetypes retain their original meaning and gain the additional implication that the pageblock is not ASI-mapped. In future extensions it's likely that more migratetypes will need to support different sensitivities; if and when that happens a more invasive change will be needed but for now this should allow developing the necessary allocator logic for flipping sensitivities by modifying the ASI page tables. For builds with ASI disabled, the two new migratetypes are aliases for one another. Some code needs to be aware of this aliasing (for example, the 'fallbacks' array needs an ifdef for the entries that would otherwise alias) while others doesn't (for example, set_pageblock_migratetype() just works regardless). Since there is now a migratetype below MIGRATE_PCPTYPES with no fallbacks, the 'fallbacks' arrays are no longer all the same size, so make them be terminated by a -1 instead of having a fixed size. On non-ASI builds, the new 'if (fallback_mt < 0)' in find_suitable_fallback() is provably always false and can be eliminated by the compiler. Clang 20 seems to be smart enough to do this regardless, but add a 'const' qualifier to the arrays to try and increase confidence anyway. Signed-off-by: Brendan Jackman --- include/linux/gfp.h | 21 +++++++++++++++++---- include/linux/mmzone.h | 19 ++++++++++++++++++- mm/memory_hotplug.c | 2 +- mm/page_alloc.c | 22 +++++++++++++++------- mm/show_mem.c | 13 +++++++------ 5 files changed, 58 insertions(+), 19 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 23289aa54b6c38a71a908e5a6e034828a75a3b66..0253dcfb66cbaa82414528a2662= 71d719bc09d6a 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -20,13 +20,26 @@ static inline int gfp_migratetype(const gfp_t gfp_flags) VM_WARN_ON((gfp_flags & GFP_MOVABLE_MASK) =3D=3D GFP_MOVABLE_MASK); =20 if (unlikely(page_group_by_mobility_disabled)) - return MIGRATE_UNMOVABLE; + goto unmovable; + + /* Only unmovable/unreclaimable pages can be nonsensitive right now. */ + VM_WARN_ONCE((gfp_flags & GFP_MOVABLE_MASK) && !(gfp_flags & __GFP_SENSIT= IVE), + "%pGg", &gfp_flags); =20 switch (gfp_flags & GFP_MOVABLE_MASK) { - case __GFP_RECLAIMABLE: return MIGRATE_RECLAIMABLE; - case __GFP_MOVABLE: return MIGRATE_MOVABLE; - default: return MIGRATE_UNMOVABLE; + case __GFP_RECLAIMABLE: + return MIGRATE_RECLAIMABLE; + case __GFP_MOVABLE: + return MIGRATE_MOVABLE; + default: + break; } + +unmovable: + if (gfp_flags & __GFP_SENSITIVE) + return MIGRATE_UNMOVABLE_SENSITIVE; + else + return MIGRATE_UNMOVABLE_NONSENSITIVE; } =20 static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9540b41894da6d67751506614e90bed142e127b4..b8215b921b71c0b1cbda2d26266= 4b1ee2a0cb750 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -46,7 +46,19 @@ #define PAGE_ALLOC_COSTLY_ORDER 3 =20 enum migratetype { - MIGRATE_UNMOVABLE, + /* + * All movable pages are sensitive for ASI. Unmovable pages might be + * either; the migratetype reflects whether they are mapped into the + * global-nonsensitive address space. + * + * TODO: what about HIGHATOMIC/RECLAIMABLE? + */ + MIGRATE_UNMOVABLE_SENSITIVE, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + MIGRATE_UNMOVABLE_NONSENSITIVE, +#else + MIGRATE_UNMOVABLE_NONSENSITIVE =3D MIGRATE_UNMOVABLE_SENSITIVE, +#endif MIGRATE_MOVABLE, MIGRATE_RECLAIMABLE, MIGRATE_PCPTYPES, /* the number of types on the pcp lists */ @@ -89,6 +101,11 @@ static inline bool is_migrate_movable(int mt) return is_migrate_cma(mt) || mt =3D=3D MIGRATE_MOVABLE; } =20 +static inline bool is_migrate_unmovable(int mt) +{ + return mt =3D=3D MIGRATE_UNMOVABLE_SENSITIVE || mt =3D=3D MIGRATE_UNMOVAB= LE_NONSENSITIVE; +} + /* * Check whether a migratetype can be merged with another migratetype. * diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 16cf9e17077e359b98a69dc4bca48f4575b9a28c..28092c51c7a49546f5b7961aaf1= 500c72de88337 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1127,7 +1127,7 @@ int mhp_init_memmap_on_memory(unsigned long pfn, unsi= gned long nr_pages, if (mhp_off_inaccessible) page_init_poison(pfn_to_page(pfn), sizeof(struct page) * nr_pages); =20 - move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_UNMOVABLE); + move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_UNMOVABLE_SENSI= TIVE); =20 for (i =3D 0; i < nr_pages; i++) { struct page *page =3D pfn_to_page(pfn + i); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index aa54ef95233a052b5e79b4994b2879a72ff9acfd..ae711025da15823e97cc8b63e02= 77fc41b5ca0f8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -418,8 +418,9 @@ void set_pfnblock_flags_mask(struct page *page, unsigne= d long flags, void set_pageblock_migratetype(struct page *page, int migratetype) { if (unlikely(page_group_by_mobility_disabled && - migratetype < MIGRATE_PCPTYPES)) - migratetype =3D MIGRATE_UNMOVABLE; + migratetype < MIGRATE_PCPTYPES && + migratetype !=3D MIGRATE_UNMOVABLE_NONSENSITIVE)) + migratetype =3D MIGRATE_UNMOVABLE_SENSITIVE; =20 set_pfnblock_flags_mask(page, (unsigned long)migratetype, page_to_pfn(page), MIGRATETYPE_MASK); @@ -1610,10 +1611,14 @@ struct page *__rmqueue_smallest(struct zone *zone, = unsigned int order, * * The other migratetypes do not have fallbacks. */ -static int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] =3D { - [MIGRATE_UNMOVABLE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE }, - [MIGRATE_MOVABLE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE }, - [MIGRATE_RECLAIMABLE] =3D { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE }, +static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] =3D { + [MIGRATE_UNMOVABLE_SENSITIVE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_MOVAB= LE }, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + /* TODO: Cannot fallback from nonsensitive */ + [MIGRATE_UNMOVABLE_NONSENSITIVE] =3D { -1 }, +#endif + [MIGRATE_MOVABLE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_UNMOV= ABLE_SENSITIVE }, + [MIGRATE_RECLAIMABLE] =3D { MIGRATE_UNMOVABLE_SENSITIVE, MIG= RATE_MOVABLE }, }; =20 #ifdef CONFIG_CMA @@ -1893,7 +1898,7 @@ static bool should_try_claim_block(unsigned int order= , int start_mt) * allocation size. Later movable allocations can always steal from this * block, which is less problematic. */ - if (start_mt =3D=3D MIGRATE_RECLAIMABLE || start_mt =3D=3D MIGRATE_UNMOVA= BLE) + if (start_mt =3D=3D MIGRATE_RECLAIMABLE || is_migrate_unmovable(start_mt)) return true; =20 if (page_group_by_mobility_disabled) @@ -1929,6 +1934,9 @@ int find_suitable_fallback(struct free_area *area, un= signed int order, *claim_block =3D false; for (i =3D 0; i < MIGRATE_PCPTYPES - 1 ; i++) { fallback_mt =3D fallbacks[migratetype][i]; + if (fallback_mt < 0) + return fallback_mt; + if (free_area_empty(area, fallback_mt)) continue; =20 diff --git a/mm/show_mem.c b/mm/show_mem.c index 43afb56abbd3e1f436452ef551dd8149545ccb13..59af6fb286354156cc2aa258fdd= 78c85797248a6 100644 --- a/mm/show_mem.c +++ b/mm/show_mem.c @@ -141,15 +141,16 @@ static bool show_mem_node_skip(unsigned int flags, in= t nid, nodemask_t *nodemask static void show_migration_types(unsigned char type) { static const char types[MIGRATE_TYPES] =3D { - [MIGRATE_UNMOVABLE] =3D 'U', - [MIGRATE_MOVABLE] =3D 'M', - [MIGRATE_RECLAIMABLE] =3D 'E', - [MIGRATE_HIGHATOMIC] =3D 'H', + [MIGRATE_UNMOVABLE_SENSITIVE] =3D 'S', + [MIGRATE_UNMOVABLE_NONSENSITIVE] =3D 'N', + [MIGRATE_MOVABLE] =3D 'M', + [MIGRATE_RECLAIMABLE] =3D 'E', + [MIGRATE_HIGHATOMIC] =3D 'H', #ifdef CONFIG_CMA - [MIGRATE_CMA] =3D 'C', + [MIGRATE_CMA] =3D 'C', #endif #ifdef CONFIG_MEMORY_ISOLATION - [MIGRATE_ISOLATE] =3D 'I', + [MIGRATE_ISOLATE] =3D 'I', #endif }; char tmp[MIGRATE_TYPES + 1]; --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E46761F3BA3 for ; Thu, 13 Mar 2025 18:12:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889524; cv=none; b=pvXIcqK8Z+ANly0EJSqttPhxwhD82SE7asMtV+8nrD1nRV9wbKor0iPpBpSwmiAfmcZkd1EwafiLBituHWNUg3FsxfKHxLZe95O4Fw0jWiwWCfo+VU7PpK1qZAAENKsLnmEyhZPLwY1y2cj3A7PX9HZ8DOR+wC4kUyLXRX3wCEY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889524; c=relaxed/simple; bh=iGdPf0L9eAmfpxDO21sCB/pch8E5xey9ZueJIm70CcA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=otAF5k8qrHS8jxfCNAakkgvKHGKQ7m14DFiW8YXfQ2U8GD2lRGsRSNEsPscZ5aw+a4JEzD0KKgavNRqkL6uB5AITIQmZYFZ3W47fDvDCTyDfuINV6rRYnARoEVG+UohnQDCyjT2dIQUxgVUk0SF7f6Ro4ON3YnPx+PVt8IN8o1E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WlZWKWYQ; arc=none smtp.client-ip=209.85.221.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WlZWKWYQ" Received: by mail-wr1-f74.google.com with SMTP id ffacd0b85a97d-3912fe32a30so633975f8f.1 for ; Thu, 13 Mar 2025 11:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889521; x=1742494321; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=WlZWKWYQcV+vEs53+NXnnJ6eYf92gOZ8Jqa/cSLR8yRrESRBRIZEAxzIsj26AuGFIs C3CfDNBk7r9AwqJbsBOgt43smNmg3A3Cjo8UHFS2P/RLcGELKgjE29MnG8rUtAnEnFmC HACqKiU9/oCEJsnLdRdDk5tVLRXLk/+92H1LQz77Xn2TxQ4mw1HWv0zpgey3y8Lt0plQ jrMyQj/qy/tGjLwNEvpcsxwbNI4rmQgHGb+ONFixdApxY5C4Q5M64oRObqiokfd7n/PO NIYi2fzxy55ygdkzEMNYXQxLhoGcSYuY9rW0zYpu5V6O0jaYIhSyPDPZy1zTSFMhmvZi phQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889521; x=1742494321; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MT67Tc4nbE6RvTfwk389B1U2q+t+QBE9ZWlf/JDHQjo=; b=B7WMJku0l5THkCngHpolmbsi1rL9Et01iug+Ma1djJXq+iaPe+b1AjwuUlCHuR/rUH mSOk8gE15+TsNrqjwrVPGb5E+fRspHEOuFuRfD+x2VJ0PkvW2C4FaZ9WxVQ3e1I8fzdT dL3fBS921IeLb1U7Nr1ePtTuy7tfABnp3sCNh96lEDbkSFT3b1wUDVeQw0Nhz8OH37jq mdhksyr1dP3JvuDViDQ1eVe4QrU1GD01b2t9K4Nhn2B1RgS278GIGUJipAhG/qsgRmxt fJaMRD7G3yA9eVl+GlfUAsm+xY3ECA+0muxI1W5xhpwC474DNJf733H1aG/E0hphzK3o wi1w== X-Gm-Message-State: AOJu0Yyv8jp4oFXu9xVRBPAXbKDW2LG0nfT196xYOLpTyau3qO1jo/lW 4Nu+EmjbjVlvTeTGpkZDiDsJ6tqxxxAqGlQhMT0z9dibPpYacB9cCxTOj7t3iXOMxiaKk+p8yuj 3f2nELunNrw== X-Google-Smtp-Source: AGHT+IGTDPbDngPkmEoWDX7m7yWVpzg47wDDczIbjQCcv6h8SnX4qWQyopmuiT4sdFB+hEFVGiIYdXMgmMnnIQ== X-Received: from wro9.prod.google.com ([2002:a05:6000:41c9:b0:391:2f9a:9a8b]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6000:1f83:b0:390:f9d0:5e3 with SMTP id ffacd0b85a97d-39263affd7amr11502380f8f.1.1741889521421; Thu, 13 Mar 2025 11:12:01 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:29 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-10-04972e046cea@google.com> Subject: [PATCH RFC 10/11] mm/page_alloc: Add support for nonsensitive allocations From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Creating nonsensitive pages from sensitive ones is pretty easy: Just call asi_map(). Since this doesn't require any allocation or synchronization (at least, the synchronization is implied by the zone lock) just do it immediately in the place where the pageblock's migratetype is updated. Implement a migratetype fallback mechanism to let this happen, restricted to requiring a whole pageblock. Now that it's possible to create nonsensitive pages, the allocator supports non-__GFP_SENSITIVE allocations, so remove the temporary hack setting this flag unconditionally in the entry point. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 5 ++++ mm/page_alloc.c | 67 +++++++++++++++++++++++++++++++++++-------= ---- 2 files changed, 57 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index cf8be544de8b108190b765e3eb337089866207a2..4208b60852ace25fc9716c276bd= 86bf920ab3340 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -2,7 +2,12 @@ #ifndef _ASM_X86_ASI_H #define _ASM_X86_ASI_H =20 +#include +#include +#include + #include +#include =20 #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ae711025da15823e97cc8b63e0277fc41b5ca0f8..0d8bbad8675c99282f308c4a412= 2d5d9c4b14dae 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -15,6 +15,7 @@ * (lots of bits borrowed from Ingo Molnar & Andrew Morton) */ =20 +#include #include #include #include @@ -422,6 +423,18 @@ void set_pageblock_migratetype(struct page *page, int = migratetype) migratetype !=3D MIGRATE_UNMOVABLE_NONSENSITIVE)) migratetype =3D MIGRATE_UNMOVABLE_SENSITIVE; =20 + /* + * TODO: doing this here makes this code pretty confusing, since in the + * asi_unmap() case it's up to the caller. Need to find a way to express + * this with more symmetry. + * + * TODO: Also need to ensure the pages have been zeroed since they were + * last in userspace. Need to figure out how to do that without too much + * cost. + */ + if (migratetype =3D=3D MIGRATE_UNMOVABLE_NONSENSITIVE) + asi_map(page, pageblock_nr_pages); + set_pfnblock_flags_mask(page, (unsigned long)migratetype, page_to_pfn(page), MIGRATETYPE_MASK); } @@ -1606,19 +1619,35 @@ struct page *__rmqueue_smallest(struct zone *zone, = unsigned int order, =20 =20 /* - * This array describes the order lists are fallen back to when - * the free lists for the desirable migrate type are depleted + * This array describes the order lists are fallen back to when the free l= ists + * for the desirable migrate type are depleted. When ASI is enabled, diffe= rent + * migratetypes have different numbers of fallbacks available, in that cas= e the + * arrays are terminated early by -1. * * The other migratetypes do not have fallbacks. + * + * Note there are no fallbacks from sensitive to nonsensitive migratetypes. */ -static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] =3D { - [MIGRATE_UNMOVABLE_SENSITIVE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_MOVAB= LE }, #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION - /* TODO: Cannot fallback from nonsensitive */ - [MIGRATE_UNMOVABLE_NONSENSITIVE] =3D { -1 }, +#define TERMINATE_FALLBACK -1 +#else +#define TERMINATE_FALLBACK #endif - [MIGRATE_MOVABLE] =3D { MIGRATE_RECLAIMABLE, MIGRATE_UNMOV= ABLE_SENSITIVE }, - [MIGRATE_RECLAIMABLE] =3D { MIGRATE_UNMOVABLE_SENSITIVE, MIG= RATE_MOVABLE }, +static const int fallbacks[MIGRATE_PCPTYPES][MIGRATE_PCPTYPES - 1] =3D { + [MIGRATE_UNMOVABLE_SENSITIVE] =3D { + MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + [MIGRATE_UNMOVABLE_NONSENSITIVE] =3D { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE + }, +#endif + [MIGRATE_MOVABLE] =3D { + MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE_SENSITIVE, TERMINATE_FALLBACK + }, + [MIGRATE_RECLAIMABLE] =3D { + MIGRATE_UNMOVABLE_SENSITIVE, MIGRATE_MOVABLE, TERMINATE_FALLBACK + }, }; =20 #ifdef CONFIG_CMA @@ -1931,6 +1960,21 @@ int find_suitable_fallback(struct free_area *area, u= nsigned int order, if (area->nr_free =3D=3D 0) return -1; =20 + /* + * It's not possible to change the sensitivity of individual pages, + * only an entire pageblock. Thus, we can only fallback to a migratetype + * of a different sensitivity when the entire block is free. + * + * This cross-sensitivity fallback occurs exactly when the start + * migratetype is MIGRATE_UNMOVABLE_NONSENSITIVE. This is because it's + * the only nonsensitive migratetype (so if it's the start type, a + * fallback will always differ in sensitivity) and it doesn't appear in + * the 'fallbacks' arrays (i.e. we never fall back to it, so if it isn't + * the start type, the fallback will never differ in sensitivity). + */ + if (migratetype =3D=3D MIGRATE_UNMOVABLE_NONSENSITIVE && order < pagebloc= k_order) + return -1; + *claim_block =3D false; for (i =3D 0; i < MIGRATE_PCPTYPES - 1 ; i++) { fallback_mt =3D fallbacks[migratetype][i]; @@ -4694,13 +4738,6 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, = unsigned int order, gfp_t alloc_gfp; /* The gfp_t that was actually used for allocation */ struct alloc_context ac =3D { }; =20 - /* - * Temporary hack: Allocation of nonsensitive pages is not possible yet, - * allocate everything sensitive. The restricted address space is never - * actually entered yet so this is fine. - */ - gfp |=3D __GFP_SENSITIVE; - /* * There are several places where we assume that the order value is sane * so bail out early if the request is out of bound. --=20 2.49.0.rc1.451.g8f38331e32-goog From nobody Wed Dec 17 22:44:34 2025 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D49C01EF0B0 for ; Thu, 13 Mar 2025 18:12:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889526; cv=none; b=Bgzw2BXzKp+MxjJfag0oTPDWdGqf4SsnrgNBu6XEUPiIDsQycMPzYbKfk3PSJg32UcpuPKBGBmYJFyE0yZ9geQpHeGyqWeXp7hsqiiguslLle5S6m8SuclM1GlieruJioLQkDvAKsvhl75fcDAIYnMtO7X8stUgisbhrau4tCng= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1741889526; c=relaxed/simple; bh=v7imAdqjD4pAPp4qMYhNCQjF4lz5o+NNzBuaHhgAwAw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=I83G05zATXeEgCZ4noHrNWMwHFW5FSKA6Leg6sAA64xc+dSagI4cvEM7XpaY1sD7BP+7KQoC1jjxlE1mmDuikbk9bCciEyI0xSBK8KZFmXC27qe9jWZQ9hQeofoP8w6VdnxgKYAaSTScNe27UwHWZxATMmNlxQpXcePsMpWC5D0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=I/VCXTBP; arc=none smtp.client-ip=209.85.128.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jackmanb.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="I/VCXTBP" Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43ceeaf1524so6822135e9.1 for ; Thu, 13 Mar 2025 11:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1741889523; x=1742494323; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=I/VCXTBPbN6WKMaoiCCAHNuAxqwWMGlbGhRcabqSEk4UV6JBHzOC7IB9w/3HG1n3lF 5aDn0gYFVC86MU7AU7uOtxXmyMjtyJ784ZMHbYBj2iE7KF+LOh46PRwwIGZk3ceBkyr7 f6Fqb6LJZMfTMAutKBMezq03I3t9gMXaRs1zcdYD/AG34klSN4vQbajuHq8INcW2yo2C MC6yK5EL6cblz9Hv4aMtr7UO/l+sfhdp+PjuppaulDCAXyC0PndSaJyjen79fXyVVa4t AsCAv3AOhFdMIuIEYNJOqplCIYEVX/Q1n2kqgZKUrY20gCIPbTAr4FO6oiDTGfr1l+AX 1eTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1741889523; x=1742494323; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jZCBSIxQMZuxgPH5nxXznTpvfPN1Wm6wtktBoLoOnVE=; b=wMmH4VU03QI4KWromxmjryQBsHuQ2EhW7IBS3MeNeoWL3ZFcjWeoyc9I8pDAfR1MBN BT7edNGqN+I4/EDK1Br3x97Hko6bor7udeCgoA7ocHQVWpzzWMeX2ugwNyCRERrUtgOV d1FpNukQtg4YK5WmndoEgJO9elDk4Vw6Xb6i39VQnAaXlruQF+cZsFnwpODpXcNv5iLk pV01WCfHv7RaW9Ns4bwu2uvKcxUMJdYkQkoW6z/8iC86obH6FHdQQ0nUzzTqafp8dmPL joNY5UytksEReR9LDQ6nE9Yp5l+7v2ZuzVNIY8jDHm72oNDzloZ8vIgXBXmOAfJrYBv/ kMDA== X-Gm-Message-State: AOJu0YywIlmYyAebW1M5pNXn6nxrAA7x6nQ+RpwqFmod/RGpdrsiJqet vuz48HQ9u2yU74OHwiNqqhadDoBZcqi5OgZkvPIyg8IsQ8B9MBFS1bUX1IaM+DHTNbquJnam240 7bu+ZX6jsqg== X-Google-Smtp-Source: AGHT+IFbs6k3vnl5QQ+C0J0hTTMoncea9PLFMkDkNtbfor/v1Jhe1WlJqAJFv/1WYWGnmKJfkaAhbBEhyKJgWQ== X-Received: from wmrn39.prod.google.com ([2002:a05:600c:5027:b0:43c:f03d:18aa]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3143:b0:439:4c1e:d810 with SMTP id 5b1f17b1804b1-43d1808593dmr31746155e9.9.1741889523325; Thu, 13 Mar 2025 11:12:03 -0700 (PDT) Date: Thu, 13 Mar 2025 18:11:30 +0000 In-Reply-To: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20250313-asi-page-alloc-v1-0-04972e046cea@google.com> X-Mailer: b4 0.14.2 Message-ID: <20250313-asi-page-alloc-v1-11-04972e046cea@google.com> Subject: [PATCH RFC 11/11] mm/page_alloc: Add support for ASI-unmapping pages From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, Andrew Morton , David Rientjes , Vlastimil Babka , David Hildenbrand Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mike Rapoport , Junaid Shahid , Reiji Watanabe , Patrick Bellasi , Brendan Jackman , Yosry Ahmed Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable While calling asi_map() is pretty easy, to unmap pages we need to ensure a TLB shootdown is complete before we allow them to be allocated. Therefore, treat this as a special case of buddy allocation. Allocate an entire block, release the zone lock and enable interrupts, then do the unmap and TLB shootdown. Once that's complete, return any unwanted pages within the block to the freelists. Signed-off-by: Brendan Jackman --- mm/internal.h | 5 ++++ mm/page_alloc.c | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++-= ---- 2 files changed, 86 insertions(+), 7 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index b82ab345fb994b7c4971e550556e24bb68f315f6..7904be86fa2c7fded62c100d84f= e572c15407ccf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1189,6 +1189,11 @@ unsigned int reclaim_clean_pages_from_list(struct zo= ne *zone, #endif #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAI= M set */ +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +#define ALLOC_ASI_UNMAP 0x1000 /* allow asi_unmap(), requiring TLB = shootdown. */ +#else +#define ALLOC_ASI_UNMAP 0x0 +#endif =20 /* Flags that allow allocations below the min watermark. */ #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC= |ALLOC_OOM) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0d8bbad8675c99282f308c4a4122d5d9c4b14dae..9ac883d7a71387d291bc04bad67= 5e2545dd7ba0f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1627,6 +1627,7 @@ struct page *__rmqueue_smallest(struct zone *zone, un= signed int order, * The other migratetypes do not have fallbacks. * * Note there are no fallbacks from sensitive to nonsensitive migratetypes. + * That's instead handled as a totally special case in __rmqueue_asi_unmap= (). */ #ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION #define TERMINATE_FALLBACK -1 @@ -2790,7 +2791,77 @@ static inline void zone_statistics(struct zone *pref= erred_zone, struct zone *z, #endif } =20 -static __always_inline +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +/* + * Allocate a page by converting some memory to sensitive, by doing an ASI + * unmap. This can't be done via __rmqueue_fallback because that unmap req= uires + * a TLB flush which can only be done with interrupts on. + */ +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int request_o= rder, + unsigned int alloc_flags, int migratetype) +{ + struct page *page; + int alloc_order; + int i; + + lockdep_assert_irqs_enabled(); + + if (!(alloc_flags & ALLOC_ASI_UNMAP)) + return NULL; + + VM_WARN_ON_ONCE(migratetype =3D=3D MIGRATE_UNMOVABLE_NONSENSITIVE); + + /* + * Need to unmap a whole pageblock (otherwise it might require + * allocating pagetables). Can't do that with the zone lock held, but we + * also can't flip the block's migratetype until the flush is complete, + * otherwise any concurrent sensitive allocations could momentarily leak + * data into the restricted address space. As a simple workaround, + * "allocate" at least the whole block, unmap it (with IRQs enabled), + * then free any remainder of the block again. + * + * An alternative to this could be to synchronize an intermediate state + * on the pageblock (since since this code can't be called in an IRQ, + * this shouldn't be too bad - it's likely OK to just busy-wait until + * any conurrent TLB flush completes.). + */ + + alloc_order =3D max(request_order, pageblock_order); + spin_lock_irq(&zone->lock); + page =3D __rmqueue_smallest(zone, alloc_order, MIGRATE_UNMOVABLE_NONSENSI= TIVE); + spin_unlock_irq(&zone->lock); + if (!page) + return NULL; + + asi_unmap(page, 1 << alloc_order); + + change_pageblock_range(page, alloc_order, migratetype); + + if (request_order >=3D alloc_order) + return page; + + spin_lock_irq(&zone->lock); + for (i =3D request_order; i < alloc_order; i++) { + struct page *page_to_free =3D page + (1 << i); + + __free_one_page(page_to_free, page_to_pfn(page_to_free), zone, i, + migratetype, FPI_SKIP_REPORT_NOTIFY); + } + spin_unlock_irq(&zone->lock); + + return page; +} +#else +static inline +struct page *__rmqueue_asi_unmap(struct zone *zone, unsigned int order, + unsigned int alloc_flags, int migratetype) +{ + return NULL; +} +#endif + +static noinline struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, unsigned int order, unsigned int alloc_flags, int migratetype) @@ -2814,13 +2885,14 @@ struct page *rmqueue_buddy(struct zone *preferred_z= one, struct zone *zone, */ if (!page && (alloc_flags & (ALLOC_OOM|ALLOC_NON_BLOCK))) page =3D __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); - - if (!page) { - spin_unlock_irqrestore(&zone->lock, flags); - return NULL; - } } spin_unlock_irqrestore(&zone->lock, flags); + + if (!page) + page =3D __rmqueue_asi_unmap(zone, order, alloc_flags, migratetype); + + if (!page) + return NULL; } while (check_new_pages(page, order)); =20 __count_zid_vm_events(PGALLOC, page_zonenum(page), 1 << order); @@ -3356,6 +3428,8 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp= _t gfp_mask, if (gfp_migratetype(gfp_mask) =3D=3D MIGRATE_MOVABLE) alloc_flags |=3D ALLOC_CMA; #endif + if (gfp_mask & __GFP_SENSITIVE && gfpflags_allow_blocking(gfp_mask)) + alloc_flags |=3D ALLOC_ASI_UNMAP; return alloc_flags; } =20 @@ -4382,7 +4456,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int o= rder, reserve_flags =3D __gfp_pfmemalloc_flags(gfp_mask); if (reserve_flags) alloc_flags =3D gfp_to_alloc_flags_cma(gfp_mask, reserve_flags) | - (alloc_flags & ALLOC_KSWAPD); + (alloc_flags & (ALLOC_KSWAPD | ALLOC_ASI_UNMAP)); =20 /* * Reset the nodemask and zonelist iterators if memory policies can be --=20 2.49.0.rc1.451.g8f38331e32-goog