From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4ED703A453A for ; Thu, 27 Aug 2026 23:31:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873480; cv=none; b=Us3bc6lce15+ut5dBNxV+zN3H5nYiyshHi3NgN1Vqi/b/elqlDxC+s/vlI/nSN3V3u9ecuu9WeXoBKwLl9jgrwuw3P0C8XrVLp6VMD2ZLdZJG9f/PH/P6jkandADjMIAcd/AqrPor44BItSRxLq9wiey9uKBxzrpiT9XC09n21M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873480; c=relaxed/simple; bh=3I5dS/SQ6pCyb3i0Ix/G3rblivqux+gR9oaHKYEaKnY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sqIsuFyJggabtS0+pEDFGgdIVfqgkkvmrdgH/arU7DUHe8zjDY2tCDHKCvn5CoKCeOBQ352R22x/ZXA24DNr/U6xiiJcREY7qOqTDG9+neSxSSTy5BL3qmZZ+fxoZ5RAdBB2RL45VmEvtAYgZgBnOsqHoYd3Yqeg454oojhi7lc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=F0WE0Vee; arc=none smtp.client-ip=209.85.214.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="F0WE0Vee" Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-2d52734fc41so4602075ad.1 for ; Thu, 27 Aug 2026 16:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873479; x=1788478279; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=brDKxupp32vlfYsuON/JY3iAK7z2zOrfUeaeP3zp25g=; b=F0WE0VeeAQdzh0CIlwv1meqQbhzCsbR96VDhR0EaTiqYsmWRamm8ir76N2R0XrYMwZ l8xcCIZJ2d/B9i7wQdrdQiM/bB2ZQEutThINttOILceOC1swN03B9ctKDgsYGG1JC8Tv FzzUJSQbp5zeJkbiG6TEnVPX9R7VbgzcZOU2HFZfA1maAZ44o2nEERunsZBYIaj22hlx zGUAMD6h4Pg6DgkubIrXCKcLC4aKaxoU0NuiL6sXg+mgXgy+3hN0kayGgHmmF49W8Wil nLiCl9VLgyTyq73qDqdmQDzGyP15I9fupqSw2+InLeOsxXDROEBFYymCS8JBf+kEW5KE 2eJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873479; x=1788478279; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=brDKxupp32vlfYsuON/JY3iAK7z2zOrfUeaeP3zp25g=; b=OrrgGwY+CEQMGcTNznRGi3tBWiPLQdFhyecfIFDMZ5asXbBt/v8NViUzi7UGEMu2aI rtoUNwDjLp1Y8peEzfyedJ9cf1NEeFONjP5j9h9+YVHq/pBnuSjnmXWkxTvOvNnWhg+1 9axfcV7NWpqUW+i5NUiun0BrMF3IMnSHdMGhqdgY1jXf0MWEQBGYJIfenShbBlKBQfsu MmvJXvhsJsARiq0SZVb075Q2X7gN9/KK4kf9SOFDkV5Hk8I5P4BffyMJ2mwJOf6eX4Tp 2Mj7sA3iQXn4PI/JqzNfbhWeiibVTniT9J+ukJgB/yHeKBlSTcYm0yVVWA4Qx+kMOZ8r yFIQ== X-Gm-Message-State: AFuF++kTbvZyAS+zZQEjk/MwCeQbs+ZhdUqLYOSCWD3clRp9GwGJi/hM iQjNAtJvHDtBX56m9jyJGaNgEV++aRnUeKU5JyLV40Ei+Tv91avcsCSp6DfgKkafaRye5AnllcP 3dy6RQA2T24PwRw== X-Received: from dybeb10.prod.google.com ([2002:a05:7300:a2ca:b0:328:4d52:8cd0]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:458b:b0:2d6:f3f0:3e7a with SMTP id d9443c01a7336-2d74dc2ade3mr36333205ad.3.1787873478170; Thu, 27 Aug 2026 16:31:18 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:39 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-2-stevensd@google.com> Subject: [RFC 01/10] Add !MEMCG memcg_list_lru_alloc implementation From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a stub implementation of memcg_list_lru_alloc() when CONFIG_MEMCG isn't enabled. Signed-off-by: David Stevens --- include/linux/list_lru.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index a450fffe1550..40eaeaa3424c 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -79,10 +79,10 @@ static inline int list_lru_init_memcg_key(struct list_l= ru *lru, struct shrinker return list_lru_init_memcg(lru, shrinker); } =20 +#ifdef CONFIG_MEMCG int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); =20 -#ifdef CONFIG_MEMCG /** * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable fol= io * @folio: the newly allocated & charged folio @@ -106,6 +106,11 @@ static inline int folio_memcg_list_lru_alloc(struct fo= lio *folio, { return 0; } + +static inline int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct li= st_lru *lru, gfp_t gfp) +{ + return 0; +} #endif =20 void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 234023DCDAC for ; Thu, 27 Aug 2026 23:31:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873482; cv=none; b=EloZSpzVDCgEArEuA31zy0imFRlrwTJQmsvo2cuZu+JJIv2a3L1UabxsVQYT1kPCcBeUecKBcP8ILPz/mQ1VMkCqUem+/koHxfei40+yDT7gnURhkvW3YntyP6I2MGSTtsuQOmOpEC9Uko+/n+WPAdyTpWMSk2myHKKiZ9zeIEo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873482; c=relaxed/simple; bh=1JDl/PiJOAZxnlSKEC5PiAyofjmTisyMsK4a9y30L6I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hGUeVqUam3QV7qJfB+J77+vmcSuO+8nQAhOc3akHCCfs7Kgvxq0yzimcChOoUbYmqoigvQjkMsofIMv5GhHZo27SEAwoQZN2zTYnhO1JvOpH4+4w4fQqLoITil13Z8s6GBmLA4mG1tiN0b2Hv6ecuxHZagz6hY5XhkQqqgcxtUk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=RNrfTuYW; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="RNrfTuYW" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-38e25e4b41cso481622a91.0 for ; Thu, 27 Aug 2026 16:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873480; x=1788478280; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=4pKguOeL4XpEhj7eOn7cPoRPiZ6Kih53q6MwTo24ARI=; b=RNrfTuYWpdkpjojbbjv4eFKXN3+IMRf/AVaHOSd6i73FEKmgyie9pXgonFC9OFH2zg v1TBEkWC+EaSirH+nYD4AVy3o9JEUqyS0sMwm0ovXNDU5dEPXIenanvU8fqaJm6FngzW jgv8FrvB8eu3ad2hqvWnSMnUvJVTppn1rxZwDlX68mPJ8sfHlDMXB3PJos4AMkUULYT1 RMFR0bUL/aq/dqr1rVn8SVHfz9XYSaPN7eMmgNiMhPXr6sU2rQhUIWM1P1fFW39MCAZX FVRiNf46KwWbFI5OJZJuAEBPUt2COhpAnp9iTZ+Rjgljcgvbc0b1nL1izHRyN5Meqb3e 7qDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873480; x=1788478280; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=4pKguOeL4XpEhj7eOn7cPoRPiZ6Kih53q6MwTo24ARI=; b=TweFvcxt2LGgMmCv5j6eokvclDKyOfLgSbth6J12vmUIaVXxq5zSQ/Z6X/4X8JQM6w +d4HcsRnylA4JxWfLFSSYGpbKk0ygcDtdNAesN5hrUrh72py36nbt28twpz95nwv5xdf k1xFru1iaYVpJpKmBKOdGCsg1C4qhILBfbu6X2Nshy0h8uDA2vifTIj/QZCsD1Hzg8aL Rde8r0CLLBHp17dLULGq1QvtHbj2DM0ihhahMQsQTmfrV/4wDgatpJFnN9dbbpNbfJRY i6i6c7iErLCv2vVgJUhO1rlWgcHnTkuqYbUBEmrzR0Xc+3jptX+hpZule86z9qXvW7TY TKHg== X-Gm-Message-State: AFuF++k4IFIv0ED/TK72ZXNNkybfjnqiGBw0gvNSfXXbOPTi2DMy0RnX 9gi1ZlEk6mhzl/e2Jkq/ioZq357UWGn8+vgE91IHIzLtjM0KdCQGvj4Wj+62fEyGflsN/0Sa3s/ AUuxE7BuC4bElbw== X-Received: from dlbtp4.prod.google.com ([2002:a05:7022:3b84:b0:141:4abc:f12d]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1a8e:b0:396:4c63:7193 with SMTP id 98e67ed59e1d1-396d0fe45damr5091530a91.11.1787873480154; Thu, 27 Aug 2026 16:31:20 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:40 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-3-stevensd@google.com> Subject: [RFC 02/10] mm/vmalloc: Skip vmallocinfo NUMA stats for VM_SPARSE From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Bail from show_numa_info() in vmalloc_info_show() for areas with VM_SPARSE set. This lets clients that allocate VM_SPARSE areas directly use vm_struct's nr_pages and pages fields. Signed-off-by: David Stevens --- mm/vmalloc.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index f4fa227a8d7f..d39897300f29 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -5294,6 +5294,9 @@ static void show_numa_info(struct seq_file *m, struct= vm_struct *v, if (!counters) return; =20 + if (v->flags & VM_SPARSE) + return; + memset(counters, 0, nr_node_ids * sizeof(unsigned int)); =20 for (nr =3D 0; nr < v->nr_pages; nr +=3D step) --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A8AD3932C3 for ; Thu, 27 Aug 2026 23:31:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.197 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873484; cv=none; b=rbUpKB3xoRR9MiV1fwxkxCJsFKdWMIpKhdNpp+CJ5VI5Vq21wG/bWh+YSQughk543VuNcdFlhJ+PDSZ/02l1U6tRrhd0F68cCmUJnKsCbFhbJnXt0FFX13amluJ6Hpsk6e5EMsc18r1rRS5iwVBbLS10eFaHMAVBwGaP2+WxZd8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873484; c=relaxed/simple; bh=dfPDgNVh62+kHVsnAbD/7Pdvo8mCB2lnJPdMqfNI/Gc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Oh6mmZz8Br3SLmvnYdVV+UvWuO3p42lAlqh8/f9dozwJHy/AffdKHsR+kdcI1qJsjJjFSb6cz2CVJH9ZOCdTjc6stkS0eDIznTR0aN2klDw0EMbMNcZwEMH+uirfMeULo3YyfguId0XLdl+8TXXxPLoAic6RX8MXnP7OU8GT4K4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ot44G7at; arc=none smtp.client-ip=209.85.215.197 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ot44G7at" Received: by mail-pg1-f197.google.com with SMTP id 41be03b00d2f7-cc1cdfb337eso660747a12.0 for ; Thu, 27 Aug 2026 16:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873482; x=1788478282; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XRF3HmEVCSbkNryj4HHFEle4K2g8xeVsDlkdo6/RfiQ=; b=Ot44G7atyvQiCmg1NPcjdfAjvjJk19lKqljvJjLPxH7VIjp1LAkK0R0AbufPd3ZhWM KqZS0HxD7rstsZrRZ0GYfNsZV+E+FhpnLp2OBwwGfjO+4myg53cKaQe2GFZxYoCPnxji onWpX7G956XlbE5nIzJo7z3VGNkMfNy+nTZAzxl9UpLvNvxy4F0sEKdU0FYN0OFaLBX7 qyEhoRBboQ1jNCx5J8WEi68Wlte+qkP3ls4UzxMncWbzeW5Fmw8+/uv3/+WPogplbP/g iJpqb5IYuOWB7ZUHoilvdr6o5pNxPQcJc2iYj7CK3AUCIZsB9FFiI6E9scYAE+IyU4hQ ChRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873482; x=1788478282; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XRF3HmEVCSbkNryj4HHFEle4K2g8xeVsDlkdo6/RfiQ=; b=BJ5jtiAhvrwjDqSTQLByhvB0EGYvhwsUdIyioCywrCaqVNUfK5nwf8C5jGPeydyfZS +scQnzt2gp1z19s12kRUeP4kfUUU/7QMMIkpCXBD7zSBeHSkSu/yJ3JIWbNXzDc40cRR //jnutwf7ET3J/zZHWkPrTJcFs0tKf4wWBp15sHm8Ki5W93OuWcfPTffG1IC10Ih/jay eCXUIZcDKLxTqQBAP82tLw3Z6/eIitY6QiqwNfAm0bsI4LDB+4GSID6LneAKEYLDt2B9 7IIKiG492S3AztxhUWlVOdvfRTjrGdP51N0D4Uvg8QfTanR1wAF7anjpM01XroqFlUC5 HWuw== X-Gm-Message-State: AFuF++nRCFzYUaVSKbIYECU1y/CBJqQuno58dZCtGJjE3CgN+6nXPE8D G8YgJES18FThZH7fpVf3KkK3GbYLLetJVaUat0tqvRtj4YhWKTt07f81AHkv+r/ZYdk8FCfEwe1 akxQqdkJY141ETQ== X-Received: from dyblf13.prod.google.com ([2002:a05:7301:a0d:b0:315:bf25:ce14]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:94c9:b0:3d1:be60:2816 with SMTP id adf61e73a8af0-3d268cf776bmr3404563637.10.1787873482070; Thu, 27 Aug 2026 16:31:22 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:41 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-4-stevensd@google.com> Subject: [RFC 03/10] fork: refactor vmap stack alloc/free into helpers From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Prepare for alternate implementations of some functions for reclaimable stacks by creating helpers for allocating and freeing vmap stacks. Also reorder some functions to consolidate the functions that will have alternate implementations. Signed-off-by: David Stevens --- kernel/fork.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index f0e2e131a9a5..10bd76f6dd20 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -207,11 +207,6 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stack= s[NR_CACHED_STACKS]); */ #define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN) =20 -struct vm_stack { - struct rcu_head rcu; - struct vm_struct *stack_vm_area; -}; - static struct vm_struct *alloc_thread_stack_node_from_cache(struct task_st= ruct *tsk, int node) { struct vm_struct *vm_area; @@ -272,6 +267,26 @@ static bool try_release_thread_stack_to_cache(struct v= m_struct *vm_area) return false; } =20 +static void free_vmap_stack(struct vm_struct *vm_area) +{ + vfree(vm_area->addr); +} + +static struct vm_struct *alloc_vmap_stack(int node) +{ + void *stack =3D __vmalloc_node(THREAD_SIZE, THREAD_ALIGN, + GFP_VMAP_STACK, + node, __builtin_return_address(0)); + if (!stack) + return NULL; + return find_vm_area(stack); +} + +struct vm_stack { + struct rcu_head rcu; + struct vm_struct *stack_vm_area; +}; + static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack =3D container_of(rh, struct vm_stack, rcu); @@ -280,7 +295,7 @@ static void thread_stack_free_rcu(struct rcu_head *rh) if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; =20 - vfree(vm_area->addr); + free_vmap_stack(vm_area); } =20 static void thread_stack_delayed_free(struct task_struct *tsk) @@ -302,7 +317,7 @@ static int free_vm_stack_cache(unsigned int cpu) if (!vm_area) continue; =20 - vfree(vm_area->addr); + free_vmap_stack(vm_area); cached_vm_stack_areas[i] =3D NULL; } =20 @@ -338,7 +353,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) vm_area =3D alloc_thread_stack_node_from_cache(tsk, node); if (vm_area) { if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); + free_vmap_stack(vm_area); return -ENOMEM; } =20 @@ -356,15 +371,13 @@ static int alloc_thread_stack_node(struct task_struct= *tsk, int node) return 0; } =20 - stack =3D __vmalloc_node(THREAD_SIZE, THREAD_ALIGN, - GFP_VMAP_STACK, - node, __builtin_return_address(0)); - if (!stack) + vm_area =3D alloc_vmap_stack(node); + if (!vm_area) return -ENOMEM; + stack =3D vm_area->addr; =20 - vm_area =3D find_vm_area(stack); if (memcg_charge_kernel_stack(vm_area)) { - vfree(stack); + free_vmap_stack(vm_area); return -ENOMEM; } /* --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 096893E16B9 for ; Thu, 27 Aug 2026 23:31:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873486; cv=none; b=nAYIN5GAm0pyS8L72nrm4q+gOhHgumAVFmPIjLU059mTRq1WhRQUQx0y+QZNkALcmo6wPdE6ZabwTZ0TfVJRHI5utIo4HjSW7BWrq8zAL4pV1nUZkFNVuO6QgavaN/RDz3jHb52psLWnoxsdoavtlkw6fse/rSju2Kkq7ERst/o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873486; c=relaxed/simple; bh=K+D5Y6brOLXG1aN+myxhErHXOB0qjYSBze5FnYnXukE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=G7c4WDRvYOh1jKh8BhkqeAAf4b62WbbcFluQFFWmy8mOFmCycNMoBLDzCbxpBWg9AIemA252Gw0kfxJjeKM8ku33tzEEh8ROpTTT8tgJsW8m3kiSqcH9whZ85yIPcHAeU5mQxNvVrJZp8eVmsjOhiCNtspFHxsH5XOWyWy8RV88= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jsHHRjku; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jsHHRjku" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-38ec0f510a9so1115637a91.2 for ; Thu, 27 Aug 2026 16:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873484; x=1788478284; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=LOrBCjVl+Gsil1QM8OKlKDVjeh9PhasgM5tVDAHOOrA=; b=jsHHRjkujr+z7AdG2T8A0WmfXHOj/0izZM6MEoJd9JfiDzNg5R1CJ4xvcAks0fI8ju EC7MaTGu8bGIo4biLfzJP6vWZOUOrrSRKSUf8c1ddtOzl2RFA022HjIrHIeY4KjN6gZQ xLVg356CwxuLv7tUvqJpjiAUig4zcx1BAQffJQaKEQ73Z6uxppIGXMKVRKatq6wN85RV DPrUtiWM4VdfG4GwACRNxRgKzDzkI62r2JLnyWyNn7gPA7LRUi1ChzUXhQu4rZ7LujW1 jJFsXtH7tdccr3kSrKUJRoSUBJYfGdBePSmn1xQrvlKxQhVyEB+/95T+sBrZ04P6zImY uqdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873484; x=1788478284; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LOrBCjVl+Gsil1QM8OKlKDVjeh9PhasgM5tVDAHOOrA=; b=ISTa5WLApMY4NTz7MRrlHGckMngyd544XNE1TP2QRMaSDb36AVZRNAmJIMQLurgf08 7hm36+6RXLYGgtL5JgdUPmuWeoqGoZxyuOnxffS2O1iXS9J0e/c+9Af6Fc8Tav4gTRHI RHfoGn1s8Hhz6tCpMoAhPUVuA5JXpCQKO10RRcUQAB+ZBiawxi3wyFSaF+froR03cAup S3Ak9Ery5erGxLd7tG+i4N+GTV/4kbbevjTWbqDeKjx36aG4p9PUhPXb0Vp8HVOYQRab /f32AXHzgvozEDnNU3ASmgR1D7j+K8jZQC8/JS2tgu5JgXD5ikQKz2Nc80I/m3mvgIol 2Zww== X-Gm-Message-State: AFuF++nfqmWNUKwZuRX0YTqXDtjPR4DPzHkaD5ohWMpB2SVd2syMQv0/ QGUAdI717XZYOnKPBoauUfhq9lN0tkULT9FJxRX2H9e/QkOXl15sK7P1VQIWLSjbJu/V4BXOha/ IHyEij56nec5jYA== X-Received: from dlbqi2.prod.google.com ([2002:a05:7022:ebc2:b0:13f:bdf3:f58]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:4cc2:b0:396:d27b:89b9 with SMTP id 98e67ed59e1d1-396d27bb20emr3178120a91.10.1787873484126; Thu, 27 Aug 2026 16:31:24 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:42 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-5-stevensd@google.com> Subject: [RFC 04/10] mm: vmalloc: support creating aligned vm areas From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Expose a get_vm_area_node() function that supports passing an alignment when allocating a vm area. Signed-off-by: David Stevens --- include/linux/vmalloc.h | 2 ++ mm/vmalloc.c | 24 +++++++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index d87dc7f77f4e..5ed9c5b25026 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -246,6 +246,8 @@ static inline size_t get_vm_area_size(const struct vm_s= truct *area) extern struct vm_struct *get_vm_area(unsigned long size, unsigned long fla= gs); extern struct vm_struct *get_vm_area_caller(unsigned long size, unsigned long flags, const void *caller); +extern struct vm_struct *get_vm_area_node(unsigned long size, unsigned lon= g align, + unsigned long flags, int node); extern struct vm_struct *__get_vm_area_caller(unsigned long size, unsigned long flags, unsigned long start, unsigned long end, diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d39897300f29..059cd0748514 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3280,7 +3280,7 @@ struct vm_struct *__get_vm_area_caller(unsigned long = size, unsigned long flags, * @flags: %VM_IOREMAP for I/O mappings or VM_ALLOC * * Search an area of @size in the kernel virtual mapping area, - * and reserved it for out purposes. Returns the area descriptor + * and reserved it for our purposes. Returns the area descriptor * on success or %NULL on failure. * * Return: the area descriptor on success or %NULL on failure. @@ -3301,6 +3301,28 @@ struct vm_struct *get_vm_area_caller(unsigned long s= ize, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } =20 +/** + * get_vm_area_node - reserve a contiguous kernel virtual area + * @size: size of the area + * @align: alignment of the area + * @flags: %VM_IOREMAP for I/O mappings or VM_ALLOC + * @node: node from which to allocate data structures + * + * Search an area of @size in the kernel virtual mapping area, + * and reserve it for our purposes. Returns the area descriptor + * on success or %NULL on failure. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align, + unsigned long flags, int node) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + node, GFP_KERNEL, + __builtin_return_address(0)); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDA163DCD9B for ; Thu, 27 Aug 2026 23:31:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.199 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873488; cv=none; b=dYrD23iJalIBXRS33akKyv71X4BqNEHE7T2xAkEgme47D/2KCPck589O41gX9FdDmdKxgJ7tKK2eCqZ/I2ATUap9LE1jn5Dls5ZND50whpRZzkjWwo3HSHe4i38kCb3W8WQhWXqOMzSOtmq3zaEbfKfEDbAl9NR5YeaPTQELTYU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873488; c=relaxed/simple; bh=OL5kyzzH3pevyaGpCfzUthvi5UyUYi/GZsUUb63MbgA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qWutvfyBIBNMYFZf5xxW/zE2CzyiqrxtcOzPdY6ZaIddv9UFK6z862+7IJQcgyI9uLOTofOhJ1S6d4/rF2y/b21aSuXDvp2akOvtdFmdSwNqR4T61bwd4zZnc31yXp+3hwRPn3qVSt/G6AgXJITGIK06d9QnnUtENp5j+DMRmho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EnnqF4V/; arc=none smtp.client-ip=209.85.214.199 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EnnqF4V/" Received: by mail-pl1-f199.google.com with SMTP id d9443c01a7336-2ccb687f82eso3260405ad.3 for ; Thu, 27 Aug 2026 16:31:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873486; x=1788478286; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=/lOzSmEZ51hzHprpWRTTa0ItgwWLWckqKQo+nDAEYGw=; b=EnnqF4V/hmj76dHndqgFoNkxdLWMWu2xhJC+n1smXB2RMIUla8CfUp+jl59LRjmCm4 3CUJ2nQi+zv4eRxpJ0qoyiObwFRb9KLA+QTYtvxdJVvT0NANAyP6/3YouiUTmk/ZU+QI OvqFvYX879aqrmkafCTOuACsXXmec7qshPzdi3Ppo2Y5GwrQE7Mp17gKh2YdrEUhf+Ct scsb1RrPw1jxfpu6fWaZ4KtAj9pWNhpkC4xVrmQSl0hoytRF/qsWNnuGg7i4HCLcv/+W sZjSxk1BCHmgfwu77n/GFhVDI5pupAU14RaeNBbvM4DwmH+wV+yIVtcUul5yaaUMhLki iowQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873486; x=1788478286; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/lOzSmEZ51hzHprpWRTTa0ItgwWLWckqKQo+nDAEYGw=; b=TgoduZh3lyLsfNXvbRC+fVq92g5WIzbOP28xijwzUcZ1SNx0BdpRwkh2aeR6GPnN+C MR1Y8Lyd0ThWG+NqHZeNaHtNT0b9kpEAQD/owfkHLvBeLeFhHrKEeE8oer6PRL5gDUkF yoVTT0BjpmSEhIZ5nq0K8a4BYI4/ARoaczx9Si3UUPIOdISV/1dYeGWbyj4j+wtf4B6o lwLZKo2d5/0vkRoAYLhEbpi2IQ5/2pkK2MvFOn3hkt68ba/Z9jZys0iTv6fFxcOiAa82 gd5sDqGGGL+GrpfrR4PV6ySFuoOH15PlymKMKsLORIC4dwF2afrmZLWaNDD8hCbeV58o MSwg== X-Gm-Message-State: AFuF++lqQZ8JD3EbiSjz3+pRON2MXNRW32pcPjQksKpYAl7v3bcEZTH0 g3zynk5xn86RtS2MO90LKRC2jwqzJg6KBSKtRqR8+HLfMg0yjxHIRD1XIGrRLfxh7j/3MJcx5+E OfZd1iFtjp/iiQQ== X-Received: from dybud10.prod.google.com ([2002:a05:7300:f60a:b0:311:6ce7:736d]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:458b:b0:2d6:f3f0:3e7a with SMTP id d9443c01a7336-2d74dc2ade3mr36340345ad.3.1787873485885; Thu, 27 Aug 2026 16:31:25 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:43 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-6-stevensd@google.com> Subject: [RFC 05/10] fork: allocate reclaimable stacks with VM_SPARSE From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Since the pages of reclaimable stacks will not always be populated, we need to set VM_SPARSE on their vm_structs to avoid crashes when vread_iter sees a partially reclaimed stack. Note that stacks will only be partially reclaimed when they are associated with a live task, so the stack management and caching code in fork.c won't actually ever see a partially reclaimed stack. Directly managing the vm_structs instead of going through vmalloc also requires doing the freeing of the stack on a work queue instead of in an RCU callback. Previously, we were relying on deferred vfree work to move much of the cleanup work from softirq to process context. The fact that vmap stack pages are included in vmalloc's vmstat is well known. We should continue that to avoid changing procfs. Signed-off-by: David Stevens --- kernel/fork.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/kernel/fork.c b/kernel/fork.c index 10bd76f6dd20..6acad0038b78 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -267,6 +267,97 @@ static bool try_release_thread_stack_to_cache(struct v= m_struct *vm_area) return false; } =20 +#ifdef CONFIG_RECLAIMABLE_STACK +static void free_vmap_stack(struct vm_struct *vm_area) +{ + int i; + + remove_vm_area(vm_area->addr); + + for (i =3D 0; i < vm_area->nr_pages; i++) { + mod_node_page_state(page_pgdat(vm_area->pages[i]), NR_VMALLOC, -1); + __free_page(vm_area->pages[i]); + } + + kfree(vm_area->pages); + kfree(vm_area); +} + +static struct vm_struct *alloc_vmap_stack(int node) +{ + struct vm_struct *vm_area; + int ret; + + vm_area =3D get_vm_area_node(THREAD_SIZE, THREAD_ALIGN, VM_MAP | VM_SPARS= E, node); + if (!vm_area) + return NULL; + + vm_area->pages =3D kcalloc_node(THREAD_SIZE >> PAGE_SHIFT, sizeof(*vm_are= a->pages), + GFP_KERNEL | __GFP_ZERO, node); + if (!vm_area->pages) + goto alloc_failure; + + while (vm_area->nr_pages < THREAD_SIZE >> PAGE_SHIFT) { + struct page *page; + gfp_t gfp =3D GFP_VMAP_STACK | __GFP_HIGHMEM; + + if (node =3D=3D NUMA_NO_NODE) + page =3D alloc_pages(gfp, 0); + else + page =3D alloc_pages_node(node, gfp, 0); + + if (!page) + goto alloc_failure; + + /* + * Non-reclaimable vmap stacks pages aren't charged against an + * memcg until account_kernel_stack(), but they are added to + * the node's NR_VMALLOC counter. Copy that behavior to avoid + * confusing userspace. + */ + mod_node_page_state(page_pgdat(page), NR_VMALLOC, 1); + vm_area->pages[vm_area->nr_pages++] =3D page; + } + + ret =3D vmap_pages_range((unsigned long)vm_area->addr, + (unsigned long)vm_area->addr + THREAD_SIZE, + PAGE_KERNEL, vm_area->pages, PAGE_SHIFT); + if (ret) + goto alloc_failure; + + return vm_area; + +alloc_failure: + free_vmap_stack(vm_area); + return NULL; +} + +struct vm_stack { + struct rcu_work work; + struct vm_struct *stack_vm_area; +}; + +static void thread_stack_free_work(struct work_struct *work) +{ + struct vm_stack *vm_stack =3D container_of(to_rcu_work(work), struct vm_s= tack, work); + struct vm_struct *vm_area =3D vm_stack->stack_vm_area; + + if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) + return; + + free_vmap_stack(vm_area); +} + +static void thread_stack_delayed_free(struct task_struct *tsk) +{ + struct vm_stack *vm_stack =3D tsk->stack; + + vm_stack->stack_vm_area =3D tsk->stack_vm_area; + INIT_RCU_WORK(&vm_stack->work, thread_stack_free_work); + queue_rcu_work(system_wq, &vm_stack->work); +} + +#else /* !CONFIG_RECLAIMABLE_STACK */ static void free_vmap_stack(struct vm_struct *vm_area) { vfree(vm_area->addr); @@ -305,6 +396,7 @@ static void thread_stack_delayed_free(struct task_struc= t *tsk) vm_stack->stack_vm_area =3D tsk->stack_vm_area; call_rcu(&vm_stack->rcu, thread_stack_free_rcu); } +#endif /* CONFIG_RECLAIMABLE_STACK */ =20 static int free_vm_stack_cache(unsigned int cpu) { --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FF4B3E16B9 for ; Thu, 27 Aug 2026 23:31:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873491; cv=none; b=JzGCDZOU7jnsisAOKAuOg7VQxUXNZqez+FFVn8bHxDgZYYWffB7z9wsnx0Jwwh21B3a0aq6rfR46XXAWR+SbuHFPYGbdIcxWIIMUeduH4k1xgqMCkahHf/GLQV9F/jMnT8vwW7znGdBPLCY974hC9cVTgRkQ9QOjSf0s3Tnbt1k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873491; c=relaxed/simple; bh=JKcdzLhT55vZUuF15Pa+/WM53yNof0qoNdhu59RgtuA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Wfnnh29C43AkEVFjcAsosyGFQAzPc0Jc+Kogqp77S2cwgIAN2Xr3VD5DogK6EvUvV2PPw/+msHlXIalOzwuC74xzF0VX4MdXnZYmB7Ih7BGAoYDcx71eXSTZU+VPBU0IXomDjeB6jKXuEW8plYwMfMsGtA2J2CNiRxaahkncHbQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=LBhkOXyJ; arc=none smtp.client-ip=209.85.216.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="LBhkOXyJ" Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-38f5ac7416eso620703a91.3 for ; Thu, 27 Aug 2026 16:31:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873489; x=1788478289; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=qsQSfVFeubNsDIXPhKgUpT/ImczFchV73yRsCSFESEY=; b=LBhkOXyJvoDO5ubGpIc21d56SCuf5IQnppajcfKdhAI+HdMHoZD+dYfTuv/GegHPX4 WJ/dK9FZzex8fM3coAoR6163voAtr+/qYyVCVFDwJPd+Ahk+kSBtf+Vt1dHrPTlPTHn9 eSTByN8eqQ+hbhRjycFVPACWn+4D3V8heowzWEoFVjCG7iGdOSykS9uOPv5sEoS614JD 6W2WefMmS+Kut4aazz3c9ylZC/w0f6RvOXfB3rjKQM9oI8GwcwSHaDf6qSWv/qWjGsFQ TfH+S2NSTSSp99cZGp5+Hogd+zSGmqkDSTHm+41spzR1HQ6OWznbInxrZBNURwcV+Ob2 QoSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873489; x=1788478289; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=qsQSfVFeubNsDIXPhKgUpT/ImczFchV73yRsCSFESEY=; b=PWnE3+mdMKCm+2GaOKHhlmv9gvMo/LF3t+Izew0nOhvbg4E6qkWrOyQom9oZPmBURd ppNC5NsrkCGTE2meB8sWk162Htu54EMZWpRrodt0dDoztwFF7obrifVrv9NhrZ//y0c/ U1OHL2hcjMhWqT4Ke39u+1d5VdtApP0xeW2Lo9gnZALueHKytpcq6wkxOBBKKgw2MF2H Lh3cEXinMo2VA89o9pBxk2jmVjIsiDrbHB0bHzatctfztAZQq2sIDjOenAB4aa91LAyj 66fx3zbMl/bAa+TZM78UBDCWElpsL6rMvSSMnGkJ1YXZYpCSjhN4xrNi3Ej6jdwGTl+0 nXlQ== X-Gm-Message-State: AFuF++lNEXiNM1ftn7hvQkiRga/Um3yKGErwYFumsdkLrUuEvTCVujwO wt4YYMqMYZGEoECIkf1o4rS41WTukx4yZu/VjLX4r/pNbAJQAwjmGXrr2bkPFiLyaxc0kQvHa8c vEHdHEaLYVPzwvg== X-Received: from dli22-n1.prod.google.com ([2002:a05:7022:296:10b0:141:ac8:b5ae]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:da83:b0:38f:efed:5445 with SMTP id 98e67ed59e1d1-396d0de796fmr5335679a91.4.1787873488329; Thu, 27 Aug 2026 16:31:28 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:44 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-7-stevensd@google.com> Subject: [RFC 06/10] Reclaim memory from blocked kernel stacks From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Although an individual kernel stack is cheap, the cost can add up on thread heavy systems. On such systems, many threads are often blocked for extended periods of time, waiting for userspace or external events. When a task is blocked, its stack is in a stable, well-defined state. Since we can tell exactly which part of those stacks are unused, we can free the unused portions and reduce the global memory footprint of kernel stacks. That said, not all blocked tasks can safely have their stacks reclaimed. Since a running task may need all of its stack, a blocked task whose stack has been partially freed cannot be rescheduled until its stack is fully repopulated. Since allocating that memory may require direct reclaim, if a task holding a lock that direct reclaim depends on has its stack reclaimed, we would run the risk of deadlock. Knowing when tasks hold locks that direct reclaim may depend on is not viable in production systems, especially without lockdep. But noting that most userspace threads will be blocked on events external to the kernel (e.g. waiting on a futex for a userspace controlled wakeup, waiting in epoll for a network packet, waiting on a timer for time to pass, etc), it is possible to enumerate a relatively small number of call sites where most userspace threads will be blocked while holding zero kernel locks. This change adds a new component that reclaims task stacks. A set of scheduler hooks are used to determine when it is safe to reclaim stacks and to ensure that a task's stack is fully repopulated before being rescheduled. A new PF_RECLAIMABLE_STACK task flag is added that will be used to annotate such safe-to-reclaim call sites. Users of the flag must ensure the task doesn't block while holding a lock while the flag is set. A new TASK_STACK_RECLAIM state is introduced for tasks that are blocked waiting for the stacks to be repopulated. While reclaimable stacks will not cause direct reclaim to deadlock, it does introduce a dependency on allocating memory before an OOM victim can exit, since any reclaimed stacks need to be repopulated before their threads can run again. This dependency can lead to further OOM kills or potentially an OOM panic if no additional victims are available. Reclaimable stacks is built on VMAP_STACK. There is no fundamental dependency on !STACK_GROWSUP, but since the only STACK_GROWSUP architecture doesn't support VMAP_STACK, support for STACK_GROWSUP is not implemented. This feature does work on PREEMPT_RT, but is likely undesirable due to the extra uncertainty. The fact that alloc_pages_nolock_noprof() cannot be called from under the scheduler lock also makes repopulating stacks more expensive. Enabling reclaimable stacks is mutually exclusive with enabling DEBUG_STACK_USAGE. Making stack_not_used() safe (including from NMIs) may be technically feasible. Even then, the number that it would report would be subtly different: max since last reclaim vs max ever used. This difference seems likely to confuse anything consuming the data, so disabling the feature seems prudent. Since kernel stacks are not visible in PROC_KCORE when reclaimable stacks is enabled, it is disabled by default when that config is enabled. If a user wants to enable the manually enable the feature and give up visibility, they can make that choice. In the followup patch, reclaiming task stacks will be done in a shrinker. However, this patch does the reclaim using per-task work structs for simplicity. Signed-off-by: David Stevens --- arch/Kconfig | 18 ++ include/linux/sched.h | 39 ++- include/linux/sched/task_stack.h | 23 ++ kernel/Makefile | 2 + kernel/fork.c | 7 + kernel/sched/core.c | 18 +- kernel/sched/sched.h | 3 + kernel/stack_shrinker.c | 464 +++++++++++++++++++++++++++++++ kernel/stack_shrinker.h | 58 ++++ 9 files changed, 627 insertions(+), 5 deletions(-) create mode 100644 kernel/stack_shrinker.c create mode 100644 kernel/stack_shrinker.h diff --git a/arch/Kconfig b/arch/Kconfig index fa7507ac8e13..adb4a5957996 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1534,6 +1534,24 @@ config VMAP_STACK backing virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. =20 +config HAVE_ARCH_RECLAIMABLE_STACK + def_bool n + +config RECLAIMABLE_STACK + default !PREEMPT_RT && !PROC_KCORE + bool "Allow stacks of some blocked threads to be reclaimed" + depends on VMAP_STACK && !STACK_GROWSUP + depends on HAVE_ARCH_RECLAIMABLE_STACK + depends on !DEBUG_STACK_USAGE + depends on !KASAN_VMALLOC # TODO: add support for this + depends on !DEBUG_KMEMLEAK # TODO: add support for this + help + Enable this to allow the unused portion of kernel stacks of most + blocked tasks to be reclaimed. + + The wakeup latency of tasks with reclaimed stacks may increase, + especially while the system is under memory pressure. + config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET def_bool n help diff --git a/include/linux/sched.h b/include/linux/sched.h index 373bcc0598d1..c93a234fac96 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -83,6 +83,7 @@ struct sched_dl_entity; struct seq_file; struct sighand_struct; struct signal_struct; +struct stack_reclaim_work; struct task_delay_info; struct task_exec_state; struct task_group; @@ -124,7 +125,8 @@ struct user_event_mm; #define TASK_FREEZABLE 0x00002000 #define __TASK_FREEZABLE_UNSAFE (0x00004000 * IS_ENABLED(CONFIG_LOC= KDEP)) #define TASK_FROZEN 0x00008000 -#define TASK_STATE_MAX 0x00010000 +#define TASK_STACK_RECLAIM 0x00010000 +#define TASK_STATE_MAX 0x00020000 =20 #define TASK_ANY (TASK_STATE_MAX-1) =20 @@ -823,6 +825,29 @@ struct kmap_ctrl { #endif }; =20 +#ifdef CONFIG_RECLAIMABLE_STACK +enum stack_reclaim_enum { + STACK_IN_USE =3D 0, + STACK_PREPARE_RECLAIM =3D 1, + STACK_RECLAIMABLE =3D 2, + STACK_RECLAIMING =3D 3, + STACK_RECLAIMING_IN_USE =3D 4, + STACK_RECLAIMED =3D 5, +} __packed; + +union stack_reclaim_state { + struct { + enum stack_reclaim_enum stack_state; + u16 node; + }; + u32 val; +}; + +union stack_reclaim_list { + struct llist_node refill_entry; +}; +#endif + struct task_struct { #ifdef CONFIG_THREAD_INFO_IN_TASK /* @@ -1587,6 +1612,16 @@ struct task_struct { #endif #ifdef CONFIG_VMAP_STACK struct vm_struct *stack_vm_area; +#ifdef CONFIG_RECLAIMABLE_STACK + union stack_reclaim_state stack_reclaim_state; + union stack_reclaim_list stack_reclaim_list; +#ifdef CONFIG_MEMCG + struct obj_cgroup *stack_obj_cgroup; +#endif + + // TODO: Replace these with a shrinker + struct stack_reclaim_work *stack_reclaim_work; +#endif #endif #ifdef CONFIG_THREAD_INFO_IN_TASK /* A live task holds one reference: */ @@ -1796,7 +1831,7 @@ extern struct pid *cad_pid; * I am cleaning dirty pages from some other bdi. */ #define PF_KTHREAD 0x00200000 /* I am a kernel thread */ #define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */ -#define PF__HOLE__00800000 0x00800000 +#define PF_RECLAIMABLE_STACK 0x00800000 /* This task's stack is reclaimabl= e */ #define PF__HOLE__01000000 0x01000000 #define PF__HOLE__02000000 0x02000000 #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle = with cpus_mask */ diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_st= ack.h index 1fab7e9043a3..5d5567899d51 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -6,6 +6,7 @@ * task->stack (kernel stack) handling interfaces: */ =20 +#include #include #include #include @@ -83,6 +84,10 @@ static inline void put_task_stack(struct task_struct *ts= k) {} =20 void exit_task_stack_account(struct task_struct *tsk); =20 +/* + * Must only be called on current or from inside __schedule() on + * prev, to avoid crashing when CONFIG_RECLAIM_STACK is enabled. + */ #define task_stack_end_corrupted(task) \ (*(end_of_stack(task)) !=3D STACK_END_MAGIC) =20 @@ -114,4 +119,22 @@ static inline int kstack_end(void *addr) return !(((unsigned long)addr+sizeof(void*)-1) & (THREAD_SIZE-sizeof(void= *))); } =20 +#ifdef CONFIG_RECLAIMABLE_STACK + +DEFINE_CLASS(allow_stack_reclaim, bool, + ({ + if (!_T) + current->flags &=3D ~PF_RECLAIMABLE_STACK; + }), + ({ + bool was_set =3D current->flags & PF_RECLAIMABLE_STACK; + + current->flags |=3D PF_RECLAIMABLE_STACK; + was_set; + }), + void) +#else /* !CONFIG_RECLAIMABLE_STACK */ +DEFINE_CLASS(allow_stack_reclaim, bool, ({ (void)_T; }), ({ false; }), voi= d) +#endif /* !CONFIG_VMAP_STACK */ + #endif /* _LINUX_SCHED_TASK_STACK_H */ diff --git a/kernel/Makefile b/kernel/Makefile index 1e1a31673577..01547102e13a 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -12,6 +12,8 @@ obj-y =3D fork.o exec_domain.o exec_state.o panic.o \ notifier.o ksysfs.o cred.o reboot.o \ async.o range.o smpboot.o ucount.o regset.o ksyms_common.o =20 +obj-$(CONFIG_RECLAIMABLE_STACK) +=3D stack_shrinker.o + obj-$(CONFIG_MULTIUSER) +=3D groups.o obj-$(CONFIG_VHOST_TASK) +=3D vhost_task.o =20 diff --git a/kernel/fork.c b/kernel/fork.c index 6acad0038b78..9b2cc3d01dd1 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -120,6 +120,8 @@ /* For dup_mmap(). */ #include "../mm/internal.h" =20 +#include "stack_shrinker.h" + #include =20 #define CREATE_TRACE_POINTS @@ -460,6 +462,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) =20 tsk->stack_vm_area =3D vm_area; tsk->stack =3D stack; + add_to_stack_shrinker(tsk, node); return 0; } =20 @@ -472,6 +475,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) free_vmap_stack(vm_area); return -ENOMEM; } + /* * We can't call find_vm_area() in interrupt context, and * free_thread_stack() can be called in interrupt context, @@ -480,6 +484,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) tsk->stack_vm_area =3D vm_area; stack =3D kasan_reset_tag(stack); tsk->stack =3D stack; + add_to_stack_shrinker(tsk, node); return 0; } =20 @@ -898,6 +903,7 @@ void __put_task_struct(struct task_struct *tsk) delayacct_tsk_free(tsk); put_signal_struct(tsk->signal); sched_core_free(tsk); + remove_from_stack_shrinker(tsk); free_task(tsk); } EXPORT_SYMBOL_GPL(__put_task_struct); @@ -1124,6 +1130,7 @@ static struct task_struct *dup_task_struct(struct tas= k_struct *orig, int node) free_stack: exit_task_stack_account(tsk); free_thread_stack(tsk); + remove_from_stack_shrinker(tsk); free_tsk: free_task_struct(tsk); return NULL; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 96226707c2f6..ab7db60d5dc2 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4252,6 +4252,7 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) { guard(preempt)(); int cpu, success =3D 0; + bool need_deferred_repopulate, do_deferred_repopulate_wake =3D false; =20 wake_flags |=3D WF_TTWU; =20 @@ -4295,8 +4296,6 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) if (!ttwu_state_match(p, state, &success)) break; =20 - trace_sched_waking(p); - /* * Ensure we load p->on_rq _after_ p->state, otherwise it would * be possible to, falsely, observe p->on_rq =3D=3D 0 and get stuck @@ -4320,8 +4319,18 @@ int try_to_wake_up(struct task_struct *p, unsigned i= nt state, int wake_flags) * A similar smp_rmb() lives in __task_needs_rq_lock(). */ smp_rmb(); - if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) + if (READ_ONCE(p->on_rq) && ttwu_runnable(p, wake_flags)) { + trace_sched_waking(p); + break; + } + + if (!ensure_stack_is_present(p, &need_deferred_repopulate)) { + WRITE_ONCE(p->__state, TASK_STACK_RECLAIM); + do_deferred_repopulate_wake =3D need_deferred_repopulate; break; + } + + trace_sched_waking(p); =20 /* * Ensure we load p->on_cpu _after_ p->on_rq, otherwise it would be @@ -4418,6 +4427,8 @@ int try_to_wake_up(struct task_struct *p, unsigned in= t state, int wake_flags) if (success) ttwu_stat(p, task_cpu(p), wake_flags); =20 + if (unlikely(do_deferred_repopulate_wake)) + wake_stack_repopulate(); return success; } =20 @@ -5354,6 +5365,7 @@ static struct rq *finish_task_switch(struct task_stru= ct *prev) prev_state =3D READ_ONCE(prev->__state); vtime_task_switch(prev); perf_event_task_sched_in(prev, current); + allow_stack_reclaim(prev); finish_task(prev); tick_nohz_task_switch(); finish_lock_switch(rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 56acf502ba26..8a81fb83fe8f 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -79,6 +79,7 @@ #include =20 #include "../workqueue_internal.h" +#include "../stack_shrinker.h" =20 struct rq; struct cfs_rq; @@ -3039,6 +3040,8 @@ static inline void __block_task(struct rq *rq, struct= task_struct *p) delayacct_blkio_start(); } =20 + prepare_stack_for_reclaim(p); + ASSERT_EXCLUSIVE_WRITER(p->on_rq); =20 /* diff --git a/kernel/stack_shrinker.c b/kernel/stack_shrinker.c new file mode 100644 index 000000000000..d7b1a7dfa716 --- /dev/null +++ b/kernel/stack_shrinker.c @@ -0,0 +1,464 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "stack_shrinker.h" + +#include "../mm/internal.h" + +struct repopulate_work { + struct work_struct work; + struct llist_head stacks; +}; + +static DEFINE_PER_CPU(struct repopulate_work, repopulate_work); + +// TODO: replace with shrinker +struct stack_reclaim_work { + struct task_struct *tsk; + struct irq_work irq_work; + struct work_struct work; +}; + +static void schedule_stack_reclaim_work(struct irq_work *w) +{ + struct stack_reclaim_work *work =3D container_of(w, typeof(*work), irq_wo= rk); + + if (!queue_work(system_wq, &work->work)) + put_task_struct(work->tsk); +} + +static void do_reclaim_stack(struct task_struct *tsk); + +static void do_stack_reclaim_work(struct work_struct *w) +{ + struct task_struct *tsk =3D container_of(w, struct stack_reclaim_work, wo= rk)->tsk; + + do_reclaim_stack(tsk); +} + +#ifdef CONFIG_MEMCG +static void set_stack_obj_cgroup(struct task_struct *tsk) +{ + tsk->stack_obj_cgroup =3D get_obj_cgroup_from_current(); +} + +static void put_stack_obj_cgroup(struct task_struct *tsk) +{ + if (tsk->stack_obj_cgroup) + obj_cgroup_put(tsk->stack_obj_cgroup); +} + +static inline struct mem_cgroup *get_stack_memcg(struct task_struct *tsk) +{ + return tsk->stack_obj_cgroup ? get_mem_cgroup_from_objcg(tsk->stack_obj_c= group) + : root_mem_cgroup; +} +#else /* !CONFIG_MEMCG */ +static void set_stack_obj_cgroup(struct task_struct *tsk) { } + +static void put_stack_obj_cgroup(struct task_struct *tsk) { } + +static inline struct mem_cgroup *get_stack_memcg(struct task_struct *tsk) +{ + return NULL; +} +#endif /* CONFIG_MEMCG */ + +void add_to_stack_shrinker(struct task_struct *tsk, int node) +{ + BUILD_BUG_ON(sizeof(tsk->stack_reclaim_state) !=3D sizeof(tsk->stack_recl= aim_state.val)); + BUILD_BUG_ON(NODES_SHIFT > 15); + + tsk->stack_reclaim_state.val =3D 0; + tsk->stack_reclaim_state.stack_state =3D STACK_IN_USE; + tsk->stack_reclaim_state.node =3D node; + set_stack_obj_cgroup(tsk); + init_llist_node(&tsk->stack_reclaim_list.refill_entry); + + // TODO: replace with shrinker + tsk->stack_reclaim_work =3D kmalloc_obj(*tsk->stack_reclaim_work, GFP_KER= NEL); + BUG_ON(!tsk->stack_reclaim_work); + + tsk->stack_reclaim_work->tsk =3D tsk; + init_irq_work(&tsk->stack_reclaim_work->irq_work, schedule_stack_reclaim_= work); + INIT_WORK(&tsk->stack_reclaim_work->work, do_stack_reclaim_work); +} + +static inline int calculate_num_unused_pages(struct task_struct *tsk) +{ + unsigned long top_of_stack =3D (unsigned long)end_of_stack(tsk); + /* + * Since tsk is !on_rq and !on_cpu, top_of_blocked_task_stack() safely + * tells us the end of the stack frame of the inner most context switch + * function. Any parts of the stack above that are stale stack frames + * or never used, and thus can be unmapped and discarded. + */ + return (top_of_blocked_task_stack(&tsk->thread) - top_of_stack) >> PAGE_S= HIFT; +} + +static bool repopulate_stack(struct task_struct *tsk, bool is_deferred, + struct llist_head *fail_list) +{ + int num_missing_pages =3D 0, nr_allocated =3D 0, ret; + struct page *pages[THREAD_SIZE >> PAGE_SHIFT] =3D {}; + struct vm_struct *vm_area =3D tsk->stack_vm_area; + unsigned long addr =3D (unsigned long)vm_area->addr; + struct mem_cgroup *tsk_memcg, *old_active_memcg; + int node =3D tsk->stack_reclaim_state.node =3D=3D U16_MAX ? NUMA_NO_NODE + : tsk->stack_reclaim_state.node; + + num_missing_pages =3D (THREAD_SIZE >> PAGE_SHIFT) - vm_area->nr_pages; + if (num_missing_pages =3D=3D 0) + return true; + + tsk_memcg =3D get_stack_memcg(tsk); + old_active_memcg =3D set_active_memcg(tsk_memcg); + + if (is_deferred) { + gfp_t gfp =3D GFP_KERNEL_ACCOUNT | __GFP_ZERO; + /* + * If the oom killer wants to free memory from this process, + * allow access to reserves so the task can hopefully run + * sooner to die and thus make progress towards freeing the + * process's non-mm memory. + */ + if (tsk_is_oom_victim(tsk)) + gfp |=3D __GFP_MEMALLOC; + + for (; nr_allocated < num_missing_pages; nr_allocated++) { + pages[nr_allocated] =3D alloc_pages_node_noprof(node, gfp, 0); + if (!pages[nr_allocated]) + goto repopulate_fail; + } + } else { + /* + * PREEMPT_RT turns spin_trylock() from an atomic cmpxchg into + * an operation that takes a rt_mutex's internal raw spin lock. + * Doing that from inside the scheduler would result in a + * circular locking dependency. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + goto repopulate_fail; + + for (; nr_allocated < num_missing_pages; nr_allocated++) { + pages[nr_allocated] =3D alloc_pages_nolock_noprof(__GFP_ACCOUNT, + node, 0); + if (!pages[nr_allocated]) + goto repopulate_fail; + } + } + + set_active_memcg(old_active_memcg); + mem_cgroup_put(tsk_memcg); + + for (int i =3D 0; i < num_missing_pages; i++) { + vm_area->pages[i] =3D pages[i]; + mod_lruvec_page_state(pages[i], NR_KERNEL_STACK_KB, PAGE_SIZE / 1024); + mod_node_page_state(page_pgdat(pages[i]), NR_VMALLOC, 1); + } + vm_area->nr_pages =3D THREAD_SIZE >> PAGE_SHIFT; + + /* + * The page tables for the stack were allocated when the stack was + * originally created, so we're guaranteed not to need to allocate new + * ones. As such, vmap_pages_range() won't acquire any locks and can be + * called under the scheduler's raw spinlocks. + * + * The only way it can fail is if we're trying to colbber an existing + * mapping or if the page allocator gave us an invalid page. Neither + * case is recoverable. + */ + ret =3D vmap_pages_range(addr, addr + num_missing_pages * PAGE_SIZE, + PAGE_KERNEL, vm_area->pages, PAGE_SHIFT); + BUG_ON(ret !=3D 0); + + // TODO: Clearing pages under the scheduler lock is probably too much + // work under a raw spinlock. We could try maintaining our own small + // pool of pre-zero'ed pages instead of using alloc_pages_nolock. + if (!is_deferred) + clear_pages((void *)addr, num_missing_pages); + + set_task_stack_end_magic(tsk); + return true; + +repopulate_fail: + set_active_memcg(old_active_memcg); + mem_cgroup_put(tsk_memcg); + + while (nr_allocated--) + free_pages_nolock(pages[nr_allocated], 0); + + if (!fail_list) { + /* + * Preemption is left disabled until wake_stack_repopulate(), to + * guarantee that we queue the correct work. + */ + preempt_disable(); + fail_list =3D &this_cpu_ptr(&repopulate_work)->stacks; + } + llist_add(&tsk->stack_reclaim_list.refill_entry, fail_list); + return false; +} + +static void release_stack(struct task_struct *tsk) +{ + struct vm_struct *vm_area =3D tsk->stack_vm_area; + unsigned long addr =3D (unsigned long)vm_area->addr; + int nr_to_free; + + nr_to_free =3D calculate_num_unused_pages(tsk); + + if (unlikely(nr_to_free =3D=3D 0)) + return; + + vm_area_unmap_pages(vm_area, addr, addr + nr_to_free * PAGE_SIZE); + for (int i =3D 0; i < nr_to_free; i++) { + mod_lruvec_page_state(vm_area->pages[i], NR_KERNEL_STACK_KB, + -(int)(PAGE_SIZE / 1024)); + mod_node_page_state(page_pgdat(vm_area->pages[i]), NR_VMALLOC, -1); + __free_pages(vm_area->pages[i], 0); + } + vm_area->nr_pages -=3D nr_to_free; +} + +static void do_reclaim_stack(struct task_struct *tsk) +{ + union stack_reclaim_state prev_state, target_state; + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + if (prev_state.stack_state =3D=3D STACK_RECLAIMABLE) + target_state.stack_state =3D STACK_RECLAIMING; + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + /* + * If target_state.stack_state =3D=3D STACK_RECLAIMING, we know tsk is st= ill + * alive and can't run until we're done, so putting the ref here is safe. + */ + put_task_struct(tsk); + if (target_state.stack_state !=3D STACK_RECLAIMING) + return; + + release_stack(tsk); + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + target_state.stack_state =3D STACK_RECLAIMED; + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + if (prev_state.stack_state =3D=3D STACK_RECLAIMING_IN_USE) { + struct repopulate_work *work =3D get_cpu_ptr(&repopulate_work); + + llist_add(&tsk->stack_reclaim_list.refill_entry, &work->stacks); + queue_work(system_highpri_wq, &work->work); + put_cpu_ptr(work); + } +} + +static void do_repopulate_stacks(struct work_struct *w) +{ + struct repopulate_work *work =3D container_of(w, struct repopulate_work, = work); + struct llist_node *head; + + while ((head =3D llist_del_all(&work->stacks))) { + struct task_struct *tsk, *tmp; + + llist_for_each_entry_safe(tsk, tmp, head, stack_reclaim_list.refill_entr= y) { + init_llist_node(&tsk->stack_reclaim_list.refill_entry); + if (repopulate_stack(tsk, true, &work->stacks)) { + wake_up_state(tsk, TASK_STACK_RECLAIM); + } else { + /* + * Repopulate only failes due to low memory. If + * that happens, give the rest of the system a + * chance to free some memory. + */ + cond_resched(); + } + } + } +} + +bool __ensure_stack_is_present(struct task_struct *tsk, bool *need_deferre= d_repopulate) +{ + union stack_reclaim_state prev_state, target_state; + + *need_deferred_repopulate =3D false; + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + + switch (prev_state.stack_state) { + case STACK_IN_USE: + return true; + case STACK_PREPARE_RECLAIM: + case STACK_RECLAIMABLE: + case STACK_RECLAIMED: + /* + * Transitioning STACK_RECLAIM -> STACK_IN_USE won't + * combine with the prior STACK_IN_USE case to lead to + * tasks with unpopulated stacks running. If immediate + * repopulation fails, then ttwu() puts the task in the + * TASK_STACK_RECLAIM state, so the only ttwu() that can + * wake up the task is after we repopulate the stack. + */ + target_state.stack_state =3D STACK_IN_USE; + break; + case STACK_RECLAIMING: + target_state.stack_state =3D STACK_RECLAIMING_IN_USE; + break; + case STACK_RECLAIMING_IN_USE: + /* + * Tasks with stacks in state STACK_RECLAIMING_IN_USE + * should have __state =3D=3D TASK_STACK_RECLAIM state, so + * ttwu_state_match() should reject any wakeups other + * than the one after the stack gets repopulated. + */ + WARN(1, "TASK_STACK_RECLAIM violation"); + return false; + } + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + switch (prev_state.stack_state) { + case STACK_RECLAIMABLE: + case STACK_PREPARE_RECLAIM: + return true; + case STACK_RECLAIMING: + return false; + case STACK_RECLAIMED: + if (repopulate_stack(tsk, false, NULL)) + return true; + *need_deferred_repopulate =3D true; + return false; + default: + // Unreachable due to return statements in cmpxchg loop + unreachable(); + } +} + +void __prepare_stack_for_reclaim(struct task_struct *tsk) +{ + union stack_reclaim_state prev_state, target_state; + + // TODO: Skip rt threads + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + + if (prev_state.stack_state =3D=3D STACK_IN_USE) { + target_state.stack_state =3D STACK_PREPARE_RECLAIM; + } else { + WARN(1, "Runnable thread with reclaimable stack state=3D%x", + prev_state.stack_state); + return; + } + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + /* + * With delayed dequeue, __allow_stack_reclaim() can be called by + * finish_task() before __block_task() calls this function. When + * that happens, __allow_stack_reclaim() sees STACK_IN_USE and is + * thus a no-op. We need a call here to progress the state machine. + * + * Note that __block_task() is called under the rq lock, so we don't + * need to worry about concurrent calls. + */ + if (!tsk->on_cpu) + __allow_stack_reclaim(tsk); +} + +void __allow_stack_reclaim(struct task_struct *tsk) +{ + union stack_reclaim_state prev_state, target_state; + + if (WARN_ON_ONCE(tsk->__state =3D=3D TASK_DEAD)) + return; + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + + if (prev_state.stack_state !=3D STACK_PREPARE_RECLAIM) { + WARN(prev_state.stack_state !=3D STACK_IN_USE, + "Reclaimable state %x for previously running task", prev_state.val= ); + return; + } + target_state.stack_state =3D STACK_RECLAIMABLE; + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + if (irq_work_queue(&tsk->stack_reclaim_work->irq_work)) { + /* + * Take a ref that gets released by do_reclaim_stack() so we don't + * have to worry about races with remove_from_stack_shrinker(). + */ + get_task_struct(tsk); + } +} + +/* + * This function is called when the task is deleted, which can happen well + * after the task releases its stack. However, a dead task will never have + * TASK_STACK_RECLAIM set, so its last context switch will leave the stack + * state as STACK_IN_USE. As such, if the shrinker processes a task after = its + * death, it will remove the task from the lru without accessing the stack. + */ +void remove_from_stack_shrinker(struct task_struct *tsk) +{ + put_stack_obj_cgroup(tsk); + kfree(tsk->stack_reclaim_work); +} + +void wake_stack_repopulate(void) +{ + queue_work(system_highpri_wq, &this_cpu_ptr(&repopulate_work)->work); + preempt_enable(); +} + +static int stack_shrinker_cpuhp_setup(unsigned int cpu) +{ + struct repopulate_work *work =3D per_cpu_ptr(&repopulate_work, cpu); + + init_llist_head(&work->stacks); + INIT_WORK(&work->work, do_repopulate_stacks); + return 0; +} + +static int stack_shrinker_cpuhp_teardown(unsigned int cpu) +{ + flush_work(&per_cpu_ptr(&repopulate_work, cpu)->work); + return 0; +} + +static int __init fork_late_init(void) +{ + int ret; + + ret =3D cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "stack_shrinker", + stack_shrinker_cpuhp_setup, + stack_shrinker_cpuhp_teardown); + if (ret < 0) { + WARN(1, "Failed to initialize stack_shrinker cpuhp %d\n", ret); + return 0; + } + + return 0; +} + +module_init(fork_late_init); diff --git a/kernel/stack_shrinker.h b/kernel/stack_shrinker.h new file mode 100644 index 000000000000..242a35bddf3e --- /dev/null +++ b/kernel/stack_shrinker.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_STACK_SHRINKER_H +#define _LINUX_STACK_SHRINKER_H + +#include +#include +#include +#include +#include + +#ifdef CONFIG_RECLAIMABLE_STACK + +void add_to_stack_shrinker(struct task_struct *tsk, int node); +void remove_from_stack_shrinker(struct task_struct *tsk); + +bool __ensure_stack_is_present(struct task_struct *tsk, bool *need_deferre= d_repopulate); +void __prepare_stack_for_reclaim(struct task_struct *tsk); +void __allow_stack_reclaim(struct task_struct *tsk); +void wake_stack_repopulate(void); + +static inline bool ensure_stack_is_present(struct task_struct *tsk, bool *= need_deferred_repopulate) +{ + if (unlikely(tsk->flags & PF_RECLAIMABLE_STACK)) + return __ensure_stack_is_present(tsk, need_deferred_repopulate); + return true; +} + +static inline void prepare_stack_for_reclaim(struct task_struct *tsk) +{ + if (unlikely(tsk->flags & PF_RECLAIMABLE_STACK)) + __prepare_stack_for_reclaim(tsk); +} + +static inline void allow_stack_reclaim(struct task_struct *tsk) +{ + if (unlikely(tsk->flags & PF_RECLAIMABLE_STACK)) + __allow_stack_reclaim(tsk); +} + +#else /* !CONFIG_RECLAIMABLE_STACK */ + +static inline void add_to_stack_shrinker(struct task_struct *tsk, int node= ) {} +static inline void remove_from_stack_shrinker(struct task_struct *tsk) {} + +static inline bool ensure_stack_is_present(struct task_struct *tsk, bool *= need_deferred_repopulate) +{ + return true; +} + +static inline void prepare_stack_for_reclaim(struct task_struct *tsk) {} + +static inline void allow_stack_reclaim(struct task_struct *tsk) {} + +static inline void wake_stack_repopulate(void) {} + +#endif /* CONFIG_RECLAIMABLE_STACK */ + +#endif /* _LINUX_STACK_SHRINKER_H */ --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5485E3E557D for ; Thu, 27 Aug 2026 23:31:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873493; cv=none; b=phOfivfmTorT3cwl/vZ2li7O8vXCewtKq2WGZIjzSwR/NBB4j6PTD8T4q0mZShrR4oXhLAkL2+l8Qp0u0/tBjrkn/ml9YrSuBFPelCYkJkZ3Q7zHt4eC2TloRLtHPekXnauC3HUjZqYL3kGpNaI3SHedxDx7OMWXi9Esy9b9QgM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873493; c=relaxed/simple; bh=QzsK4I7O4UNU9KGyJGbH5fbGdEHciGy0e3moiv3yKA8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H5ocoEnqysZih0V1CDNaQ7cSI0aqCxIq9Ntqm5N0FNC1GE2vko5a8B2XRcfKyFFulAr5ccedC545SnmTXnBV5VTJAofeBwUwenYGurGekiYSQmdIqZUzXMkMats7oUY+M3+6ro1gzOnM92Ra3tN22cir/+K74GHW8473fiTXAY0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=AOgZBrPW; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="AOgZBrPW" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-395543dc382so629104a91.0 for ; Thu, 27 Aug 2026 16:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873491; x=1788478291; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=DF4cMpT7HMhXRKSc173deaVgeyoN8RCgYF4bsP8Paw0=; b=AOgZBrPWB+c1ur8vFljYMiVxAj0rR9aP20PDea++Cbq2csbkEubDH+1PrSEq4TyTD+ R1LDxtxQMHVqy6KiYXB3Xcp1aepNRoLp0NmBfIlTrviI1r2MGnXwYImeCl0O9+VUp7hX 8aMgXVId66qmg3A9QlKcWQz+bYXVZxNSfFbDu8us48xUjCK81ukn+RmLGB9oe0cAaBJr rHtH1RGzChCjZkUbTE3RRMRTkVl9bkLFMJTtqSxszNMYPtw2Cy9P4auuS97AByNMnvkK YhcgeC0yRVOQEDXy2rYdOrTIGjAxFX2/RcaZMUmCI+/WVyNP6LVp0kbwLGqFZCHTvRRm bD4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873491; x=1788478291; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DF4cMpT7HMhXRKSc173deaVgeyoN8RCgYF4bsP8Paw0=; b=PFuqI/Z0Gudo69SiinTkI0tESXjYCR5LZbVwFdCu25kYjq06zXH8wCe6osxshflFDo PUn+MxKg+7tSMp0BRIAcJ5dz5D/Y5wJnYDr9igphw9K2exb2CgT1wAVCUDhibGe1b7Rb N4SrMaX13E2MLnQbCbUcY6nGMqSU965oOjnuGWKJLPXrEB1cfW5EsTzBLkQXrc3qUAYL SZX6I1MxouvD29r2myzbF3vuijxhPWni28poWeJ0lbmHfvug+dEb03HNMvYSTLVcammU OvgVNFDkD6Z8cQyHoBA44R4Lis70i65/HfMzf78c+mKyx85LCHTUS5VtzgP6+k2hXPxh Dy6Q== X-Gm-Message-State: AFuF++lZAfQE5hSmNLwFCbH++XH/41FhDDRKWqY7rwvW2ev9pKZp5kBS WOfC/JK6rgHro7ADyxqmFzAimu8uT+T78aky7N8Oulgdlz/lufd+9s49bnN4V4IWFaxeUsKm1al Ohi9mAk7NZbBpvg== X-Received: from dlbdx3.prod.google.com ([2002:a05:7022:e03:b0:141:59a4:247a]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3c47:b0:38d:f5bb:e0f4 with SMTP id 98e67ed59e1d1-396d0e27e11mr5441676a91.1.1787873490274; Thu, 27 Aug 2026 16:31:30 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:45 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-8-stevensd@google.com> Subject: [RFC 07/10] Reclaim stacks via a shrinker From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use a shrinker to reclaim unused portions of blocked task stacks. The core of the shrinker is a list_lru. A referenced bit is used to require that a stack remains unused for long enough to be seen by the shrinker twice. To minimize how much work is done in the scheduler path and under the pi_lock, tasks are initially put onto a regular list before being moved onto the list_lru by a shrinker. Additionally, tasks are not proactively removed from the lru when they become unblocked - again, to reduce what happens in the scheduler path. Signed-off-by: David Stevens --- include/linux/sched.h | 13 +- kernel/stack_shrinker.c | 400 ++++++++++++++++++++++++++++++++++------ 2 files changed, 357 insertions(+), 56 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index c93a234fac96..dc241d1c058f 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -83,7 +83,6 @@ struct sched_dl_entity; struct seq_file; struct sighand_struct; struct signal_struct; -struct stack_reclaim_work; struct task_delay_info; struct task_exec_state; struct task_group; @@ -835,9 +834,17 @@ enum stack_reclaim_enum { STACK_RECLAIMED =3D 5, } __packed; =20 +enum stack_reclaim_lru_state_enum { + STACK_LRU_NOT_PRESENT =3D 0, + STACK_LRU_PENDING =3D 1, + STACK_LRU_NEW =3D 2, + STACK_LRU_OLD =3D 3, +} __packed; + union stack_reclaim_state { struct { enum stack_reclaim_enum stack_state; + enum stack_reclaim_lru_state_enum lru_state; u16 node; }; u32 val; @@ -845,6 +852,7 @@ union stack_reclaim_state { =20 union stack_reclaim_list { struct llist_node refill_entry; + struct list_head reclaim_entry; }; #endif =20 @@ -1618,9 +1626,6 @@ struct task_struct { #ifdef CONFIG_MEMCG struct obj_cgroup *stack_obj_cgroup; #endif - - // TODO: Replace these with a shrinker - struct stack_reclaim_work *stack_reclaim_work; #endif #endif #ifdef CONFIG_THREAD_INFO_IN_TASK diff --git a/kernel/stack_shrinker.c b/kernel/stack_shrinker.c index d7b1a7dfa716..62cc5303d69b 100644 --- a/kernel/stack_shrinker.c +++ b/kernel/stack_shrinker.c @@ -26,28 +26,41 @@ struct repopulate_work { =20 static DEFINE_PER_CPU(struct repopulate_work, repopulate_work); =20 -// TODO: replace with shrinker -struct stack_reclaim_work { - struct task_struct *tsk; - struct irq_work irq_work; - struct work_struct work; -}; +static DEFINE_RAW_SPINLOCK(new_reclaimable_stacks_lock); +static LIST_HEAD(new_reclaimable_stacks); =20 -static void schedule_stack_reclaim_work(struct irq_work *w) -{ - struct stack_reclaim_work *work =3D container_of(w, typeof(*work), irq_wo= rk); +static struct list_lru reclaimable_stacks_lru; =20 - if (!queue_work(system_wq, &work->work)) - put_task_struct(work->tsk); -} +/* + * do_shrink_slab() wants us to scan (nr_obj / (1 << priority)), but unless + * reclaim is really struggling, that can round down to 0 for a lot of + * memcgs. Compensate for that by scaling count and batch size. + */ +#define STACK_COUNT_SHIFT DEF_PRIORITY +/* + * Large batch sizes (like the 128 default) can result in spiky behavior, = where + * deferred work acculmulates and then all of a memcg's stacks get scanned= all + * at once. + */ +#define STACK_BATCH_SIZE 4 =20 -static void do_reclaim_stack(struct task_struct *tsk); +/* + * For purposes of shrinker iteration, a stack allocated with NUMA_NO_NODE= is + * associated with the node that created it. This extra bit allows us to + * determine if NUMA_NO_NODE or the saved node should be used for repopula= tion. + */ +#define TASK_NUMA_NO_NODE_FLAG BIT(15) =20 -static void do_stack_reclaim_work(struct work_struct *w) +static int task_shrinker_node(struct task_struct *tsk) { - struct task_struct *tsk =3D container_of(w, struct stack_reclaim_work, wo= rk)->tsk; + return tsk->stack_reclaim_state.node & ~TASK_NUMA_NO_NODE_FLAG; +} =20 - do_reclaim_stack(tsk); +static int task_alloc_node(struct task_struct *tsk) +{ + if (tsk->stack_reclaim_state.node & TASK_NUMA_NO_NODE_FLAG) + return NUMA_NO_NODE; + return tsk->stack_reclaim_state.node; } =20 #ifdef CONFIG_MEMCG @@ -85,17 +98,13 @@ void add_to_stack_shrinker(struct task_struct *tsk, int= node) =20 tsk->stack_reclaim_state.val =3D 0; tsk->stack_reclaim_state.stack_state =3D STACK_IN_USE; - tsk->stack_reclaim_state.node =3D node; + tsk->stack_reclaim_state.lru_state =3D STACK_LRU_NOT_PRESENT; set_stack_obj_cgroup(tsk); - init_llist_node(&tsk->stack_reclaim_list.refill_entry); + INIT_LIST_HEAD(&tsk->stack_reclaim_list.reclaim_entry); =20 - // TODO: replace with shrinker - tsk->stack_reclaim_work =3D kmalloc_obj(*tsk->stack_reclaim_work, GFP_KER= NEL); - BUG_ON(!tsk->stack_reclaim_work); - - tsk->stack_reclaim_work->tsk =3D tsk; - init_irq_work(&tsk->stack_reclaim_work->irq_work, schedule_stack_reclaim_= work); - INIT_WORK(&tsk->stack_reclaim_work->work, do_stack_reclaim_work); + if (node =3D=3D NUMA_NO_NODE) + node =3D numa_node_id() | TASK_NUMA_NO_NODE_FLAG; + tsk->stack_reclaim_state.node =3D node; } =20 static inline int calculate_num_unused_pages(struct task_struct *tsk) @@ -118,8 +127,7 @@ static bool repopulate_stack(struct task_struct *tsk, b= ool is_deferred, struct vm_struct *vm_area =3D tsk->stack_vm_area; unsigned long addr =3D (unsigned long)vm_area->addr; struct mem_cgroup *tsk_memcg, *old_active_memcg; - int node =3D tsk->stack_reclaim_state.node =3D=3D U16_MAX ? NUMA_NO_NODE - : tsk->stack_reclaim_state.node; + int node =3D task_alloc_node(tsk); =20 num_missing_pages =3D (THREAD_SIZE >> PAGE_SHIFT) - vm_area->nr_pages; if (num_missing_pages =3D=3D 0) @@ -239,21 +247,6 @@ static void do_reclaim_stack(struct task_struct *tsk) { union stack_reclaim_state prev_state, target_state; =20 - prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); - do { - target_state.val =3D prev_state.val; - if (prev_state.stack_state =3D=3D STACK_RECLAIMABLE) - target_state.stack_state =3D STACK_RECLAIMING; - } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); - - /* - * If target_state.stack_state =3D=3D STACK_RECLAIMING, we know tsk is st= ill - * alive and can't run until we're done, so putting the ref here is safe. - */ - put_task_struct(tsk); - if (target_state.stack_state !=3D STACK_RECLAIMING) - return; - release_stack(tsk); =20 prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); @@ -271,6 +264,245 @@ static void do_reclaim_stack(struct task_struct *tsk) } } =20 +static void process_one_new_reclaimable_stack(struct task_struct *tsk, str= uct list_head *new_stacks) +{ + union stack_reclaim_state prev_state, target_state; + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + + switch (prev_state.stack_state) { + case STACK_IN_USE: + case STACK_PREPARE_RECLAIM: + target_state.lru_state =3D STACK_LRU_NOT_PRESENT; + break; + case STACK_RECLAIMABLE: + target_state.lru_state =3D STACK_LRU_NEW; + break; + case STACK_RECLAIMING: + case STACK_RECLAIMING_IN_USE: + case STACK_RECLAIMED: + /* + * These states only happen after a shrinker has started + * processing a stack, so seeing one of these means + * multiple shrinkers are somehow targeting one stack. + */ + WARN(1, "task with stack state %x on list\n", prev_state.val); + put_task_struct(tsk); + return; + } + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + if (target_state.stack_state =3D=3D STACK_RECLAIMABLE) + list_add(&tsk->stack_reclaim_list.reclaim_entry, new_stacks); + else + put_task_struct(tsk); +} + +static void process_new_reclaimable_stacks(void) +{ + LIST_HEAD(new_stacks); + struct task_struct *tsk, *tmp; + + scoped_guard(raw_spinlock_irq, &new_reclaimable_stacks_lock) { + while ((tsk =3D list_first_entry_or_null(&new_reclaimable_stacks, + typeof(*tsk), + stack_reclaim_list.reclaim_entry))) { + list_del_init(&tsk->stack_reclaim_list.reclaim_entry); + + /* + * We hold a ref on the task during the interval when a + * stack is being moved from new_reclaimable_stacks + * onto the lru, to avoid needing to deal with races + * against remove_from_stack_shrinker(). If tryget + * fails here, tsk is about to be deleted, so we can + * just skip it. + */ + if (!tryget_task_struct(tsk)) + continue; + + raw_spin_unlock_irq(&new_reclaimable_stacks_lock); + + process_one_new_reclaimable_stack(tsk, &new_stacks); + + raw_spin_lock_irq(&new_reclaimable_stacks_lock); + } + } + + list_for_each_entry_safe(tsk, tmp, &new_stacks, stack_reclaim_list.reclai= m_entry) { + struct mem_cgroup *memcg =3D get_stack_memcg(tsk); + + list_del_init(&tsk->stack_reclaim_list.reclaim_entry); + + if (memcg_list_lru_alloc(memcg, &reclaimable_stacks_lru, GFP_ATOMIC) =3D= =3D 0) { + local_bh_disable(); + list_lru_add(&reclaimable_stacks_lru, + &tsk->stack_reclaim_list.reclaim_entry, + task_shrinker_node(tsk), memcg); + local_bh_enable(); + } else { + union stack_reclaim_state prev_state, target_state; + + pr_warn_ratelimited("failed to allocate stack reclaim metadata\n"); + + /* + * Just give up if memcg_list_lru_alloc() fails. We + * could try immediately reclaiming the stack, but + * reclaiming a single stack isn't going to help if + * things are so bad a GFP_ATOMIC slab allocation fails. + */ + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + if (prev_state.stack_state =3D=3D STACK_RECLAIMABLE) + target_state.stack_state =3D STACK_RECLAIMED; + target_state.lru_state =3D STACK_LRU_NOT_PRESENT; + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, + &prev_state.val, target_state.val)); + } + + mem_cgroup_put(memcg); + put_task_struct(tsk); + } +} + +static enum lru_status isolate_lru_stack(struct list_head *item, + struct list_lru_one *lru, void *arg) +{ + struct list_head *to_reclaim =3D arg; + struct task_struct *tsk =3D container_of(item, struct task_struct, + stack_reclaim_list.reclaim_entry); + union stack_reclaim_state prev_state, target_state; + enum lru_status ret; + bool is_isolated =3D false; + + prev_state.val =3D READ_ONCE(tsk->stack_reclaim_state.val); + do { + target_state.val =3D prev_state.val; + + switch (prev_state.stack_state) { + case STACK_IN_USE: + case STACK_PREPARE_RECLAIM: + target_state.lru_state =3D STACK_LRU_NOT_PRESENT; + ret =3D LRU_REMOVED; + break; + case STACK_RECLAIMABLE: + switch (prev_state.lru_state) { + case STACK_LRU_NEW: + target_state.lru_state =3D STACK_LRU_OLD; + ret =3D LRU_ROTATE; + break; + case STACK_LRU_OLD: + target_state.stack_state =3D STACK_RECLAIMING; + target_state.lru_state =3D STACK_LRU_NOT_PRESENT; + ret =3D LRU_REMOVED; + break; + case STACK_LRU_NOT_PRESENT: + case STACK_LRU_PENDING: + WARN(1, "item on stack reclaim lru with bad state\n"); + list_lru_isolate(lru, item); + return LRU_REMOVED; + } + break; + case STACK_RECLAIMING: + case STACK_RECLAIMING_IN_USE: + case STACK_RECLAIMED: + /* + * These states only happen after a shrinker has started + * processing a stack, so seeing one of these means + * multiple shrinkers are somehow targeting one stack. + */ + WARN(1, "stack with state %x on list\n", prev_state.val); + list_lru_isolate(lru, item); + return LRU_REMOVED; + } + + /* + * We need to isolate the item before the cmpxchg to prevent + * races with process_one_new_reclaimable_stack() adding the + * entry to its new_stacks list. + */ + if (ret =3D=3D LRU_REMOVED && !is_isolated) { + is_isolated =3D true; + list_lru_isolate(lru, item); + } + } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); + + if (target_state.stack_state =3D=3D STACK_RECLAIMING) { + /* + * The task can't be freed since STACK_RECLAIMING prevents + * it from running and exiting, so no need to hold a ref. + */ + list_add_tail(item, to_reclaim); + } else if (target_state.stack_state =3D=3D STACK_RECLAIMABLE) { + /* + * If we isolated but then lost a cmpxchg race against + * __allow_stack_reclaim(), we need to undo the isolation. + * + * STACK_RECLAIMABLE means we observed the task blocked (i.e. + * not dead). Even if the task dies and its refcount hits zero, + * RCU cleanup of the task will be delayed because the lru walk + * is guarded by local_bh_disable(), so unlocking won't cause + * races with remove_from_stack_shrinker(). + */ + if (is_isolated) { + struct mem_cgroup *lru_memcg; + + spin_unlock(&lru->lock); + + lru_memcg =3D get_stack_memcg(tsk); + list_lru_add(&reclaimable_stacks_lru, item, + task_shrinker_node(tsk), lru_memcg); + mem_cgroup_put(lru_memcg); + + ret =3D LRU_REMOVED_RETRY; + + } else { + WARN(target_state.lru_state !=3D STACK_LRU_OLD, + "Reclaimable stack with bad state %x\n", target_state.val); + } + } else { + WARN(target_state.lru_state !=3D STACK_LRU_NOT_PRESENT, + "Isolate stack with bad state %x\n", target_state.val); + } + + return ret; +} + +static unsigned long scan_reclaimable_stacks(struct shrinker *shrinker, + struct shrink_control *sc) +{ + unsigned long freed =3D 0; + LIST_HEAD(to_reclaim); + struct task_struct *tsk, *tmp; + + sc->nr_to_scan >>=3D STACK_COUNT_SHIFT; + local_bh_disable(); + list_lru_shrink_walk(&reclaimable_stacks_lru, sc, + isolate_lru_stack, &to_reclaim); + local_bh_enable(); + + list_for_each_entry_safe(tsk, tmp, &to_reclaim, stack_reclaim_list.reclai= m_entry) { + freed++; + list_del_init(&tsk->stack_reclaim_list.reclaim_entry); + do_reclaim_stack(tsk); + } + + return freed << STACK_COUNT_SHIFT; +} + +static unsigned long get_reclaimable_stack_count(struct shrinker *shrinker, + struct shrink_control *sc) +{ + unsigned long count; + + process_new_reclaimable_stacks(); + count =3D list_lru_shrink_count(&reclaimable_stacks_lru, sc); + + return count ? (unsigned long)(count << STACK_COUNT_SHIFT) : SHRINK_EMPTY; +} + static void do_repopulate_stacks(struct work_struct *w) { struct repopulate_work *work =3D container_of(w, struct repopulate_work, = work); @@ -285,7 +517,7 @@ static void do_repopulate_stacks(struct work_struct *w) wake_up_state(tsk, TASK_STACK_RECLAIM); } else { /* - * Repopulate only failes due to low memory. If + * Repopulate only fails due to low memory. If * that happens, give the rest of the system a * chance to free some memory. */ @@ -387,6 +619,7 @@ void __prepare_stack_for_reclaim(struct task_struct *ts= k) void __allow_stack_reclaim(struct task_struct *tsk) { union stack_reclaim_state prev_state, target_state; + bool add_to_list =3D false; =20 if (WARN_ON_ONCE(tsk->__state =3D=3D TASK_DEAD)) return; @@ -401,14 +634,17 @@ void __allow_stack_reclaim(struct task_struct *tsk) return; } target_state.stack_state =3D STACK_RECLAIMABLE; + if (prev_state.lru_state =3D=3D STACK_LRU_NOT_PRESENT) { + target_state.lru_state =3D STACK_LRU_PENDING; + add_to_list =3D true; + } else if (prev_state.lru_state =3D=3D STACK_LRU_OLD) { + target_state.lru_state =3D STACK_LRU_NEW; + } } while (!try_cmpxchg(&tsk->stack_reclaim_state.val, &prev_state.val, tar= get_state.val)); =20 - if (irq_work_queue(&tsk->stack_reclaim_work->irq_work)) { - /* - * Take a ref that gets released by do_reclaim_stack() so we don't - * have to worry about races with remove_from_stack_shrinker(). - */ - get_task_struct(tsk); + if (add_to_list) { + guard(raw_spinlock_irqsave)(&new_reclaimable_stacks_lock); + list_add(&tsk->stack_reclaim_list.reclaim_entry, &new_reclaimable_stacks= ); } } =20 @@ -421,8 +657,34 @@ void __allow_stack_reclaim(struct task_struct *tsk) */ void remove_from_stack_shrinker(struct task_struct *tsk) { + struct mem_cgroup *lru_memcg; + + /* + * Since tsk is being deleted, the tryget_task_struct() call in + * process_new_reclaimable_stacks() excludes racing with a shrinker + * moving the task from STACK_LRU_PENDING -> STACK_LRU_NEW. A race can + * just result in the delete operation being a no-op. + */ + switch (READ_ONCE(tsk->stack_reclaim_state.lru_state)) { + case STACK_LRU_NOT_PRESENT: + break; + case STACK_LRU_PENDING: + scoped_guard(raw_spinlock_irqsave, &new_reclaimable_stacks_lock) { + if (!list_empty(&tsk->stack_reclaim_list.reclaim_entry)) + list_del_init(&tsk->stack_reclaim_list.reclaim_entry); + } + break; + case STACK_LRU_NEW: + case STACK_LRU_OLD: + lru_memcg =3D get_stack_memcg(tsk); + local_bh_disable(); + list_lru_del(&reclaimable_stacks_lru, &tsk->stack_reclaim_list.reclaim_e= ntry, + task_shrinker_node(tsk), lru_memcg); + local_bh_enable(); + mem_cgroup_put(lru_memcg); + break; + } put_stack_obj_cgroup(tsk); - kfree(tsk->stack_reclaim_work); } =20 void wake_stack_repopulate(void) @@ -431,6 +693,8 @@ void wake_stack_repopulate(void) preempt_enable(); } =20 +static struct lock_class_key stack_shrinker_key; + static int stack_shrinker_cpuhp_setup(unsigned int cpu) { struct repopulate_work *work =3D per_cpu_ptr(&repopulate_work, cpu); @@ -448,16 +712,48 @@ static int stack_shrinker_cpuhp_teardown(unsigned int= cpu) =20 static int __init fork_late_init(void) { + struct shrinker *shrinker; + const char *msg; int ret; + enum cpuhp_state cpuhp_val; =20 ret =3D cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "stack_shrinker", stack_shrinker_cpuhp_setup, stack_shrinker_cpuhp_teardown); if (ret < 0) { - WARN(1, "Failed to initialize stack_shrinker cpuhp %d\n", ret); - return 0; + msg =3D "cpuhp failure"; + goto cpuhp_setup_fail; + } + cpuhp_val =3D ret; + + shrinker =3D shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE, + "stack_shrinker"); + if (!shrinker) { + msg =3D "shrinker alloc failure"; + ret =3D -ENOMEM; + goto shrinker_alloc_fail; } =20 + ret =3D list_lru_init_memcg_key(&reclaimable_stacks_lru, shrinker, + &stack_shrinker_key); + if (ret !=3D 0) { + msg =3D "list_lru_init failure"; + goto list_lru_init_fail; + } + + shrinker->count_objects =3D get_reclaimable_stack_count; + shrinker->scan_objects =3D scan_reclaimable_stacks; + shrinker->seeks =3D 4; + shrinker->batch =3D STACK_BATCH_SIZE << STACK_COUNT_SHIFT; + shrinker_register(shrinker); + return 0; + +list_lru_init_fail: + shrinker_free(shrinker); +shrinker_alloc_fail: + cpuhp_remove_state(cpuhp_val); +cpuhp_setup_fail: + WARN(1, "Failed to initialize stack_shrinker %s: %d\n", msg, ret); return 0; } =20 --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B8193E0C46 for ; Thu, 27 Aug 2026 23:31:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873495; cv=none; b=V806FEWlUkVoSaRYnbik6D056qqV5dgOQ+4rdJaEb0bkvC5NaiY8S5elP77Dk+b1t4bPEpPc7zL2fxLZsizp5V8uGDD8bLCgm81Ss5iFLJ5iTCy40nTJKV5AaNagFiw+8qdcIGLx+zAlhWV+e5z9TKwsFS3uM2ox8Ek/kc1Zpts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873495; c=relaxed/simple; bh=MoqQQPGu0W+jsTlyJM/eR16SFJKPR4Ev04RS0g2Ytf4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=AXMUrh7L7nHtUIg53QMGwCmMRjScQ/xlJ3XoDBZn99iA9huF3Uz9yKSeiuhzmk1xFqCWX0hC4inSy5pecCnfG53cQr3J5xAEBUDEpnfPXCfRXaAjVCQlD+AXrBbVg8SpNRlLqwdkSVuKFHutkTFmA/Z59AA3MKjdcQm7NdM3gAU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=J2Qi3xaM; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="J2Qi3xaM" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-38ea32e57e2so469075a91.1 for ; Thu, 27 Aug 2026 16:31:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873493; x=1788478293; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=PgniuiCLGc5SclSuhaFn4pRhiopbh+4BL9/R1MdVpyI=; b=J2Qi3xaMrqUHswR9RIMndRRIZIitSIMzHJus76+bcNC9pzNVXjIfrHzFKAad55Aeyz wkPPDAiVjg/isFcaScrm477z4d0KgvAqK0PucADQtf0fhOC1/duP2wKNHRUf7o9mgG05 vQwieir1hBCY/xi/Lojh7LoEboBO7CfbuYY1t5YPuYkj1rqXu5mV1uVFsBScbgupRA7c 607ZCSWebE7qDuzczjg5cSeNc+VSr94PST3IuVbgdoU/j0SUD/WmnzXOzim4zUkKqDVk EsjZmb+GX5JSngnPmu2pYYVCHRBQ4bRZBNBGS81XQGkXP7h/xCiFwb4noiECWDa+EI2P 9HyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873493; x=1788478293; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=PgniuiCLGc5SclSuhaFn4pRhiopbh+4BL9/R1MdVpyI=; b=q7qCCpfnLh9j0VpocopQ6rGFqE1r3mSFby1uyJc4wd1uLpeb9E4mqcy6g414cxRHIy 5XZ4T0ZpHVCG/BVJW0T9Rcvc6wxI2VkhcgZe+ka+9uyrcem+YG9s/bosAc6+w2U8UDmT QQD40xcGZ5ZQGs20Gd8fdUY3Q2NTcxr8Ak2FOA8HGLWCJMmN3jYb3aSRbrTGnSPntLRx eBvW8SYNgwW7xwrttilngeW3Quwfdv97XnIwocvm4mcn06vsQeRdbiF2bnewzJsUhbTP T/Yp7laAx6lhqbcEDBZYuTum+VWwZ5SiDazDqUcj5Kaq80dJlqf36waKL+Lau3vw3T4D O3Gw== X-Gm-Message-State: AFuF++lq8jrNKHICGnqyGonoG7xMr4FTXiOpTuwq43D221dVOzNKEnBb aQwZTEGS2CB7C79IMcoiBvndrnao+9Gr/ozVHazuYLpwAl7Ql5TazgGogLnBE43ydSvL/NQVvdN 0zPj95YWbeRzY3w== X-Received: from dlbdm24.prod.google.com ([2002:a05:7022:6b98:b0:141:4a59:5b22]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:380a:b0:38f:de97:b06 with SMTP id 98e67ed59e1d1-396d0daf810mr4980324a91.5.1787873492204; Thu, 27 Aug 2026 16:31:32 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:46 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-9-stevensd@google.com> Subject: [RFC 08/10] Set PF_RECLAIMABLE_STACK in various places From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Annotate various blocking locations with the PF_RECLAIMABLE_STACK. Signed-off-by: David Stevens --- drivers/android/binder/thread.rs | 14 +++++++++++++ fs/eventpoll.c | 3 +++ fs/pipe.c | 28 ++++++++++++++++--------- fs/select.c | 3 +++ kernel/futex/waitwake.c | 3 +++ kernel/signal.c | 36 +++++++++++++++++++------------- kernel/time/hrtimer.c | 3 +++ rust/kernel/task.rs | 16 ++++++++++++++ 8 files changed, 82 insertions(+), 24 deletions(-) diff --git a/drivers/android/binder/thread.rs b/drivers/android/binder/thre= ad.rs index bc0ef8927905..be62f7fd43ca 100644 --- a/drivers/android/binder/thread.rs +++ b/drivers/android/binder/thread.rs @@ -538,13 +538,21 @@ fn get_work_local(self: &Arc, wait: bool) -> Re= sult, wait: bool) -> Result #include #include +#include #include #include #include @@ -2302,6 +2303,8 @@ static int ep_poll(struct eventpoll *ep, struct epoll= _event __user *events, if (signal_pending(current)) return -EINTR; =20 + guard(allow_stack_reclaim)(); + /* * Internally init_wait() uses autoremove_wake_function(), * thus wait entry is removed from the wait queue on each diff --git a/fs/pipe.c b/fs/pipe.c index 429b0714ec57..15503b12fe8f 100644 --- a/fs/pipe.c +++ b/fs/pipe.c @@ -27,6 +27,7 @@ #include #include #include +#include =20 #include #include @@ -469,16 +470,23 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *t= o) break; } mutex_unlock(&pipe->mutex); - /* - * We only get here if we didn't actually read anything. - * - * But because we didn't read anything, at this point we can - * just return directly with -ERESTARTSYS if we're interrupted, - * since we've done any required wakeups and there's no need - * to mark anything accessed. And we've dropped the lock. - */ - if (wait_event_interruptible_exclusive(pipe->rd_wait, pipe_readable(pipe= )) < 0) - return -ERESTARTSYS; + + { + guard(allow_stack_reclaim)(); + /* + * We only get here if we didn't actually read + * anything. + * + * But because we didn't read anything, at this point + * we can just return directly with -ERESTARTSYS if + * we're interrupted, since we've done any required + * wakeups and there's no need to mark anything + * accessed. And we've dropped the lock. + */ + if (wait_event_interruptible_exclusive(pipe->rd_wait, + pipe_readable(pipe)) < 0) + return -ERESTARTSYS; + } =20 wake_next_reader =3D true; mutex_lock(&pipe->mutex); diff --git a/fs/select.c b/fs/select.c index 95d76531015a..3af1bf4d74a9 100644 --- a/fs/select.c +++ b/fs/select.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -236,6 +237,8 @@ static int poll_schedule_timeout(struct poll_wqueues *p= wq, int state, { int rc =3D -EINTR; =20 + guard(allow_stack_reclaim)(); + set_current_state(state); if (!READ_ONCE(pwq->triggered)) rc =3D schedule_hrtimeout_range(expires, slack, HRTIMER_MODE_ABS); diff --git a/kernel/futex/waitwake.c b/kernel/futex/waitwake.c index d4483d15d30a..0fbf7bfcd904 100644 --- a/kernel/futex/waitwake.c +++ b/kernel/futex/waitwake.c @@ -2,6 +2,7 @@ =20 #include #include +#include #include #include =20 @@ -378,6 +379,7 @@ void futex_do_wait(struct futex_q *q, struct hrtimer_sl= eeper *timeout) * has tried to wake us, and we can skip the call to schedule(). */ if (likely(!plist_node_empty(&q->list))) { + guard(allow_stack_reclaim)(); /* * If the timer has already expired, current will already be * flagged for rescheduling. Only call schedule if there @@ -547,6 +549,7 @@ static void futex_sleep_multiple(struct futex_vector *v= s, unsigned int count, return; } =20 + guard(allow_stack_reclaim)(); schedule(); } =20 diff --git a/kernel/signal.c b/kernel/signal.c index bbc0fd4cc4d7..ca0d79ee013f 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -2718,17 +2718,21 @@ static void do_freezer_trap(void) return; } =20 - /* - * Now we're sure that there is no pending fatal signal and no - * pending traps. Clear TIF_SIGPENDING to not get out of schedule() - * immediately (if there is a non-fatal signal pending), and - * put the task into sleep. - */ - __set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE); - clear_thread_flag(TIF_SIGPENDING); - spin_unlock_irq(¤t->sighand->siglock); - cgroup_enter_frozen(); - schedule(); + { + guard(allow_stack_reclaim)(); + + /* + * Now we're sure that there is no pending fatal signal and no + * pending traps. Clear TIF_SIGPENDING to not get out of schedule() + * immediately (if there is a non-fatal signal pending), and + * put the task into sleep. + */ + __set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE); + clear_thread_flag(TIF_SIGPENDING); + spin_unlock_irq(¤t->sighand->siglock); + cgroup_enter_frozen(); + schedule(); + } =20 /* * We could've been woken by task_work, run it to clear @@ -3788,9 +3792,13 @@ static int do_sigtimedwait(const sigset_t *which, ke= rnel_siginfo_t *info, recalc_sigpending(); spin_unlock_irq(&tsk->sighand->siglock); =20 - __set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE); - ret =3D schedule_hrtimeout_range(to, tsk->timer_slack_ns, - HRTIMER_MODE_REL); + { + guard(allow_stack_reclaim)(); + __set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE); + ret =3D schedule_hrtimeout_range(to, tsk->timer_slack_ns, + HRTIMER_MODE_REL); + } + spin_lock_irq(&tsk->sighand->siglock); __set_task_blocked(tsk, &tsk->real_blocked); sigemptyset(&tsk->real_blocked); diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c index 313dcea127fe..30c85482dfd5 100644 --- a/kernel/time/hrtimer.c +++ b/kernel/time/hrtimer.c @@ -39,6 +39,7 @@ #include #include #include +#include #include #include #include @@ -2392,6 +2393,8 @@ static int __sched do_nanosleep(struct hrtimer_sleepe= r *t, enum hrtimer_mode mod struct restart_block *restart; =20 do { + guard(allow_stack_reclaim)(); + set_current_state(TASK_INTERRUPTIBLE|TASK_FREEZABLE); hrtimer_sleeper_start_expires(t, mode); =20 diff --git a/rust/kernel/task.rs b/rust/kernel/task.rs index 38273f4eedb5..6168f058343a 100644 --- a/rust/kernel/task.rs +++ b/rust/kernel/task.rs @@ -344,6 +344,22 @@ pub fn group_leader(&self) -> &Task { // only be used while `current` is still valid, thus still running. unsafe { &*ptr.cast() } } + + /// Sets the given task flag bits on the current task. + #[inline] + pub fn set_flag_bits(&self, set: u32) { + // SAFETY: The `flags` field of `current` is not modified from= other threads, so + // the non-atomic update isn't a race. + unsafe { (*self.as_ptr()).flags |=3D set } + } + + /// Clears the given task flag bits on the current task. + #[inline] + pub fn clear_flag_bits(&self, clear: u32) { + // SAFETY: The `flags` field of `current` is not modified from= other threads, so + // the non-atomic update isn't a race. + unsafe { (*self.as_ptr()).flags &=3D !clear } + } } =20 // SAFETY: The type invariants guarantee that `Task` is always refcounted. --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7B553D5668 for ; Thu, 27 Aug 2026 23:31:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.197 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873496; cv=none; b=CBVbz3S7P7yYgcbdpIrw/3YSSHVtmT2r4CLTn74/B/AgwlYInf5xHxZZ7iraMRhQtTeiyzmPF2eohvLh7tAs7j2wowzRg4lHESzAAbr3YbuTNRK0l7eN+1vx0VNuv9e4Gh+5wRoUWpNdAScjU95yIMY+WvyOUURKEeRRjqUR+sE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873496; c=relaxed/simple; bh=IaRhJZseYtmfOAwaHfBsaGhcCpvv3aaACzd/i2DLAGY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pbIGjmWNOu1qPyQt79/89JkEIQ9X3fAkEY2XUiIVlaf7TI7WjvGvNQKtU9MtWCepGSLKSr2eSYPVclRgLsR3NJOY/rbaSmvlyOarE9cDjeQkmM0fFD9QGAQ4KLR9c2DPiVQawoig3LA4yjk2P2rFAWwAS43LYEJ3XfnV8x1J+jU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ubzFfFqV; arc=none smtp.client-ip=209.85.214.197 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ubzFfFqV" Received: by mail-pl1-f197.google.com with SMTP id d9443c01a7336-2d55d8cd938so3865015ad.1 for ; Thu, 27 Aug 2026 16:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873494; x=1788478294; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=63v8voqDYPPccdvhHdT2gYF/erUoDbYrARzDm1If6YE=; b=ubzFfFqVMJhQxV67j6Q9JQZNam3vdUcUmpStIKKAoVxUhGt9BwIlvN5/Hv5N1tpShl L6/b3DIdI37jGQuHeGWVc05K4caYnlkKhrUE/Cu1mFe05GVO/gi0kBJ1niCbTS0e4I3P WyfmElEEvi/avuuywC3oWjG+QG40elboSlLyhTvRK/MDjwGz7IBIoJRgrRq7Bgg9CBG5 A4S1BFEoQ3u/jnNieM0bYBwPHgjOCfRwgPyZdECXBifvRb5kS3jAeGe68HT3T9LqMo+F CrAQePbAvEgK2bSX9LDDsT+CLxJU0H84OZnf23Nwp5h8/VS2rui6HrGfg4xCOfOHt7IW CKTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873494; x=1788478294; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=63v8voqDYPPccdvhHdT2gYF/erUoDbYrARzDm1If6YE=; b=PhK7tJEyTcYvM4dBvC5uhzl4KOz0LJsY78jpRgt89GduaFTpCyl5a2DVM64OoUNyE5 IJiB8oqTRTW+rdyXyG4vmAoUAu43PnhTa3GXne7gJSHgvjMZiepgnZl8Uf1bHgaNKdHm KnT6GG929omQHBnN2VzsjOstCSEiABTJHm3jy3sp2vAckeEKW52rrNt/kYJjtDAWCTcy A98VjsP5T/VXhH8J6hWAZiK7BQZohYVtBQhrpWFvuqfBUj3xqYJBipUi4DLu3WuYyKuj MAvLmbyhS11HO+LxYqGuLA8+OsOOSP16a8USjY+KwhiXD6u0U/NiPJURmO0rHvi9T2v7 JFRA== X-Gm-Message-State: AFuF++nukQ52bUn17hK5q/Lpm/HDTJM81/F1B3BMdtmHswlutTr+x28B 73mheM0JmkdLbA5sI8Mbh5n0k2rp+mZkZ08ZNh2Z/Vat0PrIZS+yk3PhV/9FoTdVsbF19Hz4X5v u7ae8mvGfHTO7AQ== X-Received: from dybpd7.prod.google.com ([2002:a05:7301:4547:b0:327:dce2:b13c]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1850:b0:2d5:e3ce:3988 with SMTP id d9443c01a7336-2d74dddceebmr49959775ad.11.1787873494176; Thu, 27 Aug 2026 16:31:34 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:47 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-10-stevensd@google.com> Subject: [RFC 09/10] x86: Enable reclaimable stacks From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement top_of_blocked_task_stack() to get the stack pointer out of a thread_struct and enable HAVE_ARCH_RECLAIMABLE_STACK. Signed-off-by: David Stevens --- arch/x86/Kconfig | 1 + arch/x86/include/asm/processor.h | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index bdad90f210e4..652d5a9c0830 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -211,6 +211,7 @@ config X86 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_USERFAULTFD_MINOR if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_RECLAIMABLE_STACK if X86_64 select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/proces= sor.h index 87b1d4c0727e..207e7adced64 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -690,6 +690,11 @@ extern void start_thread(struct pt_regs *regs, unsigne= d long new_ip, #define GET_TSC_CTL(adr) get_tsc_mode((adr)) #define SET_TSC_CTL(val) set_tsc_mode((val)) =20 +static inline unsigned long top_of_blocked_task_stack(struct thread_struct= *thread) +{ + return thread->sp; +} + extern int get_tsc_mode(unsigned long adr); extern int set_tsc_mode(unsigned int val); =20 --=20 2.55.0.897.gb25b4bd76c-goog From nobody Sun Sep 27 00:39:09 2026 Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AA8083E716F for ; Thu, 27 Aug 2026 23:31:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873498; cv=none; b=AW7RsYkX1PHkXZ21PzrjU5UZCrScEKYwL72OqZ79vJE+aY2VCv8u4togX/b7mbndFG7RaD5545VdzaAJxbQornzzs9ldO157RY00kVhpZqVhYlcvjhN6Yjeeo64Fh6lkZoqrcRUqR37LHYqvDt1SpNbDmjlFDXTBePabKWZprR0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1787873498; c=relaxed/simple; bh=a4DHxg2eDawzpdV6ihnCiTe31A8NEYh6oPRQKJtlYWQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QKH/Y/w0+IoVhDqFGyg6SvLCHzS1c04SsENJi2MXCiQP0FnZSBLgKJ6aj+EhyVUqjVx0ql/aaBoSqkxTl/R8xRbGFZIB8x8IbyL9379ya5xQGt9aWbZPpVWutsvgyEUJpvBhud7C7/f+6kcB04mdjwyyhn8vAVfXwCnrq4e0ZaQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A5T2MugD; arc=none smtp.client-ip=209.85.216.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--stevensd.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A5T2MugD" Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-38e11baa66eso806939a91.2 for ; Thu, 27 Aug 2026 16:31:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1787873496; x=1788478296; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=THehtAVXPzXeOISNI+hN8Wh32bn5y3s9ZVH/HPQs5Ig=; b=A5T2MugDTl03wWg5tWQDSsVhfk/z/7vYj+q2emKo4Ui5SCgwX4lNbBpjXajyIN1m5J bjx8lr/bAUE1KeXjh7Yjd4suqFSb1Qh1Rm0mdI6cTtjE1X1ystwmAYxFpSzpvHiahupf F0/eVmNM/ozVYo4EawBWlsa3JWT7BRSzaTmwChpmdAM7TKSjDlfTpxuvfahUfLCDhK97 FEs9JdLeuNJmyuq218op2Gblpgs8cGYxqp/cV8GP9P2z6HO1xWDkXzZtDgSvucgAo20Q NxQILCdwiHUSAkAihQhQQhUICzXOELdBapF6TLJjuQBnvpaMDjw5ySayU+iM4D3EfTzG cWyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1787873496; x=1788478296; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=THehtAVXPzXeOISNI+hN8Wh32bn5y3s9ZVH/HPQs5Ig=; b=bkME7JtwiwEvH7aEaL3zesYKkebn1W3dUCeYeCDD5Ef3ZjyAnpYvRNvjVHF2upplVf fvGc+JyIMjbzyxPSqSd33gvv0KL+UWodAA+ew2JndGT14joxv1fzm0InfEJoBP3HZf8q n+3yGB+Asw/IsRAzolQ6GrsNrY1lQqBM1PnPhVHyxM2T2W+tvMgXkXAHvVOunXFKNs/7 wyzCRw/IvwzNyDIPfNULoljL+H14tgQNiFptluGGEhlj8fH64vcPxsvWBen3X/R3v83H UxbhqQwPTJtUdALtR5bZ3NOKKB97sZvsCCPS0IlgG+GPmS7JSKv4zCSUsFQ6zmVxCFHa leag== X-Gm-Message-State: AFuF++mFWiamVS7haJp5scqnLno30CPo61trWgSFaEm+1qg+ooKqKYXA SPii9/tZ353VArTEBHH5hZe8Xs5yYPF0Xgpgw4JJ/cyOPuSJRKfABm/BQW5ZbKSrHYc0fQlPkRu DEaR8GXz0bN5CFw== X-Received: from dlbsj13.prod.google.com ([2002:a05:7022:f90d:b0:139:d9c7:3cf4]) (user=stevensd job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2c88:b0:396:b98b:a3c2 with SMTP id 98e67ed59e1d1-396d0f52a0amr5303479a91.8.1787873495813; Thu, 27 Aug 2026 16:31:35 -0700 (PDT) Date: Thu, 27 Aug 2026 16:29:48 -0700 In-Reply-To: <20260827232948.2520558-1-stevensd@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260827232948.2520558-1-stevensd@google.com> X-Mailer: git-send-email 2.55.0.897.gb25b4bd76c-goog Message-ID: <20260827232948.2520558-11-stevensd@google.com> Subject: [RFC 10/10] arm64: Enable reclaimable stacks From: David Stevens To: Catalin Marinas , Will Deacon , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Andrew Morton , Dave Chinner , Qi Zheng , Roman Gushchin , Muchun Song , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Uladzislau Rezki , David Hildenbrand , Lorenzo Stoakes , "Liam R . Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko , Kees Cook , Sebastian Andrzej Siewior , Clark Williams , suleiman@google.com Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-rt-devel@lists.linux.dev, David Stevens Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Implement top_of_blocked_task_stack() to get the stack pointer out of a thread_struct and enable HAVE_ARCH_RECLAIMABLE_STACK. Signed-off-by: David Stevens --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/processor.h | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index b3afe0688919..262a4fddce19 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -173,6 +173,7 @@ config ARM64 select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE select HAVE_ARCH_VMAP_STACK + select HAVE_ARCH_RECLAIMABLE_STACK select HAVE_ARM_SMCCC select HAVE_ASM_MODVERSIONS select HAVE_EBPF_JIT diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/pr= ocessor.h index c2a627f39314..f5a550835c94 100644 --- a/arch/arm64/include/asm/processor.h +++ b/arch/arm64/include/asm/processor.h @@ -204,6 +204,11 @@ struct thread_struct { #endif }; =20 +static inline unsigned long top_of_blocked_task_stack(struct thread_struct= *thread) +{ + return thread->cpu_context.sp; +} + static inline unsigned int thread_get_vl(struct thread_struct *thread, enum vec_type type) { --=20 2.55.0.897.gb25b4bd76c-goog