From nobody Mon Feb 9 07:43:11 2026 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09D67481C0 for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175608; cv=none; b=ox8gfU5Zo/C1CgVEPagd+hGjpOSzkziiaATkR4/ewia+RV7ET4NHrG4444uBkO7N5ySJsaVbB36OsfeVCUpiv/w4WTgW0RJwuJKBegQn4oBaWq2NF0DfWyFrQuEsP6Q9y9gJqr6i+4ms7oYEBCzX/zq9tFqAUDltWwyNPxPjyaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175608; c=relaxed/simple; bh=7FN5G5YkH+G4Zy6dvl7Btv8IAMrVxXa8kxhlpjyjjxY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G0RTSqPrGkkJ4tvz5274M+ZC3gTqlVJBLx2UTppX6sbwKVwxwqjlwz0fUwQMsdukaXhMwF8UatUjX+8WNGY8xv4IKRSAlfwgzjc27ByPxOvskWASkAA3uzjNFFretQHvS95iNJcp+Bl6DgzIB4Vd8JNL2fX5yEXUgDrXRF1+bYY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=qcP3FLEO; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="qcP3FLEO" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3c2138463ccso2641871b6e.2 for ; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175606; x=1710780406; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=4lhjThGfB91n24i50IPCohz8k1KRXdDof+Lt/uTAI2U=; b=qcP3FLEOYiFk85fyifZUygn8vQ4KYq7ijS+tCWLqm1H9t0EL6cSOih34kpSpKYopaK +po+fpVXorOLnvZKzwpSJbVyaZ4yAA7vZLz35TrDyT3l/4lYLLDn7Boor7gdXNVYJFvQ ZJNylAPq9BKwoysoN57iGgmiffcye51RvpsqXbc57daLHHJk4Nq+Nr5WqbgeXuLsjiuQ dz/yYhJQdR2sQbdF/8oO7uhLy/BIRBWv2atkQ5tRRCrC6np+NeDSO/XCoLbDysBvP3Fv J7OPzMhIGhIaI0dzYei+5lX7mc71i/q3vD8qNcXWleBQDHbPMQyi5Qu1TRsKoqmrOLvH EQ3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175606; x=1710780406; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4lhjThGfB91n24i50IPCohz8k1KRXdDof+Lt/uTAI2U=; b=DfVDGdcerTEHdtTT97gbhvvF9aW3JcQARaMb/xYSaIOzKnXOrAFjmvqwOr0qJc+DKb zfsCfZeaxUtR95N0gLC6dHqgZahPrgJGiF9bfOX3mqOALkP5bHMYa5r0ZQ0hyRpZxEaV KsWdK9C2kivsh0vh/uiLRoTeQRnlUt2X7cXHT2TduEBYrt4aETjETlNEx3hhBvG7nSx9 h89NqJ3fYfROg1FaEng+czXs7A/WJ0K6gZjOUV+Wz5QzuekTEgpr6wNPkPUZ++yPSjA2 f7IH4nytMFnvWuEvxqRu1cwXic8IifU5ytY12yQW4ohiEs5j4+2teCuRhysAzRcsRvkb oq9A== X-Gm-Message-State: AOJu0YzI7I1PDEkJDouULGSbfVxANvXhbSAYeaMcTVDIqNJ3kLBSu5T8 BLdBoEdeYiNWm/ZdlHUD4SEqMmF4q0WBv+ZsrKIKXVESLiovQaDdmfnCcl9yiYcRKfvRbvjdyoN JV7w= X-Google-Smtp-Source: AGHT+IEjg4Cv5VbQxeFt499tiStDwxoTwSUC+FAOlU+MVjKFmojDB4eSoAwzE63CJMnyaPIAjMQShA== X-Received: by 2002:a05:6808:ecd:b0:3c2:39d1:f105 with SMTP id q13-20020a0568080ecd00b003c239d1f105mr8453793oiv.11.1710175604451; Mon, 11 Mar 2024 09:46:44 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:44 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 01/14] task_stack.h: remove obsolete __HAVE_ARCH_KSTACK_END check Date: Mon, 11 Mar 2024 16:46:25 +0000 Message-ID: <20240311164638.2015063-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove __HAVE_ARCH_KSTACK_END as it has been osolete since removal of metag architecture in v4.17. Signed-off-by: Pasha Tatashin --- include/linux/sched/task_stack.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_st= ack.h index ccd72b978e1f..860faea06883 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -116,7 +116,6 @@ static inline unsigned long stack_not_used(struct task_= struct *p) #endif extern void set_task_stack_end_magic(struct task_struct *tsk); =20 -#ifndef __HAVE_ARCH_KSTACK_END static inline int kstack_end(void *addr) { /* Reliable end of stack detection: @@ -124,6 +123,5 @@ static inline int kstack_end(void *addr) */ return !(((unsigned long)addr+sizeof(void*)-1) & (THREAD_SIZE-sizeof(void= *))); } -#endif =20 #endif /* _LINUX_SCHED_TASK_STACK_H */ --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-oa1-f49.google.com (mail-oa1-f49.google.com [209.85.160.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 883D547F6B for ; Mon, 11 Mar 2024 16:46:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175608; cv=none; b=fQmB/FI16LpToZ7muUDS6eEt3uVYpEh6XNhedV3YaP48NkGBXdENXYgy149NQrD0n9Fk7DUx1Wmpur3R4sQWE3Io+aIppEot9kQDgD6KEHu3I357qiFHliQ61gvfSdBceTf/mVOJwX1ib3XSUy7bK0zCNwGppT1etlRtyfUNmEg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175608; c=relaxed/simple; bh=rNSw66o+XpxAmDCLp/4Y0L0j3NJA6sLvc+VkxIH0y8g=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=egiqYDUeVTxKgLghmoEQRABkGhepRAbEZsJJcw0hhJVQHjWbdb7mQMkWpwGnve7UJd9OOPMT+u1++Ik569X1cg2cWUF8CGRDHqF6r4+Me+scsTD+zKBxBTTQuMulUVkzVFafOT4zGhHdtIMBiRKkPtRbdlcP0rLxmOekxea6z60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=NdFUqRgw; arc=none smtp.client-ip=209.85.160.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="NdFUqRgw" Received: by mail-oa1-f49.google.com with SMTP id 586e51a60fabf-22181888b88so3895624fac.1 for ; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175605; x=1710780405; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=uRKHBCRgEDsAkh8Uu5g0LvaNrGLtTcS4Fv1I3hMOqvE=; b=NdFUqRgwZzPkrAPEBG6Vly8/GajM6oy9t9Cc7v6qZZiUENLwc8xDKDA40eIsvDxLke KHWZ51VzZZWxK24XkQCe5NhpFSD+7fTad2xoa/+kGX6FyYn+HE1tuSL19AuX0tZ+DB1x wyjb9eaia04Eed/6QvxNFJvZ6K4JueagJ2LXxj0PQ/gspKhR9W+Cs6WvU8bnZXyoCu93 Dwq5RtFypq0rg02fJ60zJACeYkDT2Xi9XjQ/xkE91vQ/dzxCmC+gIR+1ZTY3/8fWbijw OtEASAdUhLVKvMGzcYX6uBadWCGgmHmADgiRQOMQvVFfDiT0Z4LEnfKE7EkKvWE8xvoQ YqOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175605; x=1710780405; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uRKHBCRgEDsAkh8Uu5g0LvaNrGLtTcS4Fv1I3hMOqvE=; b=pC9yqcGs/Rz6XEzSg8iARr1P9IrFZnZP2WzsPaBXZo2vp9FreyZuk7QhzaNj/aS243 cJ+GR67uAvZFe4x9s2T9qqRL378fw4qz4Nq8qITLx0TQd+ImWU3FSHOz9mwWG6aKWXi9 +q6hy5D2nSZoAu5/Ep6Jq1brqbXg6reO4Q6cDevPN9Pm5AqNyCUDn/hGbWklzNVBsh9f OvfMheXhVR/Ac4HTh904YAoULRMR31BdBavrvY2tfAITDcLMTli1eczMbGylyKhr4co5 XXj+xz5C329OLthP+gOjs+RRREChJ6J+waOUiywn3UOnOZFss4RH48BjVaAqF0jmWRS1 iGrg== X-Gm-Message-State: AOJu0Yzc9D20zGJrM/9rM6+lBSClgOxB//2frFdRsu9SnbMBp4catV9u xVdXtRp3u/7mHbUgU6ihs4GWQOeFAwVPrEUWSIalCqHuKqkhNg+XQ5Ge9ScLzvo9AhDfR4prn86 luRA= X-Google-Smtp-Source: AGHT+IGB8z9Eh2EiOSMQqnmRI9zZB59VQQwcHf8aThINC/aQYO0+ESVp0WJ0ORdeFBqZ+2a6LmE9PQ== X-Received: by 2002:a05:6870:4721:b0:21f:4fe:9ee8 with SMTP id b33-20020a056870472100b0021f04fe9ee8mr7898763oaq.4.1710175605333; Mon, 11 Mar 2024 09:46:45 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:45 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 02/14] fork: Clean-up ifdef logic around stack allocation Date: Mon, 11 Mar 2024 16:46:26 +0000 Message-ID: <20240311164638.2015063-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There is unneeded OR in the ifdef functions that are used to allocate and free kernel stacks based on direct map or vmap. Adding dynamic stack support would complicate this logic even further. Therefore, clean up by Changing the order so OR is no longer needed. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 0d944e92a43f..32600bf2422a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -179,13 +179,7 @@ static inline void free_task_struct(struct task_struct= *tsk) kmem_cache_free(task_struct_cachep, tsk); } =20 -/* - * Allocate pages if THREAD_SIZE is >=3D PAGE_SIZE, otherwise use a - * kmemcache based allocator. - */ -# if THREAD_SIZE >=3D PAGE_SIZE || defined(CONFIG_VMAP_STACK) - -# ifdef CONFIG_VMAP_STACK +#ifdef CONFIG_VMAP_STACK /* * vmalloc() is a bit slow, and calling vfree() enough times will force a = TLB * flush. Try to minimize the number of calls by caching stacks. @@ -337,7 +331,13 @@ static void free_thread_stack(struct task_struct *tsk) tsk->stack_vm_area =3D NULL; } =20 -# else /* !CONFIG_VMAP_STACK */ +#else /* !CONFIG_VMAP_STACK */ + +/* + * Allocate pages if THREAD_SIZE is >=3D PAGE_SIZE, otherwise use a + * kmemcache based allocator. + */ +#if THREAD_SIZE >=3D PAGE_SIZE =20 static void thread_stack_free_rcu(struct rcu_head *rh) { @@ -369,8 +369,7 @@ static void free_thread_stack(struct task_struct *tsk) tsk->stack =3D NULL; } =20 -# endif /* CONFIG_VMAP_STACK */ -# else /* !(THREAD_SIZE >=3D PAGE_SIZE || defined(CONFIG_VMAP_STACK)) */ +#else /* !(THREAD_SIZE >=3D PAGE_SIZE) */ =20 static struct kmem_cache *thread_stack_cache; =20 @@ -409,7 +408,8 @@ void thread_stack_cache_init(void) BUG_ON(thread_stack_cache =3D=3D NULL); } =20 -# endif /* THREAD_SIZE >=3D PAGE_SIZE || defined(CONFIG_VMAP_STACK) */ +#endif /* THREAD_SIZE >=3D PAGE_SIZE */ +#endif /* CONFIG_VMAP_STACK */ =20 /* SLAB cache for signal_struct structures (tsk->signal) */ static struct kmem_cache *signal_cachep; --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77229482C8 for ; Mon, 11 Mar 2024 16:46:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175609; cv=none; b=jxSn2S4elh1VQO+uCOB8vgPGvj4lP8R/j7qF0fcNwZGneOjSGOjvWyFMC6XlzM1IdG+ynqRBdJSMrtnajBdrkQTk/1RFH2Cb1nzodapsHFrbMMJzTSTmi6G9q8yi0cSPSl/5hzhOKUIcAUTW01Q0GU2pwS3F91gwVhWH5EyHKT0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175609; c=relaxed/simple; bh=tEvVrQdda52ZeoTZCZGpLyifsvZ4B54Au/IQTs3CyBY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ns6TQ6VKLgGQMZXreh3KxYS6XESyfsN7IQ2Jkz6cEZGNqdmKkwsCoo3bPNIsNA3cce4cWjuXeyGWjPp9kI7+FIgomeTIF9bCAtzEvcb0C+d8DPrT+EC2AZpIgx9PvKEEsADmVF8UmTo6xQ9XtCpk1dv2dRLNybeSPIHeZNIa9dQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=mQTT3A9v; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="mQTT3A9v" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7882e94d408so382722985a.0 for ; Mon, 11 Mar 2024 09:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175606; x=1710780406; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=HoufURnUHz+cr/acEvSc8ejQyNfvHWk1Ds3/A5XAtBc=; b=mQTT3A9vqwHQNYTdHWJygzw8tcaJfBI7IIbzYESS5ds1wk3CkSvRc3G8gYZ9Yx4xd1 WuLOFK3uodx7DUGJHqwoWZUn1yu5VNDonE6bIJWoqLcmoLHzFXYZp/86Z7kgxPetBGcV XieCbZKVad8qPUsUym6gVTPlcaebw15P/vhb63W66HG6KTxZPcKiCGsSAZ6hDQSByy/D Dlb/GG1dLT3OZZEM6p5ox/BMfIPyRU51Nd/FEa0uDZIEEv9Af1xPQzjuPtmlwhfbJRqV iOF9/0/J/8AmFNXlXSnmVl58R1RUOwRpta/wAkxbP/VbLlVlFn2KBmNuu1VWFNauAWD3 gfew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175606; x=1710780406; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HoufURnUHz+cr/acEvSc8ejQyNfvHWk1Ds3/A5XAtBc=; b=TisOJKYWBzP47zQWt6T5wEXj2uTE3sO+yudfuv4yR88anHIySym1jqr8aNu4EuU74H LouTTF3dMta3/oN0Tt2eFovZZ0BjdByjy/yR73AQAEiXbf68EBZGPYJD0LJQq8lR+cKg HzINZ8HRkCwLTs/pH2f/bdY9RsVL4Xl8sXM29K2N5HIQtLQoZLNcGWYAvt8LOtM/Rmtu zi4lPd7WMuNEXjUD9n5m7rw2qi1+/yrxyAdCQ2itNCsovqWj4yA5sBEncIyq/Ej5oh7b i7XUWEtzhqtR8h4udeQKc9046cbAT6JovM7xAtlX6bR0pO8Tvx4KRHUQMpAsMmEZJVUI gElA== X-Gm-Message-State: AOJu0Ywn9U/kqzVZV0qVOOh/cmbS9oNUG8uc34G4F6XOUQSUBnpuD4PF Z6UQkupAlM7aboc+Gl4WtBLp+atvczlZbNMMOwyfRE40bXFkBSbkLUCoJIr0C0tXHh35+N1CzFL 2gag= X-Google-Smtp-Source: AGHT+IGrhJ6GByyle2ZXj6YELCySZAA2hRd+lCiymK5OoO5mkfddy6eeNIr6RoEdx/yXV9YfoWqMCg== X-Received: by 2002:a05:620a:13cd:b0:788:3c2b:42e2 with SMTP id g13-20020a05620a13cd00b007883c2b42e2mr8567991qkl.10.1710175606139; Mon, 11 Mar 2024 09:46:46 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:45 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 03/14] fork: Clean-up naming of vm_strack/vm_struct variables in vmap stacks code Date: Mon, 11 Mar 2024 16:46:27 +0000 Message-ID: <20240311164638.2015063-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" There are two data types: "struct vm_struct" and "struct vm_stack" that have the same local variable names: vm_stack, or vm, or s, which makes code confusing to read. Change the code so the naming is consisent: struct vm_struct is always called vm_area struct vm_stack is always called vm_stack Signed-off-by: Pasha Tatashin --- kernel/fork.c | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 32600bf2422a..60e812825a7a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -192,12 +192,12 @@ struct vm_stack { struct vm_struct *stack_vm_area; }; =20 -static bool try_release_thread_stack_to_cache(struct vm_struct *vm) +static bool try_release_thread_stack_to_cache(struct vm_struct *vm_area) { unsigned int i; =20 for (i =3D 0; i < NR_CACHED_STACKS; i++) { - if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm) !=3D NULL) + if (this_cpu_cmpxchg(cached_stacks[i], NULL, vm_area) !=3D NULL) continue; return true; } @@ -207,11 +207,12 @@ static bool try_release_thread_stack_to_cache(struct = vm_struct *vm) static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack =3D container_of(rh, struct vm_stack, rcu); + struct vm_struct *vm_area =3D vm_stack->stack_vm_area; =20 if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; =20 - vfree(vm_stack); + vfree(vm_area->addr); } =20 static void thread_stack_delayed_free(struct task_struct *tsk) @@ -228,12 +229,12 @@ static int free_vm_stack_cache(unsigned int cpu) int i; =20 for (i =3D 0; i < NR_CACHED_STACKS; i++) { - struct vm_struct *vm_stack =3D cached_vm_stacks[i]; + struct vm_struct *vm_area =3D cached_vm_stacks[i]; =20 - if (!vm_stack) + if (!vm_area) continue; =20 - vfree(vm_stack->addr); + vfree(vm_area->addr); cached_vm_stacks[i] =3D NULL; } =20 @@ -263,32 +264,29 @@ static int memcg_charge_kernel_stack(struct vm_struct= *vm) =20 static int alloc_thread_stack_node(struct task_struct *tsk, int node) { - struct vm_struct *vm; + struct vm_struct *vm_area; void *stack; int i; =20 for (i =3D 0; i < NR_CACHED_STACKS; i++) { - struct vm_struct *s; - - s =3D this_cpu_xchg(cached_stacks[i], NULL); - - if (!s) + vm_area =3D this_cpu_xchg(cached_stacks[i], NULL); + if (!vm_area) continue; =20 /* Reset stack metadata. */ - kasan_unpoison_range(s->addr, THREAD_SIZE); + kasan_unpoison_range(vm_area->addr, THREAD_SIZE); =20 - stack =3D kasan_reset_tag(s->addr); + stack =3D kasan_reset_tag(vm_area->addr); =20 /* Clear stale pointers from reused stack. */ memset(stack, 0, THREAD_SIZE); =20 - if (memcg_charge_kernel_stack(s)) { - vfree(s->addr); + if (memcg_charge_kernel_stack(vm_area)) { + vfree(vm_area->addr); return -ENOMEM; } =20 - tsk->stack_vm_area =3D s; + tsk->stack_vm_area =3D vm_area; tsk->stack =3D stack; return 0; } @@ -306,8 +304,8 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) if (!stack) return -ENOMEM; =20 - vm =3D find_vm_area(stack); - if (memcg_charge_kernel_stack(vm)) { + vm_area =3D find_vm_area(stack); + if (memcg_charge_kernel_stack(vm_area)) { vfree(stack); return -ENOMEM; } @@ -316,7 +314,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) * free_thread_stack() can be called in interrupt context, * so cache the vm_struct. */ - tsk->stack_vm_area =3D vm; + tsk->stack_vm_area =3D vm_area; stack =3D kasan_reset_tag(stack); tsk->stack =3D stack; return 0; --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7B374AED1 for ; Mon, 11 Mar 2024 16:46:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175610; cv=none; b=LEnwbILsB8zX69Fg933aDWUHEvtORoiRvO5lZgh0LBexKC8D5es0YTb77c6wy/yM6ZjibyBrjSY4RnDT9TxlWyBQBD1aEZCY0zfOJSIlrNNxjWCOonqTlPc9yfaGtQZxBC7rD3QS/CPBqMOm8sGDamoC4KJi5jf+RAFF2qQyDEU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175610; c=relaxed/simple; bh=1rbaxunOVQPm7FPgRClladfLQq23xPc2MbtK28dseXQ=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VyzAAD1lS6sCJJnmxTk9sGNi2LEPCwGRxg3X/OXJ14DszkpFsYLW2FiU0kObGVksNAZF5ItxvbI0AqUo9DQbJUvrrHZRmZzATflur6M5MsLpTBHnRqgdiHwVqk/NXZTyro8gx0NwueXxaAplHuu8VrcUEG9M9ZCN4qpeSGe+EOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=q9Di0cQF; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="q9Di0cQF" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-7884114feb8so119905385a.1 for ; Mon, 11 Mar 2024 09:46:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175607; x=1710780407; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=RZjnTg3rcm4xEs2wuMm2S2zMsfw/GRvQpvq6ZsCVNYQ=; b=q9Di0cQFIiTo63gXIxSVQaGOI70w8l37/sKZzk4Af7Uh7nSl+Rx6ztOW/R2b0heH7K CxS9f6ldFRJRtXWZLXifv4jqSUEHtkY+yrVDognppkBJ6Zk17TWRDmdnvx+qUskpwo6O j6qBj1PLHtd2aWkvVwth14JPdOst4Q+LJAs/vtF1jkjueuuAL0xqkQD+hlXdUO5uH/PJ 4o6HmBjqeZ6bixvnWvKujRJszPjgzkyVaHscTs1WSFaWK5h1DbwIU7bpelCnOAHnIAB7 9iV0SSQRHNvVDFXIEWhK4HU1IXkVLDCw8IqjI6QxDuLokK7Y0OfY8vvHNM7xh52M18I0 2kew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175607; x=1710780407; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RZjnTg3rcm4xEs2wuMm2S2zMsfw/GRvQpvq6ZsCVNYQ=; b=WdMnAFXOaR8Ac5pX6ByVpg3/AgcFWHfBi9JSUD4aWFOn5egrHvc2GeGVdwov/u1//+ 4yK864H1q5SVDOIG17L8hQXD60sc8KLyhrJrgvF6PxIeXGNApRkEAULF4e/OKkterVmd knDGO7ZhdjmFE6U5pPLGGjCx1lbwzDCiamxC7qjLhQZFdOvRHvzc4ZpV8PasK0PZtZWu H/HmtKRY824ruu57tWvCJJ+mnvkAucN0nXpSiHCrdZk72Jlp8THB+k6uBrB3pIyEV89w vHVcHl3/ybg2NMgHWpI9HQl97RcSf3tR7pcRp+qHgWhSHqjQs3PHKRrHOhZncfVTu9uq S+BA== X-Gm-Message-State: AOJu0YzzWcnUu8VVO3zzeziBLmfhG0J6uLE7djB5Ayn03K96FwaiF3jy JoP7EWMPhwijfOGuwuw+nLe30RTVku35IfIf3H4tNTJDxtWuzHbmYqW3kmaJzhA4TtHTHwv6WJ2 WkbY= X-Google-Smtp-Source: AGHT+IGK3ZkaISv7/cXUwYINZRSYzKZD81aDWnU5gX/aNi31ERoz+HAdpfJ4DupGmG/DqWHaLyf2Ag== X-Received: by 2002:a05:620a:3908:b0:788:7507:6e24 with SMTP id qr8-20020a05620a390800b0078875076e24mr1874945qkn.3.1710175607288; Mon, 11 Mar 2024 09:46:47 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:47 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 04/14] fork: Remove assumption that vm_area->nr_pages equals to THREAD_SIZE Date: Mon, 11 Mar 2024 16:46:28 +0000 Message-ID: <20240311164638.2015063-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In many places number of pages in the stack is detremined via (THREAD_SIZE / PAGE_SIZE). There is also a BUG_ON() that ensures that (THREAD_SIZE / PAGE_SIZE) is indeed equals to vm_area->nr_pages. However, with dynamic stacks, the number of pages in vm_area will grow with stack, therefore, use vm_area->nr_pages to determine the actual number of pages allocated in stack. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 60e812825a7a..a35f4008afa0 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -243,13 +243,11 @@ static int free_vm_stack_cache(unsigned int cpu) =20 static int memcg_charge_kernel_stack(struct vm_struct *vm) { - int i; - int ret; + int i, ret, nr_pages; int nr_charged =3D 0; =20 - BUG_ON(vm->nr_pages !=3D THREAD_SIZE / PAGE_SIZE); - - for (i =3D 0; i < THREAD_SIZE / PAGE_SIZE; i++) { + nr_pages =3D vm->nr_pages; + for (i =3D 0; i < nr_pages; i++) { ret =3D memcg_kmem_charge_page(vm->pages[i], GFP_KERNEL, 0); if (ret) goto err; @@ -531,9 +529,10 @@ static void account_kernel_stack(struct task_struct *t= sk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { struct vm_struct *vm =3D task_stack_vm_area(tsk); - int i; + int i, nr_pages; =20 - for (i =3D 0; i < THREAD_SIZE / PAGE_SIZE; i++) + nr_pages =3D vm->nr_pages; + for (i =3D 0; i < nr_pages; i++) mod_lruvec_page_state(vm->pages[i], NR_KERNEL_STACK_KB, account * (PAGE_SIZE / 1024)); } else { @@ -551,10 +550,11 @@ void exit_task_stack_account(struct task_struct *tsk) =20 if (IS_ENABLED(CONFIG_VMAP_STACK)) { struct vm_struct *vm; - int i; + int i, nr_pages; =20 vm =3D task_stack_vm_area(tsk); - for (i =3D 0; i < THREAD_SIZE / PAGE_SIZE; i++) + nr_pages =3D vm->nr_pages; + for (i =3D 0; i < nr_pages; i++) memcg_kmem_uncharge_page(vm->pages[i], 0); } } --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA5CB52F78 for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175612; cv=none; b=IpsfxuPT7cTJdfZZL7BJGE98+Zp2Xz3VEH8ozSHgJmKcZt4LSJ1MTlWqVhVggSoUhDNlXBcZ3fmIE0ZYIC68yBHPlQM+rtZc0mjlxgPqMzwt6T4v37+JltYumEaH4hNi0uv0PlvCqbo5p0Gr/p9fQ0QwV/g0Z9xlXjd16TZwWh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175612; c=relaxed/simple; bh=Bwrt2lusa+A0c0rwtSSKV9SaUdudi3XCtrQeJ9sAzzY=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=flQ0+KkugVRzlbEEfk4ov+DsViFK2z2lxfPUoBF1HndTRK4O0yx3GSKnIiItjVHn8TjCLuBzZuwFXGKHmGoWJD6OSpgtndi13jNaFmAhOm84H552Cy7fsIowgvEzQLwAnEHYOCQeZJANiYzhzQskee1KnTorE8E78yQ6YQNcpSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=vm86Zclz; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="vm86Zclz" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3c19b7d9de7so3130554b6e.2 for ; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175610; x=1710780410; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=85s92hjYUuufTOzCHiSH6hattUyZa1ZbNxNbo2SWk2c=; b=vm86ZclzNyd2Fbu+RtCevAm+UgYz83V+wJeCW+RrM6gBlK6BKeX7hPOtcMOFGCfjaM iIQ3NTyodNV3ybGzMm1w+TMnKYqoKDrtEAuH0M/yFIrS4+gIgqNXu74ixN3oj+sUIvVJ n5n9+IYLd/e+TGjO8XVj381FrxYWib3UxdR5cyAXvKCLqJPVgU6bLaajmGG5s6gfXDAn ovCccboyx8X3TlYHgICj/MXNwCQm1+BSAO+PmeraixYgFYRhO7qKSqcolxYjvzEibaze gekl9A67eeSLT7kD+nYT0NDz8DijLfsI7ta05zazaICZqic9K1iH6cvHd+E09aey2Kxm Pmsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175610; x=1710780410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=85s92hjYUuufTOzCHiSH6hattUyZa1ZbNxNbo2SWk2c=; b=Ttd+OXpBlpXw19OS13PK8ITjsvSffXiVfnbQP8iN8airWNkkBmBJOfu2LSJOTIJm5M o8/XzL3ntSYVrYFnDcLhAZd56E4JPhbzR/ztJIcHs4xXD0Kf4w5R5LeM7x7LS4d3vwCw ZXlf8zDy3Ip9qAKveYDXCL8pF34RU76nksA6TudnDIywPyXzdUv7acnXkgechB9mRajk PZRT4z+dHkJmdzplsJecWvasg8U+BZvN01XgWUb3TNCW7hFhH7395KHawRUlBQAxJdW7 qAUoOkWoZCc27T9Jq6zwKvdr+bvdoP/4RLf2r3ft7jcO+TOucq80B9MnMlmkjzBXrsxR pvmQ== X-Gm-Message-State: AOJu0YwjiE1388Zn53UwCF3gNVWzdsihgpYSfivtkrw0ioTBFaSw5gFI FhYe8TPBOL5/6Grcq+bEAPve8j8D8syT4zy34Cb9boUBe1xfAiZ+EqZBsaDCtsb7LS6JQ5j9Ir1 kmCo= X-Google-Smtp-Source: AGHT+IF0sA8TBoTxey5vp022o4Baok5dPg7vNZ4pcZt1EEyuyh5oG+THj2Y1Wu2BsuZNYuYtVQC9/Q== X-Received: by 2002:a05:6808:15aa:b0:3c2:1262:e04a with SMTP id t42-20020a05680815aa00b003c21262e04amr8114898oiw.1.1710175608379; Mon, 11 Mar 2024 09:46:48 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:47 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 05/14] fork: check charging success before zeroing stack Date: Mon, 11 Mar 2024 16:46:29 +0000 Message-ID: <20240311164638.2015063-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" No need to do zero cahced stack if memcg charge fails, so move the charging attempt before the memset operation. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index a35f4008afa0..6a2f2c85e09f 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -271,6 +271,11 @@ static int alloc_thread_stack_node(struct task_struct = *tsk, int node) if (!vm_area) continue; =20 + if (memcg_charge_kernel_stack(vm_area)) { + vfree(vm_area->addr); + return -ENOMEM; + } + /* Reset stack metadata. */ kasan_unpoison_range(vm_area->addr, THREAD_SIZE); =20 @@ -279,11 +284,6 @@ static int alloc_thread_stack_node(struct task_struct = *tsk, int node) /* Clear stale pointers from reused stack. */ memset(stack, 0, THREAD_SIZE); =20 - if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); - return -ENOMEM; - } - tsk->stack_vm_area =3D vm_area; tsk->stack =3D stack; return 0; --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-oa1-f45.google.com (mail-oa1-f45.google.com [209.85.160.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6A0DF51C52 for ; Mon, 11 Mar 2024 16:46:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175611; cv=none; b=kKKtW1beQdzGOl++HXr7odiAQTHykt04jEppq54B4+eEv0jpn2HtddHxEBhgMJloiJzfUh0Rwx0KvNnEXhwWGFfZQ+5okjCDDVZlV4XCNJi4iZ/H4rLCp0oRkBFttBYWBOXYI+vtksK8fhmq5RCrkph5E3/P7OAhXQnRd0kYuOY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175611; c=relaxed/simple; bh=YWX2hBlKKgeraszO5aQgtTxYaDusNvEWibyBtdLpKUA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RHBhVlMEsw5jOkVnJejAqb7SvWNDdkxPJ8by6qBObi7r+Cvq0FckFF9BB317ZYR0B+o2lVGeQ17Pflb7lmMyuf6aotXcn+JB3OQmqRr1vWynbmXEkhPscomAVs2rkd721ap8UXJCwiTgeRcRiLb/rzozNC0hCgtiGYuGqMij+8g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=xWQwSztA; arc=none smtp.client-ip=209.85.160.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="xWQwSztA" Received: by mail-oa1-f45.google.com with SMTP id 586e51a60fabf-21eea6aab5eso2878629fac.0 for ; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175609; x=1710780409; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=WeoI5maDr3l2Ozw1ib0LWqoOcMWVTMRirMYG/aEeKfI=; b=xWQwSztAqzOTGlXTBdhkxdWcW1v1o5EuHnBdcTwB6z2pFCyutKVaca0gOxHjj74u2c a2Ksr6DsX90ki0Q+G85gofBtxAnRpJA8o4NOgzDJhZDcdcks5E1Zfnzz0M7xjWZB6T41 RyESUrjst9s2MuCWs35naXuLy7vhYt4VVerF0hr5wytJzFMPjhN8tFs3JYXy8tnigS9m mz0iHZDcLaha4BctL4lgKI69jpfQSeREdrdKQdJR3zu3f/z0o6Unw9sle5tR8ds6GZyZ fvwwWph6s9QAmcZHXPdNq8zZopHrkbrzx0Iip9A/iaFgPgoYJDNlbMWRomxHtZDxQfqG mELw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175609; x=1710780409; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WeoI5maDr3l2Ozw1ib0LWqoOcMWVTMRirMYG/aEeKfI=; b=g4oxT2V+lTBMxKLb7Jig+6nw+B1020DM5QsBJX3Ns3DjpM6XBk7C8WBwuHhPwrMNED KvB9qyO5+/gPLEoITAinKSrZVDEYVmaWwiItTLa4eAVB1Wra9aAknFgX9ajOjkIfs+OQ MXYToaWYqyZhTo5C3PRcKtocziJNE+3brIUGxyyZw4LH/5ekCYlPJKnP9JVURlwS9wAB ukiOsfSknqrWypb17uXc7tds1pZVTftIikh58BVxeVFhiH2HOE08EP5F4YtsIsF4EP7D THvQxC+T54PFfpHwQakt1aKp1iYyKWHmEC4NgMWdYdbmw0rAQKDyUuSteje46w0F+Qax xQzQ== X-Gm-Message-State: AOJu0YxdcVa0mnQbPYbqbtjf2/ZFpWr++Dq05P4siiV0mc5sSh60D6ZL JSv8/yV3LGUceqrXxCjwsMfGE3n9zFcW9PsGu3nepzWpMk8CY9rs80ywP/5Viv+/PZyDHxY5VNA P+tk= X-Google-Smtp-Source: AGHT+IF7PMDKY7qPBLzKisHlCeRFB7P9dZ8FJtUn1b4P6yLcZu1GLHtuFZbDu9s9LwmUDKjKkIy2MQ== X-Received: by 2002:a05:6870:3307:b0:221:a43d:60ef with SMTP id x7-20020a056870330700b00221a43d60efmr8059130oae.3.1710175609309; Mon, 11 Mar 2024 09:46:49 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:49 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 06/14] fork: zero vmap stack using clear_page() instead of memset() Date: Mon, 11 Mar 2024 16:46:30 +0000 Message-ID: <20240311164638.2015063-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preporation for dynamic kernel stacks do not zero the whole span of the stack, but instead only the pages that are part of the vm_area. This is because with dynamic stacks we might have only partially populated stacks. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 6a2f2c85e09f..41e0baee79d2 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -263,8 +263,8 @@ static int memcg_charge_kernel_stack(struct vm_struct *= vm) static int alloc_thread_stack_node(struct task_struct *tsk, int node) { struct vm_struct *vm_area; + int i, j, nr_pages; void *stack; - int i; =20 for (i =3D 0; i < NR_CACHED_STACKS; i++) { vm_area =3D this_cpu_xchg(cached_stacks[i], NULL); @@ -282,7 +282,9 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) stack =3D kasan_reset_tag(vm_area->addr); =20 /* Clear stale pointers from reused stack. */ - memset(stack, 0, THREAD_SIZE); + nr_pages =3D vm_area->nr_pages; + for (j =3D 0; j < nr_pages; j++) + clear_page(page_address(vm_area->pages[j])); =20 tsk->stack_vm_area =3D vm_area; tsk->stack =3D stack; --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 833FB53812 for ; Mon, 11 Mar 2024 16:46:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175613; cv=none; b=F9+CmXZ3D2pet2fU+1QKd8SDqr/OSXkJKLboR1oXQrscKggffNKQ94Lu6gCwVSL78aOjiMnJaHsri5NV8O3hKM3AYRmtLFPI/8rxillM00GtIHayAA2FFgjttcH0g/Rvd15jXjFHoNzcoOKx1Qhcsq2FbSGccRxOCSyje8Ja1Vo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175613; c=relaxed/simple; bh=8aZV0c0PfVdZg6NDBVhtRXWK2FtIZwb93neqMuTW6Ps=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ayRgl8R+nvu6kV/U26Z8/3/39GM6O/C4jOym58HBLOSv7KUg2uoj0Os+3zTu5PJAbTasrq+X7dARLLioPKbZG5H655pZ00BVKGVzuzKCXMIWu+ZX4aeg3osW7nK1ll8slcPvVAw95oJ4fzKWWq1lwcLoHUIPu3isBMk+TS3B668= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=x4cYnBIT; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="x4cYnBIT" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7810827e54eso353132285a.2 for ; Mon, 11 Mar 2024 09:46:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175610; x=1710780410; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=rWA/mIcrC8mrdgOBIz97cH4Ahtr8zqihEDBDcy8+bis=; b=x4cYnBIT/IQbSylpX3UpfrfFC/Z72RHUFvVFLGZqwp/J4vb7SmcvaFqrwNiUb3aPRf /Rbr2D2lm/3xJSGzHcZrbkSkVCU01j/H39sKYNJjohOBKok/2scoPLXuliNpzobfHzcC oBsF6QIj1zf6NvKd/Wj899qN+g73AduprtaHmXXUB8GRCgu7MQsoVDw/QJ/8gxeLSzso nzOPrmOiWnmDgHEQ6D+0WbfUbfpg+59e0TZYDHHYoWmSjDUHnoC64ntdqFpZ38HNzKC1 +Dn1WzNZczvRmROOcT87+XwtlRV9v0d+WzJclITHEJiYpA/a+aNKakcHllbIfMKgNpBU d0VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175610; x=1710780410; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rWA/mIcrC8mrdgOBIz97cH4Ahtr8zqihEDBDcy8+bis=; b=eQonPAcciYgh1JGLPKsqtgAap6jFzQFpjHHwWhFDl1J3PQygAnhReZHq0PGL2aRts+ /+6fCIeDm85P5PcvmYljwUPCBlb50DpaDyLe7Uc3o6RiQSpleGE+boiC+YQa6qPar17V AVo/fCd09RMbgBEMkmiUrMMeX6xliG92V/ZpAyrg0gG5A+HqjqyDEakjGEvUXUJLvUT1 XW1EOjPFFFr9dIHwKINQtQY16XNRVJnJGtXp2FNWBmO7kcSnSJkEBSYgQbcX+RBUTsvU DnTBWr66D+cnF680FiQyCoh5LKdYTyLro1kdRZ7EXtZqvPzrFy37sN2HOQSNIhp0h+HH TW6A== X-Gm-Message-State: AOJu0Yx0A4G2CKIzuV7zIx95Lbl5BKAFORfS2eFVGJItIc0fg7kSceXH MQwmsolFpW6s9CoXiIsAZAPqJfSnkQ6sppboaNHJQckrYfV8Zsv3utnvi7ysp9SN6pK7nvBkooD IwZk= X-Google-Smtp-Source: AGHT+IEiaWGNP1LEtKt9OxZe9a0Ecsughs1hTC155ICY2/6rd+mLJB7PdZHKC0eSeLtAUcuC8VPmXQ== X-Received: by 2002:a05:620a:222a:b0:788:61ee:7fee with SMTP id n10-20020a05620a222a00b0078861ee7feemr6338288qkh.61.1710175610089; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:49 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 07/14] fork: use the first page in stack to store vm_stack in cached_stacks Date: Mon, 11 Mar 2024 16:46:31 +0000 Message-ID: <20240311164638.2015063-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" vmap stack are stored in a per-cpu cache_stacks in order to reduce number of allocations and free calls. However, the stacks ared stored using the buttom address of the stack. Since stacks normally grow down, this is a problem with dynamic stacks, as the lower pages might not even be allocated. Instead of the first available page from vm_area. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 41e0baee79d2..3004e6ce6c65 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -217,9 +217,10 @@ static void thread_stack_free_rcu(struct rcu_head *rh) =20 static void thread_stack_delayed_free(struct task_struct *tsk) { - struct vm_stack *vm_stack =3D tsk->stack; + struct vm_struct *vm_area =3D tsk->stack_vm_area; + struct vm_stack *vm_stack =3D page_address(vm_area->pages[0]); =20 - vm_stack->stack_vm_area =3D tsk->stack_vm_area; + vm_stack->stack_vm_area =3D vm_area; call_rcu(&vm_stack->rcu, thread_stack_free_rcu); } =20 --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 948305472A for ; Mon, 11 Mar 2024 16:46:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175614; cv=none; b=LhORN8AiGhVs5srBwctx+VN01LvmDFYwbbVOEweTzV0CCCD5DAUU+QmuZ4DMLyiXiszAChuKympQOn0Ql5AQhNgbavvlk6gDbFRhNXxL6lbwajp38J38VtU2rkpn/yoZOx05fGC3zCME/MN8hYAeAOJke9AUFPTWrCotUjJsvzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175614; c=relaxed/simple; bh=/nr7aaEAvjVHOUDe35kqc9+BWm5yZt4eYB5LCI8io0Q=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CdjypVI2aEqRA+VWkvpYkJbmF9D+HmRA6MNwO8yXxlmJe5Q+YArVH0i4W4K1DgzwJwMv5ZRwiYLVhjJc+56+k7rAMhJJcg96dRv0tjc5gDaRq+8UySmI7B5rZVaDMMV7pWLEs86zn/GGlMvrMaYfSnHitd95QeNo+tEBmCC+u2k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=2P8Bdxfd; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="2P8Bdxfd" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-78852e1e923so96445285a.3 for ; Mon, 11 Mar 2024 09:46:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175611; x=1710780411; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=2P8BdxfdiCm33uclKiECOzNcNUfb4Y8S/DMlHKAotrq30fVsjwhZghrWJjwm+s+RwP QQ0iJfAUbbjHzoq9oiyRbUFVbDjW3n3kA8xOiXewnDseUfyGBgKaDY9Vmg5Xq6fa67lJ M9Cm+5AcOYIgT/4KKdi6LccefhVIvAAcIxy/66ZdMJO5IUmCDXn/yl93sIRuo7a8B8wW zLMizkgG7tlWnVjLa4CYGt6lOlUtzkAzem5OC98X/TnbOh7uXNjkW4Fqh8bVipUY6m00 Qs/XrfumncsdaZovg3xIaGIo4KOo7/cMZE4UhpvSH/I3O+em5ZAMTbz6I/2Sc6wLWIJd TuyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175611; x=1710780411; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hkZD7ALZVm0P5UALUItr672MRcA6vth2fJOAHe6U29Q=; b=nOrNF3JYOwtwIlSEGCr9ExBBLgExOjEJurWLJYAh/egYbfpAjMLEBs2/G3VVl6vUE+ Vq1K68+WrCl830tqsG+TPFzFCrQw50AIk6pqSVPS0GbufUtPstf5SwAbB4EYdkwdyxUG NKMMsKHujcSOirlEjFjDMC0K004yA7QTaqV0fwONQS6ByprJN6dEu8LoxXZ8tOpL4/S6 upCbsjfQ42QVA0OyQr46fL4Qyh2hbdXPYjEgjmVchpuyVe/0Ei/Tjs8Y+ZfwMBGe/td9 2lWZkEGFxujgbea1PYPATx6YyZjtdogNbMuzs4AHAT/oGl+HffqxNoFpYMCyBrou+Gk3 2LZQ== X-Gm-Message-State: AOJu0YwK3TFzbsrkPXDndpxE52ZapaJfikRriT2/A+vPA7FBPVfWQKDq cDDQzh8kQRt3fa3rANgLx2t8p578wVwE0EKiq+7BdI/AhOsO1eg6XCBkbcyilunpOprpDTT9Qyq SvLA= X-Google-Smtp-Source: AGHT+IHTkS+2s/WrBF0S6nD4DTbIdKAkKUkplrTiXcc/c/8vBDolvcc8zD3FtIujCYDfupcwREXrKQ== X-Received: by 2002:a05:620a:15bc:b0:788:50aa:35f7 with SMTP id f28-20020a05620a15bc00b0078850aa35f7mr7780372qkk.30.1710175610969; Mon, 11 Mar 2024 09:46:50 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:50 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 08/14] fork: separate vmap stack alloction and free calls Date: Mon, 11 Mar 2024 16:46:32 +0000 Message-ID: <20240311164638.2015063-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In preparation for the dynamic stacks, separate out the __vmalloc_node_range and vfree calls from the vmap based stack allocations. The dynamic stacks will use their own variants of these functions. Signed-off-by: Pasha Tatashin --- kernel/fork.c | 53 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/kernel/fork.c b/kernel/fork.c index 3004e6ce6c65..bbae5f705773 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -204,6 +204,29 @@ static bool try_release_thread_stack_to_cache(struct v= m_struct *vm_area) return false; } =20 +static inline struct vm_struct *alloc_vmap_stack(int node) +{ + void *stack; + + /* + * Allocated stacks are cached and later reused by new threads, + * so memcg accounting is performed manually on assigning/releasing + * stacks to tasks. Drop __GFP_ACCOUNT. + */ + stack =3D __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, + VMALLOC_START, VMALLOC_END, + THREADINFO_GFP & ~__GFP_ACCOUNT, + PAGE_KERNEL, + 0, node, __builtin_return_address(0)); + + return (stack) ? find_vm_area(stack) : NULL; +} + +static inline void free_vmap_stack(struct vm_struct *vm_area) +{ + vfree(vm_area->addr); +} + static void thread_stack_free_rcu(struct rcu_head *rh) { struct vm_stack *vm_stack =3D container_of(rh, struct vm_stack, rcu); @@ -212,7 +235,7 @@ static void thread_stack_free_rcu(struct rcu_head *rh) if (try_release_thread_stack_to_cache(vm_stack->stack_vm_area)) return; =20 - vfree(vm_area->addr); + free_vmap_stack(vm_area); } =20 static void thread_stack_delayed_free(struct task_struct *tsk) @@ -235,7 +258,7 @@ static int free_vm_stack_cache(unsigned int cpu) if (!vm_area) continue; =20 - vfree(vm_area->addr); + free_vmap_stack(vm_area); cached_vm_stacks[i] =3D NULL; } =20 @@ -265,7 +288,6 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) { struct vm_struct *vm_area; int i, j, nr_pages; - void *stack; =20 for (i =3D 0; i < NR_CACHED_STACKS; i++) { vm_area =3D this_cpu_xchg(cached_stacks[i], NULL); @@ -273,14 +295,13 @@ static int alloc_thread_stack_node(struct task_struct= *tsk, int node) continue; =20 if (memcg_charge_kernel_stack(vm_area)) { - vfree(vm_area->addr); + free_vmap_stack(vm_area); return -ENOMEM; } =20 /* Reset stack metadata. */ kasan_unpoison_range(vm_area->addr, THREAD_SIZE); - - stack =3D kasan_reset_tag(vm_area->addr); + tsk->stack =3D kasan_reset_tag(vm_area->addr); =20 /* Clear stale pointers from reused stack. */ nr_pages =3D vm_area->nr_pages; @@ -288,26 +309,15 @@ static int alloc_thread_stack_node(struct task_struct= *tsk, int node) clear_page(page_address(vm_area->pages[j])); =20 tsk->stack_vm_area =3D vm_area; - tsk->stack =3D stack; return 0; } =20 - /* - * Allocated stacks are cached and later reused by new threads, - * so memcg accounting is performed manually on assigning/releasing - * stacks to tasks. Drop __GFP_ACCOUNT. - */ - stack =3D __vmalloc_node_range(THREAD_SIZE, THREAD_ALIGN, - VMALLOC_START, VMALLOC_END, - THREADINFO_GFP & ~__GFP_ACCOUNT, - PAGE_KERNEL, - 0, node, __builtin_return_address(0)); - if (!stack) + vm_area =3D alloc_vmap_stack(node); + if (!vm_area) return -ENOMEM; =20 - vm_area =3D find_vm_area(stack); if (memcg_charge_kernel_stack(vm_area)) { - vfree(stack); + free_vmap_stack(vm_area); return -ENOMEM; } /* @@ -316,8 +326,7 @@ static int alloc_thread_stack_node(struct task_struct *= tsk, int node) * so cache the vm_struct. */ tsk->stack_vm_area =3D vm_area; - stack =3D kasan_reset_tag(stack); - tsk->stack =3D stack; + tsk->stack =3D kasan_reset_tag(vm_area->addr); return 0; } =20 --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A5E0654FA2 for ; Mon, 11 Mar 2024 16:46:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175615; cv=none; b=bxoA8lqi8AU4NaMffJL237HyFuaKpEajNVQ5pq4wmLbMfk+Cnf9KjUWxYvwX14Wt9BblSECMGooWXybPtsEuFOw3reuyNpDn/LDNZUdKKaHvXLvChjHB45Maz8VaXcsxTHCWU73JJaSV45mhswUzi/OvpE8Y8UDWN84UEMamqlI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175615; c=relaxed/simple; bh=J5REF2bzCzAX7O0LOHW14nTnyl8YGMv3zounUf7qQr4=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Rrr3D6t5kuX9VikkA7oYAw55Mk0Z/VFpJ+EJfKBvzyOABDyINM/cl8n9oO7LAWDWCxBZ8maxfMh6FsNj6lNNYBQo+6qRHP1vEukDn3A+zmN9U9lJW5jyD8KFbtPAiAk10R3j/szE1oScfyC04C/gjsL1HYOvx3oaG2grMKSSnJk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=SUzPNr44; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="SUzPNr44" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-78850ab4075so120664885a.1 for ; Mon, 11 Mar 2024 09:46:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175612; x=1710780412; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=UUTioQgK+VFwu4bHGNUstSWJvzqUc2xbAOy5DzulCE4=; b=SUzPNr44BG72fCi5CjWAI1V5z26hOG3O70z6dpgZmoI0Hie9Bb5RZ/y/i+WpfZcpDc zqn15CuFqqma3nQj2QF2/9bV/iowcKNpwOgPKZ9n4u/UP6q2Yk9n/akcXS4zbhIT9uGe nFAgmY57GPDejAT3ytqEbDRKNcfbZ3ii+D9kTwSrZlva/a02KZRvG1F9yQCU3ywJv20s j7IRWwDmDep/P6Gw7ikDRoVbq/KuauaZw+MvlY3WQS4w+RMM2omI38z2Jnsj2SKOiqL8 pVEggM2qsy1yrmgr/yOanBVLaasw0ZT6gjEEqvkK1GVbRO88DxHb1XQknFcU9g3YawRG Gq9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175612; x=1710780412; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UUTioQgK+VFwu4bHGNUstSWJvzqUc2xbAOy5DzulCE4=; b=UVdzOxkA9jPXIgkTAZ571SIhTm+4JpBEu+GnB2mzCNerWr5oxy1/QHIh/ekg6/Wo4u lakMk0sMIY06T7ldzMkj5iF42yKH9DPBxkp/dZoaxD/ra+EByw52tnld/K5ZmUlY3S4c sHzWyn3BsxxcPIGuqEujQcwsm0NDH79Wmj5SvEfyCCSRn0cIbWQ+n3MDILZCqcLH7Hkc pmjvId48GuCGTGo3J4JF7pLtnFOb5QmAjExSvwFIpgBULCOonKZ9kPseO5WTupTrfIph gMoAaWXXIm558ktBYyhpbJzz84oTGRtE1iEpZ8mtZKwZ0d/WsVQuLcgu55+Fz1DFuI3n MZqg== X-Gm-Message-State: AOJu0YxFB249T7CnbwaD3MXFWWTn2wzvYaqIU4itBsMkDFCAfrx2q11S D7oAqsltxi7kD/L2AHC3JxaC6/XHdLrsrnR8N7eDZvoen0Hbqhr8L9grIDb1QbepwBPcoA+FHjY irJE= X-Google-Smtp-Source: AGHT+IF4clsrNYnhAmKVCQ5b1CkyUTVuvR4XaNSojXhA81ENz+LMIfO1rJKaTKEeTeVkpkuetWxx/w== X-Received: by 2002:a05:620a:5d9a:b0:788:5909:8ce5 with SMTP id xx26-20020a05620a5d9a00b0078859098ce5mr1085134qkn.34.1710175612342; Mon, 11 Mar 2024 09:46:52 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:51 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 09/14] mm/vmalloc: Add a get_vm_area_node() and vmap_pages_range_noflush() public functions Date: Mon, 11 Mar 2024 16:46:33 +0000 Message-ID: <20240311164638.2015063-10-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" get_vm_area_node() Unlike the other public get_vm_area_* variants, this one accepts node from which to allocate data structure, and also the align, which allows to create vm area with a specific alignment. This call is going to be used by dynamic stacks in order to ensure that the stack VM area of a specific alignment, and that even if there is only one page mapped, no page table allocations are going to be needed to map the other stack pages. vmap_pages_range_noflush() Is already a global function, but was exported through mm/internal.h, since we will need it from kernel/fork.c in order to map the initial stack pages, move the forward declaration of this function to the linux/vmalloc.h header. Signed-off-by: Pasha Tatashin --- include/linux/vmalloc.h | 15 +++++++++++++++ mm/internal.h | 9 --------- mm/vmalloc.c | 24 ++++++++++++++++++++++++ 3 files changed, 39 insertions(+), 9 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index c720be70c8dd..e18b6ab1584b 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -210,6 +210,9 @@ extern struct vm_struct *__get_vm_area_caller(unsigned = long size, unsigned long flags, unsigned long start, unsigned long end, const void *caller); +struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align, + unsigned long flags, int node, gfp_t gfp, + const void *caller); void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); @@ -241,10 +244,22 @@ static inline void set_vm_flush_reset_perms(void *add= r) vm->flags |=3D VM_FLUSH_RESET_PERMS; } =20 +int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned lon= g end, + pgprot_t prot, struct page **pages, + unsigned int page_shift); + #else static inline void set_vm_flush_reset_perms(void *addr) { } + +static inline +int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned lon= g end, + pgprot_t prot, struct page **pages, + unsigned int page_shift) +{ + return -EINVAL; +} #endif =20 /* for /proc/kcore */ diff --git a/mm/internal.h b/mm/internal.h index f309a010d50f..ba1e2ce68157 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -969,19 +969,10 @@ size_t splice_folio_into_pipe(struct pipe_inode_info = *pipe, */ #ifdef CONFIG_MMU void __init vmalloc_init(void); -int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages, unsigned int page_shif= t); #else static inline void vmalloc_init(void) { } - -static inline -int __must_check vmap_pages_range_noflush(unsigned long addr, unsigned lon= g end, - pgprot_t prot, struct page **pages, unsigned int page_shif= t) -{ - return -EINVAL; -} #endif =20 int __must_check __vmap_pages_range_noflush(unsigned long addr, diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d12a17fc0c17..7dcba463ff99 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2650,6 +2650,30 @@ struct vm_struct *get_vm_area_caller(unsigned long s= ize, unsigned long flags, NUMA_NO_NODE, GFP_KERNEL, caller); } =20 +/** + * get_vm_area_node - reserve a contiguous and aligned kernel virtual area + * @size: size of the area + * @align: alignment of the start address of the area + * @flags: %VM_IOREMAP for I/O mappings + * @node: NUMA node from which to allocate the area data structure + * @gfp: Flags to pass to the allocator + * @caller: Caller to be stored in the vm area data structure + * + * Search an area of @size/align in the kernel virtual mapping area, + * and reserved it for out purposes. Returns the area descriptor + * on success or %NULL on failure. + * + * Return: the area descriptor on success or %NULL on failure. + */ +struct vm_struct *get_vm_area_node(unsigned long size, unsigned long align, + unsigned long flags, int node, gfp_t gfp, + const void *caller) +{ + return __get_vm_area_node(size, align, PAGE_SHIFT, flags, + VMALLOC_START, VMALLOC_END, + node, gfp, caller); +} + /** * find_vm_area - find a continuous kernel virtual area * @addr: base address --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD91155E7B for ; Mon, 11 Mar 2024 16:46:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175616; cv=none; b=RCLL5grTeQitKF94x0Z94rBmqgLkIdJh8aUjMhCuE5iWyDfxGDKoRYEOkPnqHVP0VnygsIB9pYzgblVD/4Ksx9pe2Xtw22DAp4wjeg4oxElYll5e5AeqvYSzpnBP8xwKrp9jB5QlbU1TAY25qkXaldcHURrM6rD1roBA2vDJpO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175616; c=relaxed/simple; bh=UoabulEd/vJ9rmGjs45W0TPYHtt56amogSUkndFOzq0=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LLY8ObMmbE1if1+QlH2DSKkwTGlcmV2bhei6qk7Zm28uKWZeb4R1m9niJfK9VtWu3m2HmX+fSdbdAadcipcQKwBZvVyRqDGYGcJWncq90tgaGT9fczlkCoQtDS+opZNbBm2au9Py7l2xAm+c74k+0HyrSELwQFtZkvM+tzOgkYM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=uM/C8rdQ; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="uM/C8rdQ" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7884a9a404fso246385685a.3 for ; Mon, 11 Mar 2024 09:46:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175613; x=1710780413; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2I5VvwnC3LRR3szeP871B8eKL/IZGdjRShD9rinhwbY=; b=uM/C8rdQg22GL8jKVPmnOcUNRpKwGmzX1j3q0v9A3NcQj4s+IIhBskPIlScL2dBSGg SPn6pdmhSJWQ8ApHnyoxuGwmuay2R5DHbSbQy5Roox7a5UyVCMtcXZ/v32VyXzuWRKdp /9aQG5/G+4BP0gWmpsN2MA3W1Vuoja/s7V50gs30hqlmO/dKtaiEaN0E5+/B/O05AhJA igUlJIm1j1IcpIrv57dCAnkeXTX2ZDDp+1iGAZiXYjrRbKINyKSvIelCN0856v/gnpnS h4HEXEB5Q8e2+D/Sa+YSP6/GGe7eJgijRabMLf9jTPdHabuoBuoqj9dtTq2bORdbu0SV exXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175613; x=1710780413; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2I5VvwnC3LRR3szeP871B8eKL/IZGdjRShD9rinhwbY=; b=iE3GMVKeQsW9vT+IpqBYRhtL1awHmgTFR9gNd1RiIAL9ACenYrJ4eDcyn2mRaIsYmB QQb6Etou5GJVIzRSUkqUu7NTIyN+zxvHkl0Y1hboiX+RY53JXdu/+zwXeg5QaIx9i38d qUwTYvxt70DJz2YgvOLYI+I7m0x5qgilE2y1rUyYYTcop4v3hCAwwNHUXiQA9vNS3HZ+ IQja2l2Xt0L1bmAlZqPhKOlwcjJLxRGPVZxsf6vNWv44tCL6DmVUc+WuefbNDJQPVOOu s2VuQNqhAfByuOHQonPBEHKNDOL85/XEuEuI1mWtzOZl0Ne4F1WbKYmOzKqa6aW88nu4 Wyiw== X-Gm-Message-State: AOJu0Yy9OlOe5joEFEmfW0ppKmStWQsHRHeK3I8rhjhsZJws0YcqWdaf MxNAfCVBq1tytMX2AcLrxzVFPN88ozUCwzUUHZTfu2CrSlwzZqaYG/L/E1oVAllooWL1g1tLJud R5co= X-Google-Smtp-Source: AGHT+IH9a5LZ+7yIW2SDInIoVHV8enodZiTDdoz2gyXsxlHKxaxJ5G55cFQu5tNI4LZq2rv5sPx5sw== X-Received: by 2002:a05:620a:1a08:b0:788:4101:3d2 with SMTP id bk8-20020a05620a1a0800b00788410103d2mr10720214qkb.1.1710175613353; Mon, 11 Mar 2024 09:46:53 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:52 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 10/14] fork: Dynamic Kernel Stacks Date: Mon, 11 Mar 2024 16:46:34 +0000 Message-ID: <20240311164638.2015063-11-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The core implementation of dynamic kernel stacks. Unlike traditional kernel stacks, these stack are auto-grow as they are used. This allows to save a significant amount of memory in the fleet environments. Also, potentially the default size of kernel thread can be increased in order to prevent stack overflows without compromising on the overall memory overhead. The dynamic kernel stacks interface provides two global functions: 1. dynamic_stack_fault(). Architectures that support dynamic kernel stacks, must call this function in order to handle the fault in the stack. It allocates and maps new pages into the stack. The pages are maintained in a per-cpu data structure. 2. dynamic_stack() Must be called as a thread leaving CPU to check if the thread has allocated dynamic stack pages (tsk->flags & PF_DYNAMIC_STACK) is set. If this is the case, there are two things need to be performed: a. Charge the thread for the allocated stack pages. b. refill the per-cpu array so the next thread can also fault. Dynamic kernel threads do not support "STACK_END_MAGIC", as the last page is does not have to be faulted in. However, since they are based of vmap stacks, the guard pages always protect the dynamic kernel stacks from overflow. The average depth of a kernel thread depends on the workload, profiling, virtualization, compiler optimizations, and driver implementations. Therefore, the numbers should be tested for a specific workload. From my tests I found the following values on a freshly booted idling machines: CPU #Cores #Stacks Regular(kb) Dynamic(kb) AMD Genoa 384 5786 92576 23388 Intel Skylake 112 3182 50912 12860 AMD Rome 128 3401 54416 14784 AMD Rome 256 4908 78528 20876 Intel Haswell 72 2644 42304 10624 On all machines dynamic kernel stacks take about 25% of the original stack memory. Only 5% of active tasks performed a stack page fault in their life cycles. Signed-off-by: Pasha Tatashin --- arch/Kconfig | 34 +++++ include/linux/sched.h | 2 +- include/linux/sched/task_stack.h | 41 +++++- kernel/fork.c | 239 +++++++++++++++++++++++++++++++ kernel/sched/core.c | 1 + 5 files changed, 315 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index a5af0edd3eb8..da3df347b069 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1241,6 +1241,40 @@ config VMAP_STACK backing virtual mappings with real shadow memory, and KASAN_VMALLOC must be enabled. =20 +config HAVE_ARCH_DYNAMIC_STACK + def_bool n + help + An arch should select this symbol if it can support kernel stacks + dynamic growth. + + - Arch must have support for HAVE_ARCH_VMAP_STACK, in order to handle + stack related page faults + + - Arch must be able to faults from interrupt context. + - Arch must allows the kernel to handle stack faults gracefully, even + during interrupt handling. + + - Exceptions such as no pages available should be handled the same + in the consitent and predictable way. I.e. the exception should be + handled the same as when stack overflow occurs when guard pages are + touched with extra information about the allocation error. + +config DYNAMIC_STACK + default y + bool "Dynamically grow kernel stacks" + depends on THREAD_INFO_IN_TASK + depends on HAVE_ARCH_DYNAMIC_STACK + depends on VMAP_STACK + depends on !KASAN + depends on !DEBUG_STACK_USAGE + depends on !STACK_GROWSUP + help + Dynamic kernel stacks allow to save memory on machines with a lot of + threads by starting with small stacks, and grow them only when needed. + On workloads where most of the stack depth do not reach over one page + the memory saving can be subsentantial. The feature requires virtually + mapped kernel stacks in order to handle page faults. + config HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET def_bool n help diff --git a/include/linux/sched.h b/include/linux/sched.h index ffe8f618ab86..d3ce3cd065ce 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1628,7 +1628,7 @@ extern struct pid *cad_pid; #define PF_USED_MATH 0x00002000 /* If unset the fpu must be initialized b= efore use */ #define PF_USER_WORKER 0x00004000 /* Kernel thread cloned from userspace = thread */ #define PF_NOFREEZE 0x00008000 /* This thread should not be frozen */ -#define PF__HOLE__00010000 0x00010000 +#define PF_DYNAMIC_STACK 0x00010000 /* This thread allocated dynamic stack= pages */ #define PF_KSWAPD 0x00020000 /* I am kswapd */ #define PF_MEMALLOC_NOFS 0x00040000 /* All allocation requests will inheri= t GFP_NOFS */ #define PF_MEMALLOC_NOIO 0x00080000 /* All allocation requests will inheri= t GFP_NOIO */ diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_st= ack.h index 860faea06883..4934bfd65ad1 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -82,9 +82,49 @@ static inline void put_task_stack(struct task_struct *ts= k) {} =20 void exit_task_stack_account(struct task_struct *tsk); =20 +#ifdef CONFIG_DYNAMIC_STACK + +#define task_stack_end_corrupted(task) 0 + +#ifndef THREAD_PREALLOC_PAGES +#define THREAD_PREALLOC_PAGES 1 +#endif + +#define THREAD_DYNAMIC_PAGES \ + ((THREAD_SIZE >> PAGE_SHIFT) - THREAD_PREALLOC_PAGES) + +void dynamic_stack_refill_pages(void); +bool dynamic_stack_fault(struct task_struct *tsk, unsigned long address); + +/* + * Refill and charge for the used pages. + */ +static inline void dynamic_stack(struct task_struct *tsk) +{ + if (unlikely(tsk->flags & PF_DYNAMIC_STACK)) { + dynamic_stack_refill_pages(); + tsk->flags &=3D ~PF_DYNAMIC_STACK; + } +} + +static inline void set_task_stack_end_magic(struct task_struct *tsk) {} + +#else /* !CONFIG_DYNAMIC_STACK */ + #define task_stack_end_corrupted(task) \ (*(end_of_stack(task)) !=3D STACK_END_MAGIC) =20 +void set_task_stack_end_magic(struct task_struct *tsk); +static inline void dynamic_stack(struct task_struct *tsk) {} + +static inline bool dynamic_stack_fault(struct task_struct *tsk, + unsigned long address) +{ + return false; +} + +#endif /* CONFIG_DYNAMIC_STACK */ + static inline int object_is_on_stack(const void *obj) { void *stack =3D task_stack_page(current); @@ -114,7 +154,6 @@ static inline unsigned long stack_not_used(struct task_= struct *p) # endif } #endif -extern void set_task_stack_end_magic(struct task_struct *tsk); =20 static inline int kstack_end(void *addr) { diff --git a/kernel/fork.c b/kernel/fork.c index bbae5f705773..63e1fd661e17 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -204,6 +204,232 @@ static bool try_release_thread_stack_to_cache(struct = vm_struct *vm_area) return false; } =20 +#ifdef CONFIG_DYNAMIC_STACK + +static DEFINE_PER_CPU(struct page *, dynamic_stack_pages[THREAD_DYNAMIC_PA= GES]); + +static struct vm_struct *alloc_vmap_stack(int node) +{ + gfp_t gfp =3D THREADINFO_GFP & ~__GFP_ACCOUNT; + unsigned long addr, end; + struct vm_struct *vm_area; + int err, i; + + vm_area =3D get_vm_area_node(THREAD_SIZE, THREAD_ALIGN, VM_MAP, node, + gfp, __builtin_return_address(0)); + if (!vm_area) + return NULL; + + vm_area->pages =3D kmalloc_node(sizeof(void *) * + (THREAD_SIZE >> PAGE_SHIFT), gfp, node); + if (!vm_area->pages) + goto cleanup_err; + + for (i =3D 0; i < THREAD_PREALLOC_PAGES; i++) { + vm_area->pages[i] =3D alloc_pages(gfp, 0); + if (!vm_area->pages[i]) + goto cleanup_err; + vm_area->nr_pages++; + } + + addr =3D (unsigned long)vm_area->addr + + (THREAD_DYNAMIC_PAGES << PAGE_SHIFT); + end =3D (unsigned long)vm_area->addr + THREAD_SIZE; + err =3D vmap_pages_range_noflush(addr, end, PAGE_KERNEL, vm_area->pages, + PAGE_SHIFT); + if (err) + goto cleanup_err; + + return vm_area; +cleanup_err: + for (i =3D 0; i < vm_area->nr_pages; i++) + __free_page(vm_area->pages[i]); + kfree(vm_area->pages); + kfree(vm_area); + + return NULL; +} + +static void free_vmap_stack(struct vm_struct *vm_area) +{ + int i, nr_pages; + + remove_vm_area(vm_area->addr); + + nr_pages =3D vm_area->nr_pages; + for (i =3D 0; i < nr_pages; i++) + __free_page(vm_area->pages[i]); + + kfree(vm_area->pages); + kfree(vm_area); +} + +/* + * This flag is used to pass information from fault handler to refill about + * which pages were allocated, and should be charged to memcg. + */ +#define DYNAMIC_STACK_PAGE_AQUIRED_FLAG 0x1 + +static struct page *dynamic_stack_get_page(void) +{ + struct page **pages =3D this_cpu_ptr(dynamic_stack_pages); + int i; + + for (i =3D 0; i < THREAD_DYNAMIC_PAGES; i++) { + struct page *page =3D pages[i]; + + if (page && !((uintptr_t)page & DYNAMIC_STACK_PAGE_AQUIRED_FLAG)) { + pages[i] =3D (void *)((uintptr_t)pages[i] | DYNAMIC_STACK_PAGE_AQUIRED_= FLAG); + return page; + } + } + + return NULL; +} + +static int dynamic_stack_refill_pages_cpu(unsigned int cpu) +{ + struct page **pages =3D per_cpu_ptr(dynamic_stack_pages, cpu); + int i; + + for (i =3D 0; i < THREAD_DYNAMIC_PAGES; i++) { + if (pages[i]) + break; + pages[i] =3D alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); + if (unlikely(!pages[i])) { + pr_err("failed to allocate dynamic stack page for cpu[%d]\n", + cpu); + } + } + + return 0; +} + +static int dynamic_stack_free_pages_cpu(unsigned int cpu) +{ + struct page **pages =3D per_cpu_ptr(dynamic_stack_pages, cpu); + int i; + + for (i =3D 0; i < THREAD_DYNAMIC_PAGES; i++) { + if (!pages[i]) + continue; + __free_page(pages[i]); + pages[i] =3D NULL; + } + + return 0; +} + +void dynamic_stack_refill_pages(void) +{ + struct page **pages =3D this_cpu_ptr(dynamic_stack_pages); + int i, ret; + + for (i =3D 0; i < THREAD_DYNAMIC_PAGES; i++) { + struct page *page =3D pages[i]; + + if (!((uintptr_t)page & DYNAMIC_STACK_PAGE_AQUIRED_FLAG)) + break; + + page =3D (void *)((uintptr_t)page & ~DYNAMIC_STACK_PAGE_AQUIRED_FLAG); + ret =3D memcg_kmem_charge_page(page, GFP_KERNEL, 0); + /* + * XXX Since stack pages were already allocated, we should never + * fail charging. Therefore, we should probably induce force + * charge and oom killing if charge fails. + */ + if (unlikely(ret)) + pr_warn_ratelimited("dynamic stack: charge for allocated page failed\n"= ); + + mod_lruvec_page_state(page, NR_KERNEL_STACK_KB, + PAGE_SIZE / 1024); + + page =3D alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); + if (unlikely(!page)) + pr_err_ratelimited("failed to refill per-cpu dynamic stack\n"); + pages[i] =3D page; + } +} + +bool noinstr dynamic_stack_fault(struct task_struct *tsk, unsigned long ad= dress) +{ + unsigned long stack, hole_end, addr; + struct vm_struct *vm_area; + struct page *page; + int nr_pages; + pte_t *pte; + + /* check if address is inside the kernel stack area */ + stack =3D (unsigned long)tsk->stack; + if (address < stack || address >=3D stack + THREAD_SIZE) + return false; + + vm_area =3D tsk->stack_vm_area; + if (!vm_area) + return false; + + /* + * check if this stack can still grow, otherwise fault will be reported + * as guard page access. + */ + nr_pages =3D vm_area->nr_pages; + if (nr_pages >=3D (THREAD_SIZE >> PAGE_SHIFT)) + return false; + + /* Check if fault address is within the stack hole */ + hole_end =3D stack + THREAD_SIZE - (nr_pages << PAGE_SHIFT); + if (address >=3D hole_end) + return false; + + /* + * Most likely we faulted in the page right next to the last mapped + * page in the stack, however, it is possible (but very unlikely) that + * the faulted page is actually skips some pages in the stack. Make sure + * we do not create more than one holes in the stack, and map every + * page between the current fault address and the last page that is + * mapped in the stack. + */ + address =3D PAGE_ALIGN_DOWN(address); + for (addr =3D hole_end - PAGE_SIZE; addr >=3D address; addr -=3D PAGE_SIZ= E) { + /* Take the next page from the per-cpu list */ + page =3D dynamic_stack_get_page(); + if (!page) { + instrumentation_begin(); + pr_emerg("Failed to allocate a page during kernel_stack_fault\n"); + instrumentation_end(); + return false; + } + + /* Store the new page in the stack's vm_area */ + vm_area->pages[nr_pages] =3D page; + vm_area->nr_pages =3D nr_pages + 1; + + /* Add the new page entry to the page table */ + pte =3D virt_to_kpte(addr); + if (!pte) { + instrumentation_begin(); + pr_emerg("The PTE page table for a kernel stack is not found\n"); + instrumentation_end(); + return false; + } + + /* Make sure there are no existing mappings at this address */ + if (pte_present(*pte)) { + instrumentation_begin(); + pr_emerg("The PTE contains a mapping\n"); + instrumentation_end(); + return false; + } + set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL)); + } + + /* Refill the pcp stack pages during context switch */ + tsk->flags |=3D PF_DYNAMIC_STACK; + + return true; +} + +#else /* !CONFIG_DYNAMIC_STACK */ static inline struct vm_struct *alloc_vmap_stack(int node) { void *stack; @@ -226,6 +452,7 @@ static inline void free_vmap_stack(struct vm_struct *vm= _area) { vfree(vm_area->addr); } +#endif /* CONFIG_DYNAMIC_STACK */ =20 static void thread_stack_free_rcu(struct rcu_head *rh) { @@ -1083,6 +1310,16 @@ void __init fork_init(void) NULL, free_vm_stack_cache); #endif =20 +#ifdef CONFIG_DYNAMIC_STACK + cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "fork:dynamic_stack", + dynamic_stack_refill_pages_cpu, + dynamic_stack_free_pages_cpu); + /* + * Fill the dynamic stack pages for the boot CPU, others will be filled + * as CPUs are onlined. + */ + dynamic_stack_refill_pages_cpu(smp_processor_id()); +#endif scs_init(); =20 lockdep_init_task(&init_task); @@ -1096,6 +1333,7 @@ int __weak arch_dup_task_struct(struct task_struct *d= st, return 0; } =20 +#ifndef CONFIG_DYNAMIC_STACK void set_task_stack_end_magic(struct task_struct *tsk) { unsigned long *stackend; @@ -1103,6 +1341,7 @@ void set_task_stack_end_magic(struct task_struct *tsk) stackend =3D end_of_stack(tsk); *stackend =3D STACK_END_MAGIC; /* for overflow detection */ } +#endif =20 static struct task_struct *dup_task_struct(struct task_struct *orig, int n= ode) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 9116bcc90346..20f9523c3159 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6617,6 +6617,7 @@ static void __sched notrace __schedule(unsigned int s= ched_mode) rq =3D cpu_rq(cpu); prev =3D rq->curr; =20 + dynamic_stack(prev); schedule_debug(prev, !!sched_mode); =20 if (sched_feat(HRTICK) || sched_feat(HRTICK_DL)) --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF2755676F for ; Mon, 11 Mar 2024 16:46:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175617; cv=none; b=rDWISzrQCfC4aUZVUbwaN1Hb62LfmW8HGTGsWo8Z9csNbszwWXyxp8t4+dxnMew4ovd6GZtuslEwxnoFqnKx7TKmjW4sCG/T+HRaHMUw2Ts1NezjX1p7YsoNFPpKqKiMcc+PrnzFYHS6+NS2VJJtFBY85d+ugLmAjKWEBT2lejc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175617; c=relaxed/simple; bh=SjdSy7wze8iqdobtOSWXvuZy+HC33whE98lynSJVzCI=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=omZfznI2ehvlr+NcxDdul+hwJdYhPNqNKk3sq31YJLa2+lSMt90P+Q1trfP5wQgwcuyWoBnXqyKOEOVIlqEOgon3uithKunnXRR9SxKUQRCVCT2frTdUwpu8ZmKoFoB8eqSWhUnwydRMl/hmPlhZbAHWzHA66l7dJmvrRn57kEc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=ARTcAiqf; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="ARTcAiqf" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-783045e88a6so277630385a.0 for ; Mon, 11 Mar 2024 09:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175614; x=1710780414; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=NGLgnjdxSReHw9wlD+FW1YibHNtCAjZjXR36RT3BbYQ=; b=ARTcAiqfV/nHTM50ECHtkfKwZoR4w7P0StEgR2k4YbupLc+QL+Qvhr06tCGrCwkljM 9eNKpMpIbW860mYyIsyVagCnvWZBKDrKEik72y6JrjNvATLdwr+r/Mpv9TB6NCRh9UVC 52KypElvcNowwDatOwJboNtQ9aXFLKJNcLsaowiP2m/0+CkesFqxbFhK69sLSwU+lsKd mbl/y923RY5IMwAi07Jq0wDjnPWGk9h+WaLov6ZlmbuJmNiUbCmwD4LvoKywV1S42IK8 MADIv/W1v9ppHBrzqP9SXBFCdy3r35mG2g82lQbMVQ/eRDSMv7QrxHtJYjGqiakwv5PA nncg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175614; x=1710780414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NGLgnjdxSReHw9wlD+FW1YibHNtCAjZjXR36RT3BbYQ=; b=rZC6YWFU1xAA7u6x/Z8Yvjt08zPejEd+iPyZfIIpBC39TXfiKld4gAGVBRUMzB6Z1B Y6l+GkHmqyd/5P985lvRWMXYnibe2AoqqzHyDPDlX3A4yk2e95ICW1wZAWmduaxXyuRE iLzIrm1c6ILYzBWAnU0NezoJQc/SWulyY5C1JlWHiF88NivwFLovrJrJdC7cAIckig1X 88GjElYIx/CFMugss+Gnb4KGD4YPvpb3PMU6qi5cl+CR/vVnrnII3fuaWHHCrcQtx/zO CRdRNJweCasoH1HEwza8sVteDQad65WV/yU/jn+jqoalflpH0R1VyCGZkBF3Wq9sV4cC nmpA== X-Gm-Message-State: AOJu0Yx8T1mtYaVWH6VHyAUXyg1UU9/J2I8qwI1JhafM5fOtx2RjWyML T98BlaMXBiel1puENP7iOYGYcAGNyA3B17kT3UuKtNxEUxpKfBcToget9KrQujYNioiE3cUAvIe i8H0= X-Google-Smtp-Source: AGHT+IHiFimL/uL6CiZ0/ffUHb5GjnvMvZQk5uciXsXYpJX5LXGSw9tTbmvr67JCBeLYBzWo/uqLsw== X-Received: by 2002:a05:620a:45a1:b0:788:4e44:a9b7 with SMTP id bp33-20020a05620a45a100b007884e44a9b7mr11466669qkb.78.1710175614423; Mon, 11 Mar 2024 09:46:54 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:54 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 11/14] x86: add support for Dynamic Kernel Stacks Date: Mon, 11 Mar 2024 16:46:35 +0000 Message-ID: <20240311164638.2015063-12-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add dynamic_stack_fault() calls to the kernel faults, and also declare HAVE_ARCH_DYNAMIC_STACK =3D y, so that dynamic kernel stacks can be enabled on x86 architecture. Signed-off-by: Pasha Tatashin --- arch/x86/Kconfig | 1 + arch/x86/kernel/traps.c | 3 +++ arch/x86/mm/fault.c | 3 +++ 3 files changed, 7 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 5edec175b9bf..9bb0da3110fa 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -197,6 +197,7 @@ config X86 select HAVE_ARCH_USERFAULTFD_WP if X86_64 && USERFAULTFD select HAVE_ARCH_USERFAULTFD_MINOR if X86_64 && USERFAULTFD select HAVE_ARCH_VMAP_STACK if X86_64 + select HAVE_ARCH_DYNAMIC_STACK if X86_64 select HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET select HAVE_ARCH_WITHIN_STACK_FRAMES select HAVE_ASM_MODVERSIONS diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index c3b2f863acf0..cc05401e729f 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -413,6 +413,9 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif =20 + if (dynamic_stack_fault(current, address)) + return; + irqentry_nmi_enter(regs); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index d6375b3c633b..651c558b10eb 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1198,6 +1198,9 @@ do_kern_addr_fault(struct pt_regs *regs, unsigned lon= g hw_error_code, if (is_f00f_bug(regs, hw_error_code, address)) return; =20 + if (dynamic_stack_fault(current, address)) + return; + /* Was the fault spurious, caused by lazy TLB invalidation? */ if (spurious_kernel_fault(hw_error_code, address)) return; --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A04FB56B72 for ; Mon, 11 Mar 2024 16:46:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175618; cv=none; b=HWLd15y2qE3f7IQzxUhsYXR+k5FMTDQaDWCws6mWxZJtitIlUpKBp8ZGjRY+DL80IQtgE11cz6HfdjANjdkWtL0RBcv1YfhDPwg9kwF17yAo/B+nn4hUXPh1MEjMIa25V+RwlNsFRC9+FfaQZjSKeUSjKNBwQNj1NhfFZY4tXg8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175618; c=relaxed/simple; bh=yMyYef3NZl4G4p32yVMbEyHz15V0r/n9UEHNsI8V/OU=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gTU6rzXEBgQ9NX4SL5bUTBq55J3F9zH2vJN7KexCYJQLGiUwY8I9hTfOJWMuGG19mPYTXzc5gfpGTXSxI59uG88LT4lXpQLK367DWeA0eOXkmhpgn0KXTyl7p9++9rFtb0ueQN5tGjcJHyjpqLVYZ3oz68O6mz+bRuRxD/Qeotc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=KvYA/g7n; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="KvYA/g7n" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-783045e88a6so277632585a.0 for ; Mon, 11 Mar 2024 09:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175615; x=1710780415; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=w3lfkbgOqarNxaTYrCv0KyWjfiF6t60qYNqci1EaSrU=; b=KvYA/g7nW8657mqsa3INNq4bY3H2wyoRVWYt64PLx8aNNc1UOdVPJncyG4qm3EG7T4 kxmmtALcXLvEbGobR6d5CDb6mfMSBIVUPVfMhZm8Rm8T3MnDmKOnXvfTE5krvypWnYn9 9gWfoFmwRlJ6cCn4ZP7CBm4CS7uj5Ni2Uu1kxz1yUR+cUP3y0YfVpuV2RF8rtrLy7+7L tprYNFeJCUwBCJcYRpOahsv8WF+By1fHIzrGQ3FPkme8EIe8NTYlOqNifFhwieS3fQAJ HEYORzRRjXTfMmjbfBXN0DMZ9FR7Ai/oCIOIPzlShzmZ0F4aRiUrFVRIioH5n/dgSpVf RuQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175615; x=1710780415; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w3lfkbgOqarNxaTYrCv0KyWjfiF6t60qYNqci1EaSrU=; b=vj8jsMSdix0sgocRk2cgx+F/immNRyXgHip8rFzDizjKsDk/nWhNyBmtcqWGKPRvjS i9zrQeMGHBpSNjui8noIJdIJXY9dTZqGM6aN7b54Zeec5PzMZB0vhHN5jbSoAbqY2hSR NwDbquxne3o9A/iWZJkNg2rqgOjCyYrV3xoJ2jzLLyfMcBe6Wf7iSPGQEH6R6HTFZlD+ e4VB50IbyNWLGOwDF782rdh6PgLWt08HuHVk8ks5Pk4S1BaAwruMeiTaAhWiWO7oyjld E7hMMd2DGnLjduNm1CJU6Dme8lXpAoixyG0/Tio36FlLbZD2rQ+3qOIBo5gB7h7MdvkB 6W6g== X-Gm-Message-State: AOJu0YwPhowmijEBNMNmGFIuGMiauSyuD0kEFeVOgQIYAHqdm9Ro4PWl RvW1P2UTsHFJJ4Bpuqlb+tGpBsSMgP1cglP6OVXUqv7tVDws/2GC4nbgUYxypUN9t36HZSGePqO PPgc= X-Google-Smtp-Source: AGHT+IF4l4AZR/miARXhlieJbwXHGqXTuZHHo2Erbz9LNEfZfrLPyat4CsaJX3i07UmkC9xY9bUxgA== X-Received: by 2002:a05:620a:4609:b0:788:2740:550a with SMTP id br9-20020a05620a460900b007882740550amr9951495qkb.44.1710175615373; Mon, 11 Mar 2024 09:46:55 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:54 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 12/14] task_stack.h: Clean-up stack_not_used() implementation Date: Mon, 11 Mar 2024 16:46:36 +0000 Message-ID: <20240311164638.2015063-13-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Inside small stack_not_used() function there are several ifdefs for stack growing-up vs. regular versions. Instead just implement this function two times, one for growing-up and another regular. This is needed, because there will be a third implementation of this function for dynamic stacks. Signed-off-by: Pasha Tatashin --- include/linux/sched/task_stack.h | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_st= ack.h index 4934bfd65ad1..396d5418ae32 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -135,25 +135,30 @@ static inline int object_is_on_stack(const void *obj) extern void thread_stack_cache_init(void); =20 #ifdef CONFIG_DEBUG_STACK_USAGE +#ifdef CONFIG_STACK_GROWSUP static inline unsigned long stack_not_used(struct task_struct *p) { unsigned long *n =3D end_of_stack(p); =20 - do { /* Skip over canary */ -# ifdef CONFIG_STACK_GROWSUP + do { /* Skip over canary */ n--; -# else - n++; -# endif } while (!*n); =20 -# ifdef CONFIG_STACK_GROWSUP return (unsigned long)end_of_stack(p) - (unsigned long)n; -# else +} +#else /* !CONFIG_STACK_GROWSUP */ +static inline unsigned long stack_not_used(struct task_struct *p) +{ + unsigned long *n =3D end_of_stack(p); + + do { /* Skip over canary */ + n++; + } while (!*n); + return (unsigned long)n - (unsigned long)end_of_stack(p); -# endif } -#endif +#endif /* CONFIG_STACK_GROWSUP */ +#endif /* CONFIG_DEBUG_STACK_USAGE */ =20 static inline int kstack_end(void *addr) { --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E6B656B84 for ; Mon, 11 Mar 2024 16:46:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175619; cv=none; b=ZPk88ik0sN4kqsu2BhC5AioMmkH7PZwiydBDEEB0YcH+gdZH0fq6sgnOkVxGLNfIOVcr8sWDZLCJjglFbpU1guJ5jM0NBKOZHVNx42UDMbeKLesSfnh06ti0FwaNED/VqzcSEmHmsXx4Wnjhy8dmzO+DY537brIy9so7rsqeqd0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175619; c=relaxed/simple; bh=uG0XdE33GnOjaXgw0sEgfD6qMLI1cOfcn+gbSJbqKvs=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hMUUAVz5i/ncWvnvfPZ6tKEeMmVADB3jBBT7wA6oqhxATmlTq8FKgPdPUOvCHMnYBczInq8hQ7tQiAVzZYCOawua5bPI7H4xVp9MlSviqLBGeAP8KiBps0Novji9Gh04CdurzclqsuXzvfNDzjFmBU0+BmYOaC8zkfl5DqxbsCk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=j/q/31gH; arc=none smtp.client-ip=209.85.222.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="j/q/31gH" Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-78863dc4247so91038685a.3 for ; Mon, 11 Mar 2024 09:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175616; x=1710780416; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=cK/ZqVdlnureqUddSXvrvjnokezl2c4iuxNaHlbvXOk=; b=j/q/31gHIaFViDLSBz8X5TdFp8y/fs+HWhdHNK5zThNLm6yMOfvGs31ViTHpgsgbOh qE5EXDLQH27ltiNQw9JfOoQfD3ywoNWjLFyaF4z6HpWT38O/d5SdkViur1Xom1h9eHT7 0Lq3PGasRrdsKHUQVw+INHtLpHVdMcyxi3LxTDJussbYrp9yoC7GDKj/EnbM3h88r1VD RK5GnF9DQmqv2YHT8A/jE91DBWUWm9YgklL0w1hB6kYSlA4vFlFkZrC2DT+IOM6chxC3 IBNBMcTz1ZvcjW/8KQyBO89y1TmBiXJMVl/3vDXstI+JlnFTNh3ML0kS5H7SV1LnIoDS L1bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175616; x=1710780416; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cK/ZqVdlnureqUddSXvrvjnokezl2c4iuxNaHlbvXOk=; b=bjWCGpDAZrw1GEajhqem+EvKaWj4w9KzaF0KzXhq19khShRZaeg5sOlxppForz/SBO M+Pv3dWZeGHDZTihVZPYRyBndsc+YYIrzS9MZ+g/qvd4Vnk2PPbE3dqTv7CaZXusr/VP ddVNXrX/+lRXfeItza3HCKkrM8zLYLaJE5s0MBp3VUd1jaO5UIsa35E/Mnru7uUrdz4g M/vMz1O4uiuAeNYJ78oUolvwH2o/P96DyevxjMv1SWP8Ge4YkIrEPIAxaYDH6CnNHal2 I8bX9b/nn9bKCEmXGuNaqjCORi67qbXs/z8yOxGENitTL69we0REJzVUXfHZzMTGMox2 67Kw== X-Gm-Message-State: AOJu0Yz//QB3SsdA/zkqREjlYS7T6nkXfIYalnNRQOBik/esglVugXNl eZk3rx2QmNjAxUSIOZzAh51a2RQL7tAC/JG8yfDlxQKEywop+nvOP+yGlRWrVDn8nMkd2+ytVmo HExc= X-Google-Smtp-Source: AGHT+IEp6VoHaP5KrWImU0EWHAjL+QS9WD2gq+SL5008BAWJX/vKvEeMCLb5ZMa54jxJQaXt/mOFWQ== X-Received: by 2002:ae9:e701:0:b0:788:c09:ce3 with SMTP id m1-20020ae9e701000000b007880c090ce3mr7062849qka.46.1710175616304; Mon, 11 Mar 2024 09:46:56 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:55 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 13/14] task_stack.h: Add stack_not_used() support for dynamic stack Date: Mon, 11 Mar 2024 16:46:37 +0000 Message-ID: <20240311164638.2015063-14-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" CONFIG_DEBUG_STACK_USAGE is enabled by default on most architectures. Its purpose is to determine and print the maximum stack depth on thread exit. The way it works, is it starts from the buttom of the stack and searches the first non-zero word in the stack. With dynamic stack it does not work very well, as it means it faults every pages in every stack. Instead, add a specific version of stack_not_used() for dynamic stacks where instead of starting from the buttom of the stack, we start from the last page mapped in the stack. In addition to not doing uncessary page faulting, this search is optimized by skipping search through zero pages. Also, because dynamic stack does not end with MAGIC_NUMBER, there is no need to skeep the buttom most word in the stack. Signed-off-by: Pasha Tatashin --- arch/Kconfig | 1 - include/linux/sched/task_stack.h | 38 +++++++++++++++++++++++--------- 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index da3df347b069..759b2bb7edb6 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1266,7 +1266,6 @@ config DYNAMIC_STACK depends on HAVE_ARCH_DYNAMIC_STACK depends on VMAP_STACK depends on !KASAN - depends on !DEBUG_STACK_USAGE depends on !STACK_GROWSUP help Dynamic kernel stacks allow to save memory on machines with a lot of diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_st= ack.h index 396d5418ae32..c5fb679b31ee 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -9,6 +9,7 @@ #include #include #include +#include =20 #ifdef CONFIG_THREAD_INFO_IN_TASK =20 @@ -109,6 +110,21 @@ static inline void dynamic_stack(struct task_struct *t= sk) =20 static inline void set_task_stack_end_magic(struct task_struct *tsk) {} =20 +#ifdef CONFIG_DEBUG_STACK_USAGE +static inline unsigned long stack_not_used(struct task_struct *p) +{ + struct vm_struct *vm_area =3D p->stack_vm_area; + unsigned long alloc_size =3D vm_area->nr_pages << PAGE_SHIFT; + unsigned long stack =3D (unsigned long)p->stack; + unsigned long *n =3D (unsigned long *)(stack + THREAD_SIZE - alloc_size); + + while (!*n) + n++; + + return (unsigned long)n - stack; +} +#endif /* CONFIG_DEBUG_STACK_USAGE */ + #else /* !CONFIG_DYNAMIC_STACK */ =20 #define task_stack_end_corrupted(task) \ @@ -123,17 +139,6 @@ static inline bool dynamic_stack_fault(struct task_str= uct *tsk, return false; } =20 -#endif /* CONFIG_DYNAMIC_STACK */ - -static inline int object_is_on_stack(const void *obj) -{ - void *stack =3D task_stack_page(current); - - return (obj >=3D stack) && (obj < (stack + THREAD_SIZE)); -} - -extern void thread_stack_cache_init(void); - #ifdef CONFIG_DEBUG_STACK_USAGE #ifdef CONFIG_STACK_GROWSUP static inline unsigned long stack_not_used(struct task_struct *p) @@ -160,6 +165,17 @@ static inline unsigned long stack_not_used(struct task= _struct *p) #endif /* CONFIG_STACK_GROWSUP */ #endif /* CONFIG_DEBUG_STACK_USAGE */ =20 +#endif /* CONFIG_DYNAMIC_STACK */ + +static inline int object_is_on_stack(const void *obj) +{ + void *stack =3D task_stack_page(current); + + return (obj >=3D stack) && (obj < (stack + THREAD_SIZE)); +} + +extern void thread_stack_cache_init(void); + static inline int kstack_end(void *addr) { /* Reliable end of stack detection: --=20 2.44.0.278.ge034bb2e1d-goog From nobody Mon Feb 9 07:43:11 2026 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5E76857303 for ; Mon, 11 Mar 2024 16:46:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175619; cv=none; b=I/pDj0eJ2e0ST8APlTjt92w1yFXndxI56NAh8EH3sMW+TlM5U/ozPt8aaMYnpcslwVuIdlORWfI0tqvHyLRiA79mPq5x45bQ1jTt3jaQ233MlRAwoV6PU7wpwTD0jAsAeSz/hgZ7JPP24NkE8EnUYO0qrdawH3jMvx1UP2NeakQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710175619; c=relaxed/simple; bh=/QZnssDAeZpjT2sFmC0j/E2H3o0tSEKpT/2jMGwPGOw=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EG49q/4bW7jBtSh2JZPpGRGK/thdpD8eMzQablWP7O3os9ip0sPEUOF64o9OpnHphyZxOSwr1QielHgTpnJl6KUp3UWsNvcmMjpRHjzlytHj2bQFUtQHxKw7UkBKD099465BTaQkAAMlX5i4KHv6dXcvO8l/yXqHLJYMDGw1wKM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com; spf=pass smtp.mailfrom=soleen.com; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b=m3FrBRkb; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=soleen.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=soleen.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=soleen-com.20230601.gappssmtp.com header.i=@soleen-com.20230601.gappssmtp.com header.b="m3FrBRkb" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-7884a9a47a5so122048385a.2 for ; Mon, 11 Mar 2024 09:46:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1710175617; x=1710780417; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=q6FpZ49BpunKdsFtEl370XBCgostMe6YS8Mf7VMqlwM=; b=m3FrBRkbW1wH9x988Zv/mO0VhHRfDaVv+k5okI9z8d8zGJ0OTxhr+P98Ds7N0ymveY DW8iDwq6YwWOZtRRKh3JmyFVZduGITXhTF8Igupvz6FtCAOGVP6Djk93KeLl8etbmNuS 6TKDtIPia0vhrI976Qa7KEkf32e0W9Y9apsjFC1Akesy9WnHaJWih5+42tboxL+L/sNY tsSEOugU6XXiVDb/pe8Ve78pLnwk+0VVwxyAkZ703tfdMCublXompMRWGfb578k9cESX zOECp1/QDHeAounTyL2o0L6qo1noAYnS7tkQqDD3L9oT2IHa5eWXX/axE/L/ar1BVRvf RodA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710175617; x=1710780417; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q6FpZ49BpunKdsFtEl370XBCgostMe6YS8Mf7VMqlwM=; b=iWsVdIwff+bgNtDnEPrMcq5dG0wgO1WdLUUxNQkP/gTxFawBk/sJYjmQuFLd3ItPi5 kN7RJU/h4d/HkeHqUYVhoTki8gcv+QiBIlAuuCagynkAzuH+r6QFvCt1sww6+PR6Wfik VE4GeY3E3tfP2pcXQj0G1l/fS2LE6Z1+bK+9Bu0PBhEKpwrtBrKrtk+sfESLNMOrr602 YPHxVIcO/KO4mj6Pz8QFsgD7kxiCciX4o180fMglN8FD+5eaoBLNicETJa7QKpAduQjf pLjFjDqTnibZeYPbZDAMjHlHqMnuE1tKHJXRaSfPU0t6RZUriA40V2NKsOWbcw7bC6bX ZSEA== X-Gm-Message-State: AOJu0YxEZzkcw29uLOdsrFwCIv95bCU3cguHkaIWalymJFZu2X0yLyFM h/dnjpd8pCjAv72OGiUtrxMjl3c+i2JI740Z9yb9pQ3iwOn0u+p31Nj0ABKXzNYbE+Y06I2Dkh7 cqXo= X-Google-Smtp-Source: AGHT+IGBBQHGTyj0HK6a6nT9QQX2CzgLifOnSoon3yUl5MxZhI+SOhyqTpiNJHmFfzsFPzFEmk6ojA== X-Received: by 2002:ae9:e701:0:b0:788:2e8a:1731 with SMTP id m1-20020ae9e701000000b007882e8a1731mr6728530qka.17.1710175617252; Mon, 11 Mar 2024 09:46:57 -0700 (PDT) Received: from soleen.c.googlers.com.com (150.254.86.34.bc.googleusercontent.com. [34.86.254.150]) by smtp.gmail.com with ESMTPSA id d27-20020a05620a137b00b00788228fbe05sm2851589qkl.17.2024.03.11.09.46.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 09:46:56 -0700 (PDT) From: Pasha Tatashin To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, hpa@zytor.com, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kent.overstreet@linux.dev, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com, pasha.tatashin@soleen.com Subject: [RFC 14/14] fork: Dynamic Kernel Stack accounting Date: Mon, 11 Mar 2024 16:46:38 +0000 Message-ID: <20240311164638.2015063-15-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog In-Reply-To: <20240311164638.2015063-1-pasha.tatashin@soleen.com> References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an accounting of amount of stack pages that has been faulted is currently in use. Example use case: $ cat /proc/vmstat | grep stack nr_kernel_stack 18684 nr_dynamic_stacks_faults 156 The above shows that the kernel stacks use total 18684KiB, out of which 156KiB were faulted in. Given that the pre-allocated stacks are 4KiB, we can determine the total number of tasks: tasks =3D (nr_kernel_stack - nr_dynamic_stacks_faults) / 4 =3D 4632. The amount of kernel stack memory without dynamic stack on this machine woud be: 4632 * 16 KiB =3D 74,112 KiB Therefore, in this example dynamic stacks save: 55,428 KiB Signed-off-by: Pasha Tatashin --- include/linux/mmzone.h | 3 +++ kernel/fork.c | 13 ++++++++++++- mm/vmstat.c | 3 +++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index a497f189d988..ba4f1d148c3f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -198,6 +198,9 @@ enum node_stat_item { NR_FOLL_PIN_ACQUIRED, /* via: pin_user_page(), gup flag: FOLL_PIN */ NR_FOLL_PIN_RELEASED, /* pages returned via unpin_user_page() */ NR_KERNEL_STACK_KB, /* measured in KiB */ +#ifdef CONFIG_DYNAMIC_STACK + NR_DYNAMIC_STACKS_FAULTS_KB, /* KiB of faulted kernel stack memory */ +#endif #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) NR_KERNEL_SCS_KB, /* measured in KiB */ #endif diff --git a/kernel/fork.c b/kernel/fork.c index 63e1fd661e17..2520583d160a 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -343,6 +343,9 @@ void dynamic_stack_refill_pages(void) =20 mod_lruvec_page_state(page, NR_KERNEL_STACK_KB, PAGE_SIZE / 1024); + mod_lruvec_page_state(page, + NR_DYNAMIC_STACKS_FAULTS_KB, + PAGE_SIZE / 1024); =20 page =3D alloc_pages(THREADINFO_GFP & ~__GFP_ACCOUNT, 0); if (unlikely(!page)) @@ -771,9 +774,17 @@ static void account_kernel_stack(struct task_struct *t= sk, int account) int i, nr_pages; =20 nr_pages =3D vm->nr_pages; - for (i =3D 0; i < nr_pages; i++) + for (i =3D 0; i < nr_pages; i++) { mod_lruvec_page_state(vm->pages[i], NR_KERNEL_STACK_KB, account * (PAGE_SIZE / 1024)); +#ifdef CONFIG_DYNAMIC_STACK + if (i >=3D THREAD_PREALLOC_PAGES) { + mod_lruvec_page_state(vm->pages[i], + NR_DYNAMIC_STACKS_FAULTS_KB, + account * (PAGE_SIZE / 1024)); + } +#endif + } } else { void *stack =3D task_stack_page(tsk); =20 diff --git a/mm/vmstat.c b/mm/vmstat.c index db79935e4a54..1ad6eede3d85 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1237,6 +1237,9 @@ const char * const vmstat_text[] =3D { "nr_foll_pin_acquired", "nr_foll_pin_released", "nr_kernel_stack", +#ifdef CONFIG_DYNAMIC_STACK + "nr_dynamic_stacks_faults", +#endif #if IS_ENABLED(CONFIG_SHADOW_CALL_STACK) "nr_shadow_call_stack", #endif --=20 2.44.0.278.ge034bb2e1d-goog