From nobody Tue Dec 16 14:37:27 2025 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71C7618B47C for ; Mon, 2 Jun 2025 11:02:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748862148; cv=none; b=M9YLnhpIHOTFDnbNRJTB8R31y2PyLf9kxpWs9bFKRpQEoZ5zk7ZtVn1as8XaTKyocH+XV7GP8REsefaDchFi7gvzZB7XvrLrQmMedfXSjihQABx08COfJvMg/Z9gtc6ZDA9yqQPUb9ihPGz/+/rZY8a5hETmi0YsRF1XR7lRwUY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748862148; c=relaxed/simple; bh=sGGOF6H61Kb8NYKEAILjOo0Ob8kpMXtmxpZkU6fLnnE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=mEl2mflZyqAMr9QUsx1730RFCbuZMOIK1Sz6Taa0dz6ggxYyy0JOryWbxYTsqawvZLdTu/zbtI+0oiqc5tVfVXVO/m7skdo2d6KzjPDIU/yEXfVBuUgHDH1WVDfxs5O550gbTX1SfqZhOIJNklT7L6/uM7/PFghINRY1x4QN80o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=uiNnEm6O; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=pGAcvMIH; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=uiNnEm6O; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=pGAcvMIH; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="uiNnEm6O"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="pGAcvMIH"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="uiNnEm6O"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="pGAcvMIH" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C13FA219E4; Mon, 2 Jun 2025 11:02:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A6pctFRzU+SYdaBRtN4s3/10XePRK+pRaGKjp/WHplY=; b=uiNnEm6OLfwlIy2mTZ55O3baRpstg7j3nZF0f7NvZrLt1413wDU+hXDe49Yu2rROSp0Re8 JzLccXneTb67HAWMzTUEOJDNdYmyrN0Y/mVSspiH8TQgsirXCKrHWwZvDkdg6aFoCD3xVh 0iXDfr3nK4Qt/D58LcPqtB56GgsWgZI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A6pctFRzU+SYdaBRtN4s3/10XePRK+pRaGKjp/WHplY=; b=pGAcvMIHfx92OXumIzPMoG1qXyxWQ1YzNoFJVn17jOJu3cXXi2H73M5BBs+iRCjC/3P9uh 4ygCKeCA6MBFWiAQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A6pctFRzU+SYdaBRtN4s3/10XePRK+pRaGKjp/WHplY=; b=uiNnEm6OLfwlIy2mTZ55O3baRpstg7j3nZF0f7NvZrLt1413wDU+hXDe49Yu2rROSp0Re8 JzLccXneTb67HAWMzTUEOJDNdYmyrN0Y/mVSspiH8TQgsirXCKrHWwZvDkdg6aFoCD3xVh 0iXDfr3nK4Qt/D58LcPqtB56GgsWgZI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=A6pctFRzU+SYdaBRtN4s3/10XePRK+pRaGKjp/WHplY=; b=pGAcvMIHfx92OXumIzPMoG1qXyxWQ1YzNoFJVn17jOJu3cXXi2H73M5BBs+iRCjC/3P9uh 4ygCKeCA6MBFWiAQ== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id AAD1C13AE0; Mon, 2 Jun 2025 11:02:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id EFJlKbmEPWggHAAAD6G6ig (envelope-from ); Mon, 02 Jun 2025 11:02:17 +0000 From: Vlastimil Babka Date: Mon, 02 Jun 2025 13:02:12 +0200 Subject: [PATCH v2 1/2] mm, slab: restore NUMA policy support for large kmalloc Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250602-frozen-pages-for-large-kmalloc-v2-1-84a21f2c3640@suse.cz> References: <20250602-frozen-pages-for-large-kmalloc-v2-0-84a21f2c3640@suse.cz> In-Reply-To: <20250602-frozen-pages-for-large-kmalloc-v2-0-84a21f2c3640@suse.cz> To: Christoph Lameter , David Rientjes Cc: Andrew Morton , Roman Gushchin , Harry Yoo , Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Spam-Flag: NO X-Spam-Score: -8.30 X-Spamd-Result: default: False [-8.30 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[99.99%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[suse.cz:mid,suse.cz:email] X-Spam-Level: The slab allocator observes the task's NUMA policy in various places such as allocating slab pages. Large kmalloc() allocations used to do that too, until an unintended change by c4cab557521a ("mm/slab_common: cleanup kmalloc_large()") resulted in ignoring mempolicy and just preferring the local node. Restore the NUMA policy support. Fixes: c4cab557521a ("mm/slab_common: cleanup kmalloc_large()") Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter (Ampere) Acked-by: Roman Gushchin Reviewed-by: Harry Yoo --- mm/slub.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index dc9e729e1d269b5d362cb5bc44f824640ffd00f3..11356c701f9f857a2e8cf40bf96= 3ac3abdb5e010 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4282,7 +4282,12 @@ static void *___kmalloc_large_node(size_t size, gfp_= t flags, int node) flags =3D kmalloc_fix_flags(flags); =20 flags |=3D __GFP_COMP; - folio =3D (struct folio *)alloc_pages_node_noprof(node, flags, order); + + if (node =3D=3D NUMA_NO_NODE) + folio =3D (struct folio *)alloc_pages_noprof(flags, order); + else + folio =3D (struct folio *)__alloc_pages_noprof(flags, order, node, NULL); + if (folio) { ptr =3D folio_address(folio); lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, --=20 2.49.0 From nobody Tue Dec 16 14:37:27 2025 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1AA91EDA3A for ; Mon, 2 Jun 2025 11:02:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748862154; cv=none; b=k1vTLdV048EXZbpq2+wmH8bLyjICNwgNrmWyFsn59G7YIJxMB29tlqPxMg/AQzMDxVpFV8n/MH0PZc1K4fEwWrwbkTTDWK/zvl5UJAMIGrgD+ZRQybM51ZgA23794kzFPYWLmY0WK3gxkCdF+jOFmj33sn1BuYl9Rf/OPoEATyM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1748862154; c=relaxed/simple; bh=APcqzwsA9o4EBJdncCezmMHiwyDknfk8GwAUzFwFqPY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=KqbA1gXG/6UjK2VFcsPdULjR3FM05IG7yMMbCS3hOPwBkHY1zyiIUPss9JQuGgUu0i9PpAViUkIRVE9IW42RngLqa59jkqx+tzD+vX7+ynZKCW+O6KlxaiCMF4AfF4nJ5718DoO5dBPtOET5NNzHuOSGmUQ7YIoeoSXDsQw7Eg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz; spf=pass smtp.mailfrom=suse.cz; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=O/TPJgsh; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=mZEFNmrg; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b=OLd6pquk; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b=wmwQ15hB; arc=none smtp.client-ip=195.135.223.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.cz Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="O/TPJgsh"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="mZEFNmrg"; dkim=pass (1024-bit key) header.d=suse.cz header.i=@suse.cz header.b="OLd6pquk"; dkim=permerror (0-bit key) header.d=suse.cz header.i=@suse.cz header.b="wmwQ15hB" Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id CFC1B219EB; Mon, 2 Jun 2025 11:02:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748862138; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZztYGn3J3Hw8d9RMjSfh9mOTW5hxG+HFPyUVr4o4Ux0=; b=O/TPJgshfUfq1fyDdxPU8Hq2bPUX3ek9sFMZ4FBcOXJ2QTeWj8pMfsGbCTIt6JHowuGDYv /J/XUwWDuUjlBbqAQd1ZD7OpeLEBxyKV0qurjjwkX4MJAqaldxz96aBdFUv4WvpbvGaMes TrkeW/bTqpKbskxbwflbnU8XQl2obYQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748862138; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZztYGn3J3Hw8d9RMjSfh9mOTW5hxG+HFPyUVr4o4Ux0=; b=mZEFNmrgGtn8H93eZ9a5RDjIvBlqh1ZKnv2zyIf+iwRu+DKMsJKlmxMT1s2ZxDpsIrI1Q5 scg7maOjd/iq17AQ== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZztYGn3J3Hw8d9RMjSfh9mOTW5hxG+HFPyUVr4o4Ux0=; b=OLd6pqukHonKrFnlRtGUf+A9pMdztBM48Ym20EE7O8iczKWlybdZ27zgoxMR4zZrQHbXF+ oGQzo5yzGHDG/Xns9kn77vwGyYOQuxvz6ILaUwLGtnJfoVI7vGUHcQ2+I3VLQGwnZOe5sE hdiuQ11yh2utQ8lK5RWaNSyPgc/z5Po= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1748862137; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZztYGn3J3Hw8d9RMjSfh9mOTW5hxG+HFPyUVr4o4Ux0=; b=wmwQ15hBa42uBZfrPFKqZb0eK7nVeiHs3NC91AfoxBMvpXSuoKbWSf9Y1g523gm+Y/tk79 b/Pj3M76DNweDJBA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BA29A13AE1; Mon, 2 Jun 2025 11:02:17 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id KCArLbmEPWggHAAAD6G6ig (envelope-from ); Mon, 02 Jun 2025 11:02:17 +0000 From: Vlastimil Babka Date: Mon, 02 Jun 2025 13:02:13 +0200 Subject: [PATCH v2 2/2] mm, slab: use frozen pages for large kmalloc Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250602-frozen-pages-for-large-kmalloc-v2-2-84a21f2c3640@suse.cz> References: <20250602-frozen-pages-for-large-kmalloc-v2-0-84a21f2c3640@suse.cz> In-Reply-To: <20250602-frozen-pages-for-large-kmalloc-v2-0-84a21f2c3640@suse.cz> To: Christoph Lameter , David Rientjes Cc: Andrew Morton , Roman Gushchin , Harry Yoo , Matthew Wilcox , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.14.2 X-Spamd-Result: default: False [-8.30 / 50.00]; REPLY(-4.00)[]; BAYES_HAM(-3.00)[100.00%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_SEVEN(0.00)[9]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_SIGNED(0.00)[suse.cz:s=susede2_rsa,suse.cz:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; R_RATELIMIT(0.00)[to_ip_from(RLww46cwtmo61nf1fa4wzihnqu)]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DBL_BLOCKED_OPENRESOLVER(0.00)[oracle.com:email,suse.cz:email,suse.cz:mid] X-Spam-Level: X-Spam-Flag: NO X-Spam-Score: -8.30 Since slab pages are now frozen, it makes sense to have large kmalloc() objects behave same as small kmalloc(), as the choice between the two is an implementation detail depending on allocation size. Notably, increasing refcount on a slab page containing kmalloc() object is not possible anymore, so it should be consistent for large kmalloc pages. Therefore, change large kmalloc to use the frozen pages API. Because of some unexpected fallout in the slab pages case (see commit b9c0e49abfca ("mm: decline to manipulate the refcount on a slab page"), implement the same kind of checks and warnings as part of this change. Notably, networking code using sendpage_ok() to determine whether the page refcount can be manipulated in the network stack should continue behaving correctly. Before this change, the function returns true for large kmalloc pages and page refcount can be manipulated. After this change, the function will return false. Acked-by: Roman Gushchin Acked-by: Harry Yoo Signed-off-by: Vlastimil Babka --- include/linux/mm.h | 4 +++- mm/slub.c | 6 +++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index bf55206935c467f7508e863332063bb15f904a24..d3eb6adf9fa949fbd611470182a= 03c743b16aac7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1549,6 +1549,8 @@ static inline void get_page(struct page *page) struct folio *folio =3D page_folio(page); if (WARN_ON_ONCE(folio_test_slab(folio))) return; + if (WARN_ON_ONCE(folio_test_large_kmalloc(folio))) + return; folio_get(folio); } =20 @@ -1643,7 +1645,7 @@ static inline void put_page(struct page *page) { struct folio *folio =3D page_folio(page); =20 - if (folio_test_slab(folio)) + if (folio_test_slab(folio) || folio_test_large_kmalloc(folio)) return; =20 folio_put(folio); diff --git a/mm/slub.c b/mm/slub.c index 11356c701f9f857a2e8cf40bf963ac3abdb5e010..d87015fad2df65629050d9bcd22= 4facd3d2f4033 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4284,9 +4284,9 @@ static void *___kmalloc_large_node(size_t size, gfp_t= flags, int node) flags |=3D __GFP_COMP; =20 if (node =3D=3D NUMA_NO_NODE) - folio =3D (struct folio *)alloc_pages_noprof(flags, order); + folio =3D (struct folio *)alloc_frozen_pages_noprof(flags, order); else - folio =3D (struct folio *)__alloc_pages_noprof(flags, order, node, NULL); + folio =3D (struct folio *)__alloc_frozen_pages_noprof(flags, order, node= , NULL); =20 if (folio) { ptr =3D folio_address(folio); @@ -4783,7 +4783,7 @@ static void free_large_kmalloc(struct folio *folio, v= oid *object) lruvec_stat_mod_folio(folio, NR_SLAB_UNRECLAIMABLE_B, -(PAGE_SIZE << order)); __folio_clear_large_kmalloc(folio); - folio_put(folio); + free_frozen_pages(&folio->page, order); } =20 /* --=20 2.49.0