From nobody Sun Feb 8 02:23:16 2026 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEB6F5490B for ; Tue, 5 Mar 2024 10:10:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633430; cv=none; b=Cdhm/UxbgOgkWAStbKoFYtzLCv/QEK9jy06U+rpJNL0bM3J4lhola/oKBAQgrN2zoSCJZWut55W5B+KmCrBtLQyPYkrFBOhAs/HPhvdnYPQLxO6hsNpRU7xLUyMk1CkmSPAqh0k62UBjpSHzp06D9+pS3u9RuDdfcaISGEWxc4U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633430; c=relaxed/simple; bh=Wyo8wzm4MTDz4JStiYyO080jO0R+E8V6S6bv9ZTR4iA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=geCDlgPACiAHPxeHdBAgCbqRX6CGHA3URgC4UZ/nxjuSWDI4k2NM/1YnQ36S2X7HIv8+2cA3NV2NwlzE02OMsVkE9/bDB293Xnglm+96NuNLgMetdLuNFoRrq9C5lHmbsfyj/IyrluZZOEhged9o4jpF44POgJ925p6uNJOayGo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=VnyWQvGo; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="VnyWQvGo" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3c1ec2d05feso961355b6e.0 for ; Tue, 05 Mar 2024 02:10:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633428; x=1710238228; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=D8rMBFW0tg47V1vhR+hT7ZAeuwLlaibSycxaH+YicYQ=; b=VnyWQvGoaTrXEWac3DWanC0wt1NS9T9vvxs3BxvbDsM5KF/8Nfn8o/Pz4snNDmj72v eby0QoOYbUA1lQlEnKfMK97JUMbps0xP5dJmqaBY/2xaND9S+Z1wBBqRFkZqlI6eWpmp jPmLVRClTxt2ovLmsrd1pTzTPL0ayxxcTBz3M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633428; x=1710238228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D8rMBFW0tg47V1vhR+hT7ZAeuwLlaibSycxaH+YicYQ=; b=EANnZvQfB8GgmqbVsCsrt0J5w7hWUkFQR2+IXdJvayfOYo4buIcJ3CCagItz81SeKg HuvBL4YY0mKyJNqsvRCRM8gJ3qzivTWSLe1qcystKyuh5eiWnhX2t8tGYf7F+WELfdiZ lZ/pFF9FIn2u7kbRYBZo+gDjkS9LIl36b168nDIJW15hBWWJEEvwMCphF3gHggqefZSs XePT6WWZYwClSgXs+9qQSgTBogi6b7eAdUaAc7lj9kyI9dR/8K/8BdvwIvbZIyk/WezY zxbLwsMaNYFehcv+LnRp3WvqnwGdN2rxOkK2QevaRURJ4wHFBmEAiP+Gn5ZR/mXoKJr6 xqVQ== X-Forwarded-Encrypted: i=1; AJvYcCWD55LaSCjUvdGJprqdnOibGYQx+gLJsa8TwtlMOEGpMhoC2hjVd2odLMqXzIBCJxHdy9+/Xj01IA0J5ODhLu5MWHPtqrlFxZ5cChA1 X-Gm-Message-State: AOJu0YwK2PmsFpd1ijGErvPIEr516+PbdIwSjVIPnCALu9c8pbLZey5g b0JAL3Umw9dnJqRHIzxfLHQ85rupAwkycR812NJaDCQcmFp0WGWDFfCnOPgbGA== X-Google-Smtp-Source: AGHT+IGYfAbScDqt9gSvtMMgpvMOmJMAT2r+uZEpZv8yZZyNk7NeKQx1h1ByxW1TZ8pqZAe02uLOcw== X-Received: by 2002:a05:6808:607:b0:3c1:559b:4290 with SMTP id y7-20020a056808060700b003c1559b4290mr1272920oih.42.1709633427854; Tue, 05 Mar 2024 02:10:27 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id y185-20020a638ac2000000b005df41b00ee9sm8692546pgd.68.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 1/9] slab: Introduce kmem_buckets typedef Date: Tue, 5 Mar 2024 02:10:17 -0800 Message-Id: <20240305101026.694758-1-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1794; i=keescook@chromium.org; h=from:subject; bh=Wyo8wzm4MTDz4JStiYyO080jO0R+E8V6S6bv9ZTR4iA=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QdzwZRrBHJ6q6tZF3ljbnIROOFCv0Jfi5y dLhu0dVnMWJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JhRGEACXqOYvigduxdiLg5s3NKoZ5sAGZvlIOApR3AAy5NlVdVm3El7+ClqD7kAsaRASlCZzfCE SnrdJUJTmH8pIf9m0+2frgqPwuRmMlb09607iDkUQPLPN1YxJMR27tBwJWq9BgA3ubUWo+QsdgT YXE7VP7nPQdqREOzraeyi2MHH+U3DNUaP61lbMW6KxmdVKDTNvRohGXZ4t+fDkAx3qPc5WUijJu +T0ZdLwbHIqRgXmTleQTynxNAt+ias93uGlGLgEDjoQZXs3asvnUvukRlD4dRoL8Jw9iQijFq1Q Dt4OD7yUPU+PD4XJr3GZIJhcJtIYM4IVHGlAVfYnKDclM8zlcITotaYHXQpXMql4PslEEE4eao3 2lj3oguta9XJ6ZA8lMy1oJUDczWNwOLRtiE4ocEojO7GA2/sMkSDKFBWiQoXoqqozQ72/GOfjya z0P7wyB5UCUQt13UQzDHMpYOQ/D5+iqge17qSf3KrlYD3bDZTO0UxJ6zzyDqVGApKdVJXmzv1nS 5BCsV9JIyHRHMs1bxaWyexVgrO27nMe+VW7LxuZWBm53lHjxxXQXT0bPu26dEUkeQrRMBN6BrfR Hi8QkUDZmoqSL/wiOIGv/c+qZfMpokFWp+beQoz6lhK2cuoYx+8nFdtG2JAcDlfnXKJ5RU+DLD4 BUBdZTAp9dA4F9w== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Encapsulate the concept of a single set of kmem_caches that are used for the kmalloc size buckets. Redefine kmalloc_caches as an array of these buckets (for the different global cache buckets). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++-- mm/slab_common.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index b5f5ee8308d0..55059faf166c 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -375,8 +375,9 @@ enum kmalloc_cache_type { NR_KMALLOC_TYPES }; =20 -extern struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; +typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1]; + +extern kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES]; =20 /* * Define gfp bits that should not be set for KMALLOC_NORMAL. diff --git a/mm/slab_common.c b/mm/slab_common.c index 238293b1dbe1..8787cf17d6e4 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -649,8 +649,7 @@ static struct kmem_cache *__init create_kmalloc_cache(c= onst char *name, return s; } =20 -struct kmem_cache * -kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1] __ro_after_init = =3D +kmem_buckets kmalloc_caches[NR_KMALLOC_TYPES] __ro_after_init =3D { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=3D42570 */ }; EXPORT_SYMBOL(kmalloc_caches); =20 --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 51F77548EE for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; cv=none; b=lsyaTPSL0yV53SzXKgl6L0Ojy7MvNuP2g//QTv/0hjJdEvzNIFM4XMetI/9ZonunxSSPNuzDRjEfbJDdmA04mzwAfI7oYLbQ///BLmBNuWO8rVxCctDHdLZbCYdXppPtGLSsVUk2sJvDGm9eDb0Nvy5394M/3YKRQrW+5XLqe+s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; c=relaxed/simple; bh=dsYwePKLsCLMzShH9MEQL3PgSxerM2N+02XNBrgQB3U=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g+NV6xvj78AaevbCgRKZTMQeTFb7crZWb2ryEQK5a+4FmeP48qrJK4C4Qqi+GOv9VuiCwcfjEXLJc07ES0TLVvZWb0PvFt2GojJDojeKYVpHZK6Uz2PW83kXrg7XJP1vKpipxjrBwE+E+xd93k8d2O8rNX/zvtqqhxPnA4Wvoa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=DzqiGsMv; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="DzqiGsMv" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1dca3951ad9so52543305ad.3 for ; Tue, 05 Mar 2024 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633431; x=1710238231; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=DzqiGsMvnyuPX9FYpscrIGebjxFCgVqN8A4eMd5SDSMtEnT7/iK1cl5+LkAeASNtCH vq1PFFDl3DXQo6QPaxBMVlxhJ/NOusiSP/iXdVA2gnLBaxqMzUH+yut0IWq+rcNABnzQ 59x8w3H71un7a8F7JQRBvIZ45eZp+iuHOnu7o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633431; x=1710238231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DCac96MRWhgh1/2EEmwcpolePHb0x1wbAjOzVa8BY+0=; b=WryAcyHsIJWh1FXFjw174j5W29mUh4vG7VDk1cY1303tiSVu/2CoKPXarMz7gujrdc LMQxz+m+H308yRSa5RiYu/16+Pf5JiQwDZxPrxDlK4CwE/vYbbLY+eY4VQ+o1F5/k9RW WTnHhNrI58tfKbwUf1Ne4hpAJug5CfwamrbeDR1+vy8iw8RXLozkqtBosugWI7cTciej Xiwq3omvSFZ0jUJkaTjl0TOL36Bzk0yGr3h+frMI/XwCVN+rv5Zr8CD3GGxiQ4XwlTfv oVBmBcrkhETtn0BfLND09DF50XV29sfTLfULg+W7I0ulGqVA4dk6Fr03exKbXEUMrfgy 3eHg== X-Forwarded-Encrypted: i=1; AJvYcCWGYE031X1Dmi/eoWKbxdS5wrixdIXQc3RQn3w77J0hfB5uO9T/6CeGdieP1HrfVc9RrcNw7YTJXfx7a1U0Rmfi5IjnpZGswmzAcc/r X-Gm-Message-State: AOJu0Yy2zDtK0As4AhYRtqOe4SB9fdBwp0YEhmfL3yNzq42mBNOqsXkc 6rQWkfkUvODzYx/ccYwvFbWIdnvHOXKVqowBhCmGWEANXMSGbS2ueyLDsCh5hw== X-Google-Smtp-Source: AGHT+IGQD7egJY3N2jy+qTOLk98RlzGpfSgXtQ8qNR83/EYIZpLvDqeKxbLyOSOxTxFrRo9HzKuI6Q== X-Received: by 2002:a17:903:230a:b0:1dc:fb9d:402d with SMTP id d10-20020a170903230a00b001dcfb9d402dmr1584663plh.58.1709633430745; Tue, 05 Mar 2024 02:10:30 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n7-20020a170903110700b001dc9893b03bsm10063682plh.272.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, linux-hardening@vger.kernel.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/9] slub: Plumb kmem_buckets into __do_kmalloc_node() Date: Tue, 5 Mar 2024 02:10:18 -0800 Message-Id: <20240305101026.694758-2-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5763; i=keescook@chromium.org; h=from:subject; bh=dsYwePKLsCLMzShH9MEQL3PgSxerM2N+02XNBrgQB3U=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Qh5OvUkS2WLPejBFffVYr3X7AdnuJ+g+ss wawW9pNc9iJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jpz5D/90L2rsJJvl6+P+wDTk00KFpS0yvEOEnTFGCVFHdm1gLjjRgBEZOCCPWclrHgGTkc8hBia nOccJ3xAylGH42H9q91nYChjlw/IZUry4o+uK9hKbxXcdC57QMGm38XenTvoOd4Eeqry07ryFmS Vtt0qurq/+5ki/6NecKkeipCrrNoA5RCxIDY0d95UiJV9EO4VZiWoeJ6j+YW6sCHuYqojJUmth8 nASULi9PDpCRxUlhbDJGHsgd7vhSSVKhr+iAVeILIgKu0ABrT+JLWABfwImg0azZt5Ls1q6akOw gnS+llQjZPPJ3vLk3Yqy3vdLff1gpUvad4G0TTJ69rOlneNSxmYTtrZZcxO/7jqHYl3Y3+pZWcI Ag/9AGf2AlCALy0WdUlBXXNfJE4lwywJnBfsr7GkeTtnd8JhTwB2w92l+xuHv5JUmzB5cSk6EOK dLaYXgNTlKnMTIq8eiOLrYhtEZGOpGgC15pVSSGiUBGviTyv+Rdi59+L+NVkvcHyG9z+2qVDc9/ Lvm8+g1VDnXM8KSNOfUFbX48M+wlbPFG1fhjcwxttYIsH2AqUtWu8K48KlXf5ugIsLGlhRQmjEX 8bFw/IhfXee/9Bt9phiU1TThOqIbodEUWlAb14Y/A/0YMUIOJbkeUWTR5mjkPZDaIFUh0MEGkE/ lg0QYFHUdAVLRJg== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To be able to choose which buckets to allocate from, make the buckets available to the lower level kmalloc interfaces. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org Cc: linux-hardening@vger.kernel.org --- include/linux/slab.h | 8 ++++---- lib/fortify_kunit.c | 2 +- mm/slab.h | 6 ++++-- mm/slab_common.c | 2 +- mm/slub.c | 12 ++++++------ 5 files changed, 16 insertions(+), 14 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 55059faf166c..1cc1a7637b56 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -508,8 +508,8 @@ static __always_inline void kfree_bulk(size_t size, voi= d **p) kmem_cache_free_bulk(NULL, size, p); } =20 -void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_= alignment - __alloc_size(1); +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) + __assume_kmalloc_alignment __alloc_size(2); void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) _= _assume_slab_alignment __malloc; =20 @@ -608,7 +608,7 @@ static __always_inline __alloc_size(1) void *kmalloc_no= de(size_t size, gfp_t fla kmalloc_caches[kmalloc_type(flags, _RET_IP_)][index], flags, node, size); } - return __kmalloc_node(size, flags, node); + return __kmalloc_node(NULL, size, flags, node); } =20 /** @@ -686,7 +686,7 @@ static inline __alloc_size(1, 2) void *kmalloc_array_no= de(size_t n, size_t size, return NULL; if (__builtin_constant_p(n) && __builtin_constant_p(size)) return kmalloc_node(bytes, flags, node); - return __kmalloc_node(bytes, flags, node); + return __kmalloc_node(NULL, bytes, flags, node); } =20 static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size,= gfp_t flags, int node) diff --git a/lib/fortify_kunit.c b/lib/fortify_kunit.c index 2e4fedc81621..c44400b577f3 100644 --- a/lib/fortify_kunit.c +++ b/lib/fortify_kunit.c @@ -182,7 +182,7 @@ static void alloc_size_##allocator##_dynamic_test(struc= t kunit *test) \ checker(expected_size, __kmalloc(alloc_size, gfp), \ kfree(p)); \ checker(expected_size, \ - __kmalloc_node(alloc_size, gfp, NUMA_NO_NODE), \ + __kmalloc_node(NULL, alloc_size, gfp, NUMA_NO_NODE), \ kfree(p)); \ \ orig =3D kmalloc(alloc_size, gfp); \ diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..931f261bde48 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -404,16 +404,18 @@ static inline unsigned int size_index_elem(unsigned i= nt bytes) * KMALLOC_MAX_CACHE_SIZE and the caller must check that. */ static inline struct kmem_cache * -kmalloc_slab(size_t size, gfp_t flags, unsigned long caller) +kmalloc_slab(kmem_buckets *b, size_t size, gfp_t flags, unsigned long call= er) { unsigned int index; =20 + if (!b) + b =3D &kmalloc_caches[kmalloc_type(flags, caller)]; if (size <=3D 192) index =3D kmalloc_size_index[size_index_elem(size)]; else index =3D fls(size - 1); =20 - return kmalloc_caches[kmalloc_type(flags, caller)][index]; + return (*b)[index]; } =20 gfp_t kmalloc_fix_flags(gfp_t flags); diff --git a/mm/slab_common.c b/mm/slab_common.c index 8787cf17d6e4..1d0f25b6ae91 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -698,7 +698,7 @@ size_t kmalloc_size_roundup(size_t size) * The flags don't matter since size_index is common to all. * Neither does the caller for just getting ->object_size. */ - return kmalloc_slab(size, GFP_KERNEL, 0)->object_size; + return kmalloc_slab(NULL, size, GFP_KERNEL, 0)->object_size; } =20 /* Above the smaller buckets, size is a multiple of page size. */ diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..71220b4b1f79 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3959,7 +3959,7 @@ void *kmalloc_large_node(size_t size, gfp_t flags, in= t node) EXPORT_SYMBOL(kmalloc_large_node); =20 static __always_inline -void *__do_kmalloc_node(size_t size, gfp_t flags, int node, +void *__do_kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int nod= e, unsigned long caller) { struct kmem_cache *s; @@ -3975,7 +3975,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int= node, if (unlikely(!size)) return ZERO_SIZE_PTR; =20 - s =3D kmalloc_slab(size, flags, caller); + s =3D kmalloc_slab(b, size, flags, caller); =20 ret =3D slab_alloc_node(s, NULL, flags, node, caller, size); ret =3D kasan_kmalloc(s, ret, size, flags); @@ -3983,22 +3983,22 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, i= nt node, return ret; } =20 -void *__kmalloc_node(size_t size, gfp_t flags, int node) +void *__kmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) { - return __do_kmalloc_node(size, flags, node, _RET_IP_); + return __do_kmalloc_node(b, size, flags, node, _RET_IP_); } EXPORT_SYMBOL(__kmalloc_node); =20 void *__kmalloc(size_t size, gfp_t flags) { - return __do_kmalloc_node(size, flags, NUMA_NO_NODE, _RET_IP_); + return __do_kmalloc_node(NULL, size, flags, NUMA_NO_NODE, _RET_IP_); } EXPORT_SYMBOL(__kmalloc); =20 void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, unsigned long caller) { - return __do_kmalloc_node(size, flags, node, caller); + return __do_kmalloc_node(NULL, size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); =20 --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2679548E6 for ; Tue, 5 Mar 2024 10:10:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633431; cv=none; b=t3mHRqkCbPp4ZzHfMj5e2keWbHaHGbXtbepMvPuH24b5Je6ekWW6sYJ7yPcRqpPMgFqoqPj8GvDHR1WzCq83iHKBt9ax/60waL+WN8QqLFtzUiZINL/72eLqsitAcgyPU8cW/Sfa4qGgfFDE2l7BnJ2VNh1HtfW71s6hT6BAJSQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633431; c=relaxed/simple; bh=JqAs2ONNE7kMO16unP+0GI5vQbPsq9Mj6KA7f6Xw/go=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FQ3uWASoT6BsqEBCu6mTBnI3GaRr2F0nGVyNGnvMG5277iZWi/Kueqh/aY9j7jrd+2B62n6eQhaujKEsUJHsg+eqrFuwzv8YFmAaG5Mz0TIiauDuggwzpHDfZlOZGO6krdU92LYj8EmLKBXpDOX6PbP4m9RqaM6mKHS7PtYyvTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=c7s1eTp2; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="c7s1eTp2" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-1dcb3e6ff3fso28228905ad.2 for ; Tue, 05 Mar 2024 02:10:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633429; x=1710238229; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99J+yh1I+zD4qoBsjvOLBvxyDWRgmU0/2arb8UvstmI=; b=c7s1eTp2x2Pq2b9KQvTawnB3//dDRaQv74AoAQ0vFDZMj5jR6j9mTZSvTDYq8Rp2sv VcNa/zs5XOC0eWji8hCrNtjl+TiPm0tuokacJ5QssGeaDCO9TPwyeSAF/DzJI83YdF2B q+oYzDkdnlAHhfPKhnP9lBuczvlEVL2LeUEqk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633429; x=1710238229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99J+yh1I+zD4qoBsjvOLBvxyDWRgmU0/2arb8UvstmI=; b=EKfPgNuh09IGPLb32oxAgF3+NYjjFrAbhx5K+HPVfGOCxbz7Bnmy294Dlh+dELoh9m uI1otR0QeVXQMFxsh4huKx4WfNtCr/z3kfBzWqBbmjnRsDqbwHGsn3l7nCwfiBJmveZR lat2KZyIRflX25vH/WMEjcx+0BBs+Li2XNjnWGJ28kD90sX2CKHnMFzHuDUV1j9yq0Qs Q4L72oMFiM7LQ4QVUjdtwPM7BRoJE6SgemjM0uu0aej9If2s5dQtZ8oRNk8CSlOMm9nV Lk0zRFO+yGtHd50SqbaPaCuOmzr+Q44IagxdYNeKG5wtsBXm5/zGXXprPYWBsWMDJRzP c5lg== X-Forwarded-Encrypted: i=1; AJvYcCXr0emkwGqpQcmS+Q3V3iG2qDip23dpJZxYEWonkifcVqx43bW/3vI0A8lP3NegFa/dIxINYICtinLckNLn19E5YuYIWwYvFEiwXRWZ X-Gm-Message-State: AOJu0YwysDecixirhbWdqABHUJL3zXm8OU8Ni9e1MjzwcYMicrnOXtC+ 9C2v4vLD8KX0Mh/kqq9oCaN3advwtAqEr5OxhNVh4l9tEmvoZvCTvD0v/ElFhw== X-Google-Smtp-Source: AGHT+IFqjHRULQpfLuOIfWcrg86BprEdjHxZ7A04s1os533fRHV67h+yavqSGXCERg/CRb9UyIa8KQ== X-Received: by 2002:a17:902:d4c1:b0:1d8:ab27:d76c with SMTP id o1-20020a170902d4c100b001d8ab27d76cmr1643127plg.51.1709633429324; Tue, 05 Mar 2024 02:10:29 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n15-20020a170903110f00b001dcc3a46a6bsm10081558plh.262.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 3/9] util: Introduce __kvmalloc_node() that can take kmem_buckets argument Date: Tue, 5 Mar 2024 02:10:19 -0800 Message-Id: <20240305101026.694758-3-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2996; i=keescook@chromium.org; h=from:subject; bh=JqAs2ONNE7kMO16unP+0GI5vQbPsq9Mj6KA7f6Xw/go=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QU4fAZ6X0I2VVjtMMEATXAWJJ5dqn0j2T+ P8LqusMSw+JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JmecD/92uSdARNPOF/Ig41R9iblLk9UxCbO1fB2HKnaa/w+yLHjuEWP2fptZr1u4cl7oBi+V+7e XCMsvWK4pkOYAP7NuRX8GU6AxmvUqlZpmQXa283pIL7cWbyoBryIVD0uVp7YjUDWMT3mZYo8kTI WOxJrg2BS22j8FjJ0SmfK28wsBeG/6y4f11iNleBFW5KpXlGaRw2NoFGPv/oYzmVJdpOvno7tv1 SrJhdM++M5F4fg8dy8UZdPFSBIjoz4VC49dlXRzdsM9GoSdV+aakrKnzGaX8WPqr1QTfDcy5qEv RCskJbWRYLonWFp9KcpeVrWeXB43I5oPU6H/DHx0PGqLxhEKdHMseOCtWkAKNPVhtqOufAQLdct 2XPAIyB+CIavIHRn8oZvNB8dhF46U1bKOM9jFISSARfnGEZp8WC4c4OStJpN9nCnwYHPz32sXy5 IFPtLolXYwV9mqfABdpmFQb/QG6+XuvjFWZ+HpxdxuAD/41pG2HrHPS3XlG77pqtqM+/vtWm5LH hf8/tXhzxjo8B8vey28JSnydpkHd63SeElyfCf/JQ/Vbe5wzsRt/oHddPw1OThs95gN/vEjmHXH rP+vV3/i9F7vKu273jzC4MIKRmYWSs3/Lc000Nj3NCnpIp6uaAeGZFpJIzs1h/qLp3rQGBKT8yw TPX8rWd+pV6qhAA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Provide an API to perform kvmalloc-style allocations with a particular set of buckets. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 9 ++++++++- mm/util.c | 9 +++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 1cc1a7637b56..f26ac9a6ef9f 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -723,7 +723,14 @@ static inline __alloc_size(1) void *kzalloc_node(size_= t size, gfp_t flags, int n return kmalloc_node(size, flags | __GFP_ZERO, node); } =20 -extern void *kvmalloc_node(size_t size, gfp_t flags, int node) __alloc_siz= e(1); +void * __alloc_size(2) +__kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node); + +static inline __alloc_size(1) void *kvmalloc_node(size_t size, gfp_t flags= , int node) +{ + return __kvmalloc_node(NULL, size, flags, node); +} + static inline __alloc_size(1) void *kvmalloc(size_t size, gfp_t flags) { return kvmalloc_node(size, flags, NUMA_NO_NODE); diff --git a/mm/util.c b/mm/util.c index 5a6a9802583b..02c895b87a28 100644 --- a/mm/util.c +++ b/mm/util.c @@ -577,8 +577,9 @@ unsigned long vm_mmap(struct file *file, unsigned long = addr, EXPORT_SYMBOL(vm_mmap); =20 /** - * kvmalloc_node - attempt to allocate physically contiguous memory, but u= pon + * __kvmalloc_node - attempt to allocate physically contiguous memory, but= upon * failure, fall back to non-contiguous (vmalloc) allocation. + * @b: which set of kmalloc buckets to allocate from. * @size: size of the request. * @flags: gfp mask for the allocation - must be compatible (superset) wit= h GFP_KERNEL. * @node: numa node to allocate from @@ -592,7 +593,7 @@ EXPORT_SYMBOL(vm_mmap); * * Return: pointer to the allocated memory of %NULL in case of failure */ -void *kvmalloc_node(size_t size, gfp_t flags, int node) +void *__kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node) { gfp_t kmalloc_flags =3D flags; void *ret; @@ -614,7 +615,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) kmalloc_flags &=3D ~__GFP_NOFAIL; } =20 - ret =3D kmalloc_node(size, kmalloc_flags, node); + ret =3D __kmalloc_node(b, size, kmalloc_flags, node); =20 /* * It doesn't really make sense to fallback to vmalloc for sub page @@ -643,7 +644,7 @@ void *kvmalloc_node(size_t size, gfp_t flags, int node) flags, PAGE_KERNEL, VM_ALLOW_HUGE_VMAP, node, __builtin_return_address(0)); } -EXPORT_SYMBOL(kvmalloc_node); +EXPORT_SYMBOL(__kvmalloc_node); =20 /** * kvfree() - Free memory. --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 58556548EB for ; Tue, 5 Mar 2024 10:10:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633433; cv=none; b=a3DY5putNzpxALokEZn3ljogzcYhmmOGzkmj6zls+1Z2lloKKjJYdJ3CZY7SV/xqWRqAFZuLPU7k1Jzs7Zk5UN/MAj637/Ne6lAWYTV3pC/OqnySoURGUdfXVdDw+q5sDYTQNZIPccI9LETpbzw0GFKv6SeE9GQFT1tFBvB+o/Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633433; c=relaxed/simple; bh=y3vf/uqNKTNnOBLgqwE3tYFT2Luy4iHXIHzaxbjeLuY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=im+8owKQT4Uif/08XCglpROvFuX2DrcRiRamvyKTHLBsTKscJxjUnCR4fl/Hdt4nV2aMCp28MygEw+YjvxVKPiM/GH3eYu9t8HAZP9xJTdaA6OzTEjB2ObN2YPTppcegjUq7i9aUre9A0doHCPbJXxqqpvWJkqT0AEwMquJCCiU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=T2v9Wzno; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="T2v9Wzno" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6e63e9abf6aso286861b3a.2 for ; Tue, 05 Mar 2024 02:10:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633430; x=1710238230; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=T2v9Wznov3Rjko+n4hZHVIXdf5g1oiKwCASjTXgI5KOSchhWyill9f5IUjDGxRzSES yEJaQKUmwDPxRp1Gi+t4H51myK2dZ2s/CEHnHOrvUf3BbKMgXb2wseL6eNmws7ZNHLlR YQJ9SxjGN2xqeDKXD23BLrNvq7zMMcWkgGSW8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633430; x=1710238230; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BeUYrlEBxVnkyCc/znLfwIQ49KwaP7QNJJ/emVdVuls=; b=ZnVHJXePSsJakwpYkEfvsZKR1/4Hnh0MBB8Olc5eBRinV5lCRe8fg3cu0eRcRrOGJR 7w59bZyVbGOUrh9akRCLJWKke5hJ14ZkzLLg/SVrKs1Bw8mgyg7PngFaGO40CgBumB/Z hskmfiES70SR5KywygLpVU0flCFD/sGpx0UvTdNKL4hCuecoBkMjmNnsIfAQ8FEjaXur LQcAk7ZsotR/AiwDXjxTxxBzAWuQhsL7keOsF3r6+fEToV4oaw8V29JnikvvTEsEegnz 4yspQJUKeLyFj4/jX8Pl+4h2I6/OelUisHNtW6YXd0BKe0yCs3PbG7scaupXCDkar5xw p1LA== X-Forwarded-Encrypted: i=1; AJvYcCWcMDWuyguqtHJ52RRkTYDAD3pwcAh/KNb1rYW8J+xhOYHwLWHJdDD+DmJVXyC3sX9SXWtJ6GK//L4nMHR4VCqU6EE/GNuflAzGblcL X-Gm-Message-State: AOJu0YyP8IkZKQB0735TtyDl4ZJ+OCEmV6DmhhvOeo0Qxde+6PxASIOd lAtTS9U5qrEEGHEM5mDeDOE/BYQbw1nhpdDoMBHI1ZbzP/P5ZTrUrGy1Oe4pcw== X-Google-Smtp-Source: AGHT+IEA1CSyGq/q0upjn8FhCc79X1Sz1fnRam7a7llwA6f0dlAvS0keaevB9x35pohBL6JFKtxYYg== X-Received: by 2002:a05:6a20:244c:b0:1a1:4cbe:bf74 with SMTP id t12-20020a056a20244c00b001a14cbebf74mr1486585pzc.15.1709633429921; Tue, 05 Mar 2024 02:10:29 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id g5-20020a170902c38500b001dc78455383sm10146541plg.223.2024.03.05.02.10.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:27 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 4/9] slab: Introduce kmem_buckets_create() Date: Tue, 5 Mar 2024 02:10:20 -0800 Message-Id: <20240305101026.694758-4-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=5125; i=keescook@chromium.org; h=from:subject; bh=y3vf/uqNKTNnOBLgqwE3tYFT2Luy4iHXIHzaxbjeLuY=; b=owEBbAKT/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QPWE1fKjC5eCeZH4vakqfDEWTI/IohhLqT GoUV/5ej76JAjIEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jm/UD/jfSpO31HjXWApCYeolkFhb19QZEq8SrIOIqsqGoGZVAU3k6RWg8yQniQMy3yngBQfa2Ww h/QK0A6lEPzXyLI42kPBmua1RHFrT+Vqe3/1WofvSSsTESICw9TsnzitVinrUOM0nQo51nWISXI rTAc2Dqc1+eteEhuO2mBf61hDJZKYlE3dAE5Oy4yiPLQFOXU363T3VyPS1+jHmjCX9a7U/Fwp1L RE0Jo+IrwYWkV7JCru2Lf9QUOZcLZe1kaDT1S8KGCLBUj0qGfSQ0+6oYCVle7ZtWnAgM69VdfEM u0aSFr9DMPQLnL8DW+S2i4Y7E5d0OaG/m+vACFCAyEOjbMvqBotLX2oVqDLe3wNMWZN/GBxUDdu 9D/jUe03FvtktaMo2UTni3Qsx3d0boMxXGArJ5wo+vyGDDDBs8hxF/3ZhnEd58EUf4/WPWb7Hax 2EybOpRinpRqyal2KGNFPDIBOG4YgGqWYgGXRb3pw4pH6eFdr/PLQhggawSo/Wg96qL8WHZRvZ8 Ndwaph2SCDS3DM+r5nhF8mm7Tw5P9exoRn+a0tJNyv0i8CcN7W2IbJjkG9d/LV/uokfH1SLJWNJ VY+Op6bioLEyekWFsA4tTJyWA5wDB2UiYy2llni7suxYh5HYAzl4ULcPr/X6iHSyvXfscabaGjQ iPexMVWCtLcF/ X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dedicated caches are available For fixed size allocations via kmem_cache_alloc(), but for dynamically sized allocations there is only the global kmalloc API's set of buckets available. This means it isn't possible to separate specific sets of dynamically sized allocations into a separate collection of caches. This leads to a use-after-free exploitation weakness in the Linux kernel since many heap memory spraying/grooming attacks depend on using userspace-controllable dynamically sized allocations to collide with fixed size allocations that end up in same cache. While CONFIG_RANDOM_KMALLOC_CACHES provides a probabilistic defense against these kinds of "type confusion" attacks, including for fixed same-size heap objects, we can create a complementary deterministic defense for dynamically sized allocations. In order to isolate user-controllable sized allocations from system allocations, introduce kmem_buckets_create(), which behaves like kmem_cache_create(). (The next patch will introduce kmem_buckets_alloc(), which behaves like kmem_cache_alloc().) Allows for confining allocations to a dedicated set of sized caches (which have the same layout as the kmalloc caches). This can also be used in the future once codetag allocation annotations exist to implement per-caller allocation cache isolation[1] even for dynamic allocations. Link: https://lore.kernel.org/lkml/202402211449.401382D2AF@keescook [1] Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 5 +++ mm/slab_common.c | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index f26ac9a6ef9f..058d0e3cd181 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -493,6 +493,11 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struc= t list_lru *lru, gfp_t gfpflags) __assume_slab_alignment __malloc; void kmem_cache_free(struct kmem_cache *s, void *objp); =20 +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, unsigned int usersize, + void (*ctor)(void *)); + /* * Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building diff --git a/mm/slab_common.c b/mm/slab_common.c index 1d0f25b6ae91..03ba9aac96b6 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -392,6 +392,74 @@ kmem_cache_create(const char *name, unsigned int size,= unsigned int align, } EXPORT_SYMBOL(kmem_cache_create); =20 +static struct kmem_cache *kmem_buckets_cache __ro_after_init; + +kmem_buckets *kmem_buckets_create(const char *name, unsigned int align, + slab_flags_t flags, + unsigned int useroffset, + unsigned int usersize, + void (*ctor)(void *)) +{ + kmem_buckets *b; + int idx; + + if (WARN_ON(!kmem_buckets_cache)) + return NULL; + + b =3D kmem_cache_alloc(kmem_buckets_cache, GFP_KERNEL|__GFP_ZERO); + if (WARN_ON(!b)) + return NULL; + + flags |=3D SLAB_NO_MERGE; + + for (idx =3D 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + char *short_size, *cache_name; + unsigned int cache_useroffset, cache_usersize; + unsigned int size; + + if (!kmalloc_caches[KMALLOC_NORMAL][idx]) + continue; + + size =3D kmalloc_caches[KMALLOC_NORMAL][idx]->object_size; + if (!size) + continue; + + short_size =3D strchr(kmalloc_caches[KMALLOC_NORMAL][idx]->name, '-'); + if (WARN_ON(!short_size)) + goto fail; + + cache_name =3D kasprintf(GFP_KERNEL, "%s-%s", name, short_size + 1); + if (WARN_ON(!cache_name)) + goto fail; + + if (useroffset >=3D size) { + cache_useroffset =3D 0; + cache_usersize =3D 0; + } else { + cache_useroffset =3D useroffset; + cache_usersize =3D min(size - cache_useroffset, usersize); + } + (*b)[idx] =3D kmem_cache_create_usercopy(cache_name, size, + align, flags, cache_useroffset, + cache_usersize, ctor); + kfree(cache_name); + if (WARN_ON(!(*b)[idx])) + goto fail; + } + + return b; + +fail: + for (idx =3D 0; idx < ARRAY_SIZE(kmalloc_caches[KMALLOC_NORMAL]); idx++) { + if ((*b)[idx]) + kmem_cache_destroy((*b)[idx]); + } + kfree(b); + + return NULL; +} +EXPORT_SYMBOL(kmem_buckets_create); + #ifdef SLAB_SUPPORTS_SYSFS /* * For a given kmem_cache, kmem_cache_destroy() should only be called @@ -933,6 +1001,10 @@ void __init create_kmalloc_caches(slab_flags_t flags) =20 /* Kmalloc array is now usable */ slab_state =3D UP; + + kmem_buckets_cache =3D kmem_cache_create("kmalloc_buckets", + sizeof(kmem_buckets), + 0, 0, NULL); } =20 /** --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B951955E43 for ; Tue, 5 Mar 2024 10:10:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; cv=none; b=cagCUchZSxcnT48A3/MY/m9sL1aXZheEGI6DOjPmS6b1hhkYGGwR1Pxm5SPQOH9Towjcm7l8D4fanZXt86d0AiCS6Vp8ppQcHOmUFX5OugneSEBpo+2w8VKlRx3y7c8SR+oySg8mwZQOGMvqBxR9A6xeTmKbbEscanhc95dzRYc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; c=relaxed/simple; bh=qg6+zLyXOvclDNk40orVz0UwK8KHs3WN88KHdjuAoYc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Wz9nOdbyeo5EwPSWpJMnlGrcI/vBFghDXZVXNqjsFzn/C76lP821r+bl5Ygj82xr739LVx5NRrWWdnNqjFJRRmnlGcTJITuO2vtZiG8fcD3AizZ5f1tL610BHAGaLCtigWq2s5sjQq/lYmHM6HIct4b2wLeFjjJdGh3d4HOm5S4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=BItY9Tly; arc=none smtp.client-ip=209.85.215.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="BItY9Tly" Received: by mail-pg1-f182.google.com with SMTP id 41be03b00d2f7-5dc949f998fso405771a12.3 for ; Tue, 05 Mar 2024 02:10:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633432; x=1710238232; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YqypDeuefMJmnQ8sf5GSNnN+8VJFJPXjvaUOYfHLyEk=; b=BItY9TlyKABqFNiQZW+MiXx1NsFwp6PhFbDLdxVqGRrElTOyZtWfsiUE5wy3FGayxs EJiPId9v7hN5/s+aSRYFjGLJ/82HpupxL08simauFrLKoxGjllpyjXtx/GIIrl+W8WQ3 pDr+e49VwbW2A2mIpOfan8YkdOmwhXU8h6wjU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633432; x=1710238232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YqypDeuefMJmnQ8sf5GSNnN+8VJFJPXjvaUOYfHLyEk=; b=PjAfHNuXu5zvpYH2sM5NFh7UnFBlp8U7h3QojayILnJj5hqiOWTNMnZxoryHNSqQLV BcteCkYt7NvqkcJyWSNjfxsl3EPXeOTfFXLLQjnkUioVb9liPktm6pj0T+A16qBtxFCz 78ptE7LckZHz6DOKXdQ2IFuDtR2ozPXOlhQFSRYybbkXTyKwlDYu9EpcCr6L3eH6/VYr UiyZ5DnB6P1FR2KzF9S9BpzQ2ygKxsSukg3rr+iA9zk3mWa9ea1cDm3dR3wSSxuoKHg3 eKswdEJT2akLtQ2YVep0T/XCMIYz8puhe23EKFd9vAW4p5mdjaP9nB2xGLdYEiQ85eJe Ng6Q== X-Forwarded-Encrypted: i=1; AJvYcCXVbjcwLEllDf9exXAbJjUSOiq2Vt2J8A8EjFoSDghBeGRjdXB+QMlqfhMIoyfibewYR3RSmKLE5e0yO9SvrrYz1aOzu0GSldqsbNJA X-Gm-Message-State: AOJu0YypvsxqNaPNnw3ANH9VyizIGimivx20HiJubf1j9CoBJKlZ96Gz k7d42OcYNZZ7W3SqkrcN/PZKpmdijN7EVjh5Lc5AYJTqSXJKJYCtJcRJyFXlBg== X-Google-Smtp-Source: AGHT+IGeJZyVJ09gI9gdMeeZa2TNl9QqRwg39EBpS2gnYKvxtqTdu16IIH+Z1sVU6rZsLUOu3KxERg== X-Received: by 2002:a17:90a:2c8:b0:299:ecd0:483b with SMTP id d8-20020a17090a02c800b00299ecd0483bmr8668816pjd.10.1709633432154; Tue, 05 Mar 2024 02:10:32 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id h8-20020a17090a3d0800b00299be0e00c1sm12409317pjc.33.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 5/9] slab: Introduce kmem_buckets_alloc() Date: Tue, 5 Mar 2024 02:10:21 -0800 Message-Id: <20240305101026.694758-5-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1274; i=keescook@chromium.org; h=from:subject; bh=qg6+zLyXOvclDNk40orVz0UwK8KHs3WN88KHdjuAoYc=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QePW1cikxrAkyTkNPDsM7P6WLLogreKw78 DkUscrwcj2JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JkIeD/9SGhc8L8n+b0l4YrTF1D9VoErEfn1Ca1UNFW/W0gggb8pmBG/YuAzR/noRa8vP/Obacdv xX/kz3RVVUNa5CYWF20lMDU/Lp8oLF05rVXs0+FZ7Y+Hj4NcwGb4ZDEcTwKpevYm7+jo9WEDQ5d zMSamVXJp/wkyr1XfGZ7cxjHlDRKSjdplhbJgG0UpihMCnecqlqWCXwJbsKgeaEq4Sl26MCcrlM K0ULjcAvpm6F3d4s636IjEqcvuIG1obEke1QFfbVJC5/dJduyyEKm1Lgraw5dGP56FR+aC56qjS 65EMTIxf1kNAW+Fy8bPuL7WDntcwNqNENqpozW05vn4+JPT/wOhqoVmXjQZ3DQfbw1ikOaWsuGR +wrVI1e9aMbAxyN+NV6Dk09hIjGge2x1xZ4ikVrEqfYiJl/PlZG1rZ3YtAjKzVhhHSlziq1EXO7 eLtSu666WzevVuCoqb7mheOeAM4BFhlrgwfm+ZnWU0YM/Ycmrv70Q8134urLdvFSqmvc8qAL1CF xi+VJmmdh42aZ/SeMDt4C3lJogV3xKZGfmixjPA3HWDF8q7OPo6tVZ0GK5r+TW9BmbGeL5G/Mz5 ehqYwoqpMNk2S8s4t7WPvdOakD8dz8vt+onslIuOlwmQgDswizHCZFv3m2uwtvVSPNlj7ZIDrUi W3INC3N8UCVkL0g== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To perform allocations with the buckets allocated with kmem_buckets_create(), introduce kmem_buckets_alloc() which behaves like kmem_cache_alloc(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 058d0e3cd181..08d248f9a1ba 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -600,6 +600,12 @@ static __always_inline __alloc_size(1) void *kmalloc(s= ize_t size, gfp_t flags) return __kmalloc(size, flags); } =20 +static __always_inline __alloc_size(2) +void *kmem_buckets_alloc(kmem_buckets *b, size_t size, gfp_t flags) +{ + return __kmalloc_node(b, size, flags, NUMA_NO_NODE); +} + static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp= _t flags, int node) { if (__builtin_constant_p(size) && size) { --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19A2F55E54 for ; Tue, 5 Mar 2024 10:10:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; cv=none; b=Swb/MJ7VPwOk3Le6V959vcV44pFWaxznB+JiffB18uzAWilRkalduajsfo8NEadzBrWx7N+NONLwYDmzU+kUZ3SLiHLNX2CCCZo+5KXS+AhzWL6CHAa6wtcECJHDfw+fpz/SCCe0tgSJFQe2ZODzML39x/9pMEBqNRIA3Rz4lME= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; c=relaxed/simple; bh=5+bllf7wQXoQ1iiymcQxAaVJUnLpEaTtP4WhLyfCGPk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qI2p8Z2lbhQvfE7BIs+cBZMfop8th++hz8aXAiQEzDrk7H1KvRRoFGQAG+PifvuQdVvNQbEnOhmU24MfU42rQTAadxorO6NpNpsf1CihTH6T3n1booACXPWfRczDmg6Hrq8D+dG1ZLT4ylTx+BpS4MlzxhHApRchOqGqawQCbIU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=Jq5k6fQ7; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="Jq5k6fQ7" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6e5a232fe80so3406168b3a.0 for ; Tue, 05 Mar 2024 02:10:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633432; x=1710238232; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=W6tSmWQUz9mHEs7KKdtWRSqXpZBjknnUveJhwT+v2FU=; b=Jq5k6fQ703V9hULNmaMj51rmkGH4CZW4hnaqyjwDdexzJc3KRKc3FZohTAuw3xhVzh NAVVN1Lmmtv2M2c2IIeawk+kfjYC5A3Ez3gi9VSBRvidR0CoSLFmZ1WywTNWOPe67oA+ pbNEBGHPBJYMKL7jN67cKd9iWt4tbRxpFRel8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633432; x=1710238232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W6tSmWQUz9mHEs7KKdtWRSqXpZBjknnUveJhwT+v2FU=; b=dS/d70fokrk+tcT2bWBEFJEf88RxsrQsqWDNBsD1VE+SPbbaXhxefEqVeuueqYG+01 oV3xE6oZzO/8EzO2pKzUIVJsEsxGfZ0oPV6N3kDDJ3+0icAzhsNYMdHDFE/0HWQok9ot J7ElX+W6JGav3z0Km7sBNenuvOCf3p7cp7R4LSNgyGzkpg1xkSBjLFoJKfWYxY+rx1d4 pvljXABGbWQGNTPIANg7PHNYqOP76S1uMqqw/IHsrJJCDXN4cnh3YXNp+WFeAGoI3s36 qIjEPmtB0kW/5/E0+GW7W0C9qzs5QwY5DCDHXmw+xfPwlCi/mdfNHfvm8xEINVOT99As 7OVg== X-Forwarded-Encrypted: i=1; AJvYcCVJS4k0t5USgM7r4U21+j9MpvtzWzi0B1SkxY0ssPo2k3NSnc0peFVYZIq2WHuJ3wz1HWKW//minNLIgBgxGyFtM4f9K9LY84h4XWj+ X-Gm-Message-State: AOJu0YxJS88U4FBaieFeCwgLrnnl9IIK5kHd3VmEfN1hCeQr57NA2CCe WEQ8fvWJ3NGLFKylPHhhYO0eGEgNDtwirqb57aTUEsRH1Uqd6qunp/Fnq+pJLg== X-Google-Smtp-Source: AGHT+IGioGxnLf58krzooyXRSux/Zxx7hWwNEvHeCXchceFMByRkqxZ+PaMIxnsL+kY8mFgOqRxI3w== X-Received: by 2002:a05:6a20:729a:b0:1a1:3b27:4f3f with SMTP id o26-20020a056a20729a00b001a13b274f3fmr1585251pzk.2.1709633432471; Tue, 05 Mar 2024 02:10:32 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id m7-20020a62f207000000b006e580678dfbsm8485054pfh.193.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 6/9] slub: Introduce kmem_buckets_alloc_track_caller() Date: Tue, 5 Mar 2024 02:10:22 -0800 Message-Id: <20240305101026.694758-6-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2914; i=keescook@chromium.org; h=from:subject; bh=5+bllf7wQXoQ1iiymcQxAaVJUnLpEaTtP4WhLyfCGPk=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Q9Fez113NbGGVqjvVTslyLAH8TI+6Wb4a3 UHTy9ESMGeJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A Jum0D/9tfCyWy09BzH1L9P4M66z0u2ctSfRJPeQ1x4bgaEnW38mkbjG41ME6e+R5Zj0gpmBgZ+L BVNaHnefEXko8smuTRzZ4FGGksACNwKf0+N86253pNgtssxWUfgfa3aQ1EA8XEBCMceHWGnr84H h34x9bAbwoUCp96M4lvRDs8cKxVlleF0jjCvDpvMT3T1ZAyvoPvNHYpvj6mth+oFIZTSkt3Wz+X 919IaLbl/IDuiJVd9rtpALExUWOh+KXL6FuuwYZHWN+SaSa712xlgG/5pLuJ+2l8Ub1IJiEQVWp r/4azSgELrhUJFvtIbkEWWKreqYhRzNs/rKZUmxW7K1gGKiGdDk4ZDqhqhclVbw3MMBNLZbLerx rSCyI/khhcdBGZYwjg6IFEY/vpFVkrL44xPf13VdK07PEsrsff7tXBBEX59HFI2KvQG5OsvWVhP aHHbbGxMkh8n+z9aHH4WpY49kUBjUPRrASB5fIz5FsMzXVgyemQHiUSYTVdkJ3aT34AdKNF6pks Immb98LUo/rCvKgXzxq/u49qS0zQkVpV4BWtD/SwK4MuhHIOU8Rm9ytXz8ZSxeZ0v1pYc89qObI o3aW/HpIQI7IY2DuYh0+o+TW5k0hDDZa00o0vGiCTiQIKa61gmW3pYCwmM5FPO2U0gagUkJ30W6 V9QFinQCp7WdukA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For better capturing the caller details for allocation wrappers, introduce kmem_buckets_alloc_track_caller() by plumbing the buckets into the existing *_track_caller() interfaces. Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 11 +++++++---- mm/slub.c | 4 ++-- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 08d248f9a1ba..7d84f875dcf4 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -606,6 +606,9 @@ void *kmem_buckets_alloc(kmem_buckets *b, size_t size, = gfp_t flags) return __kmalloc_node(b, size, flags, NUMA_NO_NODE); } =20 +#define kmem_buckets_alloc_track_caller(b, size, flags) \ + __kmalloc_node_track_caller(b, size, flags, NUMA_NO_NODE, _RET_IP_) + static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp= _t flags, int node) { if (__builtin_constant_p(size) && size) { @@ -670,10 +673,10 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t= n, size_t size, gfp_t flag return kmalloc_array(n, size, flags | __GFP_ZERO); } =20 -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node, - unsigned long caller) __alloc_size(1); +void *__kmalloc_node_track_caller(kmem_buckets *b, size_t size, gfp_t flag= s, int node, + unsigned long caller) __alloc_size(2); #define kmalloc_node_track_caller(size, flags, node) \ - __kmalloc_node_track_caller(size, flags, node, \ + __kmalloc_node_track_caller(NULL, size, flags, node, \ _RET_IP_) =20 /* @@ -685,7 +688,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t fl= ags, int node, * request comes from. */ #define kmalloc_track_caller(size, flags) \ - __kmalloc_node_track_caller(size, flags, \ + __kmalloc_node_track_caller(NULL, size, flags, \ NUMA_NO_NODE, _RET_IP_) =20 static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t= size, gfp_t flags, diff --git a/mm/slub.c b/mm/slub.c index 71220b4b1f79..ae54ec452a11 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3995,10 +3995,10 @@ void *__kmalloc(size_t size, gfp_t flags) } EXPORT_SYMBOL(__kmalloc); =20 -void *__kmalloc_node_track_caller(size_t size, gfp_t flags, +void *__kmalloc_node_track_caller(kmem_buckets *b, size_t size, gfp_t flag= s, int node, unsigned long caller) { - return __do_kmalloc_node(NULL, size, flags, node, caller); + return __do_kmalloc_node(b, size, flags, node, caller); } EXPORT_SYMBOL(__kmalloc_node_track_caller); =20 --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D530054916 for ; Tue, 5 Mar 2024 10:10:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; cv=none; b=aDW41CwPfiKKEomTNNpX2FcjRLv5R9JYbXB/LydEK/Oh3Pba370SP+5w9oZrddeS+9u6eCu9zD8w8Q7qbHOmMxh4kiMb2eGCpJfjJXasvljiurs3S5s/htnEejBCKfOnFYyugNPmRUnj4P3xAVjeCeVe/qRv6u5WjxhILFDsM4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633434; c=relaxed/simple; bh=k3ROePG0wD7icc8xiWOerLraaHU6zkW2VMYF75ammOw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M1LxKhlNhmfodXmg3nG73mTrnF8+Na5DGXczB74hX1QXFCVP3USw9bxzxQFj6+6qFx9dFachpE3twHcspMxnxMl3djf/+Omcdmc15ZJXENrkP6KGeH0QbNeIFLtjaAKIyKKjUAwnVqWP+i27Jh7BNEb8VXk3BbY8gV/FlGSeAt0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=oduSlXmU; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="oduSlXmU" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1dc3b4b9b62so46673365ad.1 for ; Tue, 05 Mar 2024 02:10:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633431; x=1710238231; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J4oJMKLdbK31ulnxzAH51pNhqUof8xsZThnS45kbbx0=; b=oduSlXmUp03ACFepizrKbxgClSFCuTX3N+CRwNCngRnKw41jIiGqGjWav9PPs9KZYZ mNUmT0Q3222qygl56XeEDfMDVm8XsWYRFS4+oH2OhtoOd4/gMNL5DIK74vBvQ7l0/x0o g3q7cPDRbRp6KD7o5BUGTv/LpqjjClHFVwPNs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633431; x=1710238231; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J4oJMKLdbK31ulnxzAH51pNhqUof8xsZThnS45kbbx0=; b=FYk3PLpwQyh3dGNtAKKUAp/Yc6sAVPQmCbmW2+XCvCkjTGtVrsuoGBKFspdicMURHs msrij0CjzvUiwRhBPuBIOzrrI3+oPk3BTbDXkxOEKBQeXgAGDu3g5xzxhfUCyxnfIv3V /DVFF95UTsRtiUpMq/sxi+BWYJHzpLE6KSxRbYU1RSApGMCBn7f5PQejopsGAKbZg5CX M0lA1bDkP+1zLblO8dYRJlgtd61vFIcY/lxx2w9zIGUyhOhVc7EIDHUuH4x0ybtOsC8c zKnRxfEHPpOou+9O0duHjPy1pVGJr7NAgmOkwez4nYQ2G8BofjRkVaLWDoQ7rKZ8rQ7+ PzEA== X-Forwarded-Encrypted: i=1; AJvYcCVw/5GGlEEljq6zahec51BymdoyOArIM5LwdAy14HfPdx0B7yRk/davwTmm72WfqKJZ9OvWNODpX6zJELiKEyLkmK92kyWqrOTNBczX X-Gm-Message-State: AOJu0YxWrDJ3ZoILIxdCrLYJNNrVe79IeRKZBIaGJTXzkdONDFtx2a+t Vg9NEN1j20xhqD6SC0E8+WcudO4/tKzaEzl20/1w1u7tGgnMq51CWFWxKNOb6w== X-Google-Smtp-Source: AGHT+IHrjjh8uwOGfKisJHpLEa0Atyc/w3DC95jENvS8m/9tWBeTRxSog9SPn/yX99K3zySiGpAi+g== X-Received: by 2002:a17:902:c946:b0:1dc:f989:3116 with SMTP id i6-20020a170902c94600b001dcf9893116mr1689963pla.18.1709633431171; Tue, 05 Mar 2024 02:10:31 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id b4-20020a170902d50400b001dcf93e90a0sm6282358plg.20.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-mm@kvack.org, "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 7/9] slab: Introduce kmem_buckets_valloc() Date: Tue, 5 Mar 2024 02:10:23 -0800 Message-Id: <20240305101026.694758-7-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1268; i=keescook@chromium.org; h=from:subject; bh=k3ROePG0wD7icc8xiWOerLraaHU6zkW2VMYF75ammOw=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Q3E2uwV8NIpzb5dXDsQuDADvQD8oEdZqqt GBc07SpeUKJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JkIREACdPvVBaWa/0ViOnT81LdqxheGCGDtUUmVO8IhLpzHQI2rh/b5CfK7msJHDhECQIcRrSdi 9nheGlO4Ww0GdAzyGQiuG+uKl+8tPoMsEDmLN9mYMJzOFVR9qtmtiAj8CWKbMecilAJj7/jLqds iCJqrBzYT0DBMPzR+RZ9sy4+mqmsXB8/VLVTfQt56mISQNPQ4HxWJF2ILoadyqB9vPcefR9U1EU OzMaLq1/npProPdec6qADdEauw08Z36eJCQphHgRvJ82WAFVrMNxpshJ6EMEpBpK8K5rISvVsar aqXEgU6wRqJfn9CHNyLBKbNEsewc+MTNzTH6kW30tWM/d0EuaYIykGXqwZFz6ALQ8Drk6s3yo+u KN1Nb+BF9a2zqYfrY9DNDIRvNra8AzrHVlKP3RHbTjzJDcvZDyjoEkPmDR2fCR2H+Z0kdexLb2B kk584A9xUsXzzD2J+uVGtk6XiiRF9RMheuM13Hn5v4gt4o/evttZNtO2TvSMEgiYlV7G77+lk27 THKPCHQ33k+oMkMYU93Eun2edSp14I5OInVVbt0+av7+/C3NRYD3Li2IJNOs4cX4OpXAliq48bk jKoGMk0FoWujDweGgEaojZtYDuVb8Uyxqw3TBzo2P0/2LUFvkVfm1Ovlo8NLjCSYQ6x2M+f71mk 5bgLk5y9jX9jonw== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For allocations that may need to fallback to vmalloc, add kmem_buckets_valloc(). Signed-off-by: Kees Cook --- Cc: Vlastimil Babka Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Joonsoo Kim Cc: Andrew Morton Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: linux-mm@kvack.org --- include/linux/slab.h | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/include/linux/slab.h b/include/linux/slab.h index 7d84f875dcf4..0cf72861d5fa 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -740,6 +740,12 @@ static inline __alloc_size(1) void *kzalloc_node(size_= t size, gfp_t flags, int n void * __alloc_size(2) __kvmalloc_node(kmem_buckets *b, size_t size, gfp_t flags, int node); =20 +static __always_inline __alloc_size(2) +void *kmem_buckets_valloc(kmem_buckets *b, size_t size, gfp_t flags) +{ + return __kvmalloc_node(b, size, flags, NUMA_NO_NODE); +} + static inline __alloc_size(1) void *kvmalloc_node(size_t size, gfp_t flags= , int node) { return __kvmalloc_node(NULL, size, flags, node); --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB99758118 for ; Tue, 5 Mar 2024 10:10:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633438; cv=none; b=Y5pLvzItsUOnLS0MAJFHzNrX8x488HM3a2AlY9dWCUVA5X3jjXWMJwVs8nuEU3DjjXus3kealLfjKyNmDfnAjohm9skEBtNvA+9FkF5roAZc7C+n+Htif1M0RuSsIoUbyqeJ2QdzkIFnielyom23epYmwfdQg786o89EEdtN2sE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633438; c=relaxed/simple; bh=Ltfy4agIWW8GFQ2wB/SGVqMVh7NAol3Xo3zP80oOSxA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AlQMOFPlMGbqqX3NUq28Wvu6GTTE18WjwENgGbKTaneUIsWobWvrZK8FwOhHVlzM04jmPo8ivmg1vTauQk6KPCbp6CNOgGaFgJ9LFKXrUutBXDuShZ2bSvlha3PK5dFpQbiY7OUFkc0kPRkmrevFyaYWsQiR2j0KiI7shbeTdTU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=bR1OojoN; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="bR1OojoN" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6e09143c7bdso3678668b3a.3 for ; Tue, 05 Mar 2024 02:10:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633435; x=1710238235; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pnsMPssoThTQkcXoWA9Ekbi5Q1zZTU79wL6KH4AEkgQ=; b=bR1OojoNY9U4kBa3O3RTA/fUHlIT/7+MWl5yOH6dqCjfqG+w44d97LtHsK1Q9TEIX4 1JQQBRt2bGO+qAO5ydtCnY3axo8YGLtYN60zIKUoKUYzHS3qjcNL5mMCoUYYGxDHsYkX Pw9RO/oR7D+tQHgLT92MlZypeEfkKm1CX6leM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633435; x=1710238235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pnsMPssoThTQkcXoWA9Ekbi5Q1zZTU79wL6KH4AEkgQ=; b=Dv7SYbrY7SFlMoF5UozzHLqjdiR4Pf0MgZa/DEukERbM1XWnvpWkL7Z6py/Qo4r52R 7Emz7VOjJ36HuYD1piqyDN/L8fuvV9ISas/LemSmQxpKAn7gHNntfi58x5F6PmlZPB4X ShHmdrDhNdAmkFKmAUBrrITvXtJbqOlyhzgSj6jOzmHqAr8gfmZVZNwki0eA3fcdGh4Z RzKURM/16qRZBiS+BT/5sZTJe8SOTrl7LhhmhWqOG9iZBI04DH04D/s8fV5f2FdXkfhe H7dTREMhfI0IHfAhafS8Omrr0KjVoAvjCU3yB4MakLqenJrJqBezKk0d6x9DAZKGO1++ PZUQ== X-Forwarded-Encrypted: i=1; AJvYcCUvRb64+TgYCIDfszJbvsRTCNSUeprwslcgsuC+7lb6aRCxbHxTTvh7KYZ8luzl/c3d2y/ko5COKJmJ26Vk88S55/kCemENPVNRxXyv X-Gm-Message-State: AOJu0YzVREUHer1F5Fr+EOJ68mO6QHbrgYtUSqb/mOv/Uv75kW9Y4072 hdOBpJ/xymZ5i3NCT+pH9niZiTMDB4lxcvP/Hy4Hhk4kYoHgGetAFUv33A0erA== X-Google-Smtp-Source: AGHT+IEiAWTgdgJYXNyV1wO/0LoKLC3IKAkw+TtluHY1zuMxHwA3vOiKJTRV4nyNTDEQlhJfiH3Lwg== X-Received: by 2002:a05:6a00:2354:b0:6e5:80a4:2ff2 with SMTP id j20-20020a056a00235400b006e580a42ff2mr12396375pfj.30.1709633435073; Tue, 05 Mar 2024 02:10:35 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id n9-20020aa79849000000b006e61b0a4b17sm3625323pfq.185.2024.03.05.02.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 8/9] ipc, msg: Use dedicated slab buckets for alloc_msg() Date: Tue, 5 Mar 2024 02:10:24 -0800 Message-Id: <20240305101026.694758-8-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2182; i=keescook@chromium.org; h=from:subject; bh=Ltfy4agIWW8GFQ2wB/SGVqMVh7NAol3Xo3zP80oOSxA=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+QWH/R4wFO4/C60XHEU8ysAaQUzkDa/3+3d QWUIDhM692JAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkAAKCRCJcvTf3G3A JrvaD/wNahnrlFTqZJolsZvbWW0JKOIUHvIZ9tu5KXbYjElEyGOPqLCj2ubk3S/Q6iA9219qfa6 pGqd/gb14P9uZyXmRoI9+3bS4VDQCYU2pVYoTkpLeg4mS3bLPisEIDg13VQAB8yGYFFzgx570th kBc2psi8XalZRDb+22nGBoKA0hP91pQ00G0kJzgtIobHvLf08qFlMvNbmsE5Pme/RYM37z+uKJo 6Ht69bJWQXxP3brIzWYPe9S0kO8dNubvglHpUetBciWXQNmWqzDhkprvcsUARwd7gUvgA4Zl2bU BvI0Fxk7BBq4ju+5KUIEligRbrLMIOn2dxUIdSfr3k4opmEl5ETlonQ+wt0savTeYPsbW/lgGwn v86/q0pgtJ/5Oxcd8pnWLe6dDPg5ZGQ00IiZ9h1x5iSLqXFJ1MFi/B+uFSaLnK0guFBVXHIUCmV I6mhgVKYMZALTQfYuz9ZZDYkV9pTJCysElPoennpSXdeUJ7Pc/edHh0770uIOALNa6rXB1b+K6i Yd1XOLuB2X7tDGZOo7NgNDNoxOX71alUQybWMW+kUfUFYB5XoVlC7XOD3Ax3ixm0n6sjPCVCuY/ t2ge5/Gjz6GsJY+hlaCoZ/t+yhtmsbD3m+Gu2eA5hteP/AAmSNJGiBGNGgQ09SV6+yyLUeBYps5 sysZ56s4nY6VilQ== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The msg subsystem is a common target for exploiting[1][2][3][4][5][6] use-after-free type confusion flaws in the kernel for both read and write primitives. Avoid having a user-controlled size cache share the global kmalloc allocator by using a separate set of kmalloc buckets. Link: https://blog.hacktivesecurity.com/index.php/2022/06/13/linux-kernel-e= xploit-development-1day-case-study/ [1] Link: https://hardenedvault.net/blog/2022-11-13-msg_msg-recon-mitigation-ve= d/ [2] Link: https://www.willsroot.io/2021/08/corctf-2021-fire-of-salvation-writeu= p.html [3] Link: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html [4] Link: https://google.github.io/security-research/pocs/linux/cve-2021-22555/= writeup.html [5] Link: https://zplin.me/papers/ELOISE.pdf [6] Link: https://syst3mfailure.io/wall-of-perdition/ Signed-off-by: Kees Cook --- Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo --- ipc/msgutil.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/ipc/msgutil.c b/ipc/msgutil.c index d0a0e877cadd..f392f30a057a 100644 --- a/ipc/msgutil.c +++ b/ipc/msgutil.c @@ -42,6 +42,17 @@ struct msg_msgseg { #define DATALEN_MSG ((size_t)PAGE_SIZE-sizeof(struct msg_msg)) #define DATALEN_SEG ((size_t)PAGE_SIZE-sizeof(struct msg_msgseg)) =20 +static kmem_buckets *msg_buckets __ro_after_init; + +static int __init init_msg_buckets(void) +{ + msg_buckets =3D kmem_buckets_create("msg_msg", 0, SLAB_ACCOUNT, + sizeof(struct msg_msg), + DATALEN_MSG, NULL); + + return 0; +} +subsys_initcall(init_msg_buckets); =20 static struct msg_msg *alloc_msg(size_t len) { @@ -50,7 +61,7 @@ static struct msg_msg *alloc_msg(size_t len) size_t alen; =20 alen =3D min(len, DATALEN_MSG); - msg =3D kmalloc(sizeof(*msg) + alen, GFP_KERNEL_ACCOUNT); + msg =3D kmem_buckets_alloc(msg_buckets, sizeof(*msg) + alen, GFP_KERNEL); if (msg =3D=3D NULL) return NULL; =20 --=20 2.34.1 From nobody Sun Feb 8 02:23:16 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BEF85677D for ; Tue, 5 Mar 2024 10:10:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633436; cv=none; b=bZ/VypBwpz+mdGx3aAe9haof9FNMF7Adp6RvFZz25z9AYPAU0uvMe1zVenrlUhjyWUDgB7nvQm+lYf+G76SgZQs449aqYjP9SXZnlafAUzbAoc+05/KlcLnYFEHuxg+1UtHvElS5nYkCS00eskQKK53bWZgE9Zd0FuJzUxzdBf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709633436; c=relaxed/simple; bh=6qDL+BboHaqcTpZSUCl+S9ocvneIfXXeMs3ajiQ2O/Q=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=DlibnD2ot1RHPaz/DPqM/52pYerfOiD8HdScj2/69nxKbyHRxvqnc0usxWTs0v/EMrqiK3uRQt48srnOM3V6ifJSNw+AibO2GgP5lK+nzGhHbtNDpihP48h5aza9Wmd6FniObIHKV8jKwgBzimH0TxR9Qg79rtRSHJDzJV0ps9c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org; spf=pass smtp.mailfrom=chromium.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b=afKW85Vc; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=chromium.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="afKW85Vc" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1dc13fb0133so40526895ad.3 for ; Tue, 05 Mar 2024 02:10:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1709633434; x=1710238234; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Jdg+vWQ/glPg3PLzwoChyf7fue9eFP8PVvp8lsS5Lok=; b=afKW85VcRubI10YjZUBeM6xATHrPsAt3P4VKvRlK8VNrcMjYzRcdNQdpTMDAqmQ3ax kdq/syHQW16HUT9dJ3yWw/h71pEuxNL7YsormhU1ulz6uIajLI3lCvO0qwar2NIEOSYB dpPEwMfdU6sRJPZISTZedRn/y3vAWy0ijnM9g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709633434; x=1710238234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jdg+vWQ/glPg3PLzwoChyf7fue9eFP8PVvp8lsS5Lok=; b=aMDDsS9ddUPrSk8PaurInONDcB0M3ZMxgJnS+8xzFJIcolyXDzVsY4EKtXjxNmvNPp vmTlq+xz8KsK+vaB+KEhXlnouN/F1FaWuej7Bgif/byJkPi5eW9C1/9Qg0lQc8Bor3fh mJcexaZHJ98tyZKBKYHPFOSaaflbuuF+20QUMkrJdRKe1Z7KgroTMm/UMmXgLhF4tvWQ 5tAiZBaPqVhRC28Z2S2JdAJQThWWRReOxxLdo4MUx8DzM/7sZfGxxAFWZRF4XMO/jpNz hNEhqCDI3voZHGPJBhXkVztUx5TC+mQzNWINpH5+S+4l8mU5B5ZxANlEB/D2vaBRsVbB cPeQ== X-Forwarded-Encrypted: i=1; AJvYcCVaO90Xq6dhKmzYobM0AAceU4HwoF7emrx6b9uJZ02Md3uC8Zevw0H11b5dj3F9r/hyJA/FLVe3Oh+lTClBl5QAaXvnOEp1/oqFeDoQ X-Gm-Message-State: AOJu0YxuR3T5huvblxt/qqGDZxsB3uaR587THIqWG9RMwAXcAPCGL/4o jDH/WtPY5KF55YOlP+2d87H2u4UAFqmtjDvAmyPDdOfoY5FaSOX8q/vTAoZubQ== X-Google-Smtp-Source: AGHT+IHYCn9DBg9xzTvpQ8G0/N3dK13ZQWLK1qCcLYJXKOoBEAWGORs21YgWZoWg8YTIQaCZeo4VAA== X-Received: by 2002:a17:903:2287:b0:1dd:7da:e0a9 with SMTP id b7-20020a170903228700b001dd07dae0a9mr1791845plh.69.1709633433734; Tue, 05 Mar 2024 02:10:33 -0800 (PST) Received: from www.outflux.net ([198.0.35.241]) by smtp.gmail.com with ESMTPSA id q17-20020a170902e31100b001dbcf653017sm10134281plc.289.2024.03.05.02.10.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Mar 2024 02:10:30 -0800 (PST) From: Kees Cook To: Vlastimil Babka Cc: Kees Cook , Andrew Morton , "GONG, Ruiqi" , Xiu Jianfeng , Suren Baghdasaryan , Kent Overstreet , Jann Horn , Matteo Rizzo , linux-mm@kvack.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-hardening@vger.kernel.org Subject: [PATCH v2 9/9] mm/util: Use dedicated slab buckets for memdup_user() Date: Tue, 5 Mar 2024 02:10:25 -0800 Message-Id: <20240305101026.694758-9-keescook@chromium.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240305100933.it.923-kees@kernel.org> References: <20240305100933.it.923-kees@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=3141; i=keescook@chromium.org; h=from:subject; bh=6qDL+BboHaqcTpZSUCl+S9ocvneIfXXeMs3ajiQ2O/Q=; b=owEBbQKS/ZANAwAKAYly9N/cbcAmAcsmYgBl5u+Ruvag2LjJsl5BflMWP3F10S0bpaicfQjrP em4Flv551eJAjMEAAEKAB0WIQSlw/aPIp3WD3I+bhOJcvTf3G3AJgUCZebvkQAKCRCJcvTf3G3A Jh5rD/wJ5mDNRMwSTsAR06j+XsdZ1LXMSaSe8WeQSfww+7fSDo2bwPbB2YuP2imZfi2Z0ALaDNc QjrOdKcPLNAptwDgTvG8rgq8Ads/uaBkD6rP1GpQd9XF7ZP+IN5Tj/Ust9Zx+YZOh1j757Tk9f4 /2I7O+56tM7rqcJYTzDVwwN3lx5z9LnJAIv9cA2fc5WZw4zdqRbzRZ2x2/jTvOictVZIOz75vdw eHxukjUd1puppQSWDfMARwCUPIILs8IjoAaR8UZAfPVGp0pxZe2GBSsLG2wB+yvlBFR58gg+qkr PPA4dy4re0Tig2xdNDMKxFzPI2MGhDuckv7+bdXYa1y03Qk/hE9om+kdYryhKz3AJ7c2FU6lReH mTQ/EPfpM/DP3hVk1yOvB4YQVT+SsRzeBQyHjE7vMel9E0zPFPRlsHvTQYQfzF3xQMEWSjWW4GG 5VQ0LGp6u8ey7adQvju72Jq5NXyiMZbaTebl1RgLT0AX7iek73VAJKe9Vp1MeHbuW3KzheQjEhv V+GYAiXu51x1VsKLuN69+yg2ofManoTrfcNE9k/KiZywKTTRvzJ7GlsihkF1zScLCrOfnIJLfWn mSg61FpR2fsWqar318TaCYm5iTXskabguFfuho/W7iHJo+XHIJRToiRH7s34zAmcK3ETFBpzTUP t3ctwbz/OBT4vHA== X-Developer-Key: i=keescook@chromium.org; a=openpgp; fpr=A5C3F68F229DD60F723E6E138972F4DFDC6DC026 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Both memdup_user() and vmemdup_user() handle allocations that are regularly used for exploiting use-after-free type confusion flaws in the kernel (e.g. prctl() PR_SET_VMA_ANON_NAME[1] and setxattr[2][3][4] respectively). Since both are designed for contents coming from userspace, it allows for userspace-controlled allocation sizes. Use a dedicated set of kmalloc buckets so these allocations do not share caches with the global kmalloc buckets. After a fresh boot under Ubuntu 23.10, we can see the caches are already in active use: # grep ^memdup /proc/slabinfo memdup_user-8k 4 4 8192 4 8 : ... memdup_user-4k 8 8 4096 8 8 : ... memdup_user-2k 16 16 2048 16 8 : ... memdup_user-1k 0 0 1024 16 4 : ... memdup_user-512 0 0 512 16 2 : ... memdup_user-256 0 0 256 16 1 : ... memdup_user-128 0 0 128 32 1 : ... memdup_user-64 256 256 64 64 1 : ... memdup_user-32 512 512 32 128 1 : ... memdup_user-16 1024 1024 16 256 1 : ... memdup_user-8 2048 2048 8 512 1 : ... memdup_user-192 0 0 192 21 1 : ... memdup_user-96 168 168 96 42 1 : ... Link: https://starlabs.sg/blog/2023/07-prctl-anon_vma_name-an-amusing-heap-= spray/ [1] Link: https://duasynt.com/blog/linux-kernel-heap-spray [2] Link: https://etenal.me/archives/1336 [3] Link: https://github.com/a13xp0p0v/kernel-hack-drill/blob/master/drill_expl= oit_uaf.c [4] Signed-off-by: Kees Cook --- Cc: Andrew Morton Cc: "GONG, Ruiqi" Cc: Xiu Jianfeng Cc: Suren Baghdasaryan Cc: Kent Overstreet Cc: Jann Horn Cc: Matteo Rizzo Cc: linux-mm@kvack.org --- mm/util.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/util.c b/mm/util.c index 02c895b87a28..25b9122022a7 100644 --- a/mm/util.c +++ b/mm/util.c @@ -181,6 +181,16 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup_nul); =20 +static kmem_buckets *user_buckets __ro_after_init; + +static int __init init_user_buckets(void) +{ + user_buckets =3D kmem_buckets_create("memdup_user", 0, 0, 0, INT_MAX, NUL= L); + + return 0; +} +subsys_initcall(init_user_buckets); + /** * memdup_user - duplicate memory region from user space * @@ -194,7 +204,7 @@ void *memdup_user(const void __user *src, size_t len) { void *p; =20 - p =3D kmalloc_track_caller(len, GFP_USER | __GFP_NOWARN); + p =3D kmem_buckets_alloc_track_caller(user_buckets, len, GFP_USER | __GFP= _NOWARN); if (!p) return ERR_PTR(-ENOMEM); =20 @@ -220,7 +230,7 @@ void *vmemdup_user(const void __user *src, size_t len) { void *p; =20 - p =3D kvmalloc(len, GFP_USER); + p =3D kmem_buckets_valloc(user_buckets, len, GFP_USER); if (!p) return ERR_PTR(-ENOMEM); =20 --=20 2.34.1