From nobody Fri Sep 25 10:37:43 2026 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B39A410D3D for ; Mon, 14 Sep 2026 09:21:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377698; cv=none; b=h6Hh5f22h61Ydm4ZnCSxMKSRQ3OqoVv/SsWL/rQlkosFyeiFnBJT5xDV0dZqvcSuCNdmlV8L9tLkJDCFGONk+VvyjykGmn2uTY2P2mf13nuQ+/Y3txH+QFYa/T2Z+XYjGwKTrA/CVGWzYDhtNmog74bBPZ0QGGz3ZFAWCJqrz2Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377698; c=relaxed/simple; bh=MlleD8QjMSMowLp06JepZC8Rof4LpffQJ09I3Y+7j8w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=sjNRu6ny+QsxN22j6h4froHi7Br7nb8rpqyXO9eX9QaFlvCqqZ6eF55hcgWCsHRa92RmTIaG+QBiHM7muUVKUv1AQVtUQRlXoJ24iC0cXetIiD3/W7CPySUVFhQiUAcaP7UzShXhOZiV59OjldWCuWsg8YPyQxvA5r1cqyDayjQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=njhayYyS; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="njhayYyS" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-90e8e70fa02so39997246d6.3 for ; Mon, 14 Sep 2026 02:21:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789377695; x=1789982495; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ZLY2GYRaOljR7tphE79zbHfzaT1w5oSZifPlkEd/L4U=; b=njhayYyS49eCQCQ1F61nPWPrwrbnxaEz0p+5klyDuM5p1xF1L21EZ6rIe9V8Tux6vS 1q5eXhUhATGenpEFtrK3JsVZ1fcrTB2KUWEF7Ah2V3SADkm2AH3/aYlYTaUyQ47iO5kj klF8HxGm2eiXkTf9QRXnFFBXg0zyuqCSUPVd5CKomAiwOal/0xuEJv+8hcoVE5q04pG5 w0pmTigEQXEB8+dvS2nfk7Ne/OOQbOnOceX/WCo4NoHvZCzN7lnZdTqiPVmX7L4fZSud wOHZd4YYpF4Di3C5iNOOuallM/+L+ZRjDbFwGVrCB5oGA1PCyJzl7V4xnOU52HQPWm8Y AURA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789377695; x=1789982495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ZLY2GYRaOljR7tphE79zbHfzaT1w5oSZifPlkEd/L4U=; b=sMT59a8npbWuW5v3Q8KVVXsKewi0H+2XrEeDuM1TtvH0FHX797VTs7nE9gD3JyIApT GRQK2XAVkQZDfj8V/5ZBRWn4ZmI2wpucJx7BHP38UPQ3Uqg7E7W/s3W+Y4mL8MRxVO2V aScX3Pt6HwmtBdlBO7Wig/E2ib8z7yATMx/FxMJBUpksmyA3VG0oFGDsV8LwZn5htYWa TWbO0eBbsTqg3h4e5k7WWIhsPmz1Xk9o1cTImwNwYNysadhRwcwUwxhz25vlE/oYO1sY 0F+naxAqF+SN0mtqp/wb7ZFCtGyDASAyuCLrGi7PaMf5cepEGrOx2rQekc2VVQyRtH0h zUBQ== X-Forwarded-Encrypted: i=1; AKwUvBzJWl+T+765/pIMio7/AAght9mENOiddSJV/4U/JyO8g3g10GoVedWoWkiJpT/lNZGJ8feKveiH+4VTSbc=@vger.kernel.org X-Gm-Message-State: AFuF++nUY0+dAKO7CzmA7N5sr5ysgy01zpwykXYK+IdfsedUutmTnnye /5GwO6kfR8boRNK9zal0XSTS5O9yG1RWLn2bqyhy66wTscMNWgsb5dNa X-Gm-Gg: AYBFou21rmIE/TmdtovwyQ+YGuBBqxb6VUZkBaiHaw6r4Ne+cSDYzy0JdTAp/UvAFJC t08vdLiivx1IkGgAOMBGvZGjDDqJdoAvHoed6rV1b1glVXUB5o6JOC2hwvd+HJFY/shVFggbgcm GETHPKMNCU+hhedBPaxHCYu/TfVK7wY32WLD/rWXD8R8N8imqk5JQcUk3Lu2I7UIdgD7wfqlXRg kfUfT3QWC+fHbmTh4ahrYnyeP8p06O8UV3tGCsHofCqpwDWGWJRPgvm7SacQcPf5mwcCat/djxk ohJV9TiFAeEQcetx08RuIIREYmJg9cGp+t2F5ne/k3NREl108rxowR109HmHuaJpibGuxdGh+NY fy9QYsRn2auMRwu8V3CyqId0R48akmCwzOylJEamt79uZBYOfUKcbeL47/qNGbYWPp08EJt+Zxy EKFewQDx/S2FCHrBrykxkGdeeLCBbLGE2+nG4btUUbvXDjdXhhF+OrT2hzqmen8ckg3J/mY4tXe ylDImQ94XHEwiztNBRE9W7cKIdx9Pc9WoHSo5yQZTj31M0SFbCjK0ppIvyqtq3F7qEM X-Received: by 2002:a05:6214:8613:b0:910:6f25:8f49 with SMTP id 6a1803df08f44-9122e66152emr25233396d6.37.1789377695479; Mon, 14 Sep 2026 02:21:35 -0700 (PDT) Received: from kernel-dev.. ([2a01:4ff:f0:3ff2::1]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f49444bsm89854256d6.29.2026.09.14.02.21.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 02:21:35 -0700 (PDT) From: Uzair Beg To: io-uring@vger.kernel.org Cc: axboe@kernel.dk, asml.silence@gmail.com, Chengfeng Lin , linux-kernel@vger.kernel.org, Uzair Beg Subject: [RFC PATCH 1/3] io_uring/rsrc: allocate io_rsrc_node from a dedicated kmem_cache Date: Mon, 14 Sep 2026 09:20:47 +0000 Message-ID: <20260914092049.130079-2-uzairbeg11@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260914092049.130079-1-uzairbeg11@gmail.com> References: <20260914092049.130079-1-uzairbeg11@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" io_rsrc_node allocations come from the generic kmalloc-32 bucket via io_cache_alloc_new(). On a first fill of a sparse fixed file table the per-ring node cache is empty by construction, since io_reset_rsrc_node() returns early on a NULL slot and nothing is freed back, so every install takes an allocator round trip. Add an optional kmem_cache to io_alloc_cache and use it for the node cache. The slab pointer defaults to NULL, so other io_alloc_cache users are unchanged and imu_cache keeps using kmalloc, its element size being variable. All three free paths honour the slab. On its own this is neutral on the reported workload. It exists so that the following patches can use kmem_cache_alloc_bulk(), which has no equivalent for plain kmalloc. Reported-by: Chengfeng Lin Closes: https://lore.kernel.org/io-uring/CANGjgdmt0FQ=3Doffsdfn+wEaDxbOFoAa= 6bi92X_vEo4S6aCZ56A@mail.gmail.com/ Tested-by: Chengfeng Lin Signed-off-by: Uzair Beg --- include/linux/io_uring_types.h | 1 + io_uring/alloc_cache.c | 14 +++++++++++--- io_uring/alloc_cache.h | 8 ++++++-- io_uring/io_uring.c | 5 +++++ io_uring/io_uring.h | 1 + io_uring/rsrc.c | 2 ++ 6 files changed, 26 insertions(+), 5 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index c2ea6280901..e8d5a585a60 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -253,6 +253,7 @@ struct io_alloc_cache { unsigned int max_cached; unsigned int elem_size; unsigned int init_clear; + struct kmem_cache *slab; }; =20 struct io_ring_ctx { diff --git a/io_uring/alloc_cache.c b/io_uring/alloc_cache.c index 58423888b73..a44b82a80f1 100644 --- a/io_uring/alloc_cache.c +++ b/io_uring/alloc_cache.c @@ -10,8 +10,12 @@ void io_alloc_cache_free(struct io_alloc_cache *cache, if (!cache->entries) return; =20 - while ((entry =3D io_alloc_cache_get(cache)) !=3D NULL) - free(entry); + while ((entry =3D io_alloc_cache_get(cache)) !=3D NULL) { + if (cache->slab) + kmem_cache_free(cache->slab, entry); + else + free(entry); + } =20 kvfree(cache->entries); cache->entries =3D NULL; @@ -30,6 +34,7 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, cache->max_cached =3D max_nr; cache->elem_size =3D size; cache->init_clear =3D init_bytes; + cache->slab =3D NULL; return false; } =20 @@ -37,7 +42,10 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, g= fp_t gfp) { void *obj; =20 - obj =3D kmalloc(cache->elem_size, gfp); + if (cache->slab) + obj =3D kmem_cache_alloc(cache->slab, gfp); + else + obj =3D kmalloc(cache->elem_size, gfp); if (obj && cache->init_clear) memset(obj, 0, cache->init_clear); return obj; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index d33ce159ef3..b288bfccc91 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -61,8 +61,12 @@ static inline void *io_cache_alloc(struct io_alloc_cache= *cache, gfp_t gfp) =20 static inline void io_cache_free(struct io_alloc_cache *cache, void *obj) { - if (!io_alloc_cache_put(cache, obj)) - kfree(obj); + if (!io_alloc_cache_put(cache, obj)) { + if (cache->slab) + kmem_cache_free(cache->slab, obj); + else + kfree(obj); + } } =20 #endif diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 296667ba712..f375f0ccc0e 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -151,6 +151,7 @@ static void __io_req_caches_free(struct io_ring_ctx *ct= x); static __read_mostly DEFINE_STATIC_KEY_FALSE(io_key_has_sqarray); =20 struct kmem_cache *req_cachep; +struct kmem_cache *io_rsrc_node_cachep; static struct workqueue_struct *iou_wq __ro_after_init; =20 static int __read_mostly sysctl_io_uring_disabled; @@ -4074,6 +4075,10 @@ static int __init io_uring_init(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU); =20 + io_rsrc_node_cachep =3D kmem_cache_create("io_rsrc_node", + sizeof(struct io_rsrc_node), NULL, + SLAB_HWCACHE_ALIGN | SLAB_PANIC); + iou_wq =3D alloc_workqueue("iou_exit", WQ_UNBOUND, 64); BUG_ON(!iou_wq); =20 diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46d9141d772..6243dadd507 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -527,6 +527,7 @@ static inline bool io_req_cache_empty(struct io_ring_ct= x *ctx) } =20 extern struct kmem_cache *req_cachep; +extern struct kmem_cache *io_rsrc_node_cachep; =20 static inline struct io_kiocb *io_extract_req(struct io_ring_ctx *ctx) { diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index d787c16dc1c..6413682ebe4 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -175,6 +175,8 @@ bool io_rsrc_cache_init(struct io_ring_ctx *ctx) node_size, 0); ret |=3D io_alloc_cache_init(&ctx->imu_cache, IO_ALLOC_CACHE_MAX, imu_cache_size, 0); + if (!ret) + ctx->node_cache.slab =3D io_rsrc_node_cachep; return ret; } =20 --=20 2.43.0 From nobody Fri Sep 25 10:37:43 2026 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8AB0A42317C for ; Mon, 14 Sep 2026 09:21:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377703; cv=none; b=nnblVSL991ZY87wayxUlSUMRsROWM1WRBC7cWn7yhXmKxFE8ulHh2x4J1AkLEWJSDyVbPIbA/7WkiJuI/gEVZ4WByVukIKmCcaQ+M8Du6faWLddhVd9f0tb+klT03Gp50am/lbO7HiZdPMn85sVOeiTn/sHtIxqGGBa1g96eoTM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377703; c=relaxed/simple; bh=AWgNdelLuusMKTO3XmEV6H2qIFR8ZhjKw8eGqirrJj8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FHBNxxTpWm+YO8+OnSiuj+qbOKDWLfudtdJdrSKiP3kWCLLZrIpIr+B7N7phiAayNW5/DhpmGwIUHH++GD43vl9o1jYPCsPfruz8oXJteC7Ue7cOtQJcTIv/k4AMkz3O8ONusMtFgAfzL9ubSHaWF5kUrCJ+zLSI2sg+QuGjbIs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BCKXjLqg; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BCKXjLqg" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-530ea5dafd4so22267801cf.1 for ; Mon, 14 Sep 2026 02:21:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789377700; x=1789982500; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=DJLNgmlgQhAoJRQ2IrTohM5uVJQIzCvFDhXSRTt2DE4=; b=BCKXjLqgahPBnZIF2oa8n4IX/jW/XfHQtuXpWlaNdbMa1o2Kr2kIitX2FNE6YLaxxn IszLyAzzeP40Z/04KUGPn0PDttkyxKcWgaSeHAVO9zLVrjLfrG7/EnIvTHgaCAOBxN1a ey1G82Kz+9LeK5amNquTWc65uNNNU6EdqebrdPpx0uO2wZSow6zPsbtBqXJjy3RpBj+d 8S0ZRZLQCPbkNrHmC3zmSns2jygCFE9nYmP1tZU0ONmRfnCQrmzXzp56/6AOCaYMGD6c c3xfwRGQadddqtIlWNm7FegBB7ORdKqRNVBNXNO4QvjXakvfqzzklMpUZwdHhPQWn+gO tZFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789377700; x=1789982500; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=DJLNgmlgQhAoJRQ2IrTohM5uVJQIzCvFDhXSRTt2DE4=; b=kvGeKXtitcnlT1FolxYgLIF/1wza2UeuuU4nTtemYwtPnD5dXU1XCOlm70l1mRcugx bmjS//cIuZ0wzelAOEi5p94R5a/iE01NqplqYo36EU/3kfee68UDN4hj4pb7BYEIdJbc 4S8RQ46nHfHxsjv+Z9U+k93uAfwG+CK19v997wi+nDCcA2D1C3c4GKnfLpdzSTC0FuL7 APGg6epf7to/EvwQHpY4tJdCdak4TfBnfya/L59w8qjwbeXqQJhSRqReobwl8nsQgnOr +QcWkf47lSFEpI8MzCfB8gj5eA6bNvceWtbv4PI6vEiUfCOzZLkV7VZMSHKxHbHJ85Jh iErg== X-Forwarded-Encrypted: i=1; AKwUvBzAl+VCpQtltAexZHM1VddSvhCv9Maidxz8qtH3VC2dItVe+/MVOmr+NcXu0S5OHEM/Ckvfdc84FdHowrQ=@vger.kernel.org X-Gm-Message-State: AFuF++nQmeFjHDAkNX/096SmcdRUDacoKSYcioKXQjyDDpCT33VGd5N6 EWGr7wsYLPvOcH6HcScCeueFQ7dJwcOB9p72Yw1nZL7mRJuw2LY/39dA X-Gm-Gg: AYBFou37GeTkElvB4YNdgSe5vbTvMOROfVbXbxITeEySJxeIHtkuKHCW9NWA7fpymR6 71R1XNcgGQGOYHfGZy1mBl0PPGLucAIcF006t8D2jQtmlRyTWqrHLpzSq+ZqVXXXjZjTarguryU 7BHWy3FdA7MDJYFonzVJRnA7AyhIif8H8Sj9dF0cNUTPKYlsliEpMK98EDbt5OWps/9pWEBcEk6 sY5CgUB4gpnsjh6arpgm+EOEGhzukoJonWVSsW9TFCID5DD0n8abAY/RxQZon+eoC2YjBvrJk+h xP4o205jBdivbmNcYXt88ckwPbTmY13F2Fahtdc3RBCXrpF3OhcIBD/4+i3/Q5UaL4IphN9jffj k3UOVriMrkh6zDvgHH0ADeQQiyNIl7azIcP99tLCz/r4AoZgLTdIawRYUZhgQjn45NsUYe1/9ag mfHkg/Q+7cFTaFd7UEUptlqz0W4O+KQPBkgUMO//7TipmlpDLQenT+GhVJt107hOadYo9BybLbd IPV4lwPCf4Bm1XEjid3yqys1DVbEYrTmrh/bevV/FlyRFBug0g2eYDgrD1liWDOF0wR X-Received: by 2002:ac8:59c1:0:b0:530:db0e:e098 with SMTP id d75a77b69052e-5310cf602b0mr23311491cf.23.1789377700398; Mon, 14 Sep 2026 02:21:40 -0700 (PDT) Received: from kernel-dev.. ([2a01:4ff:f0:3ff2::1]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f49444bsm89854256d6.29.2026.09.14.02.21.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 02:21:40 -0700 (PDT) From: Uzair Beg To: io-uring@vger.kernel.org Cc: axboe@kernel.dk, asml.silence@gmail.com, Chengfeng Lin , linux-kernel@vger.kernel.org, Uzair Beg Subject: [RFC PATCH 2/3] io_uring/rsrc: bulk refill the node cache on allocation miss Date: Mon, 14 Sep 2026 09:20:48 +0000 Message-ID: <20260914092049.130079-3-uzairbeg11@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260914092049.130079-1-uzairbeg11@gmail.com> References: <20260914092049.130079-1-uzairbeg11@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A first fill of a sparse fixed file table takes one allocator round trip per install, since the per-ring node cache starts empty and nothing is freed back during the fill. At 4,096 slots that is 4,096 calls into the slab allocator. When the cache has a dedicated kmem_cache, refill it in batches on a miss: allocate up to IO_ALLOC_CACHE_REFILL objects with kmem_cache_alloc_bulk(), return one and stash the remainder in the cache. A 4,096-slot first fill then enters the allocator roughly once per batch instead of once per object. This mirrors the existing bulk allocation of requests from req_cachep. kmem_cache_alloc_bulk() may return fewer objects than requested, including zero; both cases are handled. Stashed objects have their init_clear region zeroed and are poisoned like any other cached entry, and the cache never grows past max_cached. Callers without a dedicated slab are unchanged. Bare-metal measurement shows this is neutral on the reported workload: the per-object cost is in the SLUB allocation path itself, not in the number of allocator entries. It is kept because the following patch relies on the same bulk machinery. Reported-by: Chengfeng Lin Closes: https://lore.kernel.org/io-uring/CANGjgdmt0FQ=3Doffsdfn+wEaDxbOFoAa= 6bi92X_vEo4S6aCZ56A@mail.gmail.com/ Tested-by: Chengfeng Lin Signed-off-by: Uzair Beg --- io_uring/alloc_cache.c | 29 ++++++++++++++++++++++++++--- io_uring/alloc_cache.h | 1 + 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/io_uring/alloc_cache.c b/io_uring/alloc_cache.c index a44b82a80f1..cba0e6c5d66 100644 --- a/io_uring/alloc_cache.c +++ b/io_uring/alloc_cache.c @@ -42,10 +42,33 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, = gfp_t gfp) { void *obj; =20 - if (cache->slab) - obj =3D kmem_cache_alloc(cache->slab, gfp); - else + if (cache->slab) { + unsigned int room =3D cache->max_cached - cache->nr_cached; + void **slot =3D &cache->entries[cache->nr_cached]; + unsigned int batch, got, i; + + if (unlikely(!room)) + return kmem_cache_alloc(cache->slab, gfp); + + batch =3D min_t(unsigned int, IO_ALLOC_CACHE_REFILL, room); + got =3D kmem_cache_alloc_bulk(cache->slab, gfp, batch, slot); + if (unlikely(!got)) + return NULL; + + /* return one object, stash the rest in the cache */ + obj =3D slot[got - 1]; + for (i =3D 0; i < got - 1; i++) { + if (cache->init_clear) + memset(slot[i], 0, cache->init_clear); + if (unlikely(!kasan_mempool_poison_object(slot[i]))) + break; + cache->nr_cached++; + } + for (; i < got - 1; i++) + kmem_cache_free(cache->slab, slot[i]); + } else { obj =3D kmalloc(cache->elem_size, gfp); + } if (obj && cache->init_clear) memset(obj, 0, cache->init_clear); return obj; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index b288bfccc91..82d552c7517 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -7,6 +7,7 @@ * Don't allow the cache to grow beyond this size. */ #define IO_ALLOC_CACHE_MAX 128 +#define IO_ALLOC_CACHE_REFILL 32 =20 void io_alloc_cache_free(struct io_alloc_cache *cache, void (*free)(const void *)); --=20 2.43.0 From nobody Fri Sep 25 10:37:43 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B6104252B1 for ; Mon, 14 Sep 2026 09:21:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377706; cv=none; b=Y6JQ2jpTplzrVJtE/XXUDXhVtYmhTlnrL9ClHmf4V6gEW5+aL4qRHmx7MpmuD0+3Q3h8aIgcWxoMjJgBGtB6zNSzbW4s+jYrUyM/150A2EfU4h2Gcp+kSl48+hI6LIlIO8Upg6Dd+YH3Z1PWJJfAJN12QMJECKnzacX2W/rJ0HE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789377706; c=relaxed/simple; bh=QmGWJsYTjqPI0bHKbLqcwPxNwliWQUZfC/2eQCFNoE8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R2HONfpzLx5zLhkZ0TozaklOIeRtHXyscvbP/vdKc4p5mwwjFtc0hiAHW8L+ng6Aval2GA8+KT2JXKlbRv5JYxErLU8oYhmRUTub5r3qkSqWi/QYzKI2VtYgU+dM6AmnUzXxIm/fcEPWbRt5/fk07rtrDm7echh7nwi86sygywo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=QY3g1G96; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QY3g1G96" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-530e28a62abso30385451cf.1 for ; Mon, 14 Sep 2026 02:21:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789377704; x=1789982504; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=bzaXd3BjbK6IZPh5AWHB8yMlFpjXv/VnxLmPZtypjps=; b=QY3g1G96GGri1ar1iPDUnBfbr5H8xmdU+0FwvSbYk/HCRxQhcV108Wx+cqvXfZxNTI t8M9W1VP0Azzmre9kO/pgb/crmvZPwlyDIUztxZyBlBvUNZ24eQ3s4JFHWUBHQ4PUwnz 1Ele43MjhETiCzr8Tzxf/qI7lYhWN++0Qlm7UdKMk0x95k3ICEc9qHbQMiqFWqUvCpYT cPFZdAvKY35o5lKhg8xBOKeCjYxK3iHGrT0OH30vqS+pGyd98o2KzExrOLs13s4XPs7C iOgbsA23ZoQfGMcyM4J9gZKONjBJ/VtTY9ECLyUT6RNs0l4Y7hZOZt484EFWjU/jYLbh uWOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789377704; x=1789982504; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=bzaXd3BjbK6IZPh5AWHB8yMlFpjXv/VnxLmPZtypjps=; b=nMXSETyeavvuI14WJ/b2Qzshi7i4OO9nGaG1JsUjval/klIoHuuOiOuP/ZHrG+XpWv 754MSq2fxp9tgftyHfDS3Jgydw0ItEAr13ui+TXjN6CCSzxcLsyjtFFaZUDJAuVJppbR 0C+pvUytMdU6/xjPOUpGLSf8Gm9sMyp0gjfZSe7taQVJgrvsUCSYT1r4JdkTTGBSbOTH V0Bx+rPjbKfWzJbIipl2FEJiDHvBjUzXA6SJOXWlyHewqm+nQYEqZiPreLF20Q34eyV/ IEnVu6YriKEBKnRnHqHOkQsFqpoJwhjGMe2QGB+XjiDbX65p5cltp/foXEiapZxuFx8a nzaA== X-Forwarded-Encrypted: i=1; AKwUvBxYyQ1U9DDbLN9nCp6bmbCgOi9U2bwvpZSxL7rUtXTLLXqXVlh6XrpV1K6Nu+puH4yMNOrNW19STeqcPVA=@vger.kernel.org X-Gm-Message-State: AFuF++kiBt96wbq8kD/cPWhc2VKcpYcnmtFu+27f6NRiXNIuVyJl1xB/ EJa4kH+099aNr3+Zn5fRru8LmaEOEN9nnE9ZE9kH58AnRg6Q4MEgGl6H X-Gm-Gg: AYBFou1SHDidCrY+2pCjg5dAmNHcL7GP4M60f7hX9cQNe2FJrKRJRl43NWxXUDU34UC pkOEnYYnotBv/e4XwBRMuFI2Ue2BwoHc5IhaROFOUa9zircgNDr68uLQvazoSd7G0KltoiV5iq0 g6kXGI9a/8g9VP7LyeS7EWeUc0DO2bZVLgjaIt8ZTHVmaeKAd6aZS1YARPVq47eDpRcCAExXkKz meubHC59ZpLEq7RHfDj8OgBhC1L47k0h6OmMhicCKlLhCNUSZxDpkPs39lwnUi+7jQkxrGdgBqd VSGDZFq9jP3ApWMTU1T0Ihfc+grD2e3lpukGSO8cFSGerKO9Io/dXUwjSGnCeELLUvDjNl1skwA wEXUvC7aBnjqsnc53ZAtu9sUwHMfJK1bTTCLauqNRSaexx3EkK27I0/uasMZn5jdzazR8olAmq9 KYEs9fUFfaihcgMKIo9Da46k9KOwAjhOkAJVSKb582FULJKrH6y2taoxWsrUjjljMIol26JjTGc 74MueI1jvK2WLM73g1dHeOLPsOCMxL9b/Dt0cFEO7fNLpb28RGIiXNuMQ== X-Received: by 2002:a05:622a:181e:b0:530:ce8c:b6ee with SMTP id d75a77b69052e-5310d093697mr22811311cf.60.1789377704061; Mon, 14 Sep 2026 02:21:44 -0700 (PDT) Received: from kernel-dev.. ([2a01:4ff:f0:3ff2::1]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-9120f49444bsm89854256d6.29.2026.09.14.02.21.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 02:21:43 -0700 (PDT) From: Uzair Beg To: io-uring@vger.kernel.org Cc: axboe@kernel.dk, asml.silence@gmail.com, Chengfeng Lin , linux-kernel@vger.kernel.org, Uzair Beg Subject: [RFC PATCH 3/3] io_uring/rsrc: prefill the node cache when a file table is registered empty Date: Mon, 14 Sep 2026 09:20:49 +0000 Message-ID: <20260914092049.130079-4-uzairbeg11@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260914092049.130079-1-uzairbeg11@gmail.com> References: <20260914092049.130079-1-uzairbeg11@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Registering a sparse fixed file table allocates no nodes at registration time; each node is allocated later, on the install path, where MSG_RING SEND_FD pays for it. Bare-metal measurement of the 4,096-slot first fill shows the cost is not the allocator call (bulk refill was neutral) nor fresh slab pages (priming the slab was neutral), but the per-object SLUB allocation path itself. The only way to take it off the install path is to not allocate there. When a sparse table of N slots is registered, grow the per-ring node cache to min(N, IO_ALLOC_CACHE_PREFILL_MAX) and bulk-fill it, so the subsequent installs hit the cache. Prefill is best-effort: on any failure the cache is left in a valid state (a successfully grown pointer array is retained) and registration proceeds unchanged. Non-sparse registrations are untouched, since they allocate every node inline anyway. On the reported 4,096-slot first fill this is 9.8% faster than unpatched. Because the enlarged cache also retains nodes released by FILES_UPDATE, a same-ring remove-and-refill of 4,096 files is 18.8% faster. The cost is moved to registration rather than removed: a one-shot register-then-fill is unchanged overall, and a program that registers many slots and installs few pays for nodes it never uses. Whether that trade is acceptable, or should be behind a registration flag, is the question this patch is intended to raise. The stash loop from the bulk refill path is factored into a helper so both callers share it. Reported-by: Chengfeng Lin Closes: https://lore.kernel.org/io-uring/CANGjgdmt0FQ=3Doffsdfn+wEaDxbOFoAa= 6bi92X_vEo4S6aCZ56A@mail.gmail.com/ Tested-by: Chengfeng Lin Co-developed-by: Chengfeng Lin Signed-off-by: Chengfeng Lin Signed-off-by: Uzair Beg --- io_uring/alloc_cache.c | 58 ++++++++++++++++++++++++++++++++++-------- io_uring/alloc_cache.h | 2 ++ io_uring/rsrc.c | 4 +++ 3 files changed, 54 insertions(+), 10 deletions(-) diff --git a/io_uring/alloc_cache.c b/io_uring/alloc_cache.c index cba0e6c5d66..2c6e09313d2 100644 --- a/io_uring/alloc_cache.c +++ b/io_uring/alloc_cache.c @@ -38,6 +38,22 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, return false; } =20 +static void io_cache_stash(struct io_alloc_cache *cache, void **slot, + unsigned int nr) +{ + unsigned int i; + + for (i =3D 0; i < nr; i++) { + if (cache->init_clear) + memset(slot[i], 0, cache->init_clear); + if (unlikely(!kasan_mempool_poison_object(slot[i]))) + break; + cache->nr_cached++; + } + for (; i < nr; i++) + kmem_cache_free(cache->slab, slot[i]); +} + void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp) { void *obj; @@ -45,7 +61,7 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, gf= p_t gfp) if (cache->slab) { unsigned int room =3D cache->max_cached - cache->nr_cached; void **slot =3D &cache->entries[cache->nr_cached]; - unsigned int batch, got, i; + unsigned int batch, got; =20 if (unlikely(!room)) return kmem_cache_alloc(cache->slab, gfp); @@ -57,15 +73,7 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, g= fp_t gfp) =20 /* return one object, stash the rest in the cache */ obj =3D slot[got - 1]; - for (i =3D 0; i < got - 1; i++) { - if (cache->init_clear) - memset(slot[i], 0, cache->init_clear); - if (unlikely(!kasan_mempool_poison_object(slot[i]))) - break; - cache->nr_cached++; - } - for (; i < got - 1; i++) - kmem_cache_free(cache->slab, slot[i]); + io_cache_stash(cache, slot, got - 1); } else { obj =3D kmalloc(cache->elem_size, gfp); } @@ -73,3 +81,33 @@ void *io_cache_alloc_new(struct io_alloc_cache *cache, g= fp_t gfp) memset(obj, 0, cache->init_clear); return obj; } + +void io_alloc_cache_prefill(struct io_alloc_cache *cache, unsigned int nr) +{ + gfp_t gfp =3D GFP_KERNEL | __GFP_NOWARN; + unsigned int got; + void **entries; + + if (!cache->slab || !cache->entries) + return; + + nr =3D min_t(unsigned int, nr, IO_ALLOC_CACHE_PREFILL_MAX); + if (nr <=3D cache->nr_cached) + return; + + if (nr > cache->max_cached) { + entries =3D kvmalloc_array(nr, sizeof(void *), gfp); + if (!entries) + return; + memcpy(entries, cache->entries, + cache->nr_cached * sizeof(void *)); + kvfree(cache->entries); + cache->entries =3D entries; + cache->max_cached =3D nr; + } + + got =3D kmem_cache_alloc_bulk(cache->slab, gfp, nr - cache->nr_cached, + &cache->entries[cache->nr_cached]); + if (got) + io_cache_stash(cache, &cache->entries[cache->nr_cached], got); +} diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 82d552c7517..ca6af52dd7d 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -8,6 +8,7 @@ */ #define IO_ALLOC_CACHE_MAX 128 #define IO_ALLOC_CACHE_REFILL 32 +#define IO_ALLOC_CACHE_PREFILL_MAX 4096 =20 void io_alloc_cache_free(struct io_alloc_cache *cache, void (*free)(const void *)); @@ -16,6 +17,7 @@ bool io_alloc_cache_init(struct io_alloc_cache *cache, unsigned int init_bytes); =20 void *io_cache_alloc_new(struct io_alloc_cache *cache, gfp_t gfp); +void io_alloc_cache_prefill(struct io_alloc_cache *cache, unsigned int nr); =20 static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, void *entry) diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 6413682ebe4..b7f78b6b514 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -560,6 +560,10 @@ int io_sqe_files_register(struct io_ring_ctx *ctx, voi= d __user *arg, if (!io_alloc_file_tables(ctx, &ctx->file_table, nr_args)) return -ENOMEM; =20 + /* sparse table: nodes are installed later, so cache them now */ + if (!fds) + io_alloc_cache_prefill(&ctx->node_cache, nr_args); + for (i =3D 0; i < nr_args; i++) { struct io_rsrc_node *node; u64 tag =3D 0; --=20 2.43.0