From nobody Tue Dec 2 01:30:03 2025 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54EE133B6CF for ; Fri, 21 Nov 2025 09:01:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763715664; cv=none; b=NNg2R6uyTQWFq42Xq3p3w5pFLl4ZEgZ12H3wjEokQscBedF7IIzD6ysdXgWN/cAb3wSAyhSfzT0AKHZ5WNnVcy7Flo3cdT9ug+aRCWlZRNlDl6EeCmyHkJJNokUwtgLOrHN/XdHwsnKnb3eKAINZbzZQ4ezd/ojDJ9vgHAkkPpA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763715664; c=relaxed/simple; bh=ZHoIOZsqYgxNlG+z6MWLJCh/3K8n5QbzOVNeHfh2rkY=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=ffmbN/eiI+WVycDgC3yEH/N6KNNENOMjacvANcq8TzBHcR9x/ysot2OA7um+g1Nh4pvZIVFbzbOwm/9KaD9rgTUaT1G4iPrTMjk5qaPKFMobZGoKkQXubatnn+MTFp2Vp9fibVLpQYXjes6peOvqMAJwVCjc3Q+tvlnsYlPWxG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ZxstJKO0; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZxstJKO0" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-299d40b0845so28577185ad.3 for ; Fri, 21 Nov 2025 01:01:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763715660; x=1764320460; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=7CEYg94xBFZCkQHBB1DCwySVM1sXv1HB/XC9eerLVQY=; b=ZxstJKO0XXpVtCySvxqNW0qpAZf9UZ/7bn1YxmmAuDX+FGiZrmylTLW2OA1uaq/XaT StGmkTU615gtCatcj2s3ZEmZVsJhCHRGD9ogEO0weoFgrcoNzSoPqQAFaGECUs8LBmCm I8DiEYaWN1fNnKYYABG5VryUwwDVlS1gLDemdxrq2AMwSvGVt5KLat6qtKYRUhJf+5l/ REKMKywS+k9XYEMfbgk7Oyrs5HEdb3xfEr29Kj1pIQ3SfA+VWW/8ZC57wDoqq0c6Mz3l mYgVqOQ3oRiQp+Yo3k3oFqx4jIe5x2HIrvS41td5yqTZ5EO/ZA3pablPt+2XuOQcJgNi vLzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763715660; x=1764320460; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=7CEYg94xBFZCkQHBB1DCwySVM1sXv1HB/XC9eerLVQY=; b=g0o2SO9/g610GX8p6eVo9lJZKenjQIz8vWd1Vj5qVOjSL+92Y0cXWGhRlAgQ3EsoOI Aunx6UD2SBnmeYgo7B0a4epfdihn/xApRWKvHapIqYEHFnXkMM6RL41UdeGjjtN0VOSL /HjjW1S8uYONLgibhN7rEY7hNtkhspuHs18dl7Lf5JB8j/niQ3NjW/sgQ+iZdTvIWz56 Mdu98YEip1jluLWc0MRp1of3+wFoQRlb8lEHEpExWJatGsOpPZaaaox8WYp1rapXwF9z 3vUBKSBMtuvchPNqnA7NbPLbil46TGGiEfFnWWYmcSFA1AjVwYGIL5phygKoeAbx+Zfx 9L2g== X-Forwarded-Encrypted: i=1; AJvYcCVFRJBqWwX4wkkqLbtMA87Jeinkuei6xqDHMVI41VrTcfIwSrU5q/+RqJ54NUnep/gZ2Ze2xTwfggwnVTQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yzk8P0vhoE1atK9INUNJFempgOLuRpkkRaofMtKVd5koLI7/y/c nTLweTsbTH6Aas6le3TVDRGWUWH1e7ZyC+HUd6pD+VxWr+MOLbkre3I/ X-Gm-Gg: ASbGncu07EpaPkrvw/pCkm3AnX/yWRqQVnMwgEiyn2wqTMEZKTi0F6iXvJRp7nk75Lu FLRvmeLYUKMCHXw/4BsqX4iq+tSUdpoH/h5ygzjgd1Pwlsai3SHZStKD8dOBnq0qFcbKBb/70pD 2X5k1aGatRpgnF+3EQEny7PaEs27vL29zIIcTTTE+0UuPZBqe/Xo1FkB0n8JOOm7gKrj07zKnwr uz58i+ucmNljSQ/Xz5d0vsYbyojk+Bh0UaVTkFNtie9an7214G/ffN7sLUe+lIguTRbay3xBs4m Qvz4erbKMiXmKJQRmCPZEs0ndGwzTQOBBNxUyET/eEQoio4SpSYpKO8RRlQ04snCzZ1hMOAL5+l x6CKCXkiR3YTbxbdWxXkbv1Kh+rqaNKakoi5KbOudc8KqBEzOvjKj+clGdQf6Odvi7NAfStY3Jq MY2M/Lnhae5pMXzEa+fRg3o/MT+EuP X-Google-Smtp-Source: AGHT+IE87ieZqlbUkR0VokkAR9tSsD3SSdbeKfahM6e7QiM51lcKjUC/xOt7U7NmTXamvMsyNuF7bw== X-Received: by 2002:a17:903:2985:b0:298:1830:6ada with SMTP id d9443c01a7336-29b6bf19c83mr24848795ad.30.1763715660323; Fri, 21 Nov 2025 01:01:00 -0800 (PST) Received: from n232-175-066.byted.org ([36.110.131.230]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-29b5b25e638sm50556065ad.58.2025.11.21.01.00.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Nov 2025 01:01:00 -0800 (PST) From: guzebing To: brauner@kernel.org, djwong@kernel.org Cc: linux-xfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, guzebing , Fengnan Chang Subject: [PATCH] iomap: add allocation cache for iomap_dio Date: Fri, 21 Nov 2025 17:00:52 +0800 Message-Id: <20251121090052.384823-1-guzebing1612@gmail.com> X-Mailer: git-send-email 2.20.1 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: guzebing As implemented by the bio structure, we do the same thing on the iomap-dio structure. Add a per-cpu cache for iomap_dio allocations, enabling us to quickly recycle them instead of going through the slab allocator. By making such changes, we can reduce memory allocation on the direct IO path, so that direct IO will not block due to insufficient system memory. In addition, for direct IO, the read performance of io_uring is improved by about 2.6%. Suggested-by: Fengnan Chang Signed-off-by: guzebing --- fs/iomap/direct-io.c | 92 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 87 insertions(+), 5 deletions(-) diff --git a/fs/iomap/direct-io.c b/fs/iomap/direct-io.c index 5d5d63efbd57..7a5c610ded7b 100644 --- a/fs/iomap/direct-io.c +++ b/fs/iomap/direct-io.c @@ -54,8 +54,84 @@ struct iomap_dio { struct work_struct work; } aio; }; + struct iomap_dio *dio_next; /* request queue link */ }; =20 +#define DIO_ALLOC_CACHE_THRESHOLD 16 +#define DIO_ALLOC_CACHE_MAX 256 +struct dio_alloc_cache { + struct iomap_dio *free_list; + struct iomap_dio *free_list_irq; + int nr; + int nr_irq; +}; + +static struct dio_alloc_cache __percpu *dio_cache; + +static void dio_alloc_irq_cache_splice(struct dio_alloc_cache *cache) +{ + unsigned long flags; + + /* cache->free_list must be empty */ + if (WARN_ON_ONCE(cache->free_list)) + return; + + local_irq_save(flags); + cache->free_list =3D cache->free_list_irq; + cache->free_list_irq =3D NULL; + cache->nr +=3D cache->nr_irq; + cache->nr_irq =3D 0; + local_irq_restore(flags); +} + +static struct iomap_dio *dio_alloc_percpu_cache(void) +{ + struct dio_alloc_cache *cache; + struct iomap_dio *dio; + + cache =3D per_cpu_ptr(dio_cache, get_cpu()); + if (!cache->free_list) { + if (READ_ONCE(cache->nr_irq) >=3D DIO_ALLOC_CACHE_THRESHOLD) + dio_alloc_irq_cache_splice(cache); + if (!cache->free_list) { + put_cpu(); + return NULL; + } + } + dio =3D cache->free_list; + cache->free_list =3D dio->dio_next; + cache->nr--; + put_cpu(); + return dio; +} + +static void dio_put_percpu_cache(struct iomap_dio *dio) +{ + struct dio_alloc_cache *cache; + + cache =3D per_cpu_ptr(dio_cache, get_cpu()); + if (READ_ONCE(cache->nr_irq) + cache->nr > DIO_ALLOC_CACHE_MAX) + goto out_free; + + if (in_task()) { + dio->dio_next =3D cache->free_list; + cache->free_list =3D dio; + cache->nr++; + } else if (in_hardirq()) { + lockdep_assert_irqs_disabled(); + dio->dio_next =3D cache->free_list_irq; + cache->free_list_irq =3D dio; + cache->nr_irq++; + } else { + goto out_free; + } + put_cpu(); + return; +out_free: + put_cpu(); + kfree(dio); +} + static struct bio *iomap_dio_alloc_bio(const struct iomap_iter *iter, struct iomap_dio *dio, unsigned short nr_vecs, blk_opf_t opf) { @@ -135,7 +211,7 @@ ssize_t iomap_dio_complete(struct iomap_dio *dio) ret +=3D dio->done_before; } trace_iomap_dio_complete(iocb, dio->error, ret); - kfree(dio); + dio_put_percpu_cache(dio); return ret; } EXPORT_SYMBOL_GPL(iomap_dio_complete); @@ -620,9 +696,12 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *it= er, if (!iomi.len) return NULL; =20 - dio =3D kmalloc(sizeof(*dio), GFP_KERNEL); - if (!dio) - return ERR_PTR(-ENOMEM); + dio =3D dio_alloc_percpu_cache(); + if (!dio) { + dio =3D kmalloc(sizeof(*dio), GFP_KERNEL); + if (!dio) + return ERR_PTR(-ENOMEM); + } =20 dio->iocb =3D iocb; atomic_set(&dio->ref, 1); @@ -804,7 +883,7 @@ __iomap_dio_rw(struct kiocb *iocb, struct iov_iter *ite= r, return dio; =20 out_free_dio: - kfree(dio); + dio_put_percpu_cache(dio); if (ret) return ERR_PTR(ret); return NULL; @@ -833,6 +912,9 @@ static int __init iomap_dio_init(void) =20 if (!zero_page) return -ENOMEM; + dio_cache =3D alloc_percpu(struct dio_alloc_cache); + if (!dio_cache) + return -ENOMEM; =20 return 0; } --=20 2.20.1