From nobody Sun Feb 8 02:08:31 2026 Received: from mail-lf1-f97.google.com (mail-lf1-f97.google.com [209.85.167.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 04EBB307AE9 for ; Wed, 22 Oct 2025 23:13:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174824; cv=none; b=Ab4WdTXwG8f5ZCML33kF/ERUj9vk8aFOj4PV4n7fQjsf5zoRlxN19Fvdn0Z+JkY3jNgJc6JpUv2rldb7aHtMXFvZHvf82AlpoHhcQ+u8pt/mvFbShbfUS/bvECnUp/JtpbUaVBpX3Vmm/BXSc5bBP6oQbg14VjSEWx4z6yqLppU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174824; c=relaxed/simple; bh=5e89GXt6DhKyASePnDrD7cthV1505P8u/yKg4DMfB4Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oQwS0bDdGBoLbG6rEphNyzLXZnuyicOleCnnz79DeKbWrOJ2c0vuN8uYOvrAQJbEEHj64+dhqqzWBYHfgC6xgWjOpdFwP+km4r0fOOWgirJsYrXMr0PP45RjUWSG/XkpJTiDSWhtSPJYnf+S+GZ/9F58MpRSLeyOucUSAg17Qpg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=FiKXf9Fa; arc=none smtp.client-ip=209.85.167.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="FiKXf9Fa" Received: by mail-lf1-f97.google.com with SMTP id 2adb3069b0e04-57b35d221e2so37921e87.3 for ; Wed, 22 Oct 2025 16:13:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1761174820; x=1761779620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=96BiLrpc0TjSBBUxSZ5BCVjSp2BMxOcWJzQbIZwxLNw=; b=FiKXf9FaU3BYrYwrRbv4OE29zmFm+UpmIvBYgzLofZvqaAxBVf6rq8d76ApJ/8nrrj dKOQjVnupxM/9ZAhxhDiouvB0IbrDSgQt2UJZXfTW05Nb+c7c6LmXG9YGQyNHUy7/tso Ag3RKol94LTq0KdLEKq7xptTBBKYtcylXgYz4By/KlJOml+8h0PTohxKK4cgmNyPFj0k 7Hsq1gi1Y4viEOslDbTtvtDr1d2vNbdsoehrXc81loS6llfaruWLdukonzPVGFZG/s5V JAfW1C6lBxT7GmLFJjX5mK6NMMeb7Ns+OqJCZ/hR8e7qf20C1U8Jf7sWWugZkAVuS8ZE Whbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761174820; x=1761779620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=96BiLrpc0TjSBBUxSZ5BCVjSp2BMxOcWJzQbIZwxLNw=; b=bvT1+Bq5lLEVAfrPRzRTsPebB8YdNc6SNvL9kOLPBZ9l7x9YFsY3D4GHTXA2Blwclm osR/zORHmzp7ciDfUp9XkuWDJLV6PDm3HQTyL0t7W8ZyNDRvM+yFiNUfd9JZc7loOla3 xUc8ooOLDH2YKw49T4yc+9IE8ULVsK8dMjYNPr0eG08Gd0/99DoK7pgZKgipeAI/evdA eEzlQpyB2ljK6LaRkRxcdXOl8crENvMzp7A/yq/a32XGBIot1ME0rU7dppswKlahc4q8 UnMeGA4G4TIuDiUWYdCwYCMMiljNIW7gSgXrfaySyWpfo8P+Qmw/WQ0NcML99eJlCII2 hpiw== X-Forwarded-Encrypted: i=1; AJvYcCWlmz9nSIZPOxoysypgAeuXI28TYgaIczvMlZvywCJElXlkCuBZ5GVBKZ9a9+9/ESRnEEdAwHZpgaLcD9I=@vger.kernel.org X-Gm-Message-State: AOJu0YyTlUhN4PGGRLOpGl1FdkAulDmztNZD+N/LxxIcIVRoO/qpjoKm gynsxUdZ2DgHf33qCb6Gs+AZ1CVYnPAOGUHFe821v0AR0otpmDFQCly6rqXVbGtScnJAFiEqqnt uWQlDMdAqQhJ/uCZszGCIYTHm4cExDx3ggDOo X-Gm-Gg: ASbGnctUYq3D9hYyiBtIVALHXmt8BqUN8DLBg8Fii+P3LZq7IIW5l4+Tnb5M+ZoTx0J mDNPu0yLfPlAZ1dU4oDcfQJa7g4vBDCJ/yl2ATFOSLwec5cuJtR3ukaaxLWKkB8gtg6JfeZWmtU qLINJXlKwmkawJ1Pwlhu6FsyvLwH/2rmuIfxx8g7MDWi6DaVMeZxI1xNHQFPSDqBthG+NP0mE29 ROimJw4oPVxcjKbmU18HfNPl8XuINBjRyljz/QSli2zm7XkhTx4uFZ4l2v+9tok1Lyo3823fars KW/7FeUeX8DWQc38p3GCLzae1Kjx1uvegQaSp/J2EQyUCbkqLPmUksOIWpnFA0NWYhyeDgf6HOE rtmlkwTYI6UBlyHQAXkzbqfgF8ZU3fsY= X-Google-Smtp-Source: AGHT+IFL+l6RiiF9NVqqr9YCIdVCyARHhrDoK0DxchCYTCey03a/1xzZtxLnzmVhVLR9bTz6N+i4aHRAWF2/ X-Received: by 2002:a05:6512:308d:b0:55f:6637:78f with SMTP id 2adb3069b0e04-591ea31cd1cmr1457181e87.9.1761174819856; Wed, 22 Oct 2025 16:13:39 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.129]) by smtp-relay.gmail.com with ESMTPS id 2adb3069b0e04-592f4e44e32sm33929e87.46.2025.10.22.16.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Oct 2025 16:13:39 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (unknown [IPv6:2620:125:9007:640:ffff::1199]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 372123404B4; Wed, 22 Oct 2025 17:13:38 -0600 (MDT) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id 307C1E4181C; Wed, 22 Oct 2025 17:13:38 -0600 (MDT) From: Caleb Sander Mateos To: Jens Axboe , Miklos Szeredi , Ming Lei , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chris Mason , David Sterba Cc: io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-btrfs@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH 1/3] io_uring: expose io_should_terminate_tw() Date: Wed, 22 Oct 2025 17:13:24 -0600 Message-ID: <20251022231326.2527838-2-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251022231326.2527838-1-csander@purestorage.com> References: <20251022231326.2527838-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A subsequent commit will call io_should_terminate_tw() from an inline function in include/linux/io_uring/cmd.h, so move it from an io_uring internal header to include/linux/io_uring.h. Callers outside io_uring should not call it directly. Signed-off-by: Caleb Sander Mateos --- include/linux/io_uring.h | 14 ++++++++++++++ io_uring/io_uring.h | 13 ------------- 2 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h index 85fe4e6b275c..c2a12287b821 100644 --- a/include/linux/io_uring.h +++ b/include/linux/io_uring.h @@ -1,13 +1,27 @@ /* SPDX-License-Identifier: GPL-2.0-or-later */ #ifndef _LINUX_IO_URING_H #define _LINUX_IO_URING_H =20 +#include #include #include #include =20 +/* + * Terminate the request if either of these conditions are true: + * + * 1) It's being executed by the original task, but that task is marked + * with PF_EXITING as it's exiting. + * 2) PF_KTHREAD is set, in which case the invoker of the task_work is + * our fallback task_work. + */ +static inline bool io_should_terminate_tw(struct io_ring_ctx *ctx) +{ + return (current->flags & (PF_KTHREAD | PF_EXITING)) || percpu_ref_is_dyin= g(&ctx->refs); +} + #if defined(CONFIG_IO_URING) void __io_uring_cancel(bool cancel_all); void __io_uring_free(struct task_struct *tsk); void io_uring_unreg_ringfd(void); const char *io_uring_get_opcode(u8 opcode); diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 46d9141d772a..78777bf1ea4b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -556,23 +556,10 @@ static inline bool io_allowed_run_tw(struct io_ring_c= tx *ctx) { return likely(!(ctx->flags & IORING_SETUP_DEFER_TASKRUN) || ctx->submitter_task =3D=3D current); } =20 -/* - * Terminate the request if either of these conditions are true: - * - * 1) It's being executed by the original task, but that task is marked - * with PF_EXITING as it's exiting. - * 2) PF_KTHREAD is set, in which case the invoker of the task_work is - * our fallback task_work. - */ -static inline bool io_should_terminate_tw(struct io_ring_ctx *ctx) -{ - return (current->flags & (PF_KTHREAD | PF_EXITING)) || percpu_ref_is_dyin= g(&ctx->refs); -} - static inline void io_req_queue_tw_complete(struct io_kiocb *req, s32 res) { io_req_set_res(req, res, 0); req->io_task_work.func =3D io_req_task_complete; io_req_task_work_add(req); --=20 2.45.2 From nobody Sun Feb 8 02:08:31 2026 Received: from mail-lf1-f97.google.com (mail-lf1-f97.google.com [209.85.167.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EA28307AF9 for ; Wed, 22 Oct 2025 23:13:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174825; cv=none; b=VL6juuJpiJsaAyoWqc4f9RQ4qJ1zxI8yW0t5z1jTXC3j8tfd6sXT8IGMNVCTujDF3w5wXmjgWQRn2CzyMq/xevHKfDNDlJ7Rug0XdQ5YFCrHB8me1CqftXJCEzcwuY9t5YwyCaZ0WKTtiTiEa4/6QZIkBskVNZCl+CFa5b6b6gw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174825; c=relaxed/simple; bh=1OFERuWCzicdiztTF/q7QW5F9IVu0Slo/TPxet3gSrc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FsOTuvk4Nyc2kCBLNYvNMWWJeisJBbWFdzdrbuv35HotE2vf47qe92aAoZkedeEkk9qprxmXcedDNRjvyz4BCfq44Ne3jJbsYVhzlNcB4hxOW0xzfmdRiFDnXpvbO/dXOHW+89J9FODeWamydNB95MPFsvMZNVE4NgnIXMk2y7U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=Xam4zb5r; arc=none smtp.client-ip=209.85.167.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="Xam4zb5r" Received: by mail-lf1-f97.google.com with SMTP id 2adb3069b0e04-592f2c6fe9eso21084e87.1 for ; Wed, 22 Oct 2025 16:13:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1761174820; x=1761779620; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QYyTEOzXOC/G6qb7jIG39TqesYubyS/P5gHR3qDN/K4=; b=Xam4zb5r16Y/IKCWvP6QITJYoF7heFJEsed8B8UaZanQH8/Vewr+LukGzqTyZt/+oz 7+V26lZY/FyKcX4U4BcWRvwLwazfXkLMm48UCoJwW6ztXSgfCNKFE+CaIINgpfyuJVK+ xzzxMQYBaAgQXF3rBZ2b6vQYemDhCe39JDVvycZR5FQNriI+FanqPZzQxF07BUEntRBy 5EQ2yj1jif4Qo7ynIVspsunPeJr/FaLSt23CSyRt1CXTaQJrNnWCV28gZUc9WT+9oUfa kJIEOa2OWtw4AlR32nnWWx8T1+SUFiU3M0x685wNVBcosmCIYd71NVC4CNKmK28om9hQ RaFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761174820; x=1761779620; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QYyTEOzXOC/G6qb7jIG39TqesYubyS/P5gHR3qDN/K4=; b=xGF0F1dK0NCRqRuv9qSz275BRh9zWJMJzjdGYWNf/WIQW0YvZa3M2keUcsGhV8yp+a f62DfG2Qt4qCeX/LoawEpVh534r96BTNe3iNZ5YtAIJSM9lNgIzdoRU72M5lG4zfdcfG tLyR5JYcMenyFE1DupX/Xk8ofIn4hQBCb87gkVWH4UET9rXD6K++WkLu9puj1bQx9drB Msi4OIt3aBrIsr4UWFP8RIzWfOqW/vpaDexN2Xho8NChRcb3rFQJevh5INVKfuOSEC+O qNRi2hZb0VY7xCuoodhhq3Hhz+cFacwB51DSeQIsArjjPVTYlCHDg+7Fb2ta5sOfTbab jyaQ== X-Forwarded-Encrypted: i=1; AJvYcCUGaxX+YfATAIQQ8dWbA6Sn+iQaCa5IJnwqxIB/uKeyWHqiO/h1zPX55DS+68qLOtddfpXmyA7YjSXi23M=@vger.kernel.org X-Gm-Message-State: AOJu0YxenRTPD8a5kRhK9n13rjJWLkZCYg3+KUoI9b89fm5QSqngNOqB WgT9IPxlzOWExuUzwxESFb6eHBQmEIBpDKCZ/MvIs+58LEfJjTvs2UA+4A0YeY0+fU0U3XfGXTJ VSwckoOmJo5CWvSuQkF73R7EulOLBZQFdmlDbwZAGFH2zQhyx02Eo X-Gm-Gg: ASbGnctYYYECtZJObmK0Tciz/ZVrYK3jHHTGqNQ4C3MK6JY99FhUakgZgoq9clsD1/5 HaflVYtnvu513nk5mi5SkU1ujyPQEm3OFyV0soTUwWH30TrhArK9Bwrik9CPoXyY1J1Sg/R3JQM NhQcdD8iJ3gFXX8/udf8SX5UZwoXtUy/gD54yLH8+H8Cb7tycvduAaHGVR9UztFDYYS7D/0JzfT mz9kXtCO2Cwgt8HQNXw51Jzzm4LYf6lHJSg+mddA5K1SUx2svG2pq7xyVDu3QsIK0Kone60DO09 1bb3Wprf3qlZOtdQIsbdF4gHs/E8Ca2kbMvkP1+suhaInsfYlYDSq5LjjwuWhtCNZgxNTvLoxz1 r/2wrUqMbmYhbqezm X-Google-Smtp-Source: AGHT+IGpBSpTcUsGiFW1y6xN6RAigIl6sP5f7WQURKVhSLye12F+OPVHcs9jbRx67/PqsX7m7H0LAVn7y4om X-Received: by 2002:a05:6512:2209:b0:57a:2be1:d766 with SMTP id 2adb3069b0e04-591d854a5a5mr3690398e87.4.1761174820364; Wed, 22 Oct 2025 16:13:40 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([2620:125:9017:12:36:3:5:0]) by smtp-relay.gmail.com with ESMTPS id 2adb3069b0e04-592f4ce7a08sm35160e87.29.2025.10.22.16.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Oct 2025 16:13:40 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (dev-csander.dev.purestorage.com [10.7.70.37]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id B07D23406A7; Wed, 22 Oct 2025 17:13:38 -0600 (MDT) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id AD471E4181C; Wed, 22 Oct 2025 17:13:38 -0600 (MDT) From: Caleb Sander Mateos To: Jens Axboe , Miklos Szeredi , Ming Lei , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chris Mason , David Sterba Cc: io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-btrfs@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH 2/3] io_uring/uring_cmd: call io_should_terminate_tw() when needed Date: Wed, 22 Oct 2025 17:13:25 -0600 Message-ID: <20251022231326.2527838-3-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251022231326.2527838-1-csander@purestorage.com> References: <20251022231326.2527838-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Most uring_cmd task work callbacks don't check IO_URING_F_TASK_DEAD. But it's computed unconditionally in io_uring_cmd_work(). Add a helper io_uring_cmd_should_terminate_tw() and call it instead of checking IO_URING_F_TASK_DEAD in the one callback, fuse_uring_send_in_task(). Remove the now unused IO_URING_F_TASK_DEAD. Signed-off-by: Caleb Sander Mateos --- fs/fuse/dev_uring.c | 2 +- include/linux/io_uring/cmd.h | 7 ++++++- include/linux/io_uring_types.h | 1 - io_uring/uring_cmd.c | 6 +----- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index f6b12aebb8bb..71b0c9662716 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -1214,11 +1214,11 @@ static void fuse_uring_send_in_task(struct io_uring= _cmd *cmd, { struct fuse_ring_ent *ent =3D uring_cmd_to_ring_ent(cmd); struct fuse_ring_queue *queue =3D ent->queue; int err; =20 - if (!(issue_flags & IO_URING_F_TASK_DEAD)) { + if (!io_uring_cmd_should_terminate_tw(cmd)) { err =3D fuse_uring_prepare_send(ent, ent->fuse_req); if (err) { fuse_uring_next_fuse_req(ent, queue, issue_flags); return; } diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index 7509025b4071..b84b97c21b43 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -1,11 +1,11 @@ /* SPDX-License-Identifier: GPL-2.0-or-later */ #ifndef _LINUX_IO_URING_CMD_H #define _LINUX_IO_URING_CMD_H =20 #include -#include +#include #include =20 /* only top 8 bits of sqe->uring_cmd_flags for kernel internal use */ #define IORING_URING_CMD_CANCELABLE (1U << 30) /* io_uring_cmd is being issued again */ @@ -143,10 +143,15 @@ static inline void io_uring_cmd_complete_in_task(stru= ct io_uring_cmd *ioucmd, io_uring_cmd_tw_t task_work_cb) { __io_uring_cmd_do_in_task(ioucmd, task_work_cb, 0); } =20 +static inline bool io_uring_cmd_should_terminate_tw(struct io_uring_cmd *c= md) +{ + return io_should_terminate_tw(cmd_to_io_kiocb(cmd)->ctx); +} + static inline struct task_struct *io_uring_cmd_get_task(struct io_uring_cm= d *cmd) { return cmd_to_io_kiocb(cmd)->tctx->task; } =20 diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index c2ea6280901d..278c4a25c9e8 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -37,11 +37,10 @@ enum io_uring_cmd_flags { IO_URING_F_IOPOLL =3D (1 << 10), =20 /* set when uring wants to cancel a previously issued command */ IO_URING_F_CANCEL =3D (1 << 11), IO_URING_F_COMPAT =3D (1 << 12), - IO_URING_F_TASK_DEAD =3D (1 << 13), }; =20 struct io_wq_work_node { struct io_wq_work_node *next; }; diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index d1e3ba62ee8e..35bdac35cf4d 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -114,17 +114,13 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd= *cmd, EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); =20 static void io_uring_cmd_work(struct io_kiocb *req, io_tw_token_t tw) { struct io_uring_cmd *ioucmd =3D io_kiocb_to_cmd(req, struct io_uring_cmd); - unsigned int flags =3D IO_URING_F_COMPLETE_DEFER; - - if (io_should_terminate_tw(req->ctx)) - flags |=3D IO_URING_F_TASK_DEAD; =20 /* task_work executor checks the deffered list completion */ - ioucmd->task_work_cb(ioucmd, flags); + ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); } =20 void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, io_uring_cmd_tw_t task_work_cb, unsigned flags) --=20 2.45.2 From nobody Sun Feb 8 02:08:31 2026 Received: from mail-lf1-f98.google.com (mail-lf1-f98.google.com [209.85.167.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6FDFF307AF7 for ; Wed, 22 Oct 2025 23:13:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.98 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174828; cv=none; b=hT33wPCo6UOVmR0NThooAae9V14YJ/j2phzh6uwucd1t+zDke4JnA9HURTfJ3ONun0JwgNtQK5Fmqobs7emLkSI+FQt6S3Qb9cbbdPNfx7ZkXcnPTLeI/QpmgoIFKi5Z6es1TWk8be68QCcPz5XXzNBsxgXhLY3yVN+LLBhGCD0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1761174828; c=relaxed/simple; bh=Npt8CIamNyEuH3sVd1A1KqiVJoYsn6LwQFxIQ9NJE+E=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Oa9zo3KUZCeysk6oFc6Fsjw4O5oZ73tkSth+Vcv6q9oklVx7A5Gb30jgCmKCyj54Kw1sUMJUMPVOOxM3YxJbPPAlR+UFeXwgAdAKBXor9GidFyvZRp3XrIxEmuwtJxSHiQnZW6pNckx1mPuc7wprPoCQBNQDG8U53BSb6Uti44E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=FKG+sKvH; arc=none smtp.client-ip=209.85.167.98 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="FKG+sKvH" Received: by mail-lf1-f98.google.com with SMTP id 2adb3069b0e04-57b35d221e2so37935e87.3 for ; Wed, 22 Oct 2025 16:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1761174823; x=1761779623; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NE/3amh4M7tNgNnZVfHQb2pFOExzJd/f5eLPCJymwKM=; b=FKG+sKvHTvGcglwczO8wfj2/qFsvKpmkan7Z2dvNuGaNMH4b6mohpLKvaBPjCLdnVj m5d2P3GEmi82guV2rsVe5LJn6XJLAEWJwS3tj4mBbwI+8u0emVR6vXHVpjkKp2WHmFmi HeFKKgIWd+aYDGxqxRp+ciZZFg63dHTIUFxlmIYZe/fAJbVEUX0F6/spr8ZBztFsqN7R iLYjL7LzOd9bJ//8mJZAtTMEcuM5ir/jOAFbON4z5eTfVfFJDO8qqisIFxJmIH7KXY4x 145zSd3Bjk+qmQj9S7CMJHIdPMjqqynkemZkaGRIw6kuBCws2IulbKUN7X+GN3TVbdi/ sncw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1761174823; x=1761779623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NE/3amh4M7tNgNnZVfHQb2pFOExzJd/f5eLPCJymwKM=; b=OWlFyRGJVtDYpMwbfr7aVy4aQNXeP2XGG8PcmLcwHp3COiDaJpApqoPkHljHWkVEqT P0j0lBgHDcFPNEM8JtGT3Nj2dzmrbWLWI+Vsi0nrYggpPjd69ZeAyS0p9PSiauhkk59+ DHnoHTOT3GbBzYMw3fH24PYYD/iBxk/WNTvvx/Yrwfn8JO9eeCATcGdrKkMcKVXhD1VX FPn8VifOTd2sfZdgsCbWvkL9WM/PKn2gR4SMbQxW9Fo+5iPcfo98ahVBUrFB3Iu8BBW8 HARClfAR2mev/jA6jPayS6lBNRRKzpwKVJYEE2NsBcyfvMwHWvknM2GUWMyI4wZR0vBy 84VQ== X-Forwarded-Encrypted: i=1; AJvYcCUINzn5zS06uv/pAUSsPdIA5Us7ah4mThkKzaZEARUwdXeM2JHzgoQGuq2K2gKveaRLxnzqWi1l3QZJKKE=@vger.kernel.org X-Gm-Message-State: AOJu0YxR/KAK5FYlv/yvT5PhIN6MQ/3ST4Y9MdAU/KQBMLuiZrjdCoIk 5c9cWk3KVXmQeLS3r4dq4Id6+VVC/zRCmx110f/WYC+jGTw0atgssBesyhOkZlgvmKmc2f3Iu5s I+1z/OIKGhJZemdUWubuD2To8KRdgMNDszTtSE89N+/PVO9xjSBPe X-Gm-Gg: ASbGnctb64lzg/JOxr2AN6PFtxWuhDSD/sYbDeoW7ebJyCQygzc9CQNQu7MCibnNLjD yP5WViHXVsdG5NfDM74jUqe07tkPC2N0pSMtR87CM3dd8CJvYDm+ETtfeQq62ev3zDB/XReLk4r Hoqy1HpJmwR2Qpl7bkbzdaywN+PFr11sEnwHNaRTERKOZfd9BOBanXfsnJgEedhg/xyM/CUbb1d W/4LEA6uCxaHShFMqfwbY9CpD/b00V+aFsSK2xKd++amZ6IHGURPWnpC7rWF98UiBgSowhaKreV RxmxIBZDvfV0ALHanySM6dOAmCy1P5G3WnklOBD7dIBFZCGs47e6k5K9upcnbr5jmq16SPg4Zw6 08z1DNO499yWrs7dE X-Google-Smtp-Source: AGHT+IEu6qVydvgz5FX5gNAmjAkNmVg8S3aqqXExnZUf61/Sip3v+pDczaTy7pMfPzfVmQFRBGfoB1U3Opib X-Received: by 2002:a05:6512:401b:b0:58b:212:b0b5 with SMTP id 2adb3069b0e04-591ea30a314mr1687437e87.7.1761174823283; Wed, 22 Oct 2025 16:13:43 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([2620:125:9017:12:36:3:5:0]) by smtp-relay.gmail.com with ESMTPS id 2adb3069b0e04-592f4e502d3sm34119e87.55.2025.10.22.16.13.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Oct 2025 16:13:43 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (dev-csander.dev.purestorage.com [10.7.70.37]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 2290F3407D1; Wed, 22 Oct 2025 17:13:39 -0600 (MDT) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id 20207E4181C; Wed, 22 Oct 2025 17:13:39 -0600 (MDT) From: Caleb Sander Mateos To: Jens Axboe , Miklos Szeredi , Ming Lei , Keith Busch , Christoph Hellwig , Sagi Grimberg , Chris Mason , David Sterba Cc: io-uring@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, linux-btrfs@vger.kernel.org, linux-kernel@vger.kernel.org, Caleb Sander Mateos Subject: [PATCH 3/3] io_uring/uring_cmd: avoid double indirect call in task work dispatch Date: Wed, 22 Oct 2025 17:13:26 -0600 Message-ID: <20251022231326.2527838-4-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20251022231326.2527838-1-csander@purestorage.com> References: <20251022231326.2527838-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" io_uring task work dispatch makes an indirect call to struct io_kiocb's io_task_work.func field to allow running arbitrary task work functions. In the uring_cmd case, this calls io_uring_cmd_work(), which immediately makes another indirect call to struct io_uring_cmd's task_work_cb field. Introduce a macro DEFINE_IO_URING_CMD_TASK_WORK() to define a io_req_tw_func_t function wrapping an io_uring_cmd_tw_t. Convert the io_uring_cmd_tw_t function to the io_req_tw_func_t function in io_uring_cmd_complete_in_task() and io_uring_cmd_do_in_task_lazy(). Use DEFINE_IO_URING_CMD_TASK_WORK() to define a io_req_tw_func_t function for each existing io_uring_cmd_tw_t function. Now uring_cmd task work dispatch makes a single indirect call to the io_req_tw_func_t wrapper function, which can inline the io_uring_cmd_tw_t function. This also allows removing the task_work_cb field from struct io_uring_cmd, freeing up some additional storage space. Signed-off-by: Caleb Sander Mateos --- block/ioctl.c | 1 + drivers/block/ublk_drv.c | 3 +++ drivers/nvme/host/ioctl.c | 1 + fs/btrfs/ioctl.c | 1 + fs/fuse/dev_uring.c | 1 + include/linux/io_uring/cmd.h | 45 +++++++++++++++++++++--------------- io_uring/uring_cmd.c | 13 ++--------- 7 files changed, 36 insertions(+), 29 deletions(-) diff --git a/block/ioctl.c b/block/ioctl.c index d7489a56b33c..7a3756863c9b 100644 --- a/block/ioctl.c +++ b/block/ioctl.c @@ -776,10 +776,11 @@ static void blk_cmd_complete(struct io_uring_cmd *cmd= , unsigned int issue_flags) if (bic->res =3D=3D -EAGAIN && bic->nowait) io_uring_cmd_issue_blocking(cmd); else io_uring_cmd_done(cmd, bic->res, issue_flags); } +static DEFINE_IO_URING_CMD_TASK_WORK(blk_cmd_complete) =20 static void bio_cmd_bio_end_io(struct bio *bio) { struct io_uring_cmd *cmd =3D bio->bi_private; struct blk_iou_cmd *bic =3D io_uring_cmd_to_pdu(cmd, struct blk_iou_cmd); diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 0c74a41a6753..829b049c7c75 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1354,10 +1354,11 @@ static void ublk_cmd_tw_cb(struct io_uring_cmd *cmd, struct ublk_uring_cmd_pdu *pdu =3D ublk_get_uring_cmd_pdu(cmd); struct ublk_queue *ubq =3D pdu->ubq; =20 ublk_dispatch_req(ubq, pdu->req, issue_flags); } +static DEFINE_IO_URING_CMD_TASK_WORK(ublk_cmd_tw_cb) =20 static void ublk_queue_cmd(struct ublk_queue *ubq, struct request *rq) { struct io_uring_cmd *cmd =3D ubq->ios[rq->tag].cmd; struct ublk_uring_cmd_pdu *pdu =3D ublk_get_uring_cmd_pdu(cmd); @@ -1378,10 +1379,11 @@ static void ublk_cmd_list_tw_cb(struct io_uring_cmd= *cmd, rq->rq_next =3D NULL; ublk_dispatch_req(rq->mq_hctx->driver_data, rq, issue_flags); rq =3D next; } while (rq); } +static DEFINE_IO_URING_CMD_TASK_WORK(ublk_cmd_list_tw_cb) =20 static void ublk_queue_cmd_list(struct ublk_io *io, struct rq_list *l) { struct io_uring_cmd *cmd =3D io->cmd; struct ublk_uring_cmd_pdu *pdu =3D ublk_get_uring_cmd_pdu(cmd); @@ -2529,10 +2531,11 @@ static void ublk_ch_uring_cmd_cb(struct io_uring_cm= d *cmd, int ret =3D ublk_ch_uring_cmd_local(cmd, issue_flags); =20 if (ret !=3D -EIOCBQUEUED) io_uring_cmd_done(cmd, ret, issue_flags); } +static DEFINE_IO_URING_CMD_TASK_WORK(ublk_ch_uring_cmd_cb) =20 static int ublk_ch_uring_cmd(struct io_uring_cmd *cmd, unsigned int issue_= flags) { if (unlikely(issue_flags & IO_URING_F_CANCEL)) { ublk_uring_cmd_cancel_fn(cmd, issue_flags); diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index c212fa952c0f..d4ca46b3abc7 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -405,10 +405,11 @@ static void nvme_uring_task_cb(struct io_uring_cmd *i= oucmd, =20 if (pdu->bio) blk_rq_unmap_user(pdu->bio); io_uring_cmd_done32(ioucmd, pdu->status, pdu->result, issue_flags); } +static DEFINE_IO_URING_CMD_TASK_WORK(nvme_uring_task_cb) =20 static enum rq_end_io_ret nvme_uring_cmd_end_io(struct request *req, blk_status_t err) { struct io_uring_cmd *ioucmd =3D req->end_io_data; diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 185bef0df1c2..9d395f034403 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -4704,10 +4704,11 @@ static void btrfs_uring_read_finished(struct io_uri= ng_cmd *cmd, unsigned int iss kfree(priv->pages); kfree(priv->iov); kfree(priv); kfree(bc->data); } +static DEFINE_IO_URING_CMD_TASK_WORK(btrfs_uring_read_finished) =20 void btrfs_uring_read_extent_endio(void *ctx, int err) { struct btrfs_uring_priv *priv =3D ctx; struct io_btrfs_cmd *bc =3D io_uring_cmd_to_pdu(priv->cmd, struct io_btrf= s_cmd); diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c index 71b0c9662716..e2c87c01e021 100644 --- a/fs/fuse/dev_uring.c +++ b/fs/fuse/dev_uring.c @@ -1226,10 +1226,11 @@ static void fuse_uring_send_in_task(struct io_uring= _cmd *cmd, err =3D -ECANCELED; } =20 fuse_uring_send(ent, cmd, err, issue_flags); } +static DEFINE_IO_URING_CMD_TASK_WORK(fuse_uring_send_in_task) =20 static struct fuse_ring_queue *fuse_uring_task_to_queue(struct fuse_ring *= ring) { unsigned int qid; struct fuse_ring_queue *queue; diff --git a/include/linux/io_uring/cmd.h b/include/linux/io_uring/cmd.h index b84b97c21b43..5d6e30cc9b0b 100644 --- a/include/linux/io_uring/cmd.h +++ b/include/linux/io_uring/cmd.h @@ -9,18 +9,13 @@ /* only top 8 bits of sqe->uring_cmd_flags for kernel internal use */ #define IORING_URING_CMD_CANCELABLE (1U << 30) /* io_uring_cmd is being issued again */ #define IORING_URING_CMD_REISSUE (1U << 31) =20 -typedef void (*io_uring_cmd_tw_t)(struct io_uring_cmd *cmd, - unsigned issue_flags); - struct io_uring_cmd { struct file *file; const struct io_uring_sqe *sqe; - /* callback to defer completions to task context */ - io_uring_cmd_tw_t task_work_cb; u32 cmd_op; u32 flags; u8 pdu[32]; /* available inline for free use */ }; =20 @@ -58,13 +53,29 @@ int io_uring_cmd_import_fixed_vec(struct io_uring_cmd *= ioucmd, */ void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret, u64 res2, unsigned issue_flags, bool is_cqe32); =20 void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, + io_req_tw_func_t task_work_cb, unsigned flags); =20 +/* + * uring_cmd_cb should be a function with the signature + * void (struct io_uring_cmd *cmd, unsigned issue_flags) + */ +#define IO_URING_CMD_TASK_WORK(uring_cmd_cb) uring_cmd_cb##_tw + +#define DEFINE_IO_URING_CMD_TASK_WORK(uring_cmd_cb) \ +void \ +IO_URING_CMD_TASK_WORK(uring_cmd_cb)(struct io_kiocb *req, io_tw_token_t t= w) \ +{ \ + struct io_uring_cmd *ioucmd =3D io_kiocb_to_cmd(req, struct io_uring_cmd)= ;\ + \ + /* task_work executor checks the deferred list completion */ \ + uring_cmd_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); \ +} + /* * Note: the caller should never hard code @issue_flags and only use the * mask provided by the core io_uring code. */ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, @@ -107,11 +118,11 @@ static inline int io_uring_cmd_import_fixed_vec(struc= t io_uring_cmd *ioucmd, static inline void __io_uring_cmd_done(struct io_uring_cmd *cmd, s32 ret, u64 ret2, unsigned issue_flags, bool is_cqe32) { } static inline void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, unsigned flags) + io_req_tw_func_t task_work_cb, unsigned flags) { } static inline void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, unsigned int issue_flags) { @@ -131,21 +142,19 @@ static inline bool io_uring_mshot_cmd_post_cqe(struct= io_uring_cmd *ioucmd, return true; } #endif =20 /* users must follow the IOU_F_TWQ_LAZY_WAKE semantics */ -static inline void io_uring_cmd_do_in_task_lazy(struct io_uring_cmd *ioucm= d, - io_uring_cmd_tw_t task_work_cb) -{ - __io_uring_cmd_do_in_task(ioucmd, task_work_cb, IOU_F_TWQ_LAZY_WAKE); -} - -static inline void io_uring_cmd_complete_in_task(struct io_uring_cmd *iouc= md, - io_uring_cmd_tw_t task_work_cb) -{ - __io_uring_cmd_do_in_task(ioucmd, task_work_cb, 0); -} +#define io_uring_cmd_do_in_task_lazy(ioucmd, uring_cmd_cb) \ + __io_uring_cmd_do_in_task((ioucmd), \ + IO_URING_CMD_TASK_WORK(uring_cmd_cb), \ + IOU_F_TWQ_LAZY_WAKE) + +#define io_uring_cmd_complete_in_task(ioucmd, uring_cmd_cb) \ + __io_uring_cmd_do_in_task((ioucmd), \ + IO_URING_CMD_TASK_WORK(uring_cmd_cb), \ + 0) =20 static inline bool io_uring_cmd_should_terminate_tw(struct io_uring_cmd *c= md) { return io_should_terminate_tw(cmd_to_io_kiocb(cmd)->ctx); } diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 35bdac35cf4d..5a80d35658dc 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -111,29 +111,20 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd= *cmd, io_ring_submit_unlock(ctx, issue_flags); } } EXPORT_SYMBOL_GPL(io_uring_cmd_mark_cancelable); =20 -static void io_uring_cmd_work(struct io_kiocb *req, io_tw_token_t tw) -{ - struct io_uring_cmd *ioucmd =3D io_kiocb_to_cmd(req, struct io_uring_cmd); - - /* task_work executor checks the deffered list completion */ - ioucmd->task_work_cb(ioucmd, IO_URING_F_COMPLETE_DEFER); -} - void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd, - io_uring_cmd_tw_t task_work_cb, + io_req_tw_func_t task_work_cb, unsigned flags) { struct io_kiocb *req =3D cmd_to_io_kiocb(ioucmd); =20 if (WARN_ON_ONCE(req->flags & REQ_F_APOLL_MULTISHOT)) return; =20 - ioucmd->task_work_cb =3D task_work_cb; - req->io_task_work.func =3D io_uring_cmd_work; + req->io_task_work.func =3D task_work_cb; __io_req_task_work_add(req, flags); } EXPORT_SYMBOL_GPL(__io_uring_cmd_do_in_task); =20 static inline void io_req_set_cqe32_extra(struct io_kiocb *req, --=20 2.45.2