From nobody Thu Apr 2 22:04:03 2026 Received: from mail-pg1-f225.google.com (mail-pg1-f225.google.com [209.85.215.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C0743427A03 for ; Mon, 2 Mar 2026 17:29:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.225 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772472565; cv=none; b=lldmJrYzhQiQ9EmmXVmnd8aS5TjF1z67YZ7TzZyqyfmC+5e6C1cXKqywF7e0qTOqZUyFVC4ptByz/nC47mHTWvXSGTn9zMRVRKJ4E2hZAW859fpoYHwLiI/JAHQjhLnAQ7XyqZFhHlFwVWkOmt+29UdcnKSzj0eP6jDXg4DTqUk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772472565; c=relaxed/simple; bh=EHWetMLtF/7h2Jzc9WukOzl1UiPE5W/f2SGGcvcIhqM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UbzBvQAySB3JxspVquTMqK+juAF82izzLUeU/wljDYBIvwFLQsENVMR2y7ZNsuAyOnawhKXpd9EimrRXUzMweDpnUeqIQ7K+Ij81j62M9V246ems67Y7inYtGrW2w0v7X9aKNsU78Y4pl/qC2UACw5DBlu0RJlP0cHlP1lmsCXM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=IX543Idp; arc=none smtp.client-ip=209.85.215.225 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="IX543Idp" Received: by mail-pg1-f225.google.com with SMTP id 41be03b00d2f7-c6e85f76efeso313122a12.1 for ; Mon, 02 Mar 2026 09:29:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1772472563; x=1773077363; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UJYfmLiCMCsowVS6yyBvopz5p0Wjd6fpqKLBR9qiyiM=; b=IX543Idpvg6IdhRShudCrNaH9AYqWbuEJe4zU+fFBSBzVZ0imoaSD5Iy+f93AJR87w JB9o397S8WwPnUL/2c6uJgCyYe5/mKIpsoe2736oKt+Wh3ekV7EkMD07P8peCVyRcEa1 aUMxj+J6Jz74Xlz2eSX8mWWIH7cWbwtEqpplg9lDr+UNyZ4GUUEvfU7SW6wc/d5/3ya0 sAngLNmvUZnjSMc/QZq8+V7anfYIZ+h7qZiHU7dCmFDTcUJZUXXo6F8cnxjadcGPzf+E WsJShp1KP9rOVQqf4+dhK3F+T0Q2VeCfReqaZOhM8bP3yTQbjWpuvPzCFIWUJQinJbEg WaNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772472563; x=1773077363; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=UJYfmLiCMCsowVS6yyBvopz5p0Wjd6fpqKLBR9qiyiM=; b=ZuI6nDqP8KRQhBW8GV3K1jZzfIJ79VEMPZ2cUmUE5J7SqQQG+smIu6t1QPeXRjQIm+ cyAxNmgiCCIkIS9QQrC4BBqv5veB6/xcXxn4PYnY6dpPaERKue1RvokYjRFIgUVYJfgg DXOh65UPCbuW8tpm2ft+ZQbO0FAU2ZvUEqCL99MeplrBZRaRBCxyF6RN1gCpoMewurBz uxIj5VJN+S5UCe/1hnW0NGR9CmEJCy2gVPa8Gd1fKjemDGOlJELmpm3l+RHeVRq20+6F GDnMuVl+904gGJ+XQJ/4GHEP8MzGzVQJwghHdlcwrxcJRf+uaeg9YSVSxY+eQtlZ4IcC n2tw== X-Forwarded-Encrypted: i=1; AJvYcCV/LS2pctr0Ztxz/tx4fFkHcvmya1fnu19H7TgJaBkLQKJbrFelPNYt5IPDLHH/K01LekWai856VttWqcc=@vger.kernel.org X-Gm-Message-State: AOJu0YzxNSVT/pxFrKkqQ4Ur9ml269L33aHPWWlvPWgPYt71pJ2Vj7At 1h3+TY/LIBiWfyXip9RMt/vHXU0QmwNvKko0Xq36rX9Ns4UvttIwTXFd29ABm9bJizBCtKJjxw2 /lM5C79GaZhxeOOBQM0wLnT14+awk9qClMmCT X-Gm-Gg: ATEYQzziwUhe6KudikzJBIUzVuaUq+W2X19fmxwwhbqEXo/8m4QmM4ZYrM3QjVIaAMN Ugg+lLoxBZQGngqrg1ELaDsV/J0WLmdz2a0SCK8Czdo9TtRxXjPFkD0UpwGB9J2j5Xfib7xKDuB hSD1cTzxBaE/3IhqKGhhkuByj8OMIX6JAafPTG86Ap1qQnBvNT/aTLFmDwO6rXJanH3cpUvx4hb fEbVY+cqT1+Voiz8w4m9tSzdSdFNQaXehPQv1ncSgMmsA7jO8ak+ZCpUMYDhuzyT7rrO2aqNEz1 Ob2ZW1GXbw32LC898d4JllN7B5+ngAvUxWFbDPySPVfMvylx5kcmEwPDZdmZ3nPvSrFi6fFWf7f 8A8ViO9gnZLXER2Gb3KLQvwhkUPf4qwlglCS0CmeM1TSw4O2VHEVunA== X-Received: by 2002:a17:90b:3c09:b0:359:8d95:4a57 with SMTP id 98e67ed59e1d1-3598d954d03mr2682124a91.6.1772472563024; Mon, 02 Mar 2026 09:29:23 -0800 (PST) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.128]) by smtp-relay.gmail.com with ESMTPS id 98e67ed59e1d1-3597c40a81asm921671a91.3.2026.03.02.09.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 09:29:23 -0800 (PST) X-Relaying-Domain: purestorage.com Received: from dev-csander.dev.purestorage.com (dev-csander.dev.purestorage.com [10.112.29.101]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id F0454340506; Mon, 2 Mar 2026 10:29:21 -0700 (MST) Received: by dev-csander.dev.purestorage.com (Postfix, from userid 1557716354) id EDAC2E41FBD; Mon, 2 Mar 2026 10:29:21 -0700 (MST) From: Caleb Sander Mateos To: Jens Axboe , Christoph Hellwig , Keith Busch , Sagi Grimberg Cc: io-uring@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, Anuj Gupta , Kanchan Joshi , Ming Lei , Caleb Sander Mateos Subject: [PATCH v5 3/5] io_uring: count CQEs in io_iopoll_check() Date: Mon, 2 Mar 2026 10:29:12 -0700 Message-ID: <20260302172914.2488599-4-csander@purestorage.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20260302172914.2488599-1-csander@purestorage.com> References: <20260302172914.2488599-1-csander@purestorage.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A subsequent commit will allow uring_cmds that don't use iopoll on IORING_SETUP_IOPOLL io_urings. As a result, CQEs can be posted without setting the iopoll_completed flag for a request in iopoll_list or going through task work. For example, a UBLK_U_IO_FETCH_IO_CMDS command could call io_uring_mshot_cmd_post_cqe() to directly post a CQE. The io_iopoll_check() loop currently only counts completions posted in io_do_iopoll() when determining whether the min_events threshold has been met. It also exits early if there are any existing CQEs before polling, or if any CQEs are posted while running task work. CQEs posted via io_uring_mshot_cmd_post_cqe() or other mechanisms won't be counted against min_events. Explicitly check the available CQEs in each io_iopoll_check() loop iteration to account for CQEs posted in any fashion. Signed-off-by: Caleb Sander Mateos --- io_uring/io_uring.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 46f39831d27c..b4625695bb3a 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1184,11 +1184,10 @@ __cold void io_iopoll_try_reap_events(struct io_rin= g_ctx *ctx) io_move_task_work_from_local(ctx); } =20 static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned int min_event= s) { - unsigned int nr_events =3D 0; unsigned long check_cq; =20 min_events =3D min(min_events, ctx->cq_entries); =20 lockdep_assert_held(&ctx->uring_lock); @@ -1227,34 +1226,30 @@ static int io_iopoll_check(struct io_ring_ctx *ctx,= unsigned int min_events) * the poll to the issued list. Otherwise we can spin here * forever, while the workqueue is stuck trying to acquire the * very same mutex. */ if (list_empty(&ctx->iopoll_list) || io_task_work_pending(ctx)) { - u32 tail =3D ctx->cached_cq_tail; - (void) io_run_local_work_locked(ctx, min_events); =20 if (task_work_pending(current) || list_empty(&ctx->iopoll_list)) { mutex_unlock(&ctx->uring_lock); io_run_task_work(); mutex_lock(&ctx->uring_lock); } /* some requests don't go through iopoll_list */ - if (tail !=3D ctx->cached_cq_tail || list_empty(&ctx->iopoll_list)) + if (list_empty(&ctx->iopoll_list)) break; } ret =3D io_do_iopoll(ctx, !min_events); if (unlikely(ret < 0)) return ret; =20 if (task_sigpending(current)) return -EINTR; if (need_resched()) break; - - nr_events +=3D ret; - } while (nr_events < min_events); + } while (io_cqring_events(ctx) < min_events); =20 return 0; } =20 void io_req_task_complete(struct io_tw_req tw_req, io_tw_token_t tw) --=20 2.45.2