From nobody Sun May 10 09:53:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72BB6C433EF for ; Fri, 6 May 2022 07:01:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1386999AbiEFHEs (ORCPT ); Fri, 6 May 2022 03:04:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1386846AbiEFHEo (ORCPT ); Fri, 6 May 2022 03:04:44 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7284B4BFF9; Fri, 6 May 2022 00:01:02 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id k1so6565338pll.4; Fri, 06 May 2022 00:01:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H4H/ujLwZkevJ+kG7X3STChb0etd+nfQjZIW+BhHkzQ=; b=XUsDqZyOfidnyER7oZHobTEXif6iB2Zft2CeI23d+NqfP7AG1NC3Q9qji6BUQXINFE QohXu3VmT9LU1s7SIpVl0+Ba6hsNs5aqTVpjh5vu+YLUmM8PEB/eCUx9Mg96hY0lQ7d/ 6ZE/dynGBZxGrgwV+EHrRkjWiQ5Kkbw6u4IApJcB5foRF1/4iHX3o8+LyiMGUmtXeNCl eF5E8W+HRZg66Nck4WYi72+f/RaaWT18oUZduypAKU7neKgBZJm3k20gu2L4dNzpr86p VyInWAHDZnLRR9XHOLzxfnKQ5aClNHbWfkPsr8PB8l8UNIJpEZixUQpQmrJoCanoVp/U r0BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=H4H/ujLwZkevJ+kG7X3STChb0etd+nfQjZIW+BhHkzQ=; b=6VT5sy9MQjlGH+D4lw8dWnUNpSfuaXDICjchufAYrH9E1G8id1QRwkE54JAOL49Mpx RMaZlq8uqIAL952+0a5f7YMRx/6OZhGSTAZP2FQTJrGgoPZ7wSm8wmHivGa2Qot8ky5W vl7BPsKS5Bg0a7ujwBhkXWUrfz/LlQcSiN3aja2uRKgva9IyFxyb6kkoqf+yTSWWEZSN G9cjVD1WMu0clUd3Tu6qJ6y2f+9nA7vUg98Kj1OGpgYYmK2P2p95Y6/1f3B00tXTP0Us 8ZnSVY52Wu6QDj21LNQsdD2lud093bMqh6Yz/B6EPRTnl987fNjYMxXsYQqS0oJXFzUh I8Tg== X-Gm-Message-State: AOAM531kILJOTUOtHVulRpXoda+J/vqyXphNjRxp85A/PQxENNnrrA+h nzTbWa0pdzpMuj+NKy95x9ytRI0Ujuc= X-Google-Smtp-Source: ABdhPJxzWehS2TRMetLSpGJsTjNE4wMqT85i9y24tT6h9YxGwkCmceawoI5H+G7TlsEP1yGpa4tpCg== X-Received: by 2002:a17:90b:4b4b:b0:1dc:8724:3f75 with SMTP id mi11-20020a17090b4b4b00b001dc87243f75mr10602435pjb.178.1651820461859; Fri, 06 May 2022 00:01:01 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([106.53.4.151]) by smtp.gmail.com with ESMTPSA id bh2-20020a170902a98200b0015e8d4eb2desm813112plb.296.2022.05.06.00.00.58 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 May 2022 00:01:01 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , linux-kernel@vger.kernel.org Subject: [PATCH 1/5] io_uring: add IORING_ACCEPT_MULTISHOT for accept Date: Fri, 6 May 2022 15:00:58 +0800 Message-Id: <20220506070102.26032-2-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220506070102.26032-1-haoxu.linux@gmail.com> References: <20220506070102.26032-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hao Xu add an accept_flag IORING_ACCEPT_MULTISHOT for accept, which is to support multishot. Signed-off-by: Hao Xu --- include/uapi/linux/io_uring.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h index fad63564678a..73bc7e54ac18 100644 --- a/include/uapi/linux/io_uring.h +++ b/include/uapi/linux/io_uring.h @@ -213,6 +213,11 @@ enum { #define IORING_ASYNC_CANCEL_FD (1U << 1) #define IORING_ASYNC_CANCEL_ANY (1U << 2) =20 +/* + * accept flags stored in accept_flags + */ +#define IORING_ACCEPT_MULTISHOT (1U << 15) + /* * IO completion data structure (Completion Queue Entry) */ --=20 2.36.0 From nobody Sun May 10 09:53:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73EA7C433EF for ; Fri, 6 May 2022 07:01:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235091AbiEFHEy (ORCPT ); Fri, 6 May 2022 03:04:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1386883AbiEFHEr (ORCPT ); Fri, 6 May 2022 03:04:47 -0400 Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95A8D4D63F; Fri, 6 May 2022 00:01:05 -0700 (PDT) Received: by mail-pl1-x633.google.com with SMTP id x18so6567984plg.6; Fri, 06 May 2022 00:01:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2AGe5Fmx04dYyhCcUtd7dBWkTmPQ4OwhfuehIUMvDJ4=; b=OKbex0vmp7kWmxvwwJRScYXYCARzcyb0nJnHPf3sRu+k66EkOQ06HAMncYe+24pUxi TyHl+M13kcUJRh/zR0NBQaYQTmQZA8J7i8PsjJjy4KGMZsQu7qJjcaskP605ZycJAZJV wymev/cUEgPhmDG92TcNn0+wTBmJYyiNAebidcEQK8oIJnwL2hLYrg09wF7L1VovvPoC zepjJFdTM8C15D0JR86tP09qy/GM/PKCA10/X5c+fXeTeMcw4x16T/nnm0RE/8aLjIx5 TWx3E1BBeoHH0/lP0hyEldXkHok5Wn4+coS4tC6Rq0HOauwHZICaMWl7avu0jdACO9Zs F2TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2AGe5Fmx04dYyhCcUtd7dBWkTmPQ4OwhfuehIUMvDJ4=; b=F0LxwupG29IYTRgeu8q9ro3DHSEqdPwf12xz82veuS8zLHUlw+qG/fA7cDpzoCNONi j+1jEhrh+MohrhJEK8CjezzjCgGYcaQZ4db4VW3ON0NF99Ql5rmPO6urzNnQcv94pgDT 6LLjDb1nJ9lAThVYwr0cwsX5u1y2L8ZY+gkBXlS8MRdGRPCYIQDWk+2xRLeVaVUiRNID BSCEmQDczxE5e8KIcuefNtCSLSnqzlZCGcwAkEyB2G/9rSWbyBhS6s7Fp/A4poI3Sxwz 6ek1Rtx7ttJWgZGddE0bgccLs4qULrmXQYKJ5oo+ybfLP0uJ0BYraPJa2y/13m+jeooy lSBw== X-Gm-Message-State: AOAM531giu2qywdHvIkqvUjp6b6RL0BM+iYBAksHJseMEEyb4MyQeQ4J w+iXpmdufhG80y2g7nKp7WXOsiOXyDU= X-Google-Smtp-Source: ABdhPJwR/XbbxF8XjQQ5FK4kHIN7brTqpFYIY68SsdK91qG+YI+35sIDAaeLeTNGZD53eUaBxQuscA== X-Received: by 2002:a17:902:868e:b0:15a:7c9d:b11f with SMTP id g14-20020a170902868e00b0015a7c9db11fmr2214171plo.151.1651820464993; Fri, 06 May 2022 00:01:04 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([106.53.4.151]) by smtp.gmail.com with ESMTPSA id bh2-20020a170902a98200b0015e8d4eb2desm813112plb.296.2022.05.06.00.01.02 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 May 2022 00:01:04 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , linux-kernel@vger.kernel.org Subject: [PATCH 2/5] io_uring: add REQ_F_APOLL_MULTISHOT for requests Date: Fri, 6 May 2022 15:00:59 +0800 Message-Id: <20220506070102.26032-3-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220506070102.26032-1-haoxu.linux@gmail.com> References: <20220506070102.26032-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hao Xu Add a flag to indicate multishot mode for fast poll. currently only accept use it, but there may be more operations leveraging it in the future. Signed-off-by: Hao Xu --- fs/io_uring.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fs/io_uring.c b/fs/io_uring.c index 1e7466079af7..8ebb1a794e36 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -808,6 +808,7 @@ enum { REQ_F_SINGLE_POLL_BIT, REQ_F_DOUBLE_POLL_BIT, REQ_F_PARTIAL_IO_BIT, + REQ_F_APOLL_MULTISHOT_BIT, /* keep async read/write and isreg together and in order */ REQ_F_SUPPORT_NOWAIT_BIT, REQ_F_ISREG_BIT, @@ -872,6 +873,8 @@ enum { REQ_F_DOUBLE_POLL =3D BIT(REQ_F_DOUBLE_POLL_BIT), /* request has already done partial IO */ REQ_F_PARTIAL_IO =3D BIT(REQ_F_PARTIAL_IO_BIT), + /* fast poll multishot mode */ + REQ_F_APOLL_MULTISHOT =3D BIT(REQ_F_APOLL_MULTISHOT_BIT), }; =20 struct async_poll { --=20 2.36.0 From nobody Sun May 10 09:53:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEDA7C433F5 for ; Fri, 6 May 2022 07:01:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1387429AbiEFHE7 (ORCPT ); Fri, 6 May 2022 03:04:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1386888AbiEFHEv (ORCPT ); Fri, 6 May 2022 03:04:51 -0400 Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6F0E4DF52; Fri, 6 May 2022 00:01:08 -0700 (PDT) Received: by mail-pl1-x635.google.com with SMTP id k1so6565338pll.4; Fri, 06 May 2022 00:01:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jHfXPlMnu+cF/Ne1J4JISZj0/qHHaO/EGUdUthQ8P50=; b=ClRlm7Jeu2S19jE//djbOkHvQs7fSPPS9xD2IEQl2aJDc8eT2zehyNymRDfN6FDkVr aSpZGRuPf15auLxO+3gQzy6I/h7aJ1UPGrqJZKbJE9QN54S5q7oCVjIk1BCGWvUpZphA J39lBnYKChg+JBplECHXtiBjBh+HyML66Um3Kj3T1TWDIv9keWWd6ni3dfWWtvT9zI2T dtBNFkNXSXblJvccZqRgd4Uxeda3uzdIOtkCFne8y6YeSWkTX0qnGNDNAQiqYgwOyJp4 TOGoOYIRATndN7HbwWhBaMayD7gMGDvUaiT0g6yfSGJYmLTe00fsNF+bN9Ep8rGMJxxo rvtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jHfXPlMnu+cF/Ne1J4JISZj0/qHHaO/EGUdUthQ8P50=; b=lO8mM2oWLfK10mntU+ar72yI3TLpLV1/C2CMC50nTqO/1kvBJw9+UNPwUIAlJpbEyV fjo1uOTdGTH/nKVlSkoYFil8kx3kKI7zPUqaSru34TxdmRipPpeSOiKzTxM09kLE59vv WBYGBdZY7GRe7FdN4Z40RwZKuyWxzV1fpymU58fiNak2TDOVMw9CnfdadDaKVg9HUxYu bdAtLfEffwSKCcOrN3KxhtDKJTRxGvcXX2Bcur5xhmOtin/3MZosgGWWC50ZBbj5WXlt hkmN3OQRQfukDqCdPSP17dFrBud6si8t0Wc3xTHvS/faQvXu1HliqGuwUN2JXr2+cCkd Vr/A== X-Gm-Message-State: AOAM533RZXt3fHuLuqmtfcj6H3TppKI4KgECAtsPcTnTbyt6hXlwz26a Penh0MfEfJip86wAkFl5Cv/Gc1y6dEM= X-Google-Smtp-Source: ABdhPJy6+kbqk/0tT6Y3X/ztGGuFJFowIrduB7+lS9dMFWQXywhqWTqOyhukad1jRPfbpRjG77pdbg== X-Received: by 2002:a17:902:ecc4:b0:15e:a670:6056 with SMTP id a4-20020a170902ecc400b0015ea6706056mr2083042plh.108.1651820468384; Fri, 06 May 2022 00:01:08 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([106.53.4.151]) by smtp.gmail.com with ESMTPSA id bh2-20020a170902a98200b0015e8d4eb2desm813112plb.296.2022.05.06.00.01.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 May 2022 00:01:08 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , linux-kernel@vger.kernel.org Subject: [PATCH 3/5] io_uring: let fast poll support multishot Date: Fri, 6 May 2022 15:01:00 +0800 Message-Id: <20220506070102.26032-4-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220506070102.26032-1-haoxu.linux@gmail.com> References: <20220506070102.26032-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hao Xu For operations like accept, multishot is a useful feature, since we can reduce a number of accept sqe. Let's integrate it to fast poll, it may be good for other operations in the future. Signed-off-by: Hao Xu Reported-by: kernel test robot --- fs/io_uring.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 8ebb1a794e36..d33777575faf 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -5952,7 +5952,7 @@ static void io_poll_remove_entries(struct io_kiocb *r= eq) * either spurious wakeup or multishot CQE is served. 0 when it's done with * the request, then the mask is stored in req->cqe.res. */ -static int io_poll_check_events(struct io_kiocb *req, bool locked) +static int io_poll_check_events(struct io_kiocb *req, bool *locked) { struct io_ring_ctx *ctx =3D req->ctx; int v; @@ -5981,17 +5981,26 @@ static int io_poll_check_events(struct io_kiocb *re= q, bool locked) =20 /* multishot, just fill an CQE and proceed */ if (req->cqe.res && !(req->apoll_events & EPOLLONESHOT)) { - __poll_t mask =3D mangle_poll(req->cqe.res & req->apoll_events); - bool filled; - - spin_lock(&ctx->completion_lock); - filled =3D io_fill_cqe_aux(ctx, req->cqe.user_data, mask, - IORING_CQE_F_MORE); - io_commit_cqring(ctx); - spin_unlock(&ctx->completion_lock); - if (unlikely(!filled)) - return -ECANCELED; - io_cqring_ev_posted(ctx); + if (req->flags & REQ_F_APOLL_MULTISHOT) { + io_tw_lock(req->ctx, locked); + if (likely(!(req->task->flags & PF_EXITING))) + io_queue_sqe(req); + else + return -EFAULT; + } else { + __poll_t mask =3D mangle_poll(req->cqe.res & + req->apoll_events); + bool filled; + + spin_lock(&ctx->completion_lock); + filled =3D io_fill_cqe_aux(ctx, req->cqe.user_data, + mask, IORING_CQE_F_MORE); + io_commit_cqring(ctx); + spin_unlock(&ctx->completion_lock); + if (unlikely(!filled)) + return -ECANCELED; + io_cqring_ev_posted(ctx); + } } else if (req->cqe.res) { return 0; } @@ -6010,7 +6019,7 @@ static void io_poll_task_func(struct io_kiocb *req, b= ool *locked) struct io_ring_ctx *ctx =3D req->ctx; int ret; =20 - ret =3D io_poll_check_events(req, *locked); + ret =3D io_poll_check_events(req, locked); if (ret > 0) return; =20 @@ -6035,7 +6044,7 @@ static void io_apoll_task_func(struct io_kiocb *req, = bool *locked) struct io_ring_ctx *ctx =3D req->ctx; int ret; =20 - ret =3D io_poll_check_events(req, *locked); + ret =3D io_poll_check_events(req, locked); if (ret > 0) return; =20 @@ -6275,7 +6284,7 @@ static int io_arm_poll_handler(struct io_kiocb *req, = unsigned issue_flags) struct io_ring_ctx *ctx =3D req->ctx; struct async_poll *apoll; struct io_poll_table ipt; - __poll_t mask =3D EPOLLONESHOT | POLLERR | POLLPRI; + __poll_t mask =3D POLLERR | POLLPRI; int ret; =20 if (!def->pollin && !def->pollout) @@ -6284,6 +6293,8 @@ static int io_arm_poll_handler(struct io_kiocb *req, = unsigned issue_flags) return IO_APOLL_ABORTED; if ((req->flags & (REQ_F_POLLED|REQ_F_PARTIAL_IO)) =3D=3D REQ_F_POLLED) return IO_APOLL_ABORTED; + if (!(req->flags & REQ_F_APOLL_MULTISHOT)) + mask |=3D EPOLLONESHOT; =20 if (def->pollin) { mask |=3D POLLIN | POLLRDNORM; --=20 2.36.0 From nobody Sun May 10 09:53:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5701CC433F5 for ; Fri, 6 May 2022 07:01:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1387590AbiEFHFJ (ORCPT ); Fri, 6 May 2022 03:05:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1386846AbiEFHEy (ORCPT ); Fri, 6 May 2022 03:04:54 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 331FB50050; Fri, 6 May 2022 00:01:12 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id e5so5425055pgc.5; Fri, 06 May 2022 00:01:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lEkfxhWlZ3rm6gnDVC2XNnFXDrzBYLt+qna5jxuoeL8=; b=JhPvHm9Kj9rZIgKJngzFkHQrOdfhqvsHTvp5QjdogcCQasAhI5jC5yuv5ppMuFwx+0 awGu2ntdPL3F7iO1pSwmheNHYf4g/Xfra2+JuTySbMNVdIgD5u/p1WbLVHPZ1i6zQsJt Py06dKeatANftaGneNWMrlRVbHkcwCpbXkA7ydcE3LlgvxcCUx3ny4ZZiRXu8U7AFpX1 JKGPJvPHUcjNtteXHLVKMCYKOi6ja61UHY3nZ9/1ywoVuRhGSzIcEXOQ3ApKB7vRXhY4 nBzOYdEf3zM5T5sqxmUWQuCaFcdgqkEJwb7LZlW1ZhT8tuHdM8Q5lNGOb9QyEfKSHCay eUjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lEkfxhWlZ3rm6gnDVC2XNnFXDrzBYLt+qna5jxuoeL8=; b=ETk2S1iiTXWClSSjGn7kxe4XXQ1KMrVzu7pNvs5IwvUeCEmMUUOTQRKxJ0Bd2nMKKa 7q1IDAwtp37IhuAk8xOei3Za/gAzZgTpIwMsoHp/BYSJI4P+q2wuwBuCCVHlQ9z+BnTc mQO75IHQzybiAs/M1hbmu8HF7dNB2/TXM3PzA0a4Cty1UStMfurU4jljvpPlZQZxJt1P 6owq6zK9Bp3GHq0GhGjYyCfKzLaHsuq2IbrO6JvGgnnMimOXlDT3M7CKmzTq2LvoRV6D 27Qs8IgnOkEeNduYqYvyW6BzOzECs+ILNasz3FwKaFewHzToGJokkGTL401eok/uFfEj Njmw== X-Gm-Message-State: AOAM532J76sh5HGvSmT/lEs3vJ3prvwxijH6hiYsni/SNnS/uiheXtOE k0QYOsu/4E9yeb2O1gqYj+gjBHnLB9g= X-Google-Smtp-Source: ABdhPJyxBvHXm9HEIyk97EgDCmCfrE3tPbRm2gr6KIxIM0sVv4rD/DZj0ndj754aS3Zl/MCkLtQf8A== X-Received: by 2002:a63:493:0:b0:3ab:3dda:7e90 with SMTP id 141-20020a630493000000b003ab3dda7e90mr1680529pge.106.1651820471648; Fri, 06 May 2022 00:01:11 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([106.53.4.151]) by smtp.gmail.com with ESMTPSA id bh2-20020a170902a98200b0015e8d4eb2desm813112plb.296.2022.05.06.00.01.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 May 2022 00:01:11 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , linux-kernel@vger.kernel.org Subject: [PATCH 4/5] io_uring: add a helper for poll clean Date: Fri, 6 May 2022 15:01:01 +0800 Message-Id: <20220506070102.26032-5-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220506070102.26032-1-haoxu.linux@gmail.com> References: <20220506070102.26032-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hao Xu Add a helper for poll clean, it will be used in the multishot accept in the later patches. Signed-off-by: Hao Xu Reported-by: kernel test robot --- fs/io_uring.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index d33777575faf..0a83ecc457d1 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -5711,6 +5711,23 @@ static int io_accept_prep(struct io_kiocb *req, cons= t struct io_uring_sqe *sqe) return 0; } =20 +static inline void __io_poll_clean(struct io_kiocb *req) +{ + struct io_ring_ctx *ctx =3D req->ctx; + + io_poll_remove_entries(req); + spin_lock(&ctx->completion_lock); + hash_del(&req->hash_node); + spin_unlock(&ctx->completion_lock); +} + +#define REQ_F_APOLL_MULTI_POLLED (REQ_F_APOLL_MULTISHOT | REQ_F_POLLED) +static inline void io_poll_clean(struct io_kiocb *req) +{ + if ((req->flags & REQ_F_APOLL_MULTI_POLLED) =3D=3D REQ_F_APOLL_MULTI_POLL= ED) + __io_poll_clean(req); +} + static int io_accept(struct io_kiocb *req, unsigned int issue_flags) { struct io_accept *accept =3D &req->accept; @@ -6041,17 +6058,13 @@ static void io_poll_task_func(struct io_kiocb *req,= bool *locked) =20 static void io_apoll_task_func(struct io_kiocb *req, bool *locked) { - struct io_ring_ctx *ctx =3D req->ctx; int ret; =20 ret =3D io_poll_check_events(req, locked); if (ret > 0) return; =20 - io_poll_remove_entries(req); - spin_lock(&ctx->completion_lock); - hash_del(&req->hash_node); - spin_unlock(&ctx->completion_lock); + __io_poll_clean(req); =20 if (!ret) io_req_task_submit(req, locked); --=20 2.36.0 From nobody Sun May 10 09:53:20 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C9D5C433F5 for ; Fri, 6 May 2022 07:01:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1387581AbiEFHFS (ORCPT ); Fri, 6 May 2022 03:05:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53480 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1387408AbiEFHE7 (ORCPT ); Fri, 6 May 2022 03:04:59 -0400 Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1896C4BFF9; Fri, 6 May 2022 00:01:16 -0700 (PDT) Received: by mail-pj1-x102a.google.com with SMTP id j8-20020a17090a060800b001cd4fb60dccso6121058pjj.2; Fri, 06 May 2022 00:01:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=adYCTp8/5p9sggDjQM6BWyPSAca7Ngfv3XcFgStxjkc=; b=IPmiae4btyDW39ftdngHSS1cT5GPzASdQcbJdcxTwWV2jDMoOmIZEpyDC2LMtoLgzF cFAfZ9Tut4+y3nhtZXgtYG6dOYZnSMXIbY4jtdpqRf4uEijMi2R99DJruMYx/ncGVCt6 +3Fi2YGRqCETlYIEj/VlmMAz5Yw62SxIP40MhUxbp3wa80iUeJxgmN5M0l8rfG2wdkTY Wq6TKl4ynDUV3KB3/bCD3zspniDsTkicLV1GRhbioqscktKxU62QquZUVmXCXQKNHS1L qOnYo8B6mowU+jMdL7FncrumIYtrdPadVgInIBs2jTH1XZ1Q3Qzhhda9LJsruQAewARA EJhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=adYCTp8/5p9sggDjQM6BWyPSAca7Ngfv3XcFgStxjkc=; b=PerrAkghCzKietf8Q3y3uMOx+YFZj7k0wIrC+SUsq0nvka6K4kOWPnPQC2ZEhKSSl2 d9630yEzlVPJdMO/3gk4Gy7x/EtoWpzCxJOhhQS/69KIdxbfRk/jKJiyRvg130yTDDMB wEppgkRZsN3t1o42FEKG0hEGrsHCrgmUByPldWKnIs8Mo382xB/0+c1JNnr7ApVAHYKY wkgABCEesniYajO5l+V2xLGWAlUUK5/GKGN/1mYJRaRH0Y3/YmaoninMnLlqxci7w9hr lcKhrhWmaY3rTbN+Unl9blb26Gd9sdnviBRnQSz98x2Oa07cQ+/CK1Tg31au7EKzO5e8 4p9g== X-Gm-Message-State: AOAM532UqYTnkLZJuLBHsyKylKRUA3BuqsO+NFGX4SRg/p2TUSXWLuU4 8xyPBsCisy65P1W0pc5Gc4OPyb0hI+U= X-Google-Smtp-Source: ABdhPJz0lFpQEH67zBMCtKGc7qra8QdAyFWqh2slDyV2wts8Q2oOcrNvsIbTJ13IwhXpKg1n+flkgg== X-Received: by 2002:a17:903:244c:b0:15e:b3f7:950d with SMTP id l12-20020a170903244c00b0015eb3f7950dmr2067737pls.9.1651820475432; Fri, 06 May 2022 00:01:15 -0700 (PDT) Received: from HOWEYXU-MB0.tencent.com ([106.53.4.151]) by smtp.gmail.com with ESMTPSA id bh2-20020a170902a98200b0015e8d4eb2desm813112plb.296.2022.05.06.00.01.12 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 May 2022 00:01:15 -0700 (PDT) From: Hao Xu To: io-uring@vger.kernel.org Cc: Jens Axboe , Pavel Begunkov , linux-kernel@vger.kernel.org Subject: [PATCH 5/5] io_uring: implement multishot mode for accept Date: Fri, 6 May 2022 15:01:02 +0800 Message-Id: <20220506070102.26032-6-haoxu.linux@gmail.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220506070102.26032-1-haoxu.linux@gmail.com> References: <20220506070102.26032-1-haoxu.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Hao Xu Refactor io_accept() to support multishot mode. theoretical analysis: 1) when connections come in fast - singleshot: add accept sqe(userpsace) --> accept inline ^ | |-----------------| - multishot: add accept sqe(userspace) --> accept inline ^ | |--*--| we do accept repeatedly in * place until get EAGAIN 2) when connections come in at a low pressure similar thing like 1), we reduce a lot of userspace-kernel context switch and useless vfs_poll() tests: Did some tests, which goes in this way: server client(multiple) accept connect read write write read close close Basically, raise up a number of clients(on same machine with server) to connect to the server, and then write some data to it, the server will write those data back to the client after it receives them, and then close the connection after write return. Then the client will read the data and then close the connection. Here I test 10000 clients connect one server, data size 128 bytes. And each client has a go routine for it, so they come to the server in short time. test 20 times before/after this patchset, time spent:(unit cycle, which is the return value of clock()) before: 1930136+1940725+1907981+1947601+1923812+1928226+1911087+1905897+1941075 +1934374+1906614+1912504+1949110+1908790+1909951+1941672+1969525+1934984 +1934226+1914385)/20.0 =3D 1927633.75 after: 1858905+1917104+1895455+1963963+1892706+1889208+1874175+1904753+1874112 +1874985+1882706+1884642+1864694+1906508+1916150+1924250+1869060+1889506 +1871324+1940803)/20.0 =3D 1894750.45 (1927633.75 - 1894750.45) / 1927633.75 =3D 1.65% Signed-off-by: Hao Xu --- fs/io_uring.c | 54 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 48 insertions(+), 6 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 0a83ecc457d1..9febe7774dc3 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -1254,6 +1254,7 @@ static int io_close_fixed(struct io_kiocb *req, unsig= ned int issue_flags); static enum hrtimer_restart io_link_timeout_fn(struct hrtimer *timer); static void io_eventfd_signal(struct io_ring_ctx *ctx); static void io_req_tw_post_queue(struct io_kiocb *req, s32 res, u32 cflags= ); +static void io_poll_remove_entries(struct io_kiocb *req); =20 static struct kmem_cache *req_cachep; =20 @@ -5690,24 +5691,29 @@ static int io_recv(struct io_kiocb *req, unsigned i= nt issue_flags) static int io_accept_prep(struct io_kiocb *req, const struct io_uring_sqe = *sqe) { struct io_accept *accept =3D &req->accept; + bool multishot; =20 if (unlikely(req->ctx->flags & IORING_SETUP_IOPOLL)) return -EINVAL; - if (sqe->ioprio || sqe->len || sqe->buf_index) + if (sqe->len || sqe->buf_index) return -EINVAL; =20 accept->addr =3D u64_to_user_ptr(READ_ONCE(sqe->addr)); accept->addr_len =3D u64_to_user_ptr(READ_ONCE(sqe->addr2)); accept->flags =3D READ_ONCE(sqe->accept_flags); accept->nofile =3D rlimit(RLIMIT_NOFILE); + multishot =3D !!(READ_ONCE(sqe->ioprio) & IORING_ACCEPT_MULTISHOT); =20 accept->file_slot =3D READ_ONCE(sqe->file_index); - if (accept->file_slot && (accept->flags & SOCK_CLOEXEC)) + if (accept->file_slot && ((accept->flags & SOCK_CLOEXEC) || multishot)) return -EINVAL; if (accept->flags & ~(SOCK_CLOEXEC | SOCK_NONBLOCK)) return -EINVAL; if (SOCK_NONBLOCK !=3D O_NONBLOCK && (accept->flags & SOCK_NONBLOCK)) accept->flags =3D (accept->flags & ~SOCK_NONBLOCK) | O_NONBLOCK; + if (multishot) + req->flags |=3D REQ_F_APOLL_MULTISHOT; + return 0; } =20 @@ -5730,6 +5736,7 @@ static inline void io_poll_clean(struct io_kiocb *req) =20 static int io_accept(struct io_kiocb *req, unsigned int issue_flags) { + struct io_ring_ctx *ctx =3D req->ctx; struct io_accept *accept =3D &req->accept; bool force_nonblock =3D issue_flags & IO_URING_F_NONBLOCK; unsigned int file_flags =3D force_nonblock ? O_NONBLOCK : 0; @@ -5737,10 +5744,13 @@ static int io_accept(struct io_kiocb *req, unsigned= int issue_flags) struct file *file; int ret, fd; =20 +retry: if (!fixed) { fd =3D __get_unused_fd_flags(accept->flags, accept->nofile); - if (unlikely(fd < 0)) + if (unlikely(fd < 0)) { + io_poll_clean(req); return fd; + } } file =3D do_accept(req->file, file_flags, accept->addr, accept->addr_len, accept->flags); @@ -5748,8 +5758,12 @@ static int io_accept(struct io_kiocb *req, unsigned = int issue_flags) if (!fixed) put_unused_fd(fd); ret =3D PTR_ERR(file); - if (ret =3D=3D -EAGAIN && force_nonblock) - return -EAGAIN; + if (ret =3D=3D -EAGAIN && force_nonblock) { + if ((req->flags & REQ_F_APOLL_MULTI_POLLED) =3D=3D + REQ_F_APOLL_MULTI_POLLED) + ret =3D 0; + return ret; + } if (ret =3D=3D -ERESTARTSYS) ret =3D -EINTR; req_set_fail(req); @@ -5760,7 +5774,35 @@ static int io_accept(struct io_kiocb *req, unsigned = int issue_flags) ret =3D io_install_fixed_file(req, file, issue_flags, accept->file_slot - 1); } - __io_req_complete(req, issue_flags, ret, 0); + + if (req->flags & REQ_F_APOLL_MULTISHOT) { + if (ret >=3D 0) { + bool filled; + + spin_lock(&ctx->completion_lock); + filled =3D io_fill_cqe_aux(ctx, req->cqe.user_data, ret, + IORING_CQE_F_MORE); + io_commit_cqring(ctx); + spin_unlock(&ctx->completion_lock); + if (unlikely(!filled)) { + io_poll_clean(req); + return -ECANCELED; + } + io_cqring_ev_posted(ctx); + goto retry; + } else { + /* + * the apoll multishot req should handle poll + * cancellation by itself since the upper layer + * who called io_queue_sqe() cannot get errors + * happened here. + */ + io_poll_clean(req); + return ret; + } + } else { + __io_req_complete(req, issue_flags, ret, 0); + } return 0; } =20 --=20 2.36.0