From nobody Mon Sep 29 20:18:42 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 866FFC00140 for ; Mon, 15 Aug 2022 23:11:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245234AbiHOXLv (ORCPT ); Mon, 15 Aug 2022 19:11:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1353048AbiHOXKp (ORCPT ); Mon, 15 Aug 2022 19:10:45 -0400 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC2398768D; Mon, 15 Aug 2022 13:00:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id DCD5BB80EAD; Mon, 15 Aug 2022 19:59:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0711BC433D6; Mon, 15 Aug 2022 19:59:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1660593598; bh=VfIR43MIeNQHB6wAy90NxPLlLzEU7ZpOOBFLTwYX8/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JHyTzYYsSBwgctD7x9OSwko8lKXL1RjAuovQ9ojRIDFB+7AM8ki9p1adNivcVi3pn KBu6B6+N3It6YbnhiZAbk2hACuHrONxYsHDPJ3U7OBSGMELtF6YUC7HEwaqw7TWPvZ QeOYeEI31/3STiIA0ZDJwQ08okp4UAU0BWB7xeA8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Jens Axboe , Sasha Levin Subject: [PATCH 5.19 0294/1157] io_uring: define a prep and issue handler for each opcode Date: Mon, 15 Aug 2022 19:54:10 +0200 Message-Id: <20220815180451.391660601@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220815180439.416659447@linuxfoundation.org> References: <20220815180439.416659447@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Jens Axboe [ Upstream commit 0702e5364f643bc86683d9f585edfe76dbabae39 ] Rather than have two giant switches for doing request preparation and then for doing request issue, add a prep and issue handler for each of them in the io_op_defs[] request definition. Signed-off-by: Jens Axboe Signed-off-by: Sasha Levin --- io_uring/io_uring.c | 838 +++++++++++++++++++------------------------- 1 file changed, 365 insertions(+), 473 deletions(-) diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index b63956975109..f429b68d1fc2 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -1110,231 +1110,13 @@ struct io_op_def { unsigned iopoll : 1; /* size of async data needed, if any */ unsigned short async_size; -}; =20 -static const struct io_op_def io_op_defs[] =3D { - [IORING_OP_NOP] =3D { - .audit_skip =3D 1, - .iopoll =3D 1, - }, - [IORING_OP_READV] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .buffer_select =3D 1, - .needs_async_setup =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_WRITEV] =3D { - .needs_file =3D 1, - .hash_reg_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .needs_async_setup =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_FSYNC] =3D { - .needs_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_READ_FIXED] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_WRITE_FIXED] =3D { - .needs_file =3D 1, - .hash_reg_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_POLL_ADD] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_POLL_REMOVE] =3D { - .audit_skip =3D 1, - }, - [IORING_OP_SYNC_FILE_RANGE] =3D { - .needs_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_SENDMSG] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .needs_async_setup =3D 1, - .ioprio =3D 1, - .async_size =3D sizeof(struct io_async_msghdr), - }, - [IORING_OP_RECVMSG] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .buffer_select =3D 1, - .needs_async_setup =3D 1, - .ioprio =3D 1, - .async_size =3D sizeof(struct io_async_msghdr), - }, - [IORING_OP_TIMEOUT] =3D { - .audit_skip =3D 1, - .async_size =3D sizeof(struct io_timeout_data), - }, - [IORING_OP_TIMEOUT_REMOVE] =3D { - /* used by timeout updates' prep() */ - .audit_skip =3D 1, - }, - [IORING_OP_ACCEPT] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .poll_exclusive =3D 1, - .ioprio =3D 1, /* used for flags */ - }, - [IORING_OP_ASYNC_CANCEL] =3D { - .audit_skip =3D 1, - }, - [IORING_OP_LINK_TIMEOUT] =3D { - .audit_skip =3D 1, - .async_size =3D sizeof(struct io_timeout_data), - }, - [IORING_OP_CONNECT] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .needs_async_setup =3D 1, - .async_size =3D sizeof(struct io_async_connect), - }, - [IORING_OP_FALLOCATE] =3D { - .needs_file =3D 1, - }, - [IORING_OP_OPENAT] =3D {}, - [IORING_OP_CLOSE] =3D {}, - [IORING_OP_FILES_UPDATE] =3D { - .audit_skip =3D 1, - .iopoll =3D 1, - }, - [IORING_OP_STATX] =3D { - .audit_skip =3D 1, - }, - [IORING_OP_READ] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .buffer_select =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_WRITE] =3D { - .needs_file =3D 1, - .hash_reg_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .plug =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - .iopoll =3D 1, - .async_size =3D sizeof(struct io_async_rw), - }, - [IORING_OP_FADVISE] =3D { - .needs_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_MADVISE] =3D {}, - [IORING_OP_SEND] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollout =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - }, - [IORING_OP_RECV] =3D { - .needs_file =3D 1, - .unbound_nonreg_file =3D 1, - .pollin =3D 1, - .buffer_select =3D 1, - .audit_skip =3D 1, - .ioprio =3D 1, - }, - [IORING_OP_OPENAT2] =3D { - }, - [IORING_OP_EPOLL_CTL] =3D { - .unbound_nonreg_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_SPLICE] =3D { - .needs_file =3D 1, - .hash_reg_file =3D 1, - .unbound_nonreg_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_PROVIDE_BUFFERS] =3D { - .audit_skip =3D 1, - .iopoll =3D 1, - }, - [IORING_OP_REMOVE_BUFFERS] =3D { - .audit_skip =3D 1, - .iopoll =3D 1, - }, - [IORING_OP_TEE] =3D { - .needs_file =3D 1, - .hash_reg_file =3D 1, - .unbound_nonreg_file =3D 1, - .audit_skip =3D 1, - }, - [IORING_OP_SHUTDOWN] =3D { - .needs_file =3D 1, - }, - [IORING_OP_RENAMEAT] =3D {}, - [IORING_OP_UNLINKAT] =3D {}, - [IORING_OP_MKDIRAT] =3D {}, - [IORING_OP_SYMLINKAT] =3D {}, - [IORING_OP_LINKAT] =3D {}, - [IORING_OP_MSG_RING] =3D { - .needs_file =3D 1, - .iopoll =3D 1, - }, - [IORING_OP_FSETXATTR] =3D { - .needs_file =3D 1 - }, - [IORING_OP_SETXATTR] =3D {}, - [IORING_OP_FGETXATTR] =3D { - .needs_file =3D 1 - }, - [IORING_OP_GETXATTR] =3D {}, - [IORING_OP_SOCKET] =3D { - .audit_skip =3D 1, - }, - [IORING_OP_URING_CMD] =3D { - .needs_file =3D 1, - .plug =3D 1, - .needs_async_setup =3D 1, - .async_size =3D uring_cmd_pdu_size(1), - }, + int (*prep)(struct io_kiocb *, const struct io_uring_sqe *); + int (*issue)(struct io_kiocb *, unsigned int); }; =20 +static const struct io_op_def io_op_defs[]; + /* requests with any of those set should undergo io_disarm_next() */ #define IO_DISARM_MASK (REQ_F_ARM_LTIMEOUT | REQ_F_LINK_TIMEOUT | REQ_F_FA= IL) #define IO_REQ_LINK_FLAGS (REQ_F_LINK | REQ_F_HARDLINK) @@ -8039,96 +7821,33 @@ static int io_files_update(struct io_kiocb *req, un= signed int issue_flags) return 0; } =20 -static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sq= e) +static int io_req_prep_async(struct io_kiocb *req) { + const struct io_op_def *def =3D &io_op_defs[req->opcode]; + + /* assign early for deferred execution for non-fixed file */ + if (def->needs_file && !(req->flags & REQ_F_FIXED_FILE)) + req->file =3D io_file_get_normal(req, req->cqe.fd); + if (!def->needs_async_setup) + return 0; + if (WARN_ON_ONCE(req_has_async_data(req))) + return -EFAULT; + if (io_alloc_async_data(req)) + return -EAGAIN; + switch (req->opcode) { - case IORING_OP_NOP: - return io_nop_prep(req, sqe); case IORING_OP_READV: - case IORING_OP_READ_FIXED: - case IORING_OP_READ: + return io_readv_prep_async(req); case IORING_OP_WRITEV: - case IORING_OP_WRITE_FIXED: - case IORING_OP_WRITE: - return io_prep_rw(req, sqe); - case IORING_OP_POLL_ADD: - return io_poll_add_prep(req, sqe); - case IORING_OP_POLL_REMOVE: - return io_poll_remove_prep(req, sqe); - case IORING_OP_FSYNC: - return io_fsync_prep(req, sqe); - case IORING_OP_SYNC_FILE_RANGE: - return io_sfr_prep(req, sqe); + return io_writev_prep_async(req); case IORING_OP_SENDMSG: - case IORING_OP_SEND: - return io_sendmsg_prep(req, sqe); + return io_sendmsg_prep_async(req); case IORING_OP_RECVMSG: - case IORING_OP_RECV: - return io_recvmsg_prep(req, sqe); + return io_recvmsg_prep_async(req); case IORING_OP_CONNECT: - return io_connect_prep(req, sqe); - case IORING_OP_TIMEOUT: - return io_timeout_prep(req, sqe); - case IORING_OP_TIMEOUT_REMOVE: - return io_timeout_remove_prep(req, sqe); - case IORING_OP_ASYNC_CANCEL: - return io_async_cancel_prep(req, sqe); - case IORING_OP_LINK_TIMEOUT: - return io_link_timeout_prep(req, sqe); - case IORING_OP_ACCEPT: - return io_accept_prep(req, sqe); - case IORING_OP_FALLOCATE: - return io_fallocate_prep(req, sqe); - case IORING_OP_OPENAT: - return io_openat_prep(req, sqe); - case IORING_OP_CLOSE: - return io_close_prep(req, sqe); - case IORING_OP_FILES_UPDATE: - return io_files_update_prep(req, sqe); - case IORING_OP_STATX: - return io_statx_prep(req, sqe); - case IORING_OP_FADVISE: - return io_fadvise_prep(req, sqe); - case IORING_OP_MADVISE: - return io_madvise_prep(req, sqe); - case IORING_OP_OPENAT2: - return io_openat2_prep(req, sqe); - case IORING_OP_EPOLL_CTL: - return io_epoll_ctl_prep(req, sqe); - case IORING_OP_SPLICE: - return io_splice_prep(req, sqe); - case IORING_OP_PROVIDE_BUFFERS: - return io_provide_buffers_prep(req, sqe); - case IORING_OP_REMOVE_BUFFERS: - return io_remove_buffers_prep(req, sqe); - case IORING_OP_TEE: - return io_tee_prep(req, sqe); - case IORING_OP_SHUTDOWN: - return io_shutdown_prep(req, sqe); - case IORING_OP_RENAMEAT: - return io_renameat_prep(req, sqe); - case IORING_OP_UNLINKAT: - return io_unlinkat_prep(req, sqe); - case IORING_OP_MKDIRAT: - return io_mkdirat_prep(req, sqe); - case IORING_OP_SYMLINKAT: - return io_symlinkat_prep(req, sqe); - case IORING_OP_LINKAT: - return io_linkat_prep(req, sqe); - case IORING_OP_MSG_RING: - return io_msg_ring_prep(req, sqe); - case IORING_OP_FSETXATTR: - return io_fsetxattr_prep(req, sqe); - case IORING_OP_SETXATTR: - return io_setxattr_prep(req, sqe); - case IORING_OP_FGETXATTR: - return io_fgetxattr_prep(req, sqe); - case IORING_OP_GETXATTR: - return io_getxattr_prep(req, sqe); - case IORING_OP_SOCKET: - return io_socket_prep(req, sqe); + return io_connect_prep_async(req); case IORING_OP_URING_CMD: - return io_uring_cmd_prep(req, sqe); + return io_uring_cmd_prep_async(req); } =20 printk_once(KERN_WARNING "io_uring: unhandled opcode %d\n", @@ -8136,39 +7855,6 @@ static int io_req_prep(struct io_kiocb *req, const s= truct io_uring_sqe *sqe) return -EINVAL; } =20 -static int io_req_prep_async(struct io_kiocb *req) -{ - const struct io_op_def *def =3D &io_op_defs[req->opcode]; - - /* assign early for deferred execution for non-fixed file */ - if (def->needs_file && !(req->flags & REQ_F_FIXED_FILE)) - req->file =3D io_file_get_normal(req, req->cqe.fd); - if (!def->needs_async_setup) - return 0; - if (WARN_ON_ONCE(req_has_async_data(req))) - return -EFAULT; - if (io_alloc_async_data(req)) - return -EAGAIN; - - switch (req->opcode) { - case IORING_OP_READV: - return io_readv_prep_async(req); - case IORING_OP_WRITEV: - return io_writev_prep_async(req); - case IORING_OP_SENDMSG: - return io_sendmsg_prep_async(req); - case IORING_OP_RECVMSG: - return io_recvmsg_prep_async(req); - case IORING_OP_CONNECT: - return io_connect_prep_async(req); - case IORING_OP_URING_CMD: - return io_uring_cmd_prep_async(req); - } - printk_once(KERN_WARNING "io_uring: prep_async() bad opcode %d\n", - req->opcode); - return -EFAULT; -} - static u32 io_get_sequence(struct io_kiocb *req) { u32 seq =3D req->ctx->cached_sq_head; @@ -8335,141 +8021,7 @@ static int io_issue_sqe(struct io_kiocb *req, unsig= ned int issue_flags) if (!def->audit_skip) audit_uring_entry(req->opcode); =20 - switch (req->opcode) { - case IORING_OP_NOP: - ret =3D io_nop(req, issue_flags); - break; - case IORING_OP_READV: - case IORING_OP_READ_FIXED: - case IORING_OP_READ: - ret =3D io_read(req, issue_flags); - break; - case IORING_OP_WRITEV: - case IORING_OP_WRITE_FIXED: - case IORING_OP_WRITE: - ret =3D io_write(req, issue_flags); - break; - case IORING_OP_FSYNC: - ret =3D io_fsync(req, issue_flags); - break; - case IORING_OP_POLL_ADD: - ret =3D io_poll_add(req, issue_flags); - break; - case IORING_OP_POLL_REMOVE: - ret =3D io_poll_remove(req, issue_flags); - break; - case IORING_OP_SYNC_FILE_RANGE: - ret =3D io_sync_file_range(req, issue_flags); - break; - case IORING_OP_SENDMSG: - ret =3D io_sendmsg(req, issue_flags); - break; - case IORING_OP_SEND: - ret =3D io_send(req, issue_flags); - break; - case IORING_OP_RECVMSG: - ret =3D io_recvmsg(req, issue_flags); - break; - case IORING_OP_RECV: - ret =3D io_recv(req, issue_flags); - break; - case IORING_OP_TIMEOUT: - ret =3D io_timeout(req, issue_flags); - break; - case IORING_OP_TIMEOUT_REMOVE: - ret =3D io_timeout_remove(req, issue_flags); - break; - case IORING_OP_ACCEPT: - ret =3D io_accept(req, issue_flags); - break; - case IORING_OP_CONNECT: - ret =3D io_connect(req, issue_flags); - break; - case IORING_OP_ASYNC_CANCEL: - ret =3D io_async_cancel(req, issue_flags); - break; - case IORING_OP_FALLOCATE: - ret =3D io_fallocate(req, issue_flags); - break; - case IORING_OP_OPENAT: - ret =3D io_openat(req, issue_flags); - break; - case IORING_OP_CLOSE: - ret =3D io_close(req, issue_flags); - break; - case IORING_OP_FILES_UPDATE: - ret =3D io_files_update(req, issue_flags); - break; - case IORING_OP_STATX: - ret =3D io_statx(req, issue_flags); - break; - case IORING_OP_FADVISE: - ret =3D io_fadvise(req, issue_flags); - break; - case IORING_OP_MADVISE: - ret =3D io_madvise(req, issue_flags); - break; - case IORING_OP_OPENAT2: - ret =3D io_openat2(req, issue_flags); - break; - case IORING_OP_EPOLL_CTL: - ret =3D io_epoll_ctl(req, issue_flags); - break; - case IORING_OP_SPLICE: - ret =3D io_splice(req, issue_flags); - break; - case IORING_OP_PROVIDE_BUFFERS: - ret =3D io_provide_buffers(req, issue_flags); - break; - case IORING_OP_REMOVE_BUFFERS: - ret =3D io_remove_buffers(req, issue_flags); - break; - case IORING_OP_TEE: - ret =3D io_tee(req, issue_flags); - break; - case IORING_OP_SHUTDOWN: - ret =3D io_shutdown(req, issue_flags); - break; - case IORING_OP_RENAMEAT: - ret =3D io_renameat(req, issue_flags); - break; - case IORING_OP_UNLINKAT: - ret =3D io_unlinkat(req, issue_flags); - break; - case IORING_OP_MKDIRAT: - ret =3D io_mkdirat(req, issue_flags); - break; - case IORING_OP_SYMLINKAT: - ret =3D io_symlinkat(req, issue_flags); - break; - case IORING_OP_LINKAT: - ret =3D io_linkat(req, issue_flags); - break; - case IORING_OP_MSG_RING: - ret =3D io_msg_ring(req, issue_flags); - break; - case IORING_OP_FSETXATTR: - ret =3D io_fsetxattr(req, issue_flags); - break; - case IORING_OP_SETXATTR: - ret =3D io_setxattr(req, issue_flags); - break; - case IORING_OP_FGETXATTR: - ret =3D io_fgetxattr(req, issue_flags); - break; - case IORING_OP_GETXATTR: - ret =3D io_getxattr(req, issue_flags); - break; - case IORING_OP_SOCKET: - ret =3D io_socket(req, issue_flags); - break; - case IORING_OP_URING_CMD: - ret =3D io_uring_cmd(req, issue_flags); - break; - default: - ret =3D -EINVAL; - break; - } + ret =3D def->issue(req, issue_flags); =20 if (!def->audit_skip) audit_uring_exit(!ret, ret); @@ -8898,7 +8450,7 @@ static int io_init_req(struct io_ring_ctx *ctx, struc= t io_kiocb *req, req->flags |=3D REQ_F_CREDS; } =20 - return io_req_prep(req, sqe); + return def->prep(req, sqe); } =20 static __cold int io_submit_fail_init(const struct io_uring_sqe *sqe, @@ -13200,8 +12752,343 @@ SYSCALL_DEFINE4(io_uring_register, unsigned int, = fd, unsigned int, opcode, return ret; } =20 +static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags) +{ + WARN_ON_ONCE(1); + return -ECANCELED; +} + +static const struct io_op_def io_op_defs[] =3D { + [IORING_OP_NOP] =3D { + .audit_skip =3D 1, + .iopoll =3D 1, + .prep =3D io_nop_prep, + .issue =3D io_nop, + }, + [IORING_OP_READV] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .buffer_select =3D 1, + .needs_async_setup =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_read, + }, + [IORING_OP_WRITEV] =3D { + .needs_file =3D 1, + .hash_reg_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .needs_async_setup =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_write, + }, + [IORING_OP_FSYNC] =3D { + .needs_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_fsync_prep, + .issue =3D io_fsync, + }, + [IORING_OP_READ_FIXED] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_read, + }, + [IORING_OP_WRITE_FIXED] =3D { + .needs_file =3D 1, + .hash_reg_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_write, + }, + [IORING_OP_POLL_ADD] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_poll_add_prep, + .issue =3D io_poll_add, + }, + [IORING_OP_POLL_REMOVE] =3D { + .audit_skip =3D 1, + .prep =3D io_poll_remove_prep, + .issue =3D io_poll_remove, + }, + [IORING_OP_SYNC_FILE_RANGE] =3D { + .needs_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_sfr_prep, + .issue =3D io_sync_file_range, + }, + [IORING_OP_SENDMSG] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .needs_async_setup =3D 1, + .ioprio =3D 1, + .async_size =3D sizeof(struct io_async_msghdr), + .prep =3D io_sendmsg_prep, + .issue =3D io_sendmsg, + }, + [IORING_OP_RECVMSG] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .buffer_select =3D 1, + .needs_async_setup =3D 1, + .ioprio =3D 1, + .async_size =3D sizeof(struct io_async_msghdr), + .prep =3D io_recvmsg_prep, + .issue =3D io_recvmsg, + }, + [IORING_OP_TIMEOUT] =3D { + .audit_skip =3D 1, + .async_size =3D sizeof(struct io_timeout_data), + .prep =3D io_timeout_prep, + .issue =3D io_timeout, + }, + [IORING_OP_TIMEOUT_REMOVE] =3D { + /* used by timeout updates' prep() */ + .audit_skip =3D 1, + .prep =3D io_timeout_remove_prep, + .issue =3D io_timeout_remove, + }, + [IORING_OP_ACCEPT] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .poll_exclusive =3D 1, + .ioprio =3D 1, /* used for flags */ + .prep =3D io_accept_prep, + .issue =3D io_accept, + }, + [IORING_OP_ASYNC_CANCEL] =3D { + .audit_skip =3D 1, + .prep =3D io_async_cancel_prep, + .issue =3D io_async_cancel, + }, + [IORING_OP_LINK_TIMEOUT] =3D { + .audit_skip =3D 1, + .async_size =3D sizeof(struct io_timeout_data), + .prep =3D io_link_timeout_prep, + .issue =3D io_no_issue, + }, + [IORING_OP_CONNECT] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .needs_async_setup =3D 1, + .async_size =3D sizeof(struct io_async_connect), + .prep =3D io_connect_prep, + .issue =3D io_connect, + }, + [IORING_OP_FALLOCATE] =3D { + .needs_file =3D 1, + .prep =3D io_fallocate_prep, + .issue =3D io_fallocate, + }, + [IORING_OP_OPENAT] =3D { + .prep =3D io_openat_prep, + .issue =3D io_openat, + }, + [IORING_OP_CLOSE] =3D { + .prep =3D io_close_prep, + .issue =3D io_close, + }, + [IORING_OP_FILES_UPDATE] =3D { + .audit_skip =3D 1, + .iopoll =3D 1, + .prep =3D io_files_update_prep, + .issue =3D io_files_update, + }, + [IORING_OP_STATX] =3D { + .audit_skip =3D 1, + .prep =3D io_statx_prep, + .issue =3D io_statx, + }, + [IORING_OP_READ] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .buffer_select =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_read, + }, + [IORING_OP_WRITE] =3D { + .needs_file =3D 1, + .hash_reg_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .plug =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .iopoll =3D 1, + .async_size =3D sizeof(struct io_async_rw), + .prep =3D io_prep_rw, + .issue =3D io_write, + }, + [IORING_OP_FADVISE] =3D { + .needs_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_fadvise_prep, + .issue =3D io_fadvise, + }, + [IORING_OP_MADVISE] =3D { + .prep =3D io_madvise_prep, + .issue =3D io_madvise, + }, + [IORING_OP_SEND] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollout =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .prep =3D io_sendmsg_prep, + .issue =3D io_send, + }, + [IORING_OP_RECV] =3D { + .needs_file =3D 1, + .unbound_nonreg_file =3D 1, + .pollin =3D 1, + .buffer_select =3D 1, + .audit_skip =3D 1, + .ioprio =3D 1, + .prep =3D io_recvmsg_prep, + .issue =3D io_recv, + }, + [IORING_OP_OPENAT2] =3D { + .prep =3D io_openat2_prep, + .issue =3D io_openat2, + }, + [IORING_OP_EPOLL_CTL] =3D { + .unbound_nonreg_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_epoll_ctl_prep, + .issue =3D io_epoll_ctl, + }, + [IORING_OP_SPLICE] =3D { + .needs_file =3D 1, + .hash_reg_file =3D 1, + .unbound_nonreg_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_splice_prep, + .issue =3D io_splice, + }, + [IORING_OP_PROVIDE_BUFFERS] =3D { + .audit_skip =3D 1, + .iopoll =3D 1, + .prep =3D io_provide_buffers_prep, + .issue =3D io_provide_buffers, + }, + [IORING_OP_REMOVE_BUFFERS] =3D { + .audit_skip =3D 1, + .iopoll =3D 1, + .prep =3D io_remove_buffers_prep, + .issue =3D io_remove_buffers, + }, + [IORING_OP_TEE] =3D { + .needs_file =3D 1, + .hash_reg_file =3D 1, + .unbound_nonreg_file =3D 1, + .audit_skip =3D 1, + .prep =3D io_tee_prep, + .issue =3D io_tee, + }, + [IORING_OP_SHUTDOWN] =3D { + .needs_file =3D 1, + .prep =3D io_shutdown_prep, + .issue =3D io_shutdown, + }, + [IORING_OP_RENAMEAT] =3D { + .prep =3D io_renameat_prep, + .issue =3D io_renameat, + }, + [IORING_OP_UNLINKAT] =3D { + .prep =3D io_unlinkat_prep, + .issue =3D io_unlinkat, + }, + [IORING_OP_MKDIRAT] =3D { + .prep =3D io_mkdirat_prep, + .issue =3D io_mkdirat, + }, + [IORING_OP_SYMLINKAT] =3D { + .prep =3D io_symlinkat_prep, + .issue =3D io_symlinkat, + }, + [IORING_OP_LINKAT] =3D { + .prep =3D io_linkat_prep, + .issue =3D io_linkat, + }, + [IORING_OP_MSG_RING] =3D { + .needs_file =3D 1, + .iopoll =3D 1, + .prep =3D io_msg_ring_prep, + .issue =3D io_msg_ring, + }, + [IORING_OP_FSETXATTR] =3D { + .needs_file =3D 1, + .prep =3D io_fsetxattr_prep, + .issue =3D io_fsetxattr, + }, + [IORING_OP_SETXATTR] =3D { + .prep =3D io_setxattr_prep, + .issue =3D io_setxattr, + }, + [IORING_OP_FGETXATTR] =3D { + .needs_file =3D 1, + .prep =3D io_fgetxattr_prep, + .issue =3D io_fgetxattr, + }, + [IORING_OP_GETXATTR] =3D { + .prep =3D io_getxattr_prep, + .issue =3D io_getxattr, + }, + [IORING_OP_SOCKET] =3D { + .audit_skip =3D 1, + .prep =3D io_socket_prep, + .issue =3D io_socket, + }, + [IORING_OP_URING_CMD] =3D { + .needs_file =3D 1, + .plug =3D 1, + .needs_async_setup =3D 1, + .async_size =3D uring_cmd_pdu_size(1), + .prep =3D io_uring_cmd_prep, + .issue =3D io_uring_cmd, + }, +}; + static int __init io_uring_init(void) { + int i; + #define __BUILD_BUG_VERIFY_ELEMENT(stype, eoffset, etype, ename) do { \ BUILD_BUG_ON(offsetof(stype, ename) !=3D eoffset); \ BUILD_BUG_ON(sizeof(etype) !=3D sizeof_field(stype, ename)); \ @@ -13266,6 +13153,11 @@ static int __init io_uring_init(void) =20 BUILD_BUG_ON(sizeof(struct io_uring_cmd) > 64); =20 + for (i =3D 0; i < ARRAY_SIZE(io_op_defs); i++) { + BUG_ON(!io_op_defs[i].prep); + BUG_ON(!io_op_defs[i].issue); + } + req_cachep =3D KMEM_CACHE(io_kiocb, SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT); return 0; --=20 2.35.1