From nobody Tue Feb 10 00:23:56 2026 Received: from mail-qt1-f225.google.com (mail-qt1-f225.google.com [209.85.160.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2454A198E8C for ; Wed, 9 Apr 2025 01:42:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.225 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744162957; cv=none; b=llU1fnQ1b3ShhGaEi5U/PsD9iUZQEWyHS1Rk45f2PEwgIq+T8zEhsLMXZrj5wiV9VQ9kYflLHvBx7KABgsm1s7Pkg9IUzM4brg+gx40atBLAdXOsii0aG8ZW0yoIMJXXHmaoRZnE8VmkzKeNxtilcwkjSMS6lw6QHjPGhuJYFo8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744162957; c=relaxed/simple; bh=4Ldp2l6rBuf/UdGTjzYPoDsd0i7OfOyCP25NGXo7oXo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=q2bYi7xysx4snGMmHeuHVUK8wERU+Cf9vHHrHmDCTgl9X0PButAAINn3Beh0hLiTe0sCHAmrQoIbHKgrPlg4rDowQcNjjsbNGrZMbMOZO9oRl9JBBYXf0D9VpW281V0Ksp8nCjLY97jodojuKWZc3eBZvmhLjiOzP8c53kOVDAI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=Nz2VcDE/; arc=none smtp.client-ip=209.85.160.225 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="Nz2VcDE/" Received: by mail-qt1-f225.google.com with SMTP id d75a77b69052e-4766cb762b6so2893641cf.0 for ; Tue, 08 Apr 2025 18:42:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1744162950; x=1744767750; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=MgpzTCtGeldmEWjmJqn7b6H4/Wj9w7RY3hdIrUJLwxQ=; b=Nz2VcDE/k5BPp3IiTic8YcJjiKPKXvXaiXE1qOz+GkMLh0UW5Gh7juCxTXDEQCGkWv 17YXVFLhRBLEr1P2LzNpWlzGPksBg+uh1lXKzAECgj/SqTB87zJFGsaJBvkijNgcP5mb SENdXhjLniMmUrc34LFIkJTgQirKXSh8CY5RLZplGz72P62bdtScldc7KU+t+Dkj1ySz m7TW058DwgBLhCs7Kxt5WT6OSCpM0HBEmMX6pK6YaFvCkDFfCbrntKsmh5dDpwp4pt7d /sum0jjSUErizHKuzGcQZBGVBu8Oah3ls7wluAyRccuFozscpX33JooMrs6hOpm9Ch1T x1BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744162950; x=1744767750; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MgpzTCtGeldmEWjmJqn7b6H4/Wj9w7RY3hdIrUJLwxQ=; b=KPtvp1O4WC8Huy/eXvWjU9up/XnSP6v4iu8P5V72ZMK79Xhj3/u2rWo6Uz+buI5YP7 EhJIjAOsqm6g3lO6ZTPIEPaIht+0VZqpHUYiNqg88u6uyCi1q1UnPfYpvACnyTUhqJcc jL2eZiuUYzaaWoElM4ae+R2Bmyb9plkY0nOXB8jMg0fwumhg6KMgB0jbHBgi821e3lKh rSf/2YQmoNwmx1uD0RW9fnHcfwjqehiKsvDsB4vQ6l6TkSXmR2QQIScDjfO2fAh3h9l3 G30OA1rwqrq6HcDb1iYgYM8mH7kRjcL3RIM6OZS5UywbjqA2BlBMlTDmag15nMaJwyv0 o4Dg== X-Forwarded-Encrypted: i=1; AJvYcCXyR4d3l9sZ1nbsUFOcoU5VGizi6prZaR3pyJhPXxI2wfA2MXDjBTh2aEziZlti+dsDHqDRi4RcJqMNhqs=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3pw5BmQw5V1HhFt3e2/UB+IHPOEy0a9JrKNBowRMYMTgQ0k5a qEjgGh4wrtbV1vOmUl2q/aYp1SdIVeyxgpL7ULUACty4Rx08hmS7gK/5QhGGBmRo9+ZwNmBqgQf E+g6oVdVDFxiNdZ5QWL2ll29B4FQ/sxNY X-Gm-Gg: ASbGncsv5Cwo2dHMo0t+m606lHLkoYHEoBGTfFM7sDILeG4R4MDoT8XPY1KybCOnrKe YM8LgqkBalfOAy/poZtcVkAa8JxocqsQk5TkP4hrQRNJ4elsxiFs1PFiVTJcY+3qjhfzvMldPdv wujifopzNGH4RFN8wH6zr4Gvr9OgQlBN9o+aZaR40EoTR/f7ySYlm/2pr1RkBUL8NT47qrutPEN YXi1InkCkzWXRO+SEg0+josenm5DkCMpKGYk+z8HGggvdaMu6NKwf8n97bEWk5ftIox2bm0Ol5k GYGKI7SxpXH8GmNOPZcxCOBogOtnMdnGJgoagFx/e/toq1ScFw== X-Google-Smtp-Source: AGHT+IEds+LY1jv8ZLyqlzOP6+065cbkz0eVWjeidKJKpoEWnDbEXV1YwhzCbfby9ff4B87McyMjD2mvmVqI X-Received: by 2002:a05:622a:351:b0:477:51c:d853 with SMTP id d75a77b69052e-4795f08e207mr14639391cf.9.1744162950320; Tue, 08 Apr 2025 18:42:30 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([208.88.159.129]) by smtp-relay.gmail.com with ESMTPS id d75a77b69052e-47964d6fba6sm58471cf.3.2025.04.08.18.42.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Apr 2025 18:42:30 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-ushankar.dev.purestorage.com (dev-ushankar.dev.purestorage.com [10.7.70.36]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 51E6334045B; Tue, 8 Apr 2025 19:42:29 -0600 (MDT) Received: by dev-ushankar.dev.purestorage.com (Postfix, from userid 1557716368) id 442E7E40EBE; Tue, 8 Apr 2025 19:42:29 -0600 (MDT) From: Uday Shankar Date: Tue, 08 Apr 2025 19:42:07 -0600 Subject: [PATCH v2 1/2] ublk: properly serialize all FETCH_REQs Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250408-ublk_task_per_io-v2-1-b97877e6fd50@purestorage.com> References: <20250408-ublk_task_per_io-v2-0-b97877e6fd50@purestorage.com> In-Reply-To: <20250408-ublk_task_per_io-v2-0-b97877e6fd50@purestorage.com> To: Ming Lei , Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Uday Shankar , Caleb Sander Mateos X-Mailer: b4 0.14.2 Most uring_cmds issued against ublk character devices are serialized because each command affects only one queue, and there is an early check which only allows a single task (the queue's ubq_daemon) to issue uring_cmds against that queue. However, this mechanism does not work for FETCH_REQs, since they are expected before ubq_daemon is set. Since FETCH_REQs are only used at initialization and not in the fast path, serialize them using the per-ublk-device mutex. This fixes a number of data races that were previously possible if a badly behaved ublk server decided to issue multiple FETCH_REQs against the same qid/tag concurrently. Reported-by: Caleb Sander Mateos Signed-off-by: Uday Shankar --- drivers/block/ublk_drv.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 2fd05c1bd30b03343cb6f357f8c08dd92ff47af9..5535073ccd23dfbbd25830c1722= c360146b95662 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -1809,8 +1809,8 @@ static void ublk_nosrv_work(struct work_struct *work) =20 /* device can only be started after all IOs are ready */ static void ublk_mark_io_ready(struct ublk_device *ub, struct ublk_queue *= ubq) + __must_hold(&ub->mutex) { - mutex_lock(&ub->mutex); ubq->nr_io_ready++; if (ublk_queue_ready(ubq)) { ubq->ubq_daemon =3D current; @@ -1822,7 +1822,6 @@ static void ublk_mark_io_ready(struct ublk_device *ub= , struct ublk_queue *ubq) } if (ub->nr_queues_ready =3D=3D ub->dev_info.nr_hw_queues) complete_all(&ub->completion); - mutex_unlock(&ub->mutex); } =20 static void ublk_handle_need_get_data(struct ublk_device *ub, int q_id, @@ -1962,17 +1961,18 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd = *cmd, case UBLK_IO_UNREGISTER_IO_BUF: return ublk_unregister_io_buf(cmd, ub_cmd->addr, issue_flags); case UBLK_IO_FETCH_REQ: + mutex_lock(&ub->mutex); /* UBLK_IO_FETCH_REQ is only allowed before queue is setup */ if (ublk_queue_ready(ubq)) { ret =3D -EBUSY; - goto out; + goto out_unlock; } /* * The io is being handled by server, so COMMIT_RQ is expected * instead of FETCH_REQ */ if (io->flags & UBLK_IO_FLAG_OWNED_BY_SRV) - goto out; + goto out_unlock; =20 if (ublk_need_map_io(ubq)) { /* @@ -1980,15 +1980,16 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd = *cmd, * DATA is not enabled */ if (!ub_cmd->addr && !ublk_need_get_data(ubq)) - goto out; + goto out_unlock; } else if (ub_cmd->addr) { /* User copy requires addr to be unset */ ret =3D -EINVAL; - goto out; + goto out_unlock; } =20 ublk_fill_io_cmd(io, cmd, ub_cmd->addr); ublk_mark_io_ready(ub, ubq); + mutex_unlock(&ub->mutex); break; case UBLK_IO_COMMIT_AND_FETCH_REQ: req =3D blk_mq_tag_to_rq(ub->tag_set.tags[ub_cmd->q_id], tag); @@ -2028,7 +2029,9 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *c= md, ublk_prep_cancel(cmd, issue_flags, ubq, tag); return -EIOCBQUEUED; =20 - out: +out_unlock: + mutex_unlock(&ub->mutex); +out: pr_devel("%s: complete: cmd op %d, tag %d ret %x io_flags %x\n", __func__, cmd_op, tag, ret, io->flags); return ret; --=20 2.34.1 From nobody Tue Feb 10 00:23:56 2026 Received: from mail-il1-f227.google.com (mail-il1-f227.google.com [209.85.166.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2808319938D for ; Wed, 9 Apr 2025 01:42:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.227 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744162955; cv=none; b=BXEkXmHGE0cpkSHgOo9gqZqd9yzFVa2ZXShOmpqsK3tFKUy96nOJYhoRTWsTnOTYJNskWTYdV4WWMjYBZsgdVTiRR2/mA+WghHjgEBj+eiba/sUTpjyuyo5As6NQ7l/SArswUbrjZ36HS6C+UHDZKuQPoHcxlUL4hc/BjSdFL70= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744162955; c=relaxed/simple; bh=GTrKKsElufLUl6LIESiQiKtGw4NJXYmSyb2jdCZTTTo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ISOBNlBBNg8w8Z0WSXb7VwByNShr5a7chSn1MW9jNUT0wbDHJw3Qa/ZiZI0UrpWrUnqwfWTuP9NMgNyiIs1KH7EbTJJIzkBfN6TyiJgr54NiM8E0XZY1OdZpIlSAfvvMqfZQHAPruDKGDJnZ1BF+8B5MA0NQ2RajmIAj1BQOO5U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com; spf=fail smtp.mailfrom=purestorage.com; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b=bRqVxVhM; arc=none smtp.client-ip=209.85.166.227 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=purestorage.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=purestorage.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=purestorage.com header.i=@purestorage.com header.b="bRqVxVhM" Received: by mail-il1-f227.google.com with SMTP id e9e14a558f8ab-3d450154245so51945595ab.2 for ; Tue, 08 Apr 2025 18:42:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google2022; t=1744162951; x=1744767751; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=3TFDQmxEY/oTpesIZpeWcZDgn8XPPYKLm/+q8z+3Hdw=; b=bRqVxVhMnj0e2/eEGpjMu3u5gnzEegk+5EEcFESE+Ktmj0n4RjcJct/3AyK4VlSfUx 36LUIe2XA4aKICUjRaGgzvIEOnhLByw2ZtlrVOv6lUntk7KXaawnXygFLyZmVQ3KZUdI RXNuiw5wIVf7lVRVFZyBfUcG1VsFgbV8Zfyz3m5K32i/eCkuf3lZABAtdADWJSElmweD o0P96EelGgQ2zQ8K12KbAgAmOeVyx7uhrDEEp9yr+RZWXdrrtdn5j0f2prB39BB0F1qZ QycjHeEkQPKcy4PNnAF9bUYOGIEIbHDwS8lVTGN0qTou0kvyPbM5EAbu/CUf1ManeHgW ENUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744162951; x=1744767751; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3TFDQmxEY/oTpesIZpeWcZDgn8XPPYKLm/+q8z+3Hdw=; b=RVRv6qtpc/qenX9etVrbhHNrIoMrL+E+QybKUp/Oyz0TQJNQjpVRWk/qndj67tBLuM c648ezfnO5QOG/KMm/Q64PQ0C/khhksp39zwlmW0LYmLcbwBLA9DyRnrOs6vjiRVXbfe KKK8sZDnIZFfvrce9FaMe06ogdhnRglxg5d6BIchB1+itA4YPLHtaO92NRQO66RFwnpS 21ZHGgOTJOA5Q5Zrn5yeaQNl979D8WfiFofy46prde5+NQYUAl2JaG5CjoVT+vLTN67v KLdjFtlznTsCiLEuZR180vYCe/h/qo1M8hC2zOLEIlpBpgII3QN04qx93dRVz/Gs0N+4 mT3A== X-Forwarded-Encrypted: i=1; AJvYcCXOa+H2RaMR4l1M37BHtXbGOQHsNr6Ov2M19wOS6PJMSO4pHipWWcXXxsTNtIxhM1f0l+GQ/qRsbiVI4j8=@vger.kernel.org X-Gm-Message-State: AOJu0YyRxIY0m/r4dGdEPef8h4AAQG1RPupjS6eVemuJcKhCgmAbCTy1 ocvi6LxFcZyJTggFHaX743hb7ZBRLXPvH3Wy6jEn1ixozPXD2TfsOPQn0AQe1WFiNw7PTGOW54j yGN/yQKWaVV6BSpZLdZcUhY5vD1YedcwP3kwsIV8N5RENLAf1 X-Gm-Gg: ASbGncuinjOMbzDntzYJbfhjfgoCOcA4VOtxPGujQI6qw5q03jv/JzEMtrXEtexLAba JkxBFD+wEw7fQNzvVuPA94xLlEoGSoLaptdEqsg3V97UyXv6EVObIraen3OpeLTeI/GRSX/GUPu +YMT/PtaR1EgQHEgNXbiKQaBx2O8DiqtOZfONXYgOGAT1IZWKowQhWL6Dp1EITUMYpzlAhAmR/p 0rEl5+cRKM/EGg8EcIkSERgQvi3IQEfb2qqgwEil26WZIeViGV5JiuhkUSj8KIW8RfJb8iZt4E6 Hucx+GZzd4gStQ4VbrDqtEBxGIYE4IFnv0E= X-Google-Smtp-Source: AGHT+IHPTMkBOR6/4UJLfhoawTK5/aiDlDFszMj/FEtwjJYaSspXxcx8wqq8hH1xe8z31Nb75fDS6moFSsmf X-Received: by 2002:a05:6e02:2165:b0:3d0:4e0c:2c96 with SMTP id e9e14a558f8ab-3d776c93eebmr16101355ab.2.1744162951232; Tue, 08 Apr 2025 18:42:31 -0700 (PDT) Received: from c7-smtp-2023.dev.purestorage.com ([2620:125:9017:12:36:3:5:0]) by smtp-relay.gmail.com with ESMTPS id 8926c6da1cb9f-4f505cfbc5bsm6609173.13.2025.04.08.18.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Apr 2025 18:42:31 -0700 (PDT) X-Relaying-Domain: purestorage.com Received: from dev-ushankar.dev.purestorage.com (dev-ushankar.dev.purestorage.com [10.7.70.36]) by c7-smtp-2023.dev.purestorage.com (Postfix) with ESMTP id 58ED334059A; Tue, 8 Apr 2025 19:42:30 -0600 (MDT) Received: by dev-ushankar.dev.purestorage.com (Postfix, from userid 1557716368) id 4A394E40F92; Tue, 8 Apr 2025 19:42:29 -0600 (MDT) From: Uday Shankar Date: Tue, 08 Apr 2025 19:42:08 -0600 Subject: [PATCH v2 2/2] ublk: require unique task per io instead of unique task per hctx Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250408-ublk_task_per_io-v2-2-b97877e6fd50@purestorage.com> References: <20250408-ublk_task_per_io-v2-0-b97877e6fd50@purestorage.com> In-Reply-To: <20250408-ublk_task_per_io-v2-0-b97877e6fd50@purestorage.com> To: Ming Lei , Jens Axboe Cc: linux-block@vger.kernel.org, linux-kernel@vger.kernel.org, Uday Shankar X-Mailer: b4 0.14.2 Currently, ublk_drv associates to each hardware queue (hctx) a unique task (called the queue's ubq_daemon) which is allowed to issue COMMIT_AND_FETCH commands against the hctx. If any other task attempts to do so, the command fails immediately with EINVAL. When considered together with the block layer architecture, the result is that for each CPU C on the system, there is a unique ublk server thread which is allowed to handle I/O submitted on CPU C. This can lead to suboptimal performance under imbalanced load generation. For an extreme example, suppose all the load is generated on CPUs mapping to a single ublk server thread. Then that thread may be fully utilized and become the bottleneck in the system, while other ublk server threads are totally idle. This issue can also be addressed directly in the ublk server without kernel support by having threads dequeue I/Os and pass them around to ensure even load. But this solution requires inter-thread communication at least twice for each I/O (submission and completion), which is generally a bad pattern for performance. The problem gets even worse with zero copy, as more inter-thread communication would be required to have the buffer register/unregister calls to come from the correct thread. Therefore, address this issue in ublk_drv by requiring a unique task per I/O instead of per queue/hctx. Imbalanced load can then be balanced across all ublk server threads by having threads issue FETCH_REQs in a round-robin manner. As a small toy example, consider a system with a single ublk device having 2 queues, each of queue depth 4. A ublk server having 4 threads could issue its FETCH_REQs against this device as follows (where each entry is the qid,tag pair that the FETCH_REQ targets): poller thread: T0 T1 T2 T3 0,0 0,1 0,2 0,3 1,3 1,0 1,1 1,2 Since tags appear to be allocated in sequential chunks, this setup provides a rough approximation to distributing I/Os round-robin across all ublk server threads, while letting I/Os stay fully thread-local. Signed-off-by: Uday Shankar --- drivers/block/ublk_drv.c | 75 ++++++++++++++++++++++----------------------= ---- 1 file changed, 34 insertions(+), 41 deletions(-) diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 5535073ccd23dfbbd25830c1722c360146b95662..e51c72ae1f7e2a9832e3b446bd3= 1b0d49aef98b5 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -150,6 +150,7 @@ struct ublk_io { int res; =20 struct io_uring_cmd *cmd; + struct task_struct *task; }; =20 struct ublk_queue { @@ -157,11 +158,9 @@ struct ublk_queue { int q_depth; =20 unsigned long flags; - struct task_struct *ubq_daemon; struct ublksrv_io_desc *io_cmd_buf; =20 bool force_abort; - bool timeout; bool canceling; bool fail_io; /* copy of dev->state =3D=3D UBLK_S_DEV_FAIL_IO */ unsigned short nr_io_ready; /* how many ios setup */ @@ -1072,11 +1071,6 @@ static inline struct ublk_uring_cmd_pdu *ublk_get_ur= ing_cmd_pdu( return io_uring_cmd_to_pdu(ioucmd, struct ublk_uring_cmd_pdu); } =20 -static inline bool ubq_daemon_is_dying(struct ublk_queue *ubq) -{ - return ubq->ubq_daemon->flags & PF_EXITING; -} - /* todo: handle partial completion */ static inline void __ublk_complete_rq(struct request *req) { @@ -1202,13 +1196,13 @@ static void ublk_dispatch_req(struct ublk_queue *ub= q, /* * Task is exiting if either: * - * (1) current !=3D ubq_daemon. + * (1) current !=3D io->task. * io_uring_cmd_complete_in_task() tries to run task_work - * in a workqueue if ubq_daemon(cmd's task) is PF_EXITING. + * in a workqueue if cmd's task is PF_EXITING. * * (2) current->flags & PF_EXITING. */ - if (unlikely(current !=3D ubq->ubq_daemon || current->flags & PF_EXITING)= ) { + if (unlikely(current !=3D io->task || current->flags & PF_EXITING)) { __ublk_abort_rq(ubq, req); return; } @@ -1314,23 +1308,20 @@ static void ublk_queue_cmd_list(struct ublk_queue *= ubq, struct rq_list *l) static enum blk_eh_timer_return ublk_timeout(struct request *rq) { struct ublk_queue *ubq =3D rq->mq_hctx->driver_data; + struct ublk_io *io =3D &ubq->ios[rq->tag]; unsigned int nr_inflight =3D 0; int i; =20 if (ubq->flags & UBLK_F_UNPRIVILEGED_DEV) { - if (!ubq->timeout) { - send_sig(SIGKILL, ubq->ubq_daemon, 0); - ubq->timeout =3D true; - } - + send_sig(SIGKILL, io->task, 0); return BLK_EH_DONE; } =20 - if (!ubq_daemon_is_dying(ubq)) + if (!(io->task->flags & PF_EXITING)) return BLK_EH_RESET_TIMER; =20 for (i =3D 0; i < ubq->q_depth; i++) { - struct ublk_io *io =3D &ubq->ios[i]; + io =3D &ubq->ios[i]; =20 if (!(io->flags & UBLK_IO_FLAG_ACTIVE)) nr_inflight++; @@ -1529,8 +1520,8 @@ static void ublk_commit_completion(struct ublk_device= *ub, } =20 /* - * Called from ubq_daemon context via cancel fn, meantime quiesce ublk - * blk-mq queue, so we are called exclusively with blk-mq and ubq_daemon + * Called from io task context via cancel fn, meantime quiesce ublk + * blk-mq queue, so we are called exclusively with blk-mq and io task * context, so everything is serialized. */ static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ub= q) @@ -1646,13 +1637,13 @@ static void ublk_uring_cmd_cancel_fn(struct io_urin= g_cmd *cmd, return; =20 task =3D io_uring_cmd_get_task(cmd); - if (WARN_ON_ONCE(task && task !=3D ubq->ubq_daemon)) + io =3D &ubq->ios[pdu->tag]; + if (WARN_ON_ONCE(task && task !=3D io->task)) return; =20 ub =3D ubq->dev; need_schedule =3D ublk_abort_requests(ub, ubq); =20 - io =3D &ubq->ios[pdu->tag]; WARN_ON_ONCE(io->cmd !=3D cmd); ublk_cancel_cmd(ubq, io, issue_flags); =20 @@ -1813,8 +1804,6 @@ static void ublk_mark_io_ready(struct ublk_device *ub= , struct ublk_queue *ubq) { ubq->nr_io_ready++; if (ublk_queue_ready(ubq)) { - ubq->ubq_daemon =3D current; - get_task_struct(ubq->ubq_daemon); ub->nr_queues_ready++; =20 if (capable(CAP_SYS_ADMIN)) @@ -1928,14 +1917,14 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd = *cmd, if (!ubq || ub_cmd->q_id !=3D ubq->q_id) goto out; =20 - if (ubq->ubq_daemon && ubq->ubq_daemon !=3D current) - goto out; - if (tag >=3D ubq->q_depth) goto out; =20 io =3D &ubq->ios[tag]; =20 + if (io->task && io->task !=3D current) + goto out; + /* there is pending io cmd, something must be wrong */ if (io->flags & UBLK_IO_FLAG_ACTIVE) { ret =3D -EBUSY; @@ -1989,6 +1978,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *c= md, =20 ublk_fill_io_cmd(io, cmd, ub_cmd->addr); ublk_mark_io_ready(ub, ubq); + io->task =3D get_task_struct(current); mutex_unlock(&ub->mutex); break; case UBLK_IO_COMMIT_AND_FETCH_REQ: @@ -2228,9 +2218,15 @@ static void ublk_deinit_queue(struct ublk_device *ub= , int q_id) { int size =3D ublk_queue_cmd_buf_size(ub, q_id); struct ublk_queue *ubq =3D ublk_get_queue(ub, q_id); + struct ublk_io *io; + int i; + + for (i =3D 0; i < ubq->q_depth; i++) { + io =3D &ubq->ios[i]; + if (io->task) + put_task_struct(io->task); + } =20 - if (ubq->ubq_daemon) - put_task_struct(ubq->ubq_daemon); if (ubq->io_cmd_buf) free_pages((unsigned long)ubq->io_cmd_buf, get_order(size)); } @@ -2921,15 +2917,8 @@ static void ublk_queue_reinit(struct ublk_device *ub= , struct ublk_queue *ubq) { int i; =20 - WARN_ON_ONCE(!(ubq->ubq_daemon && ubq_daemon_is_dying(ubq))); - /* All old ioucmds have to be completed */ ubq->nr_io_ready =3D 0; - /* old daemon is PF_EXITING, put it now */ - put_task_struct(ubq->ubq_daemon); - /* We have to reset it to NULL, otherwise ub won't accept new FETCH_REQ */ - ubq->ubq_daemon =3D NULL; - ubq->timeout =3D false; ubq->canceling =3D false; =20 for (i =3D 0; i < ubq->q_depth; i++) { @@ -2939,6 +2928,10 @@ static void ublk_queue_reinit(struct ublk_device *ub= , struct ublk_queue *ubq) io->flags =3D 0; io->cmd =3D NULL; io->addr =3D 0; + + WARN_ON_ONCE(!(io->task && (io->task->flags & PF_EXITING))); + put_task_struct(io->task); + io->task =3D NULL; } } =20 @@ -2979,7 +2972,7 @@ static int ublk_ctrl_start_recovery(struct ublk_devic= e *ub, pr_devel("%s: start recovery for dev id %d.\n", __func__, header->dev_id); for (i =3D 0; i < ub->dev_info.nr_hw_queues; i++) ublk_queue_reinit(ub, ublk_get_queue(ub, i)); - /* set to NULL, otherwise new ubq_daemon cannot mmap the io_cmd_buf */ + /* set to NULL, otherwise new tasks cannot mmap the io_cmd_buf */ ub->mm =3D NULL; ub->nr_queues_ready =3D 0; ub->nr_privileged_daemon =3D 0; @@ -2998,14 +2991,14 @@ static int ublk_ctrl_end_recovery(struct ublk_devic= e *ub, int ret =3D -EINVAL; int i; =20 - pr_devel("%s: Waiting for new ubq_daemons(nr: %d) are ready, dev id %d...= \n", - __func__, ub->dev_info.nr_hw_queues, header->dev_id); - /* wait until new ubq_daemon sending all FETCH_REQ */ + pr_devel("%s: Waiting for all FETCH_REQs, dev id %d...\n", __func__, + header->dev_id); + if (wait_for_completion_interruptible(&ub->completion)) return -EINTR; =20 - pr_devel("%s: All new ubq_daemons(nr: %d) are ready, dev id %d\n", - __func__, ub->dev_info.nr_hw_queues, header->dev_id); + pr_devel("%s: All FETCH_REQs received, dev id %d\n", __func__, + header->dev_id); =20 mutex_lock(&ub->mutex); if (ublk_nosrv_should_stop_dev(ub)) --=20 2.34.1