From nobody Fri Apr 3 09:50:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41DA1ECAAD8 for ; Wed, 14 Sep 2022 01:38:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230046AbiINBij (ORCPT ); Tue, 13 Sep 2022 21:38:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229773AbiINBiV (ORCPT ); Tue, 13 Sep 2022 21:38:21 -0400 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4434F6D547; Tue, 13 Sep 2022 18:38:18 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4MS2xN4mw0zlCl1; Wed, 14 Sep 2022 09:36:40 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP2 (Coremail) with SMTP id Syh0CgDHGXOGMCFj24dCAw--.23886S5; Wed, 14 Sep 2022 09:38:16 +0800 (CST) From: Yu Kuai To: song@kernel.org, logang@deltatee.com, guoqing.jiang@linux.dev, pmenzel@molgen.mpg.de Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai3@huawei.com, yukuai1@huaweicloud.com, yi.zhang@huawei.com Subject: [PATCH v2 1/4] md/raid10: cleanup wait_barrier() Date: Wed, 14 Sep 2022 09:49:11 +0800 Message-Id: <20220914014914.398712-2-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220914014914.398712-1-yukuai1@huaweicloud.com> References: <20220914014914.398712-1-yukuai1@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgDHGXOGMCFj24dCAw--.23886S5 X-Coremail-Antispam: 1UD129KBjvJXoWxGrykJF4fKF4xCFWktFWUurg_yoW5Gw4Upw 43GrW3JrWUAF9xJ398JF48CFyFq393XFWUGFyS93ykJF4jyrZ3Wr93G348KryDCrZ5XFyF qFWSyrZxKw4UtrJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUU9m14x267AKxVW5JVWrJwAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jr4l82xGYIkIc2 x26xkF7I0E14v26r1I6r4UM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJw A2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwCF04k20xvY0x0EwIxGrwCFx2 IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v2 6r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67 AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k26cxKx2IY s7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr 0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x0JUqAp5UUUUU= X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yu Kuai Currently the nasty condition is wait_barrier() is hard to read. This patch factor out the condition into a function. There are no functional changes. Signed-off-by: Yu Kuai Acked-by: Paul Menzel --- drivers/md/raid10.c | 56 ++++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 64d6e4cd8a3a..56458a53043d 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -957,44 +957,52 @@ static void lower_barrier(struct r10conf *conf) wake_up(&conf->wait_barrier); } =20 +static bool stop_waiting_barrier(struct r10conf *conf) +{ + /* barrier is dropped */ + if (!conf->barrier) + return true; + + /* + * If there are already pending requests (preventing the barrier from + * rising completely), and the pre-process bio queue isn't empty, then + * don't wait, as we need to empty that queue to get the nr_pending + * count down. + */ + if (atomic_read(&conf->nr_pending)) { + struct bio_list *bio_list =3D current->bio_list; + + if (bio_list && (!bio_list_empty(&bio_list[0]) || + !bio_list_empty(&bio_list[1]))) + return true; + } + + /* move on if recovery thread is blocked by us */ + if (conf->mddev->thread->tsk =3D=3D current && + test_bit(MD_RECOVERY_RUNNING, &conf->mddev->recovery) && + conf->nr_queued > 0) + return true; + + return false; +} + static bool wait_barrier(struct r10conf *conf, bool nowait) { bool ret =3D true; =20 spin_lock_irq(&conf->resync_lock); if (conf->barrier) { - struct bio_list *bio_list =3D current->bio_list; - conf->nr_waiting++; - /* Wait for the barrier to drop. - * However if there are already pending - * requests (preventing the barrier from - * rising completely), and the - * pre-process bio queue isn't empty, - * then don't wait, as we need to empty - * that queue to get the nr_pending - * count down. - */ /* Return false when nowait flag is set */ if (nowait) { ret =3D false; } else { + conf->nr_waiting++; raid10_log(conf->mddev, "wait barrier"); wait_event_lock_irq(conf->wait_barrier, - !conf->barrier || - (atomic_read(&conf->nr_pending) && - bio_list && - (!bio_list_empty(&bio_list[0]) || - !bio_list_empty(&bio_list[1]))) || - /* move on if recovery thread is - * blocked by us - */ - (conf->mddev->thread->tsk =3D=3D current && - test_bit(MD_RECOVERY_RUNNING, - &conf->mddev->recovery) && - conf->nr_queued > 0), + stop_waiting_barrier(conf), conf->resync_lock); + conf->nr_waiting--; } - conf->nr_waiting--; if (!conf->nr_waiting) wake_up(&conf->wait_barrier); } --=20 2.31.1 From nobody Fri Apr 3 09:50:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F47AC6FA89 for ; Wed, 14 Sep 2022 01:38:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229504AbiINBia (ORCPT ); Tue, 13 Sep 2022 21:38:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229957AbiINBiV (ORCPT ); Tue, 13 Sep 2022 21:38:21 -0400 Received: from dggsgout12.his.huawei.com (dggsgout12.his.huawei.com [45.249.212.56]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A759C6A481; Tue, 13 Sep 2022 18:38:18 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout12.his.huawei.com (SkyGuard) with ESMTP id 4MS2x50rKPz6S3MZ; Wed, 14 Sep 2022 09:36:25 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP2 (Coremail) with SMTP id Syh0CgDHGXOGMCFj24dCAw--.23886S6; Wed, 14 Sep 2022 09:38:17 +0800 (CST) From: Yu Kuai To: song@kernel.org, logang@deltatee.com, guoqing.jiang@linux.dev, pmenzel@molgen.mpg.de Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai3@huawei.com, yukuai1@huaweicloud.com, yi.zhang@huawei.com Subject: [PATCH v2 2/4] md/raid10: prevent unnecessary calls to wake_up() in fast path Date: Wed, 14 Sep 2022 09:49:12 +0800 Message-Id: <20220914014914.398712-3-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220914014914.398712-1-yukuai1@huaweicloud.com> References: <20220914014914.398712-1-yukuai1@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgDHGXOGMCFj24dCAw--.23886S6 X-Coremail-Antispam: 1UD129KBjvJXoWxXr4fXF13JrWruF1UJr18Xwb_yoWrAr48p3 yaqF45tFW5ZFZ0qw4DJFWUu3Wjgw1ktFWIkFWvk3s7ZFn5tryftF1UGFyqkryqvrZ3uryU ZFWayrWfGw48tFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUU9m14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_Jryl82xGYIkIc2 x26xkF7I0E14v26r4j6ryUM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2z4x0 Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJw A2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq3wAS 0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7IYx2 IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4UM4x0 Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwCF04k20xvY0x0EwIxGrwCFx2 IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v2 6r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67 AKxVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k26cxKx2IY s7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI0_Gr 0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x0JUc6pPUUUUU= X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yu Kuai Currently, wake_up() is called unconditionally in fast path such as raid10_make_request(), which will cause lock contention under high concurrency: raid10_make_request wake_up __wake_up_common_lock spin_lock_irqsave Improve performance by only call wake_up() if waitqueue is not empty. Signed-off-by: Yu Kuai --- drivers/md/raid10.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 56458a53043d..0edcd98461fe 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -274,6 +274,12 @@ static void put_buf(struct r10bio *r10_bio) lower_barrier(conf); } =20 +static void wake_up_barrier(struct r10conf *conf) +{ + if (wq_has_sleeper(&conf->wait_barrier)) + wake_up(&conf->wait_barrier); +} + static void reschedule_retry(struct r10bio *r10_bio) { unsigned long flags; @@ -286,7 +292,7 @@ static void reschedule_retry(struct r10bio *r10_bio) spin_unlock_irqrestore(&conf->device_lock, flags); =20 /* wake up frozen array... */ - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); =20 md_wakeup_thread(mddev->thread); } @@ -884,7 +890,7 @@ static void flush_pending_writes(struct r10conf *conf) /* flush any pending bitmap writes to disk * before proceeding w/ I/O */ md_bitmap_unplug(conf->mddev->bitmap); - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); =20 while (bio) { /* submit pending writes */ struct bio *next =3D bio->bi_next; @@ -954,7 +960,7 @@ static void lower_barrier(struct r10conf *conf) spin_lock_irqsave(&conf->resync_lock, flags); conf->barrier--; spin_unlock_irqrestore(&conf->resync_lock, flags); - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); } =20 static bool stop_waiting_barrier(struct r10conf *conf) @@ -1004,7 +1010,7 @@ static bool wait_barrier(struct r10conf *conf, bool n= owait) conf->nr_waiting--; } if (!conf->nr_waiting) - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); } /* Only increment nr_pending when we wait */ if (ret) @@ -1017,7 +1023,7 @@ static void allow_barrier(struct r10conf *conf) { if ((atomic_dec_and_test(&conf->nr_pending)) || (conf->array_freeze_pending)) - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); } =20 static void freeze_array(struct r10conf *conf, int extra) @@ -1053,7 +1059,7 @@ static void unfreeze_array(struct r10conf *conf) spin_lock_irq(&conf->resync_lock); conf->barrier--; conf->nr_waiting--; - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); spin_unlock_irq(&conf->resync_lock); } =20 @@ -1078,7 +1084,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, boo= l from_schedule) spin_lock_irq(&conf->device_lock); bio_list_merge(&conf->pending_bio_list, &plug->pending); spin_unlock_irq(&conf->device_lock); - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); md_wakeup_thread(mddev->thread); kfree(plug); return; @@ -1087,7 +1093,7 @@ static void raid10_unplug(struct blk_plug_cb *cb, boo= l from_schedule) /* we aren't scheduling, so we can do the write-out directly. */ bio =3D bio_list_get(&plug->pending); md_bitmap_unplug(mddev->bitmap); - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); =20 while (bio) { /* submit pending writes */ struct bio *next =3D bio->bi_next; @@ -1893,7 +1899,7 @@ static bool raid10_make_request(struct mddev *mddev, = struct bio *bio) __make_request(mddev, bio, sectors); =20 /* In case raid10d snuck in to freeze_array */ - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); return true; } =20 @@ -3040,7 +3046,7 @@ static void handle_write_completed(struct r10conf *co= nf, struct r10bio *r10_bio) * In case freeze_array() is waiting for condition * nr_pending =3D=3D nr_queued + extra to be true. */ - wake_up(&conf->wait_barrier); + wake_up_barrier(conf); md_wakeup_thread(conf->mddev->thread); } else { if (test_bit(R10BIO_WriteError, --=20 2.31.1 From nobody Fri Apr 3 09:50:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 335E9C54EE9 for ; Wed, 14 Sep 2022 01:38:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230006AbiINBif (ORCPT ); Tue, 13 Sep 2022 21:38:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46520 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229710AbiINBiV (ORCPT ); Tue, 13 Sep 2022 21:38:21 -0400 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23D726CF44; Tue, 13 Sep 2022 18:38:19 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4MS2xP3dMSzlFkJ; Wed, 14 Sep 2022 09:36:41 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP2 (Coremail) with SMTP id Syh0CgDHGXOGMCFj24dCAw--.23886S7; Wed, 14 Sep 2022 09:38:17 +0800 (CST) From: Yu Kuai To: song@kernel.org, logang@deltatee.com, guoqing.jiang@linux.dev, pmenzel@molgen.mpg.de Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai3@huawei.com, yukuai1@huaweicloud.com, yi.zhang@huawei.com Subject: [PATCH v2 3/4] md/raid10: fix improper BUG_ON() in raise_barrier() Date: Wed, 14 Sep 2022 09:49:13 +0800 Message-Id: <20220914014914.398712-4-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220914014914.398712-1-yukuai1@huaweicloud.com> References: <20220914014914.398712-1-yukuai1@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgDHGXOGMCFj24dCAw--.23886S7 X-Coremail-Antispam: 1UD129KBjvdXoW7XF4xtF18uFykAFy5tw48Crg_yoWfJFbE93 ZY9F129r1fK347Kw1YgFsavryIgw4kWrnF9F4rt343ua98ZFWrC3W8X3y8XwnxWay7Zr43 ZryxuFyUA3WDCjkaLaAFLSUrUUUUUb8apTn2vfkv8UJUUUU8Yxn0WfASr-VFAUDa7-sFnT 9fnUUIcSsGvfJTRUUUb-kFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k26cxKx2IYs7xG 6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUWwA2048vs2IY02 0Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xv wVC0I7IYx2IY67AKxVWDJVCq3wA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UM2 8EF7xvwVC2z280aVAFwI0_GcCE3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2AI xVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20x vE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xv r2IYc2Ij64vIr41lF7I21c0EjII2zVCS5cI20VAGYxC7MxAIw28IcxkI7VAKI48JMxC20s 026xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_ JrI_JrWlx4CE17CEb7AF67AKxVWUtVW8ZwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14 v26r1j6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8JVWxJwCI42IY6xAIw20EY4v20xva j40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JV W8JrUvcSsGvfC2KfnxnUUI43ZEXa7VUbJ73DUUUUU== X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yu Kuai 'conf->barrier' is protected by 'conf->resync_lock', reading 'conf->barrier' without holding the lock is wrong. Signed-off-by: Yu Kuai --- drivers/md/raid10.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 0edcd98461fe..377d4641bb54 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -936,8 +936,8 @@ static void flush_pending_writes(struct r10conf *conf) =20 static void raise_barrier(struct r10conf *conf, int force) { - BUG_ON(force && !conf->barrier); spin_lock_irq(&conf->resync_lock); + BUG_ON(force && !conf->barrier); =20 /* Wait until no block IO is waiting (unless 'force') */ wait_event_lock_irq(conf->wait_barrier, force || !conf->nr_waiting, --=20 2.31.1 From nobody Fri Apr 3 09:50:41 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C129CC54EE9 for ; Wed, 14 Sep 2022 01:38:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229999AbiINBio (ORCPT ); Tue, 13 Sep 2022 21:38:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229964AbiINBiW (ORCPT ); Tue, 13 Sep 2022 21:38:22 -0400 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6D816D549; Tue, 13 Sep 2022 18:38:19 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.30.67.143]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4MS2xP6qfPzlKhb; Wed, 14 Sep 2022 09:36:41 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.127.227]) by APP2 (Coremail) with SMTP id Syh0CgDHGXOGMCFj24dCAw--.23886S8; Wed, 14 Sep 2022 09:38:17 +0800 (CST) From: Yu Kuai To: song@kernel.org, logang@deltatee.com, guoqing.jiang@linux.dev, pmenzel@molgen.mpg.de Cc: linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, yukuai3@huawei.com, yukuai1@huaweicloud.com, yi.zhang@huawei.com Subject: [PATCH v2 4/4] md/raid10: convert resync_lock to use seqlock Date: Wed, 14 Sep 2022 09:49:14 +0800 Message-Id: <20220914014914.398712-5-yukuai1@huaweicloud.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220914014914.398712-1-yukuai1@huaweicloud.com> References: <20220914014914.398712-1-yukuai1@huaweicloud.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: Syh0CgDHGXOGMCFj24dCAw--.23886S8 X-Coremail-Antispam: 1UD129KBjvJXoW3Jw45Gr43CFWfCFy3Ar15CFg_yoWxWryUpw 4aqr15tFWUXrn0qr4DJa1q9r1Fgw4kKa47Ka9rW3WkZFs5tryfXF1UGr9Ygryqvr9xJFyv qFWrCFWfGw17tFJanT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUU9C14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK8VAvwI8IcIk0 rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JF0E3s1l82xGYI kIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vEj48ve4kI8wA2 z4x0Y4vE2Ix0cI8IcVAFwI0_tr0E3s1l84ACjcxK6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F 4UJwA2z4x0Y4vEx4A2jsIE14v26rxl6s0DM28EF7xvwVC2z280aVCY1x0267AKxVW0oVCq 3wAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ewAv7VC0I7 IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY6r1j6r4U M4x0Y48IcxkI7VAKI48JM4x0x7Aq67IIx4CEVc8vx2IErcIFxwCF04k20xvY0x0EwIxGrw CFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE 14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7IYx2 IY67AKxVWUCVW8JwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr0_Cr1lIxAIcVCF04k26cxK x2IYs7xG6r1j6r1xMIIF0xvEx4A2jsIE14v26r1j6r4UMIIF0xvEx4A2jsIEc7CjxVAFwI 0_Gr0_Gr1UYxBIdaVFxhVjvjDU0xZFpf9x0JUQSdkUUUUU= X-CM-SenderInfo: 51xn3trlr6x35dzhxuhorxvhhfrp/ X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Yu Kuai Currently, wait_barrier() will hold 'resync_lock' to read 'conf->barrier', and io can't be dispatched until 'barrier' is dropped. Since holding the 'barrier' is not common, convert 'resync_lock' to use seqlock so that holding lock can be avoided in fast path. Signed-off-by: Yu Kuai --- drivers/md/raid10.c | 85 +++++++++++++++++++++++++++++---------------- drivers/md/raid10.h | 2 +- 2 files changed, 57 insertions(+), 30 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 377d4641bb54..6c2396fe75a0 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -79,6 +79,21 @@ static void end_reshape(struct r10conf *conf); =20 #include "raid1-10.c" =20 +#define NULL_CMD +#define cmd_before(conf, cmd) \ + do { \ + write_sequnlock_irq(&(conf)->resync_lock); \ + cmd; \ + } while (0) +#define cmd_after(conf) write_seqlock_irq(&(conf)->resync_lock) + +#define wait_event_barrier_cmd(conf, cond, cmd) \ + wait_event_cmd((conf)->wait_barrier, cond, cmd_before(conf, cmd), \ + cmd_after(conf)) + +#define wait_event_barrier(conf, cond) \ + wait_event_barrier_cmd(conf, cond, NULL_CMD) + /* * for resync bio, r10bio pointer can be retrieved from the per-bio * 'struct resync_pages'. @@ -936,30 +951,29 @@ static void flush_pending_writes(struct r10conf *conf) =20 static void raise_barrier(struct r10conf *conf, int force) { - spin_lock_irq(&conf->resync_lock); + write_seqlock_irq(&conf->resync_lock); BUG_ON(force && !conf->barrier); =20 /* Wait until no block IO is waiting (unless 'force') */ - wait_event_lock_irq(conf->wait_barrier, force || !conf->nr_waiting, - conf->resync_lock); + wait_event_barrier(conf, force || !conf->nr_waiting); =20 /* block any new IO from starting */ - conf->barrier++; + WRITE_ONCE(conf->barrier, conf->barrier + 1); =20 /* Now wait for all pending IO to complete */ - wait_event_lock_irq(conf->wait_barrier, - !atomic_read(&conf->nr_pending) && conf->barrier < RESYNC_DEPTH, - conf->resync_lock); + wait_event_barrier(conf, !atomic_read(&conf->nr_pending) && + conf->barrier < RESYNC_DEPTH); =20 - spin_unlock_irq(&conf->resync_lock); + write_sequnlock_irq(&conf->resync_lock); } =20 static void lower_barrier(struct r10conf *conf) { unsigned long flags; - spin_lock_irqsave(&conf->resync_lock, flags); - conf->barrier--; - spin_unlock_irqrestore(&conf->resync_lock, flags); + + write_seqlock_irqsave(&conf->resync_lock, flags); + WRITE_ONCE(conf->barrier, conf->barrier - 1); + write_sequnlock_irqrestore(&conf->resync_lock, flags); wake_up_barrier(conf); } =20 @@ -992,11 +1006,29 @@ static bool stop_waiting_barrier(struct r10conf *con= f) return false; } =20 +static bool wait_barrier_nolock(struct r10conf *conf) +{ + unsigned int seq =3D read_seqbegin(&conf->resync_lock); + + if (READ_ONCE(conf->barrier)) + return false; + + atomic_inc(&conf->nr_pending); + if (!read_seqretry(&conf->resync_lock, seq)) + return true; + + atomic_dec(&conf->nr_pending); + return false; +} + static bool wait_barrier(struct r10conf *conf, bool nowait) { bool ret =3D true; =20 - spin_lock_irq(&conf->resync_lock); + if (wait_barrier_nolock(conf)) + return true; + + write_seqlock_irq(&conf->resync_lock); if (conf->barrier) { /* Return false when nowait flag is set */ if (nowait) { @@ -1004,9 +1036,7 @@ static bool wait_barrier(struct r10conf *conf, bool n= owait) } else { conf->nr_waiting++; raid10_log(conf->mddev, "wait barrier"); - wait_event_lock_irq(conf->wait_barrier, - stop_waiting_barrier(conf), - conf->resync_lock); + wait_event_barrier(conf, stop_waiting_barrier(conf)); conf->nr_waiting--; } if (!conf->nr_waiting) @@ -1015,7 +1045,7 @@ static bool wait_barrier(struct r10conf *conf, bool n= owait) /* Only increment nr_pending when we wait */ if (ret) atomic_inc(&conf->nr_pending); - spin_unlock_irq(&conf->resync_lock); + write_sequnlock_irq(&conf->resync_lock); return ret; } =20 @@ -1040,27 +1070,24 @@ static void freeze_array(struct r10conf *conf, int = extra) * must match the number of pending IOs (nr_pending) before * we continue. */ - spin_lock_irq(&conf->resync_lock); + write_seqlock_irq(&conf->resync_lock); conf->array_freeze_pending++; - conf->barrier++; + WRITE_ONCE(conf->barrier, conf->barrier + 1); conf->nr_waiting++; - wait_event_lock_irq_cmd(conf->wait_barrier, - atomic_read(&conf->nr_pending) =3D=3D conf->nr_queued+extra, - conf->resync_lock, - flush_pending_writes(conf)); - + wait_event_barrier_cmd(conf, atomic_read(&conf->nr_pending) =3D=3D + conf->nr_queued + extra, flush_pending_writes(conf)); conf->array_freeze_pending--; - spin_unlock_irq(&conf->resync_lock); + write_sequnlock_irq(&conf->resync_lock); } =20 static void unfreeze_array(struct r10conf *conf) { /* reverse the effect of the freeze */ - spin_lock_irq(&conf->resync_lock); - conf->barrier--; + write_seqlock_irq(&conf->resync_lock); + WRITE_ONCE(conf->barrier, conf->barrier - 1); conf->nr_waiting--; wake_up_barrier(conf); - spin_unlock_irq(&conf->resync_lock); + write_sequnlock_irq(&conf->resync_lock); } =20 static sector_t choose_data_offset(struct r10bio *r10_bio, @@ -4046,7 +4073,7 @@ static struct r10conf *setup_conf(struct mddev *mddev) INIT_LIST_HEAD(&conf->retry_list); INIT_LIST_HEAD(&conf->bio_end_io_list); =20 - spin_lock_init(&conf->resync_lock); + seqlock_init(&conf->resync_lock); init_waitqueue_head(&conf->wait_barrier); atomic_set(&conf->nr_pending, 0); =20 @@ -4365,7 +4392,7 @@ static void *raid10_takeover_raid0(struct mddev *mdde= v, sector_t size, int devs) rdev->new_raid_disk =3D rdev->raid_disk * 2; rdev->sectors =3D size; } - conf->barrier =3D 1; + WRITE_ONCE(conf->barrier, 1); } =20 return conf; diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h index 5c0804d8bb1f..8c072ce0bc54 100644 --- a/drivers/md/raid10.h +++ b/drivers/md/raid10.h @@ -76,7 +76,7 @@ struct r10conf { /* queue pending writes and submit them on unplug */ struct bio_list pending_bio_list; =20 - spinlock_t resync_lock; + seqlock_t resync_lock; atomic_t nr_pending; int nr_waiting; int nr_queued; --=20 2.31.1