From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91E01C433F5 for ; Thu, 7 Apr 2022 17:23:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346212AbiDGRY6 (ORCPT ); Thu, 7 Apr 2022 13:24:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346085AbiDGRYA (ORCPT ); Thu, 7 Apr 2022 13:24:00 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 981A913F42; Thu, 7 Apr 2022 10:21:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=vUaTkhLKpC517uBr5V8nSaXBUclETbcgKiMjxeP1Ciw=; b=sg+YjSBPnepVHG/AtlVk/OqAQm 0WF5GZz2jT1Gy/kUHruPwf2d3Oo1U7JsNLSLga+7ccr65ElJtOB/6sevZhXobr2cDVSBj7yEucyUA bkVleLBuhQNr5L/Oc0mY2AXuj/PGlbxhV/jTDSE7967CEcPuS4iQyhPzkKGkL0NLG8gtnAfhxp0cl sLD3G1xic607XPD/M1oeqgA0lepnwC+Bn4fgj45BYXaDt3p5nSqnNYO/E0kFZxJoaIMq/302KjX4r L2v8/2d23s+q1C2kpnsYTfHF/nXPTfZoNFA+ar5PB/wMI0uJ6juFckmAPBfOTGLY2KPFfpeJ2tvFu lYz1oT0A==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRf-002CRf-HG; Thu, 07 Apr 2022 10:57:20 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRe-0002Q1-SN; Thu, 07 Apr 2022 10:57:18 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:07 -0600 Message-Id: <20220407165713.9243-2-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 1/7] md/raid5: Cleanup setup_conf() error returns X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Be more careful about the error returns. Most errors in this function are actually ENOMEM, but it forcibly returns EIO if conf has been allocated. Instead return ret and ensure it is set appropriately before each goto abort. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 351d341a1ffa..c0e373a02d3a 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -7166,7 +7166,7 @@ static struct r5conf *setup_conf(struct mddev *mddev) int i; int group_cnt; struct r5worker_group *new_group; - int ret; + int ret =3D -ENOMEM; =20 if (mddev->new_level !=3D 5 && mddev->new_level !=3D 4 @@ -7225,6 +7225,7 @@ static struct r5conf *setup_conf(struct mddev *mddev) spin_lock_init(&conf->device_lock); seqcount_spinlock_init(&conf->gen_lock, &conf->device_lock); mutex_init(&conf->cache_size_mutex); + init_waitqueue_head(&conf->wait_for_quiescent); init_waitqueue_head(&conf->wait_for_stripe); init_waitqueue_head(&conf->wait_for_overlap); @@ -7302,11 +7303,13 @@ static struct r5conf *setup_conf(struct mddev *mdde= v) =20 conf->level =3D mddev->new_level; conf->chunk_sectors =3D mddev->new_chunk_sectors; - if (raid5_alloc_percpu(conf) !=3D 0) + ret =3D raid5_alloc_percpu(conf); + if (ret) goto abort; =20 pr_debug("raid456: run(%s) called.\n", mdname(mddev)); =20 + ret =3D -EIO; rdev_for_each(rdev, mddev) { raid_disk =3D rdev->raid_disk; if (raid_disk >=3D max_disks @@ -7370,6 +7373,7 @@ static struct r5conf *setup_conf(struct mddev *mddev) if (grow_stripes(conf, conf->min_nr_stripes)) { pr_warn("md/raid:%s: couldn't allocate %dkB for buffers\n", mdname(mddev), memory); + ret =3D -ENOMEM; goto abort; } else pr_debug("md/raid:%s: allocated %dkB\n", mdname(mddev), memory); @@ -7383,7 +7387,8 @@ static struct r5conf *setup_conf(struct mddev *mddev) conf->shrinker.count_objects =3D raid5_cache_count; conf->shrinker.batch =3D 128; conf->shrinker.flags =3D 0; - if (register_shrinker(&conf->shrinker)) { + ret =3D register_shrinker(&conf->shrinker); + if (ret) { pr_warn("md/raid:%s: couldn't register shrinker.\n", mdname(mddev)); goto abort; @@ -7394,17 +7399,16 @@ static struct r5conf *setup_conf(struct mddev *mdde= v) if (!conf->thread) { pr_warn("md/raid:%s: couldn't allocate thread.\n", mdname(mddev)); + ret =3D -ENOMEM; goto abort; } =20 return conf; =20 abort: - if (conf) { + if (conf) free_conf(conf); - return ERR_PTR(-EIO); - } else - return ERR_PTR(-ENOMEM); + return ERR_PTR(ret); } =20 static int only_parity(int raid_disk, int algo, int raid_disks, int max_de= graded) --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 255EFC433EF for ; Thu, 7 Apr 2022 17:22:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346186AbiDGRYW (ORCPT ); Thu, 7 Apr 2022 13:24:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346080AbiDGRX5 (ORCPT ); Thu, 7 Apr 2022 13:23:57 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D92664F4; Thu, 7 Apr 2022 10:21:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=ufB2LkWT2Jw65+K7Phw9dSX65Jk1XnoBPX/BNd0+QAU=; b=jiA9PBwX2dbEkxAmEkfpVbWNjb SLD4FQw2KkNztUq+G4EmSofYkyuwTapK+PYBqiap4XInK75Ne5uOxRA9hlT01/DswjYsdKrEp0/j3 ZesRYekEYuNwEJU8Dh4mHjsFnoiFAemg3rIIIUu62xBam9Br960AXvFMVi9NkPQ5t54flyv4gcMSn uZ2JK0pW1nAgvleHvX0/9V2fz8iJPef8qxHfsitItFIOt+Vekm/qO0tLOlgg4AroqiWGhw1o37QKB jdEUYBZv06ohgg0DyrBQlEfu66Hu7+zypdhKyw/72X77tNPOnau+DTQUXzRfNRzwLjCxaHjx6xia5 iNC0xu6Q==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRf-002CRg-HF; Thu, 07 Apr 2022 10:57:20 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002Q4-0w; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:08 -0600 Message-Id: <20220407165713.9243-3-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 2/7] md/raid5: Un-nest struct raid5_percpu definition X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Sparse reports many warnings of the form: drivers/md/raid5.c:1476:16: warning: dereference of noderef expression This is because all struct raid5_percpu definitions get marked as __percpu when really only the pointer in r5conf should have that annotation. Fix this by moving the defnition of raid5_precpu out of the definition of struct r5conf. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.h | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h index 9e8486a9e445..61bc2e1f1b4e 100644 --- a/drivers/md/raid5.h +++ b/drivers/md/raid5.h @@ -560,6 +560,16 @@ struct r5pending_data { struct bio_list bios; }; =20 +struct raid5_percpu { + struct page *spare_page; /* Used when checking P/Q in raid6 */ + void *scribble; /* space for constructing buffer + * lists and performing address + * conversions + */ + int scribble_obj_size; + local_lock_t lock; +}; + struct r5conf { struct hlist_head *stripe_hashtbl; /* only protect corresponding hash list and inactive_list */ @@ -635,15 +645,7 @@ struct r5conf { */ int recovery_disabled; /* per cpu variables */ - struct raid5_percpu { - struct page *spare_page; /* Used when checking P/Q in raid6 */ - void *scribble; /* space for constructing buffer - * lists and performing address - * conversions - */ - int scribble_obj_size; - local_lock_t lock; - } __percpu *percpu; + struct raid5_percpu __percpu *percpu; int scribble_disks; int scribble_sectors; struct hlist_node node; --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 072FFC433F5 for ; Thu, 7 Apr 2022 17:22:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346176AbiDGRYS (ORCPT ); Thu, 7 Apr 2022 13:24:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346076AbiDGRX4 (ORCPT ); Thu, 7 Apr 2022 13:23:56 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A787213F16; Thu, 7 Apr 2022 10:21:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=aKEoBvf3MpywGH/zk0JIbq0R6s1oOUuIxCnmt/DU/to=; b=qVvUkxTcCMuJ1LxddexP9ltzA6 uGj8NdwWHzpyNy0biEN9+fOoh5kJJQd7rZ/A5nNQUhbe/u3UL7AVClFiCWdjHC/n2/FjtUSK0dMHw 8OG2WOShOeHtK1PAMfAXpPsO1b+Oy5Kod101dMzaoQbD8VXPe+J/XLv0+eKsUQ+nw6i4P5OFgYCao ycuVM1aQlQh6w5xkEYbJY+Q/f3QgU/oiHZ7LZoktKRDWHPdSHBrG9Iq1x2H5VC+0T2WShdd4+pemC V0/fDOD3+LywdNYJHBldnFI+BM1Xo7dhfK9OD3z3zNx5PpAqSxYr+cpFHoRyIPG8jFP8TbgkKT4F7 29XDLdCQ==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRf-002CRh-KQ; Thu, 07 Apr 2022 10:57:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002Q7-5w; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:09 -0600 Message-Id: <20220407165713.9243-4-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 3/7] md/raid5: Add __rcu annotation to struct disk_info X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" rdev and replacement are protected in some circumstances with rcu_dereference and synchronize_rcu (in raid5_remove_disk()). However, they were not annotated with __rcu so a sparse warning is emitted for every rcu_dereference() call. Add the __rcu annotation and fix up the initialization with RCU_INIT_POINTER, all pointer modifications with rcu_assign_pointer(), a few cases where the pointer value is tested with rcu_access_pointer() and one case where READ_ONCE() is used instead of rcu_dereference(), a case in print_raid5_conf() that should have rcu_dereference() and rcu_read_[un]lock() calls. Additional sparse issues will be fixed up in further commits. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 46 ++++++++++++++++++++++++++-------------------- drivers/md/raid5.h | 3 ++- 2 files changed, 28 insertions(+), 21 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index c0e373a02d3a..4815f5351818 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -6288,7 +6288,7 @@ static inline sector_t raid5_sync_request(struct mdde= v *mddev, sector_t sector_n */ rcu_read_lock(); for (i =3D 0; i < conf->raid_disks; i++) { - struct md_rdev *rdev =3D READ_ONCE(conf->disks[i].rdev); + struct md_rdev *rdev =3D rcu_dereference(conf->disks[i].rdev); =20 if (rdev =3D=3D NULL || test_bit(Faulty, &rdev->flags)) still_degraded =3D 1; @@ -7320,11 +7320,11 @@ static struct r5conf *setup_conf(struct mddev *mdde= v) if (test_bit(Replacement, &rdev->flags)) { if (disk->replacement) goto abort; - disk->replacement =3D rdev; + RCU_INIT_POINTER(disk->replacement, rdev); } else { if (disk->rdev) goto abort; - disk->rdev =3D rdev; + RCU_INIT_POINTER(disk->rdev, rdev); } =20 if (test_bit(In_sync, &rdev->flags)) { @@ -7631,11 +7631,11 @@ static int raid5_run(struct mddev *mddev) rdev =3D conf->disks[i].replacement; conf->disks[i].replacement =3D NULL; clear_bit(Replacement, &rdev->flags); - conf->disks[i].rdev =3D rdev; + rcu_assign_pointer(conf->disks[i].rdev, rdev); } if (!rdev) continue; - if (conf->disks[i].replacement && + if (rcu_access_pointer(conf->disks[i].replacement) && conf->reshape_progress !=3D MaxSector) { /* replacements and reshape simply do not mix. */ pr_warn("md: cannot handle concurrent replacement and reshape.\n"); @@ -7836,8 +7836,8 @@ static void raid5_status(struct seq_file *seq, struct= mddev *mddev) =20 static void print_raid5_conf (struct r5conf *conf) { + struct md_rdev *rdev; int i; - struct disk_info *tmp; =20 pr_debug("RAID conf printout:\n"); if (!conf) { @@ -7848,14 +7848,16 @@ static void print_raid5_conf (struct r5conf *conf) conf->raid_disks, conf->raid_disks - conf->mddev->degraded); =20 + rcu_read_lock(); for (i =3D 0; i < conf->raid_disks; i++) { char b[BDEVNAME_SIZE]; - tmp =3D conf->disks + i; - if (tmp->rdev) + rdev =3D rcu_dereference(conf->disks[i].rdev); + if (rdev) pr_debug(" disk %d, o:%d, dev:%s\n", - i, !test_bit(Faulty, &tmp->rdev->flags), - bdevname(tmp->rdev->bdev, b)); + i, !test_bit(Faulty, &rdev->flags), + bdevname(rdev->bdev, b)); } + rcu_read_unlock(); } =20 static int raid5_spare_active(struct mddev *mddev) @@ -7906,8 +7908,9 @@ static int raid5_remove_disk(struct mddev *mddev, str= uct md_rdev *rdev) struct r5conf *conf =3D mddev->private; int err =3D 0; int number =3D rdev->raid_disk; - struct md_rdev **rdevp; + struct md_rdev __rcu **rdevp; struct disk_info *p =3D conf->disks + number; + struct md_rdev *tmp; =20 print_raid5_conf(conf); if (test_bit(Journal, &rdev->flags) && conf->log) { @@ -7925,9 +7928,9 @@ static int raid5_remove_disk(struct mddev *mddev, str= uct md_rdev *rdev) log_exit(conf); return 0; } - if (rdev =3D=3D p->rdev) + if (rdev =3D=3D rcu_access_pointer(p->rdev)) rdevp =3D &p->rdev; - else if (rdev =3D=3D p->replacement) + else if (rdev =3D=3D rcu_access_pointer(p->replacement)) rdevp =3D &p->replacement; else return 0; @@ -7947,7 +7950,8 @@ static int raid5_remove_disk(struct mddev *mddev, str= uct md_rdev *rdev) if (!test_bit(Faulty, &rdev->flags) && mddev->recovery_disabled !=3D conf->recovery_disabled && !has_failed(conf) && - (!p->replacement || p->replacement =3D=3D rdev) && + (!rcu_access_pointer(p->replacement) || + rcu_access_pointer(p->replacement) =3D=3D rdev) && number < conf->raid_disks) { err =3D -EBUSY; goto abort; @@ -7958,7 +7962,7 @@ static int raid5_remove_disk(struct mddev *mddev, str= uct md_rdev *rdev) if (atomic_read(&rdev->nr_pending)) { /* lost the race, try later */ err =3D -EBUSY; - *rdevp =3D rdev; + rcu_assign_pointer(*rdevp, rdev); } } if (!err) { @@ -7966,17 +7970,19 @@ static int raid5_remove_disk(struct mddev *mddev, s= truct md_rdev *rdev) if (err) goto abort; } - if (p->replacement) { + + tmp =3D rcu_access_pointer(p->replacement); + if (tmp) { /* We must have just cleared 'rdev' */ - p->rdev =3D p->replacement; - clear_bit(Replacement, &p->replacement->flags); + rcu_assign_pointer(p->rdev, tmp); + clear_bit(Replacement, &tmp->flags); smp_mb(); /* Make sure other CPUs may see both as identical * but will never see neither - if they are careful */ - p->replacement =3D NULL; + rcu_assign_pointer(p->replacement, NULL); =20 if (!err) - err =3D log_modify(conf, p->rdev, true); + err =3D log_modify(conf, tmp, true); } =20 clear_bit(WantReplacement, &rdev->flags); diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h index 61bc2e1f1b4e..638d29863503 100644 --- a/drivers/md/raid5.h +++ b/drivers/md/raid5.h @@ -473,7 +473,8 @@ enum { */ =20 struct disk_info { - struct md_rdev *rdev, *replacement; + struct md_rdev __rcu *rdev; + struct md_rdev __rcu *replacement; struct page *extra_page; /* extra page to use in prexor */ }; =20 --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00565C433F5 for ; Thu, 7 Apr 2022 17:22:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346121AbiDGRYL (ORCPT ); Thu, 7 Apr 2022 13:24:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346071AbiDGRXz (ORCPT ); Thu, 7 Apr 2022 13:23:55 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1380BE08; Thu, 7 Apr 2022 10:21:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=635ylQUcdfFmtEy7PfYRtKd/q8+5RGAxrk1aQwRlsw8=; b=PzXQRQYpXdXhwt9mfTGGsdnyJz l0BTLXXaGHofw2ZqsBOF7SczKDAW+vTZzKlHQinURnmsNceWmWlSSy6xK8dCOTdA4dReoI3ZaTwe5 whstDVfyrbkTNi2b3ycXVvkM/6RRecLsobqgJl9Q5ZWuuSTErhuTFXCbEZFIAZTcPU+BWZ4O3qShp 7PAkXTLp1UjrnzfEOFMMbLg09XP/LUbsZwx/j79QQQMxjBFrw/dZGtMuorww51II/7Qju2ltmyiH9 SPdif6fNoXFELntKTQmXINf9ZvM7uxbjHEn6E1o1xJ6bwhEcp/L52LDzTfPLuVwibdw1aNg3Ujtgh l6RzzhTg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRf-002CRm-Ow; Thu, 07 Apr 2022 10:57:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002QA-Aw; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:10 -0600 Message-Id: <20220407165713.9243-5-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 4/7] md/raid5: Annotate rdev/replacement accesses when nr_pending is elevated X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" There are a number of accesses to __rcu variables that should be safe because nr_pending in the disk is known to be elevated. Create a wrapper around rcu_dereference_protected() to annotate these accesses and verify that nr_pending is non-zero. This fixes a number of sparse warnings. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 4815f5351818..0f29a2769cb3 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -2648,6 +2648,16 @@ static void shrink_stripes(struct r5conf *conf) conf->slab_cache =3D NULL; } =20 +/* + * This helper wraps rcu_dereference_protected() and can be used when + * it is known that the nr_pending of the rdev is elevated. + */ +static struct md_rdev *rdev_pend_deref(struct md_rdev __rcu *rdev) +{ + return rcu_dereference_protected(rdev, + atomic_read(&rcu_access_pointer(rdev)->nr_pending)); +} + static void raid5_end_read_request(struct bio * bi) { struct stripe_head *sh =3D bi->bi_private; @@ -2674,9 +2684,9 @@ static void raid5_end_read_request(struct bio * bi) * In that case it moved down to 'rdev'. * rdev is not removed until all requests are finished. */ - rdev =3D conf->disks[i].replacement; + rdev =3D rdev_pend_deref(conf->disks[i].replacement); if (!rdev) - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); =20 if (use_new_offset(conf, sh)) s =3D sh->sector + rdev->new_data_offset; @@ -2790,11 +2800,11 @@ static void raid5_end_write_request(struct bio *bi) =20 for (i =3D 0 ; i < disks; i++) { if (bi =3D=3D &sh->dev[i].req) { - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); break; } if (bi =3D=3D &sh->dev[i].rreq) { - rdev =3D conf->disks[i].replacement; + rdev =3D rdev_pend_deref(conf->disks[i].replacement); if (rdev) replacement =3D 1; else @@ -2802,7 +2812,7 @@ static void raid5_end_write_request(struct bio *bi) * replaced it. rdev is not removed * until all requests are finished. */ - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); break; } } @@ -5213,23 +5223,23 @@ static void handle_stripe(struct stripe_head *sh) struct r5dev *dev =3D &sh->dev[i]; if (test_and_clear_bit(R5_WriteError, &dev->flags)) { /* We own a safe reference to the rdev */ - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); if (!rdev_set_badblocks(rdev, sh->sector, RAID5_STRIPE_SECTORS(conf), 0)) md_error(conf->mddev, rdev); rdev_dec_pending(rdev, conf->mddev); } if (test_and_clear_bit(R5_MadeGood, &dev->flags)) { - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); rdev_clear_badblocks(rdev, sh->sector, RAID5_STRIPE_SECTORS(conf), 0); rdev_dec_pending(rdev, conf->mddev); } if (test_and_clear_bit(R5_MadeGoodRepl, &dev->flags)) { - rdev =3D conf->disks[i].replacement; + rdev =3D rdev_pend_deref(conf->disks[i].replacement); if (!rdev) /* rdev have been moved down */ - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_pend_deref(conf->disks[i].rdev); rdev_clear_badblocks(rdev, sh->sector, RAID5_STRIPE_SECTORS(conf), 0); rdev_dec_pending(rdev, conf->mddev); --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B01CC433FE for ; Thu, 7 Apr 2022 17:22:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346099AbiDGRYD (ORCPT ); Thu, 7 Apr 2022 13:24:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45682 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346068AbiDGRXy (ORCPT ); Thu, 7 Apr 2022 13:23:54 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9B2025F6; Thu, 7 Apr 2022 10:21:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=dZhxBNVYkNdmeubh8z4rxDDO+YUbiSHwlMwD0uwOpJo=; b=dyvmiTvDvOJiFM/6xYqOE6F0U7 Y4rLQmgIa48GLJBIarMMTfRqry5lQP49hTba7Irc8WOPtoxakDztBr7mjwtzHM1gZMT4sHEs6nf3l R7zv0D7KBRTcfXAssW741WeBgwKTuW1ntNPMmoCOV50yNQEXfQ2FURpiv0h7Dx4AMdtWPOeVqDaA3 niqywLT6DVhSMQM3JNChMinxOekmiotuzZbKgJgzECJ/37ZSYAJKmMsOhrxSh/N+j/ccM33UwAp6r 6HQFLGyDaMd/6Jm/OZHB+b7gUNmeCLliOyj/jF126Aer7cout9XnCZpPDqFlqqVDJSqtkUvlzRyk3 Ip3S6CVA==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRg-002CRg-Th; Thu, 07 Apr 2022 10:57:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002QD-GM; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:11 -0600 Message-Id: <20220407165713.9243-6-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 5/7] md/raid5: Annotate rdev/replacement access when mddev_lock is held X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The mddev_lock should be held during raid5_remove_disk() which is when the rdev/replacement pointers are modified. So any access to these pointers marked __rcu should be safe whenever the mddev_lock is held. There are numerous such access that currently produce sparse warnings. Add a helper function, rdev_mdlock_deref() that wraps rcu_dereference_protected() in all these instances. This annotation fixes a number of sparse warnings. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 65 ++++++++++++++++++++++++++++++---------------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 0f29a2769cb3..fa955d23c88f 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -2658,6 +2658,18 @@ static struct md_rdev *rdev_pend_deref(struct md_rde= v __rcu *rdev) atomic_read(&rcu_access_pointer(rdev)->nr_pending)); } =20 +/* + * This helper wraps rcu_dereference_protected() and should be used + * when it is known that the mddev_lock() is held. This is safe + * seeing raid5_remove_disk() has the same lock held. + */ +static struct md_rdev *rdev_mdlock_deref(struct mddev *mddev, + struct md_rdev __rcu *rdev) +{ + return rcu_dereference_protected(rdev, + lockdep_is_held(&mddev->reconfig_mutex)); +} + static void raid5_end_read_request(struct bio * bi) { struct stripe_head *sh =3D bi->bi_private; @@ -7635,10 +7647,11 @@ static int raid5_run(struct mddev *mddev) =20 for (i =3D 0; i < conf->raid_disks && conf->previous_raid_disks; i++) { - rdev =3D conf->disks[i].rdev; + rdev =3D rdev_mdlock_deref(mddev, conf->disks[i].rdev); if (!rdev && conf->disks[i].replacement) { /* The replacement is all we have yet */ - rdev =3D conf->disks[i].replacement; + rdev =3D rdev_mdlock_deref(mddev, + conf->disks[i].replacement); conf->disks[i].replacement =3D NULL; clear_bit(Replacement, &rdev->flags); rcu_assign_pointer(conf->disks[i].rdev, rdev); @@ -7874,36 +7887,38 @@ static int raid5_spare_active(struct mddev *mddev) { int i; struct r5conf *conf =3D mddev->private; - struct disk_info *tmp; + struct md_rdev *rdev, *replacement; int count =3D 0; unsigned long flags; =20 for (i =3D 0; i < conf->raid_disks; i++) { - tmp =3D conf->disks + i; - if (tmp->replacement - && tmp->replacement->recovery_offset =3D=3D MaxSector - && !test_bit(Faulty, &tmp->replacement->flags) - && !test_and_set_bit(In_sync, &tmp->replacement->flags)) { + rdev =3D rdev_mdlock_deref(mddev, conf->disks[i].rdev); + replacement =3D rdev_mdlock_deref(mddev, + conf->disks[i].replacement); + if (replacement + && replacement->recovery_offset =3D=3D MaxSector + && !test_bit(Faulty, &replacement->flags) + && !test_and_set_bit(In_sync, &replacement->flags)) { /* Replacement has just become active. */ - if (!tmp->rdev - || !test_and_clear_bit(In_sync, &tmp->rdev->flags)) + if (!rdev + || !test_and_clear_bit(In_sync, &rdev->flags)) count++; - if (tmp->rdev) { + if (rdev) { /* Replaced device not technically faulty, * but we need to be sure it gets removed * and never re-added. */ - set_bit(Faulty, &tmp->rdev->flags); + set_bit(Faulty, &rdev->flags); sysfs_notify_dirent_safe( - tmp->rdev->sysfs_state); + rdev->sysfs_state); } - sysfs_notify_dirent_safe(tmp->replacement->sysfs_state); - } else if (tmp->rdev - && tmp->rdev->recovery_offset =3D=3D MaxSector - && !test_bit(Faulty, &tmp->rdev->flags) - && !test_and_set_bit(In_sync, &tmp->rdev->flags)) { + sysfs_notify_dirent_safe(replacement->sysfs_state); + } else if (rdev + && rdev->recovery_offset =3D=3D MaxSector + && !test_bit(Faulty, &rdev->flags) + && !test_and_set_bit(In_sync, &rdev->flags)) { count++; - sysfs_notify_dirent_safe(tmp->rdev->sysfs_state); + sysfs_notify_dirent_safe(rdev->sysfs_state); } } spin_lock_irqsave(&conf->device_lock, flags); @@ -7968,6 +7983,7 @@ static int raid5_remove_disk(struct mddev *mddev, str= uct md_rdev *rdev) } *rdevp =3D NULL; if (!test_bit(RemoveSynchronized, &rdev->flags)) { + lockdep_assert_held(&mddev->reconfig_mutex); synchronize_rcu(); if (atomic_read(&rdev->nr_pending)) { /* lost the race, try later */ @@ -8008,6 +8024,7 @@ static int raid5_add_disk(struct mddev *mddev, struct= md_rdev *rdev) int ret, err =3D -EEXIST; int disk; struct disk_info *p; + struct md_rdev *tmp; int first =3D 0; int last =3D conf->raid_disks - 1; =20 @@ -8065,7 +8082,8 @@ static int raid5_add_disk(struct mddev *mddev, struct= md_rdev *rdev) } for (disk =3D first; disk <=3D last; disk++) { p =3D conf->disks + disk; - if (test_bit(WantReplacement, &p->rdev->flags) && + tmp =3D rdev_mdlock_deref(mddev, p->rdev); + if (test_bit(WantReplacement, &tmp->flags) && p->replacement =3D=3D NULL) { clear_bit(In_sync, &rdev->flags); set_bit(Replacement, &rdev->flags); @@ -8356,6 +8374,7 @@ static void end_reshape(struct r5conf *conf) static void raid5_finish_reshape(struct mddev *mddev) { struct r5conf *conf =3D mddev->private; + struct md_rdev *rdev; =20 if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) { =20 @@ -8367,10 +8386,12 @@ static void raid5_finish_reshape(struct mddev *mdde= v) for (d =3D conf->raid_disks ; d < conf->raid_disks - mddev->delta_disks; d++) { - struct md_rdev *rdev =3D conf->disks[d].rdev; + rdev =3D rdev_mdlock_deref(mddev, + conf->disks[d].rdev); if (rdev) clear_bit(In_sync, &rdev->flags); - rdev =3D conf->disks[d].replacement; + rdev =3D rdev_mdlock_deref(mddev, + conf->disks[d].replacement); if (rdev) clear_bit(In_sync, &rdev->flags); } --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8B2BC433EF for ; Thu, 7 Apr 2022 17:22:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346109AbiDGRYG (ORCPT ); Thu, 7 Apr 2022 13:24:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346073AbiDGRX4 (ORCPT ); Thu, 7 Apr 2022 13:23:56 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0ABC513D47; Thu, 7 Apr 2022 10:21:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=UgtYxnZUE1oxXmbmvyRy5xJTosToTlA4CekiENW7Cls=; b=jnjOkpzVmSD5gJjs8S1zA0j10R r8dfSrudZSfcdJ6AAZAo1oZsAdoESIfQHFSPlJ7K+lH2O2+0XIKu+O97UNcy7GzUIsNQWJ3OBSHFT H0q70jDul3A8ZIXl2ZZ9agedNP8En3+70WLhOKx6bVNMRhMjANV6kEoaM6EDuCWex3Qbjl/UOq6J/ ErssYOokp0bCgJqgPK0GwYQn/1gHzMPdqqPg5e/9DU6ur/wctu3zIbxLFBZwSYr1ayzdVoUkRJOvb j717nYL9BwU6GOiHFobGxOmppewwQU5f3HkGv8QfuQE0UE3wJS80RHGyBjHxcbbVarOob8wsDd3iq bTEGeynw==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRg-002CRf-CN; Thu, 07 Apr 2022 10:57:21 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002QG-LB; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:12 -0600 Message-Id: <20220407165713.9243-7-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 6/7] md/raid5-ppl: Annotate with rcu_dereference_protected() X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To suppress the last remaining sparse warnings about accessing rdev, add rcu_dereference_protected calls to a couple places in raid5-ppl. All of these places are called under raid5_run and therefore are occurring before the array has started and is thus safe. There's no sensible check to do for the second argument of rcu_dereference_protected() so a comment is added instead. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5-ppl.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/md/raid5-ppl.c b/drivers/md/raid5-ppl.c index d3962d92df18..55d065a87b89 100644 --- a/drivers/md/raid5-ppl.c +++ b/drivers/md/raid5-ppl.c @@ -883,7 +883,9 @@ static int ppl_recover_entry(struct ppl_log *log, struc= t ppl_header_entry *e, (unsigned long long)r_sector, dd_idx, (unsigned long long)sector); =20 - rdev =3D conf->disks[dd_idx].rdev; + /* Array has not started so rcu dereference is safe */ + rdev =3D rcu_dereference_protected( + conf->disks[dd_idx].rdev, 1); if (!rdev || (!test_bit(In_sync, &rdev->flags) && sector >=3D rdev->recovery_offset)) { pr_debug("%s:%*s data member disk %d missing\n", @@ -934,7 +936,10 @@ static int ppl_recover_entry(struct ppl_log *log, stru= ct ppl_header_entry *e, parity_sector =3D raid5_compute_sector(conf, r_sector_first + i, 0, &disk, &sh); BUG_ON(sh.pd_idx !=3D le32_to_cpu(e->parity_disk)); - parity_rdev =3D conf->disks[sh.pd_idx].rdev; + + /* Array has not started so rcu dereference is safe */ + parity_rdev =3D rcu_dereference_protected( + conf->disks[sh.pd_idx].rdev, 1); =20 BUG_ON(parity_rdev->bdev->bd_dev !=3D log->rdev->bdev->bd_dev); pr_debug("%s:%*s write parity at sector %llu, disk %s\n", @@ -1404,7 +1409,9 @@ int ppl_init_log(struct r5conf *conf) =20 for (i =3D 0; i < ppl_conf->count; i++) { struct ppl_log *log =3D &ppl_conf->child_logs[i]; - struct md_rdev *rdev =3D conf->disks[i].rdev; + /* Array has not started so rcu dereference is safe */ + struct md_rdev *rdev =3D + rcu_dereference_protected(conf->disks[i].rdev, 1); =20 mutex_init(&log->io_mutex); spin_lock_init(&log->io_list_lock); --=20 2.30.2 From nobody Tue Jun 9 21:04:45 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 291E0C433FE for ; Thu, 7 Apr 2022 17:22:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346081AbiDGRX5 (ORCPT ); Thu, 7 Apr 2022 13:23:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233096AbiDGRXw (ORCPT ); Thu, 7 Apr 2022 13:23:52 -0400 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D270BC0F; Thu, 7 Apr 2022 10:21:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:MIME-Version:References:In-Reply-To: Message-Id:Date:Cc:To:From:content-disposition; bh=wpn/cFs7ICqDTKdxfQRTWuXVXhkklSg4ezYfXgfOPzE=; b=j+1wU3y0FcEcwb25+1IusqjMCD fjuyD0pji3sns2Syg1LqjS8C0aS9qh+PYurWg4idi1CEC9lNYz//3U5OwqCJXa3C6RZWGwR+U0TOz hzEBtQLnCUvCqixN8V16o8fIi3w1nWdfcHtQppY65JHvkpcVZzH67xQpEXL3HiCNqOJrM5HFZnNni EosTrt05nmIp/4j2NtrkNkoLUjMUDqmzDdlVHMp0EGCQ3IIS4veD1mctF9IlMDXef9gbUwYoM1v2g 3agmhkzHCNLPx/s8LdBs13CeyvM3BEhUjTUXl3xa+22+/7p3nI90n8uP4CX8HRLhtPyri6DfI88Bz CDxBoQzg==; Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1ncVRg-002CRe-Bt; Thu, 07 Apr 2022 10:57:22 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.94.2) (envelope-from ) id 1ncVRf-0002QJ-PW; Thu, 07 Apr 2022 10:57:19 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, Song Liu Cc: Guoqing Jiang , Logan Gunthorpe Date: Thu, 7 Apr 2022 10:57:13 -0600 Message-Id: <20220407165713.9243-8-logang@deltatee.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220407165713.9243-1-logang@deltatee.com> References: <20220407165713.9243-1-logang@deltatee.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-kernel@vger.kernel.org, linux-raid@vger.kernel.org, song@kernel.org, guoqing.jiang@linux.dev, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH v1 7/7] md/raid5: Annotate functions that hold device_lock with __must_hold X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" A handful of functions note the device_lock must be held with a comment but this is not comprehensive. Many other functions hold the lock when taken so add an __must_hold() to each call to annotate when the lock is held. This makes it a bit easier to analyse device_lock. Signed-off-by: Logan Gunthorpe Reviewed-by: Christoph Hellwig --- drivers/md/raid5.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index fa955d23c88f..b53000a917df 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -79,18 +79,21 @@ static inline int stripe_hash_locks_hash(struct r5conf = *conf, sector_t sect) } =20 static inline void lock_device_hash_lock(struct r5conf *conf, int hash) + __acquires(&conf->device_lock) { spin_lock_irq(conf->hash_locks + hash); spin_lock(&conf->device_lock); } =20 static inline void unlock_device_hash_lock(struct r5conf *conf, int hash) + __releases(&conf->device_lock) { spin_unlock(&conf->device_lock); spin_unlock_irq(conf->hash_locks + hash); } =20 static inline void lock_all_device_hash_locks_irq(struct r5conf *conf) + __acquires(&conf->device_lock) { int i; spin_lock_irq(conf->hash_locks); @@ -100,6 +103,7 @@ static inline void lock_all_device_hash_locks_irq(struc= t r5conf *conf) } =20 static inline void unlock_all_device_hash_locks_irq(struct r5conf *conf) + __releases(&conf->device_lock) { int i; spin_unlock(&conf->device_lock); @@ -164,6 +168,7 @@ static bool stripe_is_lowprio(struct stripe_head *sh) } =20 static void raid5_wakeup_stripe_thread(struct stripe_head *sh) + __must_hold(&sh->raid_conf->device_lock) { struct r5conf *conf =3D sh->raid_conf; struct r5worker_group *group; @@ -211,6 +216,7 @@ static void raid5_wakeup_stripe_thread(struct stripe_he= ad *sh) =20 static void do_release_stripe(struct r5conf *conf, struct stripe_head *sh, struct list_head *temp_inactive_list) + __must_hold(&conf->device_lock) { int i; int injournal =3D 0; /* number of date pages with R5_InJournal */ @@ -296,6 +302,7 @@ static void do_release_stripe(struct r5conf *conf, stru= ct stripe_head *sh, =20 static void __release_stripe(struct r5conf *conf, struct stripe_head *sh, struct list_head *temp_inactive_list) + __must_hold(&conf->device_lock) { if (atomic_dec_and_test(&sh->count)) do_release_stripe(conf, sh, temp_inactive_list); @@ -350,9 +357,9 @@ static void release_inactive_stripe_list(struct r5conf = *conf, } } =20 -/* should hold conf->device_lock already */ static int release_stripe_list(struct r5conf *conf, struct list_head *temp_inactive_list) + __must_hold(&conf->device_lock) { struct stripe_head *sh, *t; int count =3D 0; @@ -629,6 +636,10 @@ static struct stripe_head *__find_stripe(struct r5conf= *conf, sector_t sector, * This is because some failed devices may only affect one * of the two sections, and some non-in_sync devices may * be insync in the section most affected by failed devices. + * + * Most calls to this function hold &conf->device_lock. Calls + * in raid5_run() do not require the lock as no other threads + * have been started yet. */ int raid5_calc_degraded(struct r5conf *conf) { @@ -5278,6 +5289,7 @@ static void handle_stripe(struct stripe_head *sh) } =20 static void raid5_activate_delayed(struct r5conf *conf) + __must_hold(&conf->device_lock) { if (atomic_read(&conf->preread_active_stripes) < IO_THRESHOLD) { while (!list_empty(&conf->delayed_list)) { @@ -5295,9 +5307,9 @@ static void raid5_activate_delayed(struct r5conf *con= f) } =20 static void activate_bit_delay(struct r5conf *conf, - struct list_head *temp_inactive_list) + struct list_head *temp_inactive_list) + __must_hold(&conf->device_lock) { - /* device_lock is held */ struct list_head head; list_add(&head, &conf->bitmap_list); list_del_init(&conf->bitmap_list); @@ -5522,6 +5534,7 @@ static struct bio *chunk_aligned_read(struct mddev *m= ddev, struct bio *raid_bio) * handle_list. */ static struct stripe_head *__get_priority_stripe(struct r5conf *conf, int = group) + __must_hold(&conf->device_lock) { struct stripe_head *sh, *tmp; struct list_head *handle_list =3D NULL; @@ -6393,8 +6406,7 @@ static int retry_aligned_read(struct r5conf *conf, s= truct bio *raid_bio, static int handle_active_stripes(struct r5conf *conf, int group, struct r5worker *worker, struct list_head *temp_inactive_list) - __releases(&conf->device_lock) - __acquires(&conf->device_lock) + __must_hold(&conf->device_lock) { struct stripe_head *batch[MAX_STRIPE_BATCH], *sh; int i, batch_size =3D 0, hash; --=20 2.30.2