From nobody Fri Jul 24 05:21:57 2026 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C2064C77DF; Thu, 23 Jul 2026 13:55:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814969; cv=none; b=RmanNK2a90RRSwAwBhNF8w8padywHSx0WjAmwOUnBb+JCPRLILIJN1paeGIJlrRpt7amqftDIixOZdcFHF9JMNTopTnr6CjWAIMZbBEzvrGYk8gR757o6cgngTMUQS3nyvAyRFclLcV2Adt1m2IvXyeHJZO1g0/MXGdVobO+tBU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814969; c=relaxed/simple; bh=39mkJPwtzNFZ8nRNEFnHXySP+q5lQynf/eepMujHrrg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FLuTBRrOetJnm8LKU80NN93exK3wTh8jd68vxhJY/ab3kpHZW+7o7hlQEfCCF6UBGrrYM6fnsHsZ5RSSxa4YC1sR8Y94VqcyqCMbxFhOaKnnpRvYeym9Si98ZecB3bNr2HOolXl6zAuCrhgjfvoxn5C1SChUUZPz7Afy8JbCN1Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=TaguCRR9; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="TaguCRR9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1784814945; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=rTKyGSGgkShjZzpP3QRx1yF53Mb5XhDQUXW+TRrvk/U=; b=TaguCRR9tz2jWTkiTJ9IK2/jBMqJkkhDIIEbZCeP8KQBivwFrYzfRD+kXGiJMJ7Esz6+WtWqYbWuMpV8y4L+wfndhrEfHQF58fZcA/Ih60gILT3SH1qvOwWrSneGxon6uGk/bOLCwDq7jkC4lkQ+HDeW/bb1VJdGCV3/DBtq/ow= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R911e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045133197;MF=yunye.zhao@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0X7gKXb1_1784814940; Received: from j66c13357.sqa.eu95.tbsite.net(mailfrom:yunye.zhao@linux.alibaba.com fp:SMTPD_---0X7gKXb1_1784814940 cluster:ay36) by smtp.aliyun-inc.com; Thu, 23 Jul 2026 21:55:44 +0800 From: Yunye Zhao To: Song Liu , Yu Kuai Cc: Li Nan , Xiao Ni , Joseph Qi , linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, Yunye Zhao Subject: [PATCH v2 1/3] md/raid10: fix still_degraded being inverted in raid10_sync_request() Date: Thu, 23 Jul 2026 21:55:33 +0800 Message-Id: <20260723135535.101995-2-yunye.zhao@linux.alibaba.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> References: <20260717062743.128189-1-yunye.zhao@linux.alibaba.com> <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Commit fe6a19d40ceb ("md/md-bitmap: merge md_bitmap_start_sync() into bitmap_operations") converted still_degraded from int to bool, but inverted the assignment in the loop that checks whether the array will still be degraded after the current device is recovered: "still_degraded =3D 1" became "still_degraded =3D false". As a result, recovering a device while another mirror is still missing calls md_bitmap_start_sync() with degraded =3D=3D false, which clears bitmap bits that the still-missing device needs. When that device is re-added, its bitmap-based recovery finds the bits already cleared and skips every region written while the array was degraded, so it is marked In_sync while holding stale data: silent corruption. Reproducer (raid10 near=3D2, 4 disks, internal bitmap): - fail and remove one disk of each mirror pair - write to the degraded array - re-add both disks and let recovery finish - "check" reports mismatch_cnt=3D262272 after 256 MiB of degraded writes and file contents differ; the second disk's "recovery" completes in milliseconds because everything is skipped The same conversion in raid1 got it right (still_degraded =3D true). Restore the correct value. Fixes: fe6a19d40ceb ("md/md-bitmap: merge md_bitmap_start_sync() into bitma= p_operations") Cc: stable@vger.kernel.org Signed-off-by: Yunye Zhao --- drivers/md/raid10.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 0a3cfdd3f5df..54cddb3a98cd 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -3365,7 +3365,7 @@ static sector_t raid10_sync_request(struct mddev *mdd= ev, sector_t sector_nr, struct md_rdev *rdev =3D conf->mirrors[j].rdev; =20 if (rdev =3D=3D NULL || test_bit(Faulty, &rdev->flags)) { - still_degraded =3D false; + still_degraded =3D true; break; } } base-commit: 4539944e515183668109bdf4d0c3d7d228383d88 --=20 2.19.1.6.gb485710b From nobody Fri Jul 24 05:21:57 2026 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEFCF4D98EF; Thu, 23 Jul 2026 13:55:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.132 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814974; cv=none; b=Ejj05HLAGGdbA0bKYtldb4CRnfBF8F3Ga+U4oO7RWGQQp/tNj0vnrgU2pANI6G5hbcCozX8ZzloaGdIGcitEH/BGJL/QKS6gC1ZJA7o+1X6aT1DsY8DF01zb4hm9d7dtQf6IJyYHhcRXmIqEPuCaKFV5jKxrIli5t3C09ez6BC4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814974; c=relaxed/simple; bh=urCA+GgVdz1b+HfsPe6dOn2NDcT/bTNRXhBkmyYNk/8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nb3l91pICJC2HUGZJfTeUNuWuN7ICLu10irYcaX/KdOpEEzUlrWHW9MWh7ngFEDFHyRaz5FmQuzpKnYhf9hQLalI65VBFoZfk8bG7ccBPJs070XvZHVWt+Xv0DCN9HfmMUHf2ivCnEATfdc3uSmUiyVPRKWalfxqF7uyW1O3tNM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=b55fYvXJ; arc=none smtp.client-ip=115.124.30.132 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="b55fYvXJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1784814946; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=oZ+Cf5N5adrT2q/yc6Pli6T3HIhz3F5tsUmqStQxS9o=; b=b55fYvXJR6Dk8K1PwUcsxpnZTG8MyA8rBJL+q0rFzDlnYT2sccehmHDbzsYaLStQKTCwfBHoWTRztK8fFliYsH88ChMp3XHl7DQ/frQgPkVZ6HVi3bcy/nZXEKayW0IEI/jQeZrBpg4gb20GuqTEh7TGbN5At34ZlzYDuYzdvAw= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R281e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037009110;MF=yunye.zhao@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0X7gKXcP_1784814945; Received: from j66c13357.sqa.eu95.tbsite.net(mailfrom:yunye.zhao@linux.alibaba.com fp:SMTPD_---0X7gKXcP_1784814945 cluster:ay36) by smtp.aliyun-inc.com; Thu, 23 Jul 2026 21:55:46 +0800 From: Yunye Zhao To: Song Liu , Yu Kuai Cc: Li Nan , Xiao Ni , Joseph Qi , linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, Yunye Zhao Subject: [PATCH v2 2/3] md: add cond_resched() to md_do_sync()'s skip path Date: Thu, 23 Jul 2026 21:55:34 +0800 Message-Id: <20260723135535.101995-3-yunye.zhao@linux.alibaba.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> References: <20260717062743.128189-1-yunye.zhao@linux.alibaba.com> <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When sync_request() reports a skipped region (*skipped =3D=3D 1), md_do_sync()'s main loop advances the cursor and takes an early continue: j +=3D sectors; ... if (last_check + window > io_sectors || j =3D=3D max_sectors) continue; If the personality returns a small span per call (raid10 recovery returns only 128 sectors), syncing a large, mostly clean array iterates this branch an enormous number of times without ever yielding the CPU. On a non-preemptive kernel the resync thread then trips the soft-lockup watchdog: watchdog: BUG: soft lockup - CPU#149 stuck for 313s! [mdX_resync] md_bitmap_start_sync+0x6f/0xe0 raid10_sync_request+0x2c9/0x1530 [raid10] md_do_sync+0x810/0x1030 md_thread+0xa7/0x150 Add a cond_resched(). This does not reduce the wasted iterations; the excessive iteration count is a raid10 problem addressed separately. Cc: stable@vger.kernel.org Signed-off-by: Yunye Zhao --- drivers/md/md.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/md/md.c b/drivers/md/md.c index d1465bcd86c8..c2e8b203c4db 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -9881,8 +9881,10 @@ void md_do_sync(struct md_thread *thread) */ md_new_event(); =20 - if (last_check + window > io_sectors || j =3D=3D max_sectors) + if (last_check + window > io_sectors || j =3D=3D max_sectors) { + cond_resched(); continue; + } =20 last_check =3D io_sectors; repeat: --=20 2.19.1.6.gb485710b From nobody Fri Jul 24 05:21:57 2026 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53F154CA298; Thu, 23 Jul 2026 13:55:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.130 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814964; cv=none; b=C8zZS23owlcq47NfjscfK7TYj8niSMnq7zyboRP1jxpP/Rjxold95bP4wED2l5/1qEMz1JBZBUnY9rpwjuQ8I93avUJF/xlVd2q/4Ccvut33/ZMjlErwfMM6Za0h/+kNo1aDZIdqVUCHwoqShILaAhyZq/FtNoBIJxlLttwO4LE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784814964; c=relaxed/simple; bh=gTrhvfy0oDbzYdmL6jYNJkTvn25z5kMd/4693uGNvr0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=U3BkmKVxPa+W7qUIrjTM1DrRjVxSPBxULkVM7IplkOQ7EFG5JsK5zIWIK+60MrW1v7VlyrRXxcne35dFiSh7SJ1vZmWLpBuQyZaeVhneNJHaWlYDGVu5C9KVIXi4OaJnjpVpSv/BMN4HJjd3xs1/w8mW6pJqfonKNPMb6NInM44= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=P6I/0M80; arc=none smtp.client-ip=115.124.30.130 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="P6I/0M80" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1784814948; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=qKokoOcqtowGURSpXXhr3eJRP2q6VSeGKv/KxkaOO84=; b=P6I/0M80N72+A2f8AZH7yompEd6Cjl7H/TkQWNrM/NSgPviMPDUPw1k1s5D8R6R9Bs0yc86gCHv6864zTnnHzk0MpVbKMcf5DT/80Zxgw3ZldIZ7dM1PG0rFNy2RWaP8Q8aVSM69qJGyX6D7LTw+/UJI98jSGW/hQ5C46B5unPs= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033032089153;MF=yunye.zhao@linux.alibaba.com;NM=1;PH=DS;RN=8;SR=0;TI=SMTPD_---0X7gKXce_1784814946; Received: from j66c13357.sqa.eu95.tbsite.net(mailfrom:yunye.zhao@linux.alibaba.com fp:SMTPD_---0X7gKXce_1784814946 cluster:ay36) by smtp.aliyun-inc.com; Thu, 23 Jul 2026 21:55:47 +0800 From: Yunye Zhao To: Song Liu , Yu Kuai Cc: Li Nan , Xiao Ni , Joseph Qi , linux-raid@vger.kernel.org, linux-kernel@vger.kernel.org, Yunye Zhao Subject: [PATCH v2 3/3] md/raid10: skip clean regions in bulk during recovery Date: Thu, 23 Jul 2026 21:55:35 +0800 Message-Id: <20260723135535.101995-4-yunye.zhao@linux.alibaba.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> References: <20260717062743.128189-1-yunye.zhao@linux.alibaba.com> <20260723135535.101995-1-yunye.zhao@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" During recovery of a degraded RAID10 with a mostly-clean bitmap, the "everything skipped" path (biolist =3D=3D NULL) returns max_sync =3D=3D 128 sectors, even though md_bitmap_start_sync() already reported a much larger clean range (an unallocated bitmap page spans 2^31 sectors with a 512M bitmap chunk). On the reported 2^40-sector array md_do_sync() then crawls through 2^33 no-op iterations, burning a CPU for the whole sweep; the cond_resched() in md_do_sync()'s skip path only stops the watchdog firing. The skip path cannot simply return sync_blocks: sync_blocks is measured in array sectors while the return value advances the per-device recovery cursor, and the two spaces differ by the raid10 layout. For a near layout (far_copies =3D=3D 1 && !far_offset) the mapping is linear with slope raid_disks / near_copies; when near_copies evenly divides raid_disks the conversion is exact. Track the minimum clean span across the skipped devices, convert it to device sectors and skip it in one step. far/offset layouts are not a linear scale and keep the previous behaviour. Only devices skipped as clean feed that minimum, and the skip is suppressed when a device needed rebuilding but had no readable source (missing_source), so the cursor never jumps past sectors that still need recovery. The generic bitmap skip_sync_blocks() path cannot be used here: it does not see mrdev/mreplace (the recovery target and its replacement), and replacement targets must rebuild even bitmap-clean chunks. On a mostly-clean array with 8T per device (near=3D2, 4 disks) the recovery sweep drops from 37.2s of CPU spinning to about 10ms. Signed-off-by: Yunye Zhao --- drivers/md/raid10.c | 46 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 54cddb3a98cd..8acbd5ef618b 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -3176,6 +3176,9 @@ static sector_t raid10_sync_request(struct mddev *mdd= ev, sector_t sector_nr, int i; int max_sync; sector_t sync_blocks; + sector_t min_sync_blocks =3D MaxSector; + sector_t sync_end; + bool missing_source =3D false; sector_t chunk_mask =3D conf->geo.chunk_mask; int page_idx =3D 0; =20 @@ -3258,6 +3261,14 @@ static sector_t raid10_sync_request(struct mddev *md= dev, sector_t sector_nr, if (max_sector > mddev->resync_max) max_sector =3D mddev->resync_max; /* Don't do IO beyond here */ =20 + /* + * The chunk clamp below caps a single sync I/O to at most one chunk. + * The bitmap-clean recovery skip issues no I/O, so remember the real + * end here to bound the skip against it rather than the chunk + * boundary. + */ + sync_end =3D max_sector; + /* make sure whole request will fit in a chunk - if chunks * are meaningful */ @@ -3334,9 +3345,13 @@ static sector_t raid10_sync_request(struct mddev *md= dev, sector_t sector_nr, if (!must_sync && mreplace =3D=3D NULL && !conf->fullsync) { - /* yep, skip the sync_blocks here, but don't assume - * that there will never be anything to do here + /* Skip the clean sync_blocks here; don't + * assume there will never be anything to + * do. Only a genuinely skipped clean span + * may widen the bulk skip below. */ + if (sync_blocks < min_sync_blocks) + min_sync_blocks =3D sync_blocks; continue; } if (mrdev) @@ -3459,6 +3474,7 @@ static sector_t raid10_sync_request(struct mddev *mdd= ev, sector_t sector_nr, if (j =3D=3D conf->copies) { /* Cannot recover, so abort the recovery or * record a bad block */ + missing_source =3D true; if (any_working) { /* problem is that there are bad blocks * on other device(s) @@ -3514,6 +3530,8 @@ static sector_t raid10_sync_request(struct mddev *mdd= ev, sector_t sector_nr, } } if (biolist =3D=3D NULL) { + sector_t skip_sectors =3D max_sync; + while (r10_bio) { struct r10bio *rb2 =3D r10_bio; r10_bio =3D (struct r10bio*) rb2->master_bio; @@ -3521,7 +3539,29 @@ static sector_t raid10_sync_request(struct mddev *md= dev, sector_t sector_nr, put_buf(rb2); } *skipped =3D 1; - return max_sync; + + /* + * min_sync_blocks is in array sectors, but the return + * value advances the per-device cursor; for a near + * layout the two spaces differ by the factor + * raid_disks / near_copies. + */ + if (!missing_source && + conf->geo.far_copies =3D=3D 1 && !conf->geo.far_offset && + conf->geo.raid_disks % conf->geo.near_copies =3D=3D 0 && + min_sync_blocks !=3D MaxSector) { + sector_t clean_sectors =3D min_sync_blocks; + + clean_sectors *=3D conf->geo.near_copies; + sector_div(clean_sectors, conf->geo.raid_disks); + clean_sectors &=3D ~chunk_mask; + if (clean_sectors > sync_end - sector_nr) + clean_sectors =3D sync_end - sector_nr; + if (clean_sectors > skip_sectors) + skip_sectors =3D clean_sectors; + } + + return skip_sectors; } } else { /* resync. Schedule a read for every block at this virt offset */ --=20 2.19.1.6.gb485710b