From nobody Sun Oct 5 03:35:16 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 83B8B243951; Sun, 10 Aug 2025 13:42:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833340; cv=none; b=k/iM4lF7ga/7oobH0k4hCvzQagNVh4Vb5oSuERg2B7igHk8mS5GcheyxDqPEXGAAxXI0VUmGFNVydbU2d83pc/936NsfbYQZ7I9GsQFtd6026mNFP4i5zsJ3/tAcEgUOmjAQKqXmU2JGfd3QEYerVrewKVvXqXTAY0GB466WvwQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833340; c=relaxed/simple; bh=lYfsbUws7d3sbPCB42F9HZxMcpJ3i7N9UNMxhnM35r0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=crlNlxznxAZ7D8vXuYBwsXrSZcaO4USVgb8Swko+szyYHXHp4VGY1YVRwSqe79aCl8e7meFDxKqgfR18+hF59Q9NCYQITkEo7q1c64vA74Dm9xJom9q4ZluxmfQiRtpp6g4tCil6DXwwELh3MPfXf9LyEmtEz9EDk5Qgc/aLyLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=CiwefkbK; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="CiwefkbK" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57ABFbDa014546; Sun, 10 Aug 2025 13:42:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=PYQZgvfNcc0kXpVhI ok6r8Bxs5AdIuo/c5/UZ0YX1lA=; b=CiwefkbKnaA5w8zi5D3e11MW9beSa2Dgk Y0j1kL/LhcyIs48FZNYMvhilCnCDLm5ySprm9pQtUkS5HDCITamnSqSAcSv6HjOP WiMzHkZ0izYnusr6EuuHITJzmlHfNni+3gx5k+oO2VHNrCSrpKG5BrA9b1+uTAE3 aB93UBPCv1DiNAanj8dAUoGG7An1DAw37BB5QfYBRjvx8t4gFmZE/X2d1BbOhpqX 1/AJs3nPT9wgnm9Gwy8svMCRIt1/JTQb+0HJzbLmXu9dBtXSYTBku+3eKB50FOn+ lJ76IvZNAcUlxwnKZuW20dkXiC/dQFcb2AdYdwJ6R7lo/iMxI6Dhg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx145b19-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:12 +0000 (GMT) Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgCva020423; Sun, 10 Aug 2025 13:42:12 GMT Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx145b16-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:12 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57ADJpNk028571; Sun, 10 Aug 2025 13:42:10 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48ej5msw94-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:10 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADg8Z234013722 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:08 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CA4DD20043; Sun, 10 Aug 2025 13:42:08 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8BB2220040; Sun, 10 Aug 2025 13:42:06 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:06 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 01/11] common/rc: Add _min() and _max() helpers Date: Sun, 10 Aug 2025 19:11:52 +0530 Message-ID: <43f45a0885f28fd1d1a88122a42830dd9eeb7e2c.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: lSQiEu_RxbLos2B7hWujceCHXI9EXwDd X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXxeFTlH+R9UkW JiPg0lpnrtdgWo1uJVxfeVvFPRbIKPytRKCi3CXZ5Dy1Yg/lHZ2TpWLMmiC/Om9TJpabamPBSH7 25QIB/YbM2utTwsYwDly28LwvpD4c0wyxxozQ3sxW0gC+7LBNqDCmrvBBlKacqPti9dkFWsgfEt lns1YGBKPNBmdaUzBRmWXznrKOxgmsuY1obOI4nZs1iFvb2wL3cTookRZs82iXTwv+XwEsX5SAi ja8GR96II/tQFtvVVtATQn02mQsMMbGTMatAjCay7zpNaK4YoaHVA6EuSUDqTQOOdgRc33AIxyW SAc/lxnpLHMmAolwFqvERtpWKw0Yw06DNpLslraztgvITbiUXO4h5ZWkFBUwN5igWboYc5ZoKrH 2ct2UavOVM9mL3n9C/rqogY//vxR1e/QVXOE6z/vRhQRwEfmk0kbV5OqACwzIaXJFRuZPEfp X-Proofpoint-GUID: EcMUEfIcDq_GYqRvgpS0OWAdp8Jaj-QY X-Authority-Analysis: v=2.4 cv=fLg53Yae c=1 sm=1 tr=0 ts=6898a1b4 cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=2OwXVqhp2XgA:10 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=wDx1mg2EYdB0JXDzLs4A:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 lowpriorityscore=0 clxscore=1015 spamscore=0 priorityscore=1501 impostorscore=0 phishscore=0 mlxlogscore=967 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" Many programs open code these functionalities so add it as a generic helper in common/rc Reviewed-by: Darrick J. Wong Signed-off-by: Ojaswin Mujoo --- common/rc | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/common/rc b/common/rc index 96578d15..3858ddce 100644 --- a/common/rc +++ b/common/rc @@ -5873,6 +5873,28 @@ _require_program() { _have_program "$1" || _notrun "$tag required" } =20 +_min() { + local ret + + for arg in "$@"; do + if [ -z "$ret" ] || (( $arg < $ret )); then + ret=3D"$arg" + fi + done + echo $ret +} + +_max() { + local ret + + for arg in "$@"; do + if [ -z "$ret" ] || (( $arg > $ret )); then + ret=3D"$arg" + fi + done + echo $ret +} + ##########################################################################= ###### # make sure this script returns success /bin/true --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78241263C9E; Sun, 10 Aug 2025 13:42:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833344; cv=none; b=Pyrse0xr63aqOS1td3AWytshOgNt5z+pZR7Mq2KbN5RvM5VaxVZC0tmu/ijInzyfuCupNezPr1A36GihMYfKmzR/mKjCXlcZpSXJTlrR2RiKKnHBg6OGiNSaipTjqK191t11YMXNYSIRkBG8GAm81SYVk40RggU6WDdI3Vwn0Go= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833344; c=relaxed/simple; bh=/+UX7YI8QuUtPa998750h8tQ3mN0imbXJivKjzTgFXM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bEQPrGtOWVuK1GFS7zFCAifbK5Wo3E7xwu3bvsQTkJZqWK8sQVgp1F2ZVvaCUEmkFPObF71rq1l9WZufnqGu/UM1HEvLfWUDoRLko+EfOMQRyVH++bH7G7QQsefO/GaqwGuShyXrWmy3tma4eQ6GRVxmGMUlnNqpzHVvL5zv/xY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=LcVvMYrZ; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="LcVvMYrZ" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57AAaUju016873; Sun, 10 Aug 2025 13:42:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=WHopWYB9IEVpbApCT umqAJkSAH7oesowCcUhN6x7VP0=; b=LcVvMYrZN0NXb6iU7LiIXb1+UNB6OA1Kx W/YCy8YHuBk20aVY1TZT5RFEPlxisK7/iiCl1s3PH6wey4ZW3fnmjymYxEihYj4D PquxkdSIRARrWZLVmouwcto1afvwnV2+Yt8grBzjLqTlEDgKpyRn5w3XN2YTZSY1 keDLJJlY6abz1IPGU/L/WHWkGdzka7J2jcuTC5fbav7nFeM/tE9m9m3GUCcrH3z1 I3WdP1CSxv7yfVCaR6zFOBaJWp+nc+R0IE2n2COm+3y4kwLxQNS8XgMoQIZxr4gd HkWLkLg5TINbxIvqgupwzOaMEEmaW8Km8WdLPwt2RxJ5A+aeCjVCg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjas-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:14 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgDfI006966; Sun, 10 Aug 2025 13:42:14 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjaq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:13 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A8ZwlQ025654; Sun, 10 Aug 2025 13:42:13 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ejvm1rp3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:12 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgBxA50659614 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:11 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4531820043; Sun, 10 Aug 2025 13:42:11 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 399B620040; Sun, 10 Aug 2025 13:42:09 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:08 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 02/11] common/rc: Add a helper to run fsx on a given file Date: Sun, 10 Aug 2025 19:11:53 +0530 Message-ID: <9af96fe74e29e3300c7f5861299bbe6d8f330625.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Hh8yT4mtuUqWwEQ6j9_YF8BzqVVBgsff X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfX5JxHkea0nGGs 4Jkq0zPtbf3ITyKaFeOSmnD4OlNdsGU6P0o5uAnSqkckPrS45XaCwkFXxvVVIWULGCICzGz/y6c UAuwj9BbunvEzH4rk3/KLaAVmqaYsD7bB395EVwjEXGEsxPdQ2/2XEYNxHrBwnx4WV6PSnA/EZf 1zcKGl5FNbIw182/OlTX/JXU+sHZRwVLm1MNrY1dZL7vT/zeAXnRRwmqQho7F7Fyts6a0t/cxE4 f6oMH6PY7IAguzYasFURWfdKvkntgHf+tHI1USQTTWYHZnn7yFfvA+clBZT/8YLKTeHFvVkfE9v 8GVpEkoegAIEomBb3yngUCmRVq0HjTDoBnCHFm6V10Ovx3TxdR5VyW7TV9f5JJxdLan8fmaEgvf We7Jcje2K7HDKE8FjZS5QAEQoAIJcgiRxcuF8bQKEyZ2WeSS7IFd38e4yxPpc27sneA1aHQm X-Authority-Analysis: v=2.4 cv=QtNe3Uyd c=1 sm=1 tr=0 ts=6898a1b6 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=u_VYNAqGXo6OSSAA:21 a=2OwXVqhp2XgA:10 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=EfcJlRrZmV05EoLhlkQA:9 X-Proofpoint-ORIG-GUID: tZR7uruh8wCFz5z0kV-MspDCpgBFn83M X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 mlxlogscore=961 impostorscore=0 suspectscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" Currently run_fsx is hardcoded to run on a file in $TEST_DIR. Add a helper _run_fsx_on_file so that we can run fsx on any given file including in $SCRATCH_MNT. Also, refactor _run_fsx to use this helper. No functional change is intended in this patch. Reviewed-by: Darrick J. Wong Signed-off-by: Ojaswin Mujoo --- common/rc | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/common/rc b/common/rc index 3858ddce..dc45bff6 100644 --- a/common/rc +++ b/common/rc @@ -5169,13 +5169,24 @@ _require_hugepage_fsx() _notrun "fsx binary does not support MADV_COLLAPSE" } =20 -_run_fsx() +_run_fsx_on_file() { + local testfile=3D$1 + shift + + if ! [ -f $testfile ] + then + echo "_run_fsx_on_file: $testfile doesn't exist. Creating" >> $seqres.fu= ll + touch $testfile + fi + echo "fsx $*" local args=3D`echo $@ | sed -e "s/ BSIZE / $bsize /g" -e "s/ PSIZE / $psi= ze /g"` - set -- $FSX_PROG $args $FSX_AVOID $TEST_DIR/junk + + set -- $FSX_PROG $args $FSX_AVOID $testfile + echo "$@" >>$seqres.full - rm -f $TEST_DIR/junk + rm -f $testfile "$@" 2>&1 | tee -a $seqres.full >$tmp.fsx local res=3D${PIPESTATUS[0]} if [ $res -ne 0 ]; then @@ -5187,6 +5198,12 @@ _run_fsx() return 0 } =20 +_run_fsx() +{ + _run_fsx_on_file $TEST_DIR/junk $@ + return $? +} + # Run fsx with -h(ugepage buffers). If we can't set up a hugepage then sk= ip # the test, but if any other error occurs then exit the test. _run_hugepage_fsx() { --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EC852267B90; Sun, 10 Aug 2025 13:42:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833346; cv=none; b=OZ5dZ5SamryKKkG6+RZw1O0Ejp1ud1OnjRdMvwsVYXCX+wHFN6PI+1FDa2Hy7x0grVTGflPca0Yyc/1o8QcCqkDGJjBCmSAL5o79elERM+e9OnfKRtj81jmGi+6QasCll/v5aNkg34OJKBdzl20u62MFtI/ff3U4PYECQtyS9A4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833346; c=relaxed/simple; bh=79qiw2SuzQ0NKfic8XwozhS2RgMiI+8hAc54bqoPj8c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HzP2gfCZNeI9DF3CkEfXjFmGbq9j2W5VACUhzaq+mfhjKuJQU87YBgeOCBNmtZIJ4tPTP2/bZFlxI8a10F9xozD7EgJ0r66rhHnksPwOrKJk5x17tlKGuz8ibcYF4wauC1VQzfBlZCx+y52QjabdhFnps0ag3And5TKIXbS9Hds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=cnGefzni; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="cnGefzni" Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A8FjHZ000957; Sun, 10 Aug 2025 13:42:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=qJnJXz1zUXhVLAvTx 8/OljTBRqYdDL4cPrJBW4g2sIo=; b=cnGefznirgigd+HltdssMutrHMK7VHDfm Lckx1uB0qhJ0at8LZ+18NzBhx9qSjTAg8yHYllThcKlGlmUjOkeH4WgjR8Hs07PU jlPEDXUQvKtre+w7iRYxpKqdigk6hNZIdc3V9S0t34hKo9UjtsvQ5B0zTrqNCwdC P1L6vA0tZ2fEqS2T0YS+rbxWLlk9Mo8rs9ZbsRCyFFZ4QTKWJNvDDL1luWtHig/M fQiS4gori9skCwqU7MqLuDMwJYXTiTNB/LhN5lVZ2x+Llke7iacoKp5rjPkDM2a1 FBbOJifrgrxcl0nTYNg3l+HnIvImdy07vESqOww692A/9+ndlRyeQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48eha9ssdy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:17 +0000 (GMT) Received: from m0360072.ppops.net (m0360072.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgGXM023891; Sun, 10 Aug 2025 13:42:16 GMT Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48eha9ssdw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:16 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57AD2skc020615; Sun, 10 Aug 2025 13:42:16 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48ehnphyyt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:16 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgEA954264164 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:14 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0C8F12004B; Sun, 10 Aug 2025 13:42:14 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A92FC20043; Sun, 10 Aug 2025 13:42:11 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:11 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 03/11] ltp/fsx.c: Add atomic writes support to fsx Date: Sun, 10 Aug 2025 19:11:54 +0530 Message-ID: <8b3c42eb321b4a1a4850b7e76d53191cb20ffa41.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=KPRaDEFo c=1 sm=1 tr=0 ts=6898a1b9 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=T5_vRgAD47iBWrMonTEA:9 X-Proofpoint-ORIG-GUID: hiTZ9WUWmqOzPGQDaYLIzb7kPPfQ-Lmx X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXx02IJk0UmYf6 vdvA5t53CDyuuNaCrdyccd+SKTredkfE087P5iYmqrI1Rnqdcgrc4JAe9qC1W5TidET4bFGO9YX dL45xCM8egX3wHanKREXBQRB08Gl4j2F78zD72zL/y7FbOY+/TfvoacslOccm6vrJ+GQtU015u8 3FGt3Ehwi26sfI+OfZYVD1TKUsj32mcVf2YY0qLrT8n5oOym9NIkxPULEo2+/VvTFGkqjQQPMAJ 2C4+zh0SKTB0gYzpNC8887K47qPob0ylqoL8tKXlsSDv1CTyVHtvW9sqLsrksaTNtpg1UMV4L5o SvcVcDie2Dn5YCepyJv2McmSX78xOGa+yVq4Hv/k7NOULNv2zyo4q4E2C++eIgJx8tBHcrz04ql N8hx9jqrWLi/QmKwRLKlN/4q/ATYXU7OOIQVYIXH//MhXh/d1O7EX1hqCKhj9SK3rPBdtpJP X-Proofpoint-GUID: 37sJOD_7IDZNY1Hdd11t9L3Is5yFJW1z X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 impostorscore=0 bulkscore=0 mlxlogscore=999 spamscore=0 clxscore=1015 adultscore=0 priorityscore=1501 malwarescore=0 lowpriorityscore=0 mlxscore=0 suspectscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" Implement atomic write support to help fuzz atomic writes with fsx. Suggested-by: Ritesh Harjani (IBM) Reviewed-by: Darrick J. Wong Signed-off-by: Ojaswin Mujoo --- ltp/fsx.c | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 104 insertions(+), 5 deletions(-) diff --git a/ltp/fsx.c b/ltp/fsx.c index 163b9453..ea39ca29 100644 --- a/ltp/fsx.c +++ b/ltp/fsx.c @@ -40,6 +40,7 @@ #include #endif #include +#include "statx.h" =20 #ifndef MAP_FILE # define MAP_FILE 0 @@ -49,6 +50,10 @@ #define RWF_DONTCACHE 0x80 #endif =20 +#ifndef RWF_ATOMIC +#define RWF_ATOMIC 0x40 +#endif + #define NUMPRINTCOLUMNS 32 /* # columns of data to print on each line */ =20 /* Operation flags (bitmask) */ @@ -110,6 +115,7 @@ enum { OP_READ_DONTCACHE, OP_WRITE, OP_WRITE_DONTCACHE, + OP_WRITE_ATOMIC, OP_MAPREAD, OP_MAPWRITE, OP_MAX_LITE, @@ -200,6 +206,11 @@ int uring =3D 0; int mark_nr =3D 0; int dontcache_io =3D 1; int hugepages =3D 0; /* -h flag */ +int do_atomic_writes =3D 1; /* -a flag disables */ + +/* User for atomic writes */ +int awu_min =3D 0; +int awu_max =3D 0; =20 /* Stores info needed to periodically collapse hugepages */ struct hugepages_collapse_info { @@ -288,6 +299,7 @@ static const char *op_names[] =3D { [OP_READ_DONTCACHE] =3D "read_dontcache", [OP_WRITE] =3D "write", [OP_WRITE_DONTCACHE] =3D "write_dontcache", + [OP_WRITE_ATOMIC] =3D "write_atomic", [OP_MAPREAD] =3D "mapread", [OP_MAPWRITE] =3D "mapwrite", [OP_TRUNCATE] =3D "truncate", @@ -422,6 +434,7 @@ logdump(void) prt("\t***RRRR***"); break; case OP_WRITE_DONTCACHE: + case OP_WRITE_ATOMIC: case OP_WRITE: prt("WRITE 0x%x thru 0x%x\t(0x%x bytes)", lp->args[0], lp->args[0] + lp->args[1] - 1, @@ -1073,6 +1086,25 @@ update_file_size(unsigned offset, unsigned size) file_size =3D offset + size; } =20 +static int is_power_of_2(unsigned n) { + return ((n & (n - 1)) =3D=3D 0); +} + +/* + * Round down n to nearest power of 2. + * If n is already a power of 2, return n; + */ +static int rounddown_pow_of_2(int n) { + int i =3D 0; + + if (is_power_of_2(n)) + return n; + + for (; (1 << i) < n; i++); + + return 1 << (i - 1); +} + void dowrite(unsigned offset, unsigned size, int flags) { @@ -1081,6 +1113,27 @@ dowrite(unsigned offset, unsigned size, int flags) offset -=3D offset % writebdy; if (o_direct) size -=3D size % writebdy; + if (flags & RWF_ATOMIC) { + /* atomic write len must be inbetween awu_min and awu_max */ + if (size < awu_min) + size =3D awu_min; + if (size > awu_max) + size =3D awu_max; + + /* atomic writes need power-of-2 sizes */ + size =3D rounddown_pow_of_2(size); + + /* atomic writes need naturally aligned offsets */ + offset -=3D offset % size; + + /* Skip the write if we are crossing max filesize */ + if ((offset + size) > maxfilelen) { + if (!quiet && testcalls > simulatedopcount) + prt("skipping atomic write past maxfilelen\n"); + log4(OP_WRITE_ATOMIC, offset, size, FL_SKIPPED); + return; + } + } if (size =3D=3D 0) { if (!quiet && testcalls > simulatedopcount && !o_direct) prt("skipping zero size write\n"); @@ -1088,7 +1141,10 @@ dowrite(unsigned offset, unsigned size, int flags) return; } =20 - log4(OP_WRITE, offset, size, FL_NONE); + if (flags & RWF_ATOMIC) + log4(OP_WRITE_ATOMIC, offset, size, FL_NONE); + else + log4(OP_WRITE, offset, size, FL_NONE); =20 gendata(original_buf, good_buf, offset, size); if (offset + size > file_size) { @@ -1108,8 +1164,9 @@ dowrite(unsigned offset, unsigned size, int flags) (monitorstart =3D=3D -1 || (offset + size > monitorstart && (monitorend =3D=3D -1 || offset <=3D monitorend)))))) - prt("%lld write\t0x%x thru\t0x%x\t(0x%x bytes)\tdontcache=3D%d\n", testc= alls, - offset, offset + size - 1, size, (flags & RWF_DONTCACHE) !=3D 0); + prt("%lld write\t0x%x thru\t0x%x\t(0x%x bytes)\tdontcache=3D%d atomic_wr= =3D%d\n", testcalls, + offset, offset + size - 1, size, (flags & RWF_DONTCACHE) !=3D 0, + (flags & RWF_ATOMIC) !=3D 0); iret =3D fsxwrite(fd, good_buf + offset, size, offset, flags); if (iret !=3D size) { if (iret =3D=3D -1) @@ -1785,6 +1842,30 @@ do_dedupe_range(unsigned offset, unsigned length, un= signed dest) } #endif =20 +int test_atomic_writes(void) { + int ret; + struct statx stx; + + ret =3D xfstests_statx(AT_FDCWD, fname, 0, STATX_WRITE_ATOMIC, &stx); + if (ret < 0) { + fprintf(stderr, "main: Statx failed with %d." + " Failed to determine atomic write limits, " + " disabling!\n", ret); + return 0; + } + + if (stx.stx_attributes & STATX_ATTR_WRITE_ATOMIC && + stx.stx_atomic_write_unit_min > 0) { + awu_min =3D stx.stx_atomic_write_unit_min; + awu_max =3D stx.stx_atomic_write_unit_max; + return 1; + } + + fprintf(stderr, "main: IO Stack does not support " + "atomic writes, disabling!\n"); + return 0; +} + #ifdef HAVE_COPY_FILE_RANGE int test_copy_range(void) @@ -2356,6 +2437,12 @@ have_op: goto out; } break; + case OP_WRITE_ATOMIC: + if (!do_atomic_writes) { + log4(OP_WRITE_ATOMIC, offset, size, FL_SKIPPED); + goto out; + } + break; } =20 switch (op) { @@ -2385,6 +2472,11 @@ have_op: dowrite(offset, size, 0); break; =20 + case OP_WRITE_ATOMIC: + TRIM_OFF_LEN(offset, size, maxfilelen); + dowrite(offset, size, RWF_ATOMIC); + break; + case OP_MAPREAD: TRIM_OFF_LEN(offset, size, file_size); domapread(offset, size); @@ -2511,13 +2603,14 @@ void usage(void) { fprintf(stdout, "usage: %s", - "fsx [-dfhknqxyzBEFHIJKLORWXZ0]\n\ + "fsx [-adfhknqxyzBEFHIJKLORWXZ0]\n\ [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid]\n\ [-l flen] [-m start:end] [-o oplen] [-p progressinterval]\n\ [-r readbdy] [-s style] [-t truncbdy] [-w writebdy]\n\ [-A|-U] [-D startingop] [-N numops] [-P dirpath] [-S seed]\n\ [--replay-ops=3Dopsfile] [--record-ops[=3Dopsfile]] [--duration=3Dseco= nds]\n\ ... fname\n\ + -a: disable atomic writes\n\ -b opnum: beginning operation number (default 1)\n\ -c P: 1 in P chance of file close+open at each op (default infinity)\n\ -d: debug output for all operations\n\ @@ -3059,9 +3152,13 @@ main(int argc, char **argv) setvbuf(stdout, (char *)0, _IOLBF, 0); /* line buffered stdout */ =20 while ((ch =3D getopt_long(argc, argv, - "0b:c:de:fg:hi:j:kl:m:no:p:qr:s:t:uw:xyABD:EFJKHzCILN:OP:RS:UWXZ", + "0ab:c:de:fg:hi:j:kl:m:no:p:qr:s:t:uw:xyABD:EFJKHzCILN:OP:RS:UWXZ", longopts, NULL)) !=3D EOF) switch (ch) { + case 'a': + prt("main(): Atomic writes disabled\n"); + do_atomic_writes =3D 0; + break; case 'b': simulatedopcount =3D getnum(optarg, &endp); if (!quiet) @@ -3475,6 +3572,8 @@ main(int argc, char **argv) exchange_range_calls =3D test_exchange_range(); if (dontcache_io) dontcache_io =3D test_dontcache_io(); + if (do_atomic_writes) + do_atomic_writes =3D test_atomic_writes(); =20 while (keep_running()) if (!test()) --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E472126B0A9; Sun, 10 Aug 2025 13:42:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833348; cv=none; b=WrnQwUTADe42CfMzbnlHc+O1wjuIIg2o+0Me8szXzp9O8Mw6OJ77BVSPZ8pARnfScCW3OVPVtK2FiYGbdSFjYrMDdKN26GVC1BsIQWHjjq9gUE/nnIDFOBUQ/buY4VxkR2XuY47vkbX95DECzSG8BNNPxVHEbaI6h0uer3FFFcE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833348; c=relaxed/simple; bh=+qeVIYKnwwz47UXPhCGvi3XbtVygUZqBgYxQmHU1pW4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Fmig8bxEQAU1ZTeDOkacFWIq8uuUC3/htsOWrnP0tfpjZK0YtwcuOGpQpr1A8uAaZqhSVXpKKgl2davHDQVQukYgFpjPqDgDhMa+f0QWB/1VA2qS/7eVW1uczMQ3oJKzfvf4dXc0j2pbor18eCnH6mJIQf8Cjs0PD3qVZkRC55Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=lD9bpZVQ; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="lD9bpZVQ" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A5KRwJ031106; Sun, 10 Aug 2025 13:42:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=J32fgvwaJN5e1OPuI SrC9tvkTy13J/GzTIvPrXn0Pco=; b=lD9bpZVQfbcq7Wx4iyIVSyomgaNrHxJfT WLO3ZpArzjp8BK2mLuJM4trvjHDa7mUK3NBFf5gpEmXTweZFScYyfcdBvWo+gEFT UtcZknKfmFHny8Qvb+R1rmwUETFly0+nB+0sgFag+LZD6IARqbpoGzJdP9jPL/jJ Jh2KxeyjoK4ofxfu1ecMZzHIUYd7IaeN0YOduJ0rur9uwORS5xysxEgJIo/f4Z7B g+Wu0accdzEFd8NdTDPNagFDrYXgT3G7dCNPgtjobiuVxRsjhDDtRcrJGtaKvDDj vM4ugRtYVZHYkx8UJOwBktf4GSfhKqxC9TlTD7NSDjrxzQM7gTxfQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjax-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:19 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgJmP006980; Sun, 10 Aug 2025 13:42:19 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjav-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:19 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A963ol017606; Sun, 10 Aug 2025 13:42:18 GMT Received: from smtprelay07.fra02v.mail.ibm.com ([9.218.2.229]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ekc39nee-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:18 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay07.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgGOg44826942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:16 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7BE8520040; Sun, 10 Aug 2025 13:42:16 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6F85820043; Sun, 10 Aug 2025 13:42:14 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:14 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 04/11] generic: Add atomic write test using fio crc check verifier Date: Sun, 10 Aug 2025 19:11:55 +0530 Message-ID: <783e950d8b5ad80672a359a19ede4faeb64e3dd7.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: CNIrG4HopbdaEGioj8F1BTg1Qv_yxp-K X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXwlxz97p5tDGF M+cfN9djfMEGu2mX3FRFuqv/COspOMGQelmdRCm2saK1bU9ZLL9+h5NbLcmaOOpdcAUeRDxThHb wU0iJl/moKHZFAGHo0PLaWa0kuRz3PLOQDkFRnecQaHvSvZPK9eVhbCZqVl+smCcKxSnX7U/LVy 0fEBc5ZKNC8FT3NwNxcGXXImFJm4R/fVQrY2PWaKByDjV3RV6rL/MlZw8zvncVKyibOY0lOQ9IY PGaJexh7Rt2jS+sYLX96IJBYADPiB5VY+cXbEStoQ43JbS8ixA+EZzYjmsQnEkz7mkCaYM3qo0H KXRoZpxEBCuGmO53ORW/4z3hB9JXUfOxpLUxj2UYCkTxu1KUJFE8Id74hllI+2SOM5xwrYS1w+R Vt5cktu5x/GedMowSHlFvTA2gJ6eadR8MG9CJMQI1rOUE23sC8/HvkPEBtu5pbPeToxh10ZU X-Authority-Analysis: v=2.4 cv=QtNe3Uyd c=1 sm=1 tr=0 ts=6898a1bb cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VnNF1IyMAAAA:8 a=TDUsxBfbQoAVH7yvDjsA:9 X-Proofpoint-ORIG-GUID: vsoPwVvazyk1EMLXzeKiEm58YMB7VB23 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 suspectscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" This adds atomic write test using fio based on it's crc check verifier. fio adds a crc for each data block. If the underlying device supports atomic write then it is guaranteed that we will never have a mix data from two threads writing on the same physical block. Avoid doing overlapping parallel atomic writes because it might give unexpected results. Use offset_increment=3D, size=3D fio options to achieve this behavior. Co-developed-by: Ritesh Harjani (IBM) Signed-off-by: Ritesh Harjani (IBM) Signed-off-by: Ojaswin Mujoo Reviewed-by: "Darrick J. Wong" --- tests/generic/1226 | 107 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1226.out | 2 + 2 files changed, 109 insertions(+) create mode 100755 tests/generic/1226 create mode 100644 tests/generic/1226.out diff --git a/tests/generic/1226 b/tests/generic/1226 new file mode 100755 index 00000000..efc360e1 --- /dev/null +++ b/tests/generic/1226 @@ -0,0 +1,107 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1226 +# +# Validate FS atomic write using fio crc check verifier. +# +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto aio rw atomicwrites + +_require_scratch_write_atomic +_require_odirect +_require_aio + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount +_require_xfs_io_command "falloc" + +touch "$SCRATCH_MNT/f1" +awu_min_write=3D$(_get_atomic_write_unit_min "$SCRATCH_MNT/f1") +awu_max_write=3D$(_get_atomic_write_unit_max "$SCRATCH_MNT/f1") + +blocksize=3D$(_max "$awu_min_write" "$((awu_max_write/2))") +threads=3D$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100") +filesize=3D$((blocksize * threads * 100)) +depth=3D$threads +io_size=3D$((filesize / threads)) +io_inc=3D$io_size +testfile=3D$SCRATCH_MNT/test-file + +fio_config=3D$tmp.fio +fio_out=3D$tmp.fio.out + +fio_aw_config=3D$tmp.aw.fio +fio_verify_config=3D$tmp.verify.fio + +function create_fio_configs() +{ + create_fio_aw_config + create_fio_verify_config +} + +function create_fio_verify_config() +{ +cat >$fio_verify_config <$fio_aw_config <> $seqres.full +cat $fio_verify_config >> $seqres.full + +$XFS_IO_PROG -fc "falloc 0 $filesize" $testfile >> $seqres.full + +$FIO_PROG $fio_aw_config >> $seqres.full +ret1=3D$? +$FIO_PROG $fio_verify_config >> $seqres.full +ret2=3D$? + +[[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write failed" + +# success, all done +echo Silence is golden +status=3D0 +exit diff --git a/tests/generic/1226.out b/tests/generic/1226.out new file mode 100644 index 00000000..6dce0ea5 --- /dev/null +++ b/tests/generic/1226.out @@ -0,0 +1,2 @@ +QA output created by 1226 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE8C326D4D4; Sun, 10 Aug 2025 13:42:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833350; cv=none; b=YmAPq0Pra9OzCZDwd2QD6c/s11TNVNazCOj86DwVPVMCU4+1RfgWNiyLlvPDHSrTK7OToNG18uI6PCj52qgDauHIWA5kh6pfl75NFOLzf7M93hSdB/ZPAjJ99M7sY9qYjDzMorABizaTN7yehYuIetsxGDd55mldi+oSbjFoyKU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833350; c=relaxed/simple; bh=gNroVsPMuCxy0m+6hMmmrV8t4DGNlzsl+eGzKh1/4J4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EaMShcQaxz+u5vqwXT8oqfBH8dYw0ft5J8qvhibn6RvDaN4q/fQBpsiWjgUSpo89zzq860za89CAMTTIlauAzZkzIIgaSyoeu/Wn229JtRff6HoJRIJSCWbWeY6vONqzVuiRUei4Br7WnXnLw98RD/BHUyBAHmv7KzXZHr7yBDs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=OulLKpaV; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="OulLKpaV" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A9LVDe021434; Sun, 10 Aug 2025 13:42:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=0B3hJJmho2YHLFXzV e2sFYTqjmwhGSM+TyAncg1vENY=; b=OulLKpaVxnspNd9GRxbc1i51EhVUu/lzG M0JexYkqGv5SdBb4m6KC2j9OrGB/axXisMac9gRXqnHyWa+gPHvVSRCE5/EkA3qV vsPFHbWVb31G2JnLdLz/+kFjXBwXIdRm0CJXhmrSGxDXtjgT0OsDGEkugH9Je++u +VFii4lqhi56yWBDwWh0hmgJjCxuOxrvjwoPCys1cLTk+nnlIzq2dKvA8Q/VahAZ q05Z/DcJTGm8oFbTCKPgSVmrn85y5H+GqOq0OrXmO1wJjdKTMFy23uhlQJguRFHq ZUVf7QEEPb5/+Sjcxm2uX/Fca+eMyCUzY/KsE+zMbOEXKP9Tz/lmg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjb1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:22 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgLDn007055; Sun, 10 Aug 2025 13:42:21 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjay-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:21 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A946Tf017594; Sun, 10 Aug 2025 13:42:20 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ekc39nem-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:20 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgJ2q58458440 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:19 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 456622004F; Sun, 10 Aug 2025 13:42:19 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DF18C20040; Sun, 10 Aug 2025 13:42:16 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:16 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 05/11] generic: Add atomic write test using fio verify on file mixed mappings Date: Sun, 10 Aug 2025 19:11:56 +0530 Message-ID: <508d55ad8e3b8efde87ffbe3354e9e1d9ee8c908.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 6nkjJfpfydRyso1WUAEKta8R09VJ7QXI X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXwLXx7cp9tQzH eetFrhQbCf4amPLV052LcsI1ZH2yxAKKSvD0lUbabUxPvRcSfUEpq/NdizBjIt2DxcZ6wFnsMqH 5jaDCS5aIUkKGMWtcamANURBmZM7Z5p1oKnytstF4CowQUjI1HtPpXxHsRweKvjj9pwk9xXTeUh IKCIWuwjwDapt2ZkCTW/VxbHsEBPrZZqEJxwBK5MtJFo6rKfiw3pmtdKYtykvf042vb1m3Xl82P EYEE9nkwG+6nDAMSO6nXF8ZKXnVAcgbCJQPL44M2o2YOEygRhaE7UxkQxrOfA3ZafYwOMXPHR1o n7PBmp4i4lGUjmf26A7EU8EL4517FIMCa7LsWBdKvhmzcjixJFKLYBK7ahCsvhV6QYVtX27SAiK Hhsvqoilp/v02SRnEBdiBiOCCc1TEN11moDrM0c5rjwZf0G4DDfG58ot7ecoh3K5ibZuVQrh X-Authority-Analysis: v=2.4 cv=QtNe3Uyd c=1 sm=1 tr=0 ts=6898a1be cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VnNF1IyMAAAA:8 a=ZzjZKRc1KEgCYT2eLxwA:9 X-Proofpoint-ORIG-GUID: 2TyRVTReJBJCdTm-Ok1MgeBMLERT0JtK X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 suspectscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" This tests uses fio to first create a file with mixed mappings. Then it does atomic writes using aio dio with parallel jobs to the same file with mixed mappings. This forces the filesystem allocator to allocate extents over mixed mapping regions to stress FS block allocators. Avoid doing overlapping parallel atomic writes because it might give unexpected results. Use offset_increment=3D, size=3D fio options to achieve this behavior. Co-developed-by: Ritesh Harjani (IBM) Signed-off-by: Ritesh Harjani (IBM) Signed-off-by: Ojaswin Mujoo Reviewed-by: "Darrick J. Wong" --- tests/generic/1227 | 131 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1227.out | 2 + 2 files changed, 133 insertions(+) create mode 100755 tests/generic/1227 create mode 100644 tests/generic/1227.out diff --git a/tests/generic/1227 b/tests/generic/1227 new file mode 100755 index 00000000..7423e67c --- /dev/null +++ b/tests/generic/1227 @@ -0,0 +1,131 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1227 +# +# Validate FS atomic write using fio crc check verifier on mixed mappings +# of a file. +# +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto aio rw atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_odirect +_require_aio +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +touch "$SCRATCH_MNT/f1" +awu_min_write=3D$(_get_atomic_write_unit_min "$SCRATCH_MNT/f1") +awu_max_write=3D$(_get_atomic_write_unit_max "$SCRATCH_MNT/f1") + +aw_bsize=3D$(_max "$awu_min_write" "$((awu_max_write/4))") +fsbsize=3D$(_get_block_size $SCRATCH_MNT) + +threads=3D$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100") +filesize=3D$((aw_bsize * threads * 100)) +depth=3D$threads +aw_io_size=3D$((filesize / threads)) +aw_io_inc=3D$aw_io_size +testfile=3D$SCRATCH_MNT/test-file + +fio_prep_config=3D$tmp.prep.fio +fio_aw_config=3D$tmp.aw.fio +fio_verify_config=3D$tmp.verify.fio +fio_out=3D$tmp.fio.out + +cat >$fio_prep_config <$fio_aw_config <$fio_verify_config <> $seqres.full +cat $fio_aw_config >> $seqres.full +cat $fio_verify_config >> $seqres.full + +$XFS_IO_PROG -fc "truncate $filesize" $testfile >> $seqres.full + +#prepare file with mixed mappings +$FIO_PROG $fio_prep_config >> $seqres.full + +# do atomic writes without verifying +$FIO_PROG $fio_aw_config >> $seqres.full + +# verify data is not torn +$FIO_PROG $fio_verify_config >> $seqres.full + +# success, all done +echo Silence is golden +status=3D0 +exit diff --git a/tests/generic/1227.out b/tests/generic/1227.out new file mode 100644 index 00000000..2605d062 --- /dev/null +++ b/tests/generic/1227.out @@ -0,0 +1,2 @@ +QA output created by 1227 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 13983270EC3; Sun, 10 Aug 2025 13:42:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833354; cv=none; b=sQnoGEe/A10Y1qkpRSPNGHPN6i+AZolE7lJcrdn0sg2SooDr5JwNcdvaCfpsEJ7GsMe1lFaU3Qi4fp6ckh75WAJ5SJwz+18VWy6+v5YAn4tRSpBtXVWXowE+eW1MblcmtuXGGaKs9dgTLTZ9wZQUY4mcmFUwIZcJ6WXR/qaZ/to= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833354; c=relaxed/simple; bh=sytgdqcLbEYSb1CR1JneiuQwbBB4t3W8xsdq/VdKKMA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ORs3FeLtk8D8bvngQkRhxqpYa4i4u+IqanlLwevFi8tFCqTvn4dQWKUHpUU/Ga9Jh1PpODeHdHHN87wyJOfanIySTCPAsobEbZlgAJGRlOiMA8dnm7vCwOsqGZXluoU9nMs+Pcl5XeKKveIH7YMpGNJZhHBz3q4RTFE/TMY+TDM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=EkZSWOdR; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="EkZSWOdR" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57ABbDaG019717; Sun, 10 Aug 2025 13:42:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=M9+Rq+m14A0N34e0u DjaBhsdDMEAWYACoan5D5g/ylM=; b=EkZSWOdRPP2XtDyophCeA7aQ/L9zrGq+X 8DpLeZJsWfd++0FL9OMV8o6ZqzPIejTkhPui5XlZN/E4g9Nckw/HYssSOCGiANmj 6RjURSBdK6YvC/lkufRDqylDg2fo8T7rQgWCf8JNPH2gJPEixGnT8m9udVZ+1/aY jc/NLj1KbLIhWWjzxu0aD9wS6AlfZEDJ5vS9gYh8aRsF/FRF2mW0BDJoxWK5CqBR xiPSUdgnMfc2pWH83b6CYxqc/c4Jr6iC2Qg8lWnudoTWjN1D4zPRV54umtM6ZKl+ cG7MuueOFrGik45+i2b8hTvfYL5pnEmfX52gtf0WrNuoTEvRP+4eA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx2cnb4h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:25 +0000 (GMT) Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADcl4I005415; Sun, 10 Aug 2025 13:42:24 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx2cnb4f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:24 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A97jn7017588; Sun, 10 Aug 2025 13:42:23 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ekc39nes-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:23 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgMUh22479118 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:22 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E3EB620043; Sun, 10 Aug 2025 13:42:21 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A5BCA20040; Sun, 10 Aug 2025 13:42:19 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:19 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 06/11] generic: Add atomic write multi-fsblock O_[D]SYNC tests Date: Sun, 10 Aug 2025 19:11:57 +0530 Message-ID: X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfX6BB6ZWWthyeM +OZOSZmi3JaYvFoXbUlPbtxEMThn91Uch92rglGERe0NZCIKlbPAq2vybvFGGuOqoyw5/oZ9JWm W2V2JkVF3PsgZQ8F+W894mo9XZCSjuMwpDfc+WPTAi3kHrX973lsE1ktm9xqq1WJRKTLA+kv1nX 09ShwrGPvv+8bB9zSpK/i70cIfVgwtZOkXVIdG348q3tcdsDRnRiYXCs7Nr8/Muxba3bB9sbgbz HIjhv+ey5csbirFLpc3YLi1Ic1XPLCFZKzk6T+/I6YDXTf1VjlvaP0bUVTqU7ZcVU7bYaSaKxb1 zj9FZiaWrhA7qTMh6Q1J5OCxQ1sY/7hrzUYN2fQvJqqNFZj5g+C7Bj9xzHy8LBHhUzRtzUzLICV Du4of5fTxzs8bLLx1ypeeS4q16LUhFsbPXCpRH5nnr9Ku93ysxEjEF+zza9joBdWUEQdJuqW X-Proofpoint-GUID: Eb6OzDRH6xBFOwUPqBEwwO8w2pFIfNt7 X-Authority-Analysis: v=2.4 cv=C9zpyRP+ c=1 sm=1 tr=0 ts=6898a1c1 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VnNF1IyMAAAA:8 a=udGWdZ-Mnbz566dSa5EA:9 X-Proofpoint-ORIG-GUID: vDqJRWPpH4v_bOdZmyqv2HqQI1NdnjVa X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 impostorscore=0 clxscore=1015 malwarescore=0 spamscore=0 priorityscore=1501 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" This adds various atomic write multi-fsblock stresst tests with mixed mappings and O_SYNC, to ensure the data and metadata is atomically persisted even if there is a shutdown. Suggested-by: Ritesh Harjani (IBM) Signed-off-by: Ojaswin Mujoo Reviewed-by: "Darrick J. Wong" --- tests/generic/1228 | 137 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1228.out | 2 + 2 files changed, 139 insertions(+) create mode 100755 tests/generic/1228 create mode 100644 tests/generic/1228.out diff --git a/tests/generic/1228 b/tests/generic/1228 new file mode 100755 index 00000000..888599ce --- /dev/null +++ b/tests/generic/1228 @@ -0,0 +1,137 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1228 +# +# Atomic write multi-fsblock data integrity tests with mixed mappings +# and O_SYNC +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto quick rw atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full + +check_data_integrity() { + actual=3D$(_hexdump $testfile) + if [[ "$expected" !=3D "$actual" ]] + then + echo "Integrity check failed" + echo "Integrity check failed" >> $seqres.full + echo "# Expected file contents:" >> $seqres.full + echo "$expected" >> $seqres.full + echo "# Actual file contents:" >> $seqres.full + echo "$actual" >> $seqres.full + + _fail "Data integrity check failed. The atomic write was torn." + fi +} + +prep_mixed_mapping() { + $XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full + local off=3D0 + local mapping=3D"" + + local operations=3D("W" "H" "U") + local num_blocks=3D$((awu_max / blksz)) + for ((i=3D0; i /d= ev/null + ;; + "H") + # No operation needed for hole + ;; + "U") + $XFS_IO_PROG -c "falloc $off $blksz" $testfile >> /dev/null + ;; + esac + off=3D$((off + blksz)) + done + + echo "+ + Mixed mapping prep done. Full mapping pattern: $mapping" >> $se= qres.full + + sync $testfile +} + +verify_atomic_write() { + test $bytes_written -eq $awu_max || _fail "atomic write len=3D$awu_max as= sertion failed" + check_data_integrity +} + +mixed_mapping_test() { + prep_mixed_mapping + + echo -"+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres= .full + if [[ "$1" =3D=3D "shutdown" ]] + then + bytes_written=3D$($XFS_IO_PROG -x -dc \ + "pwrite -DA -V1 -b $awu_max 0 $awu_max" \ + -c "shutdown" $testfile | grep wrote | \ + awk -F'[/ ]' '{print $2}') + _scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed" + else + bytes_written=3D$($XFS_IO_PROG -dc \ + "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \ + grep wrote | awk -F'[/ ]' '{print $2}') + fi + + verify_atomic_write +} + +testfile=3D$SCRATCH_MNT/testfile +touch $testfile + +awu_max=3D$(_get_atomic_write_unit_max $testfile) +blksz=3D$(_get_block_size $SCRATCH_MNT) + +# Create an expected pattern to compare with +$XFS_IO_PROG -tc "pwrite -b $awu_max 0 $awu_max" $testfile >> $seqres.full +expected=3D$(_hexdump $testfile) +echo "# Expected file contents:" >> $seqres.full +echo "$expected" >> $seqres.full +echo >> $seqres.full + +echo "# Test 1: Do O_DSYNC atomic write on random mixed mapping:" >> $seqr= es.full +echo >> $seqres.full +for ((iteration=3D1; iteration<=3D10; iteration++)); do + echo "=3D=3D=3D Mixed Mapping Test Iteration $iteration =3D=3D=3D" >> $se= qres.full + + echo "+ Testing without shutdown..." >> $seqres.full + mixed_mapping_test + echo "Passed!" >> $seqres.full + + echo "+ Testing with sudden shutdown..." >> $seqres.full + mixed_mapping_test "shutdown" + echo "Passed!" >> $seqres.full + + echo "Iteration $iteration completed: OK" >> $seqres.full + echo >> $seqres.full +done +echo "# Test 1: Do O_SYNC atomic write on random mixed mapping (10 iterati= ons): OK" >> $seqres.full + + +echo >> $seqres.full +echo "# Test 2: Do extending O_SYNC atomic writes: " >> $seqres.full +bytes_written=3D$($XFS_IO_PROG -x -dstc "pwrite -A -V1 -b $awu_max 0 $awu_= max" \ + -c "shutdown" $testfile | grep wrote | awk -F'[/ ]' '{print $2}') +_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed" +verify_atomic_write +echo "# Test 2: Do extending O_SYNC atomic writes: OK" >> $seqres.full + +# success, all done +echo "Silence is golden" +status=3D0 +exit + diff --git a/tests/generic/1228.out b/tests/generic/1228.out new file mode 100644 index 00000000..1baffa91 --- /dev/null +++ b/tests/generic/1228.out @@ -0,0 +1,2 @@ +QA output created by 1228 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 407CB271A7B; Sun, 10 Aug 2025 13:42:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833355; cv=none; b=ZQ8CAJ206Af5hoZSPxgapQS0go6w5Mg9c34WSjXQgquy+gi70I/cVD4I7hwQ2hQTaXwzNJXPvdVxLMqSSyIGd1/gSki0qh5UDSTo8RnoPAEvO6hbqExXYmQqlK+hqTNzrH31/zkZoByEbFOWr/QKgBAtEVS0srlyS0ZgjA5cyYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833355; c=relaxed/simple; bh=uPasESbOKX5ySA2wW6RZodhKnEOlPArEeggRaHry10k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lvGh5b0WLlb+J2jM/1+/WcLAeuW/YkcCT6HKzqngWtgztKUxnQIQA7/7QnfXa00CVGcT9/SSmEuAwFp7ZW0f6JTdwammXnTw/lPLypXMDfFfuYyMUPhHxwQOm9kAPNpG8xbRphLlhi2bmE9ojpn8jWUMRlJmy9cUCZPlXtAnMC0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=hLAwhhcV; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="hLAwhhcV" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A9pqla008903; Sun, 10 Aug 2025 13:42:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=7AbQG57LSa3FbHWfM NrAfby5XX2AiWzch8fpH1rzmHo=; b=hLAwhhcVHo5Gdz1V9F9i3IA4P7LDe3ezV wY8cb5GRUVcV/GfDESFVi5uEf83Z35TwBz+YEPtMQGl8vPNat5aFqN/Jow69Tvq/ tp58iZcxyGhlPnVMb4rUDHw6Eu029FhRRT5o70VEyoaMRrRz6PoFd4rbgGvr5hZN 1Kjg385dYtX7VqhOG3uFEN579PnaM+x6BPaz1jMlA4C/R8gW6xmnNj6tFeZOKJJO /wkAZIxnbrJfhTITy03onu/jhdtm9VJEkAKKH6uf/jy73Xh2lmNjPIh/bHK6etJk HI1TceqEh/a0Na7CURR98YTsFPScxqDH7LT3SMKckxkYOUeyiOa1w== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dwucwcyd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:28 +0000 (GMT) Received: from m0360083.ppops.net (m0360083.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgRlw028380; Sun, 10 Aug 2025 13:42:27 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dwucwcy9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:27 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57ACKYC1026282; Sun, 10 Aug 2025 13:42:26 GMT Received: from smtprelay01.fra02v.mail.ibm.com ([9.218.2.227]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48eh20t4nk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:26 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay01.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgOXv62390778 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:24 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78F9820043; Sun, 10 Aug 2025 13:42:24 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4794620040; Sun, 10 Aug 2025 13:42:22 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:22 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 07/11] generic: Stress fsx with atomic writes enabled Date: Sun, 10 Aug 2025 19:11:58 +0530 Message-ID: <50487b2e8a510598a93888c2674df7357d371da8.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXyaXxv26N7l2/ 8/yx1ebUVhlTYhGiNJTyTlOxB/r/WD2TlfVz/9wXBQYxjVhMcCSPvkBQO5oe8PaPq4SVEH2m82k 2LHSAmIXKpeYfysWSsBtTUTwNa4HRa+7gUVa3FnlRSw9yvm8hMdekOMsdfCljB/XXfdmpjlspWK z05vZ/QojJLX8qxIC9FC9Iu522MEHvzw9GFX4oUkC6h+DjfeQg0RsPlUuY4Onrf9WcwM0qccNNU 71dcAUB4txE2iR+VYNYLn2EIC7SxLmaGRENhwLm007MvyvgIwrpUTtW/7x1chB63adQaIqCZ5qi blYto26ZQTydv2uw7Idu5vDABGfVC3ks1Lh/ZcUDZLwotKCqC/+3b7tlo4xW8plFzwH/n8rdUWO iK3Cr4/AFZ710MXDk75PwAXXigPYHuI2ReOCPkmKSb/umflha37DXrCBkb+8Q1JmTiZYU+NE X-Authority-Analysis: v=2.4 cv=d/31yQjE c=1 sm=1 tr=0 ts=6898a1c4 cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VnNF1IyMAAAA:8 a=zPW1cLLGUcrnTIp6xCMA:9 X-Proofpoint-GUID: CYLhPlzA0JAufm-1JbEkjcyz6nyRMYtt X-Proofpoint-ORIG-GUID: kdReY34iAi8zHa2SHWAM0YMeTLmfD1Kq X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 mlxscore=0 suspectscore=0 impostorscore=0 priorityscore=1501 mlxlogscore=968 phishscore=0 lowpriorityscore=0 adultscore=0 spamscore=0 bulkscore=0 malwarescore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" Stress file with atomic writes to ensure we excercise codepaths where we are mixing different FS operations with atomic writes Suggested-by: Ritesh Harjani (IBM) Signed-off-by: Ojaswin Mujoo Reviewed-by: "Darrick J. Wong" --- tests/generic/1229 | 68 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/1229.out | 2 ++ 2 files changed, 70 insertions(+) create mode 100755 tests/generic/1229 create mode 100644 tests/generic/1229.out diff --git a/tests/generic/1229 b/tests/generic/1229 new file mode 100755 index 00000000..7fa57105 --- /dev/null +++ b/tests/generic/1229 @@ -0,0 +1,68 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1229 +# +# fuzz fsx with atomic writes +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest rw auto quick atomicwrites + +_require_odirect +_require_scratch_write_atomic + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +testfile=3D$SCRATCH_MNT/testfile +touch $testfile + +awu_max=3D$(_get_atomic_write_unit_max $testfile) +blksz=3D$(_get_block_size $SCRATCH_MNT) +bsize=3D`$here/src/min_dio_alignment $SCRATCH_MNT $SCRATCH_DEV` + +set_fsx_avoid() { + local file=3D$1 + + case "$FSTYP" in + "ext4") + local dev=3D$(findmnt -n -o SOURCE --target $testfile) + + # fsx insert/collpase range support for ext4+bigalloc is + # currently broken, so disable it. Also disable incase we can't + # detect bigalloc to be on safer side. + if [ -z "$DUMPE2FS_PROG" ]; then + echo "dumpe2fs not found, disabling insert/collapse range" >> $seqres.f= ull + FSX_AVOID+=3D" -I -C" + return + fi + + $DUMPE2FS_PROG -h $dev 2>&1 | grep -q bigalloc && { + echo "fsx insert/collapse range not supported with bigalloc. Disabling.= ." >> $seqres.full + FSX_AVOID+=3D" -I -C" + } + ;; + *) + ;; + esac +} + +# fsx usage: +# +# -N numops: total # operations to do +# -l flen: the upper bound on file size +# -o oplen: the upper bound on operation size (64k default) +# -Z: O_DIRECT () + +set_fsx_avoid +_run_fsx_on_file $testfile -N 10000 -o $awu_max -A -l 500000 -r $bsize -w = $bsize -Z $FSX_AVOID >> $seqres.full +if [[ "$?" !=3D "0" ]] +then + _fail "fsx returned error: $?" +fi + +echo "Silence is golden" +status=3D0 +exit diff --git a/tests/generic/1229.out b/tests/generic/1229.out new file mode 100644 index 00000000..737d61c6 --- /dev/null +++ b/tests/generic/1229.out @@ -0,0 +1,2 @@ +QA output created by 1229 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56AF6275B08; Sun, 10 Aug 2025 13:42:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833360; cv=none; b=PZ1Ml8WD+bHh7cwNuFR8n5nJGetMU7MfCzi3Xd262eRBLjJrPOg64cwuAJ6CQ54z085k7QIK6HAhOuF79tLBNP/aWkYk28faC5g8iwI5128YctSACF2OwoYGZZiJvk7Vx0k9uWAcnwxjsFYBjcNYAVF3uI3WrkcYvYkAvNziido= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833360; c=relaxed/simple; bh=5Whlb3M258/hYH1QZfI08fWCJYDAzM4q+pN3fIkE6zg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZVC3rcG4mZbLY2CiE9WmGaMBsSCwm2Z9NmrGW2kqvCB29OmGnhOdqyI9Wlovw+soxHaJ/TIZ/t2ou3tqPkArB2i5dJgZSW1u4DsQVjC8gOAA/OwEcWn6xEZMAxtuE6Au2Kz7Kx4pCEjfKeDv3cmuiLCjVAlak9HqBvVCF7VnJRg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=F/fwmhpW; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="F/fwmhpW" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57ABmBYR002295; Sun, 10 Aug 2025 13:42:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=K/X0pllH+N+VUDutZ XHOgTw9gsmcrn73ZXWsZBojlR0=; b=F/fwmhpWdkoFjw6EH5roCDNrcj0HZr8m4 FkdziKAjIlz18NeSs7L+rlXSfQwiGtbRsVlm0r8hDi881l3IpurKJNRleCB+TBlm htrlyR1B6SZTogu9w/8njIbiemjD2uhU+X4a2m6gM1vd3Sl6L4zQnn8aKKXWrVhB auMla/Yn1kSzvmhnlhqqWXXSTnUyMthdqkRkPj//fgrLBPaUc50+dDwnrHUr7nes 75RRNDSZQcdg3cWyp+zeCNQoXSOHeITZupuezzrykNSNkfkzdyfV6H746DGL54Wn 26elLZCbjvNjQAF62K+vEK6OOVNqATC4M5GVgkOn325jB7o80C01A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjb8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:30 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADemqb003312; Sun, 10 Aug 2025 13:42:29 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48durtwjb5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:29 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A8ZwlT025654; Sun, 10 Aug 2025 13:42:28 GMT Received: from smtprelay04.fra02v.mail.ibm.com ([9.218.2.228]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ejvm1rpj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:28 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay04.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgRBj15860120 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:27 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 263D520043; Sun, 10 Aug 2025 13:42:27 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DCAA820040; Sun, 10 Aug 2025 13:42:24 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:24 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 08/11] generic: Add sudden shutdown tests for multi block atomic writes Date: Sun, 10 Aug 2025 19:11:59 +0530 Message-ID: <36f8204dfa6880956dd94a52e9d89b4f1dfeba53.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 9HcCdvpu4xlMdj4iiJ-HPeVM4v5heHRA X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfX8Lg4GGo6M0Kp +BAOgRPWQXe86PyttdWzV5osqtcqOLjc6brsnvPRwMbM+5eT4CuW0oxaWqL0dwSZPuEQ39Qi2KO akXpzi6qjIMejgKYKAXDXNkbkb7RJnlT2KFntIPioZwtfQnyq/Qdde5oTaFAnLWtEwm69u4XBow m1wNT26gtB8EXmSORGP/zBV6se7OuerjMmxk2RupkHTWd7LoYQ1/Ozh1vbyi4j4uPm3CBN5eUq+ g/R3OfJ3PRFSfvXO7b9p7ef4ckEO9+dR2QKtcPayykZPm+Zopl4BMS5JzLPBOyZmqC1d5cQ0pC4 EIyHJEDfeT5eFaxWsM8j27g7UnLV2NWCKUbXJtcFBErYuYLylZeKQvjqFll0ZUQXa5Eiq8+H/s8 atgoigwxSanKu+C/R4EmiJpPoJVFrylCrwLRLYOxgl56fqoOYq3x9+Dq1vyl++J5NgEppuua X-Authority-Analysis: v=2.4 cv=QtNe3Uyd c=1 sm=1 tr=0 ts=6898a1c6 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=jzLDx6uKbS-OZFe_CzQA:9 X-Proofpoint-ORIG-GUID: Ku93tApph_-hkBr3u6rjQoMhbkxPjknr X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 lowpriorityscore=0 clxscore=1015 mlxlogscore=999 impostorscore=0 suspectscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 spamscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" This test is intended to ensure that multi blocks atomic writes maintain atomic guarantees across sudden FS shutdowns. The way we work is that we lay out a file with random mix of written, unwritten and hole extents. Then we start performing atomic writes sequentially on the file while we parallely shutdown the FS. Then we note the last offset where the atomic write happened just before shut down and then make sure blocks around it either have completely old data or completely new data, ie the write was not torn during shutdown. We repeat the same with completely written, completely unwritten and comple= tely empty file to ensure these cases are not torn either. Finally, we have a similar test for append atomic writes Suggested-by: Ritesh Harjani (IBM) Reviewed-by: Darrick J. Wong Signed-off-by: Ojaswin Mujoo --- tests/generic/1230 | 397 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1230.out | 2 + 2 files changed, 399 insertions(+) create mode 100755 tests/generic/1230 create mode 100644 tests/generic/1230.out diff --git a/tests/generic/1230 b/tests/generic/1230 new file mode 100755 index 00000000..cff5adc0 --- /dev/null +++ b/tests/generic/1230 @@ -0,0 +1,397 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test No. 1230 +# +# Test multi block atomic writes with sudden FS shutdowns to ensure +# the FS is not tearing the write operation +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full + +testfile=3D$SCRATCH_MNT/testfile +touch $testfile + +awu_max=3D$(_get_atomic_write_unit_max $testfile) +blksz=3D$(_get_block_size $SCRATCH_MNT) +echo "Awu max: $awu_max" >> $seqres.full + +num_blocks=3D$((awu_max / blksz)) +# keep initial value high for dry run. This will be +# tweaked in dry_run() based on device write speed. +filesize=3D$(( 10 * 1024 * 1024 * 1024 )) + +_cleanup() { + [ -n "$awloop_pid" ] && kill $awloop_pid &> /dev/null + wait +} + +atomic_write_loop() { + local off=3D0 + local size=3D$awu_max + for ((i=3D0; i<$((filesize / $size )); i++)); do + # Due to sudden shutdown this can produce errors so just + # redirect them to seqres.full + $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $siz= e $off $size" >> /dev/null 2>>$seqres.full + echo "Written to offset: $off" >> $tmp.aw + off=3D$((off + $size)) + done +} + +# This test has the following flow: +# 1. Start doing sequential atomic writes in bg, upto $filesize +# 2. Sleep for 0.2s and shutdown the FS +# 3. kill the atomic write process +# 4. verify the writes were not torn +# +# We ideally want the shutdown to happen while an atomic write is ongoing +# but this gets tricky since faster devices can actually finish the whole +# atomic write loop before sleep 0.2s completes, resulting in the shutdown +# happening after the write loop which is not what we want. A simple solut= ion +# to this is to increase $filesize so step 1 takes long enough but a big +# $filesize leads to create_mixed_mappings() taking very long, which is not +# ideal. +# +# Hence, use the dry_run function to figure out the rough device speed and= set +# $filesize accordingly. +dry_run() { + echo >> $seqres.full + echo "# Estimating ideal filesize..." >> $seqres.full + atomic_write_loop & + awloop_pid=3D$! + + local i=3D0 + # Wait for atleast first write to be recorded or 10s + while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=3D$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid + + bytes_written=3D$(tail -n 1 $tmp.aw | cut -d" " -f4) + echo "# Bytes written in 0.2s: $bytes_written" >> $seqres.full + + filesize=3D$((bytes_written * 3)) + echo "# Setting \$filesize=3D$filesize" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + +} + +create_mixed_mappings() { + local file=3D$1 + local size_bytes=3D$2 + + echo "# Filling file $file with alternate mappings till size $size_bytes"= >> $seqres.full + #Fill the file with alternate written and unwritten blocks + local off=3D0 + local operations=3D("W" "U") + + for ((i=3D0; i<$((size_bytes / blksz )); i++)); do + index=3D$(($i % ${#operations[@]})) + map=3D"${operations[$index]}" + + case "$map" in + "W") + $XFS_IO_PROG -fc "pwrite -b $blksz $off $blksz" $file >> /dev/null + ;; + "U") + $XFS_IO_PROG -fc "falloc $off $blksz" $file >> /dev/null + ;; + esac + off=3D$((off + blksz)) + done + + sync $file +} + +populate_expected_data() { + # create a dummy file with expected old data for different cases + create_mixed_mappings $testfile.exp_old_mixed $awu_max + expected_data_old_mixed=3D$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_o= ld_mixed) + + $XFS_IO_PROG -fc "falloc 0 $awu_max" $testfile.exp_old_zeroes >> $seqres.= full + expected_data_old_zeroes=3D$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_= old_zeroes) + + $XFS_IO_PROG -fc "pwrite -b $awu_max 0 $awu_max" $testfile.exp_old_mapped= >> $seqres.full + expected_data_old_mapped=3D$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_= old_mapped) + + # create a dummy file with expected new data + $XFS_IO_PROG -fc "pwrite -S 0x61 -b $awu_max 0 $awu_max" $testfile.exp_ne= w >> $seqres.full + expected_data_new=3D$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_new) +} + +verify_data_blocks() { + local verify_start=3D$1 + local verify_end=3D$2 + local expected_data_old=3D"$3" + local expected_data_new=3D"$4" + + echo >> $seqres.full + echo "# Checking data integrity from $verify_start to $verify_end" >> $se= qres.full + + # After an atomic write, for every chunk we ensure that the underlying + # data is either the old data or new data as writes shouldn't get torn. + local off=3D$verify_start + while [[ "$off" -lt "$verify_end" ]] + do + #actual_data=3D$(xxd -s $off -l $awu_max -p $testfile) + actual_data=3D$(od -An -t x1 -j $off -N $awu_max $testfile) + if [[ "$actual_data" !=3D "$expected_data_new" ]] && [[ "$actual_data" != =3D "$expected_data_old" ]] + then + echo "Checksum match failed at off: $off size: $awu_max" + echo "Expected contents: (Either of the 2 below):" + echo + echo "Expected old: " + echo "$expected_data_old" + echo + echo "Expected new: " + echo "$expected_data_new" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail + fi + echo -n "Check at offset $off suceeded! " >> $seqres.full + if [[ "$actual_data" =3D=3D "$expected_data_new" ]] + then + echo "matched new" >> $seqres.full + elif [[ "$actual_data" =3D=3D "$expected_data_old" ]] + then + echo "matched old" >> $seqres.full + fi + off=3D$(( off + awu_max )) + done +} + +# test data integrity for file by shutting down in between atomic writes +test_data_integrity() { + echo >> $seqres.full + echo "# Writing atomically to file in background" >> $seqres.full + atomic_write_loop & + awloop_pid=3D$! + + local i=3D0 + # Wait for atleast first write to be recorded or 10s + while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=3D$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid + + last_offset=3D$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=3D0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + + # we want to verify all blocks around which the shutdown happended + verify_start=3D$(( last_offset - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=3D0 + fi + + verify_end=3D$(( last_offset + (awu_max * 5))) + if [[ "$verify_end" -gt "$filesize" ]] + then + verify_end=3D$filesize + fi +} + +# test data integrity for file wiht written and unwritten mappings +test_data_integrity_mixed() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with mixed mappings" >> $seqres.full + create_mixed_mappings $testfile $filesize + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mixed" "= $expected_data_new" +} + +# test data integrity for file with completely written mappings +test_data_integrity_writ() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully written mapping" >> $seqres.full + $XFS_IO_PROG -c "pwrite -b $filesize 0 $filesize" $testfile >> $seqres.fu= ll + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mapped" = "$expected_data_new" +} + +# test data integrity for file with completely unwritten mappings +test_data_integrity_unwrit() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully unwritten mappings" >> $seqres.full + $XFS_IO_PROG -c "falloc 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" = "$expected_data_new" +} + +# test data integrity for file with no mappings +test_data_integrity_hole() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with no mappings" >> $seqres.full + $XFS_IO_PROG -c "truncate $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" = "$expected_data_new" +} + +test_filesize_integrity() { + $XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Performing extending atomic writes over file in background" >> $s= eqres.full + atomic_write_loop & + awloop_pid=3D$! + + local i=3D0 + # Wait for atleast first write to be recorded or 10s + while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=3D$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid + + local last_offset=3D$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=3D0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + local filesize=3D$(_get_filesize $testfile) + echo >> $seqres.full + echo "# Filesize after shutdown: $filesize" >> $seqres.full + + # To confirm that the write went atomically, we check: + # 1. The last block should be a multiple of awu_max + # 2. The last block should be the completely new data + + if (( $filesize % $awu_max )) + then + echo "Filesize after shutdown ($filesize) not a multiple of atomic write= unit ($awu_max)" + fi + + verify_start=3D$(( filesize - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=3D0 + fi + + local verify_end=3D$filesize + + # Here the blocks should always match new data hence, for simplicity of + # code, just corrupt the $expected_data_old buffer so it never matches + local expected_data_old=3D"POISON" + verify_data_blocks $verify_start $verify_end "$expected_data_old" "$expec= ted_data_new" +} + +$XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + +dry_run + +echo >> $seqres.full +echo "# Populating expected data buffers" >> $seqres.full +populate_expected_data + +# Loop 20 times to shake out any races due to shutdown +for ((iter=3D0; iter<20; iter++)) +do + echo >> $seqres.full + echo "------ Iteration $iter ------" >> $seqres.full + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over mixed mapping= " >> $seqres.full + test_data_integrity_mixed + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over fully written= mapping" >> $seqres.full + test_data_integrity_writ + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over fully unwritt= en mapping" >> $seqres.full + test_data_integrity_unwrit + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over holes" >> $se= qres.full + test_data_integrity_hole + + echo >> $seqres.full + echo "# Starting filesize integrity test for atomic writes" >> $seqres.fu= ll + test_filesize_integrity +done + +echo "Silence is golden" +status=3D0 +exit diff --git a/tests/generic/1230.out b/tests/generic/1230.out new file mode 100644 index 00000000..d01f54ea --- /dev/null +++ b/tests/generic/1230.out @@ -0,0 +1,2 @@ +QA output created by 1230 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0A2D261593; Sun, 10 Aug 2025 13:42:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833361; cv=none; b=ZhPrgSEtUCee5glqjHUg6GJfW61wgLvFAW/ZHVXt2EEhjfhdE2C8NODJelxsxrGs+CBA+Y5MRMkl7nIOee4WSrJeoE6714xPs1Z5Ittk3wACxAUCNc3RSi90zLE6kTvaLm7kFRIGwe3Ys1TJomml3fQMCzzeyrNPwrCEUeMlAhs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833361; c=relaxed/simple; bh=wspgVSwP2KtmDPNbaP/ai2qU3wRo/iq0pVkkQNIPIXw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I+NiO0sG9BVSGR/qaKXuFhwvjJAAsXWQmhfFH9RFQ0caPxq+mCc/ovbGhASVlnlZas48G8wJOZdHM2pPrADhcpW+0tXNNe4YVyl+aAqpICD6SFYQb7I86v7qhRgMlrwoiuDAEKsyaH8A7a9qn597c5aondRp2KcEqzVHe9REfUQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=R4InZBwS; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="R4InZBwS" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57ABFHYh020424; Sun, 10 Aug 2025 13:42:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=7cAcpyoz5b2WrvbMZ 5AZRhmyr9/OG0bYIbpRqA/h1BE=; b=R4InZBwSHc5aFHNaD4VtyABGVkDaVqRfw GJ8LnLy4hHwTWpxUBF4zjnHyDf/uvB5Vh00g+xcih2B6vHP724LuRPX6TuDMRGCM 5PI/c/mhgvY25Py4wFBaqt+igf+x8b/Qi8x+NFiY1mB0GesPcESTahYVyOjRG0WG 41StrH9AzxSfkLzY53Z4bE7iGYjrUvwdgUwAJKDn9Xbg72Pzi0F/pAEGTincCJx6 GUBdx6Let6ITr+vhhnVfvkpXphVhCeZrmkQenR1x7+FQCzy8SIb4D2pLIAjt7d1G eJxk34CnYk30TGr5uFgCg9KoqZjymctgDqgoJOisgDzRYlUilcJVg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dvrnnc4e-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:32 +0000 (GMT) Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgWID027642; Sun, 10 Aug 2025 13:42:32 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dvrnnc4c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:32 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A96Uqf017600; Sun, 10 Aug 2025 13:42:31 GMT Received: from smtprelay06.fra02v.mail.ibm.com ([9.218.2.230]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ekc39nf6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:31 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay06.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgTSo14680514 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:29 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AF3AC2004B; Sun, 10 Aug 2025 13:42:29 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 70CE620040; Sun, 10 Aug 2025 13:42:27 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:27 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 09/11] ext4: Atomic writes stress test for bigalloc using fio crc verifier Date: Sun, 10 Aug 2025 19:12:00 +0530 Message-ID: <210979d189d43165fa792101cf2f4eb8ef02bc98.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfX/e8XY7776UxW qd+khSW+FvzCiGaJ95VJObbbYiQJpaHrXdOw5W1a7mDHquC1pHEqHNEQOPYrf0bjgfYt6LGuxsN RYWgaE6z8DCDvT0KM9jZUG3cqi3d1FjmlbirIzWDFYNur952gVDocUEOdovfNCx3xroT9TrBMoi aUeoqC207eIdqwS/ltkJQZfUKifFNxfDIt5tz1v5mHLWnf2XKw9g8I6NL1M7SIxYSI5cjy5HCXE ISeBZeJyiEqoNHQXb9XOGLHtvDoLU42glj0xmgIrKLw0NdrYC01vs/rF4jcu1SL4N3IcHaC4Dwu x33iCrs4y06EyBdHoQG3XM5dexyCBE5u39uHlxgQVaRm7r87pQk6c2ta6CYpN8QDuNVC6ClDcHO Z7wi5QulFz/ekoD+Yju+KErH+RbJycIwudvXldJu77yV+aqTHCMDHNNywrL1OUtqjm6F6RBm X-Authority-Analysis: v=2.4 cv=GrpC+l1C c=1 sm=1 tr=0 ts=6898a1c8 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VnNF1IyMAAAA:8 a=jHICElQ_nnKT0EPUS5YA:9 X-Proofpoint-GUID: Dxu8SrsZfkogDVdt7_UvziYjm6EiDJO1 X-Proofpoint-ORIG-GUID: ih8___XlxYzVK6FGa04wgtgjhQH5NT-S X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=922 malwarescore=0 clxscore=1015 adultscore=0 mlxscore=0 spamscore=0 lowpriorityscore=0 priorityscore=1501 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" From: "Ritesh Harjani (IBM)" We brute force all possible blocksize & clustersize combinations on a bigalloc filesystem for stressing atomic write using fio data crc verifier. We run nproc * $LOAD_FACTOR threads in parallel writing to a single $SCRATCH_MNT/test-file. With atomic writes this test ensures that we never see the mix of data contents from different threads on a given bsrange. This test might do overlapping atomic writes but that should be okay since overlapping parallel hardware atomic writes don't cause tearing as long as io size is the same for all writes. Signed-off-by: Ritesh Harjani (IBM) Signed-off-by: Ojaswin Mujoo --- tests/ext4/061 | 130 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/061.out | 2 + 2 files changed, 132 insertions(+) create mode 100755 tests/ext4/061 create mode 100644 tests/ext4/061.out diff --git a/tests/ext4/061 b/tests/ext4/061 new file mode 100755 index 00000000..a0e49249 --- /dev/null +++ b/tests/ext4/061 @@ -0,0 +1,130 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 061 +# +# Brute force all possible blocksize clustersize combination on a bigalloc +# filesystem for stressing atomic write using fio data crc verifier. We run +# nproc * 2 * $LOAD_FACTOR threads in parallel writing to a single +# $SCRATCH_MNT/test-file. With fio aio-dio atomic write this test ensures = that +# we should never see the mix of data contents from different threads for = any +# given fio blocksize. +# + +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto rw stress atomicwrites + +_require_scratch_write_atomic +_require_aiodio + +FIO_LOAD=3D$(($(nproc) * 2 * LOAD_FACTOR)) +SIZE=3D$((100*1024*1024)) +fiobsize=3D4096 + +# Calculate fsblocksize as per bdev atomic write units. +bdev_awu_min=3D$(_get_atomic_write_unit_min $SCRATCH_DEV) +bdev_awu_max=3D$(_get_atomic_write_unit_max $SCRATCH_DEV) +fsblocksize=3D$(_max 4096 "$bdev_awu_min") + +function create_fio_configs() +{ + create_fio_aw_config + create_fio_verify_config +} + +function create_fio_verify_config() +{ +cat >$fio_verify_config <$fio_aw_config <> $seqres.full 2>&1 || continue + if _try_scratch_mount >> $seqres.full 2>&1; then + echo "=3D=3D FIO test for fsblocksize=3D$fsblocksize fsclustersize=3D$= fsclustersize fiobsize=3D$fiobsize =3D=3D" >> $seqres.full + + touch $SCRATCH_MNT/f1 + create_fio_configs + + cat $fio_aw_config >> $seqres.full + echo >> $seqres.full + cat $fio_verify_config >> $seqres.full + + $FIO_PROG $fio_aw_config >> $seqres.full + ret1=3D$? + + $FIO_PROG $fio_verify_config >> $seqres.full + ret2=3D$? + + _scratch_unmount + + [[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write faile= d" + fi + done + done +done + +# success, all done +echo Silence is golden +status=3D0 +exit diff --git a/tests/ext4/061.out b/tests/ext4/061.out new file mode 100644 index 00000000..273be9e0 --- /dev/null +++ b/tests/ext4/061.out @@ -0,0 +1,2 @@ +QA output created by 061 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 12F382777E8; Sun, 10 Aug 2025 13:42:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833364; cv=none; b=galmk1K3aSoXkTEtHelIdgja2/4LDaYYez3REF35W/K8KTeaU5ij3wOvClOTGbfcO7mdASmOVawNnIhYwlumOBw5DVjEbC/aOLQBgBCgMrEzIZ1r5KVmRQ4zub4fjnddiwbJMkCmPJQ279vE1j3g2jsac/qI9LJFcJLlccUnH2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833364; c=relaxed/simple; bh=a8+l/vQCVd7zvGymmqioCgbO00ixsS34NRSk50oKXGU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kJHI5xjFRcGrLKjc9wzaAvC9RDFNwnx4FIrxSjY0IUfYh6u8/DXF/vR8c9VKT08vzXRnRxIFxcgyMsWRk5U/NHNu9opMsqSpTBh72qWDyIw93Us/fNpukmwvr+/KIemqKrNUSlBNvxnSLJFFwKknQdcOjLCYhoFOzwwKE3G9YD8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=ScAiz6GJ; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ScAiz6GJ" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A7jQxV032216; Sun, 10 Aug 2025 13:42:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=IjNJEN3tswwPw+vKz Dpz+GmwVdM6YneETCeOAS+DPLo=; b=ScAiz6GJdsJFupNOmynah+5Fk7xixlc04 SYhMAPAigErr9eGzKegvZv/J9uVaDw35fIsCrLHejj80cvEAt4iI1tC7z1yci87V +dJIfBpZHBhAuJQtadIFiLvV/02NFg4bwJB6iPKvdGgOavYmoutrJzhyg2xFJs7P sQPepyAS8HBOfKhTHddXSWSJcADBwEjunztkeVeFVrBStDmaPwrCXnsczVsq++gZ EGpgNOSPhEMbNB8r1s/oDagbufQ93SAB2A+fVKPUBTMvGF0+AaSf/9J1weDryDDC P7pfI0GlQti/JdVpeSZZXQfA4/pqkOC31lGWY8AjnriduXNvS3pQw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx145b1t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:35 +0000 (GMT) Received: from m0353729.ppops.net (m0353729.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgZbb021347; Sun, 10 Aug 2025 13:42:35 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx145b1p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:35 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57AC0Vgs026275; Sun, 10 Aug 2025 13:42:34 GMT Received: from smtprelay02.fra02v.mail.ibm.com ([9.218.2.226]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 48eh20t4nu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:34 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay02.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgWrv46334240 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:32 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 057FC2004B; Sun, 10 Aug 2025 13:42:32 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1E85D20040; Sun, 10 Aug 2025 13:42:30 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:29 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 10/11] ext4: Atomic writes test for bigalloc using fio crc verifier on multiple files Date: Sun, 10 Aug 2025 19:12:01 +0530 Message-ID: <48873bdce79f491b8b8948680afe041831af08cd.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: xzEFJrD-pyhVh85Bb_wWln9DZmGQsAC0 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfX/UnZalWhXx1J Y6iqbb7xktIRYuDaCubrw/wVmBDUkPGI2J8UHfMcN667Q2xGqqmNmObsnAoExkDHdYZTGL039M2 xn8SsZflXh/7YS3COXcOJOQeDZVbPcb513+D9Yhv7DDgFjtaiiaYvCEaUxl8mF3AGbT1FG1RSxH Yyi7EZHFZP6yvLXQrcq+wpHmy3r9QylsbgokajD4FjgBuFYCLN/f61x9hkJyByIlw20RoQi5vkH gq89Ld1F8aElvy3WKBcDb/M5zA9Fsrfzc84q02uw64ctA82X2Vgd9s7MrMi07kDIXs0eL/jRmR3 sPnRrtozvyO1dlsOW+elQDKVfeDdZSdmy1eQtP6PEjvs65dC1MI1jD+liNCcBbBfYBfQrQ3JeYI 4Vw2n3Jf1a9Oulmq/LKk0QZSl3+3ruJu1Mzath2AQnVjd4bLYkDyoRmcNvd0Zqq4sb/o2soJ X-Proofpoint-GUID: -IkvSt6TVQfCjNwQVxfB-Y5BGPGruG9U X-Authority-Analysis: v=2.4 cv=fLg53Yae c=1 sm=1 tr=0 ts=6898a1cc cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=2OwXVqhp2XgA:10 a=pGLkceISAAAA:8 a=VwQbUJbxAAAA:8 a=VnNF1IyMAAAA:8 a=yVCtIWGlblad6CDWSz4A:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 lowpriorityscore=0 clxscore=1015 spamscore=0 priorityscore=1501 impostorscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" From: "Ritesh Harjani (IBM)" Brute force all possible blocksize clustersize combination on a bigalloc filesystem for stressing atomic write using fio data crc verifier. We run multiple threads in parallel with each job writing to its own file. The parallel jobs running on a constrained filesystem size ensure that we stress the ext4 allocator to allocate contiguous extents. This test might do overlapping atomic writes but that should be okay since overlapping parallel hardware atomic writes don't cause tearing as long as io size is the same for all writes. Signed-off-by: Ritesh Harjani (IBM) Reviewed-by: Darrick J. Wong Signed-off-by: Ojaswin Mujoo --- tests/ext4/062 | 176 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/062.out | 2 + 2 files changed, 178 insertions(+) create mode 100755 tests/ext4/062 create mode 100644 tests/ext4/062.out diff --git a/tests/ext4/062 b/tests/ext4/062 new file mode 100755 index 00000000..da5e1076 --- /dev/null +++ b/tests/ext4/062 @@ -0,0 +1,176 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 062 +# +# Brute force all possible blocksize clustersize combination on a bigalloc +# filesystem for stressing atomic write using fio data crc verifier. We run +# nproc * $LOAD_FACTOR threads in parallel writing to a single +# $SCRATCH_MNT/test-file. We also create 8 such parallel jobs to run on +# a constrained filesystem size to stress the ext4 allocator to allocate +# contiguous extents. +# + +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto rw stress atomicwrites + +_require_scratch_write_atomic +_require_aiodio + +FSSIZE=3D$((360*1024*1024)) +FIO_LOAD=3D$(($(nproc) * LOAD_FACTOR)) +fiobsize=3D4096 + +# Calculate fsblocksize as per bdev atomic write units. +bdev_awu_min=3D$(_get_atomic_write_unit_min $SCRATCH_DEV) +bdev_awu_max=3D$(_get_atomic_write_unit_max $SCRATCH_DEV) +fsblocksize=3D$(_max 4096 "$bdev_awu_min") + +function create_fio_configs() +{ + create_fio_aw_config + create_fio_verify_config +} + +function create_fio_verify_config() +{ +cat >$fio_verify_config <$fio_aw_config <> $seqres.full 2>&1 || continue + if _try_scratch_mount >> $seqres.full 2>&1; then + echo "=3D=3D FIO test for fsblocksize=3D$fsblocksize fsclustersize=3D$= fsclustersize fiobsize=3D$fiobsize =3D=3D" >> $seqres.full + + touch $SCRATCH_MNT/f1 + create_fio_configs + + cat $fio_aw_config >> $seqres.full + cat $fio_verify_config >> $seqres.full + + $FIO_PROG $fio_aw_config >> $seqres.full + ret1=3D$? + + $FIO_PROG $fio_verify_config >> $seqres.full + ret2=3D$? + + _scratch_unmount + + [[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write faile= d" + fi + done + done +done + +# success, all done +echo Silence is golden +status=3D0 +exit diff --git a/tests/ext4/062.out b/tests/ext4/062.out new file mode 100644 index 00000000..a1578f48 --- /dev/null +++ b/tests/ext4/062.out @@ -0,0 +1,2 @@ +QA output created by 062 +Silence is golden --=20 2.49.0 From nobody Sun Oct 5 03:35:16 2025 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B7B1E27932D; Sun, 10 Aug 2025 13:42:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833367; cv=none; b=WiUSOheP+bRducJ+nsruEcADRyJm1IEIp68tSvx/1td/F6P59u/JAXdmVSuXhOAtcuoqsz1wTsnkocHBICWu7i10KUuUzt3mz3tYZrvk0cZeI6z2G4bDP7q1NnIP5TRyducCKEfQOIOAYTyUuKVucmiOYUi47nW42U1kvVXhtJ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1754833367; c=relaxed/simple; bh=gmaBm0x7uDbdmIXG5IfwG/w+jWZjkMAjllytUqcjUBY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=BQ1aDttRvEMYL7VopgEhFJafrCO3U0UxeqUpK26vN5sIpcLXPGYOdkdmSr4zTF/qGoHd9Lza9B8+aLUfLOS9CfwtnkfYhvuaKbYkMkGhzQ71jc9ie1ScaaOXYfp29i0vszdRUYZwdN3Z9i3O6UZJXyTHek4SVNX5rFHY6/Jd3E0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=eh+Axr50; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="eh+Axr50" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 57A3jCU8030469; Sun, 10 Aug 2025 13:42:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=0IjariK8pY14h7+X2 4bWSkvksl0nJpavjJtDkM8wVT8=; b=eh+Axr507KOyKuL9J2wu1GlZNAYkbFfbu R7ogdjUeRquixBrfUTi1htgSYsE9bM/4OQXvin7WJGsdagcfu5jSvlgrb4CHZxqo baUjqVBaw5FygAe1lWeKrXuW7BTmT8+r4dt1EIfSgqzTNsYr1GOnBIV1dIYdGVhf KK4mip/Tya1DFYzK9rbnM/iGZqi86tzPM9djwXwOT1+2Rt7NnLnVIzS8rzjsuYsZ Onr0EyPyJ3lzA3/nvm+fHGYVcuM5pcTDZIpGZlz3P1i2ZQKHCWq8ywCXgQv/xDP8 /xymsP0O7IvypS/o0xzF+Kp8qaJ7ZBWa4oDwVYMmPmWNP0MLgZ2Kg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx2cnb4u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:37 +0000 (GMT) Received: from m0356517.ppops.net (m0356517.ppops.net [127.0.0.1]) by pps.reinject (8.18.1.12/8.18.0.8) with ESMTP id 57ADgbWl013788; Sun, 10 Aug 2025 13:42:37 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 48dx2cnb4r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:37 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 57A9ApPL017581; Sun, 10 Aug 2025 13:42:36 GMT Received: from smtprelay04.fra02v.mail.ibm.com ([9.218.2.228]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 48ekc39nfb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Aug 2025 13:42:36 +0000 Received: from smtpav02.fra02v.mail.ibm.com (smtpav02.fra02v.mail.ibm.com [10.20.54.101]) by smtprelay04.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 57ADgY2429098598 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sun, 10 Aug 2025 13:42:34 GMT Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A2F72004B; Sun, 10 Aug 2025 13:42:34 +0000 (GMT) Received: from smtpav02.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 69CBD20040; Sun, 10 Aug 2025 13:42:32 +0000 (GMT) Received: from li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com (unknown [9.124.216.43]) by smtpav02.fra02v.mail.ibm.com (Postfix) with ESMTP; Sun, 10 Aug 2025 13:42:32 +0000 (GMT) From: Ojaswin Mujoo To: Zorro Lang , fstests@vger.kernel.org Cc: Ritesh Harjani , djwong@kernel.org, john.g.garry@oracle.com, tytso@mit.edu, linux-xfs@vger.kernel.org, linux-kernel@vger.kernel.org, linux-ext4@vger.kernel.org Subject: [PATCH v4 11/11] ext4: Atomic write test for extent split across leaf nodes Date: Sun, 10 Aug 2025 19:12:02 +0530 Message-ID: <2c241ea2ede39914d29aa59cd06acfc951aed160.1754833177.git.ojaswin@linux.ibm.com> X-Mailer: git-send-email 2.49.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjUwODEwMDA5NyBTYWx0ZWRfXwqhnm0fuu3Yj FUFaXx/o8z3zpAK+/aiqXE+KsslYQkZF/cvkSZaauco1lIZHbug/OUhl7pdd8gswyOqzCCEKa7m HqtqYiFX8e5GFq5HVTlaTvT7zMwIqedEGe0sO3LIPHBVFP9z4Dpwbdf1Q4eTMkzdxzgvR6dcNrS ZxrxHU+18VYC6/UTHwTPzIZesz82PzOvzATtF4/YfnXm8Rhceosnx7DZeKK6tZlKx/ta6uJg/IA +0dAQgvl1/a0k2AoaEJEehtv/4X3zaPRfhmxgDfcjmNrVNjV3iYXp1nvvA+inQT7bECz4c9L6OI ay7xaUNWi2OXC+arpUaxxip25NU/TeRVpnjJMbVHB/FO9vZwJqjjzIClk8BuAaL0JQ5DtAGZmaf vQYwUZewYgS6YkFDGgDqRrSHfyu4i79OzY+PfjmgV8HIm7YG5kd5A70uP2hfOvPRaMUQbhBt X-Proofpoint-GUID: Q73VRy-nX547rMYQyL3uRFJXu57SlkTA X-Authority-Analysis: v=2.4 cv=C9zpyRP+ c=1 sm=1 tr=0 ts=6898a1ce cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=2OwXVqhp2XgA:10 a=VnNF1IyMAAAA:8 a=BtNrGSYHu5IGmLxdGZgA:9 X-Proofpoint-ORIG-GUID: vl1O7FVlYpGgKqAdz-SvSJorus_rXwhY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1099,Hydra:6.1.9,FMLib:17.12.80.40 definitions=2025-08-10_04,2025-08-06_01,2025-03-28_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxscore=0 bulkscore=0 lowpriorityscore=0 phishscore=0 mlxlogscore=999 suspectscore=0 impostorscore=0 clxscore=1015 malwarescore=0 spamscore=0 priorityscore=1501 classifier=spam authscore=0 authtc=n/a authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.19.0-2507300000 definitions=main-2508100097 Content-Type: text/plain; charset="utf-8" In ext4, even if an allocated range is physically and logically contiguous, it can still be split into 2 extents. This is because ext4 does not merge extents across leaf nodes. This is an issue for atomic writes since even for a continuous extent the map block could (in rare cases) return a shorter map, hence tearning the write. This test creates such a file and ensures that the atomic write handles this case correctly Signed-off-by: Ojaswin Mujoo Reviewed-by: "Darrick J. Wong" --- tests/ext4/063 | 129 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/063.out | 2 + 2 files changed, 131 insertions(+) create mode 100755 tests/ext4/063 create mode 100644 tests/ext4/063.out diff --git a/tests/ext4/063 b/tests/ext4/063 new file mode 100755 index 00000000..40867acb --- /dev/null +++ b/tests/ext4/063 @@ -0,0 +1,129 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# In ext4, even if an allocated range is physically and logically contiguo= us, +# it can still be split into 2 extents. This is because ext4 does not merge +# extents across leaf nodes. This is an issue for atomic writes since even= for +# a continuous extent the map block could (in rare cases) return a shorter= map, +# hence tearning the write. This test creates such a file and ensures that= the +# atomic write handles this case correctly +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_command "$DEBUGFS_PROG" debugfs + +prep() { + local bs=3D`_get_block_size $SCRATCH_MNT` + local ex_hdr_bytes=3D12 + local ex_entry_bytes=3D12 + local entries_per_blk=3D$(( (bs - ex_hdr_bytes) / ex_entry_bytes )) + + # fill the extent tree leaf with bs len extents at alternate offsets. + # The tree should look as follows + # + # +---------+---------+ + # | index 1 | index 2 | + # +-----+---+-----+---+ + # +------+ +-----------+ + # | | + # +-------+-------+---+---------+ +-----+----+ + # | ex 1 | ex 2 | | ex n | | ex n+1 | + # | off:0 | off:2 |...| off:678 | | off:680 | + # | len:1 | len:1 | | len:1 | | len:1 | + # +-------+-------+---+---------+ +----------+ + # + for i in $(seq 0 $entries_per_blk) + do + $XFS_IO_PROG -fc "pwrite -b $bs $((i * 2 * bs)) $bs" $testfile > /dev/nu= ll + done + sync $testfile + + echo >> $seqres.full + echo "Create file with extents spanning 2 leaves. Extents:">> $seqres.full + echo "...">> $seqres.full + $DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqre= s.full + + # Now try to insert a new extent ex(new) between ex(n) and ex(n+1). + # Since this is a new FS the allocator would find continuous blocks + # such that ex(n) ex(new) ex(n+1) are physically(and logically) + # contiguous. However, since we dont merge extents across leaf we will + # end up with a tree as: + # + # +---------+---------+ + # | index 1 | index 2 | + # +-----+---+-----+---+ + # +------+ +------------+ + # | | + # +-------+-------+---+---------+ +------+-----------+ + # | ex 1 | ex 2 | | ex n | | ex n+1 (merged) | + # | off:0 | off:2 |...| off:678 | | off:679 | + # | len:1 | len:1 | | len:1 | | len:2 | + # +-------+-------+---+---------+ +------------------+ + # + echo >> $seqres.full + torn_ex_offset=3D$((((entries_per_blk * 2) - 1) * bs)) + $XFS_IO_PROG -c "pwrite $torn_ex_offset $bs" $testfile >> /dev/null + sync $testfile + + echo >> $seqres.full + echo "Perform 1 block write at $torn_ex_offset to create torn extent. Ext= ents:">> $seqres.full + echo "...">> $seqres.full + $DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqre= s.full + + _scratch_cycle_mount +} + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full + +testfile=3D$SCRATCH_MNT/testfile +touch $testfile +awu_max=3D$(_get_atomic_write_unit_max $testfile) + +echo >> $seqres.full +echo "# Prepping the file" >> $seqres.full +prep + +torn_aw_offset=3D$((torn_ex_offset - (torn_ex_offset % awu_max))) + +echo >> $seqres.full +echo "# Performing atomic IO on the torn extent range. Command: " >> $seqr= es.full +echo $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $= awu_max $torn_aw_offset $awu_max" >> $seqres.full +$XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $awu_m= ax $torn_aw_offset $awu_max" >> $seqres.full + +echo >> $seqres.full +echo "Extent state after atomic write:">> $seqres.full +echo "...">> $seqres.full +$DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqres= .full + +echo >> $seqres.full +echo "# Checking data integrity" >> $seqres.full + +# create a dummy file with expected data +$XFS_IO_PROG -fc "pwrite -S 0x61 -b $awu_max 0 $awu_max" $testfile.exp >> = /dev/null +expected_data=3D$(od -An -t x1 -j 0 -N $awu_max $testfile.exp) + +# We ensure that the data after atomic writes should match the expected da= ta +actual_data=3D$(od -An -t x1 -j $torn_aw_offset -N $awu_max $testfile) +if [[ "$actual_data" !=3D "$expected_data" ]] +then + echo "Checksum match failed at off: $torn_aw_offset size: $awu_max" + echo + echo "Expected: " + echo "$expected_data" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail +fi + +echo -n "Data verification at offset $torn_aw_offset suceeded!" >> $seqres= .full +echo "Silence is golden" +status=3D0 +exit diff --git a/tests/ext4/063.out b/tests/ext4/063.out new file mode 100644 index 00000000..de35fc52 --- /dev/null +++ b/tests/ext4/063.out @@ -0,0 +1,2 @@ +QA output created by 063 +Silence is golden --=20 2.49.0