1. Patchew
  2. linux
  3. [PATCH v3 00/13] Add more tests for multi fs block atomic writes
  4. View patch

[PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests

Ojaswin Mujoo posted 13 patches 2 months, 3 weeks ago
There is a newer version of this series
[PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests
Posted by Ojaswin Mujoo 2 months, 3 weeks ago 
This adds various atomic write multi-fsblock stresst tests
with mixed mappings and O_SYNC, to ensure the data and metadata
is atomically persisted even if there is a shutdown.

Suggested-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
---
 tests/generic/1228     | 139 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/1228.out |   2 +
 2 files changed, 141 insertions(+)
 create mode 100755 tests/generic/1228
 create mode 100644 tests/generic/1228.out

diff --git a/tests/generic/1228 b/tests/generic/1228
new file mode 100755
index 00000000..3f9a6af1
--- /dev/null
+++ b/tests/generic/1228
@@ -0,0 +1,139 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2025 IBM Corporation. All Rights Reserved.
+#
+# FS QA Test 1228
+#
+# Atomic write multi-fsblock data integrity tests with mixed mappings
+# and O_SYNC
+#
+. ./common/preamble
+. ./common/atomicwrites
+_begin_fstest auto quick rw atomicwrites
+
+_require_scratch_write_atomic_multi_fsblock
+_require_atomic_write_test_commands
+_require_scratch_shutdown
+_require_xfs_io_command "truncate"
+
+_scratch_mkfs >> $seqres.full
+_scratch_mount >> $seqres.full
+
+check_data_integrity() {
+	actual=$(_hexdump $testfile)
+	if [[ "$expected" != "$actual" ]]
+	then
+		echo "Integrity check failed"
+		echo "Integrity check failed" >> $seqres.full
+		echo "# Expected file contents:" >> $seqres.full
+		echo "$expected" >> $seqres.full
+		echo "# Actual file contents:" >> $seqres.full
+		echo "$actual" >> $seqres.full
+
+		_fail "Data integrity check failed. The atomic write was torn."
+	fi
+}
+
+prep_mixed_mapping() {
+	$XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full
+	local off=0
+	local mapping=""
+
+	local operations=("W" "H" "U")
+	local num_blocks=$((awu_max / blksz))
+	for ((i=0; i<num_blocks; i++)); do
+		local index=$((RANDOM % ${#operations[@]}))
+		local map="${operations[$index]}"
+		local mapping="${mapping}${map}"
+
+		case "$map" in
+			"W")
+				$XFS_IO_PROG -dc "pwrite -S 0x61 -b $blksz $off $blksz" $testfile > /dev/null
+				;;
+			"H")
+				# No operation needed for hole
+				;;
+			"U")
+				$XFS_IO_PROG -c "falloc $off $blksz" $testfile >> /dev/null
+				;;
+		esac
+		off=$((off + blksz))
+	done
+
+	echo "+ + Mixed mapping prep done. Full mapping pattern: $mapping" >> $seqres.full
+
+	sync $testfile
+}
+
+verify_atomic_write() {
+	if [[ "$1" == "shutdown" ]]
+	then
+		local do_shutdown=1
+	fi
+
+	test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed"
+
+	if [[ $do_shutdown -eq "1" ]]
+	then
+		echo "Shutting down filesystem" >> $seqres.full
+		_scratch_shutdown >> $seqres.full
+		_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed for Test-3"
+	fi
+
+	check_data_integrity
+}
+
+mixed_mapping_test() {
+	prep_mixed_mapping
+
+	echo "+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full
+	bytes_written=$($XFS_IO_PROG -dc "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \
+		        grep wrote | awk -F'[/ ]' '{print $2}')
+
+	verify_atomic_write $1
+}
+
+testfile=$SCRATCH_MNT/testfile
+touch $testfile
+
+awu_max=$(_get_atomic_write_unit_max $testfile)
+blksz=$(_get_block_size $SCRATCH_MNT)
+
+# Create an expected pattern to compare with
+$XFS_IO_PROG -tc "pwrite -b $awu_max 0 $awu_max" $testfile >> $seqres.full
+expected=$(_hexdump $testfile)
+echo "# Expected file contents:" >> $seqres.full
+echo "$expected" >> $seqres.full
+echo >> $seqres.full
+
+echo "# Test 1: Do O_DSYNC atomic write on random mixed mapping:" >> $seqres.full
+echo >> $seqres.full
+for ((iteration=1; iteration<=10; iteration++)); do
+	echo "=== Mixed Mapping Test Iteration $iteration ===" >> $seqres.full
+
+	echo "+ Testing without shutdown..." >> $seqres.full
+	mixed_mapping_test
+	echo "Passed!" >> $seqres.full
+
+	echo "+ Testing with sudden shutdown..." >> $seqres.full
+	mixed_mapping_test "shutdown"
+	echo "Passed!" >> $seqres.full
+
+	echo "Iteration $iteration completed: OK" >> $seqres.full
+	echo >> $seqres.full
+done
+echo "# Test 1: Do O_SYNC atomic write on random mixed mapping (10 iterations): OK" >> $seqres.full
+
+
+echo >> $seqres.full
+echo "# Test 2: Do extending O_SYNC atomic writes: " >> $seqres.full
+bytes_written=$($XFS_IO_PROG -dstc "pwrite -A -V1 -b $awu_max 0 $awu_max" $testfile | \
+                grep wrote | awk -F'[/ ]' '{print $2}')
+verify_atomic_write "shutdown"
+echo "# Test 2: Do extending O_SYNC atomic writes: OK" >> $seqres.full
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/generic/1228.out b/tests/generic/1228.out
new file mode 100644
index 00000000..1baffa91
--- /dev/null
+++ b/tests/generic/1228.out
@@ -0,0 +1,2 @@
+QA output created by 1228
+Silence is golden
-- 
2.49.0
Re: [PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests
Posted by Darrick J. Wong 2 months, 3 weeks ago 
On Sat, Jul 12, 2025 at 07:42:49PM +0530, Ojaswin Mujoo wrote:
> This adds various atomic write multi-fsblock stresst tests
> with mixed mappings and O_SYNC, to ensure the data and metadata
> is atomically persisted even if there is a shutdown.
> 
> Suggested-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
> Signed-off-by: Ojaswin Mujoo <ojaswin@linux.ibm.com>
> ---
>  tests/generic/1228     | 139 +++++++++++++++++++++++++++++++++++++++++
>  tests/generic/1228.out |   2 +
>  2 files changed, 141 insertions(+)
>  create mode 100755 tests/generic/1228
>  create mode 100644 tests/generic/1228.out
> 
> diff --git a/tests/generic/1228 b/tests/generic/1228
> new file mode 100755
> index 00000000..3f9a6af1
> --- /dev/null
> +++ b/tests/generic/1228
> @@ -0,0 +1,139 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2025 IBM Corporation. All Rights Reserved.
> +#
> +# FS QA Test 1228
> +#
> +# Atomic write multi-fsblock data integrity tests with mixed mappings
> +# and O_SYNC
> +#
> +. ./common/preamble
> +. ./common/atomicwrites
> +_begin_fstest auto quick rw atomicwrites
> +
> +_require_scratch_write_atomic_multi_fsblock
> +_require_atomic_write_test_commands
> +_require_scratch_shutdown
> +_require_xfs_io_command "truncate"
> +
> +_scratch_mkfs >> $seqres.full
> +_scratch_mount >> $seqres.full
> +
> +check_data_integrity() {
> +	actual=$(_hexdump $testfile)
> +	if [[ "$expected" != "$actual" ]]
> +	then
> +		echo "Integrity check failed"
> +		echo "Integrity check failed" >> $seqres.full
> +		echo "# Expected file contents:" >> $seqres.full
> +		echo "$expected" >> $seqres.full
> +		echo "# Actual file contents:" >> $seqres.full
> +		echo "$actual" >> $seqres.full
> +
> +		_fail "Data integrity check failed. The atomic write was torn."
> +	fi
> +}
> +
> +prep_mixed_mapping() {
> +	$XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full
> +	local off=0
> +	local mapping=""
> +
> +	local operations=("W" "H" "U")
> +	local num_blocks=$((awu_max / blksz))
> +	for ((i=0; i<num_blocks; i++)); do
> +		local index=$((RANDOM % ${#operations[@]}))
> +		local map="${operations[$index]}"
> +		local mapping="${mapping}${map}"
> +
> +		case "$map" in
> +			"W")
> +				$XFS_IO_PROG -dc "pwrite -S 0x61 -b $blksz $off $blksz" $testfile > /dev/null
> +				;;
> +			"H")
> +				# No operation needed for hole
> +				;;
> +			"U")
> +				$XFS_IO_PROG -c "falloc $off $blksz" $testfile >> /dev/null
> +				;;
> +		esac
> +		off=$((off + blksz))
> +	done
> +
> +	echo "+ + Mixed mapping prep done. Full mapping pattern: $mapping" >> $seqres.full
> +
> +	sync $testfile
> +}
> +
> +verify_atomic_write() {
> +	if [[ "$1" == "shutdown" ]]
> +	then
> +		local do_shutdown=1
> +	fi
> +
> +	test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed"
> +
> +	if [[ $do_shutdown -eq "1" ]]
> +	then
> +		echo "Shutting down filesystem" >> $seqres.full
> +		_scratch_shutdown >> $seqres.full
> +		_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed for Test-3"
> +	fi
> +
> +	check_data_integrity
> +}
> +
> +mixed_mapping_test() {
> +	prep_mixed_mapping
> +
> +	echo "+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full
> +	bytes_written=$($XFS_IO_PROG -dc "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \
> +		        grep wrote | awk -F'[/ ]' '{print $2}')
> +
> +	verify_atomic_write $1

The shutdown happens after the synchronous write completes?  If so, then
what part of recovery is this testing?

--D

> +}
> +
> +testfile=$SCRATCH_MNT/testfile
> +touch $testfile
> +
> +awu_max=$(_get_atomic_write_unit_max $testfile)
> +blksz=$(_get_block_size $SCRATCH_MNT)
> +
> +# Create an expected pattern to compare with
> +$XFS_IO_PROG -tc "pwrite -b $awu_max 0 $awu_max" $testfile >> $seqres.full
> +expected=$(_hexdump $testfile)
> +echo "# Expected file contents:" >> $seqres.full
> +echo "$expected" >> $seqres.full
> +echo >> $seqres.full
> +
> +echo "# Test 1: Do O_DSYNC atomic write on random mixed mapping:" >> $seqres.full
> +echo >> $seqres.full
> +for ((iteration=1; iteration<=10; iteration++)); do
> +	echo "=== Mixed Mapping Test Iteration $iteration ===" >> $seqres.full
> +
> +	echo "+ Testing without shutdown..." >> $seqres.full
> +	mixed_mapping_test
> +	echo "Passed!" >> $seqres.full
> +
> +	echo "+ Testing with sudden shutdown..." >> $seqres.full
> +	mixed_mapping_test "shutdown"
> +	echo "Passed!" >> $seqres.full
> +
> +	echo "Iteration $iteration completed: OK" >> $seqres.full
> +	echo >> $seqres.full
> +done
> +echo "# Test 1: Do O_SYNC atomic write on random mixed mapping (10 iterations): OK" >> $seqres.full
> +
> +
> +echo >> $seqres.full
> +echo "# Test 2: Do extending O_SYNC atomic writes: " >> $seqres.full
> +bytes_written=$($XFS_IO_PROG -dstc "pwrite -A -V1 -b $awu_max 0 $awu_max" $testfile | \
> +                grep wrote | awk -F'[/ ]' '{print $2}')
> +verify_atomic_write "shutdown"
> +echo "# Test 2: Do extending O_SYNC atomic writes: OK" >> $seqres.full
> +
> +# success, all done
> +echo "Silence is golden"
> +status=0
> +exit
> +
> diff --git a/tests/generic/1228.out b/tests/generic/1228.out
> new file mode 100644
> index 00000000..1baffa91
> --- /dev/null
> +++ b/tests/generic/1228.out
> @@ -0,0 +1,2 @@
> +QA output created by 1228
> +Silence is golden
> -- 
> 2.49.0
> 
>
Re: [PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests
Posted by Ojaswin Mujoo 2 months, 2 weeks ago 
On Thu, Jul 17, 2025 at 09:35:10AM -0700, Darrick J. Wong wrote:

<snip>

> > +verify_atomic_write() {
> > +	if [[ "$1" == "shutdown" ]]
> > +	then
> > +		local do_shutdown=1
> > +	fi
> > +
> > +	test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed"
> > +
> > +	if [[ $do_shutdown -eq "1" ]]
> > +	then
> > +		echo "Shutting down filesystem" >> $seqres.full
> > +		_scratch_shutdown >> $seqres.full
> > +		_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed for Test-3"
> > +	fi
> > +
> > +	check_data_integrity
> > +}
> > +
> > +mixed_mapping_test() {
> > +	prep_mixed_mapping
> > +
> > +	echo "+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full
> > +	bytes_written=$($XFS_IO_PROG -dc "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \
> > +		        grep wrote | awk -F'[/ ]' '{print $2}')
> > +
> > +	verify_atomic_write $1
> 
> The shutdown happens after the synchronous write completes?  If so, then
> what part of recovery is this testing?
> 
> --D

Right, it is mostly inspired by [1] where sometimes isize update could
be lost after dio completion. Although this might not exactly be
affected by atomic writes, we added it here out of caution.

[1] https://lore.kernel.org/fstests/434beffaf18d39f898518ea9eb1cea4548e77c3a.1695383715.git.ritesh.list@gmail.com/
> 
> > +}
> > +
Re: [PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests
Posted by Darrick J. Wong 2 months, 2 weeks ago 
On Wed, Jul 23, 2025 at 07:23:58PM +0530, Ojaswin Mujoo wrote:
> On Thu, Jul 17, 2025 at 09:35:10AM -0700, Darrick J. Wong wrote:
> 
> <snip>
> 
> > > +verify_atomic_write() {
> > > +	if [[ "$1" == "shutdown" ]]
> > > +	then
> > > +		local do_shutdown=1
> > > +	fi
> > > +
> > > +	test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed"
> > > +
> > > +	if [[ $do_shutdown -eq "1" ]]
> > > +	then
> > > +		echo "Shutting down filesystem" >> $seqres.full
> > > +		_scratch_shutdown >> $seqres.full
> > > +		_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed for Test-3"
> > > +	fi
> > > +
> > > +	check_data_integrity
> > > +}
> > > +
> > > +mixed_mapping_test() {
> > > +	prep_mixed_mapping
> > > +
> > > +	echo "+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full
> > > +	bytes_written=$($XFS_IO_PROG -dc "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \
> > > +		        grep wrote | awk -F'[/ ]' '{print $2}')
> > > +
> > > +	verify_atomic_write $1
> > 
> > The shutdown happens after the synchronous write completes?  If so, then
> > what part of recovery is this testing?
> > 
> > --D
> 
> Right, it is mostly inspired by [1] where sometimes isize update could
> be lost after dio completion. Although this might not exactly be
> affected by atomic writes, we added it here out of caution.
> 
> [1] https://lore.kernel.org/fstests/434beffaf18d39f898518ea9eb1cea4548e77c3a.1695383715.git.ritesh.list@gmail.com/

Ah, so we're racing with background log flush then.  Would it improve
the potential failure detection rate to call shutdown right after the
pwrite, e.g.

$XFS_IO_PROG -dxc "pwrite -DA..." -c 'shutdown' $testfile

It can take a few milliseconds to walk down the bash functions and
fork/exec another child process.

--D

> > > +}
> > > +
>
Re: [PATCH v3 07/13] generic/1228: Add atomic write multi-fsblock O_[D]SYNC tests
Posted by Ojaswin Mujoo 1 month, 3 weeks ago 
On Wed, Jul 23, 2025 at 07:54:23AM -0700, Darrick J. Wong wrote:
> On Wed, Jul 23, 2025 at 07:23:58PM +0530, Ojaswin Mujoo wrote:
> > On Thu, Jul 17, 2025 at 09:35:10AM -0700, Darrick J. Wong wrote:
> > 
> > <snip>
> > 
> > > > +verify_atomic_write() {
> > > > +	if [[ "$1" == "shutdown" ]]
> > > > +	then
> > > > +		local do_shutdown=1
> > > > +	fi
> > > > +
> > > > +	test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed"
> > > > +
> > > > +	if [[ $do_shutdown -eq "1" ]]
> > > > +	then
> > > > +		echo "Shutting down filesystem" >> $seqres.full
> > > > +		_scratch_shutdown >> $seqres.full
> > > > +		_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed for Test-3"
> > > > +	fi
> > > > +
> > > > +	check_data_integrity
> > > > +}
> > > > +
> > > > +mixed_mapping_test() {
> > > > +	prep_mixed_mapping
> > > > +
> > > > +	echo "+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full
> > > > +	bytes_written=$($XFS_IO_PROG -dc "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \
> > > > +		        grep wrote | awk -F'[/ ]' '{print $2}')
> > > > +
> > > > +	verify_atomic_write $1
> > > 
> > > The shutdown happens after the synchronous write completes?  If so, then
> > > what part of recovery is this testing?
> > > 
> > > --D
> > 
> > Right, it is mostly inspired by [1] where sometimes isize update could
> > be lost after dio completion. Although this might not exactly be
> > affected by atomic writes, we added it here out of caution.
> > 
> > [1] https://lore.kernel.org/fstests/434beffaf18d39f898518ea9eb1cea4548e77c3a.1695383715.git.ritesh.list@gmail.com/
> 
> Ah, so we're racing with background log flush then.  Would it improve
> the potential failure detection rate to call shutdown right after the
> pwrite, e.g.
> 
> $XFS_IO_PROG -dxc "pwrite -DA..." -c 'shutdown' $testfile
> 
> It can take a few milliseconds to walk down the bash functions and
> fork/exec another child process.

Sounds good, I can make that change.

Thanks!
> 
> --D
> 
> > > > +}
> > > > +
> >

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.