1. Patchew
  2. QEMU
  3. [PATCH v2 0/3] iotests improvements for mirror-sparse and others
  4. View patch

[PATCH v2 2/3] iotests: Improve mirror-sparse on ext4

Eric Blake posted 3 patches 4 months, 2 weeks ago
Maintainers: Kevin Wolf <kwolf@redhat.com>, Hanna Reitz <hreitz@redhat.com>
[PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Posted by Eric Blake 4 months, 2 weeks ago 
Fiona reported that an ext4 filesystem on top of LVM can sometimes
report over-allocation to du (based on the hueristics the filesystem
is making while observing the contents being mirrored); even though
the contents and actual size matched, about 50% of the time the size
reported by disk_usage was too large by 4k, failing the test.  In
auditing other iotests, this is a common problem we've had to deal
with.

Reported-by: Fiona Ebner <f.ebner@proxmox.com>
Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
Signed-off-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
Tested-by: Fiona Ebner <f.ebner@proxmox.com>

---
v2: Drop sync for ZFS
---
 tests/qemu-iotests/tests/mirror-sparse | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/tests/mirror-sparse b/tests/qemu-iotests/tests/mirror-sparse
index 8c52a4e2448..338d6cfbb35 100755
--- a/tests/qemu-iotests/tests/mirror-sparse
+++ b/tests/qemu-iotests/tests/mirror-sparse
@@ -96,10 +96,12 @@ _send_qemu_cmd $h1 '{"execute": "blockdev-del", "arguments":
                 {"node-name": "dst"}}' 'return' \
                   | _filter_block_job_offset | _filter_block_job_len
 $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
+# Some filesystems can fudge allocations for various reasons; rather
+# than expecting precise 2M and 20M images, it is better to allow for slop.
 result=$(disk_usage $TEST_IMG)
 if test $result -lt $((3*1024*1024)); then
     actual=sparse
-elif test $result = $((20*1024*1024)); then
+elif test $result -gt $((19*1024*1024)); then
     actual=full
 else
     actual=unknown
-- 
2.49.0
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Posted by Eric Blake 4 months, 1 week ago 
On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem
> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test.  In
> auditing other iotests, this is a common problem we've had to deal
> with.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> 
> ---
> v2: Drop sync for ZFS
> ---
>  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)

Squashing this in since Markus is reporting failures even with this
patch applied, and I'm lacking enough context to see what else might
be interfering:

diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
index 39e3196c811..a30e52de41d 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
 elif test $result -gt $((19*1024*1024)); then
     actual=full
 else
-    actual=unknown
+    actual="unexpected size ($result)"
 fi
 echo "Destination is $actual; expected $expected"
 }


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Posted by Eric Blake 4 months, 1 week ago 
On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
> > report over-allocation to du (based on the hueristics the filesystem
> > is making while observing the contents being mirrored); even though
> > the contents and actual size matched, about 50% of the time the size
> > reported by disk_usage was too large by 4k, failing the test.  In
> > auditing other iotests, this is a common problem we've had to deal
> > with.
> > 
> > Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> > Signed-off-by: Eric Blake <eblake@redhat.com>
> > Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> > Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> > 
> > ---
> > v2: Drop sync for ZFS
> > ---
> >  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> Squashing this in since Markus is reporting failures even with this
> patch applied, and I'm lacking enough context to see what else might
> be interfering:
> 
> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
> index 39e3196c811..a30e52de41d 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
>  elif test $result -gt $((19*1024*1024)); then
>      actual=full
>  else
> -    actual=unknown
> +    actual="unexpected size ($result)"

I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
install on Fedora) - it turns out that XFS rounds disk usage in terms
of allocation groups which default to a size of 1M; so I need to treat
exactly 3M as sparse, not unknown.  I'm also squashing in:

diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
index a30e52de41d..cfcaa600ab4 100755
--- i/tests/qemu-iotests/tests/mirror-sparse
+++ w/tests/qemu-iotests/tests/mirror-sparse
@@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
 # Some filesystems can fudge allocations for various reasons; rather
 # than expecting precise 2M and 20M images, it is better to allow for slop.
 result=$(disk_usage $TEST_IMG)
-if test $result -lt $((3*1024*1024)); then
+if test $result -lt $((4*1024*1024)); then
     actual=sparse
 elif test $result -gt $((19*1024*1024)); then
     actual=full


-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Posted by Markus Armbruster 4 months, 1 week ago 
Eric Blake <eblake@redhat.com> writes:

> On Wed, May 28, 2025 at 10:45:35AM -0500, Eric Blake wrote:
>> On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
>> > Fiona reported that an ext4 filesystem on top of LVM can sometimes
>> > report over-allocation to du (based on the hueristics the filesystem
>> > is making while observing the contents being mirrored); even though
>> > the contents and actual size matched, about 50% of the time the size
>> > reported by disk_usage was too large by 4k, failing the test.  In
>> > auditing other iotests, this is a common problem we've had to deal
>> > with.
>> > 
>> > Reported-by: Fiona Ebner <f.ebner@proxmox.com>
>> > Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
>> > Signed-off-by: Eric Blake <eblake@redhat.com>
>> > Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
>> > Tested-by: Fiona Ebner <f.ebner@proxmox.com>
>> > 
>> > ---
>> > v2: Drop sync for ZFS
>> > ---
>> >  tests/qemu-iotests/tests/mirror-sparse | 4 +++-
>> >  1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> Squashing this in since Markus is reporting failures even with this
>> patch applied, and I'm lacking enough context to see what else might
>> be interfering:
>> 
>> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
>> index 39e3196c811..a30e52de41d 100755
>> --- i/tests/qemu-iotests/tests/mirror-sparse
>> +++ w/tests/qemu-iotests/tests/mirror-sparse
>> @@ -105,7 +105,7 @@ if test $result -lt $((3*1024*1024)); then
>>  elif test $result -gt $((19*1024*1024)); then
>>      actual=full
>>  else
>> -    actual=unknown
>> +    actual="unexpected size ($result)"
>
> I reproduced Markus' situation on XFS (since XFS is easier than ZFS to
> install on Fedora) - it turns out that XFS rounds disk usage in terms
> of allocation groups which default to a size of 1M; so I need to treat
> exactly 3M as sparse, not unknown.  I'm also squashing in:
>
> diff --git i/tests/qemu-iotests/tests/mirror-sparse w/tests/qemu-iotests/tests/mirror-sparse
> index a30e52de41d..cfcaa600ab4 100755
> --- i/tests/qemu-iotests/tests/mirror-sparse
> +++ w/tests/qemu-iotests/tests/mirror-sparse
> @@ -100,7 +100,7 @@ $QEMU_IMG compare -U -f $IMGFMT -F raw $TEST_IMG.base $TEST_IMG
>  # Some filesystems can fudge allocations for various reasons; rather
>  # than expecting precise 2M and 20M images, it is better to allow for slop.
>  result=$(disk_usage $TEST_IMG)
> -if test $result -lt $((3*1024*1024)); then
> +if test $result -lt $((4*1024*1024)); then
>      actual=sparse
>  elif test $result -gt $((19*1024*1024)); then
>      actual=full

This one-liner appears to fix it for me.  Thanks!
Re: [PATCH v2 2/3] iotests: Improve mirror-sparse on ext4
Posted by Eric Blake 4 months, 2 weeks ago 
On Fri, May 23, 2025 at 11:27:22AM -0500, Eric Blake wrote:
> Fiona reported that an ext4 filesystem on top of LVM can sometimes
> report over-allocation to du (based on the hueristics the filesystem

and despite my other edits, I still missed fixing my typo on heuristics

> is making while observing the contents being mirrored); even though
> the contents and actual size matched, about 50% of the time the size
> reported by disk_usage was too large by 4k, failing the test.  In
> auditing other iotests, this is a common problem we've had to deal
> with.
> 
> Reported-by: Fiona Ebner <f.ebner@proxmox.com>
> Fixes: c0ddcb2c ("tests: Add iotest mirror-sparse for recent patches")
> Signed-off-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: Fiona Ebner <f.ebner@proxmox.com>
> Tested-by: Fiona Ebner <f.ebner@proxmox.com>
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.
Virtualization:  qemu.org | libguestfs.org

Patchew 2.1-devel-b67e4c2

© 2016 - 2025 Red Hat, Inc.