Currently, all tests are in the "auto" group. This is a little bit pointless.
OTOH, we need a group for the tests that we can automatically run during
"make check" each time, too. Tests in this new group are supposed to run
with every possible QEMU configuration, for example they must run with every
QEMU binary (also non-x86), without failing when an optional features is
missing (but reporting "skip" is ok), and be able to run on all kind of host
filesystems and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial
list has been determined by running the iotests with non-x86 QEMU targets
and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
macOS and FreeBSD).
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/qemu-iotests/group | 177 +++++++++++++++++++++------------------
1 file changed, 95 insertions(+), 82 deletions(-)
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 7ac9a5ea4a..a8ca5fd782 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -1,8 +1,21 @@
#
# QA groups control file
# Defines test groups
+#
+# Some notes about the groups:
+#
# - do not start group names with a digit
#
+# - quick : Tests in this group should finish within some few seconds.
+#
+# - img : Tests in this group can be used to excercise the qemu-img tool.
+#
+# - auto : Tests in this group are used during "make check" and should be
+# runnable in any case. That means they should run with every QEMU binary
+# (also non-x86), with every QEMU configuration (i.e. must not fail if
+# an optional feature is not compiled in - but reporting a "skip" is ok),
+# and work all kind of host filesystems and users (e.g. "nobody" or "root").
+#
#
# test-group association ... one line per test
@@ -32,11 +45,11 @@
023 rw auto
024 rw backing auto quick
025 rw auto quick
-026 rw blkdbg auto
+026 rw blkdbg
027 rw auto quick
-028 rw backing auto quick
+028 rw backing quick
029 rw auto quick
-030 rw auto backing
+030 rw backing
031 rw auto quick
032 rw auto quick
033 rw auto quick
@@ -46,35 +59,35 @@
037 rw auto backing quick
038 rw auto backing quick
039 rw auto quick
-040 rw auto
-041 rw auto backing
+040 rw
+041 rw backing
042 rw auto quick
043 rw auto backing
-044 rw auto
-045 rw auto quick
+044 rw
+045 rw quick
046 rw auto aio quick
047 rw auto quick
048 img auto quick
049 rw auto
050 rw auto backing quick
-051 rw auto
+051 rw
052 rw auto backing quick
053 rw auto quick
054 rw auto quick
-055 rw auto
-056 rw auto backing
-057 rw auto
-058 rw auto quick
+055 rw
+056 rw backing
+057 rw
+058 rw quick
059 rw auto quick
060 rw auto quick
061 rw auto
062 rw auto quick
063 rw auto quick
064 rw auto quick
-065 rw auto quick
+065 rw quick
066 rw auto quick
-067 rw auto quick
-068 rw auto quick
+067 rw quick
+068 rw quick
069 rw auto quick
070 rw auto quick
071 rw auto quick
@@ -91,18 +104,18 @@
082 rw auto quick
083 rw auto
084 img auto quick
-085 rw auto
+085 rw
086 rw auto quick
-087 rw auto quick
+087 rw quick
088 rw auto quick
089 rw auto quick
090 rw auto quick
091 rw auto migration
092 rw auto quick
-093 auto
+093 throttle
094 rw auto quick
-095 rw auto quick
-096 rw auto quick
+095 rw quick
+096 rw quick
097 rw auto backing
098 rw auto backing quick
099 rw auto quick
@@ -118,60 +131,60 @@
109 rw auto
110 rw auto backing quick
111 rw auto quick
-112 rw auto
+112 rw
113 rw auto quick
114 rw auto quick
-115 rw auto
+115 rw
116 rw auto quick
117 rw auto
-118 rw auto
+118 rw
119 rw auto quick
120 rw auto quick
-121 rw auto
+121 rw
122 rw auto
123 rw auto quick
-124 rw auto backing
-125 rw auto
+124 rw backing
+125 rw
126 rw auto backing
-127 rw auto backing quick
+127 rw backing quick
128 rw auto quick
-129 rw auto quick
+129 rw quick
130 rw auto quick
131 rw auto quick
-132 rw auto quick
+132 rw quick
133 auto quick
134 rw auto quick
135 rw auto
-136 rw auto
+136 rw
137 rw auto
138 rw auto quick
-139 rw auto quick
+139 rw quick
140 rw auto quick
141 rw auto quick
142 auto
143 auto quick
-144 rw auto quick
-145 auto quick
+144 rw quick
+145 quick
146 auto quick
-147 auto
-148 rw auto quick
-149 rw auto sudo
+147 img
+148 rw quick
+149 rw sudo
150 rw auto quick
-151 rw auto
-152 rw auto quick
-153 rw auto quick
+151 rw
+152 rw quick
+153 rw quick
154 rw auto backing quick
-155 rw auto
+155 rw
156 rw auto quick
-157 auto
+157 quick
158 rw auto quick
159 rw auto quick
160 rw auto quick
161 rw auto quick
-162 auto quick
-163 rw auto
-165 rw auto quick
-169 rw auto quick migration
+162 quick
+163 rw
+165 rw quick
+169 rw quick migration
170 rw auto quick
171 rw auto quick
172 auto
@@ -180,72 +193,72 @@
175 auto quick
176 rw auto backing
177 rw auto quick
-178 auto
+178 img
179 rw auto quick
181 rw auto migration
-182 rw auto quick
-183 rw auto migration
+182 rw quick
+183 rw migration
184 rw auto quick
-185 rw auto
+185 rw
186 rw auto
187 rw auto
-188 rw auto quick
-189 rw auto
+188 rw quick
+189 rw
190 rw auto quick
191 rw auto
192 rw auto quick
-194 rw auto migration quick
+194 rw migration quick
195 rw auto quick
-196 rw auto quick migration
+196 rw quick migration
197 rw auto quick
-198 rw auto
-199 rw auto migration
-200 rw auto
+198 rw
+199 rw migration
+200 rw
201 rw auto migration
-202 rw auto quick
-203 rw auto migration
-204 rw auto quick
-205 rw auto quick
-206 rw auto
+202 rw quick
+203 rw migration
+204 rw quick
+205 rw quick
+206 rw
207 rw auto
-208 rw auto quick
-209 rw auto quick
+208 rw quick
+209 rw quick
210 rw auto
211 rw auto quick
212 rw auto quick
213 rw auto quick
214 rw auto
215 rw auto quick
-216 rw auto quick
+216 rw quick
217 rw auto quick
-218 rw auto quick
-219 rw auto
+218 rw quick
+219 rw
220 rw auto
221 rw auto quick
-222 rw auto quick
-223 rw auto quick
-224 rw auto quick
+222 rw quick
+223 rw quick
+224 rw quick
225 rw auto quick
226 auto quick
-227 auto quick
-228 rw auto quick
+227 quick
+228 rw quick
229 auto quick
231 auto quick
-232 auto quick
+232 quick
233 auto quick
-234 auto quick migration
-235 auto quick
-236 auto quick
+234 quick migration
+235 quick
+236 quick
237 rw auto quick
-238 auto quick
+238 quick
239 rw auto quick
-240 auto quick
+240 quick
241 rw auto quick
-242 rw auto quick
+242 rw quick
243 rw auto quick
244 rw auto quick
-245 rw auto
-246 rw auto quick
-247 rw auto quick
-248 rw auto quick
+245 rw
+246 rw quick
+247 rw quick
+248 rw quick
249 rw auto quick
--
2.21.0
Thomas Huth <thuth@redhat.com> writes: > Currently, all tests are in the "auto" group. This is a little bit pointless. > OTOH, we need a group for the tests that we can automatically run during > "make check" each time, too. Tests in this new group are supposed to run > with every possible QEMU configuration, for example they must run with every > QEMU binary (also non-x86), without failing when an optional features is > missing (but reporting "skip" is ok), and be able to run on all kind of host > filesystems and users (i.e. also as "nobody" or "root"). > So let's use the "auto" group for this class of tests now. The initial > list has been determined by running the iotests with non-x86 QEMU targets > and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including > macOS and FreeBSD). I wonder whether we should additionally limit "make check" to "quick" tests. How slow are the non-quick auto tests for you?
On 07/05/2019 15.22, Markus Armbruster wrote: > Thomas Huth <thuth@redhat.com> writes: > >> Currently, all tests are in the "auto" group. This is a little bit pointless. >> OTOH, we need a group for the tests that we can automatically run during >> "make check" each time, too. Tests in this new group are supposed to run >> with every possible QEMU configuration, for example they must run with every >> QEMU binary (also non-x86), without failing when an optional features is >> missing (but reporting "skip" is ok), and be able to run on all kind of host >> filesystems and users (i.e. also as "nobody" or "root"). >> So let's use the "auto" group for this class of tests now. The initial >> list has been determined by running the iotests with non-x86 QEMU targets >> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >> macOS and FreeBSD). > > I wonder whether we should additionally limit "make check" to "quick" > tests. How slow are the non-quick auto tests for you? I already sorted out some of the tests that run veeeery long, since the run time on gitlab, cirrus-ci and travis is limited. "make check-block" currently takes 3 minutes on my laptop, I think that's still ok? When I run the tests from the auto group that are not in the quick group, I currently get: 003 1s ... 007 2s ... 013 5s ... 014 15s ... 015 9s ... 022 1s ... 023 18s ... 043 0s ... 049 3s ... 061 4s ... 079 2s ... 080 4s ... 091 1s ... 097 2s ... 104 0s ... 117 0s ... 122 8s ... 126 1s ... 137 1s ... 142 5s ... 172 3s ... 174 0s ... 176 5s ... 181 2s ... 186 2s ... 187 0s ... 191 5s ... 201 2s ... 214 0s ... That looks reasonable to me. Thomas
On 5/7/19 10:22 AM, Thomas Huth wrote: > On 07/05/2019 15.22, Markus Armbruster wrote: >> Thomas Huth <thuth@redhat.com> writes: >> >>> Currently, all tests are in the "auto" group. This is a little bit pointless. >>> OTOH, we need a group for the tests that we can automatically run during >>> "make check" each time, too. Tests in this new group are supposed to run >>> with every possible QEMU configuration, for example they must run with every >>> QEMU binary (also non-x86), without failing when an optional features is >>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>> filesystems and users (i.e. also as "nobody" or "root"). >>> So let's use the "auto" group for this class of tests now. The initial >>> list has been determined by running the iotests with non-x86 QEMU targets >>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>> macOS and FreeBSD). >> >> I wonder whether we should additionally limit "make check" to "quick" >> tests. How slow are the non-quick auto tests for you? > > I already sorted out some of the tests that run veeeery long, since the > run time on gitlab, cirrus-ci and travis is limited. "make check-block" > currently takes 3 minutes on my laptop, I think that's still ok? > > When I run the tests from the auto group that are not in the quick > group, I currently get: > My personal threshold is about 5 seconds for quick, so: > 003 1s ... > 007 2s ... Should these be moved to quick? > 013 5s ... this one is borderline > 014 15s ... > 015 9s ... Definitely not quick, but if you think they are still okay for auto, I can live with that. > 022 1s ... Another candidate for quick? > 023 18s ... Even longer than 14. Okay for auto? etc. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Eric Blake <eblake@redhat.com> writes: > On 5/7/19 10:22 AM, Thomas Huth wrote: >> On 07/05/2019 15.22, Markus Armbruster wrote: >>> Thomas Huth <thuth@redhat.com> writes: >>> >>>> Currently, all tests are in the "auto" group. This is a little bit pointless. >>>> OTOH, we need a group for the tests that we can automatically run during >>>> "make check" each time, too. Tests in this new group are supposed to run >>>> with every possible QEMU configuration, for example they must run with every >>>> QEMU binary (also non-x86), without failing when an optional features is >>>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>>> filesystems and users (i.e. also as "nobody" or "root"). >>>> So let's use the "auto" group for this class of tests now. The initial >>>> list has been determined by running the iotests with non-x86 QEMU targets >>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>>> macOS and FreeBSD). >>> >>> I wonder whether we should additionally limit "make check" to "quick" >>> tests. How slow are the non-quick auto tests for you? >> >> I already sorted out some of the tests that run veeeery long, since the >> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> currently takes 3 minutes on my laptop, I think that's still ok? >> >> When I run the tests from the auto group that are not in the quick >> group, I currently get: >> > > My personal threshold is about 5 seconds for quick, so: > >> 003 1s ... >> 007 2s ... > > Should these be moved to quick? > >> 013 5s ... > > this one is borderline > >> 014 15s ... >> 015 9s ... > > Definitely not quick, but if you think they are still okay for auto, I > can live with that. > >> 022 1s ... > > Another candidate for quick? > >> 023 18s ... > > Even longer than 14. Okay for auto? > > etc. Kevin, Max?
On 07/05/2019 17.50, Eric Blake wrote: > On 5/7/19 10:22 AM, Thomas Huth wrote: >> On 07/05/2019 15.22, Markus Armbruster wrote: >>> Thomas Huth <thuth@redhat.com> writes: >>> >>>> Currently, all tests are in the "auto" group. This is a little bit pointless. >>>> OTOH, we need a group for the tests that we can automatically run during >>>> "make check" each time, too. Tests in this new group are supposed to run >>>> with every possible QEMU configuration, for example they must run with every >>>> QEMU binary (also non-x86), without failing when an optional features is >>>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>>> filesystems and users (i.e. also as "nobody" or "root"). >>>> So let's use the "auto" group for this class of tests now. The initial >>>> list has been determined by running the iotests with non-x86 QEMU targets >>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>>> macOS and FreeBSD). >>> >>> I wonder whether we should additionally limit "make check" to "quick" >>> tests. How slow are the non-quick auto tests for you? >> >> I already sorted out some of the tests that run veeeery long, since the >> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> currently takes 3 minutes on my laptop, I think that's still ok? >> >> When I run the tests from the auto group that are not in the quick >> group, I currently get: >> > > My personal threshold is about 5 seconds for quick, so: > >> 003 1s ... >> 007 2s ... > > Should these be moved to quick? I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...? >> 013 5s ... > > this one is borderline > >> 014 15s ... >> 015 9s ... > > Definitely not quick, but if you think they are still okay for auto, I > can live with that. > >> 022 1s ... > > Another candidate for quick? > >> 023 18s ... > > Even longer than 14. Okay for auto? I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later. Thomas
On 08/05/2019 07.47, Thomas Huth wrote: > On 07/05/2019 17.50, Eric Blake wrote: >> On 5/7/19 10:22 AM, Thomas Huth wrote: >>> On 07/05/2019 15.22, Markus Armbruster wrote: >>>> Thomas Huth <thuth@redhat.com> writes: >>>> >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless. >>>>> OTOH, we need a group for the tests that we can automatically run during >>>>> "make check" each time, too. Tests in this new group are supposed to run >>>>> with every possible QEMU configuration, for example they must run with every >>>>> QEMU binary (also non-x86), without failing when an optional features is >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>>>> filesystems and users (i.e. also as "nobody" or "root"). >>>>> So let's use the "auto" group for this class of tests now. The initial >>>>> list has been determined by running the iotests with non-x86 QEMU targets >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>>>> macOS and FreeBSD). >>>> >>>> I wonder whether we should additionally limit "make check" to "quick" >>>> tests. How slow are the non-quick auto tests for you? >>> >>> I already sorted out some of the tests that run veeeery long, since the >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >>> currently takes 3 minutes on my laptop, I think that's still ok? >>> >>> When I run the tests from the auto group that are not in the quick >>> group, I currently get: >>> >> >> My personal threshold is about 5 seconds for quick, so: >> >>> 003 1s ... >>> 007 2s ... >> >> Should these be moved to quick? > > I'll leave that decision up to the blocklayer folks ... I thought that > there might have been a different reason that these have not been put > into "quick" yet...? > >>> 013 5s ... >> >> this one is borderline >> >>> 014 15s ... >>> 015 9s ... >> >> Definitely not quick, but if you think they are still okay for auto, I >> can live with that. >> >>> 022 1s ... >> >> Another candidate for quick? >> >>> 023 18s ... >> >> Even longer than 14. Okay for auto? > > I think I'd give it a try. If people are complaining later that "make > check" is running now way too long, we still can refine the list later. Thinking about this again, "make check" now runs quite a bit longer indeed. So I now rather tend to remove the tests that run longer than 5s from the auto group instead... I think I'll send a v4 of this patch where I'll remove them from the auto group. Thomas
Thomas Huth <thuth@redhat.com> writes: > Thinking about this again, "make check" now runs quite a bit longer > indeed. So I now rather tend to remove the tests that run longer than 5s > from the auto group instead... I think I'll send a v4 of this patch > where I'll remove them from the auto group. Appreciated! I'm all for automated testing, but "make check" is already slow enough to make certain kinds of work painful. Not quite slow enough to make me renege on commitments and go on a quest to speed it up.
Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: > On 08/05/2019 07.47, Thomas Huth wrote: > > On 07/05/2019 17.50, Eric Blake wrote: > >> On 5/7/19 10:22 AM, Thomas Huth wrote: > >>> On 07/05/2019 15.22, Markus Armbruster wrote: > >>>> Thomas Huth <thuth@redhat.com> writes: > >>>> > >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless. > >>>>> OTOH, we need a group for the tests that we can automatically run during > >>>>> "make check" each time, too. Tests in this new group are supposed to run > >>>>> with every possible QEMU configuration, for example they must run with every > >>>>> QEMU binary (also non-x86), without failing when an optional features is > >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host > >>>>> filesystems and users (i.e. also as "nobody" or "root"). > >>>>> So let's use the "auto" group for this class of tests now. The initial > >>>>> list has been determined by running the iotests with non-x86 QEMU targets > >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including > >>>>> macOS and FreeBSD). > >>>> > >>>> I wonder whether we should additionally limit "make check" to "quick" > >>>> tests. How slow are the non-quick auto tests for you? > >>> > >>> I already sorted out some of the tests that run veeeery long, since the > >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" > >>> currently takes 3 minutes on my laptop, I think that's still ok? > >>> > >>> When I run the tests from the auto group that are not in the quick > >>> group, I currently get: > >>> > >> > >> My personal threshold is about 5 seconds for quick, so: > >> > >>> 003 1s ... > >>> 007 2s ... > >> > >> Should these be moved to quick? > > > > I'll leave that decision up to the blocklayer folks ... I thought that > > there might have been a different reason that these have not been put > > into "quick" yet...? > > > >>> 013 5s ... > >> > >> this one is borderline > >> > >>> 014 15s ... > >>> 015 9s ... > >> > >> Definitely not quick, but if you think they are still okay for auto, I > >> can live with that. > >> > >>> 022 1s ... > >> > >> Another candidate for quick? > >> > >>> 023 18s ... > >> > >> Even longer than 14. Okay for auto? > > > > I think I'd give it a try. If people are complaining later that "make > > check" is running now way too long, we still can refine the list later. > > Thinking about this again, "make check" now runs quite a bit longer > indeed. So I now rather tend to remove the tests that run longer than 5s > from the auto group instead... I think I'll send a v4 of this patch > where I'll remove them from the auto group. I don't think time is everything. We should also consider how much the tests contribute to basic code coverage. There is no point in removing a test from the list because it takes 10 seconds, but if I split it in two tests taking each 5 seconds, you would include both halves. For example, 030, 040 and 041 are not that quick (14/11/42 seconds, respectively), but they are the most important tests for block jobs and covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at least. Kevin
Kevin Wolf <kwolf@redhat.com> writes: > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: >> On 08/05/2019 07.47, Thomas Huth wrote: >> > On 07/05/2019 17.50, Eric Blake wrote: >> >> On 5/7/19 10:22 AM, Thomas Huth wrote: >> >>> On 07/05/2019 15.22, Markus Armbruster wrote: >> >>>> Thomas Huth <thuth@redhat.com> writes: >> >>>> >> >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless. >> >>>>> OTOH, we need a group for the tests that we can automatically run during >> >>>>> "make check" each time, too. Tests in this new group are supposed to run >> >>>>> with every possible QEMU configuration, for example they must run with every >> >>>>> QEMU binary (also non-x86), without failing when an optional features is >> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host >> >>>>> filesystems and users (i.e. also as "nobody" or "root"). >> >>>>> So let's use the "auto" group for this class of tests now. The initial >> >>>>> list has been determined by running the iotests with non-x86 QEMU targets >> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >> >>>>> macOS and FreeBSD). >> >>>> >> >>>> I wonder whether we should additionally limit "make check" to "quick" >> >>>> tests. How slow are the non-quick auto tests for you? >> >>> >> >>> I already sorted out some of the tests that run veeeery long, since the >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> >>> currently takes 3 minutes on my laptop, I think that's still ok? >> >>> >> >>> When I run the tests from the auto group that are not in the quick >> >>> group, I currently get: >> >>> >> >> >> >> My personal threshold is about 5 seconds for quick, so: >> >> >> >>> 003 1s ... >> >>> 007 2s ... >> >> >> >> Should these be moved to quick? >> > >> > I'll leave that decision up to the blocklayer folks ... I thought that >> > there might have been a different reason that these have not been put >> > into "quick" yet...? >> > >> >>> 013 5s ... >> >> >> >> this one is borderline >> >> >> >>> 014 15s ... >> >>> 015 9s ... >> >> >> >> Definitely not quick, but if you think they are still okay for auto, I >> >> can live with that. >> >> >> >>> 022 1s ... >> >> >> >> Another candidate for quick? >> >> >> >>> 023 18s ... >> >> >> >> Even longer than 14. Okay for auto? >> > >> > I think I'd give it a try. If people are complaining later that "make >> > check" is running now way too long, we still can refine the list later. >> >> Thinking about this again, "make check" now runs quite a bit longer >> indeed. So I now rather tend to remove the tests that run longer than 5s >> from the auto group instead... I think I'll send a v4 of this patch >> where I'll remove them from the auto group. > > I don't think time is everything. We should also consider how much > the tests contribute to basic code coverage. There is no point in > removing a test from the list because it takes 10 seconds, but if I > split it in two tests taking each 5 seconds, you would include both > halves. > > For example, 030, 040 and 041 are not that quick (14/11/42 seconds, > respectively), but they are the most important tests for block jobs and > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at > least. Yes, we want block jobs covered. However, 42 seconds is a lot indeed. Can you think of ways to get a useful part of the full coverage in five seconds or less?
Am 10.05.2019 um 17:29 hat Markus Armbruster geschrieben: > Kevin Wolf <kwolf@redhat.com> writes: > > > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: > >> On 08/05/2019 07.47, Thomas Huth wrote: > >> > On 07/05/2019 17.50, Eric Blake wrote: > >> >> On 5/7/19 10:22 AM, Thomas Huth wrote: > >> >>> On 07/05/2019 15.22, Markus Armbruster wrote: > >> >>>> Thomas Huth <thuth@redhat.com> writes: > >> >>>> > >> >>>>> Currently, all tests are in the "auto" group. This is a little bit pointless. > >> >>>>> OTOH, we need a group for the tests that we can automatically run during > >> >>>>> "make check" each time, too. Tests in this new group are supposed to run > >> >>>>> with every possible QEMU configuration, for example they must run with every > >> >>>>> QEMU binary (also non-x86), without failing when an optional features is > >> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind of host > >> >>>>> filesystems and users (i.e. also as "nobody" or "root"). > >> >>>>> So let's use the "auto" group for this class of tests now. The initial > >> >>>>> list has been determined by running the iotests with non-x86 QEMU targets > >> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including > >> >>>>> macOS and FreeBSD). > >> >>>> > >> >>>> I wonder whether we should additionally limit "make check" to "quick" > >> >>>> tests. How slow are the non-quick auto tests for you? > >> >>> > >> >>> I already sorted out some of the tests that run veeeery long, since the > >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" > >> >>> currently takes 3 minutes on my laptop, I think that's still ok? > >> >>> > >> >>> When I run the tests from the auto group that are not in the quick > >> >>> group, I currently get: > >> >>> > >> >> > >> >> My personal threshold is about 5 seconds for quick, so: > >> >> > >> >>> 003 1s ... > >> >>> 007 2s ... > >> >> > >> >> Should these be moved to quick? > >> > > >> > I'll leave that decision up to the blocklayer folks ... I thought that > >> > there might have been a different reason that these have not been put > >> > into "quick" yet...? > >> > > >> >>> 013 5s ... > >> >> > >> >> this one is borderline > >> >> > >> >>> 014 15s ... > >> >>> 015 9s ... > >> >> > >> >> Definitely not quick, but if you think they are still okay for auto, I > >> >> can live with that. > >> >> > >> >>> 022 1s ... > >> >> > >> >> Another candidate for quick? > >> >> > >> >>> 023 18s ... > >> >> > >> >> Even longer than 14. Okay for auto? > >> > > >> > I think I'd give it a try. If people are complaining later that "make > >> > check" is running now way too long, we still can refine the list later. > >> > >> Thinking about this again, "make check" now runs quite a bit longer > >> indeed. So I now rather tend to remove the tests that run longer than 5s > >> from the auto group instead... I think I'll send a v4 of this patch > >> where I'll remove them from the auto group. > > > > I don't think time is everything. We should also consider how much > > the tests contribute to basic code coverage. There is no point in > > removing a test from the list because it takes 10 seconds, but if I > > split it in two tests taking each 5 seconds, you would include both > > halves. > > > > For example, 030, 040 and 041 are not that quick (14/11/42 seconds, > > respectively), but they are the most important tests for block jobs and > > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at > > least. > > Yes, we want block jobs covered. However, 42 seconds is a lot indeed. > Can you think of ways to get a useful part of the full coverage in five > seconds or less? I'm not even sure which part of the test is taking so long. Maybe it's only one of the 88 test functions that's taking long. Or we're duplicating a lot of tests because we test all combinations of setups where testing only certain combinations would give enough coverage. Answering this will probably take some effort checking what the exact cases tested are and how long each one takes. Kevin
Currently, all tests are in the "auto" group. This is a little bit pointless.
OTOH, we need a group for the tests that we can automatically run during
"make check" each time, too. Tests in this new group are supposed to run
with every possible QEMU configuration, for example they must run with every
QEMU binary (also non-x86), without failing when an optional features is
missing (but reporting "skip" is ok), run with qcow2 and file protocol,
run reasonable fast, and be able to run on all kind of host filesystems
and users (i.e. also as "nobody" or "root").
So let's use the "auto" group for this class of tests now. The initial
list has been determined by running the iotests with non-x86 QEMU targets
and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including
macOS and FreeBSD).
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
v4:
- Dropped the tests from the "auto" group that run longer than 5s
- Dropped also the tests that do not support the qcow2 format
Alex, could you replace the patch in your patch series, too, please?
tests/qemu-iotests/group | 262 +++++++++++++++++++++------------------
1 file changed, 138 insertions(+), 124 deletions(-)
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 7ac9a5ea4a..a2260d0899 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -1,8 +1,22 @@
#
# QA groups control file
# Defines test groups
+#
+# Some notes about the groups:
+#
# - do not start group names with a digit
#
+# - quick : Tests in this group should finish within some few seconds.
+#
+# - img : Tests in this group can be used to excercise the qemu-img tool.
+#
+# - auto : Tests in this group are used during "make check" and should be
+# runnable in any case. That means they should run with every QEMU binary
+# (also non-x86), with every QEMU configuration (i.e. must not fail if
+# an optional feature is not compiled in - but reporting a "skip" is ok),
+# work with the qcow2 format and file protocol, run reasonably fast (< 10s)
+# and work all kind of host filesystems and users (e.g. "nobody" or "root").
+#
#
# test-group association ... one line per test
@@ -20,8 +34,8 @@
011 rw auto quick
012 auto quick
013 rw auto
-014 rw auto
-015 rw snapshot auto
+014 rw
+015 rw snapshot
# 016 was removed, do not reuse
017 rw backing auto quick
018 rw backing auto quick
@@ -29,14 +43,14 @@
020 rw backing auto quick
021 io auto quick
022 rw snapshot auto
-023 rw auto
+023 rw
024 rw backing auto quick
025 rw auto quick
-026 rw blkdbg auto
+026 rw blkdbg
027 rw auto quick
-028 rw backing auto quick
+028 rw backing quick
029 rw auto quick
-030 rw auto backing
+030 rw backing
031 rw auto quick
032 rw auto quick
033 rw auto quick
@@ -46,206 +60,206 @@
037 rw auto backing quick
038 rw auto backing quick
039 rw auto quick
-040 rw auto
-041 rw auto backing
+040 rw
+041 rw backing
042 rw auto quick
043 rw auto backing
-044 rw auto
-045 rw auto quick
+044 rw
+045 rw quick
046 rw auto aio quick
047 rw auto quick
048 img auto quick
049 rw auto
050 rw auto backing quick
-051 rw auto
+051 rw
052 rw auto backing quick
053 rw auto quick
054 rw auto quick
-055 rw auto
-056 rw auto backing
-057 rw auto
-058 rw auto quick
-059 rw auto quick
+055 rw
+056 rw backing
+057 rw
+058 rw quick
+059 rw quick
060 rw auto quick
061 rw auto
062 rw auto quick
063 rw auto quick
-064 rw auto quick
-065 rw auto quick
+064 rw quick
+065 rw quick
066 rw auto quick
-067 rw auto quick
-068 rw auto quick
+067 rw quick
+068 rw quick
069 rw auto quick
-070 rw auto quick
+070 rw quick
071 rw auto quick
072 rw auto quick
073 rw auto quick
074 rw auto quick
-075 rw auto quick
-076 auto
-077 rw auto quick
-078 rw auto quick
+075 rw quick
+076 io
+077 rw quick
+078 rw quick
079 rw auto
080 rw auto
-081 rw auto quick
+081 rw quick
082 rw auto quick
-083 rw auto
-084 img auto quick
-085 rw auto
+083 rw
+084 img quick
+085 rw
086 rw auto quick
-087 rw auto quick
-088 rw auto quick
+087 rw quick
+088 rw quick
089 rw auto quick
090 rw auto quick
091 rw auto migration
-092 rw auto quick
-093 auto
-094 rw auto quick
-095 rw auto quick
-096 rw auto quick
+092 rw quick
+093 throttle
+094 rw quick
+095 rw quick
+096 rw quick
097 rw auto backing
098 rw auto backing quick
099 rw auto quick
# 100 was removed, do not reuse
-101 rw auto quick
+101 rw quick
102 rw auto quick
103 rw auto quick
104 rw auto
105 rw auto quick
-106 rw auto quick
+106 rw quick
107 rw auto quick
108 rw auto quick
-109 rw auto
+109 rw
110 rw auto backing quick
111 rw auto quick
-112 rw auto
-113 rw auto quick
+112 rw
+113 rw quick
114 rw auto quick
-115 rw auto
-116 rw auto quick
+115 rw
+116 rw quick
117 rw auto
-118 rw auto
-119 rw auto quick
+118 rw
+119 rw quick
120 rw auto quick
-121 rw auto
+121 rw
122 rw auto
-123 rw auto quick
-124 rw auto backing
-125 rw auto
+123 rw quick
+124 rw backing
+125 rw
126 rw auto backing
-127 rw auto backing quick
-128 rw auto quick
-129 rw auto quick
+127 rw backing quick
+128 rw quick
+129 rw quick
130 rw auto quick
-131 rw auto quick
-132 rw auto quick
+131 rw quick
+132 rw quick
133 auto quick
134 rw auto quick
-135 rw auto
-136 rw auto
+135 rw
+136 rw
137 rw auto
138 rw auto quick
-139 rw auto quick
+139 rw quick
140 rw auto quick
141 rw auto quick
142 auto
143 auto quick
-144 rw auto quick
-145 auto quick
-146 auto quick
-147 auto
-148 rw auto quick
-149 rw auto sudo
+144 rw quick
+145 quick
+146 quick
+147 img
+148 rw quick
+149 rw sudo
150 rw auto quick
-151 rw auto
-152 rw auto quick
-153 rw auto quick
+151 rw
+152 rw quick
+153 rw quick
154 rw auto backing quick
-155 rw auto
+155 rw
156 rw auto quick
-157 auto
+157 quick
158 rw auto quick
159 rw auto quick
-160 rw auto quick
+160 rw quick
161 rw auto quick
-162 auto quick
-163 rw auto
-165 rw auto quick
-169 rw auto quick migration
+162 quick
+163 rw
+165 rw quick
+169 rw quick migration
170 rw auto quick
-171 rw auto quick
+171 rw quick
172 auto
-173 rw auto
+173 rw
174 auto
-175 auto quick
+175 quick
176 rw auto backing
177 rw auto quick
-178 auto
+178 img
179 rw auto quick
181 rw auto migration
-182 rw auto quick
-183 rw auto migration
+182 rw quick
+183 rw migration
184 rw auto quick
-185 rw auto
+185 rw
186 rw auto
187 rw auto
-188 rw auto quick
-189 rw auto
+188 rw quick
+189 rw
190 rw auto quick
191 rw auto
192 rw auto quick
-194 rw auto migration quick
+194 rw migration quick
195 rw auto quick
-196 rw auto quick migration
+196 rw quick migration
197 rw auto quick
-198 rw auto
-199 rw auto migration
-200 rw auto
+198 rw
+199 rw migration
+200 rw
201 rw auto migration
-202 rw auto quick
-203 rw auto migration
-204 rw auto quick
-205 rw auto quick
-206 rw auto
-207 rw auto
-208 rw auto quick
-209 rw auto quick
-210 rw auto
-211 rw auto quick
-212 rw auto quick
-213 rw auto quick
+202 rw quick
+203 rw migration
+204 rw quick
+205 rw quick
+206 rw
+207 rw
+208 rw quick
+209 rw quick
+210 rw
+211 rw quick
+212 rw quick
+213 rw quick
214 rw auto
215 rw auto quick
-216 rw auto quick
+216 rw quick
217 rw auto quick
-218 rw auto quick
-219 rw auto
+218 rw quick
+219 rw
220 rw auto
-221 rw auto quick
-222 rw auto quick
-223 rw auto quick
-224 rw auto quick
-225 rw auto quick
+221 rw quick
+222 rw quick
+223 rw quick
+224 rw quick
+225 rw quick
226 auto quick
-227 auto quick
-228 rw auto quick
+227 quick
+228 rw quick
229 auto quick
-231 auto quick
-232 auto quick
+231 quick
+232 quick
233 auto quick
-234 auto quick migration
-235 auto quick
-236 auto quick
-237 rw auto quick
-238 auto quick
-239 rw auto quick
-240 auto quick
-241 rw auto quick
-242 rw auto quick
+234 quick migration
+235 quick
+236 quick
+237 rw quick
+238 quick
+239 rw quick
+240 quick
+241 rw quick
+242 rw quick
243 rw auto quick
244 rw auto quick
-245 rw auto
-246 rw auto quick
-247 rw auto quick
-248 rw auto quick
+245 rw
+246 rw quick
+247 rw quick
+248 rw quick
249 rw auto quick
--
2.21.0
© 2016 - 2026 Red Hat, Inc.