1. Patchew
  2. QEMU
  3. View series

[PATCH v2] tests/acceptance: allow control over tags during check-acceptance

Willian Rampazzo posted 1 patch 2 years, 9 months ago
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20210707152454.246519-1-willianr@redhat.com
There is a newer version of this series
docs/devel/testing.rst |  8 ++++++++
tests/Makefile.include | 12 +++++++++---
2 files changed, 17 insertions(+), 3 deletions(-)
[PATCH v2] tests/acceptance: allow control over tags during check-acceptance
Posted by Willian Rampazzo 2 years, 9 months ago 
Although it is possible to run a specific test using the avocado
command-line, a user may want to use a specific tag while running the
``make check-acceptance`` during the development or debugging.

This allows using the AVOCADO_TAGS environment variable where the user
takes total control of which tests should run based on the tags defined.

This also makes the check-acceptance command flexible to restrict tests
based on tags while running on CI.

e.g.:

AVOCADO_TAGS="foo bar baz" make check-acceptance

Signed-off-by: Willian Rampazzo <willianr@redhat.com>
---
 docs/devel/testing.rst |  8 ++++++++
 tests/Makefile.include | 12 +++++++++---
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 4e42392810..f6e4fc08a7 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -760,6 +760,14 @@ in the current directory, tagged as "quick", run:
 
   avocado run -t quick .
 
+It is also possible to run tests based on tags using the
+``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
+variable:
+
+.. code::
+
+   AVOCADO_TAGS=quick make check-acceptance
+
 The ``avocado_qemu.Test`` base test class
 -----------------------------------------
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8f220e15d1..d464f06e4a 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # Any number of command separated loggers are accepted.  For more
 # information please refer to "avocado --help".
 AVOCADO_SHOW=app
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+ifndef AVOCADO_TAGS
+	AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \
+						 $(filter %-softmmu,$(TARGETS)))
+else
+	AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
+endif
 
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
@@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
-            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
-            $(AVOCADO_TAGS) \
+            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
+			--filter-by-tags-include-empty-key) \
+            $(AVOCADO_CMDLINE_TAGS) \
             $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
-- 
2.31.1
Re: [PATCH v2] tests/acceptance: allow control over tags during check-acceptance
Posted by Wainer dos Santos Moschetta 2 years, 9 months ago 
Hi,

On 7/7/21 12:24 PM, Willian Rampazzo wrote:
> Although it is possible to run a specific test using the avocado
> command-line, a user may want to use a specific tag while running the
> ``make check-acceptance`` during the development or debugging.
I always wanted that feature, thanks for implementing it!
>
> This allows using the AVOCADO_TAGS environment variable where the user
> takes total control of which tests should run based on the tags defined.
>
> This also makes the check-acceptance command flexible to restrict tests
> based on tags while running on CI.
>
> e.g.:
>
> AVOCADO_TAGS="foo bar baz" make check-acceptance

If I run `AVOCADO_TAGS="machine:malta arch:mips" make check-acceptance` 
it is going to get the tests with 'machine:malta' *or* 'arch:mips' tags, 
whereas with AVOCADO_TAGS="machine:malta,arch:mips" it gets those with 
'machine:malta' *and* 'arch:mips', i.e., the semantic is different. I 
think it deserves a mention in the docs/devel/testing.rst, maybe linking 
to Avocado's doc [1] as well.

[1] 
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html

>
> Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> ---
>   docs/devel/testing.rst |  8 ++++++++
>   tests/Makefile.include | 12 +++++++++---
>   2 files changed, 17 insertions(+), 3 deletions(-)

Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

>
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 4e42392810..f6e4fc08a7 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -760,6 +760,14 @@ in the current directory, tagged as "quick", run:
>   
>     avocado run -t quick .
>   
> +It is also possible to run tests based on tags using the
> +``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
> +variable:
> +
> +.. code::
> +
> +   AVOCADO_TAGS=quick make check-acceptance
> +
>   The ``avocado_qemu.Test`` base test class
>   -----------------------------------------
>   
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 8f220e15d1..d464f06e4a 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>   # Any number of command separated loggers are accepted.  For more
>   # information please refer to "avocado --help".
>   AVOCADO_SHOW=app
> -AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
> +ifndef AVOCADO_TAGS
> +	AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \
> +						 $(filter %-softmmu,$(TARGETS)))
> +else
> +	AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS))
> +endif
>   
>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>   	$(call quiet-command, \
> @@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>   	$(call quiet-command, \
>               $(TESTS_VENV_DIR)/bin/python -m avocado \
>               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
> -            $(AVOCADO_TAGS) \
> +            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> +			--filter-by-tags-include-empty-key) \
> +            $(AVOCADO_CMDLINE_TAGS) \
>               $(if $(GITLAB_CI),,--failfast) tests/acceptance, \
>               "AVOCADO", "tests/acceptance")
>
Re: [PATCH v2] tests/acceptance: allow control over tags during check-acceptance
Posted by Willian Rampazzo 2 years, 9 months ago 
Hi, Wainer,

On Fri, Jul 9, 2021 at 4:32 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi,
>
> On 7/7/21 12:24 PM, Willian Rampazzo wrote:
> > Although it is possible to run a specific test using the avocado
> > command-line, a user may want to use a specific tag while running the
> > ``make check-acceptance`` during the development or debugging.
> I always wanted that feature, thanks for implementing it!

I'm glad it is helpful.

> >
> > This allows using the AVOCADO_TAGS environment variable where the user
> > takes total control of which tests should run based on the tags defined.
> >
> > This also makes the check-acceptance command flexible to restrict tests
> > based on tags while running on CI.
> >
> > e.g.:
> >
> > AVOCADO_TAGS="foo bar baz" make check-acceptance
>
> If I run `AVOCADO_TAGS="machine:malta arch:mips" make check-acceptance`
> it is going to get the tests with 'machine:malta' *or* 'arch:mips' tags,
> whereas with AVOCADO_TAGS="machine:malta,arch:mips" it gets those with
> 'machine:malta' *and* 'arch:mips', i.e., the semantic is different. I
> think it deserves a mention in the docs/devel/testing.rst, maybe linking
> to Avocado's doc [1] as well.

Okay, that is fair. I'll summarize something in the testing
documentation and point to the Avocado doc to avoid duplication.

>
> [1]
> https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
>
> >
> > Signed-off-by: Willian Rampazzo <willianr@redhat.com>
> > ---
> >   docs/devel/testing.rst |  8 ++++++++
> >   tests/Makefile.include | 12 +++++++++---
> >   2 files changed, 17 insertions(+), 3 deletions(-)
>
> Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>
> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>

Thanks for testing and reviewing.

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.