1. Patchew
  2. QEMU
  3. [Qemu-devel] [PULL 00/15] Python queue, 2018-10-30
  4. View patch

[Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests

Eduardo Habkost posted 15 patches 7 years, 3 months ago
[Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests
Posted by Eduardo Habkost 7 years, 3 months ago 
From: Cleber Rosa <crosa@redhat.com>

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.

The tests/requirements.txt is supposed to contain the Python
requirements that should be added to the venv created by check-venv.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-2-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/requirements.txt |  3 +++
 tests/Makefile.include | 26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 tests/requirements.txt

diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 0000000000..d39f9d1576
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
diff --git a/tests/Makefile.include b/tests/Makefile.include
index f77a495109..eabc1da2f3 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
+	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
 	@echo
 	@echo "Please note that HTML reports do not regenerate if the unit tests"
@@ -899,6 +900,30 @@ check-decodetree:
           ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
           TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+
+$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
+ifeq ($(.SHELLSTATUS),0)
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+	$(call quiet-command, \
+            $(PYTHON) -m venv --system-site-packages $@, \
+            VENV, $@)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
+            PIP, $(TESTS_VENV_REQ))
+	$(call quiet-command, touch $@)
+else
+$(TESTS_VENV_DIR):
+	$(error "venv directory for tests requires Python 3")
+endif
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -912,6 +937,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
+	rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
-- 
2.18.0.rc1.1.g3f1ff2140
Re: [Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests
Posted by Peter Maydell 7 years, 3 months ago 
On 31 October 2018 at 00:31, Eduardo Habkost <ehabkost@redhat.com> wrote:
> From: Cleber Rosa <crosa@redhat.com>
>
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
>
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
>
> The tests/requirements.txt is supposed to contain the Python
> requirements that should be added to the venv created by check-venv.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Message-Id: <20181018153134.8493-2-crosa@redhat.com>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>         @echo " $(MAKE) check-block          Run block tests"
>         @echo " $(MAKE) check-tcg            Run TCG tests"
>         @echo " $(MAKE) check-report.html    Generates an HTML test report"
> +       @echo " $(MAKE) check-venv           Creates a Python venv for tests"
>         @echo " $(MAKE) check-clean          Clean the tests"
>         @echo
>         @echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -899,6 +900,30 @@ check-decodetree:
>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>            TEST, decodetree.py)
>
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> +
> +$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> +ifeq ($(.SHELLSTATUS),0)
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +       $(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \
> +            VENV, $@)
> +       $(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> +            PIP, $(TESTS_VENV_REQ))
> +       $(call quiet-command, touch $@)
> +else
> +$(TESTS_VENV_DIR):
> +       $(error "venv directory for tests requires Python 3")
> +endif
> +
> +check-venv: $(TESTS_VENV_DIR)

Hi -- this seems to be causing one of the travis configs to fail:

https://travis-ci.org/qemu/qemu/jobs/451311466

The config includes "--python=/usr/bin/python3", but the build
fails with
     CHK version_gen.h
/home/travis/build/qemu/qemu/tests/Makefile.include:928: *** "venv
directory for tests requires Python 3". Stop.

Would you mind having a look at what's happening there ?

thanks
-- PMM
Re: [Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests
Posted by Philippe Mathieu-Daudé 7 years, 3 months ago 
On 6/11/18 14:10, Peter Maydell wrote:
> On 31 October 2018 at 00:31, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> From: Cleber Rosa <crosa@redhat.com>
>>
>> A number of QEMU tests are written in Python, and may benefit
>> from an untainted Python venv.
>>
>> By using make rules, tests that depend on specific Python libs
>> can set that rule as a requirement, along with rules that require
>> the presence or installation of specific libraries.
>>
>> The tests/requirements.txt is supposed to contain the Python
>> requirements that should be added to the venv created by check-venv.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
>> Message-Id: <20181018153134.8493-2-crosa@redhat.com>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> 
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -12,6 +12,7 @@ check-help:
>>          @echo " $(MAKE) check-block          Run block tests"
>>          @echo " $(MAKE) check-tcg            Run TCG tests"
>>          @echo " $(MAKE) check-report.html    Generates an HTML test report"
>> +       @echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>          @echo " $(MAKE) check-clean          Clean the tests"
>>          @echo
>>          @echo "Please note that HTML reports do not regenerate if the unit tests"
>> @@ -899,6 +900,30 @@ check-decodetree:
>>             ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>             TEST, decodetree.py)
>>
>> +# Python venv for running tests
>> +
>> +.PHONY: check-venv
>> +
>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
>> +
>> +$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> +ifeq ($(.SHELLSTATUS),0)
>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>> +       $(call quiet-command, \
>> +            $(PYTHON) -m venv --system-site-packages $@, \
>> +            VENV, $@)
>> +       $(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>> +            PIP, $(TESTS_VENV_REQ))
>> +       $(call quiet-command, touch $@)
>> +else
>> +$(TESTS_VENV_DIR):
>> +       $(error "venv directory for tests requires Python 3")
>> +endif
>> +
>> +check-venv: $(TESTS_VENV_DIR)
> 
> Hi -- this seems to be causing one of the travis configs to fail:
> 
> https://travis-ci.org/qemu/qemu/jobs/451311466
> 
> The config includes "--python=/usr/bin/python3", but the build
> fails with
>       CHK version_gen.h
> /home/travis/build/qemu/qemu/tests/Makefile.include:928: *** "venv
> directory for tests requires Python 3". Stop.
> 
> Would you mind having a look at what's happening there ?

I tested the failure using 'make check-venv PYTHON=python2' and the 
success using 'make check-venv PYTHON=python3' but didn't think of the 
default...

The quicker fix is to force PYTHON in .travis.yml, I'll prepare a patch.

> 
> thanks
> -- PMM
>
[Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Eduardo Habkost 7 years, 3 months ago 
The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
seems to provide an older version.  Change the existing rules to
use command output instead of exit code, to make it compatible
with older GNU make versions.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
I think that's the cause of the Travis failures.  I have
submitted a test job right now, at:
  https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
Let's see if it fixes the issue.
---
 tests/Makefile.include | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index d2e577eabb..074eece558 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # information please refer to "avocado --help".
 AVOCADO_SHOW=none
 
-$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
-ifeq ($(.SHELLSTATUS),0)
+PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
+ifeq ($(PYTHON3), 1)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
             $(PYTHON) -m venv --system-site-packages $@, \
-- 
2.18.0.rc1.1.g3f1ff2140
Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Philippe Mathieu-Daudé 7 years, 3 months ago 
On 6/11/18 15:13, Eduardo Habkost wrote:
> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.

You were quicker, I just found out :)

> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>    https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.

You can run it locally:

$ make docker-image-travis
...
$ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) \
     -u 0 --entrypoint=/bin/bash qemu:travis
root@ede09efe22fd:qemu# cd build/docker_travis/
root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
check-acceptance
   VENV    qemu/build/docker_travis/tests/venv
The virtual environment was not created successfully because ensurepip 
is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

     apt-get install python3-venv

You may need to use sudo with that command.  After installing the 
python3-venv
package, recreate your virtual environment.

root@ede09efe22fd:qemu/build/docker_travis# apt install python3-pip 
python3.4-venv
root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
check-acceptance
   VENV    qemu/build/docker_travis/tests/venv
   PIP     qemu/tests/requirements.txt
   AVOCADO tests/acceptance
JOB ID     : 5e5529e79456c388e80323acdc71f3887341a498
JOB LOG    : 
qemu/build/docker_travis/tests/results/job-2018-11-06T14.26-5e5529e/job.log
  (1/6) 
qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: 
CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
  (2/6) 
qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: 
CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
  (3/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU 
binary defined or found in the source tree (0.00 s)
  (4/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
  (5/6) 
qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
  (6/6) qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 6
JOB TIME   : 0.45 s

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>   tests/Makefile.include | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>   # information please refer to "avocado --help".
>   AVOCADO_SHOW=none
>   
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>   	$(call quiet-command, \
>               $(PYTHON) -m venv --system-site-packages $@, \
>
Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Philippe Mathieu-Daudé 7 years, 3 months ago 
Hi Peter,

Can you apply this patch as a CI bug-fix?

Thanks,

Phil.

On 6/11/18 15:27, Philippe Mathieu-Daudé wrote:
> On 6/11/18 15:13, Eduardo Habkost wrote:
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
> 
> You were quicker, I just found out :)
> 
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>    https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
> 
> You can run it locally:
> 
> $ make docker-image-travis
> ...
> $ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) \
>      -u 0 --entrypoint=/bin/bash qemu:travis
> root@ede09efe22fd:qemu# cd build/docker_travis/
> root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
> check-acceptance
>    VENV    qemu/build/docker_travis/tests/venv
> The virtual environment was not created successfully because ensurepip 
> is not
> available.  On Debian/Ubuntu systems, you need to install the python3-venv
> package using the following command.
> 
>      apt-get install python3-venv
> 
> You may need to use sudo with that command.  After installing the 
> python3-venv
> package, recreate your virtual environment.
> 
> root@ede09efe22fd:qemu/build/docker_travis# apt install python3-pip 
> python3.4-venv
> root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
> check-acceptance
>    VENV    qemu/build/docker_travis/tests/venv
>    PIP     qemu/tests/requirements.txt
>    AVOCADO tests/acceptance
> JOB ID     : 5e5529e79456c388e80323acdc71f3887341a498
> JOB LOG    : 
> qemu/build/docker_travis/tests/results/job-2018-11-06T14.26-5e5529e/job.log
>   (1/6) 
> qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: 
> CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
>   (2/6) 
> qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: 
> CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
>   (3/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU 
> binary defined or found in the source tree (0.00 s)
>   (4/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
>   (5/6) 
> qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
>   (6/6) qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
> RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 6
> JOB TIME   : 0.45 s
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
>> ---
>>   tests/Makefile.include | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>   # information please refer to "avocado --help".
>>   AVOCADO_SHOW=none
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' 
>> >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if 
>> sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>       $(call quiet-command, \
>>               $(PYTHON) -m venv --system-site-packages $@, \
>>
Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Peter Maydell 7 years, 3 months ago 
On 6 November 2018 at 14:13, Eduardo Habkost <ehabkost@redhat.com> wrote:
> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.
> ---
>  tests/Makefile.include | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=none
>
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>         $(call quiet-command, \
>              $(PYTHON) -m venv --system-site-packages $@, \
> --

Thanks, applied to master as a build fix.

-- PMM
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Markus Armbruster 7 years, 3 months ago 
Eduardo Habkost <ehabkost@redhat.com> writes:

> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.
> ---
>  tests/Makefile.include | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=none
>  
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>  	$(call quiet-command, \
>              $(PYTHON) -m venv --system-site-packages $@, \

PEP 394 recommends software distributions install Python 3 into the
default path as python3, and users use that instead of python, except
for programs that are source compatible with both 2 and 3.  So, is
finding out whether python is a Python 3 really appropriate?  Why can't
we just use python3 and be done with it?

If we can't: isn't this a configure problem?
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Peter Maydell 7 years, 3 months ago 
On 7 November 2018 at 06:05, Markus Armbruster <armbru@redhat.com> wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
>
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
>> ---
>>  tests/Makefile.include | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>  # information please refer to "avocado --help".
>>  AVOCADO_SHOW=none
>>
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>       $(call quiet-command, \
>>              $(PYTHON) -m venv --system-site-packages $@, \
>
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?
>
> If we can't: isn't this a configure problem?

You can't just use python3 and be done with it because python3
might not exist, and because (as with python 2) the user might
want to tell us the path to it. You could have configure detect
whether python3 exists and set a PYTHON3 as well as a PYTHON
(plus I guess support for the user to say "my python3 is this
binary"), and then have this code in Makefile.include handle
"PYTHON3 is not set" to mean "python 3 isn't available".

thanks
-- PMM
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Eduardo Habkost 7 years, 3 months ago 
On Wed, Nov 07, 2018 at 07:05:03AM +0100, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> > seems to provide an older version.  Change the existing rules to
> > use command output instead of exit code, to make it compatible
> > with older GNU make versions.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> > I think that's the cause of the Travis failures.  I have
> > submitted a test job right now, at:
> >   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> > Let's see if it fixes the issue.
> > ---
> >  tests/Makefile.include | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index d2e577eabb..074eece558 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >  # information please refer to "avocado --help".
> >  AVOCADO_SHOW=none
> >  
> > -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> > -ifeq ($(.SHELLSTATUS),0)
> > +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> > +ifeq ($(PYTHON3), 1)
> >  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >  	$(call quiet-command, \
> >              $(PYTHON) -m venv --system-site-packages $@, \
> 
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?

Because './configure --with-python=...' exists, and I didn't want
to break it.

Now, why do we need --with-python, and why do we need to use
$(PYTHON) when running tests?  If somebody wants to use a
different Python binary when running tests, they can already use
$PATH for that.

(That's the same argument I used for iotests a while ago:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)

> 
> If we can't: isn't this a configure problem?

It is, and I think Cleber mentioned that he planned to do it in
./configure, a while ago.  But just using the python3 binary from
$PATH would be even better.

-- 
Eduardo
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Peter Maydell 7 years, 3 months ago 
On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
> Now, why do we need --with-python, and why do we need to use
> $(PYTHON) when running tests?  If somebody wants to use a
> different Python binary when running tests, they can already use
> $PATH for that.
>
> (That's the same argument I used for iotests a while ago:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)

I'm not a great fan of requiring the user to mess with their PATH
to get configure to work. Also, the first python on the path
might be the wrong one, and we don't pass PATH from configure
to make so you end up having to make sure you specify it
right in both places.

Plus we already have --with-python, so if you want to drop
it you need to deprecate it first, and you need a justification
that's strong enough to outweigh breaking users' existing
build/packaging setups and scripts...

thanks
-- PMM
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Eduardo Habkost 7 years, 3 months ago 
On Wed, Nov 07, 2018 at 01:45:35PM +0000, Peter Maydell wrote:
> On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > Now, why do we need --with-python, and why do we need to use
> > $(PYTHON) when running tests?  If somebody wants to use a
> > different Python binary when running tests, they can already use
> > $PATH for that.
> >
> > (That's the same argument I used for iotests a while ago:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)
> 
> I'm not a great fan of requiring the user to mess with their PATH
> to get configure to work. Also, the first python on the path
> might be the wrong one, and we don't pass PATH from configure
> to make so you end up having to make sure you specify it
> right in both places.

You're assuming that this will actually require some people to
mess with their $PATH because they currently don't have Python on
their $PATH.  I don't see any evidence that this is expected to
happen.  Do you?

> 
> Plus we already have --with-python, so if you want to drop
> it you need to deprecate it first, and you need a justification
> that's strong enough to outweigh breaking users' existing
> build/packaging setups and scripts...

I would really like to remove the option as soon as we start
requiring Python 3.  Let's stop reinventing solutions to problems
already addressed by PEP 394.

-- 
Eduardo
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Markus Armbruster 7 years, 3 months ago 
Eduardo Habkost <ehabkost@redhat.com> writes:

> On Wed, Nov 07, 2018 at 01:45:35PM +0000, Peter Maydell wrote:
>> On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> > Now, why do we need --with-python, and why do we need to use
>> > $(PYTHON) when running tests?  If somebody wants to use a
>> > different Python binary when running tests, they can already use
>> > $PATH for that.
>> >
>> > (That's the same argument I used for iotests a while ago:
>> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)
>> 
>> I'm not a great fan of requiring the user to mess with their PATH
>> to get configure to work. Also, the first python on the path
>> might be the wrong one, and we don't pass PATH from configure
>> to make so you end up having to make sure you specify it
>> right in both places.
>
> You're assuming that this will actually require some people to
> mess with their $PATH because they currently don't have Python on
> their $PATH.  I don't see any evidence that this is expected to
> happen.  Do you?

What makes Python so special we must provide special means to find
it off the PATH?  Why not other tools?

>> Plus we already have --with-python, so if you want to drop
>> it you need to deprecate it first, and you need a justification
>> that's strong enough to outweigh breaking users' existing
>> build/packaging setups and scripts...
>
> I would really like to remove the option as soon as we start
> requiring Python 3.  Let's stop reinventing solutions to problems
> already addressed by PEP 394.

Concur.

We should use python3 wherever we need a Python 3.  There's no point in
letting configure check whether python3 is actually Python 3.

We should use python2 wherever we need a Python 2.  This case needs to
go away (if it isn't gone already).

We may use any of python3, python, python2 wherever we can work with
both Python 3 and 2.  Simply using python3 works on sufficiently recent
hosts.  For maximum portability, we can let configure try python3, then
python.  I wouldn't bother trying python2 as well.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Cleber Rosa 7 years, 3 months ago 

On 11/7/18 1:05 AM, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
>> ---
>>  tests/Makefile.include | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>  # information please refer to "avocado --help".
>>  AVOCADO_SHOW=none
>>  
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>  	$(call quiet-command, \
>>              $(PYTHON) -m venv --system-site-packages $@, \
> 
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?
> 

I mentioned that before, when you pointed out the issue you fix here,
that configure may be the best place to get the Python version (not only
the major version) and make it available elsewhere.  Even if not used
for other purposes, it is IMO important information to show on the
resulting "configure" output.

I'm sending patches to do that in a few.

> If we can't: isn't this a configure problem?
> 

I believe adhering to PEP394 is a good thing, but even that document
recognizes that the real world is not a perfect place: "however, end
users should be aware that python refers to python3 on at least Arch
Linux".  And it only covers *nix systems, so if we hope to care for
non-*nix systems, we have to check the Python version manually.

So, I guess the safest approach from QEMU's side is to check for the
version indeed.

- Cleber.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Markus Armbruster 7 years, 3 months ago 
Cleber Rosa <crosa@redhat.com> writes:

> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>> 
>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>>> seems to provide an older version.  Change the existing rules to
>>> use command output instead of exit code, to make it compatible
>>> with older GNU make versions.
>>>
>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>> ---
>>> I think that's the cause of the Travis failures.  I have
>>> submitted a test job right now, at:
>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>>> Let's see if it fixes the issue.
>>> ---
>>>  tests/Makefile.include | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index d2e577eabb..074eece558 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>  # information please refer to "avocado --help".
>>>  AVOCADO_SHOW=none
>>>  
>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>>> -ifeq ($(.SHELLSTATUS),0)
>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>>> +ifeq ($(PYTHON3), 1)
>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>  	$(call quiet-command, \
>>>              $(PYTHON) -m venv --system-site-packages $@, \
>> 
>> PEP 394 recommends software distributions install Python 3 into the
>> default path as python3, and users use that instead of python, except
>> for programs that are source compatible with both 2 and 3.  So, is
>> finding out whether python is a Python 3 really appropriate?  Why can't
>> we just use python3 and be done with it?
>> 
>
> I mentioned that before, when you pointed out the issue you fix here,
> that configure may be the best place to get the Python version (not only
> the major version) and make it available elsewhere.  Even if not used
> for other purposes, it is IMO important information to show on the
> resulting "configure" output.
>
> I'm sending patches to do that in a few.
>
>> If we can't: isn't this a configure problem?
>> 
>
> I believe adhering to PEP394 is a good thing, but even that document
> recognizes that the real world is not a perfect place: "however, end
> users should be aware that python refers to python3 on at least Arch
> Linux".  And it only covers *nix systems, so if we hope to care for
> non-*nix systems, we have to check the Python version manually.
>
> So, I guess the safest approach from QEMU's side is to check for the
> version indeed.

If somebody can point to a system people still use where python3 doesn't
get you a Python 3, but python does, catering for such (crappy) systems
in configure makes sense.  Until then, it's a waste of brain waves and
configure run time.

PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
recent version of Arch Linux that's still crappy in this regard?
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Philippe Mathieu-Daudé 7 years, 3 months ago 
Hi Markus,


Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :

> Cleber Rosa <crosa@redhat.com> writes:
>
> > On 11/7/18 1:05 AM, Markus Armbruster wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >>
> >>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> >>> seems to provide an older version.  Change the existing rules to
> >>> use command output instead of exit code, to make it compatible
> >>> with older GNU make versions.
> >>>
> >>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> >>> ---
> >>> I think that's the cause of the Travis failures.  I have
> >>> submitted a test job right now, at:
> >>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> >>> Let's see if it fixes the issue.
> >>> ---
> >>>  tests/Makefile.include | 4 ++--
> >>>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index d2e577eabb..074eece558 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >>>  # information please refer to "avocado --help".
> >>>  AVOCADO_SHOW=none
> >>>
> >>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)'
> >/dev/null 2>&1)
> >>> -ifeq ($(.SHELLSTATUS),0)
> >>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if
> sys.version_info >= (3, 0) else 0)')
> >>> +ifeq ($(PYTHON3), 1)
> >>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>     $(call quiet-command, \
> >>>              $(PYTHON) -m venv --system-site-packages $@, \
> >>
> >> PEP 394 recommends software distributions install Python 3 into the
> >> default path as python3, and users use that instead of python, except
> >> for programs that are source compatible with both 2 and 3.  So, is
> >> finding out whether python is a Python 3 really appropriate?  Why can't
> >> we just use python3 and be done with it?
> >>
> >
> > I mentioned that before, when you pointed out the issue you fix here,
> > that configure may be the best place to get the Python version (not only
> > the major version) and make it available elsewhere.  Even if not used
> > for other purposes, it is IMO important information to show on the
> > resulting "configure" output.
> >
> > I'm sending patches to do that in a few.
> >
> >> If we can't: isn't this a configure problem?
> >>
> >
> > I believe adhering to PEP394 is a good thing, but even that document
> > recognizes that the real world is not a perfect place: "however, end
> > users should be aware that python refers to python3 on at least Arch
> > Linux".  And it only covers *nix systems, so if we hope to care for
> > non-*nix systems, we have to check the Python version manually.
> >
> > So, I guess the safest approach from QEMU's side is to check for the
> > version indeed.
>
> If somebody can point to a system people still use where python3 doesn't
> get you a Python 3, but python does, catering for such (crappy) systems
> in configure makes sense.  Until then, it's a waste of brain waves and
> configure run time.
>
> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> recent version of Arch Linux that's still crappy in this regard?
>

Arch doesn't provide python2 by default, thus python points to python3.
I think what's crappy is scripts expecting python to be python2.

>
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Markus Armbruster 7 years, 3 months ago 
Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Markus,
>
>
> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>
>> Cleber Rosa <crosa@redhat.com> writes:
>>
>> > On 11/7/18 1:05 AM, Markus Armbruster wrote:
[...]
>> >> PEP 394 recommends software distributions install Python 3 into the
>> >> default path as python3, and users use that instead of python, except
>> >> for programs that are source compatible with both 2 and 3.  So, is
>> >> finding out whether python is a Python 3 really appropriate?  Why can't
>> >> we just use python3 and be done with it?
>> >>
>> >
>> > I mentioned that before, when you pointed out the issue you fix here,
>> > that configure may be the best place to get the Python version (not only
>> > the major version) and make it available elsewhere.  Even if not used
>> > for other purposes, it is IMO important information to show on the
>> > resulting "configure" output.
>> >
>> > I'm sending patches to do that in a few.
>> >
>> >> If we can't: isn't this a configure problem?
>> >>
>> >
>> > I believe adhering to PEP394 is a good thing, but even that document
>> > recognizes that the real world is not a perfect place: "however, end
>> > users should be aware that python refers to python3 on at least Arch
>> > Linux".  And it only covers *nix systems, so if we hope to care for
>> > non-*nix systems, we have to check the Python version manually.
>> >
>> > So, I guess the safest approach from QEMU's side is to check for the
>> > version indeed.
>>
>> If somebody can point to a system people still use where python3 doesn't
>> get you a Python 3, but python does, catering for such (crappy) systems
>> in configure makes sense.  Until then, it's a waste of brain waves and
>> configure run time.
>>
>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>> recent version of Arch Linux that's still crappy in this regard?
>>
>
> Arch doesn't provide python2 by default, thus python points to python3.

That's non-crappy as long as python3 also exists, as PEP 394 recommends.
Does it?

> I think what's crappy is scripts expecting python to be python2.

No argument.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Max Reitz 7 years, 2 months ago 
On 08.11.18 13:43, Markus Armbruster wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi Markus,
>>
>>
>> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>>
>>> Cleber Rosa <crosa@redhat.com> writes:
>>>
>>>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
> [...]
>>>>> PEP 394 recommends software distributions install Python 3 into the
>>>>> default path as python3, and users use that instead of python, except
>>>>> for programs that are source compatible with both 2 and 3.  So, is
>>>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>>>> we just use python3 and be done with it?
>>>>>
>>>>
>>>> I mentioned that before, when you pointed out the issue you fix here,
>>>> that configure may be the best place to get the Python version (not only
>>>> the major version) and make it available elsewhere.  Even if not used
>>>> for other purposes, it is IMO important information to show on the
>>>> resulting "configure" output.
>>>>
>>>> I'm sending patches to do that in a few.
>>>>
>>>>> If we can't: isn't this a configure problem?
>>>>>
>>>>
>>>> I believe adhering to PEP394 is a good thing, but even that document
>>>> recognizes that the real world is not a perfect place: "however, end
>>>> users should be aware that python refers to python3 on at least Arch
>>>> Linux".  And it only covers *nix systems, so if we hope to care for
>>>> non-*nix systems, we have to check the Python version manually.
>>>>
>>>> So, I guess the safest approach from QEMU's side is to check for the
>>>> version indeed.
>>>
>>> If somebody can point to a system people still use where python3 doesn't
>>> get you a Python 3, but python does, catering for such (crappy) systems
>>> in configure makes sense.  Until then, it's a waste of brain waves and
>>> configure run time.
>>>
>>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>>> recent version of Arch Linux that's still crappy in this regard?
>>>
>>
>> Arch doesn't provide python2 by default, thus python points to python3.
> 
> That's non-crappy as long as python3 also exists, as PEP 394 recommends.
> Does it?

Sure it does.

Arch is just problematic in how it handles "python" itself.  I don't
think there is any system that has Python 3 but no "python3".

Max
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Markus Armbruster 7 years, 2 months ago 
Max Reitz <mreitz@redhat.com> writes:

> On 08.11.18 13:43, Markus Armbruster wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> 
>>> Hi Markus,
>>>
>>>
>>> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>>>
>>>> Cleber Rosa <crosa@redhat.com> writes:
>>>>
>>>>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>> [...]
>>>>>> PEP 394 recommends software distributions install Python 3 into the
>>>>>> default path as python3, and users use that instead of python, except
>>>>>> for programs that are source compatible with both 2 and 3.  So, is
>>>>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>>>>> we just use python3 and be done with it?
>>>>>>
>>>>>
>>>>> I mentioned that before, when you pointed out the issue you fix here,
>>>>> that configure may be the best place to get the Python version (not only
>>>>> the major version) and make it available elsewhere.  Even if not used
>>>>> for other purposes, it is IMO important information to show on the
>>>>> resulting "configure" output.
>>>>>
>>>>> I'm sending patches to do that in a few.
>>>>>
>>>>>> If we can't: isn't this a configure problem?
>>>>>>
>>>>>
>>>>> I believe adhering to PEP394 is a good thing, but even that document
>>>>> recognizes that the real world is not a perfect place: "however, end
>>>>> users should be aware that python refers to python3 on at least Arch
>>>>> Linux".  And it only covers *nix systems, so if we hope to care for
>>>>> non-*nix systems, we have to check the Python version manually.
>>>>>
>>>>> So, I guess the safest approach from QEMU's side is to check for the
>>>>> version indeed.
>>>>
>>>> If somebody can point to a system people still use where python3 doesn't
>>>> get you a Python 3, but python does, catering for such (crappy) systems
>>>> in configure makes sense.  Until then, it's a waste of brain waves and
>>>> configure run time.
>>>>
>>>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>>>> recent version of Arch Linux that's still crappy in this regard?
>>>>
>>>
>>> Arch doesn't provide python2 by default, thus python points to python3.
>> 
>> That's non-crappy as long as python3 also exists, as PEP 394 recommends.
>> Does it?
>
> Sure it does.
>
> Arch is just problematic in how it handles "python" itself.  I don't
> think there is any system that has Python 3 but no "python3".

Bottom line: we can safely assume that a host that has Python 3 provides
it as python3.

Corollary:

* When python3 doesn't exist, checking whether python is Python 3 is
  futile.

* Checking whether python3 really is Python 3 is silly.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Cleber Rosa 7 years, 3 months ago 

On 11/8/18 3:45 AM, Markus Armbruster wrote:
> Cleber Rosa <crosa@redhat.com> writes:
> 
>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>>> Eduardo Habkost <ehabkost@redhat.com> writes:
>>>
>>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>>>> seems to provide an older version.  Change the existing rules to
>>>> use command output instead of exit code, to make it compatible
>>>> with older GNU make versions.
>>>>
>>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> ---
>>>> I think that's the cause of the Travis failures.  I have
>>>> submitted a test job right now, at:
>>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>>>> Let's see if it fixes the issue.
>>>> ---
>>>>  tests/Makefile.include | 4 ++--
>>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index d2e577eabb..074eece558 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>>  # information please refer to "avocado --help".
>>>>  AVOCADO_SHOW=none
>>>>  
>>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>>>> -ifeq ($(.SHELLSTATUS),0)
>>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>>>> +ifeq ($(PYTHON3), 1)
>>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>>  	$(call quiet-command, \
>>>>              $(PYTHON) -m venv --system-site-packages $@, \
>>>
>>> PEP 394 recommends software distributions install Python 3 into the
>>> default path as python3, and users use that instead of python, except
>>> for programs that are source compatible with both 2 and 3.  So, is
>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>> we just use python3 and be done with it?
>>>
>>
>> I mentioned that before, when you pointed out the issue you fix here,
>> that configure may be the best place to get the Python version (not only
>> the major version) and make it available elsewhere.  Even if not used
>> for other purposes, it is IMO important information to show on the
>> resulting "configure" output.
>>
>> I'm sending patches to do that in a few.
>>
>>> If we can't: isn't this a configure problem?
>>>
>>
>> I believe adhering to PEP394 is a good thing, but even that document
>> recognizes that the real world is not a perfect place: "however, end
>> users should be aware that python refers to python3 on at least Arch
>> Linux".  And it only covers *nix systems, so if we hope to care for
>> non-*nix systems, we have to check the Python version manually.
>>
>> So, I guess the safest approach from QEMU's side is to check for the
>> version indeed.
> 
> If somebody can point to a system people still use where python3 doesn't
> get you a Python 3, but python does, catering for such (crappy) systems
> in configure makes sense.  Until then, it's a waste of brain waves and
> configure run time.
> 
> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> recent version of Arch Linux that's still crappy in this regard?
> 

I'm not taking the mission of looking for those odd distros, I agree
it's not productive.  My point is that the Python version is an
important thing to keep track of, specially with the advent of running
Python dependent tests on environments we don't fully control.

Supposing the Python version is captured and tracked by configure, then
the "python" binary name becomes a non-issue.

- Cleber.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Eduardo Habkost 7 years, 3 months ago 
On Thu, Nov 08, 2018 at 11:06:45AM -0500, Cleber Rosa wrote:
> 
> 
> On 11/8/18 3:45 AM, Markus Armbruster wrote:
> > Cleber Rosa <crosa@redhat.com> writes:
> > 
> >> On 11/7/18 1:05 AM, Markus Armbruster wrote:
> >>> Eduardo Habkost <ehabkost@redhat.com> writes:
> >>>
> >>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> >>>> seems to provide an older version.  Change the existing rules to
> >>>> use command output instead of exit code, to make it compatible
> >>>> with older GNU make versions.
> >>>>
> >>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> >>>> ---
> >>>> I think that's the cause of the Travis failures.  I have
> >>>> submitted a test job right now, at:
> >>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> >>>> Let's see if it fixes the issue.
> >>>> ---
> >>>>  tests/Makefile.include | 4 ++--
> >>>>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>>>
> >>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>>> index d2e577eabb..074eece558 100644
> >>>> --- a/tests/Makefile.include
> >>>> +++ b/tests/Makefile.include
> >>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >>>>  # information please refer to "avocado --help".
> >>>>  AVOCADO_SHOW=none
> >>>>  
> >>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> >>>> -ifeq ($(.SHELLSTATUS),0)
> >>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> >>>> +ifeq ($(PYTHON3), 1)
> >>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>>  	$(call quiet-command, \
> >>>>              $(PYTHON) -m venv --system-site-packages $@, \
> >>>
> >>> PEP 394 recommends software distributions install Python 3 into the
> >>> default path as python3, and users use that instead of python, except
> >>> for programs that are source compatible with both 2 and 3.  So, is
> >>> finding out whether python is a Python 3 really appropriate?  Why can't
> >>> we just use python3 and be done with it?
> >>>
> >>
> >> I mentioned that before, when you pointed out the issue you fix here,
> >> that configure may be the best place to get the Python version (not only
> >> the major version) and make it available elsewhere.  Even if not used
> >> for other purposes, it is IMO important information to show on the
> >> resulting "configure" output.
> >>
> >> I'm sending patches to do that in a few.
> >>
> >>> If we can't: isn't this a configure problem?
> >>>
> >>
> >> I believe adhering to PEP394 is a good thing, but even that document
> >> recognizes that the real world is not a perfect place: "however, end
> >> users should be aware that python refers to python3 on at least Arch
> >> Linux".  And it only covers *nix systems, so if we hope to care for
> >> non-*nix systems, we have to check the Python version manually.
> >>
> >> So, I guess the safest approach from QEMU's side is to check for the
> >> version indeed.
> > 
> > If somebody can point to a system people still use where python3 doesn't
> > get you a Python 3, but python does, catering for such (crappy) systems
> > in configure makes sense.  Until then, it's a waste of brain waves and
> > configure run time.
> > 
> > PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> > recent version of Arch Linux that's still crappy in this regard?
> > 
> 
> I'm not taking the mission of looking for those odd distros, I agree
> it's not productive.  My point is that the Python version is an
> important thing to keep track of, specially with the advent of running
> Python dependent tests on environments we don't fully control.
> 
> Supposing the Python version is captured and tracked by configure, then
> the "python" binary name becomes a non-issue.

I'm not sure I agree with the "is an important thing to keep
track of" part.  I don't think we'll have any need to keep track
of the Python version in shell script or makefiles after we start
requiring Python 3.

Extra cleanups (like moving version checks to ./configure) are
still welcome, but keep in mind that this will probably be thrown
away once we drop Python 2 support.

-- 
Eduardo
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Cleber Rosa 7 years, 3 months ago 

On 11/8/18 11:51 AM, Eduardo Habkost wrote:

> I'm not sure I agree with the "is an important thing to keep
> track of" part.  I don't think we'll have any need to keep track
> of the Python version in shell script or makefiles after we start
> requiring Python 3.
> 

Well, "Python 3" is not a uniform thing.  There are many changes across
the 3.x spectrum that can bite you.  I'm speaking out of the experience
with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
we should add tests for 3.7).

> Extra cleanups (like moving version checks to ./configure) are
> still welcome, but keep in mind that this will probably be thrown
> away once we drop Python 2 support.
> 

I don't think it should.  Let me give the example of a "Python 3.0" job
on Travis, related to the following snippet:

    # Python builds
    - env: CONFIG="--target-list=x86_64-softmmu"
      python:
        - "3.0"

If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
you'll see that the intended goal was missed.  That has to do with how
Travis makes the requested Python version available, and it was only
obvious because this branch prints the Python version used.

Developers writing Python code, for instance tests, may assume a given
API that doesn't exist or behave like they expect, and not knowing the
Python version used on a remote environment makes debugging extra hard.

- Cleber.
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Eduardo Habkost 7 years, 3 months ago 
On Thu, Nov 08, 2018 at 12:36:37PM -0500, Cleber Rosa wrote:
> 
> 
> On 11/8/18 11:51 AM, Eduardo Habkost wrote:
> 
> > I'm not sure I agree with the "is an important thing to keep
> > track of" part.  I don't think we'll have any need to keep track
> > of the Python version in shell script or makefiles after we start
> > requiring Python 3.
> > 
> 
> Well, "Python 3" is not a uniform thing.  There are many changes across
> the 3.x spectrum that can bite you.  I'm speaking out of the experience
> with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
> we should add tests for 3.7).

Do you expect us to need to keep track of the exact Python
version before running the Python interpreter?  Code written in
Python doesn't count, because it can simply use sys.version_info.


> 
> > Extra cleanups (like moving version checks to ./configure) are
> > still welcome, but keep in mind that this will probably be thrown
> > away once we drop Python 2 support.
> > 
> 
> I don't think it should.  Let me give the example of a "Python 3.0" job
> on Travis, related to the following snippet:
> 
>     # Python builds
>     - env: CONFIG="--target-list=x86_64-softmmu"
>       python:
>         - "3.0"
> 
> If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
> you'll see that the intended goal was missed.  That has to do with how
> Travis makes the requested Python version available, and it was only
> obvious because this branch prints the Python version used.
> 
> Developers writing Python code, for instance tests, may assume a given
> API that doesn't exist or behave like they expect, and not knowing the
> Python version used on a remote environment makes debugging extra hard.

I'm not sure I get your point.  We can surely add diagnostic
messages to show the Python version, and we can make ./configure
fail if Python 3 is unavailable.  I'm talking about adding code
to ./configure to save Python version info for makefile rules and
shell scripts.  I expect the extra code to be unnecessary in the
future.

-- 
Eduardo
Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions
Posted by Cleber Rosa 7 years, 3 months ago 

On 11/8/18 1:26 PM, Eduardo Habkost wrote:
> On Thu, Nov 08, 2018 at 12:36:37PM -0500, Cleber Rosa wrote:
>>
>>
>> On 11/8/18 11:51 AM, Eduardo Habkost wrote:
>>
>>> I'm not sure I agree with the "is an important thing to keep
>>> track of" part.  I don't think we'll have any need to keep track
>>> of the Python version in shell script or makefiles after we start
>>> requiring Python 3.
>>>
>>
>> Well, "Python 3" is not a uniform thing.  There are many changes across
>> the 3.x spectrum that can bite you.  I'm speaking out of the experience
>> with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
>> we should add tests for 3.7).
> 
> Do you expect us to need to keep track of the exact Python
> version before running the Python interpreter?  Code written in
> Python doesn't count, because it can simply use sys.version_info.
> 
> 

You're right, Python code can do that.  My proposal is really to
facilitate other parts of the test automation, and as described later,
the debugging on remote systems.

>>
>>> Extra cleanups (like moving version checks to ./configure) are
>>> still welcome, but keep in mind that this will probably be thrown
>>> away once we drop Python 2 support.
>>>
>>
>> I don't think it should.  Let me give the example of a "Python 3.0" job
>> on Travis, related to the following snippet:
>>
>>     # Python builds
>>     - env: CONFIG="--target-list=x86_64-softmmu"
>>       python:
>>         - "3.0"
>>
>> If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
>> you'll see that the intended goal was missed.  That has to do with how
>> Travis makes the requested Python version available, and it was only
>> obvious because this branch prints the Python version used.
>>
>> Developers writing Python code, for instance tests, may assume a given
>> API that doesn't exist or behave like they expect, and not knowing the
>> Python version used on a remote environment makes debugging extra hard.
> 
> I'm not sure I get your point.  We can surely add diagnostic
> messages to show the Python version, and we can make ./configure
> fail if Python 3 is unavailable.  I'm talking about adding code
> to ./configure to save Python version info for makefile rules and
> shell scripts.  I expect the extra code to be unnecessary in the
> future.
> 

I'm definitely happy if you agree with printing the Python version,
that's where most of the value is indeed.  Then, if we need to keep the
version in the generate Makefile is something to be seen.

- Cleber.

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.