From nobody Tue Feb 10 20:52:49 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1516111344106708.4395999295084; Tue, 16 Jan 2018 06:02:24 -0800 (PST) Received: from localhost ([::1]:37231 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebRol-0006At-3O for importer@patchew.org; Tue, 16 Jan 2018 09:02:23 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40272) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ebRXJ-0008Il-9G for qemu-devel@nongnu.org; Tue, 16 Jan 2018 08:44:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ebRXG-0001Hw-IV for qemu-devel@nongnu.org; Tue, 16 Jan 2018 08:44:21 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52666) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ebRXG-0001HF-9V for qemu-devel@nongnu.org; Tue, 16 Jan 2018 08:44:18 -0500 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 4EA65D2EFF; Tue, 16 Jan 2018 13:44:17 +0000 (UTC) Received: from t460.redhat.com (unknown [10.33.36.63]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8AD805D753; Tue, 16 Jan 2018 13:44:09 +0000 (UTC) From: "Daniel P. Berrange" To: qemu-devel@nongnu.org Date: Tue, 16 Jan 2018 13:42:11 +0000 Message-Id: <20180116134217.8725-9-berrange@redhat.com> In-Reply-To: <20180116134217.8725-1-berrange@redhat.com> References: <20180116134217.8725-1-berrange@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.38]); Tue, 16 Jan 2018 13:44:17 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PATCH v5 08/14] qapi: force a UTF-8 locale for running Python X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Fam Zheng , Eduardo Habkost , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Markus Armbruster , Paolo Bonzini , =?UTF-8?q?Alex=20Benn=C3=A9e?= Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python2 did not validate locale correctness when reading input data, so would happily read UTF-8 data in non-UTF-8 locales. Python3 is strict so if you try to read UTF-8 data in the C locale, it will raise an error for any UTF-8 bytes that aren't representable in 7-bit ascii encoding. e.g. UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 54: or= dinal not in range(128) Traceback (most recent call last): File "/tmp/qemu-test/src/scripts/qapi-commands.py", line 317, in schema =3D QAPISchema(input_file) File "/tmp/qemu-test/src/scripts/qapi.py", line 1468, in __init__ parser =3D QAPISchemaParser(open(fname, 'r')) File "/tmp/qemu-test/src/scripts/qapi.py", line 301, in __init__ previously_included) File "/tmp/qemu-test/src/scripts/qapi.py", line 348, in _include exprs_include =3D QAPISchemaParser(fobj, previously_included, info) File "/tmp/qemu-test/src/scripts/qapi.py", line 271, in __init__ self.src =3D fp.read() File "/usr/lib64/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] More background on this can be seen in https://www.python.org/dev/peps/pep-0538/ Many distros support a new C.UTF-8 locale that is like the C locale, but with UTF-8 instead of 7-bit ASCII. That is not entirely portable though. This patch thus sets the LANG to "C", but overrides LC_CTYPE to be en_US.UTF-8 locale. This gets us pretty close to C.UTF-8, but in a way that should be portable to everywhere QEMU builds. This patch only forces UTF-8 for QAPI scripts, since that is the one showing the immediate error under Python3 with C locale, but potentially we ought to force this for all python scripts used in the build process. Signed-off-by: Daniel P. Berrange Reviewed-by: Eric Blake --- Makefile | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/Makefile b/Makefile index d86ecd2dd4..63767bb11f 100644 --- a/Makefile +++ b/Makefile @@ -17,6 +17,8 @@ ifneq ($(wildcard config-host.mak),) all: include config-host.mak =20 +PYTHON_UTF8 =3D LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 $(PYTHON) + git-submodule-update: =20 .PHONY: git-submodule-update @@ -471,17 +473,17 @@ qapi-py =3D $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/s= cripts/ordereddict.py =20 qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\ $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-= py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \ $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ "GEN","$@") qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\ $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-= py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \ $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ "GEN","$@") qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c= :\ $(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py $(qa= pi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \ $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \ "GEN","$@") =20 @@ -502,27 +504,27 @@ qapi-modules =3D $(SRC_PATH)/qapi-schema.json $(SRC_P= ATH)/qapi/common.json \ =20 qapi-types.c qapi-types.h :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \ $(gen-out-type) -o "." -b $<, \ "GEN","$@") qapi-visit.c qapi-visit.h :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \ $(gen-out-type) -o "." -b $<, \ "GEN","$@") qapi-event.c qapi-event.h :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-event.py \ $(gen-out-type) -o "." $<, \ "GEN","$@") qmp-commands.h qmp-marshal.c :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-commands.py \ $(gen-out-type) -o "." $<, \ "GEN","$@") qmp-introspect.h qmp-introspect.c :\ $(qapi-modules) $(SRC_PATH)/scripts/qapi-introspect.py $(qapi-py) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-introspect.py \ + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-introspect.p= y \ $(gen-out-type) -o "." $<, \ "GEN","$@") =20 @@ -792,10 +794,10 @@ qemu-img-cmds.texi: $(SRC_PATH)/qemu-img-cmds.hx $(SR= C_PATH)/scripts/hxtool docs/interop/qemu-qmp-qapi.texi docs/interop/qemu-ga-qapi.texi: $(SRC_PATH= )/scripts/qapi2texi.py $(qapi-py) =20 docs/interop/qemu-qmp-qapi.texi: $(qapi-modules) - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"= GEN","$@") + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< >= $@,"GEN","$@") =20 docs/interop/qemu-ga-qapi.texi: $(SRC_PATH)/qga/qapi-schema.json - $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi2texi.py $< > $@,"= GEN","$@") + $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi2texi.py $< >= $@,"GEN","$@") =20 qemu.1: qemu-doc.texi qemu-options.texi qemu-monitor.texi qemu-monitor-inf= o.texi qemu.1: qemu-option-trace.texi --=20 2.14.3