From nobody Thu May 2 23:45:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528991325409521.2315688546739; Thu, 14 Jun 2018 08:48:45 -0700 (PDT) Received: from localhost ([::1]:41396 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTUUO-0000JF-Hi for importer@patchew.org; Thu, 14 Jun 2018 11:48:44 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51604) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTUQB-00067c-Tp for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTUQA-0001nK-PE for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:23 -0400 Received: from tsukuyomi.43-1.org ([188.40.248.50]:46759) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fTUQA-0001my-Hw for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:22 -0400 From: Matthias Maier To: qemu-devel@nongnu.org Date: Thu, 14 Jun 2018 10:43:56 -0500 Message-Id: <20180614154357.20132-2-tamiko@43-1.org> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180614154357.20132-1-tamiko@43-1.org> References: <20180614154357.20132-1-tamiko@43-1.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 188.40.248.50 X-Mailman-Approved-At: Thu, 14 Jun 2018 11:45:41 -0400 Subject: [Qemu-devel] [PATCH 1/2] Partially revert commit d4e5ec877ca X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Arfrever Frehtes Taifersar Arahesis , Eduardo Habkost , Matthias Maier Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Arfrever Frehtes Taifersar Arahesis This commit removes the PYTHON_UTF8 workaround. The problem with setting LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 is that the en_US.UTF-8 locale might not be available. In this case setting above locales results in build errors even though another UTF-8 locale was originally set [1]. The only stable way of fixing the encoding problem is by explicitly annotating encoding/decoding in the python script. [1] https://bugs.gentoo.org/657766 Signed-off-by: Arfrever Frehtes Taifersar Arahesis Signed-off-by: Matthias Maier --- Makefile | 6 ++---- tests/Makefile.include | 6 +++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index 6c6664d9a3..9f7fd9e5dc 100644 --- a/Makefile +++ b/Makefile @@ -20,8 +20,6 @@ ifneq ($(wildcard config-host.mak),) all: include config-host.mak =20 -PYTHON_UTF8 =3D LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 $(PYTHON) - git-submodule-update: =20 .PHONY: git-submodule-update @@ -575,7 +573,7 @@ qga/qapi-generated/qga-qapi-commands.h qga/qapi-generat= ed/qga-qapi-commands.c \ qga/qapi-generated/qga-qapi-doc.texi: \ qga/qapi-generated/qapi-gen-timestamp ; qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(= qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o qga/qapi-generated -p "qga-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ @@ -675,7 +673,7 @@ qapi/qapi-introspect.h qapi/qapi-introspect.c \ qapi/qapi-doc.texi: \ qapi-gen-timestamp ; qapi-gen-timestamp: $(qapi-modules) $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o "qapi" -b $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ diff --git a/tests/Makefile.include b/tests/Makefile.include index bb08e37b9d..618ed36cbb 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -674,13 +674,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \ tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \ tests/test-qapi-gen-timestamp ; tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-t= est.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests -p "test-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ =20 tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-go= od.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests/qapi-schema -p "doc-good-" $<, \ "GEN","$@") @mv tests/qapi-schema/doc-good-qapi-doc.texi $@ @@ -938,7 +938,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-q= uick.sh qemu-img$(EXESUF) .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/= %.json $(call quiet-command, PYTHONPATH=3D$(SRC_PATH)/scripts \ - $(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ + $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ $^ >$*.test.out 2>$*.test.err; \ echo $$? >$*.test.exit, \ "TEST","$*.out") --=20 2.16.4 From nobody Thu May 2 23:45:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1528991258036681.0915513461905; Thu, 14 Jun 2018 08:47:38 -0700 (PDT) Received: from localhost ([::1]:41391 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTUTF-0007kx-73 for importer@patchew.org; Thu, 14 Jun 2018 11:47:33 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51643) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTUQG-00068b-BM for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTUQF-0001oQ-Br for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:28 -0400 Received: from tsukuyomi.43-1.org ([2a01:4f8:173:743::1:50]:50743) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fTUQF-0001nu-4v for qemu-devel@nongnu.org; Thu, 14 Jun 2018 11:44:27 -0400 From: Matthias Maier To: qemu-devel@nongnu.org Date: Thu, 14 Jun 2018 10:43:57 -0500 Message-Id: <20180614154357.20132-3-tamiko@43-1.org> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180614154357.20132-1-tamiko@43-1.org> References: <20180614154357.20132-1-tamiko@43-1.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a01:4f8:173:743::1:50 X-Mailman-Approved-At: Thu, 14 Jun 2018 11:45:41 -0400 Subject: [Qemu-devel] [PATCH 2/2] qapi: open files in binary mode and use explicit decoding/encoding in common.py X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Arfrever Frehtes Taifersar Arahesis , Eduardo Habkost , Matthias Maier Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Arfrever Frehtes Taifersar Arahesis This is a different approach to fix the locale dependent encode/decode problem in common.py utilizing the binary read/write mode [1,2] and decode/encode with explicit UTF-8 encoding arguments [3]. This approach is preferred over the fix in commit d4e5ec877ca because it is (a) locale independent, and (b) does not depend on the en_US.UTF_8 locale to be available. [1] https://docs.python.org/3.6/library/stdtypes.html#bytes.decode [2] https://docs.python.org/3.6/library/stdtypes.html#str.encode [3] https://docs.python.org/3/howto/unicode.html Signed-off-by: Arfrever Frehtes Taifersar Arahesis Signed-off-by: Matthias Maier --- scripts/qapi/common.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py index 2462fc0291..8f8ccc65c3 100644 --- a/scripts/qapi/common.py +++ b/scripts/qapi/common.py @@ -258,7 +258,7 @@ class QAPISchemaParser(object): self.fname =3D fp.name previously_included.append(os.path.abspath(fp.name)) self.incl_info =3D incl_info - self.src =3D fp.read() + self.src =3D fp.read().decode("UTF-8") if self.src =3D=3D '' or self.src[-1] !=3D '\n': self.src +=3D '\n' self.cursor =3D 0 @@ -340,7 +340,7 @@ class QAPISchemaParser(object): return None =20 try: - fobj =3D open(incl_fname, 'r') + fobj =3D open(incl_fname, 'rb') except IOError as e: raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname)) return QAPISchemaParser(fobj, previously_included, info) @@ -1492,7 +1492,7 @@ class QAPISchemaEvent(QAPISchemaEntity): class QAPISchema(object): def __init__(self, fname): self._fname =3D fname - parser =3D QAPISchemaParser(open(fname, 'r')) + parser =3D QAPISchemaParser(open(fname, 'rb')) exprs =3D check_exprs(parser.exprs) self.docs =3D parser.docs self._entity_list =3D [] @@ -2006,14 +2006,14 @@ class QAPIGen(object): if e.errno !=3D errno.EEXIST: raise fd =3D os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f =3D os.fdopen(fd, 'r+') + f =3D os.fdopen(fd, 'r+b') text =3D (self._top(fname) + self._preamble + self._body + self._bottom(fname)) - oldtext =3D f.read(len(text) + 1) + oldtext =3D f.read(len(text) + 1).decode("UTF-8") if text !=3D oldtext: f.seek(0) f.truncate(0) - f.write(text) + f.write(text.encode("UTF-8")) f.close() =20 =20 --=20 2.16.4