From nobody Sun Apr 28 20:53:30 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1529037836095909.855137486641; Thu, 14 Jun 2018 21:43:56 -0700 (PDT) Received: from localhost ([::1]:44272 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTgaW-0008Gc-8H for importer@patchew.org; Fri, 15 Jun 2018 00:43:52 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50723) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTgXf-0006gp-5b for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTgXb-0003P7-8E for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:55 -0400 Received: from tsukuyomi.43-1.org ([188.40.248.50]:49145) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fTgXb-0003OC-0U for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:51 -0400 From: Matthias Maier To: qemu-devel@nongnu.org Date: Thu, 14 Jun 2018 23:40:41 -0500 Message-Id: <20180615044042.7928-2-tamiko@43-1.org> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180615044042.7928-1-tamiko@43-1.org> References: <20180615044042.7928-1-tamiko@43-1.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 188.40.248.50 Subject: [Qemu-devel] [PATCH v2 1/2] Partially revert commit d4e5ec877ca X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matthias Maier , Eduardo Habkost , Arfrever Frehtes Taifersar Arahesis Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This commit removes the PYTHON_UTF8 workaround. The problem with setting LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 is that the en_US.UTF-8 locale might not be available. In this case setting above locales results in build errors even though another UTF-8 locale was originally set [1]. The only stable way of fixing the encoding problem is by explicitly annotating encoding/decoding in the python script. [1] https://bugs.gentoo.org/657766 Signed-off-by: Arfrever Frehtes Taifersar Arahesis Signed-off-by: Matthias Maier --- Makefile | 6 ++---- tests/Makefile.include | 6 +++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index e46f2b625a..7ed9cc4a21 100644 --- a/Makefile +++ b/Makefile @@ -20,8 +20,6 @@ ifneq ($(wildcard config-host.mak),) all: include config-host.mak =20 -PYTHON_UTF8 =3D LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 $(PYTHON) - git-submodule-update: =20 .PHONY: git-submodule-update @@ -576,7 +574,7 @@ qga/qapi-generated/qga-qapi-commands.h qga/qapi-generat= ed/qga-qapi-commands.c \ qga/qapi-generated/qga-qapi-doc.texi: \ qga/qapi-generated/qapi-gen-timestamp ; qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(= qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o qga/qapi-generated -p "qga-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ @@ -676,7 +674,7 @@ qapi/qapi-introspect.h qapi/qapi-introspect.c \ qapi/qapi-doc.texi: \ qapi-gen-timestamp ; qapi-gen-timestamp: $(qapi-modules) $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o "qapi" -b $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ diff --git a/tests/Makefile.include b/tests/Makefile.include index 607afe5bed..b5121f2851 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -674,13 +674,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \ tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \ tests/test-qapi-gen-timestamp ; tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-t= est.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests -p "test-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ =20 tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-go= od.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests/qapi-schema -p "doc-good-" $<, \ "GEN","$@") @mv tests/qapi-schema/doc-good-qapi-doc.texi $@ @@ -938,7 +938,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-q= uick.sh qemu-img$(EXESUF) .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/= %.json $(call quiet-command, PYTHONPATH=3D$(SRC_PATH)/scripts \ - $(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ + $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ $^ >$*.test.out 2>$*.test.err; \ echo $$? >$*.test.exit, \ "TEST","$*.out") --=20 2.16.4 From nobody Sun Apr 28 20:53:30 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1529037748310306.0413090473842; Thu, 14 Jun 2018 21:42:28 -0700 (PDT) Received: from localhost ([::1]:44268 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTgZ4-0007Jl-FI for importer@patchew.org; Fri, 15 Jun 2018 00:42:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50700) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fTgXc-0006gV-Gi for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fTgXb-0003PM-K8 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:52 -0400 Received: from tsukuyomi.43-1.org ([2a01:4f8:173:743::1:50]:53130) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fTgXb-0003Op-D6 for qemu-devel@nongnu.org; Fri, 15 Jun 2018 00:40:51 -0400 From: Matthias Maier To: qemu-devel@nongnu.org Date: Thu, 14 Jun 2018 23:40:42 -0500 Message-Id: <20180615044042.7928-3-tamiko@43-1.org> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20180615044042.7928-1-tamiko@43-1.org> References: <20180615044042.7928-1-tamiko@43-1.org> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a01:4f8:173:743::1:50 Subject: [Qemu-devel] [PATCH v2 2/2] qapi: open files in binary mode and use explicit decoding/encoding in common.py X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matthias Maier , Eduardo Habkost , Arfrever Frehtes Taifersar Arahesis Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" This is a different approach to fix the locale dependent encode/decode problem in common.py utilizing the binary read/write mode [1,2] and decode/encode with explicit UTF-8 encoding arguments [3]. This approach is preferred over the fix in commit d4e5ec877ca because it is (a) locale independent, and (b) does not depend on the en_US.UTF_8 locale to be available. [1] https://docs.python.org/3.6/library/stdtypes.html#bytes.decode [2] https://docs.python.org/3.6/library/stdtypes.html#str.encode [3] https://docs.python.org/3/howto/unicode.html Signed-off-by: Arfrever Frehtes Taifersar Arahesis Signed-off-by: Matthias Maier --- scripts/qapi/common.py | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py index 2462fc0291..44270cd703 100644 --- a/scripts/qapi/common.py +++ b/scripts/qapi/common.py @@ -16,6 +16,7 @@ import errno import os import re import string +import sys from collections import OrderedDict =20 builtin_types =3D { @@ -259,6 +260,8 @@ class QAPISchemaParser(object): previously_included.append(os.path.abspath(fp.name)) self.incl_info =3D incl_info self.src =3D fp.read() + if sys.version_info[0] >=3D 3: + self.src =3D self.src.decode("UTF-8") if self.src =3D=3D '' or self.src[-1] !=3D '\n': self.src +=3D '\n' self.cursor =3D 0 @@ -340,7 +343,7 @@ class QAPISchemaParser(object): return None =20 try: - fobj =3D open(incl_fname, 'r') + fobj =3D open(incl_fname, 'rb') except IOError as e: raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname)) return QAPISchemaParser(fobj, previously_included, info) @@ -1492,7 +1495,7 @@ class QAPISchemaEvent(QAPISchemaEntity): class QAPISchema(object): def __init__(self, fname): self._fname =3D fname - parser =3D QAPISchemaParser(open(fname, 'r')) + parser =3D QAPISchemaParser(open(fname, 'rb')) exprs =3D check_exprs(parser.exprs) self.docs =3D parser.docs self._entity_list =3D [] @@ -2006,9 +2009,11 @@ class QAPIGen(object): if e.errno !=3D errno.EEXIST: raise fd =3D os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f =3D os.fdopen(fd, 'r+') + f =3D os.fdopen(fd, 'r+b') text =3D (self._top(fname) + self._preamble + self._body + self._bottom(fname)) + if sys.version_info[0] >=3D 3: + text =3D text.encode("UTF-8") oldtext =3D f.read(len(text) + 1) if text !=3D oldtext: f.seek(0) --=20 2.16.4