From nobody Sat May 4 01:35:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) by mx.zohomail.com with SMTPS id 1529345028592624.4708705776926; Mon, 18 Jun 2018 11:03:48 -0700 (PDT) Received: from localhost ([::1]:36359 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUyVH-0001Hs-Vg for importer@patchew.org; Mon, 18 Jun 2018 14:03:48 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49483) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUyRf-0007WQ-LB for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUyRc-0004sl-LV for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:03 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:49176 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fUyRc-0004rp-GN for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:00 -0400 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C1686401EF00; Mon, 18 Jun 2018 17:59:59 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-237.ams2.redhat.com [10.36.117.237]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8F59110190; Mon, 18 Jun 2018 17:59:59 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 459BE113864C; Mon, 18 Jun 2018 19:59:58 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Mon, 18 Jun 2018 19:59:57 +0200 Message-Id: <20180618175958.29073-2-armbru@redhat.com> In-Reply-To: <20180618175958.29073-1-armbru@redhat.com> References: <20180618175958.29073-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 18 Jun 2018 17:59:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.5]); Mon, 18 Jun 2018 17:59:59 +0000 (UTC) for IP:'10.11.54.5' DOMAIN:'int-mx05.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v4 1/2] qapi: Open files with encoding='utf-8' X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tamiko@43-1.org, arfrever.fta@gmail.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Python 2 happily reads UTF-8 files in text mode, but Python 3 requires either UTF-8 locale or an explicit encoding passed to open(). Commit d4e5ec877ca fixed this by setting the en_US.UTF-8 locale. Falls apart when the locale isn't be available. Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use binary mode instead, with manual conversion from bytes to str. Works, but opening with an explicit encoding is simpler, so do that. Since Python 2's open() doesn't support the encoding parameter, we need to suppress it with a version check. Reported-by: Arfrever Frehtes Taifersar Arahesis Reported-by: Matthias Maier Signed-off-by: Markus Armbruster Acked-by: Eduardo Habkost Reviewed-by: Eduardo Habkost Reviewed-by: Eric Blake --- scripts/qapi/common.py | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py index 2462fc0291..832f11438a 100644 --- a/scripts/qapi/common.py +++ b/scripts/qapi/common.py @@ -16,6 +16,7 @@ import errno import os import re import string +import sys from collections import OrderedDict =20 builtin_types =3D { @@ -340,7 +341,10 @@ class QAPISchemaParser(object): return None =20 try: - fobj =3D open(incl_fname, 'r') + if sys.version_info[0] >=3D 3: + fobj =3D open(incl_fname, 'r', encoding=3D'utf-8') + else: + fobj =3D open(incl_fname, 'r') except IOError as e: raise QAPISemError(info, '%s: %s' % (e.strerror, incl_fname)) return QAPISchemaParser(fobj, previously_included, info) @@ -1492,7 +1496,11 @@ class QAPISchemaEvent(QAPISchemaEntity): class QAPISchema(object): def __init__(self, fname): self._fname =3D fname - parser =3D QAPISchemaParser(open(fname, 'r')) + if sys.version_info[0] >=3D 3: + f =3D open(fname, 'r', encoding=3D'utf-8') + else: + f =3D open(fname, 'r') + parser =3D QAPISchemaParser(f) exprs =3D check_exprs(parser.exprs) self.docs =3D parser.docs self._entity_list =3D [] @@ -2006,7 +2014,10 @@ class QAPIGen(object): if e.errno !=3D errno.EEXIST: raise fd =3D os.open(pathname, os.O_RDWR | os.O_CREAT, 0o666) - f =3D os.fdopen(fd, 'r+') + if sys.version_info[0] >=3D 3: + f =3D open(fd, 'r+', encoding=3D'utf-8') + else: + f =3D os.fdopen(fd, 'r+') text =3D (self._top(fname) + self._preamble + self._body + self._bottom(fname)) oldtext =3D f.read(len(text) + 1) --=20 2.17.1 From nobody Sat May 4 01:35:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) client-ip=208.118.235.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of gnu.org designates 208.118.235.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.gnu.org (208.118.235.17 [208.118.235.17]) by mx.zohomail.com with SMTPS id 1529344897457854.9286635278492; Mon, 18 Jun 2018 11:01:37 -0700 (PDT) Received: from localhost ([::1]:36350 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUyT3-0008F1-9A for importer@patchew.org; Mon, 18 Jun 2018 14:01:29 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49482) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fUyRf-0007WP-L5 for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fUyRc-0004sr-Le for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:03 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:49816 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fUyRc-0004rs-Gh for qemu-devel@nongnu.org; Mon, 18 Jun 2018 14:00:00 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id C2B7C87A85; Mon, 18 Jun 2018 17:59:59 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-117-237.ams2.redhat.com [10.36.117.237]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8F852111AF16; Mon, 18 Jun 2018 17:59:59 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 491F3113864D; Mon, 18 Jun 2018 19:59:58 +0200 (CEST) From: Markus Armbruster To: qemu-devel@nongnu.org Date: Mon, 18 Jun 2018 19:59:58 +0200 Message-Id: <20180618175958.29073-3-armbru@redhat.com> In-Reply-To: <20180618175958.29073-1-armbru@redhat.com> References: <20180618175958.29073-1-armbru@redhat.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 18 Jun 2018 17:59:59 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.1]); Mon, 18 Jun 2018 17:59:59 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'armbru@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: [Qemu-devel] [PATCH v4 2/2] Revert commit d4e5ec877ca X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: tamiko@43-1.org, arfrever.fta@gmail.com Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: Matthias Maier This commit removes the PYTHON_UTF8 workaround. The problem with setting LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 is that the en_US.UTF-8 locale might not be available. In this case setting above locales results in build errors even though another UTF-8 locale was originally set [1]. The only stable way of fixing the encoding problem is by explicitly annotating encoding/decoding in the python script. [1] https://bugs.gentoo.org/657766 Signed-off-by: Arfrever Frehtes Taifersar Arahesis Signed-off-by: Matthias Maier Message-Id: <20180615220205.6929-3-tamiko@43-1.org> Acked-by: Eduardo Habkost Reviewed-by: Eduardo Habkost Reviewed-by: Eric Blake --- Makefile | 6 ++---- tests/Makefile.include | 6 +++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/Makefile b/Makefile index e46f2b625a..7ed9cc4a21 100644 --- a/Makefile +++ b/Makefile @@ -20,8 +20,6 @@ ifneq ($(wildcard config-host.mak),) all: include config-host.mak =20 -PYTHON_UTF8 =3D LC_ALL=3D LANG=3DC LC_CTYPE=3Den_US.UTF-8 $(PYTHON) - git-submodule-update: =20 .PHONY: git-submodule-update @@ -576,7 +574,7 @@ qga/qapi-generated/qga-qapi-commands.h qga/qapi-generat= ed/qga-qapi-commands.c \ qga/qapi-generated/qga-qapi-doc.texi: \ qga/qapi-generated/qapi-gen-timestamp ; qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json $(= qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o qga/qapi-generated -p "qga-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ @@ -676,7 +674,7 @@ qapi/qapi-introspect.h qapi/qapi-introspect.c \ qapi/qapi-doc.texi: \ qapi-gen-timestamp ; qapi-gen-timestamp: $(qapi-modules) $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o "qapi" -b $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ diff --git a/tests/Makefile.include b/tests/Makefile.include index ca91da26cb..88f1bc1242 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -678,13 +678,13 @@ tests/test-qapi-events.c tests/test-qapi-events.h \ tests/test-qapi-introspect.c tests/test-qapi-introspect.h: \ tests/test-qapi-gen-timestamp ; tests/test-qapi-gen-timestamp: $(SRC_PATH)/tests/qapi-schema/qapi-schema-t= est.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests -p "test-" $<, \ "GEN","$(@:%-timestamp=3D%)") @>$@ =20 tests/qapi-schema/doc-good.test.texi: $(SRC_PATH)/tests/qapi-schema/doc-go= od.json $(qapi-py) - $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \ + $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \ -o tests/qapi-schema -p "doc-good-" $<, \ "GEN","$@") @mv tests/qapi-schema/doc-good-qapi-doc.texi $@ @@ -942,7 +942,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-q= uick.sh qemu-img$(EXESUF) .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/= %.json $(call quiet-command, PYTHONPATH=3D$(SRC_PATH)/scripts \ - $(PYTHON_UTF8) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ + $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ $^ >$*.test.out 2>$*.test.err; \ echo $$? >$*.test.exit, \ "TEST","$*.out") --=20 2.17.1