From nobody Mon Feb 9 09:53:26 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of _spf.google.com designates 209.85.128.51 as permitted sender) client-ip=209.85.128.51; envelope-from=philippe.mathieu.daude@gmail.com; helo=mail-wm1-f51.google.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of _spf.google.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=philippe.mathieu.daude@gmail.com ARC-Seal: i=1; a=rsa-sha256; t=1610236965; cv=none; d=zohomail.com; s=zohoarc; b=BcqpuxeXaMfMeEzwEHkgpXtBhpfkuVW6Uuvi7hjvurp6vDi+GcgpGsPS9cZSgzUu2MGA3Qt2mADCRdXpGCQdiOTV5VXM/iIL6EWNXx0BY2Hc1rKBP4XG9CKMaEpImMD4z28dhX3yfainTU4Ra5JJxLjfGYiu8z7/U+xXeKNZImw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1610236965; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:MIME-Version:Message-ID:Sender:Subject:To; bh=FDZUXvUmtcIWmbdDBGyyVWvJunFKN4yhyfU3SKU21DM=; b=EvoZ32TC73niJliyZ0aLhToF1ZtFMhvpInbSHC7RowMfcemwuUANPY0ujyrlnm9jF/CJQRo57jiDO/WgD4eGg0td94RCYzvV7Xv/i1fzeQ0N6ECI4lkucMJGWNfk2IXDpEa44DkvnnF8B9St9TRn4x4zswnfxAK0iQ4GQnL5E4s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of _spf.google.com designates 209.85.128.51 as permitted sender) smtp.mailfrom=philippe.mathieu.daude@gmail.com Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by mx.zohomail.com with SMTPS id 161023696524958.902364999962515; Sat, 9 Jan 2021 16:02:45 -0800 (PST) Received: by mail-wm1-f51.google.com with SMTP id a6so10676229wmc.2 for ; Sat, 09 Jan 2021 16:02:44 -0800 (PST) Return-Path: Return-Path: Received: from localhost.localdomain (129.red-88-21-205.staticip.rima-tde.net. [88.21.205.129]) by smtp.gmail.com with ESMTPSA id q1sm18794346wrj.8.2021.01.09.16.02.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 09 Jan 2021 16:02:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=FDZUXvUmtcIWmbdDBGyyVWvJunFKN4yhyfU3SKU21DM=; b=XdH23OyiVhhNxonYPd0SM2lThywhxGpyJT6z5C4OCYkYNSgdeknRaWxj0NMdatBlYu V8eHzQvna6LxVUpkgGKgYDJMREvY8cI5/hCl0TPfYcxaoxN1lr82/Gbq05NzYpcCJov6 WeN9WUikKn2ebos/uk6HSO2UYvyKFb+9+s1J7LZn5jPAjzmCcrj7ENzenXZ7tTrIxdXZ QaszFRxYqahmSsuPfLApfB63vtzl+g+rbYc0lUpgBqjoGEyJID98INWUhSgTFWAvQa3r jPjOrzT3qq0z+D/CVy2o6/PEkZI0h8i1gR7Bd2S+Dm7gfkDe6S2z1Jhwg25O6VeoiiZs lN/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; bh=FDZUXvUmtcIWmbdDBGyyVWvJunFKN4yhyfU3SKU21DM=; b=sI2APNsozlH68ZVRZmwzhNnphmvah07LNjed+QSigSobXqnwF9FN0jqMm0xHoIOq/I nNEkgKYhvQjmXLiaGt40CLPYvZSW2VTIApliGYgRQSvGCDRmlAIG1F8FMeHGZHCvcdNn hEbQo90vhozuAPz6xR59ek4maBPyk3GWJorgHej7qVqJEYZPh87e9bXdwqMoYd31QEPr RbVVCAR5oAZItJTm9snk7HtvgjE+m0ExwHMVof/rQHLrN0p7LdQ1/sgv7FLsA2RwEk/Z NBkBr4VfTX8DjrPRrSUJqNfYRfRC891R9CH6juC5x6dNummmmVppzUmkAKdLlnTF5Yme iDjQ== X-Gm-Message-State: AOAM531064XHHxmcXVuZlRTFl5aeL1cTaU346bS67/o44oNGhaIqBESf Oie08T4NA7UEjULWjKJbsiM= X-Google-Smtp-Source: ABdhPJx3YuAeFJoXnxPiKdqED4pzGJwFg+qB4kZVUCsVo8CIpE5dNIpL5CFl2uZ8ZCuYyUunPTwsig== X-Received: by 2002:a7b:c5d6:: with SMTP id n22mr8469520wmk.70.1610236963419; Sat, 09 Jan 2021 16:02:43 -0800 (PST) Sender: =?UTF-8?Q?Philippe_Mathieu=2DDaud=C3=A9?= From: =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= To: qemu-devel@nongnu.org Cc: Richard Henderson , Cleber Rosa , Eduardo Habkost , Paolo Bonzini , Yonggang Luo , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Peter Maydell Subject: [PATCH v3] decodetree: Open files with encoding='utf-8' Date: Sun, 10 Jan 2021 01:02:40 +0100 Message-Id: <20210110000240.761122-1-f4bug@amsat.org> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @gmail.com) When decodetree.py was added in commit 568ae7efae7, QEMU was using Python 2 which happily reads UTF-8 files in text mode. Python 3 requires either UTF-8 locale or an explicit encoding passed to open(). Now that Python 3 is required, explicit UTF-8 encoding for decodetree source files. To avoid further problems with the user locale, also explicit UTF-8 encoding for the generated C files. Explicit both input/output are plain text by using the 't' mode. This fixes: $ /usr/bin/python3 scripts/decodetree.py test.decode Traceback (most recent call last): File "scripts/decodetree.py", line 1397, in main() File "scripts/decodetree.py", line 1308, in main parse_file(f, toppat) File "scripts/decodetree.py", line 994, in parse_file for line in f: File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 80: ordinal not in range(128) Reported-by: Peter Maydell Suggested-by: Yonggang Luo Reviewed-by: Eduardo Habkost Signed-off-by: Philippe Mathieu-Daud=C3=A9 --- v3: utf-8 stdout (Eduardo and Yonggang Luo) v2: utf-8 output too (Peter) explicit default text mode. --- scripts/decodetree.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/scripts/decodetree.py b/scripts/decodetree.py index 47aa9caf6d1..4637b633e70 100644 --- a/scripts/decodetree.py +++ b/scripts/decodetree.py @@ -20,6 +20,7 @@ # See the syntax and semantics in docs/devel/decodetree.rst. # =20 +import io import os import re import sys @@ -1304,7 +1305,7 @@ def main(): =20 for filename in args: input_file =3D filename - f =3D open(filename, 'r') + f =3D open(filename, 'rt', encoding=3D'utf-8') parse_file(f, toppat) f.close() =20 @@ -1324,9 +1325,11 @@ def main(): prop_size(stree) =20 if output_file: - output_fd =3D open(output_file, 'w') + output_fd =3D open(output_file, 'wt', encoding=3D'utf-8') else: - output_fd =3D sys.stdout + output_fd =3D io.TextIOWrapper(sys.stdout.buffer, + encoding=3Dsys.stdout.encoding, + errors=3D"ignore") =20 output_autogen() for n in sorted(arguments.keys()): --=20 2.26.2