1. Patchew
  2. EDK2
  3. View series

[edk2-devel] [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools

Philippe Mathieu-Daudé posted 1 patch 4 years, 5 months ago
Failed in applying to current master (apply log)
There is a newer version of this series
BaseTools/Tests/TestTools.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
[edk2-devel] [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
Posted by Philippe Mathieu-Daudé 4 years, 5 months ago 
Under Centos 7.7 we get:

  Build environment: Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
  [...]
  ======================================================================
  ERROR: testRandomDataCycles (TianoCompress.Tests)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
    File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles
      self.compressionTestCycle(data)
    File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle
      start = self.ReadTmpFile('input')
    File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
      data = f.read()
    File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

  ----------------------------------------------------------------------

Fix by specifying the UTF-8 encoding.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
---
RFC because I'm not sure this is the best way to fix this, but
this is similar to commit 31e3eeb5e3d2d.
---
 BaseTools/Tests/TestTools.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
index 1099fd4eeaea..41cdb28b0c8c 100644
--- a/BaseTools/Tests/TestTools.py
+++ b/BaseTools/Tests/TestTools.py
@@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
         return open(os.path.join(self.testDir, fileName), mode)
 
     def ReadTmpFile(self, fileName):
-        f = open(self.GetTmpFilePath(fileName), 'r')
+        f = codecs.open(self.GetTmpFilePath(fileName), 'r', encoding='utf-8')
         data = f.read()
         f.close()
         return data
-- 
2.21.0


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51758): https://edk2.groups.io/g/devel/message/51758
Mute This Topic: https://groups.io/mt/66540766/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
Posted by Philippe Mathieu-Daudé 4 years, 5 months ago 
On 12/4/19 10:38 PM, Philippe Mathieu-Daude wrote:
> Under Centos 7.7 we get:
> 
>    Build environment: Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
>    [...]
>    ======================================================================
>    ERROR: testRandomDataCycles (TianoCompress.Tests)
>    ----------------------------------------------------------------------
>    Traceback (most recent call last):
>      File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in testRandomDataCycles
>        self.compressionTestCycle(data)
>      File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in compressionTestCycle
>        start = self.ReadTmpFile('input')
>      File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
>        data = f.read()
>      File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
>        return codecs.ascii_decode(input, self.errors)[0]
>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)
> 
>    ----------------------------------------------------------------------
> 
> Fix by specifying the UTF-8 encoding.
> 
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
> ---
> RFC because I'm not sure this is the best way to fix this, but
> this is similar to commit 31e3eeb5e3d2d.
> ---
>   BaseTools/Tests/TestTools.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
> index 1099fd4eeaea..41cdb28b0c8c 100644
> --- a/BaseTools/Tests/TestTools.py
> +++ b/BaseTools/Tests/TestTools.py
> @@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
>           return open(os.path.join(self.testDir, fileName), mode)
>   
>       def ReadTmpFile(self, fileName):
> -        f = open(self.GetTmpFilePath(fileName), 'r')
> +        f = codecs.open(self.GetTmpFilePath(fileName), 'r', encoding='utf-8')
>           data = f.read()
>           f.close()
>           return data
> 

While this fixes Python3, this also break Python2 :)

======================================================================
ERROR: testRandomDataCycles (TianoCompress.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in 
testRandomDataCycles
     self.compressionTestCycle(data)
   File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in 
compressionTestCycle
     start = self.ReadTmpFile('input')
   File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
     data = f.read() 

   File "/usr/lib/python2.7/codecs.py", line 688, in read
     return self.reader.read(size)
   File "/usr/lib/python2.7/codecs.py", line 494, in read
     newchars, decodedbytes = self.decode(data, self.errors)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 0: 
invalid start byte

This old thread recommend to use io.open:
https://web.archive.org/web/20180715024113/https://mail.python.org/pipermail/python-list/2015-March/687124.html

And it works in with both 2/3 versions, so I'll respin.


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51796): https://edk2.groups.io/g/devel/message/51796
Mute This Topic: https://groups.io/mt/66540766/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [RFC PATCH] BaseTools: Fix Python3 encoding issue in TestTools
Posted by Laszlo Ersek 4 years, 5 months ago 
On 12/05/19 19:36, Philippe Mathieu-Daudé wrote:
> On 12/4/19 10:38 PM, Philippe Mathieu-Daude wrote:
>> Under Centos 7.7 we get:
>>
>>    Build environment:
>> Linux-3.10.0-1062.7.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
>>    [...]
>>    ======================================================================
>>    ERROR: testRandomDataCycles (TianoCompress.Tests)
>>    ----------------------------------------------------------------------
>>    Traceback (most recent call last):
>>      File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in
>> testRandomDataCycles
>>        self.compressionTestCycle(data)
>>      File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in
>> compressionTestCycle
>>        start = self.ReadTmpFile('input')
>>      File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
>>        data = f.read()
>>      File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
>>        return codecs.ascii_decode(input, self.errors)[0]
>>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in
>> position 3: ordinal not in range(128)
>>
>>    ----------------------------------------------------------------------
>>
>> Fix by specifying the UTF-8 encoding.
>>
>> Cc: Bob Feng <bob.c.feng@intel.com>
>> Cc: Liming Gao <liming.gao@intel.com>
>> Signed-off-by: Philippe Mathieu-Daude <philmd@redhat.com>
>> ---
>> RFC because I'm not sure this is the best way to fix this, but
>> this is similar to commit 31e3eeb5e3d2d.
>> ---
>>   BaseTools/Tests/TestTools.py | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/BaseTools/Tests/TestTools.py b/BaseTools/Tests/TestTools.py
>> index 1099fd4eeaea..41cdb28b0c8c 100644
>> --- a/BaseTools/Tests/TestTools.py
>> +++ b/BaseTools/Tests/TestTools.py
>> @@ -135,7 +135,7 @@ class BaseToolsTest(unittest.TestCase):
>>           return open(os.path.join(self.testDir, fileName), mode)
>>         def ReadTmpFile(self, fileName):
>> -        f = open(self.GetTmpFilePath(fileName), 'r')
>> +        f = codecs.open(self.GetTmpFilePath(fileName), 'r',
>> encoding='utf-8')
>>           data = f.read()
>>           f.close()
>>           return data
>>
> 
> While this fixes Python3, this also break Python2 :)
> 
> ======================================================================
> ERROR: testRandomDataCycles (TianoCompress.Tests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "edk2/BaseTools/Tests/TianoCompress.py", line 60, in
> testRandomDataCycles
>     self.compressionTestCycle(data)
>   File "edk2/BaseTools/Tests/TianoCompress.py", line 46, in
> compressionTestCycle
>     start = self.ReadTmpFile('input')
>   File "edk2/BaseTools/Tests/TestTools.py", line 139, in ReadTmpFile
>     data = f.read()
>   File "/usr/lib/python2.7/codecs.py", line 688, in read
>     return self.reader.read(size)
>   File "/usr/lib/python2.7/codecs.py", line 494, in read
>     newchars, decodedbytes = self.decode(data, self.errors)
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 0:
> invalid start byte
> 
> This old thread recommend to use io.open:
> https://web.archive.org/web/20180715024113/https://mail.python.org/pipermail/python-list/2015-March/687124.html
> 
> 
> And it works in with both 2/3 versions, so I'll respin.

I didn't ask before (because, "commit 31e3eeb5e3d2d must have been
right, right?"), but now I can't resist anymore:

*why* do we have any such character in a *temporary* file's pathname
that is not pure ASCII? It seems wrong.

Thanks
Laszlo


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51800): https://edk2.groups.io/g/devel/message/51800
Mute This Topic: https://groups.io/mt/66540766/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.