[libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python

Daniel P. Berrangé posted 11 patches 4 years, 7 months ago
Test syntax-check failed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20190905102206.18753-1-berrange@redhat.com
Makefile.am                            |  12 +-
build-aux/augeas-gentest.pl            |  60 ----
build-aux/augeas-gentest.py            |  72 ++++
build-aux/check-spacing.pl             | 198 ----------
build-aux/check-spacing.py             | 204 +++++++++++
build-aux/header-ifdef.pl              | 182 ----------
build-aux/header-ifdef.py              | 206 +++++++++++
build-aux/minimize-po.pl               |  37 --
build-aux/minimize-po.py               |  60 ++++
build-aux/mock-noinline.pl             |  75 ----
build-aux/mock-noinline.py             |  88 +++++
build-aux/prohibit-duplicate-header.pl |  26 --
build-aux/prohibit-duplicate-header.py |  54 +++
cfg.mk                                 |  10 +-
docs/Makefile.am                       |   6 +-
docs/hvsupport.pl                      | 458 -----------------------
docs/hvsupport.py                      | 479 +++++++++++++++++++++++++
po/Makefile.am                         |   2 +-
src/Makefile.am                        |  16 +-
src/check-aclperms.pl                  |  73 ----
src/check-aclperms.py                  |  77 ++++
src/check-symfile.pl                   |  70 ----
src/check-symfile.py                   |  80 +++++
src/check-symsorting.pl                | 106 ------
src/check-symsorting.py                | 112 ++++++
25 files changed, 1455 insertions(+), 1308 deletions(-)
delete mode 100755 build-aux/augeas-gentest.pl
create mode 100755 build-aux/augeas-gentest.py
delete mode 100755 build-aux/check-spacing.pl
create mode 100755 build-aux/check-spacing.py
delete mode 100644 build-aux/header-ifdef.pl
create mode 100644 build-aux/header-ifdef.py
delete mode 100755 build-aux/minimize-po.pl
create mode 100755 build-aux/minimize-po.py
delete mode 100644 build-aux/mock-noinline.pl
create mode 100644 build-aux/mock-noinline.py
delete mode 100644 build-aux/prohibit-duplicate-header.pl
create mode 100644 build-aux/prohibit-duplicate-header.py
delete mode 100755 docs/hvsupport.pl
create mode 100755 docs/hvsupport.py
delete mode 100755 src/check-aclperms.pl
create mode 100755 src/check-aclperms.py
delete mode 100755 src/check-symfile.pl
create mode 100755 src/check-symfile.py
delete mode 100755 src/check-symsorting.pl
create mode 100755 src/check-symsorting.py
[libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Posted by Daniel P. Berrangé 4 years, 7 months ago
Less of

use re 'eval';
                                       ''=~('(?{'.(
            ('`')|                   '%').('['^'-').
         ('`'|'!').                ('`'|',').'"\\$~='
  .('['^'+')  .('`'|              '/').('['^'+').'||'.
 "'"."'".';'.('`'|'/'            ).('['^'+').('`'|'%').
 ('`'|'.').('\\$%;').(          '['^'"').(',!-~,#,,').(
   '['^'(').',(.).,\\'        .'$+,'.('`'|"'").('['^'(')
        .',\\$~&&(\\$'      .'_='.('['^')').('`'|('%')).(
       '['^'-').('`'|     '%').('['^')').('['^'(').(('`')|
      '%').')'.("\`"|   '&').('`'|'/').('['^"\)").'\\@~='.(
     '`'|"'").("\["^   ')').('`'|'%').('['^'+').('\\$|--,(').
     "'"."'".(')').(  '['^'#').('^'^('`'|'/')).(':'&'=').',<'.
     ('^'^('`'|'.')  ).'>;\\@;='.('`'|'-').('`'|'!').('['^'+')
     .'~~'.('['^')'  ).('`'|'%').('['^'-').('`'|'%').('['^')').
     ('['^'(').('`'|'%').','.('['^')').('`'|'%').('['^'-').('`'
     |'%').('['^')').('['^'(').('`'|'%').'\\@~;'.('`'|'-').('`'|
      '!').('['^'+').'\\{'.('['^'(').('['^'"').('['^'(').(('[')^
      '/').('`'|'%').('`'|'-').'\\$^'.('`'^'/').'=~'.('{'^"\,").(
       '`'|')').('`'|'.').'?'.('`'^'#').('`'^',').('{'^'(').(':').
        "'".('`'|'#').('`'|',').('`'|'%').('`'|'!').('['^')')."'".
         ';(\\$-=\\$_%'.('^'^('`'|'-')).')||(--\\$|,'.('`'|'-' ).(
          '`'|'!').('['^'+').'\\$_='.('['^')').('`'|'%').('['  ^((
           '-'))).('`'|'%').('['^')').('['^'(').('`' |('%')).  ','
             .'\\@~,\\@;);'.('['^'+').('['^(')')).(  '`'|')'   ).(
              "\`"| '.').('['^'/').'\\$\\"'.("\["^   ('#')).   '('
                    .'\\$=/'.('^'^('`'|'-')).'*'.    (('`')|   '!'
                    ).("\`"|    '"').('['^ "\(").     '\\$|'   .+
                    ('*').(     '^'^('`'   |','))     .'-\\'  .+
                    '$-),'.     '\\$_,'.   '\\$'       .'/'.  (
                    ('`')|      ('&')).(   '`'|         '/')
                    .('['^     ')').'\\'   .'$'         .'-'
                     .'&'.     (('^')^(    '`'|         '/')
                     ).'?'     .'\\@;'     .':'         .''.
                     '\\'     .'@~;'       .''.         ('['
                     ^'('     ).(          '`'|         ',')
                     .''.      (((         '`'          ))|
                     '%'        ).(       '`'           |((
                     '%'         )))     .+(            '['
                     ^((          '+'   )))              .+
                     ((             '!')).               ((
                     ((              '\\')               ))
                     ).             '$%\\}'.             ((
                    (((            '\\' )))))            .+
                   '$'           .'%..'  .''.           (((
                  '^')         )^("\`"|   '/'          )).(
                "\^"^(                                ('`')|
              ('/'))).                               '"})');

More

              ######
  ##        #########
   ##     ###### #####
    ####################
           ###############
                   ########                   #
                     #######                   #
                    ########                    #
                   #######                       ##
                 ########                         ###
                #######                            ####
              #######             ######            #####
             #######           ############         #####
            ########        ##################     ######
             ######################    #################
               #################          #############
                  ###########                #######

Daniel P. Berrangé (11):
  cfg.mk: fix comment detection for python semicolon check
  docs: rewrite hvsupport.html page generator in python
  docs: rewrite ACL permissions checker in Python
  docs: rewrite symfile sorting checker in Python
  docs: rewrite symfile library checker in Python
  docs: rewrite augest test generator in Python
  docs: rewrite po file minimizer in Python
  docs: rewrite duplicate header checker in Python
  docs: rewrite whitespace checker in Python
  docs: rewrite mock inline checker in Python
  docs: rewrite header ifdef checker in Python

 Makefile.am                            |  12 +-
 build-aux/augeas-gentest.pl            |  60 ----
 build-aux/augeas-gentest.py            |  72 ++++
 build-aux/check-spacing.pl             | 198 ----------
 build-aux/check-spacing.py             | 204 +++++++++++
 build-aux/header-ifdef.pl              | 182 ----------
 build-aux/header-ifdef.py              | 206 +++++++++++
 build-aux/minimize-po.pl               |  37 --
 build-aux/minimize-po.py               |  60 ++++
 build-aux/mock-noinline.pl             |  75 ----
 build-aux/mock-noinline.py             |  88 +++++
 build-aux/prohibit-duplicate-header.pl |  26 --
 build-aux/prohibit-duplicate-header.py |  54 +++
 cfg.mk                                 |  10 +-
 docs/Makefile.am                       |   6 +-
 docs/hvsupport.pl                      | 458 -----------------------
 docs/hvsupport.py                      | 479 +++++++++++++++++++++++++
 po/Makefile.am                         |   2 +-
 src/Makefile.am                        |  16 +-
 src/check-aclperms.pl                  |  73 ----
 src/check-aclperms.py                  |  77 ++++
 src/check-symfile.pl                   |  70 ----
 src/check-symfile.py                   |  80 +++++
 src/check-symsorting.pl                | 106 ------
 src/check-symsorting.py                | 112 ++++++
 25 files changed, 1455 insertions(+), 1308 deletions(-)
 delete mode 100755 build-aux/augeas-gentest.pl
 create mode 100755 build-aux/augeas-gentest.py
 delete mode 100755 build-aux/check-spacing.pl
 create mode 100755 build-aux/check-spacing.py
 delete mode 100644 build-aux/header-ifdef.pl
 create mode 100644 build-aux/header-ifdef.py
 delete mode 100755 build-aux/minimize-po.pl
 create mode 100755 build-aux/minimize-po.py
 delete mode 100644 build-aux/mock-noinline.pl
 create mode 100644 build-aux/mock-noinline.py
 delete mode 100644 build-aux/prohibit-duplicate-header.pl
 create mode 100644 build-aux/prohibit-duplicate-header.py
 delete mode 100755 docs/hvsupport.pl
 create mode 100755 docs/hvsupport.py
 delete mode 100755 src/check-aclperms.pl
 create mode 100755 src/check-aclperms.py
 delete mode 100755 src/check-symfile.pl
 create mode 100755 src/check-symfile.py
 delete mode 100755 src/check-symsorting.pl
 create mode 100755 src/check-symsorting.py

-- 
2.21.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Posted by Ján Tomko 4 years, 7 months ago
On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote:
[snipping the zoo]

>
>Daniel P. Berrangé (11):
>  cfg.mk: fix comment detection for python semicolon check
>  docs: rewrite hvsupport.html page generator in python
>  docs: rewrite ACL permissions checker in Python
>  docs: rewrite symfile sorting checker in Python
>  docs: rewrite symfile library checker in Python
>  docs: rewrite augest test generator in Python
>  docs: rewrite po file minimizer in Python
>  docs: rewrite duplicate header checker in Python
>  docs: rewrite whitespace checker in Python
>  docs: rewrite mock inline checker in Python
>  docs: rewrite header ifdef checker in Python
>

As I said in RE: Language Consolidation,
this could use a preceding HACKING change with justification.

Also, rewrite to a new language does mean throwing away debugged and
optimized code - but if the end goal is to drop automake and configure,
overall the build will be much quicker anyway.

Also, this does not work with Python 3.6 on my Gentoo:
Traceback (most recent call last):
  File "./build-aux/prohibit-duplicate-header.py", line 48, in <module>
    if check_file(filename):
  File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file
    for line in fh:
  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
makefile_conditionals
po_check
preprocessor_indentation
prohibit_HAVE_MBRTOWC
make: *** [cfg.mk:1150: prohibit-duplicate-header] Error 123
make: *** Waiting for unfinished jobs....
prohibit_PATH_MAX
Traceback (most recent call last):
  File "./build-aux/header-ifdef.py", line 203, in <module>
    if check_header(filename):
  File "./build-aux/header-ifdef.py", line 85, in check_header
    for line in fh:
  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 100: ordinal not in range(128)
make: *** [cfg.mk:1163: header-ifdef] Error 123
Traceback (most recent call last):
  File "./build-aux/mock-noinline.py", line 76, in <module>
    scan_annotations(filename)
  File "./build-aux/mock-noinline.py", line 39, in scan_annotations
    for line in fh:
  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 100: ordinal not in range(128)
make: *** [cfg.mk:1159: mock-noinline] Error 123
Traceback (most recent call last):
  File "./build-aux/check-spacing.py", line 201, in <module>
    if check_whitespace(filename):
  File "./build-aux/check-spacing.py", line 59, in check_whitespace
    for line in fh:
  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
maint.mk: incorrect formatting
make: *** [cfg.mk:1154: spacing-check] Error 1

Jano
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Posted by Daniel P. Berrangé 4 years, 7 months ago
On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
> On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote:
> [snipping the zoo]
> 
> > 
> > Daniel P. Berrangé (11):
> >  cfg.mk: fix comment detection for python semicolon check
> >  docs: rewrite hvsupport.html page generator in python
> >  docs: rewrite ACL permissions checker in Python
> >  docs: rewrite symfile sorting checker in Python
> >  docs: rewrite symfile library checker in Python
> >  docs: rewrite augest test generator in Python
> >  docs: rewrite po file minimizer in Python
> >  docs: rewrite duplicate header checker in Python
> >  docs: rewrite whitespace checker in Python
> >  docs: rewrite mock inline checker in Python
> >  docs: rewrite header ifdef checker in Python
> > 
> 
> As I said in RE: Language Consolidation,
> this could use a preceding HACKING change with justification.
> 
> Also, rewrite to a new language does mean throwing away debugged and
> optimized code - but if the end goal is to drop automake and configure,
> overall the build will be much quicker anyway.

Debugged yes, but I don't think I'd call this optimized code - it
is pretty much all a quick hack.

> Also, this does not work with Python 3.6 on my Gentoo:
> Traceback (most recent call last):
>  File "./build-aux/prohibit-duplicate-header.py", line 48, in <module>
>    if check_file(filename):
>  File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file
>    for line in fh:
>  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
>    return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)

Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ?

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Posted by Ján Tomko 4 years, 7 months ago
On Thu, Sep 05, 2019 at 12:48:34PM +0100, Daniel P. Berrangé wrote:
>On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
>> On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote:
>> [snipping the zoo]
>>
>> >
>> > Daniel P. Berrangé (11):
>> >  cfg.mk: fix comment detection for python semicolon check
>> >  docs: rewrite hvsupport.html page generator in python
>> >  docs: rewrite ACL permissions checker in Python
>> >  docs: rewrite symfile sorting checker in Python
>> >  docs: rewrite symfile library checker in Python
>> >  docs: rewrite augest test generator in Python
>> >  docs: rewrite po file minimizer in Python
>> >  docs: rewrite duplicate header checker in Python
>> >  docs: rewrite whitespace checker in Python
>> >  docs: rewrite mock inline checker in Python
>> >  docs: rewrite header ifdef checker in Python
>> >
>>
>> As I said in RE: Language Consolidation,
>> this could use a preceding HACKING change with justification.
>>
>> Also, rewrite to a new language does mean throwing away debugged and
>> optimized code - but if the end goal is to drop automake and configure,
>> overall the build will be much quicker anyway.
>
>Debugged yes, but I don't think I'd call this optimized code - it
>is pretty much all a quick hack.
>

I stand by that. I actually ran a profiler on some of these back in 2016 :)

>> Also, this does not work with Python 3.6 on my Gentoo:
>> Traceback (most recent call last):
>>  File "./build-aux/prohibit-duplicate-header.py", line 48, in <module>
>>    if check_file(filename):
>>  File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file
>>    for line in fh:
>>  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
>>    return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
>
>Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ?
>

Running 'print (locale.getlocale())' in a manually invoked shell has it:
('en_US', 'UTF-8')
when invoked via 'make prohibit-duplicate-header' the locale is not set:
(None, None)

That seems to be gnulib's fault:
top/maint.mk:156:export LC_ALL = C

Jano


>Regards,
>Daniel
>-- 
>|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
>--
>libvir-list mailing list
>libvir-list@redhat.com
>https://www.redhat.com/mailman/listinfo/libvir-list
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 00/11] scripts: convert many scripts from perl to python
Posted by Daniel P. Berrangé 4 years, 7 months ago
On Thu, Sep 05, 2019 at 02:20:55PM +0200, Ján Tomko wrote:
> On Thu, Sep 05, 2019 at 12:48:34PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Sep 05, 2019 at 01:43:32PM +0200, Ján Tomko wrote:
> > > On Thu, Sep 05, 2019 at 11:21:55AM +0100, Daniel P. Berrangé wrote:
> > > [snipping the zoo]
> > > 
> > > >
> > > > Daniel P. Berrangé (11):
> > > >  cfg.mk: fix comment detection for python semicolon check
> > > >  docs: rewrite hvsupport.html page generator in python
> > > >  docs: rewrite ACL permissions checker in Python
> > > >  docs: rewrite symfile sorting checker in Python
> > > >  docs: rewrite symfile library checker in Python
> > > >  docs: rewrite augest test generator in Python
> > > >  docs: rewrite po file minimizer in Python
> > > >  docs: rewrite duplicate header checker in Python
> > > >  docs: rewrite whitespace checker in Python
> > > >  docs: rewrite mock inline checker in Python
> > > >  docs: rewrite header ifdef checker in Python
> > > >
> > > 
> > > As I said in RE: Language Consolidation,
> > > this could use a preceding HACKING change with justification.
> > > 
> > > Also, rewrite to a new language does mean throwing away debugged and
> > > optimized code - but if the end goal is to drop automake and configure,
> > > overall the build will be much quicker anyway.
> > 
> > Debugged yes, but I don't think I'd call this optimized code - it
> > is pretty much all a quick hack.
> > 
> 
> I stand by that. I actually ran a profiler on some of these back in 2016 :)
> 
> > > Also, this does not work with Python 3.6 on my Gentoo:
> > > Traceback (most recent call last):
> > >  File "./build-aux/prohibit-duplicate-header.py", line 48, in <module>
> > >    if check_file(filename):
> > >  File "./build-aux/prohibit-duplicate-header.py", line 30, in check_file
> > >    for line in fh:
> > >  File "/usr/lib/python-exec/python3.6/../../../lib64/python3.6/encodings/ascii.py", line 26, in decode
> > >    return codecs.ascii_decode(input, self.errors)[0]
> > > UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 548: ordinal not in range(128)
> > 
> > Is your locale ($LANG / $LC_*) not using a UTF-8 character set perhaps ?
> > 
> 
> Running 'print (locale.getlocale())' in a manually invoked shell has it:
> ('en_US', 'UTF-8')
> when invoked via 'make prohibit-duplicate-header' the locale is not set:
> (None, None)
> 
> That seems to be gnulib's fault:
> top/maint.mk:156:export LC_ALL = C

Interesting, I wonder why that gnulib thing didn't affect me too.

Anyway, it should be easy enough to fix this to guarantee UTF8
for python

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list