This series is an effort to reduce the number of different
languages we use by eliminating most use of perl in favour
of python.

This aligns with fact that the likely future build system
we'll use (meson) is written in python, and that python
is much more commonly used/understood by developers these
days than perl.

With this applied we use perl in a handful of places only:

 - src/rpc/gendispatch.pl - this is a horrendously large
   script and very hard to understand/follow. A straight
   syntax conversion to Python would still leave a hgue
   and hard to understand/follow script. It really needs
   a clean room rewrite from scratch, with better structure.

 - src/rpc/genprotocol.pl - fairly easy to convert, but
   might be obsolete depending on approach for rewriting
   gendispatch.pl, so ignored for now

 - tests/oomtrace.pl - will be purge by the patches that
   drop OOM handling anyway

 - tools/wireshark/util/genxdrstub.pl - a very large
   script, which I haven't got the courage to tackle
   yet.

 - cfg.mk/maint.mk - many syntax rules involve regexes
   which are fed to perl. Decision on what to do
   with syntax-check rules punted to another time.

 - build-aux/gitlog-to-changelog
 - build-aux/useless-if-before-free - Both pulled in
   from gnulib. Could be rewritten quite easily if
   desired, but given that we aren't maintainers of
   them right now, they're ignored as they don't
   really impact our developers.

Note that the check-spacing.py script is significantly
slower in Python than in Perl. After researching this
it appears there is nothing that can be done. The Perl
regex engine is simply much better optimized than the
Python one.  As previously discussed we need to loook
at uncrustify or clang-format or some other tool to
validate whitespace formatting. This is ongoing. We
can either accept the slow down in the short term or
keep the Perl version in the short term.

In v4:

 - Moved all scripts into the scripts/ directory instead
   of having them scattered around source tree
 - Use re.search instead of re.match
 - Don't bother re-compiling regexes

In v3:

 - All scripts comply with all flake8 style rules with
   exception of

    E129 visually indented line with same indent as next logical line

In v2:

 - Pulled in patch to hacking file
 - Converted many more scripts
 - Forced UTF-8 character set to avoid ascii codec
   on py3 < 3.7

Daniel P. Berrangé (26):
  build: fix use of $(AUG_GENTEST) as a dependency
  build-aux: rewrite augeas test generator in Python
  build-aux: rewrite po file minimizer in Python
  build-aux: rewrite duplicate header checker in Python
  build-aux: rewrite whitespace checker in Python
  build-aux: rewrite mock inline checker in Python
  build-aux: rewrite header ifdef checker in Python
  src: rewrite ACL permissions checker in Python
  src: rewrite symfile sorting checker in Python
  src: rewrite symfile library checker in Python
  src: rewrite systemtap probe generator in Python
  src: rewrite systemtap function generator in Python
  src: rewrite driver name checker in Python
  src: rewrite driver impl checker in Python
  src: rewrite ACL rule checker in Python
  src: rewrite polkit ACL generator in Python
  src: rewrite remote protocol checker in Python
  tests: rewrite test argv line wrapper in Python
  tests: rewrite qemu capability grouper in Python
  tests: rewrite file access checker in Python
  docs: rewrite hvsupport.html page generator in python
  docs: rewrite polkit docs generator in Python
  docs: move apibuild.py to the scripts/ directory
  docs: move reformat-news.py to the scripts/ directory
  docs: move esx_vi_generator.py to the scripts/ directory
  docs: move hyperv_wmi_generator.py to the scripts/ directory

 .gitignore                                    |   3 +-
 Makefile.am                                   |  37 +-
 build-aux/augeas-gentest.pl                   |  60 ---
 build-aux/check-spacing.pl                    | 198 -------
 build-aux/header-ifdef.pl                     | 182 -------
 build-aux/minimize-po.pl                      |  37 --
 build-aux/mock-noinline.pl                    |  75 ---
 build-aux/prohibit-duplicate-header.pl        |  26 -
 cfg.mk                                        |  32 +-
 docs/Makefile.am                              |  16 +-
 docs/genaclperms.pl                           | 125 -----
 docs/hvsupport.pl                             | 458 ----------------
 po/Makefile.am                                |   2 +-
 {docs => scripts}/apibuild.py                 |   0
 scripts/augeas-gentest.py                     |  67 +++
 scripts/check-aclperms.py                     |  75 +++
 scripts/check-aclrules.py                     | 263 +++++++++
 scripts/check-driverimpls.py                  | 102 ++++
 scripts/check-drivername.py                   | 114 ++++
 scripts/check-file-access.py                  | 123 +++++
 scripts/check-remote-protocol.py              | 136 +++++
 scripts/check-spacing.py                      | 229 ++++++++
 scripts/check-symfile.py                      |  80 +++
 scripts/check-symsorting.py                   | 117 ++++
 scripts/dtrace2systemtap.py                   | 143 +++++
 {src/esx => scripts}/esx_vi_generator.py      |   0
 scripts/genaclperms.py                        | 123 +++++
 scripts/genpolkit.py                          | 122 +++++
 scripts/gensystemtap.py                       | 184 +++++++
 scripts/group-qemu-caps.py                    | 123 +++++
 scripts/header-ifdef.py                       | 231 ++++++++
 scripts/hvsupport.py                          | 503 ++++++++++++++++++
 .../hyperv_wmi_generator.py                   |   0
 scripts/minimize-po.py                        |  58 ++
 scripts/mock-noinline.py                      |  85 +++
 scripts/prohibit-duplicate-header.py          |  56 ++
 {docs => scripts}/reformat-news.py            |   0
 scripts/test-wrap-argv.py                     | 170 ++++++
 src/Makefile.am                               | 155 ++----
 src/access/Makefile.inc.am                    |   6 +-
 src/access/genpolkit.pl                       | 119 -----
 src/bhyve/Makefile.inc.am                     |   4 +-
 src/check-aclperms.pl                         |  73 ---
 src/check-aclrules.pl                         | 252 ---------
 src/check-driverimpls.pl                      |  80 ---
 src/check-drivername.pl                       |  83 ---
 src/check-symfile.pl                          |  70 ---
 src/check-symsorting.pl                       | 106 ----
 src/dtrace2systemtap.pl                       | 130 -----
 src/esx/Makefile.inc.am                       |   5 +-
 src/hyperv/Makefile.inc.am                    |   5 +-
 src/interface/Makefile.inc.am                 |   2 +-
 src/libxl/Makefile.inc.am                     |   4 +-
 src/locking/Makefile.inc.am                   |   6 +-
 src/logging/Makefile.inc.am                   |   2 +-
 src/lxc/Makefile.inc.am                       |   4 +-
 src/network/Makefile.inc.am                   |   2 +-
 src/node_device/Makefile.inc.am               |   2 +-
 src/nwfilter/Makefile.inc.am                  |   2 +-
 src/qemu/Makefile.inc.am                      |   4 +-
 src/remote/Makefile.inc.am                    |   4 +-
 src/rpc/Makefile.inc.am                       |   1 -
 src/rpc/gensystemtap.pl                       | 193 -------
 src/secret/Makefile.inc.am                    |   2 +-
 src/storage/Makefile.inc.am                   |   2 +-
 src/vbox/Makefile.inc.am                      |   2 +-
 src/vz/Makefile.inc.am                        |   2 +-
 tests/Makefile.am                             |   3 +-
 tests/check-file-access.pl                    | 126 -----
 tests/file_access_whitelist.txt               |   2 +-
 tests/group-qemu-caps.pl                      | 124 -----
 tests/test-wrap-argv.pl                       | 174 ------
 tests/testutils.c                             |  16 +-
 73 files changed, 3243 insertions(+), 2879 deletions(-)
 delete mode 100755 build-aux/augeas-gentest.pl
 delete mode 100755 build-aux/check-spacing.pl
 delete mode 100644 build-aux/header-ifdef.pl
 delete mode 100755 build-aux/minimize-po.pl
 delete mode 100644 build-aux/mock-noinline.pl
 delete mode 100644 build-aux/prohibit-duplicate-header.pl
 delete mode 100755 docs/genaclperms.pl
 delete mode 100755 docs/hvsupport.pl
 rename {docs => scripts}/apibuild.py (100%)
 create mode 100755 scripts/augeas-gentest.py
 create mode 100755 scripts/check-aclperms.py
 create mode 100755 scripts/check-aclrules.py
 create mode 100755 scripts/check-driverimpls.py
 create mode 100644 scripts/check-drivername.py
 create mode 100755 scripts/check-file-access.py
 create mode 100644 scripts/check-remote-protocol.py
 create mode 100755 scripts/check-spacing.py
 create mode 100755 scripts/check-symfile.py
 create mode 100755 scripts/check-symsorting.py
 create mode 100755 scripts/dtrace2systemtap.py
 rename {src/esx => scripts}/esx_vi_generator.py (100%)
 create mode 100755 scripts/genaclperms.py
 create mode 100755 scripts/genpolkit.py
 create mode 100755 scripts/gensystemtap.py
 create mode 100755 scripts/group-qemu-caps.py
 create mode 100644 scripts/header-ifdef.py
 create mode 100755 scripts/hvsupport.py
 rename {src/hyperv => scripts}/hyperv_wmi_generator.py (100%)
 create mode 100755 scripts/minimize-po.py
 create mode 100644 scripts/mock-noinline.py
 create mode 100644 scripts/prohibit-duplicate-header.py
 rename {docs => scripts}/reformat-news.py (100%)
 create mode 100755 scripts/test-wrap-argv.py
 delete mode 100755 src/access/genpolkit.pl
 delete mode 100755 src/check-aclperms.pl
 delete mode 100755 src/check-aclrules.pl
 delete mode 100755 src/check-driverimpls.pl
 delete mode 100755 src/check-drivername.pl
 delete mode 100755 src/check-symfile.pl
 delete mode 100755 src/check-symsorting.pl
 delete mode 100755 src/dtrace2systemtap.pl
 delete mode 100755 src/rpc/gensystemtap.pl
 delete mode 100755 tests/check-file-access.pl
 delete mode 100755 tests/group-qemu-caps.pl
 delete mode 100755 tests/test-wrap-argv.pl

-- 
2.21.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Wed, Oct 09, 2019 at 12:37:15PM +0100, Daniel P. Berrangé wrote:
>Note that the check-spacing.py script is significantly
>slower in Python than in Perl. After researching this
>it appears there is nothing that can be done. The Perl
>regex engine is simply much better optimized than the
>Python one.

I'm afraid immutability of strings does not help either.
I tried speeding it up a bit by running the regexes conditionally, e.g.:

+            if '"' in data:
+                data = quotedstringprog.sub('"XXX"', data)

That got it down from 0m6.368s to 0m3.802s (compared to perl's 1.5s),
but even before that change, I don't consider the Python version of this
one script to be more readable.

Note that perl's optimizations aren't perfect either - merely
moving all the checks into a subroutine resulted in a significant
slowdown.

>As previously discussed we need to loook
>at uncrustify or clang-format or some other tool to
>validate whitespace formatting. This is ongoing. We
>can either accept the slow down in the short term or
>keep the Perl version in the short term.
>

Just to set the expectations, running clang-format on all the
C files took me ~40 s. Not sure how much that can be efficiently
parallelized or if meson could be convinced to start clang-format
as the first thing for the 'check' target, even before starting
to compile anything.

Jano
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, Oct 10, 2019 at 03:28:39PM +0200, Ján Tomko wrote:
> On Wed, Oct 09, 2019 at 12:37:15PM +0100, Daniel P. Berrangé wrote:
> > Note that the check-spacing.py script is significantly
> > slower in Python than in Perl. After researching this
> > it appears there is nothing that can be done. The Perl
> > regex engine is simply much better optimized than the
> > Python one.
> 
> I'm afraid immutability of strings does not help either.
> I tried speeding it up a bit by running the regexes conditionally, e.g.:
> 
> +            if '"' in data:
> +                data = quotedstringprog.sub('"XXX"', data)
> 
> That got it down from 0m6.368s to 0m3.802s (compared to perl's 1.5s),
> but even before that change, I don't consider the Python version of this
> one script to be more readable.
> 
> Note that perl's optimizations aren't perfect either - merely
> moving all the checks into a subroutine resulted in a significant
> slowdown.
> 
> > As previously discussed we need to loook
> > at uncrustify or clang-format or some other tool to
> > validate whitespace formatting. This is ongoing. We
> > can either accept the slow down in the short term or
> > keep the Perl version in the short term.
> > 
> 
> Just to set the expectations, running clang-format on all the
> C files took me ~40 s. Not sure how much that can be efficiently
> parallelized or if meson could be convinced to start clang-format
> as the first thing for the 'check' target, even before starting
> to compile anything.

Not entirely surprising since it is actually doing proper parsing
of the C syntax, which is quite a bit of work. 

FWIW, when I'm writing Go code I have used emacs 'go-mode' and
this runs 'go fmt' to automatically fix your style problems
every time you save the file. So your code is basically always
correct, and you don't need a separate job to check it later.

For this approach to be viable though, you need to have an
automated CI check that is run at time of pushing to git master
to block pushes of code which is non-compliant. It is the kind 
of thing projects do when using pull requests & gating CI in 
gitlab/github though. So this is itself another can of worms.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

...

>
> FWIW, when I'm writing Go code I have used emacs 'go-mode' and
> this runs 'go fmt' to automatically fix your style problems
> every time you save the file. So your code is basically always
> correct, and you don't need a separate job to check it later.

Since you mentioned ^this, we should do the same for python code and use the
"black" tool, it has plugins for several editors (most importantly vim) as
well.

Erik

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Wed, Oct 09, 2019 at 12:37:15 +0100, Daniel Berrange wrote:

[...]

>  - src/rpc/gendispatch.pl
>  - src/rpc/genprotocol.pl
>  - tools/wireshark/util/genxdrstub.pl

[...]

> Note that the check-spacing.py script is significantly
> slower in Python than in Perl. After researching this
> it appears there is nothing that can be done. The Perl
> regex engine is simply much better optimized than the
> Python one.  As previously discussed we need to loook
> at uncrustify or clang-format or some other tool to
> validate whitespace formatting. This is ongoing. We
> can either accept the slow down in the short term or
> keep the Perl version in the short term.

I vote for keeping the perl version then. I don't see much value in
rewriting it into a much slower form just to drop it soon after.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Wed, Oct 09, 2019 at 02:01:37PM +0200, Peter Krempa wrote:
> On Wed, Oct 09, 2019 at 12:37:15 +0100, Daniel Berrange wrote:
>
> [...]
>
> >  - src/rpc/gendispatch.pl
> >  - src/rpc/genprotocol.pl
> >  - tools/wireshark/util/genxdrstub.pl
>
> [...]
>
> > Note that the check-spacing.py script is significantly
> > slower in Python than in Perl. After researching this
> > it appears there is nothing that can be done. The Perl
> > regex engine is simply much better optimized than the
> > Python one.  As previously discussed we need to loook
> > at uncrustify or clang-format or some other tool to
> > validate whitespace formatting. This is ongoing. We
> > can either accept the slow down in the short term or
> > keep the Perl version in the short term.
>
> I vote for keeping the perl version then. I don't see much value in
> rewriting it into a much slower form just to drop it soon after.

I second ^this opinion.

Regards,
Erik

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, 2019-10-10 at 08:55 +0200, Erik Skultety wrote:
> On Wed, Oct 09, 2019 at 02:01:37PM +0200, Peter Krempa wrote:
> > On Wed, Oct 09, 2019 at 12:37:15 +0100, Daniel Berrange wrote:
> > > Note that the check-spacing.py script is significantly
> > > slower in Python than in Perl. After researching this
> > > it appears there is nothing that can be done. The Perl
> > > regex engine is simply much better optimized than the
> > > Python one.  As previously discussed we need to loook
> > > at uncrustify or clang-format or some other tool to
> > > validate whitespace formatting. This is ongoing. We
> > > can either accept the slow down in the short term or
> > > keep the Perl version in the short term.
> > 
> > I vote for keeping the perl version then. I don't see much value in
> > rewriting it into a much slower form just to drop it soon after.
> 
> I second ^this opinion.

Same, especially if it puts more pressure on us finally adopting
something like clang-format.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, Oct 10, 2019 at 09:22:29AM +0200, Andrea Bolognani wrote:
> On Thu, 2019-10-10 at 08:55 +0200, Erik Skultety wrote:
> > On Wed, Oct 09, 2019 at 02:01:37PM +0200, Peter Krempa wrote:
> > > On Wed, Oct 09, 2019 at 12:37:15 +0100, Daniel Berrange wrote:
> > > > Note that the check-spacing.py script is significantly
> > > > slower in Python than in Perl. After researching this
> > > > it appears there is nothing that can be done. The Perl
> > > > regex engine is simply much better optimized than the
> > > > Python one.  As previously discussed we need to loook
> > > > at uncrustify or clang-format or some other tool to
> > > > validate whitespace formatting. This is ongoing. We
> > > > can either accept the slow down in the short term or
> > > > keep the Perl version in the short term.
> > > 
> > > I vote for keeping the perl version then. I don't see much value in
> > > rewriting it into a much slower form just to drop it soon after.
> > 
> > I second ^this opinion.
> 
> Same, especially if it puts more pressure on us finally adopting
> something like clang-format.

Since all the patches in this series are essentially independant, I'm
not going to repost this huge series just to drop that one patch.
Just consider it dropped locally.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list