[libvirt] [PATCH] docs: document that C & Python are the preferred languages

Daniel P. Berrangé posted 1 patch 2 weeks ago
Test syntax-check passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/libvirt tags/patchew/20190905114237.19748-1-berrange@redhat.com
docs/hacking.html.in | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)

[libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Daniel P. Berrangé 2 weeks ago
Blacklist Perl and Shell code in favour of Python for
sake of readability and portability.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/hacking.html.in | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/docs/hacking.html.in b/docs/hacking.html.in
index 90376968c3..8572eb65a6 100644
--- a/docs/hacking.html.in
+++ b/docs/hacking.html.in
@@ -343,6 +343,36 @@
         Richard Jones' guide to working with open source projects</a>.
     </p>
 
+    <h2><a id="lang">Language Usage</a></h2>
+
+    <p>
+      The libvirt repository makes use of a large number of programming
+      languages. There is a general desire to phase out some of the
+      existing languages used to reduce the knowledge burden on
+      developers, and facilitate introduction of new languages in
+      the future.
+    </p>
+
+    <p>
+      The preferred languages at this time are
+    </p>
+
+    <ul>
+      <li>C - for the main libvirt codebase. Dialect supported by
+        GCC/CLang only.</li>
+      <li>Python - for supporting build scripts / tools. Code must
+        run with both version 2.7 and 3.x at this time.</li>
+    </ul>
+
+    <p>
+      Languages that should not be used for any new contributions.
+    </p>
+
+    <ul>
+      <li>Perl - build scripts must be written in Python instead.</li>
+      <li>Shell - build scripts must be written in Python instead.</li>
+    </ul>
+
     <h2><a id="tooling">Tooling</a></h2>
 
     <p>
-- 
2.21.0

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Andrea Bolognani 2 weeks ago
On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
> +    <p>
> +      The libvirt repository makes use of a large number of programming
> +      languages. There is a general desire to phase out some of the
> +      existing languages used to reduce the knowledge burden on
> +      developers, and facilitate introduction of new languages in
> +      the future.

Reducing the number of languages used by the project and facilitating
the introduction of more languages seem to be contrasting goals.
Accordingly, I would leave out the last part of the sentence.

> +      <li>C - for the main libvirt codebase. Dialect supported by
> +        GCC/CLang only.</li>
> +      <li>Python - for supporting build scripts / tools. Code must
> +        run with both version 2.7 and 3.x at this time.</li>

Instead of esplicitly singling out 2.7 and 3.x, I would just say that
for both C and Python there it is required to support all platforms
listed in "platforms.html".

> +    <p>
> +      Languages that should not be used for any new contributions.

s/contributions./contributions:/

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Daniel P. Berrangé 2 weeks ago
On Thu, Sep 05, 2019 at 04:22:17PM +0200, Andrea Bolognani wrote:
> On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
> > +    <p>
> > +      The libvirt repository makes use of a large number of programming
> > +      languages. There is a general desire to phase out some of the
> > +      existing languages used to reduce the knowledge burden on
> > +      developers, and facilitate introduction of new languages in
> > +      the future.
> 
> Reducing the number of languages used by the project and facilitating
> the introduction of more languages seem to be contrasting goals.
> Accordingly, I would leave out the last part of the sentence.

That are actually directly related. The aim is to eliminate some
existing languages, so that when we add more languages, we've not
increased the overall burden.

> 
> > +      <li>C - for the main libvirt codebase. Dialect supported by
> > +        GCC/CLang only.</li>
> > +      <li>Python - for supporting build scripts / tools. Code must
> > +        run with both version 2.7 and 3.x at this time.</li>
> 
> Instead of esplicitly singling out 2.7 and 3.x, I would just say that
> for both C and Python there it is required to support all platforms
> listed in "platforms.html".

I think its important to call our both python versions explicitly
as most distros support both versions in one way or another, so
its not clear from platforms.html that we want to support both.

Adopting Meson will eventually force the matter as that requires
python 3, so we'll have to drop py2 at that point.

> > +    <p>
> > +      Languages that should not be used for any new contributions.
> 
> s/contributions./contributions:/

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Andrea Bolognani 2 weeks ago
On Thu, 2019-09-05 at 15:42 +0100, Daniel P. Berrangé wrote:
> On Thu, Sep 05, 2019 at 04:22:17PM +0200, Andrea Bolognani wrote:
> > On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
> > > +      The libvirt repository makes use of a large number of programming
> > > +      languages. There is a general desire to phase out some of the
> > > +      existing languages used to reduce the knowledge burden on
> > > +      developers, and facilitate introduction of new languages in
> > > +      the future.
> > 
> > Reducing the number of languages used by the project and facilitating
> > the introduction of more languages seem to be contrasting goals.
> > Accordingly, I would leave out the last part of the sentence.
> 
> That are actually directly related. The aim is to eliminate some
> existing languages, so that when we add more languages, we've not
> increased the overall burden.

I think the fact that we want to add more languages only makes
reducing the number of languages more pressing than it would be
otherwise, but reducing the cognitive load for contributors is a
worthy goal in its own right and alone would be enough to justify
standardizing on Python in my opinion. So I'd still prefer it if
you dropped that last part of the sentence, but I won't insist
further if you're adamant that you want to keep it.

> > > +      <li>C - for the main libvirt codebase. Dialect supported by
> > > +        GCC/CLang only.</li>
> > > +      <li>Python - for supporting build scripts / tools. Code must
> > > +        run with both version 2.7 and 3.x at this time.</li>
> > 
> > Instead of esplicitly singling out 2.7 and 3.x, I would just say that
> > for both C and Python there it is required to support all platforms
> > listed in "platforms.html".
> 
> I think its important to call our both python versions explicitly
> as most distros support both versions in one way or another, so
> its not clear from platforms.html that we want to support both.
> 
> Adopting Meson will eventually force the matter as that requires
> python 3, so we'll have to drop py2 at that point.

Alright.

> > > +    <p>
> > > +      Languages that should not be used for any new contributions.
> > 
> > s/contributions./contributions:/

If you fix this one:

  Reviewed-by: Andrea Bolognani <abologna@redhat.com>

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Laine Stump 2 weeks ago
On 9/5/19 10:51 AM, Andrea Bolognani wrote:
> On Thu, 2019-09-05 at 15:42 +0100, Daniel P. Berrangé wrote:
>> On Thu, Sep 05, 2019 at 04:22:17PM +0200, Andrea Bolognani wrote:
>>> On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
>>>> +      The libvirt repository makes use of a large number of programming
>>>> +      languages. There is a general desire to phase out some of the
>>>> +      existing languages used to reduce the knowledge burden on
>>>> +      developers, and facilitate introduction of new languages in
>>>> +      the future.
>>> Reducing the number of languages used by the project and facilitating
>>> the introduction of more languages seem to be contrasting goals.
>>> Accordingly, I would leave out the last part of the sentence.
>> That are actually directly related. The aim is to eliminate some
>> existing languages, so that when we add more languages, we've not
>> increased the overall burden.
> I think the fact that we want to add more languages only makes
> reducing the number of languages more pressing than it would be
> otherwise, but reducing the cognitive load for contributors is a
> worthy goal in its own right and alone would be enough to justify
> standardizing on Python in my opinion. So I'd still prefer it if
> you dropped that last part of the sentence, but I won't insist
> further if you're adamant that you want to keep it.


Maybe part of the reason he's saying that is so that the statement can't 
be used in the future as an argument against adding a new language?


How would you feel about something like:


"reduce the knowledge burden for old/lame languages on developers and make room in their brains for a more painless introduction of new/cool languages into the codebase in the future."


(BTW, what does the removal of perl from libvirt say about continued use of perl for libvirt-tck? There are a lot of useful tests in there that find real bugs, but they tend to languish (both in use and in enhancements) because nobody wants to do anything with perl (or with the big shell scripts that do the nwfilter testing). It would be nice if the tests were all in a language that was more accessible, but they're kind of married to the perl TAP module (and besides, who wants to spend time rewriting a bunch of test scripts when they already work?). This is mostly a moot point, because I think hardly anyone runs the libvirt-tck tests anymore, which is too bad because it has historically caught some regressions that no other testing framework did.)

>
>>>> +      <li>C - for the main libvirt codebase. Dialect supported by
>>>> +        GCC/CLang only.</li>
>>>> +      <li>Python - for supporting build scripts / tools. Code must
>>>> +        run with both version 2.7 and 3.x at this time.</li>
>>> Instead of esplicitly singling out 2.7 and 3.x, I would just say that
>>> for both C and Python there it is required to support all platforms
>>> listed in "platforms.html".
>> I think its important to call our both python versions explicitly
>> as most distros support both versions in one way or another, so
>> its not clear from platforms.html that we want to support both.
>>
>> Adopting Meson will eventually force the matter as that requires
>> python 3, so we'll have to drop py2 at that point.
> Alright.
>
>>>> +    <p>
>>>> +      Languages that should not be used for any new contributions.
>>> s/contributions./contributions:/
> If you fix this one:
>
>    Reviewed-by: Andrea Bolognani <abologna@redhat.com>
>

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Daniel P. Berrangé 2 weeks ago
On Thu, Sep 05, 2019 at 12:30:27PM -0400, Laine Stump wrote:
> On 9/5/19 10:51 AM, Andrea Bolognani wrote:
> > On Thu, 2019-09-05 at 15:42 +0100, Daniel P. Berrangé wrote:
> > > On Thu, Sep 05, 2019 at 04:22:17PM +0200, Andrea Bolognani wrote:
> > > > On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
> > > > > +      The libvirt repository makes use of a large number of programming
> > > > > +      languages. There is a general desire to phase out some of the
> > > > > +      existing languages used to reduce the knowledge burden on
> > > > > +      developers, and facilitate introduction of new languages in
> > > > > +      the future.
> > > > Reducing the number of languages used by the project and facilitating
> > > > the introduction of more languages seem to be contrasting goals.
> > > > Accordingly, I would leave out the last part of the sentence.
> > > That are actually directly related. The aim is to eliminate some
> > > existing languages, so that when we add more languages, we've not
> > > increased the overall burden.
> > I think the fact that we want to add more languages only makes
> > reducing the number of languages more pressing than it would be
> > otherwise, but reducing the cognitive load for contributors is a
> > worthy goal in its own right and alone would be enough to justify
> > standardizing on Python in my opinion. So I'd still prefer it if
> > you dropped that last part of the sentence, but I won't insist
> > further if you're adamant that you want to keep it.
> 
> 
> Maybe part of the reason he's saying that is so that the statement can't be
> used in the future as an argument against adding a new language?

Generally speaking it *should* be used as an argument against adding
new languages. Given current state of the world I'd only make exceptions
for Go or Rust, because they are the only credible options for replacing
use of C as an efficient systems programming language. I get what you're
saying though.

> How would you feel about something like:
> 
> "reduce the knowledge burden for old/lame languages on developers and 
>  make room in their brains for a more painless introduction of new/cool
>  languages into the codebase in the future."

Lemme think about it some more.

Mostly what I was struggling with was trying not to promise a bunch of
ponies that we've not yet delivered on. The HACKING file should
really reflect what current committed best practice is. THings would
be much clearer if I could set out a general "vision" or "roadmap" for
language usage, where I could talk about where we want to get to in
5 years hence, despite it not being the case today.

> (BTW, what does the removal of perl from libvirt say about continued
> use of perl for libvirt-tck? There are a lot of useful tests in there
> that find real bugs, but they tend to languish (both in use and in
> enhancements) because nobody wants to do anything with perl (or with
> the big shell scripts that do the nwfilter testing). It would be nice
> if the tests were all in a language that was more accessible, but
> they're kind of married to the perl TAP module (and besides, who
> wants to spend time rewriting a bunch of test scripts when they
> already work?). This is mostly a moot point, because I think hardly
> anyone runs the libvirt-tck tests anymore, which is too bad because
> it has historically caught some regressions that no other testing
> framework did.)

I believe they are run by the Red Hat virt QE team, as I ave
got bug reports against the Perl libvirt bindings, where the
reproducer is a TCK script.

The TCK really covered three distinct things

 - framework for controlling test execution
 - library modules/helper APIs to facilitate test creation
 - the large set of indivdual tests

There is no code binding between the tests and the framework. They
communicate exclusively through the TAP protocol. In that way we can
throw out the current framework and drop in a different one quite
easily - the new framework merely needs to be able to consume the
TAP data format. For any framework which does not natively support
the TAP format, it is quite easy to write an adapter / shim that
would convert to the desired data format. You might loose some
finese / fine grained details in the logs/progress reporting but
functionally the tests should work fine.

As a point of reference the GLib testing harness has recently
switched from using their own custom output format to using the
TAP output format. So if we adopt GLib in the libvirt.git and
use it in our unit tests those would end up using TAP. GLib
provides a shim for automake that allows it to consume the
results from GLib based units tests.

QEMU's makefiles recently switched to using TAP for its unit
and (some of) its functional tests too.

IOW, I'm perfectly happy to throw away the existing TCK framework
impl if someone proposes a better alternative.


As for the individual TCK tests themselves, there's two distinct
questions

 - Should they be re-written in a different language
 - Should they continue to use TAP output format

If we are not going to rewrite the tests, then I think they should
continue to use TAP output format. It is the easiest thing to use
when writing tests in Perl.

If we are going to rewrite the tests, then the tests should ideally
use the output format that best suits the framework invoking them.
IOW, if the framework still uses TAP, we should continue to use
TAP. If not, then don't. There is an adapter for python that can
convert its native unittest output format into TAP format.


I'm pretty ambivalent on rewriting the existing tests. The language
of existing tests only matters if you expect people to be modifying
them. Assuming a decent support library it should be no harder to
define a new test, than to extend an existing test.

IOW, if we want to write TCK tests in Python, we don't really need
to rewrite existing ones. We instead need to write a Python variant
of the helper APIs, so that you can trivially write new Python tests
which operate in the same manner as the Perl tests.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Andrea Bolognani 2 weeks ago
On Thu, 2019-09-05 at 18:14 +0100, Daniel P. Berrangé wrote:
> On Thu, Sep 05, 2019 at 12:30:27PM -0400, Laine Stump wrote:
> > On 9/5/19 10:51 AM, Andrea Bolognani wrote:
> > > On Thu, 2019-09-05 at 15:42 +0100, Daniel P. Berrangé wrote:
> > > > On Thu, Sep 05, 2019 at 04:22:17PM +0200, Andrea Bolognani wrote:
> > > > > On Thu, 2019-09-05 at 12:42 +0100, Daniel P. Berrangé wrote:
> > > > > > +      The libvirt repository makes use of a large number of programming
> > > > > > +      languages. There is a general desire to phase out some of the
> > > > > > +      existing languages used to reduce the knowledge burden on
> > > > > > +      developers, and facilitate introduction of new languages in
> > > > > > +      the future.
> > > > > 
> > > > > Reducing the number of languages used by the project and facilitating
> > > > > the introduction of more languages seem to be contrasting goals.
> > > > > Accordingly, I would leave out the last part of the sentence.
> > > > 
> > > > That are actually directly related. The aim is to eliminate some
> > > > existing languages, so that when we add more languages, we've not
> > > > increased the overall burden.
> > > 
> > > I think the fact that we want to add more languages only makes
> > > reducing the number of languages more pressing than it would be
> > > otherwise, but reducing the cognitive load for contributors is a
> > > worthy goal in its own right and alone would be enough to justify
> > > standardizing on Python in my opinion. So I'd still prefer it if
> > > you dropped that last part of the sentence, but I won't insist
> > > further if you're adamant that you want to keep it.
> > 
> > Maybe part of the reason he's saying that is so that the statement can't be
> > used in the future as an argument against adding a new language?
> 
> Generally speaking it *should* be used as an argument against adding
> new languages. Given current state of the world I'd only make exceptions
> for Go or Rust, because they are the only credible options for replacing
> use of C as an efficient systems programming language. I get what you're
> saying though.
> 
> > How would you feel about something like:
> > 
> > "reduce the knowledge burden for old/lame languages on developers and 
> >  make room in their brains for a more painless introduction of new/cool
> >  languages into the codebase in the future."
> 
> Lemme think about it some more.
> 
> Mostly what I was struggling with was trying not to promise a bunch of
> ponies that we've not yet delivered on. The HACKING file should
> really reflect what current committed best practice is. THings would
> be much clearer if I could set out a general "vision" or "roadmap" for
> language usage, where I could talk about where we want to get to in
> 5 years hence, despite it not being the case today.

How about we add a paragraph at the end of the section that says
something along the lines of

  Both the lists above are subject to change: code written in a new
  programming language might be accepted if it's generally agreed
  that the advantages of using that specific language instead of one
  of the preferred ones outweight the resulting increase in cognitive
  load, and that might in turn result in one of more of the preferred
  languages losing their status; in the same way, languages that have
  been purged from the project in the past might once again become
  acceptable, although such a reversal would require an even more
  impressive cost/benefit ratio.

?

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH] docs: document that C & Python are the preferred languages

Posted by Laine Stump 2 weeks ago
On 9/5/19 1:57 PM, Andrea Bolognani wrote:
> How about we add a paragraph at the end of the section that says
> something along the lines of
>
>    Both the lists above are subject to change: code written in a new
>    programming language might be accepted if it's generally agreed
>    that the advantages of using that specific language instead of one
>    of the preferred ones outweight the resulting increase in cognitive
>    load, and that might in turn result in one of more of the preferred
>    languages losing their status; in the same way, languages that have
>    been purged from the project in the past might once again become
>    acceptable, although such a reversal would require an even more
>    impressive cost/benefit ratio.
>
> ?


That *definitely* needs more cowbell!



--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list