[v6] Don't generate netlink .rst files inside $(srctree)

[PATCH v6 00/15] Don't generate netlink .rst files inside $(srctree)

Posted by Mauro Carvalho Chehab 3 months, 3 weeks ago

As discussed at:
   https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/

changeset f061c9f7d058 ("Documentation: Document each netlink family")
added a logic which generates *.rst files inside $(srctree). This is bad
when O=<BUILDDIR> is used.

A recent change renamed the yaml files used by Netlink, revealing a bad
side effect: as "make cleandocs" don't clean the produced files and symbols
appear duplicated for people that don't build the kernel from scratch.

This series adds an yaml parser extension and uses an index file with glob for
*. We opted to write such extension in a way that no actual yaml conversion
code is inside it. This makes it flexible enough to handle other types of yaml
files in the future. The actual yaml conversion logic were placed at 
netlink_yml_parser.py. 

As requested by YNL maintainers, this version has netlink_yml_parser.py
inside tools/net/ynl/pyynl/ directory. I don't like mixing libraries with
binaries, nor to have Python libraries spread all over the Kernel. IMO,
the best is to put all of them on a common place (scripts/lib, python/lib,
lib/python, ...) but, as this can be solved later, for now let's keep it this
way.

---

v6:
- YNL doc parser is now at tools/net/ynl/pyynl/lib/doc_generator.py;
- two patches got merged;
- added instructions to test docs with Sphinx 3.4.3 (minimal supported
  version);
- minor fixes.

v5:
- some patch reorg;
- netlink_yml_parser.py is now together with ynl tools;
- minor fixes.

v4:
- Renamed the YNL parser class;
- some minor patch cleanups and merges;
- added an extra patch to fix a insert_pattern/exclude_pattern logic when
   SPHINXDIRS is used.

v3:
- Two series got merged altogether:
  - https://lore.kernel.org/linux-doc/cover.1749723671.git.mchehab+huawei@kernel.org/T/#t
  - https://lore.kernel.org/linux-doc/cover.1749735022.git.mchehab+huawei@kernel.org

- Added an extra patch to update MAINTAINERS to point to YNL library
- Added a (somewhat unrelated) patch that remove warnings check when
  running "make cleandocs".



Mauro Carvalho Chehab (15):
  docs: conf.py: properly handle include and exclude patterns
  docs: Makefile: disable check rules on make cleandocs
  docs: netlink: netlink-raw.rst: use :ref: instead of :doc:
  tools: ynl_gen_rst.py: Split library from command line tool
  docs: netlink: index.rst: add a netlink index file
  tools: ynl_gen_rst.py: cleanup coding style
  docs: sphinx: add a parser for yaml files for Netlink specs
  docs: use parser_yaml extension to handle Netlink specs
  docs: uapi: netlink: update netlink specs link
  tools: ynl_gen_rst.py: drop support for generating index files
  docs: netlink: remove obsolete .gitignore from unused directory
  MAINTAINERS: add netlink_yml_parser.py to linux-doc
  tools: netlink_yml_parser.py: add line numbers to parsed data
  docs: parser_yaml.py: add support for line numbers from the parser
  docs: sphinx: add a file with the requirements for lowest version

 Documentation/Makefile                        |  19 +-
 Documentation/conf.py                         |  87 +++-
 Documentation/doc-guide/sphinx.rst            |  15 +
 Documentation/netlink/specs/index.rst         |  13 +
 Documentation/networking/index.rst            |   2 +-
 .../networking/netlink_spec/.gitignore        |   1 -
 .../networking/netlink_spec/readme.txt        |   4 -
 Documentation/sphinx/min_requirements.txt     |   8 +
 Documentation/sphinx/parser_yaml.py           |  84 ++++
 Documentation/userspace-api/netlink/index.rst |   2 +-
 .../userspace-api/netlink/netlink-raw.rst     |   6 +-
 Documentation/userspace-api/netlink/specs.rst |   2 +-
 MAINTAINERS                                   |   1 +
 tools/net/ynl/pyynl/lib/__init__.py           |   2 +
 tools/net/ynl/pyynl/lib/doc_generator.py      | 398 ++++++++++++++++++
 tools/net/ynl/pyynl/ynl_gen_rst.py            | 384 +----------------
 16 files changed, 614 insertions(+), 414 deletions(-)
 create mode 100644 Documentation/netlink/specs/index.rst
 delete mode 100644 Documentation/networking/netlink_spec/.gitignore
 delete mode 100644 Documentation/networking/netlink_spec/readme.txt
 create mode 100644 Documentation/sphinx/min_requirements.txt
 create mode 100755 Documentation/sphinx/parser_yaml.py
 create mode 100644 tools/net/ynl/pyynl/lib/doc_generator.py

-- 
2.49.0

Re: [PATCH v6 00/15] Don't generate netlink .rst files inside $(srctree)

Posted by Akira Yokosawa 3 months, 3 weeks ago

Hi Mauro,

On 2025/06/18 20:46, Mauro Carvalho Chehab wrote:
> As discussed at:
>    https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/
> 
> changeset f061c9f7d058 ("Documentation: Document each netlink family")
> added a logic which generates *.rst files inside $(srctree). This is bad
> when O=<BUILDDIR> is used.
> 
> A recent change renamed the yaml files used by Netlink, revealing a bad
> side effect: as "make cleandocs" don't clean the produced files and symbols
> appear duplicated for people that don't build the kernel from scratch.
> 
> This series adds an yaml parser extension and uses an index file with glob for
> *. We opted to write such extension in a way that no actual yaml conversion
> code is inside it. This makes it flexible enough to handle other types of yaml
> files in the future. The actual yaml conversion logic were placed at 
> netlink_yml_parser.py. 
> 
> As requested by YNL maintainers, this version has netlink_yml_parser.py
> inside tools/net/ynl/pyynl/ directory. I don't like mixing libraries with
> binaries, nor to have Python libraries spread all over the Kernel. IMO,
> the best is to put all of them on a common place (scripts/lib, python/lib,
> lib/python, ...) but, as this can be solved later, for now let's keep it this
> way.
> 
> ---
> 
> v6:
> - YNL doc parser is now at tools/net/ynl/pyynl/lib/doc_generator.py;
> - two patches got merged;
> - added instructions to test docs with Sphinx 3.4.3 (minimal supported
>   version);
> - minor fixes.

Quick tests against Sphinx 3.4.3 using container images based on
debian:bullseye and almalinux:9, both of which have 3.4.3 as their distro
packages, emits a *bunch* of warnings like the following:

/<srcdir>/Documentation/netlink/specs/conntrack.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
/<srcdir>/Documentation/netlink/specs/devlink.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
/<srcdir>/Documentation/netlink/specs/dpll.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
/<srcdir>/Documentation/netlink/specs/ethtool.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
/<srcdir>/Documentation/netlink/specs/fou.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
[...]

I suspect there should be a minimal required minimal version of PyYAML.

"pip freeze" based on almalinux:9 says:

    PyYAML==5.4.1

"pip freeze" based on debian:bullseye says:

    PyYAML==5.3.1

What is the minimal required version here?

And if users of those old distros need to manually upgrade PyYAML,
why don't you suggest them to upgrade Sphinx as well?

        Thanks, Akira

>
> v5:
> - some patch reorg;
> - netlink_yml_parser.py is now together with ynl tools;
> - minor fixes.
[...]

Re: [PATCH v6 00/15] Don't generate netlink .rst files inside $(srctree)

Posted by Mauro Carvalho Chehab 3 months, 3 weeks ago

Em Thu, 19 Jun 2025 00:46:15 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi Mauro,
> 
> On 2025/06/18 20:46, Mauro Carvalho Chehab wrote:
> > As discussed at:
> >    https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/
> > 
> > changeset f061c9f7d058 ("Documentation: Document each netlink family")
> > added a logic which generates *.rst files inside $(srctree). This is bad
> > when O=<BUILDDIR> is used.
> > 
> > A recent change renamed the yaml files used by Netlink, revealing a bad
> > side effect: as "make cleandocs" don't clean the produced files and symbols
> > appear duplicated for people that don't build the kernel from scratch.
> > 
> > This series adds an yaml parser extension and uses an index file with glob for
> > *. We opted to write such extension in a way that no actual yaml conversion
> > code is inside it. This makes it flexible enough to handle other types of yaml
> > files in the future. The actual yaml conversion logic were placed at 
> > netlink_yml_parser.py. 
> > 
> > As requested by YNL maintainers, this version has netlink_yml_parser.py
> > inside tools/net/ynl/pyynl/ directory. I don't like mixing libraries with
> > binaries, nor to have Python libraries spread all over the Kernel. IMO,
> > the best is to put all of them on a common place (scripts/lib, python/lib,
> > lib/python, ...) but, as this can be solved later, for now let's keep it this
> > way.
> > 
> > ---
> > 
> > v6:
> > - YNL doc parser is now at tools/net/ynl/pyynl/lib/doc_generator.py;
> > - two patches got merged;
> > - added instructions to test docs with Sphinx 3.4.3 (minimal supported
> >   version);
> > - minor fixes.  
> 
> Quick tests against Sphinx 3.4.3 using container images based on
> debian:bullseye and almalinux:9, both of which have 3.4.3 as their distro
> packages, emits a *bunch* of warnings like the following:
> 
> /<srcdir>/Documentation/netlink/specs/conntrack.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> /<srcdir>/Documentation/netlink/specs/devlink.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> /<srcdir>/Documentation/netlink/specs/dpll.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> /<srcdir>/Documentation/netlink/specs/ethtool.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> /<srcdir>/Documentation/netlink/specs/fou.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> [...]
> 
> I suspect there should be a minimal required minimal version of PyYAML.

Likely yes. From my side, I didn't change anything related to PyYAML, 
except by adding a loader at the latest patch to add line numbers.

The above warnings don't seem related. So, probably this was already
an issue.

Funny enough, I did, on my venv:

	$ pip install PyYAML==5.1
	$ tools/net/ynl/pyynl/ynl_gen_rst.py -i Documentation/netlink/specs/dpll.yaml -o Documentation/output/netlink/specs/dpll.rst -v
	...
	$ make clean; make SPHINXDIRS="netlink/specs" htmldocs
	...

but didn't get any issue (I have a later version installed outside
venv - not sure it it will do the right thing).

That's what I have at venv:

----------------------------- ---------
Package                       Version
----------------------------- ---------
alabaster                     0.7.13
babel                         2.17.0
certifi                       2025.6.15
charset-normalizer            3.4.2
docutils                      0.17.1
idna                          3.10
imagesize                     1.4.1
Jinja2                        2.8.1
MarkupSafe                    1.1.1
packaging                     25.0
pip                           25.1.1
Pygments                      2.19.1
PyYAML                        5.1
requests                      2.32.4
setuptools                    80.1.0
snowballstemmer               3.0.1
Sphinx                        3.4.3
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
urllib3                       2.4.0
----------------------------- ---------

> "pip freeze" based on almalinux:9 says:
> 
>     PyYAML==5.4.1
> 
> "pip freeze" based on debian:bullseye says:
> 
>     PyYAML==5.3.1
> 
> What is the minimal required version here?

Breno, what's the minimal version? Please update requirements.txt
to ensure that, and modify ./scripts/sphinx-pre-install to
check for it.

> 
> And if users of those old distros need to manually upgrade PyYAML,
> why don't you suggest them to upgrade Sphinx as well?

The criteria we used to define minimal version for python/sphinx
was having them released at the end of 2020/beginning 2021. So,
up to ~4 years old. We also double-checked latest LTS versions
from major distros.

With that, PyYAML 5.4.1 met the ~4 years old, and so 5.3.1 and
5.1.

funny enough:

	$ git grep tab_width Documentation/netlink/

doesn't return anything. Yet, tab_width is used by sphinx
extensions. The in-kernel ones do it the right way using
get:

	tab_width = self.options.get('tab-width',
				     self.state.document.settings.tab_width)

But perhaps some other extension you might have installed on your
environment has issues, or maybe Documentation/sphinx/parser_yaml.py
need to expand tabs with certain versions of docutils.

Please compare the versions that you're using on your test
environment with the ones I used here.

Regards,
Mauro

Re: [PATCH v6 00/15] Don't generate netlink .rst files inside $(srctree)

Posted by Akira Yokosawa 3 months, 3 weeks ago

On Wed, 18 Jun 2025 18:20:32 +0200, Mauro Carvalho Chehab wrote:
> Em Thu, 19 Jun 2025 00:46:15 +0900
> Akira Yokosawa <akiyks@gmail.com> escreveu:
> 
>> Quick tests against Sphinx 3.4.3 using container images based on
>> debian:bullseye and almalinux:9, both of which have 3.4.3 as their distro
>> packages, emits a *bunch* of warnings like the following:
>>
>> /<srcdir>/Documentation/netlink/specs/conntrack.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
>> /<srcdir>/Documentation/netlink/specs/devlink.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
>> /<srcdir>/Documentation/netlink/specs/dpll.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
>> /<srcdir>/Documentation/netlink/specs/ethtool.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
>> /<srcdir>/Documentation/netlink/specs/fou.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
>> [...]
>>
>> I suspect there should be a minimal required minimal version of PyYAML.
> 
> Likely yes. From my side, I didn't change anything related to PyYAML, 
> except by adding a loader at the latest patch to add line numbers.
> 
> The above warnings don't seem related. So, probably this was already
> an issue.
> 
> Funny enough, I did, on my venv:
> 
> 	$ pip install PyYAML==5.1
> 	$ tools/net/ynl/pyynl/ynl_gen_rst.py -i Documentation/netlink/specs/dpll.yaml -o Documentation/output/netlink/specs/dpll.rst -v
> 	...
> 	$ make clean; make SPHINXDIRS="netlink/specs" htmldocs
> 	...
> 
> but didn't get any issue (I have a later version installed outside
> venv - not sure it it will do the right thing).
> 
> That's what I have at venv:
> 
> ----------------------------- ---------
> Package                       Version
> ----------------------------- ---------
> alabaster                     0.7.13
> babel                         2.17.0
> certifi                       2025.6.15
> charset-normalizer            3.4.2
> docutils                      0.17.1
> idna                          3.10
> imagesize                     1.4.1
> Jinja2                        2.8.1
> MarkupSafe                    1.1.1
> packaging                     25.0
> pip                           25.1.1
> Pygments                      2.19.1
> PyYAML                        5.1
> requests                      2.32.4
> setuptools                    80.1.0
> snowballstemmer               3.0.1
> Sphinx                        3.4.3
> sphinxcontrib-applehelp       1.0.4
> sphinxcontrib-devhelp         1.0.2
> sphinxcontrib-htmlhelp        2.0.1
> sphinxcontrib-jsmath          1.0.1
> sphinxcontrib-qthelp          1.0.3
> sphinxcontrib-serializinghtml 1.1.5
> urllib3                       2.4.0
> ----------------------------- ---------
> 
[...]

> Please compare the versions that you're using on your test
> environment with the ones I used here.

It looks to me like the minimal required version of docutils is 0.17.1
for PyYAML integration.  Both almalinux:9 and debian:11 have 0.16.

Sphinx 4.3.2 of Ubuntu 22.04 comes with docutils 0.17.1, and it is
free of the warnings from PyYAML.

        Thanks, Akira

Re: [PATCH v6 00/15] Don't generate netlink .rst files inside $(srctree)

Posted by Mauro Carvalho Chehab 3 months, 3 weeks ago

Em Thu, 19 Jun 2025 10:34:59 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> On Wed, 18 Jun 2025 18:20:32 +0200, Mauro Carvalho Chehab wrote:
> > Em Thu, 19 Jun 2025 00:46:15 +0900
> > Akira Yokosawa <akiyks@gmail.com> escreveu:
> >   
> >> Quick tests against Sphinx 3.4.3 using container images based on
> >> debian:bullseye and almalinux:9, both of which have 3.4.3 as their distro
> >> packages, emits a *bunch* of warnings like the following:
> >>
> >> /<srcdir>/Documentation/netlink/specs/conntrack.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> >> /<srcdir>/Documentation/netlink/specs/devlink.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> >> /<srcdir>/Documentation/netlink/specs/dpll.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> >> /<srcdir>/Documentation/netlink/specs/ethtool.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> >> /<srcdir>/Documentation/netlink/specs/fou.yaml:: WARNING: YAML parsing error: AttributeError("'Values' object has no attribute 'tab_width'")
> >> [...]
> >>
> >> I suspect there should be a minimal required minimal version of PyYAML.  
> > 
> > Likely yes. From my side, I didn't change anything related to PyYAML, 
> > except by adding a loader at the latest patch to add line numbers.
> > 
> > The above warnings don't seem related. So, probably this was already
> > an issue.
> > 
> > Funny enough, I did, on my venv:
> > 
> > 	$ pip install PyYAML==5.1
> > 	$ tools/net/ynl/pyynl/ynl_gen_rst.py -i Documentation/netlink/specs/dpll.yaml -o Documentation/output/netlink/specs/dpll.rst -v
> > 	...
> > 	$ make clean; make SPHINXDIRS="netlink/specs" htmldocs
> > 	...
> > 
> > but didn't get any issue (I have a later version installed outside
> > venv - not sure it it will do the right thing).
> > 
> > That's what I have at venv:
> > 
> > ----------------------------- ---------
> > Package                       Version
> > ----------------------------- ---------
> > alabaster                     0.7.13
> > babel                         2.17.0
> > certifi                       2025.6.15
> > charset-normalizer            3.4.2
> > docutils                      0.17.1
> > idna                          3.10
> > imagesize                     1.4.1
> > Jinja2                        2.8.1
> > MarkupSafe                    1.1.1
> > packaging                     25.0
> > pip                           25.1.1
> > Pygments                      2.19.1
> > PyYAML                        5.1
> > requests                      2.32.4
> > setuptools                    80.1.0
> > snowballstemmer               3.0.1
> > Sphinx                        3.4.3
> > sphinxcontrib-applehelp       1.0.4
> > sphinxcontrib-devhelp         1.0.2
> > sphinxcontrib-htmlhelp        2.0.1
> > sphinxcontrib-jsmath          1.0.1
> > sphinxcontrib-qthelp          1.0.3
> > sphinxcontrib-serializinghtml 1.1.5
> > urllib3                       2.4.0
> > ----------------------------- ---------
> >   
> [...]
> 
> > Please compare the versions that you're using on your test
> > environment with the ones I used here.  
> 
> It looks to me like the minimal required version of docutils is 0.17.1
> for PyYAML integration.  Both almalinux:9 and debian:11 have 0.16.
> 
> Sphinx 4.3.2 of Ubuntu 22.04 comes with docutils 0.17.1, and it is
> free of the warnings from PyYAML.

Yes, it seems so. As I commented on my past e-mail, I think we need
a validation logic that will warn if versions are incompatible.
Using the experimental checks you and me did, and checking the minimal
version on Sphinx release notes (*), it seems to be that a good start
point is this:

            ========  ============  ============
            Sphinx    Min Docutils  Max Docutils
            Version   Version       Version
            --------  ------------  ------------
            < 4.0.0   0.17.1        0.17.1
            < 6.0.0   0.17.1        0.18.1
            < 7.0.0   0.18.0        0.18.1
            >= 7.0.0  0.20.0        0.21.2
            ========  ============  ============

Eventually, we may need to blacklist or whitelist other
combinations, but this would require a lot of time.

(*) I asked a LLM AI to check Sphinx release notes and docutils
    versions at the time Sphinx versions were released to aid
    creating such table. I also added your feedback about
    docutils 0.19 and your and my tests with docutils < 0.17.1.

Thanks,
Mauro