As discussed at:
   https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/

changeset f061c9f7d058 ("Documentation: Document each netlink family")
added a logic which generates *.rst files inside $(srctree). This is bad when
O=<BUILDDIR> is used.

A recent change renamed the yaml files used by Netlink, revealing a bad
side effect: as "make cleandocs" don't clean the produced files, symbols 
appear duplicated for people that don't build the kernel from scratch.

There are some possible solutions for that. The simplest one, which is what
this series address, places the build files inside Documentation/output. 
The changes to do that are simple enough, but has one drawback,
as it requires a (simple) template file for every netlink family file from
netlink/specs. The template is simple enough:

        .. kernel-include:: $BUILDDIR/networking/netlink_spec/<family>.rst

Part of the issue is that sphinx-build only produces html files for sources
inside the source tree (Documentation/). 

To address that, add an yaml parser extension to Sphinx.

It should be noticed that this version has one drawback: it increases the
documentation build time. I suspect that the culprit is inside Sphinx
glob logic and the way it handles exclude_patterns. What happens is that
sphinx/project.py uses glob, which, on my own experiences, it is slow
(due to that, I ended implementing my own glob logic for kernel-doc).

On the plus side, the extension is flexible enough to handle other types
of yaml files, as the actual yaml conversion logic is outside the extension.

With this version, there's no need to add any template file per netlink/spec
file. Yet, the Documentation/netlink/spec.index.rst require updates as
spec files are added/renamed/removed. The already-existing script can
handle it automatically by running:

            tools/net/ynl/pyynl/ynl_gen_rst.py -x  -v -o Documentation/netlink/specs/index.rst

---

v2:
- Use a Sphinx extension to handle netlink files.

v1:
- Statically add template files to as networking/netlink_spec/<family>.rst

Mauro Carvalho Chehab (12):
  tools: ynl_gen_rst.py: create a top-level reference
  docs: netlink: netlink-raw.rst: use :ref: instead of :doc:
  docs: netlink: don't ignore generated rst files
  tools: ynl_gen_rst.py: make the index parser more generic
  tools: ynl_gen_rst.py: Split library from command line tool
  scripts: lib: netlink_yml_parser.py: use classes
  tools: ynl_gen_rst.py: do some coding style cleanups
  scripts: netlink_yml_parser.py: improve index.rst generation
  docs: sphinx: add a parser template for yaml files
  docs: sphinx: parser_yaml.py: add Netlink specs parser
  docs: use parser_yaml extension to handle Netlink specs
  docs: conf.py: don't handle yaml files outside Netlink specs

 .pylintrc                                     |   2 +-
 Documentation/Makefile                        |  17 -
 Documentation/conf.py                         |  17 +-
 Documentation/netlink/specs/index.rst         |  38 ++
 Documentation/networking/index.rst            |   2 +-
 .../networking/netlink_spec/.gitignore        |   1 -
 .../networking/netlink_spec/readme.txt        |   4 -
 Documentation/sphinx/parser_yaml.py           |  80 ++++
 .../userspace-api/netlink/netlink-raw.rst     |   6 +-
 scripts/lib/netlink_yml_parser.py             | 394 ++++++++++++++++++
 tools/net/ynl/pyynl/ynl_gen_rst.py            | 378 +----------------
 11 files changed, 544 insertions(+), 395 deletions(-)
 create mode 100644 Documentation/netlink/specs/index.rst
 delete mode 100644 Documentation/networking/netlink_spec/.gitignore
 delete mode 100644 Documentation/networking/netlink_spec/readme.txt
 create mode 100755 Documentation/sphinx/parser_yaml.py
 create mode 100755 scripts/lib/netlink_yml_parser.py

-- 
2.49.0

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> As discussed at:
>    https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/
>
> changeset f061c9f7d058 ("Documentation: Document each netlink family")
> added a logic which generates *.rst files inside $(srctree). This is bad when
> O=<BUILDDIR> is used.
>
> A recent change renamed the yaml files used by Netlink, revealing a bad
> side effect: as "make cleandocs" don't clean the produced files, symbols 
> appear duplicated for people that don't build the kernel from scratch.
>
> There are some possible solutions for that. The simplest one, which is what
> this series address, places the build files inside Documentation/output. 
> The changes to do that are simple enough, but has one drawback,
> as it requires a (simple) template file for every netlink family file from
> netlink/specs. The template is simple enough:
>
>         .. kernel-include:: $BUILDDIR/networking/netlink_spec/<family>.rst

I think we could skip describing this since it was an approach that has
now been dropped.

> Part of the issue is that sphinx-build only produces html files for sources
> inside the source tree (Documentation/). 
>
> To address that, add an yaml parser extension to Sphinx.
>
> It should be noticed that this version has one drawback: it increases the
> documentation build time. I suspect that the culprit is inside Sphinx
> glob logic and the way it handles exclude_patterns. What happens is that
> sphinx/project.py uses glob, which, on my own experiences, it is slow
> (due to that, I ended implementing my own glob logic for kernel-doc).
>
> On the plus side, the extension is flexible enough to handle other types
> of yaml files, as the actual yaml conversion logic is outside the extension.

I don't think the extension would handle anything other than the Netlink
yaml specs, and I don't think that should be a goal of this patchset.

> With this version, there's no need to add any template file per netlink/spec
> file. Yet, the Documentation/netlink/spec.index.rst require updates as
> spec files are added/renamed/removed. The already-existing script can
> handle it automatically by running:
>
>             tools/net/ynl/pyynl/ynl_gen_rst.py -x  -v -o Documentation/netlink/specs/index.rst

I think this can be avoided by using the toctree glob directive in the
index, like this:

=============================
Netlink Family Specifications
=============================

.. toctree::
   :maxdepth: 1
   :glob:

   *

This would let you have a static index file.

> ---
>
> v2:
> - Use a Sphinx extension to handle netlink files.
>
> v1:
> - Statically add template files to as networking/netlink_spec/<family>.rst
>
> Mauro Carvalho Chehab (12):
>   tools: ynl_gen_rst.py: create a top-level reference
>   docs: netlink: netlink-raw.rst: use :ref: instead of :doc:

I suggest combining the first 2 patches.

>   docs: netlink: don't ignore generated rst files

Maybe leave this patch to the end and change the description to be a
cleanup of the remants of the old approach.

Further comments on specific commits

>   tools: ynl_gen_rst.py: make the index parser more generic
>   tools: ynl_gen_rst.py: Split library from command line tool
>   scripts: lib: netlink_yml_parser.py: use classes
>   tools: ynl_gen_rst.py: do some coding style cleanups
>   scripts: netlink_yml_parser.py: improve index.rst generation
>   docs: sphinx: add a parser template for yaml files
>   docs: sphinx: parser_yaml.py: add Netlink specs parser

Please combine these 2 patches. The template patch just introduces noise
into the series and makes it harder to review.

>   docs: use parser_yaml extension to handle Netlink specs
>   docs: conf.py: don't handle yaml files outside Netlink specs
>
>  .pylintrc                                     |   2 +-
>  Documentation/Makefile                        |  17 -
>  Documentation/conf.py                         |  17 +-
>  Documentation/netlink/specs/index.rst         |  38 ++
>  Documentation/networking/index.rst            |   2 +-
>  .../networking/netlink_spec/.gitignore        |   1 -
>  .../networking/netlink_spec/readme.txt        |   4 -
>  Documentation/sphinx/parser_yaml.py           |  80 ++++
>  .../userspace-api/netlink/netlink-raw.rst     |   6 +-
>  scripts/lib/netlink_yml_parser.py             | 394 ++++++++++++++++++
>  tools/net/ynl/pyynl/ynl_gen_rst.py            | 378 +----------------
>  11 files changed, 544 insertions(+), 395 deletions(-)
>  create mode 100644 Documentation/netlink/specs/index.rst
>  delete mode 100644 Documentation/networking/netlink_spec/.gitignore
>  delete mode 100644 Documentation/networking/netlink_spec/readme.txt
>  create mode 100755 Documentation/sphinx/parser_yaml.py
>  create mode 100755 scripts/lib/netlink_yml_parser.py

Em Fri, 13 Jun 2025 12:05:56 +0100
Donald Hunter <donald.hunter@gmail.com> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > As discussed at:
> >    https://lore.kernel.org/all/20250610101331.62ba466f@foz.lan/
> >
> > changeset f061c9f7d058 ("Documentation: Document each netlink family")
> > added a logic which generates *.rst files inside $(srctree). This is bad when
> > O=<BUILDDIR> is used.
> >
> > A recent change renamed the yaml files used by Netlink, revealing a bad
> > side effect: as "make cleandocs" don't clean the produced files, symbols 
> > appear duplicated for people that don't build the kernel from scratch.
> >
> > There are some possible solutions for that. The simplest one, which is what
> > this series address, places the build files inside Documentation/output. 
> > The changes to do that are simple enough, but has one drawback,
> > as it requires a (simple) template file for every netlink family file from
> > netlink/specs. The template is simple enough:
> >
> >         .. kernel-include:: $BUILDDIR/networking/netlink_spec/<family>.rst  
> 
> I think we could skip describing this since it was an approach that has
> now been dropped.

Ok. Will drop on next versions.

> 
> > Part of the issue is that sphinx-build only produces html files for sources
> > inside the source tree (Documentation/). 
> >
> > To address that, add an yaml parser extension to Sphinx.
> >
> > It should be noticed that this version has one drawback: it increases the
> > documentation build time. I suspect that the culprit is inside Sphinx
> > glob logic and the way it handles exclude_patterns. What happens is that
> > sphinx/project.py uses glob, which, on my own experiences, it is slow
> > (due to that, I ended implementing my own glob logic for kernel-doc).
> >
> > On the plus side, the extension is flexible enough to handle other types
> > of yaml files, as the actual yaml conversion logic is outside the extension.  
> 
> I don't think the extension would handle anything other than the Netlink
> yaml specs, and I don't think that should be a goal of this patchset.

Not necessarily. We do have already DT yaml files (although there's
a separate process to handle those outside the tree). Nothing prevents
we end having more. See, the way Sphinx parser works is that it will cover
all files with *.yaml extension no matter where it is located within the
tree. We may end needing to use it for something else as well (*).

(*) at the last Media Summit, we did have some discussions about using
    either yaml or rst for sensor documentation.

> > With this version, there's no need to add any template file per netlink/spec
> > file. Yet, the Documentation/netlink/spec.index.rst require updates as
> > spec files are added/renamed/removed. The already-existing script can
> > handle it automatically by running:
> >
> >             tools/net/ynl/pyynl/ynl_gen_rst.py -x  -v -o Documentation/netlink/specs/index.rst  
> 
> I think this can be avoided by using the toctree glob directive in the
> index, like this:
> 
> =============================
> Netlink Family Specifications
> =============================
> 
> .. toctree::
>    :maxdepth: 1
>    :glob:
> 
>    *
> 
> This would let you have a static index file.

Didn't know about such option. If it works with the parser, it sounds good 
enough.

> 
> > ---
> >
> > v2:
> > - Use a Sphinx extension to handle netlink files.
> >
> > v1:
> > - Statically add template files to as networking/netlink_spec/<family>.rst
> >
> > Mauro Carvalho Chehab (12):
> >   tools: ynl_gen_rst.py: create a top-level reference
> >   docs: netlink: netlink-raw.rst: use :ref: instead of :doc:  
> 
> I suggest combining the first 2 patches.
> 
> >   docs: netlink: don't ignore generated rst files  
> 
> Maybe leave this patch to the end and change the description to be a
> cleanup of the remants of the old approach.

Ok for me, but I usually prefer keeping one patch per logical change.
In this case, one patch adding support at the tool; the other one
improving docs to benefit from the new feature.

> Further comments on specific commits
> 
> >   tools: ynl_gen_rst.py: make the index parser more generic
> >   tools: ynl_gen_rst.py: Split library from command line tool
> >   scripts: lib: netlink_yml_parser.py: use classes
> >   tools: ynl_gen_rst.py: do some coding style cleanups
> >   scripts: netlink_yml_parser.py: improve index.rst generation
> >   docs: sphinx: add a parser template for yaml files
> >   docs: sphinx: parser_yaml.py: add Netlink specs parser  
> 
> Please combine these 2 patches. The template patch just introduces noise
> into the series and makes it harder to review.

Ok.

> >   docs: use parser_yaml extension to handle Netlink specs
> >   docs: conf.py: don't handle yaml files outside Netlink specs
> >
> >  .pylintrc                                     |   2 +-
> >  Documentation/Makefile                        |  17 -
> >  Documentation/conf.py                         |  17 +-
> >  Documentation/netlink/specs/index.rst         |  38 ++
> >  Documentation/networking/index.rst            |   2 +-
> >  .../networking/netlink_spec/.gitignore        |   1 -
> >  .../networking/netlink_spec/readme.txt        |   4 -
> >  Documentation/sphinx/parser_yaml.py           |  80 ++++
> >  .../userspace-api/netlink/netlink-raw.rst     |   6 +-
> >  scripts/lib/netlink_yml_parser.py             | 394 ++++++++++++++++++
> >  tools/net/ynl/pyynl/ynl_gen_rst.py            | 378 +----------------
> >  11 files changed, 544 insertions(+), 395 deletions(-)
> >  create mode 100644 Documentation/netlink/specs/index.rst
> >  delete mode 100644 Documentation/networking/netlink_spec/.gitignore
> >  delete mode 100644 Documentation/networking/netlink_spec/readme.txt
> >  create mode 100755 Documentation/sphinx/parser_yaml.py
> >  create mode 100755 scripts/lib/netlink_yml_parser.py  

Thanks,
Mauro

On Fri, 13 Jun 2025 at 13:14, Mauro Carvalho Chehab
<mchehab+huawei@kernel.org> wrote:
> >
> > >   docs: netlink: don't ignore generated rst files
> >
> > Maybe leave this patch to the end and change the description to be a
> > cleanup of the remants of the old approach.
>
> Ok for me, but I usually prefer keeping one patch per logical change.
> In this case, one patch adding support at the tool; the other one
> improving docs to benefit from the new feature.

My point is that "don't ignore generated rst files" is not a step on
the way from the old method towards the new method. The new method
does not involve generating any .rst files so the patch is something
more like "remove obsolete .gitignore from unused directory".