The parsing code from tools/net/ynl/pyynl/ynl_gen_rst.py was moved
to scripts/lib/netlink_yml_parser.py. Its maintainership
is done by Netlink maintainers. Yet, as it is used by Sphinx
build system, add it also to linux-doc maintainers, as changes
there might affect documentation builds. So, linux-docs ML
should ideally be C/C on changes to it.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
MAINTAINERS | 2 ++
1 file changed, 2 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index a92290fffa16..2c0b13e5d8fc 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7202,6 +7202,7 @@ F: scripts/get_abi.py
F: scripts/kernel-doc*
F: scripts/lib/abi/*
F: scripts/lib/kdoc/*
+F: scripts/lib/netlink_yml_parser.py
F: scripts/sphinx-pre-install
X: Documentation/ABI/
X: Documentation/admin-guide/media/
@@ -27314,6 +27315,7 @@ M: Jakub Kicinski <kuba@kernel.org>
F: Documentation/netlink/
F: Documentation/userspace-api/netlink/intro-specs.rst
F: Documentation/userspace-api/netlink/specs.rst
+F: scripts/lib/netlink_yml_parser.py
F: tools/net/ynl/
YEALINK PHONE DRIVER
--
2.49.0On Sat, 14 Jun 2025 at 09:56, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote: > > The parsing code from tools/net/ynl/pyynl/ynl_gen_rst.py was moved > to scripts/lib/netlink_yml_parser.py. Its maintainership > is done by Netlink maintainers. Yet, as it is used by Sphinx > build system, add it also to linux-doc maintainers, as changes > there might affect documentation builds. So, linux-docs ML > should ideally be C/C on changes to it. This patch can be dropped from the series when you move the library code to tools/net/ynl/pyynl/lib. > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org> > --- > MAINTAINERS | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/MAINTAINERS b/MAINTAINERS > index a92290fffa16..2c0b13e5d8fc 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > @@ -7202,6 +7202,7 @@ F: scripts/get_abi.py > F: scripts/kernel-doc* > F: scripts/lib/abi/* > F: scripts/lib/kdoc/* > +F: scripts/lib/netlink_yml_parser.py > F: scripts/sphinx-pre-install > X: Documentation/ABI/ > X: Documentation/admin-guide/media/ > @@ -27314,6 +27315,7 @@ M: Jakub Kicinski <kuba@kernel.org> > F: Documentation/netlink/ > F: Documentation/userspace-api/netlink/intro-specs.rst > F: Documentation/userspace-api/netlink/specs.rst > +F: scripts/lib/netlink_yml_parser.py > F: tools/net/ynl/ > > YEALINK PHONE DRIVER > -- > 2.49.0 >
Hi Donald, Jon, Em Sat, 14 Jun 2025 15:22:16 +0100 Donald Hunter <donald.hunter@gmail.com> escreveu: > On Sat, 14 Jun 2025 at 09:56, Mauro Carvalho Chehab > <mchehab+huawei@kernel.org> wrote: > > > > The parsing code from tools/net/ynl/pyynl/ynl_gen_rst.py was moved > > to scripts/lib/netlink_yml_parser.py. Its maintainership > > is done by Netlink maintainers. Yet, as it is used by Sphinx > > build system, add it also to linux-doc maintainers, as changes > > there might affect documentation builds. So, linux-docs ML > > should ideally be C/C on changes to it. > > This patch can be dropped from the series when you move the library > code to tools/net/ynl/pyynl/lib. > > > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org> > > --- > > MAINTAINERS | 2 ++ > > 1 file changed, 2 insertions(+) > > > > diff --git a/MAINTAINERS b/MAINTAINERS > > index a92290fffa16..2c0b13e5d8fc 100644 > > --- a/MAINTAINERS > > +++ b/MAINTAINERS > > @@ -7202,6 +7202,7 @@ F: scripts/get_abi.py > > F: scripts/kernel-doc* > > F: scripts/lib/abi/* > > F: scripts/lib/kdoc/* > > +F: scripts/lib/netlink_yml_parser.py > > F: scripts/sphinx-pre-install > > X: Documentation/ABI/ > > X: Documentation/admin-guide/media/ Adding an entry that would c/c to linux-doc for the parser is important, as problems there will affect documentation build, no matter where it is located. Perhaps one option would be to create a separate MAINTAINERS entry for it, like: YAML NETLINK (YNL) DOC GENERATOR M: Donald Hunter <donald.hunter@gmail.com> M: Jakub Kicinski <kuba@kernel.org> F: <python_lib_location>/netlink_yml_parser.py L: linux-doc@vger.kernel.org to ensure that changes to it would be C/C to linux-doc. > > @@ -27314,6 +27315,7 @@ M: Jakub Kicinski <kuba@kernel.org> > > F: Documentation/netlink/ > > F: Documentation/userspace-api/netlink/intro-specs.rst > > F: Documentation/userspace-api/netlink/specs.rst > > +F: scripts/lib/netlink_yml_parser.py > > F: tools/net/ynl/ With regards to the location itself, as I said earlier, it is up to Jon and you to decide. My preference is to have all Python libraries at the entire Kernel inside scripts/lib (or at some other common location), no matter where the caller Python command or in-kernel Sphinx extensions are located. There is also slight advantage on placing them at the same location: if we end adding parsers for other subsystems at parse_html.py, having all of them at the same directory means we don't need to do something like: lib_paths = [ "tools/net/ynl/pyynl/lib", "foo", "bar", ... ] for d in lib_paths: sys.path.insert(0, os.path.join(srctree, "scripts/lib")) from netlink_yml_parser import YnlDocGenerator # pylint: disable=C0413 from foo_yml_parser import FooYamlDocGenerator # pylint: disable=C0413 from bar_yml_parser import BarYamlDocGenerator # pylint: disable=C0413 ... Thanks, Mauro
On Sat, 14 Jun 2025 17:32:35 +0200 Mauro Carvalho Chehab wrote: > > > @@ -27314,6 +27315,7 @@ M: Jakub Kicinski <kuba@kernel.org> > > > F: Documentation/netlink/ > > > F: Documentation/userspace-api/netlink/intro-specs.rst > > > F: Documentation/userspace-api/netlink/specs.rst > > > +F: scripts/lib/netlink_yml_parser.py > > > F: tools/net/ynl/ > > With regards to the location itself, as I said earlier, it is up to > Jon and you to decide. > > My preference is to have all Python libraries at the entire Kernel > inside scripts/lib (or at some other common location), no matter where > the caller Python command or in-kernel Sphinx extensions are located. I understand that from the PoV of ease of maintenance of the docs. Is it fair to say there is a trade off here between ease of maintenance for docs maintainers and encouraging people to integrate with kernel docs in novel ways?
Em Sat, 14 Jun 2025 10:37:00 -0700
Jakub Kicinski <kuba@kernel.org> escreveu:
> On Sat, 14 Jun 2025 17:32:35 +0200 Mauro Carvalho Chehab wrote:
> > > > @@ -27314,6 +27315,7 @@ M: Jakub Kicinski <kuba@kernel.org>
> > > > F: Documentation/netlink/
> > > > F: Documentation/userspace-api/netlink/intro-specs.rst
> > > > F: Documentation/userspace-api/netlink/specs.rst
> > > > +F: scripts/lib/netlink_yml_parser.py
> > > > F: tools/net/ynl/
> >
> > With regards to the location itself, as I said earlier, it is up to
> > Jon and you to decide.
> >
> > My preference is to have all Python libraries at the entire Kernel
> > inside scripts/lib (or at some other common location), no matter where
> > the caller Python command or in-kernel Sphinx extensions are located.
>
> I understand that from the PoV of ease of maintenance of the docs.
> Is it fair to say there is a trade off here between ease of maintenance
> for docs maintainers and encouraging people to integrate with kernel
> docs in novel ways?
Placing elsewhere won't make much difference from doc maintainers and
developers.
I'm more interested on having a single place where python libraries
could be placed. Eventually, some classes might be re-used in the future
by multiple scripts and subsystems, when it makes sense, just like we do
already with Kernel's kAPIs. This also helps when checking what is the
Python's minimal version that are required by the Kernel when updating
it at:
Documentation/process/changes.rst
And writing patches documenting it like:
d2b239099cf0 ("docs: changes: update Sphinx minimal version to 3.4.3")
5e25b972a22b ("docs: changes: update Python minimal version")
Properly setting the minimal Python version is important specially to
check if the minimal version set at changes.rst is compatible with
the Makefile build targets:
$ pip install --user vermin
...
$ vermin -v scripts/lib/
Detecting python files..
Analyzing 9 files using 24 processes..
!2, 3.6 /new_devel/v4l/docs/scripts/lib/abi/abi_parser.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/abi/abi_regex.py
~2, ~3 /new_devel/v4l/docs/scripts/lib/abi/helpers.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/abi/system_symbols.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/kdoc/kdoc_files.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/kdoc/kdoc_output.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/kdoc/kdoc_parser.py
2.3, 3.0 /new_devel/v4l/docs/scripts/lib/kdoc/kdoc_re.py
!2, 3.6 /new_devel/v4l/docs/scripts/lib/netlink_yml_parser.py
Tips:
- You're using potentially backported modules: argparse, typing
If so, try using the following for better results: --backport argparse --backport typing
- Since '# novm' or '# novermin' weren't used, a speedup can be achieved using: --no-parse-comments
(disable using: --no-tips)
Minimum required versions: 3.6
Incompatible versions: 2
Thanks,
Mauro
On Sat, 14 Jun 2025 20:56:09 +0200 Mauro Carvalho Chehab wrote: > > I understand that from the PoV of ease of maintenance of the docs. > > Is it fair to say there is a trade off here between ease of maintenance > > for docs maintainers and encouraging people to integrate with kernel > > docs in novel ways? > > Placing elsewhere won't make much difference from doc maintainers and > developers. I must be missing your point. Clearly it makes a difference to Donald, who is a maintainer of the docs in question. > I'm more interested on having a single place where python libraries > could be placed. Me too, especially for selftests. But it's not clear to me that scripts/ is the right location. I thought purely user space code should live in tools/ and bulk of YNL is for user space. > Eventually, some classes might be re-used in the future > by multiple scripts and subsystems, when it makes sense, just like we do > already with Kernel's kAPIs. This also helps when checking what is the > Python's minimal version that are required by the Kernel when updating > it at: I think this is exactly the same point Donald is making, but from YNL perspective. The hope is to share more code between the ReST generator, the existing C generator and Python library. The later two are already based on a shared spec model.
Jakub Kicinski <kuba@kernel.org> writes: > On Sat, 14 Jun 2025 20:56:09 +0200 Mauro Carvalho Chehab wrote: >> I'm more interested on having a single place where python libraries >> could be placed. > > Me too, especially for selftests. But it's not clear to me that > scripts/ is the right location. I thought purely user space code > should live in tools/ and bulk of YNL is for user space. I've been out wandering the woods and canyons with no connectivity for a bit, so missed this whole discussion, sorry. Mauro and I had talked about the proper home for Python libraries when he reworked kernel-doc; we ended up with them under scripts/, which I didn't find entirely pleasing. If you were to ask me today, I'd say they should be under lib/python, but tomorrow I might say something else... In truth, I don't think it matters much, but I *do* think we should have a single location from which to import kernel-specific Python code. Spreading it throughout the tree just isn't going to lead to joy. Thanks, jon
Em Thu, 19 Jun 2025 14:06:58 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:
> Jakub Kicinski <kuba@kernel.org> writes:
>
> > On Sat, 14 Jun 2025 20:56:09 +0200 Mauro Carvalho Chehab wrote:
>
> >> I'm more interested on having a single place where python libraries
> >> could be placed.
> >
> > Me too, especially for selftests. But it's not clear to me that
> > scripts/ is the right location. I thought purely user space code
> > should live in tools/ and bulk of YNL is for user space.
>
> I've been out wandering the woods and canyons with no connectivity for a
> bit, so missed this whole discussion, sorry.
Sounds fun!
> Mauro and I had talked about the proper home for Python libraries when
> he reworked kernel-doc; we ended up with them under scripts/, which I
> didn't find entirely pleasing. If you were to ask me today, I'd say
> they should be under lib/python, but tomorrow I might say something
> else...
Yeah, I guess you proposed lib/python before... I could be wrong though.
Anyway, at least for me lib/python sounds a better alternative than
scripts. I won't mind tools/lib/python or some other place.
> In truth, I don't think it matters much, but I *do* think we should have
> a single location from which to import kernel-specific Python code.
> Spreading it throughout the tree just isn't going to lead to joy.
We're aligned with that regards: IMO, we need a single store within
the Kernel for classes that might be shared.
As I commented on one of PRs, maybe the series could be merged
with Donald proposed (tools/net/ynl/pyynl/lib/doc_generator.py),
while we're still discussing. So, let's focus on get it reviewed
and merged without needing to wait for a broader discussion
about its permanent location.
We can later shift the code once we reach an agreement.
-
To start the discussions about a permanent location, in the specific
case of YNL, we currently have there:
$ tree -d tools/net/ynl/ -I __pycache__
tools/net/ynl/
├── generated
├── lib
├── pyynl
│ └── lib
└── samples
where pyynl have executables and pyynl the python libraries.
what I would suggest is to move what it is under "pyynl/lib"
to "{prefix}/ynl", where "{prefix}" can be "lib/python",
"tools/lib/python", "scripts/lib" or whatever other location
we reach an agreement.
For now, I placed the latest version of my doc patch series
under:
https://github.com/mchehab/linux/tree/netlink_v8
to have a central place to have them on one of my scratch
trees.
I sent today for review to linux-doc ML an initial patch series
with some non-YAML related patches. I have another set of
patches after it, which I'm planning to send on Monday. At the
end, there are the YAML parser submission.
Regards,
Mauro
Em Sat, 14 Jun 2025 12:46:49 -0700
Jakub Kicinski <kuba@kernel.org> escreveu:
> On Sat, 14 Jun 2025 20:56:09 +0200 Mauro Carvalho Chehab wrote:
> > > I understand that from the PoV of ease of maintenance of the docs.
> > > Is it fair to say there is a trade off here between ease of maintenance
> > > for docs maintainers and encouraging people to integrate with kernel
> > > docs in novel ways?
> >
> > Placing elsewhere won't make much difference from doc maintainers and
> > developers.
>
> I must be missing your point. Clearly it makes a difference to Donald,
> who is a maintainer of the docs in question.
Heh, I was just saying that I missed your point ;-)
See, you said that "there is a trade off here between ease of maintenance
for docs maintainers and encouraging people to integrate with kernel
docs in novel ways".
I can't see how being easy/hard to maintain or even "integrate with
kernel docs in novel ways" would be affected by the script location.
Whatever it is located, there should be MAINTAINERS entries that would
point to YAML and network maintainers maintainers:
$ ./scripts/get_maintainer.pl tools/net/ynl/pyynl/ynl_gen_rst.py --nogit --nogit-blame --nogit-fallback
Donald Hunter <donald.hunter@gmail.com> (maintainer:YAML NETLINK (YNL))
Jakub Kicinski <kuba@kernel.org> (maintainer:YAML NETLINK (YNL))
"David S. Miller" <davem@davemloft.net> (maintainer:NETWORKING [GENERAL])
Eric Dumazet <edumazet@google.com> (maintainer:NETWORKING [GENERAL])
Paolo Abeni <pabeni@redhat.com> (maintainer:NETWORKING [GENERAL])
Simon Horman <horms@kernel.org> (reviewer:NETWORKING [GENERAL])
netdev@vger.kernel.org (open list:NETWORKING [GENERAL])
linux-kernel@vger.kernel.org (open list)
YAML NETLINK (YNL) status: Unknown
(do they all apply to YNL doc parser?)
Plus having doc ML/Maintainer on it:
Jonathan Corbet <corbet@lwn.net> (maintainer:DOCUMENTATION)
linux-doc@vger.kernel.org (open list:DOCUMENTATION)
So, at least the file called by the Sphinx class should be at the
linux-doc entry at the maintainers' file.
The rationale is that linux-doc and Jon should be c/c, just in case some
change there might end causing build issues using a version of the toolchain
that is officially supported, as documented at
Documentation/process/changes.rst, e.g. currently whatever it there is
expected to be compatible with:
====================== =============== ========================================
Program Minimal version Command to check the version
====================== =============== ========================================
...
Sphinx\ [#f1]_ 3.4.3 sphinx-build --version
...
Python (optional) 3.9.x python3 --version
...
This is independent if the YNL classes are either at scripts/lib
or at tools/net/ynl/pyynl/lib.
>
> > I'm more interested on having a single place where python libraries
> > could be placed.
>
> Me too, especially for selftests. But it's not clear to me that
> scripts/ is the right location. I thought purely user space code
> should live in tools/ and bulk of YNL is for user space.
Several scripts under scripts/ are meant to run outside build
time. One clear example is:
$ ./scripts/get_abi.py undefined
That basically checks if the userspace sysfs API is properly
documented, by reading the macine's sysfs node and comparing
with the uAPI documentation. Such tool can also used to check if
the ABI documentation Python classes are working as expected.
So, it is a mix of kernel build time and userspace.
There are also pure userspace tools like those two:
./scripts/get_dvb_firmware
./scripts/extract_xc3028.pl
Both extract firmware files from some other OS and write as a
Linux firmware file to be stored under /lib/firmware. They are
userspace-only tools.
-
From my side, I don't care where Python classes would be placed,
but I prefer having them on a single common place. It could be:
/scripts/lib
/tools/lib
/python/lib
eventually with their own sub-directories on it, like what we have
today:
${some_prefix}/kdoc
${some_prefix}/abi
In the case of netlink, it could be:
${some_prefix}/netlink
Yet, IMO, we should not have a different location for userspace
and non-userspace, as it is very hard to draw the borders on several
cases, like the ABI toolset.
> > Eventually, some classes might be re-used in the future
> > by multiple scripts and subsystems, when it makes sense, just like we do
> > already with Kernel's kAPIs. This also helps when checking what is the
> > Python's minimal version that are required by the Kernel when updating
> > it at:
>
> I think this is exactly the same point Donald is making, but from YNL
> perspective. The hope is to share more code between the ReST generator,
> the existing C generator and Python library. The later two are already
> based on a shared spec model.
That makes perfect sense to me. Yet, this doesn't preventing having
a:
${some_prefix}/ynl
directory where you would place Netlink YNL parsing, where the prefix
would be either:
- /scripts/lib
- /tools/lib
- /python/lib
- something else
It may even use some common classes under:
${some_prefix}/${some_common_prefix}
---
Now, seeing your comments, maybe the main point is wheather it is OK to
add userspace libraries to scripts/lib or not. IMO, using "/scripts/lib"
is OK, no matter if the script is kernel-build related or "pure userspace",
but if there are no consensus, we could migrate what we have to
"python/lib" or to some other place.
Thanks,
Mauro
© 2016 - 2025 Red Hat, Inc.