Hi Jon,

It is impressive how a single patch became a series with 25 ones ;-)

This version was rebased on next-20260127, which is after docs-mw
got merged with my doc patches, solving some merge conflicts.

It should now apply cleanly on the top of upstream and it will address
the kernel-doc issues with context analysis patches that are already
at linux-next.

On this version, I opted to do something I always wanted when
porting from the venerable kernel-doc perl logic: properly handle
macros without writing really complex regular expressions.

After those changes, the replacement logic becomes a lot more
easier to write, as we can have a cleaner logic to handle
function calls:

	function_xforms = [
	    (KernRe(r"^static +"), ""),
	    (KernRe(r"^extern +"), ""),
	...
	    (NestedMatch(r"__cond_acquires\s*\("), ""),
	    (NestedMatch(r"__cond_releases\s*\("), ""),
	...
	}

It can even be used to replace comma-separated arguments like here:

	struct_xforms = [
	...
	    (NestedMatch(r'\bstruct_group\s*\('), r'\2'),
	    (NestedMatch(r'\bstruct_group_attr\s*\('), r'\3'),
	    (NestedMatch(r'\bstruct_group_tagged\s*\('), r'struct \1 { \3 } \2;'),
	    (NestedMatch(r'\b__struct_group\s*\('), r'\4'),
	...
	]

As both KernRe and NestedMatch have sub methods, they can be placed
altogether at the same transforms tables.

The regex syntax for them are quite simple: they should end with a
delimiter (usually open parenthesis, but brackets and square
brackets are also supported).

The first 15 patches on this series were co-developed with Randy,
with came up with the original patch. I ended helping identifying
kernel-doc issues and doing some changes to make the parser more
reliable.

Then, I added 10 other patches to improve NestedMatch.

The final two patches on it fixes an issue when parsing
include/net/page_pool/types.h. Right now, it doesn't parse
well struct page_pool_params, as the second struct_group_tagged()
has a /* private */ on it, causing the end parenthesis to be
commented out. After the series, the documentation for it
should be ok.

-

Even NestedMatch being more complex than KernRe, on my machine,
parsing all files is 5% faster than before, because we're not
parsing anymore macro definitions.

Ah, due to the complexity of NestedMatch, I opted to write
some unit tests to verify that the logic there is correct.
We can use it to add other border cases.

Using it is as easy as running:

	$ tools/unittests/nested_match.py

(I opted to create a separate directory for it, as this
is not really documentation)

Mauro Carvalho Chehab (23):
  docs: kdoc_re: add support for groups()
  docs: kdoc_re: don't go past the end of a line
  docs: kdoc_parser: move var transformers to the beginning
  docs: kdoc_parser: don't mangle with function defines
  docs: kdoc_parser: add functions support for NestedMatch
  docs: kdoc_parser: use NestedMatch to handle __attribute__ on
    functions
  docs: kdoc_parser: fix variable regexes to work with size_t
  docs: kdoc_parser: fix the default_value logic for variables
  docs: kdoc_parser: add some debug for variable parsing
  docs: kdoc_parser: don't exclude defaults from prototype
  docs: kdoc_parser: fix parser to support multi-word types
  docs: kdoc_parser: add support for LIST_HEAD
  docs: kdoc_re: properly handle strings and escape chars on it
  docs: kdoc_re: better show KernRe() at documentation
  docs: kdoc_re: don't recompile NextMatch regex every time
  docs: kdoc_re: Change NestedMath args replacement to \0
  docs: kdoc_re: make NextedMatch use KernRe
  tools: kdoc_re: add support on NestedMatch for argument replacement
  tools: python: add helpers to run unit tests
  unittests: add tests for NestedMatch class
  tools/lib/python/unittest_helper.py
  docs: kdoc_parser: better handle struct_group macros
  docs: kdoc_re: fix a parse bug on struct page_pool_params

Randy Dunlap (2):
  docs: kdoc_parser: ignore context analysis and lock attributes
  kdoc_parser: handle struct member macro VIRTIO_DECLARE_FEATURES(name)

 tools/lib/python/kdoc/kdoc_parser.py | 109 +++--
 tools/lib/python/kdoc/kdoc_re.py     | 154 +++++--
 tools/lib/python/unittest_helper.py  | 279 +++++++++++++
 tools/unittests/nested_match.py      | 589 +++++++++++++++++++++++++++
 4 files changed, 1063 insertions(+), 68 deletions(-)
 create mode 100755 tools/lib/python/unittest_helper.py
 create mode 100755 tools/unittests/nested_match.py

-- 
2.52.0

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,
>
> It is impressive how a single patch became a series with 25 ones ;-)

*sigh*

I will try to have a good look at these shortly.  It seems pretty clear
that this isn't 7.0 material at this point, though.

One thing that jumped at me:

> Ah, due to the complexity of NestedMatch, I opted to write
> some unit tests to verify that the logic there is correct.
> We can use it to add other border cases.
>
> Using it is as easy as running:
>
> 	$ tools/unittests/nested_match.py
>
> (I opted to create a separate directory for it, as this
> is not really documentation)

Do we really need another unit-testing setup in the kernel?  I can't say
I'm familiar enough with kunit to say whether it would work for
non-kernel code; have you looked and verified that it isn't suitable?

Thanks,

jon

On 1/28/2026 9:27 AM, Jonathan Corbet wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
>> Hi Jon,
>>
>> It is impressive how a single patch became a series with 25 ones ;-)
> 
> *sigh*
> 

Splitting things up helped me understand all the changes at least :)

> I will try to have a good look at these shortly.  It seems pretty clear
> that this isn't 7.0 material at this point, though.
> 
> One thing that jumped at me:
> 
>> Ah, due to the complexity of NestedMatch, I opted to write
>> some unit tests to verify that the logic there is correct.
>> We can use it to add other border cases.
>>
>> Using it is as easy as running:
>>
>> 	$ tools/unittests/nested_match.py
>>
>> (I opted to create a separate directory for it, as this
>> is not really documentation)
> 
> Do we really need another unit-testing setup in the kernel?  I can't say
> I'm familiar enough with kunit to say whether it would work for
> non-kernel code; have you looked and verified that it isn't suitable?
> 

I'm not sure kunit would be suitable here, since its meant for running 
kernel code and does a lot of stuff to make that possible. It might be 
able to be extended, but.. this is python code. Why *shouldn't* we use 
one of the python unit test frameworks for it?

We have other python code in tree. Does any of that code have unit tests?

I agree that it doesn't make sense to build new bespoke unit tests 
different or unique per each python module, so if we want to adopt 
python unit tests we should try to pick something that works for the 
python tools in the kernel.

Perhaps finding a way to integrate this with kunit so that you can use 
"kunit run" and get python tests executed as well would make sense? 
But.. then again this isn't kernel code so I'm not sure it makes sense 
to conflate the tests with kernel unit tests.

> Thanks,
> 
> jon
>

On Wed, 28 Jan 2026 10:15:51 -0800
Jacob Keller <jacob.e.keller@intel.com> wrote:

> On 1/28/2026 9:27 AM, Jonathan Corbet wrote:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >   
> >> Hi Jon,
> >>
> >> It is impressive how a single patch became a series with 25 ones ;-)  
> > 
> > *sigh*
> >   
> 
> Splitting things up helped me understand all the changes at least :)
> 
> > I will try to have a good look at these shortly.  It seems pretty clear
> > that this isn't 7.0 material at this point, though.
> > 
> > One thing that jumped at me:
> >   
> >> Ah, due to the complexity of NestedMatch, I opted to write
> >> some unit tests to verify that the logic there is correct.
> >> We can use it to add other border cases.
> >>
> >> Using it is as easy as running:
> >>
> >> 	$ tools/unittests/nested_match.py
> >>
> >> (I opted to create a separate directory for it, as this
> >> is not really documentation)  
> > 
> > Do we really need another unit-testing setup in the kernel?  I can't say
> > I'm familiar enough with kunit to say whether it would work for
> > non-kernel code; have you looked and verified that it isn't suitable?
> >   
> 
> I'm not sure kunit would be suitable here, since its meant for running 
> kernel code and does a lot of stuff to make that possible. It might be 
> able to be extended, but.. this is python code. Why *shouldn't* we use 
> one of the python unit test frameworks for it?

This is not using kunit. It is using standard "import unittest" from
Python internal lib.

> We have other python code in tree. Does any of that code have unit tests?

Good question. On a quick grep, it sounds so:

	$ git grep "import unittest" tools scripts
	scripts/rust_is_available_test.py:import unittest
	tools/crypto/ccp/test_dbc.py:import unittest
	tools/perf/pmu-events/metric_test.py:import unittest
	tools/testing/kunit/kunit_tool_test.py:import unittest
	tools/testing/selftests/bpf/test_bpftool.py:import unittest
	tools/testing/selftests/tpm2/tpm2.py:import unittest
	tools/testing/selftests/tpm2/tpm2_tests.py:import unittest

> I agree that it doesn't make sense to build new bespoke unit tests 
> different or unique per each python module, so if we want to adopt 
> python unit tests we should try to pick something that works for the 
> python tools in the kernel.
> 
> Perhaps finding a way to integrate this with kunit so that you can use 
> "kunit run" and get python tests executed as well would make sense? 
> But.. then again this isn't kernel code so I'm not sure it makes sense 
> to conflate the tests with kernel unit tests.

It shouldn't be hard to add it there - or to have a separate script
to run python unittests.

Assuming that we place all unittests at the same directory
(tools/unittests), the enclosed path will run all of them
altogether:

	$ tools/unittests/run.py
	Ran 35 tests in 0.001s

	OK
	nested_match:
	    TestStructGroup:
	        test_struct_group_01:            OK
	        test_struct_group_02:            OK
	        test_struct_group_03:            OK
	        test_struct_group_04:            OK
	        test_struct_group_05:            OK
	        test_struct_group_06:            OK
	        test_struct_group_07:            OK
	        test_struct_group_08:            OK
	        test_struct_group_09:            OK
	        test_struct_group_10:            OK
	        test_struct_group_11:            OK
        	test_struct_group_12:            OK
	        test_struct_group_13:            OK
	        test_struct_group_14:            OK
	        test_struct_group_15:            OK
	        test_struct_group_16:            OK
	        test_struct_group_17:            OK
	        test_struct_group_18:            OK
	        test_struct_group_19:            OK
	        test_struct_group_sub:           OK
	    TestSubMacros:
	        test_acquires_multiple:          OK
        	test_acquires_nested_paren:      OK
	        test_acquires_simple:            OK
        	test_mixed_macros:               OK
	        test_must_hold:                  OK
        	test_must_hold_shared:           OK
	        test_no_false_positive:          OK
	        test_no_macro_remains:           OK
	    TestSubReplacement:
	        test_sub_count_parameter:        OK
	        test_sub_mixed_placeholders:     OK
	        test_sub_multiple_placeholders:  OK
	        test_sub_no_placeholder:         OK
	        test_sub_single_placeholder:     OK
	        test_sub_with_capture:           OK
	        test_sub_zero_placeholder:       OK


	Ran 35 tests

And the helper will also provide an argparse to allow filtering
tests, change verbosity and filtering them with a regex:

	$ tools/unittests/run.py --help
	usage: run.py [-h] [-v] [-f] [-k KEYWORD]

	Test runner with regex filtering

	options:
	  -h, --help            show this help message and exit
	  -v, --verbose
	  -f, --failfast
	  -k, --keyword KEYWORD
	                        Regex pattern to filter test methods

That's said, some integration with kunit can be interesting
to have it producing a KTAP output if needed by some CI.

---

[PATCH] [RFC] Run all tests from tools/unittests

This small example runs all unittests from tools/unittests with:

    $ tools/unittests/run.py

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

diff --git a/tools/lib/python/unittest_helper.py b/tools/lib/python/unittest_helper.py
index 3cf1075b1de4..af16acc3af17 100755
--- a/tools/lib/python/unittest_helper.py
+++ b/tools/lib/python/unittest_helper.py
@@ -213,7 +213,7 @@ class TestUnits:
                             help="Regex pattern to filter test methods")
         return parser
 
-    def run(self, caller_file, parser=None, args=None, env=None):
+    def run(self, caller_file, suite=None, parser=None, args=None, env=None):
         """Execute all tests from the unity test file"""
         if not args:
             if not parser:
@@ -232,9 +232,10 @@ class TestUnits:
             unittest.TextTestRunner(verbosity=verbose).run = lambda suite: suite
 
         # Load ONLY tests from the calling file
-        loader = unittest.TestLoader()
-        suite = loader.discover(start_dir=os.path.dirname(caller_file),
-                                pattern=os.path.basename(caller_file))
+        if not suite:
+            loader = unittest.TestLoader()
+            suite = loader.discover(start_dir=os.path.dirname(caller_file),
+                                    pattern=os.path.basename(caller_file))
 
         # Flatten the suite for environment injection
         tests_to_inject = flatten_suite(suite)
diff --git a/tools/unittests/run.py b/tools/unittests/run.py
new file mode 100755
index 000000000000..2a5a754219de
--- /dev/null
+++ b/tools/unittests/run.py
@@ -0,0 +1,17 @@
+#!/bin/env python3
+import os
+import unittest
+import sys
+
+TOOLS_DIR=os.path.join(os.path.dirname(os.path.realpath(__file__)), "..")
+sys.path.insert(0, TOOLS_DIR)
+
+from lib.python.unittest_helper import TestUnits
+
+if __name__ == "__main__":
+    loader = unittest.TestLoader()
+
+    suite = loader.discover(start_dir=os.path.join(TOOLS_DIR, "unittests"),
+                            pattern="*.py")
+
+    TestUnits().run("", suite=suite)

Thanks,
Mauro

On 1/28/2026 2:00 PM, Mauro Carvalho Chehab wrote:
> On Wed, 28 Jan 2026 10:15:51 -0800
> Jacob Keller <jacob.e.keller@intel.com> wrote:
>> On 1/28/2026 9:27 AM, Jonathan Corbet wrote:
>>> Do we really need another unit-testing setup in the kernel?  I can't say
>>> I'm familiar enough with kunit to say whether it would work for
>>> non-kernel code; have you looked and verified that it isn't suitable?
>>>    
>>
>> I'm not sure kunit would be suitable here, since its meant for running
>> kernel code and does a lot of stuff to make that possible. It might be
>> able to be extended, but.. this is python code. Why *shouldn't* we use
>> one of the python unit test frameworks for it?
> 
> This is not using kunit. It is using standard "import unittest" from
> Python internal lib.
> 

Right. I think it makes perfect sense to use unittest for python files. 
That was the point of my reply above :D

>> We have other python code in tree. Does any of that code have unit tests?
> 
> Good question. On a quick grep, it sounds so:
> 
> 	$ git grep "import unittest" tools scripts
> 	scripts/rust_is_available_test.py:import unittest
> 	tools/crypto/ccp/test_dbc.py:import unittest
> 	tools/perf/pmu-events/metric_test.py:import unittest
> 	tools/testing/kunit/kunit_tool_test.py:import unittest
> 	tools/testing/selftests/bpf/test_bpftool.py:import unittest
> 	tools/testing/selftests/tpm2/tpm2.py:import unittest
> 	tools/testing/selftests/tpm2/tpm2_tests.py:import unittest
> 
>> I agree that it doesn't make sense to build new bespoke unit tests
>> different or unique per each python module, so if we want to adopt
>> python unit tests we should try to pick something that works for the
>> python tools in the kernel.
>>
>> Perhaps finding a way to integrate this with kunit so that you can use
>> "kunit run" and get python tests executed as well would make sense?
>> But.. then again this isn't kernel code so I'm not sure it makes sense
>> to conflate the tests with kernel unit tests.
> 
> It shouldn't be hard to add it there - or to have a separate script
> to run python unittests.
> 

Right. Some way to have all unit tests run is nice so that its easy to 
hook up into various CI processes. Sounds like you have a solid idea on 
that.

> That's said, some integration with kunit can be interesting
> to have it producing a KTAP output if needed by some CI.
> 
That could also be interesting, as it would make it easier for other 
tooling to work with all the tests.

Personally I am not sure how useful that would be vs just making use of 
the unittest stuff provided as-is with python.

On Wed, 28 Jan 2026 14:08:50 -0800
Jacob Keller <jacob.e.keller@intel.com> wrote:

> On 1/28/2026 2:00 PM, Mauro Carvalho Chehab wrote:
> > On Wed, 28 Jan 2026 10:15:51 -0800
> > Jacob Keller <jacob.e.keller@intel.com> wrote:  
> >> On 1/28/2026 9:27 AM, Jonathan Corbet wrote:  
> >>> Do we really need another unit-testing setup in the kernel?  I can't say
> >>> I'm familiar enough with kunit to say whether it would work for
> >>> non-kernel code; have you looked and verified that it isn't suitable?
> >>>      
> >>
> >> I'm not sure kunit would be suitable here, since its meant for running
> >> kernel code and does a lot of stuff to make that possible. It might be
> >> able to be extended, but.. this is python code. Why *shouldn't* we use
> >> one of the python unit test frameworks for it?  
> > 
> > This is not using kunit. It is using standard "import unittest" from
> > Python internal lib.
> >   
> 
> Right. I think it makes perfect sense to use unittest for python files. 
> That was the point of my reply above :D
> 
> >> We have other python code in tree. Does any of that code have unit tests?  
> > 
> > Good question. On a quick grep, it sounds so:
> > 
> > 	$ git grep "import unittest" tools scripts
> > 	scripts/rust_is_available_test.py:import unittest
> > 	tools/crypto/ccp/test_dbc.py:import unittest
> > 	tools/perf/pmu-events/metric_test.py:import unittest
> > 	tools/testing/kunit/kunit_tool_test.py:import unittest
> > 	tools/testing/selftests/bpf/test_bpftool.py:import unittest
> > 	tools/testing/selftests/tpm2/tpm2.py:import unittest
> > 	tools/testing/selftests/tpm2/tpm2_tests.py:import unittest
> >   
> >> I agree that it doesn't make sense to build new bespoke unit tests
> >> different or unique per each python module, so if we want to adopt
> >> python unit tests we should try to pick something that works for the
> >> python tools in the kernel.
> >>
> >> Perhaps finding a way to integrate this with kunit so that you can use
> >> "kunit run" and get python tests executed as well would make sense?
> >> But.. then again this isn't kernel code so I'm not sure it makes sense
> >> to conflate the tests with kernel unit tests.  
> > 
> > It shouldn't be hard to add it there - or to have a separate script
> > to run python unittests.
> >   
> 
> Right. Some way to have all unit tests run is nice so that its easy to 
> hook up into various CI processes. Sounds like you have a solid idea on 
> that.
> 
> > That's said, some integration with kunit can be interesting
> > to have it producing a KTAP output if needed by some CI.
> >   
> That could also be interesting, as it would make it easier for other 
> tooling to work with all the tests.
> 
> Personally I am not sure how useful that would be vs just making use of 
> the unittest stuff provided as-is with python.

I'd say that, for now, we don't need a KTAP output, but as things go
more complex and more parts of the tools get unittests added, it 
could make sense to add it.

Adding proper support for it shouldn't be hard with the definitions
inside unittest_helper. All we need to do would be to write a new
inherited class from unittest.TestResult, placing there a 
printResults() method that would generate the KTAP format. We may
add a new "--ktap" argparse argument that, if enabled, it would
use the newer class instead of the Summary class.

Thanks,
Mauro