Currently testing of userspace and in-kernel API use two different
frameworks. kselftests for the userspace ones and Kunit for the
in-kernel ones. Besides their different scopes, both have different
strengths and limitations:

Kunit:
* Tests are normal kernel code.
* They use the regular kernel toolchain.
* They can be packaged and distributed as modules conveniently.

Kselftests:
* Tests are normal userspace code
* They need a userspace toolchain.
  A kernel cross toolchain is likely not enough.
* A fair amout of userland is required to run the tests,
  which means a full distro or handcrafted rootfs.
* There is no way to conveniently package and run kselftests with a
  given kernel image.
* The kselftests makefiles are not as powerful as regular kbuild.
  For example they are missing proper header dependency tracking or more
  complex compiler option modifications.

Therefore kunit is much easier to run against different kernel
configurations and architectures.
This series aims to combine kselftests and kunit, avoiding both their
limitations. It works by compiling the userspace kselftests as part of
the regular kernel build, embedding them into the kunit kernel or module
and executing them from there. If the kernel toolchain is not fit to
produce userspace because of a missing libc, the kernel's own nolibc can
be used instead.
The structured TAP output from the kselftest is integrated into the
kunit KTAP output transparently, the kunit parser can parse the combined
logs together.

Further room for improvements:
* Call each test in its completely dedicated namespace
* Handle additional test files besides the test executable through
  archives. CPIO, cramfs, etc.
* Compatibility with kselftest_harness.h (in progress)
* Expose the blobs in debugfs
* Provide some convience wrappers around compat userprogs
* Figure out a migration path/coexistence solution for
  kunit UAPI and tools/testing/selftests/

Output from the kunit example testcase, note the output of
"example_uapi_tests".

$ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
...
Running tests with:
$ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[11:53:53] ================== example (10 subtests) ===================
[11:53:53] [PASSED] example_simple_test
[11:53:53] [SKIPPED] example_skip_test
[11:53:53] [SKIPPED] example_mark_skipped_test
[11:53:53] [PASSED] example_all_expect_macros_test
[11:53:53] [PASSED] example_static_stub_test
[11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
[11:53:53] [PASSED] example_priv_test
[11:53:53] =================== example_params_test  ===================
[11:53:53] [SKIPPED] example value 3
[11:53:53] [PASSED] example value 2
[11:53:53] [PASSED] example value 1
[11:53:53] [SKIPPED] example value 0
[11:53:53] =============== [PASSED] example_params_test ===============
[11:53:53] [PASSED] example_slow_test
[11:53:53] ======================= (4 subtests) =======================
[11:53:53] [PASSED] procfs
[11:53:53] [PASSED] userspace test 2
[11:53:53] [SKIPPED] userspace test 3: some reason
[11:53:53] [PASSED] userspace test 4
[11:53:53] ================ [PASSED] example_uapi_test ================
[11:53:53] ===================== [PASSED] example =====================
[11:53:53] ============================================================
[11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
[11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running

Based on v6.16-rc1.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Changes in v5:
- Initialize output variable of kernel_wait()
- Fix .incbin with in-tree builds
- Keep requirement of KTAP tests to have a number which was removed accidentally
- Only synthesize KTAP subtest failure if the outer one is TestStatus.FAILURE
- Use -I instead of -isystem in NOLIBC_USERCFLAGS to populate dependency files
- +To filesystem developers to all patches
- +To Luis Chamberlain for discussions about usage of usermodehelper
  (see patches 6 and 12)
- Link to v4: https://lore.kernel.org/r/20250626-kunit-kselftests-v4-0-48760534fef5@linutronix.de

Changes in v4:
- Move Kconfig.nolibc from tools/ to init/
- Drop generic userprogs nolibc integration
- Drop generic blob framework
- Pick up review tags from David
- Extend new kunit TAP parser tests
- Add MAINTAINERS entry
- Allow CONFIG_KUNIT_UAPI=m
- Split /proc validation into dedicated UAPI test
- Trim recipient list a bit
- Use KUNIT_FAIL_AND_ABORT() over KUNIT_FAIL()
- Link to v3: https://lore.kernel.org/r/20250611-kunit-kselftests-v3-0-55e3d148cbc6@linutronix.de

Changes in v3:
- Reintroduce CONFIG_CC_CAN_LINK_STATIC
- Enable CONFIG_ARCH_HAS_NOLIBC for m68k and SPARC
- Properly handle 'clean' target for userprogs
- Use ramfs over tmpfs to reduce dependencies
- Inherit userprogs byte order and ABI from kernel
- Drop now unnecessary "#ifndef NOLIBC"
- Pick up review tags
- Drop usage of __private in blob.h,
  sparse complains and it is not really necessary
- Fix execution on loongarch when using clang
- Drop userprogs libgcc handling, it was ugly and is not yet necessary
- Link to v2: https://lore.kernel.org/r/20250407-kunit-kselftests-v2-0-454114e287fd@linutronix.de

Changes in v2:
- Rebase onto v6.15-rc1
- Add documentation and kernel docs
- Resolve invalid kconfig breakages
- Drop already applied patch "kbuild: implement CONFIG_HEADERS_INSTALL for Usermode Linux"
- Drop userprogs CONFIG_WERROR integration, it doesn't need to be part of this series
- Replace patch prefix "kconfig" with "kbuild"
- Rename kunit_uapi_run_executable() to kunit_uapi_run_kselftest()
- Generate private, conflict-free symbols in the blob framework
- Handle kselftest exit codes
- Handle SIGABRT
- Forward output also to kunit debugfs log
- Install a fd=0 stdin filedescriptor
- Link to v1: https://lore.kernel.org/r/20250217-kunit-kselftests-v1-0-42b4524c3b0a@linutronix.de

---
Thomas Weißschuh (15):
      kbuild: userprogs: avoid duplication of flags inherited from kernel
      kbuild: userprogs: also inherit byte order and ABI from kernel
      kbuild: doc: add label for userprogs section
      init: re-add CONFIG_CC_CAN_LINK_STATIC
      init: add nolibc build support
      fs,fork,exit: export symbols necessary for KUnit UAPI support
      kunit: tool: Add test for nested test result reporting
      kunit: tool: Don't overwrite test status based on subtest counts
      kunit: tool: Parse skipped tests from kselftest.h
      kunit: Always descend into kunit directory during build
      kunit: qemu_configs: loongarch: Enable LSX/LSAX
      kunit: Introduce UAPI testing framework
      kunit: uapi: Add example for UAPI tests
      kunit: uapi: Introduce preinit executable
      kunit: uapi: Validate usability of /proc

 Documentation/dev-tools/kunit/api/index.rst        |   5 +
 Documentation/dev-tools/kunit/api/uapi.rst         |  14 +
 Documentation/kbuild/makefiles.rst                 |   2 +
 MAINTAINERS                                        |  11 +
 Makefile                                           |   7 +-
 fs/exec.c                                          |   2 +
 fs/file.c                                          |   1 +
 fs/filesystems.c                                   |   2 +
 fs/fs_struct.c                                     |   1 +
 fs/pipe.c                                          |   2 +
 include/kunit/uapi.h                               |  77 ++++++
 init/Kconfig                                       |   7 +
 init/Kconfig.nolibc                                |  15 +
 init/Makefile.nolibc                               |  13 +
 kernel/exit.c                                      |   3 +
 kernel/fork.c                                      |   2 +
 lib/Makefile                                       |   4 -
 lib/kunit/Kconfig                                  |  14 +
 lib/kunit/Makefile                                 |  30 +-
 lib/kunit/kunit-example-test.c                     |  15 +
 lib/kunit/kunit-example-uapi.c                     |  22 ++
 lib/kunit/kunit-test-uapi.c                        |  51 ++++
 lib/kunit/kunit-test.c                             |  23 +-
 lib/kunit/kunit-uapi.c                             | 305 +++++++++++++++++++++
 lib/kunit/uapi-preinit.c                           |  63 +++++
 tools/testing/kunit/kunit_parser.py                |  11 +-
 tools/testing/kunit/kunit_tool_test.py             |  11 +
 tools/testing/kunit/qemu_configs/loongarch.py      |   2 +
 .../test_is_test_passed-failure-nested.log         |  10 +
 .../test_data/test_is_test_passed-kselftest.log    |   3 +-
 30 files changed, 715 insertions(+), 13 deletions(-)
---
base-commit: 9d5898b413d17510b2a41664a42390a2c79f8bf4
change-id: 20241015-kunit-kselftests-56273bc40442

Best regards,
-- 
Thomas Weißschuh <thomas.weissschuh@linutronix.de>

On Thu, Jul 17, 2025 at 10:48:02AM +0200, Thomas Weißschuh wrote:
> Currently testing of userspace and in-kernel API use two different
> frameworks.

Which is kinda expected as one has to run in the kernel to test
in-kernel kernel space APIs, and the other tests externally provided
kernel functionality.

> Therefore kunit is much easier to run against different kernel
> configurations and architectures.

Which is is normal.  unit tests are always easier to run than
integration tests.

> This series aims to combine kselftests and kunit, avoiding both their
> limitations. It works by compiling the userspace kselftests as part of
> the regular kernel build, embedding them into the kunit kernel or module
> and executing them from there.

This is really weird.  "Running userspace code is hard, so we package
it in the kernel".  I had my own fair share of problems with kselftests,
mostly because of the lack of structure and automated way to run them,
but adding them to the kernel (or a module) is overshooting the target
by far.

> If the kernel toolchain is not fit to
> produce userspace because of a missing libc, the kernel's own nolibc can
> be used instead.

Is nolibc enough to run all the selftests?  If so we should just do
it unconditionally, but linking to different libraries by availability
seems a bit problematic.

> The structured TAP output from the kselftest is integrated into the
> kunit KTAP output transparently, the kunit parser can parse the combined
> logs together.

Good idea!

On Thu, Jul 17, 2025 at 03:23:00PM +0200, Christoph Hellwig wrote:
> On Thu, Jul 17, 2025 at 10:48:02AM +0200, Thomas Weißschuh wrote:

(...)

> > This series aims to combine kselftests and kunit, avoiding both their
> > limitations. It works by compiling the userspace kselftests as part of
> > the regular kernel build, embedding them into the kunit kernel or module
> > and executing them from there.

(...)

> I had my own fair share of problems with kselftests,
> mostly because of the lack of structure and automated way to run them,

How did you overcome these issues? Why does everbody need to reinvent the
wheel here? KUnit already exists and provides a lot of structure and tooling.

> but adding them to the kernel (or a module) is overshooting the target
> by far.

That's a subjective statement without any reasoning I can engange with.
I would be happy to do so, but for now I can only say that I disagree.
The patches have been on the testing-related lists for
some time and so far nobody had an issue with this aspect.

> > If the kernel toolchain is not fit to
> > produce userspace because of a missing libc, the kernel's own nolibc can
> > be used instead.
> 
> Is nolibc enough to run all the selftests?

It is not and most probably won't ever be. The maintainers of each testcase
will decide which libc to use. Like it is in tools/testing/selftests/ today.
Some use glibc, some nolibc and some can do both.

> If so we should just do it unconditionally, but linking to different
> libraries by availability seems a bit problematic.

Agreed. But as mentioned above it will be the maintainers decision.

Only the preinit executable will need to support all configurations so needs
the availability check. For the framework selftest it also makes sense to
support as many configurations as possible. For the example test, any
configuration is fine. 

(...)

While having this discussion, can we also work on dealing with the symbol
exports, as discussed before?


Thomas

On Fri, Jul 18, 2025 at 08:22:26AM +0200, Thomas Weißschuh wrote:
> > I had my own fair share of problems with kselftests,
> > mostly because of the lack of structure and automated way to run them,
> 
> How did you overcome these issues? Why does everbody need to reinvent the
> wheel here?

Told people to use everything remotely file system related to use
xfstests instead, and either ignore or suffer from the rest.

> KUnit already exists and provides a lot of structure and tooling.

That's great.  Let's reuse it without having to drive running userspace
programs from kernel code.

> > but adding them to the kernel (or a module) is overshooting the target
> > by far.
> 
> That's a subjective statement without any reasoning I can engange with.

Well, then we're done here if you can't engage.

> I would be happy to do so, but for now I can only say that I disagree.
> The patches have been on the testing-related lists for
> some time and so far nobody had an issue with this aspect.

Has anyone actually chimed in and said "it's great that we bloat the
kernel to run userspace tests", or have people just mostly ignored it
like most things?

> > > If the kernel toolchain is not fit to
> > > produce userspace because of a missing libc, the kernel's own nolibc can
> > > be used instead.
> > 
> > Is nolibc enough to run all the selftests?
> 
> It is not and most probably won't ever be. The maintainers of each testcase
> will decide which libc to use. Like it is in tools/testing/selftests/ today.
> Some use glibc, some nolibc and some can do both.

So why do you want to use it here?  And how is is related to the rest
of the series?

> While having this discussion, can we also work on dealing with the symbol
> exports, as discussed before?

Well, the scope of the entire series makes it pretty clear that this
series as is simply should not go in.

You present running pure userspace tests as the solution to a problem
you don't even explain, or rather just state very highlevel.  Yes,
kselftests suck as most people will agree.  But the answer is not
to add a lot of kernel bloat to treat userspace integration tests
like kernel units tests.  How about you just fix kselftests, preferably
by reusing well known and teststed userland code?

Hi Christoph,

On Mon, Jul 21, 2025 at 09:09:58AM +0200, Christoph Hellwig wrote:
> On Fri, Jul 18, 2025 at 08:22:26AM +0200, Thomas Weißschuh wrote:
> > > I had my own fair share of problems with kselftests,
> > > mostly because of the lack of structure and automated way to run them,
> > 
> > How did you overcome these issues? Why does everbody need to reinvent the
> > wheel here?
> 
> Told people to use everything remotely file system related to use
> xfstests instead, and either ignore or suffer from the rest.

Suffering from the rest is what I am trying to avoid.
(More on that below)

> > KUnit already exists and provides a lot of structure and tooling.
> 
> That's great.  Let's reuse it without having to drive running userspace
> programs from kernel code.

Running in the kernel is the point behind KUnit. It could be done by putting
all the userspace test into a initramfs and run them on boot from there.
But that has other drawbacks:
* The tests can't be run on an existing system.
* All tests need to be loaded into memory together, and not on demand.
* The tests can not be rerun.

> > > but adding them to the kernel (or a module) is overshooting the target
> > > by far.
> > 
> > That's a subjective statement without any reasoning I can engange with.
> 
> Well, then we're done here if you can't engage.

This was a response to one specific statement. Could you be a bit more specific
in your critique? I am not sure what exactly you mean in some cases, making it
hard to respond properly. For example "bloat", it is bloaty
* source code,
* object code for users enabling the new kconfig options,
* object code for other users *not* enabling the new kconfig options?
 
> > I would be happy to do so, but for now I can only say that I disagree.
> > The patches have been on the testing-related lists for
> > some time and so far nobody had an issue with this aspect.
> 
> Has anyone actually chimed in and said "it's great that we bloat the
> kernel to run userspace tests", or have people just mostly ignored it
> like most things?

That specific wording wasn't used. Obviously...
So far nobody had any issues with the overall goal of the series.
There was criticism around implementation details and I have been and will be
working on resolving those.

Some feedback I got:

David [0]: "I've taken quite a liking to it: it'd definitely have made my
life easier more than once."
Benjamin is already playing with it, having built his own testcase [1].
I asked Shuah about it before starting development and she gave a go-ahead.
A collegue of mine is also using it to validate the PREEMPT_RT safety of
various UAPIs by combining KUnit UAPI with a runtime validator [2].

> > > > If the kernel toolchain is not fit to
> > > > produce userspace because of a missing libc, the kernel's own nolibc can
> > > > be used instead.
> > > 
> > > Is nolibc enough to run all the selftests?
> > 
> > It is not and most probably won't ever be. The maintainers of each testcase
> > will decide which libc to use. Like it is in tools/testing/selftests/ today.
> > Some use glibc, some nolibc and some can do both.
> 
> So why do you want to use it here?  And how is is related to the rest
> of the series?

To make it easier to test a wide range of architectures by not requiring a
libc from the toolchain. It also avoids relying on a bunch of out-of-tree
code (glibc) as part of the test. And there are existing kselftests which
use it over glibc for their own reasons.

But using nolibc in test code is not necessary and nobody is forced to do so.

(Maybe a disclaimer that I'm one of the nolibc maintainers is in order)

(...)

> You present running pure userspace tests as the solution to a problem
> you don't even explain, or rather just state very highlevel.

To run kselftests we need the following things:
a) A toolchain which can build userspace executables.
b) Quite a bit of supporting userland, at least glibc, coreutils and bash.
c) A rootfs assembled out of these.
d) An efficient way to incrementally rebuild the test executables and rootfs.
e) A way to put that rootfs into the system under test.
f) A way to configure a kernel which
   * is as small as possible and as fast as possible to build,
   * can run on QEMU or a real machine,
   * can run the functionality under test.
g) A way to select the tests to run in the system under test.
h) A way to communicate back the results.
i) Something to interpret the results.
j) Hook up everything into a CI system.

And for all of this there should be good in-tree tooling.

For a) and b) I am not aware of any toolchain provider or distribution which
provides this for all necessary architectures. And the existing userspace test
frameworks don't even try to address the points a) to e)/f) and let the user
figure it out. This is the case for xfstests and LTP. virtme(-ng) provide most
of it but don't support cross-architecture setups. On the other hand the tree
already contains solutions for most of those points. a) and d) are solved by
kbuild userprogs, e) to j) by KUnit and my new framework plugs b) and c).
Moving to a pure userspace solution would preclude the usage of KUnit as far as
I can see.

This all started when I worked on the generic vDSO data storage patches [3].
I needed to run the existing vDSO selftests against a bunch of architectures,
including some esoteric ones [4]. With my framework, running the vDSO selftests
for any architecture is now trivial and blazingly fast.

Does this make more sense?

> Yes, kselftests suck as most people will agree. But the answer is not
> to add a lot of kernel bloat to treat userspace integration tests
> like kernel units tests.

I fail to understand how this test code is worse than the existing KUnit test
code. This is not meant to test complex scenarios, but single system calls or
specific UAPIs, which may depend on architecture features. For example timers,
signals, vDSO, mm etc.

> How about you just fix kselftests, preferably
> by reusing well known and teststed userland code?

Is "well known and tested userland code" referring to glibc or testing
frameworks? As mentioned above, glibc can be used just fine and the frameworks
I know about are lacking.


Thomas


[0] https://lore.kernel.org/all/CABVgOSn+530YJ3OPNJQncLDQNbd9JEDtZ04Amyyxk57jOVYUyQ@mail.gmail.com/
[1] https://lore.kernel.org/all/20250626195714.2123694-3-benjamin@sipsolutions.net/
[2] https://lore.kernel.org/lkml/cover.1752088709.git.namcao@linutronix.de/
[3] https://lore.kernel.org/lkml/20250204-vdso-store-rng-v3-0-13a4669dfc8c@linutronix.de/
[4] https://lore.kernel.org/lkml/20250724-vdso-sparc64-generic-2-v1-0-e376a3bd24d1@linutronix.de/

On Mon, Aug 04, 2025 at 05:01:35PM +0200, Thomas Weißschuh wrote:
> > That's great.  Let's reuse it without having to drive running userspace
> > programs from kernel code.
> 
> Running in the kernel is the point behind KUnit.

When using kunit as is to unit test kernel functionality - obviously.

When running it to integration test the syscall boundary - not at all.

> It could be done by putting
> all the userspace test into a initramfs and run them on boot from there.
> But that has other drawbacks:
> * The tests can't be run on an existing system.
> * All tests need to be loaded into memory together, and not on demand.
> * The tests can not be rerun.

None of that is true.  While running syscall level tests from an
initramfs could be a nice feature for an automatd kernel CI system,
nothin in this tests should require running from an initramfs.

> This was a response to one specific statement. Could you be a bit more specific
> in your critique? I am not sure what exactly you mean in some cases, making it
> hard to respond properly. For example "bloat", it is bloaty
> * source code,
> * object code for users enabling the new kconfig options,
> * object code for other users *not* enabling the new kconfig options?

You are adding kernel code both at the source and object level to run
userspace tests.  That is very clearly bloat.  Even more so as it adds
functionality and exports that don't fit in with what the kernel already
does for actual kernel functionality.

> > > It is not and most probably won't ever be. The maintainers of each testcase
> > > will decide which libc to use. Like it is in tools/testing/selftests/ today.
> > > Some use glibc, some nolibc and some can do both.
> > 
> > So why do you want to use it here?  And how is is related to the rest
> > of the series?
> 
> To make it easier to test a wide range of architectures by not requiring a
> libc from the toolchain. It also avoids relying on a bunch of out-of-tree
> code (glibc) as part of the test. And there are existing kselftests which
> use it over glibc for their own reasons.
> 
> But using nolibc in test code is not necessary and nobody is forced to do so.
> 
> (Maybe a disclaimer that I'm one of the nolibc maintainers is in order)

Well, why do you even mix it up with this unrelated series then?

> To run kselftests we need the following things:
> a) A toolchain which can build userspace executables.

You'll need that for any userspace program, no matter what test
harness.

> b) Quite a bit of supporting userland, at least glibc, coreutils and bash.

Well, a libc you will need anyway.  Maybe nolibc is good enough for
some tests, but as you already stated not for very many.  The others
are just an artifcat of how you run tests.

> c) A rootfs assembled out of these.
> d) An efficient way to incrementally rebuild the test executables and rootfs.
> e) A way to put that rootfs into the system under test.

You don't need a rootfs.

> And for all of this there should be good in-tree tooling.

Absolutely.

> Moving to a pure userspace solution would preclude the usage of KUnit as far as
> I can see.

You've still failed why using kunit is the goal and not just something
that made your life easier archieving your goal.

> > Yes, kselftests suck as most people will agree. But the answer is not
> > to add a lot of kernel bloat to treat userspace integration tests
> > like kernel units tests.
> 
> I fail to understand how this test code is worse than the existing KUnit test
> code. This is not meant to test complex scenarios, but single system calls or
> specific UAPIs, which may depend on architecture features. For example timers,
> signals, vDSO, mm etc.

So now having another half-assed test framework is a good thing?

> > How about you just fix kselftests, preferably
> > by reusing well known and teststed userland code?
> 
> Is "well known and tested userland code" referring to glibc or testing
> frameworks? As mentioned above, glibc can be used just fine and the frameworks
> I know about are lacking.

Basically any kind of testing framework that has broad use.  It's not
like there aren't userland unit test frameworks if you really want to
use that.

On Thu, Jul 17, 2025 at 03:23:00PM +0200, Christoph Hellwig wrote:
> On Thu, Jul 17, 2025 at 10:48:02AM +0200, Thomas Weißschuh wrote:

> > If the kernel toolchain is not fit to
> > produce userspace because of a missing libc, the kernel's own nolibc can
> > be used instead.

> Is nolibc enough to run all the selftests?  If so we should just do
> it unconditionally, but linking to different libraries by availability
> seems a bit problematic.

There's some that rely on standard userspace libraries for accessing the
functionality they're testing or for things like crypto which would
require a bunch more work.