Hi,

The common kselftest harness provides nice features such as tracking the
number of tests, common pass/fail printouts, and setup/teardown functions.

KVM would like to benefit from the common functionality provided in the
general kselftest harness, but KVM selftests are dependent on a selftest
library.

This selftest library provides common code for setting up virtual machines,
and is designed with assertions within the setup flows to reduce KVM
selftest developer burden of making common assertions.

The common kselftest library assertions (ASSERT_EQ and EXPECT_EQ) are
dependent on access to the _metadata parameter. Any function using the
assertion macros must be parametrized by _metadata.

The KVM selftest library has many nested function calls and it would be
impractical to pass _metadata through all those layers of function calls.

Without _metadata, the KVM selftest library can still reimplement its own
assertion macro (TEST_ASSERT), but it cannot perform teardown on assertion
failure.

Sean suggested using setjmp and longjmp [1] to back to the top level
TEST_F(). I looked at [1] and found myself wishing to use TEST_F() the from
kselftest harness directly. Also, setjmp/longjmp felt like it was
introducing state that could be messed up easily. I also found recent work
that removed setjmp/longjmp from kselftest harness [2].

The kselftests harness is running tests sequentially anyway, and the
function pointers in _metadata wouldn't be changing all that often in most
selftests.

Would maintainers be open to having the kselftest harness expose a pointer
to the metadata globally?

Another option would be to expose the current teardown function pointer
globally instead of the pointer to the entire metadata struct.

kselftest harness macros ASSERT_EQ might also benefit from using the global
pointer instead, which would open the functions up to calls from nested
functions without needing to parametrize those functions with _metadata.

[1] https://lore.kernel.org/all/ZjUwqEXPA5QVItyX@google.com/
[2] https://lore.kernel.org/all/20250610141252-1ee7ae72-dbad-4a80-931c-5b4b14fb07ce@linutronix.de/

Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (4):
      selftests: harness: Move metadata structs to separate header file
      selftests: harness: Set global current_test_metadata for each test run
      KVM: selftests: Do teardown from kselftest harness if kselftest_harness is used
      HACK: Show that the teardown function is called from KVM selftests

 tools/testing/selftests/kselftest_harness.h        | 51 +++------------------
 .../testing/selftests/kselftest_harness_structs.h  | 53 ++++++++++++++++++++++
 tools/testing/selftests/kvm/Makefile.kvm           |  1 +
 .../selftests/kvm/kvm_test_harness_selftest.c      | 34 ++++++++++++++
 tools/testing/selftests/kvm/lib/assert.c           | 16 +++++++
 5 files changed, 110 insertions(+), 45 deletions(-)
---
base-commit: 5d0d3623303775d750e122a2542d1a26c8573d38
change-id: 20260414-selftest-global-metadata-7b1f04a405f3

Best regards,
--
Ackerley Tng <ackerleytng@google.com>

On Tue, Apr 14, 2026 at 11:07:47AM -0700, Ackerley Tng wrote:
> Would maintainers be open to having the kselftest harness expose a pointer
> to the metadata globally?

I would like this, yes. It's been a real pain sometimes having to pass
_metadata down into various deep helper functions.

-- 
Kees Cook

On Tue, Apr 14, 2026, Ackerley Tng wrote:
> Sean suggested using setjmp and longjmp [1] to back to the top level
> TEST_F(). I looked at [1] and found myself wishing to use TEST_F() the from
> kselftest harness directly.

Can you elaborate?  If you have a need for direct TEST_F() in KVM selftests, odds
are good someone/something else will have a similar need, sooner or later.

> Also, setjmp/longjmp felt like it was introducing state that could be messed
> up easily.

Meh, same goes for the kselftests_harness.h.  I can point you at a half dozen
bugs where enhancements to the core framework wreaked havoc for unsuspecting
subsystems.  I'm not trying to throw shade at the harness; there's a _lot_ of
goodness in there.  My point is that doing complex things that impact a huge
variety of downstream users is going to have many sharp edges, regardless of
where the complexity resides.

I'm not wedded to setjmp/longjmp by any means, but for me this isn't a compelling
argument against the approach. 

> I also found recent work that removed setjmp/longjmp from kselftest harness
> [2].

KVM selftests don't support building with nolibc.

> The kselftests harness is running tests sequentially anyway, and the
> function pointers in _metadata wouldn't be changing all that often in most
> selftests.
> 
> Would maintainers be open to having the kselftest harness expose a pointer
> to the metadata globally?
>
> Another option would be to expose the current teardown function pointer
> globally instead of the pointer to the entire metadata struct.

I'm strongly opposed to any idea effectively requires special casing KVM selftests
in the common harness.  In my experience, the common harness is already quite
brittle, in large part because there is no singular maintainer(s) that is responsible
for ensuring changes work for all downstream users.  Adding odd bits of code that
is only ever used by a handful of KVM selftests is only going to increase the
probability that that code is broken.

Sean Christopherson <seanjc@google.com> writes:

> On Tue, Apr 14, 2026, Ackerley Tng wrote:
>> Sean suggested using setjmp and longjmp [1] to back to the top level
>> TEST_F(). I looked at [1] and found myself wishing to use TEST_F() the from
>> kselftest harness directly.
>
> Can you elaborate?  If you have a need for direct TEST_F() in KVM selftests, odds
> are good someone/something else will have a similar need, sooner or later.
>

I guess I like the consistency of working with TEST_F(), there's also
TEST_F_TIMEOUT() and friends and all the usefulness of the rest of the
kselftest_harness as you described, all of which will probably need
re-wrapping if we proceed with the approach of wrapping.

>> Also, setjmp/longjmp felt like it was introducing state that could be messed
>> up easily.
>
> Meh, same goes for the kselftests_harness.h.  I can point you at a half dozen
> bugs where enhancements to the core framework wreaked havoc for unsuspecting
> subsystems.  I'm not trying to throw shade at the harness; there's a _lot_ of
> goodness in there.  My point is that doing complex things that impact a huge
> variety of downstream users is going to have many sharp edges, regardless of
> where the complexity resides.
>
> I'm not wedded to setjmp/longjmp by any means, but for me this isn't a compelling
> argument against the approach.
>
>> I also found recent work that removed setjmp/longjmp from kselftest harness
>> [2].
>
> KVM selftests don't support building with nolibc.
>

Not particularly against setting it up with setjmp/longjmp either, I
think the discussion here is mostly between

1. Wrap kselftest_harness for KVM
2. Improving kselftest_harness so KVM can benefit from it

setjmp/longjmp has already been removed from kselftest_harness so that
option doesn't make sense if we go with (2).

If we go with (1), I could (a) store the _metadata pointer in a global
variable within KVM or (b) go with setjmp/longjmp (c) some other
suggestion.

>> The kselftests harness is running tests sequentially anyway, and the
>> function pointers in _metadata wouldn't be changing all that often in most
>> selftests.
>>
>> Would maintainers be open to having the kselftest harness expose a pointer
>> to the metadata globally?
>>
>> Another option would be to expose the current teardown function pointer
>> globally instead of the pointer to the entire metadata struct.
>
> I'm strongly opposed to any idea effectively requires special casing KVM selftests
> in the common harness.  In my experience, the common harness is already quite
> brittle, in large part because there is no singular maintainer(s) that is responsible
> for ensuring changes work for all downstream users.  Adding odd bits of code that
> is only ever used by a handful of KVM selftests is only going to increase the
> probability that that code is broken.

If Kees likes the idea of exposing a pointer to the metadata globally,
does that make KVM selftests less "special"?

If the community likes the global metadata pointer, I could update the
harness to adopt the global metadata pointer and then KVM wouldn't be
special at all.

On Tue, May 19, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > On Tue, Apr 14, 2026, Ackerley Tng wrote:
> >> Sean suggested using setjmp and longjmp [1] to back to the top level
> >> TEST_F(). I looked at [1] and found myself wishing to use TEST_F() the from
> >> kselftest harness directly.
> >
> > Can you elaborate?  If you have a need for direct TEST_F() in KVM selftests, odds
> > are good someone/something else will have a similar need, sooner or later.
> >
> 
> I guess I like the consistency of working with TEST_F(), there's also
> TEST_F_TIMEOUT() and friends and all the usefulness of the rest of the
> kselftest_harness as you described, all of which will probably need
> re-wrapping if we proceed with the approach of wrapping.

FWIW, utilizing the TIMEOUT functionality could get tricky.  KVM tests often have
highly variable runtimes based on the underlying environment.  E.g. as an extreme
case, see the never ending game of whack-a-mole we've been playing with flavors
of KVM-Unit-Tests' access test[*].

I can definitely see it being useful, e.g. for tests where we *know* the runtime
is O(ms), just want to call out that this is yet another case where KVM tests tend
to have more annoying requirements than other selftests.

[*] https://lore.kernel.org/all/20260317225327.4068448-1-yosry@kernel.org

> >> Another option would be to expose the current teardown function pointer
> >> globally instead of the pointer to the entire metadata struct.
> >
> > I'm strongly opposed to any idea effectively requires special casing KVM selftests
> > in the common harness.  In my experience, the common harness is already quite
> > brittle, in large part because there is no singular maintainer(s) that is responsible
> > for ensuring changes work for all downstream users.  Adding odd bits of code that
> > is only ever used by a handful of KVM selftests is only going to increase the
> > probability that that code is broken.
> 
> If Kees likes the idea of exposing a pointer to the metadata globally,
> does that make KVM selftests less "special"?

Yes.  I don't particuarly care about the code, I just don't want to be in a
situation where KVM is doing something "weird" relative to the rest of the world.

> If the community likes the global metadata pointer, I could update the
> harness to adopt the global metadata pointer and then KVM wouldn't be
> special at all.

Sean Christopherson <seanjc@google.com> writes:

> On Tue, May 19, 2026, Ackerley Tng wrote:
>> Sean Christopherson <seanjc@google.com> writes:
>>
>> > On Tue, Apr 14, 2026, Ackerley Tng wrote:
>> >> Sean suggested using setjmp and longjmp [1] to back to the top level
>> >> TEST_F(). I looked at [1] and found myself wishing to use TEST_F() the from
>> >> kselftest harness directly.
>> >
>> > Can you elaborate?  If you have a need for direct TEST_F() in KVM selftests, odds
>> > are good someone/something else will have a similar need, sooner or later.
>> >
>>
>> I guess I like the consistency of working with TEST_F(), there's also
>> TEST_F_TIMEOUT() and friends and all the usefulness of the rest of the
>> kselftest_harness as you described, all of which will probably need
>> re-wrapping if we proceed with the approach of wrapping.
>
> FWIW, utilizing the TIMEOUT functionality could get tricky.  KVM tests often have
> highly variable runtimes based on the underlying environment.  E.g. as an extreme
> case, see the never ending game of whack-a-mole we've been playing with flavors
> of KVM-Unit-Tests' access test[*].
>
> I can definitely see it being useful, e.g. for tests where we *know* the runtime
> is O(ms), just want to call out that this is yet another case where KVM tests tend
> to have more annoying requirements than other selftests.
>
> [*] https://lore.kernel.org/all/20260317225327.4068448-1-yosry@kernel.org
>

I guess today's KVM selftests are the equivalent of "never timeout", so
that could be a separate enhancement to kselftest_harness.

We could use kselftest_harness where it is useful in KVM selftests, not
as a replacement :)

>> >> Another option would be to expose the current teardown function pointer
>> >> globally instead of the pointer to the entire metadata struct.
>> >
>> > I'm strongly opposed to any idea effectively requires special casing KVM selftests
>> > in the common harness.  In my experience, the common harness is already quite
>> > brittle, in large part because there is no singular maintainer(s) that is responsible
>> > for ensuring changes work for all downstream users.  Adding odd bits of code that
>> > is only ever used by a handful of KVM selftests is only going to increase the
>> > probability that that code is broken.
>>
>> If Kees likes the idea of exposing a pointer to the metadata globally,
>> does that make KVM selftests less "special"?
>
> Yes.  I don't particuarly care about the code, I just don't want to be in a
> situation where KVM is doing something "weird" relative to the rest of the world.
>
>> If the community likes the global metadata pointer, I could update the
>> harness to adopt the global metadata pointer and then KVM wouldn't be
>> special at all.

On the PUCK call today Sean asked if this would help with the
TEST_REQUIRE(requirement) problem [1], where we want to skip this test
if a requirement is not met.

Having a metadata pointer helps specifically for the KVM_ONE_VCPU_TEST()
wrapper. We can't define a generic TEST_REQUIRE(), but SKIP() would
work, since you can define the SKIP() action to be return, which would
return out of the __suite##_##test() function and effectively skip the
test.

The SKIP() action needs to be handled all the way up to the top level
TEST(), so there's no generic TEST_REQUIRE() to be defined.

Defining TEST_REQUIRE() as SKIP(<call teardown>, abort(), "message")
would work but that only helps with teardown and doesn't continue with
the rest of the tests. To continue with the other tests, I can't really
think of a good option other than setjmp/longjmp.


It seems like the problem KVM_ONE_VCPU_TEST() solves is a FIXTURE
configuration problem. The vCPU could be set up in the FIXTURE_SETUP(),
but there's no good way to parametrize the fixture setup code. Did I
understand correctly?

I'll have that issue in the guest_memfd conversion tests too, leading to
lots of kselftest_harness wrappers. e.g. INIT_PRIVATE [2], then
INIT_SHARED [3]. There is FIXTURE_VARIANT_ADD, but the test code isn't
the same for the INIT_PRIVATE and INIT_SHARED tests.


Is the kselftest_harness-native way to parametrize fixture setup to
create different fixtures? Like FIXTURE(gmem_conversions_init_private),
FIXTURE(gmem_conversions_init_shared) and
FIXTURE(gmem_conversions_init_shared_4_pages),
FIXTURE(gmem_conversions_init_shared_8_pages)?

I guess FIXTURE_SETUP() does have access to the fixture name so setup
parameters could also be encoded in the fixture name.

[1] https://lore.kernel.org/all/ZjUwqEXPA5QVItyX@google.com/
[2] https://lore.kernel.org/all/20260507-gmem-inplace-conversion-v6-28-91ab5a8b19a4@google.com/
[3] https://lore.kernel.org/all/20260507-gmem-inplace-conversion-v6-29-91ab5a8b19a4@google.com/