From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED7A9C001DE for ; Wed, 19 Jul 2023 22:24:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230233AbjGSWYB (ORCPT ); Wed, 19 Jul 2023 18:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230028AbjGSWXz (ORCPT ); Wed, 19 Jul 2023 18:23:55 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7D4F51FE2 for ; Wed, 19 Jul 2023 15:23:51 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cb39970630eso124874276.0 for ; Wed, 19 Jul 2023 15:23:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805430; x=1692397430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ft9xoWZD/QvTAOwbAEWStbH8Xz6u5MZ/gxDDX+RVV/w=; b=aG8hSinYA2inhlQgad33Bn5DaJjlROKEqCcLnB3LE9xb3V/fSjtcm17CPmvemFRrxr PwQoEmTLYWlPAhG66oLxIB31rXeh2BvG0Y/Hopka5+Q4ZA3t6GTE2NpKo/ZYgA5M1d7+ xb85/K2KDe883zXXe2Dj4xhSyAKRlKTO/IBz3/933ndxZ2rXG2v/D8XvL4z+KZCHZhZ6 Zatkw042zKDHziWNtlmfRydD5hjarBBrvsMDsrOzhXi72Yc3UEtnPx1lN/3msWZ5S4QQ 9KylBZWxc2P1w58+Yoy3xeCj1gLLU/1+rPfaOzjrVvJ2nENib5Y2Wz6tT+gw1zsF8BzB p+eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805430; x=1692397430; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ft9xoWZD/QvTAOwbAEWStbH8Xz6u5MZ/gxDDX+RVV/w=; b=ioZuL+LUMGZgWlGjZe8YYah8Plpvu7ti1M2YFzFjgwlbL0xgJE2SJCm5UIdF9kWWO9 Sgo+TSHP2AmTsRdLXrN17Xw55rgdEeuUZXr5mx1vYT8Eyh0Ms81dNzancNI1uOImosaX HfE7K7OcJfK4f/Tw5IMJKSHNVs5pcVjaq61hbifKMhUqIXMqEWvSR3jQwNbHoVCVkmzv QrSeBo0PYMGQZMUxL//1Z2gzW9/SiYpU49mHXlEeIOOmgvFBqn72QCuHJ81EgUP7NYbg y+54RA1HAJbVARq/4Tt0zZctKdv+oJhfug8ShoLV763zZpAxm8D6iExxl42lzcnPr0d/ xESA== X-Gm-Message-State: ABy/qLaouX+bRgs9Z5dbWn+up1oyL5Iioy9Q8aDQ6pww5j/1Nf//6F4J QGtxAOKHAlE4yTYZ0UrAeryz8wPakQ== X-Google-Smtp-Source: APBJJlGLRp+F1v7+549LUrxZwf3p5gpUjvtjhHAqzv67uGWUJrlYSvRliQEhQW/NBJbli13aBiYHfxa8LA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:6902:1005:b0:cb3:c343:19e5 with SMTP id w5-20020a056902100500b00cb3c34319e5mr39818ybt.2.1689805430803; Wed, 19 Jul 2023 15:23:50 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:28 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-2-rmoar@google.com> Subject: [PATCH v1 1/9] kunit: Add test attributes API structure From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add the basic structure of the test attribute API to KUnit, which can be used to save and access test associated data. Add attributes.c and attributes.h to hold associated structs and functions for the API. Create a struct that holds a variety of associated helper functions for each test attribute. These helper functions will be used to get the attribute value, convert the value to a string, and filter based on the value. This struct is flexible by design to allow for attributes of numerous types and contexts. Add a method to print test attributes in the format of "# [.]: ". Example for a suite: "# speed: slow" Example for a test case: "# test_case.speed: very_slow" Use this method to report attributes in the KTAP output (KTAP spec: https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when kernel's new kunit.action=3Dlist_attr option is used. Note this is derivati= ve of the kunit.action=3Dlist option. In test.h, add fields and associated helper functions to test cases and suites to hold user-inputted test attributes. Signed-off-by: Rae Moar --- Changes since RFC v2: - No major changes Changes since RFC v1: - Add option to only include attributes in the _list_tests output when this module param is set - Add printing options for attributes to print always, print only for suites, or print never. include/kunit/attributes.h | 19 +++++++++ include/kunit/test.h | 33 ++++++++++++++++ lib/kunit/Makefile | 3 +- lib/kunit/attributes.c | 80 ++++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 21 ++++++++-- lib/kunit/test.c | 17 ++++---- 6 files changed, 161 insertions(+), 12 deletions(-) create mode 100644 include/kunit/attributes.h create mode 100644 lib/kunit/attributes.c diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h new file mode 100644 index 000000000000..9fcd184cce36 --- /dev/null +++ b/include/kunit/attributes.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#ifndef _KUNIT_ATTRIBUTES_H +#define _KUNIT_ATTRIBUTES_H + +/* + * Print all test attributes for a test case or suite. + * Output format for test cases: "# .: " + * Output format for test suites: "# : " + */ +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test= _level); + +#endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/include/kunit/test.h b/include/kunit/test.h index 23120d50499e..1fc9155988e9 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -63,12 +63,16 @@ enum kunit_status { KUNIT_SKIPPED, }; =20 +/* Holds attributes for each test case and suite */ +struct kunit_attributes {}; + /** * struct kunit_case - represents an individual test case. * * @run_case: the function representing the actual test case. * @name: the name of the test case. * @generate_params: the generator function for parameterized tests. + * @attr: the attributes associated with the test * * A test case is a function with the signature, * ``void (*)(struct kunit *)`` @@ -104,6 +108,7 @@ struct kunit_case { void (*run_case)(struct kunit *test); const char *name; const void* (*generate_params)(const void *prev, char *desc); + struct kunit_attributes attr; =20 /* private: internal use only. */ enum kunit_status status; @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kun= it_status status) */ #define KUNIT_CASE(test_name) { .run_case =3D test_name, .name =3D #test_n= ame } =20 +/** + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case + * with attributes + * + * @test_name: a reference to a test case function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_ATTR(test_name, attributes) \ + { .run_case =3D test_name, .name =3D #test_name, \ + .attr =3D attributes } + /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_= case * @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kun= it_status status) { .run_case =3D test_name, .name =3D #test_name, \ .generate_params =3D gen_params } =20 +/** + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct + * kunit_case with attributes + * + * @test_name: a reference to a test case function. + * @gen_params: a reference to a parameter generator function. + * @attributes: a reference to a struct kunit_attributes object containing + * test attributes + */ +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \ + { .run_case =3D test_name, .name =3D #test_name, \ + .generate_params =3D gen_params, \ + .attr =3D attributes } + /** * struct kunit_suite - describes a related collection of &struct kunit_ca= se * @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) * @init: called before every test case. * @exit: called after every test case. * @test_cases: a null terminated array of test cases. + * @attr: the attributes associated with the test suite * * A kunit_suite is a collection of related &struct kunit_case s, such that * @init is called before every test case and @exit is called after every @@ -182,6 +214,7 @@ struct kunit_suite { int (*init)(struct kunit *test); void (*exit)(struct kunit *test); struct kunit_case *test_cases; + struct kunit_attributes attr; =20 /* private: internal use only */ char status_comment[KUNIT_STATUS_COMMENT_SIZE]; diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile index cb417f504996..46f75f23dfe4 100644 --- a/lib/kunit/Makefile +++ b/lib/kunit/Makefile @@ -6,7 +6,8 @@ kunit-objs +=3D test.o \ string-stream.o \ assert.o \ try-catch.o \ - executor.o + executor.o \ + attributes.o =20 ifeq ($(CONFIG_KUNIT_DEBUGFS),y) kunit-objs +=3D debugfs.o diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c new file mode 100644 index 000000000000..9bda5a5f4030 --- /dev/null +++ b/lib/kunit/attributes.c @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * KUnit API to save and access test attributes + * + * Copyright (C) 2023, Google LLC. + * Author: Rae Moar + */ + +#include +#include + +/* Options for printing attributes: + * PRINT_ALWAYS - attribute is printed for every test case and suite if set + * PRINT_SUITE - attribute is printed for every suite if set but not for t= est cases + * PRINT_NEVER - attribute is never printed + */ +enum print_ops { + PRINT_ALWAYS, + PRINT_SUITE, + PRINT_NEVER, +}; + +/** + * struct kunit_attr - represents a test attribute and holds flexible + * helper functions to interact with attribute. + * + * @name: name of test attribute, eg. speed + * @get_attr: function to return attribute value given a test + * @to_string: function to return string representation of given + * attribute value + * @filter: function to indicate whether a given attribute value passes a + * filter + */ +struct kunit_attr { + const char *name; + void *(*get_attr)(void *test_or_suite, bool is_test); + const char *(*to_string)(void *attr, bool *to_free); + int (*filter)(void *attr, const char *input, int *err); + void *attr_default; + enum print_ops print; +}; + +/* List of all Test Attributes */ + +static struct kunit_attr kunit_attr_list[] =3D {}; + +/* Helper Functions to Access Attributes */ + +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test= _level) +{ + int i; + bool to_free; + void *attr; + const char *attr_name, *attr_str; + struct kunit_suite *suite =3D is_test ? NULL : test_or_suite; + struct kunit_case *test =3D is_test ? test_or_suite : NULL; + + for (i =3D 0; i < ARRAY_SIZE(kunit_attr_list); i++) { + if (kunit_attr_list[i].print =3D=3D PRINT_NEVER || + (test && kunit_attr_list[i].print =3D=3D PRINT_SUITE)) + continue; + attr =3D kunit_attr_list[i].get_attr(test_or_suite, is_test); + if (attr) { + attr_name =3D kunit_attr_list[i].name; + attr_str =3D kunit_attr_list[i].to_string(attr, &to_free); + if (test) { + kunit_log(KERN_INFO, test, "%*s# %s.%s: %s", + KUNIT_INDENT_LEN * test_level, "", test->name, + attr_name, attr_str); + } else { + kunit_log(KERN_INFO, suite, "%*s# %s: %s", + KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str); + } + + /* Free to_string of attribute if needed */ + if (to_free) + kfree(attr_str); + } + } +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 74982b83707c..12e38a48a5cc 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -2,6 +2,7 @@ =20 #include #include +#include #include #include =20 @@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0); MODULE_PARM_DESC(action, "Changes KUnit executor behavior, valid values are:\n" ": run the tests like normal\n" - "'list' to list test names instead of running them.\n"); + "'list' to list test names instead of running them.\n" + "'list_attr' to list test names and attributes instead of running them.= \n"); =20 /* glob_match() needs NULL terminated strings, so we need a copy of filter= _glob_param. */ struct kunit_test_filter { @@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suit= e_set) __kunit_test_suites_init(suite_set->start, num_suites); } =20 -static void kunit_exec_list_tests(struct suite_set *suite_set) +static void kunit_exec_list_tests(struct suite_set *suite_set, bool includ= e_attr) { struct kunit_suite * const *suites; struct kunit_case *test_case; @@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *s= uite_set) /* Hack: print a ktap header so kunit.py can find the start of KUnit outp= ut. */ pr_info("KTAP version 1\n"); =20 - for (suites =3D suite_set->start; suites < suite_set->end; suites++) + for (suites =3D suite_set->start; suites < suite_set->end; suites++) { + /* Print suite name and suite attributes */ + pr_info("%s\n", (*suites)->name); + if (include_attr) + kunit_print_attr((void *)(*suites), false, 0); + + /* Print test case name and attributes in suite */ kunit_suite_for_each_test_case((*suites), test_case) { pr_info("%s.%s\n", (*suites)->name, test_case->name); + if (include_attr) + kunit_print_attr((void *)test_case, true, 0); } + } } =20 int kunit_run_all_tests(void) @@ -206,7 +217,9 @@ int kunit_run_all_tests(void) if (!action_param) kunit_exec_run_tests(&suite_set); else if (strcmp(action_param, "list") =3D=3D 0) - kunit_exec_list_tests(&suite_set); + kunit_exec_list_tests(&suite_set, false); + else if (strcmp(action_param, "list_attr") =3D=3D 0) + kunit_exec_list_tests(&suite_set, true); else pr_err("kunit executor: unknown action '%s'\n", action_param); =20 diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 84e4666555c9..9ee55139ecd1 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include #include @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *= suite) } EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases); =20 +/* Currently supported test levels */ +enum { + KUNIT_LEVEL_SUITE =3D 0, + KUNIT_LEVEL_CASE, + KUNIT_LEVEL_CASE_PARAM, +}; + static void kunit_print_suite_start(struct kunit_suite *suite) { /* @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suit= e *suite) pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n"); pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n", suite->name); + kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE); pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n", kunit_suite_num_test_cases(suite)); } =20 -/* Currently supported test levels */ -enum { - KUNIT_LEVEL_SUITE =3D 0, - KUNIT_LEVEL_CASE, - KUNIT_LEVEL_CASE_PARAM, -}; - static void kunit_print_ok_not_ok(struct kunit *test, unsigned int test_level, enum kunit_status status, @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite) } } =20 + kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE); =20 kunit_print_test_stats(&test, param_stats); =20 --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D77FCC001DF for ; Wed, 19 Jul 2023 22:24:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230326AbjGSWYE (ORCPT ); Wed, 19 Jul 2023 18:24:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230215AbjGSWX5 (ORCPT ); Wed, 19 Jul 2023 18:23:57 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09E4D1FFB for ; Wed, 19 Jul 2023 15:23:54 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-56fffdea2d0so1990047b3.1 for ; Wed, 19 Jul 2023 15:23:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805434; x=1692397434; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9sRq/L9BAwqWXNrlVe37oHbZZhdY2nOQD+Izcb3qkl8=; b=6aR9dhJ8U0+qxU7ESK1dduHa+doKJIIhbz6sedBnCkirKzlDVSXseuV04sR7qXGdv0 WQd4m9B1Pm65VneT1AmZzxUhnignOqXipQMqzkFZvnlVmzpmVRfCvcN9XMltrO6SyeFW LYHSbhvrX09byBjvzE1F5mvqQ/ACuhSHU1IPjtsw6XZTXExoarNiR/qKArVIRzIvc93+ CwYwmLEqjEKhtHugkNZZd1AHu6u/0OMdOOuYwazHWeYbfmcLNBe2P4hr00m00C9ROtKx 4BhrHHpwqn1ZU6V4XB5/L/5eu9vTFGjkZwv0dK6SmjsCUeCiuqLNGwiaEYZRvhwjcZ1R Ma8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805434; x=1692397434; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9sRq/L9BAwqWXNrlVe37oHbZZhdY2nOQD+Izcb3qkl8=; b=A1iYG0wlhqpnO9hyGUsj1n9M/lC2rySxAawnw0D6tJ4gvcygOFyNZ7e75H4rewsBH7 lbo2WUGZCvYj0sVHV7ytgoj0Qplq546DOEO8zc8rPyrJjImc6eY2Y6WTjOsAhHnJ3mnI YU9rM4jGcCsh3skm5ojCTmggsQtAgsWqcopF0le7KMN0TXBrwesjre0ytsu3KVAJACnR OFoxaWifScnVs7VxQaEA75wewus0da7mwTnrd+33qHwf49WTF8TlW+xQsdRdjyom4l/d D6uGlEIwdQqvfXW4K/UPpjsD+GQkYvUZpIHAho/fJr5obJ5vMAe2RBn8VnoDDEcYOOPn 6YPw== X-Gm-Message-State: ABy/qLab4BiiqOHmcjijHrp7z8k+ktSo4dDyN0hK9DfcxvKnwwW2jlra 4qY2P7+ViMfpg1KSb/cfOPIjeZ+92w== X-Google-Smtp-Source: APBJJlEE09FI4unmyB8EvpDDYafH7/d1DRcm2gmzizuC+we3bVianzZ/HMPSV2lbKhfRA5S3UCagemsE8Q== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:6902:1709:b0:cab:e42c:876b with SMTP id by9-20020a056902170900b00cabe42c876bmr34335ybb.3.1689805434000; Wed, 19 Jul 2023 15:23:54 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:29 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-3-rmoar@google.com> Subject: [PATCH v1 2/9] kunit: Add speed attribute From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add speed attribute to the test attribute API. This attribute will allow users to mark tests with a category of speed. Currently the categories of speed proposed are: normal, slow, and very_slow (outlined in enum kunit_speed). These are outlined in the enum kunit_speed. The assumed default speed for tests is "normal". This indicates that the test takes a relatively trivial amount of time (less than 1 second), regardless of the machine it is running on. Any test slower than this could be marked as "slow" or "very_slow". Add the macro KUNIT_CASE_SLOW to set a test as slow, as this is likely a common use of the attributes API. Add an example of marking a slow test to kunit-example-test.c. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since RFC v2: - Fixup comment. Changes since RFC v1: - Remove the "fast" category of speed. include/kunit/test.h | 32 ++++++++++++++++++++++- lib/kunit/attributes.c | 46 +++++++++++++++++++++++++++++++++- lib/kunit/kunit-example-test.c | 9 +++++++ 3 files changed, 85 insertions(+), 2 deletions(-) diff --git a/include/kunit/test.h b/include/kunit/test.h index 1fc9155988e9..ed5f5000a095 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -63,8 +63,27 @@ enum kunit_status { KUNIT_SKIPPED, }; =20 +/* Attribute struct/enum definitions */ + +/* + * Speed Attribute is stored as an enum and separated into categories of + * speed: very_slowm, slow, and normal. These speeds are relative to + * other KUnit tests. + * + * Note: unset speed attribute acts as default of KUNIT_SPEED_NORMAL. + */ +enum kunit_speed { + KUNIT_SPEED_UNSET, + KUNIT_SPEED_VERY_SLOW, + KUNIT_SPEED_SLOW, + KUNIT_SPEED_NORMAL, + KUNIT_SPEED_MAX =3D KUNIT_SPEED_NORMAL, +}; + /* Holds attributes for each test case and suite */ -struct kunit_attributes {}; +struct kunit_attributes { + enum kunit_speed speed; +}; =20 /** * struct kunit_case - represents an individual test case. @@ -150,6 +169,17 @@ static inline char *kunit_status_to_ok_not_ok(enum kun= it_status status) { .run_case =3D test_name, .name =3D #test_name, \ .attr =3D attributes } =20 +/** + * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case + * with the slow attribute + * + * @test_name: a reference to a test case function. + */ + +#define KUNIT_CASE_SLOW(test_name) \ + { .run_case =3D test_name, .name =3D #test_name, \ + .attr.speed =3D KUNIT_SPEED_SLOW } + /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_= case * diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index 9bda5a5f4030..e97096dbb3b1 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -40,9 +40,53 @@ struct kunit_attr { enum print_ops print; }; =20 +/* String Lists for enum Attributes */ + +static const char * const speed_str_list[] =3D {"unset", "very_slow", "slo= w", "normal"}; + +/* To String Methods */ + +static const char *attr_enum_to_string(void *attr, const char * const str_= list[], bool *to_free) +{ + long val =3D (long)attr; + + *to_free =3D false; + if (!val) + return NULL; + return str_list[val]; +} + +static const char *attr_speed_to_string(void *attr, bool *to_free) +{ + return attr_enum_to_string(attr, speed_str_list, to_free); +} + +/* Get Attribute Methods */ + +static void *attr_speed_get(void *test_or_suite, bool is_test) +{ + struct kunit_suite *suite =3D is_test ? NULL : test_or_suite; + struct kunit_case *test =3D is_test ? test_or_suite : NULL; + + if (test) + return ((void *) test->attr.speed); + else + return ((void *) suite->attr.speed); +} + +/* Attribute Struct Definitions */ + +static const struct kunit_attr speed_attr =3D { + .name =3D "speed", + .get_attr =3D attr_speed_get, + .to_string =3D attr_speed_to_string, + .attr_default =3D (void *)KUNIT_SPEED_NORMAL, + .print =3D PRINT_ALWAYS, +}; + /* List of all Test Attributes */ =20 -static struct kunit_attr kunit_attr_list[] =3D {}; +static struct kunit_attr kunit_attr_list[] =3D {speed_attr}; =20 /* Helper Functions to Access Attributes */ =20 diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c index b69b689ea850..01a769f35e1d 100644 --- a/lib/kunit/kunit-example-test.c +++ b/lib/kunit/kunit-example-test.c @@ -220,6 +220,14 @@ static void example_params_test(struct kunit *test) KUNIT_EXPECT_EQ(test, param->value % param->value, 0); } =20 +/* + * This test should always pass. Can be used to practice filtering attribu= tes. + */ +static void example_slow_test(struct kunit *test) +{ + KUNIT_EXPECT_EQ(test, 1 + 1, 2); +} + /* * Here we make a list of all the test cases we want to add to the test su= ite * below. @@ -237,6 +245,7 @@ static struct kunit_case example_test_cases[] =3D { KUNIT_CASE(example_all_expect_macros_test), KUNIT_CASE(example_static_stub_test), KUNIT_CASE_PARAM(example_params_test, example_gen_params), + KUNIT_CASE_SLOW(example_slow_test), {} }; =20 --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B193C0015E for ; Wed, 19 Jul 2023 22:24:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230461AbjGSWYJ (ORCPT ); Wed, 19 Jul 2023 18:24:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229622AbjGSWYA (ORCPT ); Wed, 19 Jul 2023 18:24:00 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 94BCD2112 for ; Wed, 19 Jul 2023 15:23:57 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-bd69bb4507eso112798276.2 for ; Wed, 19 Jul 2023 15:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805436; x=1692397436; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Rd3boOdAlfH2nRM3+RX0LZSIemRKG8MVf9sKC/jWVn8=; b=ZxrsLCF8zArTqOVJ6JX4jfVhGjbQDfz604yPryk2p5bCmj3ZcxkurE6OosjJIz2dqe 1wrHH00+ExltFh++DlxRaxwseeAZaN/WsUt4cc18viWQTwhfQNhUOD6xiH8fAFt546wk /Lwjx6v4/dapgFODMZg9a4bTtx0vMOXdxBluDQi6ZPE6ZULysdMCV9jn4M4xXEjamXaX QOPS7HCzF0iImxVDcPXS0qWQiWnVUwVoLrvU0zzVb0CNZWBs9BYywXExdHg9fxrc4HJZ NtXX1MKs05AABQQ/0jjgKpU2J1Mwfoz2JcXiAGPwWN3UD+1etnHzP4AYg0Psx0SKTCEi pt4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805436; x=1692397436; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rd3boOdAlfH2nRM3+RX0LZSIemRKG8MVf9sKC/jWVn8=; b=bYUp+L6SqlyIzG5+yRcMfFpeknnUxgoF22w0Y/WjsiaFh5jkSqnw8f8NZZKTWogdI/ dPY57hZupHiXuJJ2RyumanvcLpBPbUK5eFqEy1lC56+ThqPjGtu9e/nNi75V8ytWO7nz gekh6tEo508kiUu0HvJqkqBX4XCFtrH6W/sGuN1UljQq1Fq0uzN8V8r7ST0tp5ghzzxI xPybz37bc2Ihhi+YNjmlmgBWlJO7vcdp1Mi/EZcz8M3EzdqlnLSsAEUWoe913U//FVjC JABg88MOEdPqps5ptL1fRlFWw1FRLD4tDbUi5Q5YjrwHFFvzWRnclBmvorwZWT/BRg+W g+hQ== X-Gm-Message-State: ABy/qLZks0wyGMl//ivDRqrB4iavcCcZnWz+xx2daMHuM+7tPKL3tTd5 w3bXnU+bJe6PBARldoqxEyO5QcpsSw== X-Google-Smtp-Source: APBJJlFJoNuU+UE39XO0UaovGMTShePgnxiZ3V2UQyVVMe70Tv1/5UzXVkg/ZqQ+UXDG89ufoC3XGwimcA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:ab04:0:b0:bfe:ea69:91b4 with SMTP id u4-20020a25ab04000000b00bfeea6991b4mr29862ybi.4.1689805436512; Wed, 19 Jul 2023 15:23:56 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:30 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-4-rmoar@google.com> Subject: [PATCH v1 3/9] kunit: Add module attribute From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add module attribute to the test attribute API. This attribute stores the module name associated with the test using KBUILD_MODNAME. The name of a test suite and the module name often do not match. A reference to the module name associated with the suite could be extremely helpful in running tests as modules without needing to check the codebase. This attribute will be printed for each suite. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since RFC v2: - No changes. Changes: since RFC v1: - This is a new patch. include/kunit/test.h | 13 ++++++++----- lib/kunit/attributes.c | 28 +++++++++++++++++++++++++++- 2 files changed, 35 insertions(+), 6 deletions(-) diff --git a/include/kunit/test.h b/include/kunit/test.h index ed5f5000a095..011e0d6bb506 100644 --- a/include/kunit/test.h +++ b/include/kunit/test.h @@ -131,6 +131,7 @@ struct kunit_case { =20 /* private: internal use only. */ enum kunit_status status; + char *module_name; char *log; }; =20 @@ -155,7 +156,9 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) * &struct kunit_case object from it. See the documentation for * &struct kunit_case for an example on how to use it. */ -#define KUNIT_CASE(test_name) { .run_case =3D test_name, .name =3D #test_n= ame } +#define KUNIT_CASE(test_name) \ + { .run_case =3D test_name, .name =3D #test_name, \ + .module_name =3D KBUILD_MODNAME} =20 /** * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case @@ -167,7 +170,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) */ #define KUNIT_CASE_ATTR(test_name, attributes) \ { .run_case =3D test_name, .name =3D #test_name, \ - .attr =3D attributes } + .attr =3D attributes, .module_name =3D KBUILD_MODNAME} =20 /** * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case @@ -178,7 +181,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) =20 #define KUNIT_CASE_SLOW(test_name) \ { .run_case =3D test_name, .name =3D #test_name, \ - .attr.speed =3D KUNIT_SPEED_SLOW } + .attr.speed =3D KUNIT_SPEED_SLOW, .module_name =3D KBUILD_MODNAME} =20 /** * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_= case @@ -199,7 +202,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) */ #define KUNIT_CASE_PARAM(test_name, gen_params) \ { .run_case =3D test_name, .name =3D #test_name, \ - .generate_params =3D gen_params } + .generate_params =3D gen_params, .module_name =3D KBUILD_MODNAME} =20 /** * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct @@ -213,7 +216,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kuni= t_status status) #define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes) \ { .run_case =3D test_name, .name =3D #test_name, \ .generate_params =3D gen_params, \ - .attr =3D attributes } + .attr =3D attributes, .module_name =3D KBUILD_MODNAME} =20 /** * struct kunit_suite - describes a related collection of &struct kunit_ca= se diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index e97096dbb3b1..43dcb5de8b8f 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -61,6 +61,12 @@ static const char *attr_speed_to_string(void *attr, bool= *to_free) return attr_enum_to_string(attr, speed_str_list, to_free); } =20 +static const char *attr_string_to_string(void *attr, bool *to_free) +{ + *to_free =3D false; + return (char *) attr; +} + /* Get Attribute Methods */ =20 static void *attr_speed_get(void *test_or_suite, bool is_test) @@ -74,6 +80,18 @@ static void *attr_speed_get(void *test_or_suite, bool is= _test) return ((void *) suite->attr.speed); } =20 +static void *attr_module_get(void *test_or_suite, bool is_test) +{ + struct kunit_suite *suite =3D is_test ? NULL : test_or_suite; + struct kunit_case *test =3D is_test ? test_or_suite : NULL; + + // Suites get their module attribute from their first test_case + if (test) + return ((void *) test->module_name); + else + return ((void *) suite->test_cases[0].module_name); +} + /* Attribute Struct Definitions */ =20 static const struct kunit_attr speed_attr =3D { @@ -84,9 +102,17 @@ static const struct kunit_attr speed_attr =3D { .print =3D PRINT_ALWAYS, }; =20 +static const struct kunit_attr module_attr =3D { + .name =3D "module", + .get_attr =3D attr_module_get, + .to_string =3D attr_string_to_string, + .attr_default =3D (void *)"", + .print =3D PRINT_SUITE, +}; + /* List of all Test Attributes */ =20 -static struct kunit_attr kunit_attr_list[] =3D {speed_attr}; +static struct kunit_attr kunit_attr_list[] =3D {speed_attr, module_attr}; =20 /* Helper Functions to Access Attributes */ =20 --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FB2EC001DF for ; Wed, 19 Jul 2023 22:24:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230436AbjGSWYT (ORCPT ); Wed, 19 Jul 2023 18:24:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230442AbjGSWYI (ORCPT ); Wed, 19 Jul 2023 18:24:08 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23B071FFB for ; Wed, 19 Jul 2023 15:24:00 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cac66969edaso113324276.3 for ; Wed, 19 Jul 2023 15:23:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805438; x=1692397438; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=s3f3JSR7GYuMkfq2Qjp1KKE+tv6rX/vJigTO6xpPGKA=; b=nLAbeJJi8wZqEcjJsnUbro+3934ONlLn7oSKGtWDoHAtbAP916hPQbWv/wRUysgLwQ 21jHiqc95QS1Cumgoijd/UYSKCK8qotcoXGWSPxitn39++7B/A7HHWeMz8H7RQY+hMBN dl22oJz+fJ3bDgshpaqi7HBlJgllkDI/4croY3TBYo6GZs3/+QB8MVTbEH8KOGcGv0DT WGHFQFH4YNkOLXcKbPUZuxmXYEZQjBVymPgjZPPewEmoWiKpcDFliOVnR8SVLQUTxhrD G1t5emUe042PJRJssmNxrUqj/1Gq98OwSIKiUIrrL58zozNV4fQOvne4RlNWfdGQcjfL QFkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805438; x=1692397438; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s3f3JSR7GYuMkfq2Qjp1KKE+tv6rX/vJigTO6xpPGKA=; b=OYtz2uT9gNADKb9uLdjMaM888lh58KICoe0xeridA4EgZkDFri//abHfnQpI+X2iBX shZbyNZGD1pI0Q78yHhSVqPwi8FH2pCcjupwdi1J3GysHKx7kKOfvCbH2EM+nhdFT4YP zRLSpV3StmS4TWdv9HlgZzy0wd0ChhPt0vAYLoQdaQfcpwGjsj8QTQvdyJau4q3GIsMK RHGWRJ+oHjMDTucqKILD7y0OKeUL/SqpafOExO6w32uTPwOp9K/Wvd67nmGYTr2WqGGx zmprOTZnxTttAec/bdjxrNRenPLeFh5o6Z4a9OEp4jqvTCbpDyCp9pM/DxarXuGQeqli Xh5w== X-Gm-Message-State: ABy/qLbzZ4dCrrRSIQhFQZbejfY/5bgoToB77Av8cA6MpSHpOWLyCahn b3Q1oUMBq2HgHx+1wzUB51ozng1M2w== X-Google-Smtp-Source: APBJJlEngeK+hVJN5JMB3wos7ZwaSvbOjp00/POiafO2UkfpgCbiCmhqMpe7BMrLPpyz3S51drylpJSmnQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:6902:1005:b0:cb3:c343:19e5 with SMTP id w5-20020a056902100500b00cb3c34319e5mr39820ybt.2.1689805438645; Wed, 19 Jul 2023 15:23:58 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:31 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-5-rmoar@google.com> Subject: [PATCH v1 4/9] kunit: Add ability to filter attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add filtering of test attributes. Users can filter tests using the module_param called "filter". Filters are imputed in the format: Example: kunit.filter=3D"speed>slow" Operations include: >, <, >=3D, <=3D, !=3D, and =3D. These operations will = act the same for attributes of the same type but may not between types. Note multiple filters can be inputted by separating them with a comma. Example: kunit.filter=3D"speed=3Dslow, module!=3Dexample" Since both suites and test cases can have attributes, there may be conflicts. The process of filtering follows these rules: - Filtering always operates at a per-test level. - If a test has an attribute set, then the test's value is filtered on. - Otherwise, the value falls back to the suite's value. - If neither are set, the attribute has a global "default" value, which is used. Filtered tests will not be run or show in output. The tests can instead be skipped using the configurable option "kunit.filter_action=3Dskip". Note the default settings for running tests remains unfiltered. Finally, add "filter" methods for the speed and module attributes to parse and compare attribute values. Note this filtering functionality will be added to kunit.py in the next patch. Signed-off-by: Rae Moar --- Changes since RFC v2: - Change to output only one error before exiting. Changes since RFC v1: - Change method for inputting filters to allow for spaces in filtering values - Add option to skip filtered tests instead of not run or show them with the --filter_skip flag include/kunit/attributes.h | 31 +++++ lib/kunit/attributes.c | 271 +++++++++++++++++++++++++++++++++++++ lib/kunit/executor.c | 94 ++++++++++--- lib/kunit/executor_test.c | 12 +- lib/kunit/test.c | 10 +- 5 files changed, 390 insertions(+), 28 deletions(-) diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h index 9fcd184cce36..bc76a0b786d2 100644 --- a/include/kunit/attributes.h +++ b/include/kunit/attributes.h @@ -9,6 +9,20 @@ #ifndef _KUNIT_ATTRIBUTES_H #define _KUNIT_ATTRIBUTES_H =20 +/* + * struct kunit_attr_filter - representation of attributes filter with the + * attribute object and string input + */ +struct kunit_attr_filter { + struct kunit_attr *attr; + char *input; +}; + +/* + * Returns the name of the filter's attribute. + */ +const char *kunit_attr_filter_name(struct kunit_attr_filter filter); + /* * Print all test attributes for a test case or suite. * Output format for test cases: "# .: " @@ -16,4 +30,21 @@ */ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test= _level); =20 +/* + * Returns the number of fitlers in input. + */ +int kunit_get_filter_count(char *input); + +/* + * Parse attributes filter input and return an objects containing the + * attribute object and the string input of the next filter. + */ +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err); + +/* + * Returns a copy of the suite containing only tests that pass the filter. + */ +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *cons= t suite, + struct kunit_attr_filter filter, char *action, int *err); + #endif /* _KUNIT_ATTRIBUTES_H */ diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c index 43dcb5de8b8f..861b5ae3c1a1 100644 --- a/lib/kunit/attributes.c +++ b/lib/kunit/attributes.c @@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bo= ol *to_free) return (char *) attr; } =20 +/* Filter Methods */ + +static const char op_list[] =3D "<>!=3D"; + +/* + * Returns whether the inputted integer value matches the filter given + * by the operation string and inputted integer. + */ +static int int_filter(long val, const char *op, int input, int *err) +{ + if (!strncmp(op, "<=3D", 2)) + return (val <=3D input); + else if (!strncmp(op, ">=3D", 2)) + return (val >=3D input); + else if (!strncmp(op, "!=3D", 2)) + return (val !=3D input); + else if (!strncmp(op, ">", 1)) + return (val > input); + else if (!strncmp(op, "<", 1)) + return (val < input); + else if (!strncmp(op, "=3D", 1)) + return (val =3D=3D input); + *err =3D -EINVAL; + pr_err("kunit executor: invalid filter operation: %s\n", op); + return false; +} + +/* + * Returns whether the inputted enum value "attr" matches the filter given + * by the input string. Note: the str_list includes the corresponding stri= ng + * list to the enum values. + */ +static int attr_enum_filter(void *attr, const char *input, int *err, + const char * const str_list[], int max) +{ + int i, j, input_int; + long test_val =3D (long)attr; + const char *input_val; + + for (i =3D 0; input[i]; i++) { + if (!strchr(op_list, input[i])) { + input_val =3D input + i; + break; + } + } + + if (!input_val) { + *err =3D -EINVAL; + pr_err("kunit executor: filter value not found: %s\n", input); + return false; + } + + for (j =3D 0; j <=3D max; j++) { + if (!strcmp(input_val, str_list[j])) + input_int =3D j; + } + + if (!input_int) { + *err =3D -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } + + return int_filter(test_val, input, input_int, err); +} + +static int attr_speed_filter(void *attr, const char *input, int *err) +{ + return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX= ); +} + +/* + * Returns whether the inputted string value (attr) matches the filter giv= en + * by the input string. + */ +static int attr_string_filter(void *attr, const char *input, int *err) +{ + char *str =3D attr; + + if (!strncmp(input, "<", 1)) { + *err =3D -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } else if (!strncmp(input, ">", 1)) { + *err =3D -EINVAL; + pr_err("kunit executor: invalid filter input: %s\n", input); + return false; + } else if (!strncmp(input, "!=3D", 2)) { + return (strcmp(input + 2, str) !=3D 0); + } else if (!strncmp(input, "=3D", 1)) { + return (strcmp(input + 1, str) =3D=3D 0); + } + *err =3D -EINVAL; + pr_err("kunit executor: invalid filter operation: %s\n", input); + return false; +} + + /* Get Attribute Methods */ =20 static void *attr_speed_get(void *test_or_suite, bool is_test) @@ -98,6 +196,7 @@ static const struct kunit_attr speed_attr =3D { .name =3D "speed", .get_attr =3D attr_speed_get, .to_string =3D attr_speed_to_string, + .filter =3D attr_speed_filter, .attr_default =3D (void *)KUNIT_SPEED_NORMAL, .print =3D PRINT_ALWAYS, }; @@ -106,6 +205,7 @@ static const struct kunit_attr module_attr =3D { .name =3D "module", .get_attr =3D attr_module_get, .to_string =3D attr_string_to_string, + .filter =3D attr_string_filter, .attr_default =3D (void *)"", .print =3D PRINT_SUITE, }; @@ -116,6 +216,11 @@ static struct kunit_attr kunit_attr_list[] =3D {speed_= attr, module_attr}; =20 /* Helper Functions to Access Attributes */ =20 +const char *kunit_attr_filter_name(struct kunit_attr_filter filter) +{ + return filter.attr->name; +} + void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test= _level) { int i; @@ -148,3 +253,169 @@ void kunit_print_attr(void *test_or_suite, bool is_te= st, unsigned int test_level } } } + +/* Helper Functions to Filter Attributes */ + +int kunit_get_filter_count(char *input) +{ + int i, comma_index, count =3D 0; + + for (i =3D 0; input[i]; i++) { + if (input[i] =3D=3D ',') { + if ((i - comma_index) > 1) + count++; + comma_index =3D i; + } + } + if ((i - comma_index) > 0) + count++; + return count; +} + +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err) +{ + struct kunit_attr_filter filter; + int i, j, comma_index, new_start_index; + int op_index =3D -1, attr_index =3D -1; + char op; + char *input =3D *filters; + + /* Parse input until operation */ + for (i =3D 0; input[i]; i++) { + if (op_index < 0 && strchr(op_list, input[i])) { + op_index =3D i; + } else if (!comma_index && input[i] =3D=3D ',') { + comma_index =3D i; + } else if (comma_index && input[i] !=3D ' ') { + new_start_index =3D i; + break; + } + } + + if (op_index <=3D 0) { + *err =3D -EINVAL; + pr_err("kunit executor: filter operation not found: %s\n", input); + return filter; + } + + /* Temporarily set operator to \0 character. */ + op =3D input[op_index]; + input[op_index] =3D '\0'; + + /* Find associated kunit_attr object */ + for (j =3D 0; j < ARRAY_SIZE(kunit_attr_list); j++) { + if (!strcmp(input, kunit_attr_list[j].name)) { + attr_index =3D j; + break; + } + } + + input[op_index] =3D op; + + if (attr_index < 0) { + *err =3D -EINVAL; + pr_err("kunit executor: attribute not found: %s\n", input); + } else { + filter.attr =3D &kunit_attr_list[attr_index]; + } + + if (comma_index) { + input[comma_index] =3D '\0'; + filter.input =3D input + op_index; + input =3D input + new_start_index; + } else { + filter.input =3D input + op_index; + input =3D NULL; + } + + *filters =3D input; + + return filter; +} + +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *cons= t suite, + struct kunit_attr_filter filter, char *action, int *err) +{ + int n =3D 0; + struct kunit_case *filtered, *test_case; + struct kunit_suite *copy; + void *suite_val, *test_val; + bool suite_result, test_result, default_result, result; + + /* Allocate memory for new copy of suite and list of test cases */ + copy =3D kmemdup(suite, sizeof(*copy), GFP_KERNEL); + if (!copy) + return ERR_PTR(-ENOMEM); + + kunit_suite_for_each_test_case(suite, test_case) { n++; } + + filtered =3D kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL); + if (!filtered) { + kfree(copy); + return ERR_PTR(-ENOMEM); + } + + n =3D 0; + + /* Save filtering result on default value */ + default_result =3D filter.attr->filter(filter.attr->attr_default, filter.= input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* Save suite attribute value and filtering result on that value */ + suite_val =3D filter.attr->get_attr((void *)suite, false); + suite_result =3D filter.attr->filter(suite_val, filter.input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* For each test case, save test case if passes filtering. */ + kunit_suite_for_each_test_case(suite, test_case) { + test_val =3D filter.attr->get_attr((void *) test_case, true); + test_result =3D filter.attr->filter(filter.attr->get_attr(test_case, tru= e), + filter.input, err); + if (*err) { + kfree(copy); + kfree(filtered); + return NULL; + } + + /* + * If attribute value of test case is set, filter on that value. + * If not, filter on suite value if set. If not, filter on + * default value. + */ + result =3D false; + if (test_val) { + if (test_result) + result =3D true; + } else if (suite_val) { + if (suite_result) + result =3D true; + } else if (default_result) { + result =3D true; + } + + if (result) { + filtered[n++] =3D *test_case; + } else if (action && strcmp(action, "skip") =3D=3D 0) { + test_case->status =3D KUNIT_SKIPPED; + filtered[n++] =3D *test_case; + } + } + + if (n =3D=3D 0) { + kfree(copy); + kfree(filtered); + return NULL; + } + + copy->test_cases =3D filtered; + + return copy; +} diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c index 12e38a48a5cc..c286ae47435a 100644 --- a/lib/kunit/executor.c +++ b/lib/kunit/executor.c @@ -17,6 +17,9 @@ extern struct kunit_suite * const __kunit_suites_end[]; =20 static char *filter_glob_param; static char *action_param; +static char *filter_param; +static char *filter_action_param; + =20 module_param_named(filter_glob, filter_glob_param, charp, 0); MODULE_PARM_DESC(filter_glob, @@ -27,15 +30,23 @@ MODULE_PARM_DESC(action, ": run the tests like normal\n" "'list' to list test names instead of running them.\n" "'list_attr' to list test names and attributes instead of running them.= \n"); +module_param_named(filter, filter_param, charp, 0); +MODULE_PARM_DESC(filter, + "Filter which KUnit test suites/tests run at boot-time using attributes,= e.g. speed>slow"); +module_param_named(filter_action, filter_action_param, charp, 0); +MODULE_PARM_DESC(filter_action, + "Changes behavior of filtered tests using attributes, valid values are:\= n" + ": do not run filtered tests as normal\n" + "'skip': skip all filtered tests instead so tests will appear in output\= n"); =20 /* glob_match() needs NULL terminated strings, so we need a copy of filter= _glob_param. */ -struct kunit_test_filter { +struct kunit_glob_filter { char *suite_glob; char *test_glob; }; =20 /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty= . */ -static void kunit_parse_filter_glob(struct kunit_test_filter *parsed, +static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed, const char *filter_glob) { const int len =3D strlen(filter_glob); @@ -57,7 +68,7 @@ static void kunit_parse_filter_glob(struct kunit_test_fil= ter *parsed, =20 /* Create a copy of suite with only tests that match test_glob. */ static struct kunit_suite * -kunit_filter_tests(const struct kunit_suite *const suite, const char *test= _glob) +kunit_filter_glob_tests(const struct kunit_suite *const suite, const char = *test_glob) { int n =3D 0; struct kunit_case *filtered, *test_case; @@ -111,12 +122,15 @@ static void kunit_free_suite_set(struct suite_set sui= te_set) =20 static struct suite_set kunit_filter_suites(const struct suite_set *suite_= set, const char *filter_glob, + char *filters, + char *filter_action, int *err) { - int i; - struct kunit_suite **copy, *filtered_suite; + int i, j, k, filter_count; + struct kunit_suite **copy, *filtered_suite, *new_filtered_suite; struct suite_set filtered; - struct kunit_test_filter filter; + struct kunit_glob_filter parsed_glob; + struct kunit_attr_filter *parsed_filters; =20 const size_t max =3D suite_set->end - suite_set->start; =20 @@ -127,17 +141,52 @@ static struct suite_set kunit_filter_suites(const str= uct suite_set *suite_set, return filtered; } =20 - kunit_parse_filter_glob(&filter, filter_glob); - - for (i =3D 0; &suite_set->start[i] !=3D suite_set->end; i++) { - if (!glob_match(filter.suite_glob, suite_set->start[i]->name)) - continue; + if (filter_glob) + kunit_parse_glob_filter(&parsed_glob, filter_glob); =20 - filtered_suite =3D kunit_filter_tests(suite_set->start[i], filter.test_g= lob); - if (IS_ERR(filtered_suite)) { - *err =3D PTR_ERR(filtered_suite); + /* Parse attribute filters */ + if (filters) { + filter_count =3D kunit_get_filter_count(filters); + parsed_filters =3D kcalloc(filter_count + 1, sizeof(*parsed_filters), GF= P_KERNEL); + for (j =3D 0; j < filter_count; j++) + parsed_filters[j] =3D kunit_next_attr_filter(&filters, err); + if (*err) return filtered; + } + + for (i =3D 0; &suite_set->start[i] !=3D suite_set->end; i++) { + filtered_suite =3D suite_set->start[i]; + if (filter_glob) { + if (!glob_match(parsed_glob.suite_glob, filtered_suite->name)) + continue; + filtered_suite =3D kunit_filter_glob_tests(filtered_suite, + parsed_glob.test_glob); + if (IS_ERR(filtered_suite)) { + *err =3D PTR_ERR(filtered_suite); + return filtered; + } + } + if (filter_count) { + for (k =3D 0; k < filter_count; k++) { + new_filtered_suite =3D kunit_filter_attr_tests(filtered_suite, + parsed_filters[k], filter_action, err); + + /* Free previous copy of suite */ + if (k > 0 || filter_glob) + kfree(filtered_suite); + filtered_suite =3D new_filtered_suite; + + if (*err) + return filtered; + if (IS_ERR(filtered_suite)) { + *err =3D PTR_ERR(filtered_suite); + return filtered; + } + if (!filtered_suite) + break; + } } + if (!filtered_suite) continue; =20 @@ -145,8 +194,14 @@ static struct suite_set kunit_filter_suites(const stru= ct suite_set *suite_set, } filtered.end =3D copy; =20 - kfree(filter.suite_glob); - kfree(filter.test_glob); + if (filter_glob) { + kfree(parsed_glob.suite_glob); + kfree(parsed_glob.test_glob); + } + + if (filter_count) + kfree(parsed_filters); + return filtered; } =20 @@ -206,8 +261,9 @@ int kunit_run_all_tests(void) goto out; } =20 - if (filter_glob_param) { - suite_set =3D kunit_filter_suites(&suite_set, filter_glob_param, &err); + if (filter_glob_param || filter_param) { + suite_set =3D kunit_filter_suites(&suite_set, filter_glob_param, + filter_param, filter_action_param, &err); if (err) { pr_err("kunit executor: error filtering suites: %d\n", err); goto out; @@ -223,7 +279,7 @@ int kunit_run_all_tests(void) else pr_err("kunit executor: unknown action '%s'\n", action_param); =20 - if (filter_glob_param) { /* a copy was made of each suite */ + if (filter_glob_param || filter_param) { /* a copy was made of each suite= */ kunit_free_suite_set(suite_set); } =20 diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c index ce6749af374d..d7ab069324b5 100644 --- a/lib/kunit/executor_test.c +++ b/lib/kunit/executor_test.c @@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] =3D { =20 static void parse_filter_test(struct kunit *test) { - struct kunit_test_filter filter =3D {NULL, NULL}; + struct kunit_glob_filter filter =3D {NULL, NULL}; =20 - kunit_parse_filter_glob(&filter, "suite"); + kunit_parse_glob_filter(&filter, "suite"); KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite"); KUNIT_EXPECT_FALSE(test, filter.test_glob); kfree(filter.suite_glob); kfree(filter.test_glob); =20 - kunit_parse_filter_glob(&filter, "suite.test"); + kunit_parse_glob_filter(&filter, "suite.test"); KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite"); KUNIT_EXPECT_STREQ(test, filter.test_glob, "test"); kfree(filter.suite_glob); @@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test) subsuite[1] =3D alloc_fake_suite(test, "suite2", dummy_test_cases); =20 /* Want: suite1, suite2, NULL -> suite2, NULL */ - got =3D kunit_filter_suites(&suite_set, "suite2", &err); + got =3D kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *te= st) subsuite[1] =3D alloc_fake_suite(test, "suite2", dummy_test_cases); =20 /* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */ - got =3D kunit_filter_suites(&suite_set, "suite2.test2", &err); + got =3D kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err); KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); @@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *t= est) subsuite[0] =3D alloc_fake_suite(test, "suite1", dummy_test_cases); subsuite[1] =3D alloc_fake_suite(test, "suite2", dummy_test_cases); =20 - got =3D kunit_filter_suites(&suite_set, "not_found", &err); + got =3D kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err); KUNIT_ASSERT_EQ(test, err, 0); kfree_at_end(test, got.start); /* just in case */ =20 diff --git a/lib/kunit/test.c b/lib/kunit/test.c index 9ee55139ecd1..cb9797fa6303 100644 --- a/lib/kunit/test.c +++ b/lib/kunit/test.c @@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite) kunit_suite_for_each_test_case(suite, test_case) { struct kunit test =3D { .param_value =3D NULL, .param_index =3D 0 }; struct kunit_result_stats param_stats =3D { 0 }; - test_case->status =3D KUNIT_SKIPPED; =20 kunit_init_test(&test, test_case->name, test_case->log); - - if (!test_case->generate_params) { + if (test_case->status =3D=3D KUNIT_SKIPPED) { + /* Test marked as skip */ + test.status =3D KUNIT_SKIPPED; + kunit_update_stats(¶m_stats, test.status); + } else if (!test_case->generate_params) { /* Non-parameterised test. */ + test_case->status =3D KUNIT_SKIPPED; kunit_run_case_catch_errors(suite, test_case, &test); kunit_update_stats(¶m_stats, test.status); } else { /* Get initial param. */ param_desc[0] =3D '\0'; test.param_value =3D test_case->generate_params(NULL, param_desc); + test_case->status =3D KUNIT_SKIPPED; kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT "KTAP version 1\n"); kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A56EC001B0 for ; Wed, 19 Jul 2023 22:24:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229674AbjGSWYW (ORCPT ); Wed, 19 Jul 2023 18:24:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55730 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230501AbjGSWYP (ORCPT ); Wed, 19 Jul 2023 18:24:15 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 047B61FFE for ; Wed, 19 Jul 2023 15:24:01 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-c926075a50cso2134571276.1 for ; Wed, 19 Jul 2023 15:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805441; x=1692397441; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=6OtU6HKaPtInKv7GRD9j2k9uQtdnKiNR0hgZm/fuNuo=; b=UcfGvZbrMKv8SaDQsQZb13+VjYawUvXZCC5QhOFS1NxUjO5KF32QBFDoAkO8eGRVou JlviKJ/oeNVFuXkwprhUSTOYg9boSYrs6OhqZaL9WeYQfjCFAhbPqEgq6WmiQKUujtla AEuduVCppo+lhpZohhYmpxpp6LRvxYkSAgECvfFRCrHNDmV0zv8ZRnM/C88U/3HGcANu cLkAtBLnc0JGVZPDXcuUbfRbpYUmsm22bEecsCuQKCyfGNy/oNnMG0OF8v2fePDCMS5K EwScQYhnU+trDaeKDDKS0DCMDtbE+YGoIKJaz0a0L3NOk4CNXW8luqRn0hawmyDs13z8 SfyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805441; x=1692397441; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=6OtU6HKaPtInKv7GRD9j2k9uQtdnKiNR0hgZm/fuNuo=; b=cfcmaYIFzjW9qpKv2a0papzvbJTL40KN7vmCRUk5VsMc6tV8wHz1yGNNhkr9KTtZxt CHmX7tulYoaaoz2N2Veh1mDNfooiJffq6UW/iKoRmQMhzOqAPO0gH6/siMzHOrcZq3XT CS64rEOHq6waH/kVoW8AfXBLXaqk4tzi4OEbRET5YEIwuTUsFQ+mGATFNYaLpmcZ0f6I gkQP5ClOfAr4pnfa/PYhRL91BHdmDdI2WeUYM8oZ/Ycs2Ap2nMyokt+CDRtl0yqNtD/G loNKraJH0YrXydmTE87az6l2Ap1wYjj0zmGGRf6nXTpm0mLdc4gn1arPVErDjzQYYPiF jE1g== X-Gm-Message-State: ABy/qLZzx75YbtiZbMwoddshg5jykT997jZt4oEXPKFBIunW9oaGS3oh D8PW6pg9WVXIJ3F5SwRKFHcJd6Xsiw== X-Google-Smtp-Source: APBJJlHYrT1refevov/wwGKzGf7E5mcjTJpHif0Z99HcDWLAWKGz3adeGU/phPoprzDimMVENgxrRAMNsw== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:ad1f:0:b0:cf9:3564:33c7 with SMTP id y31-20020a25ad1f000000b00cf9356433c7mr27536ybi.0.1689805440994; Wed, 19 Jul 2023 15:24:00 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:32 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-6-rmoar@google.com> Subject: [PATCH v1 5/9] kunit: tool: Add command line interface to filter and report attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add ability to kunit.py to filter attributes and report a list of tests including attributes without running tests. Add flag "--filter" to input filters on test attributes. Tests will be filtered out if they do not match all inputted filters. Example: --filter speed=3Dslow (This filter would run only the tests that a= re marked as slow) Filters have operations: <, >, <=3D, >=3D, !=3D, and =3D. But note that the characters < and > are often interpreted by the shell, so they may need to be quoted or escaped. Example: --filter "speed>slow" or --filter speed\>slow (This filter would run only the tests that have the speed faster than slow. Additionally, multiple filters can be used. Example: --filter "speed=3Dslow, module!=3Dexample" (This filter would run only the tests that have the speed slow and are not in the "example" module) Note if the user wants to skip filtered tests instead of not running/showing them use the "--filter_action=3Dskip" flag instead. Expose the output of kunit.action=3Dlist option with flag "--list_tests" to output a list of tests. Additionally, add flag "--list_tests_attr" to output a list of tests and their attributes. These flags are useful to see tests and test attributes without needing to run tests. Example of the output of "--list_tests_attr": example example.test_1 example.test_2 # example.test_2.speed: slow This output includes a suite, example, with two test cases, test_1 and test_2. And in this instance test_2 has been marked as slow. Signed-off-by: Rae Moar --- Changes since RFC v2: - Remove --filter_skip flag and replace with --filter_action=3Dskip - Make KUnit executor errors visible in kunit.py and raw_output - Fix up help comments Changes since RFC v1: - Change method for inputting filters to allow for spaces in filtering values - Add option to skip filtered tests instead of not run or show them with the =E2=80=93-filter_skip flag - Separate the new feature to list tests and their attributes into both --list_tests (lists just tests) and --list_tests_attr (lists all) tools/testing/kunit/kunit.py | 70 ++++++++++++++++++++++++-- tools/testing/kunit/kunit_kernel.py | 8 ++- tools/testing/kunit/kunit_parser.py | 11 +++- tools/testing/kunit/kunit_tool_test.py | 39 +++++++------- 4 files changed, 99 insertions(+), 29 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 3905c43369c3..bc74088c458a 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -55,8 +55,12 @@ class KunitExecRequest(KunitParseRequest): build_dir: str timeout: int filter_glob: str + filter: str + filter_action: Optional[str] kernel_args: Optional[List[str]] run_isolated: Optional[str] + list_tests: bool + list_tests_attr: bool =20 @dataclass class KunitRequest(KunitExecRequest, KunitBuildRequest): @@ -102,19 +106,41 @@ def config_and_build_tests(linux: kunit_kernel.LinuxS= ourceTree, =20 def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecReq= uest) -> List[str]: args =3D ['kunit.action=3Dlist'] + + if request.kernel_args: + args.extend(request.kernel_args) + + output =3D linux.run_kernel(args=3Dargs, + timeout=3Drequest.timeout, + filter_glob=3Drequest.filter_glob, + filter=3Drequest.filter, + filter_action=3Drequest.filter_action, + build_dir=3Drequest.build_dir) + lines =3D kunit_parser.extract_tap_lines(output) + # Hack! Drop the dummy TAP version header that the executor prints out. + lines.pop() + + # Filter out any extraneous non-test output that might have gotten mixed = in. + return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)] + +def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitEx= ecRequest) -> Iterable[str]: + args =3D ['kunit.action=3Dlist_attr'] + if request.kernel_args: args.extend(request.kernel_args) =20 output =3D linux.run_kernel(args=3Dargs, timeout=3Drequest.timeout, filter_glob=3Drequest.filter_glob, + filter=3Drequest.filter, + filter_action=3Drequest.filter_action, build_dir=3Drequest.build_dir) lines =3D kunit_parser.extract_tap_lines(output) # Hack! Drop the dummy TAP version header that the executor prints out. lines.pop() =20 # Filter out any extraneous non-test output that might have gotten mixed = in. - return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)] + return lines =20 def _suites_from_test_list(tests: List[str]) -> List[str]: """Extracts all the suites from an ordered list of tests.""" @@ -128,10 +154,18 @@ def _suites_from_test_list(tests: List[str]) -> List[= str]: suites.append(suite) return suites =20 - - def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequ= est) -> KunitResult: filter_globs =3D [request.filter_glob] + if request.list_tests: + output =3D _list_tests(linux, request) + for line in output: + print(line.rstrip()) + return KunitResult(status=3DKunitStatus.SUCCESS, elapsed_time=3D0.0) + if request.list_tests_attr: + attr_output =3D _list_tests_attr(linux, request) + for line in attr_output: + print(line.rstrip()) + return KunitResult(status=3DKunitStatus.SUCCESS, elapsed_time=3D0.0) if request.run_isolated: tests =3D _list_tests(linux, request) if request.run_isolated =3D=3D 'test': @@ -155,6 +189,8 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, req= uest: KunitExecRequest) - args=3Drequest.kernel_args, timeout=3Drequest.timeout, filter_glob=3Dfilter_glob, + filter=3Drequest.filter, + filter_action=3Drequest.filter_action, build_dir=3Drequest.build_dir) =20 _, test_result =3D parse_tests(request, metadata, run_result) @@ -341,6 +377,16 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> = None: nargs=3D'?', default=3D'', metavar=3D'filter_glob') + parser.add_argument('--filter', + help=3D'Filter KUnit tests with attributes, ' + 'e.g. module=3Dexample or speed>slow', + type=3Dstr, + default=3D'') + parser.add_argument('--filter_action', + help=3D'If set to skip, filtered tests will be skipped, ' + 'e.g. --filter_action=3Dskip. Otherwise they will not run.', + type=3Dstr, + choices=3D['skip']) parser.add_argument('--kernel_args', help=3D'Kernel command-line parameters. Maybe be repeated', action=3D'append', metavar=3D'') @@ -350,6 +396,12 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> = None: 'what ran before it.', type=3Dstr, choices=3D['suite', 'test']) + parser.add_argument('--list_tests', help=3D'If set, list all tests that w= ill be ' + 'run.', + action=3D'store_true') + parser.add_argument('--list_tests_attr', help=3D'If set, list all tests a= nd test ' + 'attributes.', + action=3D'store_true') =20 def add_parse_opts(parser: argparse.ArgumentParser) -> None: parser.add_argument('--raw_output', help=3D'If set don\'t parse output fr= om kernel. ' @@ -398,8 +450,12 @@ def run_handler(cli_args: argparse.Namespace) -> None: json=3Dcli_args.json, timeout=3Dcli_args.timeout, filter_glob=3Dcli_args.filter_glob, + filter=3Dcli_args.filter, + filter_action=3Dcli_args.filter_action, kernel_args=3Dcli_args.kernel_args, - run_isolated=3Dcli_args.run_isolated) + run_isolated=3Dcli_args.run_isolated, + list_tests=3Dcli_args.list_tests, + list_tests_attr=3Dcli_args.list_tests_attr) result =3D run_tests(linux, request) if result.status !=3D KunitStatus.SUCCESS: sys.exit(1) @@ -441,8 +497,12 @@ def exec_handler(cli_args: argparse.Namespace) -> None: json=3Dcli_args.json, timeout=3Dcli_args.timeout, filter_glob=3Dcli_args.filter_glob, + filter=3Dcli_args.filter, + filter_action=3Dcli_args.filter_action, kernel_args=3Dcli_args.kernel_args, - run_isolated=3Dcli_args.run_isolated) + run_isolated=3Dcli_args.run_isolated, + list_tests=3Dcli_args.list_tests, + list_tests_attr=3Dcli_args.list_tests_attr) result =3D exec_tests(linux, exec_request) stdout.print_with_timestamp(( 'Elapsed time: %.3fs\n') % (result.elapsed_time)) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kuni= t_kernel.py index 7f648802caf6..0b6488efed47 100644 --- a/tools/testing/kunit/kunit_kernel.py +++ b/tools/testing/kunit/kunit_kernel.py @@ -330,11 +330,15 @@ class LinuxSourceTree: return False return self.validate_config(build_dir) =20 - def run_kernel(self, args: Optional[List[str]]=3DNone, build_dir: str=3D'= ', filter_glob: str=3D'', timeout: Optional[int]=3DNone) -> Iterator[str]: + def run_kernel(self, args: Optional[List[str]]=3DNone, build_dir: str=3D'= ', filter_glob: str=3D'', filter: str=3D'', filter_action: Optional[str]=3D= None, timeout: Optional[int]=3DNone) -> Iterator[str]: if not args: args =3D [] if filter_glob: - args.append('kunit.filter_glob=3D'+filter_glob) + args.append('kunit.filter_glob=3D' + filter_glob) + if filter: + args.append('kunit.filter=3D"' + filter + '"') + if filter_action: + args.append('kunit.filter_action=3D' + filter_action) args.append('kunit.enable=3D1') =20 process =3D self._ops.start(args, build_dir) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kuni= t_parser.py index fbc094f0567e..79d8832c862a 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -212,6 +212,7 @@ KTAP_START =3D re.compile(r'\s*KTAP version ([0-9]+)$') TAP_START =3D re.compile(r'\s*TAP version ([0-9]+)$') KTAP_END =3D re.compile(r'\s*(List of all partitions:|' 'Kernel panic - not syncing: VFS:|reboot: System halted)') +EXECUTOR_ERROR =3D re.compile(r'\s*kunit executor: (.*)$') =20 def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream: """Extracts KTAP lines from the kernel output.""" @@ -242,6 +243,8 @@ def extract_tap_lines(kernel_output: Iterable[str]) -> = LineStream: # remove the prefix, if any. line =3D line[prefix_len:] yield line_num, line + elif EXECUTOR_ERROR.search(line): + yield line_num, line return LineStream(lines=3Disolate_ktap_output(kernel_output)) =20 KTAP_VERSIONS =3D [1] @@ -447,7 +450,7 @@ def parse_diagnostic(lines: LineStream) -> List[str]: Log of diagnostic lines """ log =3D [] # type: List[str] - non_diagnostic_lines =3D [TEST_RESULT, TEST_HEADER, KTAP_START] + non_diagnostic_lines =3D [TEST_RESULT, TEST_HEADER, KTAP_START, TAP_START] while lines and not any(re.match(lines.peek()) for re in non_diagnostic_lines): log.append(lines.pop()) @@ -713,6 +716,11 @@ def parse_test(lines: LineStream, expected_num: int, l= og: List[str], is_subtest: """ test =3D Test() test.log.extend(log) + + # Parse any errors prior to parsing tests + err_log =3D parse_diagnostic(lines) + test.log.extend(err_log) + if not is_subtest: # If parsing the main/top-level test, parse KTAP version line and # test plan @@ -774,6 +782,7 @@ def parse_test(lines: LineStream, expected_num: int, lo= g: List[str], is_subtest: # Don't override a bad status if this test had one reported. # Assumption: no subtests means CRASHED is from Test.__init__() if test.status in (TestStatus.TEST_CRASHED, TestStatus.SUCCESS): + print_log(test.log) test.status =3D TestStatus.NO_TESTS test.add_error('0 tests run!') =20 diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/k= unit_tool_test.py index be35999bb84f..b28c1510be2e 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -597,7 +597,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', timeout=3D300) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', filter=3D'', filte= r_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_run_passes_args_pass(self): @@ -605,7 +605,7 @@ class KUnitMainTest(unittest.TestCase): self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', timeout=3D300) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', filter=3D'', filte= r_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_exec_passes_args_fail(self): @@ -629,7 +629,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run']) self.assertEqual(e.exception.code, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', timeout=3D300) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', filter=3D'', filte= r_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains(' 0 tests run!')) =20 def test_exec_raw_output(self): @@ -670,13 +670,13 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel =3D mock.Mock(return_value=3D[]) kunit.main(['run', '--raw_output', 'filter_glob']) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'filter_glob', timeout= =3D300) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'filter_glob', filter= =3D'', filter_action=3DNone, timeout=3D300) =20 def test_exec_timeout(self): timeout =3D 3453 kunit.main(['exec', '--timeout', str(timeout)]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', timeout=3Dtimeout) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', filter=3D'', filte= r_action=3DNone, timeout=3Dtimeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_run_timeout(self): @@ -684,7 +684,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--timeout', str(timeout)]) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', timeout=3Dtimeout) + args=3DNone, build_dir=3D'.kunit', filter_glob=3D'', filter=3D'', filte= r_action=3DNone, timeout=3Dtimeout) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_run_builddir(self): @@ -692,7 +692,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--build_dir=3D.kunit']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3Dbuild_dir, filter_glob=3D'', timeout=3D300) + args=3DNone, build_dir=3Dbuild_dir, filter_glob=3D'', filter=3D'', filt= er_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_config_builddir(self): @@ -710,7 +710,7 @@ class KUnitMainTest(unittest.TestCase): build_dir =3D '.kunit' kunit.main(['exec', '--build_dir', build_dir]) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3DNone, build_dir=3Dbuild_dir, filter_glob=3D'', timeout=3D300) + args=3DNone, build_dir=3Dbuild_dir, filter_glob=3D'', filter=3D'', filt= er_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_run_kunitconfig(self): @@ -786,7 +786,7 @@ class KUnitMainTest(unittest.TestCase): kunit.main(['run', '--kernel_args=3Da=3D1', '--kernel_args=3Db=3D2']) self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1) self.linux_source_mock.run_kernel.assert_called_once_with( - args=3D['a=3D1','b=3D2'], build_dir=3D'.kunit', filter_glob=3D'', = timeout=3D300) + args=3D['a=3D1','b=3D2'], build_dir=3D'.kunit', filter_glob=3D'', = filter=3D'', filter_action=3DNone, timeout=3D300) self.print_mock.assert_any_call(StrContains('Testing complete.')) =20 def test_list_tests(self): @@ -794,13 +794,11 @@ class KUnitMainTest(unittest.TestCase): self.linux_source_mock.run_kernel.return_value =3D ['TAP version 14', 'i= nit: random output'] + want =20 got =3D kunit._list_tests(self.linux_source_mock, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None,= 'suite')) - + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', N= one, None, 'suite', False, False)) self.assertEqual(got, want) # Should respect the user's filter glob when listing tests. self.linux_source_mock.run_kernel.assert_called_once_with( - args=3D['kunit.action=3Dlist'], build_dir=3D'.kunit', filter_glob=3D'su= ite*', timeout=3D300) - + args=3D['kunit.action=3Dlist'], build_dir=3D'.kunit', filter_glob=3D'su= ite*', filter=3D'', filter_action=3DNone, timeout=3D300) =20 @mock.patch.object(kunit, '_list_tests') def test_run_isolated_by_suite(self, mock_tests): @@ -809,10 +807,10 @@ class KUnitMainTest(unittest.TestCase): =20 # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*',= None, 'suite')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*',= '', None, None, 'suite', False, False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test*= ', timeout=3D300), - mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite2.test= *', timeout=3D300), + mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test*= ', filter=3D'', filter_action=3DNone, timeout=3D300), + mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite2.test= *', filter=3D'', filter_action=3DNone, timeout=3D300), ]) =20 @mock.patch.object(kunit, '_list_tests') @@ -822,13 +820,12 @@ class KUnitMainTest(unittest.TestCase): =20 # Should respect the user's filter glob when listing tests. mock_tests.assert_called_once_with(mock.ANY, - kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None,= 'test')) + kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', N= one, None, 'test', False, False)) self.linux_source_mock.run_kernel.assert_has_calls([ - mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test1= ', timeout=3D300), - mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test2= ', timeout=3D300), - mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite2.test= 1', timeout=3D300), + mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test1= ', filter=3D'', filter_action=3DNone, timeout=3D300), + mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite.test2= ', filter=3D'', filter_action=3DNone, timeout=3D300), + mock.call(args=3DNone, build_dir=3D'.kunit', filter_glob=3D'suite2.test= 1', filter=3D'', filter_action=3DNone, timeout=3D300), ]) =20 - if __name__ =3D=3D '__main__': unittest.main() --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6178AC001B0 for ; Wed, 19 Jul 2023 22:24:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229640AbjGSWYe (ORCPT ); Wed, 19 Jul 2023 18:24:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55860 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230496AbjGSWYX (ORCPT ); Wed, 19 Jul 2023 18:24:23 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A12D926A0 for ; Wed, 19 Jul 2023 15:24:04 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cf0bc5604eeso112542276.2 for ; Wed, 19 Jul 2023 15:24:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805443; x=1692397443; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=sIpA2pCI31/vstZBCLZcYLfoiRSlzDk26WYykbpKRs0=; b=aExkNcdCiwSKCRFZQRazAUw+POtV1ZY1oY9KqCPuUxyVDSTb4m7ZAUwBMuxPLeoQ+E DhWO8r0jkU27P0phJTEl3A8y68Whl8jlHqJ1E+Rx0jpAgawkmZAJ/x1Mx0BmYLFGdHVz eo20uVmy3pITzg5bS2045EFmAZwykYS14y1/Qg4NekGJRL7Bc7WwSYMfLr5at3TNAxJM jf6Y83WMI3JiuA/PuMczSd6rk8iLz2RLaiY09ZDjFbtGXVJphJ/lSohb7khozR5skgCP cceUTwiITld3gT+F4qjy1dNl1t39Ji0Z4QyZdZ1SCY3b7NeGFUJUE25UexBP6Sx+AbQb kBJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805443; x=1692397443; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=sIpA2pCI31/vstZBCLZcYLfoiRSlzDk26WYykbpKRs0=; b=I+A0jUh8pWyX0aVQrT/PX27SmUOGyE9r/JdlGvjXwIKhJlv/7dxaf2+ai3vVoN0kXx qcU5lZik+aZJfh/JgGP0oXHFvFFWKPu/8SsyopVJSQLRt+Ye6FP41TiPhJf/qThSt6X+ YYLEL64GpyNKCRNlh2QhDPjtsumX9R1F0bsk/AxSWKPsA/J7NIa0AHd0YUm9v8Lqgwd6 AS9jBm+TeYSLydAd7/NWtysrN9Y23D46UJRMrW4eQkB1YGsCZHQMRXmC8Jr9awq6HqZ3 OoUdsMA8yPRxzSNYtVF3iClYZD/nfAr+iiOhciJqVCW8oTU/eOLi+ilnDJAEdsJNrrS5 fLOg== X-Gm-Message-State: ABy/qLaeAFajnXXWdbYfb3oKnqKCQoH3MKBMy6uRQcaPOQDIakpeHdJ6 RKAaAzciZ6u8WhOyPC3q0SFxfgV9YQ== X-Google-Smtp-Source: APBJJlGeGls/YLAeNAIERgqeWPE5erbYuzQ303eiV8XxFbMU4b4hvDE8HhmtlZjiF7NRQ61XmgEJ53g1zA== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:26cb:0:b0:c22:c2e0:93bf with SMTP id m194-20020a2526cb000000b00c22c2e093bfmr29785ybm.6.1689805443549; Wed, 19 Jul 2023 15:24:03 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:33 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-7-rmoar@google.com> Subject: [PATCH v1 6/9] kunit: memcpy: Mark tests as slow using test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mark slow memcpy KUnit tests using test attributes. Tests marked as slow are as follows: memcpy_large_test, memmove_test, memmove_large_test, and memmove_overlap_test. These tests were the slowest of the memcpy tests and relatively slower to most other KUnit tests. Most of these tests are already skipped when CONFIG_MEMCPY_SLOW_KUNIT_TEST is not enabled. These tests can now be filtered using the KUnit test attribute filtering feature. Example: --filter "speed>slow". This will run only the tests that have speeds faster than slow. The slow attribute will also be outputted in KTAP. Note: This patch is intended to replace the use of CONFIG_MEMCPY_SLOW_KUNIT_TEST and to potentially deprecate this feature. This patch does not remove the config option but does add a note to the config definition commenting on this future shift. Acked-by: Kees Cook Signed-off-by: Rae Moar --- Changes since RFC v2: - No changes. Changes since RFC v1: - Added note under CONFIG_MEMCPY_SLOW_KUNIT_TEST. lib/Kconfig.debug | 3 +++ lib/memcpy_kunit.c | 8 ++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 550cb967b668..1b3894e861f2 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2701,6 +2701,9 @@ config MEMCPY_SLOW_KUNIT_TEST and bit ranges. These can be very slow, so they are split out as a separate config, in case they need to be disabled. =20 + Note this config option will be replaced by the use of KUnit test + attributes. + config IS_SIGNED_TYPE_KUNIT_TEST tristate "Test is_signed_type() macro" if !KUNIT_ALL_TESTS depends on KUNIT diff --git a/lib/memcpy_kunit.c b/lib/memcpy_kunit.c index 887926f04731..440aee705ccc 100644 --- a/lib/memcpy_kunit.c +++ b/lib/memcpy_kunit.c @@ -551,10 +551,10 @@ static void strtomem_test(struct kunit *test) static struct kunit_case memcpy_test_cases[] =3D { KUNIT_CASE(memset_test), KUNIT_CASE(memcpy_test), - KUNIT_CASE(memcpy_large_test), - KUNIT_CASE(memmove_test), - KUNIT_CASE(memmove_large_test), - KUNIT_CASE(memmove_overlap_test), + KUNIT_CASE_SLOW(memcpy_large_test), + KUNIT_CASE_SLOW(memmove_test), + KUNIT_CASE_SLOW(memmove_large_test), + KUNIT_CASE_SLOW(memmove_overlap_test), KUNIT_CASE(strtomem_test), {} }; --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB506C001DE for ; Wed, 19 Jul 2023 22:24:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230515AbjGSWYk (ORCPT ); Wed, 19 Jul 2023 18:24:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230144AbjGSWY2 (ORCPT ); Wed, 19 Jul 2023 18:24:28 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A6FC26BB for ; Wed, 19 Jul 2023 15:24:06 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-56ff81be091so2065867b3.0 for ; Wed, 19 Jul 2023 15:24:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805445; x=1692397445; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2SH8AkTFG73ufenUbtwahXNeJY1AgRA8n8Wy9Xr37zw=; b=cyNzOp15K8h6pZ+knMLm87QR3n3kAiJvbpZlQJC6RNtA8e57Us52J2RSql7c5obBHG 7Vw74UB7FcA4ZIQ6xanNmf0Rv3jdomJ/QycxcPDkCjXJxLd41WY85/19/avSnGOeGNqv 3bkYYDqbdsbczctPRXLm34M0fhB9EbTNLn7eHEYFHM1l1JGiK8tz3cRE6+g7kjTFOuFv UihO70tkubpccqZTjFYU6O9zdHRAM7lHu6fU/fEYlXfkzPmTwmeQ0X9jQHs5fk8stAie 3/pjUKq1xSXDLa3sTk8Oak78Zk6eKkeHN4k/9A2VfSOTD0ZYDUimY1JB+1cGMZ2wRG6H nzwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805445; x=1692397445; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2SH8AkTFG73ufenUbtwahXNeJY1AgRA8n8Wy9Xr37zw=; b=lk9vLggcbY+V8+SV2IQ6wXR+8umauFLkBazm7jV42mYBojuhblb5o/kKg8dyEgvIjw lRksllA9jiW+52MQSPkqYJD8VplNesTXBd5d95VXHFrYABiXROT7FjyPZFZSvSV7iMPg Jwyt3Vf7Pjx/zhkAsWP8v5IGO+rNzr8cNCd8Hq39WuUP1iOhQEfLdaEcdx5zO3nY6Rto X1RHFgW9dPuX8E2cSG0gbdY6SzUq7SavwNVdf4PNODcfqYTA85MRxXYEQU/tMmC83aWg Ukabl/EBMlOcz4iWCOv2k/APX9xu0As3jvIAbVP5ujgPVuJihmBDLuMyEKBL20m8/qqj OqOw== X-Gm-Message-State: ABy/qLZti8JGJgOpbJbZsCk76UJFXRQ+d5jQ7ugnS6A6w/KA/MpLkJbB wapqMT0buNpz2/8krKZzMwu0cBahOw== X-Google-Smtp-Source: APBJJlGSpivLuIVEbNWEM5SIqiZ8+Uauh2zz2TGgLPCMi1zsdRxCY/kefSVldz4Z6a8C9rVoIwsEQGJfvQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a81:ca48:0:b0:57a:e0b:f63 with SMTP id y8-20020a81ca48000000b0057a0e0b0f63mr3982ywk.7.1689805445487; Wed, 19 Jul 2023 15:24:05 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:34 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-8-rmoar@google.com> Subject: [PATCH v1 7/9] kunit: time: Mark test as slow using test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test attributes. This test ran relatively much slower than most other KUnit tests. By marking this test as slow, the test can now be filtered using the KUnit test attribute filtering feature. Example: --filter "speed>slow". This will run only the tests that have speeds faster than slow. The slow attribute will also be outputted in KTAP. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since RFC v2: - No changes. Changes since RFC v1: - No changes. kernel/time/time_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c index 831e8e779ace..ca058c8af6ba 100644 --- a/kernel/time/time_test.c +++ b/kernel/time/time_test.c @@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *te= st) } =20 static struct kunit_case time_test_cases[] =3D { - KUNIT_CASE(time64_to_tm_test_date_range), + KUNIT_CASE_SLOW(time64_to_tm_test_date_range), {} }; =20 --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F038DC0015E for ; Wed, 19 Jul 2023 22:24:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230511AbjGSWYr (ORCPT ); Wed, 19 Jul 2023 18:24:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230311AbjGSWYf (ORCPT ); Wed, 19 Jul 2023 18:24:35 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0B05270E for ; Wed, 19 Jul 2023 15:24:08 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-cbcffb18afeso114259276.2 for ; Wed, 19 Jul 2023 15:24:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805447; x=1692397447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uQo2ACncIiMoW4snSusP13oUfZPvI4xKN74RE19L4FM=; b=Q4Zh3SLbR44hDRlXzaT+zQxNSWeXzIjU88VDCkpERye0YEfCt+tY3H2rYz4rSQVWgQ Ie4YM59v5zsyfR3DjNd1gf5pSaj67mdRXXkCEdHJV6F3a4L6DZRqavMh3J+Vup9qEBhC xatop62wdduovazhYwCxhf3SqLB4Eg/orY5nyXoOyXGRNudft0ly0O7NeFLC1bqe/gRQ lVbeo3MA/jK9NoXJ7Nj+v0pPfwTkf2DD38Wy60se7hRWjh8hISNceZPCniu7tgb4HbBa nA61r0K6y3mXNppm4Lx16k/zJCLtVRnbUIDj2H1ygCfOPr3ISXjV4yvmGQLy+wHJHnqT JhvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805447; x=1692397447; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uQo2ACncIiMoW4snSusP13oUfZPvI4xKN74RE19L4FM=; b=HWT9huof67ewPUiavRWvtpwFbQbCXKFb3Rs8qY3IeDwgSAsAaODhxcFvg1fgrDLzCv nMzohBRZAaDYc5bjKA8I8r1Ky0y02xVc2YemsnsAa23j/rdC3FPtE6rEylyiqzXlT0l2 jUYWeK4FppIr3ICLi8LkX/FAmjwxBkIePbaVcMTwkZa8R29u9sawy+LdqESO6Glfhcbb qZN4ZQ7r9nA8EjIvJoDZdRdI2vxItycKaAd/zWcIYfO/gu50t7vHjwqXVmMqecu3Y0B4 ZDSRoHrJDaYmmM2FGE1MkS7WULsBF2wrtS0gb6f21TV0OFfmQORpT4acsdXHsyVEi0ZZ alQA== X-Gm-Message-State: ABy/qLa2immQj80JNLhD4IhcehhbVFervbpMbY1iHhwTfXtxxU1Q0Fb8 0YuEM3iO2qqFApPwgJlsOUf/wVCHXQ== X-Google-Smtp-Source: APBJJlF7DoBCoXDEt6Y3RVuCXFCgl14+OrfUPQQdZavytMrt7C26boR7YwNp5KFOYUlvmBG0TgDoMvGqIQ== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a05:6902:1005:b0:cb3:c343:19e5 with SMTP id w5-20020a056902100500b00cb3c34319e5mr39825ybt.2.1689805447446; Wed, 19 Jul 2023 15:24:07 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:35 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-9-rmoar@google.com> Subject: [PATCH v1 8/9] kunit: add tests for filtering attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add four tests to executor_test.c to test behavior of filtering attributes. - parse_filter_attr_test - to test the parsing of inputted filters - filter_attr_test - to test the filtering procedure on attributes - filter_attr_empty_test - to test the behavior when all tests are filtered out - filter_attr_skip_test - to test the configurable filter skip option Signed-off-by: Rae Moar --- Changes since RFC v2: - Change fake suite and test case names. - Add a few ASSERT statements. Changes since RFC v1: - This is a new patch lib/kunit/executor_test.c | 116 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c index d7ab069324b5..01280cb8d451 100644 --- a/lib/kunit/executor_test.c +++ b/lib/kunit/executor_test.c @@ -7,6 +7,7 @@ */ =20 #include +#include =20 static void kfree_at_end(struct kunit *test, const void *to_free); static struct kunit_suite *alloc_fake_suite(struct kunit *test, @@ -108,11 +109,126 @@ static void filter_suites_to_empty_test(struct kunit= *test) "should be empty to indicate no match"); } =20 +static void parse_filter_attr_test(struct kunit *test) +{ + int j, filter_count; + struct kunit_attr_filter *parsed_filters; + char *filters =3D "speed>slow, module!=3Dexample"; + int err =3D 0; + + filter_count =3D kunit_get_filter_count(filters); + KUNIT_EXPECT_EQ(test, filter_count, 2); + + parsed_filters =3D kunit_kcalloc(test, filter_count + 1, sizeof(*parsed_f= ilters), + GFP_KERNEL); + for (j =3D 0; j < filter_count; j++) { + parsed_filters[j] =3D kunit_next_attr_filter(&filters, &err); + KUNIT_ASSERT_EQ_MSG(test, err, 0, "failed to parse filter '%s'", filters= [j]); + } + + KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[0]), "spee= d"); + KUNIT_EXPECT_STREQ(test, parsed_filters[0].input, ">slow"); + + KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[1]), "modu= le"); + KUNIT_EXPECT_STREQ(test, parsed_filters[1].input, "!=3Dexample"); +} + +static struct kunit_case dummy_attr_test_cases[] =3D { + /* .run_case is not important, just needs to be non-NULL */ + { .name =3D "slow", .run_case =3D dummy_test, .module_name =3D "dummy", + .attr.speed =3D KUNIT_SPEED_SLOW }, + { .name =3D "normal", .run_case =3D dummy_test, .module_name =3D "dummy" = }, + {}, +}; + +static void filter_attr_test(struct kunit *test) +{ + struct kunit_suite *subsuite[3] =3D {NULL, NULL}; + struct suite_set suite_set =3D {.start =3D subsuite, .end =3D &subsuite[2= ]}; + struct suite_set got; + int err =3D 0; + + subsuite[0] =3D alloc_fake_suite(test, "normal_suite", dummy_attr_test_ca= ses); + subsuite[1] =3D alloc_fake_suite(test, "slow_suite", dummy_attr_test_case= s); + subsuite[1]->attr.speed =3D KUNIT_SPEED_SLOW; // Set suite attribute + + /* + * Want: normal_suite(slow, normal), slow_suite(slow, normal), + * NULL -> normal_suite(normal), NULL + * + * The normal test in slow_suite is filtered out because the speed + * attribute is unset and thus, the filtering is based on the parent attr= ibute + * of slow. + */ + got =3D kunit_filter_suites(&suite_set, NULL, "speed>slow", NULL, &err); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); + KUNIT_ASSERT_EQ(test, err, 0); + kfree_at_end(test, got.start); + + /* Validate we just have normal_suite */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]); + KUNIT_EXPECT_STREQ(test, got.start[0]->name, "normal_suite"); + KUNIT_ASSERT_EQ(test, got.end - got.start, 1); + + /* Now validate we just have normal test case */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases); + KUNIT_EXPECT_STREQ(test, got.start[0]->test_cases[0].name, "normal"); + KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].name); +} + +static void filter_attr_empty_test(struct kunit *test) +{ + struct kunit_suite *subsuite[3] =3D {NULL, NULL}; + struct suite_set suite_set =3D {.start =3D subsuite, .end =3D &subsuite[2= ]}; + struct suite_set got; + int err =3D 0; + + subsuite[0] =3D alloc_fake_suite(test, "suite1", dummy_attr_test_cases); + subsuite[1] =3D alloc_fake_suite(test, "suite2", dummy_attr_test_cases); + + got =3D kunit_filter_suites(&suite_set, NULL, "module!=3Ddummy", NULL, &e= rr); + KUNIT_ASSERT_EQ(test, err, 0); + kfree_at_end(test, got.start); /* just in case */ + + KUNIT_EXPECT_PTR_EQ_MSG(test, got.start, got.end, + "should be empty to indicate no match"); +} + +static void filter_attr_skip_test(struct kunit *test) +{ + struct kunit_suite *subsuite[2] =3D {NULL}; + struct suite_set suite_set =3D {.start =3D subsuite, .end =3D &subsuite[1= ]}; + struct suite_set got; + int err =3D 0; + + subsuite[0] =3D alloc_fake_suite(test, "suite", dummy_attr_test_cases); + + /* Want: suite(slow, normal), NULL -> suite(slow with SKIP, normal), NULL= */ + got =3D kunit_filter_suites(&suite_set, NULL, "speed>slow", "skip", &err); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start); + KUNIT_ASSERT_EQ(test, err, 0); + kfree_at_end(test, got.start); + + /* Validate we have both the slow and normal test */ + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases); + KUNIT_ASSERT_EQ(test, kunit_suite_num_test_cases(got.start[0]), 2); + KUNIT_EXPECT_STREQ(test, got.start[0]->test_cases[0].name, "slow"); + KUNIT_EXPECT_STREQ(test, got.start[0]->test_cases[1].name, "normal"); + + /* Now ensure slow is skipped and normal is not */ + KUNIT_EXPECT_EQ(test, got.start[0]->test_cases[0].status, KUNIT_SKIPPED); + KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].status); +} + static struct kunit_case executor_test_cases[] =3D { KUNIT_CASE(parse_filter_test), KUNIT_CASE(filter_suites_test), KUNIT_CASE(filter_suites_test_glob_test), KUNIT_CASE(filter_suites_to_empty_test), + KUNIT_CASE(parse_filter_attr_test), + KUNIT_CASE(filter_attr_test), + KUNIT_CASE(filter_attr_empty_test), + KUNIT_CASE(filter_attr_skip_test), {} }; =20 --=20 2.41.0.255.g8b1d071c50-goog From nobody Sat Feb 7 19:08:10 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 848F6C001B0 for ; Wed, 19 Jul 2023 22:24:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229751AbjGSWYz (ORCPT ); Wed, 19 Jul 2023 18:24:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230375AbjGSWYr (ORCPT ); Wed, 19 Jul 2023 18:24:47 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27A701FE1 for ; Wed, 19 Jul 2023 15:24:16 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-583312344e7so1748617b3.1 for ; Wed, 19 Jul 2023 15:24:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689805449; x=1692397449; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NHegRKijxtllrUHzL1SgBpkoHu+y+n5qaq++K8WyWmA=; b=fdqBQx83SLumVnseBEBGY8wv1JHLRmHCegfbTKpsqZwmsnq3Dm9xmh64SvghtVFTxA wHDeybiDWo9RRB9Btl/nyr8aI0jF8V56ADHIVUGP4UOVNQE9HEwaH62CkVlLzWsWdEY7 /ipNN+nLRrWA38Ov0mBVaaORiD0LU/oRw+o2FOjrIvwYfaqhXiiv35+isDfRE0vqyxJg xyLpa4ck/n4pl9RupD+kPhZ8ot8uhEK30tTJALzry75zdcVvQiiEIgtm3DvteNF/roGS ZnV9KjTtZFooTBiwo8z8fEbjtp9qYRnkT6YWvzto1xA/uyuQzoedsmwY2VUORG84r/ya XX6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689805449; x=1692397449; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NHegRKijxtllrUHzL1SgBpkoHu+y+n5qaq++K8WyWmA=; b=HSa3z1rXu95OIWruaTJl9greaDkXEh+gHWmeZAuNerL4nHVQG/Yv7wB6sPnHTGD17h psubeUkvOkxrJiXvQpBKvbkIh1LDUoYz005NgLT+Uv/DPFT+5OVBY86UxStdGAe5o9jb fBuvXzeCDpo1Md734bP2VM5CdIHKMMru+ut9d0dqUZE+8WpSf8UnGEqM0IboF4nmBho/ ahNfrlxsznyBr2lY6Zl4Tibo0Ir8ZtEg1XKPtSb8H/9W+CDQjv09ZfnLfMLqVkkQ2QYh dnM57Y67ZNaJE7xfq7X14IeUFDnE7USBIk+DaY8LevADrAjJStS6TvAxQsIKcXkA9AMW yfqA== X-Gm-Message-State: ABy/qLbTbmL7yECixZ3Mc+h40LBgiXbcLJsX6nCgig1ad12Zw1ztblaD GKuilrp9/FCmsdsUkz9fX8C0gXWdaQ== X-Google-Smtp-Source: APBJJlE8YL/XYJLuvTEOkCulIsrvD7y2CHXTDLudsi1AezvQ7W5NJYo+J0HPPSfJWCFexPio++2FX+ebfg== X-Received: from rmoar-specialist.c.googlers.com ([fda3:e722:ac3:cc00:2b:7d90:c0a8:45d3]) (user=rmoar job=sendgmr) by 2002:a25:ab0e:0:b0:cc5:c7d6:ae13 with SMTP id u14-20020a25ab0e000000b00cc5c7d6ae13mr31839ybi.5.1689805449595; Wed, 19 Jul 2023 15:24:09 -0700 (PDT) Date: Wed, 19 Jul 2023 22:23:36 +0000 In-Reply-To: <20230719222338.259684-1-rmoar@google.com> Mime-Version: 1.0 References: <20230719222338.259684-1-rmoar@google.com> X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230719222338.259684-10-rmoar@google.com> Subject: [PATCH v1 9/9] kunit: Add documentation of KUnit test attributes From: Rae Moar To: shuah@kernel.org, davidgow@google.com, dlatypov@google.com, brendan.higgins@linux.dev Cc: linux-kselftest@vger.kernel.org, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, keescook@chromium.org, linux-hardening@vger.kernel.org, jstultz@google.com, tglx@linutronix.de, sboyd@kernel.org, Rae Moar Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add documentation on the use of test attributes under the section "Tips for Running KUnit Tests" in the KUnit docs. Documentation includes three sections on how to mark tests with attributes, how attributes are reported, and how the user can filter tests using test attributes. Reviewed-by: David Gow Signed-off-by: Rae Moar --- Changes since RFC v2: - Add comment on KUNIT_CASE_SLOW() to documentation. - Add comment on how to expose raw kernel output. - Remove an extra line at the end of file. Changes since RFC v1: - This is a new patch .../dev-tools/kunit/running_tips.rst | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation= /dev-tools/kunit/running_tips.rst index 8e8c493f17d1..766f9cdea0fa 100644 --- a/Documentation/dev-tools/kunit/running_tips.rst +++ b/Documentation/dev-tools/kunit/running_tips.rst @@ -262,3 +262,169 @@ other code executed during boot, e.g. # Reset coverage counters before running the test. $ echo 0 > /sys/kernel/debug/gcov/reset $ modprobe kunit-example-test + + +Test Attributes and Filtering +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D + +Test suites and cases can be marked with test attributes, such as speed of +test. These attributes will later be printed in test output and can be use= d to +filter test execution. + +Marking Test Attributes +----------------------- + +Tests are marked with an attribute by including a ``kunit_attributes`` obj= ect +in the test definition. + +Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes= )`` +macro to define the test case instead of ``KUNIT_CASE(test_name)``. + +.. code-block:: c + + static const struct kunit_attributes example_attr =3D { + .speed =3D KUNIT_VERY_SLOW, + }; + + static struct kunit_case example_test_cases[] =3D { + KUNIT_CASE_ATTR(example_test, example_attr), + }; + +.. note:: + To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name= )``. + This is a helpful macro as the slow attribute is the most commonly used. + +Test suites can be marked with an attribute by setting the "attr" field in= the +suite definition. + +.. code-block:: c + + static const struct kunit_attributes example_attr =3D { + .speed =3D KUNIT_VERY_SLOW, + }; + + static struct kunit_suite example_test_suite =3D { + ..., + .attr =3D example_attr, + }; + +.. note:: + Not all attributes need to be set in a ``kunit_attributes`` object. Unset + attributes will remain uninitialized and act as though the attribute is s= et + to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset. + These unset attributes will not be reported and may act as a default value + for filtering purposes. + +Reporting Attributes +-------------------- + +When a user runs tests, attributes will be present in the raw kernel outpu= t (in +KTAP format). Note that attributes will be hidden by default in kunit.py o= utput +for all passing tests but the raw kernel output can be accessed using the +``--raw_output`` flag. This is an example of how test attributes for test = cases +will be formatted in kernel output: + +.. code-block:: none + + # example_test.speed: slow + ok 1 example_test + +This is an example of how test attributes for test suites will be formatte= d in +kernel output: + +.. code-block:: none + + KTAP version 2 + # Subtest: example_suite + # module: kunit_example_test + 1..3 + ... + ok 1 example_suite + +Additionally, users can output a full attribute report of tests with their +attributes, using the command line flag ``--list_tests_attr``: + +.. code-block:: bash + + kunit.py run "example" --list_tests_attr + +.. note:: + This report can be accessed when running KUnit manually by passing in the + module_param ``kunit.action=3Dlist_attr``. + +Filtering +--------- + +Users can filter tests using the ``--filter`` command line flag when runni= ng +tests. As an example: + +.. code-block:: bash + + kunit.py run --filter speed=3Dslow + + +You can also use the following operations on filters: "<", ">", "<=3D", ">= =3D", +"!=3D", and "=3D". Example: + +.. code-block:: bash + + kunit.py run --filter "speed>slow" + +This example will run all tests with speeds faster than slow. Note that the +characters < and > are often interpreted by the shell, so they may need to= be +quoted or escaped, as above. + +Additionally, you can use multiple filters at once. Simply separate filters +using commas. Example: + +.. code-block:: bash + + kunit.py run --filter "speed>slow, module=3Dkunit_example_test" + +.. note:: + You can use this filtering feature when running KUnit manually by passing + the filter as a module param: ``kunit.filter=3D"speed>slow, speed<=3Dnorm= al"``. + +Filtered tests will not run or show up in the test output. You can use the +``--filter_action=3Dskip`` flag to skip filtered tests instead. These test= s will be +shown in the test output in the test but will not run. To use this feature= when +running KUnit manually, use the module param ``kunit.filter_action=3Dskip`= `. + +Rules of Filtering Procedure +---------------------------- + +Since both suites and test cases can have attributes, there may be conflic= ts +between attributes during filtering. The process of filtering follows these +rules: + +- Filtering always operates at a per-test level. + +- If a test has an attribute set, then the test's value is filtered on. + +- Otherwise, the value falls back to the suite's value. + +- If neither are set, the attribute has a global "default" value, which is= used. + +List of Current Attributes +-------------------------- + +``speed`` + +This attribute indicates the speed of a test's execution (how slow or fast= the +test is). + +This attribute is saved as an enum with the following categories: "normal", +"slow", or "very_slow". The assumed default speed for tests is "normal". T= his +indicates that the test takes a relatively trivial amount of time (less th= an +1 second), regardless of the machine it is running on. Any test slower than +this could be marked as "slow" or "very_slow". + +The macro ``KUNIT_CASE_SLOW(test_name)`` can be easily used to set the spe= ed +of a test case to "slow". + +``module`` + +This attribute indicates the name of the module associated with the test. + +This attribute is automatically saved as a string and is printed for each = suite. +Tests can also be filtered using this attribute. --=20 2.41.0.255.g8b1d071c50-goog