From nobody Sun Jun 28 08:40:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4C59C433F5 for ; Thu, 10 Feb 2022 02:35:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231152AbiBJCfd (ORCPT ); Wed, 9 Feb 2022 21:35:33 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:47424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229764AbiBJCf1 (ORCPT ); Wed, 9 Feb 2022 21:35:27 -0500 Received: from mail-qv1-xf2e.google.com (mail-qv1-xf2e.google.com [IPv6:2607:f8b0:4864:20::f2e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FA2C237D3; Wed, 9 Feb 2022 18:35:29 -0800 (PST) Received: by mail-qv1-xf2e.google.com with SMTP id a28so3579984qvb.10; Wed, 09 Feb 2022 18:35:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=74Mm/lTty9d4MomegbPgFDlHGu1ie2dczUGBXMz/OHk=; b=Ho+oKUDtFcihYlMA4zE6jKR75hXzuXnWAHv602EU6A+VpPp4j7NJ43EQW/jLCsPa6j XNDOjjhbtlSjnLfNpIxtW5B6lkbylyeIB7peFdrGf94VGU1Tt6ClEmmbc/jByADDqIvV wUokrcjTzdLiEh/Lfi1v2PQg+zgaFNQmVbfzePkqb9GRYSMLHwT5zutJFrOsvwA2GdC8 barSLHHiyrWd1oiQ6jUI/6Zy9dTFjj1h4Kh0m7/FT7nlzylW4N24JGGj3G/5pHi6kPV8 pdO87wwKI3JtmRDkpl4I8JHrn8oU2dsaysXfGFyIbdey7RrAPfq/3MbTUm+YciEUwO4S Nirw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=74Mm/lTty9d4MomegbPgFDlHGu1ie2dczUGBXMz/OHk=; b=FROfsYGEn1G0IA49622Wmlh+Jb6ZHxKP/fsV5GM6ZAHD+rUPj7yYjvhDvnRr7uSQws aNKYd96SI3M9Xc6TFYY96/atcP/Uu6XV+pZNnICWJUBbK8KCrzATfF+Vn3lYqoMw0nIv /bUzQq2prPoOI8x9Zimj522noEYuU0TpV3nu4lvA6aws1y51DsWh83hQy4QexOr5YPsM Uo2ulIrjPCR3qHJkVM5keubefFdo4VxTk3LG0CWL/Ka793nUxI7BQhkxA4+/f9ciHK3V vgUchwHYC8BeW5Yb5N4SHWfnRZtPk6569SKTnjEa2xN7i+LEi2qrilR1GV6rEvZ7XTSs l/8w== X-Gm-Message-State: AOAM531g+ijRqrK4SNo+iSoYiX4t/z/WeggsazsKvwiziMqHk/zn0Sva R6aFTAlI9DQuPDMCFxB0IPWQYQ1dCQk= X-Google-Smtp-Source: ABdhPJzFb4Z7vhifhMjgXsUD6r3HcUwbLHNhGJXD+CRXsxLwWFNI/1nHE3oZiiGaqtEYCi5NiklFEQ== X-Received: by 2002:ad4:596d:: with SMTP id eq13mr3577233qvb.131.1644460528277; Wed, 09 Feb 2022 18:35:28 -0800 (PST) Received: from localhost.localdomain (c-67-187-90-124.hsd1.tn.comcast.net. [67.187.90.124]) by smtp.gmail.com with ESMTPSA id h11sm1333374qkp.89.2022.02.09.18.35.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Feb 2022 18:35:27 -0800 (PST) From: frowand.list@gmail.com To: Jonathan Corbet , David Gow , Shuah Khan , Kees Cook , Rae Moar , Tim.Bird@sony.com, Brendan Higgins Cc: rmr167@gmail.com, guillaume.tucker@collabora.com, dlatypov@google.com, kernelci@groups.io, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 1/2] Documentation: dev-tools: clarify KTAP specification wording Date: Wed, 9 Feb 2022 20:35:18 -0600 Message-Id: <20220210023519.3221051-2-frowand.list@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220210023519.3221051-1-frowand.list@gmail.com> References: <20220210023519.3221051-1-frowand.list@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Frank Rowand Add the spec version to the title line. Explain likely source of "Unknown lines". "Unknown lines" in nested tests are optionally indented. Add "Unknown lines" items to differences between TAP & KTAP list Reviewed-by: Tim Bird Reviewed-by: David Gow Reviewed-by: Shuah Khan Signed-off-by: Frank Rowand Reviewed-by: Kees Cook --- Changes since version 3 - Add reviewed by Shuah Changes since version 2 - Add missing quote after word: incorrect - Add Reviewed-by tags Changes since version 1 - Explain likely source of "Unknown lines" - "Unknown line" in nested tests are optionally indented - Add "Unknown lines" items to differences between TAP & KTAP list Documentation/dev-tools/ktap.rst | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/kta= p.rst index 878530cb9c27..dfb3f10a8b2d 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -1,8 +1,8 @@ .. SPDX-License-Identifier: GPL-2.0 =20 -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -The Kernel Test Anything Protocol (KTAP) -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +The Kernel Test Anything Protocol (KTAP), version 1 +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 TAP, or the Test Anything Protocol is a format for specifying test results= used by a number of projects. It's website and specification are found at this = `link @@ -174,6 +174,13 @@ There may be lines within KTAP output that do not foll= ow the format of one of the four formats for lines described above. This is allowed, however, they= will not influence the status of the tests. =20 +This is an important difference from TAP. Kernel tests may print messages +to the system console or a log file. Both of these destinations may conta= in +messages either from unrelated kernel or userspace activity, or kernel +messages from non-test code that is invoked by the test. The kernel code +invoked by the test likely is not aware that a test is in progress and +thus can not print the message as a diagnostic message. + Nested tests ------------ =20 @@ -186,10 +193,13 @@ starting with another KTAP version line and test plan= , and end with the overall result. If one of the subtests fail, for example, the parent test should a= lso fail. =20 -Additionally, all result lines in a subtest should be indented. One level = of +Additionally, all lines in a subtest should be indented. One level of indentation is two spaces: " ". The indentation should begin at the versi= on line and should end before the parent test's result line. =20 +"Unknown lines" are not considered to be lines in a subtest and thus are +allowed to be either indented or not indented. + An example of a test with two nested subtests: =20 .. code-block:: @@ -225,9 +235,11 @@ Major differences between TAP and KTAP -------------------------------------- =20 Note the major differences between the TAP and KTAP specification: -- yaml and json are not recommended in diagnostic messages -- TODO directive not recognized +- yaml and json are not recommended in KTAP diagnostic messages +- TODO directive not recognized in KTAP - KTAP allows for an arbitrary number of tests to be nested +- TAP includes "Unknown lines" in the category of "Anything else" +- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines" =20 The TAP14 specification does permit nested tests, but instead of using ano= ther nested version line, uses a line of the form --=20 Frank Rowand From nobody Sun Jun 28 08:40:01 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7F83FC433EF for ; Thu, 10 Feb 2022 02:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231287AbiBJCfg (ORCPT ); Wed, 9 Feb 2022 21:35:36 -0500 Received: from mxb-00190b01.gslb.pphosted.com ([23.128.96.19]:47434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230357AbiBJCf2 (ORCPT ); Wed, 9 Feb 2022 21:35:28 -0500 Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45251237D2; Wed, 9 Feb 2022 18:35:30 -0800 (PST) Received: by mail-qt1-x82a.google.com with SMTP id z1so3806377qto.3; Wed, 09 Feb 2022 18:35:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oJUtVaUMZZQ3Wf3eYjXBC/gGfRZDVoAst5E9J28F8Fw=; b=Mk9V3Uge8lU26SkqYaHpr1oS8I+T8nFRRG8J9Tr86RzwVjWAlYQhUqE+FIV4/6EOD2 ZqSHtbvRkDaoZxHBb9yVQJrnRTPzmbuTkHamCogMd8FB1IFg2QxlZL4w+476LwMlmjzO Zjyyck1EmKMJFRU//TdVg1fn0NsgBuFIigMdxT+K9Pv2mG5Huig7dbIilITJK3GRmC+j ryF5hNV160hFP+muelCNcwLiF6GtQvdHWDqsiZSujewBnhAcvxkxKjzezXPbCB5PJVSz ENeITHkXJcmsjhWko8yKM+hJEvI3SbJwHMWbesCwbdU8MyI92Q1RvbJBgQjHxg4djXw1 Jy1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oJUtVaUMZZQ3Wf3eYjXBC/gGfRZDVoAst5E9J28F8Fw=; b=6AzYrhrerkXYj1dKKcZYVZUkM6sRJR4TXow+7XYwNzJGprs2bXANAHkZCQJWYrD3Qm 1mVXM8A1PenIczgx2NHm9ITaUdVbl1LkX/IJsc+DHbs3o53ax88Odnw3RK+d3aCs3WVh lDQQAyy2kLV78bY6ypCPU8DG57MqUyFCIHGUqVHb2ZZ5Gdy1+aCSlAKbmFKM4O449/hW iXu0aec49ILjErq7EbKPc0Sf0yhH85cZZQrqxx2mwTTk34oy6+uCVOSCdKOkoo/8Hvm6 dzXuHhNuZY+gXqLgEAzATNq6/CDfvEPbMcJ1qgA3yV+sxvOibMAW532LlI9wz8boxQAd ig+g== X-Gm-Message-State: AOAM530Cown6M7BOFJ7U2RdAI3DjT1bB8FB9LmDJnEOJJUxk9ZGDPiuG I13XKfPRXtedXI3vwZKcPIM= X-Google-Smtp-Source: ABdhPJy9P2O4RKwq83H+EoQrtNUcaisIFb9g4ub306K5gr2IRqBRMRVBztCnm4LMQY8Rgn0es2KDKQ== X-Received: by 2002:a05:622a:50a:: with SMTP id l10mr3496252qtx.679.1644460529457; Wed, 09 Feb 2022 18:35:29 -0800 (PST) Received: from localhost.localdomain (c-67-187-90-124.hsd1.tn.comcast.net. [67.187.90.124]) by smtp.gmail.com with ESMTPSA id h11sm1333374qkp.89.2022.02.09.18.35.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Feb 2022 18:35:29 -0800 (PST) From: frowand.list@gmail.com To: Jonathan Corbet , David Gow , Shuah Khan , Kees Cook , Rae Moar , Tim.Bird@sony.com, Brendan Higgins Cc: rmr167@gmail.com, guillaume.tucker@collabora.com, dlatypov@google.com, kernelci@groups.io, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 2/2] Documentation: dev-tools: fix KTAP specification build warnings Date: Wed, 9 Feb 2022 20:35:19 -0600 Message-Id: <20220210023519.3221051-3-frowand.list@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220210023519.3221051-1-frowand.list@gmail.com> References: <20220210023519.3221051-1-frowand.list@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Frank Rowand Convert "Major differences between TAP and KTAP" from a bullet list to a table. The bullet list was being formatted as a single paragraph. Add missing required argument in code-block directives. --- Table conversion suggested by Shuah. Patch 2/2 not previously reviewed, so Reviewed-by tags not provided. Changes since version 3 - Add this commit (patch 2/2) to the series Signed-off-by: Frank Rowand --- Documentation/dev-tools/ktap.rst | 33 +++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/Documentation/dev-tools/ktap.rst b/Documentation/dev-tools/kta= p.rst index dfb3f10a8b2d..5ee735c6687f 100644 --- a/Documentation/dev-tools/ktap.rst +++ b/Documentation/dev-tools/ktap.rst @@ -68,7 +68,7 @@ Test case result lines Test case result lines indicate the final status of a test. They are required and must have the format: =20 -.. code-block:: +.. code-block:: none =20 [][ # [] []] =20 @@ -117,32 +117,32 @@ separator. =20 Example result lines include: =20 -.. code-block:: +.. code-block:: none =20 ok 1 test_case_name =20 The test "test_case_name" passed. =20 -.. code-block:: +.. code-block:: none =20 not ok 1 test_case_name =20 The test "test_case_name" failed. =20 -.. code-block:: +.. code-block:: none =20 ok 1 test # SKIP necessary dependency unavailable =20 The test "test" was SKIPPED with the diagnostic message "necessary depende= ncy unavailable". =20 -.. code-block:: +.. code-block:: none =20 not ok 1 test # TIMEOUT 30 seconds =20 The test "test" timed out, with diagnostic data "30 seconds". =20 -.. code-block:: +.. code-block:: none =20 ok 5 check return code # rcode=3D0 =20 @@ -202,7 +202,7 @@ allowed to be either indented or not indented. =20 An example of a test with two nested subtests: =20 -.. code-block:: +.. code-block:: none =20 KTAP version 1 1..1 @@ -215,7 +215,7 @@ An example of a test with two nested subtests: =20 An example format with multiple levels of nested testing: =20 -.. code-block:: +.. code-block:: none =20 KTAP version 1 1..2 @@ -234,12 +234,15 @@ An example format with multiple levels of nested test= ing: Major differences between TAP and KTAP -------------------------------------- =20 -Note the major differences between the TAP and KTAP specification: -- yaml and json are not recommended in KTAP diagnostic messages -- TODO directive not recognized in KTAP -- KTAP allows for an arbitrary number of tests to be nested -- TAP includes "Unknown lines" in the category of "Anything else" -- TAP says "Unknown lines" are "incorrect"; KTAP allows "Unknown lines" +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +Feature TAP KTAP +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D +yaml and json in diagnosic message ok not recomm= ended +TODO directive ok not recogn= ized +allows an arbitrary number of tests to be nested no yes +"Unknown lines" are in category of "Anything else" yes no +"Unknown lines" are incorrect allowed +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D =3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D =20 The TAP14 specification does permit nested tests, but instead of using ano= ther nested version line, uses a line of the form @@ -247,7 +250,7 @@ nested version line, uses a line of the form =20 Example KTAP output -------------------- -.. code-block:: +.. code-block:: none =20 KTAP version 1 1..1 --=20 Frank Rowand