From nobody Thu Sep 11 01:30:41 2025 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85581292B54 for ; Wed, 10 Sep 2025 17:00:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757523632; cv=none; b=bA7pvL+3/uS/wRojQrI4rEzdYsNZy3Gcgexdz9a/jNYiATAQPVaadiQsirWN2/9PKBErNbkT2M5pd4EekmNH5BmtvTo+pm0utxTKyrJfNQO5k9dXSajU7rhzCDiNSY5CcOE1hw1CFYU+7bQ57HU3iGf4U/ZDqZ5tGxqnB4Rjl7c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1757523632; c=relaxed/simple; bh=SD5ytb/xU/W1ovRiWGgbNzeHsrA6HUBkcwxbK3JbshE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qv6zB3KtebfVTuOF0JpnQtzuC6aqIyblLSUxkBI6rTBy6jMlqOzOFaldLxQXjWioxGszAvFneZ0yCyOC+z5UO+q4EN9WP0Tg/gbcqO6wNVJsiMptn17sSTMmaVoYvIQrmDyBBqDR3eRWXzNzLs1baWceBLDzbgPUyykqnYiY1yY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=J8xaRbWn; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="J8xaRbWn" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1757523629; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ii1zp41sejenDdX+Fc0ZqaS5dEwSpPRBnvyGP/EInUw=; b=J8xaRbWn0coV1J8szVt8HbWrgMrsz7sYHgn/lg6/9irRRGbBFMP7JXo+Nhvuu1f9NVztvf vwjDISnkV0zrEKyLugytAHiD7E9CdUWPLAi68PqIrwM1teZGHEJTvGQjuyJvZCIed0VRqM +6z1SpZp7TOkSF8Aog8Aw9kKi1wUyaY= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-38-CvBpK9KkO1SOHH-ct7Qx8w-1; Wed, 10 Sep 2025 13:00:23 -0400 X-MC-Unique: CvBpK9KkO1SOHH-ct7Qx8w-1 X-Mimecast-MFC-AGG-ID: CvBpK9KkO1SOHH-ct7Qx8w_1757523621 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D3DF9195419F; Wed, 10 Sep 2025 17:00:20 +0000 (UTC) Received: from fedora.redhat.com (unknown [10.44.22.10]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EBB781956095; Wed, 10 Sep 2025 17:00:14 +0000 (UTC) From: Gabriele Paoloni To: shuah@kernel.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, corbet@lwn.net, linux-doc@vger.kernel.org, gregkh@linuxfoundation.org Cc: linux-mm@kvack.org, safety-architecture@lists.elisa.tech, acarmina@redhat.com, kstewart@linuxfoundation.org, chuckwolber@gmail.com, Gabriele Paoloni Subject: [RFC v2 PATCH 1/3] Documentation: add guidelines for writing testable code specifications Date: Wed, 10 Sep 2025 18:59:58 +0200 Message-ID: <20250910170000.6475-2-gpaoloni@redhat.com> In-Reply-To: <20250910170000.6475-1-gpaoloni@redhat.com> References: <20250910170000.6475-1-gpaoloni@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 The Documentation/doc-guide/kernel-doc.rst chapter describes how to document the code using the kernel-doc format, however it does not specify the criteria to be followed for writing testable specifications; i.e. specifications that can be used to for the semantic description of low level requirements. This patch adds a guideline that defines criteria to formally describe developers=E2=80=99 intent at the function and subfunction level in the form of testable expectations. Signed-off-by: Gabriele Paoloni Signed-off-by: Chuck Wolber Signed-off-by: Kate Stewart --- .../doc-guide/code-specifications.rst | 208 ++++++++++++++++++ Documentation/doc-guide/index.rst | 1 + 2 files changed, 209 insertions(+) create mode 100644 Documentation/doc-guide/code-specifications.rst diff --git a/Documentation/doc-guide/code-specifications.rst b/Documentatio= n/doc-guide/code-specifications.rst new file mode 100644 index 000000000000..dee1b4f089e1 --- /dev/null +++ b/Documentation/doc-guide/code-specifications.rst @@ -0,0 +1,208 @@ +.. title:: How-to write testable code specifications + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +How-to write testable code specifications +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Introduction +------------ +The Documentation/doc-guide/kernel-doc.rst chapter describes how to docume= nt the code using the kernel-doc format, however it does not specify the cr= iteria to be followed for writing testable specifications; i.e. specificati= ons that can be used to for the semantic description of low level requireme= nts. + +This chapter defines criteria to formally describe developers=E2=80=99 int= ent at the function and subfunction level in the form of testable expectati= ons. + +A Virtuous Cycle +---------------- +By adding testable specifications at the function or (where relevant) subf= unction level, one enables the creation of a virtuous cycle when testing is= supplemented with open source code coverage tools like llvm-cov or Gcov. + +As a true reflection of developer intent, code specifications inform the c= reation of a pass/fail tests which can then be assessed in conjunction with= code coverage tools. A failing test may indicate broken code or specificat= ions that fail to capture developer intent. A gap in code coverage may indi= cate missing specifications, unintended functionalities, or insufficient te= st procedure. + +High level goals +---------------- +The code specifications: + +1. Should be maintainable together with the code. +2. Should support hierarchical traceability to allow refinement of SW depe= ndencies (i.e. cross reference critical APIs or data structures). +3. Should describe error conditions and success behaviors. +4. Should describe conditions to be met by the user to avoid unspecified o= r unwanted behaviours. +5. Should allow covering both static and dynamic aspects of the code. +6. Should be compatible with Documentation/doc-guide/kernel-doc.rst. +7. Should support the definition of a test plan (i.e. syntax should enforc= e testability as well as the avoidance of untestable specifications, e.g = =E2=80=9Cfunction_xyz() shall not do something=E2=80=9D). + +Format and Syntax +----------------- +Testable code specifications must be written according to the syntax alrea= dy defined in Documentation/doc-guide/kernel-doc.rst with additional rules = that are described below. + +Function name +~~~~~~~~~~~~~ +``* function_name() - Brief description of function.`` + +This field is to be considered informative and is not part of the testable= specifications. + +Input Arguments +~~~~~~~~~~~~~~~ +Input arguments should be specified in a way that better supports the func= tion=E2=80=99s expectations and Assumptions of Use described below. +They must not contradict the function's expectations and the function=E2= =80=99s prototype. For example:: + + * trace_set_clr_event - enable or disable an event + * @system: system name to match (NULL for any system) + * @event: event name to match (NULL for all events, within system) + * @set: 1 to enable, 0 to disable + * + [...] + * + */ + int trace_set_clr_event(const char *system, const char *event, int set) + +Above all the parameters clearly introduce the impact that they have on th= e code specifications. + +However if below we had:: + + * trace_set_clr_event - enable or disable an event + * @system: system name to match (NULL for any system) + * @event: event name to match (NULL for all events, within system) + * @set: true to enable, false to disable \ + [...] + */ + int trace_set_clr_event(const char *system, const char *event, int set) + +In this case @set would be a bad definition since it is defined as an inte= ger and not as a boolean. + +Longer Description +~~~~~~~~~~~~~~~~~~ +The `Longer Description` section is where the large part of testable code = specifications are defined. The section must be organised as follows:: + + * (Summary Description) provides an introduction of the functionalities + * provided by the function and any informal note. This text does not + * represent any testable code specification. + * + * + * Function's expectations: + * [ID1] - [code expectation] + * + * [ID2] - [code expectation] + * + * [...] + * + * [IDn] - [code expectation] + * + * Assumptions of Use: + * [ID1] - [constraint to be met by the caller] + * + * [ID2] - [constraint to be met by the caller] + * + * [IDn] - [constraint to be met by the caller] + * + +When writing the above section the following rules must be followed: + +* No rules apply to the text above ``Function=E2=80=99s expectations``; su= ch a text does not constitute testable specifications and it is just inform= ative; +* Both ``Function=E2=80=99s expectations`` and ``Assumptions of Use`` must= be listed prefixing each of them with an ID that is unique within this ker= nel-doc header. The reason for this is to facilitate cross-referencing and = traceability between tests and code specifications. +* A Function=E2=80=99s expectation is a testable behavior that the functio= n is expected to comply with (i.e. the function is expected to behave as de= fined in the function=E2=80=99s expectation). +* An Assumption of Use is a pre-condition to be met when invoking the fun= ction being documented. +* Testable functional expectations and Assumptions of Use must be construc= ted according the same rules that apply when writing software requirements: + * Statements should include a subject and a verb, together with other = elements necessary to adequately express the information content of the spe= cifications. + * The verbs are required to use the following keywords: + * For mandatory expectations the verb =E2=80=98shall=E2=80=99 is t= o be used; + * For descriptive text that do not constitute a testable expectati= on verbs such as =E2=80=98are=E2=80=99, =E2=80=98is=E2=80=99, =E2=80=98was= =E2=80=99 are to be used; + * Negative expectations must be avoided (e.g. =E2=80=98shall not= =E2=80=99 must be avoided). +* Statements must be constructed according to the following scheme: + + [**Condition**] [**Subject**] [**Verb/Action**] [**Object**] [**Constr= aint of Action**]. + + In this regard [**Condition**] and [**Constraint of Action**] could be= omitted respectively if the [**Action**] being specified must always happe= n or if there are no constraints associated with it. + +Function Context +~~~~~~~~~~~~~~~~ +The function=E2=80=99s context represents an integral part of Function=E2= =80=99s expectations and Assumptions of Use, where these can further specif= y the information contained in this section. + +Without further specifications this section is to be interpreted as per ex= ample below: + +``* Context: Any context.`` + +The function shall execute in any possible context. + +``* Context: Any context. Takes and releases the RCU lock.`` + +The function shall execute in any possible context. +The function shall take and release the RCU lock. + +``* Context: Any context. Expects to be held by caller.`` + +The function shall execute in any possible context. + is assumed to be held before this function is called. + +``* Context: Process context. May sleep if @gfp flags permit.`` + +The function shall execute in process context. +The function shall sleep according to @gfp flags definitions + +``* Context: Process context. Takes and releases .`` + +The function shall execute in process context. +The function shall take and release . + +``* Context: Softirq or process context. Takes and releases , BH-saf= e.`` + +The function shall execute in process or Softirq context. +The function shall take and release . +The function shall safely execute in bottom half contexts. + +``* Context: Interrupt context.`` + +The function shall execute in interrupt context only. + +It is a good practice to further specify the context specifications as par= t of the Function=E2=80=99s expectation (e.g. at which stage a lock is held= and released) + +Return values +~~~~~~~~~~~~~ +Return values must be written as a multiple line list in the following for= mat:: + +* Return: +* * [value-1] - [condition-1] +* * [value-2] - [condition-2] +* * [...] +* * [value-n] - [condition-n] +* * Any value returned by func-1(), func-2(),...,func-n() + +In such a format ``[value-i]`` must be a clearly identified value or range= of values that is compatible with the function prototype (e.g. for a read(= ) file operation, it is ok to define [value-i] as ``the number of bytes suc= cessfully copied to the user space buffer``). + +``[condition-i]`` must be a condition that can be unambiguously traced bac= k to the ``Function=E2=80=99s expectations`` or ``Context`` defined above; = as part of [condition-i] it is possible to refer to dependencies of invoked= functions or of internal SW or HW states. + +``Any value returned by func-1(), func-2(),...,func-n()`` defines a scenar= io where the current function is directly returning the value of an invoked= function dependency. + +Semantic aspects of testable specifications +------------------------------------------- +From a semantic point of view it is important to document the intended or = expected behavior (from a developer or integrator point of view respectivel= y) in consideration of the different design aspects impacting it. + +Such behavior shall be described in a way that makes it possible to define= test cases unambiguously. \ +To this extent it is important to document design elements impacting the e= xpected behavior and the design elements characterizing the expected behavi= or (and sometimes these can physically overlap); such design elements shall= be limited to the scope of the code being documented, that can range from = a single function to multiple ones depending on the complexity of the overa= ll code. + +**Possible elements impacting the expected behavior** of the code being do= cumented are: + +* Input parameters: parameters passed to the API being documented; +* state variables: global and static data (variables or pointers); +* software dependencies: external SW APIs invoked by the code under analys= is; +* Hardware dependencies: HW design elements directly impacting the behavio= r of the code in scope; +* Firmware dependencies: FW design elements that have an impact on the beh= avior of the API being documented (e.g. DTB or ACPI tables, or runtime serv= ices like SCMI and ACPI AML); +* Compile time configuration parameters: configuration parameters parsed w= hen compiling the Kernel Image; +* Runtime configuration parameters (AKA calibration parameters): parameter= s that can be modified at runtime. + +**Design elements characterizing the expected behavior** of the API being = documented that are in scope according to the above mentioned granularity: + +* API return values, including pointer addresses; +* Input pointers: pointers passed as input parameter to the API being docu= mented; +* state variables: global and static data (variable or pointers); +* Hardware design elements (e.g. HW registers). + +**Testability considerations**: the impact of each of the documented =E2= =80=9Cdesign elements impacting the expected behavior=E2=80=9D must be desc= ribed in terms of effect on the =E2=80=9Cdesign element characterizing the = expected behavior=E2=80=9D and, in doing so, it is important to document al= lowed or not allowed ranges of values, corner cases and error conditions; = so that it is possible to define a meaningful test plan according to differ= ent equivalence classes. + +**Scalability and maintainability considerations**: the described expected= behavior must be limited to the scope of the code under analysis so for ex= ample the Software, Firmware and Hardware dependencies shall be described i= n terms of possible impact on the invoking code deferring further details t= o the respective documentation of these. + +When deciding the scope of the code being documented, the scalability and = maintainability goals must be considered; it does not make sense to embed t= he documentation of multiple complex functions within the kernel-doc header= of the top level function as, doing so, would make it harder to review the= code changes against the documented specifications and/or to extend the sp= ecifications to new functionalities being added. + +The end goal is to build a hierarchical, scalable, maintainable documentat= ion. + +**Feasibility considerations**: Only the =E2=80=9Cmeaningful=E2=80=9D and = =E2=80=9Cuseful=E2=80=9D expected behavior, and the design elements impacti= ng it, shall be considered (e.g. a printk() logging some info may be omitte= d). There are two reasons behind this point: + +1. Specifying the expected behaviour of the code should be done, in princi= ple, in a code agnostic way. So it is not about writing a pseudo-code redun= dant implementation, but rather about defining and documenting the develope= r intent and the integrator=E2=80=99s expectations. +2. When the expected behavior is defined before implementing the code, suc= h an activity is done by experts using a level of detail that is more abstr= act than the code itself and they only refer to aspects that are relevant f= or the design expectations. diff --git a/Documentation/doc-guide/index.rst b/Documentation/doc-guide/in= dex.rst index 24d058faa75c..09e459866442 100644 --- a/Documentation/doc-guide/index.rst +++ b/Documentation/doc-guide/index.rst @@ -9,6 +9,7 @@ How to write kernel documentation =20 sphinx kernel-doc + code-specifications parse-headers contributing maintainer-profile --=20 2.48.1