Rewrite the short document on bisecting kernel bugs. The new text
improves .config handling, brings a mention of 'git bisect skip', and
explains what to do after the bisection finished -- including trying a
revert to verify the result. The rewrite at the same time removes the
unrelated and outdated section on 'Devices not appearing' and replaces
some sentences about bug reporting with a pointer to the document
covering that topic in detail.

This overall brings the approach close to the one in the recently added
text Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst.
As those two texts serve a similar purpose for different audiences,
mention that document in the head of this one and outline when the
other might be the better one to follow.

Signed-off-by: Thorsten Leemhuis <linux@leemhuis.info>
---
v3: (this version)
- incorporate a few minor changes after review feedback from Petr

v2: https://lore.kernel.org/all/fbeae4056ae8174f454c3865bc45633281bb1b31.1723997526.git.linux@leemhuis.info/
- incorporate review feedback from Jonathan
- rename to 'Bisecting a regression'

v1: https://lore.kernel.org/all/10a565e4ebca5e03a2e7abb7ffe1893136471bf9.1722846343.git.linux@leemhuis.info/
- initial version
---
 Documentation/admin-guide/bug-bisect.rst | 208 +++++++++++++++--------
 MAINTAINERS                              |   1 +
 2 files changed, 139 insertions(+), 70 deletions(-)

diff --git a/Documentation/admin-guide/bug-bisect.rst b/Documentation/admin-guide/bug-bisect.rst
index 325c5d0ed34a0a..585630d14581c7 100644
--- a/Documentation/admin-guide/bug-bisect.rst
+++ b/Documentation/admin-guide/bug-bisect.rst
@@ -1,76 +1,144 @@
-Bisecting a bug
-+++++++++++++++
+.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
+.. [see the bottom of this file for redistribution information]
 
-Last updated: 28 October 2016
+======================
+Bisecting a regression
+======================
 
-Introduction
-============
+This document describes how to use a ``git bisect`` to find the source code
+change that broke something -- for example when some functionality stopped
+working after upgrading from Linux 6.0 to 6.1.
 
-Always try the latest kernel from kernel.org and build from source. If you are
-not confident in doing that please report the bug to your distribution vendor
-instead of to a kernel developer.
+The text focuses on the gist of the process. If you are new to bisecting the
+kernel, better follow Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
+instead: it depicts everything from start to finish while covering multiple
+aspects even kernel developers occasionally forget. This includes detecting
+situations early where a bisection would be a waste of time, as nobody would
+care about the result -- for example, because the problem happens after the
+kernel marked itself as 'tainted', occurs in an abandoned version, was already
+fixed, or is caused by a .config change you or your Linux distributor performed.
 
-Finding bugs is not always easy. Have a go though. If you can't find it don't
-give up. Report as much as you have found to the relevant maintainer. See
-MAINTAINERS for who that is for the subsystem you have worked on.
+Finding the change causing a kernel issue using a bisection
+===========================================================
 
-Before you submit a bug report read
-'Documentation/admin-guide/reporting-issues.rst'.
+*Note: the following process assumes you prepared everything for a bisection.
+This includes having a Git clone with the appropriate sources, installing the
+software required to build and install kernels, as well as a .config file stored
+in a safe place (the following example assumes '~/prepared_kernel_.config') to
+use as pristine base at each bisection step; ideally, you have also worked out
+a fully reliable and straight-forward way to reproduce the regression, too.*
 
-Devices not appearing
-=====================
-
-Often this is caused by udev/systemd. Check that first before blaming it
-on the kernel.
-
-Finding patch that caused a bug
-===============================
-
-Using the provided tools with ``git`` makes finding bugs easy provided the bug
-is reproducible.
-
-Steps to do it:
-
-- build the Kernel from its git source
-- start bisect with [#f1]_::
-
-	$ git bisect start
-
-- mark the broken changeset with::
-
-	$ git bisect bad [commit]
-
-- mark a changeset where the code is known to work with::
-
-	$ git bisect good [commit]
-
-- rebuild the Kernel and test
-- interact with git bisect by using either::
-
-	$ git bisect good
-
-  or::
-
-	$ git bisect bad
-
-  depending if the bug happened on the changeset you're testing
-- After some interactions, git bisect will give you the changeset that
-  likely caused the bug.
-
-- For example, if you know that the current version is bad, and version
-  4.8 is good, you could do::
-
-           $ git bisect start
-           $ git bisect bad                 # Current version is bad
-           $ git bisect good v4.8
-
-
-.. [#f1] You can, optionally, provide both good and bad arguments at git
-	 start with ``git bisect start [BAD] [GOOD]``
-
-For further references, please read:
-
-- The man page for ``git-bisect``
-- `Fighting regressions with git bisect <https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html>`_
-- `Fully automated bisecting with "git bisect run" <https://lwn.net/Articles/317154>`_
-- `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_
+* Preparation: start the bisection and tell Git about the points in the history
+  you consider to be working and broken, which Git calls 'good' and 'bad'::
+
+     git bisect start
+     git bisect good v6.0
+     git bisect bad v6.1
+
+  Instead of Git tags like 'v6.0' and 'v6.1' you can specify commit-ids, too.
+
+1. Copy your prepared .config into the build directory and adjust it to the
+   needs of the codebase Git checked out for testing::
+
+     cp ~/prepared_kernel_.config .config
+     make olddefconfig
+
+2. Now build, install, and boot a kernel. This might fail for unrelated reasons,
+   for example, when a compile error happens at the current stage of the
+   bisection a later change resolves. In such cases run ``git bisect skip`` and
+   go back to step 1.
+
+3. Check if the functionality that regressed works in the kernel you just built.
+
+   If it works, execute::
+
+     git bisect good
+
+   If it is broken, run::
+
+     git bisect bad
+
+   Note, getting this wrong just once will send the rest of the bisection
+   totally off course. To prevent having to start anew later you thus want to
+   ensure what you tell Git is correct; it is thus often wise to spend a few
+   minutes more on testing in case your reproducer is unreliable.
+
+   After issuing one of these two commands, Git will usually check out another
+   bisection point and print something like 'Bisecting: 675 revisions left to
+   test after this (roughly 10 steps)'. In that case go back to step 1.
+
+   If Git instead prints something like 'cafecaca0c0dacafecaca0c0dacafecaca0c0da
+   is the first bad commit', then you have finished the bisection. In that case
+   move to the next point below. Note, right after displaying that line Git will
+   show some details about the culprit including its patch description; this can
+   easily fill your terminal, so you might need to scroll up to see the message
+   mentioning the culprit's commit-id.
+
+   In case you missed Git's output, you can always run ``git bisect log`` to
+   print the status: it will show how many steps remain or mention the result of
+   the bisection.
+
+* Recommended complementary task: put the bisection log and the current .config
+  file aside for the bug report; furthermore tell Git to reset the sources to
+  the state before the bisection::
+
+     git bisect log > ~/bisection-log
+     cp .config ~/bisection-config-culprit
+     git bisect reset
+
+* Recommended optional task: try reverting the culprit on top of the latest
+  codebase and check if that fixes your bug; if that is the case, it validates
+  the bisection and enables developers to resolve the regression through a
+  revert.
+
+  To try this, update your clone and check out latest mainline. Then tell Git
+  to revert the change by specifying its commit-id::
+
+     git revert --no-edit cafec0cacaca0
+
+  Git might reject this, for example when the bisection landed on a merge
+  commit. In that case, abandon the attempt. Do the same, if Git fails to revert
+  the culprit on its own because later changes depend on it -- at least unless
+  you bisected a stable or longterm kernel series, in which case you want to
+  check out its latest codebase and try a revert there.
+
+  If a revert succeeds, build and test another kernel to check if reverting
+  resolved your regression.
+
+With that the process is complete. Now report the regression as described by
+Documentation/admin-guide/reporting-issues.rst.
+
+
+Additional reading material
+---------------------------
+
+* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
+  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
+  in the Git documentation.
+* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
+  from kernel developer Nathan Chancellor.
+* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
+* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
+
+..
+   end-of-content
+..
+   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
+   you spot a typo or small mistake, feel free to let him know directly and
+   he'll fix it. You are free to do the same in a mostly informal way if you
+   want to contribute changes to the text -- but for copyright reasons please CC
+   linux-doc@vger.kernel.org and 'sign-off' your contribution as
+   Documentation/process/submitting-patches.rst explains in the section 'Sign
+   your work - the Developer's Certificate of Origin'.
+..
+   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
+   of the file. If you want to distribute this text under CC-BY-4.0 only,
+   please use 'The Linux kernel development community' for author attribution
+   and link this as source:
+   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/bug-bisect.rst
+
+..
+   Note: Only the content of this RST file as found in the Linux kernel sources
+   is available under CC-BY-4.0, as versions of this text that were processed
+   (for example by the kernel's build system) might contain content taken from
+   files which use a more restrictive license.
diff --git a/MAINTAINERS b/MAINTAINERS
index b34385f2e46d92..90c8681d4d311c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6719,6 +6719,7 @@ DOCUMENTATION REPORTING ISSUES
 M:	Thorsten Leemhuis <linux@leemhuis.info>
 L:	linux-doc@vger.kernel.org
 S:	Maintained
+F:	Documentation/admin-guide/bug-bisect.rst
 F:	Documentation/admin-guide/quickly-build-trimmed-linux.rst
 F:	Documentation/admin-guide/reporting-issues.rst
 F:	Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst

base-commit: 8663dd38a7ba5b2bfd2c7b4271e6e63bc0ef1e42
-- 
2.45.0

Thorsten Leemhuis <linux@leemhuis.info> writes:

> Rewrite the short document on bisecting kernel bugs. The new text
> improves .config handling, brings a mention of 'git bisect skip', and
> explains what to do after the bisection finished -- including trying a
> revert to verify the result. The rewrite at the same time removes the
> unrelated and outdated section on 'Devices not appearing' and replaces
> some sentences about bug reporting with a pointer to the document
> covering that topic in detail.
>
> This overall brings the approach close to the one in the recently added
> text Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst.
> As those two texts serve a similar purpose for different audiences,
> mention that document in the head of this one and outline when the
> other might be the better one to follow.
>
> Signed-off-by: Thorsten Leemhuis <linux@leemhuis.info>
> ---
> v3: (this version)
> - incorporate a few minor changes after review feedback from Petr
>
> v2: https://lore.kernel.org/all/fbeae4056ae8174f454c3865bc45633281bb1b31.1723997526.git.linux@leemhuis.info/
> - incorporate review feedback from Jonathan
> - rename to 'Bisecting a regression'
>
> v1: https://lore.kernel.org/all/10a565e4ebca5e03a2e7abb7ffe1893136471bf9.1722846343.git.linux@leemhuis.info/
> - initial version
> ---
>  Documentation/admin-guide/bug-bisect.rst | 208 +++++++++++++++--------
>  MAINTAINERS                              |   1 +
>  2 files changed, 139 insertions(+), 70 deletions(-)

I've applied this, thanks.

jon

On Thu, Aug 22, 2024 at 09:35:33AM +0200, Thorsten Leemhuis wrote:
> Rewrite the short document on bisecting kernel bugs. The new text
> improves .config handling, brings a mention of 'git bisect skip', and
> explains what to do after the bisection finished -- including trying a
> revert to verify the result. The rewrite at the same time removes the
> unrelated and outdated section on 'Devices not appearing' and replaces
> some sentences about bug reporting with a pointer to the document
> covering that topic in detail.
> 
> This overall brings the approach close to the one in the recently added
> text Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst.
> As those two texts serve a similar purpose for different audiences,
> mention that document in the head of this one and outline when the
> other might be the better one to follow.

LGTM, thanks!

Acked-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

On Thu, 22 Aug 2024 09:35:33 +0200
Thorsten Leemhuis <linux@leemhuis.info> wrote:

> Rewrite the short document on bisecting kernel bugs. The new text
> improves .config handling, brings a mention of 'git bisect skip', and
> explains what to do after the bisection finished -- including trying a
> revert to verify the result. The rewrite at the same time removes the
> unrelated and outdated section on 'Devices not appearing' and replaces
> some sentences about bug reporting with a pointer to the document
> covering that topic in detail.
> 
> This overall brings the approach close to the one in the recently added
> text Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst.
> As those two texts serve a similar purpose for different audiences,
> mention that document in the head of this one and outline when the
> other might be the better one to follow.
> 
> Signed-off-by: Thorsten Leemhuis <linux@leemhuis.info>

AFAICS it's flawless now.

Reviewed-by: Petr Tesarik <petr@tesarici.cz>

Petr T

> ---
> v3: (this version)
> - incorporate a few minor changes after review feedback from Petr
> 
> v2: https://lore.kernel.org/all/fbeae4056ae8174f454c3865bc45633281bb1b31.1723997526.git.linux@leemhuis.info/
> - incorporate review feedback from Jonathan
> - rename to 'Bisecting a regression'
> 
> v1: https://lore.kernel.org/all/10a565e4ebca5e03a2e7abb7ffe1893136471bf9.1722846343.git.linux@leemhuis.info/
> - initial version
> ---
>  Documentation/admin-guide/bug-bisect.rst | 208 +++++++++++++++--------
>  MAINTAINERS                              |   1 +
>  2 files changed, 139 insertions(+), 70 deletions(-)
> 
> diff --git a/Documentation/admin-guide/bug-bisect.rst b/Documentation/admin-guide/bug-bisect.rst
> index 325c5d0ed34a0a..585630d14581c7 100644
> --- a/Documentation/admin-guide/bug-bisect.rst
> +++ b/Documentation/admin-guide/bug-bisect.rst
> @@ -1,76 +1,144 @@
> -Bisecting a bug
> -+++++++++++++++
> +.. SPDX-License-Identifier: (GPL-2.0+ OR CC-BY-4.0)
> +.. [see the bottom of this file for redistribution information]
>  
> -Last updated: 28 October 2016
> +======================
> +Bisecting a regression
> +======================
>  
> -Introduction
> -============
> +This document describes how to use a ``git bisect`` to find the source code
> +change that broke something -- for example when some functionality stopped
> +working after upgrading from Linux 6.0 to 6.1.
>  
> -Always try the latest kernel from kernel.org and build from source. If you are
> -not confident in doing that please report the bug to your distribution vendor
> -instead of to a kernel developer.
> +The text focuses on the gist of the process. If you are new to bisecting the
> +kernel, better follow Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
> +instead: it depicts everything from start to finish while covering multiple
> +aspects even kernel developers occasionally forget. This includes detecting
> +situations early where a bisection would be a waste of time, as nobody would
> +care about the result -- for example, because the problem happens after the
> +kernel marked itself as 'tainted', occurs in an abandoned version, was already
> +fixed, or is caused by a .config change you or your Linux distributor performed.
>  
> -Finding bugs is not always easy. Have a go though. If you can't find it don't
> -give up. Report as much as you have found to the relevant maintainer. See
> -MAINTAINERS for who that is for the subsystem you have worked on.
> +Finding the change causing a kernel issue using a bisection
> +===========================================================
>  
> -Before you submit a bug report read
> -'Documentation/admin-guide/reporting-issues.rst'.
> +*Note: the following process assumes you prepared everything for a bisection.
> +This includes having a Git clone with the appropriate sources, installing the
> +software required to build and install kernels, as well as a .config file stored
> +in a safe place (the following example assumes '~/prepared_kernel_.config') to
> +use as pristine base at each bisection step; ideally, you have also worked out
> +a fully reliable and straight-forward way to reproduce the regression, too.*
>  
> -Devices not appearing
> -=====================
> -
> -Often this is caused by udev/systemd. Check that first before blaming it
> -on the kernel.
> -
> -Finding patch that caused a bug
> -===============================
> -
> -Using the provided tools with ``git`` makes finding bugs easy provided the bug
> -is reproducible.
> -
> -Steps to do it:
> -
> -- build the Kernel from its git source
> -- start bisect with [#f1]_::
> -
> -	$ git bisect start
> -
> -- mark the broken changeset with::
> -
> -	$ git bisect bad [commit]
> -
> -- mark a changeset where the code is known to work with::
> -
> -	$ git bisect good [commit]
> -
> -- rebuild the Kernel and test
> -- interact with git bisect by using either::
> -
> -	$ git bisect good
> -
> -  or::
> -
> -	$ git bisect bad
> -
> -  depending if the bug happened on the changeset you're testing
> -- After some interactions, git bisect will give you the changeset that
> -  likely caused the bug.
> -
> -- For example, if you know that the current version is bad, and version
> -  4.8 is good, you could do::
> -
> -           $ git bisect start
> -           $ git bisect bad                 # Current version is bad
> -           $ git bisect good v4.8
> -
> -
> -.. [#f1] You can, optionally, provide both good and bad arguments at git
> -	 start with ``git bisect start [BAD] [GOOD]``
> -
> -For further references, please read:
> -
> -- The man page for ``git-bisect``
> -- `Fighting regressions with git bisect <https://www.kernel.org/pub/software/scm/git/docs/git-bisect-lk2009.html>`_
> -- `Fully automated bisecting with "git bisect run" <https://lwn.net/Articles/317154>`_
> -- `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_
> +* Preparation: start the bisection and tell Git about the points in the history
> +  you consider to be working and broken, which Git calls 'good' and 'bad'::
> +
> +     git bisect start
> +     git bisect good v6.0
> +     git bisect bad v6.1
> +
> +  Instead of Git tags like 'v6.0' and 'v6.1' you can specify commit-ids, too.
> +
> +1. Copy your prepared .config into the build directory and adjust it to the
> +   needs of the codebase Git checked out for testing::
> +
> +     cp ~/prepared_kernel_.config .config
> +     make olddefconfig
> +
> +2. Now build, install, and boot a kernel. This might fail for unrelated reasons,
> +   for example, when a compile error happens at the current stage of the
> +   bisection a later change resolves. In such cases run ``git bisect skip`` and
> +   go back to step 1.
> +
> +3. Check if the functionality that regressed works in the kernel you just built.
> +
> +   If it works, execute::
> +
> +     git bisect good
> +
> +   If it is broken, run::
> +
> +     git bisect bad
> +
> +   Note, getting this wrong just once will send the rest of the bisection
> +   totally off course. To prevent having to start anew later you thus want to
> +   ensure what you tell Git is correct; it is thus often wise to spend a few
> +   minutes more on testing in case your reproducer is unreliable.
> +
> +   After issuing one of these two commands, Git will usually check out another
> +   bisection point and print something like 'Bisecting: 675 revisions left to
> +   test after this (roughly 10 steps)'. In that case go back to step 1.
> +
> +   If Git instead prints something like 'cafecaca0c0dacafecaca0c0dacafecaca0c0da
> +   is the first bad commit', then you have finished the bisection. In that case
> +   move to the next point below. Note, right after displaying that line Git will
> +   show some details about the culprit including its patch description; this can
> +   easily fill your terminal, so you might need to scroll up to see the message
> +   mentioning the culprit's commit-id.
> +
> +   In case you missed Git's output, you can always run ``git bisect log`` to
> +   print the status: it will show how many steps remain or mention the result of
> +   the bisection.
> +
> +* Recommended complementary task: put the bisection log and the current .config
> +  file aside for the bug report; furthermore tell Git to reset the sources to
> +  the state before the bisection::
> +
> +     git bisect log > ~/bisection-log
> +     cp .config ~/bisection-config-culprit
> +     git bisect reset
> +
> +* Recommended optional task: try reverting the culprit on top of the latest
> +  codebase and check if that fixes your bug; if that is the case, it validates
> +  the bisection and enables developers to resolve the regression through a
> +  revert.
> +
> +  To try this, update your clone and check out latest mainline. Then tell Git
> +  to revert the change by specifying its commit-id::
> +
> +     git revert --no-edit cafec0cacaca0
> +
> +  Git might reject this, for example when the bisection landed on a merge
> +  commit. In that case, abandon the attempt. Do the same, if Git fails to revert
> +  the culprit on its own because later changes depend on it -- at least unless
> +  you bisected a stable or longterm kernel series, in which case you want to
> +  check out its latest codebase and try a revert there.
> +
> +  If a revert succeeds, build and test another kernel to check if reverting
> +  resolved your regression.
> +
> +With that the process is complete. Now report the regression as described by
> +Documentation/admin-guide/reporting-issues.rst.
> +
> +
> +Additional reading material
> +---------------------------
> +
> +* The `man page for 'git bisect' <https://git-scm.com/docs/git-bisect>`_ and
> +  `fighting regressions with 'git bisect' <https://git-scm.com/docs/git-bisect-lk2009.html>`_
> +  in the Git documentation.
> +* `Working with git bisect <https://nathanchance.dev/posts/working-with-git-bisect/>`_
> +  from kernel developer Nathan Chancellor.
> +* `Using Git bisect to figure out when brokenness was introduced <http://webchick.net/node/99>`_.
> +* `Fully automated bisecting with 'git bisect run' <https://lwn.net/Articles/317154>`_.
> +
> +..
> +   end-of-content
> +..
> +   This document is maintained by Thorsten Leemhuis <linux@leemhuis.info>. If
> +   you spot a typo or small mistake, feel free to let him know directly and
> +   he'll fix it. You are free to do the same in a mostly informal way if you
> +   want to contribute changes to the text -- but for copyright reasons please CC
> +   linux-doc@vger.kernel.org and 'sign-off' your contribution as
> +   Documentation/process/submitting-patches.rst explains in the section 'Sign
> +   your work - the Developer's Certificate of Origin'.
> +..
> +   This text is available under GPL-2.0+ or CC-BY-4.0, as stated at the top
> +   of the file. If you want to distribute this text under CC-BY-4.0 only,
> +   please use 'The Linux kernel development community' for author attribution
> +   and link this as source:
> +   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/Documentation/admin-guide/bug-bisect.rst
> +
> +..
> +   Note: Only the content of this RST file as found in the Linux kernel sources
> +   is available under CC-BY-4.0, as versions of this text that were processed
> +   (for example by the kernel's build system) might contain content taken from
> +   files which use a more restrictive license.
> diff --git a/MAINTAINERS b/MAINTAINERS
> index b34385f2e46d92..90c8681d4d311c 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -6719,6 +6719,7 @@ DOCUMENTATION REPORTING ISSUES
>  M:	Thorsten Leemhuis <linux@leemhuis.info>
>  L:	linux-doc@vger.kernel.org
>  S:	Maintained
> +F:	Documentation/admin-guide/bug-bisect.rst
>  F:	Documentation/admin-guide/quickly-build-trimmed-linux.rst
>  F:	Documentation/admin-guide/reporting-issues.rst
>  F:	Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst
> 
> base-commit: 8663dd38a7ba5b2bfd2c7b4271e6e63bc0ef1e42