From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9316B1DFC5; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=Cfw0c3eHN6tzgQlhgxBdrsXZS/ROXdMnp4Holu+vvivqGkn8Yn7X03ppbR7RxlimBtYRh9glp1mtJ+cvH6UZddKaE0olVRgcqyZFvooS6UfH65QFidwYeYpFnWoyaPL1L5G4Qs51q9ymHP9XL40oGMRPKqofFM29kSC9zE3pYOI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=1Uigzs4n8T6q5NRH0PxQNUhRU3AhEgvWbprYJabzC1A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=eR3KvOmnPaJZQjVRjn/UqC8pIvyH2whCzV+K58iNyT799w7PpvHGt8G9HeH+M7pruSCuEbxTCP1G/XP9w9BXnw3HiEbXPqqTbXXN2bToukhDc7FaUW3a47xeNp/2i6XrpcBUKIi8sGT4YrQkLpK1Xzwxm5MfJDJBITPzZhDkKa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=ZUPupMjS; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="ZUPupMjS" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=AkZuZ+BSWwSPUu4Fb4xPXFyfctmZKJa+cg2LCD0xmwA=; t=1712647855; x=1713079855; b=ZUPupMjS+lcqwt/rz53gFXVikghL5UI3SfYJVHHPbnXNNJffBNFA7JsZCo4ut cvpIW5tBEJHVW5n7a/lv8m2aBhLywCwWWTeKbaLNfzuT4CUGrcmytr5IlNKAMywRyTUmspcG9YgVa M1hFF7ePrrW/ijuKw9rD9+omfoFu+f/s/2aP3eUGbT2pkACWYpgOGFdKA4+2XCFb7WDXG4tpDTfC5 toanSnSS6J2XAlFppZCYWD26045KuGV3djx7RqFZts9mgpNb0eJyn1oEiarSdO1J+bRs9fGnTWooe gDxMTsH1dDdakwWiFTLOPBhkUc/vAigWyJldmejSzlQtku/RZw==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wQ-0006Ld-8X; Tue, 09 Apr 2024 09:30:50 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 1/6] docs: verify/bisect: use git switch, tag kernel, and various fixes Date: Tue, 9 Apr 2024 09:30:44 +0200 Message-ID: <85029aa004447b0eeb5043fb014630f2acafacec.1712647788.git.linux@leemhuis.info> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wQ-0006Ld-8X Content-Type: text/plain; charset="utf-8" Various small improvements and fixes: * Use the more modern 'git switch' instead of 'git checkout', which makes it more obvious what's happening (among others due to the --discard-changes parameter that is more clear than --force). * Provide a hint how a mainline version number and one from a stable series look like. * When trying to validate the bisection result with a revert, add a special tag to facilitate the identification. * Sync version numbers used in various examples for consistency: stick to 6.0.13, 6.0.15, and 6.1.5. * Fix a few typos and oddities. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 117 ++++++++++-------- 1 file changed, 67 insertions(+), 50 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index d3504826f40154..c999e40c79ab7f 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -38,8 +38,8 @@ aspects, all of which might be essential in your present = case.]* **In case you want to check if a bug is present in code currently supporte= d by developers**, execute just the *preparations* and *segment 1*; while doing= so, consider the newest Linux kernel you regularly use to be the 'working' ker= nel. -In the following example that's assumed to be 6.0.13, which is why the sou= rces -of 6.0 will be used to prepare the .config file. +In the following example that's assumed to be 6.0, which is why its sources +will be used to prepare the .config file. =20 **In case you face a regression**, follow the steps at least till the end = of *segment 2*. Then you can submit a preliminary report -- or continue with @@ -61,7 +61,7 @@ will be considered the 'good' release and used to prepare= the .config file. cd ~/linux/ git remote add -t master stable \ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git - git checkout --detach v6.0 + git switch --detach v6.0 # * Hint: if you used an existing clone, ensure no stale .config is ar= ound. make olddefconfig # * Ensure the former command picked the .config of the 'working' kern= el. @@ -87,7 +87,7 @@ will be considered the 'good' release and used to prepare= the .config file. a) Checking out latest mainline code:: =20 cd ~/linux/ - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master =20 b) Build, install, and boot a kernel:: =20 @@ -125,7 +125,7 @@ will be considered the 'good' release and used to prepa= re the .config file. a) Start by checking out the sources of the 'good' version:: =20 cd ~/linux/ - git checkout --force --detach v6.0 + git switch --discard-changes --detach v6.0 =20 b) Build, install, and boot a kernel as described earlier in *segment 1, section b* -- just feel free to skip the 'du' commands, as you have a= rough @@ -157,11 +157,12 @@ will be considered the 'good' release and used to pre= pare the .config file. works with the newly built kernel. If it does, tell Git by executing ``git bisect good``; if it does not, run ``git bisect bad`` instead. =20 - All three commands will make Git checkout another commit; then re-exe= cute + All three commands will make Git check out another commit; then re-ex= ecute this step (e.g. build, install, boot, and test a kernel to then tell = Git the outcome). Do so again and again until Git shows which commit broke things. If you run short of disk space during this process, check the - "Supplementary tasks" section below. + section 'Supplementary tasks: cleanup during and after the process' + below. =20 d) Once your finished the bisection, put a few things away:: =20 @@ -172,12 +173,15 @@ will be considered the 'good' release and used to pre= pare the .config file. =20 e) Try to verify the bisection result:: =20 - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master git revert --no-edit cafec0cacaca0 + cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca= 0-reverted' =20 This is optional, as some commits are impossible to revert. But if the second command worked flawlessly, build, install, and boot one more ke= rnel - kernel, which should not show the regression. + kernel; just this time skip the first command copying the base .config= file + over, as that already has been taken care off. =20 * **Supplementary tasks**: cleanup during and after the process. =20 @@ -208,7 +212,7 @@ Step-by-step guide on how to verify bugs and bisect reg= ressions =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 This guide describes how to set up your own Linux kernels for investigatin= g bugs -or regressions you intent to report. How far you want to follow the instru= ctions +or regressions you intend to report. How far you want to follow the instru= ctions depends on your issue: =20 Execute all steps till the end of *segment 1* to **verify if your kernel p= roblem @@ -240,12 +244,17 @@ to get things rolling again. For further details on how to report Linux kernel issues or regressions ch= eck out Documentation/admin-guide/reporting-issues.rst, which works in conjunc= tion with this document. It among others explains why you need to verify bugs w= ith -the latest 'mainline' kernel, even if you face a problem with a kernel fro= m a -'stable/longterm' series; for users facing a regression it also explains t= hat -sending a preliminary report after finishing segment 2 might be wise, as t= he -regression and its culprit might be known already. For further details on -what actually qualifies as a regression check out -Documentation/admin-guide/reporting-regressions.rst. +the latest 'mainline' kernel (e.g. versions like 6.0, 6.1-rc1, or 6.1-rc6), +even if you face a problem with a kernel from a 'stable/longterm' series +(say 6.0.13). + +For users facing a regression that document also explains why sending a +preliminary report after segment 2 might be wise, as the regression and its +culprit might be known already. For further details on what actually quali= fies +as a regression check out Documentation/admin-guide/reporting-regressions.= rst. + +If you run into any problems while following this guide or have ideas how = to +improve it, :ref:`please let the kernel developers know `. =20 .. _introprep_bissbs: =20 @@ -286,7 +295,7 @@ Preparations: set up everything to build your own kerne= ls Do you follow this guide to verify if a bug is present in the code devel= opers care for? Then consider the mainline release your 'working' kernel (the = newest one you regularly use) is based on to be the 'good' version; if your 'wo= rking' - kernel for example is 6.0.11, then your 'good' kernel is 6.0. + kernel for example is 6.0.13, then your 'good' kernel is 6.0. =20 In case you face a regression, it depends on the version range where the regression was introduced: @@ -295,14 +304,14 @@ Preparations: set up everything to build your own ker= nels 6.1-rc1? Then henceforth regard 6.0 as the last known 'good' version and 6.1-rc1 as the first 'bad' one. =20 - * Some function stopped working when updating from 6.0.11 to 6.1.4? Then= for - the time being consider 6.0 as the last 'good' version and 6.1.4 as + * Some function stopped working when updating from 6.0.13 to 6.1.5? Then= for + the time being consider 6.0 as the last 'good' version and 6.1.5 as the 'bad' one. Note, at this point it is merely assumed that 6.0 is fi= ne; this assumption will be checked in segment 2. =20 - * A feature you used in 6.0.11 does not work at all or worse in 6.1.13? = In + * A feature you used in 6.0.13 does not work at all or worse in 6.1.15? = In that case you want to bisect within a stable/longterm series: consider - 6.0.11 as the last known 'good' version and 6.0.13 as the first 'bad' + 6.0.13 as the last known 'good' version and 6.0.15 as the first 'bad' one. Note, in this case you still want to compile and test a mainline = kernel as explained in segment 1: the outcome will determine if you need to r= eport your issue to the regular developers or the stable team. @@ -367,7 +376,7 @@ Preparations: set up everything to build your own kerne= ls * Start preparing a kernel build configuration (the '.config' file). =20 Before doing so, ensure you are still running the 'working' kernel an ea= rlier - step told you to boot; if you are unsure, check the current kernel relea= se + step told you to boot; if you are unsure, check the current kernelrelease identifier using ``uname -r``. =20 Afterwards check out the source code for the version earlier established= as @@ -375,7 +384,7 @@ Preparations: set up everything to build your own kerne= ls the version number in this and all later Git commands needs to be prefix= ed with a 'v':: =20 - git checkout --detach v6.0 + git switch --discard-changes --detach v6.0 =20 Now create a build configuration file:: =20 @@ -505,7 +514,7 @@ be a waste of time. [:ref:`details`] * Check out the latest Linux codebase:: =20 cd ~/linux/ - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master =20 [:ref:`details`] =20 @@ -617,7 +626,7 @@ be a waste of time. [:ref:`details`] cd ~/linux/ git remote set-branches --add stable linux-6.0.y git fetch stable - git checkout --force --detach linux-6.0.y + git switch --discard-changes --detach linux-6.0.y =20 Now use the checked out code to build and install another kernel using t= he commands the earlier steps already described in more detail:: @@ -669,7 +678,7 @@ otherwise would be a waste of time. [:ref:`details`] 'good' (once again assumed to be 6.0 here):: =20 cd ~/linux/ - git checkout --detach v6.0 + git switch --discard-changes --detach v6.0 =20 Now use the checked out code to configure, build, and install another ke= rnel using the commands the previous subsection explained in more detail:: @@ -703,7 +712,7 @@ Segment 3: perform the bisection and validate the result With all the preparations and precaution builds taken care of, you are now= ready to begin the bisection. This will make you build quite a few kernels -- us= ually about 15 in case you encountered a regression when updating to a newer ser= ies -(say from 6.0.11 to 6.1.3). But do not worry, due to the trimmed build +(say from 6.0.13 to 6.1.5). But do not worry, due to the trimmed build configuration created earlier this works a lot faster than many people ass= ume: overall on average it will often just take about 10 to 15 minutes to compi= le each kernel on commodity x86 machines. @@ -745,7 +754,7 @@ each kernel on commodity x86 machines. If compilation fails for some reason, run ``git bisect skip`` and restart executing the stack of commands from the beginning. =20 - In case you skipped the "test latest codebase" step in the guide, check = its + In case you skipped the 'test latest codebase' step in the guide, check = its description as for why the 'df [...]' and 'make -s kernelrelease [...]' commands are here. =20 @@ -823,16 +832,16 @@ each kernel on commodity x86 machines. Begin by checking out the latest codebase depending on the range you bis= ected: =20 * Did you face a regression within a stable/longterm series (say between - 6.0.11 and 6.0.13) that does not happen in mainline? Then check out the + 6.0.13 and 6.0.15) that does not happen in mainline? Then check out the latest codebase for the affected series like this:: =20 git fetch stable - git checkout --force --detach linux-6.0.y + git switch --discard-changes --detach linux-6.0.y =20 * In all other cases check out latest mainline:: =20 git fetch mainline - git checkout --force --detach mainline/master + git switch --discard-changes --detach mainline/master =20 If you bisected a regression within a stable/longterm series that also happens in mainline, there is one more thing to do: look up the mainli= ne @@ -846,21 +855,27 @@ each kernel on commodity x86 machines. =20 git revert --no-edit cafec0cacaca0 =20 - If that fails, give up trying and move on to the next step. But if it wo= rks, - build a kernel again using the familiar command sequence:: + If that fails, give up trying and move on to the next step; if it works, + adjust the tag to facilitate the identification and prevent accidentally + overwriting another kernel:: =20 cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0-r= everted' + + Build a kernel using the familiar command sequence, just without copying= the + the base .config over:: + make olddefconfig && - make -j $(nproc --all) && + make -j $(nproc --all) # * Check if the free space suffices holding another kernel: df -h /boot/ /lib/modules/ sudo make modules_install command -v installkernel && sudo make install - Make -s kernelrelease | tee -a ~/kernels-built + make -s kernelrelease | tee -a ~/kernels-built reboot =20 - Now check one last time if the feature that made you perform a bisection= work - with that kernel. + Now check one last time if the feature that made you perform a bisection= works + with that kernel: if everything went well, it should not show the regres= sion. =20 [:ref:`details`] =20 @@ -934,10 +949,12 @@ This concludes the step-by-step guide. =20 Did you run into trouble following any of the above steps not cleared up b= y the reference section below? Did you spot errors? Or do you have ideas how to -improve the guide? Then please take a moment and let the maintainer of this +improve the guide? + +If any of that applies, please take a moment and let the maintainer of this document know by email (Thorsten Leemhuis ), ideally = while CCing the Linux docs mailing list (linux-doc@vger.kernel.org). Such feedba= ck is -vital to improve this document further, which is in everybody's interest, = as it +vital to improve this text further, which is in everybody's interest, as it will enable more people to master the task described here -- and hopefully= also improve similar guides inspired by this one. =20 @@ -1059,18 +1076,18 @@ Bisection range =20 Establishing the range of commits to be checked is mostly straightforward, except when a regression occurred when switching from a release of one sta= ble -series to a release of a later series (e.g. from 6.0.11 to 6.1.4). In that= case +series to a release of a later series (e.g. from 6.0.13 to 6.1.5). In that= case Git will need some hand holding, as there is no straight line of descent. =20 That's because with the release of 6.0 mainline carried on to 6.1 while the stable series 6.0.y branched to the side. It's therefore theoretically pos= sible -that the issue you face with 6.1.4 only worked in 6.0.11, as it was fixed = by a +that the issue you face with 6.1.5 only worked in 6.0.13, as it was fixed = by a commit that went into one of the 6.0.y releases, but never hit mainline or= the 6.1.y series. Thankfully that normally should not happen due to the way the stable/longterm maintainers maintain the code. It's thus pretty safe to as= sume 6.0 as a 'good' kernel. That assumption will be tested anyway, as that ker= nel will be built and tested in the segment '2' of this guide; Git would force= you -to do this as well, if you tried bisecting between 6.0.11 and 6.1.13. +to do this as well, if you tried bisecting between 6.0.13 and 6.1.15. =20 [:ref:`back to step-by-step guide `] =20 @@ -1117,7 +1134,7 @@ These commands install a few packages that are often,= but not always needed. You for example might want to skip installing the development headers for ncur= ses, which you will only need in case you later might want to adjust the kernel= build configuration using make the targets 'menuconfig' or 'nconfig'; likewise o= mit -the headers of Qt6 is you do not plan to adjust the .config using 'xconfig= '. +the headers of Qt6 if you do not plan to adjust the .config using 'xconfig= '. =20 You furthermore might need additional libraries and their development head= ers for tasks not covered in this guide -- for example when building utilities= from @@ -1184,7 +1201,7 @@ First, execute the following command to retrieve the = latest mainline codebase:: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git =20 Now deepen your clone's history to the second predecessor of the mainline -release of your 'good' version. In case the latter are 6.0 or 6.0.11, 5.19= would +release of your 'good' version. In case the latter are 6.0 or 6.0.13, 5.19= would be the first predecessor and 5.18 the second -- hence deepen the history u= p to that version:: =20 @@ -1490,7 +1507,7 @@ highly recommended for these reasons: =20 Your report might be ignored if you send it to the wrong party -- and ev= en when you get a reply there is a decent chance that developers tell you to - evaluate which of the two cases it is before they take a closer look. + evaluate which of the two cases it is before they take a closer look. =20 [:ref:`back to step-by-step guide `] =20 @@ -1552,8 +1569,8 @@ by modifying your search terms or using another line = from the error messages. =20 In the end, most issues you run into have likely been encountered and reported by others already. That includes issues where the cause is not yo= ur -system, but lies in the code. If you run into one of those, you might thus= find a -solution (e.g. a patch) or workaround for your issue, too. +system, but lies in the code. If you run into one of those, you might thus= find +a solution (e.g. a patch) or workaround for your issue, too. =20 Package your kernel up ~~~~~~~~~~~~~~~~~~~~~~ @@ -1767,8 +1784,8 @@ multitude of reasons why this might happen. Some idea= s where to look: =20 Note, if you found and fixed problems with the .config file, you want to u= se it to build another kernel from the latest codebase, as your earlier tests wi= th -mainline and the latest version from an affected stable/longterm series we= re most -likely flawed. +mainline and the latest version from an affected stable/longterm series we= re +most likely flawed. =20 [:ref:`back to step-by-step guide `] =20 @@ -1911,7 +1928,7 @@ Now remove the boot entry for the kernel from your bo= otloader's configuration; the steps to do that vary quite a bit between Linux distributions. =20 Note, be careful with wildcards like '*' when deleting files or directories -for kernels manually: you might accidentally remove files of a 6.0.11 kern= el +for kernels manually: you might accidentally remove files of a 6.0.13 kern= el when all you want is to remove 6.0 or 6.0.1. =20 [:ref:`back to step-by-step guide `] --=20 2.44.0 From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 931E37BB1A; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=oHBRsCZ5JfNcjDKG1TTAi6CKNGONJ42+8NtgQ+uBLwvOmo16cEOIdYODl8F2dowq6wfIOpYT2iT8Si0pePSTIy+Wa8BV9I60aym19ZBVZTCeSCvfrRr/vlCPVUQ36hdjQxRtU6gTjBE2IF5GOaoIi35dfJKOWWcsxo4lwVtga5k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=pE/Y96b4Lmsy+ebjj+zVv1h1X+yVjer7tDax3IgAqLI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=JNRB5h9qjvclyul5KRj+cz7UPb3s+wbA994ZKl2hd2oaxO29Fqtre/4ALALzGn7xItiiVYwRy4jp6KWzNWkSN72LHIeN6sF86y8mwJl6vA+MPANFx0/Mma+LwI9U2MnwDQcKqhSZQRRHfHdBDBpNGGxc3ji17qxNHK2Tj8tQV4k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=PtPjQ1rS; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="PtPjQ1rS" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=mWeps612SQJGKPCMCZ4KyBqfdzQNYoeA60QuHY4EFvU=; t=1712647855; x=1713079855; b=PtPjQ1rSUV2a8BqdueeOhIn535xJFcQ9IPB9ylAfTj+iTWcCG4K2+256HX+NA OIuaJjtmV34gBHp4Xhll5MrJY0VOrdhvWsj22/Rd8AnM7MUHS4rlOm7HdZ23GXJyVPqdHhXtQDaeK X8XuuMltFvAXqAtTw0MYbnthTihyB5Jp8ZfuJGxNo4BbLc/M0AhuzM/WURLIFaE8B8m2jGU+Zucor rMmdPedAGKIJ7Ld2Fh+BGXKWUasixw6DqKGMzD39lKk9WeNamUT80FbgvA1/weUEJw88DAZ5Q5HiF WcVuXsPvnG3qc212h65CqOK39BG3PYZF3G5OWrwlqWe3Inyrjg==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wQ-0006Ld-Hl; Tue, 09 Apr 2024 09:30:50 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 2/6] docs: verify/bisect: add and fetch stable branches ahead of time Date: Tue, 9 Apr 2024 09:30:45 +0200 Message-ID: <57dcf312959476abe6151bf3d35eb79e3e9a83d1.1712647788.git.linux@leemhuis.info> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wQ-0006Ld-Hl Content-Type: text/plain; charset="utf-8" Add and fetch all required stable branches ahead of time. This fixes a bug, as readers that wanted to bisect a regression within a stable or longterm series otherwise did not have them available at the right time. This way also matches the flow somewhat better and avoids some "if you haven't already added it" phrases that otherwise become necessary in future changes. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 31 +++++++++++-------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index c999e40c79ab7f..06278501a4bdcc 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -136,8 +136,7 @@ will be considered the 'good' release and used to prepa= re the .config file. =20 * **Segment 3**: perform and validate the bisection. =20 - a) In case your 'broken' version is a stable/longterm release, add the G= it - branch holding it:: + a) Retrieve the sources for your 'bad' version:: =20 git remote set-branches --add stable linux-6.1.y git fetch stable @@ -371,6 +370,21 @@ Preparations: set up everything to build your own kern= els =20 [:ref:`details`] =20 +.. _stablesources_bissbs: + +* Retrieve the sources for any stable or longterm series you might need. + + Is the version you earlier established as 'bad' a stable or longterm rel= ease? + Then download the code for the series it belongs to ('linux-6.1.y' in th= is + example):: + + git remote set-branches --add stable linux-6.1.y + git fetch stable + + If the version earlier established as 'good' is from a different stable = or + longterm series (say 6.0.13), repeat the previous step, but this time fo= r the + branch holding the series the 'good' version belongs to (e.g. linux-6.0.= y). + .. _oldconfig_bissbs: =20 * Start preparing a kernel build configuration (the '.config' file). @@ -620,12 +634,10 @@ be a waste of time. [:ref:`details`] reproduce it with the mainline kernel you just built? One that according= to the `front page of kernel.org `_ is still supported= ? Then check if the latest codebase for the particular series might already fix= the - problem. To do so, add the stable series Git branch for your 'good' kern= el - (again, this here is assumed to be 6.0) and check out the latest version= :: + problem. To do so, check out that series latest version (again, this her= e is + assumed to be 6.0):: =20 cd ~/linux/ - git remote set-branches --add stable linux-6.0.y - git fetch stable git switch --discard-changes --detach linux-6.0.y =20 Now use the checked out code to build and install another kernel using t= he @@ -717,13 +729,6 @@ configuration created earlier this works a lot faster = than many people assume: overall on average it will often just take about 10 to 15 minutes to compi= le each kernel on commodity x86 machines. =20 -* In case your 'bad' version is a stable/longterm release (say 6.1.5), add= its - stable branch, unless you already did so earlier:: - - cd ~/linux/ - git remote set-branches --add stable linux-6.1.y - git fetch stable - .. _bisectstart_bissbs: =20 * Start the bisection and tell Git about the versions earlier established = as --=20 2.44.0 From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96F4A7C0AB; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=tkSH63eadaHc352RmbQaiMq30j0mSKAwPVChj8UeVWZ5LiENkCpFvPA/vVA6N5VMO8l8sZ37DKDqR1SN2G4M3uEMoudWL/cgQH72Z8KU7z9JPfoIEojJpAVsPv7OfwK8uAyfouXQXF5nCUySVM37XnwOpeehcqrBgV+D8gRZlJo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=eEtxTLHmaVQHaJavtnI8Gwhmf/rIFPkobr3F83J+mr8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MU7VIXSJ/inKEukAd8DWAErawY2i/NkknftCj7oy+TkwRIEwVuAbZOYPhODstLxzMfP2vGu+Aq88Pu7pE3BDdr8fSYLssiztq74RWiuJoYsc25yKsYoiewE8SzK+C3QV7apBE8HczlTeRm5kdVamwCvynTei83+E5t0bJwi72Cc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=rZYEycqh; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="rZYEycqh" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=ys+pFDTjHVyyCLZAJtZ5EOUWX+e08v/L7i+jg3ZPpmw=; t=1712647855; x=1713079855; b=rZYEycqhkydPnrgY48jzYygRqjkUbw3YmIp69kGjaKECXu4MAyV21W5ODUbQK vKyFS1H4zelBmgHhAmlnktKY5HXTLcmB03HYovcdaHSJC7zLdTsFdeVQ0ZQiXbB31Fe7kC+nnV6Y4 x11fOi4Nc9FNYQMWdcet8ewx/qHDdT+KuKwUpLuqdvyuNRgb6o7dkT6ZUSMT1R/A44S0TAPups3pe o5DScbu3CH0eyMjov02oygmCwJxOqEs3jo6dZuFZRwwFu1jOD8GKbSU5Q2AhihH0zVmU/Z9w6J02z TR6TxfYP2Lu/Q+4hxE+g2QFGVGhq8sTGcoxX3SwH8wyABa2P7Q==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wQ-0006Ld-SR; Tue, 09 Apr 2024 09:30:50 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 3/6] docs: verify/bisect: proper headlines and more spacing Date: Tue, 9 Apr 2024 09:30:46 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wQ-0006Ld-SR Content-Type: text/plain; charset="utf-8" Various small improvements and fixes: * Separate ref links from their target with a space for better readability. * Add a proper heading for the note at the end of the step-by-step guide. * Use proper 3rd and 4th level headlines in the reference section and add short intros for the 2nd level headlines that lacked one. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 194 ++++++++++-------- 1 file changed, 113 insertions(+), 81 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index 06278501a4bdcc..355c2cea5230d9 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -29,7 +29,7 @@ The essence of the process (aka 'TL;DR') =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 *[If you are new to building or bisecting Linux, ignore this section and h= ead -over to the* ":ref:`step-by-step guide`" *below. It uti= lizes +over to the* ':ref:`step-by-step guide `' *below. It ut= ilizes the same commands as this section while describing them in brief fashion. = The steps are nevertheless easy to follow and together with accompanying entri= es in a reference section mention many alternatives, pitfalls, and additional @@ -224,15 +224,15 @@ report; instead of the latter your could also head st= raight on and follow *segment 3* to **perform a bisection** for a full-fledged regression report developers are obliged to act upon. =20 - :ref:`Preparations: set up everything to build your own kernels.` + :ref:`Preparations: set up everything to build your own kernels `. =20 - :ref:`Segment 1: try to reproduce the problem with the latest codebase.` + :ref:`Segment 1: try to reproduce the problem with the latest codebase `. =20 - :ref:`Segment 2: check if the kernels you build work fine.` + :ref:`Segment 2: check if the kernels you build work fine `. =20 - :ref:`Segment 3: perform a bisection and validate the result.` + :ref:`Segment 3: perform a bisection and validate the result `. =20 - :ref:`Supplementary tasks: cleanup during and after following this guide.= ` + :ref:`Supplementary tasks: cleanup during and after following this guide = `. =20 The steps in each segment illustrate the important aspects of the process,= while a comprehensive reference section holds additional details for almost all = of the @@ -260,12 +260,14 @@ improve it, :ref:`please let the kernel developers kn= ow `. Preparations: set up everything to build your own kernels --------------------------------------------------------- =20 +The following steps lay the groundwork for all further tasks. + .. _backup_bissbs: =20 * Create a fresh backup and put system repair and restore tools at hand, j= ust to be prepared for the unlikely case of something going sideways. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _vanilla_bissbs: =20 @@ -273,7 +275,7 @@ Preparations: set up everything to build your own kerne= ls builds them automatically. That includes but is not limited to DKMS, ope= nZFS, VirtualBox, and Nvidia's graphics drivers (including the GPLed kernel mo= dule). =20 - [:ref:`details`] + [:ref:`details `] =20 .. _secureboot_bissbs: =20 @@ -284,7 +286,7 @@ Preparations: set up everything to build your own kerne= ls their restrictions through a process initiated by ``mokutil --disable-validation``. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _rangecheck_bissbs: =20 @@ -319,13 +321,13 @@ Preparations: set up everything to build your own ker= nels throughout this guide will refer to the last kernel that has been working fine.* =20 - [:ref:`details`] + [:ref:`details `] =20 .. _bootworking_bissbs: =20 * Boot into the 'working' kernel and briefly use the apparently broken fea= ture. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _diskspace_bissbs: =20 @@ -335,7 +337,7 @@ Preparations: set up everything to build your own kerne= ls debug symbols: both explain approaches reducing the amount of space, whi= ch should allow you to master these tasks with about 4 Gigabytes free space. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _buildrequires_bissbs: =20 @@ -345,7 +347,7 @@ Preparations: set up everything to build your own kerne= ls reference section shows how to quickly install those on various popular = Linux distributions. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _sources_bissbs: =20 @@ -368,7 +370,7 @@ Preparations: set up everything to build your own kerne= ls git remote add -t master stable \ https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git =20 - [:ref:`details`] + [:ref:`details `] =20 .. _stablesources_bissbs: =20 @@ -421,7 +423,7 @@ Preparations: set up everything to build your own kerne= ls 'make olddefconfig' again and check if it now picked up the right config= file as base. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _localmodconfig_bissbs: =20 @@ -455,7 +457,7 @@ Preparations: set up everything to build your own kerne= ls spending much effort on, as long as it boots and allows to properly test= the feature that causes trouble. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _tagging_bissbs: =20 @@ -465,7 +467,7 @@ Preparations: set up everything to build your own kerne= ls ./scripts/config --set-str CONFIG_LOCALVERSION '-local' ./scripts/config -e CONFIG_LOCALVERSION_AUTO =20 - [:ref:`details`] + [:ref:`details `] =20 .. _debugsymbols_bissbs: =20 @@ -484,7 +486,7 @@ Preparations: set up everything to build your own kerne= ls ./scripts/config -d DEBUG_INFO -d DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT \ -d DEBUG_INFO_DWARF4 -d DEBUG_INFO_DWARF5 -e CONFIG_DEBUG_INFO_NONE =20 - [:ref:`details`] + [:ref:`details `] =20 .. _configmods_bissbs: =20 @@ -494,14 +496,14 @@ Preparations: set up everything to build your own ker= nels * Are you running Debian? Then you want to avoid known problems by perfo= rming additional adjustments explained in the reference section. =20 - [:ref:`details`]. + [:ref:`details `]. =20 * If you want to influence other aspects of the configuration, do so now= using your preferred tool. Note, to use make targets like 'menuconfig' or 'nconfig', you will need to install the development files of ncurses; = for 'xconfig' you likewise need the Qt5 or Qt6 headers. =20 - [:ref:`details`]. + [:ref:`details `]. =20 .. _saveconfig_bissbs: =20 @@ -511,7 +513,7 @@ Preparations: set up everything to build your own kerne= ls make olddefconfig cp .config ~/kernel-config-working =20 - [:ref:`details`] + [:ref:`details `] =20 .. _introlatestcheck_bissbs: =20 @@ -521,7 +523,7 @@ Segment 1: try to reproduce the problem with the latest= codebase The following steps verify if the problem occurs with the code currently supported by developers. In case you face a regression, it also checks tha= t the problem is not caused by some .config change, as reporting the issue then = would -be a waste of time. [:ref:`details`] +be a waste of time. [:ref:`details `] =20 .. _checkoutmaster_bissbs: =20 @@ -530,7 +532,7 @@ be a waste of time. [:ref:`details`] cd ~/linux/ git switch --discard-changes --detach mainline/master =20 - [:ref:`details`] + [:ref:`details `] =20 .. _build_bissbs: =20 @@ -545,7 +547,7 @@ be a waste of time. [:ref:`details`] reference section for alternatives, which obviously will require other steps to install as well. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _install_bissbs: =20 @@ -578,7 +580,7 @@ be a waste of time. [:ref:`details`] down: if you will build more kernels as described in segment 2 and 3, yo= u will have to perform those again after executing ``command -v installkernel [= ...]``. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _storagespace_bissbs: =20 @@ -591,7 +593,7 @@ be a waste of time. [:ref:`details`] Write down or remember those two values for later: they enable you to pr= event running out of disk space accidentally during a bisection. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _kernelrelease_bissbs: =20 @@ -618,7 +620,7 @@ be a waste of time. [:ref:`details`] If that command does not return '0', check the reference section, as the= cause for this might interfere with your testing. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _recheckbroken_bissbs: =20 @@ -626,7 +628,7 @@ be a waste of time. [:ref:`details`] out the instructions in the reference section to ensure nothing went sid= eways during your tests. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _recheckstablebroken_bissbs: =20 @@ -662,12 +664,12 @@ be a waste of time. [:ref:`details`] =20 Now verify if this kernel is showing the problem. =20 - [:ref:`details`] + [:ref:`details `] =20 Do you follow this guide to verify if a problem is present in the code currently supported by Linux kernel developers? Then you are done at this point. If you later want to remove the kernel you just built, check out -:ref:`Supplementary tasks: cleanup during and after following this guide`. +:ref:`Supplementary tasks: cleanup during and after following this guide <= introclosure_bissbs>`. =20 In case you face a regression, move on and execute at least the next segme= nt as well. @@ -679,7 +681,7 @@ Segment 2: check if the kernels you build work fine =20 In case of a regression, you now want to ensure the trimmed configuration = file you created earlier works as expected; a bisection with the .config file -otherwise would be a waste of time. [:ref:`details`] +otherwise would be a waste of time. [:ref:`details `] =20 .. _recheckworking_bissbs: =20 @@ -714,7 +716,7 @@ otherwise would be a waste of time. [:ref:`details`] Now check if this kernel works as expected; if not, consult the reference section for further instructions. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _introbisect_bissbs: =20 @@ -739,7 +741,7 @@ each kernel on commodity x86 machines. git bisect good v6.0 git bisect bad v6.1.5 =20 - [:ref:`details`] + [:ref:`details `] =20 .. _bisectbuild_bissbs: =20 @@ -768,7 +770,7 @@ each kernel on commodity x86 machines. totally normal to see release identifiers like '6.0-rc1-local-gcafec0cac= aca0' if you bisect between versions 6.1 and 6.2 for example. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _bisecttest_bissbs: =20 @@ -808,7 +810,7 @@ each kernel on commodity x86 machines. might need to scroll up to see the message mentioning the culprit; alternatively, run ``git bisect log > ~/bisection-log``. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _bisectlog_bissbs: =20 @@ -820,7 +822,7 @@ each kernel on commodity x86 machines. cp .config ~/bisection-config-culprit git bisect reset =20 - [:ref:`details`] + [:ref:`details `] =20 .. _revert_bissbs: =20 @@ -882,7 +884,7 @@ each kernel on commodity x86 machines. Now check one last time if the feature that made you perform a bisection= works with that kernel: if everything went well, it should not show the regres= sion. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _introclosure_bissbs: =20 @@ -923,7 +925,7 @@ space might run out. kernel image and related files behind; in that case remove them as descr= ibed in the reference section. =20 - [:ref:`details`] + [:ref:`details `] =20 .. _finishingtouch_bissbs: =20 @@ -946,11 +948,15 @@ space might run out. the version considered 'good', and the last three or four you compiled during the actual bisection process. =20 - [:ref:`details`] + [:ref:`details `] + =20 .. _submit_improvements: =20 -This concludes the step-by-step guide. +Conclusion +---------- + +You have reached the end of the step-by-step guide. =20 Did you run into trouble following any of the above steps not cleared up b= y the reference section below? Did you spot errors? Or do you have ideas how to @@ -970,10 +976,20 @@ Reference section for the step-by-step guide This section holds additional information for almost all the items in the = above step-by-step guide. =20 +Preparations for building your own kernels +------------------------------------------ + + *The steps in this section lay the groundwork for all further tests.* + [:ref:`... `] + +The steps in all later sections of this guide depend on those described he= re. + +[:ref:`back to step-by-step guide `]. + .. _backup_bisref: =20 Prepare for emergencies ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ =20 *Create a fresh backup and put system repair and restore tools at hand.* [:ref:`... `] @@ -988,7 +1004,7 @@ for something going sideways, even if that should not = happen. .. _vanilla_bisref: =20 Remove anything related to externally maintained kernel modules ---------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Remove all software that depends on externally developed kernel drivers= or builds them automatically.* [:ref:`...`] @@ -1006,7 +1022,7 @@ explains in more detail. .. _secureboot_bisref: =20 Deal with techniques like Secure Boot -------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *On platforms with 'Secure Boot' or similar techniques, prepare everythi= ng to ensure the system will permit your self-compiled kernel to boot later.* @@ -1043,7 +1059,7 @@ Afterwards, permit MokManager to reboot the machine. .. _bootworking_bisref: =20 Boot the last kernel that was working -------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Boot into the last working kernel and briefly recheck if the feature th= at regressed really works.* [:ref:`...`] @@ -1056,7 +1072,7 @@ the right thing. .. _diskspace_bisref: =20 Space requirements ------------------- +~~~~~~~~~~~~~~~~~~ =20 *Ensure to have enough free space for building Linux.* [:ref:`... `] @@ -1074,7 +1090,7 @@ space by quite a few gigabytes. .. _rangecheck_bisref: =20 Bisection range ---------------- +~~~~~~~~~~~~~~~ =20 *Determine the kernel versions considered 'good' and 'bad' throughout th= is guide.* [:ref:`...`] @@ -1099,7 +1115,7 @@ to do this as well, if you tried bisecting between 6.= 0.13 and 6.1.15. .. _buildrequires_bisref: =20 Install build requirements --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Install all software required to build a Linux kernel.* [:ref:`...`] @@ -1150,7 +1166,7 @@ the kernel's tools/ directory. .. _sources_bisref: =20 Download the sources using Git ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Retrieve the Linux mainline sources.* [:ref:`...`] @@ -1170,7 +1186,7 @@ work better for you: .. _sources_bundle_bisref: =20 Downloading Linux mainline sources using a bundle -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""""""""""""""""""""""""""" =20 Use the following commands to retrieve the Linux mainline sources using a bundle:: @@ -1241,7 +1257,7 @@ Note, shallow clones have a few peculiar characterist= ics: .. _oldconfig_bisref: =20 Start defining the build configuration for your kernel ------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Start preparing a kernel build configuration (the '.config' file).* [:ref:`... `] @@ -1301,7 +1317,7 @@ that file to the build machine and store it as ~/linu= x/.config; afterwards run .. _localmodconfig_bisref: =20 Trim the build configuration for your kernel --------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Disable any kernel modules apparently superfluous for your setup.* [:ref:`... `] @@ -1350,7 +1366,7 @@ step-by-step guide mentions:: .. _tagging_bisref: =20 Tag the kernels about to be build ---------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Ensure all the kernels you will build are clearly identifiable using a special tag and a unique version identifier.* [:ref:`... `] @@ -1366,7 +1382,7 @@ confusing during the bisection. .. _debugsymbols_bisref: =20 Decide to enable or disable debug symbols ------------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Decide how to handle debug symbols.* [:ref:`... `] =20 @@ -1395,7 +1411,7 @@ explains this process in more detail. .. _configmods_bisref: =20 Adjust build configuration --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Check if you may want or need to adjust some other kernel configuration options:* @@ -1406,7 +1422,7 @@ kernel configuration options. .. _configmods_distros_bisref: =20 Distro specific adjustments -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""""" =20 *Are you running* [:ref:`... `] =20 @@ -1431,7 +1447,7 @@ when following this guide on a few commodity distribu= tions. .. _configmods_individual_bisref: =20 Individual adjustments -~~~~~~~~~~~~~~~~~~~~~~ +"""""""""""""""""""""" =20 *If you want to influence the other aspects of the configuration, do so now.* [:ref:`... `] @@ -1448,13 +1464,13 @@ is missing. .. _saveconfig_bisref: =20 Put the .config file aside --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Reprocess the .config after the latest changes and store it in a safe p= lace.* [:ref:`... `] =20 Put the .config you prepared aside, as you want to copy it back to the bui= ld -directory every time during this guide before you start building another +directory every time during this guide before you start building another kernel. That's because going back and forth between different versions can= alter .config files in odd ways; those occasionally cause side effects that could confuse testing or in some cases render the result of your bisection @@ -1464,8 +1480,8 @@ meaningless. =20 .. _introlatestcheck_bisref: =20 -Try to reproduce the regression ------------------------------------------ +Try to reproduce the problem with the latest codebase +----------------------------------------------------- =20 *Verify the regression is not caused by some .config change and check if= it still occurs with the latest codebase.* [:ref:`... `] @@ -1519,21 +1535,21 @@ highly recommended for these reasons: .. _checkoutmaster_bisref: =20 Check out the latest Linux codebase ------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Check out the latest Linux codebase.* - [:ref:`... `] + [:ref:`... `] =20 In case you later want to recheck if an ever newer codebase might fix the problem, remember to run that ``git fetch --shallow-exclude [...]`` command again mentioned earlier to update your local Git repository. =20 -[:ref:`back to step-by-step guide `] +[:ref:`back to step-by-step guide `] =20 .. _build_bisref: =20 Build your kernel ------------------ +~~~~~~~~~~~~~~~~~ =20 *Build the image and the modules of your first kernel using the config f= ile you prepared.* [:ref:`... `] @@ -1543,7 +1559,7 @@ yourself. Another subsection explains how to directly= package your kernel up as deb, rpm or tar file. =20 Dealing with build errors -~~~~~~~~~~~~~~~~~~~~~~~~~ +""""""""""""""""""""""""" =20 When a build error occurs, it might be caused by some aspect of your machi= ne's setup that often can be fixed quickly; other times though the problem lies= in @@ -1578,7 +1594,7 @@ system, but lies in the code. If you run into one of = those, you might thus find a solution (e.g. a patch) or workaround for your issue, too. =20 Package your kernel up -~~~~~~~~~~~~~~~~~~~~~~ +"""""""""""""""""""""" =20 The step-by-step guide uses the default make targets (e.g. 'bzImage' and 'modules' on x86) to build the image and the modules of your kernel, which= later @@ -1609,7 +1625,7 @@ distribution's kernel packages. .. _install_bisref: =20 Put the kernel in place ------------------------ +~~~~~~~~~~~~~~~~~~~~~~~ =20 *Install the kernel you just built.* [:ref:`... `] =20 @@ -1652,7 +1668,7 @@ process. Afterwards add your kernel to your bootloade= r configuration and reboot. .. _storagespace_bisref: =20 Storage requirements per kernel -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Check how much storage space the kernel, its modules, and other related= files like the initramfs consume.* [:ref:`... `] @@ -1673,7 +1689,7 @@ need to look in different places. .. _tainted_bisref: =20 Check if your newly built kernel considers itself 'tainted' ------------------------------------------------------------ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Check if the kernel marked itself as 'tainted'.* [:ref:`... `] @@ -1692,7 +1708,7 @@ interest, as your testing might be flawed otherwise. .. _recheckbroken_bisref: =20 Check the kernel built from a recent mainline codebase ------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Verify if your bug occurs with the newly built kernel.* [:ref:`... `] @@ -1718,7 +1734,7 @@ the kernel you built from the latest codebase. These = are the most frequent: .. _recheckstablebroken_bisref: =20 Check the kernel built from the latest stable/longterm codebase ---------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Are you facing a regression within a stable/longterm release, but faile= d to reproduce it with the kernel you just built using the latest mainline so= urces? @@ -1763,7 +1779,7 @@ ensure the kernel version you assumed to be 'good' ea= rlier in the process (e.g. .. _recheckworking_bisref: =20 Build your own version of the 'good' kernel -------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Build your own variant of the working kernel and check if the feature t= hat regressed works as expected with it.* [:ref:`... = `] @@ -1794,10 +1810,20 @@ most likely flawed. =20 [:ref:`back to step-by-step guide `] =20 +Perform a bisection and validate the result +------------------------------------------- + + *With all the preparations and precaution builds taken care of, you are = now + ready to begin the bisection.* [:ref:`... `] + +The steps in this segment perform and validate the bisection. + +[:ref:`back to step-by-step guide `]. + .. _bisectstart_bisref: =20 Start the bisection -------------------- +~~~~~~~~~~~~~~~~~~~ =20 *Start the bisection and tell Git about the versions earlier established= as 'good' and 'bad'.* [:ref:`... `] @@ -1811,7 +1837,7 @@ for you to test. .. _bisectbuild_bisref: =20 Build a kernel from the bisection point ---------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Build, install, and boot a kernel from the code Git checked out using t= he same commands you used earlier.* [:ref:`... `] @@ -1839,7 +1865,7 @@ There are two things worth of note here: .. _bisecttest_bisref: =20 Bisection checkpoint --------------------- +~~~~~~~~~~~~~~~~~~~~ =20 *Check if the feature that regressed works in the kernel you just built.* [:ref:`... `] @@ -1853,7 +1879,7 @@ will be for nothing. .. _bisectlog_bisref: =20 Put the bisection log away --------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Store Git's bisection log and the current .config file in a safe place.* [:ref:`... `] @@ -1873,7 +1899,7 @@ ask for it after you report the regression. .. _revert_bisref: =20 Try reverting the culprit -------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *Try reverting the culprit on top of the latest codebase to see if this = fixes your regression.* [:ref:`... `] @@ -1891,14 +1917,20 @@ succeeds, test that kernel version instead. =20 [:ref:`back to step-by-step guide `] =20 +Cleanup steps during and after following this guide +--------------------------------------------------- =20 -Supplementary tasks: cleanup during and after the bisection ------------------------------------------------------------ + *During and after following this guide you might want or need to remove = some + of the kernels you installed.* [:ref:`... `] + +The steps in this section describe clean-up procedures. + +[:ref:`back to step-by-step guide `]. =20 .. _makeroom_bisref: =20 Cleaning up during the bisection --------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 *To remove one of the kernels you installed, look up its 'kernelrelease' identifier.* [:ref:`... `] @@ -1939,7 +1971,7 @@ when all you want is to remove 6.0 or 6.0.1. [:ref:`back to step-by-step guide `] =20 Cleaning up after the bisection -------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ =20 .. _finishingtouch_bisref: =20 --=20 2.44.0 From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 931AA7BAFE; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=ZKgsvupKV+F8TWpNZwFbXKxEQBj1+FPfCq2U8fkMnboiDqB02WZkCPJYIFZBpob5T1JQxoQEPkZitOQpFGkaf0ZoRbkapz7ijAweJkNrEhld2TyzeDMIcvswRwGUxbLgBBgW5Sibu4Z66UuZ8sLiabacRFAOCWUZZzs49LxXL90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=j4STcrPuuHe6kQK78UyT+7uWG0z053vYmOT+DzSljc0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TQDqU65Mx1vfC86alXYOjxRP5phiG4NsSOFjPOJGNGXwf7f3LcRK+PbOS59wckdq0yNFeBpp0LVquSKvqB/gkpKzaFoT0LFcnI1ODoqUGOtEqu3aDQjDAk2UTXB57pzLj2pHkofmmNF/ATQdFPVEJ9bqrHP+WWe3vADEZW5an2E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=HZquhLkV; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="HZquhLkV" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=LGeHfhhpbX0NVOpB0WHhcEnxl/s/Lvh5IZ1Wpx5G0xY=; t=1712647855; x=1713079855; b=HZquhLkVzd1582+mrANxa+lFQZY0BrVsFfvIINnOBWD/HER7bPjXZCcX72yGq a76cdVIs+LJnJcgV5VymzRPuC8ZBTnHcL3pSLpSSL0AXTwdBcVs/KaNOzpoY4N/lR38xonLRvdhqk nS9ZcuPHcaq5uSjgwoY0fxZnPR7pXTkSiZiuqMipjhqI1O/wG1/5FcdMEM8IwxY8TCGOyCAYLWqa4 7cIlA1fY2KlUAIwH3gI4q2C9R+OLbBvPh5sngAWkOB+6yJ/ntr5RPAIGeVUIqUfqjZiSndvoNrVh2 Zj1A26eb9uAYVRs8rQEJia2jSpjJyaItkE1naC7YXyJqHS1q+Q==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wR-0006Ld-6C; Tue, 09 Apr 2024 09:30:51 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 4/6] docs: verify/bisect: explain testing reverts, patches and newer code Date: Tue, 9 Apr 2024 09:30:47 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wR-0006Ld-6C Content-Type: text/plain; charset="utf-8" Rename 'Supplementary tasks' to 'Complementary tasks' while introducing a section 'Optional tasks: test reverts, patches, or later versions': the latter is something readers occasionally will have to do after reporting a bug and thus is best covered here. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 128 ++++++++++++++++-- 1 file changed, 115 insertions(+), 13 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index 355c2cea5230d9..1987c827211fce 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -160,7 +160,7 @@ will be considered the 'good' release and used to prepa= re the .config file. this step (e.g. build, install, boot, and test a kernel to then tell = Git the outcome). Do so again and again until Git shows which commit broke things. If you run short of disk space during this process, check the - section 'Supplementary tasks: cleanup during and after the process' + section 'Complementary tasks: cleanup during and after the process' below. =20 d) Once your finished the bisection, put a few things away:: @@ -182,7 +182,7 @@ will be considered the 'good' release and used to prepa= re the .config file. kernel; just this time skip the first command copying the base .config= file over, as that already has been taken care off. =20 -* **Supplementary tasks**: cleanup during and after the process. +* **Complementary tasks**: cleanup during and after the process. =20 a) To avoid running out of disk space during a bisection, you might need= to remove some kernels you built earlier. You most likely want to keep t= hose @@ -205,6 +205,18 @@ will be considered the 'good' release and used to prep= are the .config file. the kernels you built earlier and later you might want to keep around= for a week or two. =20 +* **Optional task**: test a debug patch or a proposed fix later:: + + git fetch mainline + git switch --discard-changes --detach mainline/master + git apply /tmp/foobars-proposed-fix-v1.patch + cp ~/kernel-config-working .config + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1' + + Build, install, and boot a kernel as described in *segment 1, section b*= -- + but this time omit the first command copying the build configuration ove= r, + as that has been taken care of already. + .. _introguide_bissbs: =20 Step-by-step guide on how to verify bugs and bisect regressions @@ -232,7 +244,9 @@ developers are obliged to act upon. =20 :ref:`Segment 3: perform a bisection and validate the result `. =20 - :ref:`Supplementary tasks: cleanup during and after following this guide = `. + :ref:`Complementary tasks: cleanup during and after following this guide = `. + + :ref:`Optional tasks: test reverts, patches, or later versions `. =20 The steps in each segment illustrate the important aspects of the process,= while a comprehensive reference section holds additional details for almost all = of the @@ -669,7 +683,7 @@ be a waste of time. [:ref:`details `] Do you follow this guide to verify if a problem is present in the code currently supported by Linux kernel developers? Then you are done at this point. If you later want to remove the kernel you just built, check out -:ref:`Supplementary tasks: cleanup during and after following this guide <= introclosure_bissbs>`. +:ref:`Complementary tasks: cleanup during and after following this guide <= introclosure_bissbs>`. =20 In case you face a regression, move on and execute at least the next segme= nt as well. @@ -888,7 +902,7 @@ each kernel on commodity x86 machines. =20 .. _introclosure_bissbs: =20 -Supplementary tasks: cleanup during and after the bisection +Complementary tasks: cleanup during and after the bisection ----------------------------------------------------------- =20 During and after following this guide you might want or need to remove som= e of @@ -950,6 +964,81 @@ space might run out. =20 [:ref:`details `] =20 +.. _introoptional_bissbs: + +Optional: test reverts, patches, or later versions +-------------------------------------------------- + +While or after reporting a bug, you might want or potentially will be aske= d to +test reverts, debug patches, proposed fixes, or other versions. In that ca= se +follow these instructions. + +* Update your Git clone and check out the latest code. + + * In case you want to test mainline, fetch its latest changes before che= cking + its code out:: + + git fetch mainline + git switch --discard-changes --detach mainline/master + + * In case you want to test a stable or longterm kernel, first add the br= anch + holding the series you are interested in (6.2 in the example), unless = you + already did so earlier:: + + git remote set-branches --add stable linux-6.2.y + + Then fetch the latest changes and check out the latest version from the + series:: + + git fetch stable + git switch --discard-changes --detach stable/linux-6.2.y + +* Copy your kernel build configuration over:: + + cp ~/kernel-config-working .config + +* Your next step depends on what you want to do: + + * In case you just want to test the latest codebase, head to the next st= ep, + you are already all set. + + * In case you want to test if a revert fixes an issue, revert one or mul= tiple + changes by specifying their commit ids:: + + git revert --no-edit cafec0cacaca0 + + Now give that kernel a special tag to facilitates its identification a= nd + prevent accidentally overwriting another kernel:: + + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-cafec0cacaca0= -reverted' + + * In case you want to test a patch, store the patch in a file like + '/tmp/foobars-proposed-fix-v1.patch' and apply it like this:: + + git apply /tmp/foobars-proposed-fix-v1.patch + + In case of multiple patches, repeat this step with the others. + + Now give that kernel a special tag to facilitates its identification a= nd + prevent accidentally overwriting another kernel:: + + ./scripts/config --set-str CONFIG_LOCALVERSION '-local-foobars-fix-v1' + +* Build a kernel using the familiar commands, just without copying the ker= nel + build configuration over, as that has been taken care of already:: + + make olddefconfig && + make -j $(nproc --all) + # * Check if the free space suffices holding another kernel: + df -h /boot/ /lib/modules/ + sudo make modules_install + command -v installkernel && sudo make install + make -s kernelrelease | tee -a ~/kernels-built + reboot + +* Now verify you booted the newly built kernel and check it. + +[:ref:`details `] =20 .. _submit_improvements: =20 @@ -1986,20 +2075,33 @@ build artifacts and the Linux sources, but will lea= ve the Git repository (~/linux/.git/) behind -- a simple ``git reset --hard`` thus will bring the sources back. =20 -Removing the repository as well would likely be unwise at this point: ther= e is a -decent chance developers will ask you to build another kernel to perform -additional tests. This is often required to debug an issue or check propos= ed -fixes. Before doing so you want to run the ``git fetch mainline`` command = again -followed by ``git checkout mainline/master`` to bring your clone up to dat= e and -checkout the latest codebase. Then apply the patch using ``git apply -`` or ``git am `` and build yet another kernel using t= he -familiar commands. +Removing the repository as well would likely be unwise at this point: there +is a decent chance developers will ask you to build another kernel to +perform additional tests -- like testing a debug patch or a proposed fix. +Details on how to perform those can be found in the section :ref:`Optional +tasks: test reverts, patches, or later versions `. =20 Additional tests are also the reason why you want to keep the ~/kernel-config-working file around for a few weeks. =20 [:ref:`back to step-by-step guide `] =20 +.. _introoptional_bisref: + +Test reverts, patches, or later versions +---------------------------------------- + + *While or after reporting a bug, you might want or potentially will be a= sked + to test reverts, patches, proposed fixes, or other versions.* + [:ref:`... `] + +All the commands used in this section should be pretty straight forward, so +there is not much to add except one thing: when setting a kernel tag as +instructed, ensure it is not much longer than the one used in the example,= as +problems will arise if the kernelrelease identifier exceeds 63 characters. + +[:ref:`back to step-by-step guide `]. + =20 Additional reading material =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D --=20 2.44.0 From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B834F79955; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=JC7BP9HwyslE8lbiKFdd4MdU3e9W2GCSLF3XmruA0r9wjFnhW6awmbhXBWn2LAm7qveKiM3i4+jFQTBz4n47j9M8zKJQqAqvR4XRdjhLbxfODxAzJ6avZE0brdeH0lqhQbfd223Nx3A9j56ZR9pUNlNcjXQmWsp77aadjxT7+pk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=CMX6aA9wScU43YKlAsL1ntr9Wuy50Tzw1OIpyrleAiQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CiGCzo88akwT7Unbxcw6iS4VxZBl281GAtIJ7B3+UCZnIuyfFiaeCUHHXsh0osuEsobnMYNzpS65VoAnrY3CBKYP6y/L4ENZIXeSpzvjpl1T9FyX1sqieTzvwY8tq5W8n6X/WA3Y60J0ldKdIwezSJB+BCbHn2OxdTJ9nlmSB4Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=V/l8NogV; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="V/l8NogV" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=pPlFRJ8SFuDnkXHqmbQgQnUnXyCLw1rG7kPikkOu6Os=; t=1712647855; x=1713079855; b=V/l8NogVpAGIwvDzQnLqRuqtaV/4OS4p/7xfY35YcxlwizVkYsIXvsupHB99y UnJhePk5HTmd4un3fo4jwM2biLFQzbCzXoqvVBn2cbYoh4XjumKm1CoTVIvWCVtJXB8mVl4gu1x0M DqrWsxF7S1p39nungCGJlH2z9PECr48UfpfsigAiefv780B40xAxnDG7vNc+Drtsvdwe/H6hnDPNc 6HuEF5rSbZLf8z3v9C4ZzL1px12beJpPY7ToAxpEZMWGqefUvsVgDeGT/SolBQNlB5QEBqYdxUgth uuTCUS1bY3W2me0lcRhoyynzUOnEYCCxHexhKwVKtVb/f4fS6A==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wR-0006Ld-Eq; Tue, 09 Apr 2024 09:30:51 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 5/6] docs: verify/bisect: describe how to use a build host Date: Tue, 9 Apr 2024 09:30:48 +0200 Message-ID: <288160cb4769e46a3280250ca71da0abc4aa002d.1712647788.git.linux@leemhuis.info> X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wR-0006Ld-Eq Content-Type: text/plain; charset="utf-8" Describe how to build kernels on another system (with and without cross-compiling), as building locally can be quite painfully on some slow systems. This is done in an add-on section, as it would make the step-by-step guide to complicated if this special case would be described there. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 78 ++++++++++++++++++- 1 file changed, 74 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index 1987c827211fce..6193c797642732 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -276,6 +276,10 @@ Preparations: set up everything to build your own kern= els =20 The following steps lay the groundwork for all further tasks. =20 +Note: the instructions assume you are building and testing on the same +machine; if you want to compile the kernel on another system, check +:ref:`Build kernels on a different machine ` below. + .. _backup_bissbs: =20 * Create a fresh backup and put system repair and restore tools at hand, j= ust @@ -2103,11 +2107,77 @@ problems will arise if the kernelrelease identifier= exceeds 63 characters. [:ref:`back to step-by-step guide `]. =20 =20 -Additional reading material -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D +Additional information +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +.. _buildhost_bis: + +Build kernels on a different machine +------------------------------------ + +To compile kernels on another system, slightly alter the step-by-step guid= e's +instructions: + +* Start following the guide on the machine where you want to install and t= est + the kernels later. + +* After executing ':ref:`Boot into the working kernel and briefly use the + apparently broken feature `', save the list of loaded + modules to a file using ``lsmod > ~/test-machine-lsmod``. Then locate the + build configuration for the running kernel (see ':ref:`Start defining the + build configuration for your kernel `' for hints on wh= ere + to find it) and store it as '~/test-machine-config-working'. Transfer bo= th + files to the home directory of your build host. + +* Continue the guide on the build host (e.g. with ':ref:`Ensure to have en= ough + free space for building [...] `'). + +* When you reach ':ref:`Start preparing a kernel build configuration[...] + `': before running ``make olddefconfig`` for the first= time, + execute the following command to base your configuration on the one from= the + test machine's 'working' kernel:: + + cp ~/test-machine-config-working ~/linux/.config + +* During the next step to ':ref:`disable any apparently superfluous kernel + modules `' use the following command instead:: + + yes '' | make localmodconfig LSMOD=3D~/lsmod_foo-machine localmodconfig + +* Continue the guide, but ignore the instructions outlining how to compile, + install, and reboot into a kernel every time they come up. Instead build + like this:: =20 -Further sources ---------------- + cp ~/kernel-config-working .config + make olddefconfig && + make -j $(nproc --all) targz-pkg + + This will generate a gzipped tar file whose name is printed in the last + line shown; for example, a kernel with the kernelrelease identifier + '6.0.0-rc1-local-g928a87efa423' built for x86 machines usually will + be stored as '~/linux/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz'. + + Copy that file to your test machine's home directory. + +* Switch to the test machine to check if you have enough space to hold ano= ther + kernel. Then extract the file you transferred:: + + sudo tar -xvzf ~/linux-6.0.0-rc1-local-g928a87efa423-x86.tar.gz -C / + + Afterwards :ref:`generate the initramfs and add the kernel to your boot + loader's configuration `; on some distributions the foll= owing + command will take care of both these tasks:: + + sudo /sbin/installkernel 6.0.0-rc1-local-g928a87efa423 /boot/vmlinuz-6= .0.0-rc1-local-g928a87efa423 + + Now reboot and ensure you started the intended kernel. + +This approach even works when building for another architecture: just inst= all +cross-compilers and add the appropriate parameters to every invocation of = make +(e.g. ``make ARCH=3Darm64 CROSS_COMPILE=3Daarch64-linux-gnu- [...]``). + +Additional reading material +--------------------------- =20 * The `man page for 'git bisect' `_ a= nd `fighting regressions with 'git bisect' `_ --=20 2.44.0 From nobody Sun Feb 8 11:43:14 2026 Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9321B7C0A6; Tue, 9 Apr 2024 07:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=80.237.130.52 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; cv=none; b=fP2pkBMfL+wbk6Zil0/fIr1Z0w3tbiRoHbfpxazXmxx2589NWxn5dOdsxfRe6qe68zwYE/u+xkUcHK9b8Wp2/dlO1ub72dNTR9MSonjhTGZJDdfx/hByA2qDH1pmLL+f9bg7QL2LK8zp6B/Biw0sV66LChL2SfpZ9bcJsCREavs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712647858; c=relaxed/simple; bh=Z7oJ0ojzxvgN6IlNaV9Gw82cLAFxnX0SPwkszNBnPDw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VUjL9h3KIOFed0mednWNY3eryJtkWEAAKOY9rajg5ULNSYIdh4jQvYoZY6Yi0MNZ5Perb/e64koeAigwn1q5YEweW7J5SEezFMERevYEV9+qYjhUjMfTakJ91lj/SPJCpcls4mCtgrLR89tEWETn42vyFSsSybAf0NUjSNNB0+g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info; spf=pass smtp.mailfrom=leemhuis.info; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b=dhxGVrzX; arc=none smtp.client-ip=80.237.130.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=leemhuis.info Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=leemhuis.info header.i=@leemhuis.info header.b="dhxGVrzX" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=leemhuis.info; s=he214686; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:From:Sender: Reply-To:Subject:Date:Message-ID:To:Cc:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:In-Reply-To: References; bh=uCCHKQw/1e/VvfCrnRGy2B0smnW1Q61lxcP9UuK98Q8=; t=1712647855; x=1713079855; b=dhxGVrzXWpFtkaaAyeN6WjFd3bp/Jbf2XDdvHgkByRX5av70OaRzkfD1R7hq0 2bPkMdfaM9u3o6/aR5EnFTAbS/j2TKR/DpnTYYVA/noL+3gqbyJ2TWT0T0kziIbqJzcW3rWNhKjM7 xpCRWklc3ZwMRVM1WUzoHJFsUydBAgL98Wmgr/vVVgUwsnFa2FkswSi0SSP3GwpKZmRwEntgQGSUS OEEVhM9jbbDDVYEBXHHfS4o359txMB9T6srYcb1KQ5intSgM5smAYFdIqBLixD99CcgF6OwGZrN88 4HXdMz7g/bk65ropsgMJBpf07hN4NGhiW0PUNK962TUySTczhQ==; Received: from ip4d148da6.dynamic.kabel-deutschland.de ([77.20.141.166] helo=truhe.fritz.box); authenticated by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) id 1ru5wR-0006Ld-Ny; Tue, 09 Apr 2024 09:30:51 +0200 From: Thorsten Leemhuis To: Jonathan Corbet Cc: regressions@lists.linux.dev, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, Bagas Sanjaya , =?UTF-8?q?Petr=20Tesa=C5=99=C3=ADk?= Subject: [PATCH v2 6/6] docs: verify/bisect: stable regressions: first stable, then mainline Date: Tue, 9 Apr 2024 09:30:49 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-bounce-key: webpack.hosteurope.de;linux@leemhuis.info;1712647855;18e5c630; X-HE-SMSGID: 1ru5wR-0006Ld-Ny Content-Type: text/plain; charset="utf-8" Rearrange the instructions so that readers facing a regression within a stable or longterm series first test its latest release before testing mainline. This is less scary for some people. It also reduces the chance that something goes sideways for readers that compile their first kernel, as mainline can cause slightly more trouble. Signed-off-by: Thorsten Leemhuis --- .../verify-bugs-and-bisect-regressions.rst | 91 +++++++++++-------- 1 file changed, 51 insertions(+), 40 deletions(-) diff --git a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.r= st b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst index 6193c797642732..c389d4fd7599df 100644 --- a/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst +++ b/Documentation/admin-guide/verify-bugs-and-bisect-regressions.rst @@ -309,31 +309,32 @@ machine; if you want to compile the kernel on another= system, check .. _rangecheck_bissbs: =20 * Determine the kernel versions considered 'good' and 'bad' throughout this - guide. + guide: =20 - Do you follow this guide to verify if a bug is present in the code devel= opers - care for? Then consider the mainline release your 'working' kernel (the = newest - one you regularly use) is based on to be the 'good' version; if your 'wo= rking' - kernel for example is 6.0.13, then your 'good' kernel is 6.0. + * Do you follow this guide to verify if a bug is present in the code the + primary developers care for? Then consider the version of the newest k= ernel + you regularly use currently as 'good' (e.g. 6.0, 6.0.13, or 6.1-rc2). =20 - In case you face a regression, it depends on the version range where the - regression was introduced: + * Do you face a regression, e.g. something broke or works worse after + switching to a newer kernel version? In that case it depends on the ve= rsion + range during which the problem appeared: =20 - * Something which used to work in Linux 6.0 broke when switching to Linux - 6.1-rc1? Then henceforth regard 6.0 as the last known 'good' version - and 6.1-rc1 as the first 'bad' one. + * Something regressed when updating from a stable/longterm release + (say 6.0.13) to a newer mainline series (like 6.1-rc7 or 6.1) or a + stable/longterm version based on one (say 6.1.5)? Then consider the + mainline release your working kernel is based on to be the 'good' + version (e.g. 6.0) and the first version to be broken as the 'bad' o= ne + (e.g. 6.1-rc7, 6.1, or 6.1.5). Note, at this point it is merely assu= med + that 6.0 is fine; this hypothesis will be checked in segment 2. =20 - * Some function stopped working when updating from 6.0.13 to 6.1.5? Then= for - the time being consider 6.0 as the last 'good' version and 6.1.5 as - the 'bad' one. Note, at this point it is merely assumed that 6.0 is fi= ne; - this assumption will be checked in segment 2. + * Something regressed when switching from one mainline version (say 6.= 0) to + a later one (like 6.1-rc1) or a stable/longterm release based on it + (say 6.1.5)? Then regard the last working version (e.g. 6.0) as 'goo= d' and + the first broken (e.g. 6.1-rc1 or 6.1.5) as 'bad'. =20 - * A feature you used in 6.0.13 does not work at all or worse in 6.1.15? = In - that case you want to bisect within a stable/longterm series: consider - 6.0.13 as the last known 'good' version and 6.0.15 as the first 'bad' - one. Note, in this case you still want to compile and test a mainline = kernel - as explained in segment 1: the outcome will determine if you need to r= eport - your issue to the regular developers or the stable team. + * Something regressed when updating within a stable/longterm series (s= ay + from 6.0.13 to 6.0.15)? Then consider those versions as 'good' and '= bad' + (e.g. 6.0.13 and 6.0.15), as you need to bisect within that series. =20 *Note, do not confuse 'good' version with 'working' kernel; the latter t= erm throughout this guide will refer to the last kernel that has been working @@ -392,19 +393,13 @@ machine; if you want to compile the kernel on another= system, check =20 .. _stablesources_bissbs: =20 -* Retrieve the sources for any stable or longterm series you might need. - - Is the version you earlier established as 'bad' a stable or longterm rel= ease? - Then download the code for the series it belongs to ('linux-6.1.y' in th= is - example):: +* Is one of the versions you earlier established as 'good' or 'bad' a stab= le or + longterm release (say 6.1.5)? Then download the code for the series it b= elongs + to ('linux-6.1.y' in this example):: =20 git remote set-branches --add stable linux-6.1.y git fetch stable =20 - If the version earlier established as 'good' is from a different stable = or - longterm series (say 6.0.13), repeat the previous step, but this time fo= r the - branch holding the series the 'good' version belongs to (e.g. linux-6.0.= y). - .. _oldconfig_bissbs: =20 * Start preparing a kernel build configuration (the '.config' file). @@ -545,10 +540,24 @@ be a waste of time. [:ref:`details `] =20 .. _checkoutmaster_bissbs: =20 -* Check out the latest Linux codebase:: +* Check out the latest Linux codebase. =20 - cd ~/linux/ - git switch --discard-changes --detach mainline/master + * Are your 'good' and 'bad' versions from the same stable or longterm se= ries? + Then check the `front page of kernel.org `_: if it + lists a release from that series without an '[EOL]' tag, checkout the = series + latest version ('linux-6.1.y' in the following example):: + + cd ~/linux/ + git switch --discard-changes --detach stable/linux-6.1.y + + Your series is unsupported, if is not listed or carrying a 'end of lif= e' + tag. In that case you might want to check if a successor series (say + linux-6.2.y) or mainline (see next point) fix the bug. + + * In all other cases, run:: + + cd ~/linux/ + git switch --discard-changes --detach mainline/master =20 [:ref:`details `] =20 @@ -650,15 +659,15 @@ be a waste of time. [:ref:`details `] =20 .. _recheckstablebroken_bissbs: =20 -* Are you facing a problem within a stable/longterm series, but failed to - reproduce it with the mainline kernel you just built? One that according= to - the `front page of kernel.org `_ is still supported= ? Then - check if the latest codebase for the particular series might already fix= the - problem. To do so, check out that series latest version (again, this her= e is - assumed to be 6.0):: +* Did you just built a stable or longterm kernel? And were you able to rep= roduce + the regression with it? Then you should test the latest mainline codebas= e as + well, because the result determines which developers the bug must be sub= mitted + to. + + To prepare that test, check out current mainline:: =20 cd ~/linux/ - git switch --discard-changes --detach linux-6.0.y + git switch --discard-changes --detach mainline/master =20 Now use the checked out code to build and install another kernel using t= he commands the earlier steps already described in more detail:: @@ -680,7 +689,9 @@ be a waste of time. [:ref:`details `] uname -r cat /proc/sys/kernel/tainted =20 - Now verify if this kernel is showing the problem. + Now verify if this kernel is showing the problem. If it does, then you n= eed + to report the bug to the primary developers; if it does not, report it t= o the + stable team. See Documentation/admin-guide/reporting-issues.rst for deta= ils. =20 [:ref:`details `] =20 --=20 2.44.0