From nobody Fri Sep 25 09:19:53 2026 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011032.outbound.protection.outlook.com [40.93.194.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A03CF175A8A; Tue, 15 Sep 2026 00:07:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.32 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789430870; cv=fail; b=Yg+44Gk4PJ2KXLlwD/4DPR+kuS5fhIs16KVs30U+fPqOaV6ESS2gUzGCKdbf+TRw2RdnAV46DFIw9NPM3k8laXEj9v1f8NfyinT49cXqJDxPQed0XndIiNdpj76PW+pU+io1HSuWnoLm6r1ZsFtFEqc6vwJfyqHhLl07aLNT9GI= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789430870; c=relaxed/simple; bh=I8ThdAhcB7Nfj5O1GKCuaKG1TRSWnSga2c+2M8UnhEg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=bSNU77Be4OyU0nAvgLzQbfp55VfagudTFjUCaXOZLznqyD6oPfgyCM/BlRuTJSLYSB3YPNE8+jsGjXl9/bEC8lwi32+MdF4fexsBT0k9TOXqw1uG5edvVIGRlaRquZPPQ4yQj0+v3gw3pA8CCAjOsfe7KUPwBLa/qhTsX7Yx7ZQ= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=FsKsBbtn; arc=fail smtp.client-ip=40.93.194.32 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="FsKsBbtn" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nCxwH3FRPsVSNwLwtiiIeoptSJ8DiIyms7TvdwmphGwoMPiwK7WzEm/3yaAans6rCsUh9O6+SLjWFT1wKIjJ2FCvYNWdFTh+hWEnvkrcJKRu6hbMRh5Q9DMj5d+61JnYewdjGWtzjmWSTtFgscnYbRIEQoAi+sQzsyMaLOlDxBuJIFCCmWqyFIOMUephrJmbs8dOPvnq6XPT+C88ESR144kQPcCsfeU6V0S5MVbYF5l4OMYmHl5eVSoyOyWDX59tGk2AoI5ZmZP7Jokathj4oC3BbmemtxNKnXcgq07UKiKe1Tzap9LQcHQRzx+x/YPOnJE/9OH7LtLA8PCsh7VgwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9xkMU347kaSfWFryZeqGC40DMaph4QovCAQOdybBudA=; b=M2boUwiXZHeWGEJLGYzCF7DkT9VhUwg/Z4dDhdmJgFd/JHDLCJ5y0z9Qg10z3jdDA7ZsUMh1pw5KPWfkoH3my8f5jw0bXMswjWiKSWz0QerMkQDfVxzWT7nrC9x843lxdqqW+E/rJgKj8srrQuoXwCD0ygdLsnqQ1MQv3j83afcyT9SOraz90fU0Jpydj1b/E+SZ7zZ36xixmcxRpkZ239jD7GnBVmCcpQchN/TU1O4sMaUqD/KRMLE61mlEd+qGPBiyRaNiN9Cobje9522mIclVAIswc15iRusENMY9YOyn3TKIK4YU+x/3V4JruIGOvhq10Nlf2Ud/AAkLnozr5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9xkMU347kaSfWFryZeqGC40DMaph4QovCAQOdybBudA=; b=FsKsBbtn3LlOmI3YNV7VcFG8V3HhLuxktqdLbfIX8B9qAdzivYMgDfh98CNBMTdHg/JfE1UxSQVay2yEIJNvyKyn5hV3FNQ4GCGdJ7YhPhgcEx58CNPjA+qMotYPhHzXBoJPaLhkymcO6AfenRPpO7WXt5kUftrzAGH/OmjGKcu+UkbHGsuAj+p9l4iJSmIREE64Y8fe6TfmS7jq0DGbB3joBi/7IhHBTn+eG5iAC4rAaAMHThaMiVu0rM8EN1s/dg/0pUp9e1dMXBOvqJKKaZgWSLapYIYFXv3iuU89IJcBapX3q2wVA5ODHGlXSC/cs8DFQTj3ksgjW67DVnItWg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) by LV2PR12MB5823.namprd12.prod.outlook.com (2603:10b6:408:178::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.13; Tue, 15 Sep 2026 00:07:42 +0000 Received: from LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286]) by LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286%5]) with mapi id 15.21.0406.007; Tue, 15 Sep 2026 00:07:42 +0000 From: Yury Norov To: Nathan Chancellor , Nicolas Schier , linux-kbuild@vger.kernel.org Cc: Yury Norov , Jonathan Corbet , Shuah Khan , Randy Dunlap , Linus Torvalds , Lorenzo Stoakes , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/2] make: parallelize tags generation Date: Mon, 14 Sep 2026 20:07:34 -0400 Message-ID: <20260915000735.1146790-2-ynorov@nvidia.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260915000735.1146790-1-ynorov@nvidia.com> References: <20260915000735.1146790-1-ynorov@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: BN9PR03CA0305.namprd03.prod.outlook.com (2603:10b6:408:112::10) To LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV3PR12MB9356:EE_|LV2PR12MB5823:EE_ X-MS-Office365-Filtering-Correlation-Id: 7ca2b27f-3340-4b8c-d4dc-08df12bd5ce3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|7416014|376014|366016|10067099003|56012099006|5023799004|11063799006|22082099003|18002099003|6133799003; X-Microsoft-Antispam-Message-Info: DIytuZyKATAVfFTjlMduBbhFOmuB1K/Ow9d5tZZc/V1wlgfkDwXFBqbfQFJ6zOr6mAX8n9FGorZ0k2qkR28iTWEc4tuxPcM7u+netMY4TPExSTSiZxJ4/FrUuQZBW3JfPBjzS6DNJ5Hw3C8VP7MTgGRZdnrRPueGeaaRfhYFwT+rQZfLulOFejtEW3+rkLFIv8nUMzxuAn1oMMwobssRrqi9GQ0AlLb4ZyZyPVYhqlJp9WAE0fyZDv3WyElLYrxAw9W6hZ1g6QysFbGSjJ7GalLqP2YobDaJyz/Xn2i9WIpEEyo8IfsfpvGoC72LqxSTPsKmA3PZ2XDxKRKwEm/n7hAPEdMBwifUzmF+RuOr/ZmmG7Kx5d1YAAXJ0GXG0CgsFLDHp6gmUeJZTlKf1B91FNONkY0qv3Unpm4VzgJbfTv8lVlicxyqB4N0siJJZTN1XB+sjuRDy2G9N4IxzMcveSSd8GxEQnIgVI8ml3CjwYoHx7jRhsLp+W7E5V5iy5J/Fxeil3vMdOVBpOOll1XAbPGJZQHkP/UOE/Arc+HVwFG63MAc5Nf+VkFGdHpPvc9+F9S6qGGy7RZDfY36a2VNkniWFLCbGvISyq2LmHyRzugOqiA7Q+gQIv3Ss5bu9swSAy+7X8NQZsJnWJPHOttWHPF/N7SPG+cz1U2YQSTTH9Y= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV3PR12MB9356.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(7416014)(376014)(366016)(10067099003)(56012099006)(5023799004)(11063799006)(22082099003)(18002099003)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?K9ZfhKdSek8g/4E5ae7FvQTUF6I1xlCbHjtgCw803qUV7cCYAr6iFGfbuIO4?= =?us-ascii?Q?Sh3XTPMzg3sGu/wj81xIKJuQyIxzqz1ERHF1pcWodullBYWHBL76mjmiyPsU?= =?us-ascii?Q?SXh0LFTSSOsFTgmz/THwNCYtjM9jqnJykWjs4g5FLfFP1pKGs+2BLp/6Yek/?= =?us-ascii?Q?NvZDSLM3qDghrbLclec2EL6s6yCX7gP3WUYFpWK9g+rw0Wg/QQs1F5l1LVFo?= =?us-ascii?Q?0HrDOSauwRVZTA+PFW5K4lnUPDBh1zvOFq9Gvz9V8ATtnFneLW+jsV9NA6k7?= =?us-ascii?Q?PTvohQ2343qgv8ou1gvePVojGyi9dl8kDVgwVJeqDBX2qHyjT888UMFBU5/s?= =?us-ascii?Q?3kaHYpijE9HlAXOk5dbLrOqIwUt6cWQieTyxXZLepE7XphE0unxm6mONWFPX?= =?us-ascii?Q?RkDiQS97SvUNihNsZBUvMpxthEVnqcPc5DdR+cxEbyqNJ/1EH+Tw6YjwPHq9?= =?us-ascii?Q?2mcPfMX989zu/KlMxCrXFrzKksaTkDdgk8vEbAAaHCCZgQ2Zn5wuHKJThlFZ?= =?us-ascii?Q?QRxzkokxjU/h8Gm8YzkQAFv3zFBzcTd9UNUPMcFpkctdd4k1ktqrMk9uAnp7?= =?us-ascii?Q?m+ylk5X1Gu+p6LTZjN4uC6faAPe79/np5j0MH1rNNE02Id/9V2Ks9RhFb/ZQ?= =?us-ascii?Q?BlC3ACQruSH4IZg4POvQtt2Tz4cjX/bfROBI8u48mNusYMf6MZ6ZR9R1UQ2W?= =?us-ascii?Q?mCycWFMkTk9dgj9smvrPOB+vT93ugrC+RFwfRDONijavMqcKoSpUSw/RmDYS?= =?us-ascii?Q?Gz7dIVKyNHKTa9FtjWMpKGVgXfY3CV4sTWnLufNCvXW6BJfdbor8iYzN1XqM?= =?us-ascii?Q?bHVWzjNXTa8F/t4mpo6KhNIHJ/7x7s5sPHbMqJN/k+SkRXbM6UyshRpmIhhx?= =?us-ascii?Q?PNTTihGo3recK6BOV1PiQYqTvLaviPIY9n/3mcQo+wVJC492c7WT2UuVpRT9?= =?us-ascii?Q?2r/fCU7OyKwYV5P1j7RgA2O9jyKHEzDETXky33Oy3mh3dkjAq6cQRnBk4TEC?= =?us-ascii?Q?RgamznftM1NI2sVYSptGShf88Yuu8Ii9fUBBKBS7XmAkXDpZZf1KTcZxsCoS?= =?us-ascii?Q?ZhcOUFKk8lxZM6fgKVBIOWLMgAnKSYpHl2ehcBteefNbhXlwVaXbq0DH+2bz?= =?us-ascii?Q?1kEn0aXGa7sSgMy0HhcrD2/ymMzZEq84yOaih/ODpJsiNcdkz/smKCc0MwB3?= =?us-ascii?Q?OiEkOoOZehiddpKQdqXBRgpMLNS1mmtjOR6r0isSmii/H3hJlpIErIY7v/zA?= =?us-ascii?Q?z/TLNFJ3iwX2Nt9kunqQ6rUbS5paKSutX2APG8OLujhRoch9CwAMULb4QfOI?= =?us-ascii?Q?y3V98OnKDx7pE8XmgdyHmyfdZN2705kTaur3PUtS9FICg+l76Jek7xL8wCvS?= =?us-ascii?Q?jDAvyhr/imNhuhW8D+sv+RD5NdahQoGQHRO4xmAdGjFNJ6BcdIUlx38r9oTz?= =?us-ascii?Q?yfi78XuFhaeeuTK2XGQ+n9d++Qvoy024GqBqweoD8BfuoH8rivvH4/Ngf3eB?= =?us-ascii?Q?8AlBKjGMTs53Vr2cgMccGEeL0+AOCn70BrCDe4gnwFBi8dcpVYZQQNcGgoY9?= =?us-ascii?Q?Q8M0iMLx5ZThtz4lVly5af9k/A9GVey5+a60IIgnxD9wTD8q1+FTJgHFxR/y?= =?us-ascii?Q?yr5TAg+LdjV/yFTP2E7Ua4Jg3sXxknVNsA169dtC1kvynUQLv7mjQwSg2xtc?= =?us-ascii?Q?S9tI21nwnAn7ICvrVVVcRCLwgZgdW3J4l/BnBHOwZhHaG3V12QDWBDi3BbKu?= =?us-ascii?Q?6OvR5BLzyQ=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7ca2b27f-3340-4b8c-d4dc-08df12bd5ce3 X-MS-Exchange-CrossTenant-AuthSource: LV3PR12MB9356.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2026 00:07:42.7785 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jdbNZeQ6tgx/ofc3kBKVV2gjyumlyZFbetAfxJW41nrjXoT5VUJCmX1ll28VM7wjjmWapuXSWmEHI8Fw5ZLcFQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5823 Content-Type: text/plain; charset="utf-8" Generate per-directory tags in parallel with Exuberant or Universal Ctags. A recursive make shares the caller's jobserver, allowing make -jN tags to schedule directory jobs concurrently, then join them to get the traditional tags file. Allow compiled-source discovery to find no source references in a batch of .cmd files, while still propagating grep errors. The performance of the current vs parallel tags generation with time make -j8 ALLSOURCE_ARCHS=3Dall tags is: real user sys Before: 4m40.797s 4m14.229s 0m39.341s After: 1m39.203s 9m1.450s 2m7.552s Assisted-by: OpenAI Codex Signed-off-by: Yury Norov --- Documentation/kbuild/kbuild.rst | 9 +++ Makefile | 5 +- scripts/Makefile.tags | 9 +++ scripts/tags.sh | 137 +++++++++++++++++++++++++++++--- 4 files changed, 148 insertions(+), 12 deletions(-) create mode 100644 scripts/Makefile.tags diff --git a/Documentation/kbuild/kbuild.rst b/Documentation/kbuild/kbuild.= rst index 5a9013bacfb7..61587adeedba 100644 --- a/Documentation/kbuild/kbuild.rst +++ b/Documentation/kbuild/kbuild.rst @@ -317,6 +317,15 @@ To get all available archs you can also specify all. E= .g.:: =20 $ make ALLSOURCE_ARCHS=3Dall tags =20 +With Exuberant or Universal Ctags, ``make -jN tags`` generates tags for +directories in parallel and merges them into a single sorted ``tags`` file. +The jobs share make's jobserver with other build targets. For example:: + + $ make -j8 ALLSOURCE_ARCHS=3Dall tags + +Per-directory tag files are temporary and are removed after merging. +Every invocation regenerates the complete tags file. + IGNORE_DIRS ----------- For tags/TAGS/cscope targets, you can choose which directories won't diff --git a/Makefile b/Makefile index 66654fa71655..6f3945fa3fc6 100644 --- a/Makefile +++ b/Makefile @@ -2260,7 +2260,10 @@ clean: $(clean-dirs) quiet_cmd_tags =3D GEN $@ cmd_tags =3D $(BASH) $(srctree)/scripts/tags.sh $@ =20 -tags TAGS cscope gtags: FORCE +tags: FORCE + +$(call cmd,tags) + +TAGS cscope gtags: FORCE $(call cmd,tags) =20 # Generate rust-project.json (a file that describes the structure of non-C= argo diff --git a/scripts/Makefile.tags b/scripts/Makefile.tags new file mode 100644 index 000000000000..d5967a3ac496 --- /dev/null +++ b/scripts/Makefile.tags @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: GPL-2.0-only +# Directory tag shards share the parent make's jobserver. + +.PHONY: tags +tags: $(shell cat $(tags_tmp)/shards) + $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-merge $(tags_tmp) + +%.tags: %.files + $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-worker $@ diff --git a/scripts/tags.sh b/scripts/tags.sh index 41e38df96984..b001e0f78d2f 100755 --- a/scripts/tags.sh +++ b/scripts/tags.sh @@ -121,8 +121,12 @@ all_compiled_sources() { { echo include/generated/autoconf.h - find $ignore -name "*.cmd" -exec \ - grep -Poh '(?<=3D^ )\S+\.([chS]|rs)(?=3D\s)|(?<=3D=3D )\S+\.(?1)(?=3D$= )' {} \+ | + # A .cmd batch with no source references is not an error. + find $ignore -name "*.cmd" -exec sh -c ' + pattern=3D$1 + shift + grep -Poh "$pattern" "$@" || [ "$?" -eq 1 ] + ' sh '(?<=3D^ )\S+\.([chS]|rs)(?=3D\s)|(?<=3D=3D )\S+\.(?1)(?=3D$)' {} = \+ | awk '!a[$0]++' } | xargs realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative= -to=3D.) | sort -u @@ -130,7 +134,9 @@ all_compiled_sources() =20 all_target_sources() { - if [ -n "$COMPILED_SOURCE" ]; then + if [ -n "$tags_input" ]; then + sed -n 's/^s //p' "$tags_input.files" + elif [ -n "$COMPILED_SOURCE" ]; then all_compiled_sources else all_sources @@ -139,6 +145,10 @@ all_target_sources() =20 all_kconfigs() { + if [ -n "$tags_input" ]; then + sed -n 's/^k //p' "$tags_input.files" + return + fi find ${tree}arch/ -maxdepth 1 $ignore \ -name "Kconfig*" -not -type l -print; for arch in $ALLSOURCE_ARCHS; do @@ -282,12 +292,24 @@ setup_regex() done } =20 -exuberant() +setup_ctags() { CTAGS_EXTRA=3D"extra" if $1 --version 2>&1 | grep -iq universal; then CTAGS_EXTRA=3D"extras" fi + CTAGS_KCONFIG=3Dn + if $1 --list-languages | grep -iq kconfig; then + CTAGS_KCONFIG=3Dy + fi + export CTAGS_EXTRA CTAGS_KCONFIG +} + +exuberant() +{ + if [ -z "$CTAGS_EXTRA" ]; then + setup_ctags "$1" + fi setup_regex exuberant asm c # identifiers to ignore by ctags local ign=3D( @@ -312,16 +334,75 @@ exuberant() static ) all_target_sources | \ - xargs $1 -a -I "$(IFS=3D','; echo "${ign[*]}")" \ + xargs -r $1 -a "${tags_flags[@]}" -I "$(IFS=3D','; echo "${ign[*]}")" \ --$CTAGS_EXTRA=3D+fq --c-kinds=3D+px --fields=3D+iaS --langmap=3Dc:+.h \ - "${regex[@]}" + "${regex[@]}" || return =20 KCONFIG_ARGS=3D() - if ! $1 --list-languages | grep -iq kconfig; then + if [ "$CTAGS_KCONFIG" !=3D y ]; then setup_regex exuberant kconfig KCONFIG_ARGS=3D(--langdef=3Dkconfig --language-force=3Dkconfig "${regex[= @]}") fi - all_kconfigs | xargs $1 -a "${KCONFIG_ARGS[@]}" + all_kconfigs | xargs -r $1 -a "${tags_flags[@]}" "${KCONFIG_ARGS[@]}" +} + +# Call in a subshell so error handling and cleanup stay local to the opera= tion. +setup_tags_tmp() +{ + set -eo pipefail + tmp=3D$(mktemp -d .tmp_tags.XXXXXX) + trap 'rm -rf "$tmp"' EXIT + trap 'exit 1' HUP INT TERM +} + +# Let recursive make schedule directory shards using the caller's jobserve= r. +parallel_tags() +( + local tmp + setup_tags_tmp + # Workers inherit these capabilities instead of probing for each director= y. + setup_ctags ${CTAGS:-ctags} + + { + all_target_sources | sed 's/^/s /' + all_kconfigs | sed 's/^/k /' + } | LC_ALL=3DC sort -u | awk -v tmp=3D"$tmp" ' + { + dir =3D substr($0, 3) + if (!sub(/\/[^\/]*$/, "", dir)) + dir =3D "." + if (!(dir in ids)) { + ids[dir] =3D ++n + print tmp "/" n ".tags" + } + out =3D tmp "/" ids[dir] ".files" + if (out !=3D previous) { + if (previous !=3D "") + close(previous) + previous =3D out + } + print >> out + }' > "$tmp/shards" + ${MAKE:-make} -f "${tree}scripts/Makefile.tags" tags_tmp=3D"$tmp" +) + +merge_tags() +( + local tmp + setup_tags_tmp + + # Read filenames from stdin to avoid command-line length limits. + { + tr '\n' '\0' < "$1/shards" + # sort requires at least one input, even when there are no shards. + printf '/dev/null\0' + } | LC_ALL=3DC sort -m -u --files0-from=3D- | cut -f2- > "$tmp/merged" + mv "$tmp/merged" tags +) + +remove_struct_forward_declarations() +{ + LC_ALL=3DC sed -e '/^\([a-zA-Z_][a-zA-Z0-9_]*\)\t.*\t\/\^struct \1;.*\$\/= ;"\tx$/d' "$@" } =20 emacs() @@ -366,11 +447,45 @@ case "$1" in ;; =20 "tags") - rm -f tags - xtags ${CTAGS:-ctags} + # Recursive recipes also run in dry-run, touch and question modes. + # Only normal invocations may generate intermediate files. + case ${MAKEFLAGS%% *} in + *n*) exit 0 ;; + *t*) touch tags; exit $? ;; + *q*) exit 1 ;; + esac + if ${CTAGS:-ctags} --version 2>&1 | grep -Eiq 'exuberant|universal'; then + parallel_tags + exit $? + else + rm -f tags + xtags ${CTAGS:-ctags} + fi remove_structs=3Dy ;; =20 + "tags-worker") + tags_input=3D${2%.tags} + # Never expose an incomplete shard after an error or interruption. + setup_tags_tmp + tags_flags=3D(-f "$tmp/tags" --sort=3Dno --tag-relative=3Dno) + exuberant ${CTAGS:-ctags} + # Sort once per changed directory. Prefix records so the final merge + # keeps pseudo-tags ahead of all regular tag names. + remove_struct_forward_declarations "$tmp/tags" | awk ' + { + sub(/^!_TAG_FILE_SORTED\t0\t/, "!_TAG_FILE_SORTED\t1\t") + print (/^!_TAG_/ ? "0\t" : "1\t") $0 + }' | LC_ALL=3DC sort --parallel=3D1 -u > "$tmp/sorted" + mv "$tmp/sorted" "$2" + exit 0 + ;; + + "tags-merge") + merge_tags "$2" + exit $? + ;; + "TAGS") rm -f TAGS xtags etags @@ -380,5 +495,5 @@ esac =20 # Remove structure forward declarations. if [ -n "$remove_structs" ]; then - LC_ALL=3DC sed -i -e '/^\([a-zA-Z_][a-zA-Z0-9_]*\)\t.*\t\/\^struct \1;= .*\$\/;"\tx$/d' $1 + remove_struct_forward_declarations -i "$1" fi --=20 2.53.0 From nobody Fri Sep 25 09:19:53 2026 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011006.outbound.protection.outlook.com [40.93.194.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCE971A6814; Tue, 15 Sep 2026 00:07:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.6 ARC-Seal: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789430875; cv=fail; b=QvrFn256kjA0MwllMBqnDBcVNFR7Z/zZqsVWRtVwDfHvwMY7FYI+Xm+oskwYPLB13/8KRgKoID29BOVU43kLKT4b4CtPUgJdPAHR4cwjmb/q9Vh63dwXvJA2qSOigjWPwB7GROXNA6nKVvFynkJgqLMZrljlNyjtnwuvfV6taeE= ARC-Message-Signature: i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789430875; c=relaxed/simple; bh=YtmAwkg/MIjt7uW5Aw2EhmBWgG82W1vZ+XWQvjPYyuM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ouZOYK6Z4xCFLlTgHAwlqMx+Y/2taX9LocdynYnPNVzhBvUiDbVxicve6FAnOSXAXU76XPYOeSN8X6Lp9P4l5wSnF1/SH2vS3PVLWzAgotP2x/i8emyrouJx+TttR2IChMAF22dRPnsPnZ7UFGGjbzJvvhXHZjk1VSDpdWONlDU= ARC-Authentication-Results: i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=FkUzDc7U; arc=fail smtp.client-ip=40.93.194.6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="FkUzDc7U" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=odnPBL0F0fgTHTymRqsK4VP5/lzGSyTk/dWF9VD8axQbfCWpDnkEzK+u0++OwLqX5uaTRa1m79huDD22KWSRLz66VuuYuYjWFsay4rRW1/9XW4I/G6rLMzFKBXRLmAlrwHWLSgI+OTZyMWr4Ml48NvE6nJWgoDD5Hl62/gavtHV01Gw/V6KtaVdH82MxNx3fFM78FysGuLQv+k0h2Q9cCt0eTocZ9o/fNPMIJh2GGWJeobQuwyPUkFQccjXDxiZLllvXKCN6eJEJsjb4LVb10qV6KJ5rum6WN6q38YXyCJZLm4pJTVjWJJnoOE/8oxIOzvpXGITeqcUpaPaSze6JLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nBD0RMU7RV+jffkCzbcjdc2pe1wj2UEoOLyWiu9GBbM=; b=t9AJxbysu4jFLm4YjLOgWrw7c7sdYHci3GrcaiAak3NuFGTmW8zOu+3oP9b8FlUaRGm8IiDe0XzHkGapGIwd9lJhB89TfwoGFU+ZlpMpCt62cX6Itd6O4mCd29XEnk/kRGgiC/2a5o3nWnb8gBOnZCDnMsMrvDQZJhvyb4dnYuh1Xx47qLnluDpZUuEcI4XqYA2ccz5b/tRKNEfw3vB4qBG6NqibiWla9Vxi+SdcaYYYvyDRjQOaThpbe8M116t9FEEh9Rgg+PASPfuPaxp3yOxMbjN8bdkCW7lAQj1wo/UjEUUrV25bBi6jg6tsC0cOC7VwSdiOJ3wUx4XIHYDuRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nBD0RMU7RV+jffkCzbcjdc2pe1wj2UEoOLyWiu9GBbM=; b=FkUzDc7UNcGQhGRC0pKl7DlGova5LYvH0i3y/O1mXEL6B1bhJv5ecu0BQc9j9GU6cI9u9o6hs3jincXEMpfBjw9D5WygBXAYDUC+ruCAMNsX5tRLiOgTL1wPwX7fe/E43JJqxXIomSmGVFPoKcei1Hz4g3EzQVVm4BvniEBf6mJfOL+NQ5Xo9wCGdlRQt0lQe4f4ieOWNElap3LBSP7kIXOEgwd9NuBd4NSVm1hbpLBeUDYW4YhSgwJreKb2I7FgO42PI9SvaFEbBzv/zeoCDuKNaQW9qbtC00mtzXEgpJ/XXE7eRZCfvhxSopFyw9yB8P/DIRkZIEb+udufq2liDA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) by LV2PR12MB5823.namprd12.prod.outlook.com (2603:10b6:408:178::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.13; Tue, 15 Sep 2026 00:07:47 +0000 Received: from LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286]) by LV3PR12MB9356.namprd12.prod.outlook.com ([fe80::1c36:31b4:c420:6286%5]) with mapi id 15.21.0406.007; Tue, 15 Sep 2026 00:07:46 +0000 From: Yury Norov To: Nathan Chancellor , Nicolas Schier , linux-kbuild@vger.kernel.org Cc: Yury Norov , Jonathan Corbet , Shuah Khan , Randy Dunlap , Linus Torvalds , Lorenzo Stoakes , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/2] make: cache per-directory tags Date: Mon, 14 Sep 2026 20:07:35 -0400 Message-ID: <20260915000735.1146790-3-ynorov@nvidia.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260915000735.1146790-1-ynorov@nvidia.com> References: <20260915000735.1146790-1-ynorov@nvidia.com> Content-Transfer-Encoding: quoted-printable X-ClientProxiedBy: MN2PR15CA0044.namprd15.prod.outlook.com (2603:10b6:208:237::13) To LV3PR12MB9356.namprd12.prod.outlook.com (2603:10b6:408:20c::21) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV3PR12MB9356:EE_|LV2PR12MB5823:EE_ X-MS-Office365-Filtering-Correlation-Id: 9acb0992-dd90-404e-b415-08df12bd5ec6 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|23010399003|1800799024|7416014|376014|366016|10067099003|56012099006|5023799004|11063799006|22082099003|18002099003|3023799007|6133799003; X-Microsoft-Antispam-Message-Info: 3RbalKjtZbUz4osf1f6VFGuOs4ztqwuEd1Ky0dFQpuIp18l9ajek3P2ieTHFTtxZQK2CmSCdN1IaW5scAy0OdEi5FNHofs5nnT7TQZDMn67+vT3Ia/y3IFVZbwBXP2WNLJIhCIRctUw+esZFzlVQYSpfMkavMmiPUcodwmSOL++yv9Aj3K0Qyi3cACF2iHxV6dj2/gtQoDilHW+aqhqeF5kGekLpFVUAzV1fDkitIs0Ra9YwhCSEHFLnOM4VwhwYYNfbx3L16dzdx1Kf3mlHbLzKG95MdyCj+LlXE9e/HfaVkuiV6pRn8GZFF3QWCFi6Co89siEHc8lKWNAHd9DyNiXQtURaxkcIvcyZPLtzkMMTYHZqcrdUmKZnoroVYQgQvaK7NtBHcx/jiSbeGsVVrqKV5L8l+k3tcKyrreQwhYJhZU1dJHLN9wuZq8CC0xFOsr2RQ93NCcSDn4tsCuu4+4bepKZkCce6Uyc1+Wdk3Q+YmKLI+t8ckyfkbwwY4H3pQvt2eeJKIQE6LwmvhMQVcQ7/NnStJPrdabwqggX8xfbicM9WazSK+r0S7UE1Vn5HjzMbAM1MSAeNpGEPmNboRgxGVYYKglffYzVIA2hjOHQsgjoLOLZqJuREslrAaOSISnItnQlawrwTQuffQvVx0zaceh86hFTqF8iyKDn8H48= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV3PR12MB9356.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(23010399003)(1800799024)(7416014)(376014)(366016)(10067099003)(56012099006)(5023799004)(11063799006)(22082099003)(18002099003)(3023799007)(6133799003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?oYhKnlupSJ8y6X+XK6d1FTswGPV+QxXUiF9H/SOoq9A/PX3dVDWLIjY18zsF?= =?us-ascii?Q?n2CgYO7K0GYffLoyUc7StP5MN7IqZo63byuIyr1iDZyQWJCsLO8XZhrhIcnG?= =?us-ascii?Q?3e4dd/IeuCYO+/PGwQIEkKtLOf5sG+0VWcmQbaAmmOR9cOh2Jc4EhT/Hpkzq?= =?us-ascii?Q?XI/CQ6cNjGkLqQ2fccU0ZUUumItqoOqDqeNJVn4ZnFt+PHVTHUIOxShDhqX0?= =?us-ascii?Q?KKk5Y2YY0TdvrrpcaN88/WYt/PbQPp0cZBbP8baXJGY0pfOBfLOUBRLdnyk8?= =?us-ascii?Q?YWT/cGHBCzQL/X3upCFmm6l5teubyLBw2jIPBMd0roj35ivDTOv7BvijVKl3?= =?us-ascii?Q?GpmE/vRFiPX0ecm4h9AUt8/ylOpXFY81L34YC8IBFpfPPBb1txHy16yia+bv?= =?us-ascii?Q?Z/crd8oRPe5M5kHc6OQJeHYxcShir31hE2WiwfJ8rU0XlcfdtBtvCf9gQhXT?= =?us-ascii?Q?j1CbYh6NZXU5o7irEWwjr9geQEFo16xZt+u2thtLyoUX9WVEMWZ+gI2kowb1?= =?us-ascii?Q?mCcl6kSg/E5Bc6ZgR57rdxHRdLQpoJ4wIqBIuyxqLi6yDjWdjPmQml2VdBpg?= =?us-ascii?Q?8REESwKv77WiPEgdU4IVBaaw1UB3q4WySFVd6UAi47qjppnoozHUVLbYu4zE?= =?us-ascii?Q?FOgpCTOFE+s8O/J/LI7w8kzj+tgJDxMmiCMedqMSLoqpwpLK0T3ztx8n0UB5?= =?us-ascii?Q?zPARRSVqIclhf1FujYYbxcUHUdN5LHrxQ3298+j0No6shRcJSZlN8R/qdnkP?= =?us-ascii?Q?JtVjrGKxhBmhb9NqvGVksTT1bFcXP80wD2/KG067LlOaU854AUqol5ln7/Lh?= =?us-ascii?Q?78YKyTb1J0Qbqea2TgGft6nNtRwmmtyiSwsV8gx/FFn6ycKArdlW2c8eqr8r?= =?us-ascii?Q?pPxE+EVYUgXkeryKKb0y67PWRI8PCw6Ov1ZUTxgxmb+UpA4LaFZRmgk1U32C?= =?us-ascii?Q?Vt+4KE5AzAh/stCH2oKO80gsbMeHfoGhwXXauPYfQnwrlzTW16ABnmwtTciU?= =?us-ascii?Q?3mLzPov/hxuXaEHIY1LYXQKdQ2uWunpcCt8CXUzdu3Nn4NG9Fmd/qsUsHWPr?= =?us-ascii?Q?5R1QhUkS0zevEoUlypDy6Aw2gvf4a8jVsB1oGC366o69dQhUDO3mCKr6RbuM?= =?us-ascii?Q?cNs4skuPboyir5RR7qtMevAo6kyxyW02agRw11c2jOc58whHuOd0dEe9BNL/?= =?us-ascii?Q?Tlodt0ar19VxViAn+lhbvW0hqZGRH4INxUbyrWlKSuIEriOfxwimw/uhApA6?= =?us-ascii?Q?93WoIVIiApAwzaIqCStJWQl3Jzv7PCrQL6JkmLt/JRHHU1oL2x6S7XA3hcxt?= =?us-ascii?Q?xdHPw/Ujq7mZE+57XT8JM0Yqwh28dcE8FMIdnbQnFq4PSsSz9ejiEIL3B4cN?= =?us-ascii?Q?PCFqVtwTmEf9IdDCHzxg4L8+vzOfQI7iX2IYqrFQycdCItNMmCc9zGK4zf+C?= =?us-ascii?Q?A22pPu3z1yRtqR9yspcanZcn8pP4aZ+3BdZ3DScLP7ezE7HjpNn1dGIVI0I4?= =?us-ascii?Q?PsktvmrqGDTBoIwWgPgoip646uARiCrejfO57CN3s0vK6DSzg4zdCevy5HZe?= =?us-ascii?Q?rXGa4pnYghHhvJgxuVw2m3agcDP1vc2GeIlIsstajfIw9r4vllcikCfjNzpE?= =?us-ascii?Q?cqgOQ66RqHx7uF/ot3t63KFDkjSkobZBP9RwThG+y+xBbU431kAfw2sDqaDF?= =?us-ascii?Q?8DQMb6Og8KogD3U1V/grUxf8+jh3dFOjl/jlhhMZP5HCXfJwKTVOOIlCWG57?= =?us-ascii?Q?dvpXn2Mwnw=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9acb0992-dd90-404e-b415-08df12bd5ec6 X-MS-Exchange-CrossTenant-AuthSource: LV3PR12MB9356.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Sep 2026 00:07:46.1322 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vT0dW6t+UHrDmGjMkbbmfh/d/TqjOXjXgT0elsiNnCztOtqYs8hoIOeHUYo4uJOnhDA3/cS3cBQktn92wEtIRQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5823 Content-Type: text/plain; charset="utf-8" Reuse directory tag files across make tags invocations. Track source files and per-directory input lists with make dependencies so only changed entries are indexed again. Reuse the source inventory and dependency rules when the selected file list is unchanged. Ignore references to missing sources in stale .cmd files before building cache lists and dependencies. Record cache locations, so obsolete files can be pruned after a successful run, including after an earlier interrupted update. Document manual recovery from corrupt caches and require one invocation at a time per build directory, without concurrent cache cleanup. Mirror source directories for out-of-tree builds and keep external inputs under .tags.external in the build tree. Caching doesn't measurably affect clean 'make tags' execution time, and substantially improves on subsequent runs: real user sys Clean tags generation 1m39.203s 9m1.450s 2m7.552s Single file updated 0m7.738s 0m7.042s 0m4.031s Unchanged sources 0m0.722s 0m0.482s 0m0.304s The cache is kept compressed, and occupies approximately 150 MB, less than 10% of the final tags file size. Assisted-by: OpenAI Codex Signed-off-by: Yury Norov --- .gitignore | 1 + Documentation/kbuild/kbuild.rst | 43 ++++++++- Makefile | 1 + scripts/Makefile.tags | 14 ++- scripts/tags.sh | 153 +++++++++++++++++++++++++++++--- 5 files changed, 193 insertions(+), 19 deletions(-) diff --git a/.gitignore b/.gitignore index 9875120ea7bd..df1e931d0140 100644 --- a/.gitignore +++ b/.gitignore @@ -137,6 +137,7 @@ patches series =20 # ctags files +.tags.* tags !tags/ TAGS diff --git a/Documentation/kbuild/kbuild.rst b/Documentation/kbuild/kbuild.= rst index 61587adeedba..0f6c4ebb939f 100644 --- a/Documentation/kbuild/kbuild.rst +++ b/Documentation/kbuild/kbuild.rst @@ -323,8 +323,47 @@ The jobs share make's jobserver with other build targe= ts. For example:: =20 $ make -j8 ALLSOURCE_ARCHS=3Dall tags =20 -Per-directory tag files are temporary and are removed after merging. -Every invocation regenerates the complete tags file. +Each directory has a cached input list and tag file: ``mm/.tags.files`` and +``mm/.tags.zst``, for example. Out-of-tree builds mirror these paths in the +build directory, leaving the source tree untouched. Inputs outside both tr= ees +are cached under ``.tags.external/`` in the build directory. + +Each cached tag file is sorted, carries an internal header/data prefix, an= d is +compressed with ``zstd -1``. This requires zstd 1.5.6 or later (overridable +with ``ZSTD``). +The final merge temporarily decompresses the shards, uses ``sort -m``, and +removes the prefixes, producing a normal uncompressed ``tags`` file without +re-sorting unchanged directories. Temporary files are removed after mergin= g; +allow disk space for the decompressed shards as well as the final output. +Obsolete shards and their input lists are removed after a successful run. + +The build root holds the shared ``.tags.inputs``, ``.tags.settings``, +``.tags.shards``, ``.tags.dependencies``, and ``.tags.locations`` manifest= s. +The complete source list is cached in ``.tags.inputs``. When it is unchang= ed, +the per-directory lists and dependency rules are reused without regenerati= ng +or comparing them. Make still checks source timestamps on every invocation. +Missing per-directory input lists are regenerated automatically. + +Each invocation checks the source lists, but only directories with changed +sources, file lists, or the ctags command or version are indexed again. The +final ``tags`` file is merged only when needed. Added and deleted sources +and changes to architecture selection or ignored directories are detected +automatically. ``make mrproper`` removes the cache and shared manifests. + +Changes to ctags configuration files or an executable replacement with the +same command and version are not detected. After such changes, remove +``.tags.settings`` in the build directory to force regeneration on the next +``make tags`` invocation. + +Missing tag files are rebuilt automatically. Existing cache files are not +checked for corruption on unchanged runs. If a merge reports a corrupt +cache file, remove that file and rerun ``make tags``, or remove +``.tags.settings`` to rebuild all cached tags. + +Run only one ``make tags`` invocation at a time in a given build directory. +Concurrent invocations, including those selecting different architectures, +are not supported. Do not run cache cleanup or ``make mrproper`` alongside +tag generation. =20 IGNORE_DIRS ----------- diff --git a/Makefile b/Makefile index 6f3945fa3fc6..62531970d004 100644 --- a/Makefile +++ b/Makefile @@ -1810,6 +1810,7 @@ $(mrproper-dirs): $(Q)$(MAKE) $(clean)=3D$(patsubst _mrproper_%,%,$@) =20 mrproper: clean objtool_mrproper $(mrproper-dirs) + $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-clean $(call cmd,rmfiles) @find . $(RCS_FIND_IGNORE) \ \( -name '*.rmeta' \) \ diff --git a/scripts/Makefile.tags b/scripts/Makefile.tags index d5967a3ac496..2274669e06f0 100644 --- a/scripts/Makefile.tags +++ b/scripts/Makefile.tags @@ -1,9 +1,15 @@ # SPDX-License-Identifier: GPL-2.0-only # Directory tag shards share the parent make's jobserver. =20 -.PHONY: tags -tags: $(shell cat $(tags_tmp)/shards) - $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-merge $(tags_tmp) +.DEFAULT_GOAL :=3D tags +tag_shards :=3D $(shell cat .tags.shards) =20 -%.tags: %.files +include .tags.dependencies + +tags: $(tag_shards) .tags.shards .tags.settings \ + $(srctree)/scripts/tags.sh $(srctree)/scripts/Makefile.tags + $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-merge + +%.zst: %.files .tags.settings $(srctree)/scripts/tags.sh \ + $(srctree)/scripts/Makefile.tags $(Q)$(BASH) $(srctree)/scripts/tags.sh tags-worker $@ diff --git a/scripts/tags.sh b/scripts/tags.sh index b001e0f78d2f..b541228d0045 100755 --- a/scripts/tags.sh +++ b/scripts/tags.sh @@ -119,6 +119,7 @@ all_sources() =20 all_compiled_sources() { + local file { echo include/generated/autoconf.h # A .cmd batch with no source references is not an error. @@ -128,7 +129,12 @@ all_compiled_sources() grep -Poh "$pattern" "$@" || [ "$?" -eq 1 ] ' sh '(?<=3D^ )\S+\.([chS]|rs)(?=3D\s)|(?<=3D=3D )\S+\.(?1)(?=3D$)' {} = \+ | awk '!a[$0]++' - } | xargs realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --relative= -to=3D.) | + } | while IFS=3D read -r file; do + # Old .cmd files may reference sources removed since the last build. + if [ -f "$file" ]; then + printf '%s\n' "$file" + fi + done | xargs -r realpath -esq $([ -z "$KBUILD_ABS_SRCTREE" ] && echo --re= lative-to=3D.) | sort -u } =20 @@ -356,24 +362,81 @@ setup_tags_tmp() } =20 # Let recursive make schedule directory shards using the caller's jobserve= r. +# The scan is unconditional so additions and deletions are also noticed. parallel_tags() ( - local tmp + local tmp file + # Older zstd versions mishandle hidden paths with --output-dir-mirror. + if ! ${ZSTD:-zstd} --version | awk ' + match($0, /v[0-9]+\.[0-9]+\.[0-9]+/) { + split(substr($0, RSTART + 1, RLENGTH - 1), v, ".") + ok =3D (v[1] * 10000 + v[2] * 100 + v[3] >=3D 10506) + } + END { exit !ok }'; then + echo "make tags requires zstd 1.5.6 or later" >&2 + exit 1 + fi setup_tags_tmp # Workers inherit these capabilities instead of probing for each director= y. setup_ctags ${CTAGS:-ctags} =20 { + # Invalidate caches written before shards were compressed. + printf '%s\n' 'shard-format=3Ddistributed-zstd-v1' + printf '%s\n' "${CTAGS:-ctags}" + ${CTAGS:-ctags} --version + } > "$tmp/settings" + + { + # Include the generator so changes to the cache format rebuild lists. + cksum "$0" all_target_sources | sed 's/^/s /' all_kconfigs | sed 's/^/k /' - } | LC_ALL=3DC sort -u | awk -v tmp=3D"$tmp" ' - { - dir =3D substr($0, 3) + } | LC_ALL=3DC sort -u > "$tmp/inputs" + + # A matching inventory is not sufficient if a cached input list was lost. + if [ -f .tags.shards ]; then + while IFS=3D read -r file; do + if [ ! -f "${file%.zst}.files" ]; then + rm -f .tags.inputs + break + fi + done < .tags.shards + fi + + if ! cmp -s "$tmp/inputs" .tags.inputs || + [ ! -f .tags.shards ] || [ ! -f .tags.dependencies ]; then + # Publish inputs last: an interrupted update must regenerate the lists. + rm -f .tags.inputs + # Resolve source paths once so out-of-tree caches stay in the build tree. + sed -n 's/^[sk] //p' "$tmp/inputs" | tr '\n' '\0' | + xargs -0 -r realpath -e -- > "$tmp/paths" + awk -v tmp=3D"$tmp" -v source_root=3D"$(realpath "${srctree:-.}")/" \ + -v build_root=3D"$(pwd -P)/" ' + BEGIN { + printf "" > (tmp "/dependencies") + printf "" > (tmp "/lists") + } + /^[sk] / { + file =3D substr($0, 3) + getline path < (tmp "/paths") + # Prefer the more specific root when one tree contains the other. + if (index(path, source_root) =3D=3D 1 && + (length(source_root) >=3D length(build_root) || + index(path, build_root) !=3D 1)) + path =3D substr(path, length(source_root) + 1) + else if (index(path, build_root) =3D=3D 1) + path =3D substr(path, length(build_root) + 1) + else + path =3D ".tags.external" path + dir =3D path if (!sub(/\/[^\/]*$/, "", dir)) dir =3D "." + base =3D (dir =3D=3D "." ? "" : dir "/") ".tags" if (!(dir in ids)) { ids[dir] =3D ++n - print tmp "/" n ".tags" + print base ".zst" + print n, base ".files" > (tmp "/lists") } out =3D tmp "/" ids[dir] ".files" if (out !=3D previous) { @@ -382,8 +445,49 @@ parallel_tags() previous =3D out } print >> out - }' > "$tmp/shards" - ${MAKE:-make} -f "${tree}scripts/Makefile.tags" tags_tmp=3D"$tmp" + print base ".zst: " file > (tmp "/dependencies") + }' "$tmp/inputs" | LC_ALL=3DC sort > "$tmp/shards" + fi + + # Record all locations before publishing lists so interrupted builds can + # still prune files left by earlier source selections. + if [ -f "$tmp/shards" ] || [ ! -f .tags.locations ]; then + { + for file in .tags.locations .tags.shards "$tmp/shards"; do + if [ -f "$file" ]; then + cat "$file" + fi + done + } | LC_ALL=3DC sort -u > "$tmp/locations" + mv "$tmp/locations" .tags.locations + fi + + if [ -f "$tmp/lists" ]; then + local id dir + while read -r id file; do + dir=3D${file%/*} + if [ "$dir" !=3D "$file" ] && [ ! -d "$dir" ]; then + mkdir -p "$dir" + fi + if ! cmp -s "$tmp/$id.files" "$file"; then + mv "$tmp/$id.files" "$file" + fi + done < "$tmp/lists" + fi + for file in settings shards dependencies; do + if [ -f "$tmp/$file" ] && ! cmp -s "$tmp/$file" ".tags.$file"; then + mv "$tmp/$file" ".tags.$file" + fi + done + if [ ! -f .tags.inputs ]; then + mv "$tmp/inputs" .tags.inputs + fi + ${MAKE:-make} -f "${tree}scripts/Makefile.tags" + if ! cmp -s .tags.locations .tags.shards; then + LC_ALL=3DC comm -23 .tags.locations .tags.shards | remove_tag_shards + cp .tags.shards "$tmp/locations" + mv "$tmp/locations" .tags.locations + fi ) =20 merge_tags() @@ -391,15 +495,26 @@ merge_tags() local tmp setup_tags_tmp =20 + # Use only current shards, excluding directories removed since last run. + # Expand into the temporary directory; cleanup also covers decode failure= s. + xargs -r ${ZSTD:-zstd} -q -d --output-dir-mirror=3D"$tmp" -- < .tags.shar= ds # Read filenames from stdin to avoid command-line length limits. { - tr '\n' '\0' < "$1/shards" + sed "s|^|$tmp/|; s/\.zst$//" .tags.shards | tr '\n' '\0' # sort requires at least one input, even when there are no shards. printf '/dev/null\0' } | LC_ALL=3DC sort -m -u --files0-from=3D- | cut -f2- > "$tmp/merged" mv "$tmp/merged" tags ) =20 +# Remove both files belonging to each listed shard, preserving path bounda= ries. +remove_tag_shards() +{ + while IFS=3D read -r file; do + printf '%s\0' "$file" "${file%.zst}.files" + done | xargs -0 -r rm -f -- +} + remove_struct_forward_declarations() { LC_ALL=3DC sed -e '/^\([a-zA-Z_][a-zA-Z0-9_]*\)\t.*\t\/\^struct \1;.*\$\/= ;"\tx$/d' "$@" @@ -465,7 +580,7 @@ case "$1" in ;; =20 "tags-worker") - tags_input=3D${2%.tags} + tags_input=3D${2%.zst} # Never expose an incomplete shard after an error or interruption. setup_tags_tmp tags_flags=3D(-f "$tmp/tags" --sort=3Dno --tag-relative=3Dno) @@ -476,16 +591,28 @@ case "$1" in { sub(/^!_TAG_FILE_SORTED\t0\t/, "!_TAG_FILE_SORTED\t1\t") print (/^!_TAG_/ ? "0\t" : "1\t") $0 - }' | LC_ALL=3DC sort --parallel=3D1 -u > "$tmp/sorted" - mv "$tmp/sorted" "$2" + }' | LC_ALL=3DC sort --parallel=3D1 -u | + ${ZSTD:-zstd} -q -1 --single-thread -c > "$tmp/sorted.zst" + mv "$tmp/sorted.zst" "$2" exit 0 ;; =20 "tags-merge") - merge_tags "$2" + merge_tags exit $? ;; =20 + "tags-clean") + set -eo pipefail + for file in .tags.locations .tags.shards; do + if [ -f "$file" ]; then + remove_tag_shards < "$file" + fi + done + rm -f .tags.{inputs,settings,shards,dependencies,locations} + exit 0 + ;; + "TAGS") rm -f TAGS xtags etags --=20 2.53.0