From nobody Fri Dec 19 04:15:06 2025 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E223A241693; Thu, 24 Apr 2025 10:38:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491132; cv=none; b=Fyy52nYASEXkf8etqXMLByQ6Tg3xBag+Dzgc1WODSYC12R4PcFYRxLNvHLZmjjIwmzOpW0raoW1GRZ9cLJYNKp/ELom16944IIfIvC2N37cGKgMF6DSPTHXJDAFHVkitT6htOaKsfdlcmlq7QglBlXYhnIxJlrEif2hPUW8HUbA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1745491132; c=relaxed/simple; bh=TfTzeexRkea+W22p9PrysDEUq3BRt9kb/rMQRSTwMW8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=I+ISAQ3zbiNKoilshiPTxhUG4AsEMzjZr/vJLmP68isctAM5TrZdtbdUpXTzD56y8vwK9DuGwr2eUL0xmDR+njCy/HekIRaXBERxQsebz1JEr07CQwnMsSuMUjgvvxqcv+wqBIAHqcd7wjM9jqqV9h5P9GOF2GtdzQdg9Z0H06k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=mYDxu6af; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="mYDxu6af" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-476977848c4so10721421cf.1; Thu, 24 Apr 2025 03:38:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1745491128; x=1746095928; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=f626Zq4sA1x5u4WIxeymx2qYBo02OEBs1+dVlap/P9o=; b=mYDxu6afYqpvn5jqoVWXHE7UKZux3meEUCy8GBli/HdWCbQ2i0LyPgO2T9BDhZZxQ4 XRZqIHMWQrtJfYymNrYfiyojnzY6NbklD95yzkzyF/Tw+RHrVOPOk3hab7raRmFLyaQ7 KhK7Z4zN+3N8jS7JrFJRMPBsZynhT3YWy1JCeeFUu5kb9fvecNr5ba/bSRL9MAqLrS8W pNNft+uwvY6f8nqmTjB9e7GSiGU+If+Wh0cvm48PDtHAt6gkruYQtFeODFU76u1Zd+O3 RyTMVHZz0ORW8fRIG4l5KX0CAWOhbCPgBPhpMV8VCu5mDM5DOrMFUJIUYVF6vn8iiHeu XEqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1745491128; x=1746095928; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f626Zq4sA1x5u4WIxeymx2qYBo02OEBs1+dVlap/P9o=; b=Yo6F59TO8oFs7qPo5HdvfC1DIbaDGYJevTARJKLhLICk7ZK0u9NvUStlKBSpoqq3sM o18q3KQWzXJZBuyJ/qF4fs6+PHF0DKj2E3BFj/S1ZAZ3HCWSd45/jdtis3wIdwZSpwYR esak6Gr/XiBGWMxVjesIi/OwwJXUZ3R/4/UUkDpicDHPN1CRdgIGevGE/6Azioxq8D12 maHlZhWRCzlp+fkzfxWGxxIgu5qx5Uje3zXXP0pqBf20QA/lKkQ+PQDBmkB/SWX+m7zr MHSZ7TziAwT9kKs/L0hj5UCizoIbIvgXLPt+WT9SS7+if0xaHLyi7MF4OZ1bJTYpf8mH CGYA== X-Forwarded-Encrypted: i=1; AJvYcCWxUaVHDyt/9koFiv6r0/bFr59IZewbyFNC2G5mC7XrEVL+NjIPwiXWTJWjNymq+qu1HAZUDjJkle1N8Y4=@vger.kernel.org X-Gm-Message-State: AOJu0Yy36ayU5RHGAJsklGhm1J/6aIEDTLE7k9AovfQ0wl5uN3GEGE22 g3BfWUelDMqoFGADX0TVxnJqXcIFTBWG3jVsONZrROzn2tjyMMTr X-Gm-Gg: ASbGncurf0WhcYnelzzwL+1f3DE+sHKwrKQSfAVBuhQHOyovJdDxDLXgHA76W+0pkhJ MZC8w/8HAgVWJGAyCI6mZR4gtFnvMpqKu1AFZAVAJCgLGDE1TJ2JGUeezd8GXaG/peQn708m57i FwgYJijM0xcVQd+sWL0JPSIydJYpG0GvayRttuBcGUddzIaW8/AvgUwgFYyW/aTyQW4YPQz0N0B lSqUTVMlIX1gBqfloM9ozMsokBRvTE2m1JfSkRQpbaR0CVt9UgYfoMNgd96/4Rq5WGwJXvU2OIM bjnIBySUO3HyZYAtzoca5AGLiTkYOh6eoOERj9YLI/+lRz9FpSOWZWS4qgKJ/KMaxfX/RDPmCN0 dIGM= X-Google-Smtp-Source: AGHT+IHtkwn3++A75SlOwbdMJloQJtQTjKGgkfWI+4V5/ua/xt13pEbwzwYBABeDdAIQ3j26IE4FMw== X-Received: by 2002:ad4:5ec6:0:b0:6ed:cea:f63c with SMTP id 6a1803df08f44-6f4bfbc77f3mr34544906d6.6.1745491128414; Thu, 24 Apr 2025 03:38:48 -0700 (PDT) Received: from [192.168.1.159] ([2600:4041:5be7:7c00:4047:3830:186a:1bb1]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6f4c0a73b63sm7839096d6.70.2025.04.24.03.38.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Apr 2025 03:38:47 -0700 (PDT) From: Tamir Duberstein Date: Thu, 24 Apr 2025 06:38:34 -0400 Subject: [PATCH v6 06/13] scripts: generate_rust_analyzer.py: add type hints Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20250424-rust-analyzer-host-v6-6-40e67fe5c38a@gmail.com> References: <20250424-rust-analyzer-host-v6-0-40e67fe5c38a@gmail.com> In-Reply-To: <20250424-rust-analyzer-host-v6-0-40e67fe5c38a@gmail.com> To: Miguel Ojeda , Alex Gaynor , Boqun Feng , Gary Guo , =?utf-8?q?Bj=C3=B6rn_Roy_Baron?= , Benno Lossin , Andreas Hindborg , Alice Ryhl , Trevor Gross , Danilo Krummrich , Boris-Chengbiao Zhou , Kees Cook , Fiona Behrens Cc: rust-for-linux@vger.kernel.org, linux-kernel@vger.kernel.org, Lukas Wirth , Tamir Duberstein , Daniel Almeida X-Mailer: b4 0.15-dev Python type hints allow static analysis tools like mypy to detect type errors during development, improving the developer experience. Python type hints have been present in the kernel since 2019 at the latest; see commit 6ebf5866f2e8 ("kunit: tool: add Python wrappers for running KUnit tests"). Add a subclass of `argparse.Namespace` to get type checking on the CLI arguments. Move parsing of `cfg` out of `generate_crates` to reduce the number of variables in scope with `cfg` in their name. Use a defaultdict to avoid `.get("key", [])`. Run `mypy --strict scripts/generate_rust_analyzer.py --python-version 3.8` to verify. Note that `mypy` no longer supports python < 3.8. Tested-by: Daniel Almeida Reviewed-by: Daniel Almeida Reviewed-by: Trevor Gross Signed-off-by: Tamir Duberstein --- scripts/generate_rust_analyzer.py | 163 +++++++++++++++++++++++++---------= ---- 1 file changed, 106 insertions(+), 57 deletions(-) diff --git a/scripts/generate_rust_analyzer.py b/scripts/generate_rust_anal= yzer.py index a7f94f3be93e..6fa4cec0e506 100755 --- a/scripts/generate_rust_analyzer.py +++ b/scripts/generate_rust_analyzer.py @@ -10,16 +10,48 @@ import os import pathlib import subprocess import sys +from collections import defaultdict +from typing import DefaultDict, Dict, Iterable, List, Literal, Optional, T= ypedDict =20 -def args_crates_cfgs(cfgs): - crates_cfgs =3D {} - for cfg in cfgs: - crate, vals =3D cfg.split("=3D", 1) - crates_cfgs[crate] =3D vals.replace("--cfg", "").split() =20 - return crates_cfgs +class Dependency(TypedDict): + crate: int + name: str =20 -def generate_crates(srctree, objtree, sysroot_src, external_src, cfgs): + +class Source(TypedDict): + include_dirs: List[str] + exclude_dirs: List[str] + + +class Crate(TypedDict): + display_name: str + root_module: str + is_workspace_member: bool + deps: List[Dependency] + cfg: List[str] + edition: Literal["2021"] + env: Dict[str, str] + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was ad= ded in 3.11. +class ProcMacroCrate(Crate): + is_proc_macro: Literal[True] + proc_macro_dylib_path: Optional[str] # `pathlib.Path` is not JSON ser= ializable. + + +# `NotRequired` fields on `Crate` would be better but `NotRequired` was ad= ded in 3.11. +class CrateWithGenerated(Crate): + source: Optional[Source] + + +def generate_crates( + srctree: pathlib.Path, + objtree: pathlib.Path, + sysroot_src: pathlib.Path, + external_src: pathlib.Path, + crates_cfgs: DefaultDict[str, List[str]], +) -> List[Crate]: # Generate the configuration list. cfg =3D [] with open(objtree / "include" / "generated" / "rustc_cfg") as fd: @@ -31,17 +63,16 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): # Now fill the crates list -- dependencies need to come first. # # Avoid O(n^2) iterations by keeping a map of indexes. - crates =3D [] - crates_indexes =3D {} - crates_cfgs =3D args_crates_cfgs(cfgs) + crates: List[Crate] =3D [] + crates_indexes: Dict[str, int] =3D {} =20 def build_crate( - display_name, - root_module, - deps, - cfg=3D[], - is_workspace_member=3DTrue, - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] =3D [], + is_workspace_member: bool =3D True, + ) -> Crate: return { "display_name": display_name, "root_module": str(root_module), @@ -51,36 +82,30 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): "edition": "2021", "env": { "RUST_MODFILE": "This is only for rust-analyzer" - } + }, } =20 - def register_crate(crate): + def register_crate(crate: Crate) -> None: crates_indexes[crate["display_name"]] =3D len(crates) crates.append(crate) =20 def append_crate( - display_name, - root_module, - deps, - cfg=3D[], - is_workspace_member=3DTrue, - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] =3D [], + is_workspace_member: bool =3D True, + ) -> None: register_crate( - build_crate( - display_name, - root_module, - deps, - cfg, - is_workspace_member, - ) + build_crate(display_name, root_module, deps, cfg, is_workspace= _member) ) =20 def append_proc_macro_crate( - display_name, - root_module, - deps, - cfg=3D[], - ): + display_name: str, + root_module: pathlib.Path, + deps: List[str], + cfg: List[str] =3D [], + ) -> None: crate =3D build_crate(display_name, root_module, deps, cfg) proc_macro_dylib_name =3D ( subprocess.check_output( @@ -99,7 +124,7 @@ def generate_crates(srctree, objtree, sysroot_src, exter= nal_src, cfgs): .decode("utf-8") .strip() ) - proc_macro_crate =3D { + proc_macro_crate: ProcMacroCrate =3D { **crate, "is_proc_macro": True, "proc_macro_dylib_path": f"{objtree}/rust/{proc_macro_dylib_na= me}", @@ -107,10 +132,10 @@ def generate_crates(srctree, objtree, sysroot_src, ex= ternal_src, cfgs): register_crate(proc_macro_crate) =20 def append_sysroot_crate( - display_name, - deps, - cfg=3D[], - ): + display_name: str, + deps: List[str], + cfg: List[str] =3D [], + ) -> None: append_crate( display_name, sysroot_src / display_name / "src" / "lib.rs", @@ -122,7 +147,7 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): # NB: sysroot crates reexport items from one another so setting up our= transitive dependencies # here is important for ensuring that rust-analyzer can resolve symbol= s. The sources of truth # for this dependency graph are `(sysroot_src / crate / "Cargo.toml" f= or crate in crates)`. - append_sysroot_crate("core", [], cfg=3Dcrates_cfgs.get("core", [])) + append_sysroot_crate("core", [], cfg=3Dcrates_cfgs["core"]) append_sysroot_crate("alloc", ["core"]) append_sysroot_crate("std", ["alloc", "core"]) append_sysroot_crate("proc_macro", ["core", "std"]) @@ -166,9 +191,9 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): ) =20 def append_crate_with_generated( - display_name, - deps, - ): + display_name: str, + deps: List[str], + ) -> None: crate =3D build_crate( display_name, srctree / "rust" / display_name / "lib.rs", @@ -176,20 +201,23 @@ def generate_crates(srctree, objtree, sysroot_src, ex= ternal_src, cfgs): cfg=3Dcfg, ) crate["env"]["OBJTREE"] =3D str(objtree.resolve(True)) - crate["source"] =3D { - "include_dirs": [ - str(srctree / "rust" / display_name), - str(objtree / "rust") - ], - "exclude_dirs": [], + crate_with_generated: CrateWithGenerated =3D { + **crate, + "source": { + "include_dirs": [ + str(srctree / "rust" / display_name), + str(objtree / "rust") + ], + "exclude_dirs": [], + } } - register_crate(crate) + register_crate(crate_with_generated) =20 append_crate_with_generated("bindings", ["core", "ffi"]) append_crate_with_generated("uapi", ["core", "ffi"]) append_crate_with_generated("kernel", ["core", "macros", "build_error"= , "pin_init", "ffi", "bindings", "uapi"]) =20 - def is_root_crate(build_file, target): + def is_root_crate(build_file: pathlib.Path, target: str) -> bool: try: return f"{target}.o" in open(build_file).read() except FileNotFoundError: @@ -198,7 +226,7 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): # Then, the rest outside of `rust/`. # # We explicitly mention the top-level folders we want to cover. - extra_dirs =3D map(lambda dir: srctree / dir, ("samples", "drivers")) + extra_dirs: Iterable[pathlib.Path] =3D (srctree / dir for dir in ("rus= t", "drivers")) if external_src is not None: extra_dirs =3D [external_src] for folder in extra_dirs: @@ -222,7 +250,7 @@ def generate_crates(srctree, objtree, sysroot_src, exte= rnal_src, cfgs): return crates =20 =20 -def main(): +def main() -> None: parser =3D argparse.ArgumentParser() parser.add_argument("--verbose", "-v", action=3D"store_true") parser.add_argument("--cfgs", action=3D"append", default=3D[]) @@ -231,7 +259,17 @@ def main(): parser.add_argument("sysroot", type=3Dpathlib.Path) parser.add_argument("sysroot_src", type=3Dpathlib.Path) parser.add_argument("exttree", type=3Dpathlib.Path, nargs=3D"?") - args =3D parser.parse_args() + + class Args(argparse.Namespace): + verbose: bool + cfgs: List[str] + srctree: pathlib.Path + objtree: pathlib.Path + sysroot: pathlib.Path + sysroot_src: pathlib.Path + exttree: pathlib.Path + + args =3D parser.parse_args(namespace=3DArgs()) =20 logging.basicConfig( format=3D"[%(asctime)s] [%(levelname)s] %(message)s", @@ -241,8 +279,19 @@ def main(): # Making sure that the `sysroot` and `sysroot_src` belong to the same = toolchain. assert args.sysroot in args.sysroot_src.parents =20 + crates_cfg =3D { + crate: vals.lstrip("--cfg").split() + for crate, _, vals in (cfg.partition("=3D") for cfg in args.cfgs) + } + rust_project =3D { - "crates": generate_crates(args.srctree, args.objtree, args.sysroot= _src, args.exttree, args.cfgs), + "crates": generate_crates( + args.srctree, + args.objtree, + args.sysroot_src, + args.exttree, + defaultdict(list, crates_cfg), + ), "sysroot": str(args.sysroot), } =20 --=20 2.49.0