From nobody Thu Apr 18 14:04:01 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+51327+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51327+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1574829448; cv=none; d=zohomail.com; s=zohoarc; b=AHmv7or1A5ZwZpy7i2q0u1C5qEaqM/KFhiYozIxZaxQ9JrWngoshP700favU4OJWgB2NH0merIxhU1KKIA/zN5soBkbhKGjWQGcUbJpAeYWL+GzYhXMnpCPrjCub11XO1OEkyOCTuzyDiTE/CVG1lI8hdJiSqHmzUZrZMI1h8R0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1574829448; h=Content-Transfer-Encoding:Cc:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To; bh=4hgZFTf2jLdEPpxVaP2rnaBj/yC0cxyZ9EoKPCLX6Rc=; b=DcSzmSSefpp5oY/ySagT1P66acwPdeUYIXZltJnDQrzrNADJz10gbjKwSDnZhgAXz8/iF/5FGrIzGzQOCJc73hUo7nJtnmbh7YeKXcV3fR+b1ghNy8kpSK8jz1ky+k+94+nxZ+Yn96mTrCY2TtezDgqt2DDI5OCiNpdsLyZmpIE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51327+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1574829448165336.32786712725783; Tue, 26 Nov 2019 20:37:28 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id 4DQlYY1788612xERMsoOfAm0; Tue, 26 Nov 2019 20:37:27 -0800 X-Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mx.groups.io with SMTP id smtpd.web09.1831.1574829446456555022 for ; Tue, 26 Nov 2019 20:37:26 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 26 Nov 2019 20:37:25 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,248,1571727600"; d="scan'208";a="202946503" X-Received: from nldesimo-desk1.amr.corp.intel.com ([10.7.159.63]) by orsmga008.jf.intel.com with ESMTP; 26 Nov 2019 20:37:25 -0800 From: "Nate DeSimone" To: devel@edk2.groups.io Cc: Ashley E Desimone Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH] EdkRepo: Add list-repos command Date: Tue, 26 Nov 2019 20:37:07 -0800 Message-Id: <20191127043707.923-1-nathaniel.l.desimone@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,nathaniel.l.desimone@intel.com X-Gm-Message-State: hVspwBgSVduuVzCaioIo51Ubx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1574829447; bh=wIl8D0cbAv4fbrF5KiuowDuGmPec5UpHto0eYkTwUdw=; h=Cc:Date:From:Reply-To:Subject:To; b=LN8ki04mV1XJkBMG3GTq5WJ4i+pSxxnnrbrzPjmRZvOcJP8LxDeA8vZLs7xb8/7aTsS X4z4CMPi3t+B2JBW3jqTyIZ19ZKxhsWjM/hjw/Okq9uwlp28AiGMfLci9w5kNa/bu8twE talwSVXgxhcHYukVuQk7jgMpYL6QccPYTew= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" list-repos lists the git repos used by all projects and which branches those projects use. Cc: Ashley E Desimone Signed-off-by: Nate DeSimone --- edkrepo/commands/arguments/list_repos_args.py | 16 ++ edkrepo/commands/humble/__init__.py | 8 + edkrepo/commands/humble/list_repos_humble.py | 26 +++ edkrepo/commands/list_repos_command.py | 218 ++++++++++++++++++ 4 files changed, 268 insertions(+) create mode 100644 edkrepo/commands/arguments/list_repos_args.py create mode 100644 edkrepo/commands/humble/__init__.py create mode 100644 edkrepo/commands/humble/list_repos_humble.py create mode 100644 edkrepo/commands/list_repos_command.py diff --git a/edkrepo/commands/arguments/list_repos_args.py b/edkrepo/comman= ds/arguments/list_repos_args.py new file mode 100644 index 0000000..6ff9b9c --- /dev/null +++ b/edkrepo/commands/arguments/list_repos_args.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_args.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +''' Contains the help and description strings for arguments in the +list-repos command meta data. +''' + +COMMAND_DESCRIPTION =3D 'Lists the git repos used by available projects an= d the branches that are used.' +ARCHIVED_HELP =3D 'Include a listing of archived projects.' +REPOS_HELP =3D 'Only show the given subset of repos instead of all repos.' diff --git a/edkrepo/commands/humble/__init__.py b/edkrepo/commands/humble/= __init__.py new file mode 100644 index 0000000..dea6eb4 --- /dev/null +++ b/edkrepo/commands/humble/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +# +## @file +# __init__.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# diff --git a/edkrepo/commands/humble/list_repos_humble.py b/edkrepo/command= s/humble/list_repos_humble.py new file mode 100644 index 0000000..bbb05a7 --- /dev/null +++ b/edkrepo/commands/humble/list_repos_humble.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_humble.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +''' +Contains user visible strings printed by the list-repos command. +''' + +from colorama import Fore +from colorama import Style + +BRANCHES =3D 'Branches:' +BRANCH_FORMAT_STRING =3D ' {}{}{{}}{}'.format(Fore.BLUE, Style.BRIGHT, St= yle.RESET_ALL) +COMBO_FORMAT_STRING =3D ' {{}} {}{}({{}}){}'.format(Fore.CYAN, Style.B= RIGHT, Style.RESET_ALL) +DEFAULT_COMBO_FORMAT_STRING =3D ' {{}} {}{}*({{}})*{}'.format(Fore.GREE= N, Style.BRIGHT, Style.RESET_ALL) +MANIFEST_DIRECTORY =3D 'Manifest directory:' +PROJECT_NAME_FORMAT_STRING =3D '{}{}{{}}{}:'.format(Fore.YELLOW, Style.BRI= GHT, Style.RESET_ALL) +REPOSITORIES =3D 'Repositories:' +REPO_NAME_AND_URL =3D '{}{}{{}}{} - [{}{}{{}}{}]'.format(Fore.MAGENTA, Sty= le.BRIGHT, Style.RESET_ALL, Fore.RED, Style.BRIGHT, Style.RESET_ALL) +REPO_NAME_NOT_FOUND =3D 'repo_name not found' +REPO_NOT_FOUND_IN_MANIFEST =3D 'Repo {} not found in any manifest file' diff --git a/edkrepo/commands/list_repos_command.py b/edkrepo/commands/list= _repos_command.py new file mode 100644 index 0000000..de2de03 --- /dev/null +++ b/edkrepo/commands/list_repos_command.py @@ -0,0 +1,218 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_command.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +import collections +import os + +#from git import Repo +from colorama import Fore, Style + +# Our modules +from edkrepo.commands.edkrepo_command import EdkrepoCommand +from edkrepo.commands.edkrepo_command import ColorArgument +import edkrepo.commands.arguments.list_repos_args as arguments +import edkrepo.commands.humble.list_repos_humble as humble +from edkrepo.common.common_repo_functions import pull_latest_manifest_repo +from edkrepo.common.edkrepo_exception import EdkrepoInvalidParametersExcep= tion, EdkrepoManifestInvalidException +from edkrepo.common.ui_functions import init_color_console +from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml + +class ListReposCommand(EdkrepoCommand): + def __init__(self): + super().__init__() + self.repo_names =3D None + + def get_metadata(self): + metadata =3D {} + metadata['name'] =3D 'list-repos' + metadata['help-text'] =3D arguments.COMMAND_DESCRIPTION + args =3D [] + metadata['arguments'] =3D args + args.append({'name': 'repos', + 'positional': False, + 'required': False, + 'action': 'store', + 'nargs': '+', + 'help-text': arguments.REPOS_HELP}) + args.append({'name': 'archived', + 'short-name': 'a', + 'positional': False, + 'required': False, + 'help-text': arguments.ARCHIVED_HELP}) + args.append(ColorArgument) + return metadata + + def run_command(self, args, config): + print() + init_color_console(args.color) + + # Get path to global manifest file + global_manifest_directory =3D config['cfg_file'].manifest_repo_abs= _local_path + if args.verbose: + print(humble.MANIFEST_DIRECTORY) + print(global_manifest_directory) + print() + index_path =3D os.path.join(global_manifest_directory, 'CiIndex.xm= l') + + pull_latest_manifest_repo(args, config) + print() + + ci_index_xml =3D CiIndexXml(index_path) + manifests =3D {} + repo_urls =3D set() + project_list =3D list(ci_index_xml.project_list) + if args.archived: + project_list.extend(ci_index_xml.archived_project_list) + for project in project_list: + xml_file =3D ci_index_xml.get_project_xml(project) + manifest =3D ManifestXml(os.path.normpath(os.path.join(global_= manifest_directory, xml_file))) + manifests[project] =3D manifest + for combo in [c.name for c in manifest.combinations]: + sources =3D manifest.get_repo_sources(combo) + for source in sources: + repo_urls.add(self.get_repo_url(source.remote_url)) + manifests =3D collections.OrderedDict(sorted(manifests.items())) + project_justify =3D len(max(manifests.keys(), key=3Dlen)) + self.generate_repo_names(repo_urls, manifests) + print(humble.REPOSITORIES) + for repo_name in self.repo_names: + if args.repos and repo_name not in args.repos: + continue + repo =3D self.repo_names[repo_name][0] + print(humble.REPO_NAME_AND_URL.format(repo_name, repo)) + print(humble.BRANCHES) + branches =3D set() + for project_name in manifests: + for combo in [c.name for c in manifests[project_name].comb= inations]: + sources =3D manifests[project_name].get_repo_sources(c= ombo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D rep= o: + branches.add(source.branch) + branches =3D sorted(branches, key=3Dstr.casefold) + if 'master' in branches: + branches.remove('master') + branches.insert(0, 'master') + #length =3D len(max(branches, key=3Dlen)) .ljust(length) + for branch in branches: + print(humble.BRANCH_FORMAT_STRING.format(branch)) + for project_name in manifests: + combos =3D [] + for combo in [c.name for c in manifests[project_name].= combinations]: + sources =3D manifests[project_name].get_repo_sourc= es(combo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D= repo and source.branch =3D=3D branch: + combos.append(combo) + break + if len(combos) > 0: + combos =3D sorted(combos, key=3Dstr.casefold) + default_combo =3D manifests[project_name].general_= config.default_combo + if default_combo in combos: + combos.remove(default_combo) + combos.insert(0, default_combo) + first_combo =3D True + for combo in combos: + if first_combo: + project_name_print =3D humble.PROJECT_NAME= _FORMAT_STRING.format(project_name.ljust(project_justify)) + first_combo =3D False + else: + project_name_print =3D '{} '.format((' ' *= len(project_name)).ljust(project_justify)) + if default_combo =3D=3D combo: + print(humble.DEFAULT_COMBO_FORMAT_STRING.f= ormat(project_name_print, combo)) + else: + print(humble.COMBO_FORMAT_STRING.format(pr= oject_name_print, combo)) + + def get_repo_url(self, repo_url): + if repo_url[-4:].lower() =3D=3D '.git': + return repo_url[:-4] + return repo_url + + def get_repo_name(self, repo_url, manifests): + for name in self.repo_names: + if self.repo_names[name][0] =3D=3D repo_url: + return name + raise EdkrepoInvalidParametersException(humble.REPO_NAME_NOT_FOUND) + + def generate_repo_names(self, repo_urls, manifests): + self.repo_names =3D collections.OrderedDict() + for repo_url in repo_urls: + self.__repo_name_worker(repo_url, manifests) + self.repo_names =3D collections.OrderedDict(sorted(self.repo_names= .items())) + names_to_move =3D [] + for repo_name in self.repo_names: + if repo_name.lower().find('edk2') =3D=3D 0: + names_to_move.append(repo_name) + names_to_move =3D sorted(names_to_move, reverse=3DTrue) + for name_to_move in names_to_move: + self.repo_names.move_to_end(name_to_move, False) + names_to_move =3D [] + for repo_name in self.repo_names: + if repo_name.lower().find('intel') =3D=3D 0: + names_to_move.append(repo_name) + names_to_move =3D sorted(names_to_move, reverse=3DTrue) + for name_to_move in names_to_move: + self.repo_names.move_to_end(name_to_move, False) + + def __repo_name_worker(self, repo_url, manifests): + #This is a heuristic that guesses the "name" of a repository by lo= oking + #at the name given to it by the most manifest files. + names =3D collections.defaultdict(int) + for project_name in manifests: + for combo in [c.name for c in manifests[project_name].combinat= ions]: + sources =3D manifests[project_name].get_repo_sources(combo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D repo_ur= l: + names[source.root] +=3D 1 + found_unique_name =3D False + original_best_name =3D None + original_best_name_frequency =3D 0 + while not found_unique_name: + best_name =3D None + best_name_frequency =3D 0 + if len(names) <=3D 0: + if original_best_name_frequency =3D=3D 1: + #If only 1 project uses this name, then append the pro= ject + #name to the directory name to create the repo name + for project_name in manifests: + for combo in [c.name for c in manifests[project_na= me].combinations]: + sources =3D manifests[project_name].get_repo_s= ources(combo) + for source in sources: + if self.get_repo_url(source.remote_url) = =3D=3D repo_url and source.root =3D=3D original_best_name: + best_name =3D "{}-{}".format(original_= best_name, project_name) + best_name_frequency =3D original_best_= name_frequency + else: + best_name =3D repo_url + best_name_frequency =3D 0 + break + for name in names: + if names[name] > best_name_frequency: + best_name =3D name + best_name_frequency =3D names[name] + if best_name is None: + raise EdkrepoManifestInvalidException(humble.REPO_NOT_FOUN= D_IN_MANIFEST.format(repo_url)) + if original_best_name is None: + original_best_name =3D best_name + original_best_name_frequency =3D best_name_frequency + if best_name in self.repo_names: + if self.repo_names[best_name][0] =3D=3D repo_url: + found_unique_name =3D True + else: + #If there is a name collision, then which repo has the= most + #Usage of the name owns the name + if best_name_frequency > self.repo_names[best_name][1]: + old_repo_url =3D self.repo_names[name][0] + del self.repo_names[best_name] + found_unique_name =3D True + self.repo_names[best_name] =3D (repo_url, best_nam= e_frequency) + self.__repo_name_worker(old_repo_url, manifests) + else: + #Use the name given by the second most manifest fi= les + del names[best_name] + else: + found_unique_name =3D True + self.repo_names[best_name] =3D (repo_url, best_name_frequency) --=20 2.24.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#51327): https://edk2.groups.io/g/devel/message/51327 Mute This Topic: https://groups.io/mt/62229423/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-