From nobody Mon May 6 19:55:21 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+51673+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51673+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1575414379; cv=none; d=zohomail.com; s=zohoarc; b=MpKYNr3XX6JdN7FhbYsgYlRs/uQLWWbxi3MXoPgQ6F7aoHTkcHEJGfIzlYMDdr5g+fVVOhj9NXbCy+G53VXjpT1RLtmAgj5vvwKyP+kiUbhr4wN5+bHKWBXQ/xe/fTr7Rdo1PoebKsAXkPZ2qWNTOg01WE7J2vOWFnp3raz2Uv0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1575414379; h=Content-Transfer-Encoding:Cc:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To; bh=WfWboeBy7PoN/90wgzJbAg8J8xGaUxgunAoC/TJ9kNs=; b=BBWHrqVH/7H/FFvKxoI/dArOemqPjNW2PYnOuZdfV76CCuD17jxun000N1zLGFntew3o4ObeIh27O7jMTd5y64o7wiAyZxMZ9UlU3vjNfy837/PfAwrllqPzZP2y2FYdlJV0yzpgE1N0CM6gbHnIxrcTguZmSftR9BFO1/Dcg+g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51673+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1575414378896917.9752497653176; Tue, 3 Dec 2019 15:06:18 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id aaaaYY1788612xaaaaaaaaaa; Tue, 03 Dec 2019 15:06:18 -0800 X-Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mx.groups.io with SMTP id smtpd.web12.4005.1575414377525699066 for ; Tue, 03 Dec 2019 15:06:17 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 03 Dec 2019 15:06:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,275,1571727600"; d="scan'208";a="361360484" X-Received: from nldesimo-desk1.amr.corp.intel.com ([10.7.159.63]) by orsmga004.jf.intel.com with ESMTP; 03 Dec 2019 15:06:16 -0800 From: "Nate DeSimone" To: devel@edk2.groups.io Cc: Ashley E Desimone Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH V2] EdkRepo: Add list-repos command Date: Tue, 3 Dec 2019 15:05:52 -0800 Message-Id: <20191203230552.1311-1-nathaniel.l.desimone@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,nathaniel.l.desimone@intel.com X-Gm-Message-State: aaaaaaaaaaaaaaaaaaaaaaaax1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1575414378; bh=D0YbiR11Aq5KnfqFD+E95LbIplHxv2HlsQgOpGC6LAs=; h=Cc:Date:From:Reply-To:Subject:To; b=gJ9xC8tpFlbpFG2a7pN76xzJNe4tP3NF94G027mxeANdlA682/VqyFdDzEIIg9ntJFh 0QABetT+llEbnmokcSUwCkFkiHBeWJK2ZXr6J523XTP8euIeBdITblAwENFOOioEqzU3f GOkoWr1wGlqWGBpRh9G84VTxLS+FPyD0ubc= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" list-repos lists the git repos used by all projects and which branches those projects use. Cc: Ashley E Desimone Signed-off-by: Nate DeSimone Reviewed-by: Ashley E DeSimone --- edkrepo/commands/arguments/list_repos_args.py | 16 ++ edkrepo/commands/humble/__init__.py | 8 + edkrepo/commands/humble/list_repos_humble.py | 26 ++ edkrepo/commands/list_repos_command.py | 243 ++++++++++++++++++ setup.py | 5 +- 5 files changed, 295 insertions(+), 3 deletions(-) create mode 100644 edkrepo/commands/arguments/list_repos_args.py create mode 100644 edkrepo/commands/humble/__init__.py create mode 100644 edkrepo/commands/humble/list_repos_humble.py create mode 100644 edkrepo/commands/list_repos_command.py diff --git a/edkrepo/commands/arguments/list_repos_args.py b/edkrepo/comman= ds/arguments/list_repos_args.py new file mode 100644 index 0000000..6ff9b9c --- /dev/null +++ b/edkrepo/commands/arguments/list_repos_args.py @@ -0,0 +1,16 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_args.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +''' Contains the help and description strings for arguments in the +list-repos command meta data. +''' + +COMMAND_DESCRIPTION =3D 'Lists the git repos used by available projects an= d the branches that are used.' +ARCHIVED_HELP =3D 'Include a listing of archived projects.' +REPOS_HELP =3D 'Only show the given subset of repos instead of all repos.' diff --git a/edkrepo/commands/humble/__init__.py b/edkrepo/commands/humble/= __init__.py new file mode 100644 index 0000000..dea6eb4 --- /dev/null +++ b/edkrepo/commands/humble/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +# +## @file +# __init__.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# diff --git a/edkrepo/commands/humble/list_repos_humble.py b/edkrepo/command= s/humble/list_repos_humble.py new file mode 100644 index 0000000..bbb05a7 --- /dev/null +++ b/edkrepo/commands/humble/list_repos_humble.py @@ -0,0 +1,26 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_humble.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +''' +Contains user visible strings printed by the list-repos command. +''' + +from colorama import Fore +from colorama import Style + +BRANCHES =3D 'Branches:' +BRANCH_FORMAT_STRING =3D ' {}{}{{}}{}'.format(Fore.BLUE, Style.BRIGHT, St= yle.RESET_ALL) +COMBO_FORMAT_STRING =3D ' {{}} {}{}({{}}){}'.format(Fore.CYAN, Style.B= RIGHT, Style.RESET_ALL) +DEFAULT_COMBO_FORMAT_STRING =3D ' {{}} {}{}*({{}})*{}'.format(Fore.GREE= N, Style.BRIGHT, Style.RESET_ALL) +MANIFEST_DIRECTORY =3D 'Manifest directory:' +PROJECT_NAME_FORMAT_STRING =3D '{}{}{{}}{}:'.format(Fore.YELLOW, Style.BRI= GHT, Style.RESET_ALL) +REPOSITORIES =3D 'Repositories:' +REPO_NAME_AND_URL =3D '{}{}{{}}{} - [{}{}{{}}{}]'.format(Fore.MAGENTA, Sty= le.BRIGHT, Style.RESET_ALL, Fore.RED, Style.BRIGHT, Style.RESET_ALL) +REPO_NAME_NOT_FOUND =3D 'repo_name not found' +REPO_NOT_FOUND_IN_MANIFEST =3D 'Repo {} not found in any manifest file' diff --git a/edkrepo/commands/list_repos_command.py b/edkrepo/commands/list= _repos_command.py new file mode 100644 index 0000000..9035993 --- /dev/null +++ b/edkrepo/commands/list_repos_command.py @@ -0,0 +1,243 @@ +#!/usr/bin/env python3 +# +## @file +# list_repos_command.py +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +import collections +import os + +#from git import Repo +from colorama import Fore, Style + +# Our modules +from edkrepo.commands.edkrepo_command import EdkrepoCommand +from edkrepo.commands.edkrepo_command import ColorArgument +import edkrepo.commands.arguments.list_repos_args as arguments +import edkrepo.commands.humble.list_repos_humble as humble +from edkrepo.common.common_repo_functions import pull_latest_manifest_repo +from edkrepo.common.edkrepo_exception import EdkrepoInvalidParametersExcep= tion, EdkrepoManifestInvalidException +from edkrepo.common.ui_functions import init_color_console +from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml + +class ListReposCommand(EdkrepoCommand): + def __init__(self): + super().__init__() + self.repo_names =3D None + + def get_metadata(self): + metadata =3D {} + metadata['name'] =3D 'list-repos' + metadata['help-text'] =3D arguments.COMMAND_DESCRIPTION + args =3D [] + metadata['arguments'] =3D args + args.append({'name': 'repos', + 'positional': False, + 'required': False, + 'action': 'store', + 'nargs': '+', + 'help-text': arguments.REPOS_HELP}) + args.append({'name': 'archived', + 'short-name': 'a', + 'positional': False, + 'required': False, + 'help-text': arguments.ARCHIVED_HELP}) + args.append(ColorArgument) + return metadata + + def run_command(self, args, config): + print() + init_color_console(args.color) + + # Get path to global manifest file + global_manifest_directory =3D config['cfg_file'].manifest_repo_abs= _local_path + if args.verbose: + print(humble.MANIFEST_DIRECTORY) + print(global_manifest_directory) + print() + index_path =3D os.path.join(global_manifest_directory, 'CiIndex.xm= l') + + pull_latest_manifest_repo(args, config) + print() + + #Create a dictionary containing all the manifests listed in the Ci= Index.xml file + ci_index_xml =3D CiIndexXml(index_path) + manifests =3D {} + repo_urls =3D set() + project_list =3D list(ci_index_xml.project_list) + if args.archived: + project_list.extend(ci_index_xml.archived_project_list) + for project in project_list: + xml_file =3D ci_index_xml.get_project_xml(project) + manifest =3D ManifestXml(os.path.normpath(os.path.join(global_= manifest_directory, xml_file))) + manifests[project] =3D manifest + for combo in [c.name for c in manifest.combinations]: + sources =3D manifest.get_repo_sources(combo) + for source in sources: + repo_urls.add(self.get_repo_url(source.remote_url)) + + #Sort the manifests so projects will be displayed alphabetically + manifests =3D collections.OrderedDict(sorted(manifests.items())) + project_justify =3D len(max(manifests.keys(), key=3Dlen)) + + #Determine the names of the repositories + self.generate_repo_names(repo_urls, manifests) + print(humble.REPOSITORIES) + + #For each each git repository... + for repo_name in self.repo_names: + if args.repos and repo_name not in args.repos: + continue + repo =3D self.repo_names[repo_name][0] + print(humble.REPO_NAME_AND_URL.format(repo_name, repo)) + print(humble.BRANCHES) + + #Determine the list of branches that used by any branch combin= ation in any manifest + branches =3D set() + for project_name in manifests: + for combo in [c.name for c in manifests[project_name].comb= inations]: + sources =3D manifests[project_name].get_repo_sources(c= ombo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D rep= o: + branches.add(source.branch) + + #Sort the branch names so they will be displayed alphabetically + #with the exception that if a branch named "master" exists, th= en it + #will be displayed first + branches =3D sorted(branches, key=3Dstr.casefold) + if 'master' in branches: + branches.remove('master') + branches.insert(0, 'master') + + #For each interesting branch in the current git repository... + for branch in branches: + print(humble.BRANCH_FORMAT_STRING.format(branch)) + + #Determine the branch combinations that use that branch + for project_name in manifests: + combos =3D [] + for combo in [c.name for c in manifests[project_name].= combinations]: + sources =3D manifests[project_name].get_repo_sourc= es(combo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D= repo and source.branch =3D=3D branch: + combos.append(combo) + break + if len(combos) > 0: + #Sort the branch combinations so they will be disp= layed alphabetically + #with the exception that the default branch combin= ation for the manifest + #file will be displayed first + combos =3D sorted(combos, key=3Dstr.casefold) + default_combo =3D manifests[project_name].general_= config.default_combo + if default_combo in combos: + combos.remove(default_combo) + combos.insert(0, default_combo) + first_combo =3D True + for combo in combos: + #Print the project name + if first_combo: + project_name_print =3D humble.PROJECT_NAME= _FORMAT_STRING.format(project_name.ljust(project_justify)) + first_combo =3D False + else: + project_name_print =3D '{} '.format((' ' *= len(project_name)).ljust(project_justify)) + #Print the branch combination name, if this is= the default branch combination, + #then print it in green color with *'s around = it + if default_combo =3D=3D combo: + print(humble.DEFAULT_COMBO_FORMAT_STRING.f= ormat(project_name_print, combo)) + else: + print(humble.COMBO_FORMAT_STRING.format(pr= oject_name_print, combo)) + + def get_repo_url(self, repo_url): + if repo_url[-4:].lower() =3D=3D '.git': + return repo_url[:-4] + return repo_url + + def get_repo_name(self, repo_url, manifests): + for name in self.repo_names: + if self.repo_names[name][0] =3D=3D repo_url: + return name + raise EdkrepoInvalidParametersException(humble.REPO_NAME_NOT_FOUND) + + def generate_repo_names(self, repo_urls, manifests): + #Determine the names of the repositories + self.repo_names =3D collections.OrderedDict() + for repo_url in repo_urls: + self.__repo_name_worker(repo_url, manifests) + + #Sort the git repositories so they will be displayed alphabetically + self.repo_names =3D collections.OrderedDict(sorted(self.repo_names= .items())) + names_to_move =3D [] + for repo_name in self.repo_names: + if repo_name.lower().find('edk2') =3D=3D 0: + names_to_move.append(repo_name) + names_to_move =3D sorted(names_to_move, reverse=3DTrue) + for name_to_move in names_to_move: + self.repo_names.move_to_end(name_to_move, False) + names_to_move =3D [] + for repo_name in self.repo_names: + if repo_name.lower().find('intel') =3D=3D 0: + names_to_move.append(repo_name) + names_to_move =3D sorted(names_to_move, reverse=3DTrue) + for name_to_move in names_to_move: + self.repo_names.move_to_end(name_to_move, False) + + def __repo_name_worker(self, repo_url, manifests): + #This is a heuristic that guesses the "name" of a repository by lo= oking + #at the name given to it by the most manifest files. + names =3D collections.defaultdict(int) + for project_name in manifests: + for combo in [c.name for c in manifests[project_name].combinat= ions]: + sources =3D manifests[project_name].get_repo_sources(combo) + for source in sources: + if self.get_repo_url(source.remote_url) =3D=3D repo_ur= l: + names[source.root] +=3D 1 + found_unique_name =3D False + original_best_name =3D None + original_best_name_frequency =3D 0 + while not found_unique_name: + best_name =3D None + best_name_frequency =3D 0 + if len(names) <=3D 0: + if original_best_name_frequency =3D=3D 1: + #If only 1 project uses this name, then append the pro= ject + #name to the directory name to create the repo name + for project_name in manifests: + for combo in [c.name for c in manifests[project_na= me].combinations]: + sources =3D manifests[project_name].get_repo_s= ources(combo) + for source in sources: + if self.get_repo_url(source.remote_url) = =3D=3D repo_url and source.root =3D=3D original_best_name: + best_name =3D "{}-{}".format(original_= best_name, project_name) + best_name_frequency =3D original_best_= name_frequency + else: + best_name =3D repo_url + best_name_frequency =3D 0 + break + for name in names: + if names[name] > best_name_frequency: + best_name =3D name + best_name_frequency =3D names[name] + if best_name is None: + raise EdkrepoManifestInvalidException(humble.REPO_NOT_FOUN= D_IN_MANIFEST.format(repo_url)) + if original_best_name is None: + original_best_name =3D best_name + original_best_name_frequency =3D best_name_frequency + if best_name in self.repo_names: + if self.repo_names[best_name][0] =3D=3D repo_url: + found_unique_name =3D True + else: + #If there is a name collision, then which repo has the= most + #Usage of the name owns the name + if best_name_frequency > self.repo_names[best_name][1]: + old_repo_url =3D self.repo_names[name][0] + del self.repo_names[best_name] + found_unique_name =3D True + self.repo_names[best_name] =3D (repo_url, best_nam= e_frequency) + self.__repo_name_worker(old_repo_url, manifests) + else: + #Use the name given by the second most manifest fi= les + del names[best_name] + else: + found_unique_name =3D True + self.repo_names[best_name] =3D (repo_url, best_name_frequency) diff --git a/setup.py b/setup.py index 73ce9a9..1b9edad 100755 --- a/setup.py +++ b/setup.py @@ -3,7 +3,7 @@ ## @file # setup.py # -# Copyright (c) 2017- 2019, Intel Corporation. All rights reserved.
+# Copyright (c) 2017 - 2019, Intel Corporation. All rights reserved.
# SPDX-License-Identifier: BSD-2-Clause-Patent # =20 @@ -12,7 +12,7 @@ from setuptools import setup setup(name=3D'edkrepo', version=3D'2.0.0', description=3D'The edkrepo tools', - packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.common', 'edkrepo.config', 'edkrepo_manifest_parser', 'projec= t_utils'], + packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.commands.humble', 'edkrepo.common', 'edkrepo.config', 'edkrep= o_manifest_parser', 'project_utils'], package_data=3D{ }, include_package_data=3DTrue, @@ -22,4 +22,3 @@ setup(name=3D'edkrepo', ] } ) - --=20 2.24.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#51673): https://edk2.groups.io/g/devel/message/51673 Mute This Topic: https://groups.io/mt/66074555/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-