From nobody Fri Mar 29 10:01:33 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+59718+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+59718+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1589678980; cv=none; d=zohomail.com; s=zohoarc; b=CnzL21TdSJugnefBpBX70VpLBMu1IPFs2enBzR9V6AXJyRI+mZoQYsy+p0EZbpEdCxkoK+kYm9GxnTSjCAQEnFVHeKkIlYvDtzO3BXzjsf7FVvED1dGDr2/WcFH7YnhoQ7rb/+LNV4hrLpBXl+iSF/gq0EF89q0+b76/5cUS0+k= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589678980; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=dQ75acPO8zspBBmts3TDljuoOnP5ScWCzsbX84mLNEM=; b=VR1EOnD5mjg7OHVOCcQlwlvdKuJUPAjUYjFew4NWzdCjhD3j/LbYkSjlqUzlyT+d0maRzJJaoFwxaYr8vkZ/M423SLVAfqXN9iJQN2QjzR8loxhPVgCr+ncGeoB4JHTHYmlci+qEPSPpMr4ML647v4Bim0E/d6/ZAJZsuFx93gg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+59718+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1589678980049782.923471971624; Sat, 16 May 2020 18:29:40 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id uKfQYY1788612xFrvNRR9hXN; Sat, 16 May 2020 18:29:39 -0700 X-Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx.groups.io with SMTP id smtpd.web12.16194.1589678979198516244 for ; Sat, 16 May 2020 18:29:39 -0700 IronPort-SDR: L68qtA5MZ3EKcuOUwuiOIa4rZskAT/SfXvbVJqQz4a6U6VlLg4wJqBoaO5X1bKbZguvIbEhw6Z shP7Tfc3TaQA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 May 2020 18:29:38 -0700 IronPort-SDR: sr/dnZAffa+Tw+gyWt0YCUpBilu1vrbgFWnYyD43tsm0a+/6Bm6VCrHS8NKYbHJz7gYo54n/QZ eyC9B9JTEewQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,401,1583222400"; d="scan'208";a="342454492" X-Received: from ecbjorge-mobl1.amr.corp.intel.com ([10.209.101.250]) by orsmga001.jf.intel.com with ESMTP; 16 May 2020 18:29:38 -0700 From: "Bjorge, Erik C" To: devel@edk2.groups.io Cc: Ashley E Desimone , Nate DeSimone , Puja Pandya , Bret Barkelew , Prince Agyeman Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH v1 1/2] EdkRepo: Adding selective submodule init script Date: Sat, 16 May 2020 18:29:21 -0700 Message-Id: <0b6d0db81fbb9309a1644ea34eeb4ccf68b19b3b.1589678697.git.erik.c.bjorge@intel.com> In-Reply-To: References: MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,erik.c.bjorge@intel.com X-Gm-Message-State: XJnFb6NyuS2HEOoehhaYnjhnx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1589678979; bh=NCXg6IUi3LrN+q37Q1/9N/aTRoeNda4jBEFSlU3FpnE=; h=Cc:Date:From:Reply-To:Subject:To; b=D9v9WlMulV9WG+WIMgiPnpltPhNllMUlREQmGZBJDAqtK3IL2U3kqRUftST1fl4nE4S aLZt4IiBfj7TOgaPjiN95Vb0jhmyEVoUkt0TcowP8ROkDHJHa2QGPzC/EJhBeG9gY9f52 tZZyq1M55pxrJr/+RcgRRB4CPDEJJJCIlNI= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" Adds selective submodule support functions and command line scripting support. The existing support will be ported over to use this functionality. Signed-off-by: Erik Bjorge Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge --- project_utils/arguments/__init__.py | 8 + project_utils/arguments/submodule_args.py | 24 ++ project_utils/project_utils_strings.py | 24 ++ project_utils/submodule.py | 389 ++++++++++++++++++++++ setup.py | 50 +-- 5 files changed, 470 insertions(+), 25 deletions(-) create mode 100644 project_utils/arguments/__init__.py create mode 100644 project_utils/arguments/submodule_args.py create mode 100644 project_utils/project_utils_strings.py create mode 100644 project_utils/submodule.py diff --git a/project_utils/arguments/__init__.py b/project_utils/arguments/= __init__.py new file mode 100644 index 0000000..0486261 --- /dev/null +++ b/project_utils/arguments/__init__.py @@ -0,0 +1,8 @@ +#!/usr/bin/env python3 +# +## @file +# __init__.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# diff --git a/project_utils/arguments/submodule_args.py b/project_utils/argu= ments/submodule_args.py new file mode 100644 index 0000000..804ce98 --- /dev/null +++ b/project_utils/arguments/submodule_args.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +""" +Strings for command line arguments. +""" + +SUBMOD_MANIFEST_HELP =3D 'The current manifest file.' +SUBMOD_COMBO_HELP =3D 'The current branch combination in use. If a combo i= s not specified, the current ' \ + 'combo will be used.' +SUBMOD_NEW_MANIFEST_HELP =3D 'The new manifest file. Used to test manifes= t upgrade paths.' +SUBMOD_NEW_COMBO_HELP =3D 'The new branch combination to use. Used to tes= t switching combos.' +SUBMOD_DEINIT_HELP =3D 'Performs submodule deinitialization before initial= izing submodules.' +SUBMOD_INIT_FULL_HELP =3D 'Initialize all submodules (recursive).' +SUBMOD_DEINIT_FULL_HELP =3D 'Deinitialize all submodules.' +SUBMOD_WORKSPACE_HELP =3D 'The project workspace root. If not specified t= he current working directory will ' \ + 'be used.' +SUBMOD_VERBOSE_HELP =3D 'Enables verbose messaging.' diff --git a/project_utils/project_utils_strings.py b/project_utils/project= _utils_strings.py new file mode 100644 index 0000000..33c22d2 --- /dev/null +++ b/project_utils/project_utils_strings.py @@ -0,0 +1,24 @@ +#!/usr/bin/env python3 +# +## @file +# humble.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +""" +Contains strings used by the project_utils modules. +""" +# Status update messages +SUBMOD_DEINIT =3D 'Deinitializing submodules' +SUBMOD_DEINIT_FULL =3D 'Deinitializing all submodules' +SUBMOD_INIT_UPDATE =3D 'Initializing/Updating submodules' +SUBMOD_INIT_FULL =3D 'Initializing/Updating all submodules (recursive)' + +# Verbose messages +SUBMOD_INIT_PATH =3D 'Submodule init: {}' +SUBMOD_DEINIT_PATH =3D 'Submodule deinit: {}' +SUBMOD_SYNC_PATH =3D 'Submodule sync: {}' +SUBMOD_UPDATE_PATH =3D 'Submodule update: {}' +SUBMOD_EXCEPTION =3D '- Exception: {}' diff --git a/project_utils/submodule.py b/project_utils/submodule.py new file mode 100644 index 0000000..6534f2c --- /dev/null +++ b/project_utils/submodule.py @@ -0,0 +1,389 @@ +#!/usr/bin/env python3 +# +## @file +# submodule.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +import argparse +import git +import os +import sys +import traceback + +from edkrepo_manifest_parser.edk_manifest import ManifestXml +import project_utils.project_utils_strings as strings +import project_utils.arguments.submodule_args as arguments + + +def _init(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs submodule initialization. Note that if no selective submodul= e initialization + is enabled the init will be handled in _update. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + # Only handle selective initialization with this method. For fully re= cursive init + # it is cleaner to do it as part of the update process. + if submodules is not None: + for sub in submodules: + if verbose: + print(strings.SUBMOD_INIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'init', = '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + + +def _deinit(repo, submodules=3DNone, verbose=3DFalse): + """ + Performs deinitialization of submodules. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + """ + if submodules is None: + output_data =3D repo.git.execute(['git', 'submodule', 'deinit', '-= -all'], + with_extended_output=3DTrue, with_s= tdout=3DTrue) + _display_git_outupt(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_DEINIT_PATH.format(sub.path)) + output_data =3D repo.git.execute(['git', 'submodule', 'deinit'= , '--', sub.path], + with_extended_output=3DTrue, wi= th_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + + +def _update(repo, submodules=3DNone, verbose=3DFalse, recursive=3DFalse): + """ + Performs the update of submodules. This includes the sync and update = operations. + + repo - GitPython Repo object + submodules - The list of selective submodule init objects + verbose - Enable verbose messages + recursive - If submodules is None then use this parameter to determin= e if initialization + should be recursive + """ + # Now perform the update of the submodules. For a fully recursive sub= module init + # this code will update and initialize at the same time. + if submodules is None: + cmd =3D ['git', 'submodule', 'sync'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + cmd =3D ['git', 'submodule', 'update', '--init'] + if recursive: + cmd.append('--recursive') + output_data =3D repo.git.execute(cmd, with_extended_output=3DTrue,= with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + else: + for sub in submodules: + if verbose: + print(strings.SUBMOD_SYNC_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'sync'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + if verbose: + print(strings.SUBMOD_UPDATE_PATH.format(sub.path)) + cmd =3D ['git', 'submodule', 'update', '--init'] + if sub.recursive: + cmd.append('--recursive') + cmd.extend(['--', sub.path]) + output_data =3D repo.git.execute(cmd, with_extended_output=3DT= rue, with_stdout=3DTrue) + _display_git_outupt(output_data, verbose) + return + + +def _display_git_outupt(output_data, verbose=3DFalse): + """ + Displays output from GitPython git commands + + output_data - Output from the git.execute method + verbose - Enable verbose messages + """ + if verbose and output_data[0]: + print(output_data[0]) + if output_data[1]: + print(output_data[1]) + if verbose and output_data[2]: + print(output_data[2]) + + +def _compute_change(current_subs, new_subs): + """ + Determines the list of submodules that have been removed. Also needs = to + determine if any submodules need to have recursive init disabled + + current_subs - List of selective submodule init entries for the curren= t combo + new_subs - List of selective submodule init entries for the new co= mbo + """ + # Create data objects for determining what submodules need to be deini= tialized + tmp_current =3D {x.path: x for x in current_subs} + tmp_new =3D {x.path: x for x in new_subs} + + # Initial deinitialization list + deinit_paths =3D list(set(tmp_current).difference(set(tmp_new))) + + # Check for change in recursive initialization for the specific submod= ules + for path in tmp_new: + if path in tmp_new and path in tmp_current: + if tmp_current[path].recursive and not tmp_new[path].recursive: + deinit_paths.append(path) + + # Create the final list of submodules that need to be deinitialized + return [x for x in current_subs if x.path in deinit_paths] + + +def _get_submodule_enable(manifest, remote_name, combo): + """ + Determines if submodules are enabled for the current repo and combo + + manifest - Manifest object + remote_name - The name of the current remote being processed + combo - The current combo name being processed + """ + repo_sources =3D manifest.get_repo_sources(combo) + for source in repo_sources: + if source.remote_name =3D=3D remote_name: + return source.enable_submodule + return False + + +def _get_submodule_state(remote_name, start_manifest, start_combo, end_man= ifest=3DNone, end_combo=3DNone): + """ + Determines the state of submodules across manifest and combo changes. + + remote_name - The name of the current remote being processed. + start_manifest - Initial manifest parser object. + start_combo - Initial combo. + end_manifest - The manifest parser object for the project if the= manifest is being updated. + end_combo - The combination name at the end of the operation = if being modified. + """ + start_subs =3D start_manifest.get_submodule_init_paths(remote_name, st= art_combo) + start_subs_enabled =3D _get_submodule_enable(start_manifest, remote_na= me, start_combo) + if end_combo is not None: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , end_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, end_combo) + else: + end_subs =3D start_manifest.get_submodule_init_paths(remote_na= me, end_combo) + end_subs_enabled =3D _get_submodule_enable(start_manifest, rem= ote_name, end_combo) + else: + if end_manifest is not None: + end_subs =3D end_manifest.get_submodule_init_paths(remote_name= , start_combo) + end_subs_enabled =3D _get_submodule_enable(end_manifest, remot= e_name, start_combo) + else: + end_subs =3D start_subs + end_subs_enabled =3D start_subs_enabled + return start_subs, start_subs_enabled, end_subs, end_subs_enabled + + +def deinit_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule deinit based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_DEINIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _deinit(repo, None, verbose) + + +def init_full(workspace, manifest, verbose=3DFalse): + """ + Does full submodule init based on the current combo. + + workspace - Path to the current workspace. + manifest - The current manifest parser object. + """ + print(strings.SUBMOD_INIT_FULL) + current_combo =3D manifest.general_config.current_combo + repo_sources =3D manifest.get_repo_sources(current_combo) + for source in repo_sources: + if _get_submodule_enable(manifest, source.remote_name, current_com= bo): + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + _update(repo, None, verbose, True) + + +def deinit_submodules(workspace, start_manifest, start_combo, + end_manifest=3DNone, end_combo=3DNone, + verbose=3DFalse): + """ + Deinitializes the submodules for a project. + + workspace - Path to the current workspace. + start_manifest - The manifest parser object for the project. + start_combo - The combination name at the start of the operatio= n. + end_manifest - The manifest parser object for the project if the= manifest is being updated. If the + manifest file is not being updated us the value N= one. + end_combo - The combination name at the end of the operation.= If the combo will not change + use the value None. + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_DEINIT) + repo_sources =3D start_manifest.get_repo_sources(start_combo) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + start_subs, start_subs_enabled, end_subs, end_subs_enabled =3D _ge= t_submodule_state(source.remote_name, + = start_manifest, + = start_combo, + = end_manifest, + = end_combo) + if not start_subs_enabled and not end_subs_enabled: + # At this point submodules are not enabled on this repo + continue + + # Compute the list of submodules that need to be removed and added + deinit_list =3D _compute_change(start_subs, end_subs) + + # Deinitialize submodules + if (start_subs_enabled and not end_subs_enabled) or (len(start_sub= s) =3D=3D 0 and len(end_subs) !=3D 0): + # Submodules are being disabled for the entire repo so do a + # full deinit on the repo. + _deinit(repo, None, verbose) + else: + # Do the deinit based on the list + _deinit(repo, deinit_list, verbose) + + +def init_submodules(workspace, manifest, combo_name, verbose=3DFalse): + """ + Updates the submodules for a specific repo. Note that operations like= checkout will need to use + the start_combo and end_combo parameter. Operations like clone will j= ust need to use the + start_combo parameter. + + workspace - Path to the current workspace. + manifest - The manifest parser object for the project. + combo_name - The combination name at the start of the operation. + verbose - Enable verbose messages. + """ + # Process each repo that may have submodules enabled + print(strings.SUBMOD_INIT_UPDATE) + repo_sources =3D manifest.get_repo_sources(combo_name) + for source in repo_sources: + # Open the repo and process submodules + try: + repo =3D git.Repo(os.path.join(workspace, source.root)) + except Exception as repo_error: + if args.verbose: + print(strings.SUBMOD_EXCEPTION.format(repo_error)) + continue + + # Collect the submodule initialization data from manifest as well = as if submodules + # should be processed for the repo. + repo_subs =3D manifest.get_submodule_init_paths(source.remote_name= , combo_name) + repo_subs_enabled =3D _get_submodule_enable(manifest, source.remot= e_name, combo_name) + if not repo_subs_enabled: + continue + + # Initialize submodules + if len(repo_subs) > 0: + _init(repo, repo_subs, verbose) + + # Perform sync/update + if len(repo_subs) =3D=3D 0: + _update(repo, None, verbose) + else: + _update(repo, repo_subs, verbose) + + +if __name__ =3D=3D '__main__': + def parse_args(): + parser =3D argparse.ArgumentParser() + parser.add_argument('manifest_file', metavar=3D'MANIFEST', help=3D= arguments.SUBMOD_MANIFEST_HELP) + parser.add_argument('--combo', default=3DNone, help=3Darguments.SU= BMOD_COMBO_HELP) + parser.add_argument('--new-manifest', default=3DNone, help=3Dargum= ents.SUBMOD_NEW_MANIFEST_HELP) + parser.add_argument('--new-combo', default=3DNone, help=3Dargument= s.SUBMOD_NEW_COMBO_HELP) + parser.add_argument('--deinit', action=3D'store_true', help=3Dargu= ments.SUBMOD_DEINIT_HELP) + parser.add_argument('--init-full', action=3D'store_true', help=3Da= rguments.SUBMOD_INIT_FULL_HELP) + parser.add_argument('--deinit-full', action=3D'store_true', help= =3Darguments.SUBMOD_DEINIT_FULL_HELP) + parser.add_argument('--workspace', default=3D'.', help=3Darguments= .SUBMOD_WORKSPACE_HELP) + parser.add_argument('--verbose', action=3D'store_true', help=3Darg= uments.SUBMOD_VERBOSE_HELP) + return parser.parse_args() + + def main(args): + # Optional data init + new_manifest =3D None + all_new_combos =3D None + + # Extract basic manifest data + manifest =3D ManifestXml(args.manifest_file) + init_manifest =3D manifest + all_combos =3D manifest.combinations + all_combos.extend(manifest.archived_combinations) + if args.new_manifest is not None: + new_manifest =3D ManifestXml(args.new_manifest) + all_new_combos =3D new_manifest.combinations + all_new_combos.extend(new_manifest.archived_combinations) + init_manifest =3D new_manifest + + # Determine current and new combo information + current_combo =3D manifest.general_config.current_combo + new_combo =3D args.new_combo + for combo in all_combos: + if args.combo is not None: + if args.combo.lower() =3D=3D combo.name.lower(): + current_combo =3D combo.name + if args.new_combo is not None: + if args.new_combo.lower() =3D=3D combo.name.lower(): + new_combo =3D combo.name + init_combo =3D current_combo + if new_combo is not None: + init_combo =3D new_combo + + if args.deinit_full: + deinit_full(args.workspace, manifest, args.verbose) + elif args.init_full: + init_full(args.workspace, manifest, args.verbose) + else: + if args.deinit: + deinit_submodules(args.workspace, manifest, current_combo, + new_manifest, new_combo, args.verbose) + init_submodules(args.workspace, init_manifest, init_combo, arg= s.verbose) + + return 0 + + try: + args =3D parse_args() + sys.exit(main(args)) + except Exception: + if args.verbose: + traceback.print_exc() + sys.exit(1) diff --git a/setup.py b/setup.py index 5d2f55a..a855000 100755 --- a/setup.py +++ b/setup.py @@ -1,27 +1,27 @@ -#!/usr/bin/env python3 -# -## @file setup.py -# -# Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.
-# SPDX-License-Identifier: BSD-2-Clause-Patent -# - -from setuptools import setup - -setup(name=3D'edkrepo', - version=3D'2.0.0', - description=3D'The edkrepo tools', - packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.commands.humble', +#!/usr/bin/env python3 +# +## @file setup.py +# +# Copyright (c) 2017 - 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +from setuptools import setup + +setup(name=3D'edkrepo', + version=3D'2.0.0', + description=3D'The edkrepo tools', + packages=3D['edkrepo', 'edkrepo.commands', 'edkrepo.commands.argumen= ts', 'edkrepo.commands.humble', 'edkrepo.git_automation', 'edkrepo.common', 'edkrepo.commo= n.workspace_maintenance', 'edkrepo.common.workspace_maintenance.humble', 'edkrepo.co= nfig', 'edkrepo.config.humble', - 'edkrepo_manifest_parser', 'project_utils'], - package_data=3D{ - }, - include_package_data=3DTrue, - entry_points=3D{ - 'console_scripts': [ - 'edkrepo =3D edkrepo.edkrepo_entry_point:main', - 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' - ] - } - ) + 'edkrepo_manifest_parser', 'project_utils', 'project_utils= .arguments'], + package_data=3D{ + }, + include_package_data=3DTrue, + entry_points=3D{ + 'console_scripts': [ + 'edkrepo =3D edkrepo.edkrepo_entry_point:main', + 'command_completion_edkrepo =3D edkrepo.command_completion_e= dkrepo:main' + ] + } + ) --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#59718): https://edk2.groups.io/g/devel/message/59718 Mute This Topic: https://groups.io/mt/74262441/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri Mar 29 10:01:33 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+59719+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+59719+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1589678986; cv=none; d=zohomail.com; s=zohoarc; b=DtKizJTDdDnToUBLPRj4NweYJFfuIVlO55dVPlsBMfhDWkDuz+ygyhfXFjzgEphslaNSCxWLuWYT7IYq/i4pRfkrMKo+x77pRoXaU6szqwfk+5iV1WiWkQVk1Sc1Odng+Uu9g/6bGoZ71aMsHmAHKs8SIpvY/GfBXi97XCYUmus= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1589678986; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=clJ5OniogYdGM79p//ci0cqEo29suk0hG2RxeSY5tq0=; b=FgFUbLfwJYROBWBJpqLkcBTtDD9hOQ+6Xam65W+jeut5CKNaPcM7kdknsIhJpQnVRASkjmpqLWTFjtXujxvlEZxr7F4T38JICS0Gtk1zIASMo3sZzumfa+vuG5ETmJcZSOrZ8ra5cYj/PaQ5Ef37JJ2Alzfd7nyyDwJ+C/53jaA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+59719+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1589678986080296.251404389894; Sat, 16 May 2020 18:29:46 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id pYl4YY1788612x8LCx98ckOL; Sat, 16 May 2020 18:29:45 -0700 X-Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by mx.groups.io with SMTP id smtpd.web11.16492.1589678985230694148 for ; Sat, 16 May 2020 18:29:45 -0700 IronPort-SDR: 1ea2Z6FHrH3rZCTnfQmI+yqIHPYiWjhqphTH+fxLqDZBb/c4HBKQuJgcIGKyliFpKkww0bCRyy 4/6fa2elymGA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 May 2020 18:29:44 -0700 IronPort-SDR: olRFQ7ziAmMaK9MZqwCmMyL0NRRLCMHqJJmjhExpN+NSY3wBmSOKle8bk09rgdgjQ+BLrTrjWJ SagKV0C/NDCw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,401,1583222400"; d="scan'208";a="342454514" X-Received: from ecbjorge-mobl1.amr.corp.intel.com ([10.209.101.250]) by orsmga001.jf.intel.com with ESMTP; 16 May 2020 18:29:43 -0700 From: "Bjorge, Erik C" To: devel@edk2.groups.io Cc: Ashley E Desimone , Nate DeSimone , Puja Pandya , Bret Barkelew , Prince Agyeman Subject: [edk2-devel] [edk2-staging/EdkRepo] [PATCH v1 2/2] EdkRepo: Update commands to use new submodule code Date: Sat, 16 May 2020 18:29:22 -0700 Message-Id: <66a3235a3f4de04e0b8a549901f8321f8d1cf950.1589678697.git.erik.c.bjorge@intel.com> In-Reply-To: References: MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,erik.c.bjorge@intel.com X-Gm-Message-State: Xup4UkXfRb0SMUFUVf4Rlkeox1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1589678985; bh=uMRdtx6hG9WCQNdRyoQkE4TVs6YXUTkXAa1z7k9KvYk=; h=Cc:Date:From:Reply-To:Subject:To; b=ifj9OINmtLcvLqjqwloo9T9swwk9gaapMIz9ykiMgJWlbp1Sis2Geb97sGya7KVkLmU cu8wh9nmBvuNEs5HTerYMk1iznJv4xelkD7Eih1qXZ5NEdaI/d4OZ/pgF27PPHg5h3WhQ gXlUb5KeLt19dqAmkdYzF4468cdIuMD4LVM= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" Replaced use of maintain_submodules to use new common code. This allows for selective submodule initialization via the manifest file. Cc: Ashley E Desimone Cc: Nate DeSimone Cc: Puja Pandya Cc: Bret Barkelew Cc: Prince Agyeman Cc: Erik Bjorge Signed-off-by: Erik Bjorge Reviewed-by: Ashley DeSimone --- edkrepo/commands/checkout_pin_command.py | 6 ++++ edkrepo/commands/clone_command.py | 20 +++++++---- edkrepo/commands/sync_command.py | 23 +++++++++---- edkrepo/common/common_repo_functions.py | 43 +++++------------------- 4 files changed, 44 insertions(+), 48 deletions(-) diff --git a/edkrepo/commands/checkout_pin_command.py b/edkrepo/commands/ch= eckout_pin_command.py index 4aaf2b5..c250c49 100644 --- a/edkrepo/commands/checkout_pin_command.py +++ b/edkrepo/commands/checkout_pin_command.py @@ -22,6 +22,8 @@ from edkrepo.common.workspace_maintenance.manifest_repos_= maintenance import list from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impor= t find_source_manifest_repo from edkrepo.config.config_factory import get_workspace_path, get_workspac= e_manifest from edkrepo_manifest_parser.edk_manifest import ManifestXml +from project_utils.submodule import deinit_submodules, init_submodules + =20 class CheckoutPinCommand(EdkrepoCommand): def __init__(self): @@ -68,11 +70,15 @@ class CheckoutPinCommand(EdkrepoCommand): if sparse_enabled: print(SPARSE_RESET) reset_sparse_checkout(workspace_path, manifest_sources) + submodule_combo =3D pin.general_config.current_combo + deinit_submodules(workspace_path, manifest, manifest.general_confi= g.current_combo, + pin, submodule_combo, args.verbose) pin_repo_sources =3D pin.get_repo_sources(pin.general_config.curre= nt_combo) try: checkout_repos(args.verbose, args.override, pin_repo_sources, = workspace_path, manifest) manifest.write_current_combo(humble.PIN_COMBO.format(args.pinf= ile)) finally: + init_submodules(workspace_path, pin, submodule_combo, args.ver= bose) if sparse_enabled: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, pin_repo_sources, manifest) diff --git a/edkrepo/commands/clone_command.py b/edkrepo/commands/clone_com= mand.py index 179aaf7..653e10b 100644 --- a/edkrepo/commands/clone_command.py +++ b/edkrepo/commands/clone_command.py @@ -24,6 +24,7 @@ from edkrepo.common.workspace_maintenance.manifest_repos_= maintenance import pull from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impor= t list_available_manifest_repos from edkrepo.common.workspace_maintenance.humble.manifest_repos_maintenanc= e_humble import PROJ_NOT_IN_REPO, SOURCE_MANIFEST_REPO_NOT_FOUND from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml +from project_utils.submodule import init_submodules =20 =20 class CloneCommand(EdkrepoCommand): @@ -104,7 +105,6 @@ class CloneCommand(EdkrepoCommand): manifest =3D ManifestXml(local_manifest_path) =20 # Process the combination name and make sure it can be found in th= e manifest - combo_name =3D None if args.Combination is not None: try: combo_name =3D case_insensitive_single_match(args.Combinat= ion, combinations_in_manifest(manifest)) @@ -118,13 +118,15 @@ class CloneCommand(EdkrepoCommand): # Since pin files are subset of manifest files they do not hav= e a "default combo" it is set to None. In this # case use the current_combo instead. combo_name =3D manifest.general_config.current_combo + else: + # If a combo was not specified or a pin file used the default = combo should be cloned. Also ensure that the + # current combo is updated to match. + combo_name =3D manifest.general_config.default_combo + manifest.write_current_combo(combo_name) =20 # Get the list of repos to clone and clone them - manifest_config =3D manifest.general_config - if combo_name: - repo_sources_to_clone =3D manifest.get_repo_sources(combo_name) - else: - repo_sources_to_clone =3D manifest.get_repo_sources(manifest_c= onfig.default_combo) + repo_sources_to_clone =3D manifest.get_repo_sources(combo_name) + #check that the repo sources do not contain duplicated local roots local_roots =3D [r.root for r in repo_sources_to_clone] for root in local_roots: @@ -137,7 +139,11 @@ class CloneCommand(EdkrepoCommand): # Set up submodule alt url config settings prior to cloning any re= pos submodule_included_configs =3D write_included_config(manifest.remo= tes, manifest.submodule_alternate_remotes, local_manifest_dir) write_conditional_include(workspace_dir, repo_sources_to_clone, su= bmodule_included_configs) - clone_repos(args, workspace_dir, repo_sources_to_clone, project_cl= ient_side_hooks, config, args.skip_submodule, manifest) + clone_repos(args, workspace_dir, repo_sources_to_clone, project_cl= ient_side_hooks, config, manifest) + + # Init submodules + if not args.skip_submodule: + init_submodules(workspace_dir, manifest, combo_name, args.verb= ose) =20 # Perform a sparse checkout if requested. use_sparse =3D args.sparse diff --git a/edkrepo/commands/sync_command.py b/edkrepo/commands/sync_comma= nd.py index da99397..d3c724f 100644 --- a/edkrepo/commands/sync_command.py +++ b/edkrepo/commands/sync_command.py @@ -38,7 +38,7 @@ from edkrepo.common.common_repo_functions import checkout= _repos, check_dirty_rep from edkrepo.common.common_repo_functions import update_editor_config from edkrepo.common.common_repo_functions import update_repo_commit_templa= te, get_latest_sha from edkrepo.common.common_repo_functions import has_primary_repo_remote, = fetch_from_primary_repo, in_sync_with_primary -from edkrepo.common.common_repo_functions import update_hooks, maintain_su= bmodules, combinations_in_manifest +from edkrepo.common.common_repo_functions import update_hooks, combination= s_in_manifest from edkrepo.common.common_repo_functions import write_included_config, re= move_included_config from edkrepo.common.workspace_maintenance.workspace_maintenance import gen= erate_name_for_obsolete_backup from edkrepo.common.workspace_maintenance.manifest_repos_maintenance impor= t pull_workspace_manifest_repo @@ -48,6 +48,7 @@ from edkrepo.common.ui_functions import init_color_console from edkrepo.config.config_factory import get_workspace_path, get_workspac= e_manifest, get_edkrepo_global_data_directory from edkrepo.config.config_factory import get_workspace_manifest_file from edkrepo_manifest_parser.edk_manifest import CiIndexXml, ManifestXml +from project_utils.submodule import deinit_submodules, init_submodules =20 =20 class SyncCommand(EdkrepoCommand): @@ -84,6 +85,7 @@ class SyncCommand(EdkrepoCommand): current_combo =3D initial_manifest.general_config.current_combo initial_sources =3D initial_manifest.get_repo_sources(current_comb= o) initial_hooks =3D initial_manifest.repo_hooks + initial_combo =3D current_combo =20 source_global_manifest_repo =3D find_source_manifest_repo(initial_= manifest, config['cfg_file'], config['user_cfg_file'], args.source_manifest= _repo) pull_workspace_manifest_repo(initial_manifest, config['cfg_file'],= config['user_cfg_file'], args.source_manifest_repo, False) @@ -100,6 +102,7 @@ class SyncCommand(EdkrepoCommand): if not args.update_local_manifest: self.__check_for_new_manifest(args, config, initial_manifest, = workspace_path, global_manifest_directory) check_dirty_repos(initial_manifest, workspace_path) + # Determine if sparse checkout needs to be disabled for this opera= tion sparse_settings =3D initial_manifest.sparse_settings sparse_enabled =3D sparse_checkout_enabled(workspace_path, initial= _sources) @@ -113,7 +116,7 @@ class SyncCommand(EdkrepoCommand): reset_sparse_checkout(workspace_path, initial_sources) =20 # Get the latest manifest if requested - if args.update_local_manifest: #NOTE: hyphens in arg name replaced= with underscores due to argparse + if args.update_local_manifest: # NOTE: hyphens in arg name replac= ed with underscores due to argparse self.__update_local_manifest(args, config, initial_manifest, w= orkspace_path, global_manifest_directory) manifest =3D get_workspace_manifest() if args.update_local_manifest: @@ -127,6 +130,12 @@ class SyncCommand(EdkrepoCommand): repo_sources_to_sync =3D manifest.get_repo_sources(current_com= bo) manifest.write_current_combo(current_combo) =20 + # At this point both new and old manifest files are ready so we ca= n deinit any + # submodules that are removed due to a manifest update. + if not args.skip_submodule: + deinit_submodules(workspace_path, initial_manifest, initial_co= mbo, + manifest, current_combo, args.verbose) + sync_error =3D False # Calculate the hooks which need to be updated, added or removed f= or the sync if args.update_local_manifest: @@ -195,16 +204,16 @@ class SyncCommand(EdkrepoCommand): elif args.verbose: print(NO_SYNC_DETACHED_HEAD.format(repo_to_sync.root)) =20 - if not args.skip_submodule: - if repo_to_sync.enable_submodule: - # Perform submodule updates and url redirection - maintain_submodules(repo_to_sync, repo) # Update commit message templates update_repo_commit_template(workspace_path, repo, repo_to_sync= , config, source_global_manifest_repo) =20 if sync_error: print(SYNC_ERROR) =20 + # Initialize submodules + if not args.skip_submodule: + init_submodules(workspace_path, manifest, current_combo, args.= verbose) + # Restore sparse checkout state if sparse_enabled: print(SPARSE_CHECKOUT) @@ -326,7 +335,7 @@ class SyncCommand(EdkrepoCommand): print(path_to_source) if len(sources_to_remove) > 0: print(SYNC_REMOVE_LIST_END_FORMATTING) - clone_repos(args, workspace_path, sources_to_clone, new_manife= st_to_check.repo_hooks, config, args.skip_submodule, new_manifest_to_check) + clone_repos(args, workspace_path, sources_to_clone, new_manife= st_to_check.repo_hooks, config, new_manifest_to_check) # Make a list of and only checkout repos that were newly clone= d. Sync keeps repos on their initial active branches # cloning the entire combo can prevent existing repos from cor= rectly being returned to their proper branch repos_to_checkout =3D [] diff --git a/edkrepo/common/common_repo_functions.py b/edkrepo/common/commo= n_repo_functions.py index 61133f1..18e545b 100644 --- a/edkrepo/common/common_repo_functions.py +++ b/edkrepo/common/common_repo_functions.py @@ -68,12 +68,13 @@ from edkrepo_manifest_parser.edk_manifest_validation im= port validate_manifestrep from edkrepo_manifest_parser.edk_manifest_validation import get_manifest_v= alidation_status from edkrepo_manifest_parser.edk_manifest_validation import print_manifest= _errors from edkrepo_manifest_parser.edk_manifest_validation import validate_manif= estfiles +from project_utils.submodule import deinit_submodules, init_submodules =20 CLEAR_LINE =3D '\x1b[K' DEFAULT_REMOTE_NAME =3D 'origin' PRIMARY_REMOTE_NAME =3D 'primary' =20 -def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, skip_submodule, manifest): +def clone_repos(args, workspace_dir, repos_to_clone, project_client_side_h= ooks, config, manifest): for repo_to_clone in repos_to_clone: local_repo_path =3D os.path.join(workspace_dir, repo_to_clone.root) local_repo_url =3D repo_to_clone.remote_url @@ -108,10 +109,6 @@ def clone_repos(args, workspace_dir, repos_to_clone, p= roject_client_side_hooks, else: raise EdkrepoManifestInvalidException(MISSING_BRANCH_COMMIT) =20 - if not skip_submodule: - if repo_to_clone.enable_submodule: - maintain_submodules(repo_to_clone, repo, args.verbose) - try: if 'source_manifest_repo' in vars(args).keys(): src_manifest_repo =3D find_source_manifest_repo(manifest, = config['cfg_file'], config['user_cfg_file'], args.source_manifest_repo) @@ -184,32 +181,6 @@ def write_conditional_include(workspace_path, repo_sou= rces, included_configs): gitglobalconfig.add_section(section) gitglobalconfig.set(section, 'path', path) =20 -def maintain_submodules(repo_sources, repo, verbose =3D False): - try: - output_data =3D repo.git.execute(['git', 'submodule', 'init'], wit= h_extended_output=3DTrue, with_stdout=3DTrue) - if verbose and output_data[0]: - print(output_data[0]) - if output_data[1]: - print(output_data[1]) - if verbose and output_data[2]: - print(output_data[2]) - output_data =3D repo.git.execute(['git', 'submodule', 'sync', '--r= ecursive'], with_extended_output=3DTrue, with_stdout=3DTrue) - if verbose and output_data[0]: - print(output_data[0]) - if output_data[1]: - print(output_data[1]) - if verbose and output_data[2]: - print(output_data[2]) - output_data =3D repo.git.execute(['git', 'submodule', 'update', '-= -recursive'], with_extended_output=3DTrue, with_stdout=3DTrue) - if verbose and output_data[0]: - print(output_data[0]) - if output_data[1]: - print(output_data[1]) - if verbose and output_data[2]: - print(output_data[2]) - except: - raise EdkrepoGitException(SUBMODULE_FAILURE.format(repo_sources.re= mote_name)) - def install_hooks(hooks, local_repo_path, repo_for_install, config, global= _manifest_directory): # Determine the which hooks are for the repo in question and which are= from a URL based source or are in a global # manifest repo relative path @@ -399,9 +370,6 @@ def checkout_repos(verbose, override, repos_to_checkout= , workspace_path, manifes else: raise EdkrepoManifestInvalidException(MISSING_BRANCH_COMMIT) =20 - if repo_to_checkout.enable_submodule: - maintain_submodules(repo_to_checkout, repo, verbose) - def validate_manifest_repo(manifest_repo, verbose=3DFalse, archived=3DFals= e): print(VERIFY_GLOBAL) if archived: @@ -514,9 +482,11 @@ def checkout(combination_or_sha, verbose=3DFalse, over= ride=3DFalse, log=3DNone): # Create combo_or_sha so we have original input and do not introduce a= ny # unintended behavior by messing with parameters. combo_or_sha =3D combination_or_sha + submodule_combo =3D manifest.general_config.current_combo try: # Try to handle normalize combo name to match the manifest file. combo_or_sha =3D case_insensitive_single_match(combo_or_sha, combi= nations_in_manifest(manifest)) + submodule_combo =3D combo_or_sha except: # No match so leave it alone. It must be a SHA1 or a bad combo na= me. pass @@ -528,6 +498,10 @@ def checkout(combination_or_sha, verbose=3DFalse, over= ride=3DFalse, log=3DNone): log=3Dlog) initial_repo_sources =3D manifest.get_repo_sources(manifest.general_co= nfig.current_combo) =20 + # Deinit any submodules that have been removed. + deinit_submodules(workspace_path, manifest, manifest.general_config.cu= rrent_combo, + manifest, submodule_combo, verbose) + # Disable sparse checkout current_repos =3D initial_repo_sources sparse_enabled =3D sparse_checkout_enabled(workspace_path, initial_rep= o_sources) @@ -566,6 +540,7 @@ def checkout(combination_or_sha, verbose=3DFalse, overr= ide=3DFalse, log=3DNone): # Return to the initial combo, since there was an issue with cheki= ng out the selected combo checkout_repos(verbose, override, initial_repo_sources, workspace_= path, manifest) finally: + init_submodules(workspace_path, manifest, submodule_combo, verbose) if sparse_enabled or sparse_diff: print(SPARSE_CHECKOUT) sparse_checkout(workspace_path, current_repos, manifest) --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#59719): https://edk2.groups.io/g/devel/message/59719 Mute This Topic: https://groups.io/mt/74262442/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-