From nobody Thu Apr 25 14:43:28 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+83925+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83925+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1637646315; cv=none; d=zohomail.com; s=zohoarc; b=I7f7Fb7iaKQTpijUboAWhWM/QuZerjt2Sl3UEA6F0g5QK1voJDGze274e0XIF+YVAWMKJ8k4qm6+HT+m8Pz2AsPcdYy6FqDGS/iXDPXQDuzTZ1+mZCryklfZUJcB4ZUl8B4eRy1ZUIoL4lZkEQShq/USG9GAc6m4e5ihcs9cYeY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1637646315; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=rxSzqIsZm+rwYNRMCHxJEVT63uBJBTAgZGgW9Ljnv2U=; b=VPJDC8LOojy86A8McRRsV+Z3JLnEALydcLtN6LBoiB2D1jurO43NQPpPO4r4jD/KLbVeohOFSQxz7Yq9rM1/SgJz2A3HOv0dvE6eTU8Mgho+6vJTAXZ/FuymCaoIOeEFV+vg88DhKFrQoawNsXubBn2hRemKsGHXB3jvqvdrqz8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83925+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1637646315280866.9685668108963; Mon, 22 Nov 2021 21:45:15 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id Kf7hYY1788612xseqbioJV8p; Mon, 22 Nov 2021 21:45:14 -0800 X-Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mx.groups.io with SMTP id smtpd.web09.7744.1637646311113360089 for ; Mon, 22 Nov 2021 21:45:12 -0800 X-IronPort-AV: E=McAfee;i="6200,9189,10176"; a="234903537" X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="234903537" X-Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:10 -0800 X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="674336717" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.212.191.131]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:09 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao , Michael Kubacki Subject: [edk2-devel] [Patch 1/3] .pytool/Plugin/EccCheck: Remove RevertCode() Date: Mon, 22 Nov 2021 21:44:53 -0800 Message-Id: <20211123054455.600-2-michael.d.kinney@intel.com> In-Reply-To: <20211123054455.600-1-michael.d.kinney@intel.com> References: <20211123054455.600-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: dKjtdFiEAkFz7qKbrJiL0XRqx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1637646314; bh=qfq+IvPAM8fakds5XqEpt1p+t9q2LnEJmYij1w6zHDk=; h=Cc:Date:From:Reply-To:Subject:To; b=N/7b1gfjtTgEcQikKrNHGESHHVfol1cKJ8NbIwLOCJbHLC0+QG3xiGBmXHcC4AbevuT K0hkhfBk7YQjzEhJ/75fyzGwhmNE41Gv1UXB9fRDwiuBN0+iW8wo2fzdOM99BXw/KGKR4 tfmYfZfKebipOemeZOqN3reNIw9upTYPDVE= X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1637646315924100003 Content-Type: text/plain; charset="utf-8" REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2986 The RevertCode() method uses git reset which can remove local changes. Instead of modifying the local files, a copy of the package passed into the EccCheck tool is copied to a temp directory in Build/ecctemp. This same temp directory is also used for exception.xml. The working directory used by ECC is also set to this same temp directory. The combination of these changes eliminates operations that that modified the git state. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Cc: Michael Kubacki Signed-off-by: Michael D Kinney --- .pytool/Plugin/EccCheck/EccCheck.py | 96 +++++++++++++++++------------ 1 file changed, 56 insertions(+), 40 deletions(-) diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/= EccCheck.py index 2d0612269b2e..82a3f14d6d5c 100644 --- a/.pytool/Plugin/EccCheck/EccCheck.py +++ b/.pytool/Plugin/EccCheck/EccCheck.py @@ -69,32 +69,53 @@ class EccCheck(ICiBuildPlugin): env.set_shell_var('WORKSPACE', workspace_path) env.set_shell_var('PACKAGES_PATH', os.pathsep.join(Edk2pathObj.Pac= kagePathList)) self.ECC_PASS =3D True - self.ApplyConfig(pkgconfig, workspace_path, basetools_path, packag= ename) + + # Create temp directory + temp_path =3D os.path.join(workspace_path, 'Build', 'ecctemp') + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) + # Copy package being scanned to temp_path + shutil.copytree ( + os.path.join(workspace_path, packagename), + os.path.join(temp_path, packagename), + symlinks=3DTrue + ) + # Copy exception.xml to temp_path + shutil.copyfile ( + os.path.join(basetools_path, "Source", "Python", "Ecc", "excepti= on.xml"), + os.path.join(temp_path, "exception.xml") + ) + + self.ApplyConfig(pkgconfig, temp_path, packagename) modify_dir_list =3D self.GetModifyDir(packagename) patch =3D self.GetDiff(packagename) - ecc_diff_range =3D self.GetDiffRange(patch, packagename, workspace= _path) - self.GenerateEccReport(modify_dir_list, ecc_diff_range, workspace_= path, basetools_path) - ecc_log =3D os.path.join(workspace_path, "Ecc.log") - self.RevertCode() + ecc_diff_range =3D self.GetDiffRange(patch, packagename, temp_path) + # + # Set workingdir to Build output directory because Ecc generates t= emp files + # Can not set workingdir to temp_path because that can be on a dif= ferent + # drive letter for some CI platforms and RunCmd() does not work co= rrectly + # if working dir is on a different drive letter. + # + self.GenerateEccReport(modify_dir_list, ecc_diff_range, temp_path,= basetools_path, os.path.join (workspace_path, 'Build')) + ecc_log =3D os.path.join(temp_path, "Ecc.log") if self.ECC_PASS: + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) tc.SetSuccess() - self.RemoveFile(ecc_log) return 0 else: with open(ecc_log, encoding=3D'utf8') as output: ecc_output =3D output.readlines() for line in ecc_output: logging.error(line.strip()) - self.RemoveFile(ecc_log) - tc.SetFailed("EccCheck failed for {0}".format(packagename), "E= cc detected issues") + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) + tc.SetFailed("EccCheck failed for {0}".format(packagename), "C= HECK FAILED") return 1 =20 - def RevertCode(self) -> None: - submoudle_params =3D "submodule update --init" - RunCmd("git", submoudle_params) - reset_params =3D "reset HEAD --hard" - RunCmd("git", reset_params) - def GetDiff(self, pkg: str) -> List[str]: return_buffer =3D StringIO() params =3D "diff --unified=3D0 origin/master HEAD" @@ -105,11 +126,6 @@ class EccCheck(ICiBuildPlugin): =20 return patch =20 - def RemoveFile(self, file: str) -> None: - if os.path.exists(file): - os.remove(file) - return - def GetModifyDir(self, pkg: str) -> List[str]: return_buffer =3D StringIO() params =3D "diff --name-status" + ' HEAD' + ' origin/master' @@ -132,14 +148,14 @@ class EccCheck(ICiBuildPlugin): modify_dir_list =3D list(set(modify_dir_list)) return modify_dir_list =20 - def GetDiffRange(self, patch_diff: List[str], pkg: str, workingdir: st= r) -> Dict[str, List[Tuple[int, int]]]: + def GetDiffRange(self, patch_diff: List[str], pkg: str, temp_path: str= ) -> Dict[str, List[Tuple[int, int]]]: IsDelete =3D True StartCheck =3D False range_directory: Dict[str, List[Tuple[int, int]]] =3D {} for line in patch_diff: modify_file =3D self.FindModifyFile.findall(line) if modify_file and pkg in modify_file[0] and not StartCheck an= d os.path.isfile(modify_file[0]): - modify_file_comment_dic =3D self.GetCommentRange(modify_fi= le[0], workingdir) + modify_file_comment_dic =3D self.GetCommentRange(modify_fi= le[0], temp_path) IsDelete =3D False StartCheck =3D True modify_file_dic =3D modify_file[0] @@ -158,11 +174,13 @@ class EccCheck(ICiBuildPlugin): range_directory[modify_file_dic].append(i) return range_directory =20 - def GetCommentRange(self, modify_file: str, workingdir: str) -> List[T= uple[int, int]]: - modify_file_path =3D os.path.join(workingdir, modify_file) + def GetCommentRange(self, modify_file: str, temp_path: str) -> List[Tu= ple[int, int]]: + comment_range: List[Tuple[int, int]] =3D [] + modify_file_path =3D os.path.join(temp_path, modify_file) + if not os.path.exists (modify_file_path): + return comment_range with open(modify_file_path) as f: line_no =3D 1 - comment_range: List[Tuple[int, int]] =3D [] Start =3D False for line in f: if line.startswith('/**'): @@ -179,35 +197,33 @@ class EccCheck(ICiBuildPlugin): return comment_range =20 def GenerateEccReport(self, modify_dir_list: List[str], ecc_diff_range= : Dict[str, List[Tuple[int, int]]], - workspace_path: str, basetools_path: str) -> No= ne: + temp_path: str, basetools_path: str, workingdir= : str) -> None: ecc_need =3D False ecc_run =3D True - config =3D os.path.join(basetools_path, "Source", "Python", "Ecc",= "config.ini") - exception =3D os.path.join(basetools_path, "Source", "Python", "Ec= c", "exception.xml") - report =3D os.path.join(workspace_path, "Ecc.csv") + config =3D os.path.normpath(os.path.join(basetools_path, "Sourc= e", "Python", "Ecc", "config.ini")) + exception =3D os.path.normpath(os.path.join(temp_path, "exception.= xml")) + report =3D os.path.normpath(os.path.join(temp_path, "Ecc.csv")) for modify_dir in modify_dir_list: - target =3D os.path.join(workspace_path, modify_dir) + target =3D os.path.normpath(os.path.join(temp_path, modify_dir= )) logging.info('Run ECC tool for the commit in %s' % modify_dir) ecc_need =3D True ecc_params =3D "-c {0} -e {1} -t {2} -r {3}".format(config, ex= ception, target, report) - return_code =3D RunCmd("Ecc", ecc_params, workingdir=3Dworkspa= ce_path) + return_code =3D RunCmd("Ecc", ecc_params, workingdir=3Dworking= dir) if return_code !=3D 0: ecc_run =3D False break if not ecc_run: logging.error('Fail to run ECC tool') - self.ParseEccReport(ecc_diff_range, workspace_path) + self.ParseEccReport(ecc_diff_range, temp_path) =20 if not ecc_need: logging.info("Doesn't need run ECC check") =20 - revert_params =3D "checkout -- {}".format(exception) - RunCmd("git", revert_params) return =20 - def ParseEccReport(self, ecc_diff_range: Dict[str, List[Tuple[int, int= ]]], workspace_path: str) -> None: - ecc_log =3D os.path.join(workspace_path, "Ecc.log") - ecc_csv =3D os.path.join(workspace_path, "Ecc.csv") + def ParseEccReport(self, ecc_diff_range: Dict[str, List[Tuple[int, int= ]]], temp_path: str) -> None: + ecc_log =3D os.path.join(temp_path, "Ecc.log") + ecc_csv =3D os.path.join(temp_path, "Ecc.csv") row_lines =3D [] ignore_error_code =3D self.GetIgnoreErrorCode() if os.path.exists(ecc_csv): @@ -236,16 +252,16 @@ class EccCheck(ICiBuildPlugin): log.writelines(all_line) return =20 - def ApplyConfig(self, pkgconfig: Dict[str, List[str]], workspace_path:= str, basetools_path: str, pkg: str) -> None: + def ApplyConfig(self, pkgconfig: Dict[str, List[str]], temp_path: str,= pkg: str) -> None: if "IgnoreFiles" in pkgconfig: for a in pkgconfig["IgnoreFiles"]: - a =3D os.path.join(workspace_path, pkg, a) + a =3D os.path.join(temp_path, pkg, a) a =3D a.replace(os.sep, "/") =20 logging.info("Ignoring Files {0}".format(a)) if os.path.exists(a): if os.path.isfile(a): - self.RemoveFile(a) + os.remove(a) elif os.path.isdir(a): shutil.rmtree(a) else: @@ -253,7 +269,7 @@ class EccCheck(ICiBuildPlugin): =20 if "ExceptionList" in pkgconfig: exception_list =3D pkgconfig["ExceptionList"] - exception_xml =3D os.path.join(basetools_path, "Source", "Pyth= on", "Ecc", "exception.xml") + exception_xml =3D os.path.join(temp_path, "exception.xml") try: logging.info("Appending exceptions") self.AppendException(exception_list, exception_xml) --=20 2.32.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#83925): https://edk2.groups.io/g/devel/message/83925 Mute This Topic: https://groups.io/mt/87253499/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Thu Apr 25 14:43:28 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+83926+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83926+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1637646316; cv=none; d=zohomail.com; s=zohoarc; b=mHWzAPFh2VPrBtbCHMGwaskpqc1Iv3aONXoJUKkRRgvX33nFvkip19O2WG57Ye+/BNIR5SdLHp/fdSfkQF31Svjz+7kJUz0fiL3M/9dhHgPGMba0kzdG2cEiA0NnDRJUzU6TIpOlfzDVO+C0+xzsMSec2xHESqqkVurUchQTmEo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1637646316; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=Bk2AoUBJuDeEluylRDpZf9HAezfWsRaeqoaRTfx6WcY=; b=N8ridtB9VIBFfFjCmb6xRPkmjn9SYtWulA6vB7HNo3WWpua4iMCzO+c4JiLNfxJgq0UecwwAzuf3+uHPPfhd9YMhnYgnVsOG3DG4IWw2mM9B9uL/SE56AnL5naiConzgcpKNva9nQCNwqjJxKt3oEF99CSMZvqJSNI/Atx8eeh4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83926+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1637646316168966.0310055129837; Mon, 22 Nov 2021 21:45:16 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id HzqCYY1788612x2dmMhjIkWU; Mon, 22 Nov 2021 21:45:15 -0800 X-Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mx.groups.io with SMTP id smtpd.web10.7698.1637646312279977943 for ; Mon, 22 Nov 2021 21:45:12 -0800 X-IronPort-AV: E=McAfee;i="6200,9189,10176"; a="234903539" X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="234903539" X-Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:10 -0800 X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="674336719" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.212.191.131]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:10 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao , Michael Kubacki Subject: [edk2-devel] [Patch 2/3] .pytool/Plugin/EccCheck: Remove temp directory on exception Date: Mon, 22 Nov 2021 21:44:54 -0800 Message-Id: <20211123054455.600-3-michael.d.kinney@intel.com> In-Reply-To: <20211123054455.600-1-michael.d.kinney@intel.com> References: <20211123054455.600-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: J0q147pzaaCI41X0QcSGUO1Nx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1637646315; bh=7/ijlqJ/nd6/ByMnWwsEG4jprFYW/joV6P656w67x7E=; h=Cc:Date:From:Reply-To:Subject:To; b=hmI+6zsHiKfzTv0MWD5sVajYhCAfpgUxzK8uDUQhprdwi8/16xwvqVy+VyhyN1hIMye J6/93Eea1jTks7fqVML4deGnbw7KnrfnvRuZsKrnnOhjLdYmz655bqKP/zlq3SsSzeiTX jBD8jgbV/NzEtatrP+GXMxvwEZ/f9aFL3qQ= X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1637646318143100001 Content-Type: text/plain; charset="utf-8" REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2986 Add try/except to RunBuildPlugin() to remove temporary directory if a KeyboardInterrupt exception or an unexpected exception is detected. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Cc: Michael Kubacki Signed-off-by: Michael D Kinney --- .pytool/Plugin/EccCheck/EccCheck.py | 84 +++++++++++++++++------------ 1 file changed, 50 insertions(+), 34 deletions(-) diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/= EccCheck.py index 82a3f14d6d5c..25583f15bf54 100644 --- a/.pytool/Plugin/EccCheck/EccCheck.py +++ b/.pytool/Plugin/EccCheck/EccCheck.py @@ -72,48 +72,64 @@ class EccCheck(ICiBuildPlugin): =20 # Create temp directory temp_path =3D os.path.join(workspace_path, 'Build', 'ecctemp') - # Delete temp directory - if os.path.exists(temp_path): - shutil.rmtree(temp_path) - # Copy package being scanned to temp_path - shutil.copytree ( - os.path.join(workspace_path, packagename), - os.path.join(temp_path, packagename), - symlinks=3DTrue - ) - # Copy exception.xml to temp_path - shutil.copyfile ( - os.path.join(basetools_path, "Source", "Python", "Ecc", "excepti= on.xml"), - os.path.join(temp_path, "exception.xml") - ) + try: + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) + # Copy package being scanned to temp_path + shutil.copytree ( + os.path.join(workspace_path, packagename), + os.path.join(temp_path, packagename), + symlinks=3DTrue + ) + # Copy exception.xml to temp_path + shutil.copyfile ( + os.path.join(basetools_path, "Source", "Python", "Ecc", "exc= eption.xml"), + os.path.join(temp_path, "exception.xml") + ) =20 - self.ApplyConfig(pkgconfig, temp_path, packagename) - modify_dir_list =3D self.GetModifyDir(packagename) - patch =3D self.GetDiff(packagename) - ecc_diff_range =3D self.GetDiffRange(patch, packagename, temp_path) - # - # Set workingdir to Build output directory because Ecc generates t= emp files - # Can not set workingdir to temp_path because that can be on a dif= ferent - # drive letter for some CI platforms and RunCmd() does not work co= rrectly - # if working dir is on a different drive letter. - # - self.GenerateEccReport(modify_dir_list, ecc_diff_range, temp_path,= basetools_path, os.path.join (workspace_path, 'Build')) - ecc_log =3D os.path.join(temp_path, "Ecc.log") - if self.ECC_PASS: + self.ApplyConfig(pkgconfig, temp_path, packagename) + modify_dir_list =3D self.GetModifyDir(packagename) + patch =3D self.GetDiff(packagename) + ecc_diff_range =3D self.GetDiffRange(patch, packagename, temp_= path) + # + # Set workingdir to Build output directory because Ecc generat= es temp files + # Can not set workingdir to temp_path because that can be on a= different + # drive letter for some CI platforms and RunCmd() does not wor= k correctly + # if working dir is on a different drive letter. + # + self.GenerateEccReport(modify_dir_list, ecc_diff_range, temp_p= ath, basetools_path, os.path.join (workspace_path, 'Build')) + ecc_log =3D os.path.join(temp_path, "Ecc.log") + if self.ECC_PASS: + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) + tc.SetSuccess() + return 0 + else: + with open(ecc_log, encoding=3D'utf8') as output: + ecc_output =3D output.readlines() + for line in ecc_output: + logging.error(line.strip()) + # Delete temp directory + if os.path.exists(temp_path): + shutil.rmtree(temp_path) + tc.SetFailed("EccCheck failed for {0}".format(packagename)= , "CHECK FAILED") + return 1 + except KeyboardInterrupt: + # If EccCheck is interrupted by keybard interrupt, then return= failure # Delete temp directory if os.path.exists(temp_path): shutil.rmtree(temp_path) - tc.SetSuccess() - return 0 + tc.SetFailed("EccCheck interrupted for {0}".format(packagename= ), "CHECK FAILED") + return 1 else: - with open(ecc_log, encoding=3D'utf8') as output: - ecc_output =3D output.readlines() - for line in ecc_output: - logging.error(line.strip()) + # If EccCheck fails for any other exception type, raise the ex= ception # Delete temp directory if os.path.exists(temp_path): shutil.rmtree(temp_path) - tc.SetFailed("EccCheck failed for {0}".format(packagename), "C= HECK FAILED") + tc.SetFailed("EccCheck exception for {0}".format(packagename),= "CHECK FAILED") + raise return 1 =20 def GetDiff(self, pkg: str) -> List[str]: --=20 2.32.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#83926): https://edk2.groups.io/g/devel/message/83926 Mute This Topic: https://groups.io/mt/87253500/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Thu Apr 25 14:43:28 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+83927+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83927+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1637646318; cv=none; d=zohomail.com; s=zohoarc; b=idqfDFl7042qwGsGSPpnLq+CMrBk4k+XzAmIfx+NO+eNssIgwKwpTvsbX6MIJPMvSCG3Be3+TxfsPIzq0F6b9FCqv1h7ONHYVTaOv6ssCuBubnk+FYrL798uReqjxoghnE9Czr9WfW7NsP1ptwsjDGLx/j7KX6T5oQaK3rPZv4Y= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1637646318; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=RQpY6FnjdoF4WjDjW948DutG9hLrT8eHBoytPzUh/2Q=; b=n9Lil9u/UZCzTAnhY6+kdAQb6GMojM1wCCcpzNXHGkUFBHw5CEK4MbdeVzgmA4nlDt3N247sKXmSdm6w5jfsd4o9zMRr8hE32Zji229tnKRd+HX7hYQeFUNVf/4qXQQ4ZhY1t4Mt408MSv+a50nSzzSu81hSJwlmHxgFesA9wo8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83927+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1637646318473731.8754415269082; Mon, 22 Nov 2021 21:45:18 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id 0vQ7YY1788612xpXJtLFkZBd; Mon, 22 Nov 2021 21:45:16 -0800 X-Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mx.groups.io with SMTP id smtpd.web09.7744.1637646311113360089 for ; Mon, 22 Nov 2021 21:45:12 -0800 X-IronPort-AV: E=McAfee;i="6200,9189,10176"; a="234903541" X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="234903541" X-Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:10 -0800 X-IronPort-AV: E=Sophos;i="5.87,256,1631602800"; d="scan'208";a="674336721" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.212.191.131]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Nov 2021 21:45:10 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao , Michael Kubacki Subject: [edk2-devel] [Patch 3/3] .pytool/Plugin/EccCheck: Add performance optimizations Date: Mon, 22 Nov 2021 21:44:55 -0800 Message-Id: <20211123054455.600-4-michael.d.kinney@intel.com> In-Reply-To: <20211123054455.600-1-michael.d.kinney@intel.com> References: <20211123054455.600-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: 2BDBUW9wFSqn1sSBPHvHXMm4x1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1637646316; bh=th1m0CWHi+gkm8cN41Or9/a3KYb7maUZzAi6ZWNH6gw=; h=Cc:Date:From:Reply-To:Subject:To; b=U4mD+X4kqGDRFkThR2DnaXg+FoJ9WQTdPpJuj6/M9I85WSHs8uN8pS3T3rM85xLWZtw Gp3EhOgoUVLrdrmu3v1G14v+6fbJDwdQCsJgPYOmPmCCeFGX0FQ26NN+EeMcgvlDvmoxX Qo6d+NqflSgIOnIXT1qmzADfZWYfqI5K5ug= X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1637646320553100001 Content-Type: text/plain; charset="utf-8" REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2986 Improve the performance of EccCheck by using a temp file instead of stdout to capture the results of the git diff commands. If a large patch set is passed into EccCheck, using stdout could be slow and also added the large diff content to the build log that is redundant information. A second performance improvement is to filter the modified directories to remove duplicate directories. Complex libraries and modules that have subdirectories with sources would be scanned twice if there were source changes in both the main directory and subdirectories. Filter out the subdirectories from the modified directory list when this case is detected. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Cc: Michael Kubacki Signed-off-by: Michael D Kinney --- .pytool/Plugin/EccCheck/EccCheck.py | 121 +++++++++++++++++++++------- 1 file changed, 94 insertions(+), 27 deletions(-) diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/= EccCheck.py index 25583f15bf54..908655dab667 100644 --- a/.pytool/Plugin/EccCheck/EccCheck.py +++ b/.pytool/Plugin/EccCheck/EccCheck.py @@ -30,7 +30,6 @@ class EccCheck(ICiBuildPlugin): }, """ =20 - ReModifyFile =3D re.compile(r'[B-Q,S-Z]+[\d]*\t(.*)') FindModifyFile =3D re.compile(r'\+\+\+ b\/(.*)') LineScopePattern =3D (r'@@ -\d*\,*\d* \+\d*\,*\d* @@.*') LineNumRange =3D re.compile(r'@@ -\d*\,*\d* \+(\d*)\,*(\d*) @@.*') @@ -87,10 +86,12 @@ class EccCheck(ICiBuildPlugin): os.path.join(basetools_path, "Source", "Python", "Ecc", "exc= eption.xml"), os.path.join(temp_path, "exception.xml") ) + # Output file to use for git diff operations + temp_diff_output =3D os.path.join (temp_path, 'diff.txt') =20 self.ApplyConfig(pkgconfig, temp_path, packagename) - modify_dir_list =3D self.GetModifyDir(packagename) - patch =3D self.GetDiff(packagename) + modify_dir_list =3D self.GetModifyDir(packagename, temp_diff_o= utput) + patch =3D self.GetDiff(packagename, temp_diff_output) ecc_diff_range =3D self.GetDiffRange(patch, packagename, temp_= path) # # Set workingdir to Build output directory because Ecc generat= es temp files @@ -132,37 +133,103 @@ class EccCheck(ICiBuildPlugin): raise return 1 =20 - def GetDiff(self, pkg: str) -> List[str]: - return_buffer =3D StringIO() - params =3D "diff --unified=3D0 origin/master HEAD" - RunCmd("git", params, outstream=3Dreturn_buffer) - p =3D return_buffer.getvalue().strip() - patch =3D p.split("\n") - return_buffer.close() - + def GetDiff(self, pkg: str, temp_diff_output: str) -> List[str]: + patch =3D [] + # + # Generate unified diff between origin/master and HEAD. + # + params =3D "diff --output=3D{} --unified=3D0 origin/master HEAD".f= ormat(temp_diff_output) + RunCmd("git", params) + with open(temp_diff_output) as file: + patch =3D file.read().strip().split('\n') return patch =20 - def GetModifyDir(self, pkg: str) -> List[str]: - return_buffer =3D StringIO() - params =3D "diff --name-status" + ' HEAD' + ' origin/master' - RunCmd("git", params, outstream=3Dreturn_buffer) - p1 =3D return_buffer.getvalue().strip() - dir_list =3D p1.split("\n") - return_buffer.close() + def GetModifyDir(self, pkg: str, temp_diff_output: str) -> List[str]: + # + # Generate diff between origin/master and HEAD using --diff-filter= to + # exclude deleted and renamed files that do not need to be scanned= by + # ECC. Also use --name-status to only generate the names of the f= iles + # with differences. The output format of this git diff command is= a + # list of files with the change status and the filename. The file= name + # is always at the end of the line. Examples: + # + # M MdeModulePkg/Application/CapsuleApp/CapsuleApp.h + # M MdeModulePkg/Application/UiApp/FrontPage.h + # + params =3D "diff --output=3D{} --diff-filter=3Ddr --name-status or= igin/master HEAD".format(temp_diff_output) + RunCmd("git", params) + dir_list =3D [] + with open(temp_diff_output) as file: + dir_list =3D file.read().strip().split('\n') + modify_dir_list =3D [] for modify_dir in dir_list: - file_path =3D self.ReModifyFile.findall(modify_dir) - if file_path: - file_dir =3D os.path.dirname(file_path[0]) - else: + # + # Parse file name from the end of the line + # + file_path =3D modify_dir.strip().split() + # + # Skip lines that do not have at least 2 elements (status and = file name) + # + if len(file_path) < 2: continue - if pkg in file_dir and file_dir !=3D pkg: - modify_dir_list.append('%s' % file_dir) - else: + # + # Parse the directory name from the file name + # + file_dir =3D os.path.dirname(file_path[-1]) + # + # Skip directory names that do not start with the package bein= g scanned. + # + if file_dir.split('/')[0] !=3D pkg: continue + # + # Skip directory names that are identical to the package being= scanned. + # The assumption here is that there are no source files at the= package + # root. Instead, the only expected files in the package root = are + # EDK II meta data files (DEC, DSC, FDF). + # + if file_dir =3D=3D pkg: + continue + # + # Skip directory names that are already in the modified dir li= st + # + if file_dir in modify_dir_list: + continue + # + # Add the candidate directory to scan to the modified dir list + # + modify_dir_list.append(file_dir) =20 - modify_dir_list =3D list(set(modify_dir_list)) - return modify_dir_list + # + # Remove duplicates from modify_dir_list + # Given a folder path, ECC performs a recursive scan of that folde= r. + # If a parent and child folder are both present in modify_dir_list, + # then ECC will perform redudanct scans of source files. In order + # to prevent redundant scans, if a parent and child folder are both + # present, then remove all the child folders. + # + # For example, if modified_dir_list contains the following element= s: + # MdeModulePkg/Core/Dxe + # MdeModulePkg/Core/Dxe/Hand + # MdeModulePkg/Core/Dxe/Mem + # + # Then MdeModulePkg/Core/Dxe/Hand and MdeModulePkg/Core/Dxe/Mem sh= ould + # be removed because the files in those folders are covered by a s= can + # of MdeModulePkg/Core/Dxe. + # + filtered_list =3D [] + for dir1 in modify_dir_list: + Append =3D True + for dir2 in modify_dir_list: + if dir1 =3D=3D dir2: + continue + common =3D os.path.commonpath([dir1, dir2]) + if os.path.normpath(common) =3D=3D os.path.normpath(dir2): + Append =3D False + break + if Append and dir1 not in filtered_list: + filtered_list.append(dir1) + return filtered_list =20 def GetDiffRange(self, patch_diff: List[str], pkg: str, temp_path: str= ) -> Dict[str, List[Tuple[int, int]]]: IsDelete =3D True --=20 2.32.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#83927): https://edk2.groups.io/g/devel/message/83927 Mute This Topic: https://groups.io/mt/87253501/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-