From nobody Sun Feb 8 19:37:56 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+83954+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83954+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1637685083; cv=none; d=zohomail.com; s=zohoarc; b=WTlpq+hIierkJpIxedk7eETpZwZwiZMaGdCYt/atClMC52etxgdKxYNcvH3puiqkxh6Ck6AV+xoJKNBRUrVboK6sdYg1rV2yFKjooIemNmKOEqALYjsdQrHJiBNXRyLCozVwduLdTILQQCt0rVtWd81cDVVedXds9Jqbj1sb7zI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1637685083; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=vUMOCoQ5q0TQU+Br6UrsXrfNnIt7Qz1S9sE22ytHFho=; b=BqzRXzBTD9G+GFF3+GikzgiaLxONb7s2IyLl7q/ur5ZzKZsMnHTjyNmEC1kWLaVEufXsIKKSknORcSQwgdUHkPBB93CQS2PFNQP+f3qzCSDD8Es5A4rl3KxgsgdtfIbJMFfv7jwYaLd78JtEj2AkQkRd3yinkI1FnSZPEB2hRHc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+83954+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 163768508354038.43055454393209; Tue, 23 Nov 2021 08:31:23 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id kqSXYY1788612xKD5vSpui5l; Tue, 23 Nov 2021 08:31:23 -0800 X-Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mx.groups.io with SMTP id smtpd.web12.13776.1637685074761235619 for ; Tue, 23 Nov 2021 08:31:15 -0800 X-IronPort-AV: E=McAfee;i="6200,9189,10176"; a="298466170" X-IronPort-AV: E=Sophos;i="5.87,258,1631602800"; d="scan'208";a="298466170" X-Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2021 08:31:14 -0800 X-IronPort-AV: E=Sophos;i="5.87,258,1631602800"; d="scan'208";a="571112444" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.209.59.198]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 23 Nov 2021 08:31:14 -0800 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao , Michael Kubacki Subject: [edk2-devel] [Patch V2 3/3] .pytool/Plugin/EccCheck: Add performance optimizations Date: Tue, 23 Nov 2021 08:31:01 -0800 Message-Id: <20211123163101.786-4-michael.d.kinney@intel.com> In-Reply-To: <20211123163101.786-1-michael.d.kinney@intel.com> References: <20211123163101.786-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: 4EhJdzwQvQiGK9utHnmsRJsHx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1637685083; bh=1loRVaYGEPeot1jTLNWZKOm37SD28N2Egd50ZzaMvMM=; h=Cc:Date:From:Reply-To:Subject:To; b=JPAUeaakGNe85D9+SOHwOH6+BcSnMmuEeG2ibhxVpvaKco2BD9A6bJcTX0S65J5e1Ev 4KSLQrOvtrBogq/EKDdLPhpt23/dusE9sV3Z92Ddx+dWp0cyOTKm/+wSerpRhRIsPsDB5 eN4PUP+ZTMqBVBrbC9/Jst3LAwOfRhsYt9c= X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1637685085239100005 Content-Type: text/plain; charset="utf-8" REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2986 Improve the performance of EccCheck by using a temp file instead of stdout to capture the results of the git diff commands. If a large patch set is passed into EccCheck, using stdout could be slow and also added the large diff content to the build log that is redundant information. A second performance improvement is to filter the modified directories to remove duplicate directories. Complex libraries and modules that have subdirectories with sources would be scanned twice if there were source changes in both the main directory and subdirectories. Filter out the subdirectories from the modified directory list when this case is detected. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Cc: Michael Kubacki Signed-off-by: Michael D Kinney --- .pytool/Plugin/EccCheck/EccCheck.py | 121 +++++++++++++++++++++------- 1 file changed, 94 insertions(+), 27 deletions(-) diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/= EccCheck.py index de766d984f7c..4fbc67765fdf 100644 --- a/.pytool/Plugin/EccCheck/EccCheck.py +++ b/.pytool/Plugin/EccCheck/EccCheck.py @@ -30,7 +30,6 @@ class EccCheck(ICiBuildPlugin): }, """ =20 - ReModifyFile =3D re.compile(r'[B-Q,S-Z]+[\d]*\t(.*)') FindModifyFile =3D re.compile(r'\+\+\+ b\/(.*)') LineScopePattern =3D (r'@@ -\d*\,*\d* \+\d*\,*\d* @@.*') LineNumRange =3D re.compile(r'@@ -\d*\,*\d* \+(\d*)\,*(\d*) @@.*') @@ -87,10 +86,12 @@ class EccCheck(ICiBuildPlugin): os.path.join(basetools_path, "Source", "Python", "Ecc", "exc= eption.xml"), os.path.join(temp_path, "exception.xml") ) + # Output file to use for git diff operations + temp_diff_output =3D os.path.join (temp_path, 'diff.txt') =20 self.ApplyConfig(pkgconfig, temp_path, packagename) - modify_dir_list =3D self.GetModifyDir(packagename) - patch =3D self.GetDiff(packagename) + modify_dir_list =3D self.GetModifyDir(packagename, temp_diff_o= utput) + patch =3D self.GetDiff(packagename, temp_diff_output) ecc_diff_range =3D self.GetDiffRange(patch, packagename, temp_= path) # # Use temp_path as working directory when running ECC tool @@ -129,37 +130,103 @@ class EccCheck(ICiBuildPlugin): raise return 1 =20 - def GetDiff(self, pkg: str) -> List[str]: - return_buffer =3D StringIO() - params =3D "diff --unified=3D0 origin/master HEAD" - RunCmd("git", params, outstream=3Dreturn_buffer) - p =3D return_buffer.getvalue().strip() - patch =3D p.split("\n") - return_buffer.close() - + def GetDiff(self, pkg: str, temp_diff_output: str) -> List[str]: + patch =3D [] + # + # Generate unified diff between origin/master and HEAD. + # + params =3D "diff --output=3D{} --unified=3D0 origin/master HEAD".f= ormat(temp_diff_output) + RunCmd("git", params) + with open(temp_diff_output) as file: + patch =3D file.read().strip().split('\n') return patch =20 - def GetModifyDir(self, pkg: str) -> List[str]: - return_buffer =3D StringIO() - params =3D "diff --name-status" + ' HEAD' + ' origin/master' - RunCmd("git", params, outstream=3Dreturn_buffer) - p1 =3D return_buffer.getvalue().strip() - dir_list =3D p1.split("\n") - return_buffer.close() + def GetModifyDir(self, pkg: str, temp_diff_output: str) -> List[str]: + # + # Generate diff between origin/master and HEAD using --diff-filter= to + # exclude deleted and renamed files that do not need to be scanned= by + # ECC. Also use --name-status to only generate the names of the f= iles + # with differences. The output format of this git diff command is= a + # list of files with the change status and the filename. The file= name + # is always at the end of the line. Examples: + # + # M MdeModulePkg/Application/CapsuleApp/CapsuleApp.h + # M MdeModulePkg/Application/UiApp/FrontPage.h + # + params =3D "diff --output=3D{} --diff-filter=3Ddr --name-status or= igin/master HEAD".format(temp_diff_output) + RunCmd("git", params) + dir_list =3D [] + with open(temp_diff_output) as file: + dir_list =3D file.read().strip().split('\n') + modify_dir_list =3D [] for modify_dir in dir_list: - file_path =3D self.ReModifyFile.findall(modify_dir) - if file_path: - file_dir =3D os.path.dirname(file_path[0]) - else: + # + # Parse file name from the end of the line + # + file_path =3D modify_dir.strip().split() + # + # Skip lines that do not have at least 2 elements (status and = file name) + # + if len(file_path) < 2: continue - if pkg in file_dir and file_dir !=3D pkg: - modify_dir_list.append('%s' % file_dir) - else: + # + # Parse the directory name from the file name + # + file_dir =3D os.path.dirname(file_path[-1]) + # + # Skip directory names that do not start with the package bein= g scanned. + # + if file_dir.split('/')[0] !=3D pkg: continue + # + # Skip directory names that are identical to the package being= scanned. + # The assumption here is that there are no source files at the= package + # root. Instead, the only expected files in the package root = are + # EDK II meta data files (DEC, DSC, FDF). + # + if file_dir =3D=3D pkg: + continue + # + # Skip directory names that are already in the modified dir li= st + # + if file_dir in modify_dir_list: + continue + # + # Add the candidate directory to scan to the modified dir list + # + modify_dir_list.append(file_dir) =20 - modify_dir_list =3D list(set(modify_dir_list)) - return modify_dir_list + # + # Remove duplicates from modify_dir_list + # Given a folder path, ECC performs a recursive scan of that folde= r. + # If a parent and child folder are both present in modify_dir_list, + # then ECC will perform redudanct scans of source files. In order + # to prevent redundant scans, if a parent and child folder are both + # present, then remove all the child folders. + # + # For example, if modified_dir_list contains the following element= s: + # MdeModulePkg/Core/Dxe + # MdeModulePkg/Core/Dxe/Hand + # MdeModulePkg/Core/Dxe/Mem + # + # Then MdeModulePkg/Core/Dxe/Hand and MdeModulePkg/Core/Dxe/Mem sh= ould + # be removed because the files in those folders are covered by a s= can + # of MdeModulePkg/Core/Dxe. + # + filtered_list =3D [] + for dir1 in modify_dir_list: + Append =3D True + for dir2 in modify_dir_list: + if dir1 =3D=3D dir2: + continue + common =3D os.path.commonpath([dir1, dir2]) + if os.path.normpath(common) =3D=3D os.path.normpath(dir2): + Append =3D False + break + if Append and dir1 not in filtered_list: + filtered_list.append(dir1) + return filtered_list =20 def GetDiffRange(self, patch_diff: List[str], pkg: str, temp_path: str= ) -> Dict[str, List[Tuple[int, int]]]: IsDelete =3D True --=20 2.32.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#83954): https://edk2.groups.io/g/devel/message/83954 Mute This Topic: https://groups.io/mt/87262301/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-