From nobody Mon Apr 29 11:09:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+41641+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+41641+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1559147217; cv=none; d=zoho.com; s=zohoarc; b=dMd9Jo6cCXBt6a+lcq7YrgGj7hmNZ3Uhd26aK9AiaVqLWnofzA/Y/a/wK6S+O+AKjD053I1V7bzZhRQLyADlK3hyMwfIFalYzmyrBhphpfKM2XPSfZF2ZiK+0cpJD5D/12wHotUMQli8E31VAm51Sm45o7Yc5nXV37THZD44mXM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559147217; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=3gHzHQJWkUWtWisgo0S//9HKIYZl+DIQ/SwgC4NGjtg=; b=P2p1P8LEzt5iqcS8qq+ZgfmHJ7Qoq7v7pFhYtJWox24yg1Df9mKj/+2nZToaessGnvznfETAESBVZ1kojdYmkqCInh1rj/ExG7pX/RpVhweXRsL1R29t+Hp7MEXs+C6BEktKcr6s7K/2Qr8Mxtg4RiTOR/otXaUkSj/xUtryL3Q= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+41641+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1559147217125790.257500399441; Wed, 29 May 2019 09:26:57 -0700 (PDT) Return-Path: X-Received: from mga17.intel.com (mga17.intel.com []) by groups.io with SMTP; Wed, 29 May 2019 09:26:56 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 May 2019 09:26:56 -0700 X-ExtLoop1: 1 X-Received: from rodrigu3-desk.amr.corp.intel.com ([10.7.163.75]) by orsmga001.jf.intel.com with ESMTP; 29 May 2019 09:26:55 -0700 From: "Christian Rodriguez" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Yonghong Zhu Subject: [edk2-devel] [Patch V4 1/2] BaseTools: Add a checking for Sources section in INF file Date: Wed, 29 May 2019 09:26:48 -0700 Message-Id: <20190529162649.3236-2-christian.rodriguez@intel.com> In-Reply-To: <20190529162649.3236-1-christian.rodriguez@intel.com> References: <20190529162649.3236-1-christian.rodriguez@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,christian.rodriguez@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1559147216; bh=2zNrJD1AncAsRrwrGU3CkqW/1vgDVOZZPTIAD0bJhwo=; h=Cc:Date:From:Reply-To:Subject:To; b=FoBxcy0xmbt/u7IOhHzmUmyTAmrPlYeAX+i2rz2gPrDT/ZX0rhLdh3EZiHGoMzXgytA ZvGYam0GJlEf/8fjevEizfndtDGftEbBEiMx0v881Qdj9o7W6cPvWrA+5Wt67v0GKpVp/ lRExx5bJkIzvdVSlHFQ74UYkTc7Q9ivbf3A= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1804 Add a check to see if [Sources] section lists all the header type files of a module. Performance impact should be minimal with this patch since information is already being fetched for Makefile purposes. All other information is already cached in memory. No extra IO time is needed. Signed-off-by: Christian Rodriguez Cc: Bob Feng Cc: Liming Gao Cc: Yonghong Zhu Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/GenMake.py | 38 ++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 0e0f9fd9b0..5c992d7c26 100644 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -905,6 +905,44 @@ cleanlib: ForceIncludedFile, self._AutoGenObject.IncludePathList + = self._AutoGenObject.BuildOptionIncPathList ) + + # Check if header files are listed in metafile + # Get a list of unique module header source files from MetaFile + headerFilesInMetaFileSet =3D set() + for aFile in self._AutoGenObject.SourceFileList: + aFileName =3D str(aFile) + if not aFileName.endswith('.h'): + continue + headerFilesInMetaFileSet.add(aFileName.lower()) + + # Get a list of unique module autogen files + localAutoGenFileSet =3D set() + for aFile in self._AutoGenObject.AutoGenFileList: + localAutoGenFileSet.add(str(aFile).lower()) + + # Get a list of unique module dependency header files + # Exclude autogen files and files not in the source directory + headerFileDependencySet =3D set() + localSourceDir =3D str(self._AutoGenObject.SourceDir).lower() + for Dependency in FileDependencyDict.values(): + for aFile in Dependency: + aFileName =3D str(aFile).lower() + if not aFileName.endswith('.h'): + continue + if aFileName in localAutoGenFileSet: + continue + if localSourceDir not in aFileName: + continue + headerFileDependencySet.add(aFileName) + + # Check if a module dependency header file is missing from the mod= ule's MetaFile + for aFile in headerFileDependencySet: + if aFile in headerFilesInMetaFileSet: + continue + EdkLogger.warn("build","Module MetaFile [Sources] is missing l= ocal header!", + ExtraData =3D "Local Header: " + aFile + " not fou= nd in " + self._AutoGenObject.MetaFile.Path + ) + DepSet =3D None for File,Dependency in FileDependencyDict.items(): if not Dependency: --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#41641): https://edk2.groups.io/g/devel/message/41641 Mute This Topic: https://groups.io/mt/31835555/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Mon Apr 29 11:09:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+41642+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+41642+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1559147218; cv=none; d=zoho.com; s=zohoarc; b=Vg05qxKl+A1Nn7F8bU9FTP4L/+br18Qpb17xE2Gtv7RHLTnbmPgu4JP1P6jXXntgdO3QWdB6L534t6WKLbQy2BahzOiuO7OtWYxPzmimqf0PJIVikolw2B8veOmoSN47ztb+4LEArwCI5M26VyPD4vG9HKIifkd0PIhrBSCVYzk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1559147218; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=Fesq3JRSXB3TqSNsNe6pWuHzK8BLlJHd+F+/r4fDjlg=; b=Np2MJojvNR9r0CpYQBh/WNLkjpP85suzOp/xx5kdO/Vwv+xpV/GWmdr+nheS/wBvDUz/RSPq+LmyvuVpcaKlyJMQ4rF7lEsf6GeRMj240d7FRmDWMtRlGySF8tMXQQr/2v9/E/dOi8NB1JUfbidmTOu+3D6cpeyq+gP9pjrKoFo= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+41642+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1559147218107650.5562261288674; Wed, 29 May 2019 09:26:58 -0700 (PDT) Return-Path: X-Received: from mga17.intel.com (mga17.intel.com []) by groups.io with SMTP; Wed, 29 May 2019 09:26:57 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 May 2019 09:26:57 -0700 X-ExtLoop1: 1 X-Received: from rodrigu3-desk.amr.corp.intel.com ([10.7.163.75]) by orsmga001.jf.intel.com with ESMTP; 29 May 2019 09:26:56 -0700 From: "Christian Rodriguez" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Yonghong Zhu Subject: [edk2-devel] [Patch V4 2/2] BaseTools: Refactor hash tracking after checking for Sources section Date: Wed, 29 May 2019 09:26:49 -0700 Message-Id: <20190529162649.3236-3-christian.rodriguez@intel.com> In-Reply-To: <20190529162649.3236-1-christian.rodriguez@intel.com> References: <20190529162649.3236-1-christian.rodriguez@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,christian.rodriguez@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1559147217; bh=bNTv6vMTW7WoQJHUSEKn/fDJjEtvA4ZlKcilUfbG/Xg=; h=Cc:Date:From:Reply-To:Subject:To; b=YKrXcAuimZU6EpTXhoJ4w6EdyuV6SgyCrVeqxR9oE2pskTHtBVmzTQZ4vGq56ilI7wB 4SeofVdOhw5L1/eSvNjH5EDOSEbzHfXAGVJ+iOOCXgL4h5cYEJHie83ujmhjr4zC0uIYl 22gDXNZlXO3CfialmC67v/OuQNS713JQuCA= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1804 After adding a check to see if [Sources] section lists all the header type files of a module, track module and library hashes for --hash feature. If above check is not in compilance for a library or module, force hash invalidation on that library or module. Signed-off-by: Christian Rodriguez Cc: Bob Feng Cc: Liming Gao Cc: Yonghong Zhu Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGen.py | 6 +- BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/Common/GlobalData.py | 3 +- BaseTools/Source/Python/build/build.py | 65 ++++++++++++-------- 4 files changed, 53 insertions(+), 27 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index a5bef4f7c6..a376bc24d6 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -3989,7 +3989,8 @@ class ModuleAutoGen(AutoGen): for LibraryAutoGen in self.LibraryAutoGenList: LibraryAutoGen.CreateMakeFile() =20 - if self.CanSkip(): + # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping + if not GlobalData.gUseHashCache and self.CanSkip(): return =20 if len(self.CustomMakefile) =3D=3D 0: @@ -4032,7 +4033,8 @@ class ModuleAutoGen(AutoGen): for LibraryAutoGen in self.LibraryAutoGenList: LibraryAutoGen.CreateCodeFile() =20 - if self.CanSkip(): + # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping + if not GlobalData.gUseHashCache and self.CanSkip(): return =20 AutoGenList =3D [] diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 5c992d7c26..212ca0fa7f 100644 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -935,10 +935,16 @@ cleanlib: continue headerFileDependencySet.add(aFileName) =20 + # Ensure that gModuleBuildTracking has been initialized per archit= ecture + if self._AutoGenObject.Arch not in GlobalData.gModuleBuildTracking: + GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch] =3D = dict() + # Check if a module dependency header file is missing from the mod= ule's MetaFile for aFile in headerFileDependencySet: if aFile in headerFilesInMetaFileSet: continue + if GlobalData.gUseHashCache: + GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch][= self._AutoGenObject] =3D 'FAIL_METAFILE' EdkLogger.warn("build","Module MetaFile [Sources] is missing l= ocal header!", ExtraData =3D "Local Header: " + aFile + " not fou= nd in " + self._AutoGenObject.MetaFile.Path ) diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Sourc= e/Python/Common/GlobalData.py index 95e28a988f..bd45a43728 100644 --- a/BaseTools/Source/Python/Common/GlobalData.py +++ b/BaseTools/Source/Python/Common/GlobalData.py @@ -110,7 +110,8 @@ gEnableGenfdsMultiThread =3D False gSikpAutoGenCache =3D set() =20 # Dictionary for tracking Module build status as success or failure -# False -> Fail : True -> Success +# Top Dict: Key: Arch Type Value: Dictionary +# Second Dict: Key: AutoGen Obj Value: 'SUCCESS'\'FAIL'\'FAIL_METAFILE' gModuleBuildTracking =3D dict() =20 # Dictionary of booleans that dictate whether a module or diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 80ceb98310..0855d4561c 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -625,8 +625,16 @@ class BuildTask: BuildTask._ErrorFlag.set() BuildTask._ErrorMessage =3D "%s broken\n %s [%s]" % \ (threading.currentThread().getName()= , Command, WorkingDir) - if self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking a= nd not BuildTask._ErrorFlag.isSet(): - GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] = =3D True + + # Set the value used by hash invalidation flow in GlobalData.gModu= leBuildTracking to 'SUCCESS' + # If Module or Lib is being tracked, it did not fail header check = test, and built successfully + if (self.BuildItem.BuildObject.Arch in GlobalData.gModuleBuildTrac= king and + self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking[s= elf.BuildItem.BuildObject.Arch] and + GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arch= ][self.BuildItem.BuildObject] !=3D 'FAIL_METAFILE' and + not BuildTask._ErrorFlag.isSet() + ): + GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arc= h][self.BuildItem.BuildObject] =3D 'SUCCESS' + # indicate there's a thread is available for another build task BuildTask._RunningQueueLock.acquire() BuildTask._RunningQueue.pop(self.BuildItem) @@ -1154,27 +1162,30 @@ class Build(): # # def invalidateHash(self): - # GlobalData.gModuleBuildTracking contains only modules that canno= t be skipped by hash - for moduleAutoGenObj in GlobalData.gModuleBuildTracking.keys(): - # False =3D=3D FAIL : True =3D=3D Success - # Skip invalidating for Successful module builds - if GlobalData.gModuleBuildTracking[moduleAutoGenObj] =3D=3D Tr= ue: - continue + # Only for hashing feature + if not GlobalData.gUseHashCache: + return + + # GlobalData.gModuleBuildTracking contains only modules or libs th= at cannot be skipped by hash + for moduleAutoGenObjArch in GlobalData.gModuleBuildTracking.keys(): + for moduleAutoGenObj in GlobalData.gModuleBuildTracking[module= AutoGenObjArch].keys(): + # Skip invalidating for Successful Module/Lib builds + if GlobalData.gModuleBuildTracking[moduleAutoGenObjArch][m= oduleAutoGenObj] =3D=3D 'SUCCESS': + continue =20 - # The module failed to build or failed to start building, from= this point on + # The module failed to build, failed to start building, or= failed the header check test from this point on =20 - # Remove .hash from build - if GlobalData.gUseHashCache: - ModuleHashFile =3D path.join(moduleAutoGenObj.BuildDir, mo= duleAutoGenObj.Name + ".hash") + # Remove .hash from build + ModuleHashFile =3D os.path.join(moduleAutoGenObj.BuildDir,= moduleAutoGenObj.Name + ".hash") if os.path.exists(ModuleHashFile): os.remove(ModuleHashFile) =20 - # Remove .hash file from cache - if GlobalData.gBinCacheDest: - FileDir =3D path.join(GlobalData.gBinCacheDest, moduleAuto= GenObj.Arch, moduleAutoGenObj.SourceDir, moduleAutoGenObj.MetaFile.BaseName) - HashFile =3D path.join(FileDir, moduleAutoGenObj.Name + '.= hash') - if os.path.exists(HashFile): - os.remove(HashFile) + # Remove .hash file from cache + if GlobalData.gBinCacheDest: + FileDir =3D os.path.join(GlobalData.gBinCacheDest, mod= uleAutoGenObj.Arch, moduleAutoGenObj.SourceDir, moduleAutoGenObj.MetaFile.B= aseName) + HashFile =3D os.path.join(FileDir, moduleAutoGenObj.Na= me + '.hash') + if os.path.exists(HashFile): + os.remove(HashFile) =20 ## Build a module or platform # @@ -1825,9 +1836,11 @@ class Build(): if self.Target =3D=3D "genmake": return True self.BuildModules.append(Ma) - # Initialize all modules in tracking to False = (FAIL) - if Ma not in GlobalData.gModuleBuildTracking: - GlobalData.gModuleBuildTracking[Ma] =3D Fa= lse + # Initialize all modules in tracking to 'FAIL' + if Ma.Arch not in GlobalData.gModuleBuildTrack= ing: + GlobalData.gModuleBuildTracking[Ma.Arch] = =3D dict() + if Ma not in GlobalData.gModuleBuildTracking[M= a.Arch]: + GlobalData.gModuleBuildTracking[Ma.Arch][M= a] =3D 'FAIL' self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) MakeStart =3D time.time() for Ma in self.BuildModules: @@ -1911,6 +1924,7 @@ class Build(): # Save MAP buffer into MAP file. # self._SaveMapFile (MapBuffer, Wa) + self.invalidateHash() =20 def _GenFfsCmd(self,ArchList): # convert dictionary of Cmd:(Inf,Arch) @@ -2009,9 +2023,11 @@ class Build(): if self.Target =3D=3D "genmake": continue self.BuildModules.append(Ma) - # Initialize all modules in tracking to False (FAI= L) - if Ma not in GlobalData.gModuleBuildTracking: - GlobalData.gModuleBuildTracking[Ma] =3D False + # Initialize all modules in tracking to 'FAIL' + if Ma.Arch not in GlobalData.gModuleBuildTracking: + GlobalData.gModuleBuildTracking[Ma.Arch] =3D d= ict() + if Ma not in GlobalData.gModuleBuildTracking[Ma.Ar= ch]: + GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = =3D 'FAIL' self.Progress.Stop("done!") self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) MakeStart =3D time.time() @@ -2099,6 +2115,7 @@ class Build(): # Save MAP buffer into MAP file. # self._SaveMapFile(MapBuffer, Wa) + self.invalidateHash() =20 ## Generate GuidedSectionTools.txt in the FV directories. # --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#41642): https://edk2.groups.io/g/devel/message/41642 Mute This Topic: https://groups.io/mt/31835556/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-