From nobody Mon Apr 29 01:49:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45735+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45735+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565880919; cv=none; d=zoho.com; s=zohoarc; b=Wu/YH0RQ0LTTgEMHmGpEKyX+8rJJOtEh0GJ9jMp4k3f7HUw/S8k3B78sZEvqAKuNiyb/K7LYUE/6YeoZKclu51A3b1Ub/N318R9MePH2edtyXilxphXLgQ37rf6ytVq6BHrkB/r3x75xIB1ZB0pLhUX8LGjn8eK0633/P4p2A3s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565880919; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=2dMHpQfNCfdJcPcIFiu7gl8yMaortWIuY3MmUu23+sk=; b=KXGklXtL3vHXYeYT6GgGn/7oYEAtMQx3GWrx+XnN4Bo6n6etXi+3p4hMf0n35jqY8l6wq7darFzSdgBeWcmoX5p+zCWqiBneBYZ7xn0seHgYw0rV2CAdZ7TCsKAblWzkHM0zyOsEuxGJZKFYGUvlBgvgzTCepKz0Jz8lshbmkIY= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45735+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 15658809195431001.8380925089082; Thu, 15 Aug 2019 07:55:19 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by groups.io with SMTP; Thu, 15 Aug 2019 07:28:39 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2019 07:26:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,389,1559545200"; d="scan'208";a="171125274" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.104]) by orsmga008.jf.intel.com with ESMTP; 15 Aug 2019 07:26:29 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v6 1/5] BaseTools: Improve the cache hit in the edk2 build cache Date: Thu, 15 Aug 2019 22:26:17 +0800 Message-Id: <20190815142621.7644-2-steven.shi@intel.com> In-Reply-To: <20190815142621.7644-1-steven.shi@intel.com> References: <20190815142621.7644-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565880919; bh=BtK9+rIbWg4r7U2MWW3G2aKU6aBxFC6p+9oD1q1at50=; h=Cc:Date:From:Reply-To:Subject:To; b=ZjuWt5O9XGNrWM5Udfmu+TqmIvFdmhpMXKHfaIrIB9vh//3cz94SeWxDE8U6DunG4q4 ic0wPZZj7lqk69TSmBjHrypFO6d4dxdQEi/xehmd6/SOjrniwR/8FzN3u/e4+idxe137u eb5WfbWyKaIRtizog4kwNLvJiCjML0yWFR4= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1927 Current cache hash algorithm does not parse and generate the makefile to get the accurate dependency files for a module. It instead use the platform and package meta files to get the module depenedency in a quick but over approximate way. These meta files are monolithic and involve many redundant dependency for the module, which cause the module build cache miss easily. This patch introduces one more cache checkpoint and a new hash algorithm besides the current quick one. The new hash algorithm leverages the module makefile to achieve more accurate and precise dependency info for a module. When the build cache miss with the first quick hash, the Basetool will caculate new one after makefile is generated and then check again. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 21 +++++++++++++++++++= ++ BaseTools/Source/Python/AutoGen/CacheIR.py | 28 +++++++++++++++++++= +++++++++ BaseTools/Source/Python/AutoGen/DataPipe.py | 8 ++++++++ BaseTools/Source/Python/AutoGen/GenMake.py | 227 +++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++-----------------------------------------------------= ---------------------------------------------------------- BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 639 +++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= ++++++++++++++++++++++++++++++++++++++-------------------------------------= -------------------- BaseTools/Source/Python/Common/GlobalData.py | 9 +++++++++ BaseTools/Source/Python/build/build.py | 129 +++++++++++++++++++= +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++++---------------------------- 7 files changed, 865 insertions(+), 196 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py old mode 100644 new mode 100755 index e583828741..a84ed46f2e --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -182,6 +182,12 @@ class AutoGenWorkerInProcess(mp.Process): GlobalData.gDisableIncludePathCheck =3D False GlobalData.gFdfParser =3D self.data_pipe.Get("FdfParser") GlobalData.gDatabasePath =3D self.data_pipe.Get("DatabasePath") + GlobalData.gBinCacheSource =3D self.data_pipe.Get("BinCacheSou= rce") + GlobalData.gBinCacheDest =3D self.data_pipe.Get("BinCacheDest") + GlobalData.gCacheIR =3D self.data_pipe.Get("CacheIR") + GlobalData.gEnableGenfdsMultiThread =3D self.data_pipe.Get("En= ableGenfdsMultiThread") + GlobalData.file_lock =3D self.file_lock + CommandTarget =3D self.data_pipe.Get("CommandTarget") pcd_from_build_option =3D [] for pcd_tuple in self.data_pipe.Get("BuildOptPcd"): pcd_id =3D ".".join((pcd_tuple[0],pcd_tuple[1])) @@ -193,10 +199,13 @@ class AutoGenWorkerInProcess(mp.Process): FfsCmd =3D self.data_pipe.Get("FfsCommand") if FfsCmd is None: FfsCmd =3D {} + GlobalData.FfsCmd =3D FfsCmd PlatformMetaFile =3D self.GetPlatformMetaFile(self.data_pipe.G= et("P_Info").get("ActivePlatform"), self.data_pipe.Get("P_Info").= get("WorkspaceDir")) libConstPcd =3D self.data_pipe.Get("LibConstPcd") Refes =3D self.data_pipe.Get("REFS") + GlobalData.libConstPcd =3D libConstPcd + GlobalData.Refes =3D Refes while True: if self.module_queue.empty(): break @@ -223,8 +232,20 @@ class AutoGenWorkerInProcess(mp.Process): Ma.ConstPcd =3D libConstPcd[(Ma.MetaFile.File,Ma.M= etaFile.Root,Ma.Arch,Ma.MetaFile.Path)] if (Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaF= ile.Path) in Refes: Ma.ReferenceModules =3D Refes[(Ma.MetaFile.File,Ma= .MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)] + if GlobalData.gBinCacheSource and CommandTarget in [None, = "", "all"]: + Ma.GenModuleFilesHash(GlobalData.gCacheIR) + Ma.GenPreMakefileHash(GlobalData.gCacheIR) + if Ma.CanSkipbyPreMakefileCache(GlobalData.gCacheIR): + continue + Ma.CreateCodeFile(False) Ma.CreateMakeFile(False,GenFfsList=3DFfsCmd.get((Ma.MetaFi= le.File, Ma.Arch),[])) + + if GlobalData.gBinCacheSource and CommandTarget in [None, = "", "all"]: + Ma.GenMakeHeaderFilesHash(GlobalData.gCacheIR) + Ma.GenMakeHash(GlobalData.gCacheIR) + if Ma.CanSkipbyMakeCache(GlobalData.gCacheIR): + continue except Empty: pass except: diff --git a/BaseTools/Source/Python/AutoGen/CacheIR.py b/BaseTools/Source/= Python/AutoGen/CacheIR.py new file mode 100755 index 0000000000..2d9ffe3f0b --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/CacheIR.py @@ -0,0 +1,28 @@ +## @file +# Build cache intermediate result and state +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +class ModuleBuildCacheIR(): + def __init__(self, Path, Arch): + self.ModulePath =3D Path + self.ModuleArch =3D Arch + self.ModuleFilesHashDigest =3D None + self.ModuleFilesHashHexDigest =3D None + self.ModuleFilesChain =3D [] + self.PreMakefileHashHexDigest =3D None + self.CreateCodeFileDone =3D False + self.CreateMakeFileDone =3D False + self.MakefilePath =3D None + self.AutoGenFileList =3D None + self.DependencyHeaderFileSet =3D None + self.MakeHeaderFilesHashChain =3D None + self.MakeHeaderFilesHashDigest =3D None + self.MakeHeaderFilesHashChain =3D [] + self.MakeHashDigest =3D None + self.MakeHashHexDigest =3D None + self.MakeHashChain =3D [] + self.PreMakeCacheHit =3D False + self.MakeCacheHit =3D False diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py old mode 100644 new mode 100755 index 2052084bdb..84e77c301a --- a/BaseTools/Source/Python/AutoGen/DataPipe.py +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -158,3 +158,11 @@ class MemoryDataPipe(DataPipe): self.DataContainer =3D {"FdfParser": True if GlobalData.gFdfParser= else False} =20 self.DataContainer =3D {"LogLevel": EdkLogger.GetLevel()} + + self.DataContainer =3D {"BinCacheSource":GlobalData.gBinCacheSourc= e} + + self.DataContainer =3D {"BinCacheDest":GlobalData.gBinCacheDest} + + self.DataContainer =3D {"CacheIR":GlobalData.gCacheIR} + + self.DataContainer =3D {"EnableGenfdsMultiThread":GlobalData.gEnab= leGenfdsMultiThread} \ No newline at end of file diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py old mode 100644 new mode 100755 index 499ef82aea..c11b423ae1 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -906,6 +906,11 @@ cleanlib: self._AutoGenObject.IncludePathList + = self._AutoGenObject.BuildOptionIncPathList ) =20 + self.DependencyHeaderFileSet =3D set() + if FileDependencyDict: + for Dependency in FileDependencyDict.values(): + self.DependencyHeaderFileSet.update(set(Dependency)) + # Get a set of unique package includes from MetaFile parentMetaFileIncludes =3D set() for aInclude in self._AutoGenObject.PackageIncludePathList: @@ -1115,7 +1120,7 @@ cleanlib: ## For creating makefile targets for dependent libraries def ProcessDependentLibrary(self): for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList: - if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.Ca= nSkipbyHash(): + if not LibraryAutoGen.IsBinaryModule: self.LibraryBuildDirectoryList.append(self.PlaceMacro(Libr= aryAutoGen.BuildDir, self.Macros)) =20 ## Return a list containing source file's dependencies @@ -1129,114 +1134,9 @@ cleanlib: def GetFileDependency(self, FileList, ForceInculeList, SearchPathList): Dependency =3D {} for F in FileList: - Dependency[F] =3D self.GetDependencyList(F, ForceInculeList, S= earchPathList) + Dependency[F] =3D GetDependencyList(self._AutoGenObject, self.= FileCache, F, ForceInculeList, SearchPathList) return Dependency =20 - ## Find dependencies for one source file - # - # By searching recursively "#include" directive in file, find out all= the - # files needed by given source file. The dependencies will be only se= arched - # in given search path list. - # - # @param File The source file - # @param ForceInculeList The list of files which will be includ= ed forcely - # @param SearchPathList The list of search path - # - # @retval list The list of files the given source fil= e depends on - # - def GetDependencyList(self, File, ForceList, SearchPathList): - EdkLogger.debug(EdkLogger.DEBUG_1, "Try to get dependency files fo= r %s" % File) - FileStack =3D [File] + ForceList - DependencySet =3D set() - - if self._AutoGenObject.Arch not in gDependencyDatabase: - gDependencyDatabase[self._AutoGenObject.Arch] =3D {} - DepDb =3D gDependencyDatabase[self._AutoGenObject.Arch] - - while len(FileStack) > 0: - F =3D FileStack.pop() - - FullPathDependList =3D [] - if F in self.FileCache: - for CacheFile in self.FileCache[F]: - FullPathDependList.append(CacheFile) - if CacheFile not in DependencySet: - FileStack.append(CacheFile) - DependencySet.update(FullPathDependList) - continue - - CurrentFileDependencyList =3D [] - if F in DepDb: - CurrentFileDependencyList =3D DepDb[F] - else: - try: - Fd =3D open(F.Path, 'rb') - FileContent =3D Fd.read() - Fd.close() - except BaseException as X: - EdkLogger.error("build", FILE_OPEN_FAILURE, ExtraData= =3DF.Path + "\n\t" + str(X)) - if len(FileContent) =3D=3D 0: - continue - try: - if FileContent[0] =3D=3D 0xff or FileContent[0] =3D=3D= 0xfe: - FileContent =3D FileContent.decode('utf-16') - else: - FileContent =3D FileContent.decode() - except: - # The file is not txt file. for example .mcb file - continue - IncludedFileList =3D gIncludePattern.findall(FileContent) - - for Inc in IncludedFileList: - Inc =3D Inc.strip() - # if there's macro used to reference header file, expa= nd it - HeaderList =3D gMacroPattern.findall(Inc) - if len(HeaderList) =3D=3D 1 and len(HeaderList[0]) =3D= =3D 2: - HeaderType =3D HeaderList[0][0] - HeaderKey =3D HeaderList[0][1] - if HeaderType in gIncludeMacroConversion: - Inc =3D gIncludeMacroConversion[HeaderType] % = {"HeaderKey" : HeaderKey} - else: - # not known macro used in #include, always bui= ld the file by - # returning a empty dependency - self.FileCache[File] =3D [] - return [] - Inc =3D os.path.normpath(Inc) - CurrentFileDependencyList.append(Inc) - DepDb[F] =3D CurrentFileDependencyList - - CurrentFilePath =3D F.Dir - PathList =3D [CurrentFilePath] + SearchPathList - for Inc in CurrentFileDependencyList: - for SearchPath in PathList: - FilePath =3D os.path.join(SearchPath, Inc) - if FilePath in gIsFileMap: - if not gIsFileMap[FilePath]: - continue - # If isfile is called too many times, the performance = is slow down. - elif not os.path.isfile(FilePath): - gIsFileMap[FilePath] =3D False - continue - else: - gIsFileMap[FilePath] =3D True - FilePath =3D PathClass(FilePath) - FullPathDependList.append(FilePath) - if FilePath not in DependencySet: - FileStack.append(FilePath) - break - else: - EdkLogger.debug(EdkLogger.DEBUG_9, "%s included by %s = was not found "\ - "in any given path:\n\t%s" % (Inc, F, = "\n\t".join(SearchPathList))) - - self.FileCache[F] =3D FullPathDependList - DependencySet.update(FullPathDependList) - - DependencySet.update(ForceList) - if File in DependencySet: - DependencySet.remove(File) - DependencyList =3D list(DependencySet) # remove duplicate ones - - return DependencyList =20 ## CustomMakefile class # @@ -1618,7 +1518,7 @@ cleanlib: def GetLibraryBuildDirectoryList(self): DirList =3D [] for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList: - if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.Ca= nSkipbyHash(): + if not LibraryAutoGen.IsBinaryModule: DirList.append(os.path.join(self._AutoGenObject.BuildDir, = LibraryAutoGen.BuildDir)) return DirList =20 @@ -1754,11 +1654,116 @@ class TopLevelMakefile(BuildFile): def GetLibraryBuildDirectoryList(self): DirList =3D [] for LibraryAutoGen in self._AutoGenObject.LibraryAutoGenList: - if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.Ca= nSkipbyHash(): + if not LibraryAutoGen.IsBinaryModule: DirList.append(os.path.join(self._AutoGenObject.BuildDir, = LibraryAutoGen.BuildDir)) return DirList =20 +## Find dependencies for one source file +# +# By searching recursively "#include" directive in file, find out all the +# files needed by given source file. The dependencies will be only search= ed +# in given search path list. +# +# @param File The source file +# @param ForceInculeList The list of files which will be included f= orcely +# @param SearchPathList The list of search path +# +# @retval list The list of files the given source file de= pends on +# +def GetDependencyList(AutoGenObject, FileCache, File, ForceList, SearchPat= hList): + EdkLogger.debug(EdkLogger.DEBUG_1, "Try to get dependency files for %s= " % File) + FileStack =3D [File] + ForceList + DependencySet =3D set() + + if AutoGenObject.Arch not in gDependencyDatabase: + gDependencyDatabase[AutoGenObject.Arch] =3D {} + DepDb =3D gDependencyDatabase[AutoGenObject.Arch] + + while len(FileStack) > 0: + F =3D FileStack.pop() + + FullPathDependList =3D [] + if F in FileCache: + for CacheFile in FileCache[F]: + FullPathDependList.append(CacheFile) + if CacheFile not in DependencySet: + FileStack.append(CacheFile) + DependencySet.update(FullPathDependList) + continue + + CurrentFileDependencyList =3D [] + if F in DepDb: + CurrentFileDependencyList =3D DepDb[F] + else: + try: + Fd =3D open(F.Path, 'rb') + FileContent =3D Fd.read() + Fd.close() + except BaseException as X: + EdkLogger.error("build", FILE_OPEN_FAILURE, ExtraData=3DF.= Path + "\n\t" + str(X)) + if len(FileContent) =3D=3D 0: + continue + try: + if FileContent[0] =3D=3D 0xff or FileContent[0] =3D=3D 0xf= e: + FileContent =3D FileContent.decode('utf-16') + else: + FileContent =3D FileContent.decode() + except: + # The file is not txt file. for example .mcb file + continue + IncludedFileList =3D gIncludePattern.findall(FileContent) + + for Inc in IncludedFileList: + Inc =3D Inc.strip() + # if there's macro used to reference header file, expand it + HeaderList =3D gMacroPattern.findall(Inc) + if len(HeaderList) =3D=3D 1 and len(HeaderList[0]) =3D=3D = 2: + HeaderType =3D HeaderList[0][0] + HeaderKey =3D HeaderList[0][1] + if HeaderType in gIncludeMacroConversion: + Inc =3D gIncludeMacroConversion[HeaderType] % {"He= aderKey" : HeaderKey} + else: + # not known macro used in #include, always build t= he file by + # returning a empty dependency + FileCache[File] =3D [] + return [] + Inc =3D os.path.normpath(Inc) + CurrentFileDependencyList.append(Inc) + DepDb[F] =3D CurrentFileDependencyList + + CurrentFilePath =3D F.Dir + PathList =3D [CurrentFilePath] + SearchPathList + for Inc in CurrentFileDependencyList: + for SearchPath in PathList: + FilePath =3D os.path.join(SearchPath, Inc) + if FilePath in gIsFileMap: + if not gIsFileMap[FilePath]: + continue + # If isfile is called too many times, the performance is s= low down. + elif not os.path.isfile(FilePath): + gIsFileMap[FilePath] =3D False + continue + else: + gIsFileMap[FilePath] =3D True + FilePath =3D PathClass(FilePath) + FullPathDependList.append(FilePath) + if FilePath not in DependencySet: + FileStack.append(FilePath) + break + else: + EdkLogger.debug(EdkLogger.DEBUG_9, "%s included by %s was = not found "\ + "in any given path:\n\t%s" % (Inc, F, "\n\= t".join(SearchPathList))) + + FileCache[F] =3D FullPathDependList + DependencySet.update(FullPathDependList) + + DependencySet.update(ForceList) + if File in DependencySet: + DependencySet.remove(File) + DependencyList =3D list(DependencySet) # remove duplicate ones + + return DependencyList + # This acts like the main() function for the script, unless it is 'import'= ed into another script. if __name__ =3D=3D '__main__': - pass - + pass \ No newline at end of file diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py old mode 100644 new mode 100755 index 9ecf5c2dbe..613b0d2fb8 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -26,6 +26,8 @@ from Workspace.MetaFileCommentParser import UsageList from .GenPcdDb import CreatePcdDatabaseCode from Common.caching import cached_class_function from AutoGen.ModuleAutoGenHelper import PlatformInfo,WorkSpaceInfo +from AutoGen.CacheIR import ModuleBuildCacheIR +import json =20 ## Mapping Makefile type gMakeTypeMap =3D {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"} @@ -252,6 +254,8 @@ class ModuleAutoGen(AutoGen): self.AutoGenDepSet =3D set() self.ReferenceModules =3D [] self.ConstPcd =3D {} + self.Makefile =3D None + self.FileDependCache =3D {} =20 def __init_platform_info__(self): pinfo =3D self.DataPipe.Get("P_Info") @@ -1608,12 +1612,37 @@ class ModuleAutoGen(AutoGen): =20 self.IsAsBuiltInfCreated =3D True =20 + def CacheCopyFile(self, OriginDir, CopyDir, File): + sub_dir =3D os.path.relpath(File, CopyDir) + destination_file =3D os.path.join(OriginDir, sub_dir) + destination_dir =3D os.path.dirname(destination_file) + CreateDirectory(destination_dir) + try: + CopyFileOnChange(File, destination_dir) + except: + EdkLogger.quiet("[cache warning]: fail to copy file:%s to fold= er:%s" % (File, destination_dir)) + return + def CopyModuleToCache(self): - FileDir =3D path.join(GlobalData.gBinCacheDest, self.PlatformInfo.= Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, s= elf.MetaFile.BaseName) + self.GenPreMakefileHash(GlobalData.gCacheIR) + if not (self.MetaFile.Path, self.Arch) in GlobalData.gCacheIR or \ + not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].PreMak= efileHashHexDigest: + EdkLogger.quiet("[cache warning]: Cannot generate PreMakefileH= ash for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return False + + self.GenMakeHash(GlobalData.gCacheIR) + if not (self.MetaFile.Path, self.Arch) in GlobalData.gCacheIR or \ + not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHa= shChain or \ + not GlobalData.gCacheIR[(self.MetaFile.Path, self.Arch)].MakeHa= shHexDigest: + EdkLogger.quiet("[cache warning]: Cannot generate MakeHashChai= n for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return False + + MakeHashStr =3D str(GlobalData.gCacheIR[(self.MetaFile.Path, self.= Arch)].MakeHashHexDigest) + FileDir =3D path.join(GlobalData.gBinCacheDest, self.PlatformInfo.= OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceD= ir, self.MetaFile.BaseName, MakeHashStr) + FfsDir =3D path.join(GlobalData.gBinCacheDest, self.PlatformInfo.O= utputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs",= self.Guid + self.Name, MakeHashStr) + CreateDirectory (FileDir) - HashFile =3D path.join(self.BuildDir, self.Name + '.hash') - if os.path.exists(HashFile): - CopyFileOnChange(HashFile, FileDir) + self.SaveHashChainFileToCache(GlobalData.gCacheIR) ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf') if os.path.exists(ModuleFile): CopyFileOnChange(ModuleFile, FileDir) @@ -1631,38 +1660,73 @@ class ModuleAutoGen(AutoGen): CreateDirectory(destination_dir) CopyFileOnChange(File, destination_dir) =20 - def AttemptModuleCacheCopy(self): - # If library or Module is binary do not skip by hash - if self.IsBinaryModule: + def SaveHashChainFileToCache(self, gDict): + if not GlobalData.gBinCacheDest: + return False + + self.GenPreMakefileHash(gDict) + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDi= gest: + EdkLogger.quiet("[cache warning]: Cannot generate PreMakefileH= ash for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return False + + self.GenMakeHash(gDict) + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain or \ + not gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest: + EdkLogger.quiet("[cache warning]: Cannot generate MakeHashChai= n for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) return False - # .inc is contains binary information so do not skip by hash as we= ll - for f_ext in self.SourceFileList: - if '.inc' in str(f_ext): - return False - FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir,= self.MetaFile.BaseName) - HashFile =3D path.join(FileDir, self.Name + '.hash') - if os.path.exists(HashFile): - f =3D open(HashFile, 'r') - CacheHash =3D f.read() - f.close() - self.GenModuleHash() - if GlobalData.gModuleHash[self.Arch][self.Name]: - if CacheHash =3D=3D GlobalData.gModuleHash[self.Arch][self= .Name]: - for root, dir, files in os.walk(FileDir): - for f in files: - if self.Name + '.hash' in f: - CopyFileOnChange(HashFile, self.BuildDir) - else: - File =3D path.join(root, f) - sub_dir =3D os.path.relpath(File, FileDir) - destination_file =3D os.path.join(self.Out= putDir, sub_dir) - destination_dir =3D os.path.dirname(destin= ation_file) - CreateDirectory(destination_dir) - CopyFileOnChange(File, destination_dir) - if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "Pcd= Dxe": - CreatePcdDatabaseCode(self, TemplateString(), Temp= lateString()) - return True - return False + + # save the hash chain list as cache file + MakeHashStr =3D str(GlobalData.gCacheIR[(self.MetaFile.Path, self.= Arch)].MakeHashHexDigest) + CacheDestDir =3D path.join(GlobalData.gBinCacheDest, self.Platform= Info.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.So= urceDir, self.MetaFile.BaseName) + CacheHashDestDir =3D path.join(CacheDestDir, MakeHashStr) + ModuleHashPair =3D path.join(CacheDestDir, self.Name + ".ModuleHas= hPair") + MakeHashChain =3D path.join(CacheHashDestDir, self.Name + ".MakeHa= shChain") + ModuleFilesChain =3D path.join(CacheHashDestDir, self.Name + ".Mod= uleFilesChain") + + # save the HashChainDict as json file + CreateDirectory (CacheDestDir) + CreateDirectory (CacheHashDestDir) + try: + ModuleHashPairList =3D [] # tuple list: [tuple(PreMakefileHash= , MakeHash)] + if os.path.exists(ModuleHashPair): + f =3D open(ModuleHashPair, 'r') + ModuleHashPairList =3D json.load(f) + f.close() + PreMakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].PreMake= fileHashHexDigest + MakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].MakeHashHe= xDigest + ModuleHashPairList.append((PreMakeHash, MakeHash)) + ModuleHashPairList =3D list(set(map(tuple, ModuleHashPairList)= )) + with open(ModuleHashPair, 'w') as f: + json.dump(ModuleHashPairList, f, indent=3D2) + except: + EdkLogger.quiet("[cache warning]: fail to save ModuleHashPair = file in cache: %s" % ModuleHashPair) + return False + + try: + with open(MakeHashChain, 'w') as f: + json.dump(gDict[(self.MetaFile.Path, self.Arch)].MakeHashC= hain, f, indent=3D2) + except: + EdkLogger.quiet("[cache warning]: fail to save MakeHashChain f= ile in cache: %s" % MakeHashChain) + return False + + try: + with open(ModuleFilesChain, 'w') as f: + json.dump(gDict[(self.MetaFile.Path, self.Arch)].ModuleFil= esChain, f, indent=3D2) + except: + EdkLogger.quiet("[cache warning]: fail to save ModuleFilesChai= n file in cache: %s" % ModuleFilesChain) + return False + + # save the autogenfile and makefile for debug usage + CacheDebugDir =3D path.join(CacheHashDestDir, "CacheDebug") + CreateDirectory (CacheDebugDir) + CopyFileOnChange(gDict[(self.MetaFile.Path, self.Arch)].MakefilePa= th, CacheDebugDir) + if gDict[(self.MetaFile.Path, self.Arch)].AutoGenFileList: + for File in gDict[(self.MetaFile.Path, self.Arch)].AutoGenFile= List: + CopyFileOnChange(str(File), CacheDebugDir) + + return True =20 ## Create makefile for the module and its dependent libraries # @@ -1671,6 +1735,11 @@ class ModuleAutoGen(AutoGen): # @cached_class_function def CreateMakeFile(self, CreateLibraryMakeFile=3DTrue, GenFfsList =3D = []): + gDict =3D GlobalData.gCacheIR + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone: + return + # nest this function inside it's only caller. def CreateTimeStamp(): FileSet =3D {self.MetaFile.Path} @@ -1701,8 +1770,8 @@ class ModuleAutoGen(AutoGen): for LibraryAutoGen in self.LibraryAutoGenList: LibraryAutoGen.CreateMakeFile() =20 - # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping - if not GlobalData.gUseHashCache and self.CanSkip(): + # CanSkip uses timestamps to determine build skipping + if self.CanSkip(): return =20 if len(self.CustomMakefile) =3D=3D 0: @@ -1718,6 +1787,24 @@ class ModuleAutoGen(AutoGen): =20 CreateTimeStamp() =20 + MakefileType =3D Makefile._FileType + MakefileName =3D Makefile._FILE_NAME_[MakefileType] + MakefilePath =3D os.path.join(self.MakeFileDir, MakefileName) + + MewIR =3D ModuleBuildCacheIR(self.MetaFile.Path, self.Arch) + MewIR.MakefilePath =3D MakefilePath + MewIR.DependencyHeaderFileSet =3D Makefile.DependencyHeaderFileSet + MewIR.CreateMakeFileDone =3D True + with GlobalData.file_lock: + try: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.MakefilePath =3D MakefilePath + IR.DependencyHeaderFileSet =3D Makefile.DependencyHeaderFi= leSet + IR.CreateMakeFileDone =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + except: + gDict[(self.MetaFile.Path, self.Arch)] =3D MewIR + def CopyBinaryFiles(self): for File in self.Module.Binaries: SrcPath =3D File.Path @@ -1729,6 +1816,11 @@ class ModuleAutoGen(AutoGen): # dependent libraries will be cr= eated # def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): + gDict =3D GlobalData.gCacheIR + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone: + return + if self.IsCodeFileCreated: return =20 @@ -1744,8 +1836,9 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary and CreateLibraryCodeFile: for LibraryAutoGen in self.LibraryAutoGenList: LibraryAutoGen.CreateCodeFile() - # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping - if not GlobalData.gUseHashCache and self.CanSkip(): + + # CanSkip uses timestamps to determine build skipping + if self.CanSkip(): return =20 AutoGenList =3D [] @@ -1785,6 +1878,16 @@ class ModuleAutoGen(AutoGen): (" ".join(AutoGenList), " ".join(IgoredAutoGen= List), self.Name, self.Arch)) =20 self.IsCodeFileCreated =3D True + MewIR =3D ModuleBuildCacheIR(self.MetaFile.Path, self.Arch) + MewIR.CreateCodeFileDone =3D True + with GlobalData.file_lock: + try: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.CreateCodeFileDone =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + except: + gDict[(self.MetaFile.Path, self.Arch)] =3D MewIR + return AutoGenList =20 ## Summarize the ModuleAutoGen objects of all libraries used by this m= odule @@ -1854,46 +1957,468 @@ class ModuleAutoGen(AutoGen): =20 return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8') =20 + def GenModuleFilesHash(self, gDict): + # Early exit if module or library has been hashed and is in memory + if (self.MetaFile.Path, self.Arch) in gDict: + if gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain: + return gDict[(self.MetaFile.Path, self.Arch)] + + DependencyFileSet =3D set() + # Add Module Meta file + DependencyFileSet.add(self.MetaFile) + + # Add Module's source files + if self.SourceFileList: + for File in set(self.SourceFileList): + DependencyFileSet.add(File) + + # Add modules's include header files + # Search dependency file list for each source file + SourceFileList =3D [] + OutPutFileList =3D [] + for Target in self.IntroTargetList: + SourceFileList.extend(Target.Inputs) + OutPutFileList.extend(Target.Outputs) + if OutPutFileList: + for Item in OutPutFileList: + if Item in SourceFileList: + SourceFileList.remove(Item) + SearchList =3D [] + for file_path in self.IncludePathList + self.BuildOptionIncPathLis= t: + # skip the folders in platform BuildDir which are not been gen= erated yet + if file_path.startswith(os.path.abspath(self.PlatformInfo.Buil= dDir)+os.sep): + continue + SearchList.append(file_path) + FileDependencyDict =3D {} + ForceIncludedFile =3D [] + for F in SourceFileList: + # skip the files which are not been generated yet, because + # the SourceFileList usually contains intermediate build files= , e.g. AutoGen.c + if not os.path.exists(F.Path): + continue + FileDependencyDict[F] =3D GenMake.GetDependencyList(self, self= .FileDependCache, F, ForceIncludedFile, SearchList) + + if FileDependencyDict: + for Dependency in FileDependencyDict.values(): + DependencyFileSet.update(set(Dependency)) + + # Caculate all above dependency files hash + # Initialze hash object + FileList =3D [] + m =3D hashlib.md5() + for File in sorted(DependencyFileSet, key=3Dlambda x: str(x)): + if not os.path.exists(str(File)): + EdkLogger.quiet("[cache warning]: header file %s is missin= g for module: %s[%s]" % (File, self.MetaFile.Path, self.Arch)) + continue + f =3D open(str(File), 'rb') + Content =3D f.read() + f.close() + m.update(Content) + FileList.append((str(File), hashlib.md5(Content).hexdigest())) + + + MewIR =3D ModuleBuildCacheIR(self.MetaFile.Path, self.Arch) + MewIR.ModuleFilesHashDigest =3D m.digest() + MewIR.ModuleFilesHashHexDigest =3D m.hexdigest() + MewIR.ModuleFilesChain =3D FileList + with GlobalData.file_lock: + try: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.ModuleFilesHashDigest =3D m.digest() + IR.ModuleFilesHashHexDigest =3D m.hexdigest() + IR.ModuleFilesChain =3D FileList + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + except: + gDict[(self.MetaFile.Path, self.Arch)] =3D MewIR + + return gDict[(self.MetaFile.Path, self.Arch)] + + def GenPreMakefileHash(self, gDict): + # Early exit if module or library has been hashed and is in memory + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest: + return gDict[(self.MetaFile.Path, self.Arch)] + + # skip binary module + if self.IsBinaryModule: + return + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDiges= t: + self.GenModuleFilesHash(gDict) + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDiges= t: + EdkLogger.quiet("[cache warning]: Cannot generate ModuleFilesHa= shDigest for module %s[%s]" %(self.MetaFile.Path, self.Arch)) + return + + # Initialze hash object + m =3D hashlib.md5() + + # Add Platform level hash + if ('PlatformHash') in gDict: + m.update(gDict[('PlatformHash')].encode('utf-8')) + else: + EdkLogger.quiet("[cache warning]: PlatformHash is missing") + + # Add Package level hash + if self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=3Dlambda x: x= .PackageName): + if (Pkg.PackageName, 'PackageHash') in gDict: + m.update(gDict[(Pkg.PackageName, 'PackageHash')].encod= e('utf-8')) + else: + EdkLogger.quiet("[cache warning]: %s PackageHash neede= d by %s[%s] is missing" %(Pkg.PackageName, self.MetaFile.Name, self.Arch)) + + # Add Library hash + if self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=3Dlambda x: x.N= ame): + if not (Lib.MetaFile.Path, Lib.Arch) in gDict or \ + not gDict[(Lib.MetaFile.Path, Lib.Arch)].ModuleFilesHas= hDigest: + Lib.GenPreMakefileHash(gDict) + m.update(gDict[(Lib.MetaFile.Path, Lib.Arch)].ModuleFilesH= ashDigest) + + # Add Module self + m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDig= est) + + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.PreMakefileHashHexDigest =3D m.hexdigest() + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + + return gDict[(self.MetaFile.Path, self.Arch)] + + def GenMakeHeaderFilesHash(self, gDict): + # Early exit if module or library has been hashed and is in memory + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest: + return gDict[(self.MetaFile.Path, self.Arch)] + + # skip binary module + if self.IsBinaryModule: + return + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone: + if self.IsLibrary: + if (self.MetaFile.File,self.MetaFile.Root,self.Arch,self.M= etaFile.Path) in GlobalData.libConstPcd: + self.ConstPcd =3D GlobalData.libConstPcd[(self.MetaFil= e.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path)] + if (self.MetaFile.File,self.MetaFile.Root,self.Arch,self.M= etaFile.Path) in GlobalData.Refes: + self.ReferenceModules =3D GlobalData.Refes[(self.MetaF= ile.File,self.MetaFile.Root,self.Arch,self.MetaFile.Path)] + self.CreateCodeFile() + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone: + self.CreateMakeFile(GenFfsList=3DGlobalData.FfsCmd.get((self.M= etaFile.File, self.Arch),[])) + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].CreateCodeFileDone o= r \ + not gDict[(self.MetaFile.Path, self.Arch)].CreateMakeFileDone: + EdkLogger.quiet("[cache warning]: Cannot create CodeFile or Mak= efile for module %s[%s]" %(self.MetaFile.Path, self.Arch)) + return + + DependencyFileSet =3D set() + # Add Makefile + if gDict[(self.MetaFile.Path, self.Arch)].MakefilePath: + DependencyFileSet.add(gDict[(self.MetaFile.Path, self.Arch)].M= akefilePath) + else: + EdkLogger.quiet("[cache warning]: makefile is missing for modu= le %s[%s]" %(self.MetaFile.Path, self.Arch)) + + # Add header files + if gDict[(self.MetaFile.Path, self.Arch)].DependencyHeaderFileSet: + for File in gDict[(self.MetaFile.Path, self.Arch)].DependencyH= eaderFileSet: + DependencyFileSet.add(File) + else: + EdkLogger.quiet("[cache warning]: No dependency header found f= or module %s[%s]" %(self.MetaFile.Path, self.Arch)) + + # Add AutoGen files + if self.AutoGenFileList: + for File in set(self.AutoGenFileList): + DependencyFileSet.add(File) + + # Caculate all above dependency files hash + # Initialze hash object + FileList =3D [] + m =3D hashlib.md5() + for File in sorted(DependencyFileSet, key=3Dlambda x: str(x)): + if not os.path.exists(str(File)): + EdkLogger.quiet("[cache warning]: header file: %s doesn't = exist for module: %s[%s]" % (File, self.MetaFile.Path, self.Arch)) + continue + f =3D open(str(File), 'rb') + Content =3D f.read() + f.close() + m.update(Content) + FileList.append((str(File), hashlib.md5(Content).hexdigest())) + + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.AutoGenFileList =3D self.AutoGenFileList.keys() + IR.MakeHeaderFilesHashChain =3D FileList + IR.MakeHeaderFilesHashDigest =3D m.digest() + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + + return gDict[(self.MetaFile.Path, self.Arch)] + + def GenMakeHash(self, gDict): + # Early exit if module or library has been hashed and is in memory + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain: + return gDict[(self.MetaFile.Path, self.Arch)] + + # skip binary module + if self.IsBinaryModule: + return + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDiges= t: + self.GenModuleFilesHash(gDict) + if not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashD= igest: + self.GenMakeHeaderFilesHash(gDict) + + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDiges= t or \ + not gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain or \ + not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashD= igest or \ + not gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashC= hain: + EdkLogger.quiet("[cache warning]: Cannot generate ModuleFilesHa= sh or MakeHeaderFilesHash for module %s[%s]" %(self.MetaFile.Path, self.Arc= h)) + return + + # Initialze hash object + m =3D hashlib.md5() + MakeHashChain =3D [] + + # Add hash of makefile and dependency header files + m.update(gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHas= hDigest) + New =3D list(set(gDict[(self.MetaFile.Path, self.Arch)].MakeHeader= FilesHashChain) - set(MakeHashChain)) + New.sort(key=3Dlambda x: str(x)) + MakeHashChain +=3D New + + # Add Library hash + if self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=3Dlambda x: x.N= ame): + if not (Lib.MetaFile.Path, Lib.Arch) in gDict or \ + not gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashChain: + Lib.GenMakeHash(gDict) + if not gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashDigest: + print("Cannot generate MakeHash for lib module:", Lib.= MetaFile.Path, Lib.Arch) + continue + m.update(gDict[(Lib.MetaFile.Path, Lib.Arch)].MakeHashDige= st) + New =3D list(set(gDict[(Lib.MetaFile.Path, Lib.Arch)].Make= HashChain) - set(MakeHashChain)) + New.sort(key=3Dlambda x: str(x)) + MakeHashChain +=3D New + + # Add Module self + m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDig= est) + New =3D list(set(gDict[(self.MetaFile.Path, self.Arch)].ModuleFile= sChain) - set(MakeHashChain)) + New.sort(key=3Dlambda x: str(x)) + MakeHashChain +=3D New + + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.MakeHashDigest =3D m.digest() + IR.MakeHashHexDigest =3D m.hexdigest() + IR.MakeHashChain =3D MakeHashChain + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + + return gDict[(self.MetaFile.Path, self.Arch)] + + ## Decide whether we can skip the left autogen and make process + def CanSkipbyPreMakefileCache(self, gDict): + if not GlobalData.gBinCacheSource: + return False + + # If Module is binary, do not skip by cache + if self.IsBinaryModule: + return False + + # .inc is contains binary information so do not skip by hash as we= ll + for f_ext in self.SourceFileList: + if '.inc' in str(f_ext): + return False + + # Get the module hash values from stored cache and currrent build + # then check whether cache hit based on the hash values + # if cache hit, restore all the files from cache + FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.Sourc= eDir, self.MetaFile.BaseName) + FfsDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInfo= .OutputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs= ", self.Guid + self.Name) + + ModuleHashPairList =3D [] # tuple list: [tuple(PreMakefileHash, Ma= keHash)] + ModuleHashPair =3D path.join(FileDir, self.Name + ".ModuleHashPair= ") + if not os.path.exists(ModuleHashPair): + EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair f= ile: %s" % ModuleHashPair) + return False + + try: + f =3D open(ModuleHashPair, 'r') + ModuleHashPairList =3D json.load(f) + f.close() + except: + EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair = file: %s" % ModuleHashPair) + return False + + self.GenPreMakefileHash(gDict) + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDi= gest: + EdkLogger.quiet("[cache warning]: PreMakefileHashHexDigest is = missing for module %s[%s]" %(self.MetaFile.Path, self.Arch)) + return False + + MakeHashStr =3D None + CurrentPreMakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].PreM= akefileHashHexDigest + for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairL= ist): + if PreMakefileHash =3D=3D CurrentPreMakeHash: + MakeHashStr =3D str(MakeHash) + + if not MakeHashStr: + return False + + TargetHashDir =3D path.join(FileDir, MakeHashStr) + TargetFfsHashDir =3D path.join(FfsDir, MakeHashStr) + + if not os.path.exists(TargetHashDir): + EdkLogger.quiet("[cache warning]: Cache folder is missing: %s"= % TargetHashDir) + return False + + for root, dir, files in os.walk(TargetHashDir): + for f in files: + File =3D path.join(root, f) + self.CacheCopyFile(self.OutputDir, TargetHashDir, File) + if os.path.exists(TargetFfsHashDir): + for root, dir, files in os.walk(TargetFfsHashDir): + for f in files: + File =3D path.join(root, f) + self.CacheCopyFile(self.FfsOutputDir, TargetFfsHashDir= , File) + + if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": + CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) + + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.PreMakeCacheHit =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + print("[cache hit]: checkpoint_PreMakefile:", self.MetaFile.Path, = self.Arch) + #EdkLogger.quiet("cache hit: %s[%s]" % (self.MetaFile.Path, self.A= rch)) + return True + + ## Decide whether we can skip the make process + def CanSkipbyMakeCache(self, gDict): + if not GlobalData.gBinCacheSource: + return False + + # If Module is binary, do not skip by cache + if self.IsBinaryModule: + print("[cache miss]: checkpoint_Makefile: binary module:", sel= f.MetaFile.Path, self.Arch) + return False + + # .inc is contains binary information so do not skip by hash as we= ll + for f_ext in self.SourceFileList: + if '.inc' in str(f_ext): + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.MakeCacheHit =3D False + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + print("[cache miss]: checkpoint_Makefile: .inc module:", s= elf.MetaFile.Path, self.Arch) + return False + + # Get the module hash values from stored cache and currrent build + # then check whether cache hit based on the hash values + # if cache hit, restore all the files from cache + FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.Sourc= eDir, self.MetaFile.BaseName) + FfsDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInfo= .OutputDir, self.BuildTarget + "_" + self.ToolChain, TAB_FV_DIRECTORY, "Ffs= ", self.Guid + self.Name) + + ModuleHashPairList =3D [] # tuple list: [tuple(PreMakefileHash, Ma= keHash)] + ModuleHashPair =3D path.join(FileDir, self.Name + ".ModuleHashPair= ") + if not os.path.exists(ModuleHashPair): + EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair f= ile: %s" % ModuleHashPair) + return False + + try: + f =3D open(ModuleHashPair, 'r') + ModuleHashPairList =3D json.load(f) + f.close() + except: + EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair = file: %s" % ModuleHashPair) + return False + + self.GenMakeHash(gDict) + if not (self.MetaFile.Path, self.Arch) in gDict or \ + not gDict[(self.MetaFile.Path, self.Arch)].MakeHashHexDigest: + EdkLogger.quiet("[cache warning]: MakeHashHexDigest is missing= for module %s[%s]" %(self.MetaFile.Path, self.Arch)) + return False + + MakeHashStr =3D None + CurrentMakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].MakeHas= hHexDigest + for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairL= ist): + if MakeHash =3D=3D CurrentMakeHash: + MakeHashStr =3D str(MakeHash) + + if not MakeHashStr: + print("[cache miss]: checkpoint_Makefile:", self.MetaFile.Path= , self.Arch) + return False + + TargetHashDir =3D path.join(FileDir, MakeHashStr) + TargetFfsHashDir =3D path.join(FfsDir, MakeHashStr) + if not os.path.exists(TargetHashDir): + EdkLogger.quiet("[cache warning]: Cache folder is missing: %s"= % TargetHashDir) + return False + + for root, dir, files in os.walk(TargetHashDir): + for f in files: + File =3D path.join(root, f) + self.CacheCopyFile(self.OutputDir, TargetHashDir, File) + + if os.path.exists(TargetFfsHashDir): + for root, dir, files in os.walk(TargetFfsHashDir): + for f in files: + File =3D path.join(root, f) + self.CacheCopyFile(self.FfsOutputDir, TargetFfsHashDir= , File) + + if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": + CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) + with GlobalData.file_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.MakeCacheHit =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR + print("[cache hit]: checkpoint_Makefile:", self.MetaFile.Path, sel= f.Arch) + return True + ## Decide whether we can skip the ModuleAutoGen process - def CanSkipbyHash(self): + def CanSkipbyCache(self, gDict): # Hashing feature is off - if not GlobalData.gUseHashCache: + if not GlobalData.gBinCacheSource: return False =20 - # Initialize a dictionary for each arch type - if self.Arch not in GlobalData.gBuildHashSkipTracking: - GlobalData.gBuildHashSkipTracking[self.Arch] =3D dict() + if self in GlobalData.gBuildHashSkipTracking: + return GlobalData.gBuildHashSkipTracking[self] =20 # If library or Module is binary do not skip by hash if self.IsBinaryModule: + GlobalData.gBuildHashSkipTracking[self] =3D False return False =20 # .inc is contains binary information so do not skip by hash as we= ll for f_ext in self.SourceFileList: if '.inc' in str(f_ext): + GlobalData.gBuildHashSkipTracking[self] =3D False return False =20 - # Use Cache, if exists and if Module has a copy in cache - if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy(): + if not (self.MetaFile.Path, self.Arch) in gDict: + return False + + if gDict[(self.MetaFile.Path, self.Arch)].PreMakeCacheHit: + GlobalData.gBuildHashSkipTracking[self] =3D True return True =20 - # Early exit for libraries that haven't yet finished building - HashFile =3D path.join(self.BuildDir, self.Name + ".hash") - if self.IsLibrary and not os.path.exists(HashFile): - return False + if gDict[(self.MetaFile.Path, self.Arch)].MakeCacheHit: + GlobalData.gBuildHashSkipTracking[self] =3D True + return True =20 - # Return a Boolean based on if can skip by hash, either from memor= y or from IO. - if self.Name not in GlobalData.gBuildHashSkipTracking[self.Arch]: - # If hashes are the same, SaveFileOnChange() will return False. - GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] =3D no= t SaveFileOnChange(HashFile, self.GenModuleHash(), True) - return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] - else: - return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] + return False =20 ## Decide whether we can skip the ModuleAutoGen process # If any source file is newer than the module than we cannot skip # def CanSkip(self): + # Don't skip if cache feature enabled + if GlobalData.gUseHashCache or GlobalData.gBinCacheDest or GlobalD= ata.gBinCacheSource: + return False if self.MakeFileDir in GlobalData.gSikpAutoGenCache: return True if not os.path.exists(self.TimeStampPath): diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Sourc= e/Python/Common/GlobalData.py old mode 100644 new mode 100755 index bd45a43728..452dca32f0 --- a/BaseTools/Source/Python/Common/GlobalData.py +++ b/BaseTools/Source/Python/Common/GlobalData.py @@ -119,3 +119,12 @@ gModuleBuildTracking =3D dict() # Top Dict: Key: Arch Type Value: Dictionary # Second Dict: Key: Module\Library Name Value: True\False gBuildHashSkipTracking =3D dict() + +# Common dictionary to share module cache intermediate result and state +gCacheIR =3D None +# Common lock for the file access in multiple process AutoGens +file_lock =3D None +# Common dictionary to share platform libraries' constant Pcd +libConstPcd =3D None +# Common dictionary to share platform libraries' reference info +Refes =3D None \ No newline at end of file diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py old mode 100644 new mode 100755 index 4bfa54666b..d7c817b95c --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -595,7 +595,7 @@ class BuildTask: # def AddDependency(self, Dependency): for Dep in Dependency: - if not Dep.BuildObject.IsBinaryModule and not Dep.BuildObject.= CanSkipbyHash(): + if not Dep.BuildObject.IsBinaryModule and not Dep.BuildObject.= CanSkipbyCache(GlobalData.gCacheIR): self.DependencyList.append(BuildTask.New(Dep)) # BuildT= ask list =20 ## The thread wrapper of LaunchCommand function @@ -811,7 +811,7 @@ class Build(): self.AutoGenMgr =3D None EdkLogger.info("") os.chdir(self.WorkspaceDir) - self.share_data =3D Manager().dict() + GlobalData.gCacheIR =3D Manager().dict() self.log_q =3D log_q def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_dat= a): try: @@ -820,6 +820,13 @@ class Build(): feedback_q =3D mp.Queue() file_lock =3D mp.Lock() error_event =3D mp.Event() + GlobalData.file_lock =3D file_lock + FfsCmd =3D DataPipe.Get("FfsCommand") + if FfsCmd is None: + FfsCmd =3D {} + GlobalData.FfsCmd =3D FfsCmd + GlobalData.libConstPcd =3D DataPipe.Get("LibConstPcd") + GlobalData.Refes =3D DataPipe.Get("REFS") auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,share_data,self.log_q,error_event) for _ in range= (self.ThreadNumber)] self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q,err= or_event) self.AutoGenMgr.start() @@ -827,14 +834,28 @@ class Build(): w.start() if PcdMaList is not None: for PcdMa in PcdMaList: + if GlobalData.gBinCacheSource and self.Target in [None= , "", "all"]: + PcdMa.GenModuleFilesHash(share_data) + PcdMa.GenPreMakefileHash(share_data) + if PcdMa.CanSkipbyPreMakefileCache(share_data): + continue + PcdMa.CreateCodeFile(False) PcdMa.CreateMakeFile(False,GenFfsList =3D DataPipe.Get= ("FfsCommand").get((PcdMa.MetaFile.File, PcdMa.Arch),[])) =20 + if GlobalData.gBinCacheSource and self.Target in [None= , "", "all"]: + PcdMa.GenMakeHeaderFilesHash(share_data) + PcdMa.GenMakeHash(share_data) + if PcdMa.CanSkipbyMakeCache(share_data): + continue + self.AutoGenMgr.join() rt =3D self.AutoGenMgr.Status return rt, 0 - except Exception as e: - return False,e.errcode + except FatalError as e: + return False, e.args[0] + except: + return False, UNKNOWN_ERROR =20 ## Load configuration # @@ -1199,10 +1220,11 @@ class Build(): mqueue.put(m) =20 AutoGenObject.DataPipe.DataContainer =3D {"FfsCommand":FfsComm= and} + AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget": sel= f.Target} self.Progress.Start("Generating makefile and code") data_pipe_file =3D os.path.join(AutoGenObject.BuildDir, "Globa= lVar_%s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch)) AutoGenObject.DataPipe.dump(data_pipe_file) - autogen_rt, errorcode =3D self.StartAutoGen(mqueue, AutoGenObj= ect.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data) + autogen_rt,errorcode =3D self.StartAutoGen(mqueue, AutoGenObje= ct.DataPipe, self.SkipAutoGen, PcdMaList, GlobalData.gCacheIR) self.Progress.Stop("done!") if not autogen_rt: self.AutoGenMgr.TerminateWorkers() @@ -1799,6 +1821,15 @@ class Build(): CmdListDict =3D None if GlobalData.gEnableGenfdsMultiThread and self.Fdf: CmdListDict =3D self._GenFfsCmd(Wa.ArchList) + + # Add Platform and Package level hash in share_data for mo= dule hash calculation later + if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest: + GlobalData.gCacheIR[('PlatformHash')] =3D GlobalData.g= PlatformHash + for PkgName in GlobalData.gPackageHash.keys(): + GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D = GlobalData.gPackageHash[PkgName] + GlobalData.file_lock =3D mp.Lock() + GlobalData.FfsCmd =3D CmdListDict + self.Progress.Stop("done!") MaList =3D [] ExitFlag =3D threading.Event() @@ -1808,20 +1839,23 @@ class Build(): AutoGenStart =3D time.time() GlobalData.gGlobalDefines['ARCH'] =3D Arch Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTar= get, ToolChain, Arch) + GlobalData.libConstPcd =3D Pa.DataPipe.Get("LibConstPc= d") + GlobalData.Refes =3D Pa.DataPipe.Get("REFS") for Module in Pa.Platform.Modules: if self.ModuleFile.Dir =3D=3D Module.Dir and self.= ModuleFile.Name =3D=3D Module.Name: Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, = ToolChain, Arch, self.PlatformFile,Pa.DataPipe) if Ma is None: continue MaList.append(Ma) - if Ma.CanSkipbyHash(): - self.HashSkipModules.append(Ma) - if GlobalData.gBinCacheSource: - EdkLogger.quiet("cache hit: %s[%s]" % = (Ma.MetaFile.Path, Ma.Arch)) - continue - else: - if GlobalData.gBinCacheSource: - EdkLogger.quiet("cache miss: %s[%s]" %= (Ma.MetaFile.Path, Ma.Arch)) + + if GlobalData.gBinCacheSource and self.Target = in [None, "", "all"]: + Ma.GenModuleFilesHash(GlobalData.gCacheIR) + Ma.GenPreMakefileHash(GlobalData.gCacheIR) + if Ma.CanSkipbyPreMakefileCache(GlobalData= .gCacheIR): + self.HashSkipModules.append(Ma) + EdkLogger.quiet("cache hit: %s[%s]" % (= Ma.MetaFile.Path, Ma.Arch)) + continue + # Not to auto-gen for targets 'clean', 'cleanl= ib', 'cleanall', 'run', 'fds' if self.Target not in ['clean', 'cleanlib', 'c= leanall', 'run', 'fds']: # for target which must generate AutoGen c= ode and makefile @@ -1841,6 +1875,18 @@ class Build(): self.Progress.Stop("done!") if self.Target =3D=3D "genmake": return True + + if GlobalData.gBinCacheSource and self.Tar= get in [None, "", "all"]: + Ma.GenMakeHeaderFilesHash(GlobalData.g= CacheIR) + Ma.GenMakeHash(GlobalData.gCacheIR) + if Ma.CanSkipbyMakeCache(GlobalData.gC= acheIR): + self.HashSkipModules.append(Ma) + EdkLogger.quiet("cache hit: %s[%s]= " % (Ma.MetaFile.Path, Ma.Arch)) + continue + else: + EdkLogger.quiet("cache miss: %s[%s= ]" % (Ma.MetaFile.Path, Ma.Arch)) + Ma.PrintFirstMakeCacheMissFile(Glo= balData.gCacheIR) + self.BuildModules.append(Ma) # Initialize all modules in tracking to 'FAIL' if Ma.Arch not in GlobalData.gModuleBuildTrack= ing: @@ -1985,11 +2031,18 @@ class Build(): if GlobalData.gEnableGenfdsMultiThread and self.Fdf: CmdListDict =3D self._GenFfsCmd(Wa.ArchList) =20 + # Add Platform and Package level hash in share_data for mo= dule hash calculation later + if GlobalData.gBinCacheSource or GlobalData.gBinCacheDest: + GlobalData.gCacheIR[('PlatformHash')] =3D GlobalData.g= PlatformHash + for PkgName in GlobalData.gPackageHash.keys(): + GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D = GlobalData.gPackageHash[PkgName] + # multi-thread exit flag ExitFlag =3D threading.Event() ExitFlag.clear() self.AutoGenTime +=3D int(round((time.time() - WorkspaceAu= toGenTime))) self.BuildModules =3D [] + TotalModules =3D [] for Arch in Wa.ArchList: PcdMaList =3D [] AutoGenStart =3D time.time() @@ -2009,6 +2062,7 @@ class Build(): ModuleList.append(Inf) Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDic= t} Pa.DataPipe.DataContainer =3D {"Workspace_timestamp": = Wa._SrcTimeStamp} + Pa.DataPipe.DataContainer =3D {"CommandTarget": self.T= arget} for Module in ModuleList: # Get ModuleAutoGen object to generate C code file= and makefile Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile,Pa.DataPipe) @@ -2019,30 +2073,34 @@ class Build(): Ma.PlatformInfo =3D Pa Ma.Workspace =3D Wa PcdMaList.append(Ma) - if Ma.CanSkipbyHash(): - self.HashSkipModules.append(Ma) - if GlobalData.gBinCacheSource: - EdkLogger.quiet("cache hit: %s[%s]" % (Ma.= MetaFile.Path, Ma.Arch)) - continue - else: - if GlobalData.gBinCacheSource: - EdkLogger.quiet("cache miss: %s[%s]" % (Ma= .MetaFile.Path, Ma.Arch)) - - # Not to auto-gen for targets 'clean', 'cleanlib',= 'cleanall', 'run', 'fds' - # for target which must generate AutoGen code = and makefile - - self.BuildModules.append(Ma) + TotalModules.append(Ma) # Initialize all modules in tracking to 'FAIL' if Ma.Arch not in GlobalData.gModuleBuildTracking: GlobalData.gModuleBuildTracking[Ma.Arch] =3D d= ict() if Ma not in GlobalData.gModuleBuildTracking[Ma.Ar= ch]: GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = =3D 'FAIL' + mqueue =3D mp.Queue() for m in Pa.GetAllModuleInfo: mqueue.put(m) data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVa= r_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) - autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa= .DataPipe, self.SkipAutoGen, PcdMaList,self.share_data) + autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa= .DataPipe, self.SkipAutoGen, PcdMaList, GlobalData.gCacheIR) + + # Skip cache hit modules + if GlobalData.gBinCacheSource: + for Ma in TotalModules: + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.g= CacheIR and \ + GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.= Arch)].PreMakeCacheHit: + self.HashSkipModules.append(Ma) + continue + if (Ma.MetaFile.Path, Ma.Arch) in GlobalData.g= CacheIR and \ + GlobalData.gCacheIR[(Ma.MetaFile.Path, Ma.= Arch)].MakeCacheHit: + self.HashSkipModules.append(Ma) + continue + self.BuildModules.append(Ma) + else: + self.BuildModules.extend(TotalModules) =20 if not autogen_rt: self.AutoGenMgr.TerminateWorkers() @@ -2050,9 +2108,24 @@ class Build(): raise FatalError(errorcode) self.AutoGenTime +=3D int(round((time.time() - AutoGenStar= t))) self.Progress.Stop("done!") + + if GlobalData.gBinCacheSource: + EdkLogger.quiet("Total cache hit driver num: %s, cache= miss driver num: %s" % (len(set(self.HashSkipModules)), len(set(self.Build= Modules)))) + CacheHitMa =3D set() + CacheNotHitMa =3D set() + for IR in GlobalData.gCacheIR.keys(): + if 'PlatformHash' in IR or 'PackageHash' in IR: + continue + if GlobalData.gCacheIR[IR].PreMakeCacheHit or Glob= alData.gCacheIR[IR].MakeCacheHit: + CacheHitMa.add(IR) + else: + # There might be binary module or module which= has .inc files, not count for cache miss + CacheNotHitMa.add(IR) + EdkLogger.quiet("Total module num: %s, cache hit modul= e num: %s" % (len(CacheHitMa)+len(CacheNotHitMa), len(CacheHitMa))) + for Arch in Wa.ArchList: MakeStart =3D time.time() - for Ma in self.BuildModules: + for Ma in set(self.BuildModules): # Generate build task for the module if not Ma.IsBinaryModule: Bt =3D BuildTask.New(ModuleMakeUnit(Ma, Pa.Bui= ldCommand,self.Target)) --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45735): https://edk2.groups.io/g/devel/message/45735 Mute This Topic: https://groups.io/mt/32887326/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Mon Apr 29 01:49:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45736+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45736+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565880921; cv=none; d=zoho.com; s=zohoarc; b=nW6BgEPnN5JhuI49zZzOIQNqJcvJfDSQEgadeSan+liPhrK6rGxMVK1p9ai0AEb3lug2/h0pggP1nLeEH1hzWSlALh51njOnGJyX5ptofY996Rtcvlbk6xHSnAbAPsHRIkDIXvj1KwUe+SWYD4fx9l8NhIT/BKsR87dOQ6QX3vo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565880921; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=Tu/nIkLJZsi7YLmcs89Cl6q0K6dnHXnGw4i9qq1TZdA=; b=mk5W2IihvlTo93UXaUSnBBJlP0i2/KifIAJs1ifw+nTdPldLcIEcpwB6nMiROA0wDGoinSE7W7syQhVaa5gx8XvwnkhZxEzPlZmw65CTKWiPbktMKqWeE4y4SxrxTlEEUV4kRnqJeTDmBpuUbBSpsDJyhRtbCf9FzOCMoiFuvyw= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45736+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565880921199345.28121292827814; Thu, 15 Aug 2019 07:55:21 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com []) by groups.io with SMTP; Thu, 15 Aug 2019 07:28:39 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2019 07:26:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,389,1559545200"; d="scan'208";a="171125284" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.104]) by orsmga008.jf.intel.com with ESMTP; 15 Aug 2019 07:26:33 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v6 2/5] BaseTools: Print first cache missing file for build cachle Date: Thu, 15 Aug 2019 22:26:18 +0800 Message-Id: <20190815142621.7644-3-steven.shi@intel.com> In-Reply-To: <20190815142621.7644-1-steven.shi@intel.com> References: <20190815142621.7644-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565880920; bh=Z02+/zJ90hJEbkNa9617+OmjnZ6EsFbudq1u1ppgLcY=; h=Cc:Date:From:Reply-To:Subject:To; b=owpdC48zluifXFLiJAYSHoL5RLWE5/TA3f++IC9LBIem1iplAN3o88Y4fHTK4zJUmyU pzIoOlQtBsRLXTV1W6QQrd17kseCMwt2jb7Xk4KfDBvB+gsgAxGcnLVaogEagrPgkOWWx +IcgUVD5d1DN/qtq9gKI5sGUX4BfxgB1Skc= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1925 When a module build cache miss, add support to print the first cache missing file path and name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 2 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 76 ++++++++++++++++++++= ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index a84ed46f2e..30d2f96fc7 100755 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -246,6 +246,8 @@ class AutoGenWorkerInProcess(mp.Process): Ma.GenMakeHash(GlobalData.gCacheIR) if Ma.CanSkipbyMakeCache(GlobalData.gCacheIR): continue + else: + Ma.PrintFirstMakeCacheMissFile(GlobalData.gCacheIR) except Empty: pass except: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index 613b0d2fb8..6db3b47a91 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -2379,6 +2379,82 @@ class ModuleAutoGen(AutoGen): print("[cache hit]: checkpoint_Makefile:", self.MetaFile.Path, sel= f.Arch) return True =20 + ## Show the first file name which causes cache miss + def PrintFirstMakeCacheMissFile(self, gDict): + if not GlobalData.gBinCacheSource: + return + + # skip binary module + if self.IsBinaryModule: + return + + if not (self.MetaFile.Path, self.Arch) in gDict: + return + + # Only print cache miss file for the MakeCache not hit module + if gDict[(self.MetaFile.Path, self.Arch)].MakeCacheHit: + return + + if not gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain: + EdkLogger.quiet("[cache insight]: MakeHashChain is missing for= : %s[%s]" % (self.MetaFile.Path, self.Arch)) + return + + # Find the cache dir name through the .ModuleHashPair file info + FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.Sourc= eDir, self.MetaFile.BaseName) + + ModuleHashPairList =3D [] # tuple list: [tuple(PreMakefileHash, Ma= keHash)] + ModuleHashPair =3D path.join(FileDir, self.Name + ".ModuleHashPair= ") + if not os.path.exists(ModuleHashPair): + EdkLogger.quiet("[cache insight]: Cannot find ModuleHashPair f= ile for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return + + try: + f =3D open(ModuleHashPair, 'r') + ModuleHashPairList =3D json.load(f) + f.close() + except: + EdkLogger.quiet("[cache insight]: Cannot load ModuleHashPair f= ile for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return + + MakeHashSet =3D set() + for idx, (PreMakefileHash, MakeHash) in enumerate (ModuleHashPairL= ist): + TargetHashDir =3D path.join(FileDir, str(MakeHash)) + if os.path.exists(TargetHashDir): + MakeHashSet.add(MakeHash) + if not MakeHashSet: + EdkLogger.quiet("[cache insight]: Cannot find valid cache dir = for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) + return + + TargetHash =3D list(MakeHashSet)[0] + TargetHashDir =3D path.join(FileDir, str(TargetHash)) + if len(MakeHashSet) > 1 : + EdkLogger.quiet("[cache insight]: found multiple cache dirs fo= r this module, random select dir '%s' to search the first cache miss file: = %s[%s]" % (TargetHash, self.MetaFile.Path, self.Arch)) + + ListFile =3D path.join(TargetHashDir, self.Name + '.MakeHashChain') + if os.path.exists(ListFile): + try: + f =3D open(ListFile, 'r') + CachedList =3D json.load(f) + f.close() + except: + EdkLogger.quiet("[cache insight]: Cannot load MakeHashChai= n file: %s" % ListFile) + return + else: + EdkLogger.quiet("[cache insight]: Cannot find MakeHashChain fi= le: %s" % ListFile) + return + + CurrentList =3D gDict[(self.MetaFile.Path, self.Arch)].MakeHashCha= in + for idx, (file, hash) in enumerate (CurrentList): + (filecached, hashcached) =3D CachedList[idx] + if file !=3D filecached: + EdkLogger.quiet("[cache insight]: first different file in = %s[%s] is %s, the cached one is %s" % (self.MetaFile.Path, self.Arch, file,= filecached)) + break + if hash !=3D hashcached: + EdkLogger.quiet("[cache insight]: first cache miss file in= %s[%s] is %s" % (self.MetaFile.Path, self.Arch, file)) + break + + return True + ## Decide whether we can skip the ModuleAutoGen process def CanSkipbyCache(self, gDict): # Hashing feature is off --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45736): https://edk2.groups.io/g/devel/message/45736 Mute This Topic: https://groups.io/mt/32887327/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Mon Apr 29 01:49:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45737+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45737+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565880921; cv=none; d=zoho.com; s=zohoarc; b=O9fKkpNTyoHsN4N8wFP4a9f/sOCWPSRhlZgijGCjNhtAsBFt5+WtgA2UcGMHnFEgh8oKnIlcC2XpwT/IOx3BVG2ilCqpTm+j/2ABEx6OuFItMYSVKxOuy1Pz57emFxM/4qGzNsJRTDC4OgnInh/rkvl2DJdAjhE5p+4SPrX/ODE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565880921; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=5bnEwVgwzy6gNvlr19w+9EY+M8sHy0VUxnKs3bx5LAw=; b=GtJj6KLVFYKmHC5xlKw9cL7Sfh6NazN8p0VqxQcSj4x9+fKPfMazHz5mgDpKS0V3ytjmXGgMIjW2RUu9as9jLb+TptGtl4bE1R0tELDHCTlX2lGl01AavEMVjHi1cxMWpMINWdTsMMiHpMdGVPsbdZbG4Ub13XeFNeD7R+zI1xY= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45737+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 156588092117635.84638693672366; Thu, 15 Aug 2019 07:55:21 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com []) by groups.io with SMTP; Thu, 15 Aug 2019 07:28:40 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2019 07:26:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,389,1559545200"; d="scan'208";a="171125297" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.104]) by orsmga008.jf.intel.com with ESMTP; 15 Aug 2019 07:26:36 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v6 3/5] BaseTools: Change the [Arch][Name] module key in Build cache Date: Thu, 15 Aug 2019 22:26:19 +0800 Message-Id: <20190815142621.7644-4-steven.shi@intel.com> In-Reply-To: <20190815142621.7644-1-steven.shi@intel.com> References: <20190815142621.7644-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565880920; bh=pDLSjEHc0wRefq0CMjO7LX6iqQyzMkWu0UpfAGLf1S8=; h=Cc:Date:From:Reply-To:Subject:To; b=ANDx8nL604vPmPECpQ7u33GeZqUjCE0N6KVCZOHcgUAp8G6rd3k68+/STZ7E9wLTdff IdeSDxChvcWsHsxroi7put1S521iTfh0JBnTDidwXuX1SGvlnZiRvd2WGLL71b7K4u0oY a3IziDHzsP9lM0xdawSFDqSvjSGeAq77Ru8= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1951 Current build cache use the module's [self.Arch][self.Name] info as the ModuleAutoGen object key in hash list and dictionary. The [self.Arch][self.Name] is not safe as the module key because there could be two modules with same module name and arch name in one platform. E.g. A platform can override a module or library instance in another different path, the overriding module can has the same module name and arch name as the original one. Directly use the ModuleAutoGen obj self as the key, because the obj __hash__ and __repr__ attributes already contain the full path and arch name. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/GenMake.py | 6 +----- BaseTools/Source/Python/build/build.py | 48 ++++++++++++++++++++------= ---------------------- 2 files changed, 21 insertions(+), 33 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index c11b423ae1..039c913280 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -959,16 +959,12 @@ cleanlib: # Keep the file to be checked headerFileDependencySet.add(aFileName) =20 - # Ensure that gModuleBuildTracking has been initialized per archit= ecture - if self._AutoGenObject.Arch not in GlobalData.gModuleBuildTracking: - GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch] =3D = dict() - # Check if a module dependency header file is missing from the mod= ule's MetaFile for aFile in headerFileDependencySet: if aFile in headerFilesInMetaFileSet: continue if GlobalData.gUseHashCache: - GlobalData.gModuleBuildTracking[self._AutoGenObject.Arch][= self._AutoGenObject] =3D 'FAIL_METAFILE' + GlobalData.gModuleBuildTracking[self._AutoGenObject] =3D '= FAIL_METAFILE' EdkLogger.warn("build","Module MetaFile [Sources] is missing l= ocal header!", ExtraData =3D "Local Header: " + aFile + " not fou= nd in " + self._AutoGenObject.MetaFile.Path ) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index d7c817b95c..299fa64311 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -630,12 +630,11 @@ class BuildTask: =20 # Set the value used by hash invalidation flow in GlobalData.gModu= leBuildTracking to 'SUCCESS' # If Module or Lib is being tracked, it did not fail header check = test, and built successfully - if (self.BuildItem.BuildObject.Arch in GlobalData.gModuleBuildTrac= king and - self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking[s= elf.BuildItem.BuildObject.Arch] and - GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arch= ][self.BuildItem.BuildObject] !=3D 'FAIL_METAFILE' and + if (self.BuildItem.BuildObject in GlobalData.gModuleBuildTracking = and + GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] != =3D 'FAIL_METAFILE' and not BuildTask._ErrorFlag.isSet() ): - GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject.Arc= h][self.BuildItem.BuildObject] =3D 'SUCCESS' + GlobalData.gModuleBuildTracking[self.BuildItem.BuildObject] = =3D 'SUCCESS' =20 # indicate there's a thread is available for another build task BuildTask._RunningQueueLock.acquire() @@ -1171,25 +1170,24 @@ class Build(): return =20 # GlobalData.gModuleBuildTracking contains only modules or libs th= at cannot be skipped by hash - for moduleAutoGenObjArch in GlobalData.gModuleBuildTracking.keys(): - for moduleAutoGenObj in GlobalData.gModuleBuildTracking[module= AutoGenObjArch].keys(): - # Skip invalidating for Successful Module/Lib builds - if GlobalData.gModuleBuildTracking[moduleAutoGenObjArch][m= oduleAutoGenObj] =3D=3D 'SUCCESS': - continue + for Ma in GlobalData.gModuleBuildTracking: + # Skip invalidating for Successful Module/Lib builds + if GlobalData.gModuleBuildTracking[Ma] =3D=3D 'SUCCESS': + continue =20 - # The module failed to build, failed to start building, or= failed the header check test from this point on + # The module failed to build, failed to start building, or fai= led the header check test from this point on =20 - # Remove .hash from build - ModuleHashFile =3D os.path.join(moduleAutoGenObj.BuildDir,= moduleAutoGenObj.Name + ".hash") - if os.path.exists(ModuleHashFile): - os.remove(ModuleHashFile) + # Remove .hash from build + ModuleHashFile =3D os.path.join(Ma.BuildDir, Ma.Name + ".hash") + if os.path.exists(ModuleHashFile): + os.remove(ModuleHashFile) =20 - # Remove .hash file from cache - if GlobalData.gBinCacheDest: - FileDir =3D os.path.join(GlobalData.gBinCacheDest, mod= uleAutoGenObj.Arch, moduleAutoGenObj.SourceDir, moduleAutoGenObj.MetaFile.B= aseName) - HashFile =3D os.path.join(FileDir, moduleAutoGenObj.Na= me + '.hash') - if os.path.exists(HashFile): - os.remove(HashFile) + # Remove .hash file from cache + if GlobalData.gBinCacheDest: + FileDir =3D os.path.join(GlobalData.gBinCacheDest, Ma.Plat= formInfo.OutputDir, Ma.BuildTarget + "_" + Ma.ToolChain, Ma.Arch, Ma.Source= Dir, Ma.MetaFile.BaseName) + HashFile =3D os.path.join(FileDir, Ma.Name + '.hash') + if os.path.exists(HashFile): + os.remove(HashFile) =20 ## Build a module or platform # @@ -1889,10 +1887,7 @@ class Build(): =20 self.BuildModules.append(Ma) # Initialize all modules in tracking to 'FAIL' - if Ma.Arch not in GlobalData.gModuleBuildTrack= ing: - GlobalData.gModuleBuildTracking[Ma.Arch] = =3D dict() - if Ma not in GlobalData.gModuleBuildTracking[M= a.Arch]: - GlobalData.gModuleBuildTracking[Ma.Arch][M= a] =3D 'FAIL' + GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) MakeStart =3D time.time() for Ma in self.BuildModules: @@ -2075,10 +2070,7 @@ class Build(): PcdMaList.append(Ma) TotalModules.append(Ma) # Initialize all modules in tracking to 'FAIL' - if Ma.Arch not in GlobalData.gModuleBuildTracking: - GlobalData.gModuleBuildTracking[Ma.Arch] =3D d= ict() - if Ma not in GlobalData.gModuleBuildTracking[Ma.Ar= ch]: - GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = =3D 'FAIL' + GlobalData.gModuleBuildTracking[Ma] =3D 'FAIL' =20 mqueue =3D mp.Queue() for m in Pa.GetAllModuleInfo: --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45737): https://edk2.groups.io/g/devel/message/45737 Mute This Topic: https://groups.io/mt/32887328/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Mon Apr 29 01:49:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45738+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45738+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565880921; cv=none; d=zoho.com; s=zohoarc; b=T604hyps08L0UKTn8ZEr32JjMA93bP6cc0Edeasq3VQQZy0GG/QZDHZb5hHDyyUOuN95myDKae/Qo4dwg4pcxDiH8NR9Zyceyy9xr1iW4WGqZroceAzDqPxEvZcsPfUzxy5Oas6ac5TLrJzlAaqqLTyCtt/NXKzbUu1UsmdcLlY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565880921; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=gqnwUPNyCt4gwpy9vOFc7MmUZr3S929gkt2xJdCcjXk=; b=eb7M09SyRmByhf+pNhFaX83tKIQA111AjMDb/lx4xDaX5tsASTi4iwMV26oNOkbgihu2amom3gqnzenvAtd5VFkqfo0DUh+WyTLh4yTlT/9zUXkCwv+ERMMmOOQOtDDF3MHDixDHizQO5HtY3hcivDAptr/0Owq2ynt+giVY50U= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45738+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565880921310204.59113627985755; Thu, 15 Aug 2019 07:55:21 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by groups.io with SMTP; Thu, 15 Aug 2019 07:28:40 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2019 07:26:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,389,1559545200"; d="scan'208";a="171125304" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.104]) by orsmga008.jf.intel.com with ESMTP; 15 Aug 2019 07:26:39 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v6 4/5] BaseTools: Add GenFds multi-thread support in build cache Date: Thu, 15 Aug 2019 22:26:20 +0800 Message-Id: <20190815142621.7644-5-steven.shi@intel.com> In-Reply-To: <20190815142621.7644-1-steven.shi@intel.com> References: <20190815142621.7644-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565880920; bh=rsjR9ieakrBan/oCGzWsKzQyNeLR9AsOicVyQJzyr3I=; h=Cc:Date:From:Reply-To:Subject:To; b=bhP9GmzSOtAO0vPWuEHHnprUb0/KjGjcFiKbLC3/LW9PCWpzFHc+V1h27Si/EDaoFuh t3maCbqz/LM+b1hUOPdVIZSK1QCjenroTOjuT/2gEEHe5zXKARMRu/FGxllKsdHHktHRq 2VPLoZrByjGZgZbFK/PyExC7VdMmJxdGP84= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1923 Fix the issue that the GenFds multi-thread will build fail if enable the build cache together. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 23 +++++++++++++++++---= --- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index 6db3b47a91..c489c3b9c4 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -1262,11 +1262,13 @@ class ModuleAutoGen(AutoGen): fStringIO.close () fInputfile.close () return OutputName + @cached_property def OutputFile(self): retVal =3D set() OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') + FfsOutputDir =3D self.FfsOutputDir.replace('\\', '/').rstrip('/') for Item in self.CodaTargetList: File =3D Item.Target.Path.replace('\\', '/').strip('/').replac= e(DebugDir, '').replace(OutputDir, '').strip('/') retVal.add(File) @@ -1282,6 +1284,12 @@ class ModuleAutoGen(AutoGen): if File.lower().endswith('.pdb'): retVal.add(File) =20 + for Root, Dirs, Files in os.walk(FfsOutputDir): + for File in Files: + if File.lower().endswith('.ffs') or File.lower().endswith(= '.offset') or File.lower().endswith('.raw') \ + or File.lower().endswith('.raw.txt'): + retVal.add(File) + return retVal =20 ## Create AsBuilt INF file the module @@ -1652,13 +1660,16 @@ class ModuleAutoGen(AutoGen): for File in self.OutputFile: File =3D str(File) if not os.path.isabs(File): - File =3D os.path.join(self.OutputDir, File) + NewFile =3D os.path.join(self.OutputDir, File) + if not os.path.exists(NewFile): + NewFile =3D os.path.join(self.FfsOutputDir, File) + File =3D NewFile if os.path.exists(File): - sub_dir =3D os.path.relpath(File, self.OutputDir) - destination_file =3D os.path.join(FileDir, sub_dir) - destination_dir =3D os.path.dirname(destination_file) - CreateDirectory(destination_dir) - CopyFileOnChange(File, destination_dir) + if File.lower().endswith('.ffs') or File.lower().endswith(= '.offset') or File.lower().endswith('.raw') \ + or File.lower().endswith('.raw.txt'): + self.CacheCopyFile(FfsDir, self.FfsOutputDir, File) + else: + self.CacheCopyFile(FileDir, self.OutputDir, File) =20 def SaveHashChainFileToCache(self, gDict): if not GlobalData.gBinCacheDest: --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45738): https://edk2.groups.io/g/devel/message/45738 Mute This Topic: https://groups.io/mt/32887329/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Mon Apr 29 01:49:50 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45739+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45739+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565880921; cv=none; d=zoho.com; s=zohoarc; b=A542mkqrmVMg7BY4tnXISCqKtXwinq4/5b6loj7RcLkGiK1Z24pDnUeFJtYLzfDYHJJYlTGDLb8/EoV5YbGICUG0OEHitbTcayMUGFb6T7LaLDbThmutqdlMKisLo64DO74Ea8UMXX4skQ5lRGMGQTIohVu7INsgaA9TLm7CNvE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565880921; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=OaCsUEZuOtGvRMXIwdTyTGkuEOc1fjmwgF73p1DeZwo=; b=HeEWHu8WOLIv67T5XMzpa2gpoZtaFHbU9rZ/e+rtRWAV2ZcRvmqL0KTI1Vsw/0XNyWBXqseTM8EJ59Qup1zxSOpdWbS9yPI+PGFD3EbJ/8DcC4/R6o8ZbLUtCmn884iI8LB1WNQ50xsm3l3Y32BQDG2gYIRVugmvB4GzAHWvUd0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45739+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565880921524699.5922032291128; Thu, 15 Aug 2019 07:55:21 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com []) by groups.io with SMTP; Thu, 15 Aug 2019 07:28:40 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Aug 2019 07:26:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,389,1559545200"; d="scan'208";a="171125322" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.104]) by orsmga008.jf.intel.com with ESMTP; 15 Aug 2019 07:26:42 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v6 5/5] BaseTools: Improve the file saving and copying reliability Date: Thu, 15 Aug 2019 22:26:21 +0800 Message-Id: <20190815142621.7644-6-steven.shi@intel.com> In-Reply-To: <20190815142621.7644-1-steven.shi@intel.com> References: <20190815142621.7644-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565880921; bh=KuASSLaPp0gMJqxxy2pvoNOeFk1/fpmICQgEsOC6k7A=; h=Cc:Date:From:Reply-To:Subject:To; b=k5KDyB67ENl2P3v8MaIQpwWK0I4s6bnv0BM9Rr31qIAshEO1jHsQQFGqZMu7CYyI2pD 3QLAFtIQgnOhOMVt0+I9tLRvEeWX2+3IOxfNLzFLu/VDF026q7p9b3oBQiYgXbtu5LUwz xihljN8TPl/UZVwGL88dJJtatya6yX5bm+A= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D2079 The Basetool CopyFileOnChange() and SaveFileOnChange() functions might raise the IOError occasionally when build in Windows with multi-process and build cache enabled. The CopyFileOnChange() and SaveFileOnChange() might be invoked in multiple sub-processes simultaneously, and this patch adds global locks to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi Reviewed-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 6 ++++-- BaseTools/Source/Python/AutoGen/CacheIR.py | 1 + BaseTools/Source/Python/AutoGen/DataPipe.py | 2 -- BaseTools/Source/Python/AutoGen/GenC.py | 0 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 101 +++++++++++++++++++= ++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------------= ------- BaseTools/Source/Python/Common/GlobalData.py | 2 ++ BaseTools/Source/Python/Common/Misc.py | 44 +++++++++++++++++++= ++++++++++++++++++------- BaseTools/Source/Python/build/build.py | 5 ++++- 8 files changed, 119 insertions(+), 42 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 30d2f96fc7..2e68538b1c 100755 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -133,7 +133,7 @@ class AutoGenManager(threading.Thread): def kill(self): self.feedback_q.put(None) class AutoGenWorkerInProcess(mp.Process): - def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data,log_q,error_event): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k,cache_lock,share_data,log_q,error_event): mp.Process.__init__(self) self.module_queue =3D module_queue self.data_pipe_file_path =3Ddata_pipe_file_path @@ -141,6 +141,7 @@ class AutoGenWorkerInProcess(mp.Process): self.feedback_q =3D feedback_q self.PlatformMetaFileSet =3D {} self.file_lock =3D file_lock + self.cache_lock =3D cache_lock self.share_data =3D share_data self.log_q =3D log_q self.error_event =3D error_event @@ -184,9 +185,10 @@ class AutoGenWorkerInProcess(mp.Process): GlobalData.gDatabasePath =3D self.data_pipe.Get("DatabasePath") GlobalData.gBinCacheSource =3D self.data_pipe.Get("BinCacheSou= rce") GlobalData.gBinCacheDest =3D self.data_pipe.Get("BinCacheDest") - GlobalData.gCacheIR =3D self.data_pipe.Get("CacheIR") + GlobalData.gCacheIR =3D self.share_data GlobalData.gEnableGenfdsMultiThread =3D self.data_pipe.Get("En= ableGenfdsMultiThread") GlobalData.file_lock =3D self.file_lock + GlobalData.cache_lock =3D self.cache_lock CommandTarget =3D self.data_pipe.Get("CommandTarget") pcd_from_build_option =3D [] for pcd_tuple in self.data_pipe.Get("BuildOptPcd"): diff --git a/BaseTools/Source/Python/AutoGen/CacheIR.py b/BaseTools/Source/= Python/AutoGen/CacheIR.py index 2d9ffe3f0b..715be5273c 100755 --- a/BaseTools/Source/Python/AutoGen/CacheIR.py +++ b/BaseTools/Source/Python/AutoGen/CacheIR.py @@ -24,5 +24,6 @@ class ModuleBuildCacheIR(): self.MakeHashDigest =3D None self.MakeHashHexDigest =3D None self.MakeHashChain =3D [] + self.CacheCrash =3D False self.PreMakeCacheHit =3D False self.MakeCacheHit =3D False diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py index 84e77c301a..99c27dd50d 100755 --- a/BaseTools/Source/Python/AutoGen/DataPipe.py +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -163,6 +163,4 @@ class MemoryDataPipe(DataPipe): =20 self.DataContainer =3D {"BinCacheDest":GlobalData.gBinCacheDest} =20 - self.DataContainer =3D {"CacheIR":GlobalData.gCacheIR} - self.DataContainer =3D {"EnableGenfdsMultiThread":GlobalData.gEnab= leGenfdsMultiThread} \ No newline at end of file diff --git a/BaseTools/Source/Python/AutoGen/GenC.py b/BaseTools/Source/Pyt= hon/AutoGen/GenC.py old mode 100644 new mode 100755 diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index c489c3b9c4..af67244ff8 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -28,6 +28,7 @@ from Common.caching import cached_class_function from AutoGen.ModuleAutoGenHelper import PlatformInfo,WorkSpaceInfo from AutoGen.CacheIR import ModuleBuildCacheIR import json +import tempfile =20 ## Mapping Makefile type gMakeTypeMap =3D {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"} @@ -1702,9 +1703,8 @@ class ModuleAutoGen(AutoGen): try: ModuleHashPairList =3D [] # tuple list: [tuple(PreMakefileHash= , MakeHash)] if os.path.exists(ModuleHashPair): - f =3D open(ModuleHashPair, 'r') - ModuleHashPairList =3D json.load(f) - f.close() + with open(ModuleHashPair, 'r') as f: + ModuleHashPairList =3D json.load(f) PreMakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].PreMake= fileHashHexDigest MakeHash =3D gDict[(self.MetaFile.Path, self.Arch)].MakeHashHe= xDigest ModuleHashPairList.append((PreMakeHash, MakeHash)) @@ -1766,10 +1766,12 @@ class ModuleAutoGen(AutoGen): =20 if os.path.exists (self.TimeStampPath): os.remove (self.TimeStampPath) - with open(self.TimeStampPath, 'w+') as fd: + with tempfile.NamedTemporaryFile('w+', dir=3Dos.path.dirname(s= elf.TimeStampPath), delete=3DFalse) as tf: for f in FileSet: - fd.write(f) - fd.write("\n") + tf.write(f) + tf.write("\n") + tempname =3D tf.name + SaveFileOnChange(self.TimeStampPath, tempname, False) =20 # Ignore generating makefile when it is a binary module if self.IsBinaryModule: @@ -1806,7 +1808,7 @@ class ModuleAutoGen(AutoGen): MewIR.MakefilePath =3D MakefilePath MewIR.DependencyHeaderFileSet =3D Makefile.DependencyHeaderFileSet MewIR.CreateMakeFileDone =3D True - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakefilePath =3D MakefilePath @@ -1891,7 +1893,7 @@ class ModuleAutoGen(AutoGen): self.IsCodeFileCreated =3D True MewIR =3D ModuleBuildCacheIR(self.MetaFile.Path, self.Arch) MewIR.CreateCodeFileDone =3D True - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.CreateCodeFileDone =3D True @@ -1951,9 +1953,8 @@ class ModuleAutoGen(AutoGen): m.update(GlobalData.gModuleHash[self.Arch][Lib.Name].encod= e('utf-8')) =20 # Add Module self - f =3D open(str(self.MetaFile), 'rb') - Content =3D f.read() - f.close() + with open(str(self.MetaFile), 'rb') as f: + Content =3D f.read() m.update(Content) =20 # Add Module's source files @@ -1974,6 +1975,11 @@ class ModuleAutoGen(AutoGen): if gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesChain: return gDict[(self.MetaFile.Path, self.Arch)] =20 + # skip if the module cache already crashed + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return + DependencyFileSet =3D set() # Add Module Meta file DependencyFileSet.add(self.MetaFile) @@ -2021,9 +2027,8 @@ class ModuleAutoGen(AutoGen): if not os.path.exists(str(File)): EdkLogger.quiet("[cache warning]: header file %s is missin= g for module: %s[%s]" % (File, self.MetaFile.Path, self.Arch)) continue - f =3D open(str(File), 'rb') - Content =3D f.read() - f.close() + with open(str(File), 'rb') as f: + Content =3D f.read() m.update(Content) FileList.append((str(File), hashlib.md5(Content).hexdigest())) =20 @@ -2032,7 +2037,7 @@ class ModuleAutoGen(AutoGen): MewIR.ModuleFilesHashDigest =3D m.digest() MewIR.ModuleFilesHashHexDigest =3D m.hexdigest() MewIR.ModuleFilesChain =3D FileList - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.ModuleFilesHashDigest =3D m.digest() @@ -2050,6 +2055,11 @@ class ModuleAutoGen(AutoGen): gDict[(self.MetaFile.Path, self.Arch)].PreMakefileHashHexDigest: return gDict[(self.MetaFile.Path, self.Arch)] =20 + # skip if the module cache already crashed + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return + # skip binary module if self.IsBinaryModule: return @@ -2091,7 +2101,7 @@ class ModuleAutoGen(AutoGen): # Add Module self m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDig= est) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.PreMakefileHashHexDigest =3D m.hexdigest() gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2104,6 +2114,11 @@ class ModuleAutoGen(AutoGen): gDict[(self.MetaFile.Path, self.Arch)].MakeHeaderFilesHashDigest: return gDict[(self.MetaFile.Path, self.Arch)] =20 + # skip if the module cache already crashed + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return + # skip binary module if self.IsBinaryModule: return @@ -2159,7 +2174,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) FileList.append((str(File), hashlib.md5(Content).hexdigest())) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.AutoGenFileList =3D self.AutoGenFileList.keys() IR.MakeHeaderFilesHashChain =3D FileList @@ -2174,6 +2189,11 @@ class ModuleAutoGen(AutoGen): gDict[(self.MetaFile.Path, self.Arch)].MakeHashChain: return gDict[(self.MetaFile.Path, self.Arch)] =20 + # skip if the module cache already crashed + if (self.MetaFile.Path, self.Arch) in gDict and \ + gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return + # skip binary module if self.IsBinaryModule: return @@ -2222,7 +2242,7 @@ class ModuleAutoGen(AutoGen): New.sort(key=3Dlambda x: str(x)) MakeHashChain +=3D New =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeHashDigest =3D m.digest() IR.MakeHashHexDigest =3D m.hexdigest() @@ -2236,6 +2256,12 @@ class ModuleAutoGen(AutoGen): if not GlobalData.gBinCacheSource: return False =20 + if gDict[(self.MetaFile.Path, self.Arch)].PreMakeCacheHit: + return True + + if gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return False + # If Module is binary, do not skip by cache if self.IsBinaryModule: return False @@ -2255,12 +2281,15 @@ class ModuleAutoGen(AutoGen): ModuleHashPair =3D path.join(FileDir, self.Name + ".ModuleHashPair= ") if not os.path.exists(ModuleHashPair): EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair f= ile: %s" % ModuleHashPair) + with GlobalData.cache_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.CacheCrash =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR return False =20 try: - f =3D open(ModuleHashPair, 'r') - ModuleHashPairList =3D json.load(f) - f.close() + with open(ModuleHashPair, 'r') as f: + ModuleHashPairList =3D json.load(f) except: EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair = file: %s" % ModuleHashPair) return False @@ -2300,7 +2329,7 @@ class ModuleAutoGen(AutoGen): if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.PreMakeCacheHit =3D True gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2313,6 +2342,12 @@ class ModuleAutoGen(AutoGen): if not GlobalData.gBinCacheSource: return False =20 + if gDict[(self.MetaFile.Path, self.Arch)].MakeCacheHit: + return True + + if gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return False + # If Module is binary, do not skip by cache if self.IsBinaryModule: print("[cache miss]: checkpoint_Makefile: binary module:", sel= f.MetaFile.Path, self.Arch) @@ -2321,7 +2356,7 @@ class ModuleAutoGen(AutoGen): # .inc is contains binary information so do not skip by hash as we= ll for f_ext in self.SourceFileList: if '.inc' in str(f_ext): - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeCacheHit =3D False gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2338,12 +2373,15 @@ class ModuleAutoGen(AutoGen): ModuleHashPair =3D path.join(FileDir, self.Name + ".ModuleHashPair= ") if not os.path.exists(ModuleHashPair): EdkLogger.quiet("[cache warning]: Cannot find ModuleHashPair f= ile: %s" % ModuleHashPair) + with GlobalData.cache_lock: + IR =3D gDict[(self.MetaFile.Path, self.Arch)] + IR.CacheCrash =3D True + gDict[(self.MetaFile.Path, self.Arch)] =3D IR return False =20 try: - f =3D open(ModuleHashPair, 'r') - ModuleHashPairList =3D json.load(f) - f.close() + with open(ModuleHashPair, 'r') as f: + ModuleHashPairList =3D json.load(f) except: EdkLogger.quiet("[cache warning]: fail to load ModuleHashPair = file: %s" % ModuleHashPair) return False @@ -2383,7 +2421,7 @@ class ModuleAutoGen(AutoGen): =20 if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeCacheHit =3D True gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2395,6 +2433,10 @@ class ModuleAutoGen(AutoGen): if not GlobalData.gBinCacheSource: return =20 + # skip if the module cache already crashed + if gDict[(self.MetaFile.Path, self.Arch)].CacheCrash: + return + # skip binary module if self.IsBinaryModule: return @@ -2420,9 +2462,8 @@ class ModuleAutoGen(AutoGen): return =20 try: - f =3D open(ModuleHashPair, 'r') - ModuleHashPairList =3D json.load(f) - f.close() + with open(ModuleHashPair, 'r') as f: + ModuleHashPairList =3D json.load(f) except: EdkLogger.quiet("[cache insight]: Cannot load ModuleHashPair f= ile for module: %s[%s]" % (self.MetaFile.Path, self.Arch)) return diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Sourc= e/Python/Common/GlobalData.py index 452dca32f0..09e92ce08c 100755 --- a/BaseTools/Source/Python/Common/GlobalData.py +++ b/BaseTools/Source/Python/Common/GlobalData.py @@ -122,6 +122,8 @@ gBuildHashSkipTracking =3D dict() =20 # Common dictionary to share module cache intermediate result and state gCacheIR =3D None +# Common lock for the module cache intermediate data +cache_lock =3D None # Common lock for the file access in multiple process AutoGens file_lock =3D None # Common dictionary to share platform libraries' constant Pcd diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Pyth= on/Common/Misc.py old mode 100644 new mode 100755 index 554ec010dd..4799635cc4 --- a/BaseTools/Source/Python/Common/Misc.py +++ b/BaseTools/Source/Python/Common/Misc.py @@ -448,7 +448,7 @@ def RemoveDirectory(Directory, Recursively=3DFalse): # @retval True If the file content is changed and the fil= e is renewed # @retval False If the file content is the same # -def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue): +def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue, FileLock=3DNone): =20 if os.path.exists(File): if IsBinaryFile: @@ -479,6 +479,13 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTru= e): if IsBinaryFile: OpenMode =3D "wb" =20 + # use default file_lock if no input new lock + if not FileLock: + FileLock =3D GlobalData.file_lock + if FileLock: + FileLock.acquire() + + if GlobalData.gIsWindows and not os.path.exists(File): # write temp file, then rename the temp file to the real file # to make sure the file be immediate saved to disk @@ -487,14 +494,26 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTr= ue): tempname =3D tf.name try: os.rename(tempname, File) - except: - EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IOErro= r %s' % X) + except IOError as X: + if GlobalData.gBinCacheSource: + EdkLogger.quiet("[cache error]:fails to save file with err= or: %s" % (X)) + else: + EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IO= Error %s' % X) + finally: + if FileLock: + FileLock.release() else: try: with open(File, OpenMode) as Fd: Fd.write(Content) except IOError as X: - EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IOErro= r %s' % X) + if GlobalData.gBinCacheSource: + EdkLogger.quiet("[cache error]:fails to save file with err= or: %s" % (X)) + else: + EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IO= Error %s' % X) + finally: + if FileLock: + FileLock.release() =20 return True =20 @@ -510,7 +529,7 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue= ): # @retval True The two files content are different and the file= is copied # @retval False No copy really happen # -def CopyFileOnChange(SrcFile, Dst): +def CopyFileOnChange(SrcFile, Dst, FileLock=3DNone): if not os.path.exists(SrcFile): return False =20 @@ -531,6 +550,12 @@ def CopyFileOnChange(SrcFile, Dst): if not os.access(DirName, os.W_OK): EdkLogger.error(None, PERMISSION_FAILURE, "Do not have write p= ermission on directory %s" % DirName) =20 + # use default file_lock if no input new lock + if not FileLock: + FileLock =3D GlobalData.file_lock + if FileLock: + FileLock.acquire() + # os.replace and os.rename are the atomic operations in python 3 and 2. # we use these two atomic operations to ensure the file copy is atomic: # copy the src to a temp file in the dst same folder firstly, then @@ -546,9 +571,14 @@ def CopyFileOnChange(SrcFile, Dst): if GlobalData.gIsWindows and os.path.exists(DstFile): os.remove(DstFile) os.rename(tempname, DstFile) - except IOError as X: - EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError %s' = % X) + if GlobalData.gBinCacheSource: + EdkLogger.quiet("[cache error]:fails to copy file with error: = %s" % (X)) + else: + EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError = %s' % X) + finally: + if FileLock: + FileLock.release() =20 return True =20 diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 299fa64311..2c10670a69 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -820,13 +820,15 @@ class Build(): file_lock =3D mp.Lock() error_event =3D mp.Event() GlobalData.file_lock =3D file_lock + cache_lock =3D mp.Lock() + GlobalData.cache_lock =3D cache_lock FfsCmd =3D DataPipe.Get("FfsCommand") if FfsCmd is None: FfsCmd =3D {} GlobalData.FfsCmd =3D FfsCmd GlobalData.libConstPcd =3D DataPipe.Get("LibConstPcd") GlobalData.Refes =3D DataPipe.Get("REFS") - auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,share_data,self.log_q,error_event) for _ in range= (self.ThreadNumber)] + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,cache_lock,share_data,self.log_q,error_event) for= _ in range(self.ThreadNumber)] self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q,err= or_event) self.AutoGenMgr.start() for w in auto_workers: @@ -1826,6 +1828,7 @@ class Build(): for PkgName in GlobalData.gPackageHash.keys(): GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D = GlobalData.gPackageHash[PkgName] GlobalData.file_lock =3D mp.Lock() + GlobalData.cache_lock =3D mp.Lock() GlobalData.FfsCmd =3D CmdListDict =20 self.Progress.Stop("done!") --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45739): https://edk2.groups.io/g/devel/message/45739 Mute This Topic: https://groups.io/mt/32887330/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-