From nobody Sun Feb 8 09:12:59 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+51774+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51774+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1575533901; cv=none; d=zohomail.com; s=zohoarc; b=WSBfVZsFyFaKD6kkc6ypjZy36T8gwj7vaIq2BurCi8SPzjtpcv8TzINYNCqF6zUHWPcg7+ssn7dI01cLnwUYDm4y3lfiOQypOQVoPNX04ldtZM6OWBM4pqN75muRG3uD9jL5gHns1YP4AFp0fFKmnInVhFkpFPkxHfngnWdD6vo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1575533901; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=yUeBF27SY4ZrVxinDxnexsiD9nYy1hbdEbYkS5WEhYA=; b=SVF+YzpyRNoOeKGzm1NRFdAfbtT4SH8H5P77j2C9iq5QYw7FWAzgWNxRFYp9lbvJJkuca3VZWxQL+xhO0s1sZ5YhZqo8EnMygDHWBjY00UMNHaBGPmx3IKa9LViGYhmnZQ/CbO9Vustdiqjyv1P5VO3QiwUUykeWHrGgRGWv9l4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+51774+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1575533901395567.3292549703665; Thu, 5 Dec 2019 00:18:21 -0800 (PST) Return-Path: X-Received: by 127.0.0.2 with SMTP id Dr1ZYY1788612xH4rJgzQSjx; Thu, 05 Dec 2019 00:18:20 -0800 X-Received: from mga12.intel.com (mga12.intel.com []) by mx.groups.io with SMTP id smtpd.web11.6291.1575533893968797692 for ; Thu, 05 Dec 2019 00:18:19 -0800 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 05 Dec 2019 00:18:19 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,280,1571727600"; d="scan'208";a="214059975" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga006.jf.intel.com with ESMTP; 05 Dec 2019 00:18:17 -0800 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Steven Shi Subject: [edk2-devel] [Patch 4/4 V4] BaseTools: Enhance Basetool for incremental build Date: Thu, 5 Dec 2019 16:18:07 +0800 Message-Id: <20191205081807.15092-5-bob.c.feng@intel.com> In-Reply-To: <20191205081807.15092-1-bob.c.feng@intel.com> References: <20191205081807.15092-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com X-Gm-Message-State: K2StwOoRufPptZK4suZydmU1x1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1575533900; bh=47MRCGOFEg2RyJ5wxMrncuyQpRd4okhO19N2310KhQ4=; h=Cc:Date:From:Reply-To:Subject:To; b=u2RksLMLkY0ZQU3FCm+hzWoDaoT19+TeDkFwGO5cmYdkeJffyFMqSSgLrpvdiK5i0q7 tjsDpLegQe0z3eGbgqaUqrZed31JEMCdzot1bPuRONI3kZPflj2+B9Bt5mXEoBgzlUSNa 6T/W9O0X7oaIEXbQ+2AqcNpK1rnzyqtdFM0= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2311 Include dependency file in Makefile to enhance incremental build Cc: Liming Gao Cc: Steven Shi Signed-off-by: Bob Feng --- V4: Move .deps files to where corresponding .obj file locate. BaseTools/Source/Python/AutoGen/GenMake.py | 83 ++--- .../Source/Python/AutoGen/IncludesAutoGen.py | 284 ++++++++++++++++++ .../Source/Python/AutoGen/ModuleAutoGen.py | 23 ++ BaseTools/Source/Python/build/build.py | 63 +++- 4 files changed, 380 insertions(+), 73 deletions(-) create mode 100644 BaseTools/Source/Python/AutoGen/IncludesAutoGen.py diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index 59a01a7f24..fe94f9a4c2 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -183,10 +183,16 @@ class BuildFile(object): EdkLogger.error("build", PARAMETER_INVALID, "Invalid build typ= e [%s]" % FileType, ExtraData=3D"[%s]" % str(self._AutoGenObject)) self._FileType =3D FileType FileContent =3D self._TEMPLATE_.Replace(self._TemplateDict) FileName =3D self._FILE_NAME_[FileType] + if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir= , "deps.txt")): + with open(os.path.join(self._AutoGenObject.MakeFileDir, "deps.= txt"),"w+") as fd: + fd.write("") + if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir= , "dependency")): + with open(os.path.join(self._AutoGenObject.MakeFileDir, "depen= dency"),"w+") as fd: + fd.write("") return SaveFileOnChange(os.path.join(self._AutoGenObject.MakeFileD= ir, FileName), FileContent, False) =20 ## Return a list of directory creation command string # # @param DirList The list of directory to be created @@ -302,13 +308,10 @@ MAKE_FILE =3D ${makefile_path} # Build Macro # ${BEGIN}${file_macro} ${END} =20 -COMMON_DEPS =3D ${BEGIN}${common_dependency_file} \\ - ${END} - # # Overridable Target Macro Definitions # FORCE_REBUILD =3D force_build INIT_TARGET =3D init @@ -380,10 +383,12 @@ gen_libs: # gen_fds: \t@"$(MAKE)" $(MAKE_FLAGS) -f $(BUILD_DIR)${separator}${makefile_name} fds \t@cd $(MODULE_BUILD_DIR) =20 +${INCLUDETAG} + # # Individual Object Build Targets # ${BEGIN}${file_build_target} ${END} @@ -513,13 +518,10 @@ cleanlib: if Tool =3D=3D "MAKE": continue # Remove duplicated include path, if any if Attr =3D=3D "FLAGS": Value =3D RemoveDupOption(Value, IncPrefix, MyAgo.= IncludePathList) - if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_= COMPILER_MSFT and Tool =3D=3D 'CC' and '/GM' in Value: - Value =3D Value.replace(' /MP', '') - MyAgo.BuildOption[Tool][Attr] =3D Value if Tool =3D=3D "OPTROM" and PCI_COMPRESS_Flag: ValueList =3D Value.split() if ValueList: for i, v in enumerate(ValueList): if '-e' =3D=3D v: @@ -538,11 +540,11 @@ cleanlib: RespFile =3D os.path.join(MyAgo.OutputDir, str(Resp).lower= () + '.txt') StrList =3D RespDict[Resp].split(' ') UnexpandMacro =3D [] NewStr =3D [] for Str in StrList: - if '$' in Str: + if '$' in Str or '-MMD' in Str or '-MF' in Str: UnexpandMacro.append(Str) else: NewStr.append(Str) UnexpandMacroStr =3D ' '.join(UnexpandMacro) NewRespStr =3D ' '.join(NewStr) @@ -588,14 +590,13 @@ cleanlib: "source_file" : Includ= ePathList } ) FileMacroList.append(FileMacro) # Add support when compiling .nasm source files - for File in self.FileCache.keys(): - if not str(File).endswith('.nasm'): - continue - IncludePathList =3D [] + IncludePathList =3D [] + asmsource =3D [item for item in MyAgo.SourceFileList if item.File.= upper().endswith((".NASM",".ASM",".NASMB","S"))] + if asmsource: for P in MyAgo.IncludePathList: IncludePath =3D self._INC_FLAG_['NASM'] + self.PlaceMacro(= P, self.Macros) if IncludePath.endswith(os.sep): IncludePath =3D IncludePath.rstrip(os.sep) # When compiling .nasm files, need to add a literal backsl= ash at each path @@ -604,11 +605,10 @@ cleanlib: IncludePath =3D ''.join([IncludePath, '^', os.sep]) else: IncludePath =3D os.path.join(IncludePath, '') IncludePathList.append(IncludePath) FileMacroList.append(self._FILE_MACRO_TEMPLATE.Replace({"macro= _name": "NASM_INC", "source_file": IncludePathList})) - break =20 # Generate macros used to represent files containing list of input= files for ListFileMacro in self.ListFileMacros: ListFileName =3D os.path.join(MyAgo.OutputDir, "%s.lst" % List= FileMacro.lower()[:len(ListFileMacro) - 5]) FileMacroList.append("%s =3D %s" % (ListFileMacro, ListFileNam= e)) @@ -694,10 +694,11 @@ cleanlib: "dependent_library_build_directory" : self.LibraryBuildDirecto= ryList, "library_build_command" : LibraryMakeCommandList, "file_macro" : FileMacroList, "file_build_target" : self.BuildTargetList, "backward_compatible_target": BcTargetList, + "INCLUDETAG" : self._INCLUDE_CMD_[self._File= Type] + " " + os.path.join("$(MODULE_BUILD_DIR)","dependency") } =20 return MakefileTemplateDict =20 def ParserGenerateFfsCmd(self): @@ -901,20 +902,14 @@ cleanlib: if OutPutFileList: for Item in OutPutFileList: if Item in SourceFileList: SourceFileList.remove(Item) =20 - FileDependencyDict =3D self.GetFileDependency( - SourceFileList, - ForceIncludedFile, - self._AutoGenObject.IncludePathList + = self._AutoGenObject.BuildOptionIncPathList - ) - + FileDependencyDict =3D {item:ForceIncludedFile for item in SourceF= ileList} =20 - if FileDependencyDict: - for Dependency in FileDependencyDict.values(): - self.DependencyHeaderFileSet.update(set(Dependency)) + for Dependency in FileDependencyDict.values(): + self.DependencyHeaderFileSet.update(set(Dependency)) =20 # Get a set of unique package includes from MetaFile parentMetaFileIncludes =3D set() for aInclude in self._AutoGenObject.PackageIncludePathList: aIncludeName =3D str(aInclude) @@ -970,46 +965,20 @@ cleanlib: GlobalData.gModuleBuildTracking[self._AutoGenObject] =3D '= FAIL_METAFILE' EdkLogger.warn("build","Module MetaFile [Sources] is missing l= ocal header!", ExtraData =3D "Local Header: " + aFile + " not fou= nd in " + self._AutoGenObject.MetaFile.Path ) =20 - DepSet =3D None for File,Dependency in FileDependencyDict.items(): if not Dependency: - FileDependencyDict[File] =3D ['$(FORCE_REBUILD)'] continue =20 self._AutoGenObject.AutoGenDepSet |=3D set(Dependency) =20 - # skip non-C files - if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoGen.c= ": - continue - elif DepSet is None: - DepSet =3D set(Dependency) - else: - DepSet &=3D set(Dependency) - # in case nothing in SourceFileList - if DepSet is None: - DepSet =3D set() - # - # Extract common files list in the dependency files - # - for File in DepSet: - self.CommonFileDependency.append(self.PlaceMacro(File.Path, se= lf.Macros)) - CmdSumDict =3D {} CmdTargetDict =3D {} CmdCppDict =3D {} DependencyDict =3D FileDependencyDict.copy() - for File in FileDependencyDict: - # skip non-C files - if File.Ext not in [".c", ".C"] or File.Name =3D=3D "AutoGen.c= ": - continue - NewDepSet =3D set(FileDependencyDict[File]) - NewDepSet -=3D DepSet - FileDependencyDict[File] =3D ["$(COMMON_DEPS)"] + list(NewDepS= et) - DependencyDict[File] =3D list(NewDepSet) =20 # Convert target description object to target string in makefile if self._AutoGenObject.BuildRuleFamily =3D=3D TAB_COMPILER_MSFT an= d TAB_C_CODE_FILE in self._AutoGenObject.Targets: for T in self._AutoGenObject.Targets[TAB_C_CODE_FILE]: NewFile =3D self.PlaceMacro(str(T), self.Macros) @@ -1078,21 +1047,17 @@ cleanlib: if CmdCppDict.get(item.Target.SubDir): CmdCppDict[item.Target.SubDir].append(Path) else: CmdCppDict[item.Target.SubDir] =3D ['$(MAKE_FILE)'= , Path] if CppPath.Path in DependencyDict: - if '$(FORCE_REBUILD)' in DependencyDict[CppPath.Pa= th]: - if '$(FORCE_REBUILD)' not in (self.CommonFileD= ependency + CmdCppDict[item.Target.SubDir]): - CmdCppDict[item.Target.SubDir].append('$(F= ORCE_REBUILD)') - else: - for Temp in DependencyDict[CppPath.Path]: - try: - Path =3D self.PlaceMacro(Temp.Path, se= lf.Macros) - except: - continue - if Path not in (self.CommonFileDependency = + CmdCppDict[item.Target.SubDir]): - CmdCppDict[item.Target.SubDir].append(= Path) + for Temp in DependencyDict[CppPath.Path]: + try: + Path =3D self.PlaceMacro(Temp.Path, self.M= acros) + except: + continue + if Path not in (self.CommonFileDependency + Cm= dCppDict[item.Target.SubDir]): + CmdCppDict[item.Target.SubDir].append(Path) if T.Commands: CommandList =3D T.Commands[:] for Item in CommandList[:]: SingleCommandList =3D Item.split() if len(SingleCommandList) > 0 and self.CheckCCCmd(SingleCo= mmandList): @@ -1107,11 +1072,11 @@ cleanlib: CmdTargetDict[CmdSign] =3D "%s %s" % (CmdTargetDic= t[CmdSign], SingleCommandList[-1]) Index =3D CommandList.index(Item) CommandList.pop(Index) if SingleCommandList[-1].endswith("%s%s.c" % (TAB_SLAS= H, CmdSumDict[CmdSign[3:].rsplit(TAB_SLASH, 1)[0]])): Cpplist =3D CmdCppDict[T.Target.SubDir] - Cpplist.insert(0, '$(OBJLIST_%d): $(COMMON_DEPS)' = % list(self.ObjTargetDict.keys()).index(T.Target.SubDir)) + Cpplist.insert(0, '$(OBJLIST_%d): ' % list(self.Ob= jTargetDict.keys()).index(T.Target.SubDir)) T.Commands[Index] =3D '%s\n\t%s' % (' \\\n\t'.join= (Cpplist), CmdTargetDict[CmdSign]) else: T.Commands.pop(Index) return T, CmdSumDict, CmdTargetDict, CmdCppDict =20 diff --git a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools= /Source/Python/AutoGen/IncludesAutoGen.py new file mode 100644 index 0000000000..bb6e883d84 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -0,0 +1,284 @@ +## @file +# Build cache intermediate result and state +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +from Common.caching import cached_property +import Common.EdkLogger as EdkLogger +import Common.LongFilePathOs as os +from Common.BuildToolError import * +from Common.Misc import SaveFileOnChange, PathClass +from Common.Misc import TemplateString +import sys +gIsFileMap =3D {} +if sys.platform =3D=3D "win32": + _INCLUDE_DEPS_TEMPLATE =3D TemplateString(''' +${BEGIN} +!IF EXIST(${deps_file}) +!INCLUDE ${deps_file} +!ENDIF +${END} + ''') +else: + _INCLUDE_DEPS_TEMPLATE =3D TemplateString(''' +${BEGIN} +-include ${deps_file} +${END} + ''') + +DEP_FILE_TAIL =3D "# Updated \n" + +class IncludesAutoGen(): + """ This class is to manage the dependent files witch are used in Make= file to support incremental build. + 1. C files: + 1. MSVS. + cl.exe has a build option /showIncludes to display include = files on stdout. Build tool captures + that messages and generate dependency files, .deps files. + 2. CLANG and GCC + -MMD -MF build option are used to generate dependency files= by compiler. Build tool updates the + .deps files. + 2. ASL files: + 1. Trim find out all the included files with asl specific incl= ude format and generate .trim.deps file. + 2. ASL PP use c preprocessor to find out all included files wi= th #include format and generate a .deps file + 3. build tool updates the .deps file + 3. ASM files (.asm, .s or .nasm): + 1. Trim find out all the included files with asl specific incl= ude format and generate .trim.deps file. + 2. ASM PP use c preprocessor to find out all included files wi= th #include format and generate a deps file + 3. build tool updates the .deps file + """ + def __init__(self, makefile_folder, ModuleAuto): + self.d_folder =3D makefile_folder + self.makefile_folder =3D makefile_folder + self.module_autogen =3D ModuleAuto + self.ToolChainFamily =3D ModuleAuto.ToolChainFamily + self.workspace =3D ModuleAuto.WorkspaceDir + + def CreateModuleDeps(self): + SaveFileOnChange(os.path.join(self.makefile_folder,"deps.txt"),"\n= ".join(self.DepsCollection),False) + + def CreateDepsInclude(self): + deps_file =3D {'deps_file':self.deps_files} + try: + deps_include_str =3D _INCLUDE_DEPS_TEMPLATE.Replace(deps_file) + except Exception as e: + print(e) + SaveFileOnChange(os.path.join(self.makefile_folder,"dependency"),d= eps_include_str,False) + + @cached_property + def deps_files(self): + """ Get all .deps file under module build folder. """ + deps_files =3D [] + for root, _, files in os.walk(self.d_folder, topdown=3DFalse): + for name in files: + if not name.endswith(".deps"): + continue + abspath =3D os.path.join(root, name) + deps_files.append(abspath) + return deps_files + + @cached_property + def DepsCollection(self): + """ Collect all the dependency files list from all .deps files und= er a module's build folder """ + includes =3D set() + targetname =3D [item[0].Name for item in self.TargetFileList.value= s()] + for abspath in self.deps_files: + try: + with open(abspath,"r") as fd: + lines =3D fd.readlines() + + firstlineitems =3D lines[0].split(": ") + dependency_file =3D firstlineitems[1].strip(" \\\n") + dependency_file =3D dependency_file.strip('''"''') + if dependency_file: + if os.path.normpath(dependency_file +".deps") =3D=3D a= bspath: + continue + filename =3D os.path.basename(dependency_file).strip() + if filename not in self.SourceFileList and filename no= t in targetname: + includes.add(dependency_file.strip()) + + for item in lines[1:]: + if item =3D=3D DEP_FILE_TAIL: + continue + dependency_file =3D item.strip(" \\\n") + dependency_file =3D dependency_file.strip('''"''') + if os.path.normpath(dependency_file +".deps") =3D=3D a= bspath: + continue + filename =3D os.path.basename(dependency_file).strip() + if filename in self.SourceFileList: + continue + if filename in targetname: + continue + includes.add(dependency_file.strip()) + except Exception as e: + EdkLogger.error("build",FILE_NOT_FOUND, "%s doesn't exist"= % abspath, ExtraData=3Dstr(e), RaiseError=3DFalse) + continue + rt =3D sorted(list(set([item.strip(' " \\\n') for item in includes= ]))) + return rt + + @cached_property + def SourceFileList(self): + """ Get a map of module's source files name to module's source fil= es path """ + source =3D {os.path.basename(item.File):item.Path for item in self= .module_autogen.SourceFileList} + middle_file =3D {} + for afile in source: + if afile.upper().endswith(".VFR"): + middle_file.update({afile.split(".")[0]+".c":os.path.join(= self.module_autogen.DebugDir,afile.split(".")[0]+".c")}) + if afile.upper().endswith((".S","ASM")): + middle_file.update({afile.split(".")[0]+".i":os.path.join(= self.module_autogen.OutputDir,afile.split(".")[0]+".i")}) + if afile.upper().endswith(".ASL"): + middle_file.update({afile.split(".")[0]+".i":os.path.join(= self.module_autogen.OutputDir,afile.split(".")[0]+".i")}) + source.update({"AutoGen.c":os.path.join(self.module_autogen.Output= Dir,"AutoGen.c")}) + source.update(middle_file) + return source + + @cached_property + def HasNamesakeSourceFile(self): + source_base_name =3D set([os.path.basename(item.File) for item in = self.module_autogen.SourceFileList]) + rt =3D len(source_base_name) !=3D len(self.module_autogen.SourceFi= leList) + return rt + @cached_property + def CcPPCommandPathSet(self): + rt =3D set() + rt.add(self.module_autogen.BuildOption.get('CC',{}).get('PATH')) + rt.add(self.module_autogen.BuildOption.get('ASLCC',{}).get('PATH')) + rt.add(self.module_autogen.BuildOption.get('ASLPP',{}).get('PATH')) + rt.add(self.module_autogen.BuildOption.get('VFRPP',{}).get('PATH')) + rt.add(self.module_autogen.BuildOption.get('PP',{}).get('PATH')) + rt.add(self.module_autogen.BuildOption.get('APP',{}).get('PATH')) + rt.discard(None) + return rt + @cached_property + def TargetFileList(self): + """ Get a map of module's target name to a tuple of module's targe= ts path and whose input file path """ + targets =3D {} + targets["AutoGen.obj"] =3D (PathClass(os.path.join(self.module_aut= ogen.OutputDir,"AutoGen.obj")),PathClass(os.path.join(self.module_autogen.D= ebugDir,"AutoGen.c"))) + for item in self.module_autogen.Targets.values(): + for block in item: + targets[block.Target.Path] =3D (block.Target,block.Inputs[= 0]) + return targets + + def GetRealTarget(self,source_file_abs): + """ Get the final target file based on source file abspath """ + source_target_map =3D {item[1].Path:item[0].Path for item in self.= TargetFileList.values()} + source_name_map =3D {item[1].File:item[0].Path for item in self.Ta= rgetFileList.values()} + target_abs =3D source_target_map.get(source_file_abs) + if target_abs is None: + if source_file_abs.strip().endswith(".i"): + sourcefilename =3D os.path.basename(source_file_abs.strip(= )) + for sourcefile in source_name_map: + if sourcefilename.split(".")[0] =3D=3D sourcefile.spli= t(".")[0]: + target_abs =3D source_name_map[sourcefile] + break + else: + target_abs =3D source_file_abs + else: + target_abs =3D source_file_abs + return target_abs + + def CreateDepsFileForMsvc(self, DepList): + """ Generate dependency files, .deps file from /showIncludes outpu= t message """ + if not DepList: + return + ModuleDepDict =3D {} + current_source =3D "" + SourceFileAbsPathMap =3D self.SourceFileList + for line in DepList: + line =3D line.strip() + if self.HasNamesakeSourceFile: + for cc_cmd in self.CcPPCommandPathSet: + if cc_cmd in line: + if '''"'''+cc_cmd+'''"''' in line: + cc_options =3D line[len(cc_cmd)+2:].split() + else: + cc_options =3D line[len(cc_cmd):].split() + SourceFileAbsPathMap =3D {os.path.basename(item):i= tem for item in cc_options if not item.startswith("/") and os.path.exists(i= tem)} + if line in SourceFileAbsPathMap: + current_source =3D line + if current_source not in ModuleDepDict: + ModuleDepDict[SourceFileAbsPathMap[current_source]] = =3D [] + elif "Note: including file:" =3D=3D line.lstrip()[:21]: + if not current_source: + EdkLogger.error("build",BUILD_ERROR, "Parse /showInclu= des output failed. line: %s. \n" % line, RaiseError=3DFalse) + else: + ModuleDepDict[SourceFileAbsPathMap[current_source]].ap= pend(line.lstrip()[22:].strip()) + + for source_abs in ModuleDepDict: + if ModuleDepDict[source_abs]: + target_abs =3D self.GetRealTarget(source_abs) + dep_file_name =3D os.path.basename(source_abs) + ".deps" + SaveFileOnChange(os.path.join(os.path.dirname(target_abs),= dep_file_name)," \\\n".join([target_abs+":"] + ['''"''' + item +'''"''' for= item in ModuleDepDict[source_abs]]),False) + + def UpdateDepsFileforNonMsvc(self): + """ Update .deps files. + 1. Update target path to absolute path. + 2. Update middle target to final target. + """ + + for abspath in self.deps_files: + if abspath.endswith(".trim.deps"): + continue + try: + newcontent =3D [] + with open(abspath,"r") as fd: + lines =3D fd.readlines() + if lines[-1] =3D=3D DEP_FILE_TAIL: + continue + firstlineitems =3D lines[0].strip().split(" ") + + if len(firstlineitems) > 2: + sourceitem =3D firstlineitems[1] + else: + sourceitem =3D lines[1].strip().split(" ")[0] + + source_abs =3D self.SourceFileList.get(sourceitem,sourceit= em) + firstlineitems[0] =3D self.GetRealTarget(source_abs) + p_target =3D firstlineitems + if not p_target[0].strip().endswith(":"): + p_target[0] +=3D ": " + + if len(p_target) =3D=3D 2: + p_target[0] +=3D lines[1] + newcontent.append(p_target[0]) + newcontent.extend(lines[2:]) + else: + line1 =3D " ".join(p_target).strip() + line1 +=3D "\n" + newcontent.append(line1) + newcontent.extend(lines[1:]) + + newcontent.append("\n") + newcontent.append(DEP_FILE_TAIL) + with open(abspath,"w") as fw: + fw.write("".join(newcontent)) + except Exception as e: + EdkLogger.error("build",FILE_NOT_FOUND, "%s doesn't exist"= % abspath, ExtraData=3Dstr(e), RaiseError=3DFalse) + continue + + def UpdateDepsFileforTrim(self): + """ Update .deps file which generated by trim. """ + + for abspath in self.deps_files: + if not abspath.endswith(".trim.deps"): + continue + try: + newcontent =3D [] + with open(abspath,"r") as fd: + lines =3D fd.readlines() + if lines[-1] =3D=3D DEP_FILE_TAIL: + continue + + source_abs =3D lines[0].strip().split(" ")[0] + targetitem =3D self.GetRealTarget(source_abs.strip(" :")) + + targetitem +=3D ": " + targetitem +=3D lines[1] + newcontent.append(targetitem) + newcontent.extend(lines[2:]) + newcontent.append("\n") + newcontent.append(DEP_FILE_TAIL) + with open(abspath,"w") as fw: + fw.write("".join(newcontent)) + except Exception as e: + EdkLogger.error("build",FILE_NOT_FOUND, "%s doesn't exist"= % abspath, ExtraData=3Dstr(e), RaiseError=3DFalse) + continue diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index e6d6c43810..1111d5de25 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -1127,12 +1127,35 @@ class ModuleAutoGen(AutoGen): if not self.MetaFile.OriginalPath.Path.startswith(PackageD= ir): IncludesList =3D list(set(Package.Includes).difference= (set(Package._PrivateIncludes))) for Inc in IncludesList: if Inc not in RetVal: RetVal.append(str(Inc)) + RetVal.extend(self.IncPathFromBuildOptions) return RetVal =20 + @cached_property + def IncPathFromBuildOptions(self): + IncPathList =3D [] + for tool in self.BuildOption: + if 'FLAGS' in self.BuildOption[tool]: + flags =3D self.BuildOption[tool]['FLAGS'] + whitespace =3D False + for flag in flags.split(" "): + flag =3D flag.strip() + if flag.startswith(("/I","-I")): + if len(flag)>2: + if os.path.exists(flag[2:]): + IncPathList.append(flag[2:]) + else: + whitespace =3D True + continue + if whitespace and flag: + if os.path.exists(flag): + IncPathList.append(flag) + whitespace =3D False + return IncPathList + @cached_property def IncludePathLength(self): return sum(len(inc)+1 for inc in self.IncludePathList) =20 ## Get the list of include paths from the packages diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 4b31356a42..6fcfc7ec7a 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -55,11 +55,11 @@ from GenFds.GenFds import GenFds, GenFdsApi import multiprocessing as mp from multiprocessing import Manager from AutoGen.DataPipe import MemoryDataPipe from AutoGen.ModuleAutoGenHelper import WorkSpaceInfo, PlatformInfo from GenFds.FdfParser import FdfParser - +from AutoGen.IncludesAutoGen import IncludesAutoGen =20 ## standard targets of build command gSupportedTarget =3D ['all', 'genc', 'genmake', 'modules', 'libraries', 'f= ds', 'clean', 'cleanall', 'cleanlib', 'run'] =20 ## build configuration file @@ -173,33 +173,46 @@ def NormFile(FilePath, Workspace): # # @param From The stream message read from # @param To The stream message put on # @param ExitFlag The flag used to indicate stopping reading # -def ReadMessage(From, To, ExitFlag): +def ReadMessage(From, To, ExitFlag,MemTo=3DNone): while True: # read one line a time Line =3D From.readline() # empty string means "end" if Line is not None and Line !=3D b"": - To(Line.rstrip().decode(encoding=3D'utf-8', errors=3D'ignore')) + LineStr =3D Line.rstrip().decode(encoding=3D'utf-8', errors=3D= 'ignore') + if MemTo is not None: + if "Note: including file:" =3D=3D LineStr.lstrip()[:21]: + MemTo.append(LineStr) + else: + To(LineStr) + MemTo.append(LineStr) + else: + To(LineStr) else: break if ExitFlag.isSet(): break =20 +class MakeSubProc(Popen): + def __init__(self,*args, **argv): + super(MakeSubProc,self).__init__(*args, **argv) + self.ProcOut =3D [] + ## Launch an external program # # This method will call subprocess.Popen to execute an external program wi= th # given options in specified directory. Because of the dead-lock issue dur= ing # redirecting output of the external program, threads are used to to do the # redirection work. # # @param Command A list or string containing the call of th= e program # @param WorkingDir The directory in which the program will be= running # -def LaunchCommand(Command, WorkingDir): +def LaunchCommand(Command, WorkingDir,ModuleAuto =3D None): BeginTime =3D time.time() # if working directory doesn't exist, Popen() will raise an exception if not os.path.isdir(WorkingDir): EdkLogger.error("build", FILE_NOT_FOUND, ExtraData=3DWorkingDir) =20 @@ -214,23 +227,23 @@ def LaunchCommand(Command, WorkingDir): =20 Proc =3D None EndOfProcedure =3D None try: # launch the command - Proc =3D Popen(Command, stdout=3DPIPE, stderr=3DPIPE, env=3Dos.env= iron, cwd=3DWorkingDir, bufsize=3D-1, shell=3DTrue) + Proc =3D MakeSubProc(Command, stdout=3DPIPE, stderr=3DPIPE, env=3D= os.environ, cwd=3DWorkingDir, bufsize=3D-1, shell=3DTrue) =20 # launch two threads to read the STDOUT and STDERR EndOfProcedure =3D Event() EndOfProcedure.clear() if Proc.stdout: - StdOutThread =3D Thread(target=3DReadMessage, args=3D(Proc.std= out, EdkLogger.info, EndOfProcedure)) + StdOutThread =3D Thread(target=3DReadMessage, args=3D(Proc.std= out, EdkLogger.info, EndOfProcedure,Proc.ProcOut)) StdOutThread.setName("STDOUT-Redirector") StdOutThread.setDaemon(False) StdOutThread.start() =20 if Proc.stderr: - StdErrThread =3D Thread(target=3DReadMessage, args=3D(Proc.std= err, EdkLogger.quiet, EndOfProcedure)) + StdErrThread =3D Thread(target=3DReadMessage, args=3D(Proc.std= err, EdkLogger.quiet, EndOfProcedure,Proc.ProcOut)) StdErrThread.setName("STDERR-Redirector") StdErrThread.setDaemon(False) StdErrThread.start() =20 # waiting for program exit @@ -261,10 +274,19 @@ def LaunchCommand(Command, WorkingDir): RespContent =3D f.read() f.close() EdkLogger.info(RespContent) =20 EdkLogger.error("build", COMMAND_FAILURE, ExtraData=3D"%s [%s]" % = (Command, WorkingDir)) + if ModuleAuto: + iau =3D IncludesAutoGen(WorkingDir,ModuleAuto) + if ModuleAuto.ToolChainFamily =3D=3D TAB_COMPILER_MSFT: + iau.CreateDepsFileForMsvc(Proc.ProcOut) + else: + iau.UpdateDepsFileforNonMsvc() + iau.UpdateDepsFileforTrim() + iau.CreateModuleDeps() + iau.CreateDepsInclude() return "%dms" % (int(round((time.time() - BeginTime) * 1000))) =20 ## The smallest unit that can be built in multi-thread build mode # # This is the base class of build unit. The "Obj" parameter must provide @@ -606,11 +628,11 @@ class BuildTask: # @param Command A list or string contains the call of = the command # @param WorkingDir The directory in which the program wil= l be running # def _CommandThread(self, Command, WorkingDir): try: - self.BuildItem.BuildObject.BuildTime =3D LaunchCommand(Command= , WorkingDir) + self.BuildItem.BuildObject.BuildTime =3D LaunchCommand(Command= , WorkingDir,self.BuildItem.BuildObject) self.CompleteFlag =3D True =20 # Run hash operation post dependency, to account for libs if GlobalData.gUseHashCache and self.BuildItem.BuildObject.IsL= ibrary: HashFile =3D path.join(self.BuildItem.BuildObject.BuildDir= , self.BuildItem.BuildObject.Name + ".hash") @@ -1274,23 +1296,36 @@ class Build(): self.BuildModules =3D [] return True =20 # build library if Target =3D=3D 'libraries': - for Lib in AutoGenObject.LibraryBuildDirectoryList: + DirList =3D [] + for Lib in AutoGenObject.LibraryAutoGenList: + if not Lib.IsBinaryModule: + DirList.append((os.path.join(AutoGenObject.BuildDir, L= ib.BuildDir),Lib)) + for Lib, LibAutoGen in DirList: NewBuildCommand =3D BuildCommand + ['-f', os.path.normpath= (os.path.join(Lib, makefile)), 'pbuild'] - LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir) + LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir,L= ibAutoGen) return True =20 # build module if Target =3D=3D 'modules': - for Lib in AutoGenObject.LibraryBuildDirectoryList: + DirList =3D [] + for Lib in AutoGenObject.LibraryAutoGenList: + if not Lib.IsBinaryModule: + DirList.append((os.path.join(AutoGenObject.BuildDir, L= ib.BuildDir),Lib)) + for Lib, LibAutoGen in DirList: NewBuildCommand =3D BuildCommand + ['-f', os.path.normpath= (os.path.join(Lib, makefile)), 'pbuild'] - LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir) - for Mod in AutoGenObject.ModuleBuildDirectoryList: + LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir,L= ibAutoGen) + + DirList =3D [] + for ModuleAutoGen in AutoGenObject.ModuleAutoGenList: + if not ModuleAutoGen.IsBinaryModule: + DirList.append((os.path.join(AutoGenObject.BuildDir, M= oduleAutoGen.BuildDir),ModuleAutoGen)) + for Mod,ModAutoGen in DirList: NewBuildCommand =3D BuildCommand + ['-f', os.path.normpath= (os.path.join(Mod, makefile)), 'pbuild'] - LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir) + LaunchCommand(NewBuildCommand, AutoGenObject.MakeFileDir,M= odAutoGen) self.CreateAsBuiltInf() if GlobalData.gBinCacheDest: self.UpdateBuildCache() self.BuildModules =3D [] return True --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#51774): https://edk2.groups.io/g/devel/message/51774 Mute This Topic: https://groups.io/mt/66938977/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-