From nobody Mon Feb 9 00:19:46 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+44502+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+44502+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1564389922; cv=none; d=zoho.com; s=zohoarc; b=R1LG2JODC6ok1reiFc2PegVYA8h9+Bf65fDGlOFDbbfLGcQBZsAnlhm0+B46Ibsbd8k+1uMaomP616qa+teGTnw/+eHVSqG4u3vjrxYIeHks0LOt9PHQSGuQ9WHfsk+jQuOyqyeTo/qsxe9hWh9cHVKfTnHwQ/hd0d3Qp2L6agg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1564389922; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=Kg6si6qHO+IK1lL3hsO+m2/pMVCgvrqe4txDM0f8dGo=; b=Gljdbdl2arTcje6BPLK8hRQt1qwpUgIUCxAMti9Jz0uQO1NQdYrspVW2M9cJ0FcRSWyAzTU+e0iOALI2OjRMRFOORVGUIZTYWFX+Nbu0SpWTCLFLstaFDqBbJnu/L0Q+a0vnSiMD5gquZyNLuhusG8bykje+VGnWa2tQYZVVB90= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+44502+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 156438992207977.08282500481414; Mon, 29 Jul 2019 01:45:22 -0700 (PDT) Return-Path: X-Received: from mga01.intel.com (mga01.intel.com []) by groups.io with SMTP; Mon, 29 Jul 2019 01:45:21 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Jul 2019 01:45:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,322,1559545200"; d="scan'208";a="322786699" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 29 Jul 2019 01:45:19 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 11/11] BaseTools: Enhance Multiple-Process AutoGen Date: Mon, 29 Jul 2019 16:44:56 +0800 Message-Id: <20190729084456.18844-12-bob.c.feng@intel.com> In-Reply-To: <20190729084456.18844-1-bob.c.feng@intel.com> References: <20190729084456.18844-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1564389921; bh=uMORGZT7M0sflb5PcEG7fX2Ml3NqU4Wm+BXfjA0dy4U=; h=Cc:Date:From:Reply-To:Subject:To; b=M++YH9XMVJWG1mhbZpj5KO45D11VY/pFwW+SQRxd/5SX2zPT6o+HYKZzcKue5z9UR7T V/oBqhnH69O+jM3t1bAy//1QjgkxZ/xF9WDFOpUaLMaPLJjjR3OwIRovw089pqzPtY0g5 o7QelcvDas/hOIdtOadevA6ePEouFP3EvkI= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 1. Set Log queue maxsize as thread number * 10 2. enhance ModuleUniqueBaseName function 3. fix bugs of build option pcd in sub Process 4. enhance error handling 5. fix bug in the function of duplicate modules handling. Cc: Liming Gao Signed-off-by: Bob Feng --- .../Source/Python/AutoGen/AutoGenWorker.py | 55 +++++++++-- BaseTools/Source/Python/AutoGen/DataPipe.py | 11 ++- .../Python/AutoGen/ModuleAutoGenHelper.py | 9 +- .../Source/Python/AutoGen/PlatformAutoGen.py | 49 +++++++--- .../Source/Python/AutoGen/WorkspaceAutoGen.py | 2 + BaseTools/Source/Python/build/build.py | 95 ++++++++++--------- 6 files changed, 148 insertions(+), 73 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index d1c55cffa8d0..de6a17396e12 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -14,20 +14,24 @@ import Common.GlobalData as GlobalData import Common.EdkLogger as EdkLogger import os from Common.MultipleWorkspace import MultipleWorkspace as mws from AutoGen.AutoGen import AutoGen from Workspace.WorkspaceDatabase import BuildDB -import time + try: from queue import Empty except: from Queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe import logging =20 +def clearQ(q): + while not q.empty(): + q.get_nowait() + class LogAgent(threading.Thread): def __init__(self,log_q,log_level,log_file=3DNone): super(LogAgent,self).__init__() self.log_q =3D log_q self.log_level =3D log_level @@ -88,45 +92,58 @@ class LogAgent(threading.Thread): self._InfoLogger_agent.log(log_message.levelno,log_message= .getMessage()) =20 def kill(self): self.log_q.put(None) class AutoGenManager(threading.Thread): - def __init__(self,autogen_workers, feedback_q): + def __init__(self,autogen_workers, feedback_q,error_event): super(AutoGenManager,self).__init__() self.autogen_workers =3D autogen_workers self.feedback_q =3D feedback_q self.Status =3D True + self.error_event =3D error_event def run(self): try: + fin_num =3D 0 while True: badnews =3D self.feedback_q.get() if badnews is None: + break + if badnews =3D=3D "Done": + fin_num +=3D 1 + else: self.Status =3D False self.TerminateWorkers() + if fin_num =3D=3D len(self.autogen_workers): + self.clearQueue() + for w in self.autogen_workers: + w.join() break except Exception: return =20 - def kill(self): - self.feedback_q.put(None) - + def clearQueue(self): + taskq =3D self.autogen_workers[0].module_queue + logq =3D self.autogen_workers[0].log_q + clearQ(taskq) + clearQ(self.feedback_q) + clearQ(logq) def TerminateWorkers(self): - for w in self.autogen_workers: - if w.is_alive(): - w.terminate() - + self.error_event.set() + def kill(self): + self.feedback_q.put(None) class AutoGenWorkerInProcess(mp.Process): - def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data,log_q): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data,log_q,error_event): mp.Process.__init__(self) self.module_queue =3D module_queue self.data_pipe_file_path =3Ddata_pipe_file_path self.data_pipe =3D None self.feedback_q =3D feedback_q self.PlatformMetaFileSet =3D {} self.file_lock =3D file_lock self.share_data =3D share_data self.log_q =3D log_q + self.error_event =3D error_event def GetPlatformMetaFile(self,filepath,root): try: return self.PlatformMetaFileSet[(filepath,root)] except: self.PlatformMetaFileSet[(filepath,root)] =3D filepath @@ -161,17 +178,28 @@ class AutoGenWorkerInProcess(mp.Process): os.environ._data =3D self.data_pipe.Get("Env_Var") GlobalData.gWorkspace =3D workspacedir GlobalData.gDisableIncludePathCheck =3D False GlobalData.gFdfParser =3D self.data_pipe.Get("FdfParser") GlobalData.gDatabasePath =3D self.data_pipe.Get("DatabasePath") + pcd_from_build_option =3D [] + for pcd_tuple in self.data_pipe.Get("BuildOptPcd"): + pcd_id =3D ".".join((pcd_tuple[0],pcd_tuple[1])) + if pcd_tuple[2].strip(): + pcd_id =3D ".".join((pcd_id,pcd_tuple[2])) + pcd_from_build_option.append("=3D".join((pcd_id,pcd_tuple[= 3]))) + GlobalData.BuildOptionPcd =3D pcd_from_build_option module_count =3D 0 FfsCmd =3D self.data_pipe.Get("FfsCommand") if FfsCmd is None: FfsCmd =3D {} PlatformMetaFile =3D self.GetPlatformMetaFile(self.data_pipe.G= et("P_Info").get("ActivePlatform"), self.data_pipe.Get("P_Info").= get("WorkspaceDir")) + libConstPcd =3D self.data_pipe.Get("LibConstPcd") + Refes =3D self.data_pipe.Get("REFS") while not self.module_queue.empty(): + if self.error_event.is_set(): + break module_count +=3D 1 module_file,module_root,module_path,module_basename,module= _originalpath,module_arch,IsLib =3D self.module_queue.get() modulefullpath =3D os.path.join(module_root,module_file) taskname =3D " : ".join((modulefullpath,module_arch)) module_metafile =3D PathClass(module_file,module_root) @@ -184,18 +212,25 @@ class AutoGenWorkerInProcess(mp.Process): arch =3D module_arch target =3D self.data_pipe.Get("P_Info").get("Target") toolchain =3D self.data_pipe.Get("P_Info").get("ToolChain") Ma =3D ModuleAutoGen(self.Wa,module_metafile,target,toolch= ain,arch,PlatformMetaFile,self.data_pipe) Ma.IsLibrary =3D IsLib + if IsLib: + if (Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaF= ile.Path) in libConstPcd: + Ma.ConstPcd =3D libConstPcd[(Ma.MetaFile.File,Ma.M= etaFile.Root,Ma.Arch,Ma.MetaFile.Path)] + if (Ma.MetaFile.File,Ma.MetaFile.Root,Ma.Arch,Ma.MetaF= ile.Path) in Refes: + Ma.ReferenceModules =3D Refes[(Ma.MetaFile.File,Ma= .MetaFile.Root,Ma.Arch,Ma.MetaFile.Path)] Ma.CreateCodeFile() Ma.CreateMakeFile(GenFfsList=3DFfsCmd.get((Ma.MetaFile.Fil= e, Ma.Arch),[])) Ma.CreateAsBuiltInf() except Empty: pass except: traceback.print_exc(file=3Dsys.stdout) self.feedback_q.put(taskname) + finally: + self.feedback_q.put("Done") =20 def printStatus(self): print("Processs ID: %d Run %d modules in AutoGen " % (os.getpid(),= len(AutoGen.Cache()))) print("Processs ID: %d Run %d modules in AutoGenInfo " % (os.getpi= d(),len(AutoGenInfo.GetCache()))) groupobj =3D {} diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py index 33d2b14c9add..2052084bdb4b 100644 --- a/BaseTools/Source/Python/AutoGen/DataPipe.py +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -72,11 +72,11 @@ class MemoryDataPipe(DataPipe): #Platform Module Pcds ModulePcds =3D {} for m in PlatformInfo.Platform.Modules: m_pcds =3D PlatformInfo.Platform.Modules[m].Pcds if m_pcds: - ModulePcds[(m.File,m.Root)] =3D [PCD_DATA( + ModulePcds[(m.File,m.Root,m.Arch)] =3D [PCD_DATA( pcd.TokenCName,pcd.TokenSpaceGuidCName,pcd.Type, pcd.DatumType,pcd.SkuInfoList,pcd.DefaultValue, pcd.MaxDatumSize,pcd.UserDefinedDefaultStoresFlag,pcd.validate= ranges, pcd.validlists,pcd.expressions,pcd.CustomAttribute,pcd.To= kenValue) for pcd in PlatformInfo.Platform.Modules[m].Pcds.values()] @@ -84,15 +84,22 @@ class MemoryDataPipe(DataPipe): =20 self.DataContainer =3D {"MOL_PCDS":ModulePcds} =20 #Module's Library Instance ModuleLibs =3D {} + libModules =3D {} for m in PlatformInfo.Platform.Modules: module_obj =3D BuildDB.BuildObject[m,PlatformInfo.Arch,Platfor= mInfo.BuildTarget,PlatformInfo.ToolChain] Libs =3D GetModuleLibInstances(module_obj, PlatformInfo.Platfo= rm, BuildDB.BuildObject, PlatformInfo.Arch,PlatformInfo.BuildTarget,Platfor= mInfo.ToolChain) - ModuleLibs[(m.File,m.Root,module_obj.Arch)] =3D [(l.MetaFile.F= ile,l.MetaFile.Root,l.Arch) for l in Libs] + for lib in Libs: + try: + libModules[(lib.MetaFile.File,lib.MetaFile.Root,lib.Ar= ch,lib.MetaFile.Path)].append((m.File,m.Root,module_obj.Arch,m.Path)) + except: + libModules[(lib.MetaFile.File,lib.MetaFile.Root,lib.Ar= ch,lib.MetaFile.Path)] =3D [(m.File,m.Root,module_obj.Arch,m.Path)] + ModuleLibs[(m.File,m.Root,module_obj.Arch,m.Path)] =3D [(l.Met= aFile.File,l.MetaFile.Root,l.Arch,l.MetaFile.Path) for l in Libs] self.DataContainer =3D {"DEPS":ModuleLibs} + self.DataContainer =3D {"REFS":libModules} =20 #Platform BuildOptions =20 platform_build_opt =3D PlatformInfo.EdkIIBuildOption =20 diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py b/BaseT= ools/Source/Python/AutoGen/ModuleAutoGenHelper.py index 5186ca1da3e3..c7591253debd 100644 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py @@ -595,14 +595,17 @@ class PlatformInfo(AutoGenInfo): =20 def ApplyLibraryInstance(self,module): alldeps =3D self.DataPipe.Get("DEPS") if alldeps is None: alldeps =3D {} - mod_libs =3D alldeps.get((module.MetaFile.File,module.MetaFile.Roo= t,module.Arch),[]) + mod_libs =3D alldeps.get((module.MetaFile.File,module.MetaFile.Roo= t,module.Arch,module.MetaFile.Path),[]) retVal =3D [] - for (file_path,root,arch) in mod_libs: - retVal.append(self.Wa.BuildDatabase[PathClass(file_path,root),= arch, self.Target,self.ToolChain]) + for (file_path,root,arch,abs_path) in mod_libs: + libMetaFile =3D PathClass(file_path,root) + libMetaFile.OriginalPath =3D PathClass(file_path,root) + libMetaFile.Path =3D abs_path + retVal.append(self.Wa.BuildDatabase[libMetaFile, arch, self.Ta= rget,self.ToolChain]) return retVal =20 ## Parse build_rule.txt in Conf Directory. # # @retval BuildRule object diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools= /Source/Python/AutoGen/PlatformAutoGen.py index 9885c6a3a3bf..2a614e6a7134 100644 --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py @@ -131,10 +131,16 @@ class PlatformAutoGen(AutoGen): =20 self.DataPipe =3D MemoryDataPipe(self.BuildDir) self.DataPipe.FillData(self) =20 return True + def FillData_LibConstPcd(self): + libConstPcd =3D {} + for LibAuto in self.LibraryAutoGenList: + if LibAuto.ConstPcd: + libConstPcd[(LibAuto.MetaFile.File,LibAuto.MetaFile.Root,L= ibAuto.Arch,LibAuto.MetaFile.Path)] =3D LibAuto.ConstPcd + self.DataPipe.DataContainer =3D {"LibConstPcd":libConstPcd} ## hash() operator of PlatformAutoGen # # The platform file path and arch string will be used to represent # hash value of this object # @@ -1080,11 +1086,14 @@ class PlatformAutoGen(AutoGen): @cached_property def GetAllModuleInfo(self,WithoutPcd=3DTrue): ModuleLibs =3D set() for m in self.Platform.Modules: module_obj =3D self.BuildDatabase[m,self.Arch,self.BuildTarget= ,self.ToolChain] - Libs =3D GetModuleLibInstances(module_obj, self.Platform, self= .BuildDatabase, self.Arch,self.BuildTarget,self.ToolChain) + if not bool(module_obj.LibraryClass): + Libs =3D GetModuleLibInstances(module_obj, self.Platform, = self.BuildDatabase, self.Arch,self.BuildTarget,self.ToolChain) + else: + Libs =3D [] ModuleLibs.update( set([(l.MetaFile.File,l.MetaFile.Root,l.Met= aFile.Path,l.MetaFile.BaseName,l.MetaFile.OriginalPath,l.Arch,True) for l i= n Libs])) if WithoutPcd and module_obj.PcdIsDriver: continue ModuleLibs.add((m.File,m.Root,m.Path,m.BaseName,m.OriginalPath= ,module_obj.Arch,bool(module_obj.LibraryClass))) =20 @@ -1335,29 +1344,39 @@ class PlatformAutoGen(AutoGen): else: PlatformModuleOptions =3D {} =20 return ModuleTypeOptions,PlatformModuleOptions =20 + def ModuleGuid(self,Module): + if os.path.basename(Module.MetaFile.File) !=3D os.path.basename(Mo= dule.MetaFile.Path): + # + # Length of GUID is 36 + # + return os.path.basename(Module.MetaFile.Path)[:36] + return Module.Guid @cached_property def UniqueBaseName(self): retVal =3D{} - name_path_map =3D {} + ModuleNameDict =3D {} + UniqueName =3D {} for Module in self._MbList: - name_path_map[Module.BaseName] =3D set() - for Module in self._MbList: - name_path_map[Module.BaseName].add(Module.MetaFile) - for name in name_path_map: - if len(name_path_map[name]) > 1: - guidset =3D set() - for metafile in name_path_map[name]: - m =3D self.BuildDatabase[metafile, self.Arch, self.Bui= ldTarget, self.ToolChain] - retVal[name] =3D '%s_%s' % (name, m.Guid) - guidset.add(m.Guid) - samemodules =3D list(name_path_map[name]) - if len(guidset) > 1: - EdkLogger.error("build", FILE_DUPLICATED, 'Modules= have same BaseName and FILE_GUID:\n' + unique_base_name =3D '%s_%s' % (Module.BaseName,self.ModuleGui= d(Module)) + if unique_base_name not in ModuleNameDict: + ModuleNameDict[unique_base_name] =3D [] + ModuleNameDict[unique_base_name].append(Module.MetaFile) + if Module.BaseName not in UniqueName: + UniqueName[Module.BaseName] =3D set() + UniqueName[Module.BaseName].add((self.ModuleGuid(Module),Modul= e.MetaFile)) + for module_paths in ModuleNameDict.values(): + if len(module_paths) > 1 and len(set(module_paths))>1: + samemodules =3D list(set(module_paths)) + EdkLogger.error("build", FILE_DUPLICATED, 'Modules have sa= me BaseName and FILE_GUID:\n' ' %s\n %s' % (samemodules[0], samemo= dules[1])) + for name in UniqueName: + Guid_Path =3D UniqueName[name] + if len(Guid_Path) > 1: + retVal[name] =3D '%s_%s' % (name,Guid_Path.pop()[0]) return retVal ## Expand * in build option key # # @param Options Options to be expanded # @param ToolDef Use specified ToolDef instead of full version. diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTool= s/Source/Python/AutoGen/WorkspaceAutoGen.py index ab58b21772c3..4ad92653a238 100644 --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py @@ -111,10 +111,12 @@ class WorkspaceAutoGen(AutoGen): self.ProcessModuleFromPdf() self.ProcessPcdType() self.ProcessMixedPcd() self.VerifyPcdsFromFDF() self.CollectAllPcds() + for Pa in self.AutoGenObjectList: + Pa.FillData_LibConstPcd() self.GeneratePkgLevelHash() # # Check PCDs token value conflict in each DEC file. # self._CheckAllPcdsTokenValueConflict() diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 603d3aa6dad4..dc92495f3f08 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -707,11 +707,11 @@ class Build(): self.Fdf =3D BuildOptions.FdfFile self.FdList =3D BuildOptions.RomImage self.FvList =3D BuildOptions.FvImage self.CapList =3D BuildOptions.CapName self.SilentMode =3D BuildOptions.SilentMode - self.ThreadNumber =3D BuildOptions.ThreadNumber + self.ThreadNumber =3D 1 self.SkipAutoGen =3D BuildOptions.SkipAutoGen self.Reparse =3D BuildOptions.Reparse self.SkuId =3D BuildOptions.SkuId if self.SkuId: GlobalData.gSKUID_CMD =3D self.SkuId @@ -812,31 +812,32 @@ class Build(): EdkLogger.info("") os.chdir(self.WorkspaceDir) self.share_data =3D Manager().dict() self.log_q =3D log_q def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_dat= a): - if SkipAutoGen: - return - feedback_q =3D mp.Queue() - file_lock =3D mp.Lock() - auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock,share_data,self.log_q) for _ in range(self.ThreadNumb= er)] - self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q) - self.AutoGenMgr.start() - for w in auto_workers: - w.start() - if PcdMaList is not None: - for PcdMa in PcdMaList: - PcdMa.CreateCodeFile(True) - PcdMa.CreateMakeFile(GenFfsList =3D DataPipe.Get("FfsComma= nd").get((PcdMa.MetaFile.File, PcdMa.Arch),[])) - PcdMa.CreateAsBuiltInf() - for w in auto_workers: - w.join() - rt =3D self.AutoGenMgr.Status - self.AutoGenMgr.kill() - self.AutoGenMgr.join() - self.AutoGenMgr =3D None - return rt + try: + if SkipAutoGen: + return True,0 + feedback_q =3D mp.Queue() + file_lock =3D mp.Lock() + error_event =3D mp.Event() + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,share_data,self.log_q,error_event) for _ in range= (self.ThreadNumber)] + self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q,err= or_event) + self.AutoGenMgr.start() + for w in auto_workers: + w.start() + if PcdMaList is not None: + for PcdMa in PcdMaList: + PcdMa.CreateCodeFile(True) + PcdMa.CreateMakeFile(GenFfsList =3D DataPipe.Get("FfsC= ommand").get((PcdMa.MetaFile.File, PcdMa.Arch),[])) + PcdMa.CreateAsBuiltInf() + + self.AutoGenMgr.join() + rt =3D self.AutoGenMgr.Status + return rt, 0 + except Exception as e: + return False,e.errcode =20 ## Load configuration # # This method will parse target.txt and get the build configurations. # @@ -880,23 +881,10 @@ class Build(): ToolChainFamily.append(TAB_COMPILER_MSFT) else: ToolChainFamily.append(ToolDefinition[TAB_TOD_DEFINES_FAMI= LY][Tool]) self.ToolChainFamily =3D ToolChainFamily =20 - if self.ThreadNumber is None: - self.ThreadNumber =3D self.TargetTxt.TargetTxtDictionary[TAB_T= AT_DEFINES_MAX_CONCURRENT_THREAD_NUMBER] - if self.ThreadNumber =3D=3D '': - self.ThreadNumber =3D 0 - else: - self.ThreadNumber =3D int(self.ThreadNumber, 0) - - if self.ThreadNumber =3D=3D 0: - try: - self.ThreadNumber =3D multiprocessing.cpu_count() - except (ImportError, NotImplementedError): - self.ThreadNumber =3D 1 - if not self.PlatformFile: PlatformFile =3D self.TargetTxt.TargetTxtDictionary[TAB_TAT_DE= FINES_ACTIVE_PLATFORM] if not PlatformFile: # Try to find one in current directory WorkingDirectory =3D os.getcwd() @@ -911,10 +899,11 @@ class Build(): EdkLogger.error("build", RESOURCE_NOT_AVAILABLE, ExtraData=3D"No active platform specif= ied in target.txt or command line! Nothing can be built.\n") =20 self.PlatformFile =3D PathClass(NormFile(PlatformFile, self.Wo= rkspaceDir), self.WorkspaceDir) =20 + self.ThreadNumber =3D ThreadNum() ## Initialize build configuration # # This method will parse DSC file and merge the configurations from # command line and target.txt, then get the final build configuratio= ns. # @@ -1213,12 +1202,16 @@ class Build(): =20 AutoGenObject.DataPipe.DataContainer =3D {"FfsCommand":FfsComm= and} self.Progress.Start("Generating makefile and code") data_pipe_file =3D os.path.join(AutoGenObject.BuildDir, "Globa= lVar_%s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch)) AutoGenObject.DataPipe.dump(data_pipe_file) - autogen_rt =3D self.StartAutoGen(mqueue, AutoGenObject.DataPip= e, self.SkipAutoGen, PcdMaList,self.share_data) + autogen_rt,errorcode =3D self.StartAutoGen(mqueue, AutoGenObje= ct.DataPipe, self.SkipAutoGen, PcdMaList,self.share_data) self.Progress.Stop("done!") + if not autogen_rt: + self.AutoGenMgr.TerminateWorkers() + self.AutoGenMgr.join(0.1) + raise FatalError(errorcode) return autogen_rt else: # always recreate top/platform makefile when clean, just in ca= se of inconsistency AutoGenObject.CreateCodeFile(False) AutoGenObject.CreateMakeFile(False) @@ -1717,10 +1710,11 @@ class Build(): Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile,Pa.DataPipe) if Ma is None: continue if Ma.PcdIsDriver: Ma.PlatformInfo =3D Pa + Ma.Workspace =3D Wa PcdMaList.append(Ma) self.BuildModules.append(Ma) self._BuildPa(self.Target, Pa, FfsCommand=3DCmdListDic= t,PcdMaList=3DPcdMaList) =20 # Create MAP file when Load Fix Address is enabled. @@ -2045,14 +2039,15 @@ class Build(): mqueue =3D mp.Queue() for m in Pa.GetAllModuleInfo: mqueue.put(m) data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVa= r_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) - autogen_rt =3D self.StartAutoGen(mqueue, Pa.DataPipe, = self.SkipAutoGen, PcdMaList,self.share_data) - + autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa= .DataPipe, self.SkipAutoGen, PcdMaList,self.share_data) if not autogen_rt: - return + self.AutoGenMgr.TerminateWorkers() + self.AutoGenMgr.join(0.1) + raise FatalError(errorcode) self.AutoGenTime +=3D int(round((time.time() - AutoGenStar= t))) self.Progress.Stop("done!") for Arch in Wa.ArchList: MakeStart =3D time.time() for Ma in BuildModules: @@ -2286,20 +2281,35 @@ def LogBuildTime(Time): TimeDurStr =3D time.strftime("%H:%M:%S", TimeDur) return TimeDurStr else: return None =20 +def ThreadNum(): + ThreadNumber =3D BuildOption.ThreadNumber + if ThreadNumber is None: + ThreadNumber =3D TargetTxt.TargetTxtDictionary[TAB_TAT_DEFINES_MAX= _CONCURRENT_THREAD_NUMBER] + if ThreadNumber =3D=3D '': + ThreadNumber =3D 0 + else: + ThreadNumber =3D int(ThreadNumber, 0) + + if ThreadNumber =3D=3D 0: + try: + ThreadNumber =3D multiprocessing.cpu_count() + except (ImportError, NotImplementedError): + ThreadNumber =3D 1 + return ThreadNumber ## Tool entrance method # # This method mainly dispatch specific methods per the command line option= s. # If no error found, return zero value so the caller of this tool can know # if it's executed successfully or not. # # @retval 0 Tool was successful # @retval 1 Tool failed # -LogQMaxSize =3D 60 +LogQMaxSize =3D ThreadNum() * 10 def Main(): StartTime =3D time.time() =20 # # Create a log Queue @@ -2432,13 +2442,11 @@ def Main(): else: EdkLogger.error(X.ToolName, FORMAT_INVALID, File=3DX.FileName,= Line=3DX.LineNumber, ExtraData=3DX.Message, RaiseError=3DFalse) ReturnCode =3D FORMAT_INVALID except KeyboardInterrupt: if MyBuild is not None: - if MyBuild.AutoGenMgr: - MyBuild.AutoGenMgr.TerminateWorkers() - MyBuild.AutoGenMgr.kill() + # for multi-thread build exits safely MyBuild.Relinquish() ReturnCode =3D ABORT_ERROR if Option is not None and Option.debug is not None: EdkLogger.quiet("(Python %s on %s) " % (platform.python_versio= n(), sys.platform) + traceback.format_exc()) @@ -2495,10 +2503,11 @@ def Main(): Log_Agent.kill() Log_Agent.join() return ReturnCode =20 if __name__ =3D=3D '__main__': + mp.set_start_method('spawn') r =3D Main() ## 0-127 is a safe return range, and 1 is a standard default error if r < 0 or r > 127: r =3D 1 sys.exit(r) =20 --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#44502): https://edk2.groups.io/g/devel/message/44502 Mute This Topic: https://groups.io/mt/32640238/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-