From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43908+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43908+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430481; cv=none; d=zoho.com; s=zohoarc; b=E7IS+DFC0cJ65G94i7QN3SLndNer0MQuzlx+FveNOGvAmbVs+1m6vApKGzDS1jXa9+Izch3nJKYJcQr5gPg+piGAYu1HOtKpRSW8RUK34D1Ygz9Y+1pTyyjma7QNCCx8mawgX+0Ke3pVbS8lgdwgTBaTum4szCV03iYHTUul2c8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430481; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=PCfu/tDVe3pqddBn0TN/No289RL4TfG33hueaFkLSkk=; b=gBNDqKzTTIM3Ud7UsKRksvMqyrzdvdH3GM9nPC5oHaVfePNjl+pJgjL8geeKoFjB1h+LDY0FTkpaEMr0b3JrdSc7uFOTHK88Oljg8O963I80xi+lwcFUvt3tsLe3SXXVSPhshgEj44AIcuZXoE0e5rRmDbZvQPjcsYKRYKdOCGk= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43908+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430481383230.99673914963228; Wed, 17 Jul 2019 23:14:41 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:40 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319543917" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:38 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 1/9] BaseTools: Singleton the object to handle build conf file Date: Thu, 18 Jul 2019 14:14:15 +0800 Message-Id: <20190718061423.30612-2-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430481; bh=yw7In1ZcavJVwnQU1G4VmYeqzO5iHhX9VkUgyzGS8FY=; h=Cc:Date:From:Reply-To:Subject:To; b=cn2EzfdmEBYbwEuNMU0K7FtCnByi/+y6IpMneBh6MXB8jYr+3CGxbHipSaQeS0Zyk24 PuvbpEhlaWY4d+au/2oYy0OBLvHjlX0AavxqAozDLxdd5RD7pAHMl8R9IZyBECvB7LI/o gvN1E1pQcD5tgFd4WWHwoCOgkODeHjbY2PI= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 The build config files are target.txt, build rule, tooldef During a build, the config is not changed, so the object to handle them need to be singleton. Cc: Liming Gao Signed-off-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGen.py | 33 ++---------- .../Source/Python/AutoGen/BuildEngine.py | 22 ++++++++ .../Python/Common/TargetTxtClassObject.py | 2 + .../Python/Common/ToolDefClassObject.py | 6 ++- BaseTools/Source/Python/GenFds/GenFds.py | 4 +- .../Python/GenFds/GenFdsGlobalVariable.py | 54 ++++++++----------- .../Source/Python/Workspace/DscBuildData.py | 8 +-- BaseTools/Source/Python/build/build.py | 29 +++------- 8 files changed, 62 insertions(+), 96 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index 2df055a109f7..c5b3fbb0a87f 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -22,11 +22,12 @@ from . import GenC from . import GenMake from . import GenDepex from io import BytesIO =20 from .StrGather import * -from .BuildEngine import BuildRule +from .BuildEngine import BuildRuleObj as BuildRule +from .BuildEngine import gDefaultBuildRuleFile,AutoGenReqBuildRuleVerNum import shutil from Common.LongFilePathSupport import CopyLongFilePath from Common.BuildToolError import * from Common.DataType import * from Common.Misc import * @@ -76,16 +77,10 @@ gEfiVarStoreGuidPattern =3D re.compile("\s*guid\s*=3D\s= *({.*?{.*?}\s*})") =20 ## Mapping Makefile type gMakeTypeMap =3D {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"} =20 =20 -## Build rule configuration file -gDefaultBuildRuleFile =3D 'build_rule.txt' - -## Build rule default version -AutoGenReqBuildRuleVerNum =3D "0.1" - ## default file name for AutoGen gAutoGenCodeFileName =3D "AutoGen.c" gAutoGenHeaderFileName =3D "AutoGen.h" gAutoGenStringFileName =3D "%(module_name)sStrDefs.h" gAutoGenStringFormFileName =3D "%(module_name)sStrDefs.hpk" @@ -1970,32 +1965,10 @@ class PlatformAutoGen(AutoGen): ## Return the build options specific for EDKII modules in this platform @cached_property def EdkIIBuildOption(self): return self._ExpandBuildOption(self.Platform.BuildOptions, EDKII_N= AME) =20 - ## Parse build_rule.txt in Conf Directory. - # - # @retval BuildRule object - # - @cached_property - def BuildRule(self): - BuildRuleFile =3D None - if TAB_TAT_DEFINES_BUILD_RULE_CONF in self.Workspace.TargetTxt.Tar= getTxtDictionary: - BuildRuleFile =3D self.Workspace.TargetTxt.TargetTxtDictionary= [TAB_TAT_DEFINES_BUILD_RULE_CONF] - if not BuildRuleFile: - BuildRuleFile =3D gDefaultBuildRuleFile - RetVal =3D BuildRule(BuildRuleFile) - if RetVal._FileVersion =3D=3D "": - RetVal._FileVersion =3D AutoGenReqBuildRuleVerNum - else: - if RetVal._FileVersion < AutoGenReqBuildRuleVerNum : - # If Build Rule's version is less than the version number = required by the tools, halting the build. - EdkLogger.error("build", AUTOGEN_ERROR, - ExtraData=3D"The version number [%s] of bu= ild_rule.txt is less than the version number required by the AutoGen.(the m= inimum required version number is [%s])"\ - % (RetVal._FileVersion, AutoGenReqBuildRu= leVerNum)) - return RetVal - ## Summarize the packages used by modules in this platform @cached_property def PackageList(self): RetVal =3D set() for La in self.LibraryAutoGenList: @@ -3149,11 +3122,11 @@ class ModuleAutoGen(AutoGen): return RetVal =20 @cached_property def BuildRules(self): RetVal =3D {} - BuildRuleDatabase =3D self.PlatformInfo.BuildRule + BuildRuleDatabase =3D BuildRule for Type in BuildRuleDatabase.FileTypeList: #first try getting build rule by BuildRuleFamily RuleObject =3D BuildRuleDatabase[Type, self.BuildType, self.Ar= ch, self.BuildRuleFamily] if not RuleObject: # build type is always module type, but ... diff --git a/BaseTools/Source/Python/AutoGen/BuildEngine.py b/BaseTools/Sou= rce/Python/AutoGen/BuildEngine.py index 14e61140e7ba..bb9153447793 100644 --- a/BaseTools/Source/Python/AutoGen/BuildEngine.py +++ b/BaseTools/Source/Python/AutoGen/BuildEngine.py @@ -18,10 +18,13 @@ from Common.LongFilePathSupport import OpenLongFilePath= as open from Common.GlobalData import * from Common.BuildToolError import * from Common.Misc import tdict, PathClass from Common.StringUtils import NormPath from Common.DataType import * +from Common.TargetTxtClassObject import TargetTxt +gDefaultBuildRuleFile =3D 'build_rule.txt' +AutoGenReqBuildRuleVerNum =3D '0.1' =20 import Common.EdkLogger as EdkLogger =20 ## Convert file type to file list macro name # @@ -581,10 +584,29 @@ class BuildRule: _ExtraDependency : ParseCommonSubSection, _Command : ParseCommonSubSection, _UnknownSection : SkipSection, } =20 +def GetBuildRule(): + BuildRuleFile =3D None + if TAB_TAT_DEFINES_BUILD_RULE_CONF in TargetTxt.TargetTxtDictionary: + BuildRuleFile =3D TargetTxt.TargetTxtDictionary[TAB_TAT_DEFINES_BU= ILD_RULE_CONF] + if not BuildRuleFile: + BuildRuleFile =3D gDefaultBuildRuleFile + RetVal =3D BuildRule(BuildRuleFile) + if RetVal._FileVersion =3D=3D "": + RetVal._FileVersion =3D AutoGenReqBuildRuleVerNum + else: + if RetVal._FileVersion < AutoGenReqBuildRuleVerNum : + # If Build Rule's version is less than the version number requ= ired by the tools, halting the build. + EdkLogger.error("build", AUTOGEN_ERROR, + ExtraData=3D"The version number [%s] of build_= rule.txt is less than the version number required by the AutoGen.(the minim= um required version number is [%s])"\ + % (RetVal._FileVersion, AutoGenReqBuildRuleVe= rNum)) + return RetVal + +BuildRuleObj =3D GetBuildRule() + # This acts like the main() function for the script, unless it is 'import'= ed into another # script. if __name__ =3D=3D '__main__': import sys EdkLogger.Initialize() diff --git a/BaseTools/Source/Python/Common/TargetTxtClassObject.py b/BaseT= ools/Source/Python/Common/TargetTxtClassObject.py index 9d7673b41bb5..79a5acc01074 100644 --- a/BaseTools/Source/Python/Common/TargetTxtClassObject.py +++ b/BaseTools/Source/Python/Common/TargetTxtClassObject.py @@ -144,10 +144,12 @@ class TargetTxtClassObject(object): def TargetTxtDict(ConfDir): Target =3D TargetTxtClassObject() Target.LoadTargetTxtFile(os.path.normpath(os.path.join(ConfDir, gDefau= ltTargetTxtFile))) return Target =20 +TargetTxt =3D TargetTxtDict(os.path.join(os.getenv("WORKSPACE"),"Conf")) + ## # # This acts like the main() function for the script, unless it is 'import'= ed into another # script. # diff --git a/BaseTools/Source/Python/Common/ToolDefClassObject.py b/BaseToo= ls/Source/Python/Common/ToolDefClassObject.py index 4fa364942cad..063fa005840a 100644 --- a/BaseTools/Source/Python/Common/ToolDefClassObject.py +++ b/BaseTools/Source/Python/Common/ToolDefClassObject.py @@ -12,11 +12,11 @@ from __future__ import absolute_import import Common.LongFilePathOs as os import re from . import EdkLogger =20 from .BuildToolError import * -from Common.TargetTxtClassObject import TargetTxtDict +from Common.TargetTxtClassObject import TargetTxt from Common.LongFilePathSupport import OpenLongFilePath as open from Common.Misc import PathClass from Common.StringUtils import NormPath import Common.GlobalData as GlobalData from Common import GlobalData @@ -261,11 +261,11 @@ class ToolDefClassObject(object): # @param ConfDir: Conf dir # # @retval ToolDef An instance of ToolDefClassObject() with loaded tools_de= f.txt # def ToolDefDict(ConfDir): - Target =3D TargetTxtDict(ConfDir) + Target =3D TargetTxt ToolDef =3D ToolDefClassObject() if TAB_TAT_DEFINES_TOOL_CHAIN_CONF in Target.TargetTxtDictionary: ToolsDefFile =3D Target.TargetTxtDictionary[TAB_TAT_DEFINES_TOOL_C= HAIN_CONF] if ToolsDefFile: ToolDef.LoadToolDefFile(os.path.normpath(ToolsDefFile)) @@ -273,10 +273,12 @@ def ToolDefDict(ConfDir): ToolDef.LoadToolDefFile(os.path.normpath(os.path.join(ConfDir,= gDefaultToolsDefFile))) else: ToolDef.LoadToolDefFile(os.path.normpath(os.path.join(ConfDir, gDe= faultToolsDefFile))) return ToolDef =20 +ToolDef =3D ToolDefDict((os.path.join(os.getenv("WORKSPACE"),"Conf"))) + ## # # This acts like the main() function for the script, unless it is 'import'= ed into another # script. # diff --git a/BaseTools/Source/Python/GenFds/GenFds.py b/BaseTools/Source/Py= thon/GenFds/GenFds.py index 5888997761bb..51943411ad1f 100644 --- a/BaseTools/Source/Python/GenFds/GenFds.py +++ b/BaseTools/Source/Python/GenFds/GenFds.py @@ -18,11 +18,11 @@ from glob import glob from struct import unpack from linecache import getlines from io import BytesIO =20 import Common.LongFilePathOs as os -from Common.TargetTxtClassObject import TargetTxtClassObject +from Common.TargetTxtClassObject import TargetTxt from Common.DataType import * import Common.GlobalData as GlobalData from Common import EdkLogger from Common.StringUtils import NormPath from Common.Misc import DirCache, PathClass, GuidStructureStringToGuidStri= ng @@ -205,12 +205,10 @@ def GenFdsApi(FdsCommandDict, WorkSpaceDataBase=3DNon= e): GenFdsGlobalVariable.ConfDir =3D ConfDirectoryPath if not GlobalData.gConfDirectory: GlobalData.gConfDirectory =3D GenFdsGlobalVariable.ConfDir BuildConfigurationFile =3D os.path.normpath(os.path.join(ConfDirec= toryPath, "target.txt")) if os.path.isfile(BuildConfigurationFile) =3D=3D True: - TargetTxt =3D TargetTxtClassObject() - TargetTxt.LoadTargetTxtFile(BuildConfigurationFile) # if no build target given in command line, get it from target= .txt if not GenFdsGlobalVariable.TargetName: BuildTargetList =3D TargetTxt.TargetTxtDictionary[TAB_TAT_= DEFINES_TARGET] if len(BuildTargetList) !=3D 1: EdkLogger.error("GenFds", OPTION_VALUE_INVALID, ExtraD= ata=3D"Only allows one instance for Target.") diff --git a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py b/BaseT= ools/Source/Python/GenFds/GenFdsGlobalVariable.py index c9c476cf6154..f49af9371b8d 100644 --- a/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py +++ b/BaseTools/Source/Python/GenFds/GenFdsGlobalVariable.py @@ -20,13 +20,13 @@ from array import array =20 from Common.BuildToolError import COMMAND_FAILURE,GENFDS_ERROR from Common import EdkLogger from Common.Misc import SaveFileOnChange =20 -from Common.TargetTxtClassObject import TargetTxtClassObject -from Common.ToolDefClassObject import ToolDefClassObject, ToolDefDict -from AutoGen.BuildEngine import BuildRule +from Common.TargetTxtClassObject import TargetTxt +from Common.ToolDefClassObject import ToolDef +from AutoGen.BuildEngine import BuildRuleObj import Common.DataType as DataType from Common.Misc import PathClass from Common.LongFilePathSupport import OpenLongFilePath as open from Common.MultipleWorkspace import MultipleWorkspace as mws import Common.GlobalData as GlobalData @@ -93,35 +93,25 @@ class GenFdsGlobalVariable: # @staticmethod def _LoadBuildRule(): if GenFdsGlobalVariable.__BuildRuleDatabase: return GenFdsGlobalVariable.__BuildRuleDatabase - BuildConfigurationFile =3D os.path.normpath(os.path.join(GenFdsGlo= balVariable.ConfDir, "target.txt")) - TargetTxt =3D TargetTxtClassObject() - if os.path.isfile(BuildConfigurationFile) =3D=3D True: - TargetTxt.LoadTargetTxtFile(BuildConfigurationFile) - if DataType.TAB_TAT_DEFINES_BUILD_RULE_CONF in TargetTxt.Targe= tTxtDictionary: - BuildRuleFile =3D TargetTxt.TargetTxtDictionary[DataType.T= AB_TAT_DEFINES_BUILD_RULE_CONF] - if not BuildRuleFile: - BuildRuleFile =3D 'Conf/build_rule.txt' - GenFdsGlobalVariable.__BuildRuleDatabase =3D BuildRule(BuildRu= leFile) - ToolDefinitionFile =3D TargetTxt.TargetTxtDictionary[DataType.= TAB_TAT_DEFINES_TOOL_CHAIN_CONF] - if ToolDefinitionFile =3D=3D '': - ToolDefinitionFile =3D "Conf/tools_def.txt" - if os.path.isfile(ToolDefinitionFile): - ToolDef =3D ToolDefClassObject() - ToolDef.LoadToolDefFile(ToolDefinitionFile) - ToolDefinition =3D ToolDef.ToolsDefTxtDatabase - if DataType.TAB_TOD_DEFINES_BUILDRULEFAMILY in ToolDefinit= ion \ - and GenFdsGlobalVariable.ToolChainTag in ToolDefinition= [DataType.TAB_TOD_DEFINES_BUILDRULEFAMILY] \ - and ToolDefinition[DataType.TAB_TOD_DEFINES_BUILDRULEFA= MILY][GenFdsGlobalVariable.ToolChainTag]: - GenFdsGlobalVariable.BuildRuleFamily =3D ToolDefinitio= n[DataType.TAB_TOD_DEFINES_BUILDRULEFAMILY][GenFdsGlobalVariable.ToolChainT= ag] + GenFdsGlobalVariable.__BuildRuleDatabase =3D BuildRuleObj + ToolDefinitionFile =3D TargetTxt.TargetTxtDictionary[DataType.TAB_= TAT_DEFINES_TOOL_CHAIN_CONF] + if ToolDefinitionFile =3D=3D '': + ToolDefinitionFile =3D "Conf/tools_def.txt" + if os.path.isfile(ToolDefinitionFile): + ToolDefinition =3D ToolDef.ToolsDefTxtDatabase + if DataType.TAB_TOD_DEFINES_BUILDRULEFAMILY in ToolDefinition \ + and GenFdsGlobalVariable.ToolChainTag in ToolDefinition[Dat= aType.TAB_TOD_DEFINES_BUILDRULEFAMILY] \ + and ToolDefinition[DataType.TAB_TOD_DEFINES_BUILDRULEFAMILY= ][GenFdsGlobalVariable.ToolChainTag]: + GenFdsGlobalVariable.BuildRuleFamily =3D ToolDefinition[Da= taType.TAB_TOD_DEFINES_BUILDRULEFAMILY][GenFdsGlobalVariable.ToolChainTag] =20 - if DataType.TAB_TOD_DEFINES_FAMILY in ToolDefinition \ - and GenFdsGlobalVariable.ToolChainTag in ToolDefinition= [DataType.TAB_TOD_DEFINES_FAMILY] \ - and ToolDefinition[DataType.TAB_TOD_DEFINES_FAMILY][Gen= FdsGlobalVariable.ToolChainTag]: - GenFdsGlobalVariable.ToolChainFamily =3D ToolDefinitio= n[DataType.TAB_TOD_DEFINES_FAMILY][GenFdsGlobalVariable.ToolChainTag] + if DataType.TAB_TOD_DEFINES_FAMILY in ToolDefinition \ + and GenFdsGlobalVariable.ToolChainTag in ToolDefinition[Dat= aType.TAB_TOD_DEFINES_FAMILY] \ + and ToolDefinition[DataType.TAB_TOD_DEFINES_FAMILY][GenFdsG= lobalVariable.ToolChainTag]: + GenFdsGlobalVariable.ToolChainFamily =3D ToolDefinition[Da= taType.TAB_TOD_DEFINES_FAMILY][GenFdsGlobalVariable.ToolChainTag] return GenFdsGlobalVariable.__BuildRuleDatabase =20 ## GetBuildRules # @param Inf: object of InfBuildData # @param Arch: current arch @@ -834,11 +824,11 @@ class GenFdsGlobalVariable: # @param KeyStringList Filter for inputs of section generation # @param CurrentArchList Arch list # @param NameGuid The Guid name # def FindExtendTool(KeyStringList, CurrentArchList, NameGuid): - ToolDb =3D ToolDefDict(GenFdsGlobalVariable.ConfDir).ToolsDefTxtDataba= se + ToolDb =3D ToolDef.ToolsDefTxtDatabase # if user not specify filter, try to deduce it from global data. if KeyStringList is None or KeyStringList =3D=3D []: Target =3D GenFdsGlobalVariable.TargetName ToolChain =3D GenFdsGlobalVariable.ToolChainTag if ToolChain not in ToolDb['TOOL_CHAIN_TAG']: @@ -850,19 +840,19 @@ def FindExtendTool(KeyStringList, CurrentArchList, Na= meGuid): =20 if GenFdsGlobalVariable.GuidToolDefinition: if NameGuid in GenFdsGlobalVariable.GuidToolDefinition: return GenFdsGlobalVariable.GuidToolDefinition[NameGuid] =20 - ToolDefinition =3D ToolDefDict(GenFdsGlobalVariable.ConfDir).ToolsDefT= xtDictionary + ToolDefinition =3D ToolDef.ToolsDefTxtDictionary ToolPathTmp =3D None ToolOption =3D None ToolPathKey =3D None ToolOptionKey =3D None KeyList =3D None - for ToolDef in ToolDefinition.items(): - if NameGuid.lower() =3D=3D ToolDef[1].lower(): - KeyList =3D ToolDef[0].split('_') + for tool_def in ToolDefinition.items(): + if NameGuid.lower() =3D=3D tool_def[1].lower(): + KeyList =3D tool_def[0].split('_') Key =3D KeyList[0] + \ '_' + \ KeyList[1] + \ '_' + \ KeyList[2] diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/= Source/Python/Workspace/DscBuildData.py index 985f8775259d..e7ec2aba57d2 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -17,12 +17,12 @@ from Common.StringUtils import * from Common.DataType import * from Common.Misc import * from types import * from Common.Expression import * from CommonDataClass.CommonClass import SkuInfoClass -from Common.TargetTxtClassObject import TargetTxtClassObject -from Common.ToolDefClassObject import ToolDefClassObject +from Common.TargetTxtClassObject import TargetTxt +from Common.ToolDefClassObject import ToolDef from .MetaDataTable import * from .MetaFileTable import * from .MetaFileParser import * =20 from .WorkspaceCommon import GetDeclaredPcd @@ -3260,19 +3260,15 @@ class DscBuildData(PlatformBuildClassObject): @property def ToolChainFamily(self): self._ToolChainFamily =3D TAB_COMPILER_MSFT BuildConfigurationFile =3D os.path.normpath(os.path.join(GlobalDat= a.gConfDirectory, "target.txt")) if os.path.isfile(BuildConfigurationFile) =3D=3D True: - TargetTxt =3D TargetTxtClassObject() - TargetTxt.LoadTargetTxtFile(BuildConfigurationFile) ToolDefinitionFile =3D TargetTxt.TargetTxtDictionary[DataType.= TAB_TAT_DEFINES_TOOL_CHAIN_CONF] if ToolDefinitionFile =3D=3D '': ToolDefinitionFile =3D "tools_def.txt" ToolDefinitionFile =3D os.path.normpath(mws.join(self.Work= spaceDir, 'Conf', ToolDefinitionFile)) if os.path.isfile(ToolDefinitionFile) =3D=3D True: - ToolDef =3D ToolDefClassObject() - ToolDef.LoadToolDefFile(ToolDefinitionFile) ToolDefinition =3D ToolDef.ToolsDefTxtDatabase if TAB_TOD_DEFINES_FAMILY not in ToolDefinition \ or self._Toolchain not in ToolDefinition[TAB_TOD_DEFINE= S_FAMILY] \ or not ToolDefinition[TAB_TOD_DEFINES_FAMILY][self._Too= lchain]: self._ToolChainFamily =3D TAB_COMPILER_MSFT diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index d6006b651f77..cce091c4f8b5 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -28,12 +28,12 @@ import threading from optparse import OptionParser from subprocess import * from Common import Misc as Utils =20 from Common.LongFilePathSupport import OpenLongFilePath as open -from Common.TargetTxtClassObject import TargetTxtClassObject -from Common.ToolDefClassObject import ToolDefClassObject +from Common.TargetTxtClassObject import TargetTxt +from Common.ToolDefClassObject import ToolDef from Common.DataType import * from Common.BuildVersion import gBUILD_VERSION from AutoGen.AutoGen import * from Common.BuildToolError import * from Workspace.WorkspaceDatabase import WorkspaceDatabase @@ -714,12 +714,12 @@ class Build(): if self.SkuId: GlobalData.gSKUID_CMD =3D self.SkuId self.ConfDirectory =3D BuildOptions.ConfDirectory self.SpawnMode =3D True self.BuildReport =3D BuildReport(BuildOptions.ReportFile, Build= Options.ReportType) - self.TargetTxt =3D TargetTxtClassObject() - self.ToolDef =3D ToolDefClassObject() + self.TargetTxt =3D TargetTxt + self.ToolDef =3D ToolDef self.AutoGenTime =3D 0 self.MakeTime =3D 0 self.GenFdsTime =3D 0 GlobalData.BuildOptionPcd =3D BuildOptions.OptionPcd if BuildO= ptions.OptionPcd else [] #Set global flag for build mode @@ -813,12 +813,12 @@ class Build(): EdkLogger.quiet("%-16s =3D %s" % ("PREBUILD", self.Prebuild)) if self.Postbuild: EdkLogger.quiet("%-16s =3D %s" % ("POSTBUILD", self.Postbuild)) if self.Prebuild: self.LaunchPrebuild() - self.TargetTxt =3D TargetTxtClassObject() - self.ToolDef =3D ToolDefClassObject() + self.TargetTxt =3D TargetTxt + self.ToolDef =3D ToolDef if not (self.LaunchPrebuildFlag and os.path.exists(self.PlatformBu= ildPath)): self.InitBuild() =20 EdkLogger.info("") os.chdir(self.WorkspaceDir) @@ -826,27 +826,10 @@ class Build(): ## Load configuration # # This method will parse target.txt and get the build configurations. # def LoadConfiguration(self): - # - # Check target.txt and tools_def.txt and Init them - # - BuildConfigurationFile =3D os.path.normpath(os.path.join(GlobalDat= a.gConfDirectory, gBuildConfiguration)) - if os.path.isfile(BuildConfigurationFile) =3D=3D True: - StatusCode =3D self.TargetTxt.LoadTargetTxtFile(BuildConfigura= tionFile) - - ToolDefinitionFile =3D self.TargetTxt.TargetTxtDictionary[TAB_= TAT_DEFINES_TOOL_CHAIN_CONF] - if ToolDefinitionFile =3D=3D '': - ToolDefinitionFile =3D gToolsDefinition - ToolDefinitionFile =3D os.path.normpath(mws.join(self.Work= spaceDir, 'Conf', ToolDefinitionFile)) - if os.path.isfile(ToolDefinitionFile) =3D=3D True: - StatusCode =3D self.ToolDef.LoadToolDefFile(ToolDefinition= File) - else: - EdkLogger.error("build", FILE_NOT_FOUND, ExtraData=3DToolD= efinitionFile) - else: - EdkLogger.error("build", FILE_NOT_FOUND, ExtraData=3DBuildConf= igurationFile) =20 # if no ARCH given in command line, get it from target.txt if not self.ArchList: self.ArchList =3D self.TargetTxt.TargetTxtDictionary[TAB_TAT_D= EFINES_TARGET_ARCH] self.ArchList =3D tuple(self.ArchList) --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43908): https://edk2.groups.io/g/devel/message/43908 Mute This Topic: https://groups.io/mt/32512452/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43909+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43909+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430483; cv=none; d=zoho.com; s=zohoarc; b=Qplf48+UQRnsYAqulM8ts/zH7YwYT5jSYsjAtYeS9wO22nIwAGvsdeo0rVnOTsbwCXuVgjE8Ncn/ocaXt7utiUOiMZ7C3tnCp4a1J/66UEic/0PkAqP84FoPrUdL3i1pKT0FiCMMMOINdNvUWu8SWm5aTGFsOLlrQ1kIC4t9GM8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430483; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=FOmaWWvqPNB0O812f0aedOzfcX1LmO3f8WUS/9Jd8bY=; b=CWoHGSgOR9cC3/Cu/2WmMWZ+howdH3uz3Wz6lGulwENNiShJcr8AQlj0YFoveEe877VWAs+ncB4DkBxCGGG05nuvX/f1Ld5shNqIWcTPrzHSP0ao3u0evGNylTsftxJSIB66hiFrtfyTLc6IJO3r6iRmEGeu/tN5lGi5hb7Wva8= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43909+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430483132615.0098465087224; Wed, 17 Jul 2019 23:14:43 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:42 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319543932" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:40 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 2/9] BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions Date: Thu, 18 Jul 2019 14:14:16 +0800 Message-Id: <20190718061423.30612-3-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430482; bh=MhM/zet9HzexV/aTkWrPl4zTwn0Rui1kxmPTkDvv6wI=; h=Cc:Date:From:Reply-To:Subject:To; b=S4bMQmKXW1PyhSSXge7b9wTdpML/lSXtglcxTpbm571LzvseJARK12et6Ion0eE2Z7P YlDMrnMVOR0/g1rAU1w7XdEedgroJWptZ/DK8x5Dq70g3ggDB2/bRxiC+tw+qBo1y+fEC jDmdM7EzZR/v2v6HEWJPEB48VZrg+Oe7F2I= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 The WorkspaceAutoGen.__InitWorker function is too long, it's hard to read and understand. This patch is to separate the __InitWorker into multiple small ones. Cc: Liming Gao Signed-off-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGen.py | 247 +++++++++++++-------- 1 file changed, 152 insertions(+), 95 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index c5b3fbb0a87f..9e06bb942126 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -333,13 +333,58 @@ class WorkspaceAutoGen(AutoGen): self._GuidDict =3D {} =20 # there's many relative directory operations, so ... os.chdir(self.WorkspaceDir) =20 + self.MergeArch() + self.ValidateBuildTarget() + + EdkLogger.info("") + if self.ArchList: + EdkLogger.info('%-16s =3D %s' % ("Architecture(s)", ' '.join(s= elf.ArchList))) + EdkLogger.info('%-16s =3D %s' % ("Build target", self.BuildTarget)) + EdkLogger.info('%-16s =3D %s' % ("Toolchain", self.ToolChain)) + + EdkLogger.info('\n%-24s =3D %s' % ("Active Platform", self.Platfor= m)) + if BuildModule: + EdkLogger.info('%-24s =3D %s' % ("Active Module", BuildModule)) + + if self.FdfFile: + EdkLogger.info('%-24s =3D %s' % ("Flash Image Definition", sel= f.FdfFile)) + + EdkLogger.verbose("\nFLASH_DEFINITION =3D %s" % self.FdfFile) + + if Progress: + Progress.Start("\nProcessing meta-data") # - # Merge Arch + # Mark now build in AutoGen Phase # + GlobalData.gAutoGenPhase =3D True + self.ProcessModuleFromPdf() + self.ProcessPcdType() + self.ProcessMixedPcd() + self.GetPcdsFromFDF() + self.CollectAllPcds() + self.GeneratePkgLevelHash() + # + # Check PCDs token value conflict in each DEC file. + # + self._CheckAllPcdsTokenValueConflict() + # + # Check PCD type and definition between DSC and DEC + # + self._CheckPcdDefineAndType() + + self.CreateBuildOptionsFile() + self.CreatePcdTokenNumberFile() + self.CreateModuleHashInfo() + GlobalData.gAutoGenPhase =3D False + + # + # Merge Arch + # + def MergeArch(self): if not self.ArchList: ArchList =3D set(self.Platform.SupArchList) else: ArchList =3D set(self.ArchList) & set(self.Platform.SupArchLis= t) if not ArchList: @@ -349,57 +394,49 @@ class WorkspaceAutoGen(AutoGen): SkippedArchList =3D set(self.ArchList).symmetric_difference(se= t(self.Platform.SupArchList)) EdkLogger.verbose("\nArch [%s] is ignored because the platform= supports [%s] only!" % (" ".join(SkippedArchList), " ".join(self.= Platform.SupArchList))) self.ArchList =3D tuple(ArchList) =20 - # Validate build target + # Validate build target + def ValidateBuildTarget(self): if self.BuildTarget not in self.Platform.BuildTargets: EdkLogger.error("build", PARAMETER_INVALID, ExtraData=3D"Build target [%s] is not supporte= d by the platform. [Valid target: %s]" % (self.BuildTarget, " ".join(self.P= latform.BuildTargets))) - - - # parse FDF file to get PCDs in it, if any + @cached_property + def FdfProfile(self): if not self.FdfFile: self.FdfFile =3D self.Platform.FlashDefinition =20 - EdkLogger.info("") - if self.ArchList: - EdkLogger.info('%-16s =3D %s' % ("Architecture(s)", ' '.join(s= elf.ArchList))) - EdkLogger.info('%-16s =3D %s' % ("Build target", self.BuildTarget)) - EdkLogger.info('%-16s =3D %s' % ("Toolchain", self.ToolChain)) - - EdkLogger.info('\n%-24s =3D %s' % ("Active Platform", self.Platfor= m)) - if BuildModule: - EdkLogger.info('%-24s =3D %s' % ("Active Module", BuildModule)) - + FdfProfile =3D None if self.FdfFile: - EdkLogger.info('%-24s =3D %s' % ("Flash Image Definition", sel= f.FdfFile)) - - EdkLogger.verbose("\nFLASH_DEFINITION =3D %s" % self.FdfFile) - - if Progress: - Progress.Start("\nProcessing meta-data") - - if self.FdfFile: - # - # Mark now build in AutoGen Phase - # - GlobalData.gAutoGenPhase =3D True Fdf =3D FdfParser(self.FdfFile.Path) Fdf.ParseFile() GlobalData.gFdfParser =3D Fdf - GlobalData.gAutoGenPhase =3D False - PcdSet =3D Fdf.Profile.PcdDict if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile.FdDi= ct: FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] for FdRegion in FdDict.RegionList: if str(FdRegion.RegionType) is 'FILE' and self.Platfor= m.VpdToolGuid in str(FdRegion.RegionDataList): if int(FdRegion.Offset) % 8 !=3D 0: EdkLogger.error("build", FORMAT_INVALID, 'The = VPD Base Address %s must be 8-byte aligned.' % (FdRegion.Offset)) - ModuleList =3D Fdf.Profile.InfList - self.FdfProfile =3D Fdf.Profile + FdfProfile =3D Fdf.Profile + else: + if self.FdTargetList: + EdkLogger.info("No flash definition file found. FD [%s] wi= ll be ignored." % " ".join(self.FdTargetList)) + self.FdTargetList =3D [] + if self.FvTargetList: + EdkLogger.info("No flash definition file found. FV [%s] wi= ll be ignored." % " ".join(self.FvTargetList)) + self.FvTargetList =3D [] + if self.CapTargetList: + EdkLogger.info("No flash definition file found. Capsule [%= s] will be ignored." % " ".join(self.CapTargetList)) + self.CapTargetList =3D [] + + return FdfProfile + + def ProcessModuleFromPdf(self): + + if self.FdfProfile: for fvname in self.FvTargetList: if fvname.upper() not in self.FdfProfile.FvDict: EdkLogger.error("build", OPTION_VALUE_INVALID, "No such an FV in FDF file: %s" % fvna= me) =20 @@ -407,64 +444,60 @@ class WorkspaceAutoGen(AutoGen): # but the path (self.MetaFile.Path) is the real path for key in self.FdfProfile.InfDict: if key =3D=3D 'ArchTBD': MetaFile_cache =3D defaultdict(set) for Arch in self.ArchList: - Current_Platform_cache =3D self.BuildDatabase[self= .MetaFile, Arch, Target, Toolchain] + Current_Platform_cache =3D self.BuildDatabase[self= .MetaFile, Arch, self.BuildTarget, self.ToolChain] for Pkey in Current_Platform_cache.Modules: MetaFile_cache[Arch].add(Current_Platform_cach= e.Modules[Pkey].MetaFile) for Inf in self.FdfProfile.InfDict[key]: ModuleFile =3D PathClass(NormPath(Inf), GlobalData= .gWorkspace, Arch) for Arch in self.ArchList: if ModuleFile in MetaFile_cache[Arch]: break else: - ModuleData =3D self.BuildDatabase[ModuleFile, = Arch, Target, Toolchain] + ModuleData =3D self.BuildDatabase[ModuleFile, = Arch, self.BuildTarget, self.ToolChain] if not ModuleData.IsBinaryModule: EdkLogger.error('build', PARSER_ERROR, "Mo= dule %s NOT found in DSC file; Is it really a binary module?" % ModuleFile) =20 else: for Arch in self.ArchList: if Arch =3D=3D key: - Platform =3D self.BuildDatabase[self.MetaFile,= Arch, Target, Toolchain] + Platform =3D self.BuildDatabase[self.MetaFile,= Arch, self.BuildTarget, self.ToolChain] MetaFileList =3D set() for Pkey in Platform.Modules: MetaFileList.add(Platform.Modules[Pkey].Me= taFile) for Inf in self.FdfProfile.InfDict[key]: ModuleFile =3D PathClass(NormPath(Inf), Gl= obalData.gWorkspace, Arch) if ModuleFile in MetaFileList: continue - ModuleData =3D self.BuildDatabase[ModuleFi= le, Arch, Target, Toolchain] + ModuleData =3D self.BuildDatabase[ModuleFi= le, Arch, self.BuildTarget, self.ToolChain] if not ModuleData.IsBinaryModule: EdkLogger.error('build', PARSER_ERROR,= "Module %s NOT found in DSC file; Is it really a binary module?" % ModuleF= ile) =20 - else: - PcdSet =3D {} - ModuleList =3D [] - self.FdfProfile =3D None - if self.FdTargetList: - EdkLogger.info("No flash definition file found. FD [%s] wi= ll be ignored." % " ".join(self.FdTargetList)) - self.FdTargetList =3D [] - if self.FvTargetList: - EdkLogger.info("No flash definition file found. FV [%s] wi= ll be ignored." % " ".join(self.FvTargetList)) - self.FvTargetList =3D [] - if self.CapTargetList: - EdkLogger.info("No flash definition file found. Capsule [%= s] will be ignored." % " ".join(self.CapTargetList)) - self.CapTargetList =3D [] - - # apply SKU and inject PCDs from Flash Definition file + + + # parse FDF file to get PCDs in it, if any + def GetPcdsFromFDF(self): + + if self.FdfProfile: + PcdSet =3D self.FdfProfile.PcdDict + # handle the mixed pcd in FDF file + for key in PcdSet: + if key in GlobalData.MixedPcd: + Value =3D PcdSet[key] + del PcdSet[key] + for item in GlobalData.MixedPcd[key]: + PcdSet[item] =3D Value + self.VerifyPcdDeclearation(PcdSet) + + def ProcessPcdType(self): for Arch in self.ArchList: - Platform =3D self.BuildDatabase[self.MetaFile, Arch, Target, T= oolchain] - PlatformPcds =3D Platform.Pcds - self._GuidDict =3D Platform._GuidDict - SourcePcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set(),TAB_PCDS_DYNAMIC:set(),TAB_PCDS_FIXED_AT_BUILD:set()} - BinaryPcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set()} - SourcePcdDict_Keys =3D SourcePcdDict.keys() - BinaryPcdDict_Keys =3D BinaryPcdDict.keys() - + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + Platform.Pcds # generate the SourcePcdDict and BinaryPcdDict - PGen =3D PlatformAutoGen(self, self.MetaFile, Target, Toolchai= n, Arch) + PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) for BuildData in list(PGen.BuildDatabase._CACHE_.values()): if BuildData.Arch !=3D Arch: continue if BuildData.MetaFile.Ext =3D=3D '.inf': for key in BuildData.Pcds: @@ -483,11 +516,11 @@ class WorkspaceAutoGen(AutoGen): BuildData.Pcds[key].Type =3D PcdIn= Platform.Type BuildData.Pcds[key].Pending =3D Fa= lse else: #Pcd used in Library, Pcd Type from refere= nce module if Pcd Type is Pending if BuildData.Pcds[key].Pending: - MGen =3D ModuleAutoGen(self, BuildData= .MetaFile, Target, Toolchain, Arch, self.MetaFile) + MGen =3D ModuleAutoGen(self, BuildData= .MetaFile, self.BuildTarget, self.ToolChain, Arch, self.MetaFile) if MGen and MGen.IsLibrary: if MGen in PGen.LibraryAutoGenList: ReferenceModules =3D MGen.Refe= renceModules for ReferenceModule in Referen= ceModules: if ReferenceModule.MetaFil= e in Platform.Modules: @@ -497,10 +530,24 @@ class WorkspaceAutoGen(AutoGen): if PcdInReferenceM= odule.Type: BuildData.Pcds= [key].Type =3D PcdInReferenceModule.Type BuildData.Pcds= [key].Pending =3D False break =20 + def ProcessMixedPcd(self): + for Arch in self.ArchList: + SourcePcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set(),TAB_PCDS_DYNAMIC:set(),TAB_PCDS_FIXED_AT_BUILD:set()} + BinaryPcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set()} + SourcePcdDict_Keys =3D SourcePcdDict.keys() + BinaryPcdDict_Keys =3D BinaryPcdDict.keys() + + # generate the SourcePcdDict and BinaryPcdDict + PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) + for BuildData in list(PGen.BuildDatabase._CACHE_.values()): + if BuildData.Arch !=3D Arch: + continue + if BuildData.MetaFile.Ext =3D=3D '.inf': + for key in BuildData.Pcds: if TAB_PCDS_DYNAMIC_EX in BuildData.Pcds[key].Type: if BuildData.IsBinaryModule: BinaryPcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) else: SourcePcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) @@ -514,12 +561,11 @@ class WorkspaceAutoGen(AutoGen): =20 elif TAB_PCDS_DYNAMIC in BuildData.Pcds[key].Type: SourcePcdDict[TAB_PCDS_DYNAMIC].add((BuildData= .Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) elif TAB_PCDS_FIXED_AT_BUILD in BuildData.Pcds[key= ].Type: SourcePcdDict[TAB_PCDS_FIXED_AT_BUILD].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) - else: - pass + # # A PCD can only use one type for all source modules # for i in SourcePcdDict_Keys: for j in SourcePcdDict_Keys: @@ -588,27 +634,38 @@ class WorkspaceAutoGen(AutoGen): del BuildData.Pcds[key] BuildData.Pcds[newkey] =3D Value break break =20 - # handle the mixed pcd in FDF file - for key in PcdSet: - if key in GlobalData.MixedPcd: - Value =3D PcdSet[key] - del PcdSet[key] - for item in GlobalData.MixedPcd[key]: - PcdSet[item] =3D Value + #Collect package set information from INF of FDF + @cached_property + def PkgSet(self): + if not self.FdfFile: + self.FdfFile =3D self.Platform.FlashDefinition =20 - #Collect package set information from INF of FDF + if self.FdfFile: + ModuleList =3D self.FdfProfile.InfList + else: + ModuleList =3D [] + Pkgs =3D {} + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) PkgSet =3D set() for Inf in ModuleList: ModuleFile =3D PathClass(NormPath(Inf), GlobalData.gWorksp= ace, Arch) if ModuleFile in Platform.Modules: continue - ModuleData =3D self.BuildDatabase[ModuleFile, Arch, Target= , Toolchain] + ModuleData =3D self.BuildDatabase[ModuleFile, Arch, self.B= uildTarget, self.ToolChain] PkgSet.update(ModuleData.Packages) - Pkgs =3D list(PkgSet) + list(PGen.PackageList) + Pkgs[Arch] =3D list(PkgSet) + list(PGen.PackageList) + return Pkgs + + def VerifyPcdDeclearation(self,PcdSet): + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + Pkgs =3D self.PkgSet[Arch] DecPcds =3D set() DecPcdsKey =3D set() for Pkg in Pkgs: for Pcd in Pkg.Pcds: DecPcds.add((Pcd[0], Pcd[1])) @@ -636,37 +693,33 @@ class WorkspaceAutoGen(AutoGen): PARSER_ERROR, "Using Dynamic or DynamicEx type of PCD [%= s.%s] in FDF file is not allowed." % (Guid, Name), File =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][0], Line =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][1] ) + def CollectAllPcds(self): =20 - Pa =3D PlatformAutoGen(self, self.MetaFile, Target, Toolchain,= Arch) + for Arch in self.ArchList: + Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) # # Explicitly collect platform's dynamic PCDs # Pa.CollectPlatformDynamicPcds() Pa.CollectFixedAtBuildPcds() self.AutoGenObjectList.append(Pa) =20 - # - # Generate Package level hash value - # + # + # Generate Package level hash value + # + def GeneratePkgLevelHash(self): + for Arch in self.ArchList: GlobalData.gPackageHash =3D {} if GlobalData.gUseHashCache: - for Pkg in Pkgs: + for Pkg in self.PkgSet[Arch]: self._GenPkgLevelHash(Pkg) =20 - # - # Check PCDs token value conflict in each DEC file. - # - self._CheckAllPcdsTokenValueConflict() - - # - # Check PCD type and definition between DSC and DEC - # - self._CheckPcdDefineAndType() =20 + def CreateBuildOptionsFile(self): # # Create BuildOptions Macro & PCD metafile, also add the Active Pl= atform and FDF file. # content =3D 'gCommandLineDefines: ' content +=3D str(GlobalData.gCommandLineDefines) @@ -681,27 +734,31 @@ class WorkspaceAutoGen(AutoGen): content +=3D 'Flash Image Definition: ' content +=3D str(self.FdfFile) content +=3D TAB_LINE_BREAK SaveFileOnChange(os.path.join(self.BuildDir, 'BuildOptions'), cont= ent, False) =20 + def CreatePcdTokenNumberFile(self): # # Create PcdToken Number file for Dynamic/DynamicEx Pcd. # PcdTokenNumber =3D 'PcdTokenNumber: ' - if Pa.PcdTokenNumber: - if Pa.DynamicPcdList: - for Pcd in Pa.DynamicPcdList: - PcdTokenNumber +=3D TAB_LINE_BREAK - PcdTokenNumber +=3D str((Pcd.TokenCName, Pcd.TokenSpac= eGuidCName)) - PcdTokenNumber +=3D ' : ' - PcdTokenNumber +=3D str(Pa.PcdTokenNumber[Pcd.TokenCNa= me, Pcd.TokenSpaceGuidCName]) + for Arch in self.ArchList: + Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) + if Pa.PcdTokenNumber: + if Pa.DynamicPcdList: + for Pcd in Pa.DynamicPcdList: + PcdTokenNumber +=3D TAB_LINE_BREAK + PcdTokenNumber +=3D str((Pcd.TokenCName, Pcd.Token= SpaceGuidCName)) + PcdTokenNumber +=3D ' : ' + PcdTokenNumber +=3D str(Pa.PcdTokenNumber[Pcd.Toke= nCName, Pcd.TokenSpaceGuidCName]) SaveFileOnChange(os.path.join(self.BuildDir, 'PcdTokenNumber'), Pc= dTokenNumber, False) =20 + def CreateModuleHashInfo(self): # # Get set of workspace metafiles # - AllWorkSpaceMetaFiles =3D self._GetMetaFiles(Target, Toolchain, Ar= ch) + AllWorkSpaceMetaFiles =3D self._GetMetaFiles(self.BuildTarget, sel= f.ToolChain) =20 # # Retrieve latest modified time of all metafiles # SrcTimeStamp =3D 0 @@ -759,11 +816,11 @@ class WorkspaceAutoGen(AutoGen): f.close() m.update(Content) SaveFileOnChange(HashFile, m.hexdigest(), False) GlobalData.gPackageHash[Pkg.PackageName] =3D m.hexdigest() =20 - def _GetMetaFiles(self, Target, Toolchain, Arch): + def _GetMetaFiles(self, Target, Toolchain): AllWorkSpaceMetaFiles =3D set() # # add fdf # if self.FdfFile: --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43909): https://edk2.groups.io/g/devel/message/43909 Mute This Topic: https://groups.io/mt/32512453/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43910+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43910+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430484; cv=none; d=zoho.com; s=zohoarc; b=bLo50eZRWdttDJDQqvAaJRdjEdEINBqxyNgZdDD/NTuYhdoeQ0RaeglX6rh+MvycfW/0uZoY1plfOoog0+OPhEMixvaLt97jmFB1ZQ3WcaZUwAbYQTui+FNR6qAHokbdIZrUf0z7xN6UpKk6MubEWItL6WsvQW6AN6WpALHwPbs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430484; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=E9j9as4wVzldJDMCVDeKaoC5ANZCK4mmnOrUPOniP7g=; b=BwwCDA0kK4pZINxkuWtMYyUVkUoLAPksv7K3CTERi9Px2P9TqynFQ7WZh/mVs5qzCHJ1gxu6BE8neZkHl3GPKDfNGdXCmDpRVxAZkQK66WdmX7fJi8/INgKO0h9XQ4OXmXeKVRs+LvxFI3K4PckZS4R9YMIIhXV2/yvvEx95s2Q= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43910+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430484420725.0298757594247; Wed, 17 Jul 2019 23:14:44 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:43 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319543963" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:42 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 3/9] BaseTools: Add functions to get platform scope build options Date: Thu, 18 Jul 2019 14:14:17 +0800 Message-Id: <20190718061423.30612-4-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430483; bh=hSp0iSJvUmuu/6sqA2g7t1ZPg4qJgW1PFc/AT+dFtdI=; h=Cc:Date:From:Reply-To:Subject:To; b=Zukjff3sJXosyH6744bzQlbJ3XeU97zj686fH6bHq4yG4A3OogoURTA2yleChSJwXgZ 9pbxZ9Mv9lBhZm64CmQiog6OeN6+0B/dmkTZe9OU/uMvG0oimlUbK+QnRHfhcCAYqDVIi tKWQibEEseq7ftmxYv0P6STMWe5HfOsRW4c= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 These functions are used for get platform scope build options. They will be used in later patches. Cc: Liming Gao Signed-off-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGen.py | 10 +++++++++- .../Source/Python/Workspace/DscBuildData.py | 20 +++++++++++++++++++ .../Source/Python/Workspace/InfBuildData.py | 10 ++++++++++ 3 files changed, 39 insertions(+), 1 deletion(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index 9e06bb942126..792beed65e6b 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -2485,11 +2485,19 @@ class PlatformAutoGen(AutoGen): if Attr !=3D 'PATH': BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] else: BuildOptions[Tool][Attr] =3D Options[Key] return BuildOptions - + def GetGlobalBuildOptions(self,Module): + ModuleTypeOptions =3D self.Platform.GetBuildOptionsByPkg(Module, M= odule.ModuleType) + ModuleTypeOptions =3D self._ExpandBuildOption(ModuleTypeOptions) + if Module in self.Platform.Modules: + PlatformModule =3D self.Platform.Modules[str(Module)] + PlatformModuleOptions =3D self._ExpandBuildOption(PlatformModu= le.BuildOptions) + else: + PlatformModuleOptions =3D {} + return ModuleTypeOptions, PlatformModuleOptions ## Append build options in platform to a module # # @param Module The module to which the build options will be appe= nded # # @retval options The options appended with build options in pla= tform diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/= Source/Python/Workspace/DscBuildData.py index e7ec2aba57d2..dd5c3c2bd1f2 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -1222,11 +1222,31 @@ class DscBuildData(PlatformBuildClassObject): self._BuildOptions[CurKey] =3D Option else: if ' ' + Option not in self._BuildOptions[CurKey]: self._BuildOptions[CurKey] +=3D ' ' + Option return self._BuildOptions + def GetBuildOptionsByPkg(self, Module, ModuleType): =20 + local_pkg =3D os.path.split(Module.LocalPkg())[0] + if self._ModuleTypeOptions is None: + self._ModuleTypeOptions =3D OrderedDict() + if ModuleType not in self._ModuleTypeOptions: + options =3D OrderedDict() + self._ModuleTypeOptions[ ModuleType] =3D options + RecordList =3D self._RawData[MODEL_META_DATA_BUILD_OPTION, sel= f._Arch] + for ToolChainFamily, ToolChain, Option, Dummy1, Dummy2, Dummy3= , Dummy4, Dummy5 in RecordList: + if Dummy2 not in (TAB_COMMON,local_pkg.upper(),"EDKII"): + continue + Type =3D Dummy3 + if Type.upper() =3D=3D ModuleType.upper(): + Key =3D (ToolChainFamily, ToolChain) + if Key not in options or not ToolChain.endswith('_FLAG= S') or Option.startswith('=3D'): + options[Key] =3D Option + else: + if ' ' + Option not in options[Key]: + options[Key] +=3D ' ' + Option + return self._ModuleTypeOptions[ModuleType] def GetBuildOptionsByModuleType(self, Edk, ModuleType): if self._ModuleTypeOptions is None: self._ModuleTypeOptions =3D OrderedDict() if (Edk, ModuleType) not in self._ModuleTypeOptions: options =3D OrderedDict() diff --git a/BaseTools/Source/Python/Workspace/InfBuildData.py b/BaseTools/= Source/Python/Workspace/InfBuildData.py index 60970cd92836..da35391d3aff 100644 --- a/BaseTools/Source/Python/Workspace/InfBuildData.py +++ b/BaseTools/Source/Python/Workspace/InfBuildData.py @@ -817,11 +817,21 @@ class InfBuildData(ModuleBuildClassObject): for Token in TokenList: TemporaryDictionary[Arch, ModuleType] =3D TemporaryDiction= ary[Arch, ModuleType] + Token.strip() + ' ' for Arch, ModuleType in TemporaryDictionary: RetVal[Arch, ModuleType] =3D TemporaryDictionary[Arch, ModuleT= ype] return RetVal + def LocalPkg(self): + module_path =3D self.MetaFile.File + subdir =3D os.path.split(module_path)[0] + TopDir =3D "" + while subdir: + subdir,TopDir =3D os.path.split(subdir) =20 + for file_name in os.listdir(os.path.join(self.MetaFile.Root,TopDir= )): + if file_name.upper().endswith("DEC"): + pkg =3D os.path.join(TopDir,file_name) + return pkg @cached_class_function def GetGuidsUsedByPcd(self): self.Pcds return self._GuidsUsedByPcd =20 --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43910): https://edk2.groups.io/g/devel/message/43910 Mute This Topic: https://groups.io/mt/32512454/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43911+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43911+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430489; cv=none; d=zoho.com; s=zohoarc; b=MwrqqHQ+3Aw3neODGvA7iapLVIhkE9/ptgNi/tRNq23jCY23VujXDkI69YcQCiHzdqafnY9vYlPWXB6vjT0NgcR1dNluENN1Vpk0eLuYwqdUompQSxdAGS7KeMUHnEZ3v/PCNPuHGuEMuCZaRSXk3IKmE6UeX9WPnSthrRMgHJE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430489; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=lQ+6WHR61DJ8Zz6mt9yO4ElsMaFGtm4BPAnuES+N3NQ=; b=R5sdM8PNYLP78d39v5WIKnhCxNjulHSF5OlKsoeRGJzmGz0UlVd1ScQCmX6HdUZZR3flh8f2SNy6TPuF4DuheW0MF3dpmIRENuO+HzksiGPU2Xq86q0hGMMovEellkgEQgLVkPOZRCprly5impWa9pBLfD0Cppi/vRr9iCVbDNo= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43911+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430489401387.849875841835; Wed, 17 Jul 2019 23:14:49 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:48 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544020" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:43 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 4/9] BaseTools: Decouple AutoGen Objects Date: Thu, 18 Jul 2019 14:14:18 +0800 Message-Id: <20190718061423.30612-5-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430488; bh=15f3bTaRRJ/+kZ7Nk+CsG3wQXhKRGSC/Z5qvAvOyJX0=; h=Cc:Date:From:Reply-To:Subject:To; b=cp5n/RqwDI7NhRTdxm7kJZxvOewILfKiM0y3VS1g6PMkaj6FpKhG9YcT70f3rZMmmad vFYkFIYWdPkk4YEPh2izVD++1paAGkZbtqr/45Ra+3BSKgq942wsuBcpEXxOsFPJySfoR tPBp9qv5snYBb1brK5Hu0Ya97Fw+RZQvTT8= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 1. Separate the AutoGen.py into 3 small py files. One is for AutoGen base class, one is for WorkspaceAutoGen class and PlatformAutoGen class, and the one for ModuleAutoGen class. 2. Create a new class DataPipe to store the Platform scope settings. Create a new class PlatformInfo to provide the same interface as PlatformAutoGen. PlatformInfo class is initialized by DataPipe instance. Create a new class WorkspaceInfo to provide the same interface as WorkspaceAutoGen. WorkspaceInfo class is initialized by DataPipe instance. 3. Change ModuleAutoGen to depends on DataPipe, PlatformInfo and WorkspaceInfo. Remove the dependency of ModuleAutoGen to PlatformAutoGen. Cc: Liming Gao Signed-off-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGen.py | 4265 +---------------- BaseTools/Source/Python/AutoGen/DataPipe.py | 147 + BaseTools/Source/Python/AutoGen/GenC.py | 2 +- .../Source/Python/AutoGen/ModuleAutoGen.py | 1887 ++++++++ .../Python/AutoGen/ModuleAutoGenHelper.py | 616 +++ .../Source/Python/AutoGen/PlatformAutoGen.py | 1483 ++++++ .../Source/Python/AutoGen/WorkspaceAutoGen.py | 902 ++++ BaseTools/Source/Python/Common/Misc.py | 1 - .../Python/PatchPcdValue/PatchPcdValue.py | 1 - .../Source/Python/Workspace/DscBuildData.py | 10 +- .../Source/Python/Workspace/InfBuildData.py | 29 + .../Python/Workspace/WorkspaceCommon.py | 4 + .../Python/Workspace/WorkspaceDatabase.py | 3 + BaseTools/Source/Python/build/BuildReport.py | 4 +- BaseTools/Source/Python/build/build.py | 51 +- 15 files changed, 5156 insertions(+), 4249 deletions(-) create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index 792beed65e6b..d9ee699d8f30 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -10,226 +10,11 @@ =20 ## Import Modules # from __future__ import print_function from __future__ import absolute_import -import Common.LongFilePathOs as os -import re -import os.path as path -import copy -import uuid - -from . import GenC -from . import GenMake -from . import GenDepex -from io import BytesIO - -from .StrGather import * -from .BuildEngine import BuildRuleObj as BuildRule -from .BuildEngine import gDefaultBuildRuleFile,AutoGenReqBuildRuleVerNum -import shutil -from Common.LongFilePathSupport import CopyLongFilePath -from Common.BuildToolError import * -from Common.DataType import * -from Common.Misc import * -from Common.StringUtils import * -import Common.GlobalData as GlobalData -from GenFds.FdfParser import * -from CommonDataClass.CommonClass import SkuInfoClass -from GenPatchPcdTable.GenPatchPcdTable import parsePcdInfoFromMapFile -import Common.VpdInfoFile as VpdInfoFile -from .GenPcdDb import CreatePcdDatabaseCode -from Workspace.MetaFileCommentParser import UsageList -from Workspace.WorkspaceCommon import GetModuleLibInstances -from Common.MultipleWorkspace import MultipleWorkspace as mws -from . import InfSectionParser -import datetime -import hashlib -from .GenVar import VariableMgr, var_info -from collections import OrderedDict -from collections import defaultdict -from Workspace.WorkspaceCommon import OrderedListDict -from Common.ToolDefClassObject import gDefaultToolsDefFile - -from Common.caching import cached_property, cached_class_function - -## Regular expression for splitting Dependency Expression string into toke= ns -gDepexTokenPattern =3D re.compile("(\(|\)|\w+| \S+\.inf)") - -## Regular expression for match: PCD(xxxx.yyy) -gPCDAsGuidPattern =3D re.compile(r"^PCD\(.+\..+\)$") - -# -# Regular expression for finding Include Directories, the difference betwe= en MSFT and INTEL/GCC/RVCT -# is the former use /I , the Latter used -I to specify include directories -# -gBuildOptIncludePatternMsft =3D re.compile(r"(?:.*?)/I[ \t]*([^ ]*)", re.M= ULTILINE | re.DOTALL) -gBuildOptIncludePatternOther =3D re.compile(r"(?:.*?)-I[ \t]*([^ ]*)", re.= MULTILINE | re.DOTALL) - -# -# Match name =3D variable -# -gEfiVarStoreNamePattern =3D re.compile("\s*name\s*=3D\s*(\w+)") -# -# The format of guid in efivarstore statement likes following and must be = correct: -# guid =3D {0xA04A27f4, 0xDF00, 0x4D42, {0xB5, 0x52, 0x39, 0x51, 0x13, 0x0= 2, 0x11, 0x3D}} -# -gEfiVarStoreGuidPattern =3D re.compile("\s*guid\s*=3D\s*({.*?{.*?}\s*})") - -## Mapping Makefile type -gMakeTypeMap =3D {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"} - - -## default file name for AutoGen -gAutoGenCodeFileName =3D "AutoGen.c" -gAutoGenHeaderFileName =3D "AutoGen.h" -gAutoGenStringFileName =3D "%(module_name)sStrDefs.h" -gAutoGenStringFormFileName =3D "%(module_name)sStrDefs.hpk" -gAutoGenDepexFileName =3D "%(module_name)s.depex" -gAutoGenImageDefFileName =3D "%(module_name)sImgDefs.h" -gAutoGenIdfFileName =3D "%(module_name)sIdf.hpk" -gInfSpecVersion =3D "0x00010017" - -# -# Template string to generic AsBuilt INF -# -gAsBuiltInfHeaderString =3D TemplateString("""${header_comments} - -# DO NOT EDIT -# FILE auto-generated - -[Defines] - INF_VERSION =3D ${module_inf_version} - BASE_NAME =3D ${module_name} - FILE_GUID =3D ${module_guid} - MODULE_TYPE =3D ${module_module_type}${BEGIN} - VERSION_STRING =3D ${module_version_string}${END}${BEGIN} - PCD_IS_DRIVER =3D ${pcd_is_driver_string}${END}${BEGIN} - UEFI_SPECIFICATION_VERSION =3D ${module_uefi_specification_version}${END= }${BEGIN} - PI_SPECIFICATION_VERSION =3D ${module_pi_specification_version}${END}$= {BEGIN} - ENTRY_POINT =3D ${module_entry_point}${END}${BEGIN} - UNLOAD_IMAGE =3D ${module_unload_image}${END}${BEGIN} - CONSTRUCTOR =3D ${module_constructor}${END}${BEGIN} - DESTRUCTOR =3D ${module_destructor}${END}${BEGIN} - SHADOW =3D ${module_shadow}${END}${BEGIN} - PCI_VENDOR_ID =3D ${module_pci_vendor_id}${END}${BEGIN} - PCI_DEVICE_ID =3D ${module_pci_device_id}${END}${BEGIN} - PCI_CLASS_CODE =3D ${module_pci_class_code}${END}${BEGIN} - PCI_REVISION =3D ${module_pci_revision}${END}${BEGIN} - BUILD_NUMBER =3D ${module_build_number}${END}${BEGIN} - SPEC =3D ${module_spec}${END}${BEGIN} - UEFI_HII_RESOURCE_SECTION =3D ${module_uefi_hii_resource_section}${END}= ${BEGIN} - MODULE_UNI_FILE =3D ${module_uni_file}${END} - -[Packages.${module_arch}]${BEGIN} - ${package_item}${END} - -[Binaries.${module_arch}]${BEGIN} - ${binary_item}${END} - -[PatchPcd.${module_arch}]${BEGIN} - ${patchablepcd_item} -${END} - -[Protocols.${module_arch}]${BEGIN} - ${protocol_item} -${END} - -[Ppis.${module_arch}]${BEGIN} - ${ppi_item} -${END} - -[Guids.${module_arch}]${BEGIN} - ${guid_item} -${END} - -[PcdEx.${module_arch}]${BEGIN} - ${pcd_item} -${END} - -[LibraryClasses.${module_arch}] -## @LIB_INSTANCES${BEGIN} -# ${libraryclasses_item}${END} - -${depexsection_item} - -${userextension_tianocore_item} - -${tail_comments} - -[BuildOptions.${module_arch}] -## @AsBuilt${BEGIN} -## ${flags_item}${END} -""") -## Split command line option string to list -# -# subprocess.Popen needs the args to be a sequence. Otherwise there's prob= lem -# in non-windows platform to launch command -# -def _SplitOption(OptionString): - OptionList =3D [] - LastChar =3D " " - OptionStart =3D 0 - QuotationMark =3D "" - for Index in range(0, len(OptionString)): - CurrentChar =3D OptionString[Index] - if CurrentChar in ['"', "'"]: - if QuotationMark =3D=3D CurrentChar: - QuotationMark =3D "" - elif QuotationMark =3D=3D "": - QuotationMark =3D CurrentChar - continue - elif QuotationMark: - continue - - if CurrentChar in ["/", "-"] and LastChar in [" ", "\t", "\r", "\n= "]: - if Index > OptionStart: - OptionList.append(OptionString[OptionStart:Index - 1]) - OptionStart =3D Index - LastChar =3D CurrentChar - OptionList.append(OptionString[OptionStart:]) - return OptionList - -# -# Convert string to C format array -# -def _ConvertStringToByteArray(Value): - Value =3D Value.strip() - if not Value: - return None - if Value[0] =3D=3D '{': - if not Value.endswith('}'): - return None - Value =3D Value.replace(' ', '').replace('{', '').replace('}', '') - ValFields =3D Value.split(',') - try: - for Index in range(len(ValFields)): - ValFields[Index] =3D str(int(ValFields[Index], 0)) - except ValueError: - return None - Value =3D '{' + ','.join(ValFields) + '}' - return Value - - Unicode =3D False - if Value.startswith('L"'): - if not Value.endswith('"'): - return None - Value =3D Value[1:] - Unicode =3D True - elif not Value.startswith('"') or not Value.endswith('"'): - return None - - Value =3D eval(Value) # translate escape character - NewValue =3D '{' - for Index in range(0, len(Value)): - if Unicode: - NewValue =3D NewValue + str(ord(Value[Index]) % 0x10000) + ',' - else: - NewValue =3D NewValue + str(ord(Value[Index]) % 0x100) + ',' - Value =3D NewValue + '0}' - return Value - +from Common.DataType import TAB_STAR ## Base class for AutoGen # # This class just implements the cache mechanism of AutoGen objects. # class AutoGen(object): @@ -246,10 +31,11 @@ class AutoGen(object): # @param Toolchain Tool chain name # @param Arch Target arch # @param *args The specific class related parameters # @param **kwargs The specific class related dict parameters # + def __new__(cls, Workspace, MetaFile, Target, Toolchain, Arch, *args, = **kwargs): # check if the object has been created Key =3D (Target, Toolchain, Arch, MetaFile) if Key in cls.__ObjectCache: # if it exists, just return it directly @@ -279,4008 +65,49 @@ class AutoGen(object): =20 ## "=3D=3D" operator def __eq__(self, Other): return Other and self.MetaFile =3D=3D Other =20 -## Workspace AutoGen class -# -# This class is used mainly to control the whole platform build for diff= erent -# architecture. This class will generate top level makefile. -# -class WorkspaceAutoGen(AutoGen): - # call super().__init__ then call the worker function with different p= arameter count - def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): - if not hasattr(self, "_Init"): - self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,= *args, **kwargs) - self._Init =3D True - - ## Initialize WorkspaceAutoGen - # - # @param WorkspaceDir Root directory of workspace - # @param ActivePlatform Meta-file of active platform - # @param Target Build target - # @param Toolchain Tool chain name - # @param ArchList List of architecture of current bu= ild - # @param MetaFileDb Database containing meta-files - # @param BuildConfig Configuration of build - # @param ToolDefinition Tool chain definitions - # @param FlashDefinitionFile File of flash definition - # @param Fds FD list to be generated - # @param Fvs FV list to be generated - # @param Caps Capsule list to be generated - # @param SkuId SKU id from command line - # - def _InitWorker(self, WorkspaceDir, ActivePlatform, Target, Toolchain,= ArchList, MetaFileDb, - BuildConfig, ToolDefinition, FlashDefinitionFile=3D'', Fds= =3DNone, Fvs=3DNone, Caps=3DNone, SkuId=3D'', UniFlag=3DNone, - Progress=3DNone, BuildModule=3DNone): - self.BuildDatabase =3D MetaFileDb - self.MetaFile =3D ActivePlatform - self.WorkspaceDir =3D WorkspaceDir - self.Platform =3D self.BuildDatabase[self.MetaFile, TAB_ARCH= _COMMON, Target, Toolchain] - GlobalData.gActivePlatform =3D self.Platform - self.BuildTarget =3D Target - self.ToolChain =3D Toolchain - self.ArchList =3D ArchList - self.SkuId =3D SkuId - self.UniFlag =3D UniFlag - - self.TargetTxt =3D BuildConfig - self.ToolDef =3D ToolDefinition - self.FdfFile =3D FlashDefinitionFile - self.FdTargetList =3D Fds if Fds else [] - self.FvTargetList =3D Fvs if Fvs else [] - self.CapTargetList =3D Caps if Caps else [] - self.AutoGenObjectList =3D [] - self._GuidDict =3D {} - - # there's many relative directory operations, so ... - os.chdir(self.WorkspaceDir) - - self.MergeArch() - self.ValidateBuildTarget() - - EdkLogger.info("") - if self.ArchList: - EdkLogger.info('%-16s =3D %s' % ("Architecture(s)", ' '.join(s= elf.ArchList))) - EdkLogger.info('%-16s =3D %s' % ("Build target", self.BuildTarget)) - EdkLogger.info('%-16s =3D %s' % ("Toolchain", self.ToolChain)) - - EdkLogger.info('\n%-24s =3D %s' % ("Active Platform", self.Platfor= m)) - if BuildModule: - EdkLogger.info('%-24s =3D %s' % ("Active Module", BuildModule)) - - if self.FdfFile: - EdkLogger.info('%-24s =3D %s' % ("Flash Image Definition", sel= f.FdfFile)) - - EdkLogger.verbose("\nFLASH_DEFINITION =3D %s" % self.FdfFile) - - if Progress: - Progress.Start("\nProcessing meta-data") - # - # Mark now build in AutoGen Phase - # - GlobalData.gAutoGenPhase =3D True - self.ProcessModuleFromPdf() - self.ProcessPcdType() - self.ProcessMixedPcd() - self.GetPcdsFromFDF() - self.CollectAllPcds() - self.GeneratePkgLevelHash() - # - # Check PCDs token value conflict in each DEC file. - # - self._CheckAllPcdsTokenValueConflict() - # - # Check PCD type and definition between DSC and DEC - # - self._CheckPcdDefineAndType() - - self.CreateBuildOptionsFile() - self.CreatePcdTokenNumberFile() - self.CreateModuleHashInfo() - GlobalData.gAutoGenPhase =3D False - - # - # Merge Arch - # - def MergeArch(self): - if not self.ArchList: - ArchList =3D set(self.Platform.SupArchList) - else: - ArchList =3D set(self.ArchList) & set(self.Platform.SupArchLis= t) - if not ArchList: - EdkLogger.error("build", PARAMETER_INVALID, - ExtraData =3D "Invalid ARCH specified. [Valid = ARCH: %s]" % (" ".join(self.Platform.SupArchList))) - elif self.ArchList and len(ArchList) !=3D len(self.ArchList): - SkippedArchList =3D set(self.ArchList).symmetric_difference(se= t(self.Platform.SupArchList)) - EdkLogger.verbose("\nArch [%s] is ignored because the platform= supports [%s] only!" - % (" ".join(SkippedArchList), " ".join(self.= Platform.SupArchList))) - self.ArchList =3D tuple(ArchList) - - # Validate build target - def ValidateBuildTarget(self): - if self.BuildTarget not in self.Platform.BuildTargets: - EdkLogger.error("build", PARAMETER_INVALID, - ExtraData=3D"Build target [%s] is not supporte= d by the platform. [Valid target: %s]" - % (self.BuildTarget, " ".join(self.P= latform.BuildTargets))) - @cached_property - def FdfProfile(self): - if not self.FdfFile: - self.FdfFile =3D self.Platform.FlashDefinition - - FdfProfile =3D None - if self.FdfFile: - Fdf =3D FdfParser(self.FdfFile.Path) - Fdf.ParseFile() - GlobalData.gFdfParser =3D Fdf - if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile.FdDi= ct: - FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] - for FdRegion in FdDict.RegionList: - if str(FdRegion.RegionType) is 'FILE' and self.Platfor= m.VpdToolGuid in str(FdRegion.RegionDataList): - if int(FdRegion.Offset) % 8 !=3D 0: - EdkLogger.error("build", FORMAT_INVALID, 'The = VPD Base Address %s must be 8-byte aligned.' % (FdRegion.Offset)) - FdfProfile =3D Fdf.Profile - else: - if self.FdTargetList: - EdkLogger.info("No flash definition file found. FD [%s] wi= ll be ignored." % " ".join(self.FdTargetList)) - self.FdTargetList =3D [] - if self.FvTargetList: - EdkLogger.info("No flash definition file found. FV [%s] wi= ll be ignored." % " ".join(self.FvTargetList)) - self.FvTargetList =3D [] - if self.CapTargetList: - EdkLogger.info("No flash definition file found. Capsule [%= s] will be ignored." % " ".join(self.CapTargetList)) - self.CapTargetList =3D [] - - return FdfProfile - - def ProcessModuleFromPdf(self): - - if self.FdfProfile: - for fvname in self.FvTargetList: - if fvname.upper() not in self.FdfProfile.FvDict: - EdkLogger.error("build", OPTION_VALUE_INVALID, - "No such an FV in FDF file: %s" % fvna= me) - - # In DSC file may use FILE_GUID to override the module, then i= n the Platform.Modules use FILE_GUIDmodule.inf as key, - # but the path (self.MetaFile.Path) is the real path - for key in self.FdfProfile.InfDict: - if key =3D=3D 'ArchTBD': - MetaFile_cache =3D defaultdict(set) - for Arch in self.ArchList: - Current_Platform_cache =3D self.BuildDatabase[self= .MetaFile, Arch, self.BuildTarget, self.ToolChain] - for Pkey in Current_Platform_cache.Modules: - MetaFile_cache[Arch].add(Current_Platform_cach= e.Modules[Pkey].MetaFile) - for Inf in self.FdfProfile.InfDict[key]: - ModuleFile =3D PathClass(NormPath(Inf), GlobalData= .gWorkspace, Arch) - for Arch in self.ArchList: - if ModuleFile in MetaFile_cache[Arch]: - break - else: - ModuleData =3D self.BuildDatabase[ModuleFile, = Arch, self.BuildTarget, self.ToolChain] - if not ModuleData.IsBinaryModule: - EdkLogger.error('build', PARSER_ERROR, "Mo= dule %s NOT found in DSC file; Is it really a binary module?" % ModuleFile) - - else: - for Arch in self.ArchList: - if Arch =3D=3D key: - Platform =3D self.BuildDatabase[self.MetaFile,= Arch, self.BuildTarget, self.ToolChain] - MetaFileList =3D set() - for Pkey in Platform.Modules: - MetaFileList.add(Platform.Modules[Pkey].Me= taFile) - for Inf in self.FdfProfile.InfDict[key]: - ModuleFile =3D PathClass(NormPath(Inf), Gl= obalData.gWorkspace, Arch) - if ModuleFile in MetaFileList: - continue - ModuleData =3D self.BuildDatabase[ModuleFi= le, Arch, self.BuildTarget, self.ToolChain] - if not ModuleData.IsBinaryModule: - EdkLogger.error('build', PARSER_ERROR,= "Module %s NOT found in DSC file; Is it really a binary module?" % ModuleF= ile) - - - - # parse FDF file to get PCDs in it, if any - def GetPcdsFromFDF(self): - - if self.FdfProfile: - PcdSet =3D self.FdfProfile.PcdDict - # handle the mixed pcd in FDF file - for key in PcdSet: - if key in GlobalData.MixedPcd: - Value =3D PcdSet[key] - del PcdSet[key] - for item in GlobalData.MixedPcd[key]: - PcdSet[item] =3D Value - self.VerifyPcdDeclearation(PcdSet) - - def ProcessPcdType(self): - for Arch in self.ArchList: - Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] - Platform.Pcds - # generate the SourcePcdDict and BinaryPcdDict - PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) - for BuildData in list(PGen.BuildDatabase._CACHE_.values()): - if BuildData.Arch !=3D Arch: - continue - if BuildData.MetaFile.Ext =3D=3D '.inf': - for key in BuildData.Pcds: - if BuildData.Pcds[key].Pending: - if key in Platform.Pcds: - PcdInPlatform =3D Platform.Pcds[key] - if PcdInPlatform.Type: - BuildData.Pcds[key].Type =3D PcdInPlat= form.Type - BuildData.Pcds[key].Pending =3D False - - if BuildData.MetaFile in Platform.Modules: - PlatformModule =3D Platform.Modules[str(Bu= ildData.MetaFile)] - if key in PlatformModule.Pcds: - PcdInPlatform =3D PlatformModule.Pcds[= key] - if PcdInPlatform.Type: - BuildData.Pcds[key].Type =3D PcdIn= Platform.Type - BuildData.Pcds[key].Pending =3D Fa= lse - else: - #Pcd used in Library, Pcd Type from refere= nce module if Pcd Type is Pending - if BuildData.Pcds[key].Pending: - MGen =3D ModuleAutoGen(self, BuildData= .MetaFile, self.BuildTarget, self.ToolChain, Arch, self.MetaFile) - if MGen and MGen.IsLibrary: - if MGen in PGen.LibraryAutoGenList: - ReferenceModules =3D MGen.Refe= renceModules - for ReferenceModule in Referen= ceModules: - if ReferenceModule.MetaFil= e in Platform.Modules: - RefPlatformModule =3D = Platform.Modules[str(ReferenceModule.MetaFile)] - if key in RefPlatformM= odule.Pcds: - PcdInReferenceModu= le =3D RefPlatformModule.Pcds[key] - if PcdInReferenceM= odule.Type: - BuildData.Pcds= [key].Type =3D PcdInReferenceModule.Type - BuildData.Pcds= [key].Pending =3D False - break - - def ProcessMixedPcd(self): - for Arch in self.ArchList: - SourcePcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set(),TAB_PCDS_DYNAMIC:set(),TAB_PCDS_FIXED_AT_BUILD:set()} - BinaryPcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set()} - SourcePcdDict_Keys =3D SourcePcdDict.keys() - BinaryPcdDict_Keys =3D BinaryPcdDict.keys() - - # generate the SourcePcdDict and BinaryPcdDict - PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) - for BuildData in list(PGen.BuildDatabase._CACHE_.values()): - if BuildData.Arch !=3D Arch: - continue - if BuildData.MetaFile.Ext =3D=3D '.inf': - for key in BuildData.Pcds: - if TAB_PCDS_DYNAMIC_EX in BuildData.Pcds[key].Type: - if BuildData.IsBinaryModule: - BinaryPcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) - else: - SourcePcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) - - elif TAB_PCDS_PATCHABLE_IN_MODULE in BuildData.Pcd= s[key].Type: - if BuildData.MetaFile.Ext =3D=3D '.inf': - if BuildData.IsBinaryModule: - BinaryPcdDict[TAB_PCDS_PATCHABLE_IN_MO= DULE].add((BuildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGu= idCName)) - else: - SourcePcdDict[TAB_PCDS_PATCHABLE_IN_MO= DULE].add((BuildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGu= idCName)) - - elif TAB_PCDS_DYNAMIC in BuildData.Pcds[key].Type: - SourcePcdDict[TAB_PCDS_DYNAMIC].add((BuildData= .Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) - elif TAB_PCDS_FIXED_AT_BUILD in BuildData.Pcds[key= ].Type: - SourcePcdDict[TAB_PCDS_FIXED_AT_BUILD].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) - - # - # A PCD can only use one type for all source modules - # - for i in SourcePcdDict_Keys: - for j in SourcePcdDict_Keys: - if i !=3D j: - Intersections =3D SourcePcdDict[i].intersection(So= urcePcdDict[j]) - if len(Intersections) > 0: - EdkLogger.error( - 'build', - FORMAT_INVALID, - "Building modules from source INFs, following = PCD use %s and %s access method. It must be corrected to use only one acces= s method." % (i, j), - ExtraData=3D'\n\t'.join(str(P[1]+'.'+P[0]) for= P in Intersections) - ) - - # - # intersection the BinaryPCD for Mixed PCD - # - for i in BinaryPcdDict_Keys: - for j in BinaryPcdDict_Keys: - if i !=3D j: - Intersections =3D BinaryPcdDict[i].intersection(Bi= naryPcdDict[j]) - for item in Intersections: - NewPcd1 =3D (item[0] + '_' + i, item[1]) - NewPcd2 =3D (item[0] + '_' + j, item[1]) - if item not in GlobalData.MixedPcd: - GlobalData.MixedPcd[item] =3D [NewPcd1, Ne= wPcd2] - else: - if NewPcd1 not in GlobalData.MixedPcd[item= ]: - GlobalData.MixedPcd[item].append(NewPc= d1) - if NewPcd2 not in GlobalData.MixedPcd[item= ]: - GlobalData.MixedPcd[item].append(NewPc= d2) - - # - # intersection the SourcePCD and BinaryPCD for Mixed PCD - # - for i in SourcePcdDict_Keys: - for j in BinaryPcdDict_Keys: - if i !=3D j: - Intersections =3D SourcePcdDict[i].intersection(Bi= naryPcdDict[j]) - for item in Intersections: - NewPcd1 =3D (item[0] + '_' + i, item[1]) - NewPcd2 =3D (item[0] + '_' + j, item[1]) - if item not in GlobalData.MixedPcd: - GlobalData.MixedPcd[item] =3D [NewPcd1, Ne= wPcd2] - else: - if NewPcd1 not in GlobalData.MixedPcd[item= ]: - GlobalData.MixedPcd[item].append(NewPc= d1) - if NewPcd2 not in GlobalData.MixedPcd[item= ]: - GlobalData.MixedPcd[item].append(NewPc= d2) - - for BuildData in list(PGen.BuildDatabase._CACHE_.values()): - if BuildData.Arch !=3D Arch: - continue - for key in BuildData.Pcds: - for SinglePcd in GlobalData.MixedPcd: - if (BuildData.Pcds[key].TokenCName, BuildData.Pcds= [key].TokenSpaceGuidCName) =3D=3D SinglePcd: - for item in GlobalData.MixedPcd[SinglePcd]: - Pcd_Type =3D item[0].split('_')[-1] - if (Pcd_Type =3D=3D BuildData.Pcds[key].Ty= pe) or (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC_EX and BuildData.Pcds[key].Type in= PCD_DYNAMIC_EX_TYPE_SET) or \ - (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC and B= uildData.Pcds[key].Type in PCD_DYNAMIC_TYPE_SET): - Value =3D BuildData.Pcds[key] - Value.TokenCName =3D BuildData.Pcds[ke= y].TokenCName + '_' + Pcd_Type - if len(key) =3D=3D 2: - newkey =3D (Value.TokenCName, key[= 1]) - elif len(key) =3D=3D 3: - newkey =3D (Value.TokenCName, key[= 1], key[2]) - del BuildData.Pcds[key] - BuildData.Pcds[newkey] =3D Value - break - break - - #Collect package set information from INF of FDF - @cached_property - def PkgSet(self): - if not self.FdfFile: - self.FdfFile =3D self.Platform.FlashDefinition - - if self.FdfFile: - ModuleList =3D self.FdfProfile.InfList - else: - ModuleList =3D [] - Pkgs =3D {} - for Arch in self.ArchList: - Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] - PGen =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget= , self.ToolChain, Arch) - PkgSet =3D set() - for Inf in ModuleList: - ModuleFile =3D PathClass(NormPath(Inf), GlobalData.gWorksp= ace, Arch) - if ModuleFile in Platform.Modules: - continue - ModuleData =3D self.BuildDatabase[ModuleFile, Arch, self.B= uildTarget, self.ToolChain] - PkgSet.update(ModuleData.Packages) - Pkgs[Arch] =3D list(PkgSet) + list(PGen.PackageList) - return Pkgs - - def VerifyPcdDeclearation(self,PcdSet): - for Arch in self.ArchList: - Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] - Pkgs =3D self.PkgSet[Arch] - DecPcds =3D set() - DecPcdsKey =3D set() - for Pkg in Pkgs: - for Pcd in Pkg.Pcds: - DecPcds.add((Pcd[0], Pcd[1])) - DecPcdsKey.add((Pcd[0], Pcd[1], Pcd[2])) - - Platform.SkuName =3D self.SkuId - for Name, Guid,Fileds in PcdSet: - if (Name, Guid) not in DecPcds: - EdkLogger.error( - 'build', - PARSER_ERROR, - "PCD (%s.%s) used in FDF is not declared in DEC fi= les." % (Guid, Name), - File =3D self.FdfProfile.PcdFileLineDict[Name, Gui= d, Fileds][0], - Line =3D self.FdfProfile.PcdFileLineDict[Name, Gui= d, Fileds][1] - ) - else: - # Check whether Dynamic or DynamicEx PCD used in FDF f= ile. If used, build break and give a error message. - if (Name, Guid, TAB_PCDS_FIXED_AT_BUILD) in DecPcdsKey= \ - or (Name, Guid, TAB_PCDS_PATCHABLE_IN_MODULE) in D= ecPcdsKey \ - or (Name, Guid, TAB_PCDS_FEATURE_FLAG) in DecPcdsK= ey: - continue - elif (Name, Guid, TAB_PCDS_DYNAMIC) in DecPcdsKey or (= Name, Guid, TAB_PCDS_DYNAMIC_EX) in DecPcdsKey: - EdkLogger.error( - 'build', - PARSER_ERROR, - "Using Dynamic or DynamicEx type of PCD [%= s.%s] in FDF file is not allowed." % (Guid, Name), - File =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][0], - Line =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][1] - ) - def CollectAllPcds(self): - - for Arch in self.ArchList: - Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) - # - # Explicitly collect platform's dynamic PCDs - # - Pa.CollectPlatformDynamicPcds() - Pa.CollectFixedAtBuildPcds() - self.AutoGenObjectList.append(Pa) - - # - # Generate Package level hash value - # - def GeneratePkgLevelHash(self): - for Arch in self.ArchList: - GlobalData.gPackageHash =3D {} - if GlobalData.gUseHashCache: - for Pkg in self.PkgSet[Arch]: - self._GenPkgLevelHash(Pkg) - - - def CreateBuildOptionsFile(self): - # - # Create BuildOptions Macro & PCD metafile, also add the Active Pl= atform and FDF file. - # - content =3D 'gCommandLineDefines: ' - content +=3D str(GlobalData.gCommandLineDefines) - content +=3D TAB_LINE_BREAK - content +=3D 'BuildOptionPcd: ' - content +=3D str(GlobalData.BuildOptionPcd) - content +=3D TAB_LINE_BREAK - content +=3D 'Active Platform: ' - content +=3D str(self.Platform) - content +=3D TAB_LINE_BREAK - if self.FdfFile: - content +=3D 'Flash Image Definition: ' - content +=3D str(self.FdfFile) - content +=3D TAB_LINE_BREAK - SaveFileOnChange(os.path.join(self.BuildDir, 'BuildOptions'), cont= ent, False) - - def CreatePcdTokenNumberFile(self): - # - # Create PcdToken Number file for Dynamic/DynamicEx Pcd. - # - PcdTokenNumber =3D 'PcdTokenNumber: ' - for Arch in self.ArchList: - Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) - if Pa.PcdTokenNumber: - if Pa.DynamicPcdList: - for Pcd in Pa.DynamicPcdList: - PcdTokenNumber +=3D TAB_LINE_BREAK - PcdTokenNumber +=3D str((Pcd.TokenCName, Pcd.Token= SpaceGuidCName)) - PcdTokenNumber +=3D ' : ' - PcdTokenNumber +=3D str(Pa.PcdTokenNumber[Pcd.Toke= nCName, Pcd.TokenSpaceGuidCName]) - SaveFileOnChange(os.path.join(self.BuildDir, 'PcdTokenNumber'), Pc= dTokenNumber, False) - - def CreateModuleHashInfo(self): - # - # Get set of workspace metafiles - # - AllWorkSpaceMetaFiles =3D self._GetMetaFiles(self.BuildTarget, sel= f.ToolChain) - - # - # Retrieve latest modified time of all metafiles - # - SrcTimeStamp =3D 0 - for f in AllWorkSpaceMetaFiles: - if os.stat(f)[8] > SrcTimeStamp: - SrcTimeStamp =3D os.stat(f)[8] - self._SrcTimeStamp =3D SrcTimeStamp - - if GlobalData.gUseHashCache: - m =3D hashlib.md5() - for files in AllWorkSpaceMetaFiles: - if files.endswith('.dec'): - continue - f =3D open(files, 'rb') - Content =3D f.read() - f.close() - m.update(Content) - SaveFileOnChange(os.path.join(self.BuildDir, 'AutoGen.hash'), = m.hexdigest(), False) - GlobalData.gPlatformHash =3D m.hexdigest() - - # - # Write metafile list to build directory - # - AutoGenFilePath =3D os.path.join(self.BuildDir, 'AutoGen') - if os.path.exists (AutoGenFilePath): - os.remove(AutoGenFilePath) - if not os.path.exists(self.BuildDir): - os.makedirs(self.BuildDir) - with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file: - for f in AllWorkSpaceMetaFiles: - print(f, file=3Dfile) - return True - - def _GenPkgLevelHash(self, Pkg): - if Pkg.PackageName in GlobalData.gPackageHash: - return - - PkgDir =3D os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) - CreateDirectory(PkgDir) - HashFile =3D os.path.join(PkgDir, Pkg.PackageName + '.hash') - m =3D hashlib.md5() - # Get .dec file's hash value - f =3D open(Pkg.MetaFile.Path, 'rb') - Content =3D f.read() - f.close() - m.update(Content) - # Get include files hash value - if Pkg.Includes: - for inc in sorted(Pkg.Includes, key=3Dlambda x: str(x)): - for Root, Dirs, Files in os.walk(str(inc)): - for File in sorted(Files): - File_Path =3D os.path.join(Root, File) - f =3D open(File_Path, 'rb') - Content =3D f.read() - f.close() - m.update(Content) - SaveFileOnChange(HashFile, m.hexdigest(), False) - GlobalData.gPackageHash[Pkg.PackageName] =3D m.hexdigest() - - def _GetMetaFiles(self, Target, Toolchain): - AllWorkSpaceMetaFiles =3D set() - # - # add fdf - # - if self.FdfFile: - AllWorkSpaceMetaFiles.add (self.FdfFile.Path) - for f in GlobalData.gFdfParser.GetAllIncludedFile(): - AllWorkSpaceMetaFiles.add (f.FileName) - # - # add dsc - # - AllWorkSpaceMetaFiles.add(self.MetaFile.Path) - - # - # add build_rule.txt & tools_def.txt - # - AllWorkSpaceMetaFiles.add(os.path.join(GlobalData.gConfDirectory, = gDefaultBuildRuleFile)) - AllWorkSpaceMetaFiles.add(os.path.join(GlobalData.gConfDirectory, = gDefaultToolsDefFile)) - - # add BuildOption metafile - # - AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir, 'BuildOption= s')) - - # add PcdToken Number file for Dynamic/DynamicEx Pcd - # - AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir, 'PcdTokenNum= ber')) - - for Pa in self.AutoGenObjectList: - AllWorkSpaceMetaFiles.add(Pa.ToolDefinitionFile) - - for Arch in self.ArchList: - # - # add dec - # - for Package in PlatformAutoGen(self, self.MetaFile, Target, To= olchain, Arch).PackageList: - AllWorkSpaceMetaFiles.add(Package.MetaFile.Path) - - # - # add included dsc - # - for filePath in self.BuildDatabase[self.MetaFile, Arch, Target= , Toolchain]._RawData.IncludedFiles: - AllWorkSpaceMetaFiles.add(filePath.Path) - - return AllWorkSpaceMetaFiles - - def _CheckPcdDefineAndType(self): - PcdTypeSet =3D {TAB_PCDS_FIXED_AT_BUILD, - TAB_PCDS_PATCHABLE_IN_MODULE, - TAB_PCDS_FEATURE_FLAG, - TAB_PCDS_DYNAMIC, - TAB_PCDS_DYNAMIC_EX} - - # This dict store PCDs which are not used by any modules with spec= ified arches - UnusedPcd =3D OrderedDict() - for Pa in self.AutoGenObjectList: - # Key of DSC's Pcds dictionary is PcdCName, TokenSpaceGuid - for Pcd in Pa.Platform.Pcds: - PcdType =3D Pa.Platform.Pcds[Pcd].Type - - # If no PCD type, this PCD comes from FDF - if not PcdType: - continue - - # Try to remove Hii and Vpd suffix - if PcdType.startswith(TAB_PCDS_DYNAMIC_EX): - PcdType =3D TAB_PCDS_DYNAMIC_EX - elif PcdType.startswith(TAB_PCDS_DYNAMIC): - PcdType =3D TAB_PCDS_DYNAMIC - - for Package in Pa.PackageList: - # Key of DEC's Pcds dictionary is PcdCName, TokenSpace= Guid, PcdType - if (Pcd[0], Pcd[1], PcdType) in Package.Pcds: - break - for Type in PcdTypeSet: - if (Pcd[0], Pcd[1], Type) in Package.Pcds: - EdkLogger.error( - 'build', - FORMAT_INVALID, - "Type [%s] of PCD [%s.%s] in DSC file does= n't match the type [%s] defined in DEC file." \ - % (Pa.Platform.Pcds[Pcd].Type, Pcd[1], Pcd= [0], Type), - ExtraData=3DNone - ) - return - else: - UnusedPcd.setdefault(Pcd, []).append(Pa.Arch) - - for Pcd in UnusedPcd: - EdkLogger.warn( - 'build', - "The PCD was not specified by any INF module in the platfo= rm for the given architecture.\n" - "\tPCD: [%s.%s]\n\tPlatform: [%s]\n\tArch: %s" - % (Pcd[1], Pcd[0], os.path.basename(str(self.MetaFile)), s= tr(UnusedPcd[Pcd])), - ExtraData=3DNone - ) - - def __repr__(self): - return "%s [%s]" % (self.MetaFile, ", ".join(self.ArchList)) - - ## Return the directory to store FV files - @cached_property - def FvDir(self): - return path.join(self.BuildDir, TAB_FV_DIRECTORY) - - ## Return the directory to store all intermediate and final files built - @cached_property - def BuildDir(self): - return self.AutoGenObjectList[0].BuildDir - - ## Return the build output directory platform specifies - @cached_property - def OutputDir(self): - return self.Platform.OutputDirectory - - ## Return platform name - @cached_property - def Name(self): - return self.Platform.PlatformName - - ## Return meta-file GUID - @cached_property - def Guid(self): - return self.Platform.Guid - - ## Return platform version - @cached_property - def Version(self): - return self.Platform.Version - - ## Return paths of tools - @cached_property - def ToolDefinition(self): - return self.AutoGenObjectList[0].ToolDefinition - - ## Return directory of platform makefile - # - # @retval string Makefile directory - # - @cached_property - def MakeFileDir(self): - return self.BuildDir - - ## Return build command string - # - # @retval string Build command string - # - @cached_property - def BuildCommand(self): - # BuildCommand should be all the same. So just get one from platfo= rm AutoGen - return self.AutoGenObjectList[0].BuildCommand - - ## Check the PCDs token value conflict in each DEC file. - # - # Will cause build break and raise error message while two PCDs confli= ct. - # - # @return None - # - def _CheckAllPcdsTokenValueConflict(self): - for Pa in self.AutoGenObjectList: - for Package in Pa.PackageList: - PcdList =3D list(Package.Pcds.values()) - PcdList.sort(key=3Dlambda x: int(x.TokenValue, 0)) - Count =3D 0 - while (Count < len(PcdList) - 1) : - Item =3D PcdList[Count] - ItemNext =3D PcdList[Count + 1] - # - # Make sure in the same token space the TokenValue sho= uld be unique - # - if (int(Item.TokenValue, 0) =3D=3D int(ItemNext.TokenV= alue, 0)): - SameTokenValuePcdList =3D [] - SameTokenValuePcdList.append(Item) - SameTokenValuePcdList.append(ItemNext) - RemainPcdListLength =3D len(PcdList) - Count - 2 - for ValueSameCount in range(RemainPcdListLength): - if int(PcdList[len(PcdList) - RemainPcdListLen= gth + ValueSameCount].TokenValue, 0) =3D=3D int(Item.TokenValue, 0): - SameTokenValuePcdList.append(PcdList[len(P= cdList) - RemainPcdListLength + ValueSameCount]) - else: - break; - # - # Sort same token value PCD list with TokenGuid an= d TokenCName - # - SameTokenValuePcdList.sort(key=3Dlambda x: "%s.%s"= % (x.TokenSpaceGuidCName, x.TokenCName)) - SameTokenValuePcdListCount =3D 0 - while (SameTokenValuePcdListCount < len(SameTokenV= aluePcdList) - 1): - Flag =3D False - TemListItem =3D SameTokenValuePcdList[SameToke= nValuePcdListCount] - TemListItemNext =3D SameTokenValuePcdList[Same= TokenValuePcdListCount + 1] - - if (TemListItem.TokenSpaceGuidCName =3D=3D Tem= ListItemNext.TokenSpaceGuidCName) and (TemListItem.TokenCName !=3D TemListI= temNext.TokenCName): - for PcdItem in GlobalData.MixedPcd: - if (TemListItem.TokenCName, TemListIte= m.TokenSpaceGuidCName) in GlobalData.MixedPcd[PcdItem] or \ - (TemListItemNext.TokenCName, TemLi= stItemNext.TokenSpaceGuidCName) in GlobalData.MixedPcd[PcdItem]: - Flag =3D True - if not Flag: - EdkLogger.error( - 'build', - FORMAT_INVALID, - "The TokenValue [%s] of PC= D [%s.%s] is conflict with: [%s.%s] in %s"\ - % (TemListItem.TokenValue,= TemListItem.TokenSpaceGuidCName, TemListItem.TokenCName, TemListItemNext.T= okenSpaceGuidCName, TemListItemNext.TokenCName, Package), - ExtraData=3DNone - ) - SameTokenValuePcdListCount +=3D 1 - Count +=3D SameTokenValuePcdListCount - Count +=3D 1 - - PcdList =3D list(Package.Pcds.values()) - PcdList.sort(key=3Dlambda x: "%s.%s" % (x.TokenSpaceGuidCN= ame, x.TokenCName)) - Count =3D 0 - while (Count < len(PcdList) - 1) : - Item =3D PcdList[Count] - ItemNext =3D PcdList[Count + 1] - # - # Check PCDs with same TokenSpaceGuidCName.TokenCName = have same token value as well. - # - if (Item.TokenSpaceGuidCName =3D=3D ItemNext.TokenSpac= eGuidCName) and (Item.TokenCName =3D=3D ItemNext.TokenCName) and (int(Item.= TokenValue, 0) !=3D int(ItemNext.TokenValue, 0)): - EdkLogger.error( - 'build', - FORMAT_INVALID, - "The TokenValue [%s] of PCD [%s.%s] in= %s defined in two places should be same as well."\ - % (Item.TokenValue, Item.TokenSpaceGui= dCName, Item.TokenCName, Package), - ExtraData=3DNone - ) - Count +=3D 1 - ## Generate fds command - @property - def GenFdsCommand(self): - return (GenMake.TopLevelMakefile(self)._TEMPLATE_.Replace(GenMake.= TopLevelMakefile(self)._TemplateDict)).strip() - - @property - def GenFdsCommandDict(self): - FdsCommandDict =3D {} - LogLevel =3D EdkLogger.GetLevel() - if LogLevel =3D=3D EdkLogger.VERBOSE: - FdsCommandDict["verbose"] =3D True - elif LogLevel <=3D EdkLogger.DEBUG_9: - FdsCommandDict["debug"] =3D LogLevel - 1 - elif LogLevel =3D=3D EdkLogger.QUIET: - FdsCommandDict["quiet"] =3D True - - if GlobalData.gEnableGenfdsMultiThread: - FdsCommandDict["GenfdsMultiThread"] =3D True - if GlobalData.gIgnoreSource: - FdsCommandDict["IgnoreSources"] =3D True - - FdsCommandDict["OptionPcd"] =3D [] - for pcd in GlobalData.BuildOptionPcd: - if pcd[2]: - pcdname =3D '.'.join(pcd[0:3]) - else: - pcdname =3D '.'.join(pcd[0:2]) - if pcd[3].startswith('{'): - FdsCommandDict["OptionPcd"].append(pcdname + '=3D' + 'H' += '"' + pcd[3] + '"') - else: - FdsCommandDict["OptionPcd"].append(pcdname + '=3D' + pcd[3= ]) - - MacroList =3D [] - # macros passed to GenFds - MacroDict =3D {} - MacroDict.update(GlobalData.gGlobalDefines) - MacroDict.update(GlobalData.gCommandLineDefines) - for MacroName in MacroDict: - if MacroDict[MacroName] !=3D "": - MacroList.append('"%s=3D%s"' % (MacroName, MacroDict[Macro= Name].replace('\\', '\\\\'))) - else: - MacroList.append('"%s"' % MacroName) - FdsCommandDict["macro"] =3D MacroList - - FdsCommandDict["fdf_file"] =3D [self.FdfFile] - FdsCommandDict["build_target"] =3D self.BuildTarget - FdsCommandDict["toolchain_tag"] =3D self.ToolChain - FdsCommandDict["active_platform"] =3D str(self) - - FdsCommandDict["conf_directory"] =3D GlobalData.gConfDirectory - FdsCommandDict["build_architecture_list"] =3D ','.join(self.ArchLi= st) - FdsCommandDict["platform_build_directory"] =3D self.BuildDir - - FdsCommandDict["fd"] =3D self.FdTargetList - FdsCommandDict["fv"] =3D self.FvTargetList - FdsCommandDict["cap"] =3D self.CapTargetList - return FdsCommandDict - - ## Create makefile for the platform and modules in it - # - # @param CreateDepsMakeFile Flag indicating if the makefil= e for - # modules will be created as well - # - def CreateMakeFile(self, CreateDepsMakeFile=3DFalse): - if not CreateDepsMakeFile: - return - for Pa in self.AutoGenObjectList: - Pa.CreateMakeFile(True) - - ## Create autogen code for platform and modules - # - # Since there's no autogen code for platform, this method will do not= hing - # if CreateModuleCodeFile is set to False. - # - # @param CreateDepsCodeFile Flag indicating if creating mo= dule's - # autogen code file or not - # - def CreateCodeFile(self, CreateDepsCodeFile=3DFalse): - if not CreateDepsCodeFile: - return - for Pa in self.AutoGenObjectList: - Pa.CreateCodeFile(True) - - ## Create AsBuilt INF file the platform - # - def CreateAsBuiltInf(self): - return - - -## AutoGen class for platform -# -# PlatformAutoGen class will process the original information in platform -# file in order to generate makefile for platform. -# -class PlatformAutoGen(AutoGen): - # call super().__init__ then call the worker function with different p= arameter count - def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): - if not hasattr(self, "_Init"): - self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch) - self._Init =3D True - # - # Used to store all PCDs for both PEI and DXE phase, in order to gener= ate - # correct PCD database - # - _DynaPcdList_ =3D [] - _NonDynaPcdList_ =3D [] - _PlatformPcds =3D {} - - # - # The priority list while override build option - # - PrioList =3D {"0x11111" : 16, # TARGET_TOOLCHAIN_ARCH_COMMANDTYP= E_ATTRIBUTE (Highest) - "0x01111" : 15, # ******_TOOLCHAIN_ARCH_COMMANDTYPE_= ATTRIBUTE - "0x10111" : 14, # TARGET_*********_ARCH_COMMANDTYPE_= ATTRIBUTE - "0x00111" : 13, # ******_*********_ARCH_COMMANDTYPE_= ATTRIBUTE - "0x11011" : 12, # TARGET_TOOLCHAIN_****_COMMANDTYPE_= ATTRIBUTE - "0x01011" : 11, # ******_TOOLCHAIN_****_COMMANDTYPE_= ATTRIBUTE - "0x10011" : 10, # TARGET_*********_****_COMMANDTYPE_= ATTRIBUTE - "0x00011" : 9, # ******_*********_****_COMMANDTYPE_= ATTRIBUTE - "0x11101" : 8, # TARGET_TOOLCHAIN_ARCH_***********_= ATTRIBUTE - "0x01101" : 7, # ******_TOOLCHAIN_ARCH_***********_= ATTRIBUTE - "0x10101" : 6, # TARGET_*********_ARCH_***********_= ATTRIBUTE - "0x00101" : 5, # ******_*********_ARCH_***********_= ATTRIBUTE - "0x11001" : 4, # TARGET_TOOLCHAIN_****_***********_= ATTRIBUTE - "0x01001" : 3, # ******_TOOLCHAIN_****_***********_= ATTRIBUTE - "0x10001" : 2, # TARGET_*********_****_***********_= ATTRIBUTE - "0x00001" : 1} # ******_*********_****_***********_= ATTRIBUTE (Lowest) - - ## Initialize PlatformAutoGen - # - # - # @param Workspace WorkspaceAutoGen object - # @param PlatformFile Platform file (DSC file) - # @param Target Build target (DEBUG, RELEASE) - # @param Toolchain Name of tool chain - # @param Arch arch of the platform supports - # - def _InitWorker(self, Workspace, PlatformFile, Target, Toolchain, Arch= ): - EdkLogger.debug(EdkLogger.DEBUG_9, "AutoGen platform [%s] [%s]" % = (PlatformFile, Arch)) - GlobalData.gProcessingFile =3D "%s [%s, %s, %s]" % (PlatformFile, = Arch, Toolchain, Target) - - self.MetaFile =3D PlatformFile - self.Workspace =3D Workspace - self.WorkspaceDir =3D Workspace.WorkspaceDir - self.ToolChain =3D Toolchain - self.BuildTarget =3D Target - self.Arch =3D Arch - self.SourceDir =3D PlatformFile.SubDir - self.FdTargetList =3D self.Workspace.FdTargetList - self.FvTargetList =3D self.Workspace.FvTargetList - # get the original module/package/platform objects - self.BuildDatabase =3D Workspace.BuildDatabase - self.DscBuildDataObj =3D Workspace.Platform - - # flag indicating if the makefile/C-code file has been created or = not - self.IsMakeFileCreated =3D False - - self._DynamicPcdList =3D None # [(TokenCName1, TokenSpaceGuidCN= ame1), (TokenCName2, TokenSpaceGuidCName2), ...] - self._NonDynamicPcdList =3D None # [(TokenCName1, TokenSpaceGuidCN= ame1), (TokenCName2, TokenSpaceGuidCName2), ...] - - self._AsBuildInfList =3D [] - self._AsBuildModuleList =3D [] - - self.VariableInfo =3D None - - if GlobalData.gFdfParser is not None: - self._AsBuildInfList =3D GlobalData.gFdfParser.Profile.InfList - for Inf in self._AsBuildInfList: - InfClass =3D PathClass(NormPath(Inf), GlobalData.gWorkspac= e, self.Arch) - M =3D self.BuildDatabase[InfClass, self.Arch, self.BuildTa= rget, self.ToolChain] - if not M.IsBinaryModule: - continue - self._AsBuildModuleList.append(InfClass) - # get library/modules for build - self.LibraryBuildDirectoryList =3D [] - self.ModuleBuildDirectoryList =3D [] - - return True - - ## hash() operator of PlatformAutoGen - # - # The platform file path and arch string will be used to represent - # hash value of this object - # - # @retval int Hash value of the platform file path and arch - # - @cached_class_function - def __hash__(self): - return hash((self.MetaFile, self.Arch)) - - @cached_class_function - def __repr__(self): - return "%s [%s]" % (self.MetaFile, self.Arch) - - ## Create autogen code for platform and modules - # - # Since there's no autogen code for platform, this method will do not= hing - # if CreateModuleCodeFile is set to False. - # - # @param CreateModuleCodeFile Flag indicating if creating mo= dule's - # autogen code file or not - # - @cached_class_function - def CreateCodeFile(self, CreateModuleCodeFile=3DFalse): - # only module has code to be created, so do nothing if CreateModul= eCodeFile is False - if not CreateModuleCodeFile: - return - - for Ma in self.ModuleAutoGenList: - Ma.CreateCodeFile(True) - - ## Generate Fds Command - @cached_property - def GenFdsCommand(self): - return self.Workspace.GenFdsCommand - - ## Create makefile for the platform and modules in it - # - # @param CreateModuleMakeFile Flag indicating if the makefil= e for - # modules will be created as well - # - def CreateMakeFile(self, CreateModuleMakeFile=3DFalse, FfsCommand =3D = {}): - if CreateModuleMakeFile: - for Ma in self._MaList: - key =3D (Ma.MetaFile.File, self.Arch) - if key in FfsCommand: - Ma.CreateMakeFile(True, FfsCommand[key]) - else: - Ma.CreateMakeFile(True) - - # no need to create makefile for the platform more than once - if self.IsMakeFileCreated: - return - - # create library/module build dirs for platform - Makefile =3D GenMake.PlatformMakefile(self) - self.LibraryBuildDirectoryList =3D Makefile.GetLibraryBuildDirecto= ryList() - self.ModuleBuildDirectoryList =3D Makefile.GetModuleBuildDirectory= List() - - self.IsMakeFileCreated =3D True - - @property - def AllPcdList(self): - return self.DynamicPcdList + self.NonDynamicPcdList - ## Deal with Shared FixedAtBuild Pcds - # - def CollectFixedAtBuildPcds(self): - for LibAuto in self.LibraryAutoGenList: - FixedAtBuildPcds =3D {} - ShareFixedAtBuildPcdsSameValue =3D {} - for Module in LibAuto.ReferenceModules: - for Pcd in set(Module.FixedAtBuildPcds + LibAuto.FixedAtBu= ildPcds): - DefaultValue =3D Pcd.DefaultValue - # Cover the case: DSC component override the Pcd value= and the Pcd only used in one Lib - if Pcd in Module.LibraryPcdList: - Index =3D Module.LibraryPcdList.index(Pcd) - DefaultValue =3D Module.LibraryPcdList[Index].Defa= ultValue - key =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCN= ame)) - if key not in FixedAtBuildPcds: - ShareFixedAtBuildPcdsSameValue[key] =3D True - FixedAtBuildPcds[key] =3D DefaultValue - else: - if FixedAtBuildPcds[key] !=3D DefaultValue: - ShareFixedAtBuildPcdsSameValue[key] =3D False - for Pcd in LibAuto.FixedAtBuildPcds: - key =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCName)) - if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) not in self.N= onDynamicPcdDict: - continue - else: - DscPcd =3D self.NonDynamicPcdDict[(Pcd.TokenCName, Pcd= .TokenSpaceGuidCName)] - if DscPcd.Type !=3D TAB_PCDS_FIXED_AT_BUILD: - continue - if key in ShareFixedAtBuildPcdsSameValue and ShareFixedAtB= uildPcdsSameValue[key]: - LibAuto.ConstPcd[key] =3D FixedAtBuildPcds[key] - - def CollectVariables(self, DynamicPcdSet): - VpdRegionSize =3D 0 - VpdRegionBase =3D 0 - if self.Workspace.FdfFile: - FdDict =3D self.Workspace.FdfProfile.FdDict[GlobalData.gFdfPar= ser.CurrentFdName] - for FdRegion in FdDict.RegionList: - for item in FdRegion.RegionDataList: - if self.Platform.VpdToolGuid.strip() and self.Platform= .VpdToolGuid in item: - VpdRegionSize =3D FdRegion.Size - VpdRegionBase =3D FdRegion.Offset - break - - VariableInfo =3D VariableMgr(self.DscBuildDataObj._GetDefaultStore= s(), self.DscBuildDataObj.SkuIds) - VariableInfo.SetVpdRegionMaxSize(VpdRegionSize) - VariableInfo.SetVpdRegionOffset(VpdRegionBase) - Index =3D 0 - for Pcd in DynamicPcdSet: - pcdname =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCName)) - for SkuName in Pcd.SkuInfoList: - Sku =3D Pcd.SkuInfoList[SkuName] - SkuId =3D Sku.SkuId - if SkuId is None or SkuId =3D=3D '': - continue - if len(Sku.VariableName) > 0: - if Sku.VariableAttribute and 'NV' not in Sku.VariableA= ttribute: - continue - VariableGuidStructure =3D Sku.VariableGuidValue - VariableGuid =3D GuidStructureStringToGuidString(Varia= bleGuidStructure) - for StorageName in Sku.DefaultStoreDict: - VariableInfo.append_variable(var_info(Index, pcdna= me, StorageName, SkuName, StringToArray(Sku.VariableName), VariableGuid, Sk= u.VariableOffset, Sku.VariableAttribute, Sku.HiiDefaultValue, Sku.DefaultSt= oreDict[StorageName] if Pcd.DatumType in TAB_PCD_NUMERIC_TYPES else StringT= oArray(Sku.DefaultStoreDict[StorageName]), Pcd.DatumType, Pcd.CustomAttribu= te['DscPosition'], Pcd.CustomAttribute.get('IsStru',False))) - Index +=3D 1 - return VariableInfo - - def UpdateNVStoreMaxSize(self, OrgVpdFile): - if self.VariableInfo: - VpdMapFilePath =3D os.path.join(self.BuildDir, TAB_FV_DIRECTOR= Y, "%s.map" % self.Platform.VpdToolGuid) - PcdNvStoreDfBuffer =3D [item for item in self._DynamicPcdList = if item.TokenCName =3D=3D "PcdNvStoreDefaultValueBuffer" and item.TokenSpac= eGuidCName =3D=3D "gEfiMdeModulePkgTokenSpaceGuid"] - - if PcdNvStoreDfBuffer: - if os.path.exists(VpdMapFilePath): - OrgVpdFile.Read(VpdMapFilePath) - PcdItems =3D OrgVpdFile.GetOffset(PcdNvStoreDfBuffer[0= ]) - NvStoreOffset =3D list(PcdItems.values())[0].strip() i= f PcdItems else '0' - else: - EdkLogger.error("build", FILE_READ_FAILURE, "Can not f= ind VPD map file %s to fix up VPD offset." % VpdMapFilePath) - - NvStoreOffset =3D int(NvStoreOffset, 16) if NvStoreOffset.= upper().startswith("0X") else int(NvStoreOffset) - default_skuobj =3D PcdNvStoreDfBuffer[0].SkuInfoList.get(T= AB_DEFAULT) - maxsize =3D self.VariableInfo.VpdRegionSize - NvStoreOffs= et if self.VariableInfo.VpdRegionSize else len(default_skuobj.DefaultValue.= split(",")) - var_data =3D self.VariableInfo.PatchNVStoreDefaultMaxSize(= maxsize) - - if var_data and default_skuobj: - default_skuobj.DefaultValue =3D var_data - PcdNvStoreDfBuffer[0].DefaultValue =3D var_data - PcdNvStoreDfBuffer[0].SkuInfoList.clear() - PcdNvStoreDfBuffer[0].SkuInfoList[TAB_DEFAULT] =3D def= ault_skuobj - PcdNvStoreDfBuffer[0].MaxDatumSize =3D str(len(default= _skuobj.DefaultValue.split(","))) - - return OrgVpdFile - - ## Collect dynamic PCDs - # - # Gather dynamic PCDs list from each module and their settings from p= latform - # This interface should be invoked explicitly when platform action is= created. - # - def CollectPlatformDynamicPcds(self): - for key in self.Platform.Pcds: - for SinglePcd in GlobalData.MixedPcd: - if (self.Platform.Pcds[key].TokenCName, self.Platform.Pcds= [key].TokenSpaceGuidCName) =3D=3D SinglePcd: - for item in GlobalData.MixedPcd[SinglePcd]: - Pcd_Type =3D item[0].split('_')[-1] - if (Pcd_Type =3D=3D self.Platform.Pcds[key].Type) = or (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC_EX and self.Platform.Pcds[key].Type in= PCD_DYNAMIC_EX_TYPE_SET) or \ - (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC and self.Plat= form.Pcds[key].Type in PCD_DYNAMIC_TYPE_SET): - Value =3D self.Platform.Pcds[key] - Value.TokenCName =3D self.Platform.Pcds[key].T= okenCName + '_' + Pcd_Type - if len(key) =3D=3D 2: - newkey =3D (Value.TokenCName, key[1]) - elif len(key) =3D=3D 3: - newkey =3D (Value.TokenCName, key[1], key[= 2]) - del self.Platform.Pcds[key] - self.Platform.Pcds[newkey] =3D Value - break - break - - # for gathering error information - NoDatumTypePcdList =3D set() - FdfModuleList =3D [] - for InfName in self._AsBuildInfList: - InfName =3D mws.join(self.WorkspaceDir, InfName) - FdfModuleList.append(os.path.normpath(InfName)) - for M in self._MaList: -# F is the Module for which M is the module autogen - for PcdFromModule in M.ModulePcdList + M.LibraryPcdList: - # make sure that the "VOID*" kind of datum has MaxDatumSiz= e set - if PcdFromModule.DatumType =3D=3D TAB_VOID and not PcdFrom= Module.MaxDatumSize: - NoDatumTypePcdList.add("%s.%s [%s]" % (PcdFromModule.T= okenSpaceGuidCName, PcdFromModule.TokenCName, M.MetaFile)) - - # Check the PCD from Binary INF or Source INF - if M.IsBinaryModule =3D=3D True: - PcdFromModule.IsFromBinaryInf =3D True - - # Check the PCD from DSC or not - PcdFromModule.IsFromDsc =3D (PcdFromModule.TokenCName, Pcd= FromModule.TokenSpaceGuidCName) in self.Platform.Pcds - - if PcdFromModule.Type in PCD_DYNAMIC_TYPE_SET or PcdFromMo= dule.Type in PCD_DYNAMIC_EX_TYPE_SET: - if M.MetaFile.Path not in FdfModuleList: - # If one of the Source built modules listed in the= DSC is not listed - # in FDF modules, and the INF lists a PCD can only= use the PcdsDynamic - # access method (it is only listed in the DEC file= that declares the - # PCD as PcdsDynamic), then build tool will report= warning message - # notify the PI that they are attempting to build = a module that must - # be included in a flash image in order to be func= tional. These Dynamic - # PCD will not be added into the Database unless i= t is used by other - # modules that are included in the FDF file. - if PcdFromModule.Type in PCD_DYNAMIC_TYPE_SET and \ - PcdFromModule.IsFromBinaryInf =3D=3D False: - # Print warning message to let the developer m= ake a determine. - continue - # If one of the Source built modules listed in the= DSC is not listed in - # FDF modules, and the INF lists a PCD can only us= e the PcdsDynamicEx - # access method (it is only listed in the DEC file= that declares the - # PCD as PcdsDynamicEx), then DO NOT break the bui= ld; DO NOT add the - # PCD to the Platform's PCD Database. - if PcdFromModule.Type in PCD_DYNAMIC_EX_TYPE_SET: - continue - # - # If a dynamic PCD used by a PEM module/PEI module & D= XE module, - # it should be stored in Pcd PEI database, If a dynami= c only - # used by DXE module, it should be stored in DXE PCD d= atabase. - # The default Phase is DXE - # - if M.ModuleType in SUP_MODULE_SET_PEI: - PcdFromModule.Phase =3D "PEI" - if PcdFromModule not in self._DynaPcdList_: - self._DynaPcdList_.append(PcdFromModule) - elif PcdFromModule.Phase =3D=3D 'PEI': - # overwrite any the same PCD existing, if Phase is= PEI - Index =3D self._DynaPcdList_.index(PcdFromModule) - self._DynaPcdList_[Index] =3D PcdFromModule - elif PcdFromModule not in self._NonDynaPcdList_: - self._NonDynaPcdList_.append(PcdFromModule) - elif PcdFromModule in self._NonDynaPcdList_ and PcdFromMod= ule.IsFromBinaryInf =3D=3D True: - Index =3D self._NonDynaPcdList_.index(PcdFromModule) - if self._NonDynaPcdList_[Index].IsFromBinaryInf =3D=3D= False: - #The PCD from Binary INF will override the same on= e from source INF - self._NonDynaPcdList_.remove (self._NonDynaPcdList= _[Index]) - PcdFromModule.Pending =3D False - self._NonDynaPcdList_.append (PcdFromModule) - DscModuleSet =3D {os.path.normpath(ModuleInf.Path) for ModuleInf i= n self.Platform.Modules} - # add the PCD from modules that listed in FDF but not in DSC to Da= tabase - for InfName in FdfModuleList: - if InfName not in DscModuleSet: - InfClass =3D PathClass(InfName) - M =3D self.BuildDatabase[InfClass, self.Arch, self.BuildTa= rget, self.ToolChain] - # If a module INF in FDF but not in current arch's DSC mod= ule list, it must be module (either binary or source) - # for different Arch. PCDs in source module for different = Arch is already added before, so skip the source module here. - # For binary module, if in current arch, we need to list t= he PCDs into database. - if not M.IsBinaryModule: - continue - # Override the module PCD setting by platform setting - ModulePcdList =3D self.ApplyPcdSetting(M, M.Pcds) - for PcdFromModule in ModulePcdList: - PcdFromModule.IsFromBinaryInf =3D True - PcdFromModule.IsFromDsc =3D False - # Only allow the DynamicEx and Patchable PCD in AsBuil= d INF - if PcdFromModule.Type not in PCD_DYNAMIC_EX_TYPE_SET a= nd PcdFromModule.Type not in TAB_PCDS_PATCHABLE_IN_MODULE: - EdkLogger.error("build", AUTOGEN_ERROR, "PCD setti= ng error", - File=3Dself.MetaFile, - ExtraData=3D"\n\tExisted %s PCD %s= in:\n\t\t%s\n" - % (PcdFromModule.Type, PcdFromModu= le.TokenCName, InfName)) - # make sure that the "VOID*" kind of datum has MaxDatu= mSize set - if PcdFromModule.DatumType =3D=3D TAB_VOID and not Pcd= FromModule.MaxDatumSize: - NoDatumTypePcdList.add("%s.%s [%s]" % (PcdFromModu= le.TokenSpaceGuidCName, PcdFromModule.TokenCName, InfName)) - if M.ModuleType in SUP_MODULE_SET_PEI: - PcdFromModule.Phase =3D "PEI" - if PcdFromModule not in self._DynaPcdList_ and PcdFrom= Module.Type in PCD_DYNAMIC_EX_TYPE_SET: - self._DynaPcdList_.append(PcdFromModule) - elif PcdFromModule not in self._NonDynaPcdList_ and Pc= dFromModule.Type in TAB_PCDS_PATCHABLE_IN_MODULE: - self._NonDynaPcdList_.append(PcdFromModule) - if PcdFromModule in self._DynaPcdList_ and PcdFromModu= le.Phase =3D=3D 'PEI' and PcdFromModule.Type in PCD_DYNAMIC_EX_TYPE_SET: - # Overwrite the phase of any the same PCD existing= , if Phase is PEI. - # It is to solve the case that a dynamic PCD used = by a PEM module/PEI - # module & DXE module at a same time. - # Overwrite the type of the PCDs in source INF by = the type of AsBuild - # INF file as DynamicEx. - Index =3D self._DynaPcdList_.index(PcdFromModule) - self._DynaPcdList_[Index].Phase =3D PcdFromModule.= Phase - self._DynaPcdList_[Index].Type =3D PcdFromModule.T= ype - for PcdFromModule in self._NonDynaPcdList_: - # If a PCD is not listed in the DSC file, but binary INF files= used by - # this platform all (that use this PCD) list the PCD in a [Pat= chPcds] - # section, AND all source INF files used by this platform the = build - # that use the PCD list the PCD in either a [Pcds] or [PatchPc= ds] - # section, then the tools must NOT add the PCD to the Platform= 's PCD - # Database; the build must assign the access method for this P= CD as - # PcdsPatchableInModule. - if PcdFromModule not in self._DynaPcdList_: - continue - Index =3D self._DynaPcdList_.index(PcdFromModule) - if PcdFromModule.IsFromDsc =3D=3D False and \ - PcdFromModule.Type in TAB_PCDS_PATCHABLE_IN_MODULE and \ - PcdFromModule.IsFromBinaryInf =3D=3D True and \ - self._DynaPcdList_[Index].IsFromBinaryInf =3D=3D False: - Index =3D self._DynaPcdList_.index(PcdFromModule) - self._DynaPcdList_.remove (self._DynaPcdList_[Index]) - - # print out error information and break the build, if error found - if len(NoDatumTypePcdList) > 0: - NoDatumTypePcdListString =3D "\n\t\t".join(NoDatumTypePcdList) - EdkLogger.error("build", AUTOGEN_ERROR, "PCD setting error", - File=3Dself.MetaFile, - ExtraData=3D"\n\tPCD(s) without MaxDatumSize:\= n\t\t%s\n" - % NoDatumTypePcdListString) - self._NonDynamicPcdList =3D self._NonDynaPcdList_ - self._DynamicPcdList =3D self._DynaPcdList_ - # - # Sort dynamic PCD list to: - # 1) If PCD's datum type is VOID* and value is unicode string whic= h starts with L, the PCD item should - # try to be put header of dynamicd List - # 2) If PCD is HII type, the PCD item should be put after unicode = type PCD - # - # The reason of sorting is make sure the unicode string is in doub= le-byte alignment in string table. - # - UnicodePcdArray =3D set() - HiiPcdArray =3D set() - OtherPcdArray =3D set() - VpdPcdDict =3D {} - VpdFile =3D VpdInfoFile.VpdInfoFile() - NeedProcessVpdMapFile =3D False - - for pcd in self.Platform.Pcds: - if pcd not in self._PlatformPcds: - self._PlatformPcds[pcd] =3D self.Platform.Pcds[pcd] - - for item in self._PlatformPcds: - if self._PlatformPcds[item].DatumType and self._PlatformPcds[i= tem].DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32, TAB_UINT64, TAB_V= OID, "BOOLEAN"]: - self._PlatformPcds[item].DatumType =3D TAB_VOID - - if (self.Workspace.ArchList[-1] =3D=3D self.Arch): - for Pcd in self._DynamicPcdList: - # just pick the a value to determine whether is unicode st= ring type - Sku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) - Sku.VpdOffset =3D Sku.VpdOffset.strip() - - if Pcd.DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32= , TAB_UINT64, TAB_VOID, "BOOLEAN"]: - Pcd.DatumType =3D TAB_VOID - - # if found PCD which datum value is unicode string the= insert to left size of UnicodeIndex - # if found HII type PCD then insert to right of Unicod= eIndex - if Pcd.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYNAMIC_EX_= VPD]: - VpdPcdDict[(Pcd.TokenCName, Pcd.TokenSpaceGuidCName)] = =3D Pcd - - #Collect DynamicHii PCD values and assign it to DynamicExVpd P= CD gEfiMdeModulePkgTokenSpaceGuid.PcdNvStoreDefaultValueBuffer - PcdNvStoreDfBuffer =3D VpdPcdDict.get(("PcdNvStoreDefaultValue= Buffer", "gEfiMdeModulePkgTokenSpaceGuid")) - if PcdNvStoreDfBuffer: - self.VariableInfo =3D self.CollectVariables(self._DynamicP= cdList) - vardump =3D self.VariableInfo.dump() - if vardump: - # - #According to PCD_DATABASE_INIT in edk2\MdeModulePkg\I= nclude\Guid\PcdDataBaseSignatureGuid.h, - #the max size for string PCD should not exceed USHRT_M= AX 65535(0xffff). - #typedef UINT16 SIZE_INFO; - #//SIZE_INFO SizeTable[]; - if len(vardump.split(",")) > 0xffff: - EdkLogger.error("build", RESOURCE_OVERFLOW, 'The c= urrent length of PCD %s value is %d, it exceeds to the max size of String P= CD.' %(".".join([PcdNvStoreDfBuffer.TokenSpaceGuidCName,PcdNvStoreDfBuffer.= TokenCName]) ,len(vardump.split(",")))) - PcdNvStoreDfBuffer.DefaultValue =3D vardump - for skuname in PcdNvStoreDfBuffer.SkuInfoList: - PcdNvStoreDfBuffer.SkuInfoList[skuname].DefaultVal= ue =3D vardump - PcdNvStoreDfBuffer.MaxDatumSize =3D str(len(vardum= p.split(","))) - else: - #If the end user define [DefaultStores] and [XXX.Menufactu= ring] in DSC, but forget to configure PcdNvStoreDefaultValueBuffer to PcdsD= ynamicVpd - if [Pcd for Pcd in self._DynamicPcdList if Pcd.UserDefined= DefaultStoresFlag]: - EdkLogger.warn("build", "PcdNvStoreDefaultValueBuffer = should be defined as PcdsDynamicExVpd in dsc file since the DefaultStores i= s enabled for this platform.\n%s" %self.Platform.MetaFile.Path) - PlatformPcds =3D sorted(self._PlatformPcds.keys()) - # - # Add VPD type PCD into VpdFile and determine whether the VPD = PCD need to be fixed up. - # - VpdSkuMap =3D {} - for PcdKey in PlatformPcds: - Pcd =3D self._PlatformPcds[PcdKey] - if Pcd.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYNAMIC_EX_= VPD] and \ - PcdKey in VpdPcdDict: - Pcd =3D VpdPcdDict[PcdKey] - SkuValueMap =3D {} - DefaultSku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) - if DefaultSku: - PcdValue =3D DefaultSku.DefaultValue - if PcdValue not in SkuValueMap: - SkuValueMap[PcdValue] =3D [] - VpdFile.Add(Pcd, TAB_DEFAULT, DefaultSku.VpdOf= fset) - SkuValueMap[PcdValue].append(DefaultSku) - - for (SkuName, Sku) in Pcd.SkuInfoList.items(): - Sku.VpdOffset =3D Sku.VpdOffset.strip() - PcdValue =3D Sku.DefaultValue - if PcdValue =3D=3D "": - PcdValue =3D Pcd.DefaultValue - if Sku.VpdOffset !=3D TAB_STAR: - if PcdValue.startswith("{"): - Alignment =3D 8 - elif PcdValue.startswith("L"): - Alignment =3D 2 - else: - Alignment =3D 1 - try: - VpdOffset =3D int(Sku.VpdOffset) - except: - try: - VpdOffset =3D int(Sku.VpdOffset, 16) - except: - EdkLogger.error("build", FORMAT_INVALI= D, "Invalid offset value %s for PCD %s.%s." % (Sku.VpdOffset, Pcd.TokenSpac= eGuidCName, Pcd.TokenCName)) - if VpdOffset % Alignment !=3D 0: - if PcdValue.startswith("{"): - EdkLogger.warn("build", "The offset va= lue of PCD %s.%s is not 8-byte aligned!" %(Pcd.TokenSpaceGuidCName, Pcd.Tok= enCName), File=3Dself.MetaFile) - else: - EdkLogger.error("build", FORMAT_INVALI= D, 'The offset value of PCD %s.%s should be %s-byte aligned.' % (Pcd.TokenS= paceGuidCName, Pcd.TokenCName, Alignment)) - if PcdValue not in SkuValueMap: - SkuValueMap[PcdValue] =3D [] - VpdFile.Add(Pcd, SkuName, Sku.VpdOffset) - SkuValueMap[PcdValue].append(Sku) - # if the offset of a VPD is *, then it need to be = fixed up by third party tool. - if not NeedProcessVpdMapFile and Sku.VpdOffset =3D= =3D TAB_STAR: - NeedProcessVpdMapFile =3D True - if self.Platform.VpdToolGuid is None or self.P= latform.VpdToolGuid =3D=3D '': - EdkLogger.error("Build", FILE_NOT_FOUND, \ - "Fail to find third-party = BPDG tool to process VPD PCDs. BPDG Guid tool need to be defined in tools_d= ef.txt and VPD_TOOL_GUID need to be provided in DSC file.") - - VpdSkuMap[PcdKey] =3D SkuValueMap - # - # Fix the PCDs define in VPD PCD section that never referenced= by module. - # An example is PCD for signature usage. - # - for DscPcd in PlatformPcds: - DscPcdEntry =3D self._PlatformPcds[DscPcd] - if DscPcdEntry.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYN= AMIC_EX_VPD]: - if not (self.Platform.VpdToolGuid is None or self.Plat= form.VpdToolGuid =3D=3D ''): - FoundFlag =3D False - for VpdPcd in VpdFile._VpdArray: - # This PCD has been referenced by module - if (VpdPcd.TokenSpaceGuidCName =3D=3D DscPcdEn= try.TokenSpaceGuidCName) and \ - (VpdPcd.TokenCName =3D=3D DscPcdEntry.Token= CName): - FoundFlag =3D True - - # Not found, it should be signature - if not FoundFlag : - # just pick the a value to determine whether i= s unicode string type - SkuValueMap =3D {} - SkuObjList =3D list(DscPcdEntry.SkuInfoList.it= ems()) - DefaultSku =3D DscPcdEntry.SkuInfoList.get(TAB= _DEFAULT) - if DefaultSku: - defaultindex =3D SkuObjList.index((TAB_DEF= AULT, DefaultSku)) - SkuObjList[0], SkuObjList[defaultindex] = =3D SkuObjList[defaultindex], SkuObjList[0] - for (SkuName, Sku) in SkuObjList: - Sku.VpdOffset =3D Sku.VpdOffset.strip() - - # Need to iterate DEC pcd information to g= et the value & datumtype - for eachDec in self.PackageList: - for DecPcd in eachDec.Pcds: - DecPcdEntry =3D eachDec.Pcds[DecPc= d] - if (DecPcdEntry.TokenSpaceGuidCNam= e =3D=3D DscPcdEntry.TokenSpaceGuidCName) and \ - (DecPcdEntry.TokenCName =3D=3D = DscPcdEntry.TokenCName): - # Print warning message to let= the developer make a determine. - EdkLogger.warn("build", "Unref= erenced vpd pcd used!", - File=3Dself.Me= taFile, \ - ExtraData =3D = "PCD: %s.%s used in the DSC file %s is unreferenced." \ - %(DscPcdEntry.= TokenSpaceGuidCName, DscPcdEntry.TokenCName, self.Platform.MetaFile.Path)) - - DscPcdEntry.DatumType =3D D= ecPcdEntry.DatumType - DscPcdEntry.DefaultValue =3D D= ecPcdEntry.DefaultValue - DscPcdEntry.TokenValue =3D Dec= PcdEntry.TokenValue - DscPcdEntry.TokenSpaceGuidValu= e =3D eachDec.Guids[DecPcdEntry.TokenSpaceGuidCName] - # Only fix the value while no = value provided in DSC file. - if not Sku.DefaultValue: - DscPcdEntry.SkuInfoList[li= st(DscPcdEntry.SkuInfoList.keys())[0]].DefaultValue =3D DecPcdEntry.Default= Value - - if DscPcdEntry not in self._DynamicPcdList: - self._DynamicPcdList.append(DscPcdEntr= y) - Sku.VpdOffset =3D Sku.VpdOffset.strip() - PcdValue =3D Sku.DefaultValue - if PcdValue =3D=3D "": - PcdValue =3D DscPcdEntry.DefaultValue - if Sku.VpdOffset !=3D TAB_STAR: - if PcdValue.startswith("{"): - Alignment =3D 8 - elif PcdValue.startswith("L"): - Alignment =3D 2 - else: - Alignment =3D 1 - try: - VpdOffset =3D int(Sku.VpdOffset) - except: - try: - VpdOffset =3D int(Sku.VpdOffse= t, 16) - except: - EdkLogger.error("build", FORMA= T_INVALID, "Invalid offset value %s for PCD %s.%s." % (Sku.VpdOffset, DscPc= dEntry.TokenSpaceGuidCName, DscPcdEntry.TokenCName)) - if VpdOffset % Alignment !=3D 0: - if PcdValue.startswith("{"): - EdkLogger.warn("build", "The o= ffset value of PCD %s.%s is not 8-byte aligned!" %(DscPcdEntry.TokenSpaceGu= idCName, DscPcdEntry.TokenCName), File=3Dself.MetaFile) - else: - EdkLogger.error("build", FORMA= T_INVALID, 'The offset value of PCD %s.%s should be %s-byte aligned.' % (Ds= cPcdEntry.TokenSpaceGuidCName, DscPcdEntry.TokenCName, Alignment)) - if PcdValue not in SkuValueMap: - SkuValueMap[PcdValue] =3D [] - VpdFile.Add(DscPcdEntry, SkuName, Sku.= VpdOffset) - SkuValueMap[PcdValue].append(Sku) - if not NeedProcessVpdMapFile and Sku.VpdOf= fset =3D=3D TAB_STAR: - NeedProcessVpdMapFile =3D True - if DscPcdEntry.DatumType =3D=3D TAB_VOID and P= cdValue.startswith("L"): - UnicodePcdArray.add(DscPcdEntry) - elif len(Sku.VariableName) > 0: - HiiPcdArray.add(DscPcdEntry) - else: - OtherPcdArray.add(DscPcdEntry) - - # if the offset of a VPD is *, then it nee= d to be fixed up by third party tool. - VpdSkuMap[DscPcd] =3D SkuValueMap - if (self.Platform.FlashDefinition is None or self.Platform.Fla= shDefinition =3D=3D '') and \ - VpdFile.GetCount() !=3D 0: - EdkLogger.error("build", ATTRIBUTE_NOT_AVAILABLE, - "Fail to get FLASH_DEFINITION definition i= n DSC file %s which is required when DSC contains VPD PCD." % str(self.Plat= form.MetaFile)) - - if VpdFile.GetCount() !=3D 0: - - self.FixVpdOffset(VpdFile) - - self.FixVpdOffset(self.UpdateNVStoreMaxSize(VpdFile)) - PcdNvStoreDfBuffer =3D [item for item in self._DynamicPcdL= ist if item.TokenCName =3D=3D "PcdNvStoreDefaultValueBuffer" and item.Token= SpaceGuidCName =3D=3D "gEfiMdeModulePkgTokenSpaceGuid"] - if PcdNvStoreDfBuffer: - PcdName,PcdGuid =3D PcdNvStoreDfBuffer[0].TokenCName, = PcdNvStoreDfBuffer[0].TokenSpaceGuidCName - if (PcdName,PcdGuid) in VpdSkuMap: - DefaultSku =3D PcdNvStoreDfBuffer[0].SkuInfoList.g= et(TAB_DEFAULT) - VpdSkuMap[(PcdName,PcdGuid)] =3D {DefaultSku.Defau= ltValue:[SkuObj for SkuObj in PcdNvStoreDfBuffer[0].SkuInfoList.values() ]} - - # Process VPD map file generated by third party BPDG tool - if NeedProcessVpdMapFile: - VpdMapFilePath =3D os.path.join(self.BuildDir, TAB_FV_= DIRECTORY, "%s.map" % self.Platform.VpdToolGuid) - if os.path.exists(VpdMapFilePath): - VpdFile.Read(VpdMapFilePath) - - # Fixup TAB_STAR offset - for pcd in VpdSkuMap: - vpdinfo =3D VpdFile.GetVpdInfo(pcd) - if vpdinfo is None: - # just pick the a value to determine whether i= s unicode string type - continue - for pcdvalue in VpdSkuMap[pcd]: - for sku in VpdSkuMap[pcd][pcdvalue]: - for item in vpdinfo: - if item[2] =3D=3D pcdvalue: - sku.VpdOffset =3D item[1] - else: - EdkLogger.error("build", FILE_READ_FAILURE, "Can n= ot find VPD map file %s to fix up VPD offset." % VpdMapFilePath) - - # Delete the DynamicPcdList At the last time enter into this f= unction - for Pcd in self._DynamicPcdList: - # just pick the a value to determine whether is unicode st= ring type - Sku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) - Sku.VpdOffset =3D Sku.VpdOffset.strip() - - if Pcd.DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32= , TAB_UINT64, TAB_VOID, "BOOLEAN"]: - Pcd.DatumType =3D TAB_VOID - - PcdValue =3D Sku.DefaultValue - if Pcd.DatumType =3D=3D TAB_VOID and PcdValue.startswith("= L"): - # if found PCD which datum value is unicode string the= insert to left size of UnicodeIndex - UnicodePcdArray.add(Pcd) - elif len(Sku.VariableName) > 0: - # if found HII type PCD then insert to right of Unicod= eIndex - HiiPcdArray.add(Pcd) - else: - OtherPcdArray.add(Pcd) - del self._DynamicPcdList[:] - self._DynamicPcdList.extend(list(UnicodePcdArray)) - self._DynamicPcdList.extend(list(HiiPcdArray)) - self._DynamicPcdList.extend(list(OtherPcdArray)) - allskuset =3D [(SkuName, Sku.SkuId) for pcd in self._DynamicPcdLis= t for (SkuName, Sku) in pcd.SkuInfoList.items()] - for pcd in self._DynamicPcdList: - if len(pcd.SkuInfoList) =3D=3D 1: - for (SkuName, SkuId) in allskuset: - if isinstance(SkuId, str) and eval(SkuId) =3D=3D 0 or = SkuId =3D=3D 0: - continue - pcd.SkuInfoList[SkuName] =3D copy.deepcopy(pcd.SkuInfo= List[TAB_DEFAULT]) - pcd.SkuInfoList[SkuName].SkuId =3D SkuId - pcd.SkuInfoList[SkuName].SkuIdName =3D SkuName - - def FixVpdOffset(self, VpdFile ): - FvPath =3D os.path.join(self.BuildDir, TAB_FV_DIRECTORY) - if not os.path.exists(FvPath): - try: - os.makedirs(FvPath) - except: - EdkLogger.error("build", FILE_WRITE_FAILURE, "Fail to crea= te FV folder under %s" % self.BuildDir) - - VpdFilePath =3D os.path.join(FvPath, "%s.txt" % self.Platform.VpdT= oolGuid) - - if VpdFile.Write(VpdFilePath): - # retrieve BPDG tool's path from tool_def.txt according to VPD= _TOOL_GUID defined in DSC file. - BPDGToolName =3D None - for ToolDef in self.ToolDefinition.values(): - if TAB_GUID in ToolDef and ToolDef[TAB_GUID] =3D=3D self.P= latform.VpdToolGuid: - if "PATH" not in ToolDef: - EdkLogger.error("build", ATTRIBUTE_NOT_AVAILABLE, = "PATH attribute was not provided for BPDG guid tool %s in tools_def.txt" % = self.Platform.VpdToolGuid) - BPDGToolName =3D ToolDef["PATH"] - break - # Call third party GUID BPDG tool. - if BPDGToolName is not None: - VpdInfoFile.CallExtenalBPDGTool(BPDGToolName, VpdFilePath) - else: - EdkLogger.error("Build", FILE_NOT_FOUND, "Fail to find thi= rd-party BPDG tool to process VPD PCDs. BPDG Guid tool need to be defined i= n tools_def.txt and VPD_TOOL_GUID need to be provided in DSC file.") - - ## Return the platform build data object - @cached_property - def Platform(self): - return self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarg= et, self.ToolChain] - - ## Return platform name - @cached_property - def Name(self): - return self.Platform.PlatformName - - ## Return the meta file GUID - @cached_property - def Guid(self): - return self.Platform.Guid - - ## Return the platform version - @cached_property - def Version(self): - return self.Platform.Version - - ## Return the FDF file name - @cached_property - def FdfFile(self): - if self.Workspace.FdfFile: - RetVal=3D mws.join(self.WorkspaceDir, self.Workspace.FdfFile) - else: - RetVal =3D '' - return RetVal - - ## Return the build output directory platform specifies - @cached_property - def OutputDir(self): - return self.Platform.OutputDirectory - - ## Return the directory to store all intermediate and final files built - @cached_property - def BuildDir(self): - if os.path.isabs(self.OutputDir): - GlobalData.gBuildDirectory =3D RetVal =3D path.join( - path.abspath(self.OutputDir), - self.BuildTarget + "_" + self.Tool= Chain, - ) - else: - GlobalData.gBuildDirectory =3D RetVal =3D path.join( - self.WorkspaceDir, - self.OutputDir, - self.BuildTarget + "_" + self.Tool= Chain, - ) - return RetVal - - ## Return directory of platform makefile - # - # @retval string Makefile directory - # - @cached_property - def MakeFileDir(self): - return path.join(self.BuildDir, self.Arch) - - ## Return build command string - # - # @retval string Build command string - # - @cached_property - def BuildCommand(self): - RetVal =3D [] - if "MAKE" in self.ToolDefinition and "PATH" in self.ToolDefinition= ["MAKE"]: - RetVal +=3D _SplitOption(self.ToolDefinition["MAKE"]["PATH"]) - if "FLAGS" in self.ToolDefinition["MAKE"]: - NewOption =3D self.ToolDefinition["MAKE"]["FLAGS"].strip() - if NewOption !=3D '': - RetVal +=3D _SplitOption(NewOption) - if "MAKE" in self.EdkIIBuildOption: - if "FLAGS" in self.EdkIIBuildOption["MAKE"]: - Flags =3D self.EdkIIBuildOption["MAKE"]["FLAGS"] - if Flags.startswith('=3D'): - RetVal =3D [RetVal[0]] + [Flags[1:]] - else: - RetVal.append(Flags) - return RetVal - - ## Get tool chain definition - # - # Get each tool definition for given tool chain from tools_def.txt an= d platform - # - @cached_property - def ToolDefinition(self): - ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDictionary - if TAB_TOD_DEFINES_COMMAND_TYPE not in self.Workspace.ToolDef.Tool= sDefTxtDatabase: - EdkLogger.error('build', RESOURCE_NOT_AVAILABLE, "No tools fou= nd in configuration", - ExtraData=3D"[%s]" % self.MetaFile) - RetVal =3D OrderedDict() - DllPathList =3D set() - for Def in ToolDefinition: - Target, Tag, Arch, Tool, Attr =3D Def.split("_") - if Target !=3D self.BuildTarget or Tag !=3D self.ToolChain or = Arch !=3D self.Arch: - continue - - Value =3D ToolDefinition[Def] - # don't record the DLL - if Attr =3D=3D "DLL": - DllPathList.add(Value) - continue - - if Tool not in RetVal: - RetVal[Tool] =3D OrderedDict() - RetVal[Tool][Attr] =3D Value - - ToolsDef =3D '' - if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: - if "FLAGS" not in RetVal["MAKE"]: - RetVal["MAKE"]["FLAGS"] =3D "" - RetVal["MAKE"]["FLAGS"] +=3D " -s" - MakeFlags =3D '' - for Tool in RetVal: - for Attr in RetVal[Tool]: - Value =3D RetVal[Tool][Attr] - if Tool in self._BuildOptionWithToolDef(RetVal) and Attr i= n self._BuildOptionWithToolDef(RetVal)[Tool]: - # check if override is indicated - if self._BuildOptionWithToolDef(RetVal)[Tool][Attr].st= artswith('=3D'): - Value =3D self._BuildOptionWithToolDef(RetVal)[Too= l][Attr][1:] - else: - if Attr !=3D 'PATH': - Value +=3D " " + self._BuildOptionWithToolDef(= RetVal)[Tool][Attr] - else: - Value =3D self._BuildOptionWithToolDef(RetVal)= [Tool][Attr] - - if Attr =3D=3D "PATH": - # Don't put MAKE definition in the file - if Tool !=3D "MAKE": - ToolsDef +=3D "%s =3D %s\n" % (Tool, Value) - elif Attr !=3D "DLL": - # Don't put MAKE definition in the file - if Tool =3D=3D "MAKE": - if Attr =3D=3D "FLAGS": - MakeFlags =3D Value - else: - ToolsDef +=3D "%s_%s =3D %s\n" % (Tool, Attr, Valu= e) - ToolsDef +=3D "\n" - tool_def_file =3D os.path.join(self.MakeFileDir, "TOOLS_DEF." + se= lf.Arch) - SaveFileOnChange(tool_def_file, ToolsDef, False) - for DllPath in DllPathList: - os.environ["PATH"] =3D DllPath + os.pathsep + os.environ["PATH= "] - os.environ["MAKE_FLAGS"] =3D MakeFlags - - return RetVal - - ## Return the paths of tools - @cached_property - def ToolDefinitionFile(self): - tool_def_file =3D os.path.join(self.MakeFileDir, "TOOLS_DEF." + se= lf.Arch) - if not os.path.exists(tool_def_file): - self.ToolDefinition - return tool_def_file - - ## Retrieve the toolchain family of given toolchain tag. Default to 'M= SFT'. - @cached_property - def ToolChainFamily(self): - ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDatabase - if TAB_TOD_DEFINES_FAMILY not in ToolDefinition \ - or self.ToolChain not in ToolDefinition[TAB_TOD_DEFINES_FAMILY]= \ - or not ToolDefinition[TAB_TOD_DEFINES_FAMILY][self.ToolChain]: - EdkLogger.verbose("No tool chain family found in configuration= for %s. Default to MSFT." \ - % self.ToolChain) - RetVal =3D TAB_COMPILER_MSFT - else: - RetVal =3D ToolDefinition[TAB_TOD_DEFINES_FAMILY][self.ToolCha= in] - return RetVal - - @cached_property - def BuildRuleFamily(self): - ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDatabase - if TAB_TOD_DEFINES_BUILDRULEFAMILY not in ToolDefinition \ - or self.ToolChain not in ToolDefinition[TAB_TOD_DEFINES_BUILDRU= LEFAMILY] \ - or not ToolDefinition[TAB_TOD_DEFINES_BUILDRULEFAMILY][self.Too= lChain]: - EdkLogger.verbose("No tool chain family found in configuration= for %s. Default to MSFT." \ - % self.ToolChain) - return TAB_COMPILER_MSFT - - return ToolDefinition[TAB_TOD_DEFINES_BUILDRULEFAMILY][self.ToolCh= ain] - - ## Return the build options specific for all modules in this platform - @cached_property - def BuildOption(self): - return self._ExpandBuildOption(self.Platform.BuildOptions) - - def _BuildOptionWithToolDef(self, ToolDef): - return self._ExpandBuildOption(self.Platform.BuildOptions, ToolDef= =3DToolDef) - - ## Return the build options specific for EDK modules in this platform - @cached_property - def EdkBuildOption(self): - return self._ExpandBuildOption(self.Platform.BuildOptions, EDK_NAM= E) - - ## Return the build options specific for EDKII modules in this platform - @cached_property - def EdkIIBuildOption(self): - return self._ExpandBuildOption(self.Platform.BuildOptions, EDKII_N= AME) - - ## Summarize the packages used by modules in this platform - @cached_property - def PackageList(self): - RetVal =3D set() - for La in self.LibraryAutoGenList: - RetVal.update(La.DependentPackageList) - for Ma in self.ModuleAutoGenList: - RetVal.update(Ma.DependentPackageList) - #Collect package set information from INF of FDF - for ModuleFile in self._AsBuildModuleList: - if ModuleFile in self.Platform.Modules: - continue - ModuleData =3D self.BuildDatabase[ModuleFile, self.Arch, self.= BuildTarget, self.ToolChain] - RetVal.update(ModuleData.Packages) - return list(RetVal) - - @cached_property - def NonDynamicPcdDict(self): - return {(Pcd.TokenCName, Pcd.TokenSpaceGuidCName):Pcd for Pcd in s= elf.NonDynamicPcdList} - - ## Get list of non-dynamic PCDs - @property - def NonDynamicPcdList(self): - if not self._NonDynamicPcdList: - self.CollectPlatformDynamicPcds() - return self._NonDynamicPcdList - - ## Get list of dynamic PCDs - @property - def DynamicPcdList(self): - if not self._DynamicPcdList: - self.CollectPlatformDynamicPcds() - return self._DynamicPcdList - - ## Generate Token Number for all PCD - @cached_property - def PcdTokenNumber(self): - RetVal =3D OrderedDict() - TokenNumber =3D 1 - # - # Make the Dynamic and DynamicEx PCD use within different TokenNum= ber area. - # Such as: - # - # Dynamic PCD: - # TokenNumber 0 ~ 10 - # DynamicEx PCD: - # TokeNumber 11 ~ 20 - # - for Pcd in self.DynamicPcdList: - if Pcd.Phase =3D=3D "PEI" and Pcd.Type in PCD_DYNAMIC_TYPE_SET: - EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) - RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber - TokenNumber +=3D 1 - - for Pcd in self.DynamicPcdList: - if Pcd.Phase =3D=3D "PEI" and Pcd.Type in PCD_DYNAMIC_EX_TYPE_= SET: - EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) - RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber - TokenNumber +=3D 1 - - for Pcd in self.DynamicPcdList: - if Pcd.Phase =3D=3D "DXE" and Pcd.Type in PCD_DYNAMIC_TYPE_SET: - EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) - RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber - TokenNumber +=3D 1 - - for Pcd in self.DynamicPcdList: - if Pcd.Phase =3D=3D "DXE" and Pcd.Type in PCD_DYNAMIC_EX_TYPE_= SET: - EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) - RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber - TokenNumber +=3D 1 - - for Pcd in self.NonDynamicPcdList: - RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenNumber - TokenNumber +=3D 1 - return RetVal - - @cached_property - def _MaList(self): - for ModuleFile in self.Platform.Modules: - Ma =3D ModuleAutoGen( - self.Workspace, - ModuleFile, - self.BuildTarget, - self.ToolChain, - self.Arch, - self.MetaFile - ) - self.Platform.Modules[ModuleFile].M =3D Ma - return [x.M for x in self.Platform.Modules.values()] - - ## Summarize ModuleAutoGen objects of all modules to be built for this= platform - @cached_property - def ModuleAutoGenList(self): - RetVal =3D [] - for Ma in self._MaList: - if Ma not in RetVal: - RetVal.append(Ma) - return RetVal - - ## Summarize ModuleAutoGen objects of all libraries to be built for th= is platform - @cached_property - def LibraryAutoGenList(self): - RetVal =3D [] - for Ma in self._MaList: - for La in Ma.LibraryAutoGenList: - if La not in RetVal: - RetVal.append(La) - if Ma not in La.ReferenceModules: - La.ReferenceModules.append(Ma) - return RetVal - - ## Test if a module is supported by the platform - # - # An error will be raised directly if the module or its arch is not s= upported - # by the platform or current configuration - # - def ValidModule(self, Module): - return Module in self.Platform.Modules or Module in self.Platform.= LibraryInstances \ - or Module in self._AsBuildModuleList - - ## Resolve the library classes in a module to library instances - # - # This method will not only resolve library classes but also sort the = library - # instances according to the dependency-ship. - # - # @param Module The module from which the library classes will= be resolved - # - # @retval library_list List of library instances sorted - # - def ApplyLibraryInstance(self, Module): - # Cover the case that the binary INF file is list in the FDF file = but not DSC file, return empty list directly - if str(Module) not in self.Platform.Modules: - return [] - - return GetModuleLibInstances(Module, - self.Platform, - self.BuildDatabase, - self.Arch, - self.BuildTarget, - self.ToolChain, - self.MetaFile, - EdkLogger) - - ## Override PCD setting (type, value, ...) - # - # @param ToPcd The PCD to be overridden - # @param FromPcd The PCD overriding from - # - def _OverridePcd(self, ToPcd, FromPcd, Module=3D"", Msg=3D"", Library= =3D""): - # - # in case there's PCDs coming from FDF file, which have no type gi= ven. - # at this point, ToPcd.Type has the type found from dependent - # package - # - TokenCName =3D ToPcd.TokenCName - for PcdItem in GlobalData.MixedPcd: - if (ToPcd.TokenCName, ToPcd.TokenSpaceGuidCName) in GlobalData= .MixedPcd[PcdItem]: - TokenCName =3D PcdItem[0] - break - if FromPcd is not None: - if ToPcd.Pending and FromPcd.Type: - ToPcd.Type =3D FromPcd.Type - elif ToPcd.Type and FromPcd.Type\ - and ToPcd.Type !=3D FromPcd.Type and ToPcd.Type in FromPcd= .Type: - if ToPcd.Type.strip() =3D=3D TAB_PCDS_DYNAMIC_EX: - ToPcd.Type =3D FromPcd.Type - elif ToPcd.Type and FromPcd.Type \ - and ToPcd.Type !=3D FromPcd.Type: - if Library: - Module =3D str(Module) + " 's library file (" + str(Li= brary) + ")" - EdkLogger.error("build", OPTION_CONFLICT, "Mismatched PCD = type", - ExtraData=3D"%s.%s is used as [%s] in modu= le %s, but as [%s] in %s."\ - % (ToPcd.TokenSpaceGuidCName, To= kenCName, - ToPcd.Type, Module, FromPcd.T= ype, Msg), - File=3Dself.MetaFile) - - if FromPcd.MaxDatumSize: - ToPcd.MaxDatumSize =3D FromPcd.MaxDatumSize - ToPcd.MaxSizeUserSet =3D FromPcd.MaxDatumSize - if FromPcd.DefaultValue: - ToPcd.DefaultValue =3D FromPcd.DefaultValue - if FromPcd.TokenValue: - ToPcd.TokenValue =3D FromPcd.TokenValue - if FromPcd.DatumType: - ToPcd.DatumType =3D FromPcd.DatumType - if FromPcd.SkuInfoList: - ToPcd.SkuInfoList =3D FromPcd.SkuInfoList - if FromPcd.UserDefinedDefaultStoresFlag: - ToPcd.UserDefinedDefaultStoresFlag =3D FromPcd.UserDefined= DefaultStoresFlag - # Add Flexible PCD format parse - if ToPcd.DefaultValue: - try: - ToPcd.DefaultValue =3D ValueExpressionEx(ToPcd.Default= Value, ToPcd.DatumType, self.Workspace._GuidDict)(True) - except BadExpression as Value: - EdkLogger.error('Parser', FORMAT_INVALID, 'PCD [%s.%s]= Value "%s", %s' %(ToPcd.TokenSpaceGuidCName, ToPcd.TokenCName, ToPcd.Defau= ltValue, Value), - File=3Dself.MetaFile) - - # check the validation of datum - IsValid, Cause =3D CheckPcdDatum(ToPcd.DatumType, ToPcd.Defaul= tValue) - if not IsValid: - EdkLogger.error('build', FORMAT_INVALID, Cause, File=3Dsel= f.MetaFile, - ExtraData=3D"%s.%s" % (ToPcd.TokenSpaceGui= dCName, TokenCName)) - ToPcd.validateranges =3D FromPcd.validateranges - ToPcd.validlists =3D FromPcd.validlists - ToPcd.expressions =3D FromPcd.expressions - ToPcd.CustomAttribute =3D FromPcd.CustomAttribute - - if FromPcd is not None and ToPcd.DatumType =3D=3D TAB_VOID and not= ToPcd.MaxDatumSize: - EdkLogger.debug(EdkLogger.DEBUG_9, "No MaxDatumSize specified = for PCD %s.%s" \ - % (ToPcd.TokenSpaceGuidCName, TokenCName)) - Value =3D ToPcd.DefaultValue - if not Value: - ToPcd.MaxDatumSize =3D '1' - elif Value[0] =3D=3D 'L': - ToPcd.MaxDatumSize =3D str((len(Value) - 2) * 2) - elif Value[0] =3D=3D '{': - ToPcd.MaxDatumSize =3D str(len(Value.split(','))) - else: - ToPcd.MaxDatumSize =3D str(len(Value) - 1) - - # apply default SKU for dynamic PCDS if specified one is not avail= able - if (ToPcd.Type in PCD_DYNAMIC_TYPE_SET or ToPcd.Type in PCD_DYNAMI= C_EX_TYPE_SET) \ - and not ToPcd.SkuInfoList: - if self.Platform.SkuName in self.Platform.SkuIds: - SkuName =3D self.Platform.SkuName - else: - SkuName =3D TAB_DEFAULT - ToPcd.SkuInfoList =3D { - SkuName : SkuInfoClass(SkuName, self.Platform.SkuIds[SkuNa= me][0], '', '', '', '', '', ToPcd.DefaultValue) - } - - ## Apply PCD setting defined platform to a module - # - # @param Module The module from which the PCD setting will be over= ridden - # - # @retval PCD_list The list PCDs with settings from platform - # - def ApplyPcdSetting(self, Module, Pcds, Library=3D""): - # for each PCD in module - for Name, Guid in Pcds: - PcdInModule =3D Pcds[Name, Guid] - # find out the PCD setting in platform - if (Name, Guid) in self.Platform.Pcds: - PcdInPlatform =3D self.Platform.Pcds[Name, Guid] - else: - PcdInPlatform =3D None - # then override the settings if any - self._OverridePcd(PcdInModule, PcdInPlatform, Module, Msg=3D"D= SC PCD sections", Library=3DLibrary) - # resolve the VariableGuid value - for SkuId in PcdInModule.SkuInfoList: - Sku =3D PcdInModule.SkuInfoList[SkuId] - if Sku.VariableGuid =3D=3D '': continue - Sku.VariableGuidValue =3D GuidValue(Sku.VariableGuid, self= .PackageList, self.MetaFile.Path) - if Sku.VariableGuidValue is None: - PackageList =3D "\n\t".join(str(P) for P in self.Packa= geList) - EdkLogger.error( - 'build', - RESOURCE_NOT_AVAILABLE, - "Value of GUID [%s] is not found in" % Sku= .VariableGuid, - ExtraData=3DPackageList + "\n\t(used with = %s.%s from module %s)" \ - % (Guid, Name, str= (Module)), - File=3Dself.MetaFile - ) - - # override PCD settings with module specific setting - if Module in self.Platform.Modules: - PlatformModule =3D self.Platform.Modules[str(Module)] - for Key in PlatformModule.Pcds: - if GlobalData.BuildOptionPcd: - for pcd in GlobalData.BuildOptionPcd: - (TokenSpaceGuidCName, TokenCName, FieldName, pcdva= lue, _) =3D pcd - if (TokenCName, TokenSpaceGuidCName) =3D=3D Key an= d FieldName =3D=3D"": - PlatformModule.Pcds[Key].DefaultValue =3D pcdv= alue - PlatformModule.Pcds[Key].PcdValueFromComm =3D = pcdvalue - break - Flag =3D False - if Key in Pcds: - ToPcd =3D Pcds[Key] - Flag =3D True - elif Key in GlobalData.MixedPcd: - for PcdItem in GlobalData.MixedPcd[Key]: - if PcdItem in Pcds: - ToPcd =3D Pcds[PcdItem] - Flag =3D True - break - if Flag: - self._OverridePcd(ToPcd, PlatformModule.Pcds[Key], Mod= ule, Msg=3D"DSC Components Module scoped PCD section", Library=3DLibrary) - # use PCD value to calculate the MaxDatumSize when it is not speci= fied - for Name, Guid in Pcds: - Pcd =3D Pcds[Name, Guid] - if Pcd.DatumType =3D=3D TAB_VOID and not Pcd.MaxDatumSize: - Pcd.MaxSizeUserSet =3D None - Value =3D Pcd.DefaultValue - if not Value: - Pcd.MaxDatumSize =3D '1' - elif Value[0] =3D=3D 'L': - Pcd.MaxDatumSize =3D str((len(Value) - 2) * 2) - elif Value[0] =3D=3D '{': - Pcd.MaxDatumSize =3D str(len(Value.split(','))) - else: - Pcd.MaxDatumSize =3D str(len(Value) - 1) - return list(Pcds.values()) - - - - ## Calculate the priority value of the build option - # - # @param Key Build option definition contain: TARGET_TOOLCHAIN_A= RCH_COMMANDTYPE_ATTRIBUTE - # - # @retval Value Priority value based on the priority list. - # - def CalculatePriorityValue(self, Key): - Target, ToolChain, Arch, CommandType, Attr =3D Key.split('_') - PriorityValue =3D 0x11111 - if Target =3D=3D TAB_STAR: - PriorityValue &=3D 0x01111 - if ToolChain =3D=3D TAB_STAR: - PriorityValue &=3D 0x10111 - if Arch =3D=3D TAB_STAR: - PriorityValue &=3D 0x11011 - if CommandType =3D=3D TAB_STAR: - PriorityValue &=3D 0x11101 - if Attr =3D=3D TAB_STAR: - PriorityValue &=3D 0x11110 - - return self.PrioList["0x%0.5x" % PriorityValue] - - - ## Expand * in build option key - # - # @param Options Options to be expanded - # @param ToolDef Use specified ToolDef instead of full version. - # This is needed during initialization to prevent - # infinite recursion betweeh BuildOptions, - # ToolDefinition, and this function. - # - # @retval options Options expanded - # - def _ExpandBuildOption(self, Options, ModuleStyle=3DNone, ToolDef=3DNo= ne): - if not ToolDef: - ToolDef =3D self.ToolDefinition - BuildOptions =3D {} - FamilyMatch =3D False - FamilyIsNull =3D True - - OverrideList =3D {} - # - # Construct a list contain the build options which need override. - # - for Key in Options: - # - # Key[0] -- tool family - # Key[1] -- TARGET_TOOLCHAIN_ARCH_COMMANDTYPE_ATTRIBUTE - # - if (Key[0] =3D=3D self.BuildRuleFamily and - (ModuleStyle is None or len(Key) < 3 or (len(Key) > 2 and = Key[2] =3D=3D ModuleStyle))): - Target, ToolChain, Arch, CommandType, Attr =3D Key[1].spli= t('_') - if (Target =3D=3D self.BuildTarget or Target =3D=3D TAB_ST= AR) and\ - (ToolChain =3D=3D self.ToolChain or ToolChain =3D=3D T= AB_STAR) and\ - (Arch =3D=3D self.Arch or Arch =3D=3D TAB_STAR) and\ - Options[Key].startswith("=3D"): - - if OverrideList.get(Key[1]) is not None: - OverrideList.pop(Key[1]) - OverrideList[Key[1]] =3D Options[Key] - - # - # Use the highest priority value. - # - if (len(OverrideList) >=3D 2): - KeyList =3D list(OverrideList.keys()) - for Index in range(len(KeyList)): - NowKey =3D KeyList[Index] - Target1, ToolChain1, Arch1, CommandType1, Attr1 =3D NowKey= .split("_") - for Index1 in range(len(KeyList) - Index - 1): - NextKey =3D KeyList[Index1 + Index + 1] - # - # Compare two Key, if one is included by another, choo= se the higher priority one - # - Target2, ToolChain2, Arch2, CommandType2, Attr2 =3D Ne= xtKey.split("_") - if (Target1 =3D=3D Target2 or Target1 =3D=3D TAB_STAR = or Target2 =3D=3D TAB_STAR) and\ - (ToolChain1 =3D=3D ToolChain2 or ToolChain1 =3D=3D= TAB_STAR or ToolChain2 =3D=3D TAB_STAR) and\ - (Arch1 =3D=3D Arch2 or Arch1 =3D=3D TAB_STAR or Ar= ch2 =3D=3D TAB_STAR) and\ - (CommandType1 =3D=3D CommandType2 or CommandType1 = =3D=3D TAB_STAR or CommandType2 =3D=3D TAB_STAR) and\ - (Attr1 =3D=3D Attr2 or Attr1 =3D=3D TAB_STAR or At= tr2 =3D=3D TAB_STAR): - - if self.CalculatePriorityValue(NowKey) > self.Calc= ulatePriorityValue(NextKey): - if Options.get((self.BuildRuleFamily, NextKey)= ) is not None: - Options.pop((self.BuildRuleFamily, NextKey= )) - else: - if Options.get((self.BuildRuleFamily, NowKey))= is not None: - Options.pop((self.BuildRuleFamily, NowKey)) - - for Key in Options: - if ModuleStyle is not None and len (Key) > 2: - # Check Module style is EDK or EDKII. - # Only append build option for the matched style module. - if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: - continue - elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: - continue - Family =3D Key[0] - Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") - # if tool chain family doesn't match, skip it - if Tool in ToolDef and Family !=3D "": - FamilyIsNull =3D False - if ToolDef[Tool].get(TAB_TOD_DEFINES_BUILDRULEFAMILY, "") = !=3D "": - if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_BUILDRULE= FAMILY]: - continue - elif Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: - continue - FamilyMatch =3D True - # expand any wildcard - if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: - if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: - if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: - if Tool not in BuildOptions: - BuildOptions[Tool] =3D {} - if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): - BuildOptions[Tool][Attr] =3D Options[Key] - else: - # append options for the same tool except PATH - if Attr !=3D 'PATH': - BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] - else: - BuildOptions[Tool][Attr] =3D Options[Key] - # Build Option Family has been checked, which need't to be checked= again for family. - if FamilyMatch or FamilyIsNull: - return BuildOptions - - for Key in Options: - if ModuleStyle is not None and len (Key) > 2: - # Check Module style is EDK or EDKII. - # Only append build option for the matched style module. - if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: - continue - elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: - continue - Family =3D Key[0] - Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") - # if tool chain family doesn't match, skip it - if Tool not in ToolDef or Family =3D=3D "": - continue - # option has been added before - if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: - continue - - # expand any wildcard - if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: - if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: - if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: - if Tool not in BuildOptions: - BuildOptions[Tool] =3D {} - if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): - BuildOptions[Tool][Attr] =3D Options[Key] - else: - # append options for the same tool except PATH - if Attr !=3D 'PATH': - BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] - else: - BuildOptions[Tool][Attr] =3D Options[Key] - return BuildOptions - def GetGlobalBuildOptions(self,Module): - ModuleTypeOptions =3D self.Platform.GetBuildOptionsByPkg(Module, M= odule.ModuleType) - ModuleTypeOptions =3D self._ExpandBuildOption(ModuleTypeOptions) - if Module in self.Platform.Modules: - PlatformModule =3D self.Platform.Modules[str(Module)] - PlatformModuleOptions =3D self._ExpandBuildOption(PlatformModu= le.BuildOptions) - else: - PlatformModuleOptions =3D {} - return ModuleTypeOptions, PlatformModuleOptions - ## Append build options in platform to a module - # - # @param Module The module to which the build options will be appe= nded - # - # @retval options The options appended with build options in pla= tform - # - def ApplyBuildOption(self, Module): - # Get the different options for the different style module - PlatformOptions =3D self.EdkIIBuildOption - ModuleTypeOptions =3D self.Platform.GetBuildOptionsByModuleType(ED= KII_NAME, Module.ModuleType) - ModuleTypeOptions =3D self._ExpandBuildOption(ModuleTypeOptions) - ModuleOptions =3D self._ExpandBuildOption(Module.BuildOptions) - if Module in self.Platform.Modules: - PlatformModule =3D self.Platform.Modules[str(Module)] - PlatformModuleOptions =3D self._ExpandBuildOption(PlatformModu= le.BuildOptions) - else: - PlatformModuleOptions =3D {} - - BuildRuleOrder =3D None - for Options in [self.ToolDefinition, ModuleOptions, PlatformOption= s, ModuleTypeOptions, PlatformModuleOptions]: - for Tool in Options: - for Attr in Options[Tool]: - if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: - BuildRuleOrder =3D Options[Tool][Attr] - - AllTools =3D set(list(ModuleOptions.keys()) + list(PlatformOptions= .keys()) + - list(PlatformModuleOptions.keys()) + list(ModuleTyp= eOptions.keys()) + - list(self.ToolDefinition.keys())) - BuildOptions =3D defaultdict(lambda: defaultdict(str)) - for Tool in AllTools: - for Options in [self.ToolDefinition, ModuleOptions, PlatformOp= tions, ModuleTypeOptions, PlatformModuleOptions]: - if Tool not in Options: - continue - for Attr in Options[Tool]: - # - # Do not generate it in Makefile - # - if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: - continue - Value =3D Options[Tool][Attr] - # check if override is indicated - if Value.startswith('=3D'): - BuildOptions[Tool][Attr] =3D mws.handleWsMacro(Val= ue[1:]) - else: - if Attr !=3D 'PATH': - BuildOptions[Tool][Attr] +=3D " " + mws.handle= WsMacro(Value) - else: - BuildOptions[Tool][Attr] =3D mws.handleWsMacro= (Value) - - return BuildOptions, BuildRuleOrder - -# -# extend lists contained in a dictionary with lists stored in another dict= ionary -# if CopyToDict is not derived from DefaultDict(list) then this may raise = exception -# -def ExtendCopyDictionaryLists(CopyToDict, CopyFromDict): - for Key in CopyFromDict: - CopyToDict[Key].extend(CopyFromDict[Key]) - -# Create a directory specified by a set of path elements and return the fu= ll path -def _MakeDir(PathList): - RetVal =3D path.join(*PathList) - CreateDirectory(RetVal) - return RetVal - -## ModuleAutoGen class -# -# This class encapsules the AutoGen behaviors for the build tools. In addi= tion to -# the generation of AutoGen.h and AutoGen.c, it will generate *.depex file= according -# to the [depex] section in module's inf file. -# -class ModuleAutoGen(AutoGen): - # call super().__init__ then call the worker function with different p= arameter count - def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): - if not hasattr(self, "_Init"): - self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,= *args) - self._Init =3D True - - ## Cache the timestamps of metafiles of every module in a class attrib= ute - # - TimeDict =3D {} - - def __new__(cls, Workspace, MetaFile, Target, Toolchain, Arch, *args, = **kwargs): - # check if this module is employed by active platform - if not PlatformAutoGen(Workspace, args[0], Target, Toolchain, Arch= ).ValidModule(MetaFile): - EdkLogger.verbose("Module [%s] for [%s] is not employed by act= ive platform\n" \ - % (MetaFile, Arch)) - return None - return super(ModuleAutoGen, cls).__new__(cls, Workspace, MetaFile,= Target, Toolchain, Arch, *args, **kwargs) - - ## Initialize ModuleAutoGen - # - # @param Workspace EdkIIWorkspaceBuild object - # @param ModuleFile The path of module file - # @param Target Build target (DEBUG, RELEASE) - # @param Toolchain Name of tool chain - # @param Arch The arch the module supports - # @param PlatformFile Platform meta-file - # - def _InitWorker(self, Workspace, ModuleFile, Target, Toolchain, Arch, = PlatformFile): - EdkLogger.debug(EdkLogger.DEBUG_9, "AutoGen module [%s] [%s]" % (M= oduleFile, Arch)) - GlobalData.gProcessingFile =3D "%s [%s, %s, %s]" % (ModuleFile, Ar= ch, Toolchain, Target) - - self.Workspace =3D Workspace - self.WorkspaceDir =3D Workspace.WorkspaceDir - self.MetaFile =3D ModuleFile - self.PlatformInfo =3D PlatformAutoGen(Workspace, PlatformFile, Tar= get, Toolchain, Arch) - - self.SourceDir =3D self.MetaFile.SubDir - self.SourceDir =3D mws.relpath(self.SourceDir, self.WorkspaceDir) - - self.ToolChain =3D Toolchain - self.BuildTarget =3D Target - self.Arch =3D Arch - self.ToolChainFamily =3D self.PlatformInfo.ToolChainFamily - self.BuildRuleFamily =3D self.PlatformInfo.BuildRuleFamily - - self.IsCodeFileCreated =3D False - self.IsAsBuiltInfCreated =3D False - self.DepexGenerated =3D False - - self.BuildDatabase =3D self.Workspace.BuildDatabase - self.BuildRuleOrder =3D None - self.BuildTime =3D 0 - - self._PcdComments =3D OrderedListDict() - self._GuidComments =3D OrderedListDict() - self._ProtocolComments =3D OrderedListDict() - self._PpiComments =3D OrderedListDict() - self._BuildTargets =3D None - self._IntroBuildTargetList =3D None - self._FinalBuildTargetList =3D None - self._FileTypes =3D None - - self.AutoGenDepSet =3D set() - self.ReferenceModules =3D [] - self.ConstPcd =3D {} - - ## hash() operator of ModuleAutoGen - # - # The module file path and arch string will be used to represent - # hash value of this object - # - # @retval int Hash value of the module file path and arch - # - @cached_class_function - def __hash__(self): - return hash((self.MetaFile, self.Arch)) - - def __repr__(self): - return "%s [%s]" % (self.MetaFile, self.Arch) - - # Get FixedAtBuild Pcds of this Module - @cached_property - def FixedAtBuildPcds(self): - RetVal =3D [] - for Pcd in self.ModulePcdList: - if Pcd.Type !=3D TAB_PCDS_FIXED_AT_BUILD: - continue - if Pcd not in RetVal: - RetVal.append(Pcd) - return RetVal - - @cached_property - def FixedVoidTypePcds(self): - RetVal =3D {} - for Pcd in self.FixedAtBuildPcds: - if Pcd.DatumType =3D=3D TAB_VOID: - if '{}.{}'.format(Pcd.TokenSpaceGuidCName, Pcd.TokenCName)= not in RetVal: - RetVal['{}.{}'.format(Pcd.TokenSpaceGuidCName, Pcd.Tok= enCName)] =3D Pcd.DefaultValue - return RetVal - - @property - def UniqueBaseName(self): - BaseName =3D self.Name - for Module in self.PlatformInfo.ModuleAutoGenList: - if Module.MetaFile =3D=3D self.MetaFile: - continue - if Module.Name =3D=3D self.Name: - if uuid.UUID(Module.Guid) =3D=3D uuid.UUID(self.Guid): - EdkLogger.error("build", FILE_DUPLICATED, 'Modules hav= e same BaseName and FILE_GUID:\n' - ' %s\n %s' % (Module.MetaFile, self.= MetaFile)) - BaseName =3D '%s_%s' % (self.Name, self.Guid) - return BaseName - - # Macros could be used in build_rule.txt (also Makefile) - @cached_property - def Macros(self): - return OrderedDict(( - ("WORKSPACE" ,self.WorkspaceDir), - ("MODULE_NAME" ,self.Name), - ("MODULE_NAME_GUID" ,self.UniqueBaseName), - ("MODULE_GUID" ,self.Guid), - ("MODULE_VERSION" ,self.Version), - ("MODULE_TYPE" ,self.ModuleType), - ("MODULE_FILE" ,str(self.MetaFile)), - ("MODULE_FILE_BASE_NAME" ,self.MetaFile.BaseName), - ("MODULE_RELATIVE_DIR" ,self.SourceDir), - ("MODULE_DIR" ,self.SourceDir), - ("BASE_NAME" ,self.Name), - ("ARCH" ,self.Arch), - ("TOOLCHAIN" ,self.ToolChain), - ("TOOLCHAIN_TAG" ,self.ToolChain), - ("TOOL_CHAIN_TAG" ,self.ToolChain), - ("TARGET" ,self.BuildTarget), - ("BUILD_DIR" ,self.PlatformInfo.BuildDir), - ("BIN_DIR" ,os.path.join(self.PlatformInfo.BuildDir, self.Arch= )), - ("LIB_DIR" ,os.path.join(self.PlatformInfo.BuildDir, self.Arch= )), - ("MODULE_BUILD_DIR" ,self.BuildDir), - ("OUTPUT_DIR" ,self.OutputDir), - ("DEBUG_DIR" ,self.DebugDir), - ("DEST_DIR_OUTPUT" ,self.OutputDir), - ("DEST_DIR_DEBUG" ,self.DebugDir), - ("PLATFORM_NAME" ,self.PlatformInfo.Name), - ("PLATFORM_GUID" ,self.PlatformInfo.Guid), - ("PLATFORM_VERSION" ,self.PlatformInfo.Version), - ("PLATFORM_RELATIVE_DIR" ,self.PlatformInfo.SourceDir), - ("PLATFORM_DIR" ,mws.join(self.WorkspaceDir, self.PlatformInfo= .SourceDir)), - ("PLATFORM_OUTPUT_DIR" ,self.PlatformInfo.OutputDir), - ("FFS_OUTPUT_DIR" ,self.FfsOutputDir) - )) - - ## Return the module build data object - @cached_property - def Module(self): - return self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarg= et, self.ToolChain] - - ## Return the module name - @cached_property - def Name(self): - return self.Module.BaseName - - ## Return the module DxsFile if exist - @cached_property - def DxsFile(self): - return self.Module.DxsFile - - ## Return the module meta-file GUID - @cached_property - def Guid(self): - # - # To build same module more than once, the module path with FILE_G= UID overridden has - # the file name FILE_GUIDmodule.inf, but the relative path (self.M= etaFile.File) is the real path - # in DSC. The overridden GUID can be retrieved from file name - # - if os.path.basename(self.MetaFile.File) !=3D os.path.basename(self= .MetaFile.Path): - # - # Length of GUID is 36 - # - return os.path.basename(self.MetaFile.Path)[:36] - return self.Module.Guid - - ## Return the module version - @cached_property - def Version(self): - return self.Module.Version - - ## Return the module type - @cached_property - def ModuleType(self): - return self.Module.ModuleType - - ## Return the component type (for Edk.x style of module) - @cached_property - def ComponentType(self): - return self.Module.ComponentType - - ## Return the build type - @cached_property - def BuildType(self): - return self.Module.BuildType - - ## Return the PCD_IS_DRIVER setting - @cached_property - def PcdIsDriver(self): - return self.Module.PcdIsDriver - - ## Return the autogen version, i.e. module meta-file version - @cached_property - def AutoGenVersion(self): - return self.Module.AutoGenVersion - - ## Check if the module is library or not - @cached_property - def IsLibrary(self): - return bool(self.Module.LibraryClass) - - ## Check if the module is binary module or not - @cached_property - def IsBinaryModule(self): - return self.Module.IsBinaryModule - - ## Return the directory to store intermediate files of the module - @cached_property - def BuildDir(self): - return _MakeDir(( - self.PlatformInfo.BuildDir, - self.Arch, - self.SourceDir, - self.MetaFile.BaseName - )) - - ## Return the directory to store the intermediate object files of the = module - @cached_property - def OutputDir(self): - return _MakeDir((self.BuildDir, "OUTPUT")) - - ## Return the directory path to store ffs file - @cached_property - def FfsOutputDir(self): - if GlobalData.gFdfParser: - return path.join(self.PlatformInfo.BuildDir, TAB_FV_DIRECTORY,= "Ffs", self.Guid + self.Name) - return '' - - ## Return the directory to store auto-gened source files of the module - @cached_property - def DebugDir(self): - return _MakeDir((self.BuildDir, "DEBUG")) - - ## Return the path of custom file - @cached_property - def CustomMakefile(self): - RetVal =3D {} - for Type in self.Module.CustomMakefile: - MakeType =3D gMakeTypeMap[Type] if Type in gMakeTypeMap else '= nmake' - File =3D os.path.join(self.SourceDir, self.Module.CustomMakefi= le[Type]) - RetVal[MakeType] =3D File - return RetVal - - ## Return the directory of the makefile - # - # @retval string The directory string of module's makefile - # - @cached_property - def MakeFileDir(self): - return self.BuildDir - - ## Return build command string - # - # @retval string Build command string - # - @cached_property - def BuildCommand(self): - return self.PlatformInfo.BuildCommand - - ## Get object list of all packages the module and its dependent librar= ies belong to - # - # @retval list The list of package object - # - @cached_property - def DerivedPackageList(self): - PackageList =3D [] - for M in [self.Module] + self.DependentLibraryList: - for Package in M.Packages: - if Package in PackageList: - continue - PackageList.append(Package) - return PackageList - - ## Get the depex string - # - # @return : a string contain all depex expression. - def _GetDepexExpresionString(self): - DepexStr =3D '' - DepexList =3D [] - ## DPX_SOURCE IN Define section. - if self.Module.DxsFile: - return DepexStr - for M in [self.Module] + self.DependentLibraryList: - Filename =3D M.MetaFile.Path - InfObj =3D InfSectionParser.InfSectionParser(Filename) - DepexExpressionList =3D InfObj.GetDepexExpresionList() - for DepexExpression in DepexExpressionList: - for key in DepexExpression: - Arch, ModuleType =3D key - DepexExpr =3D [x for x in DepexExpression[key] if not = str(x).startswith('#')] - # the type of build module is USER_DEFINED. - # All different DEPEX section tags would be copied int= o the As Built INF file - # and there would be separate DEPEX section tags - if self.ModuleType.upper() =3D=3D SUP_MODULE_USER_DEFI= NED or self.ModuleType.upper() =3D=3D SUP_MODULE_HOST_APPLICATION: - if (Arch.upper() =3D=3D self.Arch.upper()) and (Mo= duleType.upper() !=3D TAB_ARCH_COMMON): - DepexList.append({(Arch, ModuleType): DepexExp= r}) - else: - if Arch.upper() =3D=3D TAB_ARCH_COMMON or \ - (Arch.upper() =3D=3D self.Arch.upper() and \ - ModuleType.upper() in [TAB_ARCH_COMMON, self.Mod= uleType.upper()]): - DepexList.append({(Arch, ModuleType): DepexExp= r}) - - #the type of build module is USER_DEFINED. - if self.ModuleType.upper() =3D=3D SUP_MODULE_USER_DEFINED or self.= ModuleType.upper() =3D=3D SUP_MODULE_HOST_APPLICATION: - for Depex in DepexList: - for key in Depex: - DepexStr +=3D '[Depex.%s.%s]\n' % key - DepexStr +=3D '\n'.join('# '+ val for val in Depex[key= ]) - DepexStr +=3D '\n\n' - if not DepexStr: - return '[Depex.%s]\n' % self.Arch - return DepexStr - - #the type of build module not is USER_DEFINED. - Count =3D 0 - for Depex in DepexList: - Count +=3D 1 - if DepexStr !=3D '': - DepexStr +=3D ' AND ' - DepexStr +=3D '(' - for D in Depex.values(): - DepexStr +=3D ' '.join(val for val in D) - Index =3D DepexStr.find('END') - if Index > -1 and Index =3D=3D len(DepexStr) - 3: - DepexStr =3D DepexStr[:-3] - DepexStr =3D DepexStr.strip() - DepexStr +=3D ')' - if Count =3D=3D 1: - DepexStr =3D DepexStr.lstrip('(').rstrip(')').strip() - if not DepexStr: - return '[Depex.%s]\n' % self.Arch - return '[Depex.%s]\n# ' % self.Arch + DepexStr - - ## Merge dependency expression - # - # @retval list The token list of the dependency expression af= ter parsed - # - @cached_property - def DepexList(self): - if self.DxsFile or self.IsLibrary or TAB_DEPENDENCY_EXPRESSION_FIL= E in self.FileTypes: - return {} - - DepexList =3D [] - # - # Append depex from dependent libraries, if not "BEFORE", "AFTER" = expression - # - for M in [self.Module] + self.DependentLibraryList: - Inherited =3D False - for D in M.Depex[self.Arch, self.ModuleType]: - if DepexList !=3D []: - DepexList.append('AND') - DepexList.append('(') - #replace D with value if D is FixedAtBuild PCD - NewList =3D [] - for item in D: - if '.' not in item: - NewList.append(item) - else: - FixedVoidTypePcds =3D {} - if item in self.FixedVoidTypePcds: - FixedVoidTypePcds =3D self.FixedVoidTypePcds - elif M in self.PlatformInfo.LibraryAutoGenList: - Index =3D self.PlatformInfo.LibraryAutoGenList= .index(M) - FixedVoidTypePcds =3D self.PlatformInfo.Librar= yAutoGenList[Index].FixedVoidTypePcds - if item not in FixedVoidTypePcds: - EdkLogger.error("build", FORMAT_INVALID, "{} u= sed in [Depex] section should be used as FixedAtBuild type and VOID* datum = type in the module.".format(item)) - else: - Value =3D FixedVoidTypePcds[item] - if len(Value.split(',')) !=3D 16: - EdkLogger.error("build", FORMAT_INVALID, - "{} used in [Depex] sectio= n should be used as FixedAtBuild type and VOID* datum type and 16 bytes in = the module.".format(item)) - NewList.append(Value) - DepexList.extend(NewList) - if DepexList[-1] =3D=3D 'END': # no need of a END at this= time - DepexList.pop() - DepexList.append(')') - Inherited =3D True - if Inherited: - EdkLogger.verbose("DEPEX[%s] (+%s) =3D %s" % (self.Name, M= .BaseName, DepexList)) - if 'BEFORE' in DepexList or 'AFTER' in DepexList: - break - if len(DepexList) > 0: - EdkLogger.verbose('') - return {self.ModuleType:DepexList} - - ## Merge dependency expression - # - # @retval list The token list of the dependency expression af= ter parsed - # - @cached_property - def DepexExpressionDict(self): - if self.DxsFile or self.IsLibrary or TAB_DEPENDENCY_EXPRESSION_FIL= E in self.FileTypes: - return {} - - DepexExpressionString =3D '' - # - # Append depex from dependent libraries, if not "BEFORE", "AFTER" = expresion - # - for M in [self.Module] + self.DependentLibraryList: - Inherited =3D False - for D in M.DepexExpression[self.Arch, self.ModuleType]: - if DepexExpressionString !=3D '': - DepexExpressionString +=3D ' AND ' - DepexExpressionString +=3D '(' - DepexExpressionString +=3D D - DepexExpressionString =3D DepexExpressionString.rstrip('EN= D').strip() - DepexExpressionString +=3D ')' - Inherited =3D True - if Inherited: - EdkLogger.verbose("DEPEX[%s] (+%s) =3D %s" % (self.Name, M= .BaseName, DepexExpressionString)) - if 'BEFORE' in DepexExpressionString or 'AFTER' in DepexExpres= sionString: - break - if len(DepexExpressionString) > 0: - EdkLogger.verbose('') - - return {self.ModuleType:DepexExpressionString} - - # Get the tiano core user extension, it is contain dependent library. - # @retval: a list contain tiano core userextension. - # - def _GetTianoCoreUserExtensionList(self): - TianoCoreUserExtentionList =3D [] - for M in [self.Module] + self.DependentLibraryList: - Filename =3D M.MetaFile.Path - InfObj =3D InfSectionParser.InfSectionParser(Filename) - TianoCoreUserExtenList =3D InfObj.GetUserExtensionTianoCore() - for TianoCoreUserExtent in TianoCoreUserExtenList: - for Section in TianoCoreUserExtent: - ItemList =3D Section.split(TAB_SPLIT) - Arch =3D self.Arch - if len(ItemList) =3D=3D 4: - Arch =3D ItemList[3] - if Arch.upper() =3D=3D TAB_ARCH_COMMON or Arch.upper()= =3D=3D self.Arch.upper(): - TianoCoreList =3D [] - TianoCoreList.extend([TAB_SECTION_START + Section = + TAB_SECTION_END]) - TianoCoreList.extend(TianoCoreUserExtent[Section][= :]) - TianoCoreList.append('\n') - TianoCoreUserExtentionList.append(TianoCoreList) - - return TianoCoreUserExtentionList - - ## Return the list of specification version required for the module - # - # @retval list The list of specification defined in module fi= le - # - @cached_property - def Specification(self): - return self.Module.Specification - - ## Tool option for the module build - # - # @param PlatformInfo The object of PlatformBuildInfo - # @retval dict The dict containing valid options - # - @cached_property - def BuildOption(self): - RetVal, self.BuildRuleOrder =3D self.PlatformInfo.ApplyBuildOption= (self.Module) - if self.BuildRuleOrder: - self.BuildRuleOrder =3D ['.%s' % Ext for Ext in self.BuildRule= Order.split()] - return RetVal - - ## Get include path list from tool option for the module build - # - # @retval list The include path list - # - @cached_property - def BuildOptionIncPathList(self): - # - # Regular expression for finding Include Directories, the differen= ce between MSFT and INTEL/GCC/RVCT - # is the former use /I , the Latter used -I to specify include dir= ectories - # - if self.PlatformInfo.ToolChainFamily in (TAB_COMPILER_MSFT): - BuildOptIncludeRegEx =3D gBuildOptIncludePatternMsft - elif self.PlatformInfo.ToolChainFamily in ('INTEL', 'GCC', 'RVCT'): - BuildOptIncludeRegEx =3D gBuildOptIncludePatternOther - else: - # - # New ToolChainFamily, don't known whether there is option to = specify include directories - # - return [] - - RetVal =3D [] - for Tool in ('CC', 'PP', 'VFRPP', 'ASLPP', 'ASLCC', 'APP', 'ASM'): - try: - FlagOption =3D self.BuildOption[Tool]['FLAGS'] - except KeyError: - FlagOption =3D '' - - if self.ToolChainFamily !=3D 'RVCT': - IncPathList =3D [NormPath(Path, self.Macros) for Path in B= uildOptIncludeRegEx.findall(FlagOption)] - else: - # - # RVCT may specify a list of directory separated by commas - # - IncPathList =3D [] - for Path in BuildOptIncludeRegEx.findall(FlagOption): - PathList =3D GetSplitList(Path, TAB_COMMA_SPLIT) - IncPathList.extend(NormPath(PathEntry, self.Macros) fo= r PathEntry in PathList) - - # - # EDK II modules must not reference header files outside of th= e packages they depend on or - # within the module's directory tree. Report error if violatio= n. - # - if GlobalData.gDisableIncludePathCheck =3D=3D False: - for Path in IncPathList: - if (Path not in self.IncludePathList) and (CommonPath(= [Path, self.MetaFile.Dir]) !=3D self.MetaFile.Dir): - ErrMsg =3D "The include directory for the EDK II m= odule in this line is invalid %s specified in %s FLAGS '%s'" % (Path, Tool,= FlagOption) - EdkLogger.error("build", - PARAMETER_INVALID, - ExtraData=3DErrMsg, - File=3Dstr(self.MetaFile)) - RetVal +=3D IncPathList - return RetVal - - ## Return a list of files which can be built from source - # - # What kind of files can be built is determined by build rules in - # $(CONF_DIRECTORY)/build_rule.txt and toolchain family. - # - @cached_property - def SourceFileList(self): - RetVal =3D [] - ToolChainTagSet =3D {"", TAB_STAR, self.ToolChain} - ToolChainFamilySet =3D {"", TAB_STAR, self.ToolChainFamily, self.B= uildRuleFamily} - for F in self.Module.Sources: - # match tool chain - if F.TagName not in ToolChainTagSet: - EdkLogger.debug(EdkLogger.DEBUG_9, "The toolchain [%s] for= processing file [%s] is found, " - "but [%s] is currently used" % (F.TagName,= str(F), self.ToolChain)) - continue - # match tool chain family or build rule family - if F.ToolChainFamily not in ToolChainFamilySet: - EdkLogger.debug( - EdkLogger.DEBUG_0, - "The file [%s] must be built by tools of [%s],= " \ - "but current toolchain family is [%s], buildru= le family is [%s]" \ - % (str(F), F.ToolChainFamily, self.ToolCha= inFamily, self.BuildRuleFamily)) - continue - - # add the file path into search path list for file including - if F.Dir not in self.IncludePathList: - self.IncludePathList.insert(0, F.Dir) - RetVal.append(F) - - self._MatchBuildRuleOrder(RetVal) - - for F in RetVal: - self._ApplyBuildRule(F, TAB_UNKNOWN_FILE) - return RetVal - - def _MatchBuildRuleOrder(self, FileList): - Order_Dict =3D {} - self.BuildOption - for SingleFile in FileList: - if self.BuildRuleOrder and SingleFile.Ext in self.BuildRuleOrd= er and SingleFile.Ext in self.BuildRules: - key =3D SingleFile.Path.rsplit(SingleFile.Ext,1)[0] - if key in Order_Dict: - Order_Dict[key].append(SingleFile.Ext) - else: - Order_Dict[key] =3D [SingleFile.Ext] - - RemoveList =3D [] - for F in Order_Dict: - if len(Order_Dict[F]) > 1: - Order_Dict[F].sort(key=3Dlambda i: self.BuildRuleOrder.ind= ex(i)) - for Ext in Order_Dict[F][1:]: - RemoveList.append(F + Ext) - - for item in RemoveList: - FileList.remove(item) - - return FileList - - ## Return the list of unicode files - @cached_property - def UnicodeFileList(self): - return self.FileTypes.get(TAB_UNICODE_FILE,[]) - - ## Return the list of vfr files - @cached_property - def VfrFileList(self): - return self.FileTypes.get(TAB_VFR_FILE, []) - - ## Return the list of Image Definition files - @cached_property - def IdfFileList(self): - return self.FileTypes.get(TAB_IMAGE_FILE,[]) - - ## Return a list of files which can be built from binary - # - # "Build" binary files are just to copy them to build directory. - # - # @retval list The list of files which can be built l= ater - # - @cached_property - def BinaryFileList(self): - RetVal =3D [] - for F in self.Module.Binaries: - if F.Target not in [TAB_ARCH_COMMON, TAB_STAR] and F.Target != =3D self.BuildTarget: - continue - RetVal.append(F) - self._ApplyBuildRule(F, F.Type, BinaryFileList=3DRetVal) - return RetVal - - @cached_property - def BuildRules(self): - RetVal =3D {} - BuildRuleDatabase =3D BuildRule - for Type in BuildRuleDatabase.FileTypeList: - #first try getting build rule by BuildRuleFamily - RuleObject =3D BuildRuleDatabase[Type, self.BuildType, self.Ar= ch, self.BuildRuleFamily] - if not RuleObject: - # build type is always module type, but ... - if self.ModuleType !=3D self.BuildType: - RuleObject =3D BuildRuleDatabase[Type, self.ModuleType= , self.Arch, self.BuildRuleFamily] - #second try getting build rule by ToolChainFamily - if not RuleObject: - RuleObject =3D BuildRuleDatabase[Type, self.BuildType, sel= f.Arch, self.ToolChainFamily] - if not RuleObject: - # build type is always module type, but ... - if self.ModuleType !=3D self.BuildType: - RuleObject =3D BuildRuleDatabase[Type, self.Module= Type, self.Arch, self.ToolChainFamily] - if not RuleObject: - continue - RuleObject =3D RuleObject.Instantiate(self.Macros) - RetVal[Type] =3D RuleObject - for Ext in RuleObject.SourceFileExtList: - RetVal[Ext] =3D RuleObject - return RetVal - - def _ApplyBuildRule(self, File, FileType, BinaryFileList=3DNone): - if self._BuildTargets is None: - self._IntroBuildTargetList =3D set() - self._FinalBuildTargetList =3D set() - self._BuildTargets =3D defaultdict(set) - self._FileTypes =3D defaultdict(set) - - if not BinaryFileList: - BinaryFileList =3D self.BinaryFileList - - SubDirectory =3D os.path.join(self.OutputDir, File.SubDir) - if not os.path.exists(SubDirectory): - CreateDirectory(SubDirectory) - LastTarget =3D None - RuleChain =3D set() - SourceList =3D [File] - Index =3D 0 - # - # Make sure to get build rule order value - # - self.BuildOption - - while Index < len(SourceList): - Source =3D SourceList[Index] - Index =3D Index + 1 - - if Source !=3D File: - CreateDirectory(Source.Dir) - - if File.IsBinary and File =3D=3D Source and File in BinaryFile= List: - # Skip all files that are not binary libraries - if not self.IsLibrary: - continue - RuleObject =3D self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject =3D self.BuildRules[FileType] - elif Source.Ext in self.BuildRules: - RuleObject =3D self.BuildRules[Source.Ext] - else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - - FileType =3D RuleObject.SourceFileType - self._FileTypes[FileType].add(Source) - - # stop at STATIC_LIBRARY for library - if self.IsLibrary and FileType =3D=3D TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - - Target =3D RuleObject.Apply(Source, self.BuildRuleOrder) - if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) - - self._BuildTargets[FileType].add(Target) - - if not Source.IsBinary and Source =3D=3D File: - self._IntroBuildTargetList.add(Target) - - # to avoid cyclic rule - if FileType in RuleChain: - break - - RuleChain.add(FileType) - SourceList.extend(Target.Outputs) - LastTarget =3D Target - FileType =3D TAB_UNKNOWN_FILE - - @cached_property - def Targets(self): - if self._BuildTargets is None: - self._IntroBuildTargetList =3D set() - self._FinalBuildTargetList =3D set() - self._BuildTargets =3D defaultdict(set) - self._FileTypes =3D defaultdict(set) - - #TRICK: call SourceFileList property to apply build rule for sourc= e files - self.SourceFileList - - #TRICK: call _GetBinaryFileList to apply build rule for binary fil= es - self.BinaryFileList - - return self._BuildTargets - - @cached_property - def IntroTargetList(self): - self.Targets - return self._IntroBuildTargetList - - @cached_property - def CodaTargetList(self): - self.Targets - return self._FinalBuildTargetList - - @cached_property - def FileTypes(self): - self.Targets - return self._FileTypes - - ## Get the list of package object the module depends on - # - # @retval list The package object list - # - @cached_property - def DependentPackageList(self): - return self.Module.Packages - - ## Return the list of auto-generated code file - # - # @retval list The list of auto-generated file - # - @cached_property - def AutoGenFileList(self): - AutoGenUniIdf =3D self.BuildType !=3D 'UEFI_HII' - UniStringBinBuffer =3D BytesIO() - IdfGenBinBuffer =3D BytesIO() - RetVal =3D {} - AutoGenC =3D TemplateString() - AutoGenH =3D TemplateString() - StringH =3D TemplateString() - StringIdf =3D TemplateString() - GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, = UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer) - # - # AutoGen.c is generated if there are library classes in inf, or t= here are object files - # - if str(AutoGenC) !=3D "" and (len(self.Module.LibraryClasses) > 0 - or TAB_OBJECT_FILE in self.FileTypes): - AutoFile =3D PathClass(gAutoGenCodeFileName, self.DebugDir) - RetVal[AutoFile] =3D str(AutoGenC) - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if str(AutoGenH) !=3D "": - AutoFile =3D PathClass(gAutoGenHeaderFileName, self.DebugDir) - RetVal[AutoFile] =3D str(AutoGenH) - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if str(StringH) !=3D "": - AutoFile =3D PathClass(gAutoGenStringFileName % {"module_name"= :self.Name}, self.DebugDir) - RetVal[AutoFile] =3D str(StringH) - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if UniStringBinBuffer is not None and UniStringBinBuffer.getvalue(= ) !=3D b"": - AutoFile =3D PathClass(gAutoGenStringFormFileName % {"module_n= ame":self.Name}, self.OutputDir) - RetVal[AutoFile] =3D UniStringBinBuffer.getvalue() - AutoFile.IsBinary =3D True - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if UniStringBinBuffer is not None: - UniStringBinBuffer.close() - if str(StringIdf) !=3D "": - AutoFile =3D PathClass(gAutoGenImageDefFileName % {"module_nam= e":self.Name}, self.DebugDir) - RetVal[AutoFile] =3D str(StringIdf) - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if IdfGenBinBuffer is not None and IdfGenBinBuffer.getvalue() !=3D= b"": - AutoFile =3D PathClass(gAutoGenIdfFileName % {"module_name":se= lf.Name}, self.OutputDir) - RetVal[AutoFile] =3D IdfGenBinBuffer.getvalue() - AutoFile.IsBinary =3D True - self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) - if IdfGenBinBuffer is not None: - IdfGenBinBuffer.close() - return RetVal - - ## Return the list of library modules explicitly or implicitly used by= this module - @cached_property - def DependentLibraryList(self): - # only merge library classes and PCD for non-library module - if self.IsLibrary: - return [] - return self.PlatformInfo.ApplyLibraryInstance(self.Module) - - ## Get the list of PCDs from current module - # - # @retval list The list of PCD - # - @cached_property - def ModulePcdList(self): - # apply PCD settings from platform - RetVal =3D self.PlatformInfo.ApplyPcdSetting(self.Module, self.Mod= ule.Pcds) - ExtendCopyDictionaryLists(self._PcdComments, self.Module.PcdCommen= ts) - return RetVal - - ## Get the list of PCDs from dependent libraries - # - # @retval list The list of PCD - # - @cached_property - def LibraryPcdList(self): - if self.IsLibrary: - return [] - RetVal =3D [] - Pcds =3D set() - # get PCDs from dependent libraries - for Library in self.DependentLibraryList: - PcdsInLibrary =3D OrderedDict() - ExtendCopyDictionaryLists(self._PcdComments, Library.PcdCommen= ts) - for Key in Library.Pcds: - # skip duplicated PCDs - if Key in self.Module.Pcds or Key in Pcds: - continue - Pcds.add(Key) - PcdsInLibrary[Key] =3D copy.copy(Library.Pcds[Key]) - RetVal.extend(self.PlatformInfo.ApplyPcdSetting(self.Module, P= cdsInLibrary, Library=3DLibrary)) - return RetVal - - ## Get the GUID value mapping - # - # @retval dict The mapping between GUID cname and its value - # - @cached_property - def GuidList(self): - RetVal =3D OrderedDict(self.Module.Guids) - for Library in self.DependentLibraryList: - RetVal.update(Library.Guids) - ExtendCopyDictionaryLists(self._GuidComments, Library.GuidComm= ents) - ExtendCopyDictionaryLists(self._GuidComments, self.Module.GuidComm= ents) - return RetVal - - @cached_property - def GetGuidsUsedByPcd(self): - RetVal =3D OrderedDict(self.Module.GetGuidsUsedByPcd()) - for Library in self.DependentLibraryList: - RetVal.update(Library.GetGuidsUsedByPcd()) - return RetVal - ## Get the protocol value mapping - # - # @retval dict The mapping between protocol cname and its val= ue - # - @cached_property - def ProtocolList(self): - RetVal =3D OrderedDict(self.Module.Protocols) - for Library in self.DependentLibraryList: - RetVal.update(Library.Protocols) - ExtendCopyDictionaryLists(self._ProtocolComments, Library.Prot= ocolComments) - ExtendCopyDictionaryLists(self._ProtocolComments, self.Module.Prot= ocolComments) - return RetVal - - ## Get the PPI value mapping - # - # @retval dict The mapping between PPI cname and its value - # - @cached_property - def PpiList(self): - RetVal =3D OrderedDict(self.Module.Ppis) - for Library in self.DependentLibraryList: - RetVal.update(Library.Ppis) - ExtendCopyDictionaryLists(self._PpiComments, Library.PpiCommen= ts) - ExtendCopyDictionaryLists(self._PpiComments, self.Module.PpiCommen= ts) - return RetVal - - ## Get the list of include search path - # - # @retval list The list path - # - @cached_property - def IncludePathList(self): - RetVal =3D [] - RetVal.append(self.MetaFile.Dir) - RetVal.append(self.DebugDir) - - for Package in self.Module.Packages: - PackageDir =3D mws.join(self.WorkspaceDir, Package.MetaFile.Di= r) - if PackageDir not in RetVal: - RetVal.append(PackageDir) - IncludesList =3D Package.Includes - if Package._PrivateIncludes: - if not self.MetaFile.OriginalPath.Path.startswith(PackageD= ir): - IncludesList =3D list(set(Package.Includes).difference= (set(Package._PrivateIncludes))) - for Inc in IncludesList: - if Inc not in RetVal: - RetVal.append(str(Inc)) - return RetVal - - @cached_property - def IncludePathLength(self): - return sum(len(inc)+1 for inc in self.IncludePathList) - - ## Get HII EX PCDs which maybe used by VFR - # - # efivarstore used by VFR may relate with HII EX PCDs - # Get the variable name and GUID from efivarstore and HII EX PCD - # List the HII EX PCDs in As Built INF if both name and GUID match. - # - # @retval list HII EX PCDs - # - def _GetPcdsMaybeUsedByVfr(self): - if not self.SourceFileList: - return [] - - NameGuids =3D set() - for SrcFile in self.SourceFileList: - if SrcFile.Ext.lower() !=3D '.vfr': - continue - Vfri =3D os.path.join(self.OutputDir, SrcFile.BaseName + '.i') - if not os.path.exists(Vfri): - continue - VfriFile =3D open(Vfri, 'r') - Content =3D VfriFile.read() - VfriFile.close() - Pos =3D Content.find('efivarstore') - while Pos !=3D -1: - # - # Make sure 'efivarstore' is the start of efivarstore stat= ement - # In case of the value of 'name' (name =3D efivarstore) is= equal to 'efivarstore' - # - Index =3D Pos - 1 - while Index >=3D 0 and Content[Index] in ' \t\r\n': - Index -=3D 1 - if Index >=3D 0 and Content[Index] !=3D ';': - Pos =3D Content.find('efivarstore', Pos + len('efivars= tore')) - continue - # - # 'efivarstore' must be followed by name and guid - # - Name =3D gEfiVarStoreNamePattern.search(Content, Pos) - if not Name: - break - Guid =3D gEfiVarStoreGuidPattern.search(Content, Pos) - if not Guid: - break - NameArray =3D _ConvertStringToByteArray('L"' + Name.group(= 1) + '"') - NameGuids.add((NameArray, GuidStructureStringToGuidString(= Guid.group(1)))) - Pos =3D Content.find('efivarstore', Name.end()) - if not NameGuids: - return [] - HiiExPcds =3D [] - for Pcd in self.PlatformInfo.Platform.Pcds.values(): - if Pcd.Type !=3D TAB_PCDS_DYNAMIC_EX_HII: - continue - for SkuInfo in Pcd.SkuInfoList.values(): - Value =3D GuidValue(SkuInfo.VariableGuid, self.PlatformInf= o.PackageList, self.MetaFile.Path) - if not Value: - continue - Name =3D _ConvertStringToByteArray(SkuInfo.VariableName) - Guid =3D GuidStructureStringToGuidString(Value) - if (Name, Guid) in NameGuids and Pcd not in HiiExPcds: - HiiExPcds.append(Pcd) - break - - return HiiExPcds - - def _GenOffsetBin(self): - VfrUniBaseName =3D {} - for SourceFile in self.Module.Sources: - if SourceFile.Type.upper() =3D=3D ".VFR" : - # - # search the .map file to find the offset of vfr binary in= the PE32+/TE file. - # - VfrUniBaseName[SourceFile.BaseName] =3D (SourceFile.BaseNa= me + "Bin") - elif SourceFile.Type.upper() =3D=3D ".UNI" : - # - # search the .map file to find the offset of Uni strings b= inary in the PE32+/TE file. - # - VfrUniBaseName["UniOffsetName"] =3D (self.Name + "Strings") - - if not VfrUniBaseName: - return None - MapFileName =3D os.path.join(self.OutputDir, self.Name + ".map") - EfiFileName =3D os.path.join(self.OutputDir, self.Name + ".efi") - VfrUniOffsetList =3D GetVariableOffset(MapFileName, EfiFileName, l= ist(VfrUniBaseName.values())) - if not VfrUniOffsetList: - return None - - OutputName =3D '%sOffset.bin' % self.Name - UniVfrOffsetFileName =3D os.path.join( self.OutputDir, OutputN= ame) - - try: - fInputfile =3D open(UniVfrOffsetFileName, "wb+", 0) - except: - EdkLogger.error("build", FILE_OPEN_FAILURE, "File open failed = for %s" % UniVfrOffsetFileName, None) - - # Use a instance of BytesIO to cache data - fStringIO =3D BytesIO() - - for Item in VfrUniOffsetList: - if (Item[0].find("Strings") !=3D -1): - # - # UNI offset in image. - # GUID + Offset - # { 0x8913c5e0, 0x33f6, 0x4d86, { 0x9b, 0xf1, 0x43, 0xef, = 0x89, 0xfc, 0x6, 0x66 } } - # - UniGuid =3D b'\xe0\xc5\x13\x89\xf63\x86M\x9b\xf1C\xef\x89\= xfc\x06f' - fStringIO.write(UniGuid) - UniValue =3D pack ('Q', int (Item[1], 16)) - fStringIO.write (UniValue) - else: - # - # VFR binary offset in image. - # GUID + Offset - # { 0xd0bc7cb4, 0x6a47, 0x495f, { 0xaa, 0x11, 0x71, 0x7, 0= x46, 0xda, 0x6, 0xa2 } }; - # - VfrGuid =3D b'\xb4|\xbc\xd0Gj_I\xaa\x11q\x07F\xda\x06\xa2' - fStringIO.write(VfrGuid) - VfrValue =3D pack ('Q', int (Item[1], 16)) - fStringIO.write (VfrValue) - # - # write data into file. - # - try : - fInputfile.write (fStringIO.getvalue()) - except: - EdkLogger.error("build", FILE_WRITE_FAILURE, "Write data to fi= le %s failed, please check whether the " - "file been locked or using by other applicatio= ns." %UniVfrOffsetFileName, None) - - fStringIO.close () - fInputfile.close () - return OutputName - - @cached_property - def OutputFile(self): - retVal =3D set() - OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') - DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') - for Item in self.CodaTargetList: - File =3D Item.Target.Path.replace('\\', '/').strip('/').replac= e(DebugDir, '').replace(OutputDir, '').strip('/') - retVal.add(File) - if self.DepexGenerated: - retVal.add(self.Name + '.depex') - - Bin =3D self._GenOffsetBin() - if Bin: - retVal.add(Bin) - - for Root, Dirs, Files in os.walk(OutputDir): - for File in Files: - if File.lower().endswith('.pdb'): - retVal.add(File) - - return retVal - - ## Create AsBuilt INF file the module - # - def CreateAsBuiltInf(self): - - if self.IsAsBuiltInfCreated: - return - - # Skip INF file generation for libraries - if self.IsLibrary: - return - - # Skip the following code for modules with no source files - if not self.SourceFileList: - return - - # Skip the following code for modules without any binary files - if self.BinaryFileList: - return - - ### TODO: How to handles mixed source and binary modules - - # Find all DynamicEx and PatchableInModule PCDs used by this modul= e and dependent libraries - # Also find all packages that the DynamicEx PCDs depend on - Pcds =3D [] - PatchablePcds =3D [] - Packages =3D [] - PcdCheckList =3D [] - PcdTokenSpaceList =3D [] - for Pcd in self.ModulePcdList + self.LibraryPcdList: - if Pcd.Type =3D=3D TAB_PCDS_PATCHABLE_IN_MODULE: - PatchablePcds.append(Pcd) - PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGuidCNa= me, TAB_PCDS_PATCHABLE_IN_MODULE)) - elif Pcd.Type in PCD_DYNAMIC_EX_TYPE_SET: - if Pcd not in Pcds: - Pcds.append(Pcd) - PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGui= dCName, TAB_PCDS_DYNAMIC_EX)) - PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGui= dCName, TAB_PCDS_DYNAMIC)) - PcdTokenSpaceList.append(Pcd.TokenSpaceGuidCName) - GuidList =3D OrderedDict(self.GuidList) - for TokenSpace in self.GetGuidsUsedByPcd: - # If token space is not referred by patch PCD or Ex PCD, remov= e the GUID from GUID list - # The GUIDs in GUIDs section should really be the GUIDs in sou= rce INF or referred by Ex an patch PCDs - if TokenSpace not in PcdTokenSpaceList and TokenSpace in GuidL= ist: - GuidList.pop(TokenSpace) - CheckList =3D (GuidList, self.PpiList, self.ProtocolList, PcdCheck= List) - for Package in self.DerivedPackageList: - if Package in Packages: - continue - BeChecked =3D (Package.Guids, Package.Ppis, Package.Protocols,= Package.Pcds) - Found =3D False - for Index in range(len(BeChecked)): - for Item in CheckList[Index]: - if Item in BeChecked[Index]: - Packages.append(Package) - Found =3D True - break - if Found: - break - - VfrPcds =3D self._GetPcdsMaybeUsedByVfr() - for Pkg in self.PlatformInfo.PackageList: - if Pkg in Packages: - continue - for VfrPcd in VfrPcds: - if ((VfrPcd.TokenCName, VfrPcd.TokenSpaceGuidCName, TAB_PC= DS_DYNAMIC_EX) in Pkg.Pcds or - (VfrPcd.TokenCName, VfrPcd.TokenSpaceGuidCName, TAB_PC= DS_DYNAMIC) in Pkg.Pcds): - Packages.append(Pkg) - break - - ModuleType =3D SUP_MODULE_DXE_DRIVER if self.ModuleType =3D=3D SUP= _MODULE_UEFI_DRIVER and self.DepexGenerated else self.ModuleType - DriverType =3D self.PcdIsDriver if self.PcdIsDriver else '' - Guid =3D self.Guid - MDefs =3D self.Module.Defines - - AsBuiltInfDict =3D { - 'module_name' : self.Name, - 'module_guid' : Guid, - 'module_module_type' : ModuleType, - 'module_version_string' : [MDefs['VERSION_STRING']] = if 'VERSION_STRING' in MDefs else [], - 'pcd_is_driver_string' : [], - 'module_uefi_specification_version' : [], - 'module_pi_specification_version' : [], - 'module_entry_point' : self.Module.ModuleEntryPoi= ntList, - 'module_unload_image' : self.Module.ModuleUnloadIm= ageList, - 'module_constructor' : self.Module.ConstructorLis= t, - 'module_destructor' : self.Module.DestructorList, - 'module_shadow' : [MDefs['SHADOW']] if 'SHAD= OW' in MDefs else [], - 'module_pci_vendor_id' : [MDefs['PCI_VENDOR_ID']] i= f 'PCI_VENDOR_ID' in MDefs else [], - 'module_pci_device_id' : [MDefs['PCI_DEVICE_ID']] i= f 'PCI_DEVICE_ID' in MDefs else [], - 'module_pci_class_code' : [MDefs['PCI_CLASS_CODE']] = if 'PCI_CLASS_CODE' in MDefs else [], - 'module_pci_revision' : [MDefs['PCI_REVISION']] if= 'PCI_REVISION' in MDefs else [], - 'module_build_number' : [MDefs['BUILD_NUMBER']] if= 'BUILD_NUMBER' in MDefs else [], - 'module_spec' : [MDefs['SPEC']] if 'SPEC' = in MDefs else [], - 'module_uefi_hii_resource_section' : [MDefs['UEFI_HII_RESOURCE_= SECTION']] if 'UEFI_HII_RESOURCE_SECTION' in MDefs else [], - 'module_uni_file' : [MDefs['MODULE_UNI_FILE']]= if 'MODULE_UNI_FILE' in MDefs else [], - 'module_arch' : self.Arch, - 'package_item' : [Package.MetaFile.File.rep= lace('\\', '/') for Package in Packages], - 'binary_item' : [], - 'patchablepcd_item' : [], - 'pcd_item' : [], - 'protocol_item' : [], - 'ppi_item' : [], - 'guid_item' : [], - 'flags_item' : [], - 'libraryclasses_item' : [] - } - - if 'MODULE_UNI_FILE' in MDefs: - UNIFile =3D os.path.join(self.MetaFile.Dir, MDefs['MODULE_UNI_= FILE']) - if os.path.isfile(UNIFile): - shutil.copy2(UNIFile, self.OutputDir) - - if self.AutoGenVersion > int(gInfSpecVersion, 0): - AsBuiltInfDict['module_inf_version'] =3D '0x%08x' % self.AutoG= enVersion - else: - AsBuiltInfDict['module_inf_version'] =3D gInfSpecVersion - - if DriverType: - AsBuiltInfDict['pcd_is_driver_string'].append(DriverType) - - if 'UEFI_SPECIFICATION_VERSION' in self.Specification: - AsBuiltInfDict['module_uefi_specification_version'].append(sel= f.Specification['UEFI_SPECIFICATION_VERSION']) - if 'PI_SPECIFICATION_VERSION' in self.Specification: - AsBuiltInfDict['module_pi_specification_version'].append(self.= Specification['PI_SPECIFICATION_VERSION']) - - OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') - DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') - for Item in self.CodaTargetList: - File =3D Item.Target.Path.replace('\\', '/').strip('/').replac= e(DebugDir, '').replace(OutputDir, '').strip('/') - if os.path.isabs(File): - File =3D File.replace('\\', '/').strip('/').replace(Output= Dir, '').strip('/') - if Item.Target.Ext.lower() =3D=3D '.aml': - AsBuiltInfDict['binary_item'].append('ASL|' + File) - elif Item.Target.Ext.lower() =3D=3D '.acpi': - AsBuiltInfDict['binary_item'].append('ACPI|' + File) - elif Item.Target.Ext.lower() =3D=3D '.efi': - AsBuiltInfDict['binary_item'].append('PE32|' + self.Name += '.efi') - else: - AsBuiltInfDict['binary_item'].append('BIN|' + File) - if not self.DepexGenerated: - DepexFile =3D os.path.join(self.OutputDir, self.Name + '.depex= ') - if os.path.exists(DepexFile): - self.DepexGenerated =3D True - if self.DepexGenerated: - if self.ModuleType in [SUP_MODULE_PEIM]: - AsBuiltInfDict['binary_item'].append('PEI_DEPEX|' + self.N= ame + '.depex') - elif self.ModuleType in [SUP_MODULE_DXE_DRIVER, SUP_MODULE_DXE= _RUNTIME_DRIVER, SUP_MODULE_DXE_SAL_DRIVER, SUP_MODULE_UEFI_DRIVER]: - AsBuiltInfDict['binary_item'].append('DXE_DEPEX|' + self.N= ame + '.depex') - elif self.ModuleType in [SUP_MODULE_DXE_SMM_DRIVER]: - AsBuiltInfDict['binary_item'].append('SMM_DEPEX|' + self.N= ame + '.depex') - - Bin =3D self._GenOffsetBin() - if Bin: - AsBuiltInfDict['binary_item'].append('BIN|%s' % Bin) - - for Root, Dirs, Files in os.walk(OutputDir): - for File in Files: - if File.lower().endswith('.pdb'): - AsBuiltInfDict['binary_item'].append('DISPOSABLE|' + F= ile) - HeaderComments =3D self.Module.HeaderComments - StartPos =3D 0 - for Index in range(len(HeaderComments)): - if HeaderComments[Index].find('@BinaryHeader') !=3D -1: - HeaderComments[Index] =3D HeaderComments[Index].replace('@= BinaryHeader', '@file') - StartPos =3D Index - break - AsBuiltInfDict['header_comments'] =3D '\n'.join(HeaderComments[Sta= rtPos:]).replace(':#', '://') - AsBuiltInfDict['tail_comments'] =3D '\n'.join(self.Module.TailComm= ents) - - GenList =3D [ - (self.ProtocolList, self._ProtocolComments, 'protocol_item'), - (self.PpiList, self._PpiComments, 'ppi_item'), - (GuidList, self._GuidComments, 'guid_item') - ] - for Item in GenList: - for CName in Item[0]: - Comments =3D '\n '.join(Item[1][CName]) if CName in Item[= 1] else '' - Entry =3D Comments + '\n ' + CName if Comments else CName - AsBuiltInfDict[Item[2]].append(Entry) - PatchList =3D parsePcdInfoFromMapFile( - os.path.join(self.OutputDir, self.Name + '.map= '), - os.path.join(self.OutputDir, self.Name + '.efi= ') - ) - if PatchList: - for Pcd in PatchablePcds: - TokenCName =3D Pcd.TokenCName - for PcdItem in GlobalData.MixedPcd: - if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) in Global= Data.MixedPcd[PcdItem]: - TokenCName =3D PcdItem[0] - break - for PatchPcd in PatchList: - if TokenCName =3D=3D PatchPcd[0]: - break - else: - continue - PcdValue =3D '' - if Pcd.DatumType =3D=3D 'BOOLEAN': - BoolValue =3D Pcd.DefaultValue.upper() - if BoolValue =3D=3D 'TRUE': - Pcd.DefaultValue =3D '1' - elif BoolValue =3D=3D 'FALSE': - Pcd.DefaultValue =3D '0' - - if Pcd.DatumType in TAB_PCD_NUMERIC_TYPES: - HexFormat =3D '0x%02x' - if Pcd.DatumType =3D=3D TAB_UINT16: - HexFormat =3D '0x%04x' - elif Pcd.DatumType =3D=3D TAB_UINT32: - HexFormat =3D '0x%08x' - elif Pcd.DatumType =3D=3D TAB_UINT64: - HexFormat =3D '0x%016x' - PcdValue =3D HexFormat % int(Pcd.DefaultValue, 0) - else: - if Pcd.MaxDatumSize is None or Pcd.MaxDatumSize =3D=3D= '': - EdkLogger.error("build", AUTOGEN_ERROR, - "Unknown [MaxDatumSize] of PCD [%s= .%s]" % (Pcd.TokenSpaceGuidCName, TokenCName) - ) - ArraySize =3D int(Pcd.MaxDatumSize, 0) - PcdValue =3D Pcd.DefaultValue - if PcdValue[0] !=3D '{': - Unicode =3D False - if PcdValue[0] =3D=3D 'L': - Unicode =3D True - PcdValue =3D PcdValue.lstrip('L') - PcdValue =3D eval(PcdValue) - NewValue =3D '{' - for Index in range(0, len(PcdValue)): - if Unicode: - CharVal =3D ord(PcdValue[Index]) - NewValue =3D NewValue + '0x%02x' % (CharVa= l & 0x00FF) + ', ' \ - + '0x%02x' % (CharVal >> 8) + ', ' - else: - NewValue =3D NewValue + '0x%02x' % (ord(Pc= dValue[Index]) % 0x100) + ', ' - Padding =3D '0x00, ' - if Unicode: - Padding =3D Padding * 2 - ArraySize =3D ArraySize // 2 - if ArraySize < (len(PcdValue) + 1): - if Pcd.MaxSizeUserSet: - EdkLogger.error("build", AUTOGEN_ERROR, - "The maximum size of VOID* typ= e PCD '%s.%s' is less than its actual size occupied." % (Pcd.TokenSpaceGuid= CName, TokenCName) - ) - else: - ArraySize =3D len(PcdValue) + 1 - if ArraySize > len(PcdValue) + 1: - NewValue =3D NewValue + Padding * (ArraySize -= len(PcdValue) - 1) - PcdValue =3D NewValue + Padding.strip().rstrip(','= ) + '}' - elif len(PcdValue.split(',')) <=3D ArraySize: - PcdValue =3D PcdValue.rstrip('}') + ', 0x00' * (Ar= raySize - len(PcdValue.split(','))) - PcdValue +=3D '}' - else: - if Pcd.MaxSizeUserSet: - EdkLogger.error("build", AUTOGEN_ERROR, - "The maximum size of VOID* type PC= D '%s.%s' is less than its actual size occupied." % (Pcd.TokenSpaceGuidCNam= e, TokenCName) - ) - else: - ArraySize =3D len(PcdValue) + 1 - PcdItem =3D '%s.%s|%s|0x%X' % \ - (Pcd.TokenSpaceGuidCName, TokenCName, PcdValue, PatchP= cd[1]) - PcdComments =3D '' - if (Pcd.TokenSpaceGuidCName, Pcd.TokenCName) in self._PcdC= omments: - PcdComments =3D '\n '.join(self._PcdComments[Pcd.Toke= nSpaceGuidCName, Pcd.TokenCName]) - if PcdComments: - PcdItem =3D PcdComments + '\n ' + PcdItem - AsBuiltInfDict['patchablepcd_item'].append(PcdItem) - - for Pcd in Pcds + VfrPcds: - PcdCommentList =3D [] - HiiInfo =3D '' - TokenCName =3D Pcd.TokenCName - for PcdItem in GlobalData.MixedPcd: - if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) in GlobalData= .MixedPcd[PcdItem]: - TokenCName =3D PcdItem[0] - break - if Pcd.Type =3D=3D TAB_PCDS_DYNAMIC_EX_HII: - for SkuName in Pcd.SkuInfoList: - SkuInfo =3D Pcd.SkuInfoList[SkuName] - HiiInfo =3D '## %s|%s|%s' % (SkuInfo.VariableName, Sku= Info.VariableGuid, SkuInfo.VariableOffset) - break - if (Pcd.TokenSpaceGuidCName, Pcd.TokenCName) in self._PcdComme= nts: - PcdCommentList =3D self._PcdComments[Pcd.TokenSpaceGuidCNa= me, Pcd.TokenCName][:] - if HiiInfo: - UsageIndex =3D -1 - UsageStr =3D '' - for Index, Comment in enumerate(PcdCommentList): - for Usage in UsageList: - if Comment.find(Usage) !=3D -1: - UsageStr =3D Usage - UsageIndex =3D Index - break - if UsageIndex !=3D -1: - PcdCommentList[UsageIndex] =3D '## %s %s %s' % (UsageS= tr, HiiInfo, PcdCommentList[UsageIndex].replace(UsageStr, '')) - else: - PcdCommentList.append('## UNDEFINED ' + HiiInfo) - PcdComments =3D '\n '.join(PcdCommentList) - PcdEntry =3D Pcd.TokenSpaceGuidCName + '.' + TokenCName - if PcdComments: - PcdEntry =3D PcdComments + '\n ' + PcdEntry - AsBuiltInfDict['pcd_item'].append(PcdEntry) - for Item in self.BuildOption: - if 'FLAGS' in self.BuildOption[Item]: - AsBuiltInfDict['flags_item'].append('%s:%s_%s_%s_%s_FLAGS = =3D %s' % (self.ToolChainFamily, self.BuildTarget, self.ToolChain, self.Arc= h, Item, self.BuildOption[Item]['FLAGS'].strip())) - - # Generated LibraryClasses section in comments. - for Library in self.LibraryAutoGenList: - AsBuiltInfDict['libraryclasses_item'].append(Library.MetaFile.= File.replace('\\', '/')) - - # Generated UserExtensions TianoCore section. - # All tianocore user extensions are copied. - UserExtStr =3D '' - for TianoCore in self._GetTianoCoreUserExtensionList(): - UserExtStr +=3D '\n'.join(TianoCore) - ExtensionFile =3D os.path.join(self.MetaFile.Dir, TianoCore[1]) - if os.path.isfile(ExtensionFile): - shutil.copy2(ExtensionFile, self.OutputDir) - AsBuiltInfDict['userextension_tianocore_item'] =3D UserExtStr - - # Generated depex expression section in comments. - DepexExpression =3D self._GetDepexExpresionString() - AsBuiltInfDict['depexsection_item'] =3D DepexExpression if DepexEx= pression else '' - - AsBuiltInf =3D TemplateString() - AsBuiltInf.Append(gAsBuiltInfHeaderString.Replace(AsBuiltInfDict)) - - SaveFileOnChange(os.path.join(self.OutputDir, self.Name + '.inf'),= str(AsBuiltInf), False) - - self.IsAsBuiltInfCreated =3D True - - def CopyModuleToCache(self): - FileDir =3D path.join(GlobalData.gBinCacheDest, self.PlatformInfo.= OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceD= ir, self.MetaFile.BaseName) - CreateDirectory (FileDir) - HashFile =3D path.join(self.BuildDir, self.Name + '.hash') - if os.path.exists(HashFile): - CopyFileOnChange(HashFile, FileDir) - ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf') - if os.path.exists(ModuleFile): - CopyFileOnChange(ModuleFile, FileDir) - - if not self.OutputFile: - Ma =3D self.BuildDatabase[self.MetaFile, self.Arch, self.Build= Target, self.ToolChain] - self.OutputFile =3D Ma.Binaries - - for File in self.OutputFile: - File =3D str(File) - if not os.path.isabs(File): - File =3D os.path.join(self.OutputDir, File) - if os.path.exists(File): - sub_dir =3D os.path.relpath(File, self.OutputDir) - destination_file =3D os.path.join(FileDir, sub_dir) - destination_dir =3D os.path.dirname(destination_file) - CreateDirectory(destination_dir) - CopyFileOnChange(File, destination_dir) - - def AttemptModuleCacheCopy(self): - # If library or Module is binary do not skip by hash - if self.IsBinaryModule: - return False - # .inc is contains binary information so do not skip by hash as we= ll - for f_ext in self.SourceFileList: - if '.inc' in str(f_ext): - return False - FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.OutputDir, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.Sourc= eDir, self.MetaFile.BaseName) - HashFile =3D path.join(FileDir, self.Name + '.hash') - if os.path.exists(HashFile): - f =3D open(HashFile, 'r') - CacheHash =3D f.read() - f.close() - self.GenModuleHash() - if GlobalData.gModuleHash[self.Arch][self.Name]: - if CacheHash =3D=3D GlobalData.gModuleHash[self.Arch][self= .Name]: - for root, dir, files in os.walk(FileDir): - for f in files: - if self.Name + '.hash' in f: - CopyFileOnChange(HashFile, self.BuildDir) - else: - File =3D path.join(root, f) - sub_dir =3D os.path.relpath(File, FileDir) - destination_file =3D os.path.join(self.Out= putDir, sub_dir) - destination_dir =3D os.path.dirname(destin= ation_file) - CreateDirectory(destination_dir) - CopyFileOnChange(File, destination_dir) - if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "Pcd= Dxe": - CreatePcdDatabaseCode(self, TemplateString(), Temp= lateString()) - return True - return False - - ## Create makefile for the module and its dependent libraries - # - # @param CreateLibraryMakeFile Flag indicating if or not the = makefiles of - # dependent libraries will be cr= eated - # - @cached_class_function - def CreateMakeFile(self, CreateLibraryMakeFile=3DTrue, GenFfsList =3D = []): - # nest this function inside its only caller. - def CreateTimeStamp(): - FileSet =3D {self.MetaFile.Path} - - for SourceFile in self.Module.Sources: - FileSet.add (SourceFile.Path) - - for Lib in self.DependentLibraryList: - FileSet.add (Lib.MetaFile.Path) - - for f in self.AutoGenDepSet: - FileSet.add (f.Path) - - if os.path.exists (self.TimeStampPath): - os.remove (self.TimeStampPath) - with open(self.TimeStampPath, 'w+') as file: - for f in FileSet: - print(f, file=3Dfile) - - # Ignore generating makefile when it is a binary module - if self.IsBinaryModule: - return - - self.GenFfsList =3D GenFfsList - if not self.IsLibrary and CreateLibraryMakeFile: - for LibraryAutoGen in self.LibraryAutoGenList: - LibraryAutoGen.CreateMakeFile() - - # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping - if not GlobalData.gUseHashCache and self.CanSkip(): - return - - if len(self.CustomMakefile) =3D=3D 0: - Makefile =3D GenMake.ModuleMakefile(self) - else: - Makefile =3D GenMake.CustomMakefile(self) - if Makefile.Generate(): - EdkLogger.debug(EdkLogger.DEBUG_9, "Generated makefile for mod= ule %s [%s]" % - (self.Name, self.Arch)) - else: - EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = makefile for module %s [%s]" % - (self.Name, self.Arch)) - - CreateTimeStamp() - - def CopyBinaryFiles(self): - for File in self.Module.Binaries: - SrcPath =3D File.Path - DstPath =3D os.path.join(self.OutputDir, os.path.basename(SrcP= ath)) - CopyLongFilePath(SrcPath, DstPath) - ## Create autogen code for the module and its dependent libraries - # - # @param CreateLibraryCodeFile Flag indicating if or not the = code of - # dependent libraries will be cr= eated - # - def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): - if self.IsCodeFileCreated: - return - - # Need to generate PcdDatabase even PcdDriver is binarymodule - if self.IsBinaryModule and self.PcdIsDriver !=3D '': - CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) - return - if self.IsBinaryModule: - if self.IsLibrary: - self.CopyBinaryFiles() - return - - if not self.IsLibrary and CreateLibraryCodeFile: - for LibraryAutoGen in self.LibraryAutoGenList: - LibraryAutoGen.CreateCodeFile() - - # Don't enable if hash feature enabled, CanSkip uses timestamps to= determine build skipping - if not GlobalData.gUseHashCache and self.CanSkip(): - return - - AutoGenList =3D [] - IgoredAutoGenList =3D [] - - for File in self.AutoGenFileList: - if GenC.Generate(File.Path, self.AutoGenFileList[File], File.I= sBinary): - AutoGenList.append(str(File)) - else: - IgoredAutoGenList.append(str(File)) - - - for ModuleType in self.DepexList: - # Ignore empty [depex] section or [depex] section for SUP_MODU= LE_USER_DEFINED module - if len(self.DepexList[ModuleType]) =3D=3D 0 or ModuleType =3D= =3D SUP_MODULE_USER_DEFINED or ModuleType =3D=3D SUP_MODULE_HOST_APPLICATIO= N: - continue - - Dpx =3D GenDepex.DependencyExpression(self.DepexList[ModuleTyp= e], ModuleType, True) - DpxFile =3D gAutoGenDepexFileName % {"module_name" : self.Name} - - if len(Dpx.PostfixNotation) !=3D 0: - self.DepexGenerated =3D True - - if Dpx.Generate(path.join(self.OutputDir, DpxFile)): - AutoGenList.append(str(DpxFile)) - else: - IgoredAutoGenList.append(str(DpxFile)) - - if IgoredAutoGenList =3D=3D []: - EdkLogger.debug(EdkLogger.DEBUG_9, "Generated [%s] files for m= odule %s [%s]" % - (" ".join(AutoGenList), self.Name, self.Arch)) - elif AutoGenList =3D=3D []: - EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = [%s] files for module %s [%s]" % - (" ".join(IgoredAutoGenList), self.Name, self.= Arch)) - else: - EdkLogger.debug(EdkLogger.DEBUG_9, "Generated [%s] (skipped %s= ) files for module %s [%s]" % - (" ".join(AutoGenList), " ".join(IgoredAutoGen= List), self.Name, self.Arch)) - - self.IsCodeFileCreated =3D True - return AutoGenList - - ## Summarize the ModuleAutoGen objects of all libraries used by this m= odule - @cached_property - def LibraryAutoGenList(self): - RetVal =3D [] - for Library in self.DependentLibraryList: - La =3D ModuleAutoGen( - self.Workspace, - Library.MetaFile, - self.BuildTarget, - self.ToolChain, - self.Arch, - self.PlatformInfo.MetaFile - ) - if La not in RetVal: - RetVal.append(La) - for Lib in La.CodaTargetList: - self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE) - return RetVal - - def GenModuleHash(self): - # Initialize a dictionary for each arch type - if self.Arch not in GlobalData.gModuleHash: - GlobalData.gModuleHash[self.Arch] =3D {} - - # Early exit if module or library has been hashed and is in memory - if self.Name in GlobalData.gModuleHash[self.Arch]: - return GlobalData.gModuleHash[self.Arch][self.Name].encode('ut= f-8') - - # Initialze hash object - m =3D hashlib.md5() - - # Add Platform level hash - m.update(GlobalData.gPlatformHash.encode('utf-8')) - - # Add Package level hash - if self.DependentPackageList: - for Pkg in sorted(self.DependentPackageList, key=3Dlambda x: x= .PackageName): - if Pkg.PackageName in GlobalData.gPackageHash: - m.update(GlobalData.gPackageHash[Pkg.PackageName].enco= de('utf-8')) - - # Add Library hash - if self.LibraryAutoGenList: - for Lib in sorted(self.LibraryAutoGenList, key=3Dlambda x: x.N= ame): - if Lib.Name not in GlobalData.gModuleHash[self.Arch]: - Lib.GenModuleHash() - m.update(GlobalData.gModuleHash[self.Arch][Lib.Name].encod= e('utf-8')) - - # Add Module self - f =3D open(str(self.MetaFile), 'rb') - Content =3D f.read() - f.close() - m.update(Content) - - # Add Module's source files - if self.SourceFileList: - for File in sorted(self.SourceFileList, key=3Dlambda x: str(x)= ): - f =3D open(str(File), 'rb') - Content =3D f.read() - f.close() - m.update(Content) - - GlobalData.gModuleHash[self.Arch][self.Name] =3D m.hexdigest() - - return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8') - - ## Decide whether we can skip the ModuleAutoGen process - def CanSkipbyHash(self): - # Hashing feature is off - if not GlobalData.gUseHashCache: - return False - - # Initialize a dictionary for each arch type - if self.Arch not in GlobalData.gBuildHashSkipTracking: - GlobalData.gBuildHashSkipTracking[self.Arch] =3D dict() - - # If library or Module is binary do not skip by hash - if self.IsBinaryModule: - return False - - # .inc is contains binary information so do not skip by hash as we= ll - for f_ext in self.SourceFileList: - if '.inc' in str(f_ext): - return False - - # Use Cache, if exists and if Module has a copy in cache - if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy(): - return True - - # Early exit for libraries that haven't yet finished building - HashFile =3D path.join(self.BuildDir, self.Name + ".hash") - if self.IsLibrary and not os.path.exists(HashFile): - return False - - # Return a Boolean based on if can skip by hash, either from memor= y or from IO. - if self.Name not in GlobalData.gBuildHashSkipTracking[self.Arch]: - # If hashes are the same, SaveFileOnChange() will return False. - GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] =3D no= t SaveFileOnChange(HashFile, self.GenModuleHash(), True) - return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] - else: - return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] - - ## Decide whether we can skip the ModuleAutoGen process - # If any source file is newer than the module than we cannot skip - # - def CanSkip(self): - if self.MakeFileDir in GlobalData.gSikpAutoGenCache: - return True - if not os.path.exists(self.TimeStampPath): - return False - #last creation time of the module - DstTimeStamp =3D os.stat(self.TimeStampPath)[8] - - SrcTimeStamp =3D self.Workspace._SrcTimeStamp - if SrcTimeStamp > DstTimeStamp: - return False - - with open(self.TimeStampPath,'r') as f: - for source in f: - source =3D source.rstrip('\n') - if not os.path.exists(source): - return False - if source not in ModuleAutoGen.TimeDict : - ModuleAutoGen.TimeDict[source] =3D os.stat(source)[8] - if ModuleAutoGen.TimeDict[source] > DstTimeStamp: - return False - GlobalData.gSikpAutoGenCache.add(self.MakeFileDir) - return True - - @cached_property - def TimeStampPath(self): - return os.path.join(self.MakeFileDir, 'AutoGenTimeStamp') + @classmethod + def Cache(cls): + return cls.__ObjectCache + +# +# The priority list while override build option +# +PrioList =3D {"0x11111" : 16, # TARGET_TOOLCHAIN_ARCH_COMMANDTYPE_AT= TRIBUTE (Highest) + "0x01111" : 15, # ******_TOOLCHAIN_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x10111" : 14, # TARGET_*********_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x00111" : 13, # ******_*********_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x11011" : 12, # TARGET_TOOLCHAIN_****_COMMANDTYPE_ATTR= IBUTE + "0x01011" : 11, # ******_TOOLCHAIN_****_COMMANDTYPE_ATTR= IBUTE + "0x10011" : 10, # TARGET_*********_****_COMMANDTYPE_ATTR= IBUTE + "0x00011" : 9, # ******_*********_****_COMMANDTYPE_ATTR= IBUTE + "0x11101" : 8, # TARGET_TOOLCHAIN_ARCH_***********_ATTR= IBUTE + "0x01101" : 7, # ******_TOOLCHAIN_ARCH_***********_ATTR= IBUTE + "0x10101" : 6, # TARGET_*********_ARCH_***********_ATTR= IBUTE + "0x00101" : 5, # ******_*********_ARCH_***********_ATTR= IBUTE + "0x11001" : 4, # TARGET_TOOLCHAIN_****_***********_ATTR= IBUTE + "0x01001" : 3, # ******_TOOLCHAIN_****_***********_ATTR= IBUTE + "0x10001" : 2, # TARGET_*********_****_***********_ATTR= IBUTE + "0x00001" : 1} # ******_*********_****_***********_ATTR= IBUTE (Lowest) +## Calculate the priority value of the build option +# +# @param Key Build option definition contain: TARGET_TOOLCHAIN_ARCH_= COMMANDTYPE_ATTRIBUTE +# +# @retval Value Priority value based on the priority list. +# +def CalculatePriorityValue(Key): + Target, ToolChain, Arch, CommandType, Attr =3D Key.split('_') + PriorityValue =3D 0x11111 + if Target =3D=3D TAB_STAR: + PriorityValue &=3D 0x01111 + if ToolChain =3D=3D TAB_STAR: + PriorityValue &=3D 0x10111 + if Arch =3D=3D TAB_STAR: + PriorityValue &=3D 0x11011 + if CommandType =3D=3D TAB_STAR: + PriorityValue &=3D 0x11101 + if Attr =3D=3D TAB_STAR: + PriorityValue &=3D 0x11110 + + return PrioList["0x%0.5x" % PriorityValue] diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py new file mode 100644 index 000000000000..5bcc39bd380d --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -0,0 +1,147 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +from __future__ import absolute_import +from Workspace.WorkspaceDatabase import BuildDB +from Workspace.WorkspaceCommon import GetModuleLibInstances +import Common.GlobalData as GlobalData +import os +import pickle +from pickle import HIGHEST_PROTOCOL + +class PCD_DATA(): + def __init__(self,TokenCName,TokenSpaceGuidCName,Type,DatumType,SkuInf= oList,DefaultValue, + MaxDatumSize,UserDefinedDefaultStoresFlag,validateranges, + validlists,expressions,CustomAttribute,TokenValue): + self.TokenCName =3D TokenCName + self.TokenSpaceGuidCName =3D TokenSpaceGuidCName + self.Type =3D Type + self.DatumType =3D DatumType + self.SkuInfoList =3D SkuInfoList + self.DefaultValue =3D DefaultValue + self.MaxDatumSize =3D MaxDatumSize + self.UserDefinedDefaultStoresFlag =3D UserDefinedDefaultStoresFlag + self.validateranges =3D validateranges + self.validlists =3D validlists + self.expressions =3D expressions + self.CustomAttribute =3D CustomAttribute + self.TokenValue =3D TokenValue + +class DataPipe(object): + def __init__(self, BuildDir=3DNone): + self.data_container =3D {} + self.BuildDir =3D BuildDir + +class MemoryDataPipe(DataPipe): + + def Get(self,key): + return self.data_container.get(key) + + def dump(self,file_path): + with open(file_path,'wb') as fd: + pickle.dump(self.data_container,fd,pickle.HIGHEST_PROTOCOL) + + def load(self,file_path): + with open(file_path,'rb') as fd: + self.data_container =3D pickle.load(fd) + + @property + def DataContainer(self): + return self.data_container + @DataContainer.setter + def DataContainer(self,data): + self.data_container.update(data) + + def FillData(self,PlatformInfo): + #Platform Pcds + self.DataContainer =3D { + "PLA_PCD" : [PCD_DATA( + pcd.TokenCName,pcd.TokenSpaceGuidCName,pcd.Type, + pcd.DatumType,pcd.SkuInfoList,pcd.DefaultValue, + pcd.MaxDatumSize,pcd.UserDefinedDefaultStoresFlag,pcd.validate= ranges, + pcd.validlists,pcd.expressions,pcd.CustomAttribute,pcd.To= kenValue) + for pcd in PlatformInfo.Platform.Pcds.values()] + } + + #Platform Module Pcds + ModulePcds =3D {} + for m in PlatformInfo.Platform.Modules: + m_pcds =3D PlatformInfo.Platform.Modules[m].Pcds + if m_pcds: + ModulePcds[(m.File,m.Root)] =3D [PCD_DATA( + pcd.TokenCName,pcd.TokenSpaceGuidCName,pcd.Type, + pcd.DatumType,pcd.SkuInfoList,pcd.DefaultValue, + pcd.MaxDatumSize,pcd.UserDefinedDefaultStoresFlag,pcd.validate= ranges, + pcd.validlists,pcd.expressions,pcd.CustomAttribute,pcd.To= kenValue) + for pcd in PlatformInfo.Platform.Modules[m].Pcds.values()] + + + self.DataContainer =3D {"MOL_PCDS":ModulePcds} + + #Module's Library Instance + ModuleLibs =3D {} + for m in PlatformInfo.Platform.Modules: + module_obj =3D BuildDB.BuildObject[m,PlatformInfo.Arch,Platfor= mInfo.BuildTarget,PlatformInfo.ToolChain] + Libs =3D GetModuleLibInstances(module_obj, PlatformInfo.Platfo= rm, BuildDB.BuildObject, PlatformInfo.Arch,PlatformInfo.BuildTarget,Platfor= mInfo.ToolChain) + ModuleLibs[(m.File,m.Root,module_obj.Arch)] =3D [(l.MetaFile.F= ile,l.MetaFile.Root,l.Arch) for l in Libs] + self.DataContainer =3D {"DEPS":ModuleLibs} + + #Platform BuildOptions + + platform_build_opt =3D PlatformInfo.EdkIIBuildOption + + ToolDefinition =3D PlatformInfo.ToolDefinition + module_build_opt =3D {} + for m in PlatformInfo.Platform.Modules: + ModuleTypeOptions, PlatformModuleOptions =3D PlatformInfo.GetG= lobalBuildOptions(BuildDB.BuildObject[m,PlatformInfo.Arch,PlatformInfo.Buil= dTarget,PlatformInfo.ToolChain]) + if ModuleTypeOptions or PlatformModuleOptions: + module_build_opt.update({(m.File,m.Root): {"ModuleTypeOpti= ons":ModuleTypeOptions, "PlatformModuleOptions":PlatformModuleOptions}}) + + self.DataContainer =3D {"PLA_BO":platform_build_opt, + "TOOLDEF":ToolDefinition, + "MOL_BO":module_build_opt + } + + + + #Platform Info + PInfo =3D { + "WorkspaceDir":PlatformInfo.Workspace.WorkspaceDir, + "Target":PlatformInfo.BuildTarget, + "ToolChain":PlatformInfo.Workspace.ToolChain, + "BuildRuleFile":PlatformInfo.BuildRule, + "Arch": PlatformInfo.Arch, + "ArchList":PlatformInfo.Workspace.ArchList, + "ActivePlatform":PlatformInfo.MetaFile + } + self.DataContainer =3D {'P_Info':PInfo} + + self.DataContainer =3D {'M_Name':PlatformInfo.UniqueBaseName} + + self.DataContainer =3D {"ToolChainFamily": PlatformInfo.ToolChainF= amily} + + self.DataContainer =3D {"BuildRuleFamily": PlatformInfo.BuildRuleF= amily} + + self.DataContainer =3D {"MixedPcd":GlobalData.MixedPcd} + + self.DataContainer =3D {"BuildOptPcd":GlobalData.BuildOptionPcd} + + self.DataContainer =3D {"BuildCommand": PlatformInfo.BuildCommand} + + self.DataContainer =3D {"AsBuildModuleList": PlatformInfo._AsBuild= ModuleList} + + self.DataContainer =3D {"G_defines": GlobalData.gGlobalDefines} + + self.DataContainer =3D {"CL_defines": GlobalData.gCommandLineDefin= es} + + self.DataContainer =3D {"Env_Var": {k:v for k, v in os.environ.ite= ms()}} + + self.DataContainer =3D {"PackageList": [(dec.MetaFile,dec.Arch) fo= r dec in PlatformInfo.PackageList]} + + self.DataContainer =3D {"GuidDict": PlatformInfo.Platform._GuidDic= t} + + self.DataContainer =3D {"FdfParser": True if GlobalData.gFdfParser= else False} + diff --git a/BaseTools/Source/Python/AutoGen/GenC.py b/BaseTools/Source/Pyt= hon/AutoGen/GenC.py index 4cb776206e90..4c3f4e3e55ae 100644 --- a/BaseTools/Source/Python/AutoGen/GenC.py +++ b/BaseTools/Source/Python/AutoGen/GenC.py @@ -1627,11 +1627,11 @@ def CreatePcdCode(Info, AutoGenC, AutoGenH): TokenSpaceList =3D [] for Pcd in Info.ModulePcdList: if Pcd.Type in PCD_DYNAMIC_EX_TYPE_SET and Pcd.TokenSpaceGuidCName= not in TokenSpaceList: TokenSpaceList.append(Pcd.TokenSpaceGuidCName) =20 - SkuMgr =3D Info.Workspace.Platform.SkuIdMgr + SkuMgr =3D Info.PlatformInfo.Platform.SkuIdMgr AutoGenH.Append("\n// Definition of SkuId Array\n") AutoGenH.Append("extern UINT64 _gPcd_SkuId_Array[];\n") # Add extern declarations to AutoGen.h if one or more Token Space GUID= s were found if TokenSpaceList: AutoGenH.Append("\n// Definition of PCD Token Space GUIDs used in = this module\n\n") diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py new file mode 100644 index 000000000000..5fea71a86c83 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -0,0 +1,1887 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +from __future__ import absolute_import +from AutoGen.AutoGen import AutoGen +from Common.LongFilePathSupport import CopyLongFilePath +from Common.BuildToolError import * +from Common.DataType import * +from Common.Misc import * +from Common.StringUtils import NormPath,GetSplitList +from collections import defaultdict +from Workspace.WorkspaceCommon import OrderedListDict +import os.path as path +import copy +import hashlib +from . import InfSectionParser +from . import GenC +from . import GenMake +from . import GenDepex +from io import BytesIO +from GenPatchPcdTable.GenPatchPcdTable import parsePcdInfoFromMapFile +from Workspace.MetaFileCommentParser import UsageList +from .GenPcdDb import CreatePcdDatabaseCode +from Common.caching import cached_class_function +from AutoGen.ModuleAutoGenHelper import PlatformInfo,WorkSpaceInfo + +## Mapping Makefile type +gMakeTypeMap =3D {TAB_COMPILER_MSFT:"nmake", "GCC":"gmake"} +# +# Regular expression for finding Include Directories, the difference betwe= en MSFT and INTEL/GCC/RVCT +# is the former use /I , the Latter used -I to specify include directories +# +gBuildOptIncludePatternMsft =3D re.compile(r"(?:.*?)/I[ \t]*([^ ]*)", re.M= ULTILINE | re.DOTALL) +gBuildOptIncludePatternOther =3D re.compile(r"(?:.*?)-I[ \t]*([^ ]*)", re.= MULTILINE | re.DOTALL) + +## default file name for AutoGen +gAutoGenCodeFileName =3D "AutoGen.c" +gAutoGenHeaderFileName =3D "AutoGen.h" +gAutoGenStringFileName =3D "%(module_name)sStrDefs.h" +gAutoGenStringFormFileName =3D "%(module_name)sStrDefs.hpk" +gAutoGenDepexFileName =3D "%(module_name)s.depex" +gAutoGenImageDefFileName =3D "%(module_name)sImgDefs.h" +gAutoGenIdfFileName =3D "%(module_name)sIdf.hpk" +gInfSpecVersion =3D "0x00010017" + +# +# Match name =3D variable +# +gEfiVarStoreNamePattern =3D re.compile("\s*name\s*=3D\s*(\w+)") +# +# The format of guid in efivarstore statement likes following and must be = correct: +# guid =3D {0xA04A27f4, 0xDF00, 0x4D42, {0xB5, 0x52, 0x39, 0x51, 0x13, 0x0= 2, 0x11, 0x3D}} +# +gEfiVarStoreGuidPattern =3D re.compile("\s*guid\s*=3D\s*({.*?{.*?}\s*})") + +# +# Template string to generic AsBuilt INF +# +gAsBuiltInfHeaderString =3D TemplateString("""${header_comments} + +# DO NOT EDIT +# FILE auto-generated + +[Defines] + INF_VERSION =3D ${module_inf_version} + BASE_NAME =3D ${module_name} + FILE_GUID =3D ${module_guid} + MODULE_TYPE =3D ${module_module_type}${BEGIN} + VERSION_STRING =3D ${module_version_string}${END}${BEGIN} + PCD_IS_DRIVER =3D ${pcd_is_driver_string}${END}${BEGIN} + UEFI_SPECIFICATION_VERSION =3D ${module_uefi_specification_version}${END= }${BEGIN} + PI_SPECIFICATION_VERSION =3D ${module_pi_specification_version}${END}$= {BEGIN} + ENTRY_POINT =3D ${module_entry_point}${END}${BEGIN} + UNLOAD_IMAGE =3D ${module_unload_image}${END}${BEGIN} + CONSTRUCTOR =3D ${module_constructor}${END}${BEGIN} + DESTRUCTOR =3D ${module_destructor}${END}${BEGIN} + SHADOW =3D ${module_shadow}${END}${BEGIN} + PCI_VENDOR_ID =3D ${module_pci_vendor_id}${END}${BEGIN} + PCI_DEVICE_ID =3D ${module_pci_device_id}${END}${BEGIN} + PCI_CLASS_CODE =3D ${module_pci_class_code}${END}${BEGIN} + PCI_REVISION =3D ${module_pci_revision}${END}${BEGIN} + BUILD_NUMBER =3D ${module_build_number}${END}${BEGIN} + SPEC =3D ${module_spec}${END}${BEGIN} + UEFI_HII_RESOURCE_SECTION =3D ${module_uefi_hii_resource_section}${END}= ${BEGIN} + MODULE_UNI_FILE =3D ${module_uni_file}${END} + +[Packages.${module_arch}]${BEGIN} + ${package_item}${END} + +[Binaries.${module_arch}]${BEGIN} + ${binary_item}${END} + +[PatchPcd.${module_arch}]${BEGIN} + ${patchablepcd_item} +${END} + +[Protocols.${module_arch}]${BEGIN} + ${protocol_item} +${END} + +[Ppis.${module_arch}]${BEGIN} + ${ppi_item} +${END} + +[Guids.${module_arch}]${BEGIN} + ${guid_item} +${END} + +[PcdEx.${module_arch}]${BEGIN} + ${pcd_item} +${END} + +[LibraryClasses.${module_arch}] +## @LIB_INSTANCES${BEGIN} +# ${libraryclasses_item}${END} + +${depexsection_item} + +${userextension_tianocore_item} + +${tail_comments} + +[BuildOptions.${module_arch}] +## @AsBuilt${BEGIN} +## ${flags_item}${END} +""") +# +# extend lists contained in a dictionary with lists stored in another dict= ionary +# if CopyToDict is not derived from DefaultDict(list) then this may raise = exception +# +def ExtendCopyDictionaryLists(CopyToDict, CopyFromDict): + for Key in CopyFromDict: + CopyToDict[Key].extend(CopyFromDict[Key]) + +# Create a directory specified by a set of path elements and return the fu= ll path +def _MakeDir(PathList): + RetVal =3D path.join(*PathList) + CreateDirectory(RetVal) + return RetVal + +# +# Convert string to C format array +# +def _ConvertStringToByteArray(Value): + Value =3D Value.strip() + if not Value: + return None + if Value[0] =3D=3D '{': + if not Value.endswith('}'): + return None + Value =3D Value.replace(' ', '').replace('{', '').replace('}', '') + ValFields =3D Value.split(',') + try: + for Index in range(len(ValFields)): + ValFields[Index] =3D str(int(ValFields[Index], 0)) + except ValueError: + return None + Value =3D '{' + ','.join(ValFields) + '}' + return Value + + Unicode =3D False + if Value.startswith('L"'): + if not Value.endswith('"'): + return None + Value =3D Value[1:] + Unicode =3D True + elif not Value.startswith('"') or not Value.endswith('"'): + return None + + Value =3D eval(Value) # translate escape character + NewValue =3D '{' + for Index in range(0, len(Value)): + if Unicode: + NewValue =3D NewValue + str(ord(Value[Index]) % 0x10000) + ',' + else: + NewValue =3D NewValue + str(ord(Value[Index]) % 0x100) + ',' + Value =3D NewValue + '0}' + return Value + +## ModuleAutoGen class +# +# This class encapsules the AutoGen behaviors for the build tools. In addi= tion to +# the generation of AutoGen.h and AutoGen.c, it will generate *.depex file= according +# to the [depex] section in module's inf file. +# +class ModuleAutoGen(AutoGen): + # call super().__init__ then call the worker function with different p= arameter count + def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): + if not hasattr(self, "_Init"): + self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,= *args) + self._Init =3D True + + ## Cache the timestamps of metafiles of every module in a class attrib= ute + # + TimeDict =3D {} + + def __new__(cls, Workspace, MetaFile, Target, Toolchain, Arch, *args, = **kwargs): +# check if this module is employed by active platform + if not PlatformInfo(Workspace, args[0], Target, Toolchain, Arch,ar= gs[-1]).ValidModule(MetaFile): + EdkLogger.verbose("Module [%s] for [%s] is not employed by act= ive platform\n" \ + % (MetaFile, Arch)) + return None + return super(ModuleAutoGen, cls).__new__(cls, Workspace, MetaFile,= Target, Toolchain, Arch, *args, **kwargs) + + ## Initialize ModuleAutoGen + # + # @param Workspace EdkIIWorkspaceBuild object + # @param ModuleFile The path of module file + # @param Target Build target (DEBUG, RELEASE) + # @param Toolchain Name of tool chain + # @param Arch The arch the module supports + # @param PlatformFile Platform meta-file + # + def _InitWorker(self, Workspace, ModuleFile, Target, Toolchain, Arch, = PlatformFile,DataPipe): + EdkLogger.debug(EdkLogger.DEBUG_9, "AutoGen module [%s] [%s]" % (M= oduleFile, Arch)) + GlobalData.gProcessingFile =3D "%s [%s, %s, %s]" % (ModuleFile, Ar= ch, Toolchain, Target) + + self.Workspace =3D None + self.WorkspaceDir =3D "" + self.PlatformInfo =3D None + self.DataPipe =3D DataPipe + self.__init_platform_info__() + self.MetaFile =3D ModuleFile + self.SourceDir =3D self.MetaFile.SubDir + self.SourceDir =3D mws.relpath(self.SourceDir, self.WorkspaceDir) + + self.ToolChain =3D Toolchain + self.BuildTarget =3D Target + self.Arch =3D Arch + self.ToolChainFamily =3D self.PlatformInfo.ToolChainFamily + self.BuildRuleFamily =3D self.PlatformInfo.BuildRuleFamily + + self.IsCodeFileCreated =3D False + self.IsAsBuiltInfCreated =3D False + self.DepexGenerated =3D False + + self.BuildDatabase =3D self.Workspace.BuildDatabase + self.BuildRuleOrder =3D None + self.BuildTime =3D 0 + + self._GuidComments =3D OrderedListDict() + self._ProtocolComments =3D OrderedListDict() + self._PpiComments =3D OrderedListDict() + self._BuildTargets =3D None + self._IntroBuildTargetList =3D None + self._FinalBuildTargetList =3D None + self._FileTypes =3D None + + self.AutoGenDepSet =3D set() + self.ReferenceModules =3D [] + self.ConstPcd =3D {} + + def __init_platform_info__(self): + pinfo =3D self.DataPipe.Get("P_Info") + self.Workspace =3D WorkSpaceInfo(pinfo.get("WorkspaceDir"),pinfo.g= et("ActivePlatform"),pinfo.get("Target"),pinfo.get("ToolChain"),pinfo.get("= ArchList")) + self.WorkspaceDir =3D pinfo.get("WorkspaceDir") + self.PlatformInfo =3D PlatformInfo(self.Workspace,pinfo.get("Activ= ePlatform"),pinfo.get("Target"),pinfo.get("ToolChain"),pinfo.get("Arch"),se= lf.DataPipe) + def __repr__(self): + return "%s [%s]" % (self.MetaFile, self.Arch) + + # Get FixedAtBuild Pcds of this Module + @cached_property + def FixedAtBuildPcds(self): + RetVal =3D [] + for Pcd in self.ModulePcdList: + if Pcd.Type !=3D TAB_PCDS_FIXED_AT_BUILD: + continue + if Pcd not in RetVal: + RetVal.append(Pcd) + return RetVal + + @cached_property + def FixedVoidTypePcds(self): + RetVal =3D {} + for Pcd in self.FixedAtBuildPcds: + if Pcd.DatumType =3D=3D TAB_VOID: + if '.'.join((Pcd.TokenSpaceGuidCName, Pcd.TokenCName)) not= in RetVal: + RetVal['.'.join((Pcd.TokenSpaceGuidCName, Pcd.TokenCNa= me))] =3D Pcd.DefaultValue + return RetVal + + @property + def UniqueBaseName(self): + ModuleNames =3D self.DataPipe.Get("M_Name") + if not ModuleNames: + return self.Name + return ModuleNames.get(self.Name,self.Name) + + # Macros could be used in build_rule.txt (also Makefile) + @cached_property + def Macros(self): + return OrderedDict(( + ("WORKSPACE" ,self.WorkspaceDir), + ("MODULE_NAME" ,self.Name), + ("MODULE_NAME_GUID" ,self.UniqueBaseName), + ("MODULE_GUID" ,self.Guid), + ("MODULE_VERSION" ,self.Version), + ("MODULE_TYPE" ,self.ModuleType), + ("MODULE_FILE" ,str(self.MetaFile)), + ("MODULE_FILE_BASE_NAME" ,self.MetaFile.BaseName), + ("MODULE_RELATIVE_DIR" ,self.SourceDir), + ("MODULE_DIR" ,self.SourceDir), + ("BASE_NAME" ,self.Name), + ("ARCH" ,self.Arch), + ("TOOLCHAIN" ,self.ToolChain), + ("TOOLCHAIN_TAG" ,self.ToolChain), + ("TOOL_CHAIN_TAG" ,self.ToolChain), + ("TARGET" ,self.BuildTarget), + ("BUILD_DIR" ,self.PlatformInfo.BuildDir), + ("BIN_DIR" ,os.path.join(self.PlatformInfo.BuildDir, self.Arch= )), + ("LIB_DIR" ,os.path.join(self.PlatformInfo.BuildDir, self.Arch= )), + ("MODULE_BUILD_DIR" ,self.BuildDir), + ("OUTPUT_DIR" ,self.OutputDir), + ("DEBUG_DIR" ,self.DebugDir), + ("DEST_DIR_OUTPUT" ,self.OutputDir), + ("DEST_DIR_DEBUG" ,self.DebugDir), + ("PLATFORM_NAME" ,self.PlatformInfo.Name), + ("PLATFORM_GUID" ,self.PlatformInfo.Guid), + ("PLATFORM_VERSION" ,self.PlatformInfo.Version), + ("PLATFORM_RELATIVE_DIR" ,self.PlatformInfo.SourceDir), + ("PLATFORM_DIR" ,mws.join(self.WorkspaceDir, self.PlatformInfo= .SourceDir)), + ("PLATFORM_OUTPUT_DIR" ,self.PlatformInfo.OutputDir), + ("FFS_OUTPUT_DIR" ,self.FfsOutputDir) + )) + + ## Return the module build data object + @cached_property + def Module(self): + return self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarg= et, self.ToolChain] + + ## Return the module name + @cached_property + def Name(self): + return self.Module.BaseName + + ## Return the module DxsFile if exist + @cached_property + def DxsFile(self): + return self.Module.DxsFile + + ## Return the module meta-file GUID + @cached_property + def Guid(self): + # + # To build same module more than once, the module path with FILE_G= UID overridden has + # the file name FILE_GUIDmodule.inf, but the relative path (self.M= etaFile.File) is the real path + # in DSC. The overridden GUID can be retrieved from file name + # + if os.path.basename(self.MetaFile.File) !=3D os.path.basename(self= .MetaFile.Path): + # + # Length of GUID is 36 + # + return os.path.basename(self.MetaFile.Path)[:36] + return self.Module.Guid + + ## Return the module version + @cached_property + def Version(self): + return self.Module.Version + + ## Return the module type + @cached_property + def ModuleType(self): + return self.Module.ModuleType + + ## Return the component type (for Edk.x style of module) + @cached_property + def ComponentType(self): + return self.Module.ComponentType + + ## Return the build type + @cached_property + def BuildType(self): + return self.Module.BuildType + + ## Return the PCD_IS_DRIVER setting + @cached_property + def PcdIsDriver(self): + return self.Module.PcdIsDriver + + ## Return the autogen version, i.e. module meta-file version + @cached_property + def AutoGenVersion(self): + return self.Module.AutoGenVersion + + ## Check if the module is library or not + @cached_property + def IsLibrary(self): + return bool(self.Module.LibraryClass) + + ## Check if the module is binary module or not + @cached_property + def IsBinaryModule(self): + return self.Module.IsBinaryModule + + ## Return the directory to store intermediate files of the module + @cached_property + def BuildDir(self): + return _MakeDir(( + self.PlatformInfo.BuildDir, + self.Arch, + self.SourceDir, + self.MetaFile.BaseName + )) + + ## Return the directory to store the intermediate object files of the = module + @cached_property + def OutputDir(self): + return _MakeDir((self.BuildDir, "OUTPUT")) + + ## Return the directory path to store ffs file + @cached_property + def FfsOutputDir(self): + if GlobalData.gFdfParser: + return path.join(self.PlatformInfo.BuildDir, TAB_FV_DIRECTORY,= "Ffs", self.Guid + self.Name) + return '' + + ## Return the directory to store auto-gened source files of the module + @cached_property + def DebugDir(self): + return _MakeDir((self.BuildDir, "DEBUG")) + + ## Return the path of custom file + @cached_property + def CustomMakefile(self): + RetVal =3D {} + for Type in self.Module.CustomMakefile: + MakeType =3D gMakeTypeMap[Type] if Type in gMakeTypeMap else '= nmake' + File =3D os.path.join(self.SourceDir, self.Module.CustomMakefi= le[Type]) + RetVal[MakeType] =3D File + return RetVal + + ## Return the directory of the makefile + # + # @retval string The directory string of module's makefile + # + @cached_property + def MakeFileDir(self): + return self.BuildDir + + ## Return build command string + # + # @retval string Build command string + # + @cached_property + def BuildCommand(self): + return self.PlatformInfo.BuildCommand + + ## Get object list of all packages the module and its dependent librar= ies belong to + # + # @retval list The list of package object + # + @cached_property + def DerivedPackageList(self): + PackageList =3D [] + for M in [self.Module] + self.DependentLibraryList: + for Package in M.Packages: + if Package in PackageList: + continue + PackageList.append(Package) + return PackageList + + ## Get the depex string + # + # @return : a string contain all depex expression. + def _GetDepexExpresionString(self): + DepexStr =3D '' + DepexList =3D [] + ## DPX_SOURCE IN Define section. + if self.Module.DxsFile: + return DepexStr + for M in [self.Module] + self.DependentLibraryList: + Filename =3D M.MetaFile.Path + InfObj =3D InfSectionParser.InfSectionParser(Filename) + DepexExpressionList =3D InfObj.GetDepexExpresionList() + for DepexExpression in DepexExpressionList: + for key in DepexExpression: + Arch, ModuleType =3D key + DepexExpr =3D [x for x in DepexExpression[key] if not = str(x).startswith('#')] + # the type of build module is USER_DEFINED. + # All different DEPEX section tags would be copied int= o the As Built INF file + # and there would be separate DEPEX section tags + if self.ModuleType.upper() =3D=3D SUP_MODULE_USER_DEFI= NED or self.ModuleType.upper() =3D=3D SUP_MODULE_HOST_APPLICATION: + if (Arch.upper() =3D=3D self.Arch.upper()) and (Mo= duleType.upper() !=3D TAB_ARCH_COMMON): + DepexList.append({(Arch, ModuleType): DepexExp= r}) + else: + if Arch.upper() =3D=3D TAB_ARCH_COMMON or \ + (Arch.upper() =3D=3D self.Arch.upper() and \ + ModuleType.upper() in [TAB_ARCH_COMMON, self.Mod= uleType.upper()]): + DepexList.append({(Arch, ModuleType): DepexExp= r}) + + #the type of build module is USER_DEFINED. + if self.ModuleType.upper() =3D=3D SUP_MODULE_USER_DEFINED or self.= ModuleType.upper() =3D=3D SUP_MODULE_HOST_APPLICATION: + for Depex in DepexList: + for key in Depex: + DepexStr +=3D '[Depex.%s.%s]\n' % key + DepexStr +=3D '\n'.join('# '+ val for val in Depex[key= ]) + DepexStr +=3D '\n\n' + if not DepexStr: + return '[Depex.%s]\n' % self.Arch + return DepexStr + + #the type of build module not is USER_DEFINED. + Count =3D 0 + for Depex in DepexList: + Count +=3D 1 + if DepexStr !=3D '': + DepexStr +=3D ' AND ' + DepexStr +=3D '(' + for D in Depex.values(): + DepexStr +=3D ' '.join(val for val in D) + Index =3D DepexStr.find('END') + if Index > -1 and Index =3D=3D len(DepexStr) - 3: + DepexStr =3D DepexStr[:-3] + DepexStr =3D DepexStr.strip() + DepexStr +=3D ')' + if Count =3D=3D 1: + DepexStr =3D DepexStr.lstrip('(').rstrip(')').strip() + if not DepexStr: + return '[Depex.%s]\n' % self.Arch + return '[Depex.%s]\n# ' % self.Arch + DepexStr + + ## Merge dependency expression + # + # @retval list The token list of the dependency expression af= ter parsed + # + @cached_property + def DepexList(self): + if self.DxsFile or self.IsLibrary or TAB_DEPENDENCY_EXPRESSION_FIL= E in self.FileTypes: + return {} + + DepexList =3D [] + # + # Append depex from dependent libraries, if not "BEFORE", "AFTER" = expression + # + FixedVoidTypePcds =3D {} + for M in [self] + self.LibraryAutoGenList: + FixedVoidTypePcds.update(M.FixedVoidTypePcds) + for M in [self] + self.LibraryAutoGenList: + Inherited =3D False + for D in M.Module.Depex[self.Arch, self.ModuleType]: + if DepexList !=3D []: + DepexList.append('AND') + DepexList.append('(') + #replace D with value if D is FixedAtBuild PCD + NewList =3D [] + for item in D: + if '.' not in item: + NewList.append(item) + else: + try: + Value =3D FixedVoidTypePcds[item] + if len(Value.split(',')) !=3D 16: + EdkLogger.error("build", FORMAT_INVALID, + "{} used in [Depex] sectio= n should be used as FixedAtBuild type and VOID* datum type and 16 bytes in = the module.".format(item)) + NewList.append(Value) + except: + EdkLogger.error("build", FORMAT_INVALID, "{} u= sed in [Depex] section should be used as FixedAtBuild type and VOID* datum = type in the module.".format(item)) + + DepexList.extend(NewList) + if DepexList[-1] =3D=3D 'END': # no need of a END at this= time + DepexList.pop() + DepexList.append(')') + Inherited =3D True + if Inherited: + EdkLogger.verbose("DEPEX[%s] (+%s) =3D %s" % (self.Name, M= .Module.BaseName, DepexList)) + if 'BEFORE' in DepexList or 'AFTER' in DepexList: + break + if len(DepexList) > 0: + EdkLogger.verbose('') + return {self.ModuleType:DepexList} + + ## Merge dependency expression + # + # @retval list The token list of the dependency expression af= ter parsed + # + @cached_property + def DepexExpressionDict(self): + if self.DxsFile or self.IsLibrary or TAB_DEPENDENCY_EXPRESSION_FIL= E in self.FileTypes: + return {} + + DepexExpressionString =3D '' + # + # Append depex from dependent libraries, if not "BEFORE", "AFTER" = expresion + # + for M in [self.Module] + self.DependentLibraryList: + Inherited =3D False + for D in M.DepexExpression[self.Arch, self.ModuleType]: + if DepexExpressionString !=3D '': + DepexExpressionString +=3D ' AND ' + DepexExpressionString +=3D '(' + DepexExpressionString +=3D D + DepexExpressionString =3D DepexExpressionString.rstrip('EN= D').strip() + DepexExpressionString +=3D ')' + Inherited =3D True + if Inherited: + EdkLogger.verbose("DEPEX[%s] (+%s) =3D %s" % (self.Name, M= .BaseName, DepexExpressionString)) + if 'BEFORE' in DepexExpressionString or 'AFTER' in DepexExpres= sionString: + break + if len(DepexExpressionString) > 0: + EdkLogger.verbose('') + + return {self.ModuleType:DepexExpressionString} + + # Get the tiano core user extension, it is contain dependent library. + # @retval: a list contain tiano core userextension. + # + def _GetTianoCoreUserExtensionList(self): + TianoCoreUserExtentionList =3D [] + for M in [self.Module] + self.DependentLibraryList: + Filename =3D M.MetaFile.Path + InfObj =3D InfSectionParser.InfSectionParser(Filename) + TianoCoreUserExtenList =3D InfObj.GetUserExtensionTianoCore() + for TianoCoreUserExtent in TianoCoreUserExtenList: + for Section in TianoCoreUserExtent: + ItemList =3D Section.split(TAB_SPLIT) + Arch =3D self.Arch + if len(ItemList) =3D=3D 4: + Arch =3D ItemList[3] + if Arch.upper() =3D=3D TAB_ARCH_COMMON or Arch.upper()= =3D=3D self.Arch.upper(): + TianoCoreList =3D [] + TianoCoreList.extend([TAB_SECTION_START + Section = + TAB_SECTION_END]) + TianoCoreList.extend(TianoCoreUserExtent[Section][= :]) + TianoCoreList.append('\n') + TianoCoreUserExtentionList.append(TianoCoreList) + + return TianoCoreUserExtentionList + + ## Return the list of specification version required for the module + # + # @retval list The list of specification defined in module fi= le + # + @cached_property + def Specification(self): + return self.Module.Specification + + ## Tool option for the module build + # + # @param PlatformInfo The object of PlatformBuildInfo + # @retval dict The dict containing valid options + # + @cached_property + def BuildOption(self): + RetVal, self.BuildRuleOrder =3D self.PlatformInfo.ApplyBuildOption= (self.Module) + if self.BuildRuleOrder: + self.BuildRuleOrder =3D ['.%s' % Ext for Ext in self.BuildRule= Order.split()] + return RetVal + + ## Get include path list from tool option for the module build + # + # @retval list The include path list + # + @cached_property + def BuildOptionIncPathList(self): + # + # Regular expression for finding Include Directories, the differen= ce between MSFT and INTEL/GCC/RVCT + # is the former use /I , the Latter used -I to specify include dir= ectories + # + if self.PlatformInfo.ToolChainFamily in (TAB_COMPILER_MSFT): + BuildOptIncludeRegEx =3D gBuildOptIncludePatternMsft + elif self.PlatformInfo.ToolChainFamily in ('INTEL', 'GCC', 'RVCT'): + BuildOptIncludeRegEx =3D gBuildOptIncludePatternOther + else: + # + # New ToolChainFamily, don't known whether there is option to = specify include directories + # + return [] + + RetVal =3D [] + for Tool in ('CC', 'PP', 'VFRPP', 'ASLPP', 'ASLCC', 'APP', 'ASM'): + try: + FlagOption =3D self.BuildOption[Tool]['FLAGS'] + except KeyError: + FlagOption =3D '' + + if self.ToolChainFamily !=3D 'RVCT': + IncPathList =3D [NormPath(Path, self.Macros) for Path in B= uildOptIncludeRegEx.findall(FlagOption)] + else: + # + # RVCT may specify a list of directory seperated by commas + # + IncPathList =3D [] + for Path in BuildOptIncludeRegEx.findall(FlagOption): + PathList =3D GetSplitList(Path, TAB_COMMA_SPLIT) + IncPathList.extend(NormPath(PathEntry, self.Macros) fo= r PathEntry in PathList) + + # + # EDK II modules must not reference header files outside of th= e packages they depend on or + # within the module's directory tree. Report error if violatio= n. + # + if GlobalData.gDisableIncludePathCheck =3D=3D False: + for Path in IncPathList: + if (Path not in self.IncludePathList) and (CommonPath(= [Path, self.MetaFile.Dir]) !=3D self.MetaFile.Dir): + ErrMsg =3D "The include directory for the EDK II m= odule in this line is invalid %s specified in %s FLAGS '%s'" % (Path, Tool,= FlagOption) + EdkLogger.error("build", + PARAMETER_INVALID, + ExtraData=3DErrMsg, + File=3Dstr(self.MetaFile)) + RetVal +=3D IncPathList + return RetVal + + ## Return a list of files which can be built from source + # + # What kind of files can be built is determined by build rules in + # $(CONF_DIRECTORY)/build_rule.txt and toolchain family. + # + @cached_property + def SourceFileList(self): + RetVal =3D [] + ToolChainTagSet =3D {"", TAB_STAR, self.ToolChain} + ToolChainFamilySet =3D {"", TAB_STAR, self.ToolChainFamily, self.B= uildRuleFamily} + for F in self.Module.Sources: + # match tool chain + if F.TagName not in ToolChainTagSet: + EdkLogger.debug(EdkLogger.DEBUG_9, "The toolchain [%s] for= processing file [%s] is found, " + "but [%s] is currently used" % (F.TagName,= str(F), self.ToolChain)) + continue + # match tool chain family or build rule family + if F.ToolChainFamily not in ToolChainFamilySet: + EdkLogger.debug( + EdkLogger.DEBUG_0, + "The file [%s] must be built by tools of [%s],= " \ + "but current toolchain family is [%s], buildru= le family is [%s]" \ + % (str(F), F.ToolChainFamily, self.ToolCha= inFamily, self.BuildRuleFamily)) + continue + + # add the file path into search path list for file including + if F.Dir not in self.IncludePathList: + self.IncludePathList.insert(0, F.Dir) + RetVal.append(F) + + self._MatchBuildRuleOrder(RetVal) + + for F in RetVal: + self._ApplyBuildRule(F, TAB_UNKNOWN_FILE) + return RetVal + + def _MatchBuildRuleOrder(self, FileList): + Order_Dict =3D {} + self.BuildOption + for SingleFile in FileList: + if self.BuildRuleOrder and SingleFile.Ext in self.BuildRuleOrd= er and SingleFile.Ext in self.BuildRules: + key =3D SingleFile.Path.rsplit(SingleFile.Ext,1)[0] + if key in Order_Dict: + Order_Dict[key].append(SingleFile.Ext) + else: + Order_Dict[key] =3D [SingleFile.Ext] + + RemoveList =3D [] + for F in Order_Dict: + if len(Order_Dict[F]) > 1: + Order_Dict[F].sort(key=3Dlambda i: self.BuildRuleOrder.ind= ex(i)) + for Ext in Order_Dict[F][1:]: + RemoveList.append(F + Ext) + + for item in RemoveList: + FileList.remove(item) + + return FileList + + ## Return the list of unicode files + @cached_property + def UnicodeFileList(self): + return self.FileTypes.get(TAB_UNICODE_FILE,[]) + + ## Return the list of vfr files + @cached_property + def VfrFileList(self): + return self.FileTypes.get(TAB_VFR_FILE, []) + + ## Return the list of Image Definition files + @cached_property + def IdfFileList(self): + return self.FileTypes.get(TAB_IMAGE_FILE,[]) + + ## Return a list of files which can be built from binary + # + # "Build" binary files are just to copy them to build directory. + # + # @retval list The list of files which can be built l= ater + # + @cached_property + def BinaryFileList(self): + RetVal =3D [] + for F in self.Module.Binaries: + if F.Target not in [TAB_ARCH_COMMON, TAB_STAR] and F.Target != =3D self.BuildTarget: + continue + RetVal.append(F) + self._ApplyBuildRule(F, F.Type, BinaryFileList=3DRetVal) + return RetVal + + @cached_property + def BuildRules(self): + RetVal =3D {} + BuildRuleDatabase =3D self.PlatformInfo.BuildRule + for Type in BuildRuleDatabase.FileTypeList: + #first try getting build rule by BuildRuleFamily + RuleObject =3D BuildRuleDatabase[Type, self.BuildType, self.Ar= ch, self.BuildRuleFamily] + if not RuleObject: + # build type is always module type, but ... + if self.ModuleType !=3D self.BuildType: + RuleObject =3D BuildRuleDatabase[Type, self.ModuleType= , self.Arch, self.BuildRuleFamily] + #second try getting build rule by ToolChainFamily + if not RuleObject: + RuleObject =3D BuildRuleDatabase[Type, self.BuildType, sel= f.Arch, self.ToolChainFamily] + if not RuleObject: + # build type is always module type, but ... + if self.ModuleType !=3D self.BuildType: + RuleObject =3D BuildRuleDatabase[Type, self.Module= Type, self.Arch, self.ToolChainFamily] + if not RuleObject: + continue + RuleObject =3D RuleObject.Instantiate(self.Macros) + RetVal[Type] =3D RuleObject + for Ext in RuleObject.SourceFileExtList: + RetVal[Ext] =3D RuleObject + return RetVal + + def _ApplyBuildRule(self, File, FileType, BinaryFileList=3DNone): + if self._BuildTargets is None: + self._IntroBuildTargetList =3D set() + self._FinalBuildTargetList =3D set() + self._BuildTargets =3D defaultdict(set) + self._FileTypes =3D defaultdict(set) + + if not BinaryFileList: + BinaryFileList =3D self.BinaryFileList + + SubDirectory =3D os.path.join(self.OutputDir, File.SubDir) + if not os.path.exists(SubDirectory): + CreateDirectory(SubDirectory) + LastTarget =3D None + RuleChain =3D set() + SourceList =3D [File] + Index =3D 0 + # + # Make sure to get build rule order value + # + self.BuildOption + + while Index < len(SourceList): + Source =3D SourceList[Index] + Index =3D Index + 1 + + if Source !=3D File: + CreateDirectory(Source.Dir) + + if File.IsBinary and File =3D=3D Source and File in BinaryFile= List: + # Skip all files that are not binary libraries + if not self.IsLibrary: + continue + RuleObject =3D self.BuildRules[TAB_DEFAULT_BINARY_FILE] + elif FileType in self.BuildRules: + RuleObject =3D self.BuildRules[FileType] + elif Source.Ext in self.BuildRules: + RuleObject =3D self.BuildRules[Source.Ext] + else: + # stop at no more rules + if LastTarget: + self._FinalBuildTargetList.add(LastTarget) + break + + FileType =3D RuleObject.SourceFileType + self._FileTypes[FileType].add(Source) + + # stop at STATIC_LIBRARY for library + if self.IsLibrary and FileType =3D=3D TAB_STATIC_LIBRARY: + if LastTarget: + self._FinalBuildTargetList.add(LastTarget) + break + + Target =3D RuleObject.Apply(Source, self.BuildRuleOrder) + if not Target: + if LastTarget: + self._FinalBuildTargetList.add(LastTarget) + break + elif not Target.Outputs: + # Only do build for target with outputs + self._FinalBuildTargetList.add(Target) + + self._BuildTargets[FileType].add(Target) + + if not Source.IsBinary and Source =3D=3D File: + self._IntroBuildTargetList.add(Target) + + # to avoid cyclic rule + if FileType in RuleChain: + break + + RuleChain.add(FileType) + SourceList.extend(Target.Outputs) + LastTarget =3D Target + FileType =3D TAB_UNKNOWN_FILE + + @cached_property + def Targets(self): + if self._BuildTargets is None: + self._IntroBuildTargetList =3D set() + self._FinalBuildTargetList =3D set() + self._BuildTargets =3D defaultdict(set) + self._FileTypes =3D defaultdict(set) + + #TRICK: call SourceFileList property to apply build rule for sourc= e files + self.SourceFileList + + #TRICK: call _GetBinaryFileList to apply build rule for binary fil= es + self.BinaryFileList + + return self._BuildTargets + + @cached_property + def IntroTargetList(self): + self.Targets + return self._IntroBuildTargetList + + @cached_property + def CodaTargetList(self): + self.Targets + return self._FinalBuildTargetList + + @cached_property + def FileTypes(self): + self.Targets + return self._FileTypes + + ## Get the list of package object the module depends on + # + # @retval list The package object list + # + @cached_property + def DependentPackageList(self): + return self.Module.Packages + + ## Return the list of auto-generated code file + # + # @retval list The list of auto-generated file + # + @cached_property + def AutoGenFileList(self): + AutoGenUniIdf =3D self.BuildType !=3D 'UEFI_HII' + UniStringBinBuffer =3D BytesIO() + IdfGenBinBuffer =3D BytesIO() + RetVal =3D {} + AutoGenC =3D TemplateString() + AutoGenH =3D TemplateString() + StringH =3D TemplateString() + StringIdf =3D TemplateString() + GenC.CreateCode(self, AutoGenC, AutoGenH, StringH, AutoGenUniIdf, = UniStringBinBuffer, StringIdf, AutoGenUniIdf, IdfGenBinBuffer) + # + # AutoGen.c is generated if there are library classes in inf, or t= here are object files + # + if str(AutoGenC) !=3D "" and (len(self.Module.LibraryClasses) > 0 + or TAB_OBJECT_FILE in self.FileTypes): + AutoFile =3D PathClass(gAutoGenCodeFileName, self.DebugDir) + RetVal[AutoFile] =3D str(AutoGenC) + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if str(AutoGenH) !=3D "": + AutoFile =3D PathClass(gAutoGenHeaderFileName, self.DebugDir) + RetVal[AutoFile] =3D str(AutoGenH) + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if str(StringH) !=3D "": + AutoFile =3D PathClass(gAutoGenStringFileName % {"module_name"= :self.Name}, self.DebugDir) + RetVal[AutoFile] =3D str(StringH) + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if UniStringBinBuffer is not None and UniStringBinBuffer.getvalue(= ) !=3D b"": + AutoFile =3D PathClass(gAutoGenStringFormFileName % {"module_n= ame":self.Name}, self.OutputDir) + RetVal[AutoFile] =3D UniStringBinBuffer.getvalue() + AutoFile.IsBinary =3D True + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if UniStringBinBuffer is not None: + UniStringBinBuffer.close() + if str(StringIdf) !=3D "": + AutoFile =3D PathClass(gAutoGenImageDefFileName % {"module_nam= e":self.Name}, self.DebugDir) + RetVal[AutoFile] =3D str(StringIdf) + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if IdfGenBinBuffer is not None and IdfGenBinBuffer.getvalue() !=3D= b"": + AutoFile =3D PathClass(gAutoGenIdfFileName % {"module_name":se= lf.Name}, self.OutputDir) + RetVal[AutoFile] =3D IdfGenBinBuffer.getvalue() + AutoFile.IsBinary =3D True + self._ApplyBuildRule(AutoFile, TAB_UNKNOWN_FILE) + if IdfGenBinBuffer is not None: + IdfGenBinBuffer.close() + return RetVal + + ## Return the list of library modules explicitly or implicitly used by= this module + @cached_property + def DependentLibraryList(self): + # only merge library classes and PCD for non-library module + if self.IsLibrary: + return [] + return self.PlatformInfo.ApplyLibraryInstance(self.Module) + + ## Get the list of PCDs from current module + # + # @retval list The list of PCD + # + @cached_property + def ModulePcdList(self): + # apply PCD settings from platform + RetVal =3D self.PlatformInfo.ApplyPcdSetting(self.Module, self.Mod= ule.Pcds) + + return RetVal + @cached_property + def _PcdComments(self): + ReVal =3D OrderedListDict() + ExtendCopyDictionaryLists(ReVal, self.Module.PcdComments) + if not self.IsLibrary: + for Library in self.DependentLibraryList: + ExtendCopyDictionaryLists(ReVal, Library.PcdComments) + return ReVal + + ## Get the list of PCDs from dependent libraries + # + # @retval list The list of PCD + # + @cached_property + def LibraryPcdList(self): + if self.IsLibrary: + return [] + RetVal =3D [] + Pcds =3D set() + # get PCDs from dependent libraries + for Library in self.DependentLibraryList: + PcdsInLibrary =3D OrderedDict() + for Key in Library.Pcds: + # skip duplicated PCDs + if Key in self.Module.Pcds or Key in Pcds: + continue + Pcds.add(Key) + PcdsInLibrary[Key] =3D copy.copy(Library.Pcds[Key]) + RetVal.extend(self.PlatformInfo.ApplyPcdSetting(self.Module, P= cdsInLibrary, Library=3DLibrary)) + return RetVal + + ## Get the GUID value mapping + # + # @retval dict The mapping between GUID cname and its value + # + @cached_property + def GuidList(self): + RetVal =3D self.Module.Guids + for Library in self.DependentLibraryList: + RetVal.update(Library.Guids) + ExtendCopyDictionaryLists(self._GuidComments, Library.GuidComm= ents) + ExtendCopyDictionaryLists(self._GuidComments, self.Module.GuidComm= ents) + return RetVal + + @cached_property + def GetGuidsUsedByPcd(self): + RetVal =3D OrderedDict(self.Module.GetGuidsUsedByPcd()) + for Library in self.DependentLibraryList: + RetVal.update(Library.GetGuidsUsedByPcd()) + return RetVal + ## Get the protocol value mapping + # + # @retval dict The mapping between protocol cname and its val= ue + # + @cached_property + def ProtocolList(self): + RetVal =3D OrderedDict(self.Module.Protocols) + for Library in self.DependentLibraryList: + RetVal.update(Library.Protocols) + ExtendCopyDictionaryLists(self._ProtocolComments, Library.Prot= ocolComments) + ExtendCopyDictionaryLists(self._ProtocolComments, self.Module.Prot= ocolComments) + return RetVal + + ## Get the PPI value mapping + # + # @retval dict The mapping between PPI cname and its value + # + @cached_property + def PpiList(self): + RetVal =3D OrderedDict(self.Module.Ppis) + for Library in self.DependentLibraryList: + RetVal.update(Library.Ppis) + ExtendCopyDictionaryLists(self._PpiComments, Library.PpiCommen= ts) + ExtendCopyDictionaryLists(self._PpiComments, self.Module.PpiCommen= ts) + return RetVal + + ## Get the list of include search path + # + # @retval list The list path + # + @cached_property + def IncludePathList(self): + RetVal =3D [] + RetVal.append(self.MetaFile.Dir) + RetVal.append(self.DebugDir) + + for Package in self.Module.Packages: + PackageDir =3D mws.join(self.WorkspaceDir, Package.MetaFile.Di= r) + if PackageDir not in RetVal: + RetVal.append(PackageDir) + IncludesList =3D Package.Includes + if Package._PrivateIncludes: + if not self.MetaFile.OriginalPath.Path.startswith(PackageD= ir): + IncludesList =3D list(set(Package.Includes).difference= (set(Package._PrivateIncludes))) + for Inc in IncludesList: + if Inc not in RetVal: + RetVal.append(str(Inc)) + return RetVal + + @cached_property + def IncludePathLength(self): + return sum(len(inc)+1 for inc in self.IncludePathList) + + ## Get HII EX PCDs which maybe used by VFR + # + # efivarstore used by VFR may relate with HII EX PCDs + # Get the variable name and GUID from efivarstore and HII EX PCD + # List the HII EX PCDs in As Built INF if both name and GUID match. + # + # @retval list HII EX PCDs + # + def _GetPcdsMaybeUsedByVfr(self): + if not self.SourceFileList: + return [] + + NameGuids =3D set() + for SrcFile in self.SourceFileList: + if SrcFile.Ext.lower() !=3D '.vfr': + continue + Vfri =3D os.path.join(self.OutputDir, SrcFile.BaseName + '.i') + if not os.path.exists(Vfri): + continue + VfriFile =3D open(Vfri, 'r') + Content =3D VfriFile.read() + VfriFile.close() + Pos =3D Content.find('efivarstore') + while Pos !=3D -1: + # + # Make sure 'efivarstore' is the start of efivarstore stat= ement + # In case of the value of 'name' (name =3D efivarstore) is= equal to 'efivarstore' + # + Index =3D Pos - 1 + while Index >=3D 0 and Content[Index] in ' \t\r\n': + Index -=3D 1 + if Index >=3D 0 and Content[Index] !=3D ';': + Pos =3D Content.find('efivarstore', Pos + len('efivars= tore')) + continue + # + # 'efivarstore' must be followed by name and guid + # + Name =3D gEfiVarStoreNamePattern.search(Content, Pos) + if not Name: + break + Guid =3D gEfiVarStoreGuidPattern.search(Content, Pos) + if not Guid: + break + NameArray =3D _ConvertStringToByteArray('L"' + Name.group(= 1) + '"') + NameGuids.add((NameArray, GuidStructureStringToGuidString(= Guid.group(1)))) + Pos =3D Content.find('efivarstore', Name.end()) + if not NameGuids: + return [] + HiiExPcds =3D [] + for Pcd in self.PlatformInfo.Platform.Pcds.values(): + if Pcd.Type !=3D TAB_PCDS_DYNAMIC_EX_HII: + continue + for SkuInfo in Pcd.SkuInfoList.values(): + Value =3D GuidValue(SkuInfo.VariableGuid, self.PlatformInf= o.PackageList, self.MetaFile.Path) + if not Value: + continue + Name =3D _ConvertStringToByteArray(SkuInfo.VariableName) + Guid =3D GuidStructureStringToGuidString(Value) + if (Name, Guid) in NameGuids and Pcd not in HiiExPcds: + HiiExPcds.append(Pcd) + break + + return HiiExPcds + + def _GenOffsetBin(self): + VfrUniBaseName =3D {} + for SourceFile in self.Module.Sources: + if SourceFile.Type.upper() =3D=3D ".VFR" : + # + # search the .map file to find the offset of vfr binary in= the PE32+/TE file. + # + VfrUniBaseName[SourceFile.BaseName] =3D (SourceFile.BaseNa= me + "Bin") + elif SourceFile.Type.upper() =3D=3D ".UNI" : + # + # search the .map file to find the offset of Uni strings b= inary in the PE32+/TE file. + # + VfrUniBaseName["UniOffsetName"] =3D (self.Name + "Strings") + + if not VfrUniBaseName: + return None + MapFileName =3D os.path.join(self.OutputDir, self.Name + ".map") + EfiFileName =3D os.path.join(self.OutputDir, self.Name + ".efi") + VfrUniOffsetList =3D GetVariableOffset(MapFileName, EfiFileName, l= ist(VfrUniBaseName.values())) + if not VfrUniOffsetList: + return None + + OutputName =3D '%sOffset.bin' % self.Name + UniVfrOffsetFileName =3D os.path.join( self.OutputDir, OutputN= ame) + + try: + fInputfile =3D open(UniVfrOffsetFileName, "wb+", 0) + except: + EdkLogger.error("build", FILE_OPEN_FAILURE, "File open failed = for %s" % UniVfrOffsetFileName, None) + + # Use a instance of BytesIO to cache data + fStringIO =3D BytesIO() + + for Item in VfrUniOffsetList: + if (Item[0].find("Strings") !=3D -1): + # + # UNI offset in image. + # GUID + Offset + # { 0x8913c5e0, 0x33f6, 0x4d86, { 0x9b, 0xf1, 0x43, 0xef, = 0x89, 0xfc, 0x6, 0x66 } } + # + UniGuid =3D b'\xe0\xc5\x13\x89\xf63\x86M\x9b\xf1C\xef\x89\= xfc\x06f' + fStringIO.write(UniGuid) + UniValue =3D pack ('Q', int (Item[1], 16)) + fStringIO.write (UniValue) + else: + # + # VFR binary offset in image. + # GUID + Offset + # { 0xd0bc7cb4, 0x6a47, 0x495f, { 0xaa, 0x11, 0x71, 0x7, 0= x46, 0xda, 0x6, 0xa2 } }; + # + VfrGuid =3D b'\xb4|\xbc\xd0Gj_I\xaa\x11q\x07F\xda\x06\xa2' + fStringIO.write(VfrGuid) + VfrValue =3D pack ('Q', int (Item[1], 16)) + fStringIO.write (VfrValue) + # + # write data into file. + # + try : + fInputfile.write (fStringIO.getvalue()) + except: + EdkLogger.error("build", FILE_WRITE_FAILURE, "Write data to fi= le %s failed, please check whether the " + "file been locked or using by other applicatio= ns." %UniVfrOffsetFileName, None) + + fStringIO.close () + fInputfile.close () + return OutputName + + ## Create AsBuilt INF file the module + # + def CreateAsBuiltInf(self, IsOnlyCopy =3D False): + self.OutputFile =3D set() + if IsOnlyCopy and GlobalData.gBinCacheDest: + self.CopyModuleToCache() + return + + if self.IsAsBuiltInfCreated: + return + + # Skip the following code for libraries + if self.IsLibrary: + return + + # Skip the following code for modules with no source files + if not self.SourceFileList: + return + + # Skip the following code for modules without any binary files + if self.BinaryFileList: + return + + ### TODO: How to handles mixed source and binary modules + + # Find all DynamicEx and PatchableInModule PCDs used by this modul= e and dependent libraries + # Also find all packages that the DynamicEx PCDs depend on + Pcds =3D [] + PatchablePcds =3D [] + Packages =3D [] + PcdCheckList =3D [] + PcdTokenSpaceList =3D [] + for Pcd in self.ModulePcdList + self.LibraryPcdList: + if Pcd.Type =3D=3D TAB_PCDS_PATCHABLE_IN_MODULE: + PatchablePcds.append(Pcd) + PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGuidCNa= me, TAB_PCDS_PATCHABLE_IN_MODULE)) + elif Pcd.Type in PCD_DYNAMIC_EX_TYPE_SET: + if Pcd not in Pcds: + Pcds.append(Pcd) + PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGui= dCName, TAB_PCDS_DYNAMIC_EX)) + PcdCheckList.append((Pcd.TokenCName, Pcd.TokenSpaceGui= dCName, TAB_PCDS_DYNAMIC)) + PcdTokenSpaceList.append(Pcd.TokenSpaceGuidCName) + GuidList =3D OrderedDict(self.GuidList) + for TokenSpace in self.GetGuidsUsedByPcd: + # If token space is not referred by patch PCD or Ex PCD, remov= e the GUID from GUID list + # The GUIDs in GUIDs section should really be the GUIDs in sou= rce INF or referred by Ex an patch PCDs + if TokenSpace not in PcdTokenSpaceList and TokenSpace in GuidL= ist: + GuidList.pop(TokenSpace) + CheckList =3D (GuidList, self.PpiList, self.ProtocolList, PcdCheck= List) + for Package in self.DerivedPackageList: + if Package in Packages: + continue + BeChecked =3D (Package.Guids, Package.Ppis, Package.Protocols,= Package.Pcds) + Found =3D False + for Index in range(len(BeChecked)): + for Item in CheckList[Index]: + if Item in BeChecked[Index]: + Packages.append(Package) + Found =3D True + break + if Found: + break + + VfrPcds =3D self._GetPcdsMaybeUsedByVfr() + for Pkg in self.PlatformInfo.PackageList: + if Pkg in Packages: + continue + for VfrPcd in VfrPcds: + if ((VfrPcd.TokenCName, VfrPcd.TokenSpaceGuidCName, TAB_PC= DS_DYNAMIC_EX) in Pkg.Pcds or + (VfrPcd.TokenCName, VfrPcd.TokenSpaceGuidCName, TAB_PC= DS_DYNAMIC) in Pkg.Pcds): + Packages.append(Pkg) + break + + ModuleType =3D SUP_MODULE_DXE_DRIVER if self.ModuleType =3D=3D SUP= _MODULE_UEFI_DRIVER and self.DepexGenerated else self.ModuleType + DriverType =3D self.PcdIsDriver if self.PcdIsDriver else '' + Guid =3D self.Guid + MDefs =3D self.Module.Defines + + AsBuiltInfDict =3D { + 'module_name' : self.Name, + 'module_guid' : Guid, + 'module_module_type' : ModuleType, + 'module_version_string' : [MDefs['VERSION_STRING']] = if 'VERSION_STRING' in MDefs else [], + 'pcd_is_driver_string' : [], + 'module_uefi_specification_version' : [], + 'module_pi_specification_version' : [], + 'module_entry_point' : self.Module.ModuleEntryPoi= ntList, + 'module_unload_image' : self.Module.ModuleUnloadIm= ageList, + 'module_constructor' : self.Module.ConstructorLis= t, + 'module_destructor' : self.Module.DestructorList, + 'module_shadow' : [MDefs['SHADOW']] if 'SHAD= OW' in MDefs else [], + 'module_pci_vendor_id' : [MDefs['PCI_VENDOR_ID']] i= f 'PCI_VENDOR_ID' in MDefs else [], + 'module_pci_device_id' : [MDefs['PCI_DEVICE_ID']] i= f 'PCI_DEVICE_ID' in MDefs else [], + 'module_pci_class_code' : [MDefs['PCI_CLASS_CODE']] = if 'PCI_CLASS_CODE' in MDefs else [], + 'module_pci_revision' : [MDefs['PCI_REVISION']] if= 'PCI_REVISION' in MDefs else [], + 'module_build_number' : [MDefs['BUILD_NUMBER']] if= 'BUILD_NUMBER' in MDefs else [], + 'module_spec' : [MDefs['SPEC']] if 'SPEC' = in MDefs else [], + 'module_uefi_hii_resource_section' : [MDefs['UEFI_HII_RESOURCE_= SECTION']] if 'UEFI_HII_RESOURCE_SECTION' in MDefs else [], + 'module_uni_file' : [MDefs['MODULE_UNI_FILE']]= if 'MODULE_UNI_FILE' in MDefs else [], + 'module_arch' : self.Arch, + 'package_item' : [Package.MetaFile.File.rep= lace('\\', '/') for Package in Packages], + 'binary_item' : [], + 'patchablepcd_item' : [], + 'pcd_item' : [], + 'protocol_item' : [], + 'ppi_item' : [], + 'guid_item' : [], + 'flags_item' : [], + 'libraryclasses_item' : [] + } + + if 'MODULE_UNI_FILE' in MDefs: + UNIFile =3D os.path.join(self.MetaFile.Dir, MDefs['MODULE_UNI_= FILE']) + if os.path.isfile(UNIFile): + shutil.copy2(UNIFile, self.OutputDir) + + if self.AutoGenVersion > int(gInfSpecVersion, 0): + AsBuiltInfDict['module_inf_version'] =3D '0x%08x' % self.AutoG= enVersion + else: + AsBuiltInfDict['module_inf_version'] =3D gInfSpecVersion + + if DriverType: + AsBuiltInfDict['pcd_is_driver_string'].append(DriverType) + + if 'UEFI_SPECIFICATION_VERSION' in self.Specification: + AsBuiltInfDict['module_uefi_specification_version'].append(sel= f.Specification['UEFI_SPECIFICATION_VERSION']) + if 'PI_SPECIFICATION_VERSION' in self.Specification: + AsBuiltInfDict['module_pi_specification_version'].append(self.= Specification['PI_SPECIFICATION_VERSION']) + + OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') + DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') + for Item in self.CodaTargetList: + File =3D Item.Target.Path.replace('\\', '/').strip('/').replac= e(DebugDir, '').replace(OutputDir, '').strip('/') + self.OutputFile.add(File) + if os.path.isabs(File): + File =3D File.replace('\\', '/').strip('/').replace(Output= Dir, '').strip('/') + if Item.Target.Ext.lower() =3D=3D '.aml': + AsBuiltInfDict['binary_item'].append('ASL|' + File) + elif Item.Target.Ext.lower() =3D=3D '.acpi': + AsBuiltInfDict['binary_item'].append('ACPI|' + File) + elif Item.Target.Ext.lower() =3D=3D '.efi': + AsBuiltInfDict['binary_item'].append('PE32|' + self.Name += '.efi') + else: + AsBuiltInfDict['binary_item'].append('BIN|' + File) + if not self.DepexGenerated: + DepexFile =3D os.path.join(self.OutputDir, self.Name + '.depex= ') + if os.path.exists(DepexFile): + self.DepexGenerated =3D True + if self.DepexGenerated: + self.OutputFile.add(self.Name + '.depex') + if self.ModuleType in [SUP_MODULE_PEIM]: + AsBuiltInfDict['binary_item'].append('PEI_DEPEX|' + self.N= ame + '.depex') + elif self.ModuleType in [SUP_MODULE_DXE_DRIVER, SUP_MODULE_DXE= _RUNTIME_DRIVER, SUP_MODULE_DXE_SAL_DRIVER, SUP_MODULE_UEFI_DRIVER]: + AsBuiltInfDict['binary_item'].append('DXE_DEPEX|' + self.N= ame + '.depex') + elif self.ModuleType in [SUP_MODULE_DXE_SMM_DRIVER]: + AsBuiltInfDict['binary_item'].append('SMM_DEPEX|' + self.N= ame + '.depex') + + Bin =3D self._GenOffsetBin() + if Bin: + AsBuiltInfDict['binary_item'].append('BIN|%s' % Bin) + self.OutputFile.add(Bin) + + for Root, Dirs, Files in os.walk(OutputDir): + for File in Files: + if File.lower().endswith('.pdb'): + AsBuiltInfDict['binary_item'].append('DISPOSABLE|' + F= ile) + self.OutputFile.add(File) + HeaderComments =3D self.Module.HeaderComments + StartPos =3D 0 + for Index in range(len(HeaderComments)): + if HeaderComments[Index].find('@BinaryHeader') !=3D -1: + HeaderComments[Index] =3D HeaderComments[Index].replace('@= BinaryHeader', '@file') + StartPos =3D Index + break + AsBuiltInfDict['header_comments'] =3D '\n'.join(HeaderComments[Sta= rtPos:]).replace(':#', '://') + AsBuiltInfDict['tail_comments'] =3D '\n'.join(self.Module.TailComm= ents) + + GenList =3D [ + (self.ProtocolList, self._ProtocolComments, 'protocol_item'), + (self.PpiList, self._PpiComments, 'ppi_item'), + (GuidList, self._GuidComments, 'guid_item') + ] + for Item in GenList: + for CName in Item[0]: + Comments =3D '\n '.join(Item[1][CName]) if CName in Item[= 1] else '' + Entry =3D Comments + '\n ' + CName if Comments else CName + AsBuiltInfDict[Item[2]].append(Entry) + PatchList =3D parsePcdInfoFromMapFile( + os.path.join(self.OutputDir, self.Name + '.map= '), + os.path.join(self.OutputDir, self.Name + '.efi= ') + ) + if PatchList: + for Pcd in PatchablePcds: + TokenCName =3D Pcd.TokenCName + for PcdItem in GlobalData.MixedPcd: + if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) in Global= Data.MixedPcd[PcdItem]: + TokenCName =3D PcdItem[0] + break + for PatchPcd in PatchList: + if TokenCName =3D=3D PatchPcd[0]: + break + else: + continue + PcdValue =3D '' + if Pcd.DatumType =3D=3D 'BOOLEAN': + BoolValue =3D Pcd.DefaultValue.upper() + if BoolValue =3D=3D 'TRUE': + Pcd.DefaultValue =3D '1' + elif BoolValue =3D=3D 'FALSE': + Pcd.DefaultValue =3D '0' + + if Pcd.DatumType in TAB_PCD_NUMERIC_TYPES: + HexFormat =3D '0x%02x' + if Pcd.DatumType =3D=3D TAB_UINT16: + HexFormat =3D '0x%04x' + elif Pcd.DatumType =3D=3D TAB_UINT32: + HexFormat =3D '0x%08x' + elif Pcd.DatumType =3D=3D TAB_UINT64: + HexFormat =3D '0x%016x' + PcdValue =3D HexFormat % int(Pcd.DefaultValue, 0) + else: + if Pcd.MaxDatumSize is None or Pcd.MaxDatumSize =3D=3D= '': + EdkLogger.error("build", AUTOGEN_ERROR, + "Unknown [MaxDatumSize] of PCD [%s= .%s]" % (Pcd.TokenSpaceGuidCName, TokenCName) + ) + ArraySize =3D int(Pcd.MaxDatumSize, 0) + PcdValue =3D Pcd.DefaultValue + if PcdValue[0] !=3D '{': + Unicode =3D False + if PcdValue[0] =3D=3D 'L': + Unicode =3D True + PcdValue =3D PcdValue.lstrip('L') + PcdValue =3D eval(PcdValue) + NewValue =3D '{' + for Index in range(0, len(PcdValue)): + if Unicode: + CharVal =3D ord(PcdValue[Index]) + NewValue =3D NewValue + '0x%02x' % (CharVa= l & 0x00FF) + ', ' \ + + '0x%02x' % (CharVal >> 8) + ', ' + else: + NewValue =3D NewValue + '0x%02x' % (ord(Pc= dValue[Index]) % 0x100) + ', ' + Padding =3D '0x00, ' + if Unicode: + Padding =3D Padding * 2 + ArraySize =3D ArraySize // 2 + if ArraySize < (len(PcdValue) + 1): + if Pcd.MaxSizeUserSet: + EdkLogger.error("build", AUTOGEN_ERROR, + "The maximum size of VOID* typ= e PCD '%s.%s' is less than its actual size occupied." % (Pcd.TokenSpaceGuid= CName, TokenCName) + ) + else: + ArraySize =3D len(PcdValue) + 1 + if ArraySize > len(PcdValue) + 1: + NewValue =3D NewValue + Padding * (ArraySize -= len(PcdValue) - 1) + PcdValue =3D NewValue + Padding.strip().rstrip(','= ) + '}' + elif len(PcdValue.split(',')) <=3D ArraySize: + PcdValue =3D PcdValue.rstrip('}') + ', 0x00' * (Ar= raySize - len(PcdValue.split(','))) + PcdValue +=3D '}' + else: + if Pcd.MaxSizeUserSet: + EdkLogger.error("build", AUTOGEN_ERROR, + "The maximum size of VOID* type PC= D '%s.%s' is less than its actual size occupied." % (Pcd.TokenSpaceGuidCNam= e, TokenCName) + ) + else: + ArraySize =3D len(PcdValue) + 1 + PcdItem =3D '%s.%s|%s|0x%X' % \ + (Pcd.TokenSpaceGuidCName, TokenCName, PcdValue, PatchP= cd[1]) + PcdComments =3D '' + if (Pcd.TokenSpaceGuidCName, Pcd.TokenCName) in self._PcdC= omments: + PcdComments =3D '\n '.join(self._PcdComments[Pcd.Toke= nSpaceGuidCName, Pcd.TokenCName]) + if PcdComments: + PcdItem =3D PcdComments + '\n ' + PcdItem + AsBuiltInfDict['patchablepcd_item'].append(PcdItem) + + for Pcd in Pcds + VfrPcds: + PcdCommentList =3D [] + HiiInfo =3D '' + TokenCName =3D Pcd.TokenCName + for PcdItem in GlobalData.MixedPcd: + if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) in GlobalData= .MixedPcd[PcdItem]: + TokenCName =3D PcdItem[0] + break + if Pcd.Type =3D=3D TAB_PCDS_DYNAMIC_EX_HII: + for SkuName in Pcd.SkuInfoList: + SkuInfo =3D Pcd.SkuInfoList[SkuName] + HiiInfo =3D '## %s|%s|%s' % (SkuInfo.VariableName, Sku= Info.VariableGuid, SkuInfo.VariableOffset) + break + if (Pcd.TokenSpaceGuidCName, Pcd.TokenCName) in self._PcdComme= nts: + PcdCommentList =3D self._PcdComments[Pcd.TokenSpaceGuidCNa= me, Pcd.TokenCName][:] + if HiiInfo: + UsageIndex =3D -1 + UsageStr =3D '' + for Index, Comment in enumerate(PcdCommentList): + for Usage in UsageList: + if Comment.find(Usage) !=3D -1: + UsageStr =3D Usage + UsageIndex =3D Index + break + if UsageIndex !=3D -1: + PcdCommentList[UsageIndex] =3D '## %s %s %s' % (UsageS= tr, HiiInfo, PcdCommentList[UsageIndex].replace(UsageStr, '')) + else: + PcdCommentList.append('## UNDEFINED ' + HiiInfo) + PcdComments =3D '\n '.join(PcdCommentList) + PcdEntry =3D Pcd.TokenSpaceGuidCName + '.' + TokenCName + if PcdComments: + PcdEntry =3D PcdComments + '\n ' + PcdEntry + AsBuiltInfDict['pcd_item'].append(PcdEntry) + for Item in self.BuildOption: + if 'FLAGS' in self.BuildOption[Item]: + AsBuiltInfDict['flags_item'].append('%s:%s_%s_%s_%s_FLAGS = =3D %s' % (self.ToolChainFamily, self.BuildTarget, self.ToolChain, self.Arc= h, Item, self.BuildOption[Item]['FLAGS'].strip())) + + # Generated LibraryClasses section in comments. + for Library in self.LibraryAutoGenList: + AsBuiltInfDict['libraryclasses_item'].append(Library.MetaFile.= File.replace('\\', '/')) + + # Generated UserExtensions TianoCore section. + # All tianocore user extensions are copied. + UserExtStr =3D '' + for TianoCore in self._GetTianoCoreUserExtensionList(): + UserExtStr +=3D '\n'.join(TianoCore) + ExtensionFile =3D os.path.join(self.MetaFile.Dir, TianoCore[1]) + if os.path.isfile(ExtensionFile): + shutil.copy2(ExtensionFile, self.OutputDir) + AsBuiltInfDict['userextension_tianocore_item'] =3D UserExtStr + + # Generated depex expression section in comments. + DepexExpression =3D self._GetDepexExpresionString() + AsBuiltInfDict['depexsection_item'] =3D DepexExpression if DepexEx= pression else '' + + AsBuiltInf =3D TemplateString() + AsBuiltInf.Append(gAsBuiltInfHeaderString.Replace(AsBuiltInfDict)) + + SaveFileOnChange(os.path.join(self.OutputDir, self.Name + '.inf'),= str(AsBuiltInf), False) + + self.IsAsBuiltInfCreated =3D True + if GlobalData.gBinCacheDest: + self.CopyModuleToCache() + + def CopyModuleToCache(self): + FileDir =3D path.join(GlobalData.gBinCacheDest, self.PlatformInfo.= Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir, s= elf.MetaFile.BaseName) + CreateDirectory (FileDir) + HashFile =3D path.join(self.BuildDir, self.Name + '.hash') + if os.path.exists(HashFile): + shutil.copy2(HashFile, FileDir) + if not self.IsLibrary: + ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf') + if os.path.exists(ModuleFile): + shutil.copy2(ModuleFile, FileDir) + if not self.OutputFile: + Ma =3D self.BuildDatabase[self.MetaFile, self.Arch, self.Build= Target, self.ToolChain] + self.OutputFile =3D Ma.Binaries + if self.OutputFile: + for File in self.OutputFile: + File =3D str(File) + if not os.path.isabs(File): + File =3D os.path.join(self.OutputDir, File) + if os.path.exists(File): + sub_dir =3D os.path.relpath(File, self.OutputDir) + destination_file =3D os.path.join(FileDir, sub_dir) + destination_dir =3D os.path.dirname(destination_file) + CreateDirectory(destination_dir) + shutil.copy2(File, destination_dir) + + def AttemptModuleCacheCopy(self): + # If library or Module is binary do not skip by hash + if self.IsBinaryModule: + return False + # .inc is contains binary information so do not skip by hash as we= ll + for f_ext in self.SourceFileList: + if '.inc' in str(f_ext): + return False + FileDir =3D path.join(GlobalData.gBinCacheSource, self.PlatformInf= o.Name, self.BuildTarget + "_" + self.ToolChain, self.Arch, self.SourceDir,= self.MetaFile.BaseName) + HashFile =3D path.join(FileDir, self.Name + '.hash') + if os.path.exists(HashFile): + f =3D open(HashFile, 'r') + CacheHash =3D f.read() + f.close() + self.GenModuleHash() + if GlobalData.gModuleHash[self.Arch][self.Name]: + if CacheHash =3D=3D GlobalData.gModuleHash[self.Arch][self= .Name]: + for root, dir, files in os.walk(FileDir): + for f in files: + if self.Name + '.hash' in f: + shutil.copy2(HashFile, self.BuildDir) + else: + File =3D path.join(root, f) + sub_dir =3D os.path.relpath(File, FileDir) + destination_file =3D os.path.join(self.Out= putDir, sub_dir) + destination_dir =3D os.path.dirname(destin= ation_file) + CreateDirectory(destination_dir) + shutil.copy2(File, destination_dir) + if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "Pcd= Dxe": + CreatePcdDatabaseCode(self, TemplateString(), Temp= lateString()) + return True + return False + + ## Create makefile for the module and its dependent libraries + # + # @param CreateLibraryMakeFile Flag indicating if or not the = makefiles of + # dependent libraries will be cr= eated + # + @cached_class_function + def CreateMakeFile(self, CreateLibraryMakeFile=3DTrue, GenFfsList =3D = []): + # nest this function inside it's only caller. + def CreateTimeStamp(): + FileSet =3D {self.MetaFile.Path} + + for SourceFile in self.Module.Sources: + FileSet.add (SourceFile.Path) + + for Lib in self.DependentLibraryList: + FileSet.add (Lib.MetaFile.Path) + + for f in self.AutoGenDepSet: + FileSet.add (f.Path) + + if os.path.exists (self.TimeStampPath): + os.remove (self.TimeStampPath) + with open(self.TimeStampPath, 'w+') as fd: + for f in FileSet: + fd.write(f) + fd.write("\n") + + # Ignore generating makefile when it is a binary module + if self.IsBinaryModule: + return + + self.GenFfsList =3D GenFfsList + + if not self.IsLibrary and CreateLibraryMakeFile: + for LibraryAutoGen in self.LibraryAutoGenList: + LibraryAutoGen.CreateMakeFile() + if self.CanSkip(): + return + + if len(self.CustomMakefile) =3D=3D 0: + Makefile =3D GenMake.ModuleMakefile(self) + else: + Makefile =3D GenMake.CustomMakefile(self) + if Makefile.Generate(): + EdkLogger.debug(EdkLogger.DEBUG_9, "Generated makefile for mod= ule %s [%s]" % + (self.Name, self.Arch)) + else: + EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = makefile for module %s [%s]" % + (self.Name, self.Arch)) + + CreateTimeStamp() + + def CopyBinaryFiles(self): + for File in self.Module.Binaries: + SrcPath =3D File.Path + DstPath =3D os.path.join(self.OutputDir, os.path.basename(SrcP= ath)) + CopyLongFilePath(SrcPath, DstPath) + ## Create autogen code for the module and its dependent libraries + # + # @param CreateLibraryCodeFile Flag indicating if or not the = code of + # dependent libraries will be cr= eated + # + def CreateCodeFile(self, CreateLibraryCodeFile=3DTrue): + if self.IsCodeFileCreated: + return + + # Need to generate PcdDatabase even PcdDriver is binarymodule + if self.IsBinaryModule and self.PcdIsDriver !=3D '': + CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) + return + if self.IsBinaryModule: + if self.IsLibrary: + self.CopyBinaryFiles() + return + + if not self.IsLibrary and CreateLibraryCodeFile: + for LibraryAutoGen in self.LibraryAutoGenList: + LibraryAutoGen.CreateCodeFile() + + if self.CanSkip(): + return + + AutoGenList =3D [] + IgoredAutoGenList =3D [] + + for File in self.AutoGenFileList: + if GenC.Generate(File.Path, self.AutoGenFileList[File], File.I= sBinary): + AutoGenList.append(str(File)) + else: + IgoredAutoGenList.append(str(File)) + + + for ModuleType in self.DepexList: + # Ignore empty [depex] section or [depex] section for SUP_MODU= LE_USER_DEFINED module + if len(self.DepexList[ModuleType]) =3D=3D 0 or ModuleType =3D= =3D SUP_MODULE_USER_DEFINED or ModuleType =3D=3D SUP_MODULE_HOST_APPLICATIO= N: + continue + + Dpx =3D GenDepex.DependencyExpression(self.DepexList[ModuleTyp= e], ModuleType, True) + DpxFile =3D gAutoGenDepexFileName % {"module_name" : self.Name} + + if len(Dpx.PostfixNotation) !=3D 0: + self.DepexGenerated =3D True + + if Dpx.Generate(path.join(self.OutputDir, DpxFile)): + AutoGenList.append(str(DpxFile)) + else: + IgoredAutoGenList.append(str(DpxFile)) + + if IgoredAutoGenList =3D=3D []: + EdkLogger.debug(EdkLogger.DEBUG_9, "Generated [%s] files for m= odule %s [%s]" % + (" ".join(AutoGenList), self.Name, self.Arch)) + elif AutoGenList =3D=3D []: + EdkLogger.debug(EdkLogger.DEBUG_9, "Skipped the generation of = [%s] files for module %s [%s]" % + (" ".join(IgoredAutoGenList), self.Name, self.= Arch)) + else: + EdkLogger.debug(EdkLogger.DEBUG_9, "Generated [%s] (skipped %s= ) files for module %s [%s]" % + (" ".join(AutoGenList), " ".join(IgoredAutoGen= List), self.Name, self.Arch)) + + self.IsCodeFileCreated =3D True + return AutoGenList + + ## Summarize the ModuleAutoGen objects of all libraries used by this m= odule + @cached_property + def LibraryAutoGenList(self): + RetVal =3D [] + for Library in self.DependentLibraryList: + La =3D ModuleAutoGen( + self.Workspace, + Library.MetaFile, + self.BuildTarget, + self.ToolChain, + self.Arch, + self.PlatformInfo.MetaFile, + self.DataPipe + ) + La.IsLibrary =3D True + if La not in RetVal: + RetVal.append(La) + for Lib in La.CodaTargetList: + self._ApplyBuildRule(Lib.Target, TAB_UNKNOWN_FILE) + return RetVal + + def GenModuleHash(self): + # Initialize a dictionary for each arch type + if self.Arch not in GlobalData.gModuleHash: + GlobalData.gModuleHash[self.Arch] =3D {} + + # Early exit if module or library has been hashed and is in memory + if self.Name in GlobalData.gModuleHash[self.Arch]: + return GlobalData.gModuleHash[self.Arch][self.Name].encode('ut= f-8') + + # Initialze hash object + m =3D hashlib.md5() + + # Add Platform level hash + m.update(GlobalData.gPlatformHash.encode('utf-8')) + + # Add Package level hash + if self.DependentPackageList: + for Pkg in sorted(self.DependentPackageList, key=3Dlambda x: x= .PackageName): + if Pkg.PackageName in GlobalData.gPackageHash: + m.update(GlobalData.gPackageHash[Pkg.PackageName].enco= de('utf-8')) + + # Add Library hash + if self.LibraryAutoGenList: + for Lib in sorted(self.LibraryAutoGenList, key=3Dlambda x: x.N= ame): + if Lib.Name not in GlobalData.gModuleHash[self.Arch]: + Lib.GenModuleHash() + m.update(GlobalData.gModuleHash[self.Arch][Lib.Name].encod= e('utf-8')) + + # Add Module self + f =3D open(str(self.MetaFile), 'rb') + Content =3D f.read() + f.close() + m.update(Content) + + # Add Module's source files + if self.SourceFileList: + for File in sorted(self.SourceFileList, key=3Dlambda x: str(x)= ): + f =3D open(str(File), 'rb') + Content =3D f.read() + f.close() + m.update(Content) + + GlobalData.gModuleHash[self.Arch][self.Name] =3D m.hexdigest() + + return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8') + + ## Decide whether we can skip the ModuleAutoGen process + def CanSkipbyHash(self): + # Hashing feature is off + if not GlobalData.gUseHashCache: + return False + + # Initialize a dictionary for each arch type + if self.Arch not in GlobalData.gBuildHashSkipTracking: + GlobalData.gBuildHashSkipTracking[self.Arch] =3D dict() + + # If library or Module is binary do not skip by hash + if self.IsBinaryModule: + return False + + # .inc is contains binary information so do not skip by hash as we= ll + for f_ext in self.SourceFileList: + if '.inc' in str(f_ext): + return False + + # Use Cache, if exists and if Module has a copy in cache + if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy(): + return True + + # Early exit for libraries that haven't yet finished building + HashFile =3D path.join(self.BuildDir, self.Name + ".hash") + if self.IsLibrary and not os.path.exists(HashFile): + return False + + # Return a Boolean based on if can skip by hash, either from memor= y or from IO. + if self.Name not in GlobalData.gBuildHashSkipTracking[self.Arch]: + # If hashes are the same, SaveFileOnChange() will return False. + GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] =3D no= t SaveFileOnChange(HashFile, self.GenModuleHash(), True) + return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] + else: + return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] + + ## Decide whether we can skip the ModuleAutoGen process + # If any source file is newer than the module than we cannot skip + # + def CanSkip(self): + if self.MakeFileDir in GlobalData.gSikpAutoGenCache: + return True + if not os.path.exists(self.TimeStampPath): + return False + #last creation time of the module + DstTimeStamp =3D os.stat(self.TimeStampPath)[8] + + SrcTimeStamp =3D self.Workspace._SrcTimeStamp + if SrcTimeStamp > DstTimeStamp: + return False + + with open(self.TimeStampPath,'r') as f: + for source in f: + source =3D source.rstrip('\n') + if not os.path.exists(source): + return False + if source not in ModuleAutoGen.TimeDict : + ModuleAutoGen.TimeDict[source] =3D os.stat(source)[8] + if ModuleAutoGen.TimeDict[source] > DstTimeStamp: + return False + GlobalData.gSikpAutoGenCache.add(self.MakeFileDir) + return True + + @cached_property + def TimeStampPath(self): + return os.path.join(self.MakeFileDir, 'AutoGenTimeStamp') diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py b/BaseT= ools/Source/Python/AutoGen/ModuleAutoGenHelper.py new file mode 100644 index 000000000000..5186ca1da3e3 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py @@ -0,0 +1,616 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +from __future__ import absolute_import +from Workspace.WorkspaceDatabase import WorkspaceDatabase,BuildDB +from Common.caching import cached_property +from AutoGen.BuildEngine import BuildRule,AutoGenReqBuildRuleVerNum +from AutoGen.AutoGen import CalculatePriorityValue +from Common.Misc import CheckPcdDatum,GuidValue +from Common.Expression import ValueExpressionEx +from Common.DataType import * +from CommonDataClass.Exceptions import * +from CommonDataClass.CommonClass import SkuInfoClass +import Common.EdkLogger as EdkLogger +from Common.BuildToolError import OPTION_CONFLICT,FORMAT_INVALID,RESOURCE_= NOT_AVAILABLE +from Common.MultipleWorkspace import MultipleWorkspace as mws +from collections import defaultdict +from Common.Misc import PathClass +import os + + +# +# The priority list while override build option +# +PrioList =3D {"0x11111" : 16, # TARGET_TOOLCHAIN_ARCH_COMMANDTYPE_AT= TRIBUTE (Highest) + "0x01111" : 15, # ******_TOOLCHAIN_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x10111" : 14, # TARGET_*********_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x00111" : 13, # ******_*********_ARCH_COMMANDTYPE_ATTR= IBUTE + "0x11011" : 12, # TARGET_TOOLCHAIN_****_COMMANDTYPE_ATTR= IBUTE + "0x01011" : 11, # ******_TOOLCHAIN_****_COMMANDTYPE_ATTR= IBUTE + "0x10011" : 10, # TARGET_*********_****_COMMANDTYPE_ATTR= IBUTE + "0x00011" : 9, # ******_*********_****_COMMANDTYPE_ATTR= IBUTE + "0x11101" : 8, # TARGET_TOOLCHAIN_ARCH_***********_ATTR= IBUTE + "0x01101" : 7, # ******_TOOLCHAIN_ARCH_***********_ATTR= IBUTE + "0x10101" : 6, # TARGET_*********_ARCH_***********_ATTR= IBUTE + "0x00101" : 5, # ******_*********_ARCH_***********_ATTR= IBUTE + "0x11001" : 4, # TARGET_TOOLCHAIN_****_***********_ATTR= IBUTE + "0x01001" : 3, # ******_TOOLCHAIN_****_***********_ATTR= IBUTE + "0x10001" : 2, # TARGET_*********_****_***********_ATTR= IBUTE + "0x00001" : 1} # ******_*********_****_***********_ATTR= IBUTE (Lowest) +## Base class for AutoGen +# +# This class just implements the cache mechanism of AutoGen objects. +# +class AutoGenInfo(object): + # database to maintain the objects in each child class + __ObjectCache =3D {} # (BuildTarget, ToolChain, ARCH, platform file= ): AutoGen object + + ## Factory method + # + # @param Class class object of real AutoGen class + # (WorkspaceAutoGen, ModuleAutoGen or Platfo= rmAutoGen) + # @param Workspace Workspace directory or WorkspaceAutoGen ob= ject + # @param MetaFile The path of meta file + # @param Target Build target + # @param Toolchain Tool chain name + # @param Arch Target arch + # @param *args The specific class related parameters + # @param **kwargs The specific class related dict parameters + # + @classmethod + def GetCache(cls): + return cls.__ObjectCache + def __new__(cls, Workspace, MetaFile, Target, Toolchain, Arch, *args, = **kwargs): + # check if the object has been created + Key =3D (Target, Toolchain, Arch, MetaFile) + if Key in cls.__ObjectCache: + # if it exists, just return it directly + return cls.__ObjectCache[Key] + # it didnt exist. create it, cache it, then return it + RetVal =3D cls.__ObjectCache[Key] =3D super(AutoGenInfo, cls).__ne= w__(cls) + return RetVal + + + ## hash() operator + # + # The file path of platform file will be used to represent hash value= of this object + # + # @retval int Hash value of the file path of platform file + # + def __hash__(self): + return hash(self.MetaFile) + + ## str() operator + # + # The file path of platform file will be used to represent this object + # + # @retval string String of platform file path + # + def __str__(self): + return str(self.MetaFile) + + ## "=3D=3D" operator + def __eq__(self, Other): + return Other and self.MetaFile =3D=3D Other + + ## Expand * in build option key + # + # @param Options Options to be expanded + # @param ToolDef Use specified ToolDef instead of full version. + # This is needed during initialization to prevent + # infinite recursion betweeh BuildOptions, + # ToolDefinition, and this function. + # + # @retval options Options expanded + # + def _ExpandBuildOption(self, Options, ModuleStyle=3DNone, ToolDef=3DNo= ne): + if not ToolDef: + ToolDef =3D self.ToolDefinition + BuildOptions =3D {} + FamilyMatch =3D False + FamilyIsNull =3D True + + OverrideList =3D {} + # + # Construct a list contain the build options which need override. + # + for Key in Options: + # + # Key[0] -- tool family + # Key[1] -- TARGET_TOOLCHAIN_ARCH_COMMANDTYPE_ATTRIBUTE + # + if (Key[0] =3D=3D self.BuildRuleFamily and + (ModuleStyle is None or len(Key) < 3 or (len(Key) > 2 and = Key[2] =3D=3D ModuleStyle))): + Target, ToolChain, Arch, CommandType, Attr =3D Key[1].spli= t('_') + if (Target =3D=3D self.BuildTarget or Target =3D=3D TAB_ST= AR) and\ + (ToolChain =3D=3D self.ToolChain or ToolChain =3D=3D T= AB_STAR) and\ + (Arch =3D=3D self.Arch or Arch =3D=3D TAB_STAR) and\ + Options[Key].startswith("=3D"): + + if OverrideList.get(Key[1]) is not None: + OverrideList.pop(Key[1]) + OverrideList[Key[1]] =3D Options[Key] + + # + # Use the highest priority value. + # + if (len(OverrideList) >=3D 2): + KeyList =3D list(OverrideList.keys()) + for Index in range(len(KeyList)): + NowKey =3D KeyList[Index] + Target1, ToolChain1, Arch1, CommandType1, Attr1 =3D NowKey= .split("_") + for Index1 in range(len(KeyList) - Index - 1): + NextKey =3D KeyList[Index1 + Index + 1] + # + # Compare two Key, if one is included by another, choo= se the higher priority one + # + Target2, ToolChain2, Arch2, CommandType2, Attr2 =3D Ne= xtKey.split("_") + if (Target1 =3D=3D Target2 or Target1 =3D=3D TAB_STAR = or Target2 =3D=3D TAB_STAR) and\ + (ToolChain1 =3D=3D ToolChain2 or ToolChain1 =3D=3D= TAB_STAR or ToolChain2 =3D=3D TAB_STAR) and\ + (Arch1 =3D=3D Arch2 or Arch1 =3D=3D TAB_STAR or Ar= ch2 =3D=3D TAB_STAR) and\ + (CommandType1 =3D=3D CommandType2 or CommandType1 = =3D=3D TAB_STAR or CommandType2 =3D=3D TAB_STAR) and\ + (Attr1 =3D=3D Attr2 or Attr1 =3D=3D TAB_STAR or At= tr2 =3D=3D TAB_STAR): + + if CalculatePriorityValue(NowKey) > CalculatePrior= ityValue(NextKey): + if Options.get((self.BuildRuleFamily, NextKey)= ) is not None: + Options.pop((self.BuildRuleFamily, NextKey= )) + else: + if Options.get((self.BuildRuleFamily, NowKey))= is not None: + Options.pop((self.BuildRuleFamily, NowKey)) + + for Key in Options: + if ModuleStyle is not None and len (Key) > 2: + # Check Module style is EDK or EDKII. + # Only append build option for the matched style module. + if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: + continue + elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: + continue + Family =3D Key[0] + Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") + # if tool chain family doesn't match, skip it + if Tool in ToolDef and Family !=3D "": + FamilyIsNull =3D False + if ToolDef[Tool].get(TAB_TOD_DEFINES_BUILDRULEFAMILY, "") = !=3D "": + if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_BUILDRULE= FAMILY]: + continue + elif Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: + continue + FamilyMatch =3D True + # expand any wildcard + if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: + if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: + if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: + if Tool not in BuildOptions: + BuildOptions[Tool] =3D {} + if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): + BuildOptions[Tool][Attr] =3D Options[Key] + else: + # append options for the same tool except PATH + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] + else: + BuildOptions[Tool][Attr] =3D Options[Key] + # Build Option Family has been checked, which need't to be checked= again for family. + if FamilyMatch or FamilyIsNull: + return BuildOptions + + for Key in Options: + if ModuleStyle is not None and len (Key) > 2: + # Check Module style is EDK or EDKII. + # Only append build option for the matched style module. + if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: + continue + elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: + continue + Family =3D Key[0] + Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") + # if tool chain family doesn't match, skip it + if Tool not in ToolDef or Family =3D=3D "": + continue + # option has been added before + if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: + continue + + # expand any wildcard + if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: + if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: + if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: + if Tool not in BuildOptions: + BuildOptions[Tool] =3D {} + if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): + BuildOptions[Tool][Attr] =3D Options[Key] + else: + # append options for the same tool except PATH + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] + else: + BuildOptions[Tool][Attr] =3D Options[Key] + return BuildOptions +# +#This class is the pruned WorkSpaceAutoGen for ModuleAutoGen in multiple t= hread +# +class WorkSpaceInfo(AutoGenInfo): + def __init__(self,Workspace, MetaFile, Target, ToolChain, Arch): + self._SrcTimeStamp =3D 0 + self.Db =3D BuildDB + self.BuildDatabase =3D self.Db.BuildObject + self.Target =3D Target + self.ToolChain =3D ToolChain + self.WorkspaceDir =3D Workspace + self.ActivePlatform =3D MetaFile + self.ArchList =3D Arch + + +class PlatformInfo(AutoGenInfo): + def __init__(self, Workspace, MetaFile, Target, ToolChain, Arch,DataPi= pe): + self.Wa =3D Workspace + self.WorkspaceDir =3D self.Wa.WorkspaceDir + self.MetaFile =3D MetaFile + self.Arch =3D Arch + self.Target =3D Target + self.BuildTarget =3D Target + self.ToolChain =3D ToolChain + self.Platform =3D self.Wa.BuildDatabase[self.MetaFile, self.Arch, = self.Target, self.ToolChain] + + self.SourceDir =3D MetaFile.SubDir + self.DataPipe =3D DataPipe + @cached_property + def _AsBuildModuleList(self): + retVal =3D self.DataPipe.Get("AsBuildModuleList") + if retVal is None: + retVal =3D {} + return retVal + + ## Test if a module is supported by the platform + # + # An error will be raised directly if the module or its arch is not s= upported + # by the platform or current configuration + # + def ValidModule(self, Module): + return Module in self.Platform.Modules or Module in self.Platform.= LibraryInstances \ + or Module in self._AsBuildModuleList + + @cached_property + def ToolChainFamily(self): + retVal =3D self.DataPipe.Get("ToolChainFamily") + if retVal is None: + retVal =3D {} + return retVal + + @cached_property + def BuildRuleFamily(self): + retVal =3D self.DataPipe.Get("BuildRuleFamily") + if retVal is None: + retVal =3D {} + return retVal + + @cached_property + def _MbList(self): + return [self.Wa.BuildDatabase[m, self.Arch, self.BuildTarget, self= .ToolChain] for m in self.Platform.Modules] + + @cached_property + def PackageList(self): + RetVal =3D set() + for dec_file,Arch in self.DataPipe.Get("PackageList"): + RetVal.add(self.Wa.BuildDatabase[dec_file,Arch,self.BuildTarge= t, self.ToolChain]) + return list(RetVal) + + ## Return the directory to store all intermediate and final files built + @cached_property + def BuildDir(self): + if os.path.isabs(self.OutputDir): + RetVal =3D os.path.join( + os.path.abspath(self.OutputDir), + self.Target + "_" + self.ToolChain, + ) + else: + RetVal =3D os.path.join( + self.WorkspaceDir, + self.OutputDir, + self.Target + "_" + self.ToolChain, + ) + return RetVal + + ## Return the build output directory platform specifies + @cached_property + def OutputDir(self): + return self.Platform.OutputDirectory + + ## Return platform name + @cached_property + def Name(self): + return self.Platform.PlatformName + + ## Return meta-file GUID + @cached_property + def Guid(self): + return self.Platform.Guid + + ## Return platform version + @cached_property + def Version(self): + return self.Platform.Version + + ## Return paths of tools + @cached_property + def ToolDefinition(self): + retVal =3D self.DataPipe.Get("TOOLDEF") + if retVal is None: + retVal =3D {} + return retVal + + ## Return build command string + # + # @retval string Build command string + # + @cached_property + def BuildCommand(self): + retVal =3D self.DataPipe.Get("BuildCommand") + if retVal is None: + retVal =3D [] + return retVal + + @cached_property + def PcdTokenNumber(self): + retVal =3D self.DataPipe.Get("PCD_TNUM") + if retVal is None: + retVal =3D {} + return retVal + + ## Override PCD setting (type, value, ...) + # + # @param ToPcd The PCD to be overridden + # @param FromPcd The PCD overriding from + # + def _OverridePcd(self, ToPcd, FromPcd, Module=3D"", Msg=3D"", Library= =3D""): + # + # in case there's PCDs coming from FDF file, which have no type gi= ven. + # at this point, ToPcd.Type has the type found from dependent + # package + # + TokenCName =3D ToPcd.TokenCName + for PcdItem in self.MixedPcd: + if (ToPcd.TokenCName, ToPcd.TokenSpaceGuidCName) in self.Mixed= Pcd[PcdItem]: + TokenCName =3D PcdItem[0] + break + if FromPcd is not None: + if ToPcd.Pending and FromPcd.Type: + ToPcd.Type =3D FromPcd.Type + elif ToPcd.Type and FromPcd.Type\ + and ToPcd.Type !=3D FromPcd.Type and ToPcd.Type in FromPcd= .Type: + if ToPcd.Type.strip() =3D=3D TAB_PCDS_DYNAMIC_EX: + ToPcd.Type =3D FromPcd.Type + elif ToPcd.Type and FromPcd.Type \ + and ToPcd.Type !=3D FromPcd.Type: + if Library: + Module =3D str(Module) + " 's library file (" + str(Li= brary) + ")" + EdkLogger.error("build", OPTION_CONFLICT, "Mismatched PCD = type", + ExtraData=3D"%s.%s is used as [%s] in modu= le %s, but as [%s] in %s."\ + % (ToPcd.TokenSpaceGuidCName, To= kenCName, + ToPcd.Type, Module, FromPcd.T= ype, Msg), + File=3Dself.MetaFile) + + if FromPcd.MaxDatumSize: + ToPcd.MaxDatumSize =3D FromPcd.MaxDatumSize + ToPcd.MaxSizeUserSet =3D FromPcd.MaxDatumSize + if FromPcd.DefaultValue: + ToPcd.DefaultValue =3D FromPcd.DefaultValue + if FromPcd.TokenValue: + ToPcd.TokenValue =3D FromPcd.TokenValue + if FromPcd.DatumType: + ToPcd.DatumType =3D FromPcd.DatumType + if FromPcd.SkuInfoList: + ToPcd.SkuInfoList =3D FromPcd.SkuInfoList + if FromPcd.UserDefinedDefaultStoresFlag: + ToPcd.UserDefinedDefaultStoresFlag =3D FromPcd.UserDefined= DefaultStoresFlag + # Add Flexible PCD format parse + if ToPcd.DefaultValue: + try: + ToPcd.DefaultValue =3D ValueExpressionEx(ToPcd.Default= Value, ToPcd.DatumType, self._GuidDict)(True) + except BadExpression as Value: + EdkLogger.error('Parser', FORMAT_INVALID, 'PCD [%s.%s]= Value "%s", %s' %(ToPcd.TokenSpaceGuidCName, ToPcd.TokenCName, ToPcd.Defau= ltValue, Value), + File=3Dself.MetaFile) + + # check the validation of datum + IsValid, Cause =3D CheckPcdDatum(ToPcd.DatumType, ToPcd.Defaul= tValue) + if not IsValid: + EdkLogger.error('build', FORMAT_INVALID, Cause, File=3Dsel= f.MetaFile, + ExtraData=3D"%s.%s" % (ToPcd.TokenSpaceGui= dCName, TokenCName)) + ToPcd.validateranges =3D FromPcd.validateranges + ToPcd.validlists =3D FromPcd.validlists + ToPcd.expressions =3D FromPcd.expressions + ToPcd.CustomAttribute =3D FromPcd.CustomAttribute + + if FromPcd is not None and ToPcd.DatumType =3D=3D TAB_VOID and not= ToPcd.MaxDatumSize: + EdkLogger.debug(EdkLogger.DEBUG_9, "No MaxDatumSize specified = for PCD %s.%s" \ + % (ToPcd.TokenSpaceGuidCName, TokenCName)) + Value =3D ToPcd.DefaultValue + if not Value: + ToPcd.MaxDatumSize =3D '1' + elif Value[0] =3D=3D 'L': + ToPcd.MaxDatumSize =3D str((len(Value) - 2) * 2) + elif Value[0] =3D=3D '{': + ToPcd.MaxDatumSize =3D str(len(Value.split(','))) + else: + ToPcd.MaxDatumSize =3D str(len(Value) - 1) + + # apply default SKU for dynamic PCDS if specified one is not avail= able + if (ToPcd.Type in PCD_DYNAMIC_TYPE_SET or ToPcd.Type in PCD_DYNAMI= C_EX_TYPE_SET) \ + and not ToPcd.SkuInfoList: + if self.Platform.SkuName in self.Platform.SkuIds: + SkuName =3D self.Platform.SkuName + else: + SkuName =3D TAB_DEFAULT + ToPcd.SkuInfoList =3D { + SkuName : SkuInfoClass(SkuName, self.Platform.SkuIds[SkuNa= me][0], '', '', '', '', '', ToPcd.DefaultValue) + } + + def ApplyPcdSetting(self, Module, Pcds, Library=3D""): + # for each PCD in module + for Name, Guid in Pcds: + PcdInModule =3D Pcds[Name, Guid] + # find out the PCD setting in platform + if (Name, Guid) in self.Pcds: + PcdInPlatform =3D self.Pcds[Name, Guid] + else: + PcdInPlatform =3D None + # then override the settings if any + self._OverridePcd(PcdInModule, PcdInPlatform, Module, Msg=3D"D= SC PCD sections", Library=3DLibrary) + # resolve the VariableGuid value + for SkuId in PcdInModule.SkuInfoList: + Sku =3D PcdInModule.SkuInfoList[SkuId] + if Sku.VariableGuid =3D=3D '': continue + Sku.VariableGuidValue =3D GuidValue(Sku.VariableGuid, self= .PackageList, self.MetaFile.Path) + if Sku.VariableGuidValue is None: + PackageList =3D "\n\t".join(str(P) for P in self.Packa= geList) + EdkLogger.error( + 'build', + RESOURCE_NOT_AVAILABLE, + "Value of GUID [%s] is not found in" % Sku= .VariableGuid, + ExtraData=3DPackageList + "\n\t(used with = %s.%s from module %s)" \ + % (Guid, Name, str= (Module)), + File=3Dself.MetaFile + ) + + # override PCD settings with module specific setting + if Module in self.Platform.Modules: + PlatformModule =3D self.Platform.Modules[str(Module)] + for Key in PlatformModule.Pcds: + if self.BuildOptionPcd: + for pcd in self.BuildOptionPcd: + (TokenSpaceGuidCName, TokenCName, FieldName, pcdva= lue, _) =3D pcd + if (TokenCName, TokenSpaceGuidCName) =3D=3D Key an= d FieldName =3D=3D"": + PlatformModule.Pcds[Key].DefaultValue =3D pcdv= alue + PlatformModule.Pcds[Key].PcdValueFromComm =3D = pcdvalue + break + Flag =3D False + if Key in Pcds: + ToPcd =3D Pcds[Key] + Flag =3D True + elif Key in self.MixedPcd: + for PcdItem in self.MixedPcd[Key]: + if PcdItem in Pcds: + ToPcd =3D Pcds[PcdItem] + Flag =3D True + break + if Flag: + self._OverridePcd(ToPcd, PlatformModule.Pcds[Key], Mod= ule, Msg=3D"DSC Components Module scoped PCD section", Library=3DLibrary) + # use PCD value to calculate the MaxDatumSize when it is not speci= fied + for Name, Guid in Pcds: + Pcd =3D Pcds[Name, Guid] + if Pcd.DatumType =3D=3D TAB_VOID and not Pcd.MaxDatumSize: + Pcd.MaxSizeUserSet =3D None + Value =3D Pcd.DefaultValue + if not Value: + Pcd.MaxDatumSize =3D '1' + elif Value[0] =3D=3D 'L': + Pcd.MaxDatumSize =3D str((len(Value) - 2) * 2) + elif Value[0] =3D=3D '{': + Pcd.MaxDatumSize =3D str(len(Value.split(','))) + else: + Pcd.MaxDatumSize =3D str(len(Value) - 1) + return list(Pcds.values()) + + @cached_property + def Pcds(self): + PlatformPcdData =3D self.DataPipe.Get("PLA_PCD") +# for pcd in PlatformPcdData: +# for skuid in pcd.SkuInfoList: +# pcd.SkuInfoList[skuid] =3D self.CreateSkuInfoFromDict(pc= d.SkuInfoList[skuid]) + return {(pcddata.TokenCName,pcddata.TokenSpaceGuidCName):pcddata f= or pcddata in PlatformPcdData} + + def CreateSkuInfoFromDict(self,SkuInfoDict): + return SkuInfoClass( + SkuInfoDict.get("SkuIdName"), + SkuInfoDict.get("SkuId"), + SkuInfoDict.get("VariableName"), + SkuInfoDict.get("VariableGuid"), + SkuInfoDict.get("VariableOffset"), + SkuInfoDict.get("HiiDefaultValue"), + SkuInfoDict.get("VpdOffset"), + SkuInfoDict.get("DefaultValue"), + SkuInfoDict.get("VariableGuidValue"), + SkuInfoDict.get("VariableAttribute",""), + SkuInfoDict.get("DefaultStore",None) + ) + @cached_property + def MixedPcd(self): + return self.DataPipe.Get("MixedPcd") + @cached_property + def _GuidDict(self): + RetVal =3D self.DataPipe.Get("GuidDict") + if RetVal is None: + RetVal =3D {} + return RetVal + @cached_property + def BuildOptionPcd(self): + return self.DataPipe.Get("BuildOptPcd") + def ApplyBuildOption(self,module): + PlatformOptions =3D self.DataPipe.Get("PLA_BO") + ModuleBuildOptions =3D self.DataPipe.Get("MOL_BO") + ModuleOptionFromDsc =3D ModuleBuildOptions.get((module.MetaFile.Fi= le,module.MetaFile.Root)) + if ModuleOptionFromDsc: + ModuleTypeOptions, PlatformModuleOptions =3D ModuleOptionFromD= sc["ModuleTypeOptions"],ModuleOptionFromDsc["PlatformModuleOptions"] + else: + ModuleTypeOptions, PlatformModuleOptions =3D {}, {} + ToolDefinition =3D self.DataPipe.Get("TOOLDEF") + ModuleOptions =3D self._ExpandBuildOption(module.BuildOptions) + BuildRuleOrder =3D None + for Options in [ToolDefinition, ModuleOptions, PlatformOptions, Mo= duleTypeOptions, PlatformModuleOptions]: + for Tool in Options: + for Attr in Options[Tool]: + if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: + BuildRuleOrder =3D Options[Tool][Attr] + + AllTools =3D set(list(ModuleOptions.keys()) + list(PlatformOptions= .keys()) + + list(PlatformModuleOptions.keys()) + list(ModuleTyp= eOptions.keys()) + + list(ToolDefinition.keys())) + BuildOptions =3D defaultdict(lambda: defaultdict(str)) + for Tool in AllTools: + for Options in [ToolDefinition, ModuleOptions, PlatformOptions= , ModuleTypeOptions, PlatformModuleOptions]: + if Tool not in Options: + continue + for Attr in Options[Tool]: + # + # Do not generate it in Makefile + # + if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: + continue + Value =3D Options[Tool][Attr] + # check if override is indicated + if Value.startswith('=3D'): + BuildOptions[Tool][Attr] =3D mws.handleWsMacro(Val= ue[1:]) + else: + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + mws.handle= WsMacro(Value) + else: + BuildOptions[Tool][Attr] =3D mws.handleWsMacro= (Value) + + return BuildOptions, BuildRuleOrder + + def ApplyLibraryInstance(self,module): + alldeps =3D self.DataPipe.Get("DEPS") + if alldeps is None: + alldeps =3D {} + mod_libs =3D alldeps.get((module.MetaFile.File,module.MetaFile.Roo= t,module.Arch),[]) + retVal =3D [] + for (file_path,root,arch) in mod_libs: + retVal.append(self.Wa.BuildDatabase[PathClass(file_path,root),= arch, self.Target,self.ToolChain]) + return retVal + + ## Parse build_rule.txt in Conf Directory. + # + # @retval BuildRule object + # + @cached_property + def BuildRule(self): + WInfo =3D self.DataPipe.Get("P_Info") + RetVal =3D WInfo.get("BuildRuleFile") + if RetVal._FileVersion =3D=3D "": + RetVal._FileVersion =3D AutoGenReqBuildRuleVerNum + return RetVal diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools= /Source/Python/AutoGen/PlatformAutoGen.py new file mode 100644 index 000000000000..48cf6df85ac1 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py @@ -0,0 +1,1483 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +## Import Modules +# +from __future__ import print_function +from __future__ import absolute_import +import os.path as path +import copy +from collections import defaultdict + +from .BuildEngine import BuildRule,gDefaultBuildRuleFile,AutoGenReqBuildRu= leVerNum +from .GenVar import VariableMgr, var_info +from . import GenMake +from AutoGen.DataPipe import MemoryDataPipe +from AutoGen.ModuleAutoGen import ModuleAutoGen +from AutoGen.AutoGen import AutoGen +from AutoGen.AutoGen import CalculatePriorityValue +from Workspace.WorkspaceCommon import GetModuleLibInstances +from CommonDataClass.CommonClass import SkuInfoClass +from Common.caching import cached_class_function +from Common.Expression import ValueExpressionEx +from Common.StringUtils import StringToArray,NormPath +from Common.BuildToolError import * +from Common.DataType import * +from Common.Misc import * +import Common.VpdInfoFile as VpdInfoFile + +## Split command line option string to list +# +# subprocess.Popen needs the args to be a sequence. Otherwise there's prob= lem +# in non-windows platform to launch command +# +def _SplitOption(OptionString): + OptionList =3D [] + LastChar =3D " " + OptionStart =3D 0 + QuotationMark =3D "" + for Index in range(0, len(OptionString)): + CurrentChar =3D OptionString[Index] + if CurrentChar in ['"', "'"]: + if QuotationMark =3D=3D CurrentChar: + QuotationMark =3D "" + elif QuotationMark =3D=3D "": + QuotationMark =3D CurrentChar + continue + elif QuotationMark: + continue + + if CurrentChar in ["/", "-"] and LastChar in [" ", "\t", "\r", "\n= "]: + if Index > OptionStart: + OptionList.append(OptionString[OptionStart:Index - 1]) + OptionStart =3D Index + LastChar =3D CurrentChar + OptionList.append(OptionString[OptionStart:]) + return OptionList + +## AutoGen class for platform +# +# PlatformAutoGen class will process the original information in platform +# file in order to generate makefile for platform. +# +class PlatformAutoGen(AutoGen): + # call super().__init__ then call the worker function with different p= arameter count + def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): + if not hasattr(self, "_Init"): + self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch) + self._Init =3D True + # + # Used to store all PCDs for both PEI and DXE phase, in order to gener= ate + # correct PCD database + # + _DynaPcdList_ =3D [] + _NonDynaPcdList_ =3D [] + _PlatformPcds =3D {} + + + + ## Initialize PlatformAutoGen + # + # + # @param Workspace WorkspaceAutoGen object + # @param PlatformFile Platform file (DSC file) + # @param Target Build target (DEBUG, RELEASE) + # @param Toolchain Name of tool chain + # @param Arch arch of the platform supports + # + def _InitWorker(self, Workspace, PlatformFile, Target, Toolchain, Arch= ): + EdkLogger.debug(EdkLogger.DEBUG_9, "AutoGen platform [%s] [%s]" % = (PlatformFile, Arch)) + GlobalData.gProcessingFile =3D "%s [%s, %s, %s]" % (PlatformFile, = Arch, Toolchain, Target) + + self.MetaFile =3D PlatformFile + self.Workspace =3D Workspace + self.WorkspaceDir =3D Workspace.WorkspaceDir + self.ToolChain =3D Toolchain + self.BuildTarget =3D Target + self.Arch =3D Arch + self.SourceDir =3D PlatformFile.SubDir + self.FdTargetList =3D self.Workspace.FdTargetList + self.FvTargetList =3D self.Workspace.FvTargetList + # get the original module/package/platform objects + self.BuildDatabase =3D Workspace.BuildDatabase + self.DscBuildDataObj =3D Workspace.Platform + + # flag indicating if the makefile/C-code file has been created or = not + self.IsMakeFileCreated =3D False + + self._DynamicPcdList =3D None # [(TokenCName1, TokenSpaceGuidCN= ame1), (TokenCName2, TokenSpaceGuidCName2), ...] + self._NonDynamicPcdList =3D None # [(TokenCName1, TokenSpaceGuidCN= ame1), (TokenCName2, TokenSpaceGuidCName2), ...] + + self._AsBuildInfList =3D [] + self._AsBuildModuleList =3D [] + + self.VariableInfo =3D None + + if GlobalData.gFdfParser is not None: + self._AsBuildInfList =3D GlobalData.gFdfParser.Profile.InfList + for Inf in self._AsBuildInfList: + InfClass =3D PathClass(NormPath(Inf), GlobalData.gWorkspac= e, self.Arch) + M =3D self.BuildDatabase[InfClass, self.Arch, self.BuildTa= rget, self.ToolChain] + if not M.IsBinaryModule: + continue + self._AsBuildModuleList.append(InfClass) + # get library/modules for build + self.LibraryBuildDirectoryList =3D [] + self.ModuleBuildDirectoryList =3D [] + + self.DataPipe =3D MemoryDataPipe(self.BuildDir) + self.DataPipe.FillData(self) + + return True + + @cached_class_function + def __repr__(self): + return "%s [%s]" % (self.MetaFile, self.Arch) + + ## Create autogen code for platform and modules + # + # Since there's no autogen code for platform, this method will do not= hing + # if CreateModuleCodeFile is set to False. + # + # @param CreateModuleCodeFile Flag indicating if creating mo= dule's + # autogen code file or not + # + @cached_class_function + def CreateCodeFile(self, CreateModuleCodeFile=3DFalse): + # only module has code to be created, so do nothing if CreateModul= eCodeFile is False + if not CreateModuleCodeFile: + return + + for Ma in self.ModuleAutoGenList: + Ma.CreateCodeFile(True) + + ## Generate Fds Command + @cached_property + def GenFdsCommand(self): + return self.Workspace.GenFdsCommand + + ## Create makefile for the platform and modules in it + # + # @param CreateModuleMakeFile Flag indicating if the makefil= e for + # modules will be created as well + # + def CreateMakeFile(self, CreateModuleMakeFile=3DFalse, FfsCommand =3D = {}): + if CreateModuleMakeFile: + for Ma in self._MaList: + key =3D (Ma.MetaFile.File, self.Arch) + if key in FfsCommand: + Ma.CreateMakeFile(True, FfsCommand[key]) + else: + Ma.CreateMakeFile(True) + + # no need to create makefile for the platform more than once + if self.IsMakeFileCreated: + return + + # create library/module build dirs for platform + Makefile =3D GenMake.PlatformMakefile(self) + self.LibraryBuildDirectoryList =3D Makefile.GetLibraryBuildDirecto= ryList() + self.ModuleBuildDirectoryList =3D Makefile.GetModuleBuildDirectory= List() + + self.IsMakeFileCreated =3D True + + @property + def AllPcdList(self): + return self.DynamicPcdList + self.NonDynamicPcdList + ## Deal with Shared FixedAtBuild Pcds + # + def CollectFixedAtBuildPcds(self): + for LibAuto in self.LibraryAutoGenList: + FixedAtBuildPcds =3D {} + ShareFixedAtBuildPcdsSameValue =3D {} + for Module in LibAuto.ReferenceModules: + for Pcd in set(Module.FixedAtBuildPcds + LibAuto.FixedAtBu= ildPcds): + DefaultValue =3D Pcd.DefaultValue + # Cover the case: DSC component override the Pcd value= and the Pcd only used in one Lib + if Pcd in Module.LibraryPcdList: + Index =3D Module.LibraryPcdList.index(Pcd) + DefaultValue =3D Module.LibraryPcdList[Index].Defa= ultValue + key =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCN= ame)) + if key not in FixedAtBuildPcds: + ShareFixedAtBuildPcdsSameValue[key] =3D True + FixedAtBuildPcds[key] =3D DefaultValue + else: + if FixedAtBuildPcds[key] !=3D DefaultValue: + ShareFixedAtBuildPcdsSameValue[key] =3D False + for Pcd in LibAuto.FixedAtBuildPcds: + key =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCName)) + if (Pcd.TokenCName, Pcd.TokenSpaceGuidCName) not in self.N= onDynamicPcdDict: + continue + else: + DscPcd =3D self.NonDynamicPcdDict[(Pcd.TokenCName, Pcd= .TokenSpaceGuidCName)] + if DscPcd.Type !=3D TAB_PCDS_FIXED_AT_BUILD: + continue + if key in ShareFixedAtBuildPcdsSameValue and ShareFixedAtB= uildPcdsSameValue[key]: + LibAuto.ConstPcd[key] =3D FixedAtBuildPcds[key] + + def CollectVariables(self, DynamicPcdSet): + VpdRegionSize =3D 0 + VpdRegionBase =3D 0 + if self.Workspace.FdfFile: + FdDict =3D self.Workspace.FdfProfile.FdDict[GlobalData.gFdfPar= ser.CurrentFdName] + for FdRegion in FdDict.RegionList: + for item in FdRegion.RegionDataList: + if self.Platform.VpdToolGuid.strip() and self.Platform= .VpdToolGuid in item: + VpdRegionSize =3D FdRegion.Size + VpdRegionBase =3D FdRegion.Offset + break + + VariableInfo =3D VariableMgr(self.DscBuildDataObj._GetDefaultStore= s(), self.DscBuildDataObj.SkuIds) + VariableInfo.SetVpdRegionMaxSize(VpdRegionSize) + VariableInfo.SetVpdRegionOffset(VpdRegionBase) + Index =3D 0 + for Pcd in DynamicPcdSet: + pcdname =3D ".".join((Pcd.TokenSpaceGuidCName, Pcd.TokenCName)) + for SkuName in Pcd.SkuInfoList: + Sku =3D Pcd.SkuInfoList[SkuName] + SkuId =3D Sku.SkuId + if SkuId is None or SkuId =3D=3D '': + continue + if len(Sku.VariableName) > 0: + if Sku.VariableAttribute and 'NV' not in Sku.VariableA= ttribute: + continue + VariableGuidStructure =3D Sku.VariableGuidValue + VariableGuid =3D GuidStructureStringToGuidString(Varia= bleGuidStructure) + for StorageName in Sku.DefaultStoreDict: + VariableInfo.append_variable(var_info(Index, pcdna= me, StorageName, SkuName, StringToArray(Sku.VariableName), VariableGuid, Sk= u.VariableOffset, Sku.VariableAttribute, Sku.HiiDefaultValue, Sku.DefaultSt= oreDict[StorageName] if Pcd.DatumType in TAB_PCD_NUMERIC_TYPES else StringT= oArray(Sku.DefaultStoreDict[StorageName]), Pcd.DatumType, Pcd.CustomAttribu= te['DscPosition'], Pcd.CustomAttribute.get('IsStru',False))) + Index +=3D 1 + return VariableInfo + + def UpdateNVStoreMaxSize(self, OrgVpdFile): + if self.VariableInfo: + VpdMapFilePath =3D os.path.join(self.BuildDir, TAB_FV_DIRECTOR= Y, "%s.map" % self.Platform.VpdToolGuid) + PcdNvStoreDfBuffer =3D [item for item in self._DynamicPcdList = if item.TokenCName =3D=3D "PcdNvStoreDefaultValueBuffer" and item.TokenSpac= eGuidCName =3D=3D "gEfiMdeModulePkgTokenSpaceGuid"] + + if PcdNvStoreDfBuffer: + if os.path.exists(VpdMapFilePath): + OrgVpdFile.Read(VpdMapFilePath) + PcdItems =3D OrgVpdFile.GetOffset(PcdNvStoreDfBuffer[0= ]) + NvStoreOffset =3D list(PcdItems.values())[0].strip() i= f PcdItems else '0' + else: + EdkLogger.error("build", FILE_READ_FAILURE, "Can not f= ind VPD map file %s to fix up VPD offset." % VpdMapFilePath) + + NvStoreOffset =3D int(NvStoreOffset, 16) if NvStoreOffset.= upper().startswith("0X") else int(NvStoreOffset) + default_skuobj =3D PcdNvStoreDfBuffer[0].SkuInfoList.get(T= AB_DEFAULT) + maxsize =3D self.VariableInfo.VpdRegionSize - NvStoreOffs= et if self.VariableInfo.VpdRegionSize else len(default_skuobj.DefaultValue.= split(",")) + var_data =3D self.VariableInfo.PatchNVStoreDefaultMaxSize(= maxsize) + + if var_data and default_skuobj: + default_skuobj.DefaultValue =3D var_data + PcdNvStoreDfBuffer[0].DefaultValue =3D var_data + PcdNvStoreDfBuffer[0].SkuInfoList.clear() + PcdNvStoreDfBuffer[0].SkuInfoList[TAB_DEFAULT] =3D def= ault_skuobj + PcdNvStoreDfBuffer[0].MaxDatumSize =3D str(len(default= _skuobj.DefaultValue.split(","))) + + return OrgVpdFile + + ## Collect dynamic PCDs + # + # Gather dynamic PCDs list from each module and their settings from p= latform + # This interface should be invoked explicitly when platform action is= created. + # + def CollectPlatformDynamicPcds(self): + self.CategoryPcds() + self.SortDynamicPcd() + + def CategoryPcds(self): + # Category Pcds into DynamicPcds and NonDynamicPcds + # for gathering error information + NoDatumTypePcdList =3D set() + FdfModuleList =3D [] + for InfName in self._AsBuildInfList: + InfName =3D mws.join(self.WorkspaceDir, InfName) + FdfModuleList.append(os.path.normpath(InfName)) + for M in self._MbList: +# F is the Module for which M is the module autogen + ModPcdList =3D self.ApplyPcdSetting(M, M.ModulePcdList) + LibPcdList =3D [] + for lib in M.LibraryPcdList: + LibPcdList.extend(self.ApplyPcdSetting(M, M.LibraryPcdList= [lib], lib)) + for PcdFromModule in ModPcdList + LibPcdList: + + # make sure that the "VOID*" kind of datum has MaxDatumSiz= e set + if PcdFromModule.DatumType =3D=3D TAB_VOID and not PcdFrom= Module.MaxDatumSize: + NoDatumTypePcdList.add("%s.%s [%s]" % (PcdFromModule.T= okenSpaceGuidCName, PcdFromModule.TokenCName, M.MetaFile)) + + # Check the PCD from Binary INF or Source INF + if M.IsBinaryModule =3D=3D True: + PcdFromModule.IsFromBinaryInf =3D True + + # Check the PCD from DSC or not + PcdFromModule.IsFromDsc =3D (PcdFromModule.TokenCName, Pcd= FromModule.TokenSpaceGuidCName) in self.Platform.Pcds + + if PcdFromModule.Type in PCD_DYNAMIC_TYPE_SET or PcdFromMo= dule.Type in PCD_DYNAMIC_EX_TYPE_SET: + if M.MetaFile.Path not in FdfModuleList: + # If one of the Source built modules listed in the= DSC is not listed + # in FDF modules, and the INF lists a PCD can only= use the PcdsDynamic + # access method (it is only listed in the DEC file= that declares the + # PCD as PcdsDynamic), then build tool will report= warning message + # notify the PI that they are attempting to build = a module that must + # be included in a flash image in order to be func= tional. These Dynamic + # PCD will not be added into the Database unless i= t is used by other + # modules that are included in the FDF file. + if PcdFromModule.Type in PCD_DYNAMIC_TYPE_SET and \ + PcdFromModule.IsFromBinaryInf =3D=3D False: + # Print warning message to let the developer m= ake a determine. + continue + # If one of the Source built modules listed in the= DSC is not listed in + # FDF modules, and the INF lists a PCD can only us= e the PcdsDynamicEx + # access method (it is only listed in the DEC file= that declares the + # PCD as PcdsDynamicEx), then DO NOT break the bui= ld; DO NOT add the + # PCD to the Platform's PCD Database. + if PcdFromModule.Type in PCD_DYNAMIC_EX_TYPE_SET: + continue + # + # If a dynamic PCD used by a PEM module/PEI module & D= XE module, + # it should be stored in Pcd PEI database, If a dynami= c only + # used by DXE module, it should be stored in DXE PCD d= atabase. + # The default Phase is DXE + # + if M.ModuleType in SUP_MODULE_SET_PEI: + PcdFromModule.Phase =3D "PEI" + if PcdFromModule not in self._DynaPcdList_: + self._DynaPcdList_.append(PcdFromModule) + elif PcdFromModule.Phase =3D=3D 'PEI': + # overwrite any the same PCD existing, if Phase is= PEI + Index =3D self._DynaPcdList_.index(PcdFromModule) + self._DynaPcdList_[Index] =3D PcdFromModule + elif PcdFromModule not in self._NonDynaPcdList_: + self._NonDynaPcdList_.append(PcdFromModule) + elif PcdFromModule in self._NonDynaPcdList_ and PcdFromMod= ule.IsFromBinaryInf =3D=3D True: + Index =3D self._NonDynaPcdList_.index(PcdFromModule) + if self._NonDynaPcdList_[Index].IsFromBinaryInf =3D=3D= False: + #The PCD from Binary INF will override the same on= e from source INF + self._NonDynaPcdList_.remove (self._NonDynaPcdList= _[Index]) + PcdFromModule.Pending =3D False + self._NonDynaPcdList_.append (PcdFromModule) + DscModuleSet =3D {os.path.normpath(ModuleInf.Path) for ModuleInf i= n self.Platform.Modules} + # add the PCD from modules that listed in FDF but not in DSC to Da= tabase + for InfName in FdfModuleList: + if InfName not in DscModuleSet: + InfClass =3D PathClass(InfName) + M =3D self.BuildDatabase[InfClass, self.Arch, self.BuildTa= rget, self.ToolChain] + # If a module INF in FDF but not in current arch's DSC mod= ule list, it must be module (either binary or source) + # for different Arch. PCDs in source module for different = Arch is already added before, so skip the source module here. + # For binary module, if in current arch, we need to list t= he PCDs into database. + if not M.IsBinaryModule: + continue + # Override the module PCD setting by platform setting + ModulePcdList =3D self.ApplyPcdSetting(M, M.Pcds) + for PcdFromModule in ModulePcdList: + PcdFromModule.IsFromBinaryInf =3D True + PcdFromModule.IsFromDsc =3D False + # Only allow the DynamicEx and Patchable PCD in AsBuil= d INF + if PcdFromModule.Type not in PCD_DYNAMIC_EX_TYPE_SET a= nd PcdFromModule.Type not in TAB_PCDS_PATCHABLE_IN_MODULE: + EdkLogger.error("build", AUTOGEN_ERROR, "PCD setti= ng error", + File=3Dself.MetaFile, + ExtraData=3D"\n\tExisted %s PCD %s= in:\n\t\t%s\n" + % (PcdFromModule.Type, PcdFromModu= le.TokenCName, InfName)) + # make sure that the "VOID*" kind of datum has MaxDatu= mSize set + if PcdFromModule.DatumType =3D=3D TAB_VOID and not Pcd= FromModule.MaxDatumSize: + NoDatumTypePcdList.add("%s.%s [%s]" % (PcdFromModu= le.TokenSpaceGuidCName, PcdFromModule.TokenCName, InfName)) + if M.ModuleType in SUP_MODULE_SET_PEI: + PcdFromModule.Phase =3D "PEI" + if PcdFromModule not in self._DynaPcdList_ and PcdFrom= Module.Type in PCD_DYNAMIC_EX_TYPE_SET: + self._DynaPcdList_.append(PcdFromModule) + elif PcdFromModule not in self._NonDynaPcdList_ and Pc= dFromModule.Type in TAB_PCDS_PATCHABLE_IN_MODULE: + self._NonDynaPcdList_.append(PcdFromModule) + if PcdFromModule in self._DynaPcdList_ and PcdFromModu= le.Phase =3D=3D 'PEI' and PcdFromModule.Type in PCD_DYNAMIC_EX_TYPE_SET: + # Overwrite the phase of any the same PCD existing= , if Phase is PEI. + # It is to solve the case that a dynamic PCD used = by a PEM module/PEI + # module & DXE module at a same time. + # Overwrite the type of the PCDs in source INF by = the type of AsBuild + # INF file as DynamicEx. + Index =3D self._DynaPcdList_.index(PcdFromModule) + self._DynaPcdList_[Index].Phase =3D PcdFromModule.= Phase + self._DynaPcdList_[Index].Type =3D PcdFromModule.T= ype + for PcdFromModule in self._NonDynaPcdList_: + # If a PCD is not listed in the DSC file, but binary INF files= used by + # this platform all (that use this PCD) list the PCD in a [Pat= chPcds] + # section, AND all source INF files used by this platform the = build + # that use the PCD list the PCD in either a [Pcds] or [PatchPc= ds] + # section, then the tools must NOT add the PCD to the Platform= 's PCD + # Database; the build must assign the access method for this P= CD as + # PcdsPatchableInModule. + if PcdFromModule not in self._DynaPcdList_: + continue + Index =3D self._DynaPcdList_.index(PcdFromModule) + if PcdFromModule.IsFromDsc =3D=3D False and \ + PcdFromModule.Type in TAB_PCDS_PATCHABLE_IN_MODULE and \ + PcdFromModule.IsFromBinaryInf =3D=3D True and \ + self._DynaPcdList_[Index].IsFromBinaryInf =3D=3D False: + Index =3D self._DynaPcdList_.index(PcdFromModule) + self._DynaPcdList_.remove (self._DynaPcdList_[Index]) + + # print out error information and break the build, if error found + if len(NoDatumTypePcdList) > 0: + NoDatumTypePcdListString =3D "\n\t\t".join(NoDatumTypePcdList) + EdkLogger.error("build", AUTOGEN_ERROR, "PCD setting error", + File=3Dself.MetaFile, + ExtraData=3D"\n\tPCD(s) without MaxDatumSize:\= n\t\t%s\n" + % NoDatumTypePcdListString) + self._NonDynamicPcdList =3D self._NonDynaPcdList_ + self._DynamicPcdList =3D self._DynaPcdList_ + + def SortDynamicPcd(self): + # + # Sort dynamic PCD list to: + # 1) If PCD's datum type is VOID* and value is unicode string whic= h starts with L, the PCD item should + # try to be put header of dynamicd List + # 2) If PCD is HII type, the PCD item should be put after unicode = type PCD + # + # The reason of sorting is make sure the unicode string is in doub= le-byte alignment in string table. + # + UnicodePcdArray =3D set() + HiiPcdArray =3D set() + OtherPcdArray =3D set() + VpdPcdDict =3D {} + VpdFile =3D VpdInfoFile.VpdInfoFile() + NeedProcessVpdMapFile =3D False + + for pcd in self.Platform.Pcds: + if pcd not in self._PlatformPcds: + self._PlatformPcds[pcd] =3D self.Platform.Pcds[pcd] + + for item in self._PlatformPcds: + if self._PlatformPcds[item].DatumType and self._PlatformPcds[i= tem].DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32, TAB_UINT64, TAB_V= OID, "BOOLEAN"]: + self._PlatformPcds[item].DatumType =3D TAB_VOID + + if (self.Workspace.ArchList[-1] =3D=3D self.Arch): + for Pcd in self._DynamicPcdList: + # just pick the a value to determine whether is unicode st= ring type + Sku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) + Sku.VpdOffset =3D Sku.VpdOffset.strip() + + if Pcd.DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32= , TAB_UINT64, TAB_VOID, "BOOLEAN"]: + Pcd.DatumType =3D TAB_VOID + + # if found PCD which datum value is unicode string the= insert to left size of UnicodeIndex + # if found HII type PCD then insert to right of Unicod= eIndex + if Pcd.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYNAMIC_EX_= VPD]: + VpdPcdDict[(Pcd.TokenCName, Pcd.TokenSpaceGuidCName)] = =3D Pcd + + #Collect DynamicHii PCD values and assign it to DynamicExVpd P= CD gEfiMdeModulePkgTokenSpaceGuid.PcdNvStoreDefaultValueBuffer + PcdNvStoreDfBuffer =3D VpdPcdDict.get(("PcdNvStoreDefaultValue= Buffer", "gEfiMdeModulePkgTokenSpaceGuid")) + if PcdNvStoreDfBuffer: + self.VariableInfo =3D self.CollectVariables(self._DynamicP= cdList) + vardump =3D self.VariableInfo.dump() + if vardump: + # + #According to PCD_DATABASE_INIT in edk2\MdeModulePkg\I= nclude\Guid\PcdDataBaseSignatureGuid.h, + #the max size for string PCD should not exceed USHRT_M= AX 65535(0xffff). + #typedef UINT16 SIZE_INFO; + #//SIZE_INFO SizeTable[]; + if len(vardump.split(",")) > 0xffff: + EdkLogger.error("build", RESOURCE_OVERFLOW, 'The c= urrent length of PCD %s value is %d, it exceeds to the max size of String P= CD.' %(".".join([PcdNvStoreDfBuffer.TokenSpaceGuidCName,PcdNvStoreDfBuffer.= TokenCName]) ,len(vardump.split(",")))) + PcdNvStoreDfBuffer.DefaultValue =3D vardump + for skuname in PcdNvStoreDfBuffer.SkuInfoList: + PcdNvStoreDfBuffer.SkuInfoList[skuname].DefaultVal= ue =3D vardump + PcdNvStoreDfBuffer.MaxDatumSize =3D str(len(vardum= p.split(","))) + else: + #If the end user define [DefaultStores] and [XXX.Menufactu= ring] in DSC, but forget to configure PcdNvStoreDefaultValueBuffer to PcdsD= ynamicVpd + if [Pcd for Pcd in self._DynamicPcdList if Pcd.UserDefined= DefaultStoresFlag]: + EdkLogger.warn("build", "PcdNvStoreDefaultValueBuffer = should be defined as PcdsDynamicExVpd in dsc file since the DefaultStores i= s enabled for this platform.\n%s" %self.Platform.MetaFile.Path) + PlatformPcds =3D sorted(self._PlatformPcds.keys()) + # + # Add VPD type PCD into VpdFile and determine whether the VPD = PCD need to be fixed up. + # + VpdSkuMap =3D {} + for PcdKey in PlatformPcds: + Pcd =3D self._PlatformPcds[PcdKey] + if Pcd.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYNAMIC_EX_= VPD] and \ + PcdKey in VpdPcdDict: + Pcd =3D VpdPcdDict[PcdKey] + SkuValueMap =3D {} + DefaultSku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) + if DefaultSku: + PcdValue =3D DefaultSku.DefaultValue + if PcdValue not in SkuValueMap: + SkuValueMap[PcdValue] =3D [] + VpdFile.Add(Pcd, TAB_DEFAULT, DefaultSku.VpdOf= fset) + SkuValueMap[PcdValue].append(DefaultSku) + + for (SkuName, Sku) in Pcd.SkuInfoList.items(): + Sku.VpdOffset =3D Sku.VpdOffset.strip() + PcdValue =3D Sku.DefaultValue + if PcdValue =3D=3D "": + PcdValue =3D Pcd.DefaultValue + if Sku.VpdOffset !=3D TAB_STAR: + if PcdValue.startswith("{"): + Alignment =3D 8 + elif PcdValue.startswith("L"): + Alignment =3D 2 + else: + Alignment =3D 1 + try: + VpdOffset =3D int(Sku.VpdOffset) + except: + try: + VpdOffset =3D int(Sku.VpdOffset, 16) + except: + EdkLogger.error("build", FORMAT_INVALI= D, "Invalid offset value %s for PCD %s.%s." % (Sku.VpdOffset, Pcd.TokenSpac= eGuidCName, Pcd.TokenCName)) + if VpdOffset % Alignment !=3D 0: + if PcdValue.startswith("{"): + EdkLogger.warn("build", "The offset va= lue of PCD %s.%s is not 8-byte aligned!" %(Pcd.TokenSpaceGuidCName, Pcd.Tok= enCName), File=3Dself.MetaFile) + else: + EdkLogger.error("build", FORMAT_INVALI= D, 'The offset value of PCD %s.%s should be %s-byte aligned.' % (Pcd.TokenS= paceGuidCName, Pcd.TokenCName, Alignment)) + if PcdValue not in SkuValueMap: + SkuValueMap[PcdValue] =3D [] + VpdFile.Add(Pcd, SkuName, Sku.VpdOffset) + SkuValueMap[PcdValue].append(Sku) + # if the offset of a VPD is *, then it need to be = fixed up by third party tool. + if not NeedProcessVpdMapFile and Sku.VpdOffset =3D= =3D TAB_STAR: + NeedProcessVpdMapFile =3D True + if self.Platform.VpdToolGuid is None or self.P= latform.VpdToolGuid =3D=3D '': + EdkLogger.error("Build", FILE_NOT_FOUND, \ + "Fail to find third-party = BPDG tool to process VPD PCDs. BPDG Guid tool need to be defined in tools_d= ef.txt and VPD_TOOL_GUID need to be provided in DSC file.") + + VpdSkuMap[PcdKey] =3D SkuValueMap + # + # Fix the PCDs define in VPD PCD section that never referenced= by module. + # An example is PCD for signature usage. + # + for DscPcd in PlatformPcds: + DscPcdEntry =3D self._PlatformPcds[DscPcd] + if DscPcdEntry.Type in [TAB_PCDS_DYNAMIC_VPD, TAB_PCDS_DYN= AMIC_EX_VPD]: + if not (self.Platform.VpdToolGuid is None or self.Plat= form.VpdToolGuid =3D=3D ''): + FoundFlag =3D False + for VpdPcd in VpdFile._VpdArray: + # This PCD has been referenced by module + if (VpdPcd.TokenSpaceGuidCName =3D=3D DscPcdEn= try.TokenSpaceGuidCName) and \ + (VpdPcd.TokenCName =3D=3D DscPcdEntry.Token= CName): + FoundFlag =3D True + + # Not found, it should be signature + if not FoundFlag : + # just pick the a value to determine whether i= s unicode string type + SkuValueMap =3D {} + SkuObjList =3D list(DscPcdEntry.SkuInfoList.it= ems()) + DefaultSku =3D DscPcdEntry.SkuInfoList.get(TAB= _DEFAULT) + if DefaultSku: + defaultindex =3D SkuObjList.index((TAB_DEF= AULT, DefaultSku)) + SkuObjList[0], SkuObjList[defaultindex] = =3D SkuObjList[defaultindex], SkuObjList[0] + for (SkuName, Sku) in SkuObjList: + Sku.VpdOffset =3D Sku.VpdOffset.strip() + + # Need to iterate DEC pcd information to g= et the value & datumtype + for eachDec in self.PackageList: + for DecPcd in eachDec.Pcds: + DecPcdEntry =3D eachDec.Pcds[DecPc= d] + if (DecPcdEntry.TokenSpaceGuidCNam= e =3D=3D DscPcdEntry.TokenSpaceGuidCName) and \ + (DecPcdEntry.TokenCName =3D=3D = DscPcdEntry.TokenCName): + # Print warning message to let= the developer make a determine. + EdkLogger.warn("build", "Unref= erenced vpd pcd used!", + File=3Dself.Me= taFile, \ + ExtraData =3D = "PCD: %s.%s used in the DSC file %s is unreferenced." \ + %(DscPcdEntry.= TokenSpaceGuidCName, DscPcdEntry.TokenCName, self.Platform.MetaFile.Path)) + + DscPcdEntry.DatumType =3D D= ecPcdEntry.DatumType + DscPcdEntry.DefaultValue =3D D= ecPcdEntry.DefaultValue + DscPcdEntry.TokenValue =3D Dec= PcdEntry.TokenValue + DscPcdEntry.TokenSpaceGuidValu= e =3D eachDec.Guids[DecPcdEntry.TokenSpaceGuidCName] + # Only fix the value while no = value provided in DSC file. + if not Sku.DefaultValue: + DscPcdEntry.SkuInfoList[li= st(DscPcdEntry.SkuInfoList.keys())[0]].DefaultValue =3D DecPcdEntry.Default= Value + + if DscPcdEntry not in self._DynamicPcdList: + self._DynamicPcdList.append(DscPcdEntr= y) + Sku.VpdOffset =3D Sku.VpdOffset.strip() + PcdValue =3D Sku.DefaultValue + if PcdValue =3D=3D "": + PcdValue =3D DscPcdEntry.DefaultValue + if Sku.VpdOffset !=3D TAB_STAR: + if PcdValue.startswith("{"): + Alignment =3D 8 + elif PcdValue.startswith("L"): + Alignment =3D 2 + else: + Alignment =3D 1 + try: + VpdOffset =3D int(Sku.VpdOffset) + except: + try: + VpdOffset =3D int(Sku.VpdOffse= t, 16) + except: + EdkLogger.error("build", FORMA= T_INVALID, "Invalid offset value %s for PCD %s.%s." % (Sku.VpdOffset, DscPc= dEntry.TokenSpaceGuidCName, DscPcdEntry.TokenCName)) + if VpdOffset % Alignment !=3D 0: + if PcdValue.startswith("{"): + EdkLogger.warn("build", "The o= ffset value of PCD %s.%s is not 8-byte aligned!" %(DscPcdEntry.TokenSpaceGu= idCName, DscPcdEntry.TokenCName), File=3Dself.MetaFile) + else: + EdkLogger.error("build", FORMA= T_INVALID, 'The offset value of PCD %s.%s should be %s-byte aligned.' % (Ds= cPcdEntry.TokenSpaceGuidCName, DscPcdEntry.TokenCName, Alignment)) + if PcdValue not in SkuValueMap: + SkuValueMap[PcdValue] =3D [] + VpdFile.Add(DscPcdEntry, SkuName, Sku.= VpdOffset) + SkuValueMap[PcdValue].append(Sku) + if not NeedProcessVpdMapFile and Sku.VpdOf= fset =3D=3D TAB_STAR: + NeedProcessVpdMapFile =3D True + if DscPcdEntry.DatumType =3D=3D TAB_VOID and P= cdValue.startswith("L"): + UnicodePcdArray.add(DscPcdEntry) + elif len(Sku.VariableName) > 0: + HiiPcdArray.add(DscPcdEntry) + else: + OtherPcdArray.add(DscPcdEntry) + + # if the offset of a VPD is *, then it nee= d to be fixed up by third party tool. + VpdSkuMap[DscPcd] =3D SkuValueMap + if (self.Platform.FlashDefinition is None or self.Platform.Fla= shDefinition =3D=3D '') and \ + VpdFile.GetCount() !=3D 0: + EdkLogger.error("build", ATTRIBUTE_NOT_AVAILABLE, + "Fail to get FLASH_DEFINITION definition i= n DSC file %s which is required when DSC contains VPD PCD." % str(self.Plat= form.MetaFile)) + + if VpdFile.GetCount() !=3D 0: + + self.FixVpdOffset(VpdFile) + + self.FixVpdOffset(self.UpdateNVStoreMaxSize(VpdFile)) + PcdNvStoreDfBuffer =3D [item for item in self._DynamicPcdL= ist if item.TokenCName =3D=3D "PcdNvStoreDefaultValueBuffer" and item.Token= SpaceGuidCName =3D=3D "gEfiMdeModulePkgTokenSpaceGuid"] + if PcdNvStoreDfBuffer: + PcdName,PcdGuid =3D PcdNvStoreDfBuffer[0].TokenCName, = PcdNvStoreDfBuffer[0].TokenSpaceGuidCName + if (PcdName,PcdGuid) in VpdSkuMap: + DefaultSku =3D PcdNvStoreDfBuffer[0].SkuInfoList.g= et(TAB_DEFAULT) + VpdSkuMap[(PcdName,PcdGuid)] =3D {DefaultSku.Defau= ltValue:[SkuObj for SkuObj in PcdNvStoreDfBuffer[0].SkuInfoList.values() ]} + + # Process VPD map file generated by third party BPDG tool + if NeedProcessVpdMapFile: + VpdMapFilePath =3D os.path.join(self.BuildDir, TAB_FV_= DIRECTORY, "%s.map" % self.Platform.VpdToolGuid) + if os.path.exists(VpdMapFilePath): + VpdFile.Read(VpdMapFilePath) + + # Fixup TAB_STAR offset + for pcd in VpdSkuMap: + vpdinfo =3D VpdFile.GetVpdInfo(pcd) + if vpdinfo is None: + # just pick the a value to determine whether i= s unicode string type + continue + for pcdvalue in VpdSkuMap[pcd]: + for sku in VpdSkuMap[pcd][pcdvalue]: + for item in vpdinfo: + if item[2] =3D=3D pcdvalue: + sku.VpdOffset =3D item[1] + else: + EdkLogger.error("build", FILE_READ_FAILURE, "Can n= ot find VPD map file %s to fix up VPD offset." % VpdMapFilePath) + + # Delete the DynamicPcdList At the last time enter into this f= unction + for Pcd in self._DynamicPcdList: + # just pick the a value to determine whether is unicode st= ring type + Sku =3D Pcd.SkuInfoList.get(TAB_DEFAULT) + Sku.VpdOffset =3D Sku.VpdOffset.strip() + + if Pcd.DatumType not in [TAB_UINT8, TAB_UINT16, TAB_UINT32= , TAB_UINT64, TAB_VOID, "BOOLEAN"]: + Pcd.DatumType =3D TAB_VOID + + PcdValue =3D Sku.DefaultValue + if Pcd.DatumType =3D=3D TAB_VOID and PcdValue.startswith("= L"): + # if found PCD which datum value is unicode string the= insert to left size of UnicodeIndex + UnicodePcdArray.add(Pcd) + elif len(Sku.VariableName) > 0: + # if found HII type PCD then insert to right of Unicod= eIndex + HiiPcdArray.add(Pcd) + else: + OtherPcdArray.add(Pcd) + del self._DynamicPcdList[:] + self._DynamicPcdList.extend(list(UnicodePcdArray)) + self._DynamicPcdList.extend(list(HiiPcdArray)) + self._DynamicPcdList.extend(list(OtherPcdArray)) + allskuset =3D [(SkuName, Sku.SkuId) for pcd in self._DynamicPcdLis= t for (SkuName, Sku) in pcd.SkuInfoList.items()] + for pcd in self._DynamicPcdList: + if len(pcd.SkuInfoList) =3D=3D 1: + for (SkuName, SkuId) in allskuset: + if isinstance(SkuId, str) and eval(SkuId) =3D=3D 0 or = SkuId =3D=3D 0: + continue + pcd.SkuInfoList[SkuName] =3D copy.deepcopy(pcd.SkuInfo= List[TAB_DEFAULT]) + pcd.SkuInfoList[SkuName].SkuId =3D SkuId + pcd.SkuInfoList[SkuName].SkuIdName =3D SkuName + + def FixVpdOffset(self, VpdFile ): + FvPath =3D os.path.join(self.BuildDir, TAB_FV_DIRECTORY) + if not os.path.exists(FvPath): + try: + os.makedirs(FvPath) + except: + EdkLogger.error("build", FILE_WRITE_FAILURE, "Fail to crea= te FV folder under %s" % self.BuildDir) + + VpdFilePath =3D os.path.join(FvPath, "%s.txt" % self.Platform.VpdT= oolGuid) + + if VpdFile.Write(VpdFilePath): + # retrieve BPDG tool's path from tool_def.txt according to VPD= _TOOL_GUID defined in DSC file. + BPDGToolName =3D None + for ToolDef in self.ToolDefinition.values(): + if TAB_GUID in ToolDef and ToolDef[TAB_GUID] =3D=3D self.P= latform.VpdToolGuid: + if "PATH" not in ToolDef: + EdkLogger.error("build", ATTRIBUTE_NOT_AVAILABLE, = "PATH attribute was not provided for BPDG guid tool %s in tools_def.txt" % = self.Platform.VpdToolGuid) + BPDGToolName =3D ToolDef["PATH"] + break + # Call third party GUID BPDG tool. + if BPDGToolName is not None: + VpdInfoFile.CallExtenalBPDGTool(BPDGToolName, VpdFilePath) + else: + EdkLogger.error("Build", FILE_NOT_FOUND, "Fail to find thi= rd-party BPDG tool to process VPD PCDs. BPDG Guid tool need to be defined i= n tools_def.txt and VPD_TOOL_GUID need to be provided in DSC file.") + + ## Return the platform build data object + @cached_property + def Platform(self): + return self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarg= et, self.ToolChain] + + ## Return platform name + @cached_property + def Name(self): + return self.Platform.PlatformName + + ## Return the meta file GUID + @cached_property + def Guid(self): + return self.Platform.Guid + + ## Return the platform version + @cached_property + def Version(self): + return self.Platform.Version + + ## Return the FDF file name + @cached_property + def FdfFile(self): + if self.Workspace.FdfFile: + RetVal=3D mws.join(self.WorkspaceDir, self.Workspace.FdfFile) + else: + RetVal =3D '' + return RetVal + + ## Return the build output directory platform specifies + @cached_property + def OutputDir(self): + return self.Platform.OutputDirectory + + ## Return the directory to store all intermediate and final files built + @cached_property + def BuildDir(self): + if os.path.isabs(self.OutputDir): + GlobalData.gBuildDirectory =3D RetVal =3D path.join( + path.abspath(self.OutputDir), + self.BuildTarget + "_" + self.Tool= Chain, + ) + else: + GlobalData.gBuildDirectory =3D RetVal =3D path.join( + self.WorkspaceDir, + self.OutputDir, + self.BuildTarget + "_" + self.Tool= Chain, + ) + return RetVal + + ## Return directory of platform makefile + # + # @retval string Makefile directory + # + @cached_property + def MakeFileDir(self): + return path.join(self.BuildDir, self.Arch) + + ## Return build command string + # + # @retval string Build command string + # + @cached_property + def BuildCommand(self): + RetVal =3D [] + if "MAKE" in self.ToolDefinition and "PATH" in self.ToolDefinition= ["MAKE"]: + RetVal +=3D _SplitOption(self.ToolDefinition["MAKE"]["PATH"]) + if "FLAGS" in self.ToolDefinition["MAKE"]: + NewOption =3D self.ToolDefinition["MAKE"]["FLAGS"].strip() + if NewOption !=3D '': + RetVal +=3D _SplitOption(NewOption) + if "MAKE" in self.EdkIIBuildOption: + if "FLAGS" in self.EdkIIBuildOption["MAKE"]: + Flags =3D self.EdkIIBuildOption["MAKE"]["FLAGS"] + if Flags.startswith('=3D'): + RetVal =3D [RetVal[0]] + [Flags[1:]] + else: + RetVal.append(Flags) + return RetVal + + ## Get tool chain definition + # + # Get each tool definition for given tool chain from tools_def.txt an= d platform + # + @cached_property + def ToolDefinition(self): + ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDictionary + if TAB_TOD_DEFINES_COMMAND_TYPE not in self.Workspace.ToolDef.Tool= sDefTxtDatabase: + EdkLogger.error('build', RESOURCE_NOT_AVAILABLE, "No tools fou= nd in configuration", + ExtraData=3D"[%s]" % self.MetaFile) + RetVal =3D {} + DllPathList =3D set() + for Def in ToolDefinition: + Target, Tag, Arch, Tool, Attr =3D Def.split("_") + if Target !=3D self.BuildTarget or Tag !=3D self.ToolChain or = Arch !=3D self.Arch: + continue + + Value =3D ToolDefinition[Def] + # don't record the DLL + if Attr =3D=3D "DLL": + DllPathList.add(Value) + continue + + if Tool not in RetVal: + RetVal[Tool] =3D {} + RetVal[Tool][Attr] =3D Value + + ToolsDef =3D '' + if GlobalData.gOptions.SilentMode and "MAKE" in RetVal: + if "FLAGS" not in RetVal["MAKE"]: + RetVal["MAKE"]["FLAGS"] =3D "" + RetVal["MAKE"]["FLAGS"] +=3D " -s" + MakeFlags =3D '' + for Tool in RetVal: + for Attr in RetVal[Tool]: + Value =3D RetVal[Tool][Attr] + if Tool in self._BuildOptionWithToolDef(RetVal) and Attr i= n self._BuildOptionWithToolDef(RetVal)[Tool]: + # check if override is indicated + if self._BuildOptionWithToolDef(RetVal)[Tool][Attr].st= artswith('=3D'): + Value =3D self._BuildOptionWithToolDef(RetVal)[Too= l][Attr][1:] + else: + if Attr !=3D 'PATH': + Value +=3D " " + self._BuildOptionWithToolDef(= RetVal)[Tool][Attr] + else: + Value =3D self._BuildOptionWithToolDef(RetVal)= [Tool][Attr] + + if Attr =3D=3D "PATH": + # Don't put MAKE definition in the file + if Tool !=3D "MAKE": + ToolsDef +=3D "%s =3D %s\n" % (Tool, Value) + elif Attr !=3D "DLL": + # Don't put MAKE definition in the file + if Tool =3D=3D "MAKE": + if Attr =3D=3D "FLAGS": + MakeFlags =3D Value + else: + ToolsDef +=3D "%s_%s =3D %s\n" % (Tool, Attr, Valu= e) + ToolsDef +=3D "\n" + + tool_def_file =3D os.path.join(self.MakeFileDir, "TOOLS_DEF." + se= lf.Arch) + SaveFileOnChange(tool_def_file, ToolsDef, False) + for DllPath in DllPathList: + os.environ["PATH"] =3D DllPath + os.pathsep + os.environ["PATH= "] + os.environ["MAKE_FLAGS"] =3D MakeFlags + + return RetVal + + ## Return the paths of tools + @cached_property + def ToolDefinitionFile(self): + tool_def_file =3D os.path.join(self.MakeFileDir, "TOOLS_DEF." + se= lf.Arch) + if not os.path.exists(tool_def_file): + self.ToolDefinition + return tool_def_file + + ## Retrieve the toolchain family of given toolchain tag. Default to 'M= SFT'. + @cached_property + def ToolChainFamily(self): + ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDatabase + if TAB_TOD_DEFINES_FAMILY not in ToolDefinition \ + or self.ToolChain not in ToolDefinition[TAB_TOD_DEFINES_FAMILY]= \ + or not ToolDefinition[TAB_TOD_DEFINES_FAMILY][self.ToolChain]: + EdkLogger.verbose("No tool chain family found in configuration= for %s. Default to MSFT." \ + % self.ToolChain) + RetVal =3D TAB_COMPILER_MSFT + else: + RetVal =3D ToolDefinition[TAB_TOD_DEFINES_FAMILY][self.ToolCha= in] + return RetVal + + @cached_property + def BuildRuleFamily(self): + ToolDefinition =3D self.Workspace.ToolDef.ToolsDefTxtDatabase + if TAB_TOD_DEFINES_BUILDRULEFAMILY not in ToolDefinition \ + or self.ToolChain not in ToolDefinition[TAB_TOD_DEFINES_BUILDRU= LEFAMILY] \ + or not ToolDefinition[TAB_TOD_DEFINES_BUILDRULEFAMILY][self.Too= lChain]: + EdkLogger.verbose("No tool chain family found in configuration= for %s. Default to MSFT." \ + % self.ToolChain) + return TAB_COMPILER_MSFT + + return ToolDefinition[TAB_TOD_DEFINES_BUILDRULEFAMILY][self.ToolCh= ain] + + ## Return the build options specific for all modules in this platform + @cached_property + def BuildOption(self): + return self._ExpandBuildOption(self.Platform.BuildOptions) + + def _BuildOptionWithToolDef(self, ToolDef): + return self._ExpandBuildOption(self.Platform.BuildOptions, ToolDef= =3DToolDef) + + ## Return the build options specific for EDK modules in this platform + @cached_property + def EdkBuildOption(self): + return self._ExpandBuildOption(self.Platform.BuildOptions, EDK_NAM= E) + + ## Return the build options specific for EDKII modules in this platform + @cached_property + def EdkIIBuildOption(self): + return self._ExpandBuildOption(self.Platform.BuildOptions, EDKII_N= AME) + + ## Parse build_rule.txt in Conf Directory. + # + # @retval BuildRule object + # + @cached_property + def BuildRule(self): + BuildRuleFile =3D None + if TAB_TAT_DEFINES_BUILD_RULE_CONF in self.Workspace.TargetTxt.Tar= getTxtDictionary: + BuildRuleFile =3D self.Workspace.TargetTxt.TargetTxtDictionary= [TAB_TAT_DEFINES_BUILD_RULE_CONF] + if not BuildRuleFile: + BuildRuleFile =3D gDefaultBuildRuleFile + RetVal =3D BuildRule(BuildRuleFile) + if RetVal._FileVersion =3D=3D "": + RetVal._FileVersion =3D AutoGenReqBuildRuleVerNum + else: + if RetVal._FileVersion < AutoGenReqBuildRuleVerNum : + # If Build Rule's version is less than the version number = required by the tools, halting the build. + EdkLogger.error("build", AUTOGEN_ERROR, + ExtraData=3D"The version number [%s] of bu= ild_rule.txt is less than the version number required by the AutoGen.(the m= inimum required version number is [%s])"\ + % (RetVal._FileVersion, AutoGenReqBuildRu= leVerNum)) + return RetVal + + ## Summarize the packages used by modules in this platform + @cached_property + def PackageList(self): + RetVal =3D set() + for Mb in self._MbList: + RetVal.update(Mb.Packages) + for lb in Mb.LibInstances: + RetVal.update(lb.Packages) + #Collect package set information from INF of FDF + for ModuleFile in self._AsBuildModuleList: + if ModuleFile in self.Platform.Modules: + continue + ModuleData =3D self.BuildDatabase[ModuleFile, self.Arch, self.= BuildTarget, self.ToolChain] + RetVal.update(ModuleData.Packages) + return list(RetVal) + + @cached_property + def NonDynamicPcdDict(self): + return {(Pcd.TokenCName, Pcd.TokenSpaceGuidCName):Pcd for Pcd in s= elf.NonDynamicPcdList} + + ## Get list of non-dynamic PCDs + @property + def NonDynamicPcdList(self): + if not self._NonDynamicPcdList: + self.CollectPlatformDynamicPcds() + return self._NonDynamicPcdList + + ## Get list of dynamic PCDs + @property + def DynamicPcdList(self): + if not self._DynamicPcdList: + self.CollectPlatformDynamicPcds() + return self._DynamicPcdList + + ## Generate Token Number for all PCD + @cached_property + def PcdTokenNumber(self): + RetVal =3D OrderedDict() + TokenNumber =3D 1 + # + # Make the Dynamic and DynamicEx PCD use within different TokenNum= ber area. + # Such as: + # + # Dynamic PCD: + # TokenNumber 0 ~ 10 + # DynamicEx PCD: + # TokeNumber 11 ~ 20 + # + for Pcd in self.DynamicPcdList: + if Pcd.Phase =3D=3D "PEI" and Pcd.Type in PCD_DYNAMIC_TYPE_SET: + EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) + RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber + TokenNumber +=3D 1 + + for Pcd in self.DynamicPcdList: + if Pcd.Phase =3D=3D "PEI" and Pcd.Type in PCD_DYNAMIC_EX_TYPE_= SET: + EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) + RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber + TokenNumber +=3D 1 + + for Pcd in self.DynamicPcdList: + if Pcd.Phase =3D=3D "DXE" and Pcd.Type in PCD_DYNAMIC_TYPE_SET: + EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) + RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber + TokenNumber +=3D 1 + + for Pcd in self.DynamicPcdList: + if Pcd.Phase =3D=3D "DXE" and Pcd.Type in PCD_DYNAMIC_EX_TYPE_= SET: + EdkLogger.debug(EdkLogger.DEBUG_5, "%s %s (%s) -> %d" % (P= cd.TokenCName, Pcd.TokenSpaceGuidCName, Pcd.Phase, TokenNumber)) + RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenN= umber + TokenNumber +=3D 1 + + for Pcd in self.NonDynamicPcdList: + RetVal[Pcd.TokenCName, Pcd.TokenSpaceGuidCName] =3D TokenNumber + TokenNumber +=3D 1 + return RetVal + + @cached_property + def _MbList(self): + return [self.BuildDatabase[m, self.Arch, self.BuildTarget, self.To= olChain] for m in self.Platform.Modules] + + @cached_property + def _MaList(self): + for ModuleFile in self.Platform.Modules: + Ma =3D ModuleAutoGen( + self.Workspace, + ModuleFile, + self.BuildTarget, + self.ToolChain, + self.Arch, + self.MetaFile, + self.DataPipe + ) + self.Platform.Modules[ModuleFile].M =3D Ma + return [x.M for x in self.Platform.Modules.values()] + + ## Summarize ModuleAutoGen objects of all modules to be built for this= platform + @cached_property + def ModuleAutoGenList(self): + RetVal =3D [] + for Ma in self._MaList: + if Ma not in RetVal: + RetVal.append(Ma) + return RetVal + + ## Summarize ModuleAutoGen objects of all libraries to be built for th= is platform + @cached_property + def LibraryAutoGenList(self): + RetVal =3D [] + for Ma in self._MaList: + for La in Ma.LibraryAutoGenList: + if La not in RetVal: + RetVal.append(La) + if Ma not in La.ReferenceModules: + La.ReferenceModules.append(Ma) + return RetVal + + ## Test if a module is supported by the platform + # + # An error will be raised directly if the module or its arch is not s= upported + # by the platform or current configuration + # + def ValidModule(self, Module): + return Module in self.Platform.Modules or Module in self.Platform.= LibraryInstances \ + or Module in self._AsBuildModuleList + @cached_property + def GetAllModuleInfo(self,WithoutPcd=3DTrue): + ModuleLibs =3D set() + for m in self.Platform.Modules: + module_obj =3D self.BuildDatabase[m,self.Arch,self.BuildTarget= ,self.ToolChain] + Libs =3D GetModuleLibInstances(module_obj, self.Platform, self= .BuildDatabase, self.Arch,self.BuildTarget,self.ToolChain) + ModuleLibs.update( set([(l.MetaFile.File,l.MetaFile.Root,l.Arc= h,True) for l in Libs])) + if WithoutPcd and module_obj.PcdIsDriver: + continue + ModuleLibs.add((m.File,m.Root,module_obj.Arch,False)) + + return ModuleLibs + + ## Resolve the library classes in a module to library instances + # + # This method will not only resolve library classes but also sort the = library + # instances according to the dependency-ship. + # + # @param Module The module from which the library classes will= be resolved + # + # @retval library_list List of library instances sorted + # + def ApplyLibraryInstance(self, Module): + # Cover the case that the binary INF file is list in the FDF file = but not DSC file, return empty list directly + if str(Module) not in self.Platform.Modules: + return [] + + return GetModuleLibInstances(Module, + self.Platform, + self.BuildDatabase, + self.Arch, + self.BuildTarget, + self.ToolChain, + self.MetaFile, + EdkLogger) + + ## Override PCD setting (type, value, ...) + # + # @param ToPcd The PCD to be overridden + # @param FromPcd The PCD overriding from + # + def _OverridePcd(self, ToPcd, FromPcd, Module=3D"", Msg=3D"", Library= =3D""): + # + # in case there's PCDs coming from FDF file, which have no type gi= ven. + # at this point, ToPcd.Type has the type found from dependent + # package + # + TokenCName =3D ToPcd.TokenCName + for PcdItem in GlobalData.MixedPcd: + if (ToPcd.TokenCName, ToPcd.TokenSpaceGuidCName) in GlobalData= .MixedPcd[PcdItem]: + TokenCName =3D PcdItem[0] + break + if FromPcd is not None: + if ToPcd.Pending and FromPcd.Type: + ToPcd.Type =3D FromPcd.Type + elif ToPcd.Type and FromPcd.Type\ + and ToPcd.Type !=3D FromPcd.Type and ToPcd.Type in FromPcd= .Type: + if ToPcd.Type.strip() =3D=3D TAB_PCDS_DYNAMIC_EX: + ToPcd.Type =3D FromPcd.Type + elif ToPcd.Type and FromPcd.Type \ + and ToPcd.Type !=3D FromPcd.Type: + if Library: + Module =3D str(Module) + " 's library file (" + str(Li= brary) + ")" + EdkLogger.error("build", OPTION_CONFLICT, "Mismatched PCD = type", + ExtraData=3D"%s.%s is used as [%s] in modu= le %s, but as [%s] in %s."\ + % (ToPcd.TokenSpaceGuidCName, To= kenCName, + ToPcd.Type, Module, FromPcd.T= ype, Msg), + File=3Dself.MetaFile) + + if FromPcd.MaxDatumSize: + ToPcd.MaxDatumSize =3D FromPcd.MaxDatumSize + ToPcd.MaxSizeUserSet =3D FromPcd.MaxDatumSize + if FromPcd.DefaultValue: + ToPcd.DefaultValue =3D FromPcd.DefaultValue + if FromPcd.TokenValue: + ToPcd.TokenValue =3D FromPcd.TokenValue + if FromPcd.DatumType: + ToPcd.DatumType =3D FromPcd.DatumType + if FromPcd.SkuInfoList: + ToPcd.SkuInfoList =3D FromPcd.SkuInfoList + if FromPcd.UserDefinedDefaultStoresFlag: + ToPcd.UserDefinedDefaultStoresFlag =3D FromPcd.UserDefined= DefaultStoresFlag + # Add Flexible PCD format parse + if ToPcd.DefaultValue: + try: + ToPcd.DefaultValue =3D ValueExpressionEx(ToPcd.Default= Value, ToPcd.DatumType, self.Platform._GuidDict)(True) + except BadExpression as Value: + EdkLogger.error('Parser', FORMAT_INVALID, 'PCD [%s.%s]= Value "%s", %s' %(ToPcd.TokenSpaceGuidCName, ToPcd.TokenCName, ToPcd.Defau= ltValue, Value), + File=3Dself.MetaFile) + + # check the validation of datum + IsValid, Cause =3D CheckPcdDatum(ToPcd.DatumType, ToPcd.Defaul= tValue) + if not IsValid: + EdkLogger.error('build', FORMAT_INVALID, Cause, File=3Dsel= f.MetaFile, + ExtraData=3D"%s.%s" % (ToPcd.TokenSpaceGui= dCName, TokenCName)) + ToPcd.validateranges =3D FromPcd.validateranges + ToPcd.validlists =3D FromPcd.validlists + ToPcd.expressions =3D FromPcd.expressions + ToPcd.CustomAttribute =3D FromPcd.CustomAttribute + + if FromPcd is not None and ToPcd.DatumType =3D=3D TAB_VOID and not= ToPcd.MaxDatumSize: + EdkLogger.debug(EdkLogger.DEBUG_9, "No MaxDatumSize specified = for PCD %s.%s" \ + % (ToPcd.TokenSpaceGuidCName, TokenCName)) + Value =3D ToPcd.DefaultValue + if not Value: + ToPcd.MaxDatumSize =3D '1' + elif Value[0] =3D=3D 'L': + ToPcd.MaxDatumSize =3D str((len(Value) - 2) * 2) + elif Value[0] =3D=3D '{': + ToPcd.MaxDatumSize =3D str(len(Value.split(','))) + else: + ToPcd.MaxDatumSize =3D str(len(Value) - 1) + + # apply default SKU for dynamic PCDS if specified one is not avail= able + if (ToPcd.Type in PCD_DYNAMIC_TYPE_SET or ToPcd.Type in PCD_DYNAMI= C_EX_TYPE_SET) \ + and not ToPcd.SkuInfoList: + if self.Platform.SkuName in self.Platform.SkuIds: + SkuName =3D self.Platform.SkuName + else: + SkuName =3D TAB_DEFAULT + ToPcd.SkuInfoList =3D { + SkuName : SkuInfoClass(SkuName, self.Platform.SkuIds[SkuNa= me][0], '', '', '', '', '', ToPcd.DefaultValue) + } + + ## Apply PCD setting defined platform to a module + # + # @param Module The module from which the PCD setting will be over= ridden + # + # @retval PCD_list The list PCDs with settings from platform + # + def ApplyPcdSetting(self, Module, Pcds, Library=3D""): + # for each PCD in module + for Name, Guid in Pcds: + PcdInModule =3D Pcds[Name, Guid] + # find out the PCD setting in platform + if (Name, Guid) in self.Platform.Pcds: + PcdInPlatform =3D self.Platform.Pcds[Name, Guid] + else: + PcdInPlatform =3D None + # then override the settings if any + self._OverridePcd(PcdInModule, PcdInPlatform, Module, Msg=3D"D= SC PCD sections", Library=3DLibrary) + # resolve the VariableGuid value + for SkuId in PcdInModule.SkuInfoList: + Sku =3D PcdInModule.SkuInfoList[SkuId] + if Sku.VariableGuid =3D=3D '': continue + Sku.VariableGuidValue =3D GuidValue(Sku.VariableGuid, self= .PackageList, self.MetaFile.Path) + if Sku.VariableGuidValue is None: + PackageList =3D "\n\t".join(str(P) for P in self.Packa= geList) + EdkLogger.error( + 'build', + RESOURCE_NOT_AVAILABLE, + "Value of GUID [%s] is not found in" % Sku= .VariableGuid, + ExtraData=3DPackageList + "\n\t(used with = %s.%s from module %s)" \ + % (Guid, Name, str= (Module)), + File=3Dself.MetaFile + ) + + # override PCD settings with module specific setting + if Module in self.Platform.Modules: + PlatformModule =3D self.Platform.Modules[str(Module)] + for Key in PlatformModule.Pcds: + if GlobalData.BuildOptionPcd: + for pcd in GlobalData.BuildOptionPcd: + (TokenSpaceGuidCName, TokenCName, FieldName, pcdva= lue, _) =3D pcd + if (TokenCName, TokenSpaceGuidCName) =3D=3D Key an= d FieldName =3D=3D"": + PlatformModule.Pcds[Key].DefaultValue =3D pcdv= alue + PlatformModule.Pcds[Key].PcdValueFromComm =3D = pcdvalue + break + Flag =3D False + if Key in Pcds: + ToPcd =3D Pcds[Key] + Flag =3D True + elif Key in GlobalData.MixedPcd: + for PcdItem in GlobalData.MixedPcd[Key]: + if PcdItem in Pcds: + ToPcd =3D Pcds[PcdItem] + Flag =3D True + break + if Flag: + self._OverridePcd(ToPcd, PlatformModule.Pcds[Key], Mod= ule, Msg=3D"DSC Components Module scoped PCD section", Library=3DLibrary) + # use PCD value to calculate the MaxDatumSize when it is not speci= fied + for Name, Guid in Pcds: + Pcd =3D Pcds[Name, Guid] + if Pcd.DatumType =3D=3D TAB_VOID and not Pcd.MaxDatumSize: + Pcd.MaxSizeUserSet =3D None + Value =3D Pcd.DefaultValue + if not Value: + Pcd.MaxDatumSize =3D '1' + elif Value[0] =3D=3D 'L': + Pcd.MaxDatumSize =3D str((len(Value) - 2) * 2) + elif Value[0] =3D=3D '{': + Pcd.MaxDatumSize =3D str(len(Value.split(','))) + else: + Pcd.MaxDatumSize =3D str(len(Value) - 1) + return list(Pcds.values()) + + ## Append build options in platform to a module + # + # @param Module The module to which the build options will be appe= nded + # + # @retval options The options appended with build options in pla= tform + # + def ApplyBuildOption(self, Module): + # Get the different options for the different style module + PlatformOptions =3D self.EdkIIBuildOption + ModuleTypeOptions =3D self.Platform.GetBuildOptionsByModuleType(ED= KII_NAME, Module.ModuleType) + ModuleTypeOptions =3D self._ExpandBuildOption(ModuleTypeOptions) + ModuleOptions =3D self._ExpandBuildOption(Module.BuildOptions) + if Module in self.Platform.Modules: + PlatformModule =3D self.Platform.Modules[str(Module)] + PlatformModuleOptions =3D self._ExpandBuildOption(PlatformModu= le.BuildOptions) + else: + PlatformModuleOptions =3D {} + + BuildRuleOrder =3D None + for Options in [self.ToolDefinition, ModuleOptions, PlatformOption= s, ModuleTypeOptions, PlatformModuleOptions]: + for Tool in Options: + for Attr in Options[Tool]: + if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: + BuildRuleOrder =3D Options[Tool][Attr] + + AllTools =3D set(list(ModuleOptions.keys()) + list(PlatformOptions= .keys()) + + list(PlatformModuleOptions.keys()) + list(ModuleTyp= eOptions.keys()) + + list(self.ToolDefinition.keys())) + BuildOptions =3D defaultdict(lambda: defaultdict(str)) + for Tool in AllTools: + for Options in [self.ToolDefinition, ModuleOptions, PlatformOp= tions, ModuleTypeOptions, PlatformModuleOptions]: + if Tool not in Options: + continue + for Attr in Options[Tool]: + # + # Do not generate it in Makefile + # + if Attr =3D=3D TAB_TOD_DEFINES_BUILDRULEORDER: + continue + Value =3D Options[Tool][Attr] + # check if override is indicated + if Value.startswith('=3D'): + BuildOptions[Tool][Attr] =3D mws.handleWsMacro(Val= ue[1:]) + else: + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + mws.handle= WsMacro(Value) + else: + BuildOptions[Tool][Attr] =3D mws.handleWsMacro= (Value) + + return BuildOptions, BuildRuleOrder + + + def GetGlobalBuildOptions(self,Module): + ModuleTypeOptions =3D self.Platform.GetBuildOptionsByModuleType(ED= KII_NAME, Module.ModuleType) + ModuleTypeOptions =3D self._ExpandBuildOption(ModuleTypeOptions) + + if Module in self.Platform.Modules: + PlatformModule =3D self.Platform.Modules[str(Module)] + PlatformModuleOptions =3D self._ExpandBuildOption(PlatformModu= le.BuildOptions) + else: + PlatformModuleOptions =3D {} + + return ModuleTypeOptions,PlatformModuleOptions + + @cached_property + def UniqueBaseName(self): + retVal =3D{} + name_path_map =3D {} + for Module in self._MbList: + name_path_map[Module.BaseName] =3D set() + for Module in self._MbList: + name_path_map[Module.BaseName].add(Module.MetaFile) + for name in name_path_map: + if len(name_path_map[name]) > 1: + guidset =3D set() + for metafile in name_path_map[name]: + m =3D self.BuildDatabase[metafile, self.Arch, self.Bui= ldTarget, self.ToolChain] + retVal[name] =3D '%s_%s' % (name, m.Guid) + guidset.add(m.Guid) + if len(guidset) > 1: + EdkLogger.error("build", FILE_DUPLICATED, 'Modules= have same BaseName and FILE_GUID:\n' + ' %s\n %s' % (name_path_map[name][0]= , name_path_map[name][1])) + return retVal + ## Expand * in build option key + # + # @param Options Options to be expanded + # @param ToolDef Use specified ToolDef instead of full version. + # This is needed during initialization to prevent + # infinite recursion betweeh BuildOptions, + # ToolDefinition, and this function. + # + # @retval options Options expanded + # + def _ExpandBuildOption(self, Options, ModuleStyle=3DNone, ToolDef=3DNo= ne): + if not ToolDef: + ToolDef =3D self.ToolDefinition + BuildOptions =3D {} + FamilyMatch =3D False + FamilyIsNull =3D True + + OverrideList =3D {} + # + # Construct a list contain the build options which need override. + # + for Key in Options: + # + # Key[0] -- tool family + # Key[1] -- TARGET_TOOLCHAIN_ARCH_COMMANDTYPE_ATTRIBUTE + # + if (Key[0] =3D=3D self.BuildRuleFamily and + (ModuleStyle is None or len(Key) < 3 or (len(Key) > 2 and = Key[2] =3D=3D ModuleStyle))): + Target, ToolChain, Arch, CommandType, Attr =3D Key[1].spli= t('_') + if (Target =3D=3D self.BuildTarget or Target =3D=3D TAB_ST= AR) and\ + (ToolChain =3D=3D self.ToolChain or ToolChain =3D=3D T= AB_STAR) and\ + (Arch =3D=3D self.Arch or Arch =3D=3D TAB_STAR) and\ + Options[Key].startswith("=3D"): + + if OverrideList.get(Key[1]) is not None: + OverrideList.pop(Key[1]) + OverrideList[Key[1]] =3D Options[Key] + + # + # Use the highest priority value. + # + if (len(OverrideList) >=3D 2): + KeyList =3D list(OverrideList.keys()) + for Index in range(len(KeyList)): + NowKey =3D KeyList[Index] + Target1, ToolChain1, Arch1, CommandType1, Attr1 =3D NowKey= .split("_") + for Index1 in range(len(KeyList) - Index - 1): + NextKey =3D KeyList[Index1 + Index + 1] + # + # Compare two Key, if one is included by another, choo= se the higher priority one + # + Target2, ToolChain2, Arch2, CommandType2, Attr2 =3D Ne= xtKey.split("_") + if (Target1 =3D=3D Target2 or Target1 =3D=3D TAB_STAR = or Target2 =3D=3D TAB_STAR) and\ + (ToolChain1 =3D=3D ToolChain2 or ToolChain1 =3D=3D= TAB_STAR or ToolChain2 =3D=3D TAB_STAR) and\ + (Arch1 =3D=3D Arch2 or Arch1 =3D=3D TAB_STAR or Ar= ch2 =3D=3D TAB_STAR) and\ + (CommandType1 =3D=3D CommandType2 or CommandType1 = =3D=3D TAB_STAR or CommandType2 =3D=3D TAB_STAR) and\ + (Attr1 =3D=3D Attr2 or Attr1 =3D=3D TAB_STAR or At= tr2 =3D=3D TAB_STAR): + + if CalculatePriorityValue(NowKey) > CalculatePrior= ityValue(NextKey): + if Options.get((self.BuildRuleFamily, NextKey)= ) is not None: + Options.pop((self.BuildRuleFamily, NextKey= )) + else: + if Options.get((self.BuildRuleFamily, NowKey))= is not None: + Options.pop((self.BuildRuleFamily, NowKey)) + + for Key in Options: + if ModuleStyle is not None and len (Key) > 2: + # Check Module style is EDK or EDKII. + # Only append build option for the matched style module. + if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: + continue + elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: + continue + Family =3D Key[0] + Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") + # if tool chain family doesn't match, skip it + if Tool in ToolDef and Family !=3D "": + FamilyIsNull =3D False + if ToolDef[Tool].get(TAB_TOD_DEFINES_BUILDRULEFAMILY, "") = !=3D "": + if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_BUILDRULE= FAMILY]: + continue + elif Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: + continue + FamilyMatch =3D True + # expand any wildcard + if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: + if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: + if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: + if Tool not in BuildOptions: + BuildOptions[Tool] =3D {} + if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): + BuildOptions[Tool][Attr] =3D Options[Key] + else: + # append options for the same tool except PATH + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] + else: + BuildOptions[Tool][Attr] =3D Options[Key] + # Build Option Family has been checked, which need't to be checked= again for family. + if FamilyMatch or FamilyIsNull: + return BuildOptions + + for Key in Options: + if ModuleStyle is not None and len (Key) > 2: + # Check Module style is EDK or EDKII. + # Only append build option for the matched style module. + if ModuleStyle =3D=3D EDK_NAME and Key[2] !=3D EDK_NAME: + continue + elif ModuleStyle =3D=3D EDKII_NAME and Key[2] !=3D EDKII_N= AME: + continue + Family =3D Key[0] + Target, Tag, Arch, Tool, Attr =3D Key[1].split("_") + # if tool chain family doesn't match, skip it + if Tool not in ToolDef or Family =3D=3D "": + continue + # option has been added before + if Family !=3D ToolDef[Tool][TAB_TOD_DEFINES_FAMILY]: + continue + + # expand any wildcard + if Target =3D=3D TAB_STAR or Target =3D=3D self.BuildTarget: + if Tag =3D=3D TAB_STAR or Tag =3D=3D self.ToolChain: + if Arch =3D=3D TAB_STAR or Arch =3D=3D self.Arch: + if Tool not in BuildOptions: + BuildOptions[Tool] =3D {} + if Attr !=3D "FLAGS" or Attr not in BuildOptions[T= ool] or Options[Key].startswith('=3D'): + BuildOptions[Tool][Attr] =3D Options[Key] + else: + # append options for the same tool except PATH + if Attr !=3D 'PATH': + BuildOptions[Tool][Attr] +=3D " " + Option= s[Key] + else: + BuildOptions[Tool][Attr] =3D Options[Key] + return BuildOptions diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTool= s/Source/Python/AutoGen/WorkspaceAutoGen.py new file mode 100644 index 000000000000..4372459e9a85 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py @@ -0,0 +1,902 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +## Import Modules +# +from __future__ import print_function +from __future__ import absolute_import +import os.path as path +import hashlib +from collections import defaultdict +from GenFds.FdfParser import FdfParser +from Workspace.WorkspaceCommon import GetModuleLibInstances +from AutoGen import GenMake +from AutoGen.AutoGen import AutoGen +from AutoGen.PlatformAutoGen import PlatformAutoGen +from AutoGen.BuildEngine import gDefaultBuildRuleFile +from Common.ToolDefClassObject import gDefaultToolsDefFile +from Common.StringUtils import NormPath +from Common.BuildToolError import * +from Common.DataType import * +from Common.Misc import * + +## Regular expression for splitting Dependency Expression string into toke= ns +gDepexTokenPattern =3D re.compile("(\(|\)|\w+| \S+\.inf)") + +## Regular expression for match: PCD(xxxx.yyy) +gPCDAsGuidPattern =3D re.compile(r"^PCD\(.+\..+\)$") + +## Workspace AutoGen class +# +# This class is used mainly to control the whole platform build for diff= erent +# architecture. This class will generate top level makefile. +# +class WorkspaceAutoGen(AutoGen): + # call super().__init__ then call the worker function with different p= arameter count + def __init__(self, Workspace, MetaFile, Target, Toolchain, Arch, *args= , **kwargs): + if not hasattr(self, "_Init"): + self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,= *args, **kwargs) + self._Init =3D True + + ## Initialize WorkspaceAutoGen + # + # @param WorkspaceDir Root directory of workspace + # @param ActivePlatform Meta-file of active platform + # @param Target Build target + # @param Toolchain Tool chain name + # @param ArchList List of architecture of current bu= ild + # @param MetaFileDb Database containing meta-files + # @param BuildConfig Configuration of build + # @param ToolDefinition Tool chain definitions + # @param FlashDefinitionFile File of flash definition + # @param Fds FD list to be generated + # @param Fvs FV list to be generated + # @param Caps Capsule list to be generated + # @param SkuId SKU id from command line + # + def _InitWorker(self, WorkspaceDir, ActivePlatform, Target, Toolchain,= ArchList, MetaFileDb, + BuildConfig, ToolDefinition, FlashDefinitionFile=3D'', Fds= =3DNone, Fvs=3DNone, Caps=3DNone, SkuId=3D'', UniFlag=3DNone, + Progress=3DNone, BuildModule=3DNone): + self.BuildDatabase =3D MetaFileDb + self.MetaFile =3D ActivePlatform + self.WorkspaceDir =3D WorkspaceDir + self.Platform =3D self.BuildDatabase[self.MetaFile, TAB_ARCH= _COMMON, Target, Toolchain] + GlobalData.gActivePlatform =3D self.Platform + self.BuildTarget =3D Target + self.ToolChain =3D Toolchain + self.ArchList =3D ArchList + self.SkuId =3D SkuId + self.UniFlag =3D UniFlag + + self.TargetTxt =3D BuildConfig + self.ToolDef =3D ToolDefinition + self.FdfFile =3D FlashDefinitionFile + self.FdTargetList =3D Fds if Fds else [] + self.FvTargetList =3D Fvs if Fvs else [] + self.CapTargetList =3D Caps if Caps else [] + self.AutoGenObjectList =3D [] + self._GuidDict =3D {} + + # there's many relative directory operations, so ... + os.chdir(self.WorkspaceDir) + + self.MergeArch() + self.ValidateBuildTarget() + + EdkLogger.info("") + if self.ArchList: + EdkLogger.info('%-16s =3D %s' % ("Architecture(s)", ' '.join(s= elf.ArchList))) + EdkLogger.info('%-16s =3D %s' % ("Build target", self.BuildTarget)) + EdkLogger.info('%-16s =3D %s' % ("Toolchain", self.ToolChain)) + + EdkLogger.info('\n%-24s =3D %s' % ("Active Platform", self.Platfor= m)) + if BuildModule: + EdkLogger.info('%-24s =3D %s' % ("Active Module", BuildModule)) + + if self.FdfFile: + EdkLogger.info('%-24s =3D %s' % ("Flash Image Definition", sel= f.FdfFile)) + + EdkLogger.verbose("\nFLASH_DEFINITION =3D %s" % self.FdfFile) + + if Progress: + Progress.Start("\nProcessing meta-data") + # + # Mark now build in AutoGen Phase + # + GlobalData.gAutoGenPhase =3D True + self.ProcessModuleFromPdf() + self.ProcessPcdType() + self.ProcessMixedPcd() + self.VerifyPcdsFromFDF() + self.CollectAllPcds() + self.GeneratePkgLevelHash() + # + # Check PCDs token value conflict in each DEC file. + # + self._CheckAllPcdsTokenValueConflict() + # + # Check PCD type and definition between DSC and DEC + # + self._CheckPcdDefineAndType() + + self.CreateBuildOptionsFile() + self.CreatePcdTokenNumberFile() + self.CreateModuleHashInfo() + GlobalData.gAutoGenPhase =3D False + + # + # Merge Arch + # + def MergeArch(self): + if not self.ArchList: + ArchList =3D set(self.Platform.SupArchList) + else: + ArchList =3D set(self.ArchList) & set(self.Platform.SupArchLis= t) + if not ArchList: + EdkLogger.error("build", PARAMETER_INVALID, + ExtraData =3D "Invalid ARCH specified. [Valid = ARCH: %s]" % (" ".join(self.Platform.SupArchList))) + elif self.ArchList and len(ArchList) !=3D len(self.ArchList): + SkippedArchList =3D set(self.ArchList).symmetric_difference(se= t(self.Platform.SupArchList)) + EdkLogger.verbose("\nArch [%s] is ignored because the platform= supports [%s] only!" + % (" ".join(SkippedArchList), " ".join(self.= Platform.SupArchList))) + self.ArchList =3D tuple(ArchList) + + # Validate build target + def ValidateBuildTarget(self): + if self.BuildTarget not in self.Platform.BuildTargets: + EdkLogger.error("build", PARAMETER_INVALID, + ExtraData=3D"Build target [%s] is not supporte= d by the platform. [Valid target: %s]" + % (self.BuildTarget, " ".join(self.P= latform.BuildTargets))) + @cached_property + def FdfProfile(self): + if not self.FdfFile: + self.FdfFile =3D self.Platform.FlashDefinition + + FdfProfile =3D None + if self.FdfFile: + Fdf =3D FdfParser(self.FdfFile.Path) + Fdf.ParseFile() + GlobalData.gFdfParser =3D Fdf + if Fdf.CurrentFdName and Fdf.CurrentFdName in Fdf.Profile.FdDi= ct: + FdDict =3D Fdf.Profile.FdDict[Fdf.CurrentFdName] + for FdRegion in FdDict.RegionList: + if str(FdRegion.RegionType) is 'FILE' and self.Platfor= m.VpdToolGuid in str(FdRegion.RegionDataList): + if int(FdRegion.Offset) % 8 !=3D 0: + EdkLogger.error("build", FORMAT_INVALID, 'The = VPD Base Address %s must be 8-byte aligned.' % (FdRegion.Offset)) + FdfProfile =3D Fdf.Profile + else: + if self.FdTargetList: + EdkLogger.info("No flash definition file found. FD [%s] wi= ll be ignored." % " ".join(self.FdTargetList)) + self.FdTargetList =3D [] + if self.FvTargetList: + EdkLogger.info("No flash definition file found. FV [%s] wi= ll be ignored." % " ".join(self.FvTargetList)) + self.FvTargetList =3D [] + if self.CapTargetList: + EdkLogger.info("No flash definition file found. Capsule [%= s] will be ignored." % " ".join(self.CapTargetList)) + self.CapTargetList =3D [] + + return FdfProfile + + def ProcessModuleFromPdf(self): + + if self.FdfProfile: + for fvname in self.FvTargetList: + if fvname.upper() not in self.FdfProfile.FvDict: + EdkLogger.error("build", OPTION_VALUE_INVALID, + "No such an FV in FDF file: %s" % fvna= me) + + # In DSC file may use FILE_GUID to override the module, then i= n the Platform.Modules use FILE_GUIDmodule.inf as key, + # but the path (self.MetaFile.Path) is the real path + for key in self.FdfProfile.InfDict: + if key =3D=3D 'ArchTBD': + MetaFile_cache =3D defaultdict(set) + for Arch in self.ArchList: + Current_Platform_cache =3D self.BuildDatabase[self= .MetaFile, Arch, self.BuildTarget, self.ToolChain] + for Pkey in Current_Platform_cache.Modules: + MetaFile_cache[Arch].add(Current_Platform_cach= e.Modules[Pkey].MetaFile) + for Inf in self.FdfProfile.InfDict[key]: + ModuleFile =3D PathClass(NormPath(Inf), GlobalData= .gWorkspace, Arch) + for Arch in self.ArchList: + if ModuleFile in MetaFile_cache[Arch]: + break + else: + ModuleData =3D self.BuildDatabase[ModuleFile, = Arch, self.BuildTarget, self.ToolChain] + if not ModuleData.IsBinaryModule: + EdkLogger.error('build', PARSER_ERROR, "Mo= dule %s NOT found in DSC file; Is it really a binary module?" % ModuleFile) + + else: + for Arch in self.ArchList: + if Arch =3D=3D key: + Platform =3D self.BuildDatabase[self.MetaFile,= Arch, self.BuildTarget, self.ToolChain] + MetaFileList =3D set() + for Pkey in Platform.Modules: + MetaFileList.add(Platform.Modules[Pkey].Me= taFile) + for Inf in self.FdfProfile.InfDict[key]: + ModuleFile =3D PathClass(NormPath(Inf), Gl= obalData.gWorkspace, Arch) + if ModuleFile in MetaFileList: + continue + ModuleData =3D self.BuildDatabase[ModuleFi= le, Arch, self.BuildTarget, self.ToolChain] + if not ModuleData.IsBinaryModule: + EdkLogger.error('build', PARSER_ERROR,= "Module %s NOT found in DSC file; Is it really a binary module?" % ModuleF= ile) + + + + # parse FDF file to get PCDs in it, if any + def VerifyPcdsFromFDF(self): + + if self.FdfProfile: + PcdSet =3D self.FdfProfile.PcdDict + self.VerifyPcdDeclearation(PcdSet) + + def ProcessPcdType(self): + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + Platform.Pcds + # generate the SourcePcdDict and BinaryPcdDict + Libs =3D [] + for BuildData in list(self.BuildDatabase._CACHE_.values()): + if BuildData.Arch !=3D Arch: + continue + if BuildData.MetaFile.Ext =3D=3D '.inf' and str(BuildData)= in Platform.Modules : + Libs.extend(GetModuleLibInstances(BuildData, Platform, + self.BuildDatabase, + Arch, + self.BuildTarget, + self.ToolChain + )) + for BuildData in list(self.BuildDatabase._CACHE_.values()): + if BuildData.Arch !=3D Arch: + continue + if BuildData.MetaFile.Ext =3D=3D '.inf': + for key in BuildData.Pcds: + if BuildData.Pcds[key].Pending: + if key in Platform.Pcds: + PcdInPlatform =3D Platform.Pcds[key] + if PcdInPlatform.Type: + BuildData.Pcds[key].Type =3D PcdInPlat= form.Type + BuildData.Pcds[key].Pending =3D False + + if BuildData.MetaFile in Platform.Modules: + PlatformModule =3D Platform.Modules[str(Bu= ildData.MetaFile)] + if key in PlatformModule.Pcds: + PcdInPlatform =3D PlatformModule.Pcds[= key] + if PcdInPlatform.Type: + BuildData.Pcds[key].Type =3D PcdIn= Platform.Type + BuildData.Pcds[key].Pending =3D Fa= lse + else: + #Pcd used in Library, Pcd Type from refere= nce module if Pcd Type is Pending + if BuildData.Pcds[key].Pending: + if bool(BuildData.LibraryClass): + if BuildData in set(Libs): + ReferenceModules =3D BuildData= .ReferenceModules + for ReferenceModule in Referen= ceModules: + if ReferenceModule.MetaFil= e in Platform.Modules: + RefPlatformModule =3D = Platform.Modules[str(ReferenceModule.MetaFile)] + if key in RefPlatformM= odule.Pcds: + PcdInReferenceModu= le =3D RefPlatformModule.Pcds[key] + if PcdInReferenceM= odule.Type: + BuildData.Pcds= [key].Type =3D PcdInReferenceModule.Type + BuildData.Pcds= [key].Pending =3D False + break + + def ProcessMixedPcd(self): + for Arch in self.ArchList: + SourcePcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set(),TAB_PCDS_DYNAMIC:set(),TAB_PCDS_FIXED_AT_BUILD:set()} + BinaryPcdDict =3D {TAB_PCDS_DYNAMIC_EX:set(), TAB_PCDS_PATCHAB= LE_IN_MODULE:set()} + SourcePcdDict_Keys =3D SourcePcdDict.keys() + BinaryPcdDict_Keys =3D BinaryPcdDict.keys() + + # generate the SourcePcdDict and BinaryPcdDict + + for BuildData in list(self.BuildDatabase._CACHE_.values()): + if BuildData.Arch !=3D Arch: + continue + if BuildData.MetaFile.Ext =3D=3D '.inf': + for key in BuildData.Pcds: + if TAB_PCDS_DYNAMIC_EX in BuildData.Pcds[key].Type: + if BuildData.IsBinaryModule: + BinaryPcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) + else: + SourcePcdDict[TAB_PCDS_DYNAMIC_EX].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) + + elif TAB_PCDS_PATCHABLE_IN_MODULE in BuildData.Pcd= s[key].Type: + if BuildData.MetaFile.Ext =3D=3D '.inf': + if BuildData.IsBinaryModule: + BinaryPcdDict[TAB_PCDS_PATCHABLE_IN_MO= DULE].add((BuildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGu= idCName)) + else: + SourcePcdDict[TAB_PCDS_PATCHABLE_IN_MO= DULE].add((BuildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGu= idCName)) + + elif TAB_PCDS_DYNAMIC in BuildData.Pcds[key].Type: + SourcePcdDict[TAB_PCDS_DYNAMIC].add((BuildData= .Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) + elif TAB_PCDS_FIXED_AT_BUILD in BuildData.Pcds[key= ].Type: + SourcePcdDict[TAB_PCDS_FIXED_AT_BUILD].add((Bu= ildData.Pcds[key].TokenCName, BuildData.Pcds[key].TokenSpaceGuidCName)) + + # + # A PCD can only use one type for all source modules + # + for i in SourcePcdDict_Keys: + for j in SourcePcdDict_Keys: + if i !=3D j: + Intersections =3D SourcePcdDict[i].intersection(So= urcePcdDict[j]) + if len(Intersections) > 0: + EdkLogger.error( + 'build', + FORMAT_INVALID, + "Building modules from source INFs, following = PCD use %s and %s access method. It must be corrected to use only one acces= s method." % (i, j), + ExtraData=3D'\n\t'.join(str(P[1]+'.'+P[0]) for= P in Intersections) + ) + + # + # intersection the BinaryPCD for Mixed PCD + # + for i in BinaryPcdDict_Keys: + for j in BinaryPcdDict_Keys: + if i !=3D j: + Intersections =3D BinaryPcdDict[i].intersection(Bi= naryPcdDict[j]) + for item in Intersections: + NewPcd1 =3D (item[0] + '_' + i, item[1]) + NewPcd2 =3D (item[0] + '_' + j, item[1]) + if item not in GlobalData.MixedPcd: + GlobalData.MixedPcd[item] =3D [NewPcd1, Ne= wPcd2] + else: + if NewPcd1 not in GlobalData.MixedPcd[item= ]: + GlobalData.MixedPcd[item].append(NewPc= d1) + if NewPcd2 not in GlobalData.MixedPcd[item= ]: + GlobalData.MixedPcd[item].append(NewPc= d2) + + # + # intersection the SourcePCD and BinaryPCD for Mixed PCD + # + for i in SourcePcdDict_Keys: + for j in BinaryPcdDict_Keys: + if i !=3D j: + Intersections =3D SourcePcdDict[i].intersection(Bi= naryPcdDict[j]) + for item in Intersections: + NewPcd1 =3D (item[0] + '_' + i, item[1]) + NewPcd2 =3D (item[0] + '_' + j, item[1]) + if item not in GlobalData.MixedPcd: + GlobalData.MixedPcd[item] =3D [NewPcd1, Ne= wPcd2] + else: + if NewPcd1 not in GlobalData.MixedPcd[item= ]: + GlobalData.MixedPcd[item].append(NewPc= d1) + if NewPcd2 not in GlobalData.MixedPcd[item= ]: + GlobalData.MixedPcd[item].append(NewPc= d2) + + BuildData =3D self.BuildDatabase[self.MetaFile, Arch, self.Bui= ldTarget, self.ToolChain] + for key in BuildData.Pcds: + for SinglePcd in GlobalData.MixedPcd: + if (BuildData.Pcds[key].TokenCName, BuildData.Pcds[key= ].TokenSpaceGuidCName) =3D=3D SinglePcd: + for item in GlobalData.MixedPcd[SinglePcd]: + Pcd_Type =3D item[0].split('_')[-1] + if (Pcd_Type =3D=3D BuildData.Pcds[key].Type) = or (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC_EX and BuildData.Pcds[key].Type in PCD= _DYNAMIC_EX_TYPE_SET) or \ + (Pcd_Type =3D=3D TAB_PCDS_DYNAMIC and Build= Data.Pcds[key].Type in PCD_DYNAMIC_TYPE_SET): + Value =3D BuildData.Pcds[key] + Value.TokenCName =3D BuildData.Pcds[key].T= okenCName + '_' + Pcd_Type + if len(key) =3D=3D 2: + newkey =3D (Value.TokenCName, key[1]) + elif len(key) =3D=3D 3: + newkey =3D (Value.TokenCName, key[1], = key[2]) + del BuildData.Pcds[key] + BuildData.Pcds[newkey] =3D Value + break + break + + if self.FdfProfile: + PcdSet =3D self.FdfProfile.PcdDict + # handle the mixed pcd in FDF file + for key in PcdSet: + if key in GlobalData.MixedPcd: + Value =3D PcdSet[key] + del PcdSet[key] + for item in GlobalData.MixedPcd[key]: + PcdSet[item] =3D Value + + #Collect package set information from INF of FDF + @cached_property + def PkgSet(self): + if not self.FdfFile: + self.FdfFile =3D self.Platform.FlashDefinition + + if self.FdfFile: + ModuleList =3D self.FdfProfile.InfList + else: + ModuleList =3D [] + Pkgs =3D {} + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + PkgSet =3D set() + for mb in [self.BuildDatabase[m, Arch, self.BuildTarget, self.= ToolChain] for m in Platform.Modules]: + PkgSet.update(mb.Packages) + for Inf in ModuleList: + ModuleFile =3D PathClass(NormPath(Inf), GlobalData.gWorksp= ace, Arch) + if ModuleFile in Platform.Modules: + continue + ModuleData =3D self.BuildDatabase[ModuleFile, Arch, self.B= uildTarget, self.ToolChain] + PkgSet.update(ModuleData.Packages) + Pkgs[Arch] =3D list(PkgSet) + return Pkgs + + def VerifyPcdDeclearation(self,PcdSet): + for Arch in self.ArchList: + Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] + Pkgs =3D self.PkgSet[Arch] + DecPcds =3D set() + DecPcdsKey =3D set() + for Pkg in Pkgs: + for Pcd in Pkg.Pcds: + DecPcds.add((Pcd[0], Pcd[1])) + DecPcdsKey.add((Pcd[0], Pcd[1], Pcd[2])) + + Platform.SkuName =3D self.SkuId + for Name, Guid,Fileds in PcdSet: + if (Name, Guid) not in DecPcds: + EdkLogger.error( + 'build', + PARSER_ERROR, + "PCD (%s.%s) used in FDF is not declared in DEC fi= les." % (Guid, Name), + File =3D self.FdfProfile.PcdFileLineDict[Name, Gui= d, Fileds][0], + Line =3D self.FdfProfile.PcdFileLineDict[Name, Gui= d, Fileds][1] + ) + else: + # Check whether Dynamic or DynamicEx PCD used in FDF f= ile. If used, build break and give a error message. + if (Name, Guid, TAB_PCDS_FIXED_AT_BUILD) in DecPcdsKey= \ + or (Name, Guid, TAB_PCDS_PATCHABLE_IN_MODULE) in D= ecPcdsKey \ + or (Name, Guid, TAB_PCDS_FEATURE_FLAG) in DecPcdsK= ey: + continue + elif (Name, Guid, TAB_PCDS_DYNAMIC) in DecPcdsKey or (= Name, Guid, TAB_PCDS_DYNAMIC_EX) in DecPcdsKey: + EdkLogger.error( + 'build', + PARSER_ERROR, + "Using Dynamic or DynamicEx type of PCD [%= s.%s] in FDF file is not allowed." % (Guid, Name), + File =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][0], + Line =3D self.FdfProfile.PcdFileLineDict[N= ame, Guid, Fileds][1] + ) + def CollectAllPcds(self): + + for Arch in self.ArchList: + Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) + # + # Explicitly collect platform's dynamic PCDs + # + Pa.CollectPlatformDynamicPcds() + Pa.CollectFixedAtBuildPcds() + self.AutoGenObjectList.append(Pa) + # We need to calculate the PcdTokenNumber after all Arch Pcds are = collected. + for Arch in self.ArchList: + #Pcd TokenNumber + Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) + self.UpdateModuleDataPipe(Arch, {"PCD_TNUM":Pa.PcdTokenNumber= }) + + def UpdateModuleDataPipe(self,arch, attr_dict): + for (Target, Toolchain, Arch, MetaFile) in AutoGen.Cache(): + if Arch !=3D arch: + continue + try: + AutoGen.Cache()[(Target, Toolchain, Arch, MetaFile)].DataP= ipe.DataContainer =3D attr_dict + except Exception: + pass + # + # Generate Package level hash value + # + def GeneratePkgLevelHash(self): + for Arch in self.ArchList: + GlobalData.gPackageHash =3D {} + if GlobalData.gUseHashCache: + for Pkg in self.PkgSet[Arch]: + self._GenPkgLevelHash(Pkg) + + + def CreateBuildOptionsFile(self): + # + # Create BuildOptions Macro & PCD metafile, also add the Active Pl= atform and FDF file. + # + content =3D 'gCommandLineDefines: ' + content +=3D str(GlobalData.gCommandLineDefines) + content +=3D TAB_LINE_BREAK + content +=3D 'BuildOptionPcd: ' + content +=3D str(GlobalData.BuildOptionPcd) + content +=3D TAB_LINE_BREAK + content +=3D 'Active Platform: ' + content +=3D str(self.Platform) + content +=3D TAB_LINE_BREAK + if self.FdfFile: + content +=3D 'Flash Image Definition: ' + content +=3D str(self.FdfFile) + content +=3D TAB_LINE_BREAK + SaveFileOnChange(os.path.join(self.BuildDir, 'BuildOptions'), cont= ent, False) + + def CreatePcdTokenNumberFile(self): + # + # Create PcdToken Number file for Dynamic/DynamicEx Pcd. + # + PcdTokenNumber =3D 'PcdTokenNumber: ' + for Arch in self.ArchList: + Pa =3D PlatformAutoGen(self, self.MetaFile, self.BuildTarget, = self.ToolChain, Arch) + if Pa.PcdTokenNumber: + if Pa.DynamicPcdList: + for Pcd in Pa.DynamicPcdList: + PcdTokenNumber +=3D TAB_LINE_BREAK + PcdTokenNumber +=3D str((Pcd.TokenCName, Pcd.Token= SpaceGuidCName)) + PcdTokenNumber +=3D ' : ' + PcdTokenNumber +=3D str(Pa.PcdTokenNumber[Pcd.Toke= nCName, Pcd.TokenSpaceGuidCName]) + SaveFileOnChange(os.path.join(self.BuildDir, 'PcdTokenNumber'), Pc= dTokenNumber, False) + + def CreateModuleHashInfo(self): + # + # Get set of workspace metafiles + # + AllWorkSpaceMetaFiles =3D self._GetMetaFiles(self.BuildTarget, sel= f.ToolChain) + + # + # Retrieve latest modified time of all metafiles + # + SrcTimeStamp =3D 0 + for f in AllWorkSpaceMetaFiles: + if os.stat(f)[8] > SrcTimeStamp: + SrcTimeStamp =3D os.stat(f)[8] + self._SrcTimeStamp =3D SrcTimeStamp + + if GlobalData.gUseHashCache: + m =3D hashlib.md5() + for files in AllWorkSpaceMetaFiles: + if files.endswith('.dec'): + continue + f =3D open(files, 'rb') + Content =3D f.read() + f.close() + m.update(Content) + SaveFileOnChange(os.path.join(self.BuildDir, 'AutoGen.hash'), = m.hexdigest(), False) + GlobalData.gPlatformHash =3D m.hexdigest() + + # + # Write metafile list to build directory + # + AutoGenFilePath =3D os.path.join(self.BuildDir, 'AutoGen') + if os.path.exists (AutoGenFilePath): + os.remove(AutoGenFilePath) + if not os.path.exists(self.BuildDir): + os.makedirs(self.BuildDir) + with open(os.path.join(self.BuildDir, 'AutoGen'), 'w+') as file: + for f in AllWorkSpaceMetaFiles: + print(f, file=3Dfile) + return True + + def _GenPkgLevelHash(self, Pkg): + if Pkg.PackageName in GlobalData.gPackageHash: + return + + PkgDir =3D os.path.join(self.BuildDir, Pkg.Arch, Pkg.PackageName) + CreateDirectory(PkgDir) + HashFile =3D os.path.join(PkgDir, Pkg.PackageName + '.hash') + m =3D hashlib.md5() + # Get .dec file's hash value + f =3D open(Pkg.MetaFile.Path, 'rb') + Content =3D f.read() + f.close() + m.update(Content) + # Get include files hash value + if Pkg.Includes: + for inc in sorted(Pkg.Includes, key=3Dlambda x: str(x)): + for Root, Dirs, Files in os.walk(str(inc)): + for File in sorted(Files): + File_Path =3D os.path.join(Root, File) + f =3D open(File_Path, 'rb') + Content =3D f.read() + f.close() + m.update(Content) + SaveFileOnChange(HashFile, m.hexdigest(), False) + GlobalData.gPackageHash[Pkg.PackageName] =3D m.hexdigest() + + def _GetMetaFiles(self, Target, Toolchain): + AllWorkSpaceMetaFiles =3D set() + # + # add fdf + # + if self.FdfFile: + AllWorkSpaceMetaFiles.add (self.FdfFile.Path) + for f in GlobalData.gFdfParser.GetAllIncludedFile(): + AllWorkSpaceMetaFiles.add (f.FileName) + # + # add dsc + # + AllWorkSpaceMetaFiles.add(self.MetaFile.Path) + + # + # add build_rule.txt & tools_def.txt + # + AllWorkSpaceMetaFiles.add(os.path.join(GlobalData.gConfDirectory, = gDefaultBuildRuleFile)) + AllWorkSpaceMetaFiles.add(os.path.join(GlobalData.gConfDirectory, = gDefaultToolsDefFile)) + + # add BuildOption metafile + # + AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir, 'BuildOption= s')) + + # add PcdToken Number file for Dynamic/DynamicEx Pcd + # +# AllWorkSpaceMetaFiles.add(os.path.join(self.BuildDir, 'PcdTokenNu= mber')) + + for Arch in self.ArchList: + # + # add dec + # + for Package in PlatformAutoGen(self, self.MetaFile, Target, To= olchain, Arch).PackageList: + AllWorkSpaceMetaFiles.add(Package.MetaFile.Path) + + # + # add included dsc + # + for filePath in self.BuildDatabase[self.MetaFile, Arch, Target= , Toolchain]._RawData.IncludedFiles: + AllWorkSpaceMetaFiles.add(filePath.Path) + + return AllWorkSpaceMetaFiles + + def _CheckPcdDefineAndType(self): + PcdTypeSet =3D {TAB_PCDS_FIXED_AT_BUILD, + TAB_PCDS_PATCHABLE_IN_MODULE, + TAB_PCDS_FEATURE_FLAG, + TAB_PCDS_DYNAMIC, + TAB_PCDS_DYNAMIC_EX} + + # This dict store PCDs which are not used by any modules with spec= ified arches + UnusedPcd =3D OrderedDict() + for Pa in self.AutoGenObjectList: + # Key of DSC's Pcds dictionary is PcdCName, TokenSpaceGuid + for Pcd in Pa.Platform.Pcds: + PcdType =3D Pa.Platform.Pcds[Pcd].Type + + # If no PCD type, this PCD comes from FDF + if not PcdType: + continue + + # Try to remove Hii and Vpd suffix + if PcdType.startswith(TAB_PCDS_DYNAMIC_EX): + PcdType =3D TAB_PCDS_DYNAMIC_EX + elif PcdType.startswith(TAB_PCDS_DYNAMIC): + PcdType =3D TAB_PCDS_DYNAMIC + + for Package in Pa.PackageList: + # Key of DEC's Pcds dictionary is PcdCName, TokenSpace= Guid, PcdType + if (Pcd[0], Pcd[1], PcdType) in Package.Pcds: + break + for Type in PcdTypeSet: + if (Pcd[0], Pcd[1], Type) in Package.Pcds: + EdkLogger.error( + 'build', + FORMAT_INVALID, + "Type [%s] of PCD [%s.%s] in DSC file does= n't match the type [%s] defined in DEC file." \ + % (Pa.Platform.Pcds[Pcd].Type, Pcd[1], Pcd= [0], Type), + ExtraData=3DNone + ) + return + else: + UnusedPcd.setdefault(Pcd, []).append(Pa.Arch) + + for Pcd in UnusedPcd: + EdkLogger.warn( + 'build', + "The PCD was not specified by any INF module in the platfo= rm for the given architecture.\n" + "\tPCD: [%s.%s]\n\tPlatform: [%s]\n\tArch: %s" + % (Pcd[1], Pcd[0], os.path.basename(str(self.MetaFile)), s= tr(UnusedPcd[Pcd])), + ExtraData=3DNone + ) + + def __repr__(self): + return "%s [%s]" % (self.MetaFile, ", ".join(self.ArchList)) + + ## Return the directory to store FV files + @cached_property + def FvDir(self): + return path.join(self.BuildDir, TAB_FV_DIRECTORY) + + ## Return the directory to store all intermediate and final files built + @cached_property + def BuildDir(self): + return self.AutoGenObjectList[0].BuildDir + + ## Return the build output directory platform specifies + @cached_property + def OutputDir(self): + return self.Platform.OutputDirectory + + ## Return platform name + @cached_property + def Name(self): + return self.Platform.PlatformName + + ## Return meta-file GUID + @cached_property + def Guid(self): + return self.Platform.Guid + + ## Return platform version + @cached_property + def Version(self): + return self.Platform.Version + + ## Return paths of tools + @cached_property + def ToolDefinition(self): + return self.AutoGenObjectList[0].ToolDefinition + + ## Return directory of platform makefile + # + # @retval string Makefile directory + # + @cached_property + def MakeFileDir(self): + return self.BuildDir + + ## Return build command string + # + # @retval string Build command string + # + @cached_property + def BuildCommand(self): + # BuildCommand should be all the same. So just get one from platfo= rm AutoGen + return self.AutoGenObjectList[0].BuildCommand + + ## Check the PCDs token value conflict in each DEC file. + # + # Will cause build break and raise error message while two PCDs confli= ct. + # + # @return None + # + def _CheckAllPcdsTokenValueConflict(self): + for Pa in self.AutoGenObjectList: + for Package in Pa.PackageList: + PcdList =3D list(Package.Pcds.values()) + PcdList.sort(key=3Dlambda x: int(x.TokenValue, 0)) + Count =3D 0 + while (Count < len(PcdList) - 1) : + Item =3D PcdList[Count] + ItemNext =3D PcdList[Count + 1] + # + # Make sure in the same token space the TokenValue sho= uld be unique + # + if (int(Item.TokenValue, 0) =3D=3D int(ItemNext.TokenV= alue, 0)): + SameTokenValuePcdList =3D [] + SameTokenValuePcdList.append(Item) + SameTokenValuePcdList.append(ItemNext) + RemainPcdListLength =3D len(PcdList) - Count - 2 + for ValueSameCount in range(RemainPcdListLength): + if int(PcdList[len(PcdList) - RemainPcdListLen= gth + ValueSameCount].TokenValue, 0) =3D=3D int(Item.TokenValue, 0): + SameTokenValuePcdList.append(PcdList[len(P= cdList) - RemainPcdListLength + ValueSameCount]) + else: + break; + # + # Sort same token value PCD list with TokenGuid an= d TokenCName + # + SameTokenValuePcdList.sort(key=3Dlambda x: "%s.%s"= % (x.TokenSpaceGuidCName, x.TokenCName)) + SameTokenValuePcdListCount =3D 0 + while (SameTokenValuePcdListCount < len(SameTokenV= aluePcdList) - 1): + Flag =3D False + TemListItem =3D SameTokenValuePcdList[SameToke= nValuePcdListCount] + TemListItemNext =3D SameTokenValuePcdList[Same= TokenValuePcdListCount + 1] + + if (TemListItem.TokenSpaceGuidCName =3D=3D Tem= ListItemNext.TokenSpaceGuidCName) and (TemListItem.TokenCName !=3D TemListI= temNext.TokenCName): + for PcdItem in GlobalData.MixedPcd: + if (TemListItem.TokenCName, TemListIte= m.TokenSpaceGuidCName) in GlobalData.MixedPcd[PcdItem] or \ + (TemListItemNext.TokenCName, TemLi= stItemNext.TokenSpaceGuidCName) in GlobalData.MixedPcd[PcdItem]: + Flag =3D True + if not Flag: + EdkLogger.error( + 'build', + FORMAT_INVALID, + "The TokenValue [%s] of PC= D [%s.%s] is conflict with: [%s.%s] in %s"\ + % (TemListItem.TokenValue,= TemListItem.TokenSpaceGuidCName, TemListItem.TokenCName, TemListItemNext.T= okenSpaceGuidCName, TemListItemNext.TokenCName, Package), + ExtraData=3DNone + ) + SameTokenValuePcdListCount +=3D 1 + Count +=3D SameTokenValuePcdListCount + Count +=3D 1 + + PcdList =3D list(Package.Pcds.values()) + PcdList.sort(key=3Dlambda x: "%s.%s" % (x.TokenSpaceGuidCN= ame, x.TokenCName)) + Count =3D 0 + while (Count < len(PcdList) - 1) : + Item =3D PcdList[Count] + ItemNext =3D PcdList[Count + 1] + # + # Check PCDs with same TokenSpaceGuidCName.TokenCName = have same token value as well. + # + if (Item.TokenSpaceGuidCName =3D=3D ItemNext.TokenSpac= eGuidCName) and (Item.TokenCName =3D=3D ItemNext.TokenCName) and (int(Item.= TokenValue, 0) !=3D int(ItemNext.TokenValue, 0)): + EdkLogger.error( + 'build', + FORMAT_INVALID, + "The TokenValue [%s] of PCD [%s.%s] in= %s defined in two places should be same as well."\ + % (Item.TokenValue, Item.TokenSpaceGui= dCName, Item.TokenCName, Package), + ExtraData=3DNone + ) + Count +=3D 1 + ## Generate fds command + @property + def GenFdsCommand(self): + return (GenMake.TopLevelMakefile(self)._TEMPLATE_.Replace(GenMake.= TopLevelMakefile(self)._TemplateDict)).strip() + + @property + def GenFdsCommandDict(self): + FdsCommandDict =3D {} + LogLevel =3D EdkLogger.GetLevel() + if LogLevel =3D=3D EdkLogger.VERBOSE: + FdsCommandDict["verbose"] =3D True + elif LogLevel <=3D EdkLogger.DEBUG_9: + FdsCommandDict["debug"] =3D LogLevel - 1 + elif LogLevel =3D=3D EdkLogger.QUIET: + FdsCommandDict["quiet"] =3D True + + if GlobalData.gEnableGenfdsMultiThread: + FdsCommandDict["GenfdsMultiThread"] =3D True + if GlobalData.gIgnoreSource: + FdsCommandDict["IgnoreSources"] =3D True + + FdsCommandDict["OptionPcd"] =3D [] + for pcd in GlobalData.BuildOptionPcd: + if pcd[2]: + pcdname =3D '.'.join(pcd[0:3]) + else: + pcdname =3D '.'.join(pcd[0:2]) + if pcd[3].startswith('{'): + FdsCommandDict["OptionPcd"].append(pcdname + '=3D' + 'H' += '"' + pcd[3] + '"') + else: + FdsCommandDict["OptionPcd"].append(pcdname + '=3D' + pcd[3= ]) + + MacroList =3D [] + # macros passed to GenFds + MacroDict =3D {} + MacroDict.update(GlobalData.gGlobalDefines) + MacroDict.update(GlobalData.gCommandLineDefines) + for MacroName in MacroDict: + if MacroDict[MacroName] !=3D "": + MacroList.append('"%s=3D%s"' % (MacroName, MacroDict[Macro= Name].replace('\\', '\\\\'))) + else: + MacroList.append('"%s"' % MacroName) + FdsCommandDict["macro"] =3D MacroList + + FdsCommandDict["fdf_file"] =3D [self.FdfFile] + FdsCommandDict["build_target"] =3D self.BuildTarget + FdsCommandDict["toolchain_tag"] =3D self.ToolChain + FdsCommandDict["active_platform"] =3D str(self) + + FdsCommandDict["conf_directory"] =3D GlobalData.gConfDirectory + FdsCommandDict["build_architecture_list"] =3D ','.join(self.ArchLi= st) + FdsCommandDict["platform_build_directory"] =3D self.BuildDir + + FdsCommandDict["fd"] =3D self.FdTargetList + FdsCommandDict["fv"] =3D self.FvTargetList + FdsCommandDict["cap"] =3D self.CapTargetList + return FdsCommandDict + + ## Create makefile for the platform and modules in it + # + # @param CreateDepsMakeFile Flag indicating if the makefil= e for + # modules will be created as well + # + def CreateMakeFile(self, CreateDepsMakeFile=3DFalse): + if not CreateDepsMakeFile: + return + for Pa in self.AutoGenObjectList: + Pa.CreateMakeFile(True) + + ## Create autogen code for platform and modules + # + # Since there's no autogen code for platform, this method will do not= hing + # if CreateModuleCodeFile is set to False. + # + # @param CreateDepsCodeFile Flag indicating if creating mo= dule's + # autogen code file or not + # + def CreateCodeFile(self, CreateDepsCodeFile=3DFalse): + if not CreateDepsCodeFile: + return + for Pa in self.AutoGenObjectList: + Pa.CreateCodeFile(True) + + ## Create AsBuilt INF file the platform + # + def CreateAsBuiltInf(self): + return + diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Pyth= on/Common/Misc.py index 9a63463913d0..adaecffd93c7 100644 --- a/BaseTools/Source/Python/Common/Misc.py +++ b/BaseTools/Source/Python/Common/Misc.py @@ -652,11 +652,10 @@ def GuidValue(CName, PackageList, Inffile =3D None): if not Inffile.startswith(P.MetaFile.Dir): GuidKeys =3D [x for x in P.Guids if x not in P._PrivateGui= ds] if CName in GuidKeys: return P.Guids[CName] return None - return None =20 ## A string template class # # This class implements a template for string replacement. A string templ= ate # looks like following diff --git a/BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py b/BaseT= ools/Source/Python/PatchPcdValue/PatchPcdValue.py index 02735e165ca1..d35cd792704c 100644 --- a/BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py +++ b/BaseTools/Source/Python/PatchPcdValue/PatchPcdValue.py @@ -9,11 +9,10 @@ # Import Modules # import Common.LongFilePathOs as os from Common.LongFilePathSupport import OpenLongFilePath as open import sys -import re =20 from optparse import OptionParser from optparse import make_option from Common.BuildToolError import * import Common.EdkLogger as EdkLogger diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/= Source/Python/Workspace/DscBuildData.py index dd5c3c2bd1f2..37976d067ed9 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -1371,15 +1371,15 @@ class DscBuildData(PlatformBuildClassObject): if PcdInDec.Type in [self._PCD_TYPE_STRING_[MODEL_PCD_= FIXED_AT_BUILD], self._PCD_TYPE_STRING_[MODEL_PCD_P= ATCHABLE_IN_MODULE], self._PCD_TYPE_STRING_[MODEL_PCD_F= EATURE_FLAG], self._PCD_TYPE_STRING_[MODEL_PCD_D= YNAMIC], self._PCD_TYPE_STRING_[MODEL_PCD_D= YNAMIC_EX]]: - self.Pcds[Name, Guid] =3D copy.deepcopy(PcdInDec) - self.Pcds[Name, Guid].DefaultValue =3D NoFiledValu= es[( Guid, Name)][0] + self._Pcds[Name, Guid] =3D copy.deepcopy(PcdInDec) + self._Pcds[Name, Guid].DefaultValue =3D NoFiledVal= ues[( Guid, Name)][0] if PcdInDec.Type in [self._PCD_TYPE_STRING_[MODEL_PCD_= DYNAMIC], self._PCD_TYPE_STRING_[MODEL_PCD_D= YNAMIC_EX]]: - self.Pcds[Name, Guid].SkuInfoList =3D {TAB_DEFAULT= :SkuInfoClass(TAB_DEFAULT, self.SkuIds[TAB_DEFAULT][0], '', '', '', '', '',= NoFiledValues[( Guid, Name)][0])} + self._Pcds[Name, Guid].SkuInfoList =3D {TAB_DEFAUL= T:SkuInfoClass(TAB_DEFAULT, self.SkuIds[TAB_DEFAULT][0], '', '', '', '', ''= , NoFiledValues[( Guid, Name)][0])} return AllPcds =20 def OverrideByFdfOverAll(self,AllPcds): =20 if GlobalData.gFdfParser is None: @@ -1417,12 +1417,12 @@ class DscBuildData(PlatformBuildClassObject): if PcdInDec: PcdInDec.PcdValueFromFdf =3D Value if PcdInDec.Type in [self._PCD_TYPE_STRING_[MODEL_PCD_= FIXED_AT_BUILD], self._PCD_TYPE_STRING_[MODEL_PCD_P= ATCHABLE_IN_MODULE], self._PCD_TYPE_STRING_[MODEL_PCD_F= EATURE_FLAG]]: - self.Pcds[Name, Guid] =3D copy.deepcopy(PcdInDec) - self.Pcds[Name, Guid].DefaultValue =3D Value + self._Pcds[Name, Guid] =3D copy.deepcopy(PcdInDec) + self._Pcds[Name, Guid].DefaultValue =3D Value return AllPcds =20 def ParsePcdNameStruct(self,NamePart1,NamePart2): TokenSpaceCName =3D PcdCName =3D DimensionAttr =3D Field =3D "" if "." in NamePart1: diff --git a/BaseTools/Source/Python/Workspace/InfBuildData.py b/BaseTools/= Source/Python/Workspace/InfBuildData.py index da35391d3aff..e63246b03b6e 100644 --- a/BaseTools/Source/Python/Workspace/InfBuildData.py +++ b/BaseTools/Source/Python/Workspace/InfBuildData.py @@ -152,10 +152,17 @@ class InfBuildData(ModuleBuildClassObject): self._GuidsUsedByPcd =3D OrderedDict() self._GuidComments =3D None self._PcdComments =3D None self._BuildOptions =3D None self._DependencyFileList =3D None + self.LibInstances =3D [] + self.ReferenceModules =3D set() + self.Guids + self.Pcds + def SetReferenceModule(self,Module): + self.ReferenceModules.add(Module) + return self =20 ## XXX[key] =3D value def __setitem__(self, key, value): self.__dict__[self._PROPERTY_[key]] =3D value =20 @@ -703,10 +710,29 @@ class InfBuildData(ModuleBuildClassObject): RetVal.update(self._GetPcd(MODEL_PCD_DYNAMIC)) RetVal.update(self._GetPcd(MODEL_PCD_DYNAMIC_EX)) return RetVal =20 @cached_property + def ModulePcdList(self): + RetVal =3D self.Pcds + return RetVal + @cached_property + def LibraryPcdList(self): + if bool(self.LibraryClass): + return [] + RetVal =3D {} + Pcds =3D set() + for Library in self.LibInstances: + PcdsInLibrary =3D OrderedDict() + for Key in Library.Pcds: + if Key in self.Pcds or Key in Pcds: + continue + Pcds.add(Key) + PcdsInLibrary[Key] =3D copy.copy(Library.Pcds[Key]) + RetVal[Library] =3D PcdsInLibrary + return RetVal + @cached_property def PcdsName(self): PcdsName =3D set() for Type in (MODEL_PCD_FIXED_AT_BUILD,MODEL_PCD_PATCHABLE_IN_MODUL= E,MODEL_PCD_FEATURE_FLAG,MODEL_PCD_DYNAMIC,MODEL_PCD_DYNAMIC_EX): RecordList =3D self._RawData[Type, self._Arch, self._Platform] for TokenSpaceGuid, PcdCName, _, _, _, _, _ in RecordList: @@ -1028,5 +1054,8 @@ class InfBuildData(ModuleBuildClassObject): @property def IsBinaryModule(self): if (self.Binaries and not self.Sources) or GlobalData.gIgnoreSourc= e: return True return False +def ExtendCopyDictionaryLists(CopyToDict, CopyFromDict): + for Key in CopyFromDict: + CopyToDict[Key].extend(CopyFromDict[Key]) diff --git a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py b/BaseToo= ls/Source/Python/Workspace/WorkspaceCommon.py index 41ae684d3ee9..76583f46e500 100644 --- a/BaseTools/Source/Python/Workspace/WorkspaceCommon.py +++ b/BaseTools/Source/Python/Workspace/WorkspaceCommon.py @@ -86,10 +86,12 @@ def GetDeclaredPcd(Platform, BuildDatabase, Arch, Targe= t, Toolchain, additionalP # def GetLiabraryInstances(Module, Platform, BuildDatabase, Arch, Target, To= olchain): return GetModuleLibInstances(Module, Platform, BuildDatabase, Arch, Ta= rget, Toolchain) =20 def GetModuleLibInstances(Module, Platform, BuildDatabase, Arch, Target, T= oolchain, FileName =3D '', EdkLogger =3D None): + if Module.LibInstances: + return Module.LibInstances ModuleType =3D Module.ModuleType =20 # add forced library instances (specified under LibraryClasses section= s) # # If a module has a MODULE_TYPE of USER_DEFINED, @@ -244,6 +246,8 @@ def GetModuleLibInstances(Module, Platform, BuildDataba= se, Arch, Target, Toolcha # # Build the list of constructor and destructor names # The DAG Topo sort produces the destructor order, so the list of cons= tructors must generated in the reverse order # SortedLibraryList.reverse() + Module.LibInstances =3D SortedLibraryList + SortedLibraryList =3D [lib.SetReferenceModule(Module) for lib in Sorte= dLibraryList] return SortedLibraryList diff --git a/BaseTools/Source/Python/Workspace/WorkspaceDatabase.py b/BaseT= ools/Source/Python/Workspace/WorkspaceDatabase.py index 28a975f54e51..ab7b4506c1c1 100644 --- a/BaseTools/Source/Python/Workspace/WorkspaceDatabase.py +++ b/BaseTools/Source/Python/Workspace/WorkspaceDatabase.py @@ -60,10 +60,12 @@ class WorkspaceDatabase(object): MODEL_FILE_DEC : DecBuildData, MODEL_FILE_DSC : DscBuildData, } =20 _CACHE_ =3D {} # (FilePath, Arch) : + def GetCache(self): + return self._CACHE_ =20 # constructor def __init__(self, WorkspaceDb): self.WorkspaceDb =3D WorkspaceDb =20 @@ -201,10 +203,11 @@ class WorkspaceDatabase(object): Platform =3D self.BuildObject[PathClass(Dscfile), TAB_COMMON] if Platform is None: EdkLogger.error('build', PARSER_ERROR, "Failed to parser DSC f= ile: %s" % Dscfile) return Platform =20 +BuildDB =3D WorkspaceDatabase() ## # # This acts like the main() function for the script, unless it is 'import'= ed into another # script. # diff --git a/BaseTools/Source/Python/build/BuildReport.py b/BaseTools/Sourc= e/Python/build/BuildReport.py index a3eb3b2383e4..a54c7f4ca547 100644 --- a/BaseTools/Source/Python/build/BuildReport.py +++ b/BaseTools/Source/Python/build/BuildReport.py @@ -32,11 +32,11 @@ from Common.BuildToolError import CODE_ERROR from Common.BuildToolError import COMMAND_FAILURE from Common.BuildToolError import FORMAT_INVALID from Common.LongFilePathSupport import OpenLongFilePath as open from Common.MultipleWorkspace import MultipleWorkspace as mws import Common.GlobalData as GlobalData -from AutoGen.AutoGen import ModuleAutoGen +from AutoGen.ModuleAutoGen import ModuleAutoGen from Common.Misc import PathClass from Common.StringUtils import NormPath from Common.DataType import * import collections from Common.Expression import * @@ -2138,11 +2138,11 @@ class PlatformReport(object): if GlobalData.gFdfParser is not None: if Pa.Arch in GlobalData.gFdfParser.Profile.InfDict: INFList =3D GlobalData.gFdfParser.Profile.InfDict[= Pa.Arch] for InfName in INFList: InfClass =3D PathClass(NormPath(InfName), Wa.W= orkspaceDir, Pa.Arch) - Ma =3D ModuleAutoGen(Wa, InfClass, Pa.BuildTar= get, Pa.ToolChain, Pa.Arch, Wa.MetaFile) + Ma =3D ModuleAutoGen(Wa, InfClass, Pa.BuildTar= get, Pa.ToolChain, Pa.Arch, Wa.MetaFile,Pa.DataPile) if Ma is None: continue if Ma not in ModuleAutoGenList: ModuleAutoGenList.append(Ma) for MGen in ModuleAutoGenList: diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index cce091c4f8b5..61a7cf77ac12 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -10,46 +10,49 @@ =20 ## # Import Modules # from __future__ import print_function -import Common.LongFilePathOs as os -import re +from __future__ import absolute_import +import os.path as path import sys +import os +import re import glob import time import platform import traceback -import encodings.ascii import multiprocessing - -from struct import * -from threading import * +from threading import Thread,Event,BoundedSemaphore import threading +from subprocess import Popen,PIPE +from collections import OrderedDict, defaultdict from optparse import OptionParser -from subprocess import * +from AutoGen.PlatformAutoGen import PlatformAutoGen +from AutoGen.ModuleAutoGen import ModuleAutoGen +from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen +from AutoGen import GenMake from Common import Misc as Utils =20 -from Common.LongFilePathSupport import OpenLongFilePath as open from Common.TargetTxtClassObject import TargetTxt from Common.ToolDefClassObject import ToolDef +from Common.Misc import PathClass,SaveFileOnChange,RemoveDirectory +from Common.StringUtils import NormPath +from Common.MultipleWorkspace import MultipleWorkspace as mws +from Common.BuildToolError import * from Common.DataType import * +import Common.EdkLogger as EdkLogger from Common.BuildVersion import gBUILD_VERSION -from AutoGen.AutoGen import * -from Common.BuildToolError import * -from Workspace.WorkspaceDatabase import WorkspaceDatabase -from Common.MultipleWorkspace import MultipleWorkspace as mws +from Workspace.WorkspaceDatabase import BuildDB =20 from BuildReport import BuildReport -from GenPatchPcdTable.GenPatchPcdTable import * -from PatchPcdValue.PatchPcdValue import * +from GenPatchPcdTable.GenPatchPcdTable import PeImageClass,parsePcdInfoFro= mMapFile +from PatchPcdValue.PatchPcdValue import PatchBinaryFile =20 -import Common.EdkLogger import Common.GlobalData as GlobalData from GenFds.GenFds import GenFds, GenFdsApi =20 -from collections import OrderedDict, defaultdict =20 # Version and Copyright VersionNumber =3D "0.60" + ' ' + gBUILD_VERSION __version__ =3D "%prog Version " + VersionNumber __copyright__ =3D "Copyright (c) 2007 - 2018, Intel Corporation All right= s reserved." @@ -772,11 +775,11 @@ class Build(): # Get standard WORKSPACE/Conf use the absolute path to the= WORKSPACE/Conf ConfDirectoryPath =3D mws.join(self.WorkspaceDir, 'Conf') GlobalData.gConfDirectory =3D ConfDirectoryPath GlobalData.gDatabasePath =3D os.path.normpath(os.path.join(ConfDir= ectoryPath, GlobalData.gDatabasePath)) =20 - self.Db =3D WorkspaceDatabase() + self.Db =3D BuildDB self.BuildDatabase =3D self.Db.BuildObject self.Platform =3D None self.ToolChainFamily =3D None self.LoadFixAddress =3D 0 self.UniFlag =3D BuildOptions.Flag @@ -1697,17 +1700,21 @@ class Build(): CmdListDict =3D {} if GlobalData.gEnableGenfdsMultiThread and self.Fdf: CmdListDict =3D self._GenFfsCmd(Wa.ArchList) =20 for Arch in Wa.ArchList: + PcdMaList =3D [] GlobalData.gGlobalDefines['ARCH'] =3D Arch Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTar= get, ToolChain, Arch) for Module in Pa.Platform.Modules: # Get ModuleAutoGen object to generate C code file= and makefile - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile) + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile,Pa.DataPipe) if Ma is None: continue + if Ma.PcdIsDriver: + Ma.PlatformInfo =3D Pa + PcdMaList.append(Ma) self.BuildModules.append(Ma) self._BuildPa(self.Target, Pa, FfsCommand=3DCmdListDic= t) =20 # Create MAP file when Load Fix Address is enabled. if self.Target in ["", "all", "fds"]: @@ -1799,11 +1806,11 @@ class Build(): AutoGenStart =3D time.time() GlobalData.gGlobalDefines['ARCH'] =3D Arch Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTar= get, ToolChain, Arch) for Module in Pa.Platform.Modules: if self.ModuleFile.Dir =3D=3D Module.Dir and self.= ModuleFile.Name =3D=3D Module.Name: - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, = ToolChain, Arch, self.PlatformFile) + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, = ToolChain, Arch, self.PlatformFile,Pa.DataPipe) if Ma is None: continue MaList.append(Ma) if Ma.CanSkipbyHash(): self.HashSkipModules.append(Ma) @@ -1979,10 +1986,11 @@ class Build(): # multi-thread exit flag ExitFlag =3D threading.Event() ExitFlag.clear() self.AutoGenTime +=3D int(round((time.time() - WorkspaceAu= toGenTime))) for Arch in Wa.ArchList: + PcdMaList =3D [] AutoGenStart =3D time.time() GlobalData.gGlobalDefines['ARCH'] =3D Arch Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTar= get, ToolChain, Arch) if Pa is None: continue @@ -1996,14 +2004,17 @@ class Build(): if Inf in Pa.Platform.Modules: continue ModuleList.append(Inf) for Module in ModuleList: # Get ModuleAutoGen object to generate C code file= and makefile - Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile) + Ma =3D ModuleAutoGen(Wa, Module, BuildTarget, Tool= Chain, Arch, self.PlatformFile,Pa.DataPipe) =20 if Ma is None: continue + if Ma.PcdIsDriver: + Ma.PlatformInfo =3D Pa + PcdMaList.append(Ma) if Ma.CanSkipbyHash(): self.HashSkipModules.append(Ma) if GlobalData.gBinCacheSource: EdkLogger.quiet("cache hit: %s[%s]" % (Ma.= MetaFile.Path, Ma.Arch)) continue --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43911): https://edk2.groups.io/g/devel/message/43911 Mute This Topic: https://groups.io/mt/32512455/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43912+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43912+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430490; cv=none; d=zoho.com; s=zohoarc; b=kyYKvxPJYueZBCbSydANC0dQfO04L5qytqsCiwpEcbFFW4A6MoDsjcOy8pUd2wyhsCAMkE5UwVIaz3NJVlIdciZfKfalcz4EuVefAdNJAosGOjm4HKIw2GBjHs7R1YTmqSVaKVv4uGOc2ck/zZ6yCgFLqoIhF3MZKghiToUofGU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430490; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=/8XLQxpczqYkZBBfhGgVHUlGAzpdUCW8V2sbATRuTw0=; b=bwwF4gQIIls0CSUlpSeHwHjYhwe7TSIX7JKNfOdrILpT4LneyLNr+CfOB4kg1h0TytgagERDx/15hrhN4IkD7rKL5ihfsGXy/c7TrFviWOmI9pPdU+PCFtVqtLHrPP121CByk3Efa7+uMa/Q+pnxN10mTIoRC8rEIO1gxqoc4t4= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43912+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430490010618.9308633950634; Wed, 17 Jul 2019 23:14:50 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:49 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544041" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:47 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 5/9] BaseTools: Enable Multiple Process AutoGen Date: Thu, 18 Jul 2019 14:14:19 +0800 Message-Id: <20190718061423.30612-6-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430489; bh=AtM1qCT82GwUYbrDfUjyR1OFBpVpYT+rNmuGC23wQZA=; h=Cc:Date:From:Reply-To:Subject:To; b=pinKgOOOCFy3X4k7C3jwV7klaLYNLQ4UCC3BKDp3Ymu9MYOsLKdqT4KA2O2EFhlIcuE jYZyej7FFzva0FD3Qij/VCHvT2W829V2T3rDHpH5kxO0KGXxCqoBJSEqL2iAKZ+j+rC2G GdVxJHk5AqRcGJ8gnN5W7TVjrtC6L//S5bU= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 Assign the Module AutoGen tasks into multiple sub process. Cc: Liming Gao Signed-off-by: Bob Feng --- .../Source/Python/AutoGen/AutoGenWorker.py | 160 ++++++++++++++++++ BaseTools/Source/Python/AutoGen/DataPipe.py | 6 + BaseTools/Source/Python/AutoGen/GenC.py | 4 +- .../Source/Python/AutoGen/ModuleAutoGen.py | 8 +- .../Source/Python/AutoGen/PlatformAutoGen.py | 4 +- BaseTools/Source/Python/build/build.py | 109 +++++++----- 6 files changed, 240 insertions(+), 51 deletions(-) create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py new file mode 100644 index 000000000000..a0415f0d3420 --- /dev/null +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -0,0 +1,160 @@ +## @file +# Create makefile for MS nmake and GNU make +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +# +from __future__ import absolute_import +import multiprocessing as mp +import threading +from Common.Misc import PathClass +from AutoGen.ModuleAutoGen import ModuleAutoGen +from AutoGen.ModuleAutoGenHelper import WorkSpaceInfo,AutoGenInfo +import Common.GlobalData as GlobalData +import Common.EdkLogger as EdkLogger +import os +from Common.MultipleWorkspace import MultipleWorkspace as mws +from AutoGen.AutoGen import AutoGen +from Workspace.WorkspaceDatabase import BuildDB +import time +from queue import Empty +import traceback +import sys +from AutoGen.DataPipe import MemoryDataPipe +class AutoGenManager(threading.Thread): + def __init__(self,autogen_workers, feedback_q): + super(AutoGenManager,self).__init__() + self.autogen_workers =3D autogen_workers + self.feedback_q =3D feedback_q + self.terminate =3D False + self.Status =3D True + def run(self): + try: + while True: + if self.terminate: + break + if self.feedback_q.empty(): + time.sleep(1) + continue + badnews =3D self.feedback_q.get(False) + if badnews: + print(badnews) + self.Status =3D False + self.TerminateWorkers() + break + except Exception: + return + + def kill(self): + self.terminate =3D True + + def TerminateWorkers(self): + for w in self.autogen_workers: + if w.is_alive(): + w.terminate() + +class AutoGenWorkerInProcess(mp.Process): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k): + mp.Process.__init__(self) + self.module_queue =3D module_queue + self.data_pipe_file_path =3Ddata_pipe_file_path + self.data_pipe =3D None + self.feedback_q =3D feedback_q + self.PlatformMetaFileSet =3D {} + self.file_lock =3D file_lock + def GetPlatformMetaFile(self,filepath,root): + try: + return self.PlatformMetaFileSet[(filepath,root)] + except: + self.PlatformMetaFileSet[(filepath,root)] =3D filepath + return self.PlatformMetaFileSet[(filepath,root)] + def run(self): + try: + taskname =3D "Init" + with self.file_lock: + if not os.path.exists(self.data_pipe_file_path): + self.feedback_q.put(taskname + ":" + "load data pipe %= s failed." % self.data_pipe_file_path) + self.data_pipe =3D MemoryDataPipe() + self.data_pipe.load(self.data_pipe_file_path) + EdkLogger.Initialize() + loglevel =3D self.data_pipe.Get("LogLevel") + if not loglevel: + loglevel =3D EdkLogger.INFO + EdkLogger.SetLevel(loglevel) + logfile =3D self.data_pipe.Get("LogFile") + if logfile: + EdkLogger.SetLogFile(logfile) + target =3D self.data_pipe.Get("P_Info").get("Target") + toolchain =3D self.data_pipe.Get("P_Info").get("ToolChain") + archlist =3D self.data_pipe.Get("P_Info").get("ArchList") + + active_p =3D self.data_pipe.Get("P_Info").get("ActivePlatform") + workspacedir =3D self.data_pipe.Get("P_Info").get("WorkspaceDi= r") + PackagesPath =3D os.getenv("PACKAGES_PATH") + mws.setWs(workspacedir, PackagesPath) + self.Wa =3D WorkSpaceInfo( + workspacedir,active_p,target,toolchain,archlist + ) + GlobalData.gGlobalDefines =3D self.data_pipe.Get("G_defines") + GlobalData.gCommandLineDefines =3D self.data_pipe.Get("CL_defi= nes") + os.environ._data =3D self.data_pipe.Get("Env_Var") + GlobalData.gWorkspace =3D workspacedir + GlobalData.gDisableIncludePathCheck =3D False + GlobalData.gFdfParser =3D self.data_pipe.Get("FdfParser") + GlobalData.gDatabasePath =3D self.data_pipe.Get("DatabasePath") + module_count =3D 0 + FfsCmd =3D self.data_pipe.Get("FfsCommand") + if FfsCmd is None: + FfsCmd =3D {} + PlatformMetaFile =3D self.GetPlatformMetaFile(self.data_pipe.G= et("P_Info").get("ActivePlatform"), + self.data_pipe.Get("P_Info").= get("WorkspaceDir")) + while not self.module_queue.empty(): + module_count +=3D 1 + module_file,module_root,module_path,module_basename,module= _originalpath,module_arch,IsLib =3D self.module_queue.get() + modulefullpath =3D os.path.join(module_root,module_file) + taskname =3D " : ".join((modulefullpath,module_arch)) + module_metafile =3D PathClass(module_file,module_root) + if module_path: + module_metafile.Path =3D module_path + if module_basename: + module_metafile.BaseName =3D module_basename + if module_originalpath: + module_metafile.OriginalPath =3D PathClass(module_orig= inalpath,module_root) + arch =3D module_arch + target =3D self.data_pipe.Get("P_Info").get("Target") + toolchain =3D self.data_pipe.Get("P_Info").get("ToolChain") + Ma =3D ModuleAutoGen(self.Wa,module_metafile,target,toolch= ain,arch,PlatformMetaFile,self.data_pipe) + Ma.IsLibrary =3D IsLib + Ma.CreateCodeFile() + Ma.CreateMakeFile(GenFfsList=3DFfsCmd.get((Ma.MetaFile.Fil= e, Ma.Arch),[])) + except Empty: + pass + except: + traceback.print_exc(file=3Dsys.stdout) + self.feedback_q.put(taskname) + + def printStatus(self): + print("Processs ID: %d Run %d modules in AutoGen " % (os.getpid(),= len(AutoGen.Cache()))) + print("Processs ID: %d Run %d modules in AutoGenInfo " % (os.getpi= d(),len(AutoGenInfo.GetCache()))) + groupobj =3D {} + for buildobj in BuildDB.BuildObject.GetCache().values(): + if str(buildobj).lower().endswith("dec"): + try: + groupobj['dec'].append(str(buildobj)) + except: + groupobj['dec'] =3D [str(buildobj)] + if str(buildobj).lower().endswith("dsc"): + try: + groupobj['dsc'].append(str(buildobj)) + except: + groupobj['dsc'] =3D [str(buildobj)] + + if str(buildobj).lower().endswith("inf"): + try: + groupobj['inf'].append(str(buildobj)) + except: + groupobj['inf'] =3D [str(buildobj)] + + print("Processs ID: %d Run %d pkg in WDB " % (os.getpid(),len(grou= pobj.get("dec",[])))) + print("Processs ID: %d Run %d pla in WDB " % (os.getpid(),len(grou= pobj.get("dsc",[])))) + print("Processs ID: %d Run %d inf in WDB " % (os.getpid(),len(grou= pobj.get("inf",[])))) diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py index 5bcc39bd380d..9478f41d481b 100644 --- a/BaseTools/Source/Python/AutoGen/DataPipe.py +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -9,10 +9,11 @@ from Workspace.WorkspaceDatabase import BuildDB from Workspace.WorkspaceCommon import GetModuleLibInstances import Common.GlobalData as GlobalData import os import pickle from pickle import HIGHEST_PROTOCOL +from Common import EdkLogger =20 class PCD_DATA(): def __init__(self,TokenCName,TokenSpaceGuidCName,Type,DatumType,SkuInf= oList,DefaultValue, MaxDatumSize,UserDefinedDefaultStoresFlag,validateranges, validlists,expressions,CustomAttribute,TokenValue): @@ -32,17 +33,19 @@ class PCD_DATA(): =20 class DataPipe(object): def __init__(self, BuildDir=3DNone): self.data_container =3D {} self.BuildDir =3D BuildDir + self.dump_file =3D "" =20 class MemoryDataPipe(DataPipe): =20 def Get(self,key): return self.data_container.get(key) =20 def dump(self,file_path): + self.dump_file =3D file_path with open(file_path,'wb') as fd: pickle.dump(self.data_container,fd,pickle.HIGHEST_PROTOCOL) =20 def load(self,file_path): with open(file_path,'rb') as fd: @@ -141,7 +144,10 @@ class MemoryDataPipe(DataPipe): =20 self.DataContainer =3D {"PackageList": [(dec.MetaFile,dec.Arch) fo= r dec in PlatformInfo.PackageList]} =20 self.DataContainer =3D {"GuidDict": PlatformInfo.Platform._GuidDic= t} =20 + self.DataContainer =3D {"DatabasePath":GlobalData.gDatabasePath} self.DataContainer =3D {"FdfParser": True if GlobalData.gFdfParser= else False} =20 + self.DataContainer =3D {"LogLevel": EdkLogger.GetLevel()} + self.DataContainer =3D {"LogFile": GlobalData.gOptions.LogFile if = GlobalData.gOptions.LogFile is not None else ""} diff --git a/BaseTools/Source/Python/AutoGen/GenC.py b/BaseTools/Source/Pyt= hon/AutoGen/GenC.py index 4c3f4e3e55ae..910c8fe3706c 100644 --- a/BaseTools/Source/Python/AutoGen/GenC.py +++ b/BaseTools/Source/Python/AutoGen/GenC.py @@ -1470,12 +1470,12 @@ def CreateModuleEntryPointCode(Info, AutoGenC, Auto= GenH): 'UefiSpecVersion': UefiSpecVersion + 'U' } =20 if Info.ModuleType in [SUP_MODULE_PEI_CORE, SUP_MODULE_DXE_CORE, SUP_M= ODULE_SMM_CORE, SUP_MODULE_MM_CORE_STANDALONE]: if Info.SourceFileList: - if NumEntryPoints !=3D 1: - EdkLogger.error( + if NumEntryPoints !=3D 1: + EdkLogger.error( "build", AUTOGEN_ERROR, '%s must have exactly one entry point' % Info.ModuleType, File=3Dstr(Info), ExtraData=3D ", ".join(Info.Module.ModuleEntryPointList) diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index 5fea71a86c83..69119a528621 100644 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -1662,13 +1662,11 @@ class ModuleAutoGen(AutoGen): if self.IsBinaryModule: return =20 self.GenFfsList =3D GenFfsList =20 - if not self.IsLibrary and CreateLibraryMakeFile: - for LibraryAutoGen in self.LibraryAutoGenList: - LibraryAutoGen.CreateMakeFile() + if self.CanSkip(): return =20 if len(self.CustomMakefile) =3D=3D 0: Makefile =3D GenMake.ModuleMakefile(self) @@ -1704,13 +1702,11 @@ class ModuleAutoGen(AutoGen): if self.IsBinaryModule: if self.IsLibrary: self.CopyBinaryFiles() return =20 - if not self.IsLibrary and CreateLibraryCodeFile: - for LibraryAutoGen in self.LibraryAutoGenList: - LibraryAutoGen.CreateCodeFile() + =20 if self.CanSkip(): return =20 AutoGenList =3D [] diff --git a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py b/BaseTools= /Source/Python/AutoGen/PlatformAutoGen.py index 48cf6df85ac1..5157f084eb5c 100644 --- a/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/PlatformAutoGen.py @@ -1072,14 +1072,14 @@ class PlatformAutoGen(AutoGen): def GetAllModuleInfo(self,WithoutPcd=3DTrue): ModuleLibs =3D set() for m in self.Platform.Modules: module_obj =3D self.BuildDatabase[m,self.Arch,self.BuildTarget= ,self.ToolChain] Libs =3D GetModuleLibInstances(module_obj, self.Platform, self= .BuildDatabase, self.Arch,self.BuildTarget,self.ToolChain) - ModuleLibs.update( set([(l.MetaFile.File,l.MetaFile.Root,l.Arc= h,True) for l in Libs])) + ModuleLibs.update( set([(l.MetaFile.File,l.MetaFile.Root,l.Met= aFile.Path,l.MetaFile.BaseName,l.MetaFile.OriginalPath,l.Arch,True) for l i= n Libs])) if WithoutPcd and module_obj.PcdIsDriver: continue - ModuleLibs.add((m.File,m.Root,module_obj.Arch,False)) + ModuleLibs.add((m.File,m.Root,m.Path,m.BaseName,m.OriginalPath= ,module_obj.Arch,bool(module_obj.LibraryClass))) =20 return ModuleLibs =20 ## Resolve the library classes in a module to library instances # diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 61a7cf77ac12..b4916a32171a 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -28,10 +28,11 @@ from subprocess import Popen,PIPE from collections import OrderedDict, defaultdict from optparse import OptionParser from AutoGen.PlatformAutoGen import PlatformAutoGen from AutoGen.ModuleAutoGen import ModuleAutoGen from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen +from AutoGen.AutoGenWorker import AutoGenWorkerInProcess,AutoGenManager from AutoGen import GenMake from Common import Misc as Utils =20 from Common.TargetTxtClassObject import TargetTxt from Common.ToolDefClassObject import ToolDef @@ -48,11 +49,11 @@ from BuildReport import BuildReport from GenPatchPcdTable.GenPatchPcdTable import PeImageClass,parsePcdInfoFro= mMapFile from PatchPcdValue.PatchPcdValue import PatchBinaryFile =20 import Common.GlobalData as GlobalData from GenFds.GenFds import GenFds, GenFdsApi - +import multiprocessing as mp =20 # Version and Copyright VersionNumber =3D "0.60" + ' ' + gBUILD_VERSION __version__ =3D "%prog Version " + VersionNumber __copyright__ =3D "Copyright (c) 2007 - 2018, Intel Corporation All right= s reserved." @@ -341,13 +342,13 @@ class ModuleMakeUnit(BuildUnit): # # @param self The object pointer # @param Obj The ModuleAutoGen object the build is working = on # @param Target The build target name, one of gSupportedTarget # - def __init__(self, Obj, Target): - Dependency =3D [ModuleMakeUnit(La, Target) for La in Obj.LibraryAu= toGenList] - BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Dependency= , Obj.MakeFileDir) + def __init__(self, Obj, BuildCommand,Target): + Dependency =3D [ModuleMakeUnit(La, BuildCommand,Target) for La in = Obj.LibraryAutoGenList] + BuildUnit.__init__(self, Obj, BuildCommand, Target, Dependency, Ob= j.MakeFileDir) if Target in [None, "", "all"]: self.Target =3D "tbuild" =20 ## The smallest platform unit that can be built by nmake/make command in m= ulti-thread build mode # @@ -362,14 +363,14 @@ class PlatformMakeUnit(BuildUnit): # # @param self The object pointer # @param Obj The PlatformAutoGen object the build is workin= g on # @param Target The build target name, one of gSupportedTarget # - def __init__(self, Obj, Target): - Dependency =3D [ModuleMakeUnit(Lib, Target) for Lib in self.BuildO= bject.LibraryAutoGenList] - Dependency.extend([ModuleMakeUnit(Mod, Target) for Mod in self.Bui= ldObject.ModuleAutoGenList]) - BuildUnit.__init__(self, Obj, Obj.BuildCommand, Target, Dependency= , Obj.MakeFileDir) + def __init__(self, Obj, BuildCommand, Target): + Dependency =3D [ModuleMakeUnit(Lib, BuildCommand, Target) for Lib = in self.BuildObject.LibraryAutoGenList] + Dependency.extend([ModuleMakeUnit(Mod, BuildCommand,Target) for Mo= d in self.BuildObject.ModuleAutoGenList]) + BuildUnit.__init__(self, Obj, BuildCommand, Target, Dependency, Ob= j.MakeFileDir) =20 ## The class representing the task of a module build or platform build # # This class manages the build tasks in multi-thread build mode. Its jobs = include # scheduling thread running, catching thread error, monitor the thread sta= tus, etc. @@ -821,12 +822,35 @@ class Build(): self.TargetTxt =3D TargetTxt self.ToolDef =3D ToolDef if not (self.LaunchPrebuildFlag and os.path.exists(self.PlatformBu= ildPath)): self.InitBuild() =20 + self.AutoGenMgr =3D None EdkLogger.info("") os.chdir(self.WorkspaceDir) + def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList): + if SkipAutoGen: + return + feedback_q =3D mp.Queue() + file_lock =3D mp.Lock() + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock) for _ in range(mp.cpu_count()//2)] + self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q) + self.AutoGenMgr.start() + for w in auto_workers: + w.start() + if PcdMaList is not None: + for PcdMa in PcdMaList: + PcdMa.CreateCodeFile(True) + PcdMa.CreateMakeFile(GenFfsList =3D DataPipe.Get("FfsComma= nd").get((PcdMa.MetaFile.File, PcdMa.Arch),[])) + PcdMa.CreateAsBuiltInf() + for w in auto_workers: + w.join() + rt =3D self.AutoGenMgr.Status + self.AutoGenMgr.kill() + self.AutoGenMgr.join() + self.AutoGenMgr =3D None + return rt =20 ## Load configuration # # This method will parse target.txt and get the build configurations. # @@ -1187,30 +1211,29 @@ class Build(): # @param CreateDepModuleCodeFile Flag used to indicate creating= code # for dependent modules/Libraries # @param CreateDepModuleMakeFile Flag used to indicate creating= makefile # for dependent modules/Libraries # - def _BuildPa(self, Target, AutoGenObject, CreateDepsCodeFile=3DTrue, C= reateDepsMakeFile=3DTrue, BuildModule=3DFalse, FfsCommand=3D{}): + def _BuildPa(self, Target, AutoGenObject, CreateDepsCodeFile=3DTrue, C= reateDepsMakeFile=3DTrue, BuildModule=3DFalse, FfsCommand=3DNone, PcdMaList= =3DNone): if AutoGenObject is None: return False - + if FfsCommand is None: + FfsCommand =3D {} # skip file generation for cleanxxx targets, run and fds target if Target not in ['clean', 'cleanlib', 'cleanall', 'run', 'fds']: # for target which must generate AutoGen code and makefile - if not self.SkipAutoGen or Target =3D=3D 'genc': - self.Progress.Start("Generating code") - AutoGenObject.CreateCodeFile(CreateDepsCodeFile) - self.Progress.Stop("done!") - if Target =3D=3D "genc": - return True + mqueue =3D mp.Queue() + for m in AutoGenObject.GetAllModuleInfo: + mqueue.put(m) =20 - if not self.SkipAutoGen or Target =3D=3D 'genmake': - self.Progress.Start("Generating makefile") - AutoGenObject.CreateMakeFile(CreateDepsMakeFile, FfsComman= d) - self.Progress.Stop("done!") - if Target =3D=3D "genmake": - return True + AutoGenObject.DataPipe.DataContainer =3D {"FfsCommand":FfsComm= and} + self.Progress.Start("Generating makefile and code") + data_pipe_file =3D os.path.join(self.WorkspaceDir, "GlobalVar_= %s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch)) + AutoGenObject.DataPipe.dump(data_pipe_file) + autogen_rt =3D self.StartAutoGen(mqueue, AutoGenObject.DataPip= e, self.SkipAutoGen, PcdMaList) + self.Progress.Stop("done!") + return autogen_rt else: # always recreate top/platform makefile when clean, just in ca= se of inconsistency AutoGenObject.CreateCodeFile(False) AutoGenObject.CreateMakeFile(False) =20 @@ -1712,11 +1735,11 @@ class Build(): continue if Ma.PcdIsDriver: Ma.PlatformInfo =3D Pa PcdMaList.append(Ma) self.BuildModules.append(Ma) - self._BuildPa(self.Target, Pa, FfsCommand=3DCmdListDic= t) + self._BuildPa(self.Target, Pa, FfsCommand=3DCmdListDic= t,PcdMaList=3DPcdMaList) =20 # Create MAP file when Load Fix Address is enabled. if self.Target in ["", "all", "fds"]: for Arch in Wa.ArchList: GlobalData.gGlobalDefines['ARCH'] =3D Arch @@ -1847,11 +1870,11 @@ class Build(): GlobalData.gModuleBuildTracking[Ma.Arch][M= a] =3D 'FAIL' self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) MakeStart =3D time.time() for Ma in self.BuildModules: if not Ma.IsBinaryModule: - Bt =3D BuildTask.New(ModuleMakeUnit(Ma, self.T= arget)) + Bt =3D BuildTask.New(ModuleMakeUnit(Ma, Pa.Bui= ldCommand,self.Target)) # Break build if any build thread has error if BuildTask.HasError(): # we need a full version of makefile for platf= orm ExitFlag.set() BuildTask.WaitForComplete() @@ -1977,18 +2000,19 @@ class Build(): self.LoadFixAddress =3D Wa.Platform.LoadFixAddress self.BuildReport.AddPlatformReport(Wa) Wa.CreateMakeFile(False) =20 # Add ffs build to makefile - CmdListDict =3D None + CmdListDict =3D {} if GlobalData.gEnableGenfdsMultiThread and self.Fdf: CmdListDict =3D self._GenFfsCmd(Wa.ArchList) =20 # multi-thread exit flag ExitFlag =3D threading.Event() ExitFlag.clear() self.AutoGenTime +=3D int(round((time.time() - WorkspaceAu= toGenTime))) + BuildModules =3D [] for Arch in Wa.ArchList: PcdMaList =3D [] AutoGenStart =3D time.time() GlobalData.gGlobalDefines['ARCH'] =3D Arch Pa =3D PlatformAutoGen(Wa, self.PlatformFile, BuildTar= get, ToolChain, Arch) @@ -2021,38 +2045,35 @@ class Build(): else: if GlobalData.gBinCacheSource: EdkLogger.quiet("cache miss: %s[%s]" % (Ma= .MetaFile.Path, Ma.Arch)) =20 # Not to auto-gen for targets 'clean', 'cleanlib',= 'cleanall', 'run', 'fds' - if self.Target not in ['clean', 'cleanlib', 'clean= all', 'run', 'fds']: # for target which must generate AutoGen code = and makefile - if not self.SkipAutoGen or self.Target =3D=3D = 'genc': - Ma.CreateCodeFile(True) - if self.Target =3D=3D "genc": - continue =20 - if not self.SkipAutoGen or self.Target =3D=3D = 'genmake': - if CmdListDict and self.Fdf and (Module.Fi= le, Arch) in CmdListDict: - Ma.CreateMakeFile(True, CmdListDict[Mo= dule.File, Arch]) - del CmdListDict[Module.File, Arch] - else: - Ma.CreateMakeFile(True) - if self.Target =3D=3D "genmake": - continue - self.BuildModules.append(Ma) + BuildModules.append(Ma) # Initialize all modules in tracking to 'FAIL' if Ma.Arch not in GlobalData.gModuleBuildTracking: GlobalData.gModuleBuildTracking[Ma.Arch] =3D d= ict() if Ma not in GlobalData.gModuleBuildTracking[Ma.Ar= ch]: GlobalData.gModuleBuildTracking[Ma.Arch][Ma] = =3D 'FAIL' + mqueue =3D mp.Queue() + for m in Pa.GetAllModuleInfo: + mqueue.put(m) + Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDic= t} + data_pipe_file =3D os.path.join(self.WorkspaceDir, "Gl= obalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) + Pa.DataPipe.dump(data_pipe_file) + autogen_rt =3D self.StartAutoGen(mqueue, Pa.DataPipe, = self.SkipAutoGen, PcdMaList) self.Progress.Stop("done!") self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) + if not autogen_rt: + return + for Arch in Wa.ArchList: MakeStart =3D time.time() - for Ma in self.BuildModules: + for Ma in BuildModules: # Generate build task for the module if not Ma.IsBinaryModule: - Bt =3D BuildTask.New(ModuleMakeUnit(Ma, self.T= arget)) + Bt =3D BuildTask.New(ModuleMakeUnit(Ma, Pa.Bui= ldCommand,self.Target)) # Break build if any build thread has error if BuildTask.HasError(): # we need a full version of makefile for platf= orm ExitFlag.set() BuildTask.WaitForComplete() @@ -2495,10 +2516,16 @@ def Main(): EdkLogger.quiet("(Python %s on %s) " % (platform.python_versio= n(), sys.platform) + traceback.format_exc()) else: EdkLogger.error(X.ToolName, FORMAT_INVALID, File=3DX.FileName,= Line=3DX.LineNumber, ExtraData=3DX.Message, RaiseError=3DFalse) ReturnCode =3D FORMAT_INVALID except KeyboardInterrupt: + if MyBuild is not None: + if MyBuild.AutoGenMgr: + MyBuild.AutoGenMgr.TerminateWorkers() + MyBuild.AutoGenMgr.kill() + # for multi-thread build exits safely + MyBuild.Relinquish() ReturnCode =3D ABORT_ERROR if Option is not None and Option.debug is not None: EdkLogger.quiet("(Python %s on %s) " % (platform.python_versio= n(), sys.platform) + traceback.format_exc()) except: if MyBuild is not None: --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43912): https://edk2.groups.io/g/devel/message/43912 Mute This Topic: https://groups.io/mt/32512456/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43913+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43913+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430490; cv=none; d=zoho.com; s=zohoarc; b=loJhUo1ya8IoAq3MjfTbjegLvvLr//swkxaOo8m0Q6LdJRkQS61jRP4+hJIBhuz88mcHE4IoW4rxj7uSLlxdPmoZVLHAIfivgLoDoGzKlfPolc4h2uYlhj2qPs2SNlIbbF16Hd0d1z4uXnrfDU38Yb0Sd3s7fy6MVH1MPgg8Ll8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430490; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=snBoEf4nHIU8llgbaFCQl4W+o2MfzhOvcnr/ATLPaEQ=; b=dDOufhwRSWFK0Aip8ILO6zGcbnYbIDH6pKzrZ50Y6KRsUtF2xQUSL1nEhsVVLZYInXGHwrLxmayIFY1EaSxPnpjAEsVse219m1Xb2EqElemrctcO7Ebo/75XHHFEhF4FqkkbeA6+Oa2XNrF5XUnj/3CuTd0YikIA60QbERg79pU= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43913+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430490992942.1223328897134; Wed, 17 Jul 2019 23:14:50 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:50 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544060" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:48 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 6/9] BaseTools: Add shared data for processes Date: Thu, 18 Jul 2019 14:14:20 +0800 Message-Id: <20190718061423.30612-7-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430490; bh=fPaF/Lam3tE2GttAPm35zvvX5Xsg/kDXkkdWhljQnBg=; h=Cc:Date:From:Reply-To:Subject:To; b=gehvw+ooSlhCdTwHB3bVntjLaTt4Xirso1CQt/bGCslf/aWwTjo+5Zp/PiGPu4701O2 h4w4erP1lE7PO/2NMwIr8CpM9Zoy6SDGdLzcGqTIPZHNFaliw/VgUbq8W9DvycZWbQKhp WmllqD22216cM8H3zBfbigV8tZrTwhFPCFc= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 Add shared data for autogen processes. Cc: Liming Gao Signed-off-by: Bob Feng --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 3 ++- BaseTools/Source/Python/build/build.py | 10 ++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index a0415f0d3420..150de0891c6e 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -52,18 +52,19 @@ class AutoGenManager(threading.Thread): for w in self.autogen_workers: if w.is_alive(): w.terminate() =20 class AutoGenWorkerInProcess(mp.Process): - def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data): mp.Process.__init__(self) self.module_queue =3D module_queue self.data_pipe_file_path =3Ddata_pipe_file_path self.data_pipe =3D None self.feedback_q =3D feedback_q self.PlatformMetaFileSet =3D {} self.file_lock =3D file_lock + self.share_data =3D share_data def GetPlatformMetaFile(self,filepath,root): try: return self.PlatformMetaFileSet[(filepath,root)] except: self.PlatformMetaFileSet[(filepath,root)] =3D filepath diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index b4916a32171a..46aed15bd9d1 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -50,10 +50,11 @@ from GenPatchPcdTable.GenPatchPcdTable import PeImageCl= ass,parsePcdInfoFromMapFi from PatchPcdValue.PatchPcdValue import PatchBinaryFile =20 import Common.GlobalData as GlobalData from GenFds.GenFds import GenFds, GenFdsApi import multiprocessing as mp +from multiprocessing import Manager =20 # Version and Copyright VersionNumber =3D "0.60" + ' ' + gBUILD_VERSION __version__ =3D "%prog Version " + VersionNumber __copyright__ =3D "Copyright (c) 2007 - 2018, Intel Corporation All right= s reserved." @@ -825,16 +826,17 @@ class Build(): self.InitBuild() =20 self.AutoGenMgr =3D None EdkLogger.info("") os.chdir(self.WorkspaceDir) - def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList): + self.share_data =3D Manager().dict() + def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_dat= a): if SkipAutoGen: return feedback_q =3D mp.Queue() file_lock =3D mp.Lock() - auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock) for _ in range(mp.cpu_count()//2)] + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock,share_data) for _ in range(mp.cpu_count()//2)] self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q) self.AutoGenMgr.start() for w in auto_workers: w.start() if PcdMaList is not None: @@ -1227,11 +1229,11 @@ class Build(): =20 AutoGenObject.DataPipe.DataContainer =3D {"FfsCommand":FfsComm= and} self.Progress.Start("Generating makefile and code") data_pipe_file =3D os.path.join(self.WorkspaceDir, "GlobalVar_= %s_%s.bin" % (str(AutoGenObject.Guid),AutoGenObject.Arch)) AutoGenObject.DataPipe.dump(data_pipe_file) - autogen_rt =3D self.StartAutoGen(mqueue, AutoGenObject.DataPip= e, self.SkipAutoGen, PcdMaList) + autogen_rt =3D self.StartAutoGen(mqueue, AutoGenObject.DataPip= e, self.SkipAutoGen, PcdMaList,self.share_data) self.Progress.Stop("done!") return autogen_rt else: # always recreate top/platform makefile when clean, just in ca= se of inconsistency AutoGenObject.CreateCodeFile(False) @@ -2059,11 +2061,11 @@ class Build(): for m in Pa.GetAllModuleInfo: mqueue.put(m) Pa.DataPipe.DataContainer =3D {"FfsCommand":CmdListDic= t} data_pipe_file =3D os.path.join(self.WorkspaceDir, "Gl= obalVar_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) - autogen_rt =3D self.StartAutoGen(mqueue, Pa.DataPipe, = self.SkipAutoGen, PcdMaList) + autogen_rt =3D self.StartAutoGen(mqueue, Pa.DataPipe, = self.SkipAutoGen, PcdMaList,self.share_data) self.Progress.Stop("done!") self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) if not autogen_rt: return for Arch in Wa.ArchList: --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43913): https://edk2.groups.io/g/devel/message/43913 Mute This Topic: https://groups.io/mt/32512457/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43914+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43914+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430492; cv=none; d=zoho.com; s=zohoarc; b=hl/ZH8MEY5ShHEbI5dnoXBNmZDmp073UJHt/pFwBLYoAeUBlSLmg26JO+RRKnsGo+b/JdRsHUoV4brSKasYEp+9awiyTfQfpkteNg9Eb1S6vcph+5FJGuMgPcdwLRSFLFRAWoPE+BYHf2daAP004Zu2cxXbpPQjWNFGgziaJHbE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430492; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=mbP5QhAnuYQPrbmDpqevlT0SclUSS3vgK5wnabO9094=; b=hdfNZNO2+xT0qWUQ9QHp8hJYjLSYSf+9SBq5o98uSAKc4hdQCjOqqGWrWaUh10GZmS3/h0z3tZx/+1QunP9D22cx5RWIh2l7/PpjEEMMKJsIWTFE1JBEGQL291gIDWvurQ5YtgGqoflZ5lPpp4eFnTN+Dy70G9hCNwLX/azyrnM= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43914+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 15634304928074.3393717601167054; Wed, 17 Jul 2019 23:14:52 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:52 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544066" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:50 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 7/9] BaseTools: Add LogAgent to support multiple process Autogen Date: Thu, 18 Jul 2019 14:14:21 +0800 Message-Id: <20190718061423.30612-8-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430492; bh=Nl4a0qkSLWgAXwp9Ghxjku7UmfbnqPXPweeM3LJN074=; h=Cc:Date:From:Reply-To:Subject:To; b=RKYqjob0tTWFc0HPYeYibkgW7Hk2hhuZEeaufR6HSkQktc71dUrc6yKUTTfT7mxDSUw DufRZZKS9pKEhOlrwopRuMUKDHi5Hd9UEOXPUmfj3wKwbGgGbb8VOfH7fvBvfawlf2s7X C3+MQyXwFo2shJgWGYdVYNkjpK+IxBadbn4= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 AutoGen processes race the logfile. To resolve this issue, this patch create a LogAgent thread in main process to write the log content to console or file, Other process will send the log content to the LogAgent. Cc: Liming Gao Signed-off-by: Bob Feng --- .../Source/Python/AutoGen/AutoGenWorker.py | 86 +++++++++++++++---- BaseTools/Source/Python/AutoGen/DataPipe.py | 2 +- BaseTools/Source/Python/Common/EdkLogger.py | 33 ++++++- BaseTools/Source/Python/build/build.py | 27 ++++-- 4 files changed, 120 insertions(+), 28 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 150de0891c6e..19d1cfac39fd 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -19,52 +19,111 @@ from Workspace.WorkspaceDatabase import BuildDB import time from queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe +import logging + +class LogAgent(threading.Thread): + def __init__(self,log_q,log_level,log_file=3DNone): + super(LogAgent,self).__init__() + self.log_q =3D log_q + self.log_level =3D log_level + self.log_file =3D log_file + def InitLogger(self): + # For DEBUG level (All DEBUG_0~9 are applicable) + self._DebugLogger_agent =3D logging.getLogger("tool_debug_agent") + _DebugFormatter =3D logging.Formatter("[%(asctime)s.%(msecs)d]: %(= message)s", datefmt=3D"%H:%M:%S") + self._DebugLogger_agent.setLevel(self.log_level) + _DebugChannel =3D logging.StreamHandler(sys.stdout) + _DebugChannel.setFormatter(_DebugFormatter) + self._DebugLogger_agent.addHandler(_DebugChannel) + + # For VERBOSE, INFO, WARN level + self._InfoLogger_agent =3D logging.getLogger("tool_info_agent") + _InfoFormatter =3D logging.Formatter("%(message)s") + self._InfoLogger_agent.setLevel(self.log_level) + _InfoChannel =3D logging.StreamHandler(sys.stdout) + _InfoChannel.setFormatter(_InfoFormatter) + self._InfoLogger_agent.addHandler(_InfoChannel) + + # For ERROR level + self._ErrorLogger_agent =3D logging.getLogger("tool_error_agent") + _ErrorFormatter =3D logging.Formatter("%(message)s") + self._ErrorLogger_agent.setLevel(self.log_level) + _ErrorCh =3D logging.StreamHandler(sys.stderr) + _ErrorCh.setFormatter(_ErrorFormatter) + self._ErrorLogger_agent.addHandler(_ErrorCh) + + if self.log_file: + if os.path.exists(self.log_file): + os.remove(self.log_file) + _Ch =3D logging.FileHandler(self.log_file) + _Ch.setFormatter(_DebugFormatter) + self._DebugLogger_agent.addHandler(_Ch) + + _Ch=3D logging.FileHandler(self.log_file) + _Ch.setFormatter(_InfoFormatter) + self._InfoLogger_agent.addHandler(_Ch) + + _Ch =3D logging.FileHandler(self.log_file) + _Ch.setFormatter(_ErrorFormatter) + self._ErrorLogger_agent.addHandler(_Ch) + + def run(self): + self.InitLogger() + while True: + log_message =3D self.log_q.get() + if log_message is None: + break + if log_message.name =3D=3D "tool_error": + self._ErrorLogger_agent.log(log_message.levelno,log_messag= e.getMessage()) + elif log_message.name =3D=3D "tool_info": + self._InfoLogger_agent.log(log_message.levelno,log_message= .getMessage()) + elif log_message.name =3D=3D "tool_debug": + self._DebugLogger_agent.log(log_message.levelno,log_messag= e.getMessage()) + else: + self._InfoLogger_agent.log(log_message.levelno,log_message= .getMessage()) + + def kill(self): + self.log_q.put(None) class AutoGenManager(threading.Thread): def __init__(self,autogen_workers, feedback_q): super(AutoGenManager,self).__init__() self.autogen_workers =3D autogen_workers self.feedback_q =3D feedback_q - self.terminate =3D False self.Status =3D True def run(self): try: while True: - if self.terminate: - break - if self.feedback_q.empty(): - time.sleep(1) - continue - badnews =3D self.feedback_q.get(False) - if badnews: - print(badnews) + badnews =3D self.feedback_q.get() + if badnews is None: self.Status =3D False self.TerminateWorkers() break except Exception: return =20 def kill(self): - self.terminate =3D True + self.feedback_q.put(None) =20 def TerminateWorkers(self): for w in self.autogen_workers: if w.is_alive(): w.terminate() =20 class AutoGenWorkerInProcess(mp.Process): - def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data,log_q): mp.Process.__init__(self) self.module_queue =3D module_queue self.data_pipe_file_path =3Ddata_pipe_file_path self.data_pipe =3D None self.feedback_q =3D feedback_q self.PlatformMetaFileSet =3D {} self.file_lock =3D file_lock self.share_data =3D share_data + self.log_q =3D log_q def GetPlatformMetaFile(self,filepath,root): try: return self.PlatformMetaFileSet[(filepath,root)] except: self.PlatformMetaFileSet[(filepath,root)] =3D filepath @@ -75,18 +134,15 @@ class AutoGenWorkerInProcess(mp.Process): with self.file_lock: if not os.path.exists(self.data_pipe_file_path): self.feedback_q.put(taskname + ":" + "load data pipe %= s failed." % self.data_pipe_file_path) self.data_pipe =3D MemoryDataPipe() self.data_pipe.load(self.data_pipe_file_path) - EdkLogger.Initialize() + EdkLogger.LogClientInitialize(self.log_q) loglevel =3D self.data_pipe.Get("LogLevel") if not loglevel: loglevel =3D EdkLogger.INFO EdkLogger.SetLevel(loglevel) - logfile =3D self.data_pipe.Get("LogFile") - if logfile: - EdkLogger.SetLogFile(logfile) target =3D self.data_pipe.Get("P_Info").get("Target") toolchain =3D self.data_pipe.Get("P_Info").get("ToolChain") archlist =3D self.data_pipe.Get("P_Info").get("ArchList") =20 active_p =3D self.data_pipe.Get("P_Info").get("ActivePlatform") diff --git a/BaseTools/Source/Python/AutoGen/DataPipe.py b/BaseTools/Source= /Python/AutoGen/DataPipe.py index 9478f41d481b..33d2b14c9add 100644 --- a/BaseTools/Source/Python/AutoGen/DataPipe.py +++ b/BaseTools/Source/Python/AutoGen/DataPipe.py @@ -145,9 +145,9 @@ class MemoryDataPipe(DataPipe): self.DataContainer =3D {"PackageList": [(dec.MetaFile,dec.Arch) fo= r dec in PlatformInfo.PackageList]} =20 self.DataContainer =3D {"GuidDict": PlatformInfo.Platform._GuidDic= t} =20 self.DataContainer =3D {"DatabasePath":GlobalData.gDatabasePath} + self.DataContainer =3D {"FdfParser": True if GlobalData.gFdfParser= else False} =20 self.DataContainer =3D {"LogLevel": EdkLogger.GetLevel()} - self.DataContainer =3D {"LogFile": GlobalData.gOptions.LogFile if = GlobalData.gOptions.LogFile is not None else ""} diff --git a/BaseTools/Source/Python/Common/EdkLogger.py b/BaseTools/Source= /Python/Common/EdkLogger.py index ae2070bebba3..f6a5e3b4daf9 100644 --- a/BaseTools/Source/Python/Common/EdkLogger.py +++ b/BaseTools/Source/Python/Common/EdkLogger.py @@ -8,10 +8,11 @@ ## Import modules from __future__ import absolute_import import Common.LongFilePathOs as os, sys, logging import traceback from .BuildToolError import * +import logging.handlers =20 ## Log level constants DEBUG_0 =3D 1 DEBUG_1 =3D 2 DEBUG_2 =3D 3 @@ -198,30 +199,30 @@ def error(ToolName, ErrorCode, Message=3DNone, File= =3DNone, Line=3DNone, ExtraData=3DNon =20 # Log information which should be always put out quiet =3D _ErrorLogger.error =20 ## Initialize log system -def Initialize(): +def LogClientInitialize(log_q): # # Since we use different format to log different levels of message int= o different # place (stdout or stderr), we have to use different "Logger" objects = to do this. # # For DEBUG level (All DEBUG_0~9 are applicable) _DebugLogger.setLevel(INFO) - _DebugChannel =3D logging.StreamHandler(sys.stdout) + _DebugChannel =3D logging.handlers.QueueHandler(log_q) _DebugChannel.setFormatter(_DebugFormatter) _DebugLogger.addHandler(_DebugChannel) =20 # For VERBOSE, INFO, WARN level _InfoLogger.setLevel(INFO) - _InfoChannel =3D logging.StreamHandler(sys.stdout) + _InfoChannel =3D logging.handlers.QueueHandler(log_q) _InfoChannel.setFormatter(_InfoFormatter) _InfoLogger.addHandler(_InfoChannel) =20 # For ERROR level _ErrorLogger.setLevel(INFO) - _ErrorCh =3D logging.StreamHandler(sys.stderr) + _ErrorCh =3D logging.handlers.QueueHandler(log_q) _ErrorCh.setFormatter(_ErrorFormatter) _ErrorLogger.addHandler(_ErrorCh) =20 ## Set log level # @@ -232,10 +233,34 @@ def SetLevel(Level): Level =3D INFO _DebugLogger.setLevel(Level) _InfoLogger.setLevel(Level) _ErrorLogger.setLevel(Level) =20 +## Initialize log system +def Initialize(): + # + # Since we use different format to log different levels of message int= o different + # place (stdout or stderr), we have to use different "Logger" objects = to do this. + # + # For DEBUG level (All DEBUG_0~9 are applicable) + _DebugLogger.setLevel(INFO) + _DebugChannel =3D logging.StreamHandler(sys.stdout) + _DebugChannel.setFormatter(_DebugFormatter) + _DebugLogger.addHandler(_DebugChannel) + + # For VERBOSE, INFO, WARN level + _InfoLogger.setLevel(INFO) + _InfoChannel =3D logging.StreamHandler(sys.stdout) + _InfoChannel.setFormatter(_InfoFormatter) + _InfoLogger.addHandler(_InfoChannel) + + # For ERROR level + _ErrorLogger.setLevel(INFO) + _ErrorCh =3D logging.StreamHandler(sys.stderr) + _ErrorCh.setFormatter(_ErrorFormatter) + _ErrorLogger.addHandler(_ErrorCh) + def InitializeForUnitTest(): Initialize() SetLevel(SILENT) =20 ## Get current log level diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 46aed15bd9d1..6ecb80e45ed2 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -28,11 +28,12 @@ from subprocess import Popen,PIPE from collections import OrderedDict, defaultdict from optparse import OptionParser from AutoGen.PlatformAutoGen import PlatformAutoGen from AutoGen.ModuleAutoGen import ModuleAutoGen from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen -from AutoGen.AutoGenWorker import AutoGenWorkerInProcess,AutoGenManager +from AutoGen.AutoGenWorker import AutoGenWorkerInProcess,AutoGenManager,\ + LogAgent from AutoGen import GenMake from Common import Misc as Utils =20 from Common.TargetTxtClassObject import TargetTxt from Common.ToolDefClassObject import ToolDef @@ -697,11 +698,11 @@ class Build(): # # @param Target The build command target, one of gSupp= ortedTarget # @param WorkspaceDir The directory of workspace # @param BuildOptions Build options passed from command line # - def __init__(self, Target, WorkspaceDir, BuildOptions): + def __init__(self, Target, WorkspaceDir, BuildOptions,log_q): self.WorkspaceDir =3D WorkspaceDir self.Target =3D Target self.PlatformFile =3D BuildOptions.PlatformFile self.ModuleFile =3D BuildOptions.ModuleFile self.ArchList =3D BuildOptions.TargetArch @@ -827,16 +828,17 @@ class Build(): =20 self.AutoGenMgr =3D None EdkLogger.info("") os.chdir(self.WorkspaceDir) self.share_data =3D Manager().dict() + self.log_q =3D log_q def StartAutoGen(self,mqueue, DataPipe,SkipAutoGen,PcdMaList,share_dat= a): if SkipAutoGen: return feedback_q =3D mp.Queue() file_lock =3D mp.Lock() - auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock,share_data) for _ in range(mp.cpu_count()//2)] + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_file= ,feedback_q,file_lock,share_data,self.log_q) for _ in range(mp.cpu_count()/= /2)] self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q) self.AutoGenMgr.start() for w in auto_workers: w.start() if PcdMaList is not None: @@ -2393,35 +2395,42 @@ def MyOptionParser(): # @retval 1 Tool failed # def Main(): StartTime =3D time.time() =20 + # + # Create a log Queue + # + LogQ =3D mp.Queue() # Initialize log system - EdkLogger.Initialize() + EdkLogger.LogClientInitialize(LogQ) GlobalData.gCommand =3D sys.argv[1:] # # Parse the options and args # (Option, Target) =3D MyOptionParser() GlobalData.gOptions =3D Option GlobalData.gCaseInsensitive =3D Option.CaseInsensitive =20 # Set log level + LogLevel =3D EdkLogger.INFO if Option.verbose is not None: EdkLogger.SetLevel(EdkLogger.VERBOSE) + LogLevel =3D EdkLogger.VERBOSE elif Option.quiet is not None: EdkLogger.SetLevel(EdkLogger.QUIET) + LogLevel =3D EdkLogger.QUIET elif Option.debug is not None: EdkLogger.SetLevel(Option.debug + 1) + LogLevel =3D Option.debug + 1 else: EdkLogger.SetLevel(EdkLogger.INFO) =20 - if Option.LogFile is not None: - EdkLogger.SetLogFile(Option.LogFile) - if Option.WarningAsError =3D=3D True: EdkLogger.SetWarningAsError() + Log_Agent =3D LogAgent(LogQ,LogLevel,Option.LogFile) + Log_Agent.start() =20 if platform.platform().find("Windows") >=3D 0: GlobalData.gIsWindows =3D True else: GlobalData.gIsWindows =3D False @@ -2491,11 +2500,11 @@ def Main(): EdkLogger.error("build", ErrorCode, ExtraData=3DErrorInfo) =20 if Option.Flag is not None and Option.Flag not in ['-c', '-s']: EdkLogger.error("build", OPTION_VALUE_INVALID, "UNI flag must = be one of -c or -s") =20 - MyBuild =3D Build(Target, Workspace, Option) + MyBuild =3D Build(Target, Workspace, Option,LogQ) GlobalData.gCommandLineDefines['ARCH'] =3D ' '.join(MyBuild.ArchLi= st) if not (MyBuild.LaunchPrebuildFlag and os.path.exists(MyBuild.Plat= formBuildPath)): MyBuild.Launch() =20 # @@ -2577,10 +2586,12 @@ def Main(): =20 EdkLogger.SetLevel(EdkLogger.QUIET) EdkLogger.quiet("\n- %s -" % Conclusion) EdkLogger.quiet(time.strftime("Build end time: %H:%M:%S, %b.%d %Y", ti= me.localtime())) EdkLogger.quiet("Build total time: %s\n" % BuildDurationStr) + Log_Agent.kill() + Log_Agent.join() return ReturnCode =20 if __name__ =3D=3D '__main__': r =3D Main() ## 0-127 is a safe return range, and 1 is a standard default error --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43914): https://edk2.groups.io/g/devel/message/43914 Mute This Topic: https://groups.io/mt/32512458/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43915+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43915+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430493; cv=none; d=zoho.com; s=zohoarc; b=i6oVh4bBnSoIIcyBvNxe2hL3ql+Ih57RXzV+O6FVqPTMlX31Qt3SoyVBjkOKQWNiArNqZ/DzxaqQWZMOuA39qgy3aiKRKG7N9HXoLM8ofkpbHJEQ2blwneQQ9lNihe8+NgGzdjp03uYSSFezlULsm+DwqNyx2TGhxseSUaDAkGc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430493; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=haj/2USzP4BP9sQymtmAHr/Q/IJHz2QyoUvSRG5o4XA=; b=lZLy7o4zcDW76s8qBj++l5TJN5Z6Kd3uWRm8eWuz2Kz77h8UNqpDUUttjCz0ZcFeorpohu6LMGwZWqucwImPTa2kWUHiYm59hlilsh/z7Gg9nF8cEmtcytyTUgiARvjqcNp0MpuaNuaohzjSMJqNeJnBecUwz2Y3DSxIZ/OdY+0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43915+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430493932427.8339805602907; Wed, 17 Jul 2019 23:14:53 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:53 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544074" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:51 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao Subject: [edk2-devel] [Patch 8/9] BaseTools: Move BuildOption parser out of build.py Date: Thu, 18 Jul 2019 14:14:22 +0800 Message-Id: <20190718061423.30612-9-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430493; bh=3ofGjP7vzV+B8b7wZy91ISfqESY20fqtjVm5SOYIRRc=; h=Cc:Date:From:Reply-To:Subject:To; b=FszKrBkrnMjjPV/xsV0/nyQ2Imm72TIKFr2fYVo/yHJTOLq0QUJRm5YSd46vmw8aNEH KAp4X9VW77uSmFgbTtLtyQh41gkfAwkZmaN0sPZDLA/oXzEllFL60XW9KgU1gcx4avkWN uyGOZrc9B1NN2138DRfMHFOycLNmEu5CWpo= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 Build tool supports user to specify the conf folder. To make the build options be evaluated at the beginning of launching build, extract the buildoption function from build.py to a new .py file. Signed-off-by: Bob Feng Cc: Liming Gao --- .../Python/Common/TargetTxtClassObject.py | 28 ++++- BaseTools/Source/Python/build/build.py | 108 +----------------- BaseTools/Source/Python/build/buildoptions.py | 92 +++++++++++++++ 3 files changed, 121 insertions(+), 107 deletions(-) create mode 100644 BaseTools/Source/Python/build/buildoptions.py diff --git a/BaseTools/Source/Python/Common/TargetTxtClassObject.py b/BaseT= ools/Source/Python/Common/TargetTxtClassObject.py index 79a5acc01074..16cc75ccb7c8 100644 --- a/BaseTools/Source/Python/Common/TargetTxtClassObject.py +++ b/BaseTools/Source/Python/Common/TargetTxtClassObject.py @@ -8,16 +8,19 @@ ## # Import Modules # from __future__ import print_function from __future__ import absolute_import +from buildoptions import BuildOption,BuildTarget +import Common.GlobalData as GlobalData import Common.LongFilePathOs as os from . import EdkLogger from . import DataType from .BuildToolError import * -from . import GlobalData + from Common.LongFilePathSupport import OpenLongFilePath as open +from Common.MultipleWorkspace import MultipleWorkspace as mws =20 gDefaultTargetTxtFile =3D "target.txt" =20 ## TargetTxtClassObject # @@ -139,16 +142,33 @@ class TargetTxtClassObject(object): # # @param ConfDir: Conf dir # # @retval Target An instance of TargetTxtClassObject() with loaded target.= txt # -def TargetTxtDict(ConfDir): +def TargetTxtDict(): Target =3D TargetTxtClassObject() - Target.LoadTargetTxtFile(os.path.normpath(os.path.join(ConfDir, gDefau= ltTargetTxtFile))) + if BuildOption.ConfDirectory: + # Get alternate Conf location, if it is absolute, then just use th= e absolute directory name + ConfDirectoryPath =3D os.path.normpath(BuildOption.ConfDirectory) + + if not os.path.isabs(ConfDirectoryPath): + # Since alternate directory name is not absolute, the alternat= e directory is located within the WORKSPACE + # This also handles someone specifying the Conf directory in t= he workspace. Using --conf=3DConf + ConfDirectoryPath =3D mws.join(os.environ["WORKSPACE"], ConfDi= rectoryPath) + else: + if "CONF_PATH" in os.environ: + ConfDirectoryPath =3D os.path.normcase(os.path.normpath(os.env= iron["CONF_PATH"])) + else: + # Get standard WORKSPACE/Conf use the absolute path to the WOR= KSPACE/Conf + ConfDirectoryPath =3D mws.join(os.environ["WORKSPACE"], 'Conf') + GlobalData.gConfDirectory =3D ConfDirectoryPath + targettxt =3D os.path.normpath(os.path.join(ConfDirectoryPath, gDefaul= tTargetTxtFile)) + if os.path.exists(targettxt): + Target.LoadTargetTxtFile(targettxt) return Target =20 -TargetTxt =3D TargetTxtDict(os.path.join(os.getenv("WORKSPACE"),"Conf")) +TargetTxt =3D TargetTxtDict() =20 ## # # This acts like the main() function for the script, unless it is 'import'= ed into another # script. diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 6ecb80e45ed2..33fe548de8f4 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -24,11 +24,11 @@ import traceback import multiprocessing from threading import Thread,Event,BoundedSemaphore import threading from subprocess import Popen,PIPE from collections import OrderedDict, defaultdict -from optparse import OptionParser +from buildoptions import BuildOption,BuildTarget from AutoGen.PlatformAutoGen import PlatformAutoGen from AutoGen.ModuleAutoGen import ModuleAutoGen from AutoGen.WorkspaceAutoGen import WorkspaceAutoGen from AutoGen.AutoGenWorker import AutoGenWorkerInProcess,AutoGenManager,\ LogAgent @@ -41,11 +41,11 @@ from Common.Misc import PathClass,SaveFileOnChange,Remo= veDirectory from Common.StringUtils import NormPath from Common.MultipleWorkspace import MultipleWorkspace as mws from Common.BuildToolError import * from Common.DataType import * import Common.EdkLogger as EdkLogger -from Common.BuildVersion import gBUILD_VERSION + from Workspace.WorkspaceDatabase import BuildDB =20 from BuildReport import BuildReport from GenPatchPcdTable.GenPatchPcdTable import PeImageClass,parsePcdInfoFro= mMapFile from PatchPcdValue.PatchPcdValue import PatchBinaryFile @@ -53,14 +53,10 @@ from PatchPcdValue.PatchPcdValue import PatchBinaryFile import Common.GlobalData as GlobalData from GenFds.GenFds import GenFds, GenFdsApi import multiprocessing as mp from multiprocessing import Manager =20 -# Version and Copyright -VersionNumber =3D "0.60" + ' ' + gBUILD_VERSION -__version__ =3D "%prog Version " + VersionNumber -__copyright__ =3D "Copyright (c) 2007 - 2018, Intel Corporation All right= s reserved." =20 ## standard targets of build command gSupportedTarget =3D ['all', 'genc', 'genmake', 'modules', 'libraries', 'f= ds', 'clean', 'cleanall', 'cleanlib', 'run'] =20 ## build configuration file @@ -761,26 +757,11 @@ class Build(): GlobalData.gBinCacheDest =3D BinCacheDest else: if GlobalData.gBinCacheDest is not None: EdkLogger.error("build", OPTION_VALUE_INVALID, ExtraData= =3D"Invalid value of option --binary-destination.") =20 - if self.ConfDirectory: - # Get alternate Conf location, if it is absolute, then just us= e the absolute directory name - ConfDirectoryPath =3D os.path.normpath(self.ConfDirectory) - - if not os.path.isabs(ConfDirectoryPath): - # Since alternate directory name is not absolute, the alte= rnate directory is located within the WORKSPACE - # This also handles someone specifying the Conf directory = in the workspace. Using --conf=3DConf - ConfDirectoryPath =3D mws.join(self.WorkspaceDir, ConfDire= ctoryPath) - else: - if "CONF_PATH" in os.environ: - ConfDirectoryPath =3D os.path.normcase(os.path.normpath(os= .environ["CONF_PATH"])) - else: - # Get standard WORKSPACE/Conf use the absolute path to the= WORKSPACE/Conf - ConfDirectoryPath =3D mws.join(self.WorkspaceDir, 'Conf') - GlobalData.gConfDirectory =3D ConfDirectoryPath - GlobalData.gDatabasePath =3D os.path.normpath(os.path.join(ConfDir= ectoryPath, GlobalData.gDatabasePath)) + GlobalData.gDatabasePath =3D os.path.normpath(os.path.join(GlobalD= ata.gConfDirectory, GlobalData.gDatabasePath)) =20 self.Db =3D BuildDB self.BuildDatabase =3D self.Db.BuildObject self.Platform =3D None self.ToolChainFamily =3D None @@ -2290,17 +2271,11 @@ def ParseDefines(DefineList=3D[]): DefineDict[DefineTokenList[0]] =3D "TRUE" else: DefineDict[DefineTokenList[0]] =3D DefineTokenList[1].stri= p() return DefineDict =20 -gParamCheck =3D [] -def SingleCheckCallback(option, opt_str, value, parser): - if option not in gParamCheck: - setattr(parser.values, option.dest, value) - gParamCheck.append(option) - else: - parser.error("Option %s only allows one instance in command line!"= % option) + =20 def LogBuildTime(Time): if Time: TimeDurStr =3D '' TimeDur =3D time.gmtime(Time) @@ -2310,83 +2285,10 @@ def LogBuildTime(Time): TimeDurStr =3D time.strftime("%H:%M:%S", TimeDur) return TimeDurStr else: return None =20 -## Parse command line options -# -# Using standard Python module optparse to parse command line option of th= is tool. -# -# @retval Opt A optparse.Values object containing the parsed options -# @retval Args Target of build command -# -def MyOptionParser(): - Parser =3D OptionParser(description=3D__copyright__, version=3D__versi= on__, prog=3D"build.exe", usage=3D"%prog [options] [all|fds|genc|genmake|cl= ean|cleanall|cleanlib|modules|libraries|run]") - Parser.add_option("-a", "--arch", action=3D"append", type=3D"choice", = choices=3D['IA32', 'X64', 'EBC', 'ARM', 'AARCH64'], dest=3D"TargetArch", - help=3D"ARCHS is one of list: IA32, X64, ARM, AARCH64 or EBC, whic= h overrides target.txt's TARGET_ARCH definition. To specify more archs, ple= ase repeat this option.") - Parser.add_option("-p", "--platform", action=3D"callback", type=3D"str= ing", dest=3D"PlatformFile", callback=3DSingleCheckCallback, - help=3D"Build the platform specified by the DSC file name argument= , overriding target.txt's ACTIVE_PLATFORM definition.") - Parser.add_option("-m", "--module", action=3D"callback", type=3D"strin= g", dest=3D"ModuleFile", callback=3DSingleCheckCallback, - help=3D"Build the module specified by the INF file name argument.") - Parser.add_option("-b", "--buildtarget", type=3D"string", dest=3D"Buil= dTarget", help=3D"Using the TARGET to build the platform, overriding target= .txt's TARGET definition.", - action=3D"append") - Parser.add_option("-t", "--tagname", action=3D"append", type=3D"string= ", dest=3D"ToolChain", - help=3D"Using the Tool Chain Tagname to build the platform, overri= ding target.txt's TOOL_CHAIN_TAG definition.") - Parser.add_option("-x", "--sku-id", action=3D"callback", type=3D"strin= g", dest=3D"SkuId", callback=3DSingleCheckCallback, - help=3D"Using this name of SKU ID to build the platform, overridin= g SKUID_IDENTIFIER in DSC file.") - - Parser.add_option("-n", action=3D"callback", type=3D"int", dest=3D"Thr= eadNumber", callback=3DSingleCheckCallback, - help=3D"Build the platform using multi-threaded compiler. The valu= e overrides target.txt's MAX_CONCURRENT_THREAD_NUMBER. When value is set to= 0, tool automatically detect number of "\ - "processor threads, set value to 1 means disable multi-thread= build, and set value to more than 1 means user specify the threads number = to build.") - - Parser.add_option("-f", "--fdf", action=3D"callback", type=3D"string",= dest=3D"FdfFile", callback=3DSingleCheckCallback, - help=3D"The name of the FDF file to use, which overrides the setti= ng in the DSC file.") - Parser.add_option("-r", "--rom-image", action=3D"append", type=3D"stri= ng", dest=3D"RomImage", default=3D[], - help=3D"The name of FD to be generated. The name must be from [FD]= section in FDF file.") - Parser.add_option("-i", "--fv-image", action=3D"append", type=3D"strin= g", dest=3D"FvImage", default=3D[], - help=3D"The name of FV to be generated. The name must be from [FV]= section in FDF file.") - Parser.add_option("-C", "--capsule-image", action=3D"append", type=3D"= string", dest=3D"CapName", default=3D[], - help=3D"The name of Capsule to be generated. The name must be from= [Capsule] section in FDF file.") - Parser.add_option("-u", "--skip-autogen", action=3D"store_true", dest= =3D"SkipAutoGen", help=3D"Skip AutoGen step.") - Parser.add_option("-e", "--re-parse", action=3D"store_true", dest=3D"R= eparse", help=3D"Re-parse all meta-data files.") - - Parser.add_option("-c", "--case-insensitive", action=3D"store_true", d= est=3D"CaseInsensitive", default=3DFalse, help=3D"Don't check case of file = name.") - - Parser.add_option("-w", "--warning-as-error", action=3D"store_true", d= est=3D"WarningAsError", help=3D"Treat warning in tools as error.") - Parser.add_option("-j", "--log", action=3D"store", dest=3D"LogFile", h= elp=3D"Put log in specified file as well as on console.") - - Parser.add_option("-s", "--silent", action=3D"store_true", type=3DNone= , dest=3D"SilentMode", - help=3D"Make use of silent mode of (n)make.") - Parser.add_option("-q", "--quiet", action=3D"store_true", type=3DNone,= help=3D"Disable all messages except FATAL ERRORS.") - Parser.add_option("-v", "--verbose", action=3D"store_true", type=3DNon= e, help=3D"Turn on verbose output with informational messages printed, "\ - = "including library instances selected, final dependency expression, "\ - = "and warning messages, etc.") - Parser.add_option("-d", "--debug", action=3D"store", type=3D"int", hel= p=3D"Enable debug messages at specified level.") - Parser.add_option("-D", "--define", action=3D"append", type=3D"string"= , dest=3D"Macros", help=3D"Macro: \"Name [=3D Value]\".") - - Parser.add_option("-y", "--report-file", action=3D"store", dest=3D"Rep= ortFile", help=3D"Create/overwrite the report to the specified filename.") - Parser.add_option("-Y", "--report-type", action=3D"append", type=3D"ch= oice", choices=3D['PCD', 'LIBRARY', 'FLASH', 'DEPEX', 'BUILD_FLAGS', 'FIXED= _ADDRESS', 'HASH', 'EXECUTION_ORDER'], dest=3D"ReportType", default=3D[], - help=3D"Flags that control the type of build report to generate. = Must be one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS, FIXED_ADDRESS, HA= SH, EXECUTION_ORDER]. "\ - "To specify more than one flag, repeat this option on the com= mand line and the default flag set is [PCD, LIBRARY, FLASH, DEPEX, HASH, BU= ILD_FLAGS, FIXED_ADDRESS]") - Parser.add_option("-F", "--flag", action=3D"store", type=3D"string", d= est=3D"Flag", - help=3D"Specify the specific option to parse EDK UNI file. Must be= one of: [-c, -s]. -c is for EDK framework UNI file, and -s is for EDK UEFI= UNI file. "\ - "This option can also be specified by setting *_*_*_BUILD_FLA= GS in [BuildOptions] section of platform DSC. If they are both specified, t= his value "\ - "will override the setting in [BuildOptions] section of platf= orm DSC.") - Parser.add_option("-N", "--no-cache", action=3D"store_true", dest=3D"D= isableCache", default=3DFalse, help=3D"Disable build cache mechanism") - Parser.add_option("--conf", action=3D"store", type=3D"string", dest=3D= "ConfDirectory", help=3D"Specify the customized Conf directory.") - Parser.add_option("--check-usage", action=3D"store_true", dest=3D"Chec= kUsage", default=3DFalse, help=3D"Check usage content of entries listed in = INF file.") - Parser.add_option("--ignore-sources", action=3D"store_true", dest=3D"I= gnoreSources", default=3DFalse, help=3D"Focus to a binary build and ignore = all source files") - Parser.add_option("--pcd", action=3D"append", dest=3D"OptionPcd", help= =3D"Set PCD value by command line. Format: \"PcdName=3DValue\" ") - Parser.add_option("-l", "--cmd-len", action=3D"store", type=3D"int", d= est=3D"CommandLength", help=3D"Specify the maximum line length of build com= mand. Default is 4096.") - Parser.add_option("--hash", action=3D"store_true", dest=3D"UseHashCach= e", default=3DFalse, help=3D"Enable hash-based caching during build process= .") - Parser.add_option("--binary-destination", action=3D"store", type=3D"st= ring", dest=3D"BinCacheDest", help=3D"Generate a cache of binary files in t= he specified directory.") - Parser.add_option("--binary-source", action=3D"store", type=3D"string"= , dest=3D"BinCacheSource", help=3D"Consume a cache of binary files from the= specified directory.") - Parser.add_option("--genfds-multi-thread", action=3D"store_true", dest= =3D"GenfdsMultiThread", default=3DFalse, help=3D"Enable GenFds multi thread= to generate ffs file.") - Parser.add_option("--disable-include-path-check", action=3D"store_true= ", dest=3D"DisableIncludePathCheck", default=3DFalse, help=3D"Disable the i= nclude path check for outside of package.") - (Opt, Args) =3D Parser.parse_args() - return (Opt, Args) - ## Tool entrance method # # This method mainly dispatch specific methods per the command line option= s. # If no error found, return zero value so the caller of this tool can know # if it's executed successfully or not. @@ -2405,11 +2307,11 @@ def Main(): EdkLogger.LogClientInitialize(LogQ) GlobalData.gCommand =3D sys.argv[1:] # # Parse the options and args # - (Option, Target) =3D MyOptionParser() + Option, Target =3D BuildOption, BuildTarget GlobalData.gOptions =3D Option GlobalData.gCaseInsensitive =3D Option.CaseInsensitive =20 # Set log level LogLevel =3D EdkLogger.INFO diff --git a/BaseTools/Source/Python/build/buildoptions.py b/BaseTools/Sour= ce/Python/build/buildoptions.py new file mode 100644 index 000000000000..7161aa66f23e --- /dev/null +++ b/BaseTools/Source/Python/build/buildoptions.py @@ -0,0 +1,92 @@ +## @file +# build a platform or a module +# +# Copyright (c) 2014, Hewlett-Packard Development Company, L.P.
+# Copyright (c) 2007 - 2019, Intel Corporation. All rights reserved.
+# Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.
+# +# SPDX-License-Identifier: BSD-2-Clause-Patent +# + +# Version and Copyright +from Common.BuildVersion import gBUILD_VERSION +from optparse import OptionParser +VersionNumber =3D "0.60" + ' ' + gBUILD_VERSION +__version__ =3D "%prog Version " + VersionNumber +__copyright__ =3D "Copyright (c) 2007 - 2018, Intel Corporation All right= s reserved." + +gParamCheck =3D [] +def SingleCheckCallback(option, opt_str, value, parser): + if option not in gParamCheck: + setattr(parser.values, option.dest, value) + gParamCheck.append(option) + else: + parser.error("Option %s only allows one instance in command line!"= % option) + +def MyOptionParser(): + Parser =3D OptionParser(description=3D__copyright__, version=3D__versi= on__, prog=3D"build.exe", usage=3D"%prog [options] [all|fds|genc|genmake|cl= ean|cleanall|cleanlib|modules|libraries|run]") + Parser.add_option("-a", "--arch", action=3D"append", type=3D"choice", = choices=3D['IA32', 'X64', 'EBC', 'ARM', 'AARCH64'], dest=3D"TargetArch", + help=3D"ARCHS is one of list: IA32, X64, ARM, AARCH64 or EBC, whic= h overrides target.txt's TARGET_ARCH definition. To specify more archs, ple= ase repeat this option.") + Parser.add_option("-p", "--platform", action=3D"callback", type=3D"str= ing", dest=3D"PlatformFile", callback=3DSingleCheckCallback, + help=3D"Build the platform specified by the DSC file name argument= , overriding target.txt's ACTIVE_PLATFORM definition.") + Parser.add_option("-m", "--module", action=3D"callback", type=3D"strin= g", dest=3D"ModuleFile", callback=3DSingleCheckCallback, + help=3D"Build the module specified by the INF file name argument.") + Parser.add_option("-b", "--buildtarget", type=3D"string", dest=3D"Buil= dTarget", help=3D"Using the TARGET to build the platform, overriding target= .txt's TARGET definition.", + action=3D"append") + Parser.add_option("-t", "--tagname", action=3D"append", type=3D"string= ", dest=3D"ToolChain", + help=3D"Using the Tool Chain Tagname to build the platform, overri= ding target.txt's TOOL_CHAIN_TAG definition.") + Parser.add_option("-x", "--sku-id", action=3D"callback", type=3D"strin= g", dest=3D"SkuId", callback=3DSingleCheckCallback, + help=3D"Using this name of SKU ID to build the platform, overridin= g SKUID_IDENTIFIER in DSC file.") + + Parser.add_option("-n", action=3D"callback", type=3D"int", dest=3D"Thr= eadNumber", callback=3DSingleCheckCallback, + help=3D"Build the platform using multi-threaded compiler. The valu= e overrides target.txt's MAX_CONCURRENT_THREAD_NUMBER. When value is set to= 0, tool automatically detect number of "\ + "processor threads, set value to 1 means disable multi-thread= build, and set value to more than 1 means user specify the threads number = to build.") + + Parser.add_option("-f", "--fdf", action=3D"callback", type=3D"string",= dest=3D"FdfFile", callback=3DSingleCheckCallback, + help=3D"The name of the FDF file to use, which overrides the setti= ng in the DSC file.") + Parser.add_option("-r", "--rom-image", action=3D"append", type=3D"stri= ng", dest=3D"RomImage", default=3D[], + help=3D"The name of FD to be generated. The name must be from [FD]= section in FDF file.") + Parser.add_option("-i", "--fv-image", action=3D"append", type=3D"strin= g", dest=3D"FvImage", default=3D[], + help=3D"The name of FV to be generated. The name must be from [FV]= section in FDF file.") + Parser.add_option("-C", "--capsule-image", action=3D"append", type=3D"= string", dest=3D"CapName", default=3D[], + help=3D"The name of Capsule to be generated. The name must be from= [Capsule] section in FDF file.") + Parser.add_option("-u", "--skip-autogen", action=3D"store_true", dest= =3D"SkipAutoGen", help=3D"Skip AutoGen step.") + Parser.add_option("-e", "--re-parse", action=3D"store_true", dest=3D"R= eparse", help=3D"Re-parse all meta-data files.") + + Parser.add_option("-c", "--case-insensitive", action=3D"store_true", d= est=3D"CaseInsensitive", default=3DFalse, help=3D"Don't check case of file = name.") + + Parser.add_option("-w", "--warning-as-error", action=3D"store_true", d= est=3D"WarningAsError", help=3D"Treat warning in tools as error.") + Parser.add_option("-j", "--log", action=3D"store", dest=3D"LogFile", h= elp=3D"Put log in specified file as well as on console.") + + Parser.add_option("-s", "--silent", action=3D"store_true", type=3DNone= , dest=3D"SilentMode", + help=3D"Make use of silent mode of (n)make.") + Parser.add_option("-q", "--quiet", action=3D"store_true", type=3DNone,= help=3D"Disable all messages except FATAL ERRORS.") + Parser.add_option("-v", "--verbose", action=3D"store_true", type=3DNon= e, help=3D"Turn on verbose output with informational messages printed, "\ + = "including library instances selected, final dependency expression, "\ + = "and warning messages, etc.") + Parser.add_option("-d", "--debug", action=3D"store", type=3D"int", hel= p=3D"Enable debug messages at specified level.") + Parser.add_option("-D", "--define", action=3D"append", type=3D"string"= , dest=3D"Macros", help=3D"Macro: \"Name [=3D Value]\".") + + Parser.add_option("-y", "--report-file", action=3D"store", dest=3D"Rep= ortFile", help=3D"Create/overwrite the report to the specified filename.") + Parser.add_option("-Y", "--report-type", action=3D"append", type=3D"ch= oice", choices=3D['PCD', 'LIBRARY', 'FLASH', 'DEPEX', 'BUILD_FLAGS', 'FIXED= _ADDRESS', 'HASH', 'EXECUTION_ORDER'], dest=3D"ReportType", default=3D[], + help=3D"Flags that control the type of build report to generate. = Must be one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS, FIXED_ADDRESS, HA= SH, EXECUTION_ORDER]. "\ + "To specify more than one flag, repeat this option on the com= mand line and the default flag set is [PCD, LIBRARY, FLASH, DEPEX, HASH, BU= ILD_FLAGS, FIXED_ADDRESS]") + Parser.add_option("-F", "--flag", action=3D"store", type=3D"string", d= est=3D"Flag", + help=3D"Specify the specific option to parse EDK UNI file. Must be= one of: [-c, -s]. -c is for EDK framework UNI file, and -s is for EDK UEFI= UNI file. "\ + "This option can also be specified by setting *_*_*_BUILD_FLA= GS in [BuildOptions] section of platform DSC. If they are both specified, t= his value "\ + "will override the setting in [BuildOptions] section of platf= orm DSC.") + Parser.add_option("-N", "--no-cache", action=3D"store_true", dest=3D"D= isableCache", default=3DFalse, help=3D"Disable build cache mechanism") + Parser.add_option("--conf", action=3D"store", type=3D"string", dest=3D= "ConfDirectory", help=3D"Specify the customized Conf directory.") + Parser.add_option("--check-usage", action=3D"store_true", dest=3D"Chec= kUsage", default=3DFalse, help=3D"Check usage content of entries listed in = INF file.") + Parser.add_option("--ignore-sources", action=3D"store_true", dest=3D"I= gnoreSources", default=3DFalse, help=3D"Focus to a binary build and ignore = all source files") + Parser.add_option("--pcd", action=3D"append", dest=3D"OptionPcd", help= =3D"Set PCD value by command line. Format: \"PcdName=3DValue\" ") + Parser.add_option("-l", "--cmd-len", action=3D"store", type=3D"int", d= est=3D"CommandLength", help=3D"Specify the maximum line length of build com= mand. Default is 4096.") + Parser.add_option("--hash", action=3D"store_true", dest=3D"UseHashCach= e", default=3DFalse, help=3D"Enable hash-based caching during build process= .") + Parser.add_option("--binary-destination", action=3D"store", type=3D"st= ring", dest=3D"BinCacheDest", help=3D"Generate a cache of binary files in t= he specified directory.") + Parser.add_option("--binary-source", action=3D"store", type=3D"string"= , dest=3D"BinCacheSource", help=3D"Consume a cache of binary files from the= specified directory.") + Parser.add_option("--genfds-multi-thread", action=3D"store_true", dest= =3D"GenfdsMultiThread", default=3DFalse, help=3D"Enable GenFds multi thread= to generate ffs file.") + Parser.add_option("--disable-include-path-check", action=3D"store_true= ", dest=3D"DisableIncludePathCheck", default=3DFalse, help=3D"Disable the i= nclude path check for outside of package.") + (Opt, Args) =3D Parser.parse_args() + return (Opt, Args) + +BuildOption, BuildTarget =3D MyOptionParser() --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43915): https://edk2.groups.io/g/devel/message/43915 Mute This Topic: https://groups.io/mt/32512459/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Sun Apr 28 12:36:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+43916+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43916+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1563430495; cv=none; d=zoho.com; s=zohoarc; b=UhgvBr6mgUJZ08zxihtxRr2Xzr9TkOXN44n1SnM3GUFaDwU0uBk/nN6aX1vW11duvqXoq8bzM6H1AzoK+zCIbfXVF0fVSvN1iGSKzEK7A1ZuC4kTAPXa5Miv9J9A4AvmheVQnKmGihIgpe/HVaAskyTPa+FtkeBEf8Ql4qH0SNA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1563430495; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=L+HDUVWtn6oRcFTpcRi+yj+TM9Mj0AgSMhFYvR2Wszs=; b=bHu3zzFZWN5Q0PxM8t2ijmcHsCdbOCf/uavLsgGgkMxECGkyfPUg2En0BEsK8opClm4zmJ7rttOgGUlPKfV4CQlE64tJTVVFTsSHU04SequQUeiLKnCQdnVttfbMSM6zgFtx1RKwLdbRbayLkicSieDpPDRmukh1nVvPmrjR9Uc= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+43916+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1563430495686210.86663870571067; Wed, 17 Jul 2019 23:14:55 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Wed, 17 Jul 2019 23:14:54 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jul 2019 23:14:53 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,276,1559545200"; d="scan'208";a="319544081" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 17 Jul 2019 23:14:52 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 9/9] BaseTools: Add the support for python 2 Date: Thu, 18 Jul 2019 14:14:23 +0800 Message-Id: <20190718061423.30612-10-bob.c.feng@intel.com> In-Reply-To: <20190718061423.30612-1-bob.c.feng@intel.com> References: <20190718061423.30612-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1563430495; bh=ypjbJLJ/Z61LC2Jtx02dH3d5g5e6TZHTNux5l/0LPpY=; h=Cc:Date:From:Reply-To:Subject:To; b=KfqQLDnjFU1aAIngWKsvt7phpKszdqhptK6cxpRcwu9FnIIzR8M5l9OTYxVA8J2WAh3 pla8pdKu2echJZEZSQ0gxwargA0QA77pzHz7zlqRj+8x8T2K2EH84UHYMxv6SBIUWJXLL MU5G5qwxluKJkCQoYu9X7t79N7GvenAsgUw= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 python3 change the module name of Queue to queue. python3 add a new log handler of QueueHandler. This patch is to make Multiple process AutoGen feature work for python2 Cc: Liming Gao Signed-off-by: Bob Feng --- .../Source/Python/AutoGen/AutoGenWorker.py | 5 +- BaseTools/Source/Python/Common/EdkLogger.py | 92 ++++++++++++++++++- 2 files changed, 92 insertions(+), 5 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 19d1cfac39fd..233a921e74fe 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -15,11 +15,14 @@ import Common.EdkLogger as EdkLogger import os from Common.MultipleWorkspace import MultipleWorkspace as mws from AutoGen.AutoGen import AutoGen from Workspace.WorkspaceDatabase import BuildDB import time -from queue import Empty +try: + from queue import Empty +except: + from Queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe import logging =20 diff --git a/BaseTools/Source/Python/Common/EdkLogger.py b/BaseTools/Source= /Python/Common/EdkLogger.py index f6a5e3b4daf9..15fd1458a95a 100644 --- a/BaseTools/Source/Python/Common/EdkLogger.py +++ b/BaseTools/Source/Python/Common/EdkLogger.py @@ -3,16 +3,100 @@ # # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.
# SPDX-License-Identifier: BSD-2-Clause-Patent # =20 +# Copyright 2001-2016 by Vinay Sajip. All Rights Reserved. +# +# Permission to use, copy, modify, and distribute this software and its +# documentation for any purpose and without fee is hereby granted, +# provided that the above copyright notice appear in all copies and that +# both that copyright notice and this permission notice appear in +# supporting documentation, and that the name of Vinay Sajip +# not be used in advertising or publicity pertaining to distribution +# of the software without specific, written prior permission. +# VINAY SAJIP DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLU= DING +# ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL +# VINAY SAJIP BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES= OR +# ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHET= HER +# IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING O= UT +# OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +# This copyright is for QueueHandler. + ## Import modules from __future__ import absolute_import import Common.LongFilePathOs as os, sys, logging import traceback from .BuildToolError import * -import logging.handlers +try: + from logging.handlers import QueueHandler +except: + class QueueHandler(logging.Handler): + """ + This handler sends events to a queue. Typically, it would be used = together + with a multiprocessing Queue to centralise logging to file in one = process + (in a multi-process application), so as to avoid file write conten= tion + between processes. + + This code is new in Python 3.2, but this class can be copy pasted = into + user code for use with earlier Python versions. + """ + + def __init__(self, queue): + """ + Initialise an instance, using the passed queue. + """ + logging.Handler.__init__(self) + self.queue =3D queue + + def enqueue(self, record): + """ + Enqueue a record. + + The base implementation uses put_nowait. You may want to overr= ide + this method if you want to use blocking, timeouts or custom qu= eue + implementations. + """ + self.queue.put_nowait(record) + + def prepare(self, record): + """ + Prepares a record for queuing. The object returned by this met= hod is + enqueued. + + The base implementation formats the record to merge the message + and arguments, and removes unpickleable items from the record + in-place. + + You might want to override this method if you want to convert + the record to a dict or JSON string, or send a modified copy + of the record while leaving the original intact. + """ + # The format operation gets traceback text into record.exc_text + # (if there's exception data), and also returns the formatted + # message. We can then use this to replace the original + # msg + args, as these might be unpickleable. We also zap the + # exc_info and exc_text attributes, as they are no longer + # needed and, if not None, will typically not be pickleable. + msg =3D self.format(record) + record.message =3D msg + record.msg =3D msg + record.args =3D None + record.exc_info =3D None + record.exc_text =3D None + return record + + def emit(self, record): + """ + Emit a record. + + Writes the LogRecord to the queue, preparing it for pickling f= irst. + """ + try: + self.enqueue(self.prepare(record)) + except Exception: + self.handleError(record) =20 ## Log level constants DEBUG_0 =3D 1 DEBUG_1 =3D 2 DEBUG_2 =3D 3 @@ -206,23 +290,23 @@ def LogClientInitialize(log_q): # Since we use different format to log different levels of message int= o different # place (stdout or stderr), we have to use different "Logger" objects = to do this. # # For DEBUG level (All DEBUG_0~9 are applicable) _DebugLogger.setLevel(INFO) - _DebugChannel =3D logging.handlers.QueueHandler(log_q) + _DebugChannel =3D QueueHandler(log_q) _DebugChannel.setFormatter(_DebugFormatter) _DebugLogger.addHandler(_DebugChannel) =20 # For VERBOSE, INFO, WARN level _InfoLogger.setLevel(INFO) - _InfoChannel =3D logging.handlers.QueueHandler(log_q) + _InfoChannel =3D QueueHandler(log_q) _InfoChannel.setFormatter(_InfoFormatter) _InfoLogger.addHandler(_InfoChannel) =20 # For ERROR level _ErrorLogger.setLevel(INFO) - _ErrorCh =3D logging.handlers.QueueHandler(log_q) + _ErrorCh =3D QueueHandler(log_q) _ErrorCh.setFormatter(_ErrorFormatter) _ErrorLogger.addHandler(_ErrorCh) =20 ## Set log level # --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#43916): https://edk2.groups.io/g/devel/message/43916 Mute This Topic: https://groups.io/mt/32512460/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-