From nobody Tue May 14 03:14:31 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+113934+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+113934+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1705478116; cv=none; d=zohomail.com; s=zohoarc; b=M/9axAQa8c2IfTuLykzYggKQ8eUZwYlb62bIdV+gls9mexv4/IPShvwAIAzfWDzNGeuWAI31k/8Qh8Zt38PmerpP9si5SDryrk2/gHGK+cal3y9bO4gGc7Cw21njf2aBFcPXoCBYHJwaZhtpUgFykWmxgsFU9zFOd+jcs9CvOOY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1705478116; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:Sender:Subject:Subject:To:To:Message-Id; bh=xvkUh1lXeFjhhsM1eLUkWa7MCPuezJ6Hndo19LZ7P1k=; b=ar9eJHRm/zxfTxZDolHaUyVrOrKBepwFEyrQXeEaHFTTucmixwKOPKa4IfAmtOaYIMXAfPLXrckJ6p3Wd5cPOYrn5k06Ri2ETK4zaqYtgmSfBh71T/YT+2dvkInBq0QUFG/pc/zUJ6gzF9SKpevmJQnMvTIBeQf0Ex+cYFYrz3g= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+113934+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1705478116359141.62798250420235; Tue, 16 Jan 2024 23:55:16 -0800 (PST) Return-Path: DKIM-Signature: a=rsa-sha256; bh=hEMcVe24nFeAYbRfPgsrZBFqudPvolG5ZYcWenzZLDE=; c=relaxed/simple; d=groups.io; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1705478115; v=1; b=wwID/0Hw7p+ywSPSk14LMw4cA4aV3+ZulSPRevmnhher7oFteOdmMIue4HjcyQbLvD5oHuHF JGZ50dyYQfNM3ZJ6nGxY49YWfcuLzoGCKQdtBke8b4/zk1e2qzLhrWmdrmqdMnG/EtLtJuDrBPO nZqTZh+g1RJYimXUc5rJ8Aao= X-Received: by 127.0.0.2 with SMTP id Zyb6YY1788612xXdYRcDBHuq; Tue, 16 Jan 2024 23:55:15 -0800 X-Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by mx.groups.io with SMTP id smtpd.web11.4736.1705478114818279987 for ; Tue, 16 Jan 2024 23:55:15 -0800 X-IronPort-AV: E=McAfee;i="6600,9927,10955"; a="390551783" X-IronPort-AV: E=Sophos;i="6.05,200,1701158400"; d="scan'208";a="390551783" X-Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jan 2024 23:55:14 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.05,200,1701158400"; d="scan'208";a="32747665" X-Received: from ashrafal-mobl4.gar.corp.intel.com ([10.213.110.62]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jan 2024 23:55:10 -0800 From: "Ashraf Ali S" To: devel@edk2.groups.io Cc: Ashraf Ali S , Yuwei Chen , Rebecca Cran , Liming Gao , Bob Feng , Amy Chan , Sai Chaganty , Digant H Solanki Subject: [edk2-devel] [PATCH] BaseTools: Optimize GenerateByteArrayValue and CollectPlatformGuids APIs Date: Wed, 17 Jan 2024 13:24:59 +0530 Message-Id: MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,ashraf.ali.s@intel.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: aGWQ8pAHUuLb9cdRDyzZmmGEx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1705478117830100003 Content-Type: text/plain; charset="utf-8" During the Incremental build GenerateByteArrayValue used to generate the ByteArrayValue even when there is no change in the PCD/VPDs. which is time consuming API based on the number of PCD/VPDs and SKU IDs. The optimization is that GenerateByteArrayValue is used to store the PcdRecordList in a JSON file for each of the arch. and during the Incremental build this API will check if there is any change in the PCD /VPDs then rest of the flow remains the same. if there is no change then it will return the provious build data. Flow: during the 1st build PcdRecordList.json is not exists, PcdRecordList will be dumped to json file. and it will copy the output.txt as well. Note: as the output.txt are different for different Arch, so it will be stored in the Arch folder. During the Incremental build check if there is any change in PCD/VPD. if there is a change in VPD/PCD then recreate the PcdRecordList.json. and rest of the flow remains same. if there is no change in VPD/PCD read the output.txt and return the data Cc: Yuwei Chen Cc: Rebecca Cran Cc: Liming Gao Cc: Bob Feng Cc: Amy Chan Cc: Sai Chaganty Cc: Digant H Solanki Signed-off-by: Ashraf Ali S --- .../Source/Python/AutoGen/WorkspaceAutoGen.py | 16 ++--- .../Source/Python/Workspace/DscBuildData.py | 72 +++++++++++++++---- 2 files changed, 64 insertions(+), 24 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py b/BaseTool= s/Source/Python/AutoGen/WorkspaceAutoGen.py index 160e3a3cd3..eec9280c8e 100644 --- a/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py @@ -160,22 +160,18 @@ class WorkspaceAutoGen(AutoGen): =20 def CollectPlatformGuids(self): oriInfList =3D [] - oriPkgSet =3D set() - PlatformPkg =3D set() + pkgSet =3D set() for Arch in self.ArchList: Platform =3D self.BuildDatabase[self.MetaFile, Arch, self.Buil= dTarget, self.ToolChain] oriInfList =3D Platform.Modules for ModuleFile in oriInfList: ModuleData =3D self.BuildDatabase[ModuleFile, Platform._Ar= ch, Platform._Target, Platform._Toolchain] - oriPkgSet.update(ModuleData.Packages) - for Pkg in oriPkgSet: - Guids =3D Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(ModuleData.Packages) if Platform.Packages: - PlatformPkg.update(Platform.Packages) - for Pkg in PlatformPkg: - Guids =3D Pkg.Guids - GlobalData.gGuidDict.update(Guids) + pkgSet.update(Platform.Packages) + for Pkg in pkgSet: + Guids =3D Pkg.Guids + GlobalData.gGuidDict.update(Guids) =20 @cached_property def FdfProfile(self): diff --git a/BaseTools/Source/Python/Workspace/DscBuildData.py b/BaseTools/= Source/Python/Workspace/DscBuildData.py index 4768099343..740b8e22be 100644 --- a/BaseTools/Source/Python/Workspace/DscBuildData.py +++ b/BaseTools/Source/Python/Workspace/DscBuildData.py @@ -37,6 +37,8 @@ from functools import reduce from Common.Misc import SaveFileOnChange from Workspace.BuildClassObject import PlatformBuildClassObject, Structure= Pcd, PcdClassObject, ModuleBuildClassObject from collections import OrderedDict, defaultdict +import json +import shutil =20 def _IsFieldValueAnArray (Value): Value =3D Value.strip() @@ -56,6 +58,7 @@ def _IsFieldValueAnArray (Value): =20 PcdValueInitName =3D 'PcdValueInit' PcdValueCommonName =3D 'PcdValueCommon' +PcdRecordListName =3D 'PcdRecordList.json' =20 PcdMainCHeader =3D ''' /** @@ -1599,7 +1602,7 @@ class DscBuildData(PlatformBuildClassObject): S_pcd_set =3D DscBuildData.OverrideByComm(S_pcd_set) =20 # Create a tool to caculate structure pcd value - Str_Pcd_Values =3D self.GenerateByteArrayValue(S_pcd_set) + Str_Pcd_Values =3D self.GenerateByteArrayValue(S_pcd_set, RecordLi= st) =20 if Str_Pcd_Values: for (skuname, StoreName, PcdGuid, PcdName, PcdValue) in Str_Pc= d_Values: @@ -2750,12 +2753,61 @@ class DscBuildData(PlatformBuildClassObject): ccflags.add(item) i +=3D1 return ccflags - def GenerateByteArrayValue (self, StructuredPcds): + + def GetStructurePcdSet (self, OutputValueFile): + if not os.path.isfile(OutputValueFile): + EdkLogger.error("GetStructurePcdSet", FILE_NOT_FOUND, "Output.= txt doesn't exist", ExtraData=3DOutputValueFile) + return [] + File =3D open (OutputValueFile, 'r') + FileBuffer =3D File.readlines() + File.close() + + #start update structure pcd final value + StructurePcdSet =3D [] + for Pcd in FileBuffer: + PcdValue =3D Pcd.split ('|') + PcdInfo =3D PcdValue[0].split ('.') + StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], Pc= dInfo[3], PcdValue[2].strip())) + return StructurePcdSet + + def GenerateByteArrayValue (self, StructuredPcds, PcdRecordList): # # Generate/Compile/Run C application to determine if there are any= flexible array members # if not StructuredPcds: return + # + # If the output path doesn't exists then create it + # + if not os.path.exists(self.OutputPath): + os.makedirs(self.OutputPath) + + PcdRecordListPath =3D os.path.join(self.OutputPath, self._Arch, Pc= dRecordListName) + PcdRecordOutputValueFile =3D os.path.join(self.OutputPath, self._A= rch, 'Output.txt') + + if not os.path.exists(os.path.dirname(PcdRecordListPath)): + os.makedirs(os.path.dirname(PcdRecordListPath)) + # + # Check if the PcdRecordList.json exists or not + # if exits then it might be a incremental build then check if the = PcdRecord list has been changed or not. + # if changed then proceed further, if not changed then return the = stored data from earlier build + # + if os.path.isfile(PcdRecordListPath): + with open(PcdRecordListPath, 'r') as file: + file_content_str =3D file.read() + if file_content_str: + # Use json.loads to convert the string back to a list + file_content =3D json.loads(file_content_str) + # Check if all PcdRecordList in record_set are present= in file_content + # and if there are no extra PcdRecordList in file_cont= ent + if set(map(tuple, PcdRecordList)) =3D=3D set(map(tuple= , file_content)): + return self.GetStructurePcdSet(PcdRecordOutputValu= eFile) + # + # 1st build, create the PcdRecordList.json + # update the record as PCD Input has been changed in incremental b= uild + # + with open(PcdRecordListPath, 'w') as file: + json.dump(PcdRecordList, file) =20 InitByteValue =3D "" CApp =3D PcdMainCHeader @@ -2832,8 +2884,6 @@ class DscBuildData(PlatformBuildClassObject): =20 CApp =3D CApp + PcdMainCEntry + '\n' =20 - if not os.path.exists(self.OutputPath): - os.makedirs(self.OutputPath) CAppBaseFileName =3D os.path.join(self.OutputPath, PcdValueInitNam= e) SaveFileOnChange(CAppBaseFileName + '.c', CApp, False) =20 @@ -3042,17 +3092,11 @@ class DscBuildData(PlatformBuildClassObject): if returncode !=3D 0: EdkLogger.warn('Build', COMMAND_FAILURE, 'Can not collect = output from command: %s\n%s\n%s\n' % (Command, StdOut, StdErr)) =20 - #start update structure pcd final value - File =3D open (OutputValueFile, 'r') - FileBuffer =3D File.readlines() - File.close() + # Copy output file for each Arch + shutil.copyfile(OutputValueFile, PcdRecordOutputValueFile) =20 - StructurePcdSet =3D [] - for Pcd in FileBuffer: - PcdValue =3D Pcd.split ('|') - PcdInfo =3D PcdValue[0].split ('.') - StructurePcdSet.append((PcdInfo[0], PcdInfo[1], PcdInfo[2], Pc= dInfo[3], PcdValue[2].strip())) - return StructurePcdSet + # return structure pcd final value + return self.GetStructurePcdSet(OutputValueFile) =20 @staticmethod def NeedUpdateOutput(OutputFile, ValueCFile, StructureInput): --=20 2.39.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#113934): https://edk2.groups.io/g/devel/message/113934 Mute This Topic: https://groups.io/mt/103781798/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-