From nobody Tue Apr 16 21:10:25 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+40613+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+40613+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1557856600; cv=none; d=zoho.com; s=zohoarc; b=hSo/dV9K8uEKkTNYpIxZmFDKTGsnYBcYMEw2BAWqYEotOkRyFRIsbTphLFQuw1HjEHs8JVoZglgmV5cf4jnelg/xPy75pD/uUVAa8HbIJBHkFQvHgjBthkrArkufYAtNj06NpnF5S044I2dVDI1gj3/LFTQ/GEDwNZGH8Fbz4FA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1557856600; h=Content-Transfer-Encoding:Cc:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To:ARC-Authentication-Results; bh=8LPUzkIOjkXVVWO5rG6UHDnMi88ZkDCvB23E73OYjo8=; b=bUAp/CmHVvvLsVb1G9WCQhrbdiuUO0cIkzYKKHdXKdqBkhTmq9DhJF/X2xmZHrrxaDnniPn3lexHYQcVF+RenxK9lTG7pXXoL0txQpxCWOXXjrM2RdMnHrosCrSKlvqZWOm9o49aDzrqamQdhekCZEAmb3YUl5WweqGXb9I7NQA= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+40613+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1557856600330667.1315536138886; Tue, 14 May 2019 10:56:40 -0700 (PDT) Return-Path: X-Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by groups.io with SMTP; Tue, 14 May 2019 10:56:39 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 May 2019 10:56:23 -0700 X-ExtLoop1: 1 X-Received: from rodrigu3-desk.amr.corp.intel.com ([10.7.163.75]) by orsmga001.jf.intel.com with ESMTP; 14 May 2019 10:56:22 -0700 From: "Christian Rodriguez" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Yonghong Zhu Subject: [edk2-devel] [PATCH] BaseTools: Library hashing fix and optimization for --hash feature Date: Tue, 14 May 2019 10:56:18 -0700 Message-Id: <20190514175618.10868-1-christian.rodriguez@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,christian.rodriguez@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1557856599; bh=DSdSLCFdRaL3oRZV0Homufs/K3Q4RRQGaqqQCxOfuzU=; h=Cc:Date:From:Reply-To:Subject:To; b=wlyjzkFFGnoRs4OEE7OsjAmoJt3nJYoduVLCTsuBn2K004NH8oAXWq9+I1RyS3YHbDD eOO2ulK6Wt18DwS3LmziJn4VAiNejxa6k39QpaNoBjvJiIu9wF7tvDtyiQhSC+j2mCNHv HWz3mNH2ZtRu6NE53GXUJ+z9YaeVwszU0Og= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1788 Library hashing is now supported by the --hash feature. The --hash feature implementation assumed that the hashing could be done in place once per module, but that isn't true for libraries due to the fact that they are built as dependencies. So on a clean build, we now generate the .hash after the library dependencies are complete. Added early escape as optimization, if hash already exists in memory. Signed-off-by: Christian Rodriguez Cc: Bob Feng Cc: Liming Gao Cc: Yonghong Zhu --- BaseTools/Source/Python/AutoGen/AutoGen.py | 52 ++++++++++++++++++++++++= +++++++++++++++------------- BaseTools/Source/Python/Common/GlobalData.py | 6 ++++++ BaseTools/Source/Python/build/build.py | 7 ++++++- 3 files changed, 51 insertions(+), 14 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index 31721a6f9f..cf92d07be1 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -4093,13 +4093,20 @@ class ModuleAutoGen(AutoGen): return RetVal =20 def GenModuleHash(self): + # Initialize a dictionary for each arch type if self.Arch not in GlobalData.gModuleHash: GlobalData.gModuleHash[self.Arch] =3D {} - if self.Name in GlobalData.gModuleHash[self.Arch] and GlobalData.g= BinCacheSource and self.AttemptModuleCacheCopy(): - return False + + # Early exit if module or library has been hashed and is in memory + if self.Name in GlobalData.gModuleHash[self.Arch]: + return GlobalData.gModuleHash[self.Arch][self.Name].encode('ut= f-8') + + # Initialze hash object m =3D hashlib.md5() + # Add Platform level hash m.update(GlobalData.gPlatformHash.encode('utf-8')) + # Add Package level hash if self.DependentPackageList: for Pkg in sorted(self.DependentPackageList, key=3Dlambda x: x= .PackageName): @@ -4118,6 +4125,7 @@ class ModuleAutoGen(AutoGen): Content =3D f.read() f.close() m.update(Content) + # Add Module's source files if self.SourceFileList: for File in sorted(self.SourceFileList, key=3Dlambda x: str(x)= ): @@ -4126,27 +4134,45 @@ class ModuleAutoGen(AutoGen): f.close() m.update(Content) =20 - ModuleHashFile =3D path.join(self.BuildDir, self.Name + ".hash") - if self.Name not in GlobalData.gModuleHash[self.Arch]: - GlobalData.gModuleHash[self.Arch][self.Name] =3D m.hexdigest() - if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy(): - return False - return SaveFileOnChange(ModuleHashFile, m.hexdigest(), False) + GlobalData.gModuleHash[self.Arch][self.Name] =3D m.hexdigest() + + return GlobalData.gModuleHash[self.Arch][self.Name].encode('utf-8') =20 ## Decide whether we can skip the ModuleAutoGen process def CanSkipbyHash(self): + # Hashing feature is off + if not GlobalData.gUseHashCache: + return False + + # Initialize a dictionary for each arch type + if self.Arch not in GlobalData.gBuildHashSkipTracking: + GlobalData.gBuildHashSkipTracking[self.Arch] =3D dict() + # If library or Module is binary do not skip by hash if self.IsBinaryModule: return False + # .inc is contains binary information so do not skip by hash as we= ll for f_ext in self.SourceFileList: if '.inc' in str(f_ext): return False - if GlobalData.gUseHashCache: - # If there is a valid hash or function generated a valid hash;= function will return False - # and the statement below will return True - return not self.GenModuleHash() - return False + + # Use Cache, if exists and if Module has a copy in cache + if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy(): + return True + + # Early exit for libraries that haven't yet finished building + HashFile =3D path.join(self.BuildDir, self.Name + ".hash") + if self.IsLibrary and not os.path.exists(HashFile): + return False + + # Return a Boolean based on if can skip by hash, either from memor= y or from IO. + if self.Name not in GlobalData.gBuildHashSkipTracking[self.Arch]: + # If hashes are the same, SaveFileOnChange() will return False. + GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] =3D no= t SaveFileOnChange(HashFile, self.GenModuleHash(), False) + return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] + else: + return GlobalData.gBuildHashSkipTracking[self.Arch][self.Name] =20 ## Decide whether we can skip the ModuleAutoGen process # If any source file is newer than the module than we cannot skip diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Sourc= e/Python/Common/GlobalData.py index 79f23c892d..95e28a988f 100644 --- a/BaseTools/Source/Python/Common/GlobalData.py +++ b/BaseTools/Source/Python/Common/GlobalData.py @@ -112,3 +112,9 @@ gSikpAutoGenCache =3D set() # Dictionary for tracking Module build status as success or failure # False -> Fail : True -> Success gModuleBuildTracking =3D dict() + +# Dictionary of booleans that dictate whether a module or +# library can be skiped +# Top Dict: Key: Arch Type Value: Dictionary +# Second Dict: Key: Module\Library Name Value: True\False +gBuildHashSkipTracking =3D dict() diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 7271570d29..97a4dc8ee2 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -593,7 +593,7 @@ class BuildTask: # def AddDependency(self, Dependency): for Dep in Dependency: - if not Dep.BuildObject.IsBinaryModule: + if not Dep.BuildObject.IsBinaryModule and not Dep.BuildObject.= CanSkipbyHash(): self.DependencyList.append(BuildTask.New(Dep)) # BuildT= ask list =20 ## The thread wrapper of LaunchCommand function @@ -605,6 +605,11 @@ class BuildTask: try: self.BuildItem.BuildObject.BuildTime =3D LaunchCommand(Command= , WorkingDir) self.CompleteFlag =3D True + + # Run hash operation post dependency, to account for libs + if GlobalData.gUseHashCache and self.BuildItem.BuildObject.IsL= ibrary: + HashFile =3D path.join(self.BuildItem.BuildObject.BuildDir= , self.BuildItem.BuildObject.Name + ".hash") + SaveFileOnChange(HashFile, self.BuildItem.BuildObject.GenM= oduleHash(), False) except: # # TRICK: hide the output of threads left running, so that the = user can --=20 2.19.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#40613): https://edk2.groups.io/g/devel/message/40613 Mute This Topic: https://groups.io/mt/31621303/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-