From nobody Mon Feb 9 19:52:29 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45639+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45639+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565810195; cv=none; d=zoho.com; s=zohoarc; b=HlhG7mKG2xgu8hZllfWu+KApgCXkj5fEBLiNgfFFxBJIzYmb7V3VVKk0xmub+UN+s1XgOQK6wXS+3NPKzIkl1YbBLuGV1n5FeTGa4EiW38rrGNrfSfuCK38wDyxalpay19msr/y08xOtD4c7bMLrYr/vsBQnuSE9qVYjmisZWjg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565810195; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=wA+nHv+1uG0JaC7cIC3THcIWxu2z8Yc/wySiYuJoNEg=; b=MMnn1yqi1Ak75IofN4gMWG+MrnFTvdAlo0zpPY8DZLhspHIwmpH7Kp9lWecjLVEkbhuE5FCm9+i5GK7HmdLxdmxcZRgMBMxSrko9iGOlV0zTb2ydxYhvTIM2ziYYwUBShkLE3ZH1lE3poMIQumA4IVxdRehl+I6tTb1EplTZ000= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45639+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565810195368692.9738011065546; Wed, 14 Aug 2019 12:16:35 -0700 (PDT) Return-Path: X-Received: from mga07.intel.com (mga07.intel.com []) by groups.io with SMTP; Wed, 14 Aug 2019 12:16:34 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Aug 2019 12:16:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,386,1559545200"; d="scan'208";a="176664080" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.213.161]) by fmsmga008.fm.intel.com with ESMTP; 14 Aug 2019 12:16:31 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, lersek@redhat.com, leif.lindholm@linaro.org, afish@apple.com, stephano.cetola@intel.com, michael.d.kinney@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v5 5/5] BaseTools: Improve the file saving and copying reliability Date: Thu, 15 Aug 2019 03:16:06 +0800 Message-Id: <20190814191606.16532-6-steven.shi@intel.com> In-Reply-To: <20190814191606.16532-1-steven.shi@intel.com> References: <20190814191606.16532-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565810194; bh=ajWUHKq+9vnrOM5LqkfqfRXamyDaLCxze1SmwHxHi3A=; h=Cc:Date:From:Reply-To:Subject:To; b=xQgBLfZ+ZvA/d9zVdI5MwnJttTzuP3PTfEIF+GogbcBbl+I7ZaXUbFRFKbNe8JVpfqx x0x6KE25wQSEJpuTAZ89f1hPLsE0AsFFY7xMW5JpAmAONXKpTADt7EOZvh10H48jxnDSV IWJrl237hfqh5Q83ZLRJ4lNAvPMrzq2Mgf8= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D2079 The Basetool CopyFileOnChange() and SaveFileOnChange() functions might raise the IOError occasionally when build in Windows with multi-process and build cache enabled. The CopyFileOnChange() and SaveFileOnChange() might be invoked in multiple sub-processes simultaneously, and this patch adds global locks to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGenWorker.py | 4 +++- BaseTools/Source/Python/AutoGen/GenC.py | 0 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 18 +++++++++--------- BaseTools/Source/Python/Common/GlobalData.py | 2 ++ BaseTools/Source/Python/Common/Misc.py | 29 ++++++++++++++++++++= +++------ BaseTools/Source/Python/build/build.py | 5 ++++- 6 files changed, 41 insertions(+), 17 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 30d2f96fc7..21c5efbe6a 100755 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -133,7 +133,7 @@ class AutoGenManager(threading.Thread): def kill(self): self.feedback_q.put(None) class AutoGenWorkerInProcess(mp.Process): - def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k, share_data,log_q,error_event): + def __init__(self,module_queue,data_pipe_file_path,feedback_q,file_loc= k,cache_lock,share_data,log_q,error_event): mp.Process.__init__(self) self.module_queue =3D module_queue self.data_pipe_file_path =3Ddata_pipe_file_path @@ -141,6 +141,7 @@ class AutoGenWorkerInProcess(mp.Process): self.feedback_q =3D feedback_q self.PlatformMetaFileSet =3D {} self.file_lock =3D file_lock + self.cache_lock =3D cache_lock self.share_data =3D share_data self.log_q =3D log_q self.error_event =3D error_event @@ -187,6 +188,7 @@ class AutoGenWorkerInProcess(mp.Process): GlobalData.gCacheIR =3D self.data_pipe.Get("CacheIR") GlobalData.gEnableGenfdsMultiThread =3D self.data_pipe.Get("En= ableGenfdsMultiThread") GlobalData.file_lock =3D self.file_lock + GlobalData.cache_lock =3D self.cache_lock CommandTarget =3D self.data_pipe.Get("CommandTarget") pcd_from_build_option =3D [] for pcd_tuple in self.data_pipe.Get("BuildOptPcd"): diff --git a/BaseTools/Source/Python/AutoGen/GenC.py b/BaseTools/Source/Pyt= hon/AutoGen/GenC.py old mode 100644 new mode 100755 diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index c489c3b9c4..0f319e5b26 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -1806,7 +1806,7 @@ class ModuleAutoGen(AutoGen): MewIR.MakefilePath =3D MakefilePath MewIR.DependencyHeaderFileSet =3D Makefile.DependencyHeaderFileSet MewIR.CreateMakeFileDone =3D True - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakefilePath =3D MakefilePath @@ -1891,7 +1891,7 @@ class ModuleAutoGen(AutoGen): self.IsCodeFileCreated =3D True MewIR =3D ModuleBuildCacheIR(self.MetaFile.Path, self.Arch) MewIR.CreateCodeFileDone =3D True - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.CreateCodeFileDone =3D True @@ -2032,7 +2032,7 @@ class ModuleAutoGen(AutoGen): MewIR.ModuleFilesHashDigest =3D m.digest() MewIR.ModuleFilesHashHexDigest =3D m.hexdigest() MewIR.ModuleFilesChain =3D FileList - with GlobalData.file_lock: + with GlobalData.cache_lock: try: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.ModuleFilesHashDigest =3D m.digest() @@ -2091,7 +2091,7 @@ class ModuleAutoGen(AutoGen): # Add Module self m.update(gDict[(self.MetaFile.Path, self.Arch)].ModuleFilesHashDig= est) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.PreMakefileHashHexDigest =3D m.hexdigest() gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2159,7 +2159,7 @@ class ModuleAutoGen(AutoGen): m.update(Content) FileList.append((str(File), hashlib.md5(Content).hexdigest())) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.AutoGenFileList =3D self.AutoGenFileList.keys() IR.MakeHeaderFilesHashChain =3D FileList @@ -2222,7 +2222,7 @@ class ModuleAutoGen(AutoGen): New.sort(key=3Dlambda x: str(x)) MakeHashChain +=3D New =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeHashDigest =3D m.digest() IR.MakeHashHexDigest =3D m.hexdigest() @@ -2300,7 +2300,7 @@ class ModuleAutoGen(AutoGen): if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) =20 - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.PreMakeCacheHit =3D True gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2321,7 +2321,7 @@ class ModuleAutoGen(AutoGen): # .inc is contains binary information so do not skip by hash as we= ll for f_ext in self.SourceFileList: if '.inc' in str(f_ext): - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeCacheHit =3D False gDict[(self.MetaFile.Path, self.Arch)] =3D IR @@ -2383,7 +2383,7 @@ class ModuleAutoGen(AutoGen): =20 if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "PcdDxe": CreatePcdDatabaseCode(self, TemplateString(), TemplateString()) - with GlobalData.file_lock: + with GlobalData.cache_lock: IR =3D gDict[(self.MetaFile.Path, self.Arch)] IR.MakeCacheHit =3D True gDict[(self.MetaFile.Path, self.Arch)] =3D IR diff --git a/BaseTools/Source/Python/Common/GlobalData.py b/BaseTools/Sourc= e/Python/Common/GlobalData.py index 452dca32f0..09e92ce08c 100755 --- a/BaseTools/Source/Python/Common/GlobalData.py +++ b/BaseTools/Source/Python/Common/GlobalData.py @@ -122,6 +122,8 @@ gBuildHashSkipTracking =3D dict() =20 # Common dictionary to share module cache intermediate result and state gCacheIR =3D None +# Common lock for the module cache intermediate data +cache_lock =3D None # Common lock for the file access in multiple process AutoGens file_lock =3D None # Common dictionary to share platform libraries' constant Pcd diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Pyth= on/Common/Misc.py old mode 100644 new mode 100755 index 554ec010dd..ada2be2c88 --- a/BaseTools/Source/Python/Common/Misc.py +++ b/BaseTools/Source/Python/Common/Misc.py @@ -448,7 +448,7 @@ def RemoveDirectory(Directory, Recursively=3DFalse): # @retval True If the file content is changed and the fil= e is renewed # @retval False If the file content is the same # -def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue): +def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue, FileLock=3DGlobal= Data.file_lock): =20 if os.path.exists(File): if IsBinaryFile: @@ -479,6 +479,9 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue= ): if IsBinaryFile: OpenMode =3D "wb" =20 + if FileLock: + FileLock.acquire() + if GlobalData.gIsWindows and not os.path.exists(File): # write temp file, then rename the temp file to the real file # to make sure the file be immediate saved to disk @@ -487,8 +490,11 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTru= e): tempname =3D tf.name try: os.rename(tempname, File) - except: - EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IOErro= r %s' % X) + except IOError as X: + if GlobalData.gBinCacheSource: + EdkLogger.quiet("[cache error]:fails to save file with err= or: %s" % (X)) + else: + EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IO= Error %s' % X) else: try: with open(File, OpenMode) as Fd: @@ -496,6 +502,9 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue= ): except IOError as X: EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IOErro= r %s' % X) =20 + if FileLock: + FileLock.release() + return True =20 ## Copy source file only if it is different from the destination file @@ -510,7 +519,7 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue= ): # @retval True The two files content are different and the file= is copied # @retval False No copy really happen # -def CopyFileOnChange(SrcFile, Dst): +def CopyFileOnChange(SrcFile, Dst, FileLock=3DGlobalData.file_lock): if not os.path.exists(SrcFile): return False =20 @@ -531,6 +540,9 @@ def CopyFileOnChange(SrcFile, Dst): if not os.access(DirName, os.W_OK): EdkLogger.error(None, PERMISSION_FAILURE, "Do not have write p= ermission on directory %s" % DirName) =20 + if FileLock: + FileLock.acquire() + # os.replace and os.rename are the atomic operations in python 3 and 2. # we use these two atomic operations to ensure the file copy is atomic: # copy the src to a temp file in the dst same folder firstly, then @@ -546,9 +558,14 @@ def CopyFileOnChange(SrcFile, Dst): if GlobalData.gIsWindows and os.path.exists(DstFile): os.remove(DstFile) os.rename(tempname, DstFile) - except IOError as X: - EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError %s' = % X) + if GlobalData.gBinCacheSource: + EdkLogger.quiet("[cache error]:fails to copy file with error: = %s" % (X)) + else: + EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError = %s' % X) + + if FileLock: + FileLock.release() =20 return True =20 diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 299fa64311..2c10670a69 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -820,13 +820,15 @@ class Build(): file_lock =3D mp.Lock() error_event =3D mp.Event() GlobalData.file_lock =3D file_lock + cache_lock =3D mp.Lock() + GlobalData.cache_lock =3D cache_lock FfsCmd =3D DataPipe.Get("FfsCommand") if FfsCmd is None: FfsCmd =3D {} GlobalData.FfsCmd =3D FfsCmd GlobalData.libConstPcd =3D DataPipe.Get("LibConstPcd") GlobalData.Refes =3D DataPipe.Get("REFS") - auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,share_data,self.log_q,error_event) for _ in range= (self.ThreadNumber)] + auto_workers =3D [AutoGenWorkerInProcess(mqueue,DataPipe.dump_= file,feedback_q,file_lock,cache_lock,share_data,self.log_q,error_event) for= _ in range(self.ThreadNumber)] self.AutoGenMgr =3D AutoGenManager(auto_workers,feedback_q,err= or_event) self.AutoGenMgr.start() for w in auto_workers: @@ -1826,6 +1828,7 @@ class Build(): for PkgName in GlobalData.gPackageHash.keys(): GlobalData.gCacheIR[(PkgName, 'PackageHash')] =3D = GlobalData.gPackageHash[PkgName] GlobalData.file_lock =3D mp.Lock() + GlobalData.cache_lock =3D mp.Lock() GlobalData.FfsCmd =3D CmdListDict =20 self.Progress.Stop("done!") --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45639): https://edk2.groups.io/g/devel/message/45639 Mute This Topic: https://groups.io/mt/32867361/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-