From nobody Tue Feb 10 00:00:43 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+45509+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45509+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565686270; cv=none; d=zoho.com; s=zohoarc; b=P/fZlpcgqhC8uqZVdGlDCxA1o+xMd+abTZdmaSaaMEZuOmOmiOn8EChB1qYrNrANAGPvLGr7Idw4dBdaf1/oDDiGzP4Ui+YXwZSYR5kYMcLZIJdzx9nopHC0nGKbYJuVP6KXnAOzbxO6JVWAGMop31WniWQtSl2OCBQfAukbHlE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565686270; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=j8zAf/okwNJCidlXZQITAbxpjQ2kaDhZTzL2pnetEHQ=; b=Bk+EqPj3H0svafjYLC7Ho6rJKaMogm+jnj4kLdSolqRjZO81/XCwazChyx1o/hSZatLBfD5iA7+4X710gTcVi4PLOcMQtZrilKYfEI37olyyjEGKLa6cgVcOiSh5opd/ve9AnBZzK7bU61dDdTVMJhXZRuSlLpKj2EAtf48aEzU= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+45509+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565686270599894.3290809252499; Tue, 13 Aug 2019 01:51:10 -0700 (PDT) Return-Path: X-Received: from mga09.intel.com (mga09.intel.com []) by groups.io with SMTP; Tue, 13 Aug 2019 01:51:09 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Aug 2019 01:51:09 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,380,1559545200"; d="scan'208";a="200414618" X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.211.145]) by fmsmga004.fm.intel.com with ESMTP; 13 Aug 2019 01:51:08 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com, michael.johnson@intel.com, "Shi, Steven" Subject: [edk2-devel] [PATCH v3 5/5] BaseTools: Improve the file saving and copying reliability Date: Tue, 13 Aug 2019 16:50:55 +0800 Message-Id: <20190813085055.23208-6-steven.shi@intel.com> In-Reply-To: <20190813085055.23208-1-steven.shi@intel.com> References: <20190813085055.23208-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565686270; bh=RzadLFrM9ltL+Zx8kTGsdoBtpBCuSP1PayWqa/mkHX4=; h=Cc:Date:From:Reply-To:Subject:To; b=mCHANVWvYE1N3nHJvK6wB9ff2NwlqhGdldHTwUikR/pw8E5dTEIkowLTvILAgSS2NHO 2yTQeWlVVWeZxqV8mCWIZRdHiamLrY/Jp1ev8fqAQPuetfevQOM7Bc1BTeeduh768whBF pstlt9NithCsZjHxzG9de9PT6JGHg3lBcnk= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" From: "Shi, Steven" BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D2079 The Basetool CopyFileOnChange() and SaveFileOnChange() functions might raise the IOError occasionally when build in Windows with multi-process and build cache enabled. The CopyFileOnChange() and SaveFileOnChange() might be invoked in multiple sub-processes simultaneously, and this patch adds a global lock to sync these functions invoking which can harden their reliability. Cc: Liming Gao Cc: Bob Feng Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/GenC.py | 2 +- BaseTools/Source/Python/AutoGen/GenMake.py | 2 +- BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 2 +- BaseTools/Source/Python/Common/Misc.py | 18 +++++++++++++----- 4 files changed, 16 insertions(+), 8 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenC.py b/BaseTools/Source/Pyt= hon/AutoGen/GenC.py old mode 100644 new mode 100755 index 910c8fe370..1bb3f42b46 --- a/BaseTools/Source/Python/AutoGen/GenC.py +++ b/BaseTools/Source/Python/AutoGen/GenC.py @@ -2103,5 +2103,5 @@ def CreateCode(Info, AutoGenC, AutoGenH, StringH, Uni= GenCFlag, UniGenBinBuffer, # @retval False If the file exists and the content is not chan= ged # def Generate(FilePath, Content, IsBinaryFile): - return SaveFileOnChange(FilePath, Content, IsBinaryFile) + return SaveFileOnChange(FilePath, Content, IsBinaryFile, GlobalData.fi= le_lock) =20 diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/= Python/AutoGen/GenMake.py index de820eeb2f..e159782306 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -185,7 +185,7 @@ class BuildFile(object): self._FileType =3D FileType FileContent =3D self._TEMPLATE_.Replace(self._TemplateDict) FileName =3D self._FILE_NAME_[FileType] - return SaveFileOnChange(os.path.join(self._AutoGenObject.MakeFileD= ir, FileName), FileContent, False) + return SaveFileOnChange(os.path.join(self._AutoGenObject.MakeFileD= ir, FileName), FileContent, False, GlobalData.file_lock) =20 ## Return a list of directory creation command string # diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/S= ource/Python/AutoGen/ModuleAutoGen.py index ee8518e19c..0d16d8ded7 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -1612,7 +1612,7 @@ class ModuleAutoGen(AutoGen): destination_dir =3D os.path.dirname(destination_file) CreateDirectory(destination_dir) try: - CopyFileOnChange(File, destination_dir) + CopyFileOnChange(File, destination_dir, GlobalData.file_lock) except: EdkLogger.quiet("[cache warning]: fail to copy file:%s to fold= er:%s" % (File, destination_dir)) return diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Pyth= on/Common/Misc.py old mode 100644 new mode 100755 index 26d149c270..db2c3c9589 --- a/BaseTools/Source/Python/Common/Misc.py +++ b/BaseTools/Source/Python/Common/Misc.py @@ -448,7 +448,7 @@ def RemoveDirectory(Directory, Recursively=3DFalse): # @retval True If the file content is changed and the fil= e is renewed # @retval False If the file content is the same # -def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue): +def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue, FileLock=3DNone): =20 if os.path.exists(File): if IsBinaryFile: @@ -486,8 +486,12 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTru= e): tf.write(Content) tempname =3D tf.name try: - os.rename(tempname, File) - except: + if FileLock: + with FileLock: + os.rename(tempname, File) + else: + os.rename(tempname, File) + except IOError as X: EdkLogger.error(None, FILE_CREATE_FAILURE, ExtraData=3D'IOErro= r %s' % X) else: try: @@ -510,7 +514,7 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTrue= ): # @retval True The two files content are different and the file= is copied # @retval False No copy really happen # -def CopyFileOnChange(SrcFile, Dst): +def CopyFileOnChange(SrcFile, Dst, FileLock=3DNone): if not os.path.exists(SrcFile): return False =20 @@ -545,7 +549,11 @@ def CopyFileOnChange(SrcFile, Dst): # os.rename reqire to remove the dst on Windows, otherwise OSE= rror will be raised. if GlobalData.gIsWindows and os.path.exists(DstFile): os.remove(DstFile) - os.rename(tempname, DstFile) + if FileLock: + with FileLock: + os.rename(tempname, DstFile) + else: + os.rename(tempname, DstFile) =20 except IOError as X: EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError %s' = % X) --=20 2.17.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#45509): https://edk2.groups.io/g/devel/message/45509 Mute This Topic: https://groups.io/mt/32850701/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-