From nobody Tue May 7 01:08:39 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+42436+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+42436+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1560515084; cv=none; d=zoho.com; s=zohoarc; b=M4Z5MZc5qqYzsoGjc1dw32dxRxUBVv92flVip0vRGDDAZ03OKgX5+FWPYdIjk8ZFeLAJj0MTMU2jtafW0r2pl0nbZl0atIkIaA0+mi/sm8i3KYjP7+TtFKFjayenYi78XXH4ymHDlFfxyVqOQvQcGrXj4TunOc7r2UA6K0dymew= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1560515084; h=Cc:Date:From:List-Id:List-Unsubscribe:Message-ID:Reply-To:Sender:Subject:To:ARC-Authentication-Results; bh=U0c8F4JWTZ4DdvnM3NL2IgeZBgJ/ENvpEtyJpq4Mq+I=; b=HvNAdpV2RmcpUnVzU19L4zeeSMn5UYeVW1H8ZuaKViRP5kW8iHT6pgMhqTEdUL6GpNAv1ZRrn6/MeYZmoDY0piJ9YL/B/s3Og1V07vFLs+f8E/hj7/EIxJh/mm9EhglTfEAdq5PvjmZtZk/S0HUN4JDkZExQGNx9kD3aERkgKC0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+42436+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 156051508496870.79981010077961; Fri, 14 Jun 2019 05:24:44 -0700 (PDT) Return-Path: X-Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by groups.io with SMTP; Fri, 14 Jun 2019 05:24:43 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga104.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 14 Jun 2019 05:24:42 -0700 X-ExtLoop1: 1 X-Received: from jshi19-mobl.ccr.corp.intel.com ([10.254.215.177]) by orsmga004.jf.intel.com with ESMTP; 14 Jun 2019 05:24:40 -0700 From: "Steven Shi" To: devel@edk2.groups.io Cc: liming.gao@intel.com, bob.c.feng@intel.com, christian.rodriguez@intel.com Subject: [edk2-devel] [PATCH] BaseTools:Introduce CopyFileOnChange() function to copy cache files Date: Fri, 14 Jun 2019 20:24:25 +0800 Message-Id: <20190614122425.2956-1-steven.shi@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,steven.shi@intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1560515084; bh=VesHjc4N2JkSTyNpeOrXlxsPk00Zogvz0fYqw5zvGLM=; h=Cc:Date:From:Reply-To:Subject:To; b=qDJ26K0hgav63gn8Y61KLkul764+RGP0v72Koba+b3JJYRg1r2VlvnDGWlFNoWaVy/f +9WfqydugfGog+EkdCksH3xAh6io4ZoZ2yVpKtjsQyCoG+vFoKT6HUMHvnPK0KaHz5WBq 1ywth0eQZzqHqWC31icu2QJml58WJzAEIw4= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D1894 Basetool need a CopyFileOnChange() function to avoid cache file writing race in multi-thread build. Some platforms build fail with file IO writing race issue when the build cache is enabled to store cache files in multi-threads. This is because common same library cache files (e.g. some libs in MdePkg) can be stored by many different driver modules' build threads at same time. Current build cache need a function to check whether the same cache file already exist, and only copy source file if it is different from the destination file. This patch introduces an atomic copy function to avoid duplicated cache files copy. Cc: Liming Gao Cc: Bob Feng Cc: Christian Rodriguez Signed-off-by: Steven Shi --- BaseTools/Source/Python/AutoGen/AutoGen.py | 11 ++++++----- BaseTools/Source/Python/Common/LongFilePathOs.py | 4 ++++ BaseTools/Source/Python/Common/Misc.py | 56 ++++++++++++++++++++= +++++++++++++++++++++++++++++++++++- 3 files changed, 65 insertions(+), 6 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/= Python/AutoGen/AutoGen.py index 3f41fbb507..7b933f7122 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGen.py +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py @@ -3901,11 +3901,12 @@ class ModuleAutoGen(AutoGen): CreateDirectory (FileDir) HashFile =3D path.join(self.BuildDir, self.Name + '.hash') if os.path.exists(HashFile): - shutil.copy2(HashFile, FileDir) + CopyFileOnChange(HashFile, FileDir) if not self.IsLibrary: ModuleFile =3D path.join(self.OutputDir, self.Name + '.inf') if os.path.exists(ModuleFile): - shutil.copy2(ModuleFile, FileDir) + CopyFileOnChange(ModuleFile, FileDir) else: OutputDir =3D self.OutputDir.replace('\\', '/').strip('/') DebugDir =3D self.DebugDir.replace('\\', '/').strip('/') @@ -3925,7 +3926,7 @@ class ModuleAutoGen(AutoGen): destination_file =3D os.path.join(FileDir, sub_dir) destination_dir =3D os.path.dirname(destination_file) CreateDirectory(destination_dir) - shutil.copy2(File, destination_dir) + CopyFileOnChange(File, destination_dir) =20 def AttemptModuleCacheCopy(self): # If library or Module is binary do not skip by hash @@ -3947,14 +3948,14 @@ class ModuleAutoGen(AutoGen): for root, dir, files in os.walk(FileDir): for f in files: if self.Name + '.hash' in f: - shutil.copy(HashFile, self.BuildDir) + CopyFileOnChange(HashFile, self.BuildDir) else: File =3D path.join(root, f) sub_dir =3D os.path.relpath(File, FileDir) destination_file =3D os.path.join(self.Out= putDir, sub_dir) destination_dir =3D os.path.dirname(destin= ation_file) CreateDirectory(destination_dir) - shutil.copy(File, destination_dir) + CopyFileOnChange(File, destination_dir) if self.Name =3D=3D "PcdPeim" or self.Name =3D=3D "Pcd= Dxe": CreatePcdDatabaseCode(self, TemplateString(), Temp= lateString()) return True diff --git a/BaseTools/Source/Python/Common/LongFilePathOs.py b/BaseTools/S= ource/Python/Common/LongFilePathOs.py index ae8a68e4ad..190f36d7ec 100644 --- a/BaseTools/Source/Python/Common/LongFilePathOs.py +++ b/BaseTools/Source/Python/Common/LongFilePathOs.py @@ -60,6 +60,10 @@ def listdir(path): List.append(Item) return List =20 +if hasattr(os, 'replace'): + def replace(src, dst): + return os.replace(LongFilePath(src), LongFilePath(dst)) + environ =3D os.environ getcwd =3D os.getcwd chdir =3D os.chdir diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Pyth= on/Common/Misc.py index d082c58bef..1513b9226a 100644 --- a/BaseTools/Source/Python/Common/Misc.py +++ b/BaseTools/Source/Python/Common/Misc.py @@ -18,13 +18,13 @@ import re import pickle import array import shutil +import filecmp from random import sample from struct import pack import uuid import subprocess import tempfile from collections import OrderedDict - import Common.LongFilePathOs as os from Common import EdkLogger as EdkLogger from Common import GlobalData as GlobalData @@ -502,6 +502,60 @@ def SaveFileOnChange(File, Content, IsBinaryFile=3DTru= e): =20 return True =20 +## Copy source file only if it is different from the destination file +# +# This method is used to copy file only if the source file and destination +# file content are different. This is quite useful to avoid duplicated +# file writing. +# +# @param SrcFile The path of source file +# @param Dst The path of destination file or folder +# +# @retval True The two files content are different and the file= is copied +# @retval False No copy really happen +# +def CopyFileOnChange(SrcFile, Dst): + if not os.path.exists(SrcFile): + return False + + if os.path.isdir(Dst): + DstFile =3D os.path.join(Dst, os.path.basename(SrcFile)) + else: + DstFile =3D Dst + + if os.path.exists(DstFile) and filecmp.cmp(SrcFile, DstFile): + return False + + DirName =3D os.path.dirname(DstFile) + if not CreateDirectory(DirName): + EdkLogger.error(None, FILE_CREATE_FAILURE, "Could not create direc= tory %s" % DirName) + else: + if DirName =3D=3D '': + DirName =3D os.getcwd() + if not os.access(DirName, os.W_OK): + EdkLogger.error(None, PERMISSION_FAILURE, "Do not have write p= ermission on directory %s" % DirName) + + # os.replace and os.rename are the atomic operations in python 3 and 2. + # we use these two atomic operations to ensure the file copy is atomic: + # copy the src to a temp file in the dst same folder firstly, then + # replace or rename the temp file to the destination file. + with tempfile.NamedTemporaryFile(dir=3DDirName, delete=3DFalse) as tf: + shutil.copyfile(SrcFile, tf.name) + tempname =3D tf.name + try: + if hasattr(os, 'replace'): + os.replace(tempname, DstFile) + else: + # os.rename reqire to remove the dst on Windows, otherwise OSE= rror will be raised. + if GlobalData.gIsWindows and os.path.exists(DstFile): + os.remove(DstFile) + os.rename(tempname, DstFile) + + except IOError as X: + EdkLogger.error(None, FILE_COPY_FAILURE, ExtraData=3D'IOError %s' = % X) + + return True + ## Retrieve and cache the real path name in file system # # @param Root The root directory of path relative to --=20 2.17.1.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#42436): https://edk2.groups.io/g/devel/message/42436 Mute This Topic: https://groups.io/mt/32063150/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-