From nobody Mon Sep 16 19:04:56 2024 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1695044075; cv=none; d=zohomail.com; s=zohoarc; b=ODo34HJtp0DTAJ7w+puwa1uwkhbsQPUMdo1Ow4eRa2PEWUVsm4+3h22/aJPYnP+z9NlXg7nRKOv9DFPv8uQZ2v61+jzLUEdqDKR4Lm0AiJ7X2Vu17/kLIcDWvXkE2x8QX/R7mKO7oX4iyyQA9XqG7l50J86aHUGDrZ2yFm9vwTc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1695044075; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Izi3AgniXVJCnAZ9pY2FQA6RTZ/rqmD6IBSgF3iUu+I=; b=cuo83oYdfRZOUU78pyAvYINGe6xFNUvcKXJMH7K+XQpRhlDmpFaBQUCBLZHyfSiq8g26mniqopxqddvImp8liTIRB9QrTHfWieX5Swsuu4LaaR57fMCxHfNOgsJmlfz+qR6gX5/sv11zY7+QdUzx/cv163s6mdHd8HIYI++QSng= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1695044075577882.8844438713061; Mon, 18 Sep 2023 06:34:35 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qiEOF-0000W2-WF; Mon, 18 Sep 2023 09:34:16 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qiENp-0000Bj-UK for qemu-devel@nongnu.org; Mon, 18 Sep 2023 09:33:52 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qiENl-0004gQ-NZ for qemu-devel@nongnu.org; Mon, 18 Sep 2023 09:33:48 -0400 Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-152-E3mvzwqpOe2mzyZn5OogUw-1; Mon, 18 Sep 2023 09:33:42 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 25EF7185A79C; Mon, 18 Sep 2023 13:33:42 +0000 (UTC) Received: from sirius.home.kraxel.org (unknown [10.39.193.95]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 96316140273C; Mon, 18 Sep 2023 13:33:41 +0000 (UTC) Received: by sirius.home.kraxel.org (Postfix, from userid 1000) id EBF091800D70; Mon, 18 Sep 2023 15:33:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695044024; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Izi3AgniXVJCnAZ9pY2FQA6RTZ/rqmD6IBSgF3iUu+I=; b=Wjcob5C/2oi8TtxNodqYuAjP2SSsVbaHB7xeIHqwf1MqFmZMcYSBAse7KW+q+ilUHALKWs Qv6jUdG1xWOtnlsLkSlbaEYZRxKgVlJPJq2LO59jsvseMyqaymRSb44ohzKaZrO4NInaKk XockWr2kZaR0R/Z56T1z2je+qKLZ0tQ= X-MC-Unique: E3mvzwqpOe2mzyZn5OogUw-1 From: Gerd Hoffmann To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Gerd Hoffmann , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= , Ani Sinha , Igor Mammedov Subject: [PULL 2/8] edk2: update build script Date: Mon, 18 Sep 2023 15:33:24 +0200 Message-ID: <20230918133338.2399701-3-kraxel@redhat.com> In-Reply-To: <20230918133338.2399701-1-kraxel@redhat.com> References: <20230918133338.2399701-1-kraxel@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=kraxel@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1695044078017100007 Content-Type: text/plain; charset="utf-8" Sync with latest version from gitlab.com/kraxel/edk2-build-config Signed-off-by: Gerd Hoffmann --- roms/edk2-build.py | 202 ++++++++++++++++++++++++++++----------------- 1 file changed, 128 insertions(+), 74 deletions(-) diff --git a/roms/edk2-build.py b/roms/edk2-build.py index 870893f7c8e3..e564765aaa71 100755 --- a/roms/edk2-build.py +++ b/roms/edk2-build.py @@ -6,6 +6,7 @@ """ import os import sys +import time import shutil import argparse import subprocess @@ -45,19 +46,28 @@ def get_coredir(cfg): return os.path.abspath(cfg['global']['core']) return os.getcwd() =20 -def get_version(cfg): +def get_toolchain(cfg, build): + if cfg.has_option(build, 'tool'): + return cfg[build]['tool'] + if cfg.has_option('global', 'tool'): + return cfg['global']['tool'] + return 'GCC5' + +def get_version(cfg, silent =3D False): coredir =3D get_coredir(cfg) if version_override: version =3D version_override - print('') - print(f'### version [override]: {version}') + if not silent: + print('') + print(f'### version [override]: {version}') return version if os.environ.get('RPM_PACKAGE_NAME'): version =3D os.environ.get('RPM_PACKAGE_NAME') version +=3D '-' + os.environ.get('RPM_PACKAGE_VERSION') version +=3D '-' + os.environ.get('RPM_PACKAGE_RELEASE') - print('') - print(f'### version [rpmbuild]: {version}') + if not silent: + print('') + print(f'### version [rpmbuild]: {version}') return version if os.path.exists(coredir + '/.git'): cmdline =3D [ 'git', 'describe', '--tags', '--abbrev=3D8', @@ -66,16 +76,17 @@ def get_version(cfg): stdout =3D subprocess.PIPE, check =3D True) version =3D result.stdout.decode().strip() - print('') - print(f'### version [git]: {version}') + if not silent: + print('') + print(f'### version [git]: {version}') return version return None =20 def pcd_string(name, value): return f'{name}=3DL{value}\\0' =20 -def pcd_version(cfg): - version =3D get_version(cfg) +def pcd_version(cfg, silent =3D False): + version =3D get_version(cfg, silent) if version is None: return [] return [ '--pcd', pcd_string('PcdFirmwareVersionString', version) ] @@ -85,49 +96,58 @@ def pcd_release_date(): return [] return [ '--pcd', pcd_string('PcdFirmwareReleaseDateString', release_d= ate) ] =20 -def build_message(line, line2 =3D None): +def build_message(line, line2 =3D None, silent =3D False): if os.environ.get('TERM') in [ 'xterm', 'xterm-256color' ]: # setxterm title start =3D '\x1b]2;' end =3D '\x07' print(f'{start}{rebase_prefix}{line}{end}', end =3D '') =20 - print('') - print('###') - print(f'### {rebase_prefix}{line}') - if line2: - print(f'### {line2}') - print('###', flush =3D True) + if silent: + print(f'### {rebase_prefix}{line}', flush =3D True) + else: + print('') + print('###') + print(f'### {rebase_prefix}{line}') + if line2: + print(f'### {line2}') + print('###', flush =3D True) =20 -def build_run(cmdline, name, section, silent =3D False): - print(cmdline, flush =3D True) +def build_run(cmdline, name, section, silent =3D False, nologs =3D False): if silent: - print('### building in silent mode ...', flush =3D True) + logfile =3D f'{section}.log' + if nologs: + print(f'### building in silent mode [no log] ...', flush =3D T= rue) + else: + print(f'### building in silent mode [{logfile}] ...', flush = =3D True) + start =3D time.time() result =3D subprocess.run(cmdline, check =3D False, stdout =3D subprocess.PIPE, stderr =3D subprocess.STDOUT) - - logfile =3D f'{section}.log' - print(f'### writing log to {logfile} ...') - with open(logfile, 'wb') as f: - f.write(result.stdout) + if not nologs: + with open(logfile, 'wb') as f: + f.write(result.stdout) =20 if result.returncode: print('### BUILD FAILURE') + print('### cmdline') + print(cmdline) print('### output') print(result.stdout.decode()) print(f'### exit code: {result.returncode}') else: - print('### OK') + secs =3D int(time.time() - start) + print(f'### OK ({int(secs/60)}:{secs%60:02d})') else: + print(cmdline, flush =3D True) result =3D subprocess.run(cmdline, check =3D False) if result.returncode: print(f'ERROR: {cmdline[0]} exited with {result.returncode}' f' while building {name}') sys.exit(result.returncode) =20 -def build_copy(plat, tgt, dstdir, copy): - srcdir =3D f'Build/{plat}/{tgt}_GCC5' +def build_copy(plat, tgt, toolchain, dstdir, copy): + srcdir =3D f'Build/{plat}/{tgt}_{toolchain}' names =3D copy.split() srcfile =3D names[0] if len(names) > 1: @@ -156,66 +176,68 @@ def pad_file(dstdir, pad): subprocess.run(cmdline, check =3D True) =20 # pylint: disable=3Dtoo-many-branches -def build_one(cfg, build, jobs =3D None, silent =3D False): +def build_one(cfg, build, jobs =3D None, silent =3D False, nologs =3D Fals= e): + b =3D cfg[build] + cmdline =3D [ 'build' ] - cmdline +=3D [ '-t', 'GCC5' ] - cmdline +=3D [ '-p', cfg[build]['conf'] ] + cmdline +=3D [ '-t', get_toolchain(cfg, build) ] + cmdline +=3D [ '-p', b['conf'] ] =20 - if (cfg[build]['conf'].startswith('OvmfPkg/') or - cfg[build]['conf'].startswith('ArmVirtPkg/')): - cmdline +=3D pcd_version(cfg) + if (b['conf'].startswith('OvmfPkg/') or + b['conf'].startswith('ArmVirtPkg/')): + cmdline +=3D pcd_version(cfg, silent) cmdline +=3D pcd_release_date() =20 if jobs: cmdline +=3D [ '-n', jobs ] - for arch in cfg[build]['arch'].split(): + for arch in b['arch'].split(): cmdline +=3D [ '-a', arch ] - if 'opts' in cfg[build]: - for name in cfg[build]['opts'].split(): + if 'opts' in b: + for name in b['opts'].split(): section =3D 'opts.' + name for opt in cfg[section]: cmdline +=3D [ '-D', opt + '=3D' + cfg[section][opt] ] - if 'pcds' in cfg[build]: - for name in cfg[build]['pcds'].split(): + if 'pcds' in b: + for name in b['pcds'].split(): section =3D 'pcds.' + name for pcd in cfg[section]: cmdline +=3D [ '--pcd', pcd + '=3D' + cfg[section][pcd] ] - if 'tgts' in cfg[build]: - tgts =3D cfg[build]['tgts'].split() + if 'tgts' in b: + tgts =3D b['tgts'].split() else: tgts =3D [ 'DEBUG' ] for tgt in tgts: desc =3D None - if 'desc' in cfg[build]: - desc =3D cfg[build]['desc'] - build_message(f'building: {cfg[build]["conf"]} ({cfg[build]["arch"= ]}, {tgt})', - f'description: {desc}') + if 'desc' in b: + desc =3D b['desc'] + build_message(f'building: {b["conf"]} ({b["arch"]}, {tgt})', + f'description: {desc}', + silent =3D silent) build_run(cmdline + [ '-b', tgt ], - cfg[build]['conf'], + b['conf'], build + '.' + tgt, - silent) + silent, + nologs) =20 - if 'plat' in cfg[build]: + if 'plat' in b: # copy files - for cpy in cfg[build]: + for cpy in b: if not cpy.startswith('cpy'): continue - build_copy(cfg[build]['plat'], - tgt, - cfg[build]['dest'], - cfg[build][cpy]) + build_copy(b['plat'], tgt, + get_toolchain(cfg, build), + b['dest'], b[cpy]) # pad builds - for pad in cfg[build]: + for pad in b: if not pad.startswith('pad'): continue - pad_file(cfg[build]['dest'], - cfg[build][pad]) + pad_file(b['dest'], b[pad]) =20 -def build_basetools(silent =3D False): - build_message('building: BaseTools') +def build_basetools(silent =3D False, nologs =3D False): + build_message('building: BaseTools', silent =3D silent) basedir =3D os.environ['EDK_TOOLS_PATH'] cmdline =3D [ 'make', '-C', basedir ] - build_run(cmdline, 'BaseTools', 'build.basetools', silent) + build_run(cmdline, 'BaseTools', 'build.basetools', silent, nologs) =20 def binary_exists(name): for pdir in os.environ['PATH'].split(':'): @@ -223,7 +245,7 @@ def binary_exists(name): return True return False =20 -def prepare_env(cfg): +def prepare_env(cfg, silent =3D False): """ mimic Conf/BuildEnv.sh """ workspace =3D os.getcwd() packages =3D [ workspace, ] @@ -253,7 +275,7 @@ def prepare_env(cfg): toolsdef =3D coredir + '/Conf/tools_def.txt' if not os.path.exists(toolsdef): os.makedirs(os.path.dirname(toolsdef), exist_ok =3D True) - build_message('running BaseTools/BuildEnv') + build_message('running BaseTools/BuildEnv', silent =3D silent) cmdline =3D [ 'bash', 'BaseTools/BuildEnv' ] subprocess.run(cmdline, cwd =3D coredir, check =3D True) =20 @@ -267,20 +289,32 @@ def prepare_env(cfg): os.environ['PYTHONHASHSEED'] =3D '1' =20 # for cross builds - if binary_exists('arm-linux-gnu-gcc'): + if binary_exists('arm-linux-gnueabi-gcc'): + # ubuntu + os.environ['GCC5_ARM_PREFIX'] =3D 'arm-linux-gnueabi-' + os.environ['GCC_ARM_PREFIX'] =3D 'arm-linux-gnueabi-' + elif binary_exists('arm-linux-gnu-gcc'): + # fedora os.environ['GCC5_ARM_PREFIX'] =3D 'arm-linux-gnu-' + os.environ['GCC_ARM_PREFIX'] =3D 'arm-linux-gnu-' if binary_exists('loongarch64-linux-gnu-gcc'): os.environ['GCC5_LOONGARCH64_PREFIX'] =3D 'loongarch64-linux-gnu-' + os.environ['GCC_LOONGARCH64_PREFIX'] =3D 'loongarch64-linux-gnu-' =20 hostarch =3D os.uname().machine if binary_exists('aarch64-linux-gnu-gcc') and hostarch !=3D 'aarch64': os.environ['GCC5_AARCH64_PREFIX'] =3D 'aarch64-linux-gnu-' + os.environ['GCC_AARCH64_PREFIX'] =3D 'aarch64-linux-gnu-' if binary_exists('riscv64-linux-gnu-gcc') and hostarch !=3D 'riscv64': os.environ['GCC5_RISCV64_PREFIX'] =3D 'riscv64-linux-gnu-' + os.environ['GCC_RISCV64_PREFIX'] =3D 'riscv64-linux-gnu-' if binary_exists('x86_64-linux-gnu-gcc') and hostarch !=3D 'x86_64': os.environ['GCC5_IA32_PREFIX'] =3D 'x86_64-linux-gnu-' os.environ['GCC5_X64_PREFIX'] =3D 'x86_64-linux-gnu-' os.environ['GCC5_BIN'] =3D 'x86_64-linux-gnu-' + os.environ['GCC_IA32_PREFIX'] =3D 'x86_64-linux-gnu-' + os.environ['GCC_X64_PREFIX'] =3D 'x86_64-linux-gnu-' + os.environ['GCC_BIN'] =3D 'x86_64-linux-gnu-' =20 def build_list(cfg): for build in cfg.sections(): @@ -303,10 +337,12 @@ def main(): parser.add_argument('-j', '--jobs', dest =3D 'jobs', type =3D str, help =3D 'allow up to JOBS parallel build jobs', metavar =3D 'JOBS') - parser.add_argument('-m', '--match', dest =3D 'match', type =3D str, + parser.add_argument('-m', '--match', dest =3D 'match', + type =3D str, action =3D 'append', help =3D 'only run builds matching INCLUDE (substr= ing)', metavar =3D 'INCLUDE') - parser.add_argument('-x', '--exclude', dest =3D 'exclude', type =3D st= r, + parser.add_argument('-x', '--exclude', dest =3D 'exclude', + type =3D str, action =3D 'append', help =3D 'skip builds matching EXCLUDE (substring)= ', metavar =3D 'EXCLUDE') parser.add_argument('-l', '--list', dest =3D 'list', @@ -316,6 +352,9 @@ def main(): action =3D 'store_true', default =3D False, help =3D 'write build output to logfiles, ' 'write to console only on errors') + parser.add_argument('--no-logs', dest =3D 'nologs', + action =3D 'store_true', default =3D False, + help =3D 'do not write build log files (with --sil= ent)') parser.add_argument('--core', dest =3D 'core', type =3D str, metavar = =3D 'DIR', help =3D 'location of the core edk2 repository ' '(i.e. where BuildTools are located)') @@ -323,6 +362,9 @@ def main(): type =3D str, action =3D 'append', metavar =3D 'DI= R', help =3D 'location(s) of additional packages ' '(can be specified multiple times)') + parser.add_argument('-t', '--toolchain', dest =3D 'toolchain', + type =3D str, metavar =3D 'NAME', + help =3D 'tool chain to be used to build edk2') parser.add_argument('--version-override', dest =3D 'version_override', type =3D str, metavar =3D 'VERSION', help =3D 'set firmware build version') @@ -335,7 +377,7 @@ def main(): os.chdir(options.directory) =20 if not os.path.exists(options.configfile): - print('config file "{options.configfile}" not found') + print(f'config file "{options.configfile}" not found') return 1 =20 cfg =3D configparser.ConfigParser() @@ -344,7 +386,7 @@ def main(): =20 if options.list: build_list(cfg) - return + return 0 =20 if not cfg.has_section('global'): cfg.add_section('global') @@ -352,6 +394,8 @@ def main(): cfg.set('global', 'core', options.core) if options.pkgs: cfg.set('global', 'pkgs', ' '.join(options.pkgs)) + if options.toolchain: + cfg.set('global', 'tool', options.toolchain) =20 global version_override global release_date @@ -361,18 +405,28 @@ def main(): if options.release_date: release_date =3D options.release_date =20 - prepare_env(cfg) - build_basetools(options.silent) + prepare_env(cfg, options.silent) + build_basetools(options.silent, options.nologs) for build in cfg.sections(): if not build.startswith('build.'): continue - if options.match and options.match not in build: - print(f'# skipping "{build}" (not matching "{options.match}")') - continue - if options.exclude and options.exclude in build: - print(f'# skipping "{build}" (matching "{options.exclude}")') - continue - build_one(cfg, build, options.jobs, options.silent) + if options.match: + matching =3D False + for item in options.match: + if item in build: + matching =3D True + if not matching: + print(f'# skipping "{build}" (not matching "{"|".join(opti= ons.match)}")') + continue + if options.exclude: + exclude =3D False + for item in options.exclude: + if item in build: + print(f'# skipping "{build}" (matching "{item}")') + exclude =3D True + if exclude: + continue + build_one(cfg, build, options.jobs, options.silent, options.nologs) =20 return 0 =20 --=20 2.41.0