From nobody Mon Sep 16 19:56:47 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+101064+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+101064+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1678696229; cv=none; d=zohomail.com; s=zohoarc; b=fFrjMRGh03nq4mmdsxaf8k6JQBbe7Ff/9SMF9YcqpI3VkNZiF3rZPhEAvC59UR4iaRSJdwj039B1qpGX2b9HscE6k3BFhGkQGlnTSpVcJYKg7gqwQyQau+jVeDnGqmtP2DZ2wG2bBuuDjZpOLck41W3/RMWQHVwBAMAgkSP3Ssg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1678696229; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=3XH8qR9xdTRm6IeA66mCAAvaGEGGkbzIUitqZer3GcI=; b=VZTH8lFdlORmAc4wneHOcMUzhqBDCe/8SnJQXWePeDCKkPcRICEHJJXTqP7KqtjmbDO+E/c/S4TW3MamsuFuWvbgm3P6BUufsfdoXEfHOAdvA6T2mcCv8nhudMyVf1msp3LUFWV/I5pM2Dh23aLTdwjM1Q9iimn4hRc777PDml8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+101064+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1678696229060599.7539373789667; Mon, 13 Mar 2023 01:30:29 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id 3JCJYY1788612xKkbIoxqYBO; Mon, 13 Mar 2023 01:30:28 -0700 X-Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web11.15096.1678696227185080663 for ; Mon, 13 Mar 2023 01:30:27 -0700 X-Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-364-XxnOZ83gPleoRDZ-4SEKAQ-1; Mon, 13 Mar 2023 04:30:22 -0400 X-MC-Unique: XxnOZ83gPleoRDZ-4SEKAQ-1 X-Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 68F493C10141; Mon, 13 Mar 2023 08:30:22 +0000 (UTC) X-Received: from sirius.home.kraxel.org (unknown [10.39.192.142]) by smtp.corp.redhat.com (Postfix) with ESMTPS id DD04B400F50; Mon, 13 Mar 2023 08:30:21 +0000 (UTC) X-Received: by sirius.home.kraxel.org (Postfix, from userid 1000) id 486571801E81; Mon, 13 Mar 2023 09:30:16 +0100 (CET) From: "Gerd Hoffmann" To: devel@edk2.groups.io Cc: Jian J Wang , Pawel Polawski , Xiaoyu Lu , Ard Biesheuvel , Guomin Jiang , Gerd Hoffmann , Jiewen Yao , Oliver Steffen , Jordan Justen Subject: [edk2-devel] [PATCH 04/22] CryptoPkg/openssl: add openssl3 configure scripts Date: Mon, 13 Mar 2023 09:29:58 +0100 Message-Id: <20230313083016.136448-5-kraxel@redhat.com> In-Reply-To: <20230313083016.136448-1-kraxel@redhat.com> References: <20230313083016.136448-1-kraxel@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.10 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Precedence: Bulk List-Unsubscribe: List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,kraxel@redhat.com X-Gm-Message-State: lKEsYUn9jenhTyezuLfukb6Px1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1678696228; bh=tGNXtDksbk6mYIHnX4agphGhyxmrs5xQKPgv+azoZsw=; h=Cc:Content-Type:Date:From:Reply-To:Subject:To; b=kqaAcNplCi1r4/Uo4oEuL/oxvDAvoA5i2PRwHuVxmCJWqUoLnpcuNVMCaRSRKOnTlEb aZkMSoEXvG8euT8T55DBLoBRp4Q+27/+uSmAyPEg5hYQ/ASTuT39zX7w3ffv4ZFhOirkY FxvDm2YzIpPpI64NeqFXSdGA+mo0lJFjK5s= X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1678696230942100024 Content-Type: text/plain; charset="utf-8"; x-default="true" Rewrite the script to configure openssl 3.0 from scratch. It's two scripts now: * Tiny helper script, dumping the perl configdata as json. * Actual configure.py script, written in python, which copies over the generated files to openssl-gen and updates the OpensslLib*.inf file lists and build flags. The configuration workflow has changed a bit: * All generated files are stored in the openssl-gen directory tree. * For ec/no-ec builds two different header files are used. Default is the ec variant, and the new EDK2_OPENSSL_NOEC define is used to select the no-ec build. A five line wrapper include is used to pick the one or the other. * For non-accel builds -DOPENSSL_NO_ASM on the command line is used (same as before). * For configration defines the OPENSSL_FLAGS_$(variant) variable is used, where variant is the architecture for the accelerated builds and 'NOASM' for the non-accelerated builds. Signed-off-by: Gerd Hoffmann --- CryptoPkg/Library/OpensslLib/configure.py | 365 ++++++++++++++++++++++ CryptoPkg/Library/OpensslLib/perl2json.pl | 19 ++ 2 files changed, 384 insertions(+) create mode 100755 CryptoPkg/Library/OpensslLib/configure.py create mode 100755 CryptoPkg/Library/OpensslLib/perl2json.pl diff --git a/CryptoPkg/Library/OpensslLib/configure.py b/CryptoPkg/Library/= OpensslLib/configure.py new file mode 100755 index 000000000000..d8aa0cb03edf --- /dev/null +++ b/CryptoPkg/Library/OpensslLib/configure.py @@ -0,0 +1,365 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: BSD-2-Clause-Patent +import os +import sys +import json +import shutil +import pprint +import argparse +import subprocess + +def openssl_configure(openssldir, target, ec =3D True): + """ Run openssl Configure script. """ + cmdline =3D [ + 'perl', + 'Configure', + '--config=3D../UefiAsm.conf', + '--api=3D1.1.1', + '--with-rand-seed=3Dnone', + target, + 'no-afalgeng', + 'no-async', + 'no-autoerrinit', + 'no-autoload-config', + 'no-bf', + 'no-blake2', + 'no-camellia', + 'no-capieng', + 'no-cast', + 'no-chacha', + 'no-cmac', + 'no-cms', + 'no-ct', + 'no-deprecated', + 'no-des', + 'no-dgram', + 'no-dsa', + 'no-dynamic-engine', + 'no-ec2m', + 'no-engine', + 'no-err', + 'no-filenames', + 'no-gost', + 'no-idea', + 'no-md4', + 'no-mdc2', + 'no-pic', + 'no-ocb', + 'no-poly1305', + 'no-posix-io', + 'no-rc2', + 'no-rc4', + 'no-rfc3779', + 'no-rmd160', + 'no-scrypt', + 'no-seed', + 'no-sm4', + 'no-sock', + 'no-srp', + 'no-srtp', + 'no-ssl', + 'no-stdio', + 'no-threads', + 'no-ts', + 'no-ui', + 'no-whirlpool', + ] + if not ec: + cmdline +=3D [ 'no-ec', ] + print('') + print(f'# -*- configure openssl for {target} (ec=3D{ec}) -*-') + rc =3D subprocess.run(cmdline, cwd =3D openssldir, + stdout =3D subprocess.PIPE, + stderr =3D subprocess.PIPE) + if rc.returncode: + print(rc.stdout) + print(rc.stderr) + sys.exit(rc.returncode) + +def openssl_run_make(openssldir, target): + """ + Run make utility to generate files or cleanup. + Target can be either a string or a list of strings. + """ + cmdline =3D [ 'make', '--silent' ] + if isinstance(target, list): + cmdline +=3D target + else: + cmdline +=3D [ target, ] + rc =3D subprocess.run(cmdline, cwd =3D openssldir) + rc.check_returncode() + +def get_configdata(openssldir): + """ + Slurp openssl config data as JSON, + using a little perl helper script. + """ + cmdline =3D [ + 'perl', + 'perl2json.pl', + openssldir, + ] + rc =3D subprocess.run(cmdline, stdout =3D subprocess.PIPE) + rc.check_returncode() + return json.loads(rc.stdout) + +def is_asm(filename): + """ Check whenevr the passed file is an assembler file """ + if filename.endswith('.s') or filename.endswith('.S'): + return True + return False + +def generate_files(openssldir, opensslgendir, asm, filelist): + """ + Generate files, using make, and copy over the results to the + directory tree for generated openssl files. Creates + subdirectories as needed. + """ + openssl_run_make(openssldir, filelist) + for filename in filelist: + src =3D os.path.join(openssldir, filename) + if is_asm(filename): + dst =3D os.path.join(opensslgendir, asm, filename) + else: + dst =3D os.path.join(opensslgendir, filename) + os.makedirs(os.path.dirname(dst), exist_ok =3D True) + shutil.copyfile(src, dst) + +def generate_include_files(openssldir, opensslgendir, asm, cfg): + """ Generate openssl include files """ + print('# generate include files') + filelist =3D cfg['unified_info']['generate'].keys() + filelist =3D list(filter(lambda f: 'include' in f, filelist)) + generate_files(openssldir, opensslgendir, asm, filelist) + +def generate_library_files(openssldir, opensslgendir, asm, cfg, obj): + """ + Generate openssl source files for a given library. Handles + mostly assembler files, but a few C sources are generated too. + """ + filelist =3D get_source_list(cfg, obj, True) + if filelist: + print(f'# generate source files for {obj}') + generate_files(openssldir, opensslgendir, asm, filelist) + +def generate_all_files(openssldir, opensslgendir, asm, cfg): + """ Generate all files needed. """ + generate_include_files(openssldir, opensslgendir, asm, cfg) + generate_library_files(openssldir, opensslgendir, asm, cfg, 'libcrypto= ') + generate_library_files(openssldir, opensslgendir, asm, cfg, 'providers= /libcommon.a') + generate_library_files(openssldir, opensslgendir, asm, cfg, 'libssl') + +def get_source_list(cfg, obj, gen): + """ + Gets the list of source files needed to create a specific object. + * If 'gen' is True the function returns the list of generated + files. + * If 'gen' is False the function returns the list of files not + generated (which are used from the submodule directly). + Note: Will call itself recursively to resolve nested dependencies. + """ + sources =3D cfg['unified_info']['sources'] + generate =3D cfg['unified_info']['generate'] + srclist =3D [] + if sources.get(obj): + for item in sources.get(obj): + srclist +=3D get_source_list(cfg, item, gen) + else: + is_generated =3D generate.get(obj) is not None + if is_generated =3D=3D gen: + srclist +=3D [ obj, ] + return srclist + +def get_sources(cfg, obj, asm): + """ + Get the list of all sources files. Will fetch both generated + and not generated file lists and update the paths accordingly, so + the openssl submodule or the sub-tree for generated files is + referenced as needed. + """ + srclist =3D get_source_list(cfg, obj, False) + genlist =3D get_source_list(cfg, obj, True) + srclist =3D list(map(lambda x: f'$(OPENSSL_PATH)/{x}', srclist)) + c_list =3D list(map(lambda x: f'$(OPENSSL_GEN_PATH)/{x}', + filter(lambda x: not is_asm(x), genlist))) + asm_list =3D list(map(lambda x: f'$(OPENSSL_GEN_PATH)/{asm}/{x}', + filter(is_asm, genlist))) + return srclist + c_list + asm_list + +def sources_filter_fn(filename): + """ + Filter source lists. Drops files we don't want include or + need replace with our own uefi-specific version. + """ + exclude =3D [ + 'randfile.c', + '/store/', + '/storemgmt/', + ] + for item in exclude: + if item in filename: + return False + return True + +def hash_filter_fn(filename): + """ + Filter source lists. Include source files with hash functions only. + """ + include =3D [ + '/sha/', + '/sm3/', + 'mem_clr.c', + ] + exclude =3D [ + 'sha1_one.c', + ] + for item in exclude: + if item in filename: + return False + for item in include: + if item in filename: + return True + return False + +def libcrypto_sources(cfg, asm =3D None): + """ Get source file list for libcrypto """ + files =3D get_sources(cfg, 'libcrypto', asm) + files +=3D get_sources(cfg, 'providers/libcommon.a', asm) + files =3D list(filter(sources_filter_fn, files)) + return files + +def libssl_sources(cfg, asm =3D None): + """ Get source file list for libssl """ + files =3D get_sources(cfg, 'libssl', asm) + files =3D list(filter(sources_filter_fn, files)) + return files + +def hash_sources(cfg, asm =3D None): + """ Get source file list for hash functions """ + files =3D get_sources(cfg, 'libcrypto', asm) + files =3D list(filter(hash_filter_fn, files)) + return files + +def update_inf(filename, sources, arch =3D None, defines =3D []): + """ + Update inf file, replace source file list and build flags. + """ + head =3D '' + tail =3D '' + state =3D 0 + + if arch: + section =3D f'Sources.{arch}' + flags =3D f'OPENSSL_FLAGS_{arch}' + else: + section =3D None + flags =3D f'OPENSSL_FLAGS_NOASM' + state =3D 1 + + # read and parse file + with open(filename, 'r') as f: + while True: + line =3D f.readline() + if line =3D=3D '': + break + if state in [0, 1]: + if flags in line: + (keep, replace) =3D line.split('=3D') + args =3D map(lambda x: f'-D{x}', defines) + head +=3D keep + '=3D ' + ' '.join(args) + '\r\n' + else: + head +=3D line.rstrip() + '\r\n' + if state =3D=3D 0 and section in line: + state =3D 1 + if state =3D=3D 1 and 'Autogenerated files list starts here' i= n line: + state =3D 2 + if state =3D=3D 2 and 'Autogenerated files list ends here' in = line: + state =3D 3 + if state =3D=3D 3: + tail +=3D line.rstrip() + '\r\n' + + # write updated file + with open(filename, 'w') as f: + f.write(head) + for src in sources: + f.write(f' {src}\r\n') + f.write(tail) + +def main(): + # prepare + os.chdir(os.path.dirname(__file__)) + openssldir =3D os.path.join(os.getcwd(), 'openssl') + opensslgendir =3D os.path.join(os.getcwd(), 'openssl-gen') + + # asm accel configs (see UefiAsm.conf) + for ec in [True, False]: + if ec: + inf =3D 'OpensslLibFullAccel.inf' + hdr =3D 'configuration-ec.h' + else: + inf =3D 'OpensslLibAccel.inf' + hdr =3D 'configuration-noec.h' + sources =3D {} + defines =3D {} + for asm in [ 'UEFI-IA32-MSFT', 'UEFI-IA32-GCC', + 'UEFI-X64-MSFT', 'UEFI-X64-GCC', + 'UEFI-AARCH64-GCC' ]: + (uefi, arch, cc) =3D asm.split('-') + archcc =3D f'{arch}-{cc}' + + openssl_configure(openssldir, asm, ec =3D ec); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, archcc, cfg) + shutil.move(os.path.join(opensslgendir, 'include', 'openssl', = 'configuration.h'), + os.path.join(opensslgendir, 'include', 'openssl', = hdr)) + openssl_run_make(openssldir, 'distclean') + + srclist =3D libcrypto_sources(cfg, archcc) + libssl_sources(cf= g, archcc) + sources[archcc] =3D list(map(lambda x: f'{x} | {cc}', filter(i= s_asm, srclist))) + sources[arch] =3D list(filter(lambda x: not is_asm(x), srclist= )) + defines[arch] =3D cfg['unified_info']['defines']['libcrypto'] + + ia32accel =3D sources['IA32'] + sources['IA32-MSFT'] + sources['IA= 32-GCC'] + x64accel =3D sources['X64'] + sources['X64-MSFT'] + sources['X64-G= CC'] + aa64accel =3D sources['AARCH64'] + sources['AARCH64-GCC'] + update_inf(inf, ia32accel, 'IA32', defines['IA32']) + update_inf(inf, x64accel, 'X64', defines['X64']) + update_inf(inf, aa64accel, 'AARCH64', defines['AARCH64']) + + # noaccel - ec enabled + openssl_configure(openssldir, 'UEFI', ec =3D True); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, None, cfg) + openssl_run_make(openssldir, 'distclean') + + update_inf('OpensslLibFull.inf', + libcrypto_sources(cfg) + libssl_sources(cfg), + None, cfg['unified_info']['defines']['libcrypto']) + + # noaccel - ec disabled + openssl_configure(openssldir, 'UEFI', ec =3D False); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, None, cfg) + openssl_run_make(openssldir, 'distclean') + + update_inf('OpensslLibCrypto.inf', + libcrypto_sources(cfg), + None, cfg['unified_info']['defines']['libcrypto']) + update_inf('OpensslLib.inf', + libcrypto_sources(cfg) + libssl_sources(cfg), + None, cfg['unified_info']['defines']['libcrypto']) + update_inf('OpensslLibHash.inf', + hash_sources(cfg), + None, cfg['unified_info']['defines']['libcrypto']) + + # wrap header file + confighdr =3D os.path.join(opensslgendir, 'include', 'openssl', 'confi= guration.h') + with open(confighdr, 'w') as f: + f.write('#ifdef EDK2_OPENSSL_NOEC\n' + '# include "configuration-noec.h"\n' + '#else\n' + '# include "configuration-ec.h"\n' + '#endif\n') + +if __name__ =3D=3D '__main__': + sys.exit(main()) diff --git a/CryptoPkg/Library/OpensslLib/perl2json.pl b/CryptoPkg/Library/= OpensslLib/perl2json.pl new file mode 100755 index 000000000000..f7364596a73d --- /dev/null +++ b/CryptoPkg/Library/OpensslLib/perl2json.pl @@ -0,0 +1,19 @@ +#!/usr/bin/perl +# +# write out configdata.pm as json +# +use strict; +use warnings; +use JSON; + +BEGIN { + my $openssldir =3D shift; + push @INC, $openssldir; +} +use configdata qw/%config %target %unified_info/; + +my %data; +$data{'config'} =3D \%config; +$data{'target'} =3D \%target; +$data{'unified_info'} =3D \%unified_info; +print encode_json(\%data) --=20 2.39.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#101064): https://edk2.groups.io/g/devel/message/101064 Mute This Topic: https://groups.io/mt/97576409/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-