From nobody Tue Dec 16 07:08:44 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+107343+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+107343+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1690526458; cv=none; d=zohomail.com; s=zohoarc; b=JrZ9g8ACZXlH29b97j/9TekS4Wn80Rmijy7noBjGVRyU3llsegM4Ij4KErXmhHb51jT8QmGSjJ6Mf/18d8MBMKMDRf+f+9FO05n/8i+LTBQ3Z4b/LcWewpCE7YzZCNcwFvC9cObgrAVqjguQK3OJRwdahREliln1Fk1R6cxmO2I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1690526458; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=JAfGT7Ur6OeRUUihsSU4ozMgeWA8IswAgCsDYwKJL2c=; b=J/6i1BBBPaZxftIaDOPJr6hZ8hgnCf2lU6QzyX9j7CjKolNgMbA2ehOU7Qtz9F9mYEEx9K+Le9JCplTLA6OtrVG566Yoo8JA6eSZ2enJY2tIaJPmKRcKf+Gd9JDAXCnvz+VZtBwLZsYsOso3kMVRMfzREMImAOTYa/kYdJuMqUM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+107343+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1690526458487364.0037305714551; Thu, 27 Jul 2023 23:40:58 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=mlWeKtg57V43RTO0U78QD8HrIiQJXFLcpd9MYa18MTU=; c=relaxed/simple; d=groups.io; h=X-Received:X-Received:X-IronPort-AV:X-IronPort-AV:X-Received:X-ExtLoop1:X-IronPort-AV:X-IronPort-AV:X-Received:From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:X-Gm-Message-State:Content-Transfer-Encoding; s=20140610; t=1690526458; v=1; b=U8WDgnN4mr5rJm+YL6JEIJ49RfD+g4BaHAoKAUkBjWyqN4OyDBQinj8e+H/+dVCqDaXOBhCX FgbS4JJXYmbc+iSWQo9KiVaPK4/hI+0gJ0WbY1Kq1Wwxi0lzteda/6FvZV5NH4DHevFQlC45a4Z bFn6HqYibHSAU8CdQub5+daQ= X-Received: by 127.0.0.2 with SMTP id 7ItiYY1788612xF0M2wWwIXA; Thu, 27 Jul 2023 23:40:58 -0700 X-Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by mx.groups.io with SMTP id smtpd.web10.27081.1690526456058870373 for ; Thu, 27 Jul 2023 23:40:56 -0700 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="434804183" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="434804183" X-Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 23:40:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="762495926" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="762495926" X-Received: from liyi4-desktop.ccr.corp.intel.com ([10.239.153.10]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Jul 2023 23:40:51 -0700 From: "Li, Yi" To: devel@edk2.groups.io Cc: Gerd Hoffmann , Yi Li , Jiewen Yao , Xiaoyu Lu , Guomin Jiang Subject: [edk2-devel] [PATCH 04/29] CryptoPkg/openssl: add openssl3 configure scripts Date: Fri, 28 Jul 2023 14:39:50 +0800 Message-Id: <473d3ecdd4f2bbc48889d81702b439d94dd2abd3.1690444292.git.yi1.li@intel.com> In-Reply-To: References: MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,yi1.li@intel.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: XKIvU7koWGkaTldBKckO03Tvx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1690526459997100009 Content-Type: text/plain; charset="utf-8" From: Gerd Hoffmann Rewrite the script to configure openssl 3.0 from scratch. It's two scripts now: * Tiny helper script, dumping the perl configdata as json. * Actual configure.py script, written in python, which copies over the generated files to openssl-gen and updates the OpensslLib*.inf file lists and build flags. The configuration workflow has changed a bit: * All generated files are stored in the OpensslGen directory tree. * For ec/no-ec builds two different header files are used. Default is the ec variant, and the new EDK2_OPENSSL_NOEC define is used to select the no-ec build. A five line wrapper include is used to pick the one or the other. * For non-accel builds -DOPENSSL_NO_ASM on the command line is used (same as before). * For configration defines the OPENSSL_FLAGS_$(variant) variable is used, where variant is the architecture for the accelerated builds and 'NOASM' for the non-accelerated builds. Signed-off-by: Gerd Hoffmann Signed-off-by: Yi Li Cc: Jiewen Yao Cc: Xiaoyu Lu Cc: Guomin Jiang --- .../Library/OpensslLib/OpenSSL-HOWTO.txt | 4 +- CryptoPkg/Library/OpensslLib/configure.py | 395 ++++++++++++++++++ CryptoPkg/Library/OpensslLib/perl2json.pl | 19 + 3 files changed, 416 insertions(+), 2 deletions(-) create mode 100755 CryptoPkg/Library/OpensslLib/configure.py create mode 100755 CryptoPkg/Library/OpensslLib/perl2json.pl diff --git a/CryptoPkg/Library/OpensslLib/OpenSSL-HOWTO.txt b/CryptoPkg/Lib= rary/OpensslLib/OpenSSL-HOWTO.txt index e52ee27b49..fff47d9f6b 100644 --- a/CryptoPkg/Library/OpensslLib/OpenSSL-HOWTO.txt +++ b/CryptoPkg/Library/OpensslLib/OpenSSL-HOWTO.txt @@ -28,9 +28,9 @@ on the cryptography. refer to edk2/Readme.md for how to clone the code. =20 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D - About process_files.pl + About configure.py =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D - "process_files.pl" is one Perl script which runs the OpenSSL Configure, + "configure.py" is one python script which runs the OpenSSL Configure, then processes the resulting file list into our local OpensslLib.inf and OpensslLibCrypto.inf. This only needs to be done once by the maintainer / developer when diff --git a/CryptoPkg/Library/OpensslLib/configure.py b/CryptoPkg/Library/= OpensslLib/configure.py new file mode 100755 index 0000000000..fc7f16ddb9 --- /dev/null +++ b/CryptoPkg/Library/OpensslLib/configure.py @@ -0,0 +1,395 @@ +#!/usr/bin/python3 +# SPDX-License-Identifier: BSD-2-Clause-Patent +import os +import sys +import json +import shutil +import pprint +import argparse +import subprocess + +def openssl_configure(openssldir, target, ec =3D True): + """ Run openssl Configure script. """ + cmdline =3D [ + 'perl', + 'Configure', + '--config=3D../UefiAsm.conf', + '--api=3D1.1.1', + '--with-rand-seed=3Dnone', + target, + 'no-afalgeng', + 'no-aria', + 'no-async', + 'no-autoerrinit', + 'no-autoload-config', + 'no-bf', + 'no-blake2', + 'no-camellia', + 'no-capieng', + 'no-cast', + 'no-chacha', + 'no-cmac', + 'no-cmp', + 'no-cms', + 'no-ct', + 'no-deprecated', + 'no-des', + 'no-dgram', + 'no-dsa', + 'no-dso', + 'no-dtls', + 'no-dtls1-method', + 'no-dtls1_2-method', + 'no-dynamic-engine', + 'no-ec2m', + 'no-engine', + 'no-err', + 'no-filenames', + 'no-gost', + 'no-hw', + 'no-idea', + 'no-ktls', + 'no-makedepend', + 'no-module', + 'no-md4', + 'no-mdc2', + 'no-multiblock', + 'no-nextprotoneg', + 'no-pic', + 'no-psk', + 'no-ocb', + 'no-ocsp', + 'no-padlockeng', + 'no-poly1305', + 'no-posix-io', + 'no-rc2', + 'no-rc4', + 'no-rc5', + 'no-rfc3779', + 'no-rmd160', + 'no-scrypt', + 'no-seed', + 'no-shared', + 'no-siphash', + 'no-siv', + 'no-sm2', + 'no-sm4', + 'no-sock', + 'no-srp', + 'no-srtp', + 'no-ssl', + 'no-ssl3-method', + 'no-ssl-trace', + 'no-static-engine', + 'no-stdio', + 'no-threads', + 'no-tls1_3', + 'no-ts', + 'no-ui-console', + 'no-whirlpool', + 'disable-legacy', + ] + if not ec: + cmdline +=3D [ 'no-ec', ] + print('') + print(f'# -*- configure openssl for {target} (ec=3D{ec}) -*-') + rc =3D subprocess.run(cmdline, cwd =3D openssldir, + stdout =3D subprocess.PIPE, + stderr =3D subprocess.PIPE) + if rc.returncode: + print(rc.stdout) + print(rc.stderr) + sys.exit(rc.returncode) + +def openssl_run_make(openssldir, target): + """ + Run make utility to generate files or cleanup. + Target can be either a string or a list of strings. + """ + cmdline =3D [ 'make', '--silent' ] + if isinstance(target, list): + cmdline +=3D target + else: + cmdline +=3D [ target, ] + rc =3D subprocess.run(cmdline, cwd =3D openssldir) + rc.check_returncode() + +def get_configdata(openssldir): + """ + Slurp openssl config data as JSON, + using a little perl helper script. + """ + cmdline =3D [ + 'perl', + 'perl2json.pl', + openssldir, + ] + rc =3D subprocess.run(cmdline, stdout =3D subprocess.PIPE) + rc.check_returncode() + return json.loads(rc.stdout) + +def is_asm(filename): + """ Check whenevr the passed file is an assembler file """ + if filename.endswith('.s') or filename.endswith('.S'): + return True + return False + +def copy_generated_file(src, dst): + src_file =3D [] + with open(src, 'r') as fsrc: + src_file =3D fsrc.readlines() + with open(dst, 'w') as fdst: + for lines in range(len(src_file)): + s =3D src_file[lines] + s =3D s.rstrip() + "\r\n" + fdst.write(s.expandtabs()) + +def generate_files(openssldir, opensslgendir, asm, filelist): + """ + Generate files, using make, and copy over the results to the + directory tree for generated openssl files. Creates + subdirectories as needed. + """ + openssl_run_make(openssldir, filelist) + for filename in filelist: + src =3D os.path.join(openssldir, filename) + if is_asm(filename): + """ rename MSFT asm files to .nasm """ + if 'IA32-MSFT' in asm: + filename =3D filename.replace('.S', '.nasm') + elif 'X64-MSFT' in asm: + filename =3D filename.replace('.s', '.nasm') + dst =3D os.path.join(opensslgendir, asm, filename) + else: + dst =3D os.path.join(opensslgendir, filename) + os.makedirs(os.path.dirname(dst), exist_ok =3D True) + copy_generated_file(src, dst) + +def generate_include_files(openssldir, opensslgendir, asm, cfg): + """ Generate openssl include files """ + print('# generate include files') + filelist =3D cfg['unified_info']['generate'].keys() + filelist =3D list(filter(lambda f: 'include' in f, filelist)) + generate_files(openssldir, opensslgendir, asm, filelist) + +def generate_library_files(openssldir, opensslgendir, asm, cfg, obj): + """ + Generate openssl source files for a given library. Handles + mostly assembler files, but a few C sources are generated too. + """ + filelist =3D get_source_list(cfg, obj, True) + if filelist: + print(f'# generate source files for {obj}') + generate_files(openssldir, opensslgendir, asm, filelist) + +def generate_all_files(openssldir, opensslgendir, asm, cfg): + """ Generate all files needed. """ + generate_include_files(openssldir, opensslgendir, asm, cfg) + generate_library_files(openssldir, opensslgendir, asm, cfg, 'libcrypto= ') + generate_library_files(openssldir, opensslgendir, asm, cfg, 'providers= /libcommon.a') + generate_library_files(openssldir, opensslgendir, asm, cfg, 'libssl') + +def get_source_list(cfg, obj, gen): + """ + Gets the list of source files needed to create a specific object. + * If 'gen' is True the function returns the list of generated + files. + * If 'gen' is False the function returns the list of files not + generated (which are used from the submodule directly). + Note: Will call itself recursively to resolve nested dependencies. + """ + sources =3D cfg['unified_info']['sources'] + generate =3D cfg['unified_info']['generate'] + srclist =3D [] + if sources.get(obj): + for item in sources.get(obj): + srclist +=3D get_source_list(cfg, item, gen) + else: + is_generated =3D generate.get(obj) is not None + if is_generated =3D=3D gen: + srclist +=3D [ obj, ] + return srclist + +def get_sources(cfg, obj, asm): + """ + Get the list of all sources files. Will fetch both generated + and not generated file lists and update the paths accordingly, so + the openssl submodule or the sub-tree for generated files is + referenced as needed. + """ + srclist =3D get_source_list(cfg, obj, False) + genlist =3D get_source_list(cfg, obj, True) + srclist =3D list(map(lambda x: f'$(OPENSSL_PATH)/{x}', srclist)) + c_list =3D list(map(lambda x: f'$(OPENSSL_GEN_PATH)/{x}', + filter(lambda x: not is_asm(x), genlist))) + asm_list =3D list(map(lambda x: f'$(OPENSSL_GEN_PATH)/{asm}/{x}', + filter(is_asm, genlist))) + return srclist + c_list + asm_list + +def sources_filter_fn(filename): + """ + Filter source lists. Drops files we don't want include or + need replace with our own uefi-specific version. + """ + exclude =3D [ + 'randfile.c', + '/store/', + '/storemgmt/', + '/encode_decode/encode', + '/pkcs12/', + 'statem_srvr.c', + 'extensions_srvr.c', + 'defltprov.c', + 'baseprov.c', + 'provider_predefined.c', + ] + for item in exclude: + if item in filename: + return False + return True + +def libcrypto_sources(cfg, asm =3D None): + """ Get source file list for libcrypto """ + files =3D get_sources(cfg, 'libcrypto', asm) + files +=3D get_sources(cfg, 'providers/libcommon.a', asm) + files =3D list(filter(sources_filter_fn, files)) + return files + +def libssl_sources(cfg, asm =3D None): + """ Get source file list for libssl """ + files =3D get_sources(cfg, 'libssl', asm) + files =3D list(filter(sources_filter_fn, files)) + return files + +def update_inf(filename, sources, arch =3D None, defines =3D []): + """ + Update inf file, replace source file list and build flags. + """ + head =3D '' + tail =3D '' + state =3D 0 + + if arch: + section =3D f'Sources.{arch}' + flags =3D f'OPENSSL_FLAGS_{arch}' + else: + section =3D None + flags =3D f'OPENSSL_FLAGS_NOASM' + state =3D 1 + + # read and parse file + with open(filename, 'r') as f: + while True: + line =3D f.readline() + if line =3D=3D '': + break + if state in [0, 1]: + if flags in line: + (keep, replace) =3D line.split('=3D') + args =3D map(lambda x: f'-D{x}', defines) + head +=3D keep + '=3D ' + ' '.join(args) + head =3D head.rstrip() + '\r\n' + else: + head +=3D line.rstrip() + '\r\n' + if state =3D=3D 0 and section in line: + state =3D 1 + if state =3D=3D 1 and 'Autogenerated files list starts here' i= n line: + state =3D 2 + if state =3D=3D 2 and 'Autogenerated files list ends here' in = line: + state =3D 3 + if state =3D=3D 3: + tail +=3D line.rstrip() + '\r\n' + + # write updated file + with open(filename, 'w') as f: + f.write(head) + for src in sources: + f.write(f' {src}\r\n') + f.write(tail) + +def update_MSFT_asm_format(asm, filelist): + """ rename MSFT asm files to .nasm """ + if 'IA32-MSFT' in asm: + for file_index in range(len(filelist)): + filelist[file_index] =3D filelist[file_index].replace('.S', '.= nasm') + elif 'X64-MSFT' in asm: + for file_index in range(len(filelist)): + filelist[file_index] =3D filelist[file_index].replace('.s', '.= nasm') + +def main(): + # prepare + os.chdir(os.path.dirname(os.path.abspath(__file__))) + openssldir =3D os.path.join(os.getcwd(), 'openssl') + opensslgendir =3D os.path.join(os.getcwd(), 'OpensslGen') + + # asm accel configs (see UefiAsm.conf) + for ec in [True, False]: + if ec: + inf =3D 'OpensslLibFullAccel.inf' + hdr =3D 'configuration-ec.h' + else: + inf =3D 'OpensslLibAccel.inf' + hdr =3D 'configuration-noec.h' + sources =3D {} + defines =3D {} + for asm in [ 'UEFI-IA32-MSFT', 'UEFI-IA32-GCC', + 'UEFI-X64-MSFT', 'UEFI-X64-GCC']: + (uefi, arch, cc) =3D asm.split('-') + archcc =3D f'{arch}-{cc}' + + openssl_configure(openssldir, asm, ec =3D ec); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, archcc, cfg) + shutil.move(os.path.join(opensslgendir, 'include', 'openssl', = 'configuration.h'), + os.path.join(opensslgendir, 'include', 'openssl', = hdr)) + openssl_run_make(openssldir, 'distclean') + + srclist =3D libcrypto_sources(cfg, archcc) + libssl_sources(cf= g, archcc) + sources[archcc] =3D list(map(lambda x: f'{x} | {cc}', filter(i= s_asm, srclist))) + update_MSFT_asm_format(archcc, sources[archcc]) + sources[arch] =3D list(filter(lambda x: not is_asm(x), srclist= )) + defines[arch] =3D cfg['unified_info']['defines']['libcrypto'] + + ia32accel =3D sources['IA32'] + sources['IA32-MSFT'] + sources['IA= 32-GCC'] + x64accel =3D sources['X64'] + sources['X64-MSFT'] + sources['X64-G= CC'] + update_inf(inf, ia32accel, 'IA32', defines['IA32']) + update_inf(inf, x64accel, 'X64', defines['X64']) + + # noaccel - ec enabled + openssl_configure(openssldir, 'UEFI', ec =3D True); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, None, cfg) + openssl_run_make(openssldir, 'distclean') + + defines =3D [] + if 'libcrypto' in cfg['unified_info']['defines']: + defines =3D cfg['unified_info']['defines']['libcrypto'] + + update_inf('OpensslLibFull.inf', + libcrypto_sources(cfg) + libssl_sources(cfg), + defines) + + # noaccel - ec disabled + openssl_configure(openssldir, 'UEFI', ec =3D False); + cfg =3D get_configdata(openssldir) + generate_all_files(openssldir, opensslgendir, None, cfg) + openssl_run_make(openssldir, 'distclean') + + update_inf('OpensslLibCrypto.inf', + libcrypto_sources(cfg), + None, defines) + update_inf('OpensslLib.inf', + libcrypto_sources(cfg) + libssl_sources(cfg), + None, defines) + + # wrap header file + confighdr =3D os.path.join(opensslgendir, 'include', 'openssl', 'confi= guration.h') + with open(confighdr, 'w') as f: + f.write('#ifdef EDK2_OPENSSL_NOEC\r\n' + '# include "configuration-noec.h"\r\n' + '#else\r\n' + '# include "configuration-ec.h"\r\n' + '#endif\r\n') + +if __name__ =3D=3D '__main__': + sys.exit(main()) diff --git a/CryptoPkg/Library/OpensslLib/perl2json.pl b/CryptoPkg/Library/= OpensslLib/perl2json.pl new file mode 100755 index 0000000000..e3cf29d63c --- /dev/null +++ b/CryptoPkg/Library/OpensslLib/perl2json.pl @@ -0,0 +1,19 @@ +#!/usr/bin/perl +# +# write out configdata.pm as json +# +use strict; +use warnings; +use JSON; + +BEGIN { + my $openssldir =3D shift; + push @INC, $openssldir; +} +use configdata qw/%config %target %unified_info/; + +my %data; +$data{'config'} =3D \%config; +$data{'target'} =3D \%target; +$data{'unified_info'} =3D \%unified_info; +print encode_json(\%data) --=20 2.31.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#107343): https://edk2.groups.io/g/devel/message/107343 Mute This Topic: https://groups.io/mt/100406038/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-