[v3] New Cross OS tool chain CLANG9

[edk2-devel] [Patch v3 00/11] New Cross OS tool chain CLANG9

Posted by Liming Gao 6 years, 3 months ago

In v3, add the detail commit message for patch 3. 
Update the fix in EmulatorPkg based on the comments.

In v2, drop patch 12, and update commit message for 5 & 11.

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1603
Code: https://github.com/lgao4/edk2/tree/CLANG9
Wiki: https://github.com/lgao4/edk2/wiki/CLANG9-Tools-Chain

CLANG9 tool chain is added to directly generate PE/COFF image (EFI image).
This tool chain uses LLVM clang C compiler and lld linker, generates PE/COFF
image and PDB compatible debug symbol format. Now, it supports IA32/X64 Archs.
It must use LLVM 9 or above release. LLVM 9 is ready on 
http://releases.llvm.org/download.html#9.0.0.

CLANG9 is the cross OS tool chain. It can work on Windows/Linux/Mac host OS. 
For the same source code, with the same version LLVM tool chain, 
CLANG9 can generate the same binary image. So, the developer can 
choose the different development environment and work on the same 
code base. Besides, EDKII project build also requires third party 
tools: nasm and iasl. They both keep the same version. If so, the same 
binary image can be generated on the different host OS.

LLVM tool chain provides the compiler and linker. To build EDK2 project, 
some other tools are still required. On Windows OS, nmake and Visual Studio 
are required to call Makefile and compile BaseTools C tools. 
On Linux/Mac, binutils and gcc are required to make and compile BaseTools 
C tools. Because VS or GCC are mainly used to compile BaseTools and provide 
nmake/make tool, they can keep on the stable version without update.

To build source code, CLANG9 tool chain (-t CLANG9) can be specified
on Windows OS, set CLANG_HOST_BIN=n, set CLANG9_BIN=LLVM installed directory
CLANG_HOST_BIN is used CLANG_HOST_PREFIX. Prefix n is for nmake.
For example:
*  set CLANG_HOST_BIN=n
*  set CLANG9_BIN=C:\Program Files\LLVM\bin\
*  set IASL_PREFIX=C:\Asl\

On Linux/Mac, export CLANG9_BIN=LLVM installed directory, CLANG_HOST_BIN is 
not required, because there is no prefix for make.
For example:
*  export CLANG9_BIN=/home/clang9/bin/

Now, CLANG9 tool chain has been verified in Edk2 packages and Ovmf/Emulator
with LLVM 9.0.0 on Windows and Linux OS. 
OVMF IA32/X64/IA32X64 all boots to Shell on Windows and Linux OS. 
Emulator can boot to Shell on Windows only with CLANG9.
OVMF Ia32X64 RELEASE build generates the same BIOS images on Windows and Linux OS.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>

Liming Gao (11):
  BaseTools tools_def.template: Remove unnecessary $(DEST_DIR_DEBUG)
    path
  BaseTools tools_def: Add CLANG9 tool chain to directly generate PE
    image
  BaseTools GenFw: Fix the issue to update the wrong size as SectionSize
  MdePkg Base.h: Add definition for CLANG9 tool chain
  MdePkg BaseIoLibIntrinsic: Remove __inline__ attribute for IO
    functions
  MdeModulePkg LzmaCustomDecompressLib: Update macro to be same in CLANG
    tool
  MdeModulePkg RegularExpressionDxe: Disable warning for CLANG9 tool
    chain
  CryptoPkg: Append options to make CLANG9 tool chain pass build
  CryptoPkg IntrinsicLib: Make _fltused always be used
  EmulatorPkg: Enable CLANG9 tool chain
  OvmfPkg: Enable CLANG9 tool chain

 BaseTools/Source/C/GenFw/GenFw.c                   |   8 +-
 CryptoPkg/Library/IntrinsicLib/MemoryIntrinsics.c  |  10 +-
 EmulatorPkg/Win/Host/WinHost.c                     |   6 +-
 MdePkg/Library/BaseIoLibIntrinsic/IoLibGcc.c       |   6 -
 BaseTools/Conf/build_rule.template                 |  26 +++--
 BaseTools/Conf/tools_def.template                  | 124 +++++++++++++++++++--
 CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf    |   1 +
 CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf     |   1 +
 CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf |   1 +
 CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf     |   1 +
 CryptoPkg/Library/OpensslLib/OpensslLib.inf        |   1 +
 CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf  |   1 +
 EmulatorPkg/EmulatorPkg.dsc                        |   7 +-
 EmulatorPkg/Win/Host/WinHost.inf                   |   6 +
 .../LzmaCustomDecompressLib/Sdk/C/7zTypes.h        |   2 +-
 .../RegularExpressionDxe/RegularExpressionDxe.inf  |   3 +
 MdePkg/Include/Base.h                              |   6 +-
 MdePkg/Include/Ia32/ProcessorBind.h                |   4 +-
 MdePkg/Include/X64/ProcessorBind.h                 |   2 +-
 OvmfPkg/OvmfPkgIa32.dsc                            |   4 +-
 OvmfPkg/OvmfPkgIa32X64.dsc                         |   4 +-
 OvmfPkg/OvmfPkgX64.dsc                             |   4 +-
 22 files changed, 183 insertions(+), 45 deletions(-)

-- 
2.13.0.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#49157): https://edk2.groups.io/g/devel/message/49157
Mute This Topic: https://groups.io/mt/34694424/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch v3 00/11] New Cross OS tool chain CLANG9

Posted by Leif Lindholm 6 years, 3 months ago

Hi Liming,

I had missed that this set was out for review (I would appreciate a cc
on any future revisions).

On Thu, Oct 17, 2019 at 02:55:44PM +0800, Liming Gao wrote:
> In v3, add the detail commit message for patch 3. 
> Update the fix in EmulatorPkg based on the comments.
> 
> In v2, drop patch 12, and update commit message for 5 & 11.
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1603
> Code: https://github.com/lgao4/edk2/tree/CLANG9

This branch does not appear to have been updated since 5 September, so
it does not contain the version of the code sent out in this set. (And
I can find no other obvious branch in that repository that may hold
it.) Could you push an updated one please?

One thing that strikes me as problematic with the version on the
branch is that the set imports bits from GCC48 *and* GCC5. That seems
like a maintenance hazard, but may no longer be the case for the
current version of the set?

Best Regards,

Leif

> Wiki: https://github.com/lgao4/edk2/wiki/CLANG9-Tools-Chain
> 
> CLANG9 tool chain is added to directly generate PE/COFF image (EFI image).
> This tool chain uses LLVM clang C compiler and lld linker, generates PE/COFF
> image and PDB compatible debug symbol format. Now, it supports IA32/X64 Archs.
> It must use LLVM 9 or above release. LLVM 9 is ready on 
> http://releases.llvm.org/download.html#9.0.0.
> 
> CLANG9 is the cross OS tool chain. It can work on Windows/Linux/Mac host OS. 
> For the same source code, with the same version LLVM tool chain, 
> CLANG9 can generate the same binary image. So, the developer can 
> choose the different development environment and work on the same 
> code base. Besides, EDKII project build also requires third party 
> tools: nasm and iasl. They both keep the same version. If so, the same 
> binary image can be generated on the different host OS.
> 
> LLVM tool chain provides the compiler and linker. To build EDK2 project, 
> some other tools are still required. On Windows OS, nmake and Visual Studio 
> are required to call Makefile and compile BaseTools C tools. 
> On Linux/Mac, binutils and gcc are required to make and compile BaseTools 
> C tools. Because VS or GCC are mainly used to compile BaseTools and provide 
> nmake/make tool, they can keep on the stable version without update.
> 
> To build source code, CLANG9 tool chain (-t CLANG9) can be specified
> on Windows OS, set CLANG_HOST_BIN=n, set CLANG9_BIN=LLVM installed directory
> CLANG_HOST_BIN is used CLANG_HOST_PREFIX. Prefix n is for nmake.
> For example:
> *  set CLANG_HOST_BIN=n
> *  set CLANG9_BIN=C:\Program Files\LLVM\bin\
> *  set IASL_PREFIX=C:\Asl\
> 
> On Linux/Mac, export CLANG9_BIN=LLVM installed directory, CLANG_HOST_BIN is 
> not required, because there is no prefix for make.
> For example:
> *  export CLANG9_BIN=/home/clang9/bin/
> 
> Now, CLANG9 tool chain has been verified in Edk2 packages and Ovmf/Emulator
> with LLVM 9.0.0 on Windows and Linux OS. 
> OVMF IA32/X64/IA32X64 all boots to Shell on Windows and Linux OS. 
> Emulator can boot to Shell on Windows only with CLANG9.
> OVMF Ia32X64 RELEASE build generates the same BIOS images on Windows and Linux OS.
> 
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Steven Shi <steven.shi@intel.com>
> Cc: Jordan Justen <jordan.l.justen@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Andrew Fish <afish@apple.com>
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> Cc: Jian J Wang <jian.j.wang@intel.com>
> Cc: Hao A Wu <hao.a.wu@intel.com>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> 
> Liming Gao (11):
>   BaseTools tools_def.template: Remove unnecessary $(DEST_DIR_DEBUG)
>     path
>   BaseTools tools_def: Add CLANG9 tool chain to directly generate PE
>     image
>   BaseTools GenFw: Fix the issue to update the wrong size as SectionSize
>   MdePkg Base.h: Add definition for CLANG9 tool chain
>   MdePkg BaseIoLibIntrinsic: Remove __inline__ attribute for IO
>     functions
>   MdeModulePkg LzmaCustomDecompressLib: Update macro to be same in CLANG
>     tool
>   MdeModulePkg RegularExpressionDxe: Disable warning for CLANG9 tool
>     chain
>   CryptoPkg: Append options to make CLANG9 tool chain pass build
>   CryptoPkg IntrinsicLib: Make _fltused always be used
>   EmulatorPkg: Enable CLANG9 tool chain
>   OvmfPkg: Enable CLANG9 tool chain
> 
>  BaseTools/Source/C/GenFw/GenFw.c                   |   8 +-
>  CryptoPkg/Library/IntrinsicLib/MemoryIntrinsics.c  |  10 +-
>  EmulatorPkg/Win/Host/WinHost.c                     |   6 +-
>  MdePkg/Library/BaseIoLibIntrinsic/IoLibGcc.c       |   6 -
>  BaseTools/Conf/build_rule.template                 |  26 +++--
>  BaseTools/Conf/tools_def.template                  | 124 +++++++++++++++++++--
>  CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf    |   1 +
>  CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf     |   1 +
>  CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf |   1 +
>  CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf     |   1 +
>  CryptoPkg/Library/OpensslLib/OpensslLib.inf        |   1 +
>  CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf  |   1 +
>  EmulatorPkg/EmulatorPkg.dsc                        |   7 +-
>  EmulatorPkg/Win/Host/WinHost.inf                   |   6 +
>  .../LzmaCustomDecompressLib/Sdk/C/7zTypes.h        |   2 +-
>  .../RegularExpressionDxe/RegularExpressionDxe.inf  |   3 +
>  MdePkg/Include/Base.h                              |   6 +-
>  MdePkg/Include/Ia32/ProcessorBind.h                |   4 +-
>  MdePkg/Include/X64/ProcessorBind.h                 |   2 +-
>  OvmfPkg/OvmfPkgIa32.dsc                            |   4 +-
>  OvmfPkg/OvmfPkgIa32X64.dsc                         |   4 +-
>  OvmfPkg/OvmfPkgX64.dsc                             |   4 +-
>  22 files changed, 183 insertions(+), 45 deletions(-)
> 
> -- 
> 2.13.0.windows.1
> 
> 
> 
> 

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#49187): https://edk2.groups.io/g/devel/message/49187
Mute This Topic: https://groups.io/mt/34694424/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch v3 00/11] New Cross OS tool chain CLANG9

Posted by Liming Gao 6 years, 3 months ago

Leif:

>-----Original Message-----
>From: Leif Lindholm [mailto:leif.lindholm@linaro.org]
>Sent: Friday, October 18, 2019 12:31 AM
>To: devel@edk2.groups.io; Gao, Liming <liming.gao@intel.com>
>Cc: Yao, Jiewen <jiewen.yao@intel.com>; Shi, Steven
><steven.shi@intel.com>; Justen, Jordan L <jordan.l.justen@intel.com>; Laszlo
>Ersek <lersek@redhat.com>; Andrew Fish <afish@apple.com>; Ni, Ray
><ray.ni@intel.com>; Ard Biesheuvel <ard.biesheuvel@linaro.org>; Wang, Jian
>J <jian.j.wang@intel.com>; Wu, Hao A <hao.a.wu@intel.com>; Feng, Bob C
><bob.c.feng@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
>Subject: Re: [edk2-devel] [Patch v3 00/11] New Cross OS tool chain CLANG9
>
>Hi Liming,
>
>I had missed that this set was out for review (I would appreciate a cc
>on any future revisions).
Yes. 

>
>On Thu, Oct 17, 2019 at 02:55:44PM +0800, Liming Gao wrote:
>> In v3, add the detail commit message for patch 3.
>> Update the fix in EmulatorPkg based on the comments.
>>
>> In v2, drop patch 12, and update commit message for 5 & 11.
>>
>> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1603
>> Code: https://github.com/lgao4/edk2/tree/CLANG9
>
>This branch does not appear to have been updated since 5 September, so
>it does not contain the version of the code sent out in this set. (And
>I can find no other obvious branch in that repository that may hold
>it.) Could you push an updated one please?
>
Done
Code: https://github.com/lgao4/edk2/tree/CLANG9_v3

>One thing that strikes me as problematic with the version on the
>branch is that the set imports bits from GCC48 *and* GCC5. That seems
>like a maintenance hazard, but may no longer be the case for the
>current version of the set?
>
CLANG9 tool chain refers to CLANG38 tool chain. So, it refers to GCC48 and GCC5 flags.
1. CLANG9 CC flags reuses GCC48_ALL_CC_FLAGS. GCC48_ALL_CC_FLAGS is common GCC flags 
for GCC48, GCC49, GCC5 and CLANG38. So, I think it is fine to reuse it in new CLANG9 tool chain. 
2. CLANG9 ASM_FLAGS reuses GCC5_ASM_FLAGS. It is same to common GCC_ASM_FLAGS. I 
Can use GCC_ASM_FLAGS. 

Thanks
Liming
>Best Regards,
>
>Leif
>
>> Wiki: https://github.com/lgao4/edk2/wiki/CLANG9-Tools-Chain
>>
>> CLANG9 tool chain is added to directly generate PE/COFF image (EFI image).
>> This tool chain uses LLVM clang C compiler and lld linker, generates PE/COFF
>> image and PDB compatible debug symbol format. Now, it supports IA32/X64
>Archs.
>> It must use LLVM 9 or above release. LLVM 9 is ready on
>> http://releases.llvm.org/download.html#9.0.0.
>>
>> CLANG9 is the cross OS tool chain. It can work on Windows/Linux/Mac host
>OS.
>> For the same source code, with the same version LLVM tool chain,
>> CLANG9 can generate the same binary image. So, the developer can
>> choose the different development environment and work on the same
>> code base. Besides, EDKII project build also requires third party
>> tools: nasm and iasl. They both keep the same version. If so, the same
>> binary image can be generated on the different host OS.
>>
>> LLVM tool chain provides the compiler and linker. To build EDK2 project,
>> some other tools are still required. On Windows OS, nmake and Visual
>Studio
>> are required to call Makefile and compile BaseTools C tools.
>> On Linux/Mac, binutils and gcc are required to make and compile BaseTools
>> C tools. Because VS or GCC are mainly used to compile BaseTools and
>provide
>> nmake/make tool, they can keep on the stable version without update.
>>
>> To build source code, CLANG9 tool chain (-t CLANG9) can be specified
>> on Windows OS, set CLANG_HOST_BIN=n, set CLANG9_BIN=LLVM installed
>directory
>> CLANG_HOST_BIN is used CLANG_HOST_PREFIX. Prefix n is for nmake.
>> For example:
>> *  set CLANG_HOST_BIN=n
>> *  set CLANG9_BIN=C:\Program Files\LLVM\bin\
>> *  set IASL_PREFIX=C:\Asl\
>>
>> On Linux/Mac, export CLANG9_BIN=LLVM installed directory,
>CLANG_HOST_BIN is
>> not required, because there is no prefix for make.
>> For example:
>> *  export CLANG9_BIN=/home/clang9/bin/
>>
>> Now, CLANG9 tool chain has been verified in Edk2 packages and
>Ovmf/Emulator
>> with LLVM 9.0.0 on Windows and Linux OS.
>> OVMF IA32/X64/IA32X64 all boots to Shell on Windows and Linux OS.
>> Emulator can boot to Shell on Windows only with CLANG9.
>> OVMF Ia32X64 RELEASE build generates the same BIOS images on Windows
>and Linux OS.
>>
>> Cc: Jiewen Yao <jiewen.yao@intel.com>
>> Cc: Steven Shi <steven.shi@intel.com>
>> Cc: Jordan Justen <jordan.l.justen@intel.com>
>> Cc: Laszlo Ersek <lersek@redhat.com>
>> Cc: Andrew Fish <afish@apple.com>
>> Cc: Ray Ni <ray.ni@intel.com>
>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>> Cc: Jian J Wang <jian.j.wang@intel.com>
>> Cc: Hao A Wu <hao.a.wu@intel.com>
>> Cc: Bob Feng <bob.c.feng@intel.com>
>> Cc: Michael D Kinney <michael.d.kinney@intel.com>
>>
>> Liming Gao (11):
>>   BaseTools tools_def.template: Remove unnecessary $(DEST_DIR_DEBUG)
>>     path
>>   BaseTools tools_def: Add CLANG9 tool chain to directly generate PE
>>     image
>>   BaseTools GenFw: Fix the issue to update the wrong size as SectionSize
>>   MdePkg Base.h: Add definition for CLANG9 tool chain
>>   MdePkg BaseIoLibIntrinsic: Remove __inline__ attribute for IO
>>     functions
>>   MdeModulePkg LzmaCustomDecompressLib: Update macro to be same in
>CLANG
>>     tool
>>   MdeModulePkg RegularExpressionDxe: Disable warning for CLANG9 tool
>>     chain
>>   CryptoPkg: Append options to make CLANG9 tool chain pass build
>>   CryptoPkg IntrinsicLib: Make _fltused always be used
>>   EmulatorPkg: Enable CLANG9 tool chain
>>   OvmfPkg: Enable CLANG9 tool chain
>>
>>  BaseTools/Source/C/GenFw/GenFw.c                   |   8 +-
>>  CryptoPkg/Library/IntrinsicLib/MemoryIntrinsics.c  |  10 +-
>>  EmulatorPkg/Win/Host/WinHost.c                     |   6 +-
>>  MdePkg/Library/BaseIoLibIntrinsic/IoLibGcc.c       |   6 -
>>  BaseTools/Conf/build_rule.template                 |  26 +++--
>>  BaseTools/Conf/tools_def.template                  | 124
>+++++++++++++++++++--
>>  CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf    |   1 +
>>  CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf     |   1 +
>>  CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf |   1 +
>>  CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf     |   1 +
>>  CryptoPkg/Library/OpensslLib/OpensslLib.inf        |   1 +
>>  CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf  |   1 +
>>  EmulatorPkg/EmulatorPkg.dsc                        |   7 +-
>>  EmulatorPkg/Win/Host/WinHost.inf                   |   6 +
>>  .../LzmaCustomDecompressLib/Sdk/C/7zTypes.h        |   2 +-
>>  .../RegularExpressionDxe/RegularExpressionDxe.inf  |   3 +
>>  MdePkg/Include/Base.h                              |   6 +-
>>  MdePkg/Include/Ia32/ProcessorBind.h                |   4 +-
>>  MdePkg/Include/X64/ProcessorBind.h                 |   2 +-
>>  OvmfPkg/OvmfPkgIa32.dsc                            |   4 +-
>>  OvmfPkg/OvmfPkgIa32X64.dsc                         |   4 +-
>>  OvmfPkg/OvmfPkgX64.dsc                             |   4 +-
>>  22 files changed, 183 insertions(+), 45 deletions(-)
>>
>> --
>> 2.13.0.windows.1
>>
>>
>> 
>>

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#49234): https://edk2.groups.io/g/devel/message/49234
Mute This Topic: https://groups.io/mt/34694424/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-