1. Patchew
  2. EDK2
  3. View series

[edk2-devel] [Patch 0/4 V4] Enhance Incremental Build

Bob Feng posted 4 patches 4 years, 4 months ago
Failed in applying to current master (apply log)
There is a newer version of this series
BaseTools/Conf/build_rule.template            |  94 ++++---
BaseTools/Conf/tools_def.template             | 173 ++++++------
BaseTools/Source/Python/AutoGen/GenMake.py    |  83 ++----
.../Source/Python/AutoGen/IncludesAutoGen.py  | 255 ++++++++++++++++++
.../Source/Python/AutoGen/ModuleAutoGen.py    |  23 ++
BaseTools/Source/Python/Trim/Trim.py          | 115 ++++++--
BaseTools/Source/Python/build/build.py        |  63 ++++-
7 files changed, 588 insertions(+), 218 deletions(-)
create mode 100644 BaseTools/Source/Python/AutoGen/IncludesAutoGen.py
[edk2-devel] [Patch 0/4 V4] Enhance Incremental Build
Posted by Bob Feng 4 years, 4 months ago 
V4: Add $(DEPS_FLAGS) for rule [C-Code-File.BASE.AARCH64 ...]
V3: Change CLANG9 to CLANGPDB according to commit 14672c34bd 
V2: Fixed a bug in patch 4/4.

Incremental build reduces the build time by only building
the module that need to update. Edk2 Build system is a Makefile
based build system. The incrememtal build ability is provided by
the Make program. But Edk2 build tool need to generate correct makefile
to have Make program do incremental build correctly.

The current solution in build tool to support incremental build is that build
tool find out the include file list for each source file of a module, and in module's
makefile, build tool add the include file list as the source file's dependency. 
In this way Make program can decide if it need to rebuild a source code by checking
its dependency. This solution has 2 shortcommings, one is the process of finding
include list is slow, the other is this method can't handle case that 
a MACRO in #include statement so the related source file is always built. 

This patch provides another method to support incremental build. That is to use
c preprocessor and trim tool to generate dependency files for the source file.
This method will save much time in AutoGen phase and handle MACRO in #include correctly.

For C files:
    1. MSVS.
        cl.exe has a build option /showIncludes to display include files on stdout. Build tool captures
        that messages and generate dependency files, .deps files.
    2. CLANG and GCC
        -MMD -MF build option are used to generate dependency files by preprocessor. Build tool updates the
       .deps files.

For ASL files:
    1. Trim find out all the included files, which are asl specific include format, and generate .trim.deps file.
    2. ASL PP use c preprocessor to find out all included files with #include format and generate a .deps file
    3. build tool updates the .deps file

For ASM files (.asm, .s or .nasm):
    1. Trim find out all the included files, which are asm specific include format, and generate .trim.deps file.
    2. ASM PP use c preprocessor to find out all included files with #include format and generate a deps file
    3. build tool updates the .deps file

Build tool add "include" instruction for those deps files in the Makefile.

This patch does not support RVCT tool chain for the BZ https://bugzilla.tianocore.org/show_bug.cgi?id=1750

Feng, Bob C (4):
  BaseTools: Add build option for dependency file generation
  BaseTools: Generate dependent files for ASL and ASM files
  BaseTools: Update build_rule.txt to generate dependent files.
  BaseTools: Enhance Basetool for incremental build

 BaseTools/Conf/build_rule.template            |  94 ++++---
 BaseTools/Conf/tools_def.template             | 173 ++++++------
 BaseTools/Source/Python/AutoGen/GenMake.py    |  83 ++----
 .../Source/Python/AutoGen/IncludesAutoGen.py  | 255 ++++++++++++++++++
 .../Source/Python/AutoGen/ModuleAutoGen.py    |  23 ++
 BaseTools/Source/Python/Trim/Trim.py          | 115 ++++++--
 BaseTools/Source/Python/build/build.py        |  63 ++++-
 7 files changed, 588 insertions(+), 218 deletions(-)
 create mode 100644 BaseTools/Source/Python/AutoGen/IncludesAutoGen.py

-- 
2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51588): https://edk2.groups.io/g/devel/message/51588
Mute This Topic: https://groups.io/mt/65666140/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [Patch 0/4 V4] Enhance Incremental Build
Posted by Liming Gao 4 years, 4 months ago 
Bob:
  Thanks for your update. I verify this enhancement. It is great. Now, I have some minor comments. Can you help check them?

1. This change requires tools_def and build_rule be updated together. If they are not updated, what will happen? Can build tool report the error and let user update them?
2. C compiler will generate the include file list for all other files except for ASL and Nasm,, and output the include file into the screen. Build tool catches them, and creates the file xxx.deps. Can Build tool generate xxx.deps in the same output directory with the output file? For example, if C source output file is generated into OUTPUT\X64 directory, its deps file is also generated into OUTPUT\X64 directory. 
3. Do you verify the error case? For example, if C source file doesn't pass compile, its deps will still be generated or not?

Thanks
Liming
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng
> Sent: Tuesday, December 3, 2019 4:53 PM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] [Patch 0/4 V4] Enhance Incremental Build
> 
> V4: Add $(DEPS_FLAGS) for rule [C-Code-File.BASE.AARCH64 ...]
> V3: Change CLANG9 to CLANGPDB according to commit 14672c34bd
> V2: Fixed a bug in patch 4/4.
> 
> Incremental build reduces the build time by only building
> the module that need to update. Edk2 Build system is a Makefile
> based build system. The incrememtal build ability is provided by
> the Make program. But Edk2 build tool need to generate correct makefile
> to have Make program do incremental build correctly.
> 
> The current solution in build tool to support incremental build is that build
> tool find out the include file list for each source file of a module, and in module's
> makefile, build tool add the include file list as the source file's dependency.
> In this way Make program can decide if it need to rebuild a source code by checking
> its dependency. This solution has 2 shortcommings, one is the process of finding
> include list is slow, the other is this method can't handle case that
> a MACRO in #include statement so the related source file is always built.
> 
> This patch provides another method to support incremental build. That is to use
> c preprocessor and trim tool to generate dependency files for the source file.
> This method will save much time in AutoGen phase and handle MACRO in #include correctly.
> 
> For C files:
>     1. MSVS.
>         cl.exe has a build option /showIncludes to display include files on stdout. Build tool captures
>         that messages and generate dependency files, .deps files.
>     2. CLANG and GCC
>         -MMD -MF build option are used to generate dependency files by preprocessor. Build tool updates the
>        .deps files.
> 
> For ASL files:
>     1. Trim find out all the included files, which are asl specific include format, and generate .trim.deps file.
>     2. ASL PP use c preprocessor to find out all included files with #include format and generate a .deps file
>     3. build tool updates the .deps file
> 
> For ASM files (.asm, .s or .nasm):
>     1. Trim find out all the included files, which are asm specific include format, and generate .trim.deps file.
>     2. ASM PP use c preprocessor to find out all included files with #include format and generate a deps file
>     3. build tool updates the .deps file
> 
> Build tool add "include" instruction for those deps files in the Makefile.
> 
> This patch does not support RVCT tool chain for the BZ https://bugzilla.tianocore.org/show_bug.cgi?id=1750
> 
> Feng, Bob C (4):
>   BaseTools: Add build option for dependency file generation
>   BaseTools: Generate dependent files for ASL and ASM files
>   BaseTools: Update build_rule.txt to generate dependent files.
>   BaseTools: Enhance Basetool for incremental build
> 
>  BaseTools/Conf/build_rule.template            |  94 ++++---
>  BaseTools/Conf/tools_def.template             | 173 ++++++------
>  BaseTools/Source/Python/AutoGen/GenMake.py    |  83 ++----
>  .../Source/Python/AutoGen/IncludesAutoGen.py  | 255 ++++++++++++++++++
>  .../Source/Python/AutoGen/ModuleAutoGen.py    |  23 ++
>  BaseTools/Source/Python/Trim/Trim.py          | 115 ++++++--
>  BaseTools/Source/Python/build/build.py        |  63 ++++-
>  7 files changed, 588 insertions(+), 218 deletions(-)
>  create mode 100644 BaseTools/Source/Python/AutoGen/IncludesAutoGen.py
> 
> --
> 2.20.1.windows.1
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51604): https://edk2.groups.io/g/devel/message/51604
Mute This Topic: https://groups.io/mt/65666140/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [Patch 0/4 V4] Enhance Incremental Build
Posted by Bob Feng 4 years, 4 months ago 
1. No. If user does not update build_rule and tools_def, build will not failed. No impact for clean build. Since there will no COMMON_DEPS variable generated in Makefile, if user change a header file which is in original COMMON_DEPS, build tool will not rebuild the corresponding module for incremental build. 
2. Yes. for GCC and Clang, the deps file are under the same folder as .obj file localed. But for MSVC, they are in the module_build_folder, I can update this in new patch set.
3. If the compile failed, there will no deps file generated.

Thanks,
Bob

-----Original Message-----
From: Gao, Liming 
Sent: Tuesday, December 3, 2019 10:33 PM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>
Subject: RE: [edk2-devel] [Patch 0/4 V4] Enhance Incremental Build

Bob:
  Thanks for your update. I verify this enhancement. It is great. Now, I have some minor comments. Can you help check them?

1. This change requires tools_def and build_rule be updated together. If they are not updated, what will happen? Can build tool report the error and let user update them?
2. C compiler will generate the include file list for all other files except for ASL and Nasm,, and output the include file into the screen. Build tool catches them, and creates the file xxx.deps. Can Build tool generate xxx.deps in the same output directory with the output file? For example, if C source output file is generated into OUTPUT\X64 directory, its deps file is also generated into OUTPUT\X64 directory. 
3. Do you verify the error case? For example, if C source file doesn't pass compile, its deps will still be generated or not?

Thanks
Liming
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob 
> Feng
> Sent: Tuesday, December 3, 2019 4:53 PM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] [Patch 0/4 V4] Enhance Incremental Build
> 
> V4: Add $(DEPS_FLAGS) for rule [C-Code-File.BASE.AARCH64 ...]
> V3: Change CLANG9 to CLANGPDB according to commit 14672c34bd
> V2: Fixed a bug in patch 4/4.
> 
> Incremental build reduces the build time by only building the module 
> that need to update. Edk2 Build system is a Makefile based build 
> system. The incrememtal build ability is provided by the Make program. 
> But Edk2 build tool need to generate correct makefile to have Make 
> program do incremental build correctly.
> 
> The current solution in build tool to support incremental build is 
> that build tool find out the include file list for each source file of 
> a module, and in module's makefile, build tool add the include file list as the source file's dependency.
> In this way Make program can decide if it need to rebuild a source 
> code by checking its dependency. This solution has 2 shortcommings, 
> one is the process of finding include list is slow, the other is this 
> method can't handle case that a MACRO in #include statement so the related source file is always built.
> 
> This patch provides another method to support incremental build. That 
> is to use c preprocessor and trim tool to generate dependency files for the source file.
> This method will save much time in AutoGen phase and handle MACRO in #include correctly.
> 
> For C files:
>     1. MSVS.
>         cl.exe has a build option /showIncludes to display include files on stdout. Build tool captures
>         that messages and generate dependency files, .deps files.
>     2. CLANG and GCC
>         -MMD -MF build option are used to generate dependency files by preprocessor. Build tool updates the
>        .deps files.
> 
> For ASL files:
>     1. Trim find out all the included files, which are asl specific include format, and generate .trim.deps file.
>     2. ASL PP use c preprocessor to find out all included files with #include format and generate a .deps file
>     3. build tool updates the .deps file
> 
> For ASM files (.asm, .s or .nasm):
>     1. Trim find out all the included files, which are asm specific include format, and generate .trim.deps file.
>     2. ASM PP use c preprocessor to find out all included files with #include format and generate a deps file
>     3. build tool updates the .deps file
> 
> Build tool add "include" instruction for those deps files in the Makefile.
> 
> This patch does not support RVCT tool chain for the BZ 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1750
> 
> Feng, Bob C (4):
>   BaseTools: Add build option for dependency file generation
>   BaseTools: Generate dependent files for ASL and ASM files
>   BaseTools: Update build_rule.txt to generate dependent files.
>   BaseTools: Enhance Basetool for incremental build
> 
>  BaseTools/Conf/build_rule.template            |  94 ++++---
>  BaseTools/Conf/tools_def.template             | 173 ++++++------
>  BaseTools/Source/Python/AutoGen/GenMake.py    |  83 ++----
>  .../Source/Python/AutoGen/IncludesAutoGen.py  | 255 ++++++++++++++++++
>  .../Source/Python/AutoGen/ModuleAutoGen.py    |  23 ++
>  BaseTools/Source/Python/Trim/Trim.py          | 115 ++++++--
>  BaseTools/Source/Python/build/build.py        |  63 ++++-
>  7 files changed, 588 insertions(+), 218 deletions(-)  create mode 
> 100644 BaseTools/Source/Python/AutoGen/IncludesAutoGen.py
> 
> --
> 2.20.1.windows.1
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#51712): https://edk2.groups.io/g/devel/message/51712
Mute This Topic: https://groups.io/mt/65666140/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.