[edk2-devel] [Patch 0/2] Convert Split tool to python

Bob Feng posted 2 patches 5 days, 3 hours ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/edk2 tags/patchew/20210113095609.1398-1-bob.c.feng@intel.com
BaseTools/Source/C/Split/Split.c            | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split       |  31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
BaseTools/Source/C/GNUmakefile              |   1 -
BaseTools/Source/C/Makefile                 |   1 -
BaseTools/Source/C/Split/GNUmakefile        |  17 -
BaseTools/Source/C/Split/Makefile           |  16 -
BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py   |  10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

[edk2-devel] [Patch 0/2] Convert Split tool to python

Posted by Bob Feng 5 days, 3 hours ago
There are 2 reasons to convert Split tool from C to Python.
1. We are in the process of moving the Basetools Python code
to a separate repository. But there still are many C tools under
edk2/BaseTools. To make all Basetools be in the separate repo,
we can convert the C tools to Python tools.
2. The original Split tool is very slow. This python tool can reduce
90% time

Bob Feng (2):
  BaseTools: Convert Split tool to python
  BaseTools: Add unittest for Split tool

 BaseTools/Source/C/Split/Split.c            | 466 --------------------
 BaseTools/BinWrappers/PosixLike/Split       |  31 +-
 BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
 BaseTools/Source/C/GNUmakefile              |   1 -
 BaseTools/Source/C/Makefile                 |   1 -
 BaseTools/Source/C/Split/GNUmakefile        |  17 -
 BaseTools/Source/C/Split/Makefile           |  16 -
 BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
 BaseTools/Source/Python/Split/__init__.py   |  10 +
 BaseTools/Source/Python/tests/test_split.py | 101 +++++
 10 files changed, 325 insertions(+), 524 deletions(-)
 delete mode 100644 BaseTools/Source/C/Split/Split.c
 create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
 delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
 delete mode 100644 BaseTools/Source/C/Split/Makefile
 create mode 100644 BaseTools/Source/Python/Split/Split.py
 create mode 100644 BaseTools/Source/Python/Split/__init__.py
 create mode 100644 BaseTools/Source/Python/tests/test_split.py

-- 
2.29.1.windows.1



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-


回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Posted by gaoliming 4 days, 12 hours ago
Bob:
  Have you plan to convert all C tools to Python tools? Or only some ones?
Can you share your planning for the changes in BaseTools?
  
  And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest. 

Thanks
Liming
> -----邮件原件-----
> 发件人: bounce+27952+70232+4905953+8761045@groups.io
> <bounce+27952+70232+4905953+8761045@groups.io> 代表 Bob Feng
> 发送时间: 2021年1月13日 17:56
> 收件人: devel@edk2.groups.io
> 主题: [edk2-devel] [Patch 0/2] Convert Split tool to python
> 
> There are 2 reasons to convert Split tool from C to Python.
> 
> 1. We are in the process of moving the Basetools Python code
> 
> to a separate repository. But there still are many C tools under
> 
> edk2/BaseTools. To make all Basetools be in the separate repo,
> 
> we can convert the C tools to Python tools.
> 
> 2. The original Split tool is very slow. This python tool can reduce
> 
> 90% time
> 
> Bob Feng (2):
>   BaseTools: Convert Split tool to python
>   BaseTools: Add unittest for Split tool
> 
>  BaseTools/Source/C/Split/Split.c            | 466 --------------------
>  BaseTools/BinWrappers/PosixLike/Split       |  31 +-
>  BaseTools/BinWrappers/WindowsLike/Split.bat |   3 +
>  BaseTools/Source/C/GNUmakefile              |   1 -
>  BaseTools/Source/C/Makefile                 |   1 -
>  BaseTools/Source/C/Split/GNUmakefile        |  17 -
>  BaseTools/Source/C/Split/Makefile           |  16 -
>  BaseTools/Source/Python/Split/Split.py      | 203 +++++++++
>  BaseTools/Source/Python/Split/__init__.py   |  10 +
>  BaseTools/Source/Python/tests/test_split.py | 101 +++++
>  10 files changed, 325 insertions(+), 524 deletions(-)
>  delete mode 100644 BaseTools/Source/C/Split/Split.c
>  create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
>  delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
>  delete mode 100644 BaseTools/Source/C/Split/Makefile
>  create mode 100644 BaseTools/Source/Python/Split/Split.py
>  create mode 100644 BaseTools/Source/Python/Split/__init__.py
>  create mode 100644 BaseTools/Source/Python/tests/test_split.py
> 
> --
> 2.29.1.windows.1
> 
> 
> 
> -=-=-=-=-=-=
> Groups.io Links: You receive all messages sent to this group.
> View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
> Mute This Topic: https://groups.io/mt/79647273/4905953
> Group Owner: devel+owner@edk2.groups.io
> Unsubscribe: https://edk2.groups.io/g/devel/unsub
> [gaoliming@byosoft.com.cn]
> -=-=-=-=-=-=
> 





-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70262): https://edk2.groups.io/g/devel/message/70262
Mute This Topic: https://groups.io/mt/79665827/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Posted by Bob Feng 4 days, 10 hours ago
Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:

> 
> Bob:
> Have you plan to convert all C tools to Python tools? Or only some ones?

I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048

> 
> Can you share your planning for the changes in BaseTools?

Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.

> 
> And, I see you also add unittest for new split tool. Can you introduce the
> 
> unittest framework for python tool? If so, new added tool can follow the
> same way to add unittest.

The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

> 
> 
> Thanks
> Liming
> 
>> -----邮件原件-----
>> 发件人: bounce+27952+70232+4905953+8761045@groups.io
>> <bounce+27952+70232+4905953+8761045@groups.io> 代表 Bob Feng
>> 发送时间: 2021年1月13日 17:56
>> 收件人: devel@edk2.groups.io
>> 主题: [edk2-devel] [Patch 0/2] Convert Split tool to python
>> 
>> There are 2 reasons to convert Split tool from C to Python.
>> 
>> 1. We are in the process of moving the Basetools Python code
>> 
>> to a separate repository. But there still are many C tools under
>> 
>> edk2/BaseTools. To make all Basetools be in the separate repo,
>> 
>> we can convert the C tools to Python tools.
>> 
>> 2. The original Split tool is very slow. This python tool can reduce
>> 
>> 90% time
>> 
>> Bob Feng (2):
>> BaseTools: Convert Split tool to python
>> BaseTools: Add unittest for Split tool
>> 
>> BaseTools/Source/C/Split/Split.c | 466 --------------------
>> BaseTools/BinWrappers/PosixLike/Split | 31 +-
>> BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
>> BaseTools/Source/C/GNUmakefile | 1 -
>> BaseTools/Source/C/Makefile | 1 -
>> BaseTools/Source/C/Split/GNUmakefile | 17 -
>> BaseTools/Source/C/Split/Makefile | 16 -
>> BaseTools/Source/Python/Split/Split.py | 203 +++++++++
>> BaseTools/Source/Python/Split/__init__.py | 10 +
>> BaseTools/Source/Python/tests/test_split.py | 101 +++++
>> 10 files changed, 325 insertions(+), 524 deletions(-)
>> delete mode 100644 BaseTools/Source/C/Split/Split.c
>> create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
>> delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
>> delete mode 100644 BaseTools/Source/C/Split/Makefile
>> create mode 100644 BaseTools/Source/Python/Split/Split.py
>> create mode 100644 BaseTools/Source/Python/Split/__init__.py
>> create mode 100644 BaseTools/Source/Python/tests/test_split.py
>> 
>> --
>> 2.29.1.windows.1
>> 
>> 
>> 
>> -=-=-=-=-=-=
>> Groups.io Links: You receive all messages sent to this group.
>> View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
>> Mute This Topic: https://groups.io/mt/79647273/4905953
>> Group Owner: devel+owner@edk2.groups.io
>> Unsubscribe: https://edk2.groups.io/g/devel/unsub
>> [gaoliming@byosoft.com.cn]
>> -=-=-=-=-=-=
> 
>


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70269): https://edk2.groups.io/g/devel/message/70269
Mute This Topic: https://groups.io/mt/79665827/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-


回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Posted by gaoliming 3 days, 12 hours ago
Bob:

 Thanks for your planning sharing. For BaseTools test, I still have some questions. 

 

1.      For test framework, is it the incremental test framework or the functionality test framework or both? 

2.      For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory. 

 

Thanks

Liming

发件人: tosubbounce+27952+0+0+0@groups.io <tosubbounce+27952+0+0+0@groups.io> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn>; devel@edk2.groups.io
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

 

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:

Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?

I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.

https://edk2.groups.io/g/devel/topic/76572200#64993 

https://edk2.groups.io/g/rfc/topic/74009714#270 

https://edk2.groups.io/g/devel/topic/73069134#58048

Can you share your planning for the changes in BaseTools?

Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.

And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.

The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html 
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected. 

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/ 


Thanks
Liming

-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> 
<bounce+27952+70232+4905953+8761045@groups.io <mailto:bounce+27952+70232+4905953+8761045@groups.io> > 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io <mailto:devel@edk2.groups.io> 
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io <mailto:devel+owner@edk2.groups.io> 
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70350): https://edk2.groups.io/g/devel/message/70350
Mute This Topic: https://groups.io/mt/79691688/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-


Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Posted by Bob Feng 3 days, 11 hours ago
Hi Liming,

1. The test framework can be used for both incremental test and functionality test. We can do more discussion when I send out this framework for review.
2. Yes, I agree to add a sub folder under the test folder. I’ll update the patch.

Is there any other comments before I send the patch V2?

Thanks,
Bob

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of gaoliming
Sent: Friday, January 15, 2021 9:13 AM
To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io
Subject: 回复: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Bob:
 Thanks for your planning sharing. For BaseTools test, I still have some questions.


1.     For test framework, is it the incremental test framework or the functionality test framework or both?

2.     For BaseTools\Source\Python\tests\test_split.py, it creates new directory ‘tests’ for test case scripts, and use test_split.py for split tool. So, new tool test script will follow the same style. Besides, split tool is simple. It may not have its special test case. But, Fce/FMMT are too complex. They need the different bios images as their test cases. Then, where are those test cases placed into? I would prefer to place the test script and its test cases into the same directory. So, I would propose to create the subdirectory in ‘tests’ for each tool test, such as Split for this case. Then, every tool can place its test script and test case into its own directory.

Thanks
Liming
发件人: tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io> <tosubbounce+27952+0+0+0@groups.io<mailto:tosubbounce+27952+0+0+0@groups.io>> 代表 Bob Feng
发送时间: 2021年1月14日 10:46
收件人: gaoliming <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: Re: [edk2-devel] 回复: [edk2-devel] [Patch 0/2] Convert Split tool to python

Good questions. I answer them inline.

Thanks,
Bob

On Thu, Jan 14, 2021 at 08:53 AM, gaoliming wrote:
Bob:
Have you plan to convert all C tools to Python tools? Or only some ones?
I am still investigating the feasibility of converting the C tools to Python tools. Some C tools are simple but some are complex. I hope all the C tools can be converted to Python tools so that all the tools can be moved to the BaseTools separate repo. I think we can file some BZs to track the progress of the C tools conversion. And I hope there could be more volunteers to be involved in this work.

Here are the discussions about the benefit and process of moving BaseTools out of the edk2 repo for reference.
https://edk2.groups.io/g/devel/topic/76572200#64993
https://edk2.groups.io/g/rfc/topic/74009714#270
https://edk2.groups.io/g/devel/topic/73069134#58048
Can you share your planning for the changes in BaseTools?
Except for this patch, my current work for Basetools are:
1. Re-write the FMMT/FCE with Python and propose to add these 2 tools to the edk2 BaseTools.
2. Prepare to publish the Basetools incremental build regression test cases and test framework.
And, I see you also add unittest for new split tool. Can you introduce the
unittest framework for python tool? If so, new added tool can follow the
same way to add unittest.
The unittest is the python build-in library for the unit test. https://docs.python.org/3/library/unittest.html
I provide the unit test is to help you to review this patch. From the unit test case, it will be easy to
see if this tool's behavior is as expected.

I think for the python code development, we can try our best to follow this document. https://edk2-docs.gitbook.io/edk-ii-python-development-process-specification/

Thanks
Liming
-----邮件原件-----
发件人: bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>
<bounce+27952+70232+4905953+8761045@groups.io<mailto:bounce+27952+70232+4905953+8761045@groups.io>> 代表 Bob Feng
发送时间: 2021年1月13日 17:56
收件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io>
主题: [edk2-devel] [Patch 0/2] Convert Split tool to python

There are 2 reasons to convert Split tool from C to Python.

1. We are in the process of moving the Basetools Python code

to a separate repository. But there still are many C tools under

edk2/BaseTools. To make all Basetools be in the separate repo,

we can convert the C tools to Python tools.

2. The original Split tool is very slow. This python tool can reduce

90% time

Bob Feng (2):
BaseTools: Convert Split tool to python
BaseTools: Add unittest for Split tool

BaseTools/Source/C/Split/Split.c | 466 --------------------
BaseTools/BinWrappers/PosixLike/Split | 31 +-
BaseTools/BinWrappers/WindowsLike/Split.bat | 3 +
BaseTools/Source/C/GNUmakefile | 1 -
BaseTools/Source/C/Makefile | 1 -
BaseTools/Source/C/Split/GNUmakefile | 17 -
BaseTools/Source/C/Split/Makefile | 16 -
BaseTools/Source/Python/Split/Split.py | 203 +++++++++
BaseTools/Source/Python/Split/__init__.py | 10 +
BaseTools/Source/Python/tests/test_split.py | 101 +++++
10 files changed, 325 insertions(+), 524 deletions(-)
delete mode 100644 BaseTools/Source/C/Split/Split.c
create mode 100644 BaseTools/BinWrappers/WindowsLike/Split.bat
delete mode 100644 BaseTools/Source/C/Split/GNUmakefile
delete mode 100644 BaseTools/Source/C/Split/Makefile
create mode 100644 BaseTools/Source/Python/Split/Split.py
create mode 100644 BaseTools/Source/Python/Split/__init__.py
create mode 100644 BaseTools/Source/Python/tests/test_split.py

--
2.29.1.windows.1



-=-=-=-=-=-=
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70232): https://edk2.groups.io/g/devel/message/70232
Mute This Topic: https://groups.io/mt/79647273/4905953
Group Owner: devel+owner@edk2.groups.io<mailto:devel+owner@edk2.groups.io>
Unsubscribe: https://edk2.groups.io/g/devel/unsub
[gaoliming@byosoft.com.cn]
-=-=-=-=-=-=



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#70351): https://edk2.groups.io/g/devel/message/70351
Mute This Topic: https://groups.io/mt/79692367/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-