1. Patchew
  2. EDK2
  3. View series

[edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.

Bob Feng posted 1 patch 3 years, 7 months ago
Failed in applying to current master (apply log)
BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
[edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Bob Feng 3 years, 7 months ago 
From: Mingyue Liang <mingyuex.liang@intel.com>

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880

Currently, When doing the Incremental build, the directory
macros extended to absolute path in output Makefile, which
is inconsistent with the output of Clean build.

When we do macro replacement, we can't replace macro due to
inconsistent path case, which results in inconsistent display
of incremental build and clean build in makefile.Therefore,
the path is converted to achieve the correct macro replacement.

Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Cc: Yuwei Chen <yuwei.chen@intel.com>
---
 BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index 0314d0ea34..b04d3f5436 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -786,8 +786,10 @@ cleanlib:
 
     def ReplaceMacro(self, str):
         for Macro in self.MacroList:
-            if self._AutoGenObject.Macros[Macro] and self._AutoGenObject.Macros[Macro] in str:
-                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro + ')')
+            if self._AutoGenObject.Macros[Macro] and os.path.normcase(self._AutoGenObject.Macros[Macro]) in os.path.normcase(str):
+                replace_dir = str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Macros[Macro])): os.path.normcase(str).index(
+                    os.path.normcase(self._AutoGenObject.Macros[Macro])) + len(self._AutoGenObject.Macros[Macro])]
+                str = str.replace(replace_dir, '$(' + Macro + ')')
         return str
 
     def CommandExceedLimit(self):
-- 
2.28.0.windows.1



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65527): https://edk2.groups.io/g/devel/message/65527
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Andrew Fish via groups.io 3 years, 7 months ago 
Does this work on case sensitive file systems?
> On Sep 23, 2020, at 3:58 AM, Bob Feng <bob.c.feng@intel.com> wrote:
> 
> From: Mingyue Liang <mingyuex.liang@intel.com>
> 
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880
> 
> Currently, When doing the Incremental build, the directory
> macros extended to absolute path in output Makefile, which
> is inconsistent with the output of Clean build.
> 
> When we do macro replacement, we can't replace macro due to
> inconsistent path case, which results in inconsistent display
> of incremental build and clean build in makefile.Therefore,
> the path is converted to achieve the correct macro replacement.
> 
> Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> ---
> BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
> index 0314d0ea34..b04d3f5436 100755
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -786,8 +786,10 @@ cleanlib:
> 
>     def ReplaceMacro(self, str):
>         for Macro in self.MacroList:
> -            if self._AutoGenObject.Macros[Macro] and self._AutoGenObject.Macros[Macro] in str:
> -                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro + ')')
> +            if self._AutoGenObject.Macros[Macro] and os.path.normcase(self._AutoGenObject.Macros[Macro]) in os.path.normcase(str):
> +                replace_dir = str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Macros[Macro])): os.path.normcase(str).index(
> +                    os.path.normcase(self._AutoGenObject.Macros[Macro])) + len(self._AutoGenObject.Macros[Macro])]
> +                str = str.replace(replace_dir, '$(' + Macro + ')')
>         return str
> 
>     def CommandExceedLimit(self):
> -- 
> 2.28.0.windows.1
> 
> 
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65533): https://edk2.groups.io/g/devel/message/65533
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Bob Feng 3 years, 7 months ago 
Yes. we did test on Windows and Linux.

From the https://docs.python.org/3/library/os.path.html
os.path.normcase(path)
    Normalize the case of a pathname. On Windows, convert all characters in the pathname to lowercase, and also convert forward slashes to  backward slashes. On other operating systems, return the path unchanged.

Thanks,
Bob
-----Original Message-----
From: Andrew Fish <afish@apple.com> 
Sent: Wednesday, September 23, 2020 10:24 PM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>
Cc: Liang, MingyueX <mingyuex.liang@intel.com>; Liming Gao <gaoliming@byosoft.com.cn>; Chen, Christine <yuwei.chen@intel.com>
Subject: Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.

Does this work on case sensitive file systems?
> On Sep 23, 2020, at 3:58 AM, Bob Feng <bob.c.feng@intel.com> wrote:
> 
> From: Mingyue Liang <mingyuex.liang@intel.com>
> 
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880
> 
> Currently, When doing the Incremental build, the directory macros 
> extended to absolute path in output Makefile, which is inconsistent 
> with the output of Clean build.
> 
> When we do macro replacement, we can't replace macro due to 
> inconsistent path case, which results in inconsistent display of 
> incremental build and clean build in makefile.Therefore, the path is 
> converted to achieve the correct macro replacement.
> 
> Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> ---
> BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py 
> b/BaseTools/Source/Python/AutoGen/GenMake.py
> index 0314d0ea34..b04d3f5436 100755
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -786,8 +786,10 @@ cleanlib:
> 
>     def ReplaceMacro(self, str):
>         for Macro in self.MacroList:
> -            if self._AutoGenObject.Macros[Macro] and self._AutoGenObject.Macros[Macro] in str:
> -                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro + ')')
> +            if self._AutoGenObject.Macros[Macro] and os.path.normcase(self._AutoGenObject.Macros[Macro]) in os.path.normcase(str):
> +                replace_dir = str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Macros[Macro])): os.path.normcase(str).index(
> +                    os.path.normcase(self._AutoGenObject.Macros[Macro])) + len(self._AutoGenObject.Macros[Macro])]
> +                str = str.replace(replace_dir, '$(' + Macro + ')')
>         return str
> 
>     def CommandExceedLimit(self):
> --
> 2.28.0.windows.1
> 
> 
> 
> 
> 
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65534): https://edk2.groups.io/g/devel/message/65534
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Andrew Fish via groups.io 3 years, 7 months ago 
Bob,

Sorry I was confused by the commit comment `inconsistent path case`. So this was only a bug on case insensitive file systems? 

I’m paranoid as macOS support case and case insensitive file systems and we have had a lot of strange bugs in the past. 

Thanks,

Andrew Fish

> On Sep 23, 2020, at 7:59 AM, Bob Feng <bob.c.feng@intel.com> wrote:
> 
> Yes. we did test on Windows and Linux.
> 
> From the https://docs.python.org/3/library/os.path.html <https://docs.python.org/3/library/os.path.html>
> os.path.normcase(path)
>    Normalize the case of a pathname. On Windows, convert all characters in the pathname to lowercase, and also convert forward slashes to  backward slashes. On other operating systems, return the path unchanged.
> 
> Thanks,
> Bob
> -----Original Message-----
> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>> 
> Sent: Wednesday, September 23, 2020 10:24 PM
> To: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com <mailto:bob.c.feng@intel.com>>
> Cc: Liang, MingyueX <mingyuex.liang@intel.com <mailto:mingyuex.liang@intel.com>>; Liming Gao <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>; Chen, Christine <yuwei.chen@intel.com <mailto:yuwei.chen@intel.com>>
> Subject: Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
> 
> Does this work on case sensitive file systems?
>> On Sep 23, 2020, at 3:58 AM, Bob Feng <bob.c.feng@intel.com <mailto:bob.c.feng@intel.com>> wrote:
>> 
>> From: Mingyue Liang <mingyuex.liang@intel.com <mailto:mingyuex.liang@intel.com>>
>> 
>> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880 <https://bugzilla.tianocore.org/show_bug.cgi?id=2880>
>> 
>> Currently, When doing the Incremental build, the directory macros 
>> extended to absolute path in output Makefile, which is inconsistent 
>> with the output of Clean build.
>> 
>> When we do macro replacement, we can't replace macro due to 
>> inconsistent path case, which results in inconsistent display of 
>> incremental build and clean build in makefile.Therefore, the path is 
>> converted to achieve the correct macro replacement.
>> 
>> Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com <mailto:mingyuex.liang@intel.com>>
>> Cc: Bob Feng <bob.c.feng@intel.com <mailto:bob.c.feng@intel.com>>
>> Cc: Liming Gao <gaoliming@byosoft.com.cn <mailto:gaoliming@byosoft.com.cn>>
>> Cc: Yuwei Chen <yuwei.chen@intel.com <mailto:yuwei.chen@intel.com>>
>> ---
>> BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
>> 1 file changed, 4 insertions(+), 2 deletions(-)
>> 
>> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py 
>> b/BaseTools/Source/Python/AutoGen/GenMake.py
>> index 0314d0ea34..b04d3f5436 100755
>> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
>> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
>> @@ -786,8 +786,10 @@ cleanlib:
>> 
>>    def ReplaceMacro(self, str):
>>        for Macro in self.MacroList:
>> -            if self._AutoGenObject.Macros[Macro] and self._AutoGenObject.Macros[Macro] in str:
>> -                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro + ')')
>> +            if self._AutoGenObject.Macros[Macro] and os.path.normcase(self._AutoGenObject.Macros[Macro]) in os.path.normcase(str):
>> +                replace_dir = str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Macros[Macro])): os.path.normcase(str).index(
>> +                    os.path.normcase(self._AutoGenObject.Macros[Macro])) + len(self._AutoGenObject.Macros[Macro])]
>> +                str = str.replace(replace_dir, '$(' + Macro + ')')
>>        return str
>> 
>>    def CommandExceedLimit(self):
>> --
>> 2.28.0.windows.1
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65535): https://edk2.groups.io/g/devel/message/65535
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Bob Feng 3 years, 7 months ago 
Yes. this bug only happens on case insensitive file systems.

Thanks,
Bob

From: Andrew Fish <afish@apple.com>
Sent: Wednesday, September 23, 2020 11:31 PM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com>
Cc: Liang, MingyueX <mingyuex.liang@intel.com>; Liming Gao <gaoliming@byosoft.com.cn>; Chen, Christine <yuwei.chen@intel.com>
Subject: Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.

Bob,

Sorry I was confused by the commit comment `inconsistent path case`. So this was only a bug on case insensitive file systems?

I’m paranoid as macOS support case and case insensitive file systems and we have had a lot of strange bugs in the past.

Thanks,

Andrew Fish


On Sep 23, 2020, at 7:59 AM, Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> wrote:

Yes. we did test on Windows and Linux.

From the https://docs.python.org/3/library/os.path.html
os.path.normcase(path)
   Normalize the case of a pathname. On Windows, convert all characters in the pathname to lowercase, and also convert forward slashes to  backward slashes. On other operating systems, return the path unchanged.

Thanks,
Bob
-----Original Message-----
From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Wednesday, September 23, 2020 10:24 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Cc: Liang, MingyueX <mingyuex.liang@intel.com<mailto:mingyuex.liang@intel.com>>; Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>; Chen, Christine <yuwei.chen@intel.com<mailto:yuwei.chen@intel.com>>
Subject: Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.

Does this work on case sensitive file systems?

On Sep 23, 2020, at 3:58 AM, Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> wrote:

From: Mingyue Liang <mingyuex.liang@intel.com<mailto:mingyuex.liang@intel.com>>

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880

Currently, When doing the Incremental build, the directory macros
extended to absolute path in output Makefile, which is inconsistent
with the output of Clean build.

When we do macro replacement, we can't replace macro due to
inconsistent path case, which results in inconsistent display of
incremental build and clean build in makefile.Therefore, the path is
converted to achieve the correct macro replacement.

Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com<mailto:mingyuex.liang@intel.com>>
Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>
Cc: Liming Gao <gaoliming@byosoft.com.cn<mailto:gaoliming@byosoft.com.cn>>
Cc: Yuwei Chen <yuwei.chen@intel.com<mailto:yuwei.chen@intel.com>>
---
BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
b/BaseTools/Source/Python/AutoGen/GenMake.py
index 0314d0ea34..b04d3f5436 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -786,8 +786,10 @@ cleanlib:

   def ReplaceMacro(self, str):
       for Macro in self.MacroList:
-            if self._AutoGenObject.Macros[Macro] and self._AutoGenObject.Macros[Macro] in str:
-                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro + ')')
+            if self._AutoGenObject.Macros[Macro] and os.path.normcase(self._AutoGenObject.Macros[Macro]) in os.path.normcase(str):
+                replace_dir = str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Macros[Macro])): os.path.normcase(str).index(
+                    os.path.normcase(self._AutoGenObject.Macros[Macro])) + len(self._AutoGenObject.Macros[Macro])]
+                str = str.replace(replace_dir, '$(' + Macro + ')')
       return str

   def CommandExceedLimit(self):
--
2.28.0.windows.1











-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65536): https://edk2.groups.io/g/devel/message/65536
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH] BaseTools: Normalize case of pathname when evaluating Macros.
Posted by Yuwei Chen 3 years, 7 months ago 
Reviewed-by: Yuwei Chen<yuwei.chen@intel.com>

> -----Original Message-----
> From: Feng, Bob C <bob.c.feng@intel.com>
> Sent: Wednesday, September 23, 2020 6:58 PM
> To: devel@edk2.groups.io
> Cc: Liang, MingyueX <mingyuex.liang@intel.com>; Liming Gao
> <gaoliming@byosoft.com.cn>; Chen, Christine <yuwei.chen@intel.com>
> Subject: [PATCH] BaseTools: Normalize case of pathname when evaluating
> Macros.
> 
> From: Mingyue Liang <mingyuex.liang@intel.com>
> 
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2880
> 
> Currently, When doing the Incremental build, the directory macros extended
> to absolute path in output Makefile, which is inconsistent with the output of
> Clean build.
> 
> When we do macro replacement, we can't replace macro due to inconsistent
> path case, which results in inconsistent display of incremental build and clean
> build in makefile.Therefore, the path is converted to achieve the correct
> macro replacement.
> 
> Signed-off-by: Mingyue Liang <mingyuex.liang@intel.com>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <gaoliming@byosoft.com.cn>
> Cc: Yuwei Chen <yuwei.chen@intel.com>
> ---
>  BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
> b/BaseTools/Source/Python/AutoGen/GenMake.py
> index 0314d0ea34..b04d3f5436 100755
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -786,8 +786,10 @@ cleanlib:
> 
>      def ReplaceMacro(self, str):
>          for Macro in self.MacroList:
> -            if self._AutoGenObject.Macros[Macro] and
> self._AutoGenObject.Macros[Macro] in str:
> -                str = str.replace(self._AutoGenObject.Macros[Macro], '$(' + Macro
> + ')')
> +            if self._AutoGenObject.Macros[Macro] and
> os.path.normcase(self._AutoGenObject.Macros[Macro]) in
> os.path.normcase(str):
> +                replace_dir =
> str[os.path.normcase(str).index(os.path.normcase(self._AutoGenObject.Ma
> cros[Macro])): os.path.normcase(str).index(
> +                    os.path.normcase(self._AutoGenObject.Macros[Macro])) +
> len(self._AutoGenObject.Macros[Macro])]
> +                str = str.replace(replace_dir, '$(' + Macro + ')')
>          return str
> 
>      def CommandExceedLimit(self):
> --
> 2.28.0.windows.1



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#65558): https://edk2.groups.io/g/devel/message/65558
Mute This Topic: https://groups.io/mt/77032381/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.