1. Patchew
  2. EDK2
  3. View series

[edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

tlaronde@polynum.com posted 1 patch 1 year, 5 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/edk2 tags/patchew/Y3t4mxrmhuiMG._2FnN@polynum.com
[edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by tlaronde@polynum.com 1 year, 5 months ago 
diff --git a/edksetup.sh b/edksetup.sh
index 06d2f041e6..46b295c430 100755
--- a/edksetup.sh
+++ b/edksetup.sh
@@ -105,6 +105,19 @@ function SetupEnv()
   fi
 }
 
+# whereis(1) is not a POSIX utility and, for example, its implementation
+# in NetBSD is different form the Linux one.
+#
+function whereis()
+{
+  (
+    IFS=:
+    for dir in $PATH; do
+      eval ls $dir/${1}* 2>/dev/null || true 
+    done
+  )
+}
+
 function SetupPython3()
 {
   if [ $origin_version ];then

signed-off-by: Thierry LARONDE <tlaronde@polynum.com>
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96522): https://edk2.groups.io/g/devel/message/96522
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by Pedro Falcato 1 year, 5 months ago 
On Mon, Nov 21, 2022 at 9:21 PM <tlaronde@polynum.com> wrote:

> diff --git a/edksetup.sh b/edksetup.sh
> index 06d2f041e6..46b295c430 100755
> --- a/edksetup.sh
> +++ b/edksetup.sh
> @@ -105,6 +105,19 @@ function SetupEnv()
>    fi
>  }
>
> +# whereis(1) is not a POSIX utility and, for example, its implementation
> +# in NetBSD is different form the Linux one.
> +#
> +function whereis()
> +{
> +  (
> +    IFS=:
> +    for dir in $PATH; do
> +      eval ls $dir/${1}* 2>/dev/null || true
> +    done
> +  )
> +}
>
Hi Thierry,

First of all, thanks for the patch! I had noticed this problem when running
edksetup.sh on a POSIX but not quite Linux system before.

I kind of dislike your solution. Does NetBSD ship /bin/which by default? I
think replacing whereis with "which -a" would be a lot better.
I don't think there's a 100% standard way to do this in POSIX, as which
isn't POSIX either, and your solution seems... hacky?

Also, please send patches in the standard git format (git commit -s + git
format-patch + git send-email with the proper CCs to the maintainers, see
the proper guides for more details).

Thanks,
Pedro


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96523): https://edk2.groups.io/g/devel/message/96523
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by Rebecca Cran 1 year, 5 months ago 
On 11/21/22 15:22, Pedro Falcato wrote:

> I kind of dislike your solution. Does NetBSD ship /bin/which by default? 
> I think replacing whereis with "which -a" would be a lot better.
> I don't think there's a 100% standard way to do this in POSIX, as which 
> isn't POSIX either, and your solution seems... hacky?


"command" seems to be the POSIX way to do this?
https://pubs.opengroup.org/onlinepubs/9699919799/

Though "whereis python3" shows the following on my system:

python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3 
/usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3 
/usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3 
/usr/share/man/man1/python3.1.gz

"which -a python3" returns:

/usr/bin/python3
/bin/python3

And "command -p -v" returns:

/bin/python3

I don't know if we need all the results from "whereis"?

-- 
Rebecca Cran


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96565): https://edk2.groups.io/g/devel/message/96565
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by tlaronde@polynum.com 1 year, 5 months ago 
Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> On 11/21/22 15:22, Pedro Falcato wrote:
> 
> > I kind of dislike your solution. Does NetBSD ship /bin/which by default?
> > I think replacing whereis with "which -a" would be a lot better.
> > I don't think there's a 100% standard way to do this in POSIX, as which
> > isn't POSIX either, and your solution seems... hacky?
> 
> 
> "command" seems to be the POSIX way to do this?
> https://pubs.opengroup.org/onlinepubs/9699919799/
> 
> Though "whereis python3" shows the following on my system:
> 
> python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> /usr/share/man/man1/python3.1.gz
> 
> "which -a python3" returns:
> 
> /usr/bin/python3
> /bin/python3
> 
> And "command -p -v" returns:
> 
> /bin/python3
> 
> I don't know if we need all the results from "whereis"?

The problem is when one does not know which exact version of python
is here and how, exactly, the command is named.

whereis(1) returns whatever command with python3 as prefix, while
command or "which -a" will return only exactly python3. If, on the
system, python is fully version qualified (as is the case with
pkgsrc, the opt packages framework, used on NetBSD):

which -a python3

will return nothing, since, it is python3.9 for example on the OS.


There is a way to manipulate the namespace, since edk2setup.sh sets
environment and looks, by default, in some directories, but I have
respected the logics here: since whereis(1) is used to find a python3
related command in the PATH, even if it is named python3.x, the simplest
is to provide a basic whereis implementation in the script itself. Or,
changing the behavior, will perhaps break existing installation handling
the builds.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96572): https://edk2.groups.io/g/devel/message/96572
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by Pedro Falcato 1 year, 5 months ago 
On Tue, Nov 22, 2022 at 4:26 PM <tlaronde@polynum.com> wrote:

> Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> > On 11/21/22 15:22, Pedro Falcato wrote:
> >
> > > I kind of dislike your solution. Does NetBSD ship /bin/which by
> default?
> > > I think replacing whereis with "which -a" would be a lot better.
> > > I don't think there's a 100% standard way to do this in POSIX, as which
> > > isn't POSIX either, and your solution seems... hacky?
> >
> >
> > "command" seems to be the POSIX way to do this?
> > https://pubs.opengroup.org/onlinepubs/9699919799/
> >
> > Though "whereis python3" shows the following on my system:
> >
> > python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> > /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> > /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> > /usr/share/man/man1/python3.1.gz
> >
> > "which -a python3" returns:
> >
> > /usr/bin/python3
> > /bin/python3
> >
> > And "command -p -v" returns:
> >
> > /bin/python3
> >
> > I don't know if we need all the results from "whereis"?
>
> The problem is when one does not know which exact version of python
> is here and how, exactly, the command is named.
>
> whereis(1) returns whatever command with python3 as prefix, while
> command or "which -a" will return only exactly python3. If, on the
> system, python is fully version qualified (as is the case with
> pkgsrc, the opt packages framework, used on NetBSD):
>
> which -a python3
>
> will return nothing, since, it is python3.9 for example on the OS.
>

Sorry, what? This sounds so broken. How can a script shebang ever work?

For Sane(tm) systems, I propose command -v python3 + $(command -v python3)
-c 'import sys; print(sys.version.split(" ")[0])',
which gives us the path to the Python 3 interpreter + the path in a
relatively easy, simple way.
I think this could work mostly everywhere but apparently NetBSD since you
don't provide python3? Which makes little sense in my head.

Pedro


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96573): https://edk2.groups.io/g/devel/message/96573
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by tlaronde@polynum.com 1 year, 5 months ago 
Le Tue, Nov 22, 2022 at 04:31:14PM +0000, Pedro Falcato a écrit :
> On Tue, Nov 22, 2022 at 4:26 PM <tlaronde@polynum.com> wrote:
> 
> > Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> > > On 11/21/22 15:22, Pedro Falcato wrote:
> > >
> > > > I kind of dislike your solution. Does NetBSD ship /bin/which by
> > default?
> > > > I think replacing whereis with "which -a" would be a lot better.
> > > > I don't think there's a 100% standard way to do this in POSIX, as which
> > > > isn't POSIX either, and your solution seems... hacky?
> > >
> > >
> > > "command" seems to be the POSIX way to do this?
> > > https://pubs.opengroup.org/onlinepubs/9699919799/
> > >
> > > Though "whereis python3" shows the following on my system:
> > >
> > > python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> > > /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> > > /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> > > /usr/share/man/man1/python3.1.gz
> > >
> > > "which -a python3" returns:
> > >
> > > /usr/bin/python3
> > > /bin/python3
> > >
> > > And "command -p -v" returns:
> > >
> > > /bin/python3
> > >
> > > I don't know if we need all the results from "whereis"?
> >
> > The problem is when one does not know which exact version of python
> > is here and how, exactly, the command is named.
> >
> > whereis(1) returns whatever command with python3 as prefix, while
> > command or "which -a" will return only exactly python3. If, on the
> > system, python is fully version qualified (as is the case with
> > pkgsrc, the opt packages framework, used on NetBSD):
> >
> > which -a python3
> >
> > will return nothing, since, it is python3.9 for example on the OS.
> >
> 
> Sorry, what? This sounds so broken. How can a script shebang ever work?
> 
> For Sane(tm) systems, I propose command -v python3 + $(command -v python3)
> -c 'import sys; print(sys.version.split(" ")[0])',
> which gives us the path to the Python 3 interpreter + the path in a
> relatively easy, simple way.
> I think this could work mostly everywhere but apparently NetBSD since you
> don't provide python3? Which makes little sense in my head.
> 

NetBSD (as other OSes) does not provide python in base. It is available
as an add-on. And the add-on framework on NetBSD allows to install
several versions of Python (since there seems to be so many of them...)
and in order to do that, the binary is not called "python", neither
"python3" but "python3.9" or "python3.10". And since one can
concurrently install different versions, there is no symbolic or
hardlink python3 to a preferred python 3 series binary.

As I have already said, edk2setup.sh allows to manipulate the
environment and, for example, allows to set explicitely the env var
PYTHON_COMMAND. So I know how to circumvent the problem. 

But, since the script uses whereis(1), the Linux version, that does
take "python3" as a prefix and searches for "python3*" (in more than the
directories in PATH in fact), I have simply made so that the current
behavior, intended(?), work on NetBSD too or on any system without
whereis(1) or with a whereis(1) that does not behave as the Linux one.

If you want to nuke whereis and use a POSIX2 utility instead, I'm
perfectly fine: I will have to make other adjustements for NetBSD (not
in edk2setup.sh) and I'm writing documentation on how-to, so I can
circumvent this elsewhere.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96574): https://edk2.groups.io/g/devel/message/96574
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by Pedro Falcato 1 year, 5 months ago 
On Tue, Nov 22, 2022 at 3:40 PM Rebecca Cran <quic_rcran@quicinc.com> wrote:

> On 11/21/22 15:22, Pedro Falcato wrote:
>
> > I kind of dislike your solution. Does NetBSD ship /bin/which by default?
> > I think replacing whereis with "which -a" would be a lot better.
> > I don't think there's a 100% standard way to do this in POSIX, as which
> > isn't POSIX either, and your solution seems... hacky?
>
>
> "command" seems to be the POSIX way to do this?
> https://pubs.opengroup.org/onlinepubs/9699919799/
>
> Though "whereis python3" shows the following on my system:
>
> python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> /usr/share/man/man1/python3.1.gz
>
> "which -a python3" returns:
>
> /usr/bin/python3
> /bin/python3
>
> And "command -p -v" returns:
>
> /bin/python3
>
> I don't know if we need all the results from "whereis"?
>
> --
> Rebecca Cran
>

I guess we could just use /bin/python3 (as in command -v python3) and
readlink to find the actual version?

In my local system "command -v python3" returns /usr/bin/python3
"readlink /usr/bin/python3" returns python3.10, which we could promptly
parse into a version?

I don't know how portable this is, but it's an idea. I definitely don't see
the reason to iterate through every possible python3* in PATH.

Pedro


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96571): https://edk2.groups.io/g/devel/message/96571
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)
Posted by tlaronde@polynum.com 1 year, 5 months ago 
Hello Pedro,

Le Mon, Nov 21, 2022 at 10:22:21PM +0000, Pedro Falcato a écrit :
> On Mon, Nov 21, 2022 at 9:21 PM <tlaronde@polynum.com> wrote:
> 
> > diff --git a/edksetup.sh b/edksetup.sh
> > index 06d2f041e6..46b295c430 100755
> > --- a/edksetup.sh
> > +++ b/edksetup.sh
> > @@ -105,6 +105,19 @@ function SetupEnv()
> >    fi
> >  }
> >
> > +# whereis(1) is not a POSIX utility and, for example, its implementation
> > +# in NetBSD is different form the Linux one.
> > +#
> > +function whereis()
> > +{
> > +  (
> > +    IFS=:
> > +    for dir in $PATH; do
> > +      eval ls $dir/${1}* 2>/dev/null || true
> > +    done
> > +  )
> > +}
> >
> Hi Thierry,
> 
> First of all, thanks for the patch! I had noticed this problem when running
> edksetup.sh on a POSIX but not quite Linux system before.
> 
> I kind of dislike your solution. Does NetBSD ship /bin/which by default? I
> think replacing whereis with "which -a" would be a lot better.
> I don't think there's a 100% standard way to do this in POSIX, as which
> isn't POSIX either, and your solution seems... hacky?

The problem is that which -a does not do what is required by the usage
of whereis(1) in the script and what is indeed implemented in Linux whereis(1) (and not in the obsolescent
NetBSD whereis(1)): look for "string*". "which -a python" or "which -a
'python*'" will return nothing.

This is why I ended up with this solution that uses only POSIX sh(1)
features and that is, if I'm not mistaken, an implementation of what is
used.

Best,
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#96543): https://edk2.groups.io/g/devel/message/96543
Mute This Topic: https://groups.io/mt/95182516/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.