Re: posix command search and execution

[Mike Jonkmans]

On Wed, Nov 08, 2023 at 11:52:19PM +0700, Robert Elz wrote:
>     Date:        Tue, 7 Nov 2023 23:04:10 +0100
>     From:        Mike Jonkmans <bashbug@jonkmans.nl>
>     Message-ID:  <20231107220410.GC27780@jonkmans.nl>
> 
>   | It makes sense to partition the builtins in three categories with
>   | a separate name for each.
> 
> That's more or less what has been done now, with the specials, intrinsics,
> and regular builtins.

A step in the right direction.
In 30 years or so we may lose the special builtins ;)

>   | Unusuality sketch:
>   | - the shell provides a builtin for a standard utility
>   | - the distributor provides a function for the same utility
> 
> What counts as "the implementation" is always (apparently) an unspecified
> issue, splitting things into 2 camps (as almost every standards org
> tends to do, there's "us" and "the rest of them") never works well
> in practice, yet continues to be done for all kinds of purposes, as it
> always seems to provide a nice clean answer - yet as that shows, never
> really does, as what is called what always depends upon from what point
> of view you're looking, and there tend to be many.
> 
> But:
>   | in /etc/profile
> in that case it would be not be "provided by the implementation" - anything
> loaded from profile files is a user add on (even if it is forced upon the
> user by their admins) - if the local user doesn't want that function, they
> can simply remove it from /etc/profile (or not use that at all).
> 
>   | Are scripts in /etc/profile considered part of the implementation?
> So, no, I don't think so.

I don't want to start a flame war, but Chet thought it was. :)

>From a user/application writer's perspective I would say that it is.
Having a work around is not really convincing, it is still part of the
implementation.

Code -> Distribution -> Admin -> User

The `implementation' could be including any up to the first three.

>   | It is all kind of theoretical. What wonders me is that the POSIX
>   | specifications and definitions sometimes are imprecise or lacking.
> 
> They sometimes are, often because no-one considered some weird case,
> so didn't think to actually write down what happens then - in the POSIX
> world that's called implicitly unspecified.  Occasionally there are
> simple errors (minor ones, like spelling, grammar, punctuation or
> formatting issues - which are easy to overlook, though annoying) to
> major ones where the standard says what someone thought was correct,
> but they were mistaken.
> 
> But it isn't always the case, even when it seems so - to really
> understand it, you really have to read and comprehend (almost) the
> whole damn thing (all 4000 approx pages of it).   The text in one
> place often says nothing about something which is specified in some
> entirely different context.

Writing specifications is not an easy task.
Let's try to make things better (for whoever is defining `better').

Seems there are at least a couple of goals for the POSIX standard.
- providing direction to implementers
- converging implementations
- providing an api to users

An anti goal would be to keep supporting historical burdens,
like special builtins.

>   | > Yes.   But almost all shells implement it that way, so the
>   | > seemingly logical assumption is mostly backed by experience.
>   | Again this is not too precise for a standard.
> Yes, but the point is that this isn't what the standard says, more like
> what it really should say.   This PATH search for builtins stuff is an
> invention by people trying to force something upon shells that none of
> them implemented, because they thought it would be better.

Ouch.

> It is actually messier that it first seems, if you have a builtin command,
> it isn't simply finding that command's name in a PATH search that is
> required, but that the directory (from PATH) in which it was found be the
> one "associated" with the builtin command (how exactly that is determined,
> or what it really even means is not specified anywhere).   The effect is
> that if a shell with a builtin "test" believes that the associated
> directory is /bin and someone's path is /usr/bin:/bin - and there is a
> test command in /usr/bin (even if /usr/bin and /bin are the same directory,
> or /bin/name and /usr/bin/name are linked and so invoke the same thing)
> then the filesystem command is supposed to be invoked rather than the
> builtin.    There are reasons for that, but they're really fairly stupid.

Oh dear. I thought it could not get any worse.

> [Finding the standard utilities]
>   | Good to know. I was wondering about that.
>   | But where is that being specified?
> Somewhere in the conformance stuff I believe it is, I don't have
> the time to go look for it at the minute.
> 
>   | I already can't find the definition of standard utilities.
> There really isn't one - it is just a shorthand for "utilities
> specified by the standard" - less to type (and read).

It sounds logical, but it might be unwise to leave unspecified.
As it lets readers unnecessary questioning/searching for this.

>   | Ok. I was not aware that all standard utilities have to be in PATH.
> Not the utilities, or not the way you phrased that, the directories in
> which the standard utilities are found have to be in PATH (and PATH needs
> to be ordered such that they are found, not some other utility of the
> same name).    But I expect that's what you meant.

I presume, this is related to `getconf PATH', which Chet pointed out.
 
>   | One might want to override `.' (source in bash parlance) for logging.
> In a portable script you cannot, as POSIX only requires that "name"s work
> as function names, and "." is not a "name" (a "name" is the same syntax as
> used for variable names - which doesn't include special or positional params).
> 
> Most shells will allow it (POSIX doesn't forbid it) but you cannot rely
> upon it working.

Good point. Didn't realize that.
But the use case of wanting to override (special) builtins, is valid.
That aside, I prefer `source', as it is easier to grep.

>   | Can this intrinsic list be amended with any user loaded builtins?
> I don't see why not, but the concept of loadable builtins is beyond
> the standard, so more or less anything is possible there.
>   | That referred to the POSIX utilities link.
>   | https://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html
> As I said in my recent reply to Chet (except that his insane practice of
> including a Reply-To to just him, excluding the list, which my MUA follows
> as instructed by the header unless I remember to override it, meant that
> that reply went only to him), that list doesn't include the special builtin
> utilities (they come in XCU 2.15), the section just before XCU 3, which the
> page from your URL is the index for, and the special builtins are also
> standard utilities.

Ah, that happened to me too.
Have to remember to do a `list reply' in mutt.

Thanks for your insights.

-- 
Regards, Mike Jonkmans