Re: Revisiting Error handling (errexit)

[Martin D Kealey]

On Sun, 10 Jul 2022 at 05:39, Yair Lenga <yair.lenga@gmail.com> wrote:

> Re: command prefaced by ! which is important:
> * The '!' operator 'normal' behavior is to reverse the exit status of a
> command ('if ! check-something ; then ...').
>

Unless that status is ignored, in which case, well, it's still ignored.


> * I do not think it's a good idea to change the meaning of '!' when
> running with 'error checking'.
> * I think that the existing structures ('|| true', or '|| :') to force
> success status are good enough and well understood by beginner and advanced
> developers.
>

I'm not suggesting a change; rather I'm suggesting that your new errfail
should honour the existing rule for "!" (as per POSIX.1-2008 [
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html],
under the description of the "set" built-in):

2. The *-e* setting shall be ignored when executing the compound list
> following the *while*, *until*, *if*, or *elif* reserved word, *a
> pipeline beginning with the ! reserved word*, or any command of an AND-OR
> list other than the last.
>

So the exit status of a command starting with "!" (being the inverse of the
command it prefaces) is *not* considered by errexit, regardless of whether
it in turn is "tested".

It follows that

>  ! (( expression ))
>
and

>  (( expression )) || true
>
are equivalent under errexit; the former is the preferred idiom in some
places precisely because it is expressly called out in that clause of the
standard.

If you propose to make the former unsafe under errfail, then I suggest that
the onus is on you to explain why you would break code where the author has
clearly indicated that its exit status should not be taken as indicating
success or failure.

Question: What is the penalty for "|| true" ? true is bash builtin, and in
> bash it's equivalent to ':'. (see:
> https://unix.stackexchange.com/questions/34677/what-is-the-difference-between-and-true
> )
>

Even if there were no performance difference (and I'll admit, the
performance difference is very small), there's the visual clutter and the
cognitive load this places on subsequent maintainers. (One can adopt a
strategy of pushing the "|| true" off to the right with lots of whitespace,
but then there is the converse problem that any change to the expression is
just that bit harder to read in "git diff".)

Re: yet another global setting is perpetuating the "wrong direction".
> Most other scripting solutions that I'm familiar with are using dynamic
> (rather than lexical) scoping for 'try ... catch.'.
>

You're quite right that throw+catch is dynamically scoped, though try+catch
is of course a lexically scoped block.

The problem here is that you're retrofitting; in effect, you're making a
global declaration that *removes* an implicit try+catch+ignore around every
existing statement; *that's* what I want to have under lexically-scoped
control.

Considering that bash is stable, I do not think that it is realistic to try
> to expect major changes to 'bash'.
>

Expect, maybe not. Hope for? certainly. Fork it and do it myself? I'll
think about it.


> For a more sophisticated environment, python, groovy, javascript or (your
> favorite choice) might be a better solution.
>

Agreed that there are better languages for most complex tasks.

However they're not as pervasively available as Bash; the only language
that comes close to the same availability is Perl, and much as I like Perl,
it's abjectly detested by many folk. (The Shell would be *as* abjectly
detested if people actually understood it, but they're under the delusion
that it's "simple", and so they don't bother to hate it until it bites
them, and by the time they understand why it bit them, they're hooked and
can't leave even if they do hate it.)

Question: Out of curiosity, can you share your idea for a better solution ?
>

The direction I personally would take the shell is towards a "sane"
language where the current syntax is but a subset, and the oddball
semantics are "opt in".

Obviously your intent is to amend the behaviour specified by the first part
of that POSIX rule, so that only the last pipeline before "then" or "do" is
considered exempt, and not any earlier pipelines within the compound list
that appears there.

I see some merit in that approach, but on the whole I would not opt for
that approach. It's not as if long lists of commands between if/then is the
norm, and where they do occur it's for good reason.

Consider:

if  some_cmd
>     (( $? == 0 || $? == 43 ))
> then
>     echo "some_cmd succeeded or reported 'no action necessary'"
> fi
>

Where clearly the exit status of some_cmd is being given due consideration,
and carefully made exempt from errexit.

Instead I would focus on two things:

   1. abolishing "weird effects at a distance", which are a much bigger
   problem for code maintenance: the fact that calling a function within
   if/then or while/do turns off errexit with dynamic scope is the key pain
   point.
   2. making it possible to "catch" an error, do some local clean-up, and
   re-throw it

To this end, I would introduce some way of performing most "shopt", "set
-o", and "compat" settings with lexical scope; « local -o errfail » comes
to mind, but the exact syntax is not important and can be thrashed out
later. And yes, this would be allowed at file level, not just within
functions. Indeed it would be *preferred*  at file level, effectively as a
declaration of which language variant applies to *this file*.

When a function is defined with such an option in effect, it would attach
to the function itself, so that when the function is called, those settings
are performed automatically at function entry, even if it's called from
somewhere that the setting is not in effect. Conversely, if they're *not*
in effect when the function is defined, they're turned off even if they
were on where it's called from.

When "sourcing"/"dotting" a file, the default settings apply (until it
reaches a « local -o » statement, or whatever syntax is chosen for that),
even if called from somewhere with other settings in effect.

This makes it possible to have different settings in different files, which
is important for managing large projects.

Then I would encourage *every* bash file to start with the "compat" option
for the *current* version when it's written, so that it doesn't get broken
by subsequent updates to Bash. (Yes, that means we would need to actually
define the compat option when the version is created, instead of when the
*next* version replaces it.)

Side note: this means that tab completion functions for interactive Bash
wouldn't have to be some huge monolith that needs to be immediately
re-checked and updated when Bash is updated in the future. Instead the
maintenance of individual completion functions could be safely devolved to
the projects that they apply to.

I would also take a declarative approach to whether each function
participates in "errfail" or "errexit" handling:

   1. Does its return status indicate success or failure? If not, then all
   calls to it would automatically ignore errexit and errfail, whether
   "tested" or not.
   2. Does it "catch" errors that occur inside it? That is, despite «
   errexit » being in effect when the function was called, block unwinding due
   to inner errors as if « errfail » were in effect instead.

I expect both of these could be set in three ways: as an outer-scope
setting (« local -o ») when the function is defined; or as an option flag
after « function » when defining the function; or as an extra option to «
declare -f » before or after the function is actually defined. Of course,
any time you have a setting, it also applies to subsequent ordinary
commands within that lexical scope.

A key effect of this would be to allow interoperation between "modules",
being files containing sets of functions, where some use set -e (and need
it as a failsafe), and some don't (and can't operate with it enabled). Or
with set -f, or compat40, or whatever.

I would define a lexical scope as extending from the current statement
until the end of the inner-most enclosing compound statement, or the end of
an explicit subshell, or the end of the file, or the end of the string
that's being read by "eval"; whichever comes first.

I'm in two minds whether this should be some variation on "local" or
"declare", or a new keyword such as "decl" or "using" or "with". (I find it
ironic that the effects of "local" are not, in fact, actually local.)

Lastly, all of this needs to be done in a way that won't blow up on older
versions of Bash. That's a work in progress.

Yes I can see that this is a much more substantial change than you're
proposing, but it's not insanely huge.

-Martin

PS: I would also make it possible to define a function within a function
such that it too has local scope.

PPS: my longer-term goal would be complete lexically scoped symbol tables
for variables and function names, but that's not a prerequisite for
implementing this change.