Re: Incorrect / Inconsistent behavior with nameref assignments in functions

[Binarus]

On 30.08.2020 02:59, Koichi Murase wrote:
> 2020-08-29 14:46 Binarus <lists@binarus.de>:
>> I am wondering when debian will include bash 5.1. It looks like
>> debian testing and debian unstable are on bash 5.0, so it will
>> probably take several years.
> 
> Actually the problem of the function `Dummy' will not be solved even
> in bash 5.1.  There is another but similar problem with your function.
> A user might specify `namerefArray' as the name of an outer array,
> which results in a circular-reference error.

Actually, this is what first happened to me and what led me to the
problem described in my original post.

I was trying to solve the circular reference problem by establishing a
mixture of a simple variable naming convention and copying method. The
idea was that all nameref variable names would start with the string
"nameref" (designating the nameref "type"), that all other variable
names would start with other strings, and that I would never pass a
variable which is already a nameref to other functions as a parameter.
Then, as the first action in each function which deals with namerefs, I
would copy the contents of each nameref (whose name starts with
"nameref") to a local variable (whose name doesn't start with
"nameref"), and (if needed) would write back the contents from the local
variable to the nameref variable at the end of the function.

This would make sure that circular references couldn't occur. My example
script 1 is the implementation of this idea, which worked in that it
prevented the circular references ...

In my case, this idea is not as dumb as it first sounds due to the
seemingly unnecessary copying, because most of my functions have to
operate on copies of the data they get passed (mostly arrays) anyway.

Later, while the problem of the circular reference error had indeed been
solved, I noticed that the script silently produced wrong results.

This is the key point and the difference between the two sorts of error:

The bug I have reported (I still believe that "bug" is the correct term)
makes scripts silently produce wrong results. This is an absolute no-go.
If I hadn't tested it thoroughly with edge cases, I eventually wouldn't
have noticed it, which could have led to serious damage (the scripts in
question will be part of a backup system).

In contrast, a circular reference makes bash throw a visible,
appropriate message. I am fine with bash throwing errors or warnings and
possibly aborting script execution. When I see this, I can fix the
problem. Therefore, I don't have any problem with bash 5.1 still not
allowing that sort of circular reference (as long as it keeps throwing
messages when it encounters that situation).

But again, I consider it a massive problem when data just silently is
not copied as expected and even the contents of the variable which is
referenced by the nameref get destroyed (at least as long as we are in
the respective function itself) by using it as the RHS in an assignment.

>   $ cat testR2c.sh
>   function Dummy {
>     local -n namerefArray="$1"
>     local -a -i myArray=("${namerefArray[@]}")
>     local -p
>   }
>   declare -a -i namerefArray=('1' '2' '3')
>   Dummy namerefArray
> 
>   $ bash-5.1-alpha testR2c.sh
>   testR2c.sh: line 4: local: warning: namerefArray: circular name reference
>   testR2c.sh: line 4: warning: namerefArray: circular name reference
>   testR2c.sh: line 5: warning: namerefArray: circular name reference
>   testR2c.sh: line 5: warning: namerefArray: circular name reference
>   declare -a myArray=([0]="1" [1]="2" [2]="3")
>   declare -n namerefArray="namerefArray"
> 
> If you want to work around the problem, there are several ways.
> 
> * One of the simplest ways is to use different variable names as
>   already suggested by other people.  However, when the variable name
>   is not under control for some reason (that, e.g., the functon is
>   provided to users who may use it in an arbitrary way, or it imports
>   different shell-script frameworks), the probability of the name
>   collision is not 0%.

This solution would be good enough for me, because bash warns me about
the problem if it occurs, and I then can change the script accordingly;
I (currently) don't need to provide the functions to other users.

> * Another way is to copy to the local array only when the name is
>   different from `myArray':
> 
>   function Dummy {
>     [[ $1 == myArray ]] ||
>       eval "local -a myArray=(\"\${$1[@]}\")"
>     declare -p myArray
>   }

Thank you very much for that idea!

However, eval is evil. If I ever had to provide that function to other
users (which currently is not the case), then I would have a problem if
another user would call it like that:

declare -a -i myArray1=('1' '2' '3')
Dummy 'myArray1[@]}"); echo Gotcha!; #'

Output:

root@cerberus:~/scripts# ./test6
Gotcha!
declare -a myArray=([0]="1" [1]="2" [2]="3")

I guess it would be very complicated, if possible at all, to protect the
code inside eval against every sort of such attacks.

>   When you want to add `-i' attribute to the array or to modify the
>   array without affecting the original outer array, you may first save
>   the value to another local array and next copy the array to the
>   array that you want to edit.
> 
>   function Dummy {
>     [[ $1 == inputArray ]] ||
>       eval "local -a inputArray=(\"\${$1[@]}\")"
>     local -ia myArray=("${inputArray[@]}")
>     declare -p myArray
>   }

For the reason detailed above, I'd like to avoid eval by all means
(although I currently do not need to handle attacks from evil users).

> * If you want to use namerefs to eliminate the use of `eval', maybe
>   you could do like the following but I think it is more natural and
>   readable to use eval:
> 
>   function Dummy {
>     [[ $1 == refArray ]] || local -n refArray=$1
>     [[ $1 == inputArray ]] || local -i inputArray=("${refArray[@]}")
>     local -ia myArray=("${inputArray[@]}")
>     declare -p myArray
>   }

Thank you very much! This is a good idea which I hadn't thought about
yet. It provides a clean solution to the circular reference problem.
However (hoping that I don't get flamed for a dumb question), I don't
understand why we need inputArray at all in that code. Wouldn't the
following function be sufficient?

function Dummy {
  [[ $1 == refArray ]] || local -n refArray=$1
  local -ia myArray=("${refArray[@]}")
  declare -p myArray
}

Unfortunately, these solutions (while solving the circular reference
problem) don't solve my original problem. I can't use bash 5.1 in the
production environment where the scripts will run, so I have to work
around the bug I have described in the first message of this thread.

I think I'll stick with my current (extremely ugly, but reliable)
solution: Assign a number to each function, and decorate each local
variable in each function with that number, i.e. prepend or append that
number to each local variable name.

Thank you very much again for your valuable time and knowledge!

Best regards,

Binarus