Re: Exclude builtins from command completion

[Stephane Chazelas]

2020-01-10 09:57:52 -0500, Chet Ramey:
[...]
> buildpat()
> {
>       builtins_pat=
> 
>       set -- $(printf "%q " $(compgen -b -k "$1"))    # to leave reserved 
> words, omit -k

There are a few problems with that code

$(compgen...) will take the output of compgen, split it on the
value of $IFS (SPC, TAB, NL by default), perform filename
generation on each resulting word, and pass the result as
separate arguments to printf.

With %q, printf may add some '...' or $'...' or \ around the
resulting strings. Then those things will be printed followed by
SPC characters.

Because of the unquoted $(...) again, that resulting string will
be again split on $IFS and undergo another round of filename
generation, and the list stored as positional parameters.

In the general case, that wouldn't make much sense, but
thankfully, here the output of compgen -b -k will be relatively
tamed words. No SPC, TAB, NL, ?, * in builtin/keyword names.

However, there's a [ builtin and some [[ and ]] keywords.

So if you call that function in a context where $IFS doesn't
contain newline and either the failglob or nullglob option are
set, for $(compgen...) you'd get funny behaviour.

Try for instance

bash -O failglob -c 'IFS=; echo $(compgen -bk)'

printf %q will insert some backslash before the [s and because
in bash5, backslash became a glob operator, even with the
default $IFS, failglob/nullglob will cause problem:

$ bash -O failglob -c 'set -- $(printf "%q " $(compgen -bk "$1"))'
bash: no match: \[

Even without failglob/nullglob, if that function is called from
within the /bin directory which on many systems contain a file
called "[", that "\[" will become "[", and if called from within
a directory with millions of entries, running that function will
take a long time as it needs to read the full contents of the
directory (at least twice).

A rule 101 of POSIX/bash shell scripting is that if you want to
split an expansion (that's the rare cases where you want to
leave a word expansion unquoted), you must set IFS to the
delimiter you want and disable globbing. And if you don't want
split+glob, you quote the expansion (like for that $#).

You also forgot the "--" to delimit the options from the compgen
prefix:

$ bash  -c 'set -- $(printf "%q " $(compgen -bk "$1"))' bash "-z"
bash: line 0: compgen: -z: invalid option

Here, it seems like you want to get the list of builtin and
keywords, using printf %q to insert \ before [ on the assumption
that it's the only thing that will be quoted. So that should be:

local - # restores the options upon return from the function
set -o noglob # disable globbing
local IFS=$'\n' # split on newline
set -- $(printf '%q\n' $(compgen -bk -- "$1"))

, 
>       [ $# -gt 0 ] && builtins_pat='@('"$(IFS='|' ; echo "$*")"')'

Why not just:

local IFS='|'
builtins_pat="@($*)"

Here, you'd only want to escape the [, ], (, ) *, ?, \
characters if the intent is to build a @(foo|bar) pattern.

In the output of compgen -bk, of those, only [ is likely to
occur but %q generally can't be used to escape glob operators
because it escapes more than wildcard characters and in ways
that are not compatible with wildcards. For instance, it escapes
a TAB character as $'\t'. But as a wildcard `$'\t'` would only
ever match on $'t'.

Here, you'd just need:

builtins_pat="@($(
  compgen -bk -- "$1" |
    sed 's/[][?\\*()]/\\&/g' |
    paste -sd '|' -
))"


> }
> 
> commandcomp()
> {
>       local builtins_pat w
> 
>       w="$2"
>       buildpat "$w"
>       
>       if [ -n "$builtins_pat" ]; then
>               COMPREPLY=( $(shopt -s extglob ; compgen -c -X "$builtins_pat" 
> "$w") )
>       else
>               COMPREPLY=( $(compgen -c "$w") )
>       fi
> }
> 
> complete -I -F commandcomp
[...]

You've got the same kind of problems here (IFS not set, glob not
disabled, missing --), but here, it's not fixable as compgen
outputs newline delimited records but newline is as valid a
character in a file name (and thus in a command name) as any, so
its output is not post-processable.

$ touch ~/bin/$'weird\ncommand'
$ chmod +x ~/bin/$'weird\ncommand'
$ bash -c 'compgen -c we'
weird
command
weave
weave

Here, whatever we do, we can't get back the $'weird\ncommand'
command.

compgen at the moment can't be used reliably in completion
widgets.

To fix it, you (as the bash maintainer) could add some -0 option
to compgen so it outputs NUL delimited records and use:

readarray -td '' COMPREPLY < <(
    shopt -s extglob; compgen -c -X "$builtins_pat" -- "$w")

Or add a "-O <name>" option, for "compgen" to store the
generated completions in an array directly instead of outputing
them one per line and use it here as:

local restore="$(shopt -p extglob)"
shopt -s extglob
compgen -O COMPREPLY -c -X "$builtins_pat" -- "$w"
eval "$restore"

By the way, is there no equivalent of "local -" for shopt
options?

Also note that if the nocasematch option is on, that will
exclude all the DO dO Do... It would help to have a way to have
a default set of options locally in functions (like the emulate
-L zsh of zsh).

-- 
Stephane