Re: Combination of "eval set -- ..." and $() command substitution is slow

[astian]

Dear Chet Ramey, thanks for the explanations, I assume you were able to
reproduce the issue.

First a warning that I forgot to include in the previous email: the quoted
excerpts of the gprof tables correspond to a slightly modified test script in
which I tried to magnify the overhead in order to more easily spot performance
differences.  Therefore, the absolute values displayed, e.g.  number of
function calls, are exaggerated.  The relative values (and hopefully your
conclusions) remain largely the same, though.  Sorry for any potential
confusion.

A few short comments prompted by your reply (probably not very useful, feel
free to ignore them).

Chet Ramey:
>>     - $() seems generally slightly slower than ``, but becomes pathologically
>>       so when preceded with "eval set -- ...".
>
> It is slightly slower -- POSIX requires that the shell parse the contents
> of $(...) to determine that it's a valid script as part of finding the
> closing `)'. The rules for finding the closing "`" don't have that
> requirement.
>
>>     - "eval set -- ..." itself doesn't seem slow at all, but obviously it has
>>       side-effects not captured by the "time" measurement tool.
>
> What happens is you end up with a 4900-character command string that you
> have to parse multiple times. But that's not the worst of it.

Since this statement ought to run exactly once, naïvely I would expect that by
"multiple times" you really mean at most "twice": once for the top-level
script, another time "inside" the eval "sub-script".

> The gprof output provides a clue.
>
>
>>       case 1 1 0 (pathological):
>>        %   cumulative   self              self     total
>>       time   seconds   seconds    calls  us/call  us/call  name
>>       38.89      0.21     0.21    28890     7.27     7.27  set_line_mbstate
>
> set_line_mbstate() runs through each command line before parsing, creating
> a bitmap that indicates whether each element is a single-byte character or
> part of a multi-byte character. The scanner uses this to determine whether
> a shell metacharacter should act as a delimiter or get skipped over as part
> of a multibyte character. For a single run with args `1 1 0', it gets
> called around 7300 times, with around 2400 of them for the 4900-character
> string with all the arguments.
>
> When you're in a multibyte locale (en_US.UTF-8 is one such), each one of
> those characters requires a call to mbrlen/mbrtowc. So that ends up being
> 2400 * 4900 calls to mbrlen.

I am indeed using an UTF-8 locale, but I tested also with export LC_ALL=C and 
the
behaviour did not change, I should have mentioned that.

Also, it occurs to me that since all shell metacharacters are surely 7-bit
ASCII, and all UTF-8 code units are strictly and deliberately non-ASCII 8-bit
bytes, at least in the case of UTF-8 it may not be necessary to construct such
bitmap: there cannot be a shell-metacharacter byte part of a UTF-8 sequence
representing something else, so there's no need for the distinction.  Of
course, this observation applies only to such specially crafted multibyte
encodings as UTF-8 (which nonetheless is surely by far the most common).
Furthermore, it does not explain the issue at hand.

> There is something happening here -- there's no way there should be that
> many calls to set_line_mbstate(),

Notice that there are almost as many calls (only 2 fewer) in case "0 1 0" (in
which eval is not used) yet in that case the performance is not harmed.
Quoting from the previous email:

  case 0 1 0:
   2.08      0.32     0.01    28888     0.00     0.00  set_line_mbstate
  case 1 1 0 (pathological):
  38.89      0.21     0.21    28890     7.27     7.27  set_line_mbstate

OTOH, all other test cases show less than 100 calls (these numbers correspond
to a profiling run of the given script, unmodified):

  $ grep -m1 set_line_mbstate gmon*.table
  gmon.out.0-0-X.table:  0.00      0.00     0.00       88     0.00     0.00  
set_line_mbstate
  gmon.out.0-1-0.table:  0.00      0.13     0.00     7288     0.00     0.00  
set_line_mbstate
  gmon.out.0-1-1.table:  0.00      0.18     0.00       88     0.00     0.00  
set_line_mbstate
  gmon.out.1-0-X.table:  0.00      0.01     0.00       90     0.00     0.00  
set_line_mbstate
  gmon.out.1-1-0.table: 37.50      0.06     0.06     7290     8.23     8.23  
set_line_mbstate
  gmon.out.1-1-1.table:  0.00      0.07     0.00       90     0.00     0.00  
set_line_mbstate

> even when you have to save and restore
> the input line because you have to parse the contents of $(). There must
> be some combination of the effect of `eval' on the line bitmap and the
> long string. I'll see what I can figure out.
>
> Chet

Cheers.