bug#24751: 26.0.50; Regex stack overflow not detected properly (gets "Variable binding depth exceeds max-specpdl-size")

[npostavs]

This is a followup from Bug #24358, since that thread filled up with
messages about the crash in 25.1, so I'm opening a new bug for the wrong
regex stack overflow check (this affects only the master branch).

To reproduce the error run (re-search-forward ".*\\(\n.*\\)*" nil t) in
a buffer with contents of the attached bug-24358-regex-max-specpdl.txt.

bug-24358-regex-max-specpdl.txt
Description: sample data for regex test

Here is my analysis from
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24358#27 again:

> --- Begin Message ---
>
>
>
> Subject: 
>
> bug#24358: 25.1.50; re-search-forward errors with "Variable binding depth exceeds max-specpdl-size"
>
>
>
>
> Date: 
>
> Sat, 03 Sep 2016 11:43:16 -0400
>
>
>
>
> > (I'm also on GNU/Linux, Arch) I get the same max-specpdl-size error
> > with 25.1.50 [this refers to the master branch which is now 26.0.50
> > (emacs-25 was 25.0.xx at the time)].  With emacs version 24.5 (and
> > below) I get (error "Stack overflow in regexp matcher") [as expected].
>
> The problem is that re_max_failures is set to 40001 (instead of the
> original 40000) in main()[1], which is a problem because of the
> GROW_FAIL_STACK uses (re_max_failures * TYPICAL_FAILURE_SIZE) as a cap
> on the amount to allocate, but ((fail_stack).size * sizeof
> (fail_stack_elt_t)) to calculate current allocation.
>
> Since TYPICAL_FAILURE_SIZE = 20 and sizeof (fail_stack_elt_t) = 8, and
> it seems that (fail_stack).size grows in increments of 3, when
> (fail_stack).avail is 99999 and (fail_stack).size reaches 100002:
>
> (fail_stack).size * sizeof (fail_stack_elt_t) => 800016
>   re_max_failures * TYPICAL_FAILURE_SIZE      => 800020
>
> ENSURE_FAIL_STACK(3) then loops indefinitely reallocating a stack of
> size 800020 again and again until the record_xmalloc fails to
> grow_specdl() (thus the "max-specpdl-size" error).
>
> ----------
>
> So we we might want to fix the re_max_failures setting in main, but it
> doesn't quite make sense to me that GROW_FAIL_STACK relies on
> re_max_failures being a multiple of (sizeof (fail_stack_elt_t)).  At the
> definition of TYPICAL_FAILURE_SIZE we have
>
> /* Estimate the size of data pushed by a typical failure stack entry.
>    An estimate is all we need, because all we use this for
>    is to choose a limit for how big to make the failure stack.  */
> /* BEWARE, the value `20' is hard-coded in emacs.c:main().  */
> #define TYPICAL_FAILURE_SIZE 20
>
> Why do we use an "estimate" here?  What's wrong with just using
> (re_max_failures * sizeof (fail_stack_elt_t)) as the limit?  Or should
> the limit actually be (re_max_failures * TYPICAL_FAILURE_SIZE * sizeof
> (fail_stack_elt_t))?
>
> -----------
>
> 827           long lim = rlim.rlim_cur;
> (gdb) p rlim
> $1 = {
>   rlim_cur = 8388608, 
>   rlim_max = 18446744073709551615
> }
> (gdb) next
> 833           int ratio = 20 * sizeof (char *);
> (gdb) 
> 834           ratio += ratio / 3;
> (gdb) 
> 837           int extra = 200000;
> (gdb) p ratio
> $2 = 213
> [...]
> (gdb) display ((newlim - extra) / ratio)
> 1: ((newlim - extra) / ratio) = 40000
> (gdb) next
> 856               newlim += pagesize - 1;
> 1: ((newlim - extra) / ratio) = 40000
> (gdb) 
> 857               if (0 <= rlim.rlim_max && rlim.rlim_max < newlim)
> 1: ((newlim - extra) / ratio) = 40019
> (gdb) 
> 859               newlim -= newlim % pagesize;
> 1: ((newlim - extra) / ratio) = 40019
> (gdb) 
> 861               if (pagesize <= newlim - lim)
> 1: ((newlim - extra) / ratio) = 40001
> (gdb) undisplay 1
> (gdb) next
> 863                   rlim.rlim_cur = newlim;
> (gdb) 
> 864                   if (setrlimit (RLIMIT_STACK, &rlim) == 0)
> (gdb) 
> 865                     lim = newlim;
> (gdb) 
> 870           re_max_failures = lim < extra ? 0 : min (lim - extra, SIZE_MAX) / 
> ratio;
> (gdb) 
> 875       stack_bottom = &stack_bottom_variable;
> (gdb) p re_max_failures
> $3 = 40001
>
> -----------
>
> [1]: This was the case since 9d356f62 2016-05-27 "Robustify stack-size 
> calculation"
>
>
> --- End Message ---