Re: configure[xxxx]: : is not an identifer

[Kevin R. Bulgrien]

----- Original Message -----
> From: "Eric Blake" <address@hidden>
> To: "Kevin R. Bulgrien" <address@hidden>, address@hidden
> Sent: Thursday, June 7, 2018 4:54:34 PM
> Subject: Re: configure[xxxx]: : is not an identifer
> 
> On 06/07/2018 04:30 PM, Kevin R. Bulgrien wrote:
> 
> >> Can you run under '/bin/sh -x ./configure' (or whatever shell
> >> configure
> >> actually picks in place of /bin/sh) to get a more verbose log
> >> right
> >> before the failure message of what the script was attempting?
> > 
> > Yes, I did this after posting the original e-mail, and I got a
> > result
> > that showed me the last command that ran was the ac_subst_vars
> > assignment:
> > 
> >    ...
> 
> >    + ./configure[1845]: : is not an identifier
> > 
> > This is about 4K into the value being assigned.
> 
> Then it sounds like we've found a legitimate shell bug in your
> /bin/sh.
> It would be nice to fine-tune the characteristics of the bug; is it
> merely:
> 
> a='8k wall of text'
> 
> that fails, while:
> 
> a='4k...'
> b='4k...'
> a="$a$b"
> 
> works?  Or is it that a shell variable in general cannot hold more
> than
> 4k?  Does the size of the shell script matter (as in the bug is
> present
> if the assignment also coincides with some other large boundary, such
> as
> a 64k page size)?
> 
> > 
> > I found that if I broke up this value by assigning it to different
> > variables (so as not to reduce variable store usage), the script
> > runs "fine".  I.e., I break up the ac_subst_vars along the lines
> > of:
> > 
> >    ac_subst_vars='...
> >    ...'
> >    foo='...
> >    ...'
> >    bar='...
> >    ...'
> > 
> > Where the size of each piece is similar (~4K).  When this is done,
> > the configure script runs "fine", though it is certainly damaged.
> 
> Well, yeah, if you break it up to get past tickling the overlarge
> assignment bug, but then don't reassemble it, the resulting
> substitutions are incomplete and the end product would be damaged.
>  As
> long as variables can hold more than 4k (even if we have to build it
> via
> smaller steps), that can be worked around.

I found that using the last displayed value shown in the error message
became unreliable near the end of the value.  I ended up splitting
the variable assignment into pieces of about 200 lines each to
get it to work.   This correlates to splitting the assignment into
six pieces:

$ diff -U1 configure.orig configure
--- configure.orig      2018-06-06 05:41:59.000000000 -0500
+++ configure   2018-06-06 06:22:33.000000000 -0500
@@ -629,3 +629,3 @@
 gl_getopt_required=POSIX
-ac_subst_vars='M4tests_LTLIBOBJS
+ac_subst_vars1='M4tests_LTLIBOBJS
 M4tests_LIBOBJS
@@ -838,4 +838,4 @@
 GNULIB_POSIX_SPAWNATTR_SETSIGDEFAULT
-GNULIB_POSIX_SPAWNATTR_GETSIGDEFAULT
-GNULIB_POSIX_SPAWNATTR_SETSCHEDPOLICY
+GNULIB_POSIX_SPAWNATTR_GETSIGDEFAULT'
+ac_subst_vars2='GNULIB_POSIX_SPAWNATTR_SETSCHEDPOLICY
 GNULIB_POSIX_SPAWNATTR_GETSCHEDPOLICY
@@ -1038,4 +1038,4 @@
 REPLACE_TANHF
-REPLACE_TANF
-REPLACE_SQRTL
+REPLACE_TANF'
+ac_subst_vars3='REPLACE_SQRTL
 REPLACE_SQRTF
@@ -1239,4 +1239,4 @@
 GNULIB_LOG1P
-GNULIB_LOG10L
-GNULIB_LOG10F
+GNULIB_LOG10L'
+ac_subst_vars4='GNULIB_LOG10F
 GNULIB_LOG10
@@ -1439,4 +1439,4 @@
 UNISTD_H_HAVE_WINSOCK2_H_AND_USE_SOCKETS
-UNISTD_H_HAVE_WINSOCK2_H
-REPLACE_WRITE
+UNISTD_H_HAVE_WINSOCK2_H'
+ac_subst_vars5='REPLACE_WRITE
 REPLACE_USLEEP
@@ -1639,4 +1639,4 @@
 GNULIB_CALLOC_POSIX
-GNULIB_ATOLL
-GNULIB__EXIT
+GNULIB_ATOLL'
+ac_subst_vars6='GNULIB__EXIT
 LTLIBCSTACK
@@ -1844,2 +1844,10 @@
 SHELL'
+echo $ac_subst_vars1 | wc -c >krb
+echo $ac_subst_vars2 | wc -c >>krb
+echo $ac_subst_vars3 | wc -c >>krb
+echo $ac_subst_vars4 | wc -c >>krb
+echo $ac_subst_vars5 | wc -c >>krb
+echo $ac_subst_vars6 | wc -c >>krb
+ac_subst_vars="$ac_subst_vars1\n$ac_subst_vars2\n$ac_subst_vars3\n$ac_subst_vars4\n$ac_subst_vars5\n$ac_subst_vars6"
+echo $ac_subst_vars | wc -c >>krb
 ac_subst_files=''

$ cat krb
   4030
   3609
   2745
   3136
   3162
   2540
  19222

This shows the whole constructed as a sum of the parts, with the
final result matching the original 19222 bytes assigned (as reported
in a prior post by copying the assignment to a standalone script).

With this modification, the script runs fine under this /bin/sh.

If 4K is a factor in this bug, it is subtle.  It has the appearance
that the shell error message does not reliably show the points where
the variable has to be split the further into the value you get, though
I suspect the error messages occur near 4K boundaries.

I first broke the value by splitting just before the last value
reported by the crash, then re-ran the modified script to find the
next split point, until I had 5 pieces.  This would be about right
to split a 19220 byte value into 4K pieces, but, at this point, I
found that the script bombed other places and moving the 5th split
around wasn't working, so I just ended up splitting by line count
into 6 pieces to rid myself of the error.

> > I unintentionally over-rode the  #!/bin/sh in the script with:
> > 
> >    $ bash -x ./configure -C --prefix=${HOME}
> > 
> > This resolved the problem, and ./configure ran fine with the
> > unmodified configure script.
> 
> The preferred spelling for this is:
> 
> CONFIG_SHELL=/path/to/bash ./configure CONFIG_SHELL=/path/to/bash
> -C...
> 
> (yes, this is the one case where you have to pass CONFIG_SHELL both
> in
> the environment and on the command line; it's a bit of a
> boot-strapping
> dilemma for forcing the configure script to choose a given shell
> rather
> than its normal behavior of choosing a best shell).

Thanks for this tip.  I'll be glad to know this for the future.

> > The system /bin/sh does not report version, but can be loosely
> > identified in the OpenServer ecosystem:
> > 
> > $ ls -l /bin/sh
> > lrwxrwxrwx 1 root root 30 May 5 2008 /bin/sh ->
> > /opt/K/SCO/Unix/5.0.7Hw/bin/sh
> 
> Old, but the real question is if it is still part of a supported
> distro
> (in which case it would be really nice to work around it) or if the
> vendor has long since dropped support (in which case it's probably
> not
> worth spending time on something even the vendor doesn't care about)

Well, to my knowledge, fresh updates to the OpenServer 5 product have
not been issued in years.  Based on the ftp server, I'd say the last
maintenance rollup was issued in 2013, and a shell update was issued
in late 2014 to address a number of shell vulnerabilities.
 
> > 
> > I do have BASH built for this system:
> 
> Does bash (or any other shell flavors) come pre-installed, or does a
> pristine system come with only the broken /bin/sh?  (Again, the
> bootstrapping question, of whether it is merely sufficient to detect
> the
> /bin/sh bug and avoid it altogether because we should always be able
> to
> find something else, or whether we should try hard to work in spite
> of
> the /bin/sh bug because bash won't always be found).  Obviously, it's
> worked for you to install bash, and if nothing else, we can always
> recommend that a user with a broken /bin/sh install bash, but that
> feels
> like a fallback solution (and may not work if bash itself can't be
> self-built because it has a configure script that also tickles the
> bug).

Being new to this environment, I'm not sure what comes installed by
default when installing from media, but, on this system, that 2014
shell update was packaged in an extshells-1.4.1Ab package:

  SOLUTION: This supplement addresses the 'shellshock' security 
            vulnerabilities defined by the following CVEs:

            CVE-2014-6271
            CVE-2014-6277
            CVE-2014-6278
            CVE-2014-7169
            CVE-2014-7186
            CVE-2014-7187

            These vulnerabilities could allow a regular bash user to gain
            privileges through a crafted environment variable.

            This supplement is intended for installation on OpenServer 
releases: 
           
                  SCO OpenServer Release 5.0.7 with Maintenance Pack 5
                  OpenServer 507V

            The extshells-1.4.1Ab package contains these shells:
              GNU Bourne-Again Shell (bash) 4.3.30
              The Z-Shell (zsh) 4.2.6
              Enhanced C-Shell (tcsh) 6.14.00
              Korn Shell 93r

            Only the bash shell was updated in this package. 
            bash-4.3.30 corrects the 'shellshock' vulnerability.

I do believe that BASH is not installed by default as a number of
customer systems do not have it installed.  That said, I do
believe that when the vendor's Extended Shells are installed,
bash is available.

On this system I have the following vendor-supplied volumes along
with others probably not relevant to this issue:

  Extended Shells (ver 1.4.1Ab)
  GNU Development Tools (ver 5.0.7Kj)
  GNU Development Tools (ver 5.0.7g)
  SCO OpenServer Enterprise System (ver 5.0.7Hw)

> > $ bash --version
> > GNU bash, version 4.3.30(1)-release (i586-pc-sco3.2v5.0)
> > 
> >>> I supposed that perhaps in this case, [1845] might be a line
> >>> number.
> >>
> >> Or maybe a process id?
> > 
> > Based on using /bin/sh -x to invoke the configure script, it is
> > clear
> > that [1845] is a line number.
> > 
> > It seems clear the OpenServer 5.0.7 system /bin/sh shell is
> > inadequate in
> > some way, but it also seems clear that it is not a simple matter of
> > variable length.  If, using vim, I create my own script with
> > #!/bin/sh, I
> > can copy the ac_subst_vars assignment into it and it works just
> > fine, and,
> > besides, I can create variables with much larger values in this
> > shell.
> > 
> >    #!/bin/sh
> >    ac_subst_vars='...
> >    ...
> >    ...'
> >    echo "${ac_subst_vars}" | wc -c
> > 
> > ./test.sh
> > 19222
> 
> Oh well, that's going to make characterizing the bug even harder :(
> You've identified the size of the assignment statement, and the line
> it
> starts on; can you also identify the text near various 4k points in
> the
> generated configure file lie, to see if something syntactically
> interesting is happening if the shell treats two separate parts of a
> line in an interesting manner?  (For example, splitting 'ab=cd' into
> 'a'
> and 'b=cd' has different semantic than splitting into 'ab' and '=cd';
> and the error message about an empty variable name being an invalid
> assignment might be a case of splitting right before the =).

As to syntactically interesting constructs, I'd say not.  There is
nothing present in the value list that matches [^a-zA-Z0-9_] other
than linefeeds separating each value.  Everything looks like a plain
and simple alpha-numeric values with underscores embedded.

> > Any thought as to whether I should I just move along and recognize
> > I
> > need to use BASH in these cases, or is this an indication of
> > something to address somewhere in the tool chain that does
> > not detect this problem?
> 
> Knowing that bash is a workaround is good; so now it's a question of
> whether it's also worth trying to spend effort on making autoconf
> smarter to avoid this problem with /bin/sh for other users of your
> platform, or whether your situation is niche enough that we just let
> you live with the workaround.

In summary, it is not terribly difficult to get this bash installed
via a vendor distribution file.  It is not necessary to build from
source.  I can't really speak to how many people still might use
the platform, but I can't imagine a lot of folk attempt to build
packages on it.  I have found myself supporting an effort to conjure
up support for more secure network protocols as some significant
protocol security mitigations are about to go into effect.

Though the purist part of me would like to see a fix, I also fully
realize how archaic this system is, so perhaps this thread being on
the net may be a reasonable enough way to point other people to a
reasonable workaround without draining resources from things that
do get a bigger audience.

-- 
Kevin R. Bulgrien, Network/Software Engineer
Computer Systems Design, Inc dba Systems Design