IBM z/OS compatibility issues

[Daniel Richard G.]

Hello list,

I have been testing gnulib on IBM z/OS in recent days, and have found
a few compatibility issues (some old, some new) that I'd like to
address here.

This is intended to be a comprehensive review, so it will be a little
long; please bear with me!


BROKEN LOCALE FUNCTIONS

The configure script finds the following:

    checking for xlocale.h... no
    checking for duplocale... yes
    checking for uselocale... yes
    checking for newlocale... yes
    checking for freelocale... yes

    checking whether locale.h conforms to POSIX:2001... yes
    checking whether uselocale works... no

    checking whether duplocale(LC_GLOBAL_LOCALE) works... no

    checking whether setlocale supports the C locale... yes

The problem is, those duplocale(), newlocale(), and freelocale()
functions are not usable. Not only are they not declared in locale.h,
not only does the runtime crash if you call them, the locale_t type
isn't even defined. Here is a portion of the config.log output for the
"duplocale works" test:

    configure:37076: checking whether duplocale(LC_GLOBAL_LOCALE) works
    configure:37166: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 
-qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 
-qhaltonmsg=CCN3296  conftest.c  >&5
    ERROR CCN3275 ./conftest.c:405   Unexpected text loc encountered.
    ERROR CCN3045 ./conftest.c:405   Undeclared identifier locale_t.
    ERROR CCN3045 ./conftest.c:415   Undeclared identifier LC_GLOBAL_LOCALE.
    ERROR CCN3045 ./conftest.c:415   Undeclared identifier loc.
    CCN0793(I) Compilation failed for file ./conftest.c.  Object file not 
created.
    configure:37166: $? = 12
    configure: program exited with status 12

And the "uselocale works" test, which fails in the crashing way:

    configure:24730: checking whether uselocale works
    configure:24757: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 
-qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 
-qhaltonmsg=CCN3296  conftest.c  >&5
    configure:24757: $? = 0
    configure:24757: ./conftest
    CEE3728S The use of a function, which is not supported by this release of 
Language Environment was detected.
             From compile unit /tmp/gnulib-build/conftest.c at entry point main 
at statement 275 at compile unit offset
              +00000074 at entry offset +00000074 at address 2190A9BC.
    configure:24757: $? = 137
    configure: program exited with status 137

Currently, gnulib reads this as "Oh, the system duplocale() et al. are
broken, let me replace them." Unfortunately, this results in build
errors at the replacement function prototype, due to the missing
locale_t type:

    xlc-wrap -DHAVE_CONFIG_H -I. -I/u/username/testdir/gllib -I.. 
-DGNULIB_STRICT_CHECKING=1 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 
-qhaltonmsg=CCN3296  -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o 
hard-locale.o /u/username/testdir/gllib/hard-locale.c
    ERROR CCN3166 ./locale.h:702   Definition of function locale_t requires 
parentheses.
    ERROR CCN3276 ./locale.h:702   Syntax error: possible missing '{'?

In order to get a successful build, I have to set

    ac_cv_func_duplocale=no
    ac_cv_func_newlocale=no

before configuring. Could these functions (perhaps freelocale() too) be
treated as non-existent if there is no usable locale_t type?
Alternately, checking for their (missing) declarations should also work.


PTHREAD ENVIRONMENT

z/OS has effectively two major pthread interfaces: _OPEN_THREADS, and
_UNIX95_THREADS. The latter is the one you want to use, because its API
is compatible with other systems. (_OPEN_THREADS uses e.g. a different
signature for pthread_getspecific() that comes from a draft POSIX
specification; refer to

    
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ptgetsp.htm

for the gory details. pthread_detach() is different, too, and
pthread_cond_timedwait() can return EAGAIN.)

Additional information on z/OS feature test macros, if desired:

    
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ftms.htm

More specifically, to get a suitable pthreads interface, you need to
#define _UNIX95_THREADS and _XOPEN_SOURCE=600. (This could perhaps be
added to AC_USE_SYSTEM_EXTENSIONS for this system.)

However, one annoyance of _UNIX95_THREADS is that it does not define
PTHREAD_RWLOCK_INITIALIZER, supposedly because the SUSv3 standard (which
is what that feature test macro requests) does not specify it. However,
IBM does provide the "implementation-defined"
PTHREAD_RWLOCK_INITIALIZER_NP, which can be used in its place. See the
note here:

    
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/rp0r0i.htm

I would thus suggest adding something like the following to
lib/pthread.in.h:

    #if defined(__MVS__) && !defined(PTHREAD_RWLOCK_INITIALIZER)
    # define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
    #endif


SHELL SCRIPTING SNAFUS

In the z/OS environment, if you want to use a proper Bash shell, you
typically get the build provided by Rocket Software:

    https://www.rocketsoftware.com/product-categories/mainframe/bash-zos

The problem, however, is that shell scripting with this version of Bash
can be a little tricky, because it doesn't fully embrace EBCDIC. A brief
demonstration:

    bash-4.3$ echo $BASH_VERSION
    4.3.46(51)-release

    bash-4.3$ printf ABC | od -t x1
    0000000000    41  42  43
    0000000003

    bash-4.3$ /bin/printf ABC | od -t x1
    0000000000    C1  C2  C3
    0000000003

    bash-4.3$ echo ABC | grep ABC
    (no output)

    bash-4.3$ /bin/echo ABC | grep ABC
    ABC

    bash-4.3$ echo ABC >test.txt
    bash-4.3$ cat test.txt
    ????

    bash-4.3$ od -t x1 <test.txt
    0000000000    41  42  43  0A
    0000000004

    bash-4.3$ echo `cat test.txt`
    ABC

It's very confusing and annoying, and when I first tested gnulib using
this version of Bash, it ended up looping infinitely on running
test-atexit.sh. The loop occurred in mktempd_(), in gltests/init.sh,
because the sheer brokenness of the shell environment resulted in the
MAX_TRIES_ logic not working.

In order to get the shell to work sanely, you have to set a bevy of
environment variables discussed in

    /usr/lpp/ported/bash-4.3/share/doc/bash/4.3/README.ZOS

(I cannot find a public copy to link to, unfortunately). These are the
settings I used which allow things to work:

    _ENCODE_FILE_NEW=IBM-1047
    _ENCODE_FILE_EXISTING=IBM-1047
    _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    _BPXK_AUTOCVT=ON
    _TAG_REDIR_ERR=txt
    _TAG_REDIR_IN=txt
    _TAG_REDIR_OUT=txt

(Note: This may not be a minimal set)

While I would not recommend giving to init.sh knowledge of the above
variables, I think it would be helpful to do some basic sanity checking
(like the echo|grep invocation above) to avoid more pathological
breakage later in the script. The failure message could include a hint
to the user about what's wrong with the shell, and what needs to be done
to fix it.

Note: The Rocket Software version of Bash is not the only one that
exists on this platform. My org has an older version, apparently
compiled by us long ago, that has given me a lot fewer headaches.
Therefore, no assumption can be made about Bash on z/OS being the Rocket
version. Unfortunately, as far as I can tell, there is no good way of
uniquely identifying the Rocket version, either (as you can see, the
BASH_VERSION string has not been tweaked).


ENVIRONMENT VARIABLES

One annoyance on this platform is that the default behavior of things
can sometimes be weirdly different from other platforms, in a way which
breaks programs expecting normal Unix/POSIX behavior. Unfortunately,
IBM's response to this is often not "Let us fix that so it works like
other Unix systems," but "That's too bad. We can't change the default
behavior because mumblemumble, but we can provide an environment
variable that, if set, with cause that thing to behave in the manner
you expect."

At present, the only such variable worth mentioning here is
_EDC_SIG_DFLT, which when set to 1, causes certain default signal
handlers *not* to print out messages:

    
https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/edc_sig_dflt.htm

This is something that should at least be set in the gnulib test
environment so that test-sigpipe.sh doesn't break, but may be worth
adding to the library itself so that programs can continue to expect
"normal" signal semantics.

(There are other environment variables documented at the link above that
may be of interest, but I have not found any additional ones to be
necessary to fix issues in gnulib's test suite.)


AUTOCONF ISSUES

A few issues in this environment relate to compiler quirks, which I
intend to bring to the attention of the Autoconf folks:

* -qfloat=ieee is needed to use "normal" IEEE754 floating-point format
  (the default is IBM's proprietary "hexadecimal" format)

* -qhaltonmsg=CCN3296 is needed so that the compiler treats missing
  header files as an error instead of a warning (!)

* Some extra rigmarole is needed so that a system foo.h is not favored
  over a foo.h file in some -I<dir> location; the "xlc-wrap" script I am
  using addresses this, and is as follows:

    #!/bin/sh
    exec xlc -qnosearch "$@" -qsearch=/usr/include

* Other compiler frontends are available on z/OS: c89, c99, cc, xlC,
  xlc++, xlclang, xlclang++. Not all of them are usable, and some of
  them take a strange option syntax; generally the ones starting with
  "xl" are easier to deal with

  (Some work previously done on this can be seen in GNU Gawk's
  m4/arch.m4 file)


MISCELLANEOUS BUGS

There are a handful of additional issues that I would like to mention
here for completeness, but am taking up primarily with IBM, rather than
this list. If anyone would like me to elaborate on any of these, please
feel free to ask:

* [sys/]signal.h does not #define NSIG (nor SIGMAX)

* File descriptor passed to fdopendir() no longer behaves like a normal
  fd; in particular, dup2(fd,fd) and fcntl(fd,F_GETFL) both fail (but
  close(fd) does not), causing test-fdopendir to fail spuriously (breaks
  the "fdopendir should not close fd" assertion)

* select() on /dev/null always returns 0, even though reading or writing
  to it never blocks

* IBM XLC compiler does not support the C11 _Alignas() specifier, nor
  UTF-8 literal strings, despite ostensibly supporting C11 (other
  features are mostly there, save for the next point)

* C11 _Thread_local support is utterly broken (test-thread_local fails)

* The z/OS Make utility is severely broken: does not support VPATH, nor
  recipe lines preceded with a hyphen, nor $(VAR:=.suffix) (can change a
  suffix but not add a new one); thankfully I have GNU Make here


That is basically everything I have. I'd like to get this knowledge out
into the open, and hopefully integrated into gnulib and elsewhere, so
that less manual intervention is needed when building GNU software on
this platform. I am happy to answer questions, and help with testing any
proposed changes.


--Daniel


-- 
Daniel Richard G. || address@hidden
My ASCII-art .sig got a bad case of Times New Roman.