I still don't understand what the answer to my original question is, i.e. how do I detect in my C code whether I've actually got some openpty implementation? As far as I can see from pty.m4, HAVE_OPENPTY is only defined if the system actually has the function, and REPLACE_OPENPTY is only defined if the system declares the function but not correctly. If the system utterly lacks openpty, but has _getpty and posix_openpt, then a replacement will be built, but no macro will be set.
It seems to me that HAVE_OPENPTY should end up set to 1 in any of these three cases, i.e. in both arms of the outermost if, provided that the required functions are found. Further, it seems there's another bug: the two arms of the outermost if should not be mutually exclusive, because if you come out of the top branch with REPLACE_OPENPTY=1 , then you still need to run the check in the bottom branch to make sure you can actually build the replacement. Otherwise, if you're on a system that declares openpty, but either non-const or without an implementation, and also lacks _getpty and/or posix_openpt, then you'll end up with REPLACE_OPENPTY=1 incorrectly.
Again, some feedback on my analysis would be helpful. I believe that if it's correct, the solution is pretty simple, so here it is, to make the foregoing more concrete:
if test $ac_cv_have_decl_openpty = yes; then
...
if test $gl_cv_func_openpty_const = yes; then
HAVE_OPENPTY=1
end
fi
fi
if test $gl_cv_func_openpty_const != yes; then
dnl We need gnulib's openpty.
dnl Prerequisites of lib/openpty.c in this case.
AC_CHECK_FUNCS([_getpty posix_openpt], [HAVE_OPENPTY=1; REPLACE_OPENPTY=1], [HAVE_OPENPTY=0; REPLACE_OPENPTY=0])
fi
if test $HAVE_OPENPTY = 1; then
AC_DEFINE([HAVE_OPENPTY], [1],
[Define to 1 if the system has the 'openpty' function.])
fi