ttyname_r on OSF/1

[Bruno Haible]

On OSF/1 5.1, the ttyname_r test fails:

test-ttyname_r.c:45: assertion failed
sh: 367584 Abort - core dumped
FAIL: test-ttyname_r

The reason is that ttyname_r stores its result in the buffer, ignoring
the buffer's size. This patch provides a workaround:


2010-12-22  Bruno Haible  <address@hidden>

        ttyname_r: Work around bug on OSF/1 5.1.
        * doc/posix-functions/ttyname_r.texi: Mention the OSF/1 bug.
        * m4/ttyname_r.m4 (gl_FUNC_TTYNAME_R): Detect the OSF/1 bug. Say "no"
        instead of "guessing no" when the OSF/1 bug or the Solaris bug is
        present.
        * lib/ttyname_r.c (ttyname_r): Update comments.

--- doc/posix-functions/ttyname_r.texi.orig     Wed Dec 22 17:06:47 2010
+++ doc/posix-functions/ttyname_r.texi  Wed Dec 22 16:50:30 2010
@@ -19,6 +19,10 @@
 This function has an incompatible declaration on some platforms:
 MacOS X 10.4, Solaris 11 2010-11 (when @code{_POSIX_PTHREAD_SEMANTICS} is not 
defined).
 @item
+This function ignores the size argument, thus overwriting memory after the
+buffer, on some platforms:
+OSF/1 5.1.
address@hidden
 This function refuses to do anything when the output buffer is less than 128
 bytes large, on some platforms:
 Solaris 11 2010-11.
--- lib/ttyname_r.c.orig        Wed Dec 22 17:06:47 2010
+++ lib/ttyname_r.c     Wed Dec 22 16:53:31 2010
@@ -32,8 +32,9 @@
   /* When ttyname_r exists, use it.  */
 #if HAVE_TTYNAME_R
   /* This code is multithread-safe.  */
-  /* On Solaris, ttyname_r always fails if buflen < 128.  So provide a buffer
-     that is large enough.  */
+  /* On Solaris, ttyname_r always fails if buflen < 128.  On OSF/1 5.1,
+     ttyname_r ignores the buffer size and assumes the buffer is large enough.
+     So provide a buffer that is large enough.  */
   char largerbuf[512];
 # if HAVE_POSIXDECL_TTYNAME_R
   int err =
--- m4/ttyname_r.m4.orig        Wed Dec 22 17:06:47 2010
+++ m4/ttyname_r.m4     Wed Dec 22 17:04:22 2010
@@ -1,4 +1,4 @@
-# ttyname_r.m4 serial 6
+# ttyname_r.m4 serial 7
 dnl Copyright (C) 2010 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -43,6 +43,8 @@
       dnl On Solaris 10, both ttyname_r functions (the one with the non-POSIX
       dnl declaration and the one with the POSIX declaration) refuse to do
       dnl anything when the output buffer is less than 128 bytes large.
+      dnl On OSF/1 5.1, ttyname_r ignores the buffer size and assumes the
+      dnl buffer is large enough.
       AC_REQUIRE([AC_CANONICAL_HOST])
       AC_CACHE_CHECK([whether ttyname_r works with small buffers],
         [gl_cv_func_ttyname_r_works],
@@ -53,6 +55,8 @@
           case "$host_os" in
                       # Guess no on Solaris.
             solaris*) gl_cv_func_ttyname_r_works="guessing no" ;;
+                      # Guess no on OSF/1.
+            osf*)     gl_cv_func_ttyname_r_works="guessing no" ;;
                       # Guess yes otherwise.
             *)        gl_cv_func_ttyname_r_works="guessing yes" ;;
           esac
@@ -70,13 +74,17 @@
 
   fd = open ("/dev/tty", O_RDONLY);
   if (fd < 0)
-    result |= 1;
+    result |= 16;
   else if (ttyname_r (fd, buf, sizeof (buf)) != 0)
-    result |= 2;
+    result |= 17;
+  else if (ttyname_r (fd, buf, 1) == 0)
+    result |= 18;
   return result;
 }]])],
             [gl_cv_func_ttyname_r_works=yes],
-            [:],
+            [case $? in
+               17 | 18) gl_cv_func_ttyname_r_works=no ;;
+             esac],
             [:])
         ])
       case "$gl_cv_func_ttyname_r_works" in