fcntl: fix F_DUPFD_CLOEXEC behaviour on Haiku

[Bruno Haible]

On Haiku, the fcntl tests fail. Reason:

1) fcntl (oldfd, F_DUPFD_CLOEXEC, newfd)
   sets the FD_CLOEXEC flag on oldfd, not on newfd (!).
2) Additionally, if newfd < 0, it fails with errno = EMFILE,
   where POSIX wants errno = EINVAL.

This fixes it. I'm too lazy to write an autoconf test for this one;
just putting in #ifdef __HAIKU__.


2018-09-06  Bruno Haible  <address@hidden>

        fcntl: Fix F_DUPFD_CLOEXEC behaviour on Haiku.
        * lib/fcntl.c (rpl_fcntl): For F_DUPFD_CLOEXEC, don't even try the
        system fcntl.
        * doc/posix-functions/fcntl.texi: Document the issue.

diff --git a/doc/posix-functions/fcntl.texi b/doc/posix-functions/fcntl.texi
index f606ead..df89607 100644
--- a/doc/posix-functions/fcntl.texi
+++ b/doc/posix-functions/fcntl.texi
@@ -22,13 +22,18 @@ BeOS.
 Note that the gnulib replacement code is functional but not atomic.
 
 @item
+The @code{F_DUPFD_CLOEXEC} action of this function sets the
address@hidden flag on the wrong file descriptor on some platforms:
+Haiku.
+
address@hidden
 The @code{F_DUPFD} action of this function does not reject
 out-of-range targets properly on some platforms:
 AIX 7.1, Cygwin 1.5.x, Haiku.
 
 @item
 The @code{F_DUPFD} action of this function mistakenly clears
-FD_CLOEXEC on the source descriptor on some platforms:
address@hidden on the source descriptor on some platforms:
 Haiku.
 @end itemize
 
diff --git a/lib/fcntl.c b/lib/fcntl.c
index 80ff8c4..c985a7e 100644
--- a/lib/fcntl.c
+++ b/lib/fcntl.c
@@ -329,6 +329,12 @@ rpl_fcntl (int fd, int action, /* arg */...)
         result = dupfd (fd, target, O_CLOEXEC);
         break;
 #else /* HAVE_FCNTL */
+# if defined __HAIKU__
+        /* On Haiku, the system fcntl (fd, F_DUPFD_CLOEXEC, target) sets
+           the O_CLOEXEC flag on fd, not on target.  Therefore avoid the
+           system fcntl in this case.  */
+#  define have_dupfd_cloexec -1
+# else
         /* Try the system call first, if the headers claim it exists
            (that is, if GNULIB_defined_F_DUPFD_CLOEXEC is 0), since we
            may be running with a glibc that has the macro but with an
@@ -343,10 +349,10 @@ rpl_fcntl (int fd, int action, /* arg */...)
             if (0 <= result || errno != EINVAL)
               {
                 have_dupfd_cloexec = 1;
-# if REPLACE_FCHDIR
+#  if REPLACE_FCHDIR
                 if (0 <= result)
                   result = _gl_register_dup (fd, result);
-# endif
+#  endif
               }
             else
               {
@@ -357,6 +363,7 @@ rpl_fcntl (int fd, int action, /* arg */...)
               }
           }
         else
+# endif
           result = rpl_fcntl (fd, F_DUPFD, target);
         if (0 <= result && have_dupfd_cloexec == -1)
           {