bug#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

[Georges Ko]

Glenn Morris <rgm@gnu.org> writes:

> Paul Eggert wrote:
>
>> [moving this to bug-gnu-emacs, so that it gets a proper bug number]
>
> It already has one: http://debbugs.gnu.org/19191

I have been able to "fix" the problem by trial and error.

On my system, the function allocate_pty (process.c) calls the macro 
PTY_TTY_NAME_SPRINTF, generated in config.h, which is: 

  #define PTY_TTY_NAME_SPRINTF
  {
    char *ptsname (int), *ptyname;
    int grantpt_result;
    sigset_t blocked;
    sigemptyset (&blocked);
    sigaddset (&blocked, SIGCHLD);
    pthread_sigmask (SIG_BLOCK, &blocked, 0);
    grantpt_result = grantpt (fd);
    pthread_sigmask (SIG_UNBLOCK, &blocked, 0);
    if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname 
(fd)))
      {
        emacs_close (fd);
        return -1;
      }
    snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname);
  } 

It turned out that grantpt(fd) returns -1, with errno = 13 (EACCES), 
so -1 is returned by allocate_pty.

After checking EACCES of grantpt():

  The slave pseudoterminal was opened before grantpt(), or a grantpt() was 
  already issued. In either case, slave pseudoterminal permissions and 
  ownership have already been updated. If you use grantpt() to change slave 
  pseudoterminal permissions, you must issue grantpt() between the master 
  open and the first pseudoterminal open, and grantpt() can only be issued 
  once. 
  
I changed the macro to:

    if ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = 
ptsname (fd)))

and it "fixed" the problem. I'll let the experts figure out the root cause...

The diff of configure (24.5):

$ diff configure.24.5 configure.24.5.ttyfix
17105c17105
<                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), 
*ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); 
sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); 
grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if 
(grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { 
emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, \"%s\", 
ptyname); }" >>confdefs.h
---
>                 $as_echo "#define PTY_TTY_NAME_SPRINTF { char *ptsname (int), 
> *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); 
> sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); 
> grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if 
> ( /* grantpt_result == -1 || */ unlockpt (fd) == -1 || !(ptyname = ptsname 
> (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, 
> \"%s\", ptyname); }" >>confdefs.h

Georges
-- 
 Georges Ko                     gko@gko.net                      2015-05-13