Subject: bug#20555: closed (Re: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty")

From: [email-masked] (GNU bug Tracking System)
Date: Thu, 11 Jun 2015 23:49:02 +0000

Your bug report

#20555: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
20555: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20555
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"
From: Paul Eggert <[email-masked]>
To: Akira Shirai <[email-masked]>
Cc: "Warren, John W CIV \(US\)" <[email-masked]>, [email-masked], Georges Ko <[email-masked]>
Date: Thu, 11 Jun 2015 16:48:10 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Akira Shirai, thanks for the info about grantpt and its internal setuid program; 
that made the problem obvious.  I installed the attached patch into the Emacs 
master sources and am (perhaps optimistically) marking this bug as done.
>From 099cb9f7b8b1d00293298d947fc77b8c96120820 Mon Sep 17 00:00:00 2001
From: Paul Eggert <[email-masked]>
Date: Thu, 11 Jun 2015 16:41:36 -0700
Subject: [PATCH] Fix "not a tty" bug on Solaris 10

* configure.ac (PTY_OPEN): Define to plain 'open'
on SVR4-derived hosts, so that the O_CLOEXEC flag isn't set.
* src/process.c (allocate_pty): Set the O_CLOEXEC flag after
calling PTY_TTY_NAME_SPRINTF, for the benefit of SVR4-derived
hosts that call grantpt which does its work via a setuid subcommand
(Bug#19191, Bug#19927, Bug#20555, Bug#20686).
Also, set O_CLOEXEC even if PTY_OPEN is not defined, since it
seems relevant in that case too.
---
 configure.ac  |  5 ++++-
 src/process.c | 20 +++++++++++---------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/configure.ac b/configure.ac
index 9c6a74a..070b061 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4397,14 +4397,17 @@ case $opsys in
     ;;
 
   sol2* )
-    dnl On SysVr4, grantpt(3) forks a subprocess, so keep sigchld_handler()
+    dnl On SysVr4, grantpt(3) forks a subprocess, so do not use
+    dnl O_CLOEXEC when opening the pty, and keep the SIGCHLD handler
     dnl from intercepting that death.  If any child but grantpt's should die
     dnl within, it should be caught after sigrelse(2).
+    AC_DEFINE(PTY_OPEN, [fd = open (pty_name, O_RDWR | O_NONBLOCK)])
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [{ char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1 || unlockpt (fd) == -1 || !(ptyname = ptsname (fd))) { emacs_close (fd); return -1; } snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname); }])
     ;;
 
   unixware )
     dnl Comments are as per sol2*.
+    AC_DEFINE(PTY_OPEN, [fd = open (pty_name, O_RDWR | O_NONBLOCK)])
     AC_DEFINE(PTY_TTY_NAME_SPRINTF, [{ char *ptsname (int), *ptyname; int grantpt_result; sigset_t blocked; sigemptyset (&blocked); sigaddset (&blocked, SIGCHLD); pthread_sigmask (SIG_BLOCK, &blocked, 0); grantpt_result = grantpt (fd); pthread_sigmask (SIG_UNBLOCK, &blocked, 0); if (grantpt_result == -1) fatal("could not grant slave pty"); if (unlockpt(fd) == -1) fatal("could not unlock slave pty"); if (!(ptyname = ptsname(fd))) fatal ("could not enable slave pty"); snprintf (pty_name, PTY_NAME_SIZE, "%s", ptyname); }])
     ;;
 esac
diff --git a/src/process.c b/src/process.c
index 17fe708..b4f979f 100644
--- a/src/process.c
+++ b/src/process.c
@@ -658,22 +658,24 @@ allocate_pty (char pty_name[PTY_NAME_SIZE])
 
 	if (fd >= 0)
 	  {
-#ifdef PTY_OPEN
+#ifdef PTY_TTY_NAME_SPRINTF
+	    PTY_TTY_NAME_SPRINTF
+#else
+	    sprintf (pty_name, "/dev/tty%c%x", c, i);
+#endif /* no PTY_TTY_NAME_SPRINTF */
+
 	    /* Set FD's close-on-exec flag.  This is needed even if
 	       PT_OPEN calls posix_openpt with O_CLOEXEC, since POSIX
 	       doesn't require support for that combination.
+	       Do this after PTY_TTY_NAME_SPRINTF, which on some platforms
+	       doesn't work if the close-on-exec flag is set (Bug#20555).
 	       Multithreaded platforms where posix_openpt ignores
 	       O_CLOEXEC (or where PTY_OPEN doesn't call posix_openpt)
 	       have a race condition between the PTY_OPEN and here.  */
 	    fcntl (fd, F_SETFD, FD_CLOEXEC);
-#endif
-	    /* Check to make certain that both sides are available
-	       this avoids a nasty yet stupid bug in rlogins.  */
-#ifdef PTY_TTY_NAME_SPRINTF
-	    PTY_TTY_NAME_SPRINTF
-#else
-	    sprintf (pty_name, "/dev/tty%c%x", c, i);
-#endif /* no PTY_TTY_NAME_SPRINTF */
+
+	    /* Check to make certain that both sides are available.
+	       This avoids a nasty yet stupid bug in rlogins.  */
 	    if (faccessat (AT_FDCWD, pty_name, R_OK | W_OK, AT_EACCESS) != 0)
 	      {
 		emacs_close (fd);
-- 
2.1.0

Subject: Re: Emacs 24.2 vs 24.4 on Solaris: M-x shell and "tty => not a tty"
From: Paul Eggert <[email-masked]>
To: Georges Ko <[email-masked]>, Emacs bug reports and feature requests <[email-masked]>
Date: Mon, 11 May 2015 23:08:34 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

[moving this to bug-gnu-emacs, so that it gets a proper bug number]

Georges Ko wrote:
> What were the changes done on Emacs 24.4 regarding subshells (or
> processes), compared to version 24.2?

Quite a few, I suspect.  You can use 'diff' to find out the details....

> When compiled in the same environment, "tty" in a subshell
> (M-x shell) has different outputs:
>
>    - in Emacs 24.2, "tty" outputs the tty,
>    - in Emacs 24.4, "tty" outputs "not a tty".
>
> That is, if I run the command below to spawn a shell:
>
>    (apply 'start-process "shell" "*shell*" "/bin/ksh" '("-i"))
>
>    - in Emacs 24.2, "ps" shows the shell has a normal tty,
>    - in Emacs 24.4, "ps" shows the shell has a tty with "?".

I don't observe either problem when running Emacs 24.5 under Solaris 11 x86-64 
compiled with the bundled GCC (4.5.2).  However, I do observe them when running 
Emacs 24.5 under Solaris 10 sparc compiled with Sun C 5.13.

> I couldn't compile 24.3 due to some errors, so I don't know its behavior.

Emacs 24.3 is bundled within Solaris 11.2, for what it's worth.

If you're still running into problems, I suggest replying with details of how 
you built Emacs (version of OS and compiler, 'configure' arguments, log from 
'configure' and 'make', etc.).

PS.  This looks a bit like Bug#9150 <http://bugs.gnu.org/9150>, but that bug was 
in an older Emacs, which probably means it's independent.