Subject: bug#33965: handling of closed file descriptors

From: Bruno Haible <[email-masked]>
Date: Thu, 03 Jan 2019 20:42:28 +0100

There is a problem with the handling of closed file descriptors
in diff-3.7: As mentioned in [1], the 'new-file' test fails on HP-UX.
An investigation [2] shows that different coding techniques are needed,
depending on the desired outcome for closed file descriptors.

This patch uses a variant of (A) from [2]. Namely, the result of fcntl
can apparently be used to distinguish the substitute file descriptor
stuffed in by exec() from a regular open("/dev/null",O_RDONLY).

[1] https://lists.gnu.org/archive/html/diffutils-devel/2018-12/msg00019.html
[2] https://lists.gnu.org/archive/html/bug-gnulib/2019-01/msg00012.html
>From 288ddc7c18c022f39233ca78e78e49e02833966c Mon Sep 17 00:00:00 2001
From: Bruno Haible <[email-masked]>
Date: Thu, 3 Jan 2019 20:36:35 +0100
Subject: [PATCH] Recognize file descriptors closed by the parent process on
 HP-UX.

* src/diff.c (compare_files): Use fcntl to distinguish a file descriptor closed
by the parent parent process from a file descriptor that references /dev/null.
---
 src/diff.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/src/diff.c b/src/diff.c
index 6e8c6be..2ed3ae5 100644
--- a/src/diff.c
+++ b/src/diff.c
@@ -1172,6 +1172,13 @@ compare_files (struct comparison const *parent,
 	      cmp.file[f].desc = STDIN_FILENO;
 	      if (binary && ! isatty (STDIN_FILENO))
 		set_binary_mode (STDIN_FILENO, O_BINARY);
+#ifdef __hpux
+	      /* Recognize file descriptors closed by the parent on HP-UX.  */
+	      int flags = fcntl (STDIN_FILENO, F_GETFL, NULL);
+	      if (flags >= 0 && (flags & FD_CLOEXEC) != 0)
+		cmp.file[f].desc = ERRNO_ENCODE (EBADF);
+	      else
+#endif
 	      if (fstat (STDIN_FILENO, &cmp.file[f].stat) != 0)
 		cmp.file[f].desc = ERRNO_ENCODE (errno);
 	      else
-- 
2.7.4