GNU bug report logs - #6900
mktemp: want option to make a fifo

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 6900 in the body.
You can then email your comments to 6900 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: bug-coreutils <at> gnu.org
Subject: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 08:22:27 -0700

mktemp: Please add an option which creates a fifo
instead of a file or a directory.  "--fifo" seems appropriate,
although the abbreviation "-f" might be confused as a request
for an ordinary file; and "-p" as in "--pipe" is already taken
for "prefix".

-- 

Message #8 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 10:40:00 -0600

On 08/23/2010 09:22 AM, John Reiser wrote:
> mktemp: Please add an option which creates a fifo
> instead of a file or a directory.  "--fifo" seems appropriate,
> although the abbreviation "-f" might be confused as a request
> for an ordinary file; and "-p" as in "--pipe" is already taken
> for "prefix".

Thanks for the report.  However, I'm inclined to mark this bug as a 
duplicate of 6330, for the reasons already documented in this long thread:
http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00013.html

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message #11 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 10:34:40 -0700

On 08/23/2010 09:40 AM, Eric Blake wrote:
> On 08/23/2010 09:22 AM, John Reiser wrote:
>> mktemp: Please add an option which creates a fifo
>>   [snip]
> 
> Thanks for the report.  However, I'm inclined to mark this bug as a
> duplicate of 6330, for the reasons already documented in this long thread:
> http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00013.html

The suggested alternative (use "mktemp -d" to create a uniquely-named
directory, then use mkfifo of an arbitrary fixed name within that directory)
satisfies only some of the desired properties.  Yes, it's a fifo with
a safely-created unique name.  However, removing that fifo leaves behind
the directory, and the directory also occupies its own space [often
a few kilobytes] even during the lifetime of the fifo.

It is somewhat ironic that Mr. Blake's .sig promotes "Libvirt virtual-
ization library", because the alternative suggestion is not a
faithful virtualization of what "mktemp --fifo" should be.

-- 

Message #14 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 12:00:50 -0600

On 08/23/2010 11:34 AM, John Reiser wrote:
> On 08/23/2010 09:40 AM, Eric Blake wrote:
>> On 08/23/2010 09:22 AM, John Reiser wrote:
>>> mktemp: Please add an option which creates a fifo
>>>    [snip]
>>
>> Thanks for the report.  However, I'm inclined to mark this bug as a
>> duplicate of 6330, for the reasons already documented in this long thread:
>> http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00013.html
>
> The suggested alternative (use "mktemp -d" to create a uniquely-named
> directory, then use mkfifo of an arbitrary fixed name within that directory)
> satisfies only some of the desired properties.  Yes, it's a fifo with
> a safely-created unique name.  However, removing that fifo leaves behind
> the directory, and the directory also occupies its own space [often
> a few kilobytes] even during the lifetime of the fifo.

If we add 'mktemp --fifo', then where do we stop?  What about creating a 
uniquely named symlink?  Or a new shared memory object?

http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00033.html

And did you not read in that thread about the arguments that recursively 
moving an entire temporary directory hierarchy with 'rm -rf dir' is just 
as short in shell code as 'rm -f fifo'?  A couple kilobytes of disk 
space for a temporary directory given today's storage technologies is 
not a very convincing argument.

The whole argument of Unix is that each tool does one thing well, so 
that the combination of tools can do everything.  mktemp should NOT be 
bloated just to reproduce the functionality of mkfifo or of ln, when 
those tools can already be paired nicely.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message #17 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 13:40:34 -0700

On 08/23/2010 11:00 AM, Eric Blake wrote:
  [snip]
> If we add 'mktemp --fifo', then where do we stop?  What about creating a
> uniquely named symlink?  Or a new shared memory object?

There are several possible answers to "Where do we stop?".

1) Stop at current code [no changes.]

2) Stop when user requests stop.

3) Add --symlink and --hardlink.  Each requires another string argument.
   Limiting the number of string arguments is a defensible argument:
   it is based on an extrinsic property.

4) Add --block and --character device nodes.  Each requires two
   non-negative integer arguments, and mktemp should check this
   in order to provide good Usability.

5) Stop when all file types of S_IFMT (<stat.h>) have been covered.
   This would add S_IFSOCK, which is gnarly indeed.

6) Stop after shmem, which is not even a file type.

I favor drawing the line between 2) and 3), which means adding --fifo
and not the others.  Two independent user requests [one of which
includes code!] within a few months show a definite need.

> http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00033.html
> 
> And did you not read in that thread about the arguments that recursively
> moving an entire temporary directory hierarchy with 'rm -rf dir' is just
> as short in shell code as 'rm -f fifo'?

The user of such a fifo must be taught that one unlink() or "rm -f fifo"
does recover all the resources.  That is a significant reduction in
*each* of the five software metric dimensions: Functionality, Usability,
Performance, Maintenance, and Supportability.  The hard task is finding
all the places that [should] remove objects from the filesystem, and
teaching them to remove the directory of such a fifo, too.  For shell,
a dirname() must be applied.  For C code, then the code must figure out
[or be told] that the name is special, parse the name, and perform another
unlink().  This is not trivial.

>   A couple kilobytes of disk
> space for a temporary directory given today's storage technologies is
> not a very convincing argument.

One of the places where I want such fifos [many hundreds of them]
is in a filesystem that resides on a much smaller flash memory device.
Occupying a megabyte more than necessary can be fatal.

> The whole argument of Unix is that each tool does one thing well, so
> that the combination of tools can do everything.  mktemp should NOT be
> bloated just to reproduce the functionality of mkfifo or of ln, when
> those tools can already be paired nicely.

It's not nice to get a compound object (directory plus fifo) when a
singleton is expected, and when the compound object cannot be treated
identically to the singleton.  The properties are different,
not "reproduced"; and that makes it not "bloat."

-- 

Message #20 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 14:22:38 -0700

On 08/23/2010 01:40 PM, John Reiser wrote:
> Two independent user requests [one of which
> includes code!] within a few months show a definite need.

The two requests would have been more convincing if they demonstrated
a definite need.  Neither did: they both just said the equivalent
of "we need a new option".

Ordinarily, plain "|" does a good job of creating temporary fifos
with a minimum of fuss.  Perhaps there are exceptional cases
where plain "|" doesn't suffice, but no examples have been given.
We need these examples in order to judge how important the
need is, in order to balance this need against the other need
of keeping coreutils simple and general.

Message #23 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 15:49:45 -0700

On 08/23/2010 02:22 PM, Paul Eggert wrote:
> On 08/23/2010 01:40 PM, John Reiser wrote:
>> Two independent user requests [one of which
>> includes code!] within a few months show a definite need.
> 
> The two requests would have been more convincing if they demonstrated
> a definite need.  Neither did: they both just said the equivalent
> of "we need a new option".

Consider a pipeline with three logical stages:   A | B1 | C
where B1 is a shell script.  At some point B1 notices that one
parallel process B2 becomes desirable, such that B2 also pipes
into the same C as B1, and output from a *subset* of the multiple
sequential subprocesses of B1 pipes into the same B2 [view in
constant-width font]:
A |  B1   | C
       \  /
        B2
where both slashes also indicate fifos.

The fifo from B2 to C can be the same fifo as from B1 to C,
just by forking B1 and inheriting stdout.  However the fifo
from B1 to B2 cannot be an anonymous pipe, but must be named
because some of the subprocesses of B1 continue to pipe their
output directly into C.  It's the switching back and forth
between "stdout goes directly to C" and "stdout goes to B2
first, and then to C" that requires a named fifo.

Some subprocesses of B1 will direct their output into B2 by
shell redirection "> $fifo_B1_B2".  And yes, I know that B1
must hold open the write end of the fifo; I use:
   sleep 999000  > $fifo_B1_B2 &

Finally, the $fifo_B1_B2 should be as invisible as possible
when viewed from outside the A|B1|C pipeline.  All together,
that's why I want "mktemp --fifo".

[Process C is 'cpio'.  Process A is 'find'.  Process B1 is
a filter which is mostly pass-through but sometimes uncovers
embedded filenames that must come along for the ride.
Process B2 has large setup cost, but is 100% parallelizable
with B1.  Filenames are written singly, and are small
enough to receive atomic passage through the pipes.]

-- 

Message #26 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 16:55:58 -0600

On 08/23/2010 04:49 PM, John Reiser wrote:
> On 08/23/2010 02:22 PM, Paul Eggert wrote:
>> On 08/23/2010 01:40 PM, John Reiser wrote:
>>> Two independent user requests [one of which
>>> includes code!] within a few months show a definite need.
>>
>> The two requests would have been more convincing if they demonstrated
>> a definite need.  Neither did: they both just said the equivalent
>> of "we need a new option".
>
> Consider a pipeline with three logical stages:   A | B1 | C
> where B1 is a shell script.  At some point B1 notices that one
> parallel process B2 becomes desirable, such that B2 also pipes
> into the same C as B1, and output from a *subset* of the multiple
> sequential subprocesses of B1 pipes into the same B2 [view in
> constant-width font]:
> A |  B1   | C
>         \  /
>          B2
> where both slashes also indicate fifos.

Thanks for the example scenario.  Next, could you please show the shell 
code that you are currently using, along with a version with your 
proposed 'mktemp --fifo', to show the difference in usage paradigms 
between the two approaches?  We may be able to help you efficiently 
re-write your existing code into an equally manageable but more portable 
solution using existing tools, or it may help to convince us that the 
shorthand is truly useful.  But without seeing actual usage examples, I 
can't say.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

Message #29 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eric Blake <eblake <at> redhat.com>
Cc: John Reiser <jreiser <at> bitwagon.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Mon, 23 Aug 2010 16:49:59 -0700

On 08/23/2010 03:55 PM, Eric Blake wrote:
> We may be able to help you efficiently re-write your existing code into
> an equally manageable but more portable solution using existing tools,
> or it may help to convince us that the shorthand is truly useful.

I agree that a full example would be helpful.  Here's a quick sketch
of a shell script that solves the stated problem without using a
named fifo.  Without the full example it's hard to say whether
this approach would work well in the existing code.

#! /bin/sh
nl='
'
a() { for i in 0 1 2 3 4 5; do echo B$i; echo C$i; done; }
b1() {
  start_b2=$nl
  while read command
  do
    case $command in
    (C*) printf '%s\n' "$command" >&3;;
    (B*) printf '%s\n' "$start_b2$command"; start_b2=;;
    esac
  done
}
maybe_b2() { read line && sed 's/^./D/'; }
c() { sed 's/^/E/'; }

a | (b1 | maybe_b2) 3>&1 | c

Message #32 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 10:42:08 -0700

> The two requests would have been more convincing if they demonstrated
> a definite need.

Another case: "pipe" stderr into a separate pipeline from stdout.
This is useful for stderr as a status channel.

    fifo_stderr1=$(mktemp --fifo stderr.XXXXXX)
    fifo_stderr2=$(mktemp --fifo stderr.XXXXXX)
    listener1 < $fifo_stderr1 &
    listener2 < $fifo_stderr2 &
    cmd1  2> $fifo_stderr1  |  cmd2  2> $fifo_stderr2

Another case: Use file descriptor 3 as a command and control channel.
Output has two default file descriptors (stdout and stderr), why not
input (stdin and cmdin)?  This is especially helpful for repairing a
nest of processes that are connected by pipes.

   fifo_cmdin1=$(mktemp --fifo cmdin.XXXXXX); sleep 999000 > $fifo_cmdin1 &
   fifo_cmdin2=$(mktemp --fifo cmdin.XXXXXX); sleep 999000 > $fifo_cmdin2 &
   cmd1  3< $fifo_cmdin1  |  cmd2  3< $fifo_cmdin2  &
   echo quit  > $fifo_cmdin1

-- 

Message #35 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 11:03:17 -0700

On 08/23/2010 04:49 PM, Paul Eggert wrote:
> Here's a quick sketch
> of a shell script that solves the stated problem without using a
> named fifo.

  [snip]
> maybe_b2() { read line && sed 's/^./D/'; }

> a | (b1 | maybe_b2) 3>&1 | c

Thank you for reminding me about using "3>&1" for "fan-in" multiplexing.
This can work for B2, B3, ..., B8 but cannot handle more because shell
is allowed to balk at redirecting fd greater than 9.  It can be limited
further if any of the Bi already use other low file descriptors.  I also
prefer to think about the situation as "B1 is doing fan-out multiplexing"
rather than "C is doing fan-in multiplexing", because the "C" end is
more "constant" and the "B" end is more "variable".

I dislike the part "(b1 | maybe_b2)" because in practice it tends to be
fragile.  When things go wrong it is hard to diagnose and recover.
Also, I do not see how to extend it for the case of more than one
conditional process whose startup should be delayed (B3, B4, ...),
such as the 'sed' in "read line && sed ...".

-- 

Message #38 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 11:32:49 -0700

On 08/24/2010 10:42 AM, John Reiser wrote:
> Another case: "pipe" stderr into a separate pipeline from stdout.
> This is useful for stderr as a status channel.
>
>      fifo_stderr1=$(mktemp --fifo stderr.XXXXXX)
>      fifo_stderr2=$(mktemp --fifo stderr.XXXXXX)
>      listener1<  $fifo_stderr1&
>      listener2<  $fifo_stderr2&
>      cmd1  2>  $fifo_stderr1  |  cmd2  2>  $fifo_stderr2

For this example, it's simpler (and does not use any more processes)
to follow Eric's suggestion to use mktemp -d to create a directory to
hold the named fifos, to put the named fifos in that directory, and
then to clean up with 'rm -r', with something like this:

  dir=$(mktemp -d dir.XXXXXX)
  mkfifo $dir/1 $dir/2
  listener1 < $dir/1
  listener2 < $dir/2
  cmd1 2> $dir/1 | cmd2 2> $dir/2

This mktemp -d version is easier to clean up after, as 'rm -fr $dir'
is easier to understand and read than 'rm -f $fifo_stderr1
$fifo_stderr1'.  The simplicity of cleanup becomes a greater advantage
as the number of named fifos (and other temporaries) grows.

Better yet, do not use named fifos, since pipes suffice:

  (cmd1 2>&1 >&3 | listener1) 3>&1 |
  (cmd2 2>&1 >&3 | listener2) 3>&1

Pipes are simpler and more robust, as they do not need 'rm'
afterwards, and you don't need to worry about what to do when mktemp
or mkfifo fails.

> Another case: Use file descriptor 3 as a command and control channel.
> Output has two default file descriptors (stdout and stderr), why not
> input (stdin and cmdin)?  This is especially helpful for repairing a
> nest of processes that are connected by pipes.
>
>     fifo_cmdin1=$(mktemp --fifo cmdin.XXXXXX); sleep 999000> 
$fifo_cmdin1&
>     fifo_cmdin2=$(mktemp --fifo cmdin.XXXXXX); sleep 999000> 
$fifo_cmdin2&
>     cmd1  3<  $fifo_cmdin1  |  cmd2  3<  $fifo_cmdin2&
>     echo quit>  $fifo_cmdin1

Again, once things get this complicated, it doesn't cost any more
processes to use a temporary directory, with something like this:

  dir=$(mktemp -d dir.XXXXXX)
  mkfifo $dir/1 $dir/2
  listener1 < $dir/1
  listener2 < $dir/2
  cmd1 2> $dir/1 | cmd2 2> $dir/2

This example can also be simplified and made more robust by using
pipes instead of named fifos, as follows:

  (echo quit 1 | cmd1 3<&0 <&4) 4<&0 |
  (:           | cmd2 3<&0 <&4) 4<&0

Message #41 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 11:39:08 -0700

Message #44 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 12:01:12 -0700

On 08/24/2010 11:03 AM, John Reiser wrote:
> This can work for B2, B3, ..., B8 but cannot handle more because shell
> is allowed to balk at redirecting fd greater than 9.

That is not a problem in practice.  The limit of 10 descriptors does
not mean that one can't create more than 10 pipes in one's shell
script.  All it means is that one can't have more than 10 useful pipes
active at any given position in one's shell script.  I've never had
the need to have more than 10 pipes connected to a component, and I
imagine that any component that needs more than 10 pipes connected to
it should be simplified anyway.

> I also prefer to think about the situation as "B1 is doing fan-out
> multiplexing" rather than "C is doing fan-in multiplexing", because
> the "C" end is more "constant" and the "B" end is more "variable".

OK, but that sort of high-level thinking is one level of abstraction
above named fifos and pipes, and in this case one can implement it in
terms of either named fifos or pipes.  Admittedly one needs to get
used to the implementation strategy.

> I dislike the part "(b1 | maybe_b2)" because in practice it tends to
> be fragile.  When things go wrong it is hard to diagnose and
> recover.

True, but it's hard to diagnose named fifos as well.  I can use
"tee" to debug either sort of shell script, and don't find it any
harder to debug the piped version.  I'm not sure what is meant by
"recover".

> Also, I do not see how to extend it for the case of more than one
> conditional process whose startup should be delayed (B3, B4, ...),
> such as the 'sed' in "read line&&  sed ...".

I don't quite follow the question, but if we're talking about
multiple file descriptors for b1, each treated like the pipe to b2,
surely this can be done with something like this:

  a | ((b1 3>&1 >&5 | maybe_b3) 4>&1 >&5 | maybe_b4) 5>&1 | c

Here b1's file descriptor 3 goes to b3, and its file descriptor 4 goes
to b4.

Message #47 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 12:17:36 -0700

On 08/24/2010 11:32 AM, Paul Eggert wrote:
  [snip]
> Again, once things get this complicated, it doesn't cost any more
> processes to use a temporary directory, with something like this:
> 
>   dir=$(mktemp -d dir.XXXXXX)
>   mkfifo $dir/1 $dir/2
>   listener1 < $dir/1
>   listener2 < $dir/2
>   cmd1 2> $dir/1 | cmd2 2> $dir/2
> 
> This example can also be simplified and made more robust by using
> pipes instead of named fifos, as follows:
> 
>   (echo quit 1 | cmd1 3<&0 <&4) 4<&0 |
>   (:           | cmd2 3<&0 <&4) 4<&0

That simplification loses functionality because it "binds" the cmdin
channels to a specific process or pipeline.  I gave the example
"echo quit > $fifo_cmdin1" only to emphasize that the command-
and-control inputs are separate from the "main" dataflow which goes
through the regular pipes.  When I want to control a pipeline or
nest of processes interactively, then I don't know in advance
which process, which commands, or in which sequence to give them.
Using fifos allows for arbitrary flexibility of "fanout multiplexing"
my interactive commands.  I choose which fifo for each command,
at the time that the I give the command.  That cannot be done
with fixed anonymous pipes.

-- 

Message #50 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 13:07:53 -0700

On 08/24/2010 12:17 PM, John Reiser wrote:
> Using fifos allows for arbitrary flexibility of "fanout multiplexing"
> my interactive commands.

I'm not sure what is meant by "arbitrary flexibility", but of course a
shell script that is using pipes can also decide dynamically which
processes to start up and connect to the control channel, and what
output these processes will generate, and (when there are multiple
control channels) which control channels each such process will
connect to.

I take your point that named fifos provide a different style for
creating networks of processes, a style that may be more convenient in
many cases.  However, for small examples pipes seem quite adequate,
and for large examples mktemp -d seems to simplify the code.  Also,
mktemp -d would simplify administration of large scripts: I'd far
rather see a small /tmp, one subdirectory per running script, than a
/tmp filled with countless fifos and other temporary files.

The bottom line is that so far we haven't seen a real example
exhibiting a need for mktemp --fifo that can't be satisfied just as
well with existing tools.

Message #53 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Tue, 24 Aug 2010 13:30:47 -0700

>   (echo quit 1 | cmd1 3<&0 <&4) 4<&0 |
>   (:           | cmd2 3<&0 <&4) 4<&0

When adopting this style, then remember to consider the possibility of
closing the "extra" file descritpor (fd 4 here) to reduce the number
of open files.  If there are no 'trap', then also remember to consider
the possibility of using 'exec' to reduce the number of processes created
['fork' or 'clone' system calls] and their lifetimes:

   (exec echo quit 1 | exec cmd1 3<&0 <&4 4<&-) 4<&0 |
   (     :           | exec cmd2 3<&0 <&4 4<&-) 4<&0

-- 

Message #56 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: John Reiser <jreiser <at> bitwagon.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: pitfalls of recoding named fifo as pipe
Date: Wed, 25 Aug 2010 07:14:38 -0700

On 08/24/2010 11:32 AM, Paul Eggert wrote:
>   dir=$(mktemp -d dir.XXXXXX)
>   mkfifo $dir/1 $dir/2
>   listener1 < $dir/1
>   listener2 < $dir/2
>   cmd1 2> $dir/1 | cmd2 2> $dir/2

> Better yet, do not use named fifos, since pipes suffice:
> 
>   (cmd1 2>&1 >&3 | listener1) 3>&1 |
>   (cmd2 2>&1 >&3 | listener2) 3>&1

That is not equivalent!  stdout from listener1 gets piped into
cmd2, instead of going to the common stdout of all processes
other than cmd1.  Fixing this using only anonymous pipes
is cumbersome because the additional file descriptor must be
[should be] redirected by all four processes.

Also, file descriptor 3 is left open in all processes, which
hogs system resources and is less safe.

Therefore something like this is required:
4>&1  (
  3>&1       ( 1>&3  2>&1  3>&-  4>&-  cmd1
             | 1>&4        3>&-  4>&-  listener1
             )
|
  3>&1  4>&- ( 1>&3  2>&1  3>&-        cmd2
             |             3>&-        listener2
             )
)
where the formatting emphasizes prefix actions and common elements.

Using the named fifos is much more readable and understandable.
The cost is explicit cleanup, including the transfer of the onus
for recovery at resource exhaustion (fd not available, space
not available in filesystem) from the shell to the user.

-- 

Message #59 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> CS.UCLA.EDU>
To: John Reiser <jreiser <at> bitwagon.com>
Cc: Eric Blake <eblake <at> redhat.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: pitfalls of recoding named fifo as pipe
Date: Wed, 25 Aug 2010 13:40:35 -0700

On 08/25/2010 07:14 AM, John Reiser wrote:
> That is not equivalent!  stdout from listener1 gets piped into
> cmd2, instead of going to the common stdout of all processes
> other than cmd1.

Yes, you're right.  I had assumed that the listeners did not generate
standard output.  If they do, it should have been this:

  ((cmd1 2>&1 >&3 | listener1 >&4) 3>&1 |
   (cmd2 2>&1 >&4 | listener2    )
  ) 4>&1

> Also, file descriptor 3 is left open in all processes, which
> hogs system resources and is less safe.

Those resource and safety issues are typically negligible.
If one wanted to be extremely cautious, one could write this:

  ((cmd1 2>&1 >&3 3>&- 4>&- | listener1 >&4 3>&- 4>&-) 3>&1 |
   (cmd2 2>&1 >&4      4>&- | listener2          4>&-)
  ) 4>&1

(This is a shorter and POSIX-conforming way of writing the variant
that you sent, plus it fixes a couple of bugs in the way that variant
redirected file descriptor 2.)  But almost nobody would or should
bother with this level of complication.

> Using the named fifos is much more readable and understandable.

Yes, as I said, once things get sufficiently complicated, it's quite
possible for named fifos to be easier to use.  It's still not clear to
me that this is the case here, as even the overly-complicated piped
version is a tad simpler than the named-fifo version.  But the point
remains that, once a script is complicated enough to need a lot of
named fifos, it should be using mktemp -d anyway.

Message #62 received at 6900 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Eric Blake <eblake <at> redhat.com>
Cc: John Reiser <jreiser <at> bitwagon.com>, 6900 <at> debbugs.gnu.org
Subject: Re: bug#6900: mktemp: want option to make a fifo
Date: Sun, 07 Aug 2011 18:08:24 +0200

forcemerge 6900 6330
tags 6900 notabug
close 6900
thanks

Eric Blake wrote:
> On 08/23/2010 09:22 AM, John Reiser wrote:
>> mktemp: Please add an option which creates a fifo
>> instead of a file or a directory.  "--fifo" seems appropriate,
>> although the abbreviation "-f" might be confused as a request
>> for an ordinary file; and "-p" as in "--pipe" is already taken
>> for "prefix".
>
> Thanks for the report.  However, I'm inclined to mark this bug as a
> duplicate of 6330, for the reasons already documented in this long
> thread:
> http://lists.gnu.org/archive/html/bug-coreutils/2010-06/msg00013.html

I've just merged them and closed both.
It's clear that the proposed change is neithehr required nor justified.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.