GNU bug report logs - #13075
fifo unlimited buffer size?

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 13075 in the body.
You can then email your comments to 13075 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Peng Yu <pengyu.ut <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: fifo unlimited buffer size?
Date: Mon, 3 Dec 2012 21:19:02 -0600

Hi,

I have the following script. When the number to the right of 'seq' is
large (as 100000 in the example), the script will hang. But when the
number is small (say 1000), the script can be finished correctly. I
suspect that the problem is that there is a limit on the buffer size
for fifo. Is it so? Is there a way to make the following script work
no matter how large the number is? Thanks!

~/linux/test/gnu/gnu/coreutils/mkfifo/tee$ cat main2.sh
#!/usr/bin/env bash

rm -rf a b c
mkfifo a b c
seq 100000 | tee a > b &
sort -k 1,1n a > c &
join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c)
<(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b)

-- 
Regards,
Peng

Message #8 received at 13075 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Peng Yu <pengyu.ut <at> gmail.com>
Cc: 13075 <at> debbugs.gnu.org
Subject: Re: bug#13075: fifo unlimited buffer size?
Date: Tue, 04 Dec 2012 12:24:35 +0000

tag 13075 + notabug
close 13075
thanks

On 12/04/2012 03:19 AM, Peng Yu wrote:
> Hi,
>
> I have the following script. When the number to the right of 'seq' is
> large (as 100000 in the example), the script will hang. But when the
> number is small (say 1000), the script can be finished correctly. I
> suspect that the problem is that there is a limit on the buffer size
> for fifo. Is it so? Is there a way to make the following script work
> no matter how large the number is? Thanks!
>
> ~/linux/test/gnu/gnu/coreutils/mkfifo/tee$ cat main2.sh
> #!/usr/bin/env bash
>
> rm -rf a b c
> mkfifo a b c
> seq 100000 | tee a > b &
> sort -k 1,1n a > c &
> join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c)
> <(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b)

So this is problematic due to `sort`.
That's special as it needs to consume all its input before
producing any output. Therefore unless the buffers connecting
the other commands in || can consume the data, there will be a deadlock.

This version doesn't block for example as
the input is being generated asynchronously for the sort command.

#!/usr/bin/env bash
rm -rf a b c
mkfifo a b c
join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c) \
<(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b) &
seq 100000 | sort -k 1,1n > c &
seq 100000 > b
wait

Obviously, if your input is expensive to generate,
then you'd be best copying to another file
and sorting that.

thanks,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.