GNU bug report logs - #13075
fifo unlimited buffer size?

Previous Next

Full log

View this message in rfc822 format

From: Pádraig Brady <P <at> draigBrady.com>
To: Peng Yu <pengyu.ut <at> gmail.com>
Cc: 13075 <at> debbugs.gnu.org
Subject: bug#13075: fifo unlimited buffer size?
Date: Tue, 04 Dec 2012 12:24:35 +0000

tag 13075 + notabug
close 13075
thanks

On 12/04/2012 03:19 AM, Peng Yu wrote:
> Hi,
>
> I have the following script. When the number to the right of 'seq' is
> large (as 100000 in the example), the script will hang. But when the
> number is small (say 1000), the script can be finished correctly. I
> suspect that the problem is that there is a limit on the buffer size
> for fifo. Is it so? Is there a way to make the following script work
> no matter how large the number is? Thanks!
>
> ~/linux/test/gnu/gnu/coreutils/mkfifo/tee$ cat main2.sh
> #!/usr/bin/env bash
>
> rm -rf a b c
> mkfifo a b c
> seq 100000 | tee a > b &
> sort -k 1,1n a > c &
> join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c)
> <(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b)

So this is problematic due to `sort`.
That's special as it needs to consume all its input before
producing any output. Therefore unless the buffers connecting
the other commands in || can consume the data, there will be a deadlock.

This version doesn't block for example as
the input is being generated asynchronously for the sort command.

#!/usr/bin/env bash
rm -rf a b c
mkfifo a b c
join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c) \
<(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b) &
seq 100000 | sort -k 1,1n > c &
seq 100000 > b
wait

Obviously, if your input is expensive to generate,
then you'd be best copying to another file
and sorting that.

thanks,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.