GNU bug report logs - #13089
Wish: split every n'th into n pipes

Previous Next

Full log

Message #8 received at 13089 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Ole Tange <tange <at> gnu.org>
Cc: 13089 <at> debbugs.gnu.org
Subject: Re: bug#13089: Wish: split every n'th into n pipes
Date: Wed, 05 Dec 2012 18:52:29 +0000

tag 13089 + notabug
close 13089

On 12/05/2012 02:11 PM, Ole Tange wrote:
> I often have data that can be processed in parallel.
>
> It would be great if split --filter could look at every n'th line
> instead of chunking into n chunks:
>
>    cat bigfile | split --every-nth -n 8 --filter "grep foo"
>
> The above should start 8 greps and give each a line in round robin manner.
>
> Ideally it should be possible to do so non-blocking so if some lines
> take longer for one instance of grep, then the rest of the greps are
> not blocked.

So that's mostly supported already (notice the r/ below):

$ seq 8000 | split -n r/8 --filter='wc -l' | uniq -c
      8 1000

The concurrency is achieved through standard I/O buffers
between split and the filters (note also the -u split option).

I'm not sure non blocking I/O would be of much benefit,
since the filters will be the same, and if we did that,
then we'd have to worry about internal buffering in split.
We had a similar question about tee, yesterday, and I
think the answer is the same here, that the complexity
doesn't seem warranted for such edge cases.

thanks,
Pádraig.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.